[Hexagon] Clear stale kill flags in HexagonOptAddrMode::updateAddBases (#240512)
processAddBases chains a group of A2_addi instructions so that a later
one computes its address off an earlier one's definition register
instead of the original base. If that definition register had already
been killed by an intervening instruction, the rewritten add ends up
reading a dead register, tripping the machine verifier with "Using an
undefined physical register".
Clear the kill flags on the definition register between the first
reached add and the rewritten add so its live range extends to the new
use.
Fixes #240245.
[CIR] Let llvmOp lowering forward properties to the LLVM op
The lowering patterns generated from `llvmOp` build the LLVM operation with
default properties, so a CIR op whose flags have an LLVM counterpart had to
be lowered by hand. `llvmOpProps` lists the CIR attributes and properties to
forward; the emitter derives the CIR getter and the LLVM setter from the
names, and a renamed LLVM property is given as a string, as in
`(ins "is_zero_poison":$poison_zero)`. Values go through the
convertToLLVMProperty overload for their C++ type. The emitter rejects
entries that are not attributes or properties of the op.
cir.add, cir.sub and cir.mul (overflow flags), the floating-point vector
reductions (fastmath flags), cir.clz, cir.ctz, cir.abs and cir.is_fp_class
switch to it, and cir.popcount, cir.vec.extract and cir.vec.insert use plain
`llvmOp`. Their hand-written patterns and lowerFPVectorReduction go away.
Assisted-by: Claude Code (Claude Opus 5.5).
[llubi] Add support for llvm.matrix.column.major.store. (#237646)
Add support for llvm.matrix.column.major.store: column C of the matrix
is stored as a <Rows x ElemTy> vector at offset C * Stride elements from
the base pointer.
PR: https://github.com/llvm/llvm-project/pull/237646
[CIR] Match classic codegen for saturating fixed-point negation and padded conversions
Three CIR fixed-point paths diverged from classic codegen. Unary minus on a
saturating fixed-point type emitted a plain subtraction instead of a
saturating 0 - x. With -fpadding-on-unsigned-fixed-point, float to unsigned
fixed-point conversion produced the fptosi.sat result in an unsigned CIR
type, so its clamp to 0 compared as unsigned and never fired; it now
converts and clamps in a signed type, like llvm::FixedPointBuilder, and
casts to the unsigned type afterwards. A saturating fixed-point conversion
that kept the width but changed signedness returned a value of the source
CIR type, which broke the cir.cmp verifier and needed a pointer bitcast to
store; it now casts to the destination type.
Assisted-by: Claude Code (Claude Opus 5.5).
[CIR][OpenMP] Add support for 'distribute parallel for' with 'target teams'
Adds support for handling the full '[target] teams distribute parallel for'.
Introduces setting the omp.composite attibute for the 'distribute parallel for'
composite construct.
Assisted-by: Cursor / Claude Sonnet 5 High
[CIR][OpenMP] Add initial support for distribute
This patch adds initial support for the OpenMP 'distribute' directive. Basic
code generation for the standalone and combined cases is introduced. Support
for clauses will be added in the future. The composite 'teams distribute
parallel for' form will be added in a subsequent patch.
Assisted-by: Cursor / Claude Sonnet 5 High
[ConstraintElim] A signed sum of operands with opposite signs cannot wrap
If one operand is known non-negative and the other non-positive, the signed
sum is between them, so it cannot overflow.
https://alive2.llvm.org/ce/z/ZF5Yqr
[ConstraintElim] Propagate the sign across an icmp samesign (#235158)
If one operand of `icmp samesign` is known (non-)negative, so is the
other. Add that as a fact in the signed system.
https://alive2.llvm.org/ce/z/xUgGRM
[CIR][OpenMP] Add support for omp teams
This patch adds support for the OpenMP 'teams' directive, both standalone
and as the combined `target teams` construct. isCombinableLeaf now also
recognizes teams, so an explicitly-nested `target` + `teams` pair gets
the same omp.combined marking as the combined case.
Assisted-by: Cursor / Claude Sonnet 5 High
[LTO] Keep the remarks of every partition with parallel codegen
With ParallelCodeGenParallelismLevel > 1, splitCodeGen compiles each
partition in a fresh LTOLLVMContext. Those contexts have no remark
streamer, so code generation remarks (asm-printer, prolog-epilog,
regalloc, ...) never reach the -pass-remarks-output file.
Add LLVMRemarkBuffer, which streams the remarks of one context into an
in-memory bitstream buffer and later emits them through the serializer
of another context's remark file. splitCodeGen gives each partition a
buffer with the remark settings from lto::Config and, once all
partitions are done, emits them in partition order, so the file does
not depend on thread scheduling. Remarks are copied as they are;
RemarkLinker would sort and deduplicate them and drop the ones without
a debug location.
LTOCodeGenerator now stores its -lto-pass-remarks-* options in its
Config, so llvm-lto -j gets the partition remarks too.
[4 lines not shown]
[CIR] Make sure 'generated' functions get proper target attributes (#240511)
This was discovered by trying to enable the 'init-priority-attr' test
from classic codegen. IN that case, the lack of matching 'target'
attributes (target-cpu, target-features, tune-cpu) defeated inlining
when they don't match the default triple.
This patch adds the 'default' value for these as 'top level' module
attributes, and makes sure that 'runtime' functions are tagged this way
so that LLVM will be willing to inline them.
[NFC][SimplifyLibCalls] Skip non-libcalls earlier to save ~0.1% instrs (#240218)
Instcombine calls `LibCallSimplifier::optimizeCall` on every direct
function call, but most calls are not known runtime library function
calls. Check the (cached) TargetLibraryInfo::getLibFunc result before
setting up operand bundles, the IRBuilder guard and the fortified
simplifier, and return early for non-intrinsic, non-libcall callees.
Generated code should be identical.
I gathered instruction counts for sqlite3 -O3 and got a -0.08%
reduction. InstCombine over CUTLASS GEMM kernels compiled for sm_100
executes 0.15% fewer instructions.
LLM-aided
[MLIR][TOSA] Remove unneeded MLIRFuncTransformOps dep in TOSA Transforms (#240437)
TOSA Transforms depends on FuncTransformOps even though there is no
include of any header from Dialect/Func/TransformOps/ and nothing is
used from that library. Most likely a confusion with FuncTransforms
which is already depended on.
[RISCV] Add VeeR EH1 processor definition (#238935)
Add -mcpu=chipsalliance-veer-eh1 for the CHIPS Alliance VeeR EH1
(previously known as the Western Digital SweRV EH1), a dual-issue
in-order RV32IMC core: https://github.com/chipsalliance/Cores-VeeR-EH1
The scheduling model and tuning are a separate upcoming change.
Assisted-by: Claude
pci_iov: Reuse downstream bridge bus window if it already covers VF bus
If the parent bridge's [secondary, subordinate] window already covers
the VF bus (e.g., programmed by BIOS or a prior PF), skip allocating
PCI_RES_BUS. This avoids a duplicate rman allocation in the multi-PF
case while still allocating when growth is actually needed.
Reviewed by: ssaxena
Differential Revision: https://reviews.freebsd.org/D52163
(cherry picked from commit feb0a7e19f3c2e4c7eb90668b1e3dc34b5bb6dd6)
re(4): re-arm the Tx doorbell when re_txeof() leaves a non-empty ring
On PCIe parts a TxPoll request can be lost when packets are queued
in quick succession, leaving owned descriptors with no transfer in
progress until the watchdog fires. re_txeof() runs from the interrupt
handlers, re_tick() and re_watchdog(), so re-writing TXSTART whenever
the ring is still non-empty turns a potential 5-second stall into at
most one tick.
One register write on a path that already took an interrupt;
fast path untouched.
* Sustained bidirectional load on RTL8168H; no Tx stalls, no throughput
regression at 941 Mbps line rate.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D58277
PR: kern/166724
(cherry picked from commit 4bff6b9f5cef13df8540fa6869bd6707f8e7575d)
re(4): harden re_watchdog() recovery and log controller state
Distinguish the two failure classes from the PR in a single log line
(ring indices, ISR/IMR, TXCFG, interrupt mode): lost interrupt vs
genuine DMA stall.
Bail out instead of re-initializing when the controller reads back
all-ones (fallen off the bus; reinit cannot help).
Re-assert the driver's existing ASPM-disabled policy before reinit,
since firmware/power transitions re-arming L0s/L1 is a documented
stall trigger.
Diagnostics-only on the recovered path; no fast-path change.
* Field diagnostics running on an RTL8168H production fleet; the log
format distinguishes lost-doorbell / DMA-stall / dead-controller
without a debug build.
[5 lines not shown]
re(4): recover Tx completions whose MSI was swallowed in re_intr_msi()
A Tx completion that raises a status bit between the ISR ack at the top
of re_intr_msi() and the IMR re-enable at the bottom is never re-signalled:
these controllers do not re-assert MSI for an already-set status bit
(this is why hw.re.msi_disable is a known workaround in the PR).
Re-read ISR before re-enabling; if a Tx bit is pending, ack just that bit,
reap the ring and restart the queue. Rx bits are deliberately left set so
they re-arm the interrupt normally and Rx moderation state is untouched.
Also flush the posted IMR write. Mirrors what the INTx path already
achieves via the loop in re_intr().
* MSI interrupt mode on RTL8168H under load; "missed Tx interrupts"
watchdog recoveries no longer occur.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D58278
[3 lines not shown]
re(4): quiesce RTL8168G+ and reset before freeing buffers in re_stop()
The STOPREQ command written by re_stop() is not defined for
RTL8168G and later; issuing it can wedge the MAC.
Replace it on those parts with the vendor-documented sequence:
* settle delay
* bounded poll for Tx queue empty
* clear TE/RE
* then bounded poll of the MCU command register (0xD3) FIFO-empty bits.
Also reset the controller before the Rx/Tx buffer free: a controller that
has not quiesced keeps DMAing stale, still-owned descriptors pointing at
freed mbufs (use-after-free under INVARIANTS, cross-NIC mbuf corruption
reported in the PR).
Adds the RL_MCU_* register definitions.
[13 lines not shown]
if_bnxt: avoid FTQM/STQM pg_info alias on reset
Copying the STQM backing-store context to seed FTQM's also copied
STQM's pg_info pointer verbatim. On cold load STQM's pg_info is still
NULL so this is harmless, but on a firmware reset STQM's pg_info is
already non-NULL, making FTQM alias STQM's backing-store pages. The
allocator then skips FTQM since it looks already allocated, and FTQM
is later indexed as its own array, reading out of bounds into STQM's
buffer and dereferencing a bogus DMA address.
Clear ctxm->pg_info after the memcpy so FTQM always gets its own
backing store.
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Broadcom Inc.
Differential Revision: https://reviews.freebsd.org/D58615
(cherry picked from commit f2ce672ade231a1e19f1238922a2c3b6fe696e22)
if_bnxt: add support for HW generic stats
HW generic stats provide information about PCIe transactions and
out-of-order doorbell writes, useful for debug. Allocate a DMA buffer
for them (disabling BNXT_FW_CAP_GENERIC_STATS if the allocation
fails), fetch them via HWRM_STAT_GENERIC_QSTATS once per admin-status
tick on P5+ chips, and expose them to userspace under
dev.bnxt.<unit>.hwstats.generic_stats.* via sysctl.
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Broadcom Inc.
Differential Revision: https://reviews.freebsd.org/D58614
(cherry picked from commit d1619865da1cd4c5b557158b4bbf666e0f7f040b)
if_bnxt: add support for SIOCGI2CPB page/bank i2c requests
Advertise IFLIB_I2C_PAGE_BANK and forward the requested page/bank to
firmware, so CMIS optics can be read correctly instead of falling
back to bogus legacy SFF-8472 decoding. Also fix bnxt_i2c_req() to
return positive errno values.
Signed-off-by: gallatin
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Broadcom Inc.
Differential Revision: https://reviews.freebsd.org/D59573
(cherry picked from commit fdab50f6f8107c9d825a7e6d92d7a6b89c417bbb)
if_bnxt: Fix interrupt storm on Thor2
An interrupt storm was observed on Thor2 because the Rx CQ consumer
index was always one less than it should be. Correct the Rx CQ
consumer index to fix the storm.
Reported by: gallatin
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Broadcom Inc.
Differential Revision: https://reviews.freebsd.org/D58602
(cherry picked from commit 8c69a5dc60b3630a4c5bca0bda714f791324afad)
bnxt_en: dcb: stop zeroing ETS and PFC config in bnxt_dcb_init()
bnxt_dcb_init() currently pushes all-zero ETS and PFC settings to
the firmware at every attach. This reserves bandwidth allocation
for RoCE traffic, which prevents L2 traffic from reaching line rate.
These settings should only be updated with valid values when the
RoCE driver is loaded. Therefore, we should stop programming them
during bnxt_dcb_init().
Signed-off-by: Andy Gospodarek <gospo at broadcom.com>
Reviewed by: chandrakanth.patil_broadcom.com, gallatin
MFC after: 2 weeks
Sponsored by: Broadcom Inc.
Differential Revision: https://reviews.freebsd.org/D60516
(cherry picked from commit 31cbd8f27d687ce9b10a22c858296532be400bb4)
[MLIR][SPIRV] Remove unneeded dependency on Core (#240422)
ArithToSPIRV only depends on ADT and Support headers whose
implementation are all part of Support rather than Core. Core is
therefore an unneeded dependency so this commits removes it.
[flang][CodeGen] Lower undefined fir.box to descriptor storage (#240439)
A fir.box SSA value is lowered as the address of its descriptor storage,
and fir.load/fir.store of a box are lowered as memcpy through that
address. fir.undefined of a box type was lowered to an undefined
pointer, so storing or loading a possibly undefined box (for instance a
box merged with the fir.undefined default value that MLIR mem2reg
inserts when promoting a fir.alloca of a box) copied from an undefined
address. Optimized code usually drops such copies, but unoptimized code
reads from address zero.
Lower fir.undefined of a box type to an uninitialized descriptor alloca
so that these copies are well defined. For assumed-rank boxes, whose
copy size is computed from the rank field, also set the rank to zero so
that the copy stays within the storage.
