[VPlan] Append recipes created via builder to worklist
The previous PR appended the top most created recipe to the worklist, and this PR extends it to any other nested recipes that were created, similar to InstCombine.
This removes the header mask in a good few more places on RISC-V as measured on SPEC CPU 2017, e.g. for the following loop:
```c
long f(const int *p, const int *q, long n) {
long a = 0, b = 0;
for (long i = 0;; i++) {
if (p[i] && q[i]) { a += i; b += i; }
if (i + 1 == n) break;
}
return a + b;
}
```
Before:
[49 lines not shown]
[InterleavedLoadCombine] Do not combine loads across basic blocks (#223915)
This pass turns a group of interleaved loads into a single wide load
inserted at the first load. That is only valid when all of the loads are
in the same basic block. Otherwise the wide load can read memory that
the original program only accessed on a conditional path.
The offset index keyed candidates on base pointer, type and offset only,
so it could pair loads from different blocks. Restrict each block's
matching to candidates whose loads are in that block. This also keeps
the candidate list and index per block, so they stay small.
Fixes a miscompile introduced by #213053.
[clang][LoongArch] Emit "target-abi" module flag (#223647)
The LoongArch backend reads the ABI from the "target-abi" module flag,
but clang only emits that flag for ARM, PowerPC and RISC-V. Since LTO code
generation doesn't see the -target-abi option passed to clang, it falls
back to the default ABI of the target, which is lp64d for
loongarch64-unknown-elf, regardless of the ABI the code was compiled for.
When an FPU is enabled, this silently miscompiles code built for another
ABI. For example, with -mabi=lp64s, functions get the lp64d calling
convention after LTO, passing and returning floating-point values in FP registers
instead of general-purpose ones, and the output is marked as double-float. Nothing
reports this if every object goes through LTO. For instance, a shared library
built this way links without error, and so does lp64s code built without LTO that
uses it. Without an FPU, the generated code keeps the soft-float calling
convention, but the output is still marked as double-float.
The mismatch only shows up as an error when LTO output is linked
together with relocatable objects built without LTO, which LLD rejects with "cannot
[15 lines not shown]
[LLVM][CodeGen][SME] Don't use non-streaming zeroing instruction when in streaming mode. (#222353)
When selecting scalable vector bf16-f32 converts we emit zip(0,vec).
Unfortunately, the typical zero-latency zeroing instruction is not
available when in streaming mode.
(cherry picked from commit 9679fbfea993cb2e815fe0fa5eefde310f917349)
[AMDGPU] Fix GlobalISel VGPR-to-VCC lowering types (#220987)
Register-bank legalization synthesized an untyped G_AND s32, but
instruction-selection patterns only matched typed G_AND i32, causing:
```
LLVM ERROR: cannot select: %:vgpr_32(s32) = G_AND ...
```
The source-level and i32 was valid; the incorrect type was introduced by
AMDGPU `VgprToVccCopy` lowering. SelectionDAG was unaffected because it
retained typed DAG nodes.
Assisted-by: Codex
---------
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
AMDGPU: Maintain LiveIntervals when removing blocks in SILowerControlFlow (#223642)
removeMBBifRedundant removed a redundant block's instructions from the
LiveIntervals maps but not the block itself, leaving a stale idx2MBBMap
entry that crashed getMBBFromIndex in the verifier.
Add SlotIndexes::removeMBBFromMaps to drop the erased block's slot
range, and recompute the intervals that spanned it.
Co-authored-by: Claude claude-opus-4.8 <noreply at anthropic.com>
[RISCV] Do not verify size for some Xqci instructions (#223913)
After instruction size verification was enabled by default in
https://github.com/llvm/llvm-project/pull/221687/ we were running into
crashes for certain 48-bit Xqci instructions. This was happening because
we have compress patterns for these 48-bit instructions to compress them
into 32-bit instructions which `getInstSizeInBytes` was not taking into
account.
The crash looks like:
```
Size mismatch for: QC_E_SW $x0, $x14, 0 :: (store (s32) into %stack.0)
Expected maximum size: 2
Actual size: 4
```
Do not verify size for such instructions until we can return the correct
size for them.
AMDGPU: Use LiveIntervals in SIOptimizeVGPRLiveRange when available (#222385)
LiveVariables has been long deprecated. Use LiveIntervals if available.
With the current pass structure, this will use LiveVariables.
Co-authored-by: Claude (Claude-Opus-4.8)
[IR] Simplify DIAssignID by storing its links in the node. NFC (#223324)
AssignmentTrackingPass generates DIAssignID metadata. A DIAssignID links
the instructions it is attached to, kept in
LLVMContextImpl::AssignmentIDToInstrs, with the dbg_assign records that
refer to it, found through a permanent use list (#78300 made DIAssignID
the only always-replaceable MDNode for this). Keep both lists in the
node as TinyPtrVector members instead (98% of instruction lists and
88-96% of record lists hold exactly one pointer), and drop the map and
`isAlwaysReplaceable()`.
Aided by Opus 5
[RISCV] Merge zeroing stores into wider LMUL vector stores (#223908)
memset lowers to LMUL1 vector stores (v2i64 for e64) that we rely on
store merging to widen. `storeOfVectorConstantIsCheap` counted the
number of MemVT-sized stores, so a v2i64 MemVT needed 4 stores before
merging kicked in, leaving two VL=2 m1 stores for a 32B zero region.
Use the total element count for zeroing stores so two v2i64 stores
merge into one VL=4 e64 m2 store.
Partially fixes #223906
Assisted-by: TRAE CLI (Opus 4.8)
[MustExecute] Use the loop predecessor instead of the preheader (#222096)
CanProveNotTakenFirstIteration() gets the IV start value via
LHS->getIncomingValueForBlock(CurLoop->getLoopPreheader()). If the loop
is not in simplified form, getLoopPreheader() returns null and the
lookup asserts. This is reachable from print<must-execute> on IR that is
not in LoopSimplify form (see the reproducer in #166488).
Fixed: use getLoopPredecessor() instead: it returns the unique
out-of-loop predecessor of the header even when it is not a dedicated
preheader, consequently the start value is still known. If there is no
such predecessor, bail out.
Fixes #166488.
[IR][InstSimplify] Clarify poison elements in cttz.elts results in poison
The generic expansion for cttz.elts in LegalizeVectorOps is
(reduce.umax (select input, step_vector, 0))
Because reductions produce poison if any element is poison, this will also produce poison.
ConstantFolding currently has short circuiting behaviour, e.g. `cttz.elts <0, 1, poison, poison> -> 1`, but we should probably make this consistent with reductions and just return poison. This updates LangRef and ConstantFolding to be more explicit about this.
CodeGen: Replace some getPointerRegClass() uses with operand reg classes
Try to take the register class from the contexual instruction begin emitted
instead. This is a step towards fully replacing PointerLikeRegClass with
RegClassByHwMode.
Co-authored-by: Claude (Opus 4.8) <noreply at anthropic.com>
[lldb] Support building lldbHost under Emscripten (#223206)
This builds on the new `HostInfoEmscripten` and `PlatformEmscripten`
support.
Emscripten provides many POSIX APIs, but it does not support `fork` or
native host process management. This patch allows `lldbHost` to build
under Emscripten without treating it as Linux.
It:
- avoids building the `ProcessLauncherPosixFork` implementation;
- adds a small Emscripten Host implementation;
- reports process enumeration, process lookup, launching and shell
expansion as unsupported.
The remaining generic POSIX Host functionality continues to be used.
This is enough for the current libLLDB and SB API experiment, while live
process debugging will be handled separately once we have an in-browser
execution backend.
[RISCV] Replace decodeImmThreeOperand/decodeImmFourOperand with a template function. NFC (#223924)
Make the immediate a template argument that we can be provided in the
DecoderMethod name.
[clang][AArch64] Consolidate codegen tests for SVE's DUP intrinsics
This patch completes the migration of the tests from:
* clang/test/CodeGen/AArch64/sve-intrinsics/acle_sve_dup.c
to:
* clang/test/CodeGen/AArch64/sve/dup.c
`dup.c` covers all the cases previously tested by `acle_sve_dup.c`.
However, unlike `acle_sve_dup.c`, it was only tested as C. Add `-x c++`
RUN lines to provide the equivalent C++ coverage, including lowering
via CIR.
[RISCV] Update Zvzip support to v0.3 (#210603)
This updates the experimental Zvzip implementation to match the
v0.3 draft specification from RISC-V ISA manual PR:
https://github.com/riscv/riscv-isa-manual/pull/3233
In Zvzip 0.3, `vtype` and `vl` describe the destination vector for all
five instructions:
- `vzip.vv` produces a destination with EMUL=LMUL from two source
operands with EMUL=LMUL/2. Its mask is indexed by destination
elements and therefore has the mask type associated with the result
vector. The overlap rule of `vzip.vv` matches widening vector
arithmetic instructions;
- `vunzipe.v` and `vunzipo.v` produce a destination with EMUL=LMUL
from a source operand with EMUL=2*LMUL.
[17 lines not shown]
[clang] Fix Itanium mangler crash on lambdas in top-level statements (#217041)
`clang-repl` asserts when a lambda in a top-level statement needs a
mangled name, for example `ns::call([]{});`. Since #84150 a
`TopLevelStmtDecl` is a `DeclContext`, and `manglePrefix` casts it to
`NamedDecl`.
Treat `TopLevelStmtDecl` as a local container and give it a synthesized
internal encoding (`L9__stmt__0v`), like `Ub_` for block literals.
Entities in a top-level statement then mangle as `<local-name>`s with
discriminators, so same-named locals in two statements do not collide.
Each `TopLevelStmtDecl` gets an ordinal at creation, in parse order; it
is serialized.
Tests: `Interpreter/lambda-top-level-stmt.cpp` (crashes without the fix)
and `CodeGenCXX/top-level-stmt-local-names.cpp` (two lambdas, two
same-named local classes). The interpreter test is `UNSUPPORTED:
system-windows`: MSVC compat enables `-fdelayed-template-parsing`, and
clang-repl asserts on any late-parsed template in a later fragment,
[2 lines not shown]
[clang][NVPTX] Emit !atomic.ignore.denormal.mode for CUDA atomics
CUDA's atomicAdd() family is defined in terms of PTX atom.add, whose
denormal behavior is fixed by the hardware. Without any annotation the
backend has to assume the function's denormal mode must be honored and
expands these into CAS loops whenever the two disagree. Mark them with
!atomic.ignore.denormal.mode so the native instruction is used.
That covers the __nvvm_atom_*_add_gen_f builtins that atomicAdd(),
atomicAdd_block() and atomicAdd_system() are written in terms of, plus
C11/C++11 atomics under -fatomic-ignore-denormal-mode and the
[[clang::atomic(ignore_denormal_mode)]] attribute, which requires
teaching the NVPTX target about AtomicOptions.
The condition for when the metadata is meaningful is now shared with the
AMDGPU and SPIR-V targets in addAtomicIgnoreDenormalModeMetadata(). It
takes an AllowHalf flag because whether f16 denormals are observable is
target specific: PTX exposes no FTZ control for f16 operations, so
atom.add.f16 never flushes and the opt-in is meaningful there, whereas
[3 lines not shown]
