[VPlan] Remove header masks whose tail lanes aren't used
Replace a use of header-mask && x with x if the user, transitively
through freeze, logical-and, not and mask.beforefirst(freeze ...), only
reaches masks that are and'ed with the header mask again. The header
mask is a prefix and lane i of mask.beforefirst only depends on lanes
0..i, so the lanes outside of the header mask are never used.
This avoids the i1 vp.merge that EVL tail folding needs for the inner
header mask, e.g. in the early exit mask.beforefirst(freeze(header-mask
&& c)) and in switch default edges.
[VPlan] Allow tail folding early exits with side effects
Continues the work in #179595.
This allows us to tail fold e.g. the following loop:
```c
extern char a[1024], b[1024], c[1024], y[1024];
void two_guards(void) {
for (int i = 0; i < 1000; i++) {
if (a[i] == 0)
break;
if (b[i] > 3) {
if (c[i] < 9)
y[i] = 1;
}
}
}
```
[30 lines not shown]
[VPlan] Support tail folding uncountable early exits
Enables the first step towards #179595. After #237254, the MaskedHandleExitInVectorLoop early exit style uses the CFG to handle masking for early exit loops. This allows it to compose with tail folding which is also handled via the CFG in #176143.
This allows us to tail fold the following loop:
```c
extern char x[256];
int f(char * restrict y) {
for (int i = 0; i < 256; i++) {
if (x[i] == 0)
return i;
}
return -1;
}
```
Into this on RISC-V:
[18 lines not shown]
[VPlan] Use vputils::findCanonicalIVIncrement in more places. NFC
In a future patch we want to enable more scenarios with early exits and tail folding. In these scenarios the terminator will no longer be a BranchOnCount/BranchOnCond but a BranchOnTwoConds, so a bunch of canonical IV matchers break. Use vputils::findCanonicalIVIncrement instead since it searches the users of the canonical IV instead of looking at the terminator.
[clang] Remove declarations without definitions (NFC) (#242170)
This patch removes function and method declarations that have no
definitions and no callers. This patch also explicitly deletes two
undefined default constructors so that accidental default construction
fails at compile time instead of link time.
Assisted-by: Antigravity
[VPlan] Allow early exit loops with side effects by masking in the vector loop.
When all blocks with side effects are dominated by every early exit, we can vectorize the loop if we mask off any lanes past the exit. We can do this with control flow:
```mermaid
flowchart TD
header --> exiting
exiting -->| exit mask | sideeffects --> latch --> latch.split
exiting -->| !exit mask | latch.split
```
For every early exiting block we add a branch to the latch on `!mask.beforefirst(exitcond)`. This is implemented in UncountableExitStyle::ReadOnly which is now renamed to MaskedHandleExitInVectorLoop.
This is a complementary exit style to MaskedHandleExitInScalarLoop. If any of the side effects aren't dominated by the early exit it still falls back to the scalar loop style, which is why none of the existing early exit side effect tests changed. Masking in loop is profitable for RISC-V and I'm assuming it's also profitable for AArch64 too, but if not then we can add a TTI hook for a preferred exit style.
[VPlan] Preserve SSA in early exit loops
MaskedCond is used to logically-and the condition to early exit with the current block in-mask. However because we use it in the latch block where the condition isn't defined on all paths, we have to workaround it not being in SSA in the verifier.
This fixes this by using reconstructSSA added in #212209 to preserve SSA in both the exit conditions and live outs in the exit blocks. It also means we get the right exit condition without MaskedCond and allows us to remove the MaskedCond exceptions in the verifier. MaskedCond is still kept since a followup PR will still use it.
One subtle point is that we need to freeze any branch conditions. Any lanes after and including the early exiting lane might contain poison and we can have branches on those poison lanes that didn't we didn't previously branch on in the scalar loop.
There's no miscompile on main today because we coincidentally don't keep SSA so exit conditions are read directly without being polluted by poison stemming from a branch past the early exit. But in this PR the exit conditions are polluted and we can end up with poison in the condition operand of a select, which causes an undefined result for AnyOf even if it freezes its operand. The poison in the branch will also cause problems in future if we decide to preserve uniform control flow and emit branches instead of blends. The exit_cond_merged_after_poison_branch test illustrates this.
[VPlan] Simplify BLEND X/M, false, ..., false -> M && X
If an i1 blend has false for all but one incoming value, then we can simplify this to the one non-false value. Blends by construction never have any overlapping mask values, so we don't need to worry about the order as the masks can't "shortcut" each other.
[VPlan] Move extract-lane(0, canWideIV) combine before unrolling
Some extract-lane combines can't be performed before unrolling, e.g. the one that combines it to Instruction::ExtractElement can end up extracting from a different part. But the extract-lane(0, canWideIV) can because at index 0 you're always extracting from the first part. Do this before unrolling, which improves codegen of unrolled code.
[VPlan] Support interleaving early exit loops with side effects
We couldn't do this on main previously because the exit mask `active-lane-mask(0, FirstActiveLane(Cond))` doesn't get unrolled properly across parts. Now that we use mask.beforefirst we can unroll it properly in a chain. The unrolled form is:
unroll(mask.beforefirst, P) = mask.beforefirst(cond[P] | broadcast(anyof(not(unroll(mask.beforefirst, P-1)))))
On AArch64 this can be combined to brkpb in another patch.
[VPlan] Use llvm.mask.beforefirst for computing early exit mask
This makes the cost more accurate and improves codegen for RISC-V. The codegen for an early exit loop with side effects then goes from:
.LBB0_3: # %vector.body
# =>This Inner Loop Header: Depth=1
vl1r.v v16, (a3)
vsetvli zero, zero, e8, m1, ta, ma
vmseq.vi v16, v16, 0
vfirst.m a7, v16
srli t0, a7, 63
czero.nez a7, a7, t0
czero.eqz t0, a1, t0
or a7, t0, a7
vsetvli zero, zero, e64, m8, ta, ma
vmsltu.vx v0, v8, a7
vle8.v v17, (a5), v0.t
vcpop.m t1, v16
vsetvli zero, zero, e8, m1, ta, ma
[22 lines not shown]
[llubi] Add support for vp.strided.load and vp.strided.store
Building on top of https://github.com/llvm/llvm-project/pull/241710, these are defined in terms of vp.gather and vp.scatter
Assisted-by: Opus 5.5
[RISCV] Don't restore SP from FP in the epilogue when SP never moved (#239761)
With a frame pointer and the V extension enabled, hasReservedCallFrame()
is false for every function, because hasRVVFrameObject() answers from
the subtarget rather than from the frame. emitEpilogue() then restores
SP from FP in every frame-pointer function, including the ones with no
outgoing stack arguments, no RVV stack objects and no variable-sized
objects, where SP still holds the value the prologue gave it.
Only restore SP from FP for an unreserved call frame when a call passes
arguments on the stack, the case the restore was added for, or when it
replaces the multi-instruction undo of an RVV stack area or a split SP
adjustment.
In a distribution libLLVM.so built for RVA23 with frame pointers, this
removes 133,758 of 140,895 such restores, about 535 KB of text.
Found-with riscvlint.
Assisted-by: Claude Code (Claude Fable 5.1)
[RISCV] Skip the xor with x0 when a masked atomic stores zero (#240749)
The masked merge computes old ^ ((old ^ new) & mask). When the new value
is zero it arrives in x0, and the first xor is a move spelled
`xor scratch, old, zero`. AND old with the mask directly instead. In a
masked cmpxchg the loop head has already computed old & mask into the
scratch register, so the loop tail needs only the final xor.
This is LLVM's 8- and 16-bit compare-and-exchange of zero, which is 591
of the 636 non-canonical moves riscvlint finds in Firefox's libxul.
Found via riscvlint.
Assisted-by: Claude Code (Claude Opus 5.5)
[mlir][WasmSSA] Mark wasmssa.return as ReturnLike (#243023)
Fixes #242375.
When erasing dead results of a `FunctionOpInterface` operation,
`-remove-dead-values` checks each block terminator for the statically
declared `ReturnLike` trait and erases the corresponding terminator
operands. `wasmssa.return` was only declared as a terminator, so after
the pass erased `wasmssa.func`'s dead results, `wasmssa.return` kept its
operands and the module no longer verified:
```
error: 'wasmssa.return' op has 1 operands, but enclosing function returns 0
```
This marks `wasmssa.return` as `ReturnLike` (the trait lives in
ControlFlowInterfaces.td, hence the new include) and adds a regression
test under `mlir/test/Dialect/WasmSSA/`.
[lldb] Run a module's dSYM scripts once per target (#242920)
Today, `Target::ModulesDidLoad` loads the scripting resources of every
module it is given, and it is given the same module more than once: when
the module is added to the target, when a dynamic loader gives it a load
address, and when a dynamic loader removes it and adds it back, like
DynamicLoaderMacOS does when a process launches. Each time, LLDB
reloaded the module's dSYM script, ran its `__lldb_init_module` again
and called `__lldb_module_added_to_target` again. Scripts that register
extensions on their target, like stop hooks or frame providers,
registered them twice, and the reloading the module resets the
module-level state they kept.
This patch makes the target record the UUID of each module it
auto-loaded a script for, and skip the script when it sees the module
again. A module that leaves the image list and comes back, or that LLDB
creates again for the same binary, doesn't load its script again, while
a rebuilt binary, which gets a new UUID, does. A module without a UUID
isn't recorded, so its script still runs every time.
[8 lines not shown]
[llubi] Add support for vp.gather and vp.scatter
These follow the semantics of llvm.masked.gather/llvm.masked.scatter. We need to just make sure that the implicit alignment is taken from the element type, not the vector type.
[AST] Refactor format string parsing into parser classes (NFC) (#243043)
The functions that analyze printf and scanf format strings were free
functions threading the same parameters through every call. Gather them
into classes and turn constant parameters into class members.
Since I'm changing function signatures, I took this opportunity to
rename them to lowerCamelCase to conform to the coding standard.