[InstCombine] Keep condition weights when reassociating logical selects (#239484)
Reassociating (A || B) || C and (A && B) && C leaves a select that still
tests A. Copy that select's existing weights instead of marking the new
one unknown.
[AMDGPU][~NFC] Refactor buffer intrinsic generation, add arg positions to table (#241613)
We already had a lookup table for whether an intrinsic was a resource
intrinsic, which included if any given intrinsic used an image and where
the resource argument was.
However, once we know we had a buffer intrinsic (like
amdgcn_raw_ptr_buffer_load or amdgcn_struct_ptr_tbuffer_store) we didn't
have a good way to know where other arguments (like the `voffset` field)
were ... or whether they were present at all (thus making things like
telling raw and struct variants of an operation, which differ by a
`vindex` operand, annoying). Generally, we relied on an awkward
combination of switch statements and manual reasoning around things like
"the aux/cpol argument is always last" or, more riskily, things like
"well, the offset will be 3 or 4 past the resource, depending on if
there's a vindex".
After realizing I'd just had an LLM create yet another version of this
logic on a different patch, I decided I should unleash one onto the
[40 lines not shown]
[mlir][WasmSSA] Add missing MLIRControlFlowInterfaces link dependency (#243322)
Follow-up fix for #243023 (see the llvm-ci buildbot comments there).
`ReturnLike` implies
`DeclareOpInterfaceMethods<RegionBranchTerminatorOpInterface>`, so the
regenerated `WasmSSA_ReturnOp` pulls symbols from
`MLIRControlFlowInterfaces`. Static builds don't notice (everything
resolves in the final executable link), but shared-library builders fail
at `lib/libMLIRWasmSSADialect.so` with `-Wl,-z,defs`:
- https://lab.llvm.org/buildbot/#/builders/226/builds/30137
- https://lab.llvm.org/buildbot/#/builders/203/builds/69390
Add `MLIRControlFlowInterfaces` to `LINK_LIBS`, matching Func and
MLProgram, which link it for the same trait.
Co-authored-by: Ferdinand Lemaire <flscminecraft at gmail.com>
SimplifyLibCalls: preserve allocation attributes in optimizeNew
When `LibCallSimplifier::optimizeNew` rewrites an `operator new` call into a
`__hot_cold_t` allocation call, forward the allocation attributes
(`allockind`, `allocsize`, `"alloc-family"`, and parameter `allocalign`) from
the original call onto the synthesized call instruction alongside its metadata.
This is a new patch since 2023 (unlike most of the rest of the stack)
and allows the next patch to remove more specialized C++ knowledge
from the LLVM layer, instead leaving it in clang where it seems more
sensible.
This patch was authored with the assistance of an LLM and was reviewed
by me before posting.
CGExprCXX: emit family and allockind attributes
This allows us to remove the special knowledge about C++ allocation from
LLVM's MemoryBuiltins.cpp, instead trusting attributes to communicate
the important properties.
Annoyingly, clang has to grow a little bit of hard-coded knowledge about
mangled symbol names in order to actually get the most-basic-matching
::operator::new when working out the alloc-family attribute. I spent
some time trying to do this "right" by looking up the available
operators new and finding the best one, but that's super tricky and this
has the virtue of working and being pretty obviously correct when
compared to the old code.
Happily, clang now emits allocator family attributes for custom
allocators like the `__hot_cold_t` overloads in `new_hot_cold.cpp`
(which `isReplaceableGlobalAllocationFunction` in clang recognizes),
allowing us to remove that special knowledge from LLVM.
[7 lines not shown]
arm64 pmap: combine L3C accessed bits in pmap_mincore()
The accessed bit may be set in only some of an L3C superpage's L3
PTEs. However, when pmap_mincore() found an L3C mapping, it reported
the accessed state of the specified L3 PTE only, so mincore(2) could
fail to report MINCORE_REFERENCED for an L3C superpage that had been
accessed. When the accessed bit is not set in the specified entry, use
pmap_load_l3c() to combine the accessed bits from all of the L3C
superpage's L3 PTEs.
(cherry picked from commit 3ba68f5c5dcc9a4bcd513c28efce29afdfcb76ff)
[VPlan] Remove header masks whose tail lanes aren't used
Replace a use of header-mask && x with x if the user, transitively
through freeze, logical-and, not and mask.beforefirst(freeze ...), only
reaches masks that are and'ed with the header mask again. The header
mask is a prefix and lane i of mask.beforefirst only depends on lanes
0..i, so the lanes outside of the header mask are never used.
This avoids the i1 vp.merge that EVL tail folding needs for the inner
header mask, e.g. in the early exit mask.beforefirst(freeze(header-mask
&& c)) and in switch default edges.
[VPlan] Allow tail folding early exits with side effects
Continues the work in #179595.
This allows us to tail fold e.g. the following loop:
```c
extern char a[1024], b[1024], c[1024], y[1024];
void two_guards(void) {
for (int i = 0; i < 1000; i++) {
if (a[i] == 0)
break;
if (b[i] > 3) {
if (c[i] < 9)
y[i] = 1;
}
}
}
```
[30 lines not shown]
[VPlan] Support tail folding uncountable early exits
Enables the first step towards #179595. After #237254, the MaskedHandleExitInVectorLoop early exit style uses the CFG to handle masking for early exit loops. This allows it to compose with tail folding which is also handled via the CFG in #176143.
This allows us to tail fold the following loop:
```c
extern char x[256];
int f(char * restrict y) {
for (int i = 0; i < 256; i++) {
if (x[i] == 0)
return i;
}
return -1;
}
```
Into this on RISC-V:
[18 lines not shown]
[VPlan] Use vputils::findCanonicalIVIncrement in more places. NFC
In a future patch we want to enable more scenarios with early exits and tail folding. In these scenarios the terminator will no longer be a BranchOnCount/BranchOnCond but a BranchOnTwoConds, so a bunch of canonical IV matchers break. Use vputils::findCanonicalIVIncrement instead since it searches the users of the canonical IV instead of looking at the terminator.
[VPlan] Allow early exit loops with side effects by masking in the vector loop.
When all blocks with side effects are dominated by every early exit, we can vectorize the loop if we mask off any lanes past the exit. We can do this with control flow:
```mermaid
flowchart TD
header --> exiting
exiting -->| exit mask | sideeffects --> latch --> latch.split
exiting -->| !exit mask | latch.split
```
For every early exiting block we add a branch to the latch on `!mask.beforefirst(exitcond)`. This is implemented in UncountableExitStyle::ReadOnly which is now renamed to MaskedHandleExitInVectorLoop.
This is a complementary exit style to MaskedHandleExitInScalarLoop. If any of the side effects aren't dominated by the early exit it still falls back to the scalar loop style, which is why none of the existing early exit side effect tests changed. Masking in loop is profitable for RISC-V and I'm assuming it's also profitable for AArch64 too, but if not then we can add a TTI hook for a preferred exit style.
[VPlan] Preserve SSA in early exit loops
MaskedCond is used to logically-and the condition to early exit with the current block in-mask. However because we use it in the latch block where the condition isn't defined on all paths, we have to workaround it not being in SSA in the verifier.
This fixes this by using reconstructSSA added in #212209 to preserve SSA in both the exit conditions and live outs in the exit blocks. It also means we get the right exit condition without MaskedCond and allows us to remove the MaskedCond exceptions in the verifier. MaskedCond is still kept since a followup PR will still use it.
One subtle point is that we need to freeze any branch conditions. Any lanes after and including the early exiting lane might contain poison and we can have branches on those poison lanes that didn't we didn't previously branch on in the scalar loop.
There's no miscompile on main today because we coincidentally don't keep SSA so exit conditions are read directly without being polluted by poison stemming from a branch past the early exit. But in this PR the exit conditions are polluted and we can end up with poison in the condition operand of a select, which causes an undefined result for AnyOf even if it freezes its operand. The poison in the branch will also cause problems in future if we decide to preserve uniform control flow and emit branches instead of blends. The exit_cond_merged_after_poison_branch test illustrates this.
[VPlan] Simplify BLEND X/M, false, ..., false -> M && X
If an i1 blend has false for all but one incoming value, then we can simplify this to the one non-false value. Blends by construction never have any overlapping mask values, so we don't need to worry about the order as the masks can't "shortcut" each other.
[VPlan] Move extract-lane(0, canWideIV) combine before unrolling
Some extract-lane combines can't be performed before unrolling, e.g. the one that combines it to Instruction::ExtractElement can end up extracting from a different part. But the extract-lane(0, canWideIV) can because at index 0 you're always extracting from the first part. Do this before unrolling, which improves codegen of unrolled code.
[VPlan] Support interleaving early exit loops with side effects
We couldn't do this on main previously because the exit mask `active-lane-mask(0, FirstActiveLane(Cond))` doesn't get unrolled properly across parts. Now that we use mask.beforefirst we can unroll it properly in a chain. The unrolled form is:
unroll(mask.beforefirst, P) = mask.beforefirst(cond[P] | broadcast(anyof(not(unroll(mask.beforefirst, P-1)))))
On AArch64 this can be combined to brkpb in another patch.
[VPlan] Use llvm.mask.beforefirst for computing early exit mask
This makes the cost more accurate and improves codegen for RISC-V. The codegen for an early exit loop with side effects then goes from:
.LBB0_3: # %vector.body
# =>This Inner Loop Header: Depth=1
vl1r.v v16, (a3)
vsetvli zero, zero, e8, m1, ta, ma
vmseq.vi v16, v16, 0
vfirst.m a7, v16
srli t0, a7, 63
czero.nez a7, a7, t0
czero.eqz t0, a1, t0
or a7, t0, a7
vsetvli zero, zero, e64, m8, ta, ma
vmsltu.vx v0, v8, a7
vle8.v v17, (a5), v0.t
vcpop.m t1, v16
vsetvli zero, zero, e8, m1, ta, ma
[22 lines not shown]
[llubi] Add support for vp.strided.load and vp.strided.store (#241715)
Building on top of https://github.com/llvm/llvm-project/pull/241710,
these are defined in terms of vp.gather and vp.scatter.
The base pointer must be aligned to the align attribute/default ABI
alignment, but subsequent pointers past the first lane are unaligned.
Assisted-by: Opus 5.5
[llubi] Add support for vp.gather and vp.scatter (#241710)
These follow the semantics of llvm.masked.gather/llvm.masked.scatter. We
need to just make sure that the implicit alignment is taken from the
element type, not the vector type.
Assisted-by: Opus 5.5
Update FP8 conversion test for x86 small-int ABI
b1c38d9b5839 (#236450) stopped emitting zeroext/signext on small
integer arguments for x86-64 SysV, so the autogenerated checks for
the i8 parameters no longer matched after rebasing onto main.
Regenerate the checks with update_cc_test_checks.py.
Change-Id: I2f8bb84a8d70888d0ec125e2dd1cf19dc39aaa08
Co-Authored-By: Claude Fable 5.2 <noreply at anthropic.com>
[scudo] Fix DoubleFreeFromSecondary to allow multiple messages. (#243238)
If the secondary is zeroed then a different error message will occur, so
allow that other message too.