[mlir][SPIR-V] Add LifetimeStart and LifetimeStop operations (#232191)
Add spirv.LifetimeStart and spirv.LifetimeStop with assembly syntax,
verification, and generated binary serialization and deserialization.
Both operations require function scope and the Kernel capability.
[CIR] Implement 'loop hint' lowering (#232975)
Classic codegen has a lot of infrastructure to emit the loop-info-stack,
we can do it as a pretty simple attribute. This handles lowering to CIR
for these hints, after extracting the stuff that determines the values
to a separate type in CodeGenUtils.
Flattening the loops/emitting this to LLVM-IR is left to a follow up.
AI Notice: I made claude help me with the tests, so it did much of that
to make sure we got reasonable test coverage.
[Support][NFC] Fix MSVC C4267 warning in SourceMgr.h (#233299)
`SourceMgr::getNumBuffers()` and `SourceMgr::AddNewSourceBuffer()`
return `unsigned` but implicitly narrow `std::vector::size()`
(`size_t`). When compiled with 64-bit MSVC at the default `/W3` level,
code using these functions gets:
- `warning C4267: 'return': conversion from 'size_t' to 'unsigned int',
possible loss of data`
LLVM's own MSVC build disables C4267 (`-wd4267` in
`HandleLLVMOptions.cmake`), so this does not show up in-tree, but it
does in projects that include LLVM headers without that suppression. Add
explicit `static_cast`s.
Verified with MSVC 19.36 (VS 2022 17.6), x64, `/W3`: the warning appears
before this change and is gone after it.
No functional change; existing SourceMgrTest coverage exercises both
functions. SourceMgrTest passed 42/42, and the whole SupportTests suite
[2 lines not shown]
[flang] Make LoopVersioning deterministic (#233438)
Use a `MapVector` rather than a `DenseMap` to collect accessed arrays in
LoopVersioning so that the order is deterministic and doesn't depend on
memory addresses. Similar fix to #231802.
[ShrinkWrap] Teach shrink-wrapper to look through PHIs (#232839)
When analysing shrink-wrap candidates for potential stack usage, we
currently call `getUnderlyingObject`, which does not look through PHIs.
To catch some more easy shrink-wrapping opportunities, replace with
`getUnderlyingObjects` and check that all of the underlying objects are
known non-stack pointers.
[Transforms] Preserve select profiles in combineMetadata (#233487)
getMergedProfMetadata will return nullptr in circumstances where we can
otherwise preserve the profile. Add a GVN regression test for where this
showed up.
Found from a bootstrapping LLVM build with `-mllvm
-enable-profcheck-in-pipeline`.
Allow 128-bit bounds in DISubrangeType (#231928)
When adding DISubrangeType, I neglected to account for the case of
128-bit array bounds. While unusual, this is valid in Ada, and a type
like this will cause a crash in LLVM:
type Big_Range is range 2**127 - 15 .. 2**127 - 1;
type Big_Array is array (Big_Range) of Boolean;
This patch adds support for this situation to DISubrangeType. DISubrange
is not modified, as gnat-llvm does not need this.
Some extra work was needed in the DWARF writer. In particular,
DW_AT_lower_bound (et al) are in both the 'constant' and 'exprloc'
class, and so DW_FORM_block* will be interpreted as an expression -- one
of the many weirdnesses of DWARF.
To handle this, I've added support for emitting these attributes using
DW_FORM_data16.
[4 lines not shown]
[CIR] Only scan globals and constants for block addresses in GotoSolver (#232870)
The pass walked every op in the module checking inherent attributes for
BlockAddrInfoAttr/BlockAddrDiffAttr. Block addresses only appear in
attributes carried by globals and cir.const values (process() collects
cir.block_address ops directly), so scan just those ops and fold the two
attribute walks into one. The scan assumes module scope, so the pass is
declared on ModuleOp like the other lowering passes.
Successor order now follows attribute order: constant labels precede
cir.block_address labels, and mixed attribute kinds in one initializer
follow initializer order instead of grouping by kind. This matches the
order the pass documents, program behavior is unchanged.
Saves most of the 0.3-3.0s GotoSolver time on the slowest self-host TUs
(e.g. 3.0s -> 0.14s on clang/lib/Sema/SemaExpr.cpp).
[SLP][NFCI] Cache getPointersDiff results (#232083)
SLP recomputes the same pointer distances repeatedly while sorting and
classifying loads, which dominates compile time on large blocks.
Assisted-by: Claude
[LLVMABI][AARCH64] Implement general aggregate classification (#230631)
This implements general aggregate type classification for arguments and
return values on AArch64 targets. Aggregates of at most 16 bytes are
passed or return directly in registers or on the stack. Larger
aggregates are passed or returned indirectly.
Assisted-by: Cursor / various models
[CodeGen] Fix register coalescing early-clobber defs into reserved physregs (#233450)
Problem arises due to `joinReservedPhysRegs` creating a new dead def
`getRegSlot()` without checking if the vreg definition operand was an
early-clobber.
After adding the `isEarlyClobber` check, `getNextNonNullIndex` preserves
the early-clobber index slot kind, so every step of the loop visit the
next instruction at its early-clobber slot. With that the loop get past
the COPY instruction and falsely rejected the join. So to fix this a
`MachineInstruction` iterator is used, rather than checking per slot.
Fixes : https://github.com/llvm/llvm-project/issues/229778
[CIR] Memoize FuncType purity checks in the CIR dialect (#232863)
FuncType::verify walks every type in the signature to prove it only
contains CIR types, and it runs on every FuncType construction in
asserts-enabled builds. Signatures share the same record types over and
over, so the same graphs were re-walked thousands of times per
translation unit (15% of CXXABILowering).
Memoize "purity" per type in the CIR dialect object: the dialect gives
the cache exactly the context lifetime, and a mutex guards the map
(computing outside the lock is safe since two threads store the same
value). Records gain members when completed, so a 'pure' answer is only
cached when the walk saw no incomplete record; 'impure' answers are
always stable.
CXXABILowering on clang/lib/Sema/SemaExpr.cpp (selfhosting) goes from
6.3s to 3.0s and `-emit-cir` from 49.4s to 29.3s (asserts-enabled
build); output is unchanged.
[flang][cuda] Keep delayed descriptor allocs out of CFG loops (#232992)
CUFAllocDelay places a descriptor's cuf.alloc, its initial store and its
fir.declare at the nearest common dominator of all their uses. When a DO
loop lowers to branches, for example because it contains a RETURN, and
the descriptor is used both inside the loop and after it, that block is
the loop header. The group then runs on every trip, including the exit
trip, and the code after the loop reads a fresh, empty descriptor:
```fortran
do jj = 1, 1
if (.false.) return
allocate(clvars(1))
end do
allocate(cvars(1))
cvars(:) = clvars(:) ! Assign: mismatching element counts (to 1, from 0)
```
After computing the common dominator, climb the dominator tree while the
[8 lines not shown]
[PowerPC] Accept .set directive in mem operand (#229992)
It is common practice to give register numbers
symbolic names using .set directives. Parsing
memory operands should accept such identifiers
just like other operands do. This patch just
adds that handling so asm such as:
.set SP,1
std 3, -8(SP)
is handled correctly.
Fixes: https://github.com/llvm/llvm-project/issues/41472
[mlir][ArmSME] Mark integration tests unsupported under ASan on Darwin (#233445)
In an ASan build on macOS arm64, the ArmSME integration tests fail with:
```
Assertion failed: (isInt<28>(Addend) && "Branch target is out of range.")
(RuntimeDyldMachOAArch64.h, encodeAddend)
```
What happens:
- `mlir-runner` JIT-compiles the program to machine code in memory
- That code calls the SME ABI routines (e.g., `__arm_tpidr2_save`),
which live in `libmlir_arm_sme_abi_stubs`
- On AArch64 a direct branch (bl) can only reach about +/-128 MB from
where it is, see
[here](https://support.arm.com/documentation/dui0801/l/A64-General-Instructions/BL--A64-?lang=en)
- The AArch64 backend emits these calls as direct branches and ignores
the code model, so a large code model in the JIT does not help
- Under ASan the library can be mapped farther than that, the offset
[9 lines not shown]
[IndVarSimplify] Ensure fp to int32 conversion stays in bound (#230179)
IndVarSimplify converting floating-point induction variables into integer ones, some cases Exit value of the loop overflows and gives wrong result.
[InferAddressSpaces] Do not rewrite index operands of a constant GEP (#229491)
InferAddressSpaces rewrites flat (generic) pointers into a specific
address space once it can infer one. For constant expressions it does
this in cloneConstantExprWithNewAddressSpace(), which clones the
expression with the pointer operand moved into the inferred address
space.
For a constant getelementptr, that function used to recurse into every
operand, including the indices. Only operand 0, the base pointer, has to
live in the inferred address space. An index is an integer, and it can
be built from a pointer in a different address space, for example
ptrtoint (ptr addrspacecast (ptr addrspace(1) @g to ptr) to i64).
Rewriting it for the base's address space is wrong. In an assertions
build it hits the AddrSpaceCast assert, which expects the cast's source
to already be in the inferred address space:
assert(CE->getOperand(0)->getType()->getPointerAddressSpace() ==
NewAddrSpace)
[11 lines not shown]
[MergeFunctions] Compare alignment of cmpxchg and atomicrmw (#233444)
FunctionComparator compared the alignment of loads and stores but not of
cmpxchg and atomicrmw, so MergeFunctions merged functions whose atomics only
differed in it. The alignment decides whether an atomic is emitted inline or
as a libatomic call.
[LLVM][LowerTypeTests] Drop llvm.type.checked.load in drop-type-tests<all> (#232865)
drop-type-tests<all> dropped `llvm.type.test` but left
`llvm.type.checked.load[.relative]` behind, which the backend cannot
select. Handle them there too via `FatLTOCleanup`'s cleanup helper, now
shared. All mode only: in `Assume` mode a checked load may feed a CFI
trap check that must keep trapping.
[AMDGPU][NFC] Add d16 integer coverage for raw.ptr.buffer.load.format (#231300)
raw.ptr.buffer.load.format didn't have test coverage for the d16
paths, while other related intrinsics (like the non-`.ptr` versions)
did. This commit fixes the issue.
AI disclosure: Claude wrote these tests
Co-Authored-By: Claude Opus 5 (1M context) <noreply at anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply at anthropic.com>
[AMDGPU] Let llvm.amdgcn.raw.ptr.buffer.load.format also return any legal type (#231308)
While llvm.amdgcn.raw.buffer.load.format can handle several float and
integer (and vectors of those) types, and so
can llvm.amdgcn.struct.ptr.buffer.load.format,
llvm.amdgcn.raw.ptr.buffer.format was restricted to floats for some
reason, probably somebody forgetting to loosen it.
This commit fixes the issue so integer-format typed loads can migrate
off of the non-pointer intrinsic.
AI disclosure: Claude noticed it when I asked it draft up a stronger
deprecation for the non-ptr intrinsics, and I wrote the commit message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply at anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply at anthropic.com>
[AMDGPU][NFC] Use ptr ops in ptr.buffer.atomic.fadd_rtn_errors.ll (#231298)
llvm.amdgcn.ptr.buffer.atomic.fadd_rtn_errors.ll had tests that were
using the deprecated `<4 x i32>` intrinsics. This commit fixes the
issue.
AI disclosure: Claude found this and made the change, I wrote this
message and reviewed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply at anthropic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply at anthropic.com>
[mlir][linalg] Fix memory leak in `vectorize-insert-slice-rank-reducing` test (#233435)
The e2e test added in #226490 bufferizes a `tensor.empty` with
`-one-shot-bufferize`, which allocates the buffer but never frees it.
The LeakSanitizer reports the leak in builds with ASAN, see
[here](https://github.com/llvm/llvm-project/pull/226490#issuecomment-6267203174).
With this PR, the lowered IR now contains the `free` instruction.
**Note**: I could not check the LeakSanitizer report directly, as LSan
is not supported on macOS arm64 :(
Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
[SystemZ][z/OS] Escape apostrophes and ampersands in HLASM ALIAS (#233067)
HLASM wants every apostrophe and ampersand inside a character
constant doubled (Language Reference SC26-4940-09, p.131), but the
streamer printed ALIAS C'...' with the external name as is. So a name
with an apostrophe in it, like one from an asm label, ended the string
early and broke the assembly. This adds a small helper that does the
doubling and uses it in all three places that emit ALIAS.
[libunwind] Clear the signal frame state after compact unwind steps (#232823)
`_isSignalFrame` was only updated by DWARF steps. After stepping out of a
signal frame (always DWARF, since compact unwind can't encode them; see
#186458), compact steps left it set, so later return addresses were
treated as exact IPs. Besides misreporting `unw_is_signal_frame`, this
looks up the wrong function's unwind info when a return address is the
start of the next function (e.g. if there was a noreturn call at the end
of a function). Clear the state on compact steps.
Assisted-by: Claude Opus 5.5
