[LAA] Fix off-by-EltSize in negative-step deref bounds check (#211964)
For a negative step, AR at MaxBTC is the lowest accessed address and
must not fall below StartPtr, i.e. `StartOffset >= MaxBTC * |Step|`.
evaluatePtrAddRecAtMaxBTCWillNotWrap compared StartOffset against the
full accessed size, which also includes EltSize. As a result it
rejected loops whose last iteration lands exactly on the base pointer,
and such early-exit loops were not vectorized.
Compare against `MaxBTC * |Step|` instead, and compare unsigned: both
values are unsigned offsets, and the computation type is not
necessarily wider than the index type.
Not addressed here: for a negative step, the end bound Start + EltSize
in getStartAndEndForAccess is neither checked against the deref region
nor saturated on wrap; it is marked with a FIXME for a follow-up.
Related to https://github.com/llvm/llvm-project/issues/211962
[GlobalISel] Canonicalize more constants
Move scalar constants behind casts and copies, and vector splats, to
the RHS of commutative operations. Keep two constant operands in their
existing order.
Retain commuted identity coverage for casts and splats that constant
folding cannot simplify, and update affected AArch64 and AMDGPU output.
[GlobalISel] Fixup commuted combine matching
Follow up 5348847f3c9c by matching the subtraction on either side of
(A-C1)+C2. Check constants and the use count through the matched operands.
Cover the commuted form with constant-to-RHS canonicalization disabled.
[GlobalISel] Auto-commute combine patterns
Emit both operand orders for constrained commutative generic match
patterns, including each use of a PatFrag. Skip redundant permutations
and enforce MaxPermutations before allocating expanded alternatives.
Give each permutation its own operand expansions so operand indices
cannot leak between alternatives.
Keep commuted immediate patterns available with constant-to-RHS
canonicalization, and consolidate AMDGPU carry rules while preserving
zero-addend and RHS-first matching.
Make the integer and AMDGPU multiply/select helpers accept either operand
order. Look through single-use copies when unmerging constants to avoid
extra AMDGPU materializations.
Add emitter documentation and TableGen, MIR, and codegen regressions.
AtomicExpand: Use C int width for atomic libcall ordering argument (#236964)
The memory order argument of the __atomic_* libcalls is an int, so
it is i16 on 16-bit targets. This now matches the RuntimeLibcalls
signature.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[TargetParser][clang] Parse Intel IGCA targets from --offload-arch into OffloadArch (#227468)
Add support for Intel IGCA targets, such as igca_20ca, in
--offload-arch.
Clang now parses an IGCA target from the --offload-arch value and stores
it as an OffloadArch.
This code will be used later in clang in both the driver and the
frontend to enable clang support for compiling SYCL programs with
support for specific IGCA targets.
Assisted-by: Claude code (+ some Github copilot)
---------
Co-authored-by: Tom Honermann <tom at honermann.net>
Co-authored-by: srividya sundaram <srividya.sundaram at intel.com>
[AMDGPU] Define a feature to disable s_add_pc_i64, not enable it. NFC. (#231179)
This is in the hope that the instruction is only unusable on a limited
number of subtargets. It should be usable on all future subtargets.
[bazel] Remove default license type annotations
Since at least Bazel 7.7, https://bazel.build/versions/7.7.1/reference/be/common-definitions, this has been unused:
> This is part of a deprecated licensing API that Bazel no longer uses. Don’t use this.
We could consider integrating the recommended replacement [`@rules_license`](https://github.com/bazelbuild/rules_license) to properly propagate license info, but I don't see a compelling reason to do so given that:
- downstream rules don't seem to need this, as it doesn't exist today
- llvm-project has minimal external dependencies, so the Bazel rules would have minimal benefit from license verification
[InstCombine] Fold sub nuw (and X, LowMask), (zext Y) to zext (sub nuw (trunc X), Y) (#231358)
Fold "Sub nuw (and X, LowMask), (zext Y)" to "zext (sub nuw (trunc X),
Y)",
"Sub nuw (zext Y), (and X, LowMask)" to "zext (sub nuw Y, (trunc X))",
when LowMask has exactly as many low bits set as Y's width,
Then 'and' is zext(trunc X), and the wide nuw implies the narrow
subtraction cannot wrap.
This pattern is produced for (X & 0xFFFF) / C * C: InstCombine
canonicalizes it to X - X % C, and CorrelatedValuePropagation narrows
the
urem to i16. The i32 sub of a zext'd i16 urem blocks the backend from
folding X - (X - (X / C) * C) after the urem is expanded, leaving two
redundant subtractions on AMDGPU, X86, AArch64 and ARM.
Proof: https://alive2.llvm.org/ce/z/euQXNHhttps://alive2.llvm.org/ce/z/FznUvR
[3 lines not shown]
[AMDGPU] Re-enable verifier in shufflevector test (#236871)
The scheduler bug behind #130884 was fixed by #227897.
Fixes #130884.
Co-authored-by: Petar Jovanovic <petar.jovanovic at amd.com>
[SandboxVec][LoadStoreVec] Don't pack non-integral pointer constants
getEquivalentConstantWithType() only rejected a non-integral pointer
destination type (via an assert), while a non-integral source such as
'ptr addrspace(5) null' was silently converted with ptrtoint. A
non-integral pointer has no stable integer representation, so it cannot
be packed into an integer lane.
Add VecUtils::isNonIntegralPtr() and return nullptr from
getEquivalentConstantWithType() when either type is a non-integral
pointer, which makes createConstantVector() give up on the bundle.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Add LLVM_ABI annotations to Reinterpret
Export the out-of-line static members of Reinterpret, as reported by
the LLVM_ABI annotation check. getCastChain() is private but is called
from the inline canReinterpret(), so it needs to be exported too.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Add Reinterpret helper for same-size type casts (NFC)
Replace getEquivalentConstantWithType() with a Reinterpret utility that
reinterprets the bits of a value as another type of the same size. It
builds a chain of ptrtoint / bitcast / inttoptr casts, and applies it
either to a constant (folding) or to a runtime value (inserting casts).
The runtime form will be used to unpack mixed-type vector loads and to
pack mixed-type store operands.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
RuntimeLibcalls: Move initExtensionsForTriple from TargetLibraryInfo
Move the i32 extension ABI rules into RuntimeLibcallsInfo so they can
be used to compute libcall signatures. TargetLibraryInfo now starts
calling into RuntimeLibcallsInfo to use it. Eventually these 2 will
merge.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[mlir][acc] Add barrier after thread-partitioned loops in sequential nests. (#236961)
A loop partitioned across threads inside a sequential loop leaves its
results spread over threads. Later code in the nest, or the next
iteration, can read or overwrite them before other threads have stored
them. Add a block-level barrier after such a loop when a later
access conflicts with it on the same storage (read after write, write
after read, write after write).
Accesses to thread-private memory never conflict. Compute region inputs
are traced back to the value they are passed from, so two inputs that
name
the same storage conflict. Loops nested in a worker loop are skipped,
since a per-row barrier already follows them there.
[PowerPC] fix SPE offset miscompilation (#233220)
The SPE doubleword load and store (evldd and evstdd) have a 5-bit field
for the offset, which is scaled by 8. the logic checked that the
displacement fit in a u16, but for these instructions it should use a
stricter check.
https://godbolt.org/z/Ev7nGq74b
```llvm
declare void @use(ptr)
define void @f(double %x) "target-features"="+spe" {
%d = alloca double, align 8
%a = alloca [256 x i8], align 8
%big = alloca [32768 x i8], align 8
store i8 0, ptr %a, align 8
store double %x, ptr %d, align 8
call void @use(ptr %big)
[42 lines not shown]
AMDGPU: Remove kill flags on virtual registers from MIR tests
Soon the verifier will disallow this. Delete redundant tests which
only differed from others by kill flag uses. twoaddr-bundle.mir relied
on the input kill flags in TwoAddressInstruction, so compute
LiveIntervals first as the real pipeline does.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Support mixed-type store vectorization of arbitrary values
vectorizeStores() only handled chains whose stored values were all
consecutive loads or all constants. Any other chain, e.g., storing
function arguments or computed values, was left alone.
Pack such values into a vector with createPack(). For a chain of mixed
types the vector has integer lanes, so each value is first
reinterpreted (bitcast / ptrtoint) as the lanes it occupies with
Reinterpret::reinterpretValue(), and a value spanning several lanes is
inserted lane by lane. The pack is placed right before the vector
store, where every stored value is available. Whether this pays off is
left to the existing cost check in acceptOrRevert().
Chains storing a non-integral pointer are left alone.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Vectorize store chains whose loads have external uses
vectorizeStores() used to give up on a load-store chain if any of the
loads had a user other than its store. Now that loads of mixed types
can be unpacked from a vector load, vectorize such chains as well: the
loads become a single vector load, and the external users are fed from
the lanes each load occupies, cast back to the load's type if needed.
The unpacking loop of vectorizeLoads() is factored out into
unpackLoads() (along with the canUnpackLoads() check) and shared by
both paths. Chains with a non-integral pointer load that has external
users are still left alone.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Move contiguity check into canVectorizeLSV() (NFC)
Both vectorizeStores() and vectorizeLoads() (via createVectorLoad())
checked that the bundle is consecutive before calling canVectorize().
Move that check into LegalityAnalysis::canVectorizeLSV(), reporting
ResultReason::NotConsecutive, which mirrors how canVectorize() checks
consecutive loads and stores for BundleVec.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Support mixed-type load vectorization
Remove the restriction that every load in a top-level load chain must
have the vector's element type. A chain of mixed types such as
{i32, float} or {double, i32, i32} is loaded as a vector of integers
(as chosen by VecUtils::getCombinedVectorTypeFor()), each original load
is unpacked from the lanes it occupies, and then reinterpreted back to
its own type with Reinterpret::reinterpretValue().
Chains involving a non-integral pointer are left alone, since such a
pointer cannot be recovered from an integer lane.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Add Reinterpret helper for same-size type casts
Replace getEquivalentConstantWithType() with a Reinterpret utility that
reinterprets the bits of a value as another type of the same size. It
builds a chain of ptrtoint / bitcast / inttoptr casts, and applies it
either to a constant (folding) or to a runtime value (inserting casts).
The runtime form will be used to unpack mixed-type vector loads and to
pack mixed-type store operands.
Also add VecUtils::isNonIntegralPtr(). A non-integral pointer has no
stable integer representation, so it is no longer packed into an
integer lane. Previously only a non-integral destination type was
rejected (via an assert), while a non-integral source such as
'ptr addrspace(5) null' was silently converted with ptrtoint.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Add LLVM_ABI annotations to Reinterpret
Export the out-of-line static members of Reinterpret, as reported by
the LLVM_ABI annotation check. getCastChain() is private but is called
from the inline canReinterpret(), so it needs to be exported too.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[SandboxVec][LoadStoreVec] Move contiguity check into LegalityAnalysis (NFC) (#236636)
Both `vectorizeStores()` and `vectorizeLoads()` checked that the bundle
is consecutive before calling `canVectorize()`. Move that check into
`LegalityAnalysis::canVectorizeLSV()`, reporting
`ResultReason::NotConsecutive`, which mirrors how
`LegalityAnalysis::canVectorize()` checks consecutive loads and stores
for BundleVec.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[MachineSink] Track use/def when shouldPostRASink() is false (#236267)
In `tryToSinkCopy()`, it currently is missing a call to
`LiveRegUnits::accumulateUsedDefed()` when `shouldPostRASink()`
is false. This can cause a COPY to be sunk past a user
because it looks like the COPY def doesn't have any users in
the block.
I don't have a lit test because there are currently no in-tree
targets that exercise this path.
[SandboxVec][LoadStoreVec] Don't pack non-integral pointer constants
getEquivalentConstantWithType() only rejected a non-integral pointer
destination type (via an assert), while a non-integral source such as
'ptr addrspace(5) null' was silently converted with ptrtoint. A
non-integral pointer has no stable integer representation, so it cannot
be packed into an integer lane.
Add VecUtils::isNonIntegralPtr() and return nullptr from
getEquivalentConstantWithType() when either type is a non-integral
pointer, which makes createConstantVector() give up on the bundle.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
