[LV] Add tests for divergent main/epilogue interleaved group widening decisions. nfc (#232423)
The test case is a interleaved load with tail gap, and allow scalar
epilogue by -tail-folding-policy=dont-fold-tail.
- AArch64/epilogue-vectorization-interleave-gap-divergence.ll:
Main vectorized loop VF is 16, choose interleaved load, and force scalar
epilogue. Epilogue vectorization loop VF is 4, choose to scalarize th 2
loads, and should not need to force scalar epilogue.
- RISCV/epilogue-vectorization-interleave-gap-divergence.ll
Main vectorized loop VF is 64, choose 2 strided loads, and should not
need to force scalar epilogue. Epilogue vectorization loop VF is 4,
choose interleaved load, and force scalar epilogue.
- X86/epilogue-vectorization-interleave-gap-divergence.ll
Main vectorized loop VF is 64, choose to scalarize th 2 loads, and
should not need to force scalar epilogue. Epilogue vectorization loop VF
is 4, choose interleaved load, and force scalar epilogue.
Pre-commit test for #213904
[AMDGPU][VOPD] Make VOPD pairing load-overlap check deterministic (#232317)
loadsMayOverlap() fills a lazy cache while iterating a SmallPtrSet<SUnit*>.
This introduces potential non-determinism (which we observed in some
tests) so switching to Use SmallSetVector so iteration follows insertion
order. Non-determinism first introduced by #201930.
Claude was used to assist with this fix.
[AMDGPU] Prevent meta instructions from triggering hazard checks (#233072)
getWaitStatesSince calls hazard predicates for meta instructions even
though they do not issue. As a result, a DBG_VALUE reading EXEC can look
like a real dependency to the VCMPX EXEC WAR check and cause an
unnecessary S_WAITCNT_DEPCTR when debug information is present.
Skip meta instructions centrally in getWaitStatesSince instead of
handling them in individual hazard predicates. This also matches the
existing behavior in UpdateState.
This new solution was suggested by @jayfoad.
[InferAlignment] Infer alignment from dominating branch conditions (#232022)
Use DomConditionCache when querying known pointer bits, allowing
InferAlignment to derive stronger alignment from dominating conditions.
Fixes https://github.com/llvm/llvm-project/issues/231240.
[GlobalISel] Ignore debug instructions in extracted-load scan budget (#229304)
matchCombineExtractedVectorLoad currently counts debug instructions
while bounding the scan between a vector load and its extract use. With
enough DBG_VALUE records, this suppresses scalar-load folding and makes
debug information change generated code.
Use the established instructionsWithoutDebug helper, matching the other
bounded GlobalISel load-fold scans. The regression test covers
equivalent functions with and without in-window debug values.
Assisted-by: OpenAI Codex 5.6 Sol
Assisted-by: Claude (review)
[ARM] Ignore debug instructions in shouldCoalesce (#231883)
`ARMBaseRegisterInfo::shouldCoalesce` computes its coalescing budget
from
`MBB->size()`, so debug instructions can raise the size multiplier and
change
code generation.
Use `instructionsWithoutDebug` while preserving the existing treatment
of
bundle interiors and pseudo probes. The regression test is derived from
`out-of-registers.ll` and checks that the debug and stripped forms
produce the
same result.
[InstCombine] Drop poison-generating annotations forming min/max for clamp (#231537)
When converting select to min/max intrinsics we have to be careful about
poison-propagation, because min/max propagate unconditionally, while
select does not. Normally, this is not a problem, because the select
arms are also used in the select condition, which propagates
unconditionally.
However, SPF also matches special patterns where this may not be the
case. In particular, for `(X <u C1) ? C1 : UMIN(X, C2) ==> UMAX(UMIN(X,
C2), C1)` style transforms, if the inner min/max has poison-generating
annotations, the result can be more-poisonous.
I'm fixing this in a somewhat hacky way, by dropping the poison
annotations as part of SPF canonicalization. This makes assumptions
about the kind of patterns SPF matches. Originally, I wanted to just
move this code out of SPF into a dedicated InstCombine fold, but this
did not work around because SPF also handles cases with casts in
between, and I did not want to replicate that logic.
For https://github.com/rust-lang/rust/issues/163728.
[AArch64] Prevent debug info from changing stack tag slot selection (#230935)
`AArch64StackTaggingPreRA::findFirstSlotCandidate` ranks tagged stack
slots by uses of their `TAGPstack` result. Its `use_instructions` walk
also counts `DBG_VALUE`, so adding one debug record can change which
slot is pinned to offset zero. The reduced, verifier-valid IR has two
tagged allocas and a `#dbg_value` of the first `llvm.aarch64.tagp`
result. With that record, `irg` uses `x0` and `addg` uses `x1`; without
it, those registers are reversed.
I update the `use_instructions` by `use_nodbg_instructions`. The new
`llc` test compiles the same IR both with its debug record and after
`opt -strip-debug`, checking that the slot choice stays the same. On the
unmodified base, the run with the debug record fails its `irg` check;
both runs pass with the fix. The new test and nearby stack-tagging
CodeGen tests pass on LLVM main at `52246606581f`.
The trigger uses a direct `llvm.aarch64.tagp` call in IR. The IR has
been verified by `llc -verify-machineinstrs`.
[compiler-rt] Build the builtins library for AArch64 big endian (#225996)
Clang supports aarch64_be as a target, but compiler-rt cannot be
configured for it: the builtins architecture list omits it, and the
default-target detection misclassifies an aarch64_be triple as aarch64
because CMake's MATCHES is a substring match. Match aarch64_be before
the generic aarch64 branch, add it to the ARM64 list in
builtin-config-ix.cmake, and map its sources to the aarch64 sources.
[Mips] Lower calls to private functions the same way as internal functions (#229975)
As with 85b4fbfecb65cd1f5a06e707e7ec0fc2de562407, this seems to be
another case where the backend was treating private functions
differently for no apparent reason.
This was particularly problematic for building the Zig compiler with
LLVM linked statically because we'd have enough such calls that linking
failed with thousands of these errors:
relocation R_MIPS_CALL16 out of range: 76048 is not in [-32768, 32767]
(cherry picked from commit 22371aefb10a31207de828cff7ef7511e9a552f7)
[clang][bytecode] disable tail calls on mips (#229679)
As for the other architectures, this caused a Clang build for MIPS to
fail.
(cherry picked from commit cbc387549f1d218959827d90c4d45541f74b7da2)
[mlir][bufferization] Remove memref.alloc-only check for while exit aliases (#232834)
The `scf.while` exit-alias path added in #226951 currently restricts
allocations to `memref.alloc`. With the realloc interface model fixed in
#231608, I think we can remove that restriction.
This allows any eligible allocation op that supports
`HoistingKind::Loop` to use that path, which at this time means an
extension to `memref.alloca`.
Also remove the allocation value from the hoisting state since only that
check used it.
[RISCV] Ignore debug uses in vector peephole folds (#230319)
A `DBG_VALUE` can make several RISC-V vector peephole folds see an extra
use and skip a transformation, so adding debug information changes the
resulting Machine IR. The new MIR test covers five affected folds. One
triggering input is:
```text
%x:vr = PseudoVADD_VV_M1 %passthru, $noreg, $noreg, 4, 5, 0
DBG_VALUE %x, $noreg, !8, !DIExpression(), debug-location !10
%y:vr = PseudoVMV_V_V_M1 %passthru, %x, 4, 5, 1
```
Without the debug use, the pass folds the move into the source
operation; with it, the original pass leaves both instructions. Use
`hasOneNonDBGUse()` in the affected checks. `LiveDebugVariables` handles
debug locations left by rewritten or erased definitions.
As requested in review, the first commit records the original behavior
and passes without the fix. Subsequent commits update the checks for the
corrected behavior.
[AMDGPU][NFC] Use MCRegister instead of MCPhysReg in SIRegisterInfo::get32BitRegister (#232830)
Changes the signature of `get32BitRegister ` to avoid casts at call sites. The callers pass Register or MCRegister values and keep the result in Register or MCRegister and getMatchingSuperReg returns MCRegister, so use MCRegister instead of narrowing through MCPhysReg.
Fixes several instances of MSVC warning C4244 ("possible loss of data") across multiple files in llvm/lib/Target/AMDGPU.
Assisted-by: Claude <noreply at anthropic.com>
[AMDGPU][NFC] Change disassembler callback parameters from unsigned to uint64_t (#215190)
Every call from the TableGen-generated disassembler to a decoder callback in the AMDGPU backend narrows implicitly: the caller passes a uint64_t and the callback declares its operand parameter as unsigned. There are around 5900 such call sites in AMDGPUGenDisassemblerTables.inc.
This patch changes the parameter type of the callbacks from unsigned to uint64_t so that it matches the caller, and adds a static_cast where the value is forwarded on to an interface that takes unsigned. Changing the TableGen disassembler backend to emit an explicit conversion would change the generated .inc file of every target. This patch addresses AMDGPU only.
Assisted-by: Claude <noreply at anthropic.com>
[mlir][arith] Preserve unsigned values when narrowing operations (#233323)
Require unsigned representability for `divui`, `ceildivui`, `remui`, and
`shrui`. A signed range fitting the narrower type does not preserve the
unsigned operand values, even when the result fits.
Add concrete counterexamples and coverage for safe narrowing of
zero-extended and proven non-negative operands.
Assisted-by: Codex
[mlir][arith] Check overflow when reassociating integer constants (#233155)
Folding `(x + 100 nsw) + 100 nsw` to `x + (-56) nsw` in `i8` introduces
poison for inputs such as `x = -100` that were previously defined.
Check signed and unsigned overflow when folding constants in the
add/sub/mul reassociation patterns. Retain each common nsw/nuw flag only
when its constant arithmetic does not overflow.
Assisted-by: Codex
[ELF] --print-icf-sections: print classes in input order (#233345)
... instead of the hash-sorted section order, so a change to the hash no
longer reorders the output.
[SLP] Handle REVEC when costing expanded compress mask (#232810)
With REVEC the scalars are vectors.
After https://github.com/llvm/llvm-project/pull/135174 the
masked-load-compress
shuffle that is emitted uses the compress mask expanded to the elements
of
those vectors (transformScalarShuffleIndiciesToVector), but the cost was
left
computed with the unexpanded per-vector mask, so a shuffle such as <24 x
i8> ->
<16 x i8> was costed as a <2 x i8> shuffle. This made the compress
sequence
looked cheaper than it is.
Assisted-By: Claude Sonnet 5.5
