[ARM] Optimize code for 0 and int min (#229489)
## Summary
Optimize Thumb-2 code generation for zero tests of values masked by
contiguous low-bit masks. For these tests, shifting the input can
preserve whether the masked value is zero while avoiding a `BIC`.
The four source forms covered by this pattern reduce to one of two
zero-test shapes: an `i32` test or an `i64` test lowered across two
`i32` halves.
## Changes
- Add a helper to recognize eligible single-use ANDs with contiguous low
masks and form an equivalent shifted value for a zero test.
- Apply the transform to Thumb-2 equality and inequality comparisons
against zero, including the OR of halves produced when an `i64`
comparison is lowered.
[24 lines not shown]
[MachineLICM] Don't request BFI analysis if not needed. NFC (#235638)
BFI is only needed in one place, isTgtHotterThanSrc, which is only
called when there's profiling info. Lazily request it to save 0.06%
geomean on stage2-O3:
https://llvm-compile-time-tracker.com/compare.php?from=36c4d811b59c826b416fd66518f406c96829d4be&to=c57789d7ebd8288fe063fd7b33481e88eed80527&stat=instructions:u
I also considered some sort of lambda that's called directly from
isTgtHotterThanSrc, but it turns out that's not NFC as the MachineLICM
can split the preheader before it's called so the BFI analysis returned
is slightly different. Might be worth exploring later.
[RISCV] Lazily request analyses in RISCVGatherScatterLowering and RISCVVLOptimizer. NFC (#235546)
We skip RISCVGatherScatterLowering and RISCVVLOptimizer when there's no
V extension (or fixed length vector support), but after we request any
analyses. So on no-V configurations we'll still end up computing the
analyses. Fetch these lazily so that we don't unnecessarily compute them
in the new PM. The legacy PM will still redundantly compute them.
This also has the benefit that if e.g. there are no gather/scatters in a
function or no instructions with an optimisable VL, we also won't
request the analyses.
[libc] Implement thrd_sleep and thrd_yield in threads (#232919)
Implement the standard C11/C23 thread functions `thrd_sleep` and
`thrd_yield` in `<threads.h>`.
Fixes #232378.
Assisted by Antigravity and reviewed by Aman Maurya.
[AArch64] Support FP atomic stores with memory hints
This patch extends the atomic-store-hint pseudos to accept preserved
floating-point atomic stores. It adds an FPR16 pseudo family and direct
half, bfloat, float, and double patterns for the supported addressing
modes.
The pseudos expand relaxed stores to STR and RCPC3 release and
sequentially consistent stores to STLUR, retaining memory hints without
cross-register-bank copies.
The existing AArch64::ATOMIC_STORE_HINT_H is renamed to
AArch64::ATOMIC_STORE_HINT_HH and AArch64::ATOMIC_STORE_HINT_H now means
FP16 instead of it's old association with integer half-word.
[AArch64] Support direct RCPC3 FP atomic loads and stores
Preserve half, bfloat, float, and double atomic loads and stores
through AtomicExpand for acquire and release orderings, and preserve
sequentially consistent stores, when RCPC3 and NEON are available.
Add SelectionDAG patterns for the RCPC3 LDAPUR and STLUR operations,
including unscaled offsets, and update the existing FP lane patterns to
match the preserved floating-point atomic nodes. Targets without the
required features continue to use integer lowering.
[CIR] Add long_double_size to #cir.lowering_lang_options (#234946)
It represents `LangOpts::LongDoubleSize` which is initialized if
`-mlong-double-*` command line option is present.
Relates to https://github.com/llvm/llvm-project/issues/227917 .
[mlir][NVVM] Split enums and attributes out of NVVMDialect.h
Many users of the NVVM dialect outside of the dialect itself only need its
enums and attributes but had to include `NVVMDialect.h`, which pulls in
the declarations of all NVVM ops and their includes.
Add a new `NVVMAttributes.h` header that declares only the NVVM enums and
attributes, and use it where applicable. `NVVMDialect.h` now contains
only the op declarations and drops the includes that are no longer needed.
The definitions were split out of NVVMDialect.cpp as well to keep a
traditional header-source file split.
Co-authored-by: Claude <noreply at anthropic.com>
[unittests] Skip target tests when the default triple is not built
PassBuilder and TargetMachine C API tests FAIL when LLVM_TARGETS_TO_BUILD
excludes the host triple. Skip instead, and avoid tearing down an
uninitialized TargetMachine after a skipped SetUp.
Co-authored-by: Cursor <cursoragent at cursor.com>
[SCEV] Do not consider base pointer defining scope for offset flags in `getGEPExpr` (#222713)
The defining scope of the base pointer of a GEP was previously being
used to infer no-wrap flags on the SCEV sub-expressions for the offset,
which however only depend on the index operands. Ensure the scope
bound for the offset flags is therefore derived from those alone.
Fixes: https://github.com/llvm/llvm-project/issues/221524.
[AArch64] Support direct relaxed FP atomic loads and stores
This patch preserves half, bfloat, float, and double atomic loads and
stores through AtomicExpand when they use unordered or monotonic
ordering and the target supports floating point.
It adds SelectionDAG patterns that select the accesses directly in the
floating-point register bank. Unsupported targets, stronger orderings,
and stores carrying memory hints continue to use integer lowering but
will be handled in the subsequent PRs.
[Analysis] De-templatize TBAANodeImpl and TBAAStructTagNodeImpl (NFC) (#235609)
This patch removes MutableTBAANode and MutableTBAAStructTagNode while
turning TBAANodeImpl and TBAAStructTagNodeImpl into non-template classes
TBAANode and TBAAStructTagNode, respectively.
Note that the last uses of MutableTBAANode and MutableTBAAStructTagNode
were removed on November 3, 2017 in commit
4b77f463d0838bc2bae34b05e2dab9e1d7387dae, leaving TBAANode and
TBAAStructTagNode as the only instantiations of TBAANodeImpl and
TBAAStructTagNodeImpl, respectively.
Assisted-by: Antigravity
[flang] Replace PointerUnion::dyn_cast with llvm::dyn_cast (NFC) (#235608)
PointerUnion::dyn_cast has been soft-deprecated in favor of
llvm::dyn_cast and llvm::dyn_cast_if_present. This patch replaces
PointerUnion::dyn_cast with llvm::dyn_cast.
Note that AliasAnalysis::Source::SourceOrigin::u is always initialized
with nonnull values, and other methods in AliasAnalysis.cpp already
use llvm::dyn_cast and llvm::cast on origin.u.
This patch completes the migration away from PointerUnion::dyn_cast
under flang/.
Assisted-by: Antigravity
[OpenMP] Add runtime selection for metadirective with non-constant conditions. (#192455)
This patch implements runtime evaluation and selection for `OpenMP`
metadirectives when user conditions are non-constant expressions.
[AArch64] Prevent debug instructions and pseudo probes from changing post-RA scheduling (#234920)
A `DBG_VALUE` or `PSEUDO_PROBE` between an ordinary AArch64 instruction
and a
following CFI instruction hides the CFI from
`AArch64InstrInfo::isSchedulingBoundary`. The `PostMachineScheduler` can
then
include the instruction in the preceding region and reorder it, so
non-code
metadata changes the final instruction order.
Look past debug instructions and pseudo probes when finding the
following CFI
boundary. A focused `postmisched` MIR test covers both cases.
This is related to, but distinct from, #36588. That issue concerned CFI
instructions introduced by debug-info emission. Here the CFI boundary is
already present, but an intervening debug instruction or pseudo probe
hides it.
[2 lines not shown]
[JumpTableToSwitch] Call values JT everywhere, rename jump table type to JumpTable (NFC) (#235449)
Currently `expandToSwitch` uses `JT` for what rest of the code calls
`JumpTable`. Call it `JT` everywhere. With this we also drop the
redundant `Ty` suffix in `JumpTableToTy`.
[AMDGPU] Ignore debug instructions in exec-mask search limits (#232560)
Two searches in SIOptimizeExecMasking charge debug instructions against
their instruction limits. With enough debug instructions, a nearby exec
copy is not found and an exec-copy or saveexec fold is missed.
Count only ordinary instructions in both searches.
Assisted-by: OpenAI Codex 5.6 Sol, co-reviewed by Claude and Yongqiang
Tian.
[runtimes][offload] Add missing dependencies to check-offload
After configuring an OpenMP/offload build, running `ninja check-offload`
without first running the build doesn't build all the necessary
dependencies like flang or clang-linker-wrapper, or the runtimes.
Claude assisted by iteratively trying to run check-offload and adding
dependencies as they were needed.
[compiler-rt][test] Fix sanitizer_coverage_trace_pc_guard-init.cpp on Android (#235625)
Use `%dynamiclib1` and `%ld_flags_rpath_so1`/`%ld_flags_rpath_exe1` so
`DT_NEEDED` does not record the host path to the shared library.
Fixes test failure on
https://lab.llvm.org/buildbot/#/builders/sanitizer-x86_64-linux-android
Assisted-by: Gemini
[Offload] Add properties to olGet*Info to retrieve native handles (#234186)
Adds a property to be able to retrieve the native handle of a queue,
device and context to their respective olGetInfo routines.
This is needed to implement interopability features on top of liboffload
like OpenMP's interop directive.
Assisted by Claude.
[mlir][linalg] Remap linalg.index when packing a structured op (#221430)
## Problem
Packing a structured operation splits an original loop dimension `d`
into outer and inner dimensions. The operation's region is moved to the
packed `linalg.generic` unchanged, so `linalg.index d` starts returning
only the outer tile coordinate.
The resulting IR verifies, but silently computes incorrect index values.
## Approach
Track the inner loop dimension introduced for each packed loop and
rewrite:
linalg.index d
to:
[20 lines not shown]
[CodeGen] Don't fold a load into an inline asm that may overwrite it
The peephole optimizer folds a load from a stack slot into a foldable
inline asm operand, which then reads the slot itself. An asm that writes
the slot first (through an early-clobber "=&m" output) then read what it
wrote, so a load isn't folded into an asm that stores, unless the slot
is invariant. Nor is a load that doesn't fill the register, e.g. a
32-bit load into an XMM register, since the asm would read the whole
register's worth of memory.
Assisted-by: Claude Opus 5.5
[CodeGen] Keep an inline asm's memory accesses described after a fold
The memoperands of an instruction describe all of its memory accesses.
Folding gave an inline asm a single memoperand, for the stack slot,
which said that it accessed no other memory even when it had an "m"
operand or a "memory" clobber. Without memoperands, the asm was taken to
access any memory, so it now gets one for any memory as well.
Assisted-by: Claude Opus 5.5
[CodeGen] Prefer spilling a register an inline asm can fold
InlineSpiller rematerialized a value passed to a foldable inline asm
operand (e.g. a constant) into a register right before the asm, which
the asm may not have had, since registers running out is why the value
was spilled. It's now spilled and folded instead.
An asm that can fold a register also no longer adds to the register's
spill weight, since it needs no reload or spill for it. The allocator
then spills that register before the asm's other operands, which need
registers, which saves spills in two_tied_rm and tied_rm_undef.
Assisted-by: Claude Opus 5.5
[CodeGen] Fold an inline asm subregister operand at its slot offset
A stack slot holds the whole register, and a folded operand referred to
the start of the slot, so a subregister was folded only if it was at
offset 0, and other subregisters got a reload into a register the asm
may not have. Coalescing can turn an i686 "rm" byte operand into
sub_8bit_hi of a GR32, which is at byte 1. Fold a subregister at its
offset in the slot, from getStackSlotRange(), so
getFrameIndexOperands() now takes an offset.
Also add a third asm to qm_tied_high_byte, which clobbers every GPR, so
that it reaches the folder. Without it, the value is reloaded into a
register that's free at the second asm.
Assisted-by: Claude Opus 5.5
