[CIR][AArch64] Handle constrained Neon FMA and sqrt (#218307)
Propagate expression FP options through AArch64 builtin emission and
attach the active FP environment to CIR `cir.fma` and `cir.sqrt`
operations. This allows strict FP operations to lower to constrained
LLVM intrinsics while preserving unconstrained behavior.
Co-locate classic Clang, CIR, and CIR-to-LLVM coverage under
command-line strict mode. For FP16 FMA and sqrt operations, also test a
pragma overriding a command-line maytrap setting.
Place this coverage in focused `-constrained.c` files alongside the
corresponding Neon test families. Move the matching classic constrained
checks from the legacy files and remove their superseded blocks, while
retaining unrelated legacy tests. Existing unconstrained coverage
remains in the regular Neon tests.
Track FMA operands through ABI conversions and lane selection, verify
the constrained results are returned, and check that lowering does not
[4 lines not shown]
CodeGen: Remove TRI arguments from TargetInstrInfo hooks (#228164)
Continue with cleanups enabled by #158224.. TRI can now always directly
be referenced from TargetInstrInfo
Co-authored-by: Claude Opus 5 <noreply at anthropic.com>
AMDGPU: Remove promotion of 32 and 64-bit atomic load/store types
The atomic load and store patterns now cover all 32 and 64-bit register types,
so there is no need to coerce to integer for selection. Mark the 16-bit vector
types as legal for atomic load and store rather than expanding them.
This fixes failing on atomic load and store of v2bf16 and v4bf16 which were
never promoted and ended up expanded.
The f16 and bf16 scalar cases are still promoted since the 16-bit atomic
patterns only cover i16 and i32, though that also should be fixed.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[clang] Replace PointerUnion::dyn_cast with llvm::dyn_cast (NFC) (#229682)
PointerUnion::dyn_cast has been soft-deprecated in favor of
llvm::dyn_cast and llvm::dyn_cast_if_present. This patch replaces the
former with llvm::dyn_cast where the operand is guaranteed to be
nonnull.
Note that AccessPath::Base is always initialized to a nonnull pointer.
Assisted-by: Antigravity
[AArch64] Use multi-vector intrinsics for masked load/store users of predicate-as-counter
If the user of the original wide mask is a masked load or store
intrinsic (matching the element size of the predicate-as-counter),
rewrite it directly to a masked multi-vector load/store.
This avoids materializing the vector mask and is easier to handle here
than later (e.g. in SelectionDAG), since we do not need to match the
concatenation of all `pext` segments of the predicate-as-counter.
Assisted-by: Codex
[AArch64] Avoid materializing full masks for extractelement users of predicate-as-counter
If the user of the original wide mask is an `extractelement` and the
index is known to be within the first segment of the
predicate-as-counter, replace it with `extractelement(pext(counter, 0))`.
This avoids materializing the vector mask and produces a form that can
be folded into a conditional branch when the predicate-as-counter is
produced by a `whilelo`.
Assisted-by: Codex
[AArch64] Add predicate-as-counter loop rewrite pass (#220960)
This patch adds an AArch64 IR loop pass that rewrites wide loop-carried
`llvm.get.active.lane.mask` phis to predicate-as-counter `whilelo` phis
(for SVE2.1 or streaming SME2 targets).
Users of the original mask are preserved by materializing vector
predicates with `aarch64.sve.pext`.
The element size and vector scale (VLx2 or VLx4) of the
predicate-as-counter is inferred from the mask load/store users within
the loop. These could be optimized to multi-vector loads/stores (though
that is not included in this patch).
For example, a loop like:
```
entry:
%step = vscale x 64
[31 lines not shown]
[Analysis] Remove unused CallGraphSCC methods (NFC) (#229683)
The last callers of CallGraphSCC::ReplaceNode and
CallGraphSCC::DeleteNode were removed on July 12, 2024 in commit
58bc98cd3abd72226cdbaa05bd92af9598d491db.
Their removal leaves the private member variable Context unused, so this
patch also removes Context and simplifies the CallGraphSCC constructor.
Assisted-by: Antigravity
Mips: Mark the assembler temporary def dead in inserted branches
The branch instructions reserve $at for long branch expansion, which
only uses it as a scratch. Instruction selection already marks the def
dead; branches rebuilt by later passes did not, which blocked
BranchFolding from hoisting common code past them.
Co-authored-by: Claude Opus 5 <noreply at anthropic.com>
[IRBuilder] Accept Module instead of LLVMContext in ctor (#229430)
This adds new constructor overloads, which accept `Module &` instead of
`LLVMContext &` in the ctor, and replaces nearly all usages of the old
constructors. A few more complicated cases (C API, SandboxIR context and
one unit test) are left alone for now.
The intent is to remove the old LLVMContext constructors and make
availability of the Module required at time of construction (either
directly passed to the ctor or implied by the insertion point).
The motivation for this is to ensure that IRBuilder always has an
available DataLayout. IRBuilder used to be DL-agnostic, but at this
point we've accumulated a number of key places which require a data
layout, and get it from the insertion point instead. This is not great,
because the insertion point is not actually required to be set when
producing instructions (they just won't be inserted).
Additionally, not having a required DataLayout also introduces the
[11 lines not shown]
[SPIRV] Re-sort basic blocks after LoopSimplify for non-shader targets (#229678)
SPIRVPrepareFunctions sorts basic blocks in reverse post-order so that
dominators appear before all blocks
they dominate, as required by SPIR-V.
However, since #187519 (4a773b9f35fc), LoopSimplifyPass runs later in
addISelPrepare for non-shader
targets as well. For shader targets, SPIRVStructurizer subsequently
re-runs sortBlocks(F),
but for non-shader targets no pass restored the block order, allowing
dedicated exit or preheader blocks
inserted by LoopSimplify to appear before their dominators.
Call sortBlocks(Func) for non-shader targets at the start of
SPIRVEmitIntrinsicsImpl::runOnFunction.
AMDGPU: Use normal load/store pattern type lists for atomics
Atomic load and store of vector types are now permitted in the IR. Instead of
maintaining separate scalar-only atomic pattern lists, cover atomic load/store
in the existing per-register-type pattern loops. As a side effect
-flat-for-global is respected in more cases.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[clang-tidy] Make range source classes configurable (#226193)
The check currently hardcodes the container types accepted as sources in
range-based for loops. This prevents it from recognizing
project-specific range-like containers even when they expose the
operations needed by the check.
Add a `ForRangeLoopClasses` option, analogous to `VectorLikeClasses`,
for configuring those source container classes. The existing
standard-library types remain the defaults, so the current behavior is
preserved.
The test covers a custom range-like source and verifies the generated
`reserve(range.size())` fix.
Tests:
- `ninja check-clang-extra-clang-tidy-checkers-performance` (48/48
passed)
[7 lines not shown]
[MachineCombiner] Don't count debug instructions in block size (#225765)
The method MachineCombinerImpl::combineInstructions made decisions based
on
MBB->size() > inc_threshold
in two places, and since MachineBasicBlock::size() includes DBG_VALUE,
the existence of debug info can affect the resulting code.
Use sizeWithoutDebugLargerThan() instead to get the same code with and
without debug info.
I originally found this problem for my out-of-tree target. Then I used
Claude Opus 5 to convert the mir test for my out-of-tree target to one
for AArch64.
The other place, in the register pressure reduction part, was found by
code inspection. It is guarded by MustReduceRegisterPressure, which only
PowerPC uses. The PowerPC test for it was created by Claude Opus 5.
TableGen: Allow IsTruncStore predicates on atomic PatFrags
Atomic stores can be truncating in the same way as regular stores.
Allow IsTruncStore and IsNonTruncStore to be set on atomic store
PatFrags so patterns can check the store is not truncating without
needing to specify a fixed memory size.
I still find the hierarchy of load/store PatFrags frustrating. This would be
easier if we fixed the legacy mistake of treating atomic store as an "atomic"
rather than a store.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
AMDGPU/GlobalISel: Fix buffer stores of bf16 (#229673)
The 16-bit store source fixup only handled i8, i16 and f16, so bf16
values reached register bank legalization unextended and failed. Bitcast
16-bit FP sources to i16 before any-extending, rather than any-extending
the FP value directly.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[GlobalISel] Ensure integer trunc and anyext in call lowering. (#229346)
This ensures that the trunc and anyext created for argument lowering
have integer types, possibly casting back to fp types. This helps the
Arm backend but also comes up for WebAssembly and AMDGPU too.
[clang] Honor -discard-value-names for LLVM IR input (#229674)
For IR input, CodeGenAction forces setDiscardValueNames(false) to avoid
the crash in #44241, so names created by the optimizer (e.g. the
inliner's `.i`) are kept. c53d807321f6 fixed that crash in
Value::setNameImpl. Apply CodeGenOpts.DiscardValueNames after loading
the module and remove the "ignoring -fdiscard-value-names" warning.
LLM-aided. `clang -O3 -c sqlite3.bc` executes 0.7% fewer instructions.
clang: Do not pass an empty -syslibroot for --sysroot= on Darwin
-isysroot and DEFAULT_SYSROOT, but it checked only for the presence
of the option. An empty --sysroot= then produced -syslibroot "",
where previously it fell through to -isysroot. An empty --sysroot= is
the usual way to defeat DEFAULT_SYSROOT, so treat it as absent.
Use this to fix the darwin-static-lib tests when clang is built with
DEFAULT_SYSROOT or CLANG_USE_XCSELECT.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[flang] Take the debug record's location from the declaration. (#229484)
`DeclareOpConversion` gives the `DbgDeclareOp` it creates the location
of the memref (the alloca) instead of the location of the `XDeclareOp`.
This is problematic in 2 ways.
1. The `XDeclareOp` better represents where the variable is declared.
The memref is where the variable's memory happens to be allocated, and
for a dummy argument it is a block argument, so the record ends up on
the procedure statement. For a subroutine whose dummies `n` and `a` are
declared on lines 2 and 3, their `#dbg_declare` records are both at line
1 today, and at lines 2 and 3 with this change.
2. It requires the alloca to have a valid location. If it does not, the
`DbgDeclareOp` gets none either, and a debug record without a location
is invalid, so it is dropped during the MLIR to LLVM IR translation and
the variable disappears from the debug info altogether.
This PR makes `DeclareOpConversion` use the location of the
[4 lines not shown]
AMDGPU/GlobalISel: Use integer types when narrowing loads and stores (#229547)
The narrowScalar mutation for loads and stores produced untyped scalars.
For FP-typed values this resulted in an untyped G_OR in the lowerLoad
expansion for unaligned private accesses, which failed to select.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
MachineLICM: Stop checking kill flags in register pressure estimate (#229201)
This was checking kill flags, or hasOneNonDBGUse as a kill approximation.
The kill flag case appears to be of no practical use. InstrEmitter does not
emit the kill flag in the multiple user cases that would be required and I've
only managed to trigger a different hoisting decision with hand modified MIR.
The flag disagrees with hasOneNonDBGUse in 40 CodeGen tests (mostly
custom-inserter loops such as AMDGPU waterfalls and atomic expansions),
but removing it does not change the output of any CodeGen test.
AArch64: Test trap-after-noreturn against the exception model
Unfortunately, the AArch64TargetMachine constructor modifies
TargetOptions::TrapUnreachable if MCAsmInfo::usesWindowsCFI(). Thus, by
default, a trap after a noreturn call is emitted on COFF targets with the
default exception mode. There was no test coverage for this case's
interaction with an overridden exception model. Add the missing test in
preparation for cleaning up both the MC side exception predicates and
TrapUnreachable mutation.
Co-authored-by: Claude Opus 5 <noreply at anthropic.com>
[offload][OpenMP][NFC] Add a default-grid run to the no-loop tests
Add a run without OMP_NUM_TEAMS and OMP_TEAMS_THREAD_LIMIT to show the
kernels are promoted with the runtime's default grid as well.
