[LV] Add tests for predicated udivs used by replicated recipes (NFC). (#242895)
Add tests where a predicated udiv with various uses in different regions
and cost tests
[LAA] Add tests for miscompiles and monotonic bounds (NFC). (#242898)
Add a number of tests for cases where LAA currently computes incorrect
results: access size ignored in isSafeDependenceDistance, unchecked
dependences between multiple stores through the same pointer, and header
phis with a loop-varying latch value treated as forks.
Also add tests for non-affine pointers clamped with umin/umax/smin/smax
and zext, including cases that must not be bounded.
[KnownFPClass] Refine deductions for trunc/floor/ceil (#231368)
Follow up to https://github.com/llvm/llvm-project/pull/229982
Also refactors `KnownFPClass::roundToIntegral` to use the
`applyInputDenormalMode` helper.
[ValueTracking][KnownFPClass] Support constrained fadd/fsub (#241060)
Adds a rounding mode parameter to `KnownFPClass::fadd` and
`KnownFPClass::fsub` to support `constrained_fadd`/`constrained_fsub`
and `G_STRICT_FADD`/`G_STRICT_FSUB`.
[LV] Add test with hoisted loads using an IV with non-const step (NFC) (#242871)
Add a test checking the cost of complementary predicated loads that are
hoisted and whose address is based on an induction with a non-constant
step, which cannot be versioned.
[mlir][AMDGPU] Add preferAgpr hint to MFMA operations
The AMDGPU backend accepts the `!amdgpu.mfma !{!"agpr"}` instruction
metadata on calls to the `llvm.amdgcn.mfma.*` intrinsics (including
`llvm.amdgcn.mfma.scale.*`). The metadata is a hint that asks the backend
to allocate the accumulator input (C) and the result (D) of the MFMA to
AGPRs. The backend can ignore the hint, and dropping it does not change
program behavior.
Add an optional `preferAgpr` unit attribute to the ROCDL MFMA intrinsic
operations, including the scaled MFMA operations, and to `amdgpu.mfma` and
`amdgpu.scaled_mfma`. When the attribute is present, the translation to
LLVM IR attaches `!amdgpu.mfma !{!"agpr"}` to the intrinsic call, and the
AMDGPUToROCDL conversion copies the attribute to the ROCDL operation. When
the attribute is absent, the printed operations and the translated LLVM IR
are the same as before.
The SMFMAC operations do not get the attribute because the backend ignores
the hint on `llvm.amdgcn.smfmac.*`.
[5 lines not shown]
[llvm-strings] Add --unicode option
This commit adds the --unicode option from GNU strings to llvm-strings.
The options from GNU strings are supported, plus an additional option
value --unicode=utf8 to explicitly request UTF-8 output, similar to how
we also add an --encoding=utf8 option value to explicitly specify the
input is UTF-8.
[AMDGPU] Keep metadata when shrinking scaled MFMA fp4/fp6 operands
InstCombine narrows the A/B operands of llvm.amdgcn.mfma.scale.*.f8f6f4
for fp4/fp6 formats by creating a new call, but did not copy the old
call's metadata. Copy it, as the other rewrites in this file do, so
hints such as !amdgpu.mfma survive.
[AMDGPU] Select the MFMA AGPR form from !amdgpu.mfma metadata
Problem
- Frontends (Triton) pin MFMA accumulators to AGPRs with "=a" inline asm.
- The asm forces VGPR->AGPR copies around every MFMA (v_accvgpr_mov
+ s_nop inside loops), and selection has no way to ask for the AGPR
form per instruction: it is chosen per function.
Idea: a droppable hint on the MFMA call, read when the form is selected.
IR call @llvm.amdgcn.mfma.*(...), !amdgpu.mfma !{!"agpr"}
| SelectionDAGBuilder: TLI.getTargetIntrinsicHint(I)
DAG INTRINSIC_WO_CHAIN --SDEI side table: TargetHint
| CanUseAGPR_MAI predicate
MIR V_MFMA_..._e64 (AGPR form): C/D are AReg, no copies
How
1. TargetLowering::getTargetIntrinsicHint returns metadata from an
intrinsic call; SelectionDAGBuilder stores it on the node in the
[28 lines not shown]
[AMDGPU] Check the AGPR budget before selecting the AGPR form of MFMA
With -amdgpu-mfma-vgpr-form=0, selectAGPRFormMFMA selects the AGPR form
of an MFMA if the minimum from "amdgpu-agpr-alloc" is at least the
accumulator size. That is not sufficient for an accumulator wider than
128 bits. For such an MFMA, instruction selection selects the tied form,
and if src2 is live after the MFMA, the two-address pass converts it to
the early-clobber form. The early-clobber form needs src2 and vdst in
disjoint AGPRs, which is twice the accumulator size. If the function
cannot allocate that number of AGPRs, for example a 512-bit accumulator
with "amdgpu-agpr-alloc"="16", register allocation fails with "ran out
of registers during register allocation" in SelectionDAG and GlobalISel.
Record in SIMachineFunctionInfo the number of AGPRs that the function
can allocate, from GCNSubtarget::getMaxNumVectorRegs and the
-amdgpu-stress-agpr limit. Select the AGPR form only if that number is
at least twice the accumulator size for an accumulator wider than 128
bits, and at least the accumulator size otherwise. A vdst of 128 bits or
less can overlap src2, so the register allocator can split src2 instead.
[16 lines not shown]
[clang-repl] Rework -mllvm handling (#242861)
Parse -mllvm for the host and device CompilerInstances before their
IncrementalActions run, returning a bad option as an llvm::Error from
Interpreter::create rather than exiting the process (#197133). Point the
Orc and Wasm TargetMachines, which ignore the parsed options, at the
LLVMContext's library options. Drop the Wasm executor's re-parse with
cl::ParseCommandLineOptions, which rejects library options; lld no
longer resets LLVM options (#242717).
Parse clang-repl's own command line after target initialization with
cl::parseCommandLineOptions, forwarding library options as -mllvm, so
`clang-repl -print-after-isel` works again.
Stop calling llvm_shutdown() after each unittest suite; it destroyed
the option registry that later suites need.
LLM-aided
CodeGen: Simplify def segment extension in shrinkToUses
During extendSegmentsToUses, Segments holds only one def segment per
value, and none extend past their block. Live-in segments are kept
separately in LiveIns. To extend a def to a use, find the last segment
starting before the use, check that it starts in the use's block, and
move its end to the use. This replaces extendInBlock, whose general
search and merging of overlapped segments never applies here.
Also combine the per-block live-out and live-in tables into one struct.
llc instructions:u:
CTMark x86_64: -0.07%
CTMark arm64: -0.10%
AMDGPU sample: -0.27%
Largest AMDGPU improvements:
rocprim warp_sort (gfx1100): -1.89%
rocsolver stedc (gfx942): -1.70%
[5 lines not shown]
CodeGen: Merge shrinkToUses live-in segments in one pass
extendSegmentsToUses visits blocks in worklist order and inserted each
live-in segment into the middle of the segment vector, as did
createSegmentsForValues for the def segments. Sort both and merge them
in one pass instead, with at most one live-in segment per block. The
gain is largest in cases with large live ranges across many blocks.
llc instructions:u:
CTMark x86_64: -0.07%
CTMark arm64: -0.03%
AMDGPU sample: -0.28%
Largest AMDGPU improvements:
rocsolver potf2-specialized-kernels-d (gfx942): -4.57%
rocsolver getf2-small-d (gfx942): -2.71%
rocsolver getf2-small-d (gfx1100): -2.10%
rocprim warp_sort (gfx1100): -1.52%
Blender Cycles kernel (gfx1030/gfx1100/gfx1201): -0.9%
[2 lines not shown]
[AMDGPU] Check the AGPR budget before selecting the AGPR form of MFMA
With -amdgpu-mfma-vgpr-form=0, selectAGPRFormMFMA selects the AGPR form
of an MFMA if the minimum from "amdgpu-agpr-alloc" is at least the
accumulator size. That is not sufficient for an accumulator wider than
128 bits. For such an MFMA, instruction selection selects the tied form,
and if src2 is live after the MFMA, the two-address pass converts it to
the early-clobber form. The early-clobber form needs src2 and vdst in
disjoint AGPRs, which is twice the accumulator size. If the function
cannot allocate that number of AGPRs, for example a 512-bit accumulator
with "amdgpu-agpr-alloc"="16", register allocation fails with "ran out
of registers during register allocation" in SelectionDAG and GlobalISel.
Record in SIMachineFunctionInfo the number of AGPRs that the function
can allocate, from GCNSubtarget::getMaxNumVectorRegs. Select the AGPR
form only if that number is at least twice the accumulator size for an
accumulator wider than 128 bits, and at least the accumulator size
otherwise. A vdst of 128 bits or less can overlap src2, so the register
allocator can split src2 instead. Keep the check of the minimum as it
[12 lines not shown]
[AMDGPU] Keep metadata when shrinking scaled MFMA fp4/fp6 operands
InstCombine narrows the A/B operands of llvm.amdgcn.mfma.scale.*.f8f6f4
for fp4/fp6 formats by creating a new call, but did not copy the old
call's metadata. Copy it, as the other rewrites in this file do, so
hints such as !amdgpu.mfma.agpr survive.
[AMDGPU] Select the MFMA AGPR form from !amdgpu.mfma.agpr metadata
Problem
- Frontends (Triton) pin MFMA accumulators to AGPRs with "=a" inline asm.
- The asm forces VGPR->AGPR copies around every MFMA (v_accvgpr_mov
+ s_nop inside loops), and selection has no way to ask for the AGPR
form per instruction: it is chosen per function.
Idea: a droppable hint on the MFMA call, read when the form is selected.
IR call @llvm.amdgcn.mfma.*(...), !amdgpu.mfma.agpr !{}
| SelectionDAGBuilder: TLI.getTargetIntrinsicHint(I)
DAG INTRINSIC_WO_CHAIN --SDEI side table: TargetHint
| CanUseAGPR_MAI predicate
MIR V_MFMA_..._e64 (AGPR form): C/D are AReg, no copies
How
1. TargetLowering::getTargetIntrinsicHint returns metadata from an
intrinsic call; SelectionDAGBuilder stores it on the node in the
[26 lines not shown]
[AMDGPU] Keep metadata when shrinking scaled MFMA fp4/fp6 operands
InstCombine narrows the A/B operands of llvm.amdgcn.mfma.scale.*.f8f6f4
for fp4/fp6 formats by creating a new call, but did not copy the old
call's metadata. Copy it, as the other rewrites in this file do, so
hints such as !amdgpu.mfma.agpr survive.
[AMDGPU] Select the MFMA AGPR form from !amdgpu.mfma.agpr metadata
Problem
- Frontends (Triton) pin MFMA accumulators to AGPRs with "=a" inline asm.
- The asm forces VGPR->AGPR copies around every MFMA (v_accvgpr_mov
+ s_nop inside loops), and selection has no way to ask for the AGPR
form per instruction: it is chosen per function.
Idea: a droppable hint on the MFMA call, read when the form is selected.
IR call @llvm.amdgcn.mfma.*(...), !amdgpu.mfma.agpr !{}
| SelectionDAGBuilder: TLI.getTargetIntrinsicHint(I)
DAG INTRINSIC_WO_CHAIN --SDEI side table: TargetHint
| CanUseAGPR_MAI predicate
MIR V_MFMA_..._e64 (AGPR form): C/D are AReg, no copies
How
1. TargetLowering::getTargetIntrinsicHint returns metadata from an
intrinsic call; SelectionDAGBuilder stores it on the node in the
[25 lines not shown]
[llvm-strings] Add --unicode option
This commit adds the --unicode option from GNU strings to llvm-strings.
The options from GNU strings are supported, plus an additional option
value --unicode=utf8 to explicitly request UTF-8 output, similar to how
we also add an --encoding=utf8 option value to explicitly specify the
input is UTF-8.
[clang][Sema][ThreadSafety] Accept a pointee parameter of any redeclaration (#235813)
An attribute on a function pointer may name a parameter of its pointee:
extern void (*cb)(int, Guarded *g);
void (*cb)(int, Guarded *g) REQUIRES(g->mu) = nullptr;
Importantly, this also can come up for templates, where the "later"
definition is the same definition, but re-instantiated later (templates
also have other bugs which will be fixed separately). At a call through
'cb', the analysis substituted the call's arguments for the pointee's
parameters, but it accepted only a parameter of the canonical
declaration's prototype. Since the attribute above is on a
redeclaration, it names that redeclaration's 'g', which hit the
llvm_unreachable "ParmVarDecl does not belong to current declaration".
The `getCanonicalParamDecl` previously had two purposes: identifying the
canonical identity of a parameter in the body, and this assert. There
cannot be a body on a function pointer, so we don't need to be able to
[14 lines not shown]
