[SLP] Fix FMA regression in FMA-candidate retry
When tryToVectorize is called with AllowFMACandidates=true, falling
through to tryToVectorizeList vectorizes the fmul operands of an
FMA-candidate fadd without accounting for the lost FMA opportunity.
canConvertToFMA requires those fmuls to have one use, so vectorizing
them always breaks FMA formation. The cost model for tryToVectorizeList
omits the fadd from the tree and compares "2 fmuls vs 1 vfmul", missing
the scalar FMA savings entirely.
Block tryToVectorizeList when AllowFMACandidates=true. TryToReduce is
safe because computeReductionCost accounts for FMA in the scalar
baseline via canConvertToFMA on the fadd user.
Fixes a 4.5% regression in SPEC17 imagemagick on AArch64 introduced by
Reviewers: sushgokh, bababuck
Pull Request: https://github.com/llvm/llvm-project/pull/199706
[SandboxIR][Tracker] Implement accept(/*AcceptAll*/) and revert(/*RevertAll*/) (#197289)
In the context of nested checkpoints the tracker's API was somewhat
inconsistent. Tracker::revert() would revert to the last checkpoint but
accept() would accept all changes.
This patch fixes this, and introduces `accept(bool AcceptAll)` and
`revert(bool RevertAll)`.
[AMDGPU][docs][NFC] Fix some instruction names in gfx950 doc (#199094)
In the GFX950 documentation, some instructions that should have the
_sdwa suffix were incorrectly given the _dpp suffix.
[clang] fix getTemplateInstantiationArgs
This implements a new strategy for collecting the template arguments, by
relying on the qualifiers and template parameter lists to navigate the template
context of out-of-line definitions.
This greatly simplifies the signature of that function, by removing a bunch
of workarounds, and simpliffying a couple that weren't removed yet.
Since this now relies on qualifiers and template parameter lists,
this patch expends most of its effort making sure these are placed,
transformed and propagated to template instantiations.
Also makes the explicit specialization AST nodes stop abusing the template
parameter lists by storing it's own template parameter list, creating a
dedicated field for them, similar to partial specializations.
Revert "[clang] Use FileError in FileManager::getFileRef, getDirectoryRef" (#199721)
Reverts llvm/llvm-project#199126
This caused a small compile time regression.
[AMDGPU] Diagnose unsupported fma_legacy/sudot4/sudot8 intrinsics on some subtargets (#198464)
Add proper diagnostics for `llvm.amdgcn.fma.legacy`,
`llvm.amdgcn.sudot4` and `llvm.amdgcn.sudot8` on subtargets where they
are unsupported
[AMDGPU] Implement -amdgpu-spill-cfi-saved-regs
These spills need special CFI anyway, so implementing them directly
where CFI is emitted avoids the need to invent a mechanism to track them
from ISel.
Change-Id: If4f34abb3a8e0e46b859a7c74ade21eff58c4047
Co-authored-by: Scott Linder scott.linder at amd.com
Co-authored-by: Venkata Ramanaiah Nalamothu VenkataRamanaiah.Nalamothu at amd.com
[AMDGPU] Implement CFI for CSR spills
Introduce new SPILL pseudos to allow CFI to be generated for only CSR
spills, and to make ISA-instruction-level accurate information.
Other targets either generate slightly incorrect information or rely on
conventions for how spills are placed within the entry block. The
approach in this change produces larger unwind tables, with the
increased size being spent on additional DW_CFA_advance_location
instructions needed to describe the unwinding accurately.
Change-Id: I9b09646abd2ac4e56eddf5e9aeca1a5bebbd43dd
Co-authored-by: Scott Linder <scott.linder at amd.com>
Co-authored-by: Venkata Ramanaiah Nalamothu <VenkataRamanaiah.Nalamothu at amd.com>
[AMDGPU] Implement CFI for non-kernel functions
This does not implement CSR spills other than those AMDGPU handles
during PEI. The remaining spills are handled in a subsequent patch.
Change-Id: I5e3a9a62cf9189245011a82a129790d813d49373
Co-authored-by: Scott Linder <scott.linder at amd.com>
Co-authored-by: Venkata Ramanaiah Nalamothu <VenkataRamanaiah.Nalamothu at amd.com>
[AMDGPU] Emit entry function Dwarf CFI
Entry functions represent the end of unwinding, as they are the
outer-most frame. This implies they can only have a meaningful
definition for the CFA, which AMDGPU defines using a memory location
description with a literal private address space address. The return
address is set to undefined as a sentinel value to signal the end of
unwinding.
Change-Id: I21580f6a24f4869ba32939c9c6332506032cc654
Co-authored-by: Scott Linder <scott.linder at amd.com>
Co-authored-by: Venkata Ramanaiah Nalamothu <VenkataRamanaiah.Nalamothu at amd.com>
[MC][Dwarf] Add custom CFI pseudo-ops for use in AMDGPU
While these can be represented with .cfi_escape, using these pseudo-cfi
instructions makes .s/.mir files more readable, and it is necessary to
support updating registers in CFI instructions (something that the
AMDGPU backend requires).
Change-Id: I763d0cabe5990394670281d4afb5a170981e55d0
[Clang] Default to async unwind tables for amdgcn
To avoid codegen changes when enabling debug-info (see
https://bugs.llvm.org/show_bug.cgi?id=37240) we want to
enable unwind tables by default.
There is some pessimization in post-prologepilog scheduling, and a
general solution to the problem of CFI_INSTRUCTION-as-scheduling-barrier
should be explored.
Change-Id: I83625875966928c7c4411cd7b95174dc58bda25a
[MIR] Error on signed integer in getUnsigned
Previously we effectively took the absolute value of the APSInt, instead
diagnose the unexpected negative value.
Change-Id: I4efe961e7b29fdf1d5f97df12f8139aac12c9219
[AMDGPU] Emit entry function Dwarf CFI (#183152)
Entry functions represent the end of unwinding, as they are the
outer-most frame. This implies they can only have a meaningful
definition for the CFA, which AMDGPU defines using a memory location
description with a literal private address space address. The return
address is set to undefined as a sentinel value to signal the end of
unwinding.
Change-Id: I21580f6a24f4869ba32939c9c6332506032cc654
Co-authored-by: Scott Linder <scott.linder at amd.com>
Co-authored-by: Venkata Ramanaiah Nalamothu <VenkataRamanaiah.Nalamothu at amd.com>
[1/3][RegAlloc][LiveRegMatrix] Fix inconsistency in HoistSpillHelper delegates (#197773)
HoistSpillHelper's LiveRangeEdit delegate callbacks did not keep the
LiveRegMatrix consistent when eliminateDeadDefs triggered interval
shrinking and splitting during spill hoisting.
Three issues:
1. No LRE_WillShrinkVirtReg override: when eliminateDeadDefs shrinks a
vreg's interval via shrinkToUses, the matrix was not updated. Add an
override that unassigns the vreg from the matrix and records it in
PendingReassignments for later re-assignment.
2. LRE_DidCloneVirtReg called VRM.assignVirt2Phys without
Matrix->assign: when splitSeparateComponents creates new vregs, the
clones got VRM entries but were never inserted into the matrix. Fix by
consuming PendingReassignments and properly assigning both Old (shrunk)
and New (split) intervals to the matrix.
[15 lines not shown]
[flang][cuda] Lower c_devptr value arguments in bind(c) like c_ptr (#199316)
Treat `type(c_devptr), value` arguments in BIND(C) interfaces like
`type(c_ptr), value` by passing the nested raw address value instead of
the outer derived type ABI. This keeps call signatures consistent for
CUDA Fortran generic specifics that share a C binding label and avoids
argument misclassification at the x86_64 register/stack boundary.
[DSE] Restrict partial-overlap store merging to matching orderings. (#199728)
Partial-overlap store merging folds the later killing store into the
earlier dead store and erases the killing store. That is invalid if the
killing store is volatile or has stronger-than-unordered atomic
ordering, because erasing it drops an observable write. It is also invalid
if the killing and dead stores have different atomic orderings, because
the bytes originally written by the killing store would inherit the dead
store's atomicity after the merge -- silently dropping (or adding)
atomicity for those bytes.
Require both stores to be unordered (i.e. non-volatile with ordering at
most unordered) and to share the same ordering. This preserves the
existing fold for two simple stores or two unordered-atomic stores
(e.g. simple.ll's test43a) while leaving volatile, ordered-atomic, and
atomicity-mismatched cases in place.
This bug was found by a large run of Opus 4.7 looking for bugs in LLVM.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply at anthropic.com>
[llvm][Object] Add COFF support to extractOffloadBundleFatBinary (#199574)
Use PointerToRawData from the COFF section header to compute the section
offset, replacing the previous stub that returned an error for all COFF
object files.
This enables llvm-objdump --offloading and llvm-readobj --offloading to
work on COFF fatbins produced by HIP on Windows.
---------
Co-authored-by: James Henderson <James.Henderson at sony.com>
Co-authored-by: Matt Arsenault <arsenm2 at gmail.com>
