[VectorCombine] Propagate profile metadata when combining selects (#217690)
Note: In foldDeinterleaveInterleavePair when the condition is scalar all
instructions within a step have the same condition, so propagating from
NarrowInst works.
[mlir][memref] Avoid overflow in mem2reg alloca checks (#205245)
This patch fixes a crash in memory slot promotion under extremely large
static memrefs while handling element-count overflow.
Root Cause:
`MemRefMemorySlot.cpp` currently checks promotability with: `if
(getType().getNumElements() > MaxElementsToPromote)`
For extremely large static memref such as:
`memref<9223372036854775807x3xi32>`
This will result `ShapedType::getNumElements` overflows and triggers
assertion in `BuiltinTypeInterfaces.cpp:86:`:
`assert(num.has_value() && "integer overflow in element count
computation")`
Fix:
The changes in `AllocaOp::getPromotableSlots` can detect total element
[5 lines not shown]
[clang][nfc] Formatting fix for SubobjectAdjustment struct in Expr.h (#217803)
I was hitting an unnecessary reformat in another change and it's just a
minor formatting issue.
[SCEVExpander] Consider values below the hoisted insert point for reuse. (#217452)
expand() computes an insertion point before looking for an existing
value that already computes the expression: for an expression that is
computable in the loop, the point is hoisted to the loop header so that
the result dominates any user in the loop. A value that already computes
the expression but lives below that point is then rejected on position
alone, even though it dominates the point we were asked to expand at.
Retry the lookup at the requested point, and remember the result for the
location it is available at rather than for the hoisted point for a
point it does not dominate would hand it to a later expansion there.
This improves expansions in most cases in practice on a number of
workloads:
https://github.com/dtcxzyw/llvm-opt-benchmark-nightly/pull/965. In 9 out
of 163 files, there are slight increases in instruction count, but it is
a clear improvement in the majority of changes.
PR: https://github.com/llvm/llvm-project/pull/217452
[MLIR][LLVM] Preserve unknown function metadata on import
Import representable non-debug function metadata without a kind-specific
dialect conversion into `LLVMFuncOp` `function_metadata`. Preserve repeated
metadata kinds through the generic carrier so LLVM IR import and export can
round-trip the supported generic metadata subset.
Warn and drop attachments outside that subset while continuing to import the
function.
[Clang][RISCV] Mangle the ABI tag of a standard calling convention variant (#217249)
The psABI says a function using a standard calling convention variant
has to append an extra ABI tag to its mangled name, following the "ABI
tags" rule of the Itanium C++ ABI. The only variant listed so far is the
fixed-length vector calling convention, whose tag is
riscv_vls_cc_<ABI_VLEN>:
`__attribute__((riscv_vls_cc(128))) void foo();`
now mangles as `_Z3fooB16riscv_vls_cc_128v`.
The tag comes from the calling convention of the function type, so it
also covers a convention picked up from a typedef. It is emitted like an
explicit abi_tag attribute: sorted and deduplicated together with the
explicit tags, and never dropped as a derived tag.
Note the example in the psABI writes the tag length as B12, which does
not match the 16 characters of "riscv_vls_cc_128".
[4 lines not shown]
[clang][NVPTX] Emit !atomic.ignore.denormal.mode for CUDA atomics
CUDA's atomicAdd() family is defined in terms of PTX atom.add, whose
denormal behavior is fixed by the hardware. Without any annotation the
backend has to assume the function's denormal mode must be honored and
expands these into CAS loops whenever the two disagree. Mark them with
!atomic.ignore.denormal.mode so the native instruction is used.
That covers the __nvvm_atom_*_add_gen_f builtins that atomicAdd(),
atomicAdd_block() and atomicAdd_system() are written in terms of, plus
C11/C++11 atomics under -fatomic-ignore-denormal-mode and the
[[clang::atomic(ignore_denormal_mode)]] attribute, which requires
teaching the NVPTX target about AtomicOptions.
The condition for when the metadata is meaningful is now shared with the
AMDGPU and SPIR-V targets in addAtomicIgnoreDenormalModeMetadata(). It
takes an AllowHalf flag because whether f16 denormals are observable is
target specific: PTX exposes no FTZ control for f16 operations, so
atom.add.f16 never flushes and the opt-in is meaningful there, whereas
[3 lines not shown]
[NVPTX] Honor !atomic.ignore.denormal.mode on atomicrmw fadd
PTX atom.add has a fixed denormal behavior that the program cannot
control: atom.add.f32 flushes denormals on global memory but not on
shared, and atom.add.f16 never flushes. When that disagrees with the
function's denormal mode, the backend expands the atomic into a CAS loop
so the denormal behavior is preserved.
!atomic.ignore.denormal.mode says the denormal behavior of this
particular atomic does not matter, so use the native instruction even
when it disagrees. This is the same thing -nvptx-allow-ftz-atomics does,
except per-instruction instead of per-compilation, which lets a frontend
opt in only the operations it knows about -- notably CUDA's atomicAdd(),
which is defined in terms of atom.add.
Note that -nvptx-allow-ftz-atomics defaults to true, so the new behavior
is only observable with -nvptx-allow-ftz-atomics=false.
Co-authored-by: Artem Belevich <tra at google.com>
[IR] Generalize !amdgpu.ignore.denormal.mode into !atomic.ignore.denormal.mode
The !amdgpu.ignore.denormal.mode metadata tells the backend that an
atomicrmw fadd need not honor the function's denormal mode, so a native
atomic instruction whose denormal behavior is fixed in hardware may be
used instead of a CAS loop. Nothing about that is AMDGPU specific: NVPTX
has exactly the same problem with atom.add, whose FTZ behavior depends on
the address space and cannot be controlled.
Promote it to a target independent fixed metadata kind,
!atomic.ignore.denormal.mode, and switch the AMDGPU, SPIR-V and OpenMP
producers and consumers over to it. Document it in LangRef, and point
AMDGPUUsage at that description rather than duplicating it.
Existing IR keeps working: AutoUpgrade renames the metadata on atomicrmw
instructions when parsing textual IR and when materializing bitcode. The
upgrade is deliberately scoped to atomicrmw rather than being applied to
every attachment of that name, since that is the only place the metadata
was ever meaningful. Because bitcode can be materialized one function at
[6 lines not shown]
[RISCV][Disassembler] Symbolize UImm20 and SImm12Lo operands (#217550)
he RISC-V disassembler currently decodes operands represented by
`UImm20OperandMaybeSym` and `Simm12LoOp` directly into immediate
operands. As a result, an installed `MCSymbolizer` has no opportunity to
recover relocation-backed symbolic expressions for these operands.
Enable `MCSymbolizer` handling for RISC-V `UImm20` and `SImm12Lo`
operands. Fall back to the original immediate when symbolization is not
available. `SImm12Lo` values are sign-extended, while `UImm20` values
remain unchanged.
This enables clients such as BOLT to reconstruct symbolic operands for
HI20/LO12, PC-relative, and GOT relocations during instruction decoding.
This change only adds symbolizer hooks for these operands; the actual
symbolization is delegated to the installed `MCSymbolizer`. When no
symbolizer is installed, the existing immediate-based disassembly
behavior is preserved. Test coverage may need to be added after
RISCVMCSymbolizer is implemented in BOLT.
RuntimeLibcalls: Describe register-returning divmod libcall ABIs
Teach RuntimeLibcallsInfo::getFunctionTy about the ARM AEABI
(__aeabi_*divmod) and Windows (__rt_*div*) divmod sigantures. Currently
the custom lowering to these calls hardcodes the call signature information,
but in the future this should be automatically handled.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply at anthropic.com>
[CodeGen] Fix -fsanitize=array-bounds for __sized_by / _or_null pointers
`EmitCountedByBoundsChecking()` assumed a CountAttributedType is always
a __counted_by pointer. That isn't true, there are four versions of the
attribute:
* `__counted_by`: Already handled correctly.
* `__counted_by_or_null`: Incorectly handled.
* `__sized_by`: Incorreclty handled.
* `__sized_by_or_null`: Incorreclty handled.
In particular:
* __sized_by / __sized_by_or_null: the loaded bound is a byte count, but the
element index was compared against it directly, so an access was only
flagged once the index exceeded the byte count -- missing out-of-bounds
accesses for a pointee larger than one byte. Scale the index to bytes
('index * sizeof(element)') before comparing. counted_by counts elements
and is unchanged; a void (or otherwise zero-sized) pointee uses the GNU
[15 lines not shown]
[CodeGen] Fix __builtin_dynamic_object_size for __sized_by / _or_null pointers (#213794)
`emitCountedByPointerSize()` assumed a CountAttributedType is always
a __counted_by pointer. That isn't true, there are four versions of the
attribute:
* `__counted_by`: Already handled correctly.
* `__counted_by_or_null`: Incorectly handled.
* `__sized_by`: Incorreclty handled.
* `__sized_by_or_null`: Incorreclty handled.
In particular:
* __sized_by / __sized_by_or_null: the attribute argument is a byte count, but
the object size was computed as count * sizeof(*ptr), over-reporting by the
element size for any pointee larger than one byte. Use the count directly for
the byte-counting variants.
* __counted_by_or_null / __sized_by_or_null: a null pointer describes no
[20 lines not shown]
[AMDGPU] Update the comment regarding opsel[0] for v_cvt_sr_fp8/bf8_f16, NFC (#217796)
To be consistent with the actual implementation, OPSEL[0] can be
non-zero to select high 16-bit of src0.
[VPlan] Append recipes created via builder to worklist
The previous PR appended the top most created recipe to the worklist, and this PR extends it to any other nested recipes that were created, similar to InstCombine.
This removes the header mask in a good few more places on RISC-V as measured on SPEC CPU 2017, e.g. for the following loop:
```c
long f(const int *p, const int *q, long n) {
long a = 0, b = 0;
for (long i = 0;; i++) {
if (p[i] && q[i]) { a += i; b += i; }
if (i + 1 == n) break;
}
return a + b;
}
```
Before:
[49 lines not shown]
[VPlan] Make simplifyRecipe more like InstCombine
Most combines in simplifyRecipe RAUW a value, but not all of them erase the old recipe.
Unify them and bring it in line with InstCombine by having it return a VPValue, which simplifyRecipes can then call RAUW with, and automatically erase the old recipe.
Similarly to InstCombine, combines that modify a recipe should return the same recipe.
[VPlan] Process simplifyRecipes in a worklist
This brings simplifyRecipes further in line with InstCombine, and asides from unlocking more simplifications it also helps avoid spurious test churn whenever passes are moved around simplifyRecipes.
For now just push the new recipe onto the worklist, not its users.
This uses a post order traversal so we maintain the same simplification order as before.
I've gone through and checked every simplification we do is a canonicalisation that converges, and I checked on llvm-test-suite + SPEC CPU 2017 in various configurations that we don't hit any cycles.
[VPlan] Make simplifyRecipe more like InstCombine
Most combines in simplifyRecipe RAUW a value, but not all of them erase the old recipe.
Unify them and bring it in line with InstCombine by having it return a VPValue, which simplifyRecipes can then call RAUW with, and automatically erase the old recipe.
Similarly to InstCombine, combines that modify a recipe should return the same recipe.
