[AMDGPU] Keep metadata when shrinking scaled MFMA fp4/fp6 operands
InstCombine narrows the A/B operands of llvm.amdgcn.mfma.scale.*.f8f6f4
for fp4/fp6 formats by creating a new call, but did not copy the old
call's metadata. Copy it, as the other rewrites in this file do, so
hints such as !amdgpu.mfma.agpr survive.
[AMDGPU] Select the MFMA AGPR form from !amdgpu.mfma.agpr metadata
Problem
- Frontends (Triton) pin MFMA accumulators to AGPRs with "=a" inline asm.
- The asm forces VGPR->AGPR copies around every MFMA (v_accvgpr_mov
+ s_nop inside loops), and selection has no way to ask for the AGPR
form per instruction: it is chosen per function.
Idea: a droppable hint on the MFMA call, read when the form is selected.
IR call @llvm.amdgcn.mfma.*(...), !amdgpu.mfma.agpr !{}
| SelectionDAGBuilder: TLI.getTargetIntrinsicHint(I)
DAG INTRINSIC_WO_CHAIN --SDEI side table: TargetHint
| CanUseAGPR_MAI predicate
MIR V_MFMA_..._e64 (AGPR form): C/D are AReg, no copies
How
1. TargetLowering::getTargetIntrinsicHint returns metadata from an
intrinsic call; SelectionDAGBuilder stores it on the node in the
[25 lines not shown]
[clang][Sema][ThreadSafety] Accept a pointee parameter of any redeclaration (#235813)
An attribute on a function pointer may name a parameter of its pointee:
extern void (*cb)(int, Guarded *g);
void (*cb)(int, Guarded *g) REQUIRES(g->mu) = nullptr;
Importantly, this also can come up for templates, where the "later"
definition is the same definition, but re-instantiated later (templates
also have other bugs which will be fixed separately). At a call through
'cb', the analysis substituted the call's arguments for the pointee's
parameters, but it accepted only a parameter of the canonical
declaration's prototype. Since the attribute above is on a
redeclaration, it names that redeclaration's 'g', which hit the
llvm_unreachable "ParmVarDecl does not belong to current declaration".
The `getCanonicalParamDecl` previously had two purposes: identifying the
canonical identity of a parameter in the body, and this assert. There
cannot be a body on a function pointer, so we don't need to be able to
[14 lines not shown]
Verifier: Check each string function attribute once
verifyFunctionAttrs looked up about 35 string attributes by name for
every function and call site. Iterate over the string attributes that
are present instead, and only look up the attributes they depend on.
Whole O3 instructions in opt:
kimwitu++ k.bc -1.03%, SPASS cnf.bc -0.34%, sqlite3 -0.12%.
CTMark instructions:u: ThinLTO -0.05%.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[ConstantFolding] Fold constant vector `bitcast`s to scalar bytes (#242811)
#188030 added `bitcast` constant folding for byte types.
This PR handles vector to scalar byte `bitcast`s.
Constant fold vector `bitcast`s to scalar bytes.
- A vector of integers or floating-point values folds to a byte constant
with the same bits.
- A vector with a `poison` element isn't folded, as a byte constant
can't represent partial `poison` values.
- A vector of pointers isn't folded, as a byte constant can't represent
pointer provenance.
- A vector of bytes with an element that isn't a plain constant, like a
pointer, isn't folded, as it would drop provenance.
```llvm
%a = bitcast <2 x i32> <i32 1, i32 2> to b64 ; b64 8589934593
%b = bitextract i8, b64 %a, i32 32 ; i8 2
%c = bitcast <2 x i32> <i32 1, i32 poison> to b64 ; unchanged
```
AMDGPU: Update physreg LiveIntervals accurately in SIOptimizeExecMaskingPreRA
Instead of dropping the exec/vcc/scc register unit ranges after any
change, remove all physreg defs of the instruction. This fixes the
redundant exec copy fold leaving a stale def in the exec range for
COPY $exec, implicit-def $exec, which was hidden by always
recomputing.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[InstSimplify] Simplify vector `bitinsert` and `bitextract` with out-of-range offsets (#242816)
Simplify vector `bitinsert` and `bitextract` to `poison` when every
element of a constant offset is out of range or `poison`.
#229125 added this fold for scalar and splat offsets. This extends it to
non-splat vector offsets.
```llvm
%a = bitextract <4 x i8>, <4 x b32> %x, <4 x i32> <i32 25, i32 32, i32 poison, i32 -1> ; poison
%b = bitextract <4 x i8>, <4 x b32> %x, <4 x i32> <i32 0, i32 8, i32 16, i32 25> ; unchanged, only the last element is out of range
```
[Option] Remove XXXOptions::Global. NFC (#242854)
cl::ParseCommandLineOptions rejects library options (#242692), and the C
API parses them into its own cl::LibraryOptionsContext (#242692), so
nothing sets T::Global and it always equals T::Default. Remove it, along
with cl::LibraryOptions::global() and reset().
`LibraryOptionsContext::get(nullptr)` returns T::Default, and
`cl::ResetAllOptionOccurrences` only resets cl:: option values.
LLM-aided
[mlir][python] Define __index__ on PyIntegerAttribute (#215555)
In nanobind v2.14.0+, integer casting from Python objects to C++ integer
types was updated to require `__index__` (`PyNumber_Index`) instead of
`__int__`.
https://nanobind.readthedocs.io/en/latest/changelog.html#version-2-14-0-aug-7-2026
Because `PyIntegerAttribute` defined `__int__` but not `__index__`,
passing `IntegerAttr` instances to bindings expecting integer sequences
failed. Specifically, in `mlir/test/python/ir/attributes.py`:
# CHECK: input: [IntegerAttr(4 : i64), IntegerAttr(2 : i64)] (<class
'list'>),
# CHECK-SAME: result: array<i8: 4, 2>
create_and_print(DenseI8ArrayAttr, [Attribute.parse(f"{x}") for x in [4,
2]])
Failed because `DenseI8ArrayAttr.get([IntegerAttr(4 : i64),
[12 lines not shown]
[clang] Fix target-dependent default for -mincremental-linker-compatible (#222099)
Patch #188800 modified the target specific default for this flag which
resulted in non MSVC, non UEFI targets getting this flag set. This
caused issue https://github.com/llvm/llvm-project/issues/219457 which
this patch should fix.
Assisted-by: Gemini
(cherry picked from commit 08d064a71dbdfa1b50f352d34c103ad245070611)
[libc++] Fix std::mt19937 seeding with std::seed_seq under vectorized ABI (#209860)
This PR fixes a bug in `std::mersenne_twister_engine::seed(_Sseq&)`
where the initialization twist is bypassed when the vectorized engine
optimization (`_LIBCPP_ABI_VECTORIZED_MERSENNE_TWISTER_ENGINE`) is
enabled.
In the vectorized implementation introduced in #206423, the engine state
must be pre-twisted during initialization by calling
`__update_all_states()` at the end of the `seed` functions.
However, in `seed(_Sseq& __q)`, the all-zero state check contains an
early `return;` when a non-zero state is verified. This early return
bypasses the trailing `__update_all_states()` call. Consequently, the
generator is initialized with raw, untwisted seed values (tempered)
rather than the standard twisted values.
This only manifests when compiled with unstable ABI settings
(`_LIBCPP_ABI_VERSION >= 2`) which enables the vectorized
[10 lines not shown]
[AArch64] fix invalid GISel lowering of saturating `f16` to integer conversion (#235676)
The saturating version of the code seems to be copied from the
non-saturating version, and has this comment
> The range of a fp16 value fits into an i17, so we can lower the width
to i64.
But that is wrong for the saturating conversions, because positive
infinity is mapped to `i128::MAX`.
There may be a way to recover some of the shorter codegen, but I'm
hoping this change is something we can easily backport.
cc https://github.com/rust-lang/rust/issues/164972
(cherry picked from commit 617b7da9db855f1c74e9425555ce23f98f4dc6f9)
FunctionAttrs: Check all uses when inferring captures for argument SCCs (#231536)
When merging the capture components of an SCC of arguments, only the
first use of each argument was considered, because of a misplaced break.
If an argument was passed to another argument in the SCC first and to a
capturing argument outside of it later, it was incorrectly inferred to
not be captured.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
(cherry picked from commit 05b3eeffba9f93d14a9f79af253c4769d8c01827)
[clangd] Index symbols declared inside export declarations (#234755)
SymbolCollector::shouldCollectSymbol only accepts symbols whose semantic
parent is a translation unit, namespace, linkage spec, enum, record or
one of a few ObjC containers. ExportDecl is a DeclContext and is
returned by getDeclContext(), so every entity declared with `export` or
inside an `export { ... }` block was dropped from the index. Members of
exported classes were still indexed, since their parent is the record.
In C++20 modules code this hides most of the exported interface from the
index: Go to Definition on an exported function stops at the declaration
in the module interface instead of reaching the definition in the
implementation unit, and workspace/symbol does not find it. Rename uses
the same check, so it rejects exported symbols as "not eligible for
indexing" (reported in clangd/clangd#2569).
Use getNonTransparentDeclContext() instead, so declarations inside
transparent contexts (export, extern "C", unscoped enums) are judged by
their enclosing context. Do the same in isIndexedForCodeCompletion, so
[7 lines not shown]
release/23.x: [clang codegen] Fix MS ABI with deleted template constructor (#237634)
Manaul backport of 2abfa104dc091113f844d252b4df2a22e800e4b8. (The code
change is the same, the generated IR for the testcase changed slightly.)
[X86][APX] Save R16-R31 in interrupt and no_caller_saved_registers functions (#237109)
x86_intrcc, "no_caller_saved_registers" and anyregcc functions must
preserve every register they modify. With EGPR their save lists
(CSR_64_AllRegs*) did not include R16-R31, so a handler that uses, or
calls something that clobbers, an extended GPR silently corrupts the
interrupted context. For example, a handler reading its interrupt frame
loads it into r16/r17 and returns with iretq without restoring them.
Add CSR_64_APX_AllRegs{,_NoSSE,_AVX,_AVX512}, the AllRegs lists plus
R16-R31, and use them as the save lists when the subtarget has EGPR. The
call-preserved masks are unchanged, so callers keep treating R16-R31 as
clobbered. This stays correct for callees built without EGPR or by other
or older compilers.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
(cherry picked from commit 6a30204d78bd45b56611bbd09cf8c53b78b7435d)
[LoopSimplify] Use SmallSetVector for dead predecessor removal (#231155)
In `simplifyOneLoop`, `BadPreds` was previously declared as
`SmallPtrSet<BasicBlock*, 4>`. When a non-header loop block had 5 or
more unreachable out-of-loop predecessors, `SmallPtrSet` overflowed its
inline small array and iterated over `BasicBlock*` pointers in hash
order under ASLR. Because `PHINode::removeIncomingValue` swaps each
removed predecessor slot with the last `PHINode` operand, removing dead
predecessors in pointer-hash order permuted the remaining live incoming
blocks in the `PHINode` and produced non-deterministic IR and ThinLTO
bitcode.
Replace `SmallPtrSet<BasicBlock*, 4>` with `SmallSetVector<BasicBlock *,
4>` to deduplicate and remove dead predecessors in deterministic
`predecessors(BB)` order.
Fixes #231143
(cherry picked from commit 921c2c0b929eb05dd8560c6bead0b1e5345e35ca)
[clang-tidy] Fix false positive in readability-trailing-comma (#220548)
Use the syntactic form of an empty InitListExpr instead of falling back
to the semantic form.
Fixes #220222
(cherry picked from commit 0abb55638843428fccd7a923e6126fcb74a69a51)
[Mips] Fix O32 lowering on 64-bit CPUs (#224937)
Use 32-bit GPR lowering for O32 while preserving the selected ISA and
FPR mode. Handle FP64 transfers through the stack when high-half moves
are unavailable.
(cherry picked from commit 52500486a0b723a1d9528e3098901702da837c3d)
This PR splits the `libc_generated_header` rule into two steps. Instead of directly generating a header, we introduce two new rules: `libc_hdrgen_spec` and `libc_generated_headers`. The former describes all the inputs to a `hdrgen` invocation for a header without producing a file, and the latter generates the header from the spec while letting you specify `--entrypoints=`.
This is nice because it allows rules to share definitions for how headers are produced while specifying their own sets of `entrypoints`.
---
**Note:** this is not exactly equivalent to how the CMake build works. In this PR, LLVM-libc function objects are still compiled against [unfiltered headers](https://github.com/llvm/llvm-project/blob/cbfac21cd384f6c7c480c2d0026ecedfe2d619e7/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl#L46), whereas CMake only produces [filtered headers](https://github.com/llvm/llvm-project/blob/cbfac21cd384f6c7c480c2d0026ecedfe2d619e7/libc/cmake/modules/LLVMLibCHeaderRules.cmake#L107) and compiles c++ sources against them. If we want to be closer to CMake, we could instead make the entrypoint list configurable at the command-line. This could look something like:
```
libc_entrypoint_list(
name = "default_entrypoint_list",
entrypoints = [ ... ], # names of entrypoints, or unset for all entrypoints
)
label_flag(
name = "entrypoint_list",
build_setting_default = ":default_entrypoint_list",
)
```
[2 lines not shown]
[sanitizer_common][Darwin] Fix TLS base and size on arm64 (#242018)
On arm64 Darwin, GetThreadStackAndTls() returns an empty TLS range, so
LSan
never scans a thread's TSD. Memory that is only referenced from a live
thread's TSD is then reported as leaked, for example
pthread_setspecific()
values or the continuation cache libdispatch keeps in a worker thread.
There are two bugs. TlsBaseAddr() reads TPIDRRO_EL0 and then does
`segbase &= 0x07ul`, which keeps the low 3 bits instead of clearing
them.
TlsSize() returns 0 on arm64, so the bad base was never used. The mask
was
added in 443820179a.
Up to xnu-7195 (macOS 11), xnu's `_os_tsd_get_base()` also cleared the
low
3 bits of TPIDRRO_EL0, which held the CPU number. Since xnu-8019 (macOS
[33 lines not shown]
IR: Check inline compatibility on the function attribute sets (#242855)
hasCompatibleFnAttrs queried each attribute on the caller and callee
which needs to find the function attributes in the AttributeList. Instead
pass the AttributeSet once and use them for the inline rules, which only
look at function attributes.
callgrind instructions, opt -O3: SPASS cnf.bc -0.20%, sqlite3 -0.12%,
kimwitu++ k.bc -0.11%. CTMark instructions:u: O3 -0.04%, ThinLTO -0.04%.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
AMDGPU: Fix double interval removal in SIOptimizeExecMaskingPreRA (#242790)
An exec copy folded after its use was erased from an s_endpgm block
had its interval removed twice. Fixes an assert after 6a483810f0da.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
BPF: Fix removing the blocks after the start of an unwind
BPFUnwindLower erased each dead block as soon as it unlinked it, so a
dead block that was a successor of a later one was used after being
freed. It also kept an address-taken block but not the blocks it
branches to, leaving a branch to an erased block.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
