[X86] Replace custom AND/OR/XOR reduction pattern matching with ISD::VECREDUCE_AND/OR/XOR support (#199544)
The middle-end (SLP, VectorCombine and InstCombine) recognition and
handling of vector logic reduction patterns is sufficient now, so the
backend can work with ISD::VECREDUCE_AND/OR/XOR nodes directly.
[GVN] Track visited nodes in equality propagation to avoid OOM (#212265)
The added test ended up being out-of-memory (OOM) killed, because GVN
computed the same equality facts through exponentially many paths.
Co-authored-by: Katy Thackray <katy.thackray2 at arm.com>
[flang] Add an example plugin exercising the HLFIR pipeline extension points
The HLFIR extension points, the pipeline config callback registry and fir-opt's
symbol export all exist to let an out-of-tree MLIR pass run while the HLFIR
intrinsic operations (hlfir.sum, hlfir.matmul, ...) are still present. None of
that was covered end to end, and nothing showed how to use it.
Add flang/examples/HLFIRPipelinePlugin, modelled on PrintFlangFunctionNames. It
contributes a pass that prints the HLFIR operations still present in the module,
tagged with the pipeline position it was inserted at, and exposes it through
both plugin entry points: a static initializer calling
fir::registerPassPipelineConfigCallback for the `flang -fc1 -load` path, and
mlirGetPassPluginInfo so fir-opt can load it with --load-pass-plugin.
Nothing is linked into the shared object; MLIR, FIR and flang symbols resolve
against the host tool, which is what export_executable_symbols_for_plugins on
flang and fir-opt provides.
Two tests use it. Examples/hlfir-pipeline-plugin.f90 covers -emit-fir and
[7 lines not shown]
[Docs][AMDGPU] fully specify volatile accesses in the memory model
A volatile access on AMDGPU includes store-available or load-visible semantics
at the widest scope supported by its address space:
- system scope for global/generic,
- workgroup scope for local (LDS)
This fully specifies the behavior that the LLVM memory model leaves
target-dependent, matching the implementation in SIMemoryLegalizer.
Assisted-By: Claude Opus 4.8
[flang] Add an example plugin exercising the HLFIR pipeline extension points
The HLFIR extension points, the config augmentor registry and fir-opt's symbol
export exist to let an out-of-tree MLIR pass run while the HLFIR intrinsic
operations (hlfir.sum, hlfir.matmul, ...) are still present. None of that was
covered end to end, and there was no worked example of how to use it.
Add flang/examples/HLFIRPipelinePlugin, modelled on PrintFlangFunctionNames. It
contributes a pass that prints the HLFIR operations still present in the
module, tagged with the pipeline position it was inserted at, and exposes it
through both plugin entry points: a static initializer calling
fir::registerPassPipelineConfigCallback for the `flang -fc1 -load` path, and
mlirGetPassPluginInfo so fir-opt can load it with --load-pass-plugin.
Nothing is linked into the shared object: MLIR, FIR and flang symbols resolve
against the host tool, which is what export_executable_symbols_for_plugins on
flang and fir-opt provides. Removing the fir-opt export drops it from ~105k
exported dynamic symbols to one and makes --load-pass-plugin fail to load.
[7 lines not shown]
[flang] Add a pass-pipeline config callback hook for plugins
The HLFIR-to-FIR pipeline extension points live on MLIRToLLVMPassPipelineConfig,
which the frontend builds as a local of CodeGenAction, out of reach of a plugin.
Add a process-global registry of callbacks that run on the config before the
pipeline is built. A plugin registers one from a static initializer, so it is in
place before any compilation begins, as FrontendPluginRegistry does for plugin
actions. Both code generation entry points invoke the callbacks, lowerHLFIRToFIR
for -emit-fir and generateLLVMIR for -emit-llvm/-emit-obj, and are mutually
exclusive for a given compilation, so a plugin sees the same behaviour whichever
output was asked for.
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
[LV] Make greater use of ScalarEvolution in addDiffRuntimeChecks (#213668)
When the loop vectoriser calls addDiffRuntimeChecks it goes to a lot of
effort to avoid generating multiple copies of the VF calculation by
caching the first instance. However, now that ScalarEvolution has a
getElementCount function we can simplify this code significantly,
especially since SCEVs are also implicitly cached. Make greater use of
SCEVs for computation also has the side-effect of improving code quality
in the memory check blocks, which is important when estimating costs of
these checks. You can see this in some tests like
LoopVectorize/AArch64/sve-runtime-check-size-based-threshold.ll
where the threshold for entering the loop has been relaxed.
Ideally, all computation in addDiffRuntimeChecks should be done using
SCEV because there are presumably other folds that can be applied to the
comparisons. However, I'd keep things simple in this PR and deal with
that in a follow-on PR.
[SPIR-V] Preserve sign and payload when printing NaN as hex float (#212438)
NaN was always printed as the canonical `0x1.8p+<MaxExp>`, silently
dropping the sign bit and any signaling/payload bits
Discussed in https://github.com/llvm/llvm-project/pull/212295
[MLIR][SCCP] Fix in-place folds leaking into IR during simulation (#213933)
SparseConstantPropagation restored the operation only when `fold` did
not return any fold results. But a folder can mutate the op in place and
still return out-of-place results or fail, e.g. `vector.extract` folds
constant dynamic positions into static ones before attempting further
folds.
The constants fed to `fold` are speculative lattice values, so the
mutation bakes a possibly-wrong constant into the IR. SCCP would
permanently replace a loop-carried dynamic index with its first lattice
value for example.
Fix: Restore the original operands and attributes after every fold call,
regardless of its outcome.
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
[flang] Add HLFIR-to-FIR pass pipeline extension points (#212194)
The FIR optimizer extension points (FIROptEarly, FIRInliner, FIROptLast)
all
run after HLFIR has been lowered to FIR, so the HLFIR intrinsic
operations
(hlfir.sum, hlfir.matmul, ...) are gone by the time they run.
Transformations
that need to see those operations have nowhere to attach.
Add two extension points to createHLFIRToFIRPassPipeline:
* HLFIROptEarly, at the start of the pipeline, before any HLFIR
simplification or inlining.
* HLFIROptLast, just before createLowerHLFIRIntrinsics.
Drivers register passes through registerHLFIROptEarlyEPCallbacks and
registerHLFIROptLastEPCallbacks on MLIRToLLVMPassPipelineConfig. The
invoke
[4 lines not shown]
[flang] Add a pass-pipeline config callback hook for plugins
The HLFIR-to-FIR pipeline extension points live on MLIRToLLVMPassPipelineConfig,
which the frontend builds as a local of CodeGenAction, out of reach of a plugin.
Add a process-global registry of callbacks that run on the config before the
pipeline is built. A plugin registers one from a static initializer, so it is in
place before any compilation begins, as FrontendPluginRegistry does for plugin
actions. Both code generation entry points invoke the callbacks, lowerHLFIRToFIR
for -emit-fir and generateLLVMIR for -emit-llvm/-emit-obj, and are mutually
exclusive for a given compilation, so a plugin sees the same behaviour whichever
output was asked for.
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
[lldb] Clear stale error in Target::ReadMemory on file-cache fallback (#213451)
Load the checked-in core `linux-aarch64-pac.core` from
`lldb/test/API/functionalities/postmortem/elf-core/` with its binary and ask for
a `char16_t *` summary at the start of `.text`:
```
(lldb) settings set target.max-string-summary-length 8
(lldb) expression -l c++ -- (char16_t *)0x400140
(char16_t *) $0 = 0x0000000000400140 unable to read data
(lldb) memory read -s1 -c16 0x400140
0x00400140: 3f 23 03 d5 ff 83 00 d1 fd 7b 01 a9 fd 43 00 91 ?#.......{...C..
```
`memory read` prints the very bytes the summary just claimed it could not read.
Both go through `Target::ReadMemory()`, which reuses a single `Status &error` for
the process read and for the file-cache fallback at the end of the function, and
`Target::ReadMemoryFromFileCache()` only ever sets that `Status`, it never clears
it. So when the process read fails outright and the fallback then satisfies the
[37 lines not shown]
[clang-tidy] Fix clang-tidy-diff with not producing blank lines (#213873)
clang-tidy-diff.py unconditionally wrote `stdout + "\n"` for every file
it processed, even when clang-tidy produced no output so this bloated
output with needless blank line (happened with `-quiet` flag enabled
which made tidy produce 0 diagnostics).
[SPIRV][HLSL] Fix Vulkan-invalid SPIR-V for Interlocked* on groupshared/UAV memory (#212663)
Compiling HLSL `InterlockedOr`/`InterlockedAdd`/`InterlockedXor` against
`groupshared` (or UAV) destinations through the clang HLSL -> SPIR-V
path produced SPIR-V that `spirv-val` rejects for Vulkan.
Illegal OpCapability Linkage: groupshared variables are emitted
as `external hidden addrspace(3)` (Workgroup)
declarations. `getSpirvLinkageTypeFor` gave any non-interface
declaration `Import` linkage, which adds a
`LinkageAttributes` decoration and forces `OpCapability` Linkage, which
is illegal in a Vulkan shader.
Fix: in a shader environment (no linker), Workgroup/Private declarations
get no linkage.
Fixes: https://github.com/llvm/offload-test-suite/issues/1404
Assisted by: Github Copilot
[flang] Add an example plugin exercising the HLFIR pipeline extension points
The HLFIR extension points, the config augmentor registry and fir-opt's symbol
export exist to let an out-of-tree MLIR pass run while the HLFIR intrinsic
operations (hlfir.sum, hlfir.matmul, ...) are still present. None of that was
covered end to end, and there was no worked example of how to use it.
Add flang/examples/HLFIRPipelinePlugin, modelled on PrintFlangFunctionNames. It
contributes a pass that prints the HLFIR operations still present in the
module, tagged with the pipeline position it was inserted at, and exposes it
through both plugin entry points: a static initializer calling
fir::registerPassPipelineConfigCallback for the `flang -fc1 -load` path, and
mlirGetPassPluginInfo so fir-opt can load it with --load-pass-plugin.
Nothing is linked into the shared object: MLIR, FIR and flang symbols resolve
against the host tool, which is what export_executable_symbols_for_plugins on
flang and fir-opt provides. Removing the fir-opt export drops it from ~105k
exported dynamic symbols to one and makes --load-pass-plugin fail to load.
[7 lines not shown]
[flang] Add a pass-pipeline config-augmentor hook for -load'ed plugins
The HLFIR-to-FIR pass pipeline exposes extension points on
MLIRToLLVMPassPipelineConfig, but that config is built inside the frontend, so
a -load'ed plugin has no way to reach it and register passes.
registerDefaultInlinerPass is the only augmentor today, and it is wired in by
hand.
Add a global registry of config augmentors:
* fir::registerPassPipelineConfigCallback(cb) appends a callback, to be
called from a plugin's static initializer at -load time.
* fir::invokePassPipelineConfigCallbacks(config) runs them on the config.
This mirrors what flang already does for -load'ed plugin actions via
FrontendPluginRegistry: a process-global, append-only registry populated from
static initializers, which run before any compilation begins.
Both code generation entry points invoke the callbacks after building their
[11 lines not shown]
[flang] Add HLFIR-to-FIR pass pipeline extension points
The FIR optimizer extension points (FIROptEarly, FIRInliner, FIROptLast) all
run after HLFIR has been lowered to FIR, so the HLFIR intrinsic operations
(hlfir.sum, hlfir.matmul, ...) are gone by the time they run. Transformations
that need to see those operations have nowhere to attach.
Add two extension points to createHLFIRToFIRPassPipeline:
* HLFIROptEarly, at the start of the pipeline, before any HLFIR
simplification or inlining.
* HLFIROptLast, just before createLowerHLFIRIntrinsics.
Drivers register passes through registerHLFIROptEarlyEPCallbacks and
registerHLFIROptLastEPCallbacks on MLIRToLLVMPassPipelineConfig. The invoke
methods are const so they can be called on the const config the HLFIR pipeline
receives. With no callbacks registered the pipeline is unchanged.
Co-Authored-By: Claude Opus 5 <noreply at anthropic.com>
[mlir][gpu][spirv] Convert memref<mma_matrix> to spv.array<coopmatrix> (#212806)
In a gpu.func, allow allocating local arrays of gpu.mma_matrix, e.g.
`memref.alloca() : memref<Nx!gpu.mma_matrix<HxW, ...>`. Extend the SPIRV
conversion to convert such memrefs to `%ARR = spirv.Variable:
spv.ptr<spv.array<N x coopmatrix>, Function>`, which can be conveniently
indexed with `spirv.AccessChain %ARR[i]`. This enables arrays of CoopMMA
matrices both at the `gpu` and `spirv` levels.
Signed-off-by: Fabrizio Indirli <fabrizio.indirli at arm.com>
[Clang] Define `__SIG_ATOMIC_TYPE__` macro (#213934)
Define `__SIG_ATOMIC_TYPE__` for compatibility with GCC.
This fixes `riscv32-netbsd` and `riscv64-netbsd` system headers that
define `sig_atomic_t` using this macro.
Reference:
https://gcc.gnu.org/onlinedocs/gcc-16.1.0/cpp/Common-Predefined-Macros.html
Follow up of #199678, Closes #213895
[Clang][AMDGPU] Add ``amdgpu_av("none")`` attribute for atomic expressions (#199622)
Add a statement attribute that suppresses MakeAvailable/MakeVisible
cache operations on AMDGPU atomic instructions while preserving memory
ordering (waits). The attribute takes a string argument specifying the
mode. Currently "none" is the only supported mode. The resulting atomic
or fence instruction carries !mmra !{!"amdgcn-av", !"none"} metadata.
This only works with builtins that get lowered to intrinsics or
instructions. The attribute does not survive inlining. For example, a
C++ std atomic is typically a wrapper around a Clang builtin, and
applying this attribute on the std atomic does not remove the
MakeAvailableMakeVisible semantics built into it.
Part of a stack:
- #199486
- #199621
- #199489
[5 lines not shown]
