[mlir][ODS] Add so-called "Natural Sharding" to ops
Prior to this PR, ODS had one mechanism for sharding which equally split a `<Dialect>Ops.cpp.inc` file into `n` many, as requested on the command line (using preprocessor defines for selective enablement).
The motivation for doing so is that this can speed up compile time by splitting class definition compilation to multiple translation units.
However, this approach has one large disadvantage: Since only the `<Dialect>Ops.cpp.inc` is split, the correpsonding `<Dialect>Ops.h.inc` is still included in each translation unit. For a large dialect like NVVM or LLVM this leads to **much higher** concurrent memory usage while compiling as every compilation process contains an in-memory copy of the header plus can lead to slower compilation time overall if too many shards are being used as overall work goes up (`n` header parses instead of 1).
This PR tries to address these issues by introducing a new kind of sharding called "natural sharding". The idea is that we split `<Dialect>Ops.cpp.inc` and `<Dialect>Ops.h.inc` along so called partitions. A new header can be created for every partition and every translation unit only contains the code from its partition. This means a TU no longer needs to include the entire dialect header but only the header from its partition.
One can think of these as effectively being sub-dialects from the perspective of the required boilerplate.
We call it "Natural Sharding", as the partitions are defined by the dialect authors and are meant to be categories that naturally categorizes a dialects operations (e.g., Memory ops, control flow ops, arithmetic ops, intrinsic ops, etc. in the LLVM Dialect). The new `-op-partition` then filters output generation to just a given partition.
This PR introduces the mechanism and for testing, already exercises it on the test dialect. Users of `TestOps.h` were further changed to just include a given partition if deemed useful.
Overall this yielded the following build gains vs the existing sharding:
* Building `MLIRTestDialect` sources now takes **30% less Wall time** on my 16 thread machine, 50% less CPU time and **50% less memory** (both clang and GCC)
* Building other TUs (unittests mainly) that include `MLIRTestDialect` is roughly **25% faster wall time** and CPU time and takes about **20% (clang) to 25% (GCC) less memory** to compile.
Future plan is to introduce this for types and attributes as well
[lldb] Use TargetParser for AMDGPU architectures (#223192)
Fixes #218864
These PRs enabled the authoritative GPU list in TargetParser:
- #206480: `amdgpuN.M` triple subarchitectures, compatibility, and
reverse mapping.
- #212353 and related work: TableGen-generated TargetParser GPU tables.
- #220328: `getSubArchFromGPUName()`.
Replace the manually maintained LLDB AMDGPU target list with
TargetParser GPU and subarchitecture information.
R600 is intentionally left unsupported. It is a legacy architecture
family that is not supported by amd-dbgapi. Exact CPU extraction from
ELF headers remains limited to AMDGPU HSA ABI V3 through V6; other
AMDGPU objects remain generic.
[3 lines not shown]
[libc] Implement swprintf. (#235008)
Replace the initial stub implementation with the real thing.
* The entrypoint declarations are moved out of the "experimental"
sections.
* `printf_main` and `printf_main_modular` are the last pieces in
__support/printf_core to be templated on character type.
* Test cases are added from stdio/sprintf_test.cpp and
stdio/snprintf_test.cpp.
Assisted-by: Automated tooling, human reviewed.
[flang][OpenMP] Fix sections construct emitting a barrier with a nowait clause. (#238269)
Currently the OpenMP sections construct still emits an implicit barrier
at the end of its region when a reduction clause is present, even if a
nowait clause is present. For example:
```fortran
subroutine repro(sum)
integer :: sum
!$omp sections reduction(+:sum)
!$omp section
sum = sum + 1
!$omp end sections nowait
end subroutine
```
Emits a barrier call:
```sh
flang -fopenmp -S -emit-llvm repro.f90 -o repro.ll
rg 'call.*@__kmpc_barrier' repro.ll
```
[23 lines not shown]
[AMDGPU][GlobalISel] Fold unmasked signed byte loads into D16 high loads (#236136)
A left shift by 16 discards the upper half of a 32-bit sign-extending
load. Accept the bare load in the high-half D16 matcher as well as the
existing masked and zero-extended forms, retaining the 8-bit
memory-width check.
Previously, the combiner recognized the signed load only when its result
was explicitly limited to 16 bits:
(sextload i8 → i32 & 0xffff) << 16
zext(sextload i8 → i16) << 16
It now also recognizes:
(sextload i8 → i32) << 16
Patches a regression discovered with
https://github.com/llvm/llvm-project/pull/230636
Assisted by: GPT-6.1-Sol
[TTI][NFC] Add list overload of getVectorInstrContextHint (#239062)
Move SLP's helper that computes a VectorInstrContext across a list of
values with demanded elements into TTI, next to the single-instruction
overload, so other vectorizers can reuse it.
Requested in review of https://github.com/llvm/llvm-project/pull/237651
Assisted by AI
[AMDGPU] Move the bitcast cost comments, document the i1 mask test
Put the comment about the scalarized parts of a vector bitcast above
the size check, and the comment about the shared registers inside it.
Say in the test which case @i1_mask covers.
Assisted-by: Claude Opus
[AMDGPU] Fix store wait elision when BUFFER_INV is pending (#236549)
### Overview
Starting with https://github.com/llvm/llvm-project/pull/220694, on
gfx942/950 hardware, functions whose only outstanding LOAD_CNT events at
the end of a function are stores do not wait for the stores to complete
before returning. The current PR reverts that behavior, reintroducing
the vmcnt wait.
### Background
I made that change because I misunderstood
https://github.com/llvm/llvm-project/blob/main/llvm/docs/AMDGPUUsage.md?plain=1#L3525,
i.e. I thought `On return from a function, all dependency counters must
be zero except for VScnt/STOREcnt` was talking about HWEvents, not
counters.
While fixing this I realized:
[24 lines not shown]
[flang] Treat stack save/restore and fir.unreachable as not accessing memory
These operations do not declare memory effects, so the alias analysis
assumed they may modify and reference any memory. They do not access
memory: answer NoModRef for them.
[mlir][bufferization] Drop unused MLIRSparseTensorDialect dependency (#239229)
MLIRBufferizationDialect has not used anything from the sparse tensor
dialect since #85482 removed the sparse-specific verification of
bufferization.alloc_tensor, and #147549 removed the last include of
SparseTensor.h. Drop the stale link dependency.
Co-authored-by: Claude <noreply at anthropic.com>
[Attributor] Don't move allocations passed to musttail calls to the stack
Like a tail call, a musttail call cannot access allocas of the caller,
but its marker cannot be removed. AAHeapToStack still moved an
allocation to the stack when it was passed to a musttail call, so the
callee got a pointer into a stack frame that no longer exists.
This PR treats such a use as a bad use. The free-based check still applies,
because an allocation that is always freed is freed before the musttail
call.
This PR was assisted by AI but I reviewed all the changes.
[OpenACC] Wrap multi-block acc.loop bodies in scf.execute_region (#238974)
convertACCLoopToSCFFor always cloned the loop body directly into the
innermost scf.for. cloneACCRegionInto only handles single-block regions,
so a structured acc.loop whose body has more than one block hit an
unreachable in the orphan-loop path of acc-specialize-for-host.
Wrap multi-block bodies in scf.execute_region, as
convertACCLoopToSCFParallel already does. Single-block bodies are still
cloned directly into the scf.for, so existing output is unchanged.
Add a unit test for a two-block body and a acc-specialize-for-host check
for an orphan multi-block loop.
[CLANG][OpenMP] Restore loop variable values after transformation directives (#214717)
This patch fixes loop variable finalization for OpenMP 6.0
loop-transformations constructs: tile, stripe, reverse, interchange and
fuse to comply with spec requirement page 371, lines 19-21. The spec
requires that "After the execution of the loop-transforming construct,
the loop-iteration variables of any of its transformation-affected loops
have the values that they would have without the loop-transforming
directive".
[mlir][bufferization] Drop unused MLIRSparseTensorDialect dependency (NFC)
MLIRBufferizationDialect has not used anything from the sparse tensor
dialect since #85482 removed the sparse-specific verification of
bufferization.alloc_tensor, and #147549 removed the last include of
SparseTensor.h. Drop the stale link dependency.
Co-authored-by: Claude <noreply at anthropic.com>
[AMDGPU][True16] Keep hi16 in 32-to-16-bit VGPR-to-SGPR copies
When moveToVALU turns a VGPR-to-SGPR copy from a 32-bit VGPR into a
16-bit register, it sets the source subregister to lo16. If the copy
already read a 16-bit subregister, this overwrites it:
%1:sgpr_lo16 = COPY %0.hi16:vgpr_32
became
%1:vgpr_16 = COPY %0.lo16
which reads the wrong half. Only default to lo16 when the source has no
16-bit subregister.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[AMDGPU][NFC] Pre-commit test for a true16 VGPR-to-SGPR copy of hi16
A VGPR-to-SGPR copy of a 16-bit hi16 subregister into a 16-bit SGPR
currently ends up reading lo16:
%1:sgpr_lo16 = COPY %0.hi16:vgpr_32
becomes
%1:vgpr_16 = COPY %0.lo16
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[AMDGPU][True16] Don't narrow 32-bit uses when folding a VGPR-to-SGPR copy
True16 has no 16-bit SGPRs, so a 16-bit VGPR value is copied into a
32-bit SGPR:
%3:sreg_32 = COPY %2.lo16:vgpr_32
Since #211848, moveToVALU removes this copy by replacing every use of %3
with %2.lo16. Uses that read all 32 bits become malformed, e.g. a PHI:
%5:vgpr_32 = PHI %x, %bb.0, %2.lo16, %bb.1
DetectDeadLanes then treats %2 as dead, and the load defining it is
deleted. On gfx1250 this drops the conditional f16 load of alpha/beta
in rocSPARSE and rocBLAS kernels in device pointer mode.
Now uses that read 16 bits, including %3.lo16, get %2.lo16. Uses that
read 32 bits get one 32-bit value built at the copy with REG_SEQUENCE.
[6 lines not shown]
[AMDGPU][NFC] Add a helper to widen a 16-bit VGPR value to 32 bits
Three places in SIInstrInfo and SIFixSGPRCopies build the same sequence
to put a 16-bit value into the low half of a 32-bit VGPR:
%undef:vgpr_16 = IMPLICIT_DEF
%dst:vgpr_32 = REG_SEQUENCE %src.sub, %subreg.lo16, %undef, %subreg.hi16
Move it into SIInstrInfo::buildVGPR32FromLo16(). The source subregister
is a parameter, so callers can't silently drop it.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[AMDGPU][NFC] Pre-commit tests for true16 VGPR-to-SGPR copies of 16-bit subregs
Add tests for VGPR-to-SGPR copies of a 16-bit VGPR subregister in
true16 mode. The current output is wrong in all of them:
- v2s_copy_lo16_phi (MIR) and true16-v2s-copy-subreg-phi.ll: a 32-bit
PHI gets a 16-bit subregister operand (%src.lo16). On gfx1250 the
conditional f16 load feeding the PHI is then deleted.
- v2s_copy_hi16_through_sgpr32 (MIR): a copy of %src.hi16 ends up
reading %src.lo16.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[AMDGPU] Stop publishing reserved registers from AMDGPUReserveWWMRegs (#237446)
Since #226523, the register allocators publish the reserved register set
they
freeze to the shared RegisterClassInfo. That makes the publication in
AMDGPUReserveWWMRegs redundant: it recomputes the set with
TRI->getReservedRegs(), and the VGPR allocator republishes the same set
right
after it.
The first commit asserts at the entry of AMDGPURewriteAGPRCopyMFMA, the
consumer
that was handed a reserved register before #221849, that the analysis
matches
MRI's frozen set. It adds RegisterClassInfo::hasCurrentReservedRegs()
for the
comparison. check-llvm passes with it; without the allocators'
publication,
many AMDGPU tests trip it, whereas no test notices without the
[9 lines not shown]
[MLIR][Bytecode] Fix test failure depending on version length (#239179)
This test is testing alignment, and would fail whenever the version
string (which is embedded in the bytecode as the producer) ended up
putting the resource section at a 32-byte aligned offset. This doesn't
happen with the default producer string, but can happen with an unlucky
value of `LLVM_VERSION_SUFFIX`.
Fixes an issue that was exposed by
https://github.com/llvm/llvm-project/pull/222776.
[mlir][ODS] Include headers required by generated enum declarations and definitions
Generated enum declarations and definitions use C++ constructs that are
not declared by any header they include. They therefore relied on the
surrounding source file to include these headers first.
The enum declarations and definitions now include the headers declaring
these constructs, depending on the features of the enums.
The declarations still only forward declare `AsmParser` and `FieldParser`
to keep the generated headers cheap to include.
Similar to #236155 and #238735 but for the enum generator.
Co-authored-by: Claude <noreply at anthropic.com>
[mlir] remove dead values: don't remove used symbols (#238760)
This was hitherto accidental for functions and region-carrying globals,
make it intentional. In a custom dialect, one may have a side effect
free global-like operation with a symbol referenced from elsewhere in
the translation unit. We don't want that to be removed.
Assisted-by: Gemini 4 Argon
[clang][LoongArch] Pick the default ABI of environment-less triples from the architecture (#231441)
For a LoongArch triple without an environment, such as
loongarch32-unknown-none-elf, the
driver defaults to the double-float ABI (ilp32d), but none of the 32-bit
architectures
(loongarch32, la32v1.0, la32rv1.0) has an FPU, so the driver passes
`-target-abi ilp32d`
together with features that cannot implement it. Compiling anything then
warns:
```
warning: both target-abi and the triple-implied ABI are invalid, ignoring and using feature-implied ABI
```
and the parts disagree on the ABI: the backend falls back to ilp32s,
while clang lowers calls
and defines the predefined macros for ilp32d (`__loongarch_double_float`
is 1 while
[14 lines not shown]
[LoongArch] Load scalars into vector lane 0 with FLD/VLDREPL (#234119)
Inserting a loaded integer into lane 0 of an undefined vector, which is
also how sub-128-bit vector loads are legalized, was lowered to a scalar
load plus [X]VINSGR2VR. VINSGR2VR merges into its destination, so it
carries a false dependency on the previous value of that register, and
it needs a GPR-to-vector move.
Use FLD.{S/D} for 32/64-bit elements and [X]VLDREPL.{B/H} for 8/16-bit
elements instead. They write the whole register in one instruction.
When the register allocator reuses one vector register for several such
loads, the false dependency serializes otherwise independent chains. In
the GSM autocorrelation loop (9 x i64 multiply-accumulate over i16 taps)
on LA664:
VF2: 0.953s -> 0.724s
VF4: 1.446s -> 0.856s
