[mlir][transform] Fuse selected producers in reverse topological order (#241779)
`transform.structured.fuse_into_containing_op` can leave a shared
producer partially unfused when it is processed before one of its
selected consumers.
For a dependency such as `P -> R -> Y` with an additional `P -> Y` edge,
P may be removed from the pending set before fusing R introduces another
use of P. In the standard Softmax decomposition, this leaves both
full-tensor and tiled exponential computations, even after
canonicalization and CSE.
Process selected producers in reverse topological order so that consumer
fusion exposes all relevant producer uses before the producer is
retired. Also diagnose parentless producers before sorting.
Regression tests cover Softmax with both handle orders, skip
connections, multi-result producers, distinct tiles, and invalid
top-level producers.
[5 lines not shown]
[mlir] Hoist invariant ops out of nested regions in LICM (#227145)
LICM only looked at ops directly in the loop body, so an invariant op
inside an `scf.if` or `affine.if` that stays in the loop was never
hoisted. The shared LICM utility now also looks into control-flow
regions nested in the loop body and hoists pure ops whose operands are
defined outside the loop.
Fixes #226878
Fixes #215458
Fixes #215492
Used Claude to write tests
[LoopVectorize] Add llvm.loop.vectorize.fp_unsafe.{enable,disable} metadata (#219167)
Add a new per-loop metadata key llvm.loop.vectorize.fp_unsafe to
explicitly control whether the Loop Vectorizer may apply floating-point
transforms that change the computed result:
!{!"llvm.loop.vectorize.fp_unsafe.enable"} ; allowed
!{!"llvm.loop.vectorize.fp_unsafe.disable"} ; suppressed
The hint governs both places where this is currently implied by
vectorize.enable.
- allowReordering() - reassociation of FP reductions
- isPotentiallyUnsafe() - vectorization on targets whose vector FP
semantics differ from their scalar semantics
When absent, both retain their existing behaviour: vectorize.enable or
an explicit vector width > 1 grants the relaxation. When present, the
hint takes precedence over that default.
[2 lines not shown]
[AMDGPU] Read TTMP6 only if clusters are enabled
The hardware only initializes TTMP6 when clusters are enabled, but
`llvm.amdgcn.cluster.workgroup.{id,max.id}.{x,y,z}` and
`llvm.amdgcn.cluster.workgroup.max.flat.id` read it without a check, so they
could potentially return stale values without clusters. The programming model
treats a launch without clusters as an implicit 1x1x1 cluster, so all of these
values are 0.
Fixes LCOMPILER-3241.
This PR was assisted by AI but I reviewed all the changes.
[CodeGen] Repair LiveStacks before compaction
Spill slot intervals survive between greedy runs, and their
boundaries can sit on instructions erased in between. Compaction
unlinks those entries, so the boundaries stop following renumbering
and overlapping slots can look disjoint. StackSlotColoring then
merges them, and one spill clobbers the other.
Move such boundaries onto the preceding instruction first.
[flang] Keep do concurrent structured when its branching stays in the body
A DO loop whose raw branching is confined to its body keeps its structured
form, with only the body folded into an scf.execute_region. Only increment
loops qualified. A do concurrent header puts no control flow in the body
either: its indices are always integer and its mask is a test inside the
innermost loop. So a do concurrent qualifies under the same body checks.
This lets a do concurrent with, for example, a SELECT CASE in its body lower
inside an OpenACC kernels region instead of hitting the "unstructured do loop
in acc kernels" TODO.
[flang][OpenACC] Fix lowering crash on DO WHILE loop constructs (#243885)
A loop construct associated with a DO WHILE crashed lowering: reading
the
DO WHILE's loop control as bounds aborted, both when the DO WHILE is
lowered
unstructured and, with -lower-do-while-to-scf-while, when it is lowered
structurally. A DO WHILE has no bounds and no induction variable, so
lower
the construct to an acc.loop without bounds that holds the whole DO
WHILE,
as for an unstructured loop, with nothing to privatize.
PR stack:
- https://github.com/llvm/llvm-project/pull/243884
- https://github.com/llvm/llvm-project/pull/243885 ⬅️
- https://github.com/llvm/llvm-project/pull/244176
[IRBuilder] Propagate nuw/nusw flags when canonicalizing GEP (#244592)
InstCombine's GEP index-width canonicalization in
InstCombinerImpl::visitGetElementPtrInst propagates the GEP's no-wrap
flags to the cast that adjusts the index width:
nusw/nuw give trunc nsw/nuw
nusw+nuw gives zext nneg when widening
That was added in commit a71909156e5.
Recently IRBuilder::CreateGEP started to canonicalize GEPs itself.
Support for that was added in commit ede30f6ce0. However, it used
CreateSExtOrTrunc, which drops the flags. When the index type was
already adjusted by IRBuilder, InstCombine never got to apply its
flag-preserving canonicalization, so the flags were lost.
This patch makes sure IRBuilder is using the same logic as InstCombine
when adjusting the index width.
[libc++] Remove redundant `_LIBCPP_HIDE_FROM_ABI` from detail headers j-o (#244539)
This is the third bulk-removal patch of `_LIBCPP_HIDE_FROM_ABI`. While
this is something we could do over time, actively removing the redundant
annotations significantly improves readability, and QoI.
This happens in a few patches to keep the size somewhat manageable.
[CIR] Implement relative-vtable-lookups (#243798)
This patch handles lookups from a relative-vtable. While the rest of the
relative V-table work is quite large, this ends up being pretty easy to
split up. Effectively, we now just make sure we lower to the LLVM
dialect properly for loads of a relative Vtable.
[CIR][AMDGPU] Remove redundant __builtin_amdgcn_lerp case in CIRGen (#243235)
Removes the redundant __builtin_amdgcn_lerp case from `clang/lib/CIR/CodeGen/CIRGenBuiltinAMDGPU.cpp` since EmitBuiltinExpr already lowers them automatically.
Reference PR #222204
[LLVM][Clang] Fill Windows DLL version metadata (#243543)
LLVM and Clang DLLs leave their description and original filename empty
and omit copyright information from their Windows version resources.
Fill these fields, provide descriptions for the LLVM and Clang C
interface libraries, and record each target's actual output filename.
Give each target its own resource and keep the metadata attached to that
target so library creation cannot overwrite another target's metadata.
[mlir][NVVM] Partition the NVVM dialect ops
Including `NVVMDialect.h` is expensive. The header declares all 219 ops of the dialect regardless of how many ops are being used by an includer.
The dialect library pays the same price: each of its 12 TUs includes all op declarations through a precompiled `NVVMDialect.h`, including the 8 TUs that each define an eighth of the ops and `NVVMDialect.cpp`, which implements all of them.
This patch replaces the index sharding with op partitions ("logical sharding", #239254). The ops are grouped by topic into 10 partitions.
Every partition has its own header, which declares the ops of the partition, and its own source file, which defines them together with their hand-written implementations (verifiers, parsers, printers, builders, intrinsic lowerings, etc.).
`NVVMDialect.h` stays as an umbrella header that includes all partition headers, but its use is discouraged. Its includers in MLIR and flang now include what they use instead unless they require the majority of the headers.
Measurements where "Memory" is the sum of the max RSS of every TU, and the users are the TUs in MLIR outside the dialect library that include NVVM dialect headers:
| TUs | Compiler | Wall time (s) | CPU time (s) | Total memory (GiB) |
|------------------------------|----------|---------------------|--------------------|---------------------|
| `MLIRNVVMDialect` (12 -> 14) | clang | 22.3 -> 20.1 (-10%) | 208 -> 193 (-7%) | 9.1 -> 8.6 (-5%) |
| `MLIRNVVMDialect` (12 -> 14) | clang -g | 26.4 -> 24.8 (-6%) | 241 -> 234 (-3%) | 9.7 -> 9.1 (-6%) |
| `MLIRNVVMDialect` (12 -> 14) | gcc | 25.3 -> 21.9 (-13%) | 222 -> 198 (-11%) | 14.9 -> 14.0 (-6%) |
| `MLIRNVVMDialect` (12 -> 14) | gcc -g | 36.8 -> 33.4 (-9%) | 324 -> 296 (-9%) | 21.0 -> 18.8 (-10%) |
| Users (10) | clang | 27.8 -> 24.3 (-13%) | 176 -> 149 (-15%) | 9.3 -> 8.0 (-14%) |
[3 lines not shown]
[coro] Remove renumberBlocks to fix assert (#244573)
GenericCycleInfo was not invalidated, so asserted when profcheck is
enabled that the block number epoche is outdated.
Remove the `renumberBlocks` call to fix this.
Given that e.g. `GenericCycleInfo` already uses the numbers as block
indices, they should still be sufficiently dense.
[MLIR][NVVM] Fix lowering of unscaled conversions on older architectures (#243009)
This patch fixes the lowering of unscaled `FP4` and `FP6` conversions
to be supported on older architectures by lowering down to the
existing unscaled intrinsics.
Lit tests are updated to reflect the same.
Assisted-by: Claude Code
Signed-off-by: DharuniRAcharya <dharunira at nvidia.com>