[AArch64] Fold four and eight way partial reductions with [SU]ADDLP (#214636)
#214691 lowers two-way partial reductions to [SU]ADALP, which adds each
pair of lanes together and widens the result. This adds the four-way and
eight-way folds, which are two and three of those steps.
Each is lowered by folding one step and re-entering, with [us]mull
widening the products first when there is a multiply.
Registrations added:
```
(v4i32, v16i8) four way without dotprod
(v2i64, v16i8) eight way without dotprod
(v2i32, v16i8) eight way without dotprod
(v2i64, v8i16) four way always, no neon dot product takes i16
```
Example:
```llvm
[36 lines not shown]
AMDGPU/GlobalISel: Bitcasting G_TRUNC combine
Creating the G_TRUNC that changes type as well. This is really
the LLT::scalar trunc style that we inherited from switching to
extended LLTs, very common on non-true16 targets.
Affects inst-select pattern matching that were blocked by bitcast.
AMDGPU/GlobalISel: Fold (trunc (lshr x, 16)) to unmerge high half
Huge performance impact in some shaders with high register pressure.
What this really does is selects _hi16 register classes, and true16
machinery takes over later. Trunc + shift ends up in _lo16!
This requires allocating an extra register when the _lo16 half of the
original real 32-bit VGPR is still live.
[LAA] Bound non-affine monotonic pointer expressions for runtime checks. (#210626)
We currently refuse to generate runtime checks for non-affine AddRecs (
e.g. std::bitset-style addresses involving unsigned divides, like
words[i/N])
This patch adds getNonAffineMonotonicBounds, whic h , which forms a
tight bound for such accesses, if the offset is provably monotonic and
non-decreasing over the loop.
It uses the offset value at the first iteration
(SplitIntoInitAndPostInc) and at the last iteration (getSCEVAtScope).
Initially this just supports divides of affine AddRecs, to be extended
in the follow-ups.
First step towards fixing
https://github.com/llvm/llvm-project/issues/207882.
PR: https://github.com/llvm/llvm-project/pull/210626
[LoopInterchange] Swap preheader contents before rebuilding LCSSA (#218468)
adjustLoopLinks() swapped the inner/outer preheader bodies only after
adjustLoopBranches() had already moved the reduction PHIs and rebuilt
LCSSA via formLCSSAForInstructions(). A reduction init defined in the
inner preheader was therefore still stranded below its use on the seed
edge when LCSSA was rebuilt, so formLCSSAForInstructions() was handed
dominance-broken IR.
Move the swapBBContents() call into adjustLoopBranches(), after the
replacePhiUsesWith() relabeling and before the LCSSA rebuild, so the
reduction init is relocated into the new outer preheader and dominates
the seed edge. With the swap moved, adjustLoopLinks() only forwarded to
adjustLoopBranches(), so it is inlined into its sole caller and removed.
Fixes #215511
AI tools have been used to generate portions of this patch
[mlir] Fix mlir-pdll-lsp-server link. (#220123)
This is the same fix TableGenLspServerLib got in 9e469ced42cd. This
addresses the mlir-pdll-lsp-server instance of #152371.
MLIRPdllLspServerLib transitively needs LLVMTableGen, which is not part
of libLLVM.so, so MLIRPDLLParser already brings in static LLVMSupport.
Linking the dylib here as well gives mlir-pdll-lsp-server two copies of
LLVMSupport. Linking still succeeds, but with assertions
view-output.test fails. The test trips over the two copies having
different Hashing.h seeds, which is a separate defect to be addressed in
another PR.
It has not shown up in CI because it needs the dylib and assertions at
the same time, which rarely coincide. It also only became reachable once
a3a25996b114 moved the LSP transport into libLLVM.so.
Assisted-By: Claude Opus 5
[CodeGen][RegCoalescer] Preserve valid lanes when keeping IMPLICIT_DEF (#215272)
A subregister definition without the undef flag is a read-modify-write,
including when it is defined by IMPLICIT_DEF. When the IMPLICIT_DEF cannot be
erased, retain the lanes carrying values from the preceding definition.
Otherwise the coalescer can miss the interference between %src and %dst and
turn the first two copies below into a destructive in-place permutation.
[AMDGPU] Add tests for scalar hi-half f16 fma and mad on GFX9 (#219432)
v_fma_f16 and v_mad_legacy_f16 both support op_sel on GFX9, so a 16-bit
ternary
whose sources come from the high halves of packed registers could be a
single
instruction. SelectVOP3OpSel() does not set op_sel yet, so the high
halves are
extracted and re-inserted with explicit shifts instead. Add tests for
both SDAG
and GlobalISel so that the improvement is visible if this is ever fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply at anthropic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply at anthropic.com>
[LV] Give each RUN line in smax-idx.ll its own check prefix (NFC). (#220210)
The three RUN lines share the CHECK prefix, which only works while all
three VF/IC configurations produce identical output. This fixes the
dropped checks for smax_idx_max_no_exit_user.
While touching the file, also modernize the naming a bit, to be more
uniform with existing tests.
