[mlir][vector] Only fold ext ops from the same type into vector.contract (#221298)
`FoldArithExtIntoContractionOp` checks that both contraction operands
are produced by the same kind of extension, but not that the extensions
start from the same element type. With `extsi` from i8 on one side and
from i16 on the other it rewrites the contraction to take the two
sources directly, and the result fails to verify:
```
'vector.contract' op failed to verify that lhs and rhs have same element type
```
The same happens with `extf` from f16 and bf16. Compare the source
element types and leave such contractions alone.
Test: an `extf` case in `fold-arith-extf-into-vector-contract.mlir` that
keeps the extensions.
[JumpTableToSwitch] Use ConstGlobalIterator to read jump table entries
Replace JTTS's own jump table parsing with the `ConstGlobalIterator` helper,
which is also used by instcombine's `foldCmpLoadFromIndexedGlobal`.
As a result JTTS now also handles GEPs without nusw/nuw flags, by masking the
index the same way instcombine does.
Note that the helper requires the indices to be in canonical index type, so
JTTS only works with canonical index types now. This is not a problem in
practice as JTTS only runs after instcombine.
[DAGCombiner] Drop `StartingIndex` in `calculateByteProvider`
The function comment of `calculateByteProvider` claims:
"StartingIndex is the byte position we are trying to provide for the
LoadCombine."
This is wrong:
- When not SHL, `StartingIndex` tracks the `StartingIndex` of the
current call of `calculateByteProvider`.
- When SHL, it tracks `Index`.
Since `Index` can be changed by `SHL` and `BSWAP`, this makes
`StartingIndex` track some intermediate `Index` in the chain of
recursion, which is different from the byte position.
Also, the comment claims:
"If these do not match, then we can not combine the vector loads."
This is also wrong, since MatchLoadCombine already checks that the bytes
are adjacent in memory and that the combined load is allowed.
[6 lines not shown]
[DAGCombiner] Precommit tests for vector load combine misses
The vector element check in calculateByteProvider compares the element
index with StartingIndex, which is reset by every shift, so the first
two tests are not combined. The scalable test reads past the known
minimum size of the vector load and must stay uncombined.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[DAGCombiner] Fix MatchLoadCombine address for non-zero vector offsets
When the first byte has a non-zero vector element index, it is
discarded by MatchLoadCombine, which miscompiles. Also, MatchLoadCombine
bails out if the byte offset within the scalar type is non-zero.
Fix the mis-compilation by accumulating both the vector element index
and the offset within scalar elements, and adjust the load address and
alignment accordingly. Fix the missed optimization by dropping the
`MemoryByteOffset == 0` requirement.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[DAGCombiner] Precommit tests for MatchLoadCombine non-zero offsets
The byte at the lowest address comes from element 1 of the vector load,
but the combined load currently reads from the vector's base address.
In load_i32_from_nonzero_offset, it is the second byte of an i16 load,
which is currently not combined.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[DAGCombiner] Fix load combine mask position under nested shifts
For calculateByteProvider to work correctly, `ByteMask[StartingIndex]`
needs to 'aim' at the original byte mask passed in first call of the
recursion chain. But in handling of SHL, `StartingIndex` is copied from
`Index`, while `Index` can change during recursion. This creates bug,
with the following causal chain during recursion:
1. Index is changed. Currently, SHL and BSWAP does this.
2. SHL, copy Index to `StartingIndex`.
3. AND, storing the wrong `ByteMask[StartingIndex]`.
Fix by passing `ByteMask` as a pointer to the same byte mask during
recursion.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[DAGCombiner] Precommit test for load combine mask under nested shifts
The current output applies the mask to byte 1 (0xffff0fff) instead of
byte 3 (0x0fffffff), or instead of byte 0 (0xffffff0f) when the shift
is inside a bswap.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[mlir][affine] Mask transfers in SuperVectorize instead of using in_bounds (#243425)
SuperVectorize sets `in_bounds` when the memref dim is a multiple of the
vector size, without looking at the index. With vectors of 8 on a
`memref<64xf32>`, a read of `%src[%i + 1]` reads one element past the
end, and a write to `%dst[%i + 4]` writes four past the end.
SuperVectorize now masks each transfer by `dim - index`. For that,
`createReadOrMaskedRead` now builds its mask like
`createWriteOrMaskedWrite`, from the indices and the permutation map.
SuperVectorize is the only caller that passes them.
For downstream pipelines: the masks are `vector.mask` regions, so
lowering to LLVM needs `-lower-vector-mask`. `-eliminate-vector-masks
-canonicalize` removes the masks that are provably all-true and leaves
those transfers with `in_bounds = [true]`. Main left some of them at the
default `false`, e.g. a read of `memref<100xf32>` in a loop `0 to 64`.
Transfer unrolling handles `vector.mask` since #239554.
[9 lines not shown]
[libc++][pstl] Add a serial implementation of std::rotate (#244574)
This PR adds a serial implementation of `std::rotate` to PSTL.
Currently there's no intent to provide a parallel version of this
algorithm, but later it can be added as well.
Part of #99938.
[ARM] Promote bf16 libcalls via f32 (#244879)
This add promotion rules for bf16 operations, the list of which is taken
from the current fp16 operations.
[CodeGen] Fix physical-register input latency for inline asm (#229764)
As described in #229538, the scheduling DAG classifies inline asm’s
variable inputs as pseudo uses, assigning zero latency and hiding
modeled producer latency from scheduling decisions. The dependency
preserves instruction order, but the scheduler does not account for the
time needed to produce the result.
This patch is a more generic approach which fix the issue in general
infrastructure.
Fix approach:
* In `ScheduleDAGInstrs`, recognize explicit inline-asm inputs as real
uses and allow explicit definitions feeding them to bypass the
pseudo-definition check.
* The existing `computeOperandLatency()` then supplies modeled producer
latency, including for asm-to-asm dependencies.
* Implicit bookkeeping operands and other consumers retain their
existing behavior.
[3 lines not shown]
[AMDGPU] Keep soft waitcnts unchanged until SIInsertWaitcnts converges
On the first visit to a loop header only the preheader state is known.
A soft waitcnt there (e.g. from a release fence) can have some of its
counters simplified away while another counter keeps the instruction
alive: the fence's own vmcnt(0), a preceding wait.asyncmark, or an
explicit s_waitcnt. The soft waitcnt was then rewritten with the
simplified value and promoted to a hard waitcnt, or merged into a
preceding hard one. When the back edge later brought a pending LDS
store, the original lgkmcnt(0) request was gone, and the s_barrier after
the release fence no longer waited for it.
Treat soft waitcnts as requests that depend on the incoming state: never
merge, promote or rewrite them while the analysis iterates. They are
kept unchanged in DeferredErasures, a hard waitcnt (existing or new)
carries what is needed on each visit, and the soft waitcnts are erased
once the analysis converges. This extends #220357, which did the same
only for soft waitcnts that were simplified away entirely.
[11 lines not shown]
[clang] Correct the argument declaration of 'ptrauth_vtable_pointer' in Attrs.td (#243446)
The three enum parameters of 'ptrauth_vtable_pointer' were declared with
IsString set, but its custom handler requires *identifier* arguments in
those positions and rejects anything else.
There is no observable behavioral difference since the attribute does
not take the generic path where IsString would matter.
Found by observation.
[AMDGPU] Pre-commit tests for soft waitcnt requests lost at a loop header (NFC)
Two cases where a soft waitcnt at a loop header loses its lgkmcnt(0)
request: on the first visit no LDS operation is pending, so lgkmcnt is
simplified away while vmcnt keeps the instruction alive (either the soft
waitcnt's own vmcnt, or a preceding hard vmcnt(0) it is merged into).
The DS write in the loop is then not waited for before the S_BARRIER on
the next iteration. The checks show the current, incorrect output.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[X86] lowerShuffleAsZeroOrAnyExtend - ensure we bitcast VZEXT_MOVL source to the result type (#244646)
lowerShuffleAsZeroOrAnyExtend can be used with v4i32 and v4f32 types,
but we expect VZEXT_MOVL to be used with v4i32 - isel was still working,
but improvements to SDNode verification was causing an assert
Fixes #244568
[ConstFold][NFC] Add IR denorm misfolding baseline (#236190)
This patch adds baseline tests with FIXME comments to highlight results
that are missing denormal_fpenv-aware flushing during const-folding with,
which will be addressed in a follow-up patch.
Currently ConstantFolding.cpp is the only IR code-path that tries to handle
denormal_fpenv while const-folding, and even there it misses several
cases like FMA and fmax. This results in inconsistently folded results in
several cases with a non-IEEE denormal_fpenv.
The cases ignoring denormal_fpenv that are covered in these new baseline tests are:
- InstSimplify arithmetic: FMA, rounding, comparisons, and min/max.
- Constant folding during InstCombine reassociation and factorization.
- BranchProbabilityInfo folding of loop-carried value chains.
- NVVM multiply, divide, and FMA at the subnormal rounding boundary.
[AArch64][GlobalISel][NFC] Add denormal const-fold baselines (#241382)
Add GlobalISel lit tests for AArch64 recording the current scalar
constant folding behaviour for f16, f32 and f64 under IEEE,
flush-to-zero and dynamic denormal environments.
Add TODO comments describing the desired results if const-folding were
to honour denormal_fpenv, which will be added in a follow-up commit.
[InlineSpiller] Use a new value number for the COPY range of a fold (#243055)
When folding a memory operand makes the target insert a COPY (e.g. a
reload folded into an X86 NDD instruction), foldMemoryOperand() extends
the original live interval from the COPY to the folded instruction
(#199182). That segment reused the folded instruction's value number,
though the register holds the COPY's source there.
HoistSpillHelper groups spills by (stack slot, original value number),
so rmRedundantSpills treated the store the COPY was folded into and the
later spill of the folded result as duplicates, and deleted the latter.
In memfold-origVNI-crash.ll this dropped an `or`, returning %p2 instead
of %p2 | %p3.
Give the segment its own value number at the COPY. Replace the
getVNInfoAt() guard with an assert: the sub-interval invariant already
requires OrigLI to be live at FoldMI's def.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[InlineSpiller] Do not merge or hoist a spill with no original value (#237148)
HoistSpillHelper keeps a copy of the original live interval, taken at
the first spill to each stack slot and never updated, and keys
MergeableSpills by the original VNInfo of each spill. Since #199182,
foldMemoryOperand can extend the original interval backwards over the
COPY that an X86 NDD fold inserts (#191368). A spill later folded into
that COPY lies outside the saved interval, so getVNInfoAt returns null.
The null VNInfo* is stored as a MergeableSpills key and dereferenced in
runHoistSpills (SIGSEGV).
Leave such a spill out of merging and hoisting; it stays where it was
inserted. Both are optimizations, so skipping them is safe, and inputs
that did not crash before get the same code.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[ValueTracking] Track constant ExtractElement indices in sign-bit queries (#244296)
Query only the selected lane for an in-range constant index into a fixed
vector.
This matches the behavior in SDAG.
Assisted by: GPT 6.1 Sol
[SelectionDAG] Honor denormal modes when folding (#236172)
Introduce shared denormal-aware FPFold helper functions in the Support
library.
Then use the new tryFoldFP API within SelectionDAG to correctly fold
arithmetic, FMA, comparisons, rounding, and min/max using the enclosing
MachineFunction's denormal mode.
Decline folds when dynamic flushing can change the result or IEEE
rounding can promote a subnormal FTZ result to minimum-normal. Update
the NVPTX tests added in #236060 to verify the correct behaviour.
[PowerPC] Model the CR defs of MTCRF used to restore callee-saved CRs (#242763)
The epilogue restores the non-volatile CR fields (CR2-CR4) with a single
`mtcrf`. MTCRF takes only an FXM mask immediate and a GPR, and declares
no CR operands, so the post-RA scheduler sees no dependency between the
restore and instructions that read or write the restored CR bits. It can
therefore hoist the restore above those reads, making the function use
the caller's condition bits instead of its own.
Add the restored CR fields as implicit defs when building the MTCRF in
`emitEpilogue()` and `restoreCRs()`, matching the way the prologue
already adds implicit uses to MFCR. The scheduler now models the WAR/WAW
dependencies and keeps the restore after the last read.
This is a regression from #202339 (3fc6711ccaf6), which replaced the
per-field MTOCRF sequence -- each MTOCRF defines the CR field it
restores -- with a single MTCRF, which declares no CR defs.
Fixes #242250.