[CodeGen] Fix physical-register input latency for inline asm (#229764)
As described in #229538, the scheduling DAG classifies inline asm’s
variable inputs as pseudo uses, assigning zero latency and hiding
modeled producer latency from scheduling decisions. The dependency
preserves instruction order, but the scheduler does not account for the
time needed to produce the result.
This patch is a more generic approach which fix the issue in general
infrastructure.
Fix approach:
* In `ScheduleDAGInstrs`, recognize explicit inline-asm inputs as real
uses and allow explicit definitions feeding them to bypass the
pseudo-definition check.
* The existing `computeOperandLatency()` then supplies modeled producer
latency, including for asm-to-asm dependencies.
* Implicit bookkeeping operands and other consumers retain their
existing behavior.
[3 lines not shown]
[AMDGPU] Keep soft waitcnts unchanged until SIInsertWaitcnts converges
On the first visit to a loop header only the preheader state is known.
A soft waitcnt there (e.g. from a release fence) can have some of its
counters simplified away while another counter keeps the instruction
alive: the fence's own vmcnt(0), a preceding wait.asyncmark, or an
explicit s_waitcnt. The soft waitcnt was then rewritten with the
simplified value and promoted to a hard waitcnt, or merged into a
preceding hard one. When the back edge later brought a pending LDS
store, the original lgkmcnt(0) request was gone, and the s_barrier after
the release fence no longer waited for it.
Treat soft waitcnts as requests that depend on the incoming state: never
merge, promote or rewrite them while the analysis iterates. They are
kept unchanged in DeferredErasures, a hard waitcnt (existing or new)
carries what is needed on each visit, and the soft waitcnts are erased
once the analysis converges. This extends #220357, which did the same
only for soft waitcnts that were simplified away entirely.
[11 lines not shown]
[clang] Correct the argument declaration of 'ptrauth_vtable_pointer' in Attrs.td (#243446)
The three enum parameters of 'ptrauth_vtable_pointer' were declared with
IsString set, but its custom handler requires *identifier* arguments in
those positions and rejects anything else.
There is no observable behavioral difference since the attribute does
not take the generic path where IsString would matter.
Found by observation.
[AMDGPU] Pre-commit tests for soft waitcnt requests lost at a loop header (NFC)
Two cases where a soft waitcnt at a loop header loses its lgkmcnt(0)
request: on the first visit no LDS operation is pending, so lgkmcnt is
simplified away while vmcnt keeps the instruction alive (either the soft
waitcnt's own vmcnt, or a preceding hard vmcnt(0) it is merged into).
The DS write in the loop is then not waited for before the S_BARRIER on
the next iteration. The checks show the current, incorrect output.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[X86] lowerShuffleAsZeroOrAnyExtend - ensure we bitcast VZEXT_MOVL source to the result type (#244646)
lowerShuffleAsZeroOrAnyExtend can be used with v4i32 and v4f32 types,
but we expect VZEXT_MOVL to be used with v4i32 - isel was still working,
but improvements to SDNode verification was causing an assert
Fixes #244568
[ConstFold][NFC] Add IR denorm misfolding baseline (#236190)
This patch adds baseline tests with FIXME comments to highlight results
that are missing denormal_fpenv-aware flushing during const-folding with,
which will be addressed in a follow-up patch.
Currently ConstantFolding.cpp is the only IR code-path that tries to handle
denormal_fpenv while const-folding, and even there it misses several
cases like FMA and fmax. This results in inconsistently folded results in
several cases with a non-IEEE denormal_fpenv.
The cases ignoring denormal_fpenv that are covered in these new baseline tests are:
- InstSimplify arithmetic: FMA, rounding, comparisons, and min/max.
- Constant folding during InstCombine reassociation and factorization.
- BranchProbabilityInfo folding of loop-carried value chains.
- NVVM multiply, divide, and FMA at the subnormal rounding boundary.
[AArch64][GlobalISel][NFC] Add denormal const-fold baselines (#241382)
Add GlobalISel lit tests for AArch64 recording the current scalar
constant folding behaviour for f16, f32 and f64 under IEEE,
flush-to-zero and dynamic denormal environments.
Add TODO comments describing the desired results if const-folding were
to honour denormal_fpenv, which will be added in a follow-up commit.
[InlineSpiller] Use a new value number for the COPY range of a fold (#243055)
When folding a memory operand makes the target insert a COPY (e.g. a
reload folded into an X86 NDD instruction), foldMemoryOperand() extends
the original live interval from the COPY to the folded instruction
(#199182). That segment reused the folded instruction's value number,
though the register holds the COPY's source there.
HoistSpillHelper groups spills by (stack slot, original value number),
so rmRedundantSpills treated the store the COPY was folded into and the
later spill of the folded result as duplicates, and deleted the latter.
In memfold-origVNI-crash.ll this dropped an `or`, returning %p2 instead
of %p2 | %p3.
Give the segment its own value number at the COPY. Replace the
getVNInfoAt() guard with an assert: the sub-interval invariant already
requires OrigLI to be live at FoldMI's def.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[InlineSpiller] Do not merge or hoist a spill with no original value (#237148)
HoistSpillHelper keeps a copy of the original live interval, taken at
the first spill to each stack slot and never updated, and keys
MergeableSpills by the original VNInfo of each spill. Since #199182,
foldMemoryOperand can extend the original interval backwards over the
COPY that an X86 NDD fold inserts (#191368). A spill later folded into
that COPY lies outside the saved interval, so getVNInfoAt returns null.
The null VNInfo* is stored as a MergeableSpills key and dereferenced in
runHoistSpills (SIGSEGV).
Leave such a spill out of merging and hoisting; it stays where it was
inserted. Both are optimizations, so skipping them is safe, and inputs
that did not crash before get the same code.
Assisted-by: Claude Opus 5.5 <noreply at anthropic.com>
[ValueTracking] Track constant ExtractElement indices in sign-bit queries (#244296)
Query only the selected lane for an in-range constant index into a fixed
vector.
This matches the behavior in SDAG.
Assisted by: GPT 6.1 Sol
[SelectionDAG] Honor denormal modes when folding (#236172)
Introduce shared denormal-aware FPFold helper functions in the Support
library.
Then use the new tryFoldFP API within SelectionDAG to correctly fold
arithmetic, FMA, comparisons, rounding, and min/max using the enclosing
MachineFunction's denormal mode.
Decline folds when dynamic flushing can change the result or IEEE
rounding can promote a subnormal FTZ result to minimum-normal. Update
the NVPTX tests added in #236060 to verify the correct behaviour.
[PowerPC] Model the CR defs of MTCRF used to restore callee-saved CRs (#242763)
The epilogue restores the non-volatile CR fields (CR2-CR4) with a single
`mtcrf`. MTCRF takes only an FXM mask immediate and a GPR, and declares
no CR operands, so the post-RA scheduler sees no dependency between the
restore and instructions that read or write the restored CR bits. It can
therefore hoist the restore above those reads, making the function use
the caller's condition bits instead of its own.
Add the restored CR fields as implicit defs when building the MTCRF in
`emitEpilogue()` and `restoreCRs()`, matching the way the prologue
already adds implicit uses to MFCR. The scheduler now models the WAR/WAW
dependencies and keeps the restore after the last read.
This is a regression from #202339 (3fc6711ccaf6), which replaced the
per-field MTOCRF sequence -- each MTOCRF defines the CR field it
restores -- with a single MTCRF, which declares no CR defs.
Fixes #242250.
Handle the func kind in createMIROperandComment
createMIROperandComment read a func flag's constraint code as a register
class, giving "func:FR16X" for a "p" operand, and didn't print the code.
Treat a func flag like a mem flag there, as MachineInstr::print does. The
MIR printer doesn't print this comment for a flag, so test it with a unit
test.
Also assert that a func flag isn't given a register class, and say in the
flag layout comment that a func flag holds a constraint code.
Assisted-by: Claude Opus 5.5
[VPlan] Only narrow store groups storing a single load group (#244730)
When narrowing a store group whose members are all load interleave group
values, only the defining group of each member was checked to hold it at
the matching index. Members could come from different load groups, but
the store group was narrowed to a copy of the first member's group only.
Require all members to come from the load group of member 0.
[clang][Driver] Check the existence of -gen-reproducer=off instead of -fno-crash-diagnostics
-fno-crash-diagnostic is an alias of -gen-reproducer=off. That means
The check for -fno-crash-diagnostic does not work as expected.
This patch resolves it.
[TargetInstrInfo] Address review: autogenerate greedy-inline-asm-fold.mir checks
Generate the checks with update_mir_test_checks.py so that the spills and
reloads around each folded inline asm are verified, not just the INLINEASM
itself. Fix the "=r" comment to describe the C "+r" operand, note that which
"+rm" pair greedy folds is its own choice, and add a "+&rm" case on an
uninitialized value showing that folding a pair whose tied use is undef
stores to the slot without loading from it.
Assisted-by: Claude Opus 5.5
Test a tie whose def moves when a folded operand is before it
Folding a "+rm" pair before a "+r" pair moves the "+r" pair's def, so
both ends of its tie get new indices. The other tests only moved the
use.
Assisted-by: Claude Opus 5.5