[Offload][AMDGPU] Remove AMDGPU check from checkMachine (#236974)
This check, among others, must be updated whenever a new processor is
added. Since processor compatibility is already checked later by the
AMDGPU plugin, and this is only an early sanity check, removing it
leaves one less place that can be missed when adding new GPUs.
[AMDGPU] Fix VGPR and AGPR critical pressure accounting (#234358)
* Use combined VGPR/AGPR pressure for critical-pressure scoring and
scheduling checks on targets with unified allocation.
* Preserve the full occupancy budget before clamping per-bank limits,
and apply VGPR alignment only when AGPRs are live.
* Keep per-bank excess checks to account for spills, and save and
restore the combined critical-pressure limit during rescheduling.
Related issue: #219377
[AArch64][TTI] Fix Apple runtime unroller to handle scalar epilogues (#235877)
Previously this unroller skipped loops marked as "vectorized" trusting
that LoopVectorizer did its job well. However, LoopVectorizer marks
all of the sub-loops (vector, vector epilogue and scalar epilogue) this
way even if it introduces a runtime check. In that case, scalar epilogue
might be required for the whole execution and we still don't consider
it for runtime unrolling.
Instead, we use "llvm.loop.unroll.runtime.disable" that LoopVectorizer
also puts on all of the sub-loops with one important exception: scalar
epilogues when we introduce a runtime check. That makes this attribute
a better alternative to base our decision on.
[SLP]Keep extractelement used by other gather nodes
Copyable lanes may put the same extractelement into several gathers.
Assisted-by: Cursor
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/237294
AArch64: Respect subregister operands in some peephole opts (#237091)
Get the correct operand constraint class accounting for any
subregister indexes on the operands. This previously would use the
direct register class from the virtual register, not accounting for
the subregister indexes in the operands. Also remove pointless
attempts to treat inability to constrain as a recoverable condition
since all the uses asserted anyway.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[RISCV][P-ext] Add isel pattern for mulsu.w11/maccsu.w11 (#237135)
On RV64, a signed*unsigned multiply of the upper 32-bit halves of two
GPRs is written in the DAG as
`(mul (sra rs1, 32), (srl rs2, 32))`
Without a dedicated pattern this lowered to srai + srli + mul (or
maccsu.w00 when the product fed an add). Add patterns that select
mulsu.w11 for the multiply and maccsu.w11 for the accumulate form
(add rd, (mul_oneuse ...)). They match the existing mul.w11/macc.w11
[GlobalISel] Don't set disjoint flag in lowerRotate unless the rotate amount is non-zero modulo the bitwidth. (#236993)
If the bit width is a power of 2 and the shift amount is zero modulo the
bitwidth, the left and right shift produce the same value so the Or is
not disjoint.
For the non-power of 2 lowering, we make sure to produce 0 in one of the
shifts when the rotate amount is zero modulo the bit width. This keeps
the Or disjoint.
We can't allow any element of the shift amount to be undef because the
disjoint flag produces poison which is stronger than undef.
Fixes #236025.
Assisted-by: Claude
[AMDGPU] Materialize WMMA SGPR scales to VGPRs before scheduling
v_wmma_scale scale operands accept an SGPR or a VGPR. On GFX12,
depctr_va_ssrc is a 1-deep counter, so each VALU instruction that reads
an SGPR scale takes the only scalar-source slot. A run of WMMA ops that
all read SGPR scales therefore serializes. One cross-bank copy pays for
that slot once; later WMMA ops then read a VGPR and no longer contend
for va_ssrc.
SIMaterializeWMMAScale runs at the end of the machine SSA optimizations,
after SIShrinkInstructions and before scheduling. It is skipped at -O0
and on targets older than GFX12. GFX1250 is included: the separate VALU
read-after-write hazard no longer applies there, but the dependency
counter still does.
The pass walks every WMMA that has scale_src0 and records virtual SGPR
uses of scale_src0 and scale_src1, grouped by register. Physical
registers and subregister uses are left alone. For each SGPR it inserts
a COPY into a low aligned VGPR (the lo256 class matching the scale's
[22 lines not shown]
[lldb][test] Don't expect an exact thread count in a Windows stop reply (#237038)
On Windows the process can have thread pool threads besides the 5
threads the test creates, so this test was marked as an expected failure
on Windows. When no extra thread exists at the stop, the test passes and
the run fails as an unexpected pass: 102 out of 2000 runs on a loaded
machine.
This patch checks for at least 5 threads on Windows and keeps the exact
count elsewhere.
With this patch, 2000 out of 2000 runs pass.
[llvm][cas] Fix indefinite wait for CAS shared locks (#236929)
On Darwin, flock may only wake a single waiting shared lock request
after an exclusive lock is released. If that first waiter happens to be
a long-running CAS client, it may indefinitely delay other waiting
shared accesses that are already queued up. Workaround this with a retry
loop so that all shared CAS clients will run concurrently.
Assisted-by: claude (helped with writing the test)
rdar://191274568
SelectionDAG: Build FP immediates from integer bits in the legalizer (#233767)
Teach the generic legalizer to legalize FP immediates as a bitcast of
an opaque integer constant. Migrate LoongArch to use the new helper
instead of doing it in custom lowering.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[SystemZ][z/OS] Add missing ADA reference to DWARF sections
Labels marked as DATA needs a reference to the ADA. However, this
is missing for the DWARF sections, which can lead to crashes in
certain scenarios. Fix is to set the ADA reference.
Assisted by: IBM Bob
[clang][ThreadSafety] Instantiate a parameter's attribute naming a later parameter (#232784)
Under -fexperimental-late-parse-attributes, an attribute on a parameter
may name a parameter declared after it:
template <class T> void f(void (*release)(T) RELEASE(mu), Mutex *mu);
A parameter's attributes were instantiated as the parameter was, before
any later one, so instantiating this asserted in
LocalInstantiationScope::findInstantiationOf, and the late parsing
change rejected it in a template. Defer a parameter's late-parsed
attributes, with the mechanism a class uses for its members, until every
parameter of the prototype is instantiated in
TransformFunctionTypeParams. A function whose type needs no substitution
instantiated its parameters one by one instead; route it through
SubstParmTypes, so there is one place that defers. SubstParmTypes now
takes whether to evaluate constraints, which SubstFunctionType passes in
both of its calls, so a typedef'd function type also follows the
instantiator's setting instead of always evaluating them. With that,
[3 lines not shown]
ARM: Remove kill flags on virtual registers from MIR tests (#236921)
The verifier will soon disallow these. Delete tests which only differ
from another in the kill flags.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
