[mlir][Affine] Prevent buffer hoisting out of affine.parallel (#225324)
I noticed that `buffer-loop-hoisting` can move allocations out of
`affine.parallel`, making a buffer that should be local to each
iteration shared between the parallel iterations.
`AffineParallelOp` implements `LoopLikeOpInterface` but was missing the
`HasParallelRegion` trait, so `isSequentialLoop` treated it as
sequential. This adds the missing trait to `affine.parallel`, allowing
the pass to recognize the parallel boundary without changing the
hoisting logic.
I’ve added a regression test based on the reproducer in #225149, which
checks that the allocation stays inside `affine.parallel` and that the
store and load still use that allocation.
Fixes #225149.
[Clang][RISCV] Add packed widening subtraction accumulate intrinsics (#225764)
Add Clang header intrinsics for the RISC-V P-extension packed widening
subtraction accumulate operations:
- __riscv_pwsuba_i16x4
- __riscv_pwsuba_i32x2
- __riscv_pwsubau_u16x4
- __riscv_pwsubau_u32x2
Each computes rd + (a - b). The header wrappers use generic LLVM IR,
so RV32 selects the direct `pwsuba.*` / `pwsubau.*` instructions while
RV64 lowers the generic IR to the decomposition specified by the
P-extension intrinsic spec.
[Clang][RISCV] Add packed subvector insert intrinsics (#225771)
Add __riscv_pset_i8x4_i8x8, __riscv_pset_u8x4_u8x8,
__riscv_pset_i16x2_i16x4 and __riscv_pset_u16x2_u16x4, which insert a
32-bit packed subvector into a 64-bit packed vector. The index selects
the low or high half and must be a constant, enforced with
__enable_if__ as for the element insert intrinsics.
[flang][cuda] Defer data-transfer conversion in OpenACC routines (#225906)
Add an option so cuf-convert can leave cuf.data_transfer in the
host copy of an OpenACC routine until host/device specialization.
After that, transfers in a specialized device body become
assignments. Place allocas inside IsolatedFromAbove offload
regions so later FIR lowering does not store into a temporary
defined outside acc.compute_region.
[RISCV][P-ext] Select PPAIREO.W when packing with the high word of a value (#225763)
The low-half packed subvector insert is a v2i32 whose low word is the
subvector and whose high word is the high word of the wide vector,
which is exactly `ppaireo.w` (the low word comes from `rs1`, the high
word from `rs2`). Select it directly instead of `srli` + `pack`.
Adds `rvp-ppaireo-w.ll`; `rvp-simd-64.ll` is updated because the scalar
low insert in `test_insert_vector_32` now selects `ppaireo.w`.
[SCCP] Call correct simplifyBinOp in FP operation
This simplify FPOps with correct FMF, which covers non-inlined function
that can not be simplified by InstComb pass.
Also, update the TODO as we have cover that case in simplifyBinOp. But
per-element optimization is not enable as we don't have structure-like
LatticeMap for vector.
[AMDGPU] Fold a constant add/sub into the sudot4/sudot8 accumulator
Fold a constant add into the accumulator operand of sudot4 and sudot8 when
clamping is disabled:
```
sudot(a, b, C1, false) + C2 -> sudot(a, b, C1 + C2, false)
```
Subtraction by a constant is canonicalized to addition of its negation.
[AMDGPU][InstCombine] Fold zero dot operands to accumulator
Fold AMDGPU dot intrinsics when either operand is zero.
`dot(a, 0) = 0` and `dot(0, b) = 0`, so replace the intrinsic with its accumulator.
This avoids unrelated clamp and add/sub reassociation cases.
[AMDGPU][InstCombine] Canonicalize dot constant operands (#225083)
Move constant dot product source operands to the right hand side
and add tests for signed and unsigned dot intrinsics.
[flang] Fold fir.if with a constant condition (#225542)
Use `populateRegionBranchOpInterfaceInliningPattern` to fold `fir.if`
when the condition is a constant. It inlines the region the condition
selects, replacing uses of the `fir.if` results with the operands of
that region's `fir.result`, and erases the `fir.result`. A false
condition with no else region leaves nothing behind.
[orc-rt] Make StandaloneMachOUnwindInfoRegistrar internal. (#225982)
StandaloneMachOUnwindInfoRegistrar depends on libunwind's
__unw_add_find_dynamic_unwind_sections, which is only available on
Darwin, and is only consumed within the runtime. Move its header to
orc-rt-internal/bedrock/sys/darwin/ (no longer installed), and its
sources and unit test to the corresponding sys/darwin/ directories.
The SPS CI adder header stays public, but is now installed only on Apple
platforms.
[RISCV][MC] Allow x0 as the destination of qc.inw (#225602)
According to the Xqciio specification, `qc.inw` does not impose any
restriction that excludes `x0` as the destination register. In
`qc.inw.yaml`, `rd` is defined as:
```
- name: rd
location: 11-7
```
This commit removes the incorrect `x0` restriction for `qc.inw`.
Link:
https://github.com/qualcomm/riscv-unified-db/blob/main/arch_overlay/qc_iu/inst/Xqci/qc.inw.yaml
[RISCV] Support getJumpTableIndex (#224197)
This patch implements getJumpTableIndex hook for RISC-V; it trace from
PseudoBRIND back to the %jump-table.N.
The li instruction in jumptable dispatch block exists because the phi
constant is emitted during instruction selection. LLVM chooses the
source block in which to place the phi constant, then postpones moving
it to MachineSink, which relocates the phi constant closer to its use
site. However, to sink it out of the jump table dispatch block,
MachineSink needs to split the critical edge, which requires updating
the jump table entry. It needs the getJumpTableIndex hook to know which
jump table to update.
This avoids unnecessary instructions in the hot path (the jump table
dispatch block).
[AMDGPU] Measure MFMA read hazards at each producer
Introduce more sophisticated traversal to avoid the following traps:
- order-dependent traversal and discarding seen BBs despite shorter path
- mis-matching distance and window of different producers
Record the best distance per BB instead of a visited flag and sweep the
arrivals in nondecreasing distance (bucket queue). This pairs producers
with their actual distance to a consumer in one go.
Fixed scenarios:
- MFMA reading an MFMA result as srcA, srcB or srcC
- VALU, memory or export instruction reading an MFMA result
rewrite-vgpr-mfma-to-agpr.ll gains an s_nop 2: a 4-pass XDL write that
partially overlaps the srcC read two slots later requires five wait
states, and none were emitted because the nearest producer wrote the
register in full.
AI-assisted.
[AMDGPU][NFC] Extract the MFMA read-window calculation
Move the wait states a consumer needs before reading an MFMA result out
of checkMAIHazards90A into getMFMAReadWaitStates, taking the producer as
an argument, so a caller can ask about a specific producer. The partial
srcC overlap half moves into getMFMAOverlappedSrcCWaitStates. The caller
passes the producer the walk recorded, so nothing changes.
AI-assisted.
[AMDGPU] Measure MFMA overwrite hazards at each instruction
Apply previously established processing to:
- VALU overwriting an MFMA result
- VALU overwriting a register an MFMA took as srcC
AI-assisted.
[RISCV][GlobalISel] Fix fptosi/fptoui from half to i64 on RV32 (#222316)
The fcvt.l[u].h patterns are RV64-only, so RV32 had no rule for
`G_FPTOSI/G_FPTOUI` with `{s64, s16}.` The magnitude of a half is at
most `65504`, so with Zfh the i32 result of `fcvt.w[u].h` never
overflows and can simply be extended to i64.
[AMDGPU] Use synthetic fixtures for generic feature validation tests (#225551)
Remove the classification test that includes the full AMDGPU target.
Real targets are validated when generating AMDGPUTargetParserDef.inc
during the build.
Extend the lightweight generic feature tests to cover inherited
classifications of backend-only features, support through member-side
implications, and unsupported generic-side implied features.
Addresses the timeout reported on #223179
