[mlir][vector-to-gpu] Do not convert ops masked by vector.mask (#226548)
A `vector.transfer_write` inside a `vector.mask` has no mask operand.
Since the write has no results, the `vector.mask` is not included in the
MMA conversion slice, and the write is converted inside the mask region,
producing invalid IR on both the `gpu.subgroup_mma` and NVGPU paths.
Treat operations for which `MaskableOpInterface::isMasked()` is true as
unsupported by MMA conversion. This leaves the entire slice unconverted.
The two paths differ in what masking support would require. On the
`gpu.subgroup_mma` path, the write becomes
`gpu.subgroup_mma_store_matrix`, which has no mask operand, so the
current conversion cannot preserve the mask. On the NVGPU path, the
write becomes per-lane `vector.store` operations, which could in
principle be replaced by `vector.maskedstore` operations with masks
derived from the fragment layout. This patch does not implement that and
bails out on both paths.
[4 lines not shown]
[CIR] Admit only CIR types as nested CIR types (#233846)
FuncType::verify required every type reachable from a signature to be a
CIR type, and proved it with a deep walk on every !cir.func
construction.
#232863 memoized the walk in the dialect behind a mutex, but a record
that is still incomplete cannot be cached, and during CIRGen most of
them
are.
The walk was needed because !cir.ptr, !cir.array, !cir.data_member,
!cir.bitfield and the record types took unconstrained mlir::Type
parameters. Constrain them to CIR types instead. Each type then checks
only its own parameters, a CIR type holds only CIR types at any depth,
and FuncType::verify checks only its inputs and result. The walk, the
cache and the mutex go away.
- Pointee and data member types use a new CIR_DialectType constraint,
and the array element constraint now requires it too.
[19 lines not shown]
[LoopInterchange] Look up loop cache costs by original loop (NFC)
Part of #235496
A later patch will change which IVs a Loop object holds. Therefore,
change the way CacheCost is looked up so that it does not depend on
the correspondence between IVs and Loop objects.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[AMDGPU][Codegen] Don't incorrectly shrink `S_LOAD_DWORDX{16,8}_IMM` during remat (#225209)
This came up after #214725 but is a separate issue.
`SIInstrInfo::reMaterialize` should not shrink the width of
`S_LOAD_DWORDX{16,8}_IMM` if
1. `UsedLanes` is the full lanemask, keeping this consistent with
`S_MOV_64` handling after #184333,
2. or if there is more than one user of the rematerialized register. The
current implementation only checks that the instruction being
rematerialized before has a single use of the register.
The added test fails without this patch because of the incorrect
one-user check (the load is shrunk despite the full register width being
used).
[CIR][CUDA][HIP] Skip alias emission when the aliasee is not emitted (#234627)
Copy shouldSkipAliasEmission from classic codegen
Without it, device code emits host-only aliases, which fail translation
on spirv64-amd
[LSR] Fix LCSSA preservation in RewriteForPHI (#235361)
There was a call to SplitLandingPadPredecessors that did not specify
LCSSA preservation. Set it to ShouldPreserveLCSSA so we preserve LCSSA
here in the NewPM.
Found when doing a build of the LLVM test-suite with LCSSA preservation
enabled for LSR. This was not found in my initial testing (bootstrapping
LLVM) given we compile with -fno-exceptions.
Assisted-by: LLM
[libc][math] Fix `tanpif` and `tanpif16` error handling (#235296)
Relates to
https://github.com/llvm/llvm-project/pull/147192#issuecomment-5726874002.
- Set ERANGE when tanpif and tanpif16 underflow.
- Set ERANGE instead of EDOM at half-integer poles.
- Raise underflow and inexact exceptions explicitly in tanpif16.
- Return exact results for odd multiples of one quarter without raising
inexact.
Tested both functions with CORE-MATH's exhaustive checker in all four
rounding modes. With `CORE_MATH_SUPPORT_ERRNO` and
`CORE_MATH_CHECK_INEXACT` enabled.
[InstCombine] Remove redundant add one folds (#235664)
These are fully subsumed by the `sext(bool) + C -> bool ? C - 1 : C` and
`(iN X s>> (N - 1)) + 1 --> zext (X > -1)` folds higher up in the
function.
[CGP] Remove handling of lifetime.end via bitcast (NFC) (#235712)
Nowadays lifetimes are required to have a direct alloca argument,
there cannot be a bitcast in between.
[IR] Remove handling for removed constant expressions (NFC) (#235702)
Support for these casts as constant expressions has been dropped,
so don't handle them in getWithOperands().
[AMDGPU] GFX125x: Relax global_inv/wb when NC MTYPE is not used
When the driver doesn't use MTYPE NC on gfx125x, the hardware ensures coherency
of the L2 caches. Then, the global_inv/wb instructions we emit only need to
reach the L2s to acquire/release data to device or system scope.
That's the case for gfx1250, gfx1250-strict, and gfx1251. gfx12-5-generic is
intentionally left out.
[Loads] Sign-extend narrow constant GEP indices in decomposeLinearExpression (#235092)
Constant GEP indices narrower than the index type were multiplied by the
stride without being widened, which asserted on mismatched APInt widths.
[libc] Use a syscall wrapper for `epoll_pwait2` (#235701)
This PR move `epoll_pwait2` behind the centralized syscall wrappers,
following `libc/docs/dev/syscall_wrapper_refactor.md`.
And slightly make the cmake and bazel target more sorted.
[MLIR][Index] Restrict subtraction comparison canonicalization (#216831)
Fix the unsafe canonicalization of ordered comparisons involving
`index.sub`.
Ordered comparisons may change semantics when the subtraction wraps, so
the
rewrite is now restricted to `eq` and `ne`.
For example, at a 32-bit index width, let `x = -2^31` and `y = 1`.
Then `x - y` wraps to `2^31 - 1`, so `(x - y) < 0` is false while
`x < y` is true.
Regression tests cover ordered comparisons while preserving the valid
`eq` and
`ne` canonicalizations.
Fixes #216042
[2 lines not shown]
[AArch64][GlobalISel] Split wide G_SMULH/G_UMULH instead of lowering them (#235327)
Vector `G_SMULH`/`G_UMULH` wider than 128 bits were lowered to widening
multiplies and shifts. This splits them into legal 128-bit pieces first,
as `G_MUL` already does, so they select to `smull`/`umull` pairs and
`uzp2`.
For example, `udiv <8 x i32>` by a constant:
```
define <8 x i32> @test(<8 x i32> %x) {
%r = udiv <8 x i32> %x, <i32 7, i32 6, i32 7, i32 6, i32 7, i32 6, i32 7, i32 6>
ret <8 x i32> %r
}
```
goes from 27 instructions to 22, now identical to SelectionDAG: the
half-register extractions (`mov d2, v0.d[1]`, ...) are gone and the
multiply-highs use `umull2`.
[CIR] Fix cir.try successor regions
`TryOp::getSuccessorRegions` sent every exit from the try region back to the
parent, so nothing led from the try region to the handlers. A `cir.resume`
ending the try region, left there once FlattenCFG has flattened an inner
cleanup scope, unwinds into a handler, and `CIRTryOpFlattening` branches it
to the handler dispatch block. Now every exit from the try region can enter
the handlers, and a `cir.resume` leaves the op only when there are none. The
parent keeps its edges to the handlers. An unwind out of a call in the try
region is not a region branch point, so these edges stand in for it, and
without them -remove-dead-values empties a handler that is reached only that
way.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[CIR] Keep cir.cleanup.scope cleanup regions reachable
RegionBranchOpInterface::getSuccessorRegions must return every region that
may run next, but an unwind out of a call in the body, or a cir.return,
cir.break, cir.continue or cir.goto out of it, runs the cleanup without
passing a region branch point. A cleanup reached only that way had no
predecessor, so generic passes treated it as dead, and -remove-dead-values
erased an EH cleanup that stores to memory. Now the cleanup region is a
successor of the parent and of every body exit, and a body exit that the
cleanup kind does not cover can also leave the op.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[mlir][scf] support back peeling of signed integer loops (#235260)
the signed split bound is now the arith chain instead of `affine.max`
hence supporting peeling of signed non-index loops with dynamic bounds.
from #234435
claude helped me with tests and comments
