[AMDGPU] Introduce ABI occupancy for object linking
This PR introduces ABI occupancy as the contract used to compile functions under
object linking. The default is the occupancy needed for a 1024 workitem
workgroup. It can be overridden for a module by the `amdgpu_abi_waves_per_eu`
module flag, replaced for a function by `amdgpu-flat-work-group-size`, or
tightened by an accepted `amdgpu-waves-per-eu` hint.
The ABI occupancy replaces the workgroup-derived minimum that an
`amdgpu-waves-per-eu` hint is validated against, so the hint can only make the
contract stricter.
The backend emits the selected occupancy in `.amdgpu.info` and uses it to
enforce the object linking register budget.
[IR2Vec] Add UncondBr and CondBr to the seed vocabulary (#243025)
Update `llvm/lib/Analysis/models/seedEmbeddingVocab75D.json`: PR #184027
split `Br` into `CondBr` and `UncondBr`, so all branches got zero
vectors
[clang-tidy] Keep side effects in readability-simplify-boolean-expr (#243303)
`f() || true` and `f() && false` were rewritten to the literal value,
which removed the call to `f()`. The check still warns in these cases
but if the left operand may have side effects it now doesn't give a fix.
Fixes #242838.
[AMDGPU] Fix LDS externalization for object linking
With object linking, the backend converts LDS variables and named barriers to
external declarations, and the linker allocates them. There are two cases that
could potentially make invalid IR or cause a crash:
- LDS or a named barrier with vague linkage, for example a shared variable in an
inline function, is in a comdat. The external declaration kept the comdat, and
the verifier rejected the module: "Declaration may not be in a Comdat!".
- Alloca promotion to LDS runs after LDS lowering. The LDS variable that it
created stayed an internal definition, and the asm printer asserted: "LDS
definitions should have been externalized when object linking is enabled".
This PR fixes both of them.
This PR was assisted by AI but I reviewed all the changes.
[mlir][xegpu] Reject dynamic shapes in TensorDesc type (#242540)
`-xegpu-blocking` crashes when a loop-carried `xegpu.tensor_desc` has a
dynamic dim. TensorDesc should have a static shape, but the verifier
allowed `?`. Now the verifier rejects it.
AI tool use: an AI coding assistant was used to write the code.
Fixes #203676
NAS-145250 / 27.0.0 / Isolate nvmet render failures and fix FILE namespace sizing (by bmeagherix) (by bugclerk) (#20229)
A FILE namespace whose size isn't a multiple of 512 bytes works with the
kernel target but is rejected by SPDK. That single failure aborted the
whole SPDK config render, breaking unrelated namespaces, stale-config
cleanup and HA ANA state.
Changes:
1. **Isolate per-item failures in the SPDK nvmet render.** A bad item
is logged and skipped, and the rest is applied.
2. **Add similar failure isolation for the kernel nvmet render**
3. **Validate FILE namespace size** on create and update,
for both targets.
4. **Use the actual file size when resizing FILE namespaces.** The
stored size could be unset (causing a crash on resize) or stale
(letting an "expansion" truncate the file).
5. **Skips SPDK host links for subsystems not visible on the HA standby
node.** These always failed, and previously aborted the standby's
[11 lines not shown]
[LFI] Use context register for runtime call offsets (#241549)
Since all LFI targets use a context register, we can move the rtcall
page to be offset from the context register instead of placed
immediately before the sandbox (LFI used this design to avoid needing a
context register in all configurations, a design we are no longer
looking to support). This gives us a cleaner sandbox layout. The LFI
runtime has support for ctxreg-based runtime calls.
[Clang][AMDGPU] Name the builtin in required target feature diagnostics (#218358)
`CheckAMDGCNBuiltinFunctionCall` checked the required target features of
the `image load, store, sample and gather4` builtins itself. This
duplicated `CodeGenUtils::checkTargetFeatures`, which CodeGen and CIR
run for every
builtin call they emit.
Assisted by: Claude Sonnet
sysutils/qga-guest-exec: New port: Run commands in a guest via the QEMU Guest Agent protocol
qga-guest-exec is a utility that executes commands inside a virtual machine
guest through the QEMU Guest Agent Protocol. It connects to the guest agent
socket and requests command execution, supporting both synchronous mode with
output capture and polling, and asynchronous mode that launches the process
and exits immediately.
WWW: https://github.com/DtxdF/qga-guest-exec/
[libc][math] Rename PolyEval.h to polyeval.h.
Rename src/__support/FPUtil/PolyEval.h to polyeval.h to match lowercase
snake_case header naming, and update includes, CMake, and Bazel rules.
[clang] Report errors from writing the PCM instead of crashing
The `writeImpl()` function never checks the `raw_fd_ostream` for errors.
If writing the PCM fails (e.g. because the disk is full), the stream's
destructor reports a fatal error instead of `writeImpl()` returning the
error code to the caller. This PR checks the stream for errors, closing
it explicitly to also catch errors reported on close, and returns them.
[VPlan] Allow tail folding early exits with side effects
Continues the work in #179595.
This allows us to tail fold e.g. the following loop:
```c
extern char a[1024], b[1024], c[1024], y[1024];
void two_guards(void) {
for (int i = 0; i < 1000; i++) {
if (a[i] == 0)
break;
if (b[i] > 3) {
if (c[i] < 9)
y[i] = 1;
}
}
}
```
[30 lines not shown]
[VPlan] Remove header masks whose tail lanes aren't used
Replace a use of header-mask && x with x if the user, transitively
through freeze, logical-and, not and mask.beforefirst(freeze ...), only
reaches masks that are and'ed with the header mask again. The header
mask is a prefix and lane i of mask.beforefirst only depends on lanes
0..i, so the lanes outside of the header mask are never used.
This avoids the i1 vp.merge that EVL tail folding needs for the inner
header mask, e.g. in the early exit mask.beforefirst(freeze(header-mask
&& c)) and in switch default edges.
[VPlan] Support tail folding uncountable early exits
Enables the first step towards #179595. After #237254, the MaskedHandleExitInVectorLoop early exit style uses the CFG to handle masking for early exit loops. This allows it to compose with tail folding which is also handled via the CFG in #176143.
This allows us to tail fold the following loop:
```c
extern char x[256];
int f(char * restrict y) {
for (int i = 0; i < 256; i++) {
if (x[i] == 0)
return i;
}
return -1;
}
```
Into this on RISC-V:
[18 lines not shown]
[VPlan] Use vputils::findCanonicalIVIncrement in more places. NFC
In a future patch we want to enable more scenarios with early exits and tail folding. In these scenarios the terminator will no longer be a BranchOnCount/BranchOnCond but a BranchOnTwoConds, so a bunch of canonical IV matchers break. Use vputils::findCanonicalIVIncrement instead since it searches the users of the canonical IV instead of looking at the terminator.
[VPlan] Allow early exit loops with side effects by masking in the vector loop.
When all blocks with side effects are dominated by every early exit, we can vectorize the loop if we mask off any lanes past the exit. We can do this with control flow:
```mermaid
flowchart TD
header --> exiting
exiting -->| exit mask | sideeffects --> latch --> latch.split
exiting -->| !exit mask | latch.split
```
For every early exiting block we add a branch to the latch on `!mask.beforefirst(exitcond)`. This is implemented in UncountableExitStyle::ReadOnly which is now renamed to MaskedHandleExitInVectorLoop.
This is a complementary exit style to MaskedHandleExitInScalarLoop. If any of the side effects aren't dominated by the early exit it still falls back to the scalar loop style, which is why none of the existing early exit side effect tests changed. Masking in loop is profitable for RISC-V and I'm assuming it's also profitable for AArch64 too, but if not then we can add a TTI hook for a preferred exit style.
[VPlan] Preserve SSA in early exit loops
MaskedCond is used to logically-and the condition to early exit with the current block in-mask. However because we use it in the latch block where the condition isn't defined on all paths, we have to workaround it not being in SSA in the verifier.
This fixes this by using reconstructSSA added in #212209 to preserve SSA in both the exit conditions and live outs in the exit blocks. It also means we get the right exit condition without MaskedCond and allows us to remove the MaskedCond exceptions in the verifier. MaskedCond is still kept since a followup PR will still use it.
One subtle point is that we need to freeze any branch conditions. Any lanes after and including the early exiting lane might contain poison and we can have branches on those poison lanes that didn't we didn't previously branch on in the scalar loop.
There's no miscompile on main today because we coincidentally don't keep SSA so exit conditions are read directly without being polluted by poison stemming from a branch past the early exit. But in this PR the exit conditions are polluted and we can end up with poison in the condition operand of a select, which causes an undefined result for AnyOf even if it freezes its operand. The poison in the branch will also cause problems in future if we decide to preserve uniform control flow and emit branches instead of blends. The exit_cond_merged_after_poison_branch test illustrates this.