[TailCallElim] Add profile annotations to return value selects
Tail call elimination in some cases can create selects on possible
return values conditioned on whether or not execution is currently in
what was a recursive call. That is equal to the probability with which
we recurse, which in turn can be computed from the block frequencies of
blocks that recurse and blocks that directly return.
Reviewers: mtrofin
Reviewed By: mtrofin
Pull Request: https://github.com/llvm/llvm-project/pull/202518
[SelectionDAG] Handle constants in SimplifyMultipleUseDemandedBits
Replace a non-zero constant with zero when none of its set bits are
demanded.
This allows users of `SimplifyMultipleUseDemandedBits` to eliminate
irrelevant constant bits while preserving the convention that a null
SDValue indicates no simplification.
[Clang] Add missing release note entry in #226753 (#227078)
As per the feedback from #226753, we add release note for GH-212211.
Also move the test to new-delete.cpp.
Assisted-by: Claude
[CodeGen] Enable validated AMDGPU wave spill costs by default
Use available validated wave counts for AMDGPU spill placement without an
explicit opt-in. Keep the existing target, mapping and normalization checks
and the ordinary block-frequency fallback for unavailable or rejected data.
Retain -enable-wave-profiled-spill=false for debugging and matched performance
comparisons. Test default behavior, explicit disabling, equivalence to explicit
enabling, fallback cases and the positive frequency floor.
[RISCV][P-ext] Support the missing packed pair (#226992)
This patch supports the packed pair for v2i16.
Some are inconsistent with the spec because existing shuffle-lowering
transforms them into equivalent narrowing shifts or zips.
[AMDGPU] Fold sudot intrinsics with a zero multiplicand
Replace sudot4 and sudot8 with their accumulator when either
multiplicand is zero.
For example:
```
sudot4(sign0, x, sign1, 0, acc, clamp)
->
acc
```
[AMDGPU] Canonicalize constant operands of sudot intrinsics (#226478)
Move a constant multiplicand and its sign flag to the second operand.
For example:
```
sudot4(true, 1, false, x, acc, clamp)
->
sudot4(false, x, true, 1, acc, clamp)
```
[CodeGen] Keep wave-profiled spill frequencies positive
SpillPlacement expects positive block weights, but a valid wave profile can
record zero executions for a CFG-reachable block. Giving such a block zero
spill cost can make the allocator choose a very different placement.
Clamp every accepted wave-derived frequency to at least one, as we already
do for nonzero counts that round down to zero. Unmeasured or rejected blocks
still use their existing MBFI frequency. Add a focused MIR test for a valid
zero-wave record.
This pattern arose in a profiled Composable Kernel convolution case. With
the separate spill correctness fixes and partial spilling enabled, the
zero-cost policy failed two CPU-reference checks; the positive floor passed
both. The test checks the cost directly; the application result was checked
separately on gfx950.
[CodeGen] Use validated wave counts for AMDGPU spill costs
Lane-based block frequencies can understate the cost of a spill in
divergent GPU code: a wave still executes a block with only some lanes
active. Use measured block-wave counts to weight SpillPlacement's costs
relative to the original entry-wave count.
Opt in on AMDGPU only. Accept a measured count only when its IR block
maps uniquely to a machine block with matching predecessors and
successors. Keep the existing MBFI cost for unmeasured or rejected
blocks, and leave branch probabilities and general BFI unchanged.
[InstrProf] Replace !PGOFuncName and !PGOName metadata with !guid (#214134)
!PGOFuncName and !PGOName metadata were attached to internal functions
and vtables during profile annotation to record their original
"<file>;<name>" PGO names before ThinLTO promoted and renamed them. In
post-link LTO passes, InstrProfSymtab read that metadata back so profile
records keyed by the original name's hash could still find the renamed
IR object.
Global objects now carry stable !guid metadata assigned before LTO
renaming, which records the MD5 hash of the original PGO name directly.
See: https://discourse.llvm.org/t/rfc-keep-globalvalue-guids-stable/84801
Use !guid instead of maintaining separate PGO name metadata:
- Stop emitting and reading !PGOFuncName and !PGOName in Clang and
PGOInstrumentation, and remove the metadata helper functions
(createPGOFuncNameMetadata, createPGONameMetadata,
getPGOFuncNameMetadata, and the metadata name getters).
[13 lines not shown]
[Transforms] Preserve wave profiles across CFG rewrites
HIP device PGO attaches measured wave counts to IR blocks. Later CFG
rewrites can drop counts from unchanged blocks or leave stale counts
on blocks that now execute differently. Either case makes the profile
unreliable for later optimizations.
Preserve counts through switch lowering, structurization, and loop
rotation only when a block still represents the same executions. Keep
unaffected counts and their IDs even when a loop header's count must be
invalidated. Transfer branch hints only for equivalent decisions, and
avoid assigning switch weights when default traffic cannot be traced
to one edge.
[PGO] Load dense block wave counts from device profiles
Use the profile's dense layout flag to map appended wave-only slots after
the original block/select prefix. Reuse the producer's block selection so
eligible loop and reconvergence blocks retain their directly measured wave
frequencies. Keep select slots out of the block mapping.
Test dense and sparse profiles, measured zeros, generation and metadata
switches, and exact loop block identities after critical-edge splitting.
[PGO] Add a debugging switch for wave profile metadata
Add the hidden pgo-wave-metadata option, enabled by default, for debugging,
performance comparisons and disabling wave annotations when investigating
regressions without turning off ordinary PGO or uniformity hints.
Gate wave metadata emission while retaining the existing clearing of stale
function and block annotations during profile use. Profile collection and
ordinary count reconstruction are unchanged.
Test default/explicit enablement, disabling, retained counts and uniformity
hints, and replacement profiles with missing, mismatched or zero counts.
[PGO] Load GPU wave counts into IR metadata
GPU profiles contain wave counts alongside lane counts, but profile use
does not expose them to optimizations. Wave visits do not obey scalar
flow conservation, so unmeasured blocks cannot use counts reconstructed
from neighboring blocks or ordinary branch weights.
Map wave-counter indices to the blocks selected by PGO instrumentation,
after reproducing its critical-edge splits. Attach measured counts using
wave.profile metadata, retaining measured zeros and marking other blocks
unmeasured. Require a measured entry count for normalization and exclude
select-counter slots from the block mapping.
Validate the wave-counter layout against the accepted lane profile.
Clear old wave metadata when loading a replacement profile, including
when a function has no usable record. Do not emit wave metadata for
previously profiled functions: their branch weights may change counter
placement without changing the CFG hash. Keep ordinary lane-count
reconstruction and branch weights unchanged.
[IR] Define GPU wave-profile metadata
Existing offload GPU profile counters measure lane executions, while
GPU instructions execute at wave granularity under an active-lane
mask. A block visited by every wave can therefore look cold when only
a few lanes are active. Scalar branch weights also cannot represent a
divergent wave visiting both successors before reconverging. These
profiles are a poor fit for optimizations that estimate work performed
by a wave.
Lane counters remain useful for measuring per-lane branch selectivity
and estimating work that scales with the number of active lanes.
Wave counters cannot replace them: a visit with one active lane and a
visit with all lanes active both count as one. The two profiles provide
complementary information about instruction execution and lane activity.
Introduce wave.profile and wave.profile.block metadata to represent
measured wave visits to IR blocks. Each dynamic visit with at least one
active lane contributes one to the count. A measured zero is distinct
[18 lines not shown]
[PGO] Collect dense AMDGPU block wave counts
Wave visits are not additive across divergent control flow. Sparse scalar
counter sites can leave repeated loop blocks unmeasured, so their wave
frequencies cannot be reconstructed from entry and edge counts.
Append zero-step instrumentation for eligible unmeasured AMDGPU blocks,
keeping the existing lane and select counter indices unchanged. Exclude the
appended slots from lane-flow reconstruction and uniformity annotation.
Identify the layout with a profile variant bit, preserve it through raw and
indexed readers/writers, and select it automatically during profile use.
Keep sparse profiles readable and reject incompatible merges, including
concatenated raw profiles. Ignore empty merge-worker contexts.
Enable dense collection for ordinary AMDGPU IR-PGO by default, with the
hidden -pgo-instrument-dense-wave-counts option for debugging. Leave
context-sensitive, coverage, and temporal instrumentation unchanged.
[2 lines not shown]
[mlir][scf] Add unsignedCmp to scf.parallel and use it in the tiling in-bound check (#226130)
This change mirrors `scf.for` where `scf.parallel` gets an `unsignedCmp`
unit attribute, and every pass that rebuilds or lowers a `scf.parallel`
has to respect it:
- Propagate where a loop is rebuilt from another, such as parallel loop
tiling, parallel loop fusion (which also refuses to fuse loops of
different signedness), parallel-to-nested-fors and SCF-to-CF lowering.
- Decline where the bound arithmetic assumes signed values, such as
SCF-to-GPU, SCF-to-OpenMP, async-parallel-for and the
parallel-loop-collapsing test pass.
Fixes #223233
[ProfCheck] Exclude DirectX (#227157)
I don't think anyone runs profiling on DirectX, so exclude it for now.
Also move AMDGPU to the normal exclusion list given there are efforts
around PGO for AMDGPU currently.
[lldb/test] Register xros (visionOS) as a Darwin platform (#227142)
`lldbplatform.py` never registered `xros` at all: no enum value, no
`__name_lookup` entry, and it was absent from `__darwin_embedded`/
`darwin_all`. As a result, `platformIsDarwin()` returned `False` for
`xros`, and `finalize_build_dictionary`'s fallback branch, which indexes
`platform_name_to_uname` by the raw platform name, would `KeyError`
before any visionOS test could build.
Add `xros` alongside the other embedded Darwin platforms so it's
recognized the same way `ios`/`tvos`/`watchos`/`bridgeos` already are.
Signed-off-by: Med Ismail Bennani <ismail at bennani.ma>
[Github] Temporary fix for #226230 (#227147)
We are seeing this issue in the libc++ runner sets and it can presumably
pop up in the libc runner sets as well in the case of an abnormally long
clone operation.
[lldb/test] Normalize appletvos to tvos in getPlatform() (#227138)
Mirror the existing `iphoneos` -> `ios` SDK-name normalization for
`tvOS`. Without it, an `--apple-sdk` value derived from "appletvos" made
`getPlatform()` return "appletvos", which isn't in
`lldbplatform.__name_lookup`, so `platformIsDarwin(`) returned `False`
and `finalize_build_dictionary` KeyError'd before any `remote-tvos` test
could build.
Signed-off-by: Med Ismail Bennani <ismail at bennani.ma>
[SPIR-V] Sign extend narrow G_SMIN/G_SMAX operands and mask sign sensitive results (#226957)
This PR includes `G_SMIN` and `G_SMAX` to the list of sign sensitive
ops. It also masks the result of the sign sensitive ops to their
original widths with a bitwise AND because the later users assume the
upper bits are zero and without masking they read the wrong value.
[flang][acc] Warn when a loop is above its routine parallelism level (#227041)
OpenACC spec allows a routine to parent a loop at its own parallelism
level or below. A higher level is not allowed: gang inside a worker
routine, gang or worker inside a vector routine, gang, worker, or vector
inside a seq routine, and a gang dimension above the routine's gang
dimension. A device-specific routine clause counts the same way.
Flang accepted these loops with no diagnostic. Warn that the clause is
ignored. A loop at the same level or below is left alone, and illegal
nesting of loops remains an error.
