[PGO] Load GPU wave counts into IR metadata
GPU profiles contain wave counts alongside lane counts, but profile use
does not expose them to optimizations. Wave visits do not obey scalar
flow conservation, so unmeasured blocks cannot use counts reconstructed
from neighboring blocks or ordinary branch weights.
Map wave-counter indices to the blocks selected by PGO instrumentation,
after reproducing its critical-edge splits. For dense profiles, also map the
appended wave-only slots to eligible blocks in function order, after the
original block/select prefix. Select the layout from the profile flags.
Attach measured counts using wave.profile metadata, retaining measured
zeros and marking blocks without measurements unavailable. Require a
measured entry count for normalization and exclude select-counter slots
from the block mapping. Repeated instrumentation sites for the same block
represent one measurement and are not summed.
Validate the wave-counter layout against the accepted lane profile.
Clear old wave metadata when loading a replacement profile, including
[14 lines not shown]
[IR] Define GPU wave-profile metadata
Existing offload GPU profile counters measure lane executions, while
GPU instructions execute at wave granularity under an active-lane
mask. A block visited by every wave can therefore look cold when only
a few lanes are active. Scalar branch weights also cannot represent a
divergent wave visiting both successors before reconverging. These
profiles are a poor fit for optimizations that estimate work performed
by a wave.
Lane counters remain useful for measuring per-lane branch selectivity
and estimating work that scales with the number of active lanes.
Wave counters cannot replace them: a visit with one active lane and a
visit with all lanes active both count as one. The two profiles provide
complementary information about instruction execution and lane activity.
Introduce wave.profile and wave.profile.block metadata to represent
measured wave visits to IR blocks. Each dynamic visit with at least one
active lane contributes one to the count. A measured zero is distinct
[19 lines not shown]
[RISCV][P-ext] Add isel pattern for mul.w01/mul.w11 (#236323)
`mul.w01` multiplies the sign-extended lower word of rs1 by the
sign-extended upper word of rs2, and `mul.w11` multiplies the
sign-extended upper words of both. Both produce a 64-bit result and
are RV64 only.
Previously, (sra X, 32) was selected as a separate srai feeding `mul`.
Match it directly so the shift is folded.
[Support] Add missing cast to uch (#236493)
In #174150, a call to `isdigit(PEEK())` was added, but `PEEK()` expands
to an expression of type `char` and `isdigit` requires its argument to
be `unsigned char`. Elsewhere in this file, this is done by just casting
the argument, `isalpha((uch)PEEK())`, so do the same here.
Revert "[asan] Adopt sanitizer_common operator-new framework (#196388)" (#236607)
#196388 made ASan's throwing operator new throw std::bad_alloc when
allocator_may_return_null=1. That exception does not reach the caller in
two configurations:
- arm64 Darwin linked with ld64.lld. asan_win.cpp compiles to an empty
object that is linked right before asan_new_delete.cpp.o, and ld64.lld
writes an encoding-0 (no unwind information) compact unwind entry at
operator new's address (#224949). The unwinder cannot step out of
operator new, so the std::bad_alloc it throws terminates the program
instead of reaching the caller's handler. The linker fix is #225055.
- Android. Both libclang_rt.asan-\*-android.so and libc++_shared.so link
static libc++abi.a, so the two libraries have separate new-handler state
and std::bad_alloc type info (#235557).
This also reverts #235557, which marked tests XFAIL on Android only
because of #196388.
[4 lines not shown]
[lldb] Add libstdcpp atomic data formatters (#174218)
This patch adds a new synthetic child provider and a summary formatter
for the libstdc++ implementation of std::atomic<>.
[Offload] Remove unused GenericDevice in DeviceTy::launchKernel (NFC) (#236593)
The last use of GenericDevice in DeviceTy::launchKernel was removed on
October 28, 2026 in commit 8a29b68622037494f3e678cd2832b511fe2f3a9b.
Assisted-by: Antigravity
[PGO] Use the iterator-only IRBuilder constructor
Dense wave instrumentation uses a deprecated IRBuilder constructor that
takes both a block and an insertion iterator. This fails builds that
treat deprecation warnings as errors.
Use the iterator-only constructor. It obtains the same block from the
iterator and preserves the insertion point.
[SelectionDAG] Pack promoted vector bitcasts (#235918)
# Symptom
Promoting an integer vector widens each element, leaving padding between
the original values. `PromoteIntOp_BITCAST` falls back to
`CreateStackStoreLoad` for these inputs, even when the result can be
assembled directly in registers.
For example:
```llvm
define <2 x i32> @pack(<16 x i4> %x) {
%r = bitcast <16 x i4> %x to <2 x i32>
ret <2 x i32> %r
}
```
# Fix
[7 lines not shown]
[mlir][Presburger] Add IntegerRelation::gist (context-based simplification)
Add IntegerRelation::gist(context), a port of ISL's isl_basic_map_gist
(uset_gist / uset_gist_full). It replaces the relation S by a relation
S' such that S' ∩ context == S ∩ context, with as few constraints as
possible: constraints of S that are implied by the context together with
the remaining constraints of S are removed, and the remaining constraints
may be rewritten using the equalities of the context. Outside of the
context S' may differ from S, so the result is only meaningful together
with the context. The result has at most as many constraints as S
(except that the constraints defining a division that is still used may
be restored, see below), never more local variables, and never involves
variables that S does not involve; in particular, locals of the context
are never introduced.
Trivial cases: an obviously empty context yields the universe; if the
intersection with the context is found to be empty later (inconsistent
combined equalities, a context that turns out to be rationally empty, or
an empty combined Simplex), the result is empty; an obviously empty S
[43 lines not shown]
[mlir][Presburger] Add PresburgerRelation::gist
Add PresburgerRelation::gist (and PresburgerSet forwarding overloads)
for a convex context and for a context that is itself a union of
disjuncts, following ISL's isl_map_gist / isl_map_gist_basic_map.
For a convex context, each disjunct is simplified separately with
IntegerRelation::gist; disjuncts that become empty are dropped and if any
disjunct becomes the universe, the universe is returned.
For a non-convex context, the trivial cases (universe set or context,
empty context, equal set and context, context being a subset of the
set) are handled first. Otherwise, the context is approximated by a
convex set. Instead of ISL's unshifted simple hull of the context and the
set, we use the universe together with all constraints of the disjuncts
of the context and of the set that are valid for every disjunct of the
context (checked with Simplex); including the constraints of the set
allows removing constraints that are implied by every disjunct of the
context even if they are not implied by the simple hull.
[mlir][Presburger] Add IntegerRelation::reduceUsingEqualities helper
Add a protected helper that rewrites all constraints of a relation using
the equalities of another relation in the same space, without changing
the intersection of the two. This is a port of ISL's
isl_basic_set_reduce_using_equalities together with
project_onto_involved: the equalities are first projected onto the
variables that occur in the relation being reduced, so that no new
variables (in particular, no locals of the other relation) are
introduced, and each remaining equality is then used to eliminate its
last variable with a non-zero coefficient from every constraint, so that
later variables (locals) are eliminated in favor of earlier ones.
This is a building block for context-based simplification (gist) and is
exercised by the gist tests added in a follow-up commit, since there is
no existing pattern for unit-testing protected members of
IntegerRelation.
[mlir][Presburger] NFC: factor out removeRedundantConstraintsFromSimplex
Split the compaction loops of IntegerRelation::removeRedundantConstraints
into a new protected helper, removeRedundantConstraintsFromSimplex, which
removes the constraints of the relation that are marked redundant in a
given Simplex, where the relation's constraints start at a given offset
in the Simplex (inequalities first, then each equality as a pair).
This allows reusing the same compaction logic when the relation's
constraints have been added to a Simplex that already contains other
constraints, e.g., for context-based simplification (gist), where the
context's constraints come first and are never marked redundant.
[mlir][Presburger] Add PresburgerRelation::gist
Add PresburgerRelation::gist (and PresburgerSet forwarding overloads)
for a convex context and for a context that is itself a union of
disjuncts, following ISL's isl_map_gist / isl_map_gist_basic_map.
For a convex context, each disjunct is simplified separately with
IntegerRelation::gist; disjuncts that become empty are dropped and if any
disjunct becomes the universe, the universe is returned.
For a non-convex context, the trivial cases (universe set or context,
empty context, equal set and context, context being a subset of the
set) are handled first. Otherwise, the context is approximated by a
convex set. Instead of ISL's unshifted simple hull of the context and the
set, we use the universe together with all constraints of the disjuncts
of the context and of the set that are valid for every disjunct of the
context (checked with Simplex); including the constraints of the set
allows removing constraints that are implied by every disjunct of the
context even if they are not implied by the simple hull.
[mlir][Presburger] Add IntegerRelation::gist (context-based simplification)
Add IntegerRelation::gist(context), a port of ISL's isl_basic_map_gist
(uset_gist / uset_gist_full). It replaces the relation S by a relation
S' such that S' ∩ context == S ∩ context, with as few constraints as
possible: constraints of S that are implied by the context together with
the remaining constraints of S are removed, and the remaining constraints
may be rewritten using the equalities of the context. Outside of the
context S' may differ from S, so the result is only meaningful together
with the context. The result has at most as many constraints as S
(except that the constraints defining a division that is still used may
be restored, see below), never more local variables, and never involves
variables that S does not involve; in particular, locals of the context
are never introduced.
Trivial cases: an obviously empty context yields the universe; if the
intersection with the context is found to be empty later (inconsistent
combined equalities, a context that turns out to be rationally empty, or
an empty combined Simplex), the result is empty; an obviously empty S
[43 lines not shown]
[PGO] Collect GPU wave counts alongside lane counters
Lane counts measure how many active lanes reach an instrumentation point.
They do not tell us how many times a wave executes it. For example, 32 lane
visits can come from two visits with 16 active lanes or 32 visits with one
active lane. Uniformity counters cannot distinguish these cases either:
neither has a fully active 32-lane wave.
Add a wave counter for each GPU instrumentation counter. The runtime adds
one per active wave visit, independently of the lane increment's step.
Append wave counters to the existing counter allocation and expose them
separately when reading a profile, so they do not enter lane-flow
reconstruction.
For ordinary AMDGPU IR instrumentation, also measure blocks without a
sparse counter. Preserve the sparse block and select indices, then append
zero-step counters for the remaining blocks that permit instrumentation.
Record this dense layout in the profile version flags so profile use can
reconstruct it without matching a command-line setting.
[5 lines not shown]
