[mlir][vector] Update `CastAwayTransfer{Read|Write}LeadingOneDim` (#219499)
Updates `CastAwayTransfer{Read|Write}LeadingOneDim` to use
`vector.shape_cast`, rather than `vector.extract`, as the canonical form
for stripping unit dimensions.
This change was originally implemented by @krzysz00 in #196206, but was
subsequently reverted in #199546. This PR intentionally restores only a
subset of #196206, making it easier to identify and triage any potential
regressions.
Co-authored-by: Krzysztof Drewniak <Krzysztof.Drewniak at amd.com>
[RISCV][GlobalISel] Fold large constant offsets in selectAddrRegImm (#219161)
Fold ADDI adjustment (AddiPair) for offsets in [-4096, 4094] and split
larger constants into materialized Hi + Lo12 offset, matching SDAG. Add
isWorthFoldingAdd to guard the split and extract the shared ADDI
renderer into renderAddiPair.
Assisted-by: Claude
[clang][AST] Fix infinite recursion when printing fully qualified template parameters (#219044)
This fixes an infinite recursion crash that was introduced in #206041.
When printing a `DeclRefExpr` using
`PrintingPolicy::FullyQualifiedName`, we were previously trying to print
the fully qualified name of all decls. However, when the decl is a
template parameter, its `DeclContext` is the template specialization
itself. If a template specialization's arguments depend on that same
template parameter (e.g., `template<int Count> struct
View<int[Count]>`), attempting to print the qualified name forces Clang
to recursively evaluate the enclosing context. This led to unbounded
recursion (`View<int[Count]>::Count` ->
`View<int[View<int[Count]>::Count]>::Count` and so on). Since template
parameters are inherently scoped to their template declarations and do
not require a fully qualified name, this patch resolves the issue by
skipping `printQualifiedName` if the decl is a template parameter
(`!VD->isTemplateParameter()`).
Fixes #218076.
[MemDep] Remove cached non-local defs referencing deleted instructions (#219929)
MemoryDependenceResults::removeInstruction() removed entries from
NonLocalDefsCache when the removed non-pointer instruction was itself a
cache key, but did not remove entries where that instruction was the cached
dependency result.
This could leave a stale instruction reachable through NonLocalDefsCache
after GVN erased it. A later non-local load query could then consume the
dangling dependency and crash in GVN::analyzeLoadAvailability().
Use ReverseNonLocalDefsCache to remove forward cache entries whose
dependency result references the removed instruction, keeping both maps
consistent.
Related removal paths now use the existing reverse-map helper, with
consistency assertions to catch future map divergence.
Fixes #219885.
[6 lines not shown]
[GlobalISel] Migrate various generic wip_match_opcode combines to MIR-pattern. (#220213)
This patch converts a batch of GlobalISel combine rules from
hand-written C++ matchers to declarative MIR patterns preserving the
behavior. It also along with adds two functional changes (a
rule-ordering fix and an out-of-bounds bug fix) are described below.
- `select_same_val` → `select_same_val_trivial` +
`select_same_val_equiv` group
- `select_constant_cmp` → `_false` / `_true` / `_general` group
- `simplify_add_to_sub`, `add_p2i_to_ptradd` → `GICombinePatFrag`s
- `commute_shift` → `commute_shift_frags` (C++ residue reduced to
`isDesirableToCommuteWithShift`)
- `combine_i2p_to_p2i`, `ptr_add_zero`, `sext_trunc_sext_load` →
patterns.
- funnel-shift / rotate / `ashr_lshr` / `constant_fold_fma` /
`constant_fold_cast_op` / `combine_minmax_nan` → pattern fragments
[7 lines not shown]
[AMDGPU] Route no-modifier reg-or-inline AsmParser operands through HwMode predicate
Convert the reg-or-inline operands with no modifiers (MFMA VGPR/AGPR
sources, VCSrc, v_pk_mov_b32, VOP scalar f64) from the fixed-class
isRegOrInlineNoMods to the HwMode-aware isRegOrInlineNoModsByHwMode, so an
odd-aligned tuple is rejected at the offending operand column instead of by
the validateVGPRAlign catch-all.
Co-Authored-By: Claude <noreply at anthropic.com>
[AMDGPU] Factor the AsmParser match-specificity comparison into a helper, NFC
Extract the "keep the most specific match status" comparison in
matchAndEmitInstruction into an atLeastAsSpecific() lambda ranking
Match_MnemonicFail < Match_InvalidOperand < Match_MissingFeature. No
functional change.
[AMDGPU] Make custom AsmParser matchers alignment aware
Currently custom matchers accept registers belonging to unaligned classes,
leaving the alignment check and its diagnostic to validateVGPRAlign. The problem
is that validateVGPRAlign does not account for the operand register class, which
may have a different alignment requirement on mixed-alignment targets.
The fix is to make a custom matcher resolve the operand's _AlignTarget class via
HwMode and use that resolved class to accept a register. However, doing so alone
would reject a misaligned register with a generic "invalid operand for
instruction" diagnostic, so extra infrastructure conveys the alignment diagnostic
instead: on a class miss, diagnoseRegAlign re-checks the register against the same
class with alignment relaxed (getUnalignedEquivalentRC, new in AMDGPUBaseInfo);
if it fits, the only problem is alignment, so it records
OperandMatchError::VGPRAlignMismatch on the operand. matchAndEmitInstruction
selects that reason and prints the alignment error, now at the offending operand
column instead of column 1.
This commit converts the input-mods and DP-ALU DPP matchers. The no-modifier
[4 lines not shown]
[AMDGPU] Add AsmParser tests for misaligned VGPR tuples in custom-predicate operands
MFMA VGPR sources, DP-ALU DPP, v_pk_mov_b32 and VOPD3 f64 take VGPR tuple
operands validated by custom AsmParser predicates. Odd-aligned tuples are
currently only caught late by validateVGPRAlign, so the error points at the
whole instruction (column 1). A follow-up moves the diagnostic to the
offending operand.
[offload][omp] Load and resolve device binaries through liboffload
Migrate DeviceTy::loadBinary and global/kernel symbol resolution off
GenericPluginTy::load_binary/get_global/get_function onto liboffload's
Program/Symbol API, encapsulated in a new ProgramTy abstraction that wraps
an ol_program_handle_t. Kernel symbol resolution still needs the plugin's
opaque GenericKernelTy* handle for the legacy launch path, obtained via a
temporary __ol_tgt_GetKernelFromSymbol helper rather than new public
liboffload API surface. Removes the now-dead __tgt_device_binary type and
the corresponding GenericPluginTy methods and exports entries.
Set correct stack reserve when built by Clang on Windows (#221263)
Currently it's handled correctly for MSVC's LINK and MinGW (i.e.
distributed builds of LLVM work correctly), but not lld-link, causing
such builds to crash in certain scenarios.
[CIR] Use the modern enum case classes (#220935)
The `I32EnumAttrCase` family carries an `Attr` half, and an
`IntegerAttr`
predicate with it, that a CIR enum has no use for now that the enums
derive
from `EnumInfo`. Upstream says of those forms that they "are not needed
when
using the newer `EnumCase` form".
Rename all of them to `I32EnumCase`, `I32BitEnumCaseNone`,
`I32BitEnumCaseBit` and `BitEnumCaseGroup`. The group class drops its
width
prefix because the modern spelling takes the width from its cases.
[LLVM][SelectionDAG] Fix incorrect splitting of VECTOR_COMPRESS when passthrough operand is not undef. (#215772)
When splitting VECTOR_COMPRESS the original mask is used for the final
step when inactive elements are set to their passthrough value. This is
incorrect because only the inactive lanes of the result should be set to
their passthrough value.
[lldb] Small changes to StopPointSiteList::FindInRange (#221975)
1. Make the return value indicate an empty list or not. (I would return
a list, but you cannot copy this list and I don't want to make a larger
change right now)
2. Remove a redundant write at one of the call sites.
AMDGPU: Remove llvm.amdgcn.addrspacecast.nonnull (#220928)
The intrinsic is fully replaced by the nonnull flag on addrspacecast,
so remove it.
Old bitcode/IR is autoupgraded, though this is very conservative. This
intrinsic was only inserted by the backend, and hopefully nobody was
directly emitting it.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply at anthropic.com>
AMDGPU: Set the addrspacecast nonnull flag instead of the intrinsic (#220927)
AMDGPUCodeGenPrepare proved the source of certain flat<->local/private
addrspacecasts non-null and rewrote them to
llvm.amdgcn.addrspacecast.nonnull. Now that the flag is honored in
codegen, set it in place on the existing instruction instead.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply at anthropic.com>
[libc] Simplify RPC lock handling to be scalar on the GPU (#221805)
Summary:
This interface was originally written in a complicated way as a
pessimistic bet against NVIDIA's independent thread scheduling. The idea
was that it would work regardless of which set actually executed the
function. The problem is that this forces the operation to be an
expensive vector operation.
This should be safe, we already have other safeguards against ITS in the
main loop. When this function is called, we know that the index and lane
mask are convergent in the set. ITS says the true threads executing this
could be greater than returned. But in this case the 'locked' value
would not be shuffled in, it would not see that it got the lock, and
loop again in a different mask set.
The complicated thing about ITS is mostly that any lanes that share a PC
will report convergent, but that should not bite us here as we already
guard against that before calling this.
lib/msun amd64: use AVX FMA instructions for fma(3) and fmaf(3) when available
Utilize the ARCHLEVEL framework from libc/amd64/string to provide the
way for runtime selection of the implementation, if wanted.
Reviewed by: fuz, kfv
Discussed with: kargl
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D59462
lib/msun: move the calculation of LIBC_SRCTOP earlier
which makes the variable available for machine/Makefile.inc usage.
Reviewed by: fuz, kfv
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D59462
