[mlir][OpenACC] Lower single-block thread-only array reductions (#212369)
Example:
```fortran
!$acc parallel loop vector reduction(+:b)
do i = 1, n
b(i) = i
end do
```
In this code the array reduction is thread(vector)-only with no
gang/block launch dim, so a within-block `all_reduce` is a complete
reduction. ACCCGToGPU still rejected it as NYI because the accumulate
had no block context.
Fix: keep the NYI only when the region actually launches block
dimensions; allow the existing per-element `all_reduce` path for
single-block thread-only array accumulates.
[NVPTX] Emit some more PTX directives through target streamer (#212070)
Move `.branchtargets`, `.reg`, `.local`, `.alias`, `.pragma`, and the
empty `.debug_macinfo` section out of hand-built strings in
`NVPTXAsmPrinter` and into `NVPTXTargetStreamer` hooks.
Opportunistically pulled at a few other threads that could use cleanup
as well.
Three behavior changes:
1. Jump table labels now come from `MachineFunction::getJTISymbol`, so
`$L_brx_<idx>` becomes `JTI<func>_<idx>` and is no longer shared across
functions
2. `.alias` names its aliasee by mangled symbol rather than raw IR name,
fixing `.alias a, ;` for an unnamed aliasee.
3. The rest of the test diff is a duplicated blank line that functions
with no register declarations no longer emit.
Co-Authored by Opus 5
[MacroFusion] Add RunPreRA/RunPostRA to select the scheduling stage (#212182)
MacroFusion runs in both the pre-RA and post-RA machine schedulers.
Add two `bit` fields to the `Fusion` base class, `RunPreRA` and
`RunPostRA`, both defaulting to `true`, so a fusion can opt out of a
scheduling stage. At least one of them must be `true`.
When a stage is disabled, `MacroFusionPredicatorEmitter` emits an early
`return false` guard for that stage (detected via the `NoVRegs` machine
function property) and only keeps the statistic counter for the stage
the fusion actually runs in. A pre-RA-only fusion is generated as:
```cpp
STATISTIC(NumNAMEPreRA, "Times NAME Triggered (pre-ra)");
bool isNAME(...) {
...
if (SecondMI.getMF()->getProperties().hasNoVRegs())
return false;
/* Predicates */
[9 lines not shown]
[llvm-objcopy] Add AMDGPU case to binary-output-target.test
Add test coverage for converting binary input to elf64-amdgpu format,
verifying the output has the correct format string, arch (amdgpu),
and machine type (EM_AMDGPU 0xE0). Follows the same pattern as all
other architectures in this test file.
[llvm-objcopy] Fix AMDGPU arch checks in tests
ELFObjectFile.h getArch() for EM_AMDGPU returns Triple::UnknownArch
when e_flags & EF_AMDGPU_MACH is 0 (no GPU target specified). Only
when a MACH flag in the AMDGCN range is present does it return
Triple::amdgpu.
- cross-arch-headers.test: restore EF_AMDGPU_MACH_AMDGCN_GFX900 flag
on the input ELF so that after format conversion the output correctly
reports Arch: amdgpu.
- binary-output-target.test: expect Arch: unknown since converting
from raw binary input (-I binary) produces an ELF with e_flags=0
(no MACH flags), giving UnknownArch. This is correct behavior.
[SLP]Fix dep accounting for expanded binops in non-scheduled entries
The extra dependency counted for the expanded form was never released
when the expanding entry had no schedule bundle and the block had no
copyable elements, leaving the operand's bundle unscheduled forever.
Take the pseudo-bundle path also when the instruction is an expanded
binop in some tree entry.
Fixes #212220
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/212380
[llvm-objcopy] Fix AMDGPU arch checks in tests
ELFObjectFile.h getArch() for EM_AMDGPU returns Triple::UnknownArch
when e_flags & EF_AMDGPU_MACH is 0 (no GPU target specified). Only
when a MACH flag in the AMDGCN range is present does it return
Triple::amdgpu.
- cross-arch-headers.test: restore EF_AMDGPU_MACH_AMDGCN_GFX900 flag
on the input ELF so that after format conversion the output correctly
reports Arch: amdgpu.
- binary-output-target.test: expect Arch: unknown since converting
from raw binary input (-I binary) produces an ELF with e_flags=0
(no MACH flags), giving UnknownArch. This is correct behavior.
[clang][Index][USR] Generate USRs for class-type non-type template arguments (#212356)
A class-type non-type template parameter is represented in the AST by a
TemplateParamObjectDecl, which has an empty DeclarationName.
USRGenerator had no visitor for it, so it fell through to
VisitNamedDecl, where EmitDeclName fails on the empty name and sets
IgnoreResults. That discarded the USR of the enclosing declaration.
Add a visitor that encodes the parameter object's type and value, so
specializations on distinct values get distinct USRs and equal values
agree.
Fixes #212351
[OpenMP] Propagate PRESENT to pointee entries in mapper codegen
Extend map-type-modifier propagation in emitUserDefinedMapper to the PRESENT
modifier, but only for entries that have an attach ptr (the pointee data, whose
storage differs from the struct's own). A present modifier on the outer clause
must require that pointee to be present on the device.
This is gated on a new PropagatePresentToPointee argument, which Clang sets from
CGM.getLangOpts().OpenMP >= 60. Before 6.0 the present modifier is treated as
not applying to the pointee: the spec committee confirmed the divergence
between the present motion modifier (to/from) and the present map-type modifier
(map) was unintentional, to be fixed as an OpenMP 6.0 erratum. Only propagation
is gated; present written directly in a mapper's own clause applies at all
versions.
A TODO notes PRESENT should also propagate to the struct's own members, which
is blocked while pointer members use PTR_AND_OBJ.
Update the present-check tests to their final 6.0-gated behavior.
[2 lines not shown]
[OpenMP][Clang] Enable ATTACH-style maps for mappers.
Track per-entry attach-ptr info (HasAttachPtr) through mapper codegen so that
emitUserDefinedMapper does not add a new outer MEMBER_OF to pointee/combined
entries (which occupy different storage than the struct) or to ATTACH entries.
Clang and the MLIR translator populate the per-entry array in parallel with the
other MapInfosTy arrays.
Address review:
- Rename MapSkipMemberOfArrayTy to MapHasAttachPtrArrayTy to match the
HasAttachPtr field it backs.
- Restructure the emitUserDefinedMapper comment into a bulleted (*)/(**)/(***)
list keyed to the example entries.
- Reword the Clang comments: HasAttachPtr marks pointee entries that have a
base attach-ptr; a combined entry has a base attach-ptr if its constituents
do; cross-reference emitUserDefinedMapper for the MEMBER_OF rationale.
- Update the moved present-check tests to their now-correct behavior (the
attach-style maps make the inbounds present checks pass and remove the
"explicit extension" errors).
[2 lines not shown]
[OpenMP] Propagate ALWAYS/DELETE/CLOSE map-type modifiers to mapper entries
When a map clause uses a user-defined mapper, the map-type-modifying bits
(ALWAYS, DELETE, CLOSE) on the outer clause must apply to each map the mapper
inserts (OpenMP 6.0:281:34). Propagate them in emitUserDefinedMapper by OR-ing
the imported modifier bits into each pushed component, except ATTACH entries
(ATTACH|ALWAYS is reserved for attach(always) and the other bits are
meaningless for ATTACH).
PRESENT is intentionally not propagated here yet (a TODO notes it is handled in
a follow-on, since it requires distinguishing pointee entries from the struct's
own storage).
Update the offload always-propagation tests to their now-correct behavior:
ALWAYS forces a member transfer that the ref count would otherwise suppress, so
mapper_map_always_from.c reads s.y back as 111, and the enter-data variants
(C and Fortran) see 111 in the device copy (all were 0 before this change).
Since emitUserDefinedMapper is shared between clang and flang, the Fortran test
covers the flang path as well.
[2 lines not shown]
[OpenMP][test] Add mapper ALWAYS-propagation tests with no target construct
Companions to mapper_map_always_from.c that exercise ALWAYS propagation into a
user-defined mapper's entries without entering any target region. The device
copy is inspected directly via omp_get_mapped_ptr() and omp_target_memcpy(), so
the check covers only the data-motion performed by `target enter data`.
The Fortran variant is the first Fortran offload test that runs on the host
(x86_64) target: it needs no device kernel, so it does not depend on a device
runtime being available.
Both tests currently CHECK the pre-propagation behavior (the transfer is
suppressed for the already-mapped entry) and record the expected value in a
FIXME.
[Instrumentor] Reintroduce runtime inlining (#208072)
The ability to link in and eagerly inline a specified runtime was
present in older dev versions of the instrumentor. This patch ports that
functionality to the current instrumentor pass and adds a test.
Co-authored-by: Johannes Doerfert <johannes at jdoerfert.de>
Co-authored-by: Kevin Sala <salapenades1 at llnl.gov>
Co-authored-by: Ivan Radanov Ivanov <ivanov.i.aa at m.titech.ac.jp>
[llvm-objcopy] Add AMDGPU case to binary-output-target.test
Add test coverage for converting binary input to elf64-amdgpu format,
verifying the output has the correct format string, arch (amdgpu),
and machine type (EM_AMDGPU 0xE0). Follows the same pattern as all
other architectures in this test file.
[AMDGPU] Allow scalar operands in V_PK_LSHL_ADD_U64 (#212334)
Patch creates dummy source modifiers for this instruction to use
with OPSEL. Even though these do not support modifiers we can reuse
the same mechanics as with packed fp64 and other u64 to allow scalar
operands.
[flang-rt][cuda] Keep CUFDeviceIsActive working under runtime/driver skew (#212320)
Resolve cuDevicePrimaryCtxGetState via the installed driver version, and
prefer running scope-exit cleanup when the probe cannot prove teardown,
so local device allocations are not leaked when the unversioned lookup
fails.
[clang] Fix record alignment lost via external layout on Arm64 (#212362)
An external layout source (such as LLDB reading DWARF) supplies a
record's final alignment directly, since `alignas` / `__declspec(align)`
attributes are not recoverable from debug info. `finalizeLayout`
captured `NonRequiredAlignment` before applying the external alignment,
so an externally laid out record published a stale, too-small value.
On Arm64 this field is used to fold a base's alignment into the derived
record, so a derived class picked up the stale value instead of the
base's actual alignment. This regressed
`lldb/test/API/lang/cpp/alignas_base_class` when #210461 was merged,
where `alignof(Derived)` evaluated to 1 instead of 8. Other targets were
unaffected as they do not read this field.
Treat all of an externally supplied alignment as non-required, as there
is no way to tell how much of it was imposed by an alignment attribute.
[MLIR] Allow recovering from semi-affine sets in FlatLinearValueConstraints
The IntegerSet constructor of FlatLinearValueConstraints asserts when
flattening fails, which happens for semi-affine integer sets. Since the
failure is signalled from inside a constructor, callers have no way to
detect the unsupported case ahead of time and no way to work around it;
they simply crash.
Make that constructor protected and expose the construction through
`FlatLinearValueConstraints::create` (and the corresponding
`FlatAffineValueConstraints::create`), which returns a `FailureOr` and
so lets callers handle a semi-affine set gracefully.
Existing in-tree users are updated: `addAffineIfOpDomain` keeps asserting
as before, and `simplifyIntegerSet` now returns the set unchanged instead
of asserting, since a semi-affine set simply cannot be simplified this way.
[VPlan] Add vputils::getEarlyExits helper (NFC) (#212348)
Both handleUncountableEarlyExits and the countable early-exit disconnect
in handleEarlyExits walk all exit blocks and skip predecessors coming
from the middle block to find the early exiting blocks.
Factor that walk out into vputils::getEarlyExits, returning the (early
exiting block, exit block) pairs.
[CI] Add documentation for self-hosted runners (#211622)
This patch adds (work in progress) documentation for the various
self-hosted runner groups we have access to in the LLVM monorepo.
[MLIR] Allow recovering from semi-affine sets in FlatLinearValueConst… (#212346)
…raints
The IntegerSet constructor of FlatLinearValueConstraints asserts when
flattening fails, which happens for semi-affine integer sets. Since the
failure is signalled from inside a constructor, callers have no way to
detect the unsupported case ahead of time and no way to work around it;
they simply crash.
Add an optional `bool *error` parameter to the constructor. When it is
non-null, hitting the unimplemented case sets `*error` to true and
returns instead of asserting, letting callers bail out gracefully.
`*error` is set to false on success. When the pointer is null the
previous assert behavior is retained, so existing callers are
unaffected.
AMDGPU: Handle more TargetParser queries in tablegen
Previously we had various enum switches. Start generated tables
indexed by enums. Avoid some special cases by defining the dummy
"generic" and "generic-hsa" targets as real processors.
Co-authored-by: Claude (Claude-Opus-4.8)
AMDGPU: Generate TargetParser table from TableGen
Migrate .def file to tablegen. The initial TargetParser
AMDGPU backend only handled R600. Extend to modern targets.
Stub out some tablegen definitions which will be used in future
changes.
Co-authored-by: Claude (Claude-Opus-4.8)
[SimplifyCFG] Do not thread branches into uncontrolled convergent regions
SimplifyCFG's foldCondBranchOnValueKnownInPredecessor can thread an edge past
a block that acts as a reconvergence point. If the threaded destination reaches
an uncontrolled convergent operation before returning to the threaded-through
block, the transform can change which dynamic instance of the convergent
operation is executed.
Add a conservative destination scan for this fold and skip the threading
candidate when it can reach an uncontrolled convergent call before returning
to the original block. Controlled convergent operations using convergence
control tokens are left alone.
Fixes ROCM-26496.
Apply metadirective loop checks after variant selection
Determine reachable replacements before preparing the associated loop. Apply
loop-only limitations only when a loop replacement can actually be selected.
Keep candidate-wide loop preparation for runtime cascades, where multiple
replacement arms remain reachable.
