[AArch64] Fix non-strict scalar bf16/fp128 conversions (#234868)
Fix crashes when lowering non-strict scalar conversions between bf16
and fp128. Both directions now go through fp32. The fp128-to-bf16 path
uses the existing round-to-odd helper before the final bf16 conversion.
Add codegen tests for both directions, covering scalar returns and
loads/stores with and without bf16 support.
Fixes #172400.
[libc][regex] Add '+' and '?' ERE quantifiers (#236364)
In POSIX Extended Regular Expressions (ERE), the '+' quantifier matches
one or more occurrences of the preceding atom and '?' matches zero or
one occurrence. Quantifiers with no preceding repeatable atom, following
an anchor, or chained directly after another quantifier without grouping
must be rejected with REG_BADRPT.
Lower 'r+' to Concat(r, Star(r)) and 'r?' to Alt(EmptyStr, r) in
parse_quantified_atom. Because both references to 'r' in Concat(r,
Star(r)) share the same interned ExprId handle, '+' adds at most two
ExprPool nodes and '?' adds one node regardless of subexpression
complexity, without introducing new AST node kinds.
External compliance suites: Henry Spencer suite passes 218/456 (+50)
boolean/error cases and 73/456 (+8) full-position cases; AT&T testregex
passes 333/400 (+81) ERE boolean/error cases and 335/473 (+81) overall
boolean/error cases with 0 regressions.
Assisted-by: Automated tooling, human reviewed.
[Inline] Make the landing pad clause order deterministic (#236160)
When inlining through an invoke, HandleInlinedLandingPad appends the
outer landing pad's clauses to each inlined landing pad. It visited them
through a SmallPtrSet, which iterates in pointer order once it holds
more than 16 entries. So the use-list order of the clause constants
(e.g. a C++ typeinfo global) could change from run to run.
Use a SmallSetVector to visit them in block order, and drop the unused
set parameter of forwardResume.
Assisted by Opus 5.5
[Offload][OMPT] Add OMPT profiler implementation
Add the OMPT-specific profiler types and plugin tracing implementation without compiling or activating them yet. The existing callback path remains authoritative until the tracing interface and orchestration dependencies are available.
Assisted-by: Claude Code
[Offload][OMPT] Drop device lifecycle handling from OmptProfilerTy
Since #221726, libomptarget emits the device_initialize,
device_finalize and device_load callbacks itself, and GenericProfilerTy
no longer has the hooks the plugins used to report these events. The
overrides were left without callers and still used
GenericPluginTy::getUserId(), which #221726 removed, so the file would
not compile once it is built. Remove them together with the per-profiler
callback copies and the initialization flag only they used.
Assisted-by: Claude Code
[Offload][OMPT] Replace generic profiler data hooks with OMPT API
GenericProfilerTy no longer exposes getProfilerSpecificData() or
getTraceRecordManager(), so OmptProfilerTy provides the OMPT-specific
entry points directly: trackAsyncRecord() creates the event info that
carries a trace record into the plugins, and getOmptProfiler() gives
libomptarget typed access to the process-wide profiler. Keeping the
OMPT types out of the generic interface lets the plugins stay agnostic
of the profiling backend.
The buffer manager member moves back out until the tracing buffer
infrastructure exists later in the series.
Assisted-by: Claude Code
[Offload][OMPT] Call device tracing entry points directly
The plugin-side tracing code reached the libomptarget-side entry points by
dlopen'ing "libomptarget.so" and dlsym'ing each libomptarget_ompt_* symbol
on first use. That indirection dates back to when libomptarget and the
plugins were separate shared objects. They are not: PluginOmpt is linked
into omptarget itself, so the library was dlopen'ing itself to look up its
own symbols. The callback path was already converted to direct calls; the
tracing path should never have reintroduced the pattern.
Declare the entry points in OmptCommonDefs.h and call them directly. This
removes ParentLibrary, ensureFuncPtrLoaded(), the generated function
pointers, and the libomptarget_ompt_*_t typedefs (two of which were
duplicates, and three of which were never used).
Calling directly also means the symbols no longer have to be exported, so
no version script entry is needed for them.
The per-entry-point mutexes are left untouched here: they currently wrap
[4 lines not shown]
[Offload][AMDGPU] Wire HSA profiling into GenericProfiler abstraction (#226044)
Add device profiling infrastructure to the AMDGPU plugin so that the
GenericProfiler can receive nanosecond-accurate kernel execution and
data transfer timestamps from the HSA runtime.
Key changes:
- Add ProfilingInfoTy struct to transport HSA profiling data
- Add timeKernelInNsAsync/timeDataTransferInNsAsync callbacks that
extract dispatch/copy times from HSA signals and call
handleKernelCompletion/handleDataTransfer on the profiler
- Add getOrNullProfilerSpecificData helper to extract ProfilerData
from AsyncInfoWrapperTy
- Add getDeviceTimeStamp() override using hsa_system_get_info
- Add getSystemTimestampInNs() for HSA system timestamp queries
- Add schedProfilerKernelTiming/schedProfilerDataTransferTiming to
StreamSlotTy for scheduling profiler callbacks on stream slots
- Thread ProfilerSpecificData through pushKernelLaunch,
pushMemoryCopyH2DAsync, pushMemoryCopyD2HAsync, pushMemoryCopyD2DAsync
[11 lines not shown]
[AMDGPU] Enable SlotIndexes compaction by default
Add TargetRegisterInfo::shouldCompactSlotIndexes() so targets can opt
in to dropping dead SlotIndexes entries before greedy allocation, one
target at a time. An explicit -greedy-compact-slot-indexes still
overrides the target. Enable it for AMDGPU, where the leftover entries
skew live range priorities enough to cause heavy spilling.
[mlir][tensor] Move ExtractFromTensorGenerate pattern to ExtractOp (NFC) (#223201)
This patch moves `ExtractFromTensorGenerate` from
`GenerateOp::getCanonicalizationPatterns` to
`ExtractOp::getCanonicalizationPatterns`, resolving an explicit TODO in
`TensorOps.cpp`.
Because `ExtractFromTensorGenerate` matches and rewrites
`tensor::ExtractOp`, registering it under `ExtractOp` adheres to MLIR's
convention of grouping canonicalization patterns by their root
operation.
Assisted-by: Gemini
[clang][bytecode] Reject methods with invalid parent decls (#236338)
E.g. the later call to `isEmpty()` via
`isReadByLValueToRValueConversion()` causes infinite recursion.
[mlir][IR] Add return value to `replaceUsesWithIf` indicating that a substitution happened (#235748)
This makes it easier/cleaner for clients to detect and propagate
`change.
For example patterns are expected to return `success` when a change
happens and thus must detect them. Returning a value avoid capturing a
flag and modifying by side-effect in the callback.
[X86] Don't discard over-16 stack alignment for 32-bit interrupt handlers (#225366)
calculateMaxStackAlign assigned Align(16) instead of taking the maximum,
so an alloca with align 64 in a 32-bit handler got andl $-16 and landed
only 16-byte aligned, even though PEI had laid the frame out for 64.
With AVX512 the compiler then emits vmovaps to that object, which faults.
[AMDGPU] Preserve noclobber after acquire operations from a different addrspace
Follow-up to #219991 to address reviewer comments:
AMDGPUAnnotateUniformValues should annotate uniform global loads as
`!amdgpu.noclobber` if the memory they are reading is never clobbered (since
kernel launch) before the load is executed.
PR #219991 strengthened the logic to consider acquire operations before the
load as clobbering (because they cause stores from other threads to be observed
by the load without a data race).
This PR weakens this criterion to ignore acquire operations that don't affect
the global address space (because they don't affect what the load in question
may or may not see). This addresses the performance-relevant case of s_barriers
with LDS-only fences.
The logic is still overly pessimistic for one-as load acquires, they could also
be ignored when looking for clobbering operations.
Disclaimer: Claude gave review comments and suggestions that I addressed during
[5 lines not shown]
refactor(codegen): Move shift helpers out of line
Define the shared shift helpers in CodeGenCommonISel.cpp under llvm::detail. Use function_ref callbacks to preserve lazy KnownBits computation, and retain LLVM_ABI for callers in other CodeGen component libraries.
feat(gisel): Mirror trunc/xor/select demanded bits
Extend scalar demanded-bits simplification to G_TRUNC, G_XOR, and
G_SELECT, following SelectionDAG traversal, known-bit propagation,
shared-use handling, and final folds. Honor GISel legality and target
profitability when rewriting or narrowing operations.
Share AArch64's logical-immediate search through a narrow XOR hook and
adapt XOR/shift profitability queries for both selectors. Preserve
SelectionDAG's constant-shrinking interface and node construction.
Share shift-amount validation in CodeGenCommonISel using templated
callbacks for selector-specific lane access and KnownBits queries.
Preserve recursive constant bypasses and use-specific rewrites of shared
producers. Recognize constants at the analysis depth limit, matching
SelectionDAG, and retain KnownBits-based bypasses as a fallback.
Cover AArch64 and AMDGPU with focused MIR and codegen tests, including
shared mask chains, SELECT arms, and depth-limited XOR bypasses.
[SPARC] Give the leaf epilogue's stack adjustment a debug location (#235247)
The stack adjustment in a leaf function's epilogue was built with an
empty DebugLoc, so it inherits the line of whatever precedes it and the
row for the return's line starts after it. A breakpoint on that line
then stops with the frame already released, and %sp-based variable
locations read the wrong slots.
Use the return's location, as the non-leaf restore already does.
The delay slot filler currently hides this by moving the adjustment
into the delay slot of the retl; it shows once that slot is not filled,
see #235235.
Assisted-by: Claude Code (Claude Opus 5.5)
