[libomp] OpenMP 6.0: Add device trait parser (#176164)
OpenMP 6.0 introduced a device trait specification language for the
environment variables OMP_AVAILABLE_DEVICES (4.3.7) and
OMP_DEFAULT_DEVICE (4.3.8).
This commit defines a grammar for that language and implements a parser
for a large part of this grammar.
[AMDGPU] Utilize Promote action for FMINIMUM/MAX f16 (#215678)
Utilize Promote action to convert f16 FMINIMUM/FMAXIMUM to use v2f16.
Custom lowering is no longer needed.
Support for Promote from scalar to vector was added in
https://github.com/llvm/llvm-project/pull/215042.
---------
Signed-off-by: John Lu <John.Lu at amd.com>
[DebugInfo] Do not stop after livedebugvars (#215169)
livedebugvars is an analysis. We can only -stop-after it inside the
LegacyPM due to how the LegacyPM schedules analyses. We cannot
stop-after it inside of the NewPM given analyses are dynamically
requested within each pass. Instead, just stop before the asm printer
given we're after the last livedebugvars analysis which happens pretty
late in the pipeline and it doesn't seem like there's a vastly better
spot.
[CIR] Require every record member to specify its kind
The member kind list was optional, and an all-data list was canonicalized
to an absent one, so "nobody computed this" and "everything is data" had
the same spelling. A producer that forgot to specify was assumed
correct.
The list is now required on both record types and the verifier demands
one kind per member. Normalization is no longer needed and is removed.
A producer whose members all hold data says so with getAllDataKinds.
CIRGen has to compute a list too, so record lowering funnels every
appended field through addField and marks inserted padding as pad.
A data member still prints without a mark, and `data` is now accepted on
input so every kind can be written out.
Assisted-by: Cursor / claude-opus-5
[CIR] Add AArch64TargetCIRGenInfo (#215424)
This adds an AArch64-specific implementation of TargetCIRGenInfo and
adds handlers for the functions that require AArch64-specific handling.
I've implemented the wouldInliningViolateFunctionCallABI function
(because that seemed easier than deciding when to report NYI), generated
an NYI error for isScalarizableAsmOperand in the one case where it needs
to do something other than forward the call to the based class, and
added MissingFeatures asserts for setTargetAttributes (because CIR
doesn't support the features it wants to add attributes for yet).
Assisted-by: Cursor / various models
[CIR][NFC] Add atomic compare-exchange runtime order test (#215722)
Add a test for `__atomic_compare_exchange_n` when both success and
failure memory orders are runtime values. Check the nested CIR switches
across every generated memory-order combination.
Partially addresses #156747.
[mlir] Remove dead declaration promoteSingleIterationLoops (#215738)
The corresponding function definition was removed on January 24, 2022
in a70aa7bb0d9a6066831b339e0a09a2c1bc74fe2b.
[LLVM][Maintainers] Volunteer for LoadStoreVectorizer (#214025)
I have been contributing to and reviewing changes to the
LoadStoreVectorizer for over a year now, figured I should formalize
this.
[Clang] Fix BitInt padding clearing on big-endian targets
This patch fixes the padding clearing logic of `_BitInt`s.
Before this patch, the clearing logic assumed little endian. But the
memory layout of BitInts differs between little and big endian:
- In LE, the occupied bits start from the lowest address and go on
contiguously up until the BitInt's declared size. The padding bits
then start from that point and go contiguously until the end of the
storage unit.
- In BE, since the byte order is reversed, the occupied bit interval
is not contiguous if the storage unit is larger than the BitInt's
size.
Therefore, the logic must tell the two cases apart and perform the
calculations accordingly.
[lldb] Add a reason for all requireNot* decorators. (#214701)
Previously, tests using the @requireNot* decorators documented by the
test was skipped using a trailing comment that never made it to the test
report.
Change the decorator to take a required `reason` parameter, as the test
should have a reason why it is not required.
[Inliner] Don't apply the cold-callsite threshold in non-callable functions (#211255)
Partially addresses #211132.
`InlineCostCallAnalyzer::updateThreshold` clamps the threshold to
`-inline-cold-callsite-threshold` (45) when BFI says the callsite is
cold. The reasoning behind the clamp, that cold code does not need to be
fast so it is not worth the code size, does not hold when the caller is
a hardware entry point rather than something callable. Register
allocation there is whole-function and occupancy is set by the worst
case over it, so a call left out of line costs the hot path too, however
cold the call itself is.
So guard the clamp with `isCallableCC(Caller->getCallingConv())`. This
is not AMDGPU-specific: it picks up `SPIR_KERNEL` and `PTX_Kernel` on
the same reasoning. An earlier revision of this PR added a
`TTI::applyColdCallSiteThreshold` hook and returned false from AMDGPU
for entry functions; that was dropped in review in favour of the calling
convention, which states the actual property directly and needs no new
[47 lines not shown]
[mlir][arith] Fold ceildivsi with MININT operands (#214637)
`CeilDivSIOp::fold` computes the ceiling by negating operands so that
the
division runs on two non-negative values. Negating `MININT` overflows,
so the
folder gives up on any `MININT` operand, even when the result is
perfectly
representable:
```mlir
// i8, MININT = -128. ceil(-128 / 7) = -18, which fits, but does not fold.
%0 = arith.constant 7 : i8
%min = arith.constant -128 : i8
%1 = arith.ceildivsi %min, %0 : i8
```
The existing TODO on the folder mentions only a `MININT` dividend, but a
`MININT` divisor is affected the same way, since that operand gets
[181 lines not shown]
[GlobalISel] emit G_BITCAST in widenScalarUnmergeValues when SrcTy is float (#204541)
widenScalarUnmergeValues emits G_LSHR and G_TRUNC to extract
sub-register pieces from the source. The shift amount and the shift
itself are built with SrcTy, so when the source is a floating-point
scalar (e.g. bf16) the extraction ends up in the float domain:
```
%1:_(bf16) = G_CONSTANT i16 8
%2:_(bf16) = G_LSHR %0:_(bf16), %1:_(bf16)
```
**Fix:** Before the shift/trunc extraction sequence, check if SrcTy is a
float type and bitcast it to the corresponding integer type.
[BOLT] Default heatmap block sizes to cache line, pages and hugepage (#215427)
The defaults were 64, 4K, 256K. 4K is the page size only on x86-64 and
on
AArch64 kernels built that way; AArch64 also runs 16K and 64K base
pages, and
256K corresponds to nothing in particular on either.
Use 64, 4K, 16K, 64K, 2M: the cache line, the three base page sizes in
use, and
the PMD hugepage above a 4K base page. Each granularity then maps onto a
real
capacity, which is what makes the working set numbers comparable to one
-- L1i
lines, iTLB and L2 TLB entries, frontend region-table entries.
Two more granularities cost two more passes over an already-built map,
no extra
decoding.
[5 lines not shown]
[BOLT] Keep the spelling of input heatmap block sizes (#215426)
The block-size parser turns "64K" into 65536 and discards the original
text,
keeping it only for error messages. The working set log then has to
either
reprint the raw value or reformat it back, and reformatting invents a
spelling
the user did not choose: "1MiB" comes back as "1M".
Store the spelling next to the value and echo it. Heatmap file names
keep using
the numeric value, matching the existing "dumping heatmap with bucket
size N"
message and the -<size> suffix that tests already expect.
Test Plan:
updated heatmap-preagg.test
[6 lines not shown]
[OpenMP][flang] Use auto-generated data for modifier verification
Introduce descriptors of OpenMP clauses and modifiers, auto-generated
from OpenMP specification data[*].
This replaces the preexisting modifier descriptors used in flang,
the code using it is adjusted to use the new definitions.
[*] This does not contain any actual auto-generation infrastructure.
The descriptors were generated by a custom script from the exported
OpenMP definitions.
Enforce single-operand form for llvm.loop.vectorize.scalable metadata (#215742)
Replace the two-operand boolean form
!{!"llvm.loop.vectorize.scalable.enable", i1 0/1} with a single-operand
enable/disable pair:
!{!"llvm.loop.vectorize.scalable.enable"}
!{!"llvm.loop.vectorize.scalable.disable"}
The Verifier rejects the two-operand form, AutoUpgrade rewrites old
bitcode, and the readers and producers in LLVM, Clang and MLIR are
updated.
RFC:
https://discourse.llvm.org/t/rfc-enforce-single-operand-format-for-all-enable-metadata-nodes/90571
[NFC][AST] Make nullability attributed-type construction const (#215756)
Make the nullability-specific `ASTContext::getAttributedType` overload
`const` and remove a now-unneeded `const_cast` from array-decay
handling.
This addresses feedback from
https://github.com/llvm/llvm-project/pull/215266#discussion_r3751285430.
Reviewers: @j-hui @Xazax-hun @egorzhdan
CodeGen: Remove TargetOptions::FloatABIType
This is now fully replaced with the "float-abi" module flag.
If the module flag is not present, the default is computed
from the triple. Consumers are updated to read the module flag.
RuntimeLibraryAnalysis now defers analysis until run() on a Module,
instead of during the pass constructor as before. This requires copying
all of the remaining relevant TargetOptions so they are available
when the module is seen.
Unfortunately, ARM still depends on TargetOptions for determining
the float-abi. -target-abi=aapcs16 still changes the default float-abi,
but an explicit module flag wins.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply at anthropic.com>
[LV] Use wide lane masks as the canonical form when tail-folding & interleaving (#209484)
Previously, when passing -enable-wide-lane-mask to request the use of wide
lane masks, tryToReplaceALMWithWideALM would attempt to replace several
ActiveLaneMasks created during unrolling with a single mask + extracts.
This PR removes tryToReplaceALMWithWideALM from VPTransforms and makes
wide lane masks the canonical form when interleaving a tail-folded loop
which uses ActiveLaneMask for control flow, as discussed on #193757.
During unrolling, the multiplier is set to UF for ActiveLaneMask and one
ExtractSubvectorForPart instruction for each part is created. Unnecessary
ExtractSubvectorForParts are removed during the generate stage if the loop
is not unrolled, otherwise this creates an extract subvector for each part.
This does not change the default behaviour for tail-folded loops, which
are still not considered for interleaving by default. The decision on
whether to use a single wide active lane mask with extracts or to split
it up into multiple smaller masks is now left to the backend.
[BOLT] Default heatmap block sizes to cache line, pages and hugepage
The defaults were 64, 4K, 256K. 4K is the page size only on x86-64 and on
AArch64 kernels built that way; AArch64 also runs 16K and 64K base pages, and
256K corresponds to nothing in particular on either.
Use 64, 4K, 16K, 64K, 2M: the cache line, the three base page sizes in use, and
the PMD hugepage above a 4K base page. Each granularity then maps onto a real
capacity, which is what makes the working set numbers comparable to one -- L1i
lines, iTLB and L2 TLB entries, frontend region-table entries.
Two more granularities cost two more passes over an already-built map, no extra
decoding.
[BOLT] Keep the spelling of each heatmap block size
The block-size parser turns "64K" into 65536 and discards the original text,
keeping it only for error messages. The working set log then has to either
reprint the raw value or reformat it back, and reformatting invents a spelling
the user did not choose: "1MiB" comes back as "1M".
Store the spelling next to the value and echo it. Heatmap file names keep using
the numeric value, matching the existing "dumping heatmap with bucket size N"
message and the -<size> suffix that tests already expect.
Test Plan:
updated heatmap-preagg.test
[AMDGPU] Account for aliasing reads in RewriteOutArguments (#209908)
Fix AMDGPURewriteOutArguments incorrectly moving an out-argument store
past a potentially aliasing read.
Use the full ModRef result when scanning intervening memory accesses, so
the rewrite is rejected for both reads and writes that may access the
out-argument location.
Fixes #207969.
