[CIR] Emit inbounds/inrange for vtable address points in initializers (#227520)
Classic codegen emits `inbounds inrange(...)` for vtable address points
in global initializers: VTT entries and the vptr of a
constant-initialized object. CIR never emitted any flags for
`#cir.global_view`. It only got `inbounds` and `nuw` when LLVM's
constant folder inferred them, which stopped for a while after
https://github.com/llvm/llvm-project/pull/226904 until #227651 restored
the folding. It never got `inrange`.
This patch emits the flags in CIR instead of relying on the folder:
- Add an `address_point` flag to `#cir.global_view`, spelled
`#cir.global_view<@vtable, [vtable index, slot], address_point>`.
- Set it where CIRGen builds VTT entries and constant vptrs, and keep it
when CIRGen and `CXXABILowering` rebuild the attribute.
- When lowering to LLVM, make an address point `inbounds`, with an
`inrange` covering the one vtable in the group that it points into. The
bounds come from the vtable global's type, so they also work when the
[10 lines not shown]
[AArch64][SVE] Fix predicate type for quadword gather/scatter SDNodes (#228274)
GLD1Q_MERGE_ZERO and SST1Q_PRED use a single predicate bit per 128-bit
quadword (always nxv1i1), not one bit per data element.
SDT_AArch64_GATHER_VS and SDT_AArch64_SCATTER_VS incorrectly required
the predicate to have the same number of elements as the loaded/stored
vector via SDTCisSameNumEltsAs<0,1>, which only happened to hold for the
data width these nodes were originally used with.
Give these two nodes their own SDTypeProfile that fixes the predicate
type to nxv1i1, and update the corresponding patterns in
SVEInstrFormats.td to match.
Co-authored-by: Claude Sonnet 5.5 <noreply at anthropic.com>
[AArch64] Remove stale packed extend/shift for memory ops(NFC) (#230068)
This patch removes stale extend/shift representation for memory ops.
They use packed representation which has been replaced by operands to
the instruction format.
[AMDGPU] Model gfx1250-strict WMMA latencies
gfx1250-strict shares the gfx1250 scheduling model, but some WMMA
instructions take longer on it:
- 16x16x64 FP8/BF8: 8 cycles instead of 4.
- f8f6f4: 16 cycles when any input is f8 and 8 cycles otherwise, instead of
8 and 4.
The WMMA co-execution hazard category is derived from the latency, so this
also gives these instructions the required number of wait states on
gfx1250-strict.
Read the f8f6f4 matrix formats through named operands for both MCInst and
MachineInstr, so llvm-mca models the strict latencies too. This also fixes the
both-f4 check, which read fixed operand indices and could take other
immediates for f4 formats: inline constant scales of the scaled forms, which
gave too few hazard wait states, and the neg_lo/neg_hi operands of the
unscaled form in assembler input.
[flang][OpenMP] Privatize loop IVs in the innermost parallel (#227486)
A sequential loop's iteration variable is predetermined private in the
innermost parallel, teams or task-generating construct that encloses
the loop. When the loop was nested in another construct, such as a
worksharing loop, lowering failed to privatize the variable in the
enclosing parallel region. Instead, it created a new local copy inside
the nested construct, so the parallel region's other references to the
variable used the shared host variable.
Fix this by deciding which construct privatizes a symbol based on the
scope that owns it. This also simplifies DataSharingProcessor: the
OMPConstructSymbolVisitor, which walked the parse tree to track where
symbols were defined, is no longer needed and has been removed.
I have noticed that metadirectives don't always own the symbols that
should be privatized in them, as semantics doesn't create a new scope.
I'm not very familiar with metadirectives, but it seems this causes some
privatization issues with non-explicitly specified DSAs, as
[7 lines not shown]
[InstCombine] Fix profiles in foldMulSelectToNegate (#229865)
The condition is always the same as the original select, so we can just
propagate the metadata after using a matcher to ensure that we get the
actual SI out.
AMDGPU: Remove remaining uses of -amdgpu-scalarize-global-loads=false
Stop relying on the option to select vector loads from uniform kernel
argument pointers.
Index loads by the workitem id where the test is about memory operations.
Use volatile loads where the test depends on a uniform value held in
VGPRs. Use functions with inreg pointer arguments for the MUBUF encoding
tests, which preserves the existing encodings. Convert the early
if-conversion tests that only need operand values into functions.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
AMDGPU: Use functions in more operation tests instead of kernel loads
Stop relying on -amdgpu-scalarize-global-loads=false. Inputs are passed
as VGPR arguments, or inreg for SGPR operands. Kernels that check
multiple stores index their loads by workitem id.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[CodeView] Set the S_COMPILE3 hotpatch flag from a module flag (#229868)
Previously, the `HotPatch` flag in CodeView `S_COMPILE3` was only
derived from `TargetOptions::Hotpatch`, which is set by clang's backend
setup but not by LTO code generation. Objects produced by LTO were
therefore never marked as hotpatchable, and lld's `/FUNCTIONPADMIN`,
which only pads chunks from hotpatchable objects, left their functions
unpadded, even though the `"patchable-function"` attribute still made
code generation pad the function entries.
After this PR, emit an `"ms-hotpatch"` module flag when compiling with
`/hotpatch`, and also set the CodeView flag from it. The flag uses the
`Min` merge behavior, so when LTO merges modules, it is reset to 0 if
any module was compiled without `/hotpatch`, and the result is never
wrongly marked as hotpatchable.
This came up while working on
https://github.com/llvm/llvm-project/pull/229477
[4 lines not shown]
[LoopUnroll] Snapshot loop blocks after runtime remainder generation (#229779)
Runtime remainder unrolling can simplify the enclosing loop nest and
delete blocks from the loop being unrolled. The early OriginalLoopBlocks
snapshot then contains dangling pointers, causing a crash during
dominator-tree updates.
Capture the snapshot after remainder generation and before cloning adds
new blocks to the loop. The included test aborts without the fix with
`cannot get DomTreeNode of block with different parent (exit 134)`, and
passes lit with the fix.
Assisted by AI tooling, to triage the bug and find an LLVM IR reproducer
for the crash.
[CIR] Put try_throw's normal destination in the throw's region (#229915)
`replaceThrowWithTryThrow` creates the unreachable normal destination of
`cir.try_throw` at the end of the parent function.
This is not correct for coroutines. Doing so would create a reference to
a block outside their respective regions, leading to a verification
error.
This patch moves the unreachable normal destination of `cir.try_throw`
in `replaceThrowWithTryThrow` right below the throw's region.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[lld][MachO] Write the contents of an output section in parallel (#229512)
This is part of the ld64.lld performance improvements tracked by #222068
Writer::writeSections() does actually run the output sections in
parallel, but each section's writeTo() is not done in parallel, so
there's still a lot of work done by one thread at a time.
We can parallelize the writeTo() calls pretty easily as each input
writes to its own range in the output buffer.
This commit is output-preserving.
[InstCombine] Fold fpto[su]i.sat of NaN guarded select
fpto[su]i.sat returns 0 for both NaN and $\pm 0.0$, so a select that
replaces a NaN input with zero does not change the result, and the call
can take X directly.
```llvm
%not.nan = fcmp ord float %x, 0.0
%sel = select i1 %not.nan, float %x, float 0.0
%r = call i32 @llvm.fptosi.sat.i32.f32(float %sel)
-->
%r = call i32 @llvm.fptosi.sat.i32.f32(float %x)
```
The inverted form `select (fcmp uno X, 0.0), 0.0, X` is folded too.
Only the canonical `fcmp ord/uno X, 0.0` is matched, which also covers
`oeq/une X, X` after fcmp canonicalization.
[InstCombine] Add tests for fpto[su]i.sat of NaN guarded select. NFC
fpto[su]i.sat returns 0 for NaN and +/-0.0, so the select is redundant.
Cover ord/oeq/uno/une checks, scalar and vector, and negative cases.
[flang][OpenMP] Switch clause verification to descriptor-based
Delete all the scattered pieces of clause verification that are now
replaced by the unified handling.
[flang][OpenMP] Improve diagnostics about modifier properties
Use OpenMPDeprecated and OpenMPFuture warning categories for modifier
diagnostics as well, analogously to how they are used for clauses.
[flang][OpenMP] Account for using different versions for modifier checks
When a modifier from a past/future version is accepted, use its properties
from the nearest version in which it is allowed.
[flang][OpenMP] Create generic interfaces for "container" entities
In short, clauses are containers of modifiers, and directives are
containers of clauses. With the abstractions in place, clauses and
modifiers look almost the same from the point of view of syntactic
properties.
Use these functions to implement generic GetAllowedElements and
GetElementVersionRange functions.
[flang][OpenMP] Properly use GetAllowedElements
Allowed elements are not just those explicitly listed, but also those
from the union of allowed sets.
For example, the set of modifiers allowed on a clause are those that
are listed on a given clause, plus the union of all modifier sets that
the clause allows.
[GitHub] Bump CI Tooling Container to 23.1.3 (#230178)
To pull in 5340f7cc8814b89c01a786e85e1736e10159a9f0, which is impacting
clang-format results.
Part of fixing #230169.
[MISched] Implement `UnanalyzableFrontier`-based DAG construction algorithm (#227375)
Introduces a new algorithm for constructing the control dependencies in
the schedule DAG. The new algorithm is gated behind a (temporary)
`cl::opt` (`-enable-unanalyzable-store-sequencing`), currently off by
default. With the option disabled, the change is an effective NFC: edge
insertion order changes slightly, but the DAGs do not change materially.
The difference between this algorithm and the existing algorithm is
that, rather than maintaining all unanalyzable memory operations and
repeatedly querying all of them, we instead maintain a frontier of loads
and stores since a 'sequencing store'. A sequencing store is either (a)
a store that writes to different set of base objects to the last seen
sequencing store or (b) any store if there has been a load from another
base object since the last sequencing store. We treat the sequencing
stores carefully to ensure that all previously seen unanalyzable memory
operations that do not belong to the frontier necessarily transitively
succeed the current sequencing store. This makes it sound to sequence
preceding memory operations against the sequencing store and only those
[9 lines not shown]
