[SLP] Skip getelementptr pairs that are related affine loop recurrences (#226220)
When the loop vectorizer scalarizes strided accesses, each lane computes
its address as `base + ((off + stride * (iv + k)) << 2)`, with a
loop-invariant stride and offset, which loop strength reduction turns
into pointer increments. Since #150450 made the cost model loop-aware,
`vectorizeGEPIndices` compares against that scalar arithmetic and finds
`<4 x i64>` mul/add/shl, with every lane extracted for the scalar
accesses, profitable.
`vectorizeGEPIndices` already drops pairs of getelementptrs with a
constant difference. This patch also drops pairs that are affine
recurrences of the enclosing loop with a loop-invariant difference,
since LSR computes both from a common induction variable. Since #228918,
the once-used seeds no longer pick up this index arithmetic, so no extra
seed filtering is needed.
For `strided_gather_offset` in `X86/strength-reducible-address.ll`
(#226907), the loop goes from 30 to 13 instructions with Haswell code,
[17 lines not shown]
[flang] Add fir.assert and its LLVM code generation (#231245)
Add the new fir.assert operation for which the design was introduced in
https://github.com/llvm/llvm-project/pull/228502
Verify its message and diagnostic operands, and erase constant-true
assertions. Keep attribute and optionality queries able to inspect
guarded values without making rewriting operations transparent to the
guard.
For the LLVM code generation, generate assertions to a weighted branch,
a noreturn runtime call on failure, and guarded operands on success.
Pack larger diagnostic payloads into an array on the failure path and
preserve source locations and device symbol scope. Add dialect, code
generation, effect, and variable-query tests.
This introduces the operation without emitting it from source lowering.
Assisted-by: AI
[flang][debug] Name real and complex types by their kind (#232259)
Debug info names intrinsic types <type>(kind=N), with N taken from the
size of the type in bytes. So real(kind=3) (bfloat16) is 2 bytes, so it
was named real(kind=2) and became the same debug type as IEEE half
real(kind=2).
Take the kind of real and complex types from mlirFloatTypeToKind
instead. Logical types now use their own kind as well.
Co-authored-by: Cursor <cursoragent at cursor.com>
SelectionDAG: Don't fold bitcasts of opaque constants to FP (#232633)
Folding the bitcast of an opaque integer constant into a ConstantFP
drops the opaque flag. Skip the fold like the truncate fold does, so a
hoisted constant stays a shared integer. In aarch64-pmull2.ll the
hoisted constant is now materialized with an equivalent 32-bit mov.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[ORC] Add InstanceSymbolGenerator for runtime instance symbols (#232575)
The ORC runtime gives names, called instance symbols, to objects created
at runtime, since such objects have no symbols under the standard object
models. It exposes them through its
orc_rt_ci_sps_Session_lookupInstanceSymbols controller-interface
function.
InstanceSymbolGenerator is a DefinitionGenerator that generates symbols
by looking them up in the ORC runtime Session's instance symbols (via a
Proxy). InstanceSymbolGeneratorSPS.h binds the proxy to the runtime's
SPS controller interface, using a new
rt::sps_ci::SessionLookupInstanceSymbols descriptor.
ExecutionSession's constructor now checks the bootstrap symbols for the
lookup function and, if present, adds an InstanceSymbolGenerator to the
bootstrap JITDylib.
This also adds SymbolLookupResult and RemoteSymbolLookupResult aliases
[2 lines not shown]
[llvm][test][debuginfod] Fix flaky timeout.test by using an Event (#232312)
Fixes #232238
Because this test relies on real time, the OS scheduler may delay the
client long enough to get past the server's fixed delay causing the test
to fail.
We only care that the client applies a timeout, not that it is of some
set number of seconds. So we can use an Event to stall the server until
we know the client has exited.
I also added a very long timeout on the subprocess call now in case of a
bug in the tested utility. Ideally this would be applied by lit, but one
of the lit timeout options requires the psutil library, and builders
might have their reasons for wanting one or the other option. This is
something we can address in future but right now a big number is the
solution.
[Propeller] Implement direct BBID-based matching for CFG profile inference. (#231317)
In BasicBlockMatchingAndInference, add support for mapping CFG profile
nodes and edges directly using UniqueBBIDs via -propeller-match-bbid
(default false). Assuming basic block IDs are consistently assigned,
this maps profile counts directly without requiring hash matching or
applying the infer threshold check. If a function has no Propeller CFG
profile, bypass inference so that the function falls back cleanly to
default or iFDO layout. Basic block path cloning does not run before
matching and inference, so blocks are identified by their base IDs, and
the counts of clones in the profile are attributed to the original
block.
A basic block is only matched if its successors and predecessors in the
CFG profile are also its successors and predecessors in the MIR.
Profiles may omit edges without samples, so MIR edges that are missing
from the profile are allowed. A block with a profile edge that is not in
the MIR is not matched and kept as unknown, so that
SampleProfileInference can infer its weight.
[2 lines not shown]
[AArch64][llvm] Combine SVE AND-NOT operations into BIC for _x forms
Replace NOT followed by AND with BIC for expressions such as:
return svand_u32_x(svptrue_b32(), a, svnot_u32_x(svptrue_b32(), b));
Combine ordinary AND with NOT when the NOT predicate is all-active or
its inactive values are undefined.
Also combine matching-predicate SVE AND_x-NOT pairs before lowering
discards the AND predicate. AND_x ignores inactive results, so the NOT's
inactive values do not need preserving.
This applies to all integer element widths. Update the pre-commit LLVM
expectations, including cases where combining would change the
result.
[AArch64][llvm] Combine SVE AND-NOT operations into BIC for _m forms
Replace NOT followed by merging AND with BIC for expressions such as:
return svand_u32_m(pg, a, svnot_u32_x(pg, b));
Combine when NOT is the second operand and its predicate matches the AND
predicate or is all-active.
Use predicated BIC to preserve the AND's first operand in inactive
lanes. Use unpredicated BIC when the AND predicate is all-active.
Do not commute merging AND operands, since this would change inactive
results. Leave pairs with incompatible predicates unchanged.
This applies to all integer element widths. Update the pre-commit LLVM
expectations, including cases where combining would change the
result.
[AArch64][llvm] Add pre-commit tests for SVE AND-NOT combining
Add LLVM tests for expressions where NOT followed by AND could
use BIC. Record the existing assembly so the following changes show the
missed optimisations being fixed.
Cover _x and _m AND forms using all-active and runtime predicates. Test
_m forms across all integer element widths. Include NOT with defined
inactive values and reversed AND operands.
Also test cases that must remain as NOT followed by AND because
combining would change the values in inactive lanes.
[flang] Refine fir.assert verification and target lowering
Restrict diagnostic values to i8, i16, i32, i64, and index, requiring
producers to choose how boolean values are extended. Use target global
address spaces for diagnostic strings and cast their addresses for calls.
Use runtime entry-name macros and a named fixed-value limit, remove a
redundant verifier check, and cover incompatible declarations, symbol
conflicts, explicit boolean extension, and global address spaces in tests.
[flang] Add fir.assert and its LLVM lowering
Introduce a non-speculatable assertion with typed guarded results and a
write effect on a non-addressable Fortran runtime resource. Verify its
message and diagnostic operands, and erase constant-true assertions.
Keep attribute and optionality queries able to inspect guarded values
without making rewriting operations transparent to the guard.
Lower assertions to a weighted branch, a noreturn runtime call on failure,
and guarded operands on success. Pack larger diagnostic payloads into an
array on the failure path and preserve source locations and device symbol
scope. Add dialect, code generation, effect, and variable-query tests.
This introduces the operation without emitting it from source lowering.
[flang-rt] Add fatal diagnostics with integer values (#231241)
Add noreturn runtime entries for messages with integer diagnostic
values. Use a fixed four-value signature for small messages and a
count-and-array signature for larger payloads, keeping both callable
from device code.
The array entry accepts the restricted %lld/%% format, checks argument
counts and allocation sizes, and preserves the existing source location,
crash handler, and termination behavior. Document the runtime interfaces
in the inline-check design and cover formatting and failures in unit
tests.
Assisted-by: AI
[mlir][x86] Add pattern to tile and unroll vector.contract for AMX nanokernel (#229076)
Introduces a new rewrite pattern to tile and unroll a `vector.contract`
inside an accumulation loop into a loop nest and smaller accumulation
chains on which the AMX nanokernel pattern can match.
The rationale is to provide an entry point for tile-based DSLs that
leave tiling decisions to the user, and compile directly to `vector` IR,
bypassing the usual infrastructure.
This PR covers the AMX BF16 and int8 targets. The pattern match is quite
rigid in this initial version, as the transformation reconstructs the
accumulation loop with smaller shapes instead of rewriting, and the
contraction's shape must be cleanly unrollable to the nanokernel-sized
contractions.
Assisted-by: Copilot
[Flang] Introduce common::numeric_limits (#228108)
On platforms that do not have a native (u)int128_t, the class
`Int128<IS_SIGNED>` is meant to provide its own implementation. The C++
specification allows programs to provide its own specializations of
`std::numeric_limits<T>` for such user-defined types
([[namespace.std]](http://www.eel.is/c++draft//namespace.std#2)).
However, the combination of clang-cl using the MSVC STL or libstdc++ in
strict mode poses a problem:
1. While Clang has a int128_t native type, MSVC's STL implementation
does not provide a `std::numeric_limits<T>` specialization for it
(because Microsoft's cl.exe doesn't have a 128 bit integer type).
2. Some versions if libstdc++ disable all uses of __int128 in strict
mode (-std=c++NN instead of -std=gnu++NN), including the specialization
of `std::numeric_limits<T>`. Moreover, `std::numeric_limits<__int128>`
being specialized does not mean that `std::make_unsigned_t<__int128>`
also is.
A previous solution in #216958 was to provide our own specialization of
[14 lines not shown]
[LoongArch] OptWInstrs: Sign-extend outer PHI operands to remove sext.w in loops
A sext.w inside a loop often cannot be removed only because one of the
values reaching it comes in through a PHI from outside the loop and is not
known to be sign-extended, e.g. an i32 argument that is only any-extended:
preheader:
...
loop:
%p = PHI %x, %preheader, %next, %loop
%s = ADDI_W %p, 0
%next = ADD_D %p, ...
If every other value reaching the sext.w is sign-extended, sign-extend
those PHI operands in their incoming blocks instead and remove the sext.w.
This trades a sext.w executed on every iteration for one executed once per
entry into the loop.
Only incoming blocks with a smaller loop depth than the sext.w are
[8 lines not shown]
[Inliner] Change CalleeTTI -> CallerTTI
It makes more sense to use the caller's TTI, as this way we can query
whether the instruction can be inlined into the parent function, using
state described by the caller's TTI.
For example, for AArch64 SME where a function can be in a different mode
(streaming/non-streaming), the cost-model gives different answers based
on the TTI used.
For now, we don't expect to see any changes in behaviour as there are
currently no targets where it makes different cost/inlining-decisions
based on which TTI is used.
SelectionDAG: Don't fold bitcasts of opaque constants to FP
Folding the bitcast of an opaque integer constant into a ConstantFP
drops the opaque flag. Skip the fold like the truncate fold does, so a
hoisted constant stays a shared integer. In aarch64-pmull2.ll the
hoisted constant is now materialized with an equivalent 32-bit mov.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[flang] Add check for definability for Image-selector STAT= specifier #224058 (#230031)
This MR fixes #224058, where Flang accepts a non-definable
`stat-variable` in an image selector without any diagnostic..
[AArch64][GlobalISel] Generate TBL for shuffles in single block loops (#230673)
This is the global-isel equivalent of #227593, where we generate a tbl
in a loop as opposed to multiple shuffle instructions. The only major
difference is that we do not support perfect shuffle yet (this is a
prerequisite for that), and that we pick up the single basic block from
the MachineBasicBlock directly.
[AArch64][SME] NFC: Inline getCompatibilityForChangeToStreamingMode. (#231899)
This removes the need for the `StreamingIncompatibility` enum and makes
the code a little simpler.
This PR also moves `hasPossibleIncompatibleOps` to AArch64TTIimpl, so
that we can query the costmodel in a follow-up PR.
