[mlir][vector] Unroll transfers masked by vector.mask (#239554)
`UnrollTransferReadPattern` and `UnrollTransferWritePattern` reject
transfers inside `vector.mask` (#230461). Transfers with `in_bounds =
false` unroll today, so replacing `in_bounds` with masks (#227821) would
lose this unrolling if the masks are introduced before unrolling and
survive mask elimination.
Unroll them into one `vector.mask` per tile, with the matching tile of
the mask (and passthru, if any). Both patterns now use
`MaskableOpRewritePattern`.
Limited to minor identity maps, fixed-size vectors and target shapes of
the vector's rank. The mask-operand form is left unchanged; its removal
from transfers is tracked in #239511. Other masked transfers and other
ops under `vector.mask` are still not unrolled.
After `-canonicalize`, slices of a `vector.create_mask` become per-tile
`vector.create_mask` ops. For example, with step 4 and bound 1024,
[3 lines not shown]
[VPlan] Move extract-lane(0, canWideIV) combine before unrolling
Some extract-lane combines can't be performed before unrolling, e.g. the one that combines it to Instruction::ExtractElement can end up extracting from a different part. But the extract-lane(0, canWideIV) can because at index 0 you're always extracting from the first part. Do this before unrolling, which improves codegen of unrolled code.
[VPlan] Support interleaving early exit loops with side effects
We couldn't do this on main previously because the exit mask `active-lane-mask(0, FirstActiveLane(Cond))` doesn't get unrolled properly across parts. Now that we use mask.beforefirst we can unroll it properly in a chain. The unrolled form is:
unroll(mask.beforefirst, P) = mask.beforefirst(cond[P] | broadcast(anyof(not(unroll(mask.beforefirst, P-1)))))
On AArch64 this can be combined to brkpb in another patch.
[libc] Add nl_langinfo and nl_langinfo_l implementations (#243540)
POSIX defines nl_langinfo and nl_langinfo_l so callers can query locale
parameters such as CODESET and standard date, numeric, and message
strings without parsing external locale files. Now that the internal
locale tables exist, this change implements both entrypoints across all
57 POSIX <langinfo.h> items and exposes them on Linux targets.
nl_langinfo resolves directly against the fixed compiled-in default
locale tables without consulting mutable global or thread-local state,
while nl_langinfo_l queries the caller-supplied locale_t handle (falling
back to the compiled-in default when LIBC_CONF_ENABLE_RUNTIME_LOCALE is
disabled). Unrecognised nl_item values return "" without modifying
errno.
Assisted-by: Automated tooling, human reviewed.
Reland "[RegisterCoalescer] Keep endpoints of lane-carrying IMPLICIT_DEF" (#243068)
Relands #230141, reverted in #242409.
The new test had a kill flag on a virtual register, which the
MachineVerifier rejects since #239012. Dropped it; no code change.
Original commit message:
Since #196895, pruneValues discards the endpoints of a value copied from
an erasable IMPLICIT_DEF. A partial IMPLICIT_DEF without `<read-undef>`
carries other lanes those endpoints may read, so discarding them leaves
a hole in the live range. Keep them unless the source has no ValidLanes.
Assisted-by: Claude Opus 5.5
[VPlan] Make VPlan::hasEarlyExit operate on loop regions. NFC (#243753)
This allows it to handle early exit loops with side effects, which it
currently misses. Early exit loops with side effects have
indistinguishable control flow to regular loops after the regions are
dissolved.
The only post-dissolve caller is the LoopInfo fixup for dispatch blocks.
However we don't need to check hasEarlyExit here, as it won't touch any
loops that aren't dispatch blocks.
TableGen: Index patterns by predicates when generating variants (#244106)
GenerateVariants quadractically compared each new variant against
every pattern to find duplicates, though only patterns with the same
predicates can match. Index the patterns by predicates and only
compare against those.
RISCVGenDAGISel.inc tablegen run:
Gigainstructions Gigacycles CPU seconds
Before 56.9 27.8 8.7
After 50.0 (-12.2%) 25.2 (-9.3%) 8.0 (-8.8%)
AMDGPUGenDAGISel.inc tablegen run:
Gigainstructions Gigacycles CPU seconds
Before 91.2 44.6 14.2
After 85.3 (-6.5%) 42.7 (-4.2%) 13.4 (-5.4%)
[6 lines not shown]
[Hexagon] Ignore debug uses when generating muxes (#232237)
HexagonGenMux incorrectly treats register uses from DBG_VALUE
instructions as executable register uses when building its def-use maps.
A debug-only use of the destination register can therefore prevent two
complementary conditional transfers from being folded into a C2_mux.
Keep debug instructions in the positional map but exclude them from the
executable def-use information used by mux analysis. Add a regression
test covering a DBG_VALUE between the conditional transfers.
Fixes #227919
[mlir][llvm] Import and export DIEnumerator debug info (#243873)
Import llvm::DIEnumerator as DIEnumeratorAttr and export it back, so
enumeration types keep their enumerators. This follows the attribute
added in #241969, and the discussion in
https://discourse.llvm.org/t/interest-in-adding-dienumerator-and-template-parameter-debug-info-to-the-llvm-dialect/92776.
Before this change DebugImporter did not handle DIEnumerator, so
translating an enumerator returned null, and because the importer drops
the whole elements list when one element fails to translate, an
enumeration type was imported without any of its enumerators.
The importer keeps the name (null when the LLVM name is empty), the
unsigned flag, and the value as an IntegerAttr with exactly the width of
the LLVM APInt. The exporter builds the llvm::DIEnumerator from the same
three pieces. As usual for signless integers, an unsigned value prints
in its signed form, for example value = -1 : i64, isUnsigned = true, and
a one bit value prints as true or false. LLVM's textual IR parser gives
a literal such as 0 or -1 a width of one bit, so these two import as
[12 lines not shown]
[lld-macho] Give thunks to linker-private labels no symbol
Like the labels themselves, their thunks stay out of the output symbol
table, as in ld-prime.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[orc-rt] Make NativeDylibManager's test dylib a shared input. (#244105)
The shared library added for the NativeDylibManager unit tests isn't
actually NativeDylibManager-specific. Move it to test/Inputs as
orc-rt-test-dylib, so that it can be reused by other unit and regression
tests. Unit tests find it through the ORC_RT_TEST_DYLIB_PATH compile
definition (previously NDM_TEST_LIB_PATH), and regression tests through
a new %{test-dylib} substitution.
The library is now written in C, defines its own export macro rather
than using the unit tests' TestVisibility.h, and its functions are
renamed to orc_rt_test_dylib_return42 and orc_rt_test_dylib_return7.
[Hexagon] Add a limit on the HLX vector stack size (#243174)
Report an error when a function needs more HLX vector (VTCM) stack than
allowed, including the area for outgoing HLX stack arguments. The limit
comes from the "hlx-vector-stack-limit" function attribute, or else from
the -hlx-vector-stack-limit option, and there is no limit by default.
In Clang, the -fhlx-vector-stack-limit=N driver option, which requires
-mhlx, sets the limit on all functions through the
"hlx-vector-stack-limit" function attribute in IR. A function can
override it with the hlx_vector_stack_limit(N) attribute.
Assisted by AI
---------
Co-authored-by: Ankit Aggarwal <aankit at qti.qualcomm.com>
Co-authored-by: pavani karveti <pkarveti at qti.qualcomm.com>
[flang][mlir][OMPIRBuilder][OpenMP] Support iterator modifier in declare mapper map clause (#235491)
This PR adds LLVM code generation support for the `iterator` modifier in
`map` clauses inside `declare mapper`, building on the lowering added in
[#241358](https://github.com/llvm/llvm-project/pull/241358).
**Implementation:**
- `OMPIRBuilder`: added `MapInfosTy::DynamicSegments` to describe groups
of mapper components whose count is only known at runtime (trip count, a
callback producing one iteration's base/begin/size, the map type, and an
optional child mapper). `emitUserDefinedMapper` emits a loop per segment
that pushes each component via `__tgt_push_mapper_component`, resolving
`MEMBER_OF` against `__tgt_mapper_num_components`. Added
`LeadingParentCond` to mark the first static component as a parent
synthesized only to group those segments, pushed conditionally at
runtime.
- Added runtime handling for iterator-generated map entries using
`__tgt_push_mapper_component`, since the number of elements is known
[22 lines not shown]
[Hexagon] Test that HLX pair alloca lowering keeps half-vector loads (#243916)
Check that a store to an HLX VectorPair alloca is lowered to SS16b.new
while an HLX_Vector-typed load from the same alloca stays a plain load.
Assisted by AI
TableGen: Index patterns by predicates when generating variants
GenerateVariants quadractically compared each new variant against
every pattern to find duplicates, though only patterns with the same
predicates can match. Index the patterns by predicates and only
compare against those.
RISCVGenDAGISel.inc tablegen run:
Gigainstructions Gigacycles CPU seconds
Before 56.9 27.8 8.7
After 50.0 (-12.2%) 25.2 (-9.3%) 8.0 (-8.8%)
AMDGPUGenDAGISel.inc tablegen run:
Gigainstructions Gigacycles CPU seconds
Before 91.2 44.6 14.2
After 85.3 (-6.5%) 42.7 (-4.2%) 13.4 (-5.4%)
[7 lines not shown]
[BOLT] Skip symbols with no name (#239088)
In llvm-nm, an existing bug makes smybol with no name listed in
special-syms. This makes later value unpacking fails. It has been fixed
in trunk but is not in the older release version. We workaround this by
skipping such symbols by check len < 3 or not.
See: https://github.com/llvm/llvm-project/pull/192129
[CMake] Enforce -Wglobal-constructors for MC, MCParser, NVPTXCodeGen (#244103)
They no longer have C++ dynamic initialization after their cl::opt
globals moved to TableGen.
[RISCV] Increase the vector spill/reload cost (#243766)
When spilling/reloading a vector register, the stack slot address
need to be materialized before the store/load.
[clang] Add new volatile -Wunsequenced support under -Wunsequenced-volatile (#244073)
So it can be disabled separately while people clean up violations.
Initially implemented in #180955.
Assisted with Gemini