[LoopVectorize][X86] Improve libmvec SSE/AVX/AVX-512 test coverage (#211863)
This PR updates the libmvec loop vectorization tests to use `-force-vector-width` instead of hardcoded loop metadata.
Fixes #211532
[APFloat] Don't turn a Float8E8M0FNU NaN into an Inf on conversion (#214919)
`Float8E8M0FNU` has a precision of 1, so it stores no significand bits
and its NaN carries no payload. `IEEEFloat::convert` shifts the (empty)
significand into the target format, which leaves the NaN exponent with
an all-zero significand — and that is the encoding of an infinity in any
format that has infinities.
The `APFloat` object still reports `fcNaN`, so `isNaN()` returns true,
but `bitcastToAPInt()` hands back Inf bits. Anything that stores the bit
pattern rather than the object — `ConstantFP`, MLIR's `FloatAttr` —
therefore ends up with an infinity.
In MLIR this shows up when folding `arith.extf` on an `f8E8M0FNU`
constant. The OCP MXFP spec uses the all-ones encoding as the scale of
an invalid block, so this silently replaces "this block is invalid" with
a value that poisons everything it is multiplied into:
```mlir
[27 lines not shown]
[analyzer] Fix -analyzer-output=html assert on reversed and macro ranges
HTMLDiagnostics::HighlightRange guarded against a reversed range by
comparing line numbers, so a same-line reversal - which is what the piece for
an implicit copy constructor carries - reached html::HighlightRange.
Its scan walks from begin to end, ran off the end of the buffer, and asserted:
https://godbolt.org/z/sTb5qfjjd
Invalid position to insert! (RewriteRope.h)
It also added the end token's length itself and then passed a token range to
html::HighlightRange, which measured the token again, this time from the
interior. For most tokens the two cancel, but where the tail re-lexes longer
the highlight reached past the end of the range, e.g. over a trailing ';'.
Use getExpansionRangeInFile(), which rejects reversed and cross-file ranges,
then convert once and tell html::HighlightRange the range is already
char-granular.
[4 lines not shown]
[analyzer] Fix -analyzer-output=sarif crash on macro-expanded ranges
A path piece whose range ends inside a macro expansion aborted the whole
document: https://godbolt.org/z/61vWYcsWj
Cannot create a physicalLocation from invalid SourceRange!
convertTokenRangeToCharRange() built the end with
Lexer::getLocForEndOfToken(), which returns an invalid location for a macro
ID that is not at the end of its expansion, and used it unchecked. The
analyzer's own test corpus hits this in nine files; text and plist output
were unaffected because both already map such ranges to the expansion.
- Use getExpansionRangeInFile(), so the region covers the macro use like the
other two outputs.
- Fall back to a caret when the range is unusable. A thread flow needs a
location per piece, so dropping one would truncate the reported path. This
also stops reversed ranges producing regions with endColumn < startColumn.
[4 lines not shown]
[clang] Reject ranges getExpansionRangeInFile cannot represent
getExpansionRangeInFile was extracted verbatim and inherited two shortcomings
of the original loop, fixed here before the analyzer's SARIF and HTML consumers
depend on it:
- It mapped the end with getExpansionRange(SourceLocation), which always
reports a token range, so a char-range input was widened by a whole token.
Now using the getExpansionRange(CharSourceRange) overload, which keeps the flag.
- It passed reversed ranges through. Consumers walk begin->end; now returning
nullopt for those, as Lexer::makeFileCharRange already does.
Separate from the extraction so that stays NFC, and out of the consumer fixes
because it changes the shared helper's contract rather than one output.
Both contract changes, plus the invalid- and cross-file-range guards, are
covered by a GetExpansionRangeInFile unit test in
clang/unittests/Frontend/TextDiagnosticTest.cpp.
Assisted-By: claude
[VPlan] Cost truncated widened inductions via ::computeCost. (#212786)
Follow-up to https://github.com/llvm/llvm-project/pull/202232 to also
compute costs for truncated inductions in ::computeCost, removing the
fallback to the legacy cost model unless the loop is fully unrolled.
Note that this changes vectorization decisions in both direction, e.g.
* now vectorized `@tail_predicate_without_optsize`: legacy costs <16 x
i64>, whereas we generate a narrow IV <16 x i8>, which is much cheaper
* no longer vectorized `@second_lshr_operand_zero_via_scev()`: we
generate 2 IVs (one truncated and one not truncated), which is more
expensive than the single IV LV assumes (note that previously we would
ignore the cost of the trunc in the cost computation, because the trunc
was replaced by the wide truncated IV, which was assumed free.
Both cases are due to more accurate cost computations.
PR: https://github.com/llvm/llvm-project/pull/212786
[LV] Vectorize down-counting floating-point argmin/argmax reductions
Extend the multi-use min/max reduction coupling so a floating-point
min/max value reduction can be paired with a down-counting (FindFirst)
index reduction, enabling argmin/argmax vectorization for loops whose
induction counts down.
Analysis (IVDescriptors):
- Recognize select-based FP min/max recurrences, not just the
intrinsic form.
- For the shared-compare argmin/argmax shape (the reduction compare
also feeds an index select), take the required NaN-free and
signed-zero-free facts from the compare when the select itself does
not carry them. Plain min/max reductions keep the strict
flags-on-the-select rule.
VPlan (handleMultiUseReductions):
- Accept a select as the value reduction's min/max operation and read
its value operands accordingly.
[14 lines not shown]
[GVN] Recover affine-equal select-dependent load addresses
Teach PHITransAddr to recover available select-arm addresses when a
constant GEP index adjustment is folded into a nested byte offset. This
lets GVN reuse the existing load-PRE path without depending on SCEV.
[lldb][AArch64] Detect type for many registers with a single function (#214734)
It was pointed out during review of
https://github.com/llvm/llvm-project/pull/214515
that the type for the 2 GCS registers will be created twice.
Each one will have the same ID so we will only emit one and
the other goes unused.
To account for this, and the possibility of not just 2 but N
registers later, I've changed the name in the register entry
to a list of names.
So for the 2 GCS registers we only do detection once, and both
of them will refer to the same instance of the type.
[VectorCombine] Fold reduction chains with equivalent bitcast sources (#212084)
## Description
`foldShuffleChainsToReduce` currently identifies leaf sources by SSA
value identity.
This can prevent a shuffle reduction chain from being folded when
multiple bitcast instructions represent the same vector source.
This happens when an earlier VectorCombine transform moves a bitcast
through a shuffle and creates a new bitcast of the same underlying
value.
The reduction matcher then sees the original and newly created bitcasts
as separate sources and rejects the fold.
This PR treats bitcast sources as equivalent when they have the same
result type and the same operand.
It then merges the demanded lanes of equivalent sources while preserving
duplicate-lane semantics:
[18 lines not shown]
[LoopSplitUtils] Revert, removing from tree (#214577)
This reverts commits:
- 2354dce21 ([Transforms][Utils] Add LoopSplitUtils for iteration-space
loop splitting, #205995)
- 8f1efc26 ([Transforms][Utils] Preserve branch weights in
LoopSplitUtils, #213626)
- 49ace5ab ([Transforms][Utils] Test for branch weight preservation in
LoopSplitUtils, #213647)
Removing LoopSplitUtils from the tree completely, as several crashes
were uncovered after it was added. A highly reduced initial version with
much better test coverage is proposed for the re-land.
[NVPTX] Add intrinsics for ue5m3x2 to f16x2/bf16x2 conversions (#214411)
This patch adds the following intrinsics for `ue5m3x2` to `f16x2/bf16x2`
conversions introduced in PTX 9.4:
- `cvt.rn.f16x2.ue5m3x2`
- `cvt.rn{.satfinite}{.scaled::n2::ue8m0}.bf16x2.ue5m3x2`
Tests have been verified through `ptxas-13.4`.
---------
Signed-off-by: DharuniRAcharya <dharunira at nvidia.com>
[lldb][AArch64] Detect type for many registers with a single function
It was pointed out during review of https://github.com/llvm/llvm-project/pull/214515
that the type for the 2 GCS registers will be created twice.
Each one will have the same ID so we will only emit one and
the other goes unused.
To account for this, and the possibility of not just 2 but N
registers later, I've changed the name in the register entry
to a list of names.
So for the 2 GCS registers we only do detection once, and both
of them will refer to the same instance of the type.
[lldb][AArch64] Remove size from type detector entries (#214733)
This was passed to the constructor but goes unused.
The detector functions know the size of the type
they're creating already.
[LV] Vectorize down-counting floating-point argmin/argmax reductions
Extend the multi-use min/max reduction coupling so a floating-point
min/max value reduction can be paired with a down-counting (FindFirst)
index reduction, enabling argmin/argmax vectorization for loops whose
induction counts down.
Analysis (IVDescriptors):
- Recognize select-based FP min/max recurrences, not just the
intrinsic form.
- For the shared-compare argmin/argmax shape (the reduction compare
also feeds an index select), take the required NaN-free and
signed-zero-free facts from the compare when the select itself does
not carry them. Plain min/max reductions keep the strict
flags-on-the-select rule.
VPlan (handleMultiUseReductions):
- Accept a select as the value reduction's min/max operation and read
its value operands accordingly.
[14 lines not shown]
[GVN] Replace SCEV address recovery with GEP peeling
Avoid introducing ScalarEvolution into GVN by teaching PHITransAddr to
recover affine-equivalent select-arm addresses from nested GEPs.
[ConstraintElimination] Defer removal of simplified ssub.with.overflow (#215135)
replaceSubOverflowUses erased the intrinsic as soon as it became dead.
That frees the intrinsic's operand Use array, but the worklist can still
hold UseCheck entries pointing into it, storing a now invalid pointer to
a Use *.
Instead of erasing the intrinsic in place, poison its arguments and push
it onto ToRemove.
PR: https://github.com/llvm/llvm-project/pull/215135
[lldb][AArch64] Detect type for many registers with a single function
It was pointed out during review of https://github.com/llvm/llvm-project/pull/214515
that the type for the 2 GCS registers will be created twice.
Each one will have the same ID so we will only emit one and
the other goes unused.
To account for this, and the possibility of not just 2 but N
registers later, I've changed the name in the register entry
to a list of names.
So for the 2 GCS registers we only do detection once, and both
of them will refer to the same instance of the type.
[lldb][AArch64] Remove size from type detector entries
This was passed to the constructor but goes unused.
The detector functions know the size of the type
they're creating already.
