[NVPTX] Select scalar_to_vector for bf16x2 (#236706)
Fixes #235368.
DAG combining can create a v2bf16 SCALAR_TO_VECTOR for byte-vector
concatenations, including copies of unaligned bfloat values through
local arrays. NVPTX treats the node as legal but lacks a selection
pattern.
Pack the bf16 value into the low half of a 32-bit register and zero the
unspecified high half, as already done for v2f16. Add regression
coverage for direct packing, unaligned loads, and the original
local-array copy.
[libc] Use a syscall wrapper for remap_file_pages
This PR move remap_file_pages behind the centralized syscall wrappers, following libc/docs/dev/syscall_wrapper_refactor.md.
[libc] Use a syscall wrapper for posix_madvise
This PR move posix_madvise behind the centralized syscall wrappers, following libc/docs/dev/syscall_wrapper_refactor.md.
[VPlan] Factor out VPSCEVExpander::canExpandInIR (NFC). (#240063)
Split off the check for expressions that are expanded in IR and hoisted
out of HoistOutOf, to reuse it in follow-ups.
[LV] Remove InnerLoopVectorizer (NFC) (#240055)
All that is left of InnerLoopVectorizer is a bundle of state passed to
executePlan, of which only the assumption cache, a fresh IRBuilder and
whether runtime checks were generated are used, plus fixVectorizedLoop,
which runs legacyCSE on the vector loop header.
Create the IRBuilder in executePlan, take the assumption cache from
Legal, pass whether there are runtime checks as a bool and run legacyCSE
directly. Also drop executePlan's DominatorTree argument, which is the
same as the planner's DT member, and update comments that still refer to
InnerLoopVectorizer.
PR: https://github.com/llvm/llvm-project/pull/240055
[KnownFPClass][NFC] Move fma logic out of fadd_impl, and document rounding mode assumption made by fadd/fsub (#240054)
I have plans to improve deductions made by `fadd`/`fsub` and `fma`.
However, some of these deductions are only valid for `fadd`/`fsub`, so I
have moved the `fma` logic outside of `fadd_impl` and into its own
`fma_impl` function. This allows us to make improvements to `fadd/fsub`
independently of `fma`, and it also makes it easier to audit
improvements to `fma`.
I also have plans to support the rounding mode for primitive arithmetic
functions https://github.com/llvm/llvm-project/issues/228974, so I have
documented the `NearestTiesToEven` assumption that `fadd_impl` makes
when ruling out `-0.0`, allowing `fadd_impl` to be valid for all
rounding modes.
[SLP] Give failed seed a try
Previously, with the following vectorizable code:
y = x * x * x;
where IR ordered the code as:
sq = x * x;
y = sq * x;
If the seed is [sq0, y0, sq1, y1] and the cost model rejects
vectorizing sq0 and sq1, y0 will be bypassed without any retry in `I
+= ActualVF - 1`.
To fix this, we give skipped nodes a chance by using them as the root
of the tree and rebuilding it. In the retry logic, we starts from
bring the child of failure match (sq0,sq1) but uses FirstSkipped as
the filter. In here, [y0, y1] will be the new User to go
TryVectorizeWindow. If it still fails, it tried to bring its child in
as the next level try. For example, [sq0, y0, z0, sq1, y1, z1]. [y0,
y1] will brings [z0, z1] in until converge.
[SLP] Pruning when there is no Undef
By checking all (two) of the users of isGuaranteedNotToBePoison.
1. IsNotPoisonedVec: FindReusedSplat do same check as HasUndefScalars
so it never called and can safely prunning.
2. all_of(NonConstants, [=](Value *V)...: Also do the same check as
HasUndefScalars.
Nowadays, we generates very few undef value. So the prunning here
makes sense as we try to do linear check to find a possible undef
instead of calling the very expensive isGuaranteedNotToBePoison.
[SLP][NFC] Add tests for dependent seeds in rejected windows
Precommit tests for the once-used seeds that depend on each other, such
as x * x and x * (x * x), whose window is rejected and whose users are
bypassed by the window walk.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[flang][openacc] Do not implicitly map whole common block when not needed (#239772)
A common block that is used in a compute region only as the `storage`
operand of the fir.declare of its members must not be implicitly mapped
as a
whole as it would clash with explicit mappings of the individual
members. When the common block is accessed directly, the mapping can be
done like before.
[SLP]Use fma as copyable main op without fmuladd lanes
fma(V, 1.0, -0.0) and fma(0.0, -0.0, V) equal V exactly, so fma
bundles model non-fma lanes as copyables, like fmuladd. fma lanes
never absorb fmul/fadd, so fma loses ties with fmul/fadd.
Assisted-by: Cursor
Reviewers:
Pull Request: https://github.com/llvm/llvm-project/pull/240039
OpenMP: Stop using TTI to pick the partial unroll factor in OMPIRBuilder
When the unrolled loop's CanonicalLoopInfo is needed by an enclosing
loop directive and no factor was given, unrollLoopPartial computed one
by constructing its own TargetMachine with default TargetOptions and a
standalone FunctionAnalysisManager, then running LoopUnrollPass's cost
model on the function while it was still being built.
This is a layering violation. Frontends should not be using codegen
optimization information to construct the original program. This will
not have a properly configured TargetTransformInfo from a pass
manager. The cost model is also not meaningful on incomplete,
unoptimized IR.
Use a factor of 2, matching what Sema's OpenMP loop transformation
uses for a partial clause without a factor. This removes the hidden
-openmp-ir-builder-unroll-threshold-factor option.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
OpenMP: Use TargetMachine::getTargetIRAnalysis in OMPIRBuilder
Use the TargetMachine's TargetIRAnalysis instead of wrapping
getTargetTransformInfo in a custom callback. This is a marginal
improvement. TTI should come from the pass manager. There isn't a pass
manager because this is frontend IR construction code, and it's
entirely inappropriate to be using TTI at all.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[Support] Pass tool options to parseCommandLineOptions
Follow-up to #239920: `RegisterMCToolOptions` adds `MCToolOptions` to
the process-wide registry. Once lld registers them, every later parse in
the process accepts `-crel`, including clang's `-mllvm`, which then
ignores it.
Add a `parseCommandLineOptions` overload that takes the libraries a tool
opts in to. Replace `RegisterMCToolOptions` with `mc::getToolOptions`
and `codegen::getToolOptions`, which tools pass where they parse,
including lld's `-mllvm`, gold's `-plugin-opt`, and libLTO's debug
options. `RegisterCodeGenFlags` no longer registers the MC options. When
CodeGen's options move to TableGen, their parser joins
`codegen::getToolOptions` without changing these call sites.
clang-linker-wrapper forwards `-mllvm` to the device and host link steps
and does not pass the MC options. Its wrapper module uses the module's
own context instead of the user's `-mllvm`, as discussed on #233513.
[5 lines not shown]
BPF: Stop lowering unwinding through nonexistent unwinder calls
BPF borrowed the DWARF EH pipeline in an unsound way. DwarfEHPrepare
turned resume into a call of _Unwind_Resume, which will never be
linked in. BPFUnwindLower later replaced it, and calls of
_Unwind_RaiseException, by matching callee names after register
allocation. This fixes the pass pipeline relying on these transient
calls to phantom functions.
Instead add an invokable llvm.bpf.unwind.exit intrinsic, similar to
wasm's throw intrinsics. BPFUnwindPrepare replaces resume with it so
no impossible call to _Unwind_Resume reaches selection.
The intrinsic selects to a pseudo which BPFUnwindLower turns into
either unwind_exit or a branch to the landing pad when invoked. The
pseudo clobbers nothing, so register allocation can be more aggressive
than when the function looked like an ordinary call.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[Support] Pass tool options to parseCommandLineOptions
Follow-up to #239920: `RegisterMCToolOptions` adds `MCToolOptions` to
the process-wide registry. Once lld registers them, every later parse in
the process accepts `-crel`, including clang's `-mllvm`, which then
ignores it.
Add a `parseCommandLineOptions` overload that takes the libraries a tool
opts in to. Replace `RegisterMCToolOptions` with `mc::getToolOptions`
and `codegen::getToolOptions`, which tools pass where they parse,
including lld's `-mllvm`, gold's `-plugin-opt`, and libLTO's debug
options. `RegisterCodeGenFlags` no longer registers the MC options. When
CodeGen's options move to TableGen, their parser joins
`codegen::getToolOptions` without changing these call sites.
`LibraryOptionsContext` records the library that parsed each argument,
so `assign`, which runs on each ThinLTO backend thread, replays without
looking up the registry.
LLM-aided
[mlir][vector] Fix foldInsertUseChain to return getDest() (#239714)
Fixes a regression introduced in #238070, which changed `return
insertOp.getResult()` to `return insertOp.getDest()` in
`foldInsertUseChain`. `dest` is `InsertOp`'s destination vector rather
than its result.
Because of this change, folding `vector.insert` ops at the same position
replaced `insertOp` with its destination operand instead of signaling an
in-place fold, dropping the final insertion.
We also add a test to prevent this regression from happening again.
cc @banach-space
[Support] Pass tool options to parseCommandLineOptions
`RegisterMCToolOptions` adds `MCToolOptions` to the process-wide
registry. Once lld registers them, every later parse in the process
accepts `-crel`, including clang's `-mllvm`, which then ignores it.
Add a `parseCommandLineOptions` overload that takes the libraries a tool
opts in to. Replace `RegisterMCToolOptions` with `mc::getToolOptions`
and `codegen::getToolOptions`, which tools pass where they parse,
including lld's `-mllvm`, gold's `-plugin-opt`, and libLTO's debug
options. `RegisterCodeGenFlags` no longer registers the MC options.
`LibraryOptionsContext` records the library that parsed each argument,
so `assign` replays without looking up the registry, which does not
contain tool options.
LLM-aided
