[SandboxVectorizer] Dispatch LoadStoreVec::runOnRegion on seed kind
runOnRegion() previously assumed its seed slice was always a store chain,
unconditionally casting Bndl[0] to StoreInst. This crashed (assertion in
areConsecutive<StoreInst>) whenever -sbvec-collect-seeds included "loads",
since a load-seeded region's Aux holds LoadInsts.
Add a symmetric top-level path for load-kind seed slices: createVectorLoad()
builds the vector load, and vectorizeLoads() -- unlike a load that merely
feeds a store -- has to handle arbitrary uses, so it replaces each original
load with an extract from the vector load (VecUtils::unpack()) rather than
just discarding it. runOnRegion() determines the seed kind from Bndl[0]
(asserting the slice is homogeneous, which SeedCollection guarantees) and
dispatches to vectorizeStores()/vectorizeLoads() accordingly.
No sub-run search yet: vectorizeStores()/vectorizeLoads() are each still
tried once over the whole seed slice, same as before this commit for
stores. Sub-bundle partitioning for both kinds is a separate follow-up.
[13 lines not shown]
[llc] Respect shouldDefaultToNewPM
This makes llc respect the new TM flag that allows targets to opt in to
defaulting to the NewPM.
Reviewers: vikramRH, aengelke, arsenm
Pull Request: https://github.com/llvm/llvm-project/pull/214572
[Clang] Make NewPM switch respect shouldDefaultToNewPM
shouldDefaultToNewPM is a new target flag that allows targets to specify
that the NewPM should be used by default for compilation. Wire it up
into clang.
Reviewers: arsenm, efriedma-quic, jansvoboda11
Pull Request: https://github.com/llvm/llvm-project/pull/214571
[RISCV][P-ext] Select scalar mulhr/mulhru/mulhrsu for RV32 v2i32 (#215938)
The `combinePExtTruncate` DAGCombine bailed out on RV32 for both
`v4i16` and `v2i32` 64-bit packed types. The `v4i16` case is correct
(no paired rounding multiply-high for 16-bit lanes), but `v2i32` was
incorrectly excluded.
RV32 provides scalar `mulhr`/`mulhru`/`mulhrsu` instructions. Since
`v2i32` on RV32 is a GPRPair, splitting into two scalar operations in
the combine — while the widening multiply shape is still visible —
reuses those instructions directly.
The non-rounding forms (`mulh`/`mulhu`/`mulhsu`) already scalarize
correctly through the generic legalizer, so only the rounding case
needs explicit handling here.
[SandboxVectorizer] Vectorize partial store sub-bundles in LoadStoreVec
runOnRegion() previously required an entire store seed chain to vectorize
as one unit. A seed slice can legitimately fail that as a whole while a
sub-run within it is still fine, e.g. because it spans an address gap
(SeedBundle::getSlice sorts by address but doesn't guarantee contiguity)
or a sub-range fails to schedule. Add findLegalStoreRun()/isLegalStoreRun()
to search for the longest vectorizable run starting at a given position,
and have runOnRegion() call vectorizeStores() once per such run instead of
once for the whole chain. isLegalStoreRun() is purely an address/scheduling
check now -- no operand-eligibility check is needed since packOperands()
accepts any operand kind.
The search only ever shrinks a candidate length, never grows one:
Scheduler::trySchedule() permanently commits a successful multi-instruction
bundle (a later request that overlaps it and is not an exact match returns
AlreadyScheduled and fails), so starting from the longest candidate and
shrinking on failure is the only search order compatible with the
scheduler's contract.
[18 lines not shown]
[SandboxVectorizer] Make LoadStoreVec::vectorizeStores direction-agnostic
Remove the AllLoads/AllConstants operand-kind gate: vectorizeStores() no
longer requires a store chain's value operands to be all loads, all
constants, or neither. Instead it always builds the vector value via
packOperands(), which packs any mix of loads, constants, or arbitrary SSA
values via extractelement/insertelement -- direction-agnostic in the sense
that it doesn't care what kind of operand it's given, unlike the
load-specific and constant-specific paths it replaces.
vectorizeLoads()/createVectorLoad(), added by the previous commit for
load-kind seed slices, are unaffected: packOperands() only replaces
vectorizeStores()'s old all-loads fast path, which duplicated the same
vector-load construction createVectorLoad() already does.
Update load_store_vec.ll, load_store_vec_mixed_types.ll, and
AMDGPU/basic.ll for the new pack-based codegen.
check-llvm Transforms/SandboxVectorizer passes (31/31).
[SandboxVectorizer] Make tryEraseDeadInstrs tolerate multi-use load operands
vectorizeStores() currently only ever passes single-use load operands
to tryEraseDeadInstrs(), since isFoldableLoadOperand() guarantees this.
This is NFC today, but is prep for a follow-up commit that switches
vectorizeStores() over to packOperands(), which does not make that
guarantee.
[SandboxVectorizer] Add a direction-agnostic operand packer to LoadStoreVec
Add packOperands(), which packs a list of arbitrary Values into a single
vector value via extractelement/insertelement, regardless of what kind
of Value each one is (load, constant, or any other SSA value) -- unlike
vectorizeStores()'s current load-specific and constant-specific paths,
which require every operand to be uniformly one kind or the other.
Operands are combined at the granularity of their narrowest common
scalar element type (getCombinedElementType(), the same rule
VecUtils::getCombinedVectorTypeFor() uses for a mixed-type instruction
chain, generalized to arbitrary Values rather than only Instructions).
An operand wider than that granularity is split into multiple lanes via
a bitcast, matching how a vector-typed element is split into per-lane
extracts.
reinterpretSameWidth() picks bitcast, ptrtoint, or inttoptr as needed to
reinterpret an operand at the combined granularity: a plain bitcast can't
convert between pointer and non-pointer types, and inttoptr requires an
[9 lines not shown]
[SandboxVectorizer] Dispatch LoadStoreVec::runOnRegion on seed kind
runOnRegion() previously assumed its seed slice was always a store chain,
unconditionally casting Bndl[0] to StoreInst. This crashed (assertion in
areConsecutive<StoreInst>) whenever -sbvec-collect-seeds included "loads",
since a load-seeded region's Aux holds LoadInsts.
Add a symmetric top-level path for load-kind seed slices: createVectorLoad()
builds the vector load, and vectorizeLoads() -- unlike a load that merely
feeds a store -- has to handle arbitrary uses, so it replaces each original
load with an extract from the vector load (VecUtils::unpack()) rather than
just discarding it. runOnRegion() determines the seed kind from Bndl[0]
(asserting the slice is homogeneous, which SeedCollection guarantees) and
dispatches to vectorizeStores()/vectorizeLoads() accordingly.
No sub-run search yet: vectorizeStores()/vectorizeLoads() are each still
tried once over the whole seed slice, same as before this commit for
stores. Sub-bundle partitioning for both kinds is a separate follow-up.
[6 lines not shown]
[CodeGen] Derive the regalloc pipeline from the allocator, remove -optimize-regalloc (#215740)
-regalloc selects the allocator and -optimize-regalloc independently
selects the pipeline, but only two combinations are meaningful: a
non-fast allocator with -optimize-regalloc=0 is a fatal error, and
-regalloc=fast with the optimized pipeline runs the full preparation
just to rewrite a vacuous VirtRegMap.
Derive the pipeline from the allocator instead: an explicit -regalloc
implies its pipeline; the default follows the optimization level.
-regalloc=greedy now works at -O0 and -regalloc=fast always uses the
fast pipeline. Update tests to name the allocator; regenerate three
that checked the hybrid output.
Aided by Fable 5
[SandboxVectorizer] Dispatch LoadStoreVec::runOnRegion on seed kind
runOnRegion() previously assumed its seed slice was always a store chain,
unconditionally casting Bndl[0] to StoreInst. This crashed (assertion in
areConsecutive<StoreInst>) whenever -sbvec-collect-seeds included "loads",
since a load-seeded region's Aux holds LoadInsts.
Add a symmetric top-level path for load-kind seed slices: createVectorLoad()
builds the vector load, and vectorizeLoads() -- unlike a load that merely
feeds a store -- has to handle arbitrary uses, so it replaces each original
load with an extract from the vector load (VecUtils::unpack()) rather than
just discarding it. runOnRegion() determines the seed kind from Bndl[0]
(asserting the slice is homogeneous, which SeedCollection guarantees) and
dispatches to vectorizeStores()/vectorizeLoads() accordingly.
No sub-run search yet: vectorizeStores()/vectorizeLoads() are each still
tried once over the whole seed slice, same as before this commit for
stores. Sub-bundle partitioning for both kinds is a separate follow-up.
[6 lines not shown]
[Profcheck] Enable MergeFunc tests (#215881)
Remove MergeFunc from the profcheck exclusion list.
All MergeFunc tests pass with LLVM_ENABLE_PROFCHECK enabled, so no
expected-failure entries are needed.
[ADT][docs] Document SortedVectorMap in ProgrammersManual (NFC) (#215930)
This patch adds a section for SortedVectorMap in the Programmer's
Manual, detailing its intended use cases (small maps, minimal memory
overhead vs DenseMap, iteration in sorted key order) and trade-offs
(O(log N) binary search lookup and O(N) insertion/deletion vs O(1) in
DenseMap).
We also add a cross-reference from the existing "A sorted 'vector'"
section.
[BOLT] Link pthread in Profile and Target libs to fix undefined pthread_rwlock_* symbols (#215748)
MCPlusBuilder.h uses llvm::sys::RWMutex, which aliases std::shared_mutex
on non-Apple platforms. Its inline lock/unlock calls pthread_rwlock_*
directly, so any shared library that instantiates these inline functions
must link pthread explicitly under -Wl,-z,defs.
This only shows up when building with -DBUILD_SHARED_LIBS=ON. With
static libs, the missing symbols get resolved elsewhere on the final
link line, so the problem stays hidden.
LLVMBOLTCore, LLVMBOLTRewrite, LLVMBOLTPasses, and LLVMBOLTUtils already
link ${LLVM_PTHREAD_LIB}; LLVMBOLTProfile and the LLVMBOLTTarget{X86,
AArch64,RISCV} libs were missing it, causing link failures.
[ORC] Move EPCGenericJITLinkMemoryManager to RTBridge proxies (#215797)
Reimplement EPCGenericJITLinkMemoryManager's reserve/initialize/release
calls using rt::Proxy objects rather than direct
ExecutorProcessControl::callSPSWrapperAsync calls, matching
EPCGenericMemoryAccess and EPCGenericDylibManager. The manager's
behavior is unchanged.
Details:
* SymbolAddrs (five ExecutorAddrs) becomes Bindings: the allocator
instance address plus rt::Proxy handles.
* The proxy types and SPS specs are hoisted into shared headers --
RTBridge/GenericMemoryManagerProxies.h and
RTBridge/SPS/GenericMemoryManagerProxySpecs.h -- since they now need
only Shared/SPS vocabulary. Deinitialize is included in the family for
completeness though this manager does not call it.
[9 lines not shown]
