[VPlan] Pull out reverses and splice.lefts from elementwise operations (#199234)
InstCombine pulls reverses up and out of operations, e.g.
`binop(reverse(x), reverse(y)) -> reverse(binop(x,y))`. This reduces the
overall number of reverses, and also allows the `reverse(reverse(x))`
combine to kick in much more.
This implements the same canonicalization in VPlan which allows for more
vectorization due to cost model improvements, and generally handles more
cases when there's predication involved.
If we have a reversed load and reversed store whose stores are now
eliminated, we will be left with just two reversed masks on the load and
store. But with EVL tail folding this will leave behind a
`splice.right(ops(splice.left(...)))` pair on the value from memory.
InstCombine can fold away a pair of `vp.reverse(ops(vp.reverse(...)))`,
but it can't fold a pair of splices. So to prevent regressions we also
have to pull splice.lefts like `ops(splice.left(poison, x, evl)) ->
[2 lines not shown]
[Clang] Fix missing vtable for `dynamic_cast<FinalClass &>(*this)` in a function template (#207349)
This is a follow-up to #202594, which fixed a pointer cast, but not
a reference cast. Surprisingly, `CXXDynamicCastExpr::getType()`
for a reference cast is a `RecordType` and not a `ReferenceType`.
How this happens:
In `Sema::BuildCXXNamedCast`, a `CastOperation Op` variable
is constructed. The `CastOperation` constructor initializes
`ResultType(destType.getNonLValueExprType(S.Context))`
where `QualType::getNonLValueExprType` turns a `ReferenceType` into
a `RecordType`. `Sema::BuildCXXNamedCast` then passes `Op.ResultType`
to `CXXDynamicCastExpr::Create`.
[VPlan] Introduce VPConstant VPIRValue (NFC) (#207387)
There a gap in the VPIRValue class hierarchy, where constant live-ins
are absent, when this is in fact a very common case. The motivation of
introducing this new class is to refine optimizations to account for the
fact that non-constant live-ins need broadcast.
[CodeGenPrepare] Cache known-live PHIs when deleting dead PHI chains (#207191)
This patch fixes a compile-time issue in CodeGenPrepare for huge
functions.
`DeleteDeadPHIs` may repeatedly prove overlapping PHI chains non-dead.
For very large functions, many PHIs can share the same non-dead def-use
suffix, causing the same suffix to be scanned many times.
Add an `KnownNonDeadPHIs` cache to `RecursivelyDeleteDeadPHINode`
and `DeleteDeadPHIs`. When a chain is proven non-dead, visited PHIs are
recorded so later queries can stop once they reach one of them.
This reduces the pathological CodeGenPrepare case from ~30mins to ~30s.
[Flang][MLIR][OpenMP] Move host op filtering to the omp dialect
The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.
By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.
As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
[6 lines not shown]
[Flang][MLIR][OpenMP] Fix declare_target globals visibility
This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
compiling for an OpenMP offload target. This prevents potential symbol
redefinition issues related to globals that don't actually exist on the
device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to
prevent their associated "internal" linkage from producing broken
device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
`declare_target link(...)`), the associated and unused full-storage
global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
external linkage when compiling for a target device, causing linker
errors if accessed. This mirrors Clang's behavior.
[AArch64] Add missing arithmetic to arith+cb(n)z clustering (#203721)
This patch adds a few missing opcodes for arithmetic+CB(N)Z clustering.
Most of them complement an already existing rr/rs variant for
pre/post-RA coverage. The only one which is completely new is ORN which
I think can be reasonably expected to behave similarly on AArch64
targets.
[Clang] support friend declarations with a dependent nested-name-specifier (#191268)
Fixes #104057
---
This patch adds support for friend declarations with a dependent NNS
[flang][Lower] Add alternative real expression lowering (#207371)
This is opt-in by an engineering option and disabled by default.
In section 10.1.5.2.4 of the 2023 Fortran standard "Evaluation of
numerical intrinsic operations", the standard explicitly allows
alternate mathematically equivalent lowerings. For example the source
expression X + Y + Z could be evaluated (X + Y) + Z, X + (Y + Z) or even
(X + Z) + Y, etc.
The open source benchmark SNBone shows significantly better results with
classic flang because classic flang emits real arithmetic expressions in
a different order. In the case of this benchmark it reduces dependency
depth for instructions issued to the vector unit, allowing for more of
the arithmetic to be parallelised over multiple vector execution units
in the ALU.
The lowering added by this patch tries to mimic the way classic flang
orders instructions for these expressions. I did not read any classic
[32 lines not shown]
[AArch64] Increase the relative cost of vector i64 multiply on Neoverse V3ae. (#207723)
The throughput of vector nxv2i64 multiplies on neoverse v3ae is 1/2, compared
to the throughput of 2 for integer multiplies. This large difference can mean
it is more profitable than normal to use scalar loops as opposed to vectorization.
This adds a subtarget feature that increases the cost multiple by 4 for 64bit
vector multiplies for specific CPUs. The cost model of llvm does not mean that
we can model throughputs correctly, but this should help. The same feature is
added to N2 as it has a similar difference between vector and scalar multiply
cost throughputs.
[VPlan] Forbid CSE'ing writes (NFC) (#207443)
CSE'ing two identical writes does not consider the fact that there could
be another write that writes an aliasing memory location. Fix the
potential miscompile. Note that there is currently no miscompile, as we
never remove a write, but the patch has the benefit of not processing
writes unnecessarily.
[SelectionDAG][AMDGPU] Preserve known bits for demoted sret pointers (#203468)
AMDGPU marks sret pointers with high-zero known bits so stores can be
folded into MUBUF base+offset addressing. Explicit sret arguments keep
this information through an AssertZext, but implicit sret lowering
passes the hidden return pointer through `DemoteRegister` as a
`CopyToReg/CopyFromReg` pair, where the fact is not visible to
SelectionDAG known-bits queries.
Add a `TargetLowering` hook for sret pointer known bits and use a shared
helper to materialize those bits as an `AssertZext` for both explicit
and demoted sret pointers.
Validated with llvm-test-depends, the AMDGPU function-returns test, and
the full llvm/test/CodeGen suite.
Assisted-by: Codex
[2 lines not shown]
[HashRecognize] Make `LHSAux` null if it is dead (#207231)
HashRecognize detects big-endian CRC loops with auxiliary data where the
bitwidth of `LHS` exceeds that of `LHSAux`. However, in this case,
`LHSAux` is zero-extended for the most significant bit check in each
iteration, and as such is effectively dead. Later optimization may even
miscompile in this case: for example, in the included
`crc16.be.tc8.zext.data` and `crc16.be.tc8.misalign` test cases,
`optimizeCRCLoop` emits `lshr i8 %crc.data.indexer, 8`, thereby creating
poison.
Since `LHSAux` is dead in this case, the user should receive a null
`LHSAux` instead.
[SDAG][X86] Support shrinking target-independent nodes (#206721)
X86 generally tries to shrink operations to work on smaller vector sizes
if possible. This happens in
SimplifyDemandedVectorEltsForTargetNode() for target-specific opcodes,
but it's currently not possible to do this for generic opcodes.
This introduces a getPreferredShrunkVectorSize() TLI hook to allow
shrinking generic ops based on demanded elements.
The primary motivation for this is to avoid regressions due to
https://github.com/llvm/llvm-project/pull/188489, which uplifts a
previously x86-specific node to become target-independent.
This PR enables the shrinking for fsub and fmul as examples.
Unfortunately, for many other ops we get various regressions, mostly
because horizontal operations are no longer formed.
[VectorCombine] Fold concat(binop(a,c), binop(b,d)) -> binop(concat(a,b), concat(c,d)) for i1 vectors (#206087)
Fixes #205707
`VectorCombine::foldShuffleOfBinops` rewrites `shuffle(binop(a,c),
binop(b,d))` into `binop(shuffle(a,b), shuffle(c,d))` but only when the
new form is strictly cheaper. For concat shuffles of i1 vectors, both
forms have equal cost, so the transform was silently rejected.
This fix allows the equal-cost transform when both binops are single-use
`BinaryOperators` (not icmps, to avoid widening narrow AVX-512
comparisons), canonicalising the IR to a wider binop.
The motivating case is AVX-512 mask operations: `NOT(concat(XOR(a,c),
XOR(b,d)))` becomes `NOT(XOR(concat(a,b), concat(c,d)))`, which the
backend naturally selects as `kxnorq`, reducing `kxord + kxord +
kunpckdq + knotq` (4 instructions) to `kunpckdq + kunpckdq + kxnorq` (3
instructions).
[AArch64] Fold any/sign-extend of CSET. (#207414)
This is useful to enable selecting i64 CSETM when legalisation places an
extend between a CSET and sext_inreg.
[IR][NFC] Make PassConcept immovable (#208167)
PassConcept is only ever created as a unique_ptr when added to the pass
list of a pass manager. The instances are never copied or moved. Due to
inheritance, this also wouldn't work anyway. Explicitly forbid
copying/moving and remove the dead functions.
Preliminary work for removing the vtable from PassConcept/PassModel.
