[VPlan] Pull out reverses and splice.lefts from elementwise operations (#199234)
InstCombine pulls reverses up and out of operations, e.g.
`binop(reverse(x), reverse(y)) -> reverse(binop(x,y))`. This reduces the
overall number of reverses, and also allows the `reverse(reverse(x))`
combine to kick in much more.
This implements the same canonicalization in VPlan which allows for more
vectorization due to cost model improvements, and generally handles more
cases when there's predication involved.
If we have a reversed load and reversed store whose stores are now
eliminated, we will be left with just two reversed masks on the load and
store. But with EVL tail folding this will leave behind a
`splice.right(ops(splice.left(...)))` pair on the value from memory.
InstCombine can fold away a pair of `vp.reverse(ops(vp.reverse(...)))`,
but it can't fold a pair of splices. So to prevent regressions we also
have to pull splice.lefts like `ops(splice.left(poison, x, evl)) ->
[2 lines not shown]
[Clang] Fix missing vtable for `dynamic_cast<FinalClass &>(*this)` in a function template (#207349)
This is a follow-up to #202594, which fixed a pointer cast, but not
a reference cast. Surprisingly, `CXXDynamicCastExpr::getType()`
for a reference cast is a `RecordType` and not a `ReferenceType`.
How this happens:
In `Sema::BuildCXXNamedCast`, a `CastOperation Op` variable
is constructed. The `CastOperation` constructor initializes
`ResultType(destType.getNonLValueExprType(S.Context))`
where `QualType::getNonLValueExprType` turns a `ReferenceType` into
a `RecordType`. `Sema::BuildCXXNamedCast` then passes `Op.ResultType`
to `CXXDynamicCastExpr::Create`.
[VPlan] Introduce VPConstant VPIRValue (NFC) (#207387)
There a gap in the VPIRValue class hierarchy, where constant live-ins
are absent, when this is in fact a very common case. The motivation of
introducing this new class is to refine optimizations to account for the
fact that non-constant live-ins need broadcast.
NAS-141701 / 27.0.0-BETA.1 / Split boot environment service into its own typesafe plugin (#19276)
## Context
The boot.environment service lived inside the boot plugin group
(boot_/environments.py) as a legacy dict-based Service with a
hand-written filterable query and dict-returning actions. It is
CRUD-shaped but datastore-less (query/get_instance read live ZFS via
zectl/zpool), keyed by a string BE name, with custom write actions
rather than create/update/delete.
## Solution
Moved it to a new standalone plugin (plugins/boot_environment/)
converted to the typesafe pattern: a lean
GenericCRUDService[BootEnvironmentEntry, str] delegating to a
BootEnvironmentServicePart whose query/get_instance are overridden to
build BootEnvironmentEntry models directly (no datastore).
activate/clone/destroy/keep are @api_method(check_annotations=True)
actions and promote_current_datasets stays a private boot-time hook. The
service registers under a new BootServicesContainer at self.boot, which
[17 lines not shown]
[CodeGenPrepare] Cache known-live PHIs when deleting dead PHI chains (#207191)
This patch fixes a compile-time issue in CodeGenPrepare for huge
functions.
`DeleteDeadPHIs` may repeatedly prove overlapping PHI chains non-dead.
For very large functions, many PHIs can share the same non-dead def-use
suffix, causing the same suffix to be scanned many times.
Add an `KnownNonDeadPHIs` cache to `RecursivelyDeleteDeadPHINode`
and `DeleteDeadPHIs`. When a chain is proven non-dead, visited PHIs are
recorded so later queries can stop once they reach one of them.
This reduces the pathological CodeGenPrepare case from ~30mins to ~30s.
Fix `replication.target_unmatched_snapshots` crash due to invalid result schema (#19275)
Additionally, reach 99% tests coverage for replication plugin.
[Flang][MLIR][OpenMP] Move host op filtering to the omp dialect
The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.
By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.
As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
[6 lines not shown]
[Flang][MLIR][OpenMP] Fix declare_target globals visibility
This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
compiling for an OpenMP offload target. This prevents potential symbol
redefinition issues related to globals that don't actually exist on the
device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to
prevent their associated "internal" linkage from producing broken
device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
`declare_target link(...)`), the associated and unused full-storage
global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
external linkage when compiling for a target device, causing linker
errors if accessed. This mirrors Clang's behavior.
[AArch64] Add missing arithmetic to arith+cb(n)z clustering (#203721)
This patch adds a few missing opcodes for arithmetic+CB(N)Z clustering.
Most of them complement an already existing rr/rs variant for
pre/post-RA coverage. The only one which is completely new is ORN which
I think can be reasonably expected to behave similarly on AArch64
targets.
[Clang] support friend declarations with a dependent nested-name-specifier (#191268)
Fixes #104057
---
This patch adds support for friend declarations with a dependent NNS
[flang][Lower] Add alternative real expression lowering (#207371)
This is opt-in by an engineering option and disabled by default.
In section 10.1.5.2.4 of the 2023 Fortran standard "Evaluation of
numerical intrinsic operations", the standard explicitly allows
alternate mathematically equivalent lowerings. For example the source
expression X + Y + Z could be evaluated (X + Y) + Z, X + (Y + Z) or even
(X + Z) + Y, etc.
The open source benchmark SNBone shows significantly better results with
classic flang because classic flang emits real arithmetic expressions in
a different order. In the case of this benchmark it reduces dependency
depth for instructions issued to the vector unit, allowing for more of
the arithmetic to be parallelised over multiple vector execution units
in the ALU.
The lowering added by this patch tries to mimic the way classic flang
orders instructions for these expressions. I did not read any classic
[32 lines not shown]
firmware: cleansing using output_cmd is futile
Move the code back to where it was in early 26.1. A number of
regressions could have been avoided. The last straw was buffering
of "." characters for fetching.
The read-guard is still effective. As it reads the file it does
not need to run unbuffered. GUI reads are still properly filtered.
[AArch64] Increase the relative cost of vector i64 multiply on Neoverse V3ae. (#207723)
The throughput of vector nxv2i64 multiplies on neoverse v3ae is 1/2, compared
to the throughput of 2 for integer multiplies. This large difference can mean
it is more profitable than normal to use scalar loops as opposed to vectorization.
This adds a subtarget feature that increases the cost multiple by 4 for 64bit
vector multiplies for specific CPUs. The cost model of llvm does not mean that
we can model throughputs correctly, but this should help. The same feature is
added to N2 as it has a similar difference between vector and scalar multiply
cost throughputs.
[VPlan] Forbid CSE'ing writes (NFC) (#207443)
CSE'ing two identical writes does not consider the fact that there could
be another write that writes an aliasing memory location. Fix the
potential miscompile. Note that there is currently no miscompile, as we
never remove a write, but the patch has the benefit of not processing
writes unnecessarily.
[SelectionDAG][AMDGPU] Preserve known bits for demoted sret pointers (#203468)
AMDGPU marks sret pointers with high-zero known bits so stores can be
folded into MUBUF base+offset addressing. Explicit sret arguments keep
this information through an AssertZext, but implicit sret lowering
passes the hidden return pointer through `DemoteRegister` as a
`CopyToReg/CopyFromReg` pair, where the fact is not visible to
SelectionDAG known-bits queries.
Add a `TargetLowering` hook for sret pointer known bits and use a shared
helper to materialize those bits as an `AssertZext` for both explicit
and demoted sret pointers.
Validated with llvm-test-depends, the AMDGPU function-returns test, and
the full llvm/test/CodeGen suite.
Assisted-by: Codex
[2 lines not shown]
[HashRecognize] Make `LHSAux` null if it is dead (#207231)
HashRecognize detects big-endian CRC loops with auxiliary data where the
bitwidth of `LHS` exceeds that of `LHSAux`. However, in this case,
`LHSAux` is zero-extended for the most significant bit check in each
iteration, and as such is effectively dead. Later optimization may even
miscompile in this case: for example, in the included
`crc16.be.tc8.zext.data` and `crc16.be.tc8.misalign` test cases,
`optimizeCRCLoop` emits `lshr i8 %crc.data.indexer, 8`, thereby creating
poison.
Since `LHSAux` is dead in this case, the user should receive a null
`LHSAux` instead.
