[IR][NFC] Add LoadStoreProperties to copy load/store attrs (#206470)
Introduce a small `LoadStoreProperties` struct plus get/setAttributes on
`LoadInst` and `StoreInst` so volatile/align/ordering/syncscope can be
copied together instead of one field at a time. Switch the obvious load->load
and store->store clone sites over to it.
[Debugify] Simplify debugify for locations (#207374)
This patch attempts to improve the performance of debugify for
locations, by relying on coverage tracking to replace most of the
functionality provided via the DILocations map, in exchange for losing
the ability to distinguish between "dropped" and "not-generated" bugs,
and requiring coverage tracking to determine when new bugs appear in a
pass instead of reporting the same bug repeatedly across passes.
This patch is not without cost; the justifications for using it are
that:
- Debugify locations is incredibly expensive; on a local build, without
using any coverage-tracking, this patch takes the build time for sqlite3
down from ~15 minutes to ~10 seconds.
- The difference between "dropped" and "not-generated" is a minor detail
of a bug - besides helping to determine the cause of the bug, which
origin-tracking can do with more accuracy, there's no fundamental
difference in the correctness of either. Furthermore, almost no
"dropped" bugs appear in the compiler anymore (since the debug location
[4 lines not shown]
[libc] Add a proxy header for sa_family_t (#207736)
This patch adds `hdr/types/sa_family_t.h` and updates socket tests and
helpers to use it instead of directly including `<sys/socket.h>` or
`include/llvm-libc-types/sa_family_t.h`.
The patch also adds a couple of includes of `hdr/types/socklen_t.h` for
files that are using the type, but not including it directly.
Assisted by Gemini.
[AArch64][GlobalISel] Use integer types in applySimplifyUADDO (#207962)
This avoids creating some scalar types in the IR, using integer types
for constants from a uaddo combine instead.
[DFAJumpThreading] Propagate DebugLocs to branches in select unfolding (#205851)
When DFAJumpThreading replaces a select with control flow, it generates
new blocks, new branches in those blocks, and potentially replaces the
branch in an existing block. Prior to this patch, none of these branches
were assigned debug locations; this patch replaces them as follows:
For the case where we generate two new blocks between the select block
and use block, and use a PHI of those blocks to replace the select, we
use the select's debug location for the branch instructions, since they
are doing the work of the select.
For the case where we generate one new block and replace the
unconditional branch from the select block with a conditional branch to
the new block and the use block, we treat the new branches as replacing
both the select and the original branch, so each branch takes the merged
location of the original select+br.
This patch also ensures that when we create a new path which would end
[2 lines not shown]
[InstCombine] Fold zext(sub(0, trunc(X))) to and(sub(0, X), mask) (#207564)
Problem: vector rotate and funnel shift fails to fold for vectors > 16
bytes on AVX-512. This is because of the `trunc` and `zext`
instructions.
Example:
```llvm
define dso_local <8 x i64> @baz(<8 x i64> %0, <8 x i64> %1) local_unnamed_addr {
Entry:
%2 = trunc <8 x i64> %1 to <8 x i6>
%3 = sub <8 x i6> zeroinitializer, %2
%4 = zext <8 x i6> %3 to <8 x i64>
%5 = shl <8 x i64> %0, %4
%6 = and <8 x i64> %1, splat (i64 63)
%7 = lshr <8 x i64> %0, %6
%8 = or <8 x i64> %5, %7
ret <8 x i64> %8
}
[5 lines not shown]
[CodeGen][NFC] Remove stray SDag BFI computation with NewPM (#208181)
This looks like an accident -- there's no need to compute the
BlockFrequencyInfo unconditionally and then discarding it. After this,
enabling the NewPM CodeGen pipeline is faster than the legacy PM.
[IR] Preserve !nofpclass in dropUBImplyingAttrsAndMetadata (#208186)
`!nofpclass` is a poison-generating metadata kind, so it should be
preserved by dropUBImplyingAttrsAndMetadata().
[VPlan] Pull out reverses and splice.lefts from elementwise operations (#199234)
InstCombine pulls reverses up and out of operations, e.g.
`binop(reverse(x), reverse(y)) -> reverse(binop(x,y))`. This reduces the
overall number of reverses, and also allows the `reverse(reverse(x))`
combine to kick in much more.
This implements the same canonicalization in VPlan which allows for more
vectorization due to cost model improvements, and generally handles more
cases when there's predication involved.
If we have a reversed load and reversed store whose stores are now
eliminated, we will be left with just two reversed masks on the load and
store. But with EVL tail folding this will leave behind a
`splice.right(ops(splice.left(...)))` pair on the value from memory.
InstCombine can fold away a pair of `vp.reverse(ops(vp.reverse(...)))`,
but it can't fold a pair of splices. So to prevent regressions we also
have to pull splice.lefts like `ops(splice.left(poison, x, evl)) ->
[2 lines not shown]
[Clang] Fix missing vtable for `dynamic_cast<FinalClass &>(*this)` in a function template (#207349)
This is a follow-up to #202594, which fixed a pointer cast, but not
a reference cast. Surprisingly, `CXXDynamicCastExpr::getType()`
for a reference cast is a `RecordType` and not a `ReferenceType`.
How this happens:
In `Sema::BuildCXXNamedCast`, a `CastOperation Op` variable
is constructed. The `CastOperation` constructor initializes
`ResultType(destType.getNonLValueExprType(S.Context))`
where `QualType::getNonLValueExprType` turns a `ReferenceType` into
a `RecordType`. `Sema::BuildCXXNamedCast` then passes `Op.ResultType`
to `CXXDynamicCastExpr::Create`.
[VPlan] Introduce VPConstant VPIRValue (NFC) (#207387)
There a gap in the VPIRValue class hierarchy, where constant live-ins
are absent, when this is in fact a very common case. The motivation of
introducing this new class is to refine optimizations to account for the
fact that non-constant live-ins need broadcast.
[CodeGenPrepare] Cache known-live PHIs when deleting dead PHI chains (#207191)
This patch fixes a compile-time issue in CodeGenPrepare for huge
functions.
`DeleteDeadPHIs` may repeatedly prove overlapping PHI chains non-dead.
For very large functions, many PHIs can share the same non-dead def-use
suffix, causing the same suffix to be scanned many times.
Add an `KnownNonDeadPHIs` cache to `RecursivelyDeleteDeadPHINode`
and `DeleteDeadPHIs`. When a chain is proven non-dead, visited PHIs are
recorded so later queries can stop once they reach one of them.
This reduces the pathological CodeGenPrepare case from ~30mins to ~30s.
[Flang][MLIR][OpenMP] Move host op filtering to the omp dialect
The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.
By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.
As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
[6 lines not shown]
[Flang][MLIR][OpenMP] Fix declare_target globals visibility
This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
compiling for an OpenMP offload target. This prevents potential symbol
redefinition issues related to globals that don't actually exist on the
device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to
prevent their associated "internal" linkage from producing broken
device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
`declare_target link(...)`), the associated and unused full-storage
global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
external linkage when compiling for a target device, causing linker
errors if accessed. This mirrors Clang's behavior.
[AArch64] Add missing arithmetic to arith+cb(n)z clustering (#203721)
This patch adds a few missing opcodes for arithmetic+CB(N)Z clustering.
Most of them complement an already existing rr/rs variant for
pre/post-RA coverage. The only one which is completely new is ORN which
I think can be reasonably expected to behave similarly on AArch64
targets.
[Clang] support friend declarations with a dependent nested-name-specifier (#191268)
Fixes #104057
---
This patch adds support for friend declarations with a dependent NNS
[flang][Lower] Add alternative real expression lowering (#207371)
This is opt-in by an engineering option and disabled by default.
In section 10.1.5.2.4 of the 2023 Fortran standard "Evaluation of
numerical intrinsic operations", the standard explicitly allows
alternate mathematically equivalent lowerings. For example the source
expression X + Y + Z could be evaluated (X + Y) + Z, X + (Y + Z) or even
(X + Z) + Y, etc.
The open source benchmark SNBone shows significantly better results with
classic flang because classic flang emits real arithmetic expressions in
a different order. In the case of this benchmark it reduces dependency
depth for instructions issued to the vector unit, allowing for more of
the arithmetic to be parallelised over multiple vector execution units
in the ALU.
The lowering added by this patch tries to mimic the way classic flang
orders instructions for these expressions. I did not read any classic
[32 lines not shown]
[AArch64] Increase the relative cost of vector i64 multiply on Neoverse V3ae. (#207723)
The throughput of vector nxv2i64 multiplies on neoverse v3ae is 1/2, compared
to the throughput of 2 for integer multiplies. This large difference can mean
it is more profitable than normal to use scalar loops as opposed to vectorization.
This adds a subtarget feature that increases the cost multiple by 4 for 64bit
vector multiplies for specific CPUs. The cost model of llvm does not mean that
we can model throughputs correctly, but this should help. The same feature is
added to N2 as it has a similar difference between vector and scalar multiply
cost throughputs.
[VPlan] Forbid CSE'ing writes (NFC) (#207443)
CSE'ing two identical writes does not consider the fact that there could
be another write that writes an aliasing memory location. Fix the
potential miscompile. Note that there is currently no miscompile, as we
never remove a write, but the patch has the benefit of not processing
writes unnecessarily.
