[SDAG][X86] Uplift pseudo fmin/fmax from X86 (#188489)
X86 has pseudo fmin/fmax operations that implement `x olt y ? x : y` and
`x ogt y ? x : y` respectively. There are other targets that also
support these operations, including wasm and s390x.
On wasm (where I've adopted the "pseudo" terminology from), we currently
create these ops, but incorrectly (wrt signed zero handling). On s390x
we don't, but should to avoid regressions from fixing incorrect
minnum/maxnum formation.
I figured it would make sense to uplift the existing support form X86,
so other targets can benefit from it by just marking the operations
legal.
[Flang][OpenMP] Correct ArrayElements in Reduction Clause (#196094)
Currently, when an ArrayElement is used within a Reduction clause, it
will be lowered with the reduction referencing the box containing the
array, not just the element.
To address this, adjust Flang lowering to track expressions alongside
symbol to ensure that the Array Element context is not lost and
considered when lowering a reduction with Array Element. This ensures
that, when represented in HLFIR, it will be just the element's type,
rather than the full array.
Currently this excludes DO CONCURRENT as it excludes Array Elements, and
is limited to Array Elements but there are options to expand this into
Array Sections in the future.
Assisted-by: Codex
[offload] Add cuda to the PATH for nvptx tests (#208101)
To execute the JIT tests, ptxas needs to be in PATH. This also makes
sure we use ptxas from the same cuda path that we use when compiling.
[lld][ELF][test] Accept zlib-ng compressed size (#206880)
zlib-ng 2.3.2 deflates slightly differently from zlib; the level-1
default is 0x20, just outside the LEVEL1 pattern. Widen it to accept
0x20.
Assisted-by: claude-opus
[LinkerWrapper] Fix AMDGPU Target-IDs linking in improper order (#207853)
Summary:
These target IDs are supposed to be ordered from most to least specific,
but we had no such ordering. The changes basically sort the input from
least to most specific using the target-id presence, then ensures that
their entries are listed first.
Fixes: https://github.com/llvm/llvm-project/issues/207835
[libc++] Implement LWG3918: copy elision in `std::uninitialized_move/_n` (#207692)
This implements [LWG3918](https://wg21.link/LWG3918), which guarantees
copy elision for rvalues in `std::uninitialized_move/_n`. It also
implements [LWG4452](https://wg21.link/LWG4452), a minor correction that
makes the helper added by LWG3918 constexpr.
This additionally fixes a bug in `std::uninitialized_move/_n` where they
could create and then access dangling references. The previous
implementation used the following lambda as an implementation detail:
```c++
[](auto&& __iter) -> decltype(auto) { return std::move(*__iter); }
```
When `__iter` is a prvalue, this creates a temporary object within the
lambda's body and `std::move` then returns a reference to that object.
The reference dangles as soon as control leaves the lambda. This
behavior was not permitted by the standard even before LWG3918.
The new implementation fixes the bug by using a helper,
[4 lines not shown]
[X86] Add vXi64 MULHU/MULHS lowering, keeping full-width products scalar (#206983)
Based on the discussion in the PR #169819.
This lands the unsigned (`MULHU`) and signed (`MULHS`)
`vXi64` high-multiply lowering, plus a guard so full 128-bit products
stay scalar.
### What this does
1. **Lower `ISD::MULHU` for `v4i64`/`v8i64`** via `forceExpandMultiply`
(a `vpmuludq` schoolbook), as in #169819.
2. **Lower `ISD::MULHS` for `v4i64`/`v8i64`, gated on AVX512DQ.** The
signed low multiply is `vpmullq`, so `MULHS` is only marked `Custom`
when DQ (and VL for `v4i64`) is available.
3. **Guard the full-width-product case.** When the *low* half of the
product is also used (a `wyhash`-style `lo ^ hi`), vectorizing just the
high half is redundant with the low half's wide multiply. So, this guard
let a single scalar multiply per lane yield both halves, likewise the
unpatched target already did.
[31 lines not shown]
[clang][ItaniumMangle] Fix mangling of lambdas in default member initializers of local classes (#206740)
Lambdas appearing in default member initializers of members of local
classes were previously mangled as if they belonged only to the class
scope, ignoring the enclosing function-local context.
This caused different (but same-named) local classes to produce
identical closure type mangling and corresponding RTTI/IR collisions.
Consider the following example:
```
void foo() {
{
struct T {
std::function<void()> a = [](){ std::cout << "a"; };
} t;
t.a();
}
{
[12 lines not shown]
[libc] Add a differential fuzzer for inet_ntop (#207977)
The first byte of the input is used to select the address class and the
size of the output buffer. The rest is used as the input.
We compare the results and also check that our implementation does not
overflow the buffer.
Assisted by Gemini.
[IR][NFC] Add LoadStoreProperties to copy load/store attrs (#206470)
Introduce a small `LoadStoreProperties` struct plus get/setAttributes on
`LoadInst` and `StoreInst` so volatile/align/ordering/syncscope can be
copied together instead of one field at a time. Switch the obvious load->load
and store->store clone sites over to it.
[Debugify] Simplify debugify for locations (#207374)
This patch attempts to improve the performance of debugify for
locations, by relying on coverage tracking to replace most of the
functionality provided via the DILocations map, in exchange for losing
the ability to distinguish between "dropped" and "not-generated" bugs,
and requiring coverage tracking to determine when new bugs appear in a
pass instead of reporting the same bug repeatedly across passes.
This patch is not without cost; the justifications for using it are
that:
- Debugify locations is incredibly expensive; on a local build, without
using any coverage-tracking, this patch takes the build time for sqlite3
down from ~15 minutes to ~10 seconds.
- The difference between "dropped" and "not-generated" is a minor detail
of a bug - besides helping to determine the cause of the bug, which
origin-tracking can do with more accuracy, there's no fundamental
difference in the correctness of either. Furthermore, almost no
"dropped" bugs appear in the compiler anymore (since the debug location
[4 lines not shown]
[libc] Add a proxy header for sa_family_t (#207736)
This patch adds `hdr/types/sa_family_t.h` and updates socket tests and
helpers to use it instead of directly including `<sys/socket.h>` or
`include/llvm-libc-types/sa_family_t.h`.
The patch also adds a couple of includes of `hdr/types/socklen_t.h` for
files that are using the type, but not including it directly.
Assisted by Gemini.
[AArch64][GlobalISel] Use integer types in applySimplifyUADDO (#207962)
This avoids creating some scalar types in the IR, using integer types
for constants from a uaddo combine instead.
[DFAJumpThreading] Propagate DebugLocs to branches in select unfolding (#205851)
When DFAJumpThreading replaces a select with control flow, it generates
new blocks, new branches in those blocks, and potentially replaces the
branch in an existing block. Prior to this patch, none of these branches
were assigned debug locations; this patch replaces them as follows:
For the case where we generate two new blocks between the select block
and use block, and use a PHI of those blocks to replace the select, we
use the select's debug location for the branch instructions, since they
are doing the work of the select.
For the case where we generate one new block and replace the
unconditional branch from the select block with a conditional branch to
the new block and the use block, we treat the new branches as replacing
both the select and the original branch, so each branch takes the merged
location of the original select+br.
This patch also ensures that when we create a new path which would end
[2 lines not shown]
[InstCombine] Fold zext(sub(0, trunc(X))) to and(sub(0, X), mask) (#207564)
Problem: vector rotate and funnel shift fails to fold for vectors > 16
bytes on AVX-512. This is because of the `trunc` and `zext`
instructions.
Example:
```llvm
define dso_local <8 x i64> @baz(<8 x i64> %0, <8 x i64> %1) local_unnamed_addr {
Entry:
%2 = trunc <8 x i64> %1 to <8 x i6>
%3 = sub <8 x i6> zeroinitializer, %2
%4 = zext <8 x i6> %3 to <8 x i64>
%5 = shl <8 x i64> %0, %4
%6 = and <8 x i64> %1, splat (i64 63)
%7 = lshr <8 x i64> %0, %6
%8 = or <8 x i64> %5, %7
ret <8 x i64> %8
}
[5 lines not shown]
[CodeGen][NFC] Remove stray SDag BFI computation with NewPM (#208181)
This looks like an accident -- there's no need to compute the
BlockFrequencyInfo unconditionally and then discarding it. After this,
enabling the NewPM CodeGen pipeline is faster than the legacy PM.
[IR] Preserve !nofpclass in dropUBImplyingAttrsAndMetadata (#208186)
`!nofpclass` is a poison-generating metadata kind, so it should be
preserved by dropUBImplyingAttrsAndMetadata().
[VPlan] Pull out reverses and splice.lefts from elementwise operations (#199234)
InstCombine pulls reverses up and out of operations, e.g.
`binop(reverse(x), reverse(y)) -> reverse(binop(x,y))`. This reduces the
overall number of reverses, and also allows the `reverse(reverse(x))`
combine to kick in much more.
This implements the same canonicalization in VPlan which allows for more
vectorization due to cost model improvements, and generally handles more
cases when there's predication involved.
If we have a reversed load and reversed store whose stores are now
eliminated, we will be left with just two reversed masks on the load and
store. But with EVL tail folding this will leave behind a
`splice.right(ops(splice.left(...)))` pair on the value from memory.
InstCombine can fold away a pair of `vp.reverse(ops(vp.reverse(...)))`,
but it can't fold a pair of splices. So to prevent regressions we also
have to pull splice.lefts like `ops(splice.left(poison, x, evl)) ->
[2 lines not shown]
[Clang] Fix missing vtable for `dynamic_cast<FinalClass &>(*this)` in a function template (#207349)
This is a follow-up to #202594, which fixed a pointer cast, but not
a reference cast. Surprisingly, `CXXDynamicCastExpr::getType()`
for a reference cast is a `RecordType` and not a `ReferenceType`.
How this happens:
In `Sema::BuildCXXNamedCast`, a `CastOperation Op` variable
is constructed. The `CastOperation` constructor initializes
`ResultType(destType.getNonLValueExprType(S.Context))`
where `QualType::getNonLValueExprType` turns a `ReferenceType` into
a `RecordType`. `Sema::BuildCXXNamedCast` then passes `Op.ResultType`
to `CXXDynamicCastExpr::Create`.
[VPlan] Introduce VPConstant VPIRValue (NFC) (#207387)
There a gap in the VPIRValue class hierarchy, where constant live-ins
are absent, when this is in fact a very common case. The motivation of
introducing this new class is to refine optimizations to account for the
fact that non-constant live-ins need broadcast.
[CodeGenPrepare] Cache known-live PHIs when deleting dead PHI chains (#207191)
This patch fixes a compile-time issue in CodeGenPrepare for huge
functions.
`DeleteDeadPHIs` may repeatedly prove overlapping PHI chains non-dead.
For very large functions, many PHIs can share the same non-dead def-use
suffix, causing the same suffix to be scanned many times.
Add an `KnownNonDeadPHIs` cache to `RecursivelyDeleteDeadPHINode`
and `DeleteDeadPHIs`. When a chain is proven non-dead, visited PHIs are
recorded so later queries can stop once they reach one of them.
This reduces the pathological CodeGenPrepare case from ~30mins to ~30s.
[Flang][MLIR][OpenMP] Move host op filtering to the omp dialect
The MLIR pass that removes operations exclusively intended for the host
from OpenMP target offload modules is currently defined as part of
Flang. However, this is a feature that would benefit from being reusable
by other frontends, as removing such operations is a requirement for all
OpenMP target device modules prior to LLVM IR translation.
By moving the `omp-host-op-filtering` pass out of Flang, it had to be
updated to work on a lower-level LLVM dialect-based representation,
rather than FIR. This simplified some of the existing edge cases, such
as `fir.declare` ops and `fir.boxchar` type handling. In addition, new
function arguments are introduced as placeholders and return values from
host-only functions are removed, producing a cleaner result and
simplifying the pass as compared to previously.
As a result of a later execution of this pass, dynamic dispatch of host
functions via dispatch table using `fir.dispatch`, `fir.type_info` and
`fir.dt_entry` ops would break due to the removal of `fir.dt_entry`
[6 lines not shown]
[Flang][MLIR][OpenMP] Fix declare_target globals visibility
This patch introduces various changes to the handling of
`declare_target` global variables in Flang:
- Non-`declare_target` globals are unconditionally made "internal" when
compiling for an OpenMP offload target. This prevents potential symbol
redefinition issues related to globals that don't actually exist on the
device.
- Local SAVE variables handling for OpenMP offloading programs is fixed to
prevent their associated "internal" linkage from producing broken
device code for `declare_target enter(...)`.
- When globals are indirectly accessed from the target device (e.g.
`declare_target link(...)`), the associated and unused full-storage
global is marked with "internal" linkage to facilitate later removal.
- `declare_target device_type(host) enter(...)` variables are set to
external linkage when compiling for a target device, causing linker
errors if accessed. This mirrors Clang's behavior.
