[offload] allow h2h olMemcpy with a queue (#212757)
olMemcpy currently disallows host-to-host copies with a queue, even
though that is a desirable use case (e.g. to copy a buffer as a
dependency for another operation)
This PR adds support for asynchronous host-to-host copies by adding
`GenericDeviceTy::dataMemcpy` and the corresponding required
`dataMemcpyImpl` plugin operation. Unlike `dataSubmit`, `dataRetrieve`,
and `dataExchange`, this operation does not assign a device direction to
either pointer:
- CUDA: Uses `cuMemcpyAsync`, which by itself determines the pointer
types and preserves stream ordering. Dynamic CUDA symbol declarations
and `_v2` lookup support are included.
- Level Zero: Uses the existing queue `memoryCopy` operation (backed by
`zeCommandListAppendMemoryCopy`), which appends a memory-copy command to
the command list.
- AMDGPU: No "enqueue a host-to-host copy" operation exists directly in
that API, so we enqueue a `std::memcpy` callback through
[10 lines not shown]
[X86] vector-reduce-or-cmp.ll - update mask_v3i1 IR to match middleend (#213923)
InstCombine converts vXi1 logic reductions to bitcasted scalar integer
ops - we should be testing that, not llvm.vector.reduce.or.v3i1 calls
Exposes some really poor scalarization on pre-AVX512 targets
[libc++] Trim down associative container benchmark sizes (#212252)
Based on local measurements, the associative containers are by far the
slowest benchmarks to run. While there is value in benchmarking small
and large sizes, we can significantly cut down benchmark times by
stripping down to only 2 sizes (from 4).
With this patch, the benchmarks go from 3124s to 1766s across the
associative container benchmarks, which is -43% of their execution time,
and corresponds to ~15% of the entire benchmark suite.
Towards #173032
[libc++] Fix the clang-tidy source filter to cover libcxx/src again (#213732)
The filter regex used `libcxx(abi?)`, which matches `libcxxab` and
`libcxxabi`, but never just `libcxx`.
[libc++] Remove C++20 modules tests from the C++03 frozen test suite (#213730)
These tests are never executed since they require C++20, and the C++03
frozen headers only run in C++03 mode.
[mlir][tensor] Preserve source encoding when folding insert_slice canonicalizers (#207239)
`InsertSliceOpConstantArgumentFolder` re-derives the refined source type
via `ExtractSliceOp::inferCanonicalRankReducedResultType`, which copies
the encoding of the passed-in "source template" - in this pattern,
`insertSliceOp.getDestType()`.
For a static, encoding-less destination this silently drops any encoding
the actual source carried, which downstream dialects can use for
mandatory metadata (upper bounds, layout, sparsity descriptors) lost
during `--canonicalize`.
Rebuild the refined source type carrying the original source's encoding.
If the encoding implements `VerifiableTensorEncoding`, re-verify it
against the refined shape and drop it if it no longer holds (e.g. a
rank-dependent sparse encoding); otherwise treat it as
opaque/shape-agnostic and propagate it unconditionally.
Shape refinement (the `?` -> static direction) is unchanged and still
satisfies `preservesStaticInformation`.
[2 lines not shown]
[PAC][ELF] Support R_AARCH64_AUTH_TLSDESC_CALL relocation
The R_AARCH64_AUTH_TLSDESC_CALL is introduced to allow linker relaxation of
AUTH TLSDESC call sequences for non-preemptible undefined weak symbols.
The lld patch introducing the relaxation: #194636
Corresponding ARM docs PR: https://github.com/ARM-software/abi-aa/pull/395
[RISCV] Reduce spill/reload pairs when Xqcilo extension is enabled (#212807)
[RISCV] Reduce spill/reload pairs when Xqcilo extension is enabled
Currently, `SelectAddrRegImm26` calls `SelectAddrFrameIndex` first,
causing bare frame-index loads (offset 0) to select 48-bit loads/stores at
ISel. Due to `AddedComplexity=2` on the QC48LdPat patterns, the wide
opcode won over the standard LW/SW even though the resolved frame offset
typically fits simm12.
This led to more spills and reloads in functions which are under high
register pressure because 48-bit loads and stores are not marked easily
rematerializable. Also, simply adding 48-bit loads and stores to
`isLoadFromStackSlot/isStoreToStackSlot` doesn't solve the regression
for the multi call case and only by making Isel produce the plain
32/64-bit loads and store opcodes as the baseline does RA behave
identically.
Therefor this PR fixes the issue by:
[17 lines not shown]
[X86][APX] Fix per-function V3 unwind for EGPR functions on Windows x64 (#212924)
A function that saves a callee-saved EGPR (R16-R31) cannot be encoded with V1/V2 unwind info, so it must use V3 even when the module default stays on V1/V2 (e.g. an APX clone created by auto-dispatch alongside a baseline generic clone). The previous code rejected such functions with a recoverable backend diagnostic ("EGPR (R16-R31) requires V3 unwind info on Windows x64") instead of emitting valid V3 unwind info.
Introduce a single shared predicate, requiresWinX64UnwindV3(MF), that returns true when the whole module is in V3 mode, or when the function needs an unwind table and may use EGPR. It is consumed by X86FrameLowering (SEH prolog/epilog layout), the X86WinEHUnwindV2 pass (which skips such functions), and the X86WinEHUnwindV3 pass (which stamps a per-function .seh_unwindversion 3 on every WinEH frame -- the entry block and each funclet). Also widen the SEH_UnwindVersion pseudo operand from i1imm to i8imm since it holds 1, 2 or 3.
(cherry picked from commit 9466be2c87689a07c9a171f24cb5c9b0c2f0cda8)
[ARM][MVE] Match canonical saturating negation patterns (#213085)
InstCombine pull request
https://github.com/llvm/llvm-project/pull/194519 canonicalized the
saturating negation idiom to
an llvm.ssub.sat operation. For the ARM backend we only recognized the
original
select and subtract pattern, causing vector absolute values to expand to
VQSUB
plus a compare and select instead of VQABS.
In this patch we teach the VQABS and VQNEG patterns to recognize this
ssub.sat
form.
(cherry picked from commit c99e6752b9f9e1867af88c07ee44f837bc6e4311)
[clang][Driver] Fix libc++ include path on NetBSD (#212716)
`clang++` defaults to `-stdlib=libc++` on NetBSD. When building with
both `clang` and `libcxx` included, the freshly built `clang++` fails to
find `<__config_site>`:
```
In file included from /usr/include/strings.h:68:
In file included from bin/../include/c++/v1/string.h:57:
bin/../include/c++/v1/__config:13:10: fatal
error: '__config_site' file not found
13 | #include <__config_site>
| ^~~~~~~~~~~~~~~
```
The file is present in `include/<triplet>/c++/v1`, but that isn't
searched by default. NetBSD has its own version of addLibCxxIncludePaths
which misses that directory.
[12 lines not shown]
[LLD][COFF] Replace ARM64EC TLS directory chunks with native chunks when available (#212845)
On ARM64X targets, CRT provides separate TLS directory chunks, expecting
the linker to sort it out. TLS directory uses _tls_start and _tls_end
symbols to reference .tls section. Those symbols use section sorting to
ensure that they are emitted at the start and end of .tls section, but that's
not enough when we have two separate chunks for views: only one of them
can really be the first one. Following MSVC, merge those chunks instead so
that both symbol tables point to the same chunk.
Additionally apply the same logic to _tls_used and _tls_index. This
allows entire TLS directory to be shared between EC and native views. To
achieve that, CRT additionally needs to mark each TLS callback with
-arm64xsameaddress. This matches how MSVC linker and libraries work, but
it requires EC and native views to use the same set of TLS callbacks. We
may emit separate TLS directories in the future to make it more robust.
(cherry picked from commit e096d2f60dbc6cab991d5c02a5f7125a6dc694dc)
[LoopInterchange] Prevent the transformation stage from stopping partway (#205564)
As mentioned in #205562, there are cases where the transformation stage
in LoopInterchange stops partway through, and the output IR ends up
partially modified rather than interchanged. Notably, the interchange is
recognized as having succeeded internally even in such cases. Apparently
this happens not to cause any miscompiles at the moment, but it is
clearly dangerous.
This patch removes the early exit in the transformation phase. We cannot
simply remove it, which checks for the presence of a unique successor of
the inner loop header, because there is a case where the header actually
has multiple successors. To avoid that situation, this patch changes the
code to call SplitBlock on the inner loop header unconditionally.
The test changes fall into two categories: some are simply due to newly
added redundant BBs. In the others, the expected interchanges are now
applied as intended, whereas previously the transformation failed and
the interchanges were not applied before this patch.
[2 lines not shown]
[lldb] Fix crash on creating string error (#212503)
It crashes because the `default` error string may not be a format string
compared to the `fallback` error string
(cherry picked from commit 29bc0749034c4f2bda0752360ecacd4e89d7439c)
