AMDGPU/GlobalISel: Legalize BF16 for FP math opcodes (#214059)
Make the BF16 form of G_FCONSTANT legal.
Widen BF16 G_FADD, G_FMUL, G_FMA, and G_FCANONICALIZE to F32.
[clang-tidy][docs] Note MISRA/AUTOSAR checks are not accepted (#214661)
Checks that directly reference MISRA/AUTOSAR are not accepted for
legal/license reasons. Generic overlapping checks without such links
remain fine.
Fixes #160940
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment to the
device.
Claude assisted with this patch.
[offload] Pool host and shared allocations
Route them through a memory manager, like the device allocations. Also,
move the registration as pinned memory to the plugin site since only the
plugin knows if the corresponding host/shared memory is pinned.
Improves performance of affected allocations by ballpark 1,000x.
[AMDGPU][CodeGen] Allow remat with multiple users in same region (#214725)
This relaxes one of the constraints on rematerialization candidates in
the scheduler's `PreRARematStage`. The current implementation only
allows rematerializing a register if it has a single user. This allows
it when a register has multiple users in the same region.
In such cases the register is rematerialized once just before the first
user in program order. The cost model for assessing rematerialization
opportunities stays unchanged since a register is only ever
rematerialized to a single location.
[X86] Always use 128-bit V_SET0/AVX512_128_SET0 patterns, along with SUBREG_TO_REG for extension to 256/512-bit vectors (#212950)
We often end up with duplicate "all zero" registers as they are
represented by different psuedos for 128/256/512-bit types. Whilst they
are nearly always free/cheap to rematerialize, its a waste of
instructions and registers to needlessly do this.
This patch only uses 128-bit SET0 calls and makes use of AVX's implicit
zeroing of the upper elements (via SUBREG_TO_REG). AVX512F-only targets
still use 512-bit VPXORDZrr instructions if they need to zero
xmm16-xmm31, otherwise they fallback to VXORPSrr like regular AVX.
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment to the
device.
Claude assisted with this patch.
[offload] Pool host and shared allocations
Route them through a memory manager, like the device allocations. Also,
move the registration as pinned memory to the plugin site since only the
plugin knows if the corresponding host/shared memory is pinned.
Improves performance of affected allocations by ballpark 1,000x.
[offload] Do not pool memory while allocation traces are requested (#214752)
Would otherwise hide use-after-free because memory stays valid if it's
in the pool.
[MemRef] Split elide-reinterpret-cast tests (#213026)
Narrow the scope of test files by separating them by the patterns they
exercise, for readability.
[lldb][NFC] Don't use LLDB.h in multiple-targets/debuggers tests (#210981)
LLDB.h is a header that includes (nearly) every SB API header
indirectly. This patch replaces the use of this header in some .cpp
tests by instead directly including the needed headers. This is mainly
to reduce compilation times of these files as they are recompiled on
each test run.
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment to the
device.
Claude assisted with this patch.
[offload] Pool host and shared allocations
Route them through a memory manager, like the device allocations. Also,
move the registration as pinned memory to the plugin site since only the
plugin knows if the corresponding host/shared memory is pinned.
Improves performance of affected allocations by ballpark 1,000x.
[X86] Lower vector integer division and remainder through float division (#205263)
x86 has no vector integer divide instruction so a vector `udiv` by a
variable divisor scalarizes into N scalar divides. Since there is no
vector idiv to fall back on, lowering through vector float division is
better.
This is implemented as a new DAG combine (combineIntDivRem in
X86ISelLowering.cpp) that runs before legalization and rewrites the
vector divide before it scalarizes. Handles udiv, sdiv, urem and srem.
- <=32-bit elements (i8/i16/i32): a single float divide (f32 for
<=16-bit, f64 for 32-bit) recovers the exact quotient because the
operands fit the mantissa
(https://lemire.me/blog/2017/11/16/fast-exact-integer-divisions-using-floating-point-operations/).
Enabled from SSE2.
Part of #201319.
[mlir][vector] Verify non-unit strides on `masked/expand/compress` ops (#210952)
Closes the stride-verification gap left open by #204611 and #205869.
`vector.maskedload`/`maskedstore`/`expandload`/`compressstore` lower to
LLVM masked intrinsics that read/write N *consecutive* elements from a
single pointer (see [LangRef](https://llvm.org/docs/LangRef.htm)), but
none of them verified the memref's minor-dim stride, so `strided<[2]>`
verified successfully and silently miscompiled.
This PR rejects statically-known non-unit and dynamic strides.
**Stacked on #211004**
---------
Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment to the
device.
Claude assisted with this patch.
[Clang][OpenCL] Make FP_ILOGBNAN configurable (#214535)
The OpenCL C spec allows the implementation to choose the value of the
`FP_ILOGBNAN` macro between `INT_MIN` and `INT_MAX`. This patch makes
the
value configurable by introducing a new macro
`__OPENCL_FP_ILOGBNAN_MIN`.
The default value is unchanged and is set to `INT_MAX`. If the new macro
is defined, the value of `FP_ILOGBNAN` will be set to `INT_MIN`.
This change makes the OpenCL C headers similar to the LLVM `libc`, that
also has the `FP_ILOGBNAN` macro configurable.
