[X86] Fix bf16 VBROADCAST lowering and selection without AVX512BW+VLX
`bf16` vector shuffles are lowered as `vXi16` shuffles, and
`combineTargetShuffle` turns `VBROADCAST(bitcast(x))` into
`bitcast(VBROADCAST(x))` whenever `x` has a legal type. If `x` is an
un-bitcast `v8bf16` value (e.g. a register live across basic blocks, a
volatile load, or an AVX-NE-CONVERT conversion result), this creates a
`v8bf16` or `v16bf16` `X86ISD::VBROADCAST` (sources wider than 128 bits are
extracted down to `v8bf16`). If the source is a load whose other uses are
simplified away later, it becomes an `X86ISD::VBROADCAST_LOAD` instead. Both
only had isel patterns under `[HasBWI, HasVLX]`:
LLVM ERROR: Cannot select: v8bf16 = X86ISD::VBROADCAST ...
This has been latent with AVX-NE-CONVERT (LLVM 18 already fails with
`-mcpu=sierraforest`). 93d242ed5416 (#230953) made `bf16` vectors legal with
AVX512BW, which exposed it without AVX512VL, and 3fdac7de5496 (#221225) made
`v8bf16` legal with AVX, so it now fails on every AVX2 target without
AVX512BW+VLX (e.g. `-mcpu=znver2`/`znver3`).
[19 lines not shown]
[X86] Add bf16 broadcast tests (NFC)
Add tests for `bf16` vector splats that end up as `X86ISD::VBROADCAST` or
`X86ISD::VBROADCAST_LOAD` of a `bf16` vector type:
- values live across basic blocks
- volatile loads
- loads whose other uses are simplified away after legalization
- broadcasts of `v8bf16` values that are bitcast to `vXi16`/`vXi32`
Only the default target (SSE2) and `-mattr=+avx512bw,+avx512vl` are tested
for now. AVX2 targets without AVX512BW+VLX crash on all of these: they fail
to select the `bf16` broadcast, or form an illegal `v16bf16` broadcast after
type legalization. The follow-up fix adds RUN lines for them.
Assisted-by: Antigravity
[libc] Implement posix_close in unistd (#240024)
Implement the standard POSIX.1-2024 function `posix_close` in
`<unistd.h>`.
### Changes
- Added `posix_close` entrypoint in
`libc/src/unistd/linux/posix_close.cpp` which validates `flag` (`0` or
`POSIX_CLOSE_RESTART`) and delegates to `SYS_close`.
- Added `POSIX_CLOSE_RESTART` macro definition to `unistd-macros.h`.
- Registered entrypoint across Linux targets (`x86_64`, `aarch64`,
`arm`, `riscv`).
- Added unit tests in `libc/test/src/unistd/posix_close_test.cpp`.
Fixes #240019.
[flang][OpenACC] Fix lowering crash on DO WHILE loop constructs
A loop construct associated with a DO WHILE crashed lowering: reading the
DO WHILE's loop control as bounds aborted, both when the DO WHILE is lowered
unstructured and, with -lower-do-while-to-scf-while, when it is lowered
structurally. A DO WHILE has no bounds and no induction variable, so lower
the construct to an acc.loop without bounds that holds the whole DO WHILE,
as for an unstructured loop, with nothing to privatize.
Fix check for loop promotion to parallel for
A teams loop with calls to non API routines cannot be promoted to
parallel for. The visitor overrode such a case when there was a later
call to an API routine.
[llvm][Object] Handle unknown COFF machine types (#240047)
machineToStr() previously used llvm_unreachable() to handle unknown
machine types, which could cause a crash when an unsupported machine
type was encountered. This fix makes function return "unknown" instead,
when handling machine types not supported by this function
AI assisted with investigating the issue, discussing the testing
approach, and reviewing the implementation. I implemented the fix,
adapted existing LLVM tests to add regression coverage and ran the tests
locally
Related to #239535.
[CIR] Return function pointer from `getCallableForCallee` (#237275)
Currently any indirect call will cause an assertion failure if openmp is
enabled together with CIR. This is because one of omp dialect-specific
passes (MarkDeclareTarget) will dyn_cast a pointer returned from
`getCallableForCallee`. The patch follows the same approach as FIR: if
there is no function, return first CallOp argument.
[SPIRV] Declare SPV_KHR_relaxed_extended_instruction only when used
Whether one is needed is only known while building the module-scope NSDI,
which used to be streamed in sections 7 and 10, after OpExtension had been printed.
Split NSDI generation from output: generateModuleDebugInfo() now builds
the module-scope OpStrings and debug instructions into per-section
buffers before the module header is printed, and adds the extension if a
forward reference was built.
emitNonSemanticDebugStrings() and emitNonSemanticGlobalDebugInfo() output
the buffers in sections 7 and 10.
Function-body NSDI is still streamed.
With the extension no longer declared unconditionally.
[CodeGen] Skip pseudo probes in the ImplicitNullChecks scan (#240239)
Implicit null checks (`-enable-implicit-null-checks`, used by frontends
that lower nil checks to a `SIGSEGV` handler) are silently disabled when
pseudo probes are enabled.
`PSEUDO_PROBE` is marked as having side effects so that it is not
dropped, so `canHandle()` rejects it and `analyzeBlockForNullChecks()`
gives up on the whole non-null successor. Probes start every
instrumented block, so `-fpseudo-probe-for-profiling` together with
`-enable-implicit-null-checks` loses the optimization for essentially
every candidate.
Skip debug instructions and pseudo probes while scanning the successor;
neither is the faulting operation nor counts against
`MaxInstsToConsider`. The probes stay in their block, and only the
faulting load is hoisted into the predecessor, so samples on that load
are attributed to the predecessor's probe.
[17 lines not shown]
[mlir][ArmNeon][NFC] Replace `NeonVectorOfLength` with common constraints (#239225)
Replace the custom `NeonVectorOfLength` type constraint with
`VectorOfLengthAndType` and `FixedVectorOfRankAndLengthAndType` from
`CommonTypeConstraints.td`.
This is a step towards unifying the `ArmNeon` and `ArmSVE` dialects.
More work will be required, but I am deliberately implementing it in
small, incremental steps to simplify the review process.
[libc] Fix LLVM_LIBC_ADD_FUNCTION_C_ALIAS lint namespace error (#240751)
When LIBC_COPT_PUBLIC_PACKAGING is disabled, as in internal object
targets linted by libc-lint, LLVM_LIBC_ADD_FUNCTION_C_ALIAS expanded to
static_assert(true, "Require semicolon."). Because callers place this
macro at translation-unit scope outside LIBC_NAMESPACE_DECL,
llvmlibc-implementation-in-namespace rejected the static_assert
declaration in libc/src/errno/libc_errno.cpp and the scanf sources.
Replace that stub with an extern "C" type alias referring to
LIBC_NAMESPACE::name. ImplementationInNamespaceCheck already exempts
extern "C" linkage specifications at file scope, the semicolon
requirement is preserved, and the referenced internal symbol is still
verified at compile time.
Assisted-by: Automated tooling, human reviewed.
[AArch64][DAG] Improve codegen for clearing upper vector lanes (#237259)
Add patterns to use FMOV to implicitly clear the upper lanes of a vector
while preserving lane 0
Will fix this open
[issue](https://github.com/llvm/llvm-project/issues/228317) upon the
stack being merged
[InstCombine] Canonicalise Zero Vector operand to always be RHS of a shuffle vector (#237258)
shuffle zero, x, mask --> shuffle x, zero, mask'
Canonicalize ZeroVector of a shuffle to always be the RHS operand.
The motivation can be viewed in the next PR where the zero vector on the
RHS results in the same DAG, allowing for less patterns required for
instruction selection
Some riscv intrinsic test failed. This was due to the intrinsics
emitting shuffle vectors with a zero operand.
One of the intrinsics (ppaire) were not lowering correctly due to the
change. The reason being is that a function that checks whether we lower
to the intrinsic was dependent on whether the zero was a LHS operand or
RHS operand.
I modified this so that it checks both operand orders (via commuting the
mask). The function is only used for this one intrinsic so I am not
expecting any more regressions from this
[X86] Add missing v8bf16/v16bf16 VPBROADCASTW patterns without AVX512BW+VLX
X86ISD::VBROADCAST of a v8bf16 register only had isel patterns for
[HasBWI, HasVLX] (VPBROADCASTWZ128rr). On AVX2 targets without
AVX512BW+VLX, a splat of a bf16 vector that is live across basic blocks
fails with:
LLVM ERROR: Cannot select: v8bf16 = X86ISD::VBROADCAST (v8bf16 CopyFromReg ...)
The splat itself is lowered as a v8i16 shuffle, but combineTargetShuffle
then turns VBROADCAST(bitcast(x)) into bitcast(VBROADCAST(x)), which
creates the v8bf16 node.
This has been latent since AVX-NE-CONVERT made v8bf16 a legal type (it
already fails with -mcpu=sierraforest in LLVM 18). #230953 and #221225
made v8bf16 legal with AVX512BW and with plain AVX respectively, so it
now fails on every AVX2 target without AVX512BW+VLX, including common
AMD Zen 2/Zen 3 (-mcpu=znver2/znver3) hosts.
[10 lines not shown]
[X86] Add bf16 broadcast tests (NFC)
Add tests for bf16 vector splats that end up as X86ISD::VBROADCAST or
X86ISD::VBROADCAST_LOAD:
- values live across basic blocks
- volatile loads
- loads whose other uses are simplified away after legalization
- broadcasts of v8bf16 values that are bitcast to vXi16/vXi32
Only -mattr=+avx512bw,+avx512vl is tested in this commit. Targets without
AVX512BW+VLX crash on all of these tests; the follow-up commit fixes that.
Assisted-by: Antigravity
[flang][OpenACC] Test unstructured seq loops that stay TODO in kernels
Cover the unstructured `acc loop seq` shapes in kernels that still hit
the "unstructured do loop in acc kernels" TODO on the seq loop's own
DO: a seq loop inside an unstructured IF, SELECT CASE, BLOCK or DO
WHILE, a loop that is seq only for a non-default device_type, and auto
loops. Each case checks its pre-FIR tree, and every TODO check in the
file now names the line of the loop it fires on, using split-file
--leading-lines.
[flang][OpenACC] Lower unstructured seq loops directly in kernels
An `acc loop seq` with an early exit directly in a kernels region still
hit the "unstructured do loop in acc kernels" TODO, because the kernels
construct around it stays unstructured. Its control flow stays in the
acc.kernels region, its acc.loop privatizes its induction variables,
and no parallel loop runs it on more than one thread, so lower it like
a structured seq loop in kernels.
[flang][OpenACC] Privatize nested DO variables of unstructured seq loops
A DO nested in the body of an unstructured `acc loop seq` that the
directive does not own has no acc.loop of its own when it is
unstructured, so nothing privatized its induction variable and it hit
the "unstructured do loop in acc kernels" TODO. It runs on the thread
executing the seq loop, and its induction variable is predetermined
private to that thread, so privatize it on the seq loop's acc.loop.
Factor the check for the default device_type in a list of device_type
attributes into a helper, used here and by the two existing copies.
[mlir][vector] Make `vector.gather` non-maskable and use its mask operand (#239602)
`vector.gather` already has a _mask operand_, so wrapping it in
`vector.mask` gives two ways to express the same thing.
This PR makes the following changes, to eventually avoid _double
masking_ on `vector.gather`:
- Drop `MaskableOpInterface` and `getExpectedMaskType` from
`vector.gather`, and fix the Linalg vectorizer.
- Align `vector.gather` with `vector.scatter`.
- Remove `MaskedGatherOpPattern`, its registration, and its entry in
`apply_patterns.vector.lower_masked_transfers`.
Part of #239511.
RFC:
https://discourse.llvm.org/t/rfc-unify-masking-in-the-vector-dialect/92675
Signed-off-by: Federico Bruzzone <federico.bruzzone.i at gmail.com>
[AMDGPU] Handle AVGPR pressure in a bank-aware manner (#231388)
Distribute AVGPR pressure across ArchVGPR and AGPR banks. Follow the
register allocator, first charge AVGPRs to ArchVGPRs and then to AGPRs.
Controlled by -amdgpu-avgpr-banking, ON by default.
Summary of test changes:
New tests added:
avgpr-banking-agpr-alloc.mir: Make sure amdgpu-agpr-alloc is honored by
banking codepath.
avgpr-banking-bank-excess.mir: Check candidate bank excess.
avgpr-banking-rigid-peak.mir: Check that peak ArchVGPR and AVGPR
pressure at different region points are handled properly by the banking
codepath.
avgpr-banking.ll: Most of the register pressure comes from use of
AVGPRs. Without banking, some schedules are thought to spill and are
reverted. With banking, they are kept. Neither finally spills, which the
test also checks.
[12 lines not shown]
[clang][bytecode][NFC] Explicitly set gnu++17 for arrays test. (#240438)
This test uses variable-length arrays, which are a GNU extension.
Although `-std=gnu++17` is the default, we oughtn't leave it implicit
here, as neither `-std=c++17` nor `-std=gnu++20` meet the expectations
of the test.
