[mlir][Dataflow] Revisit region arguments when operand change (#239658)
Alternative to https://github.com/llvm/llvm-project/pull/238782
`gpu.launch` body args weren't revisited after input dims lattice
changes.
* Add `GetLatticeFn` to sparse forward analysis
`visitNonControlFlowArguments`, obtaining lattice through it
automatically subscribe block on this lattice so it is revisited when
lattice change.
* Update `IntegerRangeAnalysis` and `IntegerDivisibilityAnalysis`.
* nonSuccessorInputs/nonSuccessorInputLattices may not match
successor.getSuccessor()->getArguments(), so compute index in
nonSuccessorInputs in `IntegerRangeAnalysis`.
Note to LLVM integration:
`SparseForwardDataFlowAnalysis::visitNonControlFlowArguments` now takes
an additional `GetLatticeFn getLattice` parameter. Use
[3 lines not shown]
[AMDGPU] Add UBSan checks for AMDGPU lane access builtins (#231485)
Summary:
First attempt to make some AMD specific behavior for UBSan checks. The
language manual states that using non-uniform values to readlane or
non-present indices to shuffling is undefined. There may be a few more
of these, but I thought this was low hanging fruit, hopefully
uncontroversial.
[libc] Implement sigtimedwait and sigwaitinfo in signal (#238787)
Add the standard POSIX function sigtimedwait and sigwaitinfo to
<signal.h>:
- Add low-level rt_sigtimedwait system call wrapper in
libc/src/__support/OSUtil/linux/syscall_wrappers/rt_sigtimedwait.h.
- Add public header declaration in libc/src/signal/sigtimedwait.h.
- Implement sigtimedwait entrypoint in
libc/src/signal/linux/sigtimedwait.cpp.
- Add public header declaration in libc/src/signal/sigwaitinfo.h.
- Implement sigwaitinfo entrypoint in
libc/src/signal/linux/sigwaitinfo.cpp using rt_sigtimedwait with a null
timeout.
- Declare sigtimedwait and sigwaitinfo in libc/include/signal.yaml.
- Register both entrypoints for Linux (x86_64, aarch64, riscv) in
architecture config files.
- Update CMakeLists.txt files across syscall wrappers, signal sources,
and tests.
[6 lines not shown]
[libc] Implement clock_nanosleep in time (#240025)
This PR implements standard POSIX `clock_nanosleep` in `<time.h>`.
- Adds `clock_nanosleep` entrypoint and internal Linux syscall wrapper.
- Returns error numbers directly without modifying `errno` per POSIX
specification.
- Registers public declaration in `libc/include/time.yaml`.
- Registers entrypoints across Linux targets (`x86_64`, `aarch64`,
`riscv`).
- Adds unit tests in `libc/test/src/time/clock_nanosleep_test.cpp`.
Fixes #240020.
[flang][OpenMP] Iteration distance should be (+|-) some-expr (#239289)
Verify that the operator is actually either + or -. Simplify the
pre-lowering representation to have intrinsic operator directly instead
of having a defined-operator.
The actual verification of the operators is fairly simple. The bulk of
the changes is related to extracting `IntrinsicOperatorT` out of
`DefinedOperatorT` into a standalone type and using it in
`LoopIterationT::Distance`. The functions that converted AST clauses
into the lowering representation needed to be updated, as well as the
references to the nested enum values which are now members of a
different class.
[DAG] Replace getLibraryOptions helper with getSelectionDAGOptions (#240799)
Reduces compile time impact by avoiding most
DAG.getLibraryOptions().get<SelectionDAGOptions>() lookups
[DAGCombiner][GlobalISel] Fold floor into unsigned FP-to-int conversions (#239233)
Add shared SelectionDAG/GlobalISel combines to eliminate floors before
unsigned FP-to-int conversions.
* IR conversion is equivalent (reusing an existing comment):
https://github.com/llvm/llvm-project/pull/231060#issuecomment-6120567356
* Removing the floor is a refinement for `fptoui` and preserves the
result for `fptoui.sat`.
* Shared floors are allowed, following #231060. This can enable tail
calls but can also introduce spills when the floor is a libcall.
* `InstCombine` could have this pattern as well, but AMDGPU lowering
explicitly creates a floor with two users, so these combines are still
useful.
Updated existing codegen checks for:
* `f32 -> u32`, `f64 -> u32`, and `f16 -> u16`
* 64-bit results, saturation, and vectors
* source modifiers and the normal, true16, and fake16 variants
[6 lines not shown]
[AMDGPU] Fold S_CSELECT with identical sources in SIFoldOperands (#240525)
tryFoldCndMask already folds a V_CNDMASK whose two sources are the same
register or the same immediate. Do the same for S_CSELECT_B32 and
S_CSELECT_B64.
These selects arise when two vectors with matching lanes are built in
one block and selected between in another, for example tensor
descriptors built outside a loop. Isel sees only the cross-block vector
registers, so it selects every lane, and only PeepholeOptimizer reveals
that some lanes are the same value. Folding the select lets MachineLICM
hoist the resulting move out of the loop.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[X86] Fix bf16 VBROADCAST lowering and selection without AVX512BW+VLX
`bf16` vector shuffles are lowered as `vXi16` shuffles, and
`combineTargetShuffle` turns `VBROADCAST(bitcast(x))` into
`bitcast(VBROADCAST(x))` whenever `x` has a legal type. If `x` is an
un-bitcast `v8bf16` value (e.g. a register live across basic blocks, a
volatile load, or an AVX-NE-CONVERT conversion result), this creates a
`v8bf16` or `v16bf16` `X86ISD::VBROADCAST` (sources wider than 128 bits are
extracted down to `v8bf16`). If the source is a load whose other uses are
simplified away later, it becomes an `X86ISD::VBROADCAST_LOAD` instead. Both
only had isel patterns under `[HasBWI, HasVLX]`:
LLVM ERROR: Cannot select: v8bf16 = X86ISD::VBROADCAST ...
This has been latent with AVX-NE-CONVERT (LLVM 18 already fails with
`-mcpu=sierraforest`). 93d242ed5416 (#230953) made `bf16` vectors legal with
AVX512BW, which exposed it without AVX512VL, and 3fdac7de5496 (#221225) made
`v8bf16` legal with AVX, so it now fails on every AVX2 target without
AVX512BW+VLX (e.g. `-mcpu=znver2`/`znver3`).
[19 lines not shown]
[X86] Add bf16 broadcast tests (NFC)
Add tests for `bf16` vector splats that end up as `X86ISD::VBROADCAST` or
`X86ISD::VBROADCAST_LOAD` of a `bf16` vector type:
- values live across basic blocks
- volatile loads
- loads whose other uses are simplified away after legalization
- broadcasts of `v8bf16` values that are bitcast to `vXi16`/`vXi32`
Only the default target (SSE2) and `-mattr=+avx512bw,+avx512vl` are tested
for now. AVX2 targets without AVX512BW+VLX crash on all of these: they fail
to select the `bf16` broadcast, or form an illegal `v16bf16` broadcast after
type legalization. The follow-up fix adds RUN lines for them.
Assisted-by: Antigravity
[libc] Implement posix_close in unistd (#240024)
Implement the standard POSIX.1-2024 function `posix_close` in
`<unistd.h>`.
### Changes
- Added `posix_close` entrypoint in
`libc/src/unistd/linux/posix_close.cpp` which validates `flag` (`0` or
`POSIX_CLOSE_RESTART`) and delegates to `SYS_close`.
- Added `POSIX_CLOSE_RESTART` macro definition to `unistd-macros.h`.
- Registered entrypoint across Linux targets (`x86_64`, `aarch64`,
`arm`, `riscv`).
- Added unit tests in `libc/test/src/unistd/posix_close_test.cpp`.
Fixes #240019.
[flang][OpenACC] Fix lowering crash on DO WHILE loop constructs
A loop construct associated with a DO WHILE crashed lowering: reading the
DO WHILE's loop control as bounds aborted, both when the DO WHILE is lowered
unstructured and, with -lower-do-while-to-scf-while, when it is lowered
structurally. A DO WHILE has no bounds and no induction variable, so lower
the construct to an acc.loop without bounds that holds the whole DO WHILE,
as for an unstructured loop, with nothing to privatize.
Fix check for loop promotion to parallel for
A teams loop with calls to non API routines cannot be promoted to
parallel for. The visitor overrode such a case when there was a later
call to an API routine.
[llvm][Object] Handle unknown COFF machine types (#240047)
machineToStr() previously used llvm_unreachable() to handle unknown
machine types, which could cause a crash when an unsupported machine
type was encountered. This fix makes function return "unknown" instead,
when handling machine types not supported by this function
AI assisted with investigating the issue, discussing the testing
approach, and reviewing the implementation. I implemented the fix,
adapted existing LLVM tests to add regression coverage and ran the tests
locally
Related to #239535.
[CIR] Return function pointer from `getCallableForCallee` (#237275)
Currently any indirect call will cause an assertion failure if openmp is
enabled together with CIR. This is because one of omp dialect-specific
passes (MarkDeclareTarget) will dyn_cast a pointer returned from
`getCallableForCallee`. The patch follows the same approach as FIR: if
there is no function, return first CallOp argument.
[SPIRV] Declare SPV_KHR_relaxed_extended_instruction only when used
Whether one is needed is only known while building the module-scope NSDI,
which used to be streamed in sections 7 and 10, after OpExtension had been printed.
Split NSDI generation from output: generateModuleDebugInfo() now builds
the module-scope OpStrings and debug instructions into per-section
buffers before the module header is printed, and adds the extension if a
forward reference was built.
emitNonSemanticDebugStrings() and emitNonSemanticGlobalDebugInfo() output
the buffers in sections 7 and 10.
Function-body NSDI is still streamed.
With the extension no longer declared unconditionally.
[CodeGen] Skip pseudo probes in the ImplicitNullChecks scan (#240239)
Implicit null checks (`-enable-implicit-null-checks`, used by frontends
that lower nil checks to a `SIGSEGV` handler) are silently disabled when
pseudo probes are enabled.
`PSEUDO_PROBE` is marked as having side effects so that it is not
dropped, so `canHandle()` rejects it and `analyzeBlockForNullChecks()`
gives up on the whole non-null successor. Probes start every
instrumented block, so `-fpseudo-probe-for-profiling` together with
`-enable-implicit-null-checks` loses the optimization for essentially
every candidate.
Skip debug instructions and pseudo probes while scanning the successor;
neither is the faulting operation nor counts against
`MaxInstsToConsider`. The probes stay in their block, and only the
faulting load is hoisted into the predecessor, so samples on that load
are attributed to the predecessor's probe.
[17 lines not shown]
