[AArch64][llvm] Combine SVE AND-NOT operations into BIC for _x forms (#232526)
Replace `not` followed by `and` with `bic` for expressions such as:
```
return svand_u32_x(svptrue_b32(), a, svnot_u32_x(svptrue_b32(), b));
```
Combine ordinary `and` with `not` when the `not` predicate is all-active
or its inactive values are undefined.
Also combine matching-predicate SVE `and_x`-`not` pairs before lowering
discards the `and` predicate. `and_x` ignores inactive results, so the
`not`'s inactive values do not need preserving.
This applies to all integer element widths. Update the pre-commit LLVM
expectations, including cases where combining would change the
result.
[AArch64][llvm] Combine SVE AND-NOT operations into BIC for _m forms (#232527)
Replace `not` followed by merging `and` with `bic` for expressions such
as:
```
return svand_u32_m(pg, a, svnot_u32_x(pg, b));
```
Combine when `not` is the second operand and its predicate matches the
`and` predicate or is all-active.
Use predicated `bic` to preserve the `and`'s first operand in inactive
lanes. Use unpredicated `bic` when the `and` predicate is all-active.
Do not commute merging `and` operands, since this would change inactive
results. Leave pairs with incompatible predicates unchanged.
This applies to all integer element widths. Update the pre-commit LLVM
expectations, including cases where combining would change the
result.
[AArch64][llvm] NFC: Add pre-commit tests for SVE AND-NOT combining (#232525)
Add LLVM tests for expressions where NOT followed by AND could
use BIC. Record the existing assembly so the following changes show the
missed optimisations being fixed.
Cover _x and _m AND forms using all-active and runtime predicates. Test
_m forms across all integer element widths. Include NOT with defined
inactive values and reversed AND operands.
Also test cases that must remain as NOT followed by AND because
combining would change the values in inactive lanes.
[InlineFunction] Simplify lifetime intrinsic check (NFC) (#235681)
The pointer type check always passes since the opaque pointers
migration. Additionally, we nowadays require that lifetime
intrinsics are direct users of the alloca.
[AArch64] Support FP atomic stores with memory hints
This patch extends the atomic-store-hint pseudos to accept preserved
floating-point atomic stores. It adds an FPR16 pseudo family and direct
half, bfloat, float, and double patterns for the supported addressing
modes.
The pseudos expand relaxed stores to STR and RCPC3 release and
sequentially consistent stores to STLUR, retaining memory hints without
cross-register-bank copies.
The existing AArch64::ATOMIC_STORE_HINT_H is renamed to
AArch64::ATOMIC_STORE_HINT_HH and AArch64::ATOMIC_STORE_HINT_H now means
FP16 instead of it's old association with integer half-word.
[AArch64] Support direct RCPC3 FP atomic loads and stores
Preserve half, bfloat, float, and double atomic loads and stores
through AtomicExpand for acquire and release orderings, and preserve
sequentially consistent stores, when RCPC3 and NEON are available.
Add SelectionDAG patterns for the RCPC3 LDAPUR and STLUR operations,
including unscaled offsets, and update the existing FP lane patterns to
match the preserved floating-point atomic nodes. Targets without the
required features continue to use integer lowering.
[AArch64] Support direct relaxed FP atomic loads and stores
This patch preserves half, bfloat, float, and double atomic loads and
stores through AtomicExpand when they use unordered or monotonic
ordering and the target supports floating point.
It adds SelectionDAG patterns that select the accesses directly in the
floating-point register bank. Unsupported targets, stronger orderings,
and stores carrying memory hints continue to use integer lowering but
will be handled in the subsequent PRs.
[SLP] NFC: make candidate state ownership explicit
Separate the state for the currently owned SLP graph from the graph state
used by dependency analysis, costing and emission. Keep these in private
CandidateState and SharedGraphState members of BoUpSLP.
Centralize candidate reset after scheduler invalidation, clear shared
observers before destroying graph entries, and verify ownership under
EXPENSIVE_CHECKS. Preserve existing reset boundaries and vectorization
behavior.
Assisted-by: codex
[libc] Use a syscall wrapper for `epoll_create1` (#234907)
This PR move `epoll_create1` behind the centralized syscall wrappers,
following `libc/docs/dev/syscall_wrapper_refactor.md`.
[analyzer] Remove questionable assertions in tryRearrange (#234624)
The `tryRearrange` logic in `SimpleSValBuilder.cpp` has asserted that if
a `SymIntExpr` reaches this point (as an operand of the expression that
is being rearranged), then its RHS (a concrete `APSInt`) has the same
type as the symbolic LHS.
This commit replaces these assertions with a defensive early return.
I did not observe any failure of these assertions; tests in
`svalbuilder-rearrange-comparisons.cpp` validate that they are not
triggered by some natural candidates, e.g. handling `x - 1U` where `x`
is a symbolic integer.
However, this is not a general invariant of all `SymIntExpr` instances
(there are 50+ LIT tests where one argument of `evalBinOpNN` is a
`SymIntExpr` where the types of the LHS and RHS are different) and there
isn't any concrete reason that would filter out mixed-type `SymIntExpr`s
before the `tryRearrange` logic, so the assertion isn't justified.
[12 lines not shown]
[NFC][analyzer] Replace `CheckerContext::generateSink` with `addSink` (#234351)
This change generalizes `addSink` to match the signature and behavior of
`generateSink` (but with comfortable default values for all three
parameters), then replaces use of `generateSink` with `addSink` and
deletes `generateSink`. Textually replacing `generateSink` with
`addSink` is a correct way to update downstream checkers; but it is
often possible to remove arguments that were mandatory for
`generateSink` but match the defaults of `addSink`.
This change was motivated by the fact that `CheckerContext` had three
sink-making methods (`generateSink`, `addSink` and `generateErrorNode`)
which all do the same thing and cause lots of confusion with their
separate existence. They all had their advantages and drawbacks:
- `generateErrorNode` is the most widespread (85 uses in checkers), but
it cannot be THE sink-making function, because the name is not suitable
for the dozens of cases where the sink is not used for an "error" (bug
report). It also creates the unfortunate illusion that "error nodes" are
somehow different from regular nodes.
[17 lines not shown]
