[LV] Model memory runtime checks with AddRec bounds in VPlan. (#231132)
Remove the restriction in attachRuntimeChecks that kept the IR-generated
memcheck block when a check group's bounds contain a SCEVAddRecExpr.
VPSCEVExpander now expands AddRecs of loops outside VPlan's scope as
VPInstructions, with only canonical IVs expanded to IR in the plan's
entry, which dominates the check block.
Depends on https://github.com/llvm/llvm-project/pull/230858
PR: https://github.com/llvm/llvm-project/pull/231132
[LV] Remove scalar resume phis when creating epilogue markers (NFC). (#236307)
Erase the resume phis in the main plan's scalar preheader directly in
preparePlanForMainVectorLoop, right after creating the ResumeForEpilogue
markers for them, instead of in a separate removeScalarResumePhis step
after the check blocks have been added.
[BPF] Lower the atomic operations that there is no insn for (#235215)
BPF has add, and, or, xor, xchg and cmpxchg on 32 and 64 bits, load
acquire
and store release. The rest is an error:
atomicrmw xchg ptr %p, i8 %v seq_cst
unsupported atomic operation, please use 32/64 bit version
atomicrmw max ptr %p, i64 %v seq_cst
Cannot select: AtomicLoadMax
load atomic i64, ptr %p seq_cst, align 8
sequentially consistent (seq_cst) atomic load/store is not supported
fence acquire
Runtime fence is not supported at the moment
With load acquire and store release, -mcpu=v4:
- 8 and 16 bits: the minimal size of cmpxchg is 32 bits. AtomicExpand
widens
and, or, xor, and turns the others into a loop around cmpxchg of the
[26 lines not shown]
[BPF] Select undef and poison as zero (#235216)
An undef or poison value is selected as IMPLICIT_DEF: no insn, the
register
holds what it happens to hold. The verifier rejects a read of a register
that
was not written, whatever is done with the value.
Such values are not a sign of a bug in the source. In
%e = zext i64 %b to i128
%m = mul i128 %a, %e
with bits 0-63 of %m in use the upper halves of the operands don't
matter.
They are passed to __multi3() in r2 and r4, which nobody sets: "R4
!read_ok".
Select ISD::UNDEF and ISD::POISON of i32 and i64 as 'rX = 0'. Both get
[7 lines not shown]
[BPF] Set the call frame size of the blocks that a select expands to (#236250)
A select is expanded to a branch and two new blocks. When that happens
between ADJCALLSTACKDOWN and ADJCALLSTACKUP of a call that passes
arguments on the stack, the new blocks have to say how big the call
frame is at their entry. ARM, AVR, LoongArch, M68k, PowerPC, RISC-V do
that. BPF didn't:
*** Bad machine code: Call frame size on entry does not match value
computed from predecessor ***
- function: f
- basic block: %bb.5 go
Call frame size on entry 0 does not match value computed from
predecessor 8
unwind-select-arg.ll of #235217 is the first test with such a call. It
fails with -verify-machineinstrs, which is the default in a build with
LLVM_ENABLE_EXPENSIVE_CHECKS. The invoke has nothing to do with it: a
call with six arguments, one of them a bool that is computed, fails the
[7 lines not shown]
[mlir][vector] Check the written dims when folding a write mask (#233244)
`isMaskTriviallyFoldable` compares dest dim `rankDiff + i` with index
`i`, so when the vector has fewer dims than the destination, it checks
dims against the wrong indices. `createWriteOrMaskedWrite` therefore did
not call it when given a permutation map, and kept the mask even when it
is all true. It now passes only the sizes and indices of the dims that
the map selects, and the mask shape, so the check runs for every map.
For example, vectorizing this with `vector_sizes [4, 4]`:
```mlir
%r = tensor.insert_slice %src into %dest[2, 6, 0] [4, 1, 4] [1, 1, 1]
: tensor<4x4xi32> into tensor<8x8x8xi32>
```
gave a write masked with `vector.create_mask` of sizes 6 and 8 on a
`vector<4x4xi1>`, which is all true. The write is now unmasked.
[9 lines not shown]
[SandboxVec][LoadStoreVec] Propagate Metadata (#236274)
This patch implements metatdata porpagation to the vectorized
instructions. This is the expected behavior according to the
LoadStoreVectorizer.
[ADT] Make BitmaskEnum compound assignment operators constexpr (#236003)
`~ | & ^ << >>` were already `constexpr` in `BitmaskEnum.h` while `|= &=
^= <<= >>=` were not.
Addresses build failures from
https://github.com/llvm/llvm-project/pull/220606
AI Disclosure:
I used Claude Opus 5 to help write the tests and implement the changes.
[libc][math] Fix `sinpif` and `sinpif16` error handling (#235780)
Follow up of https://github.com/llvm/llvm-project/pull/235296, tested
both functions with CORE-MATH's exhaustive checker in all four rounding
modes. With `CORE_MATH_SUPPORT_ERRNO` and `CORE_MATH_CHECK_INEXACT`
enabled.
- Set ERANGE for underflow in sinpif and sinpif16.
- Raise missing underflow exceptions in sinpif16.
- Return exact half-integer results in sinpif16 without raising
FE_INEXACT.
[AArch64][MachineOutliner] Fix up SP add/sub immediates in outlined sequences (#191120)
Extend the AArch64 machine outliner to handle ADDXri/SUBXri instructions
that read SP when the outlined function saves LR on the stack, shifting
SP down by 16.
Previously only loads/stores with an SP base were adjusted. This patch
adds support for ADDXri and SUBXri with an SP source operand, correctly
re-encoding the adjusted offset and changing the opcode when the sign
flips. Unencodable adjusted offsets cause the candidate to be rejected
during the safety check, preventing illegal outlining.
[mlir][VCIX] Check legalized type before use in math.tan test pattern (#236051)
`MathTanToVCIX` called `getElementType()` on the result of
`legalizeVectorType()` before checking whether legalization failed. For
types it can't legalize (scalars, multi-dimensional vectors, scalable
vectors with unsupported element types) that result is null, so the
pattern crashed instead of leaving the op alone like the cos, sin and
log patterns do.
This moves the null check above the first use and adds tests for a
scalar `math.tan` and a `vector<2x4xf32>` one.
Fixes #231844
[mlir][acc] Cache symbol lookups in ACCImplicitData (#236237)
ACCImplicitData resolves the global behind every candidate variable with
SymbolTable::lookupNearestSymbolFrom, through isDeviceAccessibleValue,
isInDeviceMemoryValue and the FIR type-interface models. Without a table
each lookup scans every top-level op of the module, so on a module with
thousands of symbols this dominates the pass.
Let callers pass an optional SymbolTableCollection to
isDeviceAccessibleValue, isInDeviceMemoryValue, isValidValueUse and
OpenACCSupport::isValidValueUse, and add it as a trailing parameter of
the
PointerLikeType and MappableType isDeviceAccessible / isInDeviceMemory
interface methods. ACCImplicitData keeps one collection for the duration
of a run. Passing no collection keeps the previous behavior.
This can speedup the pass by 4.5x on a large TU with a lots of symbols.
[Support][mlir] Allow custom prefixes in SourceMgrDiagnosticVerifier (#227568)
Allow callers to choose the directive prefixes (default: `expected`),
like Clang's `-verify=<prefixes>`, so that one test file can be shared
between RUN lines with different expectations.
A prefix now has to be at the start of the line or follow a character
other than `[[:alnum:]_-]`, so that `expected-error` is no longer found
inside e.g. `unexpected-error`. This also applies to MLIR.
Assisted-by: AI tooling
Pull-Request: https://github.com/llvm/llvm-project/pull/227568
[RISCV][P-ext] Add isel pattern for mulsu.h11/maccsu.h11 (#236118)
`mulsu.h11` multiplies the sign-extended upper half word of rs1 by the
zero-extended upper half word of rs2. `maccsu.h11` adds the product to
rd.
Previously, (sra X, 16) and (srl X, 16) were selected as separate
srai/srli instructions feeding `mul` or `maccsu.h00`. Match them
directly
so both shifts are folded. As with `maccsu.h00`, the maccsu pattern
requires the multiply to have a single use.
[flang] Evaluate InElemental() at most once per scope (#236235)
`CheckHelper::InElemental()` called `IsElementalProcedure()` on every
use. That characterizes the whole enclosing procedure, including every
dummy argument and the folding of its shape. `CheckObjectEntity()` calls
`InElemental()` for each object entity, so declaration checking was
quadratic in the number of dummy arguments: issue 236223 has the
reproducer and the profile.
This caches the result for the innermost procedure symbol and resets it
whenever `CheckHelper::Check(const Scope &)` changes that symbol.
Diagnostics are unchanged: `IsElementalProcedure()` is still the only
source of the answer, just evaluated once per scope instead of once per
entity.
With the issue's generator, `-fsyntax-only` is now linear in the number
of dummies: at 800 explicit-shape dummies it goes from 24.8 G to 0.44 G
instructions (about 56x). `check-flang` passes.
[2 lines not shown]
[AArch64][MTE] Tag dynamic allocas in AArch64StackTagging
Instrument dynamic allocas in the memtag-stack pass, more or less
following the the way AddressSanitizer does: by instrumenting
stackrestore to un(poison/tag) allocas when free'd.
- Dynamic allocas are given 16-byte alignment and padded to 16-bytes.
- At lifetime.start (or at the alloca if no lifetimes), the dynamic
alloca is tagged.
- The most recently executed dynamic alloca is tracked in an extra
per-func static alloca. The address of this static alloca is used
as the bottom (highest address) of the dynamic alloca section of
the frame, while the value stored there is treated as the current
top (lowest address) of the frame.
- At llvm.stackrestores, memory is untagged from the restored sp
to the last executed dynamic alloca.
- At function returns, memory is untagged from the last executed
dynamic alloca to the bottom of the (dynamic alloca section of) the
frame.
[2 lines not shown]
[AArch64][MTE] Tag dynamic allocas in AArch64StackTagging
Instrument dynamic allocas in the memtag-stack pass, more or less
following the the way AddressSanitizer does: by instrumenting
stackrestore to un(poison/tag) allocas when free'd.
- Dynamic allocas are given 16-byte alignment and padded to 16-bytes.
- At lifetime.start (or at the alloca if no lifetimes), the dynamic
alloca is tagged.
- The most recently executed dynamic alloca is tracked in an extra
per-func static alloca. The address of this static alloca is used
as the bottom (highest address) of the dynamic alloca section of
the frame, while the value stored there is treated as the current
top (lowest address) of the frame.
- At llvm.stackrestores, memory is untagged from the restored sp
to the last executed dynamic alloca.
- At function returns, memory is untagged from the last executed
dynamic alloca to the bottom of the (dynamic alloca section of) the
frame.
[2 lines not shown]
[libc++] Make detect-changepoints much faster (#236245)
The permutation test used to find changepoints was the bottleneck of the
LNT refinement job. This patch makes the test much cheaper in two ways
without changing the algorithm:
- Compute the best split of a segment only once, when the segment is
created.
- Scoring every split of a segment only needs the sums of distances
within the first t points and between the first t points and the whole
segment. The latter doesn't depend on the order of the points, so it is
computed once per segment.
This was measured to be roughly 10x faster as the previous algorithm,
and gives the same results.
Assisted by Claude
