[AMDGPU] Combine redundant ballot intrinsic calls
Suppose there is a loop where there is a call to @llvm.amdgcn.ballot,
which maps to an instruction involving the exec mask as an operand. This
instruction duplicates if the loop is unrolled. With a higher number of
unrolled iterations, the code bloats with such redundant instructions
with $exec as there is no middle-end/backend pass which could combine
such instructions in a uniform CFG.
This patch introduces a transform in AMDGPUUniformIntrinsicCombine to
combine redundant calls to @llvm.amdgcn.ballot, to mitigate this issue.
The approach is to walk over the dominator tree and collect all calls to
@llvm.amdgcn.ballot. Map the result type and condition to the calls, to
avoid combining calls of different kinds. Calls A and B can be combined
into A iff:
- A and B are identical
- A dominates B
- all paths from A to B are uniform and exec-invariant.
[2 lines not shown]
[MergeFunc] Preserve observable function pointer identity (#213604)
Fixes #213206.
MergeFunc can replace one function with a forwarding thunk to another.
FunctionComparator treated self-references as equal in every context, so
functions that observe their own address (e.g. icmp %p, @g) could be
merged incorrectly.
Compare ordinary function-pointer uses as normal global values. Treat
self-references as equal only for corresponding call targets and
blockaddress.
AI assistance was used while preparing this change.
[lldb] Handle 0 size sites in StopPointSiteList::FindInRange (#217919)
Fixes #205120
In which due to delayed breakpoints, a breakpoint that would become an
external breakpoint later (meaning managed by the debug server) was
temporarily stored as a software breakpoint (which is managed by lldb)
with a zero size breakpoint site. That zero site site tripped an
assertion when you tried to write over the site.
To fix this, I've explicitly ignored zero size sites in FindInRange by
defining them as never overlapping. FindInRange is only used for
patching reads and writes, so I think this is safe to do. I have
documented this in the docstring.
I considered adding a breakpoint type "uncommitted", but software
breakpoints are actually handled in the most conservative manner (reads
and writes are always patched). So I think as a default it's fine (also
I don't want to go and audit all the places that use that enum and end
[4 lines not shown]
[AArch64] Use LD1/ST1 in strict-align mode (#211513)
When compiling with +strict-align, we currently scalarise all fixed
width vector loads and stores where the alignment is less than the
vector size, because the LDR/STR instructions require alignment matching
the vector size. However, the LD1/ST1 instructions only require the
alignment to match the size of the individual elements, so we can use
them in more cases.
This patch:
* Adds a check in allowsMisalignedMemoryAccesses, to prevent memory
accesses being scalarised when not needed.
* Doubles the tablegen patterns for the vector LDR/STR instructions, to
restrict them to vector-size aligned pointers in +strict-align mode,
and leave them unrestricted otherwise.
[VPlan] Strip bad FIXME in expand-scev (NFC) (#218213)
7be188763a ([VPlan] Don't preserve NSW in mul -> shl conversion for bw-1
op) missed stripping it.
[AArch64][SDAG][NFC] Use CNTP intrinsic for MSTORE lowering
Currently, when lowering MSTORE the process is to Zero Extend,
VECREDUCE_ADD and then Zero Extend the result for types that are
not i64. This for i8/i16 types does not work as AArch64 cannot
zero extend these types to i64. Instead, use the CNTP intrinsic
to combine the values before storing them.
[AliasSetTracker] Use ModRefInfo (NFC) (#217917)
Use ModRefInfo instead of an equivalent custom AccessLattice.
I did not bit-pack the value (which would have required adding casts
to/from ModRefInfo everywhere) because this doesn't impact the size of
the structure on 64-bit hosts anyway.
[lldb][debugserver] Fix bugs in FindBreakpointsThatOverlapRange (#217837)
Fixes #217359
The first issue is obvious, when checking the previous breakpoint we
should push back prev_pos->second, instead of pos->second.
The second problem is what happens when lower_bound returns end().
Before, `if (pos != end)` would stop us checking the previous
breakpoint.
If lower_bound returned end() but there were breakpoints, the last
breakpoint may start before addr and extend past it. This was being
missed.
To fix that:
* Remove the `if (pos != end)` check.
* Return early if there are no breakpoints.
* If lower_bound does not find the first one, look at the previous one.
[7 lines not shown]
[MemCpyOpt] Don't perform stack-move opt for out-of-bounds copy (#217607)
There can be a copy with an out of bounds offset in dead code. If the
perform the stack-move optimization based on that, we'll end up merging
an alloca into an out-of-bounds offset of another alloca, which will
result in UB (when used in non-dead code).
Fixes https://github.com/llvm/llvm-project/issues/216566.
[lldb][debugserver] Handle breakpoint prior to addr in RemoveTrapsFromBuffer (#217851)
Fixes #217840
lower_bound finds the first breakpoint that starts >= addr. There might
be a breakpoint before that which starts before addr but extends past
addr. Therefore it might need to be patched out of the buffer.
This fix is intentionally minimal as I don't have a Mac to test it in.
A proper fix would reuse FindBreakpointsThatOverlapRange, which also has
this bug but is getting fixed. The problem with that is that
RemoveTrapsFromBuffer is const, and FindBreakpointsThatOverlapRange
returns non const pointers to the breakpoints.
Not super complex to fix but more than I want to do at a distance.
[lldb] Create mock type systems in array for deterministic order (#218009)
To fix the occasional test failure from #217789: Instead of using two
random heap allocations for the type systems, allocate them in an array
once and create aliases to the elements (similar to what David suggested
in
https://github.com/llvm/llvm-project/pull/217789#issuecomment-5372708193).
The array allocation is destroyed when the last (aliased) shared pointer
is destroyed.
[VectorCombine] Fix foldBitcastOfVPLoad reordering loads (#218336)
We were inserting the new vp.load where the bitcast was, which would
reorder loads. This should hopefully fix RISC-V buildbot failures that
were exposed after 93ac788df8ff
RISCV: Fix VectorPeephole null dereference on an undef vmerge true operand (#218328)
Also since this is an SSA pass, switch to using getVRegDef instead of
getUniqueVRegDef
Co-Authored-By: Claude <noreply at anthropic.com> (Claude Opus 4.8)
[mlir][vector] Verify multi_reduction reduction dimensions (#216854)
`vector.multi_reduction` does not bounds-check `reduction_dims`.
Out-of-range
entries verify cleanly and then crash the canonicalizer:
```mlir
%0 = vector.multi_reduction <add>, %a, %b [1, 2] : vector<1x4xf16> to vector<1xf16>
```
`getReductionMask()` builds `SmallVector<bool> res(sourceRank)` and
executes
`res[2] = true`. The `[1, -1]` variant indexes `res[(size_t)-1]`.
Reject out-of-range and duplicate dims in the verifier, following
`TransposeOp::verify`. The mask is reused by the shape-inference loop
below.
One existing test is itself the repro: `propagate-layout.mlir` uses `[1,
[10 lines not shown]
RISCV: Fix VLOptimizer null dereference on an undef slide amount (#218329)
Also since this is an SSA pass, switch from getUniqueVRegDef
to regular getVRegDef.
Co-Authored-By: Claude <noreply at anthropic.com> (Claude Opus 4.8)
Reapply [Clang] Mark new as inaccessiblememonly if sane (#217652)
Reapply https://github.com/llvm/llvm-project/pull/197199 without changes
after https://github.com/llvm/llvm-project/pull/201375 improved our
errno TBAA reasoning.
-----
If `-fassume-sane-operator-new` (the default), assume that `operator
new` does not read or write accessible memory.
Currently, this assumption already exists due to special treatment in
BasicAA. I'd like to remove this special treatment (see
https://github.com/llvm/llvm-project/pull/197180), and instead rely only
on the `memory` attribute.
It's worth noting that this is consistent with GCC's interpretation of
the flag (where it is also enabled by default):
[5 lines not shown]
[lldb] Keep DW_OP_call_frame_cfa address-sized (#216684)
`DW_OP_call_frame_cfa` currently constructs a `Scalar` directly from
`lldb::addr_t`. Since `addr_t` is a 64-bit carrier, the result remains
64 bits
when evaluating an expression for a 32-bit target, and subsequent
address
arithmetic does not wrap at the target address width.
Normalize the CFA with the evaluator's existing `to_generic` helper
after it
is pushed. This preserves the existing frame lookup, error handling, and
`LoadAddress` value type while making the scalar use the target address
width.
Add an i386 unit test that verifies both 32-bit address arithmetic and
the
resulting `APSInt` bit width.
Fixes #210980
Fix AppVerifier GetThreadId(NULL) error in llvm::thread (#216941)
A recent change (64be34c562a23761dcb48a0a6a0b3ef0576c14c4) brought up a
behaviour change to the way llvm::thread checks if it is joinable. Prior
to the patch, a thread handle is checked if it's valid. The new
behaviour is to query a thread id and check if it's non-zero.
On Windows, the new behaviour means calling GetThreadId(handle) to
retrieve the thread id. This, however, results in errors when the thread
handle is not pointing to any active thread (NULL) and when running
applications under AppVerifier as it reports that a "system function is
called with a NULL handle". For applications using LLVM infrastructure
(in our case it is a MLIR-based compiler), the call stack looks like:
```
vrfcore!VerifierStopMessageEx+0x858
vfbasics!AVrfpHandleSanityChecks+0x3c
vfbasics!AVrfpNtQueryInformationThread+0x46
KERNELBASE!GetThreadId+0x33
my_app!llvm::llvm_thread_get_id_impl
[22 lines not shown]
[Mips] Use ELF binding when expanding PIC la (#217566)
A symbol already assigned to a section is not necessarily local. Use ELF
symbol binding when expanding PIC `la`, retaining the existing
section-based fallback for non-ELF targets, so defined default-visible
globals use the correct global GOT form.
Adds `llvm/test/MC/Mips/macro-la-pic-defined-global.s` covering global,
local, and `$25` call-register expansions.
Fixes #217371
[docs] Document `-mtriple` option in opt command guide (NFC) (#213942)
`-mtriple` option has been supported for a long time, but previously
lacked documentation.
[BasicAA][TLI] Treat local-linkage globals or known environments as not aliasing errno (#170290)
Errno cannot alias global variables with internal/private-linkage,
neither can it aliases globals on known environments, where errno is
known to be defined as a function call – unless targeting
freestanding/baremetal environments.
[Support] Remove deprecated CTLog2 from MathExtras.h (#218274)
CTLog2 was deprecated in favor of ConstantLog2 and has no remaining
in-tree users, so drop it.
Co-authored-by: Claude <noreply at anthropic.com>
ModuloSchedule: Fix using getVRegDef/getUniqueVRegDef on physregs (#216795)
These should be invalid to use on physical registers, but are currently
permissive. Avoid calling them so in the future they can assert.
Also this pass seems to run in SSA, so shouldn't really be using
getUniqueVRegDef.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply at anthropic.com>
[X86] Widen non-power-of-two vector div/rem under strictfp (#217572)
Non power of two vectors bail out of the FP divide fold under strictfp
and scalarize. The SAE forms are 512-bit only so operands widen into a
zmm, and a non power of two lane count has no machine type to widen
through.
This widens to the next power of two first but only while the result
still fits one divide. Past that it splits into two chains and a scalar
divide on the odd lane is cheaper so a shape like v9i32 is unchanged.
Fixes #217558
RISCV: Fix VectorPeephole null dereference on an undef vmerge true operand
Also since this is an SSA pass, switch to using getVRegDef instead of
getUniqueVRegDef
Co-Authored-By: Claude <noreply at anthropic.com> (Claude Opus 4.8)
RISCV: Fix VLOptimizer null dereference on an undef slide amount
Also since this is an SSA pass, switch from getUniqueVRegDef
to regular getVRegDef.
Co-Authored-By: Claude <noreply at anthropic.com> (Claude Opus 4.8)
