[mlir][x86] Add pattern to tile and unroll vector.contract for AMX nanokernel (#229076)
Introduces a new rewrite pattern to tile and unroll a `vector.contract`
inside an accumulation loop into a loop nest and smaller accumulation
chains on which the AMX nanokernel pattern can match.
The rationale is to provide an entry point for tile-based DSLs that
leave tiling decisions to the user, and compile directly to `vector` IR,
bypassing the usual infrastructure.
This PR covers the AMX BF16 and int8 targets. The pattern match is quite
rigid in this initial version, as the transformation reconstructs the
accumulation loop with smaller shapes instead of rewriting, and the
contraction's shape must be cleanly unrollable to the nanokernel-sized
contractions.
Assisted-by: Copilot
[Flang] Introduce common::numeric_limits (#228108)
On platforms that do not have a native (u)int128_t, the class
`Int128<IS_SIGNED>` is meant to provide its own implementation. The C++
specification allows programs to provide its own specializations of
`std::numeric_limits<T>` for such user-defined types
([[namespace.std]](http://www.eel.is/c++draft//namespace.std#2)).
However, the combination of clang-cl using the MSVC STL or libstdc++ in
strict mode poses a problem:
1. While Clang has a int128_t native type, MSVC's STL implementation
does not provide a `std::numeric_limits<T>` specialization for it
(because Microsoft's cl.exe doesn't have a 128 bit integer type).
2. Some versions if libstdc++ disable all uses of __int128 in strict
mode (-std=c++NN instead of -std=gnu++NN), including the specialization
of `std::numeric_limits<T>`. Moreover, `std::numeric_limits<__int128>`
being specialized does not mean that `std::make_unsigned_t<__int128>`
also is.
A previous solution in #216958 was to provide our own specialization of
[14 lines not shown]
[LoongArch] OptWInstrs: Sign-extend outer PHI operands to remove sext.w in loops
A sext.w inside a loop often cannot be removed only because one of the
values reaching it comes in through a PHI from outside the loop and is not
known to be sign-extended, e.g. an i32 argument that is only any-extended:
preheader:
...
loop:
%p = PHI %x, %preheader, %next, %loop
%s = ADDI_W %p, 0
%next = ADD_D %p, ...
If every other value reaching the sext.w is sign-extended, sign-extend
those PHI operands in their incoming blocks instead and remove the sext.w.
This trades a sext.w executed on every iteration for one executed once per
entry into the loop.
Only incoming blocks with a smaller loop depth than the sext.w are
[8 lines not shown]
[Inliner] Change CalleeTTI -> CallerTTI
It makes more sense to use the caller's TTI, as this way we can query
whether the instruction can be inlined into the parent function, using
state described by the caller's TTI.
For example, for AArch64 SME where a function can be in a different mode
(streaming/non-streaming), the cost-model gives different answers based
on the TTI used.
For now, we don't expect to see any changes in behaviour as there are
currently no targets where it makes different cost/inlining-decisions
based on which TTI is used.
SelectionDAG: Don't fold bitcasts of opaque constants to FP
Folding the bitcast of an opaque integer constant into a ConstantFP
drops the opaque flag. Skip the fold like the truncate fold does, so a
hoisted constant stays a shared integer. In aarch64-pmull2.ll the
hoisted constant is now materialized with an equivalent 32-bit mov.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[flang] Add check for definability for Image-selector STAT= specifier #224058 (#230031)
This MR fixes #224058, where Flang accepts a non-definable
`stat-variable` in an image selector without any diagnostic..
[AArch64][GlobalISel] Generate TBL for shuffles in single block loops (#230673)
This is the global-isel equivalent of #227593, where we generate a tbl
in a loop as opposed to multiple shuffle instructions. The only major
difference is that we do not support perfect shuffle yet (this is a
prerequisite for that), and that we pick up the single basic block from
the MachineBasicBlock directly.
[AArch64][SME] NFC: Inline getCompatibilityForChangeToStreamingMode. (#231899)
This removes the need for the `StreamingIncompatibility` enum and makes
the code a little simpler.
This PR also moves `hasPossibleIncompatibleOps` to AArch64TTIimpl, so
that we can query the costmodel in a follow-up PR.
[CIR] Emit cir.libc.memmove and cir.libc.memset for memmove and memset (#227667)
memmove, memset and their __builtin_ forms went through calls to memmove
and memset where classic codegen emits the llvm.memmove and llvm.memset
intrinsics. Emit cir.libc.memmove and cir.libc.memset instead, following
CGBuiltin.cpp and the existing memcpy handling. They lower to the same
intrinsics. __builtin_trivially_relocate shares the memmove path and, as
in classic codegen, scales its count by the element size.
Additionally, this adds alignment to the cir.libc.memmove Op (similar to
memcpy in #225845) and properly sets the alignment for both
cir.libc.memmove and cir.libc.memset.
On top of that, the emitDirectCallee now honors the
__attribute__((no_builtin)) and variants targetting specific builtins.
Fixes https://github.com/llvm/llvm-project/issues/229378.
Assisted-by: Claude Opus 5.5
[3 lines not shown]
[lld][MachO][NFC] Don't let the output path trip cstring-dedup.s's check-not
The __cstring check uses --implicit-check-not foo/bar, but llvm-objdump
prints the input file's full path first. When that path contains "bar"
(e.g. /home/isanbard/...), the test fails. Run llvm-objdump on a
relative path from inside %t so only the section contents are checked.
Assisted-by: Claude Opus 5.5
[clang][OpenMP][NFC] Anchor "bar" CHECK-LABELs on the function definition
ordered_doacross_codegen.cpp and parallel_for_simd_codegen.cpp use a
bare "bar" as a CHECK-LABEL. That also matches the ModuleID and
source_filename lines when the source path contains "bar" (e.g. a home
directory like /home/isanbard), which splits the input at the wrong
place and makes the tests fail. Match the define of @_Z3barv instead.
Assisted-by: Claude Opus 5.5
[CodeGen] Always emit CodeView compiler info for UEFI targets (#188383)
UEFI is based on the MSVC toolchain conventions. PR #142970 made Windows
targets always emit minimal CodeView compiler information (`S_OBJNAME`
and `S_COMPILE3`) even when debug info is otherwise disabled, matching
MSVC. Make UEFI targets behave the same way.
[InstCombine] Canonicalize atomicrmw sub of a constant to add
Rewrite `atomicrmw sub ptr, C` as `atomicrmw add ptr, -C`. This
matches the canonical form InstCombine already uses for plain `sub`.
In wrapping arithmetic $old - C = old + (-C)$, so both the stored
value and the returned old value stay the same, including when C is
`INT_MIN`. Ordering, syncscope and metadata are kept.
```llvm
; before
%old = atomicrmw sub ptr %p, i32 1 syncscope("agent") monotonic
; after
%old = atomicrmw add ptr %p, i32 -1 syncscope("agent") monotonic
```
This benefits AMDGPU, for example: https://godbolt.org/z/39xsYro3o
[llvm] Adjust LLVM_ABI annotations for ELF/Mach-O (#232393)
Some annotations were not at the right level (class or method) for some
use cases. These ensure that the dylib build with default visibility set
to hidden properly works for macOS and Linux with tests.
The effort to build LLVM as a dylib is tracked in #109483.
[InstCombine] Add tests for atomicrmw sub of a constant. NFC
atomicrmw sub C can be canonicalized to atomicrmw add -C because both
produce the same stored and returned values under wrapping arithmetic.
Cover syncscope, INT_MIN, and negative cases.
Attributor: Reach fixpoint in AACallEdges if there are only direct calls
The callees when there are only direct, live calls will not change.
No other attributes matter, but this ran the update a second time to
find out.
Instructions spent in amdgpu-attributor:
rocPRIM test_device_scan, gfx942: 1265M -> 1263M (-0.2%)
Blender Cycles HIP kernel, gfx1100: 488M -> 487M (-0.2%)
rocPRIM 16 tests, full LTO, closed world: 4381M -> 4359M (-0.5%)
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Attributor: Don't create call site AACallEdges for direct calls
AACallEdgesFunction created an AACallEdges for every call-like
instruction in the function including intrinsics. For a direct call
without callback metadata, the call site attribute only ever adds the
callee, so add it directly.
Instructions spent in amdgpu-attributor:
rocPRIM test_device_scan, gfx942: 1963M -> 1768M (-9.9%)
Blender Cycles HIP kernel, gfx1100: 636M -> 622M (-2.2%)
rocPRIM 16 tests, full LTO, closed world: 7415M -> 6511M (-12.2%)
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
TableGen: Share the xforms for FP immediate bits (#232492)
AArch64 and AMDGPU defined identical bitcast_fpimm_to_iN xforms, and
Hexagon defined an equivalent ftoi. Define them once in
TargetSelectionDAG.td.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[libc] Add <libintl.h> and textdomain stub (#232351)
POSIX.1-2024 standardises the <libintl.h> message translation
interfaces. Because LLVM-libc currently supports only the "C" locale,
message catalog functions can be satisfied with stateless stubs that do
not access the filesystem.
Added the <libintl.h> header generation specification and the textdomain
entrypoint behind LLVM_LIBC_ENABLE_EXPERIMENTAL_ENTRYPOINTS, returning
"messages" when querying or resetting the domain and returning the input
domain name otherwise.
Assisted-by: Automated tooling, human reviewed.
[RISCV][P-ext] Add scalar mulq(r) intrinsics (#232316)
Add the scalar "Q-format" Multiplication intrinsics:
https://github.com/riscv/riscv-p-spec/blob/master/P-ext-intrinsics.adoc#q-format-multiplication
- `__riscv_mulq_i32`
- `__riscv_mulqr_i32`
RV32 selects `mulq`/`mulqr` directly. RV64 has no scalar forms, so the
intrinsics use `pmulq.w`/`pmulqr.w` on the low words, the same way as
the scalar `aadd`/`asub` intrinsics (#230710).
[libc] Use a syscall wrapper for `sched_getscheduler` (#232597)
This PR move `sched_getscheduler` behind the centralized syscall
wrappers, following `libc/docs/dev/syscall_wrapper_refactor.md`.
[flang] Fail fir.global conversion when the type has no LLVM equivalent
GlobalOpConversion passed the result of convertType straight to
LLVM::GlobalOp::create. For a global whose type the FIR type converter
cannot lower, such as tensor<10xi32>, that result is null and
TypeAttr::get segfaults. Return a match failure instead so the
conversion reports that fir.global could not be legalized.
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Signed-off-by: Keshav Vinayak Jha <keshavvinayakjha at gmail.com>
[clang-repl] Add OffloadType enum and HIP device parser skeleton (#231391)
This PR adds an OffloadType enum (Detect, CUDA, HIP) and
Interpreter::createWithDevice(), so device setup can be chosen in one
place. createWithCUDA() now forwards to it, so CUDA behaviour and
existing callers are unchanged. Detect and the HIP binary generation
return "unsupported" errors for now.
It also adds the IncrementalHIPDeviceParser class. Its constructor
checks that the device target triple is an AMDGPU one.
GenerateOffloadBinary() is a stub that will be implemented later.
Assisted by Claude Sonnet 5.5
Revert "[SLP][NFC]Add extra bitpack tests, NFC" (#232593)
Reverts llvm/llvm-project#232554
Reverting as the patch broke the CI builders across the board. I see the
original owner is not available to address the failure so reverting to
turn ToT green.
