[ORC] Generalize RTBridge Callers to any runtime function (#213526)
An RTBridge Caller is a controller-side handle for calling a function in
the runtime. Until now the abstraction assumed every such function was a
trampoline -- a runtime function whose job is to invoke *another*
function at an address the controller supplies (run-as-main, run-as-int,
etc.) -- so every Caller carried a dedicated ExecutorAddr parameter for
that target.
Generalize Callers to call runtime functions of any shape. Invoking a
supplied target is now just one kind of call, with the target address an
ordinary leading argument rather than a built-in parameter: e.g.
MainCaller becomes Caller<int64_t(ExecutorAddr, ArrayRef<std::string>)>.
The SPS signatures already led with an SPSExecutorAddr for the target,
so this is a pure interface change -- the SPS wrappers and all call
sites are unaffected. It lets Callers model runtime functions that do
the work themselves, such as the memory-access wrappers, rather than
only those that dispatch to another function.
[offload][OpenMP] Add atomic cross-team reductions (#209298)
Regular cross-team reductions have two phases: the intra-team reduction
and the inter-team reduction. Atomic cross-team reductions replace the
second phase with a atomic instruction which is used by the main thread
of each team to directly fold the result of the intra-team reduction
into the final result. Since this requires a combination of "data type"
and "combine operation" for which an atomic instruction is available,
only some (but very common) reductions can be transformed to atomic
reductions. In cases where multiple reductions are performed on the same
construct, the atomic path is only taken if all reductions can be
transformed. Otherwise, we fall back to the regular cross-team reduction
using a buffer with per-team slots. This is not strictly necessary, but
hybrid reductions would induce more complexity with questionable
benefit.
Selecting an atomic path might not be the best option for every
situation, which is why it is not enabled by default. Instead, it can be
enabled via `-fopenmp-target-atomic-reduction`. Note that enabling the
[17 lines not shown]
[VPlan] Handle step where sign cannot be determined optimizeFindIVRed. (#213450)
optimizeFindIVReductions uses the step to determine if min or max is
needed. Bail out if the direction of the step cannot be determined via
SCEV.
Fixes https://github.com/llvm/llvm-project/issues/213424
[lldb] std::move unique_ptrs, rather than calling .release. (#213525)
These .release() calls are legacy from the std::auto_ptr to
std::unique_ptr transition.
[LLD][COFF] Replace ARM64EC TLS directory chunks with native chunks when available (#212845)
On ARM64X targets, CRT provides separate TLS directory chunks, expecting
the linker to sort it out. TLS directory uses _tls_start and _tls_end
symbols to reference .tls section. Those symbols use section sorting to
ensure that they are emitted at the start and end of .tls section, but that's
not enough when we have two separate chunks for views: only one of them
can really be the first one. Following MSVC, merge those chunks instead so
that both symbol tables point to the same chunk.
Additionally apply the same logic to _tls_used and _tls_index. This
allows entire TLS directory to be shared between EC and native views. To
achieve that, CRT additionally needs to mark each TLS callback with
-arm64xsameaddress. This matches how MSVC linker and libraries work, but
it requires EC and native views to use the same set of TLS callbacks. We
may emit separate TLS directories in the future to make it more robust.
[SPARC] Add patterns for i64->i32 and i64->i16 BSWAP-STOREs (#210483)
The lack of those is causing instruction selection to fail.
Also, for completeness, add variants of extending/truncating ops for
LOAD-BSWAP pairs too.
(cherry picked from commit 7afc89970fc3675d77e92aa091b65c078cacdcff)
workflows/release-binaries: Move environment declaration to upload job (#212687)
This is the only job that actually needs to use the environment secrets,
so the environment must be declared. We were using secrets in the
prepare job to do a permissions check, but this is unnecessary, because
that job does not do anything that is security sensitive.
Only the upload job needs to have these permission checks and these are
already included in the upload-release-artifact composite action.
(cherry picked from commit 02bde0716776a742164941e05cb026e750763b04)
[clang][llvm][AArch64] Set hardening fn attrs on synthetic functions (#211013)
Compiler-synthesized functions such as `__llvm_gcov_writeout`,
`__llvm_gcov_reset` and `__llvm_gcov_init` were previously never
receiving the AArch64 hardening function attributes (ptrauth-returns,
ptrauth-auth-traps, ptrauth-indirect-gotos and
aarch64-jump-table-hardening) since the attributes were only emitted by
Clang and gated by `PointerAuthOptions` structure's corresponding
fields. See `setPointerAuthFnAttributes` and
`initPointerAuthFnAttributes` member functions of `TargetCodeGenInfo`.
This patch resolves this in the same manner as #83153 does for several
other attributes. Particularly, Clang now emits corresponding 4 module
flags (conditionally on whether the related feature is enabled) with Max
behavior, and LLVM's `Function::createWithDefaultAttr` derives the
matching function attributes from them. Max behavior with conditional
emission is safe because none of these features affect ABI, so promoting
an absent flag on module merge cannot break compatibility.
(cherry picked from commit 2f8eb5b4be3eba5c7f58470d6538d00df53a55fc)
[RISCV] Fix incorrect lowering of VECTOR_INTERLEAVE on fixed vectors (#212642)
This is the sibling patch of #207254, as it turns out VECTOR_INTERLEAVE
has the same problem on fixed vectors as well.
Instead of converting individual operands into scalable vectors, this
patch puts each of the operands directly onto stack using the fixed
vector version of segmented store intrinsics, before loading them back.
---------
Co-authored-by: Luke Lau <luke at igalia.com>
(cherry picked from commit ff9b99207b5d4ec73554defc1e3c1ac50f8ff1d8)
[DWARFLinker] Keep DW_TAG_enumerator children of a live enumeration_type (#212849)
Swift allows functions inside enums:
enum Foo: Int {
case bar = 0
func baz() { ... }
}
DW_TAG_enumeration_type "Foo"
DW_TAG_enumerator "bar"
DW_TAG_subprogram "baz" DW_AT_declaration
...
DW_TAG_subprogram DW_AT_low_pc(...) DW_AT_specification -> "baz"
dieNeedsChildrenToBeMeaningful() did not list DW_TAG_enumeration_type,
so the parent walk skipped the enum's children.
[4 lines not shown]
[LowerAtomic] Clear elementwise flag when lowering atomic load (#213401)
The lower atomic pass demotes atomic loads to non atomic by
calling setAtomic(NotAtomic), but left the elementwise flag untouched.
Since elementwise is only valid on atomic operations, this produced a
non-atomic elementwise load, which the verifier rejects with "non-atomic
load cannot be elementwise".
Reference: https://github.com/llvm/llvm-project/pull/204556
[InstCombine] Don't change the type of elementwise atomic loads (#213414)
Skip load type canonicalization for elementwise atomic loads, which
would
otherwise drop the vector type and produce an invalid scalar elementwise
load.
Reference: https://github.com/llvm/llvm-project/pull/204556
[test] Add irreducible CFGs with closed-form block frequencies (#213492)
The functions here have irreducible control flow, but none of them pins
down how mass is divided among the entries of an irreducible region.
Add four cases whose exact frequencies follow from the branch weights:
- equalrows: all blocks share one successor distribution; 5:3:2.
- selfloops: self edges of differing probability; ignoring them, each
block
splits evenly between the other two; 8:5:5.
- unequalrows: symmetric non-header successors, differing header row;
8:3:3.
- nonentry: a member of the region that is not an entry, so its mass is
never
adjusted; 6:4:3.
BFI computes the first two exactly and the last two not. #213488 will
show up as a diff.
[ORC] Remove EPC runAsVoidFunction/runAsIntFunction methods (#213265)
Remove runAsVoidFunction and runAsIntFunction from
ExecutorProcessControl and all its implementations, along with the
now-unused RunAs{Void,Int}FunctionWrapperName bootstrap symbols and
SimpleRemoteEPC's corresponding address fields. Their only in-tree
users, COFFPlatform and COFFVCRuntimeSupport, now use
rt::sps::Int32VoidCaller and rt::sps::Int32Int32Caller.
runAsMain is likewise routed through rt::sps::MainCaller's controller-
interface symbol, so it is looked up under the same orc_rt_ci_sps_* name
the target-process bootstrap registers.
This is a step towards decoupling the ExecutorProcessControl interface
from SPS serialization.
[CSKY] Emit build attributes in assembly output (#213507)
Fixes CodeGen/CSKY/fpu-abi-attribute.ll, failing since #212975.
emitTargetAttributes is implemented only by CSKYTargetELFStreamer, so
llc -filetype=asm drops every .csky_attribute directive. Move it to
CSKYTargetStreamer, as ARM and RISC-V do.
[MLIR][Python] Drop the LLVMSupport link dependency from the python extensions (#213509)
Follow-up to #180986, which completed the series started in #178290: it switched the last python bindings off the C++ LLVM APIs and dropped `LLVMSupport` from the support library in `AddMLIRPython.cmake`. But the per extension `PRIVATE_LINK_LIBS LLVMSupport` in `mlir/python/CMakeLists.txt` was missed, so the dependency is still there for every extension module.
Assisted by: Claude
[MLIR][Python] Drop the LLVMSupport link dependency from the python extensions
#180986 switched the last python bindings off the C++ LLVM APIs and dropped
LLVMSupport from the support library in AddMLIRPython.cmake, but the
per-extension `PRIVATE_LINK_LIBS LLVMSupport` in mlir/python/CMakeLists.txt
was missed. `declare_mlir_python_extension`'s PRIVATE_LINK_LIBS becomes
`target_link_libraries(... INTERFACE ...)`, so every extension module still
links libLLVMSupport.a. None of the extension sources reference LLVM any more:
the only remaining matches for `llvm::` are two comments and a namespace that
happens to be called `llvm`.
Statically linking LLVMSupport into every extension gives each one its own copy
of the CommandLine.cpp globals. Where the loader resolves symbols across modules
rather than keeping them module-local, whichever copy initializes second aborts
the process:
CommandLine Error: Option 'print-inst-addrs' registered more than once!
That is what happens with the emscripten build of the bindings, where the
[7 lines not shown]
[ELF,SPARC] Handle TLS IE relocations (#213500)
An initial-exec reference loads the symbol's TP-relative offset from the
GOT and adds the thread pointer:
```
sethi %tie_hi22(sym), %o0 # R_SPARC_TLS_IE_HI22
add %o0, %tie_lo10(sym), %o0 # R_SPARC_TLS_IE_LO10
ldx [%l7 + %o0], %o0, %tie_ldx(sym) # R_SPARC_TLS_IE_LDX
add %g7, %o0, %o0, %tie_add(sym) # R_SPARC_TLS_IE_ADD
```
The sethi and add form the GOT offset. In an executable a
non-preemptible symbol is optimized to Local Exec, as GNU ld does: the
sethi holds the complement, the add becomes an xor, and the load becomes
a register move, or a nop when source and destination are the same. The
thread pointer add is unchanged.
Co-authored-by: Kirill A. Korinsky <kirill at korins.ky>
Co-authored-by: LemonBoy <thatlemon at gmail.com>
Co-authored-by: Alex Rønne Petersen <alex at alexrp.com>
[mlir][x86] Decouple Accumulator from ADD-Based Vector Contractions (over loops) (#204327)
This path transforms `vector.contract(A, B, Acc)` into
`vector.contract(A, B, 0) + Acc` to decouple the contraction computation
from the accumulator update for contractions over the loop.
[ELF,SPARC] Handle GOTDATA relocations (#213497)
A PIC data reference forms the symbol's GOT offset with
R_SPARC_GOTDATA_OP_HIX22 and R_SPARC_GOTDATA_OP_LOX10 and loads through
it, with R_SPARC_GOTDATA_OP marking the load:
```
sethi %gdop_hix22(sym), %g1
xor %g1, %gdop_lox10(sym), %g1
ldx [%l7 + %g1], %g1, %gdop(sym)
```
When the symbol is neither preemptible nor absolute, optimize the load
to `add %l7, %g1, %g1` over the symbol's GOT-relative address. An
absolute symbol keeps the GOT load, as it can be arbitrarily far from
.got and we don't want the X86_64::relaxOpt complexity.
Co-authored-by: Kirill A. Korinsky <kirill at korins.ky>
Co-authored-by: LemonBoy <thatlemon at gmail.com>
Co-authored-by: Alex Rønne Petersen <alex at alexrp.com>
[JITLink] Remove LTmp workaround now that LLVM requires C++17 (#213428)
C++17 guarantees the postfix-expression naming the called function is
sequenced before evaluation of its arguments, so
L->linkPhase1(std::move(L)) is well-formed without the LTmp indirection.
This should also be implemented by MSVC now. See here (P0145R3 and
P0400R0):
https://learn.microsoft.com/ar-sa/cpp/overview/visual-cpp-language-conformance?view=msvc-170
Disclaimer: As I have no LLVM Windows machine with MSVC at hand, I was
not able to test it myself.
Co-authored-by: Claude <noreply at anthropic.com>
[lldb] Reimplement PythonCallable::GetArgInfo without executing Python code (#213378)
`b05a5d0a` added an arity-trimming step to the shared
`ScriptedPythonInterface::Dispatch`: extensions are now allowed to
define methods with trailing parameters as optional
(`num_children(self)` vs. `num_children(self, max_count)`), so before
calling into a method, `Dispatch` needs to know how many positional
arguments it actually accepts and drop any trailing ones we'd otherwise
pass.
That check calls `PythonCallable::GetArgInfo`, which ran a whole
embedded Python script through `inspect.signature` on every call, since
every scripted-extension dispatch goes through it.
For the common case `GetArgInfo()` actually needs to handle fast (plain
Python functions/methods, classes used as constructors, and callable
instances defining `__call__`, i.e. everything `Dispatch<T>()` and
`CreatePluginObject()` ever pass it), the answer is available as plain
data attributes, with no Python bytecode execution required:
[15 lines not shown]
[AMDGPU] Support partial and empty WWM pools for SGPR spills
SGPR lane spilling currently treats the WWM VGPR pool as all-or-nothing. This
can fail compilation when the requested pool cannot be formed, even though
scratch spilling or a smaller spillable pool could make progress.
This PR lets ordinary SGPR spills fall back to scratch when the pool is empty
and lets WWM register allocation use a nonempty partial pool. It keeps the
full-pool requirement for strict WWM/WQM and explicit spill-carrier
preallocation.
The no-pool fallback is recorded in SIMachineFunctionInfo so frame lowering can
provide enough emergency scavenging slots. The state is also serialized to
preserve the behavior across MIR round trips.
