[Offload][NFC] Move plugins under liboffload
Liboffload is the only library calling the plugins directly now, so move
the code inside it to reflect this change.
Also, adjust the CMake targets to also reflect this change by moving
from omptarget prefixes to LLVMOffload ones.
Assisted by Claude.
[libc] Add IPV6_PKTINFO, IPV6_TCLASS and related socket options (#243838)
This patch adds the IPV6_PKTINFO, IPV6_RECVHOPLIMIT, IPV6_RECVTCLASS and
IPV6_TCLASS socket options as well as struct in6_pktinfo (used by
IPV6_PKTINFO) to <netinet/in.h>.
The interface index in struct in6_pktinfo is unsigned int, which matches
other implementations (the kernel uapi header uses a plain int).
Assisted-by: Gemini
[OpenMP][OMPTarget] Restore attached host pointers to their value before the FROM transfer (#233650)
Since OpenMP 6.1, an attached pointer may be re-pointed on the host. A
FROM of its
containing storage must then keep the host's current value:
```c
S s = {&a[0], 0};
#pragma omp target enter data map(to : s, s.p[0 : 1])
s.p = &a[1]; // host only
#pragma omp target exit data map(from : s)
// main: s.p == &a[0], the value saved at attach time
// now: s.p == &a[1]
```
The runtime now saves the host pointers that a FROM transfer overwrites
right before
the transfer, and writes them back once the transfer completes.
Previously it wrote
[24 lines not shown]
[MLIR][OpenMP] Support device shared memory for recipe ops
Some recipe ops (`omp.private` and `omp.declare_reduction`) can hold
stack allocations inside their allocation or initialization regions, and
these regions are inlined in the corresponding blocks of their users when
translating to LLVM IR.
The issue is that, when compiling for a target device, certain stack
allocations must be replaced with device shared memory to account for
the disjoint memory spaces of GPUs when kernels run in Generic mode.
This is sometimes handled by frontends by emitting `omp.alloc_shared_mem` and
`omp.free_shared_mem` operations instead of using stack allocations, and
the `StackToShared` MLIR pass addresses any remaining stack allocations
that need to be replaced.
None of those solutions are well suited for these recipe ops, which
might need to use device shared memory for some instantiations but not
others, based on context and usage of the resulting storage. Instead,
what this patch does is to identify the cases where device shared memory
[17 lines not shown]
[MLIR][OpenMP] Introduce a ReductionVarsInfo structure, NFC
The MLIR to LLVM IR translation process for private and reduction
clauses is relatively similar, given that both are based on inlining
regions of a top-level template operation (`omp.private` and
`omp.declare_reduction`) pointed at by symbol, and mapping entry block
arguments to values as appropriate.
This similarity means that the same kind of information needs to be
passed around to helper translation functions. For privatization, this
is done using the `PrivateVarsInfo` structure but for reductions,
several vectors are passed instead.
This patch introduces a `ReductionVarsInfo` structure that mirrors the
existing `PrivateVarsInfo` to simplify the handling of this data and
converge on a similar pattern.
Assisted-by: Claude Opus 4.8
[Aarch64] lower `VECTOR_SHUFFLE_VAR` to NEON `TBL` (#240480)
TBL is byte based so this requires a magic constant multiply to line
everything up properly. SVE lowering can do this better, but I think it
makes sense to start with NEON.
---------
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[libc][math] Migrate math polynomial evaluations to estrin_polyeval and loop_polyeval. (#243741)
This is mostly NFC, the only changes it might happen is that the
`estrin_polyeval` and Horner loops templates have more code-size-aware
optimizations than the manual expansion they are replacing in this PR.
- Replace manual Estrin polynomial evaluations with
`fputil::estrin_polyeval`.
- Replace non-unrolled Horner loops in `exp2_accurate_128.h`,
`powf_double_eval.h`, and `sinhcosh_accurate_128.h`, and cold DFloat128
slow-path polynomials in `sincos_eval.h` and `tan.h`, with
`fputil::loop_polyeval`.
- Use array overloads of fputil::polyeval and fputil::altpolyeval in
atan_utils.h, sinhcosh_accurate_128.h, and tanh_accurate_128.h.
Assisted-by: Gemini
[MLIR][OpenMP] Refactor PrivateVarsInfo, NFC
The `PrivateVarsInfo` structure holds, for each privatized variable,
multiple related values such as its corresponding MLIR entry block
argument, privatizer, etc. This is represented as a vector for each
property, which has the disadvantage of potentially getting out of sync
when new values aren't added to all vectors.
The proposed change is to create a nested structure holding the
information for a single variable and have `PrivateVarsInfo` hold a
vector of these structures. This prevents the previously mentioned issue
and also simplifies its use, as it no longer requires iterating in
lockstep over multiple vectors.
Assisted-by: Claude Opus 4.8
[lldb-dap][NFC] Swap show_user and error code in DAPError (#243859)
The `show_user` args is set more often than the error code.
I prefer not to write `llvm::inconvertibleErrorCode()` every time I need
to show the error to the user.
[flang][OpenACC] Fix lowering crash on DO WHILE loop constructs
A loop construct associated with a DO WHILE crashed lowering: reading the
DO WHILE's loop control as bounds aborted, both when the DO WHILE is lowered
unstructured and, with -lower-do-while-to-scf-while, when it is lowered
structurally. A DO WHILE has no bounds and no induction variable, so lower
the construct to an acc.loop without bounds that holds the whole DO WHILE,
as for an unstructured loop, with nothing to privatize.
[flang][OpenACC] Lower unstructured seq loops in kernels
An `acc loop seq` whose loop has an early exit is lowered as an
unstructured acc.loop that privatizes its induction variables, but its
DOs then hit the "unstructured do loop in acc kernels" TODO when they
are lowered inside that acc.loop.
When privatizeInductionVariables privatizes the induction variable of a
DO owned by an unstructured acc.loop, record that DO.
genOpenACCLoopFromDoConstruct lowers a recorded DO as plain control flow
inside its acc.loop instead of reporting the TODO when that acc.loop is
seq; no parallel loop then runs the DO on more than one thread. The
record is dropped once the construct's evaluations are lowered.
The tests cover seq loops nested in structured loops, directly in a
kernels region, inside an unstructured IF, SELECT CASE, BLOCK or DO
WHILE, and a combined `kernels loop seq` construct. They also cover the
shapes that still hit the TODO: a DO the seq loop does not own (a plain
DO in its body, or a DO next to the DOs of a collapse(force:2), also
[7 lines not shown]
[AMDGPU] Only copy SGPR inputs of vector PHIs in SIFixSGPRCopies (#226213)
Since #169038, `processPHINode` legalizes every vector PHI. With AV
classes as the default (#166482, #166483), this adds a `COPY ...,
implicit $exec` for inputs that differ from the PHI only in AV-ness.
Remove the PHI path from `legalizeOperands`. `processPHINode` now copies
only the SGPR inputs of vector PHIs, at the end of the incoming block
(#168761, i1 lane masks). Other mismatched inputs, including VGPR/AGPR
mixes and subregisters, are left to PHI elimination. Propagating the
AGPR class back to operand PHIs now only applies to SGPR PHIs.
This reduces register pressure and spilling in kernels with many
divergent loop PHIs. Because the other copies now come from PHI
elimination, register allocation can change in other kernels too, in
either direction.
Co-authored-by: Matt Arsenault <arsenm2 at gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
[libc++] Destroy the elements of a vector in reverse order (#243091)
While the destruction order of the elements in a vector isn't prescribed
by the standard, a lot of programs rely on the elements being destroyed
in reverse order. This assumption was broken by #217631. Since there is
no other reason to prefer a particular direction, users being broken by
this is a good reason to undo the change in behaviour.
This adds a test in `libcxx/`, since it's simply documenting the current
behaviour to avoid changing it accidentally. We may change the behaviour
in the future if there is a reason to do so.
[X86][ACEV1] Record the program model and correct the outer products
Record the program model as the intrinsics are selected, keyed on the
intrinsic as AMX does, rather than from a 66-entry opcode table kept in
step by hand; using both models is now diagnosed, not asserted. Drop the
K dimension the internal outer products never used, and swap the TOP4MX
pseudos' sources, which reached the instruction reversed because it
takes its VVVV operand first. Widen the insert pseudos' index to i32 so
an out-of-range one truncates as for AMX, check it in clang against the
architectural 0-15, and restore the GCCRegNames grouping BSR0 reflowed.
[LSR] Only apply postincrement discount on address uses (#149341)
Cost::RateRegister currently applies the pre/postincrement discount to
all uses when the addrec is of the correct form, meaning it will be
applied even when there's no load/store that can be postincremented. Fix
this by checking that the LSRUse kind is Address.
[libc++][prototype] Add the std::cppNN release namespaces from P3866R0
Each std::cppNN namespace contains a using-directive for std, so that every
component that was never revised is reachable through it. Components that were
revised declare an alias in the release namespace next to their definition, which
hides the std entity of the same name from qualified lookup:
std::cpp29::vector<int> -> std::vector2<int> (declared in __vector/vector2.h)
std::cpp26::vector<int> -> std::vector<int> (through the using-directive)
std::cpp29::string -> std::string (through the using-directive)
Findings, captured in libcxx/test/std/library/cpp_release_namespaces:
- `using namespace std::cpp29;` makes every revised name ambiguous, since
using-directives are transitive for unqualified lookup.
- Nested namespaces must be mirrored: std::cpp29::pmr hides std::pmr unless it
re-exposes it with its own using-directive.
- std::vector and std::vector2 don't interoperate at API boundaries.
- With Clang, CTAD through the alias from an iterator pair only works if the
alias is variadic.
[9 lines not shown]
Update to zlib 1.4
ok deraadt
The security fixes are already in tree, the remaining relevant fixes
for OpenBSD include these:
- Deliver all gzip data before reporting error
- Fix truncated stream detection in gzread()
- Fix pending skip handling in gzread()
- Fix loss of zero fill on repeated gzseek()'s when writing
- Fix inaccurate total_in after Z_NEED_DICT returned
- Detect and report bad deflateSetDictionary() calls
- Avoid undefined behaviors in C language usage
- Don't assume NULL is defined in compress.c and uncompr.c
- Clarify documentation
[libc++][prototype] Turn the vector copies into std::vector2 (P3866R0)
Prototype of P3866R0 (V2: An Evolution Path for the Standard Library), pretending
that std::vector2 is introduced in C++29 as a version of std::vector that always
uses the size-based layout (begin pointer + size + capacity), regardless of the
ABI configuration.
- __vector/vector2*.h: renamed vector to vector2 and gated everything on C++29.
vector2_layout.h only keeps the size-based layout. There is intentionally no
vector2<bool> specialization.
- __fwd/vector2.h: forward declaration.
- __iterator/wrap_iter.h: __wrap_iter befriends vector by name, so it has to
befriend vector2 too.
- <vector>, CMakeLists.txt, module.modulemap.in, modules/std/vector.inc and
generate_iwyu_mapping.py: wire up the new headers.
- Tests: renamed vector to vector2, replaced the language-mode directives by
`REQUIRES: std-at-least-c++29`, added vector2 overloads/aliases to the test
support headers, and updated the ABI test with the actual vector2 sizes.
No code is shared with std::vector on purpose, except for __if_likely_else.
[CodeGenPrepare][AMDGPU] Fold urem by constant of loop increment (#242110)
Extend foldURemOfLoopIncrement to constant remainder amounts. For
constant divisors also allow a constant step smaller than the divisor,
'or disjoint' offsets, and a non-simplifiable start value (the initial
remainder is computed in the preheader). Power-of-2 divisors and urems
that do not execute on every iteration are skipped.
Co-authored-by: claude Opus5.5
[MustExecute] Remove findInContextOf and checkForAllContext
They have no users left now that the Attributor uses isInContextOf and
checkForAllTerminatorsInContext. Also remove getCurrentInst and count
from MustBeExecutedIterator, which were only used by findInContextOf.
The iterator itself is still used by the must-be-executed-contexts
printer.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[libc] Add memchr benchmark
There's a bit of a gap between LLVM libc's memchr implementation and
glibc's memchr implementation, even when both have equivalent vector
instructions available. Add some benchmarks to aid in the optimization
effort.
Assisted-By: LLM
Reviewers: kaladron
Pull Request: https://github.com/llvm/llvm-project/pull/243751
[NVPTX] Revert #191890, #215897 SETP+SELP commutativity (#243807)
Revert `SETP` commutativity and predicate inversion patch #191890 and
its subsequent `SELP` extension #215897. Adapt the reverted code to
avoid undoing subsequent refactors like replacing `SETP_i32rr`,
`SETP_i32ri`, and `SETP_i32ir` with a single flexible `SETP_i32`.
After discussions in https://github.com/llvm/llvm-project/pull/191890
highlighted several major issues with the original `SETP` commutativity
patch #191890, we need to revert it (and the follow-up `SELP` patch
#215897) so that they can be re-implemented later with a different
approach, and avoid blocking other important work while doing so.
[libc] Do not core dump when failing to meet benchmark reqs
Prefer reportFatalUsageError which doesn't core dump and just cleanly
exits.
Reviewers: kaladron
Pull Request: https://github.com/llvm/llvm-project/pull/243750