[compiler-rt] Add 'csan' library for the concurrency sanitizer (#225782)
Summary:
Adds the runtime for the concurrency sanitizer, both CPU and GPU.
Fundamentally, this works using the following pseudocode:
```c
static u64 watchpoints[N]; // Hash-indexed, zero is empty.
// Emitted before the access, so we never trip on our own write.
void check_access(volatile void *addr, u32 size, u32 type) {
// Every access probes. A read conflicts only with a watched write, a
// write conflicts with either.
if (u64 *wp = find_watchpoint(addr, size, type))
consume(wp, this_pc()); // Hand our location to the owner.
if (!should_sample()) // Wave-uniform, 1-in-N chance.
return;
[19 lines not shown]
[LLVM] Add a ConcurrencySanitizer LLVM pass (#225780)
Summary:
Add an LLVM ConcurrencySanitizer pass to instrument memory and atomic
operations for a sampling race detector. Fundamentally the pass iterates
through every candidate memory operation and adds a check above it.
Unlike TSAN, we do not replace any operations, which heavily simplifies
the instrumentation.
We do not re-use the TSAN instrumentation because the appraoches are
different enough and the passes simple enough that the extra code is not
a large concern. More differences will be added in the future, so
keeping them separate is easlier.
RFC:
https://discourse.llvm.org/t/rfc-a-thread-concurrency-sanitizer-for-gpus-in-compiler-rt/91113
[IR] Add the sanitize_concurrency function attribute (#225779)
This introduces a new concurrency sanitizer attribute to be used with
the runtime of the same name. This has no major change and simply
updates the IR to handle it properly. This attribute is invalid when
combined with the thread sanitizer one as they are mutually exclusive.
RFC:
https://discourse.llvm.org/t/rfc-a-thread-concurrency-sanitizer-for-gpus-in-compiler-rt/91113
[libc] Implement CheckedPtr utility (#234646)
This adds CheckedPtr, a wrapper around cpp::unique_ptr (along with a
make_unique helper) that asserts in !NDEBUG builds that the pointer was
checked for nullness before it is dereferenced, destroyed or overwritten
(e.g., move assignment). This way, a missing check is caught even if the
allocation succeeds.
The idea is replace AllocChecker (which was meant to serve a similar
purpose, but the checking part didn't materialize so far). See
<https://discourse.llvm.org/t/rfc-error-checking-apis-for-memory-allocation-and-management/92511>
for more details.
A few notable design choices:
- operator bool() is non-const so the checked flag does not need to be
mutable (avoiding data races on concurrent const reads)
- moving a CheckedPtr disarms the source and leaves the destination
unchecked (similar to llvm::Error), forcing the new owner to check it
- the class wraps unique_ptr rather than replacing it, as I expect we
[5 lines not shown]
[mlir][vector] Drop Linalg dependency from transfer full/partial split
The `LinalgCopy` strategy of the vector transfer full/partial split was the
only reason MLIRVectorTransforms depended on the Linalg dialect: its read
slow path filled the temporary buffer with `linalg.fill`. The copies were
already `memref.copy` since https://reviews.llvm.org/D118028.
The temporary buffer is a statically shaped alloca holding exactly one
vector, so fill it with a `vector.broadcast` of the padding value stored
through `vector.type_cast`. This is the same buffer access the
`VectorTransfer` strategy already uses.
As the strategy no longer creates any Linalg ops, rename it from
`LinalgCopy` to `MemRefCopy`
[libc] Add a hex parsing helper for network address parsing (#237359)
Both str_to_ipv6 and ether_aton need to parse short runs of hex digits,
and both were doing it in their own way. ether_aton had a hand-rolled
loop, while str_to_ipv6 used strtointeger, which meant it had to
explicitly reject all of the things strtointeger accepts but we don't
want (leading whitespace, signs, 0x prefixes), and then check the length
of the result.
This patch adds a consume_hex<T> function, which parses at most
2*sizeof(T) hex digits from the front of a string_view, and uses it in
both places. Any characters after that are left for the caller to deal
with, which works out nicely since both callers already need to check
for a separator after each component. hex_digit_value moves into the new
header as well.
I've put the header into __support/net, since that's where its users
are, though there's nothing network specific about it. The main reason
for not putting it somewhere more generic, is the question of naming it
[4 lines not shown]
[mlir][arith][linalg][tosa] Move sharding interface impls into extension libraries
The external sharding models of Arith, Linalg and Tosa were compiled into
`MLIRArithTransforms`, `MLIRLinalgTransforms` and
`MLIRTosaShardingInterfaceImpl`, so every user of these libraries linked
the Shard dialect, `MLIRShardingInterface` and, for Linalg,
`MLIRShardTransforms`, even when sharding is never used.
Move them into extension libraries, following the existing Tensor,
Func and Bufferization sharding extension convention. As there, the models are
registered by the new `registerAllExtensions` calls.
[PassBuilder] Add extra ModuleInliner/SROA/SimplifyCFG/LICM to late FullLTO passes (#197678)
This patch is in preparation to removing pre-LTO vectorisation
(https://github.com/llvm/llvm-project/pull/195886).
After removing the pre-LTO vectorisation there are various missed
optimisation opportunities which show up in benchmarks.
The pre-LTO SLP creates LTO inlining opportunities. When pre-LTO SLP is
removed, we lose the SLP -> Inliner phase sequence, so we add a second
LTO inliner to re-establish it.
The second module inliner uncovers opportunities for SROA, particularly
around passing
aggregate arguments and return values.
Likewise, we've noticed cases with missing opportunities for SimplifyCFG
and LICM, caused by the change of the order of optimisation phases.
This patch adds ModuleInliner, SROA, SimplifyCFG, and LICM late in LTO
optimisation pipeline.
[MLIR][XeGPU] Build float min/max neutral values the format can represent (#233471)
`createReductionNeutralValue` gives a float min/max reduction an
infinity (`maximumf`, `minimumf`) or a quiet NaN (the kinds that ignore
NaN). The WG-to-SG distribution and the peephole split of
`vector.multi_reduction` start their new reductions from this value.
Some float formats cannot represent these values:
- A format without infinity, such as `f8E4M3FN`: `APFloat::getInf`
returns NaN. So `maximumf` and `minimumf` start from NaN, and every
result is NaN.
- `f4E2M1FN` has neither infinity nor NaN, so building the value aborts.
- `f8E8M0FNU` has no negative values, so building -inf for `maximumf`
aborts.
This patch chooses the value by what the format supports:
- A kind that ignores NaN still uses a quiet NaN if the format has one.
- Otherwise it uses the infinity that never wins.
- A format without that infinity uses its lowest (for max) or highest
[10 lines not shown]
[PassBuilder] Remove LoopFullUnrollPass from before LoopVectorize in the FullLTO pipeline (#197176)
This patch is in preparation for disabling the pre-LTO vectorisation in
the FullLTO pipeline (https://github.com/llvm/llvm-project/pull/195886).
It removes the `LoopFullUnrollPass` run before `LoopVectorizePass` in
the full LTO pipeline.
Running the `LoopFullUnrollPass` before `LoopVectorizePass` causes
functions to be unrolled before the vectoriser has had a chance to
vectorise them (in the modified pipeline where vectorisation does not
happen in the pre-link stage).
Reland "[mlir][LLVMIR] Decouple ModuleTranslation from the OpenMP dialect" (#236777)
`ModuleTranslation.h` included `OpenMPDialect.h` and owned
OpenMP-specific state. Every consumer of the LLVM IR export, and every
dialect whose header includes `ModuleTranslation.h` (NVVM, XeVM),
therefore depended on the OpenMP dialect, and `MLIRNVVMDialect`,
`MLIRXeVMDialect` and `MLIRLLVMIRTransforms` linked `MLIROpenMPDialect`
for that reason only.
This PR decouples this by adding a TypeID-keyed
`ModuleTranslation::Extension` mechanism (similar to ones seen in
bufferization or the transform dialect) for state that lives as long as
the translation and move the OpenMP state into an extension owned by the
OpenMP translation.
Reland notes: The original commit broke `BUILD_SHARED_LIBS=ON` builds
with `-Wl,-z,defs`: `FIRDialect` attaches
`omp::DeclareTargetDefaultModel` to `fir::GlobalOp` but only linked
`MLIROpenMPDialect` transitively through
[2 lines not shown]
Keep other writers off S3 bucket datasets
## Problem
truenas_s3 relies on middleware to keep every other writer off a BUCKET bucket's dataset: a foreign write is served under stale ETags and version IDs, mints no version, skips audit and defeats object lock. The guard only matched a writer's own path at or inside a bucket, and only when the writer was saved. A writable share or task on an ancestor reached the bucket through its child mount, a cloud sync MOVE or an rsync push with `--remove-source-files` deleted from it, bucket create and recover never looked at existing writers, a dataset rename left the bucket row on the old name, and datasets could be created, cloned or renamed into a bucket. Several single-path writers skipped the check entirely (iSCSI and NVMe-oF file backends, VM and container devices, `filesystem.mkdir`/`put`, VM disk conversion, manual update uploads and a cloud backup cache path), a recursive permissions change could rewrite the bucket's private state below its root, and webshare ignored its `readonly` flag.
## Solution
- **Above a bucket.** An enabled share or task whose writes cross into child mounts, a home directory and the anonymous FTP root are refused above a BUCKET bucket's dataset, judged by the resolved path. NFS exports stop at child mounts and read-only shares only read, so both stay allowed. A disabled entry is checked when it is enabled, and rsync and cloud sync re-check before each run and write nothing when refused.
- **Writers.** A cloud sync MOVE push and an rsync push that removes its sources count as writers, and webshare honours `readonly`.
- **Reverse checks.** Each share/task attachment delegate lists its enabled writers from the same `local_path_info` the save-time check uses. Bucket create and recover, a new managed root and a rename that carries a bucket are refused where one of them, a home directory or the anonymous FTP root would reach it. The error names no writer because the S3 service relays it to its clients, so the writers are logged instead.
- **Nesting and rename.** New `validate_create`/`validate_rename` delegate hooks let S3 refuse `zfs.resource.create`, snapshot clones and renames that put a dataset inside a bucket's dataset or onto a name a bucket row still holds, and a `no_unmount` rename that moves a bucket. Bucket rows follow a dataset rename and the S3 service restarts onto the new mount, and a bucket's dataset must be mounted at `/mnt/<name>`.
- **Single-path writers.** iSCSI and NVMe-oF file backends, VM RAW and CDROM devices, container filesystem devices, `filesystem.mkdir`/`put`/`file_receive`, `vm.device.convert` and `update.file` destinations and the cloud backup cache path are refused at or inside a bucket, and a recursive permissions change inside a bucket outside `s3data` is refused.
- **Async call keyword arguments.** `Middleware._call` dropped keyword arguments when the target method was a coroutine, so `call2` never delivered them to an async method; it now forwards them as the executor path and `call_sync2` already do.
- **Alert.** A daily alert names each bucket and the writers that reach it, for configurations from before these checks; the SMB path alert leaves that case to it.
The export test now expects a writable share on a DATASET export above a BUCKET child to be refused and a read-only one to pass. The export is created NFSv4 because SMB refuses a share whose child mount has another ACL type.
[libc] Move nullptr_t definition to cstddef/nullptr_t.h (#236712)
cpp::nullptr_t was previously defined in type_traits/is_null_pointer.h,
which is an odd place for it. In standard C++, nullptr_t is defined in
<cstddef>. However, putting it directly into the umbrella cstddef.h
pulls in byte, which depends on type_traits.h for enable_if_t and
is_integral_v, creating an include cycle if type_traits wants to use
nullptr_t.
To solve this, I've placed nullptr_t into a granular
src/__support/CPP/cstddef/nullptr_t.h header (following the precedent of
headers in utility/ and libc++'s __cstddef/nullptr_t.h).
With this change:
- cstddef.h re-exports nullptr_t from cstddef/nullptr_t.h.
- is_null_pointer.h includes cstddef/nullptr_t.h and uses cpp::nullptr_t
without circular dependencies.
- unique_ptr.h, LibcTest.h, and tests include cstddef/nullptr_t.h
directly and use nullptr_t in place of decltype(nullptr).
[3 lines not shown]
