Parallelize brt_pending_apply() for multiple vdevs
Unlike DDT, sorted by the checksum prefix, BRT uses salted hash.
It means that we can't pre-sort the accesses for better caching,
and with BRT size getting bigger that dbuf cache, there is a high
chance that each lookup will require a 8KB block decompression.
It is not as bad as for 32KB DDT block, but still.
Since each vdev in BRT is completely independent, running several
brt_pending_apply_vdev() instances in parallel should increase both
CPU and (in case of ARC miss) pool utilization.
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18860
Parallelize cloning of deduplicated blocks
When block cloning is used on deduplicated blocks, the additional
references are accounted in DDT instead of BRT. Previously it was
done by brt_pending_apply_vdev(), calling ddt_addref() for each
block with DEDUP flag in the block pointer. The process was single-
threaded, and caused a bottleneck in the sync context, since with
large DDT ddt_addref() needs to decompress 32KB ZAP leaf for each
added block.
This patch changes how pending blocks with DEDUP flag are handled.
Inspead of storing them in per-vdev bv_pending_tree, sorted by
offset (which sorting makes no sense for later DDT updates), store
them in one of 16 AVL trees, sharded and sorted by their checksum
prefix. This way we can simultaneously reduce lock contention on
pools with small number of vdevs, process the shards by separate
sync threads accessing/decompressing different ZAP leaves, and
access ZAP leaves in hash order to increase chances of decompressed
dbuf cache hits.
[9 lines not shown]
arc: add a few invariant checks in release builds
Convert a couple ASSERTs invariants to VERIFYs to enforce them
in release builds to be able to root-case #18782 kernel panic,
whenever it happens again.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Andriy Tkachuk <atkachuk at wasabi.com>
Closes #18840
(cherry picked from commit 023d44b9ef68f1eff6b97d98c6a16cdb4b93cf76)
arc: fix race between arc_release() and arc_read_done()
Consider the following scenario:
1. arc_release() is called on hdr with one buf, but which has
IO_IN_PROGRESS (reading more raw data in encypted pool while
keeping decrypted data in buf).
2. arc_release() moves hdr to anon state and discards its identity.
3. arc_read_done() is called, adds the 2nd buf to hdr, increasing
b_refcnt to 2.
Now we have hdr in anon state with two bufs and without identity.
Or here's a racing scenario:
1. arc_release() checked that hdr is not in anon state, but before
taking hash_lock
2. arc_read_done() takes hash_lock, moves hdr to anon state, in
case of an error.
[25 lines not shown]
Linux: rewrite snapshot automount facility
This commit has a near-total rewrite of the snapshot mount/expire
facility, to try and modernise it, fix a lot of structural and
implementation issues and improve performance.
Because its a change in architecture, not just implementation, it's not
really been possible to incrementally move the implementation from the
old to the new. So, the source commentary is intended as a standalone
description of how it all works, while this commit message describes
what's changed for readers who know the previous version.
The main conceptual difference is that the previous version assumed that
there would only ever be one mount per snapshot dataset at most, and
that that mount would be on the matching `.zfs/snapshot/<snapname>`
mountpoint, and that this subsystem would be the one responsible for
unmounting. Those limitations just don't exist, as Linux mounts can be
duplicated, moved around, attached to other namespaces, even deleted,
without actually removing access to the filesystem and all without the
[100 lines not shown]
Linux 5.19/6.17: handle differences in how to flush delay workqueue
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
Linux 6.3: follow_down() gains flags arg
We need the flags arg to trigger the snapshot mount. For earlier
kernels, we can emulate it with vfs_path_lookup()
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
Linux 6.18 compat: vfs_parse_fs_string() takes 3 args
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: test that a foreign mount on the snapdir can be detached
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: test that ongoing access doesn't prevent snapdir detach
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: test multiple snapshot automounts via multiple base dataset mounts
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: test snapdir shutdown behaviour when automount is moved elsewhere
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: test snapdir shutdown behaviour when automount is bound elsewhere
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: snapdir_admin_create_destroy: use makedir helper
The rust-coreutils (uutils) mkdir calls llistxattr() immediately after
calling mkdir(), which triggers the automount, and so prevents rmdir
from seeing and deleting the mountpoint dir.
Switching over to our helper that we know only calls mkdir() fixes that
up.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847
ZTS: makedir helper command
The system mkdir command may do other things besides the mkdir()
syscall, which is a problem when we're trying to test the syscall
specifically. This helper only does the mkdir() syscall.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18847:w
Make systemd-udev-settle optional for the import units
systemd-udev-settle.service has been deprecated for years, recent
systemd releases warn about it at boot, and distributions have begun
shipping without it, which turns the hard Requires= in
zfs-import-cache and zfs-import-scan into a broken import: a
Requires= on a masked or removed unit keeps the service from ever
starting. Issue #10891.
Demote the dependency to Wants= and keep the After= ordering. Where
the settle unit exists and completes, the boot is what it always
was: Wants= pulls settle in, the import waits for it, and the
device-symlink guarantee it provided is intact. Where it is masked
or gone the wish is quietly dropped and the import runs anyway,
which beats not importing at all. One deliberate behavior change:
if settle itself fails, on a system so large that enumeration
overruns its timeout, the old Requires= cancelled the import while
the new units go ahead at the timeout mark with whatever has been
enumerated by then.
[29 lines not shown]
DDT: Skip DDT log lookups in ddt_prune_walk()
DDT log is not explicitly locked, and serialized only by the TXG
sync process. Since that stage of ddt_prune_walk() is not in sync
with the TXG sync process, we can't access the DDT log there. We
could possibly lock it, but it seems we can just skip it, since it
is only a performance optimization, while the final LOG lookup is
be done by prune_candidates_sync() any way.
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18889
dbuf: use dirty record size for overridden writes
A level-0 dirty record retains its ARC buffer after dmu_sync() writes it
for an indirect ZIL record. If a newer transaction group changes the
block size before the older one syncs, that retained buffer and the live
dbuf legitimately have different sizes.
The normal fallback is reachable when encryption and dedup are enabled.
dmu_sync() keeps the dedup checksum for the override BP. Syncing-context
policy then enables dedup, but zio_write_bp_init() cannot reuse an
encrypted dedup override and restores the regular write pipeline.
dbuf_write() built its ABD from the retained buffer but passed the live
dbuf size to ZIO. After growth, compression can therefore read beyond
the retained allocation. After shrink, the older transaction group's
block is instead written with the newer, smaller LSIZE and truncated
data.
This was observed as a 527 KiB compression request against a retained
[20 lines not shown]
nvpair: Fix operator precedence
59dc88602e23a436440e4164c6d9401da8f0dff2 made a mistake when doing a
check, which can cause us to continue processing when we should return
EFAULT.
Reported-by: Grok 4.5 Build Beta
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Richard Yao <richard at ryao.dev>
Closes #18874
ZTS: don't read a command's exit status as a missing binary
log_neg_expect() treats an exit status of 127 as a missing binary and
fails before it looks at what the command printed. That is only a
convention of the shell, and a command is free to return 127 for its
own reasons. fio returns the number of jobs which failed, so a run of
127 failing jobs is reported as though fio were not installed.
no_space/enospc_rm fills a pool with 200 fio jobs and requires them to
fail with ENOSPC. How many of them get that far varies with timing,
and on the occasions it comes to exactly 127 the test fails with
fio ... unexpectedly exited 127 (File not found)
even though the output holds the expected message. A dozen runs here
landed between 183 and 199, and the failure seen in CI reported 127.
Only read 127 as a missing binary when the expected output is absent.
A command which printed what was asked of it plainly ran, so nothing
[5 lines not shown]
unit/cityhash: test the cityhash hash functions
Cover known answers, the 1/2/3-argument specialized forms against
cityhash4, and distinct inputs.
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18880
unit/sha2: add Monte Carlo test
SHAVS chain MD[i] = SHA(MD[i-3] || MD[i-2] || MD[i-1]) run for 1000
rounds, for SHA-256, SHA-512 and SHA-512/256. The seeds and expected
digests are the COUNT = 0 vectors from the NIST CAVP byte-oriented test
vectors (shabytetestvectors).
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18879
build: Fix release detection when build dir is not source dir
When building outside of the root source directory, configure fails to
detect that the source for the build is a git repository because the
build directory is checked if it is a git repository. Instead check
source directory and use the source directory for generating the
release. Check for the .nogitrelease file in the source directory too.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #18891
CI: don't fail a passing job when a log reader has already exited
Once a VM's tests finish the runner kills that VM's log reader. If the
reader is already gone, kill reports ESRCH, and since the script runs
under set -eu that ends it and the job is reported failed after the
tests have already passed.
That is what turned the fedora44 run in
https://github.com/openzfs/zfs/actions/runs/30772421272 red:
vm1: Results Summary
vm1: PASS 1151
vm1: FAIL 2
vm1: SKIP 6
...
qemu-6-tests.sh: line 143: kill: (20700) - No such process
##[error]Process completed with exit code 1.
All 13 failures in that run were on the expected list and neither VM
[13 lines not shown]
CI: publish the per-VM test counter atomically
Each VM's log reader keeps a running test count in /tmp/ctr-vm$ID and
reads every other VM's counter to print the combined progress figure.
The counter is published with a plain redirect, which truncates the file
before it writes, so a reader can see it empty. `read` then returns 1,
and because the reader inherits set -eu that ends the reader subshell.
The VM keeps running and its results are recovered later from the
artifact, but its output stops being prefixed into the live log from
that point on, with nothing said about why.
Seen on fedora44 in
https://github.com/openzfs/zfs/actions/runs/30772421272. vm2's last
prefixed line is refreserv/cleanup at 01:57, carrying counter 745, while
vm1 keeps reporting vm2 frozen at 746 for the remaining 38 minutes: the
reader incremented the counter and died before printing that line. vm2
itself ran on until 02:14 and its 899/11/6 summary never reached the
live log.
[9 lines not shown]
Batch object reallocation syncs in zfs receive
When an incremental stream reallocates an object at a different dnode
slot count, receive_object() must wait for the old dnode's free to
sync out before the object number can be claimed again, because
dnode_check_slots_free() only accepts slots whose final dirty txg has
synced. The receive writer did this with one txg_wait_synced() per
reallocated object, so a dataset that ever changed dnodesize (for
example dnodesize=auto, as the root-on-ZFS guides suggest) receives
such incrementals at a few KB/s while the pool syncs mostly empty
txgs (#11353).
Batch those waits. The frees are still issued immediately, since
many frees can share one txg, but the record whose claim would need
the sync is parked on a per-receive list and the affected dnode slot
ranges are tracked in a small AVL tree. Later records that touch a
parked range park behind it in stream order while unrelated records
keep flowing. Once the batch reaches zfs_recv_defer_batch_size
(default 32 MB, 0 restores the old behavior) or the stream ends, a
[38 lines not shown]
Predict and throttle buffers dirtied by the sync context
Block cloning may cause lots of dirty buffers in the sync context
for BRT/DDT updates, while in open context dirtying only indirect
blocks, and so remaining almost invisible for write throttling.
With active cloning I was able to make dirty reach 16GB and more
while having the limit of only 4GB.
In case of cloned/deduped blocks delete the problem is not so bad,
thanks to zfs_max_async_dedup_frees limit. But on smaller systems
or with higher tunable setting it might still be a problem.
To handle the cloning make dmu_tx_count_clone() predict amount
of the dirty buffers that might be produced by the sync context
for this transaction, summarise those for each transaction group,
and account them in write throttling as real dirty buffers.
When sync context as part of DDT/BRT writing start dirtying MOS
buffers, subtract that dirty from the predicted reserve. This way
completions of the DDT/BRT write ZIOs will gradually open the
[14 lines not shown]
man: zvol_request_sync is not ignored under blk-mq
The zfs.4 entry for zvol_request_sync claims it "is ignored when
running on a kernel that supports block multiqueue (blk-mq)". This
has never been true. The sentence and the blk-mq support it describes
landed in the same commit (6f73d0216 "zvol: Support blk-mq for better
performance"), which added
if (zvol_request_sync)
force_sync = 1;
to zvol_request_impl() -- the function shared by both submission
paths, reached from zvol_submit_bio() and from zvol_mq_queue_rq()
alike. No blk-mq guard was added there then, and none exists now.
Taken literally the sentence is worse than inaccurate: HAVE_BLK_MQ
was removed in 9601eeea1 because every supported kernel has blk-mq,
so the documented condition is always satisfied and the parameter
would never do anything.
[18 lines not shown]
zstream: Add zstream raw command
Introduce a "zstream raw" command.
This command allows zvol send streams to be applied to a raw disk image
or block device, with support for incremental sends and compression.
Encrypted (raw) streams are not supported, but compression, large
blocks, and embedded write records are handled.
Contiguous writes can optionally be buffered and issued together with
pwritev(2).
Block devices supporting TRIM/UNMAP use the platform-appropriate
BLKDISCARD or SIOCGDELETE ioctl to handle FREE records.
Raw file volume images use the platform-appropriate hole punching
operation to handle FREE records if supported by the underlying
filesystem.
[7 lines not shown]
zstream: Annotate zstream_usage as noreturn
And remove dead code following its use.
Sponsored-by: Klara, Inc.
Sponsored-by: Railway Corporation
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Signed-off-by: Ryan Moeller <ryan.moeller at klarasystems.com>
Closes #18565
