L2ARC: do not feed a device while its rebuild is pending
l2arc_add_vdev() leaves the device at l2ad_start with l2ad_first set,
marks it l2ad_rebuild, and starts its feed thread. The rebuild runs
later from the spa async thread and only then restores the geometry the
on-disk header describes.
A feed in that window writes at l2ad_start believing the device is
empty, and its headers go on the head of l2ad_buflist while the rebuild
appends the restored ones at the tail. l2arc_evict() walks that list
from the tail and stops at the first header outside the range it is
clearing, so once list position stops tracking device offset the walk
exits early, headers the hand is about to overwrite are never destroyed,
and the writes that replace them charge the space twice. vs_alloc
passes vs_space and zpool(8) reports 16.0E free.
l2arc_dev_invalid() rejected a device with l2ad_rebuild or l2ad_trim_all
set until b525525b4 made the feed threads per-device; neither flag has
had a reader since. Restore the test, after SCL_L2ARC is taken so it
[7 lines not shown]
ZIO: Batch lightweight ZIOs
This further develops concept of ZIO batching from #18921, extending
it from the leaf vdev layer up the stack. Now it covers the case of
I/O scheduler enabled, not covered before. I/O aggregation of the
scheduler can complete several ZIOs at once, creating opportunity
for batching.
This change allows ZIO pipeline stages to return more than one ZIO,
chaining them via the field added in previous commit, renaming it
to more generic io_exec_next. To identify ZIOs to batch, this change
introduces a new ZIO flag ZIO_FLAG_LIGHTWEIGHT, set for members of
a batch. So once zio_done() or possibly other stage handlers return
a list of ZIOs, zio_execute() has now ability to decide which of them
to execute itself, and which to dispatch to taskqueues as before.
With this change my tests of 32KB block writes to 3x 5-wide NVMe RAIDZ1
on 64-core system with scheduler=on show throughput increase from
15.0GiB/s to 16.1GiB/s, while the taskqueue lock contention is
[12 lines not shown]
Linux: Wire RWF_DONTCACHE and POSIX_FADV_NOREUSE
Wire both to Uncached I/O, since that is what they mean -- the data
will not be accessed again after the operation. It allows us to both
reduce ARC/dbuf cache trashing and avoid extra memory copies to/from
scattered and unmapped ARC buffers.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19054
ZTS: resolve two possible divide by zero errors in tests
Resolve two divide by zero errors observed by the CI. Change
how random picking to ensure no divide by zero, and also keep
same semantics.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: tiehexue <tiehexue at hotmail.com>
Closes #19078
ZTS: do not leave a suspended pool behind in the mmp tests
The mmp tests destroy their pool from cleanup through
datasetexists $TESTPOOL && destroy_pool $TESTPOOL
and datasetexists() asks zfs, which cannot open a pool whose I/O is
suspended:
# zfs get name msus
cannot open 'msus': pool I/O is currently suspended
MMP suspends the pool by itself once its writes have not succeeded for
long enough, which is exactly the state a failing mmp test tends to
leave behind, so the guard says the pool is gone, cleanup destroys
nothing, and the pool stays imported. Every mmp test that runs after
it then fails on its own zpool create:
/dev/loop0 is part of active pool 'testpool'
[31 lines not shown]
ZTS: judge mmap_sync by a concurrent fsync, not by a fixed bound
mmap_sync fails whenever one msync() takes longer than two seconds,
and in CI it does, on machines that are merely loaded:
slow msync: 2318.177000 ms
slow msync: 4043.686000 ms
The bound was raised once already, in c4d1a19b3 ("ZTS: increase
timeout of mmap_sync_001_pos"), and the ctime_001_pos failure fixed in
8486ea36f shows the same CI stalling a creat() for twelve seconds, so
no fixed bound is going to hold there.
An msync() goes through zpl_fsync(): since a18c9edda ("Linux: sync:
remove async/sync accounting") it writes the dirty pages out with
zfs_putpage() and ends in zil_commit(), and it never waits for page
writeback. So what the bound really measures is the ZIL commit, which
is the disk. Sampling the kernel stacks of mmap_sync under an IO load
in a VM confirms it: the time goes into zil_commit_impl() waiting for
[28 lines not shown]
Skip interior dnode slots when receiving DRR_FREEOBJECTS
receive_freeobjects() looks up every object of the record with
dmu_object_info() and fails the receive on any error but ENOENT. An
interior slot of a multi-slot dnode answers EEXIST, and that is the
errno zfs receive then reports as "destination already exists". This
is how ZTS send_realloc_files fails in CI:
cannot restore to testpool/newfs at snap3: destination already exists
The sender emits such a record whenever a multi-slot dnode was freed
and a smaller one was later allocated on one of its interior slots: it
frees the leading slots, claims the new object and frees the trailing
slots as three separate records, e.g. for an 8-slot dnode at 10:
FREEOBJECTS firstobj = 10 numobjs = 3
OBJECT object = 13 ... dn_slots = 1
FREEOBJECTS firstobj = 14 numobjs = 4
[24 lines not shown]
ZTS: fill the ARC before l2arc_multidev_scaling_pos measures it
The test sets l2arc_write_max to 4MB/s, starts a background dd and then
requires that L2ARC wrote at least 80% of 4MB/s per device over a fixed
window. What that window measures, though, is not the rate limit but
how fast the dd manages to feed the ARC: a feed thread writes what it
finds on the eviction lists and goes back to sleep, so with a writer
that cannot keep ahead of it the phase falls short of the bound through
no fault of the code under test.
In CI on Linux it fails both ways, on unrelated branches:
NOTE: Single-device writes: 19MB (expected ~48MB)
Single-device writes 19MB below minimum 38MB
NOTE: Dual-device writes: 26MB (expected ~96MB)
Dual-device writes 26MB below minimum 76MB
while passing runs on the same days report 72MB and 158MB for those very
[45 lines not shown]
Wait for the txg that finished a scan to sync
dsl_scan_done() marks the scan DSS_FINISHED in syncing context, and
right before that it asks vdev_dtl_reassess() to update the DTLs and
dirty the config, which the same txg writes out later in its
spa_sync(). spa_activity_in_progress() looks at the state alone:
scanning = (scn->scn_phys.scn_state == DSS_SCANNING);
...
*in_progress = (scanning && !paused && ...);
so "zpool wait -t resilver" and "zpool wait -t scrub" return while the
labels of that txg are still being written; spa_vdev_resilver_done()
notifies the waiters from the async thread, which can run before the
sync ends, so a waiter is not even guaranteed to sleep.
A command issued right after the wait then reads a pool whose labels
are mid-rewrite. ZTS vdev_zaps_004_pos fails on this: it runs "zdb
-PC" straight after "zpool wait -t resilver" and the pool fails to
[28 lines not shown]
zdb: do not adopt an unreferenced spa when the pool fails to open
When spa_open_rewind() fails, zdb looks the pool up in the namespace
to see whether it only failed for a missing log device, and it does
so into the global spa:
if ((spa = spa_lookup(target)) != NULL &&
spa->spa_log_state == SPA_LOG_MISSING) {
spa_lookup() takes no reference, and the open that just failed had
already set spa to NULL. So whenever the pool is in the namespace
but fails to open for any other reason, zdb goes on to fatal(), and
zdb_exit() then spa_close()s a pool it holds no reference on:
zdb: can't open 'testpool': Device not configured
ASSERT at module/zfs/spa_misc.c:1004:spa_close()
zfs_refcount_count(&spa->spa_refcount) > spa->spa_minref || ...
The pool is in the namespace whenever the cachefile names it, so a
[13 lines not shown]
FreeBSD: use sys/abi_compat.h for time32_t
FreeBSD 14.5 defines time32_t in sys/abi_types.h. The SPL typedef now
duplicates it, which breaks the -std=gnu99 module build on stable/14.
Apply the same change as freebsd-src 87632ddf67b.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #19059
CI: update the FreeBSD STABLE CI runner to 14.5-STABLE
Update the FreeBSD STABLE CI from 14.4-STABLE to 14.5-STABLE. Update the
README as well.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #19058
ZTS: tighten the device_access tests
- Wrap the cachefile copy in log_must and require it to be
non-empty, so the negative import tests cannot pass vacuously
on a system without /etc/zfs/zpool.cache.
- Make check_vdevs() list only the pool it is passed instead of
every /dev-backed pool on the host.
- Skip the group with log_unsupported when $DISKS are not real
block devices, instead of failing.
- Wrap the restore_perms() chmod in log_must.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: xfcyhuang <xfcyhuang at aliyun.com>
Closes #19057
ztest: print the fishing seed with %016llx
printf("%016lx", uint64_t) is undefined on ILP32 where uint64_t
promotes to unsigned long long. Cast to u_longlong_t and use
%016llx, as is done elsewhere in this file.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Igor Ostapenko <igor.ostapenko at klarasystems.com>
Signed-off-by: xfcyhuang <xfcyhuang at aliyun.com>
Closes #19055
ztest: sleep while waiting for the fishing thread to seed
ztest_thread_create() busy-waits on zs_jumps with an empty loop
body until the newly created fishing thread jumps the shared PRNG
state. On a single-CPU system, or when the machine is heavily
loaded, the spinning parent can starve the very thread it is
waiting for. Sleep briefly in the wait loop so the fishing thread
gets a chance to run.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Igor Ostapenko <igor.ostapenko at klarasystems.com>
Signed-off-by: xfcyhuang <xfcyhuang at aliyun.com>
Closes #19055
dsl_deadlist_space_range: use correct tree when summing up space
This was found via code inspection: space_range uses dl_cache for
determining the starting point, and then uses dl_tree for the loop.
This works accidentally only because the tree and cache structs have
the same fields (specifically, node and mintxg) at the same offsets.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Mark Musante <mmusante at wasabi.com>
Closes #19053
Fix double free when cloning a block with a pending free
dmu_read_l0_bps() captures the source block pointers for a clone
without consulting dnode_block_freed(). A block whose free is recorded
but not yet synced still has a valid on-disk pointer, so the clone
registers a BRT reference to a block about to be returned to the
allocator, and it is freed twice. On a debug build this trips the
metaslab double-free verifier; on a release build the space map is
silently corrupted.
Check for a pending free and return EAGAIN, as dbuf_read_hole() already
does for buffered reads. zfs_clone_range() waits a TXG and retries, by
which point the block reads as a hole.
Two cases need more than the plain check. A hole has nothing to free,
so guard with !BP_IS_HOLE(bp), otherwise a hole inside a pending free
range fails a FICLONE that master cloned. When the head dirty record
already overrode the BP, only frees after the override count against
it, so use dnode_block_freed_after(); dbuf_free_range() leaves the
[12 lines not shown]
ZIO: Batch vdev children completions
A vdev child ZIO's life after return from the block layer is three cheap
pipeline stages, yet each child costs its own taskq dispatch and context
switch to get there, only to decrement the parent's child count and die.
Even with increased number of taskqueues this can create a huge lock
contention and scheduler overheads, especially on large systems.
To avoid that extra cost collect the leaf children of a RAIDZ, dRAID or
mirror parents as they come back, and only once the last one is in,
process all of their completions, followed by the parent's one, all on
a single thread.
This optimization is mutually exclusive with I/O scheduler, since
delayed completions there may lead to a deadlock, but for that case I
have somewhat alike optimization idea later.
With this change my tests of 32KB block writes to 3x 5-wide NVMe RAIDZ1
on 64-core system show throughput increase from 13 GiB/s to 17 GiB/s,
[4 lines not shown]
zfs-tests: add zhack mos leak/reclaim coverage
- Add a new cli_root zhack test for the mos leak/scan/reclaim flow.
- Verify dry-run leak/reclaim do not change reclaimable leak counts.
- Verify write-mode leak adds the expected reclaimable clone and
space map objects, and write-mode reclaim removes them.
- Verify reclaim idempotency by running write reclaim twice and
asserting the second pass reports "nothing to reclaim".
- Export/import the pool around write-mode operations to match the
zhack mos command import requirements.
- Register the new test in tests/zfs-tests/tests/Makefile.am and
tests/runfiles/common.run.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Martin Minkus <martin.minkus at gmail.com>
Closes #18907
zhack: add mos leak subcommand for local repro
- Add `zhack mos leak` to create synthetic leaked MOS objects for
testing scan/reclaim behavior on disposable pools, mirroring the
existing `zhack metaslab leak`.
- Support `-c` and `-s` counts for leaked DSL clone maps and
leaked SPA space maps, with dry-run default and `-w` apply.
- Print the created object IDs so test scripts can assert expected
candidates and post-reclaim cleanup.
- Document the new subcommand in zhack.1.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Martin Minkus <martin.minkus at gmail.com>
Closes #18907
zhack: add mos scan/reclaim for leaked MOS objects
- Add `zhack mos scan` to enumerate reclaim candidates for
unreferenced empty DSL clone maps and zeroed space maps.
- Add `zhack mos reclaim` with dry-run by default; `-w` applies
frees in sync context using `zap_destroy` and `space_map_free_obj`.
- Enable readable spacemaps during readonly analysis to avoid
false positives when counting referenced metaslab space maps.
- Document the new subcommands in zhack.1.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Martin Minkus <martin.minkus at gmail.com>
Closes #18907
zpool: things get messy with 0 columns
zpool iostat with terminal reporting 0 columns will print out
a very long line. Use columns 80 when query reports 0.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Toomas Soome <tsoome at me.com>
Closes #19051
ZTS: wait for the zhack pids mmp_concurrent_import started
verify_zhack() looks up the processes to wait for by name:
ZHACKPIDS=$(pgrep zhack)
for pid in $ZHACKPIDS; do
wait $pid
so it only waits for the zhacks that pgrep happens to see. One that has
not reached its exec() yet is still named after the shell that forked
it, and one that has already exited is gone; either way the test
continues while a zhack may still hold the pool imported, and the
import_activity_check() that follows then finds activity where it
requires mmp_result: 0. A zhack missed this way is also not counted,
so IMPORT_COUNT can silently disagree with what actually happened.
A CI run failed this way. Two zhacks were started, only one was
reported, the other was still running at the end and was killed by
cleanup:
[21 lines not shown]
ZTS: let the zvol settle before alloc_class_016_pos destroys the pool
The test writes 10M to a zvol and then destroys the pool a few tens of
milliseconds later:
cannot destroy 'testpool': pool is busy
ERROR: zpool destroy -f testpool exited 1
Closing the zvol makes the kernel emit a change uevent, udev opens the
device to scan it, and a destroy issued while that scan is in flight
gets EBUSY. The test already waits for udev after creating the volume
but not after writing to it, and it destroys the pool with log_must
rather than log_must_busy.
Wait for udev again after the write, and destroy the pool the way
destroy_pool() does, retrying while it reports "busy".
Verified in a VM. The race does not reproduce on an idle machine, so
udev was imitated by holding the zvol open for three seconds before the
[7 lines not shown]
zcp: support bookmarks in zfs.sync.destroy
Teach the channel-program synctask destroy path to recognize bookmark
targets and route them through the existing bookmark-destroy
implementation. This enables zfs.sync.destroy() to remove bookmarks from
Lua channel programs, matching the behavior already available for
datasets and snapshots. A functional regression test was added to cover
bookmark destruction from a channel program.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Stephen Warren <swarren at wwwdotorg.org>
Closes #18992
Revert "Linux: avoid prefaulting under the ZFS range lock"
This reverts commit 43bb1614d50fee71b8ada14ca7b3a0d968a26263.
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Issue #18872
Issue #19046
Do not complete a device removal that hit IO errors
spa_vdev_remove_thread() checks vca_read_error_bytes and
vca_write_error_bytes at the end of each metaslab it copies and sets
svr_thread_exit, so that the removal is cancelled instead of completed.
Both counters are incremented from the copy zio callbacks in
spa_vdev_copy_segment_read_done() and
spa_vdev_copy_segment_write_done(), so the errors of the segments
copied last can arrive after that check has already run. The loop then
ends with svr_thread_exit still B_FALSE and the thread calls
vdev_remove_complete(), dropping the vdev even though part of its data
was never written to the new location.
Write errors are the way to hit this. A write error is only known once
the write completes, while a read error is recorded before the write it
feeds is even issued, so read errors are almost always seen in time.
With enough data to copy, the errors of one metaslab are noticed while
the next one is being copied, which is why this mostly goes unnoticed;
it is the errors of the metaslab copied last that are missed.
[15 lines not shown]
Increasing ZTS timeout as some PRs exceed the limit
Follow up work to better balance the work between the available
CI VMs is being evaluated to speed things up, but for the moment
increase the limit to prevent these timeout failures.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: tiehexue <tiehexue at hotmail.com>
Closes #18997
Wait for the initialize and trim threads to exit
An initializing thread marks the vdev VDEV_INITIALIZE_COMPLETE, drops
vdev_initialize_lock, syncs out the new state with txg_wait_synced()
and only then clears vdev_initialize_thread. spa_vdev_activity_in_
progress_impl() looks at the state alone:
boolean_t in_progress = (activity == ZPOOL_WAIT_INITIALIZE) ?
(vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) :
(vd->vdev_trim_state == VDEV_TRIM_ACTIVE);
so "zpool wait -t initialize" and "zpool initialize -w" return while
the thread is still in that txg wait. A command issued right after
them then hits the vdev_initialize_thread != NULL checks in
spa_vdev_initialize_impl() and fails with EBUSY, both for uninit and
for starting another initialization:
cannot initialize '/var/tmp/zpool_disk2.dat': currently initializing
[16 lines not shown]
ZTS: cover change-key on a dataset whose key does not match its root
An incremental raw receive onto a dataset that was rewrapped locally
with 'zfs change-key -i' leaves it pointing at the local encryption root
while carrying the sending root's key material. Until the previous
commit, running 'zfs change-key' on such a dataset, or on its encryption
root, panicked in syncing context.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Michael Heller <75820586+mkhllr at users.noreply.github.com>
Closes #17425
Closes #18969
