ZTS: don't remove a pool mountpoint that is the root directory
destroy_pool() removes the pool's mountpoint directory after destroying
the pool. If that mountpoint is the root directory, for example a pool
with mountpoint=/ or a path such as "/." or "//", it runs rm -rf on
the root directory.
Skip the removal when the mountpoint is the same file as /, with the
same -ef check that 0daa27fea added to the default setup functions.
This reworks #19056 as suggested in its review: only this check
changes, and create_pool(), destroy_pool()'s name check and the error
messages are left alone.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Related #19049
Closes #19342
ZTS: read the MOS of an exported pool in zdb_object_range_pos
zdb_object_range_pos lists every object in the MOS of the imported
test pool with zdb, then lists it again once per type filter and
compares the two lists. A txg that syncs between two of those zdb
runs changes the space map objects, and the test fails. In one lab
failure the second run no longer had space map 258 and had a new
space map 274.
Export the pool before the MOS checks, read it with zdb -e, and
import it again afterwards. cleanup imports the pool when a check
fails while it is exported. The test also fails when zdb -e lists no
objects, since two empty lists would pass every comparison.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Patrick Domack <patrickdk at patrickdk.com>
Closes #19321
ZTS: let zpool wait see a scan's last txg sync
Since 92a3904af "zpool wait -t scrub" and "-t resilver" keep waiting
until the txg that finished the scan has synced, while zpool status
reports the scan as done as soon as that txg starts syncing.
zpool_wait_scrub_basic and zpool_wait_resilver run with the 20 ms I/O
delay of add_io_delay, so that txg can take longer to sync than the
WAIT_EXIT_GRACE of two seconds that check_while_waiting gives the
waiter after zpool status says the scan is over, and the test fails
with "zpool process took too long to exit".
Let check_while_waiting take the pool of a scan and run zpool sync on
it before the grace period starts.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19340
ZTS: import again if l2arc_dwpd_reimport_pos's rebuild stops
l2arc_dwpd_reimport_pos exports and imports its pool and then expects
the L2ARC rebuild to have repopulated the cache device. The rebuild
gives up when l2arc_hdr_limit_reached() reports that the system is
low on memory. On the FreeBSD CI runners the test fails this way,
with "L2ARC not populated after import" and "System running low on
memory, aborting L2ARC rebuild" in dmesg.
If l2_rebuild_lowmem went up, wait a little and export and import the
pool again, up to five times.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19339
ZTS: create the cache pool after l2arc_l2miss_pos's first check
l2arc_l2miss_pos reads from a pool without a cache device and checks
that arcstats.l2_misses did not change. It creates the pool with the
cache device first, though, and that pool stays imported and idle
during the check. arc_read() counts an L2ARC miss for any read from
a pool with a cache device, so any read from it in that window fails
the test. It failed this way on FreeBSD CI runners, with l2_misses
2 above the starting count.
Create the pool with the cache device after the first check, where
the test first uses it.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19338
ZTS: don't compare extraction times in snapshot_002/006
snapshot_002_pos and snapshot_006_pos archive a directory and its
snapshot with "tar cf ... .", extract both into new directories and
compare them with directory_diff. bsdtar does not restore the times
of "." when it extracts, so on FreeBSD the two directories keep the
times of their own extraction. When the two extractions straddle a
second, rsync reports ".d..t....... ./" and the test fails.
Give the extracted directories the times of the directories they
were archived from, as GNU tar does when it restores ".".
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19331
pyzfs: retry destroying test pools in _TempPool.cleanUp()
The libzfs_core tests create their pools with _TempPool. reset()
retries "zpool destroy" when it fails with "pool is busy", as that
happens on the CI builders, but cleanUp(), which destroys the pools
when the test class is done, tries once and ignores any error. When
it fails, the read-only pool (pool.<uuid>) stays imported after
pyzfs_unittest passes, and a later test that syncs all pools fails
instead: refquota_007_neg on FreeBSD with "sync 'pool.<uuid>' failed:
pool is read-only".
Move the retry loop into _destroy() and use it in both reset() and
cleanUp(). cleanUp() still ignores a final failure, but _destroy()
now prints it.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19327
ZTS: wait for the condense in zfs_destroy_dev_removal_condense
The test resumes the paused livelist condense zthr and syncs the pool
once before checking that the condense synctask counted new ALLOC
blkptrs. The zthr only notices the resume on its next tick and then
queues the synctask with dsl_sync_task_nowait(), so the synctask can
land in a later txg than the one sync_pool waits for. The counter is
then still zero and the test fails, which happens on the FreeBSD CI
runners.
In a FreeBSD 14.4 VM the test failed 20 times in 100 runs. In every
failure the counter was zero after the first sync and non-zero after
one more. Sync until the counter is non-zero, up to 10 times; the
test now passed 200 times in 200 runs.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19291
mailmap/AUTHORS: use FreeBSD.org address for Christos Longros
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <longros at FreeBSD.org>
Closes #19350
ZTS: mark send-c_stream_size_estimate as a known failure
zfs send -nP prints two summary lines to stdout. When stdout is not a
pipe, a second thread copies the send stream to it with splice(). In a
dry run there is nothing to copy, but splice() still sets the file
offset back to where it was when the call started. This is a race in the
kernel: do_splice() updates the offset without taking f_pos_lock, which
read(), write() and lseek() take.
In this test the race only shows on musl. musl's stdout is line-buffered
until its first write, so the first line is written while splice() is
still waiting, and the second line overwrites it afterwards. glibc
buffers both lines and writes them once at exit, after splice() has
returned.
List the test as an expected failure until the kernel is fixed. The test
still runs, so the entry can be removed once it passes again.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19348
CI: Update Alpine Linux to 3.24.2 and its Limine boot
Alpine 3.24.2 no longer publishes the generic BIOS and UEFI cloud
images. The replacement is a single hybrid image that boots through
Limine, with the kernels on a ~113 MB FAT partition mounted on /boot.
Download the new image. Limine has no update-extlinux equivalent and no
hook that adds entries for newly installed kernels, so point its static
limine.conf entry at the -stable kernel, which the ZTS needs for
CONFIG_SCSI_DEBUG.
Drop the "kms" mkinitfs feature before installing linux-stable. It
pulls in GPU drivers and firmware, making the initramfs ~90 MB, which
fills the partition and fails mkinitfs. The VM is headless.
The image's limine updater configs use the old key names, which the
newer limine package pulled in by the upgrade does not understand. Its
BIOS trigger then fails after replacing limine-bios.sys, so the VM no
longer boots ("limine-bios.sys size incorrect"), and the failing EFI
[5 lines not shown]
zio: prevent DDT extension across RAIDZ expansion
A flat DDT entry can gain DVAs after its initial write, but stores one
physical birth. If a pre-expansion entry receives a post-expansion RAIDZ
DVA, the BP cannot represent both width epochs. RAIDZ then reads the new
copy at the old geometry; repair can map beyond its allocation.
Before extending a live phys, compare each newly allocated RAIDZ top's
logical width at the phys birth with its width at the allocation txg.
Refuse only a mismatched epoch and retry without dedup. This produces a
fresh BP whose DVAs share one birth. Same-epoch extensions, non-RAIDZ
tops, new entries, and traditional DDT tables keep their behavior.
Add raidz_expand_008_pos, which checks the block pointers written:
extension before expansion, refusal across a 3-to-4 disk expansion for
a 128K block and for a 16K block, whose allocation has the same size
at both widths, a clean scrub, and extension of a new entry after the
expansion. The last case distinguishes this exact check from a
permanent pool-wide gate.
[5 lines not shown]
zio: consume DDT fallback errors before reporting
DDT extension deliberately returns EAGAIN when adding a copy would need
a mixed gang and non-gang block pointer. The logical write classified
that error as a non-dedup retry only after normal error reporting. It
therefore recorded a persistent data error and posted an ereport for a
hole BP even though the plain-write retry succeeded.
Classify an allocating dedup write's EAGAIN after transforms. Disable
dedup, clear the deliberate error, and request reexecution before vdev
statistics and ereports are processed. The retry terminates because
dedup changes monotonically to false. A backend EAGAIN receives at most
one non-dedup retry before ordinary error handling.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #18826
zio: balance allocation depth on refused DDT extensions
A DDT child that would mix gang and non-gang DVAs is refused at READY.
This occurs before top-vdev children can release its metaslab-group
queue-depth holds. The refusal path decremented those holds even though
only a throttled asynchronous allocation created them. Synchronous and
unthrottled writes could therefore underflow the queue depth.
Gate the per-DVA release on ZIO_FLAG_ALLOC_THROTTLED. Keep it at this
refusal site: these holds use the child's size and tag, unlike gang
header allocations that can reach generic READY error handling with
different accounting keys.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #18826
zio: fix chained DDT extension rollback
A later write requesting more copies can adopt an in-flight FDT
extension as its child, forming a chain of leads. The chain shared one
rollback snapshot. A completed child copied the live phys after newer
children could already have extended it. Failure could retain an
uncommitted DVA or discard a committed one. The first completion could
also clear a snapshot still needed by a later lead.
Treat the saved phys as a rolling rollback point. Advance it only with
the completing DDT child's private BP. Clear it only after the newest
lead is gone. Failure then restores exactly the DVAs committed by older
children, independent of newer READY completions.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #18826
vdev_raidz: contain maps that exceed their DVA allocation
RAIDZ selects a block's stripe width from the block pointer's physical
birth. Fast-dedup extension in affected releases could append a DVA
after an expansion while retaining an older birth. This makes the
birth-selected map larger than the DVA's owned allocation. A repair of
that copy can then write into the next allocation.
For dedup BPs read at a historical width, find the dispatched DVA by
top vdev and offset. Compare its owned ASIZE with the map footprint.
Reject an oversized or unmatched direct map with EIO before building
columns, so neither reads nor repair writes can cross the allocation.
Debug builds assert the routing invariant; release builds still fail
closed if it is violated.
The check is limited to dedup BPs because FDT extension is the released
producer of this state. Equal-ASIZE mixed epochs may still fail checksum
verification, but they cannot overrun the allocation. An affected copy
is reported as a RAIDZ read error while other DVA copies remain usable.
[12 lines not shown]
Allow quota to be changed even when over quota
ZFS allows datasets to use slightly more space than their quota,
typically one write operation totaling less than 1MB. Once in this state
of using more space than the quota, subsequent writes will fail with
ENOSPC. The quota can also be changed to more than the space used (with
`zfs set quota=...`). The quota can not be tightened (decreased) such
that the dataset is in an over-quota state, which is a design decision
dating back to before version 1.
The problem is that when in the state of using more space than the
quota, the quota can not be set to its current value, or relaxed
(increased) to a value that is less than the space used. Since these
operations don't make anything worse in terms of being over-quota, they
should be allowed. This enables `zfs set quota=` to be idempotent.
Original-patch-by: Matthew Ahrens <matt at mahrens.org>
External-issue: https://www.illumos.org/issues/18300
External-issue: https://www.illumos.org/issues/18332
[6 lines not shown]
Linux: Sleep instead of spinning when zfs_zget() races eviction
When igrab() fails because the VFS is evicting the inode, zfs_zget()
drops its locks, calls cond_resched() and retries. cond_resched()
only yields when a reschedule is already pending, so every lookup of
an inode that is waiting in a long dispose_list() spins at full speed,
taking and dropping the znode hold locks and allocating a znode_hold_t
on each pass. With enough of them the thread doing the eviction is
starved of both CPU and those locks, and the eviction stalls. On a
24-thread NAS serving ~180 rsync clients this turned into a livelock
with zero pool I/O that needed a power cycle.
Sleep for one tick before retrying instead, as xfs_iget() does when it
races the same VFS teardown. The retry loop is otherwise unchanged and
no locks are held while sleeping.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Karl Schulze <karl at taniustech.com>
[2 lines not shown]
Linux: Feed the superblock shrinker in batches from zfs_prune()
zfs_prune() hands the whole ARC prune request to a single
super_cache_scan() call. When arc_evict() asks for hundreds of
thousands or millions of objects, prune_icache_sb() isolates up to
that many unused inodes, marks every one of them I_FREEING, and only
then evicts them one at a time from dispose_list(). For as long as it
takes to work through that list, a lookup of any of those inodes fails
igrab() in zfs_zget() and has to retry.
The kernel's own reclaim never builds such a list: do_shrink_slab()
calls scan_objects() in chunks of shrinker->batch objects (1024 for
superblocks). Do the same in zfs_prune(), so that at most one batch
of inodes is I_FREEING at a time. The total number of objects scanned
per prune request is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Karl Schulze <karl at taniustech.com>
[2 lines not shown]
CI: verify "failing test, then fix" PRs
A bug fix is easiest to review and keep fixed when the PR first adds a
test that shows the bug, then fixes it. Add a workflow that checks
this pattern: for each commit that only changes tests/ and is followed
by a commit changing code, build that commit in a QEMU VM and check its
new or changed tests fail (or crash or hang the kernel), then build the
PR head and check the same tests pass without kernel errors.
The workflow reuses the qemu-* scripts to set up, build and boot one
test VM per build, runs each test on its own with zfs-tests.sh -t, and
restarts the VM after a crash. PRs without a test commit finish after
the detect job.
The detector gives each job 15 minutes plus 30 per test (at most 240),
and the per-test watchdog in failfirst-tests.sh drops from 40 to 25
minutes: zfs-tests.sh -t stops a test after 600 seconds, and the rest
covers the group's setup and cleanup. A job with one test times out
after 45 minutes.
[3 lines not shown]
arc: fix race between arc_release() and arc_read_done()
Consider the following scenario:
1. arc_release() is called on hdr with one buf, but which has
IO_IN_PROGRESS (reading more raw data in encypted pool while
keeping decrypted data in buf).
2. arc_release() moves hdr to anon state and discards its identity.
3. arc_read_done() is called, adds the 2nd buf to hdr, increasing
b_refcnt to 2.
Now we have hdr in anon state with two bufs and without identity.
Or here's a racing scenario:
1. arc_release() checked that hdr is not in anon state, but before
taking hash_lock
2. arc_read_done() takes hash_lock, moves hdr to anon state, in
case of an error.
[28 lines not shown]
L2ARC: Reorder header destruction for in-flight L2 writes
With multiple L2ARC devices, headers can be destroyed asynchronously
(e.g., during zpool sync) while L2_WRITING is set. The original code
destroyed L2HDR before L1HDR, causing ABDs to lose their device
association (b_l2hdr.b_dev) when arc_hdr_free_abd() is called.
This caused ABDs to be added to the global free-on-write list without
device information. When any L2ARC device completed its write and
attempted to free these orphaned ABDs, it would panic on
ASSERT(!list_link_active(&abd->abd_gang_link)) because the ABD was
still part of another device's vdev_queue I/O aggregation gang.
Fix by extending l2ad_mtx lock scope to cover L1HDR destruction and
reordering to destroy L1HDR before L2HDR when L2_WRITING is set. This
ensures arc_hdr_free_abd() can access b_l2hdr.b_dev to properly tag
ABDs with their device for deferred cleanup.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
[5 lines not shown]
L2ARC: Preserve L2HDR in arc_release() for in-flight writes
When arc_release() is called on a header with a single buffer and
L2_WRITING set, the L2HDR must be preserved for ABD cleanup (similar
to the arc_hdr_destroy() case). If we destroy the L2HDR here, later
arc_write() will allocate a new ABD and call arc_hdr_free_abd(),
which needs b_l2hdr.b_dev to properly defer ABD cleanup, causing
VERIFY(HDR_HAS_L2HDR(hdr)) to fail.
Allocate a new header for the buffer in the single_buf_l2writing
case (single buffer + L2_WRITING), leaving the original header with
L2HDR intact. The original header becomes an "orphan" (no buffers, no
b_pabd) but retains device association for ABD cleanup when
l2arc_write_done() completes.
The shared buffer case (HDR_SHARED_DATA) is excluded because L2ARC
makes its own transformed copy via l2arc_apply_transforms(), so the
original ABD is not used by the L2 write. The header can be safely
reused without allocating a new one.
[14 lines not shown]
FreeBSD: do not clear dirty bits outside a sub-block write
page_busy() shrinks the written range to DEV_BSIZE boundaries. A write
that starts and ends inside one block leaves nbytes at -DEV_BSIZE, and
vm_page_clear_dirty() then clears that block and every one above it. A
page dirtied through mmap goes clean and the store is never written.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Nick Price <nprice at FreeBSD.org>
Closes #19268
ZTS: compare pre-condense log spacemaps
The flush can write new log spacemaps in a later TXG, so total
smp_length can grow even when older logs were flushed. Compare only
entries present before condense, then sync before reading their
lengths.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: George Melikov <mail at gmelikov.ru>
Closes #19273
ZTS: restore timeout for deadman negative check
The test lowers the deadman timeout to 5 seconds for its positive
check. Restore the default before the short-delay negative check so
a long QEMU scheduling pause does not look like another deadman
failure.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: George Melikov <mail at gmelikov.ru>
Closes #19273
ZTS: retry interrupted ctime test sleeps
The ctime helper assumes sleep(2) always completes. A signal can
end it early, leaving an empty timestamp window. Retry until two
wall-clock seconds have passed before running the operation.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: George Melikov <mail at gmelikov.ru>
Closes #19273
Open only the named pool when zpool checks a pool name
While tracing zpool get on one pool I noticed that it opens every
imported pool before the one named on the command line.
zpool get, zpool set and zpool iostat call is_pool() to tell a pool
name from a vdev name. It opened every imported pool to compare names,
so a command on a small pool also paid for every other pool on the
system. is_pool() now opens only the pool it is asked about.
On a system with three pools, one of them with 1200 disks, zpool get
all on the other two pools went from about 930 ms to 15 ms and 22 ms.
On the 1200 disk pool itself it stayed at about 1.8 seconds, because
the named pool is still opened twice, once for this check and once by
the command.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Ameer Hamza <ameer.hamza at truenas.com>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Signed-off-by: Caleb St. John <yocalebo at gmail.com>
Closes #19271
Bail out of zil_create() when the pool suspends
zil_create() waits with txg_wait_synced(), a void wrapper around a
plain wait that cannot report a suspend. When the pool suspends while
it waits, the fsync() that got there does not return until the pool
resumes.
The rest of the commit path already handles this. zil_commit_flags()
and zil_commit_writer_stall() wait with TXG_WAIT_SUSPEND, call
zil_crash() on ESHUTDOWN and hand EIO back to the waiters.
zil_create() and zil_commit_activate_saxattr_feature() were left on the
old wait, so a suspend caught in either one still hangs.
Use the suspend-aware wait in all three places and return the failure.
zil_process_commit_list() already has a NULL-lwb path that signals the
nolwb waiters with the error, and its comment already says an ESHUTDOWN
there means zil_crash() was called, so the failure lands somewhere that
expects it.
[11 lines not shown]
Bail out of the objset upgrade when the pool suspends
The two objset upgrade callbacks end with txg_wait_synced(), a void
wrapper around a plain wait that cannot report a suspend. When the
pool suspends during that wait, the upgrade taskq thread stays blocked
until the pool resumes. So does dmu_objset_disown(), which calls
dmu_objset_upgrade_stop(). That waits for a running upgrade task to
finish and then waits for a txg itself.
Give all three waits TXG_WAIT_SUSPEND. The callbacks return EAGAIN,
which dmu_objset_upgrade_task_cb() records in os_upgrade_status.
zfs_ioc_userspace_upgrade() and zfs_ioc_id_quota_upgrade() return that
status, and libzfs reports EAGAIN as a suspended pool.
dmu_objset_upgrade_stop() ignores the result, as it ignored the old
wait.
On master, fstests generic/753 in the eio group hangs with the pool
suspended and z_upgrade blocked here:
[10 lines not shown]
