zloop: Add -S to seed the randomized ztest arguments
zloop picks each iteration's pool layout and ztest options with an
unseeded bash RANDOM, and ztest draws its own seed from /dev/urandom.
To rerun a crashed ztest, a developer must find the command line in
ztest.history and the seed in ztest.out and combine them by hand, and
a zloop run cannot be repeated.
Add a -S option that seeds RANDOM. Every randomized argument and the
-J seed handed to each ztest run come from that one value, so rerunning
zloop with the same -S repeats the whole loop. zloop now logs the -J
seed as part of each ztest command line in ztest.history, so a
developer can copy one line to rerun a crashed ztest. The default seed
is random and is logged at the start of the run.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Daniel Perry <dtperry at amazon.com>
Closes #19637
zfs: reject 'bookmark -r' to a dataset other than the source's
"zfs bookmark -r" only used the part of the new bookmark name after the
'#' and always started recursion from the source snapshot's dataset, so
the dataset part of the bookmark name was silently ignored:
zfs create tp/a; zfs create tp/b; zfs snapshot tp/a at s
zfs bookmark -r tp/a at s tp/b#bk
succeeded and created tp/a#bk (and the same on every descendant of
tp/a that has @s), while tp/b got nothing. Without -r the same command
is refused because the source is not an ancestor of tp/b.
With -r the bookmarks are always created on the source snapshot's
dataset and its descendants, so require the new bookmark to name that
dataset and fail with a clear message otherwise, before creating
anything. The short forms ("zfs bookmark -r fs at snap '#bk'" and
"zfs bookmark -r @snap fs#bk") are unaffected since their dataset is
taken from the other argument.
[8 lines not shown]
ZTS: test 'zfs bookmark -r' to another dataset
Extend zfs_bookmark_recursive to check that 'zfs bookmark -r' refuses
a new bookmark on a dataset other than the source snapshot's and
creates no bookmarks at all, and that the short forms ('#bookmark' and
'@snapshot') still work with -r.
This currently fails: "zfs bookmark -r tp/a at s tp/b#bk" succeeds and
creates tp/a#bk and the same bookmark on tp/a's descendants, ignoring
the dataset that was named. The fix follows in the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19298
zpool: fix 'zpool status' error count and list output
Commit e903655c5 ("zpool: Add zpool status -vv error ranges") reworked
how the data error log is printed, and introduced several regressions:
1. 'zpool status -j' (without -v) always read the error log and emitted
"errlist" (an empty array when there were no errors). Previously,
and as documented by the JSON example in zpool-status(8), the list
was only included with -v.
2. For unprivileged users, who may run 'zpool status' but not read the
error log (ZFS_IOC_ERROR_LOG), 'zpool status -j' reported
"error_count": "0" on a pool with data errors, while the plain text
output correctly said "errors: 1 data errors". It also printed
"errors: List of errors unavailable: permission denied" to stderr
even though no list was requested.
3. 'zpool status -v' printed every file name with a trailing space,
because the separator for the ranges printed by -vv was emitted
[26 lines not shown]
libzfs: reject malformed numeric ids in userquota props
userquota_propname_decode() parsed numeric user/group/project ids with
strtoul() into a uid_t and only checked that the whole string was
consumed. strtoul() accepts an empty string, leading whitespace and a
sign, and the result was silently truncated to 32 bits, so:
zfs set userquota@=10M pool # sets uid 0 (root)'s quota
zfs set userquota@-1=10M pool # sets uid 4294967295's quota
zfs set userquota at 4294968296=1G pool # sets uid 1000's quota
zfs set groupquota@' 5'=1G pool # sets gid 5's quota
all succeeded, and "zfs get userquota at root pool" then reported the
quota. The same applied to every {user,group,project}{quota,used,
objquota,objused} property. Since MAXUID is UINT32_MAX on Linux and
FreeBSD, the "id > MAXUID" check could never catch the overflow.
Require the id to be a non-empty string of decimal digits, parse it
with strtoull() checking errno, and reject values that do not fit in
[11 lines not shown]
ZTS: test malformed numeric ids in userquota properties
Add userquota_014_neg, which tries to set every quota property type
with an empty, signed, padded or out of range numeric id, and checks
that each attempt fails and that the quotas of the ids those values
used to be mistaken for (0, 5, 1000 and 4294967295) are unchanged.
This currently fails: the malformed ids are accepted, e.g.
"userquota@=10M" sets root's quota and "userquota at 4294968296=1G" sets
uid 1000's. The fix follows in the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19292
zoned_uid: print as integer, reject out-of-range uids
The zoned_uid property is registered as a plain number property, so
libzfs formatted it with zfs_nicenum() and parsed it with
zfs_nicestrtonum(). As a result:
# zfs set zoned_uid=1500 tank/fs
# zfs get -H -o value zoned_uid tank/fs
1.46K
and values that do not fit in a uid_t were accepted, then silently
truncated by the (uid_t) casts in zfs_prop_set_special(), delegating
the dataset to an unintended user namespace owner:
# zfs set zoned_uid=4294968296 tank/fs # delegated to uid 1000
# zfs set zoned_uid=4G tank/fs # delegated to uid 0
pbkdf2iters is a count, not a size, but likewise accepted size
suffixes (pbkdf2iters=150K became 153600 iterations).
[17 lines not shown]
ZTS: test zoned_uid and pbkdf2iters numeric parsing
Extend zoned_uid_001_pos to check that zoned_uid is displayed as a
plain integer, and zoned_uid_005_neg to check that values which do
not fit in a uid_t, (uid_t)-1, and values with size suffixes are
rejected. Extend zfs_change-key_pbkdf2iters to check that an
iteration count with a size suffix (150K) is rejected.
These currently fail: a zoned_uid of 1500 is displayed as "1.46K",
values such as 4294968296 or 4G are accepted and silently truncated
(to uid 1000 and uid 0), and pbkdf2iters=150K is accepted as 153600.
The fix follows in the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Colin K. Williams <colin at li-nk.org>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19297
ZTS: scope uncached dbuf counts to the test dataset
Object numbers can recur in other datasets and pools. Filter by the
pool and objset as well, so unrelated cached dbufs cannot satisfy or
fail the uncached I/O assertions.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19615
ZTS: wait for the zvol probe after the errata write
Closing the writable zvol triggers a udev probe that can hold the pool
busy during export. Wait for it with the existing block_device_wait
helper after the write, keeping the export assertion unchanged.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Co-authored-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Assisted-by: OpenAI Codex
Closes #19616
ZTS: remove TESTDIR after the cache tests
cache.cfg puts the group's vdev files in $VDIR and $VDIR2, under
$TESTDIR. setup.ksh creates both with mkdir -p, which also creates
$TESTDIR, but cleanup.ksh removes only $VDIR and $VDIR2, and none of
the group's tests removes $TESTDIR or calls default_cleanup. An empty
testdir directory is therefore left in the test directory after
zfs-tests.sh -T cache, or -t with any of its tests, unless a later
group's cleanup removes it. Remove $TESTDIR, which holds both, as
default_cleanup does.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: remove TESTDIR after the l2arc tests
l2arc.cfg puts the group's vdev files in $VDIR, $TESTDIR/disk.l2arc.
setup.ksh creates it with mkdir -p, which also creates $TESTDIR, but
cleanup.ksh removes only $VDIR, and none of the group's tests removes
$TESTDIR or calls default_cleanup. An empty testdir directory is
therefore left in the test directory after zfs-tests.sh -T l2arc, or
-t with any of its tests, unless a later group's cleanup removes it.
Remove $TESTDIR, which holds $VDIR, as default_cleanup does.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: remove TESTDIR1 after removal_condense_export
removal_condense_export.ksh sets up its pool with a container
filesystem mounted at $TESTDIR1, but its cleanup only calls
default_cleanup_noexit, which removes $TESTDIR and not $TESTDIR1, so
an empty testdir1 directory is left in the test directory after every
run. Remove it as default_container_cleanup does.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: restore zfs_remove_max_segment in removal_multiple_indirection
removal_multiple_indirection.ksh lowers zfs_remove_max_segment to
32K and its cleanup sets it back to 1M, which is not the default: the
module starts with SPA_MAXBLOCKSIZE (16M). Every test that runs after
it in the same module load, the rest of the removal group included,
therefore removes devices in 1M segments. Save the value the test
found and restore that.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: remove vdevs_in_pool's temporary file when a vdev is missing
vdevs_in_pool() saves the pool's config to a file from mktemp and
returns 1 as soon as one of the vdevs it looks for is missing, before
it removes that file. Every log_mustnot vdevs_in_pool, which the
removal, pool_checkpoint and zpool_add tests use to check that a vdev
has left the pool, therefore leaves one file behind.
On a Debian 13 VM, zfs-tests.sh -T zpool_add left nine such tmp.*
files in the test directory, all written while zpool_add_001_pos to
zpool_add_009_neg ran. Remove the file before returning 1 too.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: end zpool_add_warn_removal with log_pass
zpool_add_warn_removal.ksh has no log_pass, so it falls off its end
and exits 0. The runner records a PASS, but the cleanup registered
with log_onexit only runs from log_pass or log_fail, so the loop
device that zpool_create_add_setup attaches to vdev4 and the files
vdev0 to vdev9 are left behind after every passing run.
On a Debian 13 VM, after zfs-tests.sh -T zpool_add passed all of its
tests, /dev/loop3 was still attached to vdev4 and vdev0 to vdev9
were still present; the test's log ends without the "Performing
local cleanup via log_onexit" line that its two siblings print.
zpool_add_warn_create and zpool_add_warn_degraded use the same setup
and cleanup and end with the same log_pass, which this adds.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19652
ZTS: import_rewind_metadata_damaged: allow the checkpointed state
The test noted the last synced txg after "zpool checkpoint" and required
the rewind past the damaged txg to report exactly that txg. That txg is
not protected by the checkpoint: the checkpoint is created by a sync
task that records the previous synced uberblock, so the txg the test
noted is the one that creates the checkpoint (or a later one), and the
blocks it allocates (its MOS root among them) are freed by the overwrite
and can be reused by the txgs the export syncs. When that happens the
txg fails to load ("unable to open rootbp in dsl_pool_init", or "unable
to retrieve MOS config") and the rewind correctly goes on to the
checkpointed state, an older txg, and the test fails with "The txg N was
not reported as the rewind target". This is the same misconception as
#19077 in import_relax_metadata_damaged.
Read the txg of the checkpointed state with zdb (a new
get_checkpointed_txg helper) and accept any rewind target from that
state up to the txg noted before the overwrite. A rewind to the
checkpointed state itself leaves no checkpoint behind, so check the
[4 lines not shown]
L2ARC: Do not write encrypted buffers whose MAC differs
When compressed ARC is disabled, l2arc_apply_transforms() compresses
an encrypted buffer again and encrypts the result with the block's
salt and IV. If the compressor's output differs from the bytes on
disk, for example for a block written before an update of the
compressor, the MAC differs from the one saved in the header too. A
debug build then panics on the ASSERT0() that compares them, and a
release build writes a copy that fails its checksum when read back.
Skip the write instead, as is already done when the recompressed
buffer does not fit in its original size: the caller clears
ARC_FLAG_L2CACHE, so the buffer is not tried again until it is
accessed again, and no L2ARC accounting has been done for it yet.
Keep asserting for a buffer that was not recompressed, whose MAC can
only differ through a bug elsewhere.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19631
tests: Wait for autotrim to reclaim file vdevs
TXG syncs do not wait for the autotrim thread. Continue syncing until
the vdevs reach the existing size limit, within the existing timeout.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19623
ZTS: use native mount records in rootpool tests
Discover Linux ZFS roots and mounted datasets through procfs. Use the
FreeBSD filesystem-type column without changing the SunOS parser.
Clear the per-process exclusion before querying rootpool compression.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19647
ZTS: find wipefs through the test command path
Use the wipefs command already exposed by the test environment instead
of a distribution-specific absolute path.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Assisted-by: OpenAI Codex
Closes #19648
ZTS: run concurrent removal and zdb on Linux
Run the removal test on Linux and drop its expected-SKIP result.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19646
Keep the indirect mapping lookup from wrapping below entry 0
vdev_indirect_mapping_entry_for_offset_impl() searches with an
inclusive unsigned upper bound, "last = mid - 1". When the offset
lies below the first entry, the search reaches mid 0 and last wraps
to UINT64_MAX, so the loop goes on with indexes far outside the table
and returns whatever it finds there, instead of NULL, or instead of
the first entry for vdev_indirect_mapping_entry_for_offset_or_next().
Search [base, limit) instead: the search ends with base at the first
entry above the offset, and the first entry needs no special case.
An empty table needs none either, since the loop does not run and
both lookups return NULL, so drop the assertion that the table is not
empty.
Every block pointer to a removed vdev should fall inside a mapped
range, so in the kernel only a damaged one gets there; a caller that
looks up an arbitrary offset, as zdb -b --vdev from #19421 does, gets
there with valid input, and need not check for an empty table first.
[10 lines not shown]
Include zfs rewrite -P blocks in raw encrypted sends
zfs rewrite -P keeps the logical birth of the blocks it rewrites, and
incremental sends prune by logical birth, so they leave those blocks
out. That is right for non-raw streams, but not for raw streams of
encrypted datasets. Rewriting an encrypted block gives it a new IV and
MAC, and the dnode blocks a raw stream carries keep the MACs of the
sender, which authenticate the new MACs through the indirect blocks.
The receiver still holds the old blocks, so once its key is loaded the
affected dnode blocks fail authentication and the replica cannot be
read.
Raw sends of encrypted datasets now prune by BP_GET_BIRTH(), which
differs from the logical birth only for blocks with the rewrite bit.
dbuf_write() sets it only at level 0, so indirect blocks keep pruning
as before. The condition is the one that sets DMU_BACKUP_FEATURE_RAW:
non-raw sends, zfs send -w of unencrypted datasets, which is not a raw
stream, and zfs diff still prune by logical birth and keep the benefit
of -P. A resumed send takes the same path, since its token records
[16 lines not shown]
zio: align diagnostic flag names
Add BYPASSED_QUEUE in its bit position and include POSTREAD and
LIGHTWEIGHT so later flag names describe the actual bits.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19617
ddt: do not prune an entry that a copies=2 write may have extended
A dedup write that asks for more copies than its entry holds writes
the missing ones and adds them to the entry, while the BPs written
before keep fewer DVAs. Once the refcount falls back to 1, the entry
is in the unique class, and ddtprune clears it without freeing
anything, since the BP left refers to its block. If that BP was
written before the extension, nothing refers to the added DVAs any
more and they leak; freeing that BP later frees only its own DVAs.
On a pool with one file vdev and dedup=on, a 128K block written with
copies=1, then again with copies=2, and the second file removed: zdb
reported no leak before "zpool ddtprune -p 100" and 128K leaked after
it. A leak ztest reports about once in 10 to 16 runs of 600 s has the
same cause in the run examined.
An extended entry cannot be told from one written with as many copies
from the start, so prune only entries holding a single DVA, and gang
entries, which zio_ddt_write() never extends. Entries of blocks written
[7 lines not shown]
zfs_valstr: terminate truncated bitfield strings
Reserve a terminating byte in all bitfield renderers and avoid unsigned
subtraction underflow for small token buffers. Leave zero-length output
unwritten and retain complete-output counts.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19618
CI: Allow more time for bootstrap
FreeBSD startup installs packages before the dependency phase. Increase
the startup timeout to 20 minutes for all platforms.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19624
recv: check the temporary clone name length
Reject an incremental receive whose temporary child name cannot fit
before syncing context formats its history entry. Corrective receives
create no child. Exercise both sides of the name limit and preserve the
base snapshot GUID after rejection.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Assisted-by: OpenAI Codex
Closes #19554
ZTS: add rebuild_online_leaf
A two-way mirror leaf is offlined, misses writes, and is onlined
while a sequential rebuild to a third leaf is paused. With the scrub
after the rebuild disabled, which would otherwise repair the new leaf,
the pool is then imported from the new leaf alone, and it must hold
all data.
Without the previous commit the new leaf receives stale data from the
onlined leaf.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Jared Maxwell <128261747+ThatRetiredDude at users.noreply.github.com>
Closes #19538
