test-runner: preserve command argument boundaries
Building sudo arguments by splitting a command string inserts an empty
argument when TMPDIR is unset and splits paths containing spaces.
Construct the argument list directly and add the optional environment
assignment only when TMPDIR is set.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Assisted-by: OpenAI Codex
Closes #19575
zdb: free unpacked pool history records
Free the history records and pointer array owned by dump_history after
printing them. Share cleanup with the read-error path while preserving
its return without printing partial history.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Assisted-by: OpenAI Codex
Closes #19582
ZTS: resolve namespace fixture sleep through PATH
Resolve sleep through the constrained test PATH instead of assuming
/usr/bin/sleep exists. Alpine provides it in /bin. Keep namespace probes
and test assertions unchanged.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Assisted-by: OpenAI Codex
Closes #19583
CI: install Alpine PAM development headers
Install linux-pam-dev so configure finds security/pam_modules.h and
builds pam_zfs_key in the Alpine VM. Keep the existing module source and
installation path.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <35407790+ooonea at users.noreply.github.com>
Co-authored-by: ooonea <ooonea at noreply.codeberg.org>
Assisted-by: OpenAI Codex
Closes #19584
freebsd/zfs_context_os: remove unnecessary includes
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: move fm_panic to SPL sys/cmn_err.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: move utsname compat into SPL sys/misc.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: remove or move thread things to SPL sys/proc.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: lift fstrans stuff to SPL sys/kmem.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: lift time things to SPL sys/time.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
freebsd/zfs_context_os: lift tsd compat to SPL sys/tsd.h
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19679
ZTS: zarcsummary_001_pos: write to a reset socket
abd1561b80 (#19547) made zarcsummary exit quietly when a write fails
with ConnectionResetError, as it already did with BrokenPipeError,
without a test for it. Add one.
zarcsummary_001_pos pipes zarcsummary into head(1). ksh's pipes are
socketpairs, and zarcsummary writes unbuffered (it reexecutes itself
with python3 -u), a line at a time. When head starts late, those
writes fill the socket and zarcsummary blocks; head then exits with
output unread, the socket is reset, and the blocked write fails with
ECONNRESET rather than EPIPE. Before abd1561b80 zarcsummary died with
an unhandled ConnectionResetError, and the test failed now and then in
CI:
ConnectionResetError: [Errno 104] Connection reset by peer
ERROR: eval zarcsummary | head > /dev/null exited 1
The existing pipelines only hit that by chance. Do it deterministically
[6 lines not shown]
ZTS: keep the simulated MMP host alive through import probes
The active-import test performs more than twenty import probes.
On a stalled runner they can outlast the helper's 120-second idle
period, so later probes find an inactive host instead of testing
the intended active-host rejection.
Allow a caller-specific idle duration and use 600 seconds for this
test. Cleanup still terminates the helper explicitly, and the
finite lifetime remains a failsafe. Other callers retain 120 seconds.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Dmitry R <rdmitry0911 at gmail.com>
Co-authored-by: Dmitry R <rdmitry0911 at gmail.com>
Closes #19686
ZIL: Don't acknowledge commits after a failed log block
After a log block (lwb) fails to write, a later fsync() or O_DSYNC
write can return success, and then be lost in a crash.
When an lwb's write fails, its waiters fall back to txg_wait_synced().
zil_lwb_set_zio_dependency() passes the error on to the next lwb only
while the failed one is still in flight. An lwb issued after the
failed one reached LWB_STATE_FLUSH_DONE has no dependency on it, so it
completes without error and its waiters return success. But on disk,
the log chain ends at the failed lwb, because zil_parse() can't read
it. Until the txg holding the failed lwb's records syncs and
zil_sync() moves zh_log past it, a crash loses every record after it,
including ones zil_commit() reported as stable.
The same happens when the lwb itself is written but a dmu_sync() write
for one of its WR_INDIRECT records fails. That error also reaches the
lwb's root zio, so the lwb's own waiters fall back, and again later
lwbs complete without error. After a crash, zil_claim_log_record()
[48 lines not shown]
Fix user accounting after incremental receives into encrypted datasets
Until now, user, group and project accounting was skipped while any
receive into an encrypted dataset was in progress. A raw receive can't
account for the objects it changes, because their owners and sizes are
encrypted, and non-raw receives skipped it too so that a raw
incremental could be received on top of them later.
Since 63a26454ba, a raw incremental stream can be received into a
dataset whose base snapshot already has accounting, which happens when
sending changes back to the dataset they came from (for example,
failing back after a failover). The receive leaves the accounting
objects and each object's DNODE_FLAG_USER*USED_ACCOUNTED flags as they
were, and the upgrade that runs at the next mount only dirties every
object again. For an object that is already marked as accounted, the
old and new charges both come from the received state, so they cancel.
Objects that the stream changed or freed keep their old charges: a
chown isn't reflected, freed space stays charged, and later frees can
wrap the counts below zero. A non-raw incremental onto a base snapshot
[38 lines not shown]
Add vendor support for Macaroni OS
Macaroni OS is a source-based OpenRC distribution (a Funtoo Linux
derivative). Its zfs packages currently have to force
--with-vendor=gentoo, because the auto-detection chain in
ZFS_AC_DEFAULT_PACKAGE probes no marker file that Macaroni ships.
Add a 'macaroni' vendor, detected via /etc/macaroni-release (being
added to the OS baselayout:
https://github.com/macaroni-os/core-kit/pull/710), with behavior
identical to the existing 'gentoo' entries:
- DEFAULT_PACKAGE: tgz
- DEFAULT_INIT_SHELL: /sbin/openrc-run (IS_SYSV_RC=false)
- initconfdir: /etc/conf.d
- bashcompletiondir: /usr/share/bash-completion/completions
No behavior changes for existing vendors: each touched case only adds
one alternative to an existing pattern, and the new detection entry
[12 lines not shown]
CI: quote QEMU OS configuration expansions
Preserve the OS selection JSON and output-file path instead of applying
shell word splitting and globbing. Read the CI type and source without
interpreting backslash escapes.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Giuseppe Piscitelli <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19619
Linux: free the range on a zvol write-zeroes request
Zvols advertised no write-zeroes limit, so blkdev_issue_zeroout() fell
back to writing zero-filled pages. BLKZEROOUT, fallocate --zero-range
and a guest's zero-fill became data writes, which with compression off
took as much space as the data they replaced. fallocate --punch-hole,
which has no such fallback, failed with EOPNOTSUPP.
Accept REQ_OP_WRITE_ZEROES and handle it as a discard that keeps the
unaligned ends. dnode_free_range() zeroes the partial blocks there, so
the whole range reads back as zeroes, and the ZIL logs the same
TX_TRUNCATE a discard does. The limit is the discard limit. On 6.17
and later the zvol also reports that its write-zeroes frees the range.
REQ_NOUNMAP, set by BLKZEROOUT and zero-range, asks that the blocks
stay allocated. Copy-on-write allocates a new block on every
overwrite, so writing zeroes would reserve nothing; refreservation is
what reserves space. The range is freed anyway, as
zpl_fallocate_common() does for FALLOC_FL_ZERO_RANGE on files.
[9 lines not shown]
zed: don't count solved cases as peers of a new diagnosis
Since ddf5f34f0 the diagnosis engine skips faulting a vdev when another
leaf under the same parent has an active SERD case, on the grounds that
errors on several drives at once point at something above the drives.
A solved case also counts: its engine has fired. But a solved case
stays open for as long as its vdev stays faulted, so once ZED has
faulted one leaf of a mirror or raidz and a hot spare has replaced it,
no other leaf under that parent can be faulted by ZED again until the
first one is repaired or replaced. The second failing disk keeps its
errors, gets no spare, and the case is retired over and over ("pool ...
is tracking 1 other io error cases -- skip faulting the vdev").
Skip solved cases when counting peers. A faulted vdev gets no more
I/O, so its case no longer says anything about errors happening now.
Cases that are still accumulating errors, or whose fault is pending,
still hold off a diagnosis as before.
This makes auto_spare_multiple pass.
[5 lines not shown]
ZTS: expect auto_spare_multiple to pass
auto_spare_multiple has failed on every Linux CI run for a long time,
hidden by its "maybe" entry: after ZED faults the first device and a
hot spare replaces it, the second device's I/O errors never fault it,
and the test times out waiting for FAULTED. Remove the entry so the
failure shows.
Past that point the test also races with the scans the pool starts by
itself. Before its second scrub it polled "scrub in progress" and then
"resilver in progress" with two zpool status calls and started a scrub
once both were false. A dRAID rebuild starts a scrub in the same sync
task that ends it, so the poll could see neither and the scrub failed
with "currently scrubbing". The resilver onto the first spare, deferred
behind the first scrub, is started from the async thread after that
scrub ends, and can replace a scrub started just before it.
Wait for those with zpool wait, retry a scrub that finds one running,
and scrub again if the scrub did not run to the end.
[4 lines not shown]
Release the new znode when its ZIL commit fails
Since 967b15b88 ("ZIL: allow zil_commit() to fail with error"),
zfs_create(), zfs_mkdir() and zfs_symlink() return the result of the
zil_commit() they issue on a sync=always dataset, and since 72602f6ad
("ZIL: "crash" the ZIL if the pool suspends during fallback"), both
from #17398, that result is EIO when the pool suspends during the
commit with failmode=continue. These functions set *zpp to the held
znode before the commit, and their callers only take the znode over on
success, so the error leaks the reference: the inode is never evicted,
a removed entry stays on the delete queue, and a later unmount spins
forever in zfsvfs_teardown() waiting for z_all_znodes to drain. On
FreeBSD the leaked vnode is also left locked.
Set *zpp only once the commit succeeded, and release the znode
otherwise, unlocking the vnode first on FreeBSD. The entry itself is
kept: the transaction that created it is already committed and cannot
be undone on a suspended pool, and zfs_write() and zfs_fsync() likewise
return EIO for a change that stays applied.
[8 lines not shown]
zed: fix zpool handle leaks in zfs_retire agent
is_draid_fdomain_failure() looks up the pool twice per try, up to four
tries, via find_by_guid_impl(), but never closes the returned
zpool_handle_t. Since it is called for every removed/faulted vdev,
regardless of the pool type (the "not dRAID" check happens after the
handles are opened), zed leaks two pool handles, including their config
nvlists, on every such event, and up to eight when a dRAID with failure
domains is not (yet) in a domain failure state. The nfaults_map buffer
was also leaked on the early returns.
Rework it to open the pool only once per try, look up the top-level
vdev in the same config snapshot, and close the handle at the end of
each try. Also guard against a zero top-level guid, which previously
left nvtop uninitialized.
zfs_retire_recv() had similar leaks:
- In the resource.fs.zfs.removed/statechange path, the early returns
[20 lines not shown]
CI: stop QEMU dependency preparation on failure
The dependency wrapper continues after failed boot checks, transfers and
remote preparation. It can then wait for a guest that never powers off,
or return success after losing the original error.
Select Bash explicitly and stop on failed commands so the calling
workflow receives the preparation failure immediately.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #19462
zloop: Add -S to seed the randomized ztest arguments
zloop picks each iteration's pool layout and ztest options with an
unseeded bash RANDOM, and ztest draws its own seed from /dev/urandom.
To rerun a crashed ztest, a developer must find the command line in
ztest.history and the seed in ztest.out and combine them by hand, and
a zloop run cannot be repeated.
Add a -S option that seeds RANDOM. Every randomized argument and the
-J seed handed to each ztest run come from that one value, so rerunning
zloop with the same -S repeats the whole loop. zloop now logs the -J
seed as part of each ztest command line in ztest.history, so a
developer can copy one line to rerun a crashed ztest. The default seed
is random and is logged at the start of the run.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Daniel Perry <dtperry at amazon.com>
Closes #19637
zfs: reject 'bookmark -r' to a dataset other than the source's
"zfs bookmark -r" only used the part of the new bookmark name after the
'#' and always started recursion from the source snapshot's dataset, so
the dataset part of the bookmark name was silently ignored:
zfs create tp/a; zfs create tp/b; zfs snapshot tp/a at s
zfs bookmark -r tp/a at s tp/b#bk
succeeded and created tp/a#bk (and the same on every descendant of
tp/a that has @s), while tp/b got nothing. Without -r the same command
is refused because the source is not an ancestor of tp/b.
With -r the bookmarks are always created on the source snapshot's
dataset and its descendants, so require the new bookmark to name that
dataset and fail with a clear message otherwise, before creating
anything. The short forms ("zfs bookmark -r fs at snap '#bk'" and
"zfs bookmark -r @snap fs#bk") are unaffected since their dataset is
taken from the other argument.
[8 lines not shown]
ZTS: test 'zfs bookmark -r' to another dataset
Extend zfs_bookmark_recursive to check that 'zfs bookmark -r' refuses
a new bookmark on a dataset other than the source snapshot's and
creates no bookmarks at all, and that the short forms ('#bookmark' and
'@snapshot') still work with -r.
This currently fails: "zfs bookmark -r tp/a at s tp/b#bk" succeeds and
creates tp/a#bk and the same bookmark on tp/a's descendants, ignoring
the dataset that was named. The fix follows in the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19298
zpool: fix 'zpool status' error count and list output
Commit e903655c5 ("zpool: Add zpool status -vv error ranges") reworked
how the data error log is printed, and introduced several regressions:
1. 'zpool status -j' (without -v) always read the error log and emitted
"errlist" (an empty array when there were no errors). Previously,
and as documented by the JSON example in zpool-status(8), the list
was only included with -v.
2. For unprivileged users, who may run 'zpool status' but not read the
error log (ZFS_IOC_ERROR_LOG), 'zpool status -j' reported
"error_count": "0" on a pool with data errors, while the plain text
output correctly said "errors: 1 data errors". It also printed
"errors: List of errors unavailable: permission denied" to stderr
even though no list was requested.
3. 'zpool status -v' printed every file name with a trailing space,
because the separator for the ranges printed by -vv was emitted
[26 lines not shown]
libzfs: reject malformed numeric ids in userquota props
userquota_propname_decode() parsed numeric user/group/project ids with
strtoul() into a uid_t and only checked that the whole string was
consumed. strtoul() accepts an empty string, leading whitespace and a
sign, and the result was silently truncated to 32 bits, so:
zfs set userquota@=10M pool # sets uid 0 (root)'s quota
zfs set userquota@-1=10M pool # sets uid 4294967295's quota
zfs set userquota at 4294968296=1G pool # sets uid 1000's quota
zfs set groupquota@' 5'=1G pool # sets gid 5's quota
all succeeded, and "zfs get userquota at root pool" then reported the
quota. The same applied to every {user,group,project}{quota,used,
objquota,objused} property. Since MAXUID is UINT32_MAX on Linux and
FreeBSD, the "id > MAXUID" check could never catch the overflow.
Require the id to be a non-empty string of decimal digits, parse it
with strtoull() checking errno, and reject values that do not fit in
[11 lines not shown]
ZTS: test malformed numeric ids in userquota properties
Add userquota_014_neg, which tries to set every quota property type
with an empty, signed, padded or out of range numeric id, and checks
that each attempt fails and that the quotas of the ids those values
used to be mistaken for (0, 5, 1000 and 4294967295) are unchanged.
This currently fails: the malformed ids are accepted, e.g.
"userquota@=10M" sets root's quota and "userquota at 4294968296=1G" sets
uid 1000's. The fix follows in the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19292
zoned_uid: print as integer, reject out-of-range uids
The zoned_uid property is registered as a plain number property, so
libzfs formatted it with zfs_nicenum() and parsed it with
zfs_nicestrtonum(). As a result:
# zfs set zoned_uid=1500 tank/fs
# zfs get -H -o value zoned_uid tank/fs
1.46K
and values that do not fit in a uid_t were accepted, then silently
truncated by the (uid_t) casts in zfs_prop_set_special(), delegating
the dataset to an unintended user namespace owner:
# zfs set zoned_uid=4294968296 tank/fs # delegated to uid 1000
# zfs set zoned_uid=4G tank/fs # delegated to uid 0
pbkdf2iters is a count, not a size, but likewise accepted size
suffixes (pbkdf2iters=150K became 153600 iterations).
[17 lines not shown]
