zpool-offline.8: fix missing punctuation
Sponsored-by: ConnectWise
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alan Somers <asomers at gmail.com>
Closes #19878
CI: fetch source matching the FreeBSD image kernel
FreeBSD snapshot images can outlive their src.txz archive. If it is
missing, fetch the running image's kernel revision from the official
FreeBSD source mirror. Require an empty destination before extracting
the fallback archive.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19876
Terminate the string fields of zfs_cmd_t and drr_toname
zfsdev_ioctl_common() terminates zc_name before the secpolicy and
handler functions see it, but zc_value and zc_string are left as
userland passed them. Several secpolicy and legacy handler functions
use them as C strings (property names, snapshot and bookmark names,
vdev paths and FRUs), so an unterminated value is read on into the
fields after it and can end up stored, e.g. as a vdev FRU.
Terminate zc_value and zc_string in the same place as zc_name.
The drr_toname of a receive's BEGIN record comes from userland too,
either in zc_begin_record or in the "begin_record" nvlist pair. An
unterminated drr_toname in a resumable receive's BEGIN record (a
malformed stream, received by a user with receive permission) is read
past the end of the field in dmu_recv_begin_sync(), and panics
txg_sync on kernels built with CONFIG_FORTIFY_SOURCE. Terminate it in
dmu_recv_begin(), after the BEGIN record has been checksummed, which
covers both ioctls.
[7 lines not shown]
ZTS: strings in fixed-size ioctl fields that fill them
zfs_cmd_t's zc_value and zc_string, and the drr_toname of a receive's
BEGIN record, are copied in from userland as they are, so a caller can
fill one to the end with no NUL. Only zc_name is terminated by the
kernel; the others are read as C strings that run on into whatever
follows them.
Add libzfs_unterminated_strings and its helper ioctl_unterminated:
- ZFS_IOC_VDEV_SETFRU with an unterminated zc_value and a marker in
zc_string: the FRU in the vdev label must be zc_value less its last
byte, before and after an export and import. Today it is zc_value
followed by the marker.
- A resumable receive of a truncated stream whose drr_toname is
unterminated: the resume token must name drr_toname less its last
byte. Today drr_toname is read past the end of the field, and on a
Linux kernel built with CONFIG_FORTIFY_SOURCE that panics txg_sync.
Then abort the receive and receive the whole stream.
[4 lines not shown]
mancheck: locate the script in the source tree
Use top_srcdir so manual checks also work in an out-of-tree build.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19784
Linux: initialize tq_ksp in taskq_create()
taskq_create() allocates the taskq with kmem_alloc(), which does not
zero it, and only sets tq_ksp in taskq_kstats_init(). If starting the
taskq's threads fails, taskq_create() calls taskq_destroy() before
that, and taskq_kstats_fini() then frees whatever tq_ksp contained.
Initialize tq_ksp to NULL along with the other fields.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Closes #19828
Linux: don't use per-taskq kstats in the taskq summary
spl_taskq_kstat_data() walks tq_list under tq_list_sem and reads each
taskq's counters through its kstat, tq_ksp. But taskq_create() adds the
taskq to tq_list before taskq_kstats_init() sets tq_ksp, and
taskq_destroy() deletes the kstat before removing the taskq from
tq_list. Reading /proc/spl/kstat/taskq/summary while a pool is created
or imported can therefore follow tq_ksp before it is set or after it
has been freed, and the oops leaves tq_list_sem held, hanging every
later taskq_create().
The summary only prints counters kept in the taskq itself, which are
valid for as long as the taskq is on tq_list. Read them directly. This
also stops the summary from taking each taskq kstat's lock while
holding its own.
Add a ZTS test that reads the summary during pool create, export,
import and destroy.
[4 lines not shown]
ZTS: make dio_read_verify's direct reads fail their verify
dio_read_verify expects every vdev type's pass to see at least one
Direct I/O read checksum verify failure. A read only fails its verify
if manipulate_user_buffer's thread happens to change the buffer between
the data landing in it and the verify, so on a busy runner a whole pass
of reads can go by without a failure, and the test fails with
"Checksum verifies in zpool status -d 0. ZED dio_verify events 0".
Delay every read of the pool's vdevs by 25 ms with zinject. The vdev
copies the data into the caller's buffer before the delay, so the
buffer is always modified before the verify and the first direct read
fails. Read 100 blocks instead of 300 to keep the run time down, and
log the direct and ARC read counts of each pass.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19733
Read the objset in the fadvise handlers after entering
zpl_fadvise() and zfs_freebsd_advise() read zfsvfs->z_os before
entering the file system. If a 'zfs recv' into the file system is
suspending it, the call waits in zfs_enter() until zfs_resume_fs() has
swapped in the new objset and freed the old one, then passes the stale
pointer to dmu_prefetch_user() or dmu_evict_range(). The kernel
oopses (Linux) or panics (FreeBSD) in dnode_hold_impl(), and a backup
tool that issues fadvise while a replication stream is received can
bring a host down.
Read z_os once the file system is entered, as the other entry points
do.
This makes fadvise/fadvise_recv pass on Linux and FreeBSD.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
[2 lines not shown]
dmu_recv: split WRITE_EMBEDDED records that don't match block size
When a -L incremental stream is received into a dataset that previously
did a non-large-block receive, receive_handle_existing_object() keeps
the object's existing (smaller) block size so that data not touched by
the incremental is preserved. DRR_WRITE handles the resulting size
mismatch by falling back to dmu_write(), which splits the record across
the object's blocks. DRR_WRITE_EMBEDDED had no such fallback:
receive_write_embedded() wrote the embedded BP, with its full logical
size, into the single dbuf covering the record's offset.
The result is silent data loss (the rest of the logical block keeps its
old contents) and an on-disk BP whose LSIZE exceeds the object's block
size, which later panics any send of the received snapshot:
VERIFY3U(srdp->datablksz, ==, BP_GET_LSIZE(bp)) failed
(131072 == 1048576) in do_dump()
Reproducer:
[49 lines not shown]
ZTS: test -L embedded block receive onto smaller blocks
Add send_embedded_large_block_incremental. It receives a full stream
without -L, so the received file keeps 128k blocks, then a -L -c -e
incremental carrying a 1M WRITE_EMBEDDED record for that file, and
checks that the received data matches, that no block pointer is larger
than the file's block size, and that the received snapshot can be sent
and received again.
This currently fails: the embedded record is written as a single 1M
block pointer into a 128k block, so the rest of the logical block keeps
its old contents, and sending the received snapshot panics in do_dump()
(VERIFY3U(srdp->datablksz, ==, BP_GET_LSIZE(bp))).
Also add recv_embedded_blksz_neg, which loads a crafted stream whose
first DRR_WRITE_EMBEDDED record has drr_lsize (4096) != drr_length
(8192), the length matching an 8192-byte DRR_OBJECT block size. A
genuine sender always sets drr_length to the embedded block pointer's
logical size, so the two are equal; a record where they differ can only
[41 lines not shown]
dsl_scan: fix scan-prefetch shutdown race leaking stale queue entries
dsl_scan_prefetch_cb() checks scn_prefetch_stop without holding
spa_scrub_lock. If the check observes B_FALSE just before shutdown
sets the flag and drains/exits the prefetch worker, the callback can
still acquire spa_scrub_lock afterward and insert a new entry into
scn_prefetch_queue -- without ever rechecking the stop flag under the
lock. That entry survives past zio_wait() in the shutdown path and is
left in the queue across the txg boundary.
If the block it references is later freed and its space reused, the
next txg's prefetch worker (if the scan is still running) dequeues and
reissues the stale entry via a genuine arc_read(), which can produce a
checksum error (or, if a redundant/ditto copy of the old block still
reads back successfully, trigger a self-healing repair write back onto
the now-reused offset).
Fix this with two changes in module/zfs/dsl_scan.c:
[22 lines not shown]
ztest: add ztest_scrub_prefetch
Register a new ztest_func_t that stresses the real scan-prefetch
shutdown path at txg boundaries while replacing objects: it kicks a
scrub, replaces a set of objects (freeing their old blocks), writes
sparse data that forces indirect-block growth for prefetch to
traverse, and forces a txg sync so dsl_scan_sync()'s prefetch
dispatch/shutdown runs while freed space may be getting reused.
This does not force the specific scn_prefetch_stop race window; it
is general stress/coverage for the scan-prefetch shutdown path
running concurrently with the rest of the randomized ztest workload,
not a deterministic regression test for that race.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Allan Jude <allan at klarasystems.com>
Signed-off-by: Zhu Xiao <zxiao at wasabi.com>
Closes #19484
zio: retain device injection provenance
Handler removal can clear the global injection count while an affected
I/O is still in flight. Keep a marker on that I/O for the allocation-
throttle completion assertion instead of consulting the current handler
count.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19677
zio: report ZILWRITE in formatted flags
ZILWRITE occupies bit 15, but the formatter has no entry for it.
Add the missing entry so subsequent flag names match their bit
positions.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19677
ZTS: create gang_blocks_special_fallback's files from one process
gang_blocks_special_fallback sometimes fails with "meta-dnode blocks
were not ganged" on machines with many CPUs (3 of 50 runs on 8 vCPUs).
New objects come from a per-CPU range of the meta-dnode (128 dnodes, 4
blocks), so 80 files created by dd processes on different CPUs can leave
every 16K dnode block with only 5-13 dnodes. At ashift=9 those blocks
are written as 1K, below the 1500 bytes the test forces to gang; a block
needs 14 or more dnodes to be written as 1536 bytes and gang.
- Create the files with mkfiles, from one process, before writing them.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19867
ZTS: wait for synchronous zedlet completion
Wait for the final zedlet before checking the output order, instead of
assuming that all four zedlets finish after seven seconds. Keep the wait
bounded and retain the exact order check.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19866
ZTS: create crtime_001_pos's file without touch(1)
crtime_001_pos creates its file with touch(1) and expects crtime to
equal ctime. zfs_mknode() sets both from the same timestamp, but GNU
touch creates a new file with open(O_CREAT) and then calls
utimensat(fd, NULL), which sets ctime again. When a second boundary
falls between the two calls, ctime is one second past crtime and the
test fails with "Incorrect crtime (N != N+1)".
Create the file with a shell redirection, a single open(O_CREAT), so
crtime and ctime always match. Sleep one second before the second
touch, so the check that touch leaves crtime alone would see a reset.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19869
libzutil: wait for device nodes when udev is absent
Use the existing node wait if libudev's queue API reports udev inactive.
This lets whole-disk pool creation work without udev even when libudev
was available at build time. Keep the existing initialized-device and
devlink checks when the control socket is present.
Check for all three queue functions before using them so providers
without that API retain their existing behavior.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19861
CI: Wait for fork QEMU jobs still running on a PR push
When a PR is pushed, the fork usually starts a push run of the same
commit as the PR run starts. The fork then has no result to reuse yet,
and both repositories run every QEMU job, although OpenZFS CI is
limited to 60 concurrent runners and each of the 16 OS jobs takes two
to four hours.
When a push run of the PR head tree has QEMU jobs running, let the
Setup job of the PR run wait for them, checking every ten minutes, for
at most REUSE_WAIT_MINUTES (a repository variable: default 240, capped
at 300 to stay clear of the six hour job limit, 0 disables it). It
then reuses the jobs that succeeded and runs the others as before.
The wait holds one runner instead of one per OS. Jobs still queued 30
minutes after the wait began are not waited for, since a fork with no
runner free may not start them for hours. Successful jobs of a run
still in progress are now reused: the wait can end before the run
does, and a job's result does not change once it has finished.
[10 lines not shown]
CI: Reuse fork results of commits with the same tree
reuse-ci.py only reuses fork push runs of the exact PR head SHA. A
commit amended only to add a sign-off or reword its message, or a
squash that leaves the code as it was, gets a new SHA, so its PR runs
every job again although the fork has already tested the same files.
A push run checks out and tests the tree of its commit, which the run
records as head_commit.tree_id. Look up the tree of the PR head and
accept push runs of any commit with that tree: the latest runs of the
exact SHA, as before, and those found among the latest hundred push
runs of the workflow. The newest finished result still decides each
job.
Earlier PR runs in the base repository are still not reused. They
check out the PR head too, but the QEMU test suite and OS list of a PR
run depend on ZFS-CI-Type lines in the commit messages and on the files
the PR changes, which a message-only change can alter and which the run
does not record.
[4 lines not shown]
commitcheck: exempt all email trailers from the line limit
Only Signed-off-by and Reviewed-by lines are exempt from the
72-character body limit, so a Co-authored-by line with a long name
or address fails commitcheck. So do Reported-by, Tested-by,
Suggested-by and the other "-by" trailers used in this repository.
Exempt any trailer line that ends in an email address instead of
listing tags. This covers the two existing exemptions too.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19860
zcp: check clone names with clone_namecheck()
zfs.sync.clone() passes the new dataset's name from Lua straight to
dsl_dataset_clone_check(), which only rejects '@' and names that are
too long. zfs clone also rejects names that fail dataset_namecheck()
(the ioctl layer checks them) and, in zfs_ioc_clone(), names with '%',
which marks internal datasets such as %recv. A channel program can
therefore create clones that zfs(8) can't create, including ones zfs
list never shows.
Add clone_namecheck(): dataset_namecheck() plus rejecting '@' and '%',
reported as NAME_ERR_INVALCHAR with the character in 'what'. Use it
in zfs_ioc_clone() in place of its own '@' and '%' checks, which
doesn't change what the ioctl accepts, and in a check function for
zfs.sync.clone() that runs it before dsl_dataset_clone_check(). Doing
the check there keeps zcp_sync_task()'s handling: zfs.check.clone()
returns the error, and zfs.sync.clone() still fails in open context
(zfs program -n). Names that are too long still reach
dsl_dataset_clone_check(), which returns ENAMETOOLONG. Add
[8 lines not shown]
ZTS: test invalid clone names in channel programs
zfs.sync.clone() passes the new dataset's name from Lua to the dsl
checks, which only reject '@' and over-long names, while zfs clone
also rejects '%' and characters dataset_namecheck() refuses. Add
tst.clone_neg, which expects EINVAL for such names.
This fails now: zfs.sync.clone() creates "fs/bad%name", which zfs list
never shows because '%' marks internal datasets, and "fs/bad*name".
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19311
Keep received data when a send is resumed with -L
zfs send -t adds the -L or -w given on its command line to the
flags in the resume token, so a receive started without -L can be
resumed with -L, or with -w, which is -Lec for an unencrypted
dataset. The resumed stream then sends the OBJECT record of the
object the receive stopped in with a 1M block size, while the
receiver holds that object in 128K blocks.
receive_handle_existing_object() frees an object whose block size
changed if the stream is a full one, or if the object is not a
plain file, as for a volume with volblocksize=1M. That drops
everything received before the block that holds the resume offset.
The resumed stream does not send it again, and zfs recv exits 0.
A release-build run cut a full send of an 8M file with recordsize=1M
at a saved offset of 3538944. After a resume with -L, or with -w on
an unencrypted dataset, the received file differed from the source
from its first byte. An incremental resumed the same way matched.
[19 lines not shown]
ZTS: toggle -L when resuming a send
send-L_toggle_resume sends an 8M file from a recordsize=1m dataset
without -L and cuts the stream so the saved offset falls inside a
1M block. It resumes that receive with zfs send -L -t, and a second
cut receive with zfs send -w -t, which is -Lec for an unencrypted
dataset. It then does the same with -L for 8M written to a volume
with volblocksize=1m. The first WRITE for the resume object must be
1M, and the received file and volume must match the source.
The test needs the next commit. With only the sender fix from the
previous commit, it fails at the file comparison.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Patrick Domack <patrickdk at patrickdk.com>
Closes #19749
Resume a split large-block send at the saved offset
A send without -L splits each block larger than 128K into 128K
WRITE records. A receive that stops partway through such a block
saves the offset of the last 128K record it applied as the resume
offset, so the offset can fall inside the block. The resumed send
starts its traversal at the block that holds that offset, and
do_dump() then sends every 128K piece of the block from its start.
On debug builds the ASSERT in do_dump() requires the block to start
at or after the resume offset, so zfs send -t trips it. On release
builds the receiver gets the pieces before the resume offset a
second time and writes them again. In a release-build run of a full
send of an 8M file with recordsize=1M, the stream resumed from a
saved offset of 3538944 started with a WRITE at 3145728.
Skip the pieces that start before the resume offset, so the resumed
stream starts with the record the receive last applied, and let the
ASSERT accept a block that contains the resume offset. Resuming such
[6 lines not shown]
ZTS: resume a send of split large blocks
send_resume_split_large_block writes a 6M file and a file with one
293K block to a recordsize=1m dataset, sends them without -L, and
cuts the stream at five points, the last two near its end. It
resumes each cut receive with zfs send -t. The first WRITE for the
resume object must be at the saved resume offset, and the received
files must match. At least one saved offset must fall inside a
block.
The test fails without the next commit.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Patrick Domack <patrickdk at patrickdk.com>
Closes #19749
zdb: do not close the spa twice when an objset ID lookup fails
When the target is <pool>/<objset ID> and -N is given, or the name
does not resolve as a dataset, zdb opens the pool into the global spa
to turn the ID into a dataset name, and closes it again. The global
keeps pointing at the closed pool, so when no objset has that ID, zdb
goes on to fatal(), and zdb_exit() closes the pool a second time:
failed to hold objset 4194304: No such file or directory
zdb: can't open 'testpool/0x400000': No such file or directory
ASSERT at module/zfs/spa_misc.c:1078:spa_close()
zfs_refcount_count(&spa->spa_refcount) > spa->spa_minref || ...
zdb_objset_id runs "zdb -N" and "zdb -Nd" on such IDs and passes,
because log_mustnot takes a command killed by SIGABRT for one that
failed.
Do the lookup with a local spa, as the retry after a missing log
device already does. The global stays NULL until the dataset is
[5 lines not shown]
Don't busy-loop in txg_sync_thread() while the pool is suspended
Since c3f2f1aa2 ("vdev probe to slow disk can stall mmp write
checker"), txg_sync_thread() skips the sync of a suspended pool with a
bare "continue". Whatever ended its wait (an expired txg timer, a
txg_wait_synced() caller, an active scan or pending frees) stays true
while the pool is suspended, so the thread loops without sleeping,
holding tx_sync_lock, until the pool resumes. An exiting thread loops
forever.
A pool suspended while no txg is syncing, which is how an MMP
suspension usually happens, therefore costs a whole CPU. Kernels
without full preemption report soft lockups and RCU stalls, and every
caller of txg_wait_synced() or txg_wait_open() blocks on the mutex
instead of the condvar, so TXG_WAIT_SIGNAL and TXG_WAIT_SUSPEND waits
can no longer be interrupted or return early.
Wait on tx_sync_more_cv for up to a second instead, and let an exiting
thread go on to exit.
[6 lines not shown]
