ZTS: chattr_003_neg: accept uutils ln's wording for EPERM
The ln of Ubuntu 26.04 is uutils coreutils 0.10.0, which reports EPERM
from link(2) as "Permission denied" (uutils/coreutils#15020, fixed
after 0.12.0). The test's ln step expects "Operation not permitted",
so it fails on every ubuntu26 run. Accept either message there.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rick Galliher <kcid at kcid.io>
Co-authored-by: Claude Mythos 5.2 <noreply at anthropic.com>
Closes #19972
FreeBSD: Fix append-only semantics
On FreeBSD, ZFS_APPENDONLY was only enforced on the write path, so an
append-only file could still be truncated or extended with ftruncate(2)
through an O_APPEND descriptor, hole-punched with fspacectl(2), have its
mode changed, and be removed or renamed. Entries of an append-only
directory could also be removed or renamed. In summary:
- zfs_setattr(): apply the ZFS_IMMUTABLE attribute restrictions, which
include size changes, to ZFS_APPENDONLY files as well.
- zfs_deallocate(): refuse immutable and append-only files.
- zfs_zaccess_delete(): refuse append-only files and entries of
append-only directories, which also covers rename.
Add ZTS coverage. Linux already refuses these operations in its VFS,
so the tests are common.
zfs_clone_range() only refused ZFS_IMMUTABLE destinations. On FreeBSD,
O_APPEND can be cleared with F_SETFL after opening an append-only file,
[12 lines not shown]
ZTS: slog_replay_volume_split_*: write with stride_dd (#19917)
slog_replay_volume_split_boundary and slog_replay_volume_split_write
fail on every ubuntu26 CI run since its kernel moved to 7.0.0-43:
the replayed data matches, but no split write was logged or merged.
The tests make one 512K or 768K O_DIRECT write(2) to the zvol with
dd, so that it reaches the zvol as one request and is logged as one
TX_WRITE record split over several log blocks. Ubuntu 26.04's dd is
uutils dd, which reads into a buffer that is not 512-byte aligned.
Kernel 7.0.0-38 let that O_DIRECT write through to the zvol as one
bio; 7.0.0-43 rejects it with EINVAL, as it already did for other
block devices with the default 511 dma_alignment. uutils dd then
clears O_DIRECT and writes again through the page cache, so the zvol
gets 128 or 192 4K bios, each logged as its own small record.
Write with stride_dd -D instead: its buffer is page-aligned, and it
fails rather than retry without O_DIRECT.
[2 lines not shown]
CI: Temporarily ignore Lustre build result (#19946)
Lustre is currently having issues building against the Linux kernel.
Temporarily ignore the return code from the Lustre test build.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
CI: update the FreeBSD 15 STABLE CI runner to 15.2-STABLE
Follow the stable/15 snapshots, which are now published as
15.2-STABLE.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Christos Longros <longros at FreeBSD.org>
Closes #19899
CI: give the Linux test VMs 6 GiB, as FreeBSD's
Each zfs-qemu job runs two test VMs on a 16 GiB GitHub-hosted runner.
The Linux VMs got 8 GiB each, counting on KSM to share pages between
them. Since the end of October about one Linux job in four lands on
an AWS-hosted runner (EC2 m8id.xlarge, 15.7 GiB) whose root file
system, and so the swap file that qemu-1-setup.sh creates, sits on a
device-mapper snapshot. Every such job loses its runner: GitHub
reports "The hosted runner lost communication with the server" 48 to
52 minutes after the tests start.
A probe on the fork sampled one such runner per setting every minute.
With two 8 GiB guests, with KSM on or off, the runner stopped
reporting 3 to 4 minutes into the tests, once one guest's resident
memory had passed 6.8 GB and the host's available memory had fallen
below 4.3 GB; GitHub declared it lost about 45 minutes later. KSM had
merged no page by then. With two 6 GiB guests, the same kind of
runner ran the tests for the 76 minutes the probe allowed.
[8 lines not shown]
zpool-offline.8: fix missing punctuation
Sponsored-by: ConnectWise
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alan Somers <asomers at gmail.com>
Closes #19878
CI: fetch source matching the FreeBSD image kernel
FreeBSD snapshot images can outlive their src.txz archive. If it is
missing, fetch the running image's kernel revision from the official
FreeBSD source mirror. Require an empty destination before extracting
the fallback archive.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19876
Terminate the string fields of zfs_cmd_t and drr_toname
zfsdev_ioctl_common() terminates zc_name before the secpolicy and
handler functions see it, but zc_value and zc_string are left as
userland passed them. Several secpolicy and legacy handler functions
use them as C strings (property names, snapshot and bookmark names,
vdev paths and FRUs), so an unterminated value is read on into the
fields after it and can end up stored, e.g. as a vdev FRU.
Terminate zc_value and zc_string in the same place as zc_name.
The drr_toname of a receive's BEGIN record comes from userland too,
either in zc_begin_record or in the "begin_record" nvlist pair. An
unterminated drr_toname in a resumable receive's BEGIN record (a
malformed stream, received by a user with receive permission) is read
past the end of the field in dmu_recv_begin_sync(), and panics
txg_sync on kernels built with CONFIG_FORTIFY_SOURCE. Terminate it in
dmu_recv_begin(), after the BEGIN record has been checksummed, which
covers both ioctls.
[7 lines not shown]
ZTS: strings in fixed-size ioctl fields that fill them
zfs_cmd_t's zc_value and zc_string, and the drr_toname of a receive's
BEGIN record, are copied in from userland as they are, so a caller can
fill one to the end with no NUL. Only zc_name is terminated by the
kernel; the others are read as C strings that run on into whatever
follows them.
Add libzfs_unterminated_strings and its helper ioctl_unterminated:
- ZFS_IOC_VDEV_SETFRU with an unterminated zc_value and a marker in
zc_string: the FRU in the vdev label must be zc_value less its last
byte, before and after an export and import. Today it is zc_value
followed by the marker.
- A resumable receive of a truncated stream whose drr_toname is
unterminated: the resume token must name drr_toname less its last
byte. Today drr_toname is read past the end of the field, and on a
Linux kernel built with CONFIG_FORTIFY_SOURCE that panics txg_sync.
Then abort the receive and receive the whole stream.
[4 lines not shown]
mancheck: locate the script in the source tree
Use top_srcdir so manual checks also work in an out-of-tree build.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19784
Linux: initialize tq_ksp in taskq_create()
taskq_create() allocates the taskq with kmem_alloc(), which does not
zero it, and only sets tq_ksp in taskq_kstats_init(). If starting the
taskq's threads fails, taskq_create() calls taskq_destroy() before
that, and taskq_kstats_fini() then frees whatever tq_ksp contained.
Initialize tq_ksp to NULL along with the other fields.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Closes #19828
Linux: don't use per-taskq kstats in the taskq summary
spl_taskq_kstat_data() walks tq_list under tq_list_sem and reads each
taskq's counters through its kstat, tq_ksp. But taskq_create() adds the
taskq to tq_list before taskq_kstats_init() sets tq_ksp, and
taskq_destroy() deletes the kstat before removing the taskq from
tq_list. Reading /proc/spl/kstat/taskq/summary while a pool is created
or imported can therefore follow tq_ksp before it is set or after it
has been freed, and the oops leaves tq_list_sem held, hanging every
later taskq_create().
The summary only prints counters kept in the taskq itself, which are
valid for as long as the taskq is on tq_list. Read them directly. This
also stops the summary from taking each taskq kstat's lock while
holding its own.
Add a ZTS test that reads the summary during pool create, export,
import and destroy.
[4 lines not shown]
ZTS: make dio_read_verify's direct reads fail their verify
dio_read_verify expects every vdev type's pass to see at least one
Direct I/O read checksum verify failure. A read only fails its verify
if manipulate_user_buffer's thread happens to change the buffer between
the data landing in it and the verify, so on a busy runner a whole pass
of reads can go by without a failure, and the test fails with
"Checksum verifies in zpool status -d 0. ZED dio_verify events 0".
Delay every read of the pool's vdevs by 25 ms with zinject. The vdev
copies the data into the caller's buffer before the delay, so the
buffer is always modified before the verify and the first direct read
fails. Read 100 blocks instead of 300 to keep the run time down, and
log the direct and ARC read counts of each pass.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19733
Read the objset in the fadvise handlers after entering
zpl_fadvise() and zfs_freebsd_advise() read zfsvfs->z_os before
entering the file system. If a 'zfs recv' into the file system is
suspending it, the call waits in zfs_enter() until zfs_resume_fs() has
swapped in the new objset and freed the old one, then passes the stale
pointer to dmu_prefetch_user() or dmu_evict_range(). The kernel
oopses (Linux) or panics (FreeBSD) in dnode_hold_impl(), and a backup
tool that issues fadvise while a replication stream is received can
bring a host down.
Read z_os once the file system is entered, as the other entry points
do.
This makes fadvise/fadvise_recv pass on Linux and FreeBSD.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
[2 lines not shown]
dmu_recv: split WRITE_EMBEDDED records that don't match block size
When a -L incremental stream is received into a dataset that previously
did a non-large-block receive, receive_handle_existing_object() keeps
the object's existing (smaller) block size so that data not touched by
the incremental is preserved. DRR_WRITE handles the resulting size
mismatch by falling back to dmu_write(), which splits the record across
the object's blocks. DRR_WRITE_EMBEDDED had no such fallback:
receive_write_embedded() wrote the embedded BP, with its full logical
size, into the single dbuf covering the record's offset.
The result is silent data loss (the rest of the logical block keeps its
old contents) and an on-disk BP whose LSIZE exceeds the object's block
size, which later panics any send of the received snapshot:
VERIFY3U(srdp->datablksz, ==, BP_GET_LSIZE(bp)) failed
(131072 == 1048576) in do_dump()
Reproducer:
[49 lines not shown]
ZTS: test -L embedded block receive onto smaller blocks
Add send_embedded_large_block_incremental. It receives a full stream
without -L, so the received file keeps 128k blocks, then a -L -c -e
incremental carrying a 1M WRITE_EMBEDDED record for that file, and
checks that the received data matches, that no block pointer is larger
than the file's block size, and that the received snapshot can be sent
and received again.
This currently fails: the embedded record is written as a single 1M
block pointer into a 128k block, so the rest of the logical block keeps
its old contents, and sending the received snapshot panics in do_dump()
(VERIFY3U(srdp->datablksz, ==, BP_GET_LSIZE(bp))).
Also add recv_embedded_blksz_neg, which loads a crafted stream whose
first DRR_WRITE_EMBEDDED record has drr_lsize (4096) != drr_length
(8192), the length matching an 8192-byte DRR_OBJECT block size. A
genuine sender always sets drr_length to the embedded block pointer's
logical size, so the two are equal; a record where they differ can only
[41 lines not shown]
dsl_scan: fix scan-prefetch shutdown race leaking stale queue entries
dsl_scan_prefetch_cb() checks scn_prefetch_stop without holding
spa_scrub_lock. If the check observes B_FALSE just before shutdown
sets the flag and drains/exits the prefetch worker, the callback can
still acquire spa_scrub_lock afterward and insert a new entry into
scn_prefetch_queue -- without ever rechecking the stop flag under the
lock. That entry survives past zio_wait() in the shutdown path and is
left in the queue across the txg boundary.
If the block it references is later freed and its space reused, the
next txg's prefetch worker (if the scan is still running) dequeues and
reissues the stale entry via a genuine arc_read(), which can produce a
checksum error (or, if a redundant/ditto copy of the old block still
reads back successfully, trigger a self-healing repair write back onto
the now-reused offset).
Fix this with two changes in module/zfs/dsl_scan.c:
[22 lines not shown]
ztest: add ztest_scrub_prefetch
Register a new ztest_func_t that stresses the real scan-prefetch
shutdown path at txg boundaries while replacing objects: it kicks a
scrub, replaces a set of objects (freeing their old blocks), writes
sparse data that forces indirect-block growth for prefetch to
traverse, and forces a txg sync so dsl_scan_sync()'s prefetch
dispatch/shutdown runs while freed space may be getting reused.
This does not force the specific scn_prefetch_stop race window; it
is general stress/coverage for the scan-prefetch shutdown path
running concurrently with the rest of the randomized ztest workload,
not a deterministic regression test for that race.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Allan Jude <allan at klarasystems.com>
Signed-off-by: Zhu Xiao <zxiao at wasabi.com>
Closes #19484
zio: retain device injection provenance
Handler removal can clear the global injection count while an affected
I/O is still in flight. Keep a marker on that I/O for the allocation-
throttle completion assertion instead of consulting the current handler
count.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19677
zio: report ZILWRITE in formatted flags
ZILWRITE occupies bit 15, but the formatter has no entry for it.
Add the missing entry so subsequent flag names match their bit
positions.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Assisted-by: OpenAI Codex
Closes #19677
ZTS: create gang_blocks_special_fallback's files from one process
gang_blocks_special_fallback sometimes fails with "meta-dnode blocks
were not ganged" on machines with many CPUs (3 of 50 runs on 8 vCPUs).
New objects come from a per-CPU range of the meta-dnode (128 dnodes, 4
blocks), so 80 files created by dd processes on different CPUs can leave
every 16K dnode block with only 5-13 dnodes. At ashift=9 those blocks
are written as 1K, below the 1500 bytes the test forces to gang; a block
needs 14 or more dnodes to be written as 1536 bytes and gang.
- Create the files with mkfiles, from one process, before writing them.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19867
ZTS: wait for synchronous zedlet completion
Wait for the final zedlet before checking the output order, instead of
assuming that all four zedlets finish after seven seconds. Keep the wait
bounded and retain the exact order check.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19866
ZTS: create crtime_001_pos's file without touch(1)
crtime_001_pos creates its file with touch(1) and expects crtime to
equal ctime. zfs_mknode() sets both from the same timestamp, but GNU
touch creates a new file with open(O_CREAT) and then calls
utimensat(fd, NULL), which sets ctime again. When a second boundary
falls between the two calls, ctime is one second past crtime and the
test fails with "Incorrect crtime (N != N+1)".
Create the file with a shell redirection, a single open(O_CREAT), so
crtime and ctime always match. Sleep one second before the second
touch, so the check that touch leaves crtime alone would see a reset.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19869
libzutil: wait for device nodes when udev is absent
Use the existing node wait if libudev's queue API reports udev inactive.
This lets whole-disk pool creation work without udev even when libudev
was available at build time. Keep the existing initialized-device and
devlink checks when the control socket is present.
Check for all three queue functions before using them so providers
without that API retain their existing behavior.
Assisted-by: OpenAI Codex
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ooonea <ooonea at gmail.com>
Closes #19861
CI: Wait for fork QEMU jobs still running on a PR push
When a PR is pushed, the fork usually starts a push run of the same
commit as the PR run starts. The fork then has no result to reuse yet,
and both repositories run every QEMU job, although OpenZFS CI is
limited to 60 concurrent runners and each of the 16 OS jobs takes two
to four hours.
When a push run of the PR head tree has QEMU jobs running, let the
Setup job of the PR run wait for them, checking every ten minutes, for
at most REUSE_WAIT_MINUTES (a repository variable: default 240, capped
at 300 to stay clear of the six hour job limit, 0 disables it). It
then reuses the jobs that succeeded and runs the others as before.
The wait holds one runner instead of one per OS. Jobs still queued 30
minutes after the wait began are not waited for, since a fork with no
runner free may not start them for hours. Successful jobs of a run
still in progress are now reused: the wait can end before the run
does, and a job's result does not change once it has finished.
[10 lines not shown]
CI: Reuse fork results of commits with the same tree
reuse-ci.py only reuses fork push runs of the exact PR head SHA. A
commit amended only to add a sign-off or reword its message, or a
squash that leaves the code as it was, gets a new SHA, so its PR runs
every job again although the fork has already tested the same files.
A push run checks out and tests the tree of its commit, which the run
records as head_commit.tree_id. Look up the tree of the PR head and
accept push runs of any commit with that tree: the latest runs of the
exact SHA, as before, and those found among the latest hundred push
runs of the workflow. The newest finished result still decides each
job.
Earlier PR runs in the base repository are still not reused. They
check out the PR head too, but the QEMU test suite and OS list of a PR
run depend on ZFS-CI-Type lines in the commit messages and on the files
the PR changes, which a message-only change can alter and which the run
does not record.
[4 lines not shown]
commitcheck: exempt all email trailers from the line limit
Only Signed-off-by and Reviewed-by lines are exempt from the
72-character body limit, so a Co-authored-by line with a long name
or address fails commitcheck. So do Reported-by, Tested-by,
Suggested-by and the other "-by" trailers used in this repository.
Exempt any trailer line that ends in an email address instead of
listing tags. This covers the two existing exemptions too.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19860
zcp: check clone names with clone_namecheck()
zfs.sync.clone() passes the new dataset's name from Lua straight to
dsl_dataset_clone_check(), which only rejects '@' and names that are
too long. zfs clone also rejects names that fail dataset_namecheck()
(the ioctl layer checks them) and, in zfs_ioc_clone(), names with '%',
which marks internal datasets such as %recv. A channel program can
therefore create clones that zfs(8) can't create, including ones zfs
list never shows.
Add clone_namecheck(): dataset_namecheck() plus rejecting '@' and '%',
reported as NAME_ERR_INVALCHAR with the character in 'what'. Use it
in zfs_ioc_clone() in place of its own '@' and '%' checks, which
doesn't change what the ioctl accepts, and in a check function for
zfs.sync.clone() that runs it before dsl_dataset_clone_check(). Doing
the check there keeps zcp_sync_task()'s handling: zfs.check.clone()
returns the error, and zfs.sync.clone() still fails in open context
(zfs program -n). Names that are too long still reach
dsl_dataset_clone_check(), which returns ENAMETOOLONG. Add
[8 lines not shown]
ZTS: test invalid clone names in channel programs
zfs.sync.clone() passes the new dataset's name from Lua to the dsl
checks, which only reject '@' and over-long names, while zfs clone
also rejects '%' and characters dataset_namecheck() refuses. Add
tst.clone_neg, which expects EINVAL for such names.
This fails now: zfs.sync.clone() creates "fs/bad%name", which zfs list
never shows because '%' marks internal datasets, and "fs/bad*name".
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19311