zfs-mount-generator: fix unit names of non-ASCII mountpoints
systemd_escape() writes each byte it escapes with "\\x%02x" from
(int)*cur. Where char is signed (such as how the x86 ABIs define it), a
byte from 0x80 on is negative, so sprintf() writes \xffffff and two
more digits, and the next byte's escape overwrites all but \xff. So
/mnt/josé gets the unit mnt-jos\xff\xff.mount instead of the correct
mnt-jos\xc3\xa9.mount, and systemd refuses it, since its name doesn't
match its Where=. Mountpoints that differ only in such bytes also get
the same name, so only one of them gets a unit. And the last byte's
sprintf() writes 7 bytes past the 4 it has, into the room for the
suffix.
Print the byte as a uint8_t with %hhx, as zfs diff does with %hho, so
that it always prints as two digits.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: R. Andrew Burza <andrew.ohana at gmail.com>
Closes #19540
CI: install userspace build dependencies for unit tests
The unit job uses VM preparation that upgrades installed packages,
installs kernel and ZTS dependencies, and removes packaged ZFS modules.
None of that machine preparation is needed for the userspace unit build.
Share the Debian userspace build packages between the unit job and the
full VM installer, leaving the latter's other dependencies and setup in
place. Deliver the shared installer alongside the VM script when it runs
before the repository is copied into the guest.
Use the native unit target in the documented minimum-build example.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #19456
Keep the local key where a raw receive brings a foreign one
A raw receive writes the DSL Crypto Key from the stream over the one on
disk but leaves the encryption root it is filed under alone. The key in
the stream is wrapped by the wrapping key of the sender's encryption
root, so this only holds together for as long as ours is the same one.
Where the receiving filesystem inherits from a root with a wrapping key
of its own, which is what a replica has once 'zfs change-key -i' hands
it to a locally keyed dataset, the wrapped master key on disk becomes
the sender's while the root it is filed under is still ours, and nothing
on the pool can unwrap it. The unwrapped key stays cached until it is
unloaded, so the receive and everything after it look fine; the next
'zfs load-key' loads the root happily and the filesystem then refuses to
mount with EACCES. Getting the data back takes the sending pool's
wrapping key and a hand built encryption hierarchy, which is why
reporters have ended up rebuilding their backups instead.
The keyformat, salt and iteration count travel with the key and describe
the wrapping key it was wrapped with, so where the receiving filesystem
[30 lines not shown]
CI: Let the labels jobs ride out GitHub server errors
One transient error from GitHub fails a whole job of the labels
workflow, and with it every PR the job had not labelled yet:
area gh: Server Error: Sorry, this diff is temporarily
unavailable due to heavy server load. (HTTP 500)
conflict HTTP 502: 502 Bad Gateway (https://api.github.com/graphql)
inactive invalid character '<' looking for beginning of value
The area job is the worst off: it runs once per push, for that PR
only, so a PR it fails on has no area labels until its next push.
The area job now lists the files of a PR with git rather than with
the pulls/N/files endpoint, which needs GitHub's diff service: one
partial fetch, without file contents, of the PR heads and their base
branches into a bare repository, then git diff --name-only from the
merge base, which lists what GitHub shows, renames included. Nothing
of a PR is checked out or run. The fetch takes about 2 s and 20 MB
[22 lines not shown]
ZTS: refreserv_004_pos fails: its space limits are off
The test sets a dataset's refreservation to the dataset's available
space and expects any larger value to fail. But a refreservation also
covers the space the dataset already uses, so it can grow to available +
used; setting available + 1 succeeds. The test is listed as a known
failure on every platform.
It also writes a file into the refreservation while the parent is at its
quota. That write can fail with EDQUOT however small it is: a
transaction's worst-case space estimate is far larger than the data, and
the part that exceeds the unused refreservation is charged to the
parent. The write only goes through while the parent is a few KiB short
of its quota, which for the pool's root dataset ends as soon as the MOS
grows.
Check that available + used + 1 MiB fails. The margin is needed because
the kernel checks against the on-disk available space, which differs
from the avail property by the writes in flight. Drop both writes; the
[9 lines not shown]
Linux: log size changes and punches before dropping the range lock
zfs_freesp() makes the change under a range lock in zfs_trunc(),
zfs_extend() or zfs_free_range(), drops the lock, and only then
assigns TX_TRUNCATE in a second transaction. zfs_setattr() calls it
with log=FALSE and logs the new size later still, in TX_SETATTR.
zpl_iter_write() takes no inode lock, so a write can run in between.
Its TX_WRITE is assigned first, and replay applies the write before
the size change or the punch: a write fsync'ed after ftruncate()
returned is cut off at the new size, and one written into a punched
hole is zeroed again.
Take the range lock in zfs_freesp() and hold it until TX_TRUNCATE is
assigned, with the helpers running under it. As before, the lock
covers the whole file when the size may change and only the range
otherwise; the choice between extending, truncating and freeing is
now made under it. The page cache is still truncated after the lock
is dropped.
[27 lines not shown]
spa: keep ZIL claims relative to the loaded txg
5f1957a15 numbers the new txgs after a load of an explicitly requested
txg above every uberblock still on disk, by raising spa_first_txg.
spa_min_claim_txg() returned spa_first_txg, so the threshold for "log
block not yet committed" moved with it. A writable 'zpool import -T'
doesn't clear the log (only the rewind retry path does), so zil_claim()
then treats log blocks and WR_INDIRECT data born after the loaded txg,
which are free in its space maps, as already committed and doesn't
claim them. Debug builds fail the assertion at the end of zil_claim();
release builds free those log blocks unclaimed when the log is
destroyed: a double free, or a free of space that has been reused since.
Record the claim threshold before raising spa_first_txg and return it
from spa_min_claim_txg(). It is fixed for the life of the import, as
dsl_scan and traverse also use it after the load. The threshold is
also stored as zh_claim_txg, which replay and the log's frees compare
births against, so it has to stay the txg after the loaded one.
[11 lines not shown]
ZTS: test a writable import -T of a pool with a live ZIL
Run a synchronous writer so the ZIL is never empty and freeze the pool
while it runs: the log keeps growing, but nothing after the freeze txg
reaches the disk. Stop the writer, export, and import the pool
read-write at the txg before the newest uberblock, so the log blocks
written while the newest txg was open are born after the imported txg.
The import must succeed, and after another export and import the pool
must scrub clean and pass zdb's leak check. (Freezing keeps the test
deterministic: a pool that syncs more txgs before the export can reuse
blocks freed after the imported txg, and the rewound pool then has
data errors.)
Since 5f1957a15 the new txgs after such an import are numbered above
the newest uberblock on disk, and spa_min_claim_txg() follows them, so
the log is claimed against the wrong txg.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
[2 lines not shown]
zio: don't write dedup copies from an override BP's data
An fsync'd write reaches syncing context with an override BP that
dmu_sync() already wrote, so zio_write_bp_init() skips the compress
and encrypt stages and the zio only holds the logical data. If the
block is deduplicated against an entry with fewer DVAs than copies=
needs, zio_ddt_write() wipes the override DVAs and issues a child write
for the missing copies, pushing that logical buffer under a BP whose
PSIZE and compression come from the override.
Debug builds then fail VERIFY3U(zio->io_size, ==, BP_GET_PSIZE(bp)) in
zio_dva_allocate() and hang txg_sync. Release builds write the extra
copy from the wrong buffer, so it fails its checksum from the start:
reads use the other copy, and a scrub reports and repairs it.
When an override write needs more DVAs, restart it as a regular dedup
write instead, as zio_write_bp_init() does when it can't use an
override. The data is compressed again and the missing copies are
written from it; dbuf_write_override_done() frees the override block.
[9 lines not shown]
ZTS: test dedup of an fsync'd block that needs more copies
An fsync'd write reaches syncing context as an override BP written by
dmu_sync(). When it is deduplicated against an entry that has fewer
DVAs than its copies= needs, zio_ddt_write() issues a write for the
extra copies. Add a test that sets this up with fast dedup, lz4 and
copies=1 then 2, and checks that both files read back intact, a scrub
finds no errors and zdb finds no leaks.
This fails on a debug build with
VERIFY3U(zio->io_size, ==, BP_GET_PSIZE(bp)) in zio_dva_allocate().
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Issue #19226
Closes #19303
zvol: leave alone a minor that is being removed
zvol_os_remove_minor() clears zv_zso and drops zv_state_lock while it
tears down the OS side of a ZVOL_REMOVING zvol. zvol_set_ro() and
zvol_set_volthreading() can find the zvol by name in that window and
zvol_set_ro() dereferences the NULL zv_zso. zvol_set_volsize() drops
all its locks before it updates the OS side, so the removal can start
under it.
Since d73686867 the thread that changes volmode waits for the minor to
be replaced, but another process setting readonly or volsize on the
same volume still hits the window: the kernel oopses in
zvol_os_set_disk_ro() or zvol_os_update_volsize(), the ioctl thread
dies with the zvol's lock held and every later minor operation hangs
in the zvol taskq.
Skip a ZVOL_REMOVING zvol when setting readonly, volthreading or
volsize, as zvol_suspend() does; the property is still stored, as when
there is no minor. Hold a suspend reference across the OS side of the
[12 lines not shown]
ZTS: race volume property changes with volmode changes
Setting readonly or volsize on a volume whose minor another process is
removing, for a volmode change, dereferences the minor's OS state after
it has been cleared. Run volmode, readonly, volthreading and volsize
changes on the same volume at the same time, and check that they all
finish and the volume still works afterwards, with the properties set.
A loop still running two minutes after it should have stopped is stuck
in the kernel: fail with the stuck zfs processes' stacks rather than
wait for the test timeout.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Will Rouesnel <wrouesnel at wrouesnel.com>
Closes #19319
libzfs_input_check: Fix integer to pointer casts
The zc_nvlist_foo fields are uint64_t and must be first cast to a
pointer-sized integer (uintptr_t) before being cast to a pointer to
avoid build-breakage for some targets (e.g. FreeBSD i386 w/GCC).
Fixes: 59dc88602
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
libspl/kmem: Make prototypes safe for C++
restrict is a C99 keyword and is not available in C++, so including
this header from C++ fails to compile. FreeBSD's zfsd is one such
user, through sys/vdev_impl.h.
Use __restrict, which gcc and clang accept in both languages.
Fixes: 8b5d919d4 ("libspl: move kmem definitions from zfs_context.h")
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
Fix uint64_t format strings on 32-bit targets
These print 64-bit values with %lu, %ld or %lx, which only matches
where long is 64 bits, and breaks builds with -Werror.
Use %llu and %lld with (u_longlong_t) and (longlong_t) casts, as is
done elsewhere.
Fixes: 13ec35ce3 ("Linux/vnops: implement STATX_DIOALIGN")
Fixes: d1b0a6982 ("draid: add failure domains support")
Fixes: 816804dfd ("Add support for anyraid vdevs")
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
zstream: Use libspl atomics for memory accounting
These were the only bare __atomic calls in zstream, and Clang rejects
the 64-bit ones on i386 with -Watomic-alignment. The rest of zstream
already uses libspl.
Fixes: c878ee452 ("zstream: track and limit memory use")
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
libspl/atomic: Add sequentially consistent ops
The libspl load and store are relaxed. Add the sequentially consistent
variants that zstream needs, so that it does not have to use bare
__atomic calls or add full barriers around relaxed operations.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
libspl/atomic: Silence Clang -Watomic-alignment
Since the implementation moved into the header as inlines, the 64-bit
operations are compiled in every file that includes it, not just in
libspl where NO_ATOMIC_ALIGNMENT is applied. On 32-bit targets Clang
warns because the operations become library calls, as uint64_t may be
under-aligned (i386) or too wide to be lock-free (powerpc). This is
fatal with -Werror.
Those library calls are expected, so ignore the warning for the
inlines only.
Fixes: 32ba39f47
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Reviewed-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Signed-off-by: Ryan Moeller <rmoeller at FreeBSD.org>
Closes #19793
freebsd/kstat: clean up properly when kstat_create fails
On error, kstat_create() would free the sysctl context and the kstat
itself, but missed the lock and most importantly, any memory allocated
for ks_data. Sending those error cases through to kstat_delete() handles
all the cleanup necessary.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19789
kstat: don't set ks_data for KSTAT_TYPE RAW
It's used for internal kstat stuff, nothing to do with us.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19789
kstat: fix use of ks_data and KSTAT_FLAG_VIRTUAL
We've been holding this wrong in a few places. KSTAT_FLAG_VIRTUAL is for
when ks_data will outlive the kstat (ie continue to exist after
kstat_delete() returns). In a few places, we allocated and set up
ks_data before the kstat is published (allowed!), but free it before it
is unpublished in kstat_delete(), so if the kstat is read then, it would
access already-freed memory.
The kstat API already allows for this case. If we don't set
KSTAT_FLAG_VIRTUAL, kstat_create() will allocate ks_data for us, and
then free it at a safe time in kstat_delete() after the kstat has been
unpublished.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19789
kstat: always NULL the kstat_t* after freeing it
Doing it in some places but not others is confusing.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19789
L2ARC: Do not stop writing when l2arc_headroom is 0
zfs.4 documents l2arc_headroom=0 as removing the per-cycle headroom
limit, and l2arc_write_sublist() still treats it so. Since 3523b5f3f,
however, l2arc_write_buffers() also stops a pass once its consumed
headroom reaches the headroom, which with l2arc_headroom=0 is 0 from
the start: no sublist is scanned, and nothing is written to any cache
device. Skip that check when l2arc_headroom is 0, as the check it
replaced did.
Add l2arc_headroom_zero_pos, which reads data with l2arc_headroom=0
and checks that the cache device received buffers.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19628
ZTS: fix device_access tests on whole disks
The device_access tests fail when $DISKS are disks that ZFS uses as
whole disks, such as NVMe or SCSI disks, rather than loop devices.
ZFS partitions such a disk and opens partition 1 (nvme1n1p1, sdb1),
not the disk node whose mode the tests change.
- Compare the disk that holds each vdev in check_vdevs().
- Set the mode of a disk's partitions along with the disk, so that
an import without CAP_DAC_OVERRIDE cannot open them.
- Wait for udev before setting the mode, and set it again after a
command that relabels the disk, since udev then resets the mode.
- Import and destroy the pool in device_access_import cleanup, since
labelclear on a whole disk misses the labels in its partition.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19791
ZTS: remove scsi_debug, not NVMe, disks in reopen tests
zpool_reopen_001_pos through _005_pos and auto_online_001_pos fail
when $DISKS are NVMe disks. on_off_disk() can only remove and
reinsert SCSI disks, but these tests remove $DISK1 whenever lsblk
reports it as a disk, which NVMe disks are too. Writing "offline"
to the NVMe controller's read-only state attribute fails.
- Remove $DISK1 only when it is a SCSI or multipath disk. Otherwise
remove the scsi_debug device, as the tests already do for loop
devices.
- Clear zpool_reopen labels by short name, so that zpool finds the
partition on a whole NVMe disk (nvme1n1p1, not nvme1n11).
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19790
ZTS: removal_with_add fails on disks with 4K sectors
During a device removal, a new top-level vdev must have the pool's
ashift. The test adds a file vdev, which gets ashift 9, so on 4K-sector
disks (ashift 12) the add fails with EINVAL. libzfs reports that EINVAL
as "a pool with removing/removed vdevs does not support adding raidz or
dRAID vdevs", which hides the cause.
Add the file vdev with the largest ashift of the pool's vdevs, from the
ashift vdev property.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed by: George Wilson <gwilson at zfsmail.com>
Signed-off-by: Matthew Ahrens <matt at mahrens.org>
Co-authored-by: Claude Opus 5.5 <noreply at anthropic.com>
Closes #19787
CI: Add AlmaLinux 10 x86-64-v2 package build
EL10 is built for x86-64-v3 CPUs, so the packages built on AlmaLinux 10
cannot be installed on its x86-64-v2 variant. Add an almalinux10-v2
job to the package workflow that builds on the x86_64_v2 cloud image,
and store its RPMs in a separate x86_64_v2 directory so they do not
collide with the regular EL10 packages.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: FingerlessGlov3s <me at fingerlessgloves.me>
Closes #19811
Let zfs rewrite -W skip only blocks that a rewrite would keep
zfs rewrite -W skips blocks whose allocated size would not change, so
that a rewrite after a RAID-Z expansion can be run again without
rewriting the same blocks twice. It also skips blocks that a rewrite
would change in other ways: after the compression, checksum, copies,
dedup or special_small_blocks property of a dataset changes, the
blocks written before keep their size and are skipped. After an
expansion, it leaves in place the blocks that take the same space at
the new width, and they split the space freed around them.
Skip a block only when a rewrite would leave it as it is: compare the
block pointer with the properties that dmu_write_policy() gives a write
of the file now, the class of each DVA with the one that
spa_preferred_class() would choose (moved into spa_class_select(), with
no functional change), and, on a RAID-Z vdev, the logical width at the
block's physical birth with the width in the txg of the rewrite. A
block on a vdev that takes no allocations, or is being removed, is
rewritten.
[15 lines not shown]
spa_stats: add "misc" pool stats with suspended & resumed counters
Adds a per-pool kstat called "misc", for ad-hoc counters that don't
obviously belong elsewhere. Then, add three counters,
"pool_suspended_ioerr", "pool_suspended_mmp" and "pool_resumed", that
are incremented each time the pool is transitioned to the suspended or
resumed states in zio_suspend() or zio_resume().
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <matt at mahrens.org>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19716
raidz: take the reflow range lock when spa_ubsync advances
vdev_raidz_io_start() maps an i/o to an expanding RAID-Z vdev with the
reflow offset in spa_ubsync: rows below it use the new location only,
rows above it the old location, and also the new one where the copy has
started. raidz_reflow_sync() takes the range lock as a writer over the
range it commits, but it only writes the new offset into spa_uberblock.
spa_sync() copies that into spa_ubsync at its very end, without the
lock, so an i/o that starts in between still writes the old location of
the committed range.
Once spa_ubsync has advanced, the reflow copies other data over those
old locations, and new writes there go to the new location only. An
i/o from the window that is still in flight then (queued, slow or
retried) lands on data that has already been copied, and that data is
lost once the reflow has passed it.
Take the range lock as a writer over the range spa_ubsync is about to
pass while spa_sync() updates it, so that the update waits for those
[14 lines not shown]
