CI: install diffutils on the Alpine runner
ZTS calls cmp with skip offsets, and zfs_send_sparse does it against a
file whose hole doubles every round. GNU cmp seeks to the offset;
busybox cmp reads its way there, 512 bytes at a time. Timed per command
on the runner, the send stays flat while the compare doubles:
hole zfs send | zfs receive cmp
256MiB 0.07s 25.2s
512MiB 0.08s 51.7s
1GiB 0.07s 102.1s
so the test walks into its ten minute cap, and the group's cleanup fails
behind it and takes the next groups' setup with it.
The GNU cmp comes from diffutils, which is not in this list -- without
it the busybox applet answers to cmp instead. diffutils arrives on 3.24
only because something else happens to pull it in, so the runner works
by accident of the dependency graph rather than by asking for what it
[4 lines not shown]
libzfs: explain why a large-block stream mismatch happens
A receive fails with ZFS_ERR_STREAM_LARGE_BLOCK_MISMATCH when the
destination has large blocks and the stream does not carry them. The
error told the user to add -L, but -L cannot help when the sending
dataset has no large blocks of its own: setup_featureflags() sets
DMU_BACKUP_FEATURE_LARGE_BLOCKS only when the feature is active on the
sending dataset, so -L is silently a no-op there. That is the state a
dataset is left in by an earlier replication in the opposite direction
that ran without -L, which splits the large blocks and leaves the copy
without the feature. The two copies then have different block layouts,
and advising -L sends the user in a circle.
Reword the error to name both causes and the remedy, and document the
trap in zfs-send.8 and zfs-receive.8.
Add a test for the reverse direction, which was not covered: replicate
a dataset that has large blocks without -L, then send an incremental
back with -L and confirm it is rejected, that omitting -L fails the
[11 lines not shown]
ZTS: take the loop device name from losetup itself
mount_loopback attaches an image with "losetup -f" and then looks the
device up again with "losetup --associated". Alpine's udev populates a
/dev/loop directory with the devices the kernel pre-creates at init, and
util-linux resolves every device name to /dev/loop/N whenever that
directory exists. A device allocated beyond those, through
/dev/loop-control once the first eight are busy, has no node there and
is never found. mkfs.xfs then runs with an empty argument and the test
aborts.
That is also why it is intermittent: it depends on how many loop devices
earlier tests left attached.
Aborting leaves the loop device holding the image open, so the pool
underneath cannot be unmounted and every later test that destroys a pool
fails too.
Ask losetup to print the device it attached instead, and fail loudly if
[4 lines not shown]
CI: stop cloud-init stalling every Alpine boot
The Alpine runner takes over five minutes to answer after the build step
reboots it, and all of it is one service:
* cloud-init local ...Cloud-init v. 26.1 running 'init-local'
DataSourceLXD.py[WARNING]: /dev/lxd/sock does not exist.
<301 seconds>
[ ok ]
Everything after it takes two seconds. It is datasource detection
timing out on an LXD socket a libvirt guest will never have.
Three cloud-init services are already removed here, but the one that
stalls is cloud-init-local, which lives in the boot runlevel rather than
default and so was never touched. Remove it too. Cloud-init is still
needed on the first boot, where the ssh key arrives through it; this
runs after that.
[2 lines not shown]
ZTS: Change zpool_initialize_multiple_pools devsize
Change zpool_initialize_multiple_pools test vdev sizes from
5GB -> 1GB. This size reduction is important since the test creates
four vdevs of this size (20GB) and initializes them. This
space reduction is need for running 3 CI VMs in parallel.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #19134
ZTS: Make zpool_prefetch_001_pos faster
zpool_prefetch_001 creates 16 thousand files, and makes copies of a
fraction of them. This takes a long time, especially when run on the
slow CI VMs.
Make it faster by only creating 8k files, and do some of the creations
in parallel. Both optimizations help to bring down the test time
from 40sec -> 14sec, as measured on a local, 10-core, VM.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #19133
rpm: Add support for building akmod packages
Add akmod (Automatic Kernel Module) support to ZFS, providing a third
packaging option alongside the existing DKMS and kmod packages.
Akmod packages automatically build binary kmod RPMs when new kernels
are installed, enabling:
- Automatic kernel module rebuilds on kernel updates
- Distribution of binary kmod RPMs without recompilation
- Seamless minor version upgrades on RHEL/Fedora and clones
Key changes:
- rpm/generic/zfs-kmod.spec.in: Add conditional akmod support via
rpmbuild --define '_enable_akmod 1'. When enabled, sets
buildforkernels=akmod and repo=rpmfusion so kmodtool generates
an akmod-zfs package. Add -common subpackage for shared
license/doc files. Guard chmod and %files sections for empty
kernel_versions. Make kmod-devel packages opt-in via
%bcond_with devel to prevent akmods from installing unnecessary
[35 lines not shown]
zdb: account pending DDT-log frees in leak detection
With fast dedup, the last decref of a block lands in the DDT log, and
the physical free happens only when the log entry is flushed back into
the DDT. ddt_phys_decref() only decrements the in-memory refcount, the
log append path issues no free, and ddt_phys_free() -- the only caller
of zio_free() on this path -- runs from ddt_sync_flush_entry(). A pool
examined inside that window contains blocks that no block pointer
references but that are still allocated. zdb's leak detection never
enumerates the log, so it misreports those blocks: as "leaked space"
and a block traversal size mismatch in the generic accounting
(print-only since d63f5d7e50), and as a fatal "obsolete indirect
mapping count mismatch" when such a block sits behind a removed vdev's
indirect mapping -- which is what intermittently fails the zloop CI
job.
Count the phys that the flush will free the same way as the
deferred-free bplist, under a new "DDT pending free" bucket:
reconstruct the BP, clear the dedup bit exactly as ddt_phys_free()
[40 lines not shown]
zdb: don't index an indirect vdev's metaslabs in the --class filter
zdb_count_block()'s --class filter resolves a block's allocation class
by indexing the top vdev's metaslab array. An indirect vdev, left
behind by a device removal, normally has no metaslabs; zdb synthesizes
them in zdb_leak_init_prepare_indirect_vdevs() so that zio_claim() has
somewhere to claim into. That only happens when leak tracking is
enabled, so under -L the array is NULL and any block whose first DVA
names a removed vdev dereferences it.
Classify an indirect vdev from its allocation bias instead. The bias
determines the vdev's primary metaslab group, and is available whether
or not the synthetic metaslabs exist.
For an indirect vdev with more than zfs_embedded_slog_min_ms metaslabs
this also changes what the filter reports. The synthetic metaslabs
carry no space maps, so they all look empty, and vdev_metaslab_init()
moves the emptiest into the embedded log group; blocks in that range
then classified as "other". Measured on a ztest pool with two removed
[16 lines not shown]
ZTS: don't mistake a compiled-in kmemleak for a running one
kmemleak_late_init() creates /sys/kernel/debug/kmemleak before it checks
whether the detector came up, so on a kernel built
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF and booted without kmemleak=on the
file is there and kmemleak is not. zfs_get_009_pos then skips, and the
two send_realloc tests quietly shrink their workload, on kernels
carrying none of the overhead those concessions are for.
Ask /proc/slabinfo instead: kmemleak_init() does not create the
kmemleak_object cache when it bails out, and SLAB_NOLEAKTRACE keeps that
cache from being merged away under another name.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19123
zio_crypt: free the key unwrap uios when decryption fails
zio_crypt_key_unwrap() allocates a pair of iovec buffers through
zio_crypt_uios_init_os() and releases them with zio_crypt_uios_fini_os()
once the keys have been decrypted. When zio_decrypt_os() fails it jumps
straight to the error label, which skips that release, so both buffers
are leaked.
The failure is reachable from userland: any key unwrap that does not
authenticate leaks 64 bytes twice, so a loop of "zfs load-key" calls
with the wrong passphrase grows kernel memory without bound. It is not
Linux specific -- the FreeBSD implementation allocates these buffers as
well.
Release the uios as soon as the crypto operation returns and check the
result afterwards, matching zio_crypt_key_wrap(), which already frees
them unconditionally on both paths. The buffers are then released
exactly once, and the error label continues to own nothing.
[7 lines not shown]
ZTS: fix test-runner crashing immediately under -m
sudo has no -c option, so it exits with a usage error and the run dies
before the first test case.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19117
zpool: Add zpool status -vv error ranges
Print the byte error ranges with 'zpool status -vv'. This works
with the normal zpool status formatting flags (-p, -j, --json-int).
In addition:
- Modify ZFS_IOC_OBJ_TO_STATS ioctl to optionally return "extended"
object stats.
- Let zinject corrupt zvol data.
- Add test case.
- Add DIOCGPHYSPATH geom attribute to FreeBSD zvols. This works
basically the same as Linux's BLKZNAME ioctl, and returns the
dataset name for a zvol (like "tank/vol"). This is used by zinject.
This commit takes code from these PRs: #17502 #9781 #8902
Reviewed-by: Alek Pinchuk <Alek.Pinchuk at connectwise.com>
Reviewed-by: Akash B <akash-b at hpe.com>
[5 lines not shown]
Linux: skip hardened usercopy checks in zfs_uiomove_iter()
The check adds nothing here. It only verifies the copy stays inside
the source object, and for a vmalloc address that object is the entire
SPL slab, not the buffer. The DMU already bounds every transfer to
the individual dbuf. The user-supplied destination is validated
inside _copy_to_iter() and is unchanged.
fs/proc/kcore.c and drivers/dax/super.c call the underscore variants
for the same reason. Both are EXPORT_SYMBOL and predate 4.18.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Ameer Hamza <ahamza at ixsystems.com>
Reviewed-by: Richard Yao <richard at ryao.dev>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19115
Linux: copy whole multi-page bvecs at once
bio_add_page() merges adjacent pages into one bvec only when they are
contiguous both physically and in the memory map, so on kernels without
HIGHMEM the linear map covers an entire bvec and it can be copied with a
single mapping and memcpy() instead of one per page. Drop the blanket
PAGE_SIZE clamp in zfs_uiomove_bvec_impl(), and iterate the blk-mq path
with rq_for_each_bvec() rather than rq_for_each_segment(), which splits
every bvec at a page boundary. For a 1MB zvol read the number of
memcpy() calls per block drops by an order of magnitude, depending on
how well the caller's pages merged.
CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP maps even lowmem pages one at a time,
so PageHighMem() on its own does not tell us whether a mapping spans
the whole bvec. Add zfs_kmap_partial() for that question and use it in
abd_iter_map() too, where the same assumption was already being made.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
[2 lines not shown]
zstream: report invalid record context without assertions
Stream framing and BEGIN payload size depend on the input. Report
invalid values as ordinary errors with the record offset instead of
terminating through VERIFY assertions.
Reject oversized BEGIN payloads before allocating or reading them.
Track compound-stream conclusions explicitly so valid compound sends
remain accepted while stray END records and records after a conclusion
are rejected.
Add ZTS coverage for malformed streams in both byte orders and for
valid compound streams.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Signed-off-by: Matthias Goergens <matthias.goergens at gmail.com>
Closes #19097
zfs_ctldir: make .zfs/snapshot/<name> btime the snapshot creation time
Before the automount, '.zfs/snapshot/<name>' is a placeholder inode
using simple_dir_inode_operations, which has no getattr, so statx(2)
reports no birth time. Once mounted, it is the snapshot's root
directory and reports that directory's crtime, i.e. the dataset
creation time. A consumer therefore gets either no birth time or the
dataset creation time, never the snapshot's, and the answer changes
when the snapshot is automounted.
Report the snapshot creation time as stx_btime consistently in both
states, so it can be read with a plain statx(2) on the path. Give
the placeholder its own inode_operations with a getattr that returns
the creation time zfsctl_inode_alloc() already stores, and override
the birth time of a mounted snapshot's root in zfs_getattr_fast()
next to the existing inode number override. The STATX_BTIME copy
moves from zpl_getattr() into zfs_getattr_fast() so the override is
not overwritten. Add snapdir_stat_btime to ZTS.
[3 lines not shown]
spa_errlog: don't let one unresolvable entry hide the whole error log
process_error_log() returns as soon as process_error_block() fails on
an entry, so one error log entry whose dataset cannot be opened fails
ZFS_IOC_ERROR_LOG as a whole. "zpool status" still says "N data
errors, use '-v' for a list", but "zpool status -v" lists nothing and
prints "errors: List of errors unavailable" on stderr. Two ways to
get there: an encrypted filesystem whose key is unloaded after its
error was recorded, where the DS_HOLD_FLAG_DECRYPT hold in
find_top_affected_fs() fails with EACCES; and an error in a snapshot
recorded while the key was unloaded, which is keyed by the snapshot
itself and not purged when that snapshot is destroyed, so the hold
fails with ENOENT.
0c4064d9a0 ("Fix zpool status in case of unloaded keys") already
reports a block whose birth txg was never resolved as a bare bookmark.
Do the same when the dataset walk fails: report the bookmark, which
zpool(8) prints as "fs:<0xobj>" or "<0xobjset>:<0xobj>", and go on
with the rest of the log. The entry stays in the log until it can be
[14 lines not shown]
linux: support the FS_IOC_GETFSUUID ioctl
Populate the superblock UUID (sb->s_uuid) at mount time, through
super_set_uuid() on Linux 6.9 and later (the VFS then serves the
FS_IOC_GETFSUUID ioctl) and through a direct write plus a zpl_ioctl()
handler on older kernels. Add ZTS tests that check the UUID of
files, directories, snapshot automounts, received snapshots and
filesystems, and clones, the UUID change after "zpool reguid", and
the zfs_sb_uuid opt-out.
The UUID holds the 64-bit pool guid and the 64-bit dataset guid as
big-endian values, so its hex halves show the same numbers as "zpool
get guid" and "zfs get guid"; the pool guid keeps it unique across
pools, for example after "zpool split". It is not a valid RFC 4122
UUID, but the kernel treats sb->s_uuid as an opaque byte string.
Each mounted filesystem, snapshot and clone gets its own UUID, except
that a snapshot received into the same pool shares the guid, and thus
the UUID, of its source snapshot; a "zpool reguid" or a send/receive
re-creation changes the UUID at the next mount.
[16 lines not shown]
build: Handle case where no kernel compile tests are necessary
When caching the results of configure, a subsequent configure has the
results of all kernel compile tests cached and thus does not need to
run them. Properly handle this case.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Cache build results of unchecked kernel tests
If caching is enabled, kernel tests with a cached result will be
skipped, all others will be unconditionally run. However, then when
checking and caching the build results, not all run tests will be
checked and cached. This is because some test results don't need to
be checked depending on the result of other tests. Since these
unchecked test results were not cached, they will subsequently always
be run, even though they are never used.
Instead, after all the results checking has finished, go through all
tests whose results have not been checked and cache them.
Also, remove superfluous line in ZFS_LINUX_CONFTEST_MAKEFILE.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Ensure build dir is created for kernel option compatibility check
This bug was uncovered by the fix for CONFIG_DEBUG_LOCK_ALLOC in the
previous commit, which happened before this, whose check would never
be skipped and whose check ensured that the build directory was created.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Properly cache test results
Align the cache variable names for these tests so that they can be
skipped after having been run when caching is enabled. These tests
were checking a cache variable name that was different from the one
set after the checking the results of the test.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
libzfs_core: block delivery of SIGUSR1 in send_worker thread
This fixes a Linux-specific bug.
3a909fe33 (libzfs, libzfs_core: send: always write to pipe, 2022-02-21)
introduced a subtle bug where zfs send -RPv would not print status
information periodically anymore when the stdout is redirected to
something that isn't a pipe (e.g. >/dev/null).
This is because the send_worker thread introduced by this commit
is created with an unmodified signal mask from libzfs. When
zfs_send_space is called it creates this thread. When we request
verbose status information another thread is created which, in
theory, should periodically receive a SIGUSR1 via a POSIX timer.
The main thread blocks USR1 delivery *after* the creation of the
progress thread, leaving the send_worker thread's signal mask
unmodified. The delivery of SIGUSR1 is now random and for at least
Debian Trixie with kernel 6.12.107+deb13-amd64 this causes the
[12 lines not shown]
build: Fix building kernel module when builddir is different from srcdir
It was noticed and fixed in f2f4ada24 (Linux 6.10 compat: fix rpm-kmod
and builtin) that the kernel was changed in 6.10 to make it difficult
have the build directory be different from the source directory. The
fix is to copy the module sources to the build directory. However, the
fix was only applied to building distro packages. This same problem
exists when building outside of the root of the source tree.
Instead do the copy when building the modules-Linux target, which
initiates the kernel module build. Commit f2f4ada24 hardlinks the
source files because it knows they are on the same filesystem. However,
this is not necessarily true generally when having a separate build
directory. So make a symlink tree instead of hardlinks. Also, only do
the copy if the build directory is different from the source tree.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19105
vdev_disk: simplify alignment checks for linear ABDs
A linear ABD is a single contiguous range, so only its first page
can carry an offset and only its last one can be short. That leaves
the alignment of the buffer as the only thing to check, and no
reason to look up the pages.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19102
Suggest libtirpc-dev for Ubuntu builds
A minimal Ubuntu install does not include libtirpc and ./configure
fails with "... try installig libtirpc-devel", but that package is
named libtirpc-dev on Ubuntu.
Fix the error message to mention both package names.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Andreas Dilger <adilger at thelustrecollective.com>
Closes #19103
build: Add gitrev as a dependency to specific targets that require it
The gitrev target was added as a dependency of the "all" target in
e6c093dd94 as previous to that it was a dependency of all built
sources, including the dist target for which it is unnecessary. This
the meant that the gitrev target would only be triggered when running
the "all" target. All subtargets that have an indirect dependency would
not trigger the gitrev target. Practically this means that building
directly certain binary targets, like zdb, would fail like so:
$ make zdb
CC module/zfs/libzpool_la-abd.lo
... snip ...
CC module/zfs/libzpool_la-uberblock.lo
/home/user/zfs.git/module/zfs/spa_history.c:32:10: fatal error: zfs_gitrev.h: No such file or directory
32 | #include "zfs_gitrev.h"
| ^~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:11422: module/zfs/libzpool_la-spa_history.lo] Error 1
[20 lines not shown]
zio_crypt_os: implement platform interface for FreeBSD
This uses our internal "crypto_os" shim as the backend to the FreeBSD
kernel crypto. There's no particular reason this separate shim needs to
exist anymore, but we keep it for now to keep this interface small.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
zio_crypt_os: implement platform interface for ICP
This is now a generic implementation for any platform that wants to use
the ICP for its crypto. It exists in the "common" module source, linked
and wired for Linux and libzpool.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
