linux: support the FS_IOC_GETFSUUID ioctl
Populate the superblock UUID (sb->s_uuid) at mount time, through
super_set_uuid() on Linux 6.9 and later (the VFS then serves the
FS_IOC_GETFSUUID ioctl) and through a direct write plus a zpl_ioctl()
handler on older kernels. Add ZTS tests that check the UUID of
files, directories, snapshot automounts, received snapshots and
filesystems, and clones, the UUID change after "zpool reguid", and
the zfs_sb_uuid opt-out.
The UUID holds the 64-bit pool guid and the 64-bit dataset guid as
big-endian values, so its hex halves show the same numbers as "zpool
get guid" and "zfs get guid"; the pool guid keeps it unique across
pools, for example after "zpool split". It is not a valid RFC 4122
UUID, but the kernel treats sb->s_uuid as an opaque byte string.
Each mounted filesystem, snapshot and clone gets its own UUID, except
that a snapshot received into the same pool shares the guid, and thus
the UUID, of its source snapshot; a "zpool reguid" or a send/receive
re-creation changes the UUID at the next mount.
[16 lines not shown]
build: Handle case where no kernel compile tests are necessary
When caching the results of configure, a subsequent configure has the
results of all kernel compile tests cached and thus does not need to
run them. Properly handle this case.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Cache build results of unchecked kernel tests
If caching is enabled, kernel tests with a cached result will be
skipped, all others will be unconditionally run. However, then when
checking and caching the build results, not all run tests will be
checked and cached. This is because some test results don't need to
be checked depending on the result of other tests. Since these
unchecked test results were not cached, they will subsequently always
be run, even though they are never used.
Instead, after all the results checking has finished, go through all
tests whose results have not been checked and cache them.
Also, remove superfluous line in ZFS_LINUX_CONFTEST_MAKEFILE.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Ensure build dir is created for kernel option compatibility check
This bug was uncovered by the fix for CONFIG_DEBUG_LOCK_ALLOC in the
previous commit, which happened before this, whose check would never
be skipped and whose check ensured that the build directory was created.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
build: Properly cache test results
Align the cache variable names for these tests so that they can be
skipped after having been run when caching is enabled. These tests
were checking a cache variable name that was different from the one
set after the checking the results of the test.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19106
libzfs_core: block delivery of SIGUSR1 in send_worker thread
This fixes a Linux-specific bug.
3a909fe33 (libzfs, libzfs_core: send: always write to pipe, 2022-02-21)
introduced a subtle bug where zfs send -RPv would not print status
information periodically anymore when the stdout is redirected to
something that isn't a pipe (e.g. >/dev/null).
This is because the send_worker thread introduced by this commit
is created with an unmodified signal mask from libzfs. When
zfs_send_space is called it creates this thread. When we request
verbose status information another thread is created which, in
theory, should periodically receive a SIGUSR1 via a POSIX timer.
The main thread blocks USR1 delivery *after* the creation of the
progress thread, leaving the send_worker thread's signal mask
unmodified. The delivery of SIGUSR1 is now random and for at least
Debian Trixie with kernel 6.12.107+deb13-amd64 this causes the
[12 lines not shown]
build: Fix building kernel module when builddir is different from srcdir
It was noticed and fixed in f2f4ada24 (Linux 6.10 compat: fix rpm-kmod
and builtin) that the kernel was changed in 6.10 to make it difficult
have the build directory be different from the source directory. The
fix is to copy the module sources to the build directory. However, the
fix was only applied to building distro packages. This same problem
exists when building outside of the root of the source tree.
Instead do the copy when building the modules-Linux target, which
initiates the kernel module build. Commit f2f4ada24 hardlinks the
source files because it knows they are on the same filesystem. However,
this is not necessarily true generally when having a separate build
directory. So make a symlink tree instead of hardlinks. Also, only do
the copy if the build directory is different from the source tree.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19105
vdev_disk: simplify alignment checks for linear ABDs
A linear ABD is a single contiguous range, so only its first page
can carry an offset and only its last one can be short. That leaves
the alignment of the buffer as the only thing to check, and no
reason to look up the pages.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19102
Suggest libtirpc-dev for Ubuntu builds
A minimal Ubuntu install does not include libtirpc and ./configure
fails with "... try installig libtirpc-devel", but that package is
named libtirpc-dev on Ubuntu.
Fix the error message to mention both package names.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Andreas Dilger <adilger at thelustrecollective.com>
Closes #19103
build: Add gitrev as a dependency to specific targets that require it
The gitrev target was added as a dependency of the "all" target in
e6c093dd94 as previous to that it was a dependency of all built
sources, including the dist target for which it is unnecessary. This
the meant that the gitrev target would only be triggered when running
the "all" target. All subtargets that have an indirect dependency would
not trigger the gitrev target. Practically this means that building
directly certain binary targets, like zdb, would fail like so:
$ make zdb
CC module/zfs/libzpool_la-abd.lo
... snip ...
CC module/zfs/libzpool_la-uberblock.lo
/home/user/zfs.git/module/zfs/spa_history.c:32:10: fatal error: zfs_gitrev.h: No such file or directory
32 | #include "zfs_gitrev.h"
| ^~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:11422: module/zfs/libzpool_la-spa_history.lo] Error 1
[20 lines not shown]
zio_crypt_os: implement platform interface for FreeBSD
This uses our internal "crypto_os" shim as the backend to the FreeBSD
kernel crypto. There's no particular reason this separate shim needs to
exist anymore, but we keep it for now to keep this interface small.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
zio_crypt_os: implement platform interface for ICP
This is now a generic implementation for any platform that wants to use
the ICP for its crypto. It exists in the "common" module source, linked
and wired for Linux and libzpool.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
zio_crypt: establish platform interface; rework common code to use it
This lifts out and generalises the common parts of linux/zio_crypt and
freebsd/zio_crypt into a common version, with a
platform/backend-specific API for the not-common parts.
The common parts here involve key management, data assembly for
encrypt/decrypt, on-disk formats and so on - the "logic" side of the
equation, which are subtle and definitely shouldn't be duplicated. The
platform-specific parts meanwhile are mostly just glue to get in and out
of the platform-provided cryptographic suite.
To be clear - this is not a particular _good_ API, but further
improvement requires changes in the logic code. Establishing a clear
boundary will allow that change to be worked on more safely.
In the header, we add platform-specific types for the two places where
incompatible concepts leaked through from the implementations:
[35 lines not shown]
zio_crypt: add platform-specific headers
As we pull platform-specific things out of zio_crypt, we'll need
somewhere to move things to.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
zio_crypt: single field for mechname
On FreeBSD, it's only used in debug output anyway; there's definitely
no need for it to not just be a normal string in both.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
linux/uio: don't require wrapped data to be const
This makes it harder to work with zfs_uio_t internally ourselves, for no
apparently good reason. If a caller has a const iovec/bvec that they
want to wrap in a uio, its up to them to either cast away the const or
copy the data as appropriate. As it is, there does not appear to be any
places within OpenZFS that require such action.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
uio: add zfs_uio_iov() accessor for uio_iov
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18884
Fix metaslab count assertion in metaslab_group_alloc() for small vdevs
When creating a pool on a small vdev (< 1GB) with a low
zfs_vdev_min_ms_count value (e.g., 2), vdev_metaslab_set_size() could
compute a metaslab shift that results in fewer metaslabs than
zfs_vdev_min_ms_count, triggering an assertion failure in
metaslab_group_alloc():
ASSERT3U(mg->mg_vd->vdev_ms_count, >=, 2) failed (1 >= 2)
The root cause is that highbit64(asize / zfs_vdev_min_ms_count) returns
last set bit, which can yield a metaslab size larger than
asize / zfs_vdev_min_ms_count. For example, with a 500MB vdev and
zfs_vdev_min_ms_count=2:
- asize / 2 = 250MB
- highbit64(250MB) = 28, giving 256MB metaslabs
- 500MB / 256MB = 1 metaslab (violates minimum of 2)
Fix by subtracting 1 from highbit64() result to ensure the metaslab
size is at most asize / zfs_vdev_min_ms_count, guaranteeing at least
[22 lines not shown]
ZTS: Add import_relax_metadata_damaged exception
The newly added import_relax_metadata_damaged test case is flaky.
Add it to the exceptions script until the underlying issue is resolved.
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Issue #19077
Closes #19104
zdb: Allow filtering on zvol objects
Allow "v" flag for object ID range filters for filtering on zvol
objects. Also, fixup tests so that "v" is now considered a valid
flag.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19069
spl: Honour forced Linux slab cache selection
spl_kmem_cache_slab_limit controls automatic cache selection. An
explicit KMC_SLAB request should override that cutoff, but the Linux
slab branch applies it again. With a zero cutoff, this rejects the
forced znode cache. Pool creation later dereferences the NULL cache in
spl_kmem_cache_alloc().
Automatic selection has already applied the cutoff before this branch.
Remove the redundant check so KMC_SLAB retains its documented force
semantics, and clarify the automatic cutoff's scope in spl(4).
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthias Goergens <matthias.goergens at gmail.com>
Closes #19095
Stop the TRIM and initialize threads when a vdev goes offline
vdev_fault() and vdev_offline() leave the vdev not writeable, but the
TRIM or initialize thread running on it only notices that at its next
vdev_trim_should_stop() check, and it is that thread which records the
final state of the operation: since bedbc13da vdev_trim_thread() sets
VDEV_TRIM_CANCELED for a faulted vdev on its way out, and
vdev_initialize_thread() does the same. Neither ioctl waits for that,
even though spa_vdev_state_exit() already waits for the txg to sync
exactly so the command is synchronous -- "when the command completes,
you expect no further I/O from ZFS".
So the operation is still running when the command returns: a "zpool
status -t" issued right after "zpool offline -f" reports
loop0 FAULTED 0 0 0 external device fault (31% trimmed, ...)
on a vdev that is already FAULTED, and the TRIM thread keeps issuing IO
to the device the administrator has just faulted.
[37 lines not shown]
zfs: spa_sync_upgrades() should only take lock when needed
"zfs get -Hp ..." command got into waiting for memory allocation
while holding READER lock as dmu_objset_hold holds dp_config_rwlock READ
via dsl_pool_hold.
pageout was freeing memory by pushing some pages to swap zvol,
but got waiting for txg_wait_synced() and the memory was not freed.
txg_sync thread: spa_sync -> spa_sync_upgrades ->
rrw_enter(&dp->dp_config_rwlock, RW_WRITER) blocked.
This scenario did lead to investigation if spa_sync_upgrades() is
actually correct about requiring writer lock and it turns out that
we only do need write lock for three cases, where upgrade does change
DSL name space.
In other cases, and to check uberblock versions, we do not need
to set this lock.
[3 lines not shown]
ZTS: inject latency into initialize online/offline test
Fast vdevs can finish initialization before the online/offline test
checks progress and suspends it. Moving the suspend command earlier
only narrows this race window.
Inject a 20 ms write delay with one lane on the disk being initialized
so the test can observe progress and suspend initialization. Keep the
original online/offline assertions and clear the injection during
cleanup before destroying the pool.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matthias Goergens <matthias.goergens at gmail.com>
Closes #19024
Decline Direct I/O reads on a file handle after a benign verify failure
A Direct I/O read verifies the block checksum over the caller's buffer
after the read completes, to catch the buffer being modified while the
read is in flight. When an application recycles its O_DIRECT read
buffers across concurrent requests -- QEMU's block layer does this -- a
queued read can overwrite the buffer before the previous read's verify
runs. The verify then fails even though the data on disk is correct:
ZFS discards the direct read, re-reads the block through the ARC, and
emits an ereport.fs.zfs.dio_verify_rd. The returned data is correct and
the pool stays healthy, but under concurrent load the stream of failed
verifies and buffered re-reads is a real cost and can stall the
workload (#18610).
Once a file handle hits one of these benign failures -- a DIO read
verify that failed but whose buffered re-read then succeeded, proving
the on-disk data good and the buffer caller-modified -- decline Direct
I/O for reads on that handle for the rest of its life and route them
through the existing uncached buffered path. An application that uses a
[29 lines not shown]
Linux 7.3: make host filesystem available to kernel threads
Every kernel task has a `struct fs_struct *fs` that points to the root
of the filesystem tree for this task, and is used as the starting point
for path resolution. This is what is modified by chroot(2),
pivot_root(2), etc.
Kernel threads (as spawned by kthreadd) used to get a pointer to the
root filesystem by default. In 7.3 this changed, and kernel threads now
get a pointer to an internal nullfs; effectively a "no-op" filesystem
with no path into the true filesystem. An opt-in mechanism is provided
in the form of `scoped_with_init_fs()`, which swaps in a a pointer to
the root filesystem for the scoped code, and reverts it at end of scope.
Currently we don't have any good notion of whether or not a given thread
(including taskq tasks) will require filesystem access, and no good way
to signal it anyway. For now, to maintain the status quo, we add
`scoped_with_init_fs()` in the two places where arbitrary code may run
on a kernel thread: in `thread_generic_wrapper()`, the startup
[13 lines not shown]
Linux 7.3: inode_operations.create dropped flag arg
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Co-developed-by: TheAlmightyOgreLord <beardedtechwizard at protonmail.com>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19082
Linux: use cpu_feature_enabled() instead of static_cpu_has()
static_cpu_has() was renamed in 7.3 because since 3.18
cpu_feature_enabled() has always been the "right" way to do the same
thing. So we just switch everything over.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Co-developed-by: TheAlmightyOgreLord <beardedtechwizard at protonmail.com>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19082
Linux 7.3: bvec_iter.bi_bvec_done is now bi_offset
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Co-developed-by: TheAlmightyOgreLord <beardedtechwizard at protonmail.com>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19082
dmu: remove dmu_prealloc()
ztest is the only caller, and its ZTI_INIT entry has sat behind #if 0
since the 2010 import from OpenSolaris build 141, so nothing exercises
it. No kernel or libzpool code calls it either.
Remove the function and its export, plus the ztest code that existed
only for it: ztest_dmu_prealloc(), the ztest_prealloc() helper it
called, and the disabled ZTI_INIT entry.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Patrick Domack <patrickdk at patrickdk.com>
Closes #19090
