build: Add -dirty release tag when building packages from a dirty git repo
By adding the "-dirty" tag to the release and by extension the full
version string, it becomes more clear that the build contains changes
not checked in to the repo. This could indicated changes that are
included by mistake.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19151
zpool: explain an error log that resolves to no files
An entry is only listed when the recorded block still carries the birth
transaction recorded against it, so a log whose blocks have since been
freed or rewritten prints a header promising files and then nothing at
all. The count shown without -v comes from spa_approx_errlog_size(),
which tallies table entries without resolving them, so the two forms
disagree and neither explains why.
Say that the entries resolved to no file. Reading the log is also what
removes such entries, so no advice to scrub is given: by the next read
they are gone.
Reviewed-by: Alek Pinchuk <Alek.Pinchuk at connectwise.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Nick Price <nprice at FreeBSD.org>
Closes #19108
Fix permanent errors misfiled into the scrub error log
dsl_scan_sync() sets spa_scrub_active before it checks whether a scan is
running, behind only a test that dsl_scan_active() is true. That test is
also satisfied by an async destroy, a non-empty free bpobj, or a pending
livelist delete, and nothing clears the flag except dsl_scan_done() or
dsl_errorscrub_done(), neither of which runs when no scan was started.
So after any dataset destroy the flag stays set for the life of the spa,
and spa_log_error() files every subsequent error in the scrub error log
rather than the last error log. An error scrub reads only the last error
log and so never sees them, and zpool clear does not touch the error log
at all. When a later scan ends it rotates the scrub error log into the
last error log and deletes only the log it replaces, laundering those
entries in as though that scan had found them, after which they survive
every subsequent scrub.
This is consistent with a long-standing class of reports, among them
survive repeated scrubs, and do not respond to zpool clear. The
[20 lines not shown]
CI: mask amx-tile in the QEMU build VMs
objtool sizes its alternate signal stack with SIGSTKSZ, which musl fixes
at 8192. A runner whose CPU exposes AMX needs 11952, so sigaltstack() is
refused, objtool exits before doing any work, make deletes the object,
and configure reports
checking whether CONFIG_MODULES is defined... no
on a kernel built with CONFIG_MODULES=y. Two separate samples of forty
Alpine jobs each landed four of them on such a host; all four failed,
every other job passed.
amx-tile is the only AMX feature carrying xsave state, so masking it
alone brings the frame back under SIGSTKSZ. host-model would also avoid
this but takes AVX-512 with it, which the suite exercises. Masking on
every runner keeps the diff to one line and is a no-op on the nine in
ten without AMX; ZFS has no AMX code. This can go once a fixed objtool
reaches the distribution images.
[3 lines not shown]
rpm: Pass --nodebuginfo to rpmbuild when debuginfo is disabled by configure
This fixes an issue where the system rpmbuild configuration is setup
to generate debuginfo rpms and will fail in make rpm-kmod if the needed
debuginfo source files are empty. Since by default debuginfo generation
is not enabled, this will fail reliably. Tell rpmbuild to not try to
build debuginfo rpms, when debuginfo generation is not enabled by
configure, by passing it the --nodebuginfo option.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19146
build: Fix AM_V_GEN usage in gitrev recipe
AM_V_GEN must be placed at the start of the recipe line because it gets
expanded to "@echo" which the shell doesn't know what to do with. The
@-prefix is interpreted by make, but only at the beginning of a recipe
line.
Fixes: 7adc908d3 (build: Add gitrev as a dependency to specific targets that require it)
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19145
rpm: Add support for building akmod packages
Add akmod (Automatic Kernel Module) support to ZFS, providing a third
packaging option alongside the existing DKMS and kmod packages.
Akmod packages automatically build binary kmod RPMs when new kernels
are installed, enabling:
- Automatic kernel module rebuilds on kernel updates
- Distribution of binary kmod RPMs without recompilation
- Seamless minor version upgrades on RHEL/Fedora and clones
Key changes:
- rpm/generic/zfs-kmod.spec.in: Add conditional akmod support via
rpmbuild --define '_enable_akmod 1'. When enabled, sets
buildforkernels=akmod and repo=rpmfusion so kmodtool generates
an akmod-zfs package. Add -common subpackage for shared
license/doc files. Guard chmod and %files sections for empty
kernel_versions. Make kmod-devel packages opt-in via
%bcond_with devel to prevent akmods from installing unnecessary
[35 lines not shown]
rpm: Add support for building akmod packages
Add akmod (Automatic Kernel Module) support to ZFS, providing a third
packaging option alongside the existing DKMS and kmod packages.
Akmod packages automatically build binary kmod RPMs when new kernels
are installed, enabling:
- Automatic kernel module rebuilds on kernel updates
- Distribution of binary kmod RPMs without recompilation
- Seamless minor version upgrades on RHEL/Fedora and clones
Key changes:
- rpm/generic/zfs-kmod.spec.in: Add conditional akmod support via
rpmbuild --define '_enable_akmod 1'. When enabled, sets
buildforkernels=akmod and repo=rpmfusion so kmodtool generates an
akmod-zfs package. Add -common subpackage for shared license/doc
files. Guard chmod and %files sections for empty kernel_versions.
Make kmod-devel packages opt-in via %bcond_with devel to prevent
akmods from installing unnecessary devel packages on every kernel
[34 lines not shown]
rpm: Add support for building akmod packages
Add akmod (Automatic Kernel Module) support to ZFS, providing a third
packaging option alongside the existing DKMS and kmod packages.
Akmod packages automatically build binary kmod RPMs when new kernels
are installed, enabling:
- Automatic kernel module rebuilds on kernel updates
- Distribution of binary kmod RPMs without recompilation
- Seamless minor version upgrades on RHEL/Fedora and clones
Key changes:
- rpm/generic/zfs-kmod.spec.in: Add conditional akmod support via
rpmbuild --define '_enable_akmod 1'. When enabled, sets
buildforkernels=akmod and repo=rpmfusion so kmodtool generates an
akmod-zfs package. Add -common subpackage for shared license/doc
files. Guard chmod and %files sections for empty kernel_versions.
Make kmod-devel packages opt-in via %bcond_with devel to prevent
akmods from installing unnecessary devel packages on every kernel
[34 lines not shown]
ZTS: zfs_get_006_neg: drop argument-ordering cases
Ten of this test's thirty bad_combine entries ("all -r", "all -p", "all
-H", "all -rp", "all -rH", "all -rpH", and four "$TESTPOOL" variants)
place a valid option after the property list. GNU getopt_long(3)
reorders those into a valid invocation, so zfs get accepts them and the
negative test does not fail.
They date from the Solaris original, where getopt(3) never permutes and
they failed naturally. Rather than reassess them when the test was
enabled on Linux, 95401cb6f exported POSIXLY_CORRECT=1 to push glibc
back into non-permuting mode. It borrowed that from
zfs_unallow_008_neg, where f74b821a6 had added it with a comment saying
it is set "so the original test cases will fail".
That only ever worked against glibc. musl's getopt_long(3) selects
permutation solely from optstring[0] and never consults POSIXLY_CORRECT,
so the ten cases fail on musl systems such as Alpine Linux.
[9 lines not shown]
ZTS: look for the kmemleak file the way it is later used
The -m option checks for /sys/kernel/debug/kmemleak with
os.path.exists() before accepting the flag. That check fails on a
kernel where the detector is compiled in, booted with kmemleak=on and
running: debugfs is mounted 0700, so an unprivileged caller cannot stat
anything below it, and the suite is normally started as an ordinary
user.
Every later access to that file goes through sudo, so check for it the
same way, with -n to match the passwordless sudo the runner already
requires of the calling user.
Report what sudo said when the check does fail. The old message blamed
the kernel configuration, which is the one thing that was not wrong, and
gave no way to tell the cases apart.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19136
Avoid dd_lock in the open context space checks
dsl_dir_tempreserve_impl() and dsl_dir_willuse_space() took dd_lock of
every dsl_dir up to the pool root for each write, costing ~10% of all
CPU on a 12-thread 32KB sequential overwrite benchmark.
dd_tempreserved[] and dd_space_towrite[] are per-txg estimates of what
the open context is about to write. They never reach the on-disk
accounting, which is still updated under dd_lock from the sync context,
and all their consumers only need an estimate, as dmu_tx_try_assign()
already notes. So update them with atomics and read them unlocked.
dsl_dir_sync() no longer needs dd_lock either: it only clears the slots
of a txg that is already done, which the open context can not use any
more.
Also add a few fast paths: parent_delta() and dsl_dir_willuse_space()
have nothing to compute when there is no reservation,
dsl_dir_tempreserve_impl() needs no estimates when there is no quota
[13 lines not shown]
Aggregate dataset space accounting deltas over a txg
dsl_dataset_block_born() and dsl_dataset_block_kill() applied the space
accounting of every single block under ds_lock, and then rolled it up
under dd_lock of every dsl_dir up to the pool root. On a 12-thread
32KB sequential overwrite benchmark that accounted for ~62% of all CPU
time.
Instead accumulate the deltas in per-CPU wmsums and apply them once per
txg from dsl_dataset_sync_done(). The result is identical rather than
approximate: parent_delta() is a difference of two MAX()es around the
same counter, so it telescopes over the txg, and the rest of the
dsl_dir accounting is linear in its arguments.
Blocks moved to the deadlist are counted in a separate group from the
blocks unique to the dataset, so the hot paths need only three
wmsum_add() calls each, and ds_referenced/compressed/uncompressed_bytes
derive as the difference of the two groups when the deltas are applied.
[12 lines not shown]
ZTS/rsend: receive into a '.' or '..' component must fail
Receive a snapshot into $POOL2/. and $POOL2/.. and check that both
are rejected and that a name starting with a dot is still accepted.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18975
Closes #19010
unit/namecheck: cover dot components before a delimiter
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18975
Closes #19010
zfs_namecheck: reject '.' and '..' before a snapshot or bookmark
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18975
Closes #19010
CI: install diffutils on the Alpine runner
ZTS calls cmp with skip offsets, and zfs_send_sparse does it against a
file whose hole doubles every round. GNU cmp seeks to the offset;
busybox cmp reads its way there, 512 bytes at a time. Timed per command
on the runner, the send stays flat while the compare doubles:
hole zfs send | zfs receive cmp
256MiB 0.07s 25.2s
512MiB 0.08s 51.7s
1GiB 0.07s 102.1s
so the test walks into its ten minute cap, and the group's cleanup fails
behind it and takes the next groups' setup with it.
The GNU cmp comes from diffutils, which is not in this list -- without
it the busybox applet answers to cmp instead. diffutils arrives on 3.24
only because something else happens to pull it in, so the runner works
by accident of the dependency graph rather than by asking for what it
[4 lines not shown]
libzfs: explain why a large-block stream mismatch happens
A receive fails with ZFS_ERR_STREAM_LARGE_BLOCK_MISMATCH when the
destination has large blocks and the stream does not carry them. The
error told the user to add -L, but -L cannot help when the sending
dataset has no large blocks of its own: setup_featureflags() sets
DMU_BACKUP_FEATURE_LARGE_BLOCKS only when the feature is active on the
sending dataset, so -L is silently a no-op there. That is the state a
dataset is left in by an earlier replication in the opposite direction
that ran without -L, which splits the large blocks and leaves the copy
without the feature. The two copies then have different block layouts,
and advising -L sends the user in a circle.
Reword the error to name both causes and the remedy, and document the
trap in zfs-send.8 and zfs-receive.8.
Add a test for the reverse direction, which was not covered: replicate
a dataset that has large blocks without -L, then send an incremental
back with -L and confirm it is rejected, that omitting -L fails the
[11 lines not shown]
ZTS: take the loop device name from losetup itself
mount_loopback attaches an image with "losetup -f" and then looks the
device up again with "losetup --associated". Alpine's udev populates a
/dev/loop directory with the devices the kernel pre-creates at init, and
util-linux resolves every device name to /dev/loop/N whenever that
directory exists. A device allocated beyond those, through
/dev/loop-control once the first eight are busy, has no node there and
is never found. mkfs.xfs then runs with an empty argument and the test
aborts.
That is also why it is intermittent: it depends on how many loop devices
earlier tests left attached.
Aborting leaves the loop device holding the image open, so the pool
underneath cannot be unmounted and every later test that destroys a pool
fails too.
Ask losetup to print the device it attached instead, and fail loudly if
[4 lines not shown]
CI: stop cloud-init stalling every Alpine boot
The Alpine runner takes over five minutes to answer after the build step
reboots it, and all of it is one service:
* cloud-init local ...Cloud-init v. 26.1 running 'init-local'
DataSourceLXD.py[WARNING]: /dev/lxd/sock does not exist.
<301 seconds>
[ ok ]
Everything after it takes two seconds. It is datasource detection
timing out on an LXD socket a libvirt guest will never have.
Three cloud-init services are already removed here, but the one that
stalls is cloud-init-local, which lives in the boot runlevel rather than
default and so was never touched. Remove it too. Cloud-init is still
needed on the first boot, where the ssh key arrives through it; this
runs after that.
[2 lines not shown]
ZTS: Change zpool_initialize_multiple_pools devsize
Change zpool_initialize_multiple_pools test vdev sizes from
5GB -> 1GB. This size reduction is important since the test creates
four vdevs of this size (20GB) and initializes them. This
space reduction is need for running 3 CI VMs in parallel.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #19134
ZTS: Make zpool_prefetch_001_pos faster
zpool_prefetch_001 creates 16 thousand files, and makes copies of a
fraction of them. This takes a long time, especially when run on the
slow CI VMs.
Make it faster by only creating 8k files, and do some of the creations
in parallel. Both optimizations help to bring down the test time
from 40sec -> 14sec, as measured on a local, 10-core, VM.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #19133
rpm: Add support for building akmod packages
Add akmod (Automatic Kernel Module) support to ZFS, providing a third
packaging option alongside the existing DKMS and kmod packages.
Akmod packages automatically build binary kmod RPMs when new kernels
are installed, enabling:
- Automatic kernel module rebuilds on kernel updates
- Distribution of binary kmod RPMs without recompilation
- Seamless minor version upgrades on RHEL/Fedora and clones
Key changes:
- rpm/generic/zfs-kmod.spec.in: Add conditional akmod support via
rpmbuild --define '_enable_akmod 1'. When enabled, sets
buildforkernels=akmod and repo=rpmfusion so kmodtool generates
an akmod-zfs package. Add -common subpackage for shared
license/doc files. Guard chmod and %files sections for empty
kernel_versions. Make kmod-devel packages opt-in via
%bcond_with devel to prevent akmods from installing unnecessary
[35 lines not shown]
zdb: account pending DDT-log frees in leak detection
With fast dedup, the last decref of a block lands in the DDT log, and
the physical free happens only when the log entry is flushed back into
the DDT. ddt_phys_decref() only decrements the in-memory refcount, the
log append path issues no free, and ddt_phys_free() -- the only caller
of zio_free() on this path -- runs from ddt_sync_flush_entry(). A pool
examined inside that window contains blocks that no block pointer
references but that are still allocated. zdb's leak detection never
enumerates the log, so it misreports those blocks: as "leaked space"
and a block traversal size mismatch in the generic accounting
(print-only since d63f5d7e50), and as a fatal "obsolete indirect
mapping count mismatch" when such a block sits behind a removed vdev's
indirect mapping -- which is what intermittently fails the zloop CI
job.
Count the phys that the flush will free the same way as the
deferred-free bplist, under a new "DDT pending free" bucket:
reconstruct the BP, clear the dedup bit exactly as ddt_phys_free()
[40 lines not shown]
zdb: don't index an indirect vdev's metaslabs in the --class filter
zdb_count_block()'s --class filter resolves a block's allocation class
by indexing the top vdev's metaslab array. An indirect vdev, left
behind by a device removal, normally has no metaslabs; zdb synthesizes
them in zdb_leak_init_prepare_indirect_vdevs() so that zio_claim() has
somewhere to claim into. That only happens when leak tracking is
enabled, so under -L the array is NULL and any block whose first DVA
names a removed vdev dereferences it.
Classify an indirect vdev from its allocation bias instead. The bias
determines the vdev's primary metaslab group, and is available whether
or not the synthetic metaslabs exist.
For an indirect vdev with more than zfs_embedded_slog_min_ms metaslabs
this also changes what the filter reports. The synthetic metaslabs
carry no space maps, so they all look empty, and vdev_metaslab_init()
moves the emptiest into the embedded log group; blocks in that range
then classified as "other". Measured on a ztest pool with two removed
[16 lines not shown]
ZTS: don't mistake a compiled-in kmemleak for a running one
kmemleak_late_init() creates /sys/kernel/debug/kmemleak before it checks
whether the detector came up, so on a kernel built
CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF and booted without kmemleak=on the
file is there and kmemleak is not. zfs_get_009_pos then skips, and the
two send_realloc tests quietly shrink their workload, on kernels
carrying none of the overhead those concessions are for.
Ask /proc/slabinfo instead: kmemleak_init() does not create the
kmemleak_object cache when it bails out, and SLAB_NOLEAKTRACE keeps that
cache from being merged away under another name.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19123
zio_crypt: free the key unwrap uios when decryption fails
zio_crypt_key_unwrap() allocates a pair of iovec buffers through
zio_crypt_uios_init_os() and releases them with zio_crypt_uios_fini_os()
once the keys have been decrypted. When zio_decrypt_os() fails it jumps
straight to the error label, which skips that release, so both buffers
are leaked.
The failure is reachable from userland: any key unwrap that does not
authenticate leaks 64 bytes twice, so a loop of "zfs load-key" calls
with the wrong passphrase grows kernel memory without bound. It is not
Linux specific -- the FreeBSD implementation allocates these buffers as
well.
Release the uios as soon as the crypto operation returns and check the
result afterwards, matching zio_crypt_key_wrap(), which already frees
them unconditionally on both paths. The buffers are then released
exactly once, and the error label continues to own nothing.
[7 lines not shown]
ZTS: fix test-runner crashing immediately under -m
sudo has no -c option, so it exits with a usage error and the run dies
before the first test case.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #19117
zpool: Add zpool status -vv error ranges
Print the byte error ranges with 'zpool status -vv'. This works
with the normal zpool status formatting flags (-p, -j, --json-int).
In addition:
- Modify ZFS_IOC_OBJ_TO_STATS ioctl to optionally return "extended"
object stats.
- Let zinject corrupt zvol data.
- Add test case.
- Add DIOCGPHYSPATH geom attribute to FreeBSD zvols. This works
basically the same as Linux's BLKZNAME ioctl, and returns the
dataset name for a zvol (like "tank/vol"). This is used by zinject.
This commit takes code from these PRs: #17502 #9781 #8902
Reviewed-by: Alek Pinchuk <Alek.Pinchuk at connectwise.com>
Reviewed-by: Akash B <akash-b at hpe.com>
[5 lines not shown]
Linux: skip hardened usercopy checks in zfs_uiomove_iter()
The check adds nothing here. It only verifies the copy stays inside
the source object, and for a vmalloc address that object is the entire
SPL slab, not the buffer. The DMU already bounds every transfer to
the individual dbuf. The user-supplied destination is validated
inside _copy_to_iter() and is unchanged.
fs/proc/kcore.c and drivers/dax/super.c call the underscore variants
for the same reason. Both are EXPORT_SYMBOL and predate 4.18.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Ameer Hamza <ahamza at ixsystems.com>
Reviewed-by: Richard Yao <richard at ryao.dev>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19115
