CI: Add tzdata, libcap-utils, and ksh93's stable branch to Alpine
Missing tzdata silently breaks history_007_pos's TZ-dependent timestamp
check (stays in UTC instead of shifting). Missing libcap-utils breaks
zoned_uid_common.kshlib's capsh-based tests ("unshare: failed to
execute"). And building ksh93 from its unpinned "dev" branch, rather
than the stable "1.0" branch, causes musl crashes in ksh93's own
environment-building code, hit by the zpool_iostat/zpool_status "-c"
custom-script tests.
Combined into one commit since all three touch the same function in
qemu-3-deps-vm.sh.
Fixes the following tests on Alpine 3.24:
- history/history_007_pos
- zoned_uid/zoned_uid_023_pos
- zoned_uid/zoned_uid_025_pos
- zoned_uid/zoned_uid_026_pos
- zoned_uid/zoned_uid_030_pos
[10 lines not shown]
dmu_objset_open_impl: unregister prop callbacks on error
If dsl_prop_register() fails partway, the error path frees the objset
without removing the callbacks already registered, leaving records
that point at freed memory. Call dsl_prop_unregister_all() first.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Martin Minkus <martin.minkus at gmail.com>
Closes #18903
ZTS: fix CDDL boilerplate in zpool_expand_007_pos.ksh
Commit 1b7143557 added this file with the old OpenSolaris-style CDDL
header (CDDL HEADER START/END wrappers, wrong URL). Replace with the
current short-form CDDL boilerplate, as done in f836c9ee7 for the
block_cloning FIDEDUPERANGE tests.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18985
ZTS: replace uncompress with gunzip in history tests
Alpine Linux (musl/BusyBox) does not provide uncompress; gunzip handles
.Z (LZW compress) files and accepts the same -f flag, making it a
drop-in replacement on all supported platforms.
Fixes the following tests on Alpine 3.24:
- history/history_001_pos
- history/history_007_pos
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18981
Keep a grown vdev from adopting an older pool's labels
Labels 2 and 3 sit at offsets relative to the end of a device, so
growing a vdev moves them onto space the pool has never written.
vdev_copy_uberblocks() already says as much of the uberblock rings
found there, "either empty or contain garbage", yet the config and the
uberblocks are still read from those labels as though they were ours.
Whatever pool held the device before is usually still on it, its labels
are perfectly well formed, and vdev_label_read_config() ranks labels by
nothing but txg, so a leftover label with a higher txg wins and
vdev_validate() fails the device for belonging to a foreign pool. The
reporter grew LVM volumes laid over used disks and got a raidz child
back FAULTED with "corrupted data"; where no redundancy is left to
cover the member, the pool suspends. Issue #16144.
Read the config for validation from the two labels at the head of the
device, whose offsets do not depend on its size, and fall back to the
trailing pair only when the head cannot be read, so a damaged head
still has the second copy behind it. vdev_label_sync() writes the
[45 lines not shown]
zstream_selftest_queue: fix alignas(type) build failure under C99
alignas() with a type-name argument is C11 syntax; this file is
compiled with -std=gnu99 where _Alignas only accepts constant
expressions. Replace alignas(uint64_t) with alignas(__alignof__(
uint64_t)), which is a GCC/Clang builtin available in all modes and
produces the same alignment value.
Musl's stdalign.h defines alignas unconditionally (unlike the glibc /
compiler built-in header which guards it behind a C11 version check),
so this manifests when building against musl (tested on Alpine 3.24
x86_64 and Gentoo AArch64).
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Garth Snyder <garth at garthsnyder.com>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18971
ZTS: fix mmp_set_hostid on Alpine Linux (musl libc)
musl libc's gethostid() ignores /etc/hostid and returns 0, so the
hostid verification in mmp_set_hostid always fails after zgenhostid
writes the file. Fall back to reading /etc/hostid directly with od
when the hostid command disagrees, which works correctly on both
glibc and musl systems.
Fixes the following tests on Alpine 3.24:
- mmp/mmp_active_import
- mmp/mmp_concurrent_import
- mmp/mmp_exported_import
- mmp/mmp_hostid
- mmp/mmp_inactive_import
- mmp/mmp_on_off
- mmp/mmp_on_thread
- mmp/mmp_on_uberblocks
- mmp/mmp_on_zdb
- mmp/mmp_reset_interval
[7 lines not shown]
CI: boot linux-stable instead of linux-virt on the Alpine runner
The Alpine runner's linux-virt kernel has CONFIG_SCSI_DEBUG disabled.
Several ZTS tests use the scsi_debug kernel module to simulate disks
that support expand and fault-injection scenarios real static disks
can't easily provide. Without it, load_scsi_debug() in blkdev.shlib
calls log_unsupported and they SKIP instead of running.
linux-stable has CONFIG_SCSI_DEBUG=m. Swap linux-virt/linux-virt-dev for
linux-stable/linux-stable-dev, and switch the extlinux default kernel
from "virt" to "stable" in /etc/update-extlinux.conf.
Fixes the following tests on Alpine 3.24 (SKIP -> PASS):
- cli_root/zpool_expand/zpool_expand_001_pos
- cli_root/zpool_expand/zpool_expand_003_neg
- cli_root/zpool_expand/zpool_expand_005_pos
- cli_root/zpool_expand/zpool_expand_006_pos
- cli_root/zpool_reopen/* (7 tests)
- cli_root/zpool_split/zpool_split_wholedisk
[15 lines not shown]
libspl: consult ZFS_HOSTID on FreeBSD as well
get_system_hostid() returns gethostid() on FreeBSD, so ZFS_HOSTID has
no effect there, while on Linux it overrides the SPL hostid. zloop.sh
exports it for the iterations which need a hostid, and the ztest(1)
ENVIRONMENT VARIABLES section describes it without restricting it to
one platform, so both should read it.
Check ZFS_HOSTID first and fall back to gethostid(), parsing and
masking the value exactly as the Linux implementation does. A value
which parses as zero is ignored there, so it is ignored here too.
Drop the "On Linux" qualifier from the -M description in ztest(1),
which this change makes untrue.
Suggested-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Michael Heller <michael.heller at gmail.com>
Closes #18918
Closes #18964
CI: run zloop with multihost testing enabled
Pass -M so the zloop workflow exercises multihost on a fraction of
its iterations.
Drop the zgenhostid call added by eb5c93fa8. It was there so ztest
could import a pool it had enabled multihost on, and zloop.sh now
supplies a hostid through ZFS_HOSTID for the iterations that need
one.
Suggested-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Michael Heller <michael.heller at gmail.com>
Closes #18918
Closes #18964
zloop.sh: add -M to run some iterations with multihost
Add an option to exercise multihost testing. It is off by default,
so a local zloop run needs no hostid and behaves as before.
When it is given, one iteration in five passes -M to ztest. Running
every iteration that way would cost the loop the dozen or so
operations ztest suppresses under -M, among them the zdb consistency
check. Skipping that check also makes those iterations finish about
twice as quickly, so a time limited loop completes proportionally
more of them. A raidz expansion iteration is left alone because
ztest forces -M off for one.
ztest needs a non-zero hostid to set the multihost property. Export
ZFS_HOSTID for the iterations that use -M rather than creating
/etc/hostid: the ZTS mmp test group skips itself when that file
exists, so creating it would disable that group on the same machine.
Suggested-by: Brian Behlendorf <behlendorf1 at llnl.gov>
[4 lines not shown]
ztest: create the pool with multihost enabled under -M
The -M option advertised simulating a pool imported on a remote host,
but nothing under it enabled multihost. The only writes to
spa_multihost were in ztest_mmp_enable_disable(), the function -M
disabled, so with that removed -M merely suppresses the operations
that conflict with multihost testing.
Add the multihost property to the pool ztest creates under -M. The
property persists, so subsequent imports run the MMP activity check,
which is the point of the option.
Setting it requires a non-zero hostid, since spa_prop_validate()
returns ENOTSUP without one. ztest creates its pool with VERIFY0(),
so -M on a host with no hostid would abort in ztest_init() rather
than explain itself. Check the hostid while processing options and
exit with a message instead.
Suppress the MMP write-failure suspension for such a run. ztest sets
[18 lines not shown]
ZTS: retry the zvol_misc_fua exports while the volume is busy
zvol_misc_fua exports and re-imports the pool for each of its two
blk-mq cases while the volume device exists, so an export can race with
the udev rule which opens it and fail with "pool is busy".
zvol_misc_trim has the same structure and already uses log_must_busy for
its exports.
Do the same here, so a transiently open device does not fail the test.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Michael Heller <75820586+mkhllr at users.noreply.github.com>
Closes #18970
scan: count skipped blocks as examined
dsl_scan_scrub_cb() passes a block outside the scan's txg range to
count_block_skipped() and returns before scn_examined is touched, so
scn_skipped can exceed scn_examined and the resume in dsl_scan_init()
wraps: zpool status reports 16.0E issued. The deferred resilver check
in dsl_scan_sync() reads the same counter, and a wrapped value never
falls below zfs_resilver_defer_percent, so a resilver that should
restart early stays deferred.
Count those blocks as examined as well as skipped; they were traversed,
and the two counters have to cover the same blocks. scn_phys is on
disk, so keep clamping the resume for pools an older version skewed,
and clamp scn_to_examine - scn_skipped, which wraps the same way.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Nick Price <nprice at FreeBSD.org>
Closes #18928
zstream: remove the enqueue mutex
The current zstream code uses an enqueue mutex to avoid the possibility
of enqueue notifications being dropped.
Unfortunately, the enqueue mutex is highly contested. Workers must hold
it while searching queues for work, while enqueuers must acquire it to
complete their enqueues. This contention results in a nontrivial
performance cost.
This PR removes the enqueue mutex and replaces it with a separate
dispatch thread. It also adds a lazy 100 microseconds between an
enqueue and the signal that wakes up a thread to service it. This
coalescing of notifications encourages larger batch sizes and reduces
the number of worker loops.
Other changes:
- There's now a pool-level count of unclaimed items that's maintained
[29 lines not shown]
ZTS: fix CDDL boilerplate in FIDEDUPERANGE tests
The block_cloning FIDEDUPERANGE test files (a5b536566) used the old
OpenSolaris-style CDDL header. When 7de66233f added boilerplate
checking to spdxcheck.pl, it did not update these files, breaking
the spdxcheck CI target. Replace with the current short-form CDDL
boilerplate.
Reviewed-by: Chris Longros <chris.longros at gmail.com>
Reviewed-by: Kamil Monicz <kamil at monicz.dev>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18972
ZTS: check project ID inheritance and renames within a project directory
Covers both of the preceding changes. For the inheritance: a new symlink
and a new FIFO each add an object to the containing directory's project,
which only happens once non-regular files inherit a project ID, and a
symlink can then be renamed into a different directory carrying the same
project ID, which rename(2) permits only when the two project IDs match.
For the rename exemption: a regular file and a symlink are created
before the directory is tagged, so they carry no project ID of their
own, as every symlink on an existing pool does. Renaming each of them
within that directory afterwards fails with EXDEV unless the
cross-project check exempts renames whose source and target directories
are the same. The same renames are then repeated for objects created
after the tagging, along with replacing a symlink with "ln -sfn", and
again in an inheriting subdirectory.
A rename that does cross into a different project is still refused.
[7 lines not shown]
Allow renames within a single directory under project inheritance
A rename that keeps the object in the directory it already lives in
cannot move it between projects, so refusing it with EXDEV is never
right. Objects created before the previous commit carry no project ID of
their own, which makes this reachable on existing pools: a symlink in a
project directory cannot be renamed even to another name beside itself,
and "ln -sfn" over an existing path fails.
zfs_link() keeps its unconditional check. It has no source directory to
compare against -- a hard link names an object that may live anywhere --
so there is no equivalent "the object is already here" case to exempt.
Linking a pre-existing symlink, device node or FIFO into the project
directory that already holds it therefore still fails with EXDEV, until
the object is given a project ID of its own.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Matt Turner <mattst88 at gmail.com>
Closes #18932
Inherit the project ID for every object type
zfs_mknode() only assigned a project ID to regular files and
directories, so a symlink, device node, FIFO or socket created inside a
directory with ZFS_PROJINHERIT set was left at ZFS_DEFAULT_PROJID. The
cross-project checks in zfs_rename() and zfs_link() compare the object's
project ID against the directory's, so such an object is treated as
foreign to the very directory holding it, and cannot be renamed or
linked there at all -- "ln -sfn", which creates the new symlink under a
temporary name and renames it into place, fails with EXDEV.
ext4 and XFS store a project ID on every inode type and so do not have
this problem. Do the same, and quota-check new symlinks and rename
whiteouts against the inherited ID rather than the default one, so their
space is accounted to the project that owns them.
This changes accounting for newly created objects: symlinks, device
nodes and FIFOs now consume the project's quota where they previously
consumed none, so on a dataset already at its project quota, creating
[12 lines not shown]
[zfs-2.3.9] Add workaround for device_access ZTS test
Add workaround to get the device_access ZTS tests working on 2.3.9.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
[zfs-2.2.11] Add workaround for device_access ZTS test
Add workaround to get the device_access ZTS tests working on 2.2.11.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
dmu_redact_snap: Do proper cleanup on ENAMETOOLONG
If the redaction bookmark name exceeds ZFS_MAX_DATASET_NAME_LEN, we
leak the dataset hold, long hold, and decrypt key mapping on the target
snapshot, plus the same three references on each origin snapshot in
args[].
The leaked long holds leave those datasets busy such that destroy,
unmount, and zfs send will return EBUSY and the key mappings never drop.
Using goto out like the other error paths avoids this.
Reported-by: Grok 4.6 Build Beta
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Richard Yao <richard at ryao.dev>
Closes #18962
dmu_recv_begin_check: dsl_dataset_rele() should be called on ds
We call dsl_dataset_hold() to initialize ds, and are supposed to call
dsl_dataset_rele() on ds when we are done. In two error paths, we call
dsl_dataset_rele_flags(). If ds is an encrypted dataset, this can drop
the key mapping for the dataset, which will cause run time issues. On
unencrypted datasets, this is a no-op.
Reported-by: Grok 4.6 Build Beta
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Richard Yao <richard at ryao.dev>
Closes #18961
Linux: allow more time for snapshot automount teardown
Snapshot destruction can race with the delayed dput()/mntput() work
triggered after accessing a snapshot through .zfs/snapshot. If that
work takes longer than 20 milliseconds, the snapshot remains long-held
and zfs destroy returns EBUSY, which for example causes ZTS tests to
fail spuriously.
Relax the unmount deadline from 20 to max 40 milliseconds. The wait
continues to exit earlier than that if the teardown completes earlier.
This fixes spurious ZTS failures such as snapshot/snapshot_00
{1,2,3,5,6,7,8,12}_pos and snapshot/rollback_002_pos especially with
--enable-debug builds since e8e3076927039bc6a2094c43e3ac4e255884177f
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Rob Norris <rob.norris at klarasystems.com>
Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek at mac.com>
Closes #18965
[zfs-2.2.11] Add 'capsh' to commands.cfg
Add missing 'capsh' to commands.cfg. It was included in
master with 7839c4b5e1 but that was not backported to this branch.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
[zfs-2.3.9] Add 'capsh' to commands.cfg
Add missing 'capsh' to commands.cfg. It was included in
master with 7839c4b5e1 but that was not backported to this branch.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
[zfs-2.4.4] Add 'capsh' to commands.cfg
Add missing 'capsh' to commands.cfg. It was included in
master with 7839c4b5e1 but that was not backported to this branch.
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
