libzfs_pool: document export and initialize functions
Add brief docstrings to zpool_export(), zpool_export_force(),
zpool_initialize() and zpool_initialize_wait().
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18514
Consistently encode DRR_BEGIN packed nvlist payloads with NV_ENCODE_XDR
Currently, zfs send generates a mix of nvlist encodings in DRR_BEGIN
records, some XDR and some in native byte order. The result is that
most streams currently can't be zfs received on opposite-endian systems.
zfs send generates the outer wrappers for compound streams in userspace,
and it explicitly requests NV_ENCODE_XDR format for those records. But
the BEGIN records for individual datasets are generated on the kernel
side, in dmu_send.c, where fnvlist_pack() is used for encoding. That
routine hard-wires NV_ENCODE_NATIVE format.
This PR replaces the fnvlist_pack() call with a direct call to
nvlist_pack() that specifies NV_ENCODE_XDR.
Tests are included to verify that native-encoded nvlists are not
generated by any kernel path that attaches nvlists to BEGIN records.
There's also a check for XDR encoding in the outer wrapper of
replication streams in case there is ever a regression there.
[15 lines not shown]
zap: internal interface cleanup
Similar to previous, though a much lighter touch because these are not
"public" interfaces.
- reorganising functions into groups, by rough function class.
- matching header order to source order, to make it a little easier to
find things.
- adding light documentation to functions that had none.
Note that I've not added any documentation for the mzap_* and fzap_*
functions, as part of this commit series is laying the groundwork to
hide those functions in their backend modules; such documentation would
become obsolete very quickly.
No actual code changes.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
[4 lines not shown]
zap: public interface cleanup
- reorganising functions into groups, collections of the variants of the
same function.
- matching header order to source order, to make it a little easier to
find things.
- moving per-function documentation from source to header.
- adding light documentation to functions that had none.
No actual code changes.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Akash B <akash-b at hpe.com>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #18516
zap: split implementation out into more files
The ZAP code is mixed up across a few files without clear separation of
concerns. This splits it out from three source files to five:
- zap.c: the bulk of the "public" interface
- zap_impl.c: internals shared across all backends
- zap_micro.c: microzap backend
- zap_fat.c: fatzap backend: core logic
- zap_leaf.c: fatzap backend: leaf blocks
Note that this doesn't not change any code, just moves functions around.
Also note that right now the microzap and fatzap backends know more
about each other than is healthy. This change is simply marking out
where different things should live in the end, to make it easier for
that refactoring work to begin.
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
[4 lines not shown]
dsl_dir: avoid dd_lock during snapshots_changed updates
Avoid holding dd_lock while updating the on-disk
snapshots_changed timestamp.
Both dsl_dir_zapify() and zap_update() may dirty buffers
and recurse into space accounting, which can take dd_lock.
Holding dd_lock across either operation can therefore
preserve the lock-order inversion reported by lockdep.
Only protect the in-memory dd_snap_cmtime update
with dd_lock. Perform the zapify and ZAP update without
dd_lock held, and retry the on-disk write if another updater
advanced dd_snap_cmtime while the write was in progress.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ZhengYuan Huang <gality369 at gmail.com>
Co-authored-by: gality369 <gality369 at example.com>
Closes #18472
zpool-attach.8: add EXAMPLES section
Mirror-attach (shared with zpool.8 example 5) and raidz expansion.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Tino Reichardt <milky-zfs at mcmilk.de>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18508
CI: skip qemu matrix for documentation-only pull requests
Add a new "docs" CI type, selected when every file modified by a
pull request matches a documentation pattern (man pages, .md,
AUTHORS, COPYRIGHT, LICENSE, NOTICE, .gitignore). For this type the
os_selection is empty and the qemu matrix runs no jobs.
This affects only pull requests whose entire diff is documentation.
Any change touching a non-documentation file continues to be
classified as full, quick, linux, or freebsd by the existing
file-path rules, and a manual ZFS-CI-Type commit tag still overrides
that classification.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18518
zfs.4: Fix documentation of zfs_arc_dnode_reduce_percent
Fixes: 25458cbef Limit the amount of dnode metadata in the ARC
Fixes: 5b9f3b766 Soften pruning threshold on not evictable metadata
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Mateusz Piotrowski <0mp at FreeBSD.org>
Closes #18513
Linux: annotate nested xattr setattr znode locks
zfs_setattr() updates both the target znode and its hidden xattr
directory when ownership, mode, or project ID changes. The xattr
directory uses the same z_acl_lock and z_lock classes as the
parent znode, so lockdep reports recursive locking when the
second znode's mutexes are acquired.
This is a lockdep false positive rather than a real deadlock.
attrzp is the target file's hidden xattr directory, and the code
does not acquire these znode mutexes in the reverse order.
Acquire the attrzp mutexes with mutex_enter_nested() so lockdep
treats them as nested.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ZhengYuan Huang <gality369 at gmail.com>
Co-authored-by: gality369 <gality369 at example.com>
Closes #18506
zarcstat: detect attached L2ARC device with no data
zarcstat and zarcsummary detected L2ARC presence using the l2_size
kstat, which is data held in L2ARC, not whether a cache device is
attached. When a cache device was attached but empty (freshly added,
or fully evicted):
- zarcstat rejected "-f l2*" with "Incompatible field specified!"
- zarcsummary printed "L2ARC not detected, skipping section",
hiding cumulative I/O history and health counters
Expose the existing l2arc_ndev counter as a new kstat l2_dev_count.
It is maintained by l2arc_add_vdev() and l2arc_remove_vdev(), so it
tracks attachment in real time. Use it in both tools, falling back to
l2_size for compatibility with older kernel modules.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Ameer Hamza <ahamza at ixsystems.com>
Closes #18499
zdb: detect BRT and DDT leaks during block traversal
During -b traversal, track BRT and DDT reference counts and report
blocks claimed more times than their reference tables account for
if it causes claim errors, instead of just asserting it. Also
report entries with references not fully consumed by the traversal.
Add zdb leaks checks to cloning and dedup tests. This should make
sure the pools are in a sane state after completing the functional
tests.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18494
ZTS: redundancy_draid_spare1
Preserve the 'zpool status' output used to calculate the number of
checksum errors so it can be logged on failure. Several instances have
been observed in the CI where cksum was set to a non-zero value, yet a
subsequent run of 'zpool status' on failure showed no checksum errors.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #18500
Add some more file layout output, triggered by -v
With one -v, the block type (parity or data) is printed (matching
the ASCII-art version); with two -v, the offset into the file is
also printed.
This also updates the man page, and adds some simple
test scripts.
Sponsored-by: Klara, Inc.
Sponsored-by: Wasabi Technology, Inc.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Sean Fagan <sean.fagan at klarasystems.com>
Signed-off-by: Sean Fagan <sean.fagan at klarasystems.com>
Closes #18470
sa: fix sa_add_projid lock ordering
sa_add_projid() currently acquires hdl->sa_lock before zp->z_lock.
Several same-znode update paths take zp->z_lock and then call
sa_update() or sa_bulk_update() on the same SA handle.
On Linux, FS_IOC_FSSETXATTR reaches zfs_setattr() through
zpl_ioctl_setxattr() without outer inode serialization. This makes
the reversed lock order a real ABBA deadlock rather than a lockdep
false positive when projid is added to an old-format inode while
another thread updates the same znode.
Acquire zp->z_lock before hdl->sa_lock in sa_add_projid() to match
the existing znode update ordering.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ZhengYuan Huang <gality369 at gmail.com>
Co-authored-by: gality369 <gality369 at example.com>
Closes #18503
ZTS: zpool_iostat_002_pos remove sleep
In the CI environment commands may occasionally take longer than
expected. For zpool_iostat_002_pos this can cause a failure if fewer
than the expected numbers of lines are logged in time. To prevent
this issue relax the time constraint and simply verify the command
ran to completion and generate the correct number of lines.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #18501
Vdev allocation bias/class change
Normal, special and dedup vdevs differ only by space allocation
bias. Normal and special vdevs might even legally store blocks
targeted to other classes. Dedup vdevs don't normally do it, but
there is no real reason why they can't. Considering this, it is
not impossible to change the allocation bias for those vdevs.
This change introduces a new top-level vdev property -- alloc_bias,
reporting current bias for the vdev, and allowing to change it.
This allows to easily change vdev role in a pool, especially if
vdev removal is impossible. To not complicate the code, changes
take effect only on next pool import.
Changes to/from log vdev could also be theoretically possible, but
they are artificially blocked for now, partially due to additional
complications, and partially due to potential danger of placing
other blocks on log vdevs, that would otherwise be non-fatal.
[3 lines not shown]
ZTS: removal_with_export.ksh busy export
If the pool is active 'zpool export' will fail resulting in
a test failure. Swap log_must with log_must_busy so the export
is retried when reported as busy before failing the test.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #18498
zfs: annotate nested dd_lock in reservation sync accounting
When reservation sync updates a child's reserved space, it rolls the
delta into ancestor space accounting while still holding the child's
dd_lock. That locking order is intentional, but Linux lockdep sees
the ancestor acquisition as recursive because it lacks a nested lock
subclass annotation.
Teach the reservation-sync space-accounting path to acquire ancestor
dd_lock instances with a nested subclass. Keep the existing public
interfaces and accounting behavior unchanged by routing only the
ancestor rollup through local helpers.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: ZhengYuan Huang <gality369 at gmail.com>
Signed-off-by: gality369 <gality369 at example.com>
Closes #18497
ZTS: use 'zpool trim -w' in zpool_trim_partial.ksh
Don't use trim_progress() which is racy to wait for the pool trim
to complete. Instead use the wait (-w) option which is intended
for this.
Reviewed-by: Tino Reichardt <milky-zfs at mcmilk.de>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #18496
ZTS: Remove threadsappend_001_pos exception
Commit f828a80c may have resolved the underlying cause for
the occasional CI failures observed for this test. Remove
the exception to ensure any new occurrences are noticed.
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Tino Reichardt <milky-zfs at mcmilk.de>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #6136
Closes #18495
Zstd: rework ZSTD_isError symbol renaming
The import of Zstd v1.5.7 in a2ac9cd606ce2428c23cc89cec6f0392424e82c9
added an unconditional renaming of ZSTD_isError to zfs_ZSTD_isError
with an asm directive. Instead, do it with a define that is conditioned
on whether zstd_compat_wrapper.h is actually in use. Also add a define
to that header so that it can be detected. This allows the build to
work without using the compat wrapper.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Ryan Libby <rlibby at FreeBSD.org>
Closes #18483
linux: verify stale znodes in legacy fallocate
The mode=0 and FALLOC_FL_KEEP_SIZE preallocation path can reach
zfs_freesp() directly and call zfs_statvfs() before going through the
normal zpl_enter_verify_zp() boundary.
When zfs_rezget() tears down a failed SA reload, a stale inode may
remain alive in the VFS with z_sa_hdl cleared. The unchecked
fallocate path can then reach sa_lookup(zp->z_sa_hdl, ...) through
zfs_statvfs() or zfs_freesp() and crash on a NULL SA handle.
Use zfs_enter_verify_zp() in zfs_statvfs() so stale znodes are
rejected under the teardown lock for both fallocate and statfs.
Also wrap the direct zfs_freesp() call in
zpl_enter_verify_zp()/zfs_exit() so this path follows the same
validation rules as the other Linux ZPL file operations.
Fixes: f734301d2267
("linux: add basic fallocate(mode=0/2) compatibility")
[4 lines not shown]
Update description of spl_schedule_hrtimeout_slack_us
Clarify the effect of the non-zero value on wakeup coalescing.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18467
man: document three missing properties and tunables
Add manpage entries for parameters and properties that exist in
source but were not previously described:
- spl.4: spl_schedule_hrtimeout_slack_us
- zfsprops.7: longname
- vdevprops.7: raidz_expanding
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Christos Longros <chris.longros at gmail.com>
Closes #18467
CI: FreeBSD 15.1 PRERELEASE (#18490)
Update freebsd15-0s builder to freebsd15-1s and point it at the
15.1-PRERELEASE tag. The previous freebsd-15.0-STABLE images are
no longer available.
Additionally, add a freebsd15-0r stanza for the RELEASE.
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Fix long POSIX_FADV_DONTNEED for single block files
dbuf_whichblock() is not made to handle offsets beyond the block
end for single-block objects. Handle it in dmu_evict_range(),
similar to dmu_prefetch_by_dnode().
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18399
Closes #18489
CI/GCC: Add Fedora 44, fix build errors and threadsappend
- Add Fedora 44 to CI tests
- Fix build issues from the newer compiler. These are mostly 'char *'
to 'const char *' conversions.
- Fix threadsappend.c test waiting for the same thread TID twice.
This caused the test to hang on F44 (but strangely not other OSs?)
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #18478
Initialize vr_last_txg for rebuild
Only call txg_wait_synced() when rebuild IOs were issued for this
metaslab. This is a small optimization since in practice the first
metaslab is very likely to have allocations and cause vr_last_txg
to be initialized. After this point when processing empty metaslabs
txg_wait_synced() is called but with an already committed txg so it
will not wait. Still it's better not to call txg_wait_synced() at
all when it's not needed.
Reviewed-by: Andriy Tkachuk <atkachuk at wasabi.com>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #18482
Avoid flushing unrelated NFS exports on snapshot unmount
zfsctl_snapshot_unmount() called exportfs_flush() before every umount
attempt to drop NFS export cache references that pin the snapshot
mountpoint. The flush has global effect on the host's NFS exports and
clients, so paying it on every snapshot unmount (including auto-expire
rounds for snapshots that were never NFS-accessed) impacts unrelated
snapshots and clients.
ZFS cannot invalidate individual export cache entries because the
relevant sunrpc cache APIs are exported GPL-only. Defer the global
flush so it runs only when the umount has actually failed, then retry
once. Snapshots that are not NFS-pinned succeed on the first attempt
and never trigger the flush.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Youzhong Yang <yyang at mathworks.com>
Signed-off-by: Ameer Hamza <ahamza at ixsystems.com>
Closes #18476
