Zstd: Document update policy
Add the Zstd update policy to the subtree README.
Also update the documented location of zstd-in.c to match upstream
changes, and normalize naming from 'ZSTD' to 'Zstd'.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18089
Zstd: Restore SPDX license identifiers
When updating Zstandard to version 1.5.7 the SPDX license identifiers
were lost. This commit restores them.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18089
Zstd: Fix ASan poisoning for pooled Zstd contexts
The Zstd context mempool can reuse buffers that were previously poisoned
under AddressSanitizer, leading to false-positive use-after-poison reports
during zloop and other stress tests.
Explicitly unpoison memory when handing buffers out to Zstd and poison the
user-visible region again when buffers are returned to the pool. This makes
the allocator ASan-correct while preserving existing pooling behavior.
Also fix non-standard void * pointer arithmetic in zstd_free() and remove an
early return in zstd_dctx_alloc() so kmem_type/kmem_size are always set on
pool hits.
This only affects ASan bookkeeping in user space, does not change runtime
behavior in non-ASan configurations, and does not affect on-disk formats.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18089
Zstd: Integrate v1.5.7 into the ZFS build system
This commit builds on the previous zstd library update and adds the
necessary ZFS integration and build system changes required to make
zstd 1.5.7 compile and function correctly.
Changes:
- Add zstd_preSplit.c (new in 1.5.7) to all build systems.
- Enable x86_64 assembly in userspace (huf_decompress_amd64.S).
- Disable assembly in kernel for RETHUNK/IBT compatibility.
- Disable intrinsics in kernel for EL10 x86_64-v3 baseline.
- Disable tracing in kernel builds for AArch64 compatibility.
- Fix ZSTD_isError symbol renaming with __asm__ directive.
- Rename abs64 to ZSTD_abs64 (FreeBSD kernel conflict).
- Fix bitstream.h attributes (MEM_STATIC -> FORCE_INLINE_TEMPLATE).
- Remove xxhash.c from BSD build (now header-only).
- Update symbol names in zstd_compat_wrapper.h.
- Ignore checkstyle for zstd-in.c.
[6 lines not shown]
Zstd: Update bundled library to v1.5.7 without further adjustments
This commit only replaces the bundled source and does not include any
ZFS integration changes. Because the build depends on integration
adjustments, it will fail until the accompanying integration commit is
applied.
Upstream release: https://github.com/facebook/zstd/releases/tag/v1.5.7
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Moch <mail at alexmoch.com>
Closes #18089
FreeBSD: Remove references to DEBUG_VFS_LOCKS
This option is removed upstream in favour of plain INVARIANTS.
VNASSERT is always defined so I see no reason to use it conditionally.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Mark Johnston <markj at FreeBSD.org>
Closes #18136
FreeBSD: unbreak compilation on i386
tests/zfs-tests/cmd/mmap_seek.c: use correct printf specifier
module/zfs/vdev.c: vdev_clear(): correctly cast argument to
atomic_add_64().
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Martin Matuska <mm at FreeBSD.org>
Closes #18096
Fix --enable-invariants on FreeBSD
The make symbols were never getting forwarded to the correct make
subprocess. As far as I can tell, this has never worked. Either that,
or something has changed in the behavior of make.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alan Somers <asomers at gmail.com>
Closes #18131
Fix history logging for `zpool create -t`
`zpool create` is supposed to log the command to the new pool’s history,
as a special record that never gets evicted from the ring buffer. but
when you create a pool with `zpool create -t`, no such record is ever
logged (#18102). that bug may be the cause of issues like #16408.
`zpool create -t` (83e9986f6eefdf0afc387f06407087bba3ead4e9) and `zpool
import -t` (26b42f3f9d03f85cc7966dc2fe4dfe9216601b0e) are both designed
to override the on-disk zpool property `name` with an in-core
“temporary” name, but they work somewhat differently under the hood.
importing with a temporary name sets `spa->spa_import_flags |=
ZFS_IMPORT_TEMP_NAME` in ZFS_IOC_POOL_IMPORT, which tells
spa_write_cachefile() and spa_config_generate() to use the
ZPOOL_CONFIG_POOL_NAME in `spa->spa_config` instead of `spa->spa_name`.
creating with a temporary name permanently(!) sets the internal zpool
property `tname` (ZPOOL_PROP_TNAME) in the `zc->zc_nvlist_src` of
[17 lines not shown]
DDT: Add locking for table ZAP destruction
Similar to BRT, DDT ZAP can be destroyed by sync context when it
becomes empty. Respectively similar to BRT introduce RW-lock to
protect open context methods from the destruction.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18115
spdxcheck: enforce SPDX license tags on build system files
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18077
build: add SPDX license tags to build system files
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18077
Add fh_to_parent export definition
This commit adds support for converting a file handle to its
parent dentry. This is called in exportfs_decode_fh_raw()
when subtree checking is enabled in NFS. Defining this and
handling the expanded filehandles allows the knfsd to succeed
in handling the file handle where it might otherwise fail
with ESTALE when trying to open by filehandle.
A side effect of this change is that name_to_handle_at(2)
and open_by_handle_at(2) now support AT_HANDLE_CONNECTABLE.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Ameer Hamza <ahamza at ixsystems.com>
Signed-off-by: Andrew Walker <andrew.walker at truenas.com>
Closes #18099
spl: remove a _KERNEL check
This code is only compiled for the Linux kernel module, so that define
is always set.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18117
spl: unexport kstat_proc_entry functions
These are used to implement the kstat and procfs_list interfaces, and
aren't used from outside. There's no need to export them.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18117
spl: lift 64-bit math compat out to separate file
It's a lot of rarely-compiled code, so move it to the side to make other
code easier to read.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18117
spl: remove old atomic lock
Long ago, SPL atomics were implemented as a global spinlock over
conventional operations. In 5e9b5d832b (2009-10) they was converted to
proper atomics, with the spinlock retained as a fallback.
The switch to compile with the fallback was later removed in a91258913f
(2018-05), but the code it enabled wasn't. So lets do that.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18117
icp: emit .note.GNU-stack section for all ELF targets
On FreeBSD, linking the zfs kernel module with binutils ld 2.44 shows
the following warning:
ld: warning: aesni-gcm-avx2-vaes.o: missing .note.GNU-stack section
implies executable stack
ld: NOTE: This behaviour is deprecated and will be removed in a
future version of the linker
Some of the `.S` files under `module/icp/asm-x86_64/modes` check whether
to emit the `.note.GNU-stack` section using:
#if defined(__linux__) && defined(__ELF__)
We could add `&& defined(__FreeBSD__)` to the test, but since all other
`.S` files in the OpenZFS tree use:
#ifdef __ELF__
[6 lines not shown]
When receiving a stream with the large block flag, activate feature
ZFS send streams include a feature flag DMU_BACKUP_FEATURE_LARGE_BLOCKS
to indicate the presence of large blocks in the dataset. On the sending
side, this flag is included if the `-L` flag is passed to `zfs send`
and the feature is active in the dataset. On the receive side, the
stream is refused if the feature is active in the destination dataset
but the stream does not include the feature flag.
The problem is the feature is only activated when a large block is
born. If a large block has been born in the destination, but never
the source, the send can't work. This can arise when sending streams
back and forth between two datasets.
This commit fixes the problem by always activating the large blocks
feature when receiving a stream with the large block feature flag.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Austin Wise <AustinWise at gmail.com>
Closes #18105
Fix zfs_open() to skip zil_async_to_sync() for the snapshot
Fix zfs_open() to skip zil_async_to_sync() for the snapshot, as it won't
have any transactions. zfsvfs->z_log is NULL for the snapshot.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Jitendra Patidar <jitendra.patidar at nutanix.com>
Closes #18091
Add snapshots_changed_nsecs dataset property
Add a read-only dataset property, snapshots_changed_nsecs, which
exposes the nanosecond resolution version of snapshots_changed.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Wolfgang Hoschek <wolfgang.hoschek at mac.com>
Closes #17998
Closes #18031
ZTS: add regression test for #17180
In #17180, we fixed an interesting bug that i believe i hit in one of my
pools, but as far as i can tell, there was no test for it.
this patch adds a regression test for #17180, minimised from my attempts
to reproduce the bug in a way that resembled the history of my pool.
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Adam Moss <c at yotes.com>
Signed-off-by: delan azabani <dazabani at igalia.com>
Closes #18109
Rename several printf attributes declarations to __printf__
For kernel builds on FreeBSD, we redefine `__printf__` to
`__freebsd_kprintf__`, to support FreeBSD kernel printf(9) extensions
with clang.
In OpenZFS various printf related functions are declared with
`__attribute__((format(printf, X, Y)))`, so these won't work with the
above redefinition. With clang 21 and higher, this leads to errors
similar to:
sys/contrib/openzfs/module/zfs/spa_misc.c:414:38: error: passing
'printf' format string where 'freebsd_kprintf' format string is
expected [-Werror,-Wformat]
414 | (void) vsnprintf(buf, sizeof (buf), fmt, adx);
| ^
Since attribute names can always be spelled with leading and trailing
double underscores, rename these instances.
[7 lines not shown]
Add handling for STATX_CHANGE_COOKIE
This commit adds handling for the STATX_CHANGE_COOKIE so that
we can properly surface the ZFS znode sequence to NFS clients via
knfsd.
If knfsd does not have STATX_CHANGE_COOKIE in statx result then
it will synthesize the NFS change_info4 structure and related
change4id values algorithmically based on the ctime value of the
file. Since internally ZFS is using ktime_get_coarse_real_ts64()
for the timestamp calculation here it introduces the possiblity
that the change will not increment the change4id of directories
/ files causing a failure in the client to invalidate its attr
cache (among other things). See RFC 8881 Section 10.8 for
discussion of how clients may implement name and directory
caching.
Notable in this commit is that we are not initializing the
inode->i_version to the znode->z_seq number. The reason for this
[27 lines not shown]
kmem: don't add __GFP_RECLAIMABLE for KM_VMEM allocations
vmalloc()'d memory is not movable/reclaimable, so __GFP_RECLAIMABLE is
not a valid flag, and since 6.19 the kernel warns if you use it.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18107
cmd/zfs: clone: accept `-u` to not mount newly created datasets
Signed-off-by: Ivan Shapovalov <intelfx at intelfx.name>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #18080
CI: Add Alpine Linux 3.23 runner to the pipeline (#18087)
Add an Alpine Linux 3.23 runner to the CI chain to run OpenZFS builds
and tests against musl libc.
Currently, zfs_send_sparse is killed after 10 minutes on Alpine, causing
cascading EBUSY failures in the test suite. With zfs_send_sparse
disabled, the ZFS test suite reaches a pass rate of 94.62%.
This commit introduces the required Alpine-specific setup and a small
set of shell and cloud-init compatibility fixes that also apply to
existing Linux runners.
The Alpine runner is not enabled by default and is not executed for new
pull requests.
Sponsored-by: ERNW Research GmbH - https://ernw-research.de/
Signed-off-by: Alexander Moch <amoch at ernw.de>
[2 lines not shown]
cmd/ztest: avoid `PATH_MAX` stack allocation in `ztest_get_zdb_bin()` (#18085)
Calling realpath(path, buf) can trigger fortified header wrappers that
allocate a PATH_MAX-sized temporary buffer on the stack, exceeding the
4 KiB frame limit on some systems. Use the heap-allocating
realpath(path, NULL) form instead.
Sponsored-by: ERNW Research GmbH - https://ernw-research.de/
Signed-off-by: Alexander Moch <amoch at ernw.de>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
kmem: don't add __GFP_COMP for KM_VMEM allocations
It hasn't been necessary since Linux 3.13
(torvalds/linux at a57a49887eb33), and since 6.19 the kernel warns if you
use it.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18053
kmem: don't pass __GFP_HIGHMEM to __vmalloc
Since Linux 4.12 (torvalds/linux at 19809c2da28ae) __GFP_HIGHMEM has been
automatically added to calls to __vmalloc() internally, so we don't need
it anymore. This is good, because since 6.19 the kernel warns if you use
__GFP_HIGHMEM.
Sponsored-by: https://despairlabs.com/sponsor/
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <robn at despairlabs.com>
Closes #18053
