CI: Remove centos-streamX from default builders
In practice, we've seen limited benefit from always running the
centos-stream9 and centos-stream10 builders as part of the CI.
Remove them from the default set of builders to reduce the CI
load per PR. They may still be optionally requested.
Reviewed-by: Alexander Moch <mail at alexmoch.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #19182
build: Use release version from META for git builds
This allows easier package upgrade management for package git builds.
The previous behavior had the release number be the number of patches
on top of a ZFS release or on top of master. This is an issue if
patches are removed, thus making the release version not monotonically
increasing. This won't happen if only using official commits because
the history is treated as immutable. But for those wanting to carry
patches that have not been accepted, this can happen.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #18953
Closes #19159
pkg: Fix rpm warnings about not being able to read its database
When building on many debian machines, the RPM database, by default at
/var/lib/rpm/rpmdb.sqlite, does not exist and can not be created when
running as an unprivileged user. This will cause any rpm command that
ties to access the database to print this obnoxious error message:
error: Unable to open sqlite database /var/lib/rpm/rpmdb.sqlite: unable to open database file
error: cannot open Packages index using sqlite - Operation not permitted (1)
error: cannot open Packages database in /var/lib/rpm
It appears as though this doesn't cause any failures and is really a
warning. However, it can be printed more than 20 times when making the
deb-kmod target.
Detect when the database file is not readable at configure time and
setup various rpm and rpmbuild invocations to use the root of the build
directory as the database directory path. The alien program internally
runs various rpm queries, however it provides no good way to tell those
[7 lines not shown]
deb: Print error message when build depends are not satified
An error message is already displayed by dpkg-buildpackage when build
dependencies are not met. However, the new error message shows the
exact command to use to satisfy those dependencies.
Rename the target native-deb-local to deb-local-common because its
rules are common to both deb and native-deb package builds. And make
nateive-deb-local depend on both deb-local-common and the new
native-deb-check-build-dep.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19161
deb: Allow building when builddir is not srcdir
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19160
rpm: Make dash compatible
On some systems, notaby Debian and many of its derivatives, /bin/sh is
a symlink to /usr/bin/dash. The scripts created for the various
rpmbuild stages are run with /bin/sh. The construct {a,b,c} does not
get expanded in dash, as it does in bash's sh.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19158
FreeBSD: remove legacy ioctl support
When FreeBSD switched their ZFS to OpenZFS in 13, a compat layer was
included to allow older userspace to work with newer kernel modules and
vice-versa. Since FreeBSD 12 is now almost 3 years out of support, we
shouldn't need this compatibility support any longer, and can remove it.
The small versioned wrapping struct zfs_iocparm_t is still in use for
current ioctls (and honestly a pretty good idea), so that's retained,
and we're now explicitly enforcing the version and size fields are what
they're supposed to be.
Sponsored-by: TrueNAS
Reviewed-by: Kyle Evans <kevans at FreeBSD.org>
Reviewed-by: Ryan Moeller <ryan.moeller at klarasystems.com>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19157
Propagate vdev state change on probe failure
Currently, when probe fails on some vdev, it would became
faulted, but the parent's vdev state won't be changed. So we
can have a situation, for example, when several disks in draid
vdev are faulted (domain failure), but draid vdev itself and
pool are online, which doesn't seem right.
Solution: set isopen argument to false when calling
vdev_set_state() from spa_async_fault_vdev(). It's not quite
clear why it was set to true in the first place when the code
was introduced at commit c4f2f1aa2d.
isopen argument to vdev_set_state() is normally set to true
when we are "in the process of opening children depth-first",
in which case we don't want to propagate the change of state
to the parent, but it doesn't seem to be the case here.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
[2 lines not shown]
libzfs: do not iterate snapshots for a mountpoint changelist
The mountpoint property does not apply to snapshots, so change_one()
drops every snapshot it is given. Iterating over them while gathering
a mountpoint changelist therefore costs a full stats ioctl per snapshot
and never contributes an entry, which makes "zfs set mountpoint", "zfs
inherit mountpoint" and "zfs set volsize" scale with the number of
snapshots for no reason.
Iterate filesystems only. This also covers canmount, volsize and zoned,
since those are handled through the mountpoint changelist.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Ameer Hamza <ahamza at ixsystems.com>
Closes #19137
ZTS: Speed up replace_resilver_sit_out
Reduce replace_resilver_sit_out test time from 1:14 to 30sec.
- Use 4 "lanes" in zinject instead of 1.
- Use a 1M recordsize to speed up replacements.
- Use 1 second zfs_txg_timeout to help with replacement
Also, double the number or reads to more reliably hit the sit-out
condition. This test has been flaky in the past.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Moch <mail at alexmoch.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #19156
zstream: command line and man page cleanup
This PR makes several changes to the user-facing command line syntax and
documentation:
- The `decompress,` `drop_records,` and `recompress` subcommands now
allow naming a stream file on the command line, bringing them into
line with `dump` and `raw.` Reading from standard input remains an
option.
- Tests have been modified to exercise command-line specification of
input files.
- The man page has been updated and copy edited.
- Subcommands are presented in a consistent (alphabetical) order in the
man page, the usage message, and the source code.
- The man page and usage message now use identical names and
[71 lines not shown]
zed: fix failure domain detection for unavail case
When the domain fails and there is no traffic, zed might get
list.suspect event with the disks entering into UNAVAIL state,
instead of zfs.statechange events with the disks entering into
FAULTED state. We should handle that situation similarly and
and don't start resilvering, which would be only a huge waste
of CPU and i/o bandwidth resources.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Andriy Tkachuk <atkachuk at wasabi.com>
Related-to #18148
Closes #19154
rpm: Add configure caching if used in main configure run
When a user enables configure caching when running configure, eg. by
passing -C, enable configure caching in the corresponding rpm builds.
The configure cache from the initial configure run by the user can not
generally be used because the host and build values will be different.
Normally those values are left unset when invoking configure whereas
the configure from rpmbuild explicitly sets them.
However, the intent of the user to enable configure caching can still
be satisfied by using a cache file solely for rpm builds. This implies
that initially, when building an rpm package with configure caching
enabled, that the first two runs of configure will be uncached, the
user-initiated and rpmbuild initiated ones. The cache file is named
config.rpm.cache and is located in the root of the build directory.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19111
Closes #19148
Make the spa_config_lock reader fast path lockless
Readers took scl_lock on every spa_config_enter()/exit(), which cost
~29% of CPU time in 4KB record write workloads. Fold the writer gate
into the reference count: SCL_COUNT_WRITER in the top bit of scl_count
lets a reader take a reference and test for a writer in one atomic
operation, and lets the last reader detect a waiting writer without a
second load. Writers still serialize on scl_lock, and priority readers
still take it, since they must tell a waiting writer from a holding one.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin at TrueNAS.com>
Closes #19131
Avoid dp_lock in the dirty space accounting
dsl_pool_dirty_space() and dsl_pool_undirty_space() are called once per
logical block, from open context and from every ZIO completion, all
serializing on the single pool-wide dp_lock. On a 12-thread 32KB
overwrite benchmark they took 33% of CPU, almost all of it spinning.
The lock only protected the counters against each other, so make
dp_dirty_pertxg/dp_dirty_total and dp_sync_reserve_pertxg/_total plain
atomics. The subtractions clamp at zero via a CAS loop. Reading these
counters without the lock was already the norm.
The cv_signal() on every accounting event is only ever needed by
dmu_tx_wait(), which sleeps solely at the hard zfs_dirty_data_max wall,
so gate it on a count of waiters. Both sides need membar_sync(), since
the atomics used here provide no ordering. dmu_tx_wait() in turn only
registers as a waiter if it may really sleep, keeping dp_lock off the
soft throttle path taken by every delayed transaction.
[5 lines not shown]
freebsd: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
linux: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
icp: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
lua: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
nvpair: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
zfs: remove _KERNEL gates
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
libspl: remove platform gates from platform headers
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
linux: remove platform gates from platform headers
Sponsored-by: TrueNAS
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Rob Norris <rob.norris at truenas.com>
Closes #19153
ZIO: Order a batch arrival against the hold it releases
zio_batch_arrive() publishes onto zb_arrived and then drops its hold;
whoever drops the last one walks that list with a plain load. Neither
side carries a barrier, so on arm64, where FreeBSD's atomics are
relaxed, a thread can see zb_holds reach zero and still read a stale
zb_arrived. The missed member never leaves ZIO_VDEV_IO_DONE, its parent
never completes, and the txg sync thread waits forever in zio_wait().
Linux and FreeBSD/x86 are unaffected; their atomics are ordered. A
Cortex-A76 litmus test observes the reordering in roughly 4% of the
cases where the reader wins the batch.
Reported by: Dave Cottlehuber <dch at FreeBSD.org>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Alexander Motin <alexander.motin at TrueNAS.com>
Signed-off-by: Nick Price <nprice at FreeBSD.org>
Closes #19152
build: Add -dirty release tag when building packages from a dirty git repo
By adding the "-dirty" tag to the release and by extension the full
version string, it becomes more clear that the build contains changes
not checked in to the repo. This could indicated changes that are
included by mistake.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19151
zpool: explain an error log that resolves to no files
An entry is only listed when the recorded block still carries the birth
transaction recorded against it, so a log whose blocks have since been
freed or rewritten prints a header promising files and then nothing at
all. The count shown without -v comes from spa_approx_errlog_size(),
which tallies table entries without resolving them, so the two forms
disagree and neither explains why.
Say that the entries resolved to no file. Reading the log is also what
removes such entries, so no advice to scrub is given: by the next read
they are gone.
Reviewed-by: Alek Pinchuk <Alek.Pinchuk at connectwise.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Nick Price <nprice at FreeBSD.org>
Closes #19108
Fix permanent errors misfiled into the scrub error log
dsl_scan_sync() sets spa_scrub_active before it checks whether a scan is
running, behind only a test that dsl_scan_active() is true. That test is
also satisfied by an async destroy, a non-empty free bpobj, or a pending
livelist delete, and nothing clears the flag except dsl_scan_done() or
dsl_errorscrub_done(), neither of which runs when no scan was started.
So after any dataset destroy the flag stays set for the life of the spa,
and spa_log_error() files every subsequent error in the scrub error log
rather than the last error log. An error scrub reads only the last error
log and so never sees them, and zpool clear does not touch the error log
at all. When a later scan ends it rotates the scrub error log into the
last error log and deletes only the log it replaces, laundering those
entries in as though that scan had found them, after which they survive
every subsequent scrub.
This is consistent with a long-standing class of reports, among them
survive repeated scrubs, and do not respond to zpool clear. The
[20 lines not shown]
CI: mask amx-tile in the QEMU build VMs
objtool sizes its alternate signal stack with SIGSTKSZ, which musl fixes
at 8192. A runner whose CPU exposes AMX needs 11952, so sigaltstack() is
refused, objtool exits before doing any work, make deletes the object,
and configure reports
checking whether CONFIG_MODULES is defined... no
on a kernel built with CONFIG_MODULES=y. Two separate samples of forty
Alpine jobs each landed four of them on such a host; all four failed,
every other job passed.
amx-tile is the only AMX feature carrying xsave state, so masking it
alone brings the frame back under SIGSTKSZ. host-model would also avoid
this but takes AVX-512 with it, which the suite exercises. Masking on
every runner keeps the diff to one line and is a no-op on the nine in
ten without AMX; ZFS has no AMX code. This can go once a fixed objtool
reaches the distribution images.
[3 lines not shown]
rpm: Pass --nodebuginfo to rpmbuild when debuginfo is disabled by configure
This fixes an issue where the system rpmbuild configuration is setup
to generate debuginfo rpms and will fail in make rpm-kmod if the needed
debuginfo source files are empty. Since by default debuginfo generation
is not enabled, this will fail reliably. Tell rpmbuild to not try to
build debuginfo rpms, when debuginfo generation is not enabled by
configure, by passing it the --nodebuginfo option.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19146
build: Fix AM_V_GEN usage in gitrev recipe
AM_V_GEN must be placed at the start of the recipe line because it gets
expanded to "@echo" which the shell doesn't know what to do with. The
@-prefix is interpreted by make, but only at the beginning of a recipe
line.
Fixes: 7adc908d3 (build: Add gitrev as a dependency to specific targets that require it)
Reviewed-by: Rob Norris <rob.norris at truenas.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Glenn Washburn <development at efficientek.com>
Closes #19145
