Displaying 1 50 of 5,229 commits (0.008s)

ZFS on Linux — include/sys zfs_vfsops.h, module/zfs zpl_super.c

Fix zpl_mount() deadlock

Commit 93b43af10 inadvertently introduced the following scenario which
can result in a deadlock.  This issue was most easily reproduced by
LXD containers using a ZFS storage backend but should be reproducible
under any workload which is frequently mounting and unmounting.

-- THREAD A --
spa_sync()
  spa_sync_upgrades()
    rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG); <- Waiting on B

-- THREAD B --
mount_fs()
  zpl_mount()
    zpl_mount_impl()
      dmu_objset_hold()
        dmu_objset_hold_flags()
          dsl_pool_hold()
            dsl_pool_config_enter()
              rrw_enter(&dp->dp_config_rwlock, RW_READER, tag);
    sget()
      sget_userns()
        grab_super()

    [22 lines not shown]

ZFS on Linux — include/spl/sys isa_defs.h, lib/libspl/include/sys isa_defs.h

Fix kernel unaligned access on sparc64

Update the SA_COPY_DATA macro to check if architecture supports
efficient unaligned memory accesses at compile time.  Otherwise
fallback to using the sa_copy_data() function.

The kernel provided CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is
used to determine availability in kernel space.  In user space
the x86_64, x86, powerpc, and sometimes arm architectures will
define the HAVE_EFFICIENT_UNALIGNED_ACCESS macro.

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7642
Closes #7684

ZFS on Linux — cmd/zdb zdb.c

Reduce zdb output when pool contains checkpoint

When running zdb without additional arguments against a pool containing
a checkpoint the entire checkpoint spacemap should not be dumped.  Make
this behavior conditional upon passing the -mmmm option as described in
the zdb(8) man page.

     -mmmm   Display every spacemap record.

Reviewed-by: Serapheim Dimitropoulos <serapheim.dimitro at delphix.com>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7702
Delta File
+4 -2 cmd/zdb/zdb.c
+4 -2 1 file

ZFS on Linux — include zfs_comutil.h, include/sys dbuf.h dmu_objset.h

OpenZFS 9337 - zfs get all is slow due to uncached metadata

This project's goal is to make read-heavy channel programs and zfs(1m)
administrative commands faster by caching all the metadata that they will
need in the dbuf layer. This will prevent the data from being evicted, so
that any future call to i.e. zfs get all won't have to go to disk (very
much). There are two parts:

The dbuf_metadata_cache. We identify what to put into the cache based on
the object type of each dbuf.  Caching objset properties os
{version,normalization,utf8only,casesensitivity} in the objset_t. The reason
these needed to be cached is that although they are queried frequently,
they aren't stored in a dbuf type which we can easily recognize and cache in
the dbuf layer; instead, we have to explicitly store them. There's already
existing infrastructure for maintaining cached properties in the objset
setup code, so I simply used that.

Performance Testing:

 - Disabled kmem_flags
 - Tuned dbuf_cache_max_bytes very low (128K)
 - Tuned zfs_arc_max very low (64M)

Created test pool with 400 filesystems, and 100 snapshots per filesystem.
Later on in testing, added 600 more filesystems (with no snapshots) to make

    [22 lines not shown]
Delta File
+211 -60 module/zfs/dbuf.c
+54 -54 module/zfs/dmu.c
+43 -4 module/zfs/zfs_vfsops.c
+27 -0 man/man5/zfs-module-parameters.5
+1 -18 module/zfs/zfs_ioctl.c
+3 -15 module/zfs/zcp_iter.c
+66 -13 11 files not shown
+405 -164 17 files

ZFS on Linux — tests/test-runner/bin zts-report.py, tests/zfs-tests/tests/functional/clean_mirror default.cfg cleanup.ksh

ZTS: clean_mirror and scrub_mirror cleanup

Remove the dependency on partitionable devices for the clean_mirror
and scrub_mirror test cases.  This allows for the setup and cleanup
of the test cases to be simplified by removing the need for complex
partitioning.

This change also resolves a issue where the clean_mirror devices
were not being properly damaged since the device name was not a
full path.  The result being loopX files were being left in the
top level test_results directory.

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7434
Closes #7690

ZFS on Linux — cmd/zpool zpool_vdev.c

Default ashift for Amazon EC2 NVMe devices

Add a default 4 KiB ashift for Amazon EC2 NVMe devices on instances with
NVMe ephemeral devices, such as the types c5d, f1, i3 and m5d.
As per the official documentation [1] a 4096 byte blocksize should be
used to match the underlying hardware.

The string was identified via:

$ sudo sginfo -M /dev/nvme0n1
INQUIRY response (cmd: 0x12)
----------------------------
Device Type                        0
Vendor:                    NVMe
Product:                   Amazon EC2 NVMe
Revision level:

$ lsblk -io KNAME,TYPE,SIZE,MODEL
KNAME   TYPE    SIZE MODEL
nvme0n1 disk  442.4G Amazon EC2 NVMe Instance Storage

[1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/
    storage-optimized-instances.html
    Retrived 2018-07-03


    [5 lines not shown]
Delta File
+1 -0 cmd/zpool/zpool_vdev.c
+1 -0 1 file

ZFS on Linux — . autogen.sh

Cause autogen.sh to fail if autoreconf fails

Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Paul Dagnelie <pcd at delphix.com>
Closes #7683
Delta File
+1 -1 autogen.sh
+1 -1 1 file

ZFS on Linux — module/zfs dbuf.c

Fix coverity defects: CID 176037

CID 176037: Uninitialized scalar variable

This patch fixes an uninitialized variable defect caught by
coverity and introduced in 69830602

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7667
Delta File
+4 -3 module/zfs/dbuf.c
+4 -3 1 file

ZFS on Linux — tests/test-runner/bin zts-report.py, tests/zfs-tests/tests/functional/no_space enospc_002_pos.ksh setup.ksh

ZTS: Improve enospc tests

The enospc_002_pos test case would frequently fail due a command
succeeding when it was expected to fail due to lack of space.
In order to make this far less likely, files are created across
multiple transaction groups in order to consume as many unused
blocks as possible.

The dependency that the tests run on a partitioned block device
has been removed.  It's simpler to use sparse files.

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7663

ZFS on Linux — module/zfs dmu_send.c dmu_object.c

Fix 'zfs recv' of non large_dnode send streams

Currently, there is a bug where older send streams without the
DMU_BACKUP_FEATURE_LARGE_DNODE flag are not handled correctly.
The code in receive_object() fails to handle cases where
drro->drr_dn_slots is set to 0, which is always the case when the
sending code does not support this feature flag. This patch fixes
the issue by ensuring that that a value of 0 is treated as
DNODE_MIN_SLOTS.

Tested-by:  DHE <git at dehacked.net>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7617
Closes #7662
Delta File
+10 -8 module/zfs/dmu_send.c
+3 -0 module/zfs/dmu_object.c
+13 -8 2 files

ZFS on Linux — module/zfs dbuf.c arc.c

Raw receive fix and encrypted objset security fix

This patch fixes two problems with the encryption code. First, the
current code does not correctly prohibit the DMU from updating
dn_maxblkid during object truncation within a raw receive. This
usually only causes issues when the truncating DRR_FREE record is
aggregated with DRR_FREE records later in the receive, so it is
relatively hard to hit.

Second, this patch fixes a security issue where reading blocks
within an encrypted object did not guarantee that the dnode block
itself had ever been verified against its MAC. Usually the
verification happened anyway when the bonus buffer was read, but
some use cases (notably zvols) might never perform the check.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7632
Delta File
+103 -34 module/zfs/dbuf.c
+7 -3 module/zfs/arc.c
+6 -1 module/zfs/dnode_sync.c
+116 -38 3 files

ZFS on Linux — man/man5 zpool-features.5

Fix formatting in zpool-features(5)

The formatting of the features beginning with large_blocks was broken
when the zpool_checkpoint feature was added.

Signed-off-by: Tim Chase <tim at chase2k.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7658
Delta File
+5 -0 man/man5/zpool-features.5
+5 -0 1 file

ZFS on Linux — tests/runfiles perf-regression.run, tests/zfs-tests/tests/perf/fio random_readwrite_fixed.fio Makefile.am

OpenZFS 9184 - Add ZFS performance test for fixed blocksize random read/write IO

This change introduces a new performance test which does random reads
and writes, but instead of using `bssplit` to determine the block size,
it uses a fixed blocksize. Additionally, some new IO sizes are added to
other tests and timestamp data is recorded with the performance data.

Authored by: Ahmed Gahnem <ahmedg at delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel at delphix.com>
Reviewed by: John Kennedy <john.kennedy at delphix.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Nguyen <tony.nguyen at delphix.com>
Ported-by: John Kennedy <john.kennedy at delphix.com>
Signed-off-by: John Wren Kennedy <john.kennedy at delphix.com>
Requires-builders: perf

OpenZFS-issue: https://www.illumos.org/issues/9184
OpenZFS-commit: https://github.com/openzfs/openzfs/pull/659
External-issue: DLPX-46724
Closes #7660

ZFS on Linux — cmd/zpool zpool_main.c

Fix missing option '-e' in zpool online usage

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: WHR <msl0000023508 at gmail.com>
Closes #7655
Delta File
+1 -1 cmd/zpool/zpool_main.c
+1 -1 1 file

ZFS on Linux — man/man5 zfs-module-parameters.5

Fix duplicate "fB" typo

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: ajs124 <git at ajs124.de>
Closes #7649
Delta File
+1 -1 man/man5/zfs-module-parameters.5
+1 -1 1 file

ZFS on Linux — module/zfs spa.c

Remove suffix from zio taskq names

For zio taskq's which have multiple instances (e.g. z_rd_int_0,
z_rd_int_1, etc), each one has a unique name (the _0, _1, _2 suffix).
This makes performance analysis more difficult, because by default,
`perf` includes the thread name (which is the same as the taskq name) in
the stack trace.  This means that we get 8 different stacks, all of
which are doing the same thing, but are executed from different taskq's.

We should remove the suffix of the taskq name, so that all the
read-interrupt threads are named z_rd_int.

Note that we already support multiple taskq's with the same name.  This
happens when there are multiple pools.  In this case the taskq has a
different tq_instance, which shows up in /proc/spl/taskq-all.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Richard Elling <Richard.Elling at RichardElling.com>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed by: Pavel Zakharov <pavel.zakharov at delphix.com>
Signed-off-by: Matthew Ahrens <mahrens at delphix.com>
Closes #7646
Delta File
+3 -8 module/zfs/spa.c
+3 -8 1 file

ZFS on Linux — . configure.ac, scripts zfs-tests.sh common.sh.in

ZTS: Adopt OpenZFS test analysis script

Adopt and extend the OpenZFS ZTS results analysis script for use
with ZFS on Linux.  This allows for automatic analysis of tests
which may be skipped for a variety or reasons or which are not
entirely reliable.

In addition to the list of 'known' failures, which have been updated
for ZFS on Linux, there in a new 'maybe' section.  This mapping
include tests which might be correctly skipped depending on the
test environment.  This may be because of a missing dependency or
lack of required kernel support.  This list also includes tests
which normally pass but might on occasion fail for a harmless
reason.

The script was also extended include a reason for why a given test
might be skipped or may fail.  The reason will be included after
the test in the "results other than PASS that are expected" section.
For failures it is preferable to set the reason to the GitHub issue
number and for skipped tests several generic reasons are available.
You may also specify a custom reason if needed.

All tests were added back in to the linux.run file even if they are
expected to failed.  There is value in running tests which may not

    [16 lines not shown]

ZFS on Linux — config kernel-elevator-change.m4, include/linux blkdev_compat.h

Linux 4.14 compat: blk_queue_stackable()

The blk_queue_stackable() function was replaced in the 4.14 kernel
by queue_is_rq_based(), commit torvalds/linux at 5fdee212.  This change
resulted in the default elevator being used which can negatively
impact performance.

Rather than adding additional compatibility code to detect the
new interface unconditionally attempt to set the elevator.  Since
we expect this to fail for block devices without an elevator the
error message has been moved in to zfs_dbgmsg().

Finally, it was observed that the elevator_change() was removed
from the 4.12 kernel, commit torvalds/linux at c033269.  Update the
comment to clearly specify which are expected to export the
elevator_change() symbol.

Reviewed-by: Matthew Ahrens <mahrens at delphix.com>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7645

ZFS on Linux — config kernel-inode-times.m4 kernel-current-time.m4, include/spl/sys time.h vnode.h

Linux 4.18 compat: inode timespec -> timespec64

Commit torvalds/linux at 95582b0 changes the inode i_atime, i_mtime,
and i_ctime members form timespec's to timespec64's to make them
2038 safe.  As part of this change the current_time() function was
also updated to return the timespec64 type.

Resolve this issue by introducing a new inode_timespec_t type which
is defined to match the timespec type used by the inode.  It should
be used when working with inode timestamps to ensure matching types.

The timestruc_t type under Illumos was used in a similar fashion but
was specified to always be a timespec_t.  Rather than incorrectly
define this type all timespec_t types have been replaced by the new
inode_timespec_t type.

Finally, the kernel and user space 'sys/time.h' headers were aligned
with each other.  They define as appropriate for the context several
constants as macros and include static inline implementation of
gethrestime(), gethrestime_sec(), and gethrtime().

Reviewed-by: Chunwei Chen <tuxoko at gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7643

ZFS on Linux — cmd/zed zed_disk_event.c

Fix typo in comment, handeling->handling

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: Kash Pande <kash at tripleback.net>
Closes #7641
Delta File
+1 -1 cmd/zed/zed_disk_event.c
+1 -1 1 file

ZFS on Linux — include/sys dmu.h, module/zfs dbuf.c dsl_dataset.c

Add ASSERT to debug encryption key mapping issues

This patch simply adds an ASSERT that confirms that the last
decrypting reference on a dataset waits until the dataset is
no longer dirty. This should help to debug issues where the
ZIO layer cannot find encryption keys after a dataset has been
disowned.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7637

ZFS on Linux — tests/zfs-tests/tests/functional/history history_002_pos.ksh

Use "eval" in history_002_pos for log_must

Otherwise the output is consumed by the output redirection.

Reviewed-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tim Chase <tim at chase2k.com>
Closes #7570

ZFS on Linux — . copy-builtin

copy-builtin: SPL must be in Kbuild first

The recent SPL merge caused a regression in kernels with ZFS integrated
into the sources where our modules would be initialized in alphabetical
order, despite icp requiring the spl module be loaded first. This caused
kernels with ZFS builtin to fail to boot.

We resolve this by adding a special case for the spl that lists it
first. It is somewhat ugly, but it works.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Thode <prometheanfire at gentoo.org>
Signed-off-by: Richard Yao <ryao at gentoo.org>
Closes #7595
Closes #7606
Delta File
+2 -0 copy-builtin
+2 -0 1 file

ZFS on Linux — cmd/zfs zfs_main.c, include/sys zcp.h

Add tunables for channel programs

This patch adds tunables for modifying the maximum memory limit and
maximum instruction limit that can be specified when running a channel
program.

Reviewed-by: Matthew Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov
Reviewed-by: Sara Hartse <sara.hartse at delphix.com>
Signed-off-by: John Gallagher <john.gallagher at delphix.com>
External-issue: LX-1085
Closes #7618

ZFS on Linux — . META, config kernel-block-device-operations.m4 kernel-bdev-block-device-operations.m4

Linux compat 4.18: check_disk_size_change()

Added support for the bops->check_events() interface which was
added in the 2.6.38 kernel to replace bops->media_changed().
Fully implementing this functionality allows the volume resize
code to rely on revalidate_disk(), which is the preferred
mechanism, and removes the need to use check_disk_size_change().

In order for bops->check_events() to lookup the zvol_state_t
stored in the disk->private_data the zvol_state_lock needs to
be held.  Since the check events interface may poll the mutex
has been converted to a rwlock for better concurrently.  The
rwlock need only be taken as a writer in the zvol_free() path
when disk->private_data is set to NULL.

The configure checks for the block_device_operations structure
were consolidated in a single kernel-block-device-operations.m4
file.

The ZFS_AC_KERNEL_BDEV_BLOCK_DEVICE_OPERATIONS configure checks
and assoicated dead code was removed.  This interface was added
to the 2.6.28 kernel which predates the oldest supported 2.6.32
kernel and will therefore always be available.


    [8 lines not shown]

ZFS on Linux — include/sys zfs_ioctl.h

Reserve DMU_BACKUP_FEATURE for ZSTD

Reserve bit 25 for the ZSTD compression feature from FreeBSD.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Allan Jude <allanjude at freebsd.org>
Closes #7626
Delta File
+1 -0 include/sys/zfs_ioctl.h
+1 -0 1 file

ZFS on Linux — lib/libefi rdwr_efi.c

Remove libefi __linux__ wrappers

The ZoL version of libefi has been modified for Linux in several
places outside the existing __linux__ wrappers.  Remove them to
make the code easier to read and so as not to mislead anyone that
these are the sole modifications for Linux.

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7625
Delta File
+2 -17 lib/libefi/rdwr_efi.c
+2 -17 1 file

ZFS on Linux — cmd/ztest ztest.c

Fix ztest_vdev_add_remove() test case

Commit 2ffd89fc allowed two new errors to be reported by zil_reset()
in order to provide a descriptive error message regarding why a log
device could not be removed.  However, the new return values were
not handled in the ztest_vdev_add_remove() test case resulting in
ztest failures during automated testing.

Reviewed-by: Tim Chase <tim at chase2k.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Paul Zuchowski <pzuchowski at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7630
Delta File
+8 -1 cmd/ztest/ztest.c
+8 -1 1 file

ZFS on Linux — lib/libefi rdwr_efi.c

Fix efi_get_info() zvol detection

Partition detection for zvol devices was not working correctly
resulting inconsistent partitioning behavior when layering pools
on top of zvols.  This isn't a supported configuration but we'd
still like it to work properly.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7624
Delta File
+3 -2 lib/libefi/rdwr_efi.c
+3 -2 1 file

ZFS on Linux — module/zfs spa_config.c vdev.c, tests/runfiles linux.run

OpenZFS 9591 - ms_shift can be incorrectly changed

ms_shift can be incorrectly changed changed in MOS config for
indirect vdevs that have been historically expanded

According to spa_config_update() we expect new vdevs to have
vdev_ms_array equal to 0 and then we go ahead and set their metaslab
size. The problem is that indirect vdevs also have vdev_ms_array == 0
because their metaslabs are destroyed once their removal is done.

As a result, if a vdev was expanded and then removed may have its
ms_shift changed if another vdev was added after its removal.
Fortunately this behavior does not cause any type of crash or bad
behavior in the kernel but it can confuse zdb and anyone doing any kind
of analysis of the history of the pools.

Authored by: Serapheim Dimitropoulos <serapheim at delphix.com>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Reviewed by: George Wilson <gwilson at zfsmail.com>
Reviewed by: John Kennedy <john.kennedy at delphix.com>
Reviewed by: Prashanth Sreenivasa <pks at delphix.com>
Reviewed by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tim Chase <tim at chase2k.com>
Ported-by: Tim Chase <tim at chase2k.com>


    [4 lines not shown]

ZFS on Linux — tests/zfs-tests/tests/functional/privilege cleanup.ksh

ZTS: deletes home directories in /export/home

In the cleanup for the privilege tests, an empty variable, empty because
the corresponding setup is skipped on Linux, results in /export/home
being deleted. This patch adds an assertion that the variable is not
empty, and causes the cleanup to be skipped on Linux as well.

Reviewed by: John Wren Kennedy <jwk404 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Paul Dagnelie <pcd at delphix.com>
Signed-off-by: John Gallagher <john.gallagher at delphix.com>
External-issue: LX-1099
Closes #7615

ZFS on Linux — tests/zfs-tests/include libtest.shlib

ZTS: cleanup user_run

user_run leaves two files in /tmp, moving them to $TEST_BASE_DIR and
adding them to the default cleanup routine.

Reviewed by: John Wren Kennedy <jwk404 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Signed-off-by: bunder2015 <omfgbunder at gmail.com>
Closes #7614

ZFS on Linux — contrib/pyzfs .gitignore Makefile.am

Add pyzfs build directories to gitignore

The recent addition of pyzfs does not include the generated 'build'
and 'pyzfs.egg-info' directories in the pyzfs .gitignore or the
'make clean' target. This patch simply corrects this problem.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7612

ZFS on Linux — lib/libzfs libzfs_pool.c, module/zfs zil.c

Wrong error message when removing log device

In the case where the pool is loaded without the crypto
keys necessary to playback the intent log, and log device
removal is attempted, a generic busy message is received.
Change the message to inform the user that the datasets
must be mounted.

Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tom Caputi <tcaputi at datto.com>
Signed-off-by: Paul Zuchowski <pzuchowski at datto.com>
Closes #7518
Delta File
+18 -3 lib/libzfs/libzfs_pool.c
+5 -1 module/zfs/zil.c
+23 -4 2 files

ZFS on Linux — module/zfs aggsum.c

Fix preemptible warning in aggsum_add()

In the new aggsum counters the CPU_SEQID macro should be surrounded by
kpreempt_disable)() and kpreempt_enable() calls to prevent a Linux
kernel BUG warning.  The addsum_add() function use the cpuid to
minimize lock contention when selecting a bucket, after selection
the bucket is protected by a mutex and it is safe to reschedule the
process to a different processor at any time.

Reviewed-by: Matthew Thode <prometheanfire at gentoo.org>
Reviewed-by: Paul Dagnelie <pcd at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7609
Closes #7610
Delta File
+5 -2 module/zfs/aggsum.c
+5 -2 1 file

ZFS on Linux — . configure.ac, cmd/zed Makefile.am

Tunable directory for zfs runtime scripts

zpool and zed place scripts in subdirectories of libexecdir. Some
distributions locate architecture independent scripts in other locations
(e.g. Debian). To avoid these paths getting out of sync, centralize the
definitions.

Build zfs-test's default.cfg by Makefile.  Use the new directory
logic building tests/zfs-tests/include/default.cfg.in.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo at gmail.com>
Closes #7597

ZFS on Linux — module/zfs sa.c

Don't panic on bad SA_MAGIC in sa_build_index

If sa_build_index() encounters a corrupt buffer, don't panic.
Add info to zfs ring buffer and return EIO.  This allows for a cleaner
error recovery path.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Nathaniel Clark <nathaniel.l.clark at intel.com>
Issue #6500
Closes #7487
Delta File
+7 -1 module/zfs/sa.c
+7 -1 1 file

ZFS on Linux — etc/systemd/system-generators zfs-mount-generator.in, man/man8 zfs.8

Minor documentation, logging, and testing typos

This patch collects some minor inconsistencies and typos in the
documentation, logging and testing infrastructure.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo at gmail.com>
Closes #7608

ZFS on Linux — module/zfs arc.c

Update the correct abd in l2arc_read_done()

This patch fixes an issue where l2arc_read_done() would always
write data to b_pabd, even if raw encrypted data was requested.
This only occured in cases where the L2ARC device had a different
ashift than the main pool.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7586
Closes #7593
Delta File
+12 -9 module/zfs/arc.c
+12 -9 1 file

ZFS on Linux — include/sys dmu.h, module/zfs dmu.c arc.c

Raw receive functions must not decrypt data

This patch fixes a small bug found where receive_spill() sometimes
attempted to decrypt spill blocks when doing a raw receive. In
addition, this patch fixes another small issue in arc_buf_fill()'s
error handling where a decryption failure (which could be caused by
the first bug) would attempt to set the arc header's IO_ERROR flag
without holding the header's lock.

Reviewed-by: Matthew Thode <prometheanfire at gentoo.org>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #7564
Closes #7584
Closes #7592

ZFS on Linux — cmd/zfs zfs_main.c, tests/zfs-tests/tests/functional/cli_root/zfs_destroy zfs_destroy_005_neg.ksh

Always continue recursive destroy after error

Currently, during a recursive zfs destroy the first error that is
encountered will stop the destruction of the datasets. Errors may
happen for a variety of reasons including competing deletions
and busy datasets.
This patch switches recursive destroy to always do a best-effort
recursive dataset destroy.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Matthew Ahrens <mahrens at delphix.com>
Signed-off-by: Alek Pinchuk <apinchuk at datto.com>
Closes #7574

ZFS on Linux — tests/zfs-tests/tests/functional/history history.cfg cleanup.ksh

ZTS: history path cleanup

History tests were hard coded to use /tmp and didn't clean up
properly after testing.

Reviewed by: John Wren Kennedy <jwk404 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: bunder2015 <omfgbunder at gmail.com>
Issue #7507
Closes #7600

ZFS on Linux — . configure.ac, cmd/zpool zpool_main.c

Add pool state /proc entry, "SUSPENDED" pools

1. Add a proc entry to display the pool's state:

$ cat /proc/spl/kstat/zfs/tank/state
ONLINE

This is done without using the spa config locks, so it will
never hang.

2. Fix 'zpool status' and 'zpool list -o health' output to print
"SUSPENDED" instead of "ONLINE" for suspended pools.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Richard Elling <Richard.Elling at RichardElling.com>
Signed-off-by: Tony Hutter <hutter2 at llnl.gov>
Closes #7331
Closes #7563

ZFS on Linux — cmd/ztest ztest.c, include/sys zfs_context.h

Remove rwlock wrappers

The only remaining consumer of the rwlock compatibility wrappers
is ztest.  Remove the wrappers and convert the few remaining
calls to the underlying pthread functions.

    rwlock_init()    -> pthread_rwlock_init()
    rwlock_destroy() -> pthread_rwlock_destroy()
    rw_rdlock()      -> pthread_rwlock_rdlock()
    rw_wrlock()      -> pthread_rwlock_wrlock()
    rw_unlock()      -> pthread_rwlock_unlock()

Note pthread_rwlock_init() defaults to PTHREAD_PROCESS_PRIVATE
which is equivilant to the USYNC_THREAD behavior.  There is no
functional change.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #7591

ZFS on Linux — man/man8 zpool.8

Fix typoes in zpool man page

Fixed some highlighting in the zpool man page

Reviewed-by: Richard Laager <rlaager at wiktel.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: bunder2015 <omfgbunder at gmail.com>
Closes #7596
Delta File
+9 -4 man/man8/zpool.8
+9 -4 1 file

ZFS on Linux — include/sys vdev_disk.h, lib/libefi rdwr_efi.c

zpool reopen should detect expanded devices

Update bdev_capacity to have wholedisk vdevs query the
size of the underlying block device (correcting for the size
of the efi parition and partition alignment) and therefore detect
expanded space.

Correct vdev_get_stats_ex so that the expandsize is aligned
to metaslab size and new space is only reported if it is large
enough for a new metaslab.

Reviewed by: Don Brady <don.brady at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: George Wilson <george.wilson at delphix.com>
Reviewed-by: Matthew Ahrens <mahrens at delphix.com>
Reviewed by: John Wren Kennedy <jwk404 at gmail.com>
Signed-off-by: sara hartse <sara.hartse at delphix.com>
External-issue: LX-165
Closes #7546
Issue #7582

ZFS on Linux — include/sys dbuf.h dnode.h, module/zfs dbuf.c dnode_sync.c

OpenZFS 9577 - remove zfs_dbuf_evict_key tsd

The zfs_dbuf_evict_key TSD (thread-specific data) is not necessary -
we can instead pass a flag down in a few places to prevent recursive
dbuf eviction. Making this change has 3 benefits:

1. The code semantics are easier to understand.
2. On Linux, performance is improved, because creating/removing
   TSD values (by setting to NULL vs non-NULL) is expensive, and
   we do it very often.
3. According to Nexenta, the current semantics can cause a
   deadlock when concurrently calling dmu_objset_evict_dbufs()
   (which is rare today, but they are working on a "parallel
   unmount" change that triggers this more easily):

Porting Notes:
* Minor conflict with OpenZFS 9337 which has not yet been ported.

Authored by: Matthew Ahrens <mahrens at delphix.com>
Reviewed by: George Wilson <george.wilson at delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro at delphix.com>
Reviewed by: Brad Lewis <brad.lewis at delphix.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Ported-by: Brian Behlendorf <behlendorf1 at llnl.gov>


    [4 lines not shown]

ZFS on Linux — . Makefile.am

make install only works once

`make install` shouldn't fail if a directory it created still exists.
In this case we can blow away the spl src directory before recreating
it.  This also gracefully handles the migration from pre-spl-merge to
post-spl-merge.

Reviewed by: Richard Elling <Richard.Elling at RichardElling.com>
Reviewed-by: Pavel Zakharov <pavel.zakharov at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Matthew Ahrens <mahrens at delphix.com>
Closes #7580
Delta File
+3 -2 Makefile.am
+3 -2 1 file

ZFS on Linux — . META configure.ac

Explicitly state supported Linux versions

Add META tags Linux-Maximum and Linux-Minimum.

One pain point for package maintainers is ensuring the compatibility of
the packaged version of ZFS with the Linux kernel. By providing an
authoritative compatibility guide in the source tree, maintainers can
automate compatibility checks.

Additionally, increase META string extraction specificity.
configure.ac finds Name and Version by a very simple `grep`, which might
conceivably find other fields. Require the string be at the beginning of
a line, and be followed by a colon to avoid such confusions.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Antonio Russo <antonio.e.russo at gmail.com>
Closes #7571
Delta File
+10 -8 META
+2 -2 configure.ac
+3 -0 README.md
+15 -10 3 files

ZFS on Linux — module/zfs zap_micro.c dmu_object.c

OpenZFS 9329 - panic in zap_leaf_lookup() due to concurrent zapification

For the null pointer issue shown below, the solution is to initialize the
contents of the object before changing its type, so that concurrent accessors
will see it as non-zapified until it is ready for access via the ZAP.

    BAD TRAP: type=e (#pf Page fault) rp=ffffff00ff520440 addr=20 occurred
    in module "zfs" due to a NULL pointer dereference

    ffffff00ff520320 unix:die+df ()
    ffffff00ff520430 unix:trap+dc0 ()
    ffffff00ff520440 unix:cmntrap+e6 ()
    ffffff00ff520590 zfs:zap_leaf_lookup+46 ()
    ffffff00ff520640 zfs:fzap_lookup+a9 ()
    ffffff00ff5206e0 zfs:zap_lookup_norm+111 ()
    ffffff00ff520730 zfs:zap_contains+42 ()
    ffffff00ff520760 zfs:dsl_dataset_has_resume_receive_state+47 ()
    ffffff00ff520900 zfs:get_receive_resume_stats+3e ()
    ffffff00ff520a90 zfs:dsl_dataset_stats+262 ()
    ffffff00ff520ac0 zfs:dmu_objset_stats+2b ()
    ffffff00ff520b10 zfs:zfs_ioc_objset_stats_impl+64 ()
    ffffff00ff520b60 zfs:zfs_ioc_objset_stats+33 ()
    ffffff00ff520bd0 zfs:zfs_ioc_dataset_list_next+140 ()
    ffffff00ff520c80 zfs:zfsdev_ioctl+4d7 ()
    ffffff00ff520cc0 genunix:cdev_ioctl+39 ()

    [18 lines not shown]
Delta File
+27 -12 module/zfs/zap_micro.c
+9 -2 module/zfs/dmu_object.c
+36 -14 2 files