ZFS on Linux/src 0a10863module/zfs dsl_pool.c

ztest: creates partially initialized root dataset

Since d8fdfc2 was integrated dsl_pool_create() does not call
dmu_objset_create_impl() for the root dataset when running in
userland (ztest): this creates a pool with a partially initialized
root dataset. Trying to import and use this pool results in both
zpool and zfs executables dumping core.

Fix this by adopting an alternative change suggested in OpenZFS 8607
code review.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed by: Tom Caputi <tcaputi at datto.com>
Original-patch-by: Robert Mustacchi <rm at joyent.com>
Signed-off-by: loli10K <ezomori.nozomu at gmail.com>
Closes #8277 
DeltaFile
+10-8module/zfs/dsl_pool.c
+10-81 files

ZFS on Linux/src ad63507module/zfs zfs_vfsops.c

Remove zfs_sync() panicking kernel check

This check provides no real additional protection and unnecessarily
introduces a dependency on the "oops_in_progress" kernel symbol.
Remove the check, it there are special circumstances on other
platforms which make this a requirement it can be reintroduced
for all relevant call paths in a more portable comprehensive manor.

Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8297 

ZFS on Linux/src b194fabcmd/zdb zdb.c, include/sys metaslab_impl.h metaslab.h

Factor metaslab_load_wait() in metaslab_load()

Most callers that need to operate on a loaded metaslab, always
call metaslab_load_wait() before loading the metaslab just in
case someone else is already doing the work.

Factoring metaslab_load_wait() within metaslab_load() makes the
later more robust, as callers won't have to do the load-wait
check explicitly every time they need to load a metaslab.

Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8290 

ZFS on Linux/src 960347dmodule/zfs dmu_recv.c

Fix 0 byte memory leak in zfs receive

Currently, when a DRR_OBJECT record is read into memory in
receive_read_record(), memory is allocated for the bonus buffer.
However, if the object doesn't have a bonus buffer the code will
still "allocate" the zero bytes, but the memory will not be passed
to the processing thread for cleanup later. This causes the spl
kmem tracking code to report a leak. This patch simply changes the
code so that it only allocates this memory if it has a non-zero
length.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8266 

ZFS on Linux/src 1a75920include/sys zfs_debug.h

Document guidelines for usage of zfs_dbgmsg

Reviewed-by: Richard Elling <Richard.Elling at RichardElling.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Matt Ahrens <matt at delphix.com>
Reviewed-by: Igor Kozhukhov <igor at dilos.org>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8299 

ZFS on Linux/src e45c173scripts dkms.mkconf

dkms: Enable debuginfo option to be set with zfs sysconfig file

On some Linux distributions, the kernel module build will not
default to building with debuginfo symbols, which can make it
difficult for debugging and testing.

For this case, we provide a flag to override the build to force
debuginfo to be produced for the kernel module build.

Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Co-authored-by: Neal Gompa <ngompa at datto.com>
Co-authored-by: Simon Watson <swatson at datto.com>
Signed-off-by: Neal Gompa <ngompa at datto.com>
Signed-off-by: Simon Watson <swatson at datto.com>
Closes #8304 

ZFS on Linux/src 60b0a96module/zfs zap_leaf.c, tests/zfs-tests/tests/functional/cli_root/zfs_set user_property_003_neg.ksh user_property_001_pos.ksh

Off-by-one in zap_leaf_array_create()

Trying to set user properties with their length 1 byte shorter than the
maximum size triggers an assertion failure in zap_leaf_array_create():

  panic[cpu0]/thread=ffffff000a092c40:
  assertion failed: num_integers * integer_size < (8<<10) (0x2000 < 0x2000), file: 
../../common/fs/zfs/zap_leaf.c, line: 233

  ffffff000a092500 genunix:process_type+167c35 ()
  ffffff000a0925a0 zfs:zap_leaf_array_create+1d2 ()
  ffffff000a092650 zfs:zap_entry_create+1be ()
  ffffff000a092720 zfs:fzap_update+ed ()
  ffffff000a0927d0 zfs:zap_update+1a5 ()
  ffffff000a0928d0 zfs:dsl_prop_set_sync_impl+5c6 ()
  ffffff000a092970 zfs:dsl_props_set_sync_impl+fc ()
  ffffff000a0929b0 zfs:dsl_props_set_sync+79 ()
  ffffff000a0929f0 zfs:dsl_sync_task_sync+10a ()
  ffffff000a092a80 zfs:dsl_pool_sync+3a3 ()
  ffffff000a092b50 zfs:spa_sync+4e6 ()
  ffffff000a092c20 zfs:txg_sync_thread+297 ()
  ffffff000a092c30 unix:thread_start+8 ()

This patch simply corrects the assertion.


    [4 lines not shown]

ZFS on Linux/src 8dc2197module/zfs spa.c vdev_removal.c

Simplify spa_sync by breaking it up to smaller functions

The point of this refactoring is to break the high-level conceptual 
steps of spa_sync() to their own helper functions. In general large 
functions can enhance readability if structured well, but in this
case the amount of conceptual steps taken could use the help of 
helper functions.

Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8293 

ZFS on Linux/src ce5fb2acmd/ztest ztest.c

ztest: scrub verification

By design ztest will never inject non-repairable damage in to the
pool.  Update the ztest_scrub() test case such that it waits for
the scrub to complete and verifies the pool is always repairable.

After enabling scrub verification two scenarios were encountered
which are the result of how ztest manages failure injection.

The first case is straight forward and pertains to detaching a
mirror vdev.  In this case, the pool must always be scrubbed prior
the detach.  Failure to do so can potentially lock in previously
repairable data corruption by removing all good copies of a block
leaving only damaged ones.

The second is a little more subtle.  The child/offset selection
logic in ztest_fault_inject() depends on the calculated number of
leaves always remaining constant between injection passes.  This
is true within a single execution of ztest, but when using zloop.sh
random values are selected for each restart.  Therefore, when ztest
imports an existing pool it must be scrubbed before failure injection
can be safely enabled.  Otherwise it is possible that it will inject
non-repairable damage.

Reviewed by: Matt Ahrens <mahrens at delphix.com>

    [3 lines not shown]
DeltaFile
+63-6cmd/ztest/ztest.c
+63-61 files

ZFS on Linux/src 305781dcmd/ztest ztest.c, include/sys dmu.h

Fix error handling incallers of dbuf_hold_level()

Currently, the functions dbuf_prefetch_indirect_done() and
dmu_assign_arcbuf_by_dnode() assume that dbuf_hold_level() cannot
fail. In the event of an error the former will cause a NULL pointer
dereference and the later will trigger a VERIFY. This patch adds
error handling to these functions and their callers where necessary.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8291 

ZFS on Linux/src 75058f3include/sys vdev_impl.h, module/zfs vdev.c

Remove unused vdev_t fields

The following fields from the vdev_t struct are not used anywhere.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8285 

ZFS on Linux/src 52b6842cmd/ztest ztest.c, module/zfs ddt.c

ztest: scrub ddt repair

The ztest_ddt_repair() test is designed inflict damage to the
ddt which can be repairable by a scrub.  Unfortunately, this
repair logic was broken at some point and it went undetected.
This issue is not specific to ztest, but thankfully this extra
redundancy is rarely enabled and even more rarely needed.

The root cause was identified to be the ddt_bp_create()
function called by dsl_scan_ddt_entry() which did not set the
dedup bit of the generated block pointer.

The consequence of this was that the ZIO_DDT_READ_PIPELINE was
never enabled for the block pointer during the scrub, and the
dedup ditto repair logic was never run.  Note that for demand
reads which don't rely on ddt_bp_create() the required pipeline
stages would be enabled and the repair performed.

This was resolved by unconditionally setting the dedup bit in
ddt_bp_create().  This way all codes paths which may need to
perform a repair from a block pointer generated from the dtt
entry will be able too.  The only exception is that the dedup
bit is cleared in ddt_phys_free() which is required to avoid
leaking space.


    [5 lines not shown]

ZFS on Linux/src 419ba59module/zfs vdev.c, tests/zfs-tests/tests/functional/pool_checkpoint checkpoint_sm_scale.ksh

Update vdev_is_spacemap_addressable() for new spacemap encoding

Since the new spacemap encoding was ported to ZoL that's no longer 
a limitation. This patch updates vdev_is_spacemap_addressable() 
that was performing that check.

It also updates the appropriate test to ensure that the same 
functionality is tested.  The test does so by creating pools that 
don't have the new spacemap encoding enabled - just the checkpoint
feature. This patch also reorganizes that same tests in order to 
cut in half its memory consumption.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8286 

ZFS on Linux/src 64bdf63cmd/zdb zdb.c, cmd/ztest ztest.c

ztest: split block reconstruction

Increase the default allowed number of reconstruction attempts.
There's not an exact right number for this setting.  It needs
to be set large enough to cover any realistic failure scenarios
and small enough to avoid stalling the IO pipeline and invoking
the dead man detection.

The current value of 256 was empirically determined to be too
low based on multi-day runs of ztest.  The fault injection code
would inject more damage than could be reconstructed given the
relatively small number of attempts.  However, in all observed
cases the block could be reconstructed using a slightly higher
limit.

Based on local testing increasing the default value to 4096 was
determined to strike the best balance.  Checking all combinations
takes less than 10s in the worst case, and has so far eliminated
the vast majority of false positives detected by ztest.  This
delay is roughly on par with how long retries may be performed
to a misbehaving HDD and was deemed to be reasonable.  Better to
err on the side of a brief delay rather than fail to reconstruct
the data.

Lastly, the -Y flag has been added to zdb to make it easy to try all

    [10 lines not shown]

ZFS on Linux/src db58794tests/zfs-tests/tests/functional/pool_checkpoint pool_checkpoint.kshlib

Make zdb results for checkpoint tests consistent

This patch exports and re-imports the pool when these tests are
analyzed with zdb to get consistent results.

Reviewed by: Igor Kozhukhov <igor at dilos.org>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: George Melikov <mail at gmelikov.ru>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8292 

ZFS on Linux/src 6e91a72cmd/zfs zfs_main.c, cmd/ztest ztest.c

Disable 'zfs remap' command

The implementation of 'zfs remap' has proven to be problematic since
it modifies the objset (but not its logical contents) by dirtying
metadata without owning it.  The consequence of which is that
dmu_objset_remap_indirects() is vulnerable to certain races.

For example, if we are in the middle of receiving into the filesystem
while it is being remapped.  Then it is possible we could evict the
objset when the receive completes (see dsl_dataset_clone_swap_sync_impl,
or dmu_recv_end_sync), but dmu_objset_remap_indirects() may be still
using the objset.  The result of which would be a panic.

Extended runs of ztest(8) have exposed other possible races which
can occur when using 'zfs remap'.  Several of these have been fixed
but there may be others which have not yet been encountered and
diagnosed.

Furthermore, the ability to manually remap a filesystem is no longer
particularly useful now that the removal code can map large chunks.
Coupled with the fact that explaining what this command does and why
it may be useful requires a detailed understanding of the internals
of device removal.  These are details users should not be bothered
with.


    [13 lines not shown]

ZFS on Linux/src 5e7f3acmodule/zfs dbuf.c

Fix zio leak in dbuf_read()

Currently, dbuf_read() may decide to create a zio_root which is
used as a parent for any child zios created in dbuf_read_impl().
However, if there is an error in dbuf_read_impl(), this zio is
never executed and ends up leaked. This patch simply ensures
that we always execute the root zio, even i it has no real work
to do.

Reviewed-by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8267 
DeltaFile
+11-2module/zfs/dbuf.c
+11-21 files

ZFS on Linux/src 7b02fae. Makefile.am .gitignore

Verify .gitignore entries

This change adds a make target 'vcscheck' which scans the git workspace
for new, untracked files missing from the .gitignore configuration; this
is done to help prevent adding unwanted build artifacts to the source
tree during development.

Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: Giuseppe Di Natale <dinatale2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu at gmail.com>
Closes #8281 
DeltaFile
+7-1Makefile.am
+1-0.gitignore
+8-12 files

ZFS on Linux/src 9b626c1. META

Tag 0.8.0-rc3

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
DeltaFile
+2-2META
+2-21 files

ZFS on Linux/src d611989cmd/ztest ztest.c, module/zfs zio.c

Minor spelling corrections

Some minor spelling mistakes and typos.  No functional changes.

Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Giuseppe Di Natale <guss80 at gmail.com>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8272 

ZFS on Linux/src 61c3391include/sys zthr.h, module/zfs zthr.c spa.c

Serialize ZTHR operations to eliminate races

Adds a new lock for serializing operations on zthrs.
The commit also includes some code cleanup and
refactoring.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed by: Tom Caputi <tcaputi at datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Serapheim Dimitropoulos <serapheim at delphix.com>
Closes #8229 

ZFS on Linux/src 83c796cmodule/zfs zfs_vfsops.c, tests/runfiles linux.run

zfs filesystem skipped by df -h

On full pool when pool root filesystem references very few bytes,
the f_blocks returned to statvfs is 0 but should be at least 1.

Reviewed by: Tom Caputi <tcaputi at datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Paul Zuchowski <pzuchowski at datto.com>
Closes #8253 
Closes #8254 

ZFS on Linux/src a13392acontrib/pyzfs .gitignore

Add contrib/pyzfs/setup.py to .gitignore

As of 9ef798b77, setup.py is now generated from setup.py.in, but
this file was never moved to the .gitignore. This patch simply
corrects this issue.

Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8268 

ZFS on Linux/src e34cd80rpm/generic zfs.spec.in

Add pyzfs BuildRequires for mock(1)

When building pyzfs under mock the python-cffi and python-setuptools
packages need to be installed and have been added to the BuildRequires.

Reviewed-by: Neal Gompa <ngompa at datto.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8265 

ZFS on Linux/src 99b0b5btests/zfs-tests/tests/functional/cli_root/zpool_resilver zpool_resilver_restart.ksh zpool_resilver.cfg

ZTS: zpool_resilver_restart

Since the vdev initialize feature was integrated the ZTS
zpool_resilver_restart test has been hitting its internal
timeout more frequently.  This happens most often on
the coverage builder but not exclusively.  Increasing the
timeout for this test case prevents any false positives.

Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8273 

ZFS on Linux/src 6955b40include/sys dmu.h zap.h, module/zfs zap_micro.c dmu_object.c

Provide more flexible object allocation interface

Object allocation performance can be improved for complex operations
by providing an interface which returns the newly allocated dnode.
This allows the caller to immediately use the dnode without incurring
the expense of looking up the dnode by object number.

The functions dmu_object_alloc_hold(), zap_create_hold(), and
dmu_bonus_hold_by_dnode() were added for this purpose.

The zap_create_* functions have been updated to take advantage of
this new functionality.  The dmu_bonus_hold_impl() function should
really have never been included in sys/dmu.h and was removed.
It's sole caller was converted to use dmu_bonus_hold_by_dnode().

The new symbols have been exported for use by Lustre.

Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed by: Matt Ahrens <mahrens at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8015 

ZFS on Linux/src 58769a4module/zfs dnode.c

Don't allow dnode allocation if dn_holds != 0

This patch simply fixes a small bug where dnode_hold_impl() could
attempt to allocate a dnode that was in the process of being freed,
but which still had active references. This patch simply adds the
required check.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tom Caputi <tcaputi at datto.com>
Closes #8249 
DeltaFile
+1-0module/zfs/dnode.c
+1-01 files

ZFS on Linux/src 8bd2a28contrib/dracut README.dracut.markdown

Removed suggestion to use root dataset as bootfs

The dracut howto proposed to boot from the root dataset of a pool.
Apart from this giving problems when booting (as the code seems to
expect a child dataset and creates an illegal dataset name when using
the root dataset) the technical limitations of the root dataset
(among others the inability to rename or destroy through the `zfs`
command) resulted in the general consensus to only use it as a
container for the datasets in the pool - not as a filesystem itself.

Removed the idea to boot from the root dataset.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: Gregor Kopka <gregor at kopka.net>
Closes #8247 

ZFS on Linux/src 9ef798b. configure.ac, contrib/pyzfs setup.py.in setup.py

Use ZFS version for pyzfs & drop unused reqs file

Now that 'pyzfs' is part of the ZFS codebase, it should be
versioned the same as the rest of the source tree. This eliminates
confusion on what version of the bindings are being used, especially
for dependent Python projects that may use the Python dist metadata
to identify compatible versions of pyzfs to work from.

In addition, a trivial change to drop the unused requirements.txt
file is included, simply because it's unused and a leftover from
before it was imported into the ZFS codebase and wired into the
autotools build scripts.

Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Neal Gompa <ngompa at datto.com>
Closes #8243 

ZFS on Linux/src 0f5f238tests/zfs-tests/tests/functional/limits filesystem_count.ksh snapshot_count.ksh

zfs receive and rollback can skew filesystem_count

This commit fixes a small issue which causes both zfs receive and
rollback operations to incorrectly increase the "filesystem_count"
property value.

This change also adds a new test group "limits" to the ZFS Test Suite
to exercise both filesystem_count/limit and snapshot_count/limit
functionality.

Reviewed by: Jerry Jelinek <jerry.jelinek at joyent.com>
Reviewed by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: loli10K <ezomori.nozomu at gmail.com>
Closes #8232 

ZFS on Linux/src f384c04module/zfs vdev_mirror.c

OpenZFS 8473 - scrub does not detect errors on active spares

Scrubbing is supposed to detect and repair all errors in the pool.
However, it wrongly ignores active spare devices. The problem can
easily be reproduced in OpenZFS at git rev 0ef125d with these
commands:

    truncate -s 64m /tmp/a /tmp/b /tmp/c
    sudo zpool create testpool mirror /tmp/a /tmp/b spare /tmp/c
    sudo zpool replace testpool /tmp/a /tmp/c
    /bin/dd if=/dev/zero bs=1024k count=63 oseek=1 conv=notrunc of=/tmp/c
    sync
    sudo zpool scrub testpool
    zpool status testpool # Will show 0 errors, which is wrong
    sudo zpool offline testpool /tmp/a
    sudo zpool scrub testpool
    zpool status testpool # Will show errors on /tmp/c,
                          # which should've already been fixed

FreeBSD head is partially affected: the first scrub will detect
some errors, but the second scrub will detect more.  This same
test was run on Linux before applying the fix and the FreeBSD
head behavior was observed.

Authored by: asomers <asomers at FreeBSD.org>

    [12 lines not shown]

ZFS on Linux/src 53b5fcd. Makefile.am, lib/libzfs Makefile.am

Include third party licenses in dist tarballs

Since the merge of the Linux Solaris Porting Layer source tree into
the ZFS codebase, ZFS is now a double-licensed codebase, with the
former SPL codebase retaining its license (GPLv2+) within the ZFS
source tree.

However, the license files for SPL were not being included in the
tarballs generated by autotools. This change corrects that.

In addition, all the other third party licenses in the codebase are
now properly declared to be included in the dist tarballs.

Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Neal Gompa <ngompa at datto.com>
Closes #8242 

ZFS on Linux/src 21e000arpm/generic zfs-dkms.spec.in

Fix missing dkms modules after upgrades

If you were upgrading from say, fc28->fc29, on ZFS version X, the RPMs
macros would get called like this:

%post X.fc29
   - This is the step where fc29 gets built by dkms.
     As part of the build, dkms automatically removes the previous
     modules before building the new ones.  It then builds the new
     modules.
%preun X.fc28
   - Right before this step, X.fc29 is be built and installed, but
     since it has the same X, it's files get inadvertently removed
     by fc28's uninstall.
%postun X.fc28

This patch updates %preun X.fc28 to see if we're upgrading or
uninstalling.  If we're uninstalling, then remove our files. If we're
upgrading then do nothing, since will know dkms will have already
removed our files in %post X.fc29.

Note that since this fixes the %preun step, it's effect isn't going
to be noticed immediately.  It will only be seen when packages
with this fix are upgraded to a newer version.


    [5 lines not shown]

ZFS on Linux/src 4efb48e.github CONTRIBUTING.md, scripts commitcheck.sh

Bump commit subject length to 72 characters

There's not really a reason to keep the subject length so short,
since the reason to make it this short was for making nice renders
of a summary list of the git log. With 72 characters, this still
works out fine, so let's just raise it to that so that it's easier
to give slightly more descriptive change summaries.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: Tony Hutter <hutter2 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Neal Gompa <ngompa at datto.com>
Closes #8250 

ZFS on Linux/src 22448f0man/man8 zfs.8

zfs.8 uses wrong snapshot names in Example 15

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: Benjamin Gentil <benjamin at gentil.io>
Closes #8241 
DeltaFile
+4-4man/man8/zfs.8
+4-41 files

ZFS on Linux/src a769fb5cmd/zpool zpool_main.c, man/man8 zpool.8

Add 'zpool status -i' option

Only display the full details of the vdev initialization state
in 'zpool status' output when requested with the -i option.
By default display '(initializing)' after vdevs when they are
being actively initialized.  This is consistent with the
established precident of appending '(resilvering), etc' and
fits within the default 80 column terminal width making it
easy to read.

Additionally, updated the 'zpool initialize' documentation to
make it clear the options are mutually exclusive, but allow
duplicate options like all other zfs/zpool commands.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Tim Chase <tim at chase2k.com>
Reviewed-by: George Wilson <george.wilson at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8230

ZFS on Linux/src c10d37dcmd/ztest ztest.c, include/sys vdev_initialize.h

zfs initialize performance enhancements

PROBLEM
========

When invoking "zpool initialize" on a pool the command will
create a thread to initialize each disk. Unfortunately, it does
this serially across many transaction groups which can result
in commands taking a long time to return to the user and may
appear hung. The same thing is true when trying to suspend/cancel
the operation.

SOLUTION
=========

This change refactors the way we invoke the initialize interface
to ensure we can start or stop the intialization in just a few
transaction groups.

When stopping or cancelling a vdev initialization perform it
in two phases.  First signal each vdev initialization thread
that it should exit, then after all threads have been signaled
wait for them to exit.

On a pool with 40 leaf vdevs this reduces the vdev initialize

    [19 lines not shown]

ZFS on Linux/src 619f097cmd/zpool zpool_main.c, cmd/ztest ztest.c

OpenZFS 9102 - zfs should be able to initialize storage devices

PROBLEM
========

The first access to a block incurs a performance penalty on some platforms
(e.g. AWS's EBS, VMware VMDKs). Therefore we recommend that volumes are
"thick provisioned", where supported by the platform (VMware). This can
create a large delay in getting a new virtual machines up and running (or
adding storage to an existing Engine). If the thick provision step is
omitted, write performance will be suboptimal until all blocks on the LUN
have been written.

SOLUTION
=========

This feature introduces a way to 'initialize' the disks at install or in the
background to make sure we don't incur this first read penalty.

When an entire LUN is added to ZFS, we make all space available immediately,
and allow ZFS to find unallocated space and zero it out. This works with
concurrent writes to arbitrary offsets, ensuring that we don't zero out
something that has been (or is in the middle of being) written. This scheme
can also be applied to existing pools (affecting only free regions on the
vdev). Detailed design:

    [47 lines not shown]

ZFS on Linux/src c87db59cmd/arc_summary arc_summary2 arc_summary.py, cmd/dbufstat dbufstat

Python 2 and 3 compatibility

With Python 2 (slowly) approaching EOL and its removal from distribitions
already being planned (Fedora), the existing Python 2 code needs to be
transitioned to Python 3.  This patch stack updates the Python code to
be compatible with Python 2.7, 3.4, 3.5, 3.6, and 3.7.

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: John Wren Kennedy <john.kennedy at delphix.com>
Reviewed-by: Antonio Russo <antonio.e.russo at gmail.com>
Closes #8096

ZFS on Linux/src 9de8c0ccontrib/pyzfs/libzfs_core _libzfs_core.py _error_translation.py, contrib/pyzfs/libzfs_core/bindings __init__.py

pyzfs: python3 support (library 1/2)

These changes are efficient and valid in python 2 and 3. For the
most part, they are also pythonic.

* 2to3 conversion
* add __future__ imports
* iterator changes
* integer division
* relative import fixes

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Antonio Russo <antonio.e.russo at gmail.com>
Closes #8096

ZFS on Linux/src 6e72a5bcmd/arc_summary arc_summary2 arc_summary.py, cmd/dbufstat dbufstat dbufstat.py

pyzfs: python3 support (build system)

Almost all of the Python code in the respository has been updated
to be compatibile with Python 2.6, Python 3.4, or newer.  The only
exceptions are arc_summery3.py which requires Python 3, and pyzfs
which requires at least Python 2.7.  This allows us to maintain a
single version of the code and support most default versions of
python.  This change does the following:

* Sets the default shebang for all Python scripts to python3.  If
  only Python 2 is available, then at install time scripts which
  are compatible with Python 2 will have their shebangs replaced
  with /usr/bin/python.  This is done for compatibility until
  Python 2 goes end of life.  Since only the installed versions
  are changed this means Python 3 must be installed on the system
  for test-runner when testing in-tree.

* Added --with-python=<2|3|3.4,etc> configure option which sets
  the PYTHON environment variable to target a specific python
  version.  By default the newest installed version of Python
  will be used or the preferred distribution version when
  creating pacakges.

* Fixed --enable-pyzfs configure checks so they are run when
  --enable-pyzfs=check and --enable-pyzfs=yes.

    [29 lines not shown]

ZFS on Linux/src dffce3ctests/test-runner/bin test-runner.py

test-runner: python3 support

Updated to be compatible with Python 2.6, 2.7, 3.5 or newer.

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: John Wren Kennedy <john.kennedy at delphix.com>
Closes #8096

ZFS on Linux/src 530248dscripts zfs-tests.sh, tests/runfiles linux.run

arc_summary: consolidate test case

Since we're only installing one version of arc_summary we only
need one test case.  Update the test to determine which version
is available and then test its supported flags.

Remove files for misc tests which should have been cleaned up.

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8096

ZFS on Linux/src e5fb1dccontrib/pyzfs/libzfs_core _libzfs_core.py _error_translation.py

pyzfs: python3 support (library 2/2)

* All pool, dataset, and nvlist keys must be of type bytes.

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Signed-off-by: John Kennedy <john.kennedy at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8096

ZFS on Linux/src 4b1c406contrib/pyzfs/libzfs_core/test test_libzfs_core.py test_nvlist.py

pyzfs: python3 support (unit tests)

* Updated unit tests to be compatbile with python 2 or 3.  In most
  cases all that was required was to add the 'b' prefix to existing
  strings to convert them to type bytes for python 3 compatibility.

* There were several places where the python version need to be
  checked to remain compatible with pythong 2 and 3.  Some one
  more seasoned with Python may be able to find a way to rewrite
  these statements in a compatible fashion.

Reviewed-by: John Ramsden <johnramsden at riseup.net>
Reviewed-by: Neal Gompa <ngompa at datto.com>
Reviewed-by: loli10K <ezomori.nozomu at gmail.com>
Signed-off-by: John Wren Kennedy <john.kennedy at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8096

ZFS on Linux/src 0b8e441module/zcommon zfeature_common.c

Add zfs module feature and property compatibility

This change is required to ease the transition when upgrading
from 0.7.x to 0.8.x.  It allows 0.8.x user space utilities to
remain compatible with 0.7.x and older kernel modules.

Reviewed-by: Don Brady <don.brady at delphix.com>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8231 

ZFS on Linux/src 5365b07include libzfs.h, lib/libzfs libzfs_status.c

Add missing MMP status code to libzfs_status

When MMP was merged the status codes in libzfs_status were not
updated to add the status code for ZPOOL_STATUS_IO_FAILURE_MMP.  This
commit corrects this and adds comments to help keep track of which
code is used for which status.

Reviewed-by: Olaf Faaland <faaland1 at llnl.gov>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: bunder2015 <omfgbunder at gmail.com>
Closes #8148
Closes #8222 

ZFS on Linux/src 65ca2c1cmd/ztest ztest.c, module/zfs dmu.c

Fix 'zpool remap' freeing race

The dmu_objset_remap_indirects_impl() logic depends on dnode_hold()
returning ENOENT for dnodes which will be freed and should be skipped.

This behavior can only be relied upon when taking a new hold and
while the caller has an open transaction.  This ensures that the
open txg cannot advance and that a concurrent free will end up
in the same txg (which is critical).  Relying on an existing hold
will not prevent dnode_free() from succeeding.

The solution is to take an additional dnode_hold() after assigning
the transaction.  This ensures the remap will never dirty the dnode
if it was freed while we were waiting in dmu_tx_assign(, TXG_WAIT).

Randomly set zfs_object_remap_one_indirect_delay_ms in ztest.  This
increases the likelihood of an operation racing with the remap.
Converted from ticks to milliseconds.

Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed by: Tom Caputi <tcaputi at datto.com>
Reviewed by: Igor Kozhukhov <igor at dilos.org>
Signed-off-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Closes #8215 

ZFS on Linux/src 06f3fc2man/man8 zfs.8

Minor tweaks to zfs.8 man page for POSIX ACLs

* Capitalize POSIX in POSIX ACLs.  This change makes the POSIX 
  in POSIX ACLs all caps, which is both correct and consistent with
  the rest of the man page.

* Slightly reword part of zfs.8.  I tweaked a sentence to add a 
  missing comma, and as long as I was editing, removed a couple
  unnecessary words.

Reviewed-by: George Melikov <mail at gmelikov.ru>
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Reviewed-by: bunder2015 <omfgbunder at gmail.com>
Signed-off-by: Richard Laager <rlaager at wiktel.com>
Closes #8220 
DeltaFile
+6-6man/man8/zfs.8
+6-61 files

ZFS on Linux/src 3ec34e5include/spl/sys kmem.h, include/sys zthr.h

OpenZFS 9284 - arc_reclaim_thread has 2 jobs

Following the fix for 9018 (Replace kmem_cache_reap_now() with
kmem_cache_reap_soon), the arc_reclaim_thread() no longer blocks
while reaping.  However, the code is still confusing and error-prone,
because this thread has two responsibilities.  We should instead
separate this into two threads each with their own responsibility:

 1. keep `arc_size` under `arc_c`, by calling `arc_adjust()`, which
    improves `arc_is_overflowing()`

 2. keep enough free memory in the system, by calling
    `arc_kmem_reap_now()` plus `arc_shrink()`, which improves
    `arc_available_memory()`.

Furthermore, we can use the zthr infrastructure to separate the
"should we do something" from "do it" parts of the logic, and
normalize the start up / shut down of the threads.

Authored by: Brad Lewis <brad.lewis at delphix.com>
Reviewed by: Matt Ahrens <mahrens at delphix.com>
Reviewed by: Serapheim Dimitropoulos <serapheim at delphix.com>
Reviewed by: Pavel Zakharov <pavel.zakharov at delphix.com>
Reviewed by: Dan Kimmel <dan.kimmel at delphix.com>
Reviewed by: Paul Dagnelie <pcd at delphix.com>

    [10 lines not shown]