ZIL replay can apply part of a split WR_NEED_COPY write
After a crash while an fsync() is still writing the log, ZIL replay can
apply the first part of a write without the rest, leaving the write
torn at an arbitrary 8-byte-aligned offset. This exposes a state that
no reader could have observed before the crash: new data up to the
tear, old data after it. While POSIX doesn't specify anything before
fsync() completes, ZFS intends to preserve the atomicity of writes as
much as possible: writes within a single block are intended to be
atomic, with larger writes torn only on recordsize boundaries.
The problem occurs with WR_NEED_COPY TX_WRITE records. When such a
record does not fit in the current log block, zil_lwb_assign() splits
it into several TX_WRITE records in consecutive log blocks. If the
system crashes after the first of those blocks is on disk but before
all of them are, zil_parse() stops at the missing block, but replay
has already applied the pieces before it. Files are affected for
async writes that a later fsync() commits, and for writes of any size
with a slog. Zvols are affected too, where a tear inside a sector
[75 lines not shown]
rumdl: update to 0.3.0.
Changed
2.4–4.7× faster rumdl fmt than 0.2.79 on frozen documentation
snapshots: Cargo (99 files) 4.7×, MkDocs Material (85 files) 2.6×, and
Ruff (16 files) 2.4×. Reusing parsed Markdown and shared source analysis
reduces repeated work. Speedups compare median elapsed times from 31 paired
runs after five warmups on Apple M6: ordinary Rust 1.96.0 release builds,
default threads, warm filesystem cache, and --no-config --no-cache --silent
(--flavor=standard for Cargo/Ruff, --flavor=mkdocs for MkDocs Material).
Output includes the correctness fixes listed below and can differ from 0.2.79.
Code block tools are supported and remain opt-in. Missing tags, missing
active-mode tools and invocations with no tools checked are ignored by default.
An omitted unknown-tag policy inherits on-missing-language-definition,
preserving existing strict configurations. Valid internal cached checks count
as checked; applicable external tools rerun to reflect changes in their own
configuration and environment.
on-missing-language-definition = "warn" now reports uncovered languages;
[134 lines not shown]
Makefile.inc1: Adjust mkimg bootstrap versions
Stream-optimized VMDK support was merged to stable/15 with
__FreeBSD_version 1501504.
Sponsored by: Amazon
vdev_mirror: scrub every copy of a gang block in a sorted scrub
A sorted scrub queues one read per DVA of a block, and each of these
reads carries all of the block's DVAs so that a damaged copy can be
repaired. vdev_mirror_map_init() therefore reads only the first DVA
of a scrub read while a sorted scrub runs, since the other DVAs are
checked by their own queued reads, and reads them all only on retry.
Gang blocks are not queued: dsl_scan_enqueue() issues them at once and
whole, and their gang headers and gang members are read by gang
children of that one read. These inherit ZIO_FLAG_SCRUB and so take
the same shortcut, but nothing else ever reads their other DVAs. A
damaged copy of a gang header or gang member, other than the first, is
neither detected nor repaired by a scrub, unless the first copy is
damaged too and the retry reads them all. With the default
redundant_metadata=all every gang header has at least two copies, and
gang members have as many as the copies property asks for. A legacy
scrub (zfs_scan_legacy=1) reads and repairs every copy.
[11 lines not shown]
ZTS: test that a scrub repairs every copy of a gang block
With the default redundant_metadata=all, a gang header gets one more
copy than the block it describes, up to three, and each gang child
gets as many as the block. A scrub with the default sorted scan reads
only the first copy of each, so the other copies of a gang header or
of a gang child can be damaged without the scrub detecting or
repairing it. A legacy scrub (zfs_scan_legacy=1) reads and repairs
them.
Add a test that forces 128K blocks to gang on a file vdev, overwrites
the second copy of a gang header (copies=1) and of a gang child
(copies=2), scrubs, and checks that both copies were restored.
This fails now: the scrub repairs neither copy.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Tigerblue77 <37409593+tigerblue77 at users.noreply.github.com>
Closes #19399
EC2: Pass --vmdk to bsdec2-image-upload if needed
Starting with version 1.5.0, bsdec2-image-upload supports uploading
stream-optimized VMDK files. If we build images in that format, we
need to upload them appropriately.
MFC after: 2 weeks
Sponsored by: Amazon
(cherry picked from commit 65840afc88553f89c051b96b081061a532cb7df3)
__FreeBSD_version: Bump for mkimg -f vmdks
Also, adjust bootstrap tools code in Makefile.inc1 to reflect this; in
addition to rebuilding mkimg if it is out of date, we now need to build
libz, which was not previously needed by mkimg.
Reviewed by: kevans
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60153
X-Cherry-Pick: Logic in Makefile.inc1 was adjusted for new __FreeBSD_version
(cherry picked from commit a68ba2d2ce55f7c76174239c1f678379caec5348)
mkimg: Add image_buffer_region
This is like image_copyout_region, but copies into a buffer rather than
writing out to a file; it will be used by future support for compressed
images (and potentially other circumstances where image data must be
manipulated before being written out to disk).
Reviewed by: jrm
No objection from: Christos Komis (author)
MFC after: 2 weeks
Sponsored by: Google LLC (GSoC 2025)
Differential Revision: https://reviews.freebsd.org/D60149
(cherry picked from commit 84a3e29c72030dab16b4dc82d601d0af5da6a18c)
mkimg: Check for realloc failure
If realloc fails, clean up and return ENOMEM; don't just copy data
into NULL.
Reviewed by: emaste
Reported by: Claude Opus 5.5
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60147
(cherry picked from commit 1a0523197c5c16ce7bcb79d16c6bf8945258b428)
mkimg: Make vmdk's desc_fmt slightly more generic
This will allow it to be reused in upcoming work.
Suggested by: Claude Opus 5.5
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60150
(cherry picked from commit 498276b487c838afa2b3e4df322c43141788ba20)
mkimg: Avoid leaking a page of mmap
The image_file_map function adjusts the provided file offset to be
page-aligned, with a resulting increase in the size of the mapped
region; the increased size needs to be used when unmapping as well.
Reported by: Claude Opus 5.5
Fixes: baf4abfc39b2 ("Allow building mkimg as cross-tool")
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60148
(cherry picked from commit 434819b4fec5e508bdfe72f91e73402e9c2fca22)
mkimg: Add support for stream-optimized VMDK
FreeBSD VM/cloud images tend to be highly compressible: The generic VM
images compress roughly 3.5:1, and cloud images typically even more
since they have significant unused space in their virtual disks. This
generally doesn't matter for users who can download compressed release
images and extract them locally, but for EC2 in particular relying on
uncompressed image formats is painful: We now upload over 500 GB/week
to AWS.
This commit adds the "stream-optimized" version of the VMDK format,
which compresses each 64 kB "grain" individually (and omits grains
which are all zeroes). Compared to uncompressed formats, this can
produce much smaller images; experiments indicate that using this
format for EC2 image uploads will reduce the weekly traffic to under
100 GB/week.
Co-authored-by: Claude Opus 5.5
MFC after: 2 weeks
[4 lines not shown]
spl: avoid reclaim for speculative physical allocations
The constrained spl_kvmalloc() path allows direct reclaim while trying
to obtain physically contiguous memory, even though vmalloc can satisfy
the request with scattered pages. __GFP_NORETRY limits retries but does
not prevent that initial reclaim.
For multipage requests without __GFP_RETRY_MAYFAIL, clear direct reclaim
only from the physical attempt's flags. Keep the original flags for
fallback eligibility and vmalloc so a sleeping allocation can still
reclaim memory there. This follows the policy adopted by Linux 6.16
without changing the fallback needed by older supported kernels.
Retain the existing small-order retry policy and correct its obsolete
explanation of __GFP_RETRY_MAYFAIL.
Reviewed-by: Brian Behlendorf <behlendorf1 at llnl.gov>
Signed-off-by: Kamil Monicz <kamil at monicz.dev>
Closes #19406
[libc][math] Optimize integer-only expf and factor out shared utilities.
- Optimize `expf_integer_eval` for both code size and performance.
- Factor out the shared range reduction and `2^u` fixed point evaluation to
be used by `exp2f` and `exp10f` integer-only versions.
Assisted-by: Gemini for analysis and refactoring.
Initial import of devel/cbmc version 6.10.0.
CBMC is a Bounded Model Checker for C and C++ programs. It supports
C89, C99, most of C11, C17, C23 and most compiler extensions provided
by gcc and Visual Studio. It also supports SystemC using Scoot. It
allows verifying array bounds (buffer overflows), pointer safety,
exceptions and user-specified assertions. Furthermore, it can check
C and C++ for consistency with other languages, such as Verilog.
The verification is performed by unwinding the loops in the program
and passing the resulting equation to a decision procedure.
raidz: refuse expansions that the boot area cannot hold
A RAID-Z expansion uses the boot area of each child as scratch space,
and raidz_reflow_scratch_sync() needs it to hold one sector for every
child of the expanded vdev:
VERIFY3U(write_size, >=, raidvd->vdev_children << ashift);
vdev_raidz_attach_check() is there to refuse an attach that would
break this, but spa_vdev_attach() calls it before the new leaf is
moved under the RAID-Z vdev, while its parent is still the temporary
root vdev it was parsed under. The check took the width and the
ashift from that root, one child and an ashift of zero, so it never
refused anything. Attaching a 57th disk to a RAID-Z vdev with
ashift=16, or a 113th with ashift=15, was accepted, and the VERIFY
failed in the syncing thread as soon as the expansion started.
Pass the RAID-Z vdev to vdev_raidz_attach_check() and count the child
that is about to be added. Return a new error for it rather than
[9 lines not shown]
