arm64/vmm: Treat Rt=31 as xzr in reg & instr emul
For certain instructions like msr, mrs, ldr & str, Rt=31 means the
operand is the Zero register (XZR) as opposed to the SP, even though the
two share the encoding of 31 on AArch64.
This is currently ignored in register and instruction emulation, so
emulated guest operations like "msr <reg>, xzr" will set the value of
<reg> to the current value of the SP instead of to 0.
Skip set and get operations when Rt=31 to correct the behaviour.
Signed-off-by: Kajetan Puchalski <kajetan.puchalski at arm.com>
Reviewed by: andrew
Sponsored by: Arm Ltd
Pull Request: https://github.com/freebsd/freebsd-src/pull/2454
arm64: Relax memory barriers around DMA operations
Before and after DMA operations we need memory barriers to ensure
stores on the CPU are observed by the device, and stores by the device
are observed by the CPU.
Previously we used a "dsb sy" instruction, however this is too strong
and can lead to slower performance.
Drew Gallatin found and tested the issue with cache-coherent devices,
reducing the barriers to dmb instructions in the outer-shareable
domain. After discussing the issue with the Arm Linux DMA maintainer I
concluded it was safe for non-coherent devices as the only dsb
instructions we need are after cache operations and these already exist.
Replace the dsb before DMA with a "dmb oshst" when cache-coherent, or
the existing dsb after invalidating the cache when not coherent. This
ensures earlier stores have completed before we signal the device.
[7 lines not shown]
__FreeBSD_version: Bump for changing pkthdr layout
Changing the layout of the pkthdr requires external drivers
to be rebuilt, lest they access the wrong fields.
Sponored by: Netflix
Differential Revision: https://reviews.freebsd.org/D60691
mbuf: move leaf_rcv union farther back in the pkthdr
When the leaf_rcv union was introduced to support ktls RX offload in
4d88d81c3166 it was put at the front of the pkthhdr, and pushed the
more commonly accessed csum_flags and rsstype into the next cacheline.
Given that csum_flags is accessed on every pkt sent and every pkt freed,
but the leaf_rcv fields are only used for ktls rx offload, it makes more
sense to have csum_flags in a cacheline that is already likely to be hot.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D60691
Sponsored by: Netflix
pf: Parse the GETSTATES state id as 64 bits
The GETSTATES dump sends PF_ST_ID as a u64, but the request parser
reads it as a u32 and rejects an id taken from the dump with EINVAL.
Nothing in base sends PF_ST_ID. Parse and document it as a u64.
Reviewed by: kp
Fixes: 2cef62886dc7 ("pf: convert state retrieval to netlink")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60543
libpfctl: Return max_win and mss of state peers
The kernel sends PF_STP_MAX_WIN and PF_STP_MSS for both peers of a
state, but struct pfctl_state_peer has no fields for them and libpfctl
dropped them. Add the fields and parse the attributes.
Reviewed by: kp
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60544
if_bnxt: fix powerpc build of the MPC ring code
powerpc has no 16-bit acquire/release atomics, so the
atomic_load_acq_16() and atomic_store_rel_16() calls on the MPC TX
ring indices fail to build there. Make the producer and consumer
indices 32 bits wide and use the 32-bit primitives.
Reviewed by: gallatin, chandrakanth.patil_broadcom.com
Sponsored by: Broadcom Inc
MFC after: 1 week
Fixes: 088a42c0ff76 ("if_bnxt: add MPC (Mid-Path Channel) ring infrastructure")
Differential Revision: https://reviews.freebsd.org/D60698
pf: Set the reassembly flags under the rules lock
DIOCSETREASS stored V_pf_status.reass twice with no lock, first the
flags it was given and then, if reassembly was not among them, zero.
pf_normalize_ip() and pf_normalize_ip6() read the flags with the rules
lock held for reading, so a packet could see the intermediate value.
Compute the flags first and store them once, with the rules lock held
for writing, as DIOCSETDEBUG and DIOCSETHOSTID do.
Reviewed by: kp
Fixes: 39282ef356db ("pf: backport OpenBSD syntax of "scrub" option for "match" and "pass" rules")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60531
pf: first-match-wins for nat rules, even in anchors
Ensure we stop evaluating nat rules if we find a match in a nat-anchor.
PR: 299320
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
tests/netinet6: Turn off stable addresses in ndp_prefix_len_mismatch
ndp_prefix_len_mismatch exercises the fallback in in6_ifadd() for an
interface whose link-local address does not have the advertised prefix
length (9e792f7ef729). Since a2eb0894b79b made stable addresses
(RFC 7217) the default, in6_ifadd() takes the interface identifier from
in6_get_stableifid() and never looks at the link-local address, so the
case passes without reaching the fallback. Turn stable addresses off
on the interface.
Sponsored by: Rubicon Communications, LLC ("Netgate")
hwpmc: Add support for AMD-only IRPERF in hwpmc PERF class
The IRPERF MSR counter (0xc00000e9) exposes an instructions-retired count
when read. Its support is advertised through CPUID leaf 0x80000008 EBX bit 1.
If this bit is set, the CPU supports the IRPERF MSR. For it to be used
by the system, bit 30 of the HWCR MSR (0xC0010015) needs to be enabled.
This patch checks for availability of the feature and enables the bit. If
the feature is available, the hwpmc PERF class detects its presence and
registers a third PERF MSR counter that allows hwpmc users to measure
instructions retired using IRPERF. This is AMD-only, as Intel does not
support IRPERF.
Due to erratum #1054 affecting AMD Zen 1 before model 30h, IRPERF is only
enabled for AMD Zen 1 model 30h onwards and later microarchitectures. This
commit also implements a validation step to make sure the IRPERF counter
is actually incrementing; otherwise, the feature is not enabled. This
guarantees it is working as expected before being made available to the
system and users.
[20 lines not shown]
pfil: Avoid potential mbuf tag leak
In '398254f9999e mbuf: simplify m_free' we removed tag processing
on m_free() of M_NOFREE mbufs. It is possible that a packet filter
could attach a tag to this stack mbuf and call m_free()
on it, meaning that 398254f9999e may have introduced a tag leak.
However,if a tag is attached without anyone calling m_free(), then
this leak was pre-existing.
Reviewed by: glebius
Differential Revision: https://reviews.freebsd.org/D60686
Fixes: 398254f9999e
Sponsored by: Netflix
hwpmc_amd: fix INVARIANTS build
The variable 'amd_npmcs' is not yet present in this branch.
This is a direct commit to stable/14.
Reported by: Jenkins
Sponsored by: The FreeBSD Foundation
stress2: Add a new FS type: UFS2 to the test.
Workaround for kern.maxfiles limit exceeded.
Workaround for SU disk full issue.
Added some debug info, if umount(8) returns EBUSY.
ufs: do not let a rolled back inode write end a data-only sync
With soft updates, ffs_syncvnode() writes the inode between its passes
while buffers are still dirty. That write can go out with new block
pointers rolled back, for instance a new indirect block whose contents
are only written in a later pass, yet ffs_update() clears IN_SIZEMOD
and IN_IBLKDATA because the write was synchronous. A data-only sync
then decides at the end that the inode need not be written, and
fdatasync(2) or an O_DSYNC write that allocated the first indirect
block returned with the on-disk inode not pointing to it.
Note on entry whether a data-only sync has to write the inode, and base
the final write, done once all dependencies are flushed, on that alone:
nothing else can set IN_SIZEMOD or IN_IBLKDATA meanwhile, as the vnode
is locked. A full sync always writes the inode last and was not
affected.
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
[3 lines not shown]
tests: add UFS synchronous write durability tests
Check that once fsync(2), fdatasync(2), or a write(2) with O_SYNC or
O_DSYNC returns, the new data, the file size and every block pointer
leading to the data are on disk. The cases append a block, fill a hole,
append the first block that needs an indirect block and append below an
existing indirect block. Two more append after another write left a
new direct block unsynced, which with soft updates cuts the size and the
later pointers of an inode written meanwhile. With O_SYNC and O_DSYNC
on soft updates, those also check that the open already made the
unsynced block durable.
Each case runs on UFS without soft updates, with soft updates and with
SU+J.
The file systems live on md(4) and the checks read the inode, the
indirect block and the data block straight from the device with libufs
right after the synchronous call returns. There is no cache in front
of the disk device, so this shows what would survive a crash at that
moment without having to crash anything.
[6 lines not shown]
ufs: make fdatasync(2) and O_DSYNC writes update the inode
ffs_syncvnode() in the DATA_ONLY case and ffs_write() for IO_DATASYNC
decide whether the inode has to be written by testing IN_SIZEMOD and
IN_IBLKDATA against ip->i_flags, the chflags(2) word, instead of the
in-core state ip->i_flag. IN_SIZEMOD and IN_IBLKDATA alias UF_READONLY
and UF_ARCHIVE, so for ordinary files the test was always false: after
an fdatasync() or an O_DSYNC write that extended a file or filled a
hole, the data blocks were on disk but the inode size and block pointers
were not, and a crash before the syncer got to the inode lost the
"synced" data. Files with uarchive or urdonly set got a synchronous
inode write on every call instead.
journal_mount() has the same mixup and sets IN_MODIFIED in i_flags,
which leaves UF_OPAQUE set on the in-core .sujournal inode. The
synchronous ffs_update() that follows writes the inode regardless, so
only the stray flag is fixed here.
Fixes: 7428630b757 ("UFS: write inode block for fdatasync(2) if pointers in inode where allocated")
[9 lines not shown]
ufs: flush soft updates dependencies for O_SYNC and O_DSYNC writes
For a synchronous write, ffs_write() writes the data buffers with
bwrite() and then the inode with ffs_update(vp, 1). With soft updates,
a new block pointer can only reach the disk after the dependencies of
the block, such as the cylinder group map that allocates it, are
written. Until then initiate_write_inodeblock_ufs[12]() and
initiate_write_indirdep() roll the pointer, and for an extending write
the size, back in the buffer being written. Nothing in the write path
flushes those dependencies, so the inode went out with the old size and
pointers, ffs_update() cleared IN_SIZEMOD and IN_IBLKDATA, and a crash
before the syncer rewrote the inode lost data that O_SYNC or O_DSYNC
promised was on stable storage. The same held for a new block below an
indirect block, which ffs_balloc() writes synchronously but with the
pointer rolled back.
When soft updates are enabled and the write allocated blocks, finish it
with ffs_syncvnode(), as fsync(2) and fdatasync(2) do: it flushes the
dependencies, rewrites the rolled back indirect blocks and writes the
[30 lines not shown]
ufs: make fdatasync(2) write dirty indirect blocks
A data-only ffs_syncvnode() skips dirty indirect blocks on file systems
without soft updates, and with soft updates stops making passes once
only indirect blocks are left dirty. Indirect blocks only get dirty
when the block pointers in them change, and those pointers are what
makes a newly allocated block reachable. So when fdatasync() follows
a write that extended the file or filled a hole past the direct blocks,
the data and the inode size reached the disk but the pointer to the new
block did not, and a crash lost the "synced" data. With soft updates
the indirect block was usually written once, with the new pointer
rolled back, and never rewritten.
POSIX requires fdatasync() to write the metadata needed to retrieve the
data, which 52488b51489 already acknowledged for the inode, so treat
dirty indirect blocks the same way for a data-only sync as for a full
one. This restores the loop to what it was before 2f514f92cf0. Plain
overwrites, the case data-only sync is meant to speed up, do not dirty
indirect blocks and still write neither them nor the inode.
[8 lines not shown]
stress2: add a test for forced unmount racing mkdir at UFS_LINK_MAX
On a soft updates file system, keep a directory at UFS_LINK_MAX with
workers removing and recreating subdirectories in it, so that mkdir(2)
keeps going through ufs_sync_nlink(), and force-unmount, check and
remount the file system in a loop. The workers work relative to the
directory: a lookup through the mount point would wait for the unmount
on the covered vnode lock and never race with it. Before the fix, the
first forced unmount deadlocked with mkdir(2) sleeping in vfs_busy().
With dtrace=1, count the vfs_busy() calls made during an unmount by
caller, and the mkdir(2) calls that synced and restarted with ERELOOKUP
during an unmount, which shows that the race was hit.
Sponsored by: Sippy Software, Inc.
Reviewed by: pho
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60204
MFC after: 1 month
ufs: do not busy or reference the mount in ufs_sync_nlink()
With soft updates, mkdir(2), link(2) and rename(2) into a directory
whose i_nlink is at UFS_LINK_MAX while i_effnlink is below it call
ufs_sync_nlink1() to sync the file system and restart. It busied the
mount with vfs_busy(mp, 0), which sleeps for as long as an unmount is in
progress. The caller is still between vn_start_write() and
vn_finished_write() there, and the unmount, once it has set
MNTK_UNMOUNT, goes on to suspend writes in VFS_UNMOUNT() and waits in
vfs_write_suspend() for that write to end. Neither could make progress,
and every later vfs_busy() of the mount, such as from getfsstat(2) or a
lookup crossing the mount point, hung behind them.
The window is wide: after setting MNTK_UNMOUNT, the unmount waits for
the busy references to drain, which includes another thread already
syncing in ufs_sync_nlink1(), and for vfs_periodic(). A forced unmount
of a file system with a directory at the link limit under mkdir/rmdir
load hits it within seconds.
[19 lines not shown]
ncal(1): handle Sweden's transitional calendar (1700-1712)
Sweden used a unique intermediate calendar during 1700-1712. In 1700,
Sweden omitted the Julian leap day (Feb 29, 1700) to begin a gradual
transition to the Gregorian calendar. The transition was abandoned
mid-way due to the Great Northern War; to revert to the Julian calendar,
an extra day "Feb 30, 1712" was added. Sweden finally adopted the
Gregorian calendar in 1753 (already handled correctly by ncal).
ncal previously displayed Feb 1700 as a 29-day month and Feb 1712 as a
29-day month, both incorrect for the Swedish (-s SE) calendar mode.
Add flag_sweden, which is set when the SE country code is selected (via
-s SE or the locale) and cleared by -J, since -J selects the plain
Julian calendar. When active, sndaysr() and sdater() apply the one-day
offset that characterised the Swedish transitional calendar:
- Feb 1700: displayed with 28 days (no leap day)
- 1700-03-01 to 1712-02-30: dates run one day ahead of Julian, so
[15 lines not shown]
mbuf: simplify m_free
The check for (M_PKTHDR|M_NOFREE) was added by np@ in
"91d1b03d3617 Merge r254736 from user/np/cxl_tuning."
in order to support a clever rx mode optimization used
by cxgbe at the time. That optimization was removed in
46e1e307edb6 several years ago. This check is otherwise
unneeded and adds an extraneous test to every mbuf free,
so let's remove it.
Reviewed by: glebius, np
Differential Revision: https://reviews.freebsd.org/D60686
Sponsored by: Netflix
iflib: unify prefetching
This change makes all prefetching use the iflib prefetch wrappers, rather
than using prefetch() in some places and __builtin_prefetch() in others.
Note that __builtin_prefetch() does not work in the kernel on amd64 as clang
disables it when -mnosse is specified, so we must use the asm directives there,
and builtin_prefetch() elsewhere.
This change enables prefetching on non-x86 (notably arm64), and it enables
prefetching in netmap that has been there since day one, but which has never worked.
This is a no-op on non-netmap amd64 and i386
Differential Revision: https://reviews.freebsd.org/D60602
Reviewed by: glebius, kbowling
Sponsored by: Netflix
rms: Order a writer's stores before its readers see it leave
Readers check rms->writers without a fence, and rms_wunlock() drops
it with a plain store. rms_wlock() interrupts every CPU before the
first writer blocks readers, but nothing orders them when the last
writer leaves: on a weakly ordered CPU a reader can see the count
reach zero and still read what the writer replaced, such as callbacks
of a module unloaded under the lock.
Fence every CPU before the last writer drops the count: rename
rms_wlock_switch() to rms_block_readers() and add
rms_unblock_readers(), a wrapper for cpus_fence_seq_cst(). Readers
stay fence-free; the cost lands on the rare unlock.
Reviewed by: mjg
Discussed with: kib
Fixes: 1f162fef76a0 ("Add read-mostly sleepable locks")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60658
