nullfs: close a race when syncing inotify flags from the lower vnode
After a bypassed VOP, nullfs mirrors the lower vnode's inotify state
onto the upper vnode. The flags were checked with lockless reads
before being updated with the asserting flag set/unset primitives, so
two threads syncing the same vnode concurrently (or a sync racing a
watch being established) could both decide to make the same change;
the loser then trips the "flags already set" assertion on an
INVARIANTS kernel. On other kernels the race is harmless.
Keep the lockless check as the fast path, but re-make the decision
under the vnode interlock before actually changing the flags.
Reproduced in a 4-CPU VM with one thread cycling an inotify watch on
a lower-filesystem file while several threads stat(2) the same file
through a nullfs mount: the unpatched INVARIANTS kernel panics under
this load, the patched kernel runs it to completion.
Fixes: f1f230439fa4 ("vfs: Initial revision of inotify")
[6 lines not shown]
libc: Add strfromd, strfromf, and strfroml per C23
strfromd(), strfromf(), and strfroml() are implemented directly
in terms of gdtoa. If a non-conforming format string is passed,
the string "EDOOFUS" is returned and errno set to EDOOFUS as an
extension.
Reviewed by: fuz
MFC after: 1 month
Pull-Request: https://github.com/freebsd/freebsd-src/pull/2301
Signed-off-by: Faraz Vahedi <kfv at kfv.io>
msun: add asinpi, acospi, and atanpi
This commit implements the inverse half-cycle
trigonometric functions:
asinpi(x) = asin(x) / pi Eq. (1)
acospi(x) = acos(x) / pi
atanpi(x) = atan(x) / pi
Implemention details are contained in src/s_asinpi.c and
src/a_atanpi.c, where the details for acospi(x) appear in
the former.
*************
CAVEAT EMPTOR: The ld128 code has been only compiled. It has
not been tested for correctness due to lack of hardware.
*************
[125 lines not shown]
libc/merge.c: use memcpy() for copying
Currently mergesort() uses ICOPY_*() to copy data as four byte blocks
instead of one byte. However, this is only achievable when both size and
base arguments are aligned to four bytes.
Use of memcpy() is ideal as 1) it is cleaner and 2) the library will use
SIMD for copying when the hardware supports it. Compared to ICOPY_*(),
SIMD can support up to 64 bytes. When the SIMD-backed memcpy() find the
address is unaligned, it can first copy data up to the nearest aligned
address, and then use SIMD operations for faster transfer. Thus memcpy()
can give better performance than mergesort()'s own implementation.
This is benchmarked on amd64 where there isn't a SIMD-backed
implementation yet. However, the baseline implementation in assembly
already delivers better performance in unaligned cases although there is
some performance drops in aligned cases. The benchmark results and
script is available in the Phabricator review. Ideally, more performance
improvements will come when amd64 gets SIMD implementation of memcpy().
[5 lines not shown]
libutil++: Include <cerrno> in stringf.cc
stringf.cc uses errno and related macros without including <cerrno>.
Their availability is guaranteed only when the corresponding header
is included; transitive exposure is implementation-defined.
Modern libc++ has been progressively reducing incidental transitive
includes as part of its header removal policy (see LLVM libc++ Header
Removal Policy and D132284), making such dependencies brittle.
This change includes <cerrno> explicitly to make the dependency
well-defined. No functional or behavioural change intended.
Approved by: fuz
Signed-off-by: Faraz Vahedi <kfv at kfv.io>
Pull-Request: https://github.com/freebsd/freebsd-src/pull/2188
bhyve: tidy up bhyve_config.5
There are few warnings reported by mandoc -Tlint:
bhyve_config.5:255:31: WARNING: new sentence, new line
bhyve_config.5:257:43: WARNING: new sentence, new line
bhyve_config.5:422:2: WARNING: missing section argument: Xr nm_open
bhyve_config.5:469:24: WARNING: skipping no-space macro
bhyve_config.5:483:2: WARNING: wrong number of cells: 2 columns, 4 cells
bhyve_config.5:484:2: WARNING: wrong number of cells: 2 columns, 4 cells
bhyve_config.5:541:24: WARNING: skipping no-space macro
- "new sentence, new line" is a trivial formatting fix.
- "missing section": there is actually no nm_open() manual page,
so use .Nm instead of .Xr for it.
- "no-space macro": format without .Oc and .Ns, similarly to
how it is already done in bhyve.8 for VNC addresses.
- "wrong number of cells": also a trivial fix.
[6 lines not shown]
igc(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit 297394e995e5ea1ea9bc85e609ca116255d51e97)
igc: count TSO wire segments in the AIM counters
The transmit path bills one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64 KiB, rather than a
packet size that appears on the wire.
Count the segments the hardware emits and the header carried by each
segment. Non-TSO accounting is unchanged.
(cherry picked from commit e389a05164ccb1dd41ee8d7f09203b475322dd72)
igc: use packet-size AIM
Use the packet-size calculation introduced for igb(4) in a69ed8dfb381
and retained there until the iflib conversion in f2d6ace4a684. It
derives interrupt holdoff from average packet size, so RSS queue count
does not change its behavior.
The calculation follows the pre-iflib igb code. Retain igc's normal and
low-latency rate caps, and keep the current setting when an interval has
no usable sample.
(cherry picked from commit 01e7acd38d411c78caba1c4078bb3683f586e1c2)
igc: synchronize interrupt moderation state
Keep the saved EITR value synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Apply the packet-buffer fallback without permanently disabling AIM.
(cherry picked from commit e35533457530bb9db655e6137c2eea790e18b97b)
igc: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit 2290ea7f4311e899019fe77bf7c7775033af6b24)
igc: fix RX accounting for multi-descriptor packets
The receive path adds the running packet length to rx_bytes for every
descriptor. A packet spanning descriptors of length l1, l2, and l3 is
therefore counted as 3*l1 + 2*l2 + l3.
Add each descriptor length once. Single-descriptor accounting remains
unchanged.
(cherry picked from commit bbf0372feeb321a5bfeff7b1e79576ab01240441)
em(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit b6b379b94781da5d4328f6f57273fbe7bd9dc687)
e1000: count TSO wire segments in the AIM counters
The transmit paths billed one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64KB, so the average
size the moderation calculation sees is not a size that appears on the
wire.
Count the segments the hardware will put on the wire and the header each
of them carries.
Non-TSO accounting is unchanged.
(cherry picked from commit 072e0983d7bce80356740324973993393e77023a)
e1000: restore packet-size AIM
Restore the packet-size calculation introduced in a69ed8dfb381 and used
by igb(4) until the iflib conversion in f2d6ace4a684. It derives
interrupt holdoff from average packet size, so RSS queue count does not
change its behavior.
The calculation follows the pre-iflib code. Retain the current normal
and low-latency rate caps, and keep the current setting when an interval
has no usable sample.
Fixes: 3e501ef89667 ("e1000: Re-add AIM")
(cherry picked from commit dc4a5087b160c1a94d135ab636642defe2c71c20)
e1000: synchronize interrupt moderation state
Keep the saved EITR and PBA values synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Treat only sub-gigabit links as sub-gigabit and apply the packet-buffer
fallback without permanently disabling AIM.
(cherry picked from commit 6ef368a29b11ebc769e7929566809b75ae2c1e90)
e1000: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit bc5e7b0cbbb555ffebc7d73b273c421f9ee24c23)
ixl: Increase tx/rx ring size to 8160
I've verified the tx queue (table 8-22) in addition.
DPDK commit message
net/i40e: increase max descriptor queue length
According to the Intel X710/XXV710/XL710 Datasheet, the maximum receive
queue descriptor length is 0x1FE0 (8160 in base 10). This is specified
as QLEN in table 8-12, page 1083.
I've tested this change with an XXV710 NIC and it has positive effect on
performance under high load scenarios. Where previously I'd get
~2000 packets/sec miss rate, now I get only ~40 packets/sec miss rate.
Signed-off-by: Igor Gutorov <igootorov at gmail.com>
Acked-by: Morten Brørup <mb at smartsharesystems.com>
Acked-by: Bruce Richardson <bruce.richardson at intel.com>
[3 lines not shown]
igc(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit 297394e995e5ea1ea9bc85e609ca116255d51e97)
igc: count TSO wire segments in the AIM counters
The transmit path bills one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64 KiB, rather than a
packet size that appears on the wire.
Count the segments the hardware emits and the header carried by each
segment. Non-TSO accounting is unchanged.
(cherry picked from commit e389a05164ccb1dd41ee8d7f09203b475322dd72)
igc: use packet-size AIM
Use the packet-size calculation introduced for igb(4) in a69ed8dfb381
and retained there until the iflib conversion in f2d6ace4a684. It
derives interrupt holdoff from average packet size, so RSS queue count
does not change its behavior.
The calculation follows the pre-iflib igb code. Retain igc's normal and
low-latency rate caps, and keep the current setting when an interval has
no usable sample.
(cherry picked from commit 01e7acd38d411c78caba1c4078bb3683f586e1c2)
igc: synchronize interrupt moderation state
Keep the saved EITR value synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Apply the packet-buffer fallback without permanently disabling AIM.
(cherry picked from commit e35533457530bb9db655e6137c2eea790e18b97b)
igc: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit 2290ea7f4311e899019fe77bf7c7775033af6b24)
igc: fix RX accounting for multi-descriptor packets
The receive path adds the running packet length to rx_bytes for every
descriptor. A packet spanning descriptors of length l1, l2, and l3 is
therefore counted as 3*l1 + 2*l2 + l3.
Add each descriptor length once. Single-descriptor accounting remains
unchanged.
(cherry picked from commit bbf0372feeb321a5bfeff7b1e79576ab01240441)
em(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit b6b379b94781da5d4328f6f57273fbe7bd9dc687)
e1000: count TSO wire segments in the AIM counters
The transmit paths billed one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64KB, so the average
size the moderation calculation sees is not a size that appears on the
wire.
Count the segments the hardware will put on the wire and the header each
of them carries.
Non-TSO accounting is unchanged.
(cherry picked from commit 072e0983d7bce80356740324973993393e77023a)
e1000: restore packet-size AIM
Restore the packet-size calculation introduced in a69ed8dfb381 and used
by igb(4) until the iflib conversion in f2d6ace4a684. It derives
interrupt holdoff from average packet size, so RSS queue count does not
change its behavior.
The calculation follows the pre-iflib code. Retain the current normal
and low-latency rate caps, and keep the current setting when an interval
has no usable sample.
Fixes: 3e501ef89667 ("e1000: Re-add AIM")
(cherry picked from commit dc4a5087b160c1a94d135ab636642defe2c71c20)
e1000: synchronize interrupt moderation state
Keep the saved EITR and PBA values synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Treat only sub-gigabit links as sub-gigabit and apply the packet-buffer
fallback without permanently disabling AIM.
(cherry picked from commit 6ef368a29b11ebc769e7929566809b75ae2c1e90)
e1000: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit bc5e7b0cbbb555ffebc7d73b273c421f9ee24c23)
