nda: Tag the command set used for devctl errors
Different command sets have different encoding for op codes, etc. While
one can normally puzzle out which is which, it's better to explicitly
tag the command set used.
Sponsored by: Netflix
uvideo: import quirks infrastructure from OpenBSD
Import the device quirk system from OpenBSD to handle UVC devices
that need special handling. This includes:
- UVIDEO_FLAG_ISIGHT_STREAM_HEADER: non-standard streaming header
- UVIDEO_FLAG_REATTACH: needs reattach after firmware upload
- UVIDEO_FLAG_VENDOR_CLASS: incorrectly reports as vendor class
- UVIDEO_FLAG_NOATTACH: device not supported
- UVIDEO_FLAG_FORMAT_INDEX_IN_BMHINT: format index in bmHint
Add quirks table with known devices and lookup function.
Add iSight stream header decoder for Apple iSight cameras.
Obtained from: OpenBSD
uvideo: increase UVIDEO_MAX_PU and UVIDEO_MAX_CT to 32
Some UVC devices (e.g. Logitech C920) expose more than 8 Processing
Unit descriptors, causing "too many PU descriptors found!" errors.
Increase both limits from 8 to 32 to accommodate such devices.
hwpmc: add regression tests for detaching a live process-mode PMC
Attach a process-mode counting PMC to the current process, start it,
then detach and release it while it is still loaded on the hardware -
the case that previously leaked the PMC's runcount reference and
wedged pmc_wait_for_pmc_idle() at release. A second case does the same
from a multi-threaded process so the sibling threads' references have
to be drained too.
The tests need an allocatable process-mode counting event and skip
where none is available (hwpmc(4) not loaded, or a VM without a vPMU).
Reviewed by: adrian
MFC after: 2 weeks
Assisted-by: Claude Code (Fable 5)
Differential Revision: https://reviews.freebsd.org/D58343
hwpmc: drain a process-mode PMC's runcount when a live target detaches
A process-mode PMC's runcount tracks how many CPUs currently have it
loaded in hardware. It is decremented only by the context-switch-out
and process-exit reclaim paths, both of which the scheduler invokes
only for processes flagged P_HWPMC. Detaching a target that still has
the PMC live in hardware dropped the target and cleared P_HWPMC without
taking the PMC off the hardware or dropping the runcount reference, so
the reference leaked. A subsequent release then spun in
pmc_wait_for_pmc_idle() forever waiting for the runcount to reach zero:
on an INVARIANTS kernel this panics ("waiting too long for pmc to be
free"), otherwise it is an unkillable loop holding the hwpmc lock. Any
process able to allocate a PMC can trigger this by attaching a counting
PMC to itself and detaching it before releasing.
Take the PMC off the hardware and drop the runcount reference as part
of detaching, before P_HWPMC is cleared: reclaim it from the detaching
thread's own CPU directly, and, when the detach removes the PMC's last
target, wait for any references held by the target's other threads to
[6 lines not shown]
hwpmc: add regression tests for counting-PMC counter wraparound
Exercise a process-mode counting PMC whose accumulated count crosses,
or already exceeds, the range of the underlying hardware counter.
Before the previous commit, the first context switch after the
hardware counter wrapped panicked INVARIANTS kernels with "negative
increment" and silently corrupted the accumulated count on other
kernels.
The tests need a hardware counting event backed by a counter narrower
than 64 bits and skip where none is available (hwpmc(4) not loaded,
or a VM without a vPMU).
Reviewed by: adrian
MFC after: 2 weeks
Assisted-by: Claude Code (Fable 5)
Differential Revision: https://reviews.freebsd.org/D58341
hwpmc: handle counter wraparound for process-mode counting PMCs
The accumulated count of a process-mode counting PMC is kept in a
64-bit software counter and seeded into the hardware counter at every
context switch in. Hardware counters are narrower than that - each
PMC class discovers and records its own counter width, e.g. 48 bits
on current x86 (queried from CPUID on Intel, architectural on AMD) -
so once the accumulated count approaches the end of the hardware
counter range, the counter wraps during a time slice and the value
read back at switch out is smaller than the value seeded. The
increment was computed assuming a full 64-bit counter: on INVARIANTS
kernels a long enough counting run panics with "negative increment"
the moment the accumulated count first crosses the hardware counter
range, and on other kernels the totals silently lose a full counter
range per wrap.
Compute the increment modulo the per-class hardware counter width
instead, in both places that accumulate switch-out deltas.
[4 lines not shown]
apei: Fix i386 build over bus read, write function
bus_{read,write}_8 are macro wrappers around the corresponding bus_space
functions in sys/bus.h, so implementing bus_{read,write}_8 won't work.
Implement the underlying bus_space function instead.
Reviewed by: jrtc27, rlibby
Fixes: 9313f6b01485
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D58301
sched_ule: Fix selecting lowest priority thread early in corner case
When transferring a thread with near 100% CPU statistics (but not 100%;
up to 57.5/59≈97.46%) to a CPU where the enqueue offset is ahead of at
least 2 from the dequeue one, which requires peculiar conditions to
happen (transfer triggered by a bind request or cpuset change, or during
balancing if a thread or more existed from a brief amount of time on the
origin CPU), the transferred thread can get placed after the dequeue
offset, effectively making it appear as a high priority one unduly,
causing latency increase for other threads.
The change here was missed when changing the enqueue and dequeue offsets
update mechanism to recover pre-256-queue-runqueue ULE anti-starvation
and fairness behavior. That change opened up the possibility that these
two offsets are apart by more than one.
Reviewed by: markj
Discussed with: Minsoo Choo <minsoo at minsoo.io>
Fixes: 6792f3411f6d ("sched_ule: Recover previous nice and anti-starvation behaviors")
[3 lines not shown]
procdesc: report NOTE_PDSIGCHLD for traced and stopped process
on attach of the knote. It is same as for NOTE_EXIT when attaching to
the exiting process.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D58327
acl_from_text.c: Allow negative uid/gid numbers to be handled
getfacl / acl_to_text() incorrectly prints uid/gid numbers as signed integers.
This causes uid / gid numbers larger than 2G (2147483648) to print as
negative numbers.
The libc acl_from_text() function does not handle negative numbers.
This diff adds a backwards compatiblity fix to allow negative numbers...
(cherry picked from commit d7d71341ae7d79886143a9ce427dca0e858eda97)
i386: provide PCPU pc_small_core for amd64 compat
Provide pc_small_core for i386 too to fix an i386 build break from x86
code referring to it. It won't be set.
Reviewed by: aokblast, kib
Fixes: 7b26353a59d6 ("hwpstate_intel: Disable package control on hybrid CPU")
Differential Revision: https://reviews.freebsd.org/D58335
re(4): add hw.re.aspm_disable loader tunable
re(4) has unconditionally disabled ASPM L0s/L1 and CLKREQ at attach
for years; on laptops this costs 200mW+ (requested by adrian@ in the PR).
Make it a tunable following the existing hw.re.* pattern:
* default 1 keeps today's behavior;
* 0 preserves the firmware-configured ASPM state at attach and
skips the watchdog re-assert from the previous revision.
Documented in re.4.
* Verified on RTL8168H (XID 0x541): with hw.re.aspm_disable=0,
attach no longer logs "ASPM disabled" and pciconf -lcb shows
the firmware Link Control state preserved -- including Clock PM,
which the unconditional code previously cleared.
* Default (1) is behaviorally identical to the current driver.
[7 lines not shown]
re(4): harden re_watchdog() recovery and log controller state
Distinguish the two failure classes from the PR in a single log line
(ring indices, ISR/IMR, TXCFG, interrupt mode): lost interrupt vs
genuine DMA stall.
Bail out instead of re-initializing when the controller reads back
all-ones (fallen off the bus; reinit cannot help).
Re-assert the driver's existing ASPM-disabled policy before reinit,
since firmware/power transitions re-arming L0s/L1 is a documented
stall trigger.
Diagnostics-only on the recovered path; no fast-path change.
* Field diagnostics running on an RTL8168H production fleet; the log
format distinguishes lost-doorbell / DMA-stall / dead-controller
without a debug build.
[3 lines not shown]
re(4): recover Tx completions whose MSI was swallowed in re_intr_msi()
A Tx completion that raises a status bit between the ISR ack at the top
of re_intr_msi() and the IMR re-enable at the bottom is never re-signalled:
these controllers do not re-assert MSI for an already-set status bit
(this is why hw.re.msi_disable is a known workaround in the PR).
Re-read ISR before re-enabling; if a Tx bit is pending, ack just that bit,
reap the ring and restart the queue. Rx bits are deliberately left set so
they re-arm the interrupt normally and Rx moderation state is untouched.
Also flush the posted IMR write. Mirrors what the INTx path already
achieves via the loop in re_intr().
* MSI interrupt mode on RTL8168H under load; "missed Tx interrupts"
watchdog recoveries no longer occur.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D58278
PR: kern/166724
re(4): re-arm the Tx doorbell when re_txeof() leaves a non-empty ring
On PCIe parts a TxPoll request can be lost when packets are queued
in quick succession, leaving owned descriptors with no transfer in
progress until the watchdog fires. re_txeof() runs from the interrupt
handlers, re_tick() and re_watchdog(), so re-writing TXSTART whenever
the ring is still non-empty turns a potential 5-second stall into at
most one tick.
One register write on a path that already took an interrupt;
fast path untouched.
* Sustained bidirectional load on RTL8168H; no Tx stalls, no throughput
regression at 941 Mbps line rate.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D58277
PR: kern/166724
re(4): quiesce RTL8168G+ and reset before freeing buffers in re_stop()
The STOPREQ command written by re_stop() is not defined for
RTL8168G and later; issuing it can wedge the MAC.
Replace it on those parts with the vendor-documented sequence:
* settle delay
* bounded poll for Tx queue empty
* clear TE/RE
* then bounded poll of the MCU command register (0xD3) FIFO-empty bits.
Also reset the controller before the Rx/Tx buffer free: a controller that
has not quiesced keeps DMAing stale, still-owned descriptors pointing at
freed mbufs (use-after-free under INVARIANTS, cross-NIC mbuf corruption
reported in the PR).
Adds the RL_MCU_* register definitions.
[11 lines not shown]
aq(4): correct Atlantic 2 register access
Four Atlantic 2 register-access corrections found in bring-up.
B0 aggregate octet counters: the B0 firmware statistics interface reports
only aggregate rx/tx good octets, not the per-cast breakdown A0 and
Atlantic 1 provide, so every octet sysctl read a permanent zero while
frame counters advanced. Populate the aggregate octet fields from the B0
buffer; aq_update_hw_stats() accumulates them directly when the per-cast
octets are absent.
Drop the duplicate attach-time MCP reboot: aq_hw_mpi_create() already
reboots the A2 firmware to read its version and caps, then aq_hw_reset()
immediately rebooted it again -- a full MCP restart plus several
transaction-id-bracketed window reads, adding attach latency and a
duplicate banner. Give aq_hw_reset() a reboot flag and pass reboot=false
for A2 at attach; the load-bearing down/stop reboot (which resyncs A2 RX
DMA across ifconfig down/up) keeps reboot=true.
[14 lines not shown]
aq(4): program the Atlantic 2 multiqueue datapath
Wire up the Atlantic 2 receive datapath: the action-resolver table (ART),
multiqueue RSS, QoS, and interrupt moderation.
RX action-resolver table: Atlantic 2 replaces Atlantic 1's discrete RX
filter registers with an ART -- hardware computes a per-packet
classification tag, then walks {tag, mask, action} rows to drop, assign a
queue, or assign a TC. aq_hw_art_filter_set() installs one row under the
ART semaphore; aq_hw_init_rx_path() enables the resolver, tags L2
unicast/broadcast, installs the unicast/all-multicast and VLAN drop rows,
and assigns every 802.1p priority to TC 0 (mirroring the Atlantic 1
user-priority map, since our RX side is a single 8-ring group in TC 0).
Tag every enabled VLAN filter in the per-filter resolver-tag field -- a
register the BSD ports never write -- because the VLAN drop row matches
resolver tag 0, so without it all tagged receive was dead under VLAN
filtering. Promiscuous mode disables the drop rows rather than toggling
the Atlantic 1 promiscuous bits; all ART callers surface a semaphore
timeout consistently. The Atlantic 1 RX_TCP_RSS_HASH and TPO2
[26 lines not shown]
aq(4): add Atlantic 2 (AQC113) device support
Add support for the Marvell Atlantic 2 (AQC113/114/115/116) controllers,
a new chip generation that is not register-compatible with the Atlantic 1
parts aq(4) supports today. Adapted from the OpenBSD/NetBSD if_aq driver.
Register and device definitions (aq2_hw.h): the firmware handshake
(MIF_BOOT / MCP_HOST_REQ_INT / MIF_HOST_FINISHED), the 0x12000/0x13000
firmware interface windows, and the action-resolver table (ART) that
replaces Atlantic 1's discrete RX filters, plus the Atlantic 2 PCI device
ids and the aq_is_atlantic2() helper. Reserve a chip-feature bit
(AQ_HW_CHIP_ATLANTIC2) and add the aq_hw fields the firmware fills at boot
(ART base index, statistics interface version A0/B0). The per-VLAN-filter
resolver-tag field comes from the Linux driver; the BSD sources never
write it.
Firmware operations (aq_fwa2.c): Atlantic 2 talks to the management CPU
through the 0x12000/0x13000 register windows plus the boot handshake,
rather than Atlantic 1's mailbox in shared RAM. Implement that as a third
[31 lines not shown]
aq(4): remove dead code and tidy macros, diagnostics, and naming
Non-functional cleanup, with two diagnostic corrections.
Dead code: delete leftover commented-out AQ_DBG_ENTER/EXIT/PRINT calls
(aq_hw.c, aq_fw2x.c, aq_irq.c, aq_main.c), a commented-out aq_nic_cfg
local, the stale old-signature parameter blocks between the ring-init
declarations and their bodies (aq_ring.c), a trailing note on a live
statement, and the unused DumpHex() vendor debug helper (no callers; its
body only compiled under AQ_CFG_DEBUG_LVL > 3).
Register-write macros: parenthesize AQ_WRITE_REG_BIT's msk/shift/value
arguments so a compound argument cannot mis-bind, give AQ_HW_FLUSH() an
explicit hw parameter instead of capturing it from caller scope, and drop
the duplicate lowercase aq_hw_write_reg[_bit] aliases (converting the 43
call sites to the uppercase spelling) so there is a single form.
Diagnostics: the aq_log* family expanded through the base log macro, which
ignored its level and printed unconditionally, while the error traces
[29 lines not shown]
aq(4): harden the interrupt and MAC-statistics paths
Firmware-statistics accounting and interrupt-routing fixes.
Stats delta underflow: guard the MAC statistics delta accumulation
against counter wrap or a firmware counter reset, so a snapshot smaller
than the previous one does not underflow into a huge spurious delta.
Skip stats on a failed read: aq_update_hw_stats() ignored
aq_hw_mpi_read_stats()'s return and committed the on-stack mbox into
last_stats unconditionally. On a failed read that snapshot is garbage or
zero and poisons the delta baseline (a zeroed snapshot wipes last_stats,
so the next good read double-counts). Check the return and skip the
accumulation and the last_stats commit on failure.
Mailbox/stats separation: struct aq_hw_stats served both as the raw fw1x
MCP mailbox layout and as the driver's canonical stats snapshot, so any
field added to it would silently shift the fw1x mailbox read. Give the
fw1x mailbox its own raw layout in struct aq_hw_fw_mbox and let
[23 lines not shown]
aq(4): harden the attach, detach, and reset error paths
Correct several attach/detach/reset paths that either swallowed failures
or acted on undefined state.
MSI-X attach-failure double-free: aq_if_msix_intr_assign() freed the
per-RX-ring interrupts in its failure path and then returned an error, so
iflib's IFDI_DETACH freed the same irq structures again --
bus_teardown_intr() on a dangling tag and bus_release_resource() on an
already-released IRQ, panicking a box that should have simply failed to
attach. Let iflib own the teardown; drop the failure-path loop and the
now-dead index bookkeeping.
Detach loop bound: aq_if_detach() freed the per-ring interrupts looping
to isc_nrxqsets while indexing rx_rings[], which is sized by
rx_rings_count; index by rx_rings_count to match every other RX-ring
loop.
AQ_HW_WAIT_FOR final poll: the macro derived its result from the loop
[31 lines not shown]
aq(4): honor the kernel RSS policy and add a TX traffic-class helper
Align RX steering with the kernel RSS framework and factor out the
active-traffic-class count.
RSS key and indirection table: on an options RSS kernel the stack owns a
canonical hash key and a hash-to-bucket indirection table binding each
bucket to a CPU. aq programmed a random arc4rand() key and a plain
i % rss_qs table, so the hash it stamped in iri_flowid and the queue it
steered a flow to did not match the CPU the stack chose -- defeating RSS
affinity. Under #ifdef RSS take the key from rss_getkey() and each entry
from rss_get_indirection_to_bucket(), as e1000/ixgbe/ixl do; the non-RSS
build keeps the random key and round-robin table.
RSS hash-type policy: drop the private hw.aq.enable_rss_udp knob (RDTUN,
default on) and add aq_rss_hashconfig(), which under options RSS returns
rss_gethashconfig() and otherwise the same UDP-off default. UDP 4-tuple
hashing scatters a fragmented datagram's pieces across queues because
only the first fragment carries the L4 ports, so it is now off by default
[11 lines not shown]
