aq(4): harden the attach, detach, and reset error paths
Correct several attach/detach/reset paths that either swallowed failures
or acted on undefined state.
MSI-X attach-failure double-free: aq_if_msix_intr_assign() freed the
per-RX-ring interrupts in its failure path and then returned an error, so
iflib's IFDI_DETACH freed the same irq structures again --
bus_teardown_intr() on a dangling tag and bus_release_resource() on an
already-released IRQ, panicking a box that should have simply failed to
attach. Let iflib own the teardown; drop the failure-path loop and the
now-dead index bookkeeping.
Detach loop bound: aq_if_detach() freed the per-ring interrupts looping
to isc_nrxqsets while indexing rx_rings[], which is sized by
rx_rings_count; index by rx_rings_count to match every other RX-ring
loop.
AQ_HW_WAIT_FOR final poll: the macro derived its result from the loop
[33 lines not shown]
aq(4): harden the interrupt and MAC-statistics paths
Firmware-statistics accounting and interrupt-routing fixes.
Stats delta underflow: guard the MAC statistics delta accumulation
against counter wrap or a firmware counter reset, so a snapshot smaller
than the previous one does not underflow into a huge spurious delta.
Skip stats on a failed read: aq_update_hw_stats() ignored
aq_hw_mpi_read_stats()'s return and committed the on-stack mbox into
last_stats unconditionally. On a failed read that snapshot is garbage or
zero and poisons the delta baseline (a zeroed snapshot wipes last_stats,
so the next good read double-counts). Check the return and skip the
accumulation and the last_stats commit on failure.
Mailbox/stats separation: struct aq_hw_stats served both as the raw fw1x
MCP mailbox layout and as the driver's canonical stats snapshot, so any
field added to it would silently shift the fw1x mailbox read. Give the
fw1x mailbox its own raw layout in struct aq_hw_fw_mbox and let
[25 lines not shown]
aq(4): honor the kernel RSS policy and add a TX traffic-class helper
Align RX steering with the kernel RSS framework and factor out the
active-traffic-class count.
RSS key and indirection table: on an options RSS kernel the stack owns a
canonical hash key and a hash-to-bucket indirection table binding each
bucket to a CPU. aq programmed a random arc4rand() key and a plain
i % rss_qs table, so the hash it stamped in iri_flowid and the queue it
steered a flow to did not match the CPU the stack chose -- defeating RSS
affinity. Under #ifdef RSS take the key from rss_getkey() and each entry
from rss_get_indirection_to_bucket(), as e1000/ixgbe/ixl do; the non-RSS
build keeps the random key and round-robin table.
RSS hash-type policy: drop the private hw.aq.enable_rss_udp knob (RDTUN,
default on) and add aq_rss_hashconfig(), which under options RSS returns
rss_gethashconfig() and otherwise the same UDP-off default. UDP 4-tuple
hashing scatters a fragmented datagram's pieces across queues because
only the first fragment carries the L4 ports, so it is now off by default
[13 lines not shown]
aq(4): remove dead code and tidy macros, diagnostics, and naming
Non-functional cleanup, with two diagnostic corrections.
Dead code: delete leftover commented-out AQ_DBG_ENTER/EXIT/PRINT calls
(aq_hw.c, aq_fw2x.c, aq_irq.c, aq_main.c), a commented-out aq_nic_cfg
local, the stale old-signature parameter blocks between the ring-init
declarations and their bodies (aq_ring.c), a trailing note on a live
statement, and the unused DumpHex() vendor debug helper (no callers; its
body only compiled under AQ_CFG_DEBUG_LVL > 3).
Register-write macros: parenthesize AQ_WRITE_REG_BIT's msk/shift/value
arguments so a compound argument cannot mis-bind, give AQ_HW_FLUSH() an
explicit hw parameter instead of capturing it from caller scope, and drop
the duplicate lowercase aq_hw_write_reg[_bit] aliases (converting the 43
call sites to the uppercase spelling) so there is a single form.
Diagnostics: the aq_log* family expanded through the base log macro, which
ignored its level and printed unconditionally, while the error traces
[31 lines not shown]
aq(4): expand and correct offloads, fix VLAN/multicast filtering
Advertise the offloads the hardware already performs, correct the TX
descriptor's L3 family selection, and correct the VLAN and multicast
receive-filter paths.
Offloads: advertise IFCAP_HWCSUM_IPV6 (adding CSUM_IP6_TCP/UDP/TSO to
isc_tx_csum_flags) and IFCAP_VLAN_HWTSO, and enable the RX outer
(S-VLAN) tag parse mode in aq_hw_offload_set().
TX descriptor L3 family: aq_setup_offloads() derived tx_desc_cmd_ipv4
from CSUM_IP|CSUM_TSO, but CSUM_TSO is (CSUM_IP_TSO|CSUM_IP6_TSO) and
tcp_output() sets both bits without regard to address family, so an
IPv6 TSO frame matched on CSUM_IP_TSO and went out with the IPv4
header-checksum command set on a frame that carries no IPv4 header.
The checksum flags cannot distinguish the family; key the bit off
IPI_TX_IPV4 instead, which iflib derives from the parsed ethertype,
as the IPI_TX_INTR test below it already does. Plain IPv6 checksum
offload was unaffected, as CSUM_IP6_TCP alone never matched the mask.
[27 lines not shown]
aq(4): drop errored RX frames instead of resetting the interface
aq_isc_rxd_pkt_get() returned EBADMSG when a receive descriptor's
MAC/receive-error bit (rx_stat bit 0) was set. iflib treats any error
from isc_rxd_pkt_get() as a fatal ring fault and answers with
IFC_DO_RESET -- a full interface reinitialization. A per-frame receive
error is not a ring fault: on a marginal link or cable the Atlantic
delivers errored frames continuously, so each one triggered another
reset and the interface reset-stormed itself into carrying no traffic
instead of merely dropping the bad frames.
The Atlantic delivers errored frames to the host by design (Linux drops
them in software via buff->is_error), and iflib offers no per-frame
error return that isn't a reset. Follow the vmxnet3 model: on a receive
error zero the fragment lengths and return success. iflib then discards
the packet (assemble_segments() excludes zero-length fragments) while
still recycling the descriptors through the refill path -- no reset.
Also drop frames flagged with an RX-DMA fault (rdm_err), not just the
MAC-error bit; and keep iri_len non-zero on that drop path, since iflib
[11 lines not shown]
LinuxKPI: 802.11: add != NULL check in ieee80211_tx_status_ext()
There seems to be another possible race with net80211 state machine
changing the bss from under us (another lvif_bss_synched case).
Just do the != NULL check to avoid a NULL pointer deref in
ieee80211_ratectl_rate().
(bz extended the original comment and wrote the commit message).
Sponosred by: The FreeBSD Foundation (commit)
PR: 297184
MFC after: 3 days
ath10k: remove some early FreeBSD-specific debugging
The extra DELAY seems to no longer be needed and the dump_stack()
is definitively a problem now. Remove all this.
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
aq(4): add a runtime dev.aq.N.debug trace control
The trace_* family (trace/trace_error/trace_warn/trace_detail, used in the
F/W and init/config paths) was gated behind the compile-time
AQ_CFG_DEBUG_LVL, which is 0, so the dbg_level_/dbg_categories_ runtime
variables were dead and tracing could only be enabled by recompiling.
Decouple trace_base_ from AQ_CFG_DEBUG_LVL so it is always compiled and
gated purely at runtime on dbg_level_/dbg_categories_, make those two
variables writable (no longer const, default level 0 = off), and expose
them as dev.aq.N.debug (verbosity) and dev.aq.N.debug_categories
(subsystem mask) sysctls.
The datapath-heavy AQ_DBG_ENTER/PRINT/DUMP macros and the trace_aq_*_descr
descriptor dumps stay behind AQ_CFG_DEBUG_LVL (still 0), so the per-packet
paths are untouched -- trace_* is only used off the datapath. The two
variables are global (the trace macros reference them directly), so the
per-device sysctls share one backing store, which is fine for a debug
knob.
[10 lines not shown]
aq(4): adopt native FreeBSD errno convention
Convert the driver's internal error-handling chain from the Linux
negative-errno convention to FreeBSD positive errno everywhere.
- All `return (-EXXX)` become `return (EXXX)`, `int err = -EXXX` loses
the sign, and `if (err < 0)` checks become `if (err != 0)` across
aq_fw.c, aq_fw1x.c, aq_fw2x.c and aq_hw.c.
- mac_soft_reset_flb_ returns ETIMEDOUT/0 instead of a bool so it
matches its RBL sibling.
- The ETIME and EOK aliases in aq_common.h are removed; all sites use
ETIMEDOUT and 0 directly, and the `rc = -rc` sign flips in
aq_if_attach_pre are dropped.
Turn AQ_HW_WAIT_FOR into a statement expression evaluating to 0 on
success or ETIMEDOUT on timeout, assigned explicitly at all seven call
sites, instead of silently assigning ETIMEDOUT to a variable named err
in the caller scope. A statement expression rather than an inline
function because every call must re-evaluate its condition each
[26 lines not shown]
aq(4): modernize and de-Linuxify the vendor driver
Dead-code removal, device_printf(9) logging, style(9) de-Linuxification,
const F/W-ops tables, and readability cleanups. No change for valid
traffic.
Dead code and logging:
- Remove the sub-gigabit TSO-masking block in the link-state ISR: it
cleared IFCAP_TSO from the static isc_capabilities record (read only at
attach / SIOCSIFCAP, never on the datapath), so it never gated TSO and
only corrupted the validation mask. The Atlantic has no sub-gigabit TSO
erratum.
- Tidy the RX buffer-size handling: drop the dead switch(MCLBYTES) in
aq_if_rx_queues_alloc, rename rx_max_frame_size -> rx_buf_size, and bound
the per-fragment length from the wb.pkt_len writeback (EBADMSG on
underflow or a final fragment longer than the RX buffer).
- Drop every __FreeBSD__/__FreeBSD_version branch (FreeBSD 14.0 baseline);
the pre-13 arms used pre-opaque-if_t APIs since removed and one never
built.
[33 lines not shown]
aq(4): Fix RSS indirection table OOB write and queue distribution
Two related fixes to `aq(4)`'s RSS indirection table handling:
1. Fix an out-of-bounds stack write in `aq_hw_rss_set()`. RSS table entries are 3 bits (8 queues max), but with more than 8 RX rings `rss_table[]` holds larger values; the 32-bit write then spills one `uint16_t` past `bitary[]` and corrupts the stack, so the NIC never links or the kernel panics. Mask each value to 3 bits and pack 16 bits at a time to keep the write in bounds.
2. Build the indirection table in `aq_if_attach_post()` with a modulo over `min(rx_rings_count, HW_ATL_RSS_INDIRECTION_QUEUES_MAX)` instead of `i & (rx_rings_count - 1)`, which assumed a power-of-two ring count.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D57240
(cherry picked from commit 57f5252ff8a17aa837f677638cce5885b8a5fb3b)
aq(4): take F/W statistics off the iflib core lock (kick-and-read)
The once-per-second statistics refresh ran the whole F/W-mailbox
transaction under iflib's CTX (sx) lock: fw2x_get_stats toggled the MPI
STATISTICS control bit and busy-polled the state register for the
acknowledgement (up to ~25 ms) before downloading the counters, so a slow
F/W response blocked datapath reconfigure / ioctls for the duration.
The per-cast and error counters have no direct-register source -- the
reference Linux atlantic driver and our port both read them out of the
F/W mailbox, and the MSM registers the chip exposes are never used for the
periodic counters. So rather than poll, adopt the kick-and-read shape the
iflib peer with the same constraint uses (vmxnet3): consume the snapshot
the F/W produced for the *previous* request, then toggle the bit to
request the next one -- no wait. The F/W finished that previous refresh
~1 s ago, so the download needs no poll, and the toggle write stays
serialized against set_mode by the CTX lock exactly as before. This
removes the 25 ms poll (and the toggle_mpi_ctrl_and_wait_ helper) from
under the lock; only the fast 16-dword download remains.
[12 lines not shown]
aq(4): interrupt model and queue-count correctness
Rework the MSI-X and queue-count handling to use the standard iflib
interrupt model and to keep every ring serviced.
- Cap isc_n{tx,rx}qsets_max at the RSS indirection-table size
(HW_ATL_RSS_INDIRECTION_QUEUES_MAX, 8) instead of HW_ATL_B0_RINGS_MAX.
RSS only steers RX traffic to eight rings, so on hosts with more CPUs
the surplus TX rings never make progress: iflib flowid-steers TCP
flows across every TX ring, and a flow landing on a surplus ring has
its segments queued but never transmitted, hanging the connection.
- Add a TX-specific ifdi_tx_queue_intr_enable that reads
tx_rings[txqid]->msix. It was wired to the RX handler, which indexes
rx_rings[] with the qid; safe only while tx_rings_count ==
rx_rings_count, otherwise the lookup walks past rx_rings[] and feeds a
garbage msix value into the IRQ mask register.
- Fix three MSI-X / admin-IRQ bugs: the TX softirq was attached to
[16 lines not shown]
aq(4): RX/TX and HW-path correctness and hardening
Independent correctness fixes, plus robustness against a non-responding
device, malformed descriptor writeback, and torn MMIO reads, and the move
to the FreeBSD bus_space(9) register abstraction.
Correctness:
- aq_hw_ver_match returned true if any of major/minor/build was >=
expected; compare lexicographically so e.g. 2.0.1 is correctly seen as
older than 2.1.0.
- The VLAN hardware-filter iteration used the vlan tag directly as the
bitstring index; use vlan_tag + 1 so the active-VLAN bookkeeping lines
up with the table.
- aq_initmedia only registered IFM_AUTO in full-duplex/pause variants, so
a bare "ifconfig aq0 media autoselect" matched no entry and returned
ENXIO. Add the bare IFM_ETHER|IFM_AUTO entry, matching ix/em/igc/ixv.
- Convert the per-ring diagnostic counters to counter(9): per-CPU,
tear-free, no atomics on the increment path, fixing a data race and a
32-bit torn read against the locklessly-read sysctls. Drop three
[32 lines not shown]
aq(4): enable jumbo frames, software LRO, and suspend/resume
- Configure the RX buffer size from the interface MTU and enable jumbo
frames up to 9000 bytes, replacing the fixed standard-frame setup.
- Advertise IFCAP_LRO so iflib coalesces received TCP segments with its
software tcp_lro(9), like every other in-tree iflib driver
(ix/igc/em/vmxnet3); aq does no hardware LRO. iflib builds the
per-RX-queue LRO context unconditionally, so the capability bit is all
that is required; enabled by default via isc_capenable, toggle at
runtime with ifconfig.
- Add suspend/shutdown/resume handlers, replacing the unimplemented-
function placeholders. aq_if_shutdown/aq_if_suspend stop the interface
and deinitialize the hardware; aq_if_resume re-resets the F/W, re-reads
the mailbox address and re-selects fw_ops via aq_hw_mpi_create() before
iflib re-inits, because the runtime init path (aq_hw_init) reuses the
cached mailbox/fw_ops and a D3 power cycle can clear them. iflib calls
IFDI_RESUME unconditionally, so this also covers resuming while the
[6 lines not shown]
igbv: Fence DMA while sanitizing retained queues
The 82576 and I350 retain VF queue enable and DMA address state
across VFLR. iflib enables PCI bus mastering before driver attach, so
stale state left by a previous owner can otherwise issue DMA before
igbvf has completed its first reset and queue sanitization.
Disable PCI bus mastering immediately after mapping the VF BAR. Keep
it disabled until reset and queue sanitization succeed, verify both
disable and enable through PCI command-register readback, and wait for
pending transactions before treating the fence as complete.
Resanitize on stop before iflib releases queue mappings.
The sanitizer and recovery were exercised on I350 and 82576 VFs. Forced
queue-disable failure left the VF down, and a later administrative
down/up recovered it; successful I350 VFs passed bidirectional traffic
with no errors or drops.
Sponsored by: BBOX.io
e1000: Configure PCH low-power link modes for suspend
The PCH suspend path kept a wake link fully powered and did not restore
the negotiated EEE modes after its stop-time reset. Intel provides the
ULP entry and exit machinery in the shared code, but FreeBSD did not
invoke its Sx policy.
Enter ULP on LPT and newer PCH controllers when wake is armed without
directed-unicast, multicast, or broadcast filters, which ULP cannot
preserve. For a link retained by host wake or management, restore the
100BASE-TX and 1000BASE-T LPI controls selected by the local
advertisement and the cached link-partner ability.
Keep these power reductions best-effort: wake filters and PME are
already configured independently, and a ULP or EEE failure is logged
without converting an optional power optimization into a suspend
failure. The existing PCH resume workaround forcibly exits ULP and
clears automatic Sx LPI state before normal initialization.
[10 lines not shown]
tcp md5: improve handling of tcps_sig_err_sigopt counter
For non listening TCP endpoints, increment the tcps_sig_err_sigopt
counter when TCP MD5 is not enabled in the TCP connection, but a
segment containing a TCP MD5 option is received.
Also increment the counter when using the RACK or BBR stack.
Reported by: Hannes Elfert
Reviewed by: rrs
MFC after: 1 week
MFC to: stable/14
MFC to: stable/15
Differential Revision: https://reviews.freebsd.org/D59249
e1000: Rework Wake-on-LAN policy and programming
The driver used the NVM APME default as both the hardware-support
decision and the mutable filter mask. Consequently, an NVM-disabled
but capable port did not advertise wake support, disabling a wake mode
once could keep it disabled across later suspends, and directed-unicast
wake could never be selected.
Require the PCI power management capability to report D3hot PME support
before advertising or arming wake. A PM capability alone does not mean
the function can signal PME from the state used during system sleep.
Separate the board and port capability matrix from the NVM-selected
magic packet default. Read the proper per function NVM word on igb
controllers, cover the newer PCH generations, and retain the documented
legacy, multi-port, and OEM restrictions. Decode the distinct APM
Enable locations used by 82544, 82541EI/82547EI, and the later 8254x
parts. Do not advertise wake on the 82541ER, whose power-management
logic cannot assert PME for wake events. For I210/I211 internal iNVM,
[92 lines not shown]
cat: add -A, -E and -T flags
Support cat -A, -E and -T, which are commonly used by Linux shell
scripts. -E prints a "$" at the end of each line, -T renders tabs
as ^I, and -A is equivalent to -vET.
MFC After: 1 week
Discussed with: jrtc27
Reviewed by: jrtc27, ziaee
Differential Revision: https://reviews.freebsd.org/D59250
pf: fix securelevel off-by-one
cmd_securelevel is the securelevel at which the call should be denied.
pf (write) calls should be denied at level 3 or up (not at 2 or up as it
was), so increment these all by one.
PR: 296838
MFC after: 4 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D58377
(cherry picked from commit d13dffa150d17dc239e164ea42ddab91e6fab466)
