igb: Guard register dump during queue setup
The register-dump sysctl is installed before iflib allocates the queue
arrays and remains visible while they are freed. Return ENXIO outside
the queue lifetime instead of dereferencing a NULL or stale array.
Sponsored by: BBOX.io
(cherry picked from commit bcb62ec0e3d592892f0f304269ed2722d1bae75a)
pci_iov: Permit non-ARI VFs on a secondary bus
A non-zero VF device number does not always require ARI. The Intel
82576 and I350 [1] explicitly support a non-ARI layout that places VFs
on the next bus.
Check every requested VF RID and reject a non-zero device only when it
is on the PF bus. This retains the ARI guard for invalid same-bus
layouts while permitting the documented second-bus layout.
[1] Intel I350 Datasheet, sections 7.8.2.6.1.2, 9.6.4.6
Sponsored by: BBOX.io
(cherry picked from commit e795a31cb4d66368bdbe5ac7f61c0899d3ed39f8)
e1000: Recover from the 82574 PHY hang
The shared code provides e1000_check_phy_82574() to recognize a PHY
hang from saturated receive error and idle error counters, but em(4)
never calls it.
Run the check from timer driven admin work. Match Intel e1000e by
requiring two consecutive positive samples before requesting a full
iflib reset.
MFC after: 2 weeks
Sponsored by: BBOX.io
(cherry picked from commit 81d5356799a1db1701cb3f91146131c34dede413)
e1000: Sample statistics at timer cadence
Mailbox and link interrupts share iflib admin service with the periodic
timer. Mark timer-driven passes explicitly and run the hardware
statistics sweep only for those samples instead of repeating 66 PF MMIO
reads for every VF mailbox message.
DTrace on the I350 DUT measured the PF sweep at about 79 us on average.
The normal hz/2 timer continues to extend clear-on-read counters
safely; exported counters may trail hardware by up to 500 ms.
Sponsored by: BBOX.io
(cherry picked from commit d2cd0b57532ba35fe39744a60d53b90e6f13b5e4)
igc: Propagate hardware initialization failures
The reset helper discards igc_reset_hw and igc_init_hw errors. Runtime
initialization then continues programming rings and filters, and iflib
publishes the interface as running even though the controller did not
reach a usable state. Initial attach similarly continues into NVM and
MAC setup after a failed reset.
Return errors from the reset helper. Fail attach when the controller
cannot be reset or initialized, and report runtime failures through
iflib_init_failed() so iflib leaves the interface stopped. Also stop
register accesses and report the error when a stop path reset fails. A
later successful initialization completes pending fatal error cleanup
and re-arms FER.
Cache a requested MAC address before reset, but let init_hw program RAR0
after reset succeeds. Let iflib perform its normal attach-post failure
cleanup instead of releasing the same driver resources from both layers,
and make queue cleanup idempotent.
[4 lines not shown]
e1000: Propagate hardware initialization failures
The reset helper discards reset_hw and init_hw errors. Runtime
initialization then continues programming rings and filters, and iflib
publishes the interface as running even though the controller did not
reach a usable state. Initial attach similarly continues into NVM and
MAC setup after a failed reset.
Return errors from the reset helper. Fail attach when the controller
cannot be reset or initialized, and report runtime failures through
iflib_init_failed() so iflib leaves the interface stopped. Also stop
register accesses and report the error when a stop-path reset fails.
Sponsored by: BBOX.io
(cherry picked from commit 41a0f7a0a447ef2092faadb35f6d4c3f80c088bf)
igb: Reinitialize through iflib after media changes
A media change can require a complete controller reset. Resetting the
controller directly from the admin task leaves iflib rings, filters, and
interface state programmed for the pre-reset controller.
Request an iflib reset for every media change. This already was done
when SR-IOV was active; use the same lifecycle for the ordinary PF case.
Sponsored by: BBOX.io
(cherry picked from commit 6248e7de9d6c0f14294afc5792170a68de6a3b53)
e1000: Report 82575 memory ECC errors
82575 exposes clear-on-read, saturating counters for corrected and
uncorrected errors in the packet buffer and the receive and transmit
descriptor handlers.
Sample all three status registers together from the regular hardware
statistics update. When an unrecoverable event interrupts first, count
the values captured by the interrupt filter so the clear-on-read status
is not lost before the admin task handles it.
Expose packet buffer and descriptor handler counters under the existing
memory_errors sysctl node.
Hardware validation used an 82575EB revision 2 and the documented PBEEI,
RDHEEI, and TDHEEI injectors. Correctable and uncorrectable TX/RX
packet-buffer errors and receive/transmit descriptor-handler errors
advanced the corresponding counters. The controls and accounting
survived repeated recovery resets and an ordinary interface down/up.
[4 lines not shown]
e1000: Recover from 82575 memory errors
82575 protects its packet buffer and receive and transmit descriptor
handlers with ECC. Correctable errors are repaired in hardware.
Packet data errors are contained to the affected packet, while the
native RX_PBUR, TX_PBUR, RX_DHER, and TX_DHER interrupt causes report
unrecoverable packet buffer or descriptor handler state. The affected
traffic direction remains stopped until software resets the port.
Enable the three ECC blocks and hardware memory error reaction after
queue and filter initialization. Capture the clear-on-read status
registers in the interrupt filter and keep all four native causes masked
while the iflib admin task owns the event.
Request port reinitialization for every native PBUR or DHER cause.
Packet data errors that do not raise a native cause remain count-only
and do not disrupt the port. The captured status registers provide
diagnostics and accounting but do not independently initiate recovery.
[10 lines not shown]
e1000: Report 82576 memory ECC errors
82576 exposes clear-on-read corrected error counters for RX, TX,
switch, IPsec, descriptor-handler, PCIe retry, PCIe write, and
MSI-X memories. The packet and descriptor memories also count
uncorrectable errors.
Sample each status register exactly once from the regular hardware
statistics update and immediately before handling a memory-error
interrupt. Group the counters by packet buffer, descriptor handler,
and PCIe region. Skip the absent IPsec block on 82576NS.
PRBESTS and PMSIXESTS are shared by both LAN ports. Attribute an
indication to whichever attached port samples the clear-on-read
register first so it is not counted twice.
Hardware validation used an 82576EB revision 1. All nine implemented
status registers reported their ECC-enable bit set. The sysctl
counters remained clear across interface lifecycle, two-stream
[11 lines not shown]
e1000: Recover from 82576 memory errors
82576 reports fatal and non-fatal internal memory errors through
ICR.FER and ICR.NFER and identifies the affected memory in its
native PEIND layout. Fatal errors can stop transmit, receive, or
both until software resets and reinitializes the port.
Enable the controller-wide parity detector and implemented PEINDM
reaction bits after hardware initialization, while preserving unrelated
register state and omitting the absent IPsec memories on 82576NS.
Enable both interrupt causes and capture the read-clear PEIND register
in the interrupt filter.
Keep the causes masked while the iflib admin task owns the event.
Acknowledge non-fatal packet data errors without disrupting the port.
Request normal port reinitialization for FER, a fatal PEIND source, or
the memory hang indication. Do not apply the later I210/I350 register
layout or their special PCIe parity reset order.
[15 lines not shown]
e1000: Report corrected I350 ECC errors
I350 does not interrupt for corrected internal ECC errors. Instead,
the PCIe, DMA, packet buffer, loopback, and management memories expose
sticky status bits in their region-specific status registers.
Sample those bits with the regular hardware statistics update, preserve
the RX and TX packet buffer ECC enable state while clearing RW1C
indications, and expose counters grouped by memory region. Each counter
records observed indication bits rather than exact error counts because
repeated corrections between samples collapse into one sticky bit.
On an I350 (8086:1521 revision 1), the ECC enables remained set. All
corrected-error status registers remained clear across boot, interface
down/up, three FER recovery resets, and bidirectional line-rate traffic.
The device has no documented corrected error injector. Therefore, the
per-region paths were validated against the register definitions rather
than an injected SRAM error.
[3 lines not shown]
e1000: Recover from I350 memory errors
I350 reports uncorrectable internal memory errors through ICR.FER and
identifies the affected region in PEIND. Depending on the region and
memory, hardware stops transmit, receive, or all PCIe and DMA traffic
until the port is reset and reinitialized.
Enable FER and all regional indication masks. Capture the read-clear
status in the interrupt filter.
Record the fatal PCIe, DMA, and LAN status registers, keep FER masked
while recovery is pending, and expose per-region indication counters.
Use the datasheet required port reset before master disable order for
PCIe parity errors. Reset for PCIe, DMA, and traffic-affecting LAN
errors. Statistics and VF-mailbox parity errors only require their
status to be discarded and cleared; management-memory recovery remains
under firmware control.
Validated on an I350 (8086:1521 revision 1). Three software-set FER
[8 lines not shown]
e1000: Report corrected I210 and I211 ECC errors
I210 and I211 do not interrupt for corrected internal ECC errors.
Instead, the DMA packet-buffer and PCIe memories expose sticky status
bits in PBECCSTS and PCIEECCSTS.
Sample these bits with the regular hardware statistics update, preserve
the I210/I211 PBECCSTS enable state while clearing its RW1C indication,
and expose separate counters for the DMA packet buffer, PCIe transmit
data, and PCIe retry buffer.
The counters represent observed indications rather than exact error
counts because multiple corrections between samples collapse into one
sticky status bit.
Hardware validation used an I210 revision 3. Unlike I225 and I226, the
published I210/I211 register definitions do not expose self-clearing
injectors for these corrected ECC memories. The three counter sysctls
were present and remained zero under line-rate traffic and three fatal
[6 lines not shown]
e1000: Recover from I210 and I211 memory errors
I210 and I211 report uncorrectable internal memory errors through
ICR.FER and identify the affected region in PEIND. Depending on the
region, hardware stops transmit or all PCIe and DMA traffic until the
port is reset and reinitialized.
Enable FER and all regional indication masks. Discard indication state
left by firmware before enabling reactions, capture the read-clear
status in the interrupt filter, and keep the cause masked while recovery
is pending. Report the affected regions and expose per-region
indication counters. Management-only errors remain under firmware
control.
PCIe region parity errors require a different recovery order from the
normal reset path. Assert the port-local CTRL.RST bit, wait at least
3 ms, verify reset completion, disable master requests, clear
PCIEERRSTS, and then enter normal port reinitialization. Do not use the
device-wide CTRL.DEV_RST sequence used by I225 and I226.
[16 lines not shown]
ixgbe: Add missing mailbox API 1.6 definition
The SR-IOV status change reports mailbox API 1.6 but omitted its enum
definition, leaving main unable to compile.
API 1.6 is an established ixgbe mailbox wire revision. Add it at the
end of the revision enum, before the unknown sentinel as required by
the stable numbering contract. Naming the revision does not enable
negotiation or operations which will come with the E610 support.
Reported by: Herbert J. Skuhra <herbert at gojira.at>
(cherry picked from commit 6a1703c112ff2904feb39288f0746c8f0a04f938)
ixgbe: Add missing mailbox API 1.6 definition
The SR-IOV status change reports mailbox API 1.6 but omitted its enum
definition, leaving main unable to compile.
API 1.6 is an established ixgbe mailbox wire revision. Add it at the
end of the revision enum, before the unknown sentinel as required by
the stable numbering contract. Naming the revision does not enable
negotiation or operations which will come with the E610 support.
Reported by: Herbert J. Skuhra <herbert at gojira.at>
(cherry picked from commit 6a1703c112ff2904feb39288f0746c8f0a04f938)
e1000: Report PCH packet buffer ECC statistics
PCH packet buffer ECC status contains read-clear byte counters for
corrected and uncorrected errors. Sample them with the regular
hardware statistics update and account for the snapshot captured by
the fatal error interrupt path.
Expose the counters and the number of reset worthy interrupt
indications under dev.em.N.memory_errors. Keeping the reset counter
separate also preserves evidence when another status reader wins the
read-clear race.
Hardware validation used an I219-LM. Three documented ICS.ECCER
injections advanced fatal_resets from zero to three, exactly once per
reset. corrected_packet_buffer and uncorrected_packet_buffer remained
zero, as expected because ICS does not inject a memory error or alter
PBECCSTS.
Sponsored by: BBOX.io
[2 lines not shown]
e1000: Recover from PCH packet buffer ECC errors
PCH LAN controllers beginning with I217 report uncorrectable
packet buffer ECC errors through ICR.ECCER. Descriptor memory errors
stop the MAC and require a reset before traffic can resume.
Enable the interrupt on the PCH generations whose shared code setup
enables packet buffer ECC. Capture the read-clear PBECCSTS value in
the interrupt filter, mask ECCER while recovery is pending, and request
an iflib reset from the admin task. Reenable the cause only after
hardware initialization succeeds.
Hardware validation used an I219-LM and the documented ICS.ECCER bit to
generate the fatal interrupt. This synthesizes the interrupt cause but
does not corrupt packet buffer memory or alter its ECC byte counters.
Three injections in one boot each requested one reset and recovered
traffic without a panic or watchdog. IMS.ECCER and PBECCSTS.ECC_ENABLE
remained set after every reset.
[4 lines not shown]
ixgbe: Enable per-pool RSS on X550 family devices
X550 family devices provide a separate RSS key, redirection table,
and MRQC register for every VMDq pool. With SR-IOV enabled, the
driver continued programming only the global RSS state and never
selected MRQC.MULTIPLE_RSS. VF-local RSS programming was therefore
ineffective.
Enable multiple-RSS mode for X550, X552, X553, and E610. Initialize
the PF pool's 64-entry key, redirection table, and RSS hash controls.
Leave each VF pool untouched so its driver retains ownership of its
RSS key and mapping.
E610 folds IPv6 extension-header traffic into its base RSS selectors
and reserves the legacy EX selector bits. Translate those requested
hash types rather than programming reserved bits.
With two E610 VFs active and four PF queue sets, eight fixed TCP flows
distributed across all four PF receive queues.
[8 lines not shown]
ixv: Preserve statistics across resets
The VF statistics registers are free running and are not cleared on
read. The existing code records attach time bases and pre-reset totals,
but never uses either when publishing counters. It instead replaces
the low hardware bits directly, so counters can inherit pre-attach
traffic or jump backward after a reset.
Accumulate modular 32- and 36-bit deltas, following DPDK, while keeping
the software totals across planned resets. Establish a fresh hardware
baseline after each successful reset and invalidate the sampling epoch
when mailbox state is lost. Detect unsolicited PF resets explicitly so
a reset while link is down cannot be mistaken for counter wrap.
Remove the unused base and saved-reset bookkeeping.
On E610, packet and octet counters remained monotonic across a VF FLR
and a PF down/up cycle. Traffic after each reset advanced both RX and
TX counters.
[4 lines not shown]
ixgbe: Enable SR-IOV on E610 PFs
E610 inherits the X550-family virtualization registers, anti-spoofing
controls, and malicious-driver operations, but the frontend does not
advertise SR-IOV and cannot negotiate the mailbox revision needed by
E610 VFs.
Initialize the X550-family PF/VF mailbox registers for E610 and use
PFVFLREC for its VF reset events, following DPDK shared ixgbe code.
Advertise the E610 SR-IOV capability, accept API 1.6 only on E610, carry
the existing xcast and queue operations forward to that revision, and
return the cached physical link speed and state with the three-dword
E610 operation. Unsupported RSS and optional feature requests continue
to receive explicit failures.
SR-IOV activation also enables the existing X550-derived per-pool MDD
recovery path on E610. Document the expanded protection and link-state
coverage.
[11 lines not shown]
ixv: Support E610 mailbox API 1.6
E610 VFs no longer report the actual PF link state and speed through
VFLINKS. They can consequently report the default 10 Gb/s speed even
when the physical link uses another rate.
Negotiate mailbox API 1.6 on E610 and request the PF link state with its
three-dword operation. Retain VFLINKS as the fallback when an older PF
rejects API 1.6. Permit API 1.6 in the inherited xcast and queue
discovery helpers so negotiating the newer revision does not disable
existing operations.
Use GET_QUEUES to replace E610's one-queue fallback with the grant from
the PF. The common path continues to use one iflib queue set per data
MSI-X vector and caps the result at two queue pairs.
Preserve mailbox transport errors so the driver can distinguish an
explicit PF NACK from a transient timeout. A NACK means clear-to-send
state was lost and requires a VF reset. Preserve the last confirmed
[22 lines not shown]
pci: Export pcie_flr_supported()
Move the capability and quirk checks used by pcie_flr() into a public
side effect free helper. This lets callers determine whether an FLR
can be attempted before quiescing a device or saving state.
The helper considers the advertised PCIe FLR capability and both the
enable and disable FLR quirks.
Sponsored by: BBOX.io
(cherry picked from commit 5b48968c1a57bd1a7f086d7e09add59afa158340)
ixgbe: Enable per-pool RSS on X550 family devices
X550 family devices provide a separate RSS key, redirection table,
and MRQC register for every VMDq pool. With SR-IOV enabled, the
driver continued programming only the global RSS state and never
selected MRQC.MULTIPLE_RSS. VF-local RSS programming was therefore
ineffective.
Enable multiple-RSS mode for X550, X552, X553, and E610. Initialize
the PF pool's 64-entry key, redirection table, and RSS hash controls.
Leave each VF pool untouched so its driver retains ownership of its
RSS key and mapping.
E610 folds IPv6 extension-header traffic into its base RSS selectors
and reserves the legacy EX selector bits. Translate those requested
hash types rather than programming reserved bits.
With two E610 VFs active and four PF queue sets, eight fixed TCP flows
distributed across all four PF receive queues.
[8 lines not shown]
