ena: Put taskqueues into correct domain if !RSS
When compiled without 'options RSS', the ena driver created taskqueues
using taskqueue_start_threads_cpuset passing a mask value of NULL,
both in the ena_setup_tx_resources path (for enqueues) and in the
ena_create_io_queues path (for the completion-processing).
In the default configuration, on most EC2 instances, this results in
taskqueues running in the right NUMA domain, but only by accident; in
non-default configurations (e.g. with with multiple EBS volumes
attached and associated NVMe taskqueues) the taskqueues may land in
the wrong NUMA domain even on instance types where the one-EBS-one-ENA
case produces the desired results.
Set (struct ena_que)->domain and use that to inform the choice of CPU
sets. On a c8gn.48xlarge EC2 instance this doubles throughput on a
32-TCP-stream benchmark.
Reviewed by: akiyano
[5 lines not shown]
ena: Put taskqueues into correct domain if !RSS
When compiled without 'options RSS', the ena driver created taskqueues
using taskqueue_start_threads_cpuset passing a mask value of NULL,
both in the ena_setup_tx_resources path (for enqueues) and in the
ena_create_io_queues path (for the completion-processing).
In the default configuration, on most EC2 instances, this results in
taskqueues running in the right NUMA domain, but only by accident; in
non-default configurations (e.g. with with multiple EBS volumes
attached and associated NVMe taskqueues) the taskqueues may land in
the wrong NUMA domain even on instance types where the one-EBS-one-ENA
case produces the desired results.
Set (struct ena_que)->domain and use that to inform the choice of CPU
sets. On a c8gn.48xlarge EC2 instance this doubles throughput on a
32-TCP-stream benchmark.
Reviewed by: akiyano
[5 lines not shown]
ena: Report RX overrun errors
Extract rx_overruns from the keep alive descriptor reported by
the device and expose it via sysctl hw stats.
RX overrun errors occur when a packet arrives but there are not
enough free buffers in the RX ring to receive it.
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D56640
(cherry picked from commit e3f4a63af63bea70bc86b6c790b14aa5ee99fcd0)
ena: Budget rx descriptors, not packets
We had ENA_RX_BUDGET = 256 in order to allow up to 256 received
packets to be processed before we do other cleanups (handling tx
packets and, critically, refilling the rx buffer ring). Since the
ring holds 1024 buffers by default, this was fine for normal packets:
We refill the ring when it falls below 7/8 full, and even with a large
burst of incoming packets allowing it to fall by another 1/4 before we
consider refilling the ring still leaves it at 7/8 - 1/4 = 5/8 full.
With jumbos, the story is different: A 9k jumbo (as is used by default
within the EC2 network) consumes 3 descriptors, so a single rx cleanup
pass can consume 3/4 of the default-sized rx ring; if the rx buffer
ring wasn't completely full before a packet burst arrives, this puts
us perilously close to running out of rx buffers.
This precise failure mode has been observed on some EC2 instance types
within a Cluster Placement Group, resulting in the nominal 10 Gbps
single-flow throughput between instances dropping to ~100 Mbps as a
[21 lines not shown]
ena: Adjust ena_[rt]x_cleanup to return bool
The ena_[rt]x_cleanup functions are limited internally to a maximum
number of packets; this ensures that TX doesn't starve RX (or vice
versa) and also attempts to ensure that we get a chance to refill
the RX buffer ring before the device runs out of buffers and starts
dropping packets.
Historically these functions have returned the number of packets which
they processed which ena_cleanup compares to their respective budgets
to decide whether to reinvoke them. This is unnecessary complication;
since the precise number of packets processed is never used, adjust
the APIs of those functions to return a bool indicating if they want
to be reinvoked (aka if they hit their limits).
Since ena_tx_cleanup now only uses work_done if diagnostics are
enabled (ena_log_io macros to nothing otherwise) eliminate that
variable and pass its value (ENA_TX_BUDGET - budget) to ena_log_io
directly.
[9 lines not shown]
ena: Update driver version to v2.8.2
Bug Fixes:
* Verify that an ENA ring is in netmap only in native mode
Minor Changes:
* Move parenthesis to correct place in switch
* Add comment
* Reorder define
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55698
Sponsored by: Amazon, Inc.
(cherry picked from commit 96c5eaf0ac6b98d0832e1037d672064de43a7e00)
ena: Minor changes
1. Move parenthesis to correct place in switch and fix include order
2. Add comment at the end of an ifdef for clarity
3. Change include order.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55696
Sponsored by: Amazon, Inc.
(cherry picked from commit 2667a8454cff5896c7b467c78cd4ace5ad40f5eb)
ena: Verify that an ENA ring is in netmap only in native mode
netmap operates in two modes:
1) Emulated - netmap handling is done by the network stack, the
NIC driver operates transparently to netmap.
2) Native - netmap management is done by the NIC driver.
When checking whether a specific ENA ring is running in netmap
mode, only the following checks were done:
1. IFCAP_NETMAP - Check whether netmap capability is enabled on
the device.
2. NKR_NETMAP_ON - Check whether netmap is actively using this
ring.
The above checks implied that the netmap mode is native and the
ENA driver needs to handle the netmap logic.
The code was missing an explicit check on whether native mode
is actually on (NAF_NATIVE).
This led to a case where though emulated mode was used and
[19 lines not shown]
igc: Disable ASPM L1.2 on I226 to prevent RX stalls
PR: 279245
I226 parts advertise support for the PCIe L1.2 link substate, but a
hardware erratum makes the exit latency from that low-power state
longer than the packet buffer can absorb under load. This stalls the
inbound packet stream. Disabling ASPM system-wide (BIOS or OS ASPM
policy) does not fix it. The L1.2 enable bit must be cleared directly
in the device's own PCIe L1 PM extended capability.
Add igc_is_device_id_i226() to identify affected parts and
igc_disable_broken_aspm_l1_2() to clear the ASPM L1.2 and PCI-PM L1.2
enable bits on attach and after resume, since PCIe config space can be
reset across a suspend/resume cycle.
Ported from the Linux igc driver:
0325143b59c6 igc: disable L1.2 PCI-E link substate to avoid
[5 lines not shown]
pci: Add L1 PM definitions
Add register/bit definitions for the L1 PM substates capability
(PCIZ_L1PM) to pcireg.h.
Signed-off-by: Michael Adler <madler at tapil.com>
igc: Disable ASPM L1.2 on I226 to prevent RX stalls
PR: 279245
I226 parts advertise support for the PCIe L1.2 link substate, but a
hardware erratum makes the exit latency from that low-power state
longer than the packet buffer can absorb under load. This stalls the
inbound packet stream. Disabling ASPM system-wide (BIOS or OS ASPM
policy) does not fix it. The L1.2 enable bit must be cleared directly
in the device's own PCIe L1 PM extended capability.
Add igc_is_device_id_i226() to identify affected parts and
igc_disable_broken_aspm_l1_2() to clear the ASPM L1.2 and PCI-PM L1.2
enable bits on attach and after resume, since PCIe config space can be
reset across a suspend/resume cycle.
Ported from the Linux igc driver:
0325143b59c6 igc: disable L1.2 PCI-E link substate to avoid
[5 lines not shown]
pci: Add L1 PM definitions
Add register/bit definitions for the L1 PM substates capability
(PCIZ_L1PM) to pcireg.h.
Signed-off-by: Michael Adler <madler at tapil.com>
Attempt to repair rx issues with the igc driver.
Our testcase uses a link between two igc ports on the same platform and forcing the issue by triggering an eee status update.
Although we found an easy trigger, there are likely other events causing the same issue. This commit replaces igc_if_init() with iflib_request_reset() calls to ensure a rebuild of RX state and publishing descriptor tails.
Setup:
sysctl net.fibs=3
ifconfig igc0 media 1000baseT mediaopt full-duplex
ifconfig igc3 media 1000baseT mediaopt full-duplex
ifconfig igc3 fib 1
ifconfig igc0 inet 10.99.99.1/30 up
ifconfig igc3 inet 10.99.99.2/30 up
test correct:
ping -t 3 -c 2 10.99.99.2
[8 lines not shown]
Attempt to repair rx issues with the igc driver.
Our testcase uses a link between two igc ports on the same platform and forcing the issue by triggering an eee status update.
Although we found an easy trigger, there are likely other events causing the same issue. This commit replaces igc_if_init() with iflib_request_reset() calls to ensure a rebuild of RX state and publishing descriptor tails.
Setup:
sysctl net.fibs=3
ifconfig igc0 media 1000baseT mediaopt full-duplex
ifconfig igc3 media 1000baseT mediaopt full-duplex
ifconfig igc3 fib 1
ifconfig igc0 inet 10.99.99.1/30 up
ifconfig igc3 inet 10.99.99.2/30 up
test correct:
ping -t 3 -c 2 10.99.99.2
[8 lines not shown]
