igc(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit 297394e995e5ea1ea9bc85e609ca116255d51e97)
igc: count TSO wire segments in the AIM counters
The transmit path bills one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64 KiB, rather than a
packet size that appears on the wire.
Count the segments the hardware emits and the header carried by each
segment. Non-TSO accounting is unchanged.
(cherry picked from commit e389a05164ccb1dd41ee8d7f09203b475322dd72)
igc: use packet-size AIM
Use the packet-size calculation introduced for igb(4) in a69ed8dfb381
and retained there until the iflib conversion in f2d6ace4a684. It
derives interrupt holdoff from average packet size, so RSS queue count
does not change its behavior.
The calculation follows the pre-iflib igb code. Retain igc's normal and
low-latency rate caps, and keep the current setting when an interval has
no usable sample.
(cherry picked from commit 01e7acd38d411c78caba1c4078bb3683f586e1c2)
igc: synchronize interrupt moderation state
Keep the saved EITR value synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Apply the packet-buffer fallback without permanently disabling AIM.
(cherry picked from commit e35533457530bb9db655e6137c2eea790e18b97b)
igc: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit 2290ea7f4311e899019fe77bf7c7775033af6b24)
igc: fix RX accounting for multi-descriptor packets
The receive path adds the running packet length to rx_bytes for every
descriptor. A packet spanning descriptors of length l1, l2, and l3 is
therefore counted as 3*l1 + 2*l2 + l3.
Add each descriptor length once. Single-descriptor accounting remains
unchanged.
(cherry picked from commit bbf0372feeb321a5bfeff7b1e79576ab01240441)
em(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit b6b379b94781da5d4328f6f57273fbe7bd9dc687)
e1000: count TSO wire segments in the AIM counters
The transmit paths billed one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64KB, so the average
size the moderation calculation sees is not a size that appears on the
wire.
Count the segments the hardware will put on the wire and the header each
of them carries.
Non-TSO accounting is unchanged.
(cherry picked from commit 072e0983d7bce80356740324973993393e77023a)
e1000: restore packet-size AIM
Restore the packet-size calculation introduced in a69ed8dfb381 and used
by igb(4) until the iflib conversion in f2d6ace4a684. It derives
interrupt holdoff from average packet size, so RSS queue count does not
change its behavior.
The calculation follows the pre-iflib code. Retain the current normal
and low-latency rate caps, and keep the current setting when an interval
has no usable sample.
Fixes: 3e501ef89667 ("e1000: Re-add AIM")
(cherry picked from commit dc4a5087b160c1a94d135ab636642defe2c71c20)
e1000: synchronize interrupt moderation state
Keep the saved EITR and PBA values synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Treat only sub-gigabit links as sub-gigabit and apply the packet-buffer
fallback without permanently disabling AIM.
(cherry picked from commit 6ef368a29b11ebc769e7929566809b75ae2c1e90)
e1000: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit bc5e7b0cbbb555ffebc7d73b273c421f9ee24c23)
ixl: Increase tx/rx ring size to 8160
I've verified the tx queue (table 8-22) in addition.
DPDK commit message
net/i40e: increase max descriptor queue length
According to the Intel X710/XXV710/XL710 Datasheet, the maximum receive
queue descriptor length is 0x1FE0 (8160 in base 10). This is specified
as QLEN in table 8-12, page 1083.
I've tested this change with an XXV710 NIC and it has positive effect on
performance under high load scenarios. Where previously I'd get
~2000 packets/sec miss rate, now I get only ~40 packets/sec miss rate.
Signed-off-by: Igor Gutorov <igootorov at gmail.com>
Acked-by: Morten Brørup <mb at smartsharesystems.com>
Acked-by: Bruce Richardson <bruce.richardson at intel.com>
[3 lines not shown]
igc(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit 297394e995e5ea1ea9bc85e609ca116255d51e97)
igc: count TSO wire segments in the AIM counters
The transmit path bills one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64 KiB, rather than a
packet size that appears on the wire.
Count the segments the hardware emits and the header carried by each
segment. Non-TSO accounting is unchanged.
(cherry picked from commit e389a05164ccb1dd41ee8d7f09203b475322dd72)
igc: use packet-size AIM
Use the packet-size calculation introduced for igb(4) in a69ed8dfb381
and retained there until the iflib conversion in f2d6ace4a684. It
derives interrupt holdoff from average packet size, so RSS queue count
does not change its behavior.
The calculation follows the pre-iflib igb code. Retain igc's normal and
low-latency rate caps, and keep the current setting when an interval has
no usable sample.
(cherry picked from commit 01e7acd38d411c78caba1c4078bb3683f586e1c2)
igc: synchronize interrupt moderation state
Keep the saved EITR value synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Apply the packet-buffer fallback without permanently disabling AIM.
(cherry picked from commit e35533457530bb9db655e6137c2eea790e18b97b)
igc: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit 2290ea7f4311e899019fe77bf7c7775033af6b24)
igc: fix RX accounting for multi-descriptor packets
The receive path adds the running packet length to rx_bytes for every
descriptor. A packet spanning descriptors of length l1, l2, and l3 is
therefore counted as 3*l1 + 2*l2 + l3.
Add each descriptor length once. Single-descriptor accounting remains
unchanged.
(cherry picked from commit bbf0372feeb321a5bfeff7b1e79576ab01240441)
em(4): document adaptive interrupt moderation
Describe the disabled, adaptive, and low-latency settings and their
interrupt-rate tradeoffs.
(cherry picked from commit b6b379b94781da5d4328f6f57273fbe7bd9dc687)
e1000: count TSO wire segments in the AIM counters
The transmit paths billed one packet of ipi_len bytes per request. For
TSO that is the whole unsegmented payload, up to 64KB, so the average
size the moderation calculation sees is not a size that appears on the
wire.
Count the segments the hardware will put on the wire and the header each
of them carries.
Non-TSO accounting is unchanged.
(cherry picked from commit 072e0983d7bce80356740324973993393e77023a)
e1000: restore packet-size AIM
Restore the packet-size calculation introduced in a69ed8dfb381 and used
by igb(4) until the iflib conversion in f2d6ace4a684. It derives
interrupt holdoff from average packet size, so RSS queue count does not
change its behavior.
The calculation follows the pre-iflib code. Retain the current normal
and low-latency rate caps, and keep the current setting when an interval
has no usable sample.
Fixes: 3e501ef89667 ("e1000: Re-add AIM")
(cherry picked from commit dc4a5087b160c1a94d135ab636642defe2c71c20)
e1000: synchronize interrupt moderation state
Keep the saved EITR and PBA values synchronized with hardware across
reinitialization. Correct EITR encoding, decoding, and MSI-X register
selection, and reject nonpositive fallback rates.
Treat only sub-gigabit links as sub-gigabit and apply the packet-buffer
fallback without permanently disabling AIM.
(cherry picked from commit 6ef368a29b11ebc769e7929566809b75ae2c1e90)
e1000: make AIM counter sampling coherent
Sample free-running counters by delta instead of clearing them from the
interrupt filter, which can race their producers. Publish byte and
packet counts together at the TX and RX doorbells so each sample is
coherent.
Aggregate every TX ring assigned to the interrupt vector so unequal RX
and TX queue counts are safe. Count RX bytes only after a frame is
accepted.
(cherry picked from commit bc5e7b0cbbb555ffebc7d73b273c421f9ee24c23)
ena: Update driver version to v2.8.4
Bug Fixes:
* Fix false 'missing TX completions' warnings due to timestamp race
* Put taskqueues into correct NUMA domain if !RSS
Minor Changes:
* Batch RX statistics updates
* Swap RX/TX completions cleanup order
Submitted by: Arthur Kiyanovski <akiyano at amazon.com>
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D58242
(cherry picked from commit 605e699cd6ca4feae6c73c5c5ea8337054897116)
ena: Fix false 'missing TX completions' warnings due to timestamp race
Sporadic 'Found a Tx that wasn't completed on time' warnings appear
under sustained TX load, always reporting '1 msecs since last cleanup'
despite the 5-second timeout threshold.
The per-packet TX timestamp uses struct bintime (128 bits: two 64-bit
fields sec and frac) which is read and written non-atomically. A race
exists between the missing TX completion check
(check_missing_comp_in_tx_queue reading the timestamp) and the TX
submit path or cleanup path writing it on another CPU. Since the two
fields are not updated atomically, the check can observe a partially
written timestamp - one field from the old value and one from the new.
This can produce a timestamp with {sec=0, frac=valid}, causing the
check to compute a time offset equal to system uptime and falsely
exceeding the 5-second timeout.
Confirmed by instrumentation showing all occurrences had sec=0 with
valid frac/mbuf, cleanup_running=0, and ticks==last_cleanup_ticks.
[24 lines not shown]
ena: Batch RX statistics updates
Move per-packet counter_enter/counter_exit pairs out of the RX
processing loop and batch them into a single update after the
loop completes.
Previously, each received packet triggered two separate
counter_enter/counter_exit blocks -- one for bytes and one for
packet count. This commit accumulates totals in local variables
and updates all four counters (ring and hw stats for both packets
and bytes) in a single counter_enter/counter_exit block after the
loop.
Also move the stats update to after the refill and LRO flush
so that the error path (goto update_stats) and the normal path
converge at the same label, avoiding code duplication.
Submitted by: David Arinzon <darinzon at amazon.com>
MFC after: 2 weeks
[5 lines not shown]
ena: Swap cleanup order
As RX processing is heavier than TX completions processing, swap the
order and process TX completions first, in order to avoid starving the
completions and causing potential missing TX completions.
Submitted by: Ofir Tabachnik <ofirt at amazon.com>
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D58239
(cherry picked from commit f08def9ed97f45700eb0611a3fd9240210c9303e)
ena: Batch RX statistics updates
Move per-packet counter_enter/counter_exit pairs out of the RX
processing loop and batch them into a single update after the
loop completes.
Previously, each received packet triggered two separate
counter_enter/counter_exit blocks -- one for bytes and one for
packet count. This commit accumulates totals in local variables
and updates all four counters (ring and hw stats for both packets
and bytes) in a single counter_enter/counter_exit block after the
loop.
Also move the stats update to after the refill and LRO flush
so that the error path (goto update_stats) and the normal path
converge at the same label, avoiding code duplication.
Submitted by: David Arinzon <darinzon at amazon.com>
MFC after: 2 weeks
[5 lines not shown]
ena: Update driver version to v2.8.4
Bug Fixes:
* Fix false 'missing TX completions' warnings due to timestamp race
* Put taskqueues into correct NUMA domain if !RSS
Minor Changes:
* Batch RX statistics updates
* Swap RX/TX completions cleanup order
Submitted by: Arthur Kiyanovski <akiyano at amazon.com>
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D58242
(cherry picked from commit 605e699cd6ca4feae6c73c5c5ea8337054897116)
