tcp: Unifidef use of rss software hash in syncache
Ever since "d9c55b2e8cd6 rss: Enable portions of RSS globally.."
exposed the RSS software hashing functions, it has been possible
to use them without "ifdef RSS". Do so now in the syncache
so as to get flowids recorded.
Note that the use of the rss hash functions is conditional on IP versions,
so we must ifdef INET to ensure rss_proto_software_hash_v4() is available.
Fixes 73fe85e486d2
Sponsored by: Netflix
Reviewed by: glebius, p.mousavizadeh_protonmail.com, nickbanks_netflix.com, tuexen
Differential Revision: https://reviews.freebsd.org/D54534
(cherry picked from commit 21865c970888ad0e2cde06db843ae2a05af550e7)
tcp: Enable symmetric hashing by setting hash on outgoing conns
Now that we can trust NICs to supply an identical hash result
to software, we can setup the inpcb hash on outgoing connections.
This gives us symmetric hashing, meaning packets should enter
and leave on the same NIC queue.
Differential Revision: https://reviews.freebsd.org/D53104
Reviewed by: adrian, cc, kbowling, tuexen, zlei
Sponsored by: Netflix
(cherry picked from commit dd0e6bb996dc46e1d91f3d9aef87979716479465)
em: remove M_HASHTYPE when RSS is not enabled
Summary:
Since "73fe85e486d2 tcp: store flowid info in syncache", inp_flowid can be set
if the incoming packet is not M_HASHTYPE_NONE. But this can introduce dummy
and duplicated flowid when a virtual interface set M_HASHTYPE_OPAQUE. This
change will let the upper layer know how to deal with software hash, with
benefits like inp_flowid can be set correctly and m_pkthdr.flowid can be set
correctly in output path.
This fix is similar to "20285cad7a55"
Reviewed by: kbowling
Differential Revision: https://reviews.freebsd.org/D55137
(cherry picked from commit efcc0423d80e4048e11e997abc46bd0343537e1b)
igc: make the hardware RSS hash agree with the stack's configuration
rss_gethashconfig() is available without options RSS since d9c55b2e8cd6.
Use it to program MRQC, as ixl(4), ice(4) and iavf(4) do, instead of a
fixed field set that included UDP 4-tuple, which the configuration
excludes unless net.inet.rss.udp_4tuple is set. UDP is now hashed on
addresses only by default.
Also report the UDP hash types on receive; they were passed up as
M_HASHTYPE_NONE.
Reviewed by: kbowling
Fixes: 517904de5cca ("igc(4): Introduce new driver for the Intel I225 Ethernet controller.")
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60284
(cherry picked from commit 66ff9289b041720f29205f179337f81b5dd0c365)
ixl: verify flowid is valid before setting rsstype & flowid
According to section 8.3.2.2.1 of the XL710 datasheet,
FLTSTAT indicates if RSS hashing was used (or flow director
or nothing) to arrive at the flowid it is reporting.
We need to check this before reporting the flowid and setting the
rss type. Not checking this allows invalid flowids (0s) to be
reported for some TCP traffic (it seems to mostly impact incoming
connections). This leads to the inp flowid being 0, and egress
traffic being unbalanced (going to tx queue 0, and NIC 0 of an
lacp bundle).
Differential Revision: https://reviews.freebsd.org/D54442
Reviewed by: adrian, kbowling
Sponsored by: Netflix
(cherry picked from commit ac1cd655f647cfd8e8d8c08da00f41abfce212a1)
rss: make toeplitz.c standard part of the kernel
This will fix LINT-NOIP build. This actually adds very little to the
kernel text, e.g. 500 bytes on amd64. A perfect solution would be to
instead declare rss_config.c as 'optional inet | inet6', but that would
fail to build LINT-NOIP in several NIC drivers, that use RSS and
absolutely ignore that both INET and INET6 are optional. It is very
unlikely that vendors who maintain these drivers will will ever chase the
holy grail of a build that doesn't support IPv4 and IPv6.
Fixes: d9c55b2e8cd6b79f6926278e10a79f1bcca27a4b
(cherry picked from commit 407b1e4980189252fade78438aa08191a9883cdc)
sctp: improve handling of SO_SETFIB socket option
Since SCTP keeps fibnum in the SCTP protocol control block,
the code can't just defer the processing to the IPv4 or IPv6
layer. Therefore it needs to be implemented in the the SCTP
layer and it needs to call the socket layer function.
PR: 300091
Reviewed by: tuexen
Fixes: caccbaef8e26 ("socket: Move SO_SETFIB handling to protocol layers")
(cherry picked from commit 7b8151b0cd1b4f077b382b6be2c02c936d1988fa)
bnxt: Initialize RSS with the common key
Use rss_getkey() for the initial VNIC key instead of generating a
per-attachment private key. Preserve the writable rss_key sysctl,
its override lifetime, the hardware DMA representation, and all hash
and indirection settings.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59795
(cherry picked from commit 62c80d76f7dc960f430cb34f88c2a549612e3e93)
qlnx: Report the programmed RSS key and hash types
Implement SIOCGIFRSSKEY and SIOCGIFRSSHASH for qlnxe and qlnxev using
the cached vport RSS configuration. This lets hn(4) synchronize its
synthetic RSS configuration with the VF without assuming that every
driver programs all of the common RSS hash selections.
Serialize the queries with the device lock and reject them while the
interface is not running. Convert the firmware request's key words back
into network byte order and translate the six representable IPv4/IPv6
TCP/UDP hash selections. Report no hashing when RSS is disabled, including
the driver's single-queue configuration.
For VFs, require a successful RSS extension response from every hardware
function. The overall vport-update reply can succeed while its RSS TLV
is missing or rejected; recording that result prevents reporting an
unconfirmed configuration. Invalidate the confirmation before each RSS
request. Leave existing vport-update return semantics, queue setup, and
the receive datapath unchanged.
[6 lines not shown]
vnic nicvf: Program the common RSS key in network word order
Replace the repeated hardware-reset key with rss_getkey(). Decode each
eight-byte chunk with be64dec() into the register-word array before
programming the five key registers.
This matches the key serialization in DPDK's nicvf_rss_set_key(), added
by commit 7694fab264ac666d62a4cae04f2979991ce64259
("net/thunderx/base: add RSS and RETA configuration"). A raw copy into
the word array on little-endian hosts would reverse bytes within each
register relative to that convention.
Preserve the CPI-algorithm gate, enabled hash fields, and round-robin
indirection table.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59806
(cherry picked from commit 8c25b70120e8838135cb356c94d350105f45ff7c)
enetc: Use the common RSS key
Replace the random register words with rss_getkey() and decode each
four-byte chunk with le32dec() before writing the port's key registers.
Keep the RSS capability check and existing queue-count and indirection
table programming unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59805
(cherry picked from commit 449fc84f370b079619eaa0d70008391ac07718f5)
al_eth: Use the common RSS key
Fetch the common Toeplitz key instead of filling it with random bytes
during receive-forwarding setup. Check the key size at compile time and
retain the existing host-to-network conversion for each hardware word.
Leave forwarding, hash selection, and queue mapping unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59804
(cherry picked from commit fbe4eefbb6ca29e1f88d122619f7311896b81b1f)
qlxgb: Derive the firmware RSS key from the common key
Replace the embedded key with rss_getkey(). Retain the reversed 64-bit
word-pair order and low-word-first layout used by the existing firmware
request, matching the qlxgbe conversion.
The common default produces the same five firmware words as the old
constants. Leave hash selections, RSS enablement, and the indirection
mask unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59803
(cherry picked from commit 4ad740ad48e2b461637c85cad675202874e182e2)
qlxge: Derive both RSS keys from the common key
Replace the embedded default with rss_getkey(). Preserve the ICB's
little-endian representation of each big-endian key word, then copy
the initial 16 bytes of the serialized IPv6 key into the IPv4 key.
This retains the old default-key bytes and allows a configured common
key to take effect. Leave RSS flags and queue mapping unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59802
(cherry picked from commit a5e3c38531cd94d245372147f6f6e1d6d2f82a39)
qlxgbe: Derive the mailbox RSS key from the common key
Replace the embedded key with rss_getkey(), retaining the firmware's
reversed 64-bit word-pair order and low-word-first layout within each
pair. The common default produces the same mailbox key as the existing
constants, while configured common keys no longer get ignored.
Leave the RSS enable flags, hash types, and indirection policy unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59801
(cherry picked from commit 30fdef26de87516dbdd1dd5e3031a57ba8759437)
qlnx: Derive the hardware RSS key from the common key
Replace the embedded default key with rss_getkey(). Decode each group
of four key bytes as a big-endian word; the shared code then serializes
those host-order words as little-endian firmware fields. This preserves
the existing hardware representation for the common default and allows
a configured common key to take effect.
Keep the existing multi-queue gate, RSS hash selections, and indirection
policy unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59800
(cherry picked from commit ce68b58217ed8a9bf45b1622d3d32ec26e9025d0)
mana: Initialize ports with the common RSS key
Use rss_getkey() for each port's initial key instead of a lazily generated
driver-global random key. Remove the obsolete helper and its unsynchronized
initialization flag. Make the common-key declaration available without
options RSS.
The port still retains the key used for hardware programming and the
existing SIOCGIFRSSKEY response. Leave hash types and queue placement
unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59799
(cherry picked from commit aef4c3eb31d9c29c6aeb08023b1c9d2457d817e3)
mlx4en: Use the common RSS key for programming and queries
Replace the private fixed key with rss_getkey(). Copy the common byte
stream directly into the RSS queue-pair context and use the same helper
for SIOCGIFRSSKEY, so hardware programming and hn's query agree.
Change the helper to fill caller-owned storage instead of returning a
pointer to a static key. Check the fixed hardware and ioctl buffer sizes
at compile time. Leave RSS masks and queue steering unchanged.
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59798
(cherry picked from commit 1ad999a1c7e0efd4fd65df634a4c108d593d4027)
oce: Use the common RSS key
Fill the RSS configuration request with the common 40-byte key instead
of fresh random bytes. Preserve the mailbox representation, hash
selection, and indirection table.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59797
(cherry picked from commit fd74d1e83e18b8f014d8fcc7b61f2e9461512dca)
bxe: Program the common RSS key in searcher byte order
Use the common RSS key instead of generating private random words.
Reverse the bytes before passing the key to the shared code, which copies
the supplied bytes directly into the firmware request. The searcher
consumes this array in reverse order; a raw copy of the common key would
not produce matching Toeplitz hashes.
The hardware ordering is documented by Linux commit
d682d2bdc30650a5c7ce9908ab83ab674b658744 ("bnx2x: byte swap rss_key to
comply to Toeplitz specs"). Keep the conversion in the FreeBSD caller
rather than changing the shared-code interface.
Preserve the config_hash gate, including the PMF-only key programming
on 57710/57711, and leave hash types and indirection policy unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59796
[2 lines not shown]
axgbe: Use the common RSS key without options RSS
Fetch the common key in the PCI iflib frontend regardless of options RSS
and make the key API declaration available in both configurations.
Retain the existing hash-field selection and queue-placement branches.
The independent ARM frontend is unchanged.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59794
(cherry picked from commit e94de94f8d14ede98d38c9fc506ddf5c77ee9788)
