verbs/mlx5: Add GRE and MPLS flow specification filter
[PATCH 30/31] FreeBSD OFED support for DPDK MLX5 PMD
a) Allow verbs applications packet steering of GRE tunneled traffic.
Adding GRE flow specification based on RFC 2890.
GRE consists of flags, protocol and key fields.
IPv4 protocol 47 (IPPROTO_GRE) can be used when GRE packets are
encapsulated in IPv4.
b) verbs: Add MPLS flow specification filter
Add MPLS flow specification based on RFC 3032.
MPLS spec defined with label field which includes the
label value and additional parameters such as: BoS, TC and TTL.
MPLS allows stacking multiple labels in sequence.
In addition, the MPLS header can be encapsulated on top of different
layers, e.g.: ETH, IP (rfc4023), UDP (rfc7510), GRE (rfc4023).
Therefore, when using the flow creation verb, the application should
[14 lines not shown]
IB/mlx5: Expose GRE flow spec to user-kernel ABI header
[PATCH 29/31] FreeBSD OFED support for DPDK MLX5 PMD
a) Add ib_uverbs_flow_spec_gre to define the rule to match GRE
encapsulation protocol.
The spec includes the generic specs header, type, size and reserved
fields while the filter itself is defined as ib_uverbs_flow_gre_filter
and includes:
- Checksum present bit, key present bit and version bits in a single
16bit field.
- Protocol type field - Indicates the ether protocol type of the
encapsulated payload.
- Key field - present if key bit is set and contains an application
specific key value.
b) IB/uverbs: Expose MPLS flow spec to user-kernel ABI header
[24 lines not shown]
fixup! verbs/mlxs: mlx5: Add tunnel offloads support in direct verbs
Fix is based on upstream rdma-core commit ee54f9d5348f ("mlx5: Add
loopback flags to QP creation").
Validate create_flags against a supported-flags mask instead of rejecting
anything that is not MLX5DV_QP_CREATE_TUNNEL_OFFLOADS. A create_flags of
0 is now accepted, unknown bits are rejected, and the vendor flag is OR-ed
in rather than assigned. The loopback flags that commit also adds do not
exist in this tree, so only the validation is taken.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! IB/mlx5: Expose GRE flow spec to user-kernel ABI header
Import Linux upstream commit a93b632c4531 ("IB/mlx5: Fix GRE flow
specification").
Honour the user-supplied GRE protocol mask instead of hard-coding 0xffff,
so wildcard and partial masks work.
Also give flow_spec_data[] 8-byte alignment, so this userspace copy of
struct ib_uverbs_flow_spec_hdr matches the kernel UAPI on 32-bit builds.
The kernel header has always used __aligned_u64 here and only our copy
diverged, so there is no upstream commit for that half. The attribute is
spelled out instead of introducing an __aligned_u64 macro, which
<infiniband/types.h> does not provide and which libbnxtre already defines
for itself.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! verbs/mlx5: Add GRE and MPLS flow specification filter
Import rdma-core upstream commit ff01da2c5ac2 ("verbs: Fix typo in copying
IBV_FLOW_SPEC_UDP/TCP 'val'").
The TCP/UDP filter value was copied using the size of the IPv4 filter,
overrunning the destination subobject.
Also accept inner MPLS flow specs, matching 6d6f29721eea ("verbs: Allow
creation of inner MPLS flow spec"). Upstream deliberately has no inner
case for GRE - it is a tunnel header, matched only in the outer stack - so
that part of the review comment is not applied.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! verbs/mlx5: Expose tag matching capabilities
Reject TM-SRQ creation when tm_cap.max_ops is 0, so the internal command
QP's send queue is not sized to zero work-queue entries.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! IB/mlx5: Add tunneling offloads support
Import Linux upstream commit 4e2b53a5cb5a ("IB/mlx5: Report inner RSS
capability").
Define MLX5_RX_HASH_INNER as (1UL << 31). Shifting 1 into the sign bit of
a plain int is undefined behaviour, and user space compiles this header
too. The main.c half of that commit is already carried by the preceding
fixup.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! IB/mlx5: Expose multi-packet RQ capabilities
Import Linux upstream commit ccc870879027 ("IB/mlx5: Allow creation of a
multi-packet RQ").
Only the create_rq() part was missing: the striding RQ parameters we
validate and store on the WQ never reached the firmware. Program
two_byte_shift_en, single_stride_log_num_of_bytes and
single_wqe_log_num_of_strides, with the two log values encoded relative to
their MLX5_MIN_* bases so the 3-bit firmware fields do not truncate them.
Upstream renamed these WQ context fields later on, so the names used here
are the ones this tree's mlx5_ifc.h still carries.
Sponsored by: NVidia networking
MFC after: 1 month
verbs/mlxs: mlx5: Add tunnel offloads support in direct verbs
[PATCH 28/31] FreeBSD OFED support for DPDK MLX5 PMD
a) In order to enable offloading such as checksum and LRO for incoming
tunneling traffic, the QP should be created with tunnel offloads flag -
MLX5DV_QP_CREATE_TUNNEL_OFFLOAD.
b) Reports capability of which tunneling type supports
the tunneling offloads.
c) verbs: Add support in RSS of the inner packet
Some user space application would like to do RSS on the inner packet
fields instead of the outer. When user will set the IBV_RX_HASH_INNER
bit with one of the other hash fields, then the RSS will be on the inner
packet.
Differential revision: https://reviews.freebsd.org/D32203
MFC after: 1 month
verbs/mlx5: Allow creation of a Multi-Packet RQ using direct verbs
[PATCH 27/31] FreeBSD OFED support for DPDK MLX5 PMD
a) Add needed definitions to allow creation of a Multi-Packet RQ
using the mlx5 direct verbs interface.
In order to create a Multi-Packet RQ, one needs to provide a
mlx5dv_wq_init_attr containing the following information in its
striding_rq_attrs struct:
- single_stride_log_num_of_bytes: log of size of each stride
- single_wqe_log_num_of_strides: log of number of strides per WQE
- two_byte_shift_en: When enabled, hardware pads 2 bytes of zeros
before writing the message to memory (e.g. for IP alignment).
b) Add a helper function to verify 64 bit comp mask
The common check for a mask is as follows:
if (comp_mask & ~COMP_MASK_SUPPORTED_VALUES)
return EINVAL;
[11 lines not shown]
libmlx5: Report Multi-Packet RQ capabilities through mlx5 direct verbs
[PATCH 26/31] FreeBSD OFED support for DPDK MLX5 PMD
A Multi-Packet RQ is a receive queue where multiple packets are
written to the same WQE. Each message starts in the beginning of a
stride. The total size of the scatter elements of each WQE is
determined upon RQ creation and all the posted WQEs should meet the
determined size.
A Multi-Packet RQ reduces the number of needed post-recv operations
thus increasing performance.
It reduces memory footprint by allowing each packet to consume a
different number of strides instead of the whole WR.
Differential revision: https://reviews.freebsd.org/D32201
MFC after: 1 month
IB/mlx5: Fix ABI alignment to 64 bit
[PATCH 25/31] FreeBSD OFED support for DPDK MLX5 PMD
Struct mlx5_ib_striding_rq_caps was not aligned to 64 bit as
it should have been. Add a 32 bit reserved field.
Differential revision: https://reviews.freebsd.org/D32200
MFC after: 1 month
verbs/mlx5: Expose tag matching capabilities
[PATCH 23/31] FreeBSD OFED support for DPDK MLX5 PMD
a) Expose tag matching capabilities and show them in ibv_devinfo.
b) verbs: Introduce tag matching SRQ
Introducing tag matching SRQ (TM-SRQ), which retains basic semantic of
regular SRQ, reports completions to own CQ, and has additional tag
based message receiving mechanism.
Detailed description for the TM-SRQ usage was added into
Documentation/tag_matching.md
c) mlx5: Add support to tag matching SRQ type
Create command QP for a tag-matching SRQ. Command QP used for
inserting/removing entries from the tag matching list. This command QP
is hidden from the users in the mlx5_srq structure.
New verb ibv_post_srq_ops() will be added in next patch to use it.
[2 lines not shown]
IB/mlx5: Add tunneling offloads support
[PATCH 22/31] FreeBSD OFED support for DPDK MLX5 PMD
a) The device can support receive Stateless Offloads for the inner
packet's fields only when the packet is processed by TIR which is
enabled to support tunneling. Otherwise, the device treats the
packet as an ordinary non-tunneling packet and receive offloads
can be done only for the outer packet's field.
In order to enable receive Stateless Offloading support for incoming
tunneling traffic the TIR should be created with tunneled_offload_en.
Tunneling offloads is supported only be raw ethernet QP.
This patch includes:
- New QP creation flag for tunneling offloads.
- Reports device capabilities.
b) IB/mlx5: Add support for RSS on the inner packet
Some user space application would like to do RSS on the inner
[6 lines not shown]
fixup! libmlx5: Report SW parsing capabilities through mlx5 direct verbs
Document the full struct mlx5dv_context (cqe_comp_caps, sw_parsing_caps)
and the mlx5dv_context_comp_mask enum in mlx5dv_query_device.3, and note
that these caps are only valid when the matching comp_mask bit is asked
for on input and returned on output.
Sponsored by: NVidia networking
MFC after: 1 month
libmlx5: Report SW parsing capabilities through mlx5 direct verbs
[PATCH 24/31] FreeBSD OFED support for DPDK MLX5 PMD
Software parsing (SWP) is a feature that can be used to instruct the
device to stop using its internal parser and to parse packets on the
transmit path according to offsets set for each packet.
Through this feature, the device allows the handling of checksum and LSO
by the hardware according to the location of IP and TCP/UDP headers.
Report various SW parsing capabilities and supported QP types through
mlx5 direct verbs interface.
Differential revision: https://reviews.freebsd.org/D32199
MFC after: 1 month
fixup! IB/mlx5: Add tunneling offloads support
Drop the unused MLX5_IB_QP_TUNNEL_OFFLOAD flag. Tunnel offload is driven
by qp->tunnel_offload_en, so the enum bit was dead and misleading.
Also advertise MLX5_RX_HASH_INNER in query_device()'s
rss_caps.rx_hash_fields_mask, so user space can discover inner RSS.
Sponsored by: NVidia networking
MFC after: 1 month
IB/mlx5: Expose multi-packet RQ capabilities
[PATCH 21/31] FreeBSD OFED support for DPDK MLX5 PMD
a) This patch reports the device's striding RQ capabilities to
the user-space:
- min/max_single_stride_log_num_of_bytes: Log of min/max number of
bytes in a single stride.
- min/max_single_wqe_log_num_of_strides: Log of min/max number of
strides in a single WQE.
- supported_qpts: A bit mask to know which QP types support multi-
packet RQ, for now only Raw Packet QPs.
b) Allow creation of a multi-packet receive queue.
In order to create a multi-packet RQ, the following fields in
the mlx5_ib_rwq should be set:
- log_num_strides: Log of number of strides per WQE
- single_stride_log_num_of_bytes: Log of a single stride size
- two_byte_shift_en: When enabled, hardware pads 2 bytes of zeros
[4 lines not shown]
fixup! IB/mlx5: Expose software parsing for Raw Ethernet QP
Report sw_parsing_caps using the same condition that enables software
parsing on the SQ (both eth_net_offloads and swp), so what we advertise
matches what the driver actually does.
Sponsored by: NVidia networking
MFC after: 1 month
libmlx5: Report if kernel allows using MPW in SQ
PATCH 20/31] FreeBSD OFED support for DPDK MLX5 PMD
a) Use flag MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED to indicate hardware
supports multi packet WQE and it's enabled in SQ context.
Flag MLX5DV_CONTEXT_FLAGS_MPW is deprecated, shall not be used
in new applications.
b) Report if enhanced multi packet send WQE is supported through mlx5
direct verbs.
Differential revision: https://reviews.freebsd.org/D32194
MFC after: 1 month
IB/mlx5: Expose software parsing for Raw Ethernet QP
[PATCH 17/31] FreeBSD OFED support for DPDK MLX5 PMD
Software parsing (SWP) is a feature that can be used to instruct the
device to stop using its internal parser and to parse packets on the
transmit path according to offsets set for each packets.
Through this feature, the device allows the handling of checksum and
LSO by the hardware according to the location of IP and TCP/UDP
headers.
Enable SW parsing on Raw Ethernet send queue by default if firmware
supports it and report these capabilities to user space.
Differential revision: https://reviews.freebsd.org/D32191
MFC after: 1 month
IB/mlx5: Report mlx5 packet pacing capabilities when querying device
[PATCH 15/31] FreeBSD OFED support for DPDK MLX5 PMD
Enable mlx5 based hardware to report packet pacing capabilities
from kernel to user space. Packet pacing allows to limit the rate to any
number between the maximum and minimum, based on user settings.
The capabilities are exposed to user space through query_device by uhw.
The following capabilities are reported:
- The maximum and minimum rate limit in kbps supported by packet pacing.
- Bitmap showing which QP types are supported by packet pacing operation.
Differential revision: https://reviews.freebsd.org/D32189
MFC after: 1 month
fixup! IB/mlx5: Support Vxlan tunneling specification
Import Linux upstream commits 37da2a03c036 ("RDMA/mlx5: Use proper spec
flow label type") and 0f750966dca8 ("IB/mlx5: Add inner spec and IPv6
validation in user's flow attribute list").
The first widens set_flow_label()'s mask and value to u32, so the 20-bit
IPv6 flow label is no longer truncated to 8 bits. The second validates
the ethertype of inner flow specs as well as outer ones, and adds the IPv6
ethertype checking this tree was missing.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! libmlx5: Report if kernel allows using MPW in SQ
Interpret the kernel's raw multi_pkt_send_wqe value (0, 1 or 3) instead of
masking it against a bitmask the kernel never emits, which silently
disabled MPW.
Sponsored by: NVidia networking
MFC after: 1 month
libmlx5: External memory binding for HW resources
[PATCH 19/31] FreeBSD OFED support for DPDK MLX5 PMD
Add a new DV API mlx5dv_set_context_attr() to enable setting an external
memory allocator. This API will allow the application to use specific
decisions about the memory allocation of HW resources (e.g. DV objects).
Some examples are managing numa pinning per object, managing a
hugepages resource pool, shared memory regions.
Also extend mlx5dv_get_qp() to return UAR mapping info. This can allow a
process to share its doorbell access with secondary process by re-mmap
the UAR address on the device and make it accessible as a user space
address.
Differential revision: https://reviews.freebsd.org/D32193
MFC after: 1 month
fixup! verbs/mlx5: Annotate ibv_wc and ibv_send_wr with endian
Fix is based on upstream rdma-core commit 14a0fc824f16 ("rdma-core/irdma:
Implement device supported verb APIs").
Use ib_wr->invalidate_rkey instead of ib_wr->imm_data for
IBV_WR_SEND_WITH_INV and IBV_WR_LOCAL_INV. The two share a union, but
imm_data is __be32 while the rkey to invalidate is host order, so with the
new endian annotations this path reads the wrong member. Upstream irdma
has always used invalidate_rkey here.
Sponsored by: NVidia networking
MFC after: 1 month
verbs/mlx5: Annotate ibv_wc and ibv_send_wr with endian
PATCH 16/31] FreeBSD OFED support for DPDK MLX5 PMD
a) verbs: Annotate ibv_wc helpers with endian
This follows the scheme as used in the wc by introducing a
ibv_wc_read_invalidated_rkey to access the host endian
invalidated_rkey value with proper annotations.
This is just an inline wrapper to allow sparse to work sensibly,
not really a good reason to add another driver entry point.
b) verbs: Annotate ibv_send_wr with endian
This follows the scheme as used in the wc by introducing a
transparent union in ibv_send_wr to indicate the invalidate_rkey
is in host endian.
c) mlx5: Add sparse annotations.
Differential revision: https://reviews.freebsd.org/D32190
MFC after: 1 month
fixup! libmlx5: Fix ABI break from revising the UAR pointer
Actually provide the two ABI versions this commit describes: export
mlx5dv_init_obj@@MLX5_1.2 with the current behaviour and a compat
mlx5dv_init_obj at MLX5_1.0 that restores the historical 'void **' value, and
list the symbol under both versions in libmlx5.map. Binaries linked
against the old version keep resolving.
Sponsored by: NVidia networking
MFC after: 1 month
fixup! libmlx5: External memory binding for HW resources
Match a doorbell to its page with a range check instead of a page-boundary
mask, so mlx5_free_db() also works for external allocations that are not
page aligned and no longer leaks the doorbell page.
Sponsored by: NVidia networking
MFC after: 1 month
