iavf: Do not publish link-up while stopped
A PF link event remains cached while a VF is administratively down.
Media status queries called iavf_update_link_status() and published
that cached state as link-up, while the stopped admin path immediately
published link-down. Consumers reacting to link events could turn
this into an unbounded notification loop and prevent interface detach
from draining its link-state task.
Keep the cached PF state, but only publish link-up after iflib has
marked the VF running. A subsequent admin pass publishes the cached
state after a successful initialization.
MFC after: 2 weeks
ixl: Track and recover MDD-blocked VFs
The hardware identifies each VF with TX and RX malicious-driver
status latches, but the driver combined all events into one counter
and reported only the last VF found. It also did not record that
hardware had blocked the VF, leaving the condition invisible to
management tools.
Consume every PF and VF latch, keep per-direction VF counters,
rate-limit per-VF diagnostics, and report the blocked and
traffic-enabled state via the VF status interface. Clear the
software block only after a successful VF or PF reset reconstructs
its resources.
Match Linux i40e policy by leaving a detected VF blocked by default.
Add an opt-in hw.ixl.mdd_auto_reset_vf tunable that notifies and
resets the VF for installations that prefer availability. DPDK
provides the register clear and per-VF attribution precedent; Linux
provides the recovery policy.
[2 lines not shown]
ixl: Report PF initialization failures to iflib
ixl_if_init() returned early after AdminQ reconstruction, LAA, or
VSI initialization failures. Since IFDI_INIT has no return value,
iflib then marked the interface RUNNING and enabled its interrupts
and timers despite the incomplete hardware state.
Use iflib_init_failed() on each incomplete path. Also stop at the
first ring-enable error and tear down any partially enabled rings
before reporting failure. This keeps the interface stopped and
makes a later initialization attempt start from a bounded state.
MFC after: 2 weeks
ixl: Rebuild VF resources after a PF reset
A PF or EMP reset destroys the firmware switch topology, including
every VF VSI. The driver rebuilt only its PF VSI and left configured
VFs with stale switch element and VSI identifiers.
Notify VFs before a driver initiated reset, recreate the IOV VEB, and
rebuild each configured VF VSI and queue mapping after the PF switch
is restored. Keep a VF out of VFACTIVE if its reconstruction fails
so one failure cannot expose incomplete resources or prevent the PF
and other VFs from recovering.
Invalidate cached VF firmware identifiers and runtime state before
recreating the VEB. If VEB creation itself fails, teardown and mailbox
paths can no longer use pre-reset SEIDs or VSI data.
Factor the common VEB setup out of IOV initialization so initial
setup and post-reset reconstruction use the same topology and filter
sequence.
[2 lines not shown]
ixl: Enforce VF VLAN policy
Add access and trunk VLAN policy to the SR-IOV schema. Access VFs
use a hardware PVID and cannot alter their VLAN membership. Trunk VFs
may register up to 16 VLANs, while VLAN 0 remains implicitly admitted
for untagged and priority-tagged traffic.
Enable hardware VLAN anti-spoofing and maintain the MAC-by-VLAN filter
cross-product used by DPDK. Apply Linux's untrusted-VF limits of 18 MAC
addresses and 16 VLANs so one guest cannot consume the shared PF filter
table without bound.
Report the effective policy through the VF status interface and document
the iovctl schema.
MFC after: 2 weeks
Relnotes: yes
ixl: Make VF reset resource reconstruction fallible
Treat each stage of VF reset and VSI reconstruction as fallible. Keep
the VF out of VFACTIVE when PCIe drain, reset completion, VSI
release, or VSI allocation fails, following the DPDK PF reset model.
Propagate initial reset failures back through pci_iov_vf_add and unwind
the VF queue allocation.
Free the old software filter list before initializing a replacement VSI.
ixl_init_filters() previously replaced the list head without freeing its
entries, so every VF FLR leaked all MAC and VLAN filter objects. Reset
the associated counters and VLAN bitmap with the list.
Avoid allocating an initial VSI only to destroy it during the required
initial VF reset, and remove redundant broadcast/filter programming from
VSI setup. Also delete a partially created VSI when later Admin Queue
setup fails.
MFC after: 2 weeks
ixl: Validate VF virtchnl configuration
Bound variable-length virtchnl messages before computing their expected
length, following the newer Intel virtchnl implementation.
Validate VF ring sizes and alignments before programming HMC contexts.
DPDK uses 128-byte ring alignment and 64 through 8160 descriptors;
the virtchnl ABI further specifies TX multiples of 8 and RX multiples
of 32. Preserve the 4096-descriptor limit on X722.
Validate queue bitmaps before changing any rings, validate all queue
and interrupt contexts before applying a request, and reject invalid
RSS table entries. Also avoid sending an ACK after VLAN-strip setup
fails and reply to delete-VLAN errors with the correct opcode.
These checks prevent malformed or oversized requests from an untrusted
VF from partially programming resources outside its allocation.
MFC after: 2 weeks
pci_iov: Roll back failed VF enumeration
pci_iov_enumerate_vfs() logged a failed VF creation or driver
configuration but still reported the whole SR-IOV configuration as
successful. The PF remained enabled with the requested NumVFs and
driver state even though one or more VF children were absent.
Make VF enumeration atomic. Delete children created by the failed
attempt, invoke the PF driver cleanup, disable VF memory space and VF
Enable, release the IOV resources, and return the original error to
iovctl. Also treat failure to create a VF child as an error instead
of silently accepting a partial configuration.
MFC after: 2 weeks
ixl: Initialize VF sysctl contexts before use
The VF array is zeroed at allocation, but its sysctl contexts were
only populated after each VF was successfully added. If VF setup
failed, IOV teardown still passed every requested VF context to
sysctl_ctx_free(). An untouched context is not an initialized empty
TAILQ and caused a page fault during teardown.
Initialize every VF context with the array so both successful setup
and partial-failure cleanup have a valid lifetime.
MFC after: 2 weeks
iavf: Honor iflib transmit completion batching
iavf uses descriptor writeback by default. Hardware writes completion
status into a transmit descriptor only when it completes a descriptor
marked RS. iavf marked every packet RS even though its report-status
queue recorded and inspected only descriptors selected by iflib. The
other completion writes could not help reclaim descriptors.
iflib marks selected packets with IPI_TX_INTR as completion
checkpoints. It forces a checkpoint as deferred work or ring pressure
grows. Retain EOP on every packet, but set RS only at those
checkpoints.
The deprecated head-writeback option on 700-series VFs gets the same
batching: each RS checkpoint permits hardware to publish the completed
ring head.
DPDK uses the same sparse RS design. Let iflib choose the adaptive
interval for FreeBSD. This is a PCIe/memory bandwidth savings.
[2 lines not shown]
ixl: Honor iflib transmit completion batching
ixl uses head writeback by default. Hardware publishes the transmit
ring head through DMA only after completing a descriptor marked RS.
Marking every packet requested much more frequent head updates than
iflib needs to reclaim descriptors.
iflib marks selected packets with IPI_TX_INTR as completion
checkpoints. It forces a checkpoint as deferred work or ring pressure
grows. Retain EOP on every packet, but set RS only at those
checkpoints. This batches head writebacks while preserving bounded
descriptor reclamation.
The optional descriptor writeback mode benefits as well. ixl already
recorded only IPI_TX_INTR descriptors in its report-status queue, so
status written for every other packet was not inspected.
DPDK uses the same sparse RS design. Let iflib choose the adaptive
interval for FreeBSD. This is a PCIe/memory bandwidth savings.
[2 lines not shown]
iavf: Recover when PF communication is unavailable
A PF reset or loss of virtchnl service can make visible interface
initialization wait up to ten seconds and then return from the void
ifdi_init callback. Iflib consequently marks the interface running even
though its queues were not initialized, and no retry is scheduled when
the PF returns.
Check reset readiness without polling during reinitialization, propagate
queue-message submission errors, and bound a silent enable or disable to
one mailbox timeout. Report unsuccessful initialization to iflib and
publish link-down state without polling the stopped mailbox.
A VFLR also discards the Admin Queue and permits the PF to replace the
VF VSI. Track when full virtchnl rediscovery is required, renegotiate the
API version, refresh and validate the VF resources before using a cached
VSI ID, and replay the MAC and VLAN filters cleared by reset. Bound each
runtime discovery attempt while preserving the existing attach-time wait.
[5 lines not shown]
ufshci: do not reset the device in the XPT_RESET_DEV handler
CAM calls the SIM action callback with the SIM lock and the CAM
device lock held. The XPT_RESET_DEV handler called
ufshci_dev_reset(), which sleeps on device commands. Sleeping there
panics when another thread contends for the lock: "panic: sleeping
thread holds CAM device lock".
Report success without touching the device, as nvme_sim(4) does.
A real device reset needs the controller reset path. That rework is
planned together with in-flight request recovery.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58671
ufshci: free the taskqueue on detach
ufshci_ctrlr_destruct() never freed the taskqueue. Every load and
unload cycle leaked the taskqueue and its kernel thread. A task that
was still queued could also run after the module was gone.
Free the taskqueue in destruct. Do it after the interrupt teardown
so nothing enqueues new work. A reset task that is still queued at
this point races the queue teardown. That race is older than this
change. The planned in-flight recovery rework will close it.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58670
ufshci: check completions under the queue lock
The completion scan held only the recovery lock. The submit path sets
a slot to SCHEDULED and then rings the doorbell, both under the queue
lock. A scan running between those two steps saw a SCHEDULED slot with
a clear doorbell and completed a command the device had not started.
The command failed with OCS 0xf, and a reused slot could return wrong
read data.
Check the slot state and the doorbell under the queue lock. The submit
path holds it across both steps, so a half-submitted slot can no
longer be seen. Found with fio randrw verify on QEMU.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58668
ufshci: release the CCB after sending a start stop unit command
ufshci_sim_send_ssu() got a CCB from cam_periph_getccb() but never
returned it. Each call leaked the CCB and one slot of the device's
CCB allocation budget. When the budget runs out, the next
cam_periph_getccb() waits forever and the suspend path hangs.
Release the CCB while the periph lock is still held, as the other CAM
periph drivers do.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58669
ufshci: read UIC command results while holding the lock
The UIC result registers (UICCMDARG2/3) are only valid between a
command's completion and the next command's submission. They were read
after uic_cmd_lock was dropped, so a concurrent UIC submitter could
overwrite them in between. Read them into locals before releasing the
lock.
Also mask the generic error code to its [7:0] field when checking it,
so unrelated bits in UICCMDARG2 (such as the attribute set type echoed
for DME_SET) cannot be mistaken for an error.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58667
ufshci: byte-swap big-endian UPIU fields
The UPIU wire fields are big-endian. The task management and query
builders wrote host-order values into them. The completion paths also
read the results back without conversion. On a little-endian host an
ABORT_TASK carried a swapped task tag and LUN, a query carried a
swapped length, and attribute reads returned swapped values. Tolerant
devices masked most of the damage.
Convert with htobe*/be*toh at the wire boundary, as ufshci_sim.c
already does for its fields.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58664
ufshci: initialize desc_size for non-descriptor query requests
The flag and attribute query builders left param.desc_size
uninitialized, so stack garbage was sent as the query UPIU length
field. Devices generally ignore the length for these opcodes, which
hid the bug. Zero it explicitly.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58665
ufshci: do not free the devq twice on SIM attach failure
cam_sim_free() with free_devq set already frees the devq, so the
following cam_simq_free() call on the xpt_bus_register() and
xpt_create_path() failure paths was a double free. Also clear
ctrlr->ufshci_sim so a later ufshci_sim_detach() does not operate on
the freed SIM.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58662
ufshci: initialize alloc_units before the dedicated-buffer scan
If every unit descriptor read failed in the LU-dedicated WriteBooster
scan, alloc_units was used uninitialized. Start it at zero so that case
is treated as a zero-sized buffer and WriteBooster is disabled.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58663
ufshci: check SDB queue allocations for failure
The hardware queue and ucd_bus_addr allocations use M_NOWAIT but were
used without a NULL check, and the payload bus_dmamap_create() return
value was ignored, so a failed allocation was only discovered by
faulting on it later. Fail the construction instead. The teardown
path handles the partially constructed queue.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58661
ufshci: tolerate partially constructed queues in SDB teardown
When attach fails, ufshci_req_sdb_destroy() runs on a partially
constructed queue, and it runs twice: once from the construct error
path and once from the controller destructor.
Make that safe: NULL-check each resource before freeing it and clear
the pointer afterwards, so a second call finds nothing to do. The
construct error label no longer frees the command descriptors itself,
which fixes a double free of ucd_bus_addr. Also destroy the payload
DMA tag, which was previously leaked. Drop the mtx_initialized()
checks: the locks are always set up before any failure path can reach
the destroy.
Attach can also fail before the queues were constructed at all. The
destructor would then call a NULL qops.destroy pointer, so skip the
destroy when the queue was never set up.
Sponsored by: Samsung Electronics
[2 lines not shown]
ufshci: fix WLUN periph reference counting
The driver stored the WLUN periph pointer without holding a reference,
so the pointer went stale when the pass(4) device went away. In
addition, ufshci_sim_send_ssu() released a reference that it had never
acquired.
Define a simple ownership rule. ufshci_sim_find_periph() acquires the
periph and returns it. The cache owns one reference. The controller
destructor drops it with cam_periph_release() before taking the SIM
lock, since the release takes the CAM device lock by itself.
ufshci_sim_send_ssu() acquires its own reference and releases it when
done. Reuse the cached periph instead of searching again, so the old
reference is not leaked.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58658
