iavf: Recover when PF communication is unavailable
A PF reset or loss of virtchnl service can make visible interface
initialization wait up to ten seconds and then return from the void
ifdi_init callback. Iflib consequently marks the interface running even
though its queues were not initialized, and no retry is scheduled when
the PF returns.
Check reset readiness without polling during reinitialization, propagate
queue-message submission errors, and bound a silent enable or disable to
one mailbox timeout. Report unsuccessful initialization to iflib and
publish link-down state without polling the stopped mailbox.
A VFLR also discards the Admin Queue and permits the PF to replace the
VF VSI. Track when full virtchnl rediscovery is required, renegotiate the
API version, refresh and validate the VF resources before using a cached
VSI ID, and replay the MAC and VLAN filters cleared by reset. Bound each
runtime discovery attempt while preserving the existing attach-time wait.
[5 lines not shown]
ufshci: free the taskqueue on detach
ufshci_ctrlr_destruct() never freed the taskqueue. Every load and
unload cycle leaked the taskqueue and its kernel thread. A task that
was still queued could also run after the module was gone.
Free the taskqueue in destruct. Do it after the interrupt teardown
so nothing enqueues new work. A reset task that is still queued at
this point races the queue teardown. That race is older than this
change. The planned in-flight recovery rework will close it.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58670
ufshci: do not reset the device in the XPT_RESET_DEV handler
CAM calls the SIM action callback with the SIM lock and the CAM
device lock held. The XPT_RESET_DEV handler called
ufshci_dev_reset(), which sleeps on device commands. Sleeping there
panics when another thread contends for the lock: "panic: sleeping
thread holds CAM device lock".
Report success without touching the device, as nvme_sim(4) does.
A real device reset needs the controller reset path. That rework is
planned together with in-flight request recovery.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58671
ufshci: check completions under the queue lock
The completion scan held only the recovery lock. The submit path sets
a slot to SCHEDULED and then rings the doorbell, both under the queue
lock. A scan running between those two steps saw a SCHEDULED slot with
a clear doorbell and completed a command the device had not started.
The command failed with OCS 0xf, and a reused slot could return wrong
read data.
Check the slot state and the doorbell under the queue lock. The submit
path holds it across both steps, so a half-submitted slot can no
longer be seen. Found with fio randrw verify on QEMU.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58668
ufshci: release the CCB after sending a start stop unit command
ufshci_sim_send_ssu() got a CCB from cam_periph_getccb() but never
returned it. Each call leaked the CCB and one slot of the device's
CCB allocation budget. When the budget runs out, the next
cam_periph_getccb() waits forever and the suspend path hangs.
Release the CCB while the periph lock is still held, as the other CAM
periph drivers do.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58669
ufshci: read UIC command results while holding the lock
The UIC result registers (UICCMDARG2/3) are only valid between a
command's completion and the next command's submission. They were read
after uic_cmd_lock was dropped, so a concurrent UIC submitter could
overwrite them in between. Read them into locals before releasing the
lock.
Also mask the generic error code to its [7:0] field when checking it,
so unrelated bits in UICCMDARG2 (such as the attribute set type echoed
for DME_SET) cannot be mistaken for an error.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58667
ufshci: initialize desc_size for non-descriptor query requests
The flag and attribute query builders left param.desc_size
uninitialized, so stack garbage was sent as the query UPIU length
field. Devices generally ignore the length for these opcodes, which
hid the bug. Zero it explicitly.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58665
ufshci: byte-swap big-endian UPIU fields
The UPIU wire fields are big-endian. The task management and query
builders wrote host-order values into them. The completion paths also
read the results back without conversion. On a little-endian host an
ABORT_TASK carried a swapped task tag and LUN, a query carried a
swapped length, and attribute reads returned swapped values. Tolerant
devices masked most of the damage.
Convert with htobe*/be*toh at the wire boundary, as ufshci_sim.c
already does for its fields.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58664
ufshci: initialize alloc_units before the dedicated-buffer scan
If every unit descriptor read failed in the LU-dedicated WriteBooster
scan, alloc_units was used uninitialized. Start it at zero so that case
is treated as a zero-sized buffer and WriteBooster is disabled.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58663
ufshci: do not free the devq twice on SIM attach failure
cam_sim_free() with free_devq set already frees the devq, so the
following cam_simq_free() call on the xpt_bus_register() and
xpt_create_path() failure paths was a double free. Also clear
ctrlr->ufshci_sim so a later ufshci_sim_detach() does not operate on
the freed SIM.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58662
ufshci: tolerate partially constructed queues in SDB teardown
When attach fails, ufshci_req_sdb_destroy() runs on a partially
constructed queue, and it runs twice: once from the construct error
path and once from the controller destructor.
Make that safe: NULL-check each resource before freeing it and clear
the pointer afterwards, so a second call finds nothing to do. The
construct error label no longer frees the command descriptors itself,
which fixes a double free of ucd_bus_addr. Also destroy the payload
DMA tag, which was previously leaked. Drop the mtx_initialized()
checks: the locks are always set up before any failure path can reach
the destroy.
Attach can also fail before the queues were constructed at all. The
destructor would then call a NULL qops.destroy pointer, so skip the
destroy when the queue was never set up.
Sponsored by: Samsung Electronics
[2 lines not shown]
ufshci: check SDB queue allocations for failure
The hardware queue and ucd_bus_addr allocations use M_NOWAIT but were
used without a NULL check, and the payload bus_dmamap_create() return
value was ignored, so a failed allocation was only discovered by
faulting on it later. Fail the construction instead. The teardown
path handles the partially constructed queue.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58661
ufshci: free the correct address when DMA load fails
The bus_dmamap_load() error paths passed hwq->utrd and req_queue->ucd
to bus_dmamem_free(), but both pointers are only assigned after a
successful load and are still NULL at that point. The freshly
allocated memory was leaked. Free the local buffer instead.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58659
ufshci: fix WLUN periph reference counting
The driver stored the WLUN periph pointer without holding a reference,
so the pointer went stale when the pass(4) device went away. In
addition, ufshci_sim_send_ssu() released a reference that it had never
acquired.
Define a simple ownership rule. ufshci_sim_find_periph() acquires the
periph and returns it. The cache owns one reference. The controller
destructor drops it with cam_periph_release() before taking the SIM
lock, since the release takes the CAM device lock by itself.
ufshci_sim_send_ssu() acquires its own reference and releases it when
done. Reuse the cached periph instead of searching again, so the old
reference is not leaked.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58658
ufshci: fix SCSI I/O request failure cleanup
ufshchi_sim_scsiio() did not check the M_NOWAIT request allocation
for NULL. The CDB validation and submit failure paths also returned
without freeing the request.
Fail the CCB when the allocation returns NULL. Free the request on
every failure path. Mark the CCB as queued right before the submit,
so the failure paths above do not need to touch that flag.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58656
ufshci: free the lookup path when the periph search times out
ufshci_sim_find_periph() freed the lookup path only when it found the
periph. The timeout path returned without freeing it and leaked the
path. Free the path at the single exit instead.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58657
ufshci: handle controller command submit failures
Return submission errors from the controller command helpers and
propagate them to polled callers before waiting for completion. Free
requests that never enter a hardware queue so failure paths do not leak
or panic after the poll timeout.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58655
ufshci: fail attribute reads on a non-zero config result code
ufshci_uic_send_cmd() only logged the error code and returned success,
so a failed DME_GET gave its caller a stale value as if it were valid.
The gear and lane settings could then be programmed from that garbage.
Return ENXIO for reads instead. Writes keep logging and continuing,
because a device may reject an optional attribute and that must not
fail bring-up.
Sponsored by: Samsung Electronics
Reviewed by imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58654
ufshci: fix data direction encoding for read commands
The data_direction field in the UTP Transfer Request Descriptor is only
2 bits wide ([26:25]). UFSHCI_DATA_DIRECTION_FROM_TGT_TO_SYS was defined
as 0x10, which truncates to 0b00 (No data transfer) when stored into the
2-bit field, so every read command was described to the controller as
having no data phase. Only writes (0b01) happened to be encoded
correctly.
Define all values as 2-bit binary literals, matching the existing
RESERVED = 0b11 entry, so read is encoded as 0b10 as required by the
specification.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58652
ufshci: abort submission when payload DMA mapping fails
When bus_dmamap_load_mem() failed, ufshci_req_queue_prepare_prdt()
manually completed and released the tracker, but its caller kept going:
it built the UTRD, set the slot back to SCHEDULED, and rang the
doorbell for a tracker whose request had already been freed. Return the
mapping error and stop the submission so the released tracker is not
resurrected.
Sponsored by: Samsung Electronics
Reviewed by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D58653
pmc(8): revert unnecessary lvalue reference change from prior commit
I need to do more work before references can be accepted in other
sections of the code. This was an unnecessary drive-by change that was
not tested in `make universe`.
Reported by: CI
Fixes: fd809148 ("pmc(8): resolve -Wshadow issues")
ixgbe: supply PF transmit contexts under SR-IOV
X550-family malicious-driver detection validates the transmit
context selected by a data descriptor with Check Context set. ixgbe
sets that bit on every transmit data descriptor, but ordinary PF
packets without a VLAN or checksum offload do not create a context
descriptor. The empty context then reports an invalid MAC-header
length and blocks the PF queue as soon as MDD is enabled.
Create the existing context descriptor for every PF packet while
SR-IOV is active. This supplies the required MAC-header length and
keeps MDD from mistaking normal PF traffic for a malicious-driver
event.
(cherry picked from commit 0787b1f5b8bdfcaed97eeee7bfbd7f14ac162b0d)
ixgbe: supply PF transmit contexts under SR-IOV
X550-family malicious-driver detection validates the transmit
context selected by a data descriptor with Check Context set. ixgbe
sets that bit on every transmit data descriptor, but ordinary PF
packets without a VLAN or checksum offload do not create a context
descriptor. The empty context then reports an invalid MAC-header
length and blocks the PF queue as soon as MDD is enabled.
Create the existing context descriptor for every PF packet while
SR-IOV is active. This supplies the required MAC-header length and
keeps MDD from mistaking normal PF traffic for a malicious-driver
event.
(cherry picked from commit 0787b1f5b8bdfcaed97eeee7bfbd7f14ac162b0d)
e1000: Correct 82542 flow-control mode handling
The 82542-specific setup routine unconditionally reads the NVM
default, overwriting a flow-control mode selected by software. It
also removes transmit PAUSE support from all 82542 revisions even
though the hardware restriction applies only to rev 2.0.
Resolve the NVM default only when requested, scope the transmit
restriction to rev 2.0, and replace integer bit masking of the enum
with explicit valid mode transitions. This restores the behavior
from before the Intel shared-code split and resolves -Wassign-enum.
Reported by: glebius
MFC after: 2 weeks
ice: Report initialization failures to iflib
The primary and mirror-VSI ifdi_init callbacks can return early when
reset state or hardware queue and filter setup prevents initialization.
Iflib then marks the interface running and enables interrupts although
the driver did not finish bringing it up.
Report each non-detach failure through iflib_init_failed(). Keep the
existing ice reset and subinterface-reinitialization machinery
responsible for scheduling recovery.
MFC after: 2 weeks
bnxt: Report initialization failures to iflib
HWRM failures currently return from the void ifdi_init callback.
iflib then marks the interface running and enables interrupts despite an
incomplete ring or VNIC setup.
Move the hardware setup into an error-returning helper. The ifdi_init
wrapper can report failure through iflib_init_failed(), while firmware
recovery can propagate the same error through bnxt_open(). Also clear
the initialized state after partial setup is torn down.
MFC after: 2 weeks
ixgbe: Enable PF RSS across queues with SR-IOV
PSRTYPE is indexed by pool in VMDq+RSS mode, and its RQPL
field selects the number of receive queues available within the pool.
The PF occupies the last pool, but the driver programmed pool zero and
left the PF RQPL value at zero. As a result, all PF receive traffic
was directed to its first queue while SR-IOV was enabled.
Program PSRTYPE for the PF pool and encode its allocated receive queue
count.
MFC after: 2 weeks
ixv: Recover when the PF mailbox is unavailable
A failed VF reset or mailbox API negotiation currently returns from
the void ifdi_init callback. Iflib then marks the interface running
even though ixv left its adapter stopped. Stopped media queries can
continue polling the PF, and no timer remains active to retry when the
PF returns.
Track mailbox readiness and report unsuccessful initialization to
iflib. Stopped admin and media-status passes now publish cached
link-down state without touching the mailbox. While the VF remains
administratively up, retry complete initialization after 250 ms, one
second, four seconds, and then at a capped eight-second interval.
Preserve the requested MAC across reset, then program it once after
mailbox API negotiation. The previous two pre-reset requests each
could wait a full mailbox timeout after an established PF disappeared,
holding the iflib context lock for about two seconds before the reset
handshake.
[8 lines not shown]
igbv: Recover when the PF mailbox is unavailable
A VF reset can sanitize its retained queue registers even when the PF
does not complete the cooperative mailbox handshake. Keep those two
states separate. Do not program or enable the rings until both queue
sanitation and mailbox initialization have succeeded.
Report either initialization failure to iflib so the interface remains
stopped. Stopped admin and media-status passes now publish cached
link-down state without polling the mailbox. While the VF remains
administratively up, retry complete initialization after 250 ms, one
second, four seconds, and then at a capped eight-second interval.
Conditional iflib reset requests ensure an intervening administrative
down cancels a queued retry.
Avoid a redundant mailbox reset in the stop half of an immediate iflib
reinitialization; the following init performs the required reset.
Preserve the reset on an ordinary administrative stop and keep the
existing bounded queue-sanitation retry policy independent from
mailbox liveness recovery.
iflib: Support recoverable initialization failure
The ifdi_init method cannot report an error, so iflib always marks an
interface running and enables its interrupts after the callback returns.
Drivers whose hardware initialization depends on an unavailable peer can
only return early and leave a falsely running interface.
Add iflib_init_failed() so a callback can leave the interface stopped.
Also add a conditional reset request for asynchronous recovery: it is
discarded if the interface is administratively down when the admin task
runs, preventing a queued retry from resurrecting a stopped interface.
Do not restore saved driver flags after an MTU or capability change when
initialization failed. Restoring the pre-init flags would overwrite the
stopped result with stale RUNNING state.
Document that reset requests require the caller to schedule the admin
task, that output remains blocked during recovery, and that iflib rather
than the driver owns the driver flags.
[2 lines not shown]
