arm64/vmm: vtimer: Read CNTPCT_EL0 through a dedicated macro
CNTPCT_EL0 needs to be read after an ISB in order to ensure a consistent
value irrespective of speculative execution.
Make existing reads of CNTPCT_EL0 use a new dedicated accessor macro
that expands to the correct instruction sequence.
Signed-off-by: Kajetan Puchalski <kajetan.puchalski at arm.com>
Reviewed by: andrew
Sponsored by: Arm Ltd
Pull Request: https://github.com/freebsd/freebsd-src/pull/2423
arm64: Add ACPI support to GICv5 driver
Add ACPI support to the GICv5 IRS, ITS and IWB drivers. Device parameters are
derived from the MADT and IORT tables.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D59171
ig4(4): fix attach of ACPI-enumerated LPSS controllers
Intel LPSS I2C controllers enumerated through ACPI rather than PCI never
attach on Haswell and Broadwell, so every device behind those buses is
lost. On a Dell XPS 13 9343 that hides the I2C HID touchpad and leaves
only the PS/2 fallback, which the firmware does not restore after S3.
Three causes, all on the ACPI path:
Firmware may leave an LPSS function in D3, where its registers read as
all-ones and set_controller() fails with "controller error during
attach-1". Run _PS0 before mapping them. The PCI path does not need
this, which is why the gap went unnoticed.
INT33C2, INT33C3, INT3432 and INT3433 are Lynx Point-LP and Wildcat
Point-LP, which ig4_pci.c already classifies as IG4_HASWELL; the ACPI
path called everything but APMC0D0F an Atom SoC.
The functional clock stays gated until bit 0 of IG4_REG_CLK_PARMS is
[12 lines not shown]
ice: Add a failure injection facility
Add compile-time optional, non-sleeping fail points around every VF
creation resource boundary, before VF VSI reconstruction, and in the
GET_STATS validation path.
Provide an ICE-wide wrapper and device selector so other driver
subsystems can add scoped points without duplicating the failpoint
plumbing. Keep the current SR-IOV points and VF selector in an iov
child namespace.
Compile the facility only with options DRIVER_FAILPOINTS. This shared
option avoids a separate kernel option for every driver that provides
test-only injection hooks. Ordinary kernels contain no ICE failpoint
objects or sysctl nodes. Require an exact PF device name and
optionally a VF index before any point can fire. This prevents a stale
test setting from affecting another PF.
The hooks exposed two reset-lifetime defects while validating the
[13 lines not shown]
ice: Make VF VLAN requests idempotent
VF drivers replay their VLAN filters after a reset and may retry a
request whose reply was lost. The PF tracked only a count and sent
every requested ID back to the switch. After PF reset replay had
already restored the filters, duplicate VID 0 failed with
ICE_ERR_ALREADY_EXISTS and NACKed the entire VF batch.
Track exact VLAN membership for each VF. Compact requests to unique
IDs whose membership changes, enforce the configured limit against
those IDs, and update membership after each hardware operation so
partial failures cannot undercount filters. Treat already-present
adds and already-absent deletes as successful reconciliation and
suppress their misleading low-level error dump.
Validated on an E810-XXV with a host-attached iavf VF. A three-filter
limit was filled with VIDs 0, 1, and 4094. PF and CORE resets replayed
all three without a duplicate warning or ADD_VLAN NACK, and DTrace
confirmed a three-VID replay reached the PF. A fourth unique VID was
[8 lines not shown]
ice: Fix SR-IOV VF resource cleanup
ice_iov_uninit() freed each VF interrupt-map array without returning the
reserved indices to the device interrupt resource manager. Repeated VF
create and destroy cycles therefore exhausted the PF interrupt map even
though no VFs remained.
Return the interrupt allocation before freeing its map. Also split
software-only VSI release from hardware teardown so failures before
ice_initialize_vsi() do not issue invalid RSS, scheduler, and Free VSI
commands for an object firmware has never seen.
Keep a VF disabled until all of its resources and hardware state have
been created successfully. Clear the enabled state before teardown and
after any failed add so asynchronous mailbox processing cannot use a
partial or freed VSI. Consume VFLR status for inactive VF slots without
trying to reset a nonexistent VSI.
Track whether firmware currently owns each VSI and clear that ownership
[14 lines not shown]
powerpc/pmap: Use dcbz to zero pages in the radix pmap
pagezero() was a plain store loop (bzero), which under the kernel build
flags (-mno-vsx -msoft-float) compiles to byte stores. dcbz
establishes a zeroed cache line directly in the cache without a
read-for-ownership fetch from memory, roughly halving the memory
transactions of page zeroing.
Measured on POWER9 (Raptor Blackbird, DD2.3, bare metal), zeroing a
cold 256 MB buffer with 128-byte scalar loops:
byte stores (current libkern memset) 8.6 GB/s
doubleword (std) stores 26.8 GB/s
dcbz 34.6 GB/s
dcbz raises an alignment interrupt on caching-inhibited mappings, and
the kernel does not emulate it, so mmu_radix_zero_page() falls back to
bzero() for any page whose memattr is not the write-back default. The
internal pagezero() callers only touch freshly allocated page-table
[9 lines not shown]
ice: Report VF queue enable failures as hardware errors
The ENABLE_QUEUES handler has already validated the queue selection
before attempting to enable Rx hardware. A timeout or unexpected queue
state is an operation failure, not an invalid virtchnl parameter.
Return VIRTCHNL_STATUS_ERR_ADMIN_QUEUE_ERROR for these failures, matching
the disable path. Keep ERR_PARAM for invalid requests and attempts to
enable unconfigured queues. Preserve the recorded state of queues which
were enabled before a later queue failed.
MFC after: 2 weeks
Sponsored by: BBOX.io
ice: Make VF queue transitions idempotent
The VF mailbox handler unconditionally submitted queue-disable
commands, including when a VF repeated a request or negotiated after a
PF reset had already destroyed its queues. Firmware can reject stale
Tx queue metadata, causing a NACK and unnecessary VF recovery.
Track queue configuration and enable state across virtchnl operations,
and clear it at VF and PF reset. Apply only hardware transitions that
are not already complete while preserving progress after a partial
failure. Validate queue configurations before mutating hardware so the
state maps remain trustworthy.
FreeBSD configures Tx hardware in CONFIG_VSI_QUEUES rather than
ENABLE_QUEUES, so track Tx configuration separately from Rx
configuration and enable state. Linux ice similarly tracks per-VF Tx
and Rx queue state and skips redundant transitions.
On an E810-XXV, the prior code emitted AQ_RC_EINVAL while configuring a
[7 lines not shown]
ice: Rebuild VF VSIs after PF resets
PF and device resets discard all hardware VSI state. ice_rebuild()
only recreated the main PF VSI before replaying configuration for every
VSI. As a result, replay used stale VF VSI handles. Firmware rejected
it with AQ_RC_EACCES and the failure aborted the entire PF rebuild.
Re-add each VF VSI before replaying its configuration, and do not
publish VFACTIVE until both operations succeed. Clear the initialized
state so that the VF must negotiate again. If a VF rebuild fails, leave
it inactive and continue so that one guest cannot prevent the PF or
sibling VFs from recovering.
Mark it initialized only after the resource response is submitted
successfully, so the state follows completion of the mailbox handshake.
Make that failed state authoritative in the mailbox path. While the
firmware VSI is invalid, permit only VERSION and RESET_VF and reject
operations which require VSI state. A VFR may complete the hardware
[44 lines not shown]
ice: Quiesce queues after partial initialization
ice_if_init marks DRIVER_INITIALIZED only after all queue and filter
operations succeed, so ice_if_stop intentionally does nothing after an
initialization failure. Each failure path must therefore unwind any
hardware queues it may have configured before iflib releases their DMA
mappings.
Tx setup enables firmware scheduler queues one at a time, and Rx enable
similarly processes queues incrementally. Route failures from both
operations through cleanup paths for both the PF and mirror VSIs. The
cleanup helpers tolerate queues which were not configured, so they also
cover failures on the first queue.
This leaves failed initialization stopped as required by the
iflib_init_failed contract.
MFC after: 2 weeks
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59331
enetc: Correct transmit queue timeout detection
Test the producer and consumer indices after waiting for transmit
completion. The post-decrement loop leaves timeout at -1 on exhaustion,
so testing timeout == 0 missed the actual timeout and could report one
when the final poll completed successfully.
Leave the hardware shutdown sequence unchanged. This only corrects the
diagnostic; runtime FLR recovery and command-buffer lifetime handling
remain separate work.
MFC after: 2 weeks
Sponsored by: BBOX.io
mgb: Propagate DMA setup errors and use the requested channel
Check the receive DMA and frame transfer setup results that were
previously discarded. Return EINVAL for invalid ring/writeback addresses
rather than reporting success, and stop initialization through
iflib_init_failed() when a ring could not be configured.
Use the transmit queue count for transmit initialization and the supplied
channel index for DMA operations. The driver currently allocates one
queue in each direction, so those index/count corrections are latent.
MFC after: 2 weeks
Sponsored by: BBOX.io
axgbe: Propagate hardware and PHY initialization failures
Check hardware initialization instead of discarding its result, and
preserve the original initialization error across cleanup. Report a failed
init to iflib rather than publishing a running datapath. Return normalized
FreeBSD errors from attach_post. Reuse the existing native reset cleanup
on failure.
MFC after: 2 weeks
Sponsored by: BBOX.io
ixgbe: Report hardware initialization failures to iflib
Check the result of ixgbe_init_hw() before configuring queues. Report
failure through iflib_init_failed() instead of publishing a working
datapath after shared-code initialization failed.
MFC after: 2 weeks
Sponsored by: BBOX.io
ixl: Tie statistics baselines to hardware resource epochs
Raw VSI statistics belong to the firmware-assigned counter index and are
reset when firmware recreates the VSI. Reusing a baseline across either
event makes a counter decrease look like a full width rollover.
Reset the PF and main VSI baselines when rebuilding hardware resources.
Start a new baseline when firmware assigns a different PF counter index
and whenever it allocates a VF VSI. Ordinary interface reinits which
retain the hardware resources continue to retain their statistics.
This follows the lifecycle used by ice without placing an unconditional
statistics reset in ixl_initialize_vsi(), which also runs during ordinary
iflib initialization.
MFC after: 2 weeks
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59337
ixgbe: Defer flow control programming while admission is closed
Cache a validated flow control setting without touching hardware when
iflib has closed admission. The queues may still be live pending a
watchdog stop, or the device may already be stopped or suspended. RX
initialization and link setup replay the cached flow control policy.
For live updates, decide once under the context lock whether to touch
hardware, then mask MDD around SRRCTL writes regardless of a concurrent
watchdog closing admission. The context lock excludes actual stop and
reinitialization while these writes are in progress.
MFC after: 2 weeks
Sponsored by: BBOX.io
ixgbe: Apply DMAC changes through an if-up reset request
Serialize cached DMA coalescing policy with initialization and schedule
its application through the admin task. The old running check
preceded if_init() acquiring the context lock, so an intervening down
operation could be followed by an unconditional initialization.
Use the deferred if-up request so restart permission is checked when the
task runs. Changes made while stopped or suspended remain cached for
the next initialization. Do not schedule a reset for an unchanged value.
MFC after: 2 weeks
Sponsored by: BBOX.io
enic: Report device enable failures to iflib
Do not publish a running interface after vnic_dev_enable_wait() fails.
Run the existing stop cleanup for the queues configured before the enable
request, then report initialization failure to iflib.
Mark those queues as needing cleanup before submitting the enable request
so the stopped-state shortcut cannot skip the unwind.
MFC after: 2 weeks
Sponsored by: BBOX.io
enic: Clean the completion ring matching each queue
The transmit stop loop cleaned the receive completion ring, and the
receive stop loop cleaned the transmit completion ring. Use enic_cq_wq()
for transmit queues and enic_cq_rq() for receive queues, matching the
queue indices used by the remaining ring bookkeeping.
MFC after: 2 weeks
Sponsored by: BBOX.io
ice: Honor iflib transmit completion batching
ice marked every transmitted packet RS and recorded every last
descriptor in its report-status queue. Hardware therefore wrote
descriptor status for every packet, and the driver traversed every
packet while reclaiming completed descriptors.
iflib marks selected packets with IPI_TX_INTR as completion
checkpoints. It forces a checkpoint as deferred work or ring pressure
grows. Retain EOP on every packet, but set RS and record the descriptor
only at those checkpoints.
DPDK uses the same sparse-RS design and defaults tx_rs_thresh to 32.
Let iflib choose the adaptive interval for FreeBSD. This reduces PCIe
and memory traffic while preserving bounded descriptor reclamation.
Validated on an E810-XXV in an A-B-A test with five matched
four-stream, TSO-disabled transmit runs per phase. Median throughput
was 9.413, 9.413, and 9.414 Gbps. Median whole-system CPU was 21.54%,
[11 lines not shown]
ice: Defer RDMA critical error notifications
ice_msix_admin() runs as an interrupt filter inside a critical section.
ice_rdma_notify_pe_intr() acquires the global RDMA sx and invokes the
client event handler, both of which require sleepable thread context. A
PE or HMC critical error could therefore panic under WITNESS or sleep
from interrupt context.
Accumulate OICR causes atomically in the interrupt filter and mark them
pending in the driver state. Deliver the notification from the iflib
admin task before processing reset events. This preserves the existing
ordering, lets an iRDMA-requested reset run in the same admin pass, and
coalesces causes from multiple interrupts.
MFC after: 2 weeks
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59340
ping: do a better job with what we received from the net, part 2
When we see a difference between the payload we sent and what we
received, we dump both but we were not prepared for the case when
the received payload is less than we sent. In this case we were trying
to dump more than needed.
Funny enough, 23 years ago I already fixed a similar issue here but
didn't pay attention to this small dumping loop.
Test written by jlduran.
Reviewed by: jlduran
MFC after: 1 month
Found with: Claude Code Sonnet 5
Differential Revision: https://reviews.freebsd.org/D59556
Differential Revision: https://reviews.freebsd.org/D59582
