[VPlan] Compute the backedge-taken count with an explicit recipe. (#236655)
Instead of a symbolic VPValue for the backedge-taken count, which is
materialized after dissolving the loop regions, compute TC - 1 directly
in the vector preheader when creating the header mask that compares
against it, together with an explicit broadcast.
This removes VPlan::BackedgeTakenCount (VPlan)global state.
[SPARC] Do not fill delay slots at -O0 (#235248)
At -O0 the delay slot filler still moves the instruction before a call
into its delay slot, and at -O0 that's typically the spill of an
argument. A breakpoint on the line of the call stops on the `call`
before the store has run, so the debugger reads stale stack. GCC only
fills delay slots from -O1, and the Mips filler skips -O0 as well.
This skips the optional fill, and the restore folding, at -O0.
Two fills are needed for correctness and are kept at every level: the
restore before a tail call has to be in its delay slot (a jmp %reg reads
its target from the current window, a call writes %o7), and so does the
mov %g1, %o7 before a leaf tail call.
Until now only the optional search put them there, so
`-disable-sparc-delay-filler` produced leaf tail calls that returned
into the next function.
[33 lines not shown]
[LoopUnroll] Use DenseMap::contains instead of count (NFC) (#224529)
Replace count()-then-lookup style membership checks with contains() in
analyzeLoopUnrollCost for readability.
iflib: Use private lifecycle state for control work
Use the context-locked datapath state for netmap initialization results,
resume completion and administrative work eligibility. Keep stopped
and failed interfaces eligible for link and mailbox recovery without
using IFF_DRV_OACTIVE as an implicit indication that initialization has
been attempted. Evaluate media and deferred admin eligibility under
the context lock rather than from an earlier IFF_DRV_ flag snapshot.
Always pass through iflib_stop() before an IOV-init callback which
changes the PF queue layout, as the IOV-uninit counterpart already does.
Neither administrative down nor a cleared IFF_DRV_RUNNING proves that
queue DMA is stopped. The private lifecycle state decides whether the
hardware stop is necessary; IFF_UP only decides whether to initialize
again afterward.
Reviewed by: iflib (gallatin)
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D59597
[2 lines not shown]
ixv: Use host granted RSS queues on Hyper-V
Discover the Hyper-V queue grant through PCI configuration space for
X550, X552, X553, and E610 VFs. The X550 family exposes a byte at 0x207;
0x208 is a separate DCB flag. E610 exposes a little-endian queue count
word at those offsets. Accept grants of one, two, or four and use at most
two symmetric queue sets, subject to the existing MSI-X resource limit.
Keep one queue for invalid grants and for 82599/X540 Hyper-V VFs.
The family gates and queue limits are snooped from Windows Intel VF bus
traffic.
Update the shared code queue max so stop disables every usable queue.
Also program the receive length limit on every usable RX queue instead
of only queue zero and retain the 82599 exception for that register field.
Sponsored by: BBOX.io
(cherry picked from commit a54f83544eea2a9804ac7940d35b26ffcb34dfb3)
ixv: Recover E610 Hyper-V carrier without the PCI link query
Current Windows PF drivers do not emulate the E610 link-status query at
PCI configuration offset 0x209. The installed Intel ixw 1.8.54.0 driver
returns zero even with the physical port connected, leaving the VF at
no carrier after an otherwise successful hn handoff.
When the query returns zero, use VFLINKS for carrier only. Its speed
field is not the negotiated E610 line rate: the tested 1 Gb/s port
reports the default 10 Gb/s encoding. Publish an unknown speed instead,
including in the verbose link-up diagnostic, and document this fallback.
Keep nonzero PCI results authoritative and reject all-ones reads. Other
Hyper-V families and the native E610 mailbox path are unchanged.
A zero result cannot distinguish an unsupported query from a down link;
in either case this fallback relies on the legacy carrier indication.
The connected and disconnected ports reported VFLINKS.UP set and clear,
respectively, while both returned zero from the PCI query.
[3 lines not shown]
ixv: Add Hyper-V virtual function support
Hyper-V does not expose the normal Intel PF/VF mailbox.
Import the Hyper-V operation overrides from DPDK and complete its reset
callback with the configuration-space mechanism submitted by Microsoft.
Route ixv operations through the operation table, use mailbox API 1.0,
and limit this environment to one queue pair.
Claim the 82599, X540, X550, X550EM-X, and X550EM-A Hyper-V device IDs
listed by DPDK. Select the same path for the E610 Hyper-V subdevice
identity defined there.
For E610/Linkville, read the emulated VFLINKS-format status from PCI
configuration space at offset 0x209. Older families retain the MMIO
VFLINKS path.
Treat receive-mode changes as host-owned no-ops and do not retry VLAN
operations that Hyper-V permanently rejects. Limit the Hyper-V 82599
[15 lines not shown]
hyperv: Configure IPv6 when setting up VF failover
The default VF-up script moves IPv4 configuration from the synthetic
interface to a failover lagg, but leaves IPv6 unconfigured on the lagg.
Adding the synthetic interface as a member can remove its IPv6 addresses,
so deleting the remaining member addresses is not sufficient.
Stop accepting router advertisements on the synthetic interface before
adding it to the lagg. Leave its link-local address and IPv6 enable state
in place for lagg's address, prefix and default-router cleanup, then disable
IPv6 and explicitly remove any remaining addresses. This also covers
members without a link-local address or with IPv6 already disabled.
Replay the synthetic interface's IPv6 rc.conf configuration on the lagg,
including aliases, prefix-derived addresses and legacy configuration names.
Remap the variables in a subshell and reuse network.subr's IPv6 helpers.
For SLAAC, solicit fresh router advertisements rather than copying learned
addresses as permanent ones. Configure IPv6 independently of the IPv4
DHCP/static choice, and leave existing laggs alone on repeated invocation.
[7 lines not shown]
hyperv: Bound IPv4 address removal when configuring VF failover
The VF-up script repeatedly deletes the first IPv4 address from hn until
ifconfig fails. Netlink-based ifconfig returns success when no address
remains, so the script can loop forever and block subsequent devd work.
Use ipv4_down() from the already-sourced network.subr to enumerate the
addresses and delete each explicitly. This also handles multiple
addresses without relying on the exit status of an empty deletion. The
behavior depends on ifconfig, not the VF hardware, so no device-specific
fallback is needed.
Fixes: c68595695679 ("hyperv: Add VF bringup scripts and devd rules.")
Sponsored by: BBOX.io
(cherry picked from commit 90f7956ff61f537def0d0aeb6ee03605a671d57e)
hn: Invoke the owned VF's transmit queue flush callback
The generic if_qflush() only clears if_snd. Calling it directly for a
transparent VF bypasses the driver's callback and its private software
transmit queues. Add if_getqflushfn(), matching the existing callback
accessors, and invoke the registered callback while hn_vf_lock protects
the VF pointer.
Flush an attached VF whenever transparent mode owns it, not only while
the VF datapath is enabled. Disabling transmit during handoff does not
discard packets already queued to the VF. This also covers the inactive
interval after fallback. Non-transparent VFs remain independently owned
and are not flushed by hn. Synthetic queue flushing is unchanged.
Sponsored by: BBOX.io
(cherry picked from commit 8e816fc2e98c8c0bf0b6fef3bd5f52bc03c843ce)
hn: Serialize VF link events with datapath handoff
Hold hn_vf_lock across the VF identity/state check and link publication.
Otherwise an event can observe an enabled VF, pause during fallback, then
overwrite the freshly reported synthetic carrier with stale VF state.
Ignore events while switching or after the association generation changes.
The link notification only queues further work, so the callback does not
need the sleepable hn lock.
Document that association serial numbers are currently diagnostic:
hn_ismyvf() matches the VF by MAC, while notifications gate availability.
Sponsored by: BBOX.io
(cherry picked from commit 6b3b1d6d4f43d1521e5e10b759e3542deaf23667)
hn: Track VF association and wait for datapath switch completion
Handle the host's VF association notifications instead of ignoring them.
Require an allocated association before switching to the MAC-matched VF,
and track notification generations so a withdrawal during initialization
cannot enable an obsolete handoff. Defer association and address-event
work to the VF taskqueue; the receive channel must remain available to
deliver switch completions.
Request and wait for the empty VMBus completion for SET_DATAPATH, checking
submission and channel-revocation failures. Enable transparent VF
transmit and select its link status only after the switch completes.
Use the existing transaction lifetime and revocation handling, without a
timeout that could leave a late completion referencing a freed request.
Restore synthetic capabilities, TSO limits and hardware-assist flags on
fallback, targeting hn rather than the departing VF. Block transparent
transmit during handoff and after association withdrawal. Separate the
attach delay from saved-setting readiness, permit delayed association to
[7 lines not shown]
hyperv: Stop bus relations handling from falling through to eject
The PCI_BUS_RELATIONS2 case lacks a break after queuing the reported
devices. It consequently processes the same packet as PCI_EJECT,
interpreting device_count as the slot number. If that value matches an
existing child, the callback schedules an unintended device eject.
End the relations case after processing the device list, matching the
original PCI_BUS_RELATIONS handling.
Fixes: ea11861e434a ("arm64: Hyper-V: vPCI: Enabling v-PCI in FreeBSD in ARM64 Hyper-V")
Sponsored by: BBOX.io
(cherry picked from commit ddcf8be82b0f17c767d2edab026a4f5cfa3d217e)
hn: Refresh VF offloads after bringing it up
VF readiness can precede its first initialization. In particular,
iflib populates if_hwassist during init, so copying it while the VF is
still down can leave hn advertising checksum and TSO capabilities with
no hardware-assist flags. The stack then uses software segmentation
until a capability ioctl happens to refresh the copied state.
Synchronize enabled capabilities and hardware-assist flags after the VF
up ioctl succeeds, before switching the datapath and enabling transparent
VF transmit. Use the existing helper in the common initialization path,
covering both normal bring-up and delayed VF initialization.
Sponsored by: BBOX.io
(cherry picked from commit 7c43b915de433bf7a9988faccd6e638f02e47fea)
ixgbe: Leave run-state publication to iflib during FDIR recovery
Rebuilding the Flow Director tables does not initialize the interface
or restore its queues. Do not set IFF_DRV_RUNNING from that operation:
iflib owns the flag, and may have cleared it while a watchdog reset is
pending. Restoring it here could admit traffic before the deferred
stop and initialization have run.
Keep the table rebuild and Flow Director interrupt re-enable unchanged.
This path is conditional on IXGBE_FDIR.
Sponsored by: BBOX.io
(cherry picked from commit 3a4e037989f91672974eeedf1fa81ce0b684beb7)
ice: Use the mirror interface softc in PF callbacks
PF iflib callbacks receive struct ice_softc, not struct ice_mirr_if.
Resolve the mirror interface through sc->mirr_if before checking or
resetting subinterface state.
Use the same mirror softc when rebuilding its VSI. This records the
required subinterface reset in the state consumed by the PF callback,
rather than overlaying the PF softc and leaving rebuilt queues stopped.
Sponsored by: BBOX.io
(cherry picked from commit 33be59b660befb13cdcc38e52b1c9b4119a6a069)
ixl: Set critical-error state bits independently
ixl_set_state() takes a bit index, not a bit mask. ORing the
reset request and critical error indices produced the global reset
index, so a critical interrupt could mask its cause without scheduling
the intended PF reset.
Set both state bits explicitly.
Sponsored by: BBOX.io
(cherry picked from commit 372803d65d242b308abd6b312a40ff6987fc3452)
vmxnet3: Propagate device-enable failure to iflib
Do not let iflib publish a running interface when the virtual device
rejected its enable command. Mark initialization failed and leave the
interface stopped.
Sponsored by: BBOX.io
(cherry picked from commit 99eaf07a3cef77e61423e9431c0a8c5fd4d4be58)
iflibdd(9): Clarify resume during suspend rollback
The resume callback also runs when suspend fails, without an intervening
PCI power-state transition or configuration-space restore. Document
that drivers must not assume either has occurred.
Sponsored by: BBOX.io
(cherry picked from commit f496cd95cf9e1ed68b0df3ae2f7a24fc0f66aa6b)
ixgbe: Pause firmware recovery polling in low power
The firmware recovery callout continues polling controller registers
after a power transition. An inaccessible E610 GL_MNG_FWSM register
reads as all ones in D3, which looks like firmware recovery mode and
queues an iflib reset while the device is suspended. A later D0 poll
then reports recovery complete and queues another reset.
Pause and drain the callout before terminal stop policy is applied,
prevent an in-flight callback from rearming it, and restart polling
only after resume has cleared the wake state. Track callout
initialization so partial attach cleanup does not drain an uninitialized
callout.
The false transition was reproduced on a dual-port E610 with direct D3
and system S3. Validate the guard with wake-disabled and wake-armed D3,
three repeated D3 cycles per port, and an S3 magic-packet wake. Both
ports returned to D0 without a false recovery transition.
[3 lines not shown]
iflib: Reject media changes during suspend
iflib gates its built-in administrative and media-status callbacks once
a power transition starts, but iflib_media_change() could still invoke a
driver while the device was suspending or suspended. Several drivers
perform PHY or firmware I/O directly from this callback.
Return EBUSY before invoking IFDI_MEDIA_CHANGE() unless the device is
active. ifmedia then restores the prior selection, avoiding both
suspended hardware access and an unvalidated configuration that would
need to be replayed during resume.
Validated with device suspend on 82579LM, I210, and I225-IT
controllers. Media-selection requests returned EBUSY on every
suspended device. Resume restored the linked management interfaces at
1 Gbps with working traffic and no watchdogs; unconfigured interfaces
retained their prior admin and link state.
Reviewed by: iflib (gallatin)
[4 lines not shown]
iflib: Own queue quiescence during power transitions
Perform a terminal datapath stop before suspend and shutdown
callbacks, then drain the private configuration taskqueue before
entering low power. Track power state independently of queue
ownership and prevent built-in admin, IOV, LED, and media-status
callbacks from accessing a suspended device.
Restore driver-specific state while the datapath remains stopped.
Initialize it exactly once on resume when the interface is
administratively up, and keep an administratively-down interface
stopped. Roll back the driver when suspend or child suspension fails.
Add ifdi_power_prepare() for policy which must be established before
the terminal stop. Use it to snapshot ixgbe(4) wake policy and preserve
X550EM PHY ordering, and remove the duplicate stop from aq(4).
Reviewed by: iflib (gallatin)
Sponsored by: BBOX.io
[3 lines not shown]
iflib: don't update the admin status in if_media_status()
When _task_fn_admin() is active, it will regularly call
IFDI_UPDATE_ADMIN_STATUS(). So there is no need to do it in
iflib_media_status. This can be fairly expensive on some drivers (long
DELAY busywait loops waiting for a NIC command), and there is no need
to pause a userspace app in this DELAY() if it is happening
asynchronously anyway.
Note the logic to detect if _task_fn_admin() is regularly calling
IFDI_UPDATE_ADMIN_STATUS() was copied from that function.
Reviewed by: erj, kbowling
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D54096
(cherry picked from commit ebb93ce6d86e6c6abd9d9fada21a3b107099db8f)
iflib: Track queue datapath lifecycle
Track whether iflib queue mappings may still be accessed by the
device. Keep the state private to iflib and conservative: an unknown
or failed device must pass through IFDI_STOP() before mappings are
reused or released, while a device known to be stopped need not
receive another hardware stop.
Enter the starting state before IFDI_INIT(), publish running only
after receive buffers and framework state are ready, and stop hardware
if receive-buffer setup fails after driver initialization.
Do not initialize an administratively-down interface merely because
its MTU, capabilities, VLAN configuration, or media changed. Preserve
successful retries for an administratively-up interface whose previous
initialization failed.
This state describes ownership of iflib datapath mappings only. It
deliberately makes no claim about firmware queues, administrative DMA,
[7 lines not shown]
[llvm-mc] Report more errors through the SourceMgr (#236281)
Avoid printing to errs() directly and use the SourceMgr instead like
all the other assembler and disassembler diagnostics.
This is needed for the upcoming `--verify` option: without this, these
options interfere with the exit code from --verify and can't be matched.
Mostly NFC, although the errors now print without a `llvm-mc:` prefix.
Assisted-by: AI tooling
Pull-Request: https://github.com/llvm/llvm-project/pull/236281
[llubi] Add support for llvm.masked.gather (#235986)
Re-use the logic for llvm.masked.load to also support
llvm.masked.gather.
Instead of computing consecutive addresses from a base pointer, enabled
lanes are loaded from the pointer of the lane. Masked-off lanes are
taken from the passthru operand and are not accessed.
PR: https://github.com/llvm/llvm-project/pull/235986
[Support][mlir] Clean up list of unmatched @unknown directives (#236282)
mlir-opt calls `verify()` twice (explicitly and in the handler's
destructor), and `verify()` clears the per-file directive lists, but
did not clear the list of `@unknown` ones.
This fixes reporting unmatched `@unknown` twice.
Assisted-by: AI tooling
Pull-Request: https://github.com/llvm/llvm-project/pull/236282
