[VPlan] Narrow VPWidenGEPRecipe if only its first lane is used. (#221171)
Extend existing narrowing in legalizeAndOptimizeInductions, already
applied to VPReplicateRecipe and VPWidenRecipe, to VPWidenGEPRecipe as
well. This improves cases where an interleaved access's insert position
shares an address calculation with a strided access (see
RISCV/interleaved-load-masked-store.ll), which previously caused the
legacy cost model to classify the address as non-uniform.
iflib: Remove an unused field from struct iflib_rxq
Reported by: Alexander Sideropoulos <Alexander.Sideropoulos at netapp.com>
MFC after: 1 week
(cherry picked from commit fc09c7fee23b3cf3ddc95105ef6ef41d7956232f)
[SLP] Limit second pass to register VF in vectorizeNonVectorizableInsts. (#222755)
The second tryToVectorizeList pass in vectorizeNonVectorizableInsts
would try all roots for each possible VF between 1 and the number of
entries in the list.
This can cause super-linear compile-time, for example when there are
basic blocks with calls taking a large number of loads as arguments.
For example, running SLPVectorizer on a block with 1024 loads passed to
calls (https://llvm.godbolt.org/z/8M53G6WzW) will take a large amount of
time (timeout on godbolt, locally on Apple M1 it takes ~30s). With the
fix, it only takes 0.01s.
On large IR corpus, I did not see any difference in vectorization
decisions on AArch64.
PR: https://github.com/llvm/llvm-project/pull/222755
iflib: Permit SR-IOV configuration on a down interface
Drivers which remap PF queues need a stop/mutate/restart transaction
only when the interface has live queues. Permit their IOV
initialization callback while the interface is administratively down
and leave it down afterward.
This restores the standard boot-time iovctl.conf workflow and
lets other opt-in drivers configure VFs before netif brings the PF up.
(cherry picked from commit 2cf580c694f6f392531a63f01c3fb89c0244f89a)
iflib: Add restart transactions for IOV reconfiguration
Some devices remap the PF queues when entering or leaving SR-IOV. Add
opt-in PCI IOV helpers that hold the iflib context lock across the
complete stop, driver callback, and restart transaction.
Existing drivers continue to use the non-restarting helpers.
Sponsored by: BBOX.io
(cherry picked from commit f8fa2d77bc305bec519f9f02afe211e903c57573)
[flang] Do not hoist fir.field_index out of loops
Lowering a consumer of a !fir.field value inspects its defining operation:
for a record whose layout is known at compile time the field becomes an LLVM
GEP struct index, which must be a constant, and otherwise the `field`
attribute is read off the defining op. A field value therefore may not be a
block argument.
LICM broke that. fir.field_index is Pure and takes no operands, so it is
trivially loop-invariant and was hoisted out of the loop. Lowering emits one
inside each arm of a construct -- for example the CASEs of a SELECT CASE that
each pass a different component of the same derived type as an actual
argument -- so hoisting them left those arms as otherwise-identical blocks
differing only in that operand. Block merging then merged them and threaded
the field through a new block argument, and codegen aborted with "must be a
constant".
Leave producers of a !fir.field where they are. The arms then differ by an
operation rather than by an operand, so they are no longer merge candidates.
[7 lines not shown]
iflib: Initialize the VFLR task unconditionally
The VFLR task was initialized only from drivers MSI-X interrupt
assignment paths. ixl's legacy interrupt handler can nevertheless defer
VFLR work, leaving an uninitialized task. Even with MSI-X, the admin
interrupt was established before the task was initialized.
Initialize it alongside the other private tasks. The existing detach
check and private-taskqueue drains then cover its lifecycle for every
interrupt mode and registration failure.
Sponsored by: BBOX.io
(cherry picked from commit b4208a67edc2eb7898a9ff2a6f3990c6852910e4)
iflib: Add an admin task detach fail point
Add an exact-device fail point immediately after the admin task checks
IFC_IN_DETACH. This makes the detach race reproducible without affecting
another interface.
Use a bounded delay to keep the task active while detach enters the
taskqueue drain. Mark the point nonsleepable as a safety backstop, and
document a one-shot test for verifying that deregistration drains an
already-running task before ether_ifdetach().
Reviewed by: gallatin, kgalazka
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D58720
(cherry picked from commit ac56d36007a5a1a01fe69df370f272060e852e0b)
iflib: Drain configuration tasks before interface detach
iflib_device_deregister() sets IFC_IN_DETACH before removing the
interface, but a task which already passed its detach check can still
report a link change. This can re-arm if_linktask after
ether_ifdetach() has drained it and leave work pending across queue
teardown.
Drain the entire private taskqueue before ether_ifdetach(). Drivers
may register their own link-related configuration tasks there, so
draining only the framework admin task leaves the same race for those
drivers.
Differential Revision: https://reviews.freebsd.org/D58452
Co-authored-by: Andrew Gallatin <gallatin at FreeBSD.org>
Co-authored-by: Kevin Bowling <kbowling at FreeBSD.org>
(cherry picked from commit ba353c8950d575f9d15b82c92658e660935fba25)
iflib: Add registration failure injection points
Add six device-scoped fail(9) points at the registration milestones
needed to exercise each unwind path. An exact, runtime-only device
selector prevents unrelated iflib devices from consuming an armed point.
Mark the points non-sleepable because registration holds the ifnet and
context locks. Document one-shot operation and bus-address reprobe so a
failed attach can be recovered without another kernel build.
Reviewed by: gallatin
Sponsored by: BBOX.io
Differential Revision: https://reviews.freebsd.org/D58722
(cherry picked from commit 90e7dbe5e2ca47baff4e4c6d9e892a0554eec4db)
iflib: Allow conditional LED device support
A driver class may implement LED control even though the capability is
not available on every device or firmware version it supports. Add an
optional capability method and consult it before creating the led(4)
device. Default to supported so existing providers are unchanged.
This will be used by bnxt which blends PF and VF in the same driver.
(cherry picked from commit 2519e19f05e0c3e5925bf81b729b4c28f2ad1af6)
iflib: Create led(4) devices
When a driver implements ifdi_led_func, have the framework create its
led(4) device after attach completes and the ifnet and context locks are
released.
PR: 246885
Reported by: jlduran
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D32389
(cherry picked from commit 6591a7f6919295f2ec2b463d1ae9554a8bbf6104)
[libc] Implement pthread_attr_[gs]etschedpolicy (#222682)
This patch implements pthread_attr_setschedpolicy and
pthread_attr_getschedpolicy.
This commit only operates on the pthread_attr_t object. It does not
attempt to install the scheduling policy when creating a new thread. I'm
leaving that for a separate patch as it requires a moderately
complicated startup dance to ensure that the scheduling policy takes
effect before the startup code runs.
The validation of inputs in pthread_attr_setschedpolicy is an
interesting question. Glibc accepts only policies explicitly declared in
POSIX, while other implementations let the user pass anything, and rely
on the kernel to validate the arguments. Interestingly, even glibc does
not validate the arguments in pthread_setschedparam.
For llvm-libc, I chose not to validate the arguments either. This is
*mostly* consistent with POSIX, which says (emphasis mine):
[11 lines not shown]
iflib: Defer LED control to the device taskqueue
led(4) invokes driver callbacks while holding its mutex, including
from a callout. iflib_led_func() cannot acquire the sleepable context
lock in those contexts without causing a lock-order reversal or sleeping
from the callout.
Record the latest requested state under the iflib state lock and
enqueue the existing per-device taskqueue. The task can safely take
the context lock before invoking the driver. Coalescing requests also
avoids accumulating stale blink transitions when hardware access is
slow.
Destroy the LED device before draining its task so no new callback can
race driver detach.
(cherry picked from commit 952994751911d5d059d53e73eb874e00ee98b9ed)
[flang][OpenACC] Don't emit unstructured-loop TODO for wrappable loops (#222260)
`genOpenACCLoopFromDoConstruct` raised an NYI for any unstructured DO
inside `acc kernels`. That check ran before the wrapping in
`genFIR(DoConstruct)`, so a loop whose CFG is self-contained aborted
lowering even though the fall-back path would have wrapped it in an
`scf.execute_region`.
Skip the TODO when `isWrappableConstruct()` accepts the loop. Such a
loop is not attached to a directive; it is only nested inside the
kernels region.
Correspondingly, a DO loop sitting directly in an `acc kernels` body is
no longer wrappable, since it is the loop the directive parallelizes and
hiding its CFG would leave the kernels region with nothing to partition.
bnxt: Add led(4) identification support
Query the firmware for the LEDs on each physical port and expose
/dev/led/bnxt* only when alternate blinking is supported. Configure
every LED in the advertised group for identification and restore its
default firmware state before a function reset.
This follows the DPDK and Linux bnxt HWRM identification paths.
Reviewed against: DPDK, Linux
Reviewed by: Sumit Saxena <sumit.saxena at broadcom.com>
Sponsored by: BBOX.io
(cherry picked from commit 98b5640786f874949e441bee7c4ad5740cd6a649)
bnxt: Report initialization failures to iflib
HWRM failures currently return from the void ifdi_init callback.
iflib then marks the interface running and enables interrupts despite an
incomplete ring or VNIC setup.
Move the hardware setup into an error-returning helper. The ifdi_init
wrapper can report failure through iflib_init_failed(), while firmware
recovery can propagate the same error through bnxt_open(). Also clear
the initialized state after partial setup is torn down.
(cherry picked from commit cb0e8f5fb6de03428137c096b1c8b62710829029)
[clang-tidy][docs] Rename readability check docs to Markdown [3/5] (#221533)
Tracking issue: #201242
See the [migration guide] for more information.
[migration guide]:
https://llvm.org/docs/SphinxQuickstartTemplate.html#markdown-migration-guidelines
This is the third part of the mechanical rename of the check
documentations in readability module.
The rewrite is provided by the next PR in this stack.
ix(4): Sanitize negative error codes
Due to development history FreeBSD driver error codes are reported
the same way as in Linux (as negatives) which is inconsistent
with FreeBSD standard. It may cause unexpected behavior when driver
errors are interpreted by a kernel as syscall handler return values.
This patch converts error codes from negative to positive values for
NVM access functions.
Signed-off-by: Pawel Sobczyk <pawel.sobczyk at intel.com>
Reviewed by: kbowling, erj, milosz.linkiewicz_intel.com
Tested by: Mateusz Moga <mateusz.moga at intel.com>
Sponsored by: Intel Corporation
Differential Revision: https://reviews.freebsd.org/D57642
(cherry picked from commit 33e2eac3e3e738daa95a06f42d6c661b87ad9aac)
ixgbe: Add missing mailbox API 1.6 definition
The SR-IOV status change reports mailbox API 1.6 but omitted its enum
definition, leaving main unable to compile.
API 1.6 is an established ixgbe mailbox wire revision. Add it at the
end of the revision enum, before the unknown sentinel as required by
the stable numbering contract. Naming the revision does not enable
negotiation or operations which will come with the E610 support.
Reported by: Herbert J. Skuhra <herbert at gojira.at>
(cherry picked from commit 6a1703c112ff2904feb39288f0746c8f0a04f938)
ix(4): Remove workaround for 2.5/5G speeds on E610
The problem observed on X550 adapters with 2.5 and 5 Gbps speeds
negotiation on some switches is not affecting E610 adapters.
Remove workaround, which omitted those speeds in the list
of initially advertised speeds and advertise all speeds
supported by adapter.
Signed-off-by: Krzysztof Galazka <krzysztof.galazka at intel.com>
Reviewed by: kbowling
Tested by: Mateusz Moga <mateusz.moga at intel.com>
Sponsored by: Intel Corporation
Differential Revision: https://reviews.freebsd.org/D57339
(cherry picked from commit 62d5d119ee7d935ac05966f1c7c4333c33c4f3a9)
ixgbe: Probe additional controller variants
The shared base code already selects and configures the 82598 BX,
82599 KR, 82599 SFP Express Module, X552 XFI, X553 QSFP, and X553 N
QSFP device IDs, but the FreeBSD probe table omits them while DPDK lists
them.
Sponsored by: BBOX.io
(cherry picked from commit 27f0491e3648722a892558416a768dda77850ad9)
ixgbe: Enable per-pool RSS on X550 family devices
X550 family devices provide a separate RSS key, redirection table,
and MRQC register for every VMDq pool. With SR-IOV enabled, the
driver continued programming only the global RSS state and never
selected MRQC.MULTIPLE_RSS. VF-local RSS programming was therefore
ineffective.
Enable multiple-RSS mode for X550, X552, X553, and E610. Initialize
the PF pool's 64-entry key, redirection table, and RSS hash controls.
Leave each VF pool untouched so its driver retains ownership of its
RSS key and mapping.
E610 folds IPv6 extension-header traffic into its base RSS selectors
and reserves the legacy EX selector bits. Translate those requested
hash types rather than programming reserved bits.
With two E610 VFs active and four PF queue sets, eight fixed TCP flows
distributed across all four PF receive queues.
[8 lines not shown]
iflib: Fix panic observed while doing sysctl -a with if_bnxt unload
Observed below kernel panic calltrace while performing sysctl -a
operation while unloading the if_bnxt driver,
Fatal trap 9: general protection fault while in kernel mode
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe02a7569940
vpanic() at vpanic+0x136/frame 0xfffffe02a7569a70
panic() at panic+0x43/frame 0xfffffe02a7569ad0
trap_fatal() at trap_fatal+0x68/frame 0xfffffe02a7569af0
calltrap() at calltrap+0x8/frame 0xfffffe02a7569af0
trap 0x9, rip = 0xffffffff80c0b411, rsp = 0xfffffe02a7569bc0, rbp = 0xfffffe02a7569be0 ---
sysctl_handle_counter_u64() at sysctl_handle_counter_u64+0x61/frame 0xfffffe02a7569be0
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x9c/frame 0xfffffe02a7569c30
sysctl_root() at sysctl_root+0x22f/frame 0xfffffe02a7569cb0
userland_sysctl() at userland_sysctl+0x196/frame 0xfffffe02a7569d50
[23 lines not shown]
iflib: drain admin task and fix teardown order on register failure
When IFDI_ATTACH_POST() fails (or netmap attach fails), iflib tears down with
ether_ifdetach(), taskqueue_free(ifc_tq), and IFDI_DETACH(). CTX_LOCK is still
held after ether_ifattach. ether_ifdetach() and taskqueue_drain(admin) must not
run under CTX_LOCK.
Teardown ordering (match iflib_device_deregister):
- Free the per-interface admin taskqueue after IFDI_DETACH / IFDI_QUEUES_FREE, not before.
- Drop IFNET_WLOCK() across IFDI_DETACH / IFDI_QUEUES_FREE so driver detach can sleep in
LinuxKPI workqueue drain, then retake IFNET_WLOCK() before iflib_free_intr_mem and fail_unlock.
Reviewed by: gallatin, kgalazka, #iflib
Differential Revision: https://reviews.freebsd.org/D56316
(cherry picked from commit 439132310ae1f623f6c0a3dc241d0a34e98e040b)
ixv: Preserve statistics across resets
The VF statistics registers are free running and are not cleared on
read. The existing code records attach time bases and pre-reset totals,
but never uses either when publishing counters. It instead replaces
the low hardware bits directly, so counters can inherit pre-attach
traffic or jump backward after a reset.
Accumulate modular 32- and 36-bit deltas, following DPDK, while keeping
the software totals across planned resets. Establish a fresh hardware
baseline after each successful reset and invalidate the sampling epoch
when mailbox state is lost. Detect unsolicited PF resets explicitly so
a reset while link is down cannot be mistaken for counter wrap.
Remove the unused base and saved-reset bookkeeping.
On E610, packet and octet counters remained monotonic across a VF FLR
and a PF down/up cycle. Traffic after each reset advanced both RX and
TX counters.
[4 lines not shown]
