libthr: Cache recent pshared_key
For a pshare key, we can actually cache the last one as a fast path
instead of trapping into kernel directly. It makes sense as a lock and
unlock should be a pair on the same key and at the same time. For
example, lock(A) -> unlock(A) -> lock(B) -> unlock(B). Each unlock is
a hit and refresh later (50% hit).
For a single pshare lock, it is evne a 100% hit as the key never
changed.
For nested lock lock(A) -> lock(B) -> lock(C) -> unlock(C) ->
unlock(B) -> unlock(A). It only cache the last one. However, we should
really think about rewritting this logic as it is a pshare lock shared
by many processes and nested should be uncommon.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D60957
gve: Page-align DQO descriptor and completion rings
Currently, gve_rx_alloc_ring_dqo() and gve_tx_alloc_ring_dqo() request
CACHE_LINE_SIZE alignment when allocating DMA memory for the DQO TX/RX
descriptor and completion rings. The device requires stronger alignment
than cache line alignment, which was incidentally being met due to how
the kernel allocates memory. Explicitly meet this requirement by
requesting page aligned memory.
Signed-off-by: Tim Hostetler <thostet at google.com>
Reviewed by: markj, Jasper Tran O'Leary <jtranoleary at google.com>
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D61211
gve: Fix AdminQ buffer DMA sync directions
In gve_verify_driver_compatibility(), the driver populates the
gve_driver_info buffer for the device to read. Sync this buffer with
BUS_DMASYNC_PREWRITE instead of BUS_DMASYNC_PREREAD before issuing
the command.
In gve_adminq_describe_device(), the device populates the
gve_device_descriptor buffer for the driver to read. Sync this buffer
with BUS_DMASYNC_PREREAD instead of BUS_DMASYNC_PREWRITE before
issuing the command.
In gve_adminq_get_ptype_map_dqo(), the device populates the
gve_ptype_map buffer for the driver to read. Add the missing
BUS_DMASYNC_PREREAD and BUS_DMASYNC_POSTREAD syncs around the
command.
In normal GCP environments this did not cause a functional failure
because the underlying platforms are DMA cache-coherent and
[12 lines not shown]
ioctl: Add a macro for extracting the ioctl number from a command
Sprinkle it in various drivers which hard-coded the command layout.
No functional change intended.
Reviewed by: dumbbell, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D61045
libsecureboot: add support for more signatures
For 2027 we need to support new, stronger signatures and hashes.
Re-work Makefile.inc and local.trust.mk to define
VE_SIGN_URL_LIST or VE_SIGNATURE_LIST (can be derrived from
VE_SIGN_URL_LIST) containing tuples of sig/method/hash/flags and
for VE_SIGN_URL_LIST the same plus /url.
From VE_SIGNATURE_LIST we derrive VE_SIGNATURE_EXT_LIST used to populate
signature_ext_list which defines the order in which we look for
signatures for a manifest.
We also populate signature_info_list which maps a signature ext to the
needed hash and flags to indicate whether that hash needs to be repeated.
verify_sig will lookup signature_info_list, no match means not
supported. For asc we just call verify_asc() otherwise we call
verify_ec() or verify_rsa() depending on the key type found in the
[15 lines not shown]
nfs_clrpcops.c: Fix typo in commit db8444c1b7e4
Reported by: markj
MFC after: 1 month
Fixes: 884ee8d6c9b4 ("nfscl: Add some glue for client side NFS over RDMA")
LinuxKPI: set pci_dev msi_cap and msix_cap
Linux drivers find their MSI and MSI-X registers through pdev->msi_cap and
pdev->msix_cap, which hold where those capabilities start in PCI config
space. LinuxKPI never set them, so they stayed 0, and every access meant
for the MSI registers went to the start of config space instead.
Drivers using these fields, e.g. ath11k and amdgpu, are affected.
Fill in both fields from the capability offsets that the native PCI code
already records.
Reviewed by: dumbbell
Differential Revision: https://reviews.freebsd.org/D61125
Event: NetApp/Vendor Summit Hackathon 202611
mandoc: Drop obsolete files
These were removed by upstream long ago. Nothing in the
tree uses them. Remove them to reduce our diff to upstream.
Reviewed by: carlavilla, wosch
Reported by: Ingo Schwarze <schwarze at openbsd.org>
Differential Revision: https://reviews.freebsd.org/D61198
virtio_pci_modern: enable virtqueues before completing attach
Most of our drivers setup interrupt(s) and virtqueues are enabled at
that time, but nothing will currently enable them for devices that we're
only going to poll. This most notably affects virtio-random, and the
problem can be observed with qemu's q35 emulation and a random device
attached. The q35 implementation ignores the ring unless it's enabled,
so vtrnd_harvest() will consistently EAGAIN rather than contribute
anything of value, but with this patch it becomes functional.
This can be reproduced with a qemu invocation like so:
qemu-system-x86_64 -machine q35,accel=tcg -cpu max -smp 2 -m 2G \
-drive file=FreeBSD-16.0-CURRENT-amd64-ufs.qcow2,if=virtio \
-device pcie-root-port,id=rp1,bus=pcie.0,chassis=1 \
-object rng-random,id=rng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=rng0,bus=rp1 \
-display curses
[4 lines not shown]
virtio_random: stop on detach, don't poll
If the device is hung, then polling will block the detach and hang us.
Stop it and drain the queue instead. This fixes a hang on detach in
some circumstances without the upcoming virtio_pci_modern fix.
Reviewed by: bnovkov, kfv
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D61051
virtio_random: avoid using vtrnd_detach in the failure path
On the off chance that we discover more than one virtio-random device,
the second one will deregister the global instance or trip an invariant
when it fails to attach and calls vtrnd_detach().
We don't have that many resources to free, so just inline those. This
is purely a hygiene thing, we haven't seen an instance of this in the
wild (to my knowledge).
Reviewed by: bnovkov, kfv
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D61050
ena: Reinit RSS components during queue count change flow
The io_queues_nb sysctl is writable while the interface is down, and
setting it destroys the RSS components. When the interface is not UP
nothing reinitialises them, so the indirection table is left referencing
queues that no longer exist until the next ifconfig up.
Call ena_rss_configure() after the queue count changes when the
interface is down. When it is up, this happens through ena_up().
This can configure RSS twice if the interface is brought up afterwards,
costing three redundant admin commands. That is preferred over changing
ena_up(), which the reset path also uses.
Reviewed by: cperciva
Fixes: 56d41ad5fe9c ("Add sysctl node for ENA IO queues number adjustment")
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 2a27dd0076c55789a43415559906f795fd10c2da)
ena: Set ENA_FLAG_RSS_ACTIVE only when RSS is really configured
If the device does not report support for the RSS indirection table,
ena_rss_init_default() destroys the RSS state and returns EOPNOTSUPP.
ena_rss_configure() treats EOPNOTSUPP as success and returns 0, and
ena_up_complete() sets ENA_FLAG_RSS_ACTIVE on any 0 return.
Reading dev.ena.N.rss.key at this point causes kernel panic.
Every current device reports the indirection table, so reaching this
needs a device that is broken or misreporting.
The SYSINIT removed by a6205766b300 called ena_rss_init_default()
directly, saw the EOPNOTSUPP and cleared the flag. Routing the decision
through ena_rss_configure(), which hides EOPNOTSUPP, lost that.
Move the flag set into ena_rss_configure(), which can see whether a
table exists, and clear it when tbl_log_size is 0.
Reviewed by: cperciva
[6 lines not shown]
ena: Update driver version to v2.8.5
New Features:
* Retrieve RSS indirection table size from device
Bug Fixes:
* Limit IO IRQ setup loop to active queues only
* Reinit RSS table on device restore when queue count shrinks
* Fix RSS not working after driver reload
* Fix NULL deref before NULL check in RSS sysctl
* Reinit RSS components during queue count change flow
Minor Changes:
* Fix ENA_MEM_FREE_COHERENT with an expression size argument
Reviewed by: cperciva
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 5e66bf0df3104e212c1b24ff65284b6d1a694c59)
ena: Update driver version to v2.8.6
Bug Fixes:
* Revert "Reinit RSS components during queue count change flow"
* Fix RSS sysctl panic when the RSS table is destroyed
* Set ENA_FLAG_RSS_ACTIVE only when RSS is really configured
Reviewed by: cperciva
MFC after: 3 days
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D61059
(cherry picked from commit 6c177745c58e526bcf7451ad2765df6c63f35f97)
ena: Fix RSS sysctl panic when the RSS table is destroyed
ena_set_io_queue_count() destroys the ena-com RSS state but left
ENA_FLAG_RSS_ACTIVE set. The RSS sysctl handlers gate on that flag, so
they ran against state that was gone.
This causes 2 types of issues:
1. Kernel panic when reading dev.ena.0.rss.key or writing to
dev.ena.0.rss.indir_table
2. Returning a stale cached RSS table when reading
dev.ena.0.rss.indir_table, or a bogus 1 when reading
dev.ena.0.rss.indir_table_size
ena_set_io_queue_count() is called in 2 cases:
1. The io_queues_nb sysctl.
2. Device restore when the device comes back with fewer MSI-X vectors
than before.
Clear ENA_FLAG_RSS_ACTIVE when the RSS table is destroyed. The handlers
[14 lines not shown]
Revert "ena: Reinit RSS components during queue count change flow"
This reverts commit 2a27dd0076c5.
Changing the IO queue count destroys the RSS table, and if this is done
while the device is down a new table is not rebuilt until the next
ena_up(). That leaves 2 problems in the meantime:
1. A stale copy of the destroyed table is still returned via sysctl.
2. Reading the destroyed key, or writing the table, panics.
That commit addressed 1 by creating a new RSS table, so that sysctl
returns a fresh default table with the new number of queues. It did not
set out to fix 2, but the rebuild hid it by accident.
The rebuilt table is the default round-robin one, wired to no queues and
unusable: a write still fails, because programming the device needs the
IO queues that only exist after ena_up(). ena_up() builds that same
table anyway.
[12 lines not shown]
ena: Fix NULL deref before NULL check in RSS sysctl
In ena_sysctl_rss_indir_table(), the code dereferenced
indir->sysctl_buf before checking whether indir itself is NULL.
Move the NULL check above the dereference so the early-return path
fires before any access through indir.
Reviewed by: cperciva
Fixes: 6d1ef2abd330 ("ena: Implement full RSS reconfiguration")
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 462b6c50c7089e96c54cbd34378467f32fc709d4)
ena: Retrieve RSS table size from device
The current driver assumes a default RSS table size of 128 entries,
expecting the device's RSS table size to match this assumption.
This patch allows the driver to create RSS indirection table
with the size advertised by the device.
Reviewed by: cperciva
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 3d1e732a43eda16f20dea3fe2671ce6fca8a03a6)
ena: Fix RSS not working after driver reload
When reloading the driver (kldunload/kldload),
ena_rss_init_default_deferred() is called before ena_attach()
completes. At this point, devclass_get_softc() returns NULL for all
devices because the adapter structs have not been populated yet. As a
result, ena_rss_init_default() is never called and ENA_FLAG_RSS_ACTIVE
is never set. Later, in ena_up_complete(), the code checks the flag,
finds it unset, and skips ena_rss_configure() entirely. RSS is never
initialized, and all traffic is directed to queue 0.
Remove the ENA_FLAG_RSS_ACTIVE gate in ena_up_complete() and always call
ena_rss_configure(), which initialises the table when tbl_log_size is
zero. The deferred SYSINIT is now redundant, so remove it.
Reviewed by: cperciva
Fixes: 9b8d05b8ac78 ("Add support for Amazon Elastic Network Adapter (ENA) NIC")
MFC after: 3 days
Sponsored by: Amazon, Inc.
[2 lines not shown]
ena: Limit IO IRQ setup loop to active queues only
The loop in ena_request_io_irq() iterated over adapter->msix_vecs,
which reflects the maximum number of MSI-X vectors allocated at
init time. When the number of IO queues is reduced at runtime via
the io_queues_nb sysctl, msix_vecs is not updated, causing the
driver to bind interrupts and log CPU affinity for queues that no
longer exist.
Replace the loop bound with ENA_IO_IRQ_IDX(adapter->num_io_queues)
to iterate only over the currently active IO queues.
Also replace the loop in ena_free_io_irq() the same.
Reviewed by: cperciva
Fixes: 56d41ad5fe9c ("Add sysctl node for ENA IO queues number adjustment")
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 32c301922dc1e6f52bdd043046661ac1e2b462d8)
ena: Reinit RSS table on device restore when queue count shrinks
When ena_restore_device() is called after a reset, pci_alloc_msix()
may return fewer MSI-X vectors than requested, causing num_io_queues
to shrink. When this happens, the RSS indirection table retains
entries referencing queues that no longer exist, routing packets to
non-existent queues and causing connectivity loss.
Extract ena_set_io_queue_count() to pair the queue count update with
ena_com_rss_destroy(), ensuring the RSS table is always invalidated
when the queue count changes. Use it from both ena_update_io_rings()
and ena_restore_device().
Reviewed by: cperciva
Fixes: e2735b095b7e ("Fix assumptions about number of IO queues in the ENA")
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 2d4d2ed3fef6acda9269f545f6703d6f32f2f0b7)
ena: Update driver version to v2.8.6
Bug Fixes:
* Revert "Reinit RSS components during queue count change flow"
* Fix RSS sysctl panic when the RSS table is destroyed
* Set ENA_FLAG_RSS_ACTIVE only when RSS is really configured
Reviewed by: cperciva
MFC after: 3 days
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D61059
(cherry picked from commit 6c177745c58e526bcf7451ad2765df6c63f35f97)
ena: Fix RSS sysctl panic when the RSS table is destroyed
ena_set_io_queue_count() destroys the ena-com RSS state but left
ENA_FLAG_RSS_ACTIVE set. The RSS sysctl handlers gate on that flag, so
they ran against state that was gone.
This causes 2 types of issues:
1. Kernel panic when reading dev.ena.0.rss.key or writing to
dev.ena.0.rss.indir_table
2. Returning a stale cached RSS table when reading
dev.ena.0.rss.indir_table, or a bogus 1 when reading
dev.ena.0.rss.indir_table_size
ena_set_io_queue_count() is called in 2 cases:
1. The io_queues_nb sysctl.
2. Device restore when the device comes back with fewer MSI-X vectors
than before.
Clear ENA_FLAG_RSS_ACTIVE when the RSS table is destroyed. The handlers
[14 lines not shown]
ena: Set ENA_FLAG_RSS_ACTIVE only when RSS is really configured
If the device does not report support for the RSS indirection table,
ena_rss_init_default() destroys the RSS state and returns EOPNOTSUPP.
ena_rss_configure() treats EOPNOTSUPP as success and returns 0, and
ena_up_complete() sets ENA_FLAG_RSS_ACTIVE on any 0 return.
Reading dev.ena.N.rss.key at this point causes kernel panic.
Every current device reports the indirection table, so reaching this
needs a device that is broken or misreporting.
The SYSINIT removed by a6205766b300 called ena_rss_init_default()
directly, saw the EOPNOTSUPP and cleared the flag. Routing the decision
through ena_rss_configure(), which hides EOPNOTSUPP, lost that.
Move the flag set into ena_rss_configure(), which can see whether a
table exists, and clear it when tbl_log_size is 0.
Reviewed by: cperciva
[6 lines not shown]
ena: Update driver version to v2.8.5
New Features:
* Retrieve RSS indirection table size from device
Bug Fixes:
* Limit IO IRQ setup loop to active queues only
* Reinit RSS table on device restore when queue count shrinks
* Fix RSS not working after driver reload
* Fix NULL deref before NULL check in RSS sysctl
* Reinit RSS components during queue count change flow
Minor Changes:
* Fix ENA_MEM_FREE_COHERENT with an expression size argument
Reviewed by: cperciva
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 5e66bf0df3104e212c1b24ff65284b6d1a694c59)
Revert "ena: Reinit RSS components during queue count change flow"
This reverts commit 2a27dd0076c5.
Changing the IO queue count destroys the RSS table, and if this is done
while the device is down a new table is not rebuilt until the next
ena_up(). That leaves 2 problems in the meantime:
1. A stale copy of the destroyed table is still returned via sysctl.
2. Reading the destroyed key, or writing the table, panics.
That commit addressed 1 by creating a new RSS table, so that sysctl
returns a fresh default table with the new number of queues. It did not
set out to fix 2, but the rebuild hid it by accident.
The rebuilt table is the default round-robin one, wired to no queues and
unusable: a write still fails, because programming the device needs the
IO queues that only exist after ena_up(). ena_up() builds that same
table anyway.
[12 lines not shown]
ena: Reinit RSS components during queue count change flow
The io_queues_nb sysctl is writable while the interface is down, and
setting it destroys the RSS components. When the interface is not UP
nothing reinitialises them, so the indirection table is left referencing
queues that no longer exist until the next ifconfig up.
Call ena_rss_configure() after the queue count changes when the
interface is down. When it is up, this happens through ena_up().
This can configure RSS twice if the interface is brought up afterwards,
costing three redundant admin commands. That is preferred over changing
ena_up(), which the reset path also uses.
Reviewed by: cperciva
Fixes: 56d41ad5fe9c ("Add sysctl node for ENA IO queues number adjustment")
MFC after: 3 days
Sponsored by: Amazon, Inc.
(cherry picked from commit 2a27dd0076c55789a43415559906f795fd10c2da)
ena: Fix RSS not working after driver reload
When reloading the driver (kldunload/kldload),
ena_rss_init_default_deferred() is called before ena_attach()
completes. At this point, devclass_get_softc() returns NULL for all
devices because the adapter structs have not been populated yet. As a
result, ena_rss_init_default() is never called and ENA_FLAG_RSS_ACTIVE
is never set. Later, in ena_up_complete(), the code checks the flag,
finds it unset, and skips ena_rss_configure() entirely. RSS is never
initialized, and all traffic is directed to queue 0.
Remove the ENA_FLAG_RSS_ACTIVE gate in ena_up_complete() and always call
ena_rss_configure(), which initialises the table when tbl_log_size is
zero. The deferred SYSINIT is now redundant, so remove it.
Reviewed by: cperciva
Fixes: 9b8d05b8ac78 ("Add support for Amazon Elastic Network Adapter (ENA) NIC")
MFC after: 3 days
Sponsored by: Amazon, Inc.
[2 lines not shown]
