pause fail point: affirm sleeping without a lock
The previous patch to fail_point_pause removed a lock wrapping a sleep.
INVARIANTS assertions require we affirm we are deliberately sleeping
without a lock or timeout via the PNOLOCK flag.
Reported by: netchild
Reviewed by: netchild
Fixes: 331613ddd8a5 ("Pause failpoint: replace mtx_sleep with tsleep")
Differential Revision: https://reviews.freebsd.org/D61730
acpi_pci_link: Handle IRQ values > 255
Some systems use a stub PCI link device for hardwired interrupts
instead of static _PRT entries. These link devices can use an IRQ >
255. The current driver was designed to handle PCI link devices on
x86 used for legacy INTx IRQs which were < 255. These changes handle
larger IRQ values while making a few tradeoffs.
- Expand most IRQ fields from 8 bits to 32-bits. The BIOS IRQ
(read from the PCIR_INTLINE config register which is an 8-bit register)
remains 8 bits.
- Add a wrapper API for managing interrupt weights and avoid tracking
weights for IRQs > 255 and instead report a constant weight of 0 for
IRQs > 255.
Approved by: re (cperciva)
PR: 300476
Reported by: Aviv Bakal <avivb at amazon.com>
[4 lines not shown]
acpi_pci_link: Ignore tunables with negative values
IRQ values are unsigned and negative values do not make sense. The
invalid IRQ value (255) is already ignored, so ignoring other invalid
values is consistent with that behavior.
Approved by: re (cperciva)
Reported by: cperciva
(cherry picked from commit 81dab94a5452f996a063bdba2103eda4aa32b3d1)
(cherry picked from commit 1b71cb09074be51a87a2eb84acb46ee77fceec45)
ctld: Fix namespace IDs for exported NVMe namespaces part 2
This is the same fix as in commit
9c8be6518e2e5f396055ecdab8098ce19c4b2218 just applied to another
method that also accepts namespace IDs. The original commit handled
configuration statements assigning a named LUN to a namespace. This
change handles anonymous namespaces configured within a controller
context.
Approved by: re (cperciva)
Fixes: 66b5296f1b29 ("ctld: Add support for NVMe over Fabrics")
Sponsored by: Chelsio Communications
(cherry picked from commit 9a99fa3932ddd794234becc6eab6519d56ae6947)
(cherry picked from commit 469ac5a9c05f21526f81cb34833bfff53b07a7d5)
ctld: Fix namespace IDs for exported NVMe namespaces
NVMe namespace IDs are 1-based, but CTL LUN IDs are 0-based. The
in-kernel NVMe ctl(4) target accounts for this by adding one to the
LUN ID to compute a namespace ID. ctld was attempting to handle this
by translating namespace IDs from config files to internal LUN IDs,
but it did this incorrectly resulting in the namespace configured for
as namespace N actually showing up as namespace N + 1.
Approved by: re (cperciva)
Fixes: 66b5296f1b29 ("ctld: Add support for NVMe over Fabrics")
Sponsored by: Chelsio Communications
(cherry picked from commit 9c8be6518e2e5f396055ecdab8098ce19c4b2218)
(cherry picked from commit 033c0cd3e89ba87f3d66ceea6436cf2a37148051)
ctld: Fix namespace IDs for exported NVMe namespaces part 2
This is the same fix as in commit
9c8be6518e2e5f396055ecdab8098ce19c4b2218 just applied to another
method that also accepts namespace IDs. The original commit handled
configuration statements assigning a named LUN to a namespace. This
change handles anonymous namespaces configured within a controller
context.
Fixes: 66b5296f1b29 ("ctld: Add support for NVMe over Fabrics")
Sponsored by: Chelsio Communications
(cherry picked from commit 9a99fa3932ddd794234becc6eab6519d56ae6947)
acpi_pci_link: Ignore tunables with negative values
IRQ values are unsigned and negative values do not make sense. The
invalid IRQ value (255) is already ignored, so ignoring other invalid
values is consistent with that behavior.
Reported by: cperciva
(cherry picked from commit 81dab94a5452f996a063bdba2103eda4aa32b3d1)
ctld: Fix namespace IDs for exported NVMe namespaces
NVMe namespace IDs are 1-based, but CTL LUN IDs are 0-based. The
in-kernel NVMe ctl(4) target accounts for this by adding one to the
LUN ID to compute a namespace ID. ctld was attempting to handle this
by translating namespace IDs from config files to internal LUN IDs,
but it did this incorrectly resulting in the namespace configured for
as namespace N actually showing up as namespace N + 1.
Fixes: 66b5296f1b29 ("ctld: Add support for NVMe over Fabrics")
Sponsored by: Chelsio Communications
(cherry picked from commit 9c8be6518e2e5f396055ecdab8098ce19c4b2218)
cxgbe: Don't cache nsegs for KTLS requests
KTLS mbufs are initially parsed when they are first enqueued to
estimate the number of transmit descriptors needed so that the mbuf is
queued until enough descriptors are available. As part of this
estimate, the number of DSGL segments required by each KTLS mbuf is
calculated. Originally, the count for the first TLS record in a chain
was cached in the header mbuf to avoid having to recalculate it when
writing out the actual work request for the first TLS record, but this
requires duplicating fairly complex logic both when parsing and
transmitting requests.
Sponsored by: Chelsio Communications
(cherry picked from commit 6d587ff04b71cb82a1e98fd84b5e0257dd65fec9)
cxgbe: Add a sysctl/tunable to control KTLS offload of AES-CBC cipher suites
Disable these by default as they are less efficient and rarely used.
Sponsored by: Chelsio Communications
(cherry picked from commit 9cf774a577abbe28e03a849d1ec25f47827f07e6)
cxgbe tls: Send final data as immediate when trailing waste is trimmed
When a request needs to drop data from the crypto output (via a split
mode request), send the last 16 bytes of input as immediate data
instead of via DSGL. Requests with small payloads (16 bytes or fewer)
are now sent as immediate data only without any DSGL at all.
Sponsored by: Chelsio Communications
(cherry picked from commit 1bc81d3bb05505d1facc6884ee875ccb56e42977)
acpi_pci_link: Handle IRQ values > 255
Some systems use a stub PCI link device for hardwired interrupts
instead of static _PRT entries. These link devices can use an IRQ >
255. The current driver was designed to handle PCI link devices on
x86 used for legacy INTx IRQs which were < 255. These changes handle
larger IRQ values while making a few tradeoffs.
- Expand most IRQ fields from 8 bits to 32-bits. The BIOS IRQ
(read from the PCIR_INTLINE config register which is an 8-bit register)
remains 8 bits.
- Add a wrapper API for managing interrupt weights and avoid tracking
weights for IRQs > 255 and instead report a constant weight of 0 for
IRQs > 255.
PR: 300476
Reported by: Aviv Bakal <avivb at amazon.com>
Tested by: cperciva
[2 lines not shown]
cxgbe: Greatly simplify ktls_wr_len for T7
Don't try to fine-tune the WR size when estimating the work request
length when parsing the packet. Use a much simpler worst-case
estimate that only depends on a few fields in the mbuf metadata.
Sponsored by: Chelsio Communications
(cherry picked from commit 613e7ce43f814c3fb3e076f8509b969b5c12808f)
cxgbe: Various assertions for lengths in KTLS work requests
The construction of KTLS work requests is quite fragile, and these
assertions ensure that the constructed work requests match the length
fields encoded in some of the WR structures.
Sponsored by: Chelsio Communications
(cherry picked from commit 17126d54c40a2e3f3e2a81e31a07c299416ddf4a)
proc.h: Order thread_lock*() functions more logically
Also matches the implementation order in 'kern_mutex.c'.
No functional change.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 3a1f5a28f00770dec7c793a1c44863a824054ade)
sched_ule: tdq_move(): INVARIANTS: Remove a superfluous assertion
That the lock of the stolen thread is the same as the source runqueue is
already checked by sched_rem().
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 2210a2f8d4705eee0bf3d89a8e30f0babb40fed1)
sys/modules.h: Fix too long line
No functional change (intended).
Fixes: f4b3640475ce ("inline atomics and allow tied modules to inline locks")
Sponsored by: The FreeBSD Foundation
(cherry picked from commit cd33fd75e856f1d91412062da2bd9d98e8897039)
sched_ule: tdq_add(): INVARIANTS: Add check, fix name in assertion strings
Change the assertion on the to-add thread's lock so that it also fails
if the thread's lock is not that of the 'tdq' (the checks that it is
lock-unblocked and locked remain).
It is tempting to add another check that the thread's 'ts_cpu'
corresponds to the passed runqueue (in other words, that it has been set
correctly), but that requires to add a call to thread_lock_block_wait()
which should not be present in non-debug code, and that may prevent some
races from happening on INVARIANTS, so I gave up on it.
Fix the function name in assertion strings, as sched_add() is not the
only (direct or indirect) caller of tdq_add().
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 00add43b882c36ce657bc9d60d5644bf147f378b)
sched_ule: Document fields of 'struct runq_steal_pred_data'
While here, re-order them to have inputs before outputs.
No functional change.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 2ba9debd3ac9d3cdc49936074cef3d97c47e9346)
sched_ule: sched_rem(): Use THREAD_LOCKPTR_ASSERT() instead of MPASS()
This gives a better message.
While here, add some empty lines to make INVARIANTS-only statements
stand out for better readability.
No functional change.
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
(cherry picked from commit 136e3a3eba5b59abfae066d4caaeccd4546d8973)
sched: Fix building kernel independently of world
Regular '-I' directives must be passed to ${CPP} so that when
pre-processed, 'sched_ns.h' can find the files it includes in the proper
places, instead of relying on what is in ${WORLDTMP}, i.e., nothing if
'_includes' was not executed, which happens, e.g., when using 'make
kernel-toolchain' and then 'make buildkernel' without having executed
a 'make buildworld', or possibly some outdated copy not matching an
updated source.
Reported by: Claude Opus 5.5, imp
Fixes: 510331fa3763 ("sched: Automatically namespace schedulers' implementation functions")
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D61167
(cherry picked from commit e89d0e74e23e95fa79989a9053e788e6378eee07)
sched: Automatically namespace schedulers' implementation functions
This renames back scheduler implementation functions to their old names,
i.e., the same name as the interface functions, in the source code,
while providing the infrastructure to still produce from them C symbols
namespaced by the scheduler implementation, in order to avoid symbol
collisions with interface dispatching functions and other scheduler
implementations when multiple ones are compiled in. In a nutshell, the
implementation consists in re-routing interface names to implementation
names via the C preprocessor.
The main reasons for this change are that different implementation names:
1. Break navigating code with ctags-like tools like in-tree ctags,
Global and Universal Ctags.
2. Spoils inlining opportunities in internal scheduler code if not
careful. This alternatively could be regained by changing call
points to use scheduler-specific names, but that does not prevent
people from inadvertently adding more calls to interface functions.
Also, having to namespace calls with the implementation name in the
[77 lines not shown]