ena: Put taskqueues into correct domain if !RSS
When compiled without 'options RSS', the ena driver created taskqueues
using taskqueue_start_threads_cpuset passing a mask value of NULL,
both in the ena_setup_tx_resources path (for enqueues) and in the
ena_create_io_queues path (for the completion-processing).
In the default configuration, on most EC2 instances, this results in
taskqueues running in the right NUMA domain, but only by accident; in
non-default configurations (e.g. with with multiple EBS volumes
attached and associated NVMe taskqueues) the taskqueues may land in
the wrong NUMA domain even on instance types where the one-EBS-one-ENA
case produces the desired results.
Set (struct ena_que)->domain and use that to inform the choice of CPU
sets. On a c8gn.48xlarge EC2 instance this doubles throughput on a
32-TCP-stream benchmark.
Reviewed by: akiyano
[5 lines not shown]
[BOLT][DebugInfo] Add DWARF64 support for BOLT (#206437)
BOLT currently assumes the DWARF32 format when it rewrites debug
sections. As a result, rewriting a binary that was compiled with
`-gdwarf64` either produces a corrupted output or fails `llvm-dwarfdump
--verify`. This patch adds DWARF64 support to the BOLT DWARF rewriter,
honoring each unit's DwarfFormat and emitting correct encodings DWARF64
for binaries, including mix DWARF32 and DWARF64.
[UniformityAnalysis] Make BlockLabels transient and switch to SmallVector. NFC (#211734)
Drop the field from the descriptor and keep the labels in a
propagator-local SmallVector indexed by block number.
ena: Put taskqueues into correct domain if !RSS
When compiled without 'options RSS', the ena driver created taskqueues
using taskqueue_start_threads_cpuset passing a mask value of NULL,
both in the ena_setup_tx_resources path (for enqueues) and in the
ena_create_io_queues path (for the completion-processing).
In the default configuration, on most EC2 instances, this results in
taskqueues running in the right NUMA domain, but only by accident; in
non-default configurations (e.g. with with multiple EBS volumes
attached and associated NVMe taskqueues) the taskqueues may land in
the wrong NUMA domain even on instance types where the one-EBS-one-ENA
case produces the desired results.
Set (struct ena_que)->domain and use that to inform the choice of CPU
sets. On a c8gn.48xlarge EC2 instance this doubles throughput on a
32-TCP-stream benchmark.
Reviewed by: akiyano
[5 lines not shown]
ena: Report RX overrun errors
Extract rx_overruns from the keep alive descriptor reported by
the device and expose it via sysctl hw stats.
RX overrun errors occur when a packet arrives but there are not
enough free buffers in the RX ring to receive it.
MFC after: 2 weeks
Sponsored by: Amazon, Inc.
Differential Revision: https://reviews.freebsd.org/D56640
(cherry picked from commit e3f4a63af63bea70bc86b6c790b14aa5ee99fcd0)
ena: Budget rx descriptors, not packets
We had ENA_RX_BUDGET = 256 in order to allow up to 256 received
packets to be processed before we do other cleanups (handling tx
packets and, critically, refilling the rx buffer ring). Since the
ring holds 1024 buffers by default, this was fine for normal packets:
We refill the ring when it falls below 7/8 full, and even with a large
burst of incoming packets allowing it to fall by another 1/4 before we
consider refilling the ring still leaves it at 7/8 - 1/4 = 5/8 full.
With jumbos, the story is different: A 9k jumbo (as is used by default
within the EC2 network) consumes 3 descriptors, so a single rx cleanup
pass can consume 3/4 of the default-sized rx ring; if the rx buffer
ring wasn't completely full before a packet burst arrives, this puts
us perilously close to running out of rx buffers.
This precise failure mode has been observed on some EC2 instance types
within a Cluster Placement Group, resulting in the nominal 10 Gbps
single-flow throughput between instances dropping to ~100 Mbps as a
[21 lines not shown]
ena: Adjust ena_[rt]x_cleanup to return bool
The ena_[rt]x_cleanup functions are limited internally to a maximum
number of packets; this ensures that TX doesn't starve RX (or vice
versa) and also attempts to ensure that we get a chance to refill
the RX buffer ring before the device runs out of buffers and starts
dropping packets.
Historically these functions have returned the number of packets which
they processed which ena_cleanup compares to their respective budgets
to decide whether to reinvoke them. This is unnecessary complication;
since the precise number of packets processed is never used, adjust
the APIs of those functions to return a bool indicating if they want
to be reinvoked (aka if they hit their limits).
Since ena_tx_cleanup now only uses work_done if diagnostics are
enabled (ena_log_io macros to nothing otherwise) eliminate that
variable and pass its value (ENA_TX_BUDGET - budget) to ena_log_io
directly.
[9 lines not shown]
ena: Update driver version to v2.8.2
Bug Fixes:
* Verify that an ENA ring is in netmap only in native mode
Minor Changes:
* Move parenthesis to correct place in switch
* Add comment
* Reorder define
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55698
Sponsored by: Amazon, Inc.
(cherry picked from commit 96c5eaf0ac6b98d0832e1037d672064de43a7e00)
ena: Minor changes
1. Move parenthesis to correct place in switch and fix include order
2. Add comment at the end of an ifdef for clarity
3. Change include order.
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55696
Sponsored by: Amazon, Inc.
(cherry picked from commit 2667a8454cff5896c7b467c78cd4ace5ad40f5eb)
ena: Verify that an ENA ring is in netmap only in native mode
netmap operates in two modes:
1) Emulated - netmap handling is done by the network stack, the
NIC driver operates transparently to netmap.
2) Native - netmap management is done by the NIC driver.
When checking whether a specific ENA ring is running in netmap
mode, only the following checks were done:
1. IFCAP_NETMAP - Check whether netmap capability is enabled on
the device.
2. NKR_NETMAP_ON - Check whether netmap is actively using this
ring.
The above checks implied that the netmap mode is native and the
ENA driver needs to handle the netmap logic.
The code was missing an explicit check on whether native mode
is actually on (NAF_NATIVE).
This led to a case where though emulated mode was used and
[19 lines not shown]
[SPIR-V] Fix SPIRVTools download step failing on rerun (#211561)
Customized DOWNLOAD_COMMAND ran a raw `git clone`, which fails with
"destination path already exists" whenever ninja reruns the download
step against an already cloned checkout. Use the native
GIT_REPOSITORY/GIT_TAG (with default DOWNLOAD_COMMAND) step instead,
which handles this case correctly
[TableGen] Add register-info statistics (#211528)
Add statistics for the numbers of explicit and synthesized register
classes, along with the number of register pressure sets. This makes
growth in the generated register model easier to spot since it can be
surprising.
Assisted-by: codex
[libunwind][test][AIX] Add C API test for unwinding from AIX VAPI (as signal handler) (#209662)
Further to https://github.com/llvm/llvm-project/pull/209306, test a case
where a signal handler is a Virtual API function triggered synchronously
while the VAPI is not active. Resuming an ancestor context of the signal
frame should call the VAPI return glue.
---------
Assisted-by: IBM Bob
(cherry picked from commit dbaabbc920fcd60aeed5109c3c6353830e7605c5)
[AggressiveInstCombine] Avoid converting first table element when folding table-based cttz (#211202)
Do not attempt to zext (or truncate) the table's first element into the
cttz input type, as the element type may be wider than the input type.
Instead, convert the cttz result and emit the select directly in the
table's access type, which is the final expected type as well. Minor
opportunity to favor using APInt when the first element has more than 64
active bits to avoid crashing.
Fixes: https://github.com/llvm/llvm-project/issues/210890.
[libunwind][test][AIX] Add C API test for unwinding from AIX VAPI (non-signal-handler case) (#209306)
Test detection, during stepping, of the backchain mutation introduced by
a VAPI call (see https://github.com/llvm/llvm-project/pull/209280).
Further, test resumption of contexts using cursors obtained while a VAPI
is active on the thread.
Testing is done via FileCheck inspection of trace output enabled by
`LIBUNWIND_PRINT_UNWINDING=1`. When Live Library Update is not enabled,
synthetic trace output is generated by the test program itself.
---------
Assisted-by: IBM Bob
(cherry picked from commit 861efe03d96d93cd7e475d15bbc676f6d521f4fd)
[DWARFLinker] Treat DW_AT_LLVM_alloc_type as a type reference (#211655)
The parallel linker's getODRAttributes() did not list
`DW_AT_LLVM_alloc_type`, so the dependency tracker treated it as a live
reference rather than a type reference. A type reachable only through
DW_AT_LLVM_alloc_type was therefore never kept: the enclosing subprogram
is marked live, and its type-table-candidate children are skipped during
live marking on the assumption they are retained through the type path.
The attribute then resolved to offset 0, landing on the compile unit
header, which got correctly caught by the DWARF verifier.
The classic linker already lists `DW_AT_LLVM_alloc_type` in
isODRAttribute(), so this PR restores parity between the two.
Assisted-by: Claude
rdar://182715403
(cherry picked from commit bb17e718bc419b46bb2596c2f25c9cc0f76a2d5a)
[LoopInfo] Fix makeLoopInvariant to strip UB-implying attributes. (#211413)
isSafeToSpeculativelyExecute gained an argument to check for UB-implying
attributes in 830cf36bd4c49, but the default is to ignore them. Update
this code to account for that.
Fixes #210137
(cherry picked from commit fcff12829ecee44d715d9f901c0195fb1f30a881)
[AArch64] Reuse NeoverseN1 and N2 scheduling models for A76/A77/A78. (#211209)
We do not have native scheduling models for some of the Cortex-A
generations of CPUs, so they were still using the old Cortex-A57 model.
Whilst not perfectly accurate, the NeoverseN1 and NoeverseN2 scheduling
models should be a better fit for these CPUs. I mostly just tried to
make sure the number of pipelines matches between the neoverse core and
the cortex core. If we need something more accurate then we can start
adding new models for the CPUs.
(cherry picked from commit 54a0f730d9f8df4e7f5e4bfd4c03acbc182eb3cf)
[lldb][Windows] ignore loader breakpoints in system modules (#208233)
Currently, when debugging a program with `lldb-dap` on Windows and using
the `integratedTerminal` option, lldb-dap immediatly stops with an
`0x80000003` Exception. This is because `ntdll` executes an `int3`
breakpoint during process initialization when a debugger is attached.
This patch makes `lldb` and `lldb-server` skip the first `int3` after
launch when it originates from a system module (the loader's debugger
notification). Only that first loader breakpoint is skipped. Any later
int3, including `__debugbreak()`, `__builtin_debugtrap()` in the
debuggee's own code, still stops the debugger.
Fixes https://github.com/llvm/llvm-project/issues/198763
(cherry picked from commit 83530ce356dd55fdc981b5325ca1f5117b4a58f8)
[X86][CCMP] Fix invalid CCMP emission (#211161)
This patch ports AArch64's negation-aware conjunction algorithm to fix
invalid CCMP emission when OR nested inside an AND.
Example: https://godbolt.org/z/ave7f61hK
Before the change, the above case returns 5 rather the 9 when CCMP
enabled.
Assisted-by: Claude Opus 4.8
(cherry picked from commit 45b30651cf94ddb32676550cca7ddd3f6a5e66e0)
