pf: Free a table entry that cannot be routed
pfr_insert_kentry() returned the error of pfr_route_kentry() and
leaked the entry it had created; pfr_insert_kentries() stopped at the
first entry it could not route and leaked it and the rest of the work
queue, which no caller frees. Destroy an entry that cannot be routed
and go on with the next.
Reviewed by: kp
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60532
(cherry picked from commit 989d72fd3d35daa6a056afdd0c6b0aba4c5d8ccf)
pf: Do not hash pool address pointers into rule hashes
pf_hash_pool() hashes the first address of a pool as a whole struct
pf_addr_wrap, including the union that holds the kernel's pfi_dynaddr
or pfr_ktable pointer.
An interface pool such as "-> ($ext_if)" gets a new pfi_dynaddr on
every load, so with "set keepcounters" the rule never matches its
predecessor and loses its counters on every reload. The pointers also
differ between hosts: once a filter rule in the main ruleset has an
interface or table pool, pfsync peers disagree on the ruleset checksum
and bind every synced state to the default rule.
Hash the address with that union cleared. The rest of the address,
including the prefix length of an interface address, is hashed as
before, and the hash of a rule with a plain address pool does not
change.
Reviewed by: kp
[6 lines not shown]
pf: Set the reassembly flags under the rules lock
DIOCSETREASS stored V_pf_status.reass twice with no lock, first the
flags it was given and then, if reassembly was not among them, zero.
pf_normalize_ip() and pf_normalize_ip6() read the flags with the rules
lock held for reading, so a packet could see the intermediate value.
Compute the flags first and store them once, with the rules lock held
for writing, as DIOCSETDEBUG and DIOCSETHOSTID do.
Reviewed by: kp
Fixes: 39282ef356db ("pf: backport OpenBSD syntax of "scrub" option for "match" and "pass" rules")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60531
(cherry picked from commit b951994eddd4d274514df751da06680a32139037)
pf: Drain the overload task before a vnet's rules are freed
pf_src_connlimit() queues an entry for pf_overload_task() when a
source exceeds max-src-conn or max-src-conn-rate and the rule has an
overload table. The entry holds a pointer to the rule, and the task
runs from taskqueue_swi with the vnet of the queue. Nothing drained
the task when the vnet went away: an entry queued just before could
be applied to a rule pf_unload_vnet_purge() had freed, or to a vnet
that was gone.
Drain the task at the start of pf_unload_vnet_purge(), before the
unlinked rules are freed. No packet can queue another by then, as the
hooks are already removed.
Reviewed by: kp
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60529
(cherry picked from commit 92d66a6b474fa6fe42fbe08002803d79a5af6594)
rms: Order a writer's stores before its readers see it leave
Readers check rms->writers without a fence, and rms_wunlock() drops
it with a plain store. rms_wlock() interrupts every CPU before the
first writer blocks readers, but nothing orders them when the last
writer leaves: on a weakly ordered CPU a reader can see the count
reach zero and still read what the writer replaced, such as callbacks
of a module unloaded under the lock.
Fence every CPU before the last writer drops the count: rename
rms_wlock_switch() to rms_block_readers() and add
rms_unblock_readers(), a wrapper for cpus_fence_seq_cst(). Readers
stay fence-free; the cost lands on the rare unlock.
Reviewed by: mjg
Discussed with: kib
Fixes: 1f162fef76a0 ("Add read-mostly sleepable locks")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
[3 lines not shown]
if_gre: Use OID_AUTO for the net.link.gre sysctl node
if_gre(4) registers net.link.gre with the static OID number IFT_TUNNEL.
That number is an interface type that several tunnel drivers share, not
one that belongs to gre(4). if_me(4) used it too, which made loading
both modules panic until 21f2245de219. Nothing addresses the node by
number, so use OID_AUTO as if_me(4) now does.
Suggested by: markj
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")
(cherry picked from commit 80bc1d0e3ef732c6b58d13d0f223453a2410d8c1)
iflib: Do not ring the transmit doorbell when nothing is pending
For a lightly used ring iflib_txd_db_check() may defer zero descriptors,
so its "pending >= limit" test is true even when nothing has been queued
since the last doorbell. iflib_txq_drain() calls it before, inside and
after its loop, so a sender that drains its own packet wrote the tail
register three times per packet, twice with the value the hardware
already had.
The log of 81be655266fa ("iflib: ensure that tx interrupts enabled and
cleanups") calls skipping the doorbell when db_pending is zero "an
obvious missing optimization"; the comparison against a limit of zero
defeated it. vmx(4) and mgb(4) have dropped such repeated requests in
the driver since 2019. Return early when nothing is pending.
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60290
[2 lines not shown]
if_me: Do not reuse gre(4)'s static sysctl OID number
if_me(4) registers its net.link.me node with the static OID number
IFT_TUNNEL, the same number if_gre(4) has used for net.link.gre since
2003, so the two tunnel drivers collide under net.link. Until
d35c4cfad580 that only printed a warning and left two nodes with the
same number; since then sysctl_register_oid() panics, so loading if_me
after if_gre (or the other way round) takes the box down:
kldload if_gre
kldload if_me
panic: sysctl: OID number(131) is already in use for 'me'
Use OID_AUTO like the other tunnel drivers. Nothing addresses the node
by number.
Reviewed by: ae
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")
[3 lines not shown]
igc: Fix the flow control sysctl
dev.igc.N.fc read and wrote a function-static variable shared by every
igc device, so a read returned the last value written to any of them (3
until the first write), not the state of the device. The softc value
started as 0, which is also the value of "no flow control", while the
hardware was set up for full flow control. As a result:
- Writing 0 was taken for no change and did nothing, unless another
value had been written to that device before.
- igc_reset() took a softc value of 0 for "not set", so a device set to
0 went back to full flow control on the next init.
- With more than one receive queue the driver enabled per-queue drop
(SRRCTL.DROP_EN), which is meant for a MAC that does not send pause
frames, although the MAC was told to send them.
- Values out of range were accepted and ignored.
A write only forced the MAC's flow control bits. The pause bits
advertised to the link partner, the pause thresholds and DROP_EN stayed
[26 lines not shown]
epoch: Fix use-after-free in epoch_trace_report()
epoch_trace_report() assigned the return value of RB_INSERT() back to
the new element. When two threads report the same stack concurrently,
the loser's RB_INSERT() returns the element already in the tree, and
that element was freed while still linked, leaking the new allocation.
The next lookup touches freed memory; KASAN catches it as a
use-after-free.
Keep the return value separate and free the new element instead. The
thread that won the race prints the report, so return without printing
it a second time.
Reviewed by: markj
Fixes: 173c062a569b ("Improve EPOCH_TRACE")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60162
(cherry picked from commit dd5dc8f6e51c5132f2dc885f5c1b0492cdf3052c)
pf: modify pfik_flags atomically
The purge thread sets PFI_IFLAG_REFS on the interfaces that states
refer to without the rules lock, under which the other flags are
changed. The updates can interleave, so that a "set skip on" is lost,
or outlives its removal, until the next ruleset load.
Use atomic operations to modify the flags. In the purge thread, only
write if the flag is not already set.
Reviewed by: kp
Approved by: kp (mentor)
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60107
(cherry picked from commit c1241c6e30c792fb514f4689d2ed92ee21d3cf39)
pf: do not loop on an address that is cleared twice in pfr_clr_astats()
pfr_clr_astats() looks up each address it is given and inserts the entry
it finds at the head of a work queue. If the same address is given more
than once, the entry is inserted twice and the second insertion makes it
its own successor. pfr_clstats_kentries() then walks the queue forever,
with the rules lock held for writing, so packet processing and every
other pf operation in that vnet stop as well. To reproduce:
pfctl -e
pfctl -t foo -T add 192.0.2.1
pfctl -t foo -T zero 192.0.2.1 192.0.2.1
Do as pfr_del_addrs() does: clear pfrke_mark on the entries named, then
queue an entry only the first time it is seen. An address given more
than once is cleared, and counted, once. Validate all addresses before
any entry is touched.
Add a regression test.
[8 lines not shown]
iflib: Pass up the frames received before a descriptor error
When the driver's isc_rxd_pkt_get() fails part way through a receive
pass, iflib_rxeof() jumps straight to the reset request. The frames it
already took off the ring and chained for if_input() are neither passed
up nor freed, so up to budget - 1 frames are lost and their mbufs and
clusters leak on every such error, and the frames queued to LRO in the
pass wait for the next one. Leave the loop instead, pass the chain up
and flush LRO as on the normal exit, then request the reset as before,
without refilling.
The path is reachable from several drivers: igc, em, igb and ixgbe
return EBADMSG for frames received with errors, which the hardware
passes up only with store-bad-packets enabled; ixl, iavf and ice for
every descriptor flagged RXE; aq, enetc, mgb and enic for descriptors
they cannot use.
Reviewed by: gallatin
Sponsored by: Rubicon Communications, LLC ("Netgate")
[3 lines not shown]
e1000: Fix the flow control sysctl
dev.em.N.fc and dev.igb.N.fc read and wrote a function-static variable
shared by every em(4) and igb(4) device, so a read returned the last
value written to any of them (3 until the first write), not the state of
the device. The softc value started as 0, which is also the value of
"no flow control", while the hardware was set up for full flow control.
As a result:
- Writing 0 was taken for no change and did nothing, unless another
value had been written to that device before.
- em_reset() took a softc value of 0 for "not set", so a device set to 0
went back to full flow control on the next init.
- On igb(4) with more than one receive queue the driver enabled
per-queue drop (SRRCTL.DROP_EN), which is meant for a MAC that does
not send pause frames, although the MAC was told to send them.
- Values out of range were accepted and ignored.
A write only forced the MAC's flow control bits. The pause bits
[20 lines not shown]
pflow: Destroy exporters on module unload
Since 83641335f96c only jail removal destroys exporters. Unloading
pflow with one left trips the assertion in vnet_pflowdetach() or,
without INVARIANTS, leaves pf calling into the freed module the next
time it removes a state that pflow exports.
Reviewed by: kp
Fixes: 83641335f96c ("pf: clean up pflow sockets on jail removal")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
(cherry picked from commit c91539772b07cb78369626716a4ddf92cf6caa5f)
pf: Register the netlink family only while the vnets are set up
pf_modevent() registers the generic netlink family at MOD_LOAD, before
vnet_pf_init() has set up the state hashes and locks the handlers use,
and pf_unload() unregisters it after vnet_pf_uninit() has torn them
down. Sending PFNL_CMD_GETSTATES while pf loads or unloads panics in
pf_handle_getstates() on a NULL or freed V_pf_idhash.
Register the family from a SYSINIT ordered after vnet_pf_init(), and
unregister it from the matching SYSUNINIT, before vnet_pf_uninit().
Reviewed by: kp
Fixes: 2cef62886dc7 ("pf: convert state retrieval to netlink")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60651
(cherry picked from commit 68831d2a2cbb34e8daf24db7e4e15b8c09451f85)
pf: Do not hash pool address pointers into rule hashes
pf_hash_pool() hashes the first address of a pool as a whole struct
pf_addr_wrap, including the union that holds the kernel's pfi_dynaddr
or pfr_ktable pointer.
An interface pool such as "-> ($ext_if)" gets a new pfi_dynaddr on
every load, so with "set keepcounters" the rule never matches its
predecessor and loses its counters on every reload. The pointers also
differ between hosts: once a filter rule in the main ruleset has an
interface or table pool, pfsync peers disagree on the ruleset checksum
and bind every synced state to the default rule.
Hash the address with that union cleared. The rest of the address,
including the prefix length of an interface address, is hashed as
before, and the hash of a rule with a plain address pool does not
change.
Reviewed by: kp
[6 lines not shown]
pf: Do not corrupt neighbour advertisements when translating
When an ICMPv6 query or reply matches a translated state,
pf_test_state_icmp() copies the state's virtual id into icmp6_id. Only
echo messages have an identifier there. For all other types the
virtual id is 0 and those two bytes hold something else; in a neighbour
advertisement they hold the R, S and O flags. The checksum was not
updated either, so icmp6_input() dropped the advertisement as a bad
checksum.
Unicast NUD probes to a link-local neighbour, such as the default
router, are sent from the link-local address. With a nat rule that
matches it (e.g. "from any"), every advertisement answering a probe was
dropped: NUD failed, the neighbour cache entry was deleted, and traffic
to the neighbour stalled until it was resolved again.
Only rewrite the id of echo messages, as the ICMPv6 error path already
does, and update the checksum, as the IPv4 path does.
[10 lines not shown]
pf: Parse the GETSTATES state id as 64 bits
The GETSTATES dump sends PF_ST_ID as a u64, but the request parser
reads it as a u32 and rejects an id taken from the dump with EINVAL.
Nothing in base sends PF_ST_ID. Parse and document it as a u64.
Reviewed by: kp
Fixes: 2cef62886dc7 ("pf: convert state retrieval to netlink")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60543
(cherry picked from commit 74d2a0b79ffd8d43da61eaca36b39935594d8772)
pf: Free a table entry that cannot be routed
pfr_insert_kentry() returned the error of pfr_route_kentry() and
leaked the entry it had created; pfr_insert_kentries() stopped at the
first entry it could not route and leaked it and the rest of the work
queue, which no caller frees. Destroy an entry that cannot be routed
and go on with the next.
Reviewed by: kp
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60532
(cherry picked from commit 989d72fd3d35daa6a056afdd0c6b0aba4c5d8ccf)
pf: Set the reassembly flags under the rules lock
DIOCSETREASS stored V_pf_status.reass twice with no lock, first the
flags it was given and then, if reassembly was not among them, zero.
pf_normalize_ip() and pf_normalize_ip6() read the flags with the rules
lock held for reading, so a packet could see the intermediate value.
Compute the flags first and store them once, with the rules lock held
for writing, as DIOCSETDEBUG and DIOCSETHOSTID do.
Reviewed by: kp
Fixes: 39282ef356db ("pf: backport OpenBSD syntax of "scrub" option for "match" and "pass" rules")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60531
(cherry picked from commit b951994eddd4d274514df751da06680a32139037)
rms: Order a writer's stores before its readers see it leave
Readers check rms->writers without a fence, and rms_wunlock() drops
it with a plain store. rms_wlock() interrupts every CPU before the
first writer blocks readers, but nothing orders them when the last
writer leaves: on a weakly ordered CPU a reader can see the count
reach zero and still read what the writer replaced, such as callbacks
of a module unloaded under the lock.
Fence every CPU before the last writer drops the count: rename
rms_wlock_switch() to rms_block_readers() and add
rms_unblock_readers(), a wrapper for cpus_fence_seq_cst(). Readers
stay fence-free; the cost lands on the rare unlock.
Reviewed by: mjg
Discussed with: kib
Fixes: 1f162fef76a0 ("Add read-mostly sleepable locks")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
[3 lines not shown]
pf: Drain the overload task before a vnet's rules are freed
pf_src_connlimit() queues an entry for pf_overload_task() when a
source exceeds max-src-conn or max-src-conn-rate and the rule has an
overload table. The entry holds a pointer to the rule, and the task
runs from taskqueue_swi with the vnet of the queue. Nothing drained
the task when the vnet went away: an entry queued just before could
be applied to a rule pf_unload_vnet_purge() had freed, or to a vnet
that was gone.
Drain the task at the start of pf_unload_vnet_purge(), before the
unlinked rules are freed. No packet can queue another by then, as the
hooks are already removed.
Reviewed by: kp
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60529
(cherry picked from commit 92d66a6b474fa6fe42fbe08002803d79a5af6594)
if_gre: Use OID_AUTO for the net.link.gre sysctl node
if_gre(4) registers net.link.gre with the static OID number IFT_TUNNEL.
That number is an interface type that several tunnel drivers share, not
one that belongs to gre(4). if_me(4) used it too, which made loading
both modules panic until 21f2245de219. Nothing addresses the node by
number, so use OID_AUTO as if_me(4) now does.
Suggested by: markj
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")
(cherry picked from commit 80bc1d0e3ef732c6b58d13d0f223453a2410d8c1)
if_me: Do not reuse gre(4)'s static sysctl OID number
if_me(4) registers its net.link.me node with the static OID number
IFT_TUNNEL, the same number if_gre(4) has used for net.link.gre since
2003, so the two tunnel drivers collide under net.link. Until
d35c4cfad580 that only printed a warning and left two nodes with the
same number; since then sysctl_register_oid() panics, so loading if_me
after if_gre (or the other way round) takes the box down:
kldload if_gre
kldload if_me
panic: sysctl: OID number(131) is already in use for 'me'
Use OID_AUTO like the other tunnel drivers. Nothing addresses the node
by number.
Reviewed by: ae
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")
[3 lines not shown]
iflib: Do not ring the transmit doorbell when nothing is pending
For a lightly used ring iflib_txd_db_check() may defer zero descriptors,
so its "pending >= limit" test is true even when nothing has been queued
since the last doorbell. iflib_txq_drain() calls it before, inside and
after its loop, so a sender that drains its own packet wrote the tail
register three times per packet, twice with the value the hardware
already had.
The log of 81be655266fa ("iflib: ensure that tx interrupts enabled and
cleanups") calls skipping the doorbell when db_pending is zero "an
obvious missing optimization"; the comparison against a limit of zero
defeated it. vmx(4) and mgb(4) have dropped such repeated requests in
the driver since 2019. Return early when nothing is pending.
Reviewed by: gallatin
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60290
[2 lines not shown]
makefs/zfs: Allow the autotrim pool property to be set
Add an "autotrim" option taking "on" or "off". The value is stored in
the pool properties object alongside "bootfs", so the imported pool
starts trimming freed space automatically without a separate
"zpool set" step. When the option is omitted nothing is written and
the pool keeps the default ("off"), exactly as before.
A string option is used rather than makefs's boolean option type since
the latter ignores its value, which would make "autotrim=off" enable
the feature.
Define the spa_autotrim_t enumeration in the boot zfsimpl.h, copied
from OpenZFS's sys/spa.h, so that makefs can refer to SPA_AUTOTRIM_OFF
and SPA_AUTOTRIM_ON by name. The loader does not use it.
Document the option in makefs(8) and add a regression test.
Reviewed by: markj
[5 lines not shown]
igc: Fix the flow control sysctl
dev.igc.N.fc read and wrote a function-static variable shared by every
igc device, so a read returned the last value written to any of them (3
until the first write), not the state of the device. The softc value
started as 0, which is also the value of "no flow control", while the
hardware was set up for full flow control. As a result:
- Writing 0 was taken for no change and did nothing, unless another
value had been written to that device before.
- igc_reset() took a softc value of 0 for "not set", so a device set to
0 went back to full flow control on the next init.
- With more than one receive queue the driver enabled per-queue drop
(SRRCTL.DROP_EN), which is meant for a MAC that does not send pause
frames, although the MAC was told to send them.
- Values out of range were accepted and ignored.
A write only forced the MAC's flow control bits. The pause bits
advertised to the link partner, the pause thresholds and DROP_EN stayed
[26 lines not shown]
callout: do not retry a try-lock callout sooner than a tick
When softclock_call_cc() fails to acquire the lock of a CALLOUT_TRYLOCK
callout, it reschedules the callout half its precision after
cc_lastscan and halves the precision. Repeated failures shrink the
delay toward zero, and once the precision reaches 1 the callout is due
immediately: the timer fires again at once and softclock retries the
lock in a tight loop for as long as the lock is held.
If the lock owner runs on the callout's CPU and no other CPU is idle,
the softclock thread preempts it on every attempt, starving the thread
it is waiting on. On an 8-CPU arm64 VM, a test module holding the lock
saw 760,000 attempts per second, each with its own timer interrupt, and
progressed at 38% of its normal rate. On a 4-core amd64 system under
loopback TCP load, a netisr thread holding an inpcb lock made no
progress for 12 minutes while the TCP timer callout was retried 830,000
times per second.
Keep the half-precision retry, but never schedule it less than one
[22 lines not shown]
pf: do not leak a source hash row lock in the netlink dump
pf_handle_get_srcnodes() returns with the lock of a source hash row
held when it cannot start the message for a source node. Unlock the
row there, as the other error exit of the loop does.
Reviewed by: kp
Approved by: kp (mentor)
Fixes: 9c125336727b ("pf: convert DIOCGETSRCNODES to netlink")
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision: https://reviews.freebsd.org/D60251
(cherry picked from commit 782c6ea08603a4839c2c099842df92b6159b799f)
