mm: free PG_rcu_free folios only after an RCU grace period
A folio with the new PG_rcu_free flag is not returned to the page
allocator when its refcount reaches zero. __free_pages_prepare() clears
the flag and queues the folio on a per-CPU list, and a work item frees
it with free_unref_folios() once an RCU grace period has elapsed.
The flag is set by a holder of a reference, or copied by a split to the
folios it creates, and stays set until the folio is freed.
__free_pages_prepare() clears it before it queues the folio, and
nothing else clears it. So a load of the flag under rcu_read_lock()
that finds it set came before that folio's free was queued: the grace
period that the free waits for starts after the load, so after the
rcu_read_lock(), and the memory is not returned to the allocator before
rcu_read_unlock() (the grace-period guarantee of
Documentation/RCU/Design/Requirements/Requirements.rst). A lockless
lookup that afterwards, with the load of its slot ordered after the
load of the flag, still finds the folio where it looked it up knows
which folio that memory is. That holds only while no folio is given
[75 lines not shown]
mm: count the runs of the expedite work
For observation: the new pgfree_rcu_kick event counts the runs of
rcu_free_expedite_workfn() that found folios to free. Each of them
calls synchronize_rcu_expedited().
With only hot folios marked it stays at 0 on tmpfs fallocate() and
punch-hole loops, and with 9G free.
Assisted-by: LLM
Signed-off-by: Kiryl Shutsemau (Meta) <kas at kernel.org>
Signed-off-by: Linus Torvalds <torvalds at linux-foundation.org>
mm/filemap: read from the page cache without folio references
filemap_read() takes and drops a reference on every folio it copies
from, and small reads of one file from many CPUs contend on the
refcount.
A folio that reads keep coming back to now has PG_rcu_free and is freed
only after an RCU grace period, and a folio taken out of the page cache
is no longer given to a new owner without being freed (the splice
patch; it also describes the one exception it knows of, a gifted page
that fuse refuses). Before the slow path, look the page up in i_pages
under rcu_read_lock() and, if its folio has the flag, copy from it
straight to the destination, with user page faults disabled.
Without a reference, the folio that xas_load() returned can be freed
and its memory reused before the flag is tested, so the flag seen can
be that of a later folio in the same memory. That is why the slot is
loaded again after the flag. Memory whose folio has the flag is not
reused before a grace period, and no grace period that starts inside
[163 lines not shown]
mm/filemap: set PG_rcu_free on the folios that reads keep coming back to
Reading from the page cache without a folio reference, later in this
series, needs the folio it reads from to be freed only after an RCU
grace period. That is worth its cost only for a folio that is read
again and again: the reference that it saves is the one that many CPUs
contend on. A folio that is read once and then truncated, reclaimed or
hole-punched should be freed at once, as it is today.
So mark the folio when it turns out to be hot. filemap_read_accessed()
calls the new filemap_mark_hot(), which sets PG_rcu_free under the
reference that filemap_read() holds, when either
- folio_mark_accessed() finds it hot: the folio has been marked
accessed as often as its flags can record, or
- the read starts in the folio that the previous read of this open
file ended in, and before where that one ended, so the second read
of the same bytes through one open file marks the folio.
[62 lines not shown]
mm: expedite the grace period when a CPU has many deferred frees
A folio whose free is deferred waits one to two grace periods on a
per-CPU list, where the allocator does not see it. The memory on the
lists is the rate of deferred frees times that wait, which was 20 to
25 ms on an 8-CPU laptop (on the first form of the series, below).
With only hot folios marked, what gets deferred is a set of hot folios
leaving the page cache together: a file that was being read is
truncated or dropped with fadvise(DONTNEED), or memory pressure
reclaims what was hot a moment ago. Kiryl Shutsemau ran 16 processes
that read a shared 1G range and drop it with fadvise(DONTNEED) in a
loop, on a 2G guest with 64 CPUs: with only the marking and no
expedite work, a process next to them was OOM-killed in 5 of 8 runs.
How much waits at a given rate was measured on the first form of the
series, which deferred every folio removed from a page cache. There,
four processes that fallocate() and hole-punch 64 MiB of a memfd in a
loop freed 23.7 GiB/s and kept 565 MiB waiting on average, 1.2 GiB at
[49 lines not shown]
mm/filemap: mark a folio accessed in one place in filemap_read()
filemap_read() calls folio_mark_accessed() in two places. Before the
copy loop, for the first folio of the batch, unless the previous read
of this open file ended in that folio (pos_same_folio()): a folio that
is read in small pieces is marked accessed once. And inside the loop,
for every later folio (i > 0).
The second is the first with a test whose result is known. A later
folio is reached with ki_pos at its start and with last_pos equal to
ki_pos, because last_pos is set after each folio, and a folio that was
not read to its end is the last one: either the copy came up short, or
end_offset is before the next folio. So last_pos - 1 is in the folio
before it, and pos_same_folio() is false.
Do both with one helper, filemap_read_accessed(), called for each folio
in the loop with the position of the read and where the previous read
of this open file ended, or how far this one has got. For the first
folio the call moves below the test of end_offset against the start of
[8 lines not shown]
mm: have folio_mark_accessed() return whether the folio is hot
Each call of folio_mark_accessed() moves a folio one step up what the
LRU records of its accesses: on the classic LRU from inactive to
inactive and referenced, active, and active and referenced; on the
multi-gen LRU through PG_referenced, the refs bits and PG_workingset.
Have it return whether the folio was already at the top of that when
it was called, which is when the call leaves it as it was: active and
referenced and not idle on the classic LRU, PG_referenced, all the
refs bits and PG_workingset set on the multi-gen LRU.
lru_gen_inc_refs() returns that for the multi-gen LRU.
An unevictable folio counts as hot once it is referenced: it stays in
memory in any case. On the multi-gen LRU it is hot from the first
call, because lru_gen_inc_refs() returns before it changes anything of
an unevictable folio. A dropbehind folio is not hot, though the call
leaves it as it was too: it is to be dropped after the read.
No caller uses the value yet. A later patch of this series marks the
[5 lines not shown]
mm: count the pages waiting for a grace period and free them before OOM
Folios waiting on the per-CPU lists have refcount 0, are not PageBuddy
and are in no counter. should_reclaim_retry() counts only free and
reclaimable pages, so it can decide for the OOM killer while the memory
it needs is about to be freed.
Hot folios freed in bulk still go there. In a 2G guest with 64 vCPUs,
16 readers share a 1G range and fadvise(DONTNEED) it in a loop. With
only the marking and no expedite work, a process next to them was
OOM-killed in 5 of 8 runs.
The expedite work keeps that from piling up. But it leaves up to
RCU_FREE_EXPEDITE pages per CPU and ->batch alone, and with rcu_normal
it waits a normal grace period. With the expedite work present the
drain below did not fire on any measured load; it is the backstop.
Count the waiting pages per zone in NR_FREE_RCU_PAGES, shown as
nr_free_rcu in /proc/vmstat. rcu_free_pages() uncounts a zone's folios
[18 lines not shown]
mm: add vm events for RCU-deferred page frees
Three vm events in /proc/vmstat: pgfree_rcu_mark counts pages of folios
that filemap_mark_hot() marked, pgfree_rcu counts pages whose free
__free_pages_prepare() deferred because PG_rcu_free was set, and
pgfree_rcu_done counts pages that rcu_free_pages() hands to
free_unref_folios() after the grace period. pgfree_rcu -
pgfree_rcu_done is the number of pages waiting; it is approximate,
because the per-CPU event counters are not summed atomically. An
order-0 page that was hwpoisoned while it waited is counted as done
but kept out of the allocator, as any hwpoisoned page is.
These are for observation: how many folios get marked, and the rate at
which frees are deferred. The waiting pages are still not in
NR_FREE_PAGES, and kernel code cannot read vm events cheaply, so this is
not the counter that the allocator needs. A later patch adds that one,
per zone, as nr_free_rcu.
Assisted-by: LLM
Signed-off-by: Linus Torvalds <torvalds at linux-foundation.org>
splice: don't steal page cache folios
page_cache_pipe_buf_try_steal() takes a folio out of the page cache with
remove_mapping() and gives it to the caller to reuse.
fuse_try_move_folio() puts it into a fuse file's page cache, where
writes to that file change it.
PG_rcu_free keeps the memory of a folio from being reallocated before a
lockless lookup's rcu_read_unlock(), but a stolen folio is not freed at
all. The lockless read later in this series holds no reference, so:
reader splice into a fuse file
finds the folio in file A's
i_pages, sees PG_rcu_free, loads
the slot again: still the folio
remove_mapping() takes the folio
out of file A
replace_page_cache_folio() puts it
into the fuse file
[38 lines not shown]
Merge tag 'landlock-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/mic/linux
Pull Landlock fix from Mickaël Salaün:
"This fixes a Clang bug by removing an annotation, which was introduced
mostly for documentation"
* tag 'landlock-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/mic/linux:
landlock: Drop __counted_by from landlock_domain.handled_masks
Merge tag 'spi-fix-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
Pull spi fixes from Mark Brown:
"A small collection of driver specific fixes, plus a new device ID in
the DesignWare device tree binding.
One of the sg2044 fixes which enforces clock limits properly is more
critical than average since getting them wrong might lead to data
corruption"
* tag 'spi-fix-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi:
spi: dt-bindings: snps,dw-apb-ssi: Add Synaptics sl2610 spi
spi: sg2044-nor: Honor SPI clock limits
spi: sg2044-nor: Return transfer errors
spi: cadence-qspi: Fix status polling with octal DTR chips
spi: spi-nxp-fspi: exit stop mode before waiting for DLL lock
Merge tag 'regulator-fix-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator
Pull regulator fixes from Mark Brown:
"There's a couple of core fixes here: there were a couple of issues
with of_regulator_bulk_get_all() not behaving consistently with the
other bulk regulator APIs which caused memory leaks and error handling
issues in users.
We also have a couple of fairly standard driver specific fixes"
* tag 'regulator-fix-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regulator:
regulator: ltc3676: don't read the write-only HRST and CLIRQ registers
regulator: of: state who owns the array from of_regulator_bulk_get_all()
regulator: of: fill in supply names in of_regulator_bulk_get_all()
regulator: tps6594-regulator: Fix n_linear_ranges for TPS65224 BUCK2-4
Merge tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi
Pull SCSI fixes from James Bottomley:
"Driver (and tape media changer) only fixes. Three of the fixes are
about the same size, (ch, ufs decouble CQ and ufs avoid unsafe MMIO)
but the unsafe MMIO is the least obvious logic change"
* tag 'scsi-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi:
scsi: ufs: core: Decouple CQ sweep from request iterator in MCQ
scsi: ufs: core: Avoid unsafe MMIO reads in ufshcd_mcq_compl_all_cqes_lock()
scsi: ch: Do not keep references to data transfer element devices
scsi: ufs: mediatek: Handle mPHY power-on failures
scsi: qedi: Initialize callback state before registration
scsi: ufs: core: Hold a clock reference across the probe
Merge tag 'block-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block fixes from Jens Axboe:
- NVMe fixes via Keith:
- Fix swapped ZRASF zone state values for full and read only
- nvme-multipath revalidate head zones after unfreezing the head
queue
- Revert "nvme: do not reset controllers in NVME_CTRL_NEW state",
which broke things
- nvmet fixes for invalid NSIDs on feature 82h and using the local
P2P device on metadata allocation failure
- zloop: zero out only the unread tail of a short read
* tag 'block-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux:
Revert "nvme: do not reset controllers in NVME_CTRL_NEW state"
zloop: zero out only the unread tail of short reads
nvmet: use the local P2P device on metadata allocation failure
[3 lines not shown]
landlock: Drop __counted_by from landlock_domain.handled_masks
With clang 22 and CONFIG_FORTIFY_SOURCE, the second
landlock_restrict_self() call hits a bogus fortify report in
inherit_ruleset():
memcpy: detected buffer overflow: 4 byte write of buffer size 0
Call Trace:
__fortify_panic
landlock_merge_ruleset
__se_sys_landlock_restrict_self
handled_masks[] lives in an anonymous struct inside the anonymous union
that overlays work_free. For such a flexible array, clang computes the
address of the __counted_by counter as the start of the array instead of
&num_layers. The bound is then read from handled_masks[0], i.e. from
the first layer's access masks, which is not a count. The same layout
crashes clang itself in a standalone test.
[15 lines not shown]
Merge tag 'io_uring-7.3-20261010' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring fixes from Jens Axboe:
- Don't spin on local work the exiting task can't run.
For a ring still R_DISABLED when the task got its tctx node, the
submitter is set later and may be another task, so DEFER_TASKRUN
local work stays pending forever and the exiting task spins in
do_exit().
- Don't charge the SQ/CQ rings or provided buffer rings to
RLIMIT_MEMLOCK. The region API conversion started charging them,
which the rings had been exempt from.
- Don't exempt disabled rings from task restrictions on register
- Populate the CONNECT filter fields for a 24-byte sockaddr_in6
[6 lines not shown]
Merge tag 'xfs-fixes-7.3-rc7' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
Pull xfs fixes from Carlos Maiolino:
"This fixes a data corruption vector where a user could trick the
exchange-range feature to add shared blocks beyond the file's EOF,
which may then be used to modify data on another file"
* tag 'xfs-fixes-7.3-rc7' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
xfs: fix exchange-range-to-eof file size exchange
xfs: stop exchanging reflink flags when exchanging mappings
Merge tag 'x86-urgent-2026-10-11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull x86 fix from Ingo Molnar:
- Work around AMD TLBI Erratum #1718 (Borislav Petkov)
* tag 'x86-urgent-2026-10-11' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
x86/CPU/AMD: Fix AMD TLBI Erratum #1718
Merge tag 'riscv-for-linus-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux
Pull RISC-V fixes from Paul Walmsley:
"Several security fixes, a significant strnlen() regression fix, and
several other miscellaneous fixes:
- Fix a regression in the Zbb-enabled strnlen() implementation when
huge counts are specified
- When userspace pointer masking is enabled, strip the tag from
user-specified pointers in two spots in the RISC-V futex code where
this was missing
- When CONFIG_SHADOW_CALL_STACK=y, use a supervisor mode-only CSR bit
to determine whether the trap came from userspace or not, rather
than a userspace-writable register
- Reset the HSTATUS.HU bit during CPU initialization to avoid
unexpected behavior, as the RISC-V specification does not require
[47 lines not shown]
Merge tag 'input-for-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
Pull input fixes from Dmitry Torokhov:
- A fix for the ati_remote2 sysfs attributes handling, resolving type
confusion after conversion to dev_groups
- Updates to the xpad joystick driver adding support for the Corsair
Novablade Pro controller (wired and 2.4 GHz wireless) and a newer
hardware revision of the 8BitDo Pro 2 Wired Controller for Xbox.
* tag 'input-for-v7.3-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input:
Input: xpad - add support for 8BitDo Pro 2 Wired Controller for Xbox
Input: xpad - add support for Corsair Novablade Pro
Input: ati_remote2 - fix type confusion in device attribute handlers
xfs: fix exchange-range-to-eof file size exchange
Norbert Szetei posted a patch containing an incomplete description of a
bug in XFS_IOC_EXCHANGE_RANGE's TO_EOF flag. The bug tricks
exchange-range into modifying a file so that it has shared blocks
starting beyond EOF, which is never allowed because files never have
written data beyond EOF, and you can only share written data blocks.
This is key to all the incorrect behavior that follows.
Eventually I got from Norbert a description of what's going wrong:
> Fair. Here it is with physical block numbers, 4k blocks. peer is the file
> that gets rewritten and file1 is the one that ends up with the flag
> cleared.
>
> Start, nothing shared:
>
> peer size 4096 -> [100]
> file1 size 12288 -> [200] [201] [202]
[96 lines not shown]
xfs: stop exchanging reflink flags when exchanging mappings
When an exchange covers both files from offset 0 to i_disk_size,
xmi_can_exchange_reflink_flags() assumes every mapping moved and hands
the reflink flag over to the other file.
That assumption is wrong. A file can still own shared blocks past
i_disk_size, e.g. after EXCHANGE_RANGE_TO_EOF sets a smaller size
without unmapping anything. The flag is cleared anyway, and the next
write to such a block happens in place instead of CoW, corrupting the
other file that shares it.
Commit a23eca88448e ("xfs: fix exchange-range reflink flag clearing
issue with INO1_WRITTEN") fixed one instance of this. Rather than add
another special case, stop moving the flags.
xfs_exchmaps_ensure_reflink() already sets the flag on both inodes
before the exchange, so both just keep it.
Nothing sets CLEAR_INO{1,2}_REFLINK anymore. Intents logged by older
[22 lines not shown]
Merge tag 'edac_urgent_for_v7.3_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras
Pull EDAC fix from Borislav Petkov:
- amd64_edac: Shorten the ErrorInformation field read from the MCA_SYND
MSR to only two bits. It is perfectly fine to do so because no system
ever supported more than 2 bits of information (the Chip Selects used
were only 4 maximum) and newer hardware will use only 2 bits anyway
* tag 'edac_urgent_for_v7.3_rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras:
EDAC/amd64: Mask UMC chip select to the four implemented selects
Merge tag 'drm-fixes-2026-10-11' of https://gitlab.freedesktop.org/drm/kernel
Pull drm fixes from Dave Ailie:
"Live from Dublin Airport, it's Saturday Night drm fixes.
This week has the missing misc fixes from last week which I tracked
down and seemed to be a race/bug in my lei setup somehow, once I asked
lei to ignore it's cache I got the missing email. But there are more
misc fixes this week and amd and intel ones.
The main ones in this are amdgpu and xe, with vc4, vmwgfx, nouveau and
imagination in the middle, with a bunch of small single fixes.
Bit busier than I'd like, but the missing misc might explain it,
anyways time for me to fly home.
fb:
- defer setup when fbdev_probe() fails, not just on -EAGAIN
[84 lines not shown]
Merge tag 'i2c-fixes-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux
Pull i2c fix from Andi Shyti:
"Just one qcom-geni fix for a runtime PM reference leak
during transfer setup"
* tag 'i2c-fixes-7.3-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/andi.shyti/linux:
i2c: qcom-geni: release runtime PM reference when set_rate fails
Merge tag 'drm-misc-fixes-2026-10-09' of https://gitlab.freedesktop.org/drm/misc/kernel into drm-fixes
A pointer assignment fix for gud, a reference fix and a preallocation
size fix for imagination, a suspend fix and an allocation fix when idle
for vc4, and a hardware variant fix for nouveau.
Signed-off-by: Dave Airlie <airlied at redhat.com>
From: Maxime Ripard <self at mripard.dev>
Link: https://patch.msgid.link/asjkbknTHg2t8JGy@houat
Merge tag 'drm-xe-fixes-2026-10-08' of https://gitlab.freedesktop.org/drm/xe/kernel into drm-fixes
Fixes on:
- i2c removal (Fan)
- system Controller Maibox header handling (Mallesh)
- two bo pin/unpin accounting bugs (Thomas)
- not emitting a w/a twice (Tvrtko)
- xe_mmio_wait32() to honor delay/sleep maximums (Alan)
Signed-off-by: Dave Airlie <airlied at redhat.com>
From: Rodrigo Vivi <rodrigo.vivi at intel.com>
Link: https://patch.msgid.link/asfEfgbIjs0GJ-RF@intel.com
