kernel - Reduce/refactor nbuf and maxvnodes calculations.
* The prime motivation for this commit is to target about 1/20
(5%) of physical memory for use by the kernel. These changes
significantly reduce kernel memory usage on systems with less
than 4GB of ram (and more specific for systems with less
than 1TB of ram), and also emplace more reasonable caps on
systems with 128GB+ of ram.
These changes return 100-200MB of ram to userland on systems
with 1GB of ram, and return around 6.5GB of ram on systems
with 128G of ram.
* The nbuf calculation and related code documentation was a bit
crufty, still somewhat designed for an earlier era and was
calculating about twice the stated 5% target. For systems with
128GB of ram or less the calculation was simply creating too many
filesystem buffers, allowing as much as 10% of physical memory to
be locked up by the buffer cache.
Particularly on small systems, this 10% plus other kernel overheads
left a lot less memory available for user programs than we would
have liked. This work gets us closer to the 5% target.
* Change the base calculation from 1/10 of physical memory to 1/20
of physical memory, cutting the number of buffers in half on
most systems. The code documentation stated 1/20 but was actually
calculating 1/10.
* On large memory systems > 100GB the number of buffers is now capped
at around 400000 or so (allowing the buffer cache to use around
6.5 GBytes). This cap was previously based on a relatively
disconnected parameter relating to available memory in early boot,
and when triggered it actually miscalculating nbufs to be double
the intended number.
The new cap is based on a fixed maximum of 500MB worth of
struct bufs, roughly similar to the original intention. This
change reduces the number of buffers reserved on system with
more than around 100GB of ram from around 12GB worth of data
down to 6.5GB.
* With the BKVABIO work eliminating most SMP invltlbs on buffer
recyclement, there is no real reason to need a huge buffer
cache. Just make sure its big enough on large-memory machines
to fully cache the likely live datasets for things like bulk
compiles and such.
* For kern.maxvnodes (which can be changed at run-time if you
desire), the base calcualtion on systems with less than 1GB
of ram has been cut in half (~60K vnodes to ~30K vnodes). It
will ramp up more slowly until it roughly matches the prior
calculation at 4GB of system memory. On systems with enough
memory, maxvnodes is now explicitly capped at 4M.
There generally is no need to allow an excessive number of vnodes
to be cached.
For HAMMER1 you can set vfs.hammer.double_buffer=1 to cause it
to cache data from the underlying device, allowing it to utilize
all available free(ish) memory regardless of the maxvnodes setting.
HAMMER2 caches disk blocks in the underlying device by default.
The vnode-based vm_object caches decompressed data, so we want
to have enough vnodes for nominal heavily parallel bulk operations
to avoid unnecessary re-lookups of the vnode as well as avoid having
to decompress the same thing over and over again.
In both cases an excessively high kern.maxvnodes actually wastes
memory on both HAMMER1 and HAMMER2... or at least makes the pageout
daemon's job more difficult.
* Remove vfs.maxmallocbufspace. It is no longer connected to
anything.
UnifiedSplitRaw
