Displaying 1 50 of 38,358 commits (0.009s)

DragonFlyBSD — sbin/mount_hammer2 mount_hammer2.c

hammer2 - Fix root remount for hammer2 root

* mount_hammer2 should not fail in early boot trying to remount /
  if it is unable to connect to the cluster controller, as it might not
  be able to start it due to lo0 not being configured yet.

DragonFlyBSD — sys/vfs/hammer2 hammer2_vfsops.c

hammer2 - Fix ro -> rw mount upgrade

* Fix panic when upgrading a ro mount to rw.
Delta File
+18 -6 sys/vfs/hammer2/hammer2_vfsops.c
+18 -6 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_strategy.c

hammer2 - Fix bug in comp:autozero mode

* comp:autozero mode did not properly set the check code in the live
  de-dup case, resulting in I/O errors later on.

* Fixes issues installing /boot (newfs_hammer2 on an 'a' partition uses
  comp:autozero by default in order to disable compression, so the boot
  loader can read the files without needing decompression code).
Delta File
+11 -3 sys/vfs/hammer2/hammer2_strategy.c
+11 -3 1 file

DragonFlyBSD — sys/vfs/hammer2 hammer2_flush.c hammer2_bulkfree.c, test/debug h2chains.c

hammer2 - Fix bulkfree bugs

* Fix a bug where the HAMMER2_CHAIN_ONFLUSH flag could become desynchronized,
  preventing some modifications from chaining the flag to the root.

  This caused the bulkfree's flush to miss some chains and thus free
  blocks that should not have been freed.

  With this fix concurrent bulkfree and modifying operations appear to
  work properly.

* For now do two quick flush passes in the bulkfree code.  We probably
  need only one.

  Note on the bulkfree code: This code operates by flushing currently
  modified chains and then taking a snapshot of the vchain blockmap.
  The bulkfree code can then operate on the snapshot concurrent with
  active filesystem operations, without interfering with or being
  interfered with by those operations.

  The bulkfree code does not have to flush modified in-memory inodes or
  buffer cache buffers, it need only flush modified hammer2_chain structures
  which may have done allocations, in order to properly account for those
  allocations.  Bulkfree runs in two stages precisely in order to allow
  concurrency.

    [5 lines not shown]

DragonFlyBSD — sbin/hammer2 cmd_bulkfree.c main.c, sys/vfs/hammer2 hammer2_admin.c hammer2_bulkfree.c

hammer2 - Add kernel-thread-based async bulk free

* Add an async bulk-free feature and a kernel thread to allow the hammer2
  vfs itself to run bulk frees.

* Revamp the hammer2 thread management code a bit to support the new use,
  and clean-up the API.

DragonFlyBSD — lib Makefile, lib/libstand hammer2.c

hammer2 - buildworld components, boot2 and loader support

* Compile hammer2 components unconditionally in buildworld now.
  Leave conditionals commented out for the moment, they will be
  removed entirely later on.

* Finish revamping the boot code to properly handle H2 DIRENTs.
  The boot code can now boot from a HAMMER2 'a' partition if desired.
  UFS is no longer required.

* Change newfs_hammer2, mount_hammer2, and the hammer2 vfs to supply
  a default super-root label if none specified.  Supply "@BOOT" when
  mounting from the 'a' partition, "@ROOT" when mounting from the 'd'
  partition, and "@DATA" when mounting from any other partition.

  mount_hammer2 no longer defaults to using @LOCAL for the label when
  none is specified.  Instead we want hammer2 mounts that fail to supply
  a label to behave in a manner that is more likely to be expected by
  the superuser.

* Do not yet include hammer2 in X86_64_GENERIC.  At the moment it must
  be supplied in /boot/loader.conf if the user wishes to boot from it.

DragonFlyBSD — share/misc pci_vendors

Update the pciconf(8) database.

August 10, 2017 snapshot from http://pciids.sourceforge.net/
Delta File
+126 -10 share/misc/pci_vendors
+126 -10 1 file

DragonFlyBSD — sys/sys uio.h

<sys/uio.h>: Clean up the namespace.
Delta File
+12 -8 sys/sys/uio.h
+12 -8 1 file

DragonFlyBSD — sys/vfs/msdosfs msdosfs_vfsops.c

kernel/msdosfs: Add additional sanity checks before mounting.

This is FreeBSD revisions 113979 and 206098.

Taken-from:    FreeBSD
Reported-by:   <open.source at ribose.com>
Dragonfly-bug: <https://bugs.dragonflybsd.org/issues/3050>
Delta File
+7 -0 sys/vfs/msdosfs/msdosfs_vfsops.c
+7 -0 1 file

DragonFlyBSD — sys/vfs/msdosfs msdosfs_vfsops.c

kernel/msdosfs: Add additional sanity checks before mounting.

This is FreeBSD revisions 113979 and 206098.

Taken-from:    FreeBSD
Reported-by:   <open.source at ribose.com>
Dragonfly-bug: <https://bugs.dragonflybsd.org/issues/3050>
Delta File
+7 -0 sys/vfs/msdosfs/msdosfs_vfsops.c
+7 -0 1 file

DragonFlyBSD — sys/kern subr_csprng.c kern_synch.c

kernel - More high-process-count fixes

* The recent tsleep/wakeup work allows multiple cpumasks to associate with
  the same pcpu sleep queue.  This created a degenerate condition where
  the cpu bit would not get cleared in all masks, causing spurious IPIs
  to be sent to the cpu.

  Fixed by ensuring that the bit is cleared after receiving an IPI, if the
  queue is found to be empty.  That is, we allow spurious IPIs to happen,
  but ensure that the condition is cleared so the same driving event does
  not cause additional ones.

* Remove debugging kprintf()s from usched_dfly that were not intended to
  be committed.

* Cleanup the csprng code style a bit and add some missing parenthesis.

DragonFlyBSD — sys/kern usched_dfly.c

kernel - Fix uload calculation in usched_dfly

* When a very large number of processes are runnable (greater than 32767),
  the uload rollup field in dfly_pcpu[] can overflow.  In addition, the
  delta calculation for individual processes can also overflow.

  Fix both cases.

* Symptoms: one cpu starts getting an excessive percentage of the process
  load as the system attempts to continuously rebalance into it.
Delta File
+31 -15 sys/kern/usched_dfly.c
+31 -15 1 file

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_subpart_hammer.c flow.c, usr.sbin/installer/libinstaller diskutil.c diskutil.h

install - Fix a few issues

* sleep 1 second before [re]surveying drives to give devfs time to
  [re]probe the drive.

* fn_subpart_hammer was apparently trying to check the capacity of the
  current slice when there is no current slice.  Check the disk capacity
  instead.

DragonFlyBSD — test/sysperf Makefile

test - Add various 900,000 process tests (2)

* Connect tests to Makefile
Delta File
+11 -1 test/sysperf/Makefile
+11 -1 1 file

DragonFlyBSD — test/sysperf pipe900k.c loop900k.c

test - Add various 900,000 process tests

Add tests involving a large number of user processes.  These tests default
to creating 900,000 user processes and require a machine with at least
128GB of ram.  The machine must be booted with kern.maxproc=4000000 in
/boot/loader.conf.

These test kernel resource management in the face of a large number of
processes.  scheduler, pipe kvm, concurrent fork and exec, general kernel
handling of large numbers of processes, callout, tsleep, and wakeup.
Also concurrent teardown for 900,000 processes when you hit ^C.

* loop900k -        Fork/exec's 900,000 processes which sleep 80 seconds and
                then go into an infinite loop.

                Tests the scheduler and kernel resource management.

* pipe900k -        Fork/exec's 900,000 processes which chain 900,000 pipe()s
                between them and runs an end-to-end test.

                Tests pipe chaining, kernel pipe resource creation and
                teardown, and kernel resource management.

* sleep900k -        Fork/exec's 900,000 processes which sleep 60 seconds and
                then go into an infinite loop with a 1/10 second sleep.

    [13 lines not shown]

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_subpart_hammer.c, usr.sbin/installer/libinstaller diskutil.c diskutil.h

installer: Change fn_mapper_name() to subpartition_get_mapper_name().

Similar to subpartition_get_device_name(). It also helps cleaning up
long lines.

DragonFlyBSD — sys/kern subr_param.c

kernel - Increase flock() / posix-lock limit

* Change the scaling for kern.maxposixlocksperuid to be based on maxproc
  instead of maxusers.

* Increase the default cap to approximately 4 * maxproc.  This can lead
  to what may seem to be large numbers, but certain applications might use
  posix locks heavily and overhead is low.  We generally want the cap to
  be large enough that it never has to be overridden.
Delta File
+1 -1 sys/kern/subr_param.c
+1 -1 1 file

DragonFlyBSD — share/man/man7 tuning.7 build.7

doc - Update build(7) and tuning(7)

* Further modernization, adjust tuning to reflect the new reality.
Delta File
+63 -30 share/man/man7/tuning.7
+44 -0 share/man/man7/build.7
+107 -30 2 files

DragonFlyBSD — sys/kern uipc_mbuf.c

kernel - Increase default mbufs

* Increase default mbuf clusters by 4x

* Increase default mbuf jclusters by 2x

* Change the minimum (for now) to a smaller portion of the total mbuf
  capacity.
Delta File
+18 -12 sys/kern/uipc_mbuf.c
+18 -12 1 file

DragonFlyBSD — sys/net/ipfw ip_fw2.c

ipfw: Use SEQ_GEQ
Delta File
+3 -5 sys/net/ipfw/ip_fw2.c
+3 -5 1 file

DragonFlyBSD — sys/kern sys_pipe.c, sys/vm vm_zone.c vm_map.c

kernel - Remove bottlenecks related to mass pipe closures

* These changes fix teardown latencies involved with destroying a large
  number of pipe()s at the same time, for example when a large number
  of processes are killed simultaneously.  Testing with 900,000 processes
  chaining 900,000 pipe()s, the system could become unusable for upwards
  of an hour when the test is killed with ^C prior to these changes.

  After these changes the system balks for less than 30 seconds in the
  900,000 process case.

* Wrap pipe teardowns in a pcpu mutex.  Such teardowns only occur when the
  normal pcpu pipe cache / hysteresis gets blown out due to a large number
  of closures occurring simultaneously and are not typically in the
  critical path.

  The pcpu mutex allows good concurrency while also limiting the number of
  processes which can be tearing down pipe KVM at the same time.  If
  we have say 900,000 processes all exiting, this mechanism limits the
  teardown to (ncpus) concurrent processes.

  The mutex uses a queued lock (whereas the lockmgr spams wakeup()s),
  so we use a mutex to avoid a degenerate O(N^2) wakeup situation.
  This removes an enormous amount of overhead during the teardown.


    [14 lines not shown]
Delta File
+30 -2 sys/kern/sys_pipe.c
+15 -6 sys/vm/vm_zone.c
+7 -0 sys/vm/vm_map.c
+1 -1 sys/vm/vm_zone.h
+53 -9 4 files

DragonFlyBSD — sys/platform/pc64/acpica OsdEnvironment.c

kernel/acpica: Name a variable more appropriately (from FreebSD).

DragonFlyBSD — usr.bin/truss Makefile

i386 removal, part 62/x: Fix per-architecture config file naming in truss.
Delta File
+1 -1 usr.bin/truss/Makefile
+1 -1 1 file

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c

installer: Don't add an entry for 'root' to /etc/crypttab.

It is not necessary and will in fact generate a 'Device root already
exists.' warning because root has already been configured and mounted
at this point.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c

installer: Don't add an entry for 'root' to /etc/crypttab.

It is not necessary and will in fact generate a 'Device root already
exists.' warning because root has already been configured and mounted
at this point.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_configure.c

installer: Fix mapper name generation for encrypted installation.

fn_mapper_name() is supposed to take the mountpoint name, not the
device name.

This actually broke the install in environments where no /dev/serno/...
device was available and both root and /build were selected for
encrypting because the device name would never contain a '/' and so
the mapper name 'root' was returned for both.

DragonFlyBSD — usr.sbin/installer/dfuibe_installer fn_install.c fn_configure.c

installer: Fix mapper name generation for encrypted installation.

fn_mapper_name() is supposed to take the mountpoint name, not the
device name.

This actually broke the install in environments where no /dev/serno/...
device was available and both root and /build were selected for
encrypting because the device name would never contain a '/' and so
the mapper name 'root' was returned for both.

DragonFlyBSD — . Makefile_upgrade.inc, sys/sys sysvipc.h

<sys/sysvipc.h>: Remove another no longer used header file.

It should have been removed in 0951a8e1611f221f03c7cb219aae011dbca8a17d.
Delta File
+0 -39 sys/sys/sysvipc.h
+1 -0 Makefile_upgrade.inc
+1 -39 2 files

DragonFlyBSD — sys/kern kern_synch.c init_main.c, sys/sys proc.h kernel.h

kernel - Scale tsleep() performance vs many (thousands) of processes

* In situations where a huge number of processes or threads are present
  and sleeping (that is, more than a few thousand), the global cpumask hash
  table used by tsleep() would saturate and effectively cause any wakeup()
  call to broadcast to all CPUs.

* Refactor the tsleep initialization code to allow the global cpumask
  hash table and the pcpu hash tables to be dynamically allocated.

* Allocate a MUCH larger global cpumask hash table, and significantly
  smaller pcpu hash tables.  The global cpumask hash table is now
  sized to approximate 2 * maxproc, greatly reducing cpumask collisions
  when large numbers of processes exist in the system.

  The pcpu hash tables can be smaller without effecting performance.  This
  will simply result in more entries in each queue which are trivially
  iterated.

  Nominal maxproc ~32,000 -> in the noise (normal desktop system)
  Nominal maxproc ~250,000 -> 16MB worth of hash tables (on a 128G box)
  Maximal maxproc ~2,000,000 -> 122MB worth of hash tables (on a 128G box)

* Remove the unused sched_quantum sysctl and variable.


    [7 lines not shown]

DragonFlyBSD — sys/kern subr_param.c

kernel - Change maxproc cap calculation

* Increase the calculation for the maxproc cap based on physical ram.
  This allows a machine with 128GB of ram to maxproc past a million,
  though it should be noted that PIDs are only 6-digits, so for now
  a million processes is the actual limit.
Delta File
+4 -3 sys/kern/subr_param.c
+4 -3 1 file

DragonFlyBSD — sys/kern kern_synch.c kern_proc.c, sys/sys globaldata.h proc.h

kernel - Break up scheduler and loadavg callout

* Change the scheduler and loadavg callouts from cpu 0 to all cpus, and
  adjust the allproc_scan() and alllwp_scan() to segment the hash table
  when asked.

  Every cpu is now tasked with handling the nominal scheduler recalc and
  nominal load calculation for a portion of the process list.  The portion
  is unrelated to which cpu(s) the processes are actually scheduled on,
  it is strictly a way to spread the work around, split up by hash range.

* Significantly reduces cpu 0 stalls when a large number of user processes
  or threads are present (that is, in the tens of thousands or more).  In
  the test below, before this change, cpu 0 was straining under 40%+
  interupt load (from the callout).  After this change the load is spread
  across all cpus, approximately 1.5% per cpu.

* Tested with 400,000 running user processes on a 32-thread dual-socket
  xeon (yes, these numbers are real):

  12:27PM  up 8 mins,  3 users, load avg: 395143.28, 270541.13, 132638.33
  12:33PM  up 14 mins, 3 users, load avg: 399496.57, 361405.54, 225669.14

* NOTE: There are still a number of other non-segmented allproc scans in
        the system, particularly related to paging and swapping.

    [3 lines not shown]
Delta File
+51 -19 sys/kern/kern_synch.c
+26 -4 sys/kern/kern_proc.c
+7 -1 sys/sys/globaldata.h
+4 -4 sys/kern/kern_resource.c
+5 -2 sys/vm/vm_glue.c
+4 -2 sys/sys/proc.h
+14 -14 10 files not shown
+111 -46 16 files

DragonFlyBSD — sys/kern kern_synch.c

kernel - Bump wakeup hash size a little

* Bump from 4001 to 8191 entries to reduce chain length to help situations
  where a large numbers of user threads are in a wait state (in the tens or
  hundreds of thousands of threads).
Delta File
+1 -1 sys/kern/kern_synch.c
+1 -1 1 file

DragonFlyBSD — sys/kern kern_synch.c, sys/sys resource.h

kernel - loadavg structure 32->64 bit fields

* The loadavg structure overflows when a large number of processes
  are running.  Yes, I in fact got it to overflow.  Change the load
  fields from 32 to 64 bits.

* Tested to 400,000 runnable processes.
Delta File
+1 -1 sys/kern/kern_synch.c
+1 -1 sys/sys/resource.h
+2 -2 2 files

DragonFlyBSD — sys/kern lwkt_thread.c subr_param.c, sys/vm vm_map.c vm_page.c

kernel - Fix bottlenecks that develop when many processes are running

* When a large number of processes or threads are running (in the tens of
  thousands or more), a number of O(n) or O(ncpus) bottlenecks can develop.
  These bottlenecks do not develop when only a few thousand threads
  are present.

  By fixing these bottlenecks, and assuming kern.maxproc is autoconfigured
  or manually set high enough, DFly can now handle hundreds of thousands
  of active processes running, polling, sleeping, whatever.

  Tested to around 400,000 discrete processes (no shared VM pages) on
  a 32-thread dual-socket Xeon system.  Each process is placed in a
  1/10 second sleep loop using umtx timeouts:

  baseline                 - (before changes), system bottlenecked starting
                          at around the 30,000 process mark, eating all
                          available cpu, high IPI rate from hash
                          collisions, and other unrelated user processes
                          bogged down due to the scheduling overhead.

  200,000 processes        - System settles down to 45% idle, and low IPI
                          rate.

  220,000 processes        - System 30% idle and low IPI rate

    [58 lines not shown]

DragonFlyBSD — sys/net if_ethersubr.c

ethernet: ether_output should only run in the first netisr_ncpus netisrs
Delta File
+1 -0 sys/net/if_ethersubr.c
+1 -0 1 file

DragonFlyBSD — sys/cpu/x86_64/include signal.h, sys/kern kern_exec.c

kernel - Lower VM_MAX_USER_ADDRESS to finalize work-around for Ryzen bug

* Reduce VM_MAX_USER_ADDRESS by 2MB, effectively making the top 2MB of the
  user address space unmappable.  The user stack now starts 2MB down from
  where it did before.  Theoretically we only need to reduce the top of
  the user address space by 4KB, but doing it by 2MB may be more useful for
  future page table optimizations.

* As per AMD, Ryzen has an issue when the instruction pre-fetcher crosses
  from canonical to non-canonical address space.  This can only occur at
  the top of the user stack.

  In DragonFlyBSD, the signal trampoline resides at the top of the user stack
  and an IRETQ into it can cause a Ryzen box to lockup and destabilize due
  to this action.  The bug case was, basically two cpu threads on the same
  core, one in a cpu-bound loop of some sort while the other takes a normal
  UNIX signal (causing the IRETQ into the signal trampoline).  The IRETQ
  microcode freezes until the cpu-bound loop terminates, preventing the
  cpu thread from being able to take any interrupt or IPI whatsoever for
  the duration, and the cpu may destabilize afterwords as well.

* The pre-fetcher is somewhat heuristical, so just moving the trampoline
  down is no guarantee if the top 4KB of the user stack is mapped or mappable.
  It is better to make the boundary unmappable by userland.


    [7 lines not shown]

DragonFlyBSD — usr.bin/kcollect kcollect.8

kcollect - Fix grammar

* gunplot -> gnuplot

Reported-by: Steve Horan <steve at horan.net.au>
Delta File
+2 -2 usr.bin/kcollect/kcollect.8
+2 -2 1 file

DragonFlyBSD — sys/kern uipc_msg.c, sys/net route.c if.c

route: Routes should only be accessed from the first netisr_ncpus netisrs.

Replicate routes to the first netisr_ncpus CPUs, instead of all CPUs.
Delta File
+58 -100 sys/net/route.c
+33 -13 sys/net/if.c
+27 -7 sys/netinet/if_ether.c
+14 -12 sys/netinet/in_rmx.c
+20 -5 sys/net/radix.c
+10 -14 sys/net/route.h
+74 -34 11 files not shown
+236 -185 17 files

DragonFlyBSD — sys/kern uipc_mbuf.c

mbuf: Minor style change.
Delta File
+2 -4 sys/kern/uipc_mbuf.c
+2 -4 1 file

DragonFlyBSD — sys/net if.c netisr2.h, sys/net/altq altq_subr.c altq_rmclass.c

netisr: Simplify assertion related bits
Delta File
+10 -18 sys/net/if.c
+16 -12 sys/net/netisr2.h
+12 -12 sys/netinet6/nd6_nbr.c
+11 -11 sys/netinet/raw_ip.c
+6 -10 sys/netinet6/in6.c
+7 -7 sys/netinet/ip_divert.c
+64 -66 31 files not shown
+126 -136 37 files

DragonFlyBSD — sys/net/stf if_stf.c

stf: Make route per-cpu.  And it should run in the first netisr_ncpus netisrs.
Delta File
+28 -14 sys/net/stf/if_stf.c
+28 -14 1 file

DragonFlyBSD — sys/net/gre if_gre.c if_gre.h

gre: Rework routing facilities.

- Make route per-cpu.
- Make sure that all routing related operation happens in the first
  netisr_ncpus netisrs.
Delta File
+74 -8 sys/net/gre/if_gre.c
+1 -1 sys/net/gre/if_gre.h
+75 -9 2 files

DragonFlyBSD — bin/ps Makefile

ps(1): Remove -I${.CURDIR}/../../sys from the Makefile.

FreeBSD did that in 2001 (r76812). It's not a bootstrap tool, so there
is no reason to mix up include paths.
Delta File
+1 -2 bin/ps/Makefile
+1 -2 1 file

DragonFlyBSD — sys/cpu/x86_64/include signal.h, sys/kern kern_exec.c kern_proc.c

kernel - Move sigtramp even lower

* Attempt to work around a Ryzen cpu bug by moving sigtramp even lower than
  we have already.

DragonFlyBSD — usr.bin/kdump kdump.c mksubr

kdump(8): Add more support for printing flag etc. names.

* New support for: eaccess() modes, faccessat() modes and atflag,
    fchmodat() atflag, fchownat() atflag, fstatat() atflag,
    linkat() atflag, unlinkat() atflag, utimensat() atflag,
    getvfsstat() flags, lchflags() flags, chflagsat() flags and atflag,
    kenv() actions, usched_set() commands, sys_checkpoint() types,
    procctl() commands, mountctl() operations, and varsym_{list,set}()
    levels.

* Fix flags printing in chflags() and fchflags().

* Better separate mount() and getfsstat() flags definitions.

* Adjust grepping for fcntl() commands send/recvmsg() etc. flags to
    include flags with more than one underscore in their name, like
    F_DUPFD_CLOEXEC.

Still missing: extexit()'s 'how' argument.

DragonFlyBSD — usr.bin/kdump mksubr

kdump(1): Oops, remove forgotten include.
Delta File
+0 -1 usr.bin/kdump/mksubr
+0 -1 1 file

DragonFlyBSD — usr.bin/kdump kdump_subr.h mksubr

kdump(1): Remove unused ptraceopname(). ptrace ops are handled specially.

DragonFlyBSD — sys/vfs/procfs procfs_subr.c procfs.h

procfs - Fix blocked lock condition

* Two procfs races can result in a lock being blocked forever.  Rip out
  the old single-variable global procfs lock and per-node lock and replace
  with a normal lockmgr lock.

* The original lock existed from a time when all of procfs was wrapped with
  a global lock.  This is no longer the case.

DragonFlyBSD — sys/vfs/procfs procfs_subr.c procfs.h

procfs - Fix blocked lock condition

* Two procfs races can result in a lock being blocked forever.  Rip out
  the old single-variable global procfs lock and per-node lock and replace
  with a normal lockmgr lock.

* The original lock existed from a time when all of procfs was wrapped with
  a global lock.  This is no longer the case.

DragonFlyBSD — sys/kern vfs_nlookup.c

kernel - Fix serious permissions bug for sticky directories

* An optimization improperly bypassed the sticky-bit test, creating
  a security issue with /tmp and /var/tmp.

* Fix by disabling the optimization for the second-to-last path component.
  Any prior components retain the optimization, so long directory paths
  are still well-optimized.
Delta File
+28 -8 sys/kern/vfs_nlookup.c
+28 -8 1 file