FreshBSD

kcollect - Remove system calls from graph

* System calls make the graph too scratchy.  Remove them for now.

dsynth - Stabilization, improve compatibility

* Disable /etc/make.conf for the bulk scan (__MAKE_CONF=/dev/null)

* Add -D to turn on developer mode (from zrj)

* Add a little infrastructure to cache ports variables, starting
  with _PERL5_FROM_BIN.

dsynth - Do not count DUMMY nodes in the total

* dsynth uses DUMMY nodes to wrap all the flavors for a particular
  package when other packages have a generic (rather than flavor-specific)
  dependency to it.

* Do not count these nodes in the total for display purposes.

drm/ttm: Partially sync the vm fault code with Linux

Prevent crashes with Mesa-using applications.

drm/linux: Move vm_area_struct definition to the right header

drm/radeon: Use mm->mmap_sem

drm - Refactor task_struct and implement mm_struct

* Change td->td_linux_task from an embedded structure to a pointer.

* Add p->p_linux_mm to support tracking mm_struct's.

* Change the 'current' macro to test td->td_linux_task and call
  a support function, linux_task_alloc(), if it is NULL.

* Implement callbacks from the main kernel for thread exit and
  process exit to support functions that drop the td_linux_task and
  p_linux_mm pointers.

  Initialize and clear these callbacks in the module load/unload
  in drm_drv.c

* Implement required support functions in linux_sched.c

drm/radeon: Update to Linux 4.7.10

* Various bugfixes, especially related to Displayport and
  vblank handling

* Sea Islands and Southern Islands GPUs support improvements

* Various quirks to fix problems for specific cards, mostly
  Radeon R7 370 and Radeon R9 270X models

drm/linux: down_read() acquiress a shared lock

dsynth - Account for extraction size

* Account for the initial size of /construction after the extract
  phase.

  Note that we explicitly are not accounting for space used after this
  phase because they files (typically .o's and such) are added incrementally
  and often not accessed continuously until the ending link phase.  Including
  this space thus might cause dsynth to retard the worker count too much.

dsynth - More pkg dependency fixes

* Fix the depency check based on the depth when recursing through a
  dummy node.  Dummy nodes do not increment the depth.

* In addition, recursing through a dummy node means that the package
  only depends on the default flavor, which is only the first element
  in the idepon list under that dummy node.  Inform the next layer
  that its scan should only iterate one element.

* In addition, when build_find_leaves() recurses through a dummy node
  also use the same mechanism where only the first sub-node is
  checked, because the dependency is (again) against just the default
  flavor.

  Note that when doing a 'dsynth everything', all the dependencies
  wind up in the root list, so all the flavors get built.  But when
  building a specific subset of packages, this fix will reduce the
  number of flavors that have to be built as dependencies.

dsynth - Unbreak ncurses coloring

* Fixup the log filter for the ncurses display.

dsynth - Report phase on failure

* Report the phase a failure occurs at.

* When logging each phase, use 79 dashes as a separator.

dsynth - Use foreground color instead of background color

* Set the foreground color instead of the background color,
  just to not be too invasive.

dsynth - pty CRs, readelf, better log formatting

* Remove CR's from build logs (set tty modes on the pty).

* Parse 'readelf -x .note.tag /bin/sh' to get the operating
  system and __DragonFly_version... basically the release name.
  Compare the operating system name against what we get back
  from the system as a validator.

* Improve the log format to make it more readable.  Certain elements such
  as statistics and times are right-justified.

* Filter certain log lines from stdout when curses mode is turned off.

* Highlight stdout lines for SUCCESS in green and FAILURE and red.

dsynth - Fix VersionName (UNAME_v)

* Fix VersionName (UNAME_v) to use the format e.g.
  "DragonFly 5.7-SYNTH".

* Not yet pulled from elf info, currently using sysctl.

dsynth - Reduce number of pre-installed pkgs in worker

* The top-level scan installs all dependencies, but the deeper recursion
  only needs to install LIB and RUN dependencies.

dsynth - Fix templating bugs, flesh out environment and make.conf

* Fix a few templating bugs

* Generate a make.conf in each chroot/jail based on the configured
  environment.

* Implement the make environment in the pkglist commands as make
  variables on the command line.

* Use vfork/exec in the bulk code... generate the argument vector
  and environment ourselves.

* Ignore '/nonexistent:BLAH' dependencies.

drm/radeon: Disable acceleration on evergreen

This is a workaround for https://bugs.dragonflybsd.org/issues/3198

drm: Partially sync drm_ioctl() with Linux 4.7.10

dsynth - Add a pkgi depth calculation

* Calculate the largest tree-depth of packages that depend on PKG
  and use it in the sort as weighting to try to prioritize building
  PKG.

  The intent is to try to knock out these deep chains as quickly as
  possible so as not to get stuck on a linear chain of dependencies
  later on in the build.

dsynth - Stabilization

* Rejigger the dynamic load management calculations and fix bugs in
  how the restrictions based on pkgdep memory use are calculated.

  The pkgdep memory mechanism now supports both slow-start and
  slow-approach.  The slow-approach mechanism begins when pkgdep
  memory use goes about 1/2 the target (typically 1/4 physical memory).

  In slow-approach mode, the number of workers is increased at a
  rate of 1 every 30 seconds instead of 1 every 5 seconds in order
  to give workers time to install the packages and fill in the estimated
  memory use.  The idea is that this then allows paging to ramp up
  more slowly instead of getting 'shocked' by a dozen slots all
  issuing install-pkgs at the same time.

* Add -m gb  (in gigabytes), to override the default pkgdep memory
  target.

* For now stop mounting /usr/src in the workers.

* NCurses disablement now works as expected.

* Calculate 1 gigabyte as 1GiB.


    [2 lines not shown]

dsynth - Add Dynamic memory load management

* In addition to using the load and swap, also use the total
  estimated installed package dependencies for each worker verses
  PhysMem and reduce the number of worker slots such that this
  total is less than 1/2 physical memory.

* Fix a ^C handling bug where dsynth would try to unmount while
  still holding a file descriptor open on the filesystem it is
  trying to unmount.

* Clean up the logs a bit more.

dsynth - Add color

* Color successful builds green and failed builds red.

* Refactor some of the log mesages.

dsynth - Add logfile tracking on the ncurses display

* Track the contents of the 00*.log file below the list of
  workers.

* Handle window resize events.

* Reduce 00*.log spam by default.

* NOTE: CTRL-Z is still not properly handled, it works once or twice
  and then stops working.  Not sure why.

dsynth - Add IGNORE handling

* Add IGNORE handling (this also handles BROKEN)

* Do one final ncurses update when the build ends.

* Don't allocate "" strings, just leave the fields NULL for most of
  the strings in the pkg structure.

dsynth - Add IGNORE/BROKEN

* Implement IgnoreCount

* Only clear the 0*.log files for build directives.  Don't clear them
  for things like 'dsynth cleanup'.

dphase - Stabilization

* Fix formatting bug for 'Left', 'Failed', and 'Skipped'.  Use
  %-5d instead of %-4d. Doh!

dphase - Stabilization

* Fix a memory leak in scanit()

* When fork()/forkpty() fails, provide a reason in log 00.

kernel - Fix devfs bitmap races for pty and other devices

* Use an internal lock to protect the integrity of bitmap
  operations for devfs_clone_bitmap_*() functions, allowing
  devices to use the functions without having to lock themselves.

* Devices which use devfs_clone_bitmap_chk() + devfs_clone_bitmap_set()
  sequences have to be more careful, either having their own covering
  lock or checking the return code from devfs_clone_bitmap_set() and
  looping.

* This fixes serious /dev/ptmx pty allocation races which become obvious
  when pty's are allocated concurrently at a high rate, such as by the
  dsynth code.

dsynth - Stabilization

* Fix a double dot caused by the last commit.

* Fix missing free()'s and a few double-free()s.

* Fix failure cases in the Template copy code.

dsynth - Add #define for compression algo, default to .tgz

* Add a #define in dsynth.h to support .tar, .tgz, .txz, or .tbz.
  It is currently hardwired to .tgz.

* Switch to using .tgz.  While .txz is highly compressed, .tgz
  will decompress 10x more quickly and this seriously improves
  bulk build times.

dsynth - Fix waitpid() bug, revamp phase poll loop

* Fix an issue where the wrong wait*() status is processed as
  the error code.  This could occur especially on slow machines.

* Be sure to clean out any log data on the pty while and after
  processing the wait*() on the primary pid as there might be
  some as-yet unprocessed data.

* Use non-blocking mode on the masterpty to ensure that we
  don't accidently block on it.

dsynth - Add manual page, change configuration directory, more

* Change the configuration location from /usr/local/etc/synth.ini
  to /etc/dsynth/dsynth.ini and (as an alternate)
  /usr/local/etc/dsynth/dsynth.ini

* Add 'dsynth debug <port>' directive.  This will build all
  dependencies and then build the specified port and freeze
  its worker slot with mounts still intact.

* Add the 'dsynth init' directive.  This will initialize the
  /etc/dsynth directory and default configuration files.
  This directive will refuse to do anything if /etc/dsynth
  or /usr/local/etc/dsynth already exists.

* Allow 'Number_of_builders' and/or 'Max_jobs_per_builder' to
  be specified as 0.  If 0, these parameters will be
  auto-configured.

* Add the dsynth(1) manual page.

* Include a procfs (/proc) mount in the worker chroot.

* Change a 'cpdup' to a 'cp' to improve portability.  Also change
  how the Template is generated a bit.

dsynth - Improve DynamicMaxWorkers

* Allow the slow-start mode to be overriden with an
  options (-sN)

* Do not count swapcache swap in the swap calculation

* When dynamic worker limiting is active, display the
  limit in the upper-right-hand corner of the ncurses
  display.  When not, clear that field for brevity.

* The packaging step now pre-scans the package directory
  and deletes and '.new' files that might have been left
  over from an interrupt.

kernel - Fix dangling ref on fifo vnode

* Fix dangling refs on the fifo vnode which prevent unmount from
  succeeding.

* Fix open/close races which can occur during fifo open and close
  due to blocking conditions.  Use a lockmgr lock instead.

* Retain use of vp->v_token to help interlock read, write, and
  ioctl operations.

* This situation occurs when a filesystem fifo (aka via mkfifo)
  blocks on open() and is then interrupted.

kernel - Fix dangling ref on fifo vnode

* Fix dangling refs on the fifo vnode which prevent unmount from
  succeeding.

* Fix open/close races which can occur during fifo open and close
  due to blocking conditions.  Use a lockmgr lock instead.

* Retain use of vp->v_token to help interlock read, write, and
  ioctl operations.

* This situation occurs when a filesystem fifo (aka via mkfifo)
  blocks on open() and is then interrupted.

kernel - Rejigger midistat functions to close a race

* Make sure lock has full coverage across midistat_open() and
  midistat_read().  The temporary drop of the lock in midistat_read()
  lead to a race which allows one to read kernel memory beyond the
  end of the sbuf buffer.

* Rejigger the code to remove the global offset and just use
  uio_offset, which also fixes the same race (but leave the
  lock coverage in place regardless).

Taken-From:     FreeBSD
Security:       CVE-2019-5612

kernel - merge m_pulldown() bug fix

In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,
ensure that the subsequent mbuf contains the remainder of the bytes
the caller sought. If this is not the case, fall through to the code
which gathers the bytes in a new mbuf.

This fixes a bug where m_pulldown() could fail to gather all the desired
bytes into consecutive memory.

Patch-by:       sephe
FreeBSD-PR:     238787
Reported by:    A reddit user
Discussed with: emaste
Obtained from:  NetBSD
Obtained from:  FreeBSD

kernel - Rejigger midistat functions to close a race

* Make sure lock has full coverage across midistat_open() and
  midistat_read().  The temporary drop of the lock in midistat_read()
  lead to a race which allows one to read kernel memory beyond the
  end of the sbuf buffer.

* Rejigger the code to remove the global offset and just use
  uio_offset, which also fixes the same race (but leave the
  lock coverage in place regardless).

Taken-From:     FreeBSD
Security:       CVE-2019-5612

kernel - merge m_pulldown() bug fix

In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,
ensure that the subsequent mbuf contains the remainder of the bytes
the caller sought. If this is not the case, fall through to the code
which gathers the bytes in a new mbuf.

This fixes a bug where m_pulldown() could fail to gather all the desired
bytes into consecutive memory.

Patch-by:       sephe
FreeBSD-PR:     238787
Reported by:    A reddit user
Discussed with: emaste
Obtained from:  NetBSD
Obtained from:  FreeBSD

dsynth - Stabilization

* Set the PORTSDIR and BATCH environment variables for bulk pkglist
  operations (these were already being set properly for chroot'd
  operations).

  This will fix the PORTSDIR prefix test and removal for dependencies
  done by resolveDepString().

dsynth - Optimize pty use

* Have the WORKER process allocate the pty once and use it in all
  phases, instead of allocating and closing a new pty for each phase.

* Make N copies of /bin, /lib, /usr/bin, and /libexec into the base
  directory and mount those in the chroots instead of the system
  directory, essentially unsharing these directories.  Each worker
  gets an independent copy.

  This should reduce contention and cache-line ping-ponging overhead
  on /bin/sh, libc, make, etc, significantly.

* Create the template conditionally whether 'pkg' has been built or
  not.

dsynth - Get skips working, etc

* Get 'Skipped' working, and print a nice depdency chain for the reason
  why a packge got skipped.

* Added swap% to the dynamic worker reduction algorithm.  It startsr at
  10% swap used and maxes out at a 75% reduction with 40% of swap used.

* Add a slow-start feature so dsynth does not try to load all workers
  at startup simultaneously.  This gives the swap% and loadavg worker
  reduction code a chance to activate before the system becomes totally
  smashed.

* Cache the 0*.log logfile descirptors instead of open/write/close.

route: Remove ref counting around rt_newaddrmsg().

Doesn't seem to be needed anymore.
As proposed by sephe.

Merge remote-tracking branch 'remotes/crater/vendor/BYACC'

dsynth - Misc work

* Try to get Skipped working, propagate the NOBUILD flag upward
  and process it as a Skipped in startbuild().

* Change descriptor 0 in the forkpty()'d sub-process to /dev/null
  to try to shortcut ports which sometimes ask questions.
  (We do set BATCH as well, but some ports still ask questions).

* Change the dynamic MaxWorkers adjustment to reduce MaxWorkers
  by up to 75% at (5 * ncpus) of load.  The adjustment begins
  at 0% at (1.5 * ncpus) load.

dsynth - Reap processes more generally in the poll loop

* Adjust the dsynth worker process to reap any pid needing
  reaping within the poll loop instead of just checking the
  primary exec'd make pid.

* This catches any orphaned children that would otherwise
  accumulate during the build.

dsynth - Misc/stabilization

* Add missing environment variables PORT_DBDIR, PACKAGE_BUILDING,
  and PKG_CREATE_VERBOSE.

* Increase field width for Impulse and Pkg/hour to 5 digits.

* Fix bugs in the dynamic MaxWorkers reduction code (based on load).

dsynth - Fleshout functions

* Add dynamic MaxWorker reduction based on the load.  The number of
  workers will be reduced by (load * 2 / ncpus) when the load
  exceeds (ncpus * 2.0), with a floor of min(4, MaxWorkers).

  The idea here is to reduce unnecessary swap use when building
  larger packages that push the load up greatly.

* Clean-up terminal spam after ncurses has been turned on.

* Generate fatal error if required configured directories are
  missing.

* Add -y (yes to all) option.

* Implement 'status' and 'status-everything'

* Implement 'purge-distfiles' (initial work)

* In scans that dump percentages, print 100% at the end before
  moving onto the next step.