DragonFlyBSD/src 9afa2dausr.bin/kcollect gnuplot.c

kcollect - Remove system calls from graph

* System calls make the graph too scratchy.  Remove them for now.

DragonFlyBSD/src a67bf8dusr.bin/dsynth pkglist.c dsynth.c

dsynth - Stabilization, improve compatibility

* Disable /etc/make.conf for the bulk scan (__MAKE_CONF=/dev/null)

* Add -D to turn on developer mode (from zrj)

* Add a little infrastructure to cache ports variables, starting
  with _PERL5_FROM_BIN.

DragonFlyBSD/src 7b609b0usr.bin/dsynth build.c status.c

dsynth - Do not count DUMMY nodes in the total

* dsynth uses DUMMY nodes to wrap all the flavors for a particular
  package when other packages have a generic (rather than flavor-specific)
  dependency to it.

* Do not count these nodes in the total for display purposes.

DragonFlyBSD/src 5f38e86sys/dev/drm/include/linux mm.h, sys/dev/drm/ttm ttm_bo_vm.c

drm/ttm: Partially sync the vm fault code with Linux

Prevent crashes with Mesa-using applications.

DragonFlyBSD/src 2f7844asys/dev/drm/include/linux mm_types.h mm.h

drm/linux: Move vm_area_struct definition to the right header

DragonFlyBSD/src d6f6f23sys/dev/drm/radeon radeon_cs.c

drm/radeon: Use mm->mmap_sem

DragonFlyBSD/src 0e32b8csys/dev/drm linux_sched.c drm_drv.c, sys/dev/drm/include/asm current.h

drm - Refactor task_struct and implement mm_struct

* Change td->td_linux_task from an embedded structure to a pointer.

* Add p->p_linux_mm to support tracking mm_struct's.

* Change the 'current' macro to test td->td_linux_task and call
  a support function, linux_task_alloc(), if it is NULL.

* Implement callbacks from the main kernel for thread exit and
  process exit to support functions that drop the td_linux_task and
  p_linux_mm pointers.

  Initialize and clear these callbacks in the module load/unload
  in drm_drv.c

* Implement required support functions in linux_sched.c

DragonFlyBSD/src d78d3a2sys/dev/drm/amd/include/asic_reg/dce dce_11_2_sh_mask.h dce_11_2_d.h, sys/dev/drm/radeon radeon_drv.h cik.c

drm/radeon: Update to Linux 4.7.10

* Various bugfixes, especially related to Displayport and
  vblank handling

* Sea Islands and Southern Islands GPUs support improvements

* Various quirks to fix problems for specific cards, mostly
  Radeon R7 370 and Radeon R9 270X models

DragonFlyBSD/src 0757a5esys/dev/drm/include/linux rwsem.h

drm/linux: down_read() acquiress a shared lock

DragonFlyBSD/src 516819dusr.bin/dsynth build.c dsynth.h

dsynth - Account for extraction size

* Account for the initial size of /construction after the extract
  phase.

  Note that we explicitly are not accounting for space used after this
  phase because they files (typically .o's and such) are added incrementally
  and often not accessed continuously until the ending link phase.  Including
  this space thus might cause dsynth to retard the worker count too much.

DragonFlyBSD/src 1fa9d80usr.bin/dsynth build.c

dsynth - More pkg dependency fixes

* Fix the depency check based on the depth when recursing through a
  dummy node.  Dummy nodes do not increment the depth.

* In addition, recursing through a dummy node means that the package
  only depends on the default flavor, which is only the first element
  in the idepon list under that dummy node.  Inform the next layer
  that its scan should only iterate one element.

* In addition, when build_find_leaves() recurses through a dummy node
  also use the same mechanism where only the first sub-node is
  checked, because the dependency is (again) against just the default
  flavor.

  Note that when doing a 'dsynth everything', all the dependencies
  wind up in the root list, so all the flavors get built.  But when
  building a specific subset of packages, this fix will reduce the
  number of flavors that have to be built as dependencies.
DeltaFile
+96-33usr.bin/dsynth/build.c
+96-331 files

DragonFlyBSD/src f29c083usr.bin/dsynth gui.c

dsynth - Unbreak ncurses coloring

* Fixup the log filter for the ncurses display.

DragonFlyBSD/src 3cebe4ausr.bin/dsynth subs.c gui.c

dsynth - Report phase on failure

* Report the phase a failure occurs at.

* When logging each phase, use 79 dashes as a separator.

DragonFlyBSD/src 32811a0usr.bin/dsynth subs.c

dsynth - Use foreground color instead of background color

* Set the foreground color instead of the background color,
  just to not be too invasive.

DragonFlyBSD/src a574cbfusr.bin/dsynth config.c build.c

dsynth - pty CRs, readelf, better log formatting

* Remove CR's from build logs (set tty modes on the pty).

* Parse 'readelf -x .note.tag /bin/sh' to get the operating
  system and __DragonFly_version... basically the release name.
  Compare the operating system name against what we get back
  from the system as a validator.

* Improve the log format to make it more readable.  Certain elements such
  as statistics and times are right-justified.

* Filter certain log lines from stdout when curses mode is turned off.

* Highlight stdout lines for SUCCESS in green and FAILURE and red.

DragonFlyBSD/src 294990eusr.bin/dsynth config.c

dsynth - Fix VersionName (UNAME_v)

* Fix VersionName (UNAME_v) to use the format e.g.
  "DragonFly 5.7-SYNTH".

* Not yet pulled from elf info, currently using sysctl.

DragonFlyBSD/src 63fcce5usr.bin/dsynth build.c pkglist.c

dsynth - Reduce number of pre-installed pkgs in worker

* The top-level scan installs all dependencies, but the deeper recursion
  only needs to install LIB and RUN dependencies.

DragonFlyBSD/src 6fd6793usr.bin/dsynth build.c subs.c

dsynth - Fix templating bugs, flesh out environment and make.conf

* Fix a few templating bugs

* Generate a make.conf in each chroot/jail based on the configured
  environment.

* Implement the make environment in the pkglist commands as make
  variables on the command line.

* Use vfork/exec in the bulk code... generate the argument vector
  and environment ourselves.

* Ignore '/nonexistent:BLAH' dependencies.

DragonFlyBSD/src e4f26d7sys/dev/drm/radeon evergreen.c

drm/radeon: Disable acceleration on evergreen

This is a workaround for https://bugs.dragonflybsd.org/issues/3198

DragonFlyBSD/src c22b037sys/dev/drm drm_ioctl.c

drm: Partially sync drm_ioctl() with Linux 4.7.10

DragonFlyBSD/src 88c24d7usr.bin/dsynth build.c dsynth.h

dsynth - Add a pkgi depth calculation

* Calculate the largest tree-depth of packages that depend on PKG
  and use it in the sort as weighting to try to prioritize building
  PKG.

  The intent is to try to knock out these deep chains as quickly as
  possible so as not to get stuck on a linear chain of dependencies
  later on in the build.

DragonFlyBSD/src 7f0eca5usr.bin/dsynth build.c gui.c

dsynth - Stabilization

* Rejigger the dynamic load management calculations and fix bugs in
  how the restrictions based on pkgdep memory use are calculated.

  The pkgdep memory mechanism now supports both slow-start and
  slow-approach.  The slow-approach mechanism begins when pkgdep
  memory use goes about 1/2 the target (typically 1/4 physical memory).

  In slow-approach mode, the number of workers is increased at a
  rate of 1 every 30 seconds instead of 1 every 5 seconds in order
  to give workers time to install the packages and fill in the estimated
  memory use.  The idea is that this then allows paging to ramp up
  more slowly instead of getting 'shocked' by a dozen slots all
  issuing install-pkgs at the same time.

* Add -m gb  (in gigabytes), to override the default pkgdep memory
  target.

* For now stop mounting /usr/src in the workers.

* NCurses disablement now works as expected.

* Calculate 1 gigabyte as 1GiB.


    [2 lines not shown]

DragonFlyBSD/src fef2fc6usr.bin/dsynth build.c config.c

dsynth - Add Dynamic memory load management

* In addition to using the load and swap, also use the total
  estimated installed package dependencies for each worker verses
  PhysMem and reduce the number of worker slots such that this
  total is less than 1/2 physical memory.

* Fix a ^C handling bug where dsynth would try to unmount while
  still holding a file descriptor open on the filesystem it is
  trying to unmount.

* Clean up the logs a bit more.

DragonFlyBSD/src 3232b77usr.bin/dsynth gui.c build.c

dsynth - Add color

* Color successful builds green and failed builds red.

* Refactor some of the log mesages.

DragonFlyBSD/src 54f2fefusr.bin/dsynth gui.c build.c

dsynth - Add logfile tracking on the ncurses display

* Track the contents of the 00*.log file below the list of
  workers.

* Handle window resize events.

* Reduce 00*.log spam by default.

* NOTE: CTRL-Z is still not properly handled, it works once or twice
  and then stops working.  Not sure why.

DragonFlyBSD/src 4ea2ee4usr.bin/dsynth pkglist.c build.c

dsynth - Add IGNORE handling

* Add IGNORE handling (this also handles BROKEN)

* Do one final ncurses update when the build ends.

* Don't allocate "" strings, just leave the fields NULL for most of
  the strings in the pkg structure.

DragonFlyBSD/src 87017acusr.bin/dsynth build.c dsynth.c

dsynth - Add IGNORE/BROKEN

* Implement IgnoreCount

* Only clear the 0*.log files for build directives.  Don't clear them
  for things like 'dsynth cleanup'.

DragonFlyBSD/src 6c80619usr.bin/dsynth gui.c

dphase - Stabilization

* Fix formatting bug for 'Left', 'Failed', and 'Skipped'.  Use
  %-5d instead of %-4d. Doh!

DragonFlyBSD/src 6a3a20busr.bin/dsynth build.c repo.c

dphase - Stabilization

* Fix a memory leak in scanit()

* When fork()/forkpty() fails, provide a reason in log 00.

DragonFlyBSD/src 1991e94sys/dev/disk/vn vn.c, sys/dev/misc/snp snp.c

kernel - Fix devfs bitmap races for pty and other devices

* Use an internal lock to protect the integrity of bitmap
  operations for devfs_clone_bitmap_*() functions, allowing
  devices to use the functions without having to lock themselves.

* Devices which use devfs_clone_bitmap_chk() + devfs_clone_bitmap_set()
  sequences have to be more careful, either having their own covering
  lock or checking the return code from devfs_clone_bitmap_set() and
  looping.

* This fixes serious /dev/ptmx pty allocation races which become obvious
  when pty's are allocated concurrently at a high rate, such as by the
  dsynth code.

DragonFlyBSD/src 8d9409busr.bin/dsynth mount.c pkglist.c

dsynth - Stabilization

* Fix a double dot caused by the last commit.

* Fix missing free()'s and a few double-free()s.

* Fix failure cases in the Template copy code.

DragonFlyBSD/src 667fb2cusr.bin/dsynth dsynth.h pkglist.c

dsynth - Add #define for compression algo, default to .tgz

* Add a #define in dsynth.h to support .tar, .tgz, .txz, or .tbz.
  It is currently hardwired to .tgz.

* Switch to using .tgz.  While .txz is highly compressed, .tgz
  will decompress 10x more quickly and this seriously improves
  bulk build times.

DragonFlyBSD/src 7880dcfusr.bin/dsynth build.c

dsynth - Fix waitpid() bug, revamp phase poll loop

* Fix an issue where the wrong wait*() status is processed as
  the error code.  This could occur especially on slow machines.

* Be sure to clean out any log data on the pty while and after
  processing the wait*() on the primary pid as there might be
  some as-yet unprocessed data.

* Use non-blocking mode on the masterpty to ensure that we
  don't accidently block on it.
DeltaFile
+114-31usr.bin/dsynth/build.c
+114-311 files

DragonFlyBSD/src 8ec23causr.bin/dsynth dsynth.1 dsynth.c

dsynth - Add manual page, change configuration directory, more

* Change the configuration location from /usr/local/etc/synth.ini
  to /etc/dsynth/dsynth.ini and (as an alternate)
  /usr/local/etc/dsynth/dsynth.ini

* Add 'dsynth debug <port>' directive.  This will build all
  dependencies and then build the specified port and freeze
  its worker slot with mounts still intact.

* Add the 'dsynth init' directive.  This will initialize the
  /etc/dsynth directory and default configuration files.
  This directive will refuse to do anything if /etc/dsynth
  or /usr/local/etc/dsynth already exists.

* Allow 'Number_of_builders' and/or 'Max_jobs_per_builder' to
  be specified as 0.  If 0, these parameters will be
  auto-configured.

* Add the dsynth(1) manual page.

* Include a procfs (/proc) mount in the worker chroot.

* Change a 'cpdup' to a 'cp' to improve portability.  Also change
  how the Template is generated a bit.

DragonFlyBSD/src f7f2583usr.bin/dsynth build.c repo.c

dsynth - Improve DynamicMaxWorkers

* Allow the slow-start mode to be overriden with an
  options (-sN)

* Do not count swapcache swap in the swap calculation

* When dynamic worker limiting is active, display the
  limit in the upper-right-hand corner of the ncurses
  display.  When not, clear that field for brevity.

* The packaging step now pre-scans the package directory
  and deletes and '.new' files that might have been left
  over from an interrupt.

DragonFlyBSD/src 39d3872sys/vfs/fifofs fifo_vnops.c

kernel - Fix dangling ref on fifo vnode

* Fix dangling refs on the fifo vnode which prevent unmount from
  succeeding.

* Fix open/close races which can occur during fifo open and close
  due to blocking conditions.  Use a lockmgr lock instead.

* Retain use of vp->v_token to help interlock read, write, and
  ioctl operations.

* This situation occurs when a filesystem fifo (aka via mkfifo)
  blocks on open() and is then interrupted.
DeltaFile
+155-57sys/vfs/fifofs/fifo_vnops.c
+155-571 files

DragonFlyBSD/src f9ad110sys/vfs/fifofs fifo_vnops.c

kernel - Fix dangling ref on fifo vnode

* Fix dangling refs on the fifo vnode which prevent unmount from
  succeeding.

* Fix open/close races which can occur during fifo open and close
  due to blocking conditions.  Use a lockmgr lock instead.

* Retain use of vp->v_token to help interlock read, write, and
  ioctl operations.

* This situation occurs when a filesystem fifo (aka via mkfifo)
  blocks on open() and is then interrupted.
DeltaFile
+155-57sys/vfs/fifofs/fifo_vnops.c
+155-571 files

DragonFlyBSD/src 30a075bsys/dev/sound/midi midi.c

kernel - Rejigger midistat functions to close a race

* Make sure lock has full coverage across midistat_open() and
  midistat_read().  The temporary drop of the lock in midistat_read()
  lead to a race which allows one to read kernel memory beyond the
  end of the sbuf buffer.

* Rejigger the code to remove the global offset and just use
  uio_offset, which also fixes the same race (but leave the
  lock coverage in place regardless).

Taken-From:     FreeBSD
Security:       CVE-2019-5612

DragonFlyBSD/src c24e389sys/kern uipc_mbuf2.c

kernel - merge m_pulldown() bug fix

In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,
ensure that the subsequent mbuf contains the remainder of the bytes
the caller sought. If this is not the case, fall through to the code
which gathers the bytes in a new mbuf.

This fixes a bug where m_pulldown() could fail to gather all the desired
bytes into consecutive memory.

Patch-by:       sephe
FreeBSD-PR:     238787
Reported by:    A reddit user
Discussed with: emaste
Obtained from:  NetBSD
Obtained from:  FreeBSD

DragonFlyBSD/src f70430dsys/dev/sound/midi midi.c

kernel - Rejigger midistat functions to close a race

* Make sure lock has full coverage across midistat_open() and
  midistat_read().  The temporary drop of the lock in midistat_read()
  lead to a race which allows one to read kernel memory beyond the
  end of the sbuf buffer.

* Rejigger the code to remove the global offset and just use
  uio_offset, which also fixes the same race (but leave the
  lock coverage in place regardless).

Taken-From:     FreeBSD
Security:       CVE-2019-5612

DragonFlyBSD/src 15ce538sys/kern uipc_mbuf2.c

kernel - merge m_pulldown() bug fix

In m_pulldown(), before trying to prepend bytes to the subsequent mbuf,
ensure that the subsequent mbuf contains the remainder of the bytes
the caller sought. If this is not the case, fall through to the code
which gathers the bytes in a new mbuf.

This fixes a bug where m_pulldown() could fail to gather all the desired
bytes into consecutive memory.

Patch-by:       sephe
FreeBSD-PR:     238787
Reported by:    A reddit user
Discussed with: emaste
Obtained from:  NetBSD
Obtained from:  FreeBSD

DragonFlyBSD/src 4e6f51fusr.bin/dsynth dsynth.c

dsynth - Stabilization

* Set the PORTSDIR and BATCH environment variables for bulk pkglist
  operations (these were already being set properly for chroot'd
  operations).

  This will fix the PORTSDIR prefix test and removal for dependencies
  done by resolveDepString().

DragonFlyBSD/src 1d6e00cusr.bin/dsynth mount.c build.c

dsynth - Optimize pty use

* Have the WORKER process allocate the pty once and use it in all
  phases, instead of allocating and closing a new pty for each phase.

* Make N copies of /bin, /lib, /usr/bin, and /libexec into the base
  directory and mount those in the chroots instead of the system
  directory, essentially unsharing these directories.  Each worker
  gets an independent copy.

  This should reduce contention and cache-line ping-ponging overhead
  on /bin/sh, libc, make, etc, significantly.

* Create the template conditionally whether 'pkg' has been built or
  not.

DragonFlyBSD/src 3699ee0usr.bin/dsynth build.c subs.c

dsynth - Get skips working, etc

* Get 'Skipped' working, and print a nice depdency chain for the reason
  why a packge got skipped.

* Added swap% to the dynamic worker reduction algorithm.  It startsr at
  10% swap used and maxes out at a 75% reduction with 40% of swap used.

* Add a slow-start feature so dsynth does not try to load all workers
  at startup simultaneously.  This gives the swap% and loadavg worker
  reduction code a chance to activate before the system becomes totally
  smashed.

* Cache the 0*.log logfile descirptors instead of open/write/close.

DragonFlyBSD/src bb060a4sys/net route.c, sys/netinet6 in6.c

route: Remove ref counting around rt_newaddrmsg().

Doesn't seem to be needed anymore.
As proposed by sephe.

DragonFlyBSD/src d3df5ac

Merge remote-tracking branch 'remotes/crater/vendor/BYACC'
DeltaFile
+0-00 files

DragonFlyBSD/src 9c4c701usr.bin/dsynth build.c dsynth.h

dsynth - Misc work

* Try to get Skipped working, propagate the NOBUILD flag upward
  and process it as a Skipped in startbuild().

* Change descriptor 0 in the forkpty()'d sub-process to /dev/null
  to try to shortcut ports which sometimes ask questions.
  (We do set BATCH as well, but some ports still ask questions).

* Change the dynamic MaxWorkers adjustment to reduce MaxWorkers
  by up to 75% at (5 * ncpus) of load.  The adjustment begins
  at 0% at (1.5 * ncpus) load.

DragonFlyBSD/src e86766eusr.bin/dsynth build.c

dsynth - Reap processes more generally in the poll loop

* Adjust the dsynth worker process to reap any pid needing
  reaping within the poll loop instead of just checking the
  primary exec'd make pid.

* This catches any orphaned children that would otherwise
  accumulate during the build.

DragonFlyBSD/src 64824e6usr.bin/dsynth build.c gui.c

dsynth - Misc/stabilization

* Add missing environment variables PORT_DBDIR, PACKAGE_BUILDING,
  and PKG_CREATE_VERBOSE.

* Increase field width for Impulse and Pkg/hour to 5 digits.

* Fix bugs in the dynamic MaxWorkers reduction code (based on load).

DragonFlyBSD/src 1645cafusr.bin/dsynth status.c repo.c

dsynth - Fleshout functions

* Add dynamic MaxWorker reduction based on the load.  The number of
  workers will be reduced by (load * 2 / ncpus) when the load
  exceeds (ncpus * 2.0), with a floor of min(4, MaxWorkers).

  The idea here is to reduce unnecessary swap use when building
  larger packages that push the load up greatly.

* Clean-up terminal spam after ncurses has been turned on.

* Generate fatal error if required configured directories are
  missing.

* Add -y (yes to all) option.

* Implement 'status' and 'status-everything'

* Implement 'purge-distfiles' (initial work)

* In scans that dump percentages, print 100% at the end before
  moving onto the next step.