repo: keep the catalogue when an update fails
pkg_repo_binary_update() removed the repository database whenever an
update failed, including when the failure happened before the database
was touched (e.g. the network dropped while fetching the catalogue) or
after the previous database had been restored by
pkg_repo_binary_update_proceed(). Any transient failure therefore threw
away a usable catalogue, and the following commands failed with
"Repository X cannot be opened. 'pkg update' required" until the next
successful update.
Only remove the database when it cannot be opened anymore, so a failed
update keeps the previous catalogue while the next update can still
start fresh from an unusable one.
Fixes: #2763
scheduler: take provides/requires into account when ordering jobs
Shared libraries and named provides are resolved through the provides/requires
tables and are not listed in the regular dependencies, so they were
ignored when ordering jobs. Order a package providing a shared library
or a named provide before the packages requiring it, so that for example
libc (FreeBSD-clibs) is installed before the programs using it.
The install graph can be cyclic: FreeBSD-runtime provides libraries
required by FreeBSD-libarchive and in turn requires a library provided by
it. Such a cycle cannot be broken by splitting upgrade jobs, so the
strongly connected components of the install graph are computed with
Tarjan's algorithm and the jobs of a component are scheduled as a single
block, so that jobs depending on only one member of the component are
never interleaved between its members.
Add frontend tests covering both the shared library and the named provide
mechanisms.
checksum: fix the package digest buffer size
pkg_checksum_generate() prepends a "<version>$<type>$" prefix that
pkg_checksum_type_size() does not account for, so the *_hex types kept
the prefix and lost the digest. Size the buffer with the new
pkg_checksum_string_size().
checksum: add a standard z-base-32 encoding
The *_base32 types use a pkg-specific bit order, they are not z-base-32.
Keep them for compatibility and add the *_zbase32 types using a
standards-compliant encoder, plus blake2/s_hex variants.
Fixes: #2759
libpkg: recognize loongarch64 as loongarch:64
Add PKG_ARCH_LOONGARCH64 and map EM_LOONGARCH/ELFCLASS64 to it; no
32-bit ABI exists. Test it in the frontend suite.
external/libecc: add __loongarch__ to words.h. Upstream, from
https://github.com/libecc/libecc/pull/15/.
pkg: minor fixes to the vec iterators
Clear the caller"s pointer when the list is empty, as the previous
implementation did, and declare the index before the statements.
repo: simplify how the Hashed directory is created
Reuse tmp_name (which already holds the package directory) and append
"/" PKG_HASH_DIR with strlcat() instead of formatting it into a
dedicated buffer.
This is an attempt to fix crash on macos
tests: do not fail http cleanup without a server
The http tests start a python http server and kill it from their
cleanup. When the body is skipped (python3 is missing) no server was
ever started and httpd_cleanup() returned 1, which made atf report the
test as broken instead of skipped.
repo: fix a hang with hashed repositories
hash_file() built the Hashed directory with
snprintf(tmp_name, sizeof(tmp_name), "%s/%s", rel_dir, PKG_HASH_DIR);
where rel_dir points at tmp_name itself: passing the destination of
snprintf(3) as one of its inputs is undefined. glibc and musl produce
"/Hashed" instead of "<dir>/Hashed", so pkg_mkdirs() creates the wrong
directory and the rename(2) that follows fails with ENOENT.
The failing worker then left pkg_create_repo_thread() without
incrementing ntask nor signalling cond, so the main thread waited
forever in pthread_cond_wait(): "pkg repo" hung on every platform but
FreeBSD, whose libc happened to produce the expected string.
Use a dedicated buffer and signal the main thread before bailing out.
clean: make the keep_installed_archive test reliable
A file rebuilt within the same second keeps the same mtime, and the
file:// fetcher then answers EPKG_UPTODATE: the cached catalogue is left
unchanged and pkg clean keeps the archive of the removed package.
Wait a second before rebuilding the repository so that the new catalogue
is newer than the cached one.
fetch: test the nominal 304 up to date path
Add a test where the server honours If-Modified-Since and answers 304,
checking that pkg reports the catalogue as up to date and that the
server really answered 304, next to the test covering servers that
always answer 200.
fetch: detect up to date responses without a 304
The libfetch fetcher returned EPKG_UPTODATE only when the server
answered 304 Not Modified. A server answering 200 to a conditional
request made pkg download the repository catalogue again on every
update (issue #2742).
Compare the mtime of the response with the one sent in
If-Modified-Since and return EPKG_UPTODATE when it did not change,
like the file:// fetcher already does.
metalog: give each escaped field its own buffer
metalog_escape_path() returned a single static buffer, so escaping the
path and the link target relied on the argument evaluation order and was
not reentrant. Let the caller provide the buffer and write the link
entry with a single fprintf.
install|upgrade: confirm the removal of explicit packages
Recall how many packages will be removed in the confirmation prompt, and ask
for an extra confirmation when some were installed explicitly.
pkgdb: add pkg_attr and keep installed archives
pkg_attr holds extended package attributes. Unlike pkg_local it is tied
to the installed instance: purged on upgrade and on uninstall.
Record the archive a package is installed from in pkg_attr so that pkg
clean keeps it until the package is uninstalled, without a schema change.
pkgdb: add a local package attributes store
pkg_local holds local (user) attributes of installed packages. It is
created on demand and kept out of the versioned schema, so that
introducing a new local attribute never requires a migration. It is
keyed by package name so that local state survives upgrades.
solve: hold upgrades back instead of removing explicit packages
When the solver cannot satisfy the problem, it used to pick the last
failed assumption and negate it, which is both arbitrary (it depends on
the order in which the packages are processed) and often unnecessary: a
package the user installed explicitly ended up removed only because an
upgrade of a dependency had been asked for.
Rank the assumptions instead:
1/ explicit requests
2/ explicit removals
3/ keeping a package that was installed explicitly
4/ then upgrading such a package,
5/ finally upgrades other packages.
When the problem is unsatisfiable, give up the least important assumptions
and try again.
Forcing (-f) remains the escape hatch and the old behaviour is kept when
nothing can be given up.
[5 lines not shown]
install|upgrade: add -N/--no-remove and NEVER_REMOVE
install or upgrade sometimes has to remove installed packages that were
not requested for removal. -N/--no-remove, and the NEVER_REMOVE
configuration option, refuse those
removals.
