src: fix static build with -pie in LDFLAGS
Add -Wl,--no-pie when linking pkg-static to avoid static PIE
which requires PIC in all objects, breaking with non-PIC system
static libraries on FreeBSD.
fix: non-root reads of WAL db with missing sidecars
A read-only connection to a WAL-mode local database reports
SQLITE_OK from sqlite3_open_v2() but every query fails when the
-wal/-shm sidecars are absent (e.g. removed after root"s clean
close) or not accessible. The fallback to immutable=1 only
triggered on open failure, so it never fired. This broke
non-root pkg info in poudriere"s configure phase, causing
false "not found" and spurious install attempts.
Probe the read-only connection with a schema read after open;
on failure reopen in immutable=1, which bypasses WAL and reads
the main file directly.
fix: non-root reads of WAL db with missing sidecars
A read-only connection to a WAL-mode local database reports
SQLITE_OK from sqlite3_open_v2() but every query fails when the
-wal/-shm sidecars are absent (e.g. removed after root"s clean
close) or not accessible. The fallback to immutable=1 only
triggered on open failure, so it never fired. This broke
non-root pkg info in poudriere"s configure phase, causing
false "not found" and spurious install attempts.
Probe the read-only connection with a schema read after open;
on failure reopen in immutable=1, which bypasses WAL and reads
the main file directly.
2.7.99.3
Bump version to 2.7.99.3 and add the NEWS section covering changes
since the 2.7.99.2 release (ee453993f): DB schema/WAL overhaul,
OSVf 1.7.5, plist @for loops, security fixes including the "snap"
version parsing OOB (#2691) and pkg_printf format parsing, and an
update of the vendored SQLite amalgamation to 3.53.2.
tests/fuzz: fix plist harness
Store the fmemopen FILE* in a FILE* (not int, which truncated the
pointer) and parse through the exported plist_parse_line() instead
of the static plist_parse(), so the harness actually builds and runs.
fix: OOB and hangs in version/printf parsing
- pkg_version: fix the "snap" stage entry, whose namelen was 6 for a
4-char name, causing a heap OOB read and dangling pointer in
get_component on versions ending in "snap". Set namelen to 4 and
value to "s"-"a"+ABASE so snap sorts after alpha as intended (#2691)
- pkg_printf: stop format_code from reading f[1] past end of string,
and emit a lone "%" literally instead of looping forever when no
valid format code follows
- event: cast nbdigits to int for the %* field width
- fetch_file: print the offending URL in the "hostname too long"
error message
xstring: rename xprintf/xputs/xputc/xwrite/xflush to xstring_*
Rename the convenience macros to use the xstring_ prefix for
consistency with xstring_new/xstring_reset/xstring_renew.
pkg_osvf: Add osv_type for VuXML compatibility
As VuXML audit currently uses pkg_audit_version type
field different than pkg_osvf make sure that OSVf implementation
tries to mimic VuXML version checking code.
For keeping OSVf version compatibility in order introduce
osv_type which now contains OSVf events information
main: Don't set debug level twice
Now that pkg_ini() no longer clears the debug level, it is enough to set
it once, immediately after the getopt_long() loop.
libpkg: Fix early debug
* Initialize ctx.debug_flags to all-bits-one so pkg_dbg() can be used
before pkg_ini() read the configuration.
* In pkg_ini(), only set ctx.debug_level if the new value (from the
configuration or environment) is greater than the current value.
fix: move PRAGMA journal_mode/synchronous out of upgrade SQL
PRAGMA journal_mode and synchronous cannot be changed inside a
transaction. The v40 upgrade SQL bundled DROP INDEX statements
(which implicitly start a transaction) together with PRAGMAs,
causing a "Safety level may not be changed inside a transaction"
error.
Fix: remove PRAGMAs from db_upgrades.h v40 and set synchronous=NORMAL
in pkgdb_open right after journal_mode=WAL, where no transaction
is active.
run_prstmt: skip expensive sqlite3_expanded_sql when debug is off
sqlite3_expanded_sql formats the entire SQL with all bound values
substituted. This was called unconditionally on every prepared
statement execution, even in production (debug_level=0), wasting
CPU cycles on memory allocation and string formatting.
Guard the call with ctx.debug_level >= 4, matching the pattern
already used by pkgdb_debug().
shlibs: add missing indexes on shlib_id for faster library lookups
pkgdb_query_shlib_require/provide join through shlib_id but no index
existed on that column. Without an index, SQLite does a full table scan
of pkg_shlibs_required/provided to find matching rows for a given shlib.
Added indexes (4 for local DB, 2 for binary repo):
pkg_shlibs_required_shlib_id(shlib_id)
pkg_shlibs_required_ignore_shlib_id(shlib_id)
pkg_shlibs_provided_shlib_id(shlib_id)
pkg_shlibs_provided_ignore_shlib_id(shlib_id)
Local DB: schema bumped to v42 with upgrade path for existing databases.
Binary repo: added to binary_repo_initsql.
flavors: replace duplicated CTE with persistent view in both databases
The WITH flavors AS (...) CTE was duplicated across 5 queries in
pkgdb_query.c and repo/binary/query.c. Replace it with a persistent
VIEW created in both the local and binary repo schemas.
Changes:
- binary_private.h: add CREATE VIEW to binary_repo_initsql
- pkgdb.c: add CREATE VIEW to pkgdb_init() for fresh databases
- db_upgrades.h: add upgrade v41 to create the view on existing DBs
- pkgdb_query.c: remove the 2 duplicated CTE definitions
- repo/binary/query.c: remove the 2 duplicated CTE definitions
- pkgdb.c: bump DB_SCHEMA_MINOR 40 -> 41
binary repo: use 16K pages and synchronous=OFF during bulk update
- page_size: switch from getpagesize() (typically 4K) to 16384 for
better B-tree cache efficiency with millions of rows
- synchronous: set to OFF during bulk INSERT phase; the database is
re-opened with synchronous=FULL on subsequent access via
pkg_repo_binary_init, so durability is maintained for read queries
local db: remove 15 redundant indexes, switch to WAL+NORMAL, bump schema v40
Each *_{package_id} index is redundant: all junction tables use
PRIMARY KEY(package_id, other_id) or UNIQUE(package_id, other_id),
which implicitly creates an index with package_id as leading column.
The explicit single-column index adds storage and INSERT overhead
with zero query-plan benefit.
Dropped indexes (15 total):
pkg_script_package_id, pkg_directories_package_id,
pkg_categories_package_id, pkg_licenses_package_id,
pkg_users_package_id, pkg_groups_package_id,
pkg_shlibs_required_package_id, pkg_shlibs_required_ignore_package_id,
pkg_shlibs_provided_package_id, pkg_shlibs_provided_ignore_package_id,
pkg_annotation_package_id, pkg_conflicts_pid,
pkg_provides_id, pkg_requires_package_id, pkg_lua_script_package_id
PRAGMA changes:
- journal_mode: TRUNCATE -> WAL (better concurrent read performance)
[4 lines not shown]
pkgdb_iterator: convert implicit SQL-89 JOINs to explicit JOIN syntax
Replaced 11 comma-separated implicit JOIN queries with explicit
JOIN ... ON syntax for clarity and maintainability:
- pkg_directories, directories
- pkg_licenses, licenses AS l
- pkg_categories, categories AS c
- pkg_users, users
- pkg_groups, groups
- pkg_shlibs_required(/ignore/provided/ignore), shlibs AS s
- pkg_provides, provides AS s
- pkg_requires, requires AS s
pkgdb_query: optimized SQL queries and removed BUFSIZ limit
- Removed GROUP BY p.id from exact-match which query (path TEXT PRIMARY KEY
guarantees uniqueness per file across all packages; GROUP BY retained for
GLOB mode)
- Changed LEFT JOIN to INNER JOIN in which query (WHERE f.path = ? filters
NULLs, making LEFT JOIN misleading)
- Replaced fixed BUFSIZ buffers with xasprintf() for safe dynamic allocation
in pkgdb_query_cond, pkgdb_query_which, and pkgdb_file_exists
- Simplified pkgdb_file_exists to a const string instead of snprintf
rwhich: remove redundant pkg_files indexes
The pkg_files table has a UNIQUE(package_id, dir_id, name) constraint
which implicitly creates an index with package_id as the leading column.
This serves any WHERE package_id=? query just as efficiently as a
dedicated index, making pkg_files_package_id redundant.
The pkg_files_name index is unused: no query in the binary repo filters
by name alone - file_which either matches (dir_id, name) via the
composite index, or uses a GLOB on a concatenated expression which
cannot use any index.
Only pkg_files_dir_name(dir_id, name) is retained, as it is used by the
exact-match path in pkg_repo_binary_file_which.
rwhich: replace SQL subquery with hashmap for dir_id lookups
The FILEDIR2 prepared statement previously used a subquery:
(SELECT id FROM file_dirs WHERE path = ?2)
for every file inserted (6M+ rows on a full FreeBSD repo update).
Replace this with an in-memory pkghash built from a single SELECT after
Phase 1 directory insertion. Phase 2 looks up dir_id from the hashmap
in O(1) average instead of O(log N) per file via SQL.
Also change FILEDIR2 to accept dir_id as a direct integer parameter,
eliminating the subquery entirely.
Net improvement: ~6M subquery executions → 1 bulk SELECT + ~6M hash
lookups (all in-process, no SQL round-trips).
rwhich: remove unnecessary GROUP BY in exact-match query
The UNIQUE(package_id, dir_id, name) constraint guarantees at most
one row per package per file. No grouping is needed for the exact
match query path. The GROUP BY is retained in the glob variant,
where a single GLOB can match multiple distinct files from the same
package.
jobs/conflicts: fix data corruption in conflict_items_insert
conflict_items_insert called vec_push first, then memmove to shift
elements right at the insertion position. This overwrote the just-pushed
element at the end of the vector, duplicating the last existing element
instead.
Fix by growing capacity manually, shifting elements first, then
inserting at the correct position.
pkghash: fix deletion in open-addressing hash table
Use tombstone markers instead of NULLing keys on deletion.
In open-addressing with linear probing, setting key to NULL creates
a hole that breaks probe chains for subsequent lookups.
Track tombstones and trigger rehash when tombstone density exceeds 25%
to guarantee bounded probe sequences.
