Gate S3 auditing on appliance hardware
This commit adds changes to decide whether the S3 service audits requests from the hardware class rather than from the license, so it matches the gate the kernel audit handler already uses -- both halves of the audit trail land in the same database, so both have to answer to the same thing. The check is renamed audit_supported since it no longer reads a license, and the validation messages and API field descriptions say appliance hardware instead of Enterprise license.
Log full git output to its own log file
## Problem
`utils/git.py` shortened git's stderr to 50 characters before putting it into the `CallError`, and `textwrap.shorten` collapses newlines before it truncates. `git clone` always opens stderr with `Cloning into '/mnt/.ix-apps/truenas_catalog'...`, which is 47 characters by itself, so every clone failure produced the byte-identical message and the `fatal:` line saying what actually went wrong was discarded every time. Bad credentials, DNS, TLS, a proxy and a missing branch were indistinguishable in the UI, in the logs and in a debug bundle, and git's stderr was preserved nowhere else.
## Solution
- **A dedicated `git` log file.** The tail of git's output goes to `/var/log/git.log` rather than back into `middlewared.log`, which is what the original shortening was trying to keep clean. The syslog-ng filter and destination generate themselves from `ALL_LOG_FILES`, so `logger.py` is the only place that needed touching, plus a line in the existing logrotate stanza.
- **The raised message names the cause.** The last `fatal:` line is preferred, then the last `error:` line, then whatever git printed last. Position alone is not reliable because `reset --hard` on a bad ref prints usage boilerplate after the `fatal:` line. The message stays a single line because it is stored as the job error and rendered into the catalog sync alert, and it points at the log file for the rest.
- **Stopped discarding two other git errors.** A failed checkout or pull was caught, silently dropped and turned into a re-clone, so a repository that stopped being usable left no record of why. The clone error was also re-raised behind a second copy of the prefix `clone_repository` had already produced, which pushed the real error further right in every alert and dropped the inner errno.
NAS-143369 / 27.0.0-BETA.1 / Stop nginx closing idle VM SPICE console channels
The VM display proxy is the last websocket location in this file still on
nginx's 60s proxy_read_timeout default. NAS-140066 (5a9a2a522a) fixed the same
class of bug for /api and /websocket and noted /websocket/shell already had
its own; the VM display proxy was not in that audit.
A SPICE session opens one websocket per channel. Measured on 25.10.5 with a
live console and the WebSocket constructor instrumented: the server pings main
and display about every 15s, but inputs and cursor only every 300s, so nginx
closes those two at 61s with code 1006 while the display channel stays up. The
console keeps painting and silently ignores the keyboard and mouse. The same
console straight to websockify keeps all four channels.
3600 covers the measured 300s worst case twelve times over and matches what
NAS-140066 chose for the other two locations.
(cherry picked from commit 364f77c76e6e25d85219f0fa331fc40972974893)
NAS-143369 / 27.0.0-BETA.1 / Stop nginx closing idle VM SPICE console channels (#19646)
[NAS-143369](https://ixsystems.atlassian.net/browse/NAS-143369)
A VM SPICE console that nobody touches for a minute stops responding to
the keyboard and mouse. The picture keeps updating, so the console looks
alive.
The VM display proxy is the last websocket location in `nginx.conf.mako`
still on nginx's 60s `proxy_read_timeout` default.
[NAS-140066](https://github.com/truenas/middleware/pull/18412)
(5a9a2a522a) fixed exactly this for `/api` and `/websocket`, and its
description notes `/websocket/shell` already had its own — the VM
display proxy was not in that audit.
### Why it presents as "input is dead" rather than "console
disconnected"
A SPICE session opens one websocket per channel (`spice-html5` builds
[60 lines not shown]
Let the S3 protocol create and delete buckets
An S3 client creating a bucket sends a name and nothing about placement,
so something has to decide where the dataset goes. That decision is this
service's: the S3 daemon holds no ZFS layout, and a daemon that composed
a dataset name would be holding half of a policy whose other half it
cannot see.
The S3 service gains managed_root_dataset and sharing.s3.create's
dataset becomes optional. Omitted, the dataset is created under that
root and named after the bucket. Empty is the default and refuses a
bucket that named no dataset, which is what an appliance whose
administrator has not chosen a location should do rather than picking
one. The root must exist and is never created: a service that made a
dataset out of a string typed into its configuration would make one out
of a typo just as readily. It is checked at s3.update, because a create
is the wrong place to learn the service is misconfigured - the client
that provoked it can do nothing about it.
[41 lines not shown]
Let the S3 protocol create and delete buckets
An S3 client creating a bucket sends a name and nothing about placement,
so something has to decide where the dataset goes. That decision is this
service's: the S3 daemon holds no ZFS layout, and a daemon that composed
a dataset name would be holding half of a policy whose other half it
cannot see.
The S3 service gains managed_root_dataset and sharing.s3.create's
dataset becomes optional. Omitted, the dataset is created under that
root and named after the bucket. Empty is the default and refuses a
bucket that named no dataset, which is what an appliance whose
administrator has not chosen a location should do rather than picking
one. The root must exist and is never created: a service that made a
dataset out of a string typed into its configuration would make one out
of a typo just as readily. It is checked at s3.update, because a create
is the wrong place to learn the service is misconfigured - the client
that provoked it can do nothing about it.
[41 lines not shown]
NAS-143293 / 26.0.0 / Fail interface.commit when a bridge member cannot be added, and reject members that host a VM or container NIC in MACVLAN mode (#19659)
A NIC that hosts macvtap or macvlan ports, which is what a VM or
container NIC device attached to it directly (MACVLAN mode) creates,
cannot also be a bridge port: `br_add_if()` fails at
`netdev_rx_handler_register()` with EBUSY because the macvlan port
already owns the device's rx_handler. `configure_bridges_impl` caught
that, logged it and carried on, so `interface.sync()` reported success.
The user then either lost the network until the checkin timer rolled the
change back, because the address had already moved off the NIC, or, if
the NIC kept its address, ended up with a bridge persisted in the
database that the kernel has no member for. No error was shown either
way.
`configure_bridges_impl` now returns the failures, naming the members
that did not end up enslaved since the netlink error alone does not say
which, `interface.sync()` returns them, and `interface.commit` raises a
`CallError` and rolls back, so the user gets an immediate error instead
of a silent blackout. The boot, HA and rollback callers of
[14 lines not shown]
NAS-141885 / 26.0.0-RC.1 / Depend on websockify explicitly for SPICE display (by Qubad786) (#19663)
## Problem
VMs with a SPICE display device shell out to the `websockify` binary at
runtime (truenas_pylibvirt's display device runs `websockify --web
/usr/share/spice-html5/ ...` to proxy the console to the web UI). We
never declared that dependency though — websockify only got installed as
a transitive `Recommends:` of `spice-html5`. Once truenas_build disabled
`install_recommends` by default, the `truenas` package started
installing with `--no-install-recommends`, so websockify silently
stopped being pulled in and SPICE consoles broke.
## Solution
Add `websockify` to middlewared's `Depends`. It's a genuine hard runtime
dependency of the SPICE path, so declaring it explicitly makes it come
in regardless of the build's recommends setting and keeps us correct
even if spice-html5's packaging changes.
Original PR: https://github.com/truenas/middleware/pull/19365
Co-authored-by: Qubad786 <mrehanlm93 at gmail.com>
NAS-141885 / 26.0.0 / Depend on websockify explicitly for SPICE display (by Qubad786) (#19662)
## Problem
VMs with a SPICE display device shell out to the `websockify` binary at
runtime (truenas_pylibvirt's display device runs `websockify --web
/usr/share/spice-html5/ ...` to proxy the console to the web UI). We
never declared that dependency though — websockify only got installed as
a transitive `Recommends:` of `spice-html5`. Once truenas_build disabled
`install_recommends` by default, the `truenas` package started
installing with `--no-install-recommends`, so websockify silently
stopped being pulled in and SPICE consoles broke.
## Solution
Add `websockify` to middlewared's `Depends`. It's a genuine hard runtime
dependency of the SPICE path, so declaring it explicitly makes it come
in regardless of the build's recommends setting and keeps us correct
even if spice-html5's packaging changes.
Original PR: https://github.com/truenas/middleware/pull/19365
Co-authored-by: Qubad786 <mrehanlm93 at gmail.com>
NAS-143453 / 27.0.0-BETA.1 / Give S3 object ownership its own key (by anodos325) (#19661)
Initially, `permissions_model` determined whether the permissions should
attempt to be compatible with other processes and the S3 ownership model
(AWS-style per-bucket Object Ownership). This commit moves the object
ownership question into its own API field for bckets. It becomes
object_ownership, defaulting to BUCKET_OWNER_ENFORCED, and the model
keeps S3 and MULTIPROTOCOL alone. A MULTIPROTOCOL row folds to
OBJECT_WRITER, as the S3 service does with it.
S3_BUCKET_OWNER_ENFORCED stays spellable and stores as the pair it
always meant. It is undocumented and goes with its last consumer.
test_boto3_roundtrip no longer expects a POSIX refusal on an S3 bucket:
that model ignores the filesystem permissions entirely.
Original PR: https://github.com/truenas/middleware/pull/19658
Co-authored-by: Andrew Walker <andrew.walker at truenas.com>
NAS-143453 / 27.0.0-BETA.1 / Give S3 object ownership its own key
permissions_model carried two questions; the second was AWS's
per-bucket S3 Object Ownership setting. It becomes object_ownership,
defaulting to BUCKET_OWNER_ENFORCED, and the model keeps S3 and
MULTIPROTOCOL alone. A MULTIPROTOCOL row folds to OBJECT_WRITER, as
the S3 service does with it.
S3_BUCKET_OWNER_ENFORCED stays spellable and stores as the pair it
always meant. It is undocumented and goes with its last consumer.
test_boto3_roundtrip no longer expects a POSIX refusal on an S3
bucket: that model ignores the filesystem permissions entirely.
api/v27_0_0/s3.py takes the same change as api/v26_0_0/s3.py, the
two being identical here and 26.0 shipping the field.
(cherry picked from commit eb002f0159c3611c0f0854edb231c6d03392b431)
NAS-141885 / 27.0.0-BETA.1 / Depend on websockify explicitly for SPICE display (#19365)
## Problem
VMs with a SPICE display device shell out to the `websockify` binary at
runtime (truenas_pylibvirt's display device runs `websockify --web
/usr/share/spice-html5/ ...` to proxy the console to the web UI). We
never declared that dependency though — websockify only got installed as
a transitive `Recommends:` of `spice-html5`. Once truenas_build disabled
`install_recommends` by default, the `truenas` package started
installing with `--no-install-recommends`, so websockify silently
stopped being pulled in and SPICE consoles broke.
## Solution
Add `websockify` to middlewared's `Depends`. It's a genuine hard runtime
dependency of the SPICE path, so declaring it explicitly makes it come
in regardless of the build's recommends setting and keeps us correct
even if spice-html5's packaging changes.
(cherry picked from commit 44dd41a18e7db2c8466c6fbd72e75be226a9491e)
NAS-141885 / 27.0.0-BETA.1 / Depend on websockify explicitly for SPICE display (#19365)
## Problem
VMs with a SPICE display device shell out to the `websockify` binary at
runtime (truenas_pylibvirt's display device runs `websockify --web
/usr/share/spice-html5/ ...` to proxy the console to the web UI). We
never declared that dependency though — websockify only got installed as
a transitive `Recommends:` of `spice-html5`. Once truenas_build disabled
`install_recommends` by default, the `truenas` package started
installing with `--no-install-recommends`, so websockify silently
stopped being pulled in and SPICE consoles broke.
## Solution
Add `websockify` to middlewared's `Depends`. It's a genuine hard runtime
dependency of the SPICE path, so declaring it explicitly makes it come
in regardless of the build's recommends setting and keeps us correct
even if spice-html5's packaging changes.
(cherry picked from commit 44dd41a18e7db2c8466c6fbd72e75be226a9491e)
Rework snapshot rollback with batched destroys
## Problem
Rolling back past older snapshots destroyed the newer ones one ioctl at a time with no real pre-flight, so a hold or clone discovered midway left some snapshots already gone and, with `recursive_rollback`, a blocked child was only discovered after the parent had already been rolled back - a partially rolled-back tree. Most kernel errors (ESRCH, EBUSY, EDQUOT, ENOSPC) were flattened into `ValidationError(EINVAL)` with a raw strerror, a rollback that had already committed was still reported as a failure when the follow-up zpool-history write failed, and a thick zvol rolled back across a volsize change silently lost its refreservation.
## Solution
- **Enumerate once, up front, for the whole tree.** Newer snapshots are collected for every affected dataset before anything is touched. Without `recursive`, the rollback is refused immediately with every conflicting snapshot named and nothing destroyed; with it, hold/clone blockers are reported before any destroy, and a missing child snapshot fails the whole tree before the parent moves.
- **Batch the destroys.** All newer snapshots go in a single all-or-nothing ioctl per dataset. The kernel checks every snapshot before destroying any, so a blocker - including the long holds from an in-flight send or a `.zfs/snapshot` automount that no pre-flight can see - destroys nothing and is named from the kernel's own error list. A destroy interrupted mid-sync (empty kernel error list) re-enumerates and reports honestly that an unknown number are already gone.
- **Honest error model.** Rollback failures get errno-specific messages; blockers and operational failures surface as `CallError` (EBUSY and friends) while input problems stay `ValidationError`; a committed rollback is never reported as a failure just because the zpool-history write raised afterwards; and any failure partway through a recursive rollback names the datasets that already rolled back, since that cannot be undone.
- **Restore a thick zvol's refreservation** after a rollback that changed the volsize, matching `zfs rollback`. Volumes with a synthetic (larger) refreservation are deliberately left alone, also matching `zfs rollback`.
- **Clones are always unmounted before being destroyed** - `force` now only selects how forcefully - and destroyed via `destroy_resource()`, fixing a path that previously crashed on a method that did not exist.
- **Known limitation: bookmarks are not managed.** A bookmark newer than the target is invisible to the pre-flight and fails the rollback with EEXIST - after the newer snapshots were already destroyed when `recursive` was passed. The error says so and names the manual remedy. TrueNAS itself creates no bookmarks; this only affects externally-created ones.
NAS-143057 / 27.0.0-BETA.1 / Redact uploaded license from the audit trail (by sonicaj) (#19606)
This commit fixes an issue where `truenas.license.upload` recorded its
license argument verbatim in the audit trail, because the field was
never declared `Secret[...]`. Uploading needs FULL_ADMIN but audit
records come back with SYSTEM_AUDIT_READ, so a read-only admin could
pull out the blob a full admin installed, and the same record went to
remote syslog and into debug bundles.
Marking the field `Secret[LongNonEmptyString]` means `check_annotations`
stops accepting the old `license_: str` signature, and the
`str(license_)` that went with it turned out to be wrong already: it
stringified the `LongStringWrapper` and wrote `LongStringWrapper(<pem>)`
into the license file. The daemon's PEM scan skips the leading garbage
so nothing ever failed, but the on-disk copy never matched what was
uploaded, which defeats the TNC heartbeat's raw_license comparison and
makes every HA send_license add another wrapper layer. Unwrapping both
boxes writes the PEM back byte for byte.
[4 lines not shown]
Redact uploaded license from the audit trail
This commit fixes an issue where `truenas.license.upload` recorded its license argument verbatim in the audit trail, because the field was never declared `Secret[...]`. Uploading needs FULL_ADMIN but audit records come back with SYSTEM_AUDIT_READ, so a read-only admin could pull out the blob a full admin installed, and the same record went to remote syslog and into debug bundles.
Marking the field `Secret[LongNonEmptyString]` means `check_annotations` stops accepting the old `license_: str` signature, and the `str(license_)` that went with it turned out to be wrong already: it stringified the `LongStringWrapper` and wrote `LongStringWrapper(<pem>)` into the license file. The daemon's PEM scan skips the leading garbage so nothing ever failed, but the on-disk copy never matched what was uploaded, which defeats the TNC heartbeat's raw_license comparison and makes every HA send_license add another wrapper layer. Unwrapping both boxes writes the PEM back byte for byte.
(cherry picked from commit a64faaf2ff66648e0712390439562a6c3f35be48)
Add a regression test for license redaction in the audit log
This commit adds changes to assert that every audit entry for `truenas.license.upload` records the redaction placeholder in place of the license, so that dropping `Secret[...]` from the field cannot quietly reopen the leak while CI stays green.
The test uploads nothing and is read-only. Whatever installed the license -- CI, TrueNAS Connect or an operator -- already went through the audited API and left the entry behind, so the entries themselves are what get inspected, and it skips when nothing has called the method. It asserts the shape of the placeholder rather than its exact text, which keeps it independent of how long the placeholder is and holds for a legacy base64 blob as much as for a v2 PEM: a legacy blob carries no PEM header, so matching on one would let a leaked legacy license through.
(cherry picked from commit 82d85cdd71c08d65b1101498b3de9e38a652e6b9)
NAS-143111 / 27.0.0-BETA.1 / Write a hardware entitlement record on unlicensed iX appliances (by sonicaj) (#19657)
This PR adds changes to give an iX appliance that shipped without a
license a minimal legacy license record during upgrade, written by a new
script that runs chrooted in the new boot environment. The record is
stamped with a marker in customer_key, and parse_legacy_license grants
it exactly what bare appliance hardware already gives rather than
everything a legacy license implies. SED is the motivating case, since
its vector moves to hardware plus key later and these machines would
otherwise lose it.
LicenseInfo grows an origin field so we can tell a record we wrote from
one an issuer signed. Entitlements never read it, since a system
generated record has to resolve exactly as an issued one does; it is
there for the alert source, the unlicensed to licensed hook, TNC, and
support tickets, which all mean "iX issued this".
Original PR: https://github.com/truenas/middleware/pull/19643
Co-authored-by: Waqar Ahmed <waqarahmedjoyia at live.com>
