NAS-142154 / 27.1.0 / Reject new overlaps between Active Directory idmap ranges (by anodos325) (#20162)
The directory services refactor (368cc8cf63) removed the idmap plugin's
range validation and kept only a check between trusted domains. A
trusted domain could then share IDs with the primary domain or BUILTIN.
- Check BUILTIN, the primary domain and enabled trusted domains in
directoryservices.update.
- Leave overlaps already in the saved configuration alone, so that
existing setups can still be disabled, enabled and saved.
Original PR: https://github.com/truenas/middleware/pull/20148
---------
Co-authored-by: Andrew Walker <andrew.walker at truenas.com>
Pin dedup off on internal datasets
## Problem
Dedup can be enabled on the pool root dataset, either at pool creation or later, and the internal `.system` and `.truenas_containers` trees inherit it, so system dataset and container writes go through the dedup table. ix-apps already pins `dedup=off`, but its drift check compares raw values only, so an ix-apps tree created before that pin and inheriting `off` is never pinned locally and would follow the root once it is switched on.
## Solution
Rather than refusing dedup on the root, which is a supported pool.create workflow, the internal trees now carry a local `dedup=off`:
- **System dataset**: `.system` is created with `dedup=off` (children inherit it), and `setup_datasets` re-pins dedup whenever its source is not local, so existing systems get pinned on the next boot/import/failover reconcile.
- **Containers**: `.truenas_containers` is created with `dedup=off`, and an existing tree is re-pinned when datasets are ensured (create/pull/migrate) and on container start, which also covers boot autostart and failover.
- **ix-apps**: the drift check also fires when dedup is not set locally.
Blocks that were already written deduplicated stay in the dedup table until rewritten; only new writes are affected.
Harden container image pull
## Problem
Pulling a container image leaked raw `requests` errors, had no timeout on the tarball download so a stalled stream hung forever, and never checked the published sha256. Aborting `container.create` was ignored by the pull it was waiting on, and two concurrent pulls of the same uncached image could destroy each other's in-progress dataset through the orphan cleanup.
## Solution
- **Error wrapping**: registry index failures are raised as a readable `CallError` instead of a raw `requests` exception.
- **Timeout**: the tarball download uses `INTERNET_TIMEOUT`, so a stall fails the pull and the image dataset is destroyed.
- **Checksum**: the download is hashed as it streams and compared against the registry's sha256 before anything is extracted.
- **Registry lookup first**: the image/version lookup happens before `ensure_datasets`, the orphan destroy and the dataset create, so an unknown image or unreachable registry never touches a dataset. Cached images still return early without network access.
- **Abort**: the pull checks for abort in the download and extract loops. `container.create` aborts its pull job when it is cancelled, and raises `JobCancelledException` if the pull was aborted on its own.
- **Per-image lock**: `container.image.pull` holds a lock keyed on pool, name and version with `lock_queue_size=None`, so each create waits for its own pull job and a waiter after a successful pull hits the cached snapshot.
Fail graceful VM and container stop when the guest keeps running
## Problem
A graceful `vm.stop` / `container.stop` (no `force`, no `force_after_timeout`, the API default) waited `shutdown_timeout` and then reported SUCCESS even when the guest ignored the shutdown request and was still running. The UI showed the guest as stopped and a graceful container restart then failed on start with "already started". A suspended VM was also left suspended with a successful job, even with `force_after_timeout`, because the post-stop check only looked for `RUNNING`.
## Solution
After the shutdown request, both stop paths check the state again. If the guest has not stopped, it is powered off when `force_after_timeout` is set; otherwise the job fails with an ETIMEDOUT `CallError` saying it did not stop within the timeout. A suspended VM is resumed before the shutdown request so the guest can act on it, and library errors (e.g. stopping a guest that is not running) are raised as `CallError` like start already does. The docstrings describe the new behaviour and the container stop test now expects the graceful stop to fail.
Validate container configuration values when they are saved
## Problem
Several container settings passed create and update but only failed at start, often as a raw IndexError or ValueError: an unparseable or empty init, a cpuset with whitespace or CPUs that are not online, and a memory limit above host RAM. A missing initdir made start report success for a container that exited immediately.
## Solution
- **Save time**: init must parse to at least one token; cpuset must be well formed and online, and memory within host RAM. The cpuset and memory checks only run when the value changes, so a host that has since lost CPUs or memory does not block unrelated edits.
- **Start time**: initdir must resolve to a directory inside the container root.
- **Migration**: an incus CPU count, or a CPU set or memory limit this host can't satisfy, is dropped with a job log line instead of failing the container.
- An invalid pinned VM cpuset is a validation error instead of an unhandled ValueError, and autostart failures are logged with a traceback. Accepting `0-0` relies on the matching truenas_pylibvirt parser change.
Disable STP on the container bridge
## Problem
truenasbr0 was created with pynetif's default of STP on, so every new vnet port spent ~30s in listening/learning before forwarding. DHCP to dnsmasq was dropped during that window, delaying container networking by 30s or more on every start.
## Solution
Create the bridge with STP off. It has no physical members and only carries container NICs, so there is nothing for STP to protect; incus's bridge behaved the same way.
Validate container configuration values when they are saved
## Problem
Several container settings passed create and update but only failed at start, often as a raw IndexError or ValueError: an unparseable or empty init, a cpuset with whitespace or CPUs that are not online, and a memory limit above host RAM. A missing initdir made start report success for a container that exited immediately.
## Solution
- **Save time**: init must parse to at least one token; cpuset must be well formed and online, and memory within host RAM. The cpuset and memory checks only run when the value changes, so a host that has since lost CPUs or memory does not block unrelated edits.
- **Start time**: initdir must resolve to a directory inside the container root.
- **Migration**: an incus CPU count, or a CPU set or memory limit this host can't satisfy, is dropped with a job log line instead of failing the container.
- An invalid pinned VM cpuset is a validation error instead of an unhandled ValueError, and autostart failures are logged with a traceback. Accepting `0-0` relies on the matching truenas_pylibvirt parser change.
Validate container configuration values when they are saved
## Problem
Several container settings passed create and update but only failed at start, often as a raw IndexError or ValueError: an unparseable or empty init, a cpuset with whitespace or CPUs that are not online, and a memory limit above host RAM. A missing initdir made start report success for a container that exited immediately.
## Solution
- **Save time**: init must parse to at least one token; cpuset must be well formed and online, and memory within host RAM. The cpuset and memory checks only run when the value changes, so a host that has since lost CPUs or memory does not block unrelated edits.
- **Start time**: initdir must resolve to a directory inside the container root.
- **Migration**: an incus CPU count, or a CPU set or memory limit this host can't satisfy, is dropped with a job log line instead of failing the container.
- An invalid pinned VM cpuset is a validation error instead of an unhandled ValueError, and autostart failures are logged with a traceback. Accepting `0-0` relies on the matching truenas_pylibvirt parser change.
NAS-141340 / 28.0.0-BETA.1 / Set WebShare no_dataset_traversal when tiering is enabled (#20150)
When ZFS tiering is enabled, the tier contract is authoritative for what
each share exposes, so WebShare must not traverse from a share root into
a child dataset mounted inside it. This is the WebShare counterpart of
`shadow:no_dataset_traversal` in smb.conf. The config contract is
defined in
https://github.com/iXsystems/truenas-file-manager/issues/2232.
Jira: https://ixsystems.atlassian.net/browse/NAS-141340
- `/etc/webshare/config.json` gets a top-level `no_dataset_traversal`
boolean that follows `zfs.tier.config.enabled`.
- `zfs.tier.update` reloads WebShare when `enabled` changes, so new
sessions pick up the change.
- `tests/api2/zfs_tier/test_webshare.py` checks the rendered flag in
both states, and that a WebShare session lists and opens a child dataset
with tiering off, and hides it and refuses `directory.getHandle` with
`dataset_boundary` with tiering on. `WebshareSession` gains
[4 lines not shown]
Do not exempt the AD backend from the idmap overlap check
Samba documents idmap ranges as disjoint. If two domains hold the same
uidNumber, both accounts get the same UID and winbind does not report it.
- Reject new overlaps between domains that use the AD backend. This only
changes the primary domain against a trusted domain, as the API model
already rejects overlaps between trusted domains.
- Saved overlaps are still left alone.
(cherry picked from commit 9ca1fc88853306c9dd83239c076298cfb7c9ff2c)
Reject new overlaps between Active Directory idmap ranges
The directory services refactor (368cc8cf63) removed the idmap plugin's range
validation and kept only a check between trusted domains. A trusted domain
could then share IDs with the primary domain or BUILTIN.
- Check BUILTIN, the primary domain and enabled trusted domains in
directoryservices.update.
- Allow overlap between two domains that use the AD backend, as the idmap
plugin did (a79a0d3c6e) and as testparm does.
- Leave overlaps already in the saved configuration alone, so that existing
setups can still be disabled, enabled and saved.
(cherry picked from commit d7f41b5beb8a7486f3bc783664f89ac595f5fcd8)
NAS-142154 / 28.0.0-BETA.1 / Reject new overlaps between Active Directory idmap ranges (#20148)
The directory services refactor (368cc8cf63) removed the idmap plugin's
range validation and kept only a check between trusted domains. A
trusted domain could then share IDs with the primary domain or BUILTIN.
- Check BUILTIN, the primary domain and enabled trusted domains in
directoryservices.update.
- Leave overlaps already in the saved configuration alone, so that
existing setups can still be disabled, enabled and saved.
Set WebShare no_dataset_traversal when tiering is enabled
When ZFS tiering is enabled, WebShare must not traverse from a share root
into child datasets mounted inside it, the same as Samba does with
`shadow:no_dataset_traversal`. Render a top-level `no_dataset_traversal`
boolean in /etc/webshare/config.json that follows zfs.tier.config.enabled,
and reload WebShare when tiering is toggled so new sessions pick it up.
The zfs_tier integration tests check the rendered flag and that a WebShare
session hides and refuses a child dataset only while tiering is enabled.
Send one terminal `core.get_jobs` event per job
A successful job sent its SUCCESS event twice. `__run_body` sets the
state to SUCCESS and then calls `set_progress(100)`, which sends an
event. The `finally` block in `Job.run` then sends the same event again.
The client usually drops the second event, because `Job.result` removes
the job first. When the caller thread wakes up late, the job callback
runs twice. This made `test_client_job_callback` flaky.
`set_progress` no longer sends an event for a finished job. `Job.run`
sends the only event that reports the final state. Progress still goes
to the jobs that wrap the finished job.
`test_job_events.py` waits on `core.job_wait` before it checks the
events, so it catches a duplicate event on every run. New tests cover
failed, aborted and wrapped jobs.
`test_client_job_callback` loses its `flaky` marker. It no longer
[5 lines not shown]
