Consolidate internal dataset checks
## Problem
Three separate checks decided whether a dataset was internal and they disagreed. `has_internal_path` matched the top-level child exactly, `is_internal_dataset_name` used a substring match and was the only one aware of `.truenas_containers`, and `internal_datasets_filters` used its own substring filters with a trailing slash that missed `ix-applications` itself. As a result user datasets like `tank/ix-apps-old` were hidden and refused, creates under `.truenas_containers` left orphans that could not be deleted, destroying such datasets skipped key cleanup and leaked encryption key rows, container datasets could be destroyed or rolled back through `zfs.resource`/`pool.snapshot`, legacy `ix-applications` could not be deleted, and filesystem permission calls and local replication targets accepted internal datasets.
## Solution
- **Single source of truth**: `has_internal_path` is now the only check. `.truenas_containers` is internal, legacy `ix-applications` is visible at the top level only so users can delete it, and the substring-based checks are removed. ZFS destroy events use the same check, which fixes the key row leak.
- **Containers**: container create, image pull, rename and migration pass `bypass` for their writes. `zfs.resource.rename` gains a private `bypass` field for this, and the image clone goes through the public `zfs.resource.snapshot.clone` so the rootfs still gets mounted.
- **Pool import**: the container dataset mountpoint repair queries the dataset by explicit path since it no longer shows up in the root listing.
- **Snapshot tasks**: existing tasks on now hidden datasets stay editable as long as the dataset is unchanged.
- **Guards**: `filesystem.setperm`, `chown`, `setacl` and `mkdir` refuse paths backed by internal datasets when called over the API, while internal callers such as app ix-volume ACLs keep working. Replication tasks refuse local targets inside internal datasets.
Use an explicit bypass option for filesystem.setacl
## Problem
The internal dataset guard on `filesystem.setacl` was skipped for any internal caller by inferring it from `app is None` via `pass_app`, which made the exemption implicit.
## Solution
Added a private `bypass` field to `FilesystemSetAclOptions` and use it to skip the guard instead. `add_to_acl` passes it for app ix-volume ACLs, and `setacl` no longer needs `pass_app`.
Use an explicit bypass option for filesystem.setacl
## Problem
The internal dataset guard on `filesystem.setacl` was skipped for any internal caller by inferring it from `app is None` via `pass_app`, which made the exemption implicit.
## Solution
Added a private `bypass` field to `FilesystemSetAclOptions` and use it to skip the guard instead. `add_to_acl` passes it for app ix-volume ACLs, and `setacl` no longer needs `pass_app`.
Trim license delegate unit tests to the essentials
This commit makes some changes where the license rework tests are cut down to the few that guard severe regressions (missing delegates, stopped services or non-autostart VMs being started, TNC reverting a newer license, DS auth lockout) and drops the mock-heavy HA, propagation and per-feature edge case tests.
Simplify KMIP key pull on license revoke
This commit makes some changes where pulling KMIP keys back clears kmip_uid as soon as the key is saved locally again, with destroying the server copy staying best-effort. The escrowed key count is dropped in favour of reporting a pending revoke whenever keys are still waiting to be pulled back, and sync_keys goes back to just starting the sync jobs.
The KMIP license delegate now starts the ZFS and SED sync jobs together and waits on both, so a failing ZFS sync no longer stops the SED keys from being pulled back and every failure is reported in a single error.
Simplify KMIP key pull on license revoke
This commit makes some changes where pulling KMIP keys back clears kmip_uid as soon as the key is saved locally again, with destroying the server copy staying best-effort. The escrowed key count is dropped in favour of reporting a pending revoke whenever keys are still waiting to be pulled back, and sync_keys goes back to just starting the sync jobs.
Stop working around license daemon errors
This commit makes some changes where license entitlement snapshots, the unlicensed features alert and the KMIP key sync no longer special-case a license daemon that cannot answer. A down daemon now reads as no license everywhere except the upload path, which still refuses because it cannot validate the license without the daemon; this means a KMIP sync during a daemon outage pulls keys back locally until the next sync after it recovers.
Rework how license changes are applied
This PR adds changes to replace the license reconcile delegates and the post_license_update hook with a single apply path driven by the entitlement matrix. Every install path (upload, TNC heartbeat, HA peer propagation, legacy sync) snapshots entitlements before and after the change, diffs them and runs only the delegates of the features that actually flipped, so unaffected services are no longer reloaded/restarted on every license change.
Delegates are defined next to their plugins on top of a small base (converge, workload and custom kinds) which owns the service control flags, running/active node checks and failure reporting. Revoking a feature now revokes running functionality as well: VMs and containers are shut down (autostart ones come back on re-grant), docker is stopped while keeping its pool, webshare is stopped and masked, NFS snapshot exposure and DS auth are gated at render time (DS auth is left alone if no usable local admin exists), FC/RDMA targets are converged and KMIP keys are pulled back locally. STIG/FIPS, dedup, NVMe-oF SPDK and network FEC are alert-only. A single alert lists features which are not licensed but still in use, including revokes which did not complete.
On HA, uploads are refused on the standby, during failover and when they would drop HA on a configured pair, and the newer license is pushed to the peer when it reconnects. The pending changes are journalled in /run so a middlewared restart between install and apply does not lose them.
Address review findings in license apply rework
This commit fixes the issues found while reviewing the license apply rework: KMIP now resolves its push/pull direction through the entitlement check so a daemon blip never pulls keys and the integration tests exercise the real path, the KMIP tests expect escrow references to be kept when the server is unreachable, HA delegate steps are isolated from each other, the remote client cannot be started twice on a first HA license, and the apply/journal replay paths no longer race the upload lock.
An admin's explicit upload on the active node is now authoritative. It is propagated through a private endpoint which skips the stale/test license refusals on the standby and a marker keeps the reconnect push from reverting it until the peer has accepted it. TNC and reconnect deliveries keep those refusals.
Simplify license delegate active check and usage alert
This commit makes some changes where license delegates decided whether the node is the active controller by reading failover.status.get_local and the VIP rows themselves; they now use the existing failover.is_single_master_node check, same as the TNC heartbeat.
It also turns the unlicensed features in use alert from a 10 minute polling source into a one-shot alert that is evaluated at the end of truenas.license.apply, once at system.ready on the active node and when a node becomes master, since gates already block new use of unlicensed features so the list can only shrink between license changes. A license daemon error still leaves the existing alert untouched.
VM and container updates are gated by the license again without carving out autostart-only edits, since an autostart instance coming back on a re-grant is what autostart means and the admin can change it once licensed.
Stopping active VMs and containers is simplified as well. The libvirt reachability probe, the resume-before-stop of suspended instances and the extra private endpoints are gone; a suspended VM is powered off and a suspended container killed, shutdown and failover call stop_active directly, and the pool import license guard for containers is dropped since container.start already refuses when unlicensed.
Add middleware support for LIO ALUA HA
Wire up the middleware side of LIO ALUA high-availability: load
lio_ha.ko with per-node addresses on service start, manage ALUA
state across failover events, clean up STANDBY configfs on pool
export, and add pre-flight validation that targets have static
initiator ACLs before ALUA can be enabled.
For each target, create a portal-less phantom TPG carrying the peer
node's controller group so that a single RTPG response from any
connected port lists both ALUA groups. Write tpgt_N/rtpi explicitly
before enable so that relative target port IDs in RTPG match the
tag formula (portal.tag on Node A, portal.tag + 32000 on Node B)
rather than being auto-assigned sequentially by the kernel.
ALUA group states are driven by role and ha_state:
MASTER + synced local=OPTIMIZED remote=NONOPTIMIZED
MASTER + connected local=OPTIMIZED remote=TRANSITIONING
[4 lines not shown]
Add context_params to iscsi_scsi_connect()
Pass an optional context_params dict through to python-scsi's
init_device(), so a test can set pre-login iscsi.Context options such
as an explicit ISID. context_params_supported() lets callers skip when
the installed python-scsi predates it.
Also ruff.
NAS-144543 / 28.0.0-BETA.1 / Fix flaky scrub alert (by themylogin) (#20027)
Sometimes scrub finished before even have the time to see the alert. We
need to listen for `alert.query` events to see when it appears.
Original PR: https://github.com/truenas/middleware/pull/20026
Co-authored-by: themylogin <themylogin at gmail.com>
NAS-144543 / 27.0.0 / Fix flaky scrub alert (#20026)
Sometimes scrub finished before even have the time to see the alert. We
need to listen for `alert.query` events to see when it appears.
NAS-144542 / 27.0.0 / Skip 25.04 filterable method tests (#20025)
They never worked. The only reason this doesn't fail in 28 is because
we've removed 25.04 from 28.
Co-authored-by: Claude Opus 5 (1M context) <noreply at anthropic.com>
Consolidate internal dataset checks
## Problem
Three separate checks decided whether a dataset was internal and they disagreed. `has_internal_path` matched the top-level child exactly, `is_internal_dataset_name` used a substring match and was the only one aware of `.truenas_containers`, and `internal_datasets_filters` used its own substring filters with a trailing slash that missed `ix-applications` itself. As a result user datasets like `tank/ix-apps-old` were hidden and refused, creates under `.truenas_containers` left orphans that could not be deleted, destroying such datasets skipped key cleanup and leaked encryption key rows, container datasets could be destroyed or rolled back through `zfs.resource`/`pool.snapshot`, legacy `ix-applications` could not be deleted, and filesystem permission calls and local replication targets accepted internal datasets.
## Solution
- **Single source of truth**: `has_internal_path` is now the only check. `.truenas_containers` is internal, legacy `ix-applications` is visible at the top level only so users can delete it, and the substring-based checks are removed. ZFS destroy events use the same check, which fixes the key row leak.
- **Containers**: container create, image pull, rename and migration pass `bypass` for their writes. `zfs.resource.rename` gains a private `bypass` field for this, and the image clone goes through the public `zfs.resource.snapshot.clone` so the rootfs still gets mounted.
- **Pool import**: the container dataset mountpoint repair queries the dataset by explicit path since it no longer shows up in the root listing.
- **Snapshot tasks**: existing tasks on now hidden datasets stay editable as long as the dataset is unchanged.
- **Guards**: `filesystem.setperm`, `chown`, `setacl` and `mkdir` refuse paths backed by internal datasets when called over the API, while internal callers such as app ix-volume ACLs keep working. Replication tasks refuse local targets inside internal datasets.
NAS-144559 / 28.0.0-BETA.1 / Fix KMIP ZFS key sync with nested encryption roots (#20028)
## Problem
`get_encrypted_datasets` queried `zfs.resource` with
`get_children=True`, which rejects overlapping paths. Nested
key-encrypted roots (e.g. a key-encrypted pool root plus a child created
with its own key) are a normal layout, so every full push and pull of
ZFS keys failed with a validation error, leaving keys stuck on the KMIP
server and breaking server changes. Separately, push and pull pruned the
in-memory key cache down to the filtered database rows instead of the
datasets that exist, so the scoped push that runs whenever a
key-encrypted dataset is created or has its key changed evicted every
other KMIP-managed key from memory.
## Solution
Check existence by querying the exact paths without `get_children`, and
prune the cache only of keys whose dataset no longer exists in ZFS.
Pruning pops entries in place so concurrent sync jobs don't drop each
other's keys. The KMIP push/pull integration test now covers a nested
encryption root and checks the cache survives a scoped push.
Fix KMIP ZFS key sync with nested encryption roots
## Problem
`get_encrypted_datasets` queried `zfs.resource` with `get_children=True`, which rejects overlapping paths. Nested key-encrypted roots (e.g. a key-encrypted pool root plus a child created with its own key) are a normal layout, so every full push and pull of ZFS keys failed with a validation error, leaving keys stuck on the KMIP server and breaking server changes. Separately, push and pull pruned the in-memory key cache down to the filtered database rows instead of the datasets that exist, so the scoped push that runs whenever a key-encrypted dataset is created or has its key changed evicted every other KMIP-managed key from memory.
## Solution
Check existence by querying the exact paths without `get_children`, and prune the cache only of keys whose dataset no longer exists in ZFS. Pruning pops entries in place so concurrent sync jobs don't drop each other's keys. The KMIP push/pull integration test now covers a nested encryption root and checks the cache survives a scoped push.
NAS-144430 / 28.0.0-BETA.1 / Standardize zfs.resource.snapshot on the zfs.resource design (#20017)
## Problem
`zfs.resource.snapshot` only partly followed the design `zfs.resource`
settled on. It still exposed `query` instead of `list`, raised on
missing paths, emitted no events, took clone properties as an untyped
dict passed straight to libzfs (so user properties failed and
internal-only properties like `mountpoint` were reachable through
clone), had no `force` acknowledgement on rename, keyed most validation
errors on the bare method name, and had several wrong descriptions.
Separately, the ZFS history hook ran the whole dataset-delete path for
every snapshot destroy, sending bogus `pool.dataset.query` REMOVED
events and, on HA, syncing encryption keys to the standby once per
destroyed snapshot. `pool.snapshot.update` silently did nothing.
## Solution
- **list**: added `zfs.resource.snapshot.list`; `query` stays as a
hidden alias for pre-v28 clients and `query_impl` is now `list_impl`.
list, query and count skip a named path that does not exist; a path
[31 lines not shown]
