Relocate migrated container origins out of legacy .ix-virt
## Problem
Incus containers are ZFS clones of an image snapshot. The incus->container auto-migration relocated each container from `<pool>/.ix-virt/containers/<name>` to `<pool>/.truenas_containers/containers/<name>` with a bare `zfs rename` and did nothing else. A `zfs rename` does not change a clone's `origin`, so a migrated container stayed a clone of a snapshot still living inside `.ix-virt` - and deleting `.ix-virt` recursively destroyed those origin snapshots and cascaded into the dependent migrated clones, silently destroying migrated containers. The migration also left the legacy parents mounted, moved on without a license, gave up on every remaining pool when one failed, and could leave a container renamed into the native tree with no database row pointing at it.
## Solution
Relocate each container's origin image out of `.ix-virt` before renaming the container, so no migrated container depends on anything under `.ix-virt`, and make the surrounding migration and deletion paths recoverable.
- **Shared relocation helper** - `relocate_container_origin` reads a container's live `origin`; if it points at an image under `.ix-virt/images` or `.ix-virt/deleted/images`, it sets `canmount=noauto` on that image dataset and then renames it into the native `.truenas_containers/images/` tree. The rename goes last so it is the single atomic commit point: the image is either wholly still in `.ix-virt` or wholly relocated, and the return value describes reality. Fan-out clones auto-repoint on the rename; an origin outside `.ix-virt` is left alone; a container that is a clone of another container is refused, since the two would arrive entangled.
- **Migration path** - the migration calls the helper immediately before renaming each container and skips any container whose base image cannot be relocated, rather than producing one that looks healthy until `.ix-virt` is deleted.
- **Repair migration** - new `0020_repair_incus_clone_origins` runs the same relocation over existing `container.container` rows for systems that already ran the old migration, and restores the legacy parents' mountpoint those runs left inherited.
- **Leave the legacy tree as found** - mount properties are reverted on any container that is not migrated, and the legacy parents are restored at the end of every run, so nothing is left mounted under `/mnt/<pool>/.ix-virt` with nothing managing it. Both reverts are armed before the properties are touched, since a partial apply has to be reverted too.
- **Migration robustness** - the migration is skipped on HA capable hardware and deferred (leaving the legacy configuration intact) when the system is not licensed for containers; one unusable pool no longer stops the pools after it; and a container whose dataset was renamed but whose row was never created is moved back where the user can see it.
- **Safer deletion** - the libvirt domain is torn down first so the container's runtime mounts are gone before ZFS is touched, the dataset is destroyed next, and the database records are removed only once it is confirmed gone - so a failed destroy never orphans the dataset with no row pointing at it. An already-missing dataset is tolerated so a container whose data was lost to the old cascade can still be removed cleanly.
- **Active-instance guards** - deleting or renaming a container that is not stopped (running or suspended) is refused; delete additionally accepts `force=True`, mirroring the VM delete flow, and is now a job locked per container id. The container status model gains the `SUSPENDED` state it has always been able to report.
Relocate migrated container origins out of legacy .ix-virt
## Problem
Incus containers are ZFS clones of an image snapshot. The incus->container auto-migration relocated each container from `<pool>/.ix-virt/containers/<name>` to `<pool>/.truenas_containers/containers/<name>` with a bare `zfs rename` and did nothing else. A `zfs rename` does not change a clone's `origin`, so a migrated container stayed a clone of a snapshot still living inside `.ix-virt` - and deleting `.ix-virt` recursively destroyed those origin snapshots and cascaded into the dependent migrated clones, silently destroying migrated containers. The migration also left the legacy parents mounted, moved on without a license, gave up on every remaining pool when one failed, and could leave a container renamed into the native tree with no database row pointing at it.
## Solution
Relocate each container's origin image out of `.ix-virt` before renaming the container, so no migrated container depends on anything under `.ix-virt`, and make the surrounding migration and deletion paths recoverable.
- **Shared relocation helper** - `relocate_container_origin` reads a container's live `origin`; if it points at an image under `.ix-virt/images` or `.ix-virt/deleted/images`, it sets `canmount=noauto` on that image dataset and then renames it into the native `.truenas_containers/images/` tree. The rename goes last so it is the single atomic commit point: the image is either wholly still in `.ix-virt` or wholly relocated, and the return value describes reality. Fan-out clones auto-repoint on the rename; an origin outside `.ix-virt` is left alone; a container that is a clone of another container is refused, since the two would arrive entangled.
- **Migration path** - the migration calls the helper immediately before renaming each container and skips any container whose base image cannot be relocated, rather than producing one that looks healthy until `.ix-virt` is deleted.
- **Repair migration** - new `0020_repair_incus_clone_origins` runs the same relocation over existing `container.container` rows for systems that already ran the old migration, and restores the legacy parents' mountpoint those runs left inherited.
- **Leave the legacy tree as found** - mount properties are reverted on any container that is not migrated, and the legacy parents are restored at the end of every run, so nothing is left mounted under `/mnt/<pool>/.ix-virt` with nothing managing it. Both reverts are armed before the properties are touched, since a partial apply has to be reverted too.
- **Migration robustness** - the migration is skipped on HA capable hardware and deferred (leaving the legacy configuration intact) when the system is not licensed for containers; one unusable pool no longer stops the pools after it; and a container whose dataset was renamed but whose row was never created is moved back where the user can see it.
- **Safer deletion** - the libvirt domain is torn down first so the container's runtime mounts are gone before ZFS is touched, the dataset is destroyed next, and the database records are removed only once it is confirmed gone - so a failed destroy never orphans the dataset with no row pointing at it. An already-missing dataset is tolerated so a container whose data was lost to the old cascade can still be removed cleanly.
- **Active-instance guards** - deleting or renaming a container that is not stopped (running or suspended) is refused; delete additionally accepts `force=True`, mirroring the VM delete flow, and is now a job locked per container id. The container status model gains the `SUSPENDED` state it has always been able to report.
Route HA licensing through the entitlement engine
This commit adds changes to make is_licensed_for_ha answer out of POLICY rather than comparing the license type itself, so there is one place to change the HA rule. Behaviour is unchanged since LicenseTypeRule evaluates the same predicate over the same get_license, and it stays a plain function with no middleware dependency because ha_panic imports it directly and runs when middlewared may already be gone.
This also drops is_ha_capable from EntitlementFacts as no rule ever read it, which keeps the HA path clear of platform detection and takes a failover.hardware round-trip off every entitlement check.
Apply the VM system clock setting to the generated domain
## Problem
`pylibvirt_vm()` builds the domain configuration by unpacking `VMEntry.model_dump()` into a plain dataclass, converting `bootloader` and `cpu_mode` into their enums but not `time`. The API model types `time` as a string literal, so the configuration held `'LOCAL'`/`'UTC'` instead of a `Time` member, and the clock offset is decided by comparing that field against `Time.LOCAL`. That comparison was always false, so every VM was defined with `<clock offset="utc">` regardless of what the user picked, and Windows guests — which expect a localtime RTC — ran off by the host's UTC offset. Containers were unaffected because their equivalent helper already does the conversion.
## Solution
Convert `time` alongside the other two enums. Every caller reaches this through `VMEntry`, whose `Literal['LOCAL', 'UTC']` guarantees a valid value, so the conversion cannot raise. Added a unit test covering both offsets with and without Hyper-V enlightenments, which restores coverage that was dropped when XML generation moved out to truenas_pylibvirt.
Remove dead writers of the autotune setting
This commit adds changes to drop the two code paths that flipped the autotune advanced setting on enterprise systems, one on the post license update hook and one on firstboot. Nothing has read that column since the linux port removed its consumer, and the tuner itself runs from the installer without consulting either the column or the license, so both writers were setting a value that never did anything. The column, its API field and the legacy license key injection are deliberately left in place.
Format license utils package
This commit fixes an issue where the files added when license retrieval moved into a shared utils package were never run through ruff format, so the formatter check fails on them. Reformatting only reflows multi-line collection literals and parametrize arguments to match line length; no logic changes.
Add missing coverage for NFS snapshot entitlement
This commit adds changes to register the bespoke NFS snapshot denial message for all three deny reasons rather than just the two currently reachable, so editing a matrix cell later cannot silently drop the wording back to the generic template.
It also covers the feature against the live policy instead of only the matrix fixture, and reworks the unentitled integration fixture to mock the license rather than the entitlement check, so the negative path exercises the real engine instead of asserting a string the test itself handed in.
Gate SMB fast path and Veeam shares through the entitlement engine
This commit adds changes to gate the SMB ZFS fastpath parameters and Veeam repository shares through the entitlement engine instead of a bare is_enterprise read. Both stay separate flags with their own matrix vectors and remain in the enterprise-only legacy injection bucket, so existing licensed systems keep both features and only unlicensed HA-capable boxes lose them.
Flip NVMe-oF SPDK onto its matrix vector
This commit adds changes to move NVMEOF_SPDK off its transitional legacy rule and onto its matrix vector, which drops the is_ha_capable disjunct. That was a licensing proxy inherited from product_type rather than a physical prerequisite, so HA capable systems holding no feature key no longer get SPDK without a license.
The key is now injected into every legacy license instead of only enterprise models, so existing legacy holders keep SPDK on upgrade.
Gate TrueSearch through the entitlement engine
This commit adds changes to route the TrueSearch gate through the entitlement engine and drop the TrueNAS Connect disjunct, since a system that connects to TNC is issued a license carrying the key rather than being entitled by the connection itself. This also tightens the check from mere license presence to the feature key, which is what the feature matrix asks for, and the denial reason now comes from the entitlement instead of a hardcoded string naming TNC as an alternative.
Gate Webshare through the entitlement engine
This commit adds changes to gate Webshare through the entitlement engine. Webshare had no license check at all until now, so the key is injected into every legacy license to make sure no existing licensee loses their shares on upgrade.
Enforcement is set-time only, on share creation and on the service config. An update that leaves a share disabled is let through without the entitlement, so a system that has lost it can still turn a share off rather than having to delete it. Existing shares keep serving, since nothing re-reads the entitlement after the fact.
Note the WebUI currently hides Webshare from Enterprise systems and shows it to Community Edition, which is the inverse of this gate. That needs a matching change on the UI side before the two agree.
Gate directory services authentication through the entitlement engine
This commit adds changes to gate ds_auth, which controls whether directory services accounts may authenticate to the UI and API, through the entitlement engine instead of a bare is_enterprise read. Directory services themselves stay completely ungated on every product including Community Edition, so AD, LDAP and IPA are unaffected and only the UI and API login path is licensed. DIRECTORY_SERVICES also moves into the unconditional legacy injection bucket so every legacy licensee keeps it regardless of model.
Gate STIG and FIPS mode through the entitlement engine
This commit adds changes to route the security config gate through the entitlement engine instead of checking for the mere presence of a license. FIPS has no key of its own so the STIG entitlement governs every enterprise security option, matching the single STIG row in the feature matrix. The check stays on the write path only, since consuming it on the read path would silently unharden PAM on the next login.
Gate feature checks through the entitlement engine
This commit adds changes to route the dedup, SED, NVMe-oF SPDK, ZFS tiering and proactive support gates through truenas.entitlements.check instead of hand-composed license predicates scattered across plugins. It also fixes a crash in product_type when a license carries no hardware model, which now classifies as Community Edition.
Gate KMIP through the entitlement engine
This commit adds changes to gate KMIP key management through the entitlement engine. KMIP had no license check at all until now, so the key is injected into every legacy license to make sure no existing licensee loses key management on upgrade.
Only the enable transition is gated. Disabling KMIP is how escrowed ZFS and SED keys are pulled back to the local database, so a system that loses the entitlement must still be able to turn it off and recover its keys.
Gate apps, containers and VMs through the entitlement engine
This commit adds changes to route the apps, containers and VMs gates through the entitlement engine rather than raw feature flag reads, and injects those keys into legacy licenses so existing licensees keep both capabilities after upgrade. Note this means an injected key now overrides the legacy jails/vm bits on HA capable hardware, which is called out in a TODO next to the injection bucket.
Inject feature keys when translating legacy licenses
This commit adds changes to normalize legacy licenses at the translation layer by injecting the feature keys a legacy holder is entitled to today: capabilities gated on any valid license go to every legacy license, is_enterprise-gated capabilities go to enterprise models only, and CONTAINERS rides along with the legacy jails bit. This keeps backwards compatibility in one place so gates can evaluate the new key vocabulary uniformly.
Expose per-feature source and tier on license info
This commit adds changes to carry each licensed feature's source and tier qualifier on FeatureInfo, so entitlement rules can consume per-feature tiers (e.g. the SUPPORT contract tier) without reaching into raw license payloads.
Gate FEC mode configuration through the entitlement engine
This commit adds changes to gate FEC mode configuration on the entitlement engine instead of system.is_enterprise. The physical interface check and the ethtool capability probe stay where they are, since those are hardware facts the engine does not model.
NETWORK_FEC is now injected into every legacy license rather than only enterprise models, so HA capable systems whose license carries no model or a freenas prefixed one keep being able to configure it.
Gate NFS snapshot exposure through the entitlement engine
This commit adds changes to gate NFS snapshot exposure on the entitlement engine instead of system.is_enterprise, keeping the existing validation wording via a per-feature message override.
NFS_SNAPSHOT is now injected into every legacy license rather than only enterprise models. Its matrix vector is key-only on both hardware sides, so a legacy holder whose model is freenas-prefixed would otherwise lose the export on upgrade. The trade-off is that freenas certified systems, which are denied today, gain the feature.
Add license entitlement engine
This commit adds a pure entitlement engine that resolves whether a feature is available from hardware class and license facts, using per-feature policy rules: product-matrix vectors, a support-tier rule, a license-type rule for HA, and transitional legacy rules that reproduce current gate behavior until each feature is flipped onto its matrix vector. The full product matrix ships as reference data, and completeness tests keep the policy, matrix, display names and API vocabulary in sync so a new feature flag cannot silently skip a site.
Gate NVMe expansion shelves through the entitlement engine
This commit adds changes to gate JBOF expansion shelves through the entitlement engine rather than a bare enclosure count, keeping the shelf count itself as a conjunct and dropping the system serial check, which is a license to machine binding the license layer already owns. Legacy licenses have no way to carry a JBOF key since shelf ownership lives only in the additional hardware list, so the key is injected from the ES24N enclosure entry and licenses without a shelf are left untouched.
