Convert boot plugin to typesafe pattern
## Problem
The boot service was old-style dict-based code spread across three files (boot.py, boot_/format.py, boot_/boot_loader.py) that all declared the same BootService, so the framework silently merged them into a CompoundService. None of it was type-checked end to end, and every in-process consumer reached it through untyped string middleware.call('boot.*').
## Solution
- Consolidate everything into a single plugins/boot/ package using the port pattern: a lean BootService in __init__.py that delegates to plain, fully type-annotated module functions in disks.py/format.py/pool_ops.py, with @api_method(check_annotations=True) public methods and @private stubs. boot.get_state now returns a BootGetState model; internal callers use a dict state helper so nothing breaks on attribute-vs-key access.
- Register boot in main.py's ServiceContainer and have it own boot.environment as a sub-service, replacing BootServicesContainer.
- Expose the boot-pool name/disks cache through accessor functions instead of a module global. This also fixes a latent bug where libvirt/cdrom.py imported the global at module load (before setup runs), bound it to None, and so never filtered the boot pool out of CDROM path validation.
- Convert every same-process boot.* string call to typed call2/call_sync2 across sysdataset, zfs_events, disk, failover, and the zpool/audit/tunable/update/system_advanced plugins, the alert sources, and the cron/fstab makos, keeping each site's sync/async flavour.
- Keep the private update_initramfs/format models in the plugin (check_model_module requires private-method models to live there, not in the api package) and import them lazily from consumers, otherwise pulling boot/__init__ into the early service-init chain deadlocks on a partially-initialised middlewared.service.
- Add plugins/boot/ to mypy.yml.
Convert boot plugin to typesafe pattern
## Problem
The boot service was old-style dict-based code spread across three files (boot.py, boot_/format.py, boot_/boot_loader.py) that all declared the same BootService, so the framework silently merged them into a CompoundService. None of it was type-checked end to end, and every in-process consumer reached it through untyped string middleware.call('boot.*').
## Solution
- Consolidate everything into a single plugins/boot/ package using the port pattern: a lean BootService in __init__.py that delegates to plain, fully type-annotated module functions in disks.py/format.py/pool_ops.py, with @api_method(check_annotations=True) public methods and @private stubs. boot.get_state now returns a BootGetState model; internal callers use a dict state helper so nothing breaks on attribute-vs-key access.
- Register boot in main.py's ServiceContainer and have it own boot.environment as a sub-service, replacing BootServicesContainer.
- Expose the boot-pool name/disks cache through accessor functions instead of a module global. This also fixes a latent bug where libvirt/cdrom.py imported the global at module load (before setup runs), bound it to None, and so never filtered the boot pool out of CDROM path validation.
- Convert every same-process boot.* string call to typed call2/call_sync2 across sysdataset, zfs_events, disk, failover, and the zpool/audit/tunable/update/system_advanced plugins, the alert sources, and the cron/fstab makos, keeping each site's sync/async flavour.
- Keep the private update_initramfs/format models in the plugin (check_model_module requires private-method models to live there, not in the api package) and import them lazily from consumers, otherwise pulling boot/__init__ into the early service-init chain deadlocks on a partially-initialised middlewared.service.
- Add plugins/boot/ to mypy.yml.
Replace zectl with the truenas_bootenv engine
zectl is being retired in favor of the truenas_bootenv package that
ships with truenas_pylibzfs (companion commit there). The
boot_environment plugin now calls the engine in-process on
middleware's thread-local libzfs handle: query, clone, activate,
destroy and keep all go through it, utils.py with run_zectl_cmd is
deleted, and the boot-time promotion of installer-cloned datasets
uses the engine instead of parsing zfs list -j output.
Mutations serialize on an asyncio lock. A truenas:grub_pending
marker on the boot pool tracks pending menu regenerations and
setup() reconciles it on start. If the menu cannot be written,
activate rolls bootfs back. The boot pool is synced after every
menu write so the change is crash safe.
Re-activating the already-activated boot environment is allowed
again as the retry path after a failed menu regeneration, and
can_activate treats a missing truenas:kernel_version as not
activatable instead of only the literal "-".
Replace zectl with the truenas_bootenv engine
The typesafe boot_environment plugin now calls the engine from the
truenas_pylibzfs truenas_bootenv package on the middleware
thread-local libzfs handle instead of shelling out to zectl. Mutations
serialize under an asyncio lock, the boot menu regenerates through
etc.generate('grub') behind a truenas:grub_pending marker that setup()
reconciles after a crash, activate rolls bootfs back when the menu
cannot be written, and the pool is synced afterwards so the menu
change is crash durable.
Re-activating the activated boot environment is allowed again as the
retry path after a failed menu regeneration; the api2 test asserting
the old rejection now asserts the retry succeeds. can_activate treats
a missing truenas:kernel_version as not activatable instead of only
the literal dash. The boot-time dataset promotion uses the engine
instead of parsing zfs list -j output.
boot.environment: replace zectl with the truenas_bootenv engine
The typesafe boot_environment plugin now calls the engine from the
truenas_pylibzfs truenas_bootenv package on the middleware
thread-local libzfs handle instead of shelling out to zectl. Mutations
serialize under an asyncio lock, the boot menu regenerates through
etc.generate('grub') behind a truenas:grub_pending marker that setup()
reconciles after a crash, activate rolls bootfs back when the menu
cannot be written, and the pool is synced afterwards so the menu
change is crash durable.
Re-activating the activated boot environment is allowed again as the
retry path after a failed menu regeneration; the api2 test asserting
the old rejection now asserts the retry succeeds. can_activate treats
a missing truenas:kernel_version as not activatable instead of only
the literal dash. The boot-time dataset promotion uses the engine
instead of parsing zfs list -j output.
NAS-141719 / 26.0.0-RC.1 / Fix invalid nvmet defaults that make the test__verbose_subsys_query fail (by themylogin) (#19291)
Original PR: https://github.com/truenas/middleware/pull/19283
Co-authored-by: themylogin <themylogin at gmail.com>
boot.environment: replace zectl with the truenas_bootenv engine
The typesafe boot_environment plugin now calls the engine from the
truenas_pylibzfs truenas_bootenv package on the middleware
thread-local libzfs handle instead of shelling out to zectl. Mutations
serialize under an asyncio lock, the boot menu regenerates through
etc.generate('grub') behind a truenas:grub_pending marker that setup()
reconciles after a crash, activate rolls bootfs back when the menu
cannot be written, and the pool is synced afterwards so the menu
change is crash durable.
Re-activating the activated boot environment is allowed again as the
retry path after a failed menu regeneration; the api2 test asserting
the old rejection now asserts the retry succeeds. can_activate treats
a missing truenas:kernel_version as not activatable instead of only
the literal dash. The boot-time dataset promotion uses the engine
instead of parsing zfs list -j output.
NAS-141699 / 27.0.0-BETA.1 / fix duplicate From header in email (#19284)
`send_mail_queue()` updates the `From` address of queued messages with
`item.message["From"] = ...`, but `Message.__setitem__` *appends* a
second From header instead of replacing the existing one set by
`send_raw()`. The resulting message violates RFC 5322 and Gmail rejects
it with 550-5.7.1 "There are multiple From headers". Use
`replace_header()` so the flushed message keeps a single From header.
NAS-141699 / 26.0.0-RC.1 / fix duplicate From header and SMTP crash (#19286)
Two related bugs caused alert/test emails to be RFC 5322 non-compliant
(double From: header), which Gmail now rejects with 550-5.7.1:
1. When `fromname` is empty, `_from_addr()` returned an
`email.header.Header`; `Header.encode()` returns `str`, so the
`.encode().decode()` in `send_raw()` raised `AttributeError: 'str'
object has no attribute 'decode'` on every direct SMTP send.
`_from_addr()` now always returns `str`.
2. The failed message was then queued, and `send_mail_queue()` did
`queue.message['From'] = ...`, which *appends* a second From header
instead of replacing the existing one. Use `replace_header()` so the
flushed message keeps a single From header.
NAS-141699 / 25.10.5 / fix duplicate From header and SMTP crash (#19287)
Two related bugs caused alert/test emails to be RFC 5322 non-compliant
(double From: header), which Gmail now rejects with 550-5.7.1:
1. When `fromname` is empty, `_from_addr()` returned an
`email.header.Header`; `Header.encode()` returns `str`, so the
`.encode().decode()` in `send_raw()` raised `AttributeError: 'str'
object has no attribute 'decode'` on every direct SMTP send.
`_from_addr()` now always returns `str`.
2. The failed message was then queued, and `send_mail_queue()` did
`queue.message['From'] = ...`, which *appends* a second From header
instead of replacing the existing one. Use `replace_header()` so the
flushed message keeps a single From header.
pool: enforce vdev geometry limits and add force_topology
Enforce equal data-vdev width and cap RAIDZ/mirror width at
15/4 in pool.create and pool.update, mirroring truenas_pylibzfs.
Add force_topology (26 API) to bypass these topology policy
checks and the special/dedup redundancy rule; structural checks
still apply. Reject force_topology on Enterprise-licensed
systems.
(cherry picked from commit f07f7804cbb84ec1fe331e04485ec1b667dd3a2d)
boot.environment: replace zectl with the truenas_bootenv engine
The typesafe boot_environment plugin now calls the engine from the
truenas_pylibzfs truenas_bootenv package on the middleware
thread-local libzfs handle instead of shelling out to zectl. Mutations
serialize under an asyncio lock, the boot menu regenerates through
etc.generate('grub') behind a truenas:grub_pending marker that setup()
reconciles after a crash, activate rolls bootfs back when the menu
cannot be written, and the pool is synced afterwards so the menu
change is crash durable.
Re-activating the activated boot environment is allowed again as the
retry path after a failed menu regeneration; the api2 test asserting
the old rejection now asserts the retry succeeds. can_activate treats
a missing truenas:kernel_version as not activatable instead of only
the literal dash. The boot-time dataset promotion uses the engine
instead of parsing zfs list -j output.
boot.environment: replace zectl subprocess with truenas_bootenv engine
Proof of concept for testing alongside the truenas_pylibzfs
truenas_bootenv branch. The plugin calls the engine directly on the
middleware thread-local libzfs handle instead of shelling out to
zectl: mutations serialize under a module lock, the grub menu is
regenerated through etc.generate('grub') with a truenas:grub_pending
marker reconciled at startup after a crash, and the pool is synced so
the menu change is crash durable.
Based on the tree just before the boot_environment typesafe plugin
split; porting onto the new plugin layout is a known follow-up before
this can target master.
NAS-141711 / 26.0.0-RC.1 / Ensure truenas_zfstierd state on failover (by anodos325) (#19288)
Regenerate the truenas_zfstierd config on the promoted node.
Materialize enum_jobs() in the tier alert source so per-job get_info()
no longer opens a second read transaction inside the live iterator
(MDB_BAD_RSLOT; LMDB allows one read txn per thread per db).
Original PR: https://github.com/truenas/middleware/pull/19280
Co-authored-by: Andrew Walker <andrew.walker at truenas.com>
Ensure truenas_zfstierd state on failover
Regenerate the truenas_zfstierd config on the promoted node: the
service now owns its etc group and vrrp_master restarts it, so an
enabled database no longer surfaces a spurious tiering-disabled
error to clients.
Materialize enum_jobs() in the tier alert source so per-job
get_info() no longer opens a second read transaction inside the
live iterator (MDB_BAD_RSLOT; LMDB allows one read txn per thread).
(cherry picked from commit 41b32455f38864877a1576100c139461d0df6c3d)
NAS-141711 / 27.0.0-BETA.1 / Ensure truenas_zfstierd state on failover (#19280)
Regenerate the truenas_zfstierd config on the promoted node.
Materialize enum_jobs() in the tier alert source so per-job get_info()
no longer opens a second read transaction inside the live iterator
(MDB_BAD_RSLOT; LMDB allows one read txn per thread per db).
