[AArch64][MachineOutliner] Allow calls to be outlined when PAC is enabled (#230840)
This patch allows call sequences to be outlined when PAC-RET is active
by relaxing the stack pointer modification check for valid thunk
candidates.
Direct calls outlined as thunks are safe because the return address is
signed in the prologue and authenticated in the epilogue, while
candidates with intermediate calls that would clobber LR are still
rejected.
Fixes #230519.
> **Note on scope:** This patch specifically addresses direct and tail
calls. Indirect calls (`BLR`) under `-mbranch-protection=standard`
interact with existing BTI (`branch-target-enforcement`) constraints and
will be addressed in a follow-up patch.
camcontrol: Fix extended host identifier retrieval
The Get Features command for the host identifier always requested
the 64-bit form (cdw11 EXHID=0), but the decoding logic sized the
identifier from the controller's CTRATT extended host identifier
capability bit, or always treated it as 128 bits when -U was
given. On a controller supporting the extended host identifier,
the printed value did not match what was requested from the
controller.
Fetch the controller data before issuing the command, request the
extended form when the controller supports it, and decode the form
that was requested. Reject -U when the controller only supports
64-bit host identifiers, since an 8 byte value cannot be shown as
a UUID.
sys/cam/nvme/nvme_all.c,
sys/cam/nvme/nvme_all.h:
Add an exhid argument to nvme_get_feature_hostid(),
[19 lines not shown]
camcontrol: Don't close the CAM device in nvme_get_data()
nvme_get_data() closed the supplied CAM device on every error
path, but its callers either pass the device that main() opened
and closes, or close the device themselves, so an error resulted
in a double cam_close_device() and a double free. Move the error
path close into print_dev_nvme(), the one caller that opens its
own device, and leave the device alone in nvme_get_data().
This predates the recent camcontrol NVMe additions, but they added
more callers.
Reported by: llm-postcr
Fixes: 6ee13c54dad0
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit f7c24ef1e9a47b889eaaae8514c61e95d9738cf8)
nvmecontrol: Range-check the format command's field values
The format command accepted any 32-bit value for -m, -p, -l and
--ses and passed it into the Format NVM command. Before
eae727676169 an out-of-range value spilled into the adjacent cdw10
fields, which usually drew an error from the controller; after it,
the value is masked to the field width, so a mistyped value is
silently accepted with a different meaning. Most seriously,
--ses 8 masked to SES=0 and performed a plain format with no erase
at all, exiting 0. A value like --ses 10 masked to a cryptographic
erase while bypassing the FNA cryptographic erase capability check,
which compares the unmasked value.
Reject values outside the ranges nvmecontrol.8 documents, which are
the values the spec defines: 0-1 for -m and -l, 0-3 for -p, and
0-2 for --ses. The checks run at option validation time, before
the device is opened, and the capability check now only ever sees
valid values.
[12 lines not shown]
nvmecontrol: Mask the fields in the Format NVM command
The format command composed cdw10 by hand without masking any of
the fields and with no support for the LBAFU field. Since commit
2c41f7d8a034 the LBA format index defaulted from FLBAS can exceed
15 on an NVMe 2.0 namespace using FIDXU, and such an index would
bleed into the MSET and PI fields. An out of range index supplied
with -f for a whole-controller format was not caught either.
Build the command with nvme_admin_format_cmd(), which masks all of
the fields and encodes LBA format indexes above 15 in LBAFU, and
reject LBA format indexes greater than 63 for whole-controller
formats. Per-namespace formats already validate the index against
NLBAF.
This is the same problem that llm-postcr reported in camcontrol's
copy of this code, fixed in 3892a6453fc6; the unmasked composition
originated here (635c517ae9ca).
[9 lines not shown]
camcontrol: Fix -Wcast-align errors in the NVMe reservation report
Casting the byte buffer that holds the Reservation Report payload to
the status structures trips -Wcast-align on strict-alignment
architectures, which broke the CI builds there. Put the buffer in a
union with the two status structures instead, so it is properly
aligned and no casts are needed.
Reported by: FreeBSD CI
Fixes: ba75cf05727e ("camcontrol: Support NVMe reservations via the persist subcommand")
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
MFC after: 1 week
Sponsored by: Spectra Logic
(cherry picked from commit 68ba58a064d1004a031fac6f299f968dfc39241f)
nvme: Fix the data length in the Reservation Report command builder
nvme_ns_resv_report_cmd() put its length argument, which callers
supply in bytes, directly into cdw10. The NUMD field is a 0's
based count of dwords, so the command asked the controller to
transfer nearly four times the size of the supplied data buffer.
In practice controllers limit the transfer to the length described
by the PRP entries, which is why this worked in testing, but the
command was incorrect.
Convert the byte count to a 0's based dword count in the builder.
Reported by: llm-postcr
Fixes: 7a9e4375316b
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit 86b9d629c68f6dc63543b9c0a6da6c96c857f3bc)
nvme: Byte-swap the nsid in the Get/Set Features command builders
nvme_admin_get_feature_cmd() and nvme_admin_set_feature_cmd() stored
the namespace ID in the command without converting it to little
endian, unlike the other command building functions. Any caller
passing a non-zero namespace ID would send the wrong namespace ID on
big endian machines.
Reported by: llm-postcr
Fixes: 7a9e4375316b
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit 3a6dbe77c39d3a88b6092cfcaa820fc727d1e61d)
camcontrol: Fix swapped protection information defaults in nvformat
When no -p or -l arguments are given, the nvformat subcommand fills
in the protection information type and location from the
namespace's current settings. The two were swapped: pi was taken
from the DPS metadata location bit and pil from the protection
information type field. Formatting a namespace with protection
information enabled would therefore silently change its PI
settings, and before the Format NVM command builder masked its
fields, a namespace with PI type 2 enabled would set bit 9 of
cdw10, turning the format into a user data erase.
Reported by: llm-postcr
Fixes: 80f090111ae4
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit ef69484b4295e84c74877b439db5caf025906b0c)
nvme: Restore the host memory buffer size calculation
When the nvme_ctrlr_cmd_set_feature() call in
nvme_ctrlr_hmb_enable() was rewritten for the widened Set Features
API, the HSIZE calculation reverted to dividing by a hard coded
4096 instead of the controller page size, undoing commit
3740a8db13ba. The Host Memory Buffer Size field is in units of
the current memory page size (CC.MPS), so the wrong size would be
programmed on controllers configured with a page size other than
4KB.
Reported by: llm-postcr
Fixes: 7a9e4375316b
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit f262bdcf24f1f946edcd01f5eb5dfa2910da9e7b)
camcontrol: Require a reservation type for reserve, preempt and release
res_type was initialized to 0, but the checks that enforce the -T
argument for the reserve, preempt and release reservation actions
compare it against -1, so they could never fire and camcontrol
silently sent a reservation type of 0, which is invalid.
Initialize res_type to -1 so the checks work, and default the
reservation type to 0 for the release action's clear variant,
which does not take -T and ignores the field.
Reported by: llm-postcr
Fixes: ba75cf05727e
MFC after: 1 week
Sponsored by: Spectra Logic
Co-authored-by: Claude Fable 5 <noreply at anthropic.com>
(cherry picked from commit b95e2b2ce907cde89bc3e7be2ffeae03cfed55ab)
nvme: Mask the fields in the Format NVM command builder
nvme_admin_format_cmd() did not mask any of the fields it composes
into cdw10, so an out of range value in one field would corrupt
neighboring fields. In particular, an LBA format index greater
than 15 would bleed into the metadata settings and protection
information fields. It also had no support for LBA format indexes
above 15, which NVMe 2.0 encodes via the separate LBAFU field.
Mask all of the fields, and encode bits 5:4 of the LBA format index
in the LBAFU field (cdw10 bits 13:12).
Also reject out of range LBA format indexes in camcontrol's
nvformat global (-G) mode, which has no namespace data to validate
against. The per-namespace mode already validates the index
against NLBAF.
sys/dev/nvme/nvme.h:
Mask the SES, PIL, PI, MSET and LBAF fields in
[14 lines not shown]
cam: Add NVMe reservation, format and features CCB building functions
Add NVMe CCB (union ccb / struct ccb_nvmeio) building functions for
the four NVMe reservation opcodes, Format NVM, and Get Features /
Set Features. These use the nvme_command building blocks added to
sys/dev/nvme/nvme.h, and are used by the new camcontrol(8) nvres,
nvformat and nvfeature subcommands that will be added separately.
sys/cam/nvme/nvme_all.c:
Add NVMe CCB building functions for the four NVMe reservation
opcodes: nvme_resv_report(), nvme_resv_register(),
nvme_resv_acquire() and nvme_resv_release(). Data direction
is set internally (CAM_DIR_IN or CAM_DIR_OUT as appropriate),
so no flags parameter is needed.
Add a new nvme_format() CCB building function.
Add nvme_get_feature() and nvme_set_feature() CCB building
functions, along with nvme_get_feature_hostid() and
[18 lines not shown]
nvme: Add reservation, format and features command building support
Add command (struct nvme_command) building functions for the four
NVMe reservation opcodes (Report, Register, Acquire and Release),
Format NVM, and Get Features / Set Features, along with the
enumerations and data structures that go with them. These are
shared between the kernel and userland, and are the underpinnings
for new camcontrol(8) subcommands (nvres, nvformat, nvfeature) that
will be added separately.
The reservation type names are intentionally similar to the SCSI
names for the same reservation types.
sys/dev/nvme/nvme.h:
Add NVMe command building functions for the four NVMe
reservation opcodes: nvme_ns_resv_report_cmd(),
nvme_ns_resv_register_cmd(), nvme_ns_resv_acquire_cmd() and
nvme_ns_resv_release_cmd().
[70 lines not shown]
camcontrol: Support NVMe namespace formatting via the format subcommand
Support formatting NVMe namespaces through the existing format
subcommand. The format subcommand now dispatches on the device
protocol, in the same way the identify and persist subcommands do,
and issues the NVMe FORMAT NVM command for NVMe devices.
The SCSI syntax and behavior are unchanged. The NVMe-specific
options are -e (secure erase setting), -G (format all namespaces),
-f (LBA format), -l (protection information location), -m (metadata
setting) and -p (protection information type). The -q and -y options
work for both protocols; NVMe formats gain the same confirmation
question that SCSI formats have always had. The -r and -w options
are SCSI only; -r returns an error for NVMe devices and -w has no
effect, since the NVMe format command does not complete until the
format is finished.
The default timeout for NVMe is 30 seconds for a cryptographic erase
and 30 minutes otherwise; the generic -t option overrides it.
[25 lines not shown]
camcontrol: Add support for NVMe namespace Identify
Add a -N argument to camcontrol identify. This will fetch and
display namespace Identify data for NVMe drives, using
nvme_print_namespace() from nvmecontrol(8).
This gets the namespace Identify data via XPT_DEV_ADVINFO (i.e.
data cached in the kernel) like the standard NVMe identify command
does, and so works even on NVMe devices that do not support NVMe
Namespace Management.
sbin/camcontrol/camcontrol.c:
Add a -N argument to camcontrol identify.
Rename nvme_get_cdata() to nvme_get_data(), and give it the
ability to fetch controller or namespace Identify data.
While here, improve its error reporting with
cam_error_print(), and only look at the CAM status bits of
the returned CCB.
[24 lines not shown]
camcontrol: Add a feature subcommand for NVMe Get/Set Features
Add a feature subcommand for the NVMe Get Features and Set Features
commands:
camcontrol feature ndaX -a <get|set> -f feature [-d data]
[-s num] [-U]
There is no transport-specific prefix in the name: SCSI has no
feature command (the closest equivalents are mode pages), and the
ATA SET FEATURES subcommands are already exposed through dedicated
camcontrol subcommands (apm, aam, epc and the power commands), so
the name is unclaimed. The subcommand returns an error for
non-NVMe devices.
Currently, only the hostid feature is fully implemented. The
hostid value for set is specified with -d as a UUID or an integer
in decimal, hex, or octal, and -U outputs the hostid as a UUID on
get.
[38 lines not shown]
camcontrol: Support NVMe reservations via the persist subcommand
The NVMe reservation model is closely patterned after SCSI Persistent
Reservations, so support NVMe devices through the existing persist
subcommand rather than adding a transport-specific subcommand. The
persist subcommand now dispatches on the device protocol, in the same
way the identify subcommand does.
The SCSI syntax is unchanged. For NVMe devices, the -i modes all map
to the Reservation Report command (read_full_status requests the
extended data structure, which contains 128-bit Host Identifiers), and
the -o service actions map to the corresponding NVMe reservation
operations: register, register_ignore, reserve, preempt,
preempt_abort, release and clear, plus two NVMe-only actions,
unregister and replace. The reservation type names for -T are shared
between SCSI and NVMe. A new -c option gives explicit control over
the NVMe Persist Through Power Loss State; -p is equivalent to
-c enable. SCSI-only options return an error for NVMe devices.
[34 lines not shown]
nvme: honor FLBAS Format Index Upper when selecting the LBA format
Added nvme_ns_data_format_index() in the nvme, nda, and nvmf
host paths as well as nvmecontrol and camdd.
Reviewed by: imp, adrian
Differential Revision: https://reviews.freebsd.org/D59627
(cherry picked from commit 2c41f7d8a034ee64a0aaa69a1aa5cb07748a1a32)
nvmecontrol: Factor out namespace Identify printing
Move print_namespace() from identify.c to identify_ext.c and rename
it to nvme_print_namespace(), to match nvme_print_controller().
This allows other consumers (notably camcontrol(8)) to print NVMe
namespace Identify data.
No functional changes intended.
sbin/nvmecontrol/identify.c:
Remove print_namespace(), and call nvme_print_namespace()
instead.
sbin/nvmecontrol/identify_ext.c:
Add nvme_print_namespace(), moved here from identify.c.
sbin/nvmecontrol/nvmecontrol.h:
Remove the print_namespace() prototype.
[14 lines not shown]
[CIR][NVPTX] Support WMMA fragment loads (#234812)
Emit CIR for all 47 WMMA fragment-load builtins. Marshal scalar and
aggregate fragments through NVVM load intrinsics, preserving packed-half
representations and selecting row/column layout variants.
Add CIR and shared classic/CIR LLVM checks for all 88 supported
load/layout combinations, plus target-feature and invalid-argument
diagnostics.
Validation: 40 relevant lit tests pass. A clean assertion-enabled
compiler build and LLVM IR verification succeed. SM70/SM80 PTX
generation succeeds, with all 88 load instruction mnemonics matching
classic codegen. All 18 scalar cases directly store the intrinsic
result.
Related: https://github.com/llvm/llvm-project/issues/179278
Assisted-by: OpenAI Codex