[AMDGPU] Check the AGPR budget before selecting the AGPR form of MFMA
With -amdgpu-mfma-vgpr-form=0, selectAGPRFormMFMA selects the AGPR form
of an MFMA if the minimum from "amdgpu-agpr-alloc" is at least the
accumulator size. That is not sufficient for an accumulator wider than
128 bits. For such an MFMA, instruction selection selects the tied form,
and if src2 is live after the MFMA, the two-address pass converts it to
the early-clobber form. The early-clobber form needs src2 and vdst in
disjoint AGPRs, which is twice the accumulator size. If the function
cannot allocate that number of AGPRs, for example a 512-bit accumulator
with "amdgpu-agpr-alloc"="16", register allocation fails with "ran out
of registers during register allocation" in SelectionDAG and GlobalISel.
Record in SIMachineFunctionInfo the number of AGPRs that the function
can allocate, from GCNSubtarget::getMaxNumVectorRegs and the
-amdgpu-stress-agpr limit. Select the AGPR form only if that number is
at least twice the accumulator size for an accumulator wider than 128
bits, and at least the accumulator size otherwise. A vdst of 128 bits or
less can overlap src2, so the register allocator can split src2 instead.
[16 lines not shown]
[clang-repl] Rework -mllvm handling (#242861)
Parse -mllvm for the host and device CompilerInstances before their
IncrementalActions run, returning a bad option as an llvm::Error from
Interpreter::create rather than exiting the process (#197133). Point the
Orc and Wasm TargetMachines, which ignore the parsed options, at the
LLVMContext's library options. Drop the Wasm executor's re-parse with
cl::ParseCommandLineOptions, which rejects library options; lld no
longer resets LLVM options (#242717).
Parse clang-repl's own command line after target initialization with
cl::parseCommandLineOptions, forwarding library options as -mllvm, so
`clang-repl -print-after-isel` works again.
Stop calling llvm_shutdown() after each unittest suite; it destroyed
the option registry that later suites need.
LLM-aided
CodeGen: Simplify def segment extension in shrinkToUses
During extendSegmentsToUses, Segments holds only one def segment per
value, and none extend past their block. Live-in segments are kept
separately in LiveIns. To extend a def to a use, find the last segment
starting before the use, check that it starts in the use's block, and
move its end to the use. This replaces extendInBlock, whose general
search and merging of overlapped segments never applies here.
Also combine the per-block live-out and live-in tables into one struct.
llc instructions:u:
CTMark x86_64: -0.07%
CTMark arm64: -0.10%
AMDGPU sample: -0.27%
Largest AMDGPU improvements:
rocprim warp_sort (gfx1100): -1.89%
rocsolver stedc (gfx942): -1.70%
[5 lines not shown]
CodeGen: Merge shrinkToUses live-in segments in one pass
extendSegmentsToUses visits blocks in worklist order and inserted each
live-in segment into the middle of the segment vector, as did
createSegmentsForValues for the def segments. Sort both and merge them
in one pass instead, with at most one live-in segment per block. The
gain is largest in cases with large live ranges across many blocks.
llc instructions:u:
CTMark x86_64: -0.07%
CTMark arm64: -0.03%
AMDGPU sample: -0.28%
Largest AMDGPU improvements:
rocsolver potf2-specialized-kernels-d (gfx942): -4.57%
rocsolver getf2-small-d (gfx942): -2.71%
rocsolver getf2-small-d (gfx1100): -2.10%
rocprim warp_sort (gfx1100): -1.52%
Blender Cycles kernel (gfx1030/gfx1100/gfx1201): -0.9%
[2 lines not shown]
textproc/py-owlrl: import py-owlrl-7.6.2
A simple implementation of the OWL2 RL Profile, as well as a basic
RDFS inference, on top of RDFLib, based on forward-chaining.
[AMDGPU] Check the AGPR budget before selecting the AGPR form of MFMA
With -amdgpu-mfma-vgpr-form=0, selectAGPRFormMFMA selects the AGPR form
of an MFMA if the minimum from "amdgpu-agpr-alloc" is at least the
accumulator size. That is not sufficient for an accumulator wider than
128 bits. For such an MFMA, instruction selection selects the tied form,
and if src2 is live after the MFMA, the two-address pass converts it to
the early-clobber form. The early-clobber form needs src2 and vdst in
disjoint AGPRs, which is twice the accumulator size. If the function
cannot allocate that number of AGPRs, for example a 512-bit accumulator
with "amdgpu-agpr-alloc"="16", register allocation fails with "ran out
of registers during register allocation" in SelectionDAG and GlobalISel.
Record in SIMachineFunctionInfo the number of AGPRs that the function
can allocate, from GCNSubtarget::getMaxNumVectorRegs. Select the AGPR
form only if that number is at least twice the accumulator size for an
accumulator wider than 128 bits, and at least the accumulator size
otherwise. A vdst of 128 bits or less can overlap src2, so the register
allocator can split src2 instead. Keep the check of the minimum as it
[12 lines not shown]
asmc: make asmc_sysctl_add_system() void
Its only caller already discarded the return value with (void). Match
the other sysctl subtree builders.
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D61537
asmc: do not report absent sensors as a temperature
A key can exist for a sensor the model was never built with; the SMC
answers those with a fixed sentinel instead of an error. Return ENXIO
for the sentinel instead of publishing it as a reading.
MFC after: 3 days
Reviewed by: adrian
Differential Revision: https://reviews.freebsd.org/D61489
asmc: report temperatures in deciKelvin
The handler returned the raw key's high byte, dropping the sign and
fraction of the sp78 encoding, and published -1 on a failed read.
Read through asmc_sensor_read() instead, convert to deciKelvin, and
register the leaves as "IK". Changes the unit of dev.asmc.N.temp.*.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61487
asmc.4: fix the unterminated SYNOPSIS and FILES blocks
The SYNOPSIS .Bd and the FILES .Bl were both left open, producing two
mandoc block-nesting errors. Reduce SYNOPSIS to the .Cd line man4
style calls for, and close the FILES list.
MFC after: 3 days
Reviewed by: ziaee, ngie
Differential Revision: https://reviews.freebsd.org/D61486
asmc: propagate sysctl read and write errors
Several handlers discarded the key transport's result: fan speed, SMS
and ambient light handlers published a value regardless of read
success; a rejected fan speed or keyboard backlight write reported success;
fan id returned EPERM on an unreadable key instead of ENXIO.
asmc_fan_getvalue()/asmc_fan_setvalue() now return the error and pass
the speed out by reference instead of returning -1 for failure.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61485
asmc: register the fan id sysctl with a string format
Every fan sysctl was registered with a hardcoded "I" format, but id
is CTLTYPE_STRING. Carry the format in the table instead.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61484
asmc: clear the sensor counts in detach
The key arrays are freed but the counts were left set, leaving the
softc describing freed memory. Reachable via the attach error path,
which calls detach directly.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61483
asmc: do not drop shutdown notification when the system node fails
asmc_attach() registered the MBSS shutdown-cause handler
(EVENTHANDLER_REGISTER(shutdown_pre_sync, asmc_shutdown, ...))
after attempting to add the dev.asmc.N.system sysctl subtree. If that
subtree registration failed, the error path took goto nosms, jumping clean
over the shutdown-handler registration. On T2 Macs this silently drops
the only thing standing between a clean shutdown and bridgeOS's
watchdog power-cycling the machine.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61482
asmc: remove the unused sc_dev softc field
It was never assigned or read. legacy code.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61481
asmc: print the last field of the SMC revision correctly
Use be16dec() at the correct offset instead of an unaligned,
off-by-two-bytes read through a cast pointer.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61479
asmc: restore SMC state across suspend/resume
Reinit the SMS interrupt mask, motion thresholds, and shock window on
resume. Guard the backlight rewrite on sc_has_light. Add
device_suspend to release manual fan control before sleep and restore
it on resume.
MFC after: 3 days
Reviewed by: ngie
Differential Revision: https://reviews.freebsd.org/D61478
[AMDGPU] Keep metadata when shrinking scaled MFMA fp4/fp6 operands
InstCombine narrows the A/B operands of llvm.amdgcn.mfma.scale.*.f8f6f4
for fp4/fp6 formats by creating a new call, but did not copy the old
call's metadata. Copy it, as the other rewrites in this file do, so
hints such as !amdgpu.mfma.agpr survive.
[AMDGPU] Select the MFMA AGPR form from !amdgpu.mfma.agpr metadata
Problem
- Frontends (Triton) pin MFMA accumulators to AGPRs with "=a" inline asm.
- The asm forces VGPR->AGPR copies around every MFMA (v_accvgpr_mov
+ s_nop inside loops), and selection has no way to ask for the AGPR
form per instruction: it is chosen per function.
Idea: a droppable hint on the MFMA call, read when the form is selected.
IR call @llvm.amdgcn.mfma.*(...), !amdgpu.mfma.agpr !{}
| SelectionDAGBuilder: TLI.getTargetIntrinsicHint(I)
DAG INTRINSIC_WO_CHAIN --SDEI side table: TargetHint
| CanUseAGPR_MAI predicate
MIR V_MFMA_..._e64 (AGPR form): C/D are AReg, no copies
How
1. TargetLowering::getTargetIntrinsicHint returns metadata from an
intrinsic call; SelectionDAGBuilder stores it on the node in the
[26 lines not shown]
