[flang] Fix NOTIFY= placement checking and folding (#231660)
F2023 C933 permits NOTIFY= only in the image selector of an intrinsic
assignment's target variable. Before this patch, semantic analysis
accepted it in other contexts.
Assignment checking reuses the analyzed assignment's classification as
intrinsic or defined. For intrinsic assignment, an existing parse-tree
helper identifies the target variable's own image selector. The new
check permits a NOTIFY= specifier only if its parse-tree node belongs to
that selector's specifier list. It rejects NOTIFY= on separate
designators nested within the target variable.
If an operand error prevents assignment analysis, the checker avoids an
extra placement diagnostic on the target's own selector. Existing
duplicate-specifier diagnostics remain unchanged.
Before this patch, FoldOperation(CoarrayRef) dropped NOTIFY= from the
rebuilt reference. The change retains and folds the notification
[8 lines not shown]
[flang][NFC] Expose force-no-alias option via CommandLineOpts (#232568)
Move the force-no-alias cl::opt from a file-local static in
Pipelines.cpp
to CommandLineOpts.cpp and declare it extern in CommandLineOpts.h, so
other pipelines built on top of flang can honor the same setting without
re-registering the option.
[mlir][Diagnostics] Set parallel diagnostic order ids without a lock (#232520)
failableParallelForEach tells its ParallelDiagnosticHandler which
element each thread is processing before every element and clears it
after, so diagnostics come out in element order.
The order id now lives in a thread-local slot naming its handler by a
monotonic id (no raw pointers, so a handler destroyed with a stale slot
naming it is harmless). A thread handles one element at a time per
handler, so the slot covers every thread whose loop is not nested in
another loop on the same thread.
200K single-op func.func through per-function cse, 96 threads: sys time
22.9s to 3.8s, wall 3.4s to 3.2s. Unittest pins the nested fallback and
outer-id preservation.
[BOLT] Expand builtin_unreachable recognition heuristics (#231258)
A jump to the address right past the end of a function (the result of
__builtin_unreachable()) was replaced with a nop only when the target
was within the function's padding (MaxSize). However, there is a corner
case for the very last function in a section: MaxSize equals Size, so
BOLT would see __builtin_unreachable() instead as as an interprocedural
reference to an address outside of any section, tripping the "cannot get
section for referenced address" assertion in
processInterproceduralReferences(). Fix this.
[X86] Address review: test a tied input's untied read under -regalloc=fast
Add the case from the #229630 review, one value passed to an untied "r"
input and to the "0" input tied to an "rm" output, to show that the fast
register allocator no longer folds the output when everything fits.
Drop the -O0 RUN line from inline-asm-direct-mem-output-error.ll: it only
showed that "=rm" still picks memory at -O0, which asm-constraints-rm.ll
already tests, and there make the -O0 line name the instruction selector
and say that it tests the optimization level.
Assisted-by: Claude Opus 5.5
[TargetLowering][X86] Prefer 'r' over 'm' for foldable "rm" inline asm operands
An "rm" (register-or-memory) inline asm operand has always resolved to
'm', because getConstraintPreferences() picks the most general
constraint present, and 'm' is more general than 'r'. That's safe, since
memory can't run out, but it forces a value that could stay in a
register through a stack slot even when there's no register pressure
(https://github.com/llvm/llvm-project/issues/20571).
Prefer 'r' instead where the register allocator can fold the register
back to a stack slot when it runs out of registers, and mark the
register operand foldable (InlineAsm::Flag::setRegMayBeFolded()) so it
does. Both allocators can: the greedy allocator folds an operand when it
spills its value, and the fast allocator folds operands up front when
the asm's register operands wouldn't fit.
ParseConstraints() sets AsmOperandInfo::MayFoldRegister for an operand
whose constraint codes include 'r' and 'm' and are otherwise only
immediate codes, above -O0, on a target that opts in through the new
[26 lines not shown]
[InstCombine][AMDGPU] Fold select of intrinsics differing in one operand (#227737)
When both arms of a select are calls with one use to the same trivially
vectorizable intrinsic and differ in exactly one operand, sink the
select
into that operand:
$$\mathrm{select}(c,\ f(x, y),\ f(x, z)) \to f(x,\ \mathrm{select}(c, y,
z))$$
This removes one call. FMF are intersected, and the new call takes the
debug location of the select.
The fold is skipped when:
* the differing operand is an immarg,
* either differing operand is a constant, since that may enable a
cheaper
lowering (e.g. $x^2 \to x \cdot x$),
* the condition is a vector but the differing operand is a scalar, as
[3 lines not shown]
[RegAllocFast] Address review: don't recount a tied input's untied reads
When one value is passed to both a tied operand and an untied one, e.g.
"=rm"(r) : "r"(v), "0"(v), with an explicit -regalloc=fast above -O0 (so
TwoAddress doesn't run and the ties reach RegAllocFast), the fold estimate
counted three registers where lowerTiedUse() uses one. Tied uses are
allocated first, so the value moves into the def's register and the untied
read is rewritten to it. tiedUseNeedsOwnReg() still assumed another read of
the value could get it a register first, and the untied read was counted on
its own as well, so the asm's output was folded to the stack needlessly.
Model the tied input itself: while one of its tied defs is in a register,
it needs no register of its own unless tiedUseNeedsOwnReg() says so (only
an early-clobber def now makes another read matter), and its untied reads
share whichever register it has. Once its tied defs are folded, the untied
reads need a register again, so choose what to fold by counting again with
each candidate folded, and fold both halves when neither alone helps.
Assisted-by: Claude Opus 5.5
[clang][CIR] Add tests for SVE MAXNMV intrinsics (#232198)
adds CIR tests for `svmaxnmv` (plain SVE) intrinsics.
Moves and adapts `sve-intrinsics/acle_sve_maxnmv.c` to `sve/maxnmv.c`
Part of #223963
[clang][CIR] Add tests for SVE MINNMV intrinsics (#232199)
adds CIR tests for `svminnmv` (plain SVE) intrinsics.
Moves and adapts `sve-intrinsics/acle_sve_minnmv.c` to `sve/minnmv.c`
Part of #223963
[RISCV][Docs] Rename Extensions section in RISCVUsage to 'Ratified Extensions' (#232488)
We have separate sections for Experimental and Vendor so let's name this
what it is.
[lldb] Rename OS register metadata providers (#231564)
Rename platform/arch-specific RegisterContext* to RegisterInfo* under
Process/Utility as these files contain classes derived from
RegisterInfoInterface not RegisterContext.
[TargetLowering][X86] Prefer 'r' over 'm' for foldable "rm" inline asm operands
An "rm" (register-or-memory) inline asm operand has always resolved to
'm', because getConstraintPreferences() picks the most general
constraint present, and 'm' is more general than 'r'. That's safe, since
memory can't run out, but it forces a value that could stay in a
register through a stack slot even when there's no register pressure
(https://github.com/llvm/llvm-project/issues/20571).
Prefer 'r' instead where the register allocator can fold the register
back to a stack slot when it runs out of registers, and mark the
register operand foldable (InlineAsm::Flag::setRegMayBeFolded()) so it
does. Both allocators can: the greedy allocator folds an operand when it
spills its value, and the fast allocator folds operands up front when
the asm's register operands wouldn't fit.
ParseConstraints() sets AsmOperandInfo::MayFoldRegister for an operand
whose constraint codes include 'r' and 'm' and are otherwise only
immediate codes, above -O0, on a target that opts in through the new
[26 lines not shown]
[RegAllocFast] Fold foldable inline asm operands under register pressure
An inline asm register operand marked foldable (from an "rm" constraint)
may be replaced with a stack slot when the register allocator runs out
of registers. The greedy allocator does that when it spills the value.
The fast allocator can't: it assigns the operands of an instruction one
at a time, and folding replaces the instruction. So it reported
"inline assembly requires more registers than available" instead.
Before allocating a block, estimate from each inline asm's own operands
whether they fit in registers, and fold as many foldable registers as
needed to make them fit, so the common case without pressure still gets
a register. Values that only live across the asm don't count, since the
allocator spills them when it needs their registers. The estimate
follows allocateInstruction():
- First all defs get distinct registers, avoiding physreg defs; then all
uses, along with the defs still occupied while the uses are read
(early-clobber and tied defs, see isLiveThroughDef()), get distinct
[26 lines not shown]
[CodeGen] Address review: error for "=m|r" direct outputs too
A direct output whose chosen alternative is memory, e.g. "=m|r", still
crashed SelectionDAG, which exempted multiple-alternative operands from the
check and then asked for the address of an output with no value. Drop the
exemption and test "=m|r" with both instruction selectors. Name the
selector explicitly in the RUN lines rather than relying on -O levels,
since X86 GlobalISel can't lower inline asm and both X86 lines ran
SelectionDAG anyway. Drop "yet" from the error: Clang never emits a
direct memory output, so there are no plans to support one.
Assisted-by: Claude Opus 5.5
[CodeGen] Report an error for a direct inline asm output in memory
An inline asm output returned by value has no memory to write to, yet a
constraint such as "=rm" picks memory, the most general constraint, as
does "=m". SelectionDAG asserted on that ("Can only indirectify direct
input operands!"), and GlobalISel dereferenced a null pointer. Clang
never emits such an output, since it passes the address of a memory
output, but other IR can. Report "cannot handle direct memory outputs
yet for constraint 'm'" instead, like the other inline asm errors there.
Assisted-by: Claude Opus 5.5
[MLIR][AsmPrinter] Add alwaysUseAliases printing flag (#232518)
Per-function IR dumps print a nested op, which takes the non top-level
path where types and attributes expand in full. For passes over
functions with heavy types this buries the dump in noise that aliases
would fold away.
Add OpPrintingFlags::alwaysUseAliases plus the
mlir-print-always-use-aliases tool option. When set, printing
initializes aliases and takes the top-level path for any op, so nested
dumps use aliases. The flag takes precedence over local scope on
purpose: pass instrumentation prints nested ops with local scope, so
without precedence the per-function dumps this flag targets would never
use aliases.
Reusing one AsmState across prints must not carry the previous op
aliases or dialect resources into the next print, so initialize now
starts both fresh.
Assisted-by: Muse Code (Muse)
[TargetInstrInfo] Address review: autogenerate greedy-inline-asm-fold.mir checks
Generate the checks with update_mir_test_checks.py so that the spills and
reloads around each folded inline asm are verified, not just the INLINEASM
itself. Fix the "=r" comment to describe the C "+r" operand, note that which
"+rm" pair greedy folds is its own choice, and add a "+&rm" case on an
uninitialized value showing that folding a pair whose tied use is undef
stores to the slot without loading from it.
Assisted-by: Claude Opus 5.5
[APFloat][NFCI] Move x87DoubleExtended next to the other native IEEE types (#223193)
Reordered the `APFloatBase::Semantics` so that `S_x87DoubleExtended` is
placed after the IEEE types and before the DoubleDouble types.
Previously, `S_x87DoubleExtended` was the very last `enum` value,
meaning that `SemanticsToEnum`/`bitcastToAPInt`/`initFromAPInt` would
test if `Sem` is one of the 13 GPU floating point types before testing
if `Sem` is `x87DoubleExtended`. Given that `x87DoubleExtended` is more
commonly used and a native LLVM IR type, I thought a better heuristic
would be to test if `Sem` is a `x87DoubleExtended` type before testing
if it is one of the 13 GPU floating point types.
I also made the order of other functions/fields more consistent across
`APFloat`.
[TargetLowering][X86] Prefer 'r' over 'm' for foldable "rm" inline asm operands
An "rm" (register-or-memory) inline asm operand has always resolved to
'm', because getConstraintPreferences() picks the most general
constraint present, and 'm' is more general than 'r'. That's safe, since
memory can't run out, but it forces a value that could stay in a
register through a stack slot even when there's no register pressure
(https://github.com/llvm/llvm-project/issues/20571).
Prefer 'r' instead where the register allocator can fold the register
back to a stack slot when it runs out of registers, and mark the
register operand foldable (InlineAsm::Flag::setRegMayBeFolded()) so it
does. Both allocators can: the greedy allocator folds an operand when it
spills its value, and the fast allocator folds operands up front when
the asm's register operands wouldn't fit.
ParseConstraints() sets AsmOperandInfo::MayFoldRegister for an operand
whose constraint codes include 'r' and 'm' and are otherwise only
immediate codes, above -O0, on a target that opts in through the new
[26 lines not shown]
[RegAllocFast] Fold foldable inline asm operands under register pressure
An inline asm register operand marked foldable (from an "rm" constraint)
may be replaced with a stack slot when the register allocator runs out
of registers. The greedy allocator does that when it spills the value.
The fast allocator can't: it assigns the operands of an instruction one
at a time, and folding replaces the instruction. So it reported
"inline assembly requires more registers than available" instead.
Before allocating a block, estimate from each inline asm's own operands
whether they fit in registers, and fold as many foldable registers as
needed to make them fit, so the common case without pressure still gets
a register. Values that only live across the asm don't count, since the
allocator spills them when it needs their registers. The estimate
follows allocateInstruction():
- First all defs get distinct registers, avoiding physreg defs; then all
uses, along with the defs still occupied while the uses are read
(early-clobber and tied defs, see isLiveThroughDef()), get distinct
[26 lines not shown]
[CodeGen] Report an error for a direct inline asm output in memory
An inline asm output returned by value has no memory to write to, yet a
constraint such as "=rm" picks memory, the most general constraint, as
does "=m". SelectionDAG asserted on that ("Can only indirectify direct
input operands!"), and GlobalISel dereferenced a null pointer. Clang
never emits such an output, since it passes the address of a memory
output, but other IR can. Report "cannot handle direct memory outputs
yet for constraint 'm'" instead, like the other inline asm errors there.
Assisted-by: Claude Opus 5.5
[TargetInstrInfo] Fix folding inline asm operands next to tied operands
foldInlineAsmMemOperand() swapped a register operand for the target's
memory operands with MachineInstr::removeOperand(), which asserts when a
later operand is tied, because moving it would break the tie. Inline asm
lists every input after every output, so folding any operand that comes
before another tied pair asserted, e.g. an "rm" input followed by the
input of a "+r" operand, or one of two "+rm" operands. Without
assertions, the moved operands kept stale tie indices. Untie the
operands, rebuild the operand list, and re-tie the remaining pairs at
their new positions.
It also gave up when the register appears in more than one operand, e.g.
one value passed to two "rm" operands, which left the greedy allocator
unable to spill that value at all. Fold every such operand into the
stack slot.
Finally, take MayLoad from the folded operands rather than from every
read of the register: a folded def whose tied use is another virtual
[9 lines not shown]
[DebugInfo] Bound the probe loop in DWARFUnitIndex::getFromHash (#232375)
A CU/TU index with no empty slot and no matching signature made the
lookup iterate forever. Stop after NumBuckets probes, which visit every
slot of a well-formed table.
rdar://186841022
[MIR] Serialize the "foldable" inline asm register operand flag (#229626)
The symbolic MIR syntax for inline asm operand flags dropped
InlineAsm::Flag's RegMayBeFolded bit, which marks a register operand
(from an "rm" constraint) that the register allocator may fold to a
stack slot. Printing MIR and parsing it back therefore silently changed
what the allocator was allowed to do, e.g. with -stop-before and
-run-pass, and a test could only set the bit through a raw numeric flag.
Print the bit as a trailing "foldable", as MachineInstr::print() already
does, and parse it back. A tied use stores its matched operand number in
the same bits, so it never prints the marker.
Co-authored-by: Nick Desaulniers <ndesaulniers at google.com>
Co-authored-by: Bill Wendling <morbo at google.com>
