[AArch64] Add Neoverse V3AE scheduling info for v264 frecpe (#219115)
This was missing, falling back to the default information. It should be
like the others.
[CIR] Migrate AssumeBundleKind, AtomicFetchKind and AsmFlavor off IntegerAttr
AssumeBundleKind, AtomicFetchKind and AsmFlavor were I32EnumAttrs with the
default genSpecializedAttr, so each generated an IntegerAttr subclass with no
dialect spelling of its own. Unlike the other CIR operation enums, these
three are reached through hand-written parsers and printers rather than a
plain argument reference, so each one needs checking separately.
Each enum now sets genSpecializedAttr = 0 and gains a CIR_EnumAttr wrapper.
cir.atomic.fetch references $binop from its declarative format, so that one
gains an `enum()` wrapper.
The other two need no code changes. printAssumeBundle and parseAssumeBundle
are already typed on cir::AssumeBundleKindAttr, and the wrapper keeps that
class name, so custom<AssumeBundle> still compiles and behaves the same.
InlineAsmOp::print streams getAsmFlavor(), which returns the enum and uses
the operator<< that -gen-enum-decls emits, not the attribute.
[2 lines not shown]
[CIR] Drop lang_address_space's custom parenthesized attribute format
CIR_LangAddressSpaceAttr overrode its assembly format to
`(` custom<AddressSpaceValue>($value) `)`, so the standalone attribute
printed as `#cir<lang_address_space(offload_global)>`. The parentheses
defeated the dialect's `#cir.mnemonic<...>` syntax and forced the generic
fallback. It now uses the bracketed CIR_EnumAttr default and prints as
`#cir.lang_address_space<offload_global>`.
The override existed to make the enum read as `lang_address_space(x)` inside
`!cir.ptr` and `cir.global`. Those spellings are unaffected. Both go through
parse/printAddressSpaceValue on MemorySpaceAttrInterface in CIRTypes.cpp,
which prints the keyword and parentheses itself. The identically named
attribute-level pair in CIRAttrs.cpp was only reachable from the deleted
format, so it goes away.
[CIR] Derive lowering attr names from cppClassName, not the def name
CIRLoweringEmitter built its CXX_ABI_ALWAYS_LEGAL_ATTRS entries by calling
GetOpCppClassName on an attribute record, which splits the TableGen def name
at the first underscore. That works only while every def happens to be named
CIR_<CppClassName>Attr. When one is not, the emitter silently writes an
`isa<cir::Whatever>` for a class that does not exist, so the failure lands as
a compile error in generated code with no hint at the cause. The constraint
was real enough that CIREnumAttr.td had to document it.
Attributes already carry the authoritative name in cppClassName, and
GenerateAttrToValueVisitor two functions down was reading it correctly.
Factor that out as GetAttrCppClassRef and use it for the enum attributes and
the plain CIR attributes as well, which also drops the hardcoded `cir::` in
favour of the dialect's own cppNamespace. GetOpCppClassName stays for
operations, whose names are derived that way.
The def name is now free, so the paragraph in CIREnumAttr.td warning that it
is not goes away.
[4 lines not shown]
[CIR] Hoist genSpecializedAttr into the CIR enum base classes
Every one of the 33 CIR enum definitions set `genSpecializedAttr = 0`, and
none wanted the generated IntegerAttr subclass. That made the line pure
repetition, and worse, made "a CIR enum is never backed by IntegerAttr" a
convention each new enum had to remember rather than something the dialect
enforces.
Set it once on CIR_I32EnumAttr, CIR_I64EnumAttr and CIR_I32BitEnumAttr,
right next to the cppNamespace they already fix, and drop all 33 per-def
copies. Most of those defs carried nothing else, so they collapse from a
braced body to a plain `;`.
This is the same structure SPIRV_I32Enum, Tosa_I32Enum, GPU_I32Enum and
OpenMP's enum base class use. The `!if(genSpecializedAttr, ...)` expressions
in EnumAttrInfo resolve the same whether the bit is set on the def or
inherited.
The comment on CIR_SourceLanguage goes too. It explained the bit as being
[4 lines not shown]
[CIR] Delete the unused cir::VisibilityAttr
CIR_VisibilityAttr had no users. `cir.global` and `cir.func` carry visibility
as `DefaultValuedProp<EnumProp<CIR_VisibilityKind>>:$global_visibility`, a
property rather than an attribute, so nothing ever built or printed the
attribute and no test mentioned it.
Its only consumer was CIRGenModule::getGlobalVisibilityAttrFromDecl, itself
never called. That in turn was the only caller of
getGlobalVisibilityKindFromClangVisibility, so all three go away together.
The nearly identical getCIRVisibilityKind survives and does have a caller.
It maps clang::Visibility rather than clang::VisibilityAttr::VisibilityType,
so a future caller wiring visibility into the property has a starting point,
and git history has the rest.
CIR_VisibilityKind stays. It is the enum the property is built from.
Incidentally this removes one of the two attributes overriding their
assembly format to a bare `$value`.
[CIR] Print cir.global's TLS model without naked angle brackets
`cir.global` printed `tls_model = <tls_dyn>`. Those angle brackets were the
leftover delimiters of `#cir.tls_model<tls_dyn>` after the operation printer
stripped the dialect prefix and mnemonic. The same enum on the module printed
`cir.default_tls_model = #cir.tls_model<tls_dyn>`, so one enum had two
unrelated spellings and the per-global one was not something anyone would
write by hand.
Wrapping the argument in the `enum` directive prints the symbolic value on
its own:
cir.global external tls_model = tls_dyn @a = #cir.int<5> : !s32i
The standalone attribute is unchanged. invalid-tls.cir now sees a single
diagnostic from the operation's enum parser instead of two from the
attribute parser, and its input uses an invalid keyword rather than invalid
delimiters so it still exercises enum validation.
[CIR] Give the cleanup kind a proper standalone attribute spelling
CleanupKindAttr overrode its assembly format to a bare `$value` so that
`cir.cleanup.scope` would print `cleanup all` rather than `cleanup <all>`.
The price was that the attribute had no readable standalone form. It fell
back to the generic dialect-attribute syntax and printed as
`#cir<cleanup_kind all>`.
The `enum($attr)` operation format directive removes the tradeoff. The
attribute keeps the bracketed format CIR_EnumAttr gives it by default, and
`cir.cleanup.scope`, `cir.while` and `cir.for` ask for the bare symbolic
value instead, so the attribute now spells `#cir.cleanup<all>`.
The mnemonic drops the `_kind` suffix that the C++ enum class name carries.
`kind` says nothing a reader of `#cir.cleanup<all>` needs.
Operation syntax is unchanged. The negative test in invalid-loop-cleanup.cir
now sees a single diagnostic from the operation's enum parser, which also
lists the valid keywords, in place of the two the attribute parser emitted.
[CIR] Drop dead ceremony around the CIR enum attributes
Five things that no longer earn their place, all in the CIR enum attribute
machinery.
CIR_CleanupKindAttr carried three of them. Its cppClassName was a no-op.
CIR_EnumAttr derives from AttrDef<dialect, enumInfo.className>, and AttrDef
already defaults cppClassName to name # "Attr", which is exactly
CleanupKindAttr. Its skipDefaultBuilders plus hand-written AttrBuilder
existed only to default $value to CleanupKind::All. All three callers
(cir.while, cir.for and parseCleanupKind) pass an explicit kind, so the
default was unreachable while the generated builders stayed suppressed for
nothing. And its summary and description restated the name, overriding the
enum's own "cleanup kind" with a longer synonym. EnumAttr inherits both, so
deleting the overrides improves the generated dialect docs. Only the
isNormal / isEH / isNormalAndEH helpers were doing any work, and they stay.
CIR_TLSModelAttr's summary restated the name the same way. Its description
is informative, so only the summary goes.
[4 lines not shown]
[CIR] Migrate the FPClassTest bit enum and unquote its flags
cir.is_fp_class printed its flags inconsistently. One of the ten single-bit
values came out bare, as in `fcSNan`, while every group value and every
multi-flag combination came out quoted, as in `"fcInf"` and
`"fcSNan|fcNegInf"`. That split comes from I32BitEnumAttr setting
printBitEnumQuoted, which EnumAttr.td marks as kept only for backwards
compatibility. The bit makes the printer quote anything that is not a single
bit.
Clearing printBitEnumQuoted and using the `enum` directive selects the
separator-aware parser and printer, so every value now spells unquoted:
cir.is_fp_class %x, fcInf : (!cir.float) -> !cir.bool
cir.is_fp_class %x, fcSNan|fcNegInf : (!cir.float) -> !cir.bool
The enum also drops its specialized IntegerAttr in favour of a CIR_EnumAttr
wrapper, giving it a standalone spelling of `#cir.fp_class<fcSNan|fcNegInf>`.
The wrapper is named CIR_FPClassTestAttr rather than after the
[5 lines not shown]
[CIR] Migrate GlobalLinkageKind, CallingConv and SideEffect off IntegerAttr
GlobalLinkageKind, CallingConv and SideEffect were I32EnumAttrs with the
default genSpecializedAttr, so each generated an IntegerAttr subclass with no
dialect spelling of its own. Each now sets genSpecializedAttr = 0 and gains a
CIR_EnumAttr wrapper, and cir.global wraps $linkage in `enum()`.
cir.func and cir.call print all three through hand-written code, but they
stream stringifyGlobalLinkageKind(getLinkage()) and friends. Those take the
enum rather than the attribute, so the sites are unchanged. The
DefaultValuedAttr defaults for $linkage, $calling_conv and $side_effect keep
taking the same C++ expressions.
Operation syntax is unchanged, so no test needed updating.
GlobalLinkageKind's wrapper is spelled `#cir.linkage<internal>`, dropping
both the `global_` prefix and the `_kind` suffix that the enum class name
carries for C++ reasons.
[CIR] Record why the CUDA registration attribute parses itself
hasCustomAssemblyFormat with no explanation invites the question of whether a
declarative assemblyFormat would do, and the answer is not obvious from the
parameter list. It would not. The three flags print as presence-only keywords
and parse in any order, while an optional group anchored on a `bool`
parameter parses and prints a value, so the group would spell `extern true`.
MLIR has no presence-only flag for `bool` in an attribute format, unlike
UnitAttr in an operation format. struct(params) does round-trip, but spells
the attribute `<device_side_name = "i", kind = Variable, isExtern = true>`
instead of `<i, Variable, extern>`.
NFC.
[CIR] Migrate MemOrder and SyncScopeKind off IntegerAttr
MemOrder and SyncScopeKind, the two enums the atomic operations share, were
I32EnumAttrs with the default genSpecializedAttr, so each generated an
IntegerAttr subclass. The operations printed them symbolically, but the
attributes themselves had no dialect spelling and showed up in an attribute
dictionary as plain integers.
Both enums now set genSpecializedAttr = 0 and gain CIR_EnumAttr wrappers, so
they spell `#cir.mem_order<seq_cst>` and `#cir.sync_scope<system>`, and
cir.load, cir.store, cir.atomic.fetch, cir.atomic.xchg,
cir.atomic.cmp_xchg, cir.atomic.test_and_set, cir.atomic.clear and
cir.atomic.fence wrap the arguments in `enum()` to keep the bare keyword.
The `Arg<>`-wrapped declarations carrying operand documentation move to the
wrapper too, keeping their descriptions.
`enum()` accepts an optional-group anchor, so the `syncscope` and `atomic`
groups on cir.load and cir.store are unaffected. Operation syntax is
unchanged and no test needed updating.
[CIR] Drop the redundant suffix from the inline kind mnemonic
inline_kind was the one CIR enum attribute mnemonic still repeating what its
C++ enum class name already says. Every other mnemonic drops that suffix, so
keeping it left a single inconsistency in the file.
The attribute now spells `#cir.inline<always_inline>`. The operation argument
is still named inline_kind, since that is the accessor name and unrelated to
the mnemonic, so the printed form reads
`inline_kind = #cir.inline<always_inline>`.
While here, the enum's summary becomes "inline kind" instead of the camelCase
"inlineKind". That summary is what the generated dialect docs now show, since
CIR_InlineKindAttr no longer overrides it with a restatement of its own name.
25 CHECK lines change across four test files. Nine of them are in
CodeGen/AArch64/sme-inline-callees-streaming-attrs.c, which is
`REQUIRES: aarch64-registered-target` and so unsupported in an X86-only
build. The substitution there is identical to the one verified by
CIR/CodeGen/callsite-inline-attributes.cpp and CIR/IR/inline-attrs.cir.
[CIR] Migrate seven operation enums off IntegerAttr
CastKind, DynamicCastKind, CmpOpKind, ComplexRangeKind, InitCatchKind,
CaseOpKind and AwaitKind were I32EnumAttrs with the default
genSpecializedAttr, so each generated an IntegerAttr subclass. Operations
printed them symbolically because the format referenced the argument
directly, but the underlying attribute had no dialect spelling at all. In an
attribute dictionary or a generic-form dump it was a plain integer, so
`cir.cast bitcast` was stored as `kind = 1 : i32`.
Each enum now sets genSpecializedAttr = 0 and gets a CIR_EnumAttr wrapper,
which is the pattern EnumAttr.td documents, and the operations wrap the
argument in `enum()` to keep the bare keyword. The attributes gain a real
spelling, e.g. `#cir.cast<bitcast>`.
Operation syntax is unchanged, hence no test churn. enum-attrs.cir covers the
new standalone spellings, since nothing else in the test suite exercises
them. It also covers tls_model, which had no standalone case of its own.
[16 lines not shown]
[Support][Caching] Don't use `mmap` in `localCache`, on NFS
When the cache directory is on an NFS, we may hit a SIGBUS signal
when there is contention on the cache.
This happens typically with MPI jobs running on an NFS mounted $HOME
directory.
If we execute through the regular `read()` path, we also fail with a
`Stale file descriptor` error. But at least this path is properly handled
and we can continue without using the cache.
To achieve this, we pass `OnNFS` to `IsVolatile` and `RequiresNullTerminator`,
which makes `shouldUseMmap` return `false`.
[mlir] Disable some tests when the execution engine is disabled (#220741)
It's possible to have a scenario where the native target is enabled, but
the execution engine is not, e.g. when trying to avoid an issue with the
execution engine.
In that case, we need to disable the tests that depend on: mlir-runner,
mlir_async_runtime, mlir-capi-execution-engine-test,
mlir-capi-global-constructors-test, mlir_c_runner_utils,
mlir_runner_utils and mlir_float16_utils.
[AArch64][GlobalISel] Create uzp in shuffle(v, undefined) situations (#220535)
Stacked PR: 2/3. Preceded by
https://github.com/llvm/llvm-project/pull/219226.
In SDAG, the aarch64-isel phase checks if vector shuffles can be
expressed as uzps. To do this, it checks shuffles of type shuffle(v, v),
and shuffle(v, undefined).
GlobalISel previously only checked shuffles of type shuffle(v, v). Add a
check for the situation where one of the operands is undefined.
Notes:
An uzp de-interleaves the even and odd elements of two vectors.
e.g: `uzp <0, 1, 2, 3>, <4, 5, 6, 7> => <0, 2, 4, 6, 1, 3, 5, 7>`.
A uzp1 takes the bottom half of the result (aka. the even-indexed
elements)
e.g: `uzp1 <0, 1, 2, 3>, <4, 5, 6, 7> => <0, 2, 4, 6>`.
A shuffle is an LLVM IR generic opcode that takes 2 vectors, and places
[7 lines not shown]
X86: Handle undef push operands in call frame optimization
Avoid unchecked getVRegDef call, which will fail for undef
operands. The register-push fallback also dropped the undef flag when
rebuilding the store as a PUSH.
Co-Authored-By: Claude <noreply at anthropic.com> (Claude Opus 4.8)
[AArch64][PAC] Emit tail calls more efficiently
It may be required to insert explicit checks that LR was authenticated
successfully before performing a tail call. Previously, such checks were
inserted when expanding the TCRETURN pseudo instructions, if the stack
frame was created by the particular function. This did not take into
account the shrink-wrapping optimization, though.
This commit introduces a separate `PAUTH_CHECK_LR` pseudo instruction.
A conservative heuristic is implemented that drops the completely useless
checks. Furthermore, it moves the remaining checks to the shrink-wrapping
epilogue (if any), but only if that doesn't hurt any regular return code
paths.
[AArch64][PAC] Precommit tests on efficient LR checks before tail calls
When performing a tail call with pac-ret hardening enabled, depending on
the performance vs. security trade-off, it may be required to insert an
explicit check that the LR register contains a valid address (that is,
the authentication succeeded), as unlike a regular call, a tail call does
not dereference LR right away.
When shrink-wrapping optimization is in effect, this might be expensive
both in terms of time complexity and code size. This commit adds several
tests demonstrating the existing codegen behavior.
[AArch64][PAC] Prevent PAUTH_EPILOGUE from overwriting live registers
With shrink-wrapping, it is possible for PAUTH_EPILOGUE to be inserted
in the middle of the function where X15/X16/X17 may be alive and must
not be clobbered.
This commit implements ad-hoc spilling of the scratch registers used by
PAUTH_EPILOGUE to other GPRs. If no such registers is available at the
insertion point, an explicit compiler error is triggered.
