[NFC][BoundsSafety] Give `counted_by_or_null`, `sized_by`, and `sized_by_or_null` their own documentation (#212877)
Previously the `counted_by`, `counted_by_or_null`, `sized_by`, and
`sized_by_or_null` attributes all pointed at a single `Documentation`
object (`CountedByDocs`). The generated `AttributeReference.rst`
therefore folded all four into one entry whose text described only
`counted_by` used on a C99 flexible array member.
This was misleading because the attributes differ in important ways that
went undocumented:
* `counted_by` counts *elements* whereas `sized_by` counts *bytes*.
* `counted_by` and `sized_by` require a null pointer to have a zero
count/size, while the `_or_null` variants allow a null pointer
regardless of the count or size.
* `counted_by` may be applied to a flexible array member, but
`sized_by`, `counted_by_or_null`, and `sized_by_or_null` apply to
pointers only.
[34 lines not shown]
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment to the
device.
Claude assisted with this patch.
[offload] Pool host and shared allocations (#214754)
Route them through a memory manager, like the device allocations. Also,
move the registration as pinned memory to the plugin site since only the
plugin knows if the corresponding host/shared memory is pinned. Improves
performance of affected allocations by ballpark 1,000x.
[GVN] Limit MemorySSA reaching-value block scans (#217945)
Cap expensive non-local MemorySSA queries at the same 200-block limit
used by MemDep. On an internal workload, this closes the majority of the
compile-time gap between the MemorySSA and MemDep GVN paths.
Runtime remains almost flat.
[Verifier] Diagnose local scope chains that miss their DISubprogram
DILocalScope::getSubprogram() casts every parent to DILocalScope, so a
DILexicalBlock parented to a non-local scope such as a DIFile makes the
verifier abort on that cast before visitDILexicalBlockBase can report
"invalid local scope".
[MLIR][OpenMP] DeclareTargetInterface and DeclareTargetAttr cleanup, NFCI
The changes introduced by this patch are intended to avoid triggering
multiple by name lookups to the "omp.declare_target" attribute every
time any data from the `DeclareTargetInterface` is queried, remove the
need for keeping multiple default values for the same data, clean up
the assembly format and generally improve the usage of the interface.
List of changes:
- `DeclareTargetInterface` only provides `setDeclareTarget` and
`getDeclareTarget` methods to handle the "omp.declare_target"
attribute, maintaining a single canonical way of accessing it.
- `DeclareTargetAttr` provides direct access to the capture clause and
device type enums, rather than wrapping them into an `Attribute`.
These are now mandatory as well. Both changes together make accessing
them more straightforward and simplify the attribute representation.
[MLIR][OpenMP] Add verification for DeclareTargetInterface (#217294)
This patch introduces checks to ensure the "omp.declare_target"
attribute is only attached to `DeclareTargetInterface` operations, it is
always the right type attribute and its properties do not conflict with
the operation they are attached to.
[MLIR][OpenMP] DeclareTargetInterface and DeclareTargetAttr cleanup, NFCI
The changes introduced by this patch are intended to avoid triggering
multiple by name lookups to the "omp.declare_target" attribute every
time any data from the `DeclareTargetInterface` is queried, remove the
need for keeping multiple default values for the same data, clean up
the assembly format and generally improve the usage of the interface.
List of changes:
- `DeclareTargetInterface` only provides `setDeclareTarget` and
`getDeclareTarget` methods to handle the "omp.declare_target"
attribute, maintaining a single canonical way of accessing it.
- `DeclareTargetAttr` provides direct access to the capture clause and
device type enums, rather than wrapping them into an `Attribute`.
These are now mandatory as well. Both changes together make accessing
them more straightforward and simplify the attribute representation.
[MLIR][OpenMP] Add verification for DeclareTargetInterface
This patch introduces checks to ensure the "omp.declare_target"
attribute is only attached to `DeclareTargetInterface` operations, it is
always the right type attribute and its properties do not conflict with
the operation they are attached to.
[AMDGPU] Handle uniform control flow inside divergent (#217977)
Fix PhiIncomingAnalysis::analyze. When it detected an incoming block
that started divergent control flow ending at DefBlock, it tried to add
all blocks in that divergent region to ReachableMap. But the propagation
could be stopped by other blocks (those that did not start divergent
control flow) that were already in ReachableMap. Fix this by doing the
propagation first, before adding the remaining incoming blocks.
[RISCV] Fold vmand of mask comparisons into a masked comparison (#216264)
Fold `vmand(vmsxx(a, b), vmsxx(c, d))` into a single mask-undisturbed
masked comparison in RISCVVectorPeephole:
```
vmsxx ma, va, vb
vmsxx mb, vc, vd
vmand v0, ma, mb
->
vmsxx.vv v0, va, vb
vmsxx.vv v0, vc, vd, v0.t
```
The fold is guarded for correctness and profitability:
- Only comparisons are folded, identified as masked-pseudo-table entries
whose unmasked form has no passthru or policy operand. Mask-dependent
ops like vmsbf.m/viota.m share that shape but are excluded via
[11 lines not shown]
[Clang] Clear outer SubstIndex when normalizing PackIndexingType (#218257)
When we instantiate a fold-expression concept we expand the template
arguments outside of the TreeTransform. However a PackIndexingType
should gain its own index from the index expression, not the outer
SubstIndex.
The bug occurs because we don't support rewrite of PackIndexingType,
which the default transform unexpectedly expands the pattern in the
normalization so that the instantiation picks up the outer SubstIndex
that is set up for a fold expression.
This is identical to 410d6350ed, and in fact
we're unnecessarily transforming the pattern of PackIndexingType
repeatedly.
This is a regression since #161671, so no release note for backporting.
Fixes #218035
Tidy up buildSplitVectorBroadcast
This is used in two cases:
- Split the source operand, but keeping the output VT as is
- Split the output VT as well as the source VT
[clang-tidy] Add llvm-regex check (#207407)
This patch adds a new clang-tidy check, `llvm-regex`, which detects
malformed regular expression patterns passed to `llvm::Regex` when the
pattern can be resolved from a string literal at compile time.
The check validates regex patterns using `llvm::Regex::isValid()` and
emits a diagnostic when an invalid pattern is detected.
It currently handles patterns provided directly as string literals, as
well as string literals stored in immutable, or expected to be used as
such, string-like values such as:
- `const std::string`
- `const char *`
- `const llvm::StringRef`
- `std::string_view`
It also supports detecting invalid patterns stored in class members of
the same types with in-class initializers.
[9 lines not shown]
[NVPTX] Honor !atomic.ignore.denormal.mode on atomicrmw fadd
PTX atom.add has a fixed denormal behavior that the program cannot
control: atom.add.f32 flushes denormals on global memory but not on
shared, and atom.add.f16 never flushes. When that disagrees with the
function's denormal mode, the backend expands the atomic into a CAS loop
so the denormal behavior is preserved.
!atomic.ignore.denormal.mode says the denormal behavior of this
particular atomic does not matter, so use the native instruction even
when it disagrees. This is the same thing -nvptx-allow-ftz-atomics does,
except per-instruction instead of per-compilation, which lets a frontend
opt in only the operations it knows about -- notably CUDA's atomicAdd(),
which is defined in terms of atom.add.
Note that -nvptx-allow-ftz-atomics defaults to true, so the new behavior
is only observable with -nvptx-allow-ftz-atomics=false.
Co-authored-by: Artem Belevich <tra at google.com>
[clang][NVPTX] Emit !atomic.ignore.denormal.mode for CUDA atomics
CUDA's atomicAdd() family is defined in terms of PTX atom.add, whose
denormal behavior is fixed by the hardware. Without any annotation the
backend has to assume the function's denormal mode must be honored and
expands these into CAS loops whenever the two disagree. Mark them with
!atomic.ignore.denormal.mode so the native instruction is used.
That covers the __nvvm_atom_*_add_gen_f builtins that atomicAdd(),
atomicAdd_block() and atomicAdd_system() are written in terms of, plus
C11/C++11 atomics under -fatomic-ignore-denormal-mode and the
[[clang::atomic(ignore_denormal_mode)]] attribute, which requires
teaching the NVPTX target about AtomicOptions.
The condition for when the metadata is meaningful is now shared with the
AMDGPU and SPIR-V targets in addAtomicIgnoreDenormalModeMetadata(). It
takes an AllowHalf flag because whether f16 denormals are observable is
target specific: PTX exposes no FTZ control for f16 operations, so
atom.add.f16 never flushes and the opt-in is meaningful there, whereas
[3 lines not shown]
[IR] Generalize !amdgpu.ignore.denormal.mode into !atomic.ignore.denormal.mode
The !amdgpu.ignore.denormal.mode metadata tells the backend that an
atomicrmw fadd need not honor the function's denormal mode, so a native
atomic instruction whose denormal behavior is fixed in hardware may be
used instead of a CAS loop. Nothing about that is AMDGPU specific: NVPTX
has exactly the same problem with atom.add, whose FTZ behavior depends on
the address space and cannot be controlled.
Promote it to a target independent fixed metadata kind,
!atomic.ignore.denormal.mode, and switch the AMDGPU, SPIR-V and OpenMP
producers and consumers over to it. Document it in LangRef, and point
AMDGPUUsage at that description rather than duplicating it.
Existing IR keeps working: AutoUpgrade renames the metadata on atomicrmw
instructions when parsing textual IR and when materializing bitcode. The
upgrade is deliberately scoped to atomicrmw rather than being applied to
every attachment of that name, since that is the only place the metadata
was ever meaningful. Because bitcode can be materialized one function at
[6 lines not shown]
