[MC][X86] Reintroduce aligned instruction bundling (#175830)
Aligned bundling partitions instructions into fixed-size, naturally
aligned groups called bundles and guarantees that no instruction
crosses a bundle boundary, giving the instruction stream a single
canonical decoding. It is a building block for software-based fault
isolation: control flow cannot jump into the middle of an instruction
to manufacture a different, unchecked sequence, and when combined with
masking of indirect branch targets it constrains control flow to a
statically verifiable set of locations.
The previous target-independent implementation was removed in #148781,
which simplified MC by eliminating per-fragment BundlePadding, the
virtual emitInstToData, and BundleGroupBeforeFirstInst. This change
reimplements the feature in the X86 backend on top of the existing
MCBoundaryAlignFragment infrastructure added for branch alignment,
keeping the generic MC surface smaller:
* AsmParser parses .bundle_align_mode, .bundle_lock and .bundle_unlock
[36 lines not shown]
[Flang][OpenMP][OpenMPIRBuilder] Implement module scope declare target use rewrite mechanism (#212920)
During lowering of declare target'd variables we generate new global
variables for device that replace the use of the pre-existing global
variable. In Flang we currently rewrite this for each target region, but
that's not enough to cover indirect use cases inside of declare target
functions which can be imported into the module and utilised inside of a
target region. This PR tries to extend the scope of the rewriting to the
module than a per target region rewrite.
It does so by creating a mechanism where we can register globals for
replacement which will trigger on finalization of the OMPIRBuilder. This
is required as due to the ordering of lowering for MLIR, where we
generate the replacement global at the beginning of the module before
any uses have been generated, effectively meaning we cannot replace the
uses at that point. So, we defer the replacement to the OMPIRBuilder as
there is no deferral mechanism directly in the OpenMP MLIR lowering.
The alternative might be to rebind the global maps in ModuleTranslation
[7 lines not shown]
[CIR] Fix record layout for a union with no storage type (#213591)
A union whose CIR type ends up with no members keeps its whole size in
its
padding field, and `UnionType::getTypeSizeInBits` returned early in
exactly that
case, before reaching the padding. A union need not look empty in the
source to
land there: a lone zero-length bitfield is dropped during lowering,
leaving the
same no-storage state.
A record embedding such a union was then laid out wrong. In an unpacked
record
`insertPadding` pads whenever the end of the members placed so far,
rounded up
to the next member's alignment, falls short of that member's offset, so
a union
measuring zero earns a pad the AST layout does not have. In C++,
[22 lines not shown]
[DAGCombine] Fold (select_cc (select cond, x, y), x, a, b, eq) to (select cond, a, b) (#199688)
(select_cc (select cond, x, y), x, a, b, eq) which could be simplified
to (select cond, a, b)
[CodeGen] Fix -fsanitize=array-bounds for __sized_by / _or_null pointers
`EmitCountedByBoundsChecking()` assumed a CountAttributedType is always
a __counted_by pointer. That isn't true, there are four versions of the
attribute:
* `__counted_by`: Already handled correctly.
* `__counted_by_or_null`: Incorectly handled.
* `__sized_by`: Incorreclty handled.
* `__sized_by_or_null`: Incorreclty handled.
In particular:
* __sized_by / __sized_by_or_null: the loaded bound is a byte count, but the
element index was compared against it directly, so an access was only
flagged once the index exceeded the byte count -- missing out-of-bounds
accesses for a pointee larger than one byte. Scale the index to bytes
('index * sizeof(element)') before comparing. counted_by counts elements
and is unchanged; a void (or otherwise zero-sized) pointee uses the GNU
[13 lines not shown]
[CodeGen] Fix __builtin_dynamic_object_size for __sized_by / _or_null pointers
`emitCountedByPointerSize()` assumed a CountAttributedType is always
a __counted_by pointer. That isn't true, there are four versions of the
attribute:
* `__counted_by`: Already handled correctly.
* `__counted_by_or_null`: Incorectly handled.
* `__sized_by`: Incorreclty handled.
* `__sized_by_or_null`: Incorreclty handled.
In particular:
* __sized_by / __sized_by_or_null: the attribute argument is a byte count,
but the object size was computed as count * sizeof(*ptr), over-reporting by
the element size for any pointee larger than one byte. Use the count
directly for the byte-counting variants.
* __counted_by_or_null / __sized_by_or_null: a null pointer describes no
[20 lines not shown]
clang: Use TargetID parsing from AMDGPUTargetParser (#209845)
We had grown 2 parallel parsing implementations for
triple+gpu name+feature flag target ID strings. Mostly
eliminate the redundant clang version.
Co-authored-by: Claude (Opus 4.8)
[CodeGen][NFC] Split __sized_by tests into their own file and rename test cases
In future patches the coverage of the __counted_by family attributes is
going to be increased. To help with this patch refactors the existing
test file.
1. Split `__sized_by` tests into their own file. In later commits files
will be added for each attribute so it makes sense for each attribute
to have its own file.
2. Replace `testN` test case names with human readable descriptions. Not
all test cases that will be added in the future will apply to all
attributes. If we kept on using the `testN` naming convention it
would leave odd gaps in the test numbering because we try to keep
what a test case tests consistent between files (i.e. `testN` would
roughly test the same thing but with a different attribute). Using
named test cases completely avoids this.
[clang][darwin] armv6m Firmware crashes spilling the TLS wrapper (#213595)
Thread local storage isn't universally supported on all architectures.
Only enable it for the ones that are known to support it.
rdar://183822457
[LoopUnroll] Fix freq accuracy calculations (#213762)
This problem was reported at
<https://github.com/llvm/llvm-project/pull/182405#issuecomment-5165268733>
for the case of very large loop probabilities.
The biggest issue is that, when using linear and quadratic equations to
determine loop latch probabilities, asserts introduced by PR #182405 to
verify the accuracy of the resulting loop body frequency can fail.
Another issue is that iterations introduced by PR #182404 and PR #182405
terminate upon achieving a desired accuracy, but they can iterate longer
than necessary, wasting time achieving higher accuracy than desired.
This patch fixes the accuracy calculations to use relative differences
instead of absolute differences. It updates existing tests that reveal
the impact on the N>2 uniform case. Its adds new tests to cover the N=1,
N=2, and N>2 fast cases.
Consolidate metadirective lowering TODO tests
Group related block, loop data-environment, iteration-variable, and unsupported-replacement cases into split-file tests. This keeps each diagnostic isolated while reducing the number of TODO test files.
[SPIR-V] Legalize wide-vector atan2 by splitting first (#213341)
fixes #213340
This was simple fix we just had to change the order in which we were
doing the splitting and widdening.
This change prioritize splitting G_FATAN2 vectors wider than four
elements before attempting power-of-two widening, which G_FATAN2 does
not support.
Add float and half coverage for vector widths 6, 8, 9, 12, and 16.
AMDGPU: Validate generic processor features in TargetParser emitter
Perform some initial validation that the feature set of generic
targets is consistent with the set of covered targets. For now, this
only validates the frontend exported list so it should be good for
catching missed builtins that ought to be accepted on the generic.
In the future arbitrary features should be validated, but this is
complicated by workaround features and size features which need to
clamp to the common monimum.
Co-authored-by: Claude (Claude-Opus-4.8)
AMDGPU: Export the TargetParser feature bitset
Previously this bitset was only used to populate the feature
name string map used by clang. Eventually this will replace
the current bitmask integer. AArch64 already has a similar
interface.
Co-authored-by: Claude (Claude-Opus-4.8)
AMDGPU: Tablegenerate TargetParser feature sets (#212945)
Traditionally we maintained 2 parallel feature mechanisms,
one in clang (later moved to TargetParser), with largely
mirrored subtarget features defined in the backend. Start
directly taking feature information from the backend and putting
it into TargetParser. This is still in a compromise mid-migration
state. We still have both the legacy "ArchAttr" bitfield integer,
plus a new AMDGPUFeatureBitset field stored in the table, which
isn't yet exported.
For the moment, the new bitset is only used to populate the
feature string name map, which is the big maintainability win.
This also lists an explicit subset of exported features to
avoid churn.
Co-authored-by: Claude (Claude-Opus-4.8)
[libc] Fix elf_proxy header generation (#213737)
Followup to fix the generated proxy header after #211428.
Assisted-by: Automated tooling, human reviewed.
[lldb][minidump] write the memory after an unreadable page when saving minidump (#212641)
**Issue**
An internal failing test found a latent bug in lldb's save-core
(minidump writer). When it saved a memory range that had an unreadable
page in it, it:
- stopped at that page and threw away the readable memory after it,
Result: We couldnot get the stack traces from the minidump. in the below
example the **current logic is bailing out at the 6th region and not
writing other 70 regions.**
```
[satyajanga at devgpu011.eag2 ~/fbsource/fbcode (eacbfddefa|remote/master)]$ lldb
(lldb) file /data/users/satyajanga/fbsource/buck-out/v2/art/fbcode/55005549ebc49982/sand/tests/__Coro__/Coro
Current executable set to '/data/users/satyajanga/fbsource/buck-out/v2/art/fbcode/55005549ebc49982/sand/tests/__Coro__/Coro' (x86_64).
(lldb) b coro.cpp:44 Breakpoint 1: where = Coro`::co_main() + 197 at coro.cpp:44, address = 0x00000000002335a5
(lldb) r
Process 3374177 launched: '/data/users/satyajanga/fbsource/buck-out/v2/art/fbcode/55005549ebc49982/sand/tests/__Coro__/Coro' (x86_64)
[47 lines not shown]
[offload] Use pinned memory for KLE
Reduce kernel launch latency by using the fast path "pinned host memory
-> device memory" for submitting the kernel launch environment.
Claude assisted with this patch.
[flang][OpenMP] Lower DO and SIMD variants in metadirectives
A standalone metadirective and its associated DO are sibling PFT evaluations,
so a selected loop replacement cannot directly reuse ordinary OpenMP loop
lowering. Runtime selection must also preserve exactly one copy of the loop in
each reachable branch. Temporarily associate the evaluations while lowering
to support DO, SIMD, and DO SIMD replacements without losing or duplicating the
ordinary fallback loop.
For example:
```fortran
!$omp metadirective &
!$omp& when(user={condition(flag)}: do) &
!$omp& otherwise(nothing)
do i = 1, n
a(i) = i
end do
```
[49 lines not shown]
[CIR]Add RecordType to our dense-array optimization in lowering (#213725)
This is an issue in AMDGPUAsmParser.cpp self-build, we have a lot of
record elements (~360k+!) in an array that causes us to have this TU be
near-never-ending(hour+). Classic codegen compiles this sub-minute on my
machine. With this patch, we are only about a 30% increase in time.
Note: Claude wrote much of the tests after I got through every exception
I could think of. I think this covers everything, and I hope there is no
missing coverage.
[UBSan] Fix assertion failure in EmitCheckedInBoundsGEP for constant-… (#191278)
…folded overflowing offsets
EmitGEPOffsetInBytes has two paths: for fully constant GEPs it subtracts
pointer values and always returns OffsetOverflows=false, but for
non-constant GEPs it iterates operands using checked arithmetic. The
assertion in EmitCheckedInBoundsGEP assumed a constant TotalOffset
implies no overflow, conflating the two paths.
In the non-constant path, the offset can be entirely constant-folded
(constant index * constant element size) while the GEP itself remains
non-constant (runtime base pointer). If that arithmetic overflows
intptr_t, we get a constant TotalOffset with OffsetOverflows=true,
triggering the assertion. This is easily hit on 16-bit targets like
MSP430 with realistic struct array indices.
Remove the assertion. The existing codegen already handles this
correctly: the constant OffsetOverflows=true propagates into the check
[4 lines not shown]
