[SLP] Use costing to decide between Strided Loads and Compressed Loads (#226540)
For small vectors will small strides, the vectorized load can be
represented as either a stride or a masked compress. Rather than
preferentially choose compression, decide based on the relative costs.
[Clang][counted_by] Capture the field l-value instead of re-emitting the load
To share the pointer field's address with the -fsanitize=array-bounds
check, EmitArraySubscriptExpr emitted the field load itself instead of
going through EmitPointerWithAlignment and ScalarExprEmitter. That
duplicated the scalar member-load path by hand, and the copy missed
several things that path does: the 'align_value' assumption (fixed in
the previous commit), the member's debug location, and the
-fdebug-info-for-profiling pseudo variable for the struct base. So
enabling the sanitizer changed debug info and sample-profile
attribution, even for fields where no bounds check was emitted at all.
Instead, emit the base exactly as before, and have
ScalarExprEmitter::EmitLoadOfLValue record the l-value it loads from
when it matches an expression registered in the new
CodeGenFunction::CurLoadedLValueCapture. The bounds check then uses the
recorded field address. Since the base is emitted the same way with and
without the sanitizer, the load keeps everything the scalar path does.
This also covers a cast between the field load and the subscript (e.g.
[7 lines not shown]
[Clang][counted_by] Don't re-evaluate the base of a counted_by pointer access for -fsanitize=array-bounds
When indexing a 'counted_by' pointer field with -fsanitize=array-bounds,
EmitCountedByBoundsChecking was handed an invalid address and re-emitted
the field's l-value to locate the count. That re-evaluated the struct
base, duplicating its side-effects:
struct p { int count; int *buf __counted_by(count); };
struct p *q;
int f(int i) { return q++->buf[i]; } // 'q' incremented twice
int g(int i) { return get()->buf[i]; } // 'get' called twice
i.e. enabling the sanitizer changed the program's behavior.
When the subscript base is exactly the field load, emit the field's
l-value once, load the pointer from it (as the default case of
EmitPointerWithAlignment does, through EmitLoadOfLValue so that a
'__ptrauth'-qualified field is still authenticated), and pass the
field's address to the bounds check. When there's an intervening cast (e.g.
[6 lines not shown]
[Clang][counted_by] Keep the align_value assumption on the bounds-checked load
With -fsanitize=array-bounds, a subscript of a 'counted_by' pointer field
loads the pointer directly from the field's l-value, so that the bounds
check can reuse the field's address instead of re-evaluating the base.
That load bypassed ScalarExprEmitter, which is where the 'align_value'
assumption for a loaded l-value is emitted. Enabling array-bounds
therefore silently dropped the llvm.assume alignment bundle and, with
-fsanitize=alignment, the alignment-assumption check as well.
Move EmitLValueAlignmentAssumption from ScalarExprEmitter into
CodeGenFunction, so that the direct-load path can share it, and call it
on the loaded pointer.
Assisted-by: Claude Opus 5.5
refactor(codegen): Move shift helpers out of line
Define the shared shift helpers in CodeGenCommonISel.cpp under llvm::detail. Use function_ref callbacks to preserve lazy KnownBits computation, and retain LLVM_ABI for callers in other CodeGen component libraries.
[StructurizeCFG] Convert legacy pass from RegionPass to FunctionPass (#233940)
Convert StructurizeCFGLegacyPass from a RegionPass to a FunctionPass,
just like the NewPM version.
Possible follow-ups:
- Stop using metadata to propagate information from inner to outer
regions.
- Remove RegionPass and RGPassManager from LLVM.
Co-Authored-By: Claude Opus 5.5 <noreply at anthropic.com>
[libc][math] Add integer-only implementation for exp2f. (#233177)
- Correctly rounded to all rounding mode (statically).
- Share core computations with expf integer-only static-rounding version
- Optimize for code size.
- Based on top of https://github.com/llvm/llvm-project/pull/233118
Assisted-by: Gemini for analysis and refactoring.
[libc][math] Optimize integer-only expf and factor out shared utilities. (#233118)
- Optimize `expf_integer_eval` for both code size and performance.
- Factor out the shared range reduction and `2^u` fixed point evaluation
to be used by `exp2f` and `exp10f` integer-only versions.
Assisted-by: Gemini for analysis and refactoring.
[HIP] Fix dynamic initialization with RDC and host profiling (#233655)
HIP compilation with `-fgpu-rdc` and `-fprofile-instr-generate` fails
when a global variable needs initialization before main(). Profiling
and C++ initialization create separate startup lists, but only one
list is allowed.
Collect profiling registration and C++ initialization in the same
startup list so both run before main().
[CIR] Support CoyieldExpr in emitLValue (#232298)
In `CIRGenFunction::emitLValue`, `Expr::CoyieldExprClass` previously
called `errorNYI`. When a coroutine yields an expression whose
`yield_value()` returns an awaiter whose `await_resume()` returns an
lvalue reference (for example, `int &r = co_yield 0;` or `(co_yield 0) =
42;`), Clang evaluates the `CoyieldExpr` as an lvalue. The resume
pointer is spilled and reloaded across `cir.await` to maintain MLIR SSA
dominance in the parent scope.
Fixes #232233.
[clang][Driver] Use the Windows ARM64 defaults for aarch64-unknown-uefi
Match the Windows ARM64 driver defaults for aarch64-unknown-uefi, which
follows the same ABI:
* char is signed, so do not pass -fno-signed-char,
* the frame chain must stay valid even when frame pointers are omitted,
so -fomit-frame-pointer yields -mframe-pointer=reserved rather than
none, as the backend reserves the frame pointer for UEFI as well,
* -msign-return-address selects the B key, the only one the backend
accepts for Windows and UEFI.
Assisted-by: Claude
[clang] Add aarch64-unknown-uefi target
Add the aarch64-unknown-uefi target to clang, building on the backend
support added in the preceding patches. Like x86_64-unknown-uefi it
follows the Microsoft ABI conventions, matching the Windows ARM64 target
which is the only way EFI applications for AArch64 can be built with
Clang/LLVM today:
* add UEFIAArch64TargetInfo: LLP64 types, Microsoft C++ ABI, char*
based va_list and the Win64 calling convention kind,
* use the Windows AArch64 ABI info in CodeGen for UEFI,
* accept aarch64 for *-uefi triples in the driver.
The UEFI toolchain already handles the linker invocation (lld-link with
/subsystem:efi_application), so no further driver changes are needed.
Assisted-by: Claude
[AArch64] Apply the Windows SEH and CRT codegen constraints to UEFI
aarch64-unknown-uefi uses SEH unwind information and has no C runtime
library, so the remaining Windows specific codegen decisions that follow
from this apply to it too:
* do not align loops or basic blocks, since SEH unwind info generation
needs the exact function layout before alignment padding is inserted,
* sign return addresses with the B key by default and reject A key
signing, matching the pac_sign_lr unwind code used by the unwinder;
the branch protection parser defaults to the B key for UEFI as well,
so that -mbranch-protection=pac-ret keeps working,
* do not lower wide shifts to libcalls at -Oz, as those helpers are not
available.
The corresponding Windows tests gain aarch64-unknown-uefi run lines and
the LLVM release notes announce the new target, which completes the
backend side of aarch64-unknown-uefi support.
Assisted-by: Claude
[AArch64] Classify global references on UEFI like Windows COFF
Apply the Windows COFF rules for global references to
aarch64-unknown-uefi, which produces the same PE/COFF images:
* classify dllimport and non-dso-local globals with MO_DLLIMPORT /
MO_COFFSTUB, so extern_weak symbols are referenced through a stub that
can be resolved to zero by the linker,
* keep the existing SelectionDAG, FastISel and GlobalISel handling of
calls and tail calls to extern_weak functions in sync with that,
* default to the PIC relocation model, so jump tables and other
references stay position independent in relocatable PE images,
* lower ifuncs on UEFI as COFF has no support for them.
The extern weak and jump table tests gain aarch64-unknown-uefi run lines,
and a new test checks that the default pipelines lower ifuncs for both
Windows and UEFI.
Assisted-by: Claude
[AArch64] Probe large stack frames with __chkstk on UEFI
Like Windows, aarch64-unknown-uefi has no guard page handling of its
own, so frames larger than the probe size are probed with __chkstk
instead of inline probing, and "no-stack-arg-probe" disables it.
Tests: chkstk.ll, no-stack-arg-probe.ll and win-alloca-no-stack-probe.ll.
Assisted-by: Claude
[AArch64] Address review feedback on UEFI dynamic stack allocation
Include UEFI in the MachineSMEABI lazy ZA save assertion message and add
an -O0 run line for aarch64-unknown-uefi in win-alloca.ll to exercise the
GlobalISel dynamic alloca fallback.
Assisted-by: Claude
[AArch64] Use __chkstk for dynamic stack allocation on UEFI
Register the Windows __chkstk stack probe helper for aarch64-unknown-uefi
and lower dynamic stack allocations through it, including the SME ZA
buffer allocation in LowerFormalArguments. GlobalISel falls back to
SelectionDAG for dynamic allocas on UEFI as it does on Windows.
Tests: win-alloca.ll, the vla in win64_vararg.ll (which now runs in full
on UEFI) and sme-lazy-save-windows.ll.
Assisted-by: Claude
[AArch64] Reserve x18 as the platform register on UEFI
Both the Windows ARM64 ABI, which UEFI follows, and the firmware
ecosystem (EDK2 builds AArch64 with -ffixed-x18) treat x18 as a reserved
platform register. Reserve it by default for aarch64-unknown-uefi, as is
already done for Windows, Darwin, Android, Fuchsia and OHOS.
Assisted-by: Claude
[AArch64] Treat the Win64 calling convention as native on UEFI
Functions using the Win64 calling convention on a non-Windows target
back up x18 and are excluded from tail call optimization, because x18
is the platform register there. On UEFI the Win64 convention is the
native one, so apply the Windows behaviour.
Tests: win64cc-backup-x18.ll gains Windows and UEFI run lines showing
that neither backs up x18 and both tail call.
Assisted-by: Claude
[AArch64] Derive va_list size from the DataLayout
LowerVACOPY and the GlobalISel va_copy legalization each re-derived
sizeof(va_list) and the pointer size from target-triple conditionals,
duplicating AArch64TargetLowering::getVaListSizeInBits. Use the
DataLayout pointer size and getVaListSizeInBits in all three places so
the Darwin/Windows/UEFI vs. AAPCS decision lives in one function.
Also cover llvm.va_copy legalization for Windows and UEFI.
LLM-aided
