[MC] Make pseudo-probe divisions ordering stable (#214803)
**Problem**
`MCPseudoProbeSections::emit` sorts probe divisions only by section
ordinal.
Functions sharing a section will have a nondeterministic `unordered_map`
iteration order, producing *different .pseudo_probe bytes for identical
inputs.*
**Solution**
Use the function symbol name as a stable tie-breaker when section
ordinals match.
Add an MC regression test that reverses pseudo-probe insertion order and
requires byte-identical object files.
[4 lines not shown]
[RISC-V] Use an optional offset operand instead of zero-offset InstAliases
Introduce OptionalMemOffsetAsmOperand, which wraps a memory-offset
operand class into a variant with `IsOptional` set so that a memory
operand written without an offset, e.g. "lb a0, (a1)", parses with a
default offset of 0, and use it for a simm12_lo_optional operand. This
replaces the hand-written "(${rs1})" zero-offset InstAlias that every
load/store-style instruction needed (scalar and FP loads/stores, Zilsd,
jr/jalr and the .insn_i/.insn_s memory forms).
OptionalMemOffsetAsmOperand is somewhat complicated, but this makes it
easier to replace all the other optional zero memory operands which I
will do in follow-up commits, removing all the InstAlias duplication.
This change was assisted by AI.
Pull Request: https://github.com/llvm/llvm-project/pull/210901
[BOLT] Page out .dwo files
Split-DWARF inputs at big binaries scale ship 100+ GiB of .dwo
files. BOLT opened a fair number of them during readDebugInfo, putting
a lot of pressure on the OS memory management: mmap'd reads always
populate the page cache; with every .dwo mapped at once those pages
accumulated, refaulted, and registered as memory pressure that got the
process oomd-killed.
Now, .dwo page-cache pages are reclaimed as soon as BOLT is done with
each file: madvise(MADV_PAGEOUT) on the live mapping, then
posix_fadvise(POSIX_FADV_DONTNEED) once it is unmapped. Controlled by
-drop-dwo-page-cache, OFF by default, as it is unlikely upstream
will be processing gigantic sets of dwo files.
[BOLT] Create and release .dwo DWARF contexts incrementally
BOLT opened a DWARFContext for every .dwo during
readDebugInfo and kept them all alive until teardown. On large
split-DWARF targets that is tens of GiB held resident through
emission, the point of peak RSS.
Make the DWOCUs map a lazily-populated cache instead:
* Use the newly added DWARFUnit::clearDWO()/hasDWO() to directly
manage DWARFUnit's DIE caching mechanism.
* BinaryContext::getDWOCU() opens a context on demand (keyed off a
stable DWOId -> skeleton CU map).
* Release contexts as soon as they are done with: all of them at the
end of readDebugInfo, and per-bucket at the DWARF rewrite merge
point.
* Remove DWOCUs map, which became redundant and whose purpose can
now be served by the new id-to-skeleton map, and then fetching
the split CU from the skeleton via getNonSkeletonUnitDIE().
[5 lines not shown]
[libclc][CMake] Add cmake cache files (#201480)
This simplifies cmake configuration without need to pass verbose
variables in cmake command line. Settings in cache file can still be
overriden by passing `-D` flags.
Update `libclc/README.md` to document the usage.
[DebugInfo] Add DWARFUnit::clearDWO()
Add DWARFUnit::clearDWO() so a skeleton unit can drop the DWO context
it owns without being destroyed itself. Also add DWARFUnit::hasDWO()
to answer if that skeleton CU is currently caching a DWO context, so
users can easily look it up.
For example, BOLT opened a DWARFContext for every .dwo during
readDebugInfo and kept them all alive until teardown. On large
split-DWARF targets that is tens of GiB held
resident. clearDWO()/hasDWO() expose to users DWARFUnit's caching
capacity, allowing them to spontaneously drop the cache/look it
up/re-load it for memory management.
[CIR] Forward-declare mlir::ModuleOp in CIRGenerator.h (#214767)
Add missing forward decl for `CIRGenerator.h` to make it self
contained..
Found this while trying to create an Clang FrontEnd action in an
out-of-tree project.
Went with a forward decl since its already done in the file +
recommendations from
https://llvm.org/docs/CodingStandards.html#minimal-list-of-includes
[BOLT] Parse .eh_frame CFI programs on demand to reduce memory
BOLT read the entire .eh_frame up front via DwCtx->getEHFrame(), which
parses and caches the CFI instruction program of every CIE/FDE in the
binary for the whole run. On a large binary, this dominated
file-object discovery: CFIProgram::parse accounted for ~6.5 GB and the
cached DWARFDebugFrame ~6.9 GB of live memory. Yet the CFI programs
are only consumed in CFIReaderWriter::fillCFIInfoFor, and only for the
functions BOLT actually disassembles. discoverFileObjects itself needs
nothing but each FDE's address and range for function-boundary checks.
Here we parse .eh_frame for its index only, and decode each function's
CFI program on demand, lazily, only for the functions that really need
it. In a large binary, DWARFDebugFrame::parse drops from 6922.2 MB
to 587.6 MB, the residual being the lightweight FDE/CIE index (entries
without instruction programs), and readSpecialSections falls from
7078.7 MB to 738.6 MB on the tested binary for which BOLT's RSS is
about 80-120GB.
[DebugInfo] Allow .eh_frame CFI programs to be parsed lazily
BOLT read the entire .eh_frame up front via DwCtx->getEHFrame(), which
parses and caches the CFI instruction program of every CIE/FDE in the
binary for the whole run. On a large binary, this dominated
file-object discovery: CFIProgram::parse accounted for ~6.5 GB and the
cached DWARFDebugFrame ~6.9 GB of live memory (from 5 to 10% of total
anon peak RSS).
This new interface allows DebugInfo's users to optionally parse CFIs
on demand, only when necessary. On BOLT, this is an important lever
to manage memory utilization when processing large binaries.
[CAS][test] Disable plugin CAS tests with HWASAN (#214881)
HWASan does not tag the globals of a dlopen'ed library on Linux/glibc,
so libCASPluginTest.so traps on the first access to one of its own
globals (https://github.com/llvm/llvm-project/issues/57206).
Disables PluginCASTest.isMaterialized and the PluginCAS instantiation of
CASTest, both introduced in #213331, under HWASAN. This matches what
a0e402d41fad did for DynamicLibraryTest for the same underlying issue.
[BOLT] Skip data-hole filling unless data reordering is enabled
BinaryContext::postProcessSymbolTable unconditionally called
fixBinaryDataHoles(), which walks every allocatable section and, for
each gap in its address space, either grows a zero-sized data symbol
or creates a synthetic "HOLEat" BinaryData (plus an MCSymbol and
GlobalSymbols/BinaryDataMap entries). This machinery was introduced
(0e4d86bf, 2017) for one purpose: to give static data reordering
(-reorder-data) a movable object covering every byte of a section. It
has no other consumer.
On a large binary, these synthetic objects are live from
buildFunctionsCFG through the end of the run and, at the RSS peak
(during debug info rewriting), fixBinaryDataHoles accounted for 1669
MB (2.5%) of peak RSS -- memory spent entirely for a feature that is
off by default.
Gate fixBinaryDataHoles() (and the zero-sized-symbol validation loop
that presumes it ran) on a non-empty opts::ReorderData, keeping
generateSymbolHashes() unconditional.
[lldb] Download to a unique temporary path in the SymStore locator (#214632)
The download path was derived only from the key and the PDB name, so two
lookups (from different threads) potentially raced the same file path.
Avoid this by creating using a unique suffix.
Assisted-by: Claude
[lldb] Report a corefile whose only image is a platform binary (#214634)
found_platform_binary was tested but never assigned, so the early return
it guards was dead and LoadCoreFileImages reported failure for a
corefile whose only image a Platform plugin had already taken care of.
The caller reads that as "no binary found in the metadata" and goes on
to scan low memory for a UUID that has no reason to be there.
While here, stop assuming the module has an object file before asking it
for its sections. A binary located by an external symbol server is
turned into a Module without checking that it parses, and
LoadBinaryInTarget guards the same dereference on the other branch.
Assisted-by: Claude
[lldb] Add AddressSpaceInfo and ProcessAddress utility classes (#206370)
This is part of larger effort to support address spaces in lldb
https://discourse.llvm.org/t/rfc-address-spaces-support-in-lldb/91222/
This PR introduces the definition of AddressSpace and ProcessAddress
classes, which are foundational for next PRs
Stack:
1. #206370 (this PR) - the classes
2. #214088 ProcessAddress adoption (NFC)
3. #214089 Generic address space support
[BOLT] Key GlobalSymbols on MCContext-owned names to reduce memory
BinaryContext::registerNameAtAddress registers every symbol name
twice. It first calls MCContext::getOrCreateSymbol(Name), which
interns the name in MCContext's symbol table (the MCSymbol owns the
string via its table entry). It then also stored the name in the
GlobalSymbols map, which was a StringMap<BinaryData *>. StringMap owns
its keys, so each global name was duplicated: one copy in MCContext
and a second copy in GlobalSymbols. Both grow with the number of
symbols and, for large binaries with long mangled names, this
duplication is a meaningful source of memory use during file
object discovery.
This change makes MCContext the single owner of these name strings and
have GlobalSymbols merely reference them. GlobalSymbols becomes a
DenseMap<StringRef, BinaryData *> keyed on the MCContext-owned name
(MCSymbol::getName() of the symbol just created/looked up). No string
is copied into the map: each entry is a fixed-size (StringRef,
pointer) pair regardless of name length. Lookups (getBinaryDataByName,
[10 lines not shown]
[clang][StaticAnalyzer] Add enclosing Decl information to bug reports of RawPtrRef(LocalVars|Member)Checker (#214102)
The `RawPtrRefLocalVarsChecker` and `RawPtrRefMemberChecker` forgot to
call `Report->setDeclWithIssue()` for some bug reports. Without the
call, the HTML reports miss the enclosing Decl and have hash collision
on distinct diagnostics.
The added
`clang/test/Analysis/Checkers/WebKit/html-diag-dedup-members.cpp`
example is a reproducer of this kind of issue we observed in WebKit.
In addition, refactored `GetEnclosingDeclContextSignature` for simplicity and supporting
ObjC Decls.
rdar://183700416
Assisted-by: Claude sonnet
---------
Co-authored-by: Balázs Benics <benicsbalazs at gmail.com>
[BOLT] Compute local code section names lazily to reduce memory
Every BinaryFunction eagerly materialized two std::string members at
construction: CodeSectionName (".local.text.<name>") and
ColdCodeSectionName (".local.cold.text.<name>"), built by appending
the full function name to a fixed prefix. On ELF the name is the
(often large, mangled) symbol name, which is already owned by the
function's MCSymbol. So every function stored two extra heap copies of
its name -- for all functions, whether or not they were ever emitted
into a per-function section. During discoverFileObjects on a large
binary, this eager construction (buildSectionName -> Twine::str)
accounted for 1-2% of RSS (1770MB).
Here we make both members std::optional<std::string>, left empty by
default, and recompute the default name on demand in
getCodeSectionName() from the function's name. Only functions with an
explicitly assigned section name (via
setCodeSectionName/setColdCodeSectionName -- mostly short constants
such as ".text"/".text.cold" set during reordering, plus the
[2 lines not shown]
[BOLT] Reduce NameResolver memory usage during file object discovery (#212625)
NameResolver used a StringMap<uint64_t> to count duplicate names.
StringMap owns its keys, so every uniquify()/getUniquifiedNameCount()
query allocated a full copy of each (potentially large, mangled) symbol
name. During discoverFileObjects on a large binary, this string-key
duplication accounted for ~2 GB (1 to 2% of RSS) of allocations in
StringMap::try_emplace_with_hash -> StringMapEntry::create ->
allocateWithKey.
Replace the StringMap with a DenseMap<pair<uint64_t,uint64_t>, uint64_t>
keyed by a 128-bit xxh3 hash of the name. No string is ever stored: each
distinct name costs a fixed-size entry regardless of length. A 128-bit
hash makes collisions effectively impossible, so the per-name counts
(and therefore the generated 'Name/ID' unique names) are identical to
the string-keyed map and remain reproducible to match profile (fdata)
names.
Also clear the map at the end of discoverFileObjects, since the resolver
[2 lines not shown]
[SandboxVec][NFC] Lit tests cleanup (#214794)
This patch organizes the lit tests by introducing a directory structure,
separating them by pass, and renaming them when needed. This will become
increasingly useful for navigating through the lit tests as we keep
adding more tests and more passes.
This also drops boilerplate.ll as it was one of the early tests that
checked the vectorizer's boilerplate and is no longer needed.
[llvm][YAMLTraits] Avoid nondeterministic iteration in endMapping()
Use MapVector to avoid iterating over items in nondeterministic order.
Partially addresses https://github.com/llvm/llvm-project/issues/214872
[lldb] Pass a Target to Platform::GetSharedModule (#214633)
PlatformDarwinKernel::GetSharedModuleKernel unconditionally dereferenced
the process to reach the target's debug file search paths. The Process *
is the wrong thing to be threading through Platform::GetSharedModule.
Refactor Platform::GetSharedModule to take a `Target &` and get the
process from the target if we really need it.
[llvm][LoopVectorize] Avoid nondeterministic iteration
The Transforms/LoopVectorize/AArch64/cmp_cost.ll test fails under
LLVM_RERERSE_ITERATION, because iterating over a SmallPtrSet is
nondeterministic. Addresses part of #214872.
[SLP] Allow non-power-of-2 VF in tryToVectorizeList (#206259)
Non-power-of-2 VF support (-slp-vectorize-non-power-of-2) was wired into
the `load/store/reduction` paths but not into `tryToVectorizeList`, so
`buildvector/insertelement` seeds like <15 x half> were split into
power-of-2 pieces instead of a single vector op.
Consult `isAllowedNonPowerOf2VF()` when computing `MaxVF` and in the
inner
full-vector guard so supported widths are tried. No change unless
`-slp-vectorize-non-power-of-2` is enabled.
https://godbolt.org/z/1b8GebPEW
