AMDGPU/GlobalISel: Explicitly widen scalar to i32 for load and store (#210095)
Affects f16 and bf16. Earlier, they were widened to f32 and s32 respectively.
The actual error was the artifact combiner creating a copy between f32/i32
which fails in the machine verifier. Maybe we could create a bitcast there.
However i32 is more efficient for us and matches well with how argument
lowering keeps f16 and bf16 in i32 copies to/from physical registers.
Also starting from f16 store, G_STORE %0(f16), %1(p1) :: (store (f16),
and doing widen scalar to 32 bit type, i32 makes more sense since store
will store 16 least significant bits G_STORE %0(i32), %1(p1) :: (store (f16)
compared to G_STORE %0(f32), %1(p1) :: (store (f16), which looks incorrect if
we assume input was really in f32 format.
[clang][docs] Fix include path in InternalsManual (#209458)
The paths `include/Basic/StmtNodes.td` and `include/AST/Expr*.h` in the
Clang Internals Manual are incorrect.
[VectorCombine] load-widening.ll - add test coverage for all 4 x86-64 cpu levels and aarch64 endian test coverage (#210690)
Removed x86 endianess tests which makes no sense (and are likely to misbehave) and add equivalent test coverage to aarch64
Noticed while reviewing #209775
[ExpandMemCmp][RISCV] Expand memcmp/bcmp for aligned pointers on strict-align targets (#209738)
`RISCVTTIImpl::enableMemCmpExpansion` previously disabled `memcmp`/`bcmp`
expansion whenever the target lacks unaligned scalar memory access. This is
too conservative: when both pointers are statically known to be
sufficiently aligned (e.g. `bcmp(ptr align 8, ptr align 8, 32)`), only
naturally aligned loads are needed, which are fine on strict-align
targets.
`ExpandMemCmp` now keeps a candidate load size only if, given the known
common alignment of the two pointers, the access is naturally aligned or
the target reports it via `allowsMisalignedMemoryAccesses`; otherwise it
falls back to the libcall. The query tests whether the access is
*allowed* (not *fast*), so it is a no-op for targets that already allow
unaligned access (X86, AArch64, ...).
The known alignment now also folds in the `align` attributes on the call
arguments (`CallBase::getParamAlign`), not just the pointer value.
[5 lines not shown]
AMDGPU/GlobalISel: Fix legalizer lowering for G_EXTRACT/INSERT_VECTOR_ELT (#210094)
Use LLT::integer in bit twiddling lowering for extract/insert vector element.
[Darwin][ASan] Strip MTE-tags for inlined shadow translations (#204827)
https://github.com/llvm/llvm-project/pull/166453 stripped these tags
when the runtime performed the mem-to-shadow translation, but did not
account for ASan's inline translations. When an MTE-tagged address gets
translated, the tag bits are right-shifted too, resulting in a very high
address that faults when accessed.
This patch strips the MTE-tag bits before applying the translation on
Apple platforms.
rdar://180032780
[analyzer] Fix _BitInt support & casting behavior for Z3 symbolic execution (#210525)
Forces symbolic cast to be enabled for z3 symbolic execution, and
switches away from Ctx.getTypeSize for getting the bit width of integral
types.
The current patch might be a bit problematic for z3 cross-check, as this
relies on symbolic integer cast to be always on. I am not sure if
turning on ShouldSupportSymbolicIntegerCasts would cause issues for the
supported range-based solver, so I only kept it turned on when
AnalysisConstraintsOpt == Z3ConstraintsModel.
Assisted-by: Codex
AMDGPU/GlobalISel: Use extended LLTs in AMDGPUCombinerHelper
Here we also had to change comparisons to extended LLTs to avoid
matching bfloats. In the old scalar version, before the switch to extended
LLTs, s16 was treated as f16 and bf16 was combined as if it were f16.
AMDGPU/GlobalISel: Use integer as MMO type for loads and stores lowering
We could get away with just the type from MMO in most cases, but MMO splitting
creates MMO with LLT::scalar and we prefer integer.
AMDGPU/GlobalISel: Explicitly widen scalar to i32 for load and store
Affects f16 and bf16. Earlier, they were widened to f32 and s32 respectively.
The actual error was the artifact combiner creating a copy between f32/i32
which fails in the machine verifier. Maybe we could create a bitcast there.
However i32 is more efficient for us and matches well with how argument
lowering keeps f16 and bf16 in i32 copies to/from physical registers.
Also starting from f16 store, G_STORE %0(f16), %1(p1) :: (store (f16),
and doing widen scalar to 32 bit type, i32 makes more sense since store
will store 16 least significant bits G_STORE %0(i32), %1(p1) :: (store (f16)
compared to G_STORE %0(f32), %1(p1) :: (store (f16), which looks incorrect if
we assume input was really in f32 format.
[X86] combineShiftRightLogical - fold srl(vecreduce_umax(x),bw-1) as MOVMSK signbit reduction (#210281)
VectorCombine may have folded:
icmp_eq(vecreduce_or(splatsign(x)),0) --> icmp_sgt(vecreduce_umax(x),-1)
which DAG folds to:
srl(vecreduce_umax(x),bw-1).
This match attempts to lower:
srl(vecreduce_umax(x),bw-1) --> icmp_ne(movmsk(x),0) "any_of negative"
srl(not(vecreduce_umax(x)),bw-1) --> icmp_eq(movmsk(x),0) "none_of negative"
The correct fix would be to improve vecreduce_or costs to prevent
VectorCombine doing this, but that change is far too big to be merged
into 23.x - so I've created the narrow backend fix.
Fixes #209714
AMDGPU/GlobalISel: Fix legalizer lowering for G_EXTRACT/INSERT_VECTOR_ELT
Use LLT::integer in bit twiddling lowering for extract/insert vector element.
[AArch64] NFC: Factor out code from FP_TO_INT (SVE). (#207200)
This just moves out some of the SVE lowering code from
LowerVectorFP_TO_INT into a separate function, so that we can reuse that
in LowerVectorFP_TO_INT_SAT.
AMDGPU/GlobalISel: Use extended LLTs in AMDGPUCombinerHelper
Here we also had to change comparisons to extended LLTs to avoid
matching bfloats. In the old scalar version, before the switch to extended
LLTs, s16 was treated as f16 and bf16 was combined as if it were f16.
