[AArch64][GlobalISel] Fix invalid subregister copies for truncating stores (#213935)
The custom truncating store selector always created a subregister copy
matching the memory type. When storing an i16 value to i8 both types
used GPR32, producing an invalid copy:
```
*** Bad machine code: Invalid subregister index for virtual register ***
- function: truncstores
- basic block: %bb.0 (0xb1a58e8738c0)
- instruction: %2:gpr32 = COPY %val16.sub_32:gpr32
- operand 1: %val16.sub_32:gpr32
Register class GPR32 does not support subreg index sub_32
LLVM ERROR: Found 1 machine code errors.
```
This code path should only be used for FPRs. The existing RegBankSelect
avoids this by inserting a G_ANYEXT from i16 to i32, allowing the
imported pattern to match. It was exposed by the type-based
RegBankSelect prototype in #199040.
Assisted-by: codex
[NFC][DWARF] Prefer addSectionLabel (#214239)
This is a follow-up to a comment on #213128.
addSectionDelta was only called in sites within if statements that would
branch on whether or not split DWARF is enabled and then branching to
either addSectionDelta or addSectionLabel. Instead we can just make
addSectionLabel call addSectionDelta internally if we're in a DwoUnit to
make things simpler.
This is obviously NFC for most cases. For macro info, we were previously
branching on useSplitDWARF which can differ, but only in the case we are
emitting into the skeleton compile unit, which we should not be doing
for macro info.
[GVNSink] Do not sink @llvm.threadlocal.address (#214582)
Sinking this intrinsic with different arguments requires the use of a
phi node which violates verifier assumptions about the intrinsic. So we
mark it as unable to have operands replaced with variables.
Fixes #214565
Update `atan2` matrix tests (#214550)
This PR updates the `atan2` matrix sema error test to cover the full set
of argument/type error cases (mirroring `pow_mat-errors.hlsl`). It also
renames two tests to match the `<intrinsic>_mat-<suffix>` convention
that other intrinsics follow.
[RISCV] Increase cost of vmv.x.s and vfmv.f.s (#214481)
A change landed that enabled SLP vectorization in more places, but it
caused a 16% regression in 508.namd_r on the spacemit-k3:
https://github.com/llvm/llvm-project/issues/214417
The root cause is that we're emitting two vfmv.f.s now which are
generally slow due to the vector->scalar domain crossing:
https://godbolt.org/z/dMo55zPMa
Reciprocal throughput of vmv.x.s/vfmv.f.s is 6 times higher than
vmv.s.x/vfmv.s.f on the spacemit-k3 and 2.5 higher on the P870 according
to https://camel-cdr.github.io/rvv-bench-results/
This PR increases the cost of a vmv.x.s/vfmv.f.s from 1->2, which is
just enough to prevent unprofitable SLP vectorization and fixes the
regression on 508.namd_r.
It also has the bonus side effect of preventing loop vectorization where
[3 lines not shown]
[Flang] Coarray allocation, update error for pointer component #193829 (#194651)
The TODO message in `verify()` was not completly displayed, so it has
been replaced by `emitErrorOp`.
In addition, the test in `ConvertVariable` has been updated by adding
pointer direct component case.
Fix #193829
[VPlan] Append recipes created via builder to worklist
The previous PR appended the top most created recipe to the worklist, and this PR extends it to any other nested recipes that were created, similar to InstCombine.
This removes the header mask in a good few more places on RISC-V as measured on SPEC CPU 2017, e.g. for the following loop:
```c
long f(const int *p, const int *q, long n) {
long a = 0, b = 0;
for (long i = 0;; i++) {
if (p[i] && q[i]) { a += i; b += i; }
if (i + 1 == n) break;
}
return a + b;
}
```
Before:
[49 lines not shown]
[VPlan] Process simplifyRecipes in a worklist
This brings simplifyRecipes further in line with InstCombine, and asides from unlocking more simplifications it also helps avoid spurious test churn whenever passes are moved around simplifyRecipes.
For now just push the new recipe onto the worklist, not its users.
This uses a post order traversal so we maintain the same simplification order as before.
I've gone through and checked every simplification we do is a canonicalisation that converges, and I checked on llvm-test-suite + SPEC CPU 2017 in various configurations that we don't hit any cycles.
[VPlan] Make simplifyRecipe more like InstCombine
Most combines in simplifyRecipe RAUW a value, but not all of them erase the old recipe.
Unify them and bring it in line with InstCombine by having it return a VPValue, which simplifyRecipes can then call RAUW with, and automatically erase the old recipe.
Similarly to InstCombine, combines that modify a recipe should return the same recipe.
