[AArch64] Combine a load into GPR followed by a copy to FPR to a load into FPR directly through MIPeepholeOpt

[mingmingl]

Summary

Example pattern before:

%1:gpr64 = LDRXui ...
%2:fpr64 = COPY %1:gpr64

Pattern after:
%1:gpr64 = LDRDui

The pattern show up when elements are read from arrays, and moved into floating
point registers for calculation.

[mingmingl]

Implementation-wise, use MachineRegisterInfo::replaceRegWith to replace all uses of Reg; in this way, debug uses of the register will be updated as well.

Besides, add a dedicated MIR test for this optimization, so that machine instruction update (including MachineInstr::memoperands, etc) are visible.

[kazu]

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

108

Declare this with const. Alternatively, turn the definition into a function in the anonymous namespace outside the class.

109

Declare this with const. Alternatively, turn the definition into a function in the anonymous namespace outside the class.

110

Declare this with const. Alternatively, turn the definition into a function in the anonymous namespace outside the class.

222

Transformation before sounds a bit strange. I'd suggest Before transformation or simply`Before`.

227

Likewise.

238

Reduce indentation with an early return like so:

if (!SrcOperand.isReg() || !MRI->hasOneNonDBGUse(SrcOperand.getReg()))
  false;

251–252

Reduce indentation with an early return like so:

if (!isFPRRegister(DstRegClass) || !isGPRegister(SrcRegClass) ||
    !hasSameNumberOfBits(DstRegClass, SrcRegClass))
  return false;

286

Remove this as you have an identical line about 50 lines above.

302–328

Do these two blocks of code trigger at all? I tried ninja check-llvm with assert(0) placed in these two blocks, but neither one triggered. It may be tedious to have a test case for every case in the first switch statement, but I'd suggest at least one test case for each block in the second switch statement -- partly for correctness and partly for coverage.

332

Reduce indentation with an early return like so:

if (NewOpCode == -1)
  return false;

You might consider placing this check between the two switch statements because the second switch statement doesn't change NewOpCode.

335

Remove this is to fix the comma splice.

[dmgreen]

Do you have details on the motivating case for this? I was under the impression that it was usually handled by ISel, but the tests certainly suggest it isn't always the case. There will be times for DAG ISel which cross basic-blocks, which it won't be able to match. A number of the other test changes look like artifacts from how tablegen patterns are defined.

The AArch64MIPeepholeOptimizer seems to be a rich source of bugs. Every time we make a change in it, it takes several attempt to get it correct.

[mingmingl]

In D130100#3673897, @dmgreen wrote:

Do you have details on the motivating case for this? I was under the impression that it was usually handled by ISel, but the tests certainly suggest it isn't always the case. There will be times for DAG ISel which cross basic-blocks, which it won't be able to match. A number of the other test changes look like artifacts from how tablegen patterns are defined.

This is a good point. I understand that DAG ISel is probably a better place to handle this combine, to avoid fiddling with various kinds of load/store operations.

As shown in the .mir test case, the machine instructions are from the same basic block (not cross basic blocks, thereby code sinking won't help)

Existing UIntToFPROLoadPat or VecROLoadPat in AArch64InstrInfo.td won't catch this; in particular, the test case uses load with unsigned offset (not register offset), so not caught by VecROLoadPat.

Debug log of instruction-selection shows it's very likely due to

1. lack of type legalization (from int64 to v1i64 or something similar)
2. a pattern named VecUILoadPat, similar to what VecROLoadPat does for load with register offset.
   
   I'll look into the details and see what's missing.

btw for simplicity, the patch is about load, but I believe the same optimization opportunity exists for store operations.

The AArch64MIPeepholeOptimizer seems to be a rich source of bugs. Every time we make a change in it, it takes several attempt to get it correct.

Thanks for pointing out a different place to solve the same problem. I share the concern that general memory operations are hard to rewrite correctly, and even if they are correct, it's better to make them happen earlier in the pipeline.

[mingmingl]

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

302–328

I didn't see them being triggered from the affected test case from a cursory look (i.e., most affected load operations in the affected test case come without indices)

It definitely makes sense to add the test coverage, even if the fix would be in another place eventually.

[mingmingl]

An update:

1. For the motivating use case (around pmull intrinsic and instruction), I sent out D130548.
2. The general issue (missed combination of {load + copy -> load} exists.

For 2), peephole-opt should have folded the load (code) given AArch64InstrInfo.cpp considers this pattern (mentioned in comment). Going to see why it's not executed and update back.

[mingmingl]

In D130100#3673897, @dmgreen wrote:

Do you have details on the motivating case for this? I was under the impression that it was usually handled by ISel, but the tests certainly suggest it isn't always the case. There will be times for DAG ISel which cross basic-blocks, which it won't be able to match. A number of the other test changes look like artifacts from how tablegen patterns are defined.

The AArch64MIPeepholeOptimizer seems to be a rich source of bugs. Every time we make a change in it, it takes several attempt to get it correct.

It's true that most of the cross-register-back copies are introduced in the (multi-step) ISel pass. May I hear some feedback on whether the general idea of combining {ldr + copy} into {ldr} is good to go, given that 1) fixing ISel one-by-one could be laborious 2) peephole-opt does more clean up (say the {ldr+copy} -> ldr is exposed after ISel, as a result of multiple passes execution)

Also update the implementation by bailing out on illegal conditions

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

302–328

Leaving this open before go/no-go discussion with peephole converges.

[mingmingl]

Update implementation to bail out on illegal conditions, major ones include

1. ldr is not safe to move
2. load fold barriers exist between ldr and copy
3. ldr or copy are a part of MI bundle.

[dmgreen]

I worry about the potential for bugs in any patch in the AArch64MIPeepholeOptimizer. And loading makes that more complex. The altered tests look like they are either intrinsics like the pmull64 that was recently changed or strange types (<32 x i1>, <16 x i12>) that don't come up a lot in practice. The global-isel change in dp1.ll is a missed fold that is probably better handled in global-isel. The irregular type tests also show it only handling straight loads, not instructions like ld1 { v1.s }[1], [x12]. The current model is to try and handle almost all combines during ISel so that they can all work together in one place.

If this comes up in more places it might be worth it. From the examples in the tests I'm not so sure though. Do you have any other examples where this helps?

[mingmingl]

In D130100#3696135, @dmgreen wrote:

I worry about the potential for bugs in any patch in the AArch64MIPeepholeOptimizer. And loading makes that more complex. The altered tests look like they are either intrinsics like the pmull64 that was recently changed or strange types (<32 x i1>, <16 x i12>) that don't come up a lot in practice. The global-isel change in dp1.ll is a missed fold that is probably better handled in global-isel. The irregular type tests also show it only handling straight loads, not instructions like ld1 { v1.s }[1], [x12]. The current model is to try and handle almost all combines during ISel so that they can all work together in one place.

I totally agree with the fact that loading/storing would make things more complex here, that global-isel should be able to handle dp1.ll; strange types (<16 x i2><32xi1>) looks more interesting but I doubt if it's a result of multiple issues.

If this comes up in more places it might be worth it. From the examples in the tests I'm not so sure though. Do you have any other examples where this helps?

I'm convinced it's better to handle affected test cases in ISel (yet still made changes, hope to make it closer to correct);

I don't observe other examples where this helps -> problem hunting starts from pmull test case (llvm/test/CodeGen/AArch64/aarch64-pmull2.ll in D131047)

I'll abandon this patch for now; thanks for your timely feedback on the series of changes!

llvm/lib/Target/AArch64/AArch64MIPeepholeOpt.cpp

 //    If AArch64's 32-bit form of instruction defines the source operand of
 //    ORRWrs, we can remove the ORRWrs because the upper 32 bits of the source
 //    operand are set to zero.
 //
 //===----------------------------------------------------------------------===//
 
 #include "AArch64ExpandImm.h"
 #include "AArch64InstrInfo.h"
+#include "AArch64MachineFunctionInfo.h"
+#include "AArch64RegisterInfo.h"
 #include "MCTargetDesc/AArch64AddressingModes.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SetVector.h"
 #include "llvm/CodeGen/MachineDominators.h"
 #include "llvm/CodeGen/MachineLoopInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/Register.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "aarch64-mi-peephole-opt"
 
 namespace {
 
 struct AArch64MIPeepholeOpt : public MachineFunctionPass {
struct AArch64MIPeepholeOpt : public MachineFunctionPass {
   template <typename T>
   bool visitADDSUB(unsigned PosOpc, unsigned NegOpc, MachineInstr &MI);
   template <typename T>
   bool visitADDSSUBS(OpcodePair PosOpcs, OpcodePair NegOpcs, MachineInstr &MI);
 
   template <typename T>
   bool visitAND(unsigned Opc, MachineInstr &MI);
   bool visitORR(MachineInstr &MI);
+  bool visitCopy(MachineInstr &MI);
+
+  bool hasSameNumberOfBits(const TargetRegisterClass *FPRegClass,
+                           const TargetRegisterClass *GPRRegClass);
+  bool isGPRegister(const TargetRegisterClass *RC);
+  bool isFPRRegister(const TargetRegisterClass *RC);
   bool runOnMachineFunction(MachineFunction &MF) override;
 
   StringRef getPassName() const override {
     return "AArch64 MI Peephole Optimization pass";
   }
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
static bool splitBitmaskImm(T Imm, unsigned RegSize, T &Imm1Enc, T &Imm2Enc) {
   if (!AArch64_AM::isLogicalImmediate(NewImm2, RegSize))
     return false;
 
   Imm1Enc = AArch64_AM::encodeLogicalImmediate(NewImm1, RegSize);
   Imm2Enc = AArch64_AM::encodeLogicalImmediate(NewImm2, RegSize);
   return true;
 }
 
+bool AArch64MIPeepholeOpt::hasSameNumberOfBits(
+    const TargetRegisterClass *FPRegClass,
+    const TargetRegisterClass *GPRRegClass) {
+  return (FPRegClass == &AArch64::FPR32RegClass &&
+          GPRRegClass == &AArch64::GPR32RegClass) ||
+         (FPRegClass == &AArch64::FPR64RegClass &&
+          GPRRegClass == &AArch64::GPR64RegClass);
+}
+
+bool AArch64MIPeepholeOpt::isFPRRegister(const TargetRegisterClass *RC) {
+  return (RC == &AArch64::FPR32RegClass || RC == &AArch64::FPR64RegClass ||
+          RC == &AArch64::FPR128RegClass);
+}
+
+bool AArch64MIPeepholeOpt::isGPRegister(const TargetRegisterClass *RC) {
+  return (RC == &AArch64::GPR32RegClass || RC == &AArch64::GPR64RegClass);
+}
+
 template <typename T>
 bool AArch64MIPeepholeOpt::visitAND(
     unsigned Opc, MachineInstr &MI) {
   // Try below transformation.
   //
   // MOVi32imm + ANDWrr ==> ANDWri + ANDWri
   // MOVi64imm + ANDXrr ==> ANDXri + ANDXri
   //
return splitTwoPartImm<T>(
             .addReg(SrcReg)
             .addImm(Imm0);
         BuildMI(*MBB, MI, DL, TII->get(Opcode.second), NewDstReg)
             .addReg(NewTmpReg)
             .addImm(Imm1);
       });
 }
 
+bool AArch64MIPeepholeOpt::visitCopy(MachineInstr &MI) {
+  // Transformation before:
+  //
+  // %2:gpr64 = LDRXui %1:gpr64common, 1 :: (load (s64) from %ir.3)
+  // %3:fpr64 = COPY %2:gpr64
+  //
+  // Transformation after:
+  // %3:fpr64 = LDRDui %1:gpr64common, 1 :: (load (s64) from %ir.3)
+  const MachineOperand &DstOperand = MI.getOperand(0);
+  const MachineOperand &SrcOperand = MI.getOperand(1);
+  if (!DstOperand.isReg())
+    return false;
+
+  Register DstReg = DstOperand.getReg();
+  if (!DstReg.isVirtual())
+    return false;
+
+  if (SrcOperand.isReg() && MRI->hasOneNonDBGUse(SrcOperand.getReg())) {
+    Register SrcReg = SrcOperand.getReg();
+    if (!SrcReg.isVirtual())
+      return false;
+
+    auto *SrcMI = MRI->getUniqueVRegDef(SrcOperand.getReg());
+    if (!SrcMI)
+      return false;
+
+    int64_t NewOpCode = -1;
+
+    const TargetRegisterClass *DstRegClass = MRI->getRegClass(DstReg);
+    const TargetRegisterClass *SrcRegClass = MRI->getRegClass(SrcReg);
+    if (isFPRRegister(DstRegClass) && isGPRegister(SrcRegClass) &&
+        hasSameNumberOfBits(DstRegClass, SrcRegClass)) {
+      switch (SrcMI->getOpcode()) {
+      case AArch64::LDRXui:
+        NewOpCode = AArch64::LDRDui;
+        break;
+      case AArch64::LDRWui:
+        NewOpCode = AArch64::LDRSui;
+        break;
+      case AArch64::LDRXroW:
+        NewOpCode = AArch64::LDRDroW;
+        break;
+      case AArch64::LDRXroX:
+        NewOpCode = AArch64::LDRDroX;
+        break;
+      case AArch64::LDRWroW:
+        NewOpCode = AArch64::LDRSroW;
+        break;
+      case AArch64::LDRWroX:
+        NewOpCode = AArch64::LDRSroX;
+        break;
+      case AArch64::LDRWpre:
+        NewOpCode = AArch64::LDRSpre;
+        break;
+      case AArch64::LDRXpre:
+        NewOpCode = AArch64::LDRDpre;
+        break;
+      case AArch64::LDRWpost:
+        NewOpCode = AArch64::LDRSpost;
+        break;
+      case AArch64::LDRXpost:
+        NewOpCode = AArch64::LDRDpost;
+        break;
+      }
+
+      Register DstReg = DstOperand.getReg();
+      Register ReplacedReg;
+      switch (NewOpCode) {
+      default:
+        break;
+      case AArch64::LDRDui:
+      case AArch64::LDRSui: {
+        BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(NewOpCode),
+                DstReg)
+            .add(SrcMI->getOperand(1))
+            .add(SrcMI->getOperand(2))
+            .setMemRefs(SrcMI->memoperands())
+            .setMIFlags(SrcMI->getFlags());
+        ReplacedReg = SrcMI->getOperand(0).getReg();
+        break;
+      }
+      case AArch64::LDRDroW:
+      case AArch64::LDRDroX:
+      case AArch64::LDRSroW:
+      case AArch64::LDRSroX: {
+        BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(NewOpCode),
+                DstOperand.getReg())
+            .add(SrcMI->getOperand(1))
+            .add(SrcMI->getOperand(2))
+            .add(SrcMI->getOperand(3))
+            .setMemRefs(SrcMI->memoperands())
+            .setMIFlags(SrcMI->getFlags());
+        ReplacedReg = SrcMI->getOperand(0).getReg();
+        break;
+      }
+      case AArch64::LDRSpre:
+      case AArch64::LDRDpre:
+      case AArch64::LDRSpost:
+      case AArch64::LDRDpost: {
+        BuildMI(*MI.getParent(), MI, MI.getDebugLoc(), TII->get(NewOpCode))
+            .add(SrcMI->getOperand(0))
+            .add(DstOperand)
+            .add(SrcMI->getOperand(2))
+            .add(SrcMI->getOperand(3))
+            .setMemRefs(SrcMI->memoperands())
+            .setMIFlags(SrcMI->getFlags());
+        ReplacedReg = SrcMI->getOperand(1).getReg();
+        break;
+      }
+      }
+
+      if (NewOpCode != -1) {
+        assert(ReplacedReg.isValid() &&
+               "ReplacedReg expected to be valid when NewOpCode is chosen");
+        // Replace 'ReplacedReg' with 'DstReg', this is mainly to update all
+        // debug uses of 'ReplacedReg' to the correct value.
+        MRI->replaceRegWith(ReplacedReg, DstReg);
+        SrcMI->eraseFromParent();
+        MI.eraseFromParent();
+        return true;
+      }
+    }
+  }
+
+  return false;
+}
+
 bool AArch64MIPeepholeOpt::visitORR(MachineInstr &MI) {
   // Check this ORR comes from below zero-extend pattern.
   //
   // def : Pat<(i64 (zext GPR32:$src)),
   //           (SUBREG_TO_REG (i32 0), (ORRWrs WZR, GPR32:$src, 0), sub_32)>;
   if (MI.getOperand(3).getImm() != 0)
     return false;
 
for (MachineInstr &MI : make_early_inc_range(MBB)) {
                                           {AArch64::SUBXri, AArch64::SUBSXri},
                                           MI);
         break;
       case AArch64::SUBSXrr:
         Changed = visitADDSSUBS<uint64_t>({AArch64::SUBXri, AArch64::SUBSXri},
                                           {AArch64::ADDXri, AArch64::ADDSXri},
                                           MI);
         break;
+      case AArch64::COPY:
+        Changed = visitCopy(MI);
+        break;
       }
     }
   }
 
   return Changed;
 }
 
 FunctionPass *llvm::createAArch64MIPeepholeOptPass() {
   return new AArch64MIPeepholeOpt();
 }

llvm/test/CodeGen/AArch64/arm64-subvector-extend.ll

; CHECK-NEXT:    ret
   ret <8 x i64> %r
 }
 
 ; Extends of vectors of i1.
 
 define <32 x i8> @zext_v32i1(<32 x i1> %arg) {
 ; CHECK-LABEL: zext_v32i1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [sp, #64]
+; CHECK-NEXT:    ldr s1, [sp, #64]
 ; CHECK-NEXT:    fmov s0, w0
-; CHECK-NEXT:    ldr w9, [sp]
-; CHECK-NEXT:    ldr w10, [sp, #8]
-; CHECK-NEXT:    fmov s1, w8
 ; CHECK-NEXT:    ldr w8, [sp, #72]
+; CHECK-NEXT:    ldr w9, [sp]
 ; CHECK-NEXT:    mov.b v0[1], w1
-; CHECK-NEXT:    movi.16b v2, #1
+; CHECK-NEXT:    ldr w10, [sp, #8]
 ; CHECK-NEXT:    mov.b v1[1], w8
 ; CHECK-NEXT:    ldr w8, [sp, #80]
+; CHECK-NEXT:    movi.16b v2, #1
 ; CHECK-NEXT:    mov.b v0[2], w2
 ; CHECK-NEXT:    mov.b v1[2], w8
 ; CHECK-NEXT:    ldr w8, [sp, #88]
 ; CHECK-NEXT:    mov.b v0[3], w3
 ; CHECK-NEXT:    mov.b v1[3], w8
 ; CHECK-NEXT:    ldr w8, [sp, #96]
 ; CHECK-NEXT:    mov.b v0[4], w4
 ; CHECK-NEXT:    mov.b v1[4], w8

 ; CHECK-NEXT:    ret
   %res = zext <32 x i1> %arg to <32 x i8>
   ret <32 x i8> %res
 }
 
 define <32 x i8> @sext_v32i1(<32 x i1> %arg) {
 ; CHECK-LABEL: sext_v32i1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [sp, #64]
-; CHECK-NEXT:    fmov s0, w0
+; CHECK-NEXT:    ldr s0, [sp, #64]
+; CHECK-NEXT:    fmov s1, w0
+; CHECK-NEXT:    ldr w8, [sp, #72]
 ; CHECK-NEXT:    ldr w9, [sp]
+; CHECK-NEXT:    mov.b v1[1], w1
 ; CHECK-NEXT:    ldr w10, [sp, #8]
-; CHECK-NEXT:    fmov s1, w8
-; CHECK-NEXT:    ldr w8, [sp, #72]
-; CHECK-NEXT:    mov.b v0[1], w1
-; CHECK-NEXT:    mov.b v1[1], w8
+; CHECK-NEXT:    mov.b v0[1], w8
 ; CHECK-NEXT:    ldr w8, [sp, #80]
-; CHECK-NEXT:    mov.b v0[2], w2
-; CHECK-NEXT:    mov.b v1[2], w8
+; CHECK-NEXT:    mov.b v1[2], w2
+; CHECK-NEXT:    mov.b v0[2], w8
 ; CHECK-NEXT:    ldr w8, [sp, #88]
-; CHECK-NEXT:    mov.b v0[3], w3
-; CHECK-NEXT:    mov.b v1[3], w8
+; CHECK-NEXT:    mov.b v1[3], w3
+; CHECK-NEXT:    mov.b v0[3], w8
 ; CHECK-NEXT:    ldr w8, [sp, #96]
-; CHECK-NEXT:    mov.b v0[4], w4
-; CHECK-NEXT:    mov.b v1[4], w8
+; CHECK-NEXT:    mov.b v1[4], w4
+; CHECK-NEXT:    mov.b v0[4], w8
 ; CHECK-NEXT:    ldr w8, [sp, #104]
-; CHECK-NEXT:    mov.b v0[5], w5
-; CHECK-NEXT:    mov.b v1[5], w8
+; CHECK-NEXT:    mov.b v1[5], w5
+; CHECK-NEXT:    mov.b v0[5], w8
 ; CHECK-NEXT:    ldr w8, [sp, #112]
-; CHECK-NEXT:    mov.b v0[6], w6
-; CHECK-NEXT:    mov.b v1[6], w8
+; CHECK-NEXT:    mov.b v1[6], w6
+; CHECK-NEXT:    mov.b v0[6], w8
 ; CHECK-NEXT:    ldr w8, [sp, #120]
-; CHECK-NEXT:    mov.b v0[7], w7
-; CHECK-NEXT:    mov.b v1[7], w8
+; CHECK-NEXT:    mov.b v1[7], w7
+; CHECK-NEXT:    mov.b v0[7], w8
 ; CHECK-NEXT:    ldr w8, [sp, #128]
-; CHECK-NEXT:    mov.b v0[8], w9
+; CHECK-NEXT:    mov.b v1[8], w9
 ; CHECK-NEXT:    ldr w9, [sp, #16]
-; CHECK-NEXT:    mov.b v1[8], w8
+; CHECK-NEXT:    mov.b v0[8], w8
 ; CHECK-NEXT:    ldr w8, [sp, #136]
-; CHECK-NEXT:    mov.b v0[9], w10
+; CHECK-NEXT:    mov.b v1[9], w10
 ; CHECK-NEXT:    ldr w10, [sp, #24]
-; CHECK-NEXT:    mov.b v1[9], w8
+; CHECK-NEXT:    mov.b v0[9], w8
 ; CHECK-NEXT:    ldr w8, [sp, #144]
-; CHECK-NEXT:    mov.b v0[10], w9
+; CHECK-NEXT:    mov.b v1[10], w9
 ; CHECK-NEXT:    ldr w9, [sp, #32]
-; CHECK-NEXT:    mov.b v1[10], w8
+; CHECK-NEXT:    mov.b v0[10], w8
 ; CHECK-NEXT:    ldr w8, [sp, #152]
-; CHECK-NEXT:    mov.b v0[11], w10
+; CHECK-NEXT:    mov.b v1[11], w10
 ; CHECK-NEXT:    ldr w10, [sp, #40]
-; CHECK-NEXT:    mov.b v1[11], w8
+; CHECK-NEXT:    mov.b v0[11], w8
 ; CHECK-NEXT:    ldr w8, [sp, #160]
-; CHECK-NEXT:    mov.b v0[12], w9
+; CHECK-NEXT:    mov.b v1[12], w9
 ; CHECK-NEXT:    ldr w9, [sp, #48]
-; CHECK-NEXT:    mov.b v1[12], w8
+; CHECK-NEXT:    mov.b v0[12], w8
 ; CHECK-NEXT:    ldr w8, [sp, #168]
-; CHECK-NEXT:    mov.b v0[13], w10
+; CHECK-NEXT:    mov.b v1[13], w10
 ; CHECK-NEXT:    ldr w10, [sp, #56]
-; CHECK-NEXT:    mov.b v1[13], w8
+; CHECK-NEXT:    mov.b v0[13], w8
 ; CHECK-NEXT:    ldr w8, [sp, #176]
-; CHECK-NEXT:    mov.b v0[14], w9
-; CHECK-NEXT:    mov.b v1[14], w8
+; CHECK-NEXT:    mov.b v1[14], w9
+; CHECK-NEXT:    mov.b v0[14], w8
 ; CHECK-NEXT:    ldr w8, [sp, #184]
-; CHECK-NEXT:    mov.b v0[15], w10
-; CHECK-NEXT:    mov.b v1[15], w8
-; CHECK-NEXT:    shl.16b v0, v0, #7
+; CHECK-NEXT:    mov.b v1[15], w10
+; CHECK-NEXT:    mov.b v0[15], w8
 ; CHECK-NEXT:    shl.16b v1, v1, #7
-; CHECK-NEXT:    cmlt.16b v0, v0, #0
-; CHECK-NEXT:    cmlt.16b v1, v1, #0
+; CHECK-NEXT:    shl.16b v2, v0, #7
+; CHECK-NEXT:    cmlt.16b v0, v1, #0
+; CHECK-NEXT:    cmlt.16b v1, v2, #0
 ; CHECK-NEXT:    ret
   %res = sext <32 x i1> %arg to <32 x i8>
   ret <32 x i8> %res
 }
 
 define <64 x i8> @zext_v64i1(<64 x i1> %arg) {
 ; CHECK-LABEL: zext_v64i1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [sp, #320]
-; CHECK-NEXT:    fmov s0, w0
-; CHECK-NEXT:    ldr w9, [sp, #64]
-; CHECK-NEXT:    ldr w10, [sp, #192]
-; CHECK-NEXT:    fmov s3, w8
+; CHECK-NEXT:    ldr s0, [sp, #320]
+; CHECK-NEXT:    fmov s4, w0
 ; CHECK-NEXT:    ldr w8, [sp, #328]
-; CHECK-NEXT:    fmov s1, w9
-; CHECK-NEXT:    ldr w9, [sp, #200]
-; CHECK-NEXT:    fmov s2, w10
-; CHECK-NEXT:    ldr w10, [sp, #336]
-; CHECK-NEXT:    mov.b v3[1], w8
-; CHECK-NEXT:    ldr w8, [sp, #72]
-; CHECK-NEXT:    mov.b v0[1], w1
-; CHECK-NEXT:    ldr w11, [sp, #352]
-; CHECK-NEXT:    mov.b v2[1], w9
+; CHECK-NEXT:    ldr s1, [sp, #64]
+; CHECK-NEXT:    ldr w9, [sp, #72]
+; CHECK-NEXT:    ldr s2, [sp, #192]
+; CHECK-NEXT:    ldr w10, [sp, #200]
+; CHECK-NEXT:    mov.b v1[1], w9
 ; CHECK-NEXT:    ldr w9, [sp, #80]
-; CHECK-NEXT:    mov.b v1[1], w8
-; CHECK-NEXT:    ldr w8, [sp, #344]
-; CHECK-NEXT:    mov.b v3[2], w10
-; CHECK-NEXT:    ldr w10, [sp, #208]
-; CHECK-NEXT:    mov.b v0[2], w2
-; CHECK-NEXT:    ldr w12, [sp, #368]
-; CHECK-NEXT:    ldr w13, [sp, #384]
-; CHECK-NEXT:    mov.b v1[2], w9
-; CHECK-NEXT:    ldr w9, [sp, #360]
-; CHECK-NEXT:    mov.b v2[2], w10
+; CHECK-NEXT:    mov.b v0[1], w8
+; CHECK-NEXT:    ldr w8, [sp, #336]
+; CHECK-NEXT:    mov.b v2[1], w10
 ; CHECK-NEXT:    ldr w10, [sp, #88]
-; CHECK-NEXT:    mov.b v3[3], w8
-; CHECK-NEXT:    ldr w8, [sp, #216]
-; CHECK-NEXT:    mov.b v0[3], w3
-; CHECK-NEXT:    ldr w14, [sp, #400]
+; CHECK-NEXT:    mov.b v4[1], w1
+; CHECK-NEXT:    ldr w11, [sp, #96]
+; CHECK-NEXT:    mov.b v1[2], w9
+; CHECK-NEXT:    ldr w9, [sp, #208]
+; CHECK-NEXT:    mov.b v0[2], w8
+; CHECK-NEXT:    ldr w8, [sp, #344]
+; CHECK-NEXT:    ldr w12, [sp, #104]
+; CHECK-NEXT:    mov.b v2[2], w9
+; CHECK-NEXT:    ldr w9, [sp, #352]
+; CHECK-NEXT:    mov.b v4[2], w2
+; CHECK-NEXT:    ldr w13, [sp, #112]
 ; CHECK-NEXT:    mov.b v1[3], w10
-; CHECK-NEXT:    ldr w10, [sp, #376]
-; CHECK-NEXT:    mov.b v2[3], w8
-; CHECK-NEXT:    ldr w8, [sp, #96]
-; CHECK-NEXT:    mov.b v3[4], w11
+; CHECK-NEXT:    ldr w10, [sp, #216]
+; CHECK-NEXT:    mov.b v0[3], w8
+; CHECK-NEXT:    ldr w8, [sp, #360]
+; CHECK-NEXT:    ldr w14, [sp, #120]
+; CHECK-NEXT:    mov.b v2[3], w10
+; CHECK-NEXT:    ldr w10, [sp, #368]
+; CHECK-NEXT:    mov.b v4[3], w3
+; CHECK-NEXT:    ldr w15, [sp, #128]
+; CHECK-NEXT:    mov.b v1[4], w11
 ; CHECK-NEXT:    ldr w11, [sp, #224]
-; CHECK-NEXT:    mov.b v0[4], w4
-; CHECK-NEXT:    ldr w15, [sp, #416]
-; CHECK-NEXT:    mov.b v1[4], w8
-; CHECK-NEXT:    ldr w8, [sp, #392]
+; CHECK-NEXT:    mov.b v0[4], w9
+; CHECK-NEXT:    ldr w9, [sp, #376]
+; CHECK-NEXT:    ldr w16, [sp, #136]
 ; CHECK-NEXT:    mov.b v2[4], w11
-; CHECK-NEXT:    ldr w11, [sp, #104]
-; CHECK-NEXT:    mov.b v3[5], w9
-; CHECK-NEXT:    ldr w9, [sp, #232]
-; CHECK-NEXT:    mov.b v0[5], w5
-; CHECK-NEXT:    ldr w16, [sp, #432]
-; CHECK-NEXT:    mov.b v1[5], w11
-; CHECK-NEXT:    ldr w11, [sp, #408]
-; CHECK-NEXT:    mov.b v2[5], w9
-; CHECK-NEXT:    ldr w9, [sp, #112]
-; CHECK-NEXT:    mov.b v3[6], w12
-; CHECK-NEXT:    ldr w12, [sp, #240]
-; CHECK-NEXT:    mov.b v0[6], w6
-; CHECK-NEXT:    mov.b v1[6], w9
+; CHECK-NEXT:    ldr w11, [sp, #384]
+; CHECK-NEXT:    mov.b v4[4], w4
+; CHECK-NEXT:    mov.b v1[5], w12
+; CHECK-NEXT:    ldr w12, [sp, #232]
+; CHECK-NEXT:    mov.b v0[5], w8
+; CHECK-NEXT:    ldr w8, [sp, #392]
+; CHECK-NEXT:    movi.16b v5, #1
+; CHECK-NEXT:    mov.b v2[5], w12
+; CHECK-NEXT:    ldr w12, [sp, #400]
+; CHECK-NEXT:    mov.b v4[5], w5
+; CHECK-NEXT:    mov.b v1[6], w13
+; CHECK-NEXT:    ldr w13, [sp, #240]
+; CHECK-NEXT:    mov.b v0[6], w10
+; CHECK-NEXT:    ldr w10, [sp, #408]
+; CHECK-NEXT:    mov.b v2[6], w13
+; CHECK-NEXT:    ldr w13, [sp, #416]
+; CHECK-NEXT:    mov.b v4[6], w6
+; CHECK-NEXT:    mov.b v1[7], w14
+; CHECK-NEXT:    ldr w14, [sp, #248]
+; CHECK-NEXT:    mov.b v0[7], w9
 ; CHECK-NEXT:    ldr w9, [sp, #424]
-; CHECK-NEXT:    mov.b v2[6], w12
-; CHECK-NEXT:    ldr w12, [sp, #120]
-; CHECK-NEXT:    mov.b v3[7], w10
-; CHECK-NEXT:    ldr w10, [sp, #248]
-; CHECK-NEXT:    mov.b v0[7], w7
-; CHECK-NEXT:    mov.b v1[7], w12
-; CHECK-NEXT:    ldr w12, [sp]
-; CHECK-NEXT:    mov.b v2[7], w10
-; CHECK-NEXT:    ldr w10, [sp, #128]
-; CHECK-NEXT:    mov.b v3[8], w13
-; CHECK-NEXT:    ldr w13, [sp, #256]
-; CHECK-NEXT:    mov.b v0[8], w12
-; CHECK-NEXT:    ldr w12, [sp, #440]
-; CHECK-NEXT:    mov.b v1[8], w10
-; CHECK-NEXT:    ldr w10, [sp, #8]
-; CHECK-NEXT:    mov.b v2[8], w13
-; CHECK-NEXT:    ldr w13, [sp, #136]
-; CHECK-NEXT:    mov.b v3[9], w8
-; CHECK-NEXT:    ldr w8, [sp, #264]
-; CHECK-NEXT:    mov.b v0[9], w10
-; CHECK-NEXT:    ldr w10, [sp, #272]
-; CHECK-NEXT:    mov.b v1[9], w13
-; CHECK-NEXT:    ldr w13, [sp, #16]
-; CHECK-NEXT:    mov.b v2[9], w8
-; CHECK-NEXT:    ldr w8, [sp, #144]
-; CHECK-NEXT:    mov.b v3[10], w14
-; CHECK-NEXT:    ldr w14, [sp, #280]
-; CHECK-NEXT:    mov.b v0[10], w13
-; CHECK-NEXT:    ldr w13, [sp, #296]
-; CHECK-NEXT:    mov.b v1[10], w8
+; CHECK-NEXT:    mov.b v2[7], w14
+; CHECK-NEXT:    ldr w14, [sp, #432]
+; CHECK-NEXT:    mov.b v4[7], w7
+; CHECK-NEXT:    mov.b v1[8], w15
+; CHECK-NEXT:    ldr w15, [sp, #256]
+; CHECK-NEXT:    mov.b v0[8], w11
+; CHECK-NEXT:    ldr w11, [sp]
+; CHECK-NEXT:    mov.b v2[8], w15
+; CHECK-NEXT:    ldr w15, [sp, #440]
+; CHECK-NEXT:    mov.b v4[8], w11
+; CHECK-NEXT:    ldr w11, [sp, #144]
+; CHECK-NEXT:    mov.b v1[9], w16
+; CHECK-NEXT:    ldr w16, [sp, #264]
+; CHECK-NEXT:    mov.b v0[9], w8
+; CHECK-NEXT:    ldr w8, [sp, #8]
+; CHECK-NEXT:    mov.b v2[9], w16
+; CHECK-NEXT:    ldr w16, [sp, #272]
+; CHECK-NEXT:    mov.b v4[9], w8
+; CHECK-NEXT:    ldr w8, [sp, #16]
+; CHECK-NEXT:    mov.b v1[10], w11
+; CHECK-NEXT:    ldr w11, [sp, #152]
+; CHECK-NEXT:    mov.b v0[10], w12
+; CHECK-NEXT:    ldr w12, [sp, #280]
+; CHECK-NEXT:    mov.b v2[10], w16
+; CHECK-NEXT:    mov.b v4[10], w8
 ; CHECK-NEXT:    ldr w8, [sp, #24]
-; CHECK-NEXT:    mov.b v2[10], w10
-; CHECK-NEXT:    ldr w10, [sp, #152]
-; CHECK-NEXT:    mov.b v3[11], w11
-; CHECK-NEXT:    ldr w11, [sp, #288]
-; CHECK-NEXT:    mov.b v0[11], w8
+; CHECK-NEXT:    mov.b v1[11], w11
+; CHECK-NEXT:    ldr w11, [sp, #160]
+; CHECK-NEXT:    mov.b v0[11], w10
+; CHECK-NEXT:    ldr w10, [sp, #288]
+; CHECK-NEXT:    mov.b v2[11], w12
+; CHECK-NEXT:    mov.b v4[11], w8
 ; CHECK-NEXT:    ldr w8, [sp, #32]
-; CHECK-NEXT:    mov.b v1[11], w10
-; CHECK-NEXT:    ldr w10, [sp, #160]
-; CHECK-NEXT:    mov.b v2[11], w14
-; CHECK-NEXT:    mov.b v3[12], w15
-; CHECK-NEXT:    mov.b v0[12], w8
+; CHECK-NEXT:    mov.b v1[12], w11
+; CHECK-NEXT:    ldr w11, [sp, #168]
+; CHECK-NEXT:    mov.b v0[12], w13
+; CHECK-NEXT:    mov.b v2[12], w10
+; CHECK-NEXT:    ldr w10, [sp, #296]
+; CHECK-NEXT:    mov.b v4[12], w8
 ; CHECK-NEXT:    ldr w8, [sp, #40]
-; CHECK-NEXT:    mov.b v1[12], w10
-; CHECK-NEXT:    ldr w10, [sp, #168]
-; CHECK-NEXT:    mov.b v2[12], w11
-; CHECK-NEXT:    ldr w11, [sp, #312]
-; CHECK-NEXT:    mov.b v3[13], w9
+; CHECK-NEXT:    mov.b v1[13], w11
+; CHECK-NEXT:    ldr w11, [sp, #176]
+; CHECK-NEXT:    mov.b v0[13], w9
 ; CHECK-NEXT:    ldr w9, [sp, #304]
-; CHECK-NEXT:    mov.b v0[13], w8
+; CHECK-NEXT:    mov.b v2[13], w10
+; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    mov.b v4[13], w8
 ; CHECK-NEXT:    ldr w8, [sp, #48]
-; CHECK-NEXT:    mov.b v1[13], w10
-; CHECK-NEXT:    ldr w10, [sp, #176]
-; CHECK-NEXT:    mov.b v2[13], w13
-; CHECK-NEXT:    mov.b v3[14], w16
-; CHECK-NEXT:    mov.b v0[14], w8
-; CHECK-NEXT:    ldr w8, [sp, #56]
-; CHECK-NEXT:    mov.b v1[14], w10
+; CHECK-NEXT:    mov.b v1[14], w11
+; CHECK-NEXT:    mov.b v0[14], w14
 ; CHECK-NEXT:    mov.b v2[14], w9
-; CHECK-NEXT:    ldr w9, [sp, #184]
-; CHECK-NEXT:    movi.16b v4, #1
-; CHECK-NEXT:    mov.b v0[15], w8
-; CHECK-NEXT:    mov.b v1[15], w9
-; CHECK-NEXT:    mov.b v2[15], w11
-; CHECK-NEXT:    mov.b v3[15], w12
-; CHECK-NEXT:    and.16b v0, v0, v4
-; CHECK-NEXT:    and.16b v1, v1, v4
-; CHECK-NEXT:    and.16b v2, v2, v4
-; CHECK-NEXT:    and.16b v3, v3, v4
+; CHECK-NEXT:    ldr w9, [sp, #312]
+; CHECK-NEXT:    mov.b v4[14], w8
+; CHECK-NEXT:    ldr w8, [sp, #56]
+; CHECK-NEXT:    mov.b v1[15], w10
+; CHECK-NEXT:    mov.b v0[15], w15
+; CHECK-NEXT:    mov.b v2[15], w9
+; CHECK-NEXT:    mov.b v4[15], w8
+; CHECK-NEXT:    and.16b v1, v1, v5
+; CHECK-NEXT:    and.16b v3, v0, v5
+; CHECK-NEXT:    and.16b v2, v2, v5
+; CHECK-NEXT:    and.16b v0, v4, v5
 ; CHECK-NEXT:    ret
   %res = zext <64 x i1> %arg to <64 x i8>
   ret <64 x i8> %res
 }
 
 define <64 x i8> @sext_v64i1(<64 x i1> %arg) {
 ; CHECK-LABEL: sext_v64i1:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [sp, #320]
+; CHECK-NEXT:    ldr s0, [sp, #320]
 ; CHECK-NEXT:    fmov s3, w0
-; CHECK-NEXT:    ldr w9, [sp, #64]
-; CHECK-NEXT:    ldr w10, [sp, #192]
-; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    ldr w8, [sp, #328]
-; CHECK-NEXT:    fmov s1, w9
+; CHECK-NEXT:    ldr s1, [sp, #64]
 ; CHECK-NEXT:    ldr w9, [sp, #72]
-; CHECK-NEXT:    fmov s2, w10
-; CHECK-NEXT:    ldr w10, [sp, #80]
-; CHECK-NEXT:    mov.b v0[1], w8
-; CHECK-NEXT:    ldr w8, [sp, #200]
+; CHECK-NEXT:    ldr s2, [sp, #192]
+; CHECK-NEXT:    ldr w10, [sp, #200]
 ; CHECK-NEXT:    mov.b v1[1], w9
-; CHECK-NEXT:    ldr w9, [sp, #336]
+; CHECK-NEXT:    ldr w9, [sp, #80]
+; CHECK-NEXT:    mov.b v0[1], w8
+; CHECK-NEXT:    ldr w8, [sp, #336]
+; CHECK-NEXT:    mov.b v2[1], w10
+; CHECK-NEXT:    ldr w10, [sp, #88]
 ; CHECK-NEXT:    mov.b v3[1], w1
-; CHECK-NEXT:    ldr w11, [sp, #88]
-; CHECK-NEXT:    mov.b v2[1], w8
-; CHECK-NEXT:    ldr w8, [sp, #344]
-; CHECK-NEXT:    mov.b v0[2], w9
+; CHECK-NEXT:    ldr w11, [sp, #96]
+; CHECK-NEXT:    mov.b v1[2], w9
 ; CHECK-NEXT:    ldr w9, [sp, #208]
-; CHECK-NEXT:    mov.b v1[2], w10
-; CHECK-NEXT:    ldr w10, [sp, #352]
-; CHECK-NEXT:    mov.b v3[2], w2
-; CHECK-NEXT:    ldr w12, [sp, #96]
+; CHECK-NEXT:    mov.b v0[2], w8
+; CHECK-NEXT:    ldr w8, [sp, #344]
+; CHECK-NEXT:    ldr w12, [sp, #104]
 ; CHECK-NEXT:    mov.b v2[2], w9
-; CHECK-NEXT:    ldr w9, [sp, #360]
+; CHECK-NEXT:    ldr w9, [sp, #352]
+; CHECK-NEXT:    mov.b v3[2], w2
+; CHECK-NEXT:    ldr w13, [sp, #112]
+; CHECK-NEXT:    mov.b v1[3], w10
+; CHECK-NEXT:    ldr w10, [sp, #216]
 ; CHECK-NEXT:    mov.b v0[3], w8
-; CHECK-NEXT:    ldr w8, [sp, #216]
-; CHECK-NEXT:    mov.b v1[3], w11
-; CHECK-NEXT:    ldr w13, [sp, #104]
+; CHECK-NEXT:    ldr w8, [sp, #360]
+; CHECK-NEXT:    ldr w14, [sp, #120]
+; CHECK-NEXT:    mov.b v2[3], w10
+; CHECK-NEXT:    ldr w10, [sp, #368]
 ; CHECK-NEXT:    mov.b v3[3], w3
-; CHECK-NEXT:    ldr w11, [sp, #368]
-; CHECK-NEXT:    mov.b v2[3], w8
-; CHECK-NEXT:    ldr w14, [sp, #112]
-; CHECK-NEXT:    mov.b v0[4], w10
-; CHECK-NEXT:    ldr w10, [sp, #224]
-; CHECK-NEXT:    mov.b v1[4], w12
-; CHECK-NEXT:    ldr w8, [sp, #376]
+; CHECK-NEXT:    ldr w15, [sp, #128]
+; CHECK-NEXT:    mov.b v1[4], w11
+; CHECK-NEXT:    ldr w11, [sp, #224]
+; CHECK-NEXT:    mov.b v0[4], w9
+; CHECK-NEXT:    ldr w9, [sp, #376]
+; CHECK-NEXT:    ldr w16, [sp, #136]
+; CHECK-NEXT:    mov.b v2[4], w11
+; CHECK-NEXT:    ldr w11, [sp, #384]
 ; CHECK-NEXT:    mov.b v3[4], w4
-; CHECK-NEXT:    ldr w15, [sp, #120]
-; CHECK-NEXT:    mov.b v2[4], w10
-; CHECK-NEXT:    ldr w12, [sp, #384]
-; CHECK-NEXT:    mov.b v0[5], w9
-; CHECK-NEXT:    ldr w9, [sp, #232]
-; CHECK-NEXT:    mov.b v1[5], w13
-; CHECK-NEXT:    ldr w16, [sp, #128]
+; CHECK-NEXT:    mov.b v1[5], w12
+; CHECK-NEXT:    ldr w12, [sp, #232]
+; CHECK-NEXT:    mov.b v0[5], w8
+; CHECK-NEXT:    ldr w8, [sp, #392]
+; CHECK-NEXT:    mov.b v2[5], w12
+; CHECK-NEXT:    ldr w12, [sp, #400]
 ; CHECK-NEXT:    mov.b v3[5], w5
-; CHECK-NEXT:    ldr w10, [sp, #392]
-; CHECK-NEXT:    mov.b v2[5], w9
-; CHECK-NEXT:    ldr w13, [sp, #400]
-; CHECK-NEXT:    mov.b v0[6], w11
-; CHECK-NEXT:    ldr w11, [sp, #240]
-; CHECK-NEXT:    mov.b v1[6], w14
-; CHECK-NEXT:    ldr w9, [sp, #408]
+; CHECK-NEXT:    mov.b v1[6], w13
+; CHECK-NEXT:    ldr w13, [sp, #240]
+; CHECK-NEXT:    mov.b v0[6], w10
+; CHECK-NEXT:    ldr w10, [sp, #408]
+; CHECK-NEXT:    mov.b v2[6], w13
+; CHECK-NEXT:    ldr w13, [sp, #416]
 ; CHECK-NEXT:    mov.b v3[6], w6
-; CHECK-NEXT:    ldr w14, [sp, #416]
-; CHECK-NEXT:    mov.b v2[6], w11
-; CHECK-NEXT:    ldr w11, [sp, #424]
-; CHECK-NEXT:    mov.b v0[7], w8
-; CHECK-NEXT:    ldr w8, [sp, #248]
-; CHECK-NEXT:    mov.b v1[7], w15
-; CHECK-NEXT:    ldr w15, [sp, #432]
+; CHECK-NEXT:    mov.b v1[7], w14
+; CHECK-NEXT:    ldr w14, [sp, #248]
+; CHECK-NEXT:    mov.b v0[7], w9
+; CHECK-NEXT:    ldr w9, [sp, #424]
+; CHECK-NEXT:    mov.b v2[7], w14
+; CHECK-NEXT:    ldr w14, [sp, #432]
 ; CHECK-NEXT:    mov.b v3[7], w7
-; CHECK-NEXT:    mov.b v2[7], w8
-; CHECK-NEXT:    ldr w8, [sp]
-; CHECK-NEXT:    mov.b v0[8], w12
-; CHECK-NEXT:    ldr w12, [sp, #256]
-; CHECK-NEXT:    mov.b v1[8], w16
-; CHECK-NEXT:    ldr w16, [sp, #440]
-; CHECK-NEXT:    mov.b v3[8], w8
-; CHECK-NEXT:    ldr w8, [sp, #136]
-; CHECK-NEXT:    mov.b v2[8], w12
-; CHECK-NEXT:    ldr w12, [sp, #8]
-; CHECK-NEXT:    mov.b v0[9], w10
-; CHECK-NEXT:    ldr w10, [sp, #264]
-; CHECK-NEXT:    mov.b v1[9], w8
-; CHECK-NEXT:    ldr w8, [sp, #272]
-; CHECK-NEXT:    mov.b v3[9], w12
-; CHECK-NEXT:    ldr w12, [sp, #144]
-; CHECK-NEXT:    mov.b v2[9], w10
-; CHECK-NEXT:    ldr w10, [sp, #16]
-; CHECK-NEXT:    mov.b v0[10], w13
-; CHECK-NEXT:    ldr w13, [sp, #280]
-; CHECK-NEXT:    mov.b v1[10], w12
-; CHECK-NEXT:    ldr w12, [sp, #152]
-; CHECK-NEXT:    mov.b v3[10], w10
-; CHECK-NEXT:    ldr w10, [sp, #160]
-; CHECK-NEXT:    mov.b v2[10], w8
+; CHECK-NEXT:    mov.b v1[8], w15
+; CHECK-NEXT:    ldr w15, [sp, #256]
+; CHECK-NEXT:    mov.b v0[8], w11
+; CHECK-NEXT:    ldr w11, [sp]
+; CHECK-NEXT:    mov.b v2[8], w15
+; CHECK-NEXT:    ldr w15, [sp, #440]
+; CHECK-NEXT:    mov.b v3[8], w11
+; CHECK-NEXT:    ldr w11, [sp, #144]
+; CHECK-NEXT:    mov.b v1[9], w16
+; CHECK-NEXT:    ldr w16, [sp, #264]
+; CHECK-NEXT:    mov.b v0[9], w8
+; CHECK-NEXT:    ldr w8, [sp, #8]
+; CHECK-NEXT:    mov.b v2[9], w16
+; CHECK-NEXT:    ldr w16, [sp, #272]
+; CHECK-NEXT:    mov.b v3[9], w8
+; CHECK-NEXT:    ldr w8, [sp, #16]
+; CHECK-NEXT:    mov.b v1[10], w11
+; CHECK-NEXT:    ldr w11, [sp, #152]
+; CHECK-NEXT:    mov.b v0[10], w12
+; CHECK-NEXT:    ldr w12, [sp, #280]
+; CHECK-NEXT:    mov.b v2[10], w16
+; CHECK-NEXT:    mov.b v3[10], w8
 ; CHECK-NEXT:    ldr w8, [sp, #24]
-; CHECK-NEXT:    mov.b v0[11], w9
-; CHECK-NEXT:    ldr w9, [sp, #288]
-; CHECK-NEXT:    mov.b v1[11], w12
-; CHECK-NEXT:    ldr w12, [sp, #296]
+; CHECK-NEXT:    mov.b v1[11], w11
+; CHECK-NEXT:    ldr w11, [sp, #160]
+; CHECK-NEXT:    mov.b v0[11], w10
+; CHECK-NEXT:    ldr w10, [sp, #288]
+; CHECK-NEXT:    mov.b v2[11], w12
 ; CHECK-NEXT:    mov.b v3[11], w8
 ; CHECK-NEXT:    ldr w8, [sp, #32]
-; CHECK-NEXT:    mov.b v2[11], w13
-; CHECK-NEXT:    mov.b v0[12], w14
-; CHECK-NEXT:    mov.b v1[12], w10
-; CHECK-NEXT:    ldr w10, [sp, #168]
+; CHECK-NEXT:    mov.b v1[12], w11
+; CHECK-NEXT:    ldr w11, [sp, #168]
+; CHECK-NEXT:    mov.b v0[12], w13
+; CHECK-NEXT:    mov.b v2[12], w10
+; CHECK-NEXT:    ldr w10, [sp, #296]
 ; CHECK-NEXT:    mov.b v3[12], w8
 ; CHECK-NEXT:    ldr w8, [sp, #40]
-; CHECK-NEXT:    mov.b v2[12], w9
+; CHECK-NEXT:    mov.b v1[13], w11
+; CHECK-NEXT:    ldr w11, [sp, #176]
+; CHECK-NEXT:    mov.b v0[13], w9
 ; CHECK-NEXT:    ldr w9, [sp, #304]
-; CHECK-NEXT:    mov.b v0[13], w11
-; CHECK-NEXT:    ldr w11, [sp, #312]
-; CHECK-NEXT:    mov.b v1[13], w10
-; CHECK-NEXT:    ldr w10, [sp, #176]
+; CHECK-NEXT:    mov.b v2[13], w10
+; CHECK-NEXT:    ldr w10, [sp, #184]
 ; CHECK-NEXT:    mov.b v3[13], w8
 ; CHECK-NEXT:    ldr w8, [sp, #48]
-; CHECK-NEXT:    mov.b v2[13], w12
-; CHECK-NEXT:    mov.b v0[14], w15
-; CHECK-NEXT:    mov.b v1[14], w10
-; CHECK-NEXT:    ldr w10, [sp, #184]
+; CHECK-NEXT:    mov.b v1[14], w11
+; CHECK-NEXT:    mov.b v0[14], w14
+; CHECK-NEXT:    mov.b v2[14], w9
+; CHECK-NEXT:    ldr w9, [sp, #312]
 ; CHECK-NEXT:    mov.b v3[14], w8
 ; CHECK-NEXT:    ldr w8, [sp, #56]
-; CHECK-NEXT:    mov.b v2[14], w9
-; CHECK-NEXT:    mov.b v0[15], w16
 ; CHECK-NEXT:    mov.b v1[15], w10
+; CHECK-NEXT:    mov.b v0[15], w15
+; CHECK-NEXT:    mov.b v2[15], w9
 ; CHECK-NEXT:    mov.b v3[15], w8
-; CHECK-NEXT:    mov.b v2[15], w11
-; CHECK-NEXT:    shl.16b v4, v0, #7
 ; CHECK-NEXT:    shl.16b v1, v1, #7
-; CHECK-NEXT:    shl.16b v3, v3, #7
+; CHECK-NEXT:    shl.16b v4, v0, #7
 ; CHECK-NEXT:    shl.16b v2, v2, #7
-; CHECK-NEXT:    cmlt.16b v0, v3, #0
+; CHECK-NEXT:    shl.16b v0, v3, #7
 ; CHECK-NEXT:    cmlt.16b v1, v1, #0
 ; CHECK-NEXT:    cmlt.16b v2, v2, #0
+; CHECK-NEXT:    cmlt.16b v0, v0, #0
 ; CHECK-NEXT:    cmlt.16b v3, v4, #0
 ; CHECK-NEXT:    ret
   %res = sext <64 x i1> %arg to <64 x i8>
   ret <64 x i8> %res
 }
 
 define <1 x i128> @sext_v1x64(<1 x i64> %arg) {
   ; X0 & X1 are the real return registers, SDAG messes with v0 too for unknown reasons.

llvm/test/CodeGen/AArch64/arm64-vmul.ll

; CHECK-NEXT:    ret
   %tmp2 = load <4 x i32>, <4 x i32>* %B
   %tmp3 = call <4 x i32> @llvm.aarch64.neon.sqdmulh.v4i32(<4 x i32> %tmp1, <4 x i32> %tmp2)
   ret <4 x i32> %tmp3
 }
 
 define i32 @sqdmulh_1s(i32* %A, i32* %B) nounwind {
 ; CHECK-LABEL: sqdmulh_1s:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [x0]
-; CHECK-NEXT:    ldr w9, [x1]
-; CHECK-NEXT:    fmov s0, w8
-; CHECK-NEXT:    fmov s1, w9
+; CHECK-NEXT:    ldr s0, [x0]
+; CHECK-NEXT:    ldr s1, [x1]
 ; CHECK-NEXT:    sqdmulh s0, s0, s1
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %tmp1 = load i32, i32* %A
   %tmp2 = load i32, i32* %B
   %tmp3 = call i32 @llvm.aarch64.neon.sqdmulh.i32(i32 %tmp1, i32 %tmp2)
   ret i32 %tmp3
 }
; CHECK-NEXT:    ret
   %tmp2 = load <4 x i32>, <4 x i32>* %B
   %tmp3 = call <4 x i32> @llvm.aarch64.neon.sqrdmulh.v4i32(<4 x i32> %tmp1, <4 x i32> %tmp2)
   ret <4 x i32> %tmp3
 }
 
 define i32 @sqrdmulh_1s(i32* %A, i32* %B) nounwind {
 ; CHECK-LABEL: sqrdmulh_1s:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr w8, [x0]
-; CHECK-NEXT:    ldr w9, [x1]
-; CHECK-NEXT:    fmov s0, w8
-; CHECK-NEXT:    fmov s1, w9
+; CHECK-NEXT:    ldr s0, [x0]
+; CHECK-NEXT:    ldr s1, [x1]
 ; CHECK-NEXT:    sqrdmulh s0, s0, s1
 ; CHECK-NEXT:    fmov w0, s0
 ; CHECK-NEXT:    ret
   %tmp1 = load i32, i32* %A
   %tmp2 = load i32, i32* %B
   %tmp3 = call i32 @llvm.aarch64.neon.sqrdmulh.i32(i32 %tmp1, i32 %tmp2)
   ret i32 %tmp3
 }

llvm/test/CodeGen/AArch64/dp1.ll

 ; CHECK-SDAG-NEXT:    fmov w9, s0
 ; CHECK-SDAG-NEXT:    str x9, [x8]
 ; CHECK-SDAG-NEXT:    ret
 ;
 ; CHECK-GISEL-LABEL: ctpop_i64:
 ; CHECK-GISEL:       // %bb.0:
 ; CHECK-GISEL-NEXT:    adrp x8, :got:var64
 ; CHECK-GISEL-NEXT:    ldr x8, [x8, :got_lo12:var64]
-; CHECK-GISEL-NEXT:    ldr x9, [x8]
-; CHECK-GISEL-NEXT:    fmov d0, x9
+; CHECK-GISEL-NEXT:    ldr d0, [x8]
 ; CHECK-GISEL-NEXT:    cnt v0.8b, v0.8b
 ; CHECK-GISEL-NEXT:    uaddlv h0, v0.8b
 ; CHECK-GISEL-NEXT:    fmov w9, s0
 ; CHECK-GISEL-NEXT:    str x9, [x8]
 ; CHECK-GISEL-NEXT:    ret
   %val0_tmp = load i64, i64* @var64
   %val4_tmp = call i64 @llvm.ctpop.i64(i64 %val0_tmp)
   store volatile i64 %val4_tmp, i64* @var64

llvm/test/CodeGen/AArch64/neon-dotpattern.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple aarch64-none-linux-gnu -mattr=+dotprod    < %s | FileCheck %s
 
 define fastcc void @test_sdot_v4i8(i8* noalias nocapture %0, i8* noalias nocapture readonly %1, i8* noalias nocapture readonly %2) {
 ; CHECK-LABEL: test_sdot_v4i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr w8, [x2]
 ; CHECK-NEXT:    dup v0.2s, wzr
-; CHECK-NEXT:    ldr w9, [x1]
-; CHECK-NEXT:    fmov s1, w8
-; CHECK-NEXT:    fmov s2, w9
+; CHECK-NEXT:    ldr s1, [x2]
+; CHECK-NEXT:    ldr s2, [x1]
 ; CHECK-NEXT:    sdot v0.2s, v1.8b, v2.8b
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    str w8, [x0]
 ; CHECK-NEXT:    ret
 entry:
   %3 = bitcast i8* %0 to i32*
   %4 = load i8, i8* %1, align 1
   %5 = sext i8 %4 to i32
entry:
   %32 = add nsw i32 %31, %24
   store i32 %32, i32* %3, align 64
   ret void
 }
 
 define fastcc void @test_udot_v4i8(i8* noalias nocapture %0, i8* noalias nocapture readonly %1, i8* noalias nocapture readonly %2) {
 ; CHECK-LABEL: test_udot_v4i8:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr w8, [x2]
 ; CHECK-NEXT:    dup v0.2s, wzr
-; CHECK-NEXT:    ldr w9, [x1]
-; CHECK-NEXT:    fmov s1, w8
-; CHECK-NEXT:    fmov s2, w9
+; CHECK-NEXT:    ldr s1, [x2]
+; CHECK-NEXT:    ldr s2, [x1]
 ; CHECK-NEXT:    udot v0.2s, v1.8b, v2.8b
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    str w8, [x0]
 ; CHECK-NEXT:    ret
 entry:
   %3 = bitcast i8* %0 to i32*
   %4 = load i8, i8* %1, align 1
   %5 = zext i8 %4 to i32

llvm/test/CodeGen/AArch64/neon-extadd.ll

entry:
   %s1s = zext <20 x i8> %s1 to <20 x i32>
   %m = add <20 x i32> %s0s, %s1s
   ret <20 x i32> %m
 }
 
 define <16 x i32> @i12(<16 x i12> %s0, <16 x i12> %s1) {
 ; CHECK-LABEL: i12:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    ldr w12, [sp, #32]
-; CHECK-NEXT:    fmov s5, w0
-; CHECK-NEXT:    ldr w15, [sp]
+; CHECK-NEXT:    ldr w10, [sp, #40]
 ; CHECK-NEXT:    fmov s4, w4
-; CHECK-NEXT:    ldr w14, [sp, #40]
-; CHECK-NEXT:    fmov s0, w12
-; CHECK-NEXT:    ldr w16, [sp, #48]
-; CHECK-NEXT:    fmov s1, w15
-; CHECK-NEXT:    ldr w15, [sp, #8]
-; CHECK-NEXT:    ldr w18, [sp, #16]
-; CHECK-NEXT:    mov v0.h[1], w14
-; CHECK-NEXT:    ldr w17, [sp, #56]
-; CHECK-NEXT:    mov v1.h[1], w15
-; CHECK-NEXT:    ldr w0, [sp, #24]
-; CHECK-NEXT:    mov v5.h[1], w1
-; CHECK-NEXT:    ldr w13, [sp, #64]
-; CHECK-NEXT:    ldr w1, [sp, #128]
-; CHECK-NEXT:    mov v0.h[2], w16
-; CHECK-NEXT:    ldr w16, [sp, #96]
-; CHECK-NEXT:    mov v1.h[2], w18
-; CHECK-NEXT:    ldr w10, [sp, #72]
-; CHECK-NEXT:    mov v5.h[2], w2
-; CHECK-NEXT:    ldr w2, [sp, #160]
-; CHECK-NEXT:    mov v4.h[1], w5
-; CHECK-NEXT:    ldr w5, [sp, #168]
-; CHECK-NEXT:    mov v0.h[3], w17
-; CHECK-NEXT:    ldr w14, [sp, #104]
-; CHECK-NEXT:    mov v1.h[3], w0
-; CHECK-NEXT:    ldr w18, [sp, #136]
-; CHECK-NEXT:    fmov s6, w1
+; CHECK-NEXT:    ldr s0, [sp, #32]
+; CHECK-NEXT:    fmov s5, w0
+; CHECK-NEXT:    ldr w13, [sp, #8]
+; CHECK-NEXT:    ldr s1, [sp]
+; CHECK-NEXT:    mov v0.h[1], w10
+; CHECK-NEXT:    ldr w12, [sp, #48]
+; CHECK-NEXT:    ldr w16, [sp, #16]
+; CHECK-NEXT:    mov v1.h[1], w13
+; CHECK-NEXT:    ldr w14, [sp, #56]
+; CHECK-NEXT:    ldr w18, [sp, #24]
+; CHECK-NEXT:    mov v0.h[2], w12
+; CHECK-NEXT:    ldr w11, [sp, #72]
+; CHECK-NEXT:    ldr w13, [sp, #104]
+; CHECK-NEXT:    mov v1.h[2], w16
+; CHECK-NEXT:    ldr w17, [sp, #136]
+; CHECK-NEXT:    ldr s6, [sp, #128]
+; CHECK-NEXT:    mov v0.h[3], w14
+; CHECK-NEXT:    ldr s7, [sp, #96]
+; CHECK-NEXT:    ldr s16, [sp, #64]
+; CHECK-NEXT:    mov v1.h[3], w18
+; CHECK-NEXT:    ldr w18, [sp, #168]
 ; CHECK-NEXT:    ldr w0, [sp, #176]
-; CHECK-NEXT:    fmov s7, w16
-; CHECK-NEXT:    fmov s16, w13
 ; CHECK-NEXT:    ushll v2.4s, v0.4h, #0
 ; CHECK-NEXT:    ldr w9, [sp, #80]
 ; CHECK-NEXT:    movi v0.4s, #15, msl #8
 ; CHECK-NEXT:    ldr w12, [sp, #112]
 ; CHECK-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-NEXT:    ldr w17, [sp, #144]
-; CHECK-NEXT:    mov v6.h[1], w18
-; CHECK-NEXT:    ldr w4, [sp, #184]
-; CHECK-NEXT:    mov v7.h[1], w14
+; CHECK-NEXT:    ldr w15, [sp, #144]
+; CHECK-NEXT:    mov v4.h[1], w5
+; CHECK-NEXT:    ldr w16, [sp, #184]
+; CHECK-NEXT:    mov v5.h[1], w1
 ; CHECK-NEXT:    ldr w8, [sp, #88]
 ; CHECK-NEXT:    and v3.16b, v2.16b, v0.16b
-; CHECK-NEXT:    ldr w11, [sp, #120]
+; CHECK-NEXT:    ldr w10, [sp, #120]
 ; CHECK-NEXT:    and v2.16b, v1.16b, v0.16b
-; CHECK-NEXT:    ldr w15, [sp, #152]
-; CHECK-NEXT:    fmov s1, w2
-; CHECK-NEXT:    mov v16.h[1], w10
+; CHECK-NEXT:    ldr s1, [sp, #160]
+; CHECK-NEXT:    mov v6.h[1], w17
+; CHECK-NEXT:    ldr w14, [sp, #152]
+; CHECK-NEXT:    mov v7.h[1], w13
+; CHECK-NEXT:    mov v1.h[1], w18
+; CHECK-NEXT:    mov v16.h[1], w11
 ; CHECK-NEXT:    mov v4.h[2], w6
-; CHECK-NEXT:    mov v1.h[1], w5
-; CHECK-NEXT:    mov v6.h[2], w17
+; CHECK-NEXT:    mov v5.h[2], w2
+; CHECK-NEXT:    mov v1.h[2], w0
+; CHECK-NEXT:    mov v6.h[2], w15
 ; CHECK-NEXT:    mov v7.h[2], w12
 ; CHECK-NEXT:    mov v16.h[2], w9
-; CHECK-NEXT:    mov v1.h[2], w0
+; CHECK-NEXT:    mov v1.h[3], w16
 ; CHECK-NEXT:    mov v4.h[3], w7
 ; CHECK-NEXT:    mov v5.h[3], w3
-; CHECK-NEXT:    mov v6.h[3], w15
-; CHECK-NEXT:    mov v1.h[3], w4
-; CHECK-NEXT:    mov v7.h[3], w11
+; CHECK-NEXT:    mov v6.h[3], w14
+; CHECK-NEXT:    mov v7.h[3], w10
 ; CHECK-NEXT:    mov v16.h[3], w8
-; CHECK-NEXT:    ushll v4.4s, v4.4h, #0
 ; CHECK-NEXT:    ushll v1.4s, v1.4h, #0
+; CHECK-NEXT:    ushll v4.4s, v4.4h, #0
 ; CHECK-NEXT:    ushll v5.4s, v5.4h, #0
 ; CHECK-NEXT:    ushll v6.4s, v6.4h, #0
 ; CHECK-NEXT:    and v17.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ushll v1.4s, v7.4h, #0
 ; CHECK-NEXT:    ushll v7.4s, v16.4h, #0
 ; CHECK-NEXT:    and v4.16b, v4.16b, v0.16b
 ; CHECK-NEXT:    and v5.16b, v5.16b, v0.16b
 ; CHECK-NEXT:    and v6.16b, v6.16b, v0.16b

llvm/test/CodeGen/AArch64/peephole-ldr.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -run-pass=aarch64-mi-peephole-opt -o - -mtriple=aarch64-unknown-linux -verify-machineinstrs %s | FileCheck %s
+
+--- |
+  define void @func(ptr %0, ptr %1) {
+    %3 = getelementptr inbounds <2 x i64>, ptr %1, i64 0, i64 1
+    %4 = load i64, ptr %3, align 8
+    %5 = tail call <16 x i8> @llvm.aarch64.neon.pmull64(i64 %4, i64 2655706616)
+    store <16 x i8> %5, ptr %1, align 16
+    ret void
+  }
+
+  ; Function Attrs: nocallback nofree nosync nounwind readnone willreturn
+  declare <16 x i8> @llvm.aarch64.neon.pmull64(i64, i64)
+
+...
+---
+name:            func
+alignment:       4
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+callsEHReturn:   false
+callsUnwindInit: false
+hasEHCatchret:   false
+hasEHScopes:     false
+hasEHFunclets:   false
+failsVerification: false
+tracksDebugUserValues: false
+registers:
+  - { id: 0, class: gpr64, preferred-register: '' }
+  - { id: 1, class: gpr64common, preferred-register: '' }
+  - { id: 2, class: gpr64, preferred-register: '' }
+  - { id: 3, class: gpr32, preferred-register: '' }
+  - { id: 4, class: gpr64all, preferred-register: '' }
+  - { id: 5, class: fpr128, preferred-register: '' }
+  - { id: 6, class: fpr64, preferred-register: '' }
+  - { id: 7, class: fpr64, preferred-register: '' }
+liveins:
+  - { reg: '$x1', virtual-reg: '%1' }
+frameInfo:
+  isFrameAddressTaken: false
+  isReturnAddressTaken: false
+  hasStackMap:     false
+  hasPatchPoint:   false
+  stackSize:       0
+  offsetAdjustment: 0
+  maxAlignment:    1
+  adjustsStack:    false
+  hasCalls:        false
+  stackProtector:  ''
+  functionContext: ''
+  maxCallFrameSize: 0
+  cvBytesOfCalleeSavedRegisters: 0
+  hasOpaqueSPAdjustment: false
+  hasVAStart:      false
+  hasMustTailInVarArgFunc: false
+  hasTailCall:     false
+  localFrameSize:  0
+  savePoint:       ''
+  restorePoint:    ''
+fixedStack:      []
+stack:           []
+callSites:       []
+debugValueSubstitutions: []
+constants:       []
+machineFunctionInfo: {}
+body:             |
+  bb.0 (%ir-block.2):
+    liveins: $x1
+
+    ; CHECK-LABEL: name: func
+    ; CHECK: liveins: $x1
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: [[COPY:%[0-9]+]]:gpr64common = COPY $x1
+    ; CHECK-NEXT: [[MOVi32imm:%[0-9]+]]:gpr32 = MOVi32imm -1639260680
+    ; CHECK-NEXT: [[SUBREG_TO_REG:%[0-9]+]]:gpr64all = SUBREG_TO_REG 0, killed [[MOVi32imm]], %subreg.sub_32
+    ; CHECK-NEXT: [[LDRDui:%[0-9]+]]:fpr64 = LDRDui [[COPY]], 1 :: (load (s64) from %ir.3)
+    ; CHECK-NEXT: [[COPY1:%[0-9]+]]:fpr64 = COPY [[SUBREG_TO_REG]]
+    ; CHECK-NEXT: [[PMULLv1i64_:%[0-9]+]]:fpr128 = PMULLv1i64 killed [[LDRDui]], killed [[COPY1]]
+    ; CHECK-NEXT: STRQui killed [[PMULLv1i64_]], [[COPY]], 0 :: (store (s128) into %ir.1)
+    ; CHECK-NEXT: RET_ReallyLR
+    %1:gpr64common = COPY $x1
+    %2:gpr64 = LDRXui %1, 1 :: (load (s64) from %ir.3)
+    %3:gpr32 = MOVi32imm -1639260680
+    %4:gpr64all = SUBREG_TO_REG 0, killed %3, %subreg.sub_32
+    %6:fpr64 = COPY %2
+    %7:fpr64 = COPY %4
+    %5:fpr128 = PMULLv1i64 killed %6, killed %7
+    STRQui killed %5, %1, 0 :: (store (s128) into %ir.1)
+    RET_ReallyLR