[ARM] Convert floating point splats to integer

[dmgreen]

Summary

Under MVE a vdup will always take a gpr register, not a floating point value. During DAG combine we convert the types to a bitcast to an integer in an attempt to fold the bitcast into other code. This is OK, but only works inside the same basic block. To do the same trick across a basic block boundary we need to convert the type in codegenprepare, before the splat is sunk into the loop.

This adds a convertSplatType function to codegenprepare to do that, putting bitcasts around the splatto force the type to an integer. There is then some adjustment to the code in shouldSinkOperands to handle the extra bitcasts.

[dmgreen]

Now also hoists the first bitcast up to it's user, in case it is not in the same block.

[dmgreen]

Rebase the tests on other changes that have gone in recently.

[efriedma]

llvm/lib/CodeGen/CodeGenPrepare.cpp

6436

I guess this is only loosely related to your patch, but should this transform also be handled by tryToSinkFreeOperands?

6525

Checking whether an insertion point is legal is more complicated than this. In particular, you can't insert code after an invoke/callbr, and you can't insert code into a block terminated by a catchswitch.

I'm not sure if there's a helper for this check anywhere.

llvm/lib/Target/ARM/ARMISelLowering.cpp

15786

Is this specifically a ShuffleVectorInst?

[dmgreen]

Instruction -> ShuffleVectorInstruction and tighten the conditions for hoisting bitcasts.

[dmgreen]

llvm/lib/CodeGen/CodeGenPrepare.cpp

6436

Yeah, I think it pre-dates the tryToSinkFreeOperands version but only handles the shifts that X86 cares about. I can have a look at changing it over to see if it can use SinkFreeOperands, but @spatel is updating it in D79718. I'll at least wait until that has been done.

6525

Ah right. I've added Terminators and EHPads (mostly because there is likely little to be gained from moving the bitcast, even if it is safe). That with the phi's should exclude catchswitch too I believe.

There is ScalarEvolutionExpander::findInsertPointAfter which is trying to find an insertion point after Op. Here we don't care as much about giving up, and some of the semantics of following invokes will be different. I didn't spot much else.

I can try and make a function if you think that's better.

[spatel]

llvm/lib/CodeGen/CodeGenPrepare.cpp

6436

Thanks for cc'ing me. This function seems worse than just awkwardly structured; it can do transforms that were not intended. That's because we are walking the users of the splat shuffle without checking whether the actual use is as a shift's amount operand (operand 1 of a regular shift opcode). That could be a (hopefully minor) perf bug if it ever happens, but that's probably still not what we want to happen.

I think this transform should begin from the shift instruction, not the shuffle. tryToSinkFreeOperands() looks promising to replace this and the TLI hook. I'll take a look at updating this now, but that doesn't need to block this patch.

[efriedma]

LGTM

llvm/include/llvm/CodeGen/TargetLowering.h

public:
   /// Return true if it's significantly cheaper to shift a vector by a uniform
   /// scalar than by an amount which will vary across each lane. On x86, for
   /// example, there is a "psllw" instruction for the former case, but no simple
   /// instruction for a general "a << b" operation on vectors.
   virtual bool isVectorShiftByScalarCheap(Type *Ty) const {
     return false;
   }
 
+  /// Given a shuffle vector SVI representing a vector splat, return a new
+  /// scalar type of size equal to SVI's scalar type if the new type is more
+  /// profitable. Returns nullptr otherwise. For example under MVE float splats
+  /// are converted to integer to prevent the need to move from SPR to GPR
+  /// registers.
+  virtual Type* shouldConvertSplatType(ShuffleVectorInst* SVI) const {
+    return nullptr;
+  }
+
   /// Returns true if the opcode is a commutative binary operation.
   virtual bool isCommutativeBinOp(unsigned Opcode) const {
     // FIXME: This should get its info from the td file.
     switch (Opcode) {
     case ISD::ADD:
     case ISD::SMIN:
     case ISD::SMAX:
     case ISD::UMIN:

llvm/lib/CodeGen/CodeGenPrepare.cpp

private:
     bool optimizeGatherScatterInst(Instruction *MemoryInst, Value *Ptr);
     bool optimizeInlineAsmInst(CallInst *CS);
     bool optimizeCallInst(CallInst *CI, bool &ModifiedDT);
     bool optimizeExt(Instruction *&I);
     bool optimizeExtUses(Instruction *I);
     bool optimizeLoadExt(LoadInst *Load);
     bool optimizeShiftInst(BinaryOperator *BO);
     bool optimizeSelectInst(SelectInst *SI);
+    bool sinkShuffleVectorToShift(ShuffleVectorInst *SVI);
+    bool convertSplatType(ShuffleVectorInst *SVI);
     bool optimizeShuffleVectorInst(ShuffleVectorInst *SVI);
     bool optimizeSwitchInst(SwitchInst *SI);
     bool optimizeExtractElementInst(Instruction *Inst);
     bool dupRetToEnableTailCallOpts(BasicBlock *BB, bool &ModifiedDT);
     bool fixupDbgValue(Instruction *I);
     bool placeDbgValues(Function &F);
     bool canFormExtLd(const SmallVectorImpl<Instruction *> &MovedExts,
                       LoadInst *&LI, Instruction *&Inst, bool HasPromoted);
static bool isBroadcastShuffle(ShuffleVectorInst *SVI) {
 
   return true;
 }
 
 /// Some targets have expensive vector shifts if the lanes aren't all the same
 /// (e.g. x86 only introduced "vpsllvd" and friends with AVX2). In these cases
 /// it's often worth sinking a shufflevector splat down to its use so that
 /// codegen can spot all lanes are identical.
-bool CodeGenPrepare::optimizeShuffleVectorInst(ShuffleVectorInst *SVI) {
+bool CodeGenPrepare::sinkShuffleVectorToShift(ShuffleVectorInst *SVI) {
   BasicBlock *DefBB = SVI->getParent();
 
   // Only do this xform if variable vector shifts are particularly expensive.
   if (!TLI->isVectorShiftByScalarCheap(SVI->getType()))
     return false;
 
   // We only expect better codegen by sinking a shuffle if we can recognise a
   // constant splat.
bool CodeGenPrepare::sinkShuffleVectorToShift(ShuffleVectorInst *SVI) {
   if (SVI->use_empty()) {
     SVI->eraseFromParent();
     MadeChange = true;
   }
 
   return MadeChange;
 }
 
+/// Some targets only accept certain types for splat inputs. For example a VDUP
+/// in MVE takes a GPR (integer) register, and the instruction that incorporate
+/// a VDUP (such as a VADD qd, qm, rm) also require a gpr register.
+bool CodeGenPrepare::convertSplatType(ShuffleVectorInst *SVI) {
+  if (!match(SVI,
+             m_ShuffleVector(m_InsertElement(m_Undef(), m_Value(), m_ZeroInt()),
+                             m_Undef(), m_ZeroMask())))
+    return false;
+  Type *NewType = TLI->shouldConvertSplatType(SVI);
+  if (!NewType)
+    return false;
+
+  VectorType *SVIVecType = cast<VectorType>(SVI->getType());
+  Type *SVIType = SVIVecType->getScalarType();
+  assert(!NewType->isVectorTy() && "Expected a scalar type!");
+  assert(NewType->getScalarSizeInBits() == SVIType->getScalarSizeInBits() &&
+         "Expected a type of the same size!");
+  Type *NewVecType = VectorType::get(NewType, SVIVecType->getNumElements());
+
+  // Create a bitcast (shuffle (insert (bitcast(..))))
+  IRBuilder<> Builder(SVI->getContext());
+  Builder.SetInsertPoint(SVI);
+  Value *BC1 = Builder.CreateBitCast(
+      cast<Instruction>(SVI->getOperand(0))->getOperand(1), NewType);
+  Value *Insert = Builder.CreateInsertElement(UndefValue::get(NewVecType), BC1,
+                                              (uint64_t)0);
+  Value *Shuffle = Builder.CreateShuffleVector(
+      Insert, UndefValue::get(NewVecType), SVI->getShuffleMask());
+  Value *BC2 = Builder.CreateBitCast(Shuffle, SVIVecType);
+
+  SVI->replaceAllUsesWith(BC2);
+  RecursivelyDeleteTriviallyDeadInstructions(SVI);
+
+  // Also hoist the bitcast up to its operand if it they are not in the same
+  // block.
+  if (auto *BCI = dyn_cast<Instruction>(BC1))
+    if (auto *Op = dyn_cast<Instruction>(BCI->getOperand(0)))
+      if (BCI->getParent() != Op->getParent() && !isa<PHINode>(Op) &&
+          !Op->isTerminator() && !Op->isEHPad())
+        BCI->moveAfter(Op);
+
+  return true;
+}
+
+bool CodeGenPrepare::optimizeShuffleVectorInst(ShuffleVectorInst *SVI) {
+  if (sinkShuffleVectorToShift(SVI))
+    return true;
+  if (convertSplatType(SVI))
+    return true;
+  return false;
+}
+
 bool CodeGenPrepare::tryToSinkFreeOperands(Instruction *I) {
   // If the operands of I can be folded into a target instruction together with
   // I, duplicate and sink them.
   SmallVector<Use *, 4> OpsToSink;
   if (!TLI->shouldSinkOperands(I, OpsToSink))
     return false;
 
   // OpsToSink can contain multiple uses in a use chain (e.g.

llvm/lib/Target/ARM/ARMISelLowering.h

public:
     EVT getOptimalMemOpType(const MemOp &Op,
                             const AttributeList &FuncAttributes) const override;
 
     bool isTruncateFree(Type *SrcTy, Type *DstTy) const override;
     bool isTruncateFree(EVT SrcVT, EVT DstVT) const override;
     bool isZExtFree(SDValue Val, EVT VT2) const override;
     bool shouldSinkOperands(Instruction *I,
                             SmallVectorImpl<Use *> &Ops) const override;
+    Type* shouldConvertSplatType(ShuffleVectorInst* SVI) const override;
 
     bool isFNegFree(EVT VT) const override;
 
     bool isVectorLoadExtDesirable(SDValue ExtVal) const override;
 
     bool allowTruncateForTailCall(Type *Ty1, Type *Ty2) const override;
 
 

llvm/lib/Target/ARM/ARMISelLowering.cpp

case Instruction::Call:
       }
       return false;
     default:
       return false;
     }
   };
 
   for (auto OpIdx : enumerate(I->operands())) {
-    Value *Op = OpIdx.value().get();
+    Instruction *Op = dyn_cast<Instruction>(OpIdx.value().get());
     // Make sure we are not already sinking this operand
-    if (any_of(Ops, [&](Use *U) { return U->get() == Op; }))
+    if (!Op || any_of(Ops, [&](Use *U) { return U->get() == Op; }))
       continue;
+
+    Instruction *Shuffle = Op;
+    if (Shuffle->getOpcode() == Instruction::BitCast)
+      Shuffle = dyn_cast<Instruction>(Shuffle->getOperand(0));
     // We are looking for a splat that can be sunk.
-    if (!match(Op, m_ShuffleVector(
+    if (!Shuffle ||
+        !match(Shuffle, m_ShuffleVector(
                             m_InsertElement(m_Undef(), m_Value(), m_ZeroInt()),
                             m_Undef(), m_ZeroMask())))
       continue;
     if (!IsSinker(I, OpIdx.index()))
       continue;
 
-    Instruction *Shuffle = cast<Instruction>(Op);
     // All uses of the shuffle should be sunk to avoid duplicating it across gpr
     // and vector registers
-    for (Use &U : Shuffle->uses()) {
+    for (Use &U : Op->uses()) {
       Instruction *Insn = cast<Instruction>(U.getUser());
       if (!IsSinker(Insn, U.getOperandNo()))
         return false;
     }
+
     Ops.push_back(&Shuffle->getOperandUse(0));
+    if (Shuffle != Op)
+      Ops.push_back(&Op->getOperandUse(0));
     Ops.push_back(&OpIdx.value());
   }
   return true;
 }
 
+Type *ARMTargetLowering::shouldConvertSplatType(ShuffleVectorInst *SVI) const {
+  if (!Subtarget->hasMVEIntegerOps())
+    return nullptr;
+  Type *SVIType = SVI->getType();
+  Type *ScalarType = SVIType->getScalarType();
+
+  if (ScalarType->isFloatTy())
+    return Type::getInt32Ty(SVIType->getContext());
+  if (ScalarType->isHalfTy())
+    return Type::getInt16Ty(SVIType->getContext());
+  return nullptr;
+}
+
 bool ARMTargetLowering::isVectorLoadExtDesirable(SDValue ExtVal) const {
   EVT VT = ExtVal.getValueType();
 
   if (!isTypeLegal(VT))
     return false;
 
   if (auto *Ld = dyn_cast<MaskedLoadSDNode>(ExtVal.getOperand(0))) {
     if (Ld->isExpandingLoad())

llvm/test/CodeGen/Thumb2/mve-float16regloops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc void @test_fadd(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fadd:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vadd.f16 q0, q0, r1
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 }
 
 define arm_aapcs_vfpcc void @test_fadd_r(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fadd_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vadd.f16 q0, q0, r1
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 }
 
 define arm_aapcs_vfpcc void @test_fmul(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fmul:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vmul.f16 q0, q0, r1
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 }
 
 define arm_aapcs_vfpcc void @test_fmul_r(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fmul_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vmul.f16 q0, q0, r1
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 }
 
 define arm_aapcs_vfpcc void @test_fsub(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fsub:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vsub.f16 q0, q0, r1
 ; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 }
 
 define arm_aapcs_vfpcc void @test_fsub_r(half* noalias nocapture readonly %A, half *%BB, half* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fsub_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r1]
-; CHECK-NEXT:    vmov.f16 r1, s0
+; CHECK-NEXT:    ldrh r1, [r1]
 ; CHECK-NEXT:    vdup.16 q0, r1
 ; CHECK-NEXT:  .LBB5_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    subs r3, #8
 ; CHECK-NEXT:    vsub.f16 q1, q0, q1
 ; CHECK-NEXT:    vstrb.8 q1, [r2], #16
 ; CHECK-NEXT:    bne .LBB5_1

 
 define arm_aapcs_vfpcc void @test_fmas(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmas:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vfmas.f16 q1, q0, r2
 ; CHECK-NEXT:    vstrb.8 q1, [r3], #16
 ; CHECK-NEXT:    bne .LBB6_1

 
 define arm_aapcs_vfpcc void @test_fmas_r(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmas_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB7_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vfmas.f16 q1, q0, r2
 ; CHECK-NEXT:    vstrb.8 q1, [r3], #16
 ; CHECK-NEXT:    bne .LBB7_1

 
 define arm_aapcs_vfpcc void @test_fma(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fma:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vfma.f16 q1, q0, r2
 ; CHECK-NEXT:    vstrb.8 q1, [r3], #16
 ; CHECK-NEXT:    bne .LBB8_1

 
 define arm_aapcs_vfpcc void @test_fma_r(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fma_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vfma.f16 q1, q0, r2
 ; CHECK-NEXT:    vstrb.8 q1, [r3], #16
 ; CHECK-NEXT:    bne .LBB9_1

 
 define arm_aapcs_vfpcc void @test_fmss(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmss:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:    vneg.f16 q0, q0
 ; CHECK-NEXT:  .LBB10_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
 ; CHECK-NEXT:    vmov q3, q0
 ; CHECK-NEXT:    subs.w r12, r12, #8

 
 define arm_aapcs_vfpcc void @test_fmss_r(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmss_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:    vdup.16 q0, r2
 ; CHECK-NEXT:  .LBB11_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
 ; CHECK-NEXT:    vmov q3, q0
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vfms.f16 q3, q2, q1

 
 define arm_aapcs_vfpcc void @test_fms(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fms:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB12_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vneg.f16 q0, q0
 ; CHECK-NEXT:    vfma.f16 q0, q1, r2
 ; CHECK-NEXT:    vstrb.8 q0, [r3], #16

 
 define arm_aapcs_vfpcc void @test_fms_r(half* noalias nocapture readonly %A, half* noalias nocapture readonly %B, half *%CC, half* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fms_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    ldr.w r12, [sp]
 ; CHECK-NEXT:    cmp.w r12, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
-; CHECK-NEXT:    vldr.16 s0, [r2]
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    ldrh r2, [r2]
 ; CHECK-NEXT:  .LBB13_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    subs.w r12, r12, #8
 ; CHECK-NEXT:    vneg.f16 q0, q0
 ; CHECK-NEXT:    vfma.f16 q0, q1, r2
 ; CHECK-NEXT:    vstrb.8 q0, [r3], #16

 ; CHECK-NEXT:    ldr.w r12, [r0, #4]
 ; CHECK-NEXT:    subs r6, r5, #1
 ; CHECK-NEXT:    cmp r6, #3
 ; CHECK-NEXT:    bhi .LBB15_6
 ; CHECK-NEXT:  @ %bb.1: @ %if.then
 ; CHECK-NEXT:    ldr r7, [r0, #8]
 ; CHECK-NEXT:    add.w r4, r12, r6, lsl #1
 ; CHECK-NEXT:    lsr.w lr, r3, #2
-; CHECK-NEXT:    vldr.16 s0, [r7, #6]
-; CHECK-NEXT:    vldr.16 s2, [r7, #4]
-; CHECK-NEXT:    vldr.16 s4, [r7, #2]
-; CHECK-NEXT:    vldr.16 s6, [r7]
+; CHECK-NEXT:    ldrh.w r8, [r7, #6]
+; CHECK-NEXT:    ldrh.w r9, [r7, #4]
+; CHECK-NEXT:    ldrh r6, [r7, #2]
+; CHECK-NEXT:    ldrh r7, [r7]
 ; CHECK-NEXT:    wls lr, lr, .LBB15_5
 ; CHECK-NEXT:  @ %bb.2: @ %while.body.lr.ph
-; CHECK-NEXT:    vmov.f16 r11, s6
 ; CHECK-NEXT:    str r5, [sp, #12] @ 4-byte Spill
-; CHECK-NEXT:    vmov.f16 r10, s4
 ; CHECK-NEXT:    bic r5, r3, #3
-; CHECK-NEXT:    vmov.f16 r7, s2
-; CHECK-NEXT:    add.w r6, r12, #2
-; CHECK-NEXT:    vmov.f16 r8, s0
+; CHECK-NEXT:    add.w r10, r12, #2
 ; CHECK-NEXT:    str r5, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    add.w r5, r2, r5, lsl #1
 ; CHECK-NEXT:    str r5, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    str r1, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:  .LBB15_3: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #8
-; CHECK-NEXT:    sub.w r9, r6, #2
-; CHECK-NEXT:    adds r5, r6, #2
-; CHECK-NEXT:    vstrb.8 q2, [r4], #8
-; CHECK-NEXT:    vldrw.u32 q2, [r9]
-; CHECK-NEXT:    vldrw.u32 q3, [r6]
-; CHECK-NEXT:    vmul.f16 q2, q2, r11
-; CHECK-NEXT:    vfma.f16 q2, q3, r10
-; CHECK-NEXT:    vldrw.u32 q3, [r5]
-; CHECK-NEXT:    vfma.f16 q2, q3, r7
-; CHECK-NEXT:    vldrw.u32 q3, [r6, #4]
-; CHECK-NEXT:    adds r6, #8
-; CHECK-NEXT:    vfma.f16 q2, q3, r8
-; CHECK-NEXT:    vstrb.8 q2, [r2], #8
+; CHECK-NEXT:    vldrw.u32 q0, [r1], #8
+; CHECK-NEXT:    sub.w r11, r10, #2
+; CHECK-NEXT:    add.w r5, r10, #2
+; CHECK-NEXT:    vstrb.8 q0, [r4], #8
+; CHECK-NEXT:    vldrw.u32 q0, [r11]
+; CHECK-NEXT:    vldrw.u32 q1, [r10]
+; CHECK-NEXT:    vmul.f16 q0, q0, r7
+; CHECK-NEXT:    vfma.f16 q0, q1, r6
+; CHECK-NEXT:    vldrw.u32 q1, [r5]
+; CHECK-NEXT:    vfma.f16 q0, q1, r9
+; CHECK-NEXT:    vldrw.u32 q1, [r10, #4]
+; CHECK-NEXT:    add.w r10, r10, #8
+; CHECK-NEXT:    vfma.f16 q0, q1, r8
+; CHECK-NEXT:    vstrb.8 q0, [r2], #8
 ; CHECK-NEXT:    le lr, .LBB15_3
 ; CHECK-NEXT:  @ %bb.4: @ %while.end.loopexit
 ; CHECK-NEXT:    ldr r2, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    ldr r1, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    ldr r5, [sp, #12] @ 4-byte Reload
 ; CHECK-NEXT:    add.w r12, r12, r2, lsl #1
 ; CHECK-NEXT:    add.w r1, r1, r2, lsl #1
 ; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:  .LBB15_5: @ %while.end
-; CHECK-NEXT:    and r7, r3, #3
-; CHECK-NEXT:    vldrw.u32 q2, [r1]
-; CHECK-NEXT:    vctp.16 r7
+; CHECK-NEXT:    and lr, r3, #3
+; CHECK-NEXT:    vldrw.u32 q0, [r1]
+; CHECK-NEXT:    vctp.16 lr
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r4]
-; CHECK-NEXT:    vldrw.u32 q2, [r12]
-; CHECK-NEXT:    vmov.f16 r1, s6
-; CHECK-NEXT:    add.w r7, r12, #2
-; CHECK-NEXT:    vmul.f16 q2, q2, r1
-; CHECK-NEXT:    vmov.f16 r1, s4
-; CHECK-NEXT:    vldrw.u32 q1, [r7]
-; CHECK-NEXT:    add.w r7, r12, #6
-; CHECK-NEXT:    vfma.f16 q2, q1, r1
+; CHECK-NEXT:    vstrht.16 q0, [r4]
+; CHECK-NEXT:    vldrw.u32 q0, [r12]
+; CHECK-NEXT:    add.w r1, r12, #2
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    add.w r1, r12, #6
+; CHECK-NEXT:    vmul.f16 q0, q0, r7
+; CHECK-NEXT:    vfma.f16 q0, q1, r6
 ; CHECK-NEXT:    vldrw.u32 q1, [r12, #4]
-; CHECK-NEXT:    vmov.f16 r1, s2
-; CHECK-NEXT:    vfma.f16 q2, q1, r1
-; CHECK-NEXT:    vmov.f16 r1, s0
-; CHECK-NEXT:    vldrw.u32 q0, [r7]
-; CHECK-NEXT:    vfma.f16 q2, q0, r1
+; CHECK-NEXT:    vfma.f16 q0, q1, r9
+; CHECK-NEXT:    vldrw.u32 q1, [r1]
+; CHECK-NEXT:    vfma.f16 q0, q1, r8
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q2, [r2]
+; CHECK-NEXT:    vstrht.16 q0, [r2]
 ; CHECK-NEXT:    ldr.w r12, [r0, #4]
 ; CHECK-NEXT:  .LBB15_6: @ %if.end
 ; CHECK-NEXT:    add.w r0, r12, r3, lsl #1
 ; CHECK-NEXT:    lsr.w lr, r5, #2
 ; CHECK-NEXT:    wls lr, lr, .LBB15_10
 ; CHECK-NEXT:  @ %bb.7: @ %while.body51.preheader
 ; CHECK-NEXT:    bic r2, r5, #3
 ; CHECK-NEXT:    adds r1, r2, r3

llvm/test/CodeGen/Thumb2/mve-float32regloops.ll

 }
 
 %struct.arm_fir_instance_f32 = type { i16, float*, float* }
 define void @arm_fir_f32_1_4_mve(%struct.arm_fir_instance_f32* nocapture readonly %S, float* nocapture readonly %pSrc, float* %pDst, i32 %blockSize) {
 ; CHECK-LABEL: arm_fir_f32_1_4_mve:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
-; CHECK-NEXT:    .pad #4
-; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    .vsave {d8, d9}
-; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    .pad #8
 ; CHECK-NEXT:    sub sp, #8
-; CHECK-NEXT:    ldrh.w r10, [r0]
-; CHECK-NEXT:    mov r9, r1
+; CHECK-NEXT:    ldrh.w r9, [r0]
+; CHECK-NEXT:    mov r11, r1
 ; CHECK-NEXT:    ldr.w r12, [r0, #4]
-; CHECK-NEXT:    sub.w r1, r10, #1
+; CHECK-NEXT:    sub.w r1, r9, #1
 ; CHECK-NEXT:    cmp r1, #3
 ; CHECK-NEXT:    bhi .LBB15_6
 ; CHECK-NEXT:  @ %bb.1: @ %if.then
-; CHECK-NEXT:    ldr r7, [r0, #8]
-; CHECK-NEXT:    add.w r4, r12, r1, lsl #2
+; CHECK-NEXT:    ldr r4, [r0, #8]
 ; CHECK-NEXT:    lsr.w lr, r3, #2
-; CHECK-NEXT:    vldr s0, [r7]
-; CHECK-NEXT:    vldr s2, [r7, #4]
-; CHECK-NEXT:    vldr s4, [r7, #8]
-; CHECK-NEXT:    vldr s6, [r7, #12]
+; CHECK-NEXT:    ldrd r7, r6, [r4]
+; CHECK-NEXT:    ldrd r5, r8, [r4, #8]
+; CHECK-NEXT:    add.w r4, r12, r1, lsl #2
 ; CHECK-NEXT:    wls lr, lr, .LBB15_5
 ; CHECK-NEXT:  @ %bb.2: @ %while.body.lr.ph
-; CHECK-NEXT:    vmov r11, s4
 ; CHECK-NEXT:    bic r1, r3, #3
-; CHECK-NEXT:    vmov r5, s6
 ; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
-; CHECK-NEXT:    vmov r7, s2
+; CHECK-NEXT:    add.w r10, r12, #4
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
-; CHECK-NEXT:    vmov r8, s0
-; CHECK-NEXT:    add.w r6, r12, #4
 ; CHECK-NEXT:    str r1, [sp, #4] @ 4-byte Spill
-; CHECK-NEXT:    mov r1, r9
+; CHECK-NEXT:    mov r1, r11
 ; CHECK-NEXT:  .LBB15_3: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
-; CHECK-NEXT:    vstrb.8 q2, [r4], #16
-; CHECK-NEXT:    vldrw.u32 q2, [r6, #-4]
-; CHECK-NEXT:    vldrw.u32 q3, [r6], #16
-; CHECK-NEXT:    vmul.f32 q2, q2, r8
-; CHECK-NEXT:    vldrw.u32 q4, [r6, #-8]
-; CHECK-NEXT:    vfma.f32 q2, q3, r7
-; CHECK-NEXT:    vldrw.u32 q3, [r6, #-12]
-; CHECK-NEXT:    vfma.f32 q2, q3, r11
-; CHECK-NEXT:    vfma.f32 q2, q4, r5
-; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
+; CHECK-NEXT:    vstrb.8 q0, [r4], #16
+; CHECK-NEXT:    vldrw.u32 q0, [r10, #-4]
+; CHECK-NEXT:    vldrw.u32 q1, [r10], #16
+; CHECK-NEXT:    vmul.f32 q0, q0, r7
+; CHECK-NEXT:    vldrw.u32 q2, [r10, #-8]
+; CHECK-NEXT:    vfma.f32 q0, q1, r6
+; CHECK-NEXT:    vldrw.u32 q1, [r10, #-12]
+; CHECK-NEXT:    vfma.f32 q0, q1, r5
+; CHECK-NEXT:    vfma.f32 q0, q2, r8
+; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    le lr, .LBB15_3
 ; CHECK-NEXT:  @ %bb.4: @ %while.end.loopexit
 ; CHECK-NEXT:    ldr r1, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    add.w r12, r12, r1, lsl #2
-; CHECK-NEXT:    add.w r9, r9, r1, lsl #2
+; CHECK-NEXT:    add.w r11, r11, r1, lsl #2
 ; CHECK-NEXT:  .LBB15_5: @ %while.end
-; CHECK-NEXT:    and r6, r3, #3
-; CHECK-NEXT:    vmov lr, s6
-; CHECK-NEXT:    vmov r7, s4
-; CHECK-NEXT:    vldrw.u32 q1, [r9]
-; CHECK-NEXT:    vctp.32 r6
+; CHECK-NEXT:    and r1, r3, #3
+; CHECK-NEXT:    vldrw.u32 q0, [r11]
+; CHECK-NEXT:    vctp.32 r1
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q1, [r4]
-; CHECK-NEXT:    vmov r5, s2
-; CHECK-NEXT:    vldrw.u32 q1, [r12, #4]
-; CHECK-NEXT:    vmov r1, s0
+; CHECK-NEXT:    vstrwt.32 q0, [r4]
 ; CHECK-NEXT:    vldrw.u32 q0, [r12]
-; CHECK-NEXT:    vmul.f32 q0, q0, r1
-; CHECK-NEXT:    vfma.f32 q0, q1, r5
+; CHECK-NEXT:    vldrw.u32 q1, [r12, #4]
+; CHECK-NEXT:    vmul.f32 q0, q0, r7
+; CHECK-NEXT:    vfma.f32 q0, q1, r6
 ; CHECK-NEXT:    vldrw.u32 q1, [r12, #8]
-; CHECK-NEXT:    vfma.f32 q0, q1, r7
+; CHECK-NEXT:    vfma.f32 q0, q1, r5
 ; CHECK-NEXT:    vldrw.u32 q1, [r12, #12]
-; CHECK-NEXT:    vfma.f32 q0, q1, lr
+; CHECK-NEXT:    vfma.f32 q0, q1, r8
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q0, [r2]
 ; CHECK-NEXT:    ldr.w r12, [r0, #4]
 ; CHECK-NEXT:  .LBB15_6: @ %if.end
 ; CHECK-NEXT:    add.w r0, r12, r3, lsl #2
-; CHECK-NEXT:    lsr.w lr, r10, #2
+; CHECK-NEXT:    lsr.w lr, r9, #2
 ; CHECK-NEXT:    wls lr, lr, .LBB15_10
 ; CHECK-NEXT:  @ %bb.7: @ %while.body51.preheader
-; CHECK-NEXT:    bic r2, r10, #3
+; CHECK-NEXT:    bic r2, r9, #3
 ; CHECK-NEXT:    adds r1, r2, r3
 ; CHECK-NEXT:    mov r3, r12
 ; CHECK-NEXT:    add.w r1, r12, r1, lsl #2
 ; CHECK-NEXT:  .LBB15_8: @ %while.body51
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vstrb.8 q0, [r3], #16
 ; CHECK-NEXT:    le lr, .LBB15_8
 ; CHECK-NEXT:  @ %bb.9: @ %while.end55.loopexit
 ; CHECK-NEXT:    add.w r12, r12, r2, lsl #2
 ; CHECK-NEXT:    mov r0, r1
 ; CHECK-NEXT:  .LBB15_10: @ %while.end55
-; CHECK-NEXT:    ands r1, r10, #3
+; CHECK-NEXT:    ands r1, r9, #3
 ; CHECK-NEXT:    beq .LBB15_12
 ; CHECK-NEXT:  @ %bb.11: @ %if.then59
 ; CHECK-NEXT:    vldrw.u32 q0, [r0]
 ; CHECK-NEXT:    vctp.32 r1
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q0, [r12]
 ; CHECK-NEXT:  .LBB15_12: @ %if.end61
 ; CHECK-NEXT:    add sp, #8
-; CHECK-NEXT:    vpop {d8, d9}
-; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 entry:
   %pState1 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 1
   %0 = load float*, float** %pState1, align 4
   %pCoeffs2 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 2
   %1 = load float*, float** %pCoeffs2, align 4
   %numTaps3 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 0
   %2 = load i16, i16* %numTaps3, align 4

 %struct.arm_biquad_cascade_stereo_df2T_instance_f32 = type { i8, float*, float* }
 define arm_aapcs_vfpcc void @arm_biquad_cascade_stereo_df2T_f32(%struct.arm_biquad_cascade_stereo_df2T_instance_f32* nocapture readonly %0, float* %1, float* %2, i32 %3) {
 ; CHECK-LABEL: arm_biquad_cascade_stereo_df2T_f32:
 ; CHECK:       @ %bb.0:
 ; CHECK-NEXT:    .save {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r6, r7, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
-; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13}
-; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13}
+; CHECK-NEXT:    .vsave {d8, d9, d10, d11}
+; CHECK-NEXT:    vpush {d8, d9, d10, d11}
 ; CHECK-NEXT:    .pad #24
 ; CHECK-NEXT:    sub sp, #24
 ; CHECK-NEXT:    ldrb.w lr, [r0]
 ; CHECK-NEXT:    movs r4, #0
 ; CHECK-NEXT:    ldrd r12, r0, [r0, #4]
 ; CHECK-NEXT:    cmp r3, #0
 ; CHECK-NEXT:    strd r4, r4, [sp, #16]
 ; CHECK-NEXT:    beq .LBB17_5
 ; CHECK-NEXT:  @ %bb.1:
 ; CHECK-NEXT:    movs r5, #2
 ; CHECK-NEXT:    viwdup.u32 q0, r4, r5, #1
 ; CHECK-NEXT:    mov r4, sp
 ; CHECK-NEXT:  .LBB17_2: @ =>This Loop Header: Depth=1
 ; CHECK-NEXT:    @ Child Loop BB17_3 Depth 2
-; CHECK-NEXT:    vldr s8, [r0, #4]
-; CHECK-NEXT:    vldrw.u32 q4, [r12]
-; CHECK-NEXT:    vldr s12, [r0, #12]
 ; CHECK-NEXT:    mov r7, lr
-; CHECK-NEXT:    vldr s10, [r0, #8]
-; CHECK-NEXT:    mov r5, r2
+; CHECK-NEXT:    ldr.w lr, [r0, #12]
+; CHECK-NEXT:    ldrd r5, r6, [r0]
+; CHECK-NEXT:    vldrw.u32 q1, [r12]
+; CHECK-NEXT:    vldr s12, [r0, #8]
+; CHECK-NEXT:    vdup.32 q2, lr
 ; CHECK-NEXT:    vldr s14, [r0, #16]
-; CHECK-NEXT:    vmov.f32 s9, s8
-; CHECK-NEXT:    vmov.f32 s13, s12
-; CHECK-NEXT:    vldr s4, [r0]
-; CHECK-NEXT:    vmov.f32 s15, s14
-; CHECK-NEXT:    vstrw.32 q4, [r4]
-; CHECK-NEXT:    vmov.f32 s11, s10
+; CHECK-NEXT:    vstrw.32 q1, [r4]
+; CHECK-NEXT:    vdup.32 q1, r6
+; CHECK-NEXT:    mov r6, r2
+; CHECK-NEXT:    vmov.f32 s6, s12
 ; CHECK-NEXT:    dls lr, r3
+; CHECK-NEXT:    vmov.f32 s10, s14
+; CHECK-NEXT:    vmov.f32 s7, s12
+; CHECK-NEXT:    vmov.f32 s11, s14
 ; CHECK-NEXT:  .LBB17_3: @ Parent Loop BB17_2 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
-; CHECK-NEXT:    vldrw.u32 q5, [r1, q0, uxtw #2]
-; CHECK-NEXT:    vldrw.u32 q6, [r4, q0, uxtw #2]
-; CHECK-NEXT:    vmov r6, s4
+; CHECK-NEXT:    vldrw.u32 q4, [r1, q0, uxtw #2]
+; CHECK-NEXT:    vldrw.u32 q5, [r4, q0, uxtw #2]
 ; CHECK-NEXT:    adds r1, #8
-; CHECK-NEXT:    vfma.f32 q6, q5, r6
-; CHECK-NEXT:    vstmia r5, {s24, s25}
-; CHECK-NEXT:    adds r5, #8
-; CHECK-NEXT:    vldrw.u32 q4, [sp, #8]
-; CHECK-NEXT:    vfma.f32 q4, q6, q3
-; CHECK-NEXT:    vfma.f32 q4, q5, q2
-; CHECK-NEXT:    vstrw.32 q4, [r4]
+; CHECK-NEXT:    vfma.f32 q5, q4, r5
+; CHECK-NEXT:    vstmia r6, {s20, s21}
+; CHECK-NEXT:    adds r6, #8
+; CHECK-NEXT:    vldrw.u32 q3, [sp, #8]
+; CHECK-NEXT:    vfma.f32 q3, q5, q2
+; CHECK-NEXT:    vfma.f32 q3, q4, q1
+; CHECK-NEXT:    vstrw.32 q3, [r4]
 ; CHECK-NEXT:    le lr, .LBB17_3
 ; CHECK-NEXT:  @ %bb.4: @ in Loop: Header=BB17_2 Depth=1
 ; CHECK-NEXT:    mov lr, r7
 ; CHECK-NEXT:    adds r0, #20
 ; CHECK-NEXT:    subs.w lr, r7, #1
-; CHECK-NEXT:    vstrb.8 q4, [r12], #16
+; CHECK-NEXT:    vstrb.8 q3, [r12], #16
 ; CHECK-NEXT:    mov r1, r2
 ; CHECK-NEXT:    bne .LBB17_2
 ; CHECK-NEXT:    b .LBB17_7
 ; CHECK-NEXT:  .LBB17_5: @ %.preheader
 ; CHECK-NEXT:    mov r0, sp
 ; CHECK-NEXT:  .LBB17_6: @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r12], #16
 ; CHECK-NEXT:    vstrw.32 q0, [r0]
 ; CHECK-NEXT:    le lr, .LBB17_6
 ; CHECK-NEXT:  .LBB17_7:
 ; CHECK-NEXT:    add sp, #24
-; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
+; CHECK-NEXT:    vpop {d8, d9, d10, d11}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop {r4, r5, r6, r7, pc}
   %5 = alloca [6 x float], align 4
   %6 = getelementptr inbounds %struct.arm_biquad_cascade_stereo_df2T_instance_f32, %struct.arm_biquad_cascade_stereo_df2T_instance_f32* %0, i32 0, i32 1
   %7 = load float*, float** %6, align 4
   %8 = getelementptr inbounds %struct.arm_biquad_cascade_stereo_df2T_instance_f32, %struct.arm_biquad_cascade_stereo_df2T_instance_f32* %0, i32 0, i32 2
   %9 = load float*, float** %8, align 4
   %10 = getelementptr inbounds %struct.arm_biquad_cascade_stereo_df2T_instance_f32, %struct.arm_biquad_cascade_stereo_df2T_instance_f32* %0, i32 0, i32 0

llvm/test/CodeGen/Thumb2/mve-fma-loops.ll

 define arm_aapcs_vfpcc void @fmss1(float* nocapture readonly %x, float* nocapture readonly %y, float* noalias nocapture %z, float %a, i32 %n) {
 ; CHECK-LABEL: fmss1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    poplt {r7, pc}
-; CHECK-NEXT:    vneg.f32 s0, s0
-; CHECK-NEXT:    dlstp.32 lr, r3
 ; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    dlstp.32 lr, r3
+; CHECK-NEXT:    eor r12, r12, #-2147483648
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    vfmas.f32 q1, q0, r12
 ; CHECK-NEXT:    vstrw.32 q1, [r2], #16
 ; CHECK-NEXT:    letp lr, .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

 define arm_aapcs_vfpcc void @fms1(float* nocapture readonly %x, float* nocapture readonly %y, float* noalias nocapture %z, float %a, i32 %n) {
 ; CHECK-LABEL: fms1:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    poplt {r7, pc}
-; CHECK-NEXT:    vneg.f32 s0, s0
-; CHECK-NEXT:    dlstp.32 lr, r3
 ; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    dlstp.32 lr, r3
+; CHECK-NEXT:    eor r12, r12, #-2147483648
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    vfma.f32 q1, q0, r12
 ; CHECK-NEXT:    vstrw.32 q1, [r2], #16
 ; CHECK-NEXT:    letp lr, .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup

llvm/test/CodeGen/Thumb2/mve-intrinsics/dup.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve.fp -verify-machineinstrs -o - %s | FileCheck %s
 
 define arm_aapcs_vfpcc <8 x half> @test_vdupq_n_f16(float %a.coerce) {
 ; CHECK-LABEL: test_vdupq_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r0, s0
+; CHECK-NEXT:    vmov r0, s0
 ; CHECK-NEXT:    vdup.16 q0, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %a.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
entry:
   %.splatinsert = insertelement <4 x i32> undef, i32 %a, i32 0
   %.splat = shufflevector <4 x i32> %.splatinsert, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %.splat
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vdupq_m_n_f16(<8 x half> %inactive, float %a.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vdupq_m_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s4
+; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vdupt.16 q0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %a.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half

llvm/test/CodeGen/Thumb2/mve-intrinsics/ternary.ll

 entry:
   %0 = tail call <4 x float> @llvm.fma.v4f32(<4 x float> %b, <4 x float> %c, <4 x float> %a)
   ret <4 x float> %0
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vfmaq_n_f16(<8 x half> %a, <8 x half> %b, float %c.coerce) {
 ; CHECK-LABEL: test_vfmaq_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r0, s8
+; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vfma.f16 q0, q1, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %c.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
entry:
   %.splat = shufflevector <4 x float> %.splatinsert, <4 x float> undef, <4 x i32> zeroinitializer
   %0 = tail call <4 x float> @llvm.fma.v4f32(<4 x float> %b, <4 x float> %.splat, <4 x float> %a)
   ret <4 x float> %0
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vfmasq_n_f16(<8 x half> %a, <8 x half> %b, float %c.coerce) {
 ; CHECK-LABEL: test_vfmasq_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r0, s8
+; CHECK-NEXT:    vmov r0, s8
 ; CHECK-NEXT:    vfmas.f16 q0, q1, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %c.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
entry:
   %1 = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %0)
   %2 = tail call <4 x float> @llvm.arm.mve.fma.predicated.v4f32.v4i1(<4 x float> %b, <4 x float> %c, <4 x float> %a, <4 x i1> %1)
   ret <4 x float> %2
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vfmaq_m_n_f16(<8 x half> %a, <8 x half> %b, float %c.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vfmaq_m_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s8
+; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vfmat.f16 q0, q1, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %c.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
entry:
   %1 = tail call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %0)
   %2 = tail call <4 x float> @llvm.arm.mve.fma.predicated.v4f32.v4i1(<4 x float> %b, <4 x float> %.splat, <4 x float> %a, <4 x i1> %1)
   ret <4 x float> %2
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vfmasq_m_n_f16(<8 x half> %a, <8 x half> %b, float %c.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vfmasq_m_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s8
+; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vfmast.f16 q0, q1, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %c.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half

llvm/test/CodeGen/Thumb2/mve-intrinsics/vaddq.ll

entry:
   %.splat = shufflevector <4 x i32> %.splatinsert, <4 x i32> undef, <4 x i32> zeroinitializer
   %0 = add <4 x i32> %.splat, %a
   ret <4 x i32> %0
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vaddq_n_f16(<8 x half> %a, float %b.coerce) {
 ; CHECK-LABEL: test_vaddq_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r0, s4
+; CHECK-NEXT:    vmov r0, s4
 ; CHECK-NEXT:    vadd.f16 q0, q0, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %b.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
entry:
   %1 = call <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32 %0)
   %2 = call <8 x i16> @llvm.arm.mve.add.predicated.v8i16.v8i1(<8 x i16> %a, <8 x i16> %.splat, <8 x i1> %1, <8 x i16> undef)
   ret <8 x i16> %2
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vaddq_x_n_f16(<8 x half> %a, float %b.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vaddq_x_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s4
+; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vaddt.f16 q0, q0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %b.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
   %2 = zext i16 %p to i32
   %3 = call <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32 %2)
   %4 = call <8 x half> @llvm.arm.mve.add.predicated.v8f16.v8i1(<8 x half> %a, <8 x half> %.splat, <8 x i1> %3, <8 x half> undef)
   ret <8 x half> %4
 }

llvm/test/CodeGen/Thumb2/mve-intrinsics/vmulq.ll

entry:
   %1 = call <4 x i1> @llvm.arm.mve.pred.i2v.v4i1(i32 %0)
   %2 = call <4 x i32> @llvm.arm.mve.mul.predicated.v4i32.v4i1(<4 x i32> %a, <4 x i32> %.splat, <4 x i1> %1, <4 x i32> %inactive)
   ret <4 x i32> %2
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vmulq_m_n_f16(<8 x half> %inactive, <8 x half> %a, float %b.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vmulq_m_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s8
+; CHECK-NEXT:    vmov r1, s8
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vmult.f16 q0, q1, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %b.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half

llvm/test/CodeGen/Thumb2/mve-intrinsics/vsubq.ll

entry:
   %.splat = shufflevector <4 x i32> %.splatinsert, <4 x i32> undef, <4 x i32> zeroinitializer
   %0 = sub <4 x i32> %a, %.splat
   ret <4 x i32> %0
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vsubq_n_f16(<8 x half> %a, float %b.coerce) {
 ; CHECK-LABEL: test_vsubq_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r0, s4
+; CHECK-NEXT:    vmov r0, s4
 ; CHECK-NEXT:    vsub.f16 q0, q0, r0
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %b.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
entry:
   %1 = call <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32 %0)
   %2 = call <8 x i16> @llvm.arm.mve.sub.predicated.v8i16.v8i1(<8 x i16> %a, <8 x i16> %.splat, <8 x i1> %1, <8 x i16> undef)
   ret <8 x i16> %2
 }
 
 define arm_aapcs_vfpcc <8 x half> @test_vsubq_x_n_f16(<8 x half> %a, float %b.coerce, i16 zeroext %p) {
 ; CHECK-LABEL: test_vsubq_x_n_f16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.f16 r1, s4
+; CHECK-NEXT:    vmov r1, s4
 ; CHECK-NEXT:    vmsr p0, r0
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vsubt.f16 q0, q0, r1
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = bitcast float %b.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %.splatinsert = insertelement <8 x half> undef, half %1, i32 0
   %.splat = shufflevector <8 x half> %.splatinsert, <8 x half> undef, <8 x i32> zeroinitializer
   %2 = zext i16 %p to i32
   %3 = call <8 x i1> @llvm.arm.mve.pred.i2v.v8i1(i32 %2)
   %4 = call <8 x half> @llvm.arm.mve.sub.predicated.v8f16.v8i1(<8 x half> %a, <8 x half> %.splat, <8 x i1> %3, <8 x half> undef)
   ret <8 x half> %4
 }

llvm/test/CodeGen/Thumb2/mve-postinc-lsr.ll

 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
 ; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
 ; CHECK-NEXT:    vadd.i32 q1, q1, r0
 ; CHECK-NEXT:    vldrw.u32 q2, [q1, #64]!
 ; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
 ; CHECK-NEXT:    lsr.w lr, r0, #3
 ; CHECK-NEXT:    wls lr, lr, .LBB7_12
 ; CHECK-NEXT:  @ %bb.10:
-; CHECK-NEXT:    vldrw.u32 q3, [q1, #16]
 ; CHECK-NEXT:    vldr s0, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    vmov r0, s0
+; CHECK-NEXT:    vldrw.u32 q0, [q1, #16]
 ; CHECK-NEXT:  .LBB7_11: @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q0, [q1, #24]
+; CHECK-NEXT:    vldrw.u32 q3, [q1, #24]
 ; CHECK-NEXT:    vldrw.u32 q4, [q1, #8]
-; CHECK-NEXT:    vadd.f32 q6, q2, q3
-; CHECK-NEXT:    vsub.f32 q2, q2, q3
-; CHECK-NEXT:    vadd.f32 q5, q4, q0
-; CHECK-NEXT:    vsub.f32 q0, q4, q0
-; CHECK-NEXT:    vsub.f32 q7, q6, q5
-; CHECK-NEXT:    vcadd.f32 q4, q2, q0, #270
+; CHECK-NEXT:    vsub.f32 q6, q2, q0
+; CHECK-NEXT:    vadd.f32 q0, q2, q0
+; CHECK-NEXT:    vsub.f32 q5, q4, q3
+; CHECK-NEXT:    vadd.f32 q3, q4, q3
+; CHECK-NEXT:    vcadd.f32 q7, q6, q5, #270
+; CHECK-NEXT:    vsub.f32 q2, q0, q3
+; CHECK-NEXT:    vmul.f32 q7, q7, r0
+; CHECK-NEXT:    vadd.f32 q3, q0, q3
 ; CHECK-NEXT:    vstrw.32 q7, [sp, #32] @ 16-byte Spill
-; CHECK-NEXT:    vcadd.f32 q7, q2, q0, #90
-; CHECK-NEXT:    vadd.f32 q0, q6, q5
+; CHECK-NEXT:    vcadd.f32 q7, q6, q5, #90
+; CHECK-NEXT:    vmul.f32 q4, q2, r0
 ; CHECK-NEXT:    vldrw.u32 q2, [q1, #64]!
-; CHECK-NEXT:    vmul.f32 q0, q0, r0
-; CHECK-NEXT:    vldrw.u32 q3, [q1, #16]
-; CHECK-NEXT:    vstrw.32 q0, [q1, #-64]
-; CHECK-NEXT:    vldrw.u32 q5, [sp, #32] @ 16-byte Reload
-; CHECK-NEXT:    vmul.f32 q0, q4, r0
-; CHECK-NEXT:    vmul.f32 q4, q7, r0
-; CHECK-NEXT:    vmul.f32 q5, q5, r0
-; CHECK-NEXT:    vstrw.32 q5, [q1, #-56]
-; CHECK-NEXT:    vstrw.32 q4, [q1, #-48]
-; CHECK-NEXT:    vstrw.32 q0, [q1, #-40]
+; CHECK-NEXT:    vmul.f32 q5, q7, r0
+; CHECK-NEXT:    vmul.f32 q3, q3, r0
+; CHECK-NEXT:    vldrw.u32 q0, [q1, #16]
+; CHECK-NEXT:    vstrw.32 q3, [q1, #-64]
+; CHECK-NEXT:    vstrw.32 q4, [q1, #-56]
+; CHECK-NEXT:    vstrw.32 q5, [q1, #-48]
+; CHECK-NEXT:    vldrw.u32 q3, [sp, #32] @ 16-byte Reload
+; CHECK-NEXT:    vstrw.32 q3, [q1, #-40]
 ; CHECK-NEXT:    le lr, .LBB7_11
 ; CHECK-NEXT:  .LBB7_12:
 ; CHECK-NEXT:    add sp, #56
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 ; CHECK-NEXT:    .p2align 4
 ; CHECK-NEXT:  @ %bb.13:

llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll

 define arm_aapcs_vfpcc void @thresh_f32(float* %data, i16 zeroext %N, float %T) {
 ; CHECK-LABEL: thresh_f32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
-; CHECK-NEXT:    vneg.f32 s2, s0
 ; CHECK-NEXT:    mvn r2, #3
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
 ; CHECK-NEXT:    movs r2, #1
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #2
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s2
+; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    eor r2, r1, #-2147483648
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpte.f32 ge, q1, r2
-; CHECK-NEXT:    vcmpt.f32 le, q1, r1
+; CHECK-NEXT:    vpte.f32 ge, q1, r1
+; CHECK-NEXT:    vcmpt.f32 le, q1, r2
 ; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0

 define arm_aapcs_vfpcc void @thresh_f16(half* %data, i16 zeroext %N, float %T.coerce) {
 ; CHECK-LABEL: thresh_f16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
-; CHECK-NEXT:    mvn r2, #7
-; CHECK-NEXT:    add.w r1, r2, r1, lsl #3
-; CHECK-NEXT:    movs r2, #1
-; CHECK-NEXT:    vneg.f16 s2, s0
-; CHECK-NEXT:    add.w lr, r2, r1, lsr #3
-; CHECK-NEXT:    vmov.f16 r1, s2
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    mvn r3, #7
+; CHECK-NEXT:    add.w r1, r3, r1, lsl #3
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vneg.f16 s0, s0
+; CHECK-NEXT:    movs r3, #1
+; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
+; CHECK-NEXT:    vmov.f16 r1, s0
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
 ; CHECK-NEXT:    vpte.f16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.f16 le, q1, r1
 ; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB4_1

 define arm_aapcs_vfpcc void @thresh_rev_f32(float* %data, i16 zeroext %N, float %T) {
 ; CHECK-LABEL: thresh_rev_f32:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
-; CHECK-NEXT:    vneg.f32 s2, s0
 ; CHECK-NEXT:    mvn r2, #3
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
 ; CHECK-NEXT:    movs r2, #1
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #2
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmov r1, s2
+; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
+; CHECK-NEXT:    eor r2, r1, #-2147483648
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
-; CHECK-NEXT:    vpte.f32 ge, q1, r2
-; CHECK-NEXT:    vcmpt.f32 le, q1, r1
+; CHECK-NEXT:    vpte.f32 ge, q1, r1
+; CHECK-NEXT:    vcmpt.f32 le, q1, r2
 ; CHECK-NEXT:    vstrwe.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0

 define arm_aapcs_vfpcc void @thresh_rev_f16(half* %data, i16 zeroext %N, float %T.coerce) {
 ; CHECK-LABEL: thresh_rev_f16:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    .save {r7, lr}
 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
-; CHECK-NEXT:    mvn r2, #7
-; CHECK-NEXT:    add.w r1, r2, r1, lsl #3
-; CHECK-NEXT:    movs r2, #1
-; CHECK-NEXT:    vneg.f16 s2, s0
-; CHECK-NEXT:    add.w lr, r2, r1, lsr #3
-; CHECK-NEXT:    vmov.f16 r1, s2
-; CHECK-NEXT:    vmov.f16 r2, s0
+; CHECK-NEXT:    mvn r3, #7
+; CHECK-NEXT:    add.w r1, r3, r1, lsl #3
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vneg.f16 s0, s0
+; CHECK-NEXT:    movs r3, #1
+; CHECK-NEXT:    add.w lr, r3, r1, lsr #3
+; CHECK-NEXT:    vmov.f16 r1, s0
 ; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrh.u16 q1, [r0]
 ; CHECK-NEXT:    vpte.f16 ge, q1, r2
 ; CHECK-NEXT:    vcmpt.f16 le, q1, r1
 ; CHECK-NEXT:    vstrhe.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB9_1

llvm/test/CodeGen/Thumb2/mve-vldst4.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-none-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s
 
 define void @vldst4(half* nocapture readonly %pIn, half* nocapture %pOut, i32 %numRows, i32 %numCols, i32 %scale.coerce) #0 {
 ; CHECK-LABEL: vldst4:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    .save {r4, lr}
-; CHECK-NEXT:    push {r4, lr}
+; CHECK-NEXT:    .save {r4, r5, r7, lr}
+; CHECK-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-NEXT:    .vsave {d8, d9, d10, d11, d12, d13, d14, d15}
 ; CHECK-NEXT:    vpush {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    .pad #88
-; CHECK-NEXT:    sub sp, #88
-; CHECK-NEXT:    muls r2, r3, r2
-; CHECK-NEXT:    movs r3, #0
-; CHECK-NEXT:    cmp.w r3, r2, lsr #2
+; CHECK-NEXT:    .pad #80
+; CHECK-NEXT:    sub sp, #80
+; CHECK-NEXT:    mul r12, r3, r2
+; CHECK-NEXT:    movs r2, #0
+; CHECK-NEXT:    cmp.w r2, r12, lsr #2
 ; CHECK-NEXT:    beq.w .LBB0_3
 ; CHECK-NEXT:  @ %bb.1: @ %vector.ph
-; CHECK-NEXT:    vldr.16 s0, [sp, #160]
 ; CHECK-NEXT:    mvn r3, #7
-; CHECK-NEXT:    and.w r2, r3, r2, lsr #2
+; CHECK-NEXT:    ldr r5, [sp, #160]
+; CHECK-NEXT:    and.w r3, r3, r12, lsr #2
+; CHECK-NEXT:    sub.w r12, r3, #8
 ; CHECK-NEXT:    movs r3, #1
-; CHECK-NEXT:    vmov.f16 r12, s0
-; CHECK-NEXT:    subs r2, #8
-; CHECK-NEXT:    add.w lr, r3, r2, lsr #3
+; CHECK-NEXT:    add.w lr, r3, r12, lsr #3
 ; CHECK-NEXT:  .LBB0_2: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrh.u16 q4, [r0, #32]
+; CHECK-NEXT:    vldrh.u16 q5, [r0, #32]
 ; CHECK-NEXT:    vldrh.u16 q3, [r0, #48]
 ; CHECK-NEXT:    vldrh.u16 q7, [r0], #64
-; CHECK-NEXT:    vmov r2, s17
-; CHECK-NEXT:    vmovx.f16 s8, s13
+; CHECK-NEXT:    vmov r2, s20
+; CHECK-NEXT:    vmovx.f16 s8, s12
 ; CHECK-NEXT:    vmov.16 q0[4], r2
-; CHECK-NEXT:    vmov r3, s19
+; CHECK-NEXT:    vmov r3, s22
 ; CHECK-NEXT:    vmov.16 q0[5], r3
-; CHECK-NEXT:    vmov r2, s13
+; CHECK-NEXT:    vmov r2, s12
 ; CHECK-NEXT:    vmov.16 q0[6], r2
-; CHECK-NEXT:    vmov r2, s15
-; CHECK-NEXT:    vmov r3, s29
-; CHECK-NEXT:    vldrh.u16 q5, [r0, #-48]
-; CHECK-NEXT:    vmov.16 q0[7], r2
-; CHECK-NEXT:    vmov r2, s31
-; CHECK-NEXT:    vmov.16 q1[0], r3
-; CHECK-NEXT:    vmov.16 q1[1], r2
-; CHECK-NEXT:    vmov r2, s21
+; CHECK-NEXT:    vmov r2, s28
+; CHECK-NEXT:    vldrh.u16 q6, [r0, #-48]
+; CHECK-NEXT:    vmov.16 q1[0], r2
+; CHECK-NEXT:    vmov r3, s30
+; CHECK-NEXT:    vmov.16 q1[1], r3
+; CHECK-NEXT:    vmov r2, s24
 ; CHECK-NEXT:    vmov.16 q1[2], r2
-; CHECK-NEXT:    vmov r2, s23
+; CHECK-NEXT:    vmov r2, s14
+; CHECK-NEXT:    vmov.16 q0[7], r2
+; CHECK-NEXT:    vmov r2, s26
 ; CHECK-NEXT:    vmov.16 q1[3], r2
 ; CHECK-NEXT:    vmov.f32 s6, s2
 ; CHECK-NEXT:    vmov.f32 s7, s3
-; CHECK-NEXT:    vmul.f16 q0, q1, r12
-; CHECK-NEXT:    vmovx.f16 s4, s21
-; CHECK-NEXT:    vmov q6, q0
-; CHECK-NEXT:    vstrw.32 q0, [sp, #32] @ 16-byte Spill
-; CHECK-NEXT:    vmovx.f16 s0, s31
-; CHECK-NEXT:    vmov r3, s1
+; CHECK-NEXT:    vmul.f16 q0, q1, r5
+; CHECK-NEXT:    vmovx.f16 s4, s24
+; CHECK-NEXT:    vmov q4, q0
+; CHECK-NEXT:    vstrw.32 q0, [sp, #48] @ 16-byte Spill
+; CHECK-NEXT:    vmovx.f16 s0, s30
+; CHECK-NEXT:    vmov r3, s3
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s29
+; CHECK-NEXT:    vmovx.f16 s0, s28
 ; CHECK-NEXT:    vmov r4, s0
 ; CHECK-NEXT:    vmov.16 q0[0], r4
 ; CHECK-NEXT:    vmov.16 q0[1], r2
 ; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmovx.f16 s4, s19
+; CHECK-NEXT:    vmovx.f16 s4, s22
 ; CHECK-NEXT:    vmov.16 q0[2], r2
 ; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmovx.f16 s4, s17
+; CHECK-NEXT:    vmovx.f16 s4, s20
 ; CHECK-NEXT:    vmov r4, s4
 ; CHECK-NEXT:    vmov.16 q1[4], r4
 ; CHECK-NEXT:    vmov.16 q1[5], r2
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmovx.f16 s8, s15
+; CHECK-NEXT:    vmovx.f16 s8, s14
 ; CHECK-NEXT:    vmov.16 q1[6], r2
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmovx.f16 s8, s23
+; CHECK-NEXT:    vmovx.f16 s8, s26
 ; CHECK-NEXT:    vmov.16 q1[7], r2
 ; CHECK-NEXT:    vmov r2, s8
 ; CHECK-NEXT:    vmov.16 q0[3], r2
+; CHECK-NEXT:    vmovx.f16 s8, s13
 ; CHECK-NEXT:    vmov.f32 s2, s6
 ; CHECK-NEXT:    vmov.f32 s3, s7
-; CHECK-NEXT:    vmov.16 q1[2], r3
-; CHECK-NEXT:    vmul.f16 q0, q0, r12
-; CHECK-NEXT:    vmov r3, s18
-; CHECK-NEXT:    vmov r2, s1
-; CHECK-NEXT:    vmov q2, q0
+; CHECK-NEXT:    vmov.16 q1[0], r3
+; CHECK-NEXT:    vmul.f16 q0, q0, r5
+; CHECK-NEXT:    vmov r3, s23
+; CHECK-NEXT:    vmov r2, s3
 ; CHECK-NEXT:    vstrw.32 q0, [sp, #64] @ 16-byte Spill
-; CHECK-NEXT:    vmovx.f16 s0, s25
-; CHECK-NEXT:    vmov.16 q1[3], r2
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s9
-; CHECK-NEXT:    vmov.16 q1[6], r2
+; CHECK-NEXT:    vmovx.f16 s0, s19
+; CHECK-NEXT:    vmov.16 q1[1], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s8, s12
-; CHECK-NEXT:    vmov.16 q1[7], r2
-; CHECK-NEXT:    vmov r2, s16
+; CHECK-NEXT:    vmov.16 q1[4], r2
+; CHECK-NEXT:    vmov r2, s21
 ; CHECK-NEXT:    vmov.16 q0[4], r2
-; CHECK-NEXT:    vmov r2, s12
+; CHECK-NEXT:    vmov r2, s13
 ; CHECK-NEXT:    vmov.16 q0[5], r3
-; CHECK-NEXT:    vmov r3, s28
-; CHECK-NEXT:    vstrw.32 q1, [sp, #48] @ 16-byte Spill
+; CHECK-NEXT:    vmov r3, s29
+; CHECK-NEXT:    vstrw.32 q1, [sp, #16] @ 16-byte Spill
 ; CHECK-NEXT:    vmov.16 q0[6], r2
-; CHECK-NEXT:    vmov r2, s30
+; CHECK-NEXT:    vmov r2, s31
 ; CHECK-NEXT:    vmov.16 q1[0], r3
 ; CHECK-NEXT:    vmov.16 q1[1], r2
-; CHECK-NEXT:    vmov r2, s20
+; CHECK-NEXT:    vmov r2, s25
 ; CHECK-NEXT:    vmov.16 q1[2], r2
-; CHECK-NEXT:    vmov r2, s14
+; CHECK-NEXT:    vmov r2, s15
 ; CHECK-NEXT:    vmov.16 q0[7], r2
-; CHECK-NEXT:    vmov r2, s22
+; CHECK-NEXT:    vmov r2, s27
 ; CHECK-NEXT:    vmov.16 q1[3], r2
 ; CHECK-NEXT:    vmov.f32 s6, s2
 ; CHECK-NEXT:    vmov.f32 s7, s3
-; CHECK-NEXT:    vmovx.f16 s0, s30
+; CHECK-NEXT:    vmovx.f16 s0, s31
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s28
+; CHECK-NEXT:    vmovx.f16 s0, s29
 ; CHECK-NEXT:    vmov r4, s0
-; CHECK-NEXT:    vmul.f16 q6, q1, r12
+; CHECK-NEXT:    vmul.f16 q4, q1, r5
 ; CHECK-NEXT:    vmov.16 q0[0], r4
-; CHECK-NEXT:    vmovx.f16 s4, s20
+; CHECK-NEXT:    vmovx.f16 s4, s25
 ; CHECK-NEXT:    vmov.16 q0[1], r2
 ; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmovx.f16 s4, s18
+; CHECK-NEXT:    vmovx.f16 s4, s23
 ; CHECK-NEXT:    vmov.16 q0[2], r2
 ; CHECK-NEXT:    vmov r2, s4
-; CHECK-NEXT:    vmovx.f16 s4, s16
+; CHECK-NEXT:    vmovx.f16 s4, s21
 ; CHECK-NEXT:    vmov r4, s4
-; CHECK-NEXT:    vldrw.u32 q4, [sp, #32] @ 16-byte Reload
+; CHECK-NEXT:    vstrw.32 q4, [sp, #32] @ 16-byte Spill
 ; CHECK-NEXT:    vmov.16 q1[4], r4
-; CHECK-NEXT:    vmov r3, s25
+; CHECK-NEXT:    vmov r3, s16
 ; CHECK-NEXT:    vmov.16 q1[5], r2
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmovx.f16 s8, s14
+; CHECK-NEXT:    vmovx.f16 s8, s15
 ; CHECK-NEXT:    vmov.16 q1[6], r2
 ; CHECK-NEXT:    vmov r2, s8
-; CHECK-NEXT:    vmovx.f16 s8, s22
+; CHECK-NEXT:    vmovx.f16 s8, s27
 ; CHECK-NEXT:    vmov.16 q1[7], r2
 ; CHECK-NEXT:    vmov r2, s8
 ; CHECK-NEXT:    vmov.16 q0[3], r2
-; CHECK-NEXT:    vmov.16 q5[0], r3
+; CHECK-NEXT:    vldrw.u32 q2, [sp, #48] @ 16-byte Reload
 ; CHECK-NEXT:    vmov.f32 s2, s6
 ; CHECK-NEXT:    vmov.f32 s3, s7
+; CHECK-NEXT:    vmov.16 q1[2], r3
+; CHECK-NEXT:    vmul.f16 q6, q0, r5
+; CHECK-NEXT:    vmovx.f16 s0, s16
+; CHECK-NEXT:    vmov r2, s24
+; CHECK-NEXT:    vmov.16 q1[3], r2
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vmovx.f16 s0, s24
+; CHECK-NEXT:    vmov.16 q1[6], r2
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vmovx.f16 s0, s8
+; CHECK-NEXT:    vmov.16 q1[7], r2
+; CHECK-NEXT:    vmov r2, s8
+; CHECK-NEXT:    vstrw.32 q1, [sp] @ 16-byte Spill
 ; CHECK-NEXT:    vldrw.u32 q1, [sp, #64] @ 16-byte Reload
-; CHECK-NEXT:    vmul.f16 q2, q0, r12
-; CHECK-NEXT:    vmovx.f16 s0, s25
-; CHECK-NEXT:    vmov r2, s9
-; CHECK-NEXT:    vstrw.32 q2, [sp] @ 16-byte Spill
-; CHECK-NEXT:    vmov.16 q5[1], r2
+; CHECK-NEXT:    vmov.16 q5[0], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s9
+; CHECK-NEXT:    vmov r3, s4
+; CHECK-NEXT:    vmovx.f16 s0, s4
+; CHECK-NEXT:    vmov.16 q5[1], r3
+; CHECK-NEXT:    vmov r3, s25
 ; CHECK-NEXT:    vmov.16 q5[4], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s19
 ; CHECK-NEXT:    vmov.16 q5[5], r2
-; CHECK-NEXT:    vmov r2, s19
+; CHECK-NEXT:    vmov r2, s17
 ; CHECK-NEXT:    vmov.16 q3[2], r2
-; CHECK-NEXT:    vmov r3, s7
+; CHECK-NEXT:    vmovx.f16 s0, s17
 ; CHECK-NEXT:    vmov.16 q3[3], r3
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s7
+; CHECK-NEXT:    vmovx.f16 s0, s25
 ; CHECK-NEXT:    vmov.16 q3[6], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s27
+; CHECK-NEXT:    vmovx.f16 s0, s9
 ; CHECK-NEXT:    vmov.16 q3[7], r2
-; CHECK-NEXT:    vmov r2, s27
+; CHECK-NEXT:    vmov r2, s9
 ; CHECK-NEXT:    vmov.16 q7[0], r2
-; CHECK-NEXT:    vmov r3, s11
+; CHECK-NEXT:    vmov r3, s5
 ; CHECK-NEXT:    vmov.16 q7[1], r3
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s11
+; CHECK-NEXT:    vmovx.f16 s0, s5
 ; CHECK-NEXT:    vmov.16 q7[4], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vstrw.32 q3, [sp, #16] @ 16-byte Spill
+; CHECK-NEXT:    vldrw.u32 q0, [sp, #32] @ 16-byte Reload
 ; CHECK-NEXT:    vmov.16 q7[5], r2
-; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmovx.f16 s0, s16
-; CHECK-NEXT:    vmov.16 q3[2], r2
-; CHECK-NEXT:    vmov r3, s4
-; CHECK-NEXT:    vmovx.f16 s16, s18
-; CHECK-NEXT:    vmov.16 q3[3], r3
+; CHECK-NEXT:    vmov r3, s26
+; CHECK-NEXT:    vmov r2, s2
+; CHECK-NEXT:    vmovx.f16 s0, s2
+; CHECK-NEXT:    vmov.16 q2[2], r2
+; CHECK-NEXT:    vmov q4, q1
+; CHECK-NEXT:    vldrw.u32 q1, [sp, #48] @ 16-byte Reload
+; CHECK-NEXT:    vmov.16 q2[3], r3
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s4
-; CHECK-NEXT:    vmov.16 q3[6], r2
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s24
-; CHECK-NEXT:    vmov.16 q3[7], r2
-; CHECK-NEXT:    vmov r2, s24
-; CHECK-NEXT:    vmov.16 q1[0], r2
-; CHECK-NEXT:    vmov r3, s8
-; CHECK-NEXT:    vmov.16 q1[1], r3
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s0, s8
-; CHECK-NEXT:    vmov.16 q1[4], r2
+; CHECK-NEXT:    vmovx.f16 s0, s26
+; CHECK-NEXT:    vmov.16 q2[6], r2
 ; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vmovx.f16 s24, s26
-; CHECK-NEXT:    vmov.16 q1[5], r2
-; CHECK-NEXT:    vmov r2, s26
-; CHECK-NEXT:    vmov r3, s10
+; CHECK-NEXT:    vmov.16 q2[7], r2
+; CHECK-NEXT:    vmov r2, s6
+; CHECK-NEXT:    vmov r3, s18
 ; CHECK-NEXT:    vmov.16 q0[0], r2
+; CHECK-NEXT:    vmovx.f16 s4, s6
 ; CHECK-NEXT:    vmov.16 q0[1], r3
-; CHECK-NEXT:    vmov r2, s24
-; CHECK-NEXT:    vldrw.u32 q2, [sp, #64] @ 16-byte Reload
+; CHECK-NEXT:    vmov r2, s4
+; CHECK-NEXT:    vmovx.f16 s4, s18
+; CHECK-NEXT:    vldrw.u32 q4, [sp, #32] @ 16-byte Reload
 ; CHECK-NEXT:    vmov.16 q0[4], r2
-; CHECK-NEXT:    vmov r2, s18
-; CHECK-NEXT:    vmov.16 q6[2], r2
-; CHECK-NEXT:    vmov r3, s10
+; CHECK-NEXT:    vmov r2, s4
+; CHECK-NEXT:    vmov.16 q0[5], r2
+; CHECK-NEXT:    vmov r2, s19
+; CHECK-NEXT:    vmov r3, s27
+; CHECK-NEXT:    vmov.16 q1[2], r2
+; CHECK-NEXT:    vmovx.f16 s16, s19
+; CHECK-NEXT:    vmov.16 q1[3], r3
 ; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmovx.f16 s16, s10
-; CHECK-NEXT:    vldrw.u32 q2, [sp] @ 16-byte Reload
-; CHECK-NEXT:    vmov.16 q6[3], r3
-; CHECK-NEXT:    vmov.16 q6[6], r2
+; CHECK-NEXT:    vmovx.f16 s16, s27
+; CHECK-NEXT:    vmov.16 q1[6], r2
 ; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmovx.f16 s16, s10
-; CHECK-NEXT:    vmov.16 q6[7], r2
+; CHECK-NEXT:    vldrw.u32 q4, [sp, #64] @ 16-byte Reload
+; CHECK-NEXT:    vmov.16 q1[7], r2
+; CHECK-NEXT:    vmov.f32 s1, s9
+; CHECK-NEXT:    vldrw.u32 q6, [sp] @ 16-byte Reload
+; CHECK-NEXT:    vmovx.f16 s16, s19
+; CHECK-NEXT:    vmov.f32 s3, s11
 ; CHECK-NEXT:    vmov r2, s16
-; CHECK-NEXT:    vmov.16 q0[5], r2
-; CHECK-NEXT:    vmov.f32 s5, s13
-; CHECK-NEXT:    vmov.f32 s1, s25
-; CHECK-NEXT:    vmov.f32 s3, s27
+; CHECK-NEXT:    vldrw.u32 q4, [sp, #16] @ 16-byte Reload
+; CHECK-NEXT:    vmov.f32 s21, s25
 ; CHECK-NEXT:    vstrh.16 q0, [r1, #32]
-; CHECK-NEXT:    vldrw.u32 q0, [sp, #16] @ 16-byte Reload
-; CHECK-NEXT:    vmov.f32 s7, s15
-; CHECK-NEXT:    vmov.f32 s29, s1
-; CHECK-NEXT:    vmov.f32 s31, s3
-; CHECK-NEXT:    vldrw.u32 q0, [sp, #48] @ 16-byte Reload
-; CHECK-NEXT:    vstrh.16 q7, [r1, #48]
-; CHECK-NEXT:    vstrh.16 q1, [r1], #64
-; CHECK-NEXT:    vmov.f32 s21, s1
-; CHECK-NEXT:    vmov.f32 s23, s3
-; CHECK-NEXT:    vstrh.16 q5, [r1, #-48]
+; CHECK-NEXT:    vmov.16 q4[5], r2
+; CHECK-NEXT:    vmov.f32 s29, s13
+; CHECK-NEXT:    vmov q2, q4
+; CHECK-NEXT:    vmov.f32 s23, s27
+; CHECK-NEXT:    vmov.f32 s9, s5
+; CHECK-NEXT:    vmov.f32 s11, s7
+; CHECK-NEXT:    vstrh.16 q2, [r1, #48]
+; CHECK-NEXT:    vstrh.16 q5, [r1], #64
+; CHECK-NEXT:    vmov.f32 s31, s15
+; CHECK-NEXT:    vstrh.16 q7, [r1, #-48]
 ; CHECK-NEXT:    le lr, .LBB0_2
 ; CHECK-NEXT:  .LBB0_3: @ %while.end
-; CHECK-NEXT:    add sp, #88
+; CHECK-NEXT:    add sp, #80
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13, d14, d15}
-; CHECK-NEXT:    pop {r4, pc}
+; CHECK-NEXT:    pop {r4, r5, r7, pc}
 entry:
   %tmp.0.extract.trunc = trunc i32 %scale.coerce to i16
   %l0 = bitcast i16 %tmp.0.extract.trunc to half
   %mul = mul i32 %numCols, %numRows
   %shr = lshr i32 %mul, 2
   %cmp38 = icmp eq i32 %shr, 0
   br i1 %cmp38, label %while.end, label %vector.ph