[MachineLICM] Let targets decide to hoist cheap instructions

[dmgreen]

Summary

MachineLICM can hoist instructions out of loop, but will chooses not to do so for Cheap instructions, including all COPY instructions. Under Cortex-M cpus, where there isn't really a big difference between a MOV and any other instruction, we should really hoist these out of loops, even if they increase the immediate register pressure. MachineLICM will still test if the register pressure limit has been reached, but this puts a shouldHoistCheapInsts target hook in for hoisting instructions that wouldn't otherwise be.

This is especially true for MVE code where we sink VDUP's into blocks attempting to fold them into register variants of vector instructions. Where this scalar is a float value we are left with a COPY from SPR to GPR which needs to be hoisted. Other than that this did not seem to cause a lot of changes.

[efriedma]

I'm not sure what, exactly, is specific to Cortex-M here. Copies between int and SIMD registers aren't really cheaper on other CPUs, relatively speaking. I don't really like adding a new target hook without a better justification for why Cortex-M is different from other targets; having target-specific codepaths is going to make it harder for anyone to make improvements here in the future.

Maybe we should revisit the assumption that copies are cheap, in MachineLICMBase::IsCheapInstruction? Or maybe we should revisit the assumption that we should avoid hoisting cheap instructions?

[dmgreen]

In D76024#1918460, @efriedma wrote:

I'm not sure what, exactly, is specific to Cortex-M here. Copies between int and SIMD registers aren't really cheaper on other CPUs, relatively speaking. I don't really like adding a new target hook without a better justification for why Cortex-M is different from other targets; having target-specific codepaths is going to make it harder for anyone to make improvements here in the future.

Good point about cross copies. I was really thinking that _any_ instruction on Cortex-M isn't really much cheaper than any other (with some obvious exceptions for divisions and the like). That's different from many CPU's where movs become essentially free.

Maybe we should revisit the assumption that copies are cheap, in MachineLICMBase::IsCheapInstruction? Or maybe we should revisit the assumption that we should avoid hoisting cheap instructions?

I've tried some alternatives and put together D76135. Let me know what you think, or if anything else would be preferable.

[efriedma]

The approach in D76135 makes more sense, I think.

llvm/include/llvm/CodeGen/TargetInstrInfo.h

public:
 
   /// Return true if the instruction should be sunk by MachineSink.
   ///
   /// MachineSink determines on its own whether the instruction is safe to sink;
   /// this gives the target a hook to override the default behavior with regards
   /// to which instructions should be sunk.
   virtual bool shouldSink(const MachineInstr &MI) const { return true; }
 
+  /// Return true if the cheap instruction can be hoisted by MachineLICM, given
+  /// the register pressure cost delta in Cost.
+  virtual bool
+  shouldHoistCheapInsts(const MachineInstr &MI,
+                        const DenseMap<unsigned, int> &Cost) const {
+    return false;
+  }
+
   /// Re-issue the specified 'original' instruction at the
   /// specific location targeting a new destination register.
   /// The register in Orig->getOperand(0).getReg() will be substituted by
   /// DestReg:SubIdx. Any existing subreg index is preserved or composed with
   /// SubIdx.
   virtual void reMaterialize(MachineBasicBlock &MBB,
                              MachineBasicBlock::iterator MI, unsigned DestReg,
                              unsigned SubIdx, const MachineInstr &Orig,

llvm/lib/CodeGen/MachineLICM.cpp

private:
 
     bool HasLoopPHIUse(const MachineInstr *MI) const;
 
     bool HasHighOperandLatency(MachineInstr &MI, unsigned DefIdx,
                                unsigned Reg) const;
 
     bool IsCheapInstruction(MachineInstr &MI) const;
 
-    bool CanCauseHighRegPressure(const DenseMap<unsigned, int> &Cost,
+    bool CanCauseHighRegPressure(const MachineInstr &MI,
+                                 const DenseMap<unsigned, int> &Cost,
                                  bool Cheap);
 
     void UpdateBackTraceRegPressure(const MachineInstr *MI);
 
     bool IsProfitableToHoist(MachineInstr &MI);
 
     bool IsGuaranteedToExecute(MachineBasicBlock *BB);
 
for (unsigned i = 0, e = MI.getNumOperands(); NumDefs && i != e; ++i) {
     isCheap = true;
   }
 
   return isCheap;
 }
 
 /// Visit BBs from header to current BB, check if hoisting an instruction of the
 /// given cost matrix can cause high register pressure.
-bool
-MachineLICMBase::CanCauseHighRegPressure(const DenseMap<unsigned, int>& Cost,
-                                         bool CheapInstr) {
+bool MachineLICMBase::CanCauseHighRegPressure(
+    const MachineInstr &MI, const DenseMap<unsigned, int> &Cost,
   for (const auto &RPIdAndCost : Cost) {
     if (RPIdAndCost.second <= 0)
       continue;
 
     unsigned Class = RPIdAndCost.first;
     int Limit = RegLimit[Class];
 
     // Don't hoist cheap instructions if they would increase register pressure,
     // even if we're under the limit.
-    if (CheapInstr && !HoistCheapInsts)
+    if (CheapInstr && !(HoistCheapInsts.getNumOccurrences()
+                            ? HoistCheapInsts
+                            : TII->shouldHoistCheapInsts(MI, Cost)))
       return true;
 
     for (const auto &RP : BackTrace)
       if (static_cast<int>(RP[Class]) + RPIdAndCost.second >= Limit)
         return true;
   }
 
   return false;
bool MachineLICMBase::IsProfitableToHoist(MachineInstr &MI) {
   // moderately high pressure situation.
   // Cheap instructions will only be hoisted if they don't increase register
   // pressure at all.
   auto Cost = calcRegisterCost(&MI, /*ConsiderSeen=*/false,
                                /*ConsiderUnseenAsDef=*/false);
 
   // Visit BBs from header to current BB, if hoisting this doesn't cause
   // high register pressure, then it's safe to proceed.
-  if (!CanCauseHighRegPressure(Cost, CheapInstr)) {
+  if (!CanCauseHighRegPressure(MI, Cost, CheapInstr)) {
     LLVM_DEBUG(dbgs() << "Hoist non-reg-pressure: " << MI);
     ++NumLowRP;
     return true;
   }
 
   // Don't risk increasing register pressure if it would create copies.
   if (CreatesCopy) {
     LLVM_DEBUG(dbgs() << "Won't hoist instr with loop PHI use: " << MI);

llvm/lib/Target/ARM/ARMBaseInstrInfo.h

void loadRegFromStackSlot(MachineBasicBlock &MBB,
                             Register DestReg, int FrameIndex,
                             const TargetRegisterClass *RC,
                             const TargetRegisterInfo *TRI) const override;
 
   bool expandPostRAPseudo(MachineInstr &MI) const override;
 
   bool shouldSink(const MachineInstr &MI) const override;
 
+  bool
+  shouldHoistCheapInsts(const MachineInstr &MI,
+                        const DenseMap<unsigned, int> &Cost) const override;
+
   void reMaterialize(MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
                      unsigned DestReg, unsigned SubIdx,
                      const MachineInstr &Orig,
                      const TargetRegisterInfo &TRI) const override;
 
   MachineInstr &
   duplicate(MachineBasicBlock &MBB, MachineBasicBlock::iterator InsertBefore,
             const MachineInstr &Orig) const override;

llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp

bool ARMBaseInstrInfo::shouldSink(const MachineInstr &MI) const {
   bool IsThumb1;
   if (Next != MI.getParent()->end() &&
       analyzeCompare(*Next, SrcReg, SrcReg2, CmpMask, CmpValue) &&
       isRedundantFlagInstr(&*Next, SrcReg, SrcReg2, CmpValue, &MI, IsThumb1))
     return false;
   return true;
 }
 
+bool ARMBaseInstrInfo::shouldHoistCheapInsts(
+    const MachineInstr &MI, const DenseMap<unsigned, int> &Cost) const {
+  // Especially in tight MVE loops, most instructions are not really "cheap"
+  // (even movs/copies), and should all be pulled out of loops if available.
+  // The vector instructions that accept gpr ergisters from float values can
+  // also leave copies in the loops. Thumb2 cores have enough registers to make
+  // this generally useful (MachineLICM will already handle the case where we
+  // are actually at the register limit).
+  return Subtarget.isMClass() && Subtarget.isThumb2();
+}
+
 bool ARMBaseInstrInfo::FoldImmediate(MachineInstr &UseMI, MachineInstr &DefMI,
                                      unsigned Reg,
                                      MachineRegisterInfo *MRI) const {
   // Fold large immediates into add, sub, or, xor.
   unsigned DefOpc = DefMI.getOpcode();
   if (DefOpc != ARM::t2MOVi32imm && DefOpc != ARM::MOVi32imm)
     return false;
   if (!DefMI.getOperand(1).isImm())

llvm/test/CodeGen/Thumb2/mve-floatregloops.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve.fp -verify-machineinstrs %s -o - | FileCheck %s
 
 define arm_aapcs_vfpcc void @test_fadd(float* noalias nocapture readonly %A, float %B, float* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fadd:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:  .LBB0_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vmov r3, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.f32 q1, q1, r3
-; CHECK-NEXT:    vstrb.8 q1, [r1], #16
+; CHECK-NEXT:    vadd.f32 q0, q0, r3
+; CHECK-NEXT:    vstrb.8 q0, [r1], #16
 ; CHECK-NEXT:    bne .LBB0_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp18 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fadd_r(float* noalias nocapture readonly %A, float %B, float* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fadd_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:  .LBB1_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vmov r3, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vadd.f32 q1, q1, r3
-; CHECK-NEXT:    vstrb.8 q1, [r1], #16
+; CHECK-NEXT:    vadd.f32 q0, q0, r3
+; CHECK-NEXT:    vstrb.8 q0, [r1], #16
 ; CHECK-NEXT:    bne .LBB1_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp18 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fmul(float* noalias nocapture readonly %A, float %B, float* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fmul:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:  .LBB2_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vmov r3, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vmul.f32 q1, q1, r3
-; CHECK-NEXT:    vstrb.8 q1, [r1], #16
+; CHECK-NEXT:    vmul.f32 q0, q0, r3
+; CHECK-NEXT:    vstrb.8 q0, [r1], #16
 ; CHECK-NEXT:    bne .LBB2_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp18 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fmul_r(float* noalias nocapture readonly %A, float %B, float* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fmul_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vmov r3, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vmul.f32 q1, q1, r3
-; CHECK-NEXT:    vstrb.8 q1, [r1], #16
+; CHECK-NEXT:    vmul.f32 q0, q0, r3
+; CHECK-NEXT:    vstrb.8 q0, [r1], #16
 ; CHECK-NEXT:    bne .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp18 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fsub(float* noalias nocapture readonly %A, float %B, float* noalias nocapture %C, i32 %n) {
 ; CHECK-LABEL: test_fsub:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r2, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r3, s0
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vmov r3, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    subs r2, #4
-; CHECK-NEXT:    vsub.f32 q1, q1, r3
-; CHECK-NEXT:    vstrb.8 q1, [r1], #16
+; CHECK-NEXT:    vsub.f32 q0, q0, r3
+; CHECK-NEXT:    vstrb.8 q0, [r1], #16
 ; CHECK-NEXT:    bne .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp18 = icmp sgt i32 %n, 0

 
 
 define arm_aapcs_vfpcc void @test_fmas(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmas:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
 ; CHECK-NEXT:  .LBB6_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
-; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
+; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfmas.f32 q2, q1, r12
-; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    vfmas.f32 q1, q0, r12
+; CHECK-NEXT:    vstrb.8 q1, [r2], #16
 ; CHECK-NEXT:    bne .LBB6_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fmas_r(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmas_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
 ; CHECK-NEXT:  .LBB7_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
-; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
+; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfmas.f32 q2, q1, r12
-; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    vfmas.f32 q1, q0, r12
+; CHECK-NEXT:    vstrb.8 q1, [r2], #16
 ; CHECK-NEXT:    bne .LBB7_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fma(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fma:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
-; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
+; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfma.f32 q2, q1, r12
-; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    vfma.f32 q1, q0, r12
+; CHECK-NEXT:    vstrb.8 q1, [r2], #16
 ; CHECK-NEXT:    bne .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fma_r(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fma_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vdup.32 q0, r12
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov r12, s0
-; CHECK-NEXT:    vldrw.u32 q2, [r0], #16
-; CHECK-NEXT:    vldrw.u32 q3, [r1], #16
-; CHECK-NEXT:    vdup.32 q1, r12
+; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
+; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfma.f32 q3, q1, q2
-; CHECK-NEXT:    vstrb.8 q3, [r2], #16
+; CHECK-NEXT:    vfma.f32 q2, q0, q1
+; CHECK-NEXT:    vstrb.8 q2, [r2], #16
 ; CHECK-NEXT:    bne .LBB9_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 
 
 define arm_aapcs_vfpcc void @test_fmss(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fmss:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vdup.32 q0, r12
+; CHECK-NEXT:    vneg.f32 q0, q0
 ; CHECK-NEXT:  .LBB10_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov r12, s0
-; CHECK-NEXT:    vldrw.u32 q2, [r0], #16
-; CHECK-NEXT:    vdup.32 q1, r12
-; CHECK-NEXT:    vldrw.u32 q3, [r1], #16
-; CHECK-NEXT:    vneg.f32 q1, q1
+; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
+; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
+; CHECK-NEXT:    vmov q3, q0
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfma.f32 q1, q3, q2
-; CHECK-NEXT:    vstrb.8 q1, [r2], #16
+; CHECK-NEXT:    vfma.f32 q3, q2, q1
+; CHECK-NEXT:    vstrb.8 q3, [r2], #16
 ; CHECK-NEXT:    bne .LBB10_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fms(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fms:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
 ; CHECK-NEXT:  .LBB12_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
-; CHECK-NEXT:    vldrw.u32 q2, [r0], #16
-; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
+; CHECK-NEXT:    vldrw.u32 q1, [r0], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vneg.f32 q1, q1
-; CHECK-NEXT:    vfma.f32 q1, q2, r12
-; CHECK-NEXT:    vstrb.8 q1, [r2], #16
+; CHECK-NEXT:    vneg.f32 q0, q0
+; CHECK-NEXT:    vfma.f32 q0, q1, r12
+; CHECK-NEXT:    vstrb.8 q0, [r2], #16
 ; CHECK-NEXT:    bne .LBB12_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 }
 
 define arm_aapcs_vfpcc void @test_fms_r(float* noalias nocapture readonly %A, float* noalias nocapture readonly %B, float %C, float* noalias nocapture %D, i32 %n) {
 ; CHECK-LABEL: test_fms_r:
 ; CHECK:       @ %bb.0: @ %entry
 ; CHECK-NEXT:    cmp r3, #1
 ; CHECK-NEXT:    it lt
 ; CHECK-NEXT:    bxlt lr
+; CHECK-NEXT:    vmov r12, s0
+; CHECK-NEXT:    vdup.32 q0, r12
 ; CHECK-NEXT:  .LBB13_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r1], #16
-; CHECK-NEXT:    vmov r12, s0
-; CHECK-NEXT:    vldrw.u32 q3, [r0], #16
-; CHECK-NEXT:    vdup.32 q1, r12
-; CHECK-NEXT:    vneg.f32 q2, q2
+; CHECK-NEXT:    vldrw.u32 q1, [r1], #16
+; CHECK-NEXT:    vldrw.u32 q2, [r0], #16
 ; CHECK-NEXT:    subs r3, #4
-; CHECK-NEXT:    vfma.f32 q2, q1, q3
-; CHECK-NEXT:    vstrb.8 q2, [r2], #16
+; CHECK-NEXT:    vneg.f32 q1, q1
+; CHECK-NEXT:    vfma.f32 q1, q0, q2
+; CHECK-NEXT:    vstrb.8 q1, [r2], #16
 ; CHECK-NEXT:    bne .LBB13_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    bx lr
 entry:
   %0 = and i32 %n, 7
   %cmp = icmp eq i32 %0, 0
   tail call void @llvm.assume(i1 %cmp)
   %cmp110 = icmp sgt i32 %n, 0

 ; CHECK-NEXT:    push.w {r4, r5, r6, r7, r8, r9, r10, r11, lr}
 ; CHECK-NEXT:    .pad #4
 ; CHECK-NEXT:    sub sp, #4
 ; CHECK-NEXT:    .vsave {d8, d9}
 ; CHECK-NEXT:    vpush {d8, d9}
 ; CHECK-NEXT:    .pad #16
 ; CHECK-NEXT:    sub sp, #16
 ; CHECK-NEXT:    ldrh r4, [r0]
-; CHECK-NEXT:    ldr.w r12, [r0, #4]
+; CHECK-NEXT:    ldr r5, [r0, #4]
 ; CHECK-NEXT:    subs r7, r4, #1
 ; CHECK-NEXT:    cmp r7, #3
 ; CHECK-NEXT:    bhi .LBB15_6
 ; CHECK-NEXT:  @ %bb.1: @ %if.then
 ; CHECK-NEXT:    ldr r6, [r0, #8]
-; CHECK-NEXT:    add.w r11, r12, r7, lsl #2
+; CHECK-NEXT:    add.w r11, r5, r7, lsl #2
 ; CHECK-NEXT:    lsr.w lr, r3, #2
 ; CHECK-NEXT:    vldr s0, [r6]
 ; CHECK-NEXT:    vldr s2, [r6, #4]
 ; CHECK-NEXT:    vldr s4, [r6, #8]
 ; CHECK-NEXT:    vldr s6, [r6, #12]
 ; CHECK-NEXT:    wls lr, lr, .LBB15_5
 ; CHECK-NEXT:  @ %bb.2: @ %while.body.lr.ph
-; CHECK-NEXT:    str r4, [sp, #12] @ 4-byte Spill
-; CHECK-NEXT:    bic r4, r3, #3
+; CHECK-NEXT:    strd r3, r4, [sp, #8] @ 8-byte Folded Spill
+; CHECK-NEXT:    vmov r10, s4
+; CHECK-NEXT:    vmov r12, s6
+; CHECK-NEXT:    bic r3, r3, #3
+; CHECK-NEXT:    vmov r4, s2
+; CHECK-NEXT:    str r3, [sp] @ 4-byte Spill
+; CHECK-NEXT:    vmov r8, s0
+; CHECK-NEXT:    add.w r3, r2, r3, lsl #2
+; CHECK-NEXT:    str r3, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    movs r6, #0
-; CHECK-NEXT:    str r4, [sp, #4] @ 4-byte Spill
-; CHECK-NEXT:    add.w r4, r2, r4, lsl #2
-; CHECK-NEXT:    str r4, [sp, #8] @ 4-byte Spill
+; CHECK-NEXT:    mov r3, r5
 ; CHECK-NEXT:  .LBB15_3: @ %while.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    adds r7, r1, r6
-; CHECK-NEXT:    add.w r5, r12, r6
-; CHECK-NEXT:    vldrw.u32 q2, [r7]
+; CHECK-NEXT:    add.w r9, r1, r6
 ; CHECK-NEXT:    add.w r7, r11, r6
-; CHECK-NEXT:    vmov r10, s0
+; CHECK-NEXT:    vldrw.u32 q2, [r9]
+; CHECK-NEXT:    adds r5, r3, r6
 ; CHECK-NEXT:    vstrw.32 q2, [r7]
-; CHECK-NEXT:    vmov r9, s2
-; CHECK-NEXT:    vldrw.u32 q2, [r5]
-; CHECK-NEXT:    vmov r4, s4
 ; CHECK-NEXT:    adds r7, r2, r6
+; CHECK-NEXT:    vldrw.u32 q2, [r5]
 ; CHECK-NEXT:    adds r6, #16
-; CHECK-NEXT:    vmul.f32 q2, q2, r10
+; CHECK-NEXT:    vmul.f32 q2, q2, r8
 ; CHECK-NEXT:    vldrw.u32 q3, [r5, #4]
-; CHECK-NEXT:    vmov r8, s6
-; CHECK-NEXT:    vfma.f32 q2, q3, r9
+; CHECK-NEXT:    vfma.f32 q2, q3, r4
 ; CHECK-NEXT:    vldrw.u32 q3, [r5, #8]
 ; CHECK-NEXT:    vldrw.u32 q4, [r5, #12]
-; CHECK-NEXT:    vfma.f32 q2, q3, r4
-; CHECK-NEXT:    vfma.f32 q2, q4, r8
+; CHECK-NEXT:    vfma.f32 q2, q3, r10
+; CHECK-NEXT:    vfma.f32 q2, q4, r12
 ; CHECK-NEXT:    vstrw.32 q2, [r7]
 ; CHECK-NEXT:    le lr, .LBB15_3
 ; CHECK-NEXT:  @ %bb.4: @ %while.end.loopexit
-; CHECK-NEXT:    ldr r2, [sp, #4] @ 4-byte Reload
+; CHECK-NEXT:    ldr r2, [sp] @ 4-byte Reload
+; CHECK-NEXT:    add r4, sp, #4
 ; CHECK-NEXT:    add r11, r6
-; CHECK-NEXT:    add.w r12, r12, r2, lsl #2
+; CHECK-NEXT:    add.w r5, r3, r2, lsl #2
 ; CHECK-NEXT:    add.w r1, r1, r2, lsl #2
-; CHECK-NEXT:    ldrd r2, r4, [sp, #8] @ 8-byte Folded Reload
+; CHECK-NEXT:    ldm r4, {r2, r3, r4} @ 12-byte Folded Reload
 ; CHECK-NEXT:  .LBB15_5: @ %while.end
 ; CHECK-NEXT:    and r7, r3, #3
 ; CHECK-NEXT:    vldrw.u32 q2, [r1]
 ; CHECK-NEXT:    vctp.32 r7
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q2, [r11]
 ; CHECK-NEXT:    vmov r6, s2
-; CHECK-NEXT:    vmov r5, s0
-; CHECK-NEXT:    vldrw.u32 q0, [r12]
-; CHECK-NEXT:    vmov r1, s6
+; CHECK-NEXT:    vmov r1, s0
+; CHECK-NEXT:    vldrw.u32 q0, [r5]
+; CHECK-NEXT:    vmov r12, s6
 ; CHECK-NEXT:    vmov r7, s4
-; CHECK-NEXT:    vmul.f32 q0, q0, r5
-; CHECK-NEXT:    vldrw.u32 q1, [r12, #4]
+; CHECK-NEXT:    vmul.f32 q0, q0, r1
+; CHECK-NEXT:    vldrw.u32 q1, [r5, #4]
 ; CHECK-NEXT:    vfma.f32 q0, q1, r6
-; CHECK-NEXT:    vldrw.u32 q1, [r12, #8]
+; CHECK-NEXT:    vldrw.u32 q1, [r5, #8]
 ; CHECK-NEXT:    vfma.f32 q0, q1, r7
-; CHECK-NEXT:    vldrw.u32 q1, [r12, #12]
-; CHECK-NEXT:    vfma.f32 q0, q1, r1
+; CHECK-NEXT:    vldrw.u32 q1, [r5, #12]
+; CHECK-NEXT:    vfma.f32 q0, q1, r12
 ; CHECK-NEXT:    vpst
 ; CHECK-NEXT:    vstrwt.32 q0, [r2]
-; CHECK-NEXT:    ldr.w r12, [r0, #4]
+; CHECK-NEXT:    ldr r5, [r0, #4]
 ; CHECK-NEXT:  .LBB15_6: @ %if.end
-; CHECK-NEXT:    add.w r0, r12, r3, lsl #2
+; CHECK-NEXT:    add.w r0, r5, r3, lsl #2
+; CHECK-NEXT:    mov r2, r5
 ; CHECK-NEXT:    lsr.w lr, r4, #2
 ; CHECK-NEXT:    wls lr, lr, .LBB15_10
 ; CHECK-NEXT:  @ %bb.7: @ %while.body51.preheader
-; CHECK-NEXT:    bic r2, r4, #3
-; CHECK-NEXT:    adds r1, r2, r3
-; CHECK-NEXT:    mov r3, r12
-; CHECK-NEXT:    add.w r1, r12, r1, lsl #2
+; CHECK-NEXT:    bic r7, r4, #3
+; CHECK-NEXT:    adds r1, r7, r3
+; CHECK-NEXT:    mov r3, r2
+; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
 ; CHECK-NEXT:  .LBB15_8: @ %while.body51
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #16
 ; CHECK-NEXT:    vstrb.8 q0, [r3], #16
 ; CHECK-NEXT:    le lr, .LBB15_8
 ; CHECK-NEXT:  @ %bb.9: @ %while.end55.loopexit
-; CHECK-NEXT:    add.w r12, r12, r2, lsl #2
+; CHECK-NEXT:    add.w r2, r2, r7, lsl #2
 ; CHECK-NEXT:    mov r0, r1
 ; CHECK-NEXT:  .LBB15_10: @ %while.end55
 ; CHECK-NEXT:    ands r1, r4, #3
 ; CHECK-NEXT:    beq .LBB15_12
 ; CHECK-NEXT:  @ %bb.11: @ %if.then59
 ; CHECK-NEXT:    vldrw.u32 q0, [r0]
 ; CHECK-NEXT:    vctp.32 r1
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q0, [r12]
+; CHECK-NEXT:    vstrwt.32 q0, [r2]
 ; CHECK-NEXT:  .LBB15_12: @ %if.end61
 ; CHECK-NEXT:    add sp, #16
 ; CHECK-NEXT:    vpop {d8, d9}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 entry:
   %pState1 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 1
   %0 = load float*, float** %pState1, align 4

llvm/test/CodeGen/Thumb2/mve-pred-threshold.ll

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
 ; CHECK-NEXT:    vneg.f32 s2, s0
 ; CHECK-NEXT:    mvn r2, #3
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
 ; CHECK-NEXT:    movs r2, #1
-; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #2
+; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    dls lr, lr
+; CHECK-NEXT:    vmov r2, s2
+; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:  .LBB3_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0]
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vcmp.f32 ge, q2, r1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.f32 le, q2, r1
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vpt.f32 ge, q1, r1
+; CHECK-NEXT:    vcmpt.f32 le, q1, r2
 ; CHECK-NEXT:    vpnot
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
+; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB3_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
 ; CHECK-NEXT:    mvn r2, #7
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #3
 ; CHECK-NEXT:    movs r2, #1
 ; CHECK-NEXT:    vneg.f16 s2, s0
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #3
-; CHECK-NEXT:    vmov.i32 q1, #0x0
+; CHECK-NEXT:    vmov r1, s2
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB4_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vldrh.u16 q2, [r0]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vpt.f16 ge, q2, r2
-; CHECK-NEXT:    vcmpt.f16 le, q2, r1
+; CHECK-NEXT:    vldrh.u16 q1, [r0]
+; CHECK-NEXT:    vpt.f16 ge, q1, r2
+; CHECK-NEXT:    vcmpt.f16 le, q1, r1
 ; CHECK-NEXT:    vpnot
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q1, [r0], #16
+; CHECK-NEXT:    vstrht.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB4_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32

 ; CHECK-NEXT:    push {r7, lr}
 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
 ; CHECK-NEXT:    vneg.f32 s2, s0
 ; CHECK-NEXT:    mvn r2, #3
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #2
 ; CHECK-NEXT:    movs r2, #1
-; CHECK-NEXT:    vmov.i32 q1, #0x0
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #2
+; CHECK-NEXT:    vmov r1, s0
 ; CHECK-NEXT:    dls lr, lr
+; CHECK-NEXT:    vmov r2, s2
+; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:  .LBB8_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vldrw.u32 q2, [r0]
-; CHECK-NEXT:    vmov r1, s0
-; CHECK-NEXT:    vcmp.f32 ge, q2, r1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vcmpt.f32 le, q2, r1
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vpt.f32 ge, q1, r1
+; CHECK-NEXT:    vcmpt.f32 le, q1, r2
 ; CHECK-NEXT:    vpnot
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrwt.32 q1, [r0], #16
+; CHECK-NEXT:    vstrwt.32 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB8_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %conv = zext i16 %N to i32
   %mul = shl nuw nsw i32 %conv, 2
   %cmp15 = icmp eq i16 %N, 0
   br i1 %cmp15, label %for.cond.cleanup, label %vector.ph

 ; CHECK-NEXT:    cmp r1, #0
 ; CHECK-NEXT:    it eq
 ; CHECK-NEXT:    popeq {r7, pc}
 ; CHECK-NEXT:    mvn r2, #7
 ; CHECK-NEXT:    add.w r1, r2, r1, lsl #3
 ; CHECK-NEXT:    movs r2, #1
 ; CHECK-NEXT:    vneg.f16 s2, s0
 ; CHECK-NEXT:    add.w lr, r2, r1, lsr #3
-; CHECK-NEXT:    vmov.i32 q1, #0x0
+; CHECK-NEXT:    vmov r1, s2
+; CHECK-NEXT:    vmov r2, s0
+; CHECK-NEXT:    vmov.i32 q0, #0x0
 ; CHECK-NEXT:    dls lr, lr
 ; CHECK-NEXT:  .LBB9_1: @ %vector.body
 ; CHECK-NEXT:    @ =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    vmov r1, s2
-; CHECK-NEXT:    vldrh.u16 q2, [r0]
-; CHECK-NEXT:    vmov r2, s0
-; CHECK-NEXT:    vpt.f16 ge, q2, r2
-; CHECK-NEXT:    vcmpt.f16 le, q2, r1
+; CHECK-NEXT:    vldrh.u16 q1, [r0]
+; CHECK-NEXT:    vpt.f16 ge, q1, r2
+; CHECK-NEXT:    vcmpt.f16 le, q1, r1
 ; CHECK-NEXT:    vpnot
 ; CHECK-NEXT:    vpst
-; CHECK-NEXT:    vstrht.16 q1, [r0], #16
+; CHECK-NEXT:    vstrht.16 q0, [r0], #16
 ; CHECK-NEXT:    le lr, .LBB9_1
 ; CHECK-NEXT:  @ %bb.2: @ %for.cond.cleanup
 ; CHECK-NEXT:    pop {r7, pc}
 entry:
   %0 = bitcast float %T.coerce to i32
   %tmp.0.extract.trunc = trunc i32 %0 to i16
   %1 = bitcast i16 %tmp.0.extract.trunc to half
   %conv = zext i16 %N to i32