[ARM][LowOverheadLoops] Merge VCMP and VPST across VPT blocks

[samtebbs]

Summary

This patch adds support for combining a VPST with a dangling VCMP from a previous VPT block.

[dmgreen]

Is it possible to write some tests that are hopefully simple but still tail predicated, and contain different kinds of vpt blocks with various instructions in various orders? It probably doesn't matter if the instructions are super sensible so long as they show the different sets of blocks with vcmps followed by predicated/unpredicated instructions.

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1537–1538

I think we need to make sure this _is_ the last instruction in the block. Otherwise we can't remove it.

[samtebbs]

In D90935#2379000, @dmgreen wrote:

Is it possible to write some tests that are hopefully simple but still tail predicated, and contain different kinds of vpt blocks with various instructions in various orders? It probably doesn't matter if the instructions are super sensible so long as they show the different sets of blocks with vcmps followed by predicated/unpredicated instructions.

I can try!

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1537–1538

Shouldn't it be removable as long as instructions after it in the block don't use the VCMP's output?

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1537–1538

The instructions after the VCMP in the vpt block will use the VCMP's predicate, as opposed to the one at the start of the block.
So something like:

VPST
VCMP
VORR
..
VPST

The VORR should be using the vpr from the VCMP (and so it can't be removed), and there's probably not a lot of point combining it into the VPST, as the predicate is already set.

I couldn't spot anything that check for that already.

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1537–1538

Sure for that example the VCMP shouldn't be changed. But if the instructions after the VCMP don't use the vpr, then it can be removed right? That, rather than it being the last instruction, is the fundamental requirement I think you're getting at. This would be fine, for example:

VPST
VCMP
ADD (scalar)
SUB (scalar)
VPST

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1537–1538

Yep, that sounds fine. I meant last instruction in the VPT block, not last instruction in the basic block.

I guess put better - there cannot be any uses of vpr between the VCMP that i being removed and the VPST that is being converted into the VPT.

[samtebbs]

Add more tests and check for instructions between the VCMP and VPST that use vpr.

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1487–1507

It may be better if this takes VCMP as an argument - that would help keep the logic around here clearer where things are being used.

1537–1538

I feel like this should be using : nullptr, even if just to keep it simpler. From what I can tell the older VCMP wouldn't ever be useful after the point that we have seen something that has altered the vpr predicate.

1573

Checking that an operand is vpr may be better.

1591–1592

I _think_ this case would be invalid input, and not something we would have to worry about (it could be an assert - ideally there would be code in the verifier that checked that VPT blocks were always valid, but no one has written that yet).

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

7

If you remove the ".entry" from "bb.0.entry" below, you can probably remove these skeletons too.

171–183

A lot of this can often be removed.

348–349

I don't think this is valid input, with the VPST not creating a block with any instruction and the MVE_VORR not being in a the previous block. Same for some of the tests below.

Maybe make sure the last VPST block has some instruction in it, and the middle VPST has the correct block mask

[samtebbs]

Check for VPR use rather than predicate, pass VCMP as argument to ReplaceVCMPWithVPT, make nullptr the default VCMP value and remove impossible VPST vase.

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1487–1507

Sure!

1537–1538

Ah yes, I missed you specifying it being the last in the *vpt* block.

1537–1538

That's a good point!

1573

👍

1591–1592

I added this when I was creating test cases but it makes sense that this wouldn't happen naturally.

[samtebbs]

Clean up tests

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1567

I think this should always be true. Maybe turn it into an assert?

1569

The if VCMP can probably be hoisted to the Insts.front()->getOpcode() == ARM::MVE_VPST if.

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

284

MVE_VPST 8 I think, for this block

370

Should be MVE_VPST 4 I think

372

0 -> 1 for "ARMVCC" operand

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1567

👍

1569

👍

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

7

That's much cleaner

171–183

Very nice

284

👍

348–349

That's true! I've fixed the blocks and renamed the functions to make it more clear what they are testing.

370

👍

372

I made it 0 intentionally to simulate a non-predicated vpr use.

[samtebbs]

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

370

Actually the second instruction isn't predicated so this should be 8.

[samtebbs]

Get VPR def when merging across blocks rather than keeping a pointer to the VCMP.

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1592

Can you clang-format this?

1593

Is it possible to have:

VPST
VORR vpr
VCMP vpr
..
VPST
VORR vpr
..
VPST
VORR vpr

And would this attempt to fold the VCMP into the last vptblock? Do we have a test for that case?

1594

Do we need to do the same thing to remove kill flags as D90964?

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

188–191

Some of these still look a little funny, like they would not really be valid inputs. Are they to test the various edge cases?

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1592

Ah yes, sorry

1594

Hmm I'm not sure. From what I've seen of the pass' output. the kill flags are still valid when the instruction is removed.

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

188–191

The VPST mask does need to be changed. What else looks funny?

[samparker]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1584

This could be cleaned up by using std::any_of to iterate through the instructions and then also using the hasVPRUse helper too.

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1593

This becomes

VORR
VPT
VORR vpr
VPST
VORR vpr

And I have added a test case for this.

[samtebbs]

Add another test case and format

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1584

Oh nice, I didn't know these existed.

[samtebbs]

Use std::any_of and hasVPRUse.

[samparker]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1523–1524

Can this TODO be removed now?

1564

Don't worry about putting it up for review, but before committing would you mind adding a comment about what condition(s) this logic is handling, like there is for the other conditional blocks?

[samparker]

I'm happy if Dave has no other suggestions.

[dmgreen]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1566

If Next is only use in an assert, be careful that it does not cause a warning in release mode.

1585

Same if this is only used in an assert.

I'm still not 100% sure that this will always hold true with VPNOT or VPSEL (or anything else in the future as things change. They are not supported in TP loops yet but theoretically could be and things like this could easily with how subtle they are).

It would probably be better as an if, to be honest. It's only a minor loss of perf if it does happen, and that way it's safer.

1594

It would be something like:

VPST 4
VLDR $vpr
VCMP q0, q1, $vpr
VORR q2, killed q1, $noreg
VPST 8
VORR ... $vpr

The unpredicated VORR kills the register that the VCMP will still use.

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

198

This can be MVE_VPST 8 for a single instruction block.

[samtebbs]

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

1594

Oh I see. I thought that since the VCMP would use q1 on the next iteration it would still count as being alive.

llvm/lib/Target/ARM/ARMLowOverheadLoops.cpp

static bool isVectorPredicated(MachineInstr *MI) {
   int PIdx = llvm::findFirstVPTPredOperandIdx(*MI);
   return PIdx != -1 && MI->getOperand(PIdx + 1).getReg() == ARM::VPR;
 }
 
 static bool isVectorPredicate(MachineInstr *MI) {
   return MI->findRegisterDefOperandIdx(ARM::VPR) != -1;
 }
 
-static bool hasVPRUse(MachineInstr *MI) {
-  return MI->findRegisterUseOperandIdx(ARM::VPR) != -1;
+static bool hasVPRUse(MachineInstr &MI) {
+  return MI.findRegisterUseOperandIdx(ARM::VPR) != -1;
 }
 
 static bool isDomainMVE(MachineInstr *MI) {
   uint64_t Domain = MI->getDesc().TSFlags & ARMII::DomainMask;
   return Domain == ARMII::DomainMVE;
 }
 
 static bool shouldInspect(MachineInstr &MI) {
-  return isDomainMVE(&MI) || isVectorPredicate(&MI) ||
-    hasVPRUse(&MI);
+  return isDomainMVE(&MI) || isVectorPredicate(&MI) || hasVPRUse(MI);
 }
 
 static bool isDo(MachineInstr *MI) {
   return MI->getOpcode() != ARM::t2WhileLoopStart;
 }
 
 namespace {
 
if (int PIdx = llvm::findFirstVPTPredOperandIdx(*MI)) {
       MI->getOperand(PIdx+1).setReg(0);
     } else
       llvm_unreachable("trying to unpredicate a non-predicated instruction");
   };
 
   for (auto &Block : LoLoop.getVPTBlocks()) {
     SmallVectorImpl<MachineInstr *> &Insts = Block.getInsts();
 
-    if (VPTState::isEntryPredicatedOnVCTP(Block, /*exclusive*/true)) {
+    auto ReplaceVCMPWithVPT = [&](MachineInstr *&TheVCMP, MachineInstr *At) {
+      assert(TheVCMP && "Replacing a removed or non-existent VCMP");
+      // Replace the VCMP with a VPT
+      MachineInstrBuilder MIB =
+          BuildMI(*At->getParent(), At, At->getDebugLoc(),
+                  TII->get(VCMPOpcodeToVPT(TheVCMP->getOpcode())));
+      MIB.addImm(ARMVCC::Then);
+      // Register one
+      MIB.add(TheVCMP->getOperand(1));
+      // Register two
+      MIB.add(TheVCMP->getOperand(2));
+      // The comparison code, e.g. ge, eq, lt
+      MIB.add(TheVCMP->getOperand(3));
+      LLVM_DEBUG(dbgs() << "ARM Loops: Combining with VCMP to VPT: " << *MIB);
+      LoLoop.BlockMasksToRecompute.insert(MIB.getInstr());
+      LoLoop.ToRemove.insert(TheVCMP);
+      TheVCMP = nullptr;
+    };
+
+    if (VPTState::isEntryPredicatedOnVCTP(Block, /*exclusive*/ true)) {
+      MachineInstr *VPST = Insts.front();
       if (VPTState::hasUniformPredicate(Block)) {
         // A vpt block starting with VPST, is only predicated upon vctp and has no
         // internal vpr defs:
         // - Remove vpst.
         // - Unpredicate the remaining instructions.
-        LLVM_DEBUG(dbgs() << "ARM Loops: Removing VPST: " << *Insts.front());
-        LoLoop.ToRemove.insert(Insts.front());
+        LLVM_DEBUG(dbgs() << "ARM Loops: Removing VPST: " << *VPST);
         for (unsigned i = 1; i < Insts.size(); ++i)
           RemovePredicate(Insts[i]);
       } else {
         // The VPT block has a non-uniform predicate but it uses a vpst and its
         // entry is guarded only by a vctp, which means we:
         // - Need to remove the original vpst.
         // - Then need to unpredicate any following instructions, until
         //   we come across the divergent vpr def.
         // - Insert a new vpst to predicate the instruction(s) that following
         //   the divergent vpr def.
-        // TODO: We could be producing more VPT blocks than necessary and could
-        // fold the newly created one into a proceeding one.
         MachineInstr *Divergent = VPTState::getDivergent(Block);
-        MachineInstr *VPST = Insts.front();
         auto DivergentNext = ++MachineBasicBlock::iterator(Divergent);
         bool DivergentNextIsPredicated =
             getVPTInstrPredicate(*DivergentNext) != ARMVCC::None;
 
         for (auto I = ++MachineBasicBlock::iterator(VPST), E = DivergentNext;
              I != E; ++I)
           RemovePredicate(&*I);
 
         // Check if the instruction defining vpr is a vcmp so it can be combined
         // with the VPST This should be the divergent instruction
         MachineInstr *VCMP =
             VCMPOpcodeToVPT(Divergent->getOpcode()) != 0 ? Divergent : nullptr;
 
-        auto ReplaceVCMPWithVPT = [&]() {
-          // Replace the VCMP with a VPT
-          MachineInstrBuilder MIB = BuildMI(
-              *Divergent->getParent(), Divergent, Divergent->getDebugLoc(),
-              TII->get(VCMPOpcodeToVPT(VCMP->getOpcode())));
-          MIB.addImm(ARMVCC::Then);
-          // Register one
-          MIB.add(VCMP->getOperand(1));
-          // Register two
-          MIB.add(VCMP->getOperand(2));
-          // The comparison code, e.g. ge, eq, lt
-          MIB.add(VCMP->getOperand(3));
-          LLVM_DEBUG(dbgs()
-                     << "ARM Loops: Combining with VCMP to VPT: " << *MIB);
-          LoLoop.BlockMasksToRecompute.insert(MIB.getInstr());
-          LoLoop.ToRemove.insert(VCMP);
-        };
-
         if (DivergentNextIsPredicated) {
           // Insert a VPST at the divergent only if the next instruction
           // would actually use it. A VCMP following a VPST can be
           // merged into a VPT so do that instead if the VCMP exists.
           if (!VCMP) {
             // Create a VPST (with a null mask for now, we'll recompute it
             // later)
             MachineInstrBuilder MIB =
                 BuildMI(*Divergent->getParent(), Divergent,
                         Divergent->getDebugLoc(), TII->get(ARM::MVE_VPST));
             MIB.addImm(0);
             LLVM_DEBUG(dbgs() << "ARM Loops: Created VPST: " << *MIB);
             LoLoop.BlockMasksToRecompute.insert(MIB.getInstr());
           } else {
             // No RDA checks are necessary here since the VPST would have been
-            // directly before the VCMP
-            ReplaceVCMPWithVPT();
-          }
-        }
-        LLVM_DEBUG(dbgs() << "ARM Loops: Removing VPST: " << *VPST);
-        LoLoop.ToRemove.insert(VPST);
-      }
+            // directly after the VCMP
+            ReplaceVCMPWithVPT(VCMP, VCMP);
+          }
     } else if (Block.containsVCTP()) {
       // The vctp will be removed, so the block mask of the vp(s)t will need
       // to be recomputed.
       LoLoop.BlockMasksToRecompute.insert(Insts.front());
+    } else if (Insts.front()->getOpcode() == ARM::MVE_VPST) {
+      // If this block starts with a VPST then attempt to merge it with the
+      // preceeding un-merged VCMP into a VPT. This VCMP comes from a VPT
+      // block that no longer exists
+      MachineInstr *VPST = Insts.front();
+      auto Next = ++MachineBasicBlock::iterator(VPST);
+      assert(getVPTInstrPredicate(*Next) != ARMVCC::None &&
+             "The instruction after a VPST must be predicated");
+
+      MachineInstr *VprDef = RDA->getUniqueReachingMIDef(VPST, ARM::VPR);
+      if (VprDef && VCMPOpcodeToVPT(VprDef->getOpcode()) &&
+          !LoLoop.ToRemove.contains(VprDef)) {
+        MachineInstr *VCMP = VprDef;
+        // The VCMP and VPST can only be merged if the VCMP's operands will have
+        // the same values at the VPST
+        if (RDA->hasSameReachingDef(VCMP, VPST, VCMP->getOperand(1).getReg()) &&
+            RDA->hasSameReachingDef(VCMP, VPST, VCMP->getOperand(2).getReg())) {
+          bool IntermediateInstrsUseVPR =
+              std::any_of(++MachineBasicBlock::iterator(VCMP),
+                          MachineBasicBlock::iterator(VPST), hasVPRUse);
+          // If the instruction after the VCMP is predicated then a different
+          // code path is expected to have merged the VCMP and VPST already.
+          // This assertion protects against changes to that behaviour
+          assert(!IntermediateInstrsUseVPR &&
+                 "Instructions between the VCMP and VPST are not expected to "
+                 "be predicated");
+          ReplaceVCMPWithVPT(VCMP, VPST);
+          LLVM_DEBUG(dbgs() << "ARM Loops: Removing VPST: " << *VPST);
+          LoLoop.ToRemove.insert(VPST);
+        }
+      }
     }
   }
 
   LoLoop.ToRemove.insert(LoLoop.VCTPs.begin(), LoLoop.VCTPs.end());
 }
 
 void ARMLowOverheadLoops::Expand(LowOverheadLoop &LoLoop) {
 

llvm/test/CodeGen/Thumb2/LowOverheadLoops/vcmp-vpst-combination-across-blocks.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mtriple=thumbv8.1m.main -mattr=+mve -run-pass=arm-low-overhead-loops %s -o - --verify-machineinstrs | FileCheck %s
+
+--- |
+  define void @combine_previous() {
+  while.end6:
+    ret void
+  }
+
+  define void @combine_middle() {
+  while.end6:
+    ret void
+  }
+
+  define void @combine_last() {
+  while.end6:
+    ret void
+  }
+
+  define void @no_combination_diff_reg_value() {
+  while.end6:
+    ret void
+  }
+
+  define void @no_combination_vcmp_already_merged() {
+  while.end6:
+    ret void
+  }
+
+...
+---
+name:            combine_previous
+alignment:       8
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:
+  - id:              0
+    value:           float 0xC7EFFFFFE0000000
+    alignment:       4
+    isTargetSpecific: false
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: combine_previous
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   liveins: $r0, $r1
+  ; CHECK:   renamable $q0 = MVE_VDUP32 renamable $r1, 0, $noreg, undef renamable $q0
+  ; CHECK:   $lr = MVE_DLSTP_32 killed renamable $r1
+  ; CHECK: bb.1 (align 4):
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.2(0x04000000)
+  ; CHECK:   liveins: $lr, $q0, $r0
+  ; CHECK:   renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 0, $noreg
+  ; CHECK:   renamable $q0 = MVE_VORR renamable $q1, renamable $q1, 0, $noreg, killed renamable $q0
+  ; CHECK:   MVE_VPTv4f32 8, renamable $q1, renamable $q0, 12, implicit-def $vpr
+  ; CHECK:   renamable $q0 = MVE_VORR killed renamable $q1, killed renamable $q1, 1, killed renamable $vpr, killed renamable $q0
+  ; CHECK:   $lr = MVE_LETP killed renamable $lr, %bb.1
+  ; CHECK: bb.2:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.3 (align 4):
+  ; CHECK:   CONSTPOOL_ENTRY 0, %const.0, 4
+  bb.0:
+    successors: %bb.6(0x80000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $r3, dead $cpsr = nuw nsw tADDi3 renamable $r1, 3, 14 /* CC::al */, $noreg
+    renamable $q0 = MVE_VDUP32 killed renamable $r1, 0, $noreg, undef renamable $q0
+    renamable $r3 = t2ANDri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $lr = t2SUBri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $r3, dead $cpsr = tMOVi8 1, 14 /* CC::al */, $noreg
+    renamable $lr = nuw nsw t2ADDrs killed renamable $r3, killed renamable $lr, 19, 14 /* CC::al */, $noreg, $noreg
+    $lr = t2DoLoopStart renamable $lr
+
+  bb.6 (align 4):
+    successors: %bb.6(0x7c000000), %bb.8(0x04000000)
+    liveins: $lr, $q0, $r0, $r1, $r2
+
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    renamable $vpr = MVE_VCTP32 renamable $r1, 0, $noreg
+    renamable $r1, dead $cpsr = tSUBi8 killed renamable $r1, 4, 14 /* CC::al */, $noreg
+    MVE_VPST 8, implicit $vpr
+    renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 1, renamable $vpr
+    MVE_VPST 4, implicit $vpr
+    renamable $q0 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, renamable $q0
+    renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 1, killed renamable $vpr
+    MVE_VPST 8, implicit $vpr
+    renamable $q0 = MVE_VORR killed renamable $q1, renamable $q1, 1, killed renamable $vpr, killed renamable $q0
+    t2LoopEnd renamable $lr, %bb.6, implicit-def dead $cpsr
+    tB %bb.8, 14 /* CC::al */, $noreg
+
+  bb.8:
+    liveins: $r2, $r12, $q0
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.9 (align 4):
+    CONSTPOOL_ENTRY 0, %const.0, 4
+
+...
+---
+name:            combine_middle
+alignment:       8
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:
+  - id:              0
+    value:           float 0xC7EFFFFFE0000000
+    alignment:       4
+    isTargetSpecific: false
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: combine_middle
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   liveins: $q2, $r0, $r1
+  ; CHECK:   renamable $q0 = MVE_VDUP32 renamable $r1, 0, $noreg, undef renamable $q0
+  ; CHECK:   $lr = MVE_DLSTP_32 killed renamable $r1
+  ; CHECK: bb.1 (align 4):
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.2(0x04000000)
+  ; CHECK:   liveins: $lr, $q0, $q2, $r0
+  ; CHECK:   renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 0, $noreg
+  ; CHECK:   renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 0, $noreg, killed renamable $q2
+  ; CHECK:   MVE_VPTv4f32 8, renamable $q1, renamable $q0, 12, implicit-def $vpr
+  ; CHECK:   renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+  ; CHECK:   MVE_VPST 4, implicit $vpr
+  ; CHECK:   dead renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, killed renamable $vpr, killed renamable $q1
+  ; CHECK:   $lr = MVE_LETP killed renamable $lr, %bb.1
+  ; CHECK: bb.2:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.3 (align 4):
+  ; CHECK:   CONSTPOOL_ENTRY 0, %const.0, 4
+  bb.0:
+    successors: %bb.6(0x80000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $r3, dead $cpsr = nuw nsw tADDi3 renamable $r1, 3, 14 /* CC::al */, $noreg
+    renamable $q0 = MVE_VDUP32 killed renamable $r1, 0, $noreg, undef renamable $q0
+    renamable $r3 = t2ANDri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $lr = t2SUBri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $r3, dead $cpsr = tMOVi8 1, 14 /* CC::al */, $noreg
+    renamable $lr = nuw nsw t2ADDrs killed renamable $r3, killed renamable $lr, 19, 14 /* CC::al */, $noreg, $noreg
+    $lr = t2DoLoopStart renamable $lr
+
+  bb.6 (align 4):
+    successors: %bb.6(0x7c000000), %bb.8(0x04000000)
+    liveins: $lr, $q0, $r0, $r1, $r2, $q2
+
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    renamable $vpr = MVE_VCTP32 renamable $r1, 0, $noreg
+    renamable $r1, dead $cpsr = tSUBi8 killed renamable $r1, 4, 14 /* CC::al */, $noreg
+    MVE_VPST 2, implicit $vpr
+    renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 1, renamable $vpr
+    renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+    renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 1, killed renamable $vpr
+    MVE_VPST 8, implicit $vpr
+    renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+    MVE_VPST 4, implicit $vpr
+    renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, renamable $vpr, renamable $q1
+    t2LoopEnd renamable $lr, %bb.6, implicit-def dead $cpsr
+    tB %bb.8, 14 /* CC::al */, $noreg
+
+  bb.8:
+    liveins: $r2, $r12, $q0
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.9 (align 4):
+    CONSTPOOL_ENTRY 0, %const.0, 4
+
+...
+---
+name:            combine_last
+alignment:       8
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:
+  - id:              0
+    value:           float 0xC7EFFFFFE0000000
+    alignment:       4
+    isTargetSpecific: false
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: combine_last
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   liveins: $q2, $r0, $r1
+  ; CHECK:   renamable $q0 = MVE_VDUP32 renamable $r1, 0, $noreg, undef renamable $q0
+  ; CHECK:   $lr = MVE_DLSTP_32 killed renamable $r1
+  ; CHECK: bb.1 (align 4):
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.2(0x04000000)
+  ; CHECK:   liveins: $lr, $q0, $q2, $r0
+  ; CHECK:   renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 0, $noreg
+  ; CHECK:   MVE_VPTv4f32 8, renamable $q1, renamable $q0, 12, implicit-def $vpr
+  ; CHECK:   renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, killed renamable $vpr, killed renamable $q2
+  ; CHECK:   MVE_VPTv4f32 8, renamable $q2, renamable $q1, 12, implicit-def $vpr
+  ; CHECK:   dead renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, killed renamable $vpr, killed renamable $q1
+  ; CHECK:   $lr = MVE_LETP killed renamable $lr, %bb.1
+  ; CHECK: bb.2:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.3 (align 4):
+  ; CHECK:   CONSTPOOL_ENTRY 0, %const.0, 4
+  bb.0:
+    successors: %bb.6(0x80000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $r3, dead $cpsr = nuw nsw tADDi3 renamable $r1, 3, 14 /* CC::al */, $noreg
+    renamable $q0 = MVE_VDUP32 killed renamable $r1, 0, $noreg, undef renamable $q0
+    renamable $r3 = t2ANDri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $lr = t2SUBri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $r3, dead $cpsr = tMOVi8 1, 14 /* CC::al */, $noreg
+    renamable $lr = nuw nsw t2ADDrs killed renamable $r3, killed renamable $lr, 19, 14 /* CC::al */, $noreg, $noreg
+    $lr = t2DoLoopStart renamable $lr
+
+  bb.6 (align 4):
+    successors: %bb.6(0x7c000000), %bb.8(0x04000000)
+    liveins: $lr, $q0, $r0, $r1, $r2, $q2
+
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    renamable $vpr = MVE_VCTP32 renamable $r1, 0, $noreg
+    renamable $r1, dead $cpsr = tSUBi8 killed renamable $r1, 4, 14 /* CC::al */, $noreg
+    MVE_VPST 8, implicit $vpr
+    renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 1, renamable $vpr
+    MVE_VPST 2, implicit $vpr
+    renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 1, killed renamable $vpr
+    renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+    renamable $vpr = MVE_VCMPf32 renamable $q2, renamable $q1, 12, 1, killed renamable $vpr
+    MVE_VPST 8, implicit $vpr
+    renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, renamable $vpr, renamable $q1
+    t2LoopEnd renamable $lr, %bb.6, implicit-def dead $cpsr
+    tB %bb.8, 14 /* CC::al */, $noreg
+
+  bb.8:
+    liveins: $r2, $r12, $q0
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.9 (align 4):
+    CONSTPOOL_ENTRY 0, %const.0, 4
+
+...
+---
+name:            no_combination_diff_reg_value
+alignment:       8
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:
+  - id:              0
+    value:           float 0xC7EFFFFFE0000000
+    alignment:       4
+    isTargetSpecific: false
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: no_combination_diff_reg_value
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   liveins: $r0, $r1
+  ; CHECK:   renamable $q0 = MVE_VDUP32 renamable $r1, 0, $noreg, undef renamable $q0
+  ; CHECK:   $lr = MVE_DLSTP_32 killed renamable $r1
+  ; CHECK: bb.1 (align 4):
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.2(0x04000000)
+  ; CHECK:   liveins: $lr, $q0, $r0
+  ; CHECK:   renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 0, $noreg
+  ; CHECK:   renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 0, $noreg, killed renamable $q1
+  ; CHECK:   renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 0, killed $noreg
+  ; CHECK:   renamable $q0 = MVE_VORR renamable $q1, renamable $q1, 0, $noreg, killed renamable $q0
+  ; CHECK:   MVE_VPST 8, implicit $vpr
+  ; CHECK:   renamable $q0 = MVE_VORR killed renamable $q1, killed renamable $q1, 1, killed renamable $vpr, killed renamable $q0
+  ; CHECK:   $lr = MVE_LETP killed renamable $lr, %bb.1
+  ; CHECK: bb.2:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.3 (align 4):
+  ; CHECK:   CONSTPOOL_ENTRY 0, %const.0, 4
+  bb.0:
+    successors: %bb.6(0x80000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $r3, dead $cpsr = nuw nsw tADDi3 renamable $r1, 3, 14 /* CC::al */, $noreg
+    renamable $q0 = MVE_VDUP32 killed renamable $r1, 0, $noreg, undef renamable $q0
+    renamable $r3 = t2ANDri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $lr = t2SUBri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $r3, dead $cpsr = tMOVi8 1, 14 /* CC::al */, $noreg
+    renamable $lr = nuw nsw t2ADDrs killed renamable $r3, killed renamable $lr, 19, 14 /* CC::al */, $noreg, $noreg
+    $lr = t2DoLoopStart renamable $lr
+
+  bb.6 (align 4):
+    successors: %bb.6(0x7c000000), %bb.8(0x04000000)
+    liveins: $lr, $q0, $r0, $r1, $r2
+
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    renamable $vpr = MVE_VCTP32 renamable $r1, 0, $noreg
+    renamable $r1, dead $cpsr = tSUBi8 killed renamable $r1, 4, 14 /* CC::al */, $noreg
+    MVE_VPST 8, implicit $vpr
+    renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 1, renamable $vpr
+    MVE_VPST 4, implicit $vpr
+    renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, renamable $vpr, renamable $q1
+    renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 1, killed renamable $vpr
+    renamable $q0 = MVE_VORR renamable $q1, renamable $q1, 0, $noreg, killed renamable $q0
+    MVE_VPST 8, implicit $vpr
+    renamable $q0 = MVE_VORR killed renamable $q1, renamable $q1, 1, killed renamable $vpr, killed renamable $q0
+    t2LoopEnd renamable $lr, %bb.6, implicit-def dead $cpsr
+    tB %bb.8, 14 /* CC::al */, $noreg
+
+  bb.8:
+    liveins: $r2, $r12, $q0
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.9 (align 4):
+    CONSTPOOL_ENTRY 0, %const.0, 4
+
+...
+---
+name:            no_combination_vcmp_already_merged
+alignment:       8
+exposesReturnsTwice: false
+legalized:       false
+regBankSelected: false
+selected:        false
+failedISel:      false
+tracksRegLiveness: true
+hasWinCFI:       false
+registers:       []
+liveins:
+  - { reg: '$r0', virtual-reg: '' }
+  - { reg: '$r1', virtual-reg: '' }
+  - { reg: '$r2', virtual-reg: '' }
+frameInfo:
+fixedStack:      []
+stack:
+  - { id: 0, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+  - { id: 1, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
+      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
+      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
+callSites:       []
+debugValueSubstitutions: []
+constants:
+  - id:              0
+    value:           float 0xC7EFFFFFE0000000
+    alignment:       4
+    isTargetSpecific: false
+machineFunctionInfo: {}
+body:             |
+  ; CHECK-LABEL: name: no_combination_vcmp_already_merged
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   liveins: $q2, $r0, $r1
+  ; CHECK:   renamable $q0 = MVE_VDUP32 renamable $r1, 0, $noreg, undef renamable $q0
+  ; CHECK:   $lr = MVE_DLSTP_32 killed renamable $r1
+  ; CHECK: bb.1 (align 4):
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.2(0x04000000)
+  ; CHECK:   liveins: $lr, $q0, $q2, $r0
+  ; CHECK:   renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 0, $noreg
+  ; CHECK:   MVE_VPTv4f32 8, renamable $q1, renamable $q0, 12, implicit-def $vpr
+  ; CHECK:   renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+  ; CHECK:   MVE_VPST 8, implicit $vpr
+  ; CHECK:   dead renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, killed renamable $vpr, killed renamable $q1
+  ; CHECK:   $lr = MVE_LETP killed renamable $lr, %bb.1
+  ; CHECK: bb.2:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.3 (align 4):
+  ; CHECK:   CONSTPOOL_ENTRY 0, %const.0, 4
+  bb.0:
+    successors: %bb.6(0x80000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $r3, dead $cpsr = nuw nsw tADDi3 renamable $r1, 3, 14 /* CC::al */, $noreg
+    renamable $q0 = MVE_VDUP32 killed renamable $r1, 0, $noreg, undef renamable $q0
+    renamable $r3 = t2ANDri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $lr = t2SUBri killed renamable $r3, 4, 14 /* CC::al */, $noreg, $noreg
+    renamable $r3, dead $cpsr = tMOVi8 1, 14 /* CC::al */, $noreg
+    renamable $lr = nuw nsw t2ADDrs killed renamable $r3, killed renamable $lr, 19, 14 /* CC::al */, $noreg, $noreg
+    $lr = t2DoLoopStart renamable $lr
+
+  bb.6 (align 4):
+    successors: %bb.6(0x7c000000), %bb.8(0x04000000)
+    liveins: $lr, $q0, $r0, $r1, $r2, $q2
+
+    renamable $lr = t2LoopDec killed renamable $lr, 1
+    renamable $vpr = MVE_VCTP32 renamable $r1, 0, $noreg
+    renamable $r1, dead $cpsr = tSUBi8 killed renamable $r1, 4, 14 /* CC::al */, $noreg
+    MVE_VPST 8, implicit $vpr
+    renamable $r0, renamable $q1 = MVE_VLDRWU32_post killed renamable $r0, 16, 1, renamable $vpr
+    MVE_VPST 4, implicit $vpr
+    renamable $vpr = MVE_VCMPf32 renamable $q1, renamable $q0, 12, 1, killed renamable $vpr
+    renamable $q2 = MVE_VORR renamable $q1, renamable $q1, 1, renamable $vpr, killed renamable $q2
+    MVE_VPST 8, implicit $vpr
+    renamable $q1 = MVE_VORR killed renamable $q1, renamable $q0, 1, renamable $vpr, renamable $q1
+    t2LoopEnd renamable $lr, %bb.6, implicit-def dead $cpsr
+    tB %bb.8, 14 /* CC::al */, $noreg
+
+  bb.8:
+    liveins: $r2, $r12, $q0
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.9 (align 4):
+    CONSTPOOL_ENTRY 0, %const.0, 4
+
+...