[ARM] Simplification to ARMBlockPlacement Pass.

[malharJ]

Summary

It simplifies the logic by moving the predecessor (preHeader or it's predecessor) above the target (or loopExit),
instead of moving the target to after the predecessor.

Since the loopExit is no longer being moved, directions of any branches within/to it are unaffected.

While the predecessor is being moved, the backwards movement simplifies some considerations,
and the only consideration now required is that a forward WLS to the predecessor should not become backwards.

[malharJ]

I'm putting some details of my understanding of the existing logic, and how it is being simplified by this patch:

---

Existing logic:

When moving the loopExit forwards (after preheader), 4 things can be considered:

1. directionality of WLS (if any) within the loopExit should not regress (forward->backward).
2. directionality of LE (if any) within the loopExit should not regress (backward->forward).
3. directionality of WLS (if any) to the loopExit should not regress (forward->backward).
4. directionality of LE (if any) to the loopExit should not regress (backward->forward).

Existing code takes care to handle cases 1) and 4).
There is no need to handle 2) and 3) since they can not regress due to moving forward.

Simplification:

But if we move preHeader backwards (before loopExit), the same 4 points above can be considered but
now in the context of moving the preHeader:

• 1) & 2) above cannot occur since existing WLS is a terminator and there can't be another WLS/LE in the same block.
• 3) is the only case (ie. WLS to PreHeader) that needs to be handled
• 4) Any LE to PreHeader cannot regress if moving backwards.

[malharJ]

Rebased the patch on top of main branch to get the changes from D99649

[malharJ]

llvm/test/CodeGen/Thumb2/block-placement.mir

17–20

This test checked that the loopExit/target block is unaffected when moved forwards,
if it already contains a backwards WLS.

However, it has been removed since the updates in this patch imply that
the loopExit/target is no longer moved around.

22–25

This test was removed because it didn't seem correct/intent was not clear.
The two blocks containing the t2WhileLoopStartLR have exit branches to each other.

Just thought it might be better to remove it as the intent is unclear.
Otherwise, happy to leave it in if required.

[malharJ]

@SjoerdMeijer,

Do you think you can review this patch ?
It's not really a priority (but potentially a good simplification), but since you had reviewed the original patch, I thought I'd ask you first.

[SjoerdMeijer]

Adding @samtebbs as the original author. I need to get back into this, wrap my head around this; perhaps it's less ramping up for Sam. But I will do it anyway tomorrow.

[malharJ]

Corrected leftover (from the rebase) error in code.

[malharJ]

Updated failing tests.

[malharJ]

ping

[SjoerdMeijer]

Sorry for the delay.
First a few questions: just checking did you run all testing with this patch?

llvm/test/CodeGen/Thumb2/block-placement.mir

149

Looks like you're removing a few tests here. Are they not good tests to keep?

[malharJ]

I have run (and it passes)

make/ninja check-all

if that's what you mean by all testing ?

llvm/test/CodeGen/Thumb2/block-placement.mir

149

Please check the 2 comments here: https://reviews.llvm.org/D100094#inline-945352

[SjoerdMeijer]

In D100094#2711598, @malharJ wrote:

I have run (and it passes)

make/ninja check-all

if that's what you mean by all testing ?

I meant our downstream tests, and thus also double checking that performance is unaffected would be good too.

[dmgreen]

When comparing moving Pred before Exit, vs Exit after Pred, is either expected to be more or less efficient? Or are they both just a bit inefficient?

Please add this test case:

# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
# RUN: llc -run-pass arm-block-placement %s -o - | FileCheck %s

--- |
  target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
  target triple = "thumbv8.1m.main-arm-none-eabi"

  define i32 @c(i32 %d, i32 %e, i32 %f) #0 {
  entry:
    %g = alloca i16, align 2
    %0 = bitcast i16* %g to i8*
    call void @llvm.lifetime.start.p0i8(i64 2, i8* nonnull %0) #5
    %cond = icmp eq i32 %e, 8
    %cmp = icmp eq i32 %d, 0
    %or.cond = and i1 %cmp, %cond
    br i1 %or.cond, label %while.cond.preheader, label %sw.epilog

  while.cond.preheader:                             ; preds = %entry
    %tobool.not16 = icmp eq i32 %f, 0
    %1 = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %f)
    %2 = extractvalue { i32, i1 } %1, 1
    %3 = extractvalue { i32, i1 } %1, 0
    br i1 %2, label %while.body.preheader, label %sw.epilog

  while.body.preheader:                             ; preds = %while.cond.preheader
    br label %while.body

  while.cond1.preheader:                            ; preds = %while.body
    %4 = icmp eq i32 %f, 0
    %5 = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %f)
    %6 = extractvalue { i32, i1 } %5, 1
    %7 = extractvalue { i32, i1 } %5, 0
    br i1 %6, label %while.body3.preheader, label %sw.epilog

  while.body3.preheader:                            ; preds = %while.cond1.preheader
    br label %while.body3

  while.body:                                       ; preds = %while.body.preheader, %while.body
    %h.017 = phi i32 [ %10, %while.body ], [ undef, %while.body.preheader ]
    %8 = phi i32 [ %3, %while.body.preheader ], [ %12, %while.body ]
    %9 = call { <8 x i16>, i32 } @llvm.arm.mve.viwdup.v8i16(i32 %h.017, i32 0, i32 1)
    %10 = extractvalue { <8 x i16>, i32 } %9, 1
    %11 = extractvalue { <8 x i16>, i32 } %9, 0
    call void @llvm.arm.mve.vstr.scatter.offset.p0i16.v8i16.v8i16(i16* nonnull %g, <8 x i16> %11, <8 x i16> undef, i32 16, i32 1)
    %12 = call i32 @llvm.loop.decrement.reg.i32(i32 %8, i32 1)
    %13 = icmp ne i32 %12, 0
    br i1 %13, label %while.body, label %while.cond1.preheader

  while.body3:                                      ; preds = %while.body3.preheader, %while.body3
    %.pn = phi { <8 x i16>, i32 } [ %15, %while.body3 ], [ %9, %while.body3.preheader ]
    %14 = phi i32 [ %7, %while.body3.preheader ], [ %17, %while.body3 ]
    %h.121 = extractvalue { <8 x i16>, i32 } %.pn, 1
    %15 = call { <8 x i16>, i32 } @llvm.arm.mve.viwdup.v8i16(i32 %h.121, i32 0, i32 2)
    %16 = extractvalue { <8 x i16>, i32 } %15, 0
    call void @llvm.arm.mve.vstr.scatter.offset.p0i16.v8i16.v8i16(i16* nonnull %g, <8 x i16> %16, <8 x i16> undef, i32 16, i32 1)
    %17 = call i32 @llvm.loop.decrement.reg.i32(i32 %14, i32 1)
    %18 = icmp ne i32 %17, 0
    br i1 %18, label %while.body3, label %sw.epilog

  sw.epilog:                                        ; preds = %while.body3, %while.cond1.preheader, %while.cond.preheader, %entry
    %19 = bitcast i16* %g to i8*
    call void @llvm.lifetime.end.p0i8(i64 2, i8* nonnull %19) #5
    ret i32 undef
  }

  declare void @llvm.lifetime.start.p0i8(i64 immarg, i8* nocapture) #1
  declare { <8 x i16>, i32 } @llvm.arm.mve.viwdup.v8i16(i32, i32, i32) #2
  declare void @llvm.arm.mve.vstr.scatter.offset.p0i16.v8i16.v8i16(i16*, <8 x i16>, <8 x i16>, i32, i32) #3
  declare void @llvm.lifetime.end.p0i8(i64 immarg, i8* nocapture) #1
  declare { i32, i1 } @llvm.test.start.loop.iterations.i32(i32) #4
  declare i32 @llvm.loop.decrement.reg.i32(i32, i32) #4

  attributes #0 = { "target-features"="+armv8.1-m.main,+hwdiv,+mve.fp,+ras,+thumb-mode" }

...
---
name:            c
alignment:       2
tracksRegLiveness: true
liveins:
  - { reg: '$r0', virtual-reg: '' }
  - { reg: '$r1', virtual-reg: '' }
  - { reg: '$r2', virtual-reg: '' }
stack:
  - { id: 0, name: g, type: default, offset: -10, size: 2, alignment: 2,
      stack-id: default, callee-saved-register: '', callee-saved-restored: true,
      local-offset: -2, debug-info-variable: '', debug-info-expression: '',
      debug-info-location: '' }
  - { id: 1, name: '', type: spill-slot, offset: -4, size: 4, alignment: 4,
      stack-id: default, callee-saved-register: '$lr', callee-saved-restored: false,
      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
  - { id: 2, name: '', type: spill-slot, offset: -8, size: 4, alignment: 4,
      stack-id: default, callee-saved-register: '$r7', callee-saved-restored: true,
      debug-info-variable: '', debug-info-expression: '', debug-info-location: '' }
body:             |
  ; CHECK-LABEL: name: c
  ; CHECK: bb.0.entry:
  ; CHECK:   successors: %bb.1(0x55555555), %bb.2(0x2aaaaaab)
  ; CHECK:   liveins: $r0, $r1, $r2, $r7, $lr
  ; CHECK:   frame-setup tPUSH 14 /* CC::al */, $noreg, killed $r7, killed $lr, implicit-def $sp, implicit $sp
  ; CHECK:   frame-setup CFI_INSTRUCTION def_cfa_offset 8
  ; CHECK:   frame-setup CFI_INSTRUCTION offset $lr, -4
  ; CHECK:   frame-setup CFI_INSTRUCTION offset $r7, -8
  ; CHECK:   $sp = frame-setup tSUBspi $sp, 1, 14 /* CC::al */, $noreg
  ; CHECK:   frame-setup CFI_INSTRUCTION def_cfa_offset 12
  ; CHECK:   tCMPi8 killed renamable $r0, 0, 14 /* CC::al */, $noreg, implicit-def $cpsr
  ; CHECK:   t2IT 0, 8, implicit-def $itstate
  ; CHECK:   tCMPi8 killed renamable $r1, 8, 0 /* CC::eq */, killed $cpsr, implicit-def $cpsr, implicit killed $itstate
  ; CHECK:   t2Bcc %bb.2, 0 /* CC::eq */, killed $cpsr
  ; CHECK:   tB %bb.1, 14 /* CC::al */, $noreg
  ; CHECK: bb.5.while.cond1.preheader:
  ; CHECK:   successors: %bb.6(0x40000000), %bb.1(0x40000000)
  ; CHECK:   liveins: $r0, $r1, $r2
  ; CHECK:   renamable $lr = t2WhileLoopStartLR killed renamable $r2, %bb.1, implicit-def dead $cpsr
  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
  ; CHECK: bb.2.while.cond.preheader:
  ; CHECK:   successors: %bb.3(0x40000000), %bb.1(0x40000000)
  ; CHECK:   liveins: $r2
  ; CHECK:   renamable $lr = t2WhileLoopStartLR renamable $r2, %bb.1, implicit-def dead $cpsr
  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
  ; CHECK: bb.1.sw.epilog:
  ; CHECK:   $sp = frame-destroy tADDspi $sp, 1, 14 /* CC::al */, $noreg
  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc, implicit undef $r0
  ; CHECK: bb.3.while.body.preheader:
  ; CHECK:   successors: %bb.4(0x80000000)
  ; CHECK:   liveins: $lr, $r2
  ; CHECK:   renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
  ; CHECK:   renamable $r1 = t2ADDri $sp, 2, 14 /* CC::al */, $noreg, $noreg
  ; CHECK:   renamable $r0 = IMPLICIT_DEF
  ; CHECK: bb.4.while.body (align 4):
  ; CHECK:   successors: %bb.4(0x7c000000), %bb.5(0x04000000)
  ; CHECK:   liveins: $lr, $r0, $r1, $r2, $r3
  ; CHECK:   renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 1, 0, $noreg, undef renamable $q0
  ; CHECK:   MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
  ; CHECK: bb.6.while.body3.preheader:
  ; CHECK:   successors: %bb.7(0x80000000)
  ; CHECK:   liveins: $lr, $r0, $r1
  ; CHECK:   renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
  ; CHECK: bb.7.while.body3 (align 4):
  ; CHECK:   successors: %bb.7(0x7c000000), %bb.1(0x04000000)
  ; CHECK:   liveins: $lr, $r0, $r1, $r3
  ; CHECK:   renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 2, 0, $noreg, undef renamable $q0
  ; CHECK:   MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.7, implicit-def dead $cpsr
  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg

  bb.0.entry:
    successors: %bb.7(0x80000000), %bb.1(0x40000000)
    liveins: $r0, $r1, $r2, $r7, $lr

    frame-setup tPUSH 14 /* CC::al */, $noreg, killed $r7, killed $lr, implicit-def $sp, implicit $sp
    frame-setup CFI_INSTRUCTION def_cfa_offset 8
    frame-setup CFI_INSTRUCTION offset $lr, -4
    frame-setup CFI_INSTRUCTION offset $r7, -8
    $sp = frame-setup tSUBspi $sp, 1, 14 /* CC::al */, $noreg
    frame-setup CFI_INSTRUCTION def_cfa_offset 12
    tCMPi8 killed renamable $r0, 0, 14 /* CC::al */, $noreg, implicit-def $cpsr
    t2IT 0, 8, implicit-def $itstate
    tCMPi8 killed renamable $r1, 8, 0 /* CC::eq */, killed $cpsr, implicit-def $cpsr, implicit killed $itstate
    t2Bcc %bb.1, 0 /* CC::eq */, killed $cpsr

  bb.7.sw.epilog:
    $sp = frame-destroy tADDspi $sp, 1, 14 /* CC::al */, $noreg
    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc, implicit undef $r0

  bb.1.while.cond.preheader:
    successors: %bb.3(0x40000000), %bb.7(0x40000000)
    liveins: $r2

    renamable $lr = t2WhileLoopStartLR renamable $r2, %bb.7, implicit-def dead $cpsr
    t2B %bb.3, 14 /* CC::al */, $noreg

  bb.3.while.body.preheader:
    successors: %bb.4(0x80000000)
    liveins: $lr, $r2

    renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
    renamable $r1 = t2ADDri $sp, 2, 14 /* CC::al */, $noreg, $noreg
    renamable $r0 = IMPLICIT_DEF

  bb.4.while.body (align 4):
    successors: %bb.4(0x7c000000), %bb.2(0x04000000)
    liveins: $lr, $r0, $r1, $r2, $r3

    renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 1, 0, $noreg, undef renamable $q0
    MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
    t2B %bb.2, 14 /* CC::al */, $noreg

  bb.2.while.cond1.preheader:
    successors: %bb.5(0x40000000), %bb.7(0x40000000)
    liveins: $r0, $r1, $r2

    renamable $lr = t2WhileLoopStartLR killed renamable $r2, %bb.7, implicit-def dead $cpsr
    t2B %bb.5, 14 /* CC::al */, $noreg

  bb.5.while.body3.preheader:
    successors: %bb.6(0x80000000)
    liveins: $lr, $r0, $r1

    renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg

  bb.6.while.body3 (align 4):
    successors: %bb.6(0x7c000000), %bb.7(0x04000000)
    liveins: $lr, $r0, $r1, $r3

    renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 2, 0, $noreg, undef renamable $q0
    MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.6, implicit-def dead $cpsr
    t2B %bb.7, 14 /* CC::al */, $noreg

...

llvm/lib/Target/ARM/ARMBlockPlacement.cpp

225–227

I think it's worth just renumbering all nodes and recalculating all offsets.

[malharJ]

Addressed review comments:

• Added a test for multiple predecessors.
• Updated code to renumber all blocks + compute offset of all blocks.

[malharJ]

@SjoerdMeijer and @dmgreen , I can confirm the performance of downstream tests is unaffected by this patch.

[samtebbs]

llvm/test/CodeGen/Thumb2/block-placement.mir

22–25

Sorry for not looking at this sooner. The backwards_branch_sibling test made sure that if a backwards WLS (bb3 -> bb1) in this case was introduced by moving the loop exit (bb1) then no transformation would occur. It would still be useful to include this but adapt it to make sure that no moving happens if it would create a forwards LE.

[samtebbs]

llvm/test/CodeGen/Thumb2/block-placement.mir

22–25

I meant to write "backwards WLS (bb3 -> bb1 in this case) was introduced..."!

[malharJ]

Updated tests:

• Added a test that checks if Predecessor is moved backwards if moving it does not convert a forward WLS into backwards WLS.
• simplified the test file a bit further.

llvm/test/CodeGen/Thumb2/block-placement.mir

17–20

Instead of removing this, I've now replaced this with an equivalent test: backwards_branch_forwards_wls.

(The naming is as such because there is already another test called backwards_branch_backwards_wls, which
tests the opposite scenario)

22–25

Thanks for explaining this, but there are a few things I'd like to mention:

1. I've added a test (for checking that backwards WLS is not introduced): backwards_branch_target_already_backwards.

(It was essentially a replacement for the older test named backwards_branch_forwards_le)

So would it be ok to not include backwards_branch_sibling test since it's already being tested ?

It would still be useful to include this but adapt it to make sure that no moving happens if it would create a forwards LE.

With the new logic, a forward LE cannot be introduced since only the PreHeader is being moved backwards.
Any LE to it will remain backwards, and there is no chance of it containing a LE (since it already contains WLS which is a terminator) unless Im mistaken there.

[dmgreen]

Thanks for the tests. In the interest of fixing the bug that this fixes, LGTM.

llvm/lib/Target/ARM/ARMBlockPlacement.cpp

130

Can you format this line so it flows better.

llvm/test/CodeGen/Thumb2/block-placement.mir

195

Should this be bb.1 and bb.4?

[malharJ]

Minor updates: formatting + test

[dmgreen]

Thanks

llvm/lib/Target/ARM/ARMBlockPlacement.cpp

130

I guess it might be too long to format nicely.

llvm/lib/Target/ARM/ARMBlockPlacement.cpp

private:
   std::unique_ptr<ARMBasicBlockUtils> BBUtils = nullptr;
   MachineLoopInfo *MLI = nullptr;
 
 public:
   static char ID;
   ARMBlockPlacement() : MachineFunctionPass(ID) {}
 
   bool runOnMachineFunction(MachineFunction &MF) override;
-  void moveBasicBlock(MachineBasicBlock *BB, MachineBasicBlock *After);
+  void moveBasicBlock(MachineBasicBlock *BB, MachineBasicBlock *Before);
   bool blockIsBefore(MachineBasicBlock *BB, MachineBasicBlock *Other);
   bool fixBackwardsWLS(MachineLoop *ML);
   bool processPostOrderLoops(MachineLoop *ML);
 
   void getAnalysisUsage(AnalysisUsage &AU) const override {
     AU.setPreservesCFG();
     AU.addRequired<MachineLoopInfo>();
     MachineFunctionPass::getAnalysisUsage(AU);
static MachineInstr *findWLS(MachineLoop *ML) {
   if (WlsInstr)
     return WlsInstr;
   if (Predecessor->pred_size() == 1)
     return findWLSInBlock(*Predecessor->pred_begin());
   return nullptr;
 }
 
 /// Checks if loop has a backwards branching WLS, and if possible, fixes it.
-/// This requires checking the preheader (or it's predecessor) for a WLS and if
-/// its target is before it.
-/// If moving the target block wouldn't produce another backwards WLS or a new
-/// forwards LE branch, then move the target block after the preheader (or it's
-/// predecessor).
+/// This requires checking the predecessor (ie. preheader or it's predecessor)
+/// for a WLS and if its loopExit/target is before it.
+/// If moving the predecessor won't convert a WLS (to the predecessor) from
+/// a forward to a backward branching WLS, then move the predecessor block
+/// to before the loopExit/target.
 bool ARMBlockPlacement::fixBackwardsWLS(MachineLoop *ML) {
   MachineInstr *WlsInstr = findWLS(ML);
   if (!WlsInstr)
     return false;
 
   MachineBasicBlock *Predecessor = WlsInstr->getParent();
   MachineBasicBlock *LoopExit = WlsInstr->getOperand(2).getMBB();
-  // We don't want to move the function's entry block.
+
+  // We don't want to move Preheader to before the function's entry block.
   if (!LoopExit->getPrevNode())
     return false;
   if (blockIsBefore(Predecessor, LoopExit))
     return false;
   LLVM_DEBUG(dbgs() << DEBUG_PREFIX << "Found a backwards WLS from "
                     << Predecessor->getFullName() << " to "
                     << LoopExit->getFullName() << "\n");
 
-  // Make sure that moving the target block doesn't cause any of its WLSs
-  // that were previously not backwards to become backwards
-  bool CanMove = true;
-  MachineInstr *WlsInLoopExit = findWLSInBlock(LoopExit);
-  if (WlsInLoopExit) {
-    // An example loop structure where the LoopExit can't be moved, since
-    // bb1's WLS will become backwards once it's moved after bb3
-    // bb1:          - LoopExit
-    //      WLS bb2
-    // bb2:          - LoopExit2
-    //      ...
-    // bb3:          - Predecessor
-    //      WLS bb1
-    // bb4:          - Header
-    MachineBasicBlock *LoopExit2 = WlsInLoopExit->getOperand(2).getMBB();
-    // If the WLS from LoopExit to LoopExit2 is already backwards then
-    // moving LoopExit won't affect it, so it can be moved. If LoopExit2 is
-    // after the Predecessor then moving will keep it as a forward branch, so it
-    // can be moved. If LoopExit2 is between the Predecessor and LoopExit then
-    // moving LoopExit will make it a backwards branch, so it can't be moved
-    // since we'd fix one and introduce one backwards branch.
-    // TODO: Analyse the blocks to make a decision if it would be worth
-    // moving LoopExit even if LoopExit2 is between the Predecessor and
-    // LoopExit.
-    if (!blockIsBefore(LoopExit2, LoopExit) &&
-        (LoopExit2 == Predecessor || blockIsBefore(LoopExit2, Predecessor))) {
-      LLVM_DEBUG(dbgs() << DEBUG_PREFIX
-                        << "Can't move the target block as it would "
-                           "introduce a new backwards WLS branch\n");
-      CanMove = false;
-    }
-  }
-
-  if (CanMove) {
-    // Make sure no LEs become forwards.
-    // An example loop structure where the LoopExit can't be moved, since
-    // bb2's LE will become forwards once bb1 is moved after bb3.
-    // bb1:           - LoopExit
-    // bb2:
-    //      LE  bb1  - Terminator
-    // bb3:          - Predecessor
-    //      WLS bb1
-    // bb4:          - Header
-    for (auto It = LoopExit->getIterator(); It != Predecessor->getIterator();
-         It++) {
-      MachineBasicBlock *MBB = &*It;
-      for (auto &Terminator : MBB->terminators()) {
-        if (Terminator.getOpcode() != ARM::t2LoopEnd &&
-            Terminator.getOpcode() != ARM::t2LoopEndDec)
-          continue;
-        MachineBasicBlock *LETarget = Terminator.getOperand(2).getMBB();
-        // The LE will become forwards branching if it branches to LoopExit
-        // which isn't allowed by the architecture, so we should avoid
-        // introducing these.
-        // TODO: Analyse the blocks to make a decision if it would be worth
-        // moving LoopExit even if we'd introduce a forwards LE
-        if (LETarget == LoopExit) {
-          LLVM_DEBUG(dbgs() << DEBUG_PREFIX
-                            << "Can't move the target block as it would "
-                               "introduce a new forwards LE branch\n");
-          CanMove = false;
-          break;
-        }
+  // Make sure no forward branching WLSs to the Predecessor become backwards
+  // branching. An example loop structure where the Predecessor can't be moved,
+  // since bb2's WLS will become forwards once bb3 is moved before/above bb1.
+  //
+  //      WLS  bb3
+  for (auto It = ++LoopExit->getIterator(); It != Predecessor->getIterator();
+       ++It) {
+      if (Terminator.getOpcode() != ARM::t2WhileLoopStartLR)
+      MachineBasicBlock *WLSTarget = Terminator.getOperand(2).getMBB();
+      // moving Preheader even if we'd introduce a backwards WLS
+      if (WLSTarget == Predecessor) {
+        LLVM_DEBUG(
+            dbgs() << DEBUG_PREFIX
+                   << "Can't move Predecessor"
+                      "block as it would convert a WLS from forward to a "
+                      "backwards branching WLS\n");
+        return false;
       }
     }
   }
 
-  if (CanMove)
-    moveBasicBlock(LoopExit, Predecessor);
-
-  return CanMove;
+  moveBasicBlock(Predecessor, LoopExit);
+  return true;
 }
 
 /// Updates ordering (of WLS BB and their loopExits) in inner loops first
 /// Returns true if any change was made in any of the loops
 bool ARMBlockPlacement::processPostOrderLoops(MachineLoop *ML) {
   bool Changed = false;
   for (auto *InnerML : *ML)
     Changed |= processPostOrderLoops(InnerML);
bool ARMBlockPlacement::runOnMachineFunction(MachineFunction &MF) {
   return Changed;
 }
 
 bool ARMBlockPlacement::blockIsBefore(MachineBasicBlock *BB,
                                       MachineBasicBlock *Other) {
   return BBUtils->getOffsetOf(Other) > BBUtils->getOffsetOf(BB);
 }
 
-/// Moves a given MBB to be positioned after another MBB while maintaining
-/// existing control flow
+// Moves a BasicBlock before another, without changing the control flow
 void ARMBlockPlacement::moveBasicBlock(MachineBasicBlock *BB,
-                                       MachineBasicBlock *After) {
-  LLVM_DEBUG(dbgs() << DEBUG_PREFIX << "Moving " << BB->getName() << " after "
-                    << After->getName() << "\n");
+                                       MachineBasicBlock *Before) {
+  LLVM_DEBUG(dbgs() << DEBUG_PREFIX << "Moving " << BB->getName() << " before "
+                    << Before->getName() << "\n");
   MachineBasicBlock *BBPrevious = BB->getPrevNode();
   assert(BBPrevious && "Cannot move the function entry basic block");
-  MachineBasicBlock *AfterNext = After->getNextNode();
   MachineBasicBlock *BBNext = BB->getNextNode();
 
-  BB->moveAfter(After);
+  MachineBasicBlock *BeforePrev = Before->getPrevNode();
+  assert(BeforePrev &&
+         "Cannot move the given block to before the function entry block");
+  MachineFunction *F = BB->getParent();
+  BB->moveBefore(Before);
 
   // Since only the blocks are to be moved around (but the control flow must
   // not change), if there were any fall-throughs (to/from adjacent blocks),
   // replace with unconditional branch to the fall through block.
   auto FixFallthrough = [&](MachineBasicBlock *From, MachineBasicBlock *To) {
     LLVM_DEBUG(dbgs() << DEBUG_PREFIX << "Checking for fallthrough from "
                       << From->getName() << " to " << To->getName() << "\n");
     assert(From->isSuccessor(To) &&
if (!Terminator.isUnconditionalBranch()) {
                         << From->getName() << " to " << To->getName() << ": "
                         << *MIB.getInstr());
     }
   };
 
   // Fix fall-through to the moved BB from the one that used to be before it.
   if (BBPrevious->isSuccessor(BB))
     FixFallthrough(BBPrevious, BB);
-  // Fix fall through from the destination BB to the one that used to follow.
-  if (AfterNext && After->isSuccessor(AfterNext))
-    FixFallthrough(After, AfterNext);
+  // Fix fall through from the destination BB to the one that used to before it.
+  if (BeforePrev->isSuccessor(Before))
+    FixFallthrough(BeforePrev, Before);
   // Fix fall through from the moved BB to the one that used to follow.
   if (BBNext && BB->isSuccessor(BBNext))
     FixFallthrough(BB, BBNext);
 
-  BBUtils->adjustBBOffsetsAfter(After);
+  F->RenumberBlocks();
+  BBUtils->computeAllBlockSizes();
+  BBUtils->adjustBBOffsetsAfter(&F->front());
 }

llvm/test/CodeGen/Thumb2/block-placement.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc -mtriple=thumbv8.1m.main-none-eabi -mattr=+mve -run-pass=arm-block-placement %s -o - | FileCheck %s
 --- |
-  ; Checks that loopExitBlock gets moved (in forward direction) if there is a backwards WLS to it.
-  define void @backwards_branch(i32 %N, i32* nocapture %a, i32* nocapture readonly %b) local_unnamed_addr #0 {
+
+  ; Checks that Predecessor gets moved (to before the LoopExit) if it contains a backward WLS.
+  define void @backwards_branch(i32 %N, i32* nocapture %a, i32* nocapture readonly %b)   {
   entry:
     unreachable
   }
 
-  ; Checks that loopExitBlock does not get reordered (since it is entry block) even if there is a backwards WLS to it.
-  define void @backwards_branch_entry_block(i32 %N, i32* nocapture %a, i32* nocapture readonly %b) local_unnamed_addr #0 {
+  ; Checks that Predecessor (containing a backwards WLS) does not get moved to before the loopExit if it is the entry block.
+  define void @backwards_branch_entry_block(i32 %N, i32* nocapture %a, i32* nocapture readonly %b)   {
   entry:
     unreachable
   }
 
-  ; Checks that loopExitBlock (containing a backwards WLS) is moved (in forward direction) if there is a backwards WLS to it.
-  define void @backwards_branch_target_already_backwards(i32 %N, i32* nocapture %a, i32* nocapture readonly %b) local_unnamed_addr #0 {
+  ; Checks that Predecessor (to which a forward WLS exists) is not moved if moving it would cause the WLS to become backwards branching.
+  define void @backwards_branch_backwards_wls(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c)   {
   entry:
     unreachable
   }
 
-  define void @backwards_branch_sibling(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c) local_unnamed_addr #0 {
+  ; Checks that a MachineFunction is unaffected if it doesn't contain any WLS (pseudo) instruction.
+  define void @no_predecessor(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c)   {
   entry:
     unreachable
   }
 
-  ; Checks that loopExitBlock (to which a backwards LE exists) is not moved if moving it would cause the LE to become forwards branching.
-  define void @backwards_branch_forwards_le(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c) local_unnamed_addr #0 {
+  ; Within a nested (Both the WLS and loopExit are at depth=3 here) loop, checks that Predecessor
+  ; gets moved (in backward direction) if there exists a backdwards WLS from it to the LoopExit.
+  define void @nested_loops(i32 %n, i32 %m, i32 %l, i8* noalias %X, i8* noalias %Y)   {
   entry:
     unreachable
   }
 
-  ; Checks that a MachineFunction is unaffected if it doesn't contain any WLS (pseudo) instruction.
-  define void @no_preheader(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c) local_unnamed_addr #0 {
+  ; Checks that Predecessor (to which a forward WLS exists) is moved if moving it would NOT cause the WLS
+  ; to become backwards branching.
+  define void @backwards_branch_forwards_wls(i32 %N, i32 %M, i32* nocapture %a, i32* nocapture %b, i32* nocapture %c) {
   entry:
     unreachable
   }
 
-  ; Within a nested loop, checks that loopExit gets moved (in forward direction) if there exists a backwards WLS to it.
-  ; Both the WLS and loopExit are at depth=3.
-  define void @nested_loops(i32 %n, i32 %m, i32 %l, i8* noalias %X, i8* noalias %Y) local_unnamed_addr #0 {
+  ; Checks that multiple predecessor case is handled appropriately
+  define void @multiple_predecessors(i32 %d, i32 %e, i32 %f) {
   entry:
     unreachable
   }
 
-  declare dso_local i32 @g(...) local_unnamed_addr #1
+  declare dso_local i32 @g(...)
 
-  declare dso_local i32 @h(...) local_unnamed_addr #1
+  declare dso_local i32 @h(...)
 
 ...
 ---
 name:            backwards_branch
 body:             |
   ; CHECK-LABEL: name: backwards_branch
   ; CHECK: bb.0:
-  ; CHECK:   successors: %bb.2(0x80000000)
+  ; CHECK:   successors: %bb.1(0x80000000)
   ; CHECK:   tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
   ; CHECK:   t2IT 11, 8, implicit-def $itstate
   ; CHECK:   frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
-  ; CHECK: bb.2:
+  ; CHECK: bb.1:
   ; CHECK:   successors: %bb.3(0x80000000)
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.2, implicit-def dead $cpsr
   ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
-  ; CHECK: bb.1:
+  ; CHECK: bb.2:
   ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
   ; CHECK: bb.3:
-  ; CHECK:   successors: %bb.3(0x7c000000), %bb.1(0x04000000)
+  ; CHECK:   successors: %bb.3(0x7c000000), %bb.2(0x04000000)
   ; CHECK:   renamable $r0 = tLDRi renamable $r2, 0, 14 /* CC::al */, $noreg
   ; CHECK:   tSTRi killed renamable $r0, renamable $r1, 0, 14 /* CC::al */, $noreg
   ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
+  ; CHECK:   t2B %bb.2, 14 /* CC::al */, $noreg
   bb.0:
     successors: %bb.2(0x80000000)
     liveins: $r0, $r1, $r2, $lr
 
     tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
     t2IT 11, 8, implicit-def $itstate
     frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
 
bb.3:
 
     renamable $r0 = tLDRi renamable $r2, 0, 14 /* CC::al */, $noreg
     tSTRi killed renamable $r0, renamable $r1, 0, 14 /* CC::al */, $noreg
     renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
     t2B %bb.1, 14 /* CC::al */, $noreg
 
 ...
 ---
-name:            backwards_branch_target_already_backwards
-body:             |
-  ; CHECK-LABEL: name: backwards_branch_target_already_backwards
-  ; CHECK: bb.0:
-  ; CHECK:   successors: %bb.2(0x50000000), %bb.1(0x30000000)
-  ; CHECK:   tCMPi8 $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-  ; CHECK:   t2Bcc %bb.1, 11 /* CC::lt */, killed $cpsr
-  ; CHECK:   t2B %bb.2, 14 /* CC::al */, $noreg
-  ; CHECK: bb.2:
-  ; CHECK:   successors: %bb.3(0x80000000)
-  ; CHECK:   $lr = tMOVr $r0, 14 /* CC::al */, $noreg
-  ; CHECK:   renamable $r0 = t2ADDrs killed renamable $r2, killed $r0, 18, 14 /* CC::al */, $noreg, $noreg
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $lr, %bb.1, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
-  ; CHECK: bb.1:
-  ; CHECK:   successors: %bb.4(0x80000000)
-  ; CHECK:   tCMPi8 renamable $r1, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-  ; CHECK:   t2IT 11, 8, implicit-def $itstate
-  ; CHECK:   frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r1, %bb.0, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.4, 14 /* CC::al */, $noreg
-  ; CHECK: bb.3:
-  ; CHECK:   successors: %bb.3(0x7c000000), %bb.1(0x04000000)
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
-  ; CHECK: bb.4:
-  ; CHECK:   successors: %bb.5(0x80000000)
-  ; CHECK:   renamable $r0 = t2ADDrs killed renamable $r3, renamable $r1, 18, 14 /* CC::al */, $noreg, $noreg
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r1, %bb.6, implicit-def dead $cpsr
-  ; CHECK: bb.5:
-  ; CHECK:   successors: %bb.5(0x7c000000), %bb.6(0x04000000)
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
-  ; CHECK: bb.6:
-  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
-  bb.0:
-    successors: %bb.1(0x50000000), %bb.3(0x30000000)
-    liveins: $r0, $r1, $r2, $r3, $lr
-
-    tCMPi8 $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-    t2Bcc %bb.3, 11 /* CC::lt */, killed $cpsr
-    t2B %bb.1, 14 /* CC::al */, $noreg
-
-  bb.3:
-    successors: %bb.4(0x80000000)
-    liveins: $r1, $r3
-
-    tCMPi8 renamable $r1, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-    t2IT 11, 8, implicit-def $itstate
-    frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
-    $lr = t2WhileLoopStartLR killed renamable $r1, %bb.0, implicit-def dead $cpsr
-    t2B %bb.4, 14 /* CC::al */, $noreg
-
-  bb.1:
-    successors: %bb.2(0x80000000)
-    liveins: $r0, $r1, $r2, $r3
-
-    $lr = tMOVr $r0, 14 /* CC::al */, $noreg
-    renamable $r0 = t2ADDrs killed renamable $r2, killed $r0, 18, 14 /* CC::al */, $noreg, $noreg
-    $lr = t2WhileLoopStartLR killed renamable $lr, %bb.3, implicit-def dead $cpsr
-
-  bb.2:
-    successors: %bb.2(0x7c000000), %bb.3(0x04000000)
-    liveins: $lr, $r0, $r1, $r3
-
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.2, implicit-def dead $cpsr
-    t2B %bb.3, 14 /* CC::al */, $noreg
-
-  bb.4:
-    successors: %bb.5(0x80000000)
-    liveins: $r1, $r3
-
-    renamable $r0 = t2ADDrs killed renamable $r3, renamable $r1, 18, 14 /* CC::al */, $noreg, $noreg
-    $lr = t2WhileLoopStartLR killed renamable $r1, %bb.6, implicit-def dead $cpsr
-
-  bb.5:
-    successors: %bb.5(0x7c000000), %bb.6(0x04000000)
-    liveins: $lr, $r0
-
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
-    t2B %bb.6, 14 /* CC::al */, $noreg
-
-  bb.6:
-    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
-
-...
----
-name:            backwards_branch_sibling
-body:             |
-  ; CHECK-LABEL: name: backwards_branch_sibling
-  ; CHECK: bb.0:
-  ; CHECK:   successors: %bb.2(0x50000000), %bb.1(0x30000000)
-  ; CHECK:   tCMPi8 $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-  ; CHECK:   t2Bcc %bb.1, 11 /* CC::lt */, killed $cpsr
-  ; CHECK:   t2B %bb.2, 14 /* CC::al */, $noreg
-  ; CHECK: bb.1:
-  ; CHECK:   successors: %bb.4(0x80000000)
-  ; CHECK:   tCMPi8 renamable $r1, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-  ; CHECK:   t2IT 11, 8, implicit-def $itstate
-  ; CHECK:   frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r1, %bb.2, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.4, 14 /* CC::al */, $noreg
-  ; CHECK: bb.2:
-  ; CHECK:   successors: %bb.3(0x80000000)
-  ; CHECK:   $lr = tMOVr $r0, 14 /* CC::al */, $noreg
-  ; CHECK:   renamable $r0 = t2ADDrs killed renamable $r2, killed $r0, 18, 14 /* CC::al */, $noreg, $noreg
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $lr, %bb.1, implicit-def dead $cpsr
-  ; CHECK: bb.3:
-  ; CHECK:   successors: %bb.3(0x7c000000), %bb.1(0x04000000)
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
-  ; CHECK: bb.4:
-  ; CHECK:   successors: %bb.5(0x80000000)
-  ; CHECK:   renamable $r0 = t2ADDrs killed renamable $r3, renamable $r1, 18, 14 /* CC::al */, $noreg, $noreg
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r1, %bb.6, implicit-def dead $cpsr
-  ; CHECK: bb.5:
-  ; CHECK:   successors: %bb.5(0x7c000000), %bb.6(0x04000000)
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
-  ; CHECK: bb.6:
-  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
-  bb.0:
-    successors: %bb.1(0x50000000), %bb.3(0x30000000)
-    liveins: $r0, $r1, $r2, $r3, $lr
-
-    tCMPi8 $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-    t2Bcc %bb.3, 11 /* CC::lt */, killed $cpsr
-    t2B %bb.1, 14 /* CC::al */, $noreg
-
-  bb.3:
-    successors: %bb.4(0x80000000)
-    liveins: $r1, $r3
-
-    tCMPi8 renamable $r1, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
-    t2IT 11, 8, implicit-def $itstate
-    frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
-    $lr = t2WhileLoopStartLR killed renamable $r1, %bb.1, implicit-def dead $cpsr
-    t2B %bb.4, 14 /* CC::al */, $noreg
-
-  bb.1:
-    successors: %bb.2(0x80000000)
-    liveins: $r0, $r1, $r2, $r3
-
-    $lr = tMOVr $r0, 14 /* CC::al */, $noreg
-    renamable $r0 = t2ADDrs killed renamable $r2, killed $r0, 18, 14 /* CC::al */, $noreg, $noreg
-    $lr = t2WhileLoopStartLR killed renamable $lr, %bb.3, implicit-def dead $cpsr
-
-  bb.2:
-    successors: %bb.2(0x7c000000), %bb.3(0x04000000)
-    liveins: $lr, $r0, $r1, $r3
-
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.2, implicit-def dead $cpsr
-    t2B %bb.3, 14 /* CC::al */, $noreg
-
-  bb.4:
-    successors: %bb.5(0x80000000)
-    liveins: $r1, $r3
-
-    renamable $r0 = t2ADDrs killed renamable $r3, renamable $r1, 18, 14 /* CC::al */, $noreg, $noreg
-    $lr = t2WhileLoopStartLR killed renamable $r1, %bb.6, implicit-def dead $cpsr
-
-  bb.5:
-    successors: %bb.5(0x7c000000), %bb.6(0x04000000)
-    liveins: $lr, $r0
-
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
-    t2B %bb.6, 14 /* CC::al */, $noreg
-
-  bb.6:
-    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
-...
----
-name:            backwards_branch_forwards_le
+name:            backwards_branch_backwards_wls
 body:             |
-  ; CHECK-LABEL: name: backwards_branch_forwards_le
+  ; CHECK-LABEL: name: backwards_branch_backwards_wls
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.2(0x80000000)
   ; CHECK:   tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
   ; CHECK:   t2IT 11, 8, implicit-def $itstate
   ; CHECK:   frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
   ; CHECK: bb.1:
-  ; CHECK:   successors: %bb.1(0x80000000)
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.1, implicit-def dead $cpsr
   ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
   ; CHECK: bb.2:
-  ; CHECK:   successors: %bb.3(0x80000000)
-  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+  ; CHECK:   successors: %bb.3(0x40000000), %bb.5(0x40000000)
+  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.3, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
   ; CHECK: bb.3:
-  ; CHECK:   successors: %bb.3(0x7c000000), %bb.1(0x04000000)
-  ; CHECK:   renamable $r0 = tLDRi renamable $r2, 0, 14 /* CC::al */, $noreg
-  ; CHECK:   tSTRi killed renamable $r0, renamable $r1, 0, 14 /* CC::al */, $noreg
-  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
+  ; CHECK:   successors: %bb.1(0x7c000000), %bb.4(0x04000000)
+  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.4, 14 /* CC::al */, $noreg
+  ; CHECK: bb.4:
+  ; CHECK:   successors: %bb.1(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
   ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
+  ; CHECK: bb.5:
+  ; CHECK:   successors: %bb.5(0x40000000), %bb.3(0x40000000)
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
   bb.0:
     successors: %bb.2(0x80000000)
     liveins: $r0, $r1, $r2, $lr
 
     tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
     t2IT 11, 8, implicit-def $itstate
     frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
 
   bb.1:
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.1, implicit-def dead $cpsr
     frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
 
   bb.2:
-    successors: %bb.3(0x80000000)
+    successors: %bb.3(0x80000000), %bb.5(0x80000000)
     liveins: $r0, $r1, $r2
 
-    $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+    $lr = t2WhileLoopStartLR killed renamable $r0, %bb.3, implicit-def dead $cpsr
+    t2B %bb.5, 14 /* CC::al */, $noreg
 
   bb.3:
-    successors: %bb.3(0x7c000000), %bb.1(0x04000000)
+    successors: %bb.1(0x7c000000), %bb.4(0x04000000)
     liveins: $lr, $r1, $r2
 
-    renamable $r0 = tLDRi renamable $r2, 0, 14 /* CC::al */, $noreg
-    tSTRi killed renamable $r0, renamable $r1, 0, 14 /* CC::al */, $noreg
-    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.3, implicit-def dead $cpsr
+    $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+    t2B %bb.4, 14 /* CC::al */, $noreg
+
+  bb.4:
+    successors: %bb.1, %bb.4
+    liveins: $lr, $r1, $r2
+
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
     t2B %bb.1, 14 /* CC::al */, $noreg
 
+  bb.5:
+    successors: %bb.5, %bb.3
+    liveins: $lr, $r1, $r2
+
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
+    t2B %bb.3, 14 /* CC::al */, $noreg
 ...
 ---
-name:            no_preheader
+name:            no_predecessor
 body:             |
-  ; CHECK-LABEL: name: no_preheader
+  ; CHECK-LABEL: name: no_predecessor
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.2(0x30000000), %bb.1(0x50000000)
   ; CHECK:   frame-setup tPUSH 14 /* CC::al */, $noreg, killed $r4, killed $r5, $r7, killed $lr, implicit-def $sp, implicit $sp
   ; CHECK:   frame-setup CFI_INSTRUCTION def_cfa_offset 16
   ; CHECK:   frame-setup CFI_INSTRUCTION offset $lr, -4
   ; CHECK:   frame-setup CFI_INSTRUCTION offset $r7, -8
   ; CHECK:   frame-setup CFI_INSTRUCTION offset $r5, -12
   ; CHECK:   frame-setup CFI_INSTRUCTION offset $r4, -16
body:             |
   ; CHECK:   renamable $r12 = t2ADDri killed renamable $r12, 1, 14 /* CC::al */, $noreg, $noreg
   ; CHECK:   t2Bcc %bb.9, 0 /* CC::eq */, killed $cpsr
   ; CHECK: bb.3:
   ; CHECK:   successors: %bb.4(0x50000000), %bb.2(0x30000000)
   ; CHECK:   liveins: $r0, $r1, $r3, $r8, $r9, $r12
   ; CHECK:   tCMPi8 renamable $r1, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
   ; CHECK:   t2Bcc %bb.2, 11 /* CC::lt */, killed $cpsr
   ; CHECK: bb.4:
-  ; CHECK:   successors: %bb.6(0x80000000)
+  ; CHECK:   successors: %bb.5(0x80000000)
   ; CHECK:   liveins: $r0, $r1, $r3, $r8, $r9, $r12
   ; CHECK:   renamable $r4, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
   ; CHECK:   $r10 = tMOVr $r12, 14 /* CC::al */, $noreg
   ; CHECK:   $r2 = tMOVr $r3, 14 /* CC::al */, $noreg
-  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
-  ; CHECK: bb.6:
-  ; CHECK:   successors: %bb.7(0x50000000), %bb.5(0x30000000)
+  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
+  ; CHECK: bb.5:
+  ; CHECK:   successors: %bb.7(0x50000000), %bb.6(0x30000000)
   ; CHECK:   liveins: $r0, $r1, $r2, $r3, $r4, $r8, $r9, $r10, $r12
-  ; CHECK:   renamable $lr = t2WhileLoopStartLR killed renamable $r9, %bb.5, implicit-def dead $cpsr
+  ; CHECK:   renamable $lr = t2WhileLoopStartLR killed renamable $r9, %bb.6, implicit-def dead $cpsr
   ; CHECK:   t2B %bb.7, 14 /* CC::al */, $noreg
-  ; CHECK: bb.5:
-  ; CHECK:   successors: %bb.2(0x04000000), %bb.6(0x7c000000)
+  ; CHECK: bb.6:
+  ; CHECK:   successors: %bb.2(0x04000000), %bb.5(0x7c000000)
   ; CHECK:   liveins: $r0, $r1, $r2, $r3, $r4, $r8, $r9, $r10, $r12
   ; CHECK:   renamable $r4, dead $cpsr = nuw nsw tADDi8 killed renamable $r4, 1, 14 /* CC::al */, $noreg
   ; CHECK:   renamable $r2, dead $cpsr = tADDi8 killed renamable $r2, 1, 14 /* CC::al */, $noreg
   ; CHECK:   tCMPr renamable $r4, renamable $r1, 14 /* CC::al */, $noreg, implicit-def $cpsr
   ; CHECK:   renamable $r10 = t2ADDri killed renamable $r10, 1, 14 /* CC::al */, $noreg, $noreg
   ; CHECK:   t2Bcc %bb.2, 0 /* CC::eq */, killed $cpsr
-  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
+  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
   ; CHECK: bb.7:
   ; CHECK:   successors: %bb.8(0x80000000)
   ; CHECK:   liveins: $r0, $r1, $r2, $r3, $r4, $r8, $r9, $r10, $r12
   ; CHECK:   $r5 = tMOVr $r10, 14 /* CC::al */, $noreg
   ; CHECK:   $r6 = tMOVr $r2, 14 /* CC::al */, $noreg
   ; CHECK:   t2B %bb.8, 14 /* CC::al */, $noreg
   ; CHECK: bb.8:
-  ; CHECK:   successors: %bb.8(0x7c000000), %bb.5(0x04000000)
+  ; CHECK:   successors: %bb.8(0x7c000000), %bb.6(0x04000000)
   ; CHECK:   liveins: $lr, $r0, $r1, $r2, $r3, $r4, $r5, $r6, $r8, $r9, $r10, $r12
   ; CHECK:   tSTRi killed $r0, $r1, 0, 14 /* CC::al */, $noreg
   ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.8, implicit-def dead $cpsr
-  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
+  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
   ; CHECK: bb.9:
   ; CHECK:   $sp = frame-destroy t2LDMIA_RET $sp, 14 /* CC::al */, $noreg, def $r4, def $r5, def $r6, def $r7, def $r8, def $r9, def $r10, def $pc
   bb.0:
     successors: %bb.1
     liveins: $r0, $r1, $r2, $r3, $r4, $r5, $r6, $r7, $r8, $r9, $r10, $lr
 
     $sp = frame-setup t2STMDB_UPD $sp, 14 /* CC::al */, $noreg, killed $r4, killed $r5, killed $r6, killed $r7, killed $r8, killed $r9, killed $r10, killed $lr
     frame-setup CFI_INSTRUCTION def_cfa_offset 32
bb.6:
     tSTRi killed $r0, $r1, 0, 14 /* CC::al */, $noreg
     renamable $lr = t2LoopEndDec killed renamable $lr, %bb.6, implicit-def dead $cpsr
     t2B %bb.7, 14 /* CC::al */, $noreg
 
   bb.9:
     $sp = frame-destroy t2LDMIA_RET $sp, 14 /* CC::al */, $noreg, def $r4, def $r5, def $r6, def $r7, def $r8, def $r9, def $r10, def $pc
 
 ...
+---
+name:            backwards_branch_forwards_wls
+body:             |
+  ; CHECK-LABEL: name: backwards_branch_forwards_wls
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.1(0x80000000)
+  ; CHECK:   tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
+  ; CHECK:   t2IT 11, 8, implicit-def $itstate
+  ; CHECK:   frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.2(0x40000000), %bb.5(0x40000000)
+  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.2, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.5, 14 /* CC::al */, $noreg
+  ; CHECK: bb.2:
+  ; CHECK:   successors: %bb.4(0x40000000), %bb.3(0x40000000)
+  ; CHECK:   $lr = t2WhileLoopStartLR killed renamable $r0, %bb.3, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.4, 14 /* CC::al */, $noreg
+  ; CHECK: bb.3:
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+  ; CHECK: bb.4:
+  ; CHECK:   successors: %bb.3(0x40000000), %bb.4(0x40000000)
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
+  ; CHECK: bb.5:
+  ; CHECK:   successors: %bb.5(0x40000000), %bb.2(0x40000000)
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.2, 14 /* CC::al */, $noreg
+  bb.0:
+    successors: %bb.2
+    liveins: $r0, $r1, $r2, $lr
+
+    tCMPi8 renamable $r0, 1, 14 /* CC::al */, $noreg, implicit-def $cpsr
+    t2IT 11, 8, implicit-def $itstate
+    frame-destroy tPOP_RET 11 /* CC::lt */, killed $cpsr, def $r7, def $pc, implicit killed $itstate
+
+  bb.2:
+    successors: %bb.3, %bb.5
+    liveins: $r0, $r1, $r2
+
+    $lr = t2WhileLoopStartLR killed renamable $r0, %bb.3, implicit-def dead $cpsr
+    t2B %bb.5, 14 /* CC::al */, $noreg
+
+  bb.1:
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc
+
+  bb.3:
+    successors: %bb.4, %bb.1
+    liveins: $lr, $r1, $r2
+
+    $lr = t2WhileLoopStartLR killed renamable $r0, %bb.1, implicit-def dead $cpsr
+    t2B %bb.4, 14 /* CC::al */, $noreg
+
+  bb.4:
+    successors: %bb.1, %bb.4
+    liveins: $lr, $r1, $r2
+
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
+    t2B %bb.1, 14 /* CC::al */, $noreg
+
+  bb.5:
+    successors: %bb.5, %bb.3
+    liveins: $lr, $r1, $r2
+
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
+    t2B %bb.3, 14 /* CC::al */, $noreg
+
+...
+---
+name:            multiple_predecessors
+body:             |
+  ; CHECK-LABEL: name: multiple_predecessors
+  ; CHECK: bb.0:
+  ; CHECK:   successors: %bb.3(0x55555555), %bb.2(0x2aaaaaab)
+  ; CHECK:   frame-setup tPUSH 14 /* CC::al */, $noreg, killed $r7, killed $lr, implicit-def $sp, implicit $sp
+  ; CHECK:   frame-setup CFI_INSTRUCTION def_cfa_offset 8
+  ; CHECK:   frame-setup CFI_INSTRUCTION offset $lr, -4
+  ; CHECK:   frame-setup CFI_INSTRUCTION offset $r7, -8
+  ; CHECK:   $sp = frame-setup tSUBspi $sp, 1, 14 /* CC::al */, $noreg
+  ; CHECK:   frame-setup CFI_INSTRUCTION def_cfa_offset 12
+  ; CHECK:   tCMPi8 killed renamable $r0, 0, 14 /* CC::al */, $noreg, implicit-def $cpsr
+  ; CHECK:   t2IT 0, 8, implicit-def $itstate
+  ; CHECK:   tCMPi8 killed renamable $r1, 8, 0 /* CC::eq */, killed $cpsr, implicit-def $cpsr, implicit killed $itstate
+  ; CHECK:   t2Bcc %bb.2, 0 /* CC::eq */, killed $cpsr
+  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
+  ; CHECK: bb.1:
+  ; CHECK:   successors: %bb.6(0x40000000), %bb.3(0x40000000)
+  ; CHECK:   renamable $lr = t2WhileLoopStartLR killed renamable $r2, %bb.3, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.6, 14 /* CC::al */, $noreg
+  ; CHECK: bb.2:
+  ; CHECK:   successors: %bb.4(0x40000000), %bb.3(0x40000000)
+  ; CHECK:   renamable $lr = t2WhileLoopStartLR renamable $r2, %bb.3, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.4, 14 /* CC::al */, $noreg
+  ; CHECK: bb.3:
+  ; CHECK:   $sp = frame-destroy tADDspi $sp, 1, 14 /* CC::al */, $noreg
+  ; CHECK:   frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc, implicit undef $r0
+  ; CHECK: bb.4:
+  ; CHECK:   successors: %bb.5(0x80000000)
+  ; CHECK:   renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
+  ; CHECK:   renamable $r1 = t2ADDri $sp, 2, 14 /* CC::al */, $noreg, $noreg
+  ; CHECK:   renamable $r0 = IMPLICIT_DEF
+  ; CHECK: bb.5:
+  ; CHECK:   successors: %bb.5(0x7c000000), %bb.1(0x04000000)
+  ; CHECK:   renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 1, 0, $noreg, undef renamable $q0
+  ; CHECK:   MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.5, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
+  ; CHECK: bb.6:
+  ; CHECK:   successors: %bb.7(0x80000000)
+  ; CHECK:   renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
+  ; CHECK: bb.7:
+  ; CHECK:   successors: %bb.7(0x7c000000), %bb.3(0x04000000)
+  ; CHECK:   renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 2, 0, $noreg, undef renamable $q0
+  ; CHECK:   MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
+  ; CHECK:   renamable $lr = t2LoopEndDec killed renamable $lr, %bb.7, implicit-def dead $cpsr
+  ; CHECK:   t2B %bb.3, 14 /* CC::al */, $noreg
+  bb.0:
+    successors: %bb.7(0x80000000), %bb.1(0x40000000)
+    liveins: $r0, $r1, $r2, $r7, $lr
+
+    frame-setup tPUSH 14 /* CC::al */, $noreg, killed $r7, killed $lr, implicit-def $sp, implicit $sp
+    frame-setup CFI_INSTRUCTION def_cfa_offset 8
+    frame-setup CFI_INSTRUCTION offset $lr, -4
+    frame-setup CFI_INSTRUCTION offset $r7, -8
+    $sp = frame-setup tSUBspi $sp, 1, 14 /* CC::al */, $noreg
+    frame-setup CFI_INSTRUCTION def_cfa_offset 12
+    tCMPi8 killed renamable $r0, 0, 14 /* CC::al */, $noreg, implicit-def $cpsr
+    t2IT 0, 8, implicit-def $itstate
+    tCMPi8 killed renamable $r1, 8, 0 /* CC::eq */, killed $cpsr, implicit-def $cpsr, implicit killed $itstate
+    t2Bcc %bb.1, 0 /* CC::eq */, killed $cpsr
+
+  bb.7:
+    $sp = frame-destroy tADDspi $sp, 1, 14 /* CC::al */, $noreg
+    frame-destroy tPOP_RET 14 /* CC::al */, $noreg, def $r7, def $pc, implicit undef $r0
+
+  bb.1:
+    successors: %bb.3(0x40000000), %bb.7(0x40000000)
+    liveins: $r2
+
+    renamable $lr = t2WhileLoopStartLR renamable $r2, %bb.7, implicit-def dead $cpsr
+    t2B %bb.3, 14 /* CC::al */, $noreg
+
+  bb.3:
+    successors: %bb.4(0x80000000)
+    liveins: $lr, $r2
+
+    renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
+    renamable $r1 = t2ADDri $sp, 2, 14 /* CC::al */, $noreg, $noreg
+    renamable $r0 = IMPLICIT_DEF
+
+  bb.4:
+    successors: %bb.4(0x7c000000), %bb.2(0x04000000)
+    liveins: $lr, $r0, $r1, $r2, $r3
+
+    renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 1, 0, $noreg, undef renamable $q0
+    MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.4, implicit-def dead $cpsr
+    t2B %bb.2, 14 /* CC::al */, $noreg
+
+  bb.2:
+    successors: %bb.5(0x40000000), %bb.7(0x40000000)
+    liveins: $r0, $r1, $r2
+
+    renamable $lr = t2WhileLoopStartLR killed renamable $r2, %bb.7, implicit-def dead $cpsr
+    t2B %bb.5, 14 /* CC::al */, $noreg
+
+  bb.5:
+    successors: %bb.6(0x80000000)
+    liveins: $lr, $r0, $r1
+
+    renamable $r3, dead $cpsr = tMOVi8 0, 14 /* CC::al */, $noreg
+
+  bb.6:
+    successors: %bb.6(0x7c000000), %bb.7(0x04000000)
+    liveins: $lr, $r0, $r1, $r3
+
+    renamable $q0, renamable $r0 = MVE_VIWDUPu16 killed renamable $r0, renamable $r3, 2, 0, $noreg, undef renamable $q0
+    MVE_VSTRH16_rq undef renamable $q0, renamable $r1, killed renamable $q0, 0, $noreg
+    renamable $lr = t2LoopEndDec killed renamable $lr, %bb.6, implicit-def dead $cpsr
+    t2B %bb.7, 14 /* CC::al */, $noreg
+
+...

llvm/test/CodeGen/Thumb2/mve-float16regloops.ll

 ; CHECK-NEXT:    str r1, [sp] @ 4-byte Spill
 ; CHECK-NEXT:    subs r1, r7, #2
 ; CHECK-NEXT:    rsbs r7, r4, #0
 ; CHECK-NEXT:    str r7, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:    add.w r7, r3, #16
 ; CHECK-NEXT:    str r4, [sp, #12] @ 4-byte Spill
 ; CHECK-NEXT:    str r7, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    str r0, [sp, #16] @ 4-byte Spill
-; CHECK-NEXT:    b .LBB16_3
-; CHECK-NEXT:  .LBB16_3: @ %while.body
+; CHECK-NEXT:    b .LBB16_5
+; CHECK-NEXT:  .LBB16_3: @ %for.end
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
+; CHECK-NEXT:    ldr r0, [sp, #16] @ 4-byte Reload
+; CHECK-NEXT:    wls lr, r0, .LBB16_4
+; CHECK-NEXT:    b .LBB16_9
+; CHECK-NEXT:  .LBB16_4: @ %while.end
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
+; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
+; CHECK-NEXT:    subs.w r12, r12, #1
+; CHECK-NEXT:    vstrb.8 q0, [r2], #8
+; CHECK-NEXT:    add.w r0, r5, r0, lsl #1
+; CHECK-NEXT:    add.w r5, r0, #8
+; CHECK-NEXT:    beq.w .LBB16_12
+; CHECK-NEXT:  .LBB16_5: @ %while.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
-; CHECK-NEXT:    @ Child Loop BB16_5 Depth 2
+; CHECK-NEXT:    @ Child Loop BB16_7 Depth 2
 ; CHECK-NEXT:    @ Child Loop BB16_10 Depth 2
 ; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
 ; CHECK-NEXT:    ldrh.w lr, [r3, #14]
 ; CHECK-NEXT:    vldrw.u32 q0, [r0], #8
 ; CHECK-NEXT:    ldrh.w r8, [r3, #12]
 ; CHECK-NEXT:    ldrh r7, [r3, #10]
 ; CHECK-NEXT:    ldrh r4, [r3, #8]
 ; CHECK-NEXT:    ldrh r6, [r3, #6]

 ; CHECK-NEXT:    vfma.f16 q0, q1, r7
 ; CHECK-NEXT:    vldrw.u32 q1, [r5, #12]
 ; CHECK-NEXT:    adds r5, #16
 ; CHECK-NEXT:    vfma.f16 q0, q1, r8
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
 ; CHECK-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
 ; CHECK-NEXT:    vfma.f16 q0, q1, lr
 ; CHECK-NEXT:    cmp r0, #16
-; CHECK-NEXT:    blo .LBB16_6
-; CHECK-NEXT:  @ %bb.4: @ %for.body.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    blo .LBB16_8
+; CHECK-NEXT:  @ %bb.6: @ %for.body.preheader
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r0, [sp] @ 4-byte Reload
 ; CHECK-NEXT:    dls lr, r0
 ; CHECK-NEXT:    ldr r6, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:  .LBB16_5: @ %for.body
-; CHECK-NEXT:    @ Parent Loop BB16_3 Depth=1
+; CHECK-NEXT:  .LBB16_7: @ %for.body
+; CHECK-NEXT:    @ Parent Loop BB16_5 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    ldrh r0, [r6], #16
 ; CHECK-NEXT:    vldrw.u32 q1, [r5]
 ; CHECK-NEXT:    adds r4, r5, #2
 ; CHECK-NEXT:    vfma.f16 q0, q1, r0
 ; CHECK-NEXT:    vldrw.u32 q1, [r4]
 ; CHECK-NEXT:    ldrh r0, [r6, #-14]
 ; CHECK-NEXT:    adds r4, r5, #6

 ; CHECK-NEXT:    vfma.f16 q0, q1, r0
 ; CHECK-NEXT:    ldrh r0, [r6, #-4]
 ; CHECK-NEXT:    vldrw.u32 q1, [r5, #12]
 ; CHECK-NEXT:    vfma.f16 q0, q1, r0
 ; CHECK-NEXT:    add.w r0, r5, #14
 ; CHECK-NEXT:    vldrw.u32 q1, [r0]
 ; CHECK-NEXT:    adds r5, #16
 ; CHECK-NEXT:    vfma.f16 q0, q1, r4
-; CHECK-NEXT:    le lr, .LBB16_5
-; CHECK-NEXT:    b .LBB16_7
-; CHECK-NEXT:  .LBB16_6: @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    le lr, .LBB16_7
+; CHECK-NEXT:    b .LBB16_3
+; CHECK-NEXT:  .LBB16_8: @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r6, [sp, #4] @ 4-byte Reload
-; CHECK-NEXT:  .LBB16_7: @ %for.end
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
-; CHECK-NEXT:    ldr r0, [sp, #16] @ 4-byte Reload
-; CHECK-NEXT:    wls lr, r0, .LBB16_8
-; CHECK-NEXT:    b .LBB16_9
-; CHECK-NEXT:  .LBB16_8: @ %while.end
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
-; CHECK-NEXT:    ldr r0, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:    subs.w r12, r12, #1
-; CHECK-NEXT:    vstrb.8 q0, [r2], #8
-; CHECK-NEXT:    add.w r0, r5, r0, lsl #1
-; CHECK-NEXT:    add.w r5, r0, #8
-; CHECK-NEXT:    beq .LBB16_12
 ; CHECK-NEXT:    b .LBB16_3
 ; CHECK-NEXT:  .LBB16_9: @ %while.body76.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    mov r0, r5
 ; CHECK-NEXT:  .LBB16_10: @ %while.body76
-; CHECK-NEXT:    @ Parent Loop BB16_3 Depth=1
+; CHECK-NEXT:    @ Parent Loop BB16_5 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    ldrh r4, [r6], #2
 ; CHECK-NEXT:    vldrh.u16 q1, [r0], #2
 ; CHECK-NEXT:    vfma.f16 q0, q1, r4
 ; CHECK-NEXT:    le lr, .LBB16_10
 ; CHECK-NEXT:  @ %bb.11: @ %while.end.loopexit
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r0, [sp, #16] @ 4-byte Reload
 ; CHECK-NEXT:    add.w r5, r5, r0, lsl #1
-; CHECK-NEXT:    b .LBB16_8
+; CHECK-NEXT:    b .LBB16_4
 ; CHECK-NEXT:  .LBB16_12: @ %if.end
 ; CHECK-NEXT:    add sp, #24
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 entry:
   %pState1 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 1
   %0 = load half*, half** %pState1, align 4
   %pCoeffs2 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 2
   %1 = load half*, half** %pCoeffs2, align 4

llvm/test/CodeGen/Thumb2/mve-float32regloops.ll

 ; CHECK-NEXT:    sub.w r9, r3, #4
 ; CHECK-NEXT:    rsbs r3, r6, #0
 ; CHECK-NEXT:    str r3, [sp, #12] @ 4-byte Spill
 ; CHECK-NEXT:    add.w r3, r10, #32
 ; CHECK-NEXT:    str r5, [sp, #4] @ 4-byte Spill
 ; CHECK-NEXT:    str r6, [sp, #16] @ 4-byte Spill
 ; CHECK-NEXT:    str r3, [sp, #8] @ 4-byte Spill
 ; CHECK-NEXT:    str r0, [sp, #20] @ 4-byte Spill
-; CHECK-NEXT:    b .LBB16_3
-; CHECK-NEXT:  .LBB16_3: @ %while.body
+; CHECK-NEXT:    b .LBB16_5
+; CHECK-NEXT:  .LBB16_3: @ %for.end
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
+; CHECK-NEXT:    ldr r1, [sp, #28] @ 4-byte Reload
+; CHECK-NEXT:    ldrd r0, r9, [sp, #20] @ 8-byte Folded Reload
+; CHECK-NEXT:    wls lr, r0, .LBB16_4
+; CHECK-NEXT:    b .LBB16_9
+; CHECK-NEXT:  .LBB16_4: @ %while.end
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
+; CHECK-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
+; CHECK-NEXT:    subs.w r12, r12, #1
+; CHECK-NEXT:    vstrb.8 q0, [r2], #16
+; CHECK-NEXT:    add.w r0, r4, r0, lsl #2
+; CHECK-NEXT:    add.w r4, r0, #16
+; CHECK-NEXT:    beq .LBB16_12
+; CHECK-NEXT:  .LBB16_5: @ %while.body
 ; CHECK-NEXT:    @ =>This Loop Header: Depth=1
-; CHECK-NEXT:    @ Child Loop BB16_5 Depth 2
+; CHECK-NEXT:    @ Child Loop BB16_7 Depth 2
 ; CHECK-NEXT:    @ Child Loop BB16_10 Depth 2
 ; CHECK-NEXT:    add.w lr, r10, #8
 ; CHECK-NEXT:    vldrw.u32 q0, [r1], #16
 ; CHECK-NEXT:    ldrd r3, r7, [r10]
 ; CHECK-NEXT:    ldm.w lr, {r0, r5, r6, lr}
 ; CHECK-NEXT:    ldrd r11, r8, [r10, #24]
 ; CHECK-NEXT:    vstrb.8 q0, [r9], #16
 ; CHECK-NEXT:    vldrw.u32 q0, [r4], #32

 ; CHECK-NEXT:    vldrw.u32 q3, [r4, #-8]
 ; CHECK-NEXT:    vfma.f32 q0, q5, r6
 ; CHECK-NEXT:    ldr r0, [sp, #16] @ 4-byte Reload
 ; CHECK-NEXT:    vfma.f32 q0, q2, lr
 ; CHECK-NEXT:    vldrw.u32 q1, [r4, #-4]
 ; CHECK-NEXT:    vfma.f32 q0, q3, r11
 ; CHECK-NEXT:    cmp r0, #16
 ; CHECK-NEXT:    vfma.f32 q0, q1, r8
-; CHECK-NEXT:    blo .LBB16_6
-; CHECK-NEXT:  @ %bb.4: @ %for.body.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    blo .LBB16_8
+; CHECK-NEXT:  @ %bb.6: @ %for.body.preheader
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r0, [sp, #4] @ 4-byte Reload
 ; CHECK-NEXT:    dls lr, r0
 ; CHECK-NEXT:    ldr r7, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:  .LBB16_5: @ %for.body
-; CHECK-NEXT:    @ Parent Loop BB16_3 Depth=1
+; CHECK-NEXT:  .LBB16_7: @ %for.body
+; CHECK-NEXT:    @ Parent Loop BB16_5 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    ldm.w r7, {r0, r3, r5, r6, r8, r11}
 ; CHECK-NEXT:    vldrw.u32 q1, [r4], #32
 ; CHECK-NEXT:    vldrw.u32 q6, [r4, #-24]
 ; CHECK-NEXT:    vldrw.u32 q4, [r4, #-20]
 ; CHECK-NEXT:    vfma.f32 q0, q1, r0
 ; CHECK-NEXT:    vldrw.u32 q1, [r4, #-28]
 ; CHECK-NEXT:    vldrw.u32 q5, [r4, #-16]
 ; CHECK-NEXT:    vldrw.u32 q2, [r4, #-12]
 ; CHECK-NEXT:    vfma.f32 q0, q1, r3
 ; CHECK-NEXT:    ldrd r9, r1, [r7, #24]
 ; CHECK-NEXT:    vfma.f32 q0, q6, r5
 ; CHECK-NEXT:    vldrw.u32 q3, [r4, #-8]
 ; CHECK-NEXT:    vfma.f32 q0, q4, r6
 ; CHECK-NEXT:    vldrw.u32 q1, [r4, #-4]
 ; CHECK-NEXT:    vfma.f32 q0, q5, r8
 ; CHECK-NEXT:    adds r7, #32
 ; CHECK-NEXT:    vfma.f32 q0, q2, r11
 ; CHECK-NEXT:    vfma.f32 q0, q3, r9
 ; CHECK-NEXT:    vfma.f32 q0, q1, r1
-; CHECK-NEXT:    le lr, .LBB16_5
-; CHECK-NEXT:    b .LBB16_7
-; CHECK-NEXT:  .LBB16_6: @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    le lr, .LBB16_7
+; CHECK-NEXT:    b .LBB16_3
+; CHECK-NEXT:  .LBB16_8: @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r7, [sp, #8] @ 4-byte Reload
-; CHECK-NEXT:  .LBB16_7: @ %for.end
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
-; CHECK-NEXT:    ldr r1, [sp, #28] @ 4-byte Reload
-; CHECK-NEXT:    ldrd r0, r9, [sp, #20] @ 8-byte Folded Reload
-; CHECK-NEXT:    wls lr, r0, .LBB16_8
-; CHECK-NEXT:    b .LBB16_9
-; CHECK-NEXT:  .LBB16_8: @ %while.end
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
-; CHECK-NEXT:    ldr r0, [sp, #12] @ 4-byte Reload
-; CHECK-NEXT:    subs.w r12, r12, #1
-; CHECK-NEXT:    vstrb.8 q0, [r2], #16
-; CHECK-NEXT:    add.w r0, r4, r0, lsl #2
-; CHECK-NEXT:    add.w r4, r0, #16
-; CHECK-NEXT:    beq .LBB16_12
 ; CHECK-NEXT:    b .LBB16_3
 ; CHECK-NEXT:  .LBB16_9: @ %while.body76.preheader
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    mov r3, r4
 ; CHECK-NEXT:  .LBB16_10: @ %while.body76
-; CHECK-NEXT:    @ Parent Loop BB16_3 Depth=1
+; CHECK-NEXT:    @ Parent Loop BB16_5 Depth=1
 ; CHECK-NEXT:    @ => This Inner Loop Header: Depth=2
 ; CHECK-NEXT:    ldr r0, [r7], #4
 ; CHECK-NEXT:    vldrw.u32 q1, [r3], #4
 ; CHECK-NEXT:    vfma.f32 q0, q1, r0
 ; CHECK-NEXT:    le lr, .LBB16_10
 ; CHECK-NEXT:  @ %bb.11: @ %while.end.loopexit
-; CHECK-NEXT:    @ in Loop: Header=BB16_3 Depth=1
+; CHECK-NEXT:    @ in Loop: Header=BB16_5 Depth=1
 ; CHECK-NEXT:    ldr r0, [sp, #20] @ 4-byte Reload
 ; CHECK-NEXT:    add.w r4, r4, r0, lsl #2
-; CHECK-NEXT:    b .LBB16_8
+; CHECK-NEXT:    b .LBB16_4
 ; CHECK-NEXT:  .LBB16_12: @ %if.end
 ; CHECK-NEXT:    add sp, #32
 ; CHECK-NEXT:    vpop {d8, d9, d10, d11, d12, d13}
 ; CHECK-NEXT:    add sp, #4
 ; CHECK-NEXT:    pop.w {r4, r5, r6, r7, r8, r9, r10, r11, pc}
 entry:
   %pState1 = getelementptr inbounds %struct.arm_fir_instance_f32, %struct.arm_fir_instance_f32* %S, i32 0, i32 1
   %0 = load float*, float** %pState1, align 4