[Thumb] Fix assembler error 'cannot honor width suffix pop {lr}'

[tyomitch]

Summary

Hi Artyom and Quentin,

It looks like we are still generating .s files with invalid pop operand with Thumb1.

This is a problem when not using integrated assembler, because it
produces errors like 'cannot honor width suffix -- `pop {r7,lr}''.

See the attached test case to reproduce the issue.

The function emitPopSpecialFixUp replaces instructions
like pop {r7, lr} with:
pop {r7}
pop {r0}
mov lr, r0
since lt cannot be encoded with Thumb1.

But it only deletes lr operand from pop instruction when
the pop is the last instruction in the BB.
We should handle cases where the pop instruction
might be followed by other BB terminator instructions.

I tried to fix the issue, let me know what else you suggest.

[t.p.northover]

Bother, I considered that but thought I'd investigated all the possible terminators when reviewing that patch and convinced myself they couldn't occur after a tPOP. Which one was it?

The patch looks fine to me, though.

Tim.

[tyomitch]

In D15707#315285, @t.p.northover wrote:

Bother, I considered that but thought I'd investigated all the possible terminators when reviewing that patch and convinced myself they couldn't occur after a tPOP. Which one was it?

It's followed by an unconditional branch to a tBX_RET, and my patch only handled falling-through to a tBX_RET :-/

(Note that the same code compiles correctly with -enable-shrink-wrap=false, so it's all @qcolombet's oversight in the first place! :-P)

[tyomitch]

test/CodeGen/Thumb/thumb-shrink-wrapping.ll includes a few more cases when pop {lr} can still be generated; all of them are wrong, and need addressing.

[tyomitch]

Updating the patch to handle the aforementioned additional cases of pop {lr} in test/CodeGen/Thumb/thumb-shrink-wrapping.ll

[apazos]

Thanks Tim, the test case looks fine. I will rerun the updated patch with my other workloads and let you know if I find any other corner case.

[tyomitch]

Updating the patch to simplify ARMLoadStoreOpt::CombineMovBx

[apazos]

Artyom,

Several tests are still failing due to failure to remove lr operand from the pop instruction.

It will take me some time to reduce each test cases. But I looked at some of the failures and it seems we need to handle conditional branches tBcc to the return as well.

See example IR:

tPOP pred:14, pred:%noreg, %R4<def>, %LR<def>, %SP<imp-def>, %SP<imp-use>
tPOP pred:14, pred:%noreg, %R3<def>, %SP<imp-def>, %SP<imp-use>
%LR<def> = tMOVr %R3<kill>, pred:14, pred:%noreg
tBcc <BB#4>, pred:0, pred:%CPSR<kill>
  Successors according to CFG: BB#4(0x40000000 / 0x80000000 = 50.00%) BB#3(0x40000000 / 0x80000000 = 50.00%)

BB#3: derived from LLVM BB %if.end6

  Live Ins: %R1 %R2 %LR %R4
  Predecessors according to CFG: BB#2
%R0<def>, %CPSR<def,dead> = tMOVi8 0, pred:14, pred:%noreg
tSTRi %R0, %R1, 12, pred:14, pred:%noreg; mem:ST4[%done](tbaa=!14)
tSTRi %R1<kill>, %R2<kill>, 8, pred:14, pred:%noreg; mem:ST4[%head7](tbaa=!11)
  Successors according to CFG: BB#4(?%)

BB#4: derived from LLVM BB %cleanup

  Live Ins: %R0 %LR %R4
  Predecessors according to CFG: BB#2 BB#3 BB#0 BB#1
tBX_RET pred:14, pred:%noreg, %R0<imp-use>

[apazos]

Here is a simplified test case for the tBcc case, we can add it to thumb-pop.ll or move them all to thumb-shrinkwrapping.ll.

; RUN: llc < %s -mtriple=thumbv5e-none-linux-gnueabi-eabi -verify-machineinstrs -o - | FileCheck %s
%struct.a = type { i8*, i32, i32, i8*, i32, i32, i8*, %struct.b*, i8* (i8*, i32, i32)*, void (i8*, i8*)*,
i8*, i32, i32, i32 }
%struct.b = type opaque
%struct.c = type { i32, i32, i32, i32, i8*, i32, i32, i8*, i32, i8*, i32, i32, i32 }
%struct.e = type { i32, i32, i32, i32, i32, i32, i32, i32, %struct.c*, i32, i32, i32, i32, i8*, i32, i32,
i32, i32, i32, %struct.f*, %struct.f*, i32, i32, i32, i32, i32, i32, %struct.f*, [320 x i16], [288 x i16
], [1444 x %struct.f], i32, i32, i32 }
%struct.f = type { i8, i8, i16 }

define i32 @test2(%struct.a* %v, %struct.c* %head) {
; CHECK-LABEL: test2:
; CHECK-NOT: pop {[[POP_REG:r[0-7]]], lr}
entry:

%cmp = icmp eq %struct.a* %v, null
br i1 %cmp, label %cleanup, label %if.end

if.end: ; preds = %entry

%state1 = getelementptr inbounds %struct.a, %struct.a* %v, i32 0, i32 7
%0 = bitcast %struct.b** %state1 to %struct.e**
%1 = load %struct.e*, %struct.e** %0, align 4
%wrap = getelementptr inbounds %struct.e, %struct.e* %1, i32 0, i32 2
%2 = load i32, i32* %wrap, align 4
%and = and i32 %2, 2
%cmp2 = icmp eq i32 %and, 0
br i1 %cmp2, label %cleanup, label %if.end4

if.end4: ; preds = %if.end

%head5 = getelementptr inbounds %struct.e, %struct.e* %1, i32 0, i32 8
store %struct.c* %head, %struct.c** %head5, align 4
br label %cleanup

cleanup: ; preds = %if.end, %entry, %if.end4

%retval.0 = phi i32 [ 0, %if.end4 ], [ -2, %entry ], [ -2, %if.end ]
ret i32 %retval.0

}

[apazos]

Artyom,

These additional changes solve my problem. See if you agree with them.

• a/lib/Target/ARM/Thumb1FrameLowering.cpp

+++ b/lib/Target/ARM/Thumb1FrameLowering.cpp
@@ -433,18 +433,23 @@ bool Thumb1FrameLowering::emitPopSpecialFixUp(MachineBasicBlock &MBB,

auto MBBI = MBB.getFirstTerminator();
bool CanRestoreDirectly = STI.hasV5TOps() && !ArgRegsSaveSize;
if (CanRestoreDirectly) {

• if (MBBI != MBB.end() && MBBI->getOpcode() != ARM::tB)

+ if (MBBI != MBB.end() && MBBI->getOpcode() != ARM::tB &&
+ MBBI->getOpcode() != ARM::tBcc)

  CanRestoreDirectly = (MBBI->getOpcode() == ARM::tBX_RET ||
                        MBBI->getOpcode() == ARM::tPOP_RET);
else {
  auto MBBI_prev = MBBI;
  MBBI_prev--;
  assert(MBBI_prev->getOpcode() == ARM::tPOP);

• assert(MBB.succ_size() == 1);
• if ((*MBB.succ_begin())->begin()->getOpcode() == ARM::tBX_RET)
• MBBI = MBBI_prev; // Replace the final tPOP with a tPOP_RET.
• else
• CanRestoreDirectly = false;

+ if (MBB.succ_size() > 1)
+ CanRestoreDirectly = false;
+ else {
+ assert(MBB.succ_size() == 1);
+ if ((*MBB.succ_begin())->begin()->getOpcode() == ARM::tBX_RET)
+ MBBI = MBBI_prev; // Replace the final tPOP with a tPOP_RET.
+ else
+ CanRestoreDirectly = false;
+ }

  }
}

@@ -531,7 +536,8 @@ bool Thumb1FrameLowering::emitPopSpecialFixUp(MachineBasicBlock &MBB,

                     .addReg(PopReg, RegState::Kill));
}

• if (MBBI == MBB.end() || MBBI->getOpcode() == ARM::tB) {

+ if (MBBI == MBB.end() || MBBI->getOpcode() == ARM::tB ||
+ MBBI->getOpcode() == ARM::tBcc) {

auto Pop = MBBI;
Pop--;
assert(Pop->getOpcode() == ARM::tPOP);

[apazos]

Still some remaining issues... it does not seem you can assume that the instruction preceding the branch is the POP instruction. Looks like we just need to search back in the block for a POP that has LR as operand and then remove the operand (like I had in my original patch).

See the piece of code that is triggering a new assertion in Thumb1FrameLowering.cpp:543:

tPOP pred:14, pred:%noreg, %R4<def>, %R5<def>, %R7<def>, %LR<def>, %SP<imp-def>, %SP<imp-use>
 %R12<def> = tMOVr %R0<kill>, pred:14, pred:%noreg
 tBcc <BB#4>, pred:1, pred:%CPSR<kill>

[tyomitch]

Ana, thank you for these new test cases.

I think it's a better idea to avoid generating pop {lr} in Thumb1 epilogues than to go hunting around for them later; so I'm updating the patch to do just that.

[apazos]

Hi Artyom,

So far my tests are passing. I think you can proceed with merging the latest patch.

Thanks for the help!

lib/Target/ARM/ARMLoadStoreOptimizer.cpp

private:
     MachineInstr *MergeOpsUpdate(const MergeCandidate &Cand);
     bool FixInvalidRegPairOp(MachineBasicBlock &MBB,
                              MachineBasicBlock::iterator &MBBI);
     bool MergeBaseUpdateLoadStore(MachineInstr *MI);
     bool MergeBaseUpdateLSMultiple(MachineInstr *MI);
     bool MergeBaseUpdateLSDouble(MachineInstr &MI) const;
     bool LoadStoreMultipleOpti(MachineBasicBlock &MBB);
     bool MergeReturnIntoLDM(MachineBasicBlock &MBB);
+    bool CombineMovBx(MachineBasicBlock &MBB);
   };
   char ARMLoadStoreOpt::ID = 0;
 }
 
 INITIALIZE_PASS(ARMLoadStoreOpt, "arm-load-store-opt", ARM_LOAD_STORE_OPT_NAME, false, false)
 
 static bool definesCPSR(const MachineInstr *MI) {
   for (const auto &MO : MI->operands()) {
if (Opcode == ARM::LDMIA_UPD || Opcode == ARM::LDMDA_UPD ||
       PrevMI->copyImplicitOps(*MBB.getParent(), &*MBBI);
       MBB.erase(MBBI);
       return true;
     }
   }
   return false;
 }
 
+bool ARMLoadStoreOpt::CombineMovBx(MachineBasicBlock &MBB) {
+  MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
+  if (MBBI == MBB.begin() || MBBI == MBB.end() ||
+      MBBI->getOpcode() != ARM::tBX_RET)
+    return false;
+
+  MachineBasicBlock::iterator Prev = MBBI;
+  --Prev;
+  if (Prev->getOpcode() != ARM::tMOVr || !Prev->definesRegister(ARM::LR))
+    return false;
+
+  for (auto Use : Prev->uses())
+    if (Use.isKill()) {
+      AddDefaultPred(BuildMI(MBB, MBBI, MBBI->getDebugLoc(), TII->get(ARM::tBX))
+          .addReg(Use.getReg(), RegState::Kill))
+          .copyImplicitOps(&*MBBI);
+      MBB.erase(MBBI);
+      MBB.erase(Prev);
+      return true;
+    }
+
+  llvm_unreachable("tMOVr doesn't kill a reg before tBX_RET?");
+}
+
 bool ARMLoadStoreOpt::runOnMachineFunction(MachineFunction &Fn) {
   MF = &Fn;
   STI = &static_cast<const ARMSubtarget &>(Fn.getSubtarget());
   TL = STI->getTargetLowering();
   AFI = Fn.getInfo<ARMFunctionInfo>();
   TII = STI->getInstrInfo();
   TRI = STI->getRegisterInfo();
 
   RegClassInfoValid = false;
   isThumb2 = AFI->isThumb2Function();
   isThumb1 = AFI->isThumbFunction() && !isThumb2;
 
   bool Modified = false;
   for (MachineFunction::iterator MFI = Fn.begin(), E = Fn.end(); MFI != E;
        ++MFI) {
     MachineBasicBlock &MBB = *MFI;
     Modified |= LoadStoreMultipleOpti(MBB);
     if (STI->hasV5TOps())
       Modified |= MergeReturnIntoLDM(MBB);
+    if (isThumb1)
+      Modified |= CombineMovBx(MBB);
   }
 
   Allocator.DestroyAll();
   return Modified;
 }
 
 namespace llvm {
 void initializeARMPreAllocLoadStoreOptPass(PassRegistry &);

lib/Target/ARM/Thumb1FrameLowering.cpp

bool Thumb1FrameLowering::emitPopSpecialFixUp(MachineBasicBlock &MBB,
   // LR in the PC.
   // This is only possible with v5T ops (v4T can't change the Thumb bit via
   // a POP PC instruction), and only if we do not need to emit any SP update.
   // Otherwise, we need a temporary register to pop the value
   // and copy that value into LR.
   auto MBBI = MBB.getFirstTerminator();
   bool CanRestoreDirectly = STI.hasV5TOps() && !ArgRegsSaveSize;
   if (CanRestoreDirectly) {
-    if (MBBI != MBB.end())
+    if (MBBI != MBB.end() && MBBI->getOpcode() != ARM::tB)
       CanRestoreDirectly = (MBBI->getOpcode() == ARM::tBX_RET ||
                             MBBI->getOpcode() == ARM::tPOP_RET);
     else {
-      assert(MBB.back().getOpcode() == ARM::tPOP);
+      auto MBBI_prev = MBBI;
+      MBBI_prev--;
+      assert(MBBI_prev->getOpcode() == ARM::tPOP);
       assert(MBB.succ_size() == 1);
       if ((*MBB.succ_begin())->begin()->getOpcode() == ARM::tBX_RET)
-        MBBI--; // Replace the final tPOP with a tPOP_RET.
+        MBBI = MBBI_prev; // Replace the final tPOP with a tPOP_RET.
       else
         CanRestoreDirectly = false;
     }
   }
 
   if (CanRestoreDirectly) {
     if (!DoIt || MBBI->getOpcode() == ARM::tPOP_RET)
       return true;
bool Thumb1FrameLowering::emitPopSpecialFixUp(MachineBasicBlock &MBB,
   if (TemporaryReg) {
     assert(!PopReg && "Unnecessary MOV is about to be inserted");
     PopReg = PopFriendly.find_first();
     AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tMOVr))
                        .addReg(TemporaryReg, RegState::Define)
                        .addReg(PopReg, RegState::Kill));
   }
 
-  bool AddBx = false;
-  if (MBBI == MBB.end()) {
-    MachineInstr& Pop = MBB.back();
-    assert(Pop.getOpcode() == ARM::tPOP);
-    Pop.RemoveOperand(Pop.findRegisterDefOperandIdx(ARM::LR));
+  if (MBBI == MBB.end() || MBBI->getOpcode() == ARM::tB) {
+    auto Pop = MBBI;
+    Pop--;
+    assert(Pop->getOpcode() == ARM::tPOP);
+    Pop->RemoveOperand(Pop->findRegisterDefOperandIdx(ARM::LR));
   } else if (MBBI->getOpcode() == ARM::tPOP_RET) {
     // We couldn't use the direct restoration above, so
     // perform the opposite conversion: tPOP_RET to tPOP.
     MachineInstrBuilder MIB =
         AddDefaultPred(
             BuildMI(MBB, MBBI, MBBI->getDebugLoc(), TII.get(ARM::tPOP)));
-    unsigned Popped = 0;
+    bool Popped = false;
     for (auto MO: MBBI->operands())
       if (MO.isReg() && (MO.isImplicit() || MO.isDef()) &&
           MO.getReg() != ARM::PC) {
         MIB.addOperand(MO);
         if (!MO.isImplicit())
-          Popped++;
+          Popped = true;
       }
     // Is there anything left to pop?
     if (!Popped)
       MBB.erase(MIB.getInstr());
     // Erase the old instruction.
     MBB.erase(MBBI);
-    MBBI = MBB.end();
-    AddBx = true;
+    MBBI = AddDefaultPred(BuildMI(MBB, MBB.end(), dl, TII.get(ARM::tBX_RET)));
   }
 
   assert(PopReg && "Do not know how to get LR");
   AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tPOP)))
       .addReg(PopReg, RegState::Define);
 
   emitSPUpdate(MBB, MBBI, TII, dl, *RegInfo, ArgRegsSaveSize);
 
-  if (!TemporaryReg && MBBI != MBB.end() && MBBI->getOpcode() == ARM::tBX_RET) {
-    MachineInstrBuilder MIB = BuildMI(MBB, MBBI, dl, TII.get(ARM::tBX))
-                                  .addReg(PopReg, RegState::Kill);
-    AddDefaultPred(MIB);
-    MIB.copyImplicitOps(&*MBBI);
-    // erase the old tBX_RET instruction
-    MBB.erase(MBBI);
-    return true;
-  }
-
-  if (AddBx && !TemporaryReg) {
-    AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tBX))
-                       .addReg(PopReg, RegState::Kill));
-  } else {
-    AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tMOVr))
-                       .addReg(ARM::LR, RegState::Define)
-                       .addReg(PopReg, RegState::Kill));
-  }
-  if (TemporaryReg) {
+
+  if (TemporaryReg) 
     AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tMOVr))
                        .addReg(PopReg, RegState::Define)
                        .addReg(TemporaryReg, RegState::Kill));
-    if (AddBx)
-      AddDefaultPred(BuildMI(MBB, MBBI, dl, TII.get(ARM::tBX_RET)));
-  }
 
   return true;
 }
 
 bool Thumb1FrameLowering::
 spillCalleeSavedRegisters(MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MI,
                           const std::vector<CalleeSavedInfo> &CSI,

test/CodeGen/Thumb/thumb-pop.ll

+; RUN: llc < %s -mtriple=thumbv5e-none-linux-gnueabi-eabi -verify-machineinstrs -o - | FileCheck %s
+
+define i32 @test(i32 %value) {
+; CHECK-LABEL: test:
+; CHECK-NOT: pop {[[POP_REG:r[0-7]]], lr}
+entry:
+  %cmp = icmp slt i32 %value, 50
+  br i1 %cmp, label %if.then, label %if.else
+
+if.then:                                          ; preds = %entry
+  %div = sdiv i32 5000, %value
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+  %mul = shl nsw i32 %value, 1
+  %sub = sub nsw i32 200, %mul
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %if.then
+  %value.addr.0 = phi i32 [ %div, %if.then ], [ %sub, %if.else ]
+  ret i32 %value.addr.0
+}
+

test/CodeGen/Thumb/thumb-shrink-wrapping.ll

 ; Shift second argument by one and store into returned register.
 ; CHECK: lsls r0, r1, #1
 ; DISABLE-V5T-NEXT: pop {r4, pc}
 ; DISABLE-V4T-NEXT: [[END_LABEL]]: @ %if.end
 ; DISABLE-V4T-NEXT: pop {r4}
 ; DISABLE-V4T-NEXT: pop {r1}
 ; DISABLE-V4T-NEXT: bx r1
 ;
+; ENABLE-V5T-NEXT: {{LBB[0-9_]+}}: @ %if.end
 ; ENABLE-NEXT: bx lr
 define i32 @freqSaveAndRestoreOutsideLoop(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0
   br i1 %tobool, label %if.else, label %for.preheader
 
 for.preheader:
   tail call void asm "nop", ""()

 ; CHECK: adds [[SUM]], [[TMP]], [[SUM]]
 ; CHECK-NEXT: subs [[IV]], [[IV]], #1
 ; CHECK-NEXT: cmp [[IV]], #0
 ; CHECK-NEXT: bne [[LOOP]]
 ;
 ; Next BB.
 ; SUM << 3.
 ; CHECK: lsls [[SUM]], [[SUM]], #3
-; ENABLE-NEXT: pop {r4, lr}
+; ENABLE-V5T-NEXT: pop {r4, pc}
+; ENABLE-V4T-NEXT: pop {r4}
+; ENABLE-V4T-NEXT: pop {r1}
+; ENABLE-V4T-NEXT: bx r1
 ;
 ; Duplicated epilogue.
 ; DISABLE-V5T: pop {r4, pc}
 ; DISABLE-V4T: b [[END_LABEL:LBB[0-9_]+]]
 ;
 ; CHECK: [[ELSE_LABEL]]: @ %if.else
 ; Shift second argument by one and store into returned register.
 ; CHECK: lsls r0, r1, #1
 ; DISABLE-V5T-NEXT: pop {r4, pc}
 ; DISABLE-V4T-NEXT: [[END_LABEL]]: @ %if.end
 ; DISABLE-V4T-NEXT: pop {r4}
 ; DISABLE-V4T-NEXT: pop {r1}
 ; DISABLE-V4T-NEXT: bx r1
 ;
+; ENABLE-V5T-NEXT: {{LBB[0-9_]+}}: @ %if.end
 ; ENABLE-NEXT: bx lr
 define i32 @loopInfoSaveOutsideLoop(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0
   br i1 %tobool, label %if.else, label %for.preheader
 
 for.preheader:
   tail call void asm "nop", ""()

 ; CHECK: adds [[SUM]], [[TMP]], [[SUM]]
 ; CHECK-NEXT: subs [[IV]], [[IV]], #1
 ; CHECK-NEXT: cmp [[IV]], #0
 ; CHECK-NEXT: bne [[LOOP]]
 ;
 ; Next BB.
 ; SUM << 3.
 ; CHECK: lsls [[SUM]], [[SUM]], #3
-; ENABLE: pop {r4, lr}
+; ENABLE-V5T-NEXT: pop {r4, pc}
+; ENABLE-V4T-NEXT: pop {r4}
+; ENABLE-V4T-NEXT: pop {r1}
+; ENABLE-V4T-NEXT: bx r1
 ;
 ; Duplicated epilogue.
 ; DISABLE-V5T: pop {r4, pc}
 ; DISABLE-V4T: b [[END_LABEL:LBB[0-9_]+]]
 ;
 ; CHECK: [[ELSE_LABEL]]: @ %if.else
 ; Shift second argument by one and store into returned register.
 ; CHECK: lsls r0, r1, #1
 ; DISABLE-V5T-NEXT: pop {r4, pc}
 ; DISABLE-V4T-NEXT: [[END_LABEL]]: @ %if.end
 ; DISABLE-V4T-NEXT: pop {r4}
 ; DISABLE-V4T-NEXT: pop {r1}
 ; DISABLE-V4T-NEXT: bx r1
 ;
+; ENABLE-V5T-NEXT: {{LBB[0-9_]+}}: @ %if.end
 ; ENABLE-NEXT: bx lr
 define i32 @loopInfoRestoreOutsideLoop(i32 %cond, i32 %N) #0 {
 entry:
   %tobool = icmp eq i32 %cond, 0
   br i1 %tobool, label %if.else, label %if.then
 
 if.then:                                          ; preds = %entry
   tail call void asm "nop", "~{r4}"()

 ; CHECK: [[LOOP:LBB[0-9_]+]]: @ %for.body
 ; CHECK: movs r4, #1
 ; CHECK: subs [[IV]], [[IV]], #1
 ; CHECK-NEXT: cmp [[IV]], #0
 ; CHECK-NEXT: bne [[LOOP]]
 ;
 ; Next BB.
 ; CHECK: movs r0, #0
-; ENABLE-NEXT: pop {r4, lr}
+; ENABLE-V5T-NEXT: pop {r4, pc}
+; ENABLE-V4T-NEXT: pop {r4}
+; ENABLE-V4T-NEXT: pop {r1}
+; ENABLE-V4T-NEXT: bx r1
 ;
 ; Duplicated epilogue.
 ; DISABLE-V5T-NEXT: pop {r4, pc}
 ; DISABLE-V4T-NEXT: b [[END_LABEL:LBB[0-9_]+]]
 ;
 ; CHECK: [[ELSE_LABEL]]: @ %if.else
 ; Shift second argument by one and store into returned register.
 ; CHECK: lsls r0, r1, #1
 ; DISABLE-V5T-NEXT: pop {r4, pc}
 ; DISABLE-V4T-NEXT: [[END_LABEL]]: @ %if.end
 ; DISABLE-V4T-NEXT: pop {r4}
 ; DISABLE-V4T-NEXT: pop {r1}
 ; DISABLE-V4T-NEXT: bx r1
 ;
+; ENABLE-V5T-NEXT: {{LBB[0-9_]+}}: @ %if.end
 ; ENABLE-NEXT: bx lr
 define i32 @inlineAsm(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0
   br i1 %tobool, label %if.else, label %for.preheader
 
 for.preheader:
   tail call void asm "nop", ""()

 ; CHECK: push {r1}
 ; CHECK-NEXT: pop {r2}
 ; CHECK: push {r1}
 ; CHECK-NEXT: pop {r3}
 ; CHECK-NEXT: bl
 ; CHECK-NEXT: lsls r0, r0, #3
 ;
 ; ENABLE-NEXT: add sp, #16
-; ENABLE-NEXT: pop {[[TMP]], lr}
+; ENABLE-V5T-NEXT: pop {[[TMP]], pc}
+; ENABLE-V4T-NEXT: pop {[[TMP]]}
+; ENABLE-V4T-NEXT: pop {r1}
+; ENABLE-V4T-NEXT: bx r1
 ;
 ; Duplicated epilogue.
 ; DISABLE-V5T-NEXT: add sp, #16
 ; DISABLE-V5T-NEXT: pop {[[TMP]], pc}
 ; DISABLE-V4T-NEXT: b [[END_LABEL:LBB[0-9_]+]]
 ;
 ; CHECK: [[ELSE_LABEL]]: @ %if.else
 ; Shift second argument by one and store into returned register.
 ; CHECK: lsls r0, r1, #1
 ;
 ; Epilogue code.
+; ENABLE-V5T-NEXT: {{LBB[0-9_]+}}: @ %if.end
 ; ENABLE-NEXT: bx lr
 ;
 ; DISABLE-V4T-NEXT: [[END_LABEL]]: @ %if.end
 ; DISABLE-NEXT: add sp, #16
 ; DISABLE-V5T-NEXT: pop {[[TMP]], pc}
 ; DISABLE-V4T-NEXT: pop {[[TMP]]}
 ; DISABLE-V4T-NEXT: pop {r1}
 ; DISABLE-V4T-NEXT: bx r1