[MBP] Rotate should bring more fallthrough

[Carrot]

Summary

Current implementation of rotateLoop does rotate without carefully compare the number of exit fallthrough and top fallthrough, so it may create more taken branches, as demonstrated by the test case no_rotate.

This patch makes sure the new exit fallthrough is larger than top fallthrough before rotation.

[davidxl]

I like the direction this patch is going, but I think it can do better.

The block layout algorithm here tries to maximize total FallThrough frequency. In the context of loop rotation, this can be formulated as such -- comparing the total fall through frequency before and after. If the post rotation fall through freq is lower, do not do it. In fact, the same idea can be used to find the optimal rotation (optimal bb rotated to the bottom).

In the no_rotate.ll, assuming the Freq(header->middle) = F0, Freq(middle->latch1) = F1, Freq(latch1->latch2) = F2, Freq(latch2->header) = F3, Freq(header->end) = F4, we have F2 == F3, F0 > F1 > F2

Before rotation, the total fallthrough freq = F0 + F1 + F2
After rotation, the total fall through = F1 + F2 + F3

Before - After = F 0 - F2 > 0, clearly not good to rotate.

[Carrot]

Precisely compute the fallthrough frequency is very helpful in layout of loop header and exit BB. It is included in my next patch.

Current implementation of findBestLoopExit finds the exit edge with most frequency, and rotateLoop exposes that edge as fallthrough. This patch fixed a problem in a very common case

• The loop header has only one out-of-loop predecessor
• The bottom of the loop chain doesn't exit the loop

In this case, the exit edge can't have larger frequency than the fallthrough to header, so the loop chain should not be rotated.

[davidxl]

Loop rotation not only affects fall through placement of incoming and exit edges, but internal fall through as well -- it converts the original backedge into a new fallthrough while converts an original internal fallthrough into the backedge. In other words, the overall cost/benefit needs to consider all changes. Doing pattern matching like this can lead to wrong decisions.

I assume this case can be handled by your next complete patch, so we should try to go that direction directly.

[davidxl]

Does this patch handle the case in D62079? If yes, you can move the tests added in that patch here.

lib/CodeGen/MachineBlockPlacement.cpp

void fillWorkLists(const MachineBasicBlock *MBB,
                      const BlockFilterSet *BlockFilter);
 
   void buildChain(const MachineBasicBlock *BB, BlockChain &Chain,
                   BlockFilterSet *BlockFilter = nullptr);
   bool canMoveBottomBlockToTop(const MachineBasicBlock *BottomBlock,
                                const MachineBasicBlock *OldTop);
   bool hasViableTopFallthrough(const MachineBasicBlock *Top,
                                const BlockFilterSet &LoopBlockSet);
+  BlockFrequency TopFallThroughFreq(const MachineBasicBlock *Top,
+                                    const BlockFilterSet &LoopBlockSet);
   MachineBasicBlock *findBestLoopTop(
       const MachineLoop &L, const BlockFilterSet &LoopBlockSet);
   MachineBasicBlock *findBestLoopExit(
-      const MachineLoop &L, const BlockFilterSet &LoopBlockSet);
+      const MachineLoop &L, const BlockFilterSet &LoopBlockSet,
+      BlockFrequency &ExitFreq);
   BlockFilterSet collectLoopBlockSet(const MachineLoop &L);
   void buildLoopChains(const MachineLoop &L);
   void rotateLoop(
       BlockChain &LoopChain, const MachineBasicBlock *ExitingBB,
-      const BlockFilterSet &LoopBlockSet);
+      BlockFrequency ExitFreq, const BlockFilterSet &LoopBlockSet);
   void rotateLoopWithProfile(
       BlockChain &LoopChain, const MachineLoop &L,
       const BlockFilterSet &LoopBlockSet);
   void buildCFGChains();
   void optimizeBranches();
   void alignBlocks();
   /// Returns true if a block should be tail-duplicated to increase fallthrough
   /// opportunities.

 
 /// Find the best loop exiting block for layout.
 ///
 /// This routine implements the logic to analyze the loop looking for the best
 /// block to layout at the top of the loop. Typically this is done to maximize
 /// fallthrough opportunities.
 MachineBasicBlock *
 MachineBlockPlacement::findBestLoopExit(const MachineLoop &L,
-                                        const BlockFilterSet &LoopBlockSet) {
+                                        const BlockFilterSet &LoopBlockSet,
+                                        BlockFrequency &ExitFreq) {
   // We don't want to layout the loop linearly in all cases. If the loop header
   // is just a normal basic block in the loop, we want to look for what block
   // within the loop is the best one to layout at the top. However, if the loop
   // header has be pre-merged into a chain due to predecessors not having
   // analyzable branches, *and* the predecessor it is merged with is *not* part
   // of the loop, rotating the header into the middle of the loop will create
   // a non-contiguous range of blocks which is Very Bad. So start with the
   // header and only rotate if safe.
MachineBlockPlacement::findBestLoopExit(const MachineLoop &L,
   // one of them as the exiting block we are rotating toward, disable loop
   // rotation altogether.
   if (!BlocksExitingToOuterLoop.empty() &&
       !BlocksExitingToOuterLoop.count(ExitingBB))
     return nullptr;
 
   LLVM_DEBUG(dbgs() << "  Best exiting block: " << getBlockName(ExitingBB)
                     << "\n");
+  ExitFreq = BestExitEdgeFreq;
   return ExitingBB;
 }
 
 /// Check if there is a fallthrough to loop header Top.
 ///
 ///   1. Look for a Pred that can be layout before Top.
 ///   2. Check if Top is the most possible successor of Pred.
 bool
if (!LoopBlockSet.count(Pred) &&
       }
       if (TopOK)
         return true;
     }
   }
   return false;
 }
 
+// Find out the possible fall through frequence to the top of a loop.
+BlockFrequency
+MachineBlockPlacement::TopFallThroughFreq(
+    const MachineBasicBlock *Top,
+    const BlockFilterSet &LoopBlockSet) {
+  BlockFrequency MaxFreq = 0;
+  for (MachineBasicBlock *Pred : Top->predecessors()) {
+    BlockChain *PredChain = BlockToChain[Pred];
+    if (!LoopBlockSet.count(Pred) &&
+        (!PredChain || Pred == *std::prev(PredChain->end()))) {
+      // Found a Pred block can be placed before Top.
+      // Check if Top is the best successor of Pred.
+      auto TopProb = MBPI->getEdgeProbability(Pred, Top);
+      bool TopOK = true;
+      for (MachineBasicBlock *Succ : Pred->successors()) {
+        auto SuccProb = MBPI->getEdgeProbability(Pred, Succ);
+        BlockChain *SuccChain = BlockToChain[Succ];
+        // Check if Succ can be placed after Pred.
+        // Succ should not be in any chain, or it is the head of some chain.
+        if (!LoopBlockSet.count(Succ) && (SuccProb > TopProb) &&
+            (!SuccChain || Succ == *SuccChain->begin())) {
+          TopOK = false;
+          break;
+        }
+      }
+      if (TopOK) {
+        BlockFrequency EdgeFreq = MBFI->getBlockFreq(Pred) *
+                                  MBPI->getEdgeProbability(Pred, Top);
+        if (EdgeFreq > MaxFreq)
+          MaxFreq = EdgeFreq;
+      }
+    }
+  }
+  return MaxFreq;
+}
+
 /// Attempt to rotate an exiting block to the bottom of the loop.
 ///
 /// Once we have built a chain, try to rotate it to line up the hot exit block
 /// with fallthrough out of the loop if doing so doesn't introduce unnecessary
 /// branches. For example, if the loop has fallthrough into its header and out
 /// of its bottom already, don't rotate it.
 void MachineBlockPlacement::rotateLoop(BlockChain &LoopChain,
                                        const MachineBasicBlock *ExitingBB,
+                                       BlockFrequency ExitFreq,
                                        const BlockFilterSet &LoopBlockSet) {
   if (!ExitingBB)
     return;
 
   MachineBasicBlock *Top = *LoopChain.begin();
   MachineBasicBlock *Bottom = *std::prev(LoopChain.end());
 
   // If ExitingBB is already the last one in a chain then nothing to do.
   if (Bottom == ExitingBB)
     return;
 
   bool ViableTopFallthrough = hasViableTopFallthrough(Top, LoopBlockSet);
 
   // If the header has viable fallthrough, check whether the current loop
   // bottom is a viable exiting block. If so, bail out as rotating will
   // introduce an unnecessary branch.
   if (ViableTopFallthrough) {
     for (MachineBasicBlock *Succ : Bottom->successors()) {
       BlockChain *SuccChain = BlockToChain[Succ];
       if (!LoopBlockSet.count(Succ) &&
           (!SuccChain || Succ == *SuccChain->begin()))
         return;
     }
+
+    // Rotate will destroy the top fallthrough, we need to ensure the new exit
+    // frequency is larger than top fallthrough.
+    BlockFrequency FallThrough2Top = TopFallThroughFreq(Top, LoopBlockSet);
+    if (FallThrough2Top >= ExitFreq)
+      return;
   }
 
   BlockChain::iterator ExitIt = llvm::find(LoopChain, ExitingBB);
   if (ExitIt == LoopChain.end())
     return;
 
   // Rotating a loop exit to the bottom when there is a fallthrough to top
   // trades the entry fallthrough for an exit fallthrough.
void MachineBlockPlacement::buildLoopChains(const MachineLoop &L) {
 
   // If we selected just the header for the loop top, look for a potentially
   // profitable exit block in the event that rotating the loop can eliminate
   // branches by placing an exit edge at the bottom.
   //
   // Loops are processed innermost to uttermost, make sure we clear
   // PreferredLoopExit before processing a new loop.
   PreferredLoopExit = nullptr;
+  BlockFrequency ExitFreq;
   if (!RotateLoopWithProfile && LoopTop == L.getHeader())
-    PreferredLoopExit = findBestLoopExit(L, LoopBlockSet);
+    PreferredLoopExit = findBestLoopExit(L, LoopBlockSet, ExitFreq);
 
   BlockChain &LoopChain = *BlockToChain[LoopTop];
 
   // FIXME: This is a really lame way of walking the chains in the loop: we
   // walk the blocks, and use a set to prevent visiting a particular chain
   // twice.
   SmallPtrSet<BlockChain *, 4> UpdatedPreds;
   assert(LoopChain.UnscheduledPredecessors == 0 &&
          "LoopChain should not have unscheduled predecessors.");
   UpdatedPreds.insert(&LoopChain);
 
   for (const MachineBasicBlock *LoopBB : LoopBlockSet)
     fillWorkLists(LoopBB, UpdatedPreds, &LoopBlockSet);
 
   buildChain(LoopTop, LoopChain, &LoopBlockSet);
 
   if (RotateLoopWithProfile)
     rotateLoopWithProfile(LoopChain, L, LoopBlockSet);
   else
-    rotateLoop(LoopChain, PreferredLoopExit, LoopBlockSet);
+    rotateLoop(LoopChain, PreferredLoopExit, ExitFreq, LoopBlockSet);
 
   LLVM_DEBUG({
     // Crash at the end so we get all of the debugging output first.
     bool BadLoop = false;
     if (LoopChain.UnscheduledPredecessors) {
       BadLoop = true;
       dbgs() << "Loop chain contains a block without its preds placed!\n"
              << "  Loop header:  " << getBlockName(*L.block_begin()) << "\n"

test/CodeGen/AArch64/cmpxchg-idioms.ll

 ; CHECK: [[FAILED]]:
 ; CHECK-NOT: cmp {{w[0-9]+}}, {{w[0-9]+}}
 
 ; verify the preheader is simplified by simplifycfg.
 ; CHECK: [[PH]]:
 ; CHECK: mov w22, #2
 ; CHECK-NOT: mov w22, #4
 ; CHECK-NOT: cmn w22, #4
-; CHECK: b [[LOOP2:LBB[0-9]+_[0-9]+]]
+; CHECK: [[LOOP2:LBB[0-9]+_[0-9]+]]: ; %for.cond
 ; CHECK-NOT: b.ne [[LOOP2]]
 ; CHECK-NOT: b {{LBB[0-9]+_[0-9]+}}
 ; CHECK: bl _foo
 entry:
   %pair = cmpxchg i32* %c, i32 %a, i32 %b seq_cst seq_cst
   %success = extractvalue { i32, i1 } %pair, 1
   br label %for.cond
 

test/CodeGen/AMDGPU/si-annotate-cf.ll

 }
 
 declare float @llvm.fabs.f32(float) nounwind readnone
 
 ; This broke the old AMDIL cfg structurizer
 ; FUNC-LABEL: {{^}}loop_land_info_assert:
 ; SI:      v_cmp_lt_i32_e64 [[CMP4:s\[[0-9:]+\]]], s{{[0-9]+}}, 4{{$}}
 ; SI:      s_and_b64        [[CMP4M:s\[[0-9]+:[0-9]+\]]], exec, [[CMP4]]
-; SI:      s_branch         [[INFLOOP:BB[0-9]+_[0-9]+]]
+
+; SI: [[WHILELOOP:BB[0-9]+_[0-9]+]]: ; %while.cond
+; SI:      s_cbranch_vccz [[FOR_COND_PH:BB[0-9]+_[0-9]+]]
 
 ; SI:      [[CONVEX_EXIT:BB[0-9_]+]]
 ; SI:      s_mov_b64        vcc,
 ; SI-NEXT: s_cbranch_vccnz  [[ENDPGM:BB[0-9]+_[0-9]+]]
-; SI:      s_cbranch_vccnz  [[INFLOOP]]
+
+; SI:      s_cbranch_vccnz  [[WHILELOOP]]
 
 ; SI: ; %if.else
 ; SI:      buffer_store_dword
 
-; SI:      [[INFLOOP]]:
-; SI:      s_cbranch_vccnz [[CONVEX_EXIT]]
-
-; SI: ; %for.cond.preheader
+; SI: [[FOR_COND_PH]]: ; %for.cond.preheader
 ; SI:      s_cbranch_vccz [[ENDPGM]]
 
 ; SI:      [[ENDPGM]]:
 ; SI-NEXT: s_endpgm
 define amdgpu_kernel void @loop_land_info_assert(i32 %c0, i32 %c1, i32 %c2, i32 %c3, i32 %x, i32 %y, i1 %arg) nounwind {
 entry:
   %cmp = icmp sgt i32 %c0, 0
   br label %while.cond.outer

test/CodeGen/PowerPC/licm-remat.ll

 
 define linkonce_odr void @ZN6snappyDecompressor_(%"class.snappy::SnappyDecompressor"* %this, %"class.snappy::SnappyIOVecWriter"* %writer) {
 ; CHECK-LABEL: ZN6snappyDecompressor_:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK:       addis 3, 2, _ZN6snappy8internalL8wordmaskE@toc@ha
 ; CHECK-DAG:   addi 25, 3, _ZN6snappy8internalL8wordmaskE@toc@l
 ; CHECK-DAG:   addis 5, 2, _ZN6snappy8internalL10char_tableE@toc@ha
 ; CHECK-DAG:   addi 24, 5, _ZN6snappy8internalL10char_tableE@toc@l
-; CHECK:       b .[[LABEL1:[A-Z0-9_]+]]
-; CHECK:       .[[LABEL1]]: # %for.cond
+; CHECK:       .LBB0_1: # %for.cond
 ; CHECK-NOT:   addis {{[0-9]+}}, 2, _ZN6snappy8internalL8wordmaskE@toc@ha
 ; CHECK-NOT:   addis {{[0-9]+}}, 2, _ZN6snappy8internalL10char_tableE@toc@ha
 ; CHECK:       bctrl
 entry:
   %ip_limit_ = getelementptr inbounds %"class.snappy::SnappyDecompressor", %"class.snappy::SnappyDecompressor"* %this, i64 0, i32 2
   %0 = bitcast i8** %ip_limit_ to i64*
   %curr_iov_index_.i = getelementptr inbounds %"class.snappy::SnappyIOVecWriter", %"class.snappy::SnappyIOVecWriter"* %writer, i64 0, i32 2
   %curr_iov_written_.i = getelementptr inbounds %"class.snappy::SnappyIOVecWriter", %"class.snappy::SnappyIOVecWriter"* %writer, i64 0, i32 3

test/CodeGen/Thumb/consthoist-physical-addr.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc %s -o - -asm-verbose=false | FileCheck %s
 
 target datalayout = "e-m:e-p:32:32-Fi8-i64:64-v128:64:128-a:0:32-n32-S64"
 target triple = "thumbv6m-arm-none-eabi"
 
 define i32 @C(i32 %x, i32* nocapture %y) #0 {
 ; CHECK-LABEL: C:
 ; CHECK:         .save {r4, r5, r7, lr}
 ; CHECK-NEXT:    push {r4, r5, r7, lr}
 ; CHECK-NEXT:    movs r2, #0
 ; CHECK-NEXT:    ldr r3, .LCPI0_0
-; CHECK-NEXT:    b .LBB0_4
 ; CHECK-NEXT:  .LBB0_1:
+; CHECK-NEXT:    cmp r2, #128
+; CHECK-NEXT:    beq .LBB0_5
 ; CHECK-NEXT:    movs r4, #0
 ; CHECK-NEXT:    str r4, [r3, #8]
 ; CHECK-NEXT:    lsls r4, r2, #2
 ; CHECK-NEXT:    adds r5, r4, r0
 ; CHECK-NEXT:    str r5, [r3]
 ; CHECK-NEXT:    movs r5, #1
 ; CHECK-NEXT:    str r5, [r3, #12]
 ; CHECK-NEXT:    isb sy
-; CHECK-NEXT:  .LBB0_2:
+; CHECK-NEXT:  .LBB0_3:
 ; CHECK-NEXT:    ldr r5, [r3, #12]
 ; CHECK-NEXT:    cmp r5, #0
-; CHECK-NEXT:    bne .LBB0_2
+; CHECK-NEXT:    bne .LBB0_3
 ; CHECK-NEXT:    ldr r5, [r3, #4]
 ; CHECK-NEXT:    str r5, [r1, r4]
 ; CHECK-NEXT:    adds r2, r2, #1
-; CHECK-NEXT:  .LBB0_4:
-; CHECK-NEXT:    cmp r2, #128
-; CHECK-NEXT:    bne .LBB0_1
+; CHECK-NEXT:    b .LBB0_1
+; CHECK-NEXT:  .LBB0_5:
 ; CHECK-NEXT:    movs r0, #0
 ; CHECK-NEXT:    pop {r4, r5, r7, pc}
 ; CHECK-NEXT:    .p2align 2
 ; CHECK-NEXT:  .LCPI0_0:
 ; CHECK-NEXT:    .long 805355524
 entry:
   br label %for.cond
 

test/CodeGen/X86/block-placement.ll

 exit:
   ret i32 %b
 }
 
 define i32 @test_loop_cold_blocks(i32 %i, i32* %a) {
 ; Check that we sink cold loop blocks after the hot loop body.
 ; CHECK-LABEL: test_loop_cold_blocks:
 ; CHECK: %entry
-; CHECK-NOT: .p2align
-; CHECK: %unlikely1
-; CHECK-NOT: .p2align
-; CHECK: %unlikely2
 ; CHECK: .p2align
 ; CHECK: %body1
 ; CHECK: %body2
 ; CHECK: %body3
+; CHECK-NOT: .p2align
+; CHECK: %unlikely1
+; CHECK-NOT: .p2align
+; CHECK: %unlikely2
 ; CHECK: %exit
 
 entry:
   br label %body1
 
 body1:
   %iv = phi i32 [ 0, %entry ], [ %next, %body3 ]
   %base = phi i32 [ 0, %entry ], [ %sum, %body3 ]

 }
 
 define i32 @not_rotate_if_extra_branch_regression(i32 %count, i32 %init) {
 ; This is a regression test against patch avoid loop rotation if
 ; it introduce an extra btanch.
 ; CHECK-LABEL: not_rotate_if_extra_branch_regression
 ; CHECK: %.entry
 ; CHECK: %.first_backedge
-; CHECK: %.slow
 ; CHECK: %.second_header
+; CHECK: %.slow
 .entry:
   %sum.0 = shl nsw i32 %count, 1
   br label %.first_header
 
 .first_header:
   %i = phi i32 [ %i.1, %.first_backedge ], [ 0, %.entry ]
   %is_bo1 = icmp sgt i32 %i, 9000000
   br i1 %is_bo1, label %.bailout, label %.first_backedge, !prof !14

test/CodeGen/X86/code_placement_cold_loop_blocks.ll

end:
   ret void
 }
 
 define void @nested_loop_0(i1 %flag) !prof !1 {
 ; Test if a block that is cold in the inner loop but not cold in the outer loop
 ; will merged to the outer loop chain.
 ;
 ; CHECK-LABEL: nested_loop_0:
+; CHECK: callq b
 ; CHECK: callq c
 ; CHECK: callq d
 ; CHECK: callq e
-; CHECK: callq b
 ; CHECK: callq f
 
 entry:
   br label %header
 
 header:
   call void @b()
   %call4 = call zeroext i1 @a()

test/CodeGen/X86/code_placement_no_header_change.ll

 ; RUN: llc -mtriple=i686-linux < %s | FileCheck %s
 
 
 define i32 @bar(i32 %count) {
 ; Test checks that basic block backedge2 is not moved before header,
 ; because it can't reduce taken branches.
 ; Later backedge1 and backedge2 is rotated before loop header.
 ; CHECK-LABEL: bar
 ; CHECK: %.entry
+; CHECK: %.header
 ; CHECK: %.backedge1
 ; CHECK: %.backedge2
-; CHECK: %.header
 ; CHECK: %.exit
 .entry:
   %c = shl nsw i32 %count, 2
   br label %.header
 
 .header:
   %val1 = call i32 @foo()
   %cond1 = icmp sgt i32 %val1, 1

test/CodeGen/X86/loop-blocks.ll

block101:
   br label %loop
 
 block102:
   call void @bar102()
   br label %loop
 }
 
 ; CHECK-LABEL: check_minsize:
-;      CHECK:   jmp   .LBB4_1
 ; CHECK-NOT:   align
-; CHECK-NEXT: .LBB4_2:
-; CHECK-NEXT:   callq loop_latch
-; CHECK-NEXT: .LBB4_1:
+; CHECK:      .LBB4_1:
 ; CHECK-NEXT:   callq loop_header
+; CHECK:        callq loop_latch
+; CHECK:      .LBB4_3:
+; CHECK:        callq exit
 
 
 define void @check_minsize() minsize nounwind {
 entry:
   br label %loop
 
 loop:
   call void @loop_header()

test/CodeGen/X86/no_rotate.ll

+; RUN: llc -mtriple=i686-linux < %s | FileCheck %s
+
+; Don't rotate the loop if the number of fall through to exit is not larger
+; than the number of fall through to header.
+
+define void @no_rotate() {
+; CHECK-LABEL: no_rotate
+; CHECK: %entry
+; CHECK: %header
+; CHECK: %middle
+; CHECK: %latch1
+; CHECK: %latch2
+; CHECK: %end
+entry:
+  br label %header
+
+header:
+  %val1 = call i1 @foo()
+  br i1 %val1, label %middle, label %end
+
+middle:
+  %val2 = call i1 @foo()
+  br i1 %val2, label %latch1, label %end
+
+latch1:
+  %val3 = call i1 @foo()
+  br i1 %val3, label %latch2, label %header
+
+latch2:
+  %val4 = call i1 @foo()
+  br label %header
+
+end:
+  ret void
+}
+
+define void @do_rotate() {
+; CHECK-LABEL: do_rotate
+; CHECK: %entry
+; CHECK: %then
+; CHECK: %else
+; CHECK: %latch1
+; CHECK: %latch2
+; CHECK: %header
+; CHECK: %end
+entry:
+  %val0 = call i1 @foo()
+  br i1 %val0, label %then, label %else
+
+then:
+  call void @a()
+  br label %header
+
+else:
+  call void @b()
+  br label %header
+
+header:
+  %val1 = call i1 @foo()
+  br i1 %val1, label %latch1, label %end
+
+latch1:
+  %val3 = call i1 @foo()
+  br i1 %val3, label %latch2, label %header
+
+latch2:
+  %val4 = call i1 @foo()
+  br label %header
+
+end:
+  ret void
+}
+
+declare i1 @foo()
+declare void @a()
+declare void @b()

test/DebugInfo/X86/PR37234.ll

 ;            }
 ;            return aa;
 ;    }
 
 
 ; CHECK-LABEL: # %bb.{{.*}}:
 ; CHECK:        #DEBUG_VALUE: main:aa <- 0
 ; CHECK:  #DEBUG_VALUE: main:aa <- $[[REG:[0-9a-z]+]]
-; CHECK:  jmp .LBB0_1
-; CHECK: .LBB0_2:
+; CHECK: .LBB0_1:
+; CHECK:        #DEBUG_VALUE: main:aa <- $[[REG]]
+; CHECK:        je      .LBB0_4
+; CHECK: # %bb.{{.*}}:
 ; CHECK:        #DEBUG_VALUE: main:aa <- $[[REG]]
 ; CHECK:        jne     .LBB0_1
 ; CHECK: # %bb.{{.*}}:
 ; CHECK:        #DEBUG_VALUE: main:aa <- $[[REG]]
 ; CHECK:        incl    %[[REG]]
 ; CHECK:        #DEBUG_VALUE: main:aa <- $[[REG]]
-; CHECK: .LBB0_1:
-; CHECK:  #DEBUG_VALUE: main:aa <- $[[REG]]
-; CHECK:        jne     .LBB0_2
-; CHECK: # %bb.{{.*}}:
+; CHECK:        jmp     .LBB0_1
+; CHECK: .LBB0_4:
 ; CHECK:  #DEBUG_VALUE: main:aa <- $[[REG]]
 ; CHECK:  retq
 
 source_filename = "PR37234.cpp"
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 @.str = private unnamed_addr constant [4 x i8] c"aaa\00", align 1