GlobalISel/AArch64: don't optimize away redundant branches at -O0

[aprantl]

Summary

This patch prevents GlobalISel from optimizing out redundant branch instructions when compiling without optimizations.

The motivating example is code like the following common pattern in Swift, where users expect to be able to set a breakpoint on the early exit:

public func f(b: Bool) {
  guard b else { 
    return // I would like to set a breakpoint here.
  }
  ...
}

The patch modifies two places in GlobalISEL: The first one is in IRTranslator.cpp where the removal of redundant branches is made conditional on the optimization level. The second one is in AArch64InstructionSelector.cpp where an -O0 *only* optimization is being removed.

rdar://79515454

[dmgreen]

Sounds sensible to me. I wouldn't expect -O0 to be doing many optimizations.

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

2095–2097

The formatting is off now.

llvm/test/DebugInfo/AArch64/fallthrough-branch.ll

5

This test looks like it could be cleaned up quite a bit. It doesn't need debug info for example, as far as I understand. Just a block that fallsthrough.

[paquette]

I think that prioritizing debuggability at -O0 is in general the right call, but I am a little curious about the impact on code size. Does this increase -O0 CTMark code size a lot?

[aemerson]

In D105238#2853357, @paquette wrote:

I think that prioritizing debuggability at -O0 is in general the right call, but I am a little curious about the impact on code size. Does this increase -O0 CTMark code size a lot?

In fact it does, a lot. 8% geomean. However, SelectionDAG does the same thing so this patch would be making us match its behavior. It is unfortunate though.

[aemerson]

@aprantl by the way, I'm pretty sure there are a fair number of tests that need updating with this change.

[aprantl]

In D105238#2853402, @aemerson wrote:

@aprantl by the way, I'm pretty sure there are a fair number of tests that need updating with this change.

Yes that' right:

Failed Tests (12):
  LLVM :: CodeGen/AArch64/GlobalISel/arm64-atomic.ll
  LLVM :: CodeGen/AArch64/GlobalISel/arm64-irtranslator.ll
  LLVM :: CodeGen/AArch64/unwind-preserved.ll
  LLVM :: CodeGen/BPF/BTF/func-func-ptr.ll
  LLVM :: CodeGen/BPF/BTF/func-non-void.ll
  LLVM :: CodeGen/BPF/BTF/func-typedef.ll
  LLVM :: CodeGen/BPF/BTF/func-unused-arg.ll
  LLVM :: CodeGen/BPF/BTF/func-void.ll
  LLVM :: CodeGen/Mips/GlobalISel/llvm-ir/jump_table_and_brjt.ll
  LLVM :: CodeGen/Mips/GlobalISel/llvm-ir/long_ambiguous_chain_s32.ll
  LLVM :: CodeGen/Mips/GlobalISel/llvm-ir/long_ambiguous_chain_s64.ll
  LLVM :: CodeGen/Mips/GlobalISel/llvm-ir/phi.ll

Before I go through updating all the tests, do we think that this is a reasonable change?

Pros:

• Consistent with SelectionDAG behavior (fixes a "regression" in GlobalISEL)
• Enables debugging scenarios (setting breakpoints on early returns, etc)

Cons:

• Code size increase (-O0 only)

The patch improves debuggability, and debug builds are the primary reason why developers compile without optimizations. From my point of view this seems to be the right trade-off, but please let me know if there are any other interests we need to consider.

[aemerson]

I agree that debug info quality is more important at -O0, so while 8% is a tough regression to take, I think it's the right thing to do here.

[paquette]

+1, debuggability should be the priority here.

[aprantl]

Updated the rest of the codegen tests.

[aprantl]

Upload correct patch...

[aprantl]

And fixed a whitespace issue.

[aemerson]

LGTM.

[omjavaid]

This change appears to have broken LLDB testsuite failure https://lab.llvm.org/buildbot#builders/96/builds/9404

[aemerson]

In D105238#2863435, @omjavaid wrote:

This change appears to have broken LLDB testsuite failure https://lab.llvm.org/buildbot#builders/96/builds/9404

Sounds like a familiar fragile test.

[omjavaid]

This broke LLDB buildbot testcase where breakpoint set at start of loop failed to hit.

https://lab.llvm.org/buildbot/#/builders/96/builds/9404

https://github.com/llvm/llvm-project/blob/main/lldb/test/API/commands/process/attach/main.cpp#L15

I have temporarily reverted this change.

[aprantl]

In D105238#2866200, @omjavaid wrote:

This broke LLDB buildbot testcase where breakpoint set at start of loop failed to hit.

https://lab.llvm.org/buildbot/#/builders/96/builds/9404

https://github.com/llvm/llvm-project/blob/main/lldb/test/API/commands/process/attach/main.cpp#L15

I have temporarily reverted this change.

I agree the bot data seems to point to this commit. I'll try to figure out what happened there.

[aprantl]

The LLDB test that was failing was badly written and was trying to set a breakpoint on a line that was not guaranteed to have any code associated with it. I reapplied the change, fixed the test and the bot green again.

llvm/lib/CodeGen/GlobalISel/IRTranslator.cpp

 
 bool IRTranslator::translateBr(const User &U, MachineIRBuilder &MIRBuilder) {
   const BranchInst &BrInst = cast<BranchInst>(U);
   auto &CurMBB = MIRBuilder.getMBB();
   auto *Succ0MBB = &getMBB(*BrInst.getSuccessor(0));
 
   if (BrInst.isUnconditional()) {
     // If the unconditional target is the layout successor, fallthrough.
-    if (!CurMBB.isLayoutSuccessor(Succ0MBB))
+    if (OptLevel == CodeGenOpt::None || !CurMBB.isLayoutSuccessor(Succ0MBB))
       MIRBuilder.buildBr(*Succ0MBB);
 
     // Link successors.
     for (const BasicBlock *Succ : successors(&BrInst))
       CurMBB.addSuccessor(&getMBB(*Succ));
     return true;
   }
 

llvm/lib/Target/AArch64/GISel/AArch64InstructionSelector.cpp

case AArch64::G_DUP: {
     auto *CV = ConstantDataVector::getSplat(
         MRI.getType(Dst).getNumElements(),
         ConstantInt::get(Type::getIntNTy(Ctx, MRI.getType(Src).getSizeInBits()),
                          ValAndVReg->Value));
     if (!emitConstantVector(Dst, CV, MIB, MRI))
       return false;
     I.eraseFromParent();
     return true;
   }
-  case TargetOpcode::G_BR: {
-    // If the branch jumps to the fallthrough block, don't bother emitting it.
-    // Only do this for -O0 for a good code size improvement, because when
-    // optimizations are enabled we want to leave this choice to
-    // MachineBlockPlacement.
-    bool EnableOpt = MF.getTarget().getOptLevel() != CodeGenOpt::None;
-    if (EnableOpt || !MBB.isLayoutSuccessor(I.getOperand(0).getMBB()))
-      return false;
-    I.eraseFromParent();
-    return true;
-  }
+  case TargetOpcode::G_BR:
   case TargetOpcode::G_SHL:
     return earlySelectSHL(I, MRI);
   case TargetOpcode::G_CONSTANT: {
     bool IsZero = false;
     if (I.getOperand(1).isCImm())
       IsZero = I.getOperand(1).getCImm()->getZExtValue() == 0;
     else if (I.getOperand(1).isImm())
       IsZero = I.getOperand(1).getImm() == 0;

llvm/test/CodeGen/AArch64/GlobalISel/arm64-atomic.ll

 ; CHECK-NOLSE-O1-NEXT:    ret
 ;
 ; CHECK-NOLSE-O0-LABEL: fetch_and_nand:
 ; CHECK-NOLSE-O0:       ; %bb.0:
 ; CHECK-NOLSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    ldr w8, [x0]
 ; CHECK-NOLSE-O0-NEXT:    str w8, [sp, #28] ; 4-byte Folded Spill
+; CHECK-NOLSE-O0-NEXT:  b LBB6_1
 ; CHECK-NOLSE-O0-NEXT:  LBB6_1: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; =>This Loop Header: Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; Child Loop BB6_2 Depth 2
 ; CHECK-NOLSE-O0-NEXT:    ldr w8, [sp, #28] ; 4-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    and w9, w8, #0x7
 ; CHECK-NOLSE-O0-NEXT:    mvn w12, w9
 ; CHECK-NOLSE-O0-NEXT:  LBB6_2: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; Parent Loop BB6_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; => This Inner Loop Header: Depth=2
 ; CHECK-NOLSE-O0-NEXT:    ldaxr w9, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cmp w9, w8
 ; CHECK-NOLSE-O0-NEXT:    b.ne LBB6_4
 ; CHECK-NOLSE-O0-NEXT:  ; %bb.3: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB6_2 Depth=2
 ; CHECK-NOLSE-O0-NEXT:    stlxr w10, w12, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cbnz w10, LBB6_2
 ; CHECK-NOLSE-O0-NEXT:  LBB6_4: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB6_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    str w9, [sp, #12] ; 4-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    subs w8, w9, w8
 ; CHECK-NOLSE-O0-NEXT:    cset w8, eq
 ; CHECK-NOLSE-O0-NEXT:    str w9, [sp, #28] ; 4-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    tbz w8, #0, LBB6_1
-; CHECK-NOLSE-O0-NEXT:  ; %bb.5: ; %atomicrmw.end
+; CHECK-NOLSE-O0-NEXT:    b LBB6_5
+; CHECK-NOLSE-O0-NEXT:  LBB6_5: ; %atomicrmw.end
 ; CHECK-NOLSE-O0-NEXT:    ldr w0, [sp, #12] ; 4-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    ret
 ;
 ; CHECK-LSE-O1-LABEL: fetch_and_nand:
 ; CHECK-LSE-O1:       ; %bb.0:
 ; CHECK-LSE-O1-NEXT:  LBB6_1: ; %atomicrmw.start
 ; CHECK-LSE-O1-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-LSE-O1-NEXT:    ldxr w8, [x0]
 ; CHECK-LSE-O1-NEXT:    and w9, w8, #0x7
 ; CHECK-LSE-O1-NEXT:    mvn w9, w9
 ; CHECK-LSE-O1-NEXT:    stlxr w10, w9, [x0]
 ; CHECK-LSE-O1-NEXT:    cbnz w10, LBB6_1
 ; CHECK-LSE-O1-NEXT:  ; %bb.2: ; %atomicrmw.end
 ; CHECK-LSE-O1-NEXT:    mov x0, x8
 ; CHECK-LSE-O1-NEXT:    ret
 ;
 ; CHECK-LSE-O0-LABEL: fetch_and_nand:
 ; CHECK-LSE-O0:       ; %bb.0:
 ; CHECK-LSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-LSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    ldr w8, [x0]
 ; CHECK-LSE-O0-NEXT:    str w8, [sp, #28] ; 4-byte Folded Spill
+; CHECK-LSE-O0-NEXT:    b LBB6_1
 ; CHECK-LSE-O0-NEXT:  LBB6_1: ; %atomicrmw.start
 ; CHECK-LSE-O0-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-LSE-O0-NEXT:    ldr w8, [sp, #28] ; 4-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    and w9, w8, #0x7
 ; CHECK-LSE-O0-NEXT:    mvn w10, w9
 ; CHECK-LSE-O0-NEXT:    mov x9, x8
 ; CHECK-LSE-O0-NEXT:    casl w9, w10, [x11]
 ; CHECK-LSE-O0-NEXT:    str w9, [sp, #12] ; 4-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    subs w8, w9, w8
 ; CHECK-LSE-O0-NEXT:    cset w8, eq
 ; CHECK-LSE-O0-NEXT:    str w9, [sp, #28] ; 4-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    tbz w8, #0, LBB6_1
-; CHECK-LSE-O0-NEXT:  ; %bb.2: ; %atomicrmw.end
+; CHECK-LSE-O0-NEXT:    b LBB6_2
+; CHECK-LSE-O0-NEXT:  LBB6_2: ; %atomicrmw.end
 ; CHECK-LSE-O0-NEXT:    ldr w0, [sp, #12] ; 4-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-LSE-O0-NEXT:    ret
   %val = atomicrmw nand i32* %p, i32 7 release
   ret i32 %val
 }
 
 define i64 @fetch_and_nand_64(i64* %p) #0 {

 ; CHECK-NOLSE-O1-NEXT:    ret
 ;
 ; CHECK-NOLSE-O0-LABEL: fetch_and_nand_64:
 ; CHECK-NOLSE-O0:       ; %bb.0:
 ; CHECK-NOLSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    ldr x8, [x0]
 ; CHECK-NOLSE-O0-NEXT:    str x8, [sp, #24] ; 8-byte Folded Spill
+; CHECK-NOLSE-O0-NEXT:    b LBB7_1
 ; CHECK-NOLSE-O0-NEXT:  LBB7_1: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; =>This Loop Header: Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; Child Loop BB7_2 Depth 2
 ; CHECK-NOLSE-O0-NEXT:    ldr x8, [sp, #24] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    and x9, x8, #0x7
 ; CHECK-NOLSE-O0-NEXT:    mvn x12, x9
 ; CHECK-NOLSE-O0-NEXT:  LBB7_2: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; Parent Loop BB7_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; => This Inner Loop Header: Depth=2
 ; CHECK-NOLSE-O0-NEXT:    ldaxr x9, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cmp x9, x8
 ; CHECK-NOLSE-O0-NEXT:    b.ne LBB7_4
 ; CHECK-NOLSE-O0-NEXT:  ; %bb.3: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB7_2 Depth=2
 ; CHECK-NOLSE-O0-NEXT:    stlxr w10, x12, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cbnz w10, LBB7_2
 ; CHECK-NOLSE-O0-NEXT:  LBB7_4: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB7_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    str x9, [sp, #8] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    subs x8, x9, x8
 ; CHECK-NOLSE-O0-NEXT:    cset w8, eq
 ; CHECK-NOLSE-O0-NEXT:    str x9, [sp, #24] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    tbz w8, #0, LBB7_1
-; CHECK-NOLSE-O0-NEXT:  ; %bb.5: ; %atomicrmw.end
+; CHECK-NOLSE-O0-NEXT:    b LBB7_5
+; CHECK-NOLSE-O0-NEXT:  LBB7_5: ; %atomicrmw.end
 ; CHECK-NOLSE-O0-NEXT:    ldr x0, [sp, #8] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    ret
 ;
 ; CHECK-LSE-O1-LABEL: fetch_and_nand_64:
 ; CHECK-LSE-O1:       ; %bb.0:
 ; CHECK-LSE-O1-NEXT:  LBB7_1: ; %atomicrmw.start
 ; CHECK-LSE-O1-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-LSE-O1-NEXT:    ldaxr x8, [x0]
 ; CHECK-LSE-O1-NEXT:    and x9, x8, #0x7
 ; CHECK-LSE-O1-NEXT:    mvn x9, x9
 ; CHECK-LSE-O1-NEXT:    stlxr w10, x9, [x0]
 ; CHECK-LSE-O1-NEXT:    cbnz w10, LBB7_1
 ; CHECK-LSE-O1-NEXT:  ; %bb.2: ; %atomicrmw.end
 ; CHECK-LSE-O1-NEXT:    mov x0, x8
 ; CHECK-LSE-O1-NEXT:    ret
 ;
 ; CHECK-LSE-O0-LABEL: fetch_and_nand_64:
 ; CHECK-LSE-O0:       ; %bb.0:
 ; CHECK-LSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-LSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    ldr x8, [x0]
 ; CHECK-LSE-O0-NEXT:    str x8, [sp, #24] ; 8-byte Folded Spill
+; CHECK-LSE-O0-NEXT:    b LBB7_1
 ; CHECK-LSE-O0-NEXT:  LBB7_1: ; %atomicrmw.start
 ; CHECK-LSE-O0-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-LSE-O0-NEXT:    ldr x8, [sp, #24] ; 8-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    and x9, x8, #0x7
 ; CHECK-LSE-O0-NEXT:    mvn x10, x9
 ; CHECK-LSE-O0-NEXT:    mov x9, x8
 ; CHECK-LSE-O0-NEXT:    casal x9, x10, [x11]
 ; CHECK-LSE-O0-NEXT:    str x9, [sp, #8] ; 8-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    subs x8, x9, x8
 ; CHECK-LSE-O0-NEXT:    cset w8, eq
 ; CHECK-LSE-O0-NEXT:    str x9, [sp, #24] ; 8-byte Folded Spill
 ; CHECK-LSE-O0-NEXT:    tbz w8, #0, LBB7_1
-; CHECK-LSE-O0-NEXT:  ; %bb.2: ; %atomicrmw.end
+; CHECK-LSE-O0-NEXT:    b LBB7_2
+; CHECK-LSE-O0-NEXT:  LBB7_2: ; %atomicrmw.end
 ; CHECK-LSE-O0-NEXT:    ldr x0, [sp, #8] ; 8-byte Folded Reload
 ; CHECK-LSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-LSE-O0-NEXT:    ret
   %val = atomicrmw nand i64* %p, i64 7 acq_rel
   ret i64 %val
 }
 
 define i32 @fetch_and_or(i32* %p) #0 {

 ; CHECK-NOLSE-O1-NEXT:    ret
 ;
 ; CHECK-NOLSE-O0-LABEL: fetch_and_or:
 ; CHECK-NOLSE-O0:       ; %bb.0:
 ; CHECK-NOLSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    ldr w8, [x0]
 ; CHECK-NOLSE-O0-NEXT:    str w8, [sp, #28] ; 4-byte Folded Spill
+; CHECK-NOLSE-O0-NEXT:    b LBB8_1
 ; CHECK-NOLSE-O0-NEXT:  LBB8_1: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; =>This Loop Header: Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; Child Loop BB8_2 Depth 2
 ; CHECK-NOLSE-O0-NEXT:    ldr w8, [sp, #28] ; 4-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    mov w9, #5
 ; CHECK-NOLSE-O0-NEXT:    orr w12, w8, w9
 ; CHECK-NOLSE-O0-NEXT:  LBB8_2: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; Parent Loop BB8_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; => This Inner Loop Header: Depth=2
 ; CHECK-NOLSE-O0-NEXT:    ldaxr w9, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cmp w9, w8
 ; CHECK-NOLSE-O0-NEXT:    b.ne LBB8_4
 ; CHECK-NOLSE-O0-NEXT:  ; %bb.3: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB8_2 Depth=2
 ; CHECK-NOLSE-O0-NEXT:    stlxr w10, w12, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cbnz w10, LBB8_2
 ; CHECK-NOLSE-O0-NEXT:  LBB8_4: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB8_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    str w9, [sp, #12] ; 4-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    subs w8, w9, w8
 ; CHECK-NOLSE-O0-NEXT:    cset w8, eq
 ; CHECK-NOLSE-O0-NEXT:    str w9, [sp, #28] ; 4-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    tbz w8, #0, LBB8_1
-; CHECK-NOLSE-O0-NEXT:  ; %bb.5: ; %atomicrmw.end
+; CHECK-NOLSE-O0-NEXT:    LBB8_5
+; CHECK-NOLSE-O0-NEXT:  LBB8_5: ; %atomicrmw.end
 ; CHECK-NOLSE-O0-NEXT:    ldr w0, [sp, #12] ; 4-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    ret
 ;
 ; CHECK-LSE-O1-LABEL: fetch_and_or:
 ; CHECK-LSE-O1:       ; %bb.0:
 ; CHECK-LSE-O1-NEXT:    mov w8, #5
 ; CHECK-LSE-O1-NEXT:    ldsetal w8, w0, [x0]

 ; CHECK-NOLSE-O1-NEXT:    ret
 ;
 ; CHECK-NOLSE-O0-LABEL: fetch_and_or_64:
 ; CHECK-NOLSE-O0:       ; %bb.0:
 ; CHECK-NOLSE-O0-NEXT:    sub sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    str x0, [sp, #16] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    ldr x8, [x0]
 ; CHECK-NOLSE-O0-NEXT:    str x8, [sp, #24] ; 8-byte Folded Spill
+; CHECK-NOLSE-O0-NEXT:    b LBB9_1
 ; CHECK-NOLSE-O0-NEXT:  LBB9_1: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; =>This Loop Header: Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; Child Loop BB9_2 Depth 2
 ; CHECK-NOLSE-O0-NEXT:    ldr x8, [sp, #24] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    ldr x11, [sp, #16] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    orr x12, x8, #0x7
 ; CHECK-NOLSE-O0-NEXT:  LBB9_2: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; Parent Loop BB9_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    ; => This Inner Loop Header: Depth=2
 ; CHECK-NOLSE-O0-NEXT:    ldaxr x9, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cmp x9, x8
 ; CHECK-NOLSE-O0-NEXT:    b.ne LBB9_4
 ; CHECK-NOLSE-O0-NEXT:  ; %bb.3: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB9_2 Depth=2
 ; CHECK-NOLSE-O0-NEXT:    stlxr w10, x12, [x11]
 ; CHECK-NOLSE-O0-NEXT:    cbnz w10, LBB9_2
 ; CHECK-NOLSE-O0-NEXT:  LBB9_4: ; %atomicrmw.start
 ; CHECK-NOLSE-O0-NEXT:    ; in Loop: Header=BB9_1 Depth=1
 ; CHECK-NOLSE-O0-NEXT:    str x9, [sp, #8] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    subs x8, x9, x8
 ; CHECK-NOLSE-O0-NEXT:    cset w8, eq
 ; CHECK-NOLSE-O0-NEXT:    str x9, [sp, #24] ; 8-byte Folded Spill
 ; CHECK-NOLSE-O0-NEXT:    tbz w8, #0, LBB9_1
-; CHECK-NOLSE-O0-NEXT:  ; %bb.5: ; %atomicrmw.end
+; CHECK-NOLSE-O0-NEXT:    b LBB9_5
+; CHECK-NOLSE-O0-NEXT:  LBB9_5: ; %atomicrmw.end
 ; CHECK-NOLSE-O0-NEXT:    ldr x0, [sp, #8] ; 8-byte Folded Reload
 ; CHECK-NOLSE-O0-NEXT:    add sp, sp, #32 ; =32
 ; CHECK-NOLSE-O0-NEXT:    ret
 ;
 ; CHECK-LSE-O1-LABEL: fetch_and_or_64:
 ; CHECK-LSE-O1:       ; %bb.0:
 ; CHECK-LSE-O1-NEXT:    mov w8, #7
 ; CHECK-LSE-O1-NEXT:    ldset x8, x0, [x0]

llvm/test/CodeGen/AArch64/GlobalISel/arm64-irtranslator.ll

 bb2:
   br label %end
 }
 
 ; CHECK-LABEL: name: uncondbr_fallthrough
 ; CHECK: body:
 ; CHECK: bb.{{[0-9]+}}.{{[a-zA-Z0-9.]+}}:
 ; CHECK-NEXT: successors: %[[END:bb.[0-9]+]](0x80000000)
-; We don't emit a branch here, as we can fallthrough to the successor.
-; CHECK-NOT: G_BR
 ; CHECK: [[END]].{{[a-zA-Z0-9.]+}}:
 ; CHECK-NEXT: RET_ReallyLR
 define void @uncondbr_fallthrough() {
 entry:
   br label %end
 end:
   ret void
 }

 ; Tests for indirect br.
 ; CHECK-LABEL: name: indirectbr
 ; CHECK: body:
 ;
 ; ABI/constant lowering and IR-level entry basic block.
 ; CHECK: bb.{{[0-9]+.[a-zA-Z0-9.]+}}:
 ; Make sure we have one successor
 ; CHECK-NEXT: successors: %[[BB_L1:bb.[0-9]+]](0x80000000)
-; CHECK-NOT: G_BR
 ;
 ; Check basic block L1 has 2 successors: BBL1 and BBL2
 ; CHECK: [[BB_L1]].{{[a-zA-Z0-9.]+}} (address-taken):
 ; CHECK-NEXT: successors: %[[BB_L1]](0x40000000),
 ; CHECK:                  %[[BB_L2:bb.[0-9]+]](0x40000000)
 ; CHECK: G_BRINDIRECT %{{[0-9]+}}(p0)
 ;
 ; Check basic block L2 is the return basic block

llvm/test/CodeGen/AArch64/unwind-preserved.ll

 ; GISEL-NEXT:    .cfi_offset b21, -240
 ; GISEL-NEXT:    .cfi_offset b22, -256
 ; GISEL-NEXT:    .cfi_offset b23, -272
 ; GISEL-NEXT:    str q0, [sp] // 16-byte Folded Spill
 ; GISEL-NEXT:  .Ltmp3:
 ; GISEL-NEXT:    bl may_throw_neon
 ; GISEL-NEXT:    str q0, [sp, #16] // 16-byte Folded Spill
 ; GISEL-NEXT:  .Ltmp4:
-; GISEL-NEXT:  // %bb.1: // %.Lcontinue
+; GISEL-NEXT:    b .LBB1_1 
+; GISEL-NEXT:  .LBB1_1: // %.Lcontinue
 ; GISEL-NEXT:    ldr q0, [sp, #16] // 16-byte Folded Reload
 ; GISEL-NEXT:    ldp x29, x30, [sp, #288] // 16-byte Folded Reload
 ; GISEL-NEXT:    ldp q9, q8, [sp, #256] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q11, q10, [sp, #224] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q13, q12, [sp, #192] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q15, q14, [sp, #160] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q17, q16, [sp, #128] // 32-byte Folded Reload
 ; GISEL-NEXT:    ldp q19, q18, [sp, #96] // 32-byte Folded Reload

llvm/test/CodeGen/Mips/GlobalISel/llvm-ir/jump_table_and_brjt.ll

 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_5: # %sw.bb3
 ; MIPS32-NEXT:    lw $2, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 32
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_6: # %sw.default
-; MIPS32-NEXT:    .insn
-; MIPS32-NEXT:  # %bb.7: # %sw.epilog
+; MIPS32-NEXT:    j $BB0_7
+; MIPS32-NEXT:    nop
+; MIPS32-NEXT:  $BB0_7: # %sw.epilog
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ori $3, $zero, 8
 ; MIPS32-NEXT:    subu $2, $2, $3
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sltu $1, $1, $2
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    bnez $1, $BB0_13

 ; MIPS32_PIC-NEXT:    jr $ra
 ; MIPS32_PIC-NEXT:    nop
 ; MIPS32_PIC-NEXT:  $BB0_5: # %sw.bb3
 ; MIPS32_PIC-NEXT:    lw $2, 16($sp) # 4-byte Folded Reload
 ; MIPS32_PIC-NEXT:    addiu $sp, $sp, 40
 ; MIPS32_PIC-NEXT:    jr $ra
 ; MIPS32_PIC-NEXT:    nop
 ; MIPS32_PIC-NEXT:  $BB0_6: # %sw.default
-; MIPS32_PIC-NEXT:    .insn
-; MIPS32_PIC-NEXT:  # %bb.7: # %sw.epilog
+; MIPS32_PIC-NEXT:    b $BB0_7
+; MIPS32_PIC-NEXT:    nop
+; MIPS32_PIC-NEXT:  $BB0_7: # %sw.epilog
 ; MIPS32_PIC-NEXT:    lw $1, 16($sp) # 4-byte Folded Reload
 ; MIPS32_PIC-NEXT:    lw $2, 12($sp) # 4-byte Folded Reload
 ; MIPS32_PIC-NEXT:    ori $3, $zero, 8
 ; MIPS32_PIC-NEXT:    subu $2, $2, $3
 ; MIPS32_PIC-NEXT:    sw $2, 4($sp) # 4-byte Folded Spill
 ; MIPS32_PIC-NEXT:    sltu $1, $1, $2
 ; MIPS32_PIC-NEXT:    andi $1, $1, 1
 ; MIPS32_PIC-NEXT:    bnez $1, $BB0_13

llvm/test/CodeGen/Mips/GlobalISel/llvm-ir/long_ambiguous_chain_s32.ll

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 16($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB0_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 40($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 16($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB0_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $1, 28($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 16($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 8($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sw $2, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB0_11
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB0_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 36($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB0_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 24($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sw $2, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB0_19
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 20($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB1_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 44($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 20($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB1_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $2, 52($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 32($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $3, 20($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $3, 8($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sw $3, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $2, 16($sp) # 4-byte Folded Spill

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB1_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 40($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB1_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 28($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    move $3, $2
 ; MIPS32-NEXT:    sw $3, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $2, 16($sp) # 4-byte Folded Spill

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 16($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB2_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 40($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 16($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB2_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $1, 28($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 16($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 8($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sw $2, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB2_11
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB2_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 36($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($1)
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB2_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 24($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sw $2, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB2_19
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    lwc1 $f0, 0($1)
 ; MIPS32-NEXT:    swc1 $f0, 20($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB3_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 44($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lwc1 $f0, 0($1)
 ; MIPS32-NEXT:    swc1 $f0, 20($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB3_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_9: # %b.PHI.1
 ; MIPS32-NEXT:    lwc1 $f0, 52($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 32($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lwc1 $f1, 20($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    swc1 $f1, 8($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    swc1 $f1, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    swc1 $f0, 16($sp) # 4-byte Folded Spill

 ; MIPS32-NEXT:    lwc1 $f0, 0($1)
 ; MIPS32-NEXT:    swc1 $f0, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB3_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 40($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lwc1 $f0, 0($1)
 ; MIPS32-NEXT:    swc1 $f0, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB3_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 28($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lwc1 $f0, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    swc1 $f0, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    mov.s $f1, $f0
 ; MIPS32-NEXT:    swc1 $f1, 12($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    swc1 $f0, 16($sp) # 4-byte Folded Spill

llvm/test/CodeGen/Mips/GlobalISel/llvm-ir/long_ambiguous_chain_s64.ll

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB0_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 64($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB0_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $1, 52($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 32($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 16($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sdc1 $f0, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB0_11
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB0_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 60($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB0_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 48($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 8($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sdc1 $f0, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB0_19
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    j $BB1_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 68($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 0($1)
 ; MIPS32-NEXT:    lw $1, 4($1)
 ; MIPS32-NEXT:    sw $2, 40($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $1, 44($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB1_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $2, 76($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 56($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $4, 40($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $3, 44($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $3, 16($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $4, 20($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1

 ; MIPS32-NEXT:    j $BB1_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 64($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 0($1)
 ; MIPS32-NEXT:    lw $1, 4($1)
 ; MIPS32-NEXT:    sw $2, 8($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $1, 12($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB1_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 52($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $3, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $3, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    move $4, $3

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB2_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 64($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB2_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_9: # %b.PHI.1
 ; MIPS32-NEXT:    lw $1, 52($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 32($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 16($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sdc1 $f0, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB2_11
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB2_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 60($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB2_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 48($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 8($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sdc1 $f0, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    bnez $1, $BB2_19
 ; MIPS32-NEXT:    nop

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 40($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB3_9
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_8: # %b.PHI.1.2
 ; MIPS32-NEXT:    lw $1, 72($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 40($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB3_9
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_9: # %b.PHI.1
 ; MIPS32-NEXT:    ldc1 $f0, 80($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 60($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f2, 40($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f2, 16($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    sdc1 $f2, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill

 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB3_16
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_15: # %b.PHI.2.1
 ; MIPS32-NEXT:    lw $1, 68($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($1)
 ; MIPS32-NEXT:    sdc1 $f0, 8($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB3_16
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_16: # %b.PHI.2
 ; MIPS32-NEXT:    lw $1, 56($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 8($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    andi $1, $1, 1
 ; MIPS32-NEXT:    mov.d $f2, $f0
 ; MIPS32-NEXT:    sdc1 $f2, 24($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    sdc1 $f0, 32($sp) # 8-byte Folded Spill

llvm/test/CodeGen/Mips/GlobalISel/llvm-ir/phi.ll

 ; MIPS32-NEXT:  $BB0_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB0_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB0_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB0_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false
 

 ; MIPS32-NEXT:  $BB1_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB1_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB1_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB1_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false
 

 ; MIPS32-NEXT:  $BB2_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB2_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB2_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB2_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false
 

 ; MIPS32-NEXT:  $BB3_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB3_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB3_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB3_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false
 

 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB4_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB4_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 20($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $2, 16($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $2, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB4_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB4_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 0($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $3, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 24
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false

 ; MIPS32-NEXT:  $BB5_2: # %cond.true
 ; MIPS32-NEXT:    ldc1 $f0, 16($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB5_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB5_3: # %cond.false
 ; MIPS32-NEXT:    ldc1 $f0, 24($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB5_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB5_4: # %cond.end
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    ldc1 $f0, 0($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($1)
 ; MIPS32-NEXT:    addiu $sp, $sp, 32
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:

 ; MIPS32-NEXT:  $BB6_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB6_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB6_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 4($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB6_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB6_4: # %cond.end
 ; MIPS32-NEXT:    lw $1, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    mtc1 $1, $f0
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false

 ; MIPS32-NEXT:  $BB7_2: # %cond.true
 ; MIPS32-NEXT:    lw $1, 8($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 0($sp) # 4-byte Folded Spill
 ; MIPS32-NEXT:    j $BB7_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB7_3: # %cond.false
 ; MIPS32-NEXT:    lw $1, 12($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 0($sp) # 4-byte Folded Spill
+; MIPS32-NEXT:    j $BB7_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB7_4: # %cond.end
 ; MIPS32-NEXT:    lw $2, 4($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    lw $1, 0($sp) # 4-byte Folded Reload
 ; MIPS32-NEXT:    sw $1, 0($2)
 ; MIPS32-NEXT:    addiu $sp, $sp, 16
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:

 ; MIPS32-NEXT:  $BB8_2: # %cond.true
 ; MIPS32-NEXT:    ldc1 $f0, 8($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
 ; MIPS32-NEXT:    j $BB8_4
 ; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB8_3: # %cond.false
 ; MIPS32-NEXT:    ldc1 $f0, 16($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    sdc1 $f0, 0($sp) # 8-byte Folded Spill
+; MIPS32-NEXT:    j $BB8_4
+; MIPS32-NEXT:    nop
 ; MIPS32-NEXT:  $BB8_4: # %cond.end
 ; MIPS32-NEXT:    ldc1 $f0, 0($sp) # 8-byte Folded Reload
 ; MIPS32-NEXT:    addiu $sp, $sp, 24
 ; MIPS32-NEXT:    jr $ra
 ; MIPS32-NEXT:    nop
 entry:
   br i1 %cnd, label %cond.true, label %cond.false
 

llvm/test/DebugInfo/AArch64/fallthrough-branch.ll

+; RUN: llc -O0 -stop-before=livedebugvalues < %s | FileCheck %s
+
+; ModuleID = '/tmp/t.o'
+source_filename = "/tmp/t.o"
+target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
+target triple = "arm64-apple-macosx11.0.0"
+
+define swiftcc void @"$s1t1f1bySb_tF"(i1 %0) !dbg !35 {
+  %2 = alloca i1, align 8
+  %3 = bitcast i1* %2 to i8*
+  call void @llvm.memset.p0i8.i64(i8* align 8 %3, i8 0, i64 1, i1 false)
+  store i1 %0, i1* %2, align 8, !dbg !37
+; CHECK:   B %bb.1, debug-location !{{[0-9]+}}
+  br i1 %0, label %4, label %5, !dbg !38
+
+4:                                                ; preds = %1
+; Check that at -O0 the branches and their debug locations are not eliminated.
+; CHECK:   B %bb.3, debug-location !{{[0-9]+}}
+  br label %6, !dbg !39
+
+5:                                                ; preds = %1
+; CHECK:   B %bb.3, debug-location !{{[0-9]+}}
+  br label %6, !dbg !40
+
+6:                                                ; preds = %4, %5
+  ret void, !dbg !39
+}
+
+; Function Attrs: argmemonly nofree nosync nounwind willreturn writeonly
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg) #1
+attributes #1 = { argmemonly nofree nosync nounwind willreturn writeonly }
+
+!llvm.module.flags = !{!6, !7, !14}
+!llvm.dbg.cu = !{!15, !27}
+
+!6 = !{i32 7, !"Dwarf Version", i32 4}
+!7 = !{i32 2, !"Debug Info Version", i32 3}
+!14 = !{i32 1, !"Swift Version", i32 7}
+!15 = distinct !DICompileUnit(language: DW_LANG_Swift, file: !16, producer: "Swift", emissionKind: LineTablesOnly)
+!16 = !DIFile(filename: "t.swift", directory: "/tmp")
+!17 = !{}
+!27 = distinct !DICompileUnit(language: DW_LANG_ObjC, file: !16, emissionKind: LineTablesOnly)
+!35 = distinct !DISubprogram(name: "f", linkageName: "$s1t1f1bySb_tF", scope: !15, file: !16, line: 1, type: !36, scopeLine: 1, spFlags: DISPFlagDefinition, unit: !15, retainedNodes: !17)
+!36 = !DISubroutineType(types: null)
+!37 = !DILocation(line: 0, scope: !35)
+!38 = !DILocation(line: 2, column: 9, scope: !35)
+!39 = !DILocation(line: 3, column: 1, scope: !35)
+!40 = !DILocation(line: 2, column: 18, scope: !35)