Change the INLINEASM_BR MachineInstr to be a non-terminating instruction.

[jyknight]

Summary

Before this instruction supported output values, it fit fairly
naturally as a terminator. However, being a terminator while also
supporting outputs causes some trouble, as the physreg->vreg COPY
operations cannot be in the same block.

Modeling it as a non-terminator allows it to be handled the same way
as invoke is handled already.

Most of the changes here were created by auditing all the existing
users of MachineBasicBlock::isEHPad() and
MachineBasicBlock::hasEHPadSuccessor(), and adding calls to
isInlineAsmBrIndirectTarget or hasInlineAsmBr, as appropriate.

[nickdesaulniers]

An interesting approach. It certainly seems to simplify some of the special handling. I'm happy that it seems to delete much of the existing code. Thank you for taking the time to write up this patch.

I don't fully understand what "shrink wrapping" is, or the changes to BranchFolding, but the rest of the patch looks pretty good to me. This obviously has implications for asm goto without outputs, so I'd like to run this through a couple kernel builds to ensure we haven't regressed anything. From there, I can test kernel builds that use outputs.

Three other things to check:

1. check we don't spill post terminators (https://reviews.llvm.org/D78166). Probably no issue there.
2. check that BranchFolder::RemoveDeadBlock isn't removing any MachineBasicBlock that have their address taken. (https://reviews.llvm.org/D78234) tries to do this, but we've seen cases where asm goto w/ outputs results in the indirect successor being removed by BranchFolder. We have a case from the kernel that triggered the above two patches, and is still a problem for https://reviews.llvm.org/D75098 that I can send you.
3. That live ins match live outs (https://reviews.llvm.org/D78586). Probably no issue there.

llvm/include/llvm/CodeGen/MachineBasicBlock.h

483

I think @efriedma has been noting that the API at the MIR level doesn't feel symmetric with the LLVM IR level.

https://reviews.llvm.org/D78234#1987870 and
https://reviews.llvm.org/D78234#1989382

In LLVM IR, you have BasicBlock::HasAddressTaken, but at the MIR level the operands are still BlockAddress (which reference a Function and BasicBlock, two LLVM IR level concepts). It's too bad we don't lower these to just MachineBasicBlocks (or a new MachineBlockAddress) as operands, and have equivalent machinery for detecting whether a MachineBasicBlock has its address taken.

llvm/include/llvm/Target/Target.td

1022

Delete

llvm/lib/CodeGen/BranchFolding.cpp

1706

That's neat, I didn't know you could use initializer lists as ranges for range based for loops.s

llvm/lib/CodeGen/MachineBasicBlock.cpp

281–284

return any_of(successors(), [](MachineBasicBlock* Succ) {
  return Succ->isInlineAsmBrIndirectTarget(); };

or better yet, why not be precise and iterate terminators() checking the getOpcode() == TargetOpcode::INLINEASM_BR?

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

410 ↗

(On Diff #263461)

Unused?

llvm/lib/CodeGen/SplitKit.cpp

101

Is this assignment ok to remove?

llvm/lib/Target/Hexagon/BitTracker.cpp

959–960

DefaultToAll |= B.hasInlineAsmBr();

llvm/test/CodeGen/ARM/ifcvt-diamond-unanalyzable-common.mir

66

INLINEASM_BR just disappears from this test?

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

61

does this -1 get overwritten immediately on the next instruction?

[craig.topper]

In D79794#2032052, @nickdesaulniers wrote:

An interesting approach. It certainly seems to simplify some of the special handling. I'm happy that it seems to delete much of the existing code. Thank you for taking the time to write up this patch.

This is funny. The original asm goto patches started off with just using INLINEASM and trying to follow exception handling. We switched to INLINEASM_BR as a terminator to "simplify" things.

[kparzysz]

llvm/lib/Target/Hexagon/BitTracker.cpp

958

It would defeat the purpose of this function. It calculates the set of possible targets for each branch, given the updated register states (i.e. branch conditions), which can be a proper subset of the set of targets listed in the branch.

959–960

Ok.

llvm/lib/Target/Hexagon/HexagonConstPropagation.cpp

758

Same reason as in BitTracker.

760

Ok.

815

...

816

Ok.

821

Ok.

[void]

I think we should rename INLINEASM_BR to something that doesn't involve the _BR bit, since it's no longer a branch.

In general, I'm okay with this approach. (I thought we shouldn't make it a terminator in the first place.)

llvm/lib/CodeGen/MachineBasicBlock.cpp

281–284

It's no longer a terminator, but could be written:

return any_of(*this, [](const MachineInstr &MI) {
  return MI.getOpcode() == TargetOpcode::INLINEASM_BR;
}

289

Is this correct? We should be able to hoist into a block with INLINEASM_BR if it's coming from the default target.

llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

1036

Yay!

[nickdesaulniers]

With this series applied (all 3), I can build and boot an x86 defconfig Linux kernel that makes use of asm goto with outputs.

without outputs, just vanilla asm goto:

x86 allyesconfig builds (we don't try to boot allyesconfigs).

aarch32 defconfig kernels build and boot.

aarch64 defconfig kernels build, but they've regressed booting. No panic, just a dead machine. That will take a while to debug.

[void]

Quick ping to ask about the status of this change. :-)

[jyknight]

In the review on the prerequisite patch, it turned out there's another thing that needs to be fixed first before this -- disambiguating whether the end of a block falls through, or is actually unreachable.

I'm working on creating a patch to do that, after which I'll get back to this patch to address the comments here.

[jyknight]

Just rebase. (Doesn't address review comments)

[void]

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

61

It's the same on line 52. Is it a problem with the PHI elimination pass?

[void]

I think you need the following patch. We shouldn't allow instructions to be rescheduled after an INLINEASM_BR, because all values who's definition dominate the INLINEASM_BR are assumed to be valid on the indirect branch.

diff --git a/llvm/lib/CodeGen/MachineScheduler.cpp b/llvm/lib/CodeGen/MachineScheduler.cpp
index 0f21c97a30f..8cbbd79b661 100644
--- a/llvm/lib/CodeGen/MachineScheduler.cpp
+++ b/llvm/lib/CodeGen/MachineScheduler.cpp
@@ -443,7 +443,8 @@ static bool isSchedBoundary(MachineBasicBlock::iterator MI,
                             MachineBasicBlock *MBB,
                             MachineFunction *MF,
                             const TargetInstrInfo *TII) {
-  return MI->isCall() || TII->isSchedulingBoundary(*MI, MBB, *MF);
+  return MI->isCall() || TII->isSchedulingBoundary(*MI, MBB, *MF) ||
+      (MI->isInlineAsm() && MI->getOpcode() == TargetOpcode::INLINEASM_BR);
 }
 
 /// A region of an MBB for scheduling.
diff --git a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
index 61baa31074e..a4447bc15f1 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
@@ -41,6 +41,7 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    .cfi_offset %edi, -8
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    cmpl %edi, %esi
 ; CHECK-NEXT:    jge .LBB1_2
 ; CHECK-NEXT:  # %bb.1: # %if.then
@@ -49,7 +50,6 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    testl %edi, %esi
 ; CHECK-NEXT:    jne .Ltmp1
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    jmp .LBB1_3
 ; CHECK-NEXT:  .LBB1_2: # %if.else
 ; CHECK-NEXT:    #APP
@@ -57,7 +57,6 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    testl %esi, %edi
 ; CHECK-NEXT:    jne .Ltmp2
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:  .LBB1_3:
 ; CHECK-NEXT:    movl %esi, %eax
 ; CHECK-NEXT:    addl %edi, %eax

[nickdesaulniers]

@nathanchance tested this (version Diff 270500) and also observed boot failures for mainline linux (v5.8-rc1) for aarch64, but additionally aarch64+5.4 LTS stable, and WSL2 kernel for x86_64. I have only briefly started debugging; there were some unrelated issues with RELR relocations and PAC/BTI aarch64 ISA extensions.

[nickdesaulniers]

@void 's additional diff fixes the arm64 mainline boot issue I was observing. Maybe @nathanchance can help test with that additional diff applied atop this patch that we're in good shape?

[nathanchance]

In D79794#2104506, @nickdesaulniers wrote:

@void 's additional diff fixes the arm64 mainline boot issue I was observing. Maybe @nathanchance can help test with that additional diff applied atop this patch that we're in good shape?

Unfortunately, my test case is still broken. With https://gist.github.com/nathanchance/f69a0281d63d6e72a1449d2f5a98636b on top of 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544, this kernel boots (v5.7):

$ make -skj"$(nproc)" LLVM=1 O=out/x86_64 distclean defconfig bzImage
...

$ ../../cbl/github/boot-utils/boot-qemu.sh -a x86_64 -k out/x86_64 -t 30s |& grep -A1 qemu-system-x86_64
+ timeout --foreground 30s unbuffer qemu-system-x86_64 -cpu host -d unimp,guest_errors -enable-kvm -smp 64 -append 'console=ttyS0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/x86_64/rootfs.cpio -kernel /home/nathan/src/linux/out/x86_64/arch/x86_64/boot/bzImage -m 512m -nodefaults -serial mon:stdio
[    0.000000] Linux version 5.7.0 (nathan@ubuntu-n2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project 07bde6d2c4646c888b1011aa079bbaaa250f79b8), LLD 11.0.0 (https://github.com/llvm/llvm-project 07bde6d2c4646c888b1011aa079bbaaa250f79b8)) #1 SMP Fri Jun 19 22:07:10 MST 2020

but this one does not:

$ make -skj"$(nproc)" KCFLAGS=-march=znver2 LLVM=1 LOCALVERSION=-znver2 O=out/x86_64 distclean defconfig bzImage
...

$ ../../cbl/github/boot-utils/boot-qemu.sh -a x86_64 -k out/x86_64 -t 30s |& grep -A1 qemu-system-x86_64
+ timeout --foreground 30s unbuffer qemu-system-x86_64 -cpu host -d unimp,guest_errors -enable-kvm -smp 64 -append 'console=ttyS0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/x86_64/rootfs.cpio -kernel /home/nathan/src/linux/out/x86_64/arch/x86_64/boot/bzImage -m 512m -nodefaults -serial mon:stdio
+ RET=124

At 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544, both kernels boot without any issue.

$ make -skj"$(nproc)" LLVM=1 O=out/x86_64 distclean defconfig bzImage
...

$ ../../cbl/github/boot-utils/boot-qemu.sh -a x86_64 -k out/x86_64 -t 30s |& grep -A1 qemu-system-x86_64
+ timeout --foreground 30s unbuffer qemu-system-x86_64 -cpu host -d unimp,guest_errors -enable-kvm -smp 64 -append 'console=ttyS0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/x86_64/rootfs.cpio -kernel /home/nathan/src/linux/out/x86_64/arch/x86_64/boot/bzImage -m 512m -nodefaults -serial mon:stdio
[    0.000000] Linux version 5.7.0 (nathan@ubuntu-n2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544), LLD 11.0.0 (https://github.com/llvm/llvm-project 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544)) #1 SMP Fri Jun 19 22:02:11 MST 2020

$ make -skj"$(nproc)" KCFLAGS=-march=znver2 LLVM=1 LOCALVERSION=-znver2 O=out/x86_64 distclean defconfig bzImage
...

$ ../../cbl/github/boot-utils/boot-qemu.sh -a x86_64 -k out/x86_64 -t 30s |& grep -A1 qemu-system-x86_64
+ timeout --foreground 30s unbuffer qemu-system-x86_64 -cpu host -d unimp,guest_errors -enable-kvm -smp 64 -append 'console=ttyS0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/x86_64/rootfs.cpio -kernel /home/nathan/src/linux/out/x86_64/arch/x86_64/boot/bzImage -m 512m -nodefaults -serial mon:stdio
[    0.000000] Linux version 5.7.0-znver2 (nathan@ubuntu-n2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544), LLD 11.0.0 (https://github.com/llvm/llvm-project 1cfdda57fa63dd6d770ecb4411bd4d2b59e78544)) #1 SMP Fri Jun 19 22:00:23 MST 2020

[void]

@nathanchance & @nickdesaulniers: I forgot the post scheduler. Try the patch below.

diff --git a/llvm/lib/CodeGen/MachineScheduler.cpp b/llvm/lib/CodeGen/MachineScheduler.cpp
index 0f21c97a30f..e28d25bbfae 100644
--- a/llvm/lib/CodeGen/MachineScheduler.cpp
+++ b/llvm/lib/CodeGen/MachineScheduler.cpp
@@ -443,7 +443,8 @@ static bool isSchedBoundary(MachineBasicBlock::iterator MI,
                             MachineBasicBlock *MBB,
                             MachineFunction *MF,
                             const TargetInstrInfo *TII) {
-  return MI->isCall() || TII->isSchedulingBoundary(*MI, MBB, *MF);
+  return MI->isCall() || MI->getOpcode() == TargetOpcode::INLINEASM_BR ||
+      TII->isSchedulingBoundary(*MI, MBB, *MF);
 }
 
 /// A region of an MBB for scheduling.
diff --git a/llvm/lib/CodeGen/PostRASchedulerList.cpp b/llvm/lib/CodeGen/PostRASchedulerList.cpp
index b85f00a61ea..1a2a1d753cd 100644
--- a/llvm/lib/CodeGen/PostRASchedulerList.cpp
+++ b/llvm/lib/CodeGen/PostRASchedulerList.cpp
@@ -338,7 +338,8 @@ bool PostRAScheduler::runOnMachineFunction(MachineFunction &Fn) {
       // Calls are not scheduling boundaries before register allocation, but
       // post-ra we don't gain anything by scheduling across calls since we
       // don't need to worry about register pressure.
-      if (MI.isCall() || TII->isSchedulingBoundary(MI, &MBB, Fn)) {
+      if (MI.isCall() || MI.getOpcode() == TargetOpcode::INLINEASM_BR ||
+          TII->isSchedulingBoundary(MI, &MBB, Fn)) {
         Scheduler.enterRegion(&MBB, I, Current, CurrentCount - Count);
         Scheduler.setEndIndex(CurrentCount);
         Scheduler.schedule();
diff --git a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
index 61baa31074e..a4447bc15f1 100644
--- a/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
+++ b/llvm/test/CodeGen/X86/callbr-asm-outputs.ll
@@ -41,6 +41,7 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    .cfi_offset %edi, -8
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
+; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    cmpl %edi, %esi
 ; CHECK-NEXT:    jge .LBB1_2
 ; CHECK-NEXT:  # %bb.1: # %if.then
@@ -49,7 +50,6 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    testl %edi, %esi
 ; CHECK-NEXT:    jne .Ltmp1
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    jmp .LBB1_3
 ; CHECK-NEXT:  .LBB1_2: # %if.else
 ; CHECK-NEXT:    #APP
@@ -57,7 +57,6 @@ define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-NEXT:    testl %esi, %edi
 ; CHECK-NEXT:    jne .Ltmp2
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:  .LBB1_3:
 ; CHECK-NEXT:    movl %esi, %eax
 ; CHECK-NEXT:    addl %edi, %eax

[nathanchance]

@void that diff on top of this revision resolves the issue I was seeing, thanks!

+ timeout --foreground 30s unbuffer qemu-system-x86_64 -cpu host -d unimp,guest_errors -enable-kvm -smp 64 -append 'console=ttyS0 ' -display none -initrd /home/nathan/cbl/github/boot-utils/images/x86_64/rootfs.cpio -kernel /home/nathan/src/linux/out/x86_64/arch/x86_64/boot/bzImage -m 512m -nodefaults -serial mon:stdio
[    0.000000] Linux version 5.7.0-znver2 (nathan@ubuntu-n2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (https://github.com/llvm/llvm-project a3c12c2214b98696d2f465d9413d91dfcd02160c), LLD 11.0.0 (https://github.com/llvm/llvm-project a3c12c2214b98696d2f465d9413d91dfcd02160c)) #1 SMP Sun Jun 21 12:00:34 MST 2020

[nickdesaulniers]

In D79794#2105792, @nathanchance wrote:

@void that diff on top of this revision resolves the issue I was seeing, thanks!

That's great! @void is there a test case we can add for that post RA schedule, so that we don't regress that? @jyknight would you mind rolling @void 's changes up into this and rebasing, please? I'm curious if there's anything we can do to combine isSchedBoundary and isSchedulingBoundary?

[void]

In D79794#2107203, @nickdesaulniers wrote:

In D79794#2105792, @nathanchance wrote:

@void that diff on top of this revision resolves the issue I was seeing, thanks!

That's great! @void is there a test case we can add for that post RA schedule, so that we don't regress that? @jyknight would you mind rolling @void 's changes up into this and rebasing, please? I'm curious if there's anything we can do to combine isSchedBoundary and isSchedulingBoundary?

I'm working on getting the test case ready.

In D79794#2107203, @nickdesaulniers wrote:

In D79794#2105792, @nathanchance wrote:

@void that diff on top of this revision resolves the issue I was seeing, thanks!

That's great! @void is there a test case we can add for that post RA schedule, so that we don't regress that? @jyknight would you mind rolling @void 's changes up into this and rebasing, please? I'm curious if there's anything we can do to combine isSchedBoundary and isSchedulingBoundary?

Here's the testcase:

$ more llvm/test/CodeGen/X86/callbr-asm-instr-scheduling.ll
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -mcpu=znver2 -O2 -frame-pointer=none < %s | FileCheck %s

; Make sure that instructions aren't scheduled after the "callbr". In the
; example below, we don't want the "shrxq" through "leaq" instructions to be
; moved after the "callbr".

%struct.cpuinfo_x86 = type { i8, i8, i8, i8, i32, [3 x i32], i8, i8, i8, i8, i32, i32, %union.anon.83, [16 x i8], [64 x i8], i32, i32, i32, i32, i32, i32, i64, i16, i16, i16, i16, i16, i16, i16, i16, i16, i16, i16, i32, i8, i8 }
%union.anon.83 = type { i64, [72 x i8] }
%struct.pgd_t = type { i64 }
%struct.p4d_t = type { i64 }
%struct.pud_t = type { i64 }

@boot_cpu_data = external dso_local global %struct.cpuinfo_x86, align 8
@page_offset_base = external dso_local local_unnamed_addr global i64, align 8
@pgdir_shift = external dso_local local_unnamed_addr global i32, align 4
@__force_order = external dso_local global i64, align 8
@ptrs_per_p4d = external dso_local local_unnamed_addr global i32, align 4

define i64 @early_ioremap_pmd(i64 %addr) {
; CHECK-LABEL: early_ioremap_pmd:
; CHECK:       # %bb.0: # %entry
; CHECK-NEXT:    #APP
; CHECK-NEXT:    movq %cr3, %rax
; CHECK-EMPTY:
; CHECK-NEXT:    #NO_APP
; CHECK-NEXT:    movabsq $9223372036854771712, %rdx # imm = 0x7FFFFFFFFFFFF000
; CHECK-NEXT:    andq %rax, %rdx
; CHECK-NEXT:    movb {{.*}}(%rip), %al
; CHECK-NEXT:    movq {{.*}}(%rip), %rcx
; CHECK-NEXT:    shrxq %rax, %rdi, %rax
; CHECK-NEXT:    addq %rcx, %rdx
; CHECK-NEXT:    andl $511, %eax # imm = 0x1FF
; CHECK-NEXT:    leaq (%rdx,%rax,8), %rax
; CHECK-NEXT:    #APP
; CHECK-NEXT:  .Ltmp2:
; CHECK-NEXT:    jmp .Ltmp3
; CHECK-NEXT:  .Ltmp4:
; CHECK-NEXT:    .zero (-(((.Ltmp5-.Ltmp6)-(.Ltmp4-.Ltmp2))>0))*((.Ltmp5-.Ltmp6)-(.Ltmp4-.Ltmp2)),144
; CHECK-NEXT:  .Ltmp7:
entry:
  %0 = tail call i64 asm sideeffect "mov %cr3,$0\0A\09", "=r,=*m,~{dirflag},~{fpsr},~{flags}"(i64* nonnull @__force_order)
  %and.i = and i64 %0, 9223372036854771712
  %1 = load i64, i64* @page_offset_base, align 8
  %add = add i64 %and.i, %1
  %2 = inttoptr i64 %add to %struct.pgd_t*
  %3 = load i32, i32* @pgdir_shift, align 4
  %sh_prom = zext i32 %3 to i64
  %shr = lshr i64 %addr, %sh_prom
  %and = and i64 %shr, 511
  %arrayidx = getelementptr %struct.pgd_t, %struct.pgd_t* %2, i64 %and
  callbr void asm sideeffect "1: jmp 6f\0A2:\0A.skip -(((5f-4f) - (2b-1b)) > 0) * ((5f-4f) - (2b-1b)),0x90\0A3:\0A.section .altinstructions,\22a\22\0A .long 1b - .\0A .long 4f - .\0A .word ${1:P}\0A .byte 3b - 1b\0A .byte 5f - 4f\0
A .byte 3b - 2b\0A.previous\0A.section .altinstr_replacement,\22ax\22\0A4: jmp ${5:l}\0A5:\0A.previous\0A.section .altinstructions,\22a\22\0A .long 1b - .\0A .long 0\0A .word ${0:P}\0A .byte 3b - 1b\0A .byte 0\0A .byte 0\0A.previou
s\0A.section .altinstr_aux,\22ax\22\0A6:\0A testb $2,$3\0A jnz ${4:l}\0A jmp ${5:l}\0A.previous\0A", "i,i,i,*m,X,X,~{dirflag},~{fpsr},~{flags}"(i16 528, i32 117, i32 1, i8* getelementptr inbounds (%struct.cpuinfo_x86, %struct.cpuin
fo_x86* @boot_cpu_data, i64 0, i32 12, i32 1, i64 58), i8* blockaddress(@early_ioremap_pmd, %if.end.i), i8* blockaddress(@early_ioremap_pmd, %if.then.i))
          to label %_static_cpu_has.exit.thread.i [label %if.end.i, label %if.then.i]

_static_cpu_has.exit.thread.i:                    ; preds = %entry
  br label %if.end.i

if.then.i:                                        ; preds = %entry
  %4 = bitcast %struct.pgd_t* %arrayidx to %struct.p4d_t*
  br label %p4d_offset.exit

if.end.i:                                         ; preds = %_static_cpu_has.exit.thread.i, %entry
  %coerce.dive.i = getelementptr inbounds %struct.pgd_t, %struct.pgd_t* %arrayidx, i64 0, i32 0
  %5 = load i64, i64* %coerce.dive.i, align 8
  %6 = inttoptr i64 %5 to %struct.p4d_t*
  %7 = load i32, i32* @ptrs_per_p4d, align 4
  %sub.i.i = add i32 %7, 33554431
  %8 = and i32 %sub.i.i, 33554431
  %and.i1.i = zext i32 %8 to i64
  %add.ptr.i = getelementptr %struct.p4d_t, %struct.p4d_t* %6, i64 %and.i1.i
  br label %p4d_offset.exit

p4d_offset.exit:                                  ; preds = %if.end.i, %if.then.i
  %retval.0.i = phi %struct.p4d_t* [ %add.ptr.i, %if.end.i ], [ %4, %if.then.i ]
  %coerce.dive.i12 = getelementptr inbounds %struct.p4d_t, %struct.p4d_t* %retval.0.i, i64 0, i32 0
  %9 = load i64, i64* %coerce.dive.i12, align 8
  %and.i.i13 = and i64 %9, 4503599627366400
  %add.i.i14 = add i64 %and.i.i13, %1
  %10 = inttoptr i64 %add.i.i14 to %struct.pud_t*
  %coerce.dive.i16 = getelementptr %struct.pud_t, %struct.pud_t* %10, i64 511, i32 0
  %11 = load i64, i64* %coerce.dive.i16, align 8
  %tobool.i.i.i = icmp slt i64 %11, 0
  %..i.i.i = select i1 %tobool.i.i.i, i64 4503598553628672, i64 4503599627366400
  ret i64 %..i.i.i
}

[jyknight]

Differences in new patch, since previous one:

• Folded in void's change (but I made it in the isSchedulingBoundary() functions, rather than the two callers.
• Marked the INLINEASM_BR instruction as always having unmodeled side-effects, rather than only if it's marked thus. (not to fix any known miscompilation -- clang always marked it as such anyhow.)
• Removed FIXME comments, per review comments.

[jyknight]

After working on the previously mentioned unreachable-block-end cleanup for a bit (not done yet), and thinking about this patch more, I actually think it's okay to go ahead with this change now. The potential problem I was worried about is that the code may think that if you have INLINEASM_BR in a basic-block, where the block-end is unreachable, and the block is followed by a basic-block which is one of the indirect targets of the INLINEASM_BR, the code will think the first block falls through, when it does not.

I'm no longer concerned about this now, for two reasons:

1. I currently _suspect_ it's not actually possible for this to happen, because of the way we restrict handling/merging blocks. It will never be created that way (because callbr is a terminator in IR), and I currently believe we'll never merge the blocks to create that situation, because inlineasm_br block has multiple successors. I am not certain it can never happen, but at least I've not been able to induce it to occur. (This relates to another worry I had previously -- that we might end up with a block that has both a Call instruction that could throw, and an inlineasm_br. Or, for that matter, two inlineasm_br in the same MachineBasicBlock. But, even though it's not a terminator in MI, I believe this will not occur.)
2. Even if there _is_ some way that might happen, the badness that might occur is limited. Assuming there's a possible fallthrough, when it is in fact impossible, and the block is only an exceptional successor, seems not likely be a problem. We may insert an extraneous jump upon block rearrangement, for example, but that would not be truly problematic.

So, I think this is ready for another round of review and then to submit.

Thanks!

llvm/include/llvm/CodeGen/MachineBasicBlock.h

483

(we do actually have MachineBasicBlock::hasAddressTaken().)

I believe the change I have here resolves efriedma's concerns you've linked, by removing the mapping of source-block to target-block. It now overestimates, but should be conservatively correct, even in the face of instructions being moved/split/etc, because the _target_ cannot be so transformed.

llvm/lib/CodeGen/MachineBasicBlock.cpp

281–284

I'm worried that iterating over all the instructions in the block to retrieve this information would be bad for performance.

I don't think the overestimating here is likely to be a major optimization issue, but if it turns out to be, I'd like to revisit in a future patch.

289

I think it's correct -- in that it returns false strictly more often than it needs to. Just the same as we could sometimes hoist into a block which has an invoke in it, we could sometimes do so for an inlineasm_br.

llvm/lib/CodeGen/SplitKit.cpp

101

The code below which is invoked when LIP.second is set is simply to decide which of LIP.first or LIP.second to return. Letting it take the !LIP.second codepath has the same effect, with marginally less work.

llvm/lib/Target/Hexagon/BitTracker.cpp

958

Thanks for the explanation.

llvm/test/CodeGen/ARM/ifcvt-diamond-unanalyzable-common.mir

66

I changed the test to a different mechanism of testing unanalyzable branch sequences, since INLINEASM_BR doesn't trigger the problem anymore.

[void]

Thanks! :-)

[nickdesaulniers]

The only other concern I have is whether we have enough test coverage of the scheduling boundary changes? (Does removing the added checks there cause any existing or new test from the change to file? If not, seems like a lack of test coverage.) In the case of PostRASchedulerList, that implies a MIR test (writing MIR tests isn't something I'd wish on my worst enemy though).

llvm/include/llvm/CodeGen/MachineBasicBlock.h

474

Sorry to bikeshed the name of this method, but I find it currently doesn't match the implementation well.

I'd take hasInlineAsmBr to mean "this MachineBasicBlock has an INLINEASM_BR MCInst." Instead, the implementation is checking whether any of the successors of this MachineBasicBlock is the indirect target of an INLINEASM_BR MCInst. Those two aren't the same thing. In fact, you could have a MachineBasicBlock where the current implementation of hasInlineAsmBR would return true, and yet the MachineBasicBlock does not actually contain a INLINEASM_BR MCInst.

I know that's what you're alluding to in the comment, but I think the method should be named hasInlineAsmBrSuccessors or something of the sort. Maybe MaybeHasInlineAsmBr since it sounds like you'd rather it not be precise by scanning all instructions for the relevant opcode? (Couldn't ISEL mark these when creating the MBB? I guess having a bool member is tricky, since transforms would have to update that correctly. In this case, rematerializing the value by rescanning is simpler, and harder to get wrong.)

I guess the name sounds precise, though the comment and implementation denote that it's not, and that's what I'm most hung up on.

llvm/lib/CodeGen/BranchFolding.cpp

1711

was removing the !isEHPad check intentional?

llvm/lib/CodeGen/MachineBasicBlock.cpp

281–284

This could at least use a range-for:

for (const MachineBasicBlock *Succ : successors()) {
  if (Succ->isInlineAsmBrIndirectTarget())
    ...

llvm/lib/CodeGen/MachineVerifier.cpp

588

should the report string be updated, too, to mention INLINEASM_BR fallthrough/default target?

llvm/lib/CodeGen/SplitKit.cpp

109–110

I wonder why we don't use a reverse const iterator here?

llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp

2020

I think there's one more override of this method that was missed: AArch64InstrInfo::isSchedulingBoundary in llvm/lib/Target/AArch64/AArch64InstrInfo.cpp? Ah, nvm it defers to TargetInstrInfo::isSchedulingBoundary first.

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

33

@void 's earlier comments mentioned modifications to this test case's movl $-1, %eax. Are those still needed?

[void]

In D79794#2121018, @nickdesaulniers wrote:

The only other concern I have is whether we have enough test coverage of the scheduling boundary changes? (Does removing the added checks there cause any existing or new test from the change to file? If not, seems like a lack of test coverage.) In the case of PostRASchedulerList, that implies a MIR test (writing MIR tests isn't something I'd wish on my worst enemy though).

The reason it wasn't failing before was because INLINEASM_BR was a terminator, and the boundary check automatically caught it. The existing test, llvm/test/CodeGen/X86/callbr-asm-outputs.ll, did fail though. So at least we're covered there.

Don't get me wrong, I'm always up for adding more tests, but I think this change is okay in that regard. :-)

[void]

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

33

No, those are taken care of by the scheduling boundary check.

[jyknight]

llvm/include/llvm/CodeGen/MachineBasicBlock.h

474

Renamed to mayHaveInlineAsmBr. I don't like "hasInlineAsmBrSuccessors", because the important property is whether there is such an instruction in the block. That it currently tests via looking at the successor in the current implementation is just an implementation detail.

It would be nice for it to be precise, but as I mentioned in the other thread, it's probably not really worth the expense.

llvm/lib/CodeGen/BranchFolding.cpp

1711

Yes, no longer need to filter out EHpads, because it's now only looking at the analyzeBranch results CurFBB/CurTBB, not all successors.

llvm/lib/CodeGen/MachineVerifier.cpp

588

Maybe it should've before, but this change _removed_ the allowance for that.

llvm/lib/CodeGen/SplitKit.cpp

109–110

Who knows. =) Changed.

[jyknight]

Minor fixups.

[MatzeB]

So effectively we have an extended basic-block model in the LLVM backend now (blocks have 1 entry, but multiple exits that can be in the middle of the block). I know this was already the case for exception control-flow, but this here cemented it further. With that not being documented anywhere, and being different from LLVM IR/middleend, we are probably asking for trouble...

[MatzeB]

See also: https://bugs.llvm.org/show_bug.cgi?id=27659

We used to consider the the invoke mechanism more of a bug/misdesign. But I guess without anyone cleaning this up it was just a matter of time for other uses like this to do the same :-(

[void]

In D79794#3040491, @MatzeB wrote:

So effectively we have an extended basic-block model in the LLVM backend now (blocks have 1 entry, but multiple exits that can be in the middle of the block). I know this was already the case for exception control-flow, but this here cemented it further. With that not being documented anywhere, and being different from LLVM IR/middleend, we are probably asking for trouble...

We took great pains to ensure that nothing could be moved past it, so that the semantics won't change and we can reload registers, etc.. Not being able to 100% model the control flow of an assembly block already existed with current assembly blocks. There's no guarantee that a given assembly block won't have branches, or loops, or calls, or exits. None of which is modeled in MIR.

[MatzeB]

We took great pains to ensure that nothing could be moved past it, so that the semantics won't change and we can reload registers, etc..

Yes, I see that this is going through great pain to ensure things still work. But this all feels to me like you are making a form of EBBs work here. There is a way to jump out of the basic block without having all instructions in it executed. There must be some cases where COPYs or SPILLs insructions end up after the branch now which brings us into this trouble.

• I realize that this was painful to model since LLVM was never prepared to have terminator instructions produce values. I guess we lack the time+expertise to fix this after the fact now :-(
• The design was already messed up before INLINEASM_BR for exception handling.

So I don't know what to do here except rant and predict random bugs popping up over the years because intuition about how basic blocks work is broken.

Not being able to 100% model the control flow of an assembly block already existed with current assembly blocks. There's no guarantee that a given assembly block won't have branches, or loops, or calls, or exits. None of which is modeled in MIR.

Control flow within an assembly instruction is not a problem; Similar to a CALL instruction can also trigger arbitrary control flow, we can ignore that in the modeling since we have a guarnatee that control continues after the instruction eventually.

[nickdesaulniers]

In D79794#3040927, @MatzeB wrote:

We took great pains to ensure that nothing could be moved past it, so that the semantics won't change and we can reload registers, etc..

Yes, I see that this is going through great pain to ensure things still work. But this all feels to me like you are making a form of EBBs work here. There is a way to jump out of the basic block without having all instructions in it executed. There must be some cases where COPYs or SPILLs insructions end up after the branch now which brings us into this trouble.

• I realize that this was painful to model since LLVM was never prepared to have terminator instructions produce values. I guess we lack the time+expertise to fix this after the fact now :-(
• The design was already messed up before INLINEASM_BR for exception handling.

So I don't know what to do here except rant and predict random bugs popping up over the years because intuition about how basic blocks work is broken.

Any thoughts on how to fix this without regressing support for asm goto w/ outputs?

[void]

In D79794#3040927, @MatzeB wrote:

We took great pains to ensure that nothing could be moved past it, so that the semantics won't change and we can reload registers, etc..

Yes, I see that this is going through great pain to ensure things still work. But this all feels to me like you are making a form of EBBs work here. There is a way to jump out of the basic block without having all instructions in it executed. There must be some cases where COPYs or SPILLs insructions end up after the branch now which brings us into this trouble.

This is a general issue with "asm goto", not just with "asm goto with outputs". Someone could stomp on registers, memory, and the like and leap to BFE without the necessary COPY / SPILL instructions to clean things up. We skirt around some of these issues by saying that outputs aren't valid on the indirect branch.

There's a reason why "asm goto" wasn't implemented in LLVM until we were forced to do it...

• I realize that this was painful to model since LLVM was never prepared to have terminator instructions produce values. I guess we lack the time+expertise to fix this after the fact now :-(
• The design was already messed up before INLINEASM_BR for exception handling.

So I don't know what to do here except rant and predict random bugs popping up over the years because intuition about how basic blocks work is broken.

One method to perhaps address your concerns would be to create a "terminating" COPY instruction. This allows one to reload registers after the ASM block. There were some sticky issues with it. Going this method was much easier. (Yes, I know that's not the best reason to do something, but...) However, the only issue a hypothetical terminating COPY would solve is issues with instructions invalidly moving past the INLINEASM_BR.

I would *love* to fix this. Chris claims that MLIR would solve all of this. (nudge...nudge) :-)

[MatzeB]

This is a general issue with "asm goto", not just with "asm goto with outputs". Someone could stomp on registers, memory, and the like and leap to BFE without the necessary COPY / SPILL instructions to clean things up. We skirt around some of these issues by saying that outputs aren't valid on the indirect branch.

Not sure I follow here... if someone stomps on registers in inline-asm (without declaring that in the inputs/outputs) then it's obviously a bug in the input and not something the compiler can fix.

Any thoughts on how to fix this without regressing support for asm goto w/ outputs?

For the record: I am not advocating to revert any patches at this point; I am a obviously a year to late for the review and despite the asm-goto the same problem exists for exception control too. I just feel that we should be aware of what we are doing and call things by their name :)

So unless I am missing something here the whole problem is about terminator instructions producing values. Correct me if I am missing some other aspect of the discussion.

So about supporting outputs in terminator instructions: The tricky part is that regalloc typically expects to be able to place things behind the definition of a value (typically spills or COPYs). The reason this is tricky, is that for a terminator you are forced to place those things into the successor block(s) instead. Placing the operations into the successor blocks in turn can be tricky if the successors have multiple predecessor ("critical edges") because then we can end up executing the instructions wrongfully when coming from the other predecessors. So that would require to either be in a position to split the critical edges as needed or have the register allocator be robust enough to deal with spills/COPYs being executed in more situations than desired in case of critical edges. If I understand the asm-goto construct correctly, then the asm-labels have enough abstraction that breaking critical edges is always possible (you are forced to list all possible targets in the ASM instructions and the compiler could replace 1 block for another, and cannot jump to arbitrary computed destinations like asm-goto, right?).

[MatzeB]

One method to perhaps address your concerns would be to create a "terminating" COPY instruction. This allows one to reload registers after the ASM block. There were some sticky issues with it. Going this method was much easier. (Yes, I know that's not the best reason to do something, but...) However, the only issue a hypothetical terminating COPY would solve is issues with instructions invalidly moving past the INLINEASM_BR.

But that still would mean we end up with terminator instructions that assign registers, so we're still in this regalloc grey-area because that is technically not supported.

[void]

In D79794#3041059, @MatzeB wrote:

One method to perhaps address your concerns would be to create a "terminating" COPY instruction. This allows one to reload registers after the ASM block. There were some sticky issues with it. Going this method was much easier. (Yes, I know that's not the best reason to do something, but...) However, the only issue a hypothetical terminating COPY would solve is issues with instructions invalidly moving past the INLINEASM_BR.

But that still would mean we end up with terminator instructions that assign registers, so we're still in this regalloc grey-area because that is technically not supported.

Yup! You identified the issue we faced. :-)

[jyknight]

The design was already messed up before INLINEASM_BR for exception handling.

To me, this is the main point.

Given the existing handling for invoke, I still believe that using the same underlying mechanisms for inlineasm_br was the best implementation choice, because inlineasm_br behaves almost exactly like an invoke with a user-defined custom calling convention.

If you can come up with a proposal to handle invoke differently, then I expect inlineasm_br should fit easily into that.

llvm/include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
   ///     ... however many operands ...
   ///   Operand #last: Optional, an incoming flag.
   ///
   /// The variable width operands are required to represent target addressing
   /// modes as a single "operand", even though they may have multiple
   /// SDOperands.
   INLINEASM,
 
-  /// INLINEASM_BR - Terminator version of inline asm. Used by asm-goto.
+  /// INLINEASM_BR - Branching version of inline asm. Used by asm-goto.
   INLINEASM_BR,
 
   /// EH_LABEL - Represents a label in mid basic block used to track
   /// locations needed for debug and exception handling tables.  These nodes
   /// take a chain as input and return a chain.
   EH_LABEL,
 
   /// ANNOTATION_LABEL - Represents a mid basic block label used by

llvm/include/llvm/CodeGen/MachineBasicBlock.h

private:
   MBBSectionID SectionID{0};
 
   // Indicate that this basic block begins a section.
   bool IsBeginSection = false;
 
   // Indicate that this basic block ends a section.
   bool IsEndSection = false;
 
-  /// Default target of the callbr of a basic block.
-  bool InlineAsmBrDefaultTarget = false;
-
-  /// List of indirect targets of the callbr of a basic block.
-  SmallPtrSet<const MachineBasicBlock*, 4> InlineAsmBrIndirectTargets;
+  /// Indicate that this basic block is the indirect dest of an INLINEASM_BR.
+  bool IsInlineAsmBrIndirectTarget = false;
 
   /// since getSymbol is a relatively heavy-weight operation, the symbol
   /// is only computed once and is cached.
   mutable MCSymbol *CachedMCSymbol = nullptr;
 
   /// Used during basic block sections to mark the end of a basic block.
   MCSymbol *EndMCSymbol = nullptr;
 
#endif
   void setIsEndSection(bool V = true) { IsEndSection = V; }
 
   /// Returns the section ID of this basic block.
   MBBSectionID getSectionID() const { return SectionID; }
 
   /// Sets the section ID for this basic block.
   void setSectionID(MBBSectionID V) { SectionID = V; }
 
+  /// Returns true if this block may have an INLINEASM_BR (overestimate, by
+  /// checking if any of the successors are indirect targets of any inlineasm_br
+  /// in the function).
+  bool mayHaveInlineAsmBr() const;
+
   /// Returns true if this is the indirect dest of an INLINEASM_BR.
-  bool isInlineAsmBrIndirectTarget(const MachineBasicBlock *Tgt) const {
-    return InlineAsmBrIndirectTargets.count(Tgt);
+  bool isInlineAsmBrIndirectTarget() const {
+    return IsInlineAsmBrIndirectTarget;
   }
 
   /// Indicates if this is the indirect dest of an INLINEASM_BR.
-  void addInlineAsmBrIndirectTarget(const MachineBasicBlock *Tgt) {
-    InlineAsmBrIndirectTargets.insert(Tgt);
-  }
-
-  /// Transfers indirect targets to INLINEASM_BR's copy block.
-  void transferInlineAsmBrIndirectTargets(MachineBasicBlock *CopyBB) {
-    for (auto *Target : InlineAsmBrIndirectTargets)
-      CopyBB->addInlineAsmBrIndirectTarget(Target);
-    return InlineAsmBrIndirectTargets.clear();
-  }
-
-  /// Returns true if this is the default dest of an INLINEASM_BR.
-  bool isInlineAsmBrDefaultTarget() const {
-    return InlineAsmBrDefaultTarget;
-  }
-
-  /// Indicates if this is the default deft of an INLINEASM_BR.
-  void setInlineAsmBrDefaultTarget() {
-    InlineAsmBrDefaultTarget = true;
+  void setIsInlineAsmBrIndirectTarget(bool V = true) {
+    IsInlineAsmBrIndirectTarget = V;
   }
 
   /// Returns true if it is legal to hoist instructions into this block.
   bool isLegalToHoistInto() const;
 
   // Code Layout methods.
 
   /// Move 'this' block before or after the specified block.  This only moves

llvm/include/llvm/Target/Target.td

def INLINEASM : StandardPseudoInstruction {
   let InOperandList = (ins variable_ops);
   let AsmString = "";
   let hasSideEffects = 0;  // Note side effect is encoded in an operand.
 }
 def INLINEASM_BR : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins variable_ops);
   let AsmString = "";
-  let hasSideEffects = 0;  // Note side effect is encoded in an operand.
-  let isTerminator = 1;
-  let isBranch = 1;
-  let isIndirectBranch = 1;
+  // Unlike INLINEASM, this is always treated as having side-effects.
+  let hasSideEffects = 1;
+  // Despite potentially branching, this instruction is intentionally _not_
+  // marked as a terminator or a branch.
 }
 def CFI_INSTRUCTION : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "";
   let hasCtrlDep = 1;
   let hasSideEffects = 0;
   let isNotDuplicable = 1;

llvm/lib/CodeGen/BranchFolding.cpp

for (MachineBasicBlock *PBB : I->predecessors()) {
       // Skip blocks that loop to themselves, can't tail merge these.
       if (PBB == IBB)
         continue;
 
       // Visit each predecessor only once.
       if (!UniquePreds.insert(PBB).second)
         continue;
 
-      // Skip blocks which may jump to a landing pad. Can't tail merge these.
-      if (PBB->hasEHPadSuccessor())
+      // Skip blocks which may jump to a landing pad or jump from an asm blob.
+      // Can't tail merge these.
+      if (PBB->hasEHPadSuccessor() || PBB->mayHaveInlineAsmBr())
         continue;
 
       // After block placement, only consider predecessors that belong to the
       // same loop as IBB.  The reason is the same as above when skipping loop
       // header.
       if (AfterBlockPlacement && MLI)
         if (ML != MLI->getLoopFor(PBB))
           continue;
ReoptimizeBlock:
   // place to move this block where a fall-through will happen.
   if (!PrevBB.canFallThrough()) {
     // Now we know that there was no fall-through into this block, check to
     // see if it has a fall-through into its successor.
     bool CurFallsThru = MBB->canFallThrough();
 
     if (!MBB->isEHPad()) {
       // Check all the predecessors of this block.  If one of them has no fall
-      // throughs, move this block right after it.
+      // throughs, and analyzeBranch thinks it _could_ fallthrough to this
+      // block, move this block right after it.
       for (MachineBasicBlock *PredBB : MBB->predecessors()) {
         // Analyze the branch at the end of the pred.
         MachineBasicBlock *PredTBB = nullptr, *PredFBB = nullptr;
         SmallVector<MachineOperand, 4> PredCond;
         if (PredBB != MBB && !PredBB->canFallThrough() &&
             !TII->analyzeBranch(*PredBB, PredTBB, PredFBB, PredCond, true) &&
+            (PredTBB == MBB || PredFBB == MBB) &&
             (!CurFallsThru || !CurTBB || !CurFBB) &&
             (!CurFallsThru || MBB->getNumber() >= PredBB->getNumber())) {
           // If the current block doesn't fall through, just move it.
           // If the current block can fall through and does not end with a
           // conditional branch, we need to append an unconditional jump to
           // the (current) next block.  To avoid a possible compile-time
           // infinite loop, move blocks only backward in this case.
           // Also, if there are already 2 branches here, we cannot add a third;
if (!MBB->isEHPad()) {
           MBB->moveAfter(PredBB);
           MadeChange = true;
           goto ReoptimizeBlock;
         }
       }
     }
 
     if (!CurFallsThru) {
-      // Check all successors to see if we can move this block before it.
-      for (MachineBasicBlock *SuccBB : MBB->successors()) {
+      // Check analyzable branch-successors to see if we can move this block
+      // before one.
+      if (!CurUnAnalyzable) {
+        for (MachineBasicBlock *SuccBB : {CurFBB, CurTBB}) {
+          if (!SuccBB)
+            continue;
           // Analyze the branch at the end of the block before the succ.
           MachineFunction::iterator SuccPrev = --SuccBB->getIterator();
 
-        // If this block doesn't already fall-through to that successor, and if
-        // the succ doesn't already have a block that can fall through into it,
-        // and if the successor isn't an EH destination, we can arrange for the
-        // fallthrough to happen.
+          // If this block doesn't already fall-through to that successor, and
+          // if the succ doesn't already have a block that can fall through into
+          // it, we can arrange for the fallthrough to happen.
           if (SuccBB != MBB && &*SuccPrev != MBB &&
-            !SuccPrev->canFallThrough() && !CurUnAnalyzable &&
-            !SuccBB->isEHPad()) {
+              !SuccPrev->canFallThrough()) {
             MBB->moveBefore(SuccBB);
             MadeChange = true;
             goto ReoptimizeBlock;
           }
         }
+      }
 
       // Okay, there is no really great place to put this block.  If, however,
       // the block before this one would be a fall-through if this block were
       // removed, move this block to the end of the function. There is no real
       // advantage in "falling through" to an EH block, so we don't want to
       // perform this transformation for that case.
       //
       // Also, Windows EH introduced the possibility of an arbitrary number of

llvm/lib/CodeGen/MachineBasicBlock.cpp

 }
 
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void MachineBasicBlock::dump() const {
   print(dbgs());
 }
 #endif
 
+bool MachineBasicBlock::mayHaveInlineAsmBr() const {
+  for (const MachineBasicBlock *Succ : successors()) {
+    if (Succ->isInlineAsmBrIndirectTarget())
+      return true;
+  }
+  return false;
+}
+
 bool MachineBasicBlock::isLegalToHoistInto() const {
-  if (isReturnBlock() || hasEHPadSuccessor())
+  if (isReturnBlock() || hasEHPadSuccessor() || mayHaveInlineAsmBr())
     return false;
   return true;
 }
 
 StringRef MachineBasicBlock::getName() const {
   if (const BasicBlock *LBB = getBasicBlock())
     return LBB->getName();
   else
bool MachineBasicBlock::canSplitCriticalEdge(
     const MachineBasicBlock *Succ) const {
   // Splitting the critical edge to a landing pad block is non-trivial. Don't do
   // it in this generic function.
   if (Succ->isEHPad())
     return false;
 
   // Splitting the critical edge to a callbr's indirect block isn't advised.
   // Don't do it in this generic function.
-  if (isInlineAsmBrIndirectTarget(Succ))
+  if (Succ->isInlineAsmBrIndirectTarget())
     return false;
 
   const MachineFunction *MF = getParent();
   // Performance might be harmed on HW that implements branching using exec mask
   // where both sides of the branches are always executed.
   if (MF->getTarget().requiresStructuredCFG())
     return false;
 

llvm/lib/CodeGen/MachineSink.cpp

MachineSinking::FindSuccToSinkTo(MachineInstr &MI, MachineBasicBlock *MBB,
   if (MBB == SuccToSinkTo)
     return nullptr;
 
   // It's not safe to sink instructions to EH landing pad. Control flow into
   // landing pad is implicitly defined.
   if (SuccToSinkTo && SuccToSinkTo->isEHPad())
     return nullptr;
 
+  // It ought to be okay to sink instructions into an INLINEASM_BR target, but
+  // only if we make sure that MI occurs _before_ an INLINEASM_BR instruction in
+  // the source block (which this code does not yet do). So for now, forbid
+  // doing so.
+  if (SuccToSinkTo && SuccToSinkTo->isInlineAsmBrIndirectTarget())
+    return nullptr;
+
   return SuccToSinkTo;
 }
 
 /// Return true if MI is likely to be usable as a memory operation by the
 /// implicit null check optimization.
 ///
 /// This is a "best effort" heuristic, and should not be relied upon for
 /// correctness.  This returning true does not guarantee that the implicit null

llvm/lib/CodeGen/MachineVerifier.cpp

MachineVerifier::visitMachineBasicBlockBefore(const MachineBasicBlock *MBB) {
   FirstNonPHI = nullptr;
 
   if (!MF->getProperties().hasProperty(
       MachineFunctionProperties::Property::NoPHIs) && MRI->tracksLiveness()) {
     // If this block has allocatable physical registers live-in, check that
     // it is an entry block or landing pad.
     for (const auto &LI : MBB->liveins()) {
       if (isAllocatable(LI.PhysReg) && !MBB->isEHPad() &&
-          !MBB->isInlineAsmBrDefaultTarget() &&
           MBB->getIterator() != MBB->getParent()->begin()) {
         report("MBB has allocatable live-in, but isn't entry or landing-pad.", MBB);
         report_context(LI.PhysReg);
       }
     }
   }
 
   // Count the number of landing pad successors.
   SmallPtrSet<const MachineBasicBlock*, 4> LandingPadSuccs;
   for (const auto *succ : MBB->successors()) {
for (const MachineBasicBlock *SuccMBB : MBB->successors()) {
       if (SuccMBB == TBB || SuccMBB == FBB)
         continue;
       // If we might have a fallthrough, and the successor is the fallthrough
       // block, that's also ok.
       if (Fallthrough && SuccMBB == MBB->getNextNode())
         continue;
       // Also accept successors which are for exception-handling or might be
       // inlineasm_br targets.
-      if (SuccMBB->isEHPad() || MBB->isInlineAsmBrIndirectTarget(SuccMBB))
+      if (SuccMBB->isEHPad() || SuccMBB->isInlineAsmBrIndirectTarget())
         continue;
       report("MBB has unexpected successors which are not branch targets, "
              "fallthrough, EHPads, or inlineasm_br targets.",
              MBB);
     }
   }
 
   regsLive.clear();

llvm/lib/CodeGen/PHIEliminationUtils.cpp

 llvm::findPHICopyInsertPoint(MachineBasicBlock* MBB, MachineBasicBlock* SuccMBB,
                              unsigned SrcReg) {
   // Handle the trivial case trivially.
   if (MBB->empty())
     return MBB->begin();
 
   // Usually, we just want to insert the copy before the first terminator
   // instruction. However, for the edge going to a landing pad, we must insert
-  // the copy before the call/invoke instruction.
-  if (!SuccMBB->isEHPad())
+  // the copy before the call/invoke instruction. Similarly for an INLINEASM_BR
+  // going to an indirect target.
+  if (!SuccMBB->isEHPad() && !SuccMBB->isInlineAsmBrIndirectTarget())
     return MBB->getFirstTerminator();
 
   // Discover any defs/uses in this basic block.
   SmallPtrSet<MachineInstr*, 8> DefUsesInMBB;
   MachineRegisterInfo& MRI = MBB->getParent()->getRegInfo();
   for (MachineInstr &RI : MRI.reg_instructions(SrcReg)) {
     if (RI.getParent() == MBB)
       DefUsesInMBB.insert(&RI);

llvm/lib/CodeGen/RegisterCoalescer.cpp

 /// the movement of copy is beneficial.
 bool RegisterCoalescer::removePartialRedundancy(const CoalescerPair &CP,
                                                 MachineInstr &CopyMI) {
   assert(!CP.isPhys());
   if (!CopyMI.isFullCopy())
     return false;
 
   MachineBasicBlock &MBB = *CopyMI.getParent();
-  if (MBB.isEHPad())
+  // If this block is the target of an invoke/inlineasm_br, moving the copy into
+  // the predecessor is tricker, and we don't handle it.
+  if (MBB.isEHPad() || MBB.isInlineAsmBrIndirectTarget())
     return false;
 
   if (MBB.pred_size() != 2)
     return false;
 
   LiveInterval &IntA =
       LIS->getInterval(CP.isFlipped() ? CP.getDstReg() : CP.getSrcReg());
   LiveInterval &IntB =

llvm/lib/CodeGen/SelectionDAG/ScheduleDAGSDNodes.cpp

for (const auto &InstrOrder : Orders) {
       }
       if (DLI == DLE)
         break;
 
       LastOrder = Order;
     }
   }
 
-  // Split after an INLINEASM_BR block with outputs. This allows us to keep the
-  // copy to/from register instructions from being between two terminator
-  // instructions, which causes the machine instruction verifier agita.
-  auto TI = llvm::find_if(*BB, [](const MachineInstr &MI){
-    return MI.getOpcode() == TargetOpcode::INLINEASM_BR;
-  });
-  auto SplicePt = TI != BB->end() ? std::next(TI) : BB->end();
-  if (TI != BB->end() && SplicePt != BB->end() &&
-      TI->getOpcode() == TargetOpcode::INLINEASM_BR &&
-      SplicePt->getOpcode() == TargetOpcode::COPY) {
-    MachineBasicBlock *FallThrough = BB->getFallThrough();
-    if (!FallThrough)
-      for (const MachineOperand &MO : BB->back().operands())
-        if (MO.isMBB()) {
-          FallThrough = MO.getMBB();
-          break;
-        }
-    assert(FallThrough && "Cannot find default dest block for callbr!");
-
-    MachineBasicBlock *CopyBB = MF.CreateMachineBasicBlock(BB->getBasicBlock());
-    MachineFunction::iterator BBI(*BB);
-    MF.insert(++BBI, CopyBB);
-
-    CopyBB->splice(CopyBB->begin(), BB, SplicePt, BB->end());
-    CopyBB->setInlineAsmBrDefaultTarget();
-
-    CopyBB->addSuccessor(FallThrough, BranchProbability::getOne());
-    BB->removeSuccessor(FallThrough);
-    BB->addSuccessor(CopyBB, BranchProbability::getOne());
-
-    // Mark all physical registers defined in the original block as being live
-    // on entry to the copy block.
-    for (const auto &MI : *CopyBB)
-      for (const MachineOperand &MO : MI.operands())
-        if (MO.isReg()) {
-          Register reg = MO.getReg();
-          if (Register::isPhysicalRegister(reg)) {
-            CopyBB->addLiveIn(reg);
-            break;
-          }
-        }
-
-    CopyBB->normalizeSuccProbs();
-    BB->normalizeSuccProbs();
-
-    BB->transferInlineAsmBrIndirectTargets(CopyBB);
-
-    InsertPos = CopyBB->end();
-    return CopyBB;
-  }
-
   InsertPos = Emitter.getInsertPos();
   return Emitter.getBlock();
 }
 
 /// Return the basic block label.
 std::string ScheduleDAGSDNodes::getDAGName() const {
   return "sunit-dag." + BB->getFullName();
 }

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

assert(!I.hasOperandBundlesOtherThan(
          "Cannot lower callbrs with arbitrary operand bundles yet!");
 
   assert(I.isInlineAsm() && "Only know how to handle inlineasm callbr");
   visitInlineAsm(I);
   CopyToExportRegsIfNeeded(&I);
 
   // Retrieve successors.
   MachineBasicBlock *Return = FuncInfo.MBBMap[I.getDefaultDest()];
-  Return->setInlineAsmBrDefaultTarget();
 
   // Update successor info.
   addSuccessorWithProb(CallBrMBB, Return, BranchProbability::getOne());
   for (unsigned i = 0, e = I.getNumIndirectDests(); i < e; ++i) {
     MachineBasicBlock *Target = FuncInfo.MBBMap[I.getIndirectDest(i)];
     addSuccessorWithProb(CallBrMBB, Target, BranchProbability::getZero());
-    CallBrMBB->addInlineAsmBrIndirectTarget(Target);
+    Target->setIsInlineAsmBrIndirectTarget();
   }
   CallBrMBB->normalizeSuccProbs();
 
   // Drop into default successor.
   DAG.setRoot(DAG.getNode(ISD::BR, getCurSDLoc(),
                           MVT::Other, getControlRoot(),
                           DAG.getBasicBlock(Return)));
 }
void SelectionDAGBuilder::UpdateSplitBlock(MachineBasicBlock *First,
   for (unsigned i = 0, e = SL->JTCases.size(); i != e; ++i)
     if (SL->JTCases[i].first.HeaderBB == First)
       SL->JTCases[i].first.HeaderBB = Last;
 
   // Update BitTestCases.
   for (unsigned i = 0, e = SL->BitTestCases.size(); i != e; ++i)
     if (SL->BitTestCases[i].Parent == First)
       SL->BitTestCases[i].Parent = Last;
-
-  // SelectionDAGISel::FinishBasicBlock will add PHI operands for the
-  // successors of the fallthrough block. Here, we add PHI operands for the
-  // successors of the INLINEASM_BR block itself.
-  if (First->getFirstTerminator()->getOpcode() == TargetOpcode::INLINEASM_BR)
-    for (std::pair<MachineInstr *, unsigned> &pair : FuncInfo.PHINodesToUpdate)
-      if (First->isSuccessor(pair.first->getParent()))
-        MachineInstrBuilder(*First->getParent(), pair.first)
-            .addReg(pair.second)
-            .addMBB(First);
 }
 
 void SelectionDAGBuilder::visitIndirectBr(const IndirectBrInst &I) {
   MachineBasicBlock *IndirectBrMBB = FuncInfo.MBB;
 
   // Update machine-CFG edges with unique successors.
   SmallSet<BasicBlock*, 32> Done;
   for (unsigned i = 0, e = I.getNumSuccessors(); i != e; ++i) {
if (!OpTy->isSingleValueType() && OpTy->isSized()) {
         break;
       }
     }
 
     return TLI.getValueType(DL, OpTy, true);
   }
 };
 
-using SDISelAsmOperandInfoVector = SmallVector<SDISelAsmOperandInfo, 16>;
 
 } // end anonymous namespace
 
 /// Make sure that the output operand \p OpInfo and its corresponding input
 /// operand \p MatchingOpInfo have compatible constraint types (otherwise error
 /// out).
 static void patchMatchingInput(const SDISelAsmOperandInfo &OpInfo,
                                SDISelAsmOperandInfo &MatchingOpInfo,

 
 } // end anonymous namespace
 
 /// visitInlineAsm - Handle a call to an InlineAsm object.
 void SelectionDAGBuilder::visitInlineAsm(const CallBase &Call) {
   const InlineAsm *IA = cast<InlineAsm>(Call.getCalledOperand());
 
   /// ConstraintOperands - Information about all of the constraints.
-  SDISelAsmOperandInfoVector ConstraintOperands;
+  SmallVector<SDISelAsmOperandInfo, 16> ConstraintOperands;
 
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   TargetLowering::AsmOperandInfoVector TargetConstraints = TLI.ParseConstraints(
       DAG.getDataLayout(), DAG.getSubtarget().getRegisterInfo(), Call);
 
   // First Pass: Calculate HasSideEffects and ExtraFlags (AlignStack,
   // AsmDialect, MayLoad, MayStore).
   bool HasSideEffect = IA->hasSideEffects();

llvm/lib/CodeGen/ShrinkWrap.cpp

for (MachineBasicBlock &MBB : MF) {
     LLVM_DEBUG(dbgs() << "Look into: " << MBB.getNumber() << ' '
                       << MBB.getName() << '\n');
 
     if (MBB.isEHFuncletEntry())
       return giveUpWithRemarks(ORE, "UnsupportedEHFunclets",
                                "EH Funclets are not supported yet.",
                                MBB.front().getDebugLoc(), &MBB);
 
-    if (MBB.isEHPad()) {
-      // Push the prologue and epilogue outside of
-      // the region that may throw by making sure
-      // that all the landing pads are at least at the
-      // boundary of the save and restore points.
-      // The problem with exceptions is that the throw
-      // is not properly modeled and in particular, a
-      // basic block can jump out from the middle.
+    if (MBB.isEHPad() || MBB.isInlineAsmBrIndirectTarget()) {
+      // Push the prologue and epilogue outside of the region that may throw (or
+      // jump out via inlineasm_br), by making sure that all the landing pads
+      // are at least at the boundary of the save and restore points.  The
+      // problem is that a basic block can jump out from the middle in these
+      // cases, which we do not handle.
       updateSaveRestorePoints(MBB, RS.get());
       if (!ArePointsInteresting()) {
-        LLVM_DEBUG(dbgs() << "EHPad prevents shrink-wrapping\n");
+        LLVM_DEBUG(dbgs() << "EHPad/inlineasm_br prevents shrink-wrapping\n");
         return false;
       }
       continue;
     }
 
     for (const MachineInstr &MI : MBB) {
       if (!useOrDefCSROrFI(MI, RS.get()))
         continue;

llvm/lib/CodeGen/SplitKit.h

 /// spill or other instruction related with CurLI.
 class LLVM_LIBRARY_VISIBILITY InsertPointAnalysis {
 private:
   const LiveIntervals &LIS;
 
   /// Last legal insert point in each basic block in the current function.
   /// The first entry is the first terminator, the second entry is the
   /// last valid point to insert a split or spill for a variable that is
-  /// live into a landing pad successor.
+  /// live into a landing pad or inlineasm_br successor.
   SmallVector<std::pair<SlotIndex, SlotIndex>, 8> LastInsertPoint;
 
   SlotIndex computeLastInsertPoint(const LiveInterval &CurLI,
                                    const MachineBasicBlock &MBB);
 
 public:
   InsertPointAnalysis(const LiveIntervals &lis, unsigned BBNum);
 

llvm/lib/CodeGen/SplitKit.cpp

 
 SlotIndex
 InsertPointAnalysis::computeLastInsertPoint(const LiveInterval &CurLI,
                                             const MachineBasicBlock &MBB) {
   unsigned Num = MBB.getNumber();
   std::pair<SlotIndex, SlotIndex> &LIP = LastInsertPoint[Num];
   SlotIndex MBBEnd = LIS.getMBBEndIdx(&MBB);
 
-  SmallVector<const MachineBasicBlock *, 1> EHPadSuccessors;
-  for (const MachineBasicBlock *SMBB : MBB.successors())
-    if (SMBB->isEHPad())
-      EHPadSuccessors.push_back(SMBB);
+  SmallVector<const MachineBasicBlock *, 1> ExceptionalSuccessors;
+  bool EHPadSuccessor = false;
+  for (const MachineBasicBlock *SMBB : MBB.successors()) {
+    if (SMBB->isEHPad()) {
+      ExceptionalSuccessors.push_back(SMBB);
+      EHPadSuccessor = true;
+    } else if (SMBB->isInlineAsmBrIndirectTarget())
+      ExceptionalSuccessors.push_back(SMBB);
+  }
 
   // Compute insert points on the first call. The pair is independent of the
   // current live interval.
   if (!LIP.first.isValid()) {
     MachineBasicBlock::const_iterator FirstTerm = MBB.getFirstTerminator();
     if (FirstTerm == MBB.end())
       LIP.first = MBBEnd;
     else
       LIP.first = LIS.getInstructionIndex(*FirstTerm);
 
-    // If there is a landing pad successor, also find the call instruction.
-    if (EHPadSuccessors.empty())
+    // If there is a landing pad or inlineasm_br successor, also find the
+    // instruction. If there is no such instruction, we don't need to do
+    // anything special.  We assume there cannot be multiple instructions that
+    // are Calls with EHPad successors or INLINEASM_BR in a block. Further, we
+    // assume that if there are any, they will be after any other call
+    // instructions in the block.
+    if (ExceptionalSuccessors.empty())
       return LIP.first;
-    // There may not be a call instruction (?) in which case we ignore LPad.
-    LIP.second = LIP.first;
-    for (MachineBasicBlock::const_iterator I = MBB.end(), E = MBB.begin();
-         I != E;) {
-      --I;
-      if (I->isCall()) {
+    for (auto I = MBB.rbegin(), E = MBB.rend(); I != E; ++I) {
+      if ((EHPadSuccessor && I->isCall()) ||
+          I->getOpcode() == TargetOpcode::INLINEASM_BR) {
         LIP.second = LIS.getInstructionIndex(*I);
         break;
       }
     }
   }
 
   // If CurLI is live into a landing pad successor, move the last insert point
   // back to the call that may throw.
   if (!LIP.second)
     return LIP.first;
 
-  if (none_of(EHPadSuccessors, [&](const MachineBasicBlock *EHPad) {
+  if (none_of(ExceptionalSuccessors, [&](const MachineBasicBlock *EHPad) {
         return LIS.isLiveInToMBB(CurLI, EHPad);
       }))
     return LIP.first;
 
   // Find the value leaving MBB.
   const VNInfo *VNI = CurLI.getVNInfoBefore(MBBEnd);
   if (!VNI)
     return LIP.first;

llvm/lib/CodeGen/TailDuplicator.cpp

bool TailDuplicator::duplicateSimpleBB(
     const DenseSet<Register> &UsedByPhi,
     SmallVectorImpl<MachineInstr *> &Copies) {
   SmallPtrSet<MachineBasicBlock *, 8> Succs(TailBB->succ_begin(),
                                             TailBB->succ_end());
   SmallVector<MachineBasicBlock *, 8> Preds(TailBB->pred_begin(),
                                             TailBB->pred_end());
   bool Changed = false;
   for (MachineBasicBlock *PredBB : Preds) {
-    if (PredBB->hasEHPadSuccessor())
+    if (PredBB->hasEHPadSuccessor() || PredBB->mayHaveInlineAsmBr())
       continue;
 
     if (bothUsedInPHI(*PredBB, Succs))
       continue;
 
     MachineBasicBlock *PredTBB = nullptr, *PredFBB = nullptr;
     SmallVector<MachineOperand, 4> PredCond;
     if (TII->analyzeBranch(*PredBB, PredTBB, PredFBB, PredCond))

llvm/lib/CodeGen/TargetInstrInfo.cpp

 /// and terminators.
 bool TargetInstrInfo::isSchedulingBoundary(const MachineInstr &MI,
                                            const MachineBasicBlock *MBB,
                                            const MachineFunction &MF) const {
   // Terminators and labels can't be scheduled around.
   if (MI.isTerminator() || MI.isPosition())
     return true;
 
+  // INLINEASM_BR can jump to another block
+  if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+    return true;
+
   // Don't attempt to schedule around any instruction that defines
   // a stack-oriented pointer, as it's unlikely to be profitable. This
   // saves compile time, because it doesn't require every single
   // stack slot reference to depend on the instruction that does the
   // modification.
   const TargetLowering &TLI = *MF.getSubtarget().getTargetLowering();
   const TargetRegisterInfo *TRI = MF.getSubtarget().getRegisterInfo();
   return MI.modifiesRegister(TLI.getStackPointerRegisterToSaveRestore(), TRI);

llvm/lib/Target/AMDGPU/SIInstrInfo.cpp

bool SIInstrInfo::isSchedulingBoundary(const MachineInstr &MI,
   // TODO: Do we really want this barrier? It triggers unnecessary hazard nops
   // but is probably avoidable.
 
   // Copied from base implementation.
   // Terminators and labels can't be scheduled around.
   if (MI.isTerminator() || MI.isPosition())
     return true;
 
+  // INLINEASM_BR can jump to another block
+  if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+    return true;
+
   // Target-independent instructions do not have an implicit-use of EXEC, even
   // when they operate on VGPRs. Treating EXEC modifications as scheduling
   // boundaries prevents incorrect movements of such instructions.
 
   // TODO: Don't treat setreg with known constant that only changes MODE as
   // barrier.
   return MI.modifiesRegister(AMDGPU::EXEC, &RI) ||
          MI.getOpcode() == AMDGPU::S_SETREG_IMM32_B32 ||

llvm/lib/Target/ARM/ARMBaseInstrInfo.cpp

bool ARMBaseInstrInfo::isSchedulingBoundary(const MachineInstr &MI,
   // when debug info is not present.
   if (MI.isDebugInstr())
     return false;
 
   // Terminators and labels can't be scheduled around.
   if (MI.isTerminator() || MI.isPosition())
     return true;
 
+  // INLINEASM_BR can jump to another block
+  if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+    return true;
+
   // Treat the start of the IT block as a scheduling boundary, but schedule
   // t2IT along with all instructions following it.
   // FIXME: This is a big hammer. But the alternative is to add all potential
   // true and anti dependencies to IT block instructions as implicit operands
   // to the t2IT instruction. The added compile time and complexity does not
   // seem worth it.
   MachineBasicBlock::const_iterator I = MI;
   // Make sure to skip any debug instructions

llvm/lib/Target/Hexagon/BitTracker.cpp

if (!Eval) {
         else
           dbgs() << "\n  does not fall through\n";
       }
       Targets.insert(BTs.begin(), BTs.end());
     }
     ++It;
   } while (FallsThrough && It != End);
 
+  if (B.mayHaveInlineAsmBr())
+    DefaultToAll = true;
+
   if (!DefaultToAll) {
     // Need to add all CFG successors that lead to EH landing pads.
     // There won't be explicit branches to these blocks, but they must
     // be processed.
     for (const MachineBasicBlock *SB : B.successors()) {
       if (SB->isEHPad())
         Targets.insert(SB);
     }
     if (FallsThrough) {

llvm/lib/Target/Hexagon/HexagonConstPropagation.cpp

while (It != End) {
     EvalOk = EvalOk && MCE.evaluate(MI, Cells, Targets, FallsThru);
     if (!EvalOk)
       FallsThru = true;
     if (!FallsThru)
       break;
     ++It;
   }
 
+  if (B.mayHaveInlineAsmBr())
+    EvalOk = false;
+
   if (EvalOk) {
     // Need to add all CFG successors that lead to EH landing pads.
     // There won't be explicit branches to these blocks, but they must
     // be processed.
     for (const MachineBasicBlock *SB : B.successors()) {
       if (SB->isEHPad())
         Targets.insert(SB);
     }
     if (FallsThru) {
for (MachineInstr &MI : MRI->use_nodbg_instructions(Reg)) {
     else if (!MI.isBranch())
       visitNonBranch(MI);
     else
       visitBranchesFrom(MI);
   }
 }
 
 bool MachineConstPropagator::computeBlockSuccessors(const MachineBasicBlock *MB,
       SetVector<const MachineBasicBlock*> &Targets) {
+  Targets.clear();
+
   MachineBasicBlock::const_iterator FirstBr = MB->end();
   for (const MachineInstr &MI : *MB) {
+    if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+      return false;
     if (MI.isDebugInstr())
       continue;
     if (MI.isBranch()) {
       FirstBr = MI.getIterator();
       break;
     }
   }
 
-  Targets.clear();
   MachineBasicBlock::const_iterator End = MB->end();
 
   bool DoNext = true;
   for (MachineBasicBlock::const_iterator I = FirstBr; I != End; ++I) {
     const MachineInstr &MI = *I;
     // Can there be debug instructions between branches?
     if (MI.isDebugInstr())
       continue;

llvm/lib/Target/Hexagon/HexagonInstrInfo.cpp

for (auto I : MBB->successors())
       if (I->isEHPad())
         return true;
   }
 
   // Terminators and labels can't be scheduled around.
   if (MI.getDesc().isTerminator() || MI.isPosition())
     return true;
 
+  // INLINEASM_BR can jump to another block
+  if (MI.getOpcode() == TargetOpcode::INLINEASM_BR)
+    return true;
+
   if (MI.isInlineAsm() && !ScheduleInlineAsm)
     return true;
 
   return false;
 }
 
 /// Measure the specified inline asm to determine an approximation of its
 /// length.

llvm/lib/Target/PowerPC/PPCBranchCoalescing.cpp

for (auto &I : Cand.BranchBlock->terminators()) {
     }
   }
 
   if (Cand.BranchBlock->isEHPad() || Cand.BranchBlock->hasEHPadSuccessor()) {
     LLVM_DEBUG(dbgs() << "EH Pad - skip\n");
     return false;
   }
 
+  if (Cand.BranchBlock->mayHaveInlineAsmBr()) {
+    LLVM_DEBUG(dbgs() << "Inline Asm Br - skip\n");
+    return false;
+  }
+
   // For now only consider triangles (i.e, BranchTargetBlock is set,
   // FalseMBB is null, and BranchTargetBlock is a successor to BranchBlock)
   if (!Cand.BranchTargetBlock || FalseMBB ||
       !Cand.BranchBlock->isSuccessor(Cand.BranchTargetBlock)) {
     LLVM_DEBUG(dbgs() << "Does not form a triangle - skip\n");
     return false;
   }
 

llvm/test/CodeGen/AArch64/callbr-asm-label.ll

 ; RUN: llc < %s -mtriple=aarch64-linux-gnu | FileCheck %s
 
 @X = common local_unnamed_addr global i32 0, align 4
 
 define i32 @test1() {
 ; CHECK-LABEL: test1:
 ; CHECK:         .word b
 ; CHECK-NEXT:    .word .Ltmp0
-; CHECK-LABEL: .LBB0_1: // %cleanup
-; CHECK-LABEL: .Ltmp0:
-; CHECK-LABEL: .LBB0_2: // %indirect
+; CHECK: // %bb.1:
+; CHECK: .Ltmp0:
+; CHECK: .LBB0_2: // %indirect
 entry:
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test1, %indirect))
           to label %cleanup [label %indirect]
 
 indirect:
   br label %cleanup
 
 cleanup:
   %retval.0 = phi i32 [ 1, %indirect ], [ 0, %entry ]
   ret i32 %retval.0
 }
 
 define void @test2() {
 ; CHECK-LABEL: test2:
 entry:
   %0 = load i32, i32* @X, align 4
   %and = and i32 %0, 1
   %tobool = icmp eq i32 %and, 0
   br i1 %tobool, label %if.end10, label %if.then
 
 if.then:
 ; CHECK:       .word b
 ; CHECK-NEXT:  .word .Ltmp2
-; CHECK-LABEL: .Ltmp2:
+; CHECK:       .Ltmp2:
 ; CHECK-NEXT:  .LBB1_3: // %if.end6
   callbr void asm sideeffect "1:\0A\09.word b, ${0:l}\0A\09", "X"(i8* blockaddress(@test2, %if.end6))
           to label %if.then4 [label %if.end6]
 
 if.then4:
   %call5 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   br label %if.end6
 
 if.end6:
   %.pre = load i32, i32* @X, align 4
   %.pre13 = and i32 %.pre, 1
   %phitmp = icmp eq i32 %.pre13, 0
   br i1 %phitmp, label %if.end10, label %if.then9
 
 if.then9:
-; CHECK-LABEL: .Ltmp4:
+; CHECK: .Ltmp4:
 ; CHECK-NEXT:  .LBB1_5: // %l_yes
   callbr void asm sideeffect "", "X"(i8* blockaddress(@test2, %l_yes))
           to label %if.end10 [label %l_yes]
 
 if.end10:
   br label %l_yes
 
 l_yes:
   ret void
 }
 
 declare i32 @g(...)

llvm/test/CodeGen/AArch64/callbr-asm-obj-file.ll

 ; RUN: llc < %s -mtriple=aarch64-unknown-linux-gnu -filetype=obj -o - \
 ; RUN:  | llvm-objdump --triple=aarch64-unknown-linux-gnu -d - \
 ; RUN:  | FileCheck %s
 
 %struct.c = type { i1 (...)* }
 
 @l = common hidden local_unnamed_addr global i32 0, align 4
 
 ; CHECK-LABEL: <test1>:
 ; CHECK-LABEL: <$d.1>:
 ; CHECK-LABEL: <$x.2>:
-; CHECK-NEXT:    b 0x30 <$x.4+0x4>
+; CHECK-NEXT:    b 0x2c <$x.4>
 ; CHECK-LABEL: <$x.4>:
-; CHECK-NEXT:    b 0x30 <$x.4+0x4>
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ldr x30, [sp], #16
 ; CHECK-NEXT:    ret
 define hidden i32 @test1() {
   %1 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   %2 = icmp eq i32 %1, 0
   br i1 %2, label %3, label %5
 
10:                                               ; preds = %7, %0, %6, %9
   ret i32 undef
 }
 
 ; CHECK-LABEL: <test3>:
 ; CHECK-LABEL: <$d.9>:
 ; CHECK-LABEL: <$x.10>:
 ; CHECK-NEXT:    b {{.*}} <test3+0x18>
 ; CHECK-LABEL: <$x.12>:
-; CHECK-NEXT:    b {{.*}} <$x.12+0x4>
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ldr x30, [sp], #16
 ; CHECK-NEXT:    ret
 define internal i1 @test3() {
   %1 = tail call i32 bitcast (i32 (...)* @g to i32 ()*)()
   %2 = icmp eq i32 %1, 0
   br i1 %2, label %3, label %5
 

llvm/test/CodeGen/ARM/ifcvt-diamond-unanalyzable-common.mir

 # NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
 # RUN: llc %s -o - -run-pass=if-converter -verify-machineinstrs | FileCheck %s
-# Make sure we correctly if-convert blocks containing an INLINEASM_BR.
+# Make sure we correctly if-convert blocks containing an unanalyzable branch sequence.
+# (In this case, multiple conditional branches)
 
 --- |
   target triple = "thumbv7-unknown-linux-gnueabi"
 
   define dso_local void @fn1() {
   l_yes:
     ret void
   }
body:             |
   ; CHECK: bb.0:
   ; CHECK:   successors: %bb.1(0x40000000), %bb.2(0x40000000)
   ; CHECK:   liveins: $r0, $r1, $r2, $r4, $lr
   ; CHECK:   $sp = frame-setup t2STMDB_UPD $sp, 14 /* CC::al */, $noreg, killed $r4, killed $lr
   ; CHECK:   t2CMPri killed renamable $r2, 34, 14 /* CC::al */, $noreg, implicit-def $cpsr
   ; CHECK:   $r0 = t2MOVi 2, 1 /* CC::ne */, $cpsr, $noreg
   ; CHECK:   $r0 = t2MOVi 3, 0 /* CC::eq */, killed $cpsr, $noreg, implicit killed $r0
   ; CHECK:   tBL 14 /* CC::al */, $noreg, @fn2, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit killed $r0, implicit killed $r1, implicit-def $sp, implicit-def dead $r0
-  ; CHECK:   INLINEASM_BR &"", 9 /* sideeffect mayload attdialect */, 13 /* imm */, 0, 13 /* imm */, blockaddress(@fn1, %ir-block.l_yes)
+  ; CHECK:   t2CMPri $sp, 34, 14 /* CC::al */, $noreg, implicit-def $cpsr
+  ; CHECK:   t2Bcc %bb.2, 1 /* CC::ne */, $cpsr
+  ; CHECK:   t2Bcc %bb.2, 2 /* CC::hs */, killed $cpsr
   ; CHECK:   t2B %bb.1, 14 /* CC::al */, $noreg
   ; CHECK: bb.1:
-  ; CHECK:   INLINEASM &"", 1 /* sideeffect attdialect */
+  ; CHECK:   INLINEASM &"", 1
   ; CHECK:   $sp = t2LDMIA_RET $sp, 14 /* CC::al */, $noreg, def $r4, def $pc
   ; CHECK: bb.2.l_yes (address-taken):
   ; CHECK:   $sp = t2LDMIA_RET $sp, 14 /* CC::al */, $noreg, def $r4, def $pc
   bb.0:
     successors: %bb.1(0x40000000), %bb.2(0x40000000)
     liveins: $r0, $r1, $r2, $r4, $lr
 
     $sp = frame-setup t2STMDB_UPD $sp, 14, $noreg, killed $r4, killed $lr
     t2CMPri killed renamable $r2, 34, 14, $noreg, implicit-def $cpsr
     t2Bcc %bb.2, 1, killed $cpsr
 
   bb.1:
     successors: %bb.3(0x40000000), %bb.4(0x40000000)
     liveins: $r1
 
     $r0 = t2MOVi 3, 14, $noreg, $noreg
     tBL 14, $noreg, @fn2, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit $r0, implicit $r1, implicit-def $sp, implicit-def dead $r0
-    INLINEASM_BR &"", 9, 13, 0, 13, blockaddress(@fn1, %ir-block.l_yes)
+    t2CMPri $sp, 34, 14, $noreg, implicit-def $cpsr
+    t2Bcc %bb.4, 1, $cpsr
+    t2Bcc %bb.4, 2, killed $cpsr
     t2B %bb.3, 14, $noreg
 
   bb.2:
     successors: %bb.3(0x40000000), %bb.4(0x40000000)
     liveins: $r1
 
     $r0 = t2MOVi 2, 14, $noreg, $noreg
     tBL 14, $noreg, @fn2, csr_aapcs, implicit-def dead $lr, implicit $sp, implicit $r0, implicit $r1, implicit-def $sp, implicit-def dead $r0
-    INLINEASM_BR &"", 9, 13, 0, 13, blockaddress(@fn1, %ir-block.l_yes)
+    t2CMPri $sp, 34, 14, $noreg, implicit-def $cpsr
+    t2Bcc %bb.4, 1, $cpsr
+    t2Bcc %bb.4, 2, killed $cpsr
     t2B %bb.3, 14, $noreg
 
   bb.3:
     INLINEASM &"", 1
     $sp = t2LDMIA_RET $sp, 14, $noreg, def $r4, def $pc
 
   bb.4.l_yes (address-taken):
     $sp = t2LDMIA_RET $sp, 14, $noreg, def $r4, def $pc
 
 ...

llvm/test/CodeGen/ARM/ifcvt-size.mir

 # CHECK:      tCMPi8
 # CHECK-NEXT: tLDRi
 # CHECK-NEXT: tLDRi
 # CHECK-NEXT: tLDRi
 # CHECK-NEXT: t2LDRSHi12
 # CHECK-NEXT: INLINEASM_BR
 
 # DEBUG-LABEL: Ifcvt: function ({{[0-9]+}}) 'fn9'
-# DEBUG: MeetIfcvtSizeLimit(BranchBytes=2, CommonBytes=6, NumPredicatedInstructions=4, ExtraPredicateBytes=2)
+# DEBUG: MeetIfcvtSizeLimit(BranchBytes=2, CommonBytes=8, NumPredicatedInstructions=4, ExtraPredicateBytes=2)
 
 body:             |
   bb.0.entry:
     successors: %bb.1(0x30000000), %bb.3(0x50000000)
     liveins: $r0, $r1, $r2
 
-    tCMPi8 killed renamable $r2, 42, 14, $noreg, implicit-def $cpsr
+    tCMPi8 renamable $r2, 42, 14, $noreg, implicit-def $cpsr
     t2Bcc %bb.3, 1, killed $cpsr
 
   bb.1.if.then:
     successors:  %bb.5(0x7fffffff)
-    liveins: $r0
+    liveins: $r0, $r2
 
     renamable $r0 = tLDRi killed renamable $r0, 0, 14, $noreg
     INLINEASM_BR &"b ${0:l}", 1, 13, blockaddress(@fn9, %ir-block.lab1)
+    tBX_RET 14, $noreg, implicit $r2
 
   bb.3.if.else:
     successors: %bb.5(0x7fffffff)
-    liveins: $r1
+    liveins: $r1, $r2
 
     renamable $r0 = tLDRi killed renamable $r1, 0, 14, $noreg
     renamable $r0 = tLDRi killed renamable $r0, 0, 14, $noreg
     renamable $r0 = t2LDRSHi12 killed renamable $r0, 0, 14, $noreg
     INLINEASM_BR &"b ${0:l}", 1, 13, blockaddress(@fn9, %ir-block.lab1)
+    tBX_RET 14, $noreg, implicit $r2
     
   bb.5.lab1 (address-taken):
     liveins: $r0
 
     renamable $r0, dead $cpsr = nsw tADDi8 killed renamable $r0, 5, 14, $noreg
     tBX_RET 14, $noreg, implicit $r0
 ...

llvm/test/CodeGen/X86/callbr-asm-blockplacement.ll

 ; CHECK-NEXT:    testb %r13b, %r13b
 ; CHECK-NEXT:    jne .LBB0_2
 ; CHECK-NEXT:  # %bb.3: # %bb15
 ; CHECK-NEXT:    leaq (%rbp,%rbp,2), %rax
 ; CHECK-NEXT:    movq %rbx, global+16(,%rax,8)
 ; CHECK-NEXT:    movabsq $-2305847407260205056, %rbx # imm = 0xDFFFFC0000000000
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB0_4: # %bb17
+; CHECK-NEXT:  # %bb.4: # %bb17
 ; CHECK-NEXT:    callq widget
 ; CHECK-NEXT:  .Ltmp0: # Block address taken
 ; CHECK-NEXT:  .LBB0_5: # %bb18
 ; CHECK-NEXT:    movw $0, 14(%rbx)
 ; CHECK-NEXT:    addq $8, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13

llvm/test/CodeGen/X86/callbr-asm-branch-folding.ll

 ; CHECK-NEXT:    testl %ebp, %ebp
 ; CHECK-NEXT:    je .LBB0_9
 ; CHECK-NEXT:  # %bb.7: # %if.then14
 ; CHECK-NEXT:    movl {{[-0-9]+}}(%r{{[sb]}}p), %eax # 4-byte Reload
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    jmp .LBB0_10
 ; CHECK-NEXT:  .Ltmp0: # Block address taken
-; CHECK-NEXT:  .LBB0_8: # %if.then20.critedge
+; CHECK-NEXT:  # %bb.8: # %if.then20.critedge
 ; CHECK-NEXT:    movl {{.*}}(%rip), %edi
 ; CHECK-NEXT:    movslq %eax, %rcx
 ; CHECK-NEXT:    movl $1, %esi
 ; CHECK-NEXT:    movq %r15, %rdx
 ; CHECK-NEXT:    addq $8, %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %r12
 ; CHECK-NEXT:    popq %r13

llvm/test/CodeGen/X86/callbr-asm-instr-scheduling.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -verify-machineinstrs -mcpu=znver2 -O2 -frame-pointer=none < %s | FileCheck %s
+
+; Make sure that instructions aren't scheduled after the "callbr". In the
+; example below, we don't want the "shrxq" through "leaq" instructions to be
+; moved after the "callbr".
+
+%struct.cpuinfo_x86 = type { i8, i8, i8, i8, i32, [3 x i32], i8, i8, i8, i8, i32, i32, %union.anon.83, [16 x i8], [64 x i8], i32, i32, i32, i32, i32, i32, i64, i16, i16, i16, i16, i16, i16, i16, i16, i16, i16, i16, i32, i8, i8 }
+%union.anon.83 = type { i64, [72 x i8] }
+%struct.pgd_t = type { i64 }
+%struct.p4d_t = type { i64 }
+%struct.pud_t = type { i64 }
+
+@boot_cpu_data = external dso_local global %struct.cpuinfo_x86, align 8
+@page_offset_base = external dso_local local_unnamed_addr global i64, align 8
+@pgdir_shift = external dso_local local_unnamed_addr global i32, align 4
+@__force_order = external dso_local global i64, align 8
+@ptrs_per_p4d = external dso_local local_unnamed_addr global i32, align 4
+
+define i64 @early_ioremap_pmd(i64 %addr) {
+; CHECK-LABEL: early_ioremap_pmd:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    movq %cr3, %rax
+; CHECK-EMPTY:
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    movabsq $9223372036854771712, %rdx # imm = 0x7FFFFFFFFFFFF000
+; CHECK-NEXT:    andq %rax, %rdx
+; CHECK-NEXT:    movb {{.*}}(%rip), %al
+; CHECK-NEXT:    movq {{.*}}(%rip), %rcx
+; CHECK-NEXT:    shrxq %rax, %rdi, %rax
+; CHECK-NEXT:    addq %rcx, %rdx
+; CHECK-NEXT:    andl $511, %eax # imm = 0x1FF
+; CHECK-NEXT:    leaq (%rdx,%rax,8), %rax
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:  .Ltmp2:
+; CHECK-NEXT:    jmp .Ltmp3
+; CHECK-NEXT:  .Ltmp4:
+; CHECK-NEXT:    .zero (-(((.Ltmp5-.Ltmp6)-(.Ltmp4-.Ltmp2))>0))*((.Ltmp5-.Ltmp6)-(.Ltmp4-.Ltmp2)),144
+; CHECK-NEXT:  .Ltmp7:
+entry:
+  %0 = tail call i64 asm sideeffect "mov %cr3,$0\0A\09", "=r,=*m,~{dirflag},~{fpsr},~{flags}"(i64* nonnull @__force_order)
+  %and.i = and i64 %0, 9223372036854771712
+  %1 = load i64, i64* @page_offset_base, align 8
+  %add = add i64 %and.i, %1
+  %2 = inttoptr i64 %add to %struct.pgd_t*
+  %3 = load i32, i32* @pgdir_shift, align 4
+  %sh_prom = zext i32 %3 to i64
+  %shr = lshr i64 %addr, %sh_prom
+  %and = and i64 %shr, 511
+  %arrayidx = getelementptr %struct.pgd_t, %struct.pgd_t* %2, i64 %and
+  callbr void asm sideeffect "1: jmp 6f\0A2:\0A.skip -(((5f-4f) - (2b-1b)) > 0) * ((5f-4f) - (2b-1b)),0x90\0A3:\0A.section .altinstructions,\22a\22\0A .long 1b - .\0A .long 4f - .\0A .word ${1:P}\0A .byte 3b - 1b\0A .byte 5f - 4f\0A .byte 3b - 2b\0A.previous\0A.section .altinstr_replacement,\22ax\22\0A4: jmp ${5:l}\0A5:\0A.previous\0A.section .altinstructions,\22a\22\0A .long 1b - .\0A .long 0\0A .word ${0:P}\0A .byte 3b - 1b\0A .byte 0\0A .byte 0\0A.previous\0A.section .altinstr_aux,\22ax\22\0A6:\0A testb $2,$3\0A jnz ${4:l}\0A jmp ${5:l}\0A.previous\0A", "i,i,i,*m,X,X,~{dirflag},~{fpsr},~{flags}"(i16 528, i32 117, i32 1, i8* getelementptr inbounds (%struct.cpuinfo_x86, %struct.cpuinfo_x86* @boot_cpu_data, i64 0, i32 12, i32 1, i64 58), i8* blockaddress(@early_ioremap_pmd, %if.end.i), i8* blockaddress(@early_ioremap_pmd, %if.then.i))
+          to label %_static_cpu_has.exit.thread.i [label %if.end.i, label %if.then.i]
+
+_static_cpu_has.exit.thread.i:                    ; preds = %entry
+  br label %if.end.i
+
+if.then.i:                                        ; preds = %entry
+  %4 = bitcast %struct.pgd_t* %arrayidx to %struct.p4d_t*
+  br label %p4d_offset.exit
+
+if.end.i:                                         ; preds = %_static_cpu_has.exit.thread.i, %entry
+  %coerce.dive.i = getelementptr inbounds %struct.pgd_t, %struct.pgd_t* %arrayidx, i64 0, i32 0
+  %5 = load i64, i64* %coerce.dive.i, align 8
+  %6 = inttoptr i64 %5 to %struct.p4d_t*
+  %7 = load i32, i32* @ptrs_per_p4d, align 4
+  %sub.i.i = add i32 %7, 33554431
+  %8 = and i32 %sub.i.i, 33554431
+  %and.i1.i = zext i32 %8 to i64
+  %add.ptr.i = getelementptr %struct.p4d_t, %struct.p4d_t* %6, i64 %and.i1.i
+  br label %p4d_offset.exit
+
+p4d_offset.exit:                                  ; preds = %if.end.i, %if.then.i
+  %retval.0.i = phi %struct.p4d_t* [ %add.ptr.i, %if.end.i ], [ %4, %if.then.i ]
+  %coerce.dive.i12 = getelementptr inbounds %struct.p4d_t, %struct.p4d_t* %retval.0.i, i64 0, i32 0
+  %9 = load i64, i64* %coerce.dive.i12, align 8
+  %and.i.i13 = and i64 %9, 4503599627366400
+  %add.i.i14 = add i64 %and.i.i13, %1
+  %10 = inttoptr i64 %add.i.i14 to %struct.pud_t*
+  %coerce.dive.i16 = getelementptr %struct.pud_t, %struct.pud_t* %10, i64 511, i32 0
+  %11 = load i64, i64* %coerce.dive.i16, align 8
+  %tobool.i.i.i = icmp slt i64 %11, 0
+  %..i.i.i = select i1 %tobool.i.i.i, i64 4503598553628672, i64 4503599627366400
+  ret i64 %..i.i.i
+}

llvm/test/CodeGen/X86/callbr-asm-label-addr.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
 
 define i32 @test1(i32 %x) {
 ; CHECK-LABEL: test1:
 ; CHECK:         .quad .Ltmp0
 ; CHECK-NEXT:    .quad .Ltmp1
-; CHECK-LABEL: .Ltmp1:
-; CHECK-LABEL: .LBB0_1: # %bar
+; CHECK: .Ltmp1:
+; CHECK-NEXT: # %bb.1: # %bar
 ; CHECK-NEXT:    callq foo
-; CHECK-LABEL: .Ltmp0:
+; CHECK-NEXT: .Ltmp0:
 ; CHECK-NEXT:  # %bb.2: # %baz
 entry:
   callbr void asm sideeffect ".quad ${0:l}\0A\09.quad ${1:l}", "i,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test1, %baz), i8* blockaddress(@test1, %bar))
           to label %asm.fallthrough [label %bar]
 
 asm.fallthrough:
   br label %bar
 

llvm/test/CodeGen/X86/callbr-asm-outputs-pred-succ.ll

 ; Tests that InstrEmitter::EmitMachineNode correctly sets predecessors and
 ; successors.
 
 ; RUN: llc -stop-after=finalize-isel -print-after=finalize-isel -mtriple=i686-- < %s 2>&1 | FileCheck %s
 
 ; The block containting the INLINEASM_BR should have a fallthrough and its
-; indirect targets as its successors. The fallthrough is a block we synthesized
-; in InstrEmitter::EmitMachineNode. Fallthrough should have 100% branch weight,
+; indirect targets as its successors. Fallthrough should have 100% branch weight,
 ; while the indirect targets have 0%.
 ; CHECK: bb.0 (%ir-block.2):
-; CHECK-NEXT: successors: %bb.4(0x00000000), %bb.6(0x80000000); %bb.4(0.00%), %bb.6(100.00%)
+; CHECK-NEXT: successors: %bb.1(0x80000000), %bb.4(0x00000000); %bb.1(100.00%), %bb.4(0.00%)
 
-; The fallthrough block is predaccessed by the block containing INLINEASM_BR,
-; and succeeded by the INLINEASM_BR's original fallthrough block pre-splitting.
-; CHECK: bb.6 (%ir-block.2):
-; CHECK-NEXT: predecessors: %bb.0
-; CHECK-NEXT: successors: %bb.1(0x80000000); %bb.1(100.00%)
-
-; Another block containing a second INLINEASM_BR. Check it has two successors,
-; and the the probability for fallthrough is 100%. Predecessor check irrelevant.
+; The fallthrough is a block containing a second INLINEASM_BR. Check it has two successors,
+; and the the probability for fallthrough is 100%.
 ; CHECK: bb.1 (%ir-block.4):
-; CHECK: successors: %bb.2(0x00000000), %bb.7(0x80000000); %bb.2(0.00%), %bb.7(100.00%)
-
-; Check the synthesized fallthrough block for the second INLINEASM_BR is
-; preceded correctly, and has the original successor pre-splitting.
-; CHECK: bb.7 (%ir-block.4):
-; CHECK-NEXT: predecessors: %bb.1
-; CHECK-NEXT: successors: %bb.3(0x80000000); %bb.3(100.00%)
+; CHECK-NEXT: predecessors: %bb.0
+; CHECK-NEXT: successors: %bb.3(0x80000000), %bb.2(0x00000000); %bb.3(100.00%), %bb.2(0.00%)
 
 ; Check the second INLINEASM_BR target block is preceded by the block with the
 ; second INLINEASM_BR.
 ; CHECK: bb.2 (%ir-block.7, address-taken):
 ; CHECK-NEXT: predecessors: %bb.1
 
 ; Check the first INLINEASM_BR target block is predecessed by the block with
 ; the first INLINEASM_BR.

llvm/test/CodeGen/X86/callbr-asm-outputs.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=i686-- -verify-machineinstrs < %s | FileCheck %s
 
 ; A test for asm-goto output
 
 define i32 @test1(i32 %x) {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    addl $4, %eax
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    jmp .Ltmp0
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB0_1: # %normal
+; CHECK-NEXT:  # %bb.1: # %normal
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .Ltmp0: # Block address taken
 ; CHECK-NEXT:  .LBB0_2: # %abnormal
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    retl
 entry:
   %add = add nsw i32 %x, 4
   %ret = callbr i32 asm "xorl $1, $0; jmp ${2:l}", "=r,r,X,~{dirflag},~{fpsr},~{flags}"(i32 %add, i8* blockaddress(@test1, %abnormal))
           to label %normal [label %abnormal]
 
 normal:
   ret i32 %ret
 
 abnormal:
   ret i32 1
 }
 
 define i32 @test2(i32 %out1, i32 %out2) {
 ; CHECK-LABEL: test2:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %edi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    pushl %esi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 12
 ; CHECK-NEXT:    .cfi_offset %esi, -12
 ; CHECK-NEXT:    .cfi_offset %edi, -8
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %edi
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %esi
 ; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    cmpl %edi, %esi
-; CHECK-NEXT:    jge .LBB1_3
+; CHECK-NEXT:    jge .LBB1_2
 ; CHECK-NEXT:  # %bb.1: # %if.then
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    testl %esi, %esi
 ; CHECK-NEXT:    testl %edi, %esi
 ; CHECK-NEXT:    jne .Ltmp1
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB1_2: # %if.then
-; CHECK-NEXT:    movl %edi, %eax
-; CHECK-NEXT:    addl %esi, %eax
+; CHECK-NEXT:    jmp .LBB1_3
+; CHECK-NEXT:  .LBB1_2: # %if.else
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    testl %esi, %edi
+; CHECK-NEXT:    testl %esi, %edi
+; CHECK-NEXT:    jne .Ltmp2
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  .LBB1_3:
+; CHECK-NEXT:    movl %esi, %eax
+; CHECK-NEXT:    addl %edi, %eax
 ; CHECK-NEXT:  .Ltmp2: # Block address taken
-; CHECK-NEXT:  .LBB1_6: # %return
+; CHECK-NEXT:  .LBB1_5: # %return
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %edi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
-; CHECK-NEXT:  .LBB1_3: # %if.else
-; CHECK-NEXT:    .cfi_def_cfa_offset 12
-; CHECK-NEXT:    #APP
-; CHECK-NEXT:    testl %esi, %edi
-; CHECK-NEXT:    testl %esi, %edi
-; CHECK-NEXT:    jne .Ltmp2
-; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB1_4: # %if.else
-; CHECK-NEXT:    jmp .LBB1_2
 ; CHECK-NEXT:  .Ltmp1: # Block address taken
-; CHECK-NEXT:  .LBB1_5: # %label_true
+; CHECK-NEXT:  .LBB1_4: # %label_true
+; CHECK-NEXT:    .cfi_def_cfa_offset 12
 ; CHECK-NEXT:    movl $-2, %eax
-; CHECK-NEXT:    jmp .LBB1_6
+; CHECK-NEXT:    jmp .LBB1_5
 entry:
   %cmp = icmp slt i32 %out1, %out2
   br i1 %cmp, label %if.then, label %if.else
 
 if.then:                                          ; preds = %entry
   %0 = callbr { i32, i32 } asm sideeffect "testl $0, $0; testl $1, $2; jne ${3:l}", "={si},={di},r,X,X,0,1,~{dirflag},~{fpsr},~{flags}"(i32 %out1, i8* blockaddress(@test2, %label_true), i8* blockaddress(@test2, %return), i32 %out1, i32 %out2)
           to label %if.end [label %label_true, label %return]
 

 ; CHECK-NEXT:    .cfi_offset %edi, -8
 ; CHECK-NEXT:    testb $1, {{[0-9]+}}(%esp)
 ; CHECK-NEXT:    je .LBB2_3
 ; CHECK-NEXT:  # %bb.1: # %true
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    .short %esi
 ; CHECK-NEXT:    .short %edi
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB2_2: # %true
+; CHECK-NEXT:  # %bb.2:
 ; CHECK-NEXT:    movl %edi, %eax
 ; CHECK-NEXT:    jmp .LBB2_5
 ; CHECK-NEXT:  .LBB2_3: # %false
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    .short %eax
 ; CHECK-NEXT:    .short %edx
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB2_4: # %false
+; CHECK-NEXT:  # %bb.4:
 ; CHECK-NEXT:    movl %edx, %eax
 ; CHECK-NEXT:  .LBB2_5: # %asm.fallthrough
 ; CHECK-NEXT:    popl %esi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    popl %edi
 ; CHECK-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .Ltmp3: # Block address taken

 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movl $-1, %eax
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %ecx
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    testl %ecx, %ecx
 ; CHECK-NEXT:    testl %edx, %ecx
 ; CHECK-NEXT:    jne .Ltmp4
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB3_1: # %asm.fallthrough
+; CHECK-NEXT:  # %bb.1: # %asm.fallthrough
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    testl %ecx, %edx
 ; CHECK-NEXT:    testl %ecx, %edx
 ; CHECK-NEXT:    jne .Ltmp5
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB3_2: # %asm.fallthrough
+; CHECK-NEXT:  # %bb.2: # %asm.fallthrough2
 ; CHECK-NEXT:    addl %edx, %ecx
 ; CHECK-NEXT:    movl %ecx, %eax
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .Ltmp4: # Block address taken
 ; CHECK-NEXT:  .LBB3_3: # %label_true
 ; CHECK-NEXT:    movl $-2, %eax
 ; CHECK-NEXT:  .Ltmp5: # Block address taken
 ; CHECK-NEXT:  .LBB3_4: # %return

llvm/test/CodeGen/X86/callbr-asm.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=i686-- -O3 -verify-machineinstrs | FileCheck %s
 
 ; Tests for using callbr as an asm-goto wrapper
 
 ; Test 1 - fallthrough label gets removed, but the fallthrough code that is
 ; unreachable due to asm ending on a jmp is still left in.
 define i32 @test1(i32 %a) {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    addl $4, %eax
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    jmp .Ltmp0
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB0_1: # %normal
+; CHECK-NEXT:  # %bb.1: # %normal
 ; CHECK-NEXT:    xorl %eax, %eax
 ; CHECK-NEXT:    retl
 ; CHECK-NEXT:  .Ltmp0: # Block address taken
 ; CHECK-NEXT:  .LBB0_2: # %fail
 ; CHECK-NEXT:    movl $1, %eax
 ; CHECK-NEXT:    retl
 entry:
   %0 = add i32 %a, 4

 ; CHECK-NEXT:    # Parent Loop BB2_2 Depth=2
 ; CHECK-NEXT:    # Parent Loop BB2_3 Depth=3
 ; CHECK-NEXT:    # => This Inner Loop Header: Depth=4
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    jmp .Ltmp1
 ; CHECK-NEXT:    jmp .Ltmp2
 ; CHECK-NEXT:    jmp .Ltmp3
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB2_5: # %normal0
+; CHECK-NEXT:  # %bb.5: # %normal0
 ; CHECK-NEXT:    # in Loop: Header=BB2_4 Depth=4
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    jmp .Ltmp1
 ; CHECK-NEXT:    jmp .Ltmp2
 ; CHECK-NEXT:    jmp .Ltmp3
 ; CHECK-NEXT:    jmp .Ltmp4
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB2_6: # %normal1
+; CHECK-NEXT:  # %bb.6: # %normal1
 ; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; CHECK-NEXT:    retl
 entry:
   %a.addr = alloca i32, align 4
   store i32 %a, i32* %a.addr, align 4
   br label %label01
 
 label01:                                          ; preds = %normal0, %label04, %entry
normal1:                                          ; preds = %normal0
   ret i32 %1
 }
 
 ; Test 4 - asm-goto referenced with the 'l' (ell) modifier and not.
 define void @test4() {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    ja .Ltmp5{{$}}
+; CHECK-NEXT:    ja .Ltmp5
 ; CHECK-NEXT:    #NO_APP
-; CHECK-NEXT:  .LBB3_1: # %asm.fallthrough
+; CHECK-NEXT:  # %bb.1: # %asm.fallthrough
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    ja .Ltmp5{{$}}
+; CHECK-NEXT:    ja .Ltmp5
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:  .Ltmp5: # Block address taken
 ; CHECK-NEXT:  .LBB3_3: # %quux
 ; CHECK-NEXT:    retl
 entry:
   callbr void asm sideeffect "ja $0", "X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test4, %quux))
           to label %asm.fallthrough [label %quux]
 

llvm/test/CodeGen/X86/shrinkwrap-callbr.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -enable-shrink-wrap=true | FileCheck %s
+
+;; Ensure that shrink-wrapping understands that INLINEASM_BR may exit
+;; the block before the end, and you cannot simply place stack
+;; adjustment at the end of that block.
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+declare i32 @fn()
+
+; Function Attrs: uwtable
+define i32 @test1(i32 %v) {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    testl %edi, %edi
+; CHECK-NEXT:    je .LBB0_3
+; CHECK-NEXT:  # %bb.1: # %if.end
+; CHECK-NEXT:    callq fn
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    # jump to .Ltmp0
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.2: # %return
+; CHECK-NEXT:    movl $4, %eax
+; CHECK-NEXT:    popq %rcx
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  .LBB0_3: # %ret0
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    popq %rcx
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  .Ltmp0: # Block address taken
+; CHECK-NEXT:  .LBB0_4: # %two
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    popq %rax
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    jmp fn # TAILCALL
+entry:
+  %tobool = icmp eq i32 %v, 0
+  br i1 %tobool, label %ret0, label %if.end
+
+ret0:
+  ret i32 0
+
+if.end:
+  %call = tail call i32 @fn()
+  callbr void asm sideeffect "# jump to $0", "X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test1, %two))
+          to label %return [label %two]
+
+two:
+  %call1 = tail call i32 @fn()
+  br label %return
+
+return:
+  %retval.1 = phi i32 [ %call1, %two ], [ 4, %if.end ]
+  ret i32 %retval.1
+}

llvm/test/Verifier/callbr.ll

 ; RUN: not opt -S %s -verify 2>&1 | FileCheck %s
 
 ; CHECK: Indirect label missing from arglist.
-define void @foo() {
+; CHECK-NEXT: #test1
+define void @test1() {
   ; The %4 in the indirect label list is not found in the blockaddresses in the
   ; arg list (bad).
-  callbr void asm sideeffect "${0:l} {1:l}", "X,X"(i8* blockaddress(@foo, %3), i8* blockaddress(@foo, %2))
+  callbr void asm sideeffect "#test1", "X,X"(i8* blockaddress(@test1, %3), i8* blockaddress(@test1, %2))
   to label %1 [label %4, label %2]
 1:
   ret void
 2:
   ret void
 3:
   ret void
 4:
   ret void
 }
 
 ; CHECK-NOT: Indirect label missing from arglist.
-define void @bar() {
+define void @test2() {
   ; %4 and %2 are both in the indirect label list and the arg list (good).
-  callbr void asm sideeffect "${0:l} ${1:l}", "X,X"(i8* blockaddress(@bar, %4), i8* blockaddress(@bar, %2))
+  callbr void asm sideeffect "${0:l} ${1:l}", "X,X"(i8* blockaddress(@test2, %4), i8* blockaddress(@test2, %2))
   to label %1 [label %4, label %2]
 1:
   ret void
 2:
   ret void
 3:
   ret void
 4:
   ret void
 }
 
 ; CHECK-NOT: Indirect label missing from arglist.
-define void @baz() {
+define void @test3() {
   ; note %2 blockaddress. Such a case is possible when passing the address of
   ; a label as an input to the inline asm (both address of label and asm goto
   ; use blockaddress constants; we're testing that the indirect label list from
   ; the asm goto is in the arg list to the asm).
-  callbr void asm sideeffect "${0:l} ${1:l} ${2:l}", "X,X,X"(i8* blockaddress(@baz, %4), i8* blockaddress(@baz, %2), i8* blockaddress(@baz, %3))
+  callbr void asm sideeffect "${0:l} ${1:l} ${2:l}", "X,X,X"(i8* blockaddress(@test3, %4), i8* blockaddress(@test3, %2), i8* blockaddress(@test3, %3))
   to label %1 [label %3, label %4]
 1:
   ret void
 2:
   ret void
 3:
   ret void
 4:
   ret void
 }
+
+;; Ensure you cannot use the return value of a callbr in indirect targets.
+; CHECK: Instruction does not dominate all uses!
+; CHECK-NEXT: #test4
+define i32 @test4(i1 %var) {
+entry:
+  %ret = callbr i32 asm sideeffect "#test4", "=r,X"(i8* blockaddress(@test4, %abnormal)) to label %normal [label %abnormal]
+
+normal:
+  ret i32 0
+
+abnormal:
+  ret i32 %ret
+}
+
+;; Ensure you cannot specify the same label as both normal and indirect targets.
+; CHECK: Duplicate callbr destination!
+; CHECK-NEXT: #test5
+define i32 @test5() {
+entry:
+  %ret = callbr i32 asm sideeffect "#test5", "=r,X"(i8* blockaddress(@test5, %both)) to label %both [label %both]
+
+both:
+  ret i32 0
+}