[RFC prototype] Implementation of asm-goto support in LLVM

[craig.topper]

Summary

This patch accompanies the RFC posted here:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/127239.html

Once there is a consensus wrt the design of the feature, I will modify
this patch and split it into pieces for a proper review.

Even though this patch is definitely a prototype, it was tested quite
well, including the heavy-lifting of building the linux kernel
(kudos to Nick Desaulniers for doing that!).

One known problem in the implementation that I would like to mention
is that terminating callbr instruction is translated into INLINEASM
machine instruction, which is not a terminator. There are several crude
workarounds that we had to put in the code (e.g. see AsmPrinter.cpp) for
that. It has to be addressed in a better way and I would appreciate your
thoughts on that.

Authored By: Alexander Ivchenko, Mikhail Dvoretckii

[nickdesaulniers]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3010

Is this comment about terminator instruction still relevant?

3016

This comment is also added to lib/CodeGen/MachineBasicBlock.cpp. What does that mean for "asm-goto sometimes seems to be the second-to-last operand?" Second to last operand to what?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2527–2528

These can probably be moved closer to their uses below?

lib/Transforms/Utils/LoopSimplify.cpp

134

From this part of the patch through below, there's multiple cases of:

if (indirectbr)
  foo();
if (callbr)
  foo();

That could be:

if (indirectbr || callbr)
  foo();

[craig.topper]

Attempt to fix a bug reported by @nickdesaulniers . We weren't using the label created for the blockaddress of the basic block when we printed the inline assembly block. So there was no guarantee the label we were using wouldn't be removed. To fix this I've kept the blockaddress object in the SelectionDAG instead of stripping it in the builder. I've created it directly as TargetBlockAddress to hide it from isel. Then in the printer I look for the operand being a blockaddress and grab the symbol for it. I had to make a few checks for TargetBlockAddress in other places to make this work. I admit I have no idea if this is the right thing to do at all, and open to any input on this.

[craig.topper]

lib/Bitcode/Reader/BitcodeReader.cpp

4278–4280

That would change the meaning of the else on the first if wouldn't it?

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3010

I think so. The CodeGen layer INLINEASM is not a terminator instruction.

3016

I think this meant "instruction" not "operand"

[craig.topper]

Address some review comments. Include context this time

[void]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3015–3016

This comment and the FIXME above it have me confused. Isn't the point of callbr to convert *all* asm goto statements into callbr? If so, then they should automatically be at the end of the basic block. What are the situations where it won't follow this?

[craig.topper]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3015–3016

This is MachineIR not IR. INLINEASM has its own MachineInstr opcode which is not a terminator. Currently asm-goto is still using that.

[void]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3015–3016

*smacks forehead* Of course. Nevermind.

[chandlerc]

Adding Hans so he is aware of the progress here.

[nickdesaulniers]

test/Bitcode/callbr.ll

2

This test case failed for me when running ninja check-all locally:

: 'RUN: at line 1';    llvm/build/bin/llvm-dis < llvm/test/Bitcode/callbr.ll.bc | llvm/build/bin/FileCheck llvm/test/Bitcode/callbr.ll
llvm/build/bin/llvm-dis: error: Invalid bitcode signature

[void]

I like these changes. Could you add the documentation for the new instruction?

[craig.topper]

Add some documentation to the LangRef. Probably needs additional work.

Try to upload the patch as binary so the bitcode file go through correctly.

Add MO_BlockAddress to X86AsmPrinter's printOperand to fix a crash on some invalid inline assembly reported by @jyu2.

[nickdesaulniers]

docs/LangRef.rst

6865

s/adress/address/

6871

s/calledd/called/

6876

s/callbrs/callbr/

[craig.topper]

docs/LangRef.rst

6865

That typo exists in Call and Invoke too. Fixed those in r351279

[craig.topper]

Fix LangRef typos

[void]

There appears to be an ICE with this patch: https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-454675962

[craig.topper]

In D53765#1360880, @void wrote:

There appears to be an ICE with this patch: https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-454675962

I think I see the issue. We need to return false for callbr from llvm::isSafeToSpeculativelyExecute. We probably also should fix to computeKnownBitsFromAssume to not visit an assume that is also the CxtI instruction. In that case we end up searching through the basic block until the terminator. And only because every terminator returns false from llvm::isSafeToSpeculativelyExecute do we end up terminating the loop without walking off the end of the basic block.

[craig.topper]

Add CallBr to isSafeToSpeculativelyExecute to keep isValidAssumeForContext from walking off the end of the basic block.

[nickdesaulniers]

A few additional errors we're hitting when compiling an x86_64 kernel with allyesconfig (all configurable options and drivers "enabled").
https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-455360304

Working to triage these and provide smaller test cases. Please do not land until we understand these build failures.

[craig.topper]

include/llvm/Bitcode/LLVMBitCodes.h

482

Can this use the 14 or does it need a new number? Not sure how holes get created here.

include/llvm/IR/Instructions.h

3880

I should probably drop this comment as I'm not sure what the previous author meant.

lib/Bitcode/Reader/BitcodeReader.cpp

4295

Should this be "(CCInfo >> bitc::CALL_CCONV) & CallingConv::MaxID" instead?

lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

440

I think this is an 80 column violation

lib/IR/Instruction.cpp

788

instructions is misspelled

lib/Transforms/InstCombine/InstCombineCalls.cpp

4203

Can this be merged using CallBase?

4223

Merge with above check for invoke

4339

Should we have an interface that doesn't require copying the transfer list? Just ask for a specific transfer number instead?

lib/Transforms/InstCombine/InstructionCombining.cpp

928

Should we merge these by doing isa<InvokeInst> || isa<CallBrInst> then just casting to Instruction to get the parent?

lib/Transforms/Utils/Local.cpp

1003

Merge the dyn_cast into the if

[nickdesaulniers]

This looks like a codegen bug, from compiling more of the Linux kernel. It seems that an optimization from -O1 is causing the problem: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. I'm not sure how to help narrow it down further (which pass?). The output is from creduce and I've narrowed down the set of flags to Clang.

lib/Transforms/Utils/InlineFunction.cpp

1509

While I can understand punting in v1, this does cause at least one -Wsection like warning from the kernel: https://github.com/ClangBuiltLinux/linux/issues/317. I call it section like, because the kernel has its own tool that it uses to validate sections; some run-once kernel code and data is put into init sections that get discarded after first use to help save kernel memory.

The issue is that this return false; makes __attribute__((always_inline)) fail. We can hack around that by converting always inline functions into macros, but I don't see that flying upstream.

So not the highest priority (it's just a warning, with no issues at runtime), but would be nice to have.

[craig.topper]

In D53765#1362744, @nickdesaulniers wrote:

This looks like a codegen bug, from compiling more of the Linux kernel. It seems that an optimization from -O1 is causing the problem: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. I'm not sure how to help narrow it down further (which pass?). The output is from creduce and I've narrowed down the set of flags to Clang.

You try using -opt-bisect-limit to narrow down the pass https://llvm.org/docs/OptBisect.html

[void]

I encountered another ICE here: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455456506

[craig.topper]

Fix most of my own review comments. Fix what was probably a bug in SimplifyCFG.

I have not looked at the inlinining issue, the link error, or the ICE from @void yet.

[craig.topper]

Hopefully fix https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. Had to exclude CallBr predecessors in JumpThreading.

[nickdesaulniers]

Hopefully fix https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. Had to exclude CallBr predecessors in JumpThreading.

Yes looks good; I've closed that issue as unreproducible.

For https://github.com/ClangBuiltLinux/linux/issues/317, I think we're hitting https://bugs.llvm.org/show_bug.cgi?id=39560. Reproducer https://gist.github.com/nickdesaulniers/a69b44159f4d42a34fc1b5d18cb7b608. If we fix that, then I should be able to drop my out of tree kernel patch.

Finally, https://github.com/ClangBuiltLinux/linux/issues/319 looks troubling, but I haven't really dug into it. It *may* be due to my out of tree kernel patch (https://github.com/ClangBuiltLinux/linux/commit/0670525ded48caf958b6535d388e6478b6b14a4b), but if we can support inlining then it looks like my out of tree patch should not be necessary: https://godbolt.org/z/exFrxS. Otherwise, it may be another codegen bug due to this patch.

[majnemer]

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

[craig.topper]

In D53765#1367170, @majnemer wrote:

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

No, and Verifier.cpp currently errors for it.

[majnemer]

In D53765#1367315, @craig.topper wrote:

In D53765#1367170, @majnemer wrote:

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

No, and Verifier.cpp currently errors for it.

I think it would be good to mention in this explicitly in the LangRef.

[craig.topper]

Make the LangRef statement about only supporting inline assembly a little stronger.

[craig.topper]

-Add a new isAsmGotoTarget flag to MachineBasicBlock. Use it to preserve predecessors/successors in a few places similar to isEHPad. Probably more work to do here, but I think it at least fixes https://github.com/ClangBuiltLinux/linux/issues/319
-Remove SelectionDAGBuilder support for doing anything with CallBr other than inline assembly. The code can't be tested today so there's really no point in having it. Assuming inline assembly only made setting the flag above a little simpler.

[nickdesaulniers]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2533

needs a (void)Callee; otherwise release builds produce:

../lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:2531:16: warning: unused variable 'Callee' [-Wunused-variable]
  const Value *Callee(I.getCalledValue());
               ^

or can the whole expression just fit in the assert?

[craig.topper]

Fix unused variable in release builds

[craig.topper]

Hopefully make callbr.ll.bc binary again

[craig.topper]

Really make it binary

[nickdesaulniers]

Add a new isAsmGotoTarget flag to MachineBasicBlock

Is there a way to add a test for this case that is fixed by this, so that it never regresses without setting off alarms?

[craig.topper]

-Block tail merging in BranchFolding.cpp if the predecessor block contains an asm goto. This is what we do for exception handling.

[craig.topper]

-Simplify the code in isBlockOnlyReachableByFallthrough by using the isAsmGotoTarget flag.
-Support isAsmGotoTarget() in MachineBasicBlock::print

[nickdesaulniers]

I think llvm is failing to reach a fixed point now: https://github.com/ClangBuiltLinux/linux/issues/319#issuecomment-457874051

[craig.topper]

-Add another isAsmGotoTarget check to BranchFolding.cpp in place where we already checked isEHPad. Seems to prevent hang on fault.c reported here https://github.com/ClangBuiltLinux/linux/issues/330

[craig.topper]

-Disable critical edge splitting for the non-fallthrough edge of a callbr. Otherwise the blockaddress and the control flow edge get out of sync.

[craig.topper]

-Disable GVN PRE across callbr. This prevents it from queueing up a critical edge split that we can't perform causing an infinite loop. Might be able to do the the fallthrough edge, but left fixme for now.

[craig.topper]

-Block GVN Load PRE for CallBr critical edges.

[nickdesaulniers]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2533

Still seeing this in Diff 183921.

[craig.topper]

Restore a lost fix for a unused variable warning

[craig.topper]

-Teach shrink wrapping pass to ensure the asm goto targets are at least at the boundary of prologue/epilogue. This is what we do for EHPad, but may not be quite right for asm-goto.

[chandlerc]

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

I haven't looked as closely at the actual optimizer changes yet as I'd like, but I think those are by-and-large going to be pretty straightforward.

include/llvm/IR/CallSite.h

53

Bleh. We should just finish killing CallSite. But I guess this patch shouldn't wait on that.

88–94

FWIW, I.getInt() == 0 seems way more clear.

260–261

Can this be simplified by just delegating to CallBase? The amount of magic constants / implicit contracts here worries me.

317–321

Can we just sink this into CallBase?

595–598

Delegate to CallBase?

[chandlerc]

include/llvm/IR/InstVisitor.h

273–279

Need to update this code path.

include/llvm/IR/Instructions.h

4033–4035

See above, this I think should sink to CallBase.

lib/AsmParser/LLParser.cpp

6197–6198

bundles missing here?

6197–6198

The syntax here diverges from both switch and indirectbr in surprising ways here. They don't use any keywords like or jump to introduce thi list of labels, they just directly put it there.

While abstractly, I somewhat like this syntax, I wonder if it would be better te *exactly* match the syntax for switch which has the same default label with a list of other labels structure to it. Consistency seems good here unless it causes problems?

6250

Not a small vector?

6269

Weird indentatino.

6279

I'd use int and llvm::seq, but maybe better to remain consistent.

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2973–2976

Isn't one of the targets a fallthrough? Maybe i've not made it far enough, and the default target is different from the target here.

lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

439–447

I'm a bit surprised we don't model this as an MBB operand to begin with if it *has* to be a direct block address anyways? Curious why.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1266

Maybe have the verifier reject if we can't lower here? Regardless, the comment still seems to need some amount of fixing.

deopt bundles are a reasonable thing, and just not really a reasonable thing (at this stage) to support on callbr.

1279

This complex to thread through?

lib/CodeGen/IndirectBrExpandPass.cpp

151–154

I think we have to fix this before this goes very far. I don't actually know how this is working in practice in our biggest customer (the Linux Kernel) as it should be using things like retpolines and needing this code to work.

I also think that this entire pass, while a reasonable thing to have, was a bad idea for retpoline lowering. We found retpolines to be faster than the branch trees created by this in most cases w/ many branches. So this needs a non-trivial update. Not sure who has the time to do this at this point though, I'll ask around w/ folks familiar with this bit of code.

It would be good to at least update the comment to English prose.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536

Wait, we handle catchswitch combined with this? That seems .... ambitious.

2540–2547

Weird that we handle probabilities here but not in Global ISel

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Wait, they do?

I don't understand -- the callbr is always a terminator. I understand that it may expand to lots of instructions, but isn't that opaque at this level?

lib/Target/X86/X86AsmPrinter.cpp

256–260

The fact that we have to add an entire machine operand layer here makes me feel more strongly that we should just be using the existing support for MBB machine operands rather than this.... But maybe there is a reason I'm just missing so far.

[craig.topper]

In D53765#1375112, @chandlerc wrote:

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

Maybe I went the wrong way on this, but here's what I saw. The existence of the BlockAddress instruction in IR caused some portion of MachineIR handling to know that it needed to create a label of the form .Ltmp[0-9]+. This is helpfully annotated in the output assembly as "block address taken" and sometimes when there's a bug "address taken of block removed by CodeGen". The first version of this patch I inherited stripped the BlockAddress when the SelectionDAG was created. The resulting assembly would then print the .LBB label for the block in the inline asm instead. We ended up in a case where the LBB label didn't exist in the final output, but the Ltmp did. This failed to assemble obviously. Maybe that was really pointing to passes like BranchFolding.cpp doing bad things which we encountered later. But at the time I tried to fix it so that the inline asm printing would reference the .Ltmp label because that seemed like the label we really wanted.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

I haven't looked as closely at the actual optimizer changes yet as I'd like, but I think those are by-and-large going to be pretty straightforward.

Maybe it shouldn't be special to asm-goto. But its clear that we've already been treating exception handling pads specially, but not address-taken blocks so I've just been special casing asm goto to avoid changing anything about existing address taken. I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2973–2976

The only blocks that are marked as isAsmGotoTarget are the ones that aren't the default. See the code in SelectionDAGBuilder.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1279

Original author copied that comment out of visitInvokeInst

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536

That was copy and pasted from Invoke. I'll remove it

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

This is MachineIR. CallBr isn't a thing there. CallBr was expanded to an INLINEASM instruction followed by an unconditional jump during SelectionDAGBuilder. The INLINEASM block is not a terminator so its effectively "the middle of a basic block". This pass tried to shove a RBP pop between the INLINEASM instruction and the unconditional jump. In the case that exposed the bug, the unconditional jump went to a block that also jumped to the same inline asm target. So as far as the dominator tree was concerned, the block containing the INLINEASM dominated the other blocks so it was chosen to hold the RBP pop. But since the INLINEASM itself can jump that wasn't correct.

[efriedma]

I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

Did you consider making blocks which contain an asm goto unanalyzable? Intuitively, that's what I would expect given an asm goto is a terminator at the IR level. But I guess that makes handling the fallthrough case efficiently more difficult?

[craig.topper]

lib/AsmParser/LLParser.cpp

6250

Call and Invoke both use std::vector here...

[craig.topper]

Address some of Chandler's comments

[chandlerc]

Looking at updated patch and some of the optimizer bits, but some inline responses to discussion points below:

In D53765#1375547, @craig.topper wrote:

In D53765#1375112, @chandlerc wrote:

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

Maybe I went the wrong way on this, but here's what I saw. The existence of the BlockAddress instruction in IR caused some portion of MachineIR handling to know that it needed to create a label of the form .Ltmp[0-9]+. This is helpfully annotated in the output assembly as "block address taken" and sometimes when there's a bug "address taken of block removed by CodeGen". The first version of this patch I inherited stripped the BlockAddress when the SelectionDAG was created. The resulting assembly would then print the .LBB label for the block in the inline asm instead. We ended up in a case where the LBB label didn't exist in the final output, but the Ltmp did. This failed to assemble obviously. Maybe that was really pointing to passes like BranchFolding.cpp doing bad things which we encountered later. But at the time I tried to fix it so that the inline asm printing would reference the .Ltmp label because that seemed like the label we really wanted.

I've spent more time thinking about this. I think the difference here is the following... With the "block address" behavior we assume the address more than taken -- it is *captured* and its uses may be unanalyzable. As a consequence, it *cannot* be updated when we do things like rework the control flow.

On the flip side, the MBB operands are likely often assumed to be easily enumerated and updated when we're rewriting the control flow, and so the fact that we can't do that results in breakages.

At least, this is my understanding. Match yours?

Given this, I think the current "block address" stuff may be correct -- I don't see anything that indicates the asm-goto couldn't escape or even capture the addresses of its successors and then re-use them in some way, expecting them to be "real" addresses. If this also makes sense to you, likely good to add some comments for future readers.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

Maybe it shouldn't be special to asm-goto. But its clear that we've already been treating exception handling pads specially, but not address-taken blocks so I've just been special casing asm goto to avoid changing anything about existing address taken. I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

Ugh, yeah.

I think that we should consolidate all of this behind something like address-taken, but I think that can be follow-up work. I suspect that a decent number of the things you've fixed here are actually latent bugs w/ indirectbr and address-of-label extensions that we've just never managed to hit. =[[[[

But I think it's fine to fix that incrementally going forward by digging in and commoning these representations as much as possible.

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Ah, that explains the thing I was missing...

I had assumed that we would expand this to an INLINEASM instruction that *is* a terminator. In MIR we can have multiple terminators at the end of a block, and this is a common pattern, so could we not have both instructions be marked as terminators and avoid special casing this?

[chandlerc]

Made it through the optimizer code. Really minor changes here, I think this is looking good. Probable the biggest question marks are in the MI representation.

Also, how best to track on-going work that is going to be prioritized in the short term? Mostyl looking for what would be helpful for you all if the FIXME suggested below (or already in the code) are likely to be lost.

lib/Transforms/InstCombine/InstCombineCalls.cpp

4217–4219

Check for .isTerminator() would I think make this a bit more self-explanatory.

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925

May also be a tiny bit clearer by using terminator based terminology and APIs.

lib/Transforms/Scalar/GVN.cpp

2177–2184

Can just directly test on the CallBase now.

lib/Transforms/Utils/LoopSimplify.cpp

245–246

Ok, I've seen this pattern too many times. ;]

I'd suggest adding isIndirectTerminator or hasIndirectControlFlow or some such?

lib/Transforms/Utils/SimplifyCFG.cpp

1269–1270

FIXME as we can likely build an analogous safety predicate to the invoke one for CallBr.

1451–1456

Fold into a test on CallBase.

[craig.topper]

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

It looks like whether an instruction is a terminator is currently encoded in the MCInstrDesc entry corresponding to an opcode. So I think that means we would need a new flavor of INLINEASM opcode?

[chandlerc]

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Bleh, yeah.

I think that's (marginally) cleaner, but I have no idea how much work it is to thread an INLINEASM_BR node through the backend. If it's too much, could defer this.

Reason that I think it is cleaner: I suspect we'll keep finding ways in which we mishandle this, but if it turns into a terminator sequence I think the rest of the backend will start to behave correctly.

[craig.topper]

-More review comments addressed.
-Rebase after committing some of the CallBase related changes

[craig.topper]

Update the CallBrInst creation APIs to consistently accept a callee-type argument. This makes CallBrInst consistent with call and invoke after r351121

[craig.topper]

Add a bailout for asm goto target to Machine Block Placement. Fixes export4.zip from here https://github.com/ClangBuiltLinux/linux/issues/336

[craig.topper]

Update terminology to replace Transfers with IndirectDests. And Fallthrough with DefaultDest

[craig.topper]

Rebase

[craig.topper]

Update CallBrInst creation interfaces to use FunctionCallee instead of Function* to match CallInst and InvokeInst

[chandlerc]

A few minor comments.

I assume the big outstanding item here is figuring out the MIR representation? Anything else?

docs/LangRef.rst

6900

Maybe "branch to" or "jump to"? I don't want to imply that it has to be done via call/return.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1266

(still seems to need update)

1279

(comment remains)

lib/CodeGen/IndirectBrExpandPass.cpp

151–154

FWIW, I think I now understand -- I think the Linux kernel tester we have isn't building w/ retpolines enabled in the compiler, so we just haven't seen this explode. =[

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536

(still needs fixing)

[craig.topper]

include/llvm/Bitcode/LLVMBitCodes.h

482

@chandlerc do you know the answer to the question I left here?

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925

I'm just going to remove this change because it would require a CallBrInst that returned a value, but we don't support that right now.

[craig.topper]

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925

Oops I meant to remove this comment. I've now fixed the code to use terminator terminology and apis

[chandlerc]

include/llvm/Bitcode/LLVMBitCodes.h

482

Not directly... But I think I can explain how to answer it.

Does any version of LLVM bitcode that we still support reading use 14? If so, then we need to leave it alone. If not, I think you can take it.

But maybe good to double check w/ someone who has thought more fully about bitcode backwards compat.

[craig.topper]

lib/CodeGen/GlobalISel/IRTranslator.cpp

1266

Turns out this comment was copied directly from the InvokeInst handler above.

1279

Does global isel do anything with probabilities anywhere else? I can't see any code obviously doing it. SelectionDAGBuilder has some helper functions that we used for CallBr, but I don't see equivalent helpers here or any calls passing probability to addSuccessor.

[craig.topper]

-Change the bitcode encoding to a value that's never been used so I don't have to figure out the compatibility story.
-Add the bitcode value to the llvm-bcanalyzer
-Fix "returns" in LangRef
-Remove comment from SelectionDAGBuilder

[craig.topper]

-Update vim syntax highlight file to include 'callbr'
-Add test cases for the 2 GVN hangs related to critical edge splitting that I had fixed previously.

[kees]

I found a weird mis-compilation bug. Not sure if in LLVM or Clang half. Details here: https://reviews.llvm.org/D56571#1386973

[craig.topper]

-Fix crash in bitcode writer when writing out callbr with attributes
-Add test backend test cases and jump threading test case for previously fixed bugs

[craig.topper]

-Remove GlobalISel implementation completely. It wasn't even trying to be correct for asm-goto. It assumed a call to a function not InlineAsm. I think we can deal with this later.
-Introduce an INLINEASM_BR instruction in machine IR that is like INLINEASM but is also a terminator. Hopefully I got the right flags set on this.
-Remove isAsmGotoTarget/hasAsmGotoTargetSuccessor from MachineBasicBlock

From visual inspection of the existing tests this seems to be working, but we'll see what the kernel builds have to say

[kees]

In D53765#1386975, @kees wrote:

I found a weird mis-compilation bug. Not sure if in LLVM or Clang half. Details here: https://reviews.llvm.org/D56571#1386973

This appears to be fixed with latest llvm, clang, and D53765.185703.

[chandlerc]

This is looking really close. I think we should get some basic sanity checks w/ the kernel, and even if there is an obscure bug or two, land this and move to follow-up patches.

So, LGTM pending some basic sanity checks and addressing some nits below.

include/llvm/Analysis/SparsePropagation.h

333–334

Use TI.isIndirectTerminator() here?

lib/IR/Instruction.cpp

788

This should probably be isa<CallBase> now with a cleaner string, but not critical to fix in this patch.

lib/Transforms/InstCombine/InstCombineCalls.cpp

9

nit: I always prefer the Oxford comma to reduce ambiguity.

4329–4345

Wow this seems crazily expensive.

I guess it's OK because currently it can only occur for invoke. But the big-O here is quadratic: NumberOfUses * NumberOfSuccessors.

To make matters worse, this change moves to the generic successor APIs which will put a type check in the inner loop. =[

Maybe the simpler approach is to leave the old code for Invoke, and for CallBr simply return false if there are >0 users (which should be never currently).

[craig.topper]

Address the last round of comments from Chandler. I'll commit this shortly.

[nickdesaulniers]

With
clang asm goto diff 185489
llvm asm goto diff 185868
I see a few more warnings with the kernel's x86 object file validator (that has found bugs previously with this patch set):
https://github.com/ClangBuiltLinux/linux/issues/351
https://github.com/ClangBuiltLinux/linux/issues/336#issuecomment-461631864

I'll try to provide reproducers for these ASAP.

• EDIT **

Looks unrelated to this patch.

[nickdesaulniers]

llvm/trunk/lib/Transforms/InstCombine/InstCombineCalls.cpp

4353 ↗

(On Diff #186034)

[582/1886] Building CXX object lib/Transf...LVMInstCombine.dir/InstCombineCalls.cpp.o
../lib/Transforms/InstCombine/InstCombineCalls.cpp: In member function ‘bool llvm::InstCombiner::transformConstExprCastCall(llvm::CallBase&)’:
../lib/Transforms/InstCombine/InstCombineCalls.cpp:4353:23: warning: unused variable ‘CBI’ [-Wunused-variable]
       if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller))
                       ^~~

[nickdesaulniers]

This comment was removed by nickdesaulniers.

[nickdesaulniers]

llvm/trunk/lib/Transforms/InstCombine/InstCombineCalls.cpp

4353 ↗

(On Diff #186034)

Fixed in r353630.
https://github.com/llvm/llvm-project/commit/a97857b5b536179f79e9616235905cab8241eb68

docs/LangRef.rst

 block should be executed after the current block is finished. These
 terminator instructions typically yield a '``void``' value: they produce
 control flow, not values (the one exception being the
 ':ref:`invoke <i_invoke>`' instruction).
 
 The terminator instructions are: ':ref:`ret <i_ret>`',
 ':ref:`br <i_br>`', ':ref:`switch <i_switch>`',
 ':ref:`indirectbr <i_indirectbr>`', ':ref:`invoke <i_invoke>`',
+':ref:`callbr <i_callbr>`'
 ':ref:`resume <i_resume>`', ':ref:`catchswitch <i_catchswitch>`',
 ':ref:`catchret <i_catchret>`',
 ':ref:`cleanupret <i_cleanupret>`',
 and ':ref:`unreachable <i_unreachable>`'.
 
 .. _i_ret:
 
 '``ret``' Instruction

 
 .. code-block:: llvm
 
       %retval = invoke i32 @Test(i32 15) to label %Continue
                   unwind label %TestCleanup              ; i32:retval set
       %retval = invoke coldcc i32 %Testfnptr(i32 15) to label %Continue
                   unwind label %TestCleanup              ; i32:retval set
 
+.. _i_callbr:
+
+'``callbr``' Instruction
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      <result> = callbr [cconv] [ret attrs] [addrspace(<num>)] [<ty>|<fnty> <fnptrval>(<function args>) [fn attrs]
+                    [operand bundles] to label <normal label> or jump [other labels]
+
+Overview:
+"""""""""
+
+The '``callbr``' instruction causes control to transfer to a specified
+function, with the possibility of control flow transfer to either the
+'``normal``' label or one of the '``other``' labels.
+
+This instruction should only be used to implement the "goto" feature of gcc
+style inline assembly. Any other usage is an error in the IR verifier.
+
+Arguments:
+""""""""""
+
+This instruction requires several arguments:
+
+#. The optional "cconv" marker indicates which :ref:`calling
+   convention <callingconv>` the call should use. If none is
+   specified, the call defaults to using C calling conventions.
+#. The optional :ref:`Parameter Attributes <paramattrs>` list for return
+   values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
+   are valid here.
+#. The optional addrspace attribute can be used to indicate the address space
+   of the called function. If it is not specified, the program address space
+   from the :ref:`datalayout string<langref_datalayout>` will be used.
+#. '``ty``': the type of the call instruction itself which is also the
+   type of the return value. Functions that return no value are marked
+   ``void``.
+#. '``fnty``': shall be the signature of the function being called. The
+   argument types must match the types implied by this signature. This
+   type can be omitted if the function is not varargs.
+#. '``fnptrval``': An LLVM value containing a pointer to a function to
+   be called. In most cases, this is a direct function call, but
+   indirect ``callbr``'s are just as possible, calling an arbitrary pointer
+   to function value.
+#. '``function args``': argument list whose types match the function
+   signature argument types and parameter attributes. All arguments must
+   be of :ref:`first class <t_firstclass>` type. If the function signature
+   indicates the function accepts a variable number of arguments, the
+   extra arguments can be specified.
+#. '``normal label``': the label reached when the called function
+   executes a '``ret``' instruction.
+#. '``other labels``': the labels reached when a callee transfers control
+   to a location other than the normal '``normal label``'
+#. The optional :ref:`function attributes <fnattrs>` list.
+#. The optional :ref:`operand bundles <opbundles>` list.
+
+Semantics:
+""""""""""
+
+This instruction is designed to operate as a standard '``call``'
+instruction in most regards. The primary difference is that it
+establishes an association with additional labels to define where control
+flow goes after the call.
+
+The only use of this today is to implement the "goto" feature of gcc inline
+assembly where additional labels can be provided as locations for the inline
+assembly to return to.
+
+Example:
+""""""""
+
+.. code-block:: llvm
+
+      callbr void asm "", "r,x"(i32 %x, i8 *blockaddress(@foo, %fail))
+                  to label %normal or jump [label %fail]
+
 .. _i_resume:
 
 '``resume``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 

include/llvm-c/Core.h

typedef enum {
   /* Terminator Instructions */
   LLVMRet            = 1,
   LLVMBr             = 2,
   LLVMSwitch         = 3,
   LLVMIndirectBr     = 4,
   LLVMInvoke         = 5,
   /* removed 6 due to API changes */
   LLVMUnreachable    = 7,
+  LLVMCallBr         = 67,
 
   /* Standard Unary Operators */
   LLVMFNeg           = 66,
 
   /* Standard Binary Operators */
   LLVMAdd            = 8,
   LLVMFAdd           = 9,
   LLVMSub            = 10,

include/llvm/Analysis/SparsePropagation.h

if (!C || !isa<ConstantInt>(C)) {
       return;
     }
 
     // Constant condition variables mean the branch can only go a single way
     Succs[C->isNullValue()] = true;
     return;
   }
 
-  if (TI.isExceptionalTerminator()) {
-    Succs.assign(Succs.size(), true);
-    return;
-  }
-
-  if (isa<IndirectBrInst>(TI)) {
+  if (TI.isExceptionalTerminator() ||
+      isa<IndirectBrInst>(TI) ||
+      isa<CallBrInst>(TI)) {
     Succs.assign(Succs.size(), true);
     return;
   }
 
   SwitchInst &SI = cast<SwitchInst>(TI);
   LatticeVal SCValue;
   if (AggressiveUndef)
     SCValue = getValueState(KeyInfo::getLatticeKeyFromValue(SI.getCondition()));
   else
     SCValue = getExistingValueState(
         KeyInfo::getLatticeKeyFromValue(SI.getCondition()));
 
   if (SCValue == LatticeFunc->getOverdefinedVal() ||
       SCValue == LatticeFunc->getUntrackedVal()) {
     // All destinations are executable!
     Succs.assign(TI.getNumSuccessors(), true);
     return;

include/llvm/Bitcode/LLVMBitCodes.h

enum FunctionCodes {
   FUNC_CODE_INST_INSERTELT = 7,  // INSERTELT:  [ty, opval, opval, opval]
   FUNC_CODE_INST_SHUFFLEVEC = 8, // SHUFFLEVEC: [ty, opval, opval, opval]
   FUNC_CODE_INST_CMP = 9,        // CMP:        [opty, opval, opval, pred]
 
   FUNC_CODE_INST_RET = 10,    // RET:        [opty,opval<both optional>]
   FUNC_CODE_INST_BR = 11,     // BR:         [bb#, bb#, cond] or [bb#]
   FUNC_CODE_INST_SWITCH = 12, // SWITCH:     [opty, op0, op1, ...]
   FUNC_CODE_INST_INVOKE = 13, // INVOKE:     [attr, fnty, op0,op1, ...]
-  // 14 is unused.
+  FUNC_CODE_INST_CALLBR = 14, // CALLBR:     [attr, cc, norm, transfs,
+                              //              fnty, fnid, args...]
   FUNC_CODE_INST_UNREACHABLE = 15, // UNREACHABLE
 
   FUNC_CODE_INST_PHI = 16, // PHI:        [ty, val0,bb0, ...]
   // 17 is unused.
   // 18 is unused.
   FUNC_CODE_INST_ALLOCA = 19, // ALLOCA:     [instty, opty, op, align]
   FUNC_CODE_INST_LOAD = 20,   // LOAD:       [opty, op, align, vol]
   // 21 is unused.
   // 22 is unused.
   FUNC_CODE_INST_VAARG = 23, // VAARG:      [valistty, valist, instty]
   // This store code encodes the pointer type, rather than the value type

include/llvm/CodeGen/GlobalISel/IRTranslator.h

bool valueIsSplit(const Value &V,
                     SmallVectorImpl<uint64_t> *Offsets = nullptr);
 
   /// Translate call instruction.
   /// \pre \p U is a call instruction.
   bool translateCall(const User &U, MachineIRBuilder &MIRBuilder);
 
   bool translateInvoke(const User &U, MachineIRBuilder &MIRBuilder);
 
+  bool translateCallBr(const User &U, MachineIRBuilder &MIRBuilder);
+
   bool translateLandingPad(const User &U, MachineIRBuilder &MIRBuilder);
 
   /// Translate one of LLVM's cast instructions into MachineInstrs, with the
   /// given generic Opcode.
   bool translateCast(unsigned Opcode, const User &U,
                      MachineIRBuilder &MIRBuilder);
 
   /// Translate a phi instruction.

include/llvm/CodeGen/MachineBasicBlock.h

private:
   bool IsEHScopeEntry = false;
 
   /// Indicate that this basic block is the entry block of an EH funclet.
   bool IsEHFuncletEntry = false;
 
   /// Indicate that this basic block is the entry block of a cleanup funclet.
   bool IsCleanupFuncletEntry = false;
 
+  /// Indicate that this basic block is the target of an asm-goto.
+  bool IsAsmGotoTarget = false;
+
   /// since getSymbol is a relatively heavy-weight operation, the symbol
   /// is only computed once and is cached.
   mutable MCSymbol *CachedMCSymbol = nullptr;
 
   // Intrusive list support
   MachineBasicBlock() = default;
 
   explicit MachineBasicBlock(MachineFunction &MF, const BasicBlock *BB);
#endif
   void setIsEHFuncletEntry(bool V = true) { IsEHFuncletEntry = V; }
 
   /// Returns true if this is the entry block of a cleanup funclet.
   bool isCleanupFuncletEntry() const { return IsCleanupFuncletEntry; }
 
   /// Indicates if this is the entry block of a cleanup funclet.
   void setIsCleanupFuncletEntry(bool V = true) { IsCleanupFuncletEntry = V; }
 
+  /// Returns true if the block is an asm-goto target. That is the block might
+  /// be jumped to by inline assembly.
+  bool isAsmGotoTarget() const { return IsAsmGotoTarget; }
+
+  /// Indicates this block an asm-goto target. That is the block might be
+  /// jumped to by inline assembly.
+  void setIsAsmGotoTarget(bool V = true) { IsAsmGotoTarget = V; }
+
+  bool hasAsmGotoTargetSuccessor() const;
+
   /// Returns true if it is legal to hoist instructions into this block.
   bool isLegalToHoistInto() const;
 
   // Code Layout methods.
 
   /// Move 'this' block before or after the specified block.  This only moves
   /// the block, it does not modify the CFG or adjust potential fall-throughs at
   /// the end of the block.

include/llvm/IR/CallSite.h

 //===- CallSite.h - Abstract Call & Invoke instrs ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the CallSite class, which is a handy wrapper for code that
-// wants to treat Call and Invoke instructions in a generic way. When in non-
-// mutation context (e.g. an analysis) ImmutableCallSite should be used.
+// wants to treat Call, Invoke and CallBr instructions in a generic way. When
+// in non-mutation context (e.g. an analysis) ImmutableCallSite should be used.
 // Finally, when some degree of customization is necessary between these two
 // extremes, CallSiteBase<> can be supplied with fine-tuned parameters.
 //
 // NOTE: These classes are supposed to have "value semantics". So they should be
 // passed by value, not by reference; they should not be "new"ed or "delete"d.
 // They are efficiently copyable, assignable and constructable, with cost
 // equivalent to copying a pointer (notice that they have only a single data
 // member). The internal representation carries a flag which indicates which of
-// the two variants is enclosed. This allows for cheaper checks when various
+// the three variants is enclosed. This allows for cheaper checks when various
 // accessors of CallSite are employed.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_CALLSITE_H
 #define LLVM_IR_CALLSITE_H
 
 #include "llvm/ADT/Optional.h"

 #include <iterator>
 
 namespace llvm {
 
 namespace Intrinsic {
 enum ID : unsigned;
 }
 
-template <typename FunTy = const Function,
-          typename BBTy = const BasicBlock,
-          typename ValTy = const Value,
-          typename UserTy = const User,
-          typename UseTy = const Use,
-          typename InstrTy = const Instruction,
+template <typename FunTy = const Function, typename BBTy = const BasicBlock,
+          typename ValTy = const Value, typename UserTy = const User,
+          typename UseTy = const Use, typename InstrTy = const Instruction,
           typename CallTy = const CallInst,
           typename InvokeTy = const InvokeInst,
+          typename CallBrTy = const CallBrInst,
           typename IterTy = User::const_op_iterator>
 class CallSiteBase {
 protected:
-  PointerIntPair<InstrTy*, 1, bool> I;
+  PointerIntPair<InstrTy *, 2, int> I;
 
   CallSiteBase() = default;
-  CallSiteBase(CallTy *CI) : I(CI, true) { assert(CI); }
-  CallSiteBase(InvokeTy *II) : I(II, false) { assert(II); }
+  CallSiteBase(CallTy *CI) : I(CI, 1) { assert(CI); }
+  CallSiteBase(InvokeTy *II) : I(II, 0) { assert(II); }
+  CallSiteBase(CallBrTy *CBI) : I(CBI, 2) { assert(CBI); }
   explicit CallSiteBase(ValTy *II) { *this = get(II); }
 
 private:
   /// This static method is like a constructor. It will create an appropriate
-  /// call site for a Call or Invoke instruction, but it can also create a null
-  /// initialized CallSiteBase object for something which is NOT a call site.
+  /// call site for a Call, Invoke or CallBr instruction, but it can also create
+  /// a null initialized CallSiteBase object for something which is NOT a call
+  /// site.
   static CallSiteBase get(ValTy *V) {
     if (InstrTy *II = dyn_cast<InstrTy>(V)) {
       if (II->getOpcode() == Instruction::Call)
         return CallSiteBase(static_cast<CallTy*>(II));
-      else if (II->getOpcode() == Instruction::Invoke)
+      if (II->getOpcode() == Instruction::Invoke)
         return CallSiteBase(static_cast<InvokeTy*>(II));
+      if (II->getOpcode() == Instruction::CallBr)
+        return CallSiteBase(static_cast<CallBrTy *>(II));
     }
     return CallSiteBase();
   }
 
 public:
-  /// Return true if a CallInst is enclosed. Note that !isCall() does not mean
-  /// an InvokeInst is enclosed. It may also signify a NULL instruction pointer.
-  bool isCall() const { return I.getInt(); }
+  /// Return true if a CallInst is enclosed.
+  bool isCall() const { return I.getInt() == 1; }
 
-  /// Return true if a InvokeInst is enclosed.
-  bool isInvoke() const { return getInstruction() && !I.getInt(); }
+  /// Return true if a InvokeInst is enclosed. !I.getInt() may also signify a
+  /// NULL instruction pointer, so check that.
+  bool isInvoke() const { return getInstruction() && I.getInt() == 0; }
+
+  /// Return true if a CallBrInst is enclosed.
+  bool isCallBr() const { return I.getInt() == 2; }
 
   InstrTy *getInstruction() const { return I.getPointer(); }
   InstrTy *operator->() const { return I.getPointer(); }
   explicit operator bool() const { return I.getPointer(); }
 
   /// Get the basic block containing the call site.
   BBTy* getParent() const { return getInstruction()->getParent(); }
 
   /// Return the pointer to function that is being called.
   ValTy *getCalledValue() const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     return *getCallee();
   }
 
   /// Return the function being called if this is a direct call, otherwise
   /// return null (if it's an indirect call).
   FunTy *getCalledFunction() const {
     return dyn_cast<FunTy>(getCalledValue());
   }
 
   /// Return true if the callsite is an indirect call.
   bool isIndirectCall() const {
     const Value *V = getCalledValue();
     if (!V)
       return false;
     if (isa<FunTy>(V) || isa<Constant>(V))
       return false;
     if (const CallInst *CI = dyn_cast<CallInst>(getInstruction())) {
       if (CI->isInlineAsm())
         return false;
     }
+    if (const CallBrInst *CBI = dyn_cast<CallBrInst>(getInstruction())) {
+      if (CBI->isInlineAsm())
+        return false;
+    }
     return true;
   }
 
   /// Set the callee to the specified value.
   void setCalledFunction(Value *V) {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     *getCallee() = V;
   }
 
   /// Return the intrinsic ID of the intrinsic called by this CallSite,
   /// or Intrinsic::not_intrinsic if the called function is not an
   /// intrinsic, or if this CallSite is an indirect call.
   Intrinsic::ID getIntrinsicID() const {
     if (auto *F = getCalledFunction())
public:
   }
 
   ValTy *getArgument(unsigned ArgNo) const {
     assert(arg_begin() + ArgNo < arg_end() && "Argument # out of range!");
     return *(arg_begin() + ArgNo);
   }
 
   void setArgument(unsigned ArgNo, Value* newVal) {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(arg_begin() + ArgNo < arg_end() && "Argument # out of range!");
     getInstruction()->setOperand(ArgNo, newVal);
   }
 
   /// Given a value use iterator, returns the argument that corresponds to it.
   /// Iterator must actually correspond to an argument.
   unsigned getArgumentNo(Value::const_user_iterator I) const {
     return getArgumentNo(&I.getUse());
   }
 
   /// Given a use for an argument, get the argument number that corresponds to
   /// it.
   unsigned getArgumentNo(const Use *U) const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(isArgOperand(U) && "Argument # out of range!");
     return U - arg_begin();
   }
 
   /// The type of iterator to use when looping over actual arguments at this
   /// call site.
   using arg_iterator = IterTy;
 
   iterator_range<IterTy> args() const {
     return make_range(arg_begin(), arg_end());
   }
   bool arg_empty() const { return arg_end() == arg_begin(); }
   unsigned arg_size() const { return unsigned(arg_end() - arg_begin()); }
 
   /// Given a value use iterator, return the data operand corresponding to it.
   /// Iterator must actually correspond to a data operand.
   unsigned getDataOperandNo(Value::const_user_iterator UI) const {
     return getDataOperandNo(&UI.getUse());
   }
 
   /// Given a use for a data operand, get the data operand number that
   /// corresponds to it.
   unsigned getDataOperandNo(const Use *U) const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(isDataOperand(U) && "Data operand # out of range!");
     return U - data_operands_begin();
   }
 
   /// Type of iterator to use when looping over data operands at this call site
   /// (see below).
   using data_operand_iterator = IterTy;
 
   /// data_operands_begin/data_operands_end - Return iterators iterating over
-  /// the call / invoke argument list and bundle operands.  For invokes, this is
-  /// the set of instruction operands except the invoke target and the two
-  /// successor blocks; and for calls this is the set of instruction operands
-  /// except the call target.
+  /// the call / invoke / callbr argument list and bundle operands. For invokes,
+  /// this is the set of instruction operands except the invoke target and the
+  /// two successor blocks; for calls this is the set of instruction operands
+  /// except the call target; for callbrs the number of labels to skip must be
+  /// determined first.
 
   IterTy data_operands_begin() const {
     assert(getInstruction() && "Not a call or invoke instruction!");
-    return (*this)->op_begin();
+    return cast<CallBase>(getInstruction())->data_operands_begin();
   }
   IterTy data_operands_end() const {
     assert(getInstruction() && "Not a call or invoke instruction!");
-    return (*this)->op_end() - (isCall() ? 1 : 3);
+    return cast<CallBase>(getInstruction())->data_operands_end();
   }
   iterator_range<IterTy> data_ops() const {
     return make_range(data_operands_begin(), data_operands_end());
   }
   bool data_operands_empty() const {
     return data_operands_end() == data_operands_begin();
   }
   unsigned data_operands_size() const {
     return std::distance(data_operands_begin(), data_operands_end());
bool isMustTailCall() const {
     return isCall() && cast<CallInst>(getInstruction())->isMustTailCall();
   }
 
   /// Tests if this call site is marked as a tail call.
   bool isTailCall() const {
     return isCall() && cast<CallInst>(getInstruction())->isTailCall();
   }
 
-#define CALLSITE_DELEGATE_GETTER(METHOD) \
-  InstrTy *II = getInstruction();    \
-  return isCall()                        \
-    ? cast<CallInst>(II)->METHOD         \
+#define CALLSITE_DELEGATE_GETTER(METHOD)                                       \
+  InstrTy *II = getInstruction();                                              \
+  return isCall() ? cast<CallInst>(II)->METHOD                                 \
+                  : isCallBr() ? cast<CallBrInst>(II)->METHOD                  \
                                 : cast<InvokeInst>(II)->METHOD
 
-#define CALLSITE_DELEGATE_SETTER(METHOD) \
-  InstrTy *II = getInstruction();    \
-  if (isCall())                          \
-    cast<CallInst>(II)->METHOD;          \
-  else                                   \
+#define CALLSITE_DELEGATE_SETTER(METHOD)                                       \
+  InstrTy *II = getInstruction();                                              \
+  if (isCall())                                                                \
+    cast<CallInst>(II)->METHOD;                                                \
+  else if (isCallBr())                                                         \
+    cast<CallBrInst>(II)->METHOD;                                              \
+  else                                                                         \
     cast<InvokeInst>(II)->METHOD
 
   unsigned getNumArgOperands() const {
     CALLSITE_DELEGATE_GETTER(getNumArgOperands());
   }
 
   ValTy *getArgOperand(unsigned i) const {
     CALLSITE_DELEGATE_GETTER(getArgOperand(i));
   }
 
   ValTy *getReturnedArgOperand() const {
     CALLSITE_DELEGATE_GETTER(getReturnedArgOperand());
   }
 
   bool isInlineAsm() const {
     if (isCall())
       return cast<CallInst>(getInstruction())->isInlineAsm();
+    if (isCallBr())
+      return cast<CallBrInst>(getInstruction())->isInlineAsm();
     return false;
   }
 
   /// Get the calling convention of the call.
   CallingConv::ID getCallingConv() const {
     CALLSITE_DELEGATE_GETTER(getCallingConv());
   }
   /// Set the calling convention of the call.
   void setCallingConv(CallingConv::ID CC) {
#define CALLSITE_DELEGATE_SETTER(METHOD)                                       \
 
   Attribute getAttribute(unsigned i, StringRef Kind) const {
     CALLSITE_DELEGATE_GETTER(getAttribute(i, Kind));
   }
 
   /// Return true if the data operand at index \p i directly or indirectly has
   /// the attribute \p A.
   ///
-  /// Normal call or invoke arguments have per operand attributes, as specified
-  /// in the attribute set attached to this instruction, while operand bundle
-  /// operands may have some attributes implied by the type of its containing
-  /// operand bundle.
+  /// Normal call, invoke or callbr arguments have per operand attributes, as
+  /// specified in the attribute set attached to this instruction, while operand
+  /// bundle operands may have some attributes implied by the type of its
+  /// containing operand bundle.
   bool dataOperandHasImpliedAttr(unsigned i, Attribute::AttrKind Kind) const {
     CALLSITE_DELEGATE_GETTER(dataOperandHasImpliedAttr(i, Kind));
   }
 
   /// Extract the alignment of the return value.
   unsigned getRetAlignment() const {
     CALLSITE_DELEGATE_GETTER(getRetAlignment());
   }
#define CALLSITE_DELEGATE_SETTER(METHOD)                                       \
 
   IterTy arg_end() const {
     CALLSITE_DELEGATE_GETTER(arg_end());
   }
 
 #undef CALLSITE_DELEGATE_GETTER
 #undef CALLSITE_DELEGATE_SETTER
 
   void getOperandBundlesAsDefs(SmallVectorImpl<OperandBundleDef> &Defs) const {
-    const Instruction *II = getInstruction();
     // Since this is actually a getter that "looks like" a setter, don't use the
     // above macros to avoid confusion.
-    if (isCall())
-      cast<CallInst>(II)->getOperandBundlesAsDefs(Defs);
-    else
-      cast<InvokeInst>(II)->getOperandBundlesAsDefs(Defs);
+    cast<CallBase>(getInstruction())->getOperandBundlesAsDefs(Defs);
   }
 
   /// Determine whether this data operand is not captured.
   bool doesNotCapture(unsigned OpNo) const {
     return dataOperandHasImpliedAttr(OpNo + 1, Attribute::NoCapture);
   }
 
   /// Determine whether this argument is passed by value.

 private:
   IterTy getCallee() const {
     return cast<CallBase>(getInstruction())->op_end() - 1;
   }
 };
 
 class CallSite : public CallSiteBase<Function, BasicBlock, Value, User, Use,
                                      Instruction, CallInst, InvokeInst,
-                                     User::op_iterator> {
+                                     CallBrInst, User::op_iterator> {
 public:
   CallSite() = default;
   CallSite(CallSiteBase B) : CallSiteBase(B) {}
   CallSite(CallInst *CI) : CallSiteBase(CI) {}
   CallSite(InvokeInst *II) : CallSiteBase(II) {}
+  CallSite(CallBrInst *CBI) : CallSiteBase(CBI) {}
   explicit CallSite(Instruction *II) : CallSiteBase(II) {}
   explicit CallSite(Value *V) : CallSiteBase(V) {}
 
   bool operator==(const CallSite &CS) const { return I == CS.I; }
   bool operator!=(const CallSite &CS) const { return I != CS.I; }
   bool operator<(const CallSite &CS) const {
     return getInstruction() < CS.getInstruction();
   }

 };
 
 /// Establish a view to a call site for examination.
 class ImmutableCallSite : public CallSiteBase<> {
 public:
   ImmutableCallSite() = default;
   ImmutableCallSite(const CallInst *CI) : CallSiteBase(CI) {}
   ImmutableCallSite(const InvokeInst *II) : CallSiteBase(II) {}
+  ImmutableCallSite(const CallBrInst *CBI) : CallSiteBase(CBI) {}
   explicit ImmutableCallSite(const Instruction *II) : CallSiteBase(II) {}
   explicit ImmutableCallSite(const Value *V) : CallSiteBase(V) {}
   ImmutableCallSite(CallSite CS) : CallSiteBase(CS.getInstruction()) {}
 };
 
 } // end namespace llvm
 
 #endif // LLVM_IR_CALLSITE_H

include/llvm/IR/IRBuilder.h

InvokeInst *CreateInvoke(Value *Callee, BasicBlock *NormalDest,
                            ArrayRef<Value *> Args = None,
                            const Twine &Name = "") {
     return CreateInvoke(
         cast<FunctionType>(
             cast<PointerType>(Callee->getType())->getElementType()),
         Callee, NormalDest, UnwindDest, Args, Name);
   }
 
+  /// \brief Create a callbr instruction.
+  CallBrInst *CreateCallBr(Value *Callee, BasicBlock *Fallthrough,
+                           SmallVector<BasicBlock *, 16> Transfers,
+                           ArrayRef<Value *> Args = None,
+                           const Twine &Name = "") {
+    return Insert(CallBrInst::Create(Callee, Fallthrough, Transfers, Args),
+                  Name);
+  }
+  CallBrInst *CreateCallBr(Value *Callee, BasicBlock *Fallthrough,
+                           SmallVector<BasicBlock *, 16> Transfers,
+                           ArrayRef<Value *> Args,
+                           ArrayRef<OperandBundleDef> OpBundles,
+                           const Twine &Name = "") {
+    return Insert(
+        CallBrInst::Create(Callee, Fallthrough, Transfers, Args, OpBundles),
+        Name);
+  }
+
   ResumeInst *CreateResume(Value *Exn) {
     return Insert(ResumeInst::Create(Exn));
   }
 
   CleanupReturnInst *CreateCleanupRet(CleanupPadInst *CleanupPad,
                                       BasicBlock *UnwindBB = nullptr) {
     return Insert(CleanupReturnInst::Create(CleanupPad, UnwindBB));
   }

include/llvm/IR/InstVisitor.h

#include "llvm/IR/Instruction.def"
   RetTy visitMemMoveInst(MemMoveInst &I)          { DELEGATE(MemTransferInst); }
   RetTy visitMemTransferInst(MemTransferInst &I)  { DELEGATE(MemIntrinsic); }
   RetTy visitMemIntrinsic(MemIntrinsic &I)        { DELEGATE(IntrinsicInst); }
   RetTy visitVAStartInst(VAStartInst &I)          { DELEGATE(IntrinsicInst); }
   RetTy visitVAEndInst(VAEndInst &I)              { DELEGATE(IntrinsicInst); }
   RetTy visitVACopyInst(VACopyInst &I)            { DELEGATE(IntrinsicInst); }
   RetTy visitIntrinsicInst(IntrinsicInst &I)      { DELEGATE(CallInst); }
 
-  // Call and Invoke are slightly different as they delegate first through
-  // a generic CallSite visitor.
+  // Call, Invoke and CallBr are slightly different as they delegate first
+  // through a generic CallSite visitor.
   RetTy visitCallInst(CallInst &I) {
     return static_cast<SubClass*>(this)->visitCallSite(&I);
   }
   RetTy visitInvokeInst(InvokeInst &I) {
     return static_cast<SubClass*>(this)->visitCallSite(&I);
   }
+  RetTy visitCallBrInst(CallBrInst &I) {
+    return static_cast<SubClass *>(this)->visitCallSite(&I);
+  }
 
   // While terminators don't have a distinct type modeling them, we support
   // intercepting them with dedicated a visitor callback.
   RetTy visitReturnInst(ReturnInst &I) {
     return static_cast<SubClass *>(this)->visitTerminator(I);
   }
   RetTy visitBranchInst(BranchInst &I) {
     return static_cast<SubClass *>(this)->visitTerminator(I);
#include "llvm/IR/Instruction.def"
   // of instructions...
   //
   RetTy visitCastInst(CastInst &I)                { DELEGATE(UnaryInstruction);}
   RetTy visitUnaryOperator(UnaryOperator &I)      { DELEGATE(UnaryInstruction);}
   RetTy visitBinaryOperator(BinaryOperator &I)    { DELEGATE(Instruction);}
   RetTy visitCmpInst(CmpInst &I)                  { DELEGATE(Instruction);}
   RetTy visitUnaryInstruction(UnaryInstruction &I){ DELEGATE(Instruction);}
 
   // The next level delegation for `CallBase` is slightly more complex in order
   // to support visiting cases where the call is also a terminator.
   RetTy visitCallBase(CallBase &I) {
-    if (isa<InvokeInst>(I))
+    if (isa<InvokeInst>(I) || isa<CallBrInst>(I))
       return static_cast<SubClass *>(this)->visitTerminator(I);
 
     DELEGATE(Instruction);
   }
 
-  // Provide a legacy visitor for a 'callsite' that visits both calls and
-  // invokes.
+  // Provide a legacy visitor for a 'callsite' that visits calls, invokes,
+  // and calbrs.
   //
   // Prefer overriding the type system based `CallBase` instead.
   RetTy visitCallSite(CallSite CS) {
     assert(CS);
     Instruction &I = *CS.getInstruction();
     DELEGATE(CallBase);
   }
 

include/llvm/IR/InstrTypes.h

protected:
   bool hasDescriptor() const { return Value::HasDescriptor; }
 
   unsigned getNumSubclassExtraOperands() const {
     switch (getOpcode()) {
     case Instruction::Call:
       return 0;
     case Instruction::Invoke:
       return 2;
+    case Instruction::CallBr:
+      return getNumSubclassExtraOperandsDynamic();
     }
     llvm_unreachable("Invalid opcode!");
   }
 
+  /// Get the number of extra operands for instructions that don't have a fixed
+  /// number of extra operands.
+  unsigned getNumSubclassExtraOperandsDynamic() const;
+
 public:
   using Instruction::getContext;
 
   static bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::Call ||
-           I->getOpcode() == Instruction::Invoke;
+           I->getOpcode() == Instruction::Invoke ||
+           I->getOpcode() == Instruction::CallBr;
   }
   static bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 
   FunctionType *getFunctionType() const { return FTy; }
 
   void mutateFunctionType(FunctionType *FTy) {
public:
 
   void setCallingConv(CallingConv::ID CC) {
     auto ID = static_cast<unsigned>(CC);
     assert(!(ID & ~CallingConv::MaxID) && "Unsupported calling convention");
     setInstructionSubclassData((getSubclassDataFromInstruction() & 3) |
                                (ID << 2));
   }
 
+  /// Check if this call is an inline asm statement.
+  bool isInlineAsm() const { return isa<InlineAsm>(getCalledOperand()); }
+
   /// \name Attribute API
   ///
   /// These methods access and modify attributes on this call (including
   /// looking through to the attributes on the called function when necessary).
   ///@{
 
   /// Return the parameter attributes for this call.
   ///

include/llvm/IR/Instruction.def

 HANDLE_TERM_INST  ( 3, Switch        , SwitchInst)
 HANDLE_TERM_INST  ( 4, IndirectBr    , IndirectBrInst)
 HANDLE_TERM_INST  ( 5, Invoke        , InvokeInst)
 HANDLE_TERM_INST  ( 6, Resume        , ResumeInst)
 HANDLE_TERM_INST  ( 7, Unreachable   , UnreachableInst)
 HANDLE_TERM_INST  ( 8, CleanupRet    , CleanupReturnInst)
 HANDLE_TERM_INST  ( 9, CatchRet      , CatchReturnInst)
 HANDLE_TERM_INST  (10, CatchSwitch   , CatchSwitchInst)
-  LAST_TERM_INST  (10)
+HANDLE_TERM_INST  (11, CallBr        , CallBrInst) // A call-site terminator
+  LAST_TERM_INST  (11)
 
 // Standard unary operators...
- FIRST_UNARY_INST(11)
-HANDLE_UNARY_INST(11, FNeg  , UnaryOperator)
-  LAST_UNARY_INST(11)
+ FIRST_UNARY_INST(12)
+HANDLE_UNARY_INST(12, FNeg  , UnaryOperator)
+  LAST_UNARY_INST(12)
 
 // Standard binary operators...
- FIRST_BINARY_INST(12)
-HANDLE_BINARY_INST(12, Add  , BinaryOperator)
-HANDLE_BINARY_INST(13, FAdd , BinaryOperator)
-HANDLE_BINARY_INST(14, Sub  , BinaryOperator)
-HANDLE_BINARY_INST(15, FSub , BinaryOperator)
-HANDLE_BINARY_INST(16, Mul  , BinaryOperator)
-HANDLE_BINARY_INST(17, FMul , BinaryOperator)
-HANDLE_BINARY_INST(18, UDiv , BinaryOperator)
-HANDLE_BINARY_INST(19, SDiv , BinaryOperator)
-HANDLE_BINARY_INST(20, FDiv , BinaryOperator)
-HANDLE_BINARY_INST(21, URem , BinaryOperator)
-HANDLE_BINARY_INST(22, SRem , BinaryOperator)
-HANDLE_BINARY_INST(23, FRem , BinaryOperator)
+ FIRST_BINARY_INST(13)
+HANDLE_BINARY_INST(13, Add  , BinaryOperator)
+HANDLE_BINARY_INST(14, FAdd , BinaryOperator)
+HANDLE_BINARY_INST(15, Sub  , BinaryOperator)
+HANDLE_BINARY_INST(16, FSub , BinaryOperator)
+HANDLE_BINARY_INST(17, Mul  , BinaryOperator)
+HANDLE_BINARY_INST(18, FMul , BinaryOperator)
+HANDLE_BINARY_INST(19, UDiv , BinaryOperator)
+HANDLE_BINARY_INST(20, SDiv , BinaryOperator)
+HANDLE_BINARY_INST(21, FDiv , BinaryOperator)
+HANDLE_BINARY_INST(22, URem , BinaryOperator)
+HANDLE_BINARY_INST(23, SRem , BinaryOperator)
+HANDLE_BINARY_INST(24, FRem , BinaryOperator)
 
 // Logical operators (integer operands)
-HANDLE_BINARY_INST(24, Shl  , BinaryOperator) // Shift left  (logical)
-HANDLE_BINARY_INST(25, LShr , BinaryOperator) // Shift right (logical)
-HANDLE_BINARY_INST(26, AShr , BinaryOperator) // Shift right (arithmetic)
-HANDLE_BINARY_INST(27, And  , BinaryOperator)
-HANDLE_BINARY_INST(28, Or   , BinaryOperator)
-HANDLE_BINARY_INST(29, Xor  , BinaryOperator)
-  LAST_BINARY_INST(29)
+HANDLE_BINARY_INST(25, Shl  , BinaryOperator) // Shift left  (logical)
+HANDLE_BINARY_INST(26, LShr , BinaryOperator) // Shift right (logical)
+HANDLE_BINARY_INST(27, AShr , BinaryOperator) // Shift right (arithmetic)
+HANDLE_BINARY_INST(28, And  , BinaryOperator)
+HANDLE_BINARY_INST(29, Or   , BinaryOperator)
+HANDLE_BINARY_INST(30, Xor  , BinaryOperator)
+  LAST_BINARY_INST(30)
 
 // Memory operators...
- FIRST_MEMORY_INST(30)
-HANDLE_MEMORY_INST(30, Alloca, AllocaInst)  // Stack management
-HANDLE_MEMORY_INST(31, Load  , LoadInst  )  // Memory manipulation instrs
-HANDLE_MEMORY_INST(32, Store , StoreInst )
-HANDLE_MEMORY_INST(33, GetElementPtr, GetElementPtrInst)
-HANDLE_MEMORY_INST(34, Fence , FenceInst )
-HANDLE_MEMORY_INST(35, AtomicCmpXchg , AtomicCmpXchgInst )
-HANDLE_MEMORY_INST(36, AtomicRMW , AtomicRMWInst )
-  LAST_MEMORY_INST(36)
+ FIRST_MEMORY_INST(31)
+HANDLE_MEMORY_INST(31, Alloca, AllocaInst)  // Stack management
+HANDLE_MEMORY_INST(32, Load  , LoadInst  )  // Memory manipulation instrs
+HANDLE_MEMORY_INST(33, Store , StoreInst )
+HANDLE_MEMORY_INST(34, GetElementPtr, GetElementPtrInst)
+HANDLE_MEMORY_INST(35, Fence , FenceInst )
+HANDLE_MEMORY_INST(36, AtomicCmpXchg , AtomicCmpXchgInst )
+HANDLE_MEMORY_INST(37, AtomicRMW , AtomicRMWInst )
+  LAST_MEMORY_INST(37)
 
 // Cast operators ...
 // NOTE: The order matters here because CastInst::isEliminableCastPair
 // NOTE: (see Instructions.cpp) encodes a table based on this ordering.
- FIRST_CAST_INST(37)
-HANDLE_CAST_INST(37, Trunc   , TruncInst   )  // Truncate integers
-HANDLE_CAST_INST(38, ZExt    , ZExtInst    )  // Zero extend integers
-HANDLE_CAST_INST(39, SExt    , SExtInst    )  // Sign extend integers
-HANDLE_CAST_INST(40, FPToUI  , FPToUIInst  )  // floating point -> UInt
-HANDLE_CAST_INST(41, FPToSI  , FPToSIInst  )  // floating point -> SInt
-HANDLE_CAST_INST(42, UIToFP  , UIToFPInst  )  // UInt -> floating point
-HANDLE_CAST_INST(43, SIToFP  , SIToFPInst  )  // SInt -> floating point
-HANDLE_CAST_INST(44, FPTrunc , FPTruncInst )  // Truncate floating point
-HANDLE_CAST_INST(45, FPExt   , FPExtInst   )  // Extend floating point
-HANDLE_CAST_INST(46, PtrToInt, PtrToIntInst)  // Pointer -> Integer
-HANDLE_CAST_INST(47, IntToPtr, IntToPtrInst)  // Integer -> Pointer
-HANDLE_CAST_INST(48, BitCast , BitCastInst )  // Type cast
-HANDLE_CAST_INST(49, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
-  LAST_CAST_INST(49)
-
- FIRST_FUNCLETPAD_INST(50)
-HANDLE_FUNCLETPAD_INST(50, CleanupPad, CleanupPadInst)
-HANDLE_FUNCLETPAD_INST(51, CatchPad  , CatchPadInst)
-  LAST_FUNCLETPAD_INST(51)
+ FIRST_CAST_INST(38)
+HANDLE_CAST_INST(38, Trunc   , TruncInst   )  // Truncate integers
+HANDLE_CAST_INST(39, ZExt    , ZExtInst    )  // Zero extend integers
+HANDLE_CAST_INST(40, SExt    , SExtInst    )  // Sign extend integers
+HANDLE_CAST_INST(41, FPToUI  , FPToUIInst  )  // floating point -> UInt
+HANDLE_CAST_INST(42, FPToSI  , FPToSIInst  )  // floating point -> SInt
+HANDLE_CAST_INST(43, UIToFP  , UIToFPInst  )  // UInt -> floating point
+HANDLE_CAST_INST(44, SIToFP  , SIToFPInst  )  // SInt -> floating point
+HANDLE_CAST_INST(45, FPTrunc , FPTruncInst )  // Truncate floating point
+HANDLE_CAST_INST(46, FPExt   , FPExtInst   )  // Extend floating point
+HANDLE_CAST_INST(47, PtrToInt, PtrToIntInst)  // Pointer -> Integer
+HANDLE_CAST_INST(48, IntToPtr, IntToPtrInst)  // Integer -> Pointer
+HANDLE_CAST_INST(49, BitCast , BitCastInst )  // Type cast
+HANDLE_CAST_INST(50, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
+  LAST_CAST_INST(50)
+
+ FIRST_FUNCLETPAD_INST(51)
+HANDLE_FUNCLETPAD_INST(51, CleanupPad, CleanupPadInst)
+HANDLE_FUNCLETPAD_INST(52, CatchPad  , CatchPadInst)
+  LAST_FUNCLETPAD_INST(52)
 
 // Other operators...
- FIRST_OTHER_INST(52)
-HANDLE_OTHER_INST(52, ICmp   , ICmpInst   )  // Integer comparison instruction
-HANDLE_OTHER_INST(53, FCmp   , FCmpInst   )  // Floating point comparison instr.
-HANDLE_OTHER_INST(54, PHI    , PHINode    )  // PHI node instruction
-HANDLE_OTHER_INST(55, Call   , CallInst   )  // Call a function
-HANDLE_OTHER_INST(56, Select , SelectInst )  // select instruction
-HANDLE_USER_INST (57, UserOp1, Instruction)  // May be used internally in a pass
-HANDLE_USER_INST (58, UserOp2, Instruction)  // Internal to passes only
-HANDLE_OTHER_INST(59, VAArg  , VAArgInst  )  // vaarg instruction
-HANDLE_OTHER_INST(60, ExtractElement, ExtractElementInst)// extract from vector
-HANDLE_OTHER_INST(61, InsertElement, InsertElementInst)  // insert into vector
-HANDLE_OTHER_INST(62, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
-HANDLE_OTHER_INST(63, ExtractValue, ExtractValueInst)// extract from aggregate
-HANDLE_OTHER_INST(64, InsertValue, InsertValueInst)  // insert into aggregate
-HANDLE_OTHER_INST(65, LandingPad, LandingPadInst)  // Landing pad instruction.
-  LAST_OTHER_INST(65)
+ FIRST_OTHER_INST(53)
+HANDLE_OTHER_INST(53, ICmp   , ICmpInst   )  // Integer comparison instruction
+HANDLE_OTHER_INST(54, FCmp   , FCmpInst   )  // Floating point comparison instr.
+HANDLE_OTHER_INST(55, PHI    , PHINode    )  // PHI node instruction
+HANDLE_OTHER_INST(56, Call   , CallInst   )  // Call a function
+HANDLE_OTHER_INST(57, Select , SelectInst )  // select instruction
+HANDLE_USER_INST (58, UserOp1, Instruction)  // May be used internally in a pass
+HANDLE_USER_INST (59, UserOp2, Instruction)  // Internal to passes only
+HANDLE_OTHER_INST(60, VAArg  , VAArgInst  )  // vaarg instruction
+HANDLE_OTHER_INST(61, ExtractElement, ExtractElementInst)// extract from vector
+HANDLE_OTHER_INST(62, InsertElement, InsertElementInst)  // insert into vector
+HANDLE_OTHER_INST(63, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
+HANDLE_OTHER_INST(64, ExtractValue, ExtractValueInst)// extract from aggregate
+HANDLE_OTHER_INST(65, InsertValue, InsertValueInst)  // insert into aggregate
+HANDLE_OTHER_INST(66, LandingPad, LandingPadInst)  // Landing pad instruction.
+  LAST_OTHER_INST(66)
 
 #undef  FIRST_TERM_INST
 #undef HANDLE_TERM_INST
 #undef   LAST_TERM_INST
 
 #undef  FIRST_UNARY_INST
 #undef HANDLE_UNARY_INST
 #undef   LAST_UNARY_INST

include/llvm/IR/Instructions.h

public:
   }
 
   /// Return true if the call can return twice
   bool canReturnTwice() const { return hasFnAttr(Attribute::ReturnsTwice); }
   void setCanReturnTwice() {
     addAttribute(AttributeList::FunctionIndex, Attribute::ReturnsTwice);
   }
 
-  /// Check if this call is an inline asm statement.
-  bool isInlineAsm() const { return isa<InlineAsm>(getCalledOperand()); }
-
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::Call;
   }
   static bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 
InvokeInst::InvokeInst(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
                        const Twine &NameStr, BasicBlock *InsertAtEnd)
     : CallBase(Ty->getReturnType(), Instruction::Invoke,
                OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
                InsertAtEnd) {
   init(Ty, Func, IfNormal, IfException, Args, Bundles, NameStr);
 }
 
 //===----------------------------------------------------------------------===//
+//                              CallBrInst Class
+//===----------------------------------------------------------------------===//
+
+/// CallBr instruction, tracking function calls that may not return control but
+/// instead transfer it to a third location. The SubclassData field is used to
+/// hold the calling convention of the call.
+///
+class CallBrInst : public CallBase {
+
+  unsigned NumTransfers;
+
+  CallBrInst(const CallBrInst &BI);
+
+  /// Construct a CallBrInst given a range of arguments.
+  ///
+  /// Construct a CallBrInst from a range of arguments
+  inline CallBrInst(Value *Func, BasicBlock *Fallthrough,
+                    ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                    const Twine &NameStr, Instruction *InsertBefore)
+      : CallBrInst(cast<FunctionType>(
+                       cast<PointerType>(Func->getType())->getElementType()),
+                   Func, Fallthrough, Transfers, Args, Bundles, NumOperands,
+                   NameStr, InsertBefore) { NumTransfers = Transfers.size(); }
+
+  inline CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *Fallthrough,
+                    ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                    const Twine &NameStr, Instruction *InsertBefore);
+  /// Construct a CallBrInst given a range of arguments.
+  ///
+  /// Construct a CallBrInst from a range of arguments
+  inline CallBrInst(Value *Func, BasicBlock *Fallthrough,
+                    ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                    const Twine &NameStr, BasicBlock *InsertAtEnd);
+
+  void init(Value *Func, BasicBlock *Fallthrough,
+            ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+            ArrayRef<OperandBundleDef> Bundles, const Twine &NameStr) {
+    init(cast<FunctionType>(
+             cast<PointerType>(Func->getType())->getElementType()),
+         Func, Fallthrough, Transfers, Args, Bundles, NameStr);
+  }
+
+  void init(FunctionType *FTy, Value *Func, BasicBlock *Fallthrough,
+            ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+            ArrayRef<OperandBundleDef> Bundles, const Twine &NameStr);
+
+  /// Compute the number of operands to allocate.
+  static int ComputeNumOperands(int NumArgs, int NumTransfers,
+                                int NumBundleInputs = 0) {
+    // We need one operand for the called function, plus our extra operands and
+    // the input operand counts provided.
+    return 2 + NumTransfers + NumArgs + NumBundleInputs;
+  }
+
+protected:
+  // Note: Instruction needs to be a friend here to call cloneImpl.
+  friend class Instruction;
+
+  CallBrInst *cloneImpl() const;
+
+public:
+  static CallBrInst *Create(Value *Func, BasicBlock *Fallthrough,
+                            ArrayRef<BasicBlock *> Transfers,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            Instruction *InsertBefore = nullptr) {
+    return Create(cast<FunctionType>(
+                      cast<PointerType>(Func->getType())->getElementType()),
+                  Func, Fallthrough, Transfers, Args, None, NameStr,
+                  InsertBefore);
+  }
+
+  static CallBrInst *
+  Create(Value *Func, BasicBlock *Fallthrough, ArrayRef<BasicBlock *> Transfers,
+         ArrayRef<Value *> Args, ArrayRef<OperandBundleDef> Bundles = None,
+         const Twine &NameStr = "", Instruction *InsertBefore = nullptr) {
+    return Create(cast<FunctionType>(
+                      cast<PointerType>(Func->getType())->getElementType()),
+                  Func, Fallthrough, Transfers, Args, Bundles, NameStr,
+                  InsertBefore);
+  }
+
+  static CallBrInst *Create(FunctionType *Ty, Value *Func,
+                            BasicBlock *Fallthrough,
+                            ArrayRef<BasicBlock *> Transfers,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            Instruction *InsertBefore = nullptr) {
+    int NumOperands = ComputeNumOperands(Args.size(), Transfers.size());
+    return new (NumOperands)
+        CallBrInst(Ty, Func, Fallthrough, Transfers, Args, None, NumOperands,
+                   NameStr, InsertBefore);
+  }
+
+  static CallBrInst *
+  Create(FunctionType *Ty, Value *Func, BasicBlock *Fallthrough,
+         ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+         ArrayRef<OperandBundleDef> Bundles = None, const Twine &NameStr = "",
+         Instruction *InsertBefore = nullptr) {
+    int NumOperands = ComputeNumOperands(Args.size(), Transfers.size(),
+                                         CountBundleInputs(Bundles));
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (NumOperands, DescriptorBytes)
+        CallBrInst(Ty, Func, Fallthrough, Transfers, Args, Bundles, NumOperands,
+                   NameStr, InsertBefore);
+  }
+
+  static CallBrInst *Create(Value *Func, BasicBlock *Fallthrough,
+                            ArrayRef<BasicBlock *> Transfers,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            BasicBlock *InsertAtEnd) {
+    int NumOperands = ComputeNumOperands(Args.size(), Transfers.size());
+    return new (NumOperands)
+        CallBrInst(Func, Fallthrough, Transfers, Args, None, NumOperands,
+                   NameStr, InsertAtEnd);
+  }
+
+  static CallBrInst *Create(Value *Func, BasicBlock *Fallthrough,
+                            ArrayRef<BasicBlock *> Transfers,
+                            ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleDef> Bundles,
+                            const Twine &NameStr, BasicBlock *InsertAtEnd) {
+    int NumOperands = ComputeNumOperands(Args.size(), Transfers.size(),
+                                         CountBundleInputs(Bundles));
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (NumOperands, DescriptorBytes)
+        CallBrInst(Func, Fallthrough, Transfers, Args, Bundles, NumOperands,
+                   NameStr, InsertAtEnd);
+  }
+
+  /// Create a clone of \p CBI with a different set of operand bundles and
+  /// insert it before \p InsertPt.
+  ///
+  /// The returned callbr instruction is identical to \p CBI in every way
+  /// except that the operand bundles for the new instruction are set to the
+  /// operand bundles in \p Bundles.
+  static CallBrInst *Create(CallBrInst *CBI,
+                            ArrayRef<OperandBundleDef> Bundles,
+                            Instruction *InsertPt = nullptr);
+
+  /// Return the number of callbr transfer labels.
+  ///
+  unsigned getNumTransfers() const { return NumTransfers; }
+
+  /// getTransferLabel - Return the i-th transfer label.
+  ///
+  Value *getTransferLabel(unsigned i) const {
+    assert(i < getNumTransfers() && "Out of bounds!");
+    return getOperand(i + getNumArgOperands() + getNumTotalBundleOperands() +
+                      1);
+  }
+
+  Value *getTransferLabelUse(unsigned i) const {
+    assert(i < getNumTransfers() && "Out of bounds!");
+    return getOperandUse(i + getNumArgOperands() + getNumTotalBundleOperands() +
+                         1);
+  }
+
+  // Return the destination basic blocks...
+  BasicBlock *getFallthrough() const {
+    return cast<BasicBlock>(*(&Op<-1>() - getNumTransfers() - 1));
+  }
+  BasicBlock *getTransfer(unsigned i) const {
+    return cast<BasicBlock>(*(&Op<-1>() - getNumTransfers() + i));
+  }
+  SmallVector<BasicBlock *, 16> getTransfers() const {
+    SmallVector<BasicBlock *, 16> Transfers;
+    for (unsigned i = 0, e = getNumTransfers(); i < e; ++i)
+      Transfers.push_back(getTransfer(i));
+    return Transfers;
+  }
+  void setFallthrough(BasicBlock *B) {
+    *(&Op<-1>() - getNumTransfers() - 1) = reinterpret_cast<Value *>(B);
+  }
+  void setTransfer(unsigned i, BasicBlock *B) {
+    *(&Op<-1>() - getNumTransfers() + i) = reinterpret_cast<Value *>(B);
+  }
+
+  BasicBlock *getSuccessor(unsigned i) const {
+    assert(i < getNumTransfers() + 1 && "Successor # out of range for callbr!");
+    return i == 0 ? getFallthrough() : getTransfer(i - 1);
+  }
+
+  void setSuccessor(unsigned idx, BasicBlock *NewSucc) {
+    assert(idx < getNumTransfers() + 1 &&
+           "Successor # out of range for callbr!");
+    *(&Op<-1>() - getNumTransfers() -1 + idx) =
+        reinterpret_cast<Value *>(NewSucc);
+  }
+
+  unsigned getNumSuccessors() const { return getNumTransfers() + 1; }
+
+  // Methods for support type inquiry through isa, cast, and dyn_cast:
+  static bool classof(const Instruction *I) {
+    return (I->getOpcode() == Instruction::CallBr);
+  }
+  static bool classof(const Value *V) {
+    return isa<Instruction>(V) && classof(cast<Instruction>(V));
+  }
+
+private:
+
+  // Shadow Instruction::setInstructionSubclassData with a private forwarding
+  // method so that subclasses cannot accidentally use it.
+  void setInstructionSubclassData(unsigned short D) {
+    Instruction::setInstructionSubclassData(D);
+  }
+};
+
+CallBrInst::CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *Fallthrough,
+                       ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                       ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                       const Twine &NameStr, Instruction *InsertBefore)
+    : CallBase(Ty->getReturnType(), Instruction::CallBr,
+               OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
+               InsertBefore) {
+  init(Ty, Func, Fallthrough, Transfers, Args, Bundles, NameStr);
+}
+
+CallBrInst::CallBrInst(Value *Func, BasicBlock *Fallthrough,
+                       ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                       ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                       const Twine &NameStr, BasicBlock *InsertAtEnd)
+    : CallBase(
+          cast<FunctionType>(
+              cast<PointerType>(Func->getType())->getElementType())
+              ->getReturnType(),
+          Instruction::CallBr,
+          OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
+          InsertAtEnd) {
+  init(Func, Fallthrough, Transfers, Args, Bundles, NameStr);
+}
+
+//===----------------------------------------------------------------------===//
 //                              ResumeInst Class
 //===----------------------------------------------------------------------===//
 
 //===---------------------------------------------------------------------------
 /// Resume the propagation of an exception.
 ///
 class ResumeInst : public Instruction {
   ResumeInst(const ResumeInst &RI);

lib/Analysis/ValueTracking.cpp

case Instruction::Call: {
 
     // The called function could have undefined behavior or side-effects, even
     // if marked readnone nounwind.
     return Callee && Callee->isSpeculatable();
   }
   case Instruction::VAArg:
   case Instruction::Alloca:
   case Instruction::Invoke:
+  case Instruction::CallBr:
   case Instruction::PHI:
   case Instruction::Store:
   case Instruction::Ret:
   case Instruction::Br:
   case Instruction::IndirectBr:
   case Instruction::Switch:
   case Instruction::Unreachable:
   case Instruction::Fence:

lib/AsmParser/LLLexer.cpp

#define INSTKEYWORD(STR, Enum)                                                 \
   INSTKEYWORD(va_arg,      VAArg);
   INSTKEYWORD(ret,         Ret);
   INSTKEYWORD(br,          Br);
   INSTKEYWORD(switch,      Switch);
   INSTKEYWORD(indirectbr,  IndirectBr);
   INSTKEYWORD(invoke,      Invoke);
   INSTKEYWORD(resume,      Resume);
   INSTKEYWORD(unreachable, Unreachable);
+  INSTKEYWORD(callbr,      CallBr);
 
   INSTKEYWORD(alloca,      Alloca);
   INSTKEYWORD(load,        Load);
   INSTKEYWORD(store,       Store);
   INSTKEYWORD(cmpxchg,     AtomicCmpXchg);
   INSTKEYWORD(atomicrmw,   AtomicRMW);
   INSTKEYWORD(fence,       Fence);
   INSTKEYWORD(getelementptr, GetElementPtr);

lib/AsmParser/LLParser.h

#include "llvm/IR/Metadata.def"
     bool ParseIndirectBr(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseInvoke(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseResume(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchSwitch(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchPad(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupPad(Instruction *&Inst, PerFunctionState &PFS);
+    bool ParseCallBr(Instruction *&Inst, PerFunctionState &PFS);
 
     bool ParseUnaryOp(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc,
                       unsigned OperandType);
     bool ParseArithmetic(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc,
                          unsigned OperandType);
     bool ParseLogical(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);
     bool ParseCompare(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);
     bool ParseCast(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);

lib/AsmParser/LLParser.cpp

for (const auto &RAG : ForwardRefAttrGroups) {
     } else if (InvokeInst *II = dyn_cast<InvokeInst>(V)) {
       AttributeList AS = II->getAttributes();
       AttrBuilder FnAttrs(AS.getFnAttributes());
       AS = AS.removeAttributes(Context, AttributeList::FunctionIndex);
       FnAttrs.merge(B);
       AS = AS.addAttributes(Context, AttributeList::FunctionIndex,
                             AttributeSet::get(Context, FnAttrs));
       II->setAttributes(AS);
+    } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(V)) {
+      AttributeList AS = CBI->getAttributes();
+      AttrBuilder FnAttrs(AS.getFnAttributes());
+      AS = AS.removeAttributes(Context, AttributeList::FunctionIndex);
+      FnAttrs.merge(B);
+      AS = AS.addAttributes(Context, AttributeList::FunctionIndex,
+                            AttributeSet::get(Context, FnAttrs));
+      CBI->setAttributes(AS);
     } else if (auto *GV = dyn_cast<GlobalVariable>(V)) {
       AttrBuilder Attrs(GV->getAttributes());
       Attrs.merge(B);
       GV->setAttributes(AttributeSet::get(Context,Attrs));
     } else {
       llvm_unreachable("invalid object with forward attribute group reference");
     }
   }
int LLParser::ParseInstruction(Instruction *&Inst, BasicBlock *BB,
   case lltok::kw_indirectbr:  return ParseIndirectBr(Inst, PFS);
   case lltok::kw_invoke:      return ParseInvoke(Inst, PFS);
   case lltok::kw_resume:      return ParseResume(Inst, PFS);
   case lltok::kw_cleanupret:  return ParseCleanupRet(Inst, PFS);
   case lltok::kw_catchret:    return ParseCatchRet(Inst, PFS);
   case lltok::kw_catchswitch: return ParseCatchSwitch(Inst, PFS);
   case lltok::kw_catchpad:    return ParseCatchPad(Inst, PFS);
   case lltok::kw_cleanuppad:  return ParseCleanupPad(Inst, PFS);
+  case lltok::kw_callbr:      return ParseCallBr(Inst, PFS);
   // Unary Operators.
   case lltok::kw_fneg: {
     FastMathFlags FMF = EatFastMathFlagsIfPresent();
     int Res = ParseUnaryOp(Inst, PFS, KeywordVal, 2);
     if (Res != 0)
       return Res;
     if (FMF.any())
       Inst->setFastMathFlags(FMF);
bool LLParser::ParseUnaryOp(Instruction *&Inst, PerFunctionState &PFS,
 
   if (!Valid)
     return Error(Loc, "invalid operand type for instruction");
 
   Inst = UnaryOperator::Create((Instruction::UnaryOps)Opc, LHS);
   return false;
 }
 
+/// ParseCallBr
+///   ::= 'callbr' OptionalCallingConv OptionalAttrs Type Value ParamList
+///       OptionalAttrs OptionalOperandBundles 'to' TypeAndValue
+///       '[' LabelList ']'
+bool LLParser::ParseCallBr(Instruction *&Inst, PerFunctionState &PFS) {
+  LocTy CallLoc = Lex.getLoc();
+  AttrBuilder RetAttrs, FnAttrs;
+  std::vector<unsigned> FwdRefAttrGrps;
+  LocTy NoBuiltinLoc;
+  unsigned CC;
+  Type *RetType = nullptr;
+  LocTy RetTypeLoc;
+  ValID CalleeID;
+  SmallVector<ParamInfo, 16> ArgList;
+  SmallVector<OperandBundleDef, 2> BundleList;
+
+  BasicBlock *Fallthrough;
+  if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+      ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
+      ParseValID(CalleeID) || ParseParameterList(ArgList, PFS) ||
+      ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false,
+                                 NoBuiltinLoc) ||
+      ParseOptionalOperandBundles(BundleList, PFS) ||
+      ParseToken(lltok::kw_to, "expected 'to' in callbr") ||
+      ParseTypeAndBasicBlock(Fallthrough, PFS) ||
+      ParseToken(lltok::lsquare, "expected '[' in callbr"))
+    return true;
+
+  // Parse the destination list.
+  SmallVector<BasicBlock *, 16> Transfers;
+
+  if (Lex.getKind() != lltok::rsquare) {
+    BasicBlock *DestBB;
+    if (ParseTypeAndBasicBlock(DestBB, PFS))
+      return true;
+    Transfers.push_back(DestBB);
+
+    while (EatIfPresent(lltok::comma)) {
+      if (ParseTypeAndBasicBlock(DestBB, PFS))
+        return true;
+      Transfers.push_back(DestBB);
+    }
+  }
+
+  if (ParseToken(lltok::rsquare, "expected ']' at end of block list"))
+    return true;
+
+  // If RetType is a non-function pointer type, then this is the short syntax
+  // for the call, which means that RetType is just the return type.  Infer the
+  // rest of the function argument types from the arguments that are present.
+  FunctionType *Ty = dyn_cast<FunctionType>(RetType);
+  if (!Ty) {
+    // Pull out the types of all of the arguments...
+    std::vector<Type *> ParamTypes;
+    for (unsigned i = 0, e = ArgList.size(); i != e; ++i)
+      ParamTypes.push_back(ArgList[i].V->getType());
+
+    if (!FunctionType::isValidReturnType(RetType))
+      return Error(RetTypeLoc, "Invalid result type for LLVM function");
+
+    Ty = FunctionType::get(RetType, ParamTypes, false);
+  }
+
+  CalleeID.FTy = Ty;
+
+  // Look up the callee.
+  Value *Callee;
+  if (ConvertValIDToValue(PointerType::getUnqual(Ty), CalleeID, Callee, &PFS,
+                          /*IsCall=*/true))
+    return true;
+
+  if (isa<InlineAsm>(Callee) && !Ty->getReturnType()->isVoidTy())
+    return Error(RetTypeLoc, "asm-goto outputs not supported");
+
+  // Set up the Attribute for the function.
+  SmallVector<Value *, 8> Args;
+  SmallVector<AttributeSet, 8> ArgAttrs;
+
+  // Loop through FunctionType's arguments and ensure they are specified
+  // correctly.  Also, gather any parameter attributes.
+  FunctionType::param_iterator I = Ty->param_begin();
+  FunctionType::param_iterator E = Ty->param_end();
+  for (unsigned i = 0, e = ArgList.size(); i != e; ++i) {
+    Type *ExpectedTy = nullptr;
+    if (I != E) {
+      ExpectedTy = *I++;
+    } else if (!Ty->isVarArg()) {
+      return Error(ArgList[i].Loc, "too many arguments specified");
+    }
+
+    if (ExpectedTy && ExpectedTy != ArgList[i].V->getType())
+      return Error(ArgList[i].Loc, "argument is not of expected type '" +
+                                       getTypeString(ExpectedTy) + "'");
+    Args.push_back(ArgList[i].V);
+    ArgAttrs.push_back(ArgList[i].Attrs);
+  }
+
+  if (I != E)
+    return Error(CallLoc, "not enough parameters specified for call");
+
+  if (FnAttrs.hasAlignmentAttr())
+    return Error(CallLoc, "callbr instructions may not have an alignment");
+
+  // Finish off the Attribute and check them
+  AttributeList PAL =
+      AttributeList::get(Context, AttributeSet::get(Context, FnAttrs),
+                         AttributeSet::get(Context, RetAttrs), ArgAttrs);
+
+  CallBrInst *CBI =
+      CallBrInst::Create(Ty, Callee, Fallthrough, Transfers, Args, BundleList);
+  CBI->setCallingConv(CC);
+  CBI->setAttributes(PAL);
+  ForwardRefAttrGroups[CBI] = FwdRefAttrGrps;
+  Inst = CBI;
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 // Binary Operators.
 //===----------------------------------------------------------------------===//
 
 /// ParseArithmetic
 ///  ::= ArithmeticOps TypeAndValue ',' Value
 ///
 /// If OperandType is 0, then any FP or integer operand is allowed.  If it is 1,

lib/AsmParser/LLToken.h

enum Kind {
   kw_invoke,
   kw_resume,
   kw_unreachable,
   kw_cleanupret,
   kw_catchswitch,
   kw_catchret,
   kw_catchpad,
   kw_cleanuppad,
+  kw_callbr,
 
   kw_alloca,
   kw_load,
   kw_store,
   kw_fence,
   kw_cmpxchg,
   kw_atomicrmw,
   kw_getelementptr,

lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_RESUME: { // RESUME: [opval]
       unsigned Idx = 0;
       Value *Val = nullptr;
       if (getValueTypePair(Record, Idx, NextValueNo, Val))
         return error("Invalid record");
       I = ResumeInst::Create(Val);
       InstructionList.push_back(I);
       break;
     }
+    case bitc::FUNC_CODE_INST_CALLBR: {
+      // CALLBR: [attr, cc, norm, transfs, fty, fnid, args]
+      unsigned OpNum = 0;
+      AttributeList PAL = getAttributes(Record[OpNum++]);
+      unsigned CCInfo = Record[OpNum++];
+
+      BasicBlock *Fallthrough = getBasicBlock(Record[OpNum++]);
+      unsigned NumTransfers = Record[OpNum++];
+      SmallVector<BasicBlock *, 16> Transfers;
+      for (unsigned i = 0, e = NumTransfers; i != e; ++i)
+        Transfers.push_back(getBasicBlock(Record[OpNum++]));
+
+      FunctionType *FTy = nullptr;
+      if (CCInfo >> bitc::CALL_EXPLICIT_TYPE & 1 &&
+          !(FTy = dyn_cast<FunctionType>(getTypeByID(Record[OpNum++]))))
+        return error("Explicit call type is not a function type");
+
+      Value *Callee;
+      if (getValueTypePair(Record, OpNum, NextValueNo, Callee))
+        return error("Invalid record");
+
+      PointerType *OpTy = dyn_cast<PointerType>(Callee->getType());
+      if (!OpTy)
+        return error("Callee is not a pointer type");
+      if (!FTy) {
+        FTy = dyn_cast<FunctionType>(OpTy->getElementType());
+        if (!FTy)
+          return error("Callee is not of pointer to function type");
+      } else if (OpTy->getElementType() != FTy)
+        return error("Explicit call type does not match pointee type of "
+                     "callee operand");
+      if (Record.size() < FTy->getNumParams() + OpNum)
+        return error("Insufficient operands to call");
+
+      SmallVector<Value*, 16> Args;
+      // Read the fixed params.
+      for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i, ++OpNum) {
+        if (FTy->getParamType(i)->isLabelTy())
+          Args.push_back(getBasicBlock(Record[OpNum]));
+        else
+          Args.push_back(getValue(Record, OpNum, NextValueNo,
+                                  FTy->getParamType(i)));
+        if (!Args.back())
+          return error("Invalid record");
+      }
+
+      // Read type/value pairs for varargs params.
+      if (!FTy->isVarArg()) {
+        if (OpNum != Record.size())
+          return error("Invalid record");
+      } else {
+        while (OpNum != Record.size()) {
+          Value *Op;
+          if (getValueTypePair(Record, OpNum, NextValueNo, Op))
+            return error("Invalid record");
+          Args.push_back(Op);
+        }
+      }
+
+      I = CallBrInst::Create(FTy, Callee, Fallthrough, Transfers, Args,
+                             OperandBundles);
+      OperandBundles.clear();
+      InstructionList.push_back(I);
+      cast<CallBrInst>(I)->setCallingConv(
+          static_cast<CallingConv::ID>((0x7ff & CCInfo) >> bitc::CALL_CCONV));
+      cast<CallBrInst>(I)->setAttributes(PAL);
+      break;
+    }
     case bitc::FUNC_CODE_INST_UNREACHABLE: // UNREACHABLE
       I = new UnreachableInst(Context);
       InstructionList.push_back(I);
       break;
     case bitc::FUNC_CODE_INST_PHI: { // PHI: [ty, val0,bb0, ...]
       if (Record.size() < 1 || ((Record.size()-1)&1))
         return error("Invalid record");
       Type *Ty = getTypeByID(Record[0]);

lib/Bitcode/Writer/BitcodeWriter.cpp

case Instruction::CatchSwitch: {
     Vals.push_back(NumHandlers);
     for (const BasicBlock *CatchPadBB : CatchSwitch.handlers())
       Vals.push_back(VE.getValueID(CatchPadBB));
 
     if (CatchSwitch.hasUnwindDest())
       Vals.push_back(VE.getValueID(CatchSwitch.getUnwindDest()));
     break;
   }
+  case Instruction::CallBr: {
+    const CallBrInst *CBI = cast<CallBrInst>(&I);
+    const Value *Callee = CBI->getCalledValue();
+    FunctionType *FTy = CBI->getFunctionType();
+
+    if (CBI->hasOperandBundles())
+      writeOperandBundles(CBI, InstID);
+
+    Code = bitc::FUNC_CODE_INST_CALLBR;
+
+    Vals.push_back(VE.getAttributeListID(CBI->getAttributes()));
+
+    Vals.push_back(CBI->getCallingConv() << bitc::CALL_CCONV |
+                   1 << bitc::CALL_EXPLICIT_TYPE);
+
+    Vals.push_back(VE.getValueID(CBI->getFallthrough()));
+    Vals.push_back(CBI->getNumTransfers());
+    for (unsigned i = 0, e = CBI->getNumTransfers(); i != e; ++i)
+      Vals.push_back(VE.getValueID(CBI->getTransfer(i)));
+
+    Vals.push_back(VE.getTypeID(FTy));
+    pushValueAndType(Callee, InstID, Vals);
+
+    // Emit value #'s for the fixed parameters.
+    for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i)
+      pushValue(I.getOperand(i), InstID, Vals); // fixed param.
+
+    // Emit type/value pairs for varargs params.
+    if (FTy->isVarArg()) {
+      for (unsigned i = FTy->getNumParams(), e = CBI->getNumArgOperands();
+           i != e; ++i)
+        pushValueAndType(I.getOperand(i), InstID, Vals); // vararg
+    }
+    break;
+  }
   case Instruction::Unreachable:
     Code = bitc::FUNC_CODE_INST_UNREACHABLE;
     AbbrevToUse = FUNCTION_INST_UNREACHABLE_ABBREV;
     break;
 
   case Instruction::PHI: {
     const PHINode &PN = cast<PHINode>(I);
     Code = bitc::FUNC_CODE_INST_PHI;

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

 }
 
 /// isBlockOnlyReachableByFallthough - Return true if the basic block has
 /// exactly one predecessor and the control transfer mechanism between
 /// the predecessor and this block is a fall-through.
 bool AsmPrinter::
 isBlockOnlyReachableByFallthrough(const MachineBasicBlock *MBB) const {
   // If this is a landing pad, it isn't a fall through.  If it has no preds,
-  // then nothing falls through to it.
-  if (MBB->isEHPad() || MBB->pred_empty())
+  // then nothing falls through to it. If it is an asm-goto target it isn't
+  // a fallthrough.
+  if (MBB->isEHPad() || MBB->pred_empty() || MBB->isAsmGotoTarget())
     return false;
 
   // If there isn't exactly one predecessor, it can't be a fall through.
   if (MBB->pred_size() > 1)
     return false;
 
   // The predecessor has to be immediately before this block.
   MachineBasicBlock *Pred = *MBB->pred_begin();
   if (!Pred->isLayoutSuccessor(MBB))
for (ConstMIBundleOperands OP(MI); OP.isValid(); ++OP) {
         return false;
       if (OP->isMBB() && OP->getMBB() == MBB)
         return false;
     }
   }
 
   return true;
 }
 
 GCMetadataPrinter *AsmPrinter::GetOrCreateGCPrinter(GCStrategy &S) {
   if (!S.usesMetadata())
     return nullptr;
 
   gcp_map_type &GCMap = getGCMap(GCMetadataPrinters);
   gcp_map_type::iterator GCPI = GCMap.find(&S);
   if (GCPI != GCMap.end())
     return GCPI->second.get();
 
   auto Name = S.getName();
 
   for (GCMetadataPrinterRegistry::iterator
          I = GCMetadataPrinterRegistry::begin(),
          E = GCMetadataPrinterRegistry::end(); I != E; ++I)

lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

case '$': {
             MI->getOperand(OpNo).isMetadata()) {
           Error = true;
         } else {
           unsigned OpFlags = MI->getOperand(OpNo).getImm();
           ++OpNo;  // Skip over the ID number.
 
           if (Modifier[0] == 'l') { // Labels are target independent.
             // FIXME: What if the operand isn't an MBB, report error?
+            if (MI->getOperand(OpNo).isBlockAddress()) {
+              const BlockAddress *BA = MI->getOperand(OpNo).getBlockAddress();
+              MCSymbol *Sym =AP->GetBlockAddressSymbol(BA);
+              Sym->print(OS, AP->MAI);
+            } else {
               const MCSymbol *Sym = MI->getOperand(OpNo).getMBB()->getSymbol();
               Sym->print(OS, AP->MAI);
+            }
           } else {
             if (InlineAsm::isMemKind(OpFlags)) {
               Error = AP->PrintAsmMemoryOperand(MI, OpNo, InlineAsmVariant,
                                                 Modifier[0] ? Modifier : nullptr,
                                                 OS);
             } else {
               Error = AP->PrintAsmOperand(MI, OpNo, InlineAsmVariant,
                                           Modifier[0] ? Modifier : nullptr, OS);
             }

lib/CodeGen/BranchFolding.cpp

for (MachineBasicBlock *PBB : I->predecessors()) {
       if (PBB == IBB)
         continue;
 
       // Visit each predecessor only once.
       if (!UniquePreds.insert(PBB).second)
         continue;
 
       // Skip blocks which may jump to a landing pad. Can't tail merge these.
-      if (PBB->hasEHPadSuccessor())
+      // Same for asm goto.
+      // FIXME: Can we make asm-goto work?
+      if (PBB->hasEHPadSuccessor() || PBB->hasAsmGotoTargetSuccessor())
         continue;
 
       // After block placement, only consider predecessors that belong to the
       // same loop as IBB.  The reason is the same as above when skipping loop
       // header.
       if (AfterBlockPlacement && MLI)
         if (ML != MLI->getLoopFor(PBB))
           continue;
ReoptimizeBlock:
   // If the prior block doesn't fall through into this block, and if this
   // block doesn't fall through into some other block, see if we can find a
   // place to move this block where a fall-through will happen.
   if (!PrevBB.canFallThrough()) {
     // Now we know that there was no fall-through into this block, check to
     // see if it has a fall-through into its successor.
     bool CurFallsThru = MBB->canFallThrough();
 
-    if (!MBB->isEHPad()) {
+    if (!MBB->isEHPad() && !MBB->isAsmGotoTarget()) {
       // Check all the predecessors of this block.  If one of them has no fall
       // throughs, move this block right after it.
       for (MachineBasicBlock *PredBB : MBB->predecessors()) {
         // Analyze the branch at the end of the pred.
         MachineBasicBlock *PredTBB = nullptr, *PredFBB = nullptr;
         SmallVector<MachineOperand, 4> PredCond;
         if (PredBB != MBB && !PredBB->canFallThrough() &&
             !TII->analyzeBranch(*PredBB, PredTBB, PredFBB, PredCond, true) &&
if (!PrevBB.canFallThrough()) {
     if (!CurFallsThru) {
       // Check all successors to see if we can move this block before it.
       for (MachineBasicBlock *SuccBB : MBB->successors()) {
         // Analyze the branch at the end of the block before the succ.
         MachineFunction::iterator SuccPrev = --SuccBB->getIterator();
 
         // If this block doesn't already fall-through to that successor, and if
         // the succ doesn't already have a block that can fall through into it,
-        // and if the successor isn't an EH destination, we can arrange for the
-        // fallthrough to happen.
+        // and if the successor isn't an EH destination or asm-goto target, we
+        // can arrange for the fallthrough to happen.
         if (SuccBB != MBB && &*SuccPrev != MBB &&
             !SuccPrev->canFallThrough() && !CurUnAnalyzable &&
-            !SuccBB->isEHPad()) {
+            !SuccBB->isEHPad() && !SuccBB->isAsmGotoTarget()) {
           MBB->moveBefore(SuccBB);
           MadeChange = true;
           goto ReoptimizeBlock;
         }
       }
 
       // Okay, there is no really great place to put this block.  If, however,
       // the block before this one would be a fall-through if this block were

lib/CodeGen/CodeGenPrepare.cpp

bool CodeGenPrepare::isMergingEmptyBlockProfitable(BasicBlock *BB,
   // Loop preheaders can be good locations to spill registers. If the
   // preheader is deleted and we create a critical edge, registers may be
   // spilled in the loop body instead.
   if (!DisablePreheaderProtect && isPreheader &&
       !(BB->getSinglePredecessor() &&
         BB->getSinglePredecessor()->getSingleSuccessor()))
     return false;
 
+  // Skip merging if the block's successor is also a successor to any callbr
+  // that leads to this block.
+  for (pred_iterator PI = pred_begin(BB), E = pred_end(BB); PI != E; ++PI) {
+    if (auto *CBI = dyn_cast<CallBrInst>((*PI)->getTerminator())) {
+      if (DestBB == CBI->getFallthrough())
+        return false;
+      for (BasicBlock *CBBB : CBI->getTransfers())
+        if (DestBB == CBBB)
+          return false;
+    }
+  }
+
   // Try to skip merging if the unique predecessor of BB is terminated by a
   // switch or indirect branch instruction, and BB is used as an incoming block
   // of PHIs in DestBB. In such case, merging BB and DestBB would cause ISel to
   // add COPY instructions in the predecessor of BB instead of BB (if it is not
   // merged). Note that the critical edge created by merging such blocks wont be
   // split in MachineSink because the jump table is not analyzable. By keeping
   // such empty block (BB), ISel will place COPY instructions in BB, not in the
   // predecessor of BB.

lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateInvoke(const User &U,
   MF->addInvoke(&EHPadMBB, BeginSymbol, EndSymbol);
   MIRBuilder.getMBB().addSuccessor(&ReturnMBB);
   MIRBuilder.getMBB().addSuccessor(&EHPadMBB);
   MIRBuilder.buildBr(ReturnMBB);
 
   return true;
 }
 
+bool IRTranslator::translateCallBr(const User &U,
+                                   MachineIRBuilder &MIRBuilder) {
+  const CallBrInst &I = cast<CallBrInst>(U);
+
+  const BasicBlock *ReturnBB = I.getFallthrough();
+  SmallVector<BasicBlock *, 16> Transfers = I.getTransfers();
+
+  const Value *Callee = I.getCalledValue();
+  const Function *Fn = dyn_cast<Function>(Callee);
+
+  if (Fn && Fn->isIntrinsic())
+    return false;
+
+  // FIXME: support whatever these are.
+  if (I.countOperandBundlesOfType(LLVMContext::OB_deopt))
+    return false;
+
+  unsigned Res = I.getType()->isVoidTy() ? 0 : getOrCreateVReg(I);
+  SmallVector<unsigned, 8> Args;
+  for (auto &Arg : I.arg_operands())
+    Args.push_back(getOrCreateVReg(*Arg));
+
+  if (!CLI->lowerCall(MIRBuilder, &I, Res, Args,
+                      [&]() { return getOrCreateVReg(*I.getCalledValue()); }))
+    return false;
+
+  // FIXME: track probabilities.
+  MachineBasicBlock &ReturnMBB = getMBB(*ReturnBB);
+  MIRBuilder.getMBB().addSuccessor(&ReturnMBB);
+  for (unsigned i = 0, e = Transfers.size(); i < e; ++i)
+    MIRBuilder.getMBB().addSuccessor(&getMBB(*Transfers[i]));
+  MIRBuilder.buildBr(ReturnMBB);
+
+  return true;
+}
+
 bool IRTranslator::translateLandingPad(const User &U,
                                        MachineIRBuilder &MIRBuilder) {
   const LandingPadInst &LP = cast<LandingPadInst>(U);
 
   MachineBasicBlock &MBB = MIRBuilder.getMBB();
 
   MBB.setIsEHPad();
 

lib/CodeGen/IndirectBrExpandPass.cpp

for (BasicBlock &BB : F) {
     // because null can be compared with block addresses.
     int BBIndex = BBs.size() + 1;
     BBs.push_back(&BB);
 
     auto *ITy = cast<IntegerType>(DL.getIntPtrType(BA->getType()));
     ConstantInt *BBIndexC = ConstantInt::get(ITy, BBIndex);
 
     // Now rewrite the blockaddress to an integer constant based on the index.
-    // FIXME: We could potentially preserve the uses as arguments to inline asm.
-    // This would allow some uses such as diagnostic information in crashes to
-    // have higher quality even when this transform is enabled, but would break
-    // users that round-trip blockaddresses through inline assembly and then
-    // back into an indirectbr.
+    // FIXME: this part doesn't properly recognize other uses of blockaddress
+    // expressions, for instance, where they are used to pass labels to
+    // asm-goto. This part of the pass needs a rework.
     BA->replaceAllUsesWith(ConstantExpr::getIntToPtr(BBIndexC, BA->getType()));
   }
 
   if (BBs.empty()) {
     // There are no blocks whose address is taken, so any indirectbr instruction
     // cannot get a valid input and we can replace all of them with unreachable.
     for (auto *IBr : IndirectBrs) {
       (void)new UnreachableInst(F.getContext(), IBr);
       IBr->eraseFromParent();

lib/CodeGen/MachineBasicBlock.cpp

 
 bool MachineBasicBlock::hasEHPadSuccessor() const {
   for (const_succ_iterator I = succ_begin(), E = succ_end(); I != E; ++I)
     if ((*I)->isEHPad())
       return true;
   return false;
 }
 
+bool MachineBasicBlock::hasAsmGotoTargetSuccessor() const {
+  for (const_succ_iterator I = succ_begin(), E = succ_end(); I != E; ++I)
+    if ((*I)->isAsmGotoTarget())
+      return true;
+  return false;
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD void MachineBasicBlock::dump() const {
   print(dbgs());
 }
 #endif
 
 bool MachineBasicBlock::isLegalToHoistInto() const {
   if (isReturnBlock() || hasEHPadSuccessor())
if (hasAddressTaken()) {
     OS << "address-taken";
     HasAttributes = true;
   }
   if (isEHPad()) {
     OS << (HasAttributes ? ", " : " (");
     OS << "landing-pad";
     HasAttributes = true;
   }
+  if (isAsmGotoTarget()) {
+    OS << (HasAttributes ? ", " : " (");
+    OS << "asm-goto-target";
+    HasAttributes = true;
+  }
   if (getAlignment()) {
     OS << (HasAttributes ? ", " : " (");
     OS << "align " << getAlignment();
     HasAttributes = true;
   }
   if (HasAttributes)
     OS << ")";
   OS << ":\n";
if (TBB) {
       // layout successor, delete the branch.
       if (isLayoutSuccessor(TBB))
         TII->removeBranch(*this);
     } else {
       // The block has an unconditional fallthrough. If its successor is not its
       // layout successor, insert a branch. First we have to locate the only
       // non-landing-pad successor, as that is the fallthrough block.
       for (succ_iterator SI = succ_begin(), SE = succ_end(); SI != SE; ++SI) {
-        if ((*SI)->isEHPad())
+        if ((*SI)->isEHPad() || (*SI)->isAsmGotoTarget())
           continue;
         assert(!TBB && "Found more than one non-landing-pad successor!");
         TBB = *SI;
       }
 
       // If there is no non-landing-pad successor, the block has no fall-through
       // edges to be concerned with.
       if (!TBB)

 
 /// Given a machine basic block that branched to 'Old', change the code and CFG
 /// so that it branches to 'New' instead.
 void MachineBasicBlock::ReplaceUsesOfBlockWith(MachineBasicBlock *Old,
                                                MachineBasicBlock *New) {
   assert(Old != New && "Cannot replace self with self!");
 
   MachineBasicBlock::instr_iterator I = instr_end();
+  int Stop = 0;
   while (I != instr_begin()) {
     --I;
-    if (!I->isTerminator()) break;
+    if (!I->isTerminator() && !I->isInlineAsm()) ++Stop;
+    // FIXME: this is crude, but the asm-goto sometimes seems to be the second-
+    // to-last instruction. Find a prettier way to check this.
+    if (Stop > 1 || I->isPHI()) break;
 
     // Scan the operands of this machine instruction, replacing any uses of Old
     // with New.
     for (unsigned i = 0, e = I->getNumOperands(); i != e; ++i)
       if (I->getOperand(i).isMBB() &&
           I->getOperand(i).getMBB() == Old)
         I->getOperand(i).setMBB(New);
   }
if (IsCond)
       // Block ends in conditional jump that falls through to successor.
       DestB = FallThru;
   } else {
     assert(DestA && DestB && IsCond &&
            "CFG in a bad state. Cannot correct CFG edges");
   }
 
   // Remove superfluous edges. I.e., those which aren't destinations of this
-  // basic block, duplicate edges, or landing pads.
+  // basic block, duplicate edges, landing pads, or asm-goto targets.
   SmallPtrSet<const MachineBasicBlock*, 8> SeenMBBs;
   MachineBasicBlock::succ_iterator SI = succ_begin();
   while (SI != succ_end()) {
     const MachineBasicBlock *MBB = *SI;
     if (!SeenMBBs.insert(MBB).second ||
-        (MBB != DestA && MBB != DestB && !MBB->isEHPad())) {
+        (MBB != DestA && MBB != DestB && !MBB->isEHPad() &&
+         !MBB->isAsmGotoTarget())) {
       // This is a superfluous edge, remove it.
       SI = removeSuccessor(SI);
       Changed = true;
     } else {
       ++SI;
     }
   }
 

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

 namespace llvm {
 
 class AllocaInst;
 class AtomicCmpXchgInst;
 class AtomicRMWInst;
 class BasicBlock;
 class BranchInst;
 class CallInst;
+class CallBrInst;
 class CatchPadInst;
 class CatchReturnInst;
 class CatchSwitchInst;
 class CleanupPadInst;
 class CleanupReturnInst;
 class Constant;
 class ConstantInt;
 class ConstrainedFPIntrinsic;
void visitBitTestCase(BitTestBlock &BB,
                         MachineBasicBlock *SwitchBB);
   void visitJumpTable(JumpTable &JT);
   void visitJumpTableHeader(JumpTable &JT, JumpTableHeader &JTH,
                             MachineBasicBlock *SwitchBB);
 
 private:
   // These all get lowered before this pass.
   void visitInvoke(const InvokeInst &I);
+  void visitCallBr(const CallBrInst &I);
   void visitResume(const ResumeInst &I);
 
   void visitUnary(const User &I, unsigned Opcode);
   void visitFNeg(const User &I) { visitUnary(I, ISD::FNEG); }
 
   void visitBinary(const User &I, unsigned Opcode);
   void visitShift(const User &I, unsigned Opcode);
   void visitAdd(const User &I)  { visitBinary(I, ISD::ADD); }

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitInvoke(const InvokeInst &I) {
   addSuccessorWithProb(InvokeMBB, Return);
   for (auto &UnwindDest : UnwindDests) {
     UnwindDest.first->setIsEHPad();
     addSuccessorWithProb(InvokeMBB, UnwindDest.first, UnwindDest.second);
   }
   InvokeMBB->normalizeSuccProbs();
 
   // Drop into normal successor.
+  DAG.setRoot(DAG.getNode(ISD::BR, getCurSDLoc(), MVT::Other, getControlRoot(),
+                          DAG.getBasicBlock(Return)));
+}
+
+void SelectionDAGBuilder::visitCallBr(const CallBrInst &I) {
+  MachineBasicBlock *CallBrMBB = FuncInfo.MBB;
+
+  // Deopt bundles are lowered in LowerCallSiteWithDeoptBundle, and we don't
+  // have to do anything here to lower funclet bundles.
+  assert(!I.hasOperandBundlesOtherThan(
+             {LLVMContext::OB_deopt, LLVMContext::OB_funclet}) &&
+         "Cannot lower callbrs with arbitrary operand bundles yet!");
+
+  assert(isa<InlineAsm>(I.getCalledValue()) &&
+         "Only know how to handle inlineasm callbr");
+  visitInlineAsm(&I);
+
+  // Retrieve successors. Look through artificial IR level blocks like
+  // catchswitch for successors.
+  MachineBasicBlock *Return = FuncInfo.MBBMap[I.getSuccessor(0)];
+  SmallVector<BasicBlock *, 16> Transfers = I.getTransfers();
+
+  // Update successor info.
+  addSuccessorWithProb(CallBrMBB, Return);
+  for (auto *Transfer : Transfers) {
+    MachineBasicBlock *Target = FuncInfo.MBBMap[Transfer];
+    Target->setIsAsmGotoTarget();
+    addSuccessorWithProb(CallBrMBB, Target);
+  }
+  CallBrMBB->normalizeSuccProbs();
+
+  // Drop into normal successor.
   DAG.setRoot(DAG.getNode(ISD::BR, getCurSDLoc(),
                           MVT::Other, getControlRoot(),
                           DAG.getBasicBlock(Return)));
 }
 
 void SelectionDAGBuilder::visitResume(const ResumeInst &RI) {
   llvm_unreachable("SelectionDAGBuilder shouldn't visit resume instructions!");
 }
for (auto &T : TargetConstraints) {
 
     // Compute the value type for each operand.
     if (OpInfo.Type == InlineAsm::isInput ||
         (OpInfo.Type == InlineAsm::isOutput && OpInfo.isIndirect)) {
       OpInfo.CallOperandVal = const_cast<Value *>(CS.getArgument(ArgNo++));
 
       // Process the call argument. BasicBlocks are labels, currently appearing
       // only in asm's.
-      if (const BasicBlock *BB = dyn_cast<BasicBlock>(OpInfo.CallOperandVal)) {
+      if (isa<CallBrInst>(CS.getInstruction()) &&
+          (ArgNo - 1) >= (cast<CallBrInst>(CS.getInstruction())->getNumArgOperands() -
+                          cast<CallBrInst>(CS.getInstruction())->getNumTransfers())) {
+        const auto *BA = cast<BlockAddress>(OpInfo.CallOperandVal);
+        EVT VT = TLI.getValueType(DAG.getDataLayout(), BA->getType(), true);
+        OpInfo.CallOperand = DAG.getTargetBlockAddress(BA, VT);
+      } else if (const auto *BB = dyn_cast<BasicBlock>(OpInfo.CallOperandVal)) {
         OpInfo.CallOperand = DAG.getBasicBlock(FuncInfo.MBBMap[BB]);
       } else {
         OpInfo.CallOperand = getValue(OpInfo.CallOperandVal);
       }
 
       OpInfo.ConstraintVT =
           OpInfo
               .getCallOperandValEVT(*DAG.getContext(), TLI, DAG.getDataLayout())

lib/CodeGen/SelectionDAG/TargetLowering.cpp

void TargetLowering::LowerAsmOperandForConstraint(SDValue Op,
                                                   SelectionDAG &DAG) const {
 
   if (Constraint.length() > 1) return;
 
   char ConstraintLetter = Constraint[0];
   switch (ConstraintLetter) {
   default: break;
   case 'X':     // Allows any operand; labels (basic block) use this.
-    if (Op.getOpcode() == ISD::BasicBlock) {
+    if (Op.getOpcode() == ISD::BasicBlock ||
+        Op.getOpcode() == ISD::TargetBlockAddress) {
       Ops.push_back(Op);
       return;
     }
     LLVM_FALLTHROUGH;
   case 'i':    // Simple Integer or Relocatable Constant
   case 'n':    // Simple Integer
   case 's': {  // Relocatable Constant
     // These operands are interested in values of the form (GV+C), where C may
if (OpInfo.ConstraintCode == "X" && OpInfo.CallOperandVal) {
     // that matches labels).  For Functions, the type here is the type of
     // the result, which is not what we want to look at; leave them alone.
     Value *v = OpInfo.CallOperandVal;
     if (isa<BasicBlock>(v) || isa<ConstantInt>(v) || isa<Function>(v)) {
       OpInfo.CallOperandVal = v;
       return;
     }
 
+    if (Op.getNode() && Op.getOpcode() == ISD::TargetBlockAddress)
+      return;
+
     // Otherwise, try to resolve it to something we know about by looking at
     // the actual operand type.
     if (const char *Repl = LowerXConstraint(OpInfo.ConstraintVT)) {
       OpInfo.ConstraintCode = Repl;
       OpInfo.ConstraintType = getConstraintType(OpInfo.ConstraintCode);
     }
   }
 }

lib/CodeGen/TargetLoweringBase.cpp

 #include "llvm/IR/Instruction.def"
   };
   switch (static_cast<InstructionOpcodes>(Opcode)) {
   case Ret:            return 0;
   case Br:             return 0;
   case Switch:         return 0;
   case IndirectBr:     return 0;
   case Invoke:         return 0;
+  case CallBr:         return 0;
   case Resume:         return 0;
   case Unreachable:    return 0;
   case CleanupRet:     return 0;
   case CatchRet:       return 0;
   case CatchPad:       return 0;
   case CatchSwitch:    return 0;
   case CleanupPad:     return 0;
   case FNeg:           return ISD::FNEG;

lib/IR/AsmWriter.cpp

if (PAL.hasAttributes(AttributeList::FunctionIndex))
       Out << " #" << Machine.getAttributeGroupSlot(PAL.getFnAttributes());
 
     writeOperandBundles(II);
 
     Out << "\n          to ";
     writeOperand(II->getNormalDest(), true);
     Out << " unwind ";
     writeOperand(II->getUnwindDest(), true);
+  } else if (const CallBrInst *CBI = dyn_cast<CallBrInst>(&I)) {
+    Operand = CBI->getCalledValue();
+    FunctionType *FTy = CBI->getFunctionType();
+    Type *RetTy = FTy->getReturnType();
+    const AttributeList &PAL = CBI->getAttributes();
+
+    // Print the calling convention being used.
+    if (CBI->getCallingConv() != CallingConv::C) {
+      Out << " ";
+      PrintCallingConv(CBI->getCallingConv(), Out);
+    }
+
+    if (PAL.hasAttributes(AttributeList::ReturnIndex))
+      Out << ' ' << PAL.getAsString(AttributeList::ReturnIndex);
+
+    // If possible, print out the short form of the callbr instruction. We can
+    // only do this if the first argument is a pointer to a nonvararg function,
+    // and if the return type is not a pointer to a function.
+    //
+    Out << ' ';
+    TypePrinter.print(FTy->isVarArg() ? FTy : RetTy, Out);
+    Out << ' ';
+    writeOperand(Operand, false);
+    Out << '(';
+    for (unsigned op = 0, Eop = CBI->getNumArgOperands(); op < Eop; ++op) {
+      if (op)
+        Out << ", ";
+      writeParamOperand(CBI->getArgOperand(op), PAL.getParamAttributes(op));
+    }
+
+    Out << ')';
+    if (PAL.hasAttributes(AttributeList::FunctionIndex))
+      Out << " #" << Machine.getAttributeGroupSlot(PAL.getFnAttributes());
+
+    writeOperandBundles(CBI);
+
+    Out << "\n          to ";
+    writeOperand(CBI->getFallthrough(), true);
+    Out << " [";
+    SmallVector<BasicBlock *, 16> Transfers = CBI->getTransfers();
+    for (unsigned i = 0, e = Transfers.size(); i != e; ++i) {
+      if (i != 0)
+        Out << ", ";
+      writeOperand(Transfers[i], true);
+    }
+    Out << ']';
   } else if (const AllocaInst *AI = dyn_cast<AllocaInst>(&I)) {
     Out << ' ';
     if (AI->isUsedWithInAlloca())
       Out << "inalloca ";
     if (AI->isSwiftError())
       Out << "swifterror ";
     TypePrinter.print(AI->getAllocatedType(), Out);
 

lib/IR/Instruction.cpp

const char *Instruction::getOpcodeName(unsigned OpCode) {
   case IndirectBr: return "indirectbr";
   case Invoke: return "invoke";
   case Resume: return "resume";
   case Unreachable: return "unreachable";
   case CleanupRet: return "cleanupret";
   case CatchRet: return "catchret";
   case CatchPad: return "catchpad";
   case CatchSwitch: return "catchswitch";
+  case CallBr: return "callbr";
 
   // Standard unary operators...
   case FNeg: return "fneg";
 
   // Standard binary operators...
   case Add: return "add";
   case FAdd: return "fadd";
   case Sub: return "sub";
if (const CallInst *CI = dyn_cast<CallInst>(I1))
     return CI->isTailCall() == cast<CallInst>(I2)->isTailCall() &&
            CI->getCallingConv() == cast<CallInst>(I2)->getCallingConv() &&
            CI->getAttributes() == cast<CallInst>(I2)->getAttributes() &&
            CI->hasIdenticalOperandBundleSchema(*cast<CallInst>(I2));
   if (const InvokeInst *CI = dyn_cast<InvokeInst>(I1))
     return CI->getCallingConv() == cast<InvokeInst>(I2)->getCallingConv() &&
            CI->getAttributes() == cast<InvokeInst>(I2)->getAttributes() &&
            CI->hasIdenticalOperandBundleSchema(*cast<InvokeInst>(I2));
+  if (const CallBrInst *CI = dyn_cast<CallBrInst>(I1))
+    return CI->getCallingConv() == cast<CallBrInst>(I2)->getCallingConv() &&
+           CI->getAttributes() == cast<CallBrInst>(I2)->getAttributes() &&
+           CI->hasIdenticalOperandBundleSchema(*cast<CallBrInst>(I2));
   if (const InsertValueInst *IVI = dyn_cast<InsertValueInst>(I1))
     return IVI->getIndices() == cast<InsertValueInst>(I2)->getIndices();
   if (const ExtractValueInst *EVI = dyn_cast<ExtractValueInst>(I1))
     return EVI->getIndices() == cast<ExtractValueInst>(I2)->getIndices();
   if (const FenceInst *FI = dyn_cast<FenceInst>(I1))
     return FI->getOrdering() == cast<FenceInst>(I2)->getOrdering() &&
            FI->getSyncScopeID() == cast<FenceInst>(I2)->getSyncScopeID();
   if (const AtomicCmpXchgInst *CXI = dyn_cast<AtomicCmpXchgInst>(I1))
bool Instruction::mayReadFromMemory() const {
   case Instruction::AtomicRMW:
   case Instruction::CatchPad:
   case Instruction::CatchRet:
     return true;
   case Instruction::Call:
     return !cast<CallInst>(this)->doesNotAccessMemory();
   case Instruction::Invoke:
     return !cast<InvokeInst>(this)->doesNotAccessMemory();
+  case Instruction::CallBr:
+    return !cast<CallBrInst>(this)->doesNotAccessMemory();
   case Instruction::Store:
     return !cast<StoreInst>(this)->isUnordered();
   }
 }
 
 bool Instruction::mayWriteToMemory() const {
   switch (getOpcode()) {
   default: return false;
   case Instruction::Fence: // FIXME: refine definition of mayWriteToMemory
   case Instruction::Store:
   case Instruction::VAArg:
   case Instruction::AtomicCmpXchg:
   case Instruction::AtomicRMW:
   case Instruction::CatchPad:
   case Instruction::CatchRet:
     return true;
   case Instruction::Call:
     return !cast<CallInst>(this)->onlyReadsMemory();
   case Instruction::Invoke:
     return !cast<InvokeInst>(this)->onlyReadsMemory();
+  case Instruction::CallBr:
+    return !cast<CallBrInst>(this)->onlyReadsMemory();
   case Instruction::Load:
     return !cast<LoadInst>(this)->isUnordered();
   }
 }
 
 bool Instruction::isAtomic() const {
   switch (getOpcode()) {
   default:
for (unsigned i = 1; i < ProfileData->getNumOperands(); i += 2) {
       Vals.push_back(MDB.createConstant(
           ConstantInt::get(Type::getInt64Ty(getContext()),
                            Val.udiv(APT).getLimitedValue())));
     }
   setMetadata(LLVMContext::MD_prof, MDNode::get(getContext(), Vals));
 }
 
 void Instruction::setProfWeight(uint64_t W) {
-  assert((isa<CallInst>(this) || isa<InvokeInst>(this)) &&
-         "Can only set weights for call and invoke instrucitons");
+  assert((isa<CallInst>(this) || isa<InvokeInst>(this) ||
+          isa<CallBrInst>(this)) &&
+         "Can only set weights for call, invoke and callbr instructions");
   SmallVector<uint32_t, 1> Weights;
   Weights.push_back(W);
   MDBuilder MDB(getContext());
   setMetadata(LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
 }

lib/IR/Instructions.cpp

 }
 
 //===----------------------------------------------------------------------===//
 //                        CallBase Implementation
 //===----------------------------------------------------------------------===//
 
 Function *CallBase::getCaller() { return getParent()->getParent(); }
 
+unsigned CallBase::getNumSubclassExtraOperandsDynamic() const {
+  assert(getOpcode() == Instruction::CallBr && "Unexpected opcode!");
+  return cast<CallBrInst>(this)->getNumTransfers() + 1;
+}
+
 bool CallBase::isIndirectCall() const {
   const Value *V = getCalledValue();
   if (isa<Function>(V) || isa<Constant>(V))
     return false;
   if (const CallInst *CI = dyn_cast<CallInst>(this))
     if (CI->isInlineAsm())
       return false;
   return true;

 }
 
 
 LandingPadInst *InvokeInst::getLandingPadInst() const {
   return cast<LandingPadInst>(getUnwindDest()->getFirstNonPHI());
 }
 
 //===----------------------------------------------------------------------===//
+//                        CallBrInst Implementation
+//===----------------------------------------------------------------------===//
+
+void CallBrInst::init(FunctionType *FTy, Value *Fn, BasicBlock *Fallthrough,
+                      ArrayRef<BasicBlock *> Transfers, ArrayRef<Value *> Args,
+                      ArrayRef<OperandBundleDef> Bundles,
+                      const Twine &NameStr) {
+  this->FTy = FTy;
+
+  assert(getNumOperands() ==
+             2 + Transfers.size() + Args.size() + CountBundleInputs(Bundles) &&
+         "NumOperands not set up?");
+  NumTransfers = Transfers.size();
+  setCalledFunction(Fn);
+  setFallthrough(Fallthrough);
+  for (unsigned i = 0, e = Transfers.size(); i < e; ++i)
+    setTransfer(i, Transfers[i]);
+
+#ifndef NDEBUG
+  assert(((Args.size() == FTy->getNumParams()) ||
+          (FTy->isVarArg() && Args.size() > FTy->getNumParams())) &&
+         "Calling a function with bad signature");
+
+  for (unsigned i = 0, e = Args.size(); i != e; i++)
+    assert((i >= FTy->getNumParams() ||
+            FTy->getParamType(i) == Args[i]->getType()) &&
+           "Calling a function with a bad signature!");
+#endif
+
+  std::copy(Args.begin(), Args.end(), op_begin());
+
+  auto It = populateBundleOperandInfos(Bundles, Args.size());
+  (void)It;
+  assert(It + 2 + Transfers.size() == op_end() && "Should add up!");
+
+  setName(NameStr);
+}
+
+CallBrInst::CallBrInst(const CallBrInst &CBI)
+    : CallBase(CBI.Attrs, CBI.FTy, CBI.getType(), Instruction::CallBr,
+               OperandTraits<CallBase>::op_end(this) - CBI.getNumOperands(),
+               CBI.getNumOperands()) {
+  setCallingConv(CBI.getCallingConv());
+  std::copy(CBI.op_begin(), CBI.op_end(), op_begin());
+  std::copy(CBI.bundle_op_info_begin(), CBI.bundle_op_info_end(),
+            bundle_op_info_begin());
+  SubclassOptionalData = CBI.SubclassOptionalData;
+  NumTransfers = CBI.NumTransfers;
+}
+
+CallBrInst *CallBrInst::Create(CallBrInst *CBI, ArrayRef<OperandBundleDef> OpB,
+                               Instruction *InsertPt) {
+  std::vector<Value *> Args(CBI->arg_begin(), CBI->arg_end());
+
+  auto *NewCBI = CallBrInst::Create(CBI->getCalledValue(),
+                                    CBI->getFallthrough(), CBI->getTransfers(),
+                                    Args, OpB, CBI->getName(), InsertPt);
+  NewCBI->setCallingConv(CBI->getCallingConv());
+  NewCBI->SubclassOptionalData = CBI->SubclassOptionalData;
+  NewCBI->setAttributes(CBI->getAttributes());
+  NewCBI->setDebugLoc(CBI->getDebugLoc());
+  NewCBI->NumTransfers = CBI->NumTransfers;
+  return NewCBI;
+}
+
+//===----------------------------------------------------------------------===//
 //                        ReturnInst Implementation
 //===----------------------------------------------------------------------===//
 
 ReturnInst::ReturnInst(const ReturnInst &RI)
     : Instruction(Type::getVoidTy(RI.getContext()), Instruction::Ret,
                   OperandTraits<ReturnInst>::op_end(this) - RI.getNumOperands(),
                   RI.getNumOperands()) {
   if (RI.getNumOperands())

 InvokeInst *InvokeInst::cloneImpl() const {
   if (hasOperandBundles()) {
     unsigned DescriptorBytes = getNumOperandBundles() * sizeof(BundleOpInfo);
     return new(getNumOperands(), DescriptorBytes) InvokeInst(*this);
   }
   return new(getNumOperands()) InvokeInst(*this);
 }
 
+CallBrInst *CallBrInst::cloneImpl() const {
+  if (hasOperandBundles()) {
+    unsigned DescriptorBytes = getNumOperandBundles() * sizeof(BundleOpInfo);
+    return new (getNumOperands(), DescriptorBytes) CallBrInst(*this);
+  }
+  return new (getNumOperands()) CallBrInst(*this);
+}
+
 ResumeInst *ResumeInst::cloneImpl() const { return new (1) ResumeInst(*this); }
 
 CleanupReturnInst *CleanupReturnInst::cloneImpl() const {
   return new (getNumOperands()) CleanupReturnInst(*this);
 }
 
 CatchReturnInst *CatchReturnInst::cloneImpl() const {
   return new (getNumOperands()) CatchReturnInst(*this);

lib/IR/Value.cpp

 Value::Value(Type *ty, unsigned scid)
     : VTy(checkType(ty)), UseList(nullptr), SubclassID(scid),
       HasValueHandle(0), SubclassOptionalData(0), SubclassData(0),
       NumUserOperands(0), IsUsedByMD(false), HasName(false) {
   static_assert(ConstantFirstVal == 0, "!(SubclassID < ConstantFirstVal)");
   // FIXME: Why isn't this in the subclass gunk??
   // Note, we cannot call isa<CallInst> before the CallInst has been
   // constructed.
-  if (SubclassID == Instruction::Call || SubclassID == Instruction::Invoke)
+  if (SubclassID == Instruction::Call || SubclassID == Instruction::Invoke ||
+      SubclassID == Instruction::CallBr)
     assert((VTy->isFirstClassType() || VTy->isVoidTy() || VTy->isStructTy()) &&
            "invalid CallInst type!");
   else if (SubclassID != BasicBlockVal &&
            (/*SubclassID < ConstantFirstVal ||*/ SubclassID > ConstantLastVal))
     assert((VTy->isFirstClassType() || VTy->isVoidTy()) &&
            "Cannot create non-first-class values except for constants!");
   static_assert(sizeof(Value) == 2 * sizeof(void *) + 2 * sizeof(unsigned),
                 "Value too big");

lib/IR/Verifier.cpp

#include "llvm/IR/Metadata.def"
   void visitStoreInst(StoreInst &SI);
   void verifyDominatesUse(Instruction &I, unsigned i);
   void visitInstruction(Instruction &I);
   void visitTerminator(Instruction &I);
   void visitBranchInst(BranchInst &BI);
   void visitReturnInst(ReturnInst &RI);
   void visitSwitchInst(SwitchInst &SI);
   void visitIndirectBrInst(IndirectBrInst &BI);
+  void visitCallBrInst(CallBrInst &CBI);
   void visitSelectInst(SelectInst &SI);
   void visitUserOp1(Instruction &I);
   void visitUserOp2(Instruction &I) { visitUserOp1(I); }
   void visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call);
   void visitConstrainedFPIntrinsic(ConstrainedFPIntrinsic &FPI);
   void visitDbgIntrinsic(StringRef Kind, DbgVariableIntrinsic &DII);
   void visitDbgLabelIntrinsic(StringRef Kind, DbgLabelInst &DLI);
   void visitAtomicCmpXchgInst(AtomicCmpXchgInst &CXI);
Assert(BI.getAddress()->getType()->isPointerTy(),
          "Indirectbr operand must have pointer type!", &BI);
   for (unsigned i = 0, e = BI.getNumDestinations(); i != e; ++i)
     Assert(BI.getDestination(i)->getType()->isLabelTy(),
            "Indirectbr destinations must all have pointer type!", &BI);
 
   visitTerminator(BI);
 }
 
+void Verifier::visitCallBrInst(CallBrInst &CBI) {
+  Assert(CBI.isInlineAsm(), "Callbr is currently only used for asm-goto!",
+         &CBI);
+  for (unsigned i = 0, e = CBI.getNumSuccessors(); i != e; ++i)
+    Assert(CBI.getSuccessor(i)->getType()->isLabelTy(),
+           "Callbr successors must all have pointer type!", &CBI);
+  for (unsigned i = 0, e = CBI.getNumOperands(); i != e; ++i) {
+    Assert(i >= CBI.getNumArgOperands() || !isa<BasicBlock>(CBI.getOperand(i)),
+           "Using an unescaped label as a callbr argument!", &CBI);
+    if (isa<BasicBlock>(CBI.getOperand(i)))
+      for (unsigned j = i + 1; j != e; ++j)
+        Assert(CBI.getOperand(i) != CBI.getOperand(j),
+               "Duplicate callbr destination!", &CBI);
+  }
+
+  visitTerminator(CBI);
+}
+
 void Verifier::visitSelectInst(SelectInst &SI) {
   Assert(!SelectInst::areInvalidOperands(SI.getOperand(0), SI.getOperand(1),
                                          SI.getOperand(2)),
          "Invalid operands for select instruction!", &SI);
 
   Assert(SI.getTrueValue()->getType() == SI.getType(),
          "Select values must have same type as select instruction!", &SI);
   visitInstruction(SI);

lib/Target/X86/X86AsmPrinter.cpp

case MachineOperand::MO_Immediate:
     O << MO.getImm();
     return;
 
   case MachineOperand::MO_GlobalAddress: {
     if (AsmVariant == 0) O << '$';
     printSymbolOperand(P, MO, O);
     break;
   }
+  case MachineOperand::MO_BlockAddress: {
+    MCSymbol *Sym = P.GetBlockAddressSymbol(MO.getBlockAddress());
+    Sym->print(O, P.MAI);
+    break;
+  }
   }
 }
 
 static void printLeaMemReference(X86AsmPrinter &P, const MachineInstr *MI,
                                  unsigned Op, raw_ostream &O,
                                  const char *Modifier = nullptr) {
   const MachineOperand &BaseReg  = MI->getOperand(Op+X86::AddrBaseReg);
   const MachineOperand &IndexReg = MI->getOperand(Op+X86::AddrIndexReg);

lib/Target/X86/X86InstrInfo.cpp

bool X86InstrInfo::AnalyzeBranchImpl(
   MachineBasicBlock::iterator UnCondBrIter = MBB.end();
   while (I != MBB.begin()) {
     --I;
     if (I->isDebugInstr())
       continue;
 
     // Working from the bottom, when we see a non-terminator instruction, we're
     // done.
-    if (!isUnpredicatedTerminator(*I))
+    if (!isUnpredicatedTerminator(*I)) {
+      // If the non-terminator is an inline assembly block, we pessimistically
+      // assume it's an asm-goto that can't be handled by this analysis.
+      // In cases of non-void returns, the asm may be the second-to-last
+      // non-terminator instead.
+      // TODO: make a more precise check for asm-goto with outputs to reduce
+      // false positives
+      if (I->getOpcode() == X86::INLINEASM ||
+          (I != MBB.begin() && I->getOpcode() == TargetOpcode::COPY &&
+           (--I)->getOpcode() == X86::INLINEASM))
+        return true;
+      else
         break;
+    }
 
     // A terminator that isn't a branch can't easily be handled by this
     // analysis.
     if (!I->isBranch())
       return true;
 
     // Handle unconditional branches.
     if (I->getOpcode() == X86::JMP_1) {

lib/Transforms/InstCombine/InstCombineCalls.cpp

 //===- InstCombineCalls.cpp -----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements the visitCall and visitInvoke functions.
+// This file implements the visitCall, visitInvoke and visitCallBr functions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "InstCombineInternal.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/None.h"
Instruction *InstCombiner::visitCallInst(CallInst &CI) {
   if (CI.getFunction()->doesNotThrow() && !CI.doesNotThrow()) {
     CI.setDoesNotThrow();
     return &CI;
   }
 
   IntrinsicInst *II = dyn_cast<IntrinsicInst>(&CI);
   if (!II) return visitCallSite(&CI);
 
-  // Intrinsics cannot occur in an invoke, so handle them here instead of in
-  // visitCallSite.
+  // Intrinsics cannot occur in an invoke or a callbr, so handle them here
+  // instead of in visitCallSite.
   if (auto *MI = dyn_cast<AnyMemIntrinsic>(II)) {
     bool Changed = false;
 
     // memmove/cpy/set of zero bytes is a noop.
     if (Constant *NumBytes = dyn_cast<Constant>(MI->getLength())) {
       if (NumBytes->isNullValue())
         return eraseInstFromFunction(CI);
 
Instruction *InstCombiner::visitFenceInst(FenceInst &FI) {
   return nullptr;
 }
 
 // InvokeInst simplification
 Instruction *InstCombiner::visitInvokeInst(InvokeInst &II) {
   return visitCallSite(&II);
 }
 
+// CallBrInst simplification
+Instruction *InstCombiner::visitCallBrInst(CallBrInst &CBI) {
+  return visitCallSite(&CBI);
+}
+
 /// If this cast does not affect the value passed through the varargs area, we
 /// can eliminate the use of the cast.
 static bool isSafeToEliminateVarargsCast(const CallSite CS,
                                          const DataLayout &DL,
                                          const CastInst *const CI,
                                          const int ix) {
   if (!CI->isLosslessCast())
     return false;
static IntrinsicInst *findInitTrampoline(Value *Callee) {
 
   if (IntrinsicInst *IT = findInitTrampolineFromAlloca(TrampMem))
     return IT;
   if (IntrinsicInst *IT = findInitTrampolineFromBB(AdjustTramp, TrampMem))
     return IT;
   return nullptr;
 }
 
-/// Improvements for call and invoke instructions.
+/// Improvements for call, invoke and callbr instructions.
 Instruction *InstCombiner::visitCallSite(CallSite CS) {
   if (isAllocLikeFn(CS.getInstruction(), &TLI))
     return visitAllocSite(*CS.getInstruction());
 
   bool Changed = false;
 
   // Mark any parameters that are known to be non-null with the nonnull
   // attribute.  This is helpful for inlining calls to functions with null
if (!ArgNos.empty()) {
     LLVMContext &Ctx = CS.getInstruction()->getContext();
     AS = AS.addParamAttribute(Ctx, ArgNos,
                               Attribute::get(Ctx, Attribute::NonNull));
     CS.setAttributes(AS);
     Changed = true;
   }
 
   // If the callee is a pointer to a function, attempt to move any casts to the
-  // arguments of the call/invoke.
+  // arguments of the call/invoke/callbr.
   Value *Callee = CS.getCalledValue();
   if (!isa<Function>(Callee) && transformConstExprCastCall(CS))
     return nullptr;
 
   if (Function *CalleeF = dyn_cast<Function>(Callee)) {
     // Remove the convergent attr on calls when the callee is not convergent.
     if (CS.isConvergent() && !CalleeF->isConvergent() &&
         !CalleeF->isIntrinsic()) {
if (CalleeF->getCallingConv() != CS.getCallingConv() &&
                                   OldCall);
       // If OldCall does not return void then replaceAllUsesWith undef.
       // This allows ValueHandlers and custom metadata to adjust itself.
       if (!OldCall->getType()->isVoidTy())
         replaceInstUsesWith(*OldCall, UndefValue::get(OldCall->getType()));
       if (isa<CallInst>(OldCall))
         return eraseInstFromFunction(*OldCall);
 
-      // We cannot remove an invoke, because it would change the CFG, just
-      // change the callee to a null pointer.
-      cast<InvokeInst>(OldCall)->setCalledFunction(
-                                    Constant::getNullValue(CalleeF->getType()));
+      // We cannot remove an invoke or a callbr, because it would change the
+      // CFG, just change the callee to a null pointer.
+      cast<CallBase>(OldCall)->setCalledFunction(
       return nullptr;
     }
   }
 
   if ((isa<ConstantPointerNull>(Callee) &&
        !NullPointerIsDefined(CS.getInstruction()->getFunction())) ||
       isa<UndefValue>(Callee)) {
     // If CS does not return void then replaceAllUsesWith undef.
     // This allows ValueHandlers and custom metadata to adjust itself.
     if (!CS.getInstruction()->getType()->isVoidTy())
       replaceInstUsesWith(*CS.getInstruction(),
                           UndefValue::get(CS.getInstruction()->getType()));
 
-    if (isa<InvokeInst>(CS.getInstruction())) {
-      // Can't remove an invoke because we cannot change the CFG.
+    if (isa<InvokeInst>(CS.getInstruction()) ||
+        isa<CallBrInst>(CS.getInstruction())) {
+      // Can't remove an invoke or callbr because we cannot change the CFG.
       return nullptr;
     }
 
     // This instruction is not reachable, just remove it.  We insert a store to
     // undef so that we know that this code is not reachable, despite the fact
     // that we can't modify the CFG here.
     new StoreInst(ConstantInt::getTrue(Callee->getContext()),
                   UndefValue::get(Type::getInt1PtrTy(Callee->getContext())),
                   CS.getInstruction());
 
     return eraseInstFromFunction(*CS.getInstruction());
   }
if (CallInst *CI = dyn_cast<CallInst>(CS.getInstruction())) {
     // the fallthrough check.
     if (I) return eraseInstFromFunction(*I);
   }
 
   return Changed ? CS.getInstruction() : nullptr;
 }
 
 /// If the callee is a constexpr cast of a function, attempt to move the cast to
-/// the arguments of the call/invoke.
+/// the arguments of the call/invoke/callbr.
 bool InstCombiner::transformConstExprCastCall(CallSite CS) {
   auto *Callee = dyn_cast<Function>(CS.getCalledValue()->stripPointerCasts());
   if (!Callee)
     return false;
 
   // If this is a call to a thunk function, don't remove the cast. Thunks are
   // used to transparently forward all incoming parameters and outgoing return
   // values, so it's important to leave the cast in place.
if (OldRetTy != NewRetTy) {
     }
 
     if (!CallerPAL.isEmpty() && !Caller->use_empty()) {
       AttrBuilder RAttrs(CallerPAL, AttributeList::ReturnIndex);
       if (RAttrs.overlaps(AttributeFuncs::typeIncompatible(NewRetTy)))
         return false;   // Attribute not compatible with transformed value.
     }
 
-    // If the callsite is an invoke instruction, and the return value is used by
-    // a PHI node in a successor, we cannot change the return type of the call
-    // because there is no place to put the cast instruction (without breaking
-    // the critical edge).  Bail out in this case.
-    if (!Caller->use_empty())
-      if (InvokeInst *II = dyn_cast<InvokeInst>(Caller))
+    // If the callsite is an invoke/callbr instruction, and the return value is
+    // used by a PHI node in a successor, we cannot change the return type of
+    // the call because there is no place to put the cast instruction (without
+    // breaking the critical edge).  Bail out in this case.
+    if (!Caller->use_empty()) {
+      if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
         for (User *U : II->users())
           if (PHINode *PN = dyn_cast<PHINode>(U))
             if (PN->getParent() == II->getNormalDest() ||
                 PN->getParent() == II->getUnwindDest())
               return false;
+      } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+        for (User *U : CBI->users())
+          if (PHINode *PN = dyn_cast<PHINode>(U)) {
+            if (PN->getParent() == CBI->getFallthrough())
+              return false;
+            for (unsigned i = 0, e = CBI->getNumTransfers(); i != e; ++i)
+              if (PN->getParent() == CBI->getTransfer(i))
+                return false;
+          }
+      }
+    }
   }
 
   unsigned NumActualArgs = CS.arg_size();
   unsigned NumCommonArgs = std::min(FT->getNumParams(), NumActualArgs);
 
   // Prevent us turning:
   // declare void @takes_i32_inalloca(i32* inalloca)
   //  call void bitcast (void (i32*)* @takes_i32_inalloca to void (i32)*)(i32 0)
   //
bool InstCombiner::transformConstExprCastCall(CallSite CS) {
 
   SmallVector<OperandBundleDef, 1> OpBundles;
   CS.getOperandBundlesAsDefs(OpBundles);
 
   CallSite NewCS;
   if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
     NewCS = Builder.CreateInvoke(Callee, II->getNormalDest(),
                                  II->getUnwindDest(), Args, OpBundles);
+  } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+    NewCS = Builder.CreateCallBr(Callee, CBI->getFallthrough(),
+                                 CBI->getTransfers(), Args, OpBundles);
   } else {
     NewCS = Builder.CreateCall(Callee, Args, OpBundles);
     cast<CallInst>(NewCS.getInstruction())
         ->setTailCallKind(cast<CallInst>(Caller)->getTailCallKind());
   }
   NewCS->takeName(Caller);
   NewCS.setCallingConv(CS.getCallingConv());
   NewCS.setAttributes(NewCallerPAL);
 
   // Preserve the weight metadata for the new call instruction. The metadata
   // is used by SamplePGO to check callsite's hotness.
   uint64_t W;
   if (Caller->extractProfTotalWeight(W))
     NewCS->setProfWeight(W);
 
   // Insert a cast of the return type as necessary.
   Instruction *NC = NewCS.getInstruction();
   Value *NV = NC;
   if (OldRetTy != NV->getType() && !Caller->use_empty()) {
     if (!NV->getType()->isVoidTy()) {
       NV = NC = CastInst::CreateBitOrPointerCast(NC, OldRetTy);
       NC->setDebugLoc(Caller->getDebugLoc());
 
-      // If this is an invoke instruction, we should insert it after the first
-      // non-phi, instruction in the normal successor block.
+      // If this is an invoke/callbr instruction, we should insert it after the
+      // first non-phi instruction in the normal successor block.
       if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
         BasicBlock::iterator I = II->getNormalDest()->getFirstInsertionPt();
         InsertNewInstBefore(NC, *I);
+      } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+        BasicBlock::iterator I = CBI->getFallthrough()->getFirstInsertionPt();
+        InsertNewInstBefore(NC, *I);
       } else {
         // Otherwise, it's a call, just insert cast right after the call.
         InsertNewInstBefore(NC, *Caller);
       }
       Worklist.AddUsersToWorkList(*Caller);
     } else {
       NV = UndefValue::get(Caller->getType());
     }
if (NestTy) {
 
       Instruction *NewCaller;
       if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
         NewCaller = InvokeInst::Create(NewCallee,
                                        II->getNormalDest(), II->getUnwindDest(),
                                        NewArgs, OpBundles);
         cast<InvokeInst>(NewCaller)->setCallingConv(II->getCallingConv());
         cast<InvokeInst>(NewCaller)->setAttributes(NewPAL);
+      } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+        NewCaller =
+            CallBrInst::Create(NewCallee, CBI->getFallthrough(),
+                               CBI->getTransfers(), NewArgs, OpBundles);
+        cast<CallBrInst>(NewCaller)->setCallingConv(CBI->getCallingConv());
+        cast<CallBrInst>(NewCaller)->setAttributes(NewPAL);
       } else {
         NewCaller = CallInst::Create(NewCallee, NewArgs, OpBundles);
         cast<CallInst>(NewCaller)->setTailCallKind(
             cast<CallInst>(Caller)->getTailCallKind());
         cast<CallInst>(NewCaller)->setCallingConv(
             cast<CallInst>(Caller)->getCallingConv());
         cast<CallInst>(NewCaller)->setAttributes(NewPAL);
       }

lib/Transforms/InstCombine/InstCombineInternal.h

public:
   Instruction *visitPtrToInt(PtrToIntInst &CI);
   Instruction *visitIntToPtr(IntToPtrInst &CI);
   Instruction *visitBitCast(BitCastInst &CI);
   Instruction *visitAddrSpaceCast(AddrSpaceCastInst &CI);
   Instruction *FoldItoFPtoI(Instruction &FI);
   Instruction *visitSelectInst(SelectInst &SI);
   Instruction *visitCallInst(CallInst &CI);
   Instruction *visitInvokeInst(InvokeInst &II);
+  Instruction *visitCallBrInst(CallBrInst &CBI);
 
   Instruction *SliceUpIllegalIntegerPHI(PHINode &PN);
   Instruction *visitPHINode(PHINode &PN);
   Instruction *visitGetElementPtrInst(GetElementPtrInst &GEP);
   Instruction *visitAllocaInst(AllocaInst &AI);
   Instruction *visitAllocSite(Instruction &FI);
   Instruction *visitFree(CallInst &FI);
   Instruction *visitLoadInst(LoadInst &LI);

lib/Transforms/InstCombine/InstructionCombining.cpp

if (isa<Constant>(InVal) && !isa<ConstantExpr>(InVal))
       continue;
 
     if (isa<PHINode>(InVal)) return nullptr;  // Itself a phi.
     if (NonConstBB) return nullptr;  // More than one non-const value.
 
     NonConstBB = PN->getIncomingBlock(i);
 
     // If the InVal is an invoke at the end of the pred block, then we can't
-    // insert a computation after it without breaking the edge.
-    if (InvokeInst *II = dyn_cast<InvokeInst>(InVal))
-      if (II->getParent() == NonConstBB)
+    // insert a computation after it without breaking the edge. Same for callbr.
+    if (isa<InvokeInst>(InVal) || isa<CallBrInst>(InVal))
+      if (cast<Instruction>(InVal)->getParent() == NonConstBB)
         return nullptr;
 
     // If the incoming non-constant value is in I's block, we will remove one
     // instruction, but insert another equivalent one, leading to infinite
     // instcombine.
     if (isPotentiallyReachable(I.getParent(), NonConstBB, &DT, LI))
       return nullptr;
   }
 
   // If there is exactly one non-constant value, we can insert a copy of the
   // operation in that block.  However, if this is a critical edge, we would be
if (isAllocSiteRemovable(&MI, Users, &TLI)) {
     if (InvokeInst *II = dyn_cast<InvokeInst>(&MI)) {
       // Replace invoke with a NOP intrinsic to maintain the original CFG
       Module *M = II->getModule();
       Function *F = Intrinsic::getDeclaration(M, Intrinsic::donothing);
       InvokeInst::Create(F, II->getNormalDest(), II->getUnwindDest(),
                          None, "", II->getParent());
     }
 
+    if (CallBrInst *CBI = dyn_cast<CallBrInst>(&MI)) {
+      // Replace callbr with a NOP intrinsic to maintain the original CFG
+      Module *M = CBI->getModule();
+      Function *F = Intrinsic::getDeclaration(M, Intrinsic::donothing);
+      CallBrInst::Create(F, CBI->getFallthrough(), CBI->getTransfers(), None,
+                         "", CBI->getParent());
+    }
+
     for (auto *DII : DIIs)
       eraseInstFromFunction(*DII);
 
     return eraseInstFromFunction(MI);
   }
   return nullptr;
 }
 

lib/Transforms/Scalar/GVN.cpp

for (BasicBlock *Pred : predecessors(LoadBB)) {
     if (Pred->getTerminator()->getNumSuccessors() != 1) {
       if (isa<IndirectBrInst>(Pred->getTerminator())) {
         LLVM_DEBUG(
             dbgs() << "COULD NOT PRE LOAD BECAUSE OF INDBR CRITICAL EDGE '"
                    << Pred->getName() << "': " << *LI << '\n');
         return false;
       }
 
+      // FIXME: Can we support the fallthrough edge?
+      if (isa<CallBrInst>(Pred->getTerminator())) {
+        LLVM_DEBUG(
+            dbgs() << "COULD NOT PRE LOAD BECAUSE OF CALLBR CRITICAL EDGE '"
+                   << Pred->getName() << "': " << *LI << '\n');
+        return false;
+      }
+
       if (LoadBB->isEHPad()) {
         LLVM_DEBUG(
             dbgs() << "COULD NOT PRE LOAD BECAUSE OF AN EH PAD CRITICAL EDGE '"
                    << Pred->getName() << "': " << *LI << '\n');
         return false;
       }
 
       CriticalEdgePred.push_back(Pred);
bool GVN::performScalarPRE(Instruction *CurInst) {
   // GEP's live range increases the register pressure, and therefore it can
   // introduce unnecessary spills.
   //
   // This doesn't prevent Load PRE. PHI translation will make the GEP available
   // to the load by moving it to the predecessor block if necessary.
   if (isa<GetElementPtrInst>(CurInst))
     return false;
 
   // We don't currently value number ANY inline asm calls.
   if (CallInst *CallI = dyn_cast<CallInst>(CurInst))
     if (CallI->isInlineAsm())
       return false;
 
+  if (CallBrInst *CBI = dyn_cast<CallBrInst>(CurInst))
+    if (CBI->isInlineAsm())
+      return false;
+
   uint32_t ValNo = VN.lookup(CurInst);
 
   // Look for the predecessors for PRE opportunities.  We're
   // only trying to solve the basic diamond case, where
   // a value is computed in the successor and one predecessor,
   // but not the other.  We also explicitly disallow cases
   // where the successor is its own predecessor, because they're
   // more complicated to get right.
if (!isSafeToSpeculativelyExecute(CurInst)) {
       if (ICF->isDominatedByICFIFromSameBlock(CurInst))
         return false;
     }
 
     // Don't do PRE across indirect branch.
     if (isa<IndirectBrInst>(PREPred->getTerminator()))
       return false;
 
+    // Don't do PRE across callbr.
+    // FIXME: Can we do this across the fallthrough edge?
+    if (isa<CallBrInst>(PREPred->getTerminator()))
+      return false;
+
     // We can't do PRE safely on a critical edge, so instead we schedule
     // the edge to be split and perform the PRE the next time we iterate
     // on the function.
     unsigned SuccNum = GetSuccessorNumber(PREPred, CurrentBlock);
     if (isCriticalEdge(PREPred->getTerminator(), SuccNum)) {
       toSplit.push_back(std::make_pair(PREPred->getTerminator(), SuccNum));
       return false;
     }

lib/Transforms/Scalar/JumpThreading.cpp

bool JumpThreadingPass::ProcessBlock(BasicBlock *BB) {
   } else if (SwitchInst *SI = dyn_cast<SwitchInst>(Terminator)) {
     Condition = SI->getCondition();
   } else if (IndirectBrInst *IB = dyn_cast<IndirectBrInst>(Terminator)) {
     // Can't thread indirect branch with no successors.
     if (IB->getNumSuccessors() == 0) return false;
     Condition = IB->getAddress()->stripPointerCasts();
     Preference = WantBlockAddress;
   } else {
-    return false; // Must be an invoke.
+    return false; // Must be an invoke or callbr.
   }
 
   // Run constant folding to see if we can reduce the condition to a simple
   // constant.
   if (Instruction *I = dyn_cast<Instruction>(Condition)) {
     Value *SimpleVal =
         ConstantFoldInstruction(I, BB->getModule()->getDataLayout(), TLI);
     if (SimpleVal) {
if (PredsScanned.size() == AvailablePreds.size()+1 &&
     SmallPtrSet<BasicBlock*, 8> AvailablePredSet;
 
     for (const auto &AvailablePred : AvailablePreds)
       AvailablePredSet.insert(AvailablePred.first);
 
     // Add all the unavailable predecessors to the PredsToSplit list.
     for (BasicBlock *P : predecessors(LoadBB)) {
       // If the predecessor is an indirect goto, we can't split the edge.
-      if (isa<IndirectBrInst>(P->getTerminator()))
+      // Same for CallBr.
+      if (isa<IndirectBrInst>(P->getTerminator()) ||
+          isa<CallBrInst>(P->getTerminator()))
         return false;
 
       if (!AvailablePredSet.count(P))
         PredsToSplit.push_back(P);
     }
 
     // Split them out to their own block.
     UnavailablePred = SplitBlockPreds(LoadBB, PredsToSplit, "thread-pre-split");
if (PredToDestList.empty()) {
       if (Val != OnlyVal)
         OnlyVal = MultipleVal;
     }
 
     // We know where this predecessor is going.
     ++PredWithKnownDest;
 
     // If the predecessor ends with an indirect goto, we can't change its
-    // destination.
-    if (isa<IndirectBrInst>(Pred->getTerminator()))
+    // destination. Same for CallBr.
+    if (isa<IndirectBrInst>(Pred->getTerminator()) ||
+        isa<CallBrInst>(Pred->getTerminator()))
       continue;
 
     PredToDestList.push_back(std::make_pair(Pred, DestBB));
   }
 
   // If all edges were unthreadable, we fail.
   if (PredToDestList.empty())
     return false;

lib/Transforms/Scalar/SCCP.cpp

void visitCallInst      (CallInst &I) {
     visitCallSite(&I);
   }
 
   void visitInvokeInst    (InvokeInst &II) {
     visitCallSite(&II);
     visitTerminator(II);
   }
 
+  void visitCallBrInst    (CallBrInst &CBI) {
+    visitCallSite(&CBI);
+    visitTerminator(CBI);
+  }
+
   void visitCallSite      (CallSite CS);
   void visitResumeInst    (ResumeInst &I) { /*returns void*/ }
   void visitUnreachableInst(UnreachableInst &I) { /*returns void*/ }
   void visitFenceInst     (FenceInst &I) { /*returns void*/ }
 
   void visitInstruction(Instruction &I) {
     // All the instructions we don't do any special handling for just
     // go to overdefined.
for (unsigned i = 0; i < IBR->getNumSuccessors(); ++i) {
       }
     }
 
     // If we didn't find our destination in the IBR successor list, then we
     // have undefined behavior. Its ok to assume no successor is executable.
     return;
   }
 
+  // In case of callbr, we pessimistically assume that all successors are
+  // feasible.
+  if (isa<CallBrInst>(&TI)) {
+    Succs.assign(TI.getNumSuccessors(), true);
+    return;
+  }
+
   LLVM_DEBUG(dbgs() << "Unknown terminator instruction: " << TI << '\n');
   llvm_unreachable("SCCP: Don't know how to handle this terminator!");
 }
 
 // isEdgeFeasible - Return true if the control flow edge from the 'From' basic
 // block to the 'To' basic block is currently feasible.
 bool SCCPSolver::isEdgeFeasible(BasicBlock *From, BasicBlock *To) {
   // Check if we've called markEdgeExecutable on the edge yet. (We could
for (Instruction &I : BB) {
 
         if ((Op0LV.isUnknown() || Op1LV.isUnknown()) &&
             cast<ICmpInst>(&I)->isEquality())
           break;
         markOverdefined(&I);
         return true;
       case Instruction::Call:
       case Instruction::Invoke:
+      case Instruction::CallBr:
         // There are two reasons a call can have an undef result
         // 1. It could be tracked.
         // 2. It could be constant-foldable.
         // Because of the way we solve return values, tracked calls must
         // never be marked overdefined in ResolvedUndefsIn.
         if (Function *F = CallSite(&I).getCalledFunction())
           if (TrackedRetVals.count(F))
             break;

lib/Transforms/Utils/BasicBlockUtils.cpp

BasicBlock *llvm::SplitBlockPredecessors(BasicBlock *BB,
 
   // Move the edges from Preds to point to NewBB instead of BB.
   for (unsigned i = 0, e = Preds.size(); i != e; ++i) {
     // This is slightly more strict than necessary; the minimum requirement
     // is that there be no more than one indirectbr branching to BB. And
     // all BlockAddress uses would need to be updated.
     assert(!isa<IndirectBrInst>(Preds[i]->getTerminator()) &&
            "Cannot split an edge from an IndirectBrInst");
+    assert(!isa<CallBrInst>(Preds[i]->getTerminator()) &&
+           "Cannot split an edge from a CallBrInst");
     Preds[i]->getTerminator()->replaceUsesOfWith(BB, NewBB);
   }
 
   // Insert a new PHI node into NewBB for every PHI node in BB and that new PHI
   // node becomes an incoming value for BB's phi node.  However, if the Preds
   // list is empty, we need to insert dummy entries into the PHI nodes in BB to
   // account for the newly created predecessor.
   if (Preds.empty()) {

lib/Transforms/Utils/BreakCriticalEdges.cpp

llvm::SplitCriticalEdge(Instruction *TI, unsigned SuccNum,
 
   BasicBlock *TIBB = TI->getParent();
   BasicBlock *DestBB = TI->getSuccessor(SuccNum);
 
   // Splitting the critical edge to a pad block is non-trivial. Don't do
   // it in this generic function.
   if (DestBB->isEHPad()) return nullptr;
 
+  // Don't split the non-fallthrough edge from a callbr.
+  if (isa<CallBrInst>(TI) && SuccNum > 0)
+    return nullptr;
+
   // Create a new basic block, linking it into the CFG.
   BasicBlock *NewBB = BasicBlock::Create(TI->getContext(),
                       TIBB->getName() + "." + DestBB->getName() + "_crit_edge");
   // Create our unconditional branch.
   BranchInst *NewBI = BranchInst::Create(DestBB, NewBB);
   NewBI->setDebugLoc(TI->getDebugLoc());
 
   // Branch to the new block, breaking the edge.

lib/Transforms/Utils/InlineFunction.cpp

 llvm::InlineResult llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
                                         AAResults *CalleeAAR,
                                         bool InsertLifetime,
                                         Function *ForwardVarArgsTo) {
   Instruction *TheCall = CS.getInstruction();
   assert(TheCall->getParent() && TheCall->getFunction()
          && "Instruction not in function!");
 
+  // FIXME: we don't inline callbr yet.
+  if (isa<CallBrInst>(TheCall))
+    return false;
+
   // If IFI has any state in it, zap it before we fill it in.
   IFI.reset();
 
   Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||               // Can't inline external function or indirect
       CalledFunc->isDeclaration()) // call!
     return "external or indirect";
 
if (ParentDeopt) {
                                  ChildOB.Inputs.end());
 
           OpDefs.emplace_back("deopt", std::move(MergedDeoptArgs));
         }
 
         Instruction *NewI = nullptr;
         if (isa<CallInst>(I))
           NewI = CallInst::Create(cast<CallInst>(I), OpDefs, I);
+        else if (isa<CallBrInst>(I))
+          NewI = CallBrInst::Create(cast<CallBrInst>(I), OpDefs, I);
         else
           NewI = InvokeInst::Create(cast<InvokeInst>(I), OpDefs, I);
 
         // Note: the RAUW does the appropriate fixup in VMap, so we need to do
         // this even if the call returns void.
         I->replaceAllUsesWith(NewI);
 
         VH = nullptr;
for (Function::iterator BB = FirstNewBlock->getIterator(),
           continue;
 
         CS.getOperandBundlesAsDefs(OpBundles);
         OpBundles.emplace_back("funclet", CallSiteEHPad);
 
         Instruction *NewInst;
         if (CS.isCall())
           NewInst = CallInst::Create(cast<CallInst>(I), OpBundles, I);
+        else if (CS.isCallBr())
+          NewInst = CallBrInst::Create(cast<CallBrInst>(I), OpBundles, I);
         else
           NewInst = InvokeInst::Create(cast<InvokeInst>(I), OpBundles, I);
         NewInst->takeName(I);
         I->replaceAllUsesWith(NewInst);
         I->eraseFromParent();
 
         OpBundles.clear();
       }

lib/Transforms/Utils/Local.cpp

while (isa<PHINode>(*BBI)) {
         } else {
           return false;
         }
       }
       ++BBI;
     }
   }
 
+  // We cannot fold the block if it's a branch to an already present callbr
+  // successor because that creates duplicate successors.
+  for (auto I = pred_begin(BB), E = pred_end(BB); I != E; ++I) {
+    if (auto *CBI = dyn_cast<CallBrInst>((*I)->getTerminator())) {
+      if (Succ == CBI->getFallthrough())
+        return false;
+      for (unsigned i = 0, e = CBI->getNumTransfers(); i != e; ++i)
+        if (Succ == CBI->getTransfer(i))
+          return false;
+    }
+  }
+
   LLVM_DEBUG(dbgs() << "Killing Trivial BB: \n" << *BB);
 
   SmallVector<DominatorTree::UpdateType, 32> Updates;
   if (DTU) {
     Updates.push_back({DominatorTree::Delete, BB, Succ});
     // All predecessors of BB will be moved to Succ.
     for (auto I = pred_begin(BB), E = pred_end(BB); I != E; ++I) {
       Updates.push_back({DominatorTree::Delete, *I, BB});

lib/Transforms/Utils/LoopSimplify.cpp

 //
 // This pass also guarantees that loops will have exactly one backedge.
 //
 // Indirectbr instructions introduce several complications. If the loop
 // contains or is entered by an indirectbr instruction, it may not be possible
 // to transform the loop and make these guarantees. Client code should check
 // that these conditions are true before relying on them.
 //
+// Similar complications arise from callbr instructions, particularly in
+// asm-goto where blockaddress expressions are used.
+//
 // Note that the simplifycfg pass will clean up blocks which are split out but
 // end up being unnecessary, so usage of this pass should not pessimize
 // generated code.
 //
 // This pass obviously modifies the CFG, but updates loop information and
 // dominator information.
 //
 //===----------------------------------------------------------------------===//
BasicBlock *llvm::InsertPreheaderForLoop(Loop *L, DominatorTree *DT,
   SmallVector<BasicBlock*, 8> OutsideBlocks;
   for (pred_iterator PI = pred_begin(Header), PE = pred_end(Header);
        PI != PE; ++PI) {
     BasicBlock *P = *PI;
     if (!L->contains(P)) {         // Coming in from outside the loop?
       // If the loop is branched to from an indirect branch, we won't
       // be able to fully transform the loop, because it prohibits
       // edge splitting.
-      if (isa<IndirectBrInst>(P->getTerminator())) return nullptr;
+      // Same for callbr.
+      if (isa<IndirectBrInst>(P->getTerminator()) ||
+          isa<CallBrInst>(P->getTerminator()))
+        return nullptr;
 
       // Keep track of it.
       OutsideBlocks.push_back(P);
     }
   }
 
   // Split out the loop pre-header.
   BasicBlock *PreheaderBB;
static Loop *separateNestedLoop(Loop *L, BasicBlock *Preheader,
 
   // Pull out all predecessors that have varying values in the loop.  This
   // handles the case when a PHI node has multiple instances of itself as
   // arguments.
   SmallVector<BasicBlock*, 8> OuterLoopPreds;
   for (unsigned i = 0, e = PN->getNumIncomingValues(); i != e; ++i) {
     if (PN->getIncomingValue(i) != PN ||
         !L->contains(PN->getIncomingBlock(i))) {
-      // We can't split indirectbr edges.
-      if (isa<IndirectBrInst>(PN->getIncomingBlock(i)->getTerminator()))
+      // We can't split indirectbr or callbr edges.
+      if (isa<IndirectBrInst>(PN->getIncomingBlock(i)->getTerminator()) ||
+          isa<CallBrInst>(PN->getIncomingBlock(i)->getTerminator()))
         return nullptr;
       OuterLoopPreds.push_back(PN->getIncomingBlock(i));
     }
   }
   LLVM_DEBUG(dbgs() << "LoopSimplify: Splitting out a new outer loop\n");
 
   // If ScalarEvolution is around and knows anything about values in
   // this loop, tell it to forget them, because we're about to
static BasicBlock *insertUniqueBackedgeBlock(Loop *L, BasicBlock *Preheader,
   assert(!Header->isEHPad() && "Can't insert backedge to EH pad");
 
   // Figure out which basic blocks contain back-edges to the loop header.
   std::vector<BasicBlock*> BackedgeBlocks;
   for (pred_iterator I = pred_begin(Header), E = pred_end(Header); I != E; ++I){
     BasicBlock *P = *I;
 
     // Indirectbr edges cannot be split, so we must fail if we find one.
-    if (isa<IndirectBrInst>(P->getTerminator()))
+    // Same for callbr.
+    if (isa<IndirectBrInst>(P->getTerminator()) ||
+        isa<CallBrInst>(P->getTerminator()))
       return nullptr;
 
     if (P != Preheader) BackedgeBlocks.push_back(P);
   }
 
   // Create and insert the new backedge block...
   BasicBlock *BEBlock = BasicBlock::Create(Header->getContext(),
                                            Header->getName() + ".backedge", F);

lib/Transforms/Utils/LoopUtils.cpp

auto RewriteExit = [&](BasicBlock *BB) {
     // See if there are any non-loop predecessors of this exit block and
     // keep track of the in-loop predecessors.
     bool IsDedicatedExit = true;
     for (auto *PredBB : predecessors(BB))
       if (L->contains(PredBB)) {
         if (isa<IndirectBrInst>(PredBB->getTerminator()))
           // We cannot rewrite exiting edges from an indirectbr.
           return false;
+        if (isa<CallBrInst>(PredBB->getTerminator()))
+          // We cannot rewrite exiting edges from a callbr.
+          return false;
 
         InLoopPredecessors.push_back(PredBB);
       } else {
         IsDedicatedExit = false;
       }
 
     assert(!InLoopPredecessors.empty() && "Must have *some* loop predecessor!");
 

lib/Transforms/Utils/SimplifyCFG.cpp

static bool HoistThenElseCodeToIf(BranchInst *BI,
   DbgInfoIntrinsic *DBI2 = dyn_cast<DbgInfoIntrinsic>(I2);
   if (!DBI1 || !DBI2 || !DBI1->isIdenticalToWhenDefined(DBI2)) {
     while (isa<DbgInfoIntrinsic>(I1))
       I1 = &*BB1_Itr++;
     while (isa<DbgInfoIntrinsic>(I2))
       I2 = &*BB2_Itr++;
   }
   if (isa<PHINode>(I1) || !I1->isIdenticalToWhenDefined(I2) ||
-      (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2)))
+      (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2)) ||
+      isa<CallBrInst>(I1))
     return false;
 
   BasicBlock *BIParent = BI->getParent();
 
   bool Changed = false;
   do {
     // If we are hoisting the terminator instruction, don't move one (making a
     // broken BB), instead clone it, and remove BI.
static bool HoistThenElseCodeToIf(BranchInst *BI,
 
   return true;
 
 HoistTerminator:
   // It may not be possible to hoist an invoke.
   if (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2))
     return Changed;
 
+  // TODO: callbr hoisting currently disabled pending further study.
+  if (isa<CallBrInst>(I1))
+    return Changed;
+
   for (BasicBlock *Succ : successors(BB1)) {
     for (PHINode &PN : Succ->phis()) {
       Value *BB1V = PN.getIncomingValueForBlock(BB1);
       Value *BB2V = PN.getIncomingValueForBlock(BB2);
       if (BB1V == BB2V)
         continue;
 
       // Check for passingValueIsAlwaysUndefined here because we would rather
for (auto *I : Insts) {
     // These instructions may change or break semantics if moved.
     if (isa<PHINode>(I) || I->isEHPad() || isa<AllocaInst>(I) ||
         I->getType()->isTokenTy())
       return false;
 
     // Conservatively return false if I is an inline-asm instruction. Sinking
     // and merging inline-asm instructions can potentially create arguments
     // that cannot satisfy the inline-asm constraints.
     if (const auto *C = dyn_cast<CallInst>(I))
       if (C->isInlineAsm())
         return false;
+    if (const auto *CB = dyn_cast<CallBrInst>(I))
+      if (CB->isInlineAsm())
+        return false;
 
     // Everything must have only one use too, apart from stores which
     // have no uses.
     if (!isa<StoreInst>(I) && !I->hasOneUse())
       return false;
   }
 
   const Instruction *I0 = Insts.front();
auto SameAsI0 = [&I0, OI](const Instruction *I) {
       assert(I->getNumOperands() == I0->getNumOperands());
       return I->getOperand(OI) == I0->getOperand(OI);
     };
     if (!all_of(Insts, SameAsI0)) {
       if (!canReplaceOperandWithVariable(I0, OI))
         // We can't create a PHI from this GEP.
         return false;
       // Don't create indirect calls! The called value is the final operand.
-      if ((isa<CallInst>(I0) || isa<InvokeInst>(I0)) && OI == OE - 1) {
+      if (isa<CallBase>(I0) && OI == OE - 1) {
         // FIXME: if the call was *already* indirect, we should do this.
         return false;
       }
       for (auto *I : Insts)
         PHIOperands[I].push_back(I->getOperand(OI));
     }
   }
   return true;

test/Bitcode/callbr.ll

+; RUN:  llvm-dis < %s.bc | FileCheck %s
+
+; callbr.ll.bc was generated by passing this file to llvm-as.
+
+define i32 @test_asm_goto(i32 %x){
+entry:
+; CHECK:      callbr void asm "", "r,X"(i32 %x, i8* blockaddress(@test_asm_goto, %fail))
+; CHECK-NEXT: to label %normal [label %fail]
+  callbr void asm "", "r,X"(i32 %x, i8* blockaddress(@test_asm_goto, %fail)) to label %normal [label %fail]
+normal:
+  ret i32 1
+fail:
+  ret i32 0
+}

test/CodeGen/X86/callbr-asm-destinations.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: Duplicate callbr destination
+
+; A test for asm-goto duplicate labels limitation
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail)) to label %fail [label %fail]
+
+fail:
+  ret i32 1
+}

test/CodeGen/X86/callbr-asm-errors.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: Duplicate callbr destination
+
+; A test for asm-goto duplicate labels limitation
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail), i8* blockaddress(@test, %fail)) to label %normal [label %fail, label %fail]
+
+normal:
+  ret i32 %0
+
+fail:
+  ret i32 1
+}

test/CodeGen/X86/callbr-asm-outputs.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: error: asm-goto outputs not supported
+
+; A test for asm-goto output prohibition
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  %1 = callbr i32 asm "xorl $1, $1; jmp ${1:l}", "=&r,r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 %1
+
+fail:
+  ret i32 1
+}

test/CodeGen/X86/callbr-asm.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=i686-- -O3 | FileCheck %s
+
+; Tests for using callbr as an asm-goto wrapper
+
+; Test 1 - fallthrough label gets removed, but the fallthrough code that is
+; unreachable due to asm ending on a jmp is still left in.
+define i32 @test1(i32 %a) {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    addl $4, %eax
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    jmp .Ltmp00
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.1: # %normal
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retl
+; CHECK-NEXT:  .Ltmp0: # Block address taken
+; CHECK-NEXT:  .LBB0_2: # %fail
+; CHECK-NEXT:    movl $1, %eax
+; CHECK-NEXT:    retl
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test1, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 0
+
+fail:
+  ret i32 1
+}
+
+; Test 2 - callbr terminates an unreachable block, function gets simplified
+; to a trivial zero return.
+define i32 @test2(i32 %a) {
+; CHECK-LABEL: test2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retl
+entry:
+  br label %normal
+
+unreachableasm:
+  %0 = add i32 %a, 4
+  callbr void asm sideeffect "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test2, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 0
+
+fail:
+  ret i32 1
+}
+
+
+; Test 3 - asm-goto implements a loop. The loop gets recognized, but many loop
+; transforms fail due to canonicalization having callbr exceptions. Trivial
+; blocks at labels 1 and 3 also don't get simplified due to callbr.
+define dso_local i32 @test3(i32 %a) {
+; CHECK-LABEL: test3:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:  .Ltmp1: # Block address taken
+; CHECK-NEXT:  .LBB2_1: # %label01
+; CHECK-NEXT:    # =>This Loop Header: Depth=1
+; CHECK-NEXT:    # Child Loop BB2_2 Depth 2
+; CHECK-NEXT:    # Child Loop BB2_3 Depth 3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:  .Ltmp2: # Block address taken
+; CHECK-NEXT:  .LBB2_2: # %label02
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # => This Loop Header: Depth=2
+; CHECK-NEXT:    # Child Loop BB2_3 Depth 3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:    addl $4, {{[0-9]+}}(%esp)
+; CHECK-NEXT:  .Ltmp3: # Block address taken
+; CHECK-NEXT:  .LBB2_3: # %label03
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # Parent Loop BB2_2 Depth=2
+; CHECK-NEXT:    # => This Loop Header: Depth=3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:    .p2align 4, 0x90
+; CHECK-NEXT:  .Ltmp4: # Block address taken
+; CHECK-NEXT:  .LBB2_4: # %label04
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # Parent Loop BB2_2 Depth=2
+; CHECK-NEXT:    # Parent Loop BB2_3 Depth=3
+; CHECK-NEXT:    # => This Inner Loop Header: Depth=4
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    jmp .Ltmp10
+; CHECK-NEXT:    jmp .Ltmp20
+; CHECK-NEXT:    jmp .Ltmp30
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.5: # %normal0
+; CHECK-NEXT:    # in Loop: Header=BB2_4 Depth=4
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    jmp .Ltmp10
+; CHECK-NEXT:    jmp .Ltmp20
+; CHECK-NEXT:    jmp .Ltmp30
+; CHECK-NEXT:    jmp .Ltmp40
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  # %bb.6: # %normal1
+; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    retl
+entry:
+  %a.addr = alloca i32, align 4
+  store i32 %a, i32* %a.addr, align 4
+  br label %label01
+
+label01:                                          ; preds = %normal0, %label04, %entry
+  br label %label02
+
+label02:                                          ; preds = %normal0, %label04, %label01
+  %0 = load i32, i32* %a.addr, align 4
+  %add = add nsw i32 %0, 4
+  store i32 %add, i32* %a.addr, align 4
+  br label %label03
+
+label03:                                          ; preds = %normal0, %label04, %label02
+  br label %label04
+
+label04:                                          ; preds = %normal0, %label03
+  callbr void asm sideeffect "jmp ${0:l}; jmp ${1:l}; jmp ${2:l}", "X,X,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test3, %label01), i8* blockaddress(@test3, %label02), i8* blockaddress(@test3, %label03))
+          to label %normal0 [label %label01, label %label02, label %label03]
+
+normal0:                                          ; preds = %label04
+  callbr void asm sideeffect "jmp ${0:l}; jmp ${1:l}; jmp ${2:l}; jmp ${3:l}", "X,X,X,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test3, %label01), i8* blockaddress(@test3, %label02), i8* blockaddress(@test3, %label03), i8* blockaddress(@test3, %label04))
+          to label %normal1 [label %label01, label %label02, label %label03, label %label04]
+
+normal1:                                          ; preds = %normal0
+  %1 = load i32, i32* %a.addr, align 4
+  ret i32 %1
+}

test/Transforms/MergeFunc/call-and-invoke-with-ranges.ll

 next:
   ret i8 %out
 
 lpad:
   %pad = landingpad { i8*, i32 } cleanup
   resume { i8*, i32 } zeroinitializer
 }
 
-define i8 @call_with_same_range() {
-; CHECK-LABEL: @call_with_same_range
-; CHECK: tail call i8 @call_with_range
-  bitcast i8 0 to i8
-  %out = call i8 @dummy(), !range !0
-  ret i8 %out
-}
-
 define i8 @invoke_with_same_range() personality i8* undef {
 ; CHECK-LABEL: @invoke_with_same_range()
 ; CHECK: tail call i8 @invoke_with_range()
   %out = invoke i8 @dummy() to label %next unwind label %lpad, !range !0
 
 next:
   ret i8 %out
 
 lpad:
   %pad = landingpad { i8*, i32 } cleanup
   resume { i8*, i32 } zeroinitializer
 }
 
+define i8 @call_with_same_range() {
+; CHECK-LABEL: @call_with_same_range
+; CHECK: tail call i8 @call_with_range
+  bitcast i8 0 to i8
+  %out = call i8 @dummy(), !range !0
+  ret i8 %out
+}
 
 
 declare i8 @dummy();
 declare i32 @__gxx_personality_v0(...)
 
 !0 = !{i8 0, i8 2}
 !1 = !{i8 5, i8 7}

test/Transforms/MergeFunc/inline-asm.ll

 ; RUN: opt -mergefunc -S < %s | FileCheck %s
 
 ; CHECK-LABEL: @int_ptr_arg_different
 ; CHECK-NEXT: call void asm
 
+; CHECK-LABEL: @int_ptr_null
+; CHECK-NEXT: tail call void @float_ptr_null()
+
 ; CHECK-LABEL: @int_ptr_arg_same
 ; CHECK-NEXT: %2 = bitcast i32* %0 to float*
 ; CHECK-NEXT: tail call void @float_ptr_arg_same(float* %2)
 
-; CHECK-LABEL: @int_ptr_null
-; CHECK-NEXT: tail call void @float_ptr_null()
-
 ; Used to satisfy minimum size limit
 declare void @stuff()
 
 ; Can be merged
 define void @float_ptr_null() {
   call void asm "nop", "r"(float* null)
   call void @stuff()
   ret void