[RFC prototype] Implementation of asm-goto support in LLVM

[craig.topper]

Summary

This patch accompanies the RFC posted here:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/127239.html

Once there is a consensus wrt the design of the feature, I will modify
this patch and split it into pieces for a proper review.

Even though this patch is definitely a prototype, it was tested quite
well, including the heavy-lifting of building the linux kernel
(kudos to Nick Desaulniers for doing that!).

One known problem in the implementation that I would like to mention
is that terminating callbr instruction is translated into INLINEASM
machine instruction, which is not a terminator. There are several crude
workarounds that we had to put in the code (e.g. see AsmPrinter.cpp) for
that. It has to be addressed in a better way and I would appreciate your
thoughts on that.

Authored By: Alexander Ivchenko, Mikhail Dvoretckii

[nickdesaulniers]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3001 ↗

(On Diff #181183)

Is this comment about terminator instruction still relevant?

3007 ↗

(On Diff #181183)

This comment is also added to lib/CodeGen/MachineBasicBlock.cpp. What does that mean for "asm-goto sometimes seems to be the second-to-last operand?" Second to last operand to what?

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2540–2541 ↗

(On Diff #181183)

These can probably be moved closer to their uses below?

lib/Transforms/Utils/LoopSimplify.cpp

135 ↗

(On Diff #181183)

From this part of the patch through below, there's multiple cases of:

if (indirectbr)
  foo();
if (callbr)
  foo();

That could be:

if (indirectbr || callbr)
  foo();

[craig.topper]

Attempt to fix a bug reported by @nickdesaulniers . We weren't using the label created for the blockaddress of the basic block when we printed the inline assembly block. So there was no guarantee the label we were using wouldn't be removed. To fix this I've kept the blockaddress object in the SelectionDAG instead of stripping it in the builder. I've created it directly as TargetBlockAddress to hide it from isel. Then in the printer I look for the operand being a blockaddress and grab the symbol for it. I had to make a few checks for TargetBlockAddress in other places to make this work. I admit I have no idea if this is the right thing to do at all, and open to any input on this.

[craig.topper]

lib/Bitcode/Reader/BitcodeReader.cpp

4277–4279 ↗

(On Diff #181183)

That would change the meaning of the else on the first if wouldn't it?

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3001 ↗

(On Diff #181183)

I think so. The CodeGen layer INLINEASM is not a terminator instruction.

3007 ↗

(On Diff #181183)

I think this meant "instruction" not "operand"

[craig.topper]

Address some review comments. Include context this time

[void]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3006–3007 ↗

(On Diff #181602)

This comment and the FIXME above it have me confused. Isn't the point of callbr to convert *all* asm goto statements into callbr? If so, then they should automatically be at the end of the basic block. What are the situations where it won't follow this?

[craig.topper]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3006–3007 ↗

(On Diff #181602)

This is MachineIR not IR. INLINEASM has its own MachineInstr opcode which is not a terminator. Currently asm-goto is still using that.

[void]

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

3006–3007 ↗

(On Diff #181602)

*smacks forehead* Of course. Nevermind.

[chandlerc]

Adding Hans so he is aware of the progress here.

[nickdesaulniers]

test/Bitcode/callbr.ll

1 ↗

(On Diff #181602)

This test case failed for me when running ninja check-all locally:

: 'RUN: at line 1';    llvm/build/bin/llvm-dis < llvm/test/Bitcode/callbr.ll.bc | llvm/build/bin/FileCheck llvm/test/Bitcode/callbr.ll
llvm/build/bin/llvm-dis: error: Invalid bitcode signature

[void]

I like these changes. Could you add the documentation for the new instruction?

[craig.topper]

Add some documentation to the LangRef. Probably needs additional work.

Try to upload the patch as binary so the bitcode file go through correctly.

Add MO_BlockAddress to X86AsmPrinter's printOperand to fix a crash on some invalid inline assembly reported by @jyu2.

[nickdesaulniers]

docs/LangRef.rst

6758 ↗

(On Diff #181888)

s/adress/address/

6764 ↗

(On Diff #181888)

s/calledd/called/

6769 ↗

(On Diff #181888)

s/callbrs/callbr/

[craig.topper]

docs/LangRef.rst

6758 ↗

(On Diff #181888)

That typo exists in Call and Invoke too. Fixed those in r351279

[craig.topper]

Fix LangRef typos

[void]

There appears to be an ICE with this patch: https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-454675962

[craig.topper]

In D53765#1360880, @void wrote:

There appears to be an ICE with this patch: https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-454675962

I think I see the issue. We need to return false for callbr from llvm::isSafeToSpeculativelyExecute. We probably also should fix to computeKnownBitsFromAssume to not visit an assume that is also the CxtI instruction. In that case we end up searching through the basic block until the terminator. And only because every terminator returns false from llvm::isSafeToSpeculativelyExecute do we end up terminating the loop without walking off the end of the basic block.

[craig.topper]

Add CallBr to isSafeToSpeculativelyExecute to keep isValidAssumeForContext from walking off the end of the basic block.

[nickdesaulniers]

A few additional errors we're hitting when compiling an x86_64 kernel with allyesconfig (all configurable options and drivers "enabled").
https://github.com/ClangBuiltLinux/linux/issues/6#issuecomment-455360304

Working to triage these and provide smaller test cases. Please do not land until we understand these build failures.

[craig.topper]

include/llvm/Bitcode/LLVMBitCodes.h

477 ↗

(On Diff #182342)

Can this use the 14 or does it need a new number? Not sure how holes get created here.

include/llvm/IR/Instructions.h

3864 ↗

(On Diff #182342)

I should probably drop this comment as I'm not sure what the previous author meant.

lib/Bitcode/Reader/BitcodeReader.cpp

4294 ↗

(On Diff #182342)

Should this be "(CCInfo >> bitc::CALL_CCONV) & CallingConv::MaxID" instead?

lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

441 ↗

(On Diff #182342)

I think this is an 80 column violation

lib/IR/Instruction.cpp

789 ↗

(On Diff #182342)

instructions is misspelled

lib/Transforms/InstCombine/InstCombineCalls.cpp

4259 ↗

(On Diff #182342)

Can this be merged using CallBase?

4282 ↗

(On Diff #182342)

Merge with above check for invoke

4403 ↗

(On Diff #182342)

Should we have an interface that doesn't require copying the transfer list? Just ask for a specific transfer number instead?

lib/Transforms/InstCombine/InstructionCombining.cpp

929 ↗

(On Diff #182342)

Should we merge these by doing isa<InvokeInst> || isa<CallBrInst> then just casting to Instruction to get the parent?

lib/Transforms/Utils/Local.cpp

1004 ↗

(On Diff #182342)

Merge the dyn_cast into the if

[nickdesaulniers]

This looks like a codegen bug, from compiling more of the Linux kernel. It seems that an optimization from -O1 is causing the problem: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. I'm not sure how to help narrow it down further (which pass?). The output is from creduce and I've narrowed down the set of flags to Clang.

lib/Transforms/Utils/InlineFunction.cpp

1512 ↗

(On Diff #182342)

While I can understand punting in v1, this does cause at least one -Wsection like warning from the kernel: https://github.com/ClangBuiltLinux/linux/issues/317. I call it section like, because the kernel has its own tool that it uses to validate sections; some run-once kernel code and data is put into init sections that get discarded after first use to help save kernel memory.

The issue is that this return false; makes __attribute__((always_inline)) fail. We can hack around that by converting always inline functions into macros, but I don't see that flying upstream.

So not the highest priority (it's just a warning, with no issues at runtime), but would be nice to have.

[craig.topper]

In D53765#1362744, @nickdesaulniers wrote:

This looks like a codegen bug, from compiling more of the Linux kernel. It seems that an optimization from -O1 is causing the problem: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. I'm not sure how to help narrow it down further (which pass?). The output is from creduce and I've narrowed down the set of flags to Clang.

You try using -opt-bisect-limit to narrow down the pass https://llvm.org/docs/OptBisect.html

[void]

I encountered another ICE here: https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455456506

[craig.topper]

Fix most of my own review comments. Fix what was probably a bug in SimplifyCFG.

I have not looked at the inlinining issue, the link error, or the ICE from @void yet.

[craig.topper]

Hopefully fix https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. Had to exclude CallBr predecessors in JumpThreading.

[nickdesaulniers]

Hopefully fix https://github.com/ClangBuiltLinux/linux/issues/320#issuecomment-455435791. Had to exclude CallBr predecessors in JumpThreading.

Yes looks good; I've closed that issue as unreproducible.

For https://github.com/ClangBuiltLinux/linux/issues/317, I think we're hitting https://bugs.llvm.org/show_bug.cgi?id=39560. Reproducer https://gist.github.com/nickdesaulniers/a69b44159f4d42a34fc1b5d18cb7b608. If we fix that, then I should be able to drop my out of tree kernel patch.

Finally, https://github.com/ClangBuiltLinux/linux/issues/319 looks troubling, but I haven't really dug into it. It *may* be due to my out of tree kernel patch (https://github.com/ClangBuiltLinux/linux/commit/0670525ded48caf958b6535d388e6478b6b14a4b), but if we can support inlining then it looks like my out of tree patch should not be necessary: https://godbolt.org/z/exFrxS. Otherwise, it may be another codegen bug due to this patch.

[majnemer]

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

[craig.topper]

In D53765#1367170, @majnemer wrote:

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

No, and Verifier.cpp currently errors for it.

[majnemer]

In D53765#1367315, @craig.topper wrote:

In D53765#1367170, @majnemer wrote:

Do you have any tests which perform a callbr with a normal function target rather than inline asm?

No, and Verifier.cpp currently errors for it.

I think it would be good to mention in this explicitly in the LangRef.

[craig.topper]

Make the LangRef statement about only supporting inline assembly a little stronger.

[craig.topper]

-Add a new isAsmGotoTarget flag to MachineBasicBlock. Use it to preserve predecessors/successors in a few places similar to isEHPad. Probably more work to do here, but I think it at least fixes https://github.com/ClangBuiltLinux/linux/issues/319
-Remove SelectionDAGBuilder support for doing anything with CallBr other than inline assembly. The code can't be tested today so there's really no point in having it. Assuming inline assembly only made setting the flag above a little simpler.

[nickdesaulniers]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2533 ↗

(On Diff #183623)

needs a (void)Callee; otherwise release builds produce:

../lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp:2531:16: warning: unused variable 'Callee' [-Wunused-variable]
  const Value *Callee(I.getCalledValue());
               ^

or can the whole expression just fit in the assert?

[craig.topper]

Fix unused variable in release builds

[craig.topper]

Hopefully make callbr.ll.bc binary again

[craig.topper]

Really make it binary

[nickdesaulniers]

Add a new isAsmGotoTarget flag to MachineBasicBlock

Is there a way to add a test for this case that is fixed by this, so that it never regresses without setting off alarms?

[craig.topper]

-Block tail merging in BranchFolding.cpp if the predecessor block contains an asm goto. This is what we do for exception handling.

[craig.topper]

-Simplify the code in isBlockOnlyReachableByFallthrough by using the isAsmGotoTarget flag.
-Support isAsmGotoTarget() in MachineBasicBlock::print

[nickdesaulniers]

I think llvm is failing to reach a fixed point now: https://github.com/ClangBuiltLinux/linux/issues/319#issuecomment-457874051

[craig.topper]

-Add another isAsmGotoTarget check to BranchFolding.cpp in place where we already checked isEHPad. Seems to prevent hang on fault.c reported here https://github.com/ClangBuiltLinux/linux/issues/330

[craig.topper]

-Disable critical edge splitting for the non-fallthrough edge of a callbr. Otherwise the blockaddress and the control flow edge get out of sync.

[craig.topper]

-Disable GVN PRE across callbr. This prevents it from queueing up a critical edge split that we can't perform causing an infinite loop. Might be able to do the the fallthrough edge, but left fixme for now.

[craig.topper]

-Block GVN Load PRE for CallBr critical edges.

[nickdesaulniers]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2533 ↗

(On Diff #183623)

Still seeing this in Diff 183921.

[craig.topper]

Restore a lost fix for a unused variable warning

[craig.topper]

-Teach shrink wrapping pass to ensure the asm goto targets are at least at the boundary of prologue/epilogue. This is what we do for EHPad, but may not be quite right for asm-goto.

[chandlerc]

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

I haven't looked as closely at the actual optimizer changes yet as I'd like, but I think those are by-and-large going to be pretty straightforward.

include/llvm/IR/CallSite.h

53 ↗

(On Diff #184024)

Bleh. We should just finish killing CallSite. But I guess this patch shouldn't wait on that.

89–91 ↗

(On Diff #184024)

FWIW, I.getInt() == 0 seems way more clear.

259–264 ↗

(On Diff #184024)

Can this be simplified by just delegating to CallBase? The amount of magic constants / implicit contracts here worries me.

321–324 ↗

(On Diff #184024)

Can we just sink this into CallBase?

600–608 ↗

(On Diff #184024)

Delegate to CallBase?

[chandlerc]

include/llvm/IR/InstVisitor.h

273–279 ↗

(On Diff #184024)

Need to update this code path.

include/llvm/IR/Instructions.h

4036–4038 ↗

(On Diff #184024)

See above, this I think should sink to CallBase.

lib/AsmParser/LLParser.cpp

6197–6198 ↗

(On Diff #184024)

bundles missing here?

6197–6198 ↗

(On Diff #184024)

The syntax here diverges from both switch and indirectbr in surprising ways here. They don't use any keywords like or jump to introduce thi list of labels, they just directly put it there.

While abstractly, I somewhat like this syntax, I wonder if it would be better te *exactly* match the syntax for switch which has the same default label with a list of other labels structure to it. Consistency seems good here unless it causes problems?

6250 ↗

(On Diff #184024)

Not a small vector?

6269 ↗

(On Diff #184024)

Weird indentatino.

6279 ↗

(On Diff #184024)

I'd use int and llvm::seq, but maybe better to remain consistent.

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2973–2974 ↗

(On Diff #184024)

Isn't one of the targets a fallthrough? Maybe i've not made it far enough, and the default target is different from the target here.

lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

439–443 ↗

(On Diff #184024)

I'm a bit surprised we don't model this as an MBB operand to begin with if it *has* to be a direct block address anyways? Curious why.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1246 ↗

(On Diff #181183)

Maybe have the verifier reject if we can't lower here? Regardless, the comment still seems to need some amount of fixing.

deopt bundles are a reasonable thing, and just not really a reasonable thing (at this stage) to support on callbr.

1300 ↗

(On Diff #184024)

This complex to thread through?

lib/CodeGen/IndirectBrExpandPass.cpp

151–153 ↗

(On Diff #184024)

I think we have to fix this before this goes very far. I don't actually know how this is working in practice in our biggest customer (the Linux Kernel) as it should be using things like retpolines and needing this code to work.

I also think that this entire pass, while a reasonable thing to have, was a bad idea for retpoline lowering. We found retpolines to be faster than the branch trees created by this in most cases w/ many branches. So this needs a non-trivial update. Not sure who has the time to do this at this point though, I'll ask around w/ folks familiar with this bit of code.

It would be good to at least update the comment to English prose.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536 ↗

(On Diff #184024)

Wait, we handle catchswitch combined with this? That seems .... ambitious.

2540–2547 ↗

(On Diff #184024)

Weird that we handle probabilities here but not in Global ISel

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Wait, they do?

I don't understand -- the callbr is always a terminator. I understand that it may expand to lots of instructions, but isn't that opaque at this level?

lib/Target/X86/X86AsmPrinter.cpp

256–260 ↗

(On Diff #184024)

The fact that we have to add an entire machine operand layer here makes me feel more strongly that we should just be using the existing support for MBB machine operands rather than this.... But maybe there is a reason I'm just missing so far.

[craig.topper]

In D53765#1375112, @chandlerc wrote:

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

Maybe I went the wrong way on this, but here's what I saw. The existence of the BlockAddress instruction in IR caused some portion of MachineIR handling to know that it needed to create a label of the form .Ltmp[0-9]+. This is helpfully annotated in the output assembly as "block address taken" and sometimes when there's a bug "address taken of block removed by CodeGen". The first version of this patch I inherited stripped the BlockAddress when the SelectionDAG was created. The resulting assembly would then print the .LBB label for the block in the inline asm instead. We ended up in a case where the LBB label didn't exist in the final output, but the Ltmp did. This failed to assemble obviously. Maybe that was really pointing to passes like BranchFolding.cpp doing bad things which we encountered later. But at the time I tried to fix it so that the inline asm printing would reference the .Ltmp label because that seemed like the label we really wanted.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

I haven't looked as closely at the actual optimizer changes yet as I'd like, but I think those are by-and-large going to be pretty straightforward.

Maybe it shouldn't be special to asm-goto. But its clear that we've already been treating exception handling pads specially, but not address-taken blocks so I've just been special casing asm goto to avoid changing anything about existing address taken. I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

lib/CodeGen/AsmPrinter/AsmPrinter.cpp

2973–2974 ↗

(On Diff #184024)

The only blocks that are marked as isAsmGotoTarget are the ones that aren't the default. See the code in SelectionDAGBuilder.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1300 ↗

(On Diff #184024)

Original author copied that comment out of visitInvokeInst

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536 ↗

(On Diff #184024)

That was copy and pasted from Invoke. I'll remove it

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

This is MachineIR. CallBr isn't a thing there. CallBr was expanded to an INLINEASM instruction followed by an unconditional jump during SelectionDAGBuilder. The INLINEASM block is not a terminator so its effectively "the middle of a basic block". This pass tried to shove a RBP pop between the INLINEASM instruction and the unconditional jump. In the case that exposed the bug, the unconditional jump went to a block that also jumped to the same inline asm target. So as far as the dominator tree was concerned, the block containing the INLINEASM dominated the other blocks so it was chosen to hold the RBP pop. But since the INLINEASM itself can jump that wasn't correct.

[efriedma]

I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

Did you consider making blocks which contain an asm goto unanalyzable? Intuitively, that's what I would expect given an asm goto is a terminator at the IR level. But I guess that makes handling the fallthrough case efficiently more difficult?

[craig.topper]

lib/AsmParser/LLParser.cpp

6250 ↗

(On Diff #184024)

Call and Invoke both use std::vector here...

[craig.topper]

Address some of Chandler's comments

[chandlerc]

Looking at updated patch and some of the optimizer bits, but some inline responses to discussion points below:

In D53765#1375547, @craig.topper wrote:

In D53765#1375112, @chandlerc wrote:

Sorry it took so long, but made a full pass through this. Just want to point out that this patch is already in really great shape due to the huge amount of work from the original author, Craig taking it over, and the multiple rounds of review from Bill and Nick among others.

Most of my comments are pretty superficial and not hard to address I suspect. I have one big question and one big comment. I've referenced these in-line below as I went through the code but wanted to call them out at a highlevel:

1. Why do we need the block address machine operand structure? I'm not saying to get rid of it (yet), I just don't understand the need to support both it and direct MBB operands.

Maybe I went the wrong way on this, but here's what I saw. The existence of the BlockAddress instruction in IR caused some portion of MachineIR handling to know that it needed to create a label of the form .Ltmp[0-9]+. This is helpfully annotated in the output assembly as "block address taken" and sometimes when there's a bug "address taken of block removed by CodeGen". The first version of this patch I inherited stripped the BlockAddress when the SelectionDAG was created. The resulting assembly would then print the .LBB label for the block in the inline asm instead. We ended up in a case where the LBB label didn't exist in the final output, but the Ltmp did. This failed to assemble obviously. Maybe that was really pointing to passes like BranchFolding.cpp doing bad things which we encountered later. But at the time I tried to fix it so that the inline asm printing would reference the .Ltmp label because that seemed like the label we really wanted.

I've spent more time thinking about this. I think the difference here is the following... With the "block address" behavior we assume the address more than taken -- it is *captured* and its uses may be unanalyzable. As a consequence, it *cannot* be updated when we do things like rework the control flow.

On the flip side, the MBB operands are likely often assumed to be easily enumerated and updated when we're rewriting the control flow, and so the fact that we can't do that results in breakages.

At least, this is my understanding. Match yours?

Given this, I think the current "block address" stuff may be correct -- I don't see anything that indicates the asm-goto couldn't escape or even capture the addresses of its successors and then re-use them in some way, expecting them to be "real" addresses. If this also makes sense to you, likely good to add some comments for future readers.

2. Why do we need to special case asm-goto targets in MI? I feel like this should somewhat fall out of the handling of address-taken basic blocks that can be jumped to somewhat opaquely. It would be really nice if there were a relatively common way of handling these rather than special casing the targets of asm-goto everywhere when it feels like the fundamental special treatment isn't actually about asm-goto at all. But maybe I've missed the critical thing that actually makes it special to asm-goto. If so, just help mme spot it1 =D

Maybe it shouldn't be special to asm-goto. But its clear that we've already been treating exception handling pads specially, but not address-taken blocks so I've just been special casing asm goto to avoid changing anything about existing address taken. I'm not sure what we do for indirect branches, but I suspect analyzeBranch returning unanalyzable for indirect jumps is part of it. But that doesn't work for asm goto because our terminator is a direct jump which is analyzable.

Ugh, yeah.

I think that we should consolidate all of this behind something like address-taken, but I think that can be follow-up work. I suspect that a decent number of the things you've fixed here are actually latent bugs w/ indirectbr and address-of-label extensions that we've just never managed to hit. =[[[[

But I think it's fine to fix that incrementally going forward by digging in and commoning these representations as much as possible.

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Ah, that explains the thing I was missing...

I had assumed that we would expand this to an INLINEASM instruction that *is* a terminator. In MIR we can have multiple terminators at the end of a block, and this is a common pattern, so could we not have both instructions be marked as terminators and avoid special casing this?

[chandlerc]

Made it through the optimizer code. Really minor changes here, I think this is looking good. Probable the biggest question marks are in the MI representation.

Also, how best to track on-going work that is going to be prioritized in the short term? Mostyl looking for what would be helpful for you all if the FIXME suggested below (or already in the code) are likely to be lost.

lib/Transforms/InstCombine/InstCombineCalls.cpp

4217–4219 ↗

(On Diff #184223)

Check for .isTerminator() would I think make this a bit more self-explanatory.

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925 ↗

(On Diff #184223)

May also be a tiny bit clearer by using terminator based terminology and APIs.

lib/Transforms/Scalar/GVN.cpp

2177–2184 ↗

(On Diff #184223)

Can just directly test on the CallBase now.

lib/Transforms/Utils/LoopSimplify.cpp

245–246 ↗

(On Diff #184223)

Ok, I've seen this pattern too many times. ;]

I'd suggest adding isIndirectTerminator or hasIndirectControlFlow or some such?

lib/Transforms/Utils/SimplifyCFG.cpp

1269–1270 ↗

(On Diff #184223)

FIXME as we can likely build an analogous safety predicate to the invoke one for CallBr.

1451–1456 ↗

(On Diff #184223)

Fold into a test on CallBase.

[craig.topper]

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

It looks like whether an instruction is a terminator is currently encoded in the MCInstrDesc entry corresponding to an opcode. So I think that means we would need a new flavor of INLINEASM opcode?

[chandlerc]

lib/CodeGen/ShrinkWrap.cpp

505–507 ↗

(On Diff #184024)

Bleh, yeah.

I think that's (marginally) cleaner, but I have no idea how much work it is to thread an INLINEASM_BR node through the backend. If it's too much, could defer this.

Reason that I think it is cleaner: I suspect we'll keep finding ways in which we mishandle this, but if it turns into a terminator sequence I think the rest of the backend will start to behave correctly.

[craig.topper]

-More review comments addressed.
-Rebase after committing some of the CallBase related changes

[craig.topper]

Update the CallBrInst creation APIs to consistently accept a callee-type argument. This makes CallBrInst consistent with call and invoke after r351121

[craig.topper]

Add a bailout for asm goto target to Machine Block Placement. Fixes export4.zip from here https://github.com/ClangBuiltLinux/linux/issues/336

[craig.topper]

Update terminology to replace Transfers with IndirectDests. And Fallthrough with DefaultDest

[craig.topper]

Rebase

[craig.topper]

Update CallBrInst creation interfaces to use FunctionCallee instead of Function* to match CallInst and InvokeInst

[chandlerc]

A few minor comments.

I assume the big outstanding item here is figuring out the MIR representation? Anything else?

docs/LangRef.rst

6910 ↗

(On Diff #185140)

Maybe "branch to" or "jump to"? I don't want to imply that it has to be done via call/return.

lib/CodeGen/GlobalISel/IRTranslator.cpp

1246 ↗

(On Diff #181183)

(still seems to need update)

1300 ↗

(On Diff #184024)

(comment remains)

lib/CodeGen/IndirectBrExpandPass.cpp

151–153 ↗

(On Diff #184024)

FWIW, I think I now understand -- I think the Linux kernel tester we have isn't building w/ retpolines enabled in the compiler, so we just haven't seen this explode. =[

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

2535–2536 ↗

(On Diff #184024)

(still needs fixing)

[craig.topper]

include/llvm/Bitcode/LLVMBitCodes.h

477 ↗

(On Diff #182342)

@chandlerc do you know the answer to the question I left here?

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925 ↗

(On Diff #184223)

I'm just going to remove this change because it would require a CallBrInst that returned a value, but we don't support that right now.

[craig.topper]

lib/Transforms/InstCombine/InstructionCombining.cpp

923–925 ↗

(On Diff #184223)

Oops I meant to remove this comment. I've now fixed the code to use terminator terminology and apis

[chandlerc]

include/llvm/Bitcode/LLVMBitCodes.h

477 ↗

(On Diff #182342)

Not directly... But I think I can explain how to answer it.

Does any version of LLVM bitcode that we still support reading use 14? If so, then we need to leave it alone. If not, I think you can take it.

But maybe good to double check w/ someone who has thought more fully about bitcode backwards compat.

[craig.topper]

lib/CodeGen/GlobalISel/IRTranslator.cpp

1246 ↗

(On Diff #181183)

Turns out this comment was copied directly from the InvokeInst handler above.

1300 ↗

(On Diff #184024)

Does global isel do anything with probabilities anywhere else? I can't see any code obviously doing it. SelectionDAGBuilder has some helper functions that we used for CallBr, but I don't see equivalent helpers here or any calls passing probability to addSuccessor.

[craig.topper]

-Change the bitcode encoding to a value that's never been used so I don't have to figure out the compatibility story.
-Add the bitcode value to the llvm-bcanalyzer
-Fix "returns" in LangRef
-Remove comment from SelectionDAGBuilder

[craig.topper]

-Update vim syntax highlight file to include 'callbr'
-Add test cases for the 2 GVN hangs related to critical edge splitting that I had fixed previously.

[kees]

I found a weird mis-compilation bug. Not sure if in LLVM or Clang half. Details here: https://reviews.llvm.org/D56571#1386973

[craig.topper]

-Fix crash in bitcode writer when writing out callbr with attributes
-Add test backend test cases and jump threading test case for previously fixed bugs

[craig.topper]

-Remove GlobalISel implementation completely. It wasn't even trying to be correct for asm-goto. It assumed a call to a function not InlineAsm. I think we can deal with this later.
-Introduce an INLINEASM_BR instruction in machine IR that is like INLINEASM but is also a terminator. Hopefully I got the right flags set on this.
-Remove isAsmGotoTarget/hasAsmGotoTargetSuccessor from MachineBasicBlock

From visual inspection of the existing tests this seems to be working, but we'll see what the kernel builds have to say

[kees]

In D53765#1386975, @kees wrote:

I found a weird mis-compilation bug. Not sure if in LLVM or Clang half. Details here: https://reviews.llvm.org/D56571#1386973

This appears to be fixed with latest llvm, clang, and D53765.185703.

[chandlerc]

This is looking really close. I think we should get some basic sanity checks w/ the kernel, and even if there is an obscure bug or two, land this and move to follow-up patches.

So, LGTM pending some basic sanity checks and addressing some nits below.

include/llvm/Analysis/SparsePropagation.h

333–334 ↗

(On Diff #185703)

Use TI.isIndirectTerminator() here?

lib/IR/Instruction.cpp

789 ↗

(On Diff #182342)

This should probably be isa<CallBase> now with a cleaner string, but not critical to fix in this patch.

lib/Transforms/InstCombine/InstCombineCalls.cpp

9 ↗

(On Diff #185703)

nit: I always prefer the Oxford comma to reduce ambiguity.

4346–4350 ↗

(On Diff #185703)

Wow this seems crazily expensive.

I guess it's OK because currently it can only occur for invoke. But the big-O here is quadratic: NumberOfUses * NumberOfSuccessors.

To make matters worse, this change moves to the generic successor APIs which will put a type check in the inner loop. =[

Maybe the simpler approach is to leave the old code for Invoke, and for CallBr simply return false if there are >0 users (which should be never currently).

[craig.topper]

Address the last round of comments from Chandler. I'll commit this shortly.

[nickdesaulniers]

With
clang asm goto diff 185489
llvm asm goto diff 185868
I see a few more warnings with the kernel's x86 object file validator (that has found bugs previously with this patch set):
https://github.com/ClangBuiltLinux/linux/issues/351
https://github.com/ClangBuiltLinux/linux/issues/336#issuecomment-461631864

I'll try to provide reproducers for these ASAP.

• EDIT **

Looks unrelated to this patch.

[nickdesaulniers]

llvm/trunk/lib/Transforms/InstCombine/InstCombineCalls.cpp

4353

[582/1886] Building CXX object lib/Transf...LVMInstCombine.dir/InstCombineCalls.cpp.o
../lib/Transforms/InstCombine/InstCombineCalls.cpp: In member function ‘bool llvm::InstCombiner::transformConstExprCastCall(llvm::CallBase&)’:
../lib/Transforms/InstCombine/InstCombineCalls.cpp:4353:23: warning: unused variable ‘CBI’ [-Wunused-variable]
       if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller))
                       ^~~

[nickdesaulniers]

This comment was removed by nickdesaulniers.

[nickdesaulniers]

llvm/trunk/lib/Transforms/InstCombine/InstCombineCalls.cpp

4353

Fixed in r353630.
https://github.com/llvm/llvm-project/commit/a97857b5b536179f79e9616235905cab8241eb68

llvm/trunk/docs/LangRef.rst

 block should be executed after the current block is finished. These
 terminator instructions typically yield a '``void``' value: they produce
 control flow, not values (the one exception being the
 ':ref:`invoke <i_invoke>`' instruction).
 
 The terminator instructions are: ':ref:`ret <i_ret>`',
 ':ref:`br <i_br>`', ':ref:`switch <i_switch>`',
 ':ref:`indirectbr <i_indirectbr>`', ':ref:`invoke <i_invoke>`',
+':ref:`callbr <i_callbr>`'
 ':ref:`resume <i_resume>`', ':ref:`catchswitch <i_catchswitch>`',
 ':ref:`catchret <i_catchret>`',
 ':ref:`cleanupret <i_cleanupret>`',
 and ':ref:`unreachable <i_unreachable>`'.
 
 .. _i_ret:
 
 '``ret``' Instruction

 
 .. code-block:: llvm
 
       %retval = invoke i32 @Test(i32 15) to label %Continue
                   unwind label %TestCleanup              ; i32:retval set
       %retval = invoke coldcc i32 %Testfnptr(i32 15) to label %Continue
                   unwind label %TestCleanup              ; i32:retval set
 
+.. _i_callbr:
+
+'``callbr``' Instruction
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      <result> = callbr [cconv] [ret attrs] [addrspace(<num>)] [<ty>|<fnty> <fnptrval>(<function args>) [fn attrs]
+                    [operand bundles] to label <normal label> or jump [other labels]
+
+Overview:
+"""""""""
+
+The '``callbr``' instruction causes control to transfer to a specified
+function, with the possibility of control flow transfer to either the
+'``normal``' label or one of the '``other``' labels.
+
+This instruction should only be used to implement the "goto" feature of gcc
+style inline assembly. Any other usage is an error in the IR verifier.
+
+Arguments:
+""""""""""
+
+This instruction requires several arguments:
+
+#. The optional "cconv" marker indicates which :ref:`calling
+   convention <callingconv>` the call should use. If none is
+   specified, the call defaults to using C calling conventions.
+#. The optional :ref:`Parameter Attributes <paramattrs>` list for return
+   values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
+   are valid here.
+#. The optional addrspace attribute can be used to indicate the address space
+   of the called function. If it is not specified, the program address space
+   from the :ref:`datalayout string<langref_datalayout>` will be used.
+#. '``ty``': the type of the call instruction itself which is also the
+   type of the return value. Functions that return no value are marked
+   ``void``.
+#. '``fnty``': shall be the signature of the function being called. The
+   argument types must match the types implied by this signature. This
+   type can be omitted if the function is not varargs.
+#. '``fnptrval``': An LLVM value containing a pointer to a function to
+   be called. In most cases, this is a direct function call, but
+   indirect ``callbr``'s are just as possible, calling an arbitrary pointer
+   to function value.
+#. '``function args``': argument list whose types match the function
+   signature argument types and parameter attributes. All arguments must
+   be of :ref:`first class <t_firstclass>` type. If the function signature
+   indicates the function accepts a variable number of arguments, the
+   extra arguments can be specified.
+#. '``normal label``': the label reached when the called function
+   executes a '``ret``' instruction.
+#. '``other labels``': the labels reached when a callee transfers control
+   to a location other than the normal '``normal label``'
+#. The optional :ref:`function attributes <fnattrs>` list.
+#. The optional :ref:`operand bundles <opbundles>` list.
+
+Semantics:
+""""""""""
+
+This instruction is designed to operate as a standard '``call``'
+instruction in most regards. The primary difference is that it
+establishes an association with additional labels to define where control
+flow goes after the call.
+
+The only use of this today is to implement the "goto" feature of gcc inline
+assembly where additional labels can be provided as locations for the inline
+assembly to jump to.
+
+Example:
+""""""""
+
+.. code-block:: llvm
+
+      callbr void asm "", "r,x"(i32 %x, i8 *blockaddress(@foo, %fail))
+                  to label %normal or jump [label %fail]
+
 .. _i_resume:
 
 '``resume``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 

llvm/trunk/include/llvm-c/Core.h

typedef enum {
   /* Terminator Instructions */
   LLVMRet            = 1,
   LLVMBr             = 2,
   LLVMSwitch         = 3,
   LLVMIndirectBr     = 4,
   LLVMInvoke         = 5,
   /* removed 6 due to API changes */
   LLVMUnreachable    = 7,
+  LLVMCallBr         = 67,
 
   /* Standard Unary Operators */
   LLVMFNeg           = 66,
 
   /* Standard Binary Operators */
   LLVMAdd            = 8,
   LLVMFAdd           = 9,
   LLVMSub            = 10,

llvm/trunk/include/llvm/Analysis/SparsePropagation.h

if (!C || !isa<ConstantInt>(C)) {
       return;
     }
 
     // Constant condition variables mean the branch can only go a single way
     Succs[C->isNullValue()] = true;
     return;
   }
 
-  if (TI.isExceptionalTerminator()) {
-    Succs.assign(Succs.size(), true);
-    return;
-  }
-
-  if (isa<IndirectBrInst>(TI)) {
+  if (TI.isExceptionalTerminator() ||
+      TI.isIndirectTerminator()) {
     Succs.assign(Succs.size(), true);
     return;
   }
 
   SwitchInst &SI = cast<SwitchInst>(TI);
   LatticeVal SCValue;
   if (AggressiveUndef)
     SCValue = getValueState(KeyInfo::getLatticeKeyFromValue(SI.getCondition()));

llvm/trunk/include/llvm/Bitcode/LLVMBitCodes.h

enum FunctionCodes {
   FUNC_CODE_INST_CATCHPAD = 50,    // CATCHPAD: [bb#,bb#,num,args...]
   FUNC_CODE_INST_CLEANUPPAD = 51,  // CLEANUPPAD: [num,args...]
   FUNC_CODE_INST_CATCHSWITCH =
       52, // CATCHSWITCH: [num,args...] or [num,args...,bb]
   // 53 is unused.
   // 54 is unused.
   FUNC_CODE_OPERAND_BUNDLE = 55, // OPERAND_BUNDLE: [tag#, value...]
   FUNC_CODE_INST_UNOP = 56,      // UNOP:       [opcode, ty, opval]
+  FUNC_CODE_INST_CALLBR = 57,    // CALLBR:     [attr, cc, norm, transfs,
+                                 //              fnty, fnid, args...]
 };
 
 enum UseListCodes {
   USELIST_CODE_DEFAULT = 1, // DEFAULT: [index..., value-id]
   USELIST_CODE_BB = 2       // BB: [index..., bb-id]
 };
 
 enum AttributeKindCodes {

llvm/trunk/include/llvm/CodeGen/GlobalISel/IRTranslator.h

bool valueIsSplit(const Value &V,
                     SmallVectorImpl<uint64_t> *Offsets = nullptr);
 
   /// Translate call instruction.
   /// \pre \p U is a call instruction.
   bool translateCall(const User &U, MachineIRBuilder &MIRBuilder);
 
   bool translateInvoke(const User &U, MachineIRBuilder &MIRBuilder);
 
+  bool translateCallBr(const User &U, MachineIRBuilder &MIRBuilder);
+
   bool translateLandingPad(const User &U, MachineIRBuilder &MIRBuilder);
 
   /// Translate one of LLVM's cast instructions into MachineInstrs, with the
   /// given generic Opcode.
   bool translateCast(unsigned Opcode, const User &U,
                      MachineIRBuilder &MIRBuilder);
 
   /// Translate a phi instruction.

llvm/trunk/include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     ///     ... however many operands ...
     ///   Operand #last: Optional, an incoming flag.
     ///
     /// The variable width operands are required to represent target addressing
     /// modes as a single "operand", even though they may have multiple
     /// SDOperands.
     INLINEASM,
 
+    /// INLINEASM_BR - Terminator version of inline asm. Used by asm-goto.
+    INLINEASM_BR,
+
     /// EH_LABEL - Represents a label in mid basic block used to track
     /// locations needed for debug and exception handling tables.  These nodes
     /// take a chain as input and return a chain.
     EH_LABEL,
 
     /// ANNOTATION_LABEL - Represents a mid basic block label used by
     /// annotations. This should remain within the basic block and be ordered
     /// with respect to other call instructions, but loads and stores may float

llvm/trunk/include/llvm/CodeGen/MachineInstr.h

public:
   }
 
   bool isPHI() const {
     return getOpcode() == TargetOpcode::PHI ||
            getOpcode() == TargetOpcode::G_PHI;
   }
   bool isKill() const { return getOpcode() == TargetOpcode::KILL; }
   bool isImplicitDef() const { return getOpcode()==TargetOpcode::IMPLICIT_DEF; }
-  bool isInlineAsm() const { return getOpcode() == TargetOpcode::INLINEASM; }
+  bool isInlineAsm() const {
+    return getOpcode() == TargetOpcode::INLINEASM ||
+           getOpcode() == TargetOpcode::INLINEASM_BR;
+  }
 
   bool isMSInlineAsm() const {
     return isInlineAsm() && getInlineAsmDialect() == InlineAsm::AD_Intel;
   }
 
   bool isStackAligningInlineAsm() const;
   InlineAsm::AsmDialect getInlineAsmDialect() const;
 

llvm/trunk/include/llvm/CodeGen/SelectionDAGISel.h

virtual bool ComplexPatternFuncMutatesDAG() const {
     return false;
   }
 
   bool isOrEquivalentToAdd(const SDNode *N) const;
 
 private:
 
   // Calls to these functions are generated by tblgen.
-  void Select_INLINEASM(SDNode *N);
+  void Select_INLINEASM(SDNode *N, bool Branch);
   void Select_READ_REGISTER(SDNode *Op);
   void Select_WRITE_REGISTER(SDNode *Op);
   void Select_UNDEF(SDNode *N);
   void CannotYetSelect(SDNode *N);
 
 private:
   void DoInstructionSelection();
   SDNode *MorphNode(SDNode *Node, unsigned TargetOpc, SDVTList VTList,

llvm/trunk/include/llvm/IR/CallSite.h

 //===- CallSite.h - Abstract Call & Invoke instrs ---------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the CallSite class, which is a handy wrapper for code that
-// wants to treat Call and Invoke instructions in a generic way. When in non-
-// mutation context (e.g. an analysis) ImmutableCallSite should be used.
+// wants to treat Call, Invoke and CallBr instructions in a generic way. When
+// in non-mutation context (e.g. an analysis) ImmutableCallSite should be used.
 // Finally, when some degree of customization is necessary between these two
 // extremes, CallSiteBase<> can be supplied with fine-tuned parameters.
 //
 // NOTE: These classes are supposed to have "value semantics". So they should be
 // passed by value, not by reference; they should not be "new"ed or "delete"d.
 // They are efficiently copyable, assignable and constructable, with cost
 // equivalent to copying a pointer (notice that they have only a single data
 // member). The internal representation carries a flag which indicates which of
-// the two variants is enclosed. This allows for cheaper checks when various
+// the three variants is enclosed. This allows for cheaper checks when various
 // accessors of CallSite are employed.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_IR_CALLSITE_H
 #define LLVM_IR_CALLSITE_H
 
 #include "llvm/ADT/Optional.h"

 #include <iterator>
 
 namespace llvm {
 
 namespace Intrinsic {
 enum ID : unsigned;
 }
 
-template <typename FunTy = const Function,
-          typename BBTy = const BasicBlock,
-          typename ValTy = const Value,
-          typename UserTy = const User,
-          typename UseTy = const Use,
-          typename InstrTy = const Instruction,
+template <typename FunTy = const Function, typename BBTy = const BasicBlock,
+          typename ValTy = const Value, typename UserTy = const User,
+          typename UseTy = const Use, typename InstrTy = const Instruction,
           typename CallTy = const CallInst,
           typename InvokeTy = const InvokeInst,
+          typename CallBrTy = const CallBrInst,
           typename IterTy = User::const_op_iterator>
 class CallSiteBase {
 protected:
-  PointerIntPair<InstrTy*, 1, bool> I;
+  PointerIntPair<InstrTy *, 2, int> I;
 
   CallSiteBase() = default;
-  CallSiteBase(CallTy *CI) : I(CI, true) { assert(CI); }
-  CallSiteBase(InvokeTy *II) : I(II, false) { assert(II); }
+  CallSiteBase(CallTy *CI) : I(CI, 1) { assert(CI); }
+  CallSiteBase(InvokeTy *II) : I(II, 0) { assert(II); }
+  CallSiteBase(CallBrTy *CBI) : I(CBI, 2) { assert(CBI); }
   explicit CallSiteBase(ValTy *II) { *this = get(II); }
 
 private:
   /// This static method is like a constructor. It will create an appropriate
-  /// call site for a Call or Invoke instruction, but it can also create a null
-  /// initialized CallSiteBase object for something which is NOT a call site.
+  /// call site for a Call, Invoke or CallBr instruction, but it can also create
+  /// a null initialized CallSiteBase object for something which is NOT a call
+  /// site.
   static CallSiteBase get(ValTy *V) {
     if (InstrTy *II = dyn_cast<InstrTy>(V)) {
       if (II->getOpcode() == Instruction::Call)
         return CallSiteBase(static_cast<CallTy*>(II));
-      else if (II->getOpcode() == Instruction::Invoke)
+      if (II->getOpcode() == Instruction::Invoke)
         return CallSiteBase(static_cast<InvokeTy*>(II));
+      if (II->getOpcode() == Instruction::CallBr)
+        return CallSiteBase(static_cast<CallBrTy *>(II));
     }
     return CallSiteBase();
   }
 
 public:
-  /// Return true if a CallInst is enclosed. Note that !isCall() does not mean
-  /// an InvokeInst is enclosed. It may also signify a NULL instruction pointer.
-  bool isCall() const { return I.getInt(); }
+  /// Return true if a CallInst is enclosed.
+  bool isCall() const { return I.getInt() == 1; }
 
-  /// Return true if a InvokeInst is enclosed.
-  bool isInvoke() const { return getInstruction() && !I.getInt(); }
+  /// Return true if a InvokeInst is enclosed. !I.getInt() may also signify a
+  /// NULL instruction pointer, so check that.
+  bool isInvoke() const { return getInstruction() && I.getInt() == 0; }
+
+  /// Return true if a CallBrInst is enclosed.
+  bool isCallBr() const { return I.getInt() == 2; }
 
   InstrTy *getInstruction() const { return I.getPointer(); }
   InstrTy *operator->() const { return I.getPointer(); }
   explicit operator bool() const { return I.getPointer(); }
 
   /// Get the basic block containing the call site.
   BBTy* getParent() const { return getInstruction()->getParent(); }
 
   /// Return the pointer to function that is being called.
   ValTy *getCalledValue() const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     return *getCallee();
   }
 
   /// Return the function being called if this is a direct call, otherwise
   /// return null (if it's an indirect call).
   FunTy *getCalledFunction() const {
     return dyn_cast<FunTy>(getCalledValue());
   }
 
   /// Return true if the callsite is an indirect call.
   bool isIndirectCall() const {
     const Value *V = getCalledValue();
     if (!V)
       return false;
     if (isa<FunTy>(V) || isa<Constant>(V))
       return false;
-    if (const CallInst *CI = dyn_cast<CallInst>(getInstruction())) {
-      if (CI->isInlineAsm())
+    if (const CallBase *CB = dyn_cast<CallBase>(getInstruction()))
+      if (CB->isInlineAsm())
         return false;
-    }
     return true;
   }
 
   /// Set the callee to the specified value.  Unlike the function of the same
   /// name on CallBase, does not modify the type!
   void setCalledFunction(Value *V) {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, callbr, or invoke instruction!");
     assert(cast<PointerType>(V->getType())->getElementType() ==
                cast<CallBase>(getInstruction())->getFunctionType() &&
            "New callee type does not match FunctionType on call");
     *getCallee() = V;
   }
 
   /// Return the intrinsic ID of the intrinsic called by this CallSite,
   /// or Intrinsic::not_intrinsic if the called function is not an
public:
   }
 
   ValTy *getArgument(unsigned ArgNo) const {
     assert(arg_begin() + ArgNo < arg_end() && "Argument # out of range!");
     return *(arg_begin() + ArgNo);
   }
 
   void setArgument(unsigned ArgNo, Value* newVal) {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(arg_begin() + ArgNo < arg_end() && "Argument # out of range!");
     getInstruction()->setOperand(ArgNo, newVal);
   }
 
   /// Given a value use iterator, returns the argument that corresponds to it.
   /// Iterator must actually correspond to an argument.
   unsigned getArgumentNo(Value::const_user_iterator I) const {
     return getArgumentNo(&I.getUse());
   }
 
   /// Given a use for an argument, get the argument number that corresponds to
   /// it.
   unsigned getArgumentNo(const Use *U) const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(isArgOperand(U) && "Argument # out of range!");
     return U - arg_begin();
   }
 
   /// The type of iterator to use when looping over actual arguments at this
   /// call site.
   using arg_iterator = IterTy;
 
   iterator_range<IterTy> args() const {
     return make_range(arg_begin(), arg_end());
   }
   bool arg_empty() const { return arg_end() == arg_begin(); }
   unsigned arg_size() const { return unsigned(arg_end() - arg_begin()); }
 
   /// Given a value use iterator, return the data operand corresponding to it.
   /// Iterator must actually correspond to a data operand.
   unsigned getDataOperandNo(Value::const_user_iterator UI) const {
     return getDataOperandNo(&UI.getUse());
   }
 
   /// Given a use for a data operand, get the data operand number that
   /// corresponds to it.
   unsigned getDataOperandNo(const Use *U) const {
-    assert(getInstruction() && "Not a call or invoke instruction!");
+    assert(getInstruction() && "Not a call, invoke or callbr instruction!");
     assert(isDataOperand(U) && "Data operand # out of range!");
     return U - data_operands_begin();
   }
 
   /// Type of iterator to use when looping over data operands at this call site
   /// (see below).
   using data_operand_iterator = IterTy;
 
   /// data_operands_begin/data_operands_end - Return iterators iterating over
-  /// the call / invoke argument list and bundle operands.  For invokes, this is
-  /// the set of instruction operands except the invoke target and the two
-  /// successor blocks; and for calls this is the set of instruction operands
-  /// except the call target.
+  /// the call / invoke / callbr argument list and bundle operands. For invokes,
+  /// this is the set of instruction operands except the invoke target and the
+  /// two successor blocks; for calls this is the set of instruction operands
+  /// except the call target; for callbrs the number of labels to skip must be
+  /// determined first.
 
   IterTy data_operands_begin() const {
     assert(getInstruction() && "Not a call or invoke instruction!");
     return cast<CallBase>(getInstruction())->data_operands_begin();
   }
   IterTy data_operands_end() const {
     assert(getInstruction() && "Not a call or invoke instruction!");
     return cast<CallBase>(getInstruction())->data_operands_end();
bool isMustTailCall() const {
     return isCall() && cast<CallInst>(getInstruction())->isMustTailCall();
   }
 
   /// Tests if this call site is marked as a tail call.
   bool isTailCall() const {
     return isCall() && cast<CallInst>(getInstruction())->isTailCall();
   }
 
-#define CALLSITE_DELEGATE_GETTER(METHOD) \
-  InstrTy *II = getInstruction();    \
-  return isCall()                        \
-    ? cast<CallInst>(II)->METHOD         \
+#define CALLSITE_DELEGATE_GETTER(METHOD)                                       \
+  InstrTy *II = getInstruction();                                              \
+  return isCall() ? cast<CallInst>(II)->METHOD                                 \
+                  : isCallBr() ? cast<CallBrInst>(II)->METHOD                  \
                                 : cast<InvokeInst>(II)->METHOD
 
-#define CALLSITE_DELEGATE_SETTER(METHOD) \
-  InstrTy *II = getInstruction();    \
-  if (isCall())                          \
-    cast<CallInst>(II)->METHOD;          \
-  else                                   \
+#define CALLSITE_DELEGATE_SETTER(METHOD)                                       \
+  InstrTy *II = getInstruction();                                              \
+  if (isCall())                                                                \
+    cast<CallInst>(II)->METHOD;                                                \
+  else if (isCallBr())                                                         \
+    cast<CallBrInst>(II)->METHOD;                                              \
+  else                                                                         \
     cast<InvokeInst>(II)->METHOD
 
   unsigned getNumArgOperands() const {
     CALLSITE_DELEGATE_GETTER(getNumArgOperands());
   }
 
   ValTy *getArgOperand(unsigned i) const {
     CALLSITE_DELEGATE_GETTER(getArgOperand(i));
   }
 
   ValTy *getReturnedArgOperand() const {
     CALLSITE_DELEGATE_GETTER(getReturnedArgOperand());
   }
 
   bool isInlineAsm() const {
-    if (isCall())
-      return cast<CallInst>(getInstruction())->isInlineAsm();
-    return false;
+    return cast<CallBase>(getInstruction())->isInlineAsm();
   }
 
   /// Get the calling convention of the call.
   CallingConv::ID getCallingConv() const {
     CALLSITE_DELEGATE_GETTER(getCallingConv());
   }
   /// Set the calling convention of the call.
   void setCallingConv(CallingConv::ID CC) {
#define CALLSITE_DELEGATE_SETTER(METHOD)                                       \
 
   Attribute getAttribute(unsigned i, StringRef Kind) const {
     CALLSITE_DELEGATE_GETTER(getAttribute(i, Kind));
   }
 
   /// Return true if the data operand at index \p i directly or indirectly has
   /// the attribute \p A.
   ///
-  /// Normal call or invoke arguments have per operand attributes, as specified
-  /// in the attribute set attached to this instruction, while operand bundle
-  /// operands may have some attributes implied by the type of its containing
-  /// operand bundle.
+  /// Normal call, invoke or callbr arguments have per operand attributes, as
+  /// specified in the attribute set attached to this instruction, while operand
+  /// bundle operands may have some attributes implied by the type of its
+  /// containing operand bundle.
   bool dataOperandHasImpliedAttr(unsigned i, Attribute::AttrKind Kind) const {
     CALLSITE_DELEGATE_GETTER(dataOperandHasImpliedAttr(i, Kind));
   }
 
   /// Extract the alignment of the return value.
   unsigned getRetAlignment() const {
     CALLSITE_DELEGATE_GETTER(getRetAlignment());
   }

 private:
   IterTy getCallee() const {
     return cast<CallBase>(getInstruction())->op_end() - 1;
   }
 };
 
 class CallSite : public CallSiteBase<Function, BasicBlock, Value, User, Use,
                                      Instruction, CallInst, InvokeInst,
-                                     User::op_iterator> {
+                                     CallBrInst, User::op_iterator> {
 public:
   CallSite() = default;
   CallSite(CallSiteBase B) : CallSiteBase(B) {}
   CallSite(CallInst *CI) : CallSiteBase(CI) {}
   CallSite(InvokeInst *II) : CallSiteBase(II) {}
+  CallSite(CallBrInst *CBI) : CallSiteBase(CBI) {}
   explicit CallSite(Instruction *II) : CallSiteBase(II) {}
   explicit CallSite(Value *V) : CallSiteBase(V) {}
 
   bool operator==(const CallSite &CS) const { return I == CS.I; }
   bool operator!=(const CallSite &CS) const { return I != CS.I; }
   bool operator<(const CallSite &CS) const {
     return getInstruction() < CS.getInstruction();
   }

 };
 
 /// Establish a view to a call site for examination.
 class ImmutableCallSite : public CallSiteBase<> {
 public:
   ImmutableCallSite() = default;
   ImmutableCallSite(const CallInst *CI) : CallSiteBase(CI) {}
   ImmutableCallSite(const InvokeInst *II) : CallSiteBase(II) {}
+  ImmutableCallSite(const CallBrInst *CBI) : CallSiteBase(CBI) {}
   explicit ImmutableCallSite(const Instruction *II) : CallSiteBase(II) {}
   explicit ImmutableCallSite(const Value *V) : CallSiteBase(V) {}
   ImmutableCallSite(CallSite CS) : CallSiteBase(CS.getInstruction()) {}
 };
 
 } // end namespace llvm
 
 #endif // LLVM_IR_CALLSITE_H

llvm/trunk/include/llvm/IR/IRBuilder.h

InvokeInst *CreateInvoke(Value *Callee, BasicBlock *NormalDest,
                            ArrayRef<Value *> Args = None,
                            const Twine &Name = "") {
     return CreateInvoke(
         cast<FunctionType>(
             cast<PointerType>(Callee->getType())->getElementType()),
         Callee, NormalDest, UnwindDest, Args, Name);
   }
 
+  /// \brief Create a callbr instruction.
+  CallBrInst *CreateCallBr(FunctionType *Ty, Value *Callee,
+                           BasicBlock *DefaultDest,
+                           ArrayRef<BasicBlock *> IndirectDests,
+                           ArrayRef<Value *> Args = None,
+                           const Twine &Name = "") {
+    return Insert(CallBrInst::Create(Ty, Callee, DefaultDest, IndirectDests,
+                                     Args), Name);
+  }
+  CallBrInst *CreateCallBr(FunctionType *Ty, Value *Callee,
+                           BasicBlock *DefaultDest,
+                           ArrayRef<BasicBlock *> IndirectDests,
+                           ArrayRef<Value *> Args,
+                           ArrayRef<OperandBundleDef> OpBundles,
+                           const Twine &Name = "") {
+    return Insert(
+        CallBrInst::Create(Ty, Callee, DefaultDest, IndirectDests, Args,
+                           OpBundles), Name);
+  }
+
+  CallBrInst *CreateCallBr(FunctionCallee Callee, BasicBlock *DefaultDest,
+                           ArrayRef<BasicBlock *> IndirectDests,
+                           ArrayRef<Value *> Args = None,
+                           const Twine &Name = "") {
+    return CreateCallBr(Callee.getFunctionType(), Callee.getCallee(),
+                        DefaultDest, IndirectDests, Args, Name);
+  }
+  CallBrInst *CreateCallBr(FunctionCallee Callee, BasicBlock *DefaultDest,
+                           ArrayRef<BasicBlock *> IndirectDests,
+                           ArrayRef<Value *> Args,
+                           ArrayRef<OperandBundleDef> OpBundles,
+                           const Twine &Name = "") {
+    return CreateCallBr(Callee.getFunctionType(), Callee.getCallee(),
+                        DefaultDest, IndirectDests, Args, Name);
+  }
+
   ResumeInst *CreateResume(Value *Exn) {
     return Insert(ResumeInst::Create(Exn));
   }
 
   CleanupReturnInst *CreateCleanupRet(CleanupPadInst *CleanupPad,
                                       BasicBlock *UnwindBB = nullptr) {
     return Insert(CleanupReturnInst::Create(CleanupPad, UnwindBB));
   }

llvm/trunk/include/llvm/IR/InstVisitor.h

#include "llvm/IR/Instruction.def"
   RetTy visitMemMoveInst(MemMoveInst &I)          { DELEGATE(MemTransferInst); }
   RetTy visitMemTransferInst(MemTransferInst &I)  { DELEGATE(MemIntrinsic); }
   RetTy visitMemIntrinsic(MemIntrinsic &I)        { DELEGATE(IntrinsicInst); }
   RetTy visitVAStartInst(VAStartInst &I)          { DELEGATE(IntrinsicInst); }
   RetTy visitVAEndInst(VAEndInst &I)              { DELEGATE(IntrinsicInst); }
   RetTy visitVACopyInst(VACopyInst &I)            { DELEGATE(IntrinsicInst); }
   RetTy visitIntrinsicInst(IntrinsicInst &I)      { DELEGATE(CallInst); }
 
-  // Call and Invoke are slightly different as they delegate first through
-  // a generic CallSite visitor.
+  // Call, Invoke and CallBr are slightly different as they delegate first
+  // through a generic CallSite visitor.
   RetTy visitCallInst(CallInst &I) {
     return static_cast<SubClass*>(this)->visitCallSite(&I);
   }
   RetTy visitInvokeInst(InvokeInst &I) {
     return static_cast<SubClass*>(this)->visitCallSite(&I);
   }
+  RetTy visitCallBrInst(CallBrInst &I) {
+    return static_cast<SubClass *>(this)->visitCallSite(&I);
+  }
 
   // While terminators don't have a distinct type modeling them, we support
   // intercepting them with dedicated a visitor callback.
   RetTy visitReturnInst(ReturnInst &I) {
     return static_cast<SubClass *>(this)->visitTerminator(I);
   }
   RetTy visitBranchInst(BranchInst &I) {
     return static_cast<SubClass *>(this)->visitTerminator(I);
#include "llvm/IR/Instruction.def"
   RetTy visitUnaryOperator(UnaryOperator &I)      { DELEGATE(UnaryInstruction);}
   RetTy visitBinaryOperator(BinaryOperator &I)    { DELEGATE(Instruction);}
   RetTy visitCmpInst(CmpInst &I)                  { DELEGATE(Instruction);}
   RetTy visitUnaryInstruction(UnaryInstruction &I){ DELEGATE(Instruction);}
 
   // The next level delegation for `CallBase` is slightly more complex in order
   // to support visiting cases where the call is also a terminator.
   RetTy visitCallBase(CallBase &I) {
-    if (isa<InvokeInst>(I))
+    if (isa<InvokeInst>(I) || isa<CallBrInst>(I))
       return static_cast<SubClass *>(this)->visitTerminator(I);
 
     DELEGATE(Instruction);
   }
 
-  // Provide a legacy visitor for a 'callsite' that visits both calls and
-  // invokes.
+  // Provide a legacy visitor for a 'callsite' that visits calls, invokes,
+  // and calbrs.
   //
   // Prefer overriding the type system based `CallBase` instead.
   RetTy visitCallSite(CallSite CS) {
     assert(CS);
     Instruction &I = *CS.getInstruction();
     DELEGATE(CallBase);
   }
 

llvm/trunk/include/llvm/IR/InstrTypes.h

protected:
   bool hasDescriptor() const { return Value::HasDescriptor; }
 
   unsigned getNumSubclassExtraOperands() const {
     switch (getOpcode()) {
     case Instruction::Call:
       return 0;
     case Instruction::Invoke:
       return 2;
+    case Instruction::CallBr:
+      return getNumSubclassExtraOperandsDynamic();
     }
     llvm_unreachable("Invalid opcode!");
   }
 
+  /// Get the number of extra operands for instructions that don't have a fixed
+  /// number of extra operands.
+  unsigned getNumSubclassExtraOperandsDynamic() const;
+
 public:
   using Instruction::getContext;
 
   static bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::Call ||
-           I->getOpcode() == Instruction::Invoke;
+           I->getOpcode() == Instruction::Invoke ||
+           I->getOpcode() == Instruction::CallBr;
   }
   static bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 
   FunctionType *getFunctionType() const { return FTy; }
 
   void mutateFunctionType(FunctionType *FTy) {

llvm/trunk/include/llvm/IR/Instruction.h

public:
   bool isBinaryOp() const { return isBinaryOp(getOpcode()); }
   bool isIntDivRem() const { return isIntDivRem(getOpcode()); }
   bool isShift() { return isShift(getOpcode()); }
   bool isCast() const { return isCast(getOpcode()); }
   bool isFuncletPad() const { return isFuncletPad(getOpcode()); }
   bool isExceptionalTerminator() const {
     return isExceptionalTerminator(getOpcode());
   }
+  bool isIndirectTerminator() const {
+    return isIndirectTerminator(getOpcode());
+  }
 
   static const char* getOpcodeName(unsigned OpCode);
 
   static inline bool isTerminator(unsigned OpCode) {
     return OpCode >= TermOpsBegin && OpCode < TermOpsEnd;
   }
 
   static inline bool isUnaryOp(unsigned Opcode) {
static inline bool isExceptionalTerminator(unsigned OpCode) {
     case Instruction::Invoke:
     case Instruction::Resume:
       return true;
     default:
       return false;
     }
   }
 
+  /// Returns true if the OpCode is a terminator with indirect targets.
+  static inline bool isIndirectTerminator(unsigned OpCode) {
+    switch (OpCode) {
+    case Instruction::IndirectBr:
+    case Instruction::CallBr:
+      return true;
+    default:
+      return false;
+    }
+  }
+
   //===--------------------------------------------------------------------===//
   // Metadata manipulation.
   //===--------------------------------------------------------------------===//
 
   /// Return true if this instruction has any metadata attached to it.
   bool hasMetadata() const { return DbgLoc || hasMetadataHashEntry(); }
 
   /// Return true if this instruction has metadata attached to it other than a

llvm/trunk/include/llvm/IR/Instruction.def

 HANDLE_TERM_INST  ( 3, Switch        , SwitchInst)
 HANDLE_TERM_INST  ( 4, IndirectBr    , IndirectBrInst)
 HANDLE_TERM_INST  ( 5, Invoke        , InvokeInst)
 HANDLE_TERM_INST  ( 6, Resume        , ResumeInst)
 HANDLE_TERM_INST  ( 7, Unreachable   , UnreachableInst)
 HANDLE_TERM_INST  ( 8, CleanupRet    , CleanupReturnInst)
 HANDLE_TERM_INST  ( 9, CatchRet      , CatchReturnInst)
 HANDLE_TERM_INST  (10, CatchSwitch   , CatchSwitchInst)
-  LAST_TERM_INST  (10)
+HANDLE_TERM_INST  (11, CallBr        , CallBrInst) // A call-site terminator
+  LAST_TERM_INST  (11)
 
 // Standard unary operators...
- FIRST_UNARY_INST(11)
-HANDLE_UNARY_INST(11, FNeg  , UnaryOperator)
-  LAST_UNARY_INST(11)
+ FIRST_UNARY_INST(12)
+HANDLE_UNARY_INST(12, FNeg  , UnaryOperator)
+  LAST_UNARY_INST(12)
 
 // Standard binary operators...
- FIRST_BINARY_INST(12)
-HANDLE_BINARY_INST(12, Add  , BinaryOperator)
-HANDLE_BINARY_INST(13, FAdd , BinaryOperator)
-HANDLE_BINARY_INST(14, Sub  , BinaryOperator)
-HANDLE_BINARY_INST(15, FSub , BinaryOperator)
-HANDLE_BINARY_INST(16, Mul  , BinaryOperator)
-HANDLE_BINARY_INST(17, FMul , BinaryOperator)
-HANDLE_BINARY_INST(18, UDiv , BinaryOperator)
-HANDLE_BINARY_INST(19, SDiv , BinaryOperator)
-HANDLE_BINARY_INST(20, FDiv , BinaryOperator)
-HANDLE_BINARY_INST(21, URem , BinaryOperator)
-HANDLE_BINARY_INST(22, SRem , BinaryOperator)
-HANDLE_BINARY_INST(23, FRem , BinaryOperator)
+ FIRST_BINARY_INST(13)
+HANDLE_BINARY_INST(13, Add  , BinaryOperator)
+HANDLE_BINARY_INST(14, FAdd , BinaryOperator)
+HANDLE_BINARY_INST(15, Sub  , BinaryOperator)
+HANDLE_BINARY_INST(16, FSub , BinaryOperator)
+HANDLE_BINARY_INST(17, Mul  , BinaryOperator)
+HANDLE_BINARY_INST(18, FMul , BinaryOperator)
+HANDLE_BINARY_INST(19, UDiv , BinaryOperator)
+HANDLE_BINARY_INST(20, SDiv , BinaryOperator)
+HANDLE_BINARY_INST(21, FDiv , BinaryOperator)
+HANDLE_BINARY_INST(22, URem , BinaryOperator)
+HANDLE_BINARY_INST(23, SRem , BinaryOperator)
+HANDLE_BINARY_INST(24, FRem , BinaryOperator)
 
 // Logical operators (integer operands)
-HANDLE_BINARY_INST(24, Shl  , BinaryOperator) // Shift left  (logical)
-HANDLE_BINARY_INST(25, LShr , BinaryOperator) // Shift right (logical)
-HANDLE_BINARY_INST(26, AShr , BinaryOperator) // Shift right (arithmetic)
-HANDLE_BINARY_INST(27, And  , BinaryOperator)
-HANDLE_BINARY_INST(28, Or   , BinaryOperator)
-HANDLE_BINARY_INST(29, Xor  , BinaryOperator)
-  LAST_BINARY_INST(29)
+HANDLE_BINARY_INST(25, Shl  , BinaryOperator) // Shift left  (logical)
+HANDLE_BINARY_INST(26, LShr , BinaryOperator) // Shift right (logical)
+HANDLE_BINARY_INST(27, AShr , BinaryOperator) // Shift right (arithmetic)
+HANDLE_BINARY_INST(28, And  , BinaryOperator)
+HANDLE_BINARY_INST(29, Or   , BinaryOperator)
+HANDLE_BINARY_INST(30, Xor  , BinaryOperator)
+  LAST_BINARY_INST(30)
 
 // Memory operators...
- FIRST_MEMORY_INST(30)
-HANDLE_MEMORY_INST(30, Alloca, AllocaInst)  // Stack management
-HANDLE_MEMORY_INST(31, Load  , LoadInst  )  // Memory manipulation instrs
-HANDLE_MEMORY_INST(32, Store , StoreInst )
-HANDLE_MEMORY_INST(33, GetElementPtr, GetElementPtrInst)
-HANDLE_MEMORY_INST(34, Fence , FenceInst )
-HANDLE_MEMORY_INST(35, AtomicCmpXchg , AtomicCmpXchgInst )
-HANDLE_MEMORY_INST(36, AtomicRMW , AtomicRMWInst )
-  LAST_MEMORY_INST(36)
+ FIRST_MEMORY_INST(31)
+HANDLE_MEMORY_INST(31, Alloca, AllocaInst)  // Stack management
+HANDLE_MEMORY_INST(32, Load  , LoadInst  )  // Memory manipulation instrs
+HANDLE_MEMORY_INST(33, Store , StoreInst )
+HANDLE_MEMORY_INST(34, GetElementPtr, GetElementPtrInst)
+HANDLE_MEMORY_INST(35, Fence , FenceInst )
+HANDLE_MEMORY_INST(36, AtomicCmpXchg , AtomicCmpXchgInst )
+HANDLE_MEMORY_INST(37, AtomicRMW , AtomicRMWInst )
+  LAST_MEMORY_INST(37)
 
 // Cast operators ...
 // NOTE: The order matters here because CastInst::isEliminableCastPair
 // NOTE: (see Instructions.cpp) encodes a table based on this ordering.
- FIRST_CAST_INST(37)
-HANDLE_CAST_INST(37, Trunc   , TruncInst   )  // Truncate integers
-HANDLE_CAST_INST(38, ZExt    , ZExtInst    )  // Zero extend integers
-HANDLE_CAST_INST(39, SExt    , SExtInst    )  // Sign extend integers
-HANDLE_CAST_INST(40, FPToUI  , FPToUIInst  )  // floating point -> UInt
-HANDLE_CAST_INST(41, FPToSI  , FPToSIInst  )  // floating point -> SInt
-HANDLE_CAST_INST(42, UIToFP  , UIToFPInst  )  // UInt -> floating point
-HANDLE_CAST_INST(43, SIToFP  , SIToFPInst  )  // SInt -> floating point
-HANDLE_CAST_INST(44, FPTrunc , FPTruncInst )  // Truncate floating point
-HANDLE_CAST_INST(45, FPExt   , FPExtInst   )  // Extend floating point
-HANDLE_CAST_INST(46, PtrToInt, PtrToIntInst)  // Pointer -> Integer
-HANDLE_CAST_INST(47, IntToPtr, IntToPtrInst)  // Integer -> Pointer
-HANDLE_CAST_INST(48, BitCast , BitCastInst )  // Type cast
-HANDLE_CAST_INST(49, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
-  LAST_CAST_INST(49)
-
- FIRST_FUNCLETPAD_INST(50)
-HANDLE_FUNCLETPAD_INST(50, CleanupPad, CleanupPadInst)
-HANDLE_FUNCLETPAD_INST(51, CatchPad  , CatchPadInst)
-  LAST_FUNCLETPAD_INST(51)
+ FIRST_CAST_INST(38)
+HANDLE_CAST_INST(38, Trunc   , TruncInst   )  // Truncate integers
+HANDLE_CAST_INST(39, ZExt    , ZExtInst    )  // Zero extend integers
+HANDLE_CAST_INST(40, SExt    , SExtInst    )  // Sign extend integers
+HANDLE_CAST_INST(41, FPToUI  , FPToUIInst  )  // floating point -> UInt
+HANDLE_CAST_INST(42, FPToSI  , FPToSIInst  )  // floating point -> SInt
+HANDLE_CAST_INST(43, UIToFP  , UIToFPInst  )  // UInt -> floating point
+HANDLE_CAST_INST(44, SIToFP  , SIToFPInst  )  // SInt -> floating point
+HANDLE_CAST_INST(45, FPTrunc , FPTruncInst )  // Truncate floating point
+HANDLE_CAST_INST(46, FPExt   , FPExtInst   )  // Extend floating point
+HANDLE_CAST_INST(47, PtrToInt, PtrToIntInst)  // Pointer -> Integer
+HANDLE_CAST_INST(48, IntToPtr, IntToPtrInst)  // Integer -> Pointer
+HANDLE_CAST_INST(49, BitCast , BitCastInst )  // Type cast
+HANDLE_CAST_INST(50, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
+  LAST_CAST_INST(50)
+
+ FIRST_FUNCLETPAD_INST(51)
+HANDLE_FUNCLETPAD_INST(51, CleanupPad, CleanupPadInst)
+HANDLE_FUNCLETPAD_INST(52, CatchPad  , CatchPadInst)
+  LAST_FUNCLETPAD_INST(52)
 
 // Other operators...
- FIRST_OTHER_INST(52)
-HANDLE_OTHER_INST(52, ICmp   , ICmpInst   )  // Integer comparison instruction
-HANDLE_OTHER_INST(53, FCmp   , FCmpInst   )  // Floating point comparison instr.
-HANDLE_OTHER_INST(54, PHI    , PHINode    )  // PHI node instruction
-HANDLE_OTHER_INST(55, Call   , CallInst   )  // Call a function
-HANDLE_OTHER_INST(56, Select , SelectInst )  // select instruction
-HANDLE_USER_INST (57, UserOp1, Instruction)  // May be used internally in a pass
-HANDLE_USER_INST (58, UserOp2, Instruction)  // Internal to passes only
-HANDLE_OTHER_INST(59, VAArg  , VAArgInst  )  // vaarg instruction
-HANDLE_OTHER_INST(60, ExtractElement, ExtractElementInst)// extract from vector
-HANDLE_OTHER_INST(61, InsertElement, InsertElementInst)  // insert into vector
-HANDLE_OTHER_INST(62, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
-HANDLE_OTHER_INST(63, ExtractValue, ExtractValueInst)// extract from aggregate
-HANDLE_OTHER_INST(64, InsertValue, InsertValueInst)  // insert into aggregate
-HANDLE_OTHER_INST(65, LandingPad, LandingPadInst)  // Landing pad instruction.
-  LAST_OTHER_INST(65)
+ FIRST_OTHER_INST(53)
+HANDLE_OTHER_INST(53, ICmp   , ICmpInst   )  // Integer comparison instruction
+HANDLE_OTHER_INST(54, FCmp   , FCmpInst   )  // Floating point comparison instr.
+HANDLE_OTHER_INST(55, PHI    , PHINode    )  // PHI node instruction
+HANDLE_OTHER_INST(56, Call   , CallInst   )  // Call a function
+HANDLE_OTHER_INST(57, Select , SelectInst )  // select instruction
+HANDLE_USER_INST (58, UserOp1, Instruction)  // May be used internally in a pass
+HANDLE_USER_INST (59, UserOp2, Instruction)  // Internal to passes only
+HANDLE_OTHER_INST(60, VAArg  , VAArgInst  )  // vaarg instruction
+HANDLE_OTHER_INST(61, ExtractElement, ExtractElementInst)// extract from vector
+HANDLE_OTHER_INST(62, InsertElement, InsertElementInst)  // insert into vector
+HANDLE_OTHER_INST(63, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
+HANDLE_OTHER_INST(64, ExtractValue, ExtractValueInst)// extract from aggregate
+HANDLE_OTHER_INST(65, InsertValue, InsertValueInst)  // insert into aggregate
+HANDLE_OTHER_INST(66, LandingPad, LandingPadInst)  // Landing pad instruction.
+  LAST_OTHER_INST(66)
 
 #undef  FIRST_TERM_INST
 #undef HANDLE_TERM_INST
 #undef   LAST_TERM_INST
 
 #undef  FIRST_UNARY_INST
 #undef HANDLE_UNARY_INST
 #undef   LAST_UNARY_INST

llvm/trunk/include/llvm/IR/Instructions.h

InvokeInst::InvokeInst(FunctionType *Ty, Value *Func, BasicBlock *IfNormal,
                        const Twine &NameStr, BasicBlock *InsertAtEnd)
     : CallBase(Ty->getReturnType(), Instruction::Invoke,
                OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
                InsertAtEnd) {
   init(Ty, Func, IfNormal, IfException, Args, Bundles, NameStr);
 }
 
 //===----------------------------------------------------------------------===//
+//                              CallBrInst Class
+//===----------------------------------------------------------------------===//
+
+/// CallBr instruction, tracking function calls that may not return control but
+/// instead transfer it to a third location. The SubclassData field is used to
+/// hold the calling convention of the call.
+///
+class CallBrInst : public CallBase {
+
+  unsigned NumIndirectDests;
+
+  CallBrInst(const CallBrInst &BI);
+
+  /// Construct a CallBrInst given a range of arguments.
+  ///
+  /// Construct a CallBrInst from a range of arguments
+  inline CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *DefaultDest,
+                    ArrayRef<BasicBlock *> IndirectDests,
+                    ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                    const Twine &NameStr, Instruction *InsertBefore);
+
+  inline CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *DefaultDest,
+                    ArrayRef<BasicBlock *> IndirectDests,
+                    ArrayRef<Value *> Args,
+                    ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                    const Twine &NameStr, BasicBlock *InsertAtEnd);
+
+  void init(FunctionType *FTy, Value *Func, BasicBlock *DefaultDest,
+            ArrayRef<BasicBlock *> IndirectDests, ArrayRef<Value *> Args,
+            ArrayRef<OperandBundleDef> Bundles, const Twine &NameStr);
+
+  /// Compute the number of operands to allocate.
+  static int ComputeNumOperands(int NumArgs, int NumIndirectDests,
+                                int NumBundleInputs = 0) {
+    // We need one operand for the called function, plus our extra operands and
+    // the input operand counts provided.
+    return 2 + NumIndirectDests + NumArgs + NumBundleInputs;
+  }
+
+protected:
+  // Note: Instruction needs to be a friend here to call cloneImpl.
+  friend class Instruction;
+
+  CallBrInst *cloneImpl() const;
+
+public:
+  static CallBrInst *Create(FunctionType *Ty, Value *Func,
+                            BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            Instruction *InsertBefore = nullptr) {
+    int NumOperands = ComputeNumOperands(Args.size(), IndirectDests.size());
+    return new (NumOperands)
+        CallBrInst(Ty, Func, DefaultDest, IndirectDests, Args, None,
+                   NumOperands, NameStr, InsertBefore);
+  }
+
+  static CallBrInst *Create(FunctionType *Ty, Value *Func,
+                            BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleDef> Bundles = None,
+                            const Twine &NameStr = "",
+                            Instruction *InsertBefore = nullptr) {
+    int NumOperands = ComputeNumOperands(Args.size(), IndirectDests.size(),
+                                         CountBundleInputs(Bundles));
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (NumOperands, DescriptorBytes)
+        CallBrInst(Ty, Func, DefaultDest, IndirectDests, Args, Bundles,
+                   NumOperands, NameStr, InsertBefore);
+  }
+
+  static CallBrInst *Create(FunctionType *Ty, Value *Func,
+                            BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            BasicBlock *InsertAtEnd) {
+    int NumOperands = ComputeNumOperands(Args.size(), IndirectDests.size());
+    return new (NumOperands)
+        CallBrInst(Ty, Func, DefaultDest, IndirectDests, Args, None,
+                   NumOperands, NameStr, InsertAtEnd);
+  }
+
+  static CallBrInst *Create(FunctionType *Ty, Value *Func,
+                            BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleDef> Bundles,
+                            const Twine &NameStr, BasicBlock *InsertAtEnd) {
+    int NumOperands = ComputeNumOperands(Args.size(), IndirectDests.size(),
+                                         CountBundleInputs(Bundles));
+    unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
+
+    return new (NumOperands, DescriptorBytes)
+        CallBrInst(Ty, Func, DefaultDest, IndirectDests, Args, Bundles,
+                   NumOperands, NameStr, InsertAtEnd);
+  }
+
+  static CallBrInst *Create(FunctionCallee Func, BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            Instruction *InsertBefore = nullptr) {
+    return Create(Func.getFunctionType(), Func.getCallee(), DefaultDest,
+                  IndirectDests, Args, NameStr, InsertBefore);
+  }
+
+  static CallBrInst *Create(FunctionCallee Func, BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleDef> Bundles = None,
+                            const Twine &NameStr = "",
+                            Instruction *InsertBefore = nullptr) {
+    return Create(Func.getFunctionType(), Func.getCallee(), DefaultDest,
+                  IndirectDests, Args, Bundles, NameStr, InsertBefore);
+  }
+
+  static CallBrInst *Create(FunctionCallee Func, BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args, const Twine &NameStr,
+                            BasicBlock *InsertAtEnd) {
+    return Create(Func.getFunctionType(), Func.getCallee(), DefaultDest,
+                  IndirectDests, Args, NameStr, InsertAtEnd);
+  }
+
+  static CallBrInst *Create(FunctionCallee Func,
+                            BasicBlock *DefaultDest,
+                            ArrayRef<BasicBlock *> IndirectDests,
+                            ArrayRef<Value *> Args,
+                            ArrayRef<OperandBundleDef> Bundles,
+                            const Twine &NameStr, BasicBlock *InsertAtEnd) {
+    return Create(Func.getFunctionType(), Func.getCallee(), DefaultDest,
+                  IndirectDests, Args, Bundles, NameStr, InsertAtEnd);
+  }
+
+  /// Create a clone of \p CBI with a different set of operand bundles and
+  /// insert it before \p InsertPt.
+  ///
+  /// The returned callbr instruction is identical to \p CBI in every way
+  /// except that the operand bundles for the new instruction are set to the
+  /// operand bundles in \p Bundles.
+  static CallBrInst *Create(CallBrInst *CBI,
+                            ArrayRef<OperandBundleDef> Bundles,
+                            Instruction *InsertPt = nullptr);
+
+  /// Return the number of callbr indirect dest labels.
+  ///
+  unsigned getNumIndirectDests() const { return NumIndirectDests; }
+
+  /// getIndirectDestLabel - Return the i-th indirect dest label.
+  ///
+  Value *getIndirectDestLabel(unsigned i) const {
+    assert(i < getNumIndirectDests() && "Out of bounds!");
+    return getOperand(i + getNumArgOperands() + getNumTotalBundleOperands() +
+                      1);
+  }
+
+  Value *getIndirectDestLabelUse(unsigned i) const {
+    assert(i < getNumIndirectDests() && "Out of bounds!");
+    return getOperandUse(i + getNumArgOperands() + getNumTotalBundleOperands() +
+                         1);
+  }
+
+  // Return the destination basic blocks...
+  BasicBlock *getDefaultDest() const {
+    return cast<BasicBlock>(*(&Op<-1>() - getNumIndirectDests() - 1));
+  }
+  BasicBlock *getIndirectDest(unsigned i) const {
+    return cast<BasicBlock>(*(&Op<-1>() - getNumIndirectDests() + i));
+  }
+  SmallVector<BasicBlock *, 16> getIndirectDests() const {
+    SmallVector<BasicBlock *, 16> IndirectDests;
+    for (unsigned i = 0, e = getNumIndirectDests(); i < e; ++i)
+      IndirectDests.push_back(getIndirectDest(i));
+    return IndirectDests;
+  }
+  void setDefaultDest(BasicBlock *B) {
+    *(&Op<-1>() - getNumIndirectDests() - 1) = reinterpret_cast<Value *>(B);
+  }
+  void setIndirectDest(unsigned i, BasicBlock *B) {
+    *(&Op<-1>() - getNumIndirectDests() + i) = reinterpret_cast<Value *>(B);
+  }
+
+  BasicBlock *getSuccessor(unsigned i) const {
+    assert(i < getNumSuccessors() + 1 &&
+           "Successor # out of range for callbr!");
+    return i == 0 ? getDefaultDest() : getIndirectDest(i - 1);
+  }
+
+  void setSuccessor(unsigned idx, BasicBlock *NewSucc) {
+    assert(idx < getNumIndirectDests() + 1 &&
+           "Successor # out of range for callbr!");
+    *(&Op<-1>() - getNumIndirectDests() -1 + idx) =
+        reinterpret_cast<Value *>(NewSucc);
+  }
+
+  unsigned getNumSuccessors() const { return getNumIndirectDests() + 1; }
+
+  // Methods for support type inquiry through isa, cast, and dyn_cast:
+  static bool classof(const Instruction *I) {
+    return (I->getOpcode() == Instruction::CallBr);
+  }
+  static bool classof(const Value *V) {
+    return isa<Instruction>(V) && classof(cast<Instruction>(V));
+  }
+
+private:
+
+  // Shadow Instruction::setInstructionSubclassData with a private forwarding
+  // method so that subclasses cannot accidentally use it.
+  void setInstructionSubclassData(unsigned short D) {
+    Instruction::setInstructionSubclassData(D);
+  }
+};
+
+CallBrInst::CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *DefaultDest,
+                       ArrayRef<BasicBlock *> IndirectDests,
+                       ArrayRef<Value *> Args,
+                       ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                       const Twine &NameStr, Instruction *InsertBefore)
+    : CallBase(Ty->getReturnType(), Instruction::CallBr,
+               OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
+               InsertBefore) {
+  init(Ty, Func, DefaultDest, IndirectDests, Args, Bundles, NameStr);
+}
+
+CallBrInst::CallBrInst(FunctionType *Ty, Value *Func, BasicBlock *DefaultDest,
+                       ArrayRef<BasicBlock *> IndirectDests,
+                       ArrayRef<Value *> Args,
+                       ArrayRef<OperandBundleDef> Bundles, int NumOperands,
+                       const Twine &NameStr, BasicBlock *InsertAtEnd)
+    : CallBase(
+          cast<FunctionType>(
+              cast<PointerType>(Func->getType())->getElementType())
+              ->getReturnType(),
+          Instruction::CallBr,
+          OperandTraits<CallBase>::op_end(this) - NumOperands, NumOperands,
+          InsertAtEnd) {
+  init(Ty, Func, DefaultDest, IndirectDests, Args, Bundles, NameStr);
+}
+
+//===----------------------------------------------------------------------===//
 //                              ResumeInst Class
 //===----------------------------------------------------------------------===//
 
 //===---------------------------------------------------------------------------
 /// Resume the propagation of an exception.
 ///
 class ResumeInst : public Instruction {
   ResumeInst(const ResumeInst &RI);

llvm/trunk/include/llvm/Support/TargetOpcodes.def

 #ifndef HANDLE_TARGET_OPCODE_MARKER
 #define HANDLE_TARGET_OPCODE_MARKER(IDENT, OPC)
 #endif
 
 /// Every instruction defined here must also appear in Target.td.
 ///
 HANDLE_TARGET_OPCODE(PHI)
 HANDLE_TARGET_OPCODE(INLINEASM)
+HANDLE_TARGET_OPCODE(INLINEASM_BR)
 HANDLE_TARGET_OPCODE(CFI_INSTRUCTION)
 HANDLE_TARGET_OPCODE(EH_LABEL)
 HANDLE_TARGET_OPCODE(GC_LABEL)
 HANDLE_TARGET_OPCODE(ANNOTATION_LABEL)
 
 /// KILL - This instruction is a noop that is used only to adjust the
 /// liveness of registers. This can be useful when dealing with
 /// sub-registers.

llvm/trunk/include/llvm/Target/Target.td

def PHI : StandardPseudoInstruction {
   let hasSideEffects = 0;
 }
 def INLINEASM : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins variable_ops);
   let AsmString = "";
   let hasSideEffects = 0;  // Note side effect is encoded in an operand.
 }
+def INLINEASM_BR : StandardPseudoInstruction {
+  let OutOperandList = (outs);
+  let InOperandList = (ins variable_ops);
+  let AsmString = "";
+  let hasSideEffects = 0;  // Note side effect is encoded in an operand.
+  let isTerminator = 1;
+  let isBranch = 1;
+  let isIndirectBranch = 1;
+}
 def CFI_INSTRUCTION : StandardPseudoInstruction {
   let OutOperandList = (outs);
   let InOperandList = (ins i32imm:$id);
   let AsmString = "";
   let hasCtrlDep = 1;
   let hasSideEffects = 0;
   let isNotDuplicable = 1;
 }

llvm/trunk/lib/Analysis/ValueTracking.cpp

case Instruction::Call: {
 
     // The called function could have undefined behavior or side-effects, even
     // if marked readnone nounwind.
     return Callee && Callee->isSpeculatable();
   }
   case Instruction::VAArg:
   case Instruction::Alloca:
   case Instruction::Invoke:
+  case Instruction::CallBr:
   case Instruction::PHI:
   case Instruction::Store:
   case Instruction::Ret:
   case Instruction::Br:
   case Instruction::IndirectBr:
   case Instruction::Switch:
   case Instruction::Unreachable:
   case Instruction::Fence:

llvm/trunk/lib/AsmParser/LLLexer.cpp

#define INSTKEYWORD(STR, Enum)                                                 \
   INSTKEYWORD(va_arg,      VAArg);
   INSTKEYWORD(ret,         Ret);
   INSTKEYWORD(br,          Br);
   INSTKEYWORD(switch,      Switch);
   INSTKEYWORD(indirectbr,  IndirectBr);
   INSTKEYWORD(invoke,      Invoke);
   INSTKEYWORD(resume,      Resume);
   INSTKEYWORD(unreachable, Unreachable);
+  INSTKEYWORD(callbr,      CallBr);
 
   INSTKEYWORD(alloca,      Alloca);
   INSTKEYWORD(load,        Load);
   INSTKEYWORD(store,       Store);
   INSTKEYWORD(cmpxchg,     AtomicCmpXchg);
   INSTKEYWORD(atomicrmw,   AtomicRMW);
   INSTKEYWORD(fence,       Fence);
   INSTKEYWORD(getelementptr, GetElementPtr);

llvm/trunk/lib/AsmParser/LLParser.h

#include "llvm/IR/Metadata.def"
     bool ParseIndirectBr(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseInvoke(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseResume(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchSwitch(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchPad(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupPad(Instruction *&Inst, PerFunctionState &PFS);
+    bool ParseCallBr(Instruction *&Inst, PerFunctionState &PFS);
 
     bool ParseUnaryOp(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc,
                       unsigned OperandType);
     bool ParseArithmetic(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc,
                          unsigned OperandType);
     bool ParseLogical(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);
     bool ParseCompare(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);
     bool ParseCast(Instruction *&Inst, PerFunctionState &PFS, unsigned Opc);

llvm/trunk/lib/AsmParser/LLParser.cpp

for (const auto &RAG : ForwardRefAttrGroups) {
     } else if (InvokeInst *II = dyn_cast<InvokeInst>(V)) {
       AttributeList AS = II->getAttributes();
       AttrBuilder FnAttrs(AS.getFnAttributes());
       AS = AS.removeAttributes(Context, AttributeList::FunctionIndex);
       FnAttrs.merge(B);
       AS = AS.addAttributes(Context, AttributeList::FunctionIndex,
                             AttributeSet::get(Context, FnAttrs));
       II->setAttributes(AS);
+    } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(V)) {
+      AttributeList AS = CBI->getAttributes();
+      AttrBuilder FnAttrs(AS.getFnAttributes());
+      AS = AS.removeAttributes(Context, AttributeList::FunctionIndex);
+      FnAttrs.merge(B);
+      AS = AS.addAttributes(Context, AttributeList::FunctionIndex,
+                            AttributeSet::get(Context, FnAttrs));
+      CBI->setAttributes(AS);
     } else if (auto *GV = dyn_cast<GlobalVariable>(V)) {
       AttrBuilder Attrs(GV->getAttributes());
       Attrs.merge(B);
       GV->setAttributes(AttributeSet::get(Context,Attrs));
     } else {
       llvm_unreachable("invalid object with forward attribute group reference");
     }
   }
int LLParser::ParseInstruction(Instruction *&Inst, BasicBlock *BB,
   case lltok::kw_indirectbr:  return ParseIndirectBr(Inst, PFS);
   case lltok::kw_invoke:      return ParseInvoke(Inst, PFS);
   case lltok::kw_resume:      return ParseResume(Inst, PFS);
   case lltok::kw_cleanupret:  return ParseCleanupRet(Inst, PFS);
   case lltok::kw_catchret:    return ParseCatchRet(Inst, PFS);
   case lltok::kw_catchswitch: return ParseCatchSwitch(Inst, PFS);
   case lltok::kw_catchpad:    return ParseCatchPad(Inst, PFS);
   case lltok::kw_cleanuppad:  return ParseCleanupPad(Inst, PFS);
+  case lltok::kw_callbr:      return ParseCallBr(Inst, PFS);
   // Unary Operators.
   case lltok::kw_fneg: {
     FastMathFlags FMF = EatFastMathFlagsIfPresent();
     int Res = ParseUnaryOp(Inst, PFS, KeywordVal, 2);
     if (Res != 0)
       return Res;
     if (FMF.any())
       Inst->setFastMathFlags(FMF);
bool LLParser::ParseUnaryOp(Instruction *&Inst, PerFunctionState &PFS,
 
   if (!Valid)
     return Error(Loc, "invalid operand type for instruction");
 
   Inst = UnaryOperator::Create((Instruction::UnaryOps)Opc, LHS);
   return false;
 }
 
+/// ParseCallBr
+///   ::= 'callbr' OptionalCallingConv OptionalAttrs Type Value ParamList
+///       OptionalAttrs OptionalOperandBundles 'to' TypeAndValue
+///       '[' LabelList ']'
+bool LLParser::ParseCallBr(Instruction *&Inst, PerFunctionState &PFS) {
+  LocTy CallLoc = Lex.getLoc();
+  AttrBuilder RetAttrs, FnAttrs;
+  std::vector<unsigned> FwdRefAttrGrps;
+  LocTy NoBuiltinLoc;
+  unsigned CC;
+  Type *RetType = nullptr;
+  LocTy RetTypeLoc;
+  ValID CalleeID;
+  SmallVector<ParamInfo, 16> ArgList;
+  SmallVector<OperandBundleDef, 2> BundleList;
+
+  BasicBlock *DefaultDest;
+  if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+      ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
+      ParseValID(CalleeID) || ParseParameterList(ArgList, PFS) ||
+      ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false,
+                                 NoBuiltinLoc) ||
+      ParseOptionalOperandBundles(BundleList, PFS) ||
+      ParseToken(lltok::kw_to, "expected 'to' in callbr") ||
+      ParseTypeAndBasicBlock(DefaultDest, PFS) ||
+      ParseToken(lltok::lsquare, "expected '[' in callbr"))
+    return true;
+
+  // Parse the destination list.
+  SmallVector<BasicBlock *, 16> IndirectDests;
+
+  if (Lex.getKind() != lltok::rsquare) {
+    BasicBlock *DestBB;
+    if (ParseTypeAndBasicBlock(DestBB, PFS))
+      return true;
+    IndirectDests.push_back(DestBB);
+
+    while (EatIfPresent(lltok::comma)) {
+      if (ParseTypeAndBasicBlock(DestBB, PFS))
+        return true;
+      IndirectDests.push_back(DestBB);
+    }
+  }
+
+  if (ParseToken(lltok::rsquare, "expected ']' at end of block list"))
+    return true;
+
+  // If RetType is a non-function pointer type, then this is the short syntax
+  // for the call, which means that RetType is just the return type.  Infer the
+  // rest of the function argument types from the arguments that are present.
+  FunctionType *Ty = dyn_cast<FunctionType>(RetType);
+  if (!Ty) {
+    // Pull out the types of all of the arguments...
+    std::vector<Type *> ParamTypes;
+    for (unsigned i = 0, e = ArgList.size(); i != e; ++i)
+      ParamTypes.push_back(ArgList[i].V->getType());
+
+    if (!FunctionType::isValidReturnType(RetType))
+      return Error(RetTypeLoc, "Invalid result type for LLVM function");
+
+    Ty = FunctionType::get(RetType, ParamTypes, false);
+  }
+
+  CalleeID.FTy = Ty;
+
+  // Look up the callee.
+  Value *Callee;
+  if (ConvertValIDToValue(PointerType::getUnqual(Ty), CalleeID, Callee, &PFS,
+                          /*IsCall=*/true))
+    return true;
+
+  if (isa<InlineAsm>(Callee) && !Ty->getReturnType()->isVoidTy())
+    return Error(RetTypeLoc, "asm-goto outputs not supported");
+
+  // Set up the Attribute for the function.
+  SmallVector<Value *, 8> Args;
+  SmallVector<AttributeSet, 8> ArgAttrs;
+
+  // Loop through FunctionType's arguments and ensure they are specified
+  // correctly.  Also, gather any parameter attributes.
+  FunctionType::param_iterator I = Ty->param_begin();
+  FunctionType::param_iterator E = Ty->param_end();
+  for (unsigned i = 0, e = ArgList.size(); i != e; ++i) {
+    Type *ExpectedTy = nullptr;
+    if (I != E) {
+      ExpectedTy = *I++;
+    } else if (!Ty->isVarArg()) {
+      return Error(ArgList[i].Loc, "too many arguments specified");
+    }
+
+    if (ExpectedTy && ExpectedTy != ArgList[i].V->getType())
+      return Error(ArgList[i].Loc, "argument is not of expected type '" +
+                                       getTypeString(ExpectedTy) + "'");
+    Args.push_back(ArgList[i].V);
+    ArgAttrs.push_back(ArgList[i].Attrs);
+  }
+
+  if (I != E)
+    return Error(CallLoc, "not enough parameters specified for call");
+
+  if (FnAttrs.hasAlignmentAttr())
+    return Error(CallLoc, "callbr instructions may not have an alignment");
+
+  // Finish off the Attribute and check them
+  AttributeList PAL =
+      AttributeList::get(Context, AttributeSet::get(Context, FnAttrs),
+                         AttributeSet::get(Context, RetAttrs), ArgAttrs);
+
+  CallBrInst *CBI =
+      CallBrInst::Create(Ty, Callee, DefaultDest, IndirectDests, Args,
+                         BundleList);
+  CBI->setCallingConv(CC);
+  CBI->setAttributes(PAL);
+  ForwardRefAttrGroups[CBI] = FwdRefAttrGrps;
+  Inst = CBI;
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 // Binary Operators.
 //===----------------------------------------------------------------------===//
 
 /// ParseArithmetic
 ///  ::= ArithmeticOps TypeAndValue ',' Value
 ///
 /// If OperandType is 0, then any FP or integer operand is allowed.  If it is 1,

llvm/trunk/lib/AsmParser/LLToken.h

enum Kind {
   kw_invoke,
   kw_resume,
   kw_unreachable,
   kw_cleanupret,
   kw_catchswitch,
   kw_catchret,
   kw_catchpad,
   kw_cleanuppad,
+  kw_callbr,
 
   kw_alloca,
   kw_load,
   kw_store,
   kw_fence,
   kw_cmpxchg,
   kw_atomicrmw,
   kw_getelementptr,

llvm/trunk/lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_RESUME: { // RESUME: [opval]
       unsigned Idx = 0;
       Value *Val = nullptr;
       if (getValueTypePair(Record, Idx, NextValueNo, Val))
         return error("Invalid record");
       I = ResumeInst::Create(Val);
       InstructionList.push_back(I);
       break;
     }
+    case bitc::FUNC_CODE_INST_CALLBR: {
+      // CALLBR: [attr, cc, norm, transfs, fty, fnid, args]
+      unsigned OpNum = 0;
+      AttributeList PAL = getAttributes(Record[OpNum++]);
+      unsigned CCInfo = Record[OpNum++];
+
+      BasicBlock *DefaultDest = getBasicBlock(Record[OpNum++]);
+      unsigned NumIndirectDests = Record[OpNum++];
+      SmallVector<BasicBlock *, 16> IndirectDests;
+      for (unsigned i = 0, e = NumIndirectDests; i != e; ++i)
+        IndirectDests.push_back(getBasicBlock(Record[OpNum++]));
+
+      FunctionType *FTy = nullptr;
+      if (CCInfo >> bitc::CALL_EXPLICIT_TYPE & 1 &&
+          !(FTy = dyn_cast<FunctionType>(getTypeByID(Record[OpNum++]))))
+        return error("Explicit call type is not a function type");
+
+      Value *Callee;
+      if (getValueTypePair(Record, OpNum, NextValueNo, Callee))
+        return error("Invalid record");
+
+      PointerType *OpTy = dyn_cast<PointerType>(Callee->getType());
+      if (!OpTy)
+        return error("Callee is not a pointer type");
+      if (!FTy) {
+        FTy = dyn_cast<FunctionType>(OpTy->getElementType());
+        if (!FTy)
+          return error("Callee is not of pointer to function type");
+      } else if (OpTy->getElementType() != FTy)
+        return error("Explicit call type does not match pointee type of "
+                     "callee operand");
+      if (Record.size() < FTy->getNumParams() + OpNum)
+        return error("Insufficient operands to call");
+
+      SmallVector<Value*, 16> Args;
+      // Read the fixed params.
+      for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i, ++OpNum) {
+        if (FTy->getParamType(i)->isLabelTy())
+          Args.push_back(getBasicBlock(Record[OpNum]));
+        else
+          Args.push_back(getValue(Record, OpNum, NextValueNo,
+                                  FTy->getParamType(i)));
+        if (!Args.back())
+          return error("Invalid record");
+      }
+
+      // Read type/value pairs for varargs params.
+      if (!FTy->isVarArg()) {
+        if (OpNum != Record.size())
+          return error("Invalid record");
+      } else {
+        while (OpNum != Record.size()) {
+          Value *Op;
+          if (getValueTypePair(Record, OpNum, NextValueNo, Op))
+            return error("Invalid record");
+          Args.push_back(Op);
+        }
+      }
+
+      I = CallBrInst::Create(FTy, Callee, DefaultDest, IndirectDests, Args,
+                             OperandBundles);
+      OperandBundles.clear();
+      InstructionList.push_back(I);
+      cast<CallBrInst>(I)->setCallingConv(
+          static_cast<CallingConv::ID>((0x7ff & CCInfo) >> bitc::CALL_CCONV));
+      cast<CallBrInst>(I)->setAttributes(PAL);
+      break;
+    }
     case bitc::FUNC_CODE_INST_UNREACHABLE: // UNREACHABLE
       I = new UnreachableInst(Context);
       InstructionList.push_back(I);
       break;
     case bitc::FUNC_CODE_INST_PHI: { // PHI: [ty, val0,bb0, ...]
       if (Record.size() < 1 || ((Record.size()-1)&1))
         return error("Invalid record");
       Type *Ty = getTypeByID(Record[0]);

llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp

case Instruction::CatchSwitch: {
     Vals.push_back(NumHandlers);
     for (const BasicBlock *CatchPadBB : CatchSwitch.handlers())
       Vals.push_back(VE.getValueID(CatchPadBB));
 
     if (CatchSwitch.hasUnwindDest())
       Vals.push_back(VE.getValueID(CatchSwitch.getUnwindDest()));
     break;
   }
+  case Instruction::CallBr: {
+    const CallBrInst *CBI = cast<CallBrInst>(&I);
+    const Value *Callee = CBI->getCalledValue();
+    FunctionType *FTy = CBI->getFunctionType();
+
+    if (CBI->hasOperandBundles())
+      writeOperandBundles(CBI, InstID);
+
+    Code = bitc::FUNC_CODE_INST_CALLBR;
+
+    Vals.push_back(VE.getAttributeListID(CBI->getAttributes()));
+
+    Vals.push_back(CBI->getCallingConv() << bitc::CALL_CCONV |
+                   1 << bitc::CALL_EXPLICIT_TYPE);
+
+    Vals.push_back(VE.getValueID(CBI->getDefaultDest()));
+    Vals.push_back(CBI->getNumIndirectDests());
+    for (unsigned i = 0, e = CBI->getNumIndirectDests(); i != e; ++i)
+      Vals.push_back(VE.getValueID(CBI->getIndirectDest(i)));
+
+    Vals.push_back(VE.getTypeID(FTy));
+    pushValueAndType(Callee, InstID, Vals);
+
+    // Emit value #'s for the fixed parameters.
+    for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i)
+      pushValue(I.getOperand(i), InstID, Vals); // fixed param.
+
+    // Emit type/value pairs for varargs params.
+    if (FTy->isVarArg()) {
+      for (unsigned i = FTy->getNumParams(), e = CBI->getNumArgOperands();
+           i != e; ++i)
+        pushValueAndType(I.getOperand(i), InstID, Vals); // vararg
+    }
+    break;
+  }
   case Instruction::Unreachable:
     Code = bitc::FUNC_CODE_INST_UNREACHABLE;
     AbbrevToUse = FUNCTION_INST_UNREACHABLE_ABBREV;
     break;
 
   case Instruction::PHI: {
     const PHINode &PN = cast<PHINode>(I);
     Code = bitc::FUNC_CODE_INST_PHI;

llvm/trunk/lib/Bitcode/Writer/ValueEnumerator.cpp

for (const BasicBlock &BB : F)
 
           // Local metadata is enumerated during function-incorporation.
           if (isa<LocalAsMetadata>(MD->getMetadata()))
             continue;
 
           EnumerateMetadata(&F, MD->getMetadata());
         }
         EnumerateType(I.getType());
-        if (const CallInst *CI = dyn_cast<CallInst>(&I))
-          EnumerateAttributes(CI->getAttributes());
-        else if (const InvokeInst *II = dyn_cast<InvokeInst>(&I))
-          EnumerateAttributes(II->getAttributes());
+        if (const auto *Call = dyn_cast<CallBase>(&I))
+          EnumerateAttributes(Call->getAttributes());
 
         // Enumerate metadata attached with this instruction.
         MDs.clear();
         I.getAllMetadataOtherThanDebugLoc(MDs);
         for (unsigned i = 0, e = MDs.size(); i != e; ++i)
           EnumerateMetadata(&F, MDs[i].second);
 
         // Don't enumerate the location directly -- it has a special record

llvm/trunk/lib/CodeGen/AsmPrinter/AsmPrinter.cpp

for (auto &MI : MBB) {
       case TargetOpcode::LOCAL_ESCAPE:
         emitFrameAlloc(MI);
         break;
       case TargetOpcode::EH_LABEL:
       case TargetOpcode::GC_LABEL:
         OutStreamer->EmitLabel(MI.getOperand(0).getMCSymbol());
         break;
       case TargetOpcode::INLINEASM:
+      case TargetOpcode::INLINEASM_BR:
         EmitInlineAsm(&MI);
         break;
       case TargetOpcode::DBG_VALUE:
         if (isVerbose()) {
           if (!emitDebugValueComment(&MI, *this))
             EmitInstruction(&MI);
         }
         break;

llvm/trunk/lib/CodeGen/AsmPrinter/AsmPrinterInlineAsm.cpp

case '$': {
         if (OpNo >= MI->getNumOperands() ||
             MI->getOperand(OpNo).isMetadata()) {
           Error = true;
         } else {
           unsigned OpFlags = MI->getOperand(OpNo).getImm();
           ++OpNo;  // Skip over the ID number.
 
           if (Modifier[0] == 'l') { // Labels are target independent.
-            // FIXME: What if the operand isn't an MBB, report error?
+            if (MI->getOperand(OpNo).isBlockAddress()) {
+              const BlockAddress *BA = MI->getOperand(OpNo).getBlockAddress();
+              MCSymbol *Sym = AP->GetBlockAddressSymbol(BA);
+              Sym->print(OS, AP->MAI);
+            } else if (MI->getOperand(OpNo).isMBB()) {
               const MCSymbol *Sym = MI->getOperand(OpNo).getMBB()->getSymbol();
               Sym->print(OS, AP->MAI);
             } else {
+              Error = true;
+            }
+          } else {
             if (InlineAsm::isMemKind(OpFlags)) {
               Error = AP->PrintAsmMemoryOperand(MI, OpNo, InlineAsmVariant,
                                                 Modifier[0] ? Modifier : nullptr,
                                                 OS);
             } else {
               Error = AP->PrintAsmOperand(MI, OpNo, InlineAsmVariant,
                                           Modifier[0] ? Modifier : nullptr, OS);
             }

llvm/trunk/lib/CodeGen/CodeGenPrepare.cpp

bool CodeGenPrepare::isMergingEmptyBlockProfitable(BasicBlock *BB,
   // Loop preheaders can be good locations to spill registers. If the
   // preheader is deleted and we create a critical edge, registers may be
   // spilled in the loop body instead.
   if (!DisablePreheaderProtect && isPreheader &&
       !(BB->getSinglePredecessor() &&
         BB->getSinglePredecessor()->getSingleSuccessor()))
     return false;
 
+  // Skip merging if the block's successor is also a successor to any callbr
+  // that leads to this block.
+  // FIXME: Is this really needed? Is this a correctness issue?
+  for (pred_iterator PI = pred_begin(BB), E = pred_end(BB); PI != E; ++PI) {
+    if (auto *CBI = dyn_cast<CallBrInst>((*PI)->getTerminator()))
+      for (unsigned i = 0, e = CBI->getNumSuccessors(); i != e; ++i)
+        if (DestBB == CBI->getSuccessor(i))
+          return false;
+  }
+
   // Try to skip merging if the unique predecessor of BB is terminated by a
   // switch or indirect branch instruction, and BB is used as an incoming block
   // of PHIs in DestBB. In such case, merging BB and DestBB would cause ISel to
   // add COPY instructions in the predecessor of BB instead of BB (if it is not
   // merged). Note that the critical edge created by merging such blocks wont be
   // split in MachineSink because the jump table is not analyzable. By keeping
   // such empty block (BB), ISel will place COPY instructions in BB, not in the
   // predecessor of BB.

llvm/trunk/lib/CodeGen/GlobalISel/IRTranslator.cpp

bool IRTranslator::translateInvoke(const User &U,
   MF->addInvoke(&EHPadMBB, BeginSymbol, EndSymbol);
   MIRBuilder.getMBB().addSuccessor(&ReturnMBB);
   MIRBuilder.getMBB().addSuccessor(&EHPadMBB);
   MIRBuilder.buildBr(ReturnMBB);
 
   return true;
 }
 
+bool IRTranslator::translateCallBr(const User &U,
+                                   MachineIRBuilder &MIRBuilder) {
+  // FIXME: Implement this.
+  return false;
+}
+
 bool IRTranslator::translateLandingPad(const User &U,
                                        MachineIRBuilder &MIRBuilder) {
   const LandingPadInst &LP = cast<LandingPadInst>(U);
 
   MachineBasicBlock &MBB = MIRBuilder.getMBB();
 
   MBB.setIsEHPad();
 

llvm/trunk/lib/CodeGen/IndirectBrExpandPass.cpp

for (BasicBlock &BB : F) {
     // because null can be compared with block addresses.
     int BBIndex = BBs.size() + 1;
     BBs.push_back(&BB);
 
     auto *ITy = cast<IntegerType>(DL.getIntPtrType(BA->getType()));
     ConstantInt *BBIndexC = ConstantInt::get(ITy, BBIndex);
 
     // Now rewrite the blockaddress to an integer constant based on the index.
-    // FIXME: We could potentially preserve the uses as arguments to inline asm.
-    // This would allow some uses such as diagnostic information in crashes to
-    // have higher quality even when this transform is enabled, but would break
-    // users that round-trip blockaddresses through inline assembly and then
-    // back into an indirectbr.
+    // FIXME: This part doesn't properly recognize other uses of blockaddress
+    // expressions, for instance, where they are used to pass labels to
+    // asm-goto. This part of the pass needs a rework.
     BA->replaceAllUsesWith(ConstantExpr::getIntToPtr(BBIndexC, BA->getType()));
   }
 
   if (BBs.empty()) {
     // There are no blocks whose address is taken, so any indirectbr instruction
     // cannot get a valid input and we can replace all of them with unreachable.
     for (auto *IBr : IndirectBrs) {
       (void)new UnreachableInst(F.getContext(), IBr);

llvm/trunk/lib/CodeGen/SelectionDAG/InstrEmitter.cpp

case ISD::LIFETIME_END: {
     TargetOpcode::LIFETIME_START : TargetOpcode::LIFETIME_END;
 
     FrameIndexSDNode *FI = dyn_cast<FrameIndexSDNode>(Node->getOperand(1));
     BuildMI(*MBB, InsertPos, Node->getDebugLoc(), TII->get(TarOp))
     .addFrameIndex(FI->getIndex());
     break;
   }
 
-  case ISD::INLINEASM: {
+  case ISD::INLINEASM:
+  case ISD::INLINEASM_BR: {
     unsigned NumOps = Node->getNumOperands();
     if (Node->getOperand(NumOps-1).getValueType() == MVT::Glue)
       --NumOps;  // Ignore the glue operand.
 
     // Create the inline asm machine instruction.
-    MachineInstrBuilder MIB = BuildMI(*MF, Node->getDebugLoc(),
-                                      TII->get(TargetOpcode::INLINEASM));
+    unsigned TgtOpc = Node->getOpcode() == ISD::INLINEASM_BR
+                          ? TargetOpcode::INLINEASM_BR
+                          : TargetOpcode::INLINEASM;
+    MachineInstrBuilder MIB =
+        BuildMI(*MF, Node->getDebugLoc(), TII->get(TgtOpc));
 
     // Add the asm string as an external symbol operand.
     SDValue AsmStrV = Node->getOperand(InlineAsm::Op_AsmString);
     const char *AsmStr = cast<ExternalSymbolSDNode>(AsmStrV)->getSymbol();
     MIB.addExternalSymbol(AsmStr);
 
     // Add the HasSideEffect, isAlignStack, AsmDialect, MayLoad and MayStore
     // bits.

llvm/trunk/lib/CodeGen/SelectionDAG/ResourcePriorityQueue.cpp

for (SDep &Pred : SU->Preds) {
     // If value is passed to CopyToReg, it is probably
     // live outside BB.
     switch (ScegN->getOpcode()) {
       default:  break;
       case ISD::TokenFactor:    break;
       case ISD::CopyFromReg:    NumberDeps++;  break;
       case ISD::CopyToReg:      break;
       case ISD::INLINEASM:      break;
+      case ISD::INLINEASM_BR:   break;
     }
     if (!ScegN->isMachineOpcode())
       continue;
 
     for (unsigned i = 0, e = ScegN->getNumValues(); i != e; ++i) {
       MVT VT = ScegN->getSimpleValueType(i);
       if (TLI->isTypeLegal(VT)
           && (TLI->getRegClassFor(VT)->getID() == RCId)) {
for (const SDep &Succ : SU->Succs) {
     // If value is passed to CopyToReg, it is probably
     // live outside BB.
     switch (ScegN->getOpcode()) {
       default:  break;
       case ISD::TokenFactor:    break;
       case ISD::CopyFromReg:    break;
       case ISD::CopyToReg:      NumberDeps++;  break;
       case ISD::INLINEASM:      break;
+      case ISD::INLINEASM_BR:   break;
     }
     if (!ScegN->isMachineOpcode())
       continue;
 
     for (unsigned i = 0, e = ScegN->getNumOperands(); i != e; ++i) {
       const SDValue &Op = ScegN->getOperand(i);
       MVT VT = Op.getNode()->getSimpleValueType(Op.getResNo());
       if (TLI->isTypeLegal(VT)
else
       default:  break;
       case ISD::TokenFactor:
       case ISD::CopyFromReg:
       case ISD::CopyToReg:
         ResCount += PriorityFour;
         break;
 
       case ISD::INLINEASM:
+      case ISD::INLINEASM_BR:
         ResCount += PriorityThree;
         break;
       }
   }
   return ResCount;
 }
 
 
for (SDNode *N = SU->getNode(); N; N = N->getGluedNode())
     }
     else
       switch(N->getOpcode()) {
         default:     break;
         case ISD::CopyFromReg:
           NodeNumDefs++;
           break;
         case ISD::INLINEASM:
+        case ISD::INLINEASM_BR:
           NodeNumDefs++;
           break;
       }
 
   SU->NumRegDefsLeft = NodeNumDefs;
 }
 
 /// adjustPriorityOfUnscheduledPreds - One of the predecessors of SU was just

llvm/trunk/lib/CodeGen/SelectionDAG/ScheduleDAGFast.cpp

bool ScheduleDAGFast::DelayForLiveRegsBottomUp(SUnit *SU,
   for (SDep &Pred : SU->Preds) {
     if (Pred.isAssignedRegDep()) {
       CheckForLiveRegDef(Pred.getSUnit(), Pred.getReg(), LiveRegDefs,
                          RegAdded, LRegs, TRI);
     }
   }
 
   for (SDNode *Node = SU->getNode(); Node; Node = Node->getGluedNode()) {
-    if (Node->getOpcode() == ISD::INLINEASM) {
+    if (Node->getOpcode() == ISD::INLINEASM ||
+        Node->getOpcode() == ISD::INLINEASM_BR) {
       // Inline asm can clobber physical defs.
       unsigned NumOps = Node->getNumOperands();
       if (Node->getOperand(NumOps-1).getValueType() == MVT::Glue)
         --NumOps;  // Ignore the glue operand.
 
       for (unsigned i = InlineAsm::Op_FirstOperand; i != NumOps;) {
         unsigned Flags =
           cast<ConstantSDNode>(Node->getOperand(i))->getZExtValue();

llvm/trunk/lib/CodeGen/SelectionDAG/ScheduleDAGRRList.cpp

void ScheduleDAGRRList::EmitNode(SUnit *SU) {
   case ISD::LIFETIME_END:
   case ISD::CopyToReg:
   case ISD::CopyFromReg:
   case ISD::EH_LABEL:
     // Noops don't affect the scoreboard state. Copies are likely to be
     // removed.
     return;
   case ISD::INLINEASM:
+  case ISD::INLINEASM_BR:
     // For inline asm, clear the pipeline state.
     HazardRec->Reset();
     return;
   }
   if (SU->isCall) {
     // Calls are scheduled with their preceding instructions. For bottom-up
     // scheduling, clear the pipeline state before emitting.
     HazardRec->Reset();
DelayForLiveRegsBottomUp(SUnit *SU, SmallVectorImpl<unsigned> &LRegs) {
   // then we are free to schedule it.
   for (SDep &Pred : SU->Preds) {
     if (Pred.isAssignedRegDep() && LiveRegDefs[Pred.getReg()] != SU)
       CheckForLiveRegDef(Pred.getSUnit(), Pred.getReg(), LiveRegDefs.get(),
                          RegAdded, LRegs, TRI);
   }
 
   for (SDNode *Node = SU->getNode(); Node; Node = Node->getGluedNode()) {
-    if (Node->getOpcode() == ISD::INLINEASM) {
+    if (Node->getOpcode() == ISD::INLINEASM ||
+        Node->getOpcode() == ISD::INLINEASM_BR) {
       // Inline asm can clobber physical defs.
       unsigned NumOps = Node->getNumOperands();
       if (Node->getOperand(NumOps-1).getValueType() == MVT::Glue)
         --NumOps;  // Ignore the glue operand.
 
       for (unsigned i = InlineAsm::Op_FirstOperand; i != NumOps;) {
         unsigned Flags =
           cast<ConstantSDNode>(Node->getOperand(i))->getZExtValue();

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

 namespace llvm {
 
 class AllocaInst;
 class AtomicCmpXchgInst;
 class AtomicRMWInst;
 class BasicBlock;
 class BranchInst;
 class CallInst;
+class CallBrInst;
 class CatchPadInst;
 class CatchReturnInst;
 class CatchSwitchInst;
 class CleanupPadInst;
 class CleanupReturnInst;
 class Constant;
 class ConstantInt;
 class ConstrainedFPIntrinsic;
void visitBitTestCase(BitTestBlock &BB,
                         MachineBasicBlock *SwitchBB);
   void visitJumpTable(JumpTable &JT);
   void visitJumpTableHeader(JumpTable &JT, JumpTableHeader &JTH,
                             MachineBasicBlock *SwitchBB);
 
 private:
   // These all get lowered before this pass.
   void visitInvoke(const InvokeInst &I);
+  void visitCallBr(const CallBrInst &I);
   void visitResume(const ResumeInst &I);
 
   void visitUnary(const User &I, unsigned Opcode);
   void visitFNeg(const User &I) { visitUnary(I, ISD::FNEG); }
 
   void visitBinary(const User &I, unsigned Opcode);
   void visitShift(const User &I, unsigned Opcode);
   void visitAdd(const User &I)  { visitBinary(I, ISD::ADD); }

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitInvoke(const InvokeInst &I) {
   addSuccessorWithProb(InvokeMBB, Return);
   for (auto &UnwindDest : UnwindDests) {
     UnwindDest.first->setIsEHPad();
     addSuccessorWithProb(InvokeMBB, UnwindDest.first, UnwindDest.second);
   }
   InvokeMBB->normalizeSuccProbs();
 
   // Drop into normal successor.
+  DAG.setRoot(DAG.getNode(ISD::BR, getCurSDLoc(), MVT::Other, getControlRoot(),
+                          DAG.getBasicBlock(Return)));
+}
+
+void SelectionDAGBuilder::visitCallBr(const CallBrInst &I) {
+  MachineBasicBlock *CallBrMBB = FuncInfo.MBB;
+
+  // Deopt bundles are lowered in LowerCallSiteWithDeoptBundle, and we don't
+  // have to do anything here to lower funclet bundles.
+  assert(!I.hasOperandBundlesOtherThan(
+             {LLVMContext::OB_deopt, LLVMContext::OB_funclet}) &&
+         "Cannot lower callbrs with arbitrary operand bundles yet!");
+
+  assert(isa<InlineAsm>(I.getCalledValue()) &&
+         "Only know how to handle inlineasm callbr");
+  visitInlineAsm(&I);
+
+  // Retrieve successors.
+  MachineBasicBlock *Return = FuncInfo.MBBMap[I.getDefaultDest()];
+
+  // Update successor info.
+  addSuccessorWithProb(CallBrMBB, Return);
+  for (unsigned i = 0, e = I.getNumIndirectDests(); i < e; ++i) {
+    MachineBasicBlock *Target = FuncInfo.MBBMap[I.getIndirectDest(i)];
+    addSuccessorWithProb(CallBrMBB, Target);
+  }
+  CallBrMBB->normalizeSuccProbs();
+
+  // Drop into default successor.
   DAG.setRoot(DAG.getNode(ISD::BR, getCurSDLoc(),
                           MVT::Other, getControlRoot(),
                           DAG.getBasicBlock(Return)));
 }
 
 void SelectionDAGBuilder::visitResume(const ResumeInst &RI) {
   llvm_unreachable("SelectionDAGBuilder shouldn't visit resume instructions!");
 }
for (auto &T : TargetConstraints) {
 
     // Compute the value type for each operand.
     if (OpInfo.Type == InlineAsm::isInput ||
         (OpInfo.Type == InlineAsm::isOutput && OpInfo.isIndirect)) {
       OpInfo.CallOperandVal = const_cast<Value *>(CS.getArgument(ArgNo++));
 
       // Process the call argument. BasicBlocks are labels, currently appearing
       // only in asm's.
-      if (const BasicBlock *BB = dyn_cast<BasicBlock>(OpInfo.CallOperandVal)) {
+      const Instruction *I = CS.getInstruction();
+      if (isa<CallBrInst>(I) &&
+          (ArgNo - 1) >= (cast<CallBrInst>(I)->getNumArgOperands() -
+                          cast<CallBrInst>(I)->getNumIndirectDests())) {
+        const auto *BA = cast<BlockAddress>(OpInfo.CallOperandVal);
+        EVT VT = TLI.getValueType(DAG.getDataLayout(), BA->getType(), true);
+        OpInfo.CallOperand = DAG.getTargetBlockAddress(BA, VT);
+      } else if (const auto *BB = dyn_cast<BasicBlock>(OpInfo.CallOperandVal)) {
         OpInfo.CallOperand = DAG.getBasicBlock(FuncInfo.MBBMap[BB]);
       } else {
         OpInfo.CallOperand = getValue(OpInfo.CallOperandVal);
       }
 
       OpInfo.ConstraintVT =
           OpInfo
               .getCallOperandValEVT(*DAG.getContext(), TLI, DAG.getDataLayout())
case InlineAsm::isClobber:
       break;
     }
   }
 
   // Finish up input operands.  Set the input chain and add the flag last.
   AsmNodeOperands[InlineAsm::Op_InputChain] = Chain;
   if (Flag.getNode()) AsmNodeOperands.push_back(Flag);
 
-  Chain = DAG.getNode(ISD::INLINEASM, getCurSDLoc(),
+  unsigned ISDOpc = isa<CallBrInst>(CS.getInstruction()) ? ISD::INLINEASM_BR : ISD::INLINEASM;
+  Chain = DAG.getNode(ISDOpc, getCurSDLoc(),
                       DAG.getVTList(MVT::Other, MVT::Glue), AsmNodeOperands);
   Flag = Chain.getValue(1);
 
   // Do additional work to generate outputs.
 
   SmallVector<EVT, 1> ResultVTs;
   SmallVector<SDValue, 1> ResultValues;
   SmallVector<SDValue, 8> OutChains;

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::MCSymbol:                   return "MCSymbol";
   case ISD::TargetBlockAddress:         return "TargetBlockAddress";
 
   case ISD::CopyToReg:                  return "CopyToReg";
   case ISD::CopyFromReg:                return "CopyFromReg";
   case ISD::UNDEF:                      return "undef";
   case ISD::MERGE_VALUES:               return "merge_values";
   case ISD::INLINEASM:                  return "inlineasm";
+  case ISD::INLINEASM_BR:               return "inlineasm_br";
   case ISD::EH_LABEL:                   return "eh_label";
   case ISD::HANDLENODE:                 return "handlenode";
 
   // Unary operators
   case ISD::FABS:                       return "fabs";
   case ISD::FMINNUM:                    return "fminnum";
   case ISD::STRICT_FMINNUM:             return "strict_fminnum";
   case ISD::FMAXNUM:                    return "fmaxnum";

llvm/trunk/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp

while (VT == MVT::Glue) {
     // the chain, HandleMergeInputChains will not consider it.  Because of
     // this, we cannot ignore chains in this predicate.
     IgnoreChains = false;
   }
 
   return !findNonImmUse(Root, N.getNode(), U, IgnoreChains);
 }
 
-void SelectionDAGISel::Select_INLINEASM(SDNode *N) {
+void SelectionDAGISel::Select_INLINEASM(SDNode *N, bool Branch) {
   SDLoc DL(N);
 
   std::vector<SDValue> Ops(N->op_begin(), N->op_end());
   SelectInlineAsmMemoryOperands(Ops, DL);
 
   const EVT VTs[] = {MVT::Other, MVT::Glue};
-  SDValue New = CurDAG->getNode(ISD::INLINEASM, DL, VTs, Ops);
+  SDValue New = CurDAG->getNode(Branch ? ISD::INLINEASM_BR : ISD::INLINEASM, DL, VTs, Ops);
   New->setNodeId(-1);
   ReplaceUses(N, New.getNode());
   CurDAG->RemoveDeadNode(N);
 }
 
 void SelectionDAGISel::Select_READ_REGISTER(SDNode *Op) {
   SDLoc dl(Op);
   MDNodeSDNode *MD = dyn_cast<MDNodeSDNode>(Op->getOperand(1));
case ISD::LIFETIME_END:
     NodeToMatch->setNodeId(-1); // Mark selected.
     return;
   case ISD::AssertSext:
   case ISD::AssertZext:
     ReplaceUses(SDValue(NodeToMatch, 0), NodeToMatch->getOperand(0));
     CurDAG->RemoveDeadNode(NodeToMatch);
     return;
   case ISD::INLINEASM:
-    Select_INLINEASM(NodeToMatch);
+  case ISD::INLINEASM_BR:
+    Select_INLINEASM(NodeToMatch,
+                     NodeToMatch->getOpcode() == ISD::INLINEASM_BR);
     return;
   case ISD::READ_REGISTER:
     Select_READ_REGISTER(NodeToMatch);
     return;
   case ISD::WRITE_REGISTER:
     Select_WRITE_REGISTER(NodeToMatch);
     return;
   case ISD::UNDEF:

llvm/trunk/lib/CodeGen/SelectionDAG/TargetLowering.cpp

void TargetLowering::LowerAsmOperandForConstraint(SDValue Op,
                                                   SelectionDAG &DAG) const {
 
   if (Constraint.length() > 1) return;
 
   char ConstraintLetter = Constraint[0];
   switch (ConstraintLetter) {
   default: break;
   case 'X':     // Allows any operand; labels (basic block) use this.
-    if (Op.getOpcode() == ISD::BasicBlock) {
+    if (Op.getOpcode() == ISD::BasicBlock ||
+        Op.getOpcode() == ISD::TargetBlockAddress) {
       Ops.push_back(Op);
       return;
     }
     LLVM_FALLTHROUGH;
   case 'i':    // Simple Integer or Relocatable Constant
   case 'n':    // Simple Integer
   case 's': {  // Relocatable Constant
     // These operands are interested in values of the form (GV+C), where C may
if (OpInfo.ConstraintCode == "X" && OpInfo.CallOperandVal) {
     // that matches labels).  For Functions, the type here is the type of
     // the result, which is not what we want to look at; leave them alone.
     Value *v = OpInfo.CallOperandVal;
     if (isa<BasicBlock>(v) || isa<ConstantInt>(v) || isa<Function>(v)) {
       OpInfo.CallOperandVal = v;
       return;
     }
 
+    if (Op.getNode() && Op.getOpcode() == ISD::TargetBlockAddress)
+      return;
+
     // Otherwise, try to resolve it to something we know about by looking at
     // the actual operand type.
     if (const char *Repl = LowerXConstraint(OpInfo.ConstraintVT)) {
       OpInfo.ConstraintCode = Repl;
       OpInfo.ConstraintType = getConstraintType(OpInfo.ConstraintCode);
     }
   }
 }

llvm/trunk/lib/CodeGen/TargetLoweringBase.cpp

 #include "llvm/IR/Instruction.def"
   };
   switch (static_cast<InstructionOpcodes>(Opcode)) {
   case Ret:            return 0;
   case Br:             return 0;
   case Switch:         return 0;
   case IndirectBr:     return 0;
   case Invoke:         return 0;
+  case CallBr:         return 0;
   case Resume:         return 0;
   case Unreachable:    return 0;
   case CleanupRet:     return 0;
   case CatchRet:       return 0;
   case CatchPad:       return 0;
   case CatchSwitch:    return 0;
   case CleanupPad:     return 0;
   case FNeg:           return ISD::FNEG;

llvm/trunk/lib/IR/AsmWriter.cpp

if (PAL.hasAttributes(AttributeList::FunctionIndex))
       Out << " #" << Machine.getAttributeGroupSlot(PAL.getFnAttributes());
 
     writeOperandBundles(II);
 
     Out << "\n          to ";
     writeOperand(II->getNormalDest(), true);
     Out << " unwind ";
     writeOperand(II->getUnwindDest(), true);
+  } else if (const CallBrInst *CBI = dyn_cast<CallBrInst>(&I)) {
+    Operand = CBI->getCalledValue();
+    FunctionType *FTy = CBI->getFunctionType();
+    Type *RetTy = FTy->getReturnType();
+    const AttributeList &PAL = CBI->getAttributes();
+
+    // Print the calling convention being used.
+    if (CBI->getCallingConv() != CallingConv::C) {
+      Out << " ";
+      PrintCallingConv(CBI->getCallingConv(), Out);
+    }
+
+    if (PAL.hasAttributes(AttributeList::ReturnIndex))
+      Out << ' ' << PAL.getAsString(AttributeList::ReturnIndex);
+
+    // If possible, print out the short form of the callbr instruction. We can
+    // only do this if the first argument is a pointer to a nonvararg function,
+    // and if the return type is not a pointer to a function.
+    //
+    Out << ' ';
+    TypePrinter.print(FTy->isVarArg() ? FTy : RetTy, Out);
+    Out << ' ';
+    writeOperand(Operand, false);
+    Out << '(';
+    for (unsigned op = 0, Eop = CBI->getNumArgOperands(); op < Eop; ++op) {
+      if (op)
+        Out << ", ";
+      writeParamOperand(CBI->getArgOperand(op), PAL.getParamAttributes(op));
+    }
+
+    Out << ')';
+    if (PAL.hasAttributes(AttributeList::FunctionIndex))
+      Out << " #" << Machine.getAttributeGroupSlot(PAL.getFnAttributes());
+
+    writeOperandBundles(CBI);
+
+    Out << "\n          to ";
+    writeOperand(CBI->getDefaultDest(), true);
+    Out << " [";
+    for (unsigned i = 0, e = CBI->getNumIndirectDests(); i != e; ++i) {
+      if (i != 0)
+        Out << ", ";
+      writeOperand(CBI->getIndirectDest(i), true);
+    }
+    Out << ']';
   } else if (const AllocaInst *AI = dyn_cast<AllocaInst>(&I)) {
     Out << ' ';
     if (AI->isUsedWithInAlloca())
       Out << "inalloca ";
     if (AI->isSwiftError())
       Out << "swifterror ";
     TypePrinter.print(AI->getAllocatedType(), Out);
 

llvm/trunk/lib/IR/Instruction.cpp

const char *Instruction::getOpcodeName(unsigned OpCode) {
   case IndirectBr: return "indirectbr";
   case Invoke: return "invoke";
   case Resume: return "resume";
   case Unreachable: return "unreachable";
   case CleanupRet: return "cleanupret";
   case CatchRet: return "catchret";
   case CatchPad: return "catchpad";
   case CatchSwitch: return "catchswitch";
+  case CallBr: return "callbr";
 
   // Standard unary operators...
   case FNeg: return "fneg";
 
   // Standard binary operators...
   case Add: return "add";
   case FAdd: return "fadd";
   case Sub: return "sub";
if (const CallInst *CI = dyn_cast<CallInst>(I1))
     return CI->isTailCall() == cast<CallInst>(I2)->isTailCall() &&
            CI->getCallingConv() == cast<CallInst>(I2)->getCallingConv() &&
            CI->getAttributes() == cast<CallInst>(I2)->getAttributes() &&
            CI->hasIdenticalOperandBundleSchema(*cast<CallInst>(I2));
   if (const InvokeInst *CI = dyn_cast<InvokeInst>(I1))
     return CI->getCallingConv() == cast<InvokeInst>(I2)->getCallingConv() &&
            CI->getAttributes() == cast<InvokeInst>(I2)->getAttributes() &&
            CI->hasIdenticalOperandBundleSchema(*cast<InvokeInst>(I2));
+  if (const CallBrInst *CI = dyn_cast<CallBrInst>(I1))
+    return CI->getCallingConv() == cast<CallBrInst>(I2)->getCallingConv() &&
+           CI->getAttributes() == cast<CallBrInst>(I2)->getAttributes() &&
+           CI->hasIdenticalOperandBundleSchema(*cast<CallBrInst>(I2));
   if (const InsertValueInst *IVI = dyn_cast<InsertValueInst>(I1))
     return IVI->getIndices() == cast<InsertValueInst>(I2)->getIndices();
   if (const ExtractValueInst *EVI = dyn_cast<ExtractValueInst>(I1))
     return EVI->getIndices() == cast<ExtractValueInst>(I2)->getIndices();
   if (const FenceInst *FI = dyn_cast<FenceInst>(I1))
     return FI->getOrdering() == cast<FenceInst>(I2)->getOrdering() &&
            FI->getSyncScopeID() == cast<FenceInst>(I2)->getSyncScopeID();
   if (const AtomicCmpXchgInst *CXI = dyn_cast<AtomicCmpXchgInst>(I1))
bool Instruction::mayReadFromMemory() const {
   case Instruction::Fence: // FIXME: refine definition of mayReadFromMemory
   case Instruction::AtomicCmpXchg:
   case Instruction::AtomicRMW:
   case Instruction::CatchPad:
   case Instruction::CatchRet:
     return true;
   case Instruction::Call:
   case Instruction::Invoke:
+  case Instruction::CallBr:
     return !cast<CallBase>(this)->doesNotAccessMemory();
   case Instruction::Store:
     return !cast<StoreInst>(this)->isUnordered();
   }
 }
 
 bool Instruction::mayWriteToMemory() const {
   switch (getOpcode()) {
   default: return false;
   case Instruction::Fence: // FIXME: refine definition of mayWriteToMemory
   case Instruction::Store:
   case Instruction::VAArg:
   case Instruction::AtomicCmpXchg:
   case Instruction::AtomicRMW:
   case Instruction::CatchPad:
   case Instruction::CatchRet:
     return true;
   case Instruction::Call:
   case Instruction::Invoke:
+  case Instruction::CallBr:
     return !cast<CallBase>(this)->onlyReadsMemory();
   case Instruction::Load:
     return !cast<LoadInst>(this)->isUnordered();
   }
 }
 
 bool Instruction::isAtomic() const {
   switch (getOpcode()) {
for (unsigned i = 1; i < ProfileData->getNumOperands(); i += 2) {
       Vals.push_back(MDB.createConstant(
           ConstantInt::get(Type::getInt64Ty(getContext()),
                            Val.udiv(APT).getLimitedValue())));
     }
   setMetadata(LLVMContext::MD_prof, MDNode::get(getContext(), Vals));
 }
 
 void Instruction::setProfWeight(uint64_t W) {
-  assert((isa<CallInst>(this) || isa<InvokeInst>(this)) &&
-         "Can only set weights for call and invoke instrucitons");
+  assert(isa<CallBase>(this) &&
+         "Can only set weights for call like instructions");
   SmallVector<uint32_t, 1> Weights;
   Weights.push_back(W);
   MDBuilder MDB(getContext());
   setMetadata(LLVMContext::MD_prof, MDB.createBranchWeights(Weights));
 }

llvm/trunk/lib/IR/Instructions.cpp

 }
 
 //===----------------------------------------------------------------------===//
 //                        CallBase Implementation
 //===----------------------------------------------------------------------===//
 
 Function *CallBase::getCaller() { return getParent()->getParent(); }
 
+unsigned CallBase::getNumSubclassExtraOperandsDynamic() const {
+  assert(getOpcode() == Instruction::CallBr && "Unexpected opcode!");
+  return cast<CallBrInst>(this)->getNumIndirectDests() + 1;
+}
+
 bool CallBase::isIndirectCall() const {
   const Value *V = getCalledValue();
   if (isa<Function>(V) || isa<Constant>(V))
     return false;
   if (const CallInst *CI = dyn_cast<CallInst>(this))
     if (CI->isInlineAsm())
       return false;
   return true;

 }
 
 
 LandingPadInst *InvokeInst::getLandingPadInst() const {
   return cast<LandingPadInst>(getUnwindDest()->getFirstNonPHI());
 }
 
 //===----------------------------------------------------------------------===//
+//                        CallBrInst Implementation
+//===----------------------------------------------------------------------===//
+
+void CallBrInst::init(FunctionType *FTy, Value *Fn, BasicBlock *Fallthrough,
+                      ArrayRef<BasicBlock *> IndirectDests,
+                      ArrayRef<Value *> Args,
+                      ArrayRef<OperandBundleDef> Bundles,
+                      const Twine &NameStr) {
+  this->FTy = FTy;
+
+  assert((int)getNumOperands() ==
+             ComputeNumOperands(Args.size(), IndirectDests.size(),
+                                CountBundleInputs(Bundles)) &&
+         "NumOperands not set up?");
+  NumIndirectDests = IndirectDests.size();
+  setDefaultDest(Fallthrough);
+  for (unsigned i = 0; i != NumIndirectDests; ++i)
+    setIndirectDest(i, IndirectDests[i]);
+  setCalledOperand(Fn);
+
+#ifndef NDEBUG
+  assert(((Args.size() == FTy->getNumParams()) ||
+          (FTy->isVarArg() && Args.size() > FTy->getNumParams())) &&
+         "Calling a function with bad signature");
+
+  for (unsigned i = 0, e = Args.size(); i != e; i++)
+    assert((i >= FTy->getNumParams() ||
+            FTy->getParamType(i) == Args[i]->getType()) &&
+           "Calling a function with a bad signature!");
+#endif
+
+  std::copy(Args.begin(), Args.end(), op_begin());
+
+  auto It = populateBundleOperandInfos(Bundles, Args.size());
+  (void)It;
+  assert(It + 2 + IndirectDests.size() == op_end() && "Should add up!");
+
+  setName(NameStr);
+}
+
+CallBrInst::CallBrInst(const CallBrInst &CBI)
+    : CallBase(CBI.Attrs, CBI.FTy, CBI.getType(), Instruction::CallBr,
+               OperandTraits<CallBase>::op_end(this) - CBI.getNumOperands(),
+               CBI.getNumOperands()) {
+  setCallingConv(CBI.getCallingConv());
+  std::copy(CBI.op_begin(), CBI.op_end(), op_begin());
+  std::copy(CBI.bundle_op_info_begin(), CBI.bundle_op_info_end(),
+            bundle_op_info_begin());
+  SubclassOptionalData = CBI.SubclassOptionalData;
+  NumIndirectDests = CBI.NumIndirectDests;
+}
+
+CallBrInst *CallBrInst::Create(CallBrInst *CBI, ArrayRef<OperandBundleDef> OpB,
+                               Instruction *InsertPt) {
+  std::vector<Value *> Args(CBI->arg_begin(), CBI->arg_end());
+
+  auto *NewCBI = CallBrInst::Create(CBI->getFunctionType(),
+                                    CBI->getCalledValue(),
+                                    CBI->getDefaultDest(),
+                                    CBI->getIndirectDests(),
+                                    Args, OpB, CBI->getName(), InsertPt);
+  NewCBI->setCallingConv(CBI->getCallingConv());
+  NewCBI->SubclassOptionalData = CBI->SubclassOptionalData;
+  NewCBI->setAttributes(CBI->getAttributes());
+  NewCBI->setDebugLoc(CBI->getDebugLoc());
+  NewCBI->NumIndirectDests = CBI->NumIndirectDests;
+  return NewCBI;
+}
+
+//===----------------------------------------------------------------------===//
 //                        ReturnInst Implementation
 //===----------------------------------------------------------------------===//
 
 ReturnInst::ReturnInst(const ReturnInst &RI)
     : Instruction(Type::getVoidTy(RI.getContext()), Instruction::Ret,
                   OperandTraits<ReturnInst>::op_end(this) - RI.getNumOperands(),
                   RI.getNumOperands()) {
   if (RI.getNumOperands())

 InvokeInst *InvokeInst::cloneImpl() const {
   if (hasOperandBundles()) {
     unsigned DescriptorBytes = getNumOperandBundles() * sizeof(BundleOpInfo);
     return new(getNumOperands(), DescriptorBytes) InvokeInst(*this);
   }
   return new(getNumOperands()) InvokeInst(*this);
 }
 
+CallBrInst *CallBrInst::cloneImpl() const {
+  if (hasOperandBundles()) {
+    unsigned DescriptorBytes = getNumOperandBundles() * sizeof(BundleOpInfo);
+    return new (getNumOperands(), DescriptorBytes) CallBrInst(*this);
+  }
+  return new (getNumOperands()) CallBrInst(*this);
+}
+
 ResumeInst *ResumeInst::cloneImpl() const { return new (1) ResumeInst(*this); }
 
 CleanupReturnInst *CleanupReturnInst::cloneImpl() const {
   return new (getNumOperands()) CleanupReturnInst(*this);
 }
 
 CatchReturnInst *CatchReturnInst::cloneImpl() const {
   return new (getNumOperands()) CatchReturnInst(*this);

llvm/trunk/lib/IR/Value.cpp

 Value::Value(Type *ty, unsigned scid)
     : VTy(checkType(ty)), UseList(nullptr), SubclassID(scid),
       HasValueHandle(0), SubclassOptionalData(0), SubclassData(0),
       NumUserOperands(0), IsUsedByMD(false), HasName(false) {
   static_assert(ConstantFirstVal == 0, "!(SubclassID < ConstantFirstVal)");
   // FIXME: Why isn't this in the subclass gunk??
   // Note, we cannot call isa<CallInst> before the CallInst has been
   // constructed.
-  if (SubclassID == Instruction::Call || SubclassID == Instruction::Invoke)
+  if (SubclassID == Instruction::Call || SubclassID == Instruction::Invoke ||
+      SubclassID == Instruction::CallBr)
     assert((VTy->isFirstClassType() || VTy->isVoidTy() || VTy->isStructTy()) &&
            "invalid CallInst type!");
   else if (SubclassID != BasicBlockVal &&
            (/*SubclassID < ConstantFirstVal ||*/ SubclassID > ConstantLastVal))
     assert((VTy->isFirstClassType() || VTy->isVoidTy()) &&
            "Cannot create non-first-class values except for constants!");
   static_assert(sizeof(Value) == 2 * sizeof(void *) + 2 * sizeof(unsigned),
                 "Value too big");

llvm/trunk/lib/IR/Verifier.cpp

#include "llvm/IR/Metadata.def"
   void visitStoreInst(StoreInst &SI);
   void verifyDominatesUse(Instruction &I, unsigned i);
   void visitInstruction(Instruction &I);
   void visitTerminator(Instruction &I);
   void visitBranchInst(BranchInst &BI);
   void visitReturnInst(ReturnInst &RI);
   void visitSwitchInst(SwitchInst &SI);
   void visitIndirectBrInst(IndirectBrInst &BI);
+  void visitCallBrInst(CallBrInst &CBI);
   void visitSelectInst(SelectInst &SI);
   void visitUserOp1(Instruction &I);
   void visitUserOp2(Instruction &I) { visitUserOp1(I); }
   void visitIntrinsicCall(Intrinsic::ID ID, CallBase &Call);
   void visitConstrainedFPIntrinsic(ConstrainedFPIntrinsic &FPI);
   void visitDbgIntrinsic(StringRef Kind, DbgVariableIntrinsic &DII);
   void visitDbgLabelIntrinsic(StringRef Kind, DbgLabelInst &DLI);
   void visitAtomicCmpXchgInst(AtomicCmpXchgInst &CXI);
Assert(BI.getAddress()->getType()->isPointerTy(),
          "Indirectbr operand must have pointer type!", &BI);
   for (unsigned i = 0, e = BI.getNumDestinations(); i != e; ++i)
     Assert(BI.getDestination(i)->getType()->isLabelTy(),
            "Indirectbr destinations must all have pointer type!", &BI);
 
   visitTerminator(BI);
 }
 
+void Verifier::visitCallBrInst(CallBrInst &CBI) {
+  Assert(CBI.isInlineAsm(), "Callbr is currently only used for asm-goto!",
+         &CBI);
+  Assert(CBI.getType()->isVoidTy(), "Callbr return value is not supported!",
+         &CBI);
+  for (unsigned i = 0, e = CBI.getNumSuccessors(); i != e; ++i)
+    Assert(CBI.getSuccessor(i)->getType()->isLabelTy(),
+           "Callbr successors must all have pointer type!", &CBI);
+  for (unsigned i = 0, e = CBI.getNumOperands(); i != e; ++i) {
+    Assert(i >= CBI.getNumArgOperands() || !isa<BasicBlock>(CBI.getOperand(i)),
+           "Using an unescaped label as a callbr argument!", &CBI);
+    if (isa<BasicBlock>(CBI.getOperand(i)))
+      for (unsigned j = i + 1; j != e; ++j)
+        Assert(CBI.getOperand(i) != CBI.getOperand(j),
+               "Duplicate callbr destination!", &CBI);
+  }
+
+  visitTerminator(CBI);
+}
+
 void Verifier::visitSelectInst(SelectInst &SI) {
   Assert(!SelectInst::areInvalidOperands(SI.getOperand(0), SI.getOperand(1),
                                          SI.getOperand(2)),
          "Invalid operands for select instruction!", &SI);
 
   Assert(SI.getTrueValue()->getType() == SI.getType(),
          "Select values must have same type as select instruction!", &SI);
   visitInstruction(SI);

llvm/trunk/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

if (isa<MemSDNode>(N))
     return false;
 
   switch (N->getOpcode()) {
   case ISD::CopyToReg:
   case ISD::SELECT:
   case ISD::FDIV:
   case ISD::FREM:
   case ISD::INLINEASM:
+  case ISD::INLINEASM_BR:
   case AMDGPUISD::INTERP_P1:
   case AMDGPUISD::INTERP_P2:
   case AMDGPUISD::DIV_SCALE:
 
   // TODO: Should really be looking at the users of the bitcast. These are
   // problematic because bitcasts are used to legalize all stores to integer
   // types.
   case ISD::BITCAST:

llvm/trunk/lib/Target/AMDGPU/SIISelLowering.cpp

 }
 
 LLVM_ATTRIBUTE_UNUSED
 static bool isCopyFromRegOfInlineAsm(const SDNode *N) {
   assert(N->getOpcode() == ISD::CopyFromReg);
   do {
     // Follow the chain until we find an INLINEASM node.
     N = N->getOperand(0).getNode();
-    if (N->getOpcode() == ISD::INLINEASM)
+    if (N->getOpcode() == ISD::INLINEASM ||
+        N->getOpcode() == ISD::INLINEASM_BR)
       return true;
   } while (N->getOpcode() == ISD::CopyFromReg);
   return false;
 }
 
 bool SITargetLowering::isSDNodeSourceOfDivergence(const SDNode * N,
   FunctionLoweringInfo * FLI, LegacyDivergenceAnalysis * KDA) const
 {

llvm/trunk/lib/Target/AMDGPU/SIInstrInfo.cpp

unsigned SIInstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   switch (Opc) {
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::DBG_VALUE:
   case TargetOpcode::EH_LABEL:
     return 0;
   case TargetOpcode::BUNDLE:
     return getInstBundleSize(MI);
-  case TargetOpcode::INLINEASM: {
+  case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR: {
     const MachineFunction *MF = MI.getParent()->getParent();
     const char *AsmStr = MI.getOperand(0).getSymbolName();
     return getInlineAsmLength(AsmStr, *MF->getTarget().getMCAsmInfo());
   }
   default:
     return DescSize;
   }
 }

llvm/trunk/lib/Target/ARM/ARMISelDAGToDAG.cpp

case ISD::WRITE_REGISTER:
     if (tryWriteRegister(N))
       return;
     break;
   case ISD::READ_REGISTER:
     if (tryReadRegister(N))
       return;
     break;
   case ISD::INLINEASM:
+  case ISD::INLINEASM_BR:
     if (tryInlineAsm(N))
       return;
     break;
   case ISD::XOR:
     // Select special operations if XOR node forms integer ABS pattern
     if (tryABSOp(N))
       return;
     // Other cases are autogenerated.
for(unsigned i = 0, e = N->getGluedNode() ? NumOps - 1 : NumOps; i < e; ++i) {
     }
   }
 
   if (Glue.getNode())
     AsmNodeOperands.push_back(Glue);
   if (!Changed)
     return false;
 
-  SDValue New = CurDAG->getNode(ISD::INLINEASM, SDLoc(N),
+  SDValue New = CurDAG->getNode(N->getOpcode(), SDLoc(N),
       CurDAG->getVTList(MVT::Other, MVT::Glue), AsmNodeOperands);
   New->setNodeId(-1);
   ReplaceNode(N, New.getNode());
   return true;
 }
 
 
 bool ARMDAGToDAGISel::

llvm/trunk/lib/Target/AVR/AVRInstrInfo.cpp

default: {
     const MCInstrDesc &Desc = get(Opcode);
     return Desc.getSize();
   }
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::DBG_VALUE:
     return 0;
-  case TargetOpcode::INLINEASM: {
+  case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR: {
     const MachineFunction &MF = *MI.getParent()->getParent();
     const AVRTargetMachine &TM = static_cast<const AVRTargetMachine&>(MF.getTarget());
     const AVRSubtarget &STI = MF.getSubtarget<AVRSubtarget>();
     const TargetInstrInfo &TII = *STI.getInstrInfo();
 
     return TII.getInlineAsmLength(MI.getOperand(0).getSymbolName(),
                                   *TM.getMCAsmInfo());
   }

llvm/trunk/lib/Target/Hexagon/HexagonISelLowering.cpp

 
 SDValue
 HexagonTargetLowering::LowerINLINEASM(SDValue Op, SelectionDAG &DAG) const {
   MachineFunction &MF = DAG.getMachineFunction();
   auto &HMFI = *MF.getInfo<HexagonMachineFunctionInfo>();
   const HexagonRegisterInfo &HRI = *Subtarget.getRegisterInfo();
   unsigned LR = HRI.getRARegister();
 
-  if (Op.getOpcode() != ISD::INLINEASM || HMFI.hasClobberLR())
+  if ((Op.getOpcode() != ISD::INLINEASM &&
+       Op.getOpcode() != ISD::INLINEASM_BR) || HMFI.hasClobberLR())
     return Op;
 
   unsigned NumOps = Op.getNumOperands();
   if (Op.getOperand(NumOps-1).getValueType() == MVT::Glue)
     --NumOps;  // Ignore the flag operand.
 
   for (unsigned i = InlineAsm::Op_FirstOperand; i != NumOps;) {
     unsigned Flags = cast<ConstantSDNode>(Op.getOperand(i))->getZExtValue();
HexagonTargetLowering::HexagonTargetLowering(const TargetMachine &TM,
   setOperationAction(ISD::ConstantFP,           MVT::f32,   Legal);
   setOperationAction(ISD::ConstantFP,           MVT::f64,   Legal);
   setOperationAction(ISD::TRAP,                 MVT::Other, Legal);
   setOperationAction(ISD::ConstantPool,         MVT::i32,   Custom);
   setOperationAction(ISD::JumpTable,            MVT::i32,   Custom);
   setOperationAction(ISD::BUILD_PAIR,           MVT::i64,   Expand);
   setOperationAction(ISD::SIGN_EXTEND_INREG,    MVT::i1,    Expand);
   setOperationAction(ISD::INLINEASM,            MVT::Other, Custom);
+  setOperationAction(ISD::INLINEASM_BR,         MVT::Other, Custom);
   setOperationAction(ISD::PREFETCH,             MVT::Other, Custom);
   setOperationAction(ISD::READCYCLECOUNTER,     MVT::i64,   Custom);
   setOperationAction(ISD::INTRINSIC_VOID,       MVT::Other, Custom);
   setOperationAction(ISD::EH_RETURN,            MVT::Other, Custom);
   setOperationAction(ISD::GLOBAL_OFFSET_TABLE,  MVT::i32,   Custom);
   setOperationAction(ISD::GlobalTLSAddress,     MVT::i32,   Custom);
   setOperationAction(ISD::ATOMIC_FENCE,         MVT::Other, Custom);
 
HexagonTargetLowering::LowerEH_RETURN(SDValue Op, SelectionDAG &DAG) const {
   return DAG.getNode(HexagonISD::EH_RETURN, dl, MVT::Other, Chain);
 }
 
 SDValue
 HexagonTargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   unsigned Opc = Op.getOpcode();
 
   // Handle INLINEASM first.
-  if (Opc == ISD::INLINEASM)
+  if (Opc == ISD::INLINEASM || Opc == ISD::INLINEASM_BR)
     return LowerINLINEASM(Op, DAG);
 
   if (isHvxOperation(Op)) {
     // If HVX lowering returns nothing, try the default lowering.
     if (SDValue V = LowerHvxOperation(Op, DAG))
       return V;
   }
 

llvm/trunk/lib/Target/Hexagon/HexagonMachineScheduler.cpp

default:
     break;
   case TargetOpcode::EXTRACT_SUBREG:
   case TargetOpcode::INSERT_SUBREG:
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::COPY:
   case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR:
     break;
   }
 
   MachineBasicBlock *MBB = SU->getInstr()->getParent();
   auto &QST = MBB->getParent()->getSubtarget<HexagonSubtarget>();
   const auto &QII = *QST.getInstrInfo();
 
   // Now see if there are no other dependencies to instructions already
bool VLIWResourceModel::reserveResources(SUnit *SU, bool IsTop) {
   case TargetOpcode::SUBREG_TO_REG:
   case TargetOpcode::REG_SEQUENCE:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::CFI_INSTRUCTION:
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::COPY:
   case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR:
     break;
   }
   Packet.push_back(SU);
 
 #ifndef NDEBUG
   LLVM_DEBUG(dbgs() << "Packet[" << TotalPackets << "]:\n");
   for (unsigned i = 0, e = Packet.size(); i != e; ++i) {
     LLVM_DEBUG(dbgs() << "\t[" << i << "] SU(");

llvm/trunk/lib/Target/MSP430/MSP430InstrInfo.cpp

unsigned MSP430InstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
 
   switch (Desc.getOpcode()) {
   case TargetOpcode::CFI_INSTRUCTION:
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::DBG_VALUE:
     return 0;
-  case TargetOpcode::INLINEASM: {
+  case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR: {
     const MachineFunction *MF = MI.getParent()->getParent();
     const TargetInstrInfo &TII = *MF->getSubtarget().getInstrInfo();
     return TII.getInlineAsmLength(MI.getOperand(0).getSymbolName(),
                                   *MF->getTarget().getMCAsmInfo());
   }
   }
 
   return Desc.getSize();
 }

llvm/trunk/lib/Target/Mips/MipsInstrInfo.cpp

bool MipsInstrInfo::HasForbiddenSlot(const MachineInstr &MI) const {
   return (MI.getDesc().TSFlags & MipsII::HasForbiddenSlot) != 0;
 }
 
 /// Return the number of bytes of code the specified instruction may be.
 unsigned MipsInstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   switch (MI.getOpcode()) {
   default:
     return MI.getDesc().getSize();
-  case  TargetOpcode::INLINEASM: {       // Inline Asm: Variable size.
+  case  TargetOpcode::INLINEASM:
+  case  TargetOpcode::INLINEASM_BR: {       // Inline Asm: Variable size.
     const MachineFunction *MF = MI.getParent()->getParent();
     const char *AsmStr = MI.getOperand(0).getSymbolName();
     return getInlineAsmLength(AsmStr, *MF->getTarget().getMCAsmInfo());
   }
   case Mips::CONSTPOOL_ENTRY:
     // If this machine instr is a constant pool entry, its size is recorded as
     // operand #2.
     return MI.getOperand(2).getImm();

llvm/trunk/lib/Target/PowerPC/PPCRegisterInfo.cpp

PPCRegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
   // Convert into indexed form of the instruction:
   //
   //   sth 0:rA, 1:imm 2:(rB) ==> sthx 0:rA, 2:rB, 1:r0
   //   addi 0:rA 1:rB, 2, imm ==> add 0:rA, 1:rB, 2:r0
   unsigned OperandBase;
 
   if (noImmForm)
     OperandBase = 1;
-  else if (OpC != TargetOpcode::INLINEASM) {
+  else if (OpC != TargetOpcode::INLINEASM &&
+           OpC != TargetOpcode::INLINEASM_BR) {
     assert(ImmToIdxMap.count(OpC) &&
            "No indexed form of load or store available!");
     unsigned NewOpcode = ImmToIdxMap.find(OpC)->second;
     MI.setDesc(TII.get(NewOpcode));
     OperandBase = 1;
   } else {
     OperandBase = OffsetOperandNo;
   }

llvm/trunk/lib/Target/RISCV/RISCVInstrInfo.cpp

unsigned RISCVInstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   case TargetOpcode::EH_LABEL:
   case TargetOpcode::IMPLICIT_DEF:
   case TargetOpcode::KILL:
   case TargetOpcode::DBG_VALUE:
     return 0;
   case RISCV::PseudoCALL:
   case RISCV::PseudoTAIL:
     return 8;
-  case TargetOpcode::INLINEASM: {
+  case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR: {
     const MachineFunction &MF = *MI.getParent()->getParent();
     const auto &TM = static_cast<const RISCVTargetMachine &>(MF.getTarget());
     return getInlineAsmLength(MI.getOperand(0).getSymbolName(),
                               *TM.getMCAsmInfo());
   }
   }
 }
 

llvm/trunk/lib/Target/Sparc/SparcISelDAGToDAG.cpp

bool SparcDAGToDAGISel::tryInlineAsm(SDNode *N){
 
   if (Glue.getNode())
     AsmNodeOperands.push_back(Glue);
   if (!Changed)
     return false;
 
   SelectInlineAsmMemoryOperands(AsmNodeOperands, SDLoc(N));
 
-  SDValue New = CurDAG->getNode(ISD::INLINEASM, SDLoc(N),
+  SDValue New = CurDAG->getNode(N->getOpcode(), SDLoc(N),
       CurDAG->getVTList(MVT::Other, MVT::Glue), AsmNodeOperands);
   New->setNodeId(-1);
   ReplaceNode(N, New.getNode());
   return true;
 }
 
 void SparcDAGToDAGISel::Select(SDNode *N) {
   SDLoc dl(N);
   if (N->isMachineOpcode()) {
     N->setNodeId(-1);
     return;   // Already selected.
   }
 
   switch (N->getOpcode()) {
   default: break;
-  case ISD::INLINEASM: {
+  case ISD::INLINEASM:
+  case ISD::INLINEASM_BR: {
     if (tryInlineAsm(N))
       return;
     break;
   }
   case SPISD::GLOBAL_BASE_REG:
     ReplaceNode(N, getGlobalBaseReg());
     return;
 

llvm/trunk/lib/Target/X86/X86AsmPrinter.cpp

case MachineOperand::MO_Immediate:
     O << MO.getImm();
     return;
 
   case MachineOperand::MO_GlobalAddress: {
     if (AsmVariant == 0) O << '$';
     printSymbolOperand(P, MO, O);
     break;
   }
+  case MachineOperand::MO_BlockAddress: {
+    MCSymbol *Sym = P.GetBlockAddressSymbol(MO.getBlockAddress());
+    Sym->print(O, P.MAI);
+    break;
+  }
   }
 }
 
 static void printLeaMemReference(X86AsmPrinter &P, const MachineInstr *MI,
                                  unsigned Op, raw_ostream &O,
                                  const char *Modifier = nullptr) {
   const MachineOperand &BaseReg  = MI->getOperand(Op+X86::AddrBaseReg);
   const MachineOperand &IndexReg = MI->getOperand(Op+X86::AddrIndexReg);

llvm/trunk/lib/Target/X86/X86FloatingPoint.cpp

case TargetOpcode::IMPLICIT_DEF: {
     // All FP registers must be explicitly defined, so load a 0 instead.
     unsigned Reg = MI.getOperand(0).getReg() - X86::FP0;
     LLVM_DEBUG(dbgs() << "Emitting LD_F0 for implicit FP" << Reg << '\n');
     BuildMI(*MBB, Inst, MI.getDebugLoc(), TII->get(X86::LD_F0));
     pushReg(Reg);
     break;
   }
 
-  case TargetOpcode::INLINEASM: {
+  case TargetOpcode::INLINEASM:
+  case TargetOpcode::INLINEASM_BR: {
     // The inline asm MachineInstr currently only *uses* FP registers for the
     // 'f' constraint.  These should be turned into the current ST(x) register
     // in the machine instr.
     //
     // There are special rules for x87 inline assembly. The compiler must know
     // exactly how many registers are popped and pushed implicitly by the asm.
     // Otherwise it is not possible to restore the stack state after the inline
     // asm.

llvm/trunk/lib/Transforms/InstCombine/InstCombineCalls.cpp

 //===- InstCombineCalls.cpp -----------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
-// This file implements the visitCall and visitInvoke functions.
+// This file implements the visitCall, visitInvoke, and visitCallBr functions.
 //
 //===----------------------------------------------------------------------===//
 
 #include "InstCombineInternal.h"
 #include "llvm/ADT/APFloat.h"
 #include "llvm/ADT/APInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/None.h"
Instruction *InstCombiner::visitCallInst(CallInst &CI) {
   if (CI.getFunction()->doesNotThrow() && !CI.doesNotThrow()) {
     CI.setDoesNotThrow();
     return &CI;
   }
 
   IntrinsicInst *II = dyn_cast<IntrinsicInst>(&CI);
   if (!II) return visitCallBase(CI);
 
-  // Intrinsics cannot occur in an invoke, so handle them here instead of in
-  // visitCallBase.
+  // Intrinsics cannot occur in an invoke or a callbr, so handle them here
+  // instead of in visitCallBase.
   if (auto *MI = dyn_cast<AnyMemIntrinsic>(II)) {
     bool Changed = false;
 
     // memmove/cpy/set of zero bytes is a noop.
     if (Constant *NumBytes = dyn_cast<Constant>(MI->getLength())) {
       if (NumBytes->isNullValue())
         return eraseInstFromFunction(CI);
 
Instruction *InstCombiner::visitFenceInst(FenceInst &FI) {
   return nullptr;
 }
 
 // InvokeInst simplification
 Instruction *InstCombiner::visitInvokeInst(InvokeInst &II) {
   return visitCallBase(II);
 }
 
+// CallBrInst simplification
+Instruction *InstCombiner::visitCallBrInst(CallBrInst &CBI) {
+  return visitCallBase(CBI);
+}
+
 /// If this cast does not affect the value passed through the varargs area, we
 /// can eliminate the use of the cast.
 static bool isSafeToEliminateVarargsCast(const CallBase &Call,
                                          const DataLayout &DL,
                                          const CastInst *const CI,
                                          const int ix) {
   if (!CI->isLosslessCast())
     return false;
static IntrinsicInst *findInitTrampoline(Value *Callee) {
 
   if (IntrinsicInst *IT = findInitTrampolineFromAlloca(TrampMem))
     return IT;
   if (IntrinsicInst *IT = findInitTrampolineFromBB(AdjustTramp, TrampMem))
     return IT;
   return nullptr;
 }
 
-/// Improvements for call and invoke instructions.
+/// Improvements for call, callbr and invoke instructions.
 Instruction *InstCombiner::visitCallBase(CallBase &Call) {
   if (isAllocLikeFn(&Call, &TLI))
     return visitAllocSite(Call);
 
   bool Changed = false;
 
   // Mark any parameters that are known to be non-null with the nonnull
   // attribute.  This is helpful for inlining calls to functions with null
if (!ArgNos.empty()) {
     LLVMContext &Ctx = Call.getContext();
     AS = AS.addParamAttribute(Ctx, ArgNos,
                               Attribute::get(Ctx, Attribute::NonNull));
     Call.setAttributes(AS);
     Changed = true;
   }
 
   // If the callee is a pointer to a function, attempt to move any casts to the
-  // arguments of the call/invoke.
+  // arguments of the call/callbr/invoke.
   Value *Callee = Call.getCalledValue();
   if (!isa<Function>(Callee) && transformConstExprCastCall(Call))
     return nullptr;
 
   if (Function *CalleeF = dyn_cast<Function>(Callee)) {
     // Remove the convergent attr on calls when the callee is not convergent.
     if (Call.isConvergent() && !CalleeF->isConvergent() &&
         !CalleeF->isIntrinsic()) {
if (CalleeF->getCallingConv() != Call.getCallingConv() &&
                                   OldCall);
       // If OldCall does not return void then replaceAllUsesWith undef.
       // This allows ValueHandlers and custom metadata to adjust itself.
       if (!OldCall->getType()->isVoidTy())
         replaceInstUsesWith(*OldCall, UndefValue::get(OldCall->getType()));
       if (isa<CallInst>(OldCall))
         return eraseInstFromFunction(*OldCall);
 
-      // We cannot remove an invoke, because it would change the CFG, just
-      // change the callee to a null pointer.
-      cast<InvokeInst>(OldCall)->setCalledFunction(
+      // We cannot remove an invoke or a callbr, because it would change thexi
+      // CFG, just change the callee to a null pointer.
+      cast<CallBase>(OldCall)->setCalledFunction(
           CalleeF->getFunctionType(),
           Constant::getNullValue(CalleeF->getType()));
       return nullptr;
     }
   }
 
   if ((isa<ConstantPointerNull>(Callee) &&
        !NullPointerIsDefined(Call.getFunction())) ||
       isa<UndefValue>(Callee)) {
     // If Call does not return void then replaceAllUsesWith undef.
     // This allows ValueHandlers and custom metadata to adjust itself.
     if (!Call.getType()->isVoidTy())
       replaceInstUsesWith(Call, UndefValue::get(Call.getType()));
 
-    if (isa<InvokeInst>(Call)) {
-      // Can't remove an invoke because we cannot change the CFG.
+    if (Call.isTerminator()) {
+      // Can't remove an invoke or callbr because we cannot change the CFG.
       return nullptr;
     }
 
     // This instruction is not reachable, just remove it.  We insert a store to
     // undef so that we know that this code is not reachable, despite the fact
     // that we can't modify the CFG here.
     new StoreInst(ConstantInt::getTrue(Callee->getContext()),
                   UndefValue::get(Type::getInt1PtrTy(Callee->getContext())),
if (CallInst *CI = dyn_cast<CallInst>(&Call)) {
     // the fallthrough check.
     if (I) return eraseInstFromFunction(*I);
   }
 
   return Changed ? &Call : nullptr;
 }
 
 /// If the callee is a constexpr cast of a function, attempt to move the cast to
-/// the arguments of the call/invoke.
+/// the arguments of the call/callbr/invoke.
 bool InstCombiner::transformConstExprCastCall(CallBase &Call) {
   auto *Callee = dyn_cast<Function>(Call.getCalledValue()->stripPointerCasts());
   if (!Callee)
     return false;
 
   // If this is a call to a thunk function, don't remove the cast. Thunks are
   // used to transparently forward all incoming parameters and outgoing return
   // values, so it's important to leave the cast in place.
if (OldRetTy != NewRetTy) {
     }
 
     if (!CallerPAL.isEmpty() && !Caller->use_empty()) {
       AttrBuilder RAttrs(CallerPAL, AttributeList::ReturnIndex);
       if (RAttrs.overlaps(AttributeFuncs::typeIncompatible(NewRetTy)))
         return false;   // Attribute not compatible with transformed value.
     }
 
-    // If the callbase is an invoke instruction, and the return value is used by
-    // a PHI node in a successor, we cannot change the return type of the call
-    // because there is no place to put the cast instruction (without breaking
-    // the critical edge).  Bail out in this case.
-    if (!Caller->use_empty())
+    // If the callbase is an invoke/callbr instruction, and the return value is
+    // used by a PHI node in a successor, we cannot change the return type of
+    // the call because there is no place to put the cast instruction (without
+    // breaking the critical edge).  Bail out in this case.
+    if (!Caller->use_empty()) {
       if (InvokeInst *II = dyn_cast<InvokeInst>(Caller))
         for (User *U : II->users())
           if (PHINode *PN = dyn_cast<PHINode>(U))
             if (PN->getParent() == II->getNormalDest() ||
                 PN->getParent() == II->getUnwindDest())
               return false;
+      // FIXME: Be conservative for callbr to avoid a quadratic search.
+      if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller))
+        return false;
+    }
   }
 
   unsigned NumActualArgs = Call.arg_size();
   unsigned NumCommonArgs = std::min(FT->getNumParams(), NumActualArgs);
 
   // Prevent us turning:
   // declare void @takes_i32_inalloca(i32* inalloca)
   //  call void bitcast (void (i32*)* @takes_i32_inalloca to void (i32)*)(i32 0)
bool InstCombiner::transformConstExprCastCall(CallBase &Call) {
 
   SmallVector<OperandBundleDef, 1> OpBundles;
   Call.getOperandBundlesAsDefs(OpBundles);
 
   CallBase *NewCall;
   if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
     NewCall = Builder.CreateInvoke(Callee, II->getNormalDest(),
                                    II->getUnwindDest(), Args, OpBundles);
+  } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+    NewCall = Builder.CreateCallBr(Callee, CBI->getDefaultDest(),
+                                   CBI->getIndirectDests(), Args, OpBundles);
   } else {
     NewCall = Builder.CreateCall(Callee, Args, OpBundles);
     cast<CallInst>(NewCall)->setTailCallKind(
         cast<CallInst>(Caller)->getTailCallKind());
   }
   NewCall->takeName(Caller);
   NewCall->setCallingConv(Call.getCallingConv());
   NewCall->setAttributes(NewCallerPAL);
 
   // Preserve the weight metadata for the new call instruction. The metadata
   // is used by SamplePGO to check callsite's hotness.
   uint64_t W;
   if (Caller->extractProfTotalWeight(W))
     NewCall->setProfWeight(W);
 
   // Insert a cast of the return type as necessary.
   Instruction *NC = NewCall;
   Value *NV = NC;
   if (OldRetTy != NV->getType() && !Caller->use_empty()) {
     if (!NV->getType()->isVoidTy()) {
       NV = NC = CastInst::CreateBitOrPointerCast(NC, OldRetTy);
       NC->setDebugLoc(Caller->getDebugLoc());
 
-      // If this is an invoke instruction, we should insert it after the first
-      // non-phi, instruction in the normal successor block.
+      // If this is an invoke/callbr instruction, we should insert it after the
+      // first non-phi instruction in the normal successor block.
       if (InvokeInst *II = dyn_cast<InvokeInst>(Caller)) {
         BasicBlock::iterator I = II->getNormalDest()->getFirstInsertionPt();
         InsertNewInstBefore(NC, *I);
+      } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(Caller)) {
+        BasicBlock::iterator I = CBI->getDefaultDest()->getFirstInsertionPt();
+        InsertNewInstBefore(NC, *I);
       } else {
         // Otherwise, it's a call, just insert cast right after the call.
         InsertNewInstBefore(NC, *Caller);
       }
       Worklist.AddUsersToWorkList(*Caller);
     } else {
       NV = UndefValue::get(Caller->getType());
     }
if (NestTy) {
 
       Instruction *NewCaller;
       if (InvokeInst *II = dyn_cast<InvokeInst>(&Call)) {
         NewCaller = InvokeInst::Create(NewFTy, NewCallee,
                                        II->getNormalDest(), II->getUnwindDest(),
                                        NewArgs, OpBundles);
         cast<InvokeInst>(NewCaller)->setCallingConv(II->getCallingConv());
         cast<InvokeInst>(NewCaller)->setAttributes(NewPAL);
+      } else if (CallBrInst *CBI = dyn_cast<CallBrInst>(&Call)) {
+        NewCaller =
+            CallBrInst::Create(NewFTy, NewCallee, CBI->getDefaultDest(),
+                               CBI->getIndirectDests(), NewArgs, OpBundles);
+        cast<CallBrInst>(NewCaller)->setCallingConv(CBI->getCallingConv());
+        cast<CallBrInst>(NewCaller)->setAttributes(NewPAL);
       } else {
         NewCaller = CallInst::Create(NewFTy, NewCallee, NewArgs, OpBundles);
         cast<CallInst>(NewCaller)->setTailCallKind(
             cast<CallInst>(Call).getTailCallKind());
         cast<CallInst>(NewCaller)->setCallingConv(
             cast<CallInst>(Call).getCallingConv());
         cast<CallInst>(NewCaller)->setAttributes(NewPAL);
       }

llvm/trunk/lib/Transforms/InstCombine/InstCombineInternal.h

public:
   Instruction *visitPtrToInt(PtrToIntInst &CI);
   Instruction *visitIntToPtr(IntToPtrInst &CI);
   Instruction *visitBitCast(BitCastInst &CI);
   Instruction *visitAddrSpaceCast(AddrSpaceCastInst &CI);
   Instruction *FoldItoFPtoI(Instruction &FI);
   Instruction *visitSelectInst(SelectInst &SI);
   Instruction *visitCallInst(CallInst &CI);
   Instruction *visitInvokeInst(InvokeInst &II);
+  Instruction *visitCallBrInst(CallBrInst &CBI);
 
   Instruction *SliceUpIllegalIntegerPHI(PHINode &PN);
   Instruction *visitPHINode(PHINode &PN);
   Instruction *visitGetElementPtrInst(GetElementPtrInst &GEP);
   Instruction *visitAllocaInst(AllocaInst &AI);
   Instruction *visitAllocSite(Instruction &FI);
   Instruction *visitFree(CallInst &FI);
   Instruction *visitLoadInst(LoadInst &LI);

llvm/trunk/lib/Transforms/InstCombine/InstructionCombining.cpp

for (unsigned i = 0; i != NumPHIValues; ++i) {
 
     if (isa<PHINode>(InVal)) return nullptr;  // Itself a phi.
     if (NonConstBB) return nullptr;  // More than one non-const value.
 
     NonConstBB = PN->getIncomingBlock(i);
 
     // If the InVal is an invoke at the end of the pred block, then we can't
     // insert a computation after it without breaking the edge.
-    if (InvokeInst *II = dyn_cast<InvokeInst>(InVal))
-      if (II->getParent() == NonConstBB)
+    if (isa<InvokeInst>(InVal))
+      if (cast<Instruction>(InVal)->getParent() == NonConstBB)
         return nullptr;
 
     // If the incoming non-constant value is in I's block, we will remove one
     // instruction, but insert another equivalent one, leading to infinite
     // instcombine.
     if (isPotentiallyReachable(I.getParent(), NonConstBB, &DT, LI))
       return nullptr;
   }

llvm/trunk/lib/Transforms/Scalar/GVN.cpp

for (BasicBlock *Pred : predecessors(LoadBB)) {
     if (Pred->getTerminator()->getNumSuccessors() != 1) {
       if (isa<IndirectBrInst>(Pred->getTerminator())) {
         LLVM_DEBUG(
             dbgs() << "COULD NOT PRE LOAD BECAUSE OF INDBR CRITICAL EDGE '"
                    << Pred->getName() << "': " << *LI << '\n');
         return false;
       }
 
+      // FIXME: Can we support the fallthrough edge?
+      if (isa<CallBrInst>(Pred->getTerminator())) {
+        LLVM_DEBUG(
+            dbgs() << "COULD NOT PRE LOAD BECAUSE OF CALLBR CRITICAL EDGE '"
+                   << Pred->getName() << "': " << *LI << '\n');
+        return false;
+      }
+
       if (LoadBB->isEHPad()) {
         LLVM_DEBUG(
             dbgs() << "COULD NOT PRE LOAD BECAUSE OF AN EH PAD CRITICAL EDGE '"
                    << Pred->getName() << "': " << *LI << '\n');
         return false;
       }
 
       CriticalEdgePred.push_back(Pred);
bool GVN::performScalarPRE(Instruction *CurInst) {
   // introduce unnecessary spills.
   //
   // This doesn't prevent Load PRE. PHI translation will make the GEP available
   // to the load by moving it to the predecessor block if necessary.
   if (isa<GetElementPtrInst>(CurInst))
     return false;
 
   // We don't currently value number ANY inline asm calls.
-  if (CallInst *CallI = dyn_cast<CallInst>(CurInst))
-    if (CallI->isInlineAsm())
+  if (auto *CallB = dyn_cast<CallBase>(CurInst))
+    if (CallB->isInlineAsm())
       return false;
 
   uint32_t ValNo = VN.lookup(CurInst);
 
   // Look for the predecessors for PRE opportunities.  We're
   // only trying to solve the basic diamond case, where
   // a value is computed in the successor and one predecessor,
   // but not the other.  We also explicitly disallow cases
if (!isSafeToSpeculativelyExecute(CurInst)) {
       if (ICF->isDominatedByICFIFromSameBlock(CurInst))
         return false;
     }
 
     // Don't do PRE across indirect branch.
     if (isa<IndirectBrInst>(PREPred->getTerminator()))
       return false;
 
+    // Don't do PRE across callbr.
+    // FIXME: Can we do this across the fallthrough edge?
+    if (isa<CallBrInst>(PREPred->getTerminator()))
+      return false;
+
     // We can't do PRE safely on a critical edge, so instead we schedule
     // the edge to be split and perform the PRE the next time we iterate
     // on the function.
     unsigned SuccNum = GetSuccessorNumber(PREPred, CurrentBlock);
     if (isCriticalEdge(PREPred->getTerminator(), SuccNum)) {
       toSplit.push_back(std::make_pair(PREPred->getTerminator(), SuccNum));
       return false;
     }

llvm/trunk/lib/Transforms/Scalar/JumpThreading.cpp

bool JumpThreadingPass::ProcessBlock(BasicBlock *BB) {
   } else if (SwitchInst *SI = dyn_cast<SwitchInst>(Terminator)) {
     Condition = SI->getCondition();
   } else if (IndirectBrInst *IB = dyn_cast<IndirectBrInst>(Terminator)) {
     // Can't thread indirect branch with no successors.
     if (IB->getNumSuccessors() == 0) return false;
     Condition = IB->getAddress()->stripPointerCasts();
     Preference = WantBlockAddress;
   } else {
-    return false; // Must be an invoke.
+    return false; // Must be an invoke or callbr.
   }
 
   // Run constant folding to see if we can reduce the condition to a simple
   // constant.
   if (Instruction *I = dyn_cast<Instruction>(Condition)) {
     Value *SimpleVal =
         ConstantFoldInstruction(I, BB->getModule()->getDataLayout(), TLI);
     if (SimpleVal) {
if (PredsScanned.size() == AvailablePreds.size()+1 &&
     SmallPtrSet<BasicBlock*, 8> AvailablePredSet;
 
     for (const auto &AvailablePred : AvailablePreds)
       AvailablePredSet.insert(AvailablePred.first);
 
     // Add all the unavailable predecessors to the PredsToSplit list.
     for (BasicBlock *P : predecessors(LoadBB)) {
       // If the predecessor is an indirect goto, we can't split the edge.
-      if (isa<IndirectBrInst>(P->getTerminator()))
+      // Same for CallBr.
+      if (isa<IndirectBrInst>(P->getTerminator()) ||
+          isa<CallBrInst>(P->getTerminator()))
         return false;
 
       if (!AvailablePredSet.count(P))
         PredsToSplit.push_back(P);
     }
 
     // Split them out to their own block.
     UnavailablePred = SplitBlockPreds(LoadBB, PredsToSplit, "thread-pre-split");
if (PredToDestList.empty()) {
       if (Val != OnlyVal)
         OnlyVal = MultipleVal;
     }
 
     // We know where this predecessor is going.
     ++PredWithKnownDest;
 
     // If the predecessor ends with an indirect goto, we can't change its
-    // destination.
-    if (isa<IndirectBrInst>(Pred->getTerminator()))
+    // destination. Same for CallBr.
+    if (isa<IndirectBrInst>(Pred->getTerminator()) ||
+        isa<CallBrInst>(Pred->getTerminator()))
       continue;
 
     PredToDestList.push_back(std::make_pair(Pred, DestBB));
   }
 
   // If all edges were unthreadable, we fail.
   if (PredToDestList.empty())
     return false;

llvm/trunk/lib/Transforms/Scalar/SCCP.cpp

void visitCallInst      (CallInst &I) {
     visitCallSite(&I);
   }
 
   void visitInvokeInst    (InvokeInst &II) {
     visitCallSite(&II);
     visitTerminator(II);
   }
 
+  void visitCallBrInst    (CallBrInst &CBI) {
+    visitCallSite(&CBI);
+    visitTerminator(CBI);
+  }
+
   void visitCallSite      (CallSite CS);
   void visitResumeInst    (ResumeInst &I) { /*returns void*/ }
   void visitUnreachableInst(UnreachableInst &I) { /*returns void*/ }
   void visitFenceInst     (FenceInst &I) { /*returns void*/ }
 
   void visitInstruction(Instruction &I) {
     // All the instructions we don't do any special handling for just
     // go to overdefined.
for (unsigned i = 0; i < IBR->getNumSuccessors(); ++i) {
       }
     }
 
     // If we didn't find our destination in the IBR successor list, then we
     // have undefined behavior. Its ok to assume no successor is executable.
     return;
   }
 
+  // In case of callbr, we pessimistically assume that all successors are
+  // feasible.
+  if (isa<CallBrInst>(&TI)) {
+    Succs.assign(TI.getNumSuccessors(), true);
+    return;
+  }
+
   LLVM_DEBUG(dbgs() << "Unknown terminator instruction: " << TI << '\n');
   llvm_unreachable("SCCP: Don't know how to handle this terminator!");
 }
 
 // isEdgeFeasible - Return true if the control flow edge from the 'From' basic
 // block to the 'To' basic block is currently feasible.
 bool SCCPSolver::isEdgeFeasible(BasicBlock *From, BasicBlock *To) {
   // Check if we've called markEdgeExecutable on the edge yet. (We could
for (Instruction &I : BB) {
 
         if ((Op0LV.isUnknown() || Op1LV.isUnknown()) &&
             cast<ICmpInst>(&I)->isEquality())
           break;
         markOverdefined(&I);
         return true;
       case Instruction::Call:
       case Instruction::Invoke:
+      case Instruction::CallBr:
         // There are two reasons a call can have an undef result
         // 1. It could be tracked.
         // 2. It could be constant-foldable.
         // Because of the way we solve return values, tracked calls must
         // never be marked overdefined in ResolvedUndefsIn.
         if (Function *F = CallSite(&I).getCalledFunction())
           if (TrackedRetVals.count(F))
             break;

llvm/trunk/lib/Transforms/Utils/BasicBlockUtils.cpp

BasicBlock *llvm::SplitBlockPredecessors(BasicBlock *BB,
 
   // Move the edges from Preds to point to NewBB instead of BB.
   for (unsigned i = 0, e = Preds.size(); i != e; ++i) {
     // This is slightly more strict than necessary; the minimum requirement
     // is that there be no more than one indirectbr branching to BB. And
     // all BlockAddress uses would need to be updated.
     assert(!isa<IndirectBrInst>(Preds[i]->getTerminator()) &&
            "Cannot split an edge from an IndirectBrInst");
+    assert(!isa<CallBrInst>(Preds[i]->getTerminator()) &&
+           "Cannot split an edge from a CallBrInst");
     Preds[i]->getTerminator()->replaceUsesOfWith(BB, NewBB);
   }
 
   // Insert a new PHI node into NewBB for every PHI node in BB and that new PHI
   // node becomes an incoming value for BB's phi node.  However, if the Preds
   // list is empty, we need to insert dummy entries into the PHI nodes in BB to
   // account for the newly created predecessor.
   if (Preds.empty()) {

llvm/trunk/lib/Transforms/Utils/BreakCriticalEdges.cpp

llvm::SplitCriticalEdge(Instruction *TI, unsigned SuccNum,
 
   BasicBlock *TIBB = TI->getParent();
   BasicBlock *DestBB = TI->getSuccessor(SuccNum);
 
   // Splitting the critical edge to a pad block is non-trivial. Don't do
   // it in this generic function.
   if (DestBB->isEHPad()) return nullptr;
 
+  // Don't split the non-fallthrough edge from a callbr.
+  if (isa<CallBrInst>(TI) && SuccNum > 0)
+    return nullptr;
+
   // Create a new basic block, linking it into the CFG.
   BasicBlock *NewBB = BasicBlock::Create(TI->getContext(),
                       TIBB->getName() + "." + DestBB->getName() + "_crit_edge");
   // Create our unconditional branch.
   BranchInst *NewBI = BranchInst::Create(DestBB, NewBB);
   NewBI->setDebugLoc(TI->getDebugLoc());
 
   // Branch to the new block, breaking the edge.

llvm/trunk/lib/Transforms/Utils/InlineFunction.cpp

 llvm::InlineResult llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
                                         AAResults *CalleeAAR,
                                         bool InsertLifetime,
                                         Function *ForwardVarArgsTo) {
   Instruction *TheCall = CS.getInstruction();
   assert(TheCall->getParent() && TheCall->getFunction()
          && "Instruction not in function!");
 
+  // FIXME: we don't inline callbr yet.
+  if (isa<CallBrInst>(TheCall))
+    return false;
+
   // If IFI has any state in it, zap it before we fill it in.
   IFI.reset();
 
   Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||               // Can't inline external function or indirect
       CalledFunc->isDeclaration()) // call!
     return "external or indirect";
 
if (ParentDeopt) {
                                  ChildOB.Inputs.end());
 
           OpDefs.emplace_back("deopt", std::move(MergedDeoptArgs));
         }
 
         Instruction *NewI = nullptr;
         if (isa<CallInst>(I))
           NewI = CallInst::Create(cast<CallInst>(I), OpDefs, I);
+        else if (isa<CallBrInst>(I))
+          NewI = CallBrInst::Create(cast<CallBrInst>(I), OpDefs, I);
         else
           NewI = InvokeInst::Create(cast<InvokeInst>(I), OpDefs, I);
 
         // Note: the RAUW does the appropriate fixup in VMap, so we need to do
         // this even if the call returns void.
         I->replaceAllUsesWith(NewI);
 
         VH = nullptr;
for (Function::iterator BB = FirstNewBlock->getIterator(),
           continue;
 
         CS.getOperandBundlesAsDefs(OpBundles);
         OpBundles.emplace_back("funclet", CallSiteEHPad);
 
         Instruction *NewInst;
         if (CS.isCall())
           NewInst = CallInst::Create(cast<CallInst>(I), OpBundles, I);
+        else if (CS.isCallBr())
+          NewInst = CallBrInst::Create(cast<CallBrInst>(I), OpBundles, I);
         else
           NewInst = InvokeInst::Create(cast<InvokeInst>(I), OpBundles, I);
         NewInst->takeName(I);
         I->replaceAllUsesWith(NewInst);
         I->eraseFromParent();
 
         OpBundles.clear();
       }

llvm/trunk/lib/Transforms/Utils/Local.cpp

while (isa<PHINode>(*BBI)) {
         } else {
           return false;
         }
       }
       ++BBI;
     }
   }
 
+  // We cannot fold the block if it's a branch to an already present callbr
+  // successor because that creates duplicate successors.
+  for (auto I = pred_begin(BB), E = pred_end(BB); I != E; ++I) {
+    if (auto *CBI = dyn_cast<CallBrInst>((*I)->getTerminator())) {
+      if (Succ == CBI->getDefaultDest())
+        return false;
+      for (unsigned i = 0, e = CBI->getNumIndirectDests(); i != e; ++i)
+        if (Succ == CBI->getIndirectDest(i))
+          return false;
+    }
+  }
+
   LLVM_DEBUG(dbgs() << "Killing Trivial BB: \n" << *BB);
 
   SmallVector<DominatorTree::UpdateType, 32> Updates;
   if (DTU) {
     Updates.push_back({DominatorTree::Delete, BB, Succ});
     // All predecessors of BB will be moved to Succ.
     for (auto I = pred_begin(BB), E = pred_end(BB); I != E; ++I) {
       Updates.push_back({DominatorTree::Delete, *I, BB});

llvm/trunk/lib/Transforms/Utils/LoopSimplify.cpp

 //
 // This pass also guarantees that loops will have exactly one backedge.
 //
 // Indirectbr instructions introduce several complications. If the loop
 // contains or is entered by an indirectbr instruction, it may not be possible
 // to transform the loop and make these guarantees. Client code should check
 // that these conditions are true before relying on them.
 //
+// Similar complications arise from callbr instructions, particularly in
+// asm-goto where blockaddress expressions are used.
+//
 // Note that the simplifycfg pass will clean up blocks which are split out but
 // end up being unnecessary, so usage of this pass should not pessimize
 // generated code.
 //
 // This pass obviously modifies the CFG, but updates loop information and
 // dominator information.
 //
 //===----------------------------------------------------------------------===//
BasicBlock *llvm::InsertPreheaderForLoop(Loop *L, DominatorTree *DT,
   BasicBlock *Header = L->getHeader();
 
   // Compute the set of predecessors of the loop that are not in the loop.
   SmallVector<BasicBlock*, 8> OutsideBlocks;
   for (pred_iterator PI = pred_begin(Header), PE = pred_end(Header);
        PI != PE; ++PI) {
     BasicBlock *P = *PI;
     if (!L->contains(P)) {         // Coming in from outside the loop?
-      // If the loop is branched to from an indirect branch, we won't
+      // If the loop is branched to from an indirect terminator, we won't
       // be able to fully transform the loop, because it prohibits
       // edge splitting.
-      if (isa<IndirectBrInst>(P->getTerminator())) return nullptr;
+      if (P->getTerminator()->isIndirectTerminator())
+        return nullptr;
 
       // Keep track of it.
       OutsideBlocks.push_back(P);
     }
   }
 
   // Split out the loop pre-header.
   BasicBlock *PreheaderBB;
static Loop *separateNestedLoop(Loop *L, BasicBlock *Preheader,
 
   // Pull out all predecessors that have varying values in the loop.  This
   // handles the case when a PHI node has multiple instances of itself as
   // arguments.
   SmallVector<BasicBlock*, 8> OuterLoopPreds;
   for (unsigned i = 0, e = PN->getNumIncomingValues(); i != e; ++i) {
     if (PN->getIncomingValue(i) != PN ||
         !L->contains(PN->getIncomingBlock(i))) {
-      // We can't split indirectbr edges.
-      if (isa<IndirectBrInst>(PN->getIncomingBlock(i)->getTerminator()))
+      // We can't split indirect control flow edges.
+      if (PN->getIncomingBlock(i)->getTerminator()->isIndirectTerminator())
         return nullptr;
       OuterLoopPreds.push_back(PN->getIncomingBlock(i));
     }
   }
   LLVM_DEBUG(dbgs() << "LoopSimplify: Splitting out a new outer loop\n");
 
   // If ScalarEvolution is around and knows anything about values in
   // this loop, tell it to forget them, because we're about to
static BasicBlock *insertUniqueBackedgeBlock(Loop *L, BasicBlock *Preheader,
   // The header is not an EH pad; preheader insertion should ensure this.
   assert(!Header->isEHPad() && "Can't insert backedge to EH pad");
 
   // Figure out which basic blocks contain back-edges to the loop header.
   std::vector<BasicBlock*> BackedgeBlocks;
   for (pred_iterator I = pred_begin(Header), E = pred_end(Header); I != E; ++I){
     BasicBlock *P = *I;
 
-    // Indirectbr edges cannot be split, so we must fail if we find one.
-    if (isa<IndirectBrInst>(P->getTerminator()))
+    // Indirect edges cannot be split, so we must fail if we find one.
+    if (P->getTerminator()->isIndirectTerminator())
       return nullptr;
 
     if (P != Preheader) BackedgeBlocks.push_back(P);
   }
 
   // Create and insert the new backedge block...
   BasicBlock *BEBlock = BasicBlock::Create(Header->getContext(),
                                            Header->getName() + ".backedge", F);

llvm/trunk/lib/Transforms/Utils/LoopUtils.cpp

auto RewriteExit = [&](BasicBlock *BB) {
     // See if there are any non-loop predecessors of this exit block and
     // keep track of the in-loop predecessors.
     bool IsDedicatedExit = true;
     for (auto *PredBB : predecessors(BB))
       if (L->contains(PredBB)) {
         if (isa<IndirectBrInst>(PredBB->getTerminator()))
           // We cannot rewrite exiting edges from an indirectbr.
           return false;
+        if (isa<CallBrInst>(PredBB->getTerminator()))
+          // We cannot rewrite exiting edges from a callbr.
+          return false;
 
         InLoopPredecessors.push_back(PredBB);
       } else {
         IsDedicatedExit = false;
       }
 
     assert(!InLoopPredecessors.empty() && "Must have *some* loop predecessor!");
 

llvm/trunk/lib/Transforms/Utils/SimplifyCFG.cpp

static bool HoistThenElseCodeToIf(BranchInst *BI,
   DbgInfoIntrinsic *DBI1 = dyn_cast<DbgInfoIntrinsic>(I1);
   DbgInfoIntrinsic *DBI2 = dyn_cast<DbgInfoIntrinsic>(I2);
   if (!DBI1 || !DBI2 || !DBI1->isIdenticalToWhenDefined(DBI2)) {
     while (isa<DbgInfoIntrinsic>(I1))
       I1 = &*BB1_Itr++;
     while (isa<DbgInfoIntrinsic>(I2))
       I2 = &*BB2_Itr++;
   }
+  // FIXME: Can we define a safety predicate for CallBr?
   if (isa<PHINode>(I1) || !I1->isIdenticalToWhenDefined(I2) ||
-      (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2)))
+      (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2)) ||
+      isa<CallBrInst>(I1))
     return false;
 
   BasicBlock *BIParent = BI->getParent();
 
   bool Changed = false;
   do {
     // If we are hoisting the terminator instruction, don't move one (making a
     // broken BB), instead clone it, and remove BI.
if (!DBI1 || !DBI2 || !DBI1->isIdenticalToWhenDefined(DBI2)) {
         I2 = &*BB2_Itr++;
     }
   } while (I1->isIdenticalToWhenDefined(I2));
 
   return true;
 
 HoistTerminator:
   // It may not be possible to hoist an invoke.
+  // FIXME: Can we define a safety predicate for CallBr?
   if (isa<InvokeInst>(I1) && !isSafeToHoistInvoke(BB1, BB2, I1, I2))
     return Changed;
 
+  // TODO: callbr hoisting currently disabled pending further study.
+  if (isa<CallBrInst>(I1))
+    return Changed;
+
   for (BasicBlock *Succ : successors(BB1)) {
     for (PHINode &PN : Succ->phis()) {
       Value *BB1V = PN.getIncomingValueForBlock(BB1);
       Value *BB2V = PN.getIncomingValueForBlock(BB2);
       if (BB1V == BB2V)
         continue;
 
       // Check for passingValueIsAlwaysUndefined here because we would rather
for (auto *I : Insts) {
     // These instructions may change or break semantics if moved.
     if (isa<PHINode>(I) || I->isEHPad() || isa<AllocaInst>(I) ||
         I->getType()->isTokenTy())
       return false;
 
     // Conservatively return false if I is an inline-asm instruction. Sinking
     // and merging inline-asm instructions can potentially create arguments
     // that cannot satisfy the inline-asm constraints.
-    if (const auto *C = dyn_cast<CallInst>(I))
+    if (const auto *C = dyn_cast<CallBase>(I))
       if (C->isInlineAsm())
         return false;
 
     // Everything must have only one use too, apart from stores which
     // have no uses.
     if (!isa<StoreInst>(I) && !I->hasOneUse())
       return false;
   }
auto SameAsI0 = [&I0, OI](const Instruction *I) {
       assert(I->getNumOperands() == I0->getNumOperands());
       return I->getOperand(OI) == I0->getOperand(OI);
     };
     if (!all_of(Insts, SameAsI0)) {
       if (!canReplaceOperandWithVariable(I0, OI))
         // We can't create a PHI from this GEP.
         return false;
       // Don't create indirect calls! The called value is the final operand.
-      if ((isa<CallInst>(I0) || isa<InvokeInst>(I0)) && OI == OE - 1) {
+      if (isa<CallBase>(I0) && OI == OE - 1) {
         // FIXME: if the call was *already* indirect, we should do this.
         return false;
       }
       for (auto *I : Insts)
         PHIOperands[I].push_back(I->getOperand(OI));
     }
   }
   return true;

llvm/trunk/test/Bitcode/callbr.ll

+; RUN:  llvm-dis < %s.bc | FileCheck %s
+
+; callbr.ll.bc was generated by passing this file to llvm-as.
+
+define i32 @test_asm_goto(i32 %x){
+entry:
+; CHECK:      callbr void asm "", "r,X"(i32 %x, i8* blockaddress(@test_asm_goto, %fail))
+; CHECK-NEXT: to label %normal [label %fail]
+  callbr void asm "", "r,X"(i32 %x, i8* blockaddress(@test_asm_goto, %fail)) to label %normal [label %fail]
+normal:
+  ret i32 1
+fail:
+  ret i32 0
+}

llvm/trunk/test/CodeGen/X86/callbr-asm-blockplacement.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
+
+; This test asserted in MachineBlockPlacement during asm-goto bring up.
+
+%struct.wibble = type { %struct.pluto, i32, i8* }
+%struct.pluto = type { i32, i32, i32 }
+
+@global = external global [0 x %struct.wibble]
+
+define i32 @foo(i32 %arg, i32 (i8*)* %arg3) nounwind {
+; CHECK-LABEL: foo:
+; CHECK:       # %bb.0: # %bb
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    pushq %r15
+; CHECK-NEXT:    pushq %r14
+; CHECK-NEXT:    pushq %r13
+; CHECK-NEXT:    pushq %r12
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    movabsq $-2305847407260205056, %rbx # imm = 0xDFFFFC0000000000
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    testb %al, %al
+; CHECK-NEXT:    jne .LBB0_5
+; CHECK-NEXT:  # %bb.1: # %bb5
+; CHECK-NEXT:    movq %rsi, %r14
+; CHECK-NEXT:    movslq %edi, %rbp
+; CHECK-NEXT:    leaq (,%rbp,8), %rax
+; CHECK-NEXT:    leaq global(%rax,%rax,2), %r15
+; CHECK-NEXT:    leaq global+4(%rax,%rax,2), %r12
+; CHECK-NEXT:    xorl %r13d, %r13d
+; CHECK-NEXT:    .p2align 4, 0x90
+; CHECK-NEXT:  .LBB0_2: # %bb8
+; CHECK-NEXT:    # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    callq bar
+; CHECK-NEXT:    movq %rax, %rbx
+; CHECK-NEXT:    movq %rax, %rdi
+; CHECK-NEXT:    callq *%r14
+; CHECK-NEXT:    movq %r15, %rdi
+; CHECK-NEXT:    callq hoge
+; CHECK-NEXT:    movq %r12, %rdi
+; CHECK-NEXT:    callq hoge
+; CHECK-NEXT:    testb %r13b, %r13b
+; CHECK-NEXT:    jne .LBB0_2
+; CHECK-NEXT:  # %bb.3: # %bb15
+; CHECK-NEXT:    leaq (%rbp,%rbp,2), %rax
+; CHECK-NEXT:    movq %rbx, global+16(,%rax,8)
+; CHECK-NEXT:    movabsq $-2305847407260205056, %rbx # imm = 0xDFFFFC0000000000
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  .LBB0_4: # %bb17
+; CHECK-NEXT:    callq widget
+; CHECK-NEXT:  .Ltmp0: # Block address taken
+; CHECK-NEXT:  .LBB0_5: # %bb18
+; CHECK-NEXT:    movw $0, 14(%rbx)
+; CHECK-NEXT:    addq $8, %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %r12
+; CHECK-NEXT:    popq %r13
+; CHECK-NEXT:    popq %r14
+; CHECK-NEXT:    popq %r15
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    retq
+bb:
+  %tmp = add i64 0, -2305847407260205056
+  %tmp4 = sext i32 %arg to i64
+  br i1 undef, label %bb18, label %bb5
+
+bb5:                                              ; preds = %bb
+  %tmp6 = getelementptr [0 x %struct.wibble], [0 x %struct.wibble]* @global, i64 0, i64 %tmp4, i32 0, i32 0
+  %tmp7 = getelementptr [0 x %struct.wibble], [0 x %struct.wibble]* @global, i64 0, i64 %tmp4, i32 0, i32 1
+  br label %bb8
+
+bb8:                                              ; preds = %bb8, %bb5
+  %tmp9 = call i8* @bar(i64 undef)
+  %tmp10 = call i32 %arg3(i8* nonnull %tmp9)
+  %tmp11 = ptrtoint i32* %tmp6 to i64
+  call void @hoge(i64 %tmp11)
+  %tmp12 = ptrtoint i32* %tmp7 to i64
+  %tmp13 = add i64 undef, -2305847407260205056
+  call void @hoge(i64 %tmp12)
+  %tmp14 = icmp eq i32 0, 0
+  br i1 %tmp14, label %bb15, label %bb8
+
+bb15:                                             ; preds = %bb8
+  %tmp16 = getelementptr [0 x %struct.wibble], [0 x %struct.wibble]* @global, i64 0, i64 %tmp4, i32 2
+  store i8* %tmp9, i8** %tmp16
+  callbr void asm sideeffect "", "X"(i8* blockaddress(@foo, %bb18))
+          to label %bb17 [label %bb18]
+
+bb17:                                             ; preds = %bb15
+  call void @widget()
+  br label %bb18
+
+bb18:                                             ; preds = %bb17, %bb15, %bb
+  %tmp19 = add i64 %tmp, 14
+  %tmp20 = inttoptr i64 %tmp19 to i16*
+  store i16 0, i16* %tmp20
+  ret i32 undef
+}
+
+declare i8* @bar(i64)
+
+declare void @widget()
+
+declare void @hoge(i64)

llvm/trunk/test/CodeGen/X86/callbr-asm-branch-folding.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu | FileCheck %s
+
+; This test hung in the BranchFolding pass during asm-goto bring up
+
+@e = global i32 0
+@j = global i32 0
+
+define void @n(i32* %o, i32 %p, i32 %u) nounwind {
+; CHECK-LABEL: n:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    pushq %r15
+; CHECK-NEXT:    pushq %r14
+; CHECK-NEXT:    pushq %r13
+; CHECK-NEXT:    pushq %r12
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    pushq %rax
+; CHECK-NEXT:    movl %edx, %ebx
+; CHECK-NEXT:    movl %esi, %r12d
+; CHECK-NEXT:    movq %rdi, %r15
+; CHECK-NEXT:    callq c
+; CHECK-NEXT:    movl %eax, %r13d
+; CHECK-NEXT:    movq %r15, %rdi
+; CHECK-NEXT:    callq l
+; CHECK-NEXT:    testl %eax, %eax
+; CHECK-NEXT:    je .LBB0_1
+; CHECK-NEXT:  .LBB0_10: # %cleanup
+; CHECK-NEXT:    addq $8, %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %r12
+; CHECK-NEXT:    popq %r13
+; CHECK-NEXT:    popq %r14
+; CHECK-NEXT:    popq %r15
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    retq
+; CHECK-NEXT:  .LBB0_1: # %if.end
+; CHECK-NEXT:    movl %ebx, {{[-0-9]+}}(%r{{[sb]}}p) # 4-byte Spill
+; CHECK-NEXT:    cmpl $0, {{.*}}(%rip)
+; CHECK-NEXT:    # implicit-def: $ebx
+; CHECK-NEXT:    # implicit-def: $r14d
+; CHECK-NEXT:    je .LBB0_4
+; CHECK-NEXT:  # %bb.2: # %if.then4
+; CHECK-NEXT:    movslq %r12d, %rdi
+; CHECK-NEXT:    callq m
+; CHECK-NEXT:    # implicit-def: $ebx
+; CHECK-NEXT:    # implicit-def: $ebp
+; CHECK-NEXT:  .LBB0_3: # %r
+; CHECK-NEXT:    callq c
+; CHECK-NEXT:    movl %ebp, %r14d
+; CHECK-NEXT:  .LBB0_4: # %if.end8
+; CHECK-NEXT:    movl %ebx, %edi
+; CHECK-NEXT:    callq i
+; CHECK-NEXT:    movl %eax, %ebp
+; CHECK-NEXT:    orl %r14d, %ebp
+; CHECK-NEXT:    testl %r13d, %r13d
+; CHECK-NEXT:    je .LBB0_6
+; CHECK-NEXT:  # %bb.5:
+; CHECK-NEXT:    andl $4, %ebx
+; CHECK-NEXT:    jmp .LBB0_3
+; CHECK-NEXT:  .LBB0_6: # %if.end12
+; CHECK-NEXT:    testl %ebp, %ebp
+; CHECK-NEXT:    je .LBB0_9
+; CHECK-NEXT:  # %bb.7: # %if.then14
+; CHECK-NEXT:    movl {{[-0-9]+}}(%r{{[sb]}}p), %eax # 4-byte Reload
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    jmp .LBB0_10
+; CHECK-NEXT:  .Ltmp0: # Block address taken
+; CHECK-NEXT:  .LBB0_8: # %if.then20.critedge
+; CHECK-NEXT:    movl {{.*}}(%rip), %edi
+; CHECK-NEXT:    movslq %eax, %rcx
+; CHECK-NEXT:    movl $1, %esi
+; CHECK-NEXT:    movq %r15, %rdx
+; CHECK-NEXT:    addq $8, %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %r12
+; CHECK-NEXT:    popq %r13
+; CHECK-NEXT:    popq %r14
+; CHECK-NEXT:    popq %r15
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    jmp k # TAILCALL
+; CHECK-NEXT:  .LBB0_9: # %if.else
+; CHECK-NEXT:    incq 0
+; CHECK-NEXT:    jmp .LBB0_10
+entry:
+  %call = tail call i32 @c()
+  %call1 = tail call i32 @l(i32* %o)
+  %tobool = icmp eq i32 %call1, 0
+  br i1 %tobool, label %if.end, label %cleanup
+
+if.end:                                           ; preds = %entry
+  %0 = load i32, i32* @e
+  %tobool3 = icmp eq i32 %0, 0
+  br i1 %tobool3, label %if.end8, label %if.then4, !prof !0
+
+if.then4:                                         ; preds = %if.end
+  %conv5 = sext i32 %p to i64
+  %call6 = tail call i32 @m(i64 %conv5)
+  br label %r
+
+r:                                                ; preds = %if.end8, %if.then4
+  %flags.0 = phi i32 [ undef, %if.then4 ], [ %and, %if.end8 ]
+  %major.0 = phi i32 [ undef, %if.then4 ], [ %or, %if.end8 ]
+  %call7 = tail call i32 @c()
+  br label %if.end8
+
+if.end8:                                          ; preds = %r, %if.end
+  %flags.1 = phi i32 [ %flags.0, %r ], [ undef, %if.end ]
+  %major.1 = phi i32 [ %major.0, %r ], [ undef, %if.end ]
+  %call9 = tail call i32 @i(i32 %flags.1)
+  %or = or i32 %call9, %major.1
+  %and = and i32 %flags.1, 4
+  %tobool10 = icmp eq i32 %call, 0
+  br i1 %tobool10, label %if.end12, label %r
+
+if.end12:                                         ; preds = %if.end8
+  %tobool13 = icmp eq i32 %or, 0
+  br i1 %tobool13, label %if.else, label %if.then14
+
+if.then14:                                        ; preds = %if.end12
+  callbr void asm sideeffect "", "X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@n, %if.then20.critedge))
+          to label %cleanup [label %if.then20.critedge]
+
+if.then20.critedge:                               ; preds = %if.then14
+  %1 = load i32, i32* @j
+  %conv21 = sext i32 %u to i64
+  %call22 = tail call i32 @k(i32 %1, i64 1, i32* %o, i64 %conv21)
+  br label %cleanup
+
+if.else:                                          ; preds = %if.end12
+  %2 = load i64, i64* null
+  %inc = add i64 %2, 1
+  store i64 %inc, i64* null
+  br label %cleanup
+
+cleanup:                                          ; preds = %if.else, %if.then20.critedge, %if.then14, %entry
+  ret void
+}
+
+declare i32 @c()
+
+declare i32 @l(i32*)
+
+declare i32 @m(i64)
+
+declare i32 @i(i32)
+
+declare i32 @k(i32, i64, i32*, i64)
+
+!0 = !{!"branch_weights", i32 2000, i32 1}

llvm/trunk/test/CodeGen/X86/callbr-asm-destinations.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: Duplicate callbr destination
+
+; A test for asm-goto duplicate labels limitation
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail)) to label %fail [label %fail]
+
+fail:
+  ret i32 1
+}

llvm/trunk/test/CodeGen/X86/callbr-asm-errors.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: Duplicate callbr destination
+
+; A test for asm-goto duplicate labels limitation
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail), i8* blockaddress(@test, %fail)) to label %normal [label %fail, label %fail]
+
+normal:
+  ret i32 %0
+
+fail:
+  ret i32 1
+}

llvm/trunk/test/CodeGen/X86/callbr-asm-outputs.ll

+; RUN: not llc -mtriple=i686-- < %s 2> %t
+; RUN: FileCheck %s < %t
+
+; CHECK: error: asm-goto outputs not supported
+
+; A test for asm-goto output prohibition
+
+define i32 @test(i32 %a) {
+entry:
+  %0 = add i32 %a, 4
+  %1 = callbr i32 asm "xorl $1, $1; jmp ${1:l}", "=&r,r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 %1
+
+fail:
+  ret i32 1
+}

llvm/trunk/test/CodeGen/X86/callbr-asm.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=i686-- -O3 | FileCheck %s
+
+; Tests for using callbr as an asm-goto wrapper
+
+; Test 1 - fallthrough label gets removed, but the fallthrough code that is
+; unreachable due to asm ending on a jmp is still left in.
+define i32 @test1(i32 %a) {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    addl $4, %eax
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    jmp .Ltmp00
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  .LBB0_1: # %normal
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retl
+; CHECK-NEXT:  .Ltmp0: # Block address taken
+; CHECK-NEXT:  .LBB0_2: # %fail
+; CHECK-NEXT:    movl $1, %eax
+; CHECK-NEXT:    retl
+entry:
+  %0 = add i32 %a, 4
+  callbr void asm "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test1, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 0
+
+fail:
+  ret i32 1
+}
+
+; Test 2 - callbr terminates an unreachable block, function gets simplified
+; to a trivial zero return.
+define i32 @test2(i32 %a) {
+; CHECK-LABEL: test2:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    xorl %eax, %eax
+; CHECK-NEXT:    retl
+entry:
+  br label %normal
+
+unreachableasm:
+  %0 = add i32 %a, 4
+  callbr void asm sideeffect "xorl $0, $0; jmp ${1:l}", "r,X,~{dirflag},~{fpsr},~{flags}"(i32 %0, i8* blockaddress(@test2, %fail)) to label %normal [label %fail]
+
+normal:
+  ret i32 0
+
+fail:
+  ret i32 1
+}
+
+
+; Test 3 - asm-goto implements a loop. The loop gets recognized, but many loop
+; transforms fail due to canonicalization having callbr exceptions. Trivial
+; blocks at labels 1 and 3 also don't get simplified due to callbr.
+define dso_local i32 @test3(i32 %a) {
+; CHECK-LABEL: test3:
+; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:  .Ltmp1: # Block address taken
+; CHECK-NEXT:  .LBB2_1: # %label01
+; CHECK-NEXT:    # =>This Loop Header: Depth=1
+; CHECK-NEXT:    # Child Loop BB2_2 Depth 2
+; CHECK-NEXT:    # Child Loop BB2_3 Depth 3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:  .Ltmp2: # Block address taken
+; CHECK-NEXT:  .LBB2_2: # %label02
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # => This Loop Header: Depth=2
+; CHECK-NEXT:    # Child Loop BB2_3 Depth 3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:    addl $4, {{[0-9]+}}(%esp)
+; CHECK-NEXT:  .Ltmp3: # Block address taken
+; CHECK-NEXT:  .LBB2_3: # %label03
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # Parent Loop BB2_2 Depth=2
+; CHECK-NEXT:    # => This Loop Header: Depth=3
+; CHECK-NEXT:    # Child Loop BB2_4 Depth 4
+; CHECK-NEXT:    .p2align 4, 0x90
+; CHECK-NEXT:  .Ltmp4: # Block address taken
+; CHECK-NEXT:  .LBB2_4: # %label04
+; CHECK-NEXT:    # Parent Loop BB2_1 Depth=1
+; CHECK-NEXT:    # Parent Loop BB2_2 Depth=2
+; CHECK-NEXT:    # Parent Loop BB2_3 Depth=3
+; CHECK-NEXT:    # => This Inner Loop Header: Depth=4
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    jmp .Ltmp10
+; CHECK-NEXT:    jmp .Ltmp20
+; CHECK-NEXT:    jmp .Ltmp30
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  .LBB2_5: # %normal0
+; CHECK-NEXT:    # in Loop: Header=BB2_4 Depth=4
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    jmp .Ltmp10
+; CHECK-NEXT:    jmp .Ltmp20
+; CHECK-NEXT:    jmp .Ltmp30
+; CHECK-NEXT:    jmp .Ltmp40
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:  .LBB2_6: # %normal1
+; CHECK-NEXT:    movl {{[0-9]+}}(%esp), %eax
+; CHECK-NEXT:    retl
+entry:
+  %a.addr = alloca i32, align 4
+  store i32 %a, i32* %a.addr, align 4
+  br label %label01
+
+label01:                                          ; preds = %normal0, %label04, %entry
+  br label %label02
+
+label02:                                          ; preds = %normal0, %label04, %label01
+  %0 = load i32, i32* %a.addr, align 4
+  %add = add nsw i32 %0, 4
+  store i32 %add, i32* %a.addr, align 4
+  br label %label03
+
+label03:                                          ; preds = %normal0, %label04, %label02
+  br label %label04
+
+label04:                                          ; preds = %normal0, %label03
+  callbr void asm sideeffect "jmp ${0:l}; jmp ${1:l}; jmp ${2:l}", "X,X,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test3, %label01), i8* blockaddress(@test3, %label02), i8* blockaddress(@test3, %label03))
+          to label %normal0 [label %label01, label %label02, label %label03]
+
+normal0:                                          ; preds = %label04
+  callbr void asm sideeffect "jmp ${0:l}; jmp ${1:l}; jmp ${2:l}; jmp ${3:l}", "X,X,X,X,~{dirflag},~{fpsr},~{flags}"(i8* blockaddress(@test3, %label01), i8* blockaddress(@test3, %label02), i8* blockaddress(@test3, %label03), i8* blockaddress(@test3, %label04))
+          to label %normal1 [label %label01, label %label02, label %label03, label %label04]
+
+normal1:                                          ; preds = %normal0
+  %1 = load i32, i32* %a.addr, align 4
+  ret i32 %1
+}

llvm/trunk/test/Transforms/GVN/callbr-loadpre-critedge.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -gvn -S | FileCheck %s
+
+; This test checks that we don't hang trying to split a critical edge in loadpre
+; when the control flow uses a callbr instruction.
+
+%struct.pluto = type <{ i8, i8 }>
+
+define void @widget(%struct.pluto** %tmp1) {
+; CHECK-LABEL: @widget(
+; CHECK-NEXT:  bb:
+; CHECK-NEXT:    callbr void asm sideeffect "", "X,X"(i8* blockaddress(@widget, [[BB5:%.*]]), i8* blockaddress(@widget, [[BB8:%.*]]))
+; CHECK-NEXT:    to label [[BB4:%.*]] [label [[BB5]], label %bb8]
+; CHECK:       bb4:
+; CHECK-NEXT:    br label [[BB5]]
+; CHECK:       bb5:
+; CHECK-NEXT:    [[TMP6:%.*]] = load %struct.pluto*, %struct.pluto** [[TMP1:%.*]]
+; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr inbounds [[STRUCT_PLUTO:%.*]], %struct.pluto* [[TMP6]], i64 0, i32 1
+; CHECK-NEXT:    br label [[BB8]]
+; CHECK:       bb8:
+; CHECK-NEXT:    [[TMP9:%.*]] = phi i8* [ [[TMP7]], [[BB5]] ], [ null, [[BB:%.*]] ]
+; CHECK-NEXT:    [[TMP10:%.*]] = load %struct.pluto*, %struct.pluto** [[TMP1]]
+; CHECK-NEXT:    [[TMP11:%.*]] = getelementptr inbounds [[STRUCT_PLUTO]], %struct.pluto* [[TMP10]], i64 0, i32 0
+; CHECK-NEXT:    [[TMP12:%.*]] = load i8, i8* [[TMP11]]
+; CHECK-NEXT:    tail call void @spam(i8* [[TMP9]], i8 [[TMP12]])
+; CHECK-NEXT:    ret void
+;
+bb:
+  callbr void asm sideeffect "", "X,X"(i8* blockaddress(@widget, %bb5), i8* blockaddress(@widget, %bb8))
+  to label %bb4 [label %bb5, label %bb8]
+
+bb4:                                              ; preds = %bb
+  br label %bb5
+
+bb5:                                              ; preds = %bb4, %bb
+  %tmp6 = load %struct.pluto*, %struct.pluto** %tmp1
+  %tmp7 = getelementptr inbounds %struct.pluto, %struct.pluto* %tmp6, i64 0, i32 1
+  br label %bb8
+
+bb8:                                              ; preds = %bb5, %bb
+  %tmp9 = phi i8* [ %tmp7, %bb5 ], [ null, %bb ]
+  %tmp10 = load %struct.pluto*, %struct.pluto** %tmp1
+  %tmp11 = getelementptr inbounds %struct.pluto, %struct.pluto* %tmp10, i64 0, i32 0
+  %tmp12 = load i8, i8* %tmp11
+  tail call void @spam(i8* %tmp9, i8 %tmp12)
+  ret void
+}
+
+declare void @spam(i8*, i8)

llvm/trunk/test/Transforms/GVN/callbr-scalarpre-critedge.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -gvn -S | FileCheck %s
+
+; This test checks that we don't hang trying to split a critical edge in scalar
+; PRE when the control flow uses a callbr instruction.
+
+define void @wombat(i64 %arg, i64* %arg1, i64 %arg2, i32* %arg3) {
+; CHECK-LABEL: @wombat(
+; CHECK-NEXT:  bb:
+; CHECK-NEXT:    [[TMP5:%.*]] = or i64 [[ARG2:%.*]], [[ARG:%.*]]
+; CHECK-NEXT:    callbr void asm sideeffect "", "X,X"(i8* blockaddress(@wombat, [[BB7:%.*]]), i8* blockaddress(@wombat, [[BB9:%.*]]))
+; CHECK-NEXT:    to label [[BB6:%.*]] [label [[BB7]], label %bb9]
+; CHECK:       bb6:
+; CHECK-NEXT:    br label [[BB7]]
+; CHECK:       bb7:
+; CHECK-NEXT:    [[TMP8:%.*]] = trunc i64 [[TMP5]] to i32
+; CHECK-NEXT:    tail call void @barney(i32 [[TMP8]])
+; CHECK-NEXT:    br label [[BB9]]
+; CHECK:       bb9:
+; CHECK-NEXT:    [[TMP10:%.*]] = trunc i64 [[TMP5]] to i32
+; CHECK-NEXT:    store i32 [[TMP10]], i32* [[ARG3:%.*]]
+; CHECK-NEXT:    ret void
+;
+bb:
+  %tmp5 = or i64 %arg2, %arg
+  callbr void asm sideeffect "", "X,X"(i8* blockaddress(@wombat, %bb7), i8* blockaddress(@wombat, %bb9))
+          to label %bb6 [label %bb7, label %bb9]
+
+bb6:                                              ; preds = %bb
+  br label %bb7
+
+bb7:                                              ; preds = %bb6, %bb
+  %tmp8 = trunc i64 %tmp5 to i32
+  tail call void @barney(i32 %tmp8)
+  br label %bb9
+
+bb9:                                              ; preds = %bb7, %bb
+  %tmp10 = trunc i64 %tmp5 to i32
+  store i32 %tmp10, i32* %arg3
+  ret void
+}
+
+declare void @barney(i32)

llvm/trunk/test/Transforms/JumpThreading/callbr-edge-split.ll

+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -S -jump-threading | FileCheck %s
+
+; This test used to cause jump threading to try to split an edge of a callbr.
+
+@a = global i32 0
+
+define i32 @c() {
+; CHECK-LABEL: @c(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* @a
+; CHECK-NEXT:    [[TOBOOL:%.*]] = icmp eq i32 [[TMP0]], 0
+; CHECK-NEXT:    br i1 [[TOBOOL]], label [[IF_ELSE:%.*]], label [[IF_THEN:%.*]]
+; CHECK:       if.then:
+; CHECK-NEXT:    [[CALL:%.*]] = call i32 @b()
+; CHECK-NEXT:    [[PHITMP:%.*]] = icmp ne i32 [[CALL]], 0
+; CHECK-NEXT:    br i1 [[PHITMP]], label [[IF_THEN2:%.*]], label [[IF_END4:%.*]]
+; CHECK:       if.else:
+; CHECK-NEXT:    callbr void asm sideeffect "", "X"(i8* blockaddress(@c, [[IF_THEN2]]))
+; CHECK-NEXT:    to label [[IF_END_THREAD:%.*]] [label %if.then2]
+; CHECK:       if.end.thread:
+; CHECK-NEXT:    br label [[IF_THEN2]]
+; CHECK:       if.then2:
+; CHECK-NEXT:    [[CALL3:%.*]] = call i32 @b()
+; CHECK-NEXT:    br label [[IF_END4]]
+; CHECK:       if.end4:
+; CHECK-NEXT:    ret i32 undef
+;
+entry:
+  %0 = load i32, i32* @a
+  %tobool = icmp eq i32 %0, 0
+  br i1 %tobool, label %if.else, label %if.then
+
+if.then:                                          ; preds = %entry
+  %call = call i32 @b() #2
+  %phitmp = icmp ne i32 %call, 0
+  br label %if.end
+
+if.else:                                          ; preds = %entry
+  callbr void asm sideeffect "", "X"(i8* blockaddress(@c, %if.end)) #2
+  to label %normal [label %if.end]
+
+normal:                                           ; preds = %if.else
+  br label %if.end
+
+if.end:                                           ; preds = %if.else, %normal, %if.then
+  %d.0 = phi i1 [ %phitmp, %if.then ], [ undef, %normal ], [ undef, %if.else ]
+  br i1 %d.0, label %if.then2, label %if.end4
+
+if.then2:                                         ; preds = %if.end
+  %call3 = call i32 @b()
+  br label %if.end4
+
+if.end4:                                          ; preds = %if.then2, %if.end
+  ret i32 undef
+}
+
+declare i32 @b()

llvm/trunk/test/Transforms/MergeFunc/call-and-invoke-with-ranges.ll

 next:
   ret i8 %out
 
 lpad:
   %pad = landingpad { i8*, i32 } cleanup
   resume { i8*, i32 } zeroinitializer
 }
 
-define i8 @call_with_same_range() {
-; CHECK-LABEL: @call_with_same_range
-; CHECK: tail call i8 @call_with_range
-  bitcast i8 0 to i8
-  %out = call i8 @dummy(), !range !0
-  ret i8 %out
-}
-
 define i8 @invoke_with_same_range() personality i8* undef {
 ; CHECK-LABEL: @invoke_with_same_range()
 ; CHECK: tail call i8 @invoke_with_range()
   %out = invoke i8 @dummy() to label %next unwind label %lpad, !range !0
 
 next:
   ret i8 %out
 
 lpad:
   %pad = landingpad { i8*, i32 } cleanup
   resume { i8*, i32 } zeroinitializer
 }
 
+define i8 @call_with_same_range() {
+; CHECK-LABEL: @call_with_same_range
+; CHECK: tail call i8 @call_with_range
+  bitcast i8 0 to i8
+  %out = call i8 @dummy(), !range !0
+  ret i8 %out
+}
 
 
 declare i8 @dummy();
 declare i32 @__gxx_personality_v0(...)
 
 !0 = !{i8 0, i8 2}
 !1 = !{i8 5, i8 7}

llvm/trunk/test/Transforms/MergeFunc/inline-asm.ll

 ; RUN: opt -mergefunc -S < %s | FileCheck %s
 
 ; CHECK-LABEL: @int_ptr_arg_different
 ; CHECK-NEXT: call void asm
 
+; CHECK-LABEL: @int_ptr_null
+; CHECK-NEXT: tail call void @float_ptr_null()
+
 ; CHECK-LABEL: @int_ptr_arg_same
 ; CHECK-NEXT: %2 = bitcast i32* %0 to float*
 ; CHECK-NEXT: tail call void @float_ptr_arg_same(float* %2)
 
-; CHECK-LABEL: @int_ptr_null
-; CHECK-NEXT: tail call void @float_ptr_null()
-
 ; Used to satisfy minimum size limit
 declare void @stuff()
 
 ; Can be merged
 define void @float_ptr_null() {
   call void asm "nop", "r"(float* null)
   call void @stuff()
   ret void

llvm/trunk/tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

default: return nullptr;
       STRINGIFY_CODE(FUNC_CODE, DEBUG_LOC)
       STRINGIFY_CODE(FUNC_CODE, INST_GEP)
       STRINGIFY_CODE(FUNC_CODE, OPERAND_BUNDLE)
       STRINGIFY_CODE(FUNC_CODE, INST_FENCE)
       STRINGIFY_CODE(FUNC_CODE, INST_ATOMICRMW)
       STRINGIFY_CODE(FUNC_CODE, INST_LOADATOMIC)
       STRINGIFY_CODE(FUNC_CODE, INST_STOREATOMIC)
       STRINGIFY_CODE(FUNC_CODE, INST_CMPXCHG)
+      STRINGIFY_CODE(FUNC_CODE, INST_CALLBR)
     }
   case bitc::VALUE_SYMTAB_BLOCK_ID:
     switch (CodeID) {
     default: return nullptr;
     STRINGIFY_CODE(VST_CODE, ENTRY)
     STRINGIFY_CODE(VST_CODE, BBENTRY)
     STRINGIFY_CODE(VST_CODE, FNENTRY)
     STRINGIFY_CODE(VST_CODE, COMBINED_ENTRY)

llvm/trunk/utils/vim/syntax/llvm.vim

 syn keyword llvmType label metadata x86_mmx
 syn keyword llvmType type label opaque token
 syn match   llvmType /\<i\d\+\>/
 
 " Instructions.
 " The true and false tokens can be used for comparison opcodes, but it's
 " much more common for these tokens to be used for boolean constants.
 syn keyword llvmStatement add addrspacecast alloca and arcp ashr atomicrmw
-syn keyword llvmStatement bitcast br catchpad catchswitch catchret call
+syn keyword llvmStatement bitcast br catchpad catchswitch catchret call callbr
 syn keyword llvmStatement cleanuppad cleanupret cmpxchg eq exact extractelement
 syn keyword llvmStatement extractvalue fadd fast fcmp fdiv fence fmul fpext
 syn keyword llvmStatement fptosi fptoui fptrunc free frem fsub fneg getelementptr
 syn keyword llvmStatement icmp inbounds indirectbr insertelement insertvalue
 syn keyword llvmStatement inttoptr invoke landingpad load lshr malloc max min
 syn keyword llvmStatement mul nand ne ninf nnan nsw nsz nuw oeq oge ogt ole
 syn keyword llvmStatement olt one or ord phi ptrtoint resume ret sdiv select
 syn keyword llvmStatement sext sge sgt shl shufflevector sitofp sle slt srem