[llvm][CallBrPrepare] add llvm.callbr.landingpad intrinsic

[nickdesaulniers]

Summary

Insert a new intrinsic call after splitting critical edges, and verify
it. Later commits will update the SSA values to use this new value along
indirect branches rather than the callbr's value, and have SelectionDAG
consume this new value.

Part 2b of
https://discourse.llvm.org/t/rfc-syncing-asm-goto-with-outputs-with-gcc/65453/8.

[nickdesaulniers]

llvm/lib/CodeGen/CallBrPrepare.cpp

94

This needs to insert intrinsics even when there is no critical edge to split?

[nickdesaulniers]

• insert intrinsic call regardless of critical edge splitting

[nickdesaulniers]

• move insertion of intrinsic calls out of SplitCriticalEdges (prefer small readable functions)

[nickdesaulniers]

• s/Synth/Dest

[nickdesaulniers]

llvm/include/llvm/IR/Intrinsics.td

623

Looks like I'll need to re-add IntrNoMerge; I'm trying to write tests for llc invocations and I see simplifycfg folding these again...

[nickdesaulniers]

• add IntrNoMerge

[aeubanks]

langref?

llvm/lib/CodeGen/CallBrPrepare.cpp

115

can pass this directly to the constructor

[nickdesaulniers]

llvm/lib/IR/Verifier.cpp

5856–5887

We should also check that the landingpad is the first instruction (isFirstNonPhi or non debug or some such). Otherwise you could have:

declare i64 @foo(i64)
define i64 @test_callbr_landingpad_not_first_inst() {
entry:
  ; Specifically, output into %rdi
  %0 = callbr i64 asm "", "={di},!i"()
          to label %asm.fallthrough [label %landingpad]

asm.fallthrough:
  ret i64 42

landingpad:
  ; By the calling convention on x86_64-linux-gnu, 42 must be passed in %rdi.
  %foo = call i64 @foo(i64 42)
  %out = call i64 @llvm.callbr.landingpad.i64()
  ret i64 %out
}

declare i64 @llvm.callbr.landingpad.i64()

[nickdesaulniers]

• langref, remove Builder::SetInsertPoint call, add verifification test

[nickdesaulniers]

Ready for review:

1. https://reviews.llvm.org/D135997
2. https://reviews.llvm.org/D140166
3. https://reviews.llvm.org/D139861
4. https://reviews.llvm.org/D139872
5. https://reviews.llvm.org/D139883
6. https://reviews.llvm.org/D139970
7. https://reviews.llvm.org/D140160
8. https://reviews.llvm.org/D137113
9. https://reviews.llvm.org/D136497

[void]

As we talked before, I *really* hate the use of an intrinsic here. We used an intrinsic for the old exception handling and it was nothing but buggy. Theoretically they work just fine, but the can't take into account issues due to inlining, block splitting, and other code motion changes (at least not without a hefty dose of special checks throughout the compiler). If we use an intrinsic here, I believe we're just going to revisit the errors of the past.

[nickdesaulniers]

In D139883#4011907, @void wrote:

As we talked before, I *really* hate the use of an intrinsic here. We used an intrinsic for the old exception handling and it was nothing but buggy. Theoretically they work just fine, but the can't take into account issues due to inlining, block splitting, and other code motion changes (at least not without a hefty dose of special checks throughout the compiler). If we use an intrinsic here, I believe we're just going to revisit the errors of the past.

The use of an instruction was tried here: https://reviews.llvm.org/D139565.

Specifically, please see the feedback:

• https://reviews.llvm.org/D139565#3979289
• https://reviews.llvm.org/D139565#3982939

While I agree with your points about intrinsic vs instruction; with the approach of the series (as mentioned in the RFC), I'll note that the intrinsic doesn't live long enough to be relevant to inlining, block splitting, or other code motion changes. It is produced by callbrprepare shortly before being consumed by selectiondag.

Please see the pipeline test changes in https://reviews.llvm.org/D140180 to get a sense of the pass "distance" between Prepare callbr (which introduces these intrinsics) and <arch> DAG->DAG Pattern Instruction Selection which consumes them.

I need @void and @efriedma (and me and maybe @jyknight ) to be in agreement as the feedback around instruction vs intrinsic is giving me whiplash in code review. It's up to you two to agree; I can't have @void reject intrinsics while @efriedma rejects instructions.

[void]

In D139883#4012018, @nickdesaulniers wrote:

In D139883#4011907, @void wrote:

As we talked before, I *really* hate the use of an intrinsic here. We used an intrinsic for the old exception handling and it was nothing but buggy. Theoretically they work just fine, but the can't take into account issues due to inlining, block splitting, and other code motion changes (at least not without a hefty dose of special checks throughout the compiler). If we use an intrinsic here, I believe we're just going to revisit the errors of the past.

The use of an instruction was tried here: https://reviews.llvm.org/D139565.

Specifically, please see the feedback:

• https://reviews.llvm.org/D139565#3979289
• https://reviews.llvm.org/D139565#3982939

While I agree with your points about intrinsic vs instruction; with the approach of the series (as mentioned in the RFC), I'll note that the intrinsic doesn't live long enough to be relevant to inlining, block splitting, or other code motion changes. It is produced by callbrprepare shortly before being consumed by selectiondag.

Please see the pipeline test changes in https://reviews.llvm.org/D140180 to get a sense of the pass "distance" between Prepare callbr (which introduces these intrinsics) and <arch> DAG->DAG Pattern Instruction Selection which consumes them.

I need @void and @efriedma (and me and maybe @jyknight ) to be in agreement as the feedback around instruction vs intrinsic is giving me whiplash in code review. It's up to you two to agree; I can't have @void reject intrinsics while @efriedma rejects instructions.

Thanks for the information. (And sorry I wasn't involved in those discussions.)

The main issue with the way EH used to be handled was that the intrinsic that held all of the information about what EH values are used, etc., could be moved away from the landing pad (via inlining and other normal code motion). It made associating the intrinsic with the proper invoke instruction(s) more-or-less impossible to get correct without having to jump through a ton of hoops. Creating the landingpad instruction, and forcing it to be the first non-PHI instruction in a BB, basically solved all EH issues that existed at that time.

So, if you can ensure that the intrinsic won't be moved around due to normal passes, or, if it is moved, that it won't matter (i.e., you'll be able to associate it with the callbr when necessary), then I'll withdraw my opposition to using an intrinsic.

[efriedma]

The use of an IR pass here is basically just a hack to work around the fact that SelectionDAG doesn't have a reasonable way to do this lowering inside SelectionDAG itself. If we were using GlobalISel exclusively, we would probably just implement this as a GlobalISel pass instead of an IR pass.

If we need some additional infrastructure to make the IR pass work reliably, we can look at that, but implementing an IR instruction that only exists for a couple passes inside codegen seems like overkill.

[efriedma]

llvm/docs/LangRef.rst

13714 ↗

(On Diff #484635)

I'm not sure we need to document this at all (it's an unstable implementation detail users of LLVM shouldn't need to be aware of). But if you do think it's necessary to document it, please call out that it's an unstable implementation detail, and users should not call it.

[void]

In D139883#4024498, @efriedma wrote:

The use of an IR pass here is basically just a hack to work around the fact that SelectionDAG doesn't have a reasonable way to do this lowering inside SelectionDAG itself. If we were using GlobalISel exclusively, we would probably just implement this as a GlobalISel pass instead of an IR pass.

If we need some additional infrastructure to make the IR pass work reliably, we can look at that, but implementing an IR instruction that only exists for a couple passes inside codegen seems like overkill.

Thanks. Okay, I agree with @efriedma and you that an intrinsic should work just fine here. It seems like the lifespan of this intrinsic is going to be very short, so much so that few if any IR changes will occur.

[nickdesaulniers]

• rebase, format, remove intrinsic docs as per @efriedma

[nickdesaulniers]

• rebase

[efriedma]

LGTM

[void]

llvm/lib/IR/Verifier.cpp

5852

Please add a break; at the end of this case.

[nickdesaulniers]

• add missing break statement, as per @void

[jyknight]

llvm/lib/CodeGen/CallBrPrepare.cpp

113

Doesn't seem like much point in having this a separate variable, can just pass directly to CreateIntrinsic. And I'd merge this fn into InsertIntrinsicCalls -- with construction of Builder outside the loop, and SetInsertPoint+CreateIntrinsic in the loop.

llvm/test/Transforms/SimplifyCFG/callbr-destinations.ll

62

In the expected pass pipeline, do we simplifycfg after callbrprepare? That seems a bit surprising if so?

(Not that it's wrong to mark it IntrNoMerge, of course.)

[nickdesaulniers]

• inline InsertIntrinsicCall, as per @jyknight

llvm/test/Transforms/SimplifyCFG/callbr-destinations.ll

62

We do not; this is an artifact from when the frontend would produce the intrinsic at which point we would need to guard against SimplifyCFG merging two basic blocks that just contained this intrinsic call.

I can remove it if you feel that it's unnecessary to retain this guard. WDYT?

[nickdesaulniers]

• rebase (add header I no longer needed to add earlier but do now)

[jyknight]

llvm/test/Transforms/SimplifyCFG/callbr-destinations.ll

62

Oh I see. No, fine as is.

[nickdesaulniers]

• rebase

[nickdesaulniers]

• merge D142940, final rebase

llvm/include/llvm/IR/Intrinsics.td

def int_instrprof_value_profile : Intrinsic<[],
                                             [llvm_ptr_ty, llvm_i64_ty,
                                              llvm_i64_ty, llvm_i32_ty,
                                              llvm_i32_ty]>;
 
 def int_call_preallocated_setup : DefaultAttrsIntrinsic<[llvm_token_ty], [llvm_i32_ty]>;
 def int_call_preallocated_arg : DefaultAttrsIntrinsic<[llvm_ptr_ty], [llvm_token_ty, llvm_i32_ty]>;
 def int_call_preallocated_teardown : DefaultAttrsIntrinsic<[], [llvm_token_ty]>;
 
+// This intrinsic is intentionally undocumented and users shouldn't call it;
+// it's produced then quickly consumed during codegen.
+def int_callbr_landingpad : Intrinsic<[llvm_any_ty], [LLVMMatchType<0>],
+                                      [IntrNoMerge]>;
+
 //===------------------- Standard C Library Intrinsics --------------------===//
 //
 
 def int_memcpy  : Intrinsic<[],
                             [llvm_anyptr_ty, llvm_anyptr_ty, llvm_anyint_ty,
                              llvm_i1_ty],
                             [IntrArgMemOnly, IntrWillReturn, IntrNoFree,
                              IntrNoCallback,

llvm/lib/CodeGen/CallBrPrepare.cpp

 // MachineInstruction representation, without the use of an IR-level intrinsic.
 // But, within the current framework, it’s simpler to implement as an IR pass.
 // (If support for callbr in GlobalISel is implemented, it’s worth considering
 // whether this is still required.)
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Analysis/CFG.h"
 #include "llvm/CodeGen/Passes.h"
 #include "llvm/IR/BasicBlock.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/Function.h"
+#include "llvm/IR/IRBuilder.h"
 #include "llvm/IR/Instructions.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Intrinsics.h"
 #include "llvm/InitializePasses.h"
 #include "llvm/Pass.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 
 using namespace llvm;
 
 #define DEBUG_TYPE "callbrprepare"
 

 }
 
 bool CallBrPrepare::SplitCriticalEdges(ArrayRef<CallBrInst *> CBRs,
                                        DominatorTree &DT) {
   bool Changed = false;
   CriticalEdgeSplittingOptions Options(&DT);
   Options.setMergeIdenticalEdges();
 
   // The indirect destination might be duplicated between another parameter...
   //   %0 = callbr ... [label %x, label %x]
   // ...hence MergeIdenticalEdges and AllowIndentical edges, but we don't need
   // to split the default destination if it's duplicated between an indirect
   // destination...
   //   %1 = callbr ... to label %x [label %x]
   // ...hence starting at 1 and checking against successor 0 (aka the default
   // destination).
   for (CallBrInst *CBR : CBRs)
     for (unsigned i = 1, e = CBR->getNumSuccessors(); i != e; ++i)
       if (CBR->getSuccessor(i) == CBR->getSuccessor(0) ||
           isCriticalEdge(CBR, i, /*AllowIdenticalEdges*/ true))
         if (SplitKnownCriticalEdge(CBR, i, Options))
           Changed = true;
   return Changed;
 }
 
+static bool InsertIntrinsicCalls(ArrayRef<CallBrInst *> CBRs) {
+  bool Changed = false;
+  SmallPtrSet<const BasicBlock *, 4> Visited;
+  IRBuilder<> Builder(CBRs[0]->getContext());
+  for (CallBrInst *CBR : CBRs) {
+    for (BasicBlock *IndDest : CBR->getIndirectDests()) {
+      if (!Visited.insert(IndDest).second)
+        continue;
+      Builder.SetInsertPoint(&*IndDest->begin());
+      Builder.CreateIntrinsic(CBR->getType(), Intrinsic::callbr_landingpad,
+                              {CBR});
+      Changed = true;
+    }
+  }
+  return Changed;
+}
+
 bool CallBrPrepare::runOnFunction(Function &Fn) {
   bool Changed = false;
   SmallVector<CallBrInst *, 2> CBRs = FindCallBrs(Fn);
 
   if (CBRs.empty())
     return Changed;
 
   // It's highly likely that most programs do not contain CallBrInsts. Follow a
bool CallBrPrepare::runOnFunction(Function &Fn) {
   else {
     LazilyComputedDomTree.emplace(Fn);
     DT = &*LazilyComputedDomTree;
   }
 
   if (SplitCriticalEdges(CBRs, *DT))
     Changed = true;
 
+  if (InsertIntrinsicCalls(CBRs))
+    Changed = true;
+
   return Changed;
 }

llvm/lib/IR/Verifier.cpp

case Intrinsic::aarch64_prefetch: {
     Check(cast<ConstantInt>(Call.getArgOperand(2))->getZExtValue() < 4,
           "target argument to llvm.aarch64.prefetch must be 0-3", Call);
     Check(cast<ConstantInt>(Call.getArgOperand(3))->getZExtValue() < 2,
           "stream argument to llvm.aarch64.prefetch must be 0 or 1", Call);
     Check(cast<ConstantInt>(Call.getArgOperand(4))->getZExtValue() < 2,
           "isdata argument to llvm.aarch64.prefetch must be 0 or 1", Call);
     break;
   }
+  case Intrinsic::callbr_landingpad: {
+    const auto *CBR = dyn_cast<CallBrInst>(Call.getOperand(0));
+    Check(CBR, "intrinstic requires callbr operand", &Call);
+    if (!CBR)
+      break;
+
+    const BasicBlock *LandingPadBB = Call.getParent();
+    const BasicBlock *PredBB = LandingPadBB->getUniquePredecessor();
+    if (!PredBB) {
+      CheckFailed("Intrinsic in block must have 1 unique predecessor", &Call);
+      break;
+    }
+    if (!isa<CallBrInst>(PredBB->getTerminator())) {
+      CheckFailed("Intrinsic must have corresponding callbr in predecessor",
+                  &Call);
+      break;
+    }
+    Check(llvm::any_of(CBR->getIndirectDests(),
+                       [LandingPadBB](const BasicBlock *IndDest) {
+                         return IndDest == LandingPadBB;
+                       }),
+          "Intrinsic's corresponding callbr must have intrinsic's parent basic "
+          "block in indirect destination list",
+          &Call);
+    const Instruction &First = *LandingPadBB->begin();
+    Check(&First == &Call, "No other instructions may proceed intrinsic",
+          &Call);
+    break;
+  }
   };
 
   // Verify that there aren't any unmediated control transfers between funclets.
   if (IntrinsicInst::mayLowerToFunctionCall(ID)) {
     Function *F = Call.getParent()->getParent();
     if (F->hasPersonalityFn() &&
         isScopedEHPersonality(classifyEHPersonality(F->getPersonalityFn()))) {
       // Run EH funclet coloring on-demand and cache results for other intrinsic
       // calls in this function
       if (BlockEHFuncletColors.empty())
         BlockEHFuncletColors = colorEHFunclets(*F);
 
       // Check for catch-/cleanup-pad in first funclet block
       bool InEHFunclet = false;
       BasicBlock *CallBB = Call.getParent();
       const ColorVector &CV = BlockEHFuncletColors.find(CallBB)->second;
       assert(CV.size() > 0 && "Uncolored block");
       for (BasicBlock *ColorFirstBB : CV)
         if (dyn_cast_or_null<FuncletPadInst>(ColorFirstBB->getFirstNonPHI()))
           InEHFunclet = true;
 
       // Check for funclet operand bundle
       bool HasToken = false;
       for (unsigned I = 0, E = Call.getNumOperandBundles(); I != E; ++I)
         if (Call.getOperandBundleAt(I).getTagID() == LLVMContext::OB_funclet)
           HasToken = true;
 
       // This would cause silent code truncation in WinEHPrepare
       if (InEHFunclet)
         Check(HasToken, "Missing funclet token on intrinsic call", &Call);
     }
   }
 }
 
 /// Carefully grab the subprogram from a local scope.
 ///
 /// This carefully grabs the subprogram from a local scope, avoiding the
 /// built-in assertions that would typically fire.
 static DISubprogram *getSubprogram(Metadata *LocalScope) {
   if (!LocalScope)

llvm/test/CodeGen/AArch64/callbr-prepare.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt %s -callbrprepare -S -o - | FileCheck %s
 
 define i32 @test0() {
 ; CHECK-LABEL: @test0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[OUT:%.*]] = callbr i32 asm "# $0", "=r,!i"()
 ; CHECK-NEXT:    to label [[DIRECT:%.*]] [label %entry.indirect_crit_edge]
 ; CHECK:       entry.indirect_crit_edge:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[OUT]])
 ; CHECK-NEXT:    br label [[INDIRECT:%.*]]
 ; CHECK:       direct:
 ; CHECK-NEXT:    [[OUT2:%.*]] = callbr i32 asm "# $0", "=r,!i"()
 ; CHECK-NEXT:    to label [[DIRECT2:%.*]] [label %direct.indirect_crit_edge]
 ; CHECK:       direct.indirect_crit_edge:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[OUT2]])
 ; CHECK-NEXT:    br label [[INDIRECT]]
 ; CHECK:       direct2:
 ; CHECK-NEXT:    ret i32 0
 ; CHECK:       indirect:
 ; CHECK-NEXT:    [[OUT3:%.*]] = phi i32 [ [[OUT]], [[ENTRY_INDIRECT_CRIT_EDGE:%.*]] ], [ [[OUT2]], [[DIRECT_INDIRECT_CRIT_EDGE:%.*]] ]
 ; CHECK-NEXT:    ret i32 [[OUT3]]
 ;
 entry:

 define i32 @dont_split1() {
 ; CHECK-LABEL: @dont_split1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i"()
 ; CHECK-NEXT:    to label [[X:%.*]] [label %y]
 ; CHECK:       x:
 ; CHECK-NEXT:    ret i32 42
 ; CHECK:       y:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[TMP0]])
 ; CHECK-NEXT:    ret i32 [[TMP0]]
 ;
 entry:
   %0 = callbr i32 asm "", "=r,!i"()
   to label %x [label %y]
 
 x:
   ret i32 42

 ; that output is used.
 ; Here we have output and a critical edge along an indirect branch.
 define i32 @split_me0() {
 ; CHECK-LABEL: @split_me0(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i"()
 ; CHECK-NEXT:    to label [[X:%.*]] [label %entry.y_crit_edge]
 ; CHECK:       entry.y_crit_edge:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[TMP0]])
 ; CHECK-NEXT:    br label [[Y:%.*]]
 ; CHECK:       x:
 ; CHECK-NEXT:    br label [[Y]]
 ; CHECK:       y:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi i32 [ [[TMP0]], [[ENTRY_Y_CRIT_EDGE:%.*]] ], [ 42, [[X]] ]
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[TMP2:%.*]] = phi i32 [ [[TMP0]], [[ENTRY_Y_CRIT_EDGE:%.*]] ], [ 42, [[X]] ]
+; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
 entry:
   %0 = callbr i32 asm "", "=r,!i"()
   to label %x [label %y]
 
 x:
   br label %y
 
 y:
   %1 = phi i32 [ %0, %entry ], [ 42, %x ]
   ret i32 %1
 }
 
 ; Here we have output and a critical edge along an indirect branch.
 ; Ensure that if we repeat the indirect destination, that we only split it
 ; once.
 define i32 @split_me1(i1 %z) {
 ; CHECK-LABEL: @split_me1(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[Z:%.*]], label [[W:%.*]], label [[V:%.*]]
 ; CHECK:       w:
 ; CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,!i"()
 ; CHECK-NEXT:    to label [[X:%.*]] [label [[W_V_CRIT_EDGE:%.*]], label %w.v_crit_edge]
 ; CHECK:       w.v_crit_edge:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[TMP0]])
 ; CHECK-NEXT:    br label [[V]]
 ; CHECK:       x:
 ; CHECK-NEXT:    ret i32 42
 ; CHECK:       v:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi i32 [ [[TMP0]], [[W_V_CRIT_EDGE]] ], [ undef, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[TMP2:%.*]] = phi i32 [ [[TMP0]], [[W_V_CRIT_EDGE]] ], [ undef, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
 entry:
   br i1 %z, label %w, label %v
 
 w:
   %0 = callbr i32 asm "", "=r,!i,!i"()
   to label %x [label %v, label %v]
 

 define i32 @split_me2(i1 %z) {
 ; CHECK-LABEL: @split_me2(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br i1 [[Z:%.*]], label [[W:%.*]], label [[V:%.*]]
 ; CHECK:       w:
 ; CHECK-NEXT:    [[TMP0:%.*]] = callbr i32 asm "", "=r,!i,!i"()
 ; CHECK-NEXT:    to label [[X:%.*]] [label [[W_V_CRIT_EDGE:%.*]], label %w.v_crit_edge]
 ; CHECK:       w.v_crit_edge:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[TMP0]])
 ; CHECK-NEXT:    br label [[V]]
 ; CHECK:       x:
 ; CHECK-NEXT:    ret i32 42
 ; CHECK:       v:
-; CHECK-NEXT:    [[TMP1:%.*]] = phi i32 [ [[TMP0]], [[W_V_CRIT_EDGE]] ], [ 42, [[ENTRY:%.*]] ]
-; CHECK-NEXT:    ret i32 [[TMP1]]
+; CHECK-NEXT:    [[TMP2:%.*]] = phi i32 [ [[TMP0]], [[W_V_CRIT_EDGE]] ], [ 42, [[ENTRY:%.*]] ]
+; CHECK-NEXT:    ret i32 [[TMP2]]
 ;
 entry:
   br i1 %z, label %w, label %v
 
 w:
   %0 = callbr i32 asm "", "=r,!i,!i"()
   to label %x [label %v, label %v]
 
 x:
   ret i32 42
 
 v:
   %1 = phi i32 [ %0, %w ], [ 42, %entry ], [ %0, %w ]
   ret i32 %1
 }

llvm/test/Transforms/SimplifyCFG/callbr-destinations.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -S -passes=simplifycfg -simplifycfg-require-and-preserve-domtree=1 | FileCheck %s
+; RUN: opt < %s -S -passes="simplifycfg<sink-common-insts>" \
+; RUN:   -simplifycfg-require-and-preserve-domtree=1 | FileCheck %s
 
 define void @callbr_duplicate_dest() {
 ; CHECK-LABEL: @callbr_duplicate_dest(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    callbr void asm sideeffect "", "!i"()
 ; CHECK-NEXT:    to label [[BB:%.*]] [label %bb]
 ; CHECK:       bb:
 ; CHECK-NEXT:    ret void
bb1:
   ret void
 
 bb2:
   ret void
 
 bb3:
   ret void
 }
+
+; Validate that callbr landingpad intrinsics do not get merged (via the
+; IntrNoMerge attribute).
+define i32 @callbr_landingpad_nomerge() {
+; CHECK-LABEL: @callbr_landingpad_nomerge(
+; CHECK-NEXT:  entry:
+; CHECK-NEXT:    [[OUT:%.*]] = callbr i32 asm "# $0", "=r,!i"()
+; CHECK-NEXT:    to label [[DIRECT:%.*]] [label %entry.indirect_crit_edge]
+; CHECK:       entry.indirect_crit_edge:
+; CHECK-NEXT:    [[TMP0:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[OUT]])
+; CHECK-NEXT:    br label [[COMMON_RET:%.*]]
+; CHECK:       direct:
+; CHECK-NEXT:    [[OUT2:%.*]] = callbr i32 asm "# $0", "=r,!i"()
+; CHECK-NEXT:    to label [[COMMON_RET]] [label %direct.indirect_crit_edge]
+; CHECK:       direct.indirect_crit_edge:
+; CHECK-NEXT:    [[TMP1:%.*]] = call i32 @llvm.callbr.landingpad.i32(i32 [[OUT2]])
+; CHECK-NEXT:    br label [[COMMON_RET]]
+; CHECK:       common.ret:
+; CHECK-NEXT:    [[COMMON_RET_OP:%.*]] = phi i32 [ 0, [[DIRECT]] ], [ [[TMP0]], [[ENTRY_INDIRECT_CRIT_EDGE:%.*]] ], [ [[TMP1]], [[DIRECT_INDIRECT_CRIT_EDGE:%.*]] ]
+; CHECK-NEXT:    ret i32 [[COMMON_RET_OP]]
+;
+entry:
+  %out = callbr i32 asm "# $0", "=r,!i"()
+  to label %direct [label %entry.indirect_crit_edge]
+
+entry.indirect_crit_edge:
+  %0 = call i32 @llvm.callbr.landingpad.i32(i32 %out)
+  br label %indirect
+
+direct:
+  %out2 = callbr i32 asm "# $0", "=r,!i"()
+  to label %direct2 [label %direct.indirect_crit_edge]
+
+direct.indirect_crit_edge:
+  %1 = call i32 @llvm.callbr.landingpad.i32(i32 %out2)
+  br label %indirect
+
+direct2:
+  ret i32 0
+
+indirect:
+  %out3 = phi i32 [ %0, %entry.indirect_crit_edge ], [ %1, %direct.indirect_crit_edge ]
+  ret i32 %out3
+}
+
+declare i32 @llvm.callbr.landingpad.i32(i32)

llvm/test/Verifier/callbr.ll

entry:
   %ret = callbr i32 asm sideeffect "#test4", "=r,!i"() to label %normal [label %abnormal]
 
 normal:
   ret i32 0
 
 abnormal:
   ret i32 %ret
 }
+
+;; Tests of the callbr.landingpad intrinsic function.
+declare i32 @llvm.callbr.landingpad.i64(i64)
+define void @callbrpad_bad_type() {
+entry:
+; CHECK: Intrinsic has incorrect argument type!
+; CHECK-NEXT: ptr @llvm.callbr.landingpad.i64
+  %foo = call i32 @llvm.callbr.landingpad.i64(i64 42)
+  ret void
+}
+
+declare i32 @llvm.callbr.landingpad.i32(i32)
+define i32 @callbrpad_multi_preds() {
+entry:
+  %foo = callbr i32 asm "", "=r,!i"() to label %direct [label %indirect]
+direct:
+  br label %indirect
+indirect:
+; CHECK-NEXT: Intrinsic in block must have 1 unique predecessor
+; CHECK-NEXT: %out = call i32 @llvm.callbr.landingpad.i32(i32 %foo)
+  %out = call i32 @llvm.callbr.landingpad.i32(i32 %foo)
+  ret i32 %out
+}
+
+define void @callbrpad_wrong_callbr() {
+entry:
+  %foo = callbr i32 asm "", "=r,!i"() to label %direct [label %indirect]
+direct:
+; CHECK-NEXT: Intrinsic's corresponding callbr must have intrinsic's parent basic block in indirect destination list
+; CHECK-NEXT: %x = call i32 @llvm.callbr.landingpad.i32(i32 %foo)
+  %x = call i32 @llvm.callbr.landingpad.i32(i32 %foo)
+  ret void
+indirect:
+  ret void
+}
+
+declare i32 @foo(i32)
+define i32 @test_callbr_landingpad_not_first_inst() {
+entry:
+  %0 = callbr i32 asm "", "=r,!i"()
+          to label %asm.fallthrough [label %landingpad]
+
+asm.fallthrough:
+  ret i32 42
+
+landingpad:
+  %foo = call i32 @foo(i32 42)
+; CHECK-NEXT: No other instructions may proceed intrinsic
+; CHECK-NEXT: %out = call i32 @llvm.callbr.landingpad.i32(i32 %0)
+  %out = call i32 @llvm.callbr.landingpad.i32(i32 %0)
+  ret i32 %out
+}