Teach the inliner to track deoptimization state

[sanjoy]

Summary

This change teaches LLVM's inliner to track and suitably adjust
deoptimization state (tracked via deoptimization operand bundles) as it
inlines through call sites. The operation is described in more detail
in the LangRef changes.

Note:
The changes to CALLSITE_DELEGATE_GETTER were needed to allow
unification of the CallInst * and InvokeInst * types into an
Instruction * type in cloneWithOperandBundles.

[reames]

docs/LangRef.rst

1541 ↗

(On Diff #39848)

I'd add something here about it being the frontend responsibility to ensure that the bundle format is sufficient so that individual abstract frames can be reconstructed from the concatenated form if required.

include/llvm/IR/CallSite.h

215 ↗

(On Diff #39848)

This really looks like a white space change?

If you're going to introduce control flow, please use the "do { X } while(false)" idiom to make it less prone to miscompies.

include/llvm/IR/Instructions.h

1463 ↗

(On Diff #39848)

This comment is unclear. Is the argument list a white list? Is it an entirely new list of operand bundles? If so, where'd it come from?

lib/IR/Instructions.cpp

301 ↗

(On Diff #39848)

Don't we have an args_begin(), args_end?

Or better, an args() which returns an ArrayRef?

585 ↗

(On Diff #39848)

Same as above?

lib/Transforms/Utils/InlineFunction.cpp

211 ↗

(On Diff #39848)

This really seems like we need a getOperandBundles that returns an ArrayRef?

1039 ↗

(On Diff #39848)

I'd just inline this into the if and swap the comparison directions.

p.s. Don't you need to worry about a bundle type in the callee which is not deopt?

1151 ↗

(On Diff #39848)

I'm not sure about the order of the code here, but the fact we're replacing the same instruction twice (once for call to invoke, once for deopt bundles) seems likely to be error prone, particularly with ValueHandles being involved. Is there a way to avoid the complexity here? Don't have an obvious one myself, just asking.

1175 ↗

(On Diff #39848)

This is already enforced by the verifier, please remove.

1185 ↗

(On Diff #39848)

TagName? Why not tagID? And why not hard code OB_deopt here since it's implied by the surrounding code?

[sanjoy]

Replies inline.

Summarizing my plan to address the more complex review commments, I've proposed three post-commit changes inline:

• Change OperandBundleDef such that it can (alternative to containing an std::string tag name) contain an integral tag ID
• Change CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER to use the safer do { ... } while (false) idiom
• Introduce an iterator for getOperandBundleAt(I) and use that as much as possible

include/llvm/IR/CallSite.h

215 ↗

(On Diff #39848)

I'm just copying the pattern from CALLSITE_DELEGATE_SETTER. I agree that the do { ... } while (false) form is better -- is it okay if I fix CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER together in a separate checkin?

include/llvm/IR/Instructions.h

1463 ↗

(On Diff #39848)

It is an entirely new list of operand bundles. Will update the doc.

lib/IR/Instructions.cpp

301 ↗

(On Diff #39848)

They're there on CallSite, not on CallInst. Does changing this to

std::vector<...> Args = CallSite(this).args()

sound reasonable?

585 ↗

(On Diff #39848)

Same reply as above.

lib/Transforms/Utils/InlineFunction.cpp

211 ↗

(On Diff #39848)

That's difficult -- the object returned by getOperandBundleAt is a flyweight object (like StringRef etc.) that does not have a direct backing store (i.e. there is no internal array of OperandBundleUse objects) that I can create an ArrayRef<OperandBundleUse> with. I can definitely write a stateful iterator that iterates through all the operand bundles for a CallSite, but I've been waiting to do that sort of thing after bring-up.

1039 ↗

(On Diff #39848)

I'd just inline this into the if and swap the comparison directions.

SGTM.

p.s. Don't you need to worry about a bundle type in the callee which is not deopt?

The current semantics is that the caller's deopt state gets prepended to all of the "deopt" operand bundles in the callee. Call sites in the callee that have arbitrary deopt operand bundles are left intact. The test cases demonstrate this.

1151 ↗

(On Diff #39848)

I don't think there should be problems with ValueHandle s, but I do think this is somewhat inefficient. However, I don't think how I can fix that without changing CloneFunction to be aware of whether it is inlining into a call site that's a call instruction, or an invoke; and changing that is outside the scope of this patch.

1185 ↗

(On Diff #39848)

Right now the OperandBundleDef structure only carries an std::string that gets re-interned into LLVMContextImpl when the host instruction is constructed -- it has no way of holding a direct integer ID. This needs to be fixed (for performance, if nothing else), but doing that in this change will make it more complex. How about I change ChildOB.getTagName() to "deopt" for now, and in a later change add the functionality to create an OperandBundleDef holding a direct integral ID?

[sanjoy]

• address review

lib/Transforms/Utils/InlineFunction.cpp

1039 ↗

(On Diff #39848)

I'd just inline this into the if and swap the comparison directions.

I decided to keep this as-is (with a fix to the comment). I like how it reads currently -- "if CS has operand bundles attached, it better be of this form". Let me know if you feel strongly otherwise.

[sanjoy]

gentle ping!

[reames]

Responses to minor comments inline.

In D14552#286638, @sanjoy wrote:

• Change OperandBundleDef such that it can (alternative to containing an std::string tag name) contain an integral tag ID

Seems useful. SGTM

• Change CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER to use the safer do { ... } while (false) idiom

SGTM

• Introduce an iterator for getOperandBundleAt(I) and use that as much as possible

Not sure this is worth it.

lib/IR/Instructions.cpp

301 ↗

(On Diff #39876)

Yes.

lib/Transforms/Utils/InlineFunction.cpp

1153 ↗

(On Diff #39876)

Makes sense.

1187 ↗

(On Diff #39876)

SGTM

[reames]

include/llvm/IR/Instructions.h

1469 ↗

(On Diff #39876)

Offline, we decided a Create(InvokeInst, OperandBundles, InsertPt) would be a cleaner interface here.

lib/Transforms/Utils/InlineFunction.cpp

211–217 ↗

(On Diff #39876)

Offline we realized the confusion here. This loop is converting uses to defs, and is not simply a copy.

[sanjoy]

• address review

[sanjoy]

• use a literal "deopt" string as the tag

[reames]

LGTM w/comments addressed. I don't think this needs another round of review.

include/llvm/IR/Instructions.h

1463 ↗

(On Diff #40461)

The difference in insertion logic here is confusing and different from every other create function. Please stick to convention.

lib/Transforms/Utils/CloneFunction.cpp

386 ↗

(On Diff #40461)

CodeInfo might be null, please guard here and other as well.

lib/Transforms/Utils/InlineFunction.cpp

1171 ↗

(On Diff #40461)

minor: reserve final size

1175 ↗

(On Diff #40461)

Minor:
if (is_deopt) {}
else { }

might be clearer than if !deopt, continue

1185 ↗

(On Diff #40461)

minor: reserve the final size

llvm/trunk/docs/LangRef.rst

 From the compiler's perspective, deoptimization operand bundles make
 the call sites they're attached to at least ``readonly``.  They read
 through all of their pointer typed operands (even if they're not
 otherwise escaped) and the entire visible heap.  Deoptimization
 operand bundles do not capture their operands except during
 deoptimization, in which case control will not be returned to the
 compiled frame.
 
+The inliner knows how to inline through calls that have deoptimization
+operand bundles.  Just like inlining through a normal call site
+involves composing the normal and exceptional continuations, inlining
+through a call site with a deoptimization operand bundle needs to
+appropriately compose the "safe" deoptimization continuation.  The
+inliner does this by prepending the parent's deoptimization
+continuation to every deoptimization continuation in the inlined body.
+E.g. inlining ``@f`` into ``@g`` in the following example
+
+.. code-block:: llvm
+
+    define void @f() {
+      call void @x()  ;; no deopt state
+      call void @y() [ "deopt"(i32 10) ]
+      call void @y() [ "deopt"(i32 10), "unknown"(i8* null) ]
+      ret void
+    }
+
+    define void @g() {
+      call void @f() [ "deopt"(i32 20) ]
+      ret void
+    }
+
+will result in
+
+.. code-block:: llvm
+
+    define void @g() {
+      call void @x()  ;; still no deopt state
+      call void @y() [ "deopt"(i32 20, i32 10) ]
+      call void @y() [ "deopt"(i32 20, i32 10), "unknown"(i8* null) ]
+      ret void
+    }
+
+It is the frontend's responsibility to structure or encode the
+deoptimization state in a way that syntactically prepending the
+caller's deoptimization state to the callee's deoptimization state is
+semantically equivalent to composing the caller's deoptimization
+continuation after the callee's deoptimization continuation.
+
 .. _moduleasm:
 
 Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
 to the GCC "file scope inline asm" blocks. These blocks are internally
 concatenated by LLVM and treated as a single unit, but may be separated

llvm/trunk/include/llvm/IR/InstrTypes.h

 /// OperandBundleDef and ConstOperandBundleDef.
 template <typename InputTy> struct OperandBundleDefT {
   std::string Tag;
   std::vector<InputTy> Inputs;
 
   OperandBundleDefT() {}
   explicit OperandBundleDefT(StringRef Tag, const std::vector<InputTy> &Inputs)
       : Tag(Tag), Inputs(Inputs) {}
+
+  explicit OperandBundleDefT(StringRef Tag, std::vector<InputTy> &&Inputs)
+      : Tag(Tag), Inputs(Inputs) {}
+
+  explicit OperandBundleDefT(const OperandBundleUse &OBU) {
+    Tag = OBU.getTagName();
+    Inputs.insert(Inputs.end(), OBU.Inputs.begin(), OBU.Inputs.end());
+  }
 };
 
 typedef OperandBundleDefT<Value *> OperandBundleDef;
 typedef OperandBundleDefT<const Value *> ConstOperandBundleDef;
 
 /// \brief A mixin to add operand bundle functionality to llvm instruction
 /// classes.
 ///

llvm/trunk/include/llvm/IR/Instructions.h

public:
   static CallInst *Create(Value *F, const Twine &NameStr = "",
                           Instruction *InsertBefore = nullptr) {
     return new(1) CallInst(F, NameStr, InsertBefore);
   }
   static CallInst *Create(Value *F, const Twine &NameStr,
                           BasicBlock *InsertAtEnd) {
     return new(1) CallInst(F, NameStr, InsertAtEnd);
   }
+
+  /// \brief Create a clone of \p CI with a different set of operand bundles and
+  /// insert it before \p InsertPt.
+  ///
+  /// The returned call instruction is identical \p CI in every way except that
+  /// the operand bundles for the new instruction are set to the operand bundles
+  /// in \p Bundles.
+  static CallInst *Create(CallInst *CI, ArrayRef<OperandBundleDef> Bundles,
+                          Instruction *InsertPt = nullptr);
+
   /// CreateMalloc - Generate the IR for a call to malloc:
   /// 1. Compute the malloc call's argument as the specified type's size,
   ///    possibly multiplied by the array size if the array size is not
   ///    constant 1.
   /// 2. Call malloc with that argument.
   /// 3. Bitcast the result of the malloc call to the specified type.
   static Instruction *CreateMalloc(Instruction *InsertBefore,
                                    Type *IntPtrTy, Type *AllocTy,
static InvokeInst *Create(Value *Func, BasicBlock *IfNormal,
     unsigned Values = unsigned(Args.size()) + CountBundleInputs(Bundles) + 3;
     unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
 
     return new (Values, DescriptorBytes)
         InvokeInst(Func, IfNormal, IfException, Args, Bundles, Values, NameStr,
                    InsertAtEnd);
   }
 
+  /// \brief Create a clone of \p II with a different set of operand bundles and
+  /// insert it before \p InsertPt.
+  ///
+  /// The returned invoke instruction is identical to \p II in every way except
+  /// that the operand bundles for the new instruction are set to the operand
+  /// bundles in \p Bundles.
+  static InvokeInst *Create(InvokeInst *II, ArrayRef<OperandBundleDef> Bundles,
+                            Instruction *InsertPt = nullptr);
+
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   FunctionType *getFunctionType() const { return FTy; }
 
   void mutateFunctionType(FunctionType *FTy) {
     mutateType(FTy->getReturnType());
     this->FTy = FTy;

llvm/trunk/include/llvm/Transforms/Utils/Cloning.h

struct ClonedCodeInfo {
   bool ContainsCalls;
 
   /// ContainsDynamicAllocas - This is set to true if the cloned code contains
   /// a 'dynamic' alloca.  Dynamic allocas are allocas that are either not in
   /// the entry block or they are in the entry block but are not a constant
   /// size.
   bool ContainsDynamicAllocas;
 
+  /// All cloned call sites that have operand bundles attached are appended to
+  /// this vector.
+  std::vector<AssertingVH<Instruction>> OperandBundleCallSites;
+
   ClonedCodeInfo() : ContainsCalls(false), ContainsDynamicAllocas(false) {}
 };
 
 /// CloneBasicBlock - Return a copy of the specified basic block, but without
 /// embedding the block into a particular function.  The block returned is an
 /// exact copy of the specified basic block, without any remapping having been
 /// performed.  Because of this, this is only suitable for applications where
 /// the basic block will be inserted into the same function that it was cloned

llvm/trunk/lib/IR/Instructions.cpp

CallInst::CallInst(const CallInst &CI)
   setCallingConv(CI.getCallingConv());
 
   std::copy(CI.op_begin(), CI.op_end(), op_begin());
   std::copy(CI.bundle_op_info_begin(), CI.bundle_op_info_end(),
             bundle_op_info_begin());
   SubclassOptionalData = CI.SubclassOptionalData;
 }
 
+CallInst *CallInst::Create(CallInst *CI, ArrayRef<OperandBundleDef> OpB,
+                           Instruction *InsertPt) {
+  CallSite CS(CI);
+  std::vector<Value *> Args(CS.arg_begin(), CS.arg_end());
+
+  auto *NewCI = CallInst::Create(CI->getCalledValue(), Args, OpB, CI->getName(),
+                                 InsertPt);
+  NewCI->setTailCallKind(CI->getTailCallKind());
+  NewCI->setCallingConv(CI->getCallingConv());
+  NewCI->SubclassOptionalData = CI->SubclassOptionalData;
+  return NewCI;
+}
+
 void CallInst::addAttribute(unsigned i, Attribute::AttrKind attr) {
   AttributeSet PAL = getAttributes();
   PAL = PAL.addAttribute(getContext(), i, attr);
   setAttributes(PAL);
 }
 
 void CallInst::addAttribute(unsigned i, StringRef Kind, StringRef Value) {
   AttributeSet PAL = getAttributes();
: TerminatorInst(II.getType(), Instruction::Invoke,
       AttributeList(II.AttributeList), FTy(II.FTy) {
   setCallingConv(II.getCallingConv());
   std::copy(II.op_begin(), II.op_end(), op_begin());
   std::copy(II.bundle_op_info_begin(), II.bundle_op_info_end(),
             bundle_op_info_begin());
   SubclassOptionalData = II.SubclassOptionalData;
 }
 
+InvokeInst *InvokeInst::Create(InvokeInst *II, ArrayRef<OperandBundleDef> OpB,
+                               Instruction *InsertPt) {
+  CallSite CS(II);
+  std::vector<Value *> Args(CS.arg_begin(), CS.arg_end());
+
+  auto *NewII = InvokeInst::Create(II->getCalledValue(), II->getNormalDest(),
+                                   II->getUnwindDest(), Args, OpB,
+                                   II->getName(), InsertPt);
+  NewII->setCallingConv(II->getCallingConv());
+  NewII->SubclassOptionalData = II->SubclassOptionalData;
+  return NewII;
+}
+
 BasicBlock *InvokeInst::getSuccessorV(unsigned idx) const {
   return getSuccessor(idx);
 }
 unsigned InvokeInst::getNumSuccessorsV() const {
   return getNumSuccessors();
 }
 void InvokeInst::setSuccessorV(unsigned idx, BasicBlock *B) {
   return setSuccessor(idx, B);

llvm/trunk/lib/Transforms/Utils/CloneFunction.cpp

if (!isa<PHINode>(NewInst)) {
       }
     }
 
     if (II->hasName())
       NewInst->setName(II->getName()+NameSuffix);
     VMap[&*II] = NewInst; // Add instruction map to value.
     NewBB->getInstList().push_back(NewInst);
     hasCalls |= (isa<CallInst>(II) && !isa<DbgInfoIntrinsic>(II));
+
+    if (CodeInfo)
+      if (auto CS = ImmutableCallSite(&*II))
+        if (CS.hasOperandBundles())
+          CodeInfo->OperandBundleCallSites.push_back(NewInst);
+
     if (const AllocaInst *AI = dyn_cast<AllocaInst>(II)) {
       if (isa<ConstantInt>(AI->getArraySize()))
         hasStaticAllocas = true;
       else
         hasDynamicAllocas = true;
     }
   }
   
void PruningFunctionCloner::CloneBlock(const BasicBlock *BB,
   }
   
   if (!TerminatorDone) {
     Instruction *NewInst = OldTI->clone();
     if (OldTI->hasName())
       NewInst->setName(OldTI->getName()+NameSuffix);
     NewBB->getInstList().push_back(NewInst);
     VMap[OldTI] = NewInst;             // Add instruction map to value.
-    
+    if (CodeInfo)
+      if (auto CS = ImmutableCallSite(OldTI))
+        if (CS.hasOperandBundles())
+          CodeInfo->OperandBundleCallSites.push_back(NewInst);
+
     // Recursively clone any reachable successor blocks.
     const TerminatorInst *TI = BB->getTerminator();
     for (const BasicBlock *Succ : TI->successors())
       ToClone.push_back(Succ);
   }
   
   if (CodeInfo) {
     CodeInfo->ContainsCalls          |= hasCalls;

llvm/trunk/lib/Transforms/Utils/InlineFunction.cpp

BasicBlock *Split =
         BB->splitBasicBlock(CI->getIterator(), CI->getName() + ".noexc");
 
     // Delete the unconditional branch inserted by splitBasicBlock
     BB->getInstList().pop_back();
 
     // Create the new invoke instruction.
     ImmutableCallSite CS(CI);
     SmallVector<Value*, 8> InvokeArgs(CS.arg_begin(), CS.arg_end());
-    InvokeInst *II = InvokeInst::Create(CI->getCalledValue(), Split, UnwindEdge,
-                                        InvokeArgs, CI->getName(), BB);
+    SmallVector<OperandBundleDef, 1> OpBundles;
+
+    // Copy the OperandBundeUse instances to OperandBundleDefs.  These two are
+    // *different* representations of operand bundles: see the documentation in
+    // InstrTypes.h for more details.
+    for (unsigned i = 0, e = CS.getNumOperandBundles(); i != e; ++i)
+      OpBundles.emplace_back(CS.getOperandBundleAt(i));
+
+    // Note: we're round tripping operand bundles through memory here, and that
+    // can potentially be avoided with a cleverer API design that we do not have
+    // as of this time.
+
+    InvokeInst *II =
+        InvokeInst::Create(CI->getCalledValue(), Split, UnwindEdge, InvokeArgs,
+                           OpBundles, CI->getName(), BB);
     II->setDebugLoc(CI->getDebugLoc());
     II->setCallingConv(CI->getCallingConv());
     II->setAttributes(CI->getAttributes());
     
     // Make sure that anything using the call now uses the invoke!  This also
     // updates the CallGraph if present, because it uses a WeakVH.
     CI->replaceAllUsesWith(II);
 
bool llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
   // If IFI has any state in it, zap it before we fill it in.
   IFI.reset();
   
   const Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||              // Can't inline external function or indirect
       CalledFunc->isDeclaration() || // call, or call to a vararg function!
       CalledFunc->getFunctionType()->isVarArg()) return false;
 
-  // The inliner does not know how to inline through calls with operand bundles.
-  if (CS.hasOperandBundles())
+  // The inliner does not know how to inline through calls with operand bundles
+  // in general ...
+  if (CS.hasOperandBundles()) {
+    // ... but it knows how to inline through "deopt" operand bundles.
+    bool CanInline =
+        CS.getNumOperandBundles() == 1 &&
+        CS.getOperandBundleAt(0).getTagID() == LLVMContext::OB_deopt;
+    if (!CanInline)
       return false;
+  }
 
   // If the call to the callee cannot throw, set the 'nounwind' flag on any
   // calls that we inline.
   bool MarkNoUnwind = CS.doesNotThrow();
 
   BasicBlock *OrigBB = TheCall->getParent();
   Function *Caller = OrigBB->getParent();
 
Function::iterator FirstNewBlock;
     // Remember the first block that is newly cloned over.
     FirstNewBlock = LastBlock; ++FirstNewBlock;
 
     // Inject byval arguments initialization.
     for (std::pair<Value*, Value*> &Init : ByValInit)
       HandleByValArgumentInit(Init.first, Init.second, Caller->getParent(),
                               &*FirstNewBlock, IFI);
 
+    if (CS.hasOperandBundles()) {
+      auto ParentDeopt = CS.getOperandBundleAt(0);
+      assert(ParentDeopt.getTagID() == LLVMContext::OB_deopt &&
+             "Checked on entry!");
+
+      SmallVector<OperandBundleDef, 2> OpDefs;
+
+      for (auto &VH : InlinedFunctionInfo.OperandBundleCallSites) {
+        Instruction *I = VH;
+
+        OpDefs.clear();
+
+        CallSite ICS(I);
+        OpDefs.reserve(ICS.getNumOperandBundles());
+
+        for (unsigned i = 0, e = ICS.getNumOperandBundles(); i < e; ++i) {
+          auto ChildOB = ICS.getOperandBundleAt(i);
+          if (ChildOB.getTagID() != LLVMContext::OB_deopt) {
+            // If the inlined call has other operand bundles, let them be
+            OpDefs.emplace_back(ChildOB);
+            continue;
+          }
+
+          // It may be useful to separate this logic (of handling operand
+          // bundles) out to a separate "policy" component if this gets crowded.
+          // Prepend the parent's deoptimization continuation to the newly
+          // inlined call's deoptimization continuation.
+          std::vector<Value *> MergedDeoptArgs;
+          MergedDeoptArgs.reserve(ParentDeopt.Inputs.size() +
+                                  ChildOB.Inputs.size());
+
+          MergedDeoptArgs.insert(MergedDeoptArgs.end(),
+                                 ParentDeopt.Inputs.begin(),
+                                 ParentDeopt.Inputs.end());
+          MergedDeoptArgs.insert(MergedDeoptArgs.end(), ChildOB.Inputs.begin(),
+                                 ChildOB.Inputs.end());
+
+          OpDefs.emplace_back("deopt", std::move(MergedDeoptArgs));
+        }
+
+        Instruction *NewI = nullptr;
+        if (isa<CallInst>(I))
+          NewI = CallInst::Create(cast<CallInst>(I), OpDefs, I);
+        else
+          NewI = InvokeInst::Create(cast<InvokeInst>(I), OpDefs, I);
+
+        // Note: the RAUW does the appropriate fixup in VMap, so we need to do
+        // this even if the call returns void.
+        I->replaceAllUsesWith(NewI);
+
+        VH = nullptr;
+        I->eraseFromParent();
+      }
+    }
+
     // Update the callgraph if requested.
     if (IFI.CG)
       UpdateCallGraphAfterInlining(CS, FirstNewBlock, VMap, IFI);
 
     // Update inlined instructions' line number information.
     fixupLineNumbers(Caller, FirstNewBlock, TheCall);
 
     // Clone existing noalias metadata if necessary.

llvm/trunk/test/Transforms/Inline/deopt-bundles.ll

+; RUN: opt -S -always-inline < %s | FileCheck %s
+
+declare void @f()
+declare i32 @g()
+
+define i32 @callee_0() alwaysinline {
+ entry:
+  call void @f()
+  ret i32 2
+}
+
+define i32 @caller_0() {
+; CHECK-LABEL: @caller_0(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT: call void @f()
+; CHECK-NEXT: ret i32 2
+  %x = call i32 @callee_0() [ "deopt"(i32 5) ]
+  ret i32 %x
+}
+
+define i32 @callee_1() alwaysinline {
+ entry:
+  call void @f() [ "deopt"() ]
+  call void @f() [ "deopt"(i32 0, i32 1) ]
+  call void @f() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 2
+}
+
+define i32 @caller_1() {
+; CHECK-LABEL: @caller_1(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5) ]
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5, i32 0, i32 1) ]
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+; CHECK-NEXT:  ret i32 2
+
+  %x = call i32 @callee_1() [ "deopt"(i32 5) ]
+  ret i32 %x
+}
+
+define i32 @callee_2() alwaysinline {
+ entry:
+  %v = call i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 %v
+}
+
+define i32 @caller_2(i32 %val) {
+; CHECK-LABEL: @caller_2(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT:   [[RVAL:%[^ ]+]] = call i32 @g() [ "deopt"(i32 %val, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+; CHECK-NEXT:   ret i32 [[RVAL]]
+  %x = call i32 @callee_2() [ "deopt"(i32 %val) ]
+  ret i32 %x
+}
+
+define i32 @callee_3() alwaysinline {
+ entry:
+  %v = call i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 %v
+}
+
+define i32 @caller_3() personality i8 3 {
+; CHECK-LABEL: @caller_3(
+ entry:
+  %x = invoke i32 @callee_3() [ "deopt"(i32 7) ] to label %normal unwind label %unwind
+; CHECK: invoke i32 @g() [ "deopt"(i32 7, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+
+ normal:
+  ret i32 %x
+
+ unwind:
+  %cleanup = landingpad i8 cleanup
+  ret i32 101
+}
+
+define i32 @callee_4() alwaysinline personality i8 3 {
+ entry:
+  %v = invoke i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ] to label %normal unwind label %unwind
+
+ normal:
+  ret i32 %v
+
+ unwind:
+  %cleanup = landingpad i8 cleanup
+  ret i32 100
+}
+
+define i32 @caller_4() {
+; CHECK-LABEL: @caller_4(
+ entry:
+; CHECK: invoke i32 @g() [ "deopt"(i32 7, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+  %x = call i32 @callee_4() [ "deopt"(i32 7) ]
+  ret i32 %x
+}