Teach the inliner to track deoptimization state

[sanjoy]

Summary

This change teaches LLVM's inliner to track and suitably adjust
deoptimization state (tracked via deoptimization operand bundles) as it
inlines through call sites. The operation is described in more detail
in the LangRef changes.

Note:
The changes to CALLSITE_DELEGATE_GETTER were needed to allow
unification of the CallInst * and InvokeInst * types into an
Instruction * type in cloneWithOperandBundles.

[reames]

docs/LangRef.rst

1541

I'd add something here about it being the frontend responsibility to ensure that the bundle format is sufficient so that individual abstract frames can be reconstructed from the concatenated form if required.

include/llvm/IR/CallSite.h

215

This really looks like a white space change?

If you're going to introduce control flow, please use the "do { X } while(false)" idiom to make it less prone to miscompies.

include/llvm/IR/Instructions.h

1463

This comment is unclear. Is the argument list a white list? Is it an entirely new list of operand bundles? If so, where'd it come from?

lib/IR/Instructions.cpp

301

Don't we have an args_begin(), args_end?

Or better, an args() which returns an ArrayRef?

585

Same as above?

lib/Transforms/Utils/InlineFunction.cpp

211

This really seems like we need a getOperandBundles that returns an ArrayRef?

1039

I'd just inline this into the if and swap the comparison directions.

p.s. Don't you need to worry about a bundle type in the callee which is not deopt?

1151

I'm not sure about the order of the code here, but the fact we're replacing the same instruction twice (once for call to invoke, once for deopt bundles) seems likely to be error prone, particularly with ValueHandles being involved. Is there a way to avoid the complexity here? Don't have an obvious one myself, just asking.

1175

This is already enforced by the verifier, please remove.

1185

TagName? Why not tagID? And why not hard code OB_deopt here since it's implied by the surrounding code?

[sanjoy]

Replies inline.

Summarizing my plan to address the more complex review commments, I've proposed three post-commit changes inline:

• Change OperandBundleDef such that it can (alternative to containing an std::string tag name) contain an integral tag ID
• Change CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER to use the safer do { ... } while (false) idiom
• Introduce an iterator for getOperandBundleAt(I) and use that as much as possible

include/llvm/IR/CallSite.h

215

I'm just copying the pattern from CALLSITE_DELEGATE_SETTER. I agree that the do { ... } while (false) form is better -- is it okay if I fix CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER together in a separate checkin?

include/llvm/IR/Instructions.h

1463

It is an entirely new list of operand bundles. Will update the doc.

lib/IR/Instructions.cpp

301

They're there on CallSite, not on CallInst. Does changing this to

std::vector<...> Args = CallSite(this).args()

sound reasonable?

585

Same reply as above.

lib/Transforms/Utils/InlineFunction.cpp

211

That's difficult -- the object returned by getOperandBundleAt is a flyweight object (like StringRef etc.) that does not have a direct backing store (i.e. there is no internal array of OperandBundleUse objects) that I can create an ArrayRef<OperandBundleUse> with. I can definitely write a stateful iterator that iterates through all the operand bundles for a CallSite, but I've been waiting to do that sort of thing after bring-up.

1039

I'd just inline this into the if and swap the comparison directions.

SGTM.

p.s. Don't you need to worry about a bundle type in the callee which is not deopt?

The current semantics is that the caller's deopt state gets prepended to all of the "deopt" operand bundles in the callee. Call sites in the callee that have arbitrary deopt operand bundles are left intact. The test cases demonstrate this.

1151

I don't think there should be problems with ValueHandle s, but I do think this is somewhat inefficient. However, I don't think how I can fix that without changing CloneFunction to be aware of whether it is inlining into a call site that's a call instruction, or an invoke; and changing that is outside the scope of this patch.

1185

Right now the OperandBundleDef structure only carries an std::string that gets re-interned into LLVMContextImpl when the host instruction is constructed -- it has no way of holding a direct integer ID. This needs to be fixed (for performance, if nothing else), but doing that in this change will make it more complex. How about I change ChildOB.getTagName() to "deopt" for now, and in a later change add the functionality to create an OperandBundleDef holding a direct integral ID?

[sanjoy]

• address review

lib/Transforms/Utils/InlineFunction.cpp

1039

I'd just inline this into the if and swap the comparison directions.

I decided to keep this as-is (with a fix to the comment). I like how it reads currently -- "if CS has operand bundles attached, it better be of this form". Let me know if you feel strongly otherwise.

[sanjoy]

gentle ping!

[reames]

Responses to minor comments inline.

In D14552#286638, @sanjoy wrote:

• Change OperandBundleDef such that it can (alternative to containing an std::string tag name) contain an integral tag ID

Seems useful. SGTM

• Change CALLSITE_DELEGATE_SETTER and CALLSITE_DELEGATE_GETTER to use the safer do { ... } while (false) idiom

SGTM

• Introduce an iterator for getOperandBundleAt(I) and use that as much as possible

Not sure this is worth it.

lib/IR/Instructions.cpp

301

Yes.

lib/Transforms/Utils/InlineFunction.cpp

1151

Makes sense.

1185

SGTM

[reames]

include/llvm/IR/Instructions.h

1469

Offline, we decided a Create(InvokeInst, OperandBundles, InsertPt) would be a cleaner interface here.

lib/Transforms/Utils/InlineFunction.cpp

211–218

Offline we realized the confusion here. This loop is converting uses to defs, and is not simply a copy.

[sanjoy]

• address review

[sanjoy]

• use a literal "deopt" string as the tag

[reames]

LGTM w/comments addressed. I don't think this needs another round of review.

include/llvm/IR/Instructions.h

1463

The difference in insertion logic here is confusing and different from every other create function. Please stick to convention.

lib/Transforms/Utils/CloneFunction.cpp

379

CodeInfo might be null, please guard here and other as well.

lib/Transforms/Utils/InlineFunction.cpp

1161

minor: reserve final size

1165

Minor:
if (is_deopt) {}
else { }

might be clearer than if !deopt, continue

1175

minor: reserve the final size

docs/LangRef.rst

 compiled frame into a set of interpreted frames.
 
 From the compiler's perspective, deoptimization operand bundles make
 the call sites they're attached to at least ``readonly``.  They read
 through all of their pointer typed operands (even if they're not
 otherwise escaped) and the entire visible heap.  Deoptimization
 operand bundles do not capture their operands.
 
+The inliner knows how to inline through calls that have deoptimization
+operand bundles.  Just like inlining through a normal call site
+involves composing the normal and exceptional continuations, inlining
+through a call site with a deoptimization operand bundle needs to
+appropriately compose the "safe" deoptimization continuation.  The
+inliner does this by prepending the parent's deoptimization
+continuation to every deoptimization continuation in the inlined body.
+E.g. inlining ``@f`` into ``@g`` in the following example
+
+.. code-block:: llvm
+
+    define void @f() {
+      call void @x()  ;; no deopt state
+      call void @y() [ "deopt"(i32 10) ]
+      ret void
+    }
+
+    define void @g() {
+      call void @f() [ "deopt"(i32 20) ]
+      ret void
+    }
+
+will result in
+
+.. code-block:: llvm
+
+    define void @g() {
+      call void @x()  ;; still no deopt state
+      call void @y() [ "deopt"(i32 20, i32 10) ]
+      ret void
+    }
+
 .. _moduleasm:
 
 Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
 to the GCC "file scope inline asm" blocks. These blocks are internally
 concatenated by LLVM and treated as a single unit, but may be separated

include/llvm/IR/CallSite.h

public:
   }
 
   /// \brief Tests if this call site is marked as a tail call.
   bool isTailCall() const {
     return isCall() && cast<CallInst>(getInstruction())->isTailCall();
   }
 
 #define CALLSITE_DELEGATE_GETTER(METHOD) \
-  InstrTy *II = getInstruction();    \
-  return isCall()                        \
-    ? cast<CallInst>(II)->METHOD         \
-    : cast<InvokeInst>(II)->METHOD
+  InstrTy *II = getInstruction();        \
+  if (isCall())                          \
+    return cast<CallInst>(II)->METHOD;   \
+  else                                   \
+    return cast<InvokeInst>(II)->METHOD  \
+
 
 #define CALLSITE_DELEGATE_SETTER(METHOD) \
   InstrTy *II = getInstruction();    \
   if (isCall())                          \
     cast<CallInst>(II)->METHOD;          \
   else                                   \
     cast<InvokeInst>(II)->METHOD
 
#define CALLSITE_DELEGATE_SETTER(METHOD) \
   Optional<OperandBundleUse> getOperandBundle(StringRef Name) const {
     CALLSITE_DELEGATE_GETTER(getOperandBundle(Name));
   }
 
   Optional<OperandBundleUse> getOperandBundle(uint32_t ID) const {
     CALLSITE_DELEGATE_GETTER(getOperandBundle(ID));
   }
 
+  Instruction *cloneWithOperandBundles(ArrayRef<OperandBundleDef> OpB) {
+    CALLSITE_DELEGATE_GETTER(cloneWithOperandBundles(OpB));
+  }
+
 #undef CALLSITE_DELEGATE_GETTER
 #undef CALLSITE_DELEGATE_SETTER
 
   /// @brief Determine whether this data operand is not captured.
   bool doesNotCapture(unsigned OpNo) const {
     return dataOperandHasImpliedAttr(OpNo + 1, Attribute::NoCapture);
   }
 

include/llvm/IR/InstrTypes.h

 /// OperandBundleDef and ConstOperandBundleDef.
 template <typename InputTy> struct OperandBundleDefT {
   std::string Tag;
   std::vector<InputTy> Inputs;
 
   OperandBundleDefT() {}
   explicit OperandBundleDefT(StringRef Tag, const std::vector<InputTy> &Inputs)
       : Tag(Tag), Inputs(Inputs) {}
+
+  explicit OperandBundleDefT(StringRef Tag, std::vector<InputTy> &&Inputs)
+      : Tag(Tag), Inputs(Inputs) {}
+
+  explicit OperandBundleDefT(const OperandBundleUse &OBU) {
+    Tag = OBU.getTagName();
+    Inputs.insert(Inputs.end(), OBU.Inputs.begin(), OBU.Inputs.end());
+  }
 };
 
 typedef OperandBundleDefT<Value *> OperandBundleDef;
 typedef OperandBundleDefT<const Value *> ConstOperandBundleDef;
 
 /// \brief A mixin to add operand bundle functionality to llvm instruction
 /// classes.
 ///

include/llvm/IR/Instructions.h

public:
   static CallInst *Create(Value *F, const Twine &NameStr = "",
                           Instruction *InsertBefore = nullptr) {
     return new(1) CallInst(F, NameStr, InsertBefore);
   }
   static CallInst *Create(Value *F, const Twine &NameStr,
                           BasicBlock *InsertAtEnd) {
     return new(1) CallInst(F, NameStr, InsertAtEnd);
   }
+
+  /// \brief Clone this call instruction with a different set of operand
+  /// bundles.
+  CallInst *cloneWithOperandBundles(ArrayRef<OperandBundleDef> Bundles);
+
   /// CreateMalloc - Generate the IR for a call to malloc:
   /// 1. Compute the malloc call's argument as the specified type's size,
   ///    possibly multiplied by the array size if the array size is not
   ///    constant 1.
   /// 2. Call malloc with that argument.
   /// 3. Bitcast the result of the malloc call to the specified type.
   static Instruction *CreateMalloc(Instruction *InsertBefore,
                                    Type *IntPtrTy, Type *AllocTy,
                                    Value *AllocSize, Value *ArraySize = nullptr,
                                    Function* MallocF = nullptr,
                                    const Twine &Name = "");
static InvokeInst *Create(Value *Func, BasicBlock *IfNormal,
     unsigned Values = unsigned(Args.size()) + CountBundleInputs(Bundles) + 3;
     unsigned DescriptorBytes = Bundles.size() * sizeof(BundleOpInfo);
 
     return new (Values, DescriptorBytes)
         InvokeInst(Func, IfNormal, IfException, Args, Bundles, Values, NameStr,
                    InsertAtEnd);
   }
 
+  /// \brief Clone this invoke instruction with a different set of operand
+  /// bundles.
+  InvokeInst *cloneWithOperandBundles(ArrayRef<OperandBundleDef> Bundles);
+
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   FunctionType *getFunctionType() const { return FTy; }
 
   void mutateFunctionType(FunctionType *FTy) {
     mutateType(FTy->getReturnType());
     this->FTy = FTy;

include/llvm/Transforms/Utils/Cloning.h

struct ClonedCodeInfo {
   bool ContainsCalls;
 
   /// ContainsDynamicAllocas - This is set to true if the cloned code contains
   /// a 'dynamic' alloca.  Dynamic allocas are allocas that are either not in
   /// the entry block or they are in the entry block but are not a constant
   /// size.
   bool ContainsDynamicAllocas;
 
+  /// All cloned call sites that have operand bundles attached are appended to
+  /// this vector.
+  std::vector<AssertingVH<Instruction>> OperandBundleCallSites;
+
   ClonedCodeInfo() : ContainsCalls(false), ContainsDynamicAllocas(false) {}
 };
 
 /// CloneBasicBlock - Return a copy of the specified basic block, but without
 /// embedding the block into a particular function.  The block returned is an
 /// exact copy of the specified basic block, without any remapping having been
 /// performed.  Because of this, this is only suitable for applications where
 /// the basic block will be inserted into the same function that it was cloned

lib/IR/Instructions.cpp

CallInst::CallInst(const CallInst &CI)
   setCallingConv(CI.getCallingConv());
 
   std::copy(CI.op_begin(), CI.op_end(), op_begin());
   std::copy(CI.bundle_op_info_begin(), CI.bundle_op_info_end(),
             bundle_op_info_begin());
   SubclassOptionalData = CI.SubclassOptionalData;
 }
 
+CallInst *CallInst::cloneWithOperandBundles(ArrayRef<OperandBundleDef> OpB) {
+  std::vector<Value *> Args(op_begin(), op_begin() + getNumArgOperands());
+  auto *CI = CallInst::Create(getCalledValue(), Args, OpB, getName());
+  CI->setTailCallKind(getTailCallKind());
+  CI->setCallingConv(getCallingConv());
+  CI->SubclassOptionalData = SubclassOptionalData;
+  return CI;
+}
+
 void CallInst::addAttribute(unsigned i, Attribute::AttrKind attr) {
   AttributeSet PAL = getAttributes();
   PAL = PAL.addAttribute(getContext(), i, attr);
   setAttributes(PAL);
 }
 
 void CallInst::addAttribute(unsigned i, StringRef Kind, StringRef Value) {
   AttributeSet PAL = getAttributes();
: TerminatorInst(II.getType(), Instruction::Invoke,
       AttributeList(II.AttributeList), FTy(II.FTy) {
   setCallingConv(II.getCallingConv());
   std::copy(II.op_begin(), II.op_end(), op_begin());
   std::copy(II.bundle_op_info_begin(), II.bundle_op_info_end(),
             bundle_op_info_begin());
   SubclassOptionalData = II.SubclassOptionalData;
 }
 
+InvokeInst *
+InvokeInst::cloneWithOperandBundles(ArrayRef<OperandBundleDef> OpB) {
+  std::vector<Value *> Args(op_begin(), op_begin() + getNumArgOperands());
+  auto *II = InvokeInst::Create(getCalledValue(), getNormalDest(),
+                                getUnwindDest(), Args, OpB, getName());
+  II->setCallingConv(getCallingConv());
+  II->SubclassOptionalData = SubclassOptionalData;
+  return II;
+}
+
 BasicBlock *InvokeInst::getSuccessorV(unsigned idx) const {
   return getSuccessor(idx);
 }
 unsigned InvokeInst::getNumSuccessorsV() const {
   return getNumSuccessors();
 }
 void InvokeInst::setSuccessorV(unsigned idx, BasicBlock *B) {
   return setSuccessor(idx, B);

lib/Transforms/Utils/CloneFunction.cpp

if (!isa<PHINode>(NewInst)) {
       }
     }
 
     if (II->hasName())
       NewInst->setName(II->getName()+NameSuffix);
     VMap[&*II] = NewInst; // Add instruction map to value.
     NewBB->getInstList().push_back(NewInst);
     hasCalls |= (isa<CallInst>(II) && !isa<DbgInfoIntrinsic>(II));
+
+    if (auto CS = ImmutableCallSite(&*II))
+      if (CS.hasOperandBundles())
+        CodeInfo->OperandBundleCallSites.push_back(NewInst);
+
     if (const AllocaInst *AI = dyn_cast<AllocaInst>(II)) {
       if (isa<ConstantInt>(AI->getArraySize()))
         hasStaticAllocas = true;
       else
         hasDynamicAllocas = true;
     }
   }
   
void PruningFunctionCloner::CloneBlock(const BasicBlock *BB,
   }
   
   if (!TerminatorDone) {
     Instruction *NewInst = OldTI->clone();
     if (OldTI->hasName())
       NewInst->setName(OldTI->getName()+NameSuffix);
     NewBB->getInstList().push_back(NewInst);
     VMap[OldTI] = NewInst;             // Add instruction map to value.
-    
+    if (auto CS = ImmutableCallSite(OldTI))
+      if (CS.hasOperandBundles())
+        CodeInfo->OperandBundleCallSites.push_back(NewInst);
+
     // Recursively clone any reachable successor blocks.
     const TerminatorInst *TI = BB->getTerminator();
     for (const BasicBlock *Succ : TI->successors())
       ToClone.push_back(Succ);
   }
   
   if (CodeInfo) {
     CodeInfo->ContainsCalls          |= hasCalls;

lib/Transforms/Utils/InlineFunction.cpp

BasicBlock *Split =
         BB->splitBasicBlock(CI->getIterator(), CI->getName() + ".noexc");
 
     // Delete the unconditional branch inserted by splitBasicBlock
     BB->getInstList().pop_back();
 
     // Create the new invoke instruction.
     ImmutableCallSite CS(CI);
     SmallVector<Value*, 8> InvokeArgs(CS.arg_begin(), CS.arg_end());
-    InvokeInst *II = InvokeInst::Create(CI->getCalledValue(), Split, UnwindEdge,
-                                        InvokeArgs, CI->getName(), BB);
+    SmallVector<OperandBundleDef, 1> OpBundles;
+    for (unsigned i = 0, e = CS.getNumOperandBundles(); i != e; ++i)
+      OpBundles.emplace_back(CS.getOperandBundleAt(i));
+
+    InvokeInst *II =
+        InvokeInst::Create(CI->getCalledValue(), Split, UnwindEdge, InvokeArgs,
+                           OpBundles, CI->getName(), BB);
     II->setDebugLoc(CI->getDebugLoc());
     II->setCallingConv(CI->getCallingConv());
     II->setAttributes(CI->getAttributes());
     
     // Make sure that anything using the call now uses the invoke!  This also
     // updates the CallGraph if present, because it uses a WeakVH.
     CI->replaceAllUsesWith(II);
 
     // Delete the original call
bool llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
   IFI.reset();
   
   const Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||              // Can't inline external function or indirect
       CalledFunc->isDeclaration() || // call, or call to a vararg function!
       CalledFunc->getFunctionType()->isVarArg()) return false;
 
   // The inliner does not know how to inline through calls with operand bundles.
-  if (CS.hasOperandBundles())
+  if (CS.hasOperandBundles()) {
+    bool CanInline =
+        CS.getNumOperandBundles() == 1 &&
+        CS.getOperandBundleAt(0).getTagID() == LLVMContext::OB_deopt;
+    if (!CanInline)
       return false;
+  }
 
   // If the call to the callee cannot throw, set the 'nounwind' flag on any
   // calls that we inline.
   bool MarkNoUnwind = CS.doesNotThrow();
 
   BasicBlock *OrigBB = TheCall->getParent();
   Function *Caller = OrigBB->getParent();
 
Function::iterator FirstNewBlock;
     // Remember the first block that is newly cloned over.
     FirstNewBlock = LastBlock; ++FirstNewBlock;
 
     // Inject byval arguments initialization.
     for (std::pair<Value*, Value*> &Init : ByValInit)
       HandleByValArgumentInit(Init.first, Init.second, Caller->getParent(),
                               &*FirstNewBlock, IFI);
 
+    if (CS.hasOperandBundles()) {
+      auto ParentDeopt = CS.getOperandBundleAt(0);
+      assert(ParentDeopt.getTagID() == LLVMContext::OB_deopt &&
+             "Checked on entry!");
+
+      SmallVector<OperandBundleDef, 2> OpDefs;
+
+      for (auto &VH : InlinedFunctionInfo.OperandBundleCallSites) {
+        Instruction *I = VH;
+
+        OpDefs.clear();
+        CallSite ICS(I);
+        bool Found = false;
+        for (unsigned i = 0, e = ICS.getNumOperandBundles(); i < e; ++i) {
+          auto ChildOB = ICS.getOperandBundleAt(i);
+          if (ChildOB.getTagID() != LLVMContext::OB_deopt) {
+            // If the inlined call has other operand bundles, let them be
+            OpDefs.emplace_back(ChildOB);
+            continue;
+          }
+
+          // It may be useful to separate this logic (of handling operand
+          // bundles) out to a separate "policy" component if this gets crowded.
+
+          assert(!Found && "Only one deopt operand bundle allowed!");
+          Found = true;
+
+          // Prepend the parent's deoptimization continuation to the newly
+          // inlined call's deoptimization continuation.
+          std::vector<Value *> MergedDeoptArgs(ParentDeopt.Inputs.begin(),
+                                               ParentDeopt.Inputs.end());
+          MergedDeoptArgs.insert(MergedDeoptArgs.end(), ChildOB.Inputs.begin(),
+                                 ChildOB.Inputs.end());
+
+          OpDefs.emplace_back(ChildOB.getTagName(), std::move(MergedDeoptArgs));
+        }
+
+        Instruction *NewI = ICS.cloneWithOperandBundles(OpDefs);
+        I->getParent()->getInstList().insert(I->getIterator(), NewI);
+
+        // Note: the RAUW does the appropriate fixup in VMap, so we need to do
+        // this even if the call returns void.
+        I->replaceAllUsesWith(NewI);
+
+        VH = nullptr;
+        I->eraseFromParent();
+      }
+    }
+
     // Update the callgraph if requested.
     if (IFI.CG)
       UpdateCallGraphAfterInlining(CS, FirstNewBlock, VMap, IFI);
 
     // Update inlined instructions' line number information.
     fixupLineNumbers(Caller, FirstNewBlock, TheCall);
 
     // Clone existing noalias metadata if necessary.

test/Transforms/Inline/deopt-bundles.ll

+; RUN: opt -S -always-inline < %s | FileCheck %s
+
+declare void @f()
+declare i32 @g()
+
+define i32 @callee_0() alwaysinline {
+ entry:
+  call void @f()
+  ret i32 2
+}
+
+define i32 @caller_0() {
+; CHECK-LABEL: @caller_0(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT: call void @f()
+; CHECK-NEXT: ret i32 2
+  %x = call i32 @callee_0() [ "deopt"(i32 5) ]
+  ret i32 %x
+}
+
+define i32 @callee_1() alwaysinline {
+ entry:
+  call void @f() [ "deopt"() ]
+  call void @f() [ "deopt"(i32 0, i32 1) ]
+  call void @f() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 2
+}
+
+define i32 @caller_1() {
+; CHECK-LABEL: @caller_1(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5) ]
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5, i32 0, i32 1) ]
+; CHECK-NEXT:  call void @f() [ "deopt"(i32 5, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+; CHECK-NEXT:  ret i32 2
+
+  %x = call i32 @callee_1() [ "deopt"(i32 5) ]
+  ret i32 %x
+}
+
+define i32 @callee_2() alwaysinline {
+ entry:
+  %v = call i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 %v
+}
+
+define i32 @caller_2(i32 %val) {
+; CHECK-LABEL: @caller_2(
+ entry:
+; CHECK: entry:
+; CHECK-NEXT:   [[RVAL:%[^ ]+]] = call i32 @g() [ "deopt"(i32 %val, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+; CHECK-NEXT:   ret i32 [[RVAL]]
+  %x = call i32 @callee_2() [ "deopt"(i32 %val) ]
+  ret i32 %x
+}
+
+define i32 @callee_3() alwaysinline {
+ entry:
+  %v = call i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ]
+  ret i32 %v
+}
+
+define i32 @caller_3() personality i8 3 {
+; CHECK-LABEL: @caller_3(
+ entry:
+  %x = invoke i32 @callee_3() [ "deopt"(i32 7) ] to label %normal unwind label %unwind
+; CHECK: invoke i32 @g() [ "deopt"(i32 7, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+
+ normal:
+  ret i32 %x
+
+ unwind:
+  %cleanup = landingpad i8 cleanup
+  ret i32 101
+}
+
+define i32 @callee_4() alwaysinline personality i8 3 {
+ entry:
+  %v = invoke i32 @g() [ "deopt"(i32 0, i32 1), "foo"(double 0.0) ] to label %normal unwind label %unwind
+
+ normal:
+  ret i32 %v
+
+ unwind:
+  %cleanup = landingpad i8 cleanup
+  ret i32 100
+}
+
+define i32 @caller_4() {
+; CHECK-LABEL: @caller_4(
+ entry:
+; CHECK: invoke i32 @g() [ "deopt"(i32 7, i32 0, i32 1), "foo"(double 0.000000e+00) ]
+  %x = call i32 @callee_4() [ "deopt"(i32 7) ]
+  ret i32 %x
+}