[WinEH] Use operand bundles to describe call sites

[majnemer]

Summary

SimplifyCFG allows tail merging with code which terminates in
unreachable which, in turn, makes it possible for an invoke to end up in
a funclet which it was not originally part of.

Using operand bundles on invokes allows us to determine whether or not
an invoke was part of a funclet in the source program.

Furthermore, it allows us to unambiguously answer questions about the
legality of inlining into call sites which the personality may have
trouble with.

[rnk]

lgtm, but let's wait for more feedback.

I personally like the name "funclet" for the operand bundle. It always has to be a FuncletPadInst value, so it really always is a "funclet".

[sanjoy]

Operand bundles bits lgtm with some minor comments. Doesn't have to be this patch, but please add a small blurb to the langref about the "funclet" operand bundle.

lib/CodeGen/WinEHPrepare.cpp

795 ↗

(On Diff #42814)

Minor: will this look cleaner with an auto?

lib/Transforms/Utils/InlineFunction.cpp

1446 ↗

(On Diff #42814)

Why do you need to declare Inputs separately? Why not just OpBundles.emplace_back("funclet", CallSiteEHPad);?

1449 ↗

(On Diff #42814)

Minor: braces unnecessary. This may even be clearer as a ternary operator.

[JosephTremoulet]

lib/CodeGen/WinEHPrepare.cpp

804–809 ↗

(On Diff #42814)

I think you should do this part for CallInst uses too.

lib/IR/Verifier.cpp

2397–2398 ↗

(On Diff #42814)

Want to check that it has token type? Or specifically that it is a FuncletPadInst?

lib/Transforms/Utils/InlineFunction.cpp

1419 ↗

(On Diff #42814)

s/funclets/funclets and callsites/?

1434–1438 ↗

(On Diff #42814)

If you agree with my feedback in WinEHPrepare about making mis-annotated calls equally UB to mis-annotated invokes (and it seems to me they should agree), then this part will need to agree. It might make sense to have nounwind intrinsics be a special case (and the only one) in both places. Or neither.

1454 ↗

(On Diff #42814)

The Create overloads you call a few lines above pass the old instruction's name as the name argument for the new instruction. Does takeName do something different than (but still compatible with) that?

Also, supposing one of these calls/invokes has debug info (line number etc) attached, is what's here sufficient to propagate it to the new one?

[majnemer]

• Address review comments

[JosephTremoulet]

LGTM, thanks.

llvm/trunk/docs/LangRef.rst

.. code-block:: llvm
     }
 
 It is the frontend's responsibility to structure or encode the
 deoptimization state in a way that syntactically prepending the
 caller's deoptimization state to the callee's deoptimization state is
 semantically equivalent to composing the caller's deoptimization
 continuation after the callee's deoptimization continuation.
 
+Funclet Operand Bundles
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Funclet operand bundles are characterized by the ``"funclet"``
+operand bundle tag.  These operand bundles indicate that a call site
+is within a particular funclet.  There can be at most one
+``"funclet"`` operand bundle attached to a call site and it must have
+exactly one bundle operand.
+
 .. _moduleasm:
 
 Module-Level Inline Assembly
 ----------------------------
 
 Modules may contain "module-level inline asm" blocks, which corresponds
 to the GCC "file scope inline asm" blocks. These blocks are internally
 concatenated by LLVM and treated as a single unit, but may be separated

llvm/trunk/include/llvm/IR/InstrTypes.h

uint32_t getTagID() const {
     return Tag->getValue();
   }
 
   /// \brief Return true if this is a "deopt" operand bundle.
   bool isDeoptOperandBundle() const {
     return getTagID() == LLVMContext::OB_deopt;
   }
 
+  /// \brief Return true if this is a "funclet" operand bundle.
+  bool isFuncletOperandBundle() const {
+    return getTagID() == LLVMContext::OB_funclet;
+  }
+
 private:
   /// \brief Pointer to an entry in LLVMContextImpl::getOrInsertBundleTag.
   StringMapEntry<uint32_t> *Tag;
 };
 
 /// \brief A container for an operand bundle being viewed as a set of values
 /// rather than a set of uses.
 ///
 /// Unlike OperandBundleUse, OperandBundleDefT owns the memory it carries, and
 /// so it is possible to create and pass around "self-contained" instances of
 /// OperandBundleDef and ConstOperandBundleDef.
 template <typename InputTy> class OperandBundleDefT {
   std::string Tag;
   std::vector<InputTy> Inputs;
 
 public:
   explicit OperandBundleDefT(std::string Tag, std::vector<InputTy> Inputs)
       : Tag(std::move(Tag)), Inputs(std::move(Inputs)) {}
+  explicit OperandBundleDefT(std::string Tag, ArrayRef<InputTy> Inputs)
+      : Tag(std::move(Tag)), Inputs(Inputs) {}
 
   explicit OperandBundleDefT(const OperandBundleUse &OBU) {
     Tag = OBU.getTagName();
     Inputs.insert(Inputs.end(), OBU.Inputs.begin(), OBU.Inputs.end());
   }
 
   ArrayRef<InputTy> inputs() const { return Inputs; }
 
bool hasReadingOperandBundles() const {
     // least readonly.
     return hasOperandBundles();
   }
 
   /// \brief Return true if this operand bundle user has operand bundles that
   /// may write to the heap.
   bool hasClobberingOperandBundles() const {
     for (auto &BOI : bundle_op_infos()) {
-      if (BOI.Tag->second == LLVMContext::OB_deopt)
+      if (BOI.Tag->second == LLVMContext::OB_deopt ||
+          BOI.Tag->second == LLVMContext::OB_funclet)
         continue;
 
-      // This instruction has an operand bundle that is not a "deopt" operand
-      // bundle.  Assume the worst.
+      // This instruction has an operand bundle that is not known to us.
+      // Assume the worst.
       return true;
     }
 
     return false;
   }
 
   /// \brief Return true if the bundle operand at index \p OpIdx has the
   /// attribute \p A.

llvm/trunk/include/llvm/IR/LLVMContext.h

enum {
     MD_align = 17 // "align"
   };
 
   /// Known operand bundle tag IDs, which always have the same value.  All
   /// operand bundle tags that LLVM has special knowledge of are listed here.
   /// Additionally, this scheme allows LLVM to efficiently check for specific
   /// operand bundle tags without comparing strings.
   enum {
-    OB_deopt = 0,  // "deopt"
+    OB_deopt = 0,   // "deopt"
+    OB_funclet = 1, // "funclet"
   };
 
   /// getMDKindID - Return a unique non-zero ID for the specified metadata kind.
   /// This ID is uniqued across modules in the current LLVMContext.
   unsigned getMDKindID(StringRef Name) const;
 
   /// getMDKindNames - Populate client supplied SmallVector with the name for
   /// custom metadata IDs registered in this LLVMContext.

llvm/trunk/include/llvm/Transforms/Utils/Local.h

 /// \brief Replaces llvm.dbg.declare instruction when the alloca it describes
 /// is replaced with a new value. If Deref is true, an additional DW_OP_deref is
 /// prepended to the expression. If Offset is non-zero, a constant displacement
 /// is added to the expression (after the optional Deref). Offset can be
 /// negative. New llvm.dbg.declare is inserted immediately before AI.
 bool replaceDbgDeclareForAlloca(AllocaInst *AI, Value *NewAllocaAddress,
                                 DIBuilder &Builder, bool Deref, int Offset = 0);
 
+/// \brief Insert an unreachable instruction before the specified
+/// instruction, making it and the rest of the code in the block dead.
+void changeToUnreachable(Instruction *I, bool UseLLVMTrap);
+
 /// Replace 'BB's terminator with one that does not have an unwind successor
 /// block.  Rewrites `invoke` to `call`, etc.  Updates any PHIs in unwind
 /// successor.
 ///
 /// \param BB  Block whose terminator will be replaced.  Its terminator must
 ///            have an unwind successor.
 void removeUnwindEdge(BasicBlock *BB);
 

llvm/trunk/lib/CodeGen/WinEHPrepare.cpp

private:
   AllocaInst *insertPHILoads(PHINode *PN, Function &F);
   void replaceUseWithLoad(Value *V, Use &U, AllocaInst *&SpillSlot,
                           DenseMap<BasicBlock *, Value *> &Loads, Function &F);
   bool prepareExplicitEH(Function &F);
   void colorFunclets(Function &F);
 
   void demotePHIsOnFunclets(Function &F);
   void cloneCommonBlocks(Function &F);
-  void removeImplausibleTerminators(Function &F);
+  void removeImplausibleInstructions(Function &F);
   void cleanupPreparedFunclets(Function &F);
   void verifyPreparedFunclets(Function &F);
 
   // All fields are reset by runOnFunction.
   EHPersonality Personality = EHPersonality::Unknown;
 
   DenseMap<BasicBlock *, ColorVector> BlockColors;
   MapVector<BasicBlock *, std::vector<BasicBlock *>> FuncletBlocks;
for (ValueToValueMapTy::value_type VT : VMap) {
       SSAUpdate.AddAvailableValue(NewI->getParent(), NewI);
 
       while (!UsesToRename.empty())
         SSAUpdate.RewriteUseAfterInsertions(*UsesToRename.pop_back_val());
     }
   }
 }
 
-void WinEHPrepare::removeImplausibleTerminators(Function &F) {
+void WinEHPrepare::removeImplausibleInstructions(Function &F) {
   // Remove implausible terminators and replace them with UnreachableInst.
   for (auto &Funclet : FuncletBlocks) {
     BasicBlock *FuncletPadBB = Funclet.first;
     std::vector<BasicBlock *> &BlocksInFunclet = Funclet.second;
-    Instruction *FuncletPadInst = FuncletPadBB->getFirstNonPHI();
-    auto *CatchPad = dyn_cast<CatchPadInst>(FuncletPadInst);
-    auto *CleanupPad = dyn_cast<CleanupPadInst>(FuncletPadInst);
+    Instruction *FirstNonPHI = FuncletPadBB->getFirstNonPHI();
+    auto *FuncletPad = dyn_cast<FuncletPadInst>(FirstNonPHI);
+    auto *CatchPad = dyn_cast_or_null<CatchPadInst>(FuncletPad);
+    auto *CleanupPad = dyn_cast_or_null<CleanupPadInst>(FuncletPad);
 
     for (BasicBlock *BB : BlocksInFunclet) {
+      for (Instruction &I : *BB) {
+        CallSite CS(&I);
+        if (!CS)
+          continue;
+
+        Value *FuncletBundleOperand = nullptr;
+        if (auto BU = CS.getOperandBundle(LLVMContext::OB_funclet))
+          FuncletBundleOperand = BU->Inputs.front();
+
+        if (FuncletBundleOperand == FuncletPad)
+          continue;
+
+        // Skip call sites which are nounwind intrinsics.
+        auto *CalledFn =
+            dyn_cast<Function>(CS.getCalledValue()->stripPointerCasts());
+        if (CalledFn && CalledFn->isIntrinsic() && CS.doesNotThrow())
+          continue;
+
+        // This call site was not part of this funclet, remove it.
+        if (CS.isInvoke()) {
+          // Remove the unwind edge if it was an invoke.
+          removeUnwindEdge(BB);
+          // Get a pointer to the new call.
+          BasicBlock::iterator CallI =
+              std::prev(BB->getTerminator()->getIterator());
+          auto *CI = cast<CallInst>(&*CallI);
+          changeToUnreachable(CI, /*UseLLVMTrap=*/false);
+        } else {
+          changeToUnreachable(&I, /*UseLLVMTrap=*/false);
+        }
+
+        // There are no more instructions in the block (except for unreachable),
+        // we are done.
+        break;
+      }
+
       TerminatorInst *TI = BB->getTerminator();
       // CatchPadInst and CleanupPadInst can't transfer control to a ReturnInst.
-      bool IsUnreachableRet = isa<ReturnInst>(TI) && (CatchPad || CleanupPad);
+      bool IsUnreachableRet = isa<ReturnInst>(TI) && FuncletPad;
       // The token consumed by a CatchReturnInst must match the funclet token.
       bool IsUnreachableCatchret = false;
       if (auto *CRI = dyn_cast<CatchReturnInst>(TI))
         IsUnreachableCatchret = CRI->getCatchPad() != CatchPad;
       // The token consumed by a CleanupReturnInst must match the funclet token.
       bool IsUnreachableCleanupret = false;
       if (auto *CRI = dyn_cast<CleanupReturnInst>(TI))
         IsUnreachableCleanupret = CRI->getCleanupPad() != CleanupPad;
       if (IsUnreachableRet || IsUnreachableCatchret ||
           IsUnreachableCleanupret) {
-        // Loop through all of our successors and make sure they know that one
-        // of their predecessors is going away.
-        for (BasicBlock *SuccBB : TI->successors())
-          SuccBB->removePredecessor(BB);
-
-        new UnreachableInst(BB->getContext(), TI);
-        TI->eraseFromParent();
+        changeToUnreachable(TI, /*UseLLVMTrap=*/false);
       } else if (isa<InvokeInst>(TI)) {
+        if (Personality == EHPersonality::MSVC_CXX && CleanupPad) {
           // Invokes within a cleanuppad for the MSVC++ personality never
           // transfer control to their unwind edge: the personality will
           // terminate the program.
-        if (Personality == EHPersonality::MSVC_CXX && CleanupPad)
           removeUnwindEdge(BB);
         }
       }
     }
   }
+}
 
 void WinEHPrepare::cleanupPreparedFunclets(Function &F) {
   // Clean-up some of the mess we made by removing useles PHI nodes, trivial
   // branches, etc.
   for (Function::iterator FI = F.begin(), FE = F.end(); FI != FE;) {
     BasicBlock *BB = &*FI++;
     SimplifyInstructionsInBlock(BB);
     ConstantFoldTerminator(BB, /*DeleteDeadConditions=*/true);
bool WinEHPrepare::prepareExplicitEH(Function &F) {
   colorFunclets(F);
 
   cloneCommonBlocks(F);
 
   if (!DisableDemotion)
     demotePHIsOnFunclets(F);
 
   if (!DisableCleanups) {
-    removeImplausibleTerminators(F);
+    removeImplausibleInstructions(F);
 
     cleanupPreparedFunclets(F);
   }
 
   verifyPreparedFunclets(F);
 
   BlockColors.clear();
   FuncletBlocks.clear();

llvm/trunk/lib/IR/LLVMContext.cpp

LLVMContext::LLVMContext() : pImpl(new LLVMContextImpl(*this)) {
   unsigned AlignID = getMDKindID("align");
   assert(AlignID == MD_align && "align kind id drifted");
   (void)AlignID;
 
   auto *DeoptEntry = pImpl->getOrInsertBundleTag("deopt");
   assert(DeoptEntry->second == LLVMContext::OB_deopt &&
          "deopt operand bundle id drifted!");
   (void)DeoptEntry;
+
+  auto *FuncletEntry = pImpl->getOrInsertBundleTag("funclet");
+  assert(FuncletEntry->second == LLVMContext::OB_funclet &&
+         "funclet operand bundle id drifted!");
+  (void)FuncletEntry;
 }
 LLVMContext::~LLVMContext() { delete pImpl; }
 
 void LLVMContext::addModule(Module *M) {
   pImpl->OwnedModules.insert(M);
 }
 
 void LLVMContext::removeModule(Module *M) {

llvm/trunk/lib/IR/Verifier.cpp

void Verifier::VerifyCallSite(CallSite CS) {
   if (CS.getCalledFunction() == nullptr)
     Assert(!FTy->getReturnType()->isTokenTy(),
            "Return type cannot be token for indirect call!");
 
   if (Function *F = CS.getCalledFunction())
     if (Intrinsic::ID ID = (Intrinsic::ID)F->getIntrinsicID())
       visitIntrinsicCallSite(ID, CS);
 
-  // Verify that a callsite has at most one "deopt" operand bundle.
-  bool FoundDeoptBundle = false;
+  // Verify that a callsite has at most one "deopt" and one "funclet" operand
+  // bundle.
+  bool FoundDeoptBundle = false, FoundFuncletBundle = false;
   for (unsigned i = 0, e = CS.getNumOperandBundles(); i < e; ++i) {
-    if (CS.getOperandBundleAt(i).getTagID() == LLVMContext::OB_deopt) {
+    OperandBundleUse BU = CS.getOperandBundleAt(i);
+    uint32_t Tag = BU.getTagID();
+    if (Tag == LLVMContext::OB_deopt) {
       Assert(!FoundDeoptBundle, "Multiple deopt operand bundles", I);
       FoundDeoptBundle = true;
     }
+    if (Tag == LLVMContext::OB_funclet) {
+      Assert(!FoundFuncletBundle, "Multiple funclet operand bundles", I);
+      FoundFuncletBundle = true;
+      Assert(BU.Inputs.size() == 1,
+             "Expected exactly one funclet bundle operand", I);
+      Assert(isa<FuncletPadInst>(BU.Inputs.front()),
+             "Funclet bundle operands should correspond to a FuncletPadInst",
+             I);
+    }
   }
 
   visitInstruction(*I);
 }
 
 /// Two types are "congruent" if they are identical, or if they are both pointer
 /// types with different pointee types and the same address space.
 static bool isTypeCongruent(Type *L, Type *R) {
void Verifier::visitCleanupPadInst(CleanupPadInst &CPI) {
   User *FirstUser = nullptr;
   BasicBlock *FirstUnwindDest = nullptr;
   for (User *U : CPI.users()) {
     BasicBlock *UnwindDest;
     if (CleanupReturnInst *CRI = dyn_cast<CleanupReturnInst>(U)) {
       UnwindDest = CRI->getUnwindDest();
     } else if (isa<CleanupPadInst>(U) || isa<CatchSwitchInst>(U)) {
       continue;
+    } else if (CallSite(U)) {
+      continue;
     } else {
       Assert(false, "bogus cleanuppad use", &CPI);
     }
 
     if (!FirstUser) {
       FirstUser = U;
       FirstUnwindDest = UnwindDest;
     } else {

llvm/trunk/lib/Transforms/Utils/InlineFunction.cpp

bool llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
   const Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||              // Can't inline external function or indirect
       CalledFunc->isDeclaration() || // call, or call to a vararg function!
       CalledFunc->getFunctionType()->isVarArg()) return false;
 
   // The inliner does not know how to inline through calls with operand bundles
   // in general ...
   if (CS.hasOperandBundles()) {
-    // ... but it knows how to inline through "deopt" operand bundles.
-    bool CanInline =
-        CS.getNumOperandBundles() == 1 &&
-        CS.getOperandBundleAt(0).getTagID() == LLVMContext::OB_deopt;
-    if (!CanInline)
+    for (int i = 0, e = CS.getNumOperandBundles(); i != e; ++i) {
+      uint32_t Tag = CS.getOperandBundleAt(i).getTagID();
+      // ... but it knows how to inline through "deopt" operand bundles ...
+      if (Tag == LLVMContext::OB_deopt)
+        continue;
+      // ... and "funclet" operand bundles.
+      if (Tag == LLVMContext::OB_funclet)
+        continue;
+
       return false;
     }
+  }
 
   // If the call to the callee cannot throw, set the 'nounwind' flag on any
   // calls that we inline.
   bool MarkNoUnwind = CS.doesNotThrow();
 
   BasicBlock *OrigBB = TheCall->getParent();
   Function *Caller = OrigBB->getParent();
 
if (CalledPersonality) {
     //       supersets of others and can be used in place of the other.
     else if (CalledPersonality != CallerPersonality)
       return false;
   }
 
   // We need to figure out which funclet the callsite was in so that we may
   // properly nest the callee.
   Instruction *CallSiteEHPad = nullptr;
-  if (CalledPersonality && CallerPersonality) {
-    EHPersonality Personality = classifyEHPersonality(CalledPersonality);
+  if (CallerPersonality) {
+    EHPersonality Personality = classifyEHPersonality(CallerPersonality);
     if (isFuncletEHPersonality(Personality)) {
-      DenseMap<BasicBlock *, ColorVector> CallerBlockColors =
-          colorEHFunclets(*Caller);
-      ColorVector &CallSiteColors = CallerBlockColors[OrigBB];
-      size_t NumColors = CallSiteColors.size();
-      // There is no single parent, inlining will not succeed.
-      if (NumColors > 1)
-        return false;
-      if (NumColors == 1) {
-        BasicBlock *CallSiteFuncletBB = CallSiteColors.front();
-        if (CallSiteFuncletBB != Caller->begin()) {
-          CallSiteEHPad = CallSiteFuncletBB->getFirstNonPHI();
-          assert(CallSiteEHPad->isEHPad() && "Expected an EHPad!");
-        }
-      }
+      Optional<OperandBundleUse> ParentFunclet =
+          CS.getOperandBundle(LLVMContext::OB_funclet);
+      if (ParentFunclet)
+        CallSiteEHPad = cast<FuncletPadInst>(ParentFunclet->Inputs.front());
 
       // OK, the inlining site is legal.  What about the target function?
 
       if (CallSiteEHPad) {
         if (Personality == EHPersonality::MSVC_CXX) {
           // The MSVC personality cannot tolerate catches getting inlined into
           // cleanup funclets.
           if (isa<CleanupPadInst>(CallSiteEHPad)) {
             // Ok, the call site is within a cleanuppad.  Let's check the callee
             // for catchpads.
             for (const BasicBlock &CalledBB : *CalledFunc) {
-              if (isa<CatchPadInst>(CalledBB.getFirstNonPHI()))
+              if (isa<CatchSwitchInst>(CalledBB.getFirstNonPHI()))
                 return false;
             }
           }
         } else if (isAsynchronousEHPersonality(Personality)) {
           // SEH is even less tolerant, there may not be any sort of exceptional
           // funclet in the callee.
           for (const BasicBlock &CalledBB : *CalledFunc) {
             if (CalledBB.isEHPad())
Function::iterator FirstNewBlock;
     // Remember the first block that is newly cloned over.
     FirstNewBlock = LastBlock; ++FirstNewBlock;
 
     // Inject byval arguments initialization.
     for (std::pair<Value*, Value*> &Init : ByValInit)
       HandleByValArgumentInit(Init.first, Init.second, Caller->getParent(),
                               &*FirstNewBlock, IFI);
 
-    if (CS.hasOperandBundles()) {
-      auto ParentDeopt = CS.getOperandBundleAt(0);
-      assert(ParentDeopt.getTagID() == LLVMContext::OB_deopt &&
-             "Checked on entry!");
-
+    Optional<OperandBundleUse> ParentDeopt =
+        CS.getOperandBundle(LLVMContext::OB_deopt);
+    if (ParentDeopt) {
       SmallVector<OperandBundleDef, 2> OpDefs;
 
       for (auto &VH : InlinedFunctionInfo.OperandBundleCallSites) {
         if (!VH) continue;  // instruction was DCE'd after being cloned
 
         Instruction *I = cast<Instruction>(VH);
 
         OpDefs.clear();
if (ParentDeopt) {
             continue;
           }
 
           // It may be useful to separate this logic (of handling operand
           // bundles) out to a separate "policy" component if this gets crowded.
           // Prepend the parent's deoptimization continuation to the newly
           // inlined call's deoptimization continuation.
           std::vector<Value *> MergedDeoptArgs;
-          MergedDeoptArgs.reserve(ParentDeopt.Inputs.size() +
+          MergedDeoptArgs.reserve(ParentDeopt->Inputs.size() +
                                   ChildOB.Inputs.size());
 
           MergedDeoptArgs.insert(MergedDeoptArgs.end(),
-                                 ParentDeopt.Inputs.begin(),
-                                 ParentDeopt.Inputs.end());
+                                 ParentDeopt->Inputs.begin(),
+                                 ParentDeopt->Inputs.end());
           MergedDeoptArgs.insert(MergedDeoptArgs.end(), ChildOB.Inputs.begin(),
                                  ChildOB.Inputs.end());
 
           OpDefs.emplace_back("deopt", std::move(MergedDeoptArgs));
         }
 
         Instruction *NewI = nullptr;
         if (isa<CallInst>(I))
for (ReturnInst *RI : Returns) {
       // Don't insert llvm.stackrestore calls between a musttail call and a
       // return.  The return will restore the stack pointer.
       if (InlinedMustTailCalls && RI->getParent()->getTerminatingMustTailCall())
         continue;
       IRBuilder<>(RI).CreateCall(StackRestore, SavedPtr);
     }
   }
 
-  // Update the lexical scopes of the new funclets.  Anything that had 'none' as
-  // its parent is now nested inside the callsite's EHPad.
+  // Update the lexical scopes of the new funclets and callsites.
+  // Anything that had 'none' as its parent is now nested inside the callsite's
+  // EHPad.
+
   if (CallSiteEHPad) {
     for (Function::iterator BB = FirstNewBlock->getIterator(),
                             E = Caller->end();
          BB != E; ++BB) {
+      // Add bundle operands to any top-level call sites.
+      SmallVector<OperandBundleDef, 1> OpBundles;
+      for (BasicBlock::iterator BBI = BB->begin(), E = BB->end(); BBI != E;) {
+        Instruction *I = &*BBI++;
+        CallSite CS(I);
+        if (!CS)
+          continue;
+
+        // Skip call sites which are nounwind intrinsics.
+        auto *CalledFn =
+            dyn_cast<Function>(CS.getCalledValue()->stripPointerCasts());
+        if (CalledFn && CalledFn->isIntrinsic() && CS.doesNotThrow())
+          continue;
+
+        // Skip call sites which already have a "funclet" bundle.
+        if (CS.getOperandBundle(LLVMContext::OB_funclet))
+          continue;
+
+        CS.getOperandBundlesAsDefs(OpBundles);
+        OpBundles.emplace_back("funclet", CallSiteEHPad);
+
+        Instruction *NewInst;
+        if (CS.isCall())
+          NewInst = CallInst::Create(cast<CallInst>(I), OpBundles, I);
+        else
+          NewInst = InvokeInst::Create(cast<InvokeInst>(I), OpBundles, I);
+        NewInst->setDebugLoc(I->getDebugLoc());
+        NewInst->takeName(I);
+        I->replaceAllUsesWith(NewInst);
+        I->eraseFromParent();
+
+        OpBundles.clear();
+      }
+
       Instruction *I = BB->getFirstNonPHI();
       if (!I->isEHPad())
         continue;
 
       if (auto *CatchSwitch = dyn_cast<CatchSwitchInst>(I)) {
         if (isa<ConstantTokenNone>(CatchSwitch->getParentPad()))
           CatchSwitch->setParentPad(CallSiteEHPad);
       } else {

llvm/trunk/lib/Transforms/Utils/Local.cpp

 }
 
 bool llvm::replaceDbgDeclareForAlloca(AllocaInst *AI, Value *NewAllocaAddress,
                                       DIBuilder &Builder, bool Deref, int Offset) {
   return replaceDbgDeclare(AI, NewAllocaAddress, AI->getNextNode(), Builder,
                            Deref, Offset);
 }
 
-/// changeToUnreachable - Insert an unreachable instruction before the specified
-/// instruction, making it and the rest of the code in the block dead.
-static void changeToUnreachable(Instruction *I, bool UseLLVMTrap) {
+void llvm::changeToUnreachable(Instruction *I, bool UseLLVMTrap) {
   BasicBlock *BB = I->getParent();
   // Loop over all of the successors, removing BB's entry from any PHI
   // nodes.
   for (succ_iterator SI = succ_begin(BB), SE = succ_end(BB); SI != SE; ++SI)
     (*SI)->removePredecessor(BB);
 
   // Insert a call to llvm.trap right before this.  This turns the undefined
   // behavior into a hard fail instead of falling through into random code.

llvm/trunk/test/CodeGen/WinEH/wineh-cloning.ll

 ; RUN: opt -mtriple=x86_64-pc-windows-msvc -S -winehprepare  < %s | FileCheck %s
 
 declare i32 @__CxxFrameHandler3(...)
 declare i32 @__C_specific_handler(...)
 
 declare void @f()
-declare i32 @g()
-declare void @h(i32)
-declare i1 @b()
 
+declare void @llvm.foo(i32) nounwind
+declare void @llvm.bar() nounwind
+declare i32 @llvm.qux() nounwind
+declare i1 @llvm.baz() nounwind
 
 define void @test1() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; %x def colors: {entry} subset of use colors; must spill
-  %x = call i32 @g()
+  %x = call i32 @llvm.qux()
   invoke void @f()
     to label %noreturn unwind label %catch.switch
 catch.switch:
   %cs = catchswitch within none [label %catch] unwind to caller
 catch:
-  catchpad within %cs []
+  %cp = catchpad within %cs []
   br label %noreturn
 noreturn:
   ; %x use colors: {entry, cleanup}
-  call void @h(i32 %x)
+  call void @llvm.foo(i32 %x)
   unreachable
 }
 ; Need two copies of the call to @h, one under entry and one under catch.
 ; Currently we generate a load for each, though we shouldn't need one
 ; for the use in entry's copy.
 ; CHECK-LABEL: define void @test1(
 ; CHECK: entry:
-; CHECK:   %x = call i32 @g()
+; CHECK:   %x = call i32 @llvm.qux()
 ; CHECK:   invoke void @f()
 ; CHECK:     to label %[[EntryCopy:[^ ]+]] unwind label %catch
 ; CHECK: catch.switch:
 ; CHECK:   %cs = catchswitch within none [label %catch] unwind to caller
 ; CHECK: catch:
 ; CHECK:   catchpad within %cs []
-; CHECK-NEXT: call void @h(i32 %x)
+; CHECK-NEXT: call void @llvm.foo(i32 %x)
 ; CHECK: [[EntryCopy]]:
-; CHECK:   call void @h(i32 %x)
+; CHECK:   call void @llvm.foo(i32 %x)
 
 
 define void @test2() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   invoke void @f()
     to label %exit unwind label %cleanup
 cleanup:
   cleanuppad within none []
   br label %exit
 exit:
-  call void @f()
+  call void @llvm.bar()
   ret void
 }
 ; Need two copies of %exit's call to @f -- the subsequent ret is only
 ; valid when coming from %entry, but on the path from %cleanup, this
 ; might be a valid call to @f which might dynamically not return.
 ; CHECK-LABEL: define void @test2(
 ; CHECK: entry:
 ; CHECK:   invoke void @f()
 ; CHECK:     to label %[[exit:[^ ]+]] unwind label %cleanup
 ; CHECK: cleanup:
 ; CHECK:   cleanuppad within none []
-; CHECK:   call void @f()
+; CHECK:   call void @llvm.bar()
 ; CHECK-NEXT: unreachable
 ; CHECK: [[exit]]:
-; CHECK:   call void @f()
+; CHECK:   call void @llvm.bar()
 ; CHECK-NEXT: ret void
 
 
 define void @test3() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   invoke void @f()
     to label %invoke.cont unwind label %catch.switch
 invoke.cont:
   invoke void @f()
     to label %exit unwind label %cleanup
 catch.switch:
   %cs = catchswitch within none [label %catch] unwind to caller
 catch:
   catchpad within %cs []
   br label %shared
 cleanup:
   cleanuppad within none []
   br label %shared
 shared:
-  call void @f()
+  call void @llvm.bar()
   br label %exit
 exit:
   ret void
 }
 ; Need two copies of %shared's call to @f (similar to @test2 but
 ; the two regions here are siblings, not parent-child).
 ; CHECK-LABEL: define void @test3(
 ; CHECK:   invoke void @f()
 ; CHECK:   invoke void @f()
 ; CHECK:     to label %[[exit:[^ ]+]] unwind
 ; CHECK: catch:
 ; CHECK:   catchpad within %cs []
-; CHECK-NEXT: call void @f()
+; CHECK-NEXT: call void @llvm.bar()
 ; CHECK-NEXT: unreachable
 ; CHECK: cleanup:
 ; CHECK:   cleanuppad within none []
-; CHECK:   call void @f()
+; CHECK:   call void @llvm.bar()
 ; CHECK-NEXT: unreachable
 ; CHECK: [[exit]]:
 ; CHECK:   ret void
 
 
 define void @test4() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   invoke void @f()
     to label %shared unwind label %catch.switch
 catch.switch:
   %cs = catchswitch within none [label %catch] unwind to caller
 catch:
   catchpad within %cs []
   br label %shared
 shared:
-  %x = call i32 @g()
-  %i = call i32 @g()
+  %x = call i32 @llvm.qux()
+  %i = call i32 @llvm.qux()
   %zero.trip = icmp eq i32 %i, 0
   br i1 %zero.trip, label %exit, label %loop
 loop:
   %i.loop = phi i32 [ %i, %shared ], [ %i.dec, %loop.tail ]
-  %b = call i1 @b()
+  %b = call i1 @llvm.baz()
   br i1 %b, label %left, label %right
 left:
-  %y = call i32 @g()
+  %y = call i32 @llvm.qux()
   br label %loop.tail
 right:
-  call void @h(i32 %x)
+  call void @llvm.foo(i32 %x)
   br label %loop.tail
 loop.tail:
   %i.dec = sub i32 %i.loop, 1
   %done = icmp eq i32 %i.dec, 0
   br i1 %done, label %exit, label %loop
 exit:
-  call void @h(i32 %x)
+  call void @llvm.foo(i32 %x)
   unreachable
 }
 ; Make sure we can clone regions that have internal control
 ; flow and SSA values.  Here we need two copies of everything
 ; from %shared to %exit.
 ; CHECK-LABEL: define void @test4(
 ; CHECK:  entry:
 ; CHECK:    to label %[[shared_E:[^ ]+]] unwind label %catch.switch
 ; CHECK:  catch:
 ; CHECK:    catchpad within %cs []
-; CHECK:    [[x_C:%[^ ]+]] = call i32 @g()
-; CHECK:    [[i_C:%[^ ]+]] = call i32 @g()
+; CHECK:    [[x_C:%[^ ]+]] = call i32 @llvm.qux()
+; CHECK:    [[i_C:%[^ ]+]] = call i32 @llvm.qux()
 ; CHECK:    [[zt_C:%[^ ]+]] = icmp eq i32 [[i_C]], 0
 ; CHECK:    br i1 [[zt_C]], label %[[exit_C:[^ ]+]], label %[[loop_C:[^ ]+]]
 ; CHECK:  [[shared_E]]:
-; CHECK:    [[x_E:%[^ ]+]] = call i32 @g()
-; CHECK:    [[i_E:%[^ ]+]] = call i32 @g()
+; CHECK:    [[x_E:%[^ ]+]] = call i32 @llvm.qux()
+; CHECK:    [[i_E:%[^ ]+]] = call i32 @llvm.qux()
 ; CHECK:    [[zt_E:%[^ ]+]] = icmp eq i32 [[i_E]], 0
 ; CHECK:    br i1 [[zt_E]], label %[[exit_E:[^ ]+]], label %[[loop_E:[^ ]+]]
 ; CHECK:  [[loop_C]]:
 ; CHECK:    [[iloop_C:%[^ ]+]] = phi i32 [ [[i_C]], %catch ], [ [[idec_C:%[^ ]+]], %[[looptail_C:[^ ]+]] ]
-; CHECK:    [[b_C:%[^ ]+]] = call i1 @b()
+; CHECK:    [[b_C:%[^ ]+]] = call i1 @llvm.baz()
 ; CHECK:    br i1 [[b_C]], label %[[left_C:[^ ]+]], label %[[right_C:[^ ]+]]
 ; CHECK:  [[loop_E]]:
 ; CHECK:    [[iloop_E:%[^ ]+]] = phi i32 [ [[i_E]], %[[shared_E]] ], [ [[idec_E:%[^ ]+]], %[[looptail_E:[^ ]+]] ]
-; CHECK:    [[b_E:%[^ ]+]] = call i1 @b()
+; CHECK:    [[b_E:%[^ ]+]] = call i1 @llvm.baz()
 ; CHECK:    br i1 [[b_E]], label %[[left_E:[^ ]+]], label %[[right_E:[^ ]+]]
 ; CHECK:  [[left_C]]:
-; CHECK:    [[y_C:%[^ ]+]] = call i32 @g()
+; CHECK:    [[y_C:%[^ ]+]] = call i32 @llvm.qux()
 ; CHECK:    br label %[[looptail_C]]
 ; CHECK:  [[left_E]]:
-; CHECK:    [[y_E:%[^ ]+]] = call i32 @g()
+; CHECK:    [[y_E:%[^ ]+]] = call i32 @llvm.qux()
 ; CHECK:    br label %[[looptail_E]]
 ; CHECK:  [[right_C]]:
-; CHECK:    call void @h(i32 [[x_C]])
+; CHECK:    call void @llvm.foo(i32 [[x_C]])
 ; CHECK:    br label %[[looptail_C]]
 ; CHECK:  [[right_E]]:
-; CHECK:    call void @h(i32 [[x_E]])
+; CHECK:    call void @llvm.foo(i32 [[x_E]])
 ; CHECK:    br label %[[looptail_E]]
 ; CHECK:  [[looptail_C]]:
 ; CHECK:    [[idec_C]] = sub i32 [[iloop_C]], 1
 ; CHECK:    [[done_C:%[^ ]+]] = icmp eq i32 [[idec_C]], 0
 ; CHECK:    br i1 [[done_C]], label %[[exit_C]], label %[[loop_C]]
 ; CHECK:  [[looptail_E]]:
 ; CHECK:    [[idec_E]] = sub i32 [[iloop_E]], 1
 ; CHECK:    [[done_E:%[^ ]+]] = icmp eq i32 [[idec_E]], 0
 ; CHECK:    br i1 [[done_E]], label %[[exit_E]], label %[[loop_E]]
 ; CHECK:  [[exit_C]]:
-; CHECK:    call void @h(i32 [[x_C]])
+; CHECK:    call void @llvm.foo(i32 [[x_C]])
 ; CHECK:    unreachable
 ; CHECK:  [[exit_E]]:
-; CHECK:    call void @h(i32 [[x_E]])
+; CHECK:    call void @llvm.foo(i32 [[x_E]])
 ; CHECK:    unreachable
 
 
 define void @test5() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %exit unwind label %outer
 outer:
   %o = cleanuppad within none []
-  %x = call i32 @g()
-  invoke void @f()
+  %x = call i32 @llvm.qux()
+  invoke void @f() [ "funclet"(token %o) ]
     to label %outer.ret unwind label %catch.switch
 catch.switch:
   %cs = catchswitch within %o [label %inner] unwind to caller
 inner:
   %i = catchpad within %cs []
   catchret from %i to label %outer.post-inner
 outer.post-inner:
-  call void @h(i32 %x)
+  call void @llvm.foo(i32 %x)
   br label %outer.ret
 outer.ret:
   cleanupret from %o unwind to caller
 exit:
   ret void
 }
 ; Simple nested case (catch-inside-cleanup).  Nothing needs
 ; to be cloned.  The def and use of %x are both in %outer
 ; and so don't need to be spilled.
 ; CHECK-LABEL: define void @test5(
 ; CHECK:      outer:
-; CHECK:        %x = call i32 @g()
+; CHECK:        %x = call i32 @llvm.qux()
 ; CHECK-NEXT:   invoke void @f()
 ; CHECK-NEXT:     to label %outer.ret unwind label %catch.switch
 ; CHECK:      inner:
 ; CHECK-NEXT:   %i = catchpad within %cs []
 ; CHECK-NEXT:   catchret from %i to label %outer.post-inner
 ; CHECK:      outer.post-inner:
-; CHECK-NEXT:   call void @h(i32 %x)
+; CHECK-NEXT:   call void @llvm.foo(i32 %x)
 ; CHECK-NEXT:   br label %outer.ret
 
 
-define void @test6() personality i32 (...)* @__C_specific_handler {
-entry:
-  invoke void @f()
-    to label %invoke.cont unwind label %left
-invoke.cont:
-  invoke void @f()
-    to label %exit unwind label %right
-left:
-  cleanuppad within none []
-  br label %shared
-right:
-  %cs = catchswitch within none [label %right.catch] unwind to caller
-right.catch:
-  catchpad within %cs []
-  br label %shared
-shared:
-  %x = call i32 @g()
-  invoke void @f()
-    to label %shared.cont unwind label %inner
-shared.cont:
-  unreachable
-inner:
-  %i = cleanuppad within none []
-  call void @h(i32 %x)
-  cleanupret from %i unwind to caller
-exit:
-  ret void
-}
-; CHECK-LABEL: define void @test6(
-; CHECK:     left:
-; CHECK:       %x.for.left = call i32 @g()
-; CHECK:       invoke void @f()
-; CHECK:           to label %shared.cont.for.left unwind label %inner
-; CHECK:     right.catch:
-; CHECK:       catchpad
-; CHECK:       %x = call i32 @g()
-; CHECK:           to label %shared.cont unwind label %inner
-; CHECK:     shared.cont:
-; CHECK:       unreachable
-; CHECK:     shared.cont.for.left:
-; CHECK:       unreachable
-; CHECK:     inner:
-; CHECK:       %i = cleanuppad within none []
-; CHECK:       call void @h(i32 %x1.wineh.reload)
-; CHECK:       cleanupret from %i unwind to caller
-
-
 define void @test9() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %invoke.cont unwind label %left
 invoke.cont:
   invoke void @f()
     to label %unreachable unwind label %right
 left:
-  cleanuppad within none []
-  call void @h(i32 1)
-  invoke void @f()
+  %cp.left = cleanuppad within none []
+  call void @llvm.foo(i32 1)
+  invoke void @f() [ "funclet"(token %cp.left) ]
     to label %unreachable unwind label %right
 right:
-  cleanuppad within none []
-  call void @h(i32 2)
-  invoke void @f()
+  %cp.right = cleanuppad within none []
+  call void @llvm.foo(i32 2)
+  invoke void @f() [ "funclet"(token %cp.right) ]
     to label %unreachable unwind label %left
 unreachable:
   unreachable
 }
 ; This is an irreducible loop with two funclets that enter each other.
 ; CHECK-LABEL: define void @test9(
 ; CHECK:     entry:
 ; CHECK:               to label %invoke.cont unwind label %[[LEFT:.+]]
 ; CHECK:     invoke.cont:
 ; CHECK:               to label %[[UNREACHABLE_ENTRY:.+]] unwind label %[[RIGHT:.+]]
 ; CHECK:     [[LEFT]]:
-; CHECK:       call void @h(i32 1)
+; CHECK:       call void @llvm.foo(i32 1)
 ; CHECK:       invoke void @f()
 ; CHECK:               to label %[[UNREACHABLE_LEFT:.+]] unwind label %[[RIGHT]]
 ; CHECK:     [[RIGHT]]:
-; CHECK:       call void @h(i32 2)
+; CHECK:       call void @llvm.foo(i32 2)
 ; CHECK:       invoke void @f()
 ; CHECK:               to label %[[UNREACHABLE_RIGHT:.+]] unwind label %[[LEFT]]
 ; CHECK:     [[UNREACHABLE_RIGHT]]:
 ; CHECK:       unreachable
 ; CHECK:     [[UNREACHABLE_LEFT]]:
 ; CHECK:       unreachable
 ; CHECK:     [[UNREACHABLE_ENTRY]]:
 ; CHECK:       unreachable

 ; CHECK-NEXT:   ret void
 
 define void @test11() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %exit unwind label %cleanup.outer
 cleanup.outer:
   %outer = cleanuppad within none []
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %outer) ]
     to label %outer.cont unwind label %cleanup.inner
 outer.cont:
   br label %merge
 cleanup.inner:
   %inner = cleanuppad within %outer []
   br label %merge
 merge:
-  call void @f()
+  call void @llvm.bar()
   unreachable
 exit:
   ret void
 }
 ; merge.end will get cloned for outer and inner, but is implausible
 ; from inner, so the call @f() in inner's copy of merge should be
 ; rewritten to call @f()
 ; CHECK-LABEL: define void @test11()
 ; CHECK:      %inner = cleanuppad within %outer []
-; CHECK-NEXT: call void @f()
+; CHECK-NEXT: call void @llvm.bar()
 ; CHECK-NEXT: unreachable
 
 define void @test12() personality i32 (...)* @__CxxFrameHandler3 !dbg !5 {
 entry:
   invoke void @f()
     to label %cont unwind label %left, !dbg !8
 cont:
   invoke void @f()
     to label %exit unwind label %right
 left:
   cleanuppad within none []
   br label %join
 right:
   cleanuppad within none []
   br label %join
 join:
   ; This call will get cloned; make sure we can handle cloning
   ; instructions with debug metadata attached.
-  call void @f(), !dbg !9
+  call void @llvm.bar(), !dbg !9
   unreachable
 exit:
   ret void
 }
 
 ; CHECK-LABEL: define void @test13()
 ; CHECK: ret void
 define void @test13() personality i32 (...)* @__CxxFrameHandler3 {

llvm/trunk/test/CodeGen/WinEH/wineh-demotion.ll

 ; RUN: opt -mtriple=x86_64-pc-windows-msvc -S -winehprepare  < %s | FileCheck %s
 
 declare i32 @__CxxFrameHandler3(...)
 
 declare void @f()
 
 declare i32 @g()
 
 declare void @h(i32)
 
 declare i1 @i()
 
+declare void @llvm.bar() nounwind
+
 ; CHECK-LABEL: @test1(
 define void @test1(i1 %B) personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; Spill slot should be inserted here
   ; CHECK: [[Slot:%[^ ]+]] = alloca
   ; Can't store for %phi at these defs because the lifetimes overlap
   ; CHECK-NOT: store
   %x = call i32 @g()
merge:
   %phi = phi i32 [ %x, %left ], [ %y, %right ]
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:
   %cp = catchpad within %cs1 []
   ; CHECK: catch:
   ; CHECK: [[Reload:%[^ ]+]] = load i32, i32* [[Slot]]
   ; CHECK-NEXT: call void @h(i32 [[Reload]])
-  call void @h(i32 %phi)
+  call void @h(i32 %phi) [ "funclet"(token %cp) ]
   catchret from %cp to label %exit
 
 exit:
   ret void
 }
 
 ; CHECK-LABEL: @test2(
 define void @test2(i1 %B) personality i32 (...)* @__CxxFrameHandler3 {
merge.inner:
   ; CHECK: catchswitch within none
   %x = phi i32 [ 1, %left ], [ 2, %right ]
   %cs1 = catchswitch within none [label %catch.inner] unwind label %merge.outer
 
 catch.inner:
   %cpinner = catchpad within %cs1 []
   ; Need just one store here because only %y is affected
   ; CHECK: catch.inner:
-  %z = call i32 @g()
+  %z = call i32 @g() [ "funclet"(token %cpinner) ]
   ; CHECK:   store i32 %z
   ; CHECK-NEXT: invoke void @f
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cpinner) ]
           to label %catchret.inner unwind label %merge.outer
 
 catchret.inner:
   catchret from %cpinner to label %exit
 
 merge.outer:
   %y = phi i32 [ %x, %merge.inner ], [ %z, %catch.inner ]
   ; CHECK: merge.outer:
   ; CHECK-NOT: = phi
   ; CHECK: catchswitch within none
   %cs2 = catchswitch within none [label %catch.outer] unwind to caller
 
 catch.outer:
   %cpouter = catchpad within %cs2 []
   ; CHECK: catch.outer:
   ; CHECK: [[CatchPad:%[^ ]+]] = catchpad within %cs2 []
   ; Need to load x and y from two different slots since they're both live
   ; and can have different values (if we came from catch.inner)
   ; CHECK-DAG: load i32, i32* [[Slot1]]
   ; CHECK-DAG: load i32, i32* [[Slot2]]
   ; CHECK: catchret from [[CatchPad]] to label
-  call void @h(i32 %x)
-  call void @h(i32 %y)
+  call void @h(i32 %x) [ "funclet"(token %cpouter) ]
+  call void @h(i32 %y) [ "funclet"(token %cpouter) ]
   catchret from %cpouter to label %exit
 
 exit:
   ret void
 }
 
 ; test4: don't need stores for %phi.inner, as its only use is to feed %phi.outer
 ;        %phi.outer needs stores in %left, %right, and %join
catchpad.outer:
    ; CHECK-NEXT: catchswitch within none
    %phi.outer = phi i32 [ %phi.inner, %catchpad.inner ], [ %j, %join ]
    %cs2 = catchswitch within none [label %catch.outer] unwind to caller
 catch.outer:
    ; CHECK: catch.outer:
    ; CHECK:   [[Reload:%[^ ]+]] = load i32, i32* [[Slot]]
    ; CHECK:   call void @h(i32 [[Reload]])
    %cp2 = catchpad within %cs2 []
-   call void @h(i32 %phi.outer)
+   call void @h(i32 %phi.outer) [ "funclet"(token %cp2) ]
    catchret from %cp2 to label %exit
 exit:
    ret void
 }
 
 ; CHECK-LABEL: @test5(
 define void @test5() personality i32 (...)* @__CxxFrameHandler3 {
 entry:

 
 cleanup:
   ; cleanup phi can be loaded at cleanup entry
   ; CHECK: cleanup:
   ; CHECK-NEXT: cleanuppad within none []
   ; CHECK: [[CleanupReload:%[^ ]+]] = load i32, i32* [[CleanupSlot]]
   %phi.cleanup = phi i32 [ 1, %entry ], [ 2, %invoke.cont ]
   %cp = cleanuppad within none []
-  %b = call i1 @i()
+  %b = call i1 @i() [ "funclet"(token %cp) ]
   br i1 %b, label %left, label %right
 
 left:
   ; CHECK: left:
   ; CHECK:   call void @h(i32 [[CleanupReload]]
-  call void @h(i32 %phi.cleanup)
+  call void @h(i32 %phi.cleanup) [ "funclet"(token %cp) ]
   br label %merge
 
 right:
   ; CHECK: right:
   ; CHECK:   call void @h(i32 [[CleanupReload]]
-  call void @h(i32 %phi.cleanup)
+  call void @h(i32 %phi.cleanup) [ "funclet"(token %cp) ]
   br label %merge
 
 merge:
   ; need store for %phi.catch
   ; CHECK:      merge:
   ; CHECK-NEXT:   store i32 [[CleanupReload]], i32* [[CatchSlot:%[^ ]+]]
   ; CHECK-NEXT:   cleanupret
   cleanupret from %cp unwind label %catchswitch
catchswitch:
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:
   ; CHECK: catch:
   ; CHECK:   catchpad within %cs1
   ; CHECK:   [[CatchReload:%[^ ]+]] = load i32, i32* [[CatchSlot]]
   ; CHECK:   call void @h(i32 [[CatchReload]]
   %cp2 = catchpad within %cs1 []
-  call void @h(i32 %phi.catch)
+  call void @h(i32 %phi.catch) [ "funclet"(token %cp2) ]
   catchret from %cp2 to label %exit
 
 exit:
   ret void
 }
 
 ; We used to demote %x, but we don't need to anymore.
 ; CHECK-LABEL: @test6(
to_caller:
   cleanupret from %cp1 unwind to caller
 loop:
   invoke void @f()
           to label %loop unwind label %cleanup
 cleanup:
   ; CHECK: cleanup:
   ; CHECK:   call void @h(i32 %x)
   %cp2 = cleanuppad within none []
-  call void @h(i32 %x)
+  call void @h(i32 %x) [ "funclet"(token %cp2) ]
   cleanupret from %cp2 unwind to caller
 }
 
 ; CHECK-LABEL: @test7(
 define void @test7() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; %x is an EH pad phi, so gets stored in pred here
   ; CHECK: entry:
catchpad:
   ; CHECK: catchpad:
   ; CHECK-NEXT: catchswitch within none
   %x = phi i32 [ 1, %entry ], [ 2, %invoke.cont ]
   %cs1 = catchswitch within none [label %catch] unwind to caller
 catch:
   ; CHECK: catch:
   ; CHECK-NEXT: %[[CatchPad:[^ ]+]] = catchpad within %cs1 []
   %cp = catchpad within %cs1 []
-  %b = call i1 @i()
+  %b = call i1 @i() [ "funclet"(token %cp) ]
   br i1 %b, label %left, label %right
 left:
   ; Edge from %left to %join needs to be split so that
   ; the load of %x can be inserted *after* the catchret
   ; CHECK: left:
   ; CHECK-NEXT: catchret from %[[CatchPad]] to label %[[SplitLeft:[^ ]+]]
   catchret from %cp to label %join
   ; CHECK: [[SplitLeft]]:
   ; CHECK:   [[LoadX:%[^ ]+]] = load i32, i32* [[SlotX]]
   ; CHECK:   br label %join
 right:
   ; Edge from %right to %join needs to be split so that
   ; the load of %y can be inserted *after* the catchret
   ; CHECK: right:
   ; CHECK:   %y = call i32 @g()
   ; CHECK:   catchret from %[[CatchPad]] to label %join
-  %y = call i32 @g()
+  %y = call i32 @g() [ "funclet"(token %cp) ]
   catchret from %cp to label %join
 join:
   ; CHECK: join:
   ; CHECK:   %phi = phi i32 [ [[LoadX]], %[[SplitLeft]] ], [ %y, %right ]
   %phi = phi i32 [ %x, %left ], [ %y, %right ]
   call void @h(i32 %phi)
   br label %exit
 exit:
   ret void
 }
 
 ; CHECK-LABEL: @test8(
 define void @test8() personality i32 (...)* @__CxxFrameHandler3 { entry:
   invoke void @f()
           to label %done unwind label %cleanup1
   invoke void @f()
           to label %done unwind label %cleanup2
 
 done:
   ret void
 
 cleanup1:
   ; CHECK: [[CleanupPad1:%[^ ]+]] = cleanuppad within none []
-  ; CHECK-NEXT: call void @f()
+  ; CHECK-NEXT: call void @llvm.bar()
   ; CHECK-NEXT: cleanupret from [[CleanupPad1]]
   %cp0 = cleanuppad within none []
   br label %cleanupexit
 
 cleanup2:
   ; CHECK: cleanuppad within none []
-  ; CHECK-NEXT: call void @f()
+  ; CHECK-NEXT: call void @llvm.bar()
   ; CHECK-NEXT: unreachable
   %cp1 = cleanuppad within none []
   br label %cleanupexit
 
 cleanupexit:
-  call void @f()
+  call void @llvm.bar()
   cleanupret from %cp0 unwind label %cleanup2
 }

llvm/trunk/test/CodeGen/WinEH/wineh-no-demotion.ll

-; RUN: opt -mtriple=x86_x64-pc-windows-msvc -S -winehprepare -disable-demotion < %s | FileCheck %s
+; RUN: opt -mtriple=x86_x64-pc-windows-msvc -S -winehprepare -disable-demotion -disable-cleanups < %s | FileCheck %s
 
 declare i32 @__CxxFrameHandler3(...)
 
 declare i32 @__C_specific_handler(...)
 
 declare void @f()
 
 declare i32 @g()
 
 declare void @h(i32)
 
 ; CHECK-LABEL: @test1(
-define void @test1() personality i32 (...)* @__C_specific_handler {
+define void @test1(i1 %bool) personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
-          to label %invoke.cont1 unwind label %left
-
-invoke.cont1:
-  invoke void @f()
-          to label %invoke.cont2 unwind label %right
+          to label %invoke.cont unwind label %left
 
-invoke.cont2:
+invoke.cont:
   invoke void @f()
           to label %exit unwind label %inner
 
 left:
   %0 = cleanuppad within none []
-  br label %shared
+  br i1 %bool, label %shared, label %cleanupret
+
+cleanupret:
+  cleanupret from %0 unwind label %right
 
 right:
   %1 = cleanuppad within none []
   br label %shared
 
 shared:
   %x = call i32 @g()
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %0) ]
           to label %shared.cont unwind label %inner
 
 shared.cont:
   unreachable
 
 inner:
-  %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont2 ]
+  %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont ]
   %i = cleanuppad within none []
   call void @h(i32 %phi)
   unreachable
 
-; CHECK: %phi = phi i32 [ %x, %right ], [ 0, %invoke.cont2 ], [ %x.for.left, %left ]
+; CHECK: %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont ], [ %x.for.left, %shared.for.left ]
 ; CHECK: %i = cleanuppad within none []
 ; CHECK: call void @h(i32 %phi)
 
 exit:
   unreachable
 }
 
 ; CHECK-LABEL: @test2(
-define void @test2() personality i32 (...)* @__C_specific_handler {
+define void @test2(i1 %bool) personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
-          to label %invoke.cont unwind label %left
-
-invoke.cont:
-  invoke void @f()
-          to label %exit unwind label %right
+          to label %shared.cont unwind label %left
 
 left:
-  cleanuppad within none []
-  br label %shared
+  %0 = cleanuppad within none []
+  br i1 %bool, label %shared, label %cleanupret
+
+cleanupret:
+  cleanupret from %0 unwind label %right
 
 right:
-  cleanuppad within none []
+  %1 = cleanuppad within none []
   br label %shared
 
 shared:
   %x = call i32 @g()
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %0) ]
           to label %shared.cont unwind label %inner
 
 shared.cont:
   unreachable
 
 inner:
   %i = cleanuppad within none []
   call void @h(i32 %x)
   unreachable
 
-; CHECK:  %x1 = phi i32 [ %x.for.left, %left ], [ %x, %right ]
-; CHECK:  %i = cleanuppad within none []
-; CHECK:  call void @h(i32 %x1)
+; CHECK: %x1 = phi i32 [ %x.for.left, %shared.for.left ], [ %x, %shared ]
+; CHECK: %i = cleanuppad within none []
+; CHECK: call void @h(i32 %x1)
 
 exit:
   unreachable
 }
 
 ; CHECK-LABEL: @test4(
 define void @test4(i1 %x) personality i32 (...)* @__CxxFrameHandler3 {
 entry:

 right.other:
   br label %shared
 
 shared:
   %phi = phi i32 [ 1, %left ], [ 0, %right ], [ -1, %right.other ]
   call void @h(i32 %phi)
   unreachable
 
-; CHECK: %0 = cleanuppad within none []
-; CHECK: call void @h(i32 1)
-
-; CHECK: %1 = cleanuppad within none []
 ; CHECK: %phi = phi i32 [ 0, %right ], [ -1, %right.other ]
 ; CHECK: call void @h(i32 %phi)
 
+; CHECK: %phi.for.left = phi i32 [ 1, %left ]
+; CHECK: call void @h(i32 %phi.for.left)
+
 exit:
   unreachable
 }
 
 declare void @__std_terminate()

llvm/trunk/test/CodeGen/X86/catchpad-weight.ll

catch.dispatch.2:                                 ; preds = %catch.dispatch.1
   %cs3 = catchswitch within none [label %catch] unwind label %ehcleanup
 
 catch:                                            ; preds = %catch.dispatch.2
   %3 = catchpad within %cs3 [%rtti.TypeDescriptor7* @"\01??_R0?AUC@@@8", i32 0, i8* null]
   catchret from %3 to label %try.cont
 
 ehcleanup:                                        ; preds = %catchendblock
   %4 = cleanuppad within none []
-  call void @"\01??1HasDtor@@QEAA@XZ"(%struct.HasDtor* nonnull %o) #4
+  call void @"\01??1HasDtor@@QEAA@XZ"(%struct.HasDtor* nonnull %o) #4 [ "funclet"(token %4) ]
   cleanupret from %4 unwind to caller
 }
 
 ; Function Attrs: nounwind argmemonly
 declare void @llvm.lifetime.start(i64, i8* nocapture) #1
 
 declare void @"\01?may_throw@@YAXXZ"() #2
 

llvm/trunk/test/CodeGen/X86/cleanuppad-inalloca.ll

%call1 = invoke x86_thiscallcc %struct.A* @"\01??0A@@QAE@XZ"(%struct.A* %1)
           to label %invoke.cont unwind label %ehcleanup
 
 invoke.cont:                                      ; preds = %entry
   call void @takes_two(<{ %struct.A, %struct.A }>* inalloca nonnull %argmem)
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %2 = cleanuppad within none []
-  call x86_thiscallcc void @"\01??1A@@QAE@XZ"(%struct.A* %0)
+  call x86_thiscallcc void @"\01??1A@@QAE@XZ"(%struct.A* %0) [ "funclet"(token %2) ]
   cleanupret from %2 unwind to caller
 }
 
 ; CHECK: _passes_two:
 ; CHECK: pushl %ebp
 ; CHECK: movl %esp, %ebp
 ; CHECK: subl ${{[0-9]+}}, %esp
 ; CHECK: movl $8, %eax

llvm/trunk/test/CodeGen/X86/cleanuppad-large-codemodel.ll

 ; RUN: llc -mtriple=x86_64-pc-windows-msvc -code-model=large -o - < %s | FileCheck %s
 
 declare i32 @__CxxFrameHandler3(...)
 
 declare void @bar()
 
 define void @foo() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   invoke void @bar()
     to label %exit unwind label %cleanup
 cleanup:
   %c = cleanuppad within none []
-  call void @bar()
+  call void @bar() [ "funclet"(token %c) ]
   cleanupret from %c unwind to caller
 exit:
   ret void
 }
 
 ; CHECK: foo: # @foo
 ; CHECK: movabsq $bar, %[[reg:[^ ]*]]
 ; CHECK: callq *%[[reg]]
 ; CHECK: retq
 
 ; CHECK: "?dtor$2@?0?foo@4HA":
 ; CHECK: movabsq $bar, %[[reg:[^ ]*]]
 ; CHECK: callq *%[[reg]]
 ; CHECK: retq                            # CLEANUPRET

llvm/trunk/test/CodeGen/X86/cleanuppad-realign.ll

invoke void @f(i32 1)
           to label %invoke.cont unwind label %ehcleanup
 
 invoke.cont:                                      ; preds = %entry
   call void @Dtor(i64* %o)
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %0 = cleanuppad within none []
-  call void @Dtor(i64* %o)
+  call void @Dtor(i64* %o) [ "funclet"(token %0) ]
   cleanupret from %0 unwind to caller
 }
 
 ; X86-LABEL: _realigned_cleanup: # @realigned_cleanup
 ; X86:         pushl   %ebp
 ; X86:         movl    %esp, %ebp
 ; X86:         pushl   %ebx
 ; X86:         pushl   %edi

llvm/trunk/test/CodeGen/X86/funclet-layout.ll

entry:
   invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1
           to label %unreachable unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:                                            ; preds = %catch.dispatch
   %0 = catchpad within %cs1 [i8* null, i32 64, i8* null]
-  invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1
+  invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1 ["funclet"(token %0)]
           to label %unreachable unwind label %catch.dispatch.1
 
 catch.dispatch.1:                                 ; preds = %catch
   %cs2 = catchswitch within %0 [label %catch.3] unwind to caller
 
 catch.3:                                          ; preds = %catch.dispatch.1
   %1 = catchpad within %cs2 [i8* null, i32 64, i8* null]
   catchret from %1 to label %try.cont
entry:
   invoke void @g()
           to label %try.cont unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %catch.2] unwind label %catch.dispatch.1
 
 catch.2:                                          ; preds = %catch.dispatch
   %0 = catchpad within %cs1 [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i8* null]
-  tail call void @exit(i32 0) #2
+  tail call void @exit(i32 0) #2 [ "funclet"(token %0) ]
   unreachable
 
 catch.dispatch.1:                                 ; preds = %catch.dispatch
   %cs2 = catchswitch within none [label %catch] unwind to caller
 
 catch:                                            ; preds = %catch.dispatch.1
   %1 = catchpad within %cs2 [i8* null, i32 64, i8* null]
-  tail call void @exit(i32 0) #2
+  tail call void @exit(i32 0) #2 [ "funclet"(token %1) ]
   unreachable
 
 try.cont:                                         ; preds = %entry
   br i1 %V, label %exit_one, label %exit_two
 
 exit_one:
   tail call void @exit(i32 0)
   unreachable

llvm/trunk/test/CodeGen/X86/seh-catch-all-win32.ll

invoke void @crash() #5
           to label %__try.cont unwind label %lpad
 
 lpad:                                             ; preds = %entry
   %cs1 = catchswitch within none [label %__except] unwind to caller
 
 __except:                                         ; preds = %lpad
   %p = catchpad within %cs1 [i8* bitcast (i32 ()* @"filt$main" to i8*)]
   %code = load i32, i32* %__exceptioncode, align 4
-  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i32 0, i32 0), i32 %code) #4
+  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i32 0, i32 0), i32 %code) #4 [ "funclet"(token %p) ]
   catchret from %p to label %__try.cont
 
 __try.cont:                                       ; preds = %entry, %__except
   ret i32 0
 }
 
 define internal i32 @"filt$main"() {
 entry:

llvm/trunk/test/CodeGen/X86/seh-catch-all.ll

__try.cont:
   ret i32 0
 
 lpad:
   %cs1 = catchswitch within none [label %catchall] unwind to caller
 
 catchall:
   %p = catchpad within %cs1 [i8* null, i32 64, i8* null]
   %code = call i32 @llvm.eh.exceptioncode(token %p)
-  call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i64 0, i64 0), i32 %code)
+  call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i64 0, i64 0), i32 %code) [ "funclet"(token %p) ]
   catchret from %p to label %__try.cont
 }
 
 ; Check that we can get the exception code from eax to the printf.
 
 ; CHECK-LABEL: main:
 ; CHECK: callq crash
 ; CHECK: retq

llvm/trunk/test/CodeGen/X86/seh-catchpad.ll

catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %__except] unwind label %catch.dispatch.7
 
 __except:                                         ; preds = %catch.dispatch
   %cp1 = catchpad within %cs1 [i8* null]
   catchret from %cp1 to label %__except.2
 
 ehcleanup:                                        ; preds = %__except.2
   %cp2 = cleanuppad within none []
-  invoke fastcc void @"\01?fin$0@0@main@@"() #4
+  invoke fastcc void @"\01?fin$0@0@main@@"() #4 [ "funclet"(token %cp2) ]
           to label %invoke.cont.6 unwind label %catch.dispatch.7
 
 invoke.cont.6:                                    ; preds = %ehcleanup
   cleanupret from %cp2 unwind label %catch.dispatch.7
 
 catch.dispatch.7:
   %cs2 = catchswitch within none [label %__except.ret] unwind to caller
 

llvm/trunk/test/CodeGen/X86/seh-except-finally.ll

invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext false, i8* %0) #5
           to label %invoke.cont2 unwind label %catch.dispatch
 
 invoke.cont2:                                     ; preds = %invoke.cont
   br label %__try.cont
 
 __finally:                                             ; preds = %entry
   %cleanuppad = cleanuppad within none []
   %locals = call i8* @llvm.localaddress()
-  invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext true, i8* %locals) #5
+  invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext true, i8* %locals) #5 [ "funclet"(token %cleanuppad) ]
           to label %invoke.cont3 unwind label %catch.dispatch
 
 invoke.cont3:                                     ; preds = %__finally
   cleanupret from %cleanuppad unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %invoke.cont3, %lpad1
   %cs1 = catchswitch within none [label %__except] unwind to caller
 
 __except:                                         ; preds = %catch.dispatch
   %catchpad = catchpad within %cs1 [i8* bitcast (i32 (i8*, i8*)* @"\01?filt$0@0@use_both@@" to i8*)]
-  %call = call i32 @puts(i8* getelementptr inbounds ([9 x i8], [9 x i8]* @"\01??_C@_08MLCMLGHM@__except?$AA@", i32 0, i32 0))
+  %call = call i32 @puts(i8* getelementptr inbounds ([9 x i8], [9 x i8]* @"\01??_C@_08MLCMLGHM@__except?$AA@", i32 0, i32 0)) [ "funclet"(token %catchpad) ]
   catchret from %catchpad to label %__try.cont
 
 __try.cont:                                       ; preds = %__except, %invoke.cont2
   ret void
 }
 
 ; CHECK-LABEL: use_both:
 ; CHECK: .Ltmp0

llvm/trunk/test/CodeGen/X86/seh-finally.ll

 
 invoke.cont:                                      ; preds = %entry
   %call = call i32 @puts(i8* getelementptr inbounds ([10 x i8], [10 x i8]* @str_recovered, i64 0, i64 0))
   call void @abort()
   ret i32 0
 
 lpad:                                             ; preds = %entry
   %p = cleanuppad within none []
-  %call2 = call i32 @puts(i8* getelementptr inbounds ([10 x i8], [10 x i8]* @str_recovered, i64 0, i64 0))
+  %call2 = call i32 @puts(i8* getelementptr inbounds ([10 x i8], [10 x i8]* @str_recovered, i64 0, i64 0)) [ "funclet"(token %p) ]
   cleanupret from %p unwind to caller
 }
 
 ; X64-LABEL: main:
 ; X64: retq
 
 ; X64: .seh_handlerdata
 ; X64-NEXT: .long   (.Llsda_end0-.Llsda_begin0)/16

llvm/trunk/test/CodeGen/X86/seh-safe-div-win32.ll

entry:
   invoke void @try_body(i32* %r, i32* %n, i32* %d)
           to label %__try.cont unwind label %lpad0
 
 lpad0:
   %cs0 = catchswitch within none [label %handler0] unwind label %lpad1
 
 handler0:
   %p0 = catchpad within %cs0 [i8* bitcast (i32 ()* @safe_div_filt0 to i8*)]
-  call void @puts(i8* getelementptr ([27 x i8], [27 x i8]* @str1, i32 0, i32 0))
+  call void @puts(i8* getelementptr ([27 x i8], [27 x i8]* @str1, i32 0, i32 0)) [ "funclet"(token %p0) ]
   store i32 -1, i32* %r, align 4
   catchret from %p0 to label %__try.cont
 
 lpad1:
   %cs1 = catchswitch within none [label %handler1] unwind to caller
 
 handler1:
   %p1 = catchpad within %cs1 [i8* bitcast (i32 ()* @safe_div_filt1 to i8*)]
-  call void @puts(i8* getelementptr ([29 x i8], [29 x i8]* @str2, i32 0, i32 0))
+  call void @puts(i8* getelementptr ([29 x i8], [29 x i8]* @str2, i32 0, i32 0)) [ "funclet"(token %p1) ]
   store i32 -2, i32* %r, align 4
   catchret from %p1 to label %__try.cont
 
 __try.cont:
   %safe_ret = load i32, i32* %r, align 4
   ret i32 %safe_ret
 }
 

llvm/trunk/test/CodeGen/X86/seh-safe-div.ll

entry:
   invoke void @try_body(i32* %r, i32* %n, i32* %d)
           to label %__try.cont unwind label %lpad0
 
 lpad0:
   %cs0 = catchswitch within none [label %handler0] unwind label %lpad1
 
 handler0:
   %p0 = catchpad within %cs0 [i8* bitcast (i32 (i8*, i8*)* @safe_div_filt0 to i8*)]
-  call void @puts(i8* getelementptr ([27 x i8], [27 x i8]* @str1, i32 0, i32 0))
+  call void @puts(i8* getelementptr ([27 x i8], [27 x i8]* @str1, i32 0, i32 0)) [ "funclet"(token %p0) ]
   store i32 -1, i32* %r, align 4
   catchret from %p0 to label %__try.cont
 
 lpad1:
   %cs1 = catchswitch within none [label %handler1] unwind to caller
 
 handler1:
   %p1 = catchpad within %cs1 [i8* bitcast (i32 (i8*, i8*)* @safe_div_filt1 to i8*)]
-  call void @puts(i8* getelementptr ([29 x i8], [29 x i8]* @str2, i32 0, i32 0))
+  call void @puts(i8* getelementptr ([29 x i8], [29 x i8]* @str2, i32 0, i32 0)) [ "funclet"(token %p1) ]
   store i32 -2, i32* %r, align 4
   catchret from %p1 to label %__try.cont
 
 __try.cont:
   %safe_ret = load i32, i32* %r, align 4
   ret i32 %safe_ret
 }
 

llvm/trunk/test/CodeGen/X86/seh-stack-realign.ll

invoke void @crash() #5
           to label %__try.cont unwind label %lpad
 
 lpad:                                             ; preds = %entry
   %cs1 = catchswitch within none [label %__except] unwind to caller
 
 __except:                                         ; preds = %lpad
   %p = catchpad within %cs1 [i8* bitcast (i32 ()* @"filt$main" to i8*)]
   %code = load i32, i32* %__exceptioncode, align 4
-  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i32 0, i32 0), i32 %code) #4
+  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([27 x i8], [27 x i8]* @str, i32 0, i32 0), i32 %code) #4 [ "funclet"(token %p) ]
   catchret from %p to label %__try.cont
 
 __try.cont:                                       ; preds = %entry, %__except
   ret i32 0
 }
 
 define internal i32 @"filt$main"() {
 entry:

llvm/trunk/test/CodeGen/X86/win-catchpad-csrs.ll

 try.cont:
   ret i32 0
 
 catch.dispatch:
   %cs = catchswitch within none [label %handler1] unwind to caller
 
 handler1:
   %h1 = catchpad within %cs [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i8* null]
-  call void @f(i32 2)
+  call void @f(i32 2) [ "funclet"(token %h1) ]
   catchret from %h1 to label %try.cont
 }
 
 ; X86-LABEL: _try_catch_catch:
 ; X86: pushl %ebp
 ; X86: movl %esp, %ebp
 ; X86: pushl %ebx
 ; X86: pushl %edi

llvm/trunk/test/CodeGen/X86/win-catchpad-nested-cxx.ll

invoke void @f(i32 1)
           to label %try.cont unwind label %catch.dispatch.1
 try.cont:
   ret i32 0
 
 catch.dispatch.1:
   %cs1 = catchswitch within none [label %handler1] unwind to caller
 handler1:
   %h1 = catchpad within %cs1 [i8* null, i32 64, i8* null]
-  invoke void @f(i32 2)
+  invoke void @f(i32 2) [ "funclet"(token %h1) ]
           to label %catchret1 unwind label %catch.dispatch.2
 catchret1:
   catchret from %h1 to label %try.cont
 
 catch.dispatch.2:
   %cs2 = catchswitch within %h1 [label %handler2] unwind to caller
 handler2:
   %h2 = catchpad within %cs2 [i8* null, i32 64, i8* null]

llvm/trunk/test/CodeGen/X86/win-catchpad-nested.ll

 exit:
   ret void
 
 catch.dispatch.1:
   %cs1 = catchswitch within none [label %outer.catch] unwind to caller
 
 outer.catch:
   %cp1 = catchpad within %cs1 [i32 1]
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cp1) ]
           to label %outer.ret unwind label %catch.dispatch.2
 outer.ret:
   catchret from %cp1 to label %exit
 
 catch.dispatch.2:
   %cs2 = catchswitch within %cp1 [label %inner.catch] unwind to caller
 inner.catch:
   %cp2 = catchpad within %cs2 [i32 2]

llvm/trunk/test/CodeGen/X86/win-catchpad.ll

invoke void @f(i32 1, i32* %local)
           to label %try.cont unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %cs = catchswitch within none [label %handler1, label %handler2] unwind to caller
 
 handler1:
   %h1 = catchpad within %cs [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i32* %e.addr]
   %e = load i32, i32* %e.addr
-  call void @f(i32 %e, i32* %local)
+  call void @f(i32 %e, i32* %local) [ "funclet"(token %h1) ]
   catchret from %h1 to label %try.cont
 
 handler2:
   %h2 = catchpad within %cs [i8* null, i32 64, i8* null]
-  call void @f(i32 3, i32* %local)
+  call void @f(i32 3, i32* %local) [ "funclet"(token %h2) ]
   catchret from %h2 to label %try.cont
 
 try.cont:
   ret i32 0
 }
 
 ; X86-LABEL: _try_catch_catch:
 ; X86: movl %esp, -[[sp_offset:[0-9]+]](%ebp)

 catch.dispatch:
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:
   %cp1 = catchpad within %cs1 [i8* null, i32 64, i8* null]
   br label %loop
 
 loop:
-  %V = call i1 @getbool()
+  %V = call i1 @getbool() [ "funclet"(token %cp1) ]
   br i1 %V, label %loop, label %catch.done
 
 catch.done:
   catchret from %cp1 to label %try.cont
 
 try.cont:
   ret i32 0
 }

llvm/trunk/test/CodeGen/X86/win-cleanuppad.ll

invoke void @f(i32 1)
           to label %invoke.cont unwind label %ehcleanup
 
 invoke.cont:                                      ; preds = %entry
   call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o) #2
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %0 = cleanuppad within none []
-  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o) #2
+  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o) #2 [ "funclet"(token %0) ]
   cleanupret from %0 unwind to caller
 }
 
 ; CHECK: simple_cleanup:                         # @simple_cleanup
 ; CHECK:         pushq   %rbp
 ; CHECK:         subq    $48, %rsp
 ; CHECK:         leaq    48(%rsp), %rbp
 ; CHECK:         movq    $-2, -8(%rbp)
invoke void @f(i32 3)
           to label %invoke.cont.2 unwind label %cleanup.outer
 
 invoke.cont.2:                                    ; preds = %invoke.cont.1
   call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o1) #2
   ret void
 
 cleanup.inner:                                        ; preds = %invoke.cont
   %0 = cleanuppad within none []
-  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o2) #2
+  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o2) #2 [ "funclet"(token %0) ]
   cleanupret from %0 unwind label %cleanup.outer
 
 cleanup.outer:                                      ; preds = %invoke.cont.1, %cleanup.inner, %entry
   %1 = cleanuppad within none []
-  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o1) #2
+  call x86_thiscallcc void @"\01??1Dtor@@QAE@XZ"(%struct.Dtor* %o1) #2 [ "funclet"(token %1) ]
   cleanupret from %1 unwind to caller
 }
 
 ; X86-LABEL: _nested_cleanup:
 ; X86: movl    $1, (%esp)
 ; X86: calll   _f
 ; X86: movl    $2, (%esp)
 ; X86: calll   _f

llvm/trunk/test/CodeGen/X86/win-funclet-cfi.ll

 ; RUN: llc -mtriple=x86_64-windows-msvc < %s | FileCheck %s
 
 target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc"
 
 define void @"\01?f@@YAXXZ"(i1 %B) personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   invoke void @g()
           to label %unreachable unwind label %cleanupblock
 
 cleanupblock:
   %cleanp = cleanuppad within none []
-  call void @g()
+  call void @g() [ "funclet"(token %cleanp) ]
   cleanupret from %cleanp unwind label %catch.dispatch
 
 catch.dispatch:
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:
   %cp = catchpad within %cs1 [i8* null, i32 64, i8* null]
-  call void @g()
+  call void @g() [ "funclet"(token %cp) ]
   catchret from %cp to label %try.cont
 
 try.cont:
   ret void
 
 unreachable:
   unreachable
 }

llvm/trunk/test/CodeGen/X86/win32-eh-states.ll

 try.cont.9:                                       ; preds = %invoke.cont.3, %invoke.cont, %catch.7
   ret void
 
 lpad:                                             ; preds = %catch, %entry
   %cs1 = catchswitch within none [label %catch] unwind label %lpad.1
 
 catch:                                            ; preds = %lpad.1
   %p1 = catchpad within %cs1 [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i8* null]
-  invoke void @may_throw(i32 3)
+  invoke void @may_throw(i32 3) [ "funclet"(token %p1) ]
           to label %invoke.cont.3 unwind label %lpad.1
 
 invoke.cont.3:                                    ; preds = %catch
   catchret from %p1 to label %try.cont.9
 
 lpad.1:                                           ; preds = %invoke.cont
   %cs2 = catchswitch within none [label %catch.7] unwind to caller
 
 catch.7:
   %p2 = catchpad within %cs2 [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i8* null]
-  call void @may_throw(i32 4)
+  call void @may_throw(i32 4) [ "funclet"(token %p2) ]
   catchret from %p2 to label %try.cont.9
 }
 
 ; X86-LABEL: _f:
 ; X86: movl $-1, [[state:[-0-9]+]](%ebp)
 ; X86: movl $___ehhandler$f, {{.*}}
 ;
 ; X86: movl $0, [[state]](%ebp)
entry:
   invoke void @may_throw(i32 -1)
           to label %unreachable unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %0 = catchswitch within none [label %catch] unwind label %ehcleanup5
 
 catch:                                            ; preds = %catch.dispatch
   %1 = catchpad within %0 [i8* null, i32 64, i8* null]
-  invoke void @may_throw(i32 0)
+  invoke void @may_throw(i32 0) [ "funclet"(token %1) ]
           to label %invoke.cont unwind label %ehcleanup5
 
 invoke.cont:                                      ; preds = %catch
-  invoke void @may_throw(i32 1)
+  invoke void @may_throw(i32 1) [ "funclet"(token %1) ]
           to label %invoke.cont2 unwind label %ehcleanup
 
 invoke.cont2:                                     ; preds = %invoke.cont
-  invoke void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y)
+  invoke void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y) [ "funclet"(token %1) ]
           to label %invoke.cont3 unwind label %ehcleanup5
 
 invoke.cont3:                                     ; preds = %invoke.cont2
-  invoke void @may_throw(i32 2)
+  invoke void @may_throw(i32 2) [ "funclet"(token %1) ]
           to label %invoke.cont4 unwind label %ehcleanup5
 
 invoke.cont4:                                     ; preds = %invoke.cont3
   catchret from %1 to label %try.cont
 
 try.cont:                                         ; preds = %invoke.cont4
   call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x)
   ret void
 
 ehcleanup:                                        ; preds = %invoke.cont
   %2 = cleanuppad within %1 []
-  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y)
+  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y) [ "funclet"(token %2) ]
   cleanupret from %2 unwind label %ehcleanup5
 
 ehcleanup5:                                       ; preds = %invoke.cont2, %invoke.cont3, %ehcleanup, %catch, %catch.dispatch
   %3 = cleanuppad within none []
-  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x)
+  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x) [ "funclet"(token %3) ]
   cleanupret from %3 unwind to caller
 
 unreachable:                                      ; preds = %entry
   unreachable
 }
 
 ; X86-LABEL: _g:
 ; X86: movl $-1, [[state:[-0-9]+]](%ebp)

llvm/trunk/test/CodeGen/X86/win32-seh-catchpad.ll

entry:
   invoke void @f(i32 1) #3
           to label %__except unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %__except.ret] unwind to caller
 
 __except.ret:                                     ; preds = %catch.dispatch
   %0 = catchpad within %cs1 [i8* bitcast (i32 ()* @try_except_filter_catchall to i8*)]
-  call void @f(i32 2)
+  call void @f(i32 2) [ "funclet"(token %0) ]
   catchret from %0 to label %__except
 
 __except:
   ret void
 }
 
 ; CHECK-LABEL: _code_in_catchpad:
 ; CHECK: # %__except.ret

llvm/trunk/test/CodeGen/X86/win32-seh-nested-finally.ll

invoke void @f(i32 2) #3
           to label %invoke.cont.1 unwind label %ehcleanup.3
 
 invoke.cont.1:                                    ; preds = %invoke.cont
   call void @f(i32 3) #3
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %0 = cleanuppad within none []
-  invoke void @f(i32 2) #3
+  invoke void @f(i32 2) #3 [ "funclet"(token %0) ]
           to label %invoke.cont.2 unwind label %ehcleanup.3
 
 invoke.cont.2:                                    ; preds = %ehcleanup
   cleanupret from %0 unwind label %ehcleanup.3
 
 ehcleanup.3:                                      ; preds = %invoke.cont.2, %ehcleanup.end, %invoke.cont
   %1 = cleanuppad within none []
-  call void @f(i32 3) #3
+  call void @f(i32 3) #3 [ "funclet"(token %1) ]
   cleanupret from %1 unwind to caller
 }
 
 declare void @f(i32) #0
 
 declare i32 @_except_handler3(...)
 
 attributes #0 = { "disable-tail-calls"="false" "less-precise-fpmad"="false" "no-frame-pointer-elim"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-realign-stack" "stack-protector-buffer-size"="8" "unsafe-fp-math"="false" "use-soft-float"="false" }

llvm/trunk/test/CodeGen/X86/wineh-coreclr.ll

 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK: movq %rdx, %rcx
 ;             ^ exception pointer passed in rdx
 ; CHECK-NEXT: callq g
   %exn1 = call i8 addrspace(1)* @llvm.eh.exceptionpointer.p1i8(token %catch1)
-  call void @g(i8 addrspace(1)* %exn1)
+  call void @g(i8 addrspace(1)* %exn1) [ "funclet"(token %catch1) ]
 ; CHECK: [[L_before_f3:.+]]:
 ; CHECK-NEXT: movl $3, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f3:.+]]:
-  invoke void @f(i32 3)
+  invoke void @f(i32 3) [ "funclet"(token %catch1) ]
     to label %catch1.ret unwind label %finally.pad
 catch1.ret:
   catchret from %catch1 to label %finally.clone
 catch2.body:
   %catch2 = catchpad within %cs1 [i32 2]
 ; CHECK: .seh_proc [[L_catch2:[^ ]+]]
 ; CHECK: .seh_stackalloc [[FuncletFrameSize:[0-9]+]]
 ;                        ^ all funclets use the same frame size
 ; CHECK: movq [[PSPSymOffset]](%rcx), %rcx
 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK: movq %rdx, %rcx
 ;             ^ exception pointer passed in rdx
 ; CHECK-NEXT: callq g
   %exn2 = call i8 addrspace(1)* @llvm.eh.exceptionpointer.p1i8(token %catch2)
-  call void @g(i8 addrspace(1)* %exn2)
+  call void @g(i8 addrspace(1)* %exn2) [ "funclet"(token %catch2) ]
 ; CHECK: [[L_before_f4:.+]]:
 ; CHECK-NEXT: movl $4, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f4:.+]]:
-  invoke void @f(i32 4)
+  invoke void @f(i32 4) [ "funclet"(token %catch2) ]
     to label %try_in_catch unwind label %finally.pad
 try_in_catch:
 ; CHECK: # %try_in_catch
 ; CHECK: [[L_before_f5:.+]]:
 ; CHECK-NEXT: movl $5, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f5:.+]]:
-  invoke void @f(i32 5)
+  invoke void @f(i32 5) [ "funclet"(token %catch2) ]
     to label %catch2.ret unwind label %fault.pad
 fault.pad:
 ; CHECK: .seh_proc [[L_fault:[^ ]+]]
   %fault = cleanuppad within none [i32 undef]
 ; CHECK: .seh_stackalloc [[FuncletFrameSize:[0-9]+]]
 ;                        ^ all funclets use the same frame size
 ; CHECK: movq [[PSPSymOffset]](%rcx), %rcx
 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK: [[L_before_f6:.+]]:
 ; CHECK-NEXT: movl $6, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f6:.+]]:
-  invoke void @f(i32 6)
+  invoke void @f(i32 6) [ "funclet"(token %fault) ]
     to label %fault.ret unwind label %finally.pad
 fault.ret:
   cleanupret from %fault unwind label %finally.pad
 catch2.ret:
   catchret from %catch2 to label %finally.clone
 finally.clone:
   call void @f(i32 7)
   br label %tail
 finally.pad:
 ; CHECK: .seh_proc [[L_finally:[^ ]+]]
   %finally = cleanuppad within none []
 ; CHECK: .seh_stackalloc [[FuncletFrameSize:[0-9]+]]
 ;                        ^ all funclets use the same frame size
 ; CHECK: movq [[PSPSymOffset]](%rcx), %rcx
 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK-NEXT: movl $7, %ecx
 ; CHECK-NEXT: callq f
-  call void @f(i32 7)
+  call void @f(i32 7) [ "funclet"(token %finally) ]
   cleanupret from %finally unwind to caller
 tail:
   call void @f(i32 8)
   ret void
 ; CHECK: [[L_end:.*func_end.*]]:
 }
 
 ; FIXME: Verify that the new clauses are correct and re-enable these checks.

llvm/trunk/test/CodeGen/X86/wineh-exceptionpointer.ll

catch.pad:
   %cs1 = catchswitch within none [label %catch.body] unwind to caller
 catch.body:
   ; CHECK: {{^[^: ]+}}: # %catch.body
   ; CHECK: movq %rdx, %rcx
   ; CHECK-NEXT: callq g
   %catch = catchpad within %cs1 [i32 5]
   %exn = call i8 addrspace(1)* @llvm.eh.exceptionpointer.p1i8(token %catch)
   %cast_exn = bitcast i8 addrspace(1)* %exn to i32 addrspace(1)*
-  call void @g(i32 addrspace(1)* %cast_exn)
+  call void @g(i32 addrspace(1)* %cast_exn) [ "funclet"(token %catch) ]
   catchret from %catch to label %exit
 exit:
   ret void
 }