[WinEH] Use operand bundles to describe call sites

[majnemer]

Summary

SimplifyCFG allows tail merging with code which terminates in
unreachable which, in turn, makes it possible for an invoke to end up in
a funclet which it was not originally part of.

Using operand bundles on invokes allows us to determine whether or not
an invoke was part of a funclet in the source program.

Furthermore, it allows us to unambiguously answer questions about the
legality of inlining into call sites which the personality may have
trouble with.

[rnk]

lgtm, but let's wait for more feedback.

I personally like the name "funclet" for the operand bundle. It always has to be a FuncletPadInst value, so it really always is a "funclet".

[sanjoy]

Operand bundles bits lgtm with some minor comments. Doesn't have to be this patch, but please add a small blurb to the langref about the "funclet" operand bundle.

lib/CodeGen/WinEHPrepare.cpp

795

Minor: will this look cleaner with an auto?

lib/Transforms/Utils/InlineFunction.cpp

1446

Why do you need to declare Inputs separately? Why not just OpBundles.emplace_back("funclet", CallSiteEHPad);?

1449

Minor: braces unnecessary. This may even be clearer as a ternary operator.

[JosephTremoulet]

lib/CodeGen/WinEHPrepare.cpp

804–809

I think you should do this part for CallInst uses too.

lib/IR/Verifier.cpp

2397–2398

Want to check that it has token type? Or specifically that it is a FuncletPadInst?

lib/Transforms/Utils/InlineFunction.cpp

1419

s/funclets/funclets and callsites/?

1434–1438

If you agree with my feedback in WinEHPrepare about making mis-annotated calls equally UB to mis-annotated invokes (and it seems to me they should agree), then this part will need to agree. It might make sense to have nounwind intrinsics be a special case (and the only one) in both places. Or neither.

1454

The Create overloads you call a few lines above pass the old instruction's name as the name argument for the new instruction. Does takeName do something different than (but still compatible with) that?

Also, supposing one of these calls/invokes has debug info (line number etc) attached, is what's here sufficient to propagate it to the new one?

[majnemer]

• Address review comments

[JosephTremoulet]

LGTM, thanks.

include/llvm/IR/InstrTypes.h

uint32_t getTagID() const {
     return Tag->getValue();
   }
 
   /// \brief Return true if this is a "deopt" operand bundle.
   bool isDeoptOperandBundle() const {
     return getTagID() == LLVMContext::OB_deopt;
   }
 
+  /// \brief Return true if this is a "funclet" operand bundle.
+  bool isFuncletOperandBundle() const {
+    return getTagID() == LLVMContext::OB_funclet;
+  }
+
 private:
   /// \brief Pointer to an entry in LLVMContextImpl::getOrInsertBundleTag.
   StringMapEntry<uint32_t> *Tag;
 };
 
 /// \brief A container for an operand bundle being viewed as a set of values
 /// rather than a set of uses.
 ///
bool hasReadingOperandBundles() const {
     // least readonly.
     return hasOperandBundles();
   }
 
   /// \brief Return true if this operand bundle user has operand bundles that
   /// may write to the heap.
   bool hasClobberingOperandBundles() const {
     for (auto &BOI : bundle_op_infos()) {
-      if (BOI.Tag->second == LLVMContext::OB_deopt)
+      if (BOI.Tag->second == LLVMContext::OB_deopt ||
+          BOI.Tag->second == LLVMContext::OB_funclet)
         continue;
 
-      // This instruction has an operand bundle that is not a "deopt" operand
-      // bundle.  Assume the worst.
+      // This instruction has an operand bundle that is not known to us.
+      // Assume the worst.
       return true;
     }
 
     return false;
   }
 
   /// \brief Return true if the bundle operand at index \p OpIdx has the
   /// attribute \p A.

include/llvm/IR/LLVMContext.h

enum {
     MD_align = 17 // "align"
   };
 
   /// Known operand bundle tag IDs, which always have the same value.  All
   /// operand bundle tags that LLVM has special knowledge of are listed here.
   /// Additionally, this scheme allows LLVM to efficiently check for specific
   /// operand bundle tags without comparing strings.
   enum {
-    OB_deopt = 0,  // "deopt"
+    OB_deopt = 0,   // "deopt"
+    OB_funclet = 1, // "funclet"
   };
 
   /// getMDKindID - Return a unique non-zero ID for the specified metadata kind.
   /// This ID is uniqued across modules in the current LLVMContext.
   unsigned getMDKindID(StringRef Name) const;
 
   /// getMDKindNames - Populate client supplied SmallVector with the name for
   /// custom metadata IDs registered in this LLVMContext.

include/llvm/Transforms/Utils/Local.h

 /// \brief Replaces llvm.dbg.declare instruction when the alloca it describes
 /// is replaced with a new value. If Deref is true, an additional DW_OP_deref is
 /// prepended to the expression. If Offset is non-zero, a constant displacement
 /// is added to the expression (after the optional Deref). Offset can be
 /// negative. New llvm.dbg.declare is inserted immediately before AI.
 bool replaceDbgDeclareForAlloca(AllocaInst *AI, Value *NewAllocaAddress,
                                 DIBuilder &Builder, bool Deref, int Offset = 0);
 
+/// \brief Insert an unreachable instruction before the specified
+/// instruction, making it and the rest of the code in the block dead.
+void changeToUnreachable(Instruction *I, bool UseLLVMTrap);
+
 /// Replace 'BB's terminator with one that does not have an unwind successor
 /// block.  Rewrites `invoke` to `call`, etc.  Updates any PHIs in unwind
 /// successor.
 ///
 /// \param BB  Block whose terminator will be replaced.  Its terminator must
 ///            have an unwind successor.
 void removeUnwindEdge(BasicBlock *BB);
 

lib/CodeGen/WinEHPrepare.cpp

void WinEHPrepare::cloneCommonBlocks(Function &F) {
   }
 }
 
 void WinEHPrepare::removeImplausibleTerminators(Function &F) {
   // Remove implausible terminators and replace them with UnreachableInst.
   for (auto &Funclet : FuncletBlocks) {
     BasicBlock *FuncletPadBB = Funclet.first;
     std::vector<BasicBlock *> &BlocksInFunclet = Funclet.second;
-    Instruction *FuncletPadInst = FuncletPadBB->getFirstNonPHI();
-    auto *CatchPad = dyn_cast<CatchPadInst>(FuncletPadInst);
-    auto *CleanupPad = dyn_cast<CleanupPadInst>(FuncletPadInst);
+    Instruction *FirstNonPHI = FuncletPadBB->getFirstNonPHI();
+    auto *FuncletPad = dyn_cast<FuncletPadInst>(FirstNonPHI);
+    auto *CatchPad = dyn_cast_or_null<CatchPadInst>(FuncletPad);
+    auto *CleanupPad = dyn_cast_or_null<CleanupPadInst>(FuncletPad);
 
     for (BasicBlock *BB : BlocksInFunclet) {
       TerminatorInst *TI = BB->getTerminator();
       // CatchPadInst and CleanupPadInst can't transfer control to a ReturnInst.
-      bool IsUnreachableRet = isa<ReturnInst>(TI) && (CatchPad || CleanupPad);
+      bool IsUnreachableRet = isa<ReturnInst>(TI) && FuncletPad;
       // The token consumed by a CatchReturnInst must match the funclet token.
       bool IsUnreachableCatchret = false;
       if (auto *CRI = dyn_cast<CatchReturnInst>(TI))
         IsUnreachableCatchret = CRI->getCatchPad() != CatchPad;
       // The token consumed by a CleanupReturnInst must match the funclet token.
       bool IsUnreachableCleanupret = false;
       if (auto *CRI = dyn_cast<CleanupReturnInst>(TI))
         IsUnreachableCleanupret = CRI->getCleanupPad() != CleanupPad;
       if (IsUnreachableRet || IsUnreachableCatchret ||
           IsUnreachableCleanupret) {
-        // Loop through all of our successors and make sure they know that one
-        // of their predecessors is going away.
-        for (BasicBlock *SuccBB : TI->successors())
-          SuccBB->removePredecessor(BB);
-
-        new UnreachableInst(BB->getContext(), TI);
-        TI->eraseFromParent();
-      } else if (isa<InvokeInst>(TI)) {
+        changeToUnreachable(TI, /*UseLLVMTrap=*/false);
+      } else if (auto *II = dyn_cast<InvokeInst>(TI)) {
+        Value *FuncletBundleOperand = nullptr;
+        if (Optional<OperandBundleUse> BU =
+                II->getOperandBundle(LLVMContext::OB_funclet))
+          FuncletBundleOperand = BU->Inputs.front();
+
+        // If this invoke was not part of this funclet, remove it.
+        if (FuncletBundleOperand != FuncletPad) {
+          // First, remove the unwind edge.
+          removeUnwindEdge(BB);
+
+          // Get a pointer to the new call.
+          TI = BB->getTerminator();
+          BasicBlock::iterator CallI = std::prev(TI->getIterator());
+          auto *CI = cast<CallInst>(&*CallI);
+
+          changeToUnreachable(CI, /*UseLLVMTrap=*/false);
+        } else if (Personality == EHPersonality::MSVC_CXX && CleanupPad) {
           // Invokes within a cleanuppad for the MSVC++ personality never
           // transfer control to their unwind edge: the personality will
           // terminate the program.
-        if (Personality == EHPersonality::MSVC_CXX && CleanupPad)
           removeUnwindEdge(BB);
         }
       }
     }
   }
+}
 
 void WinEHPrepare::cleanupPreparedFunclets(Function &F) {
   // Clean-up some of the mess we made by removing useles PHI nodes, trivial
   // branches, etc.
   for (Function::iterator FI = F.begin(), FE = F.end(); FI != FE;) {
     BasicBlock *BB = &*FI++;
     SimplifyInstructionsInBlock(BB);
     ConstantFoldTerminator(BB, /*DeleteDeadConditions=*/true);

lib/IR/LLVMContext.cpp

LLVMContext::LLVMContext() : pImpl(new LLVMContextImpl(*this)) {
   unsigned AlignID = getMDKindID("align");
   assert(AlignID == MD_align && "align kind id drifted");
   (void)AlignID;
 
   auto *DeoptEntry = pImpl->getOrInsertBundleTag("deopt");
   assert(DeoptEntry->second == LLVMContext::OB_deopt &&
          "deopt operand bundle id drifted!");
   (void)DeoptEntry;
+
+  auto *FuncletEntry = pImpl->getOrInsertBundleTag("funclet");
+  assert(FuncletEntry->second == LLVMContext::OB_funclet &&
+         "funclet operand bundle id drifted!");
+  (void)FuncletEntry;
 }
 LLVMContext::~LLVMContext() { delete pImpl; }
 
 void LLVMContext::addModule(Module *M) {
   pImpl->OwnedModules.insert(M);
 }
 
 void LLVMContext::removeModule(Module *M) {

lib/IR/Verifier.cpp

void Verifier::VerifyCallSite(CallSite CS) {
   if (CS.getCalledFunction() == nullptr)
     Assert(!FTy->getReturnType()->isTokenTy(),
            "Return type cannot be token for indirect call!");
 
   if (Function *F = CS.getCalledFunction())
     if (Intrinsic::ID ID = (Intrinsic::ID)F->getIntrinsicID())
       visitIntrinsicCallSite(ID, CS);
 
-  // Verify that a callsite has at most one "deopt" operand bundle.
-  bool FoundDeoptBundle = false;
+  // Verify that a callsite has at most one "deopt" and one "funclet" operand
+  // bundle.
+  bool FoundDeoptBundle = false, FoundFuncletBundle = false;
   for (unsigned i = 0, e = CS.getNumOperandBundles(); i < e; ++i) {
-    if (CS.getOperandBundleAt(i).getTagID() == LLVMContext::OB_deopt) {
+    OperandBundleUse BU = CS.getOperandBundleAt(i);
+    uint32_t Tag = BU.getTagID();
+    if (Tag == LLVMContext::OB_deopt) {
       Assert(!FoundDeoptBundle, "Multiple deopt operand bundles", I);
       FoundDeoptBundle = true;
     }
+    if (Tag == LLVMContext::OB_funclet) {
+      Assert(!FoundFuncletBundle, "Multiple funclet operand bundles", I);
+      FoundFuncletBundle = true;
+      Assert(BU.Inputs.size() == 1,
+             "Expected exactly one funclet operand bundle input", I);
+    }
   }
 
   visitInstruction(*I);
 }
 
 /// Two types are "congruent" if they are identical, or if they are both pointer
 /// types with different pointee types and the same address space.
 static bool isTypeCongruent(Type *L, Type *R) {
void Verifier::visitCleanupPadInst(CleanupPadInst &CPI) {
   User *FirstUser = nullptr;
   BasicBlock *FirstUnwindDest = nullptr;
   for (User *U : CPI.users()) {
     BasicBlock *UnwindDest;
     if (CleanupReturnInst *CRI = dyn_cast<CleanupReturnInst>(U)) {
       UnwindDest = CRI->getUnwindDest();
     } else if (isa<CleanupPadInst>(U) || isa<CatchSwitchInst>(U)) {
       continue;
+    } else if (CallSite(U)) {
+      continue;
     } else {
       Assert(false, "bogus cleanuppad use", &CPI);
     }
 
     if (!FirstUser) {
       FirstUser = U;
       FirstUnwindDest = UnwindDest;
     } else {

lib/Transforms/Utils/InlineFunction.cpp

bool llvm::InlineFunction(CallSite CS, InlineFunctionInfo &IFI,
   const Function *CalledFunc = CS.getCalledFunction();
   if (!CalledFunc ||              // Can't inline external function or indirect
       CalledFunc->isDeclaration() || // call, or call to a vararg function!
       CalledFunc->getFunctionType()->isVarArg()) return false;
 
   // The inliner does not know how to inline through calls with operand bundles
   // in general ...
   if (CS.hasOperandBundles()) {
-    // ... but it knows how to inline through "deopt" operand bundles.
-    bool CanInline =
-        CS.getNumOperandBundles() == 1 &&
-        CS.getOperandBundleAt(0).getTagID() == LLVMContext::OB_deopt;
-    if (!CanInline)
+    for (int i = 0, e = CS.getNumOperandBundles(); i != e; ++i) {
+      uint32_t Tag = CS.getOperandBundleAt(i).getTagID();
+      // ... but it knows how to inline through "deopt" operand bundles ...
+      if (Tag == LLVMContext::OB_deopt)
+        continue;
+      // ... and "funclet" operand bundles.
+      if (Tag == LLVMContext::OB_funclet)
+        continue;
+
       return false;
     }
+  }
 
   // If the call to the callee cannot throw, set the 'nounwind' flag on any
   // calls that we inline.
   bool MarkNoUnwind = CS.doesNotThrow();
 
   BasicBlock *OrigBB = TheCall->getParent();
   Function *Caller = OrigBB->getParent();
 
if (CalledPersonality) {
     //       supersets of others and can be used in place of the other.
     else if (CalledPersonality != CallerPersonality)
       return false;
   }
 
   // We need to figure out which funclet the callsite was in so that we may
   // properly nest the callee.
   Instruction *CallSiteEHPad = nullptr;
-  if (CalledPersonality && CallerPersonality) {
-    EHPersonality Personality = classifyEHPersonality(CalledPersonality);
+  if (CallerPersonality) {
+    EHPersonality Personality = classifyEHPersonality(CallerPersonality);
     if (isFuncletEHPersonality(Personality)) {
-      DenseMap<BasicBlock *, ColorVector> CallerBlockColors =
-          colorEHFunclets(*Caller);
-      ColorVector &CallSiteColors = CallerBlockColors[OrigBB];
-      size_t NumColors = CallSiteColors.size();
-      // There is no single parent, inlining will not succeed.
-      if (NumColors > 1)
-        return false;
-      if (NumColors == 1) {
-        BasicBlock *CallSiteFuncletBB = CallSiteColors.front();
-        if (CallSiteFuncletBB != Caller->begin()) {
-          CallSiteEHPad = CallSiteFuncletBB->getFirstNonPHI();
-          assert(CallSiteEHPad->isEHPad() && "Expected an EHPad!");
-        }
-      }
+      Optional<OperandBundleUse> ParentFunclet =
+          CS.getOperandBundle(LLVMContext::OB_funclet);
+      if (ParentFunclet)
+        CallSiteEHPad = cast<FuncletPadInst>(ParentFunclet->Inputs.front());
 
       // OK, the inlining site is legal.  What about the target function?
 
       if (CallSiteEHPad) {
         if (Personality == EHPersonality::MSVC_CXX) {
           // The MSVC personality cannot tolerate catches getting inlined into
           // cleanup funclets.
           if (isa<CleanupPadInst>(CallSiteEHPad)) {
             // Ok, the call site is within a cleanuppad.  Let's check the callee
             // for catchpads.
             for (const BasicBlock &CalledBB : *CalledFunc) {
-              if (isa<CatchPadInst>(CalledBB.getFirstNonPHI()))
+              if (isa<CatchSwitchInst>(CalledBB.getFirstNonPHI()))
                 return false;
             }
           }
         } else if (isAsynchronousEHPersonality(Personality)) {
           // SEH is even less tolerant, there may not be any sort of exceptional
           // funclet in the callee.
           for (const BasicBlock &CalledBB : *CalledFunc) {
             if (CalledBB.isEHPad())
Function::iterator FirstNewBlock;
     // Remember the first block that is newly cloned over.
     FirstNewBlock = LastBlock; ++FirstNewBlock;
 
     // Inject byval arguments initialization.
     for (std::pair<Value*, Value*> &Init : ByValInit)
       HandleByValArgumentInit(Init.first, Init.second, Caller->getParent(),
                               &*FirstNewBlock, IFI);
 
-    if (CS.hasOperandBundles()) {
-      auto ParentDeopt = CS.getOperandBundleAt(0);
-      assert(ParentDeopt.getTagID() == LLVMContext::OB_deopt &&
-             "Checked on entry!");
-
+    Optional<OperandBundleUse> ParentDeopt =
+        CS.getOperandBundle(LLVMContext::OB_deopt);
+    if (ParentDeopt) {
       SmallVector<OperandBundleDef, 2> OpDefs;
 
       for (auto &VH : InlinedFunctionInfo.OperandBundleCallSites) {
         if (!VH) continue;  // instruction was DCE'd after being cloned
 
         Instruction *I = cast<Instruction>(VH);
 
         OpDefs.clear();
if (ParentDeopt) {
             continue;
           }
 
           // It may be useful to separate this logic (of handling operand
           // bundles) out to a separate "policy" component if this gets crowded.
           // Prepend the parent's deoptimization continuation to the newly
           // inlined call's deoptimization continuation.
           std::vector<Value *> MergedDeoptArgs;
-          MergedDeoptArgs.reserve(ParentDeopt.Inputs.size() +
+          MergedDeoptArgs.reserve(ParentDeopt->Inputs.size() +
                                   ChildOB.Inputs.size());
 
           MergedDeoptArgs.insert(MergedDeoptArgs.end(),
-                                 ParentDeopt.Inputs.begin(),
-                                 ParentDeopt.Inputs.end());
+                                 ParentDeopt->Inputs.begin(),
+                                 ParentDeopt->Inputs.end());
           MergedDeoptArgs.insert(MergedDeoptArgs.end(), ChildOB.Inputs.begin(),
                                  ChildOB.Inputs.end());
 
           OpDefs.emplace_back("deopt", std::move(MergedDeoptArgs));
         }
 
         Instruction *NewI = nullptr;
         if (isa<CallInst>(I))
for (ReturnInst *RI : Returns) {
       // Don't insert llvm.stackrestore calls between a musttail call and a
       // return.  The return will restore the stack pointer.
       if (InlinedMustTailCalls && RI->getParent()->getTerminatingMustTailCall())
         continue;
       IRBuilder<>(RI).CreateCall(StackRestore, SavedPtr);
     }
   }
 
   // Update the lexical scopes of the new funclets.  Anything that had 'none' as
   // its parent is now nested inside the callsite's EHPad.
+
   if (CallSiteEHPad) {
     for (Function::iterator BB = FirstNewBlock->getIterator(),
                             E = Caller->end();
          BB != E; ++BB) {
+      // Add bundle operands to any top-level call sites.
+      SmallVector<OperandBundleDef, 1> OpBundles;
+      for (BasicBlock::iterator BBI = BB->begin(), E = BB->end(); BBI != E;) {
+        Instruction *I = &*BBI++;
+        CallSite CS(I);
+        if (!CS)
+          continue;
+
+        // Skip call sites which are nounwind intrinsics.
+        auto *CalledFn =
+            dyn_cast<Function>(CS.getCalledValue()->stripPointerCasts());
+        if (CalledFn && CalledFn->isIntrinsic() && CS.doesNotThrow())
+          continue;
+
+        // Skip call sites which already have a "funclet" bundle.
+        if (CS.getOperandBundle(LLVMContext::OB_funclet))
+          continue;
+
+        CS.getOperandBundlesAsDefs(OpBundles);
+        ArrayRef<Value *> Inputs = CallSiteEHPad;
+        OpBundles.emplace_back("funclet", Inputs);
+
+        Instruction *NewInst;
+        if (CS.isCall()) {
+          NewInst = CallInst::Create(cast<CallInst>(I), OpBundles, I);
+        } else {
+          NewInst = InvokeInst::Create(cast<InvokeInst>(I), OpBundles, I);
+        }
+        NewInst->takeName(I);
+        I->replaceAllUsesWith(NewInst);
+        I->eraseFromParent();
+
+        OpBundles.clear();
+      }
+
       Instruction *I = BB->getFirstNonPHI();
       if (!I->isEHPad())
         continue;
 
       if (auto *CatchSwitch = dyn_cast<CatchSwitchInst>(I)) {
         if (isa<ConstantTokenNone>(CatchSwitch->getParentPad()))
           CatchSwitch->setParentPad(CallSiteEHPad);
       } else {

lib/Transforms/Utils/Local.cpp

 }
 
 bool llvm::replaceDbgDeclareForAlloca(AllocaInst *AI, Value *NewAllocaAddress,
                                       DIBuilder &Builder, bool Deref, int Offset) {
   return replaceDbgDeclare(AI, NewAllocaAddress, AI->getNextNode(), Builder,
                            Deref, Offset);
 }
 
-/// changeToUnreachable - Insert an unreachable instruction before the specified
-/// instruction, making it and the rest of the code in the block dead.
-static void changeToUnreachable(Instruction *I, bool UseLLVMTrap) {
+void llvm::changeToUnreachable(Instruction *I, bool UseLLVMTrap) {
   BasicBlock *BB = I->getParent();
   // Loop over all of the successors, removing BB's entry from any PHI
   // nodes.
   for (succ_iterator SI = succ_begin(BB), SE = succ_end(BB); SI != SE; ++SI)
     (*SI)->removePredecessor(BB);
 
   // Insert a call to llvm.trap right before this.  This turns the undefined
   // behavior into a hard fail instead of falling through into random code.

test/CodeGen/WinEH/wineh-cloning.ll

 
 define void @test5() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %exit unwind label %outer
 outer:
   %o = cleanuppad within none []
   %x = call i32 @g()
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %o) ]
     to label %outer.ret unwind label %catch.switch
 catch.switch:
   %cs = catchswitch within %o [label %inner] unwind to caller
 inner:
   %i = catchpad within %cs []
   catchret from %i to label %outer.post-inner
 outer.post-inner:
   call void @h(i32 %x)

 ; CHECK:      inner:
 ; CHECK-NEXT:   %i = catchpad within %cs []
 ; CHECK-NEXT:   catchret from %i to label %outer.post-inner
 ; CHECK:      outer.post-inner:
 ; CHECK-NEXT:   call void @h(i32 %x)
 ; CHECK-NEXT:   br label %outer.ret
 
 
-define void @test6() personality i32 (...)* @__C_specific_handler {
-entry:
-  invoke void @f()
-    to label %invoke.cont unwind label %left
-invoke.cont:
-  invoke void @f()
-    to label %exit unwind label %right
-left:
-  cleanuppad within none []
-  br label %shared
-right:
-  %cs = catchswitch within none [label %right.catch] unwind to caller
-right.catch:
-  catchpad within %cs []
-  br label %shared
-shared:
-  %x = call i32 @g()
-  invoke void @f()
-    to label %shared.cont unwind label %inner
-shared.cont:
-  unreachable
-inner:
-  %i = cleanuppad within none []
-  call void @h(i32 %x)
-  cleanupret from %i unwind to caller
-exit:
-  ret void
-}
-; CHECK-LABEL: define void @test6(
-; CHECK:     left:
-; CHECK:       %x.for.left = call i32 @g()
-; CHECK:       invoke void @f()
-; CHECK:           to label %shared.cont.for.left unwind label %inner
-; CHECK:     right.catch:
-; CHECK:       catchpad
-; CHECK:       %x = call i32 @g()
-; CHECK:           to label %shared.cont unwind label %inner
-; CHECK:     shared.cont:
-; CHECK:       unreachable
-; CHECK:     shared.cont.for.left:
-; CHECK:       unreachable
-; CHECK:     inner:
-; CHECK:       %i = cleanuppad within none []
-; CHECK:       call void @h(i32 %x1.wineh.reload)
-; CHECK:       cleanupret from %i unwind to caller
-
-
 define void @test9() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %invoke.cont unwind label %left
 invoke.cont:
   invoke void @f()
     to label %unreachable unwind label %right
 left:
-  cleanuppad within none []
+  %cp.left = cleanuppad within none []
   call void @h(i32 1)
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cp.left) ]
     to label %unreachable unwind label %right
 right:
-  cleanuppad within none []
+  %cp.right = cleanuppad within none []
   call void @h(i32 2)
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cp.right) ]
     to label %unreachable unwind label %left
 unreachable:
   unreachable
 }
 ; This is an irreducible loop with two funclets that enter each other.
 ; CHECK-LABEL: define void @test9(
 ; CHECK:     entry:
 ; CHECK:               to label %invoke.cont unwind label %[[LEFT:.+]]

 ; CHECK-NEXT:   ret void
 
 define void @test11() personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
     to label %exit unwind label %cleanup.outer
 cleanup.outer:
   %outer = cleanuppad within none []
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %outer) ]
     to label %outer.cont unwind label %cleanup.inner
 outer.cont:
   br label %merge
 cleanup.inner:
   %inner = cleanuppad within %outer []
   br label %merge
 merge:
   call void @f()

test/CodeGen/WinEH/wineh-demotion.ll

 
 catch.inner:
   %cpinner = catchpad within %cs1 []
   ; Need just one store here because only %y is affected
   ; CHECK: catch.inner:
   %z = call i32 @g()
   ; CHECK:   store i32 %z
   ; CHECK-NEXT: invoke void @f
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cpinner) ]
           to label %catchret.inner unwind label %merge.outer
 
 catchret.inner:
   catchret from %cpinner to label %exit
 
 merge.outer:
   %y = phi i32 [ %x, %merge.inner ], [ %z, %catch.inner ]
   ; CHECK: merge.outer:

test/CodeGen/WinEH/wineh-no-demotion.ll

-; RUN: opt -mtriple=x86_x64-pc-windows-msvc -S -winehprepare -disable-demotion < %s | FileCheck %s
+; RUN: opt -mtriple=x86_x64-pc-windows-msvc -S -winehprepare -disable-demotion -disable-cleanups < %s | FileCheck %s
 
 declare i32 @__CxxFrameHandler3(...)
 
 declare i32 @__C_specific_handler(...)
 
 declare void @f()
 
 declare i32 @g()
 
 declare void @h(i32)
 
 ; CHECK-LABEL: @test1(
-define void @test1() personality i32 (...)* @__C_specific_handler {
+define void @test1(i1 %bool) personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
-          to label %invoke.cont1 unwind label %left
-
-invoke.cont1:
-  invoke void @f()
-          to label %invoke.cont2 unwind label %right
+          to label %invoke.cont unwind label %left
 
-invoke.cont2:
+invoke.cont:
   invoke void @f()
           to label %exit unwind label %inner
 
 left:
   %0 = cleanuppad within none []
-  br label %shared
+  br i1 %bool, label %shared, label %cleanupret
+
+cleanupret:
+  cleanupret from %0 unwind label %right
 
 right:
   %1 = cleanuppad within none []
   br label %shared
 
 shared:
   %x = call i32 @g()
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %0) ]
           to label %shared.cont unwind label %inner
 
 shared.cont:
   unreachable
 
 inner:
-  %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont2 ]
+  %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont ]
   %i = cleanuppad within none []
   call void @h(i32 %phi)
   unreachable
 
-; CHECK: %phi = phi i32 [ %x, %right ], [ 0, %invoke.cont2 ], [ %x.for.left, %left ]
+; CHECK: %phi = phi i32 [ %x, %shared ], [ 0, %invoke.cont ], [ %x.for.left, %shared.for.left ]
 ; CHECK: %i = cleanuppad within none []
 ; CHECK: call void @h(i32 %phi)
 
 exit:
   unreachable
 }
 
 ; CHECK-LABEL: @test2(
-define void @test2() personality i32 (...)* @__C_specific_handler {
+define void @test2(i1 %bool) personality i32 (...)* @__C_specific_handler {
 entry:
   invoke void @f()
-          to label %invoke.cont unwind label %left
-
-invoke.cont:
-  invoke void @f()
-          to label %exit unwind label %right
+          to label %shared.cont unwind label %left
 
 left:
-  cleanuppad within none []
-  br label %shared
+  %0 = cleanuppad within none []
+  br i1 %bool, label %shared, label %cleanupret
+
+cleanupret:
+  cleanupret from %0 unwind label %right
 
 right:
-  cleanuppad within none []
+  %1 = cleanuppad within none []
   br label %shared
 
 shared:
   %x = call i32 @g()
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %0) ]
           to label %shared.cont unwind label %inner
 
 shared.cont:
   unreachable
 
 inner:
   %i = cleanuppad within none []
   call void @h(i32 %x)
   unreachable
 
-; CHECK:  %x1 = phi i32 [ %x.for.left, %left ], [ %x, %right ]
-; CHECK:  %i = cleanuppad within none []
-; CHECK:  call void @h(i32 %x1)
+; CHECK: %x1 = phi i32 [ %x.for.left, %shared.for.left ], [ %x, %shared ]
+; CHECK: %i = cleanuppad within none []
+; CHECK: call void @h(i32 %x1)
 
 exit:
   unreachable
 }
 
 ; CHECK-LABEL: @test4(
 define void @test4(i1 %x) personality i32 (...)* @__CxxFrameHandler3 {
 entry:

 right.other:
   br label %shared
 
 shared:
   %phi = phi i32 [ 1, %left ], [ 0, %right ], [ -1, %right.other ]
   call void @h(i32 %phi)
   unreachable
 
-; CHECK: %0 = cleanuppad within none []
-; CHECK: call void @h(i32 1)
-
-; CHECK: %1 = cleanuppad within none []
 ; CHECK: %phi = phi i32 [ 0, %right ], [ -1, %right.other ]
 ; CHECK: call void @h(i32 %phi)
 
+; CHECK: %phi.for.left = phi i32 [ 1, %left ]
+; CHECK: call void @h(i32 %phi.for.left)
+
 exit:
   unreachable
 }
 
 declare void @__std_terminate()

test/CodeGen/X86/funclet-layout.ll

entry:
   invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1
           to label %unreachable unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %catch] unwind to caller
 
 catch:                                            ; preds = %catch.dispatch
   %0 = catchpad within %cs1 [i8* null, i32 64, i8* null]
-  invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1
+  invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1 ["funclet"(token %0)]
           to label %unreachable unwind label %catch.dispatch.1
 
 catch.dispatch.1:                                 ; preds = %catch
   %cs2 = catchswitch within %0 [label %catch.3] unwind to caller
 
 catch.3:                                          ; preds = %catch.dispatch.1
   %1 = catchpad within %cs2 [i8* null, i32 64, i8* null]
   catchret from %1 to label %try.cont

test/CodeGen/X86/seh-catchpad.ll

catch.dispatch:                                   ; preds = %entry
   %cs1 = catchswitch within none [label %__except] unwind label %catch.dispatch.7
 
 __except:                                         ; preds = %catch.dispatch
   %cp1 = catchpad within %cs1 [i8* null]
   catchret from %cp1 to label %__except.2
 
 ehcleanup:                                        ; preds = %__except.2
   %cp2 = cleanuppad within none []
-  invoke fastcc void @"\01?fin$0@0@main@@"() #4
+  invoke fastcc void @"\01?fin$0@0@main@@"() #4 [ "funclet"(token %cp2) ]
           to label %invoke.cont.6 unwind label %catch.dispatch.7
 
 invoke.cont.6:                                    ; preds = %ehcleanup
   cleanupret from %cp2 unwind label %catch.dispatch.7
 
 catch.dispatch.7:
   %cs2 = catchswitch within none [label %__except.ret] unwind to caller
 

test/CodeGen/X86/seh-except-finally.ll

invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext false, i8* %0) #5
           to label %invoke.cont2 unwind label %catch.dispatch
 
 invoke.cont2:                                     ; preds = %invoke.cont
   br label %__try.cont
 
 __finally:                                             ; preds = %entry
   %cleanuppad = cleanuppad within none []
   %locals = call i8* @llvm.localaddress()
-  invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext true, i8* %locals) #5
+  invoke void @"\01?fin$0@0@use_both@@"(i1 zeroext true, i8* %locals) #5 [ "funclet"(token %cleanuppad) ]
           to label %invoke.cont3 unwind label %catch.dispatch
 
 invoke.cont3:                                     ; preds = %__finally
   cleanupret from %cleanuppad unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %invoke.cont3, %lpad1
   %cs1 = catchswitch within none [label %__except] unwind to caller
 

test/CodeGen/X86/win-catchpad-nested-cxx.ll

invoke void @f(i32 1)
           to label %try.cont unwind label %catch.dispatch.1
 try.cont:
   ret i32 0
 
 catch.dispatch.1:
   %cs1 = catchswitch within none [label %handler1] unwind to caller
 handler1:
   %h1 = catchpad within %cs1 [i8* null, i32 64, i8* null]
-  invoke void @f(i32 2)
+  invoke void @f(i32 2) [ "funclet"(token %h1) ]
           to label %catchret1 unwind label %catch.dispatch.2
 catchret1:
   catchret from %h1 to label %try.cont
 
 catch.dispatch.2:
   %cs2 = catchswitch within %h1 [label %handler2] unwind to caller
 handler2:
   %h2 = catchpad within %cs2 [i8* null, i32 64, i8* null]

test/CodeGen/X86/win-catchpad-nested.ll

 exit:
   ret void
 
 catch.dispatch.1:
   %cs1 = catchswitch within none [label %outer.catch] unwind to caller
 
 outer.catch:
   %cp1 = catchpad within %cs1 [i32 1]
-  invoke void @f()
+  invoke void @f() [ "funclet"(token %cp1) ]
           to label %outer.ret unwind label %catch.dispatch.2
 outer.ret:
   catchret from %cp1 to label %exit
 
 catch.dispatch.2:
   %cs2 = catchswitch within %cp1 [label %inner.catch] unwind to caller
 inner.catch:
   %cp2 = catchpad within %cs2 [i32 2]

test/CodeGen/X86/win32-eh-states.ll

 try.cont.9:                                       ; preds = %invoke.cont.3, %invoke.cont, %catch.7
   ret void
 
 lpad:                                             ; preds = %catch, %entry
   %cs1 = catchswitch within none [label %catch] unwind label %lpad.1
 
 catch:                                            ; preds = %lpad.1
   %p1 = catchpad within %cs1 [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i8* null]
-  invoke void @may_throw(i32 3)
+  invoke void @may_throw(i32 3) [ "funclet"(token %p1) ]
           to label %invoke.cont.3 unwind label %lpad.1
 
 invoke.cont.3:                                    ; preds = %catch
   catchret from %p1 to label %try.cont.9
 
 lpad.1:                                           ; preds = %invoke.cont
   %cs2 = catchswitch within none [label %catch.7] unwind to caller
 
entry:
   invoke void @may_throw(i32 -1)
           to label %unreachable unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %0 = catchswitch within none [label %catch] unwind label %ehcleanup5
 
 catch:                                            ; preds = %catch.dispatch
   %1 = catchpad within %0 [i8* null, i32 64, i8* null]
-  invoke void @may_throw(i32 0)
+  invoke void @may_throw(i32 0) [ "funclet"(token %1) ]
           to label %invoke.cont unwind label %ehcleanup5
 
 invoke.cont:                                      ; preds = %catch
-  invoke void @may_throw(i32 1)
+  invoke void @may_throw(i32 1) [ "funclet"(token %1) ]
           to label %invoke.cont2 unwind label %ehcleanup
 
 invoke.cont2:                                     ; preds = %invoke.cont
-  invoke void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y)
+  invoke void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y) [ "funclet"(token %1) ]
           to label %invoke.cont3 unwind label %ehcleanup5
 
 invoke.cont3:                                     ; preds = %invoke.cont2
-  invoke void @may_throw(i32 2)
+  invoke void @may_throw(i32 2) [ "funclet"(token %1) ]
           to label %invoke.cont4 unwind label %ehcleanup5
 
 invoke.cont4:                                     ; preds = %invoke.cont3
   catchret from %1 to label %try.cont
 
 try.cont:                                         ; preds = %invoke.cont4
   call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x)
   ret void
 
 ehcleanup:                                        ; preds = %invoke.cont
   %2 = cleanuppad within %1 []
-  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y)
+  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %y) [ "funclet"(token %2) ]
   cleanupret from %2 unwind label %ehcleanup5
 
 ehcleanup5:                                       ; preds = %invoke.cont2, %invoke.cont3, %ehcleanup, %catch, %catch.dispatch
   %3 = cleanuppad within none []
-  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x)
+  call void @"\01??1S@@QEAA@XZ"(%struct.S* nonnull %x) [ "funclet"(token %3) ]
   cleanupret from %3 unwind to caller
 
 unreachable:                                      ; preds = %entry
   unreachable
 }
 
 ; X86-LABEL: _g:
 ; X86: movl $-1, [[state:[-0-9]+]](%ebp)

test/CodeGen/X86/win32-seh-nested-finally.ll

invoke void @f(i32 2) #3
           to label %invoke.cont.1 unwind label %ehcleanup.3
 
 invoke.cont.1:                                    ; preds = %invoke.cont
   call void @f(i32 3) #3
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %0 = cleanuppad within none []
-  invoke void @f(i32 2) #3
+  invoke void @f(i32 2) #3 [ "funclet"(token %0) ]
           to label %invoke.cont.2 unwind label %ehcleanup.3
 
 invoke.cont.2:                                    ; preds = %ehcleanup
   cleanupret from %0 unwind label %ehcleanup.3
 
 ehcleanup.3:                                      ; preds = %invoke.cont.2, %ehcleanup.end, %invoke.cont
   %1 = cleanuppad within none []
   call void @f(i32 3) #3

test/CodeGen/X86/wineh-coreclr.ll

 ;             ^ exception pointer passed in rdx
 ; CHECK-NEXT: callq g
   %exn1 = call i8 addrspace(1)* @llvm.eh.exceptionpointer.p1i8(token %catch1)
   call void @g(i8 addrspace(1)* %exn1)
 ; CHECK: [[L_before_f3:.+]]:
 ; CHECK-NEXT: movl $3, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f3:.+]]:
-  invoke void @f(i32 3)
+  invoke void @f(i32 3) [ "funclet"(token %catch1) ]
     to label %catch1.ret unwind label %finally.pad
 catch1.ret:
   catchret from %catch1 to label %finally.clone
 catch2.body:
   %catch2 = catchpad within %cs1 [i32 2]
 ; CHECK: .seh_proc [[L_catch2:[^ ]+]]
 ; CHECK: .seh_stackalloc [[FuncletFrameSize:[0-9]+]]
 ;                        ^ all funclets use the same frame size
 ; CHECK: movq [[PSPSymOffset]](%rcx), %rcx
 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK: movq %rdx, %rcx
 ;             ^ exception pointer passed in rdx
 ; CHECK-NEXT: callq g
   %exn2 = call i8 addrspace(1)* @llvm.eh.exceptionpointer.p1i8(token %catch2)
   call void @g(i8 addrspace(1)* %exn2)
 ; CHECK: [[L_before_f4:.+]]:
 ; CHECK-NEXT: movl $4, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f4:.+]]:
-  invoke void @f(i32 4)
+  invoke void @f(i32 4) [ "funclet"(token %catch2) ]
     to label %try_in_catch unwind label %finally.pad
 try_in_catch:
 ; CHECK: # %try_in_catch
 ; CHECK: [[L_before_f5:.+]]:
 ; CHECK-NEXT: movl $5, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f5:.+]]:
-  invoke void @f(i32 5)
+  invoke void @f(i32 5) [ "funclet"(token %catch2) ]
     to label %catch2.ret unwind label %fault.pad
 fault.pad:
 ; CHECK: .seh_proc [[L_fault:[^ ]+]]
   %fault = cleanuppad within none [i32 undef]
 ; CHECK: .seh_stackalloc [[FuncletFrameSize:[0-9]+]]
 ;                        ^ all funclets use the same frame size
 ; CHECK: movq [[PSPSymOffset]](%rcx), %rcx
 ;                              ^ establisher frame pointer passed in rcx
 ; CHECK: movq %rcx, [[PSPSymOffset]](%rsp)
 ; CHECK: leaq [[FPOffset]](%rcx), %rbp
 ; CHECK: .seh_endprologue
 ; CHECK: [[L_before_f6:.+]]:
 ; CHECK-NEXT: movl $6, %ecx
 ; CHECK-NEXT: callq f
 ; CHECK-NEXT: [[L_after_f6:.+]]:
-  invoke void @f(i32 6)
+  invoke void @f(i32 6) [ "funclet"(token %fault) ]
     to label %fault.ret unwind label %finally.pad
 fault.ret:
   cleanupret from %fault unwind label %finally.pad
 catch2.ret:
   catchret from %catch2 to label %finally.clone
 finally.clone:
   call void @f(i32 7)
   br label %tail