[WinEH] Require token linkage in EH pad/ret signatures

[JosephTremoulet]

Summary

WinEHPrepare is going to require that cleanuppad and catchpad produce values
of token type which are consumed by any cleanupret or catchret exiting the
pad. This change updates the signatures of those operators to require/enforce
that the type produced by the pads is token type and that the rets have an
appropriate argument.

The catchpad argument of a CatchReturnInst must be a CatchPadInst (and
similarly for CleanupReturnInst/CleanupPadInst). To accommodate that
restriction, this change adds a notion of an operator constraint to both
LLParser and BitcodeReader, allowing appropriate sentinels to be constructed
for forward references and appropriate error messages to be emitted for
illegal inputs.

Also add a verifier rule (noted in LangRef) that a catchpad with a catchpad
predecessor must have no other predecessors; this ensures that WinEHPrepare
will see the expected linear relationship between sibling catches on the
same try.

Lastly, remove some superfluous/vestigial casts from instruction operand
setters operating on BasicBlocks.

[JosephTremoulet]

Rebase/update to include state numbering change

[rnk]

docs/LangRef.rst

5197–5198

s/consumes the catchpad's value/consumes the catchpad/ maybe?

I don't think the parenthetical makes this any clearer. WDYT?

include/llvm/IR/Instructions.h

3585–3586

Should we tighten up these Value* types to be CleanupBlockInst*?

3630–3631

Regardless of whether we tighten the constructor, these should probably use CleanupPadInst* instead of Value*. Use cast<> to pull it out, it'll assert if things go wrong.

4028–4032

Ditto, should we be more strict, or does that have wide-ranging complexity? At the very least, we should assert in init if we don't already.

4055–4059

Ditto, we should tighten the accesssors.

[JosephTremoulet]

Tightening the constructors and accessors isn't straightforward because we allow UndefValue in addition to FooPadInst. I thought it was important to allow UndefValue so that it's legal to replaceAllUsesWith(UndefValue) on, say, an unreachable instruction that happens to be an EH pad.

Since we allow UndefValue, we need to be able to serialize it, which in turn means we need to deserialize it, so we need a constructor that takes UndefValue *. I thought typing the constructor as Value * was cleanest, but I could switch to different overloads for the Undef and non-Undef cases if you prefer (assuming I was misinterpreting the dummy operands thing).

I actually first tried this change with the accessors tightened. The getFooPad() methods were straightforward enough; just dynamic_cast and let UndefValue get translated to nullptr. The setFooPad() methods are where I got stuck. I felt like you should be able to round-trip FooRetX->setFooPad(FooRetY->getFooPad()), like we do in the copy ctor, and since getFooPad() can return nullptr that meant that setFooPad needed to accept nullptr and translate it to UndefValue. With setFooPad doing that translation, I thought that FooPadInst * was a more appropriate type for the argument than Value *. But once I made setFooPad take a FooPadInst *, the constructor (that I had taking a Value * for the reasons above) couldn't pass it straight through, and that didn't feel right either.

docs/LangRef.rst

5197–5198

s/consumes the catchpad's value/consumes the catchpad/ maybe?

Yeah, that's better, will update.

I don't think the parenthetical makes this any clearer. WDYT?

I wish the parenthetical were more clear, but I think the sentence would be incorrect without it. I need EH Prep to be able to assume that there isn't a cycle from a pad to itself in a single function, but entering the same pad twice is of course totally fine in different activations of a recursive function. I'd be happy to reword if you have a suggestion that makes it clear. And I can remove the parenthetical if you think it's obvious that the recursive case is allowed, but that was my thinking.

include/llvm/IR/Instructions.h

4028–4032

At the very least, we should assert in init if we don't already

I originally had an assert in init, but it was failing running llvm-as over feature/exception.ll. I hypothesized that maybe dummy operands are created during parsing to facilitate lexically-forward references? I will put that assert back and debug deeper.

[JosephTremoulet]

I suppose that the accessors could be tightened by presenting it as two logical properties -- a bool indicating whether the value is undef (what would be a good name for that?) and a FooPadInst *, each with a get/set pair. The constructors could take Value*s and translate, or maybe have different overloads (one taking a FooPadInst * and one not) and force callers to decide.

[rnk]

Oh, shoot, undef. I think we may be able to get by without allowing undef then. LLVM's main utility for removing unreachable code is removeUnreachableBlocks(), and it doesn't do RAUW(undef). I think it assumes that if something is unreachable, then all direct uses of it must also be unreachable, since they must be dominated by unreachable code. Because of the token type rules, the EH pads will always dominate their EH return instructions, and they'll all get knocked out at once.

docs/LangRef.rst

5197–5198

I think the recursive case is eliminated because the recursive call would produce and consume a new catchpad value, sort of like how re-executing the same alloca in a recursive frame gives you new stack memory.

[JosephTremoulet]

Ok, if we can disallow undef that will certainly simplify this.

I'm not sure what to do about the llvm-as case, though. I debugged into it, and it is indeed a forward reference. What happens is we process this in exception.ll:

  catchret %cbv to label %exit
...
  %cbv = catchpad [i7 4] to label %bb unwind label %bb3

and then we go to construct the CatchReturnInst with this code in LLParser.cpp:

bool LLParser::ParseCatchRet(Instruction *&Inst, PerFunctionState &PFS) {
  Value *CatchPad = nullptr;

  if (ParseValue(Type::getTokenTy(Context), CatchPad, PFS))
    return true;

  BasicBlock *BB;
  if (ParseToken(lltok::kw_to, "expected 'to' in catchret") ||
      ParseTypeAndBasicBlock(BB, PFS))
      return true;

  Inst = CatchReturnInst::Create(CatchPad, BB);

that call to ParseValue eventually winds up in this code in PerFunctionState::GetVal:

// Otherwise, create a new forward reference for this value and remember it.
Value *FwdVal;
if (Ty->isLabelTy())
  FwdVal = BasicBlock::Create(F.getContext(), Name, &F);
else
  FwdVal = new Argument(Ty, Name);

so we need a way to create a CatchReturnInst with an Argument. Should I just reinterpret_cast the Argument * to a CatchPadInst * in LLParser? Maybe tighten the accessor and the public Create method (and IRBuilder's method) but leave the private constructors typed as Value * , make LLParser a friend class, and have the parser call the private constructor directly? The bitcode reader has the same issue btw (BitcodeReader::parseFunctionBody calls geValue which gets down into BitcodeReaderValueList::getValueFwdRef that creates an Argument).

[majnemer]

I think it's fine to let them take Value * instead of something more specific. The verifier should catch any mistakes.

[JosephTremoulet]

• Tidy up some LangRef language per review feedback
• Fix a gcc build warning

[JosephTremoulet]

I suppose I could change just the FooReturnInst::getFooPad() accessors' return types to FooPad * (and remove support for undef from the verifier, LangRef, examples, and tests). That way:

• Parsers could still sneak in an Argument through the public API (and likewise for any transforms that temporarily use placeholder values during a rewrite)
• Create and setFooPad() would agree with each other in accepting arbitrary Value * (but the verifier would catch non-FooPads if any were left)
• Callers of getFooPad() wanting a FooPadInst * wouldn't have to downcast, and wouldn't have to deal with the possibility of undef

That would mean we'll have values that can't have all their uses replaced with undef (unless those uses are rewritten/removed before the next verifier pass), but if that's not an important property to preserve for LLVM (I may have been biased by other codebases) then callers of getFooPad will benefit.

Would that be a happy compromise?

[majnemer]

In D12108#227888, @JosephTremoulet wrote:

I suppose I could change just the FooReturnInst::getFooPad() accessors' return types to FooPad * (and remove support for undef from the verifier, LangRef, examples, and tests). That way:

• Parsers could still sneak in an Argument through the public API (and likewise for any transforms that temporarily use placeholder values during a rewrite)
• Create and setFooPad() would agree with each other in accepting arbitrary Value * (but the verifier would catch non-FooPads if any were left)
• Callers of getFooPad() wanting a FooPadInst * wouldn't have to downcast, and wouldn't have to deal with the possibility of undef

That would mean we'll have values that can't have all their uses replaced with undef (unless those uses are rewritten/removed before the next verifier pass), but if that's not an important property to preserve for LLVM (I may have been biased by other codebases) then callers of getFooPad will benefit.

Would that be a happy compromise?

I think it's OK to let non-pad operands have undef so long as they are transient and will get wiped out before the next pass runs. We permit this sort of thing for degenerate phi and basic blocks.

docs/ExceptionHandling.rst

676

Shouldn't the cleanuppad be followed by []

681

Should the void be removed?

[JosephTremoulet]

• Fix some bad syntax in docs per review feedback

[JosephTremoulet]

Updating just the accessors' return types didn't work like I'd hoped. It would require the verifier to reject mismatches, but in order to do so with a proper error message both the verifier and the asm printer would have needed to get the bogus operand without calling getFooPad(), which meant either hardcoding the pad operand index in the verifier/printer (seems bad) or adding another method to the FooReturnInst classes returning the index or returning the pad operand as a Value *, at which point the API surface on the FooRetInstrs looked pretty convoluted. I also explored using a dynamic_cast in the parsers, but of course that just leads to the same issues with the verifier/printer. So I think that if we don't want the FooReturnInst ctors/accessors/mutators to all take Value * (which certainly seems in-line with keeping the IR approachable), the only thing to do that doesn't involve horrible hacks is to teach the parsers that some forward references need to be FooPads. I will look into doing that, but let me know if that seems off-base, and feel free to suggest something better.

docs/ExceptionHandling.rst

676

Yep, thanks.

681

Yes, updated.

[JosephTremoulet]

• Disallow undef in catchret/cleanupret
• Change signatures of ctors/getters/setters to on FooReturnInsts to use FooPad * rather than Value *
• Update parsers to understand operator contraints and create appropriate sentinels for forward references
• Add negative assembler tests for operator constraints

[majnemer]

include/llvm/IR/Instructions.h

37

Please sort this relative to the other forward declarations.

3713

Why is the reinterpret_cast necessary here?

4042

We can also assert that CatchPad isn't null.

lib/AsmParser/LLParser.cpp

2237–2253

Could we add a case for none? I think it'd let us reduce the indentation here.

lib/AsmParser/LLParser.h

114–116

Enumerators should start with a capital letter per the coding standards: http://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly

lib/Bitcode/Reader/BitcodeReader.cpp

46–50

Likewise with this these enumerators.

[JosephTremoulet]

include/llvm/IR/Instructions.h

3713

Evidently it's not, or the current ToT would be broken. When I tried to omit the cast from CatchPadInst * or CleanupPadInst * to Value * in setCatchPad or setCleanupPad, MSVC rejected it, which I presume has to do with some rule about concatenating implicit and explicit conversions that I didn't fully dig into. I took a look, and in the rest of this file (outside the new EH instructions) all the setters that take BasicBlock * use reinterpret_cast<Value *>, so I decided to follow suit and reinterpret_cast<Value *> in all the new EH instruction setters taking CatchPadInst * or CleanupPadInst * or BasicBlock *, for fear that different host compilers accept/reject different patterns. Since this one was clearly working before, I'm happy to revert it if you want me to, but that was my reasoning.

[JosephTremoulet]

• sort forward declarations
• assert catchpads/cleanuppads are non-null in catchret/cleanupret constructors and setters
• PascalCase new enum values, add prefix and drop qualifier in references
• fold OC_None into switches to save indentation

[JosephTremoulet]

Feedback incorporated.

lib/AsmParser/LLParser.h

114–116

Updated, thanks for the pointer.

[majnemer]

include/llvm/IR/Instructions.h

3713

I am having trouble imagining what use those reinterpret_cast are doing. If there are places where MSVC rejects, can you use static_cast instead of reinterpret_cast?

[JosephTremoulet]

• Remove unnecessary casts (c-style casts and reinterpret_casts where an implicit upcast suffices) from Instructions.h; move CleanupReturnInst down in the file to accommodate this
• Make sure switch enums are covered xor have defaults, per http://llvm.org/docs/CodingStandards.html#don-t-use-default-labels-in-fully-covered-switches-over-enumerations

[JosephTremoulet]

include/llvm/IR/Instructions.h

3713

I dug deeper into this. It turned out that the case MSVC was rejecting was in CleanupReturnInst::setCleanupPad, which was in fact quite reasonable since at that point CleanupPadInst was just forward-referenced, not defined, so of course it couldn't have known how to statically cast it, let alone realize that it could do so implicitly. Obvious in hindsight :/ I moved CleanupReturnInst later in the file and removed the cast.

I also took a closer look at the similar casts in the file. I was incorrect when I said "the rest of this file (outside the new EH instructions) all the setters that take BasicBlock * use reinterpret_cast<Value *>, " -- what was actually the case was that switch and invoke used reinterpret_cast, and br and indirectbr used c-style casts. I looked back through the history of InvokeInst, and it actually has bounced back and forth between c-style and reinterpret casts, but the commit messages don't discuss it. At any rate, either kind of cast appears unnecessary now (Instructions.h #includes Function.h which #includes BasicBlock.h), so I've removed those as well.

[majnemer]

LGTM, thanks so much for working on this!

docs/ExceptionHandling.rst

 frontend encounters a call site that may throw an exception, it should emit an
 invoke that unwinds to a ``catchpad`` block. Similarly, inside the scope of a
 C++ object with a destructor, invokes should unwind to a ``cleanuppad``. The
 ``terminatepad`` instruction exists to represent ``noexcept`` and throw
 specifications with one combined instruction. All potentially throwing calls in
 a ``noexcept`` function should transitively unwind to a terminateblock. Throw
 specifications are not implemented by MSVC, and are not yet supported.
 
-Each of these new EH pad instructions has a label operand that indicates which
+Each of these new EH pad instructions has a way to identify which
 action should be considered after this action. The ``catchpad`` and
-``terminatepad`` instructions are terminators, and this label is considered to
-be an unwind destination analogous to the unwind destination of an invoke. The
+``terminatepad`` instructions are terminators, and have a label operand considered
+to be an unwind destination analogous to the unwind destination of an invoke. The
 ``cleanuppad`` instruction is different from the other two in that it is not a
-terminator, and this label operand is not an edge in the CFG. The code inside a
-cleanuppad runs before transferring control to the next action, so the
-``cleanupret`` instruction is the instruction that unwinds to the next EH pad.
-All of these "unwind edges" may refer to a basic block that contains an EH pad
-instruction, or they may simply unwind to the caller. Unwinding to the caller
-has roughly the same semantics as the ``resume`` instruction in the
-``landingpad`` model. When inlining through an invoke, instructions that unwind
-to the caller are hooked up to unwind to the unwind destination of the call
-site.
+terminator. The code inside a cleanuppad runs before transferring control to the
+next action, so the ``cleanupret`` instruction is the instruction that holds a
+label operand and unwinds to the next EH pad. All of these "unwind edges" may
+refer to a basic block that contains an EH pad instruction, or they may simply
+unwind to the caller. Unwinding to the caller has roughly the same semantics as
+the ``resume`` instruction in the ``landingpad`` model. When inlining through an
+invoke, instructions that unwind to the caller are hooked up to unwind to the
+unwind destination of the call site.
 
 Putting things together, here is a hypothetical lowering of some C++ that uses
 all of the new IR instructions:
 
 .. code-block:: c
 
   struct Cleanup {
     Cleanup();
.. code-block:: llvm
 
   return:                                           ; preds = %invoke.cont.2, %catch
     %retval.0 = phi i32 [ 0, %invoke.cont.2 ], [ %9, %catch ]
     ret i32 %retval.0
 
   ; EH scope code, ordered innermost to outermost:
 
   lpad.cleanup:                                     ; preds = %invoke.cont
-    cleanuppad [label %lpad.catch]
+    %cleanup = cleanuppad []
     call void @"\01??_DCleanup@@QEAA@XZ"(%struct.Cleanup* nonnull %obj) nounwind
-    cleanupret unwind label %lpad.catch
+    cleanupret %cleanup unwind label %lpad.catch
 
   lpad.catch:                                       ; preds = %entry, %lpad.cleanup
-    catchpad void [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i32* %e]
+    %catch = catchpad [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i32* %e]
             to label %catch unwind label %lpad.terminate
 
   catch:                                            ; preds = %lpad.catch
     %9 = load i32, i32* %e, align 4
-    catchret label %return
+    catchret %catch label %return
 
   lpad.terminate:
     terminatepad [void ()* @"\01?terminate@@YAXXZ"]
             unwind to caller
   }

docs/LangRef.rst

 '``catchpad``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <resultval> = catchpad <resultty> [<args>*]
+      <resultval> = catchpad [<args>*]
           to label <normal label> unwind label <exception label>
 
 Overview:
 """""""""
 
 The '``catchpad``' instruction is used by `LLVM's exception handling
 system <ExceptionHandling.html#overview>`_ to specify that a basic block
 is a catch block --- one where a personality routine attempts to transfer
 control to catch an exception.
 The ``args`` correspond to whatever information the personality
 routine requires to know if this is an appropriate place to catch the
 exception. Control is tranfered to the ``exception`` label if the
 ``catchpad`` is not an appropriate handler for the in-flight exception.
 The ``normal`` label should contain the code found in the ``catch``
-portion of a ``try``/``catch`` sequence. It defines values supplied by
-the :ref:`personality function <personalityfn>` upon re-entry to the
-function. The ``resultval`` has the type ``resultty``.
+portion of a ``try``/``catch`` sequence. The ``resultval`` has the type
+:ref:`token <t_token>` and is used to match the ``catchpad`` to
+corresponding :ref:`catchrets <i_catchret>`.
 
 Arguments:
 """"""""""
 
 The instruction takes a list of arbitrary values which are interpreted
 by the :ref:`personality function <personalityfn>`.
 
 The ``catchpad`` must be provided a ``normal`` label to transfer control
 to if the ``catchpad`` matches the exception and an ``exception``
 label to transfer control to if it doesn't.
 
 Semantics:
 """"""""""
 
-The '``catchpad``' instruction defines the values which are set by the
-:ref:`personality function <personalityfn>` upon re-entry to the function, and
-therefore the "result type" of the ``catchpad`` instruction. As with
-calling conventions, how the personality function results are
-represented in LLVM IR is target specific.
-
 When the call stack is being unwound due to an exception being thrown,
 the exception is compared against the ``args``. If it doesn't match,
 then control is transfered to the ``exception`` basic block.
+As with calling conventions, how the personality function results are
+represented in LLVM IR is target specific.
 
 The ``catchpad`` instruction has several restrictions:
 
 -  A catch block is a basic block which is the unwind destination of
    an exceptional instruction.
 -  A catch block must have a '``catchpad``' instruction as its
    first non-PHI instruction.
 -  A catch block's ``exception`` edge must refer to a catch block or a
    catch-end block.
 -  There can be only one '``catchpad``' instruction within the
    catch block.
 -  A basic block that is not a catch block may not include a
    '``catchpad``' instruction.
+-  A catch block which has another catch block as a predecessor may not have
+   any other predecessors.
 -  It is undefined behavior for control to transfer from a ``catchpad`` to a
-   ``cleanupret`` without first executing a ``catchret`` and a subsequent
-   ``cleanuppad``.
--  It is undefined behavior for control to transfer from a ``catchpad`` to a
-   ``ret`` without first executing a ``catchret``.
+   ``ret`` without first executing a ``catchret`` that consumes the
+   ``catchpad`` or unwinding through its ``catchendpad``.
+-  It is undefined behavior for control to transfer from a ``catchpad`` to
+   itself without first executing a ``catchret`` that consumes the
+   ``catchpad`` or unwinding through its ``catchendpad``.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       ;; A catch block which can catch an integer.
-      %res = catchpad { i8*, i32 } [i8** @_ZTIi]
+      %tok = catchpad [i8** @_ZTIi]
         to label %int.handler unwind label %terminate
 
 .. _i_catchendpad:
 
 '``catchendpad``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:

 -  A catch-end block must have a '``catchendpad``' instruction as its
    first non-PHI instruction.
 -  There can be only one '``catchendpad``' instruction within the
    catch block.
 -  A basic block that is not a catch-end block may not include a
    '``catchendpad``' instruction.
 -  Exactly one catch block may unwind to a ``catchendpad``.
 -  The unwind target of invokes between a ``catchpad`` and a
-   corresponding ``catchret`` must be its ``catchendpad``.
+   corresponding ``catchret`` must be its ``catchendpad`` or
+   an inner EH pad.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       catchendpad unwind label %terminate
       catchendpad unwind to caller
 
 .. _i_catchret:
 
 '``catchret``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      catchret <type> <value> to label <normal>
+      catchret <value> to label <normal>
 
 Overview:
 """""""""
 
 The '``catchret``' instruction is a terminator instruction that has a
 single successor.
 
 
 Arguments:
 """"""""""
 
-The '``catchret``' instruction requires one argument which specifies
-where control will transfer to next.
+The first argument to a '``catchret``' indicates which ``catchpad`` it
+exits.  It must be a :ref:`catchpad <i_catchpad>`.
+The second argument to a '``catchret``' specifies where control will
+transfer to next.
 
 Semantics:
 """"""""""
 
 The '``catchret``' instruction ends the existing (in-flight) exception
 whose unwinding was interrupted with a
 :ref:`catchpad <i_catchpad>` instruction.
 The :ref:`personality function <personalityfn>` gets a chance to execute
 arbitrary code to, for example, run a C++ destructor.
 Control then transfers to ``normal``.
 It may be passed an optional, personality specific, value.
+It is undefined behavior to execute a ``catchret`` whose ``catchpad`` has
+not been executed.
+It is undefined behavior to execute a ``catchret`` if any ``catchpad`` or
+``cleanuppad`` has been executed, without subsequently executing a
+corresponding ``catchret``/``cleanupret`` or unwinding out of the inner
+pad, following the most recent execution of the ``catchret``'s corresponding
+``catchpad``.
+
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
-      catchret label %continue
+      catchret %catch label %continue
 
 .. _i_cleanupret:
 
 '``cleanupret``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      cleanupret <type> <value> unwind label <continue>
-      cleanupret <type> <value> unwind to caller
+      cleanupret <value> unwind label <continue>
+      cleanupret <value> unwind to caller
 
 Overview:
 """""""""
 
 The '``cleanupret``' instruction is a terminator instruction that has
 an optional successor.
 
 
 Arguments:
 """"""""""
 
-The '``cleanupret``' instruction requires one argument, which must have the
-same type as the result of any '``cleanuppad``' instruction in the same
-function. It also has an optional successor,  ``continue``.
+The '``cleanupret``' instruction requires one argument, which indicates
+which ``cleanuppad`` it exits, and must be a :ref:`cleanuppad <i_cleanuppad>`.
+It also has an optional successor, ``continue``.
 
 Semantics:
 """"""""""
 
 The '``cleanupret``' instruction indicates to the
 :ref:`personality function <personalityfn>` that one
 :ref:`cleanuppad <i_cleanuppad>` it transferred control to has ended.
 It transfers control to ``continue`` or unwinds out of the function.
+It is undefined behavior to execute a ``cleanupret`` whose ``cleanuppad`` has
+not been executed.
+It is undefined behavior to execute a ``cleanupret`` if any ``catchpad`` or
+``cleanuppad`` has been executed, without subsequently executing a
+corresponding ``catchret``/``cleanupret`` or unwinding out of the inner pad,
+following the most recent execution of the ``cleanupret``'s corresponding
+``cleanuppad``.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
-      cleanupret void unwind to caller
-      cleanupret { i8*, i32 } %exn unwind label %continue
+      cleanupret %cleanup unwind to caller
+      cleanupret %cleanup unwind label %continue
 
 .. _i_terminatepad:
 
 '``terminatepad``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""

 '``cleanuppad``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <resultval> = cleanuppad <resultty> [<args>*]
+      <resultval> = cleanuppad [<args>*]
 
 Overview:
 """""""""
 
 The '``cleanuppad``' instruction is used by `LLVM's exception handling
 system <ExceptionHandling.html#overview>`_ to specify that a basic block
 is a cleanup block --- one where a personality routine attempts to
 transfer control to run cleanup actions.
 The ``args`` correspond to whatever additional
 information the :ref:`personality function <personalityfn>` requires to
 execute the cleanup.
-The ``resultval`` has the type ``resultty``.
+The ``resultval`` has the type :ref:`token <t_token>` and is used to
+match the ``cleanuppad`` to corresponding :ref:`cleanuprets <i_cleanupret>`.
 
 Arguments:
 """"""""""
 
 The instruction takes a list of arbitrary values which are interpreted
 by the :ref:`personality function <personalityfn>`.
 
 Semantics:
 """"""""""
 
 The '``cleanuppad``' instruction defines the values which are set by the
-:ref:`personality function <personalityfn>` upon re-entry to the function, and
-therefore the "result type" of the ``cleanuppad`` instruction. As with
-calling conventions, how the personality function results are
+:ref:`personality function <personalityfn>` upon re-entry to the function.
+As with calling conventions, how the personality function results are
 represented in LLVM IR is target specific.
 
 When the call stack is being unwound due to an exception being thrown,
 the :ref:`personality function <personalityfn>` transfers control to the
 ``cleanuppad`` with the aid of the personality-specific arguments.
 
 The ``cleanuppad`` instruction has several restrictions:
 
 -  A cleanup block is a basic block which is the unwind destination of
    an exceptional instruction.
 -  A cleanup block must have a '``cleanuppad``' instruction as its
    first non-PHI instruction.
 -  There can be only one '``cleanuppad``' instruction within the
    cleanup block.
 -  A basic block that is not a cleanup block may not include a
    '``cleanuppad``' instruction.
+-  All ``cleanupret``s which exit a cleanuppad must have the same
+   exceptional successor.
 -  It is undefined behavior for control to transfer from a ``cleanuppad`` to a
-   ``catchret`` without first executing a ``cleanupret`` and a subsequent
-   ``catchpad``.
--  It is undefined behavior for control to transfer from a ``cleanuppad`` to a
-   ``ret`` without first executing a ``cleanupret``.
+   ``ret`` without first executing a ``cleanupret`` that consumes the
+   ``cleanuppad`` or unwinding out of the ``cleanuppad``.
+-  It is undefined behavior for control to transfer from a ``cleanuppad`` to
+   itself without first executing a ``cleanupret`` that consumes the
+   ``cleanuppad`` or unwinding out of the ``cleanuppad``.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
-      %res = cleanuppad { i8*, i32 } [label %nextaction]
+      %tok = cleanuppad []
 
 .. _intrinsics:
 
 Intrinsic Functions
 ===================
 
 LLVM supports the notion of an "intrinsic function". These functions
 have well known names and semantics and are required to follow certain

include/llvm/Bitcode/LLVMBitCodes.h

enum FunctionCodes {
     FUNC_CODE_INST_STOREATOMIC_OLD = 42, // STORE: [ptrty,ptr,val, align, vol
                                      //         ordering, synchscope]
     FUNC_CODE_INST_GEP         = 43, // GEP:  [inbounds, n x operands]
     FUNC_CODE_INST_STORE       = 44, // STORE: [ptrty,ptr,valty,val, align, vol]
     FUNC_CODE_INST_STOREATOMIC = 45, // STORE: [ptrty,ptr,val, align, vol
     FUNC_CODE_INST_CMPXCHG     = 46, // CMPXCHG: [ptrty,ptr,valty,cmp,new, align,
                                      //           vol,ordering,synchscope]
     FUNC_CODE_INST_LANDINGPAD  = 47, // LANDINGPAD: [ty,val,num,id0,val0...]
-    FUNC_CODE_INST_CLEANUPRET  = 48, // CLEANUPRET: [] or [val] or [bb#] or [val,bb#]
-    FUNC_CODE_INST_CATCHRET    = 49, // CATCHRET: [bb#]
-    FUNC_CODE_INST_CATCHPAD  = 50, // CATCHPAD: [ty,val,val,num,args...]
+    FUNC_CODE_INST_CLEANUPRET  = 48, // CLEANUPRET: [val] or [val,bb#]
+    FUNC_CODE_INST_CATCHRET    = 49, // CATCHRET: [val,bb#]
+    FUNC_CODE_INST_CATCHPAD  = 50, // CATCHPAD: [bb#,bb#,num,args...]
     FUNC_CODE_INST_TERMINATEPAD = 51, // TERMINATEPAD: [bb#,num,args...]
     FUNC_CODE_INST_CLEANUPPAD = 52, // CLEANUPPAD: [num,args...]
     FUNC_CODE_INST_CATCHENDPAD = 53, // CATCHENDPAD: [] or [bb#]
   };
 
   enum UseListCodes {
     USELIST_CODE_DEFAULT = 1, // DEFAULT: [index..., value-id]
     USELIST_CODE_BB      = 2  // BB: [index..., bb-id]

include/llvm/IR/IRBuilder.h

InvokeInst *CreateInvoke(Value *Callee, BasicBlock *NormalDest,
     return Insert(InvokeInst::Create(Callee, NormalDest, UnwindDest, Args),
                   Name);
   }
 
   ResumeInst *CreateResume(Value *Exn) {
     return Insert(ResumeInst::Create(Exn));
   }
 
-  CleanupReturnInst *CreateCleanupRet(BasicBlock *UnwindBB = nullptr,
-                                      Value *RetVal = nullptr) {
-    return Insert(CleanupReturnInst::Create(Context, RetVal, UnwindBB));
+  CleanupReturnInst *CreateCleanupRet(CleanupPadInst *CleanupPad,
+                                      BasicBlock *UnwindBB = nullptr) {
+    return Insert(CleanupReturnInst::Create(CleanupPad, UnwindBB));
   }
 
-  CatchPadInst *CreateCatchPad(Type *Ty, BasicBlock *NormalDest,
-                               BasicBlock *UnwindDest, ArrayRef<Value *> Args,
-                               const Twine &Name = "") {
-    return Insert(CatchPadInst::Create(Ty, NormalDest, UnwindDest, Args), Name);
+  CatchPadInst *CreateCatchPad(BasicBlock *NormalDest, BasicBlock *UnwindDest,
+                               ArrayRef<Value *> Args, const Twine &Name = "") {
+    return Insert(CatchPadInst::Create(NormalDest, UnwindDest, Args), Name);
   }
 
   CatchEndPadInst *CreateCatchEndPad(BasicBlock *UnwindBB = nullptr) {
     return Insert(CatchEndPadInst::Create(Context, UnwindBB));
   }
 
   TerminatePadInst *CreateTerminatePad(BasicBlock *UnwindBB = nullptr,
                                        ArrayRef<Value *> Args = {},
                                        const Twine &Name = "") {
     return Insert(TerminatePadInst::Create(Context, UnwindBB, Args), Name);
   }
 
-  CleanupPadInst *CreateCleanupPad(Type *Ty, ArrayRef<Value *> Args,
+  CleanupPadInst *CreateCleanupPad(ArrayRef<Value *> Args,
                                    const Twine &Name = "") {
-    return Insert(CleanupPadInst::Create(Ty, Args), Name);
+    return Insert(CleanupPadInst::Create(Context, Args), Name);
   }
 
-  CatchReturnInst *CreateCatchRet(BasicBlock *BB, Value *RetVal = nullptr) {
-    return Insert(CatchReturnInst::Create(BB, RetVal));
+  CatchReturnInst *CreateCatchRet(CatchPadInst *CatchPad, BasicBlock *BB) {
+    return Insert(CatchReturnInst::Create(CatchPad, BB));
   }
 
   UnreachableInst *CreateUnreachable() {
     return Insert(new UnreachableInst(Context));
   }
 
   //===--------------------------------------------------------------------===//
   // Instruction creation methods: Binary Operators

include/llvm/IR/Instructions.h

 
 namespace llvm {
 
 class APInt;
 class ConstantInt;
 class ConstantRange;
 class DataLayout;
 class LLVMContext;
+class CleanupPadInst;
 
 enum AtomicOrdering {
   NotAtomic = 0,
   Unordered = 1,
   Monotonic = 2,
   // Consume = 3,  // Not specified yet.
   Acquire = 4,
   Release = 5,

 //===----------------------------------------------------------------------===//
 //                               CleanupReturnInst Class
 //===----------------------------------------------------------------------===//
 
 class CleanupReturnInst : public TerminatorInst {
   CleanupReturnInst(const CleanupReturnInst &RI);
 
 private:
-  void init(Value *RetVal, BasicBlock *UnwindBB);
-  CleanupReturnInst(LLVMContext &C, Value *RetVal, BasicBlock *UnwindBB,
+  void init(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB);
+  CleanupReturnInst(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB,
                     unsigned Values, Instruction *InsertBefore = nullptr);
-  CleanupReturnInst(LLVMContext &C, Value *RetVal, BasicBlock *UnwindBB,
+  CleanupReturnInst(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB,
                     unsigned Values, BasicBlock *InsertAtEnd);
 
   int getUnwindLabelOpIdx() const {
     assert(hasUnwindDest());
     return 0;
   }
 
-  int getRetValOpIdx() const {
-    assert(hasReturnValue());
-    if (hasUnwindDest())
-      return 1;
-    return 0;
-  }
-
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   CleanupReturnInst *cloneImpl() const;
 
 public:
-  static CleanupReturnInst *Create(LLVMContext &C,
-                                   Value *RetVal = nullptr,
+  static CleanupReturnInst *Create(CleanupPadInst *CleanupPad,
                                    BasicBlock *UnwindBB = nullptr,
                                    Instruction *InsertBefore = nullptr) {
-    unsigned Values = 0;
-    if (RetVal)
-      ++Values;
+    unsigned Values = 1;
     if (UnwindBB)
       ++Values;
     return new (Values)
-        CleanupReturnInst(C, RetVal, UnwindBB, Values, InsertBefore);
+        CleanupReturnInst(CleanupPad, UnwindBB, Values, InsertBefore);
   }
-  static CleanupReturnInst *Create(LLVMContext &C, Value *RetVal,
+  static CleanupReturnInst *Create(CleanupPadInst *CleanupPad,
                                    BasicBlock *UnwindBB,
                                    BasicBlock *InsertAtEnd) {
-    unsigned Values = 0;
-    if (RetVal)
-      ++Values;
+    unsigned Values = 1;
     if (UnwindBB)
       ++Values;
     return new (Values)
-        CleanupReturnInst(C, RetVal, UnwindBB, Values, InsertAtEnd);
+        CleanupReturnInst(CleanupPad, UnwindBB, Values, InsertAtEnd);
   }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   bool hasUnwindDest() const { return getSubclassDataFromInstruction() & 1; }
   bool unwindsToCaller() const { return !hasUnwindDest(); }
-  bool hasReturnValue() const { return getSubclassDataFromInstruction() & 2; }
 
-  /// Convenience accessor. Returns null if there is no return value.
-  Value *getReturnValue() const {
-    if (!hasReturnValue())
-      return nullptr;
-    return getOperand(getRetValOpIdx());
+  /// Convenience accessor.
+  CleanupPadInst *getCleanupPad() const {
+    return cast<CleanupPadInst>(Op<-1>());
   }
-  void setReturnValue(Value *RetVal) {
-    assert(hasReturnValue());
-    setOperand(getRetValOpIdx(), RetVal);
+  void setCleanupPad(CleanupPadInst *CleanupPad) {
+    Op<-1>() = reinterpret_cast<Value *>(CleanupPad);
   }
 
   unsigned getNumSuccessors() const { return hasUnwindDest() ? 1 : 0; }
 
-  BasicBlock *getUnwindDest() const;
-  void setUnwindDest(BasicBlock *NewDest);
+  BasicBlock *getUnwindDest() const {
+    return hasUnwindDest() ? cast<BasicBlock>(Op<-2>()) : nullptr;
+  }
+  void setUnwindDest(BasicBlock *NewDest) {
+    assert(NewDest);
+    assert(hasUnwindDest());
+    Op<-2>() = reinterpret_cast<Value *>(NewDest);
+  }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static inline bool classof(const Instruction *I) {
     return (I->getOpcode() == Instruction::CleanupRet);
   }
   static inline bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 
 private:
   BasicBlock *getSuccessorV(unsigned Idx) const override;
   unsigned getNumSuccessorsV() const override;
   void setSuccessorV(unsigned Idx, BasicBlock *B) override;
 
   // Shadow Instruction::setInstructionSubclassData with a private forwarding
   // method so that subclasses cannot accidentally use it.
   void setInstructionSubclassData(unsigned short D) {
     Instruction::setInstructionSubclassData(D);
   }
 };
 
 template <>
 struct OperandTraits<CleanupReturnInst>
-    : public VariadicOperandTraits<CleanupReturnInst> {};
+    : public VariadicOperandTraits<CleanupReturnInst, /*MINARITY=*/1> {};
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CleanupReturnInst, Value)
 
 //===----------------------------------------------------------------------===//
 //                               CatchEndPadInst Class
 //===----------------------------------------------------------------------===//
 
 class CatchEndPadInst : public TerminatorInst {
public:
   /// Convenience accessor. Returns null if there is no return value.
   unsigned getNumSuccessors() const { return hasUnwindDest() ? 1 : 0; }
 
   BasicBlock *getUnwindDest() const {
     return hasUnwindDest() ? cast<BasicBlock>(Op<-1>()) : nullptr;
   }
   void setUnwindDest(BasicBlock *NewDest) {
     assert(NewDest);
-    Op<-1>() = NewDest;
+    Op<-1>() = reinterpret_cast<Value *>(NewDest);
   }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static inline bool classof(const Instruction *I) {
     return (I->getOpcode() == Instruction::CatchEndPad);
   }
   static inline bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));

 
 class CatchPadInst : public TerminatorInst {
 private:
   void init(BasicBlock *IfNormal, BasicBlock *IfException,
             ArrayRef<Value *> Args, const Twine &NameStr);
 
   CatchPadInst(const CatchPadInst &CPI);
 
-  explicit CatchPadInst(Type *RetTy, BasicBlock *IfNormal,
-                        BasicBlock *IfException, ArrayRef<Value *> Args,
-                        unsigned Values, const Twine &NameStr,
-                        Instruction *InsertBefore);
-  explicit CatchPadInst(Type *RetTy, BasicBlock *IfNormal,
-                        BasicBlock *IfException, ArrayRef<Value *> Args,
-                        unsigned Values, const Twine &NameStr,
-                        BasicBlock *InsertAtEnd);
+  explicit CatchPadInst(BasicBlock *IfNormal, BasicBlock *IfException,
+                        ArrayRef<Value *> Args, unsigned Values,
+                        const Twine &NameStr, Instruction *InsertBefore);
+  explicit CatchPadInst(BasicBlock *IfNormal, BasicBlock *IfException,
+                        ArrayRef<Value *> Args, unsigned Values,
+                        const Twine &NameStr, BasicBlock *InsertAtEnd);
 
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   CatchPadInst *cloneImpl() const;
 
 public:
-  static CatchPadInst *Create(Type *RetTy, BasicBlock *IfNormal,
-                              BasicBlock *IfException, ArrayRef<Value *> Args,
-                              const Twine &NameStr = "",
+  static CatchPadInst *Create(BasicBlock *IfNormal, BasicBlock *IfException,
+                              ArrayRef<Value *> Args, const Twine &NameStr = "",
                               Instruction *InsertBefore = nullptr) {
     unsigned Values = unsigned(Args.size()) + 2;
-    return new (Values) CatchPadInst(RetTy, IfNormal, IfException, Args, Values,
+    return new (Values) CatchPadInst(IfNormal, IfException, Args, Values,
                                      NameStr, InsertBefore);
   }
-  static CatchPadInst *Create(Type *RetTy, BasicBlock *IfNormal,
-                              BasicBlock *IfException, ArrayRef<Value *> Args,
-                              const Twine &NameStr, BasicBlock *InsertAtEnd) {
+  static CatchPadInst *Create(BasicBlock *IfNormal, BasicBlock *IfException,
+                              ArrayRef<Value *> Args, const Twine &NameStr,
+                              BasicBlock *InsertAtEnd) {
     unsigned Values = unsigned(Args.size()) + 2;
-    return new (Values) CatchPadInst(RetTy, IfNormal, IfException, Args, Values,
-                                     NameStr, InsertAtEnd);
+    return new (Values)
+        CatchPadInst(IfNormal, IfException, Args, Values, NameStr, InsertAtEnd);
   }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   /// getNumArgOperands - Return the number of catchpad arguments.
   ///
   unsigned getNumArgOperands() const { return getNumOperands() - 2; }

 //===----------------------------------------------------------------------===//
 
 class CleanupPadInst : public Instruction {
 private:
   void init(ArrayRef<Value *> Args, const Twine &NameStr);
 
   CleanupPadInst(const CleanupPadInst &CPI);
 
-  explicit CleanupPadInst(Type *RetTy, ArrayRef<Value *> Args,
+  explicit CleanupPadInst(LLVMContext &C, ArrayRef<Value *> Args,
                           const Twine &NameStr, Instruction *InsertBefore);
-  explicit CleanupPadInst(Type *RetTy, ArrayRef<Value *> Args,
+  explicit CleanupPadInst(LLVMContext &C, ArrayRef<Value *> Args,
                           const Twine &NameStr, BasicBlock *InsertAtEnd);
 
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   CleanupPadInst *cloneImpl() const;
 
 public:
-  static CleanupPadInst *Create(Type *RetTy, ArrayRef<Value *> Args,
+  static CleanupPadInst *Create(LLVMContext &C, ArrayRef<Value *> Args,
                                 const Twine &NameStr = "",
                                 Instruction *InsertBefore = nullptr) {
-    return new (Args.size()) CleanupPadInst(RetTy, Args, NameStr, InsertBefore);
+    return new (Args.size()) CleanupPadInst(C, Args, NameStr, InsertBefore);
   }
-  static CleanupPadInst *Create(Type *RetTy, ArrayRef<Value *> Args,
+  static CleanupPadInst *Create(LLVMContext &C, ArrayRef<Value *> Args,
                                 const Twine &NameStr, BasicBlock *InsertAtEnd) {
-    return new (Args.size()) CleanupPadInst(RetTy, Args, NameStr, InsertAtEnd);
+    return new (Args.size()) CleanupPadInst(C, Args, NameStr, InsertAtEnd);
   }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static inline bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::CleanupPad;

 //===----------------------------------------------------------------------===//
 //                               CatchReturnInst Class
 //===----------------------------------------------------------------------===//
 
 class CatchReturnInst : public TerminatorInst {
   CatchReturnInst(const CatchReturnInst &RI);
 
 private:
-  void init(BasicBlock *BB, Value *RetVal);
-  CatchReturnInst(BasicBlock *BB, Value *RetVal, unsigned Values,
+  void init(CatchPadInst *CatchPad, BasicBlock *BB);
+  CatchReturnInst(CatchPadInst *CatchPad, BasicBlock *BB,
                   Instruction *InsertBefore = nullptr);
-  CatchReturnInst(BasicBlock *BB, Value *RetVal, unsigned Values,
+  CatchReturnInst(CatchPadInst *CatchPad, BasicBlock *BB,
                   BasicBlock *InsertAtEnd);
 
 protected:
   // Note: Instruction needs to be a friend here to call cloneImpl.
   friend class Instruction;
   CatchReturnInst *cloneImpl() const;
 
 public:
-  static CatchReturnInst *Create(BasicBlock *BB, Value *RetVal = nullptr,
+  static CatchReturnInst *Create(CatchPadInst *CatchPad, BasicBlock *BB,
                                  Instruction *InsertBefore = nullptr) {
     assert(BB);
-    unsigned Values = 1;
-    if (RetVal)
-      ++Values;
-    return new (Values) CatchReturnInst(BB, RetVal, Values, InsertBefore);
+    return new (2) CatchReturnInst(CatchPad, BB, InsertBefore);
   }
-  static CatchReturnInst *Create(BasicBlock *BB, Value *RetVal,
+  static CatchReturnInst *Create(CatchPadInst *CatchPad, BasicBlock *BB,
                                  BasicBlock *InsertAtEnd) {
     assert(BB);
-    unsigned Values = 1;
-    if (RetVal)
-      ++Values;
-    return new (Values) CatchReturnInst(BB, RetVal, Values, InsertAtEnd);
+    return new (2) CatchReturnInst(CatchPad, BB, InsertAtEnd);
   }
 
   /// Provide fast operand accessors
   DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
 
   /// Convenience accessors.
-  BasicBlock *getSuccessor() const { return cast<BasicBlock>(Op<-1>()); }
-  void setSuccessor(BasicBlock *NewSucc) { Op<-1>() = (Value *)NewSucc; }
-  unsigned getNumSuccessors() const { return 1; }
+  CatchPadInst *getCatchPad() const { return cast<CatchPadInst>(Op<0>()); }
+  void setCatchPad(CatchPadInst *CatchPad) {
+    Op<0>() = reinterpret_cast<Value *>(CatchPad);
+  }
 
-  bool hasReturnValue() const { return getNumOperands() > 1; }
-  Value *getReturnValue() const { return Op<-2>(); }
-  void setReturnValue(Value *RetVal) { Op<-2>() = RetVal; }
+  BasicBlock *getSuccessor() const { return cast<BasicBlock>(Op<1>()); }
+  void setSuccessor(BasicBlock *NewSucc) {
+    Op<1>() = reinterpret_cast<Value *>(NewSucc);
+  }
+  unsigned getNumSuccessors() const { return 1; }
 
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static inline bool classof(const Instruction *I) {
     return (I->getOpcode() == Instruction::CatchRet);
   }
   static inline bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 
 private:
   BasicBlock *getSuccessorV(unsigned Idx) const override;
   unsigned getNumSuccessorsV() const override;
   void setSuccessorV(unsigned Idx, BasicBlock *B) override;
 };
 
 template <>
 struct OperandTraits<CatchReturnInst>
-    : public VariadicOperandTraits<CatchReturnInst> {};
+    : public FixedNumOperandTraits<CatchReturnInst, 2> {};
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CatchReturnInst, Value)
 
 //===----------------------------------------------------------------------===//
 //                           UnreachableInst Class
 //===----------------------------------------------------------------------===//
 
 //===---------------------------------------------------------------------------

lib/Analysis/InlineCost.cpp

 bool CallAnalyzer::visitCleanupReturnInst(CleanupReturnInst &CRI) {
   // FIXME: It's not clear that a single instruction is an accurate model for
   // the inline cost of a cleanupret instruction.
   return false;
 }
 
 bool CallAnalyzer::visitCatchReturnInst(CatchReturnInst &CRI) {
   // FIXME: It's not clear that a single instruction is an accurate model for
-  // the inline cost of a cleanupret instruction.
+  // the inline cost of a catchret instruction.
   return false;
 }
 
 bool CallAnalyzer::visitUnreachableInst(UnreachableInst &I) {
   // FIXME: It might be reasonably to discount the cost of instructions leading
   // to unreachable as they have the lowest possible impact on both runtime and
   // code size.
   return true; // No actual code is needed for unreachable.

lib/AsmParser/LLParser.h

private:
     // lists. This means that RAUW doesn't work, even on temporary MDNodes
     // which otherwise support RAUW. Instead, we defer resolving MDNode
     // references until the definitions have been processed.
     struct MDRef {
       SMLoc Loc;
       unsigned MDKind, MDSlot;
     };
 
+    /// Indicates which operator an operand allows (for the few operands that
+    /// may only reference a certain operator).
+    enum OperatorConstraint {
+      none = 0,  // No constraint
+      catchpad,  // Must be CatchPadInst
+      cleanuppad // Must be CleanupPadInst
+    };
+
     SmallVector<Instruction*, 64> InstsWithTBAATag;
 
     // Type resolution handling data structures.  The location is set when we
     // have processed a use of the type but not a definition yet.
     StringMap<std::pair<Type*, LocTy> > NamedTypes;
     std::map<unsigned, std::pair<Type*, LocTy> > NumberedTypes;
 
     std::map<unsigned, TrackingMDNodeRef> NumberedMetadata;
public:
 
       Function &getFunction() const { return F; }
 
       bool FinishFunction();
 
       /// GetVal - Get a value with the specified name or ID, creating a
       /// forward reference record if needed.  This can return null if the value
       /// exists but does not have the right type.
-      Value *GetVal(const std::string &Name, Type *Ty, LocTy Loc);
-      Value *GetVal(unsigned ID, Type *Ty, LocTy Loc);
+      Value *GetVal(const std::string &Name, Type *Ty, LocTy Loc,
+                    OperatorConstraint OC = OperatorConstraint::none);
+      Value *GetVal(unsigned ID, Type *Ty, LocTy Loc,
+                    OperatorConstraint OC = OperatorConstraint::none);
 
       /// SetInstName - After an instruction is parsed and inserted into its
       /// basic block, this installs its name.
       bool SetInstName(int NameID, const std::string &NameStr, LocTy NameLoc,
                        Instruction *Inst);
 
       /// GetBB - Get a basic block with the specified name or ID, creating a
       /// forward reference record if needed.  This can return null if the value
       /// is not a BasicBlock.
       BasicBlock *GetBB(const std::string &Name, LocTy Loc);
       BasicBlock *GetBB(unsigned ID, LocTy Loc);
 
       /// DefineBB - Define the specified basic block, which is either named or
       /// unnamed.  If there is an error, this returns null otherwise it returns
       /// the block being defined.
       BasicBlock *DefineBB(const std::string &Name, LocTy Loc);
 
       bool resolveForwardRefBlockAddresses();
     };
 
     bool ConvertValIDToValue(Type *Ty, ValID &ID, Value *&V,
-                             PerFunctionState *PFS);
+                             PerFunctionState *PFS,
+                             OperatorConstraint OC = OperatorConstraint::none);
 
     bool parseConstantValue(Type *Ty, Constant *&C);
-    bool ParseValue(Type *Ty, Value *&V, PerFunctionState *PFS);
-    bool ParseValue(Type *Ty, Value *&V, PerFunctionState &PFS) {
-      return ParseValue(Ty, V, &PFS);
+    bool ParseValue(Type *Ty, Value *&V, PerFunctionState *PFS,
+                    OperatorConstraint OC = OperatorConstraint::none);
+    bool ParseValue(Type *Ty, Value *&V, PerFunctionState &PFS,
+                    OperatorConstraint OC = OperatorConstraint::none) {
+      return ParseValue(Ty, V, &PFS, OC);
     }
     bool ParseValue(Type *Ty, Value *&V, LocTy &Loc,
                     PerFunctionState &PFS) {
       Loc = Lex.getLoc();
       return ParseValue(Ty, V, &PFS);
     }
 
     bool ParseTypeAndValue(Value *&V, PerFunctionState *PFS);

lib/AsmParser/LLParser.cpp

return P.Error(ForwardRefValIDs.begin()->second.second,
                    Twine(ForwardRefValIDs.begin()->first) + "'");
   return false;
 }
 
 
 /// GetVal - Get a value with the specified name or ID, creating a
 /// forward reference record if needed.  This can return null if the value
 /// exists but does not have the right type.
-Value *LLParser::PerFunctionState::GetVal(const std::string &Name,
-                                          Type *Ty, LocTy Loc) {
+Value *LLParser::PerFunctionState::GetVal(const std::string &Name, Type *Ty,
+                                          LocTy Loc, OperatorConstraint OC) {
   // Look this name up in the normal function symbol table.
   Value *Val = F.getValueSymbolTable().lookup(Name);
 
   // If this is a forward reference for the value, see if we already created a
   // forward ref record.
   if (!Val) {
     std::map<std::string, std::pair<Value*, LocTy> >::iterator
       I = ForwardRefVals.find(Name);
     if (I != ForwardRefVals.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
   if (Val) {
+    if (OC)
+      switch (OC) {
+      case OperatorConstraint::catchpad:
+        if (!isa<CatchPadInst>(Val)) {
+          P.Error(Loc, "'%" + Name + "' is not a catchpad");
+          return nullptr;
+        }
+        break;
+      case OperatorConstraint::cleanuppad:
+        if (!isa<CleanupPadInst>(Val)) {
+          P.Error(Loc, "'%" + Name + "' is not a cleanuppad");
+          return nullptr;
+        }
+        break;
+      default:
+        llvm_unreachable("unexpected constraint");
+      }
     if (Val->getType() == Ty) return Val;
     if (Ty->isLabelTy())
       P.Error(Loc, "'%" + Name + "' is not a basic block");
     else
       P.Error(Loc, "'%" + Name + "' defined with type '" +
               getTypeString(Val->getType()) + "'");
     return nullptr;
   }
 
   // Don't make placeholders with invalid type.
   if (!Ty->isFirstClassType()) {
     P.Error(Loc, "invalid use of a non-first-class type");
     return nullptr;
   }
 
   // Otherwise, create a new forward reference for this value and remember it.
   Value *FwdVal;
-  if (Ty->isLabelTy())
+  if (Ty->isLabelTy()) {
+    assert(!OC);
     FwdVal = BasicBlock::Create(F.getContext(), Name, &F);
-  else
+  } else {
+    switch (OC) {
+    case OperatorConstraint::none:
       FwdVal = new Argument(Ty, Name);
+      break;
+    case OperatorConstraint::catchpad:
+      FwdVal = CatchPadInst::Create(&F.getEntryBlock(), &F.getEntryBlock(), {},
+                                    Name);
+      break;
+    case OperatorConstraint::cleanuppad:
+      FwdVal = CleanupPadInst::Create(F.getContext(), {}, Name);
+      break;
+    default:
+      llvm_unreachable("unexpected constraint");
+    }
+  }
 
   ForwardRefVals[Name] = std::make_pair(FwdVal, Loc);
   return FwdVal;
 }
 
-Value *LLParser::PerFunctionState::GetVal(unsigned ID, Type *Ty,
-                                          LocTy Loc) {
+Value *LLParser::PerFunctionState::GetVal(unsigned ID, Type *Ty, LocTy Loc,
+                                          OperatorConstraint OC) {
   // Look this name up in the normal function symbol table.
   Value *Val = ID < NumberedVals.size() ? NumberedVals[ID] : nullptr;
 
   // If this is a forward reference for the value, see if we already created a
   // forward ref record.
   if (!Val) {
     std::map<unsigned, std::pair<Value*, LocTy> >::iterator
       I = ForwardRefValIDs.find(ID);
     if (I != ForwardRefValIDs.end())
       Val = I->second.first;
   }
 
   // If we have the value in the symbol table or fwd-ref table, return it.
   if (Val) {
+    if (OC)
+      switch (OC) {
+      case OperatorConstraint::catchpad:
+        if (!isa<CatchPadInst>(Val)) {
+          P.Error(Loc, "'%" + Twine(ID) + "' is not a catchpad");
+          return nullptr;
+        }
+        break;
+      case OperatorConstraint::cleanuppad:
+        if (!isa<CleanupPadInst>(Val)) {
+          P.Error(Loc, "'%" + Twine(ID) + "' is not a cleanuppad");
+          return nullptr;
+        }
+        break;
+      default:
+        llvm_unreachable("unexpected constraint");
+      }
     if (Val->getType() == Ty) return Val;
     if (Ty->isLabelTy())
       P.Error(Loc, "'%" + Twine(ID) + "' is not a basic block");
     else
       P.Error(Loc, "'%" + Twine(ID) + "' defined with type '" +
               getTypeString(Val->getType()) + "'");
     return nullptr;
   }
 
   if (!Ty->isFirstClassType()) {
     P.Error(Loc, "invalid use of a non-first-class type");
     return nullptr;
   }
 
   // Otherwise, create a new forward reference for this value and remember it.
   Value *FwdVal;
-  if (Ty->isLabelTy())
+  if (Ty->isLabelTy()) {
+    assert(!OC);
     FwdVal = BasicBlock::Create(F.getContext(), "", &F);
-  else
+  } else {
+    switch (OC) {
+    case OperatorConstraint::none:
       FwdVal = new Argument(Ty);
+      break;
+    case OperatorConstraint::catchpad:
+      FwdVal = CatchPadInst::Create(&F.getEntryBlock(), &F.getEntryBlock(), {});
+      break;
+    case OperatorConstraint::cleanuppad:
+      FwdVal = CleanupPadInst::Create(F.getContext(), {});
+      break;
+    default:
+      llvm_unreachable("unexpected constraint");
+    }
+  }
 
   ForwardRefValIDs[ID] = std::make_pair(FwdVal, Loc);
   return FwdVal;
 }
 
 /// SetInstName - After an instruction is parsed and inserted into its
 /// basic block, this installs its name.
 bool LLParser::PerFunctionState::SetInstName(int NameID,
if (NameStr.empty()) {
 
     if (unsigned(NameID) != NumberedVals.size())
       return P.Error(NameLoc, "instruction expected to be numbered '%" +
                      Twine(NumberedVals.size()) + "'");
 
     std::map<unsigned, std::pair<Value*, LocTy> >::iterator FI =
       ForwardRefValIDs.find(NameID);
     if (FI != ForwardRefValIDs.end()) {
-      if (FI->second.first->getType() != Inst->getType())
+      Value *Sentinel = FI->second.first;
+      if (Sentinel->getType() != Inst->getType())
         return P.Error(NameLoc, "instruction forward referenced with type '" +
                        getTypeString(FI->second.first->getType()) + "'");
-      FI->second.first->replaceAllUsesWith(Inst);
-      delete FI->second.first;
+      // Check operator constraints.  We only put cleanuppads or catchpads in
+      // the forward value map if the value is constrained to match.
+      if (isa<CatchPadInst>(Sentinel)) {
+        if (!isa<CatchPadInst>(Inst))
+          return P.Error(FI->second.second,
+                         "'%" + Twine(NameID) + "' is not a catchpad");
+      } else if (isa<CleanupPadInst>(Sentinel)) {
+        if (!isa<CleanupPadInst>(Inst))
+          return P.Error(FI->second.second,
+                         "'%" + Twine(NameID) + "' is not a cleanuppad");
+      }
+
+      Sentinel->replaceAllUsesWith(Inst);
+      delete Sentinel;
       ForwardRefValIDs.erase(FI);
     }
 
     NumberedVals.push_back(Inst);
     return false;
   }
 
   // Otherwise, the instruction had a name.  Resolve forward refs and set it.
   std::map<std::string, std::pair<Value*, LocTy> >::iterator
     FI = ForwardRefVals.find(NameStr);
   if (FI != ForwardRefVals.end()) {
-    if (FI->second.first->getType() != Inst->getType())
+    Value *Sentinel = FI->second.first;
+    if (Sentinel->getType() != Inst->getType())
       return P.Error(NameLoc, "instruction forward referenced with type '" +
                      getTypeString(FI->second.first->getType()) + "'");
-    FI->second.first->replaceAllUsesWith(Inst);
-    delete FI->second.first;
+    // Check operator constraints.  We only put cleanuppads or catchpads in
+    // the forward value map if the value is constrained to match.
+    if (isa<CatchPadInst>(Sentinel)) {
+      if (!isa<CatchPadInst>(Inst))
+        return P.Error(FI->second.second,
+                       "'%" + NameStr + "' is not a catchpad");
+    } else if (isa<CleanupPadInst>(Sentinel)) {
+      if (!isa<CleanupPadInst>(Inst))
+        return P.Error(FI->second.second,
+                       "'%" + NameStr + "' is not a cleanuppad");
+    }
+
+    Sentinel->replaceAllUsesWith(Inst);
+    delete Sentinel;
     ForwardRefVals.erase(FI);
   }
 
   // Set the name on the instruction.
   Inst->setName(NameStr);
 
   if (Inst->getName() != NameStr)
     return P.Error(NameLoc, "multiple definition of local value named '" +

 }
 
 
 //===----------------------------------------------------------------------===//
 // Function Parsing.
 //===----------------------------------------------------------------------===//
 
 bool LLParser::ConvertValIDToValue(Type *Ty, ValID &ID, Value *&V,
-                                   PerFunctionState *PFS) {
+                                   PerFunctionState *PFS,
+                                   OperatorConstraint OC) {
   if (Ty->isFunctionTy())
     return Error(ID.Loc, "functions are not values, refer to them as pointers");
 
+  if (OC && ID.Kind != ValID::t_LocalID && ID.Kind != ValID::t_LocalName) {
+    switch (OC) {
+    case OperatorConstraint::catchpad:
+      return Error(ID.Loc, "Catchpad value required in this position");
+    case OperatorConstraint::cleanuppad:
+      return Error(ID.Loc, "Cleanuppad value required in this position");
+    default:
+      llvm_unreachable("Unexpected constraint kind");
+    }
+  }
+
   switch (ID.Kind) {
   case ValID::t_LocalID:
     if (!PFS) return Error(ID.Loc, "invalid use of function-local name");
-    V = PFS->GetVal(ID.UIntVal, Ty, ID.Loc);
+    V = PFS->GetVal(ID.UIntVal, Ty, ID.Loc, OC);
     return V == nullptr;
   case ValID::t_LocalName:
     if (!PFS) return Error(ID.Loc, "invalid use of function-local name");
-    V = PFS->GetVal(ID.StrVal, Ty, ID.Loc);
+    V = PFS->GetVal(ID.StrVal, Ty, ID.Loc, OC);
     return V == nullptr;
   case ValID::t_InlineAsm: {
     assert(ID.FTy);
     if (!InlineAsm::Verify(ID.FTy, ID.StrVal2))
       return Error(ID.Loc, "invalid type for inline asm constraint string");
     V = InlineAsm::get(ID.FTy, ID.StrVal, ID.StrVal2, ID.UIntVal & 1,
                        (ID.UIntVal >> 1) & 1,
                        (InlineAsm::AsmDialect(ID.UIntVal >> 2)));
case ValID::t_PackedConstantStruct: {
     C = cast<Constant>(V);
     return false;
   }
   default:
     return Error(Loc, "expected a constant value");
   }
 }
 
-bool LLParser::ParseValue(Type *Ty, Value *&V, PerFunctionState *PFS) {
+bool LLParser::ParseValue(Type *Ty, Value *&V, PerFunctionState *PFS,
+                          OperatorConstraint OC) {
   V = nullptr;
   ValID ID;
-  return ParseValID(ID, PFS) ||
-         ConvertValIDToValue(Ty, ID, V, PFS);
+  return ParseValID(ID, PFS) || ConvertValIDToValue(Ty, ID, V, PFS, OC);
 }
 
 bool LLParser::ParseTypeAndValue(Value *&V, PerFunctionState *PFS) {
   Type *Ty = nullptr;
   return ParseType(Ty) ||
          ParseValue(Ty, V, PFS);
 }
 
bool LLParser::ParseResume(Instruction *&Inst, PerFunctionState &PFS) {
 
   ResumeInst *RI = ResumeInst::Create(Exn);
   Inst = RI;
   return false;
 }
 
 bool LLParser::ParseExceptionArgs(SmallVectorImpl<Value *> &Args,
                                   PerFunctionState &PFS) {
-  if (ParseToken(lltok::lsquare, "expected '[' in cleanuppad"))
+  if (ParseToken(lltok::lsquare, "expected '[' in catchpad/cleanuppad"))
     return true;
 
   while (Lex.getKind() != lltok::rsquare) {
     // If this isn't the first argument, we need a comma.
     if (!Args.empty() &&
         ParseToken(lltok::comma, "expected ',' in argument list"))
       return true;
 
while (Lex.getKind() != lltok::rsquare) {
     Args.push_back(V);
   }
 
   Lex.Lex();  // Lex the ']'.
   return false;
 }
 
 /// ParseCleanupRet
-///   ::= 'cleanupret' ('void' | TypeAndValue) unwind ('to' 'caller' | TypeAndValue)
+///   ::= 'cleanupret' Value unwind ('to' 'caller' | TypeAndValue)
 bool LLParser::ParseCleanupRet(Instruction *&Inst, PerFunctionState &PFS) {
-  Type *RetTy = nullptr;
-  Value *RetVal = nullptr;
-  if (ParseType(RetTy, /*AllowVoid=*/true))
-    return true;
+  Value *CleanupPad = nullptr;
 
-  if (!RetTy->isVoidTy())
-    if (ParseValue(RetTy, RetVal, PFS))
-      return true;
+  if (ParseValue(Type::getTokenTy(Context), CleanupPad, PFS,
+                 OperatorConstraint::cleanuppad))
 
   if (ParseToken(lltok::kw_unwind, "expected 'unwind' in cleanupret"))
     return true;
 
   BasicBlock *UnwindBB = nullptr;
   if (Lex.getKind() == lltok::kw_to) {
     Lex.Lex();
     if (ParseToken(lltok::kw_caller, "expected 'caller' in cleanupret"))
       return true;
   } else {
     if (ParseTypeAndBasicBlock(UnwindBB, PFS)) {
       return true;
     }
   }
 
-  Inst = CleanupReturnInst::Create(Context, RetVal, UnwindBB);
+  Inst = CleanupReturnInst::Create(cast<CleanupPadInst>(CleanupPad), UnwindBB);
   return false;
 }
 
 /// ParseCatchRet
-///   ::= 'catchret' ('void' | TypeAndValue) 'to' TypeAndValue
+///   ::= 'catchret' Value 'to' TypeAndValue
 bool LLParser::ParseCatchRet(Instruction *&Inst, PerFunctionState &PFS) {
-  Type *RetTy = nullptr;
-  Value *RetVal = nullptr;
-
-  if (ParseType(RetTy, /*AllowVoid=*/true))
-    return true;
+  Value *CatchPad = nullptr;
 
-  if (!RetTy->isVoidTy())
-    if (ParseValue(RetTy, RetVal, PFS))
-      return true;
+  if (ParseValue(Type::getTokenTy(Context), CatchPad, PFS,
+                 OperatorConstraint::catchpad))
 
   BasicBlock *BB;
   if (ParseToken(lltok::kw_to, "expected 'to' in catchret") ||
       ParseTypeAndBasicBlock(BB, PFS))
       return true;
 
-  Inst = CatchReturnInst::Create(BB, RetVal);
+  Inst = CatchReturnInst::Create(cast<CatchPadInst>(CatchPad), BB);
   return false;
 }
 
 /// ParseCatchPad
-///   ::= 'catchpad' Type ParamList 'to' TypeAndValue 'unwind' TypeAndValue
+///   ::= 'catchpad' ParamList 'to' TypeAndValue 'unwind' TypeAndValue
 bool LLParser::ParseCatchPad(Instruction *&Inst, PerFunctionState &PFS) {
-  Type *RetType = nullptr;
-
   SmallVector<Value *, 8> Args;
-  if (ParseType(RetType, /*AllowVoid=*/true) || ParseExceptionArgs(Args, PFS))
+  if (ParseExceptionArgs(Args, PFS))
     return true;
 
   BasicBlock *NormalBB, *UnwindBB;
   if (ParseToken(lltok::kw_to, "expected 'to' in catchpad") ||
       ParseTypeAndBasicBlock(NormalBB, PFS) ||
       ParseToken(lltok::kw_unwind, "expected 'unwind' in catchpad") ||
       ParseTypeAndBasicBlock(UnwindBB, PFS))
     return true;
 
-  Inst = CatchPadInst::Create(RetType, NormalBB, UnwindBB, Args);
+  Inst = CatchPadInst::Create(NormalBB, UnwindBB, Args);
   return false;
 }
 
 /// ParseTerminatePad
 ///   ::= 'terminatepad' ParamList 'to' TypeAndValue
 bool LLParser::ParseTerminatePad(Instruction *&Inst, PerFunctionState &PFS) {
   SmallVector<Value *, 8> Args;
   if (ParseExceptionArgs(Args, PFS))
bool LLParser::ParseTerminatePad(Instruction *&Inst, PerFunctionState &PFS) {
 
   Inst = TerminatePadInst::Create(Context, UnwindBB, Args);
   return false;
 }
 
 /// ParseCleanupPad
 ///   ::= 'cleanuppad' ParamList
 bool LLParser::ParseCleanupPad(Instruction *&Inst, PerFunctionState &PFS) {
-  Type *RetType = nullptr;
-
   SmallVector<Value *, 8> Args;
-  if (ParseType(RetType, /*AllowVoid=*/true) || ParseExceptionArgs(Args, PFS))
+  if (ParseExceptionArgs(Args, PFS))
     return true;
 
-  Inst = CleanupPadInst::Create(RetType, Args);
+  Inst = CleanupPadInst::Create(Context, Args);
   return false;
 }
 
 /// ParseCatchEndPad
 ///   ::= 'catchendpad' unwind ('to' 'caller' | TypeAndValue)
 bool LLParser::ParseCatchEndPad(Instruction *&Inst, PerFunctionState &PFS) {
   if (ParseToken(lltok::kw_unwind, "expected 'unwind' in catchendpad"))
     return true;

lib/Bitcode/Reader/BitcodeReader.cpp

 #include <deque>
 using namespace llvm;
 
 namespace {
 enum {
   SWITCH_INST_MAGIC = 0x4B5 // May 2012 => 1205 => Hex
 };
 
+/// Indicates which operator an operand allows (for the few operands that may
+/// only reference a certain operator).
+enum OperatorConstraint {
+  none = 0,  // No constraint
+  catchpad,  // Must be CatchPadInst
+  cleanuppad // Must be CleanupPadInst
+};
+
 class BitcodeReaderValueList {
   std::vector<WeakVH> ValuePtrs;
 
   /// As we resolve forward-referenced constants, we add information about them
   /// to this vector.  This allows us to resolve them in bulk instead of
   /// resolving each reference at a time.  See the code in
   /// ResolveConstantForwardRefs for more information about this.
   ///
Value *back() const { return ValuePtrs.back(); }
     void pop_back() { ValuePtrs.pop_back(); }
   bool empty() const { return ValuePtrs.empty(); }
   void shrinkTo(unsigned N) {
     assert(N <= size() && "Invalid shrinkTo request!");
     ValuePtrs.resize(N);
   }
 
   Constant *getConstantFwdRef(unsigned Idx, Type *Ty);
-  Value *getValueFwdRef(unsigned Idx, Type *Ty);
+  Value *getValueFwdRef(unsigned Idx, Type *Ty,
+                        OperatorConstraint OC = OperatorConstraint::none);
 
-  void assignValue(Value *V, unsigned Idx);
+  bool assignValue(Value *V, unsigned Idx);
 
   /// Once all constants are read, this method bulk resolves any forward
   /// references.
   void resolveConstantForwardRefs();
 };
 
 class BitcodeReaderMDValueList {
   unsigned NumFwdRefs;
public:
   void setStripDebugInfo() override;
 
 private:
   std::vector<StructType *> IdentifiedStructTypes;
   StructType *createIdentifiedStructType(LLVMContext &Context, StringRef Name);
   StructType *createIdentifiedStructType(LLVMContext &Context);
 
   Type *getTypeByID(unsigned ID);
-  Value *getFnValueByID(unsigned ID, Type *Ty) {
+  Value *getFnValueByID(unsigned ID, Type *Ty,
+                        OperatorConstraint OC = OperatorConstraint::none) {
     if (Ty && Ty->isMetadataTy())
       return MetadataAsValue::get(Ty->getContext(), getFnMetadataByID(ID));
-    return ValueList.getValueFwdRef(ID, Ty);
+    return ValueList.getValueFwdRef(ID, Ty, OC);
   }
   Metadata *getFnMetadataByID(unsigned ID) {
     return MDValueList.getValueFwdRef(ID);
   }
   BasicBlock *getBasicBlock(unsigned ID) const {
     if (ID >= FunctionBBs.size()) return nullptr; // Invalid ID
     return FunctionBBs[ID];
   }
bool getValueTypePair(SmallVectorImpl<uint64_t> &Record, unsigned &Slot,
     ResVal = getFnValueByID(ValNo, getTypeByID(TypeNo));
     return ResVal == nullptr;
   }
 
   /// Read a value out of the specified record from slot 'Slot'. Increment Slot
   /// past the number of slots used by the value in the record. Return true if
   /// there is an error.
   bool popValue(SmallVectorImpl<uint64_t> &Record, unsigned &Slot,
-                unsigned InstNum, Type *Ty, Value *&ResVal) {
-    if (getValue(Record, Slot, InstNum, Ty, ResVal))
+                unsigned InstNum, Type *Ty, Value *&ResVal,
+                OperatorConstraint OC = OperatorConstraint::none) {
+    if (getValue(Record, Slot, InstNum, Ty, ResVal, OC))
       return true;
     // All values currently take a single record slot.
     ++Slot;
     return false;
   }
 
   /// Like popValue, but does not increment the Slot number.
   bool getValue(SmallVectorImpl<uint64_t> &Record, unsigned Slot,
-                unsigned InstNum, Type *Ty, Value *&ResVal) {
-    ResVal = getValue(Record, Slot, InstNum, Ty);
+                unsigned InstNum, Type *Ty, Value *&ResVal,
+                OperatorConstraint OC = OperatorConstraint::none) {
+    ResVal = getValue(Record, Slot, InstNum, Ty, OC);
     return ResVal == nullptr;
   }
 
   /// Version of getValue that returns ResVal directly, or 0 if there is an
   /// error.
   Value *getValue(SmallVectorImpl<uint64_t> &Record, unsigned Slot,
-                  unsigned InstNum, Type *Ty) {
+                  unsigned InstNum, Type *Ty,
+                  OperatorConstraint OC = OperatorConstraint::none) {
     if (Slot == Record.size()) return nullptr;
     unsigned ValNo = (unsigned)Record[Slot];
     // Adjust the ValNo, if it was encoded relative to the InstNum.
     if (UseRelativeIDs)
       ValNo = InstNum - ValNo;
-    return getFnValueByID(ValNo, Ty);
+    return getFnValueByID(ValNo, Ty, OC);
   }
 
   /// Like getValue, but decodes signed VBRs.
   Value *getValueSigned(SmallVectorImpl<uint64_t> &Record, unsigned Slot,
-                        unsigned InstNum, Type *Ty) {
+                        unsigned InstNum, Type *Ty,
+                        OperatorConstraint OC = OperatorConstraint::none) {
     if (Slot == Record.size()) return nullptr;
     unsigned ValNo = (unsigned)decodeSignRotatedValue(Record[Slot]);
     // Adjust the ValNo, if it was encoded relative to the InstNum.
     if (UseRelativeIDs)
       ValNo = InstNum - ValNo;
-    return getFnValueByID(ValNo, Ty);
+    return getFnValueByID(ValNo, Ty, OC);
   }
 
   /// Converts alignment exponent (i.e. power of two (or zero)) to the
   /// corresponding alignment to use. If alignment is too large, returns
   /// a corresponding error code.
   std::error_code parseAlignmentValue(uint64_t Exponent, unsigned &Alignment);
   std::error_code parseAttrKind(uint64_t Code, Attribute::AttrKind *Kind);
   std::error_code parseModule(bool Resume, bool ShouldLazyLoadMetadata = false);

 // FIXME: can we inherit this from ConstantExpr?
 template <>
 struct OperandTraits<ConstantPlaceHolder> :
   public FixedNumOperandTraits<ConstantPlaceHolder, 1> {
 };
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(ConstantPlaceHolder, Value)
 }
 
-void BitcodeReaderValueList::assignValue(Value *V, unsigned Idx) {
+bool BitcodeReaderValueList::assignValue(Value *V, unsigned Idx) {
   if (Idx == size()) {
     push_back(V);
-    return;
+    return false;
   }
 
   if (Idx >= size())
     resize(Idx+1);
 
   WeakVH &OldV = ValuePtrs[Idx];
   if (!OldV) {
     OldV = V;
-    return;
+    return false;
   }
 
   // Handle constants and non-constants (e.g. instrs) differently for
   // efficiency.
   if (Constant *PHC = dyn_cast<Constant>(&*OldV)) {
     ResolveConstants.push_back(std::make_pair(PHC, Idx));
     OldV = V;
   } else {
     // If there was a forward reference to this value, replace it.
     Value *PrevVal = OldV;
+    // Check operator constraints.  We only put cleanuppads or catchpads in
+    // the forward value map if the value is constrained to match.
+    if (CatchPadInst *CatchPad = dyn_cast<CatchPadInst>(PrevVal)) {
+      if (!isa<CatchPadInst>(V))
+        return true;
+      // Delete the dummy basic block that was created with the sentinel
+      // catchpad.
+      BasicBlock *DummyBlock = CatchPad->getUnwindDest();
+      assert(DummyBlock == CatchPad->getNormalDest());
+      CatchPad->dropAllReferences();
+      delete DummyBlock;
+    } else if (isa<CleanupPadInst>(PrevVal)) {
+      if (!isa<CleanupPadInst>(V))
+        return true;
+    }
     OldV->replaceAllUsesWith(V);
     delete PrevVal;
   }
+
+  return false;
 }
 
 
 Constant *BitcodeReaderValueList::getConstantFwdRef(unsigned Idx,
                                                     Type *Ty) {
   if (Idx >= size())
     resize(Idx + 1);
 
   if (Value *V = ValuePtrs[Idx]) {
     if (Ty != V->getType())
       report_fatal_error("Type mismatch in constant table!");
     return cast<Constant>(V);
   }
 
   // Create and return a placeholder, which will later be RAUW'd.
   Constant *C = new ConstantPlaceHolder(Ty, Context);
   ValuePtrs[Idx] = C;
   return C;
 }
 
-Value *BitcodeReaderValueList::getValueFwdRef(unsigned Idx, Type *Ty) {
+Value *BitcodeReaderValueList::getValueFwdRef(unsigned Idx, Type *Ty,
+                                              OperatorConstraint OC) {
   // Bail out for a clearly invalid value. This would make us call resize(0)
   if (Idx == UINT_MAX)
     return nullptr;
 
   if (Idx >= size())
     resize(Idx + 1);
 
   if (Value *V = ValuePtrs[Idx]) {
     // If the types don't match, it's invalid.
     if (Ty && Ty != V->getType())
       return nullptr;
+    if (!OC)
       return V;
+    // Use dyn_cast to enforce operator constraints
+    switch (OC) {
+    case OperatorConstraint::catchpad:
+      return dyn_cast<CatchPadInst>(V);
+    case OperatorConstraint::cleanuppad:
+      return dyn_cast<CleanupPadInst>(V);
+    default:
+      llvm_unreachable("Unexpected operator constraint");
+    }
   }
 
   // No type specified, must be invalid reference.
   if (!Ty) return nullptr;
 
   // Create and return a placeholder, which will later be RAUW'd.
-  Value *V = new Argument(Ty);
+  Value *V;
+  if (!OC)
+    V = new Argument(Ty);
+  else
+    switch (OC) {
+    case OperatorConstraint::catchpad: {
+      BasicBlock *BB = BasicBlock::Create(Context);
+      V = CatchPadInst::Create(BB, BB, {});
+      break;
+    }
+    case OperatorConstraint::cleanuppad: {
+      V = CleanupPadInst::Create(Context, {});
+      break;
+    }
+    default:
+      llvm_unreachable("Unexpected operator constraint");
+    }
   ValuePtrs[Idx] = V;
   return V;
 }
 
 /// Once all constants are read, this method bulk resolves any forward
 /// references.  The idea behind this is that we sometimes get constants (such
 /// as large arrays) which reference *many* forward ref constants.  Replacing
 /// each of these causes a lot of thrashing when building/reuniquing the
case bitc::CST_CODE_BLOCKADDRESS:{
           FwdBBs[BBID] = BasicBlock::Create(Context);
         BB = FwdBBs[BBID];
       }
       V = BlockAddress::get(Fn, BB);
       break;
     }
     }
 
-    ValueList.assignValue(V, NextCstNo);
+    if (ValueList.assignValue(V, NextCstNo))
+      return error("Invalid forward reference");
     ++NextCstNo;
   }
 }
 
 std::error_code BitcodeReader::parseUseLists() {
   if (Stream.EnterSubBlock(bitc::USELIST_BLOCK_ID))
     return error("Invalid record");
 
case bitc::FUNC_CODE_INST_BR: { // BR: [bb#, bb#, opval] or [bb#]
                                Type::getInt1Ty(Context));
         if (!FalseDest || !Cond)
           return error("Invalid record");
         I = BranchInst::Create(TrueDest, FalseDest, Cond);
         InstructionList.push_back(I);
       }
       break;
     }
-    // CLEANUPRET: [] or [ty,val] or [bb#] or [ty,val,bb#]
+    // CLEANUPRET: [val] or [val,bb#]
     case bitc::FUNC_CODE_INST_CLEANUPRET: {
-      if (Record.size() < 2)
+      if (Record.size() != 1 && Record.size() != 2)
         return error("Invalid record");
       unsigned Idx = 0;
-      bool HasReturnValue = !!Record[Idx++];
-      bool HasUnwindDest = !!Record[Idx++];
-      Value *RetVal = nullptr;
-      BasicBlock *UnwindDest = nullptr;
-
-      if (HasReturnValue && getValueTypePair(Record, Idx, NextValueNo, RetVal))
-        return error("Invalid record");
-      if (HasUnwindDest) {
-        if (Idx == Record.size())
-          return error("Invalid record");
+      Value *CleanupPad =
+          getValue(Record, Idx++, NextValueNo, Type::getTokenTy(Context),
+                   OperatorConstraint::cleanuppad);
+      if (!CleanupPad)
+      BasicBlock *UnwindDest = nullptr;
+      if (Record.size() == 2) {
         UnwindDest = getBasicBlock(Record[Idx++]);
         if (!UnwindDest)
           return error("Invalid record");
       }
 
-      if (Record.size() != Idx)
-        return error("Invalid record");
-
-      I = CleanupReturnInst::Create(Context, RetVal, UnwindDest);
+      I = CleanupReturnInst::Create(cast<CleanupPadInst>(CleanupPad),
+                                    UnwindDest);
       InstructionList.push_back(I);
       break;
     }
-    case bitc::FUNC_CODE_INST_CATCHRET: { // CATCHRET: [bb#]
-      if (Record.size() != 1 && Record.size() != 3)
+    case bitc::FUNC_CODE_INST_CATCHRET: { // CATCHRET: [val,bb#]
+      if (Record.size() != 2)
         return error("Invalid record");
       unsigned Idx = 0;
+      Value *CatchPad =
+          getValue(Record, Idx++, NextValueNo, Type::getTokenTy(Context),
+                   OperatorConstraint::catchpad);
+      if (!CatchPad)
+        return error("Invalid record");
       BasicBlock *BB = getBasicBlock(Record[Idx++]);
       if (!BB)
         return error("Invalid record");
-      Value *RetVal = nullptr;
-      if (Record.size() == 3 &&
-          getValueTypePair(Record, Idx, NextValueNo, RetVal))
-        return error("Invalid record");
 
-      I = CatchReturnInst::Create(BB, RetVal);
+      I = CatchReturnInst::Create(cast<CatchPadInst>(CatchPad), BB);
       InstructionList.push_back(I);
       break;
     }
-    case bitc::FUNC_CODE_INST_CATCHPAD: { // CATCHPAD: [ty,bb#,bb#,num,(ty,val)*]
-      if (Record.size() < 4)
+    case bitc::FUNC_CODE_INST_CATCHPAD: { // CATCHPAD: [bb#,bb#,num,(ty,val)*]
+      if (Record.size() < 3)
         return error("Invalid record");
       unsigned Idx = 0;
-      Type *Ty = getTypeByID(Record[Idx++]);
-      if (!Ty)
-        return error("Invalid record");
       BasicBlock *NormalBB = getBasicBlock(Record[Idx++]);
       if (!NormalBB)
         return error("Invalid record");
       BasicBlock *UnwindBB = getBasicBlock(Record[Idx++]);
       if (!UnwindBB)
         return error("Invalid record");
       unsigned NumArgOperands = Record[Idx++];
       SmallVector<Value *, 2> Args;
       for (unsigned Op = 0; Op != NumArgOperands; ++Op) {
         Value *Val;
         if (getValueTypePair(Record, Idx, NextValueNo, Val))
           return error("Invalid record");
         Args.push_back(Val);
       }
       if (Record.size() != Idx)
         return error("Invalid record");
 
-      I = CatchPadInst::Create(Ty, NormalBB, UnwindBB, Args);
+      I = CatchPadInst::Create(NormalBB, UnwindBB, Args);
       InstructionList.push_back(I);
       break;
     }
     case bitc::FUNC_CODE_INST_TERMINATEPAD: { // TERMINATEPAD: [bb#,num,(ty,val)*]
       if (Record.size() < 1)
         return error("Invalid record");
       unsigned Idx = 0;
       bool HasUnwindDest = !!Record[Idx++];
case bitc::FUNC_CODE_INST_TERMINATEPAD: { // TERMINATEPAD: [bb#,num,(ty,val)*]
       }
       if (Record.size() != Idx)
         return error("Invalid record");
 
       I = TerminatePadInst::Create(Context, UnwindDest, Args);
       InstructionList.push_back(I);
       break;
     }
-    case bitc::FUNC_CODE_INST_CLEANUPPAD: { // CLEANUPPAD: [ty, num,(ty,val)*]
-      if (Record.size() < 2)
+    case bitc::FUNC_CODE_INST_CLEANUPPAD: { // CLEANUPPAD: [num,(ty,val)*]
+      if (Record.size() < 1)
         return error("Invalid record");
       unsigned Idx = 0;
-      Type *Ty = getTypeByID(Record[Idx++]);
-      if (!Ty)
-        return error("Invalid record");
       unsigned NumArgOperands = Record[Idx++];
       SmallVector<Value *, 2> Args;
       for (unsigned Op = 0; Op != NumArgOperands; ++Op) {
         Value *Val;
         if (getValueTypePair(Record, Idx, NextValueNo, Val))
           return error("Invalid record");
         Args.push_back(Val);
       }
       if (Record.size() != Idx)
         return error("Invalid record");
 
-      I = CleanupPadInst::Create(Ty, Args);
+      I = CleanupPadInst::Create(Context, Args);
       InstructionList.push_back(I);
       break;
     }
     case bitc::FUNC_CODE_INST_CATCHENDPAD: { // CATCHENDPADINST: [bb#] or []
       if (Record.size() > 1)
         return error("Invalid record");
       BasicBlock *BB = nullptr;
       if (Record.size() == 1) {

     // If this was a terminator instruction, move to the next block.
     if (isa<TerminatorInst>(I)) {
       ++CurBBNo;
       CurBB = CurBBNo < FunctionBBs.size() ? FunctionBBs[CurBBNo] : nullptr;
     }
 
     // Non-void values get registered in the value table for future use.
     if (I && !I->getType()->isVoidTy())
-      ValueList.assignValue(I, NextValueNo++);
+      if (ValueList.assignValue(I, NextValueNo++))
+        return error("Invalid forward reference");
   }
 
 OutOfRecordLoop:
 
   // Check the function list for unresolved values.
   if (Argument *A = dyn_cast<Argument>(ValueList.back())) {
     if (!A->getParent()) {
       // We found at least one unresolved value.  Nuke them all to avoid leaks.

lib/Bitcode/Writer/BitcodeWriter.cpp

static void WriteInstruction(const Instruction &I, unsigned InstID,
   }
   case Instruction::Resume:
     Code = bitc::FUNC_CODE_INST_RESUME;
     PushValueAndType(I.getOperand(0), InstID, Vals, VE);
     break;
   case Instruction::CleanupRet: {
     Code = bitc::FUNC_CODE_INST_CLEANUPRET;
     const auto &CRI = cast<CleanupReturnInst>(I);
-    Vals.push_back(CRI.hasReturnValue());
-    Vals.push_back(CRI.hasUnwindDest());
-    if (CRI.hasReturnValue())
-      PushValueAndType(CRI.getReturnValue(), InstID, Vals, VE);
+    pushValue(CRI.getCleanupPad(), InstID, Vals, VE);
     if (CRI.hasUnwindDest())
       Vals.push_back(VE.getValueID(CRI.getUnwindDest()));
     break;
   }
   case Instruction::CatchRet: {
     Code = bitc::FUNC_CODE_INST_CATCHRET;
     const auto &CRI = cast<CatchReturnInst>(I);
+    pushValue(CRI.getCatchPad(), InstID, Vals, VE);
     Vals.push_back(VE.getValueID(CRI.getSuccessor()));
-    if (CRI.hasReturnValue())
-      PushValueAndType(CRI.getReturnValue(), InstID, Vals, VE);
     break;
   }
   case Instruction::CatchPad: {
     Code = bitc::FUNC_CODE_INST_CATCHPAD;
     const auto &CPI = cast<CatchPadInst>(I);
-    Vals.push_back(VE.getTypeID(CPI.getType()));
     Vals.push_back(VE.getValueID(CPI.getNormalDest()));
     Vals.push_back(VE.getValueID(CPI.getUnwindDest()));
     unsigned NumArgOperands = CPI.getNumArgOperands();
     Vals.push_back(NumArgOperands);
     for (unsigned Op = 0; Op != NumArgOperands; ++Op)
       PushValueAndType(CPI.getArgOperand(Op), InstID, Vals, VE);
     break;
   }
   case Instruction::TerminatePad: {
     Code = bitc::FUNC_CODE_INST_TERMINATEPAD;
     const auto &TPI = cast<TerminatePadInst>(I);
     Vals.push_back(TPI.hasUnwindDest());
     if (TPI.hasUnwindDest())
       Vals.push_back(VE.getValueID(TPI.getUnwindDest()));
     unsigned NumArgOperands = TPI.getNumArgOperands();
     Vals.push_back(NumArgOperands);
     for (unsigned Op = 0; Op != NumArgOperands; ++Op)
       PushValueAndType(TPI.getArgOperand(Op), InstID, Vals, VE);
     break;
   }
   case Instruction::CleanupPad: {
     Code = bitc::FUNC_CODE_INST_CLEANUPPAD;
     const auto &CPI = cast<CleanupPadInst>(I);
-    Vals.push_back(VE.getTypeID(CPI.getType()));
     unsigned NumOperands = CPI.getNumOperands();
     Vals.push_back(NumOperands);
     for (unsigned Op = 0; Op != NumOperands; ++Op)
       PushValueAndType(CPI.getOperand(Op), InstID, Vals, VE);
     break;
   }
   case Instruction::CatchEndPad: {
     Code = bitc::FUNC_CODE_INST_CATCHENDPAD;

lib/CodeGen/WinEHPrepare.cpp

 // to. If the unwind edge came from an invoke, return null.
 static const BasicBlock *getEHPadFromPredecessor(const BasicBlock *BB) {
   const TerminatorInst *TI = BB->getTerminator();
   if (isa<InvokeInst>(TI))
     return nullptr;
   if (isa<CatchPadInst>(TI) || isa<CatchEndPadInst>(TI) ||
       isa<TerminatePadInst>(TI))
     return BB;
-  return cast<CleanupPadInst>(cast<CleanupReturnInst>(TI)->getReturnValue())
-      ->getParent();
+  return cast<CleanupReturnInst>(TI)->getCleanupPad()->getParent();
 }
 
 static void calculateExplicitStateNumbers(WinEHFuncInfo &FuncInfo,
                                           const BasicBlock &BB,
                                           int ParentState) {
   assert(BB.isEHPad());
   const Instruction *FirstNonPHI = BB.getFirstNonPHI();
   // All catchpad instructions will be handled when we process their
for (auto &Funclet : FuncletBlocks) {
 
     for (BasicBlock *BB : BlocksInFunclet) {
       TerminatorInst *TI = BB->getTerminator();
       // CatchPadInst and CleanupPadInst can't transfer control to a ReturnInst.
       bool IsUnreachableRet = isa<ReturnInst>(TI) && (CatchPad || CleanupPad);
       // The token consumed by a CatchReturnInst must match the funclet token.
       bool IsUnreachableCatchret = false;
       if (auto *CRI = dyn_cast<CatchReturnInst>(TI))
-        IsUnreachableCatchret = CRI->getReturnValue() != CatchPad;
+        IsUnreachableCatchret = CRI->getCatchPad() != CatchPad;
       // The token consumed by a CleanupPadInst must match the funclet token.
       bool IsUnreachableCleanupret = false;
       if (auto *CRI = dyn_cast<CleanupReturnInst>(TI))
-        IsUnreachableCleanupret = CRI->getReturnValue() != CleanupPad;
+        IsUnreachableCleanupret = CRI->getCleanupPad() != CleanupPad;
       if (IsUnreachableRet || IsUnreachableCatchret || IsUnreachableCleanupret) {
         new UnreachableInst(BB->getContext(), TI);
         TI->eraseFromParent();
       }
     }
   }
 
   // Clean-up some of the mess we made by removing useles PHI nodes, trivial

lib/IR/AsmWriter.cpp

for (unsigned i = 0, e = LPI->getNumClauses(); i != e; ++i) {
       if (LPI->isCatch(i))
         Out << "          catch ";
       else
         Out << "          filter ";
 
       writeOperand(LPI->getClause(i), true);
     }
   } else if (const auto *CPI = dyn_cast<CatchPadInst>(&I)) {
-    Out << ' ';
-    TypePrinter.print(I.getType(), Out);
-
     Out << " [";
     for (unsigned Op = 0, NumOps = CPI->getNumArgOperands(); Op < NumOps;
          ++Op) {
       if (Op > 0)
         Out << ", ";
       writeOperand(CPI->getArgOperand(Op), /*PrintType=*/true);
     }
     Out << "] to ";
for (unsigned Op = 0, NumOps = TPI->getNumArgOperands(); Op < NumOps;
       writeOperand(TPI->getArgOperand(Op), /*PrintType=*/true);
     }
     Out << "] unwind ";
     if (TPI->hasUnwindDest())
       writeOperand(TPI->getUnwindDest(), /*PrintType=*/true);
     else
       Out << "to caller";
   } else if (const auto *CPI = dyn_cast<CleanupPadInst>(&I)) {
-    Out << ' ';
-    TypePrinter.print(I.getType(), Out);
-
     Out << " [";
     for (unsigned Op = 0, NumOps = CPI->getNumOperands(); Op < NumOps; ++Op) {
       if (Op > 0)
         Out << ", ";
       writeOperand(CPI->getOperand(Op), /*PrintType=*/true);
     }
     Out << "]";
   } else if (isa<ReturnInst>(I) && !Operand) {
     Out << " void";
   } else if (const auto *CRI = dyn_cast<CatchReturnInst>(&I)) {
-    if (CRI->hasReturnValue()) {
-      Out << ' ';
-      writeOperand(CRI->getReturnValue(), /*PrintType=*/true);
-    } else {
-      Out << " void";
-    }
+    writeOperand(CRI->getCatchPad(), /*PrintType=*/false);
 
     Out << " to ";
     writeOperand(CRI->getSuccessor(), /*PrintType=*/true);
   } else if (const auto *CRI = dyn_cast<CleanupReturnInst>(&I)) {
-    if (CRI->hasReturnValue()) {
-      Out << ' ';
-      writeOperand(CRI->getReturnValue(), /*PrintType=*/true);
-    } else {
-      Out << " void";
-    }
+    writeOperand(CRI->getCleanupPad(), /*PrintType=*/false);
 
     Out << " unwind ";
     if (CRI->hasUnwindDest())
       writeOperand(CRI->getUnwindDest(), /*PrintType=*/true);
     else
       Out << "to caller";
   } else if (const auto *CEPI = dyn_cast<CatchEndPadInst>(&I)) {
     Out << " unwind ";

lib/IR/Instruction.cpp

const char *Instruction::getOpcodeName(unsigned OpCode) {
   case AShr:           return "ashr";
   case VAArg:          return "va_arg";
   case ExtractElement: return "extractelement";
   case InsertElement:  return "insertelement";
   case ShuffleVector:  return "shufflevector";
   case ExtractValue:   return "extractvalue";
   case InsertValue:    return "insertvalue";
   case LandingPad:     return "landingpad";
-  case CleanupPad:   return "cleanuppad";
+  case CleanupPad:     return "cleanuppad";
 
   default: return "<Invalid operator> ";
   }
 }
 
 /// Return true if both instructions have the same special state
 /// This must be kept in sync with lib/Transforms/IPO/MergeFunctions.cpp.
 static bool haveSameSpecialState(const Instruction *I1, const Instruction *I2,

lib/IR/Instructions.cpp

 
 CleanupReturnInst::CleanupReturnInst(const CleanupReturnInst &CRI)
     : TerminatorInst(CRI.getType(), Instruction::CleanupRet,
                      OperandTraits<CleanupReturnInst>::op_end(this) -
                          CRI.getNumOperands(),
                      CRI.getNumOperands()) {
   SubclassOptionalData = CRI.SubclassOptionalData;
   setInstructionSubclassData(CRI.getSubclassDataFromInstruction());
-  if (Value *RetVal = CRI.getReturnValue())
-    setReturnValue(RetVal);
-  if (BasicBlock *UnwindDest = CRI.getUnwindDest())
-    setUnwindDest(UnwindDest);
+  Op<-1>() = CRI.Op<-1>();
+  if (CRI.hasUnwindDest())
+    Op<-2>() = CRI.Op<-2>();
 }
 
-void CleanupReturnInst::init(Value *RetVal, BasicBlock *UnwindBB) {
+void CleanupReturnInst::init(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB) {
   SubclassOptionalData = 0;
   if (UnwindBB)
     setInstructionSubclassData(getSubclassDataFromInstruction() | 1);
-  if (RetVal)
-    setInstructionSubclassData(getSubclassDataFromInstruction() | 2);
 
+  Op<-1>() = CleanupPad;
   if (UnwindBB)
-    setUnwindDest(UnwindBB);
-  if (RetVal)
-    setReturnValue(RetVal);
+    Op<-2>() = UnwindBB;
 }
 
-CleanupReturnInst::CleanupReturnInst(LLVMContext &C, Value *RetVal,
+CleanupReturnInst::CleanupReturnInst(CleanupPadInst *CleanupPad,
                                      BasicBlock *UnwindBB, unsigned Values,
                                      Instruction *InsertBefore)
-    : TerminatorInst(Type::getVoidTy(C), Instruction::CleanupRet,
+    : TerminatorInst(Type::getVoidTy(CleanupPad->getContext()),
+                     Instruction::CleanupRet,
                      OperandTraits<CleanupReturnInst>::op_end(this) - Values,
                      Values, InsertBefore) {
-  init(RetVal, UnwindBB);
+  init(CleanupPad, UnwindBB);
 }
 
-CleanupReturnInst::CleanupReturnInst(LLVMContext &C, Value *RetVal,
+CleanupReturnInst::CleanupReturnInst(CleanupPadInst *CleanupPad,
                                      BasicBlock *UnwindBB, unsigned Values,
                                      BasicBlock *InsertAtEnd)
-    : TerminatorInst(Type::getVoidTy(C), Instruction::CleanupRet,
+    : TerminatorInst(Type::getVoidTy(CleanupPad->getContext()),
+                     Instruction::CleanupRet,
                      OperandTraits<CleanupReturnInst>::op_end(this) - Values,
                      Values, InsertAtEnd) {
-  init(RetVal, UnwindBB);
-}
-
-BasicBlock *CleanupReturnInst::getUnwindDest() const {
-  if (hasUnwindDest())
-    return cast<BasicBlock>(getOperand(getUnwindLabelOpIdx()));
-  return nullptr;
-}
-void CleanupReturnInst::setUnwindDest(BasicBlock *NewDest) {
-  assert(NewDest);
-  setOperand(getUnwindLabelOpIdx(), NewDest);
+  init(CleanupPad, UnwindBB);
 }
 
 BasicBlock *CleanupReturnInst::getSuccessorV(unsigned Idx) const {
   assert(Idx == 0);
   return getUnwindDest();
 }
 unsigned CleanupReturnInst::getNumSuccessorsV() const {
   return getNumSuccessors();

 void CatchEndPadInst::setSuccessorV(unsigned Idx, BasicBlock *B) {
   assert(Idx == 0);
   setUnwindDest(B);
 }
 
 //===----------------------------------------------------------------------===//
 //                        CatchReturnInst Implementation
 //===----------------------------------------------------------------------===//
-void CatchReturnInst::init(BasicBlock *BB, Value *RetVal) {
-  Op<-1>() = BB;
-  if (RetVal)
-    Op<-2>() = RetVal;
+void CatchReturnInst::init(CatchPadInst *CatchPad, BasicBlock *BB) {
+  Op<0>() = CatchPad;
+  Op<1>() = BB;
 }
 
 CatchReturnInst::CatchReturnInst(const CatchReturnInst &CRI)
     : TerminatorInst(Type::getVoidTy(CRI.getContext()), Instruction::CatchRet,
-                     OperandTraits<CatchReturnInst>::op_end(this) -
-                         CRI.getNumOperands(),
-                     CRI.getNumOperands()) {
-  Op<-1>() = CRI.Op<-1>();
-  if (CRI.getNumOperands() != 1) {
-    assert(CRI.getNumOperands() == 2);
-    Op<-2>() = CRI.Op<-2>();
-  }
+                     OperandTraits<CatchReturnInst>::op_begin(this), 2) {
+  Op<0>() = CRI.Op<0>();
+  Op<1>() = CRI.Op<1>();
 }
 
-CatchReturnInst::CatchReturnInst(BasicBlock *BB, Value *RetVal, unsigned Values,
+CatchReturnInst::CatchReturnInst(CatchPadInst *CatchPad, BasicBlock *BB,
                                  Instruction *InsertBefore)
     : TerminatorInst(Type::getVoidTy(BB->getContext()), Instruction::CatchRet,
-                     OperandTraits<CatchReturnInst>::op_end(this) - Values,
-                     Values, InsertBefore) {
-  init(BB, RetVal);
+                     OperandTraits<CatchReturnInst>::op_begin(this), 2,
+                     InsertBefore) {
+  init(CatchPad, BB);
 }
 
-CatchReturnInst::CatchReturnInst(BasicBlock *BB, Value *RetVal, unsigned Values,
+CatchReturnInst::CatchReturnInst(CatchPadInst *CatchPad, BasicBlock *BB,
                                  BasicBlock *InsertAtEnd)
     : TerminatorInst(Type::getVoidTy(BB->getContext()), Instruction::CatchRet,
-                     OperandTraits<CatchReturnInst>::op_end(this) - Values,
-                     Values, InsertAtEnd) {
-  init(BB, RetVal);
+                     OperandTraits<CatchReturnInst>::op_begin(this), 2,
+                     InsertAtEnd) {
+  init(CatchPad, BB);
 }
 
 BasicBlock *CatchReturnInst::getSuccessorV(unsigned Idx) const {
   assert(Idx == 0);
   return getSuccessor();
 }
 unsigned CatchReturnInst::getNumSuccessorsV() const {
   return getNumSuccessors();

 CatchPadInst::CatchPadInst(const CatchPadInst &CPI)
     : TerminatorInst(CPI.getType(), Instruction::CatchPad,
                      OperandTraits<CatchPadInst>::op_end(this) -
                          CPI.getNumOperands(),
                      CPI.getNumOperands()) {
   std::copy(CPI.op_begin(), CPI.op_end(), op_begin());
 }
 
-CatchPadInst::CatchPadInst(Type *RetTy, BasicBlock *IfNormal,
-                           BasicBlock *IfException, ArrayRef<Value *> Args,
-                           unsigned Values, const Twine &NameStr,
-                           Instruction *InsertBefore)
-    : TerminatorInst(RetTy, Instruction::CatchPad,
+CatchPadInst::CatchPadInst(BasicBlock *IfNormal, BasicBlock *IfException,
+                           ArrayRef<Value *> Args, unsigned Values,
+                           const Twine &NameStr, Instruction *InsertBefore)
+    : TerminatorInst(Type::getTokenTy(IfNormal->getContext()),
+                     Instruction::CatchPad,
                      OperandTraits<CatchPadInst>::op_end(this) - Values, Values,
                      InsertBefore) {
   init(IfNormal, IfException, Args, NameStr);
 }
 
-CatchPadInst::CatchPadInst(Type *RetTy, BasicBlock *IfNormal,
-                           BasicBlock *IfException, ArrayRef<Value *> Args,
-                           unsigned Values, const Twine &NameStr,
-                           BasicBlock *InsertAtEnd)
-    : TerminatorInst(RetTy, Instruction::CatchPad,
+CatchPadInst::CatchPadInst(BasicBlock *IfNormal, BasicBlock *IfException,
+                           ArrayRef<Value *> Args, unsigned Values,
+                           const Twine &NameStr, BasicBlock *InsertAtEnd)
+    : TerminatorInst(Type::getTokenTy(IfNormal->getContext()),
+                     Instruction::CatchPad,
                      OperandTraits<CatchPadInst>::op_end(this) - Values, Values,
                      InsertAtEnd) {
   init(IfNormal, IfException, Args, NameStr);
 }
 
 BasicBlock *CatchPadInst::getSuccessorV(unsigned Idx) const {
   return getSuccessor(Idx);
 }

 CleanupPadInst::CleanupPadInst(const CleanupPadInst &CPI)
     : Instruction(CPI.getType(), Instruction::CleanupPad,
                   OperandTraits<CleanupPadInst>::op_end(this) -
                       CPI.getNumOperands(),
                   CPI.getNumOperands()) {
   std::copy(CPI.op_begin(), CPI.op_end(), op_begin());
 }
 
-CleanupPadInst::CleanupPadInst(Type *RetTy, ArrayRef<Value *> Args,
+CleanupPadInst::CleanupPadInst(LLVMContext &C, ArrayRef<Value *> Args,
                                const Twine &NameStr, Instruction *InsertBefore)
-    : Instruction(RetTy, Instruction::CleanupPad,
+    : Instruction(Type::getTokenTy(C), Instruction::CleanupPad,
                   OperandTraits<CleanupPadInst>::op_end(this) - Args.size(),
                   Args.size(), InsertBefore) {
   init(Args, NameStr);
 }
 
-CleanupPadInst::CleanupPadInst(Type *RetTy, ArrayRef<Value *> Args,
+CleanupPadInst::CleanupPadInst(LLVMContext &C, ArrayRef<Value *> Args,
                                const Twine &NameStr, BasicBlock *InsertAtEnd)
-    : Instruction(RetTy, Instruction::CleanupPad,
+    : Instruction(Type::getTokenTy(C), Instruction::CleanupPad,
                   OperandTraits<CleanupPadInst>::op_end(this) - Args.size(),
                   Args.size(), InsertAtEnd) {
   init(Args, NameStr);
 }
 
 //===----------------------------------------------------------------------===//
 //                      UnreachableInst Implementation
 //===----------------------------------------------------------------------===//

lib/IR/Verifier.cpp

class Verifier : public InstVisitor<Verifier>, VerifierSupport {
   SmallPtrSet<Instruction *, 16> InstsInThisBlock;
 
   /// \brief Keep track of the metadata nodes that have been checked already.
   SmallPtrSet<const Metadata *, 32> MDNodes;
 
   /// \brief Track unresolved string-based type references.
   SmallDenseMap<const MDString *, const MDNode *, 32> UnresolvedTypeRefs;
 
-  /// \brief The result type for a catchpad.
-  Type *CatchPadResultTy;
-
-  /// \brief The result type for a cleanuppad.
-  Type *CleanupPadResultTy;
-
   /// \brief The result type for a landingpad.
   Type *LandingPadResultTy;
 
   /// \brief Whether we've seen a call to @llvm.localescape in this function
   /// already.
   bool SawFrameEscape;
 
   /// Stores the count of how many objects were passed to llvm.localescape for a
   /// given function and the largest index passed to llvm.localrecover.
   DenseMap<Function *, std::pair<unsigned, unsigned>> FrameEscapeInfo;
 
 public:
   explicit Verifier(raw_ostream &OS)
-      : VerifierSupport(OS), Context(nullptr), CatchPadResultTy(nullptr),
-        CleanupPadResultTy(nullptr), LandingPadResultTy(nullptr),
+      : VerifierSupport(OS), Context(nullptr), LandingPadResultTy(nullptr),
         SawFrameEscape(false) {}
 
   bool verify(const Function &F) {
     M = F.getParent();
     Context = &M->getContext();
 
     // First ensure the function is well-enough formed to compute dominance
     // information.
bool verify(const Function &F) {
     // run this code outside of a pass manager.
     // FIXME: It's really gross that we have to cast away constness here.
     DT.recalculate(const_cast<Function &>(F));
 
     Broken = false;
     // FIXME: We strip const here because the inst visitor strips const.
     visit(const_cast<Function &>(F));
     InstsInThisBlock.clear();
-    CatchPadResultTy = nullptr;
-    CleanupPadResultTy = nullptr;
     LandingPadResultTy = nullptr;
     SawFrameEscape = false;
 
     return !Broken;
   }
 
   bool verify(const Module &M) {
     this->M = &M;
void Verifier::visitLandingPadInst(LandingPadInst &LPI) {
   }
 
   visitInstruction(LPI);
 }
 
 void Verifier::visitCatchPadInst(CatchPadInst &CPI) {
   visitEHPadPredecessors(CPI);
 
-  if (!CatchPadResultTy)
-    CatchPadResultTy = CPI.getType();
-  else
-    Assert(CatchPadResultTy == CPI.getType(),
-           "The catchpad instruction should have a consistent result type "
-           "inside a function.",
-           &CPI);
-
   BasicBlock *BB = CPI.getParent();
   Function *F = BB->getParent();
   Assert(F->hasPersonalityFn(),
          "CatchPadInst needs to be in a function with a personality.", &CPI);
 
   // The catchpad instruction must be the first non-PHI instruction in the
   // block.
   Assert(BB->getFirstNonPHI() == &CPI,
          "CatchPadInst not the first non-PHI instruction in the block.",
          &CPI);
 
+  if (!BB->getSinglePredecessor())
+    for (BasicBlock *PredBB : predecessors(BB)) {
+      Assert(!isa<CatchPadInst>(PredBB->getTerminator()),
+             "CatchPadInst with CatchPadInst predecessor cannot have any other "
+             "predecessors.",
+             &CPI);
+    }
+
   BasicBlock *UnwindDest = CPI.getUnwindDest();
   Instruction *I = UnwindDest->getFirstNonPHI();
   Assert(
       isa<CatchPadInst>(I) || isa<CatchEndPadInst>(I),
       "CatchPadInst must unwind to a CatchPadInst or a CatchEndPadInst.",
       &CPI);
 
   visitTerminatorInst(CPI);
void Verifier::visitCatchEndPadInst(CatchEndPadInst &CEPI) {
   visitTerminatorInst(CEPI);
 }
 
 void Verifier::visitCleanupPadInst(CleanupPadInst &CPI) {
   visitEHPadPredecessors(CPI);
 
   BasicBlock *BB = CPI.getParent();
 
-  if (!CleanupPadResultTy)
-    CleanupPadResultTy = CPI.getType();
-  else
-    Assert(CleanupPadResultTy == CPI.getType(),
-           "The cleanuppad instruction should have a consistent result type "
-           "inside a function.",
-           &CPI);
-
   Function *F = BB->getParent();
   Assert(F->hasPersonalityFn(),
          "CleanupPadInst needs to be in a function with a personality.", &CPI);
 
   // The cleanuppad instruction must be the first non-PHI instruction in the
   // block.
   Assert(BB->getFirstNonPHI() == &CPI,
          "CleanupPadInst not the first non-PHI instruction in the block.",
          &CPI);
 
+  CleanupReturnInst *FirstCRI = nullptr;
+  for (User *U : CPI.users())
+    if (CleanupReturnInst *CRI = dyn_cast<CleanupReturnInst>(U)) {
+      if (!FirstCRI)
+        FirstCRI = CRI;
+      else
+        Assert(CRI->getUnwindDest() == FirstCRI->getUnwindDest(),
+               "Cleanuprets from same cleanuppad have different exceptional "
+               "successors.",
+               FirstCRI, CRI);
+    }
+
   visitInstruction(CPI);
 }
 
 void Verifier::visitCleanupReturnInst(CleanupReturnInst &CRI) {
   if (BasicBlock *UnwindDest = CRI.getUnwindDest()) {
     Instruction *I = UnwindDest->getFirstNonPHI();
     Assert(I->isEHPad() && !isa<LandingPadInst>(I),
            "CleanupReturnInst must unwind to an EH block which is not a "

lib/Transforms/Instrumentation/MemorySanitizer.cpp

#undef GET_INTRINSIC_MODREF_BEHAVIOR
   void visitLandingPadInst(LandingPadInst &I) {
     // Do nothing.
     // See http://code.google.com/p/memory-sanitizer/issues/detail?id=1
     setShadow(&I, getCleanShadow(&I));
     setOrigin(&I, getCleanOrigin());
   }
 
   void visitCleanupPadInst(CleanupPadInst &I) {
-    if (!I.getType()->isVoidTy()) {
-      setShadow(&I, getCleanShadow(&I));
-      setOrigin(&I, getCleanOrigin());
-    }
-  }
 
   void visitCatchPad(CatchPadInst &I) {
-    if (!I.getType()->isVoidTy()) {
-      setShadow(&I, getCleanShadow(&I));
-      setOrigin(&I, getCleanOrigin());
-    }
-  }
 
   void visitTerminatePad(TerminatePadInst &I) {
     DEBUG(dbgs() << "TerminatePad: " << I << "\n");
     // Nothing to do here.
   }
 
   void visitCatchEndPadInst(CatchEndPadInst &I) {
     DEBUG(dbgs() << "CatchEndPad: " << I << "\n");

lib/Transforms/Utils/InlineFunction.cpp

if (I->isEHPad()) {
         }
       } else {
         assert(isa<CatchPadInst>(I) || isa<CleanupPadInst>(I));
       }
     }
 
     if (auto *CRI = dyn_cast<CleanupReturnInst>(BB->getTerminator())) {
       if (CRI->unwindsToCaller()) {
-        CleanupReturnInst::Create(CRI->getContext(), CRI->getReturnValue(),
-                                  UnwindDest, CRI);
+        CleanupReturnInst::Create(CRI->getCleanupPad(), UnwindDest, CRI);
         CRI->eraseFromParent();
         UpdatePHINodes(BB);
       }
     }
   }
 
   if (InlinedCodeInfo.ContainsCalls)
     for (Function::iterator BB = FirstNewBlock, E = Caller->end(); BB != E;

test/Assembler/invalid-OperatorConstraint.ll

+; RUN: sed -e s/.T1:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK1 %s
+; RUN: sed -e s/.T2:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK2 %s
+; RUN: sed -e s/.T3:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK3 %s
+; RUN: sed -e s/.T4:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK4 %s
+; RUN: sed -e s/.T5:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK5 %s
+; RUN: sed -e s/.T6:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK6 %s
+
+;T1: define void @f() {
+;T1:   entry:
+;T1:     ; operator constraint requires an operator
+;T1:     catchret undef to label %entry
+;T1:     ; CHECK1: [[@LINE-1]]:15: error: Catchpad value required in this position
+;T1: }
+
+;T2: define void @f() {
+;T2:   entry:
+;T2:     %x = cleanuppad []
+;T2:     ; catchret's first operand's operator must be catchpad
+;T2:     catchret %x to label %entry
+;T2:     ; CHECK2: [[@LINE-1]]:15: error: '%x' is not a catchpad
+;T2: }
+
+;T3: define void @f() {
+;T3:   entry:
+;T3:     ; catchret's first operand's operator must be catchpad
+;T3:     ; (forward reference case)
+;T3:     catchret %x to label %next
+;T3:     ; CHECK3: [[@LINE-1]]:15: error: '%x' is not a catchpad
+;T3:   next:
+;T3:     %x = cleanuppad []
+;T3:     ret void
+;T3: }
+
+;T4: define void @f() {
+;T4:   entry:
+;T4:     ; operator constraint requires an operator
+;T4:     cleanupret undef unwind label %entry
+;T4:     ; CHECK4: [[@LINE-1]]:17: error: Cleanuppad value required in this position
+;T4: }
+
+;T5: define void @f() {
+;T5:   entry:
+;T5:     %x = catchpad []
+;T5:             to label %next unwind label %entry
+;T5:   next:
+;T5:     ; cleanupret first operand's operator must be cleanuppad
+;T5:     cleanupret %x unwind to caller
+;T5:     ; CHECK5: [[@LINE-1]]:17: error: '%x' is not a cleanuppad
+;T5: }
+
+;T6: define void @f() {
+;T6:   entry:
+;T6:     ; cleanupret's first operand's operator must be cleanuppad
+;T6:     ; (forward reference case)
+;T6:     cleanupret %x unwind label %next
+;T6:     ; CHECK6: [[@LINE-1]]:17: error: '%x' is not a cleanuppad
+;T6:   next:
+;T6:     %x = catchpad [] to label %entry unwind label %next
+;T6: }

test/CodeGen/WinEH/wineh-demotion.ll

right:
   ; CHECK-NEXT: store i32 %y, i32* [[Slot]]
   ; CHECK-NEXT: invoke void @f
   invoke void @f()
           to label %exit unwind label %merge
 merge:
   ; CHECK: merge:
   ; CHECK-NOT: = phi
   %phi = phi i32 [ %x, %left ], [ %y, %right ]
-  %cp = catchpad token [] to label %catch unwind label %catchend
+  %cp = catchpad [] to label %catch unwind label %catchend
 
 catch:
   ; CHECK: catch:
   ; CHECK: [[Reload:%[^ ]+]] = load i32, i32* [[Slot]]
   ; CHECK-NEXT: call void @h(i32 [[Reload]])
   call void @h(i32 %phi)
-  catchret token %cp to label %exit
+  catchret %cp to label %exit
 
 catchend:
   catchendpad unwind to caller
 
 exit:
   ret void
 }
 
right:
   ; CHECK-DAG:   store i32 2, i32* [[Slot1]]
   ; CHECK-DAG:   store i32 2, i32* [[Slot2]]
   ; CHECK: invoke void @f
   invoke void @f()
           to label %exit unwind label %merge.inner
 merge.inner:
   ; CHECK: merge.inner:
   ; CHECK-NOT: = phi
-  ; CHECK: catchpad token
+  ; CHECK: catchpad []
   %x = phi i32 [ 1, %left ], [ 2, %right ]
-  %cpinner = catchpad token [] to label %catch.inner unwind label %catchend.inner
+  %cpinner = catchpad [] to label %catch.inner unwind label %catchend.inner
 
 catch.inner:
   ; Need just one store here because only %y is affected
   ; CHECK: catch.inner:
   %z = call i32 @g()
   ; CHECK:   store i32 %z
   ; CHECK-NEXT: invoke void @f
   invoke void @f()
           to label %catchret.inner unwind label %merge.outer
 
 catchret.inner:
-  catchret token %cpinner to label %exit
+  catchret %cpinner to label %exit
 catchend.inner:
   catchendpad unwind label %merge.outer
 
 merge.outer:
   ; CHECK: merge.outer:
   ; CHECK-NOT: = phi
-  ; CHECK: [[CatchPad:%[^ ]+]] = catchpad token
+  ; CHECK: [[CatchPad:%[^ ]+]] = catchpad []
   %y = phi i32 [ %x, %catchend.inner ], [ %z, %catch.inner ]
-  %cpouter = catchpad token [] to label %catch.outer unwind label %catchend.outer
+  %cpouter = catchpad [] to label %catch.outer unwind label %catchend.outer
 
 catchend.outer:
   catchendpad unwind to caller
 
 catch.outer:
   ; Need to load x and y from two different slots since they're both live
   ; and can have different values (if we came from catch.inner)
   ; CHECK: catch.outer:
   ; CHECK-DAG: load i32, i32* [[Slot1]]
   ; CHECK-DAG: load i32, i32* [[Slot2]]
-  ; CHECK: catchret token [[CatchPad]] to label
+  ; CHECK: catchret [[CatchPad]] to label
   call void @h(i32 %x)
   call void @h(i32 %y)
-  catchret token %cpouter to label %exit
+  catchret %cpouter to label %exit
 
 exit:
   ret void
 }
 
 ; CHECK-LABEL: @test3(
 define void @test3(i1 %B) personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; need to spill parameter %B and def %x since they're used in a funclet
   ; CHECK: entry:
   ; CHECK-DAG: store i1 %B, i1* [[SlotB:%[^ ]+]]
   ; CHECK-DAG: store i32 %x, i32* [[SlotX:%[^ ]+]]
   ; CHECK: invoke void @f
   %x = call i32 @g()
   invoke void @f()
           to label %exit unwind label %catchpad
 
 catchpad:
-  %cp = catchpad token [] to label %catch unwind label %catchend
+  %cp = catchpad [] to label %catch unwind label %catchend
 
 catch:
   ; Need to reload %B here
   ; CHECK: catch:
   ; CHECK: [[ReloadB:%[^ ]+]] = load i1, i1* [[SlotB]]
   ; CHECK: br i1 [[ReloadB]]
   br i1 %B, label %left, label %right
 left:
   ; Use of %x is in a phi, so need reload here in pred
   ; CHECK: left:
   ; CHECK: [[ReloadX:%[^ ]+]] = load i32, i32* [[SlotX]]
   ; CHECK: br label %merge
   br label %merge
 right:
   br label %merge
 merge:
   ; CHECK: merge:
   ; CHECK:   %phi = phi i32 [ [[ReloadX]], %left ]
   %phi = phi i32 [ %x, %left ], [ 42, %right ]
   call void @h(i32 %phi)
-  catchret token %cp to label %exit
+  catchret %cp to label %exit
 
 catchend:
   catchendpad unwind to caller
 
 exit:
   ret void
 }
 
right:
   ; CHECK-NOT: store
   ; CHECK: store i32 %r, i32* [[Slot]]
   ; CHECK-NEXT: invoke void @f
   %r = call i32 @g()
   invoke void @f()
           to label %join unwind label %catchpad.inner
 catchpad.inner:
    ; CHECK: catchpad.inner:
-   ; CHECK-NEXT: catchpad token
+   ; CHECK-NEXT: catchpad []
    %phi.inner = phi i32 [ %l, %left ], [ %r, %right ]
-   %cp1 = catchpad token [] to label %catch.inner unwind label %catchend.inner
+   %cp1 = catchpad [] to label %catch.inner unwind label %catchend.inner
 catch.inner:
-   catchret token %cp1 to label %join
+   catchret %cp1 to label %join
 catchend.inner:
    catchendpad unwind label  %catchpad.outer
 join:
   ; CHECK: join:
   ; CHECK-NOT: store
   ; CHECK: store i32 %j, i32* [[Slot]]
   ; CHECK-NEXT: invoke void @f
    %j = call i32 @g()
    invoke void @f()
            to label %exit unwind label %catchpad.outer
 catchpad.outer:
    ; CHECK: catchpad.outer:
-   ; CHECK-NEXT: catchpad token
+   ; CHECK-NEXT: catchpad []
    %phi.outer = phi i32 [ %phi.inner, %catchend.inner ], [ %j, %join ]
-   %cp2 = catchpad token [] to label %catch.outer unwind label %catchend.outer
+   %cp2 = catchpad [] to label %catch.outer unwind label %catchend.outer
 catch.outer:
    ; CHECK: catch.outer:
    ; CHECK:   [[Reload:%[^ ]+]] = load i32, i32* [[Slot]]
    ; CHECK:   call void @h(i32 [[Reload]])
    call void @h(i32 %phi.outer)
-   catchret token %cp2 to label %exit
+   catchret %cp2 to label %exit
 catchend.outer:
    catchendpad unwind to caller
 exit:
    ret void
 }
 
 ; CHECK-LABEL: @test5(
 define void @test5() personality i32 (...)* @__CxxFrameHandler3 {
invoke.cont:
   ; CHECK-NEXT:   store i32 2, i32* [[CleanupSlot]]
   ; CHECK-NEXT:   invoke void @f
   invoke void @f()
           to label %invoke.cont2 unwind label %cleanup
 
 cleanup:
   ; cleanup phi can be loaded at cleanup entry
   ; CHECK: cleanup:
-  ; CHECK-NEXT: cleanuppad token
+  ; CHECK-NEXT: cleanuppad []
   ; CHECK: [[CleanupReload:%[^ ]+]] = load i32, i32* [[CleanupSlot]]
   %phi.cleanup = phi i32 [ 1, %entry ], [ 2, %invoke.cont ]
-  %cp = cleanuppad token []
+  %cp = cleanuppad []
   %b = call i1 @i()
   br i1 %b, label %left, label %right
 
 left:
   ; CHECK: left:
   ; CHECK:   call void @h(i32 [[CleanupReload]]
   call void @h(i32 %phi.cleanup)
   br label %merge
 
 right:
   ; CHECK: right:
   ; CHECK:   call void @h(i32 [[CleanupReload]]
   call void @h(i32 %phi.cleanup)
   br label %merge
 
 merge:
   ; need store for %phi.catch
   ; CHECK:      merge:
   ; CHECK-NEXT:   store i32 [[CleanupReload]], i32* [[CatchSlot:%[^ ]+]]
-  ; CHECK-NEXT:   cleanupret token
-  cleanupret token %cp unwind label %catchpad
+  ; CHECK-NEXT:   cleanupret
+  cleanupret %cp unwind label %catchpad
 
 invoke.cont2:
   ; need store for %phi.catch
   ; CHECK:      invoke.cont2:
   ; CHECK-NEXT:   store i32 3, i32* [[CatchSlot]]
   ; CHECK-NEXT:   invoke void @f
   invoke void @f()
           to label %exit unwind label %catchpad
 
 catchpad:
   ; CHECK: catchpad:
-  ; CHECK-NEXT: catchpad token
+  ; CHECK-NEXT: catchpad []
   %phi.catch = phi i32 [ %phi.cleanup, %merge ], [ 3, %invoke.cont2 ]
-  %cp2 = catchpad token [] to label %catch unwind label %catchend
+  %cp2 = catchpad [] to label %catch unwind label %catchend
 
 catch:
   ; CHECK: catch:
   ; CHECK:   [[CatchReload:%[^ ]+]] = load i32, i32* [[CatchSlot]]
   ; CHECK:   call void @h(i32 [[CatchReload]]
   call void @h(i32 %phi.catch)
-  catchret token %cp2 to label %exit
+  catchret %cp2 to label %exit
 
 catchend:
   catchendpad unwind to caller
 
 exit:
   ret void
 }
 
 ; CHECK-LABEL: @test6(
 define void @test6() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; Since %x needs to be stored but the edge to loop is critical,
   ; it needs to be split
   ; CHECK: entry:
   ; CHECK: invoke i32 @g
   ; CHECK-NEXT: to label %[[SplitBlock:[^ ]+]] unwind label %to_caller
   %x = invoke i32 @g()
           to label %loop unwind label %to_caller
   ; The store should be in the split block
   ; CHECK: [[SplitBlock]]:
   ; CHECK: store i32 %x, i32* [[SpillSlot:%[^ ]+]]
   ; CHECK: br label %loop
 to_caller:
-  %cp1 = cleanuppad token []
-  cleanupret token %cp1 unwind to caller
+  %cp1 = cleanuppad []
+  cleanupret %cp1 unwind to caller
 loop:
   invoke void @f()
           to label %loop unwind label %cleanup
 cleanup:
   ; CHECK: cleanup:
   ; CHECK:   [[Load:%[^ ]+]] = load i32, i32* [[SpillSlot]]
   ; CHECK:   call void @h(i32 [[Load]])
-  %cp2 = cleanuppad token []
+  %cp2 = cleanuppad []
   call void @h(i32 %x)
-  cleanupret token %cp2 unwind to caller
+  cleanupret %cp2 unwind to caller
 }
 
 ; CHECK-LABEL: @test7(
 define void @test7() personality i32 (...)* @__CxxFrameHandler3 {
 entry:
   ; %x is an EH pad phi, so gets stored in pred here
   ; CHECK: entry:
   ; CHECK:   store i32 1, i32* [[SlotX:%[^ ]+]]
   ; CHECK:   invoke void @f()
   invoke void @f()
      to label %invoke.cont unwind label %catchpad
 invoke.cont:
   ; %x is an EH pad phi, so gets stored in pred here
   ; CHECK: invoke.cont:
   ; CHECK:   store i32 2, i32* [[SlotX]]
   ; CHECK:   invoke void @f()
   invoke void @f()
     to label %exit unwind label %catchpad
 catchpad:
   ; %x phi should be eliminated
   ; CHECK: catchpad:
-  ; CHECK-NEXT: %[[CatchPad:[^ ]+]] = catchpad token
+  ; CHECK-NEXT: %[[CatchPad:[^ ]+]] = catchpad []
   %x = phi i32 [ 1, %entry ], [ 2, %invoke.cont ]
-  %cp = catchpad token [] to label %catch unwind label %catchend
+  %cp = catchpad [] to label %catch unwind label %catchend
 catch:
   %b = call i1 @i()
   br i1 %b, label %left, label %right
 left:
   ; Edge from %left to %join needs to be split so that
   ; the load of %x can be inserted *after* the catchret
   ; CHECK: left:
-  ; CHECK-NEXT: catchret token %[[CatchPad]] to label %[[SplitLeft:[^ ]+]]
-  catchret token %cp to label %join
+  ; CHECK-NEXT: catchret %[[CatchPad]] to label %[[SplitLeft:[^ ]+]]
+  catchret %cp to label %join
   ; CHECK: [[SplitLeft]]:
   ; CHECK:   [[LoadX:%[^ ]+]] = load i32, i32* [[SlotX]]
   ; CHECK:   br label %join
 right:
   ; Edge from %right to %join needs to be split so that
   ; the load of %y can be inserted *after* the catchret
   ; CHECK: right:
   ; CHECK:   store i32 %y, i32* [[SlotY:%[^ ]+]]
-  ; CHECK:   catchret token %[[CatchPad]] to label %[[SplitRight:[^ ]+]]
+  ; CHECK:   catchret %[[CatchPad]] to label %[[SplitRight:[^ ]+]]
   %y = call i32 @g()
-  catchret token %cp to label %join
+  catchret %cp to label %join
   ; CHECK: [[SplitRight]]:
   ; CHECK:   [[LoadY:%[^ ]+]] = load i32, i32* [[SlotY]]
   ; CHECK:   br label %join
 catchend:
   catchendpad unwind to caller
 join:
   ; CHECK: join:
   ; CHECK:   %phi = phi i32 [ [[LoadX]], %[[SplitLeft]] ], [ [[LoadY]], %[[SplitRight]] ]
invoke void @f()
           to label %done unwind label %cleanup1
   invoke void @f()
           to label %done unwind label %cleanup2
 
 done:
   ret void
 
 cleanup1:
-  ; CHECK: [[CleanupPad1:%[^ ]+]] = cleanuppad token
+  ; CHECK: [[CleanupPad1:%[^ ]+]] = cleanuppad []
   ; CHECK-NEXT: call void @f()
-  ; CHECK-NEXT: cleanupret token [[CleanupPad1]]
-  %cp0 = cleanuppad token []
+  ; CHECK-NEXT: cleanupret [[CleanupPad1]]
+  %cp0 = cleanuppad []
   br label %cleanupexit
 
 cleanup2:
-  ; CHECK: cleanuppad token
+  ; CHECK: cleanuppad []
   ; CHECK-NEXT: call void @f()
   ; CHECK-NEXT: unreachable
-  %cp1 = cleanuppad token []
+  %cp1 = cleanuppad []
   br label %cleanupexit
 
 cleanupexit:
   call void @f()
-  cleanupret token %cp0 unwind label %cleanup2
+  cleanupret %cp0 unwind label %cleanup2
 }

test/CodeGen/WinEH/wineh-statenumbering.ll

entry:
   store i32 0, i32* %tmp, align 4
   %0 = bitcast i32* %tmp to i8*
   ; CHECK:   store i32 0
   ; CHECK:   invoke void @_CxxThrowException(
   invoke void @_CxxThrowException(i8* %0, %eh.ThrowInfo* nonnull @_TI1H) #1
           to label %unreachable.for.entry unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
-  %1 = catchpad token [i8* null, i8* null] to label %catch unwind label %catchendblock
+  %1 = catchpad [i8* null, i8* null] to label %catch unwind label %catchendblock
 
 catch:                                            ; preds = %catch.dispatch
   ; CHECK: catch:
   ; CHECK:   store i32 2
   ; CHECK:   invoke void @_CxxThrowException(
   invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null) #1
           to label %unreachable unwind label %catch.dispatch.1
 
 catch.dispatch.1:                                 ; preds = %catch
-  %2 = catchpad token [i8* null, i8* null] to label %catch.3 unwind label %catchendblock.2
+  %2 = catchpad [i8* null, i8* null] to label %catch.3 unwind label %catchendblock.2
 
 catch.3:                                          ; preds = %catch.dispatch.1
   ; CHECK: catch.3:
   ; CHECK:   store i32 3
   ; CHECK:   invoke void @g(i32 1)
   invoke void @g(i32 1)
           to label %invoke.cont unwind label %catchendblock.2
 
 invoke.cont:                                      ; preds = %catch.3
-  catchret token %2 to label %try.cont
+  catchret %2 to label %try.cont
 
 try.cont:                                         ; preds = %invoke.cont
   ; CHECK: try.cont:
   ; CHECK:   store i32 1
   ; CHECK:   invoke void @g(i32 2)
   invoke void @g(i32 2)
           to label %invoke.cont.4 unwind label %catchendblock
 

test/Feature/exception.ll

 }
 
 declare void @_Z3quxv() optsize
 
 declare i32 @__gxx_personality_v0(...)
 
 define void @cleanupret0() personality i32 (...)* @__gxx_personality_v0 {
 entry:
-  br label %try.cont
-
-try.cont:
   invoke void @_Z3quxv() optsize
-          to label %try.cont unwind label %bb
-bb:
-  cleanuppad void [i7 4]
-  cleanupret i8 0 unwind label %bb
+          to label %exit unwind label %pad
+pad:
+  %cp = cleanuppad [i7 4]
+  cleanupret %cp unwind to caller
+exit:
+  ret void
 }
 
+; forward ref by name
 define void @cleanupret1() personality i32 (...)* @__gxx_personality_v0 {
 entry:
-  br label %try.cont
-
-try.cont:
   invoke void @_Z3quxv() optsize
-          to label %try.cont unwind label %bb
-bb:
-  cleanuppad void [i7 4]
-  cleanupret void unwind label %bb
+          to label %exit unwind label %pad
+cleanup:
+  cleanupret %cp unwind label %pad
+pad:
+  %cp = cleanuppad []
+  br label %cleanup
+exit:
+  ret void
 }
 
+; forward ref by ID
 define void @cleanupret2() personality i32 (...)* @__gxx_personality_v0 {
 entry:
-  cleanupret i8 0 unwind to caller
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+cleanup:
+  cleanupret %0 unwind label %pad
+pad:
+  %0 = cleanuppad []
+  br label %cleanup
+exit:
+  ret void
 }
 
-define void @cleanupret3() personality i32 (...)* @__gxx_personality_v0 {
-  cleanupret void unwind to caller
+define void @catchret0() personality i32 (...)* @__gxx_personality_v0 {
+entry:
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+pad:
+  %cp = catchpad [i7 4]
+          to label %catch unwind label %endpad
+catch:
+  catchret %cp to label %exit
+endpad:
+  catchendpad unwind to caller
+exit:
+  ret void
 }
 
-define void @catchret() personality i32 (...)* @__gxx_personality_v0 {
+; forward ref by name
+define void @catchret1() personality i32 (...)* @__gxx_personality_v0 {
 entry:
-  br label %bb
-bb:
-  catchret void to label %bb
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+catch:
+  catchret %cp to label %exit
+pad:
+  %cp = catchpad []
+          to label %catch unwind label %endpad
+endpad:
+  catchendpad unwind to caller
+exit:
+  ret void
 }
 
-define i8 @catchpad() personality i32 (...)* @__gxx_personality_v0 {
+; forward ref by ID
+define void @catchret2() personality i32 (...)* @__gxx_personality_v0 {
 entry:
-  br label %try.cont
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+catch:
+  catchret %0 to label %exit
+pad:
+  %0 = catchpad []
+          to label %catch unwind label %endpad
+endpad:
+  catchendpad unwind to caller
+exit:
+  ret void
+}
 
-try.cont:
+define i8 @catchpad() personality i32 (...)* @__gxx_personality_v0 {
+entry:
   invoke void @_Z3quxv() optsize
           to label %exit unwind label %bb2
-bb:
-  catchret token %cbv to label %exit
-
-exit:
-  ret i8 0
 bb2:
-  %cbv = catchpad token [i7 4] to label %bb unwind label %bb3
+  catchpad [i7 4] to label %exit unwind label %bb3
 bb3:
   catchendpad unwind to caller
+exit:
+  ret i8 0
 }
 
 define void @terminatepad0() personality i32 (...)* @__gxx_personality_v0 {
 entry:
   br label %try.cont
 
 try.cont:
   invoke void @_Z3quxv() optsize

 define void @cleanuppad() personality i32 (...)* @__gxx_personality_v0 {
 entry:
   br label %try.cont
 
 try.cont:
   invoke void @_Z3quxv() optsize
           to label %try.cont unwind label %bb
 bb:
-  cleanuppad void [i7 4]
+  cleanuppad [i7 4]
   ret void
 }
 
 define void @catchendpad0() personality i32 (...)* @__gxx_personality_v0 {
 entry:
   br label %try.cont
 
 try.cont: