[WinEH] Add cleanupendpad instruction

[JosephTremoulet]

Summary

Add a cleanupendpad instruction, used to mark exceptional exits out of
cleanups (for languages/targets that can abort a cleanup with another
exception). The cleanupendpad instruction is similar to the catchendpad
instruction in that it is an EH pad which is the target of unwind edges in
the handler and which itself has an unwind edge to the next EH action.
The cleanupendpad instruction, similar to cleanupret has a cleanuppad
argument indicating which cleanup it exits. The unwind successors of a
cleanuppad's cleanupendpads must agree with each other and with its
cleanuprets.

Update WinEHPrepare (and docs/tests) to accomodate cleanupendpad.

[JosephTremoulet]

lib/CodeGen/WinEHPrepare.cpp

3308–3313

This part is likely wrong or incomplete; I have some questions about how to proceed that I felt were best asked in the context of a code review:

1. As written, this will leave an unreachable as the first instruction in a block that's the target of unwind edges, which doesn't verify. I can see that if we were to run SimplifyCFGOpt::SimplifyUnreachable on it, it would rewrite any invoke predecessors as calls. Does that mean we need to run SimplifyCFGOpt on the function?
2. Should SimplifyCFGOpt::SimplifyUnreachable be extended to rewrite catchendpad/cleanupendpad/terminatepad/cleanupret predecessors as 'unwind to caller'? Does it matter that doing so would make those predecessors 'mayThrow()' where they weren't before? From the comment "This would be a good place to note that the call does not throw though" in SimplifyUnreachable, I'm guessing the same happens for the invoke->call rewrite so it's ok (or at least won't make things worse), but not sure I'm intepreting that correctly.
3. I think we'll need to do something similar for unwind edges that target implausible catchendpads, but since the plausiblity check there is 'is this catchendpad in the parent funclet', I think that will need to happen after the treeifying duplication that makes "*the* parent funclet" well-defined. [I guess this one was a comment, not a question.]

[JosephTremoulet]

• rebase
• update mergefunc test (changing the numeric opcode values changed its output order)

[JosephTremoulet]

lib/CodeGen/WinEHPrepare.cpp

3308–3314

Seems like the right thing to do here will be to share the utility discussed in D12434

[majnemer]

This LGTM modulo the change to "Remove implausible terminators and replace them with UnreachableInst". I think that the right thing to do there is, as mentioned, to update the cleanupendpad's predecessor to unwind to caller.

docs/ExceptionHandling.rst

 frontend encounters a call site that may throw an exception, it should emit an
 invoke that unwinds to a ``catchpad`` block. Similarly, inside the scope of a
 C++ object with a destructor, invokes should unwind to a ``cleanuppad``. The
 ``terminatepad`` instruction exists to represent ``noexcept`` and throw
 specifications with one combined instruction. All potentially throwing calls in
 a ``noexcept`` function should transitively unwind to a terminateblock. Throw
 specifications are not implemented by MSVC, and are not yet supported.
 
+New instructions are also used to mark the points where control is transferred
+out of a catch/cleanup handler (which will correspond to exits from the
+generated funclet).  A catch handler which reaches its end by normal execution
+executes a ``catchret`` instruction, which is a terminator indicating where in
+the function control is returned to.  A cleanup handler which reaches its end
+by normal execution executes a ``cleanupret`` instruction, which is a terminator
+indicating where the active exception will unwind to next.  A catch or cleanup
+handler which is exited by another exception being raised during its execution will
+unwind through a ``catchendpad`` or ``cleanuupendpad`` (respectively).  The
+``catchendpad`` and ``cleanupendpad`` instructions are considered "exception
+handling pads" in the same sense that ``catchpad``, ``cleanuppad``, and
+``terminatepad`` are.
+
 Each of these new EH pad instructions has a way to identify which
 action should be considered after this action. The ``catchpad`` and
 ``terminatepad`` instructions are terminators, and have a label operand considered
 to be an unwind destination analogous to the unwind destination of an invoke. The
 ``cleanuppad`` instruction is different from the other two in that it is not a
 terminator. The code inside a cleanuppad runs before transferring control to the
-next action, so the ``cleanupret`` instruction is the instruction that holds a
-label operand and unwinds to the next EH pad. All of these "unwind edges" may
-refer to a basic block that contains an EH pad instruction, or they may simply
-unwind to the caller. Unwinding to the caller has roughly the same semantics as
-the ``resume`` instruction in the ``landingpad`` model. When inlining through an
-invoke, instructions that unwind to the caller are hooked up to unwind to the
-unwind destination of the call site.
+next action, so the ``cleanupret`` and ``cleanupendpad`` instructions are the
+instructions that hold a label operand and unwind to the next EH pad. All of
+these "unwind edges" may refer to a basic block that contains an EH pad instruction,
+or they may simply unwind to the caller. Unwinding to the caller has roughly the
+same semantics as the ``resume`` instruction in the ``landingpad`` model. When
+inlining through an invoke, instructions that unwind to the caller are hooked
+up to unwind to the unwind destination of the call site.
 
 Putting things together, here is a hypothetical lowering of some C++ that uses
 all of the new IR instructions:
 
 .. code-block:: c
 
   struct Cleanup {
     Cleanup();
     ~Cleanup();
     int m;
   };
   void may_throw();
   int f() noexcept {
     try {
       Cleanup obj;
       may_throw();
     } catch (int e) {
+      may_throw();
       return e;
     }
     return 0;
   }
 
 .. code-block:: llvm
 
   define i32 @f() nounwind personality i32 (...)* @__CxxFrameHandler3 {
   entry:
     %obj = alloca %struct.Cleanup, align 4
     %e = alloca i32, align 4
     %call = invoke %struct.Cleanup* @"\01??0Cleanup@@QEAA@XZ"(%struct.Cleanup* nonnull %obj)
             to label %invoke.cont unwind label %lpad.catch
 
   invoke.cont:                                      ; preds = %entry
     invoke void @"\01?may_throw@@YAXXZ"()
             to label %invoke.cont.2 unwind label %lpad.cleanup
 
   invoke.cont.2:                                    ; preds = %invoke.cont
     call void @"\01??_DCleanup@@QEAA@XZ"(%struct.Cleanup* nonnull %obj) nounwind
     br label %return
 
-  return:                                           ; preds = %invoke.cont.2, %catch
+  return:                                           ; preds = %invoke.cont.2, %invoke.cont.3
     %retval.0 = phi i32 [ 0, %invoke.cont.2 ], [ %9, %catch ]
     ret i32 %retval.0
 
   ; EH scope code, ordered innermost to outermost:
 
   lpad.cleanup:                                     ; preds = %invoke.cont
     %cleanup = cleanuppad []
     call void @"\01??_DCleanup@@QEAA@XZ"(%struct.Cleanup* nonnull %obj) nounwind
     cleanupret %cleanup unwind label %lpad.catch
 
   lpad.catch:                                       ; preds = %entry, %lpad.cleanup
     %catch = catchpad [%rtti.TypeDescriptor2* @"\01??_R0H@8", i32 0, i32* %e]
-            to label %catch unwind label %lpad.terminate
+            to label %catch.body unwind label %catchend
 
-  catch:                                            ; preds = %lpad.catch
+  catch.body:                                       ; preds = %lpad.catch
+    invoke void @"\01?may_throw@@YAXXZ"()
+            to label %invoke.cont.3 unwind label %catchend
+
+  invoke.cont.3:                                    ; preds = %catch.body
     %9 = load i32, i32* %e, align 4
     catchret %catch label %return
 
-  lpad.terminate:
+  catchend:                                         ; preds = %lpad.catch, %catch.body
+    catchendpad unwind label %lpad.terminate
+
+  lpad.terminate:                                   ; preds = %catchend
     terminatepad [void ()* @"\01?terminate@@YAXXZ"]
             unwind to caller
   }

docs/LangRef.rst

 ':ref:`invoke <i_invoke>`' instruction).
 
 The terminator instructions are: ':ref:`ret <i_ret>`',
 ':ref:`br <i_br>`', ':ref:`switch <i_switch>`',
 ':ref:`indirectbr <i_indirectbr>`', ':ref:`invoke <i_invoke>`',
 ':ref:`resume <i_resume>`', ':ref:`catchpad <i_catchpad>`',
 ':ref:`catchendpad <i_catchendpad>`',
 ':ref:`catchret <i_catchret>`',
+':ref:`cleanupendpad <i_cleanupendpad>`',
 ':ref:`cleanupret <i_cleanupret>`',
 ':ref:`terminatepad <i_terminatepad>`',
 and ':ref:`unreachable <i_unreachable>`'.
 
 .. _i_ret:
 
 '``ret``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^
::
       catchendpad unwind to caller
 
 Overview:
 """""""""
 
 The '``catchendpad``' instruction is used by `LLVM's exception handling
 system <ExceptionHandling.html#overview>`_ to communicate to the
 :ref:`personality function <personalityfn>` which invokes are associated
-with a chain of :ref:`catchpad <i_catchpad>` instructions.
+with a chain of :ref:`catchpad <i_catchpad>` instructions; propagating an
+exception out of a catch handler is represented by unwinding through its
+``catchendpad``.  Unwinding to the outer scope when a chain of catch handlers
+do not handle an exception is also represented by unwinding through their
+``catchendpad``.
 
 The ``nextaction`` label indicates where control should transfer to if
 none of the ``catchpad`` instructions are suitable for catching the
 in-flight exception.
 
 If a ``nextaction`` label is not present, the instruction unwinds out of
 its parent function. The
 :ref:`personality function <personalityfn>` will continue processing

 
 The ``catchendpad`` instruction has several restrictions:
 
 -  A catch-end block is a basic block which is the unwind destination of
    an exceptional instruction.
 -  A catch-end block must have a '``catchendpad``' instruction as its
    first non-PHI instruction.
 -  There can be only one '``catchendpad``' instruction within the
-   catch block.
+   catch-end block.
 -  A basic block that is not a catch-end block may not include a
    '``catchendpad``' instruction.
 -  Exactly one catch block may unwind to a ``catchendpad``.
--  The unwind target of invokes between a ``catchpad`` and a
-   corresponding ``catchret`` must be its ``catchendpad`` or
-   an inner EH pad.
+- It is undefined behavior to execute a ``catchendpad`` if none of the
+  '``catchpad``'s chained to it have been executed.
+- It is undefined behavior to execute a ``catchendpad`` twice without an
+  intervening execution of one or more of the '``catchpad``'s chained to it.
+- It is undefined behavior to execute a ``catchendpad`` if, after the most
+  recent execution of the normal successor edge of any ``catchpad`` chained
+  to it, some ``catchret`` consuming that ``catchpad`` has already been
+  executed.
+- It is undefined behavior to execute a ``catchendpad`` if, after the most
+  recent execution of the normal successor edge of any ``catchpad`` chained
+  to it, any other ``catchpad`` or ``cleanuppad`` has been executed but has
+  not had a corresponding
+  ``catchret``/``cleanupret``/``catchendpad``/``cleanupendpad`` executed.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       catchendpad unwind label %terminate
       catchendpad unwind to caller

 
 The '``catchret``' instruction ends the existing (in-flight) exception
 whose unwinding was interrupted with a
 :ref:`catchpad <i_catchpad>` instruction.
 The :ref:`personality function <personalityfn>` gets a chance to execute
 arbitrary code to, for example, run a C++ destructor.
 Control then transfers to ``normal``.
 It may be passed an optional, personality specific, value.
+
 It is undefined behavior to execute a ``catchret`` whose ``catchpad`` has
 not been executed.
-It is undefined behavior to execute a ``catchret`` if any ``catchpad`` or
-``cleanuppad`` has been executed, without subsequently executing a
-corresponding ``catchret``/``cleanupret`` or unwinding out of the inner
-pad, following the most recent execution of the ``catchret``'s corresponding
-``catchpad``.
 
+It is undefined behavior to execute a ``catchret`` if, after the most recent
+execution of its ``catchpad``, some ``catchret`` or ``catchendpad`` linked
+to the same ``catchpad`` has already been executed.
+
+It is undefined behavior to execute a ``catchret`` if, after the most recent
+execution of its ``catchpad``, any other ``catchpad`` or ``cleanuppad`` has
+been executed but has not had a corresponding
+``catchret``/``cleanupret``/``catchendpad``/``cleanupendpad`` executed.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       catchret %catch label %continue
 
+.. _i_cleanupendpad:
+
+'``cleanupendpad``' Instruction
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+
+::
+
+      cleanupendpad <value> unwind label <nextaction>
+      cleanupendpad <value> unwind to caller
+
+Overview:
+"""""""""
+
+The '``cleanupendpad``' instruction is used by `LLVM's exception handling
+system <ExceptionHandling.html#overview>`_ to communicate to the
+:ref:`personality function <personalityfn>` which invokes are associated
+with a :ref:`cleanuppad <i_cleanuppad>` instructions; propagating an exception
+out of a cleanup is represented by unwinding through its ``cleanupendpad``.
+
+The ``nextaction`` label indicates where control should unwind to next, in the
+event that a cleanup is exited by means of an(other) exception being raised.
+
+If a ``nextaction`` label is not present, the instruction unwinds out of
+its parent function. The
+:ref:`personality function <personalityfn>` will continue processing
+exception handling actions in the caller.
+
+Arguments:
+""""""""""
+
+The '``cleanupendpad``' instruction requires one argument, which indicates
+which ``cleanuppad`` it exits, and must be a :ref:`cleanuppad <i_cleanuppad>`.
+It also has an optional successor, ``nextaction``, indicating where control
+should transfer to.
+
+Semantics:
+""""""""""
+
+When and exception propagates to a ``cleanupendpad``, control is transfered to
+``nextaction`` if it is present. If it is not present, control is transfered to
+the caller.
+
+The ``cleanupendpad`` instruction has several restrictions:
+
+-  A cleanup-end block is a basic block which is the unwind destination of
+   an exceptional instruction.
+-  A cleanup-end block must have a '``cleanupendpad``' instruction as its
+   first non-PHI instruction.
+-  There can be only one '``cleanupendpad``' instruction within the
+   cleanup-end block.
+-  A basic block that is not a cleanup-end block may not include a
+   '``cleanupendpad``' instruction.
+- It is undefined behavior to execute a ``cleanupendpad`` whose ``cleanuppad``
+  has not been executed.
+- It is undefined behavior to execute a ``cleanupendpad`` if, after the most
+  recent execution of its ``cleanuppad``, some ``cleanupret`` or ``cleanupendpad``
+  consuming the same ``cleanuppad`` has already been executed.
+- It is undefined behavior to execute a ``cleanupendpad`` if, after the most
+  recent execution of its ``cleanuppad``, any other ``cleanuppad`` or
+  ``catchpad`` has been executed but has not had a corresponding
+  ``cleanupret``/``catchret``/``cleanupendpad``/``catchendpad`` executed.
+
+Example:
+""""""""
+
+.. code-block:: llvm
+
+      cleanupendpad %cleanup unwind label %terminate
+      cleanupendpad %cleanup unwind to caller
+
 .. _i_cleanupret:
 
 '``cleanupret``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 

 
 Semantics:
 """"""""""
 
 The '``cleanupret``' instruction indicates to the
 :ref:`personality function <personalityfn>` that one
 :ref:`cleanuppad <i_cleanuppad>` it transferred control to has ended.
 It transfers control to ``continue`` or unwinds out of the function.
+
 It is undefined behavior to execute a ``cleanupret`` whose ``cleanuppad`` has
 not been executed.
-It is undefined behavior to execute a ``cleanupret`` if any ``catchpad`` or
-``cleanuppad`` has been executed, without subsequently executing a
-corresponding ``catchret``/``cleanupret`` or unwinding out of the inner pad,
-following the most recent execution of the ``cleanupret``'s corresponding
-``cleanuppad``.
+
+It is undefined behavior to execute a ``cleanupret`` if, after the most recent
+execution of its ``cleanuppad``, some ``cleanupret`` or ``cleanupendpad``
+consuming the same ``cleanuppad`` has already been executed.
+
+It is undefined behavior to execute a ``cleanupret`` if, after the most recent
+execution of its ``cleanuppad``, any other ``cleanuppad`` or ``catchpad`` has
+been executed but has not had a corresponding
+``cleanupret``/``catchret``/``cleanupendpad``/``catchendpad`` executed.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       cleanupret %cleanup unwind to caller
       cleanupret %cleanup unwind label %continue

 The '``cleanuppad``' instruction is used by `LLVM's exception handling
 system <ExceptionHandling.html#overview>`_ to specify that a basic block
 is a cleanup block --- one where a personality routine attempts to
 transfer control to run cleanup actions.
 The ``args`` correspond to whatever additional
 information the :ref:`personality function <personalityfn>` requires to
 execute the cleanup.
 The ``resultval`` has the type :ref:`token <t_token>` and is used to
-match the ``cleanuppad`` to corresponding :ref:`cleanuprets <i_cleanupret>`.
+match the ``cleanuppad`` to corresponding :ref:`cleanuprets <i_cleanupret>`
+and :ref:`cleanupendpads <i_cleanupendpad>`.
 
 Arguments:
 """"""""""
 
 The instruction takes a list of arbitrary values which are interpreted
 by the :ref:`personality function <personalityfn>`.
 
 Semantics:
 """"""""""
 
-The '``cleanuppad``' instruction defines the values which are set by the
-:ref:`personality function <personalityfn>` upon re-entry to the function.
-As with calling conventions, how the personality function results are
-represented in LLVM IR is target specific.
-
 When the call stack is being unwound due to an exception being thrown,
 the :ref:`personality function <personalityfn>` transfers control to the
 ``cleanuppad`` with the aid of the personality-specific arguments.
+As with calling conventions, how the personality function results are
+represented in LLVM IR is target specific.
 
 The ``cleanuppad`` instruction has several restrictions:
 
 -  A cleanup block is a basic block which is the unwind destination of
    an exceptional instruction.
 -  A cleanup block must have a '``cleanuppad``' instruction as its
    first non-PHI instruction.
 -  There can be only one '``cleanuppad``' instruction within the
    cleanup block.
 -  A basic block that is not a cleanup block may not include a
    '``cleanuppad``' instruction.
--  All '``cleanupret``'s which exit a ``cleanuppad`` must have the same
-   exceptional successor.
+-  All '``cleanupret``'s and '``cleanupendpad``'s which consume a ``cleanuppad``
+   must have the same exceptional successor.
 -  It is undefined behavior for control to transfer from a ``cleanuppad`` to a
-   ``ret`` without first executing a ``cleanupret`` that consumes the
-   ``cleanuppad`` or unwinding out of the ``cleanuppad``.
+   ``ret`` without first executing a ``cleanupret`` or ``cleanupendpad`` that
+   consumes the ``cleanuppad``.
 -  It is undefined behavior for control to transfer from a ``cleanuppad`` to
-   itself without first executing a ``cleanupret`` that consumes the
-   ``cleanuppad`` or unwinding out of the ``cleanuppad``.
+   itself without first executing a ``cleanupret`` or ``cleanupendpad`` that
+   consumes the ``cleanuppad``.
 
 Example:
 """"""""
 
 .. code-block:: llvm
 
       %tok = cleanuppad []
 

include/llvm-c/Core.h

typedef enum {
   LLVMAtomicCmpXchg  = 56,
   LLVMAtomicRMW      = 57,
 
   /* Exception Handling Operators */
   LLVMResume         = 58,
   LLVMLandingPad     = 59,
   LLVMCleanupRet     = 61,
   LLVMCatchRet       = 62,
-  LLVMCatchPad     = 63,
-  LLVMTerminatePad = 64,
-  LLVMCleanupPad   = 65,
-  LLVMCatchEndPad  = 66
+  LLVMCatchPad       = 63,
+  LLVMTerminatePad   = 64,
+  LLVMCleanupPad     = 65,
+  LLVMCatchEndPad    = 66,
+  LLVMCleanupEndPad  = 67
 
 } LLVMOpcode;
 
 typedef enum {
   LLVMVoidTypeKind,        /**< type with no size */
   LLVMHalfTypeKind,        /**< 16 bit floating point type */
   LLVMFloatTypeKind,       /**< 32 bit floating point type */
   LLVMDoubleTypeKind,      /**< 64 bit floating point type */
macro(Instruction)                      \
         macro(SwitchInst)                   \
         macro(UnreachableInst)              \
         macro(ResumeInst)                   \
         macro(CleanupReturnInst)            \
         macro(CatchReturnInst)              \
         macro(CatchPadInst)               \
         macro(TerminatePadInst)           \
         macro(CatchEndPadInst)            \
+        macro(CleanupEndPadInst)            \
       macro(UnaryInstruction)               \
         macro(AllocaInst)                   \
         macro(CastInst)                     \
           macro(AddrSpaceCastInst)          \
           macro(BitCastInst)                \
           macro(FPExtInst)                  \
           macro(FPToSIInst)                 \
           macro(FPToUIInst)                 \

include/llvm/Bitcode/LLVMBitCodes.h

FUNC_CODE_INST_CMPXCHG     = 46, // CMPXCHG: [ptrty,ptr,valty,cmp,new, align,
                                      //           vol,ordering,synchscope]
     FUNC_CODE_INST_LANDINGPAD  = 47, // LANDINGPAD: [ty,val,num,id0,val0...]
     FUNC_CODE_INST_CLEANUPRET  = 48, // CLEANUPRET: [val] or [val,bb#]
     FUNC_CODE_INST_CATCHRET    = 49, // CATCHRET: [val,bb#]
     FUNC_CODE_INST_CATCHPAD  = 50, // CATCHPAD: [bb#,bb#,num,args...]
     FUNC_CODE_INST_TERMINATEPAD = 51, // TERMINATEPAD: [bb#,num,args...]
     FUNC_CODE_INST_CLEANUPPAD = 52, // CLEANUPPAD: [num,args...]
     FUNC_CODE_INST_CATCHENDPAD = 53, // CATCHENDPAD: [] or [bb#]
+    FUNC_CODE_INST_CLEANUPENDPAD = 54, // CLEANUPENDPAD: [val] or [val,bb#]
   };
 
   enum UseListCodes {
     USELIST_CODE_DEFAULT = 1, // DEFAULT: [index..., value-id]
     USELIST_CODE_BB      = 2  // BB: [index..., bb-id]
   };
 
   enum AttributeKindCodes {

include/llvm/IR/IRBuilder.h

ResumeInst *CreateResume(Value *Exn) {
     return Insert(ResumeInst::Create(Exn));
   }
 
   CleanupReturnInst *CreateCleanupRet(CleanupPadInst *CleanupPad,
                                       BasicBlock *UnwindBB = nullptr) {
     return Insert(CleanupReturnInst::Create(CleanupPad, UnwindBB));
   }
 
+  CatchEndPadInst *CreateCleanupEndPad(CleanupPadInst *CleanupPad,
+                                       BasicBlock *UnwindBB = nullptr) {
+    return Insert(CleanupEndPadInst::Create(CleanupPad, UnwindBB));
+  }
+
   CatchPadInst *CreateCatchPad(BasicBlock *NormalDest, BasicBlock *UnwindDest,
                                ArrayRef<Value *> Args, const Twine &Name = "") {
     return Insert(CatchPadInst::Create(NormalDest, UnwindDest, Args), Name);
   }
 
   CatchEndPadInst *CreateCatchEndPad(BasicBlock *UnwindBB = nullptr) {
     return Insert(CatchEndPadInst::Create(Context, UnwindBB));
   }

include/llvm/IR/InstVisitor.h

#include "llvm/IR/Instruction.def"
   //
   RetTy visitReturnInst(ReturnInst &I)            { DELEGATE(TerminatorInst);}
   RetTy visitBranchInst(BranchInst &I)            { DELEGATE(TerminatorInst);}
   RetTy visitSwitchInst(SwitchInst &I)            { DELEGATE(TerminatorInst);}
   RetTy visitIndirectBrInst(IndirectBrInst &I)    { DELEGATE(TerminatorInst);}
   RetTy visitResumeInst(ResumeInst &I)            { DELEGATE(TerminatorInst);}
   RetTy visitUnreachableInst(UnreachableInst &I)  { DELEGATE(TerminatorInst);}
   RetTy visitCleanupReturnInst(CleanupReturnInst &I) { DELEGATE(TerminatorInst);}
-  RetTy visitCatchReturnInst(CatchReturnInst &I)  { DELEGATE(TerminatorInst);}
+  RetTy visitCleanupEndPadInst(CleanupEndPadInst &I) { DELEGATE(TerminatorInst); }
+  RetTy visitCatchReturnInst(CatchReturnInst &I)  { DELEGATE(TerminatorInst); }
   RetTy visitCatchPadInst(CatchPadInst &I)    { DELEGATE(TerminatorInst);}
   RetTy visitCatchEndPadInst(CatchEndPadInst &I) { DELEGATE(TerminatorInst); }
   RetTy visitTerminatePadInst(TerminatePadInst &I) { DELEGATE(TerminatorInst);}
   RetTy visitICmpInst(ICmpInst &I)                { DELEGATE(CmpInst);}
   RetTy visitFCmpInst(FCmpInst &I)                { DELEGATE(CmpInst);}
   RetTy visitAllocaInst(AllocaInst &I)            { DELEGATE(UnaryInstruction);}
   RetTy visitLoadInst(LoadInst     &I)            { DELEGATE(UnaryInstruction);}
   RetTy visitStoreInst(StoreInst   &I)            { DELEGATE(Instruction);}

include/llvm/IR/InstrTypes.h

public:
   }
 
   // \brief Returns true if this terminator relates to exception handling.
   bool isExceptional() const {
     switch (getOpcode()) {
     case Instruction::CatchPad:
     case Instruction::CatchEndPad:
     case Instruction::CatchRet:
+    case Instruction::CleanupEndPad:
     case Instruction::CleanupRet:
     case Instruction::Invoke:
     case Instruction::Resume:
     case Instruction::TerminatePad:
       return true;
     default:
       return false;
     }

include/llvm/IR/Instruction.h

public:
   }
 
   /// \brief Return true if the instruction is a variety of EH-block.
   bool isEHPad() const {
     switch (getOpcode()) {
     case Instruction::CatchPad:
     case Instruction::CatchEndPad:
     case Instruction::CleanupPad:
+    case Instruction::CleanupEndPad:
     case Instruction::LandingPad:
     case Instruction::TerminatePad:
       return true;
     default:
       return false;
     }
   }
 

include/llvm/IR/Instruction.def

 HANDLE_TERM_INST  ( 2, Br            , BranchInst)
 HANDLE_TERM_INST  ( 3, Switch        , SwitchInst)
 HANDLE_TERM_INST  ( 4, IndirectBr    , IndirectBrInst)
 HANDLE_TERM_INST  ( 5, Invoke        , InvokeInst)
 HANDLE_TERM_INST  ( 6, Resume        , ResumeInst)
 HANDLE_TERM_INST  ( 7, Unreachable   , UnreachableInst)
 HANDLE_TERM_INST  ( 8, CleanupRet    , CleanupReturnInst)
 HANDLE_TERM_INST  ( 9, CatchRet      , CatchReturnInst)
-HANDLE_TERM_INST  (10, CatchPad    , CatchPadInst)
-HANDLE_TERM_INST  (11, TerminatePad, TerminatePadInst)
-HANDLE_TERM_INST  (12, CatchEndPad , CatchEndPadInst)
-  LAST_TERM_INST  (12)
+HANDLE_TERM_INST  (10, CatchPad      , CatchPadInst)
+HANDLE_TERM_INST  (11, TerminatePad  , TerminatePadInst)
+HANDLE_TERM_INST  (12, CatchEndPad   , CatchEndPadInst)
+HANDLE_TERM_INST  (13, CleanupEndPad , CleanupEndPadInst)
+  LAST_TERM_INST  (13)
 
 // Standard binary operators...
- FIRST_BINARY_INST(13)
-HANDLE_BINARY_INST(13, Add  , BinaryOperator)
-HANDLE_BINARY_INST(14, FAdd , BinaryOperator)
-HANDLE_BINARY_INST(15, Sub  , BinaryOperator)
-HANDLE_BINARY_INST(16, FSub , BinaryOperator)
-HANDLE_BINARY_INST(17, Mul  , BinaryOperator)
-HANDLE_BINARY_INST(18, FMul , BinaryOperator)
-HANDLE_BINARY_INST(19, UDiv , BinaryOperator)
-HANDLE_BINARY_INST(20, SDiv , BinaryOperator)
-HANDLE_BINARY_INST(21, FDiv , BinaryOperator)
-HANDLE_BINARY_INST(22, URem , BinaryOperator)
-HANDLE_BINARY_INST(23, SRem , BinaryOperator)
-HANDLE_BINARY_INST(24, FRem , BinaryOperator)
+ FIRST_BINARY_INST(14)
+HANDLE_BINARY_INST(14, Add  , BinaryOperator)
+HANDLE_BINARY_INST(15, FAdd , BinaryOperator)
+HANDLE_BINARY_INST(16, Sub  , BinaryOperator)
+HANDLE_BINARY_INST(17, FSub , BinaryOperator)
+HANDLE_BINARY_INST(18, Mul  , BinaryOperator)
+HANDLE_BINARY_INST(19, FMul , BinaryOperator)
+HANDLE_BINARY_INST(20, UDiv , BinaryOperator)
+HANDLE_BINARY_INST(21, SDiv , BinaryOperator)
+HANDLE_BINARY_INST(22, FDiv , BinaryOperator)
+HANDLE_BINARY_INST(23, URem , BinaryOperator)
+HANDLE_BINARY_INST(24, SRem , BinaryOperator)
+HANDLE_BINARY_INST(25, FRem , BinaryOperator)
 
 // Logical operators (integer operands)
-HANDLE_BINARY_INST(25, Shl  , BinaryOperator) // Shift left  (logical)
-HANDLE_BINARY_INST(26, LShr , BinaryOperator) // Shift right (logical)
-HANDLE_BINARY_INST(27, AShr , BinaryOperator) // Shift right (arithmetic)
-HANDLE_BINARY_INST(28, And  , BinaryOperator)
-HANDLE_BINARY_INST(29, Or   , BinaryOperator)
-HANDLE_BINARY_INST(30, Xor  , BinaryOperator)
-  LAST_BINARY_INST(30)
+HANDLE_BINARY_INST(26, Shl  , BinaryOperator) // Shift left  (logical)
+HANDLE_BINARY_INST(27, LShr , BinaryOperator) // Shift right (logical)
+HANDLE_BINARY_INST(28, AShr , BinaryOperator) // Shift right (arithmetic)
+HANDLE_BINARY_INST(29, And  , BinaryOperator)
+HANDLE_BINARY_INST(30, Or   , BinaryOperator)
+HANDLE_BINARY_INST(31, Xor  , BinaryOperator)
+  LAST_BINARY_INST(31)
 
 // Memory operators...
- FIRST_MEMORY_INST(31)
-HANDLE_MEMORY_INST(31, Alloca, AllocaInst)  // Stack management
-HANDLE_MEMORY_INST(32, Load  , LoadInst  )  // Memory manipulation instrs
-HANDLE_MEMORY_INST(33, Store , StoreInst )
-HANDLE_MEMORY_INST(34, GetElementPtr, GetElementPtrInst)
-HANDLE_MEMORY_INST(35, Fence , FenceInst )
-HANDLE_MEMORY_INST(36, AtomicCmpXchg , AtomicCmpXchgInst )
-HANDLE_MEMORY_INST(37, AtomicRMW , AtomicRMWInst )
-  LAST_MEMORY_INST(37)
+ FIRST_MEMORY_INST(32)
+HANDLE_MEMORY_INST(32, Alloca, AllocaInst)  // Stack management
+HANDLE_MEMORY_INST(33, Load  , LoadInst  )  // Memory manipulation instrs
+HANDLE_MEMORY_INST(34, Store , StoreInst )
+HANDLE_MEMORY_INST(35, GetElementPtr, GetElementPtrInst)
+HANDLE_MEMORY_INST(36, Fence , FenceInst )
+HANDLE_MEMORY_INST(37, AtomicCmpXchg , AtomicCmpXchgInst )
+HANDLE_MEMORY_INST(38, AtomicRMW , AtomicRMWInst )
+  LAST_MEMORY_INST(38)
 
 // Cast operators ...
 // NOTE: The order matters here because CastInst::isEliminableCastPair 
 // NOTE: (see Instructions.cpp) encodes a table based on this ordering.
- FIRST_CAST_INST(38)
-HANDLE_CAST_INST(38, Trunc   , TruncInst   )  // Truncate integers
-HANDLE_CAST_INST(39, ZExt    , ZExtInst    )  // Zero extend integers
-HANDLE_CAST_INST(40, SExt    , SExtInst    )  // Sign extend integers
-HANDLE_CAST_INST(41, FPToUI  , FPToUIInst  )  // floating point -> UInt
-HANDLE_CAST_INST(42, FPToSI  , FPToSIInst  )  // floating point -> SInt
-HANDLE_CAST_INST(43, UIToFP  , UIToFPInst  )  // UInt -> floating point
-HANDLE_CAST_INST(44, SIToFP  , SIToFPInst  )  // SInt -> floating point
-HANDLE_CAST_INST(45, FPTrunc , FPTruncInst )  // Truncate floating point
-HANDLE_CAST_INST(46, FPExt   , FPExtInst   )  // Extend floating point
-HANDLE_CAST_INST(47, PtrToInt, PtrToIntInst)  // Pointer -> Integer
-HANDLE_CAST_INST(48, IntToPtr, IntToPtrInst)  // Integer -> Pointer
-HANDLE_CAST_INST(49, BitCast , BitCastInst )  // Type cast
-HANDLE_CAST_INST(50, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
-  LAST_CAST_INST(50)
+ FIRST_CAST_INST(39)
+HANDLE_CAST_INST(39, Trunc   , TruncInst   )  // Truncate integers
+HANDLE_CAST_INST(40, ZExt    , ZExtInst    )  // Zero extend integers
+HANDLE_CAST_INST(41, SExt    , SExtInst    )  // Sign extend integers
+HANDLE_CAST_INST(42, FPToUI  , FPToUIInst  )  // floating point -> UInt
+HANDLE_CAST_INST(43, FPToSI  , FPToSIInst  )  // floating point -> SInt
+HANDLE_CAST_INST(44, UIToFP  , UIToFPInst  )  // UInt -> floating point
+HANDLE_CAST_INST(45, SIToFP  , SIToFPInst  )  // SInt -> floating point
+HANDLE_CAST_INST(46, FPTrunc , FPTruncInst )  // Truncate floating point
+HANDLE_CAST_INST(47, FPExt   , FPExtInst   )  // Extend floating point
+HANDLE_CAST_INST(48, PtrToInt, PtrToIntInst)  // Pointer -> Integer
+HANDLE_CAST_INST(49, IntToPtr, IntToPtrInst)  // Integer -> Pointer
+HANDLE_CAST_INST(50, BitCast , BitCastInst )  // Type cast
+HANDLE_CAST_INST(51, AddrSpaceCast, AddrSpaceCastInst)  // addrspace cast
+  LAST_CAST_INST(51)
 
 // Other operators...
- FIRST_OTHER_INST(51)
-HANDLE_OTHER_INST(51, ICmp   , ICmpInst   )  // Integer comparison instruction
-HANDLE_OTHER_INST(52, FCmp   , FCmpInst   )  // Floating point comparison instr.
-HANDLE_OTHER_INST(53, PHI    , PHINode    )  // PHI node instruction
-HANDLE_OTHER_INST(54, Call   , CallInst   )  // Call a function
-HANDLE_OTHER_INST(55, Select , SelectInst )  // select instruction
-HANDLE_OTHER_INST(56, UserOp1, Instruction)  // May be used internally in a pass
-HANDLE_OTHER_INST(57, UserOp2, Instruction)  // Internal to passes only
-HANDLE_OTHER_INST(58, VAArg  , VAArgInst  )  // vaarg instruction
-HANDLE_OTHER_INST(59, ExtractElement, ExtractElementInst)// extract from vector
-HANDLE_OTHER_INST(60, InsertElement, InsertElementInst)  // insert into vector
-HANDLE_OTHER_INST(61, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
-HANDLE_OTHER_INST(62, ExtractValue, ExtractValueInst)// extract from aggregate
-HANDLE_OTHER_INST(63, InsertValue, InsertValueInst)  // insert into aggregate
-HANDLE_OTHER_INST(64, LandingPad, LandingPadInst)  // Landing pad instruction.
-HANDLE_OTHER_INST(65, CleanupPad, CleanupPadInst)
-  LAST_OTHER_INST(65)
+ FIRST_OTHER_INST(52)
+HANDLE_OTHER_INST(52, ICmp   , ICmpInst   )  // Integer comparison instruction
+HANDLE_OTHER_INST(53, FCmp   , FCmpInst   )  // Floating point comparison instr.
+HANDLE_OTHER_INST(54, PHI    , PHINode    )  // PHI node instruction
+HANDLE_OTHER_INST(55, Call   , CallInst   )  // Call a function
+HANDLE_OTHER_INST(56, Select , SelectInst )  // select instruction
+HANDLE_OTHER_INST(57, UserOp1, Instruction)  // May be used internally in a pass
+HANDLE_OTHER_INST(58, UserOp2, Instruction)  // Internal to passes only
+HANDLE_OTHER_INST(59, VAArg  , VAArgInst  )  // vaarg instruction
+HANDLE_OTHER_INST(60, ExtractElement, ExtractElementInst)// extract from vector
+HANDLE_OTHER_INST(61, InsertElement, InsertElementInst)  // insert into vector
+HANDLE_OTHER_INST(62, ShuffleVector, ShuffleVectorInst)  // shuffle two vectors.
+HANDLE_OTHER_INST(63, ExtractValue, ExtractValueInst)// extract from aggregate
+HANDLE_OTHER_INST(64, InsertValue, InsertValueInst)  // insert into aggregate
+HANDLE_OTHER_INST(65, LandingPad, LandingPadInst)  // Landing pad instruction.
+HANDLE_OTHER_INST(66, CleanupPad, CleanupPadInst)
+  LAST_OTHER_INST(66)
 
 #undef  FIRST_TERM_INST
 #undef HANDLE_TERM_INST
 #undef   LAST_TERM_INST
 
 #undef  FIRST_BINARY_INST
 #undef HANDLE_BINARY_INST
 #undef   LAST_BINARY_INST

include/llvm/IR/Instructions.h

 
 template <>
 struct OperandTraits<CatchReturnInst>
     : public FixedNumOperandTraits<CatchReturnInst, 2> {};
 
 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CatchReturnInst, Value)
 
 //===----------------------------------------------------------------------===//
+//                               CleanupEndPadInst Class
+//===----------------------------------------------------------------------===//
+
+class CleanupEndPadInst : public TerminatorInst {
+private:
+  CleanupEndPadInst(const CleanupEndPadInst &CEPI);
+
+  void init(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB);
+  CleanupEndPadInst(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB,
+                    unsigned Values, Instruction *InsertBefore = nullptr);
+  CleanupEndPadInst(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB,
+                    unsigned Values, BasicBlock *InsertAtEnd);
+
+protected:
+  // Note: Instruction needs to be a friend here to call cloneImpl.
+  friend class Instruction;
+  CleanupEndPadInst *cloneImpl() const;
+
+public:
+  static CleanupEndPadInst *Create(CleanupPadInst *CleanupPad,
+                                   BasicBlock *UnwindBB = nullptr,
+                                   Instruction *InsertBefore = nullptr) {
+    unsigned Values = UnwindBB ? 2 : 1;
+    return new (Values)
+        CleanupEndPadInst(CleanupPad, UnwindBB, Values, InsertBefore);
+  }
+  static CleanupEndPadInst *Create(CleanupPadInst *CleanupPad,
+                                   BasicBlock *UnwindBB,
+                                   BasicBlock *InsertAtEnd) {
+    unsigned Values = UnwindBB ? 2 : 1;
+    return new (Values)
+        CleanupEndPadInst(CleanupPad, UnwindBB, Values, InsertAtEnd);
+  }
+
+  /// Provide fast operand accessors
+  DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
+
+  bool hasUnwindDest() const { return getSubclassDataFromInstruction() & 1; }
+  bool unwindsToCaller() const { return !hasUnwindDest(); }
+
+  unsigned getNumSuccessors() const { return hasUnwindDest() ? 1 : 0; }
+
+  /// Convenience accessors
+  CleanupPadInst *getCleanupPad() const {
+    return cast<CleanupPadInst>(Op<-1>());
+  }
+  void setCleanupPad(CleanupPadInst *CleanupPad) {
+    assert(CleanupPad);
+    Op<-1>() = CleanupPad;
+  }
+
+  BasicBlock *getUnwindDest() const {
+    return hasUnwindDest() ? cast<BasicBlock>(Op<-2>()) : nullptr;
+  }
+  void setUnwindDest(BasicBlock *NewDest) {
+    assert(hasUnwindDest());
+    assert(NewDest);
+    Op<-2>() = NewDest;
+  }
+
+  // Methods for support type inquiry through isa, cast, and dyn_cast:
+  static inline bool classof(const Instruction *I) {
+    return (I->getOpcode() == Instruction::CleanupEndPad);
+  }
+  static inline bool classof(const Value *V) {
+    return isa<Instruction>(V) && classof(cast<Instruction>(V));
+  }
+
+private:
+  BasicBlock *getSuccessorV(unsigned Idx) const override;
+  unsigned getNumSuccessorsV() const override;
+  void setSuccessorV(unsigned Idx, BasicBlock *B) override;
+
+  // Shadow Instruction::setInstructionSubclassData with a private forwarding
+  // method so that subclasses cannot accidentally use it.
+  void setInstructionSubclassData(unsigned short D) {
+    Instruction::setInstructionSubclassData(D);
+  }
+};
+
+template <>
+struct OperandTraits<CleanupEndPadInst>
+    : public VariadicOperandTraits<CleanupEndPadInst, /*MINARITY=*/1> {};
+
+DEFINE_TRANSPARENT_OPERAND_ACCESSORS(CleanupEndPadInst, Value)
+
+//===----------------------------------------------------------------------===//
 //                               CleanupReturnInst Class
 //===----------------------------------------------------------------------===//
 
 class CleanupReturnInst : public TerminatorInst {
 private:
   CleanupReturnInst(const CleanupReturnInst &RI);
 
   void init(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB);

lib/Analysis/ValueTracking.cpp

bool llvm::isSafeToSpeculativelyExecute(const Value *V,
   case Instruction::AtomicRMW:
   case Instruction::AtomicCmpXchg:
   case Instruction::LandingPad:
   case Instruction::Resume:
   case Instruction::CatchPad:
   case Instruction::CatchEndPad:
   case Instruction::CatchRet:
   case Instruction::CleanupPad:
+  case Instruction::CleanupEndPad:
   case Instruction::CleanupRet:
   case Instruction::TerminatePad:
     return false; // Misc instructions which have effects
   }
 }
 
 bool llvm::mayBeMemoryDependent(const Instruction &I) {
   return I.mayReadOrWriteMemory() || !isSafeToSpeculativelyExecute(&I);

lib/AsmParser/LLLexer.cpp

#define INSTKEYWORD(STR, Enum)                                                 \
   INSTKEYWORD(insertvalue,    InsertValue);
   INSTKEYWORD(landingpad,     LandingPad);
   INSTKEYWORD(cleanupret,     CleanupRet);
   INSTKEYWORD(catchret,       CatchRet);
   INSTKEYWORD(catchpad,     CatchPad);
   INSTKEYWORD(terminatepad, TerminatePad);
   INSTKEYWORD(cleanuppad,   CleanupPad);
   INSTKEYWORD(catchendpad,  CatchEndPad);
+  INSTKEYWORD(cleanupendpad, CleanupEndPad);
 #undef INSTKEYWORD
 
 #define DWKEYWORD(TYPE, TOKEN)                                                 \
   do {                                                                         \
     if (Keyword.startswith("DW_" #TYPE "_")) {                                 \
       StrVal.assign(Keyword.begin(), Keyword.end());                           \
       return lltok::TOKEN;                                                     \
     }                                                                          \

lib/AsmParser/LLParser.h

#include "llvm/IR/Metadata.def"
     bool ParseInvoke(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseResume(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchRet(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchPad(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseTerminatePad(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCleanupPad(Instruction *&Inst, PerFunctionState &PFS);
     bool ParseCatchEndPad(Instruction *&Inst, PerFunctionState &PFS);
+    bool ParseCleanupEndPad(Instruction *&Inst, PerFunctionState &PFS);
 
     bool ParseArithmetic(Instruction *&I, PerFunctionState &PFS, unsigned Opc,
                          unsigned OperandType);
     bool ParseLogical(Instruction *&I, PerFunctionState &PFS, unsigned Opc);
     bool ParseCompare(Instruction *&I, PerFunctionState &PFS, unsigned Opc);
     bool ParseCast(Instruction *&I, PerFunctionState &PFS, unsigned Opc);
     bool ParseSelect(Instruction *&I, PerFunctionState &PFS);
     bool ParseVA_Arg(Instruction *&I, PerFunctionState &PFS);

lib/AsmParser/LLParser.cpp

int LLParser::ParseInstruction(Instruction *&Inst, BasicBlock *BB,
   case lltok::kw_invoke:      return ParseInvoke(Inst, PFS);
   case lltok::kw_resume:      return ParseResume(Inst, PFS);
   case lltok::kw_cleanupret:  return ParseCleanupRet(Inst, PFS);
   case lltok::kw_catchret:    return ParseCatchRet(Inst, PFS);
   case lltok::kw_catchpad:  return ParseCatchPad(Inst, PFS);
   case lltok::kw_terminatepad: return ParseTerminatePad(Inst, PFS);
   case lltok::kw_cleanuppad: return ParseCleanupPad(Inst, PFS);
   case lltok::kw_catchendpad: return ParseCatchEndPad(Inst, PFS);
+  case lltok::kw_cleanupendpad: return ParseCleanupEndPad(Inst, PFS);
   // Binary Operators.
   case lltok::kw_add:
   case lltok::kw_sub:
   case lltok::kw_mul:
   case lltok::kw_shl: {
     bool NUW = EatIfPresent(lltok::kw_nuw);
     bool NSW = EatIfPresent(lltok::kw_nsw);
     if (!NUW) NUW = EatIfPresent(lltok::kw_nuw);
if (ParseTypeAndBasicBlock(UnwindBB, PFS)) {
       return true;
     }
   }
 
   Inst = CatchEndPadInst::Create(Context, UnwindBB);
   return false;
 }
 
+/// ParseCatchEndPad
+///   ::= 'cleanupendpad' Value unwind ('to' 'caller' | TypeAndValue)
+bool LLParser::ParseCleanupEndPad(Instruction *&Inst, PerFunctionState &PFS) {
+  Value *CleanupPad = nullptr;
+
+  if (ParseValue(Type::getTokenTy(Context), CleanupPad, PFS, OC_CleanupPad))
+    return true;
+
+  if (ParseToken(lltok::kw_unwind, "expected 'unwind' in catchendpad"))
+    return true;
+
+  BasicBlock *UnwindBB = nullptr;
+  if (Lex.getKind() == lltok::kw_to) {
+    Lex.Lex();
+    if (Lex.getKind() == lltok::kw_caller) {
+      Lex.Lex();
+    } else {
+      return true;
+    }
+  } else {
+    if (ParseTypeAndBasicBlock(UnwindBB, PFS)) {
+      return true;
+    }
+  }
+
+  Inst = CleanupEndPadInst::Create(cast<CleanupPadInst>(CleanupPad), UnwindBB);
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 // Binary Operators.
 //===----------------------------------------------------------------------===//
 
 /// ParseArithmetic
 ///  ::= ArithmeticOps TypeAndValue ',' Value
 ///
 /// If OperandType is 0, then any FP or integer operand is allowed.  If it is 1,

lib/AsmParser/LLToken.h

enum Kind {
     kw_fptoui, kw_fptosi, kw_inttoptr, kw_ptrtoint, kw_bitcast,
     kw_addrspacecast,
     kw_select, kw_va_arg,
 
     kw_landingpad, kw_personality, kw_cleanup, kw_catch, kw_filter,
 
     kw_ret, kw_br, kw_switch, kw_indirectbr, kw_invoke, kw_resume,
     kw_unreachable, kw_cleanupret, kw_catchret, kw_catchpad,
-    kw_terminatepad, kw_cleanuppad, kw_catchendpad,
+    kw_terminatepad, kw_cleanuppad, kw_catchendpad, kw_cleanupendpad,
 
     kw_alloca, kw_load, kw_store, kw_fence, kw_cmpxchg, kw_atomicrmw,
     kw_getelementptr,
 
     kw_extractelement, kw_insertelement, kw_shufflevector,
     kw_extractvalue, kw_insertvalue, kw_blockaddress,
 
     // Metadata types.

lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_CATCHENDPAD: { // CATCHENDPADINST: [bb#] or []
         BB = getBasicBlock(Record[0]);
         if (!BB)
           return error("Invalid record");
       }
       I = CatchEndPadInst::Create(Context, BB);
       InstructionList.push_back(I);
       break;
     }
+    case bitc::FUNC_CODE_INST_CLEANUPENDPAD: { // CLEANUPENDPADINST: [val] or [val,bb#]
+      if (Record.size() != 1 && Record.size() != 2)
+        return error("Invalid record");
+      unsigned Idx = 0;
+      Value *CleanupPad = getValue(Record, Idx++, NextValueNo,
+                                   Type::getTokenTy(Context), OC_CleanupPad);
+      if (!CleanupPad)
+        return error("Invalid record");
+
+      BasicBlock *BB = nullptr;
+      if (Record.size() == 2) {
+        BB = getBasicBlock(Record[Idx++]);
+        if (!BB)
+          return error("Invalid record");
+      }
+      I = CleanupEndPadInst::Create(cast<CleanupPadInst>(CleanupPad), BB);
+      InstructionList.push_back(I);
+      break;
+    }
     case bitc::FUNC_CODE_INST_SWITCH: { // SWITCH: [opty, op0, op1, ...]
       // Check magic
       if ((Record[0] >> 16) == SWITCH_INST_MAGIC) {
         // "New" SwitchInst format with case ranges. The changes to write this
         // format were reverted but we still recognize bitcode that uses it.
         // Hopefully someday we will have support for case ranges and can use
         // this format again.
 

lib/Bitcode/Writer/BitcodeWriter.cpp

static void WriteInstruction(const Instruction &I, unsigned InstID,
   }
   case Instruction::CatchEndPad: {
     Code = bitc::FUNC_CODE_INST_CATCHENDPAD;
     const auto &CEPI = cast<CatchEndPadInst>(I);
     if (CEPI.hasUnwindDest())
       Vals.push_back(VE.getValueID(CEPI.getUnwindDest()));
     break;
   }
+  case Instruction::CleanupEndPad: {
+    Code = bitc::FUNC_CODE_INST_CLEANUPENDPAD;
+    const auto &CEPI = cast<CleanupEndPadInst>(I);
+    pushValue(CEPI.getCleanupPad(), InstID, Vals, VE);
+    if (CEPI.hasUnwindDest())
+      Vals.push_back(VE.getValueID(CEPI.getUnwindDest()));
+    break;
+  }
   case Instruction::Unreachable:
     Code = bitc::FUNC_CODE_INST_UNREACHABLE;
     AbbrevToUse = FUNCTION_INST_UNREACHABLE_ABBREV;
     break;
 
   case Instruction::PHI: {
     const PHINode &PN = cast<PHINode>(I);
     Code = bitc::FUNC_CODE_INST_PHI;

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.h

std::pair<SDValue, SDValue> lowerInvokable(
           MachineBasicBlock *LandingPad);
 
   // Terminator instructions.
   void visitRet(const ReturnInst &I);
   void visitBr(const BranchInst &I);
   void visitSwitch(const SwitchInst &I);
   void visitIndirectBr(const IndirectBrInst &I);
   void visitUnreachable(const UnreachableInst &I);
+  void visitCleanupEndPad(const CleanupEndPadInst &I);
   void visitCleanupRet(const CleanupReturnInst &I);
   void visitCatchEndPad(const CatchEndPadInst &I);
   void visitCatchRet(const CatchReturnInst &I);
   void visitCatchPad(const CatchPadInst &I);
   void visitTerminatePad(const TerminatePadInst &TPI);
   void visitCleanupPad(const CleanupPadInst &CPI);
 
   uint32_t getEdgeWeight(const MachineBasicBlock *Src,

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

void SelectionDAGBuilder::visitCleanupPad(const CleanupPadInst &CPI) {
   MMI.setHasEHFunclets(true);
   report_fatal_error("visitCleanupPad not yet implemented!");
 }
 
 void SelectionDAGBuilder::visitCleanupRet(const CleanupReturnInst &I) {
   report_fatal_error("visitCleanupRet not yet implemented!");
 }
 
+void SelectionDAGBuilder::visitCleanupEndPad(const CleanupEndPadInst &I) {
+  report_fatal_error("visitCleanupEndPad not yet implemented!");
+}
+
 void SelectionDAGBuilder::visitTerminatePad(const TerminatePadInst &TPI) {
   report_fatal_error("visitTerminatePad not yet implemented!");
 }
 
 void SelectionDAGBuilder::visitRet(const ReturnInst &I) {
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   auto &DL = DAG.getDataLayout();
   SDValue Chain = getControlRoot();

lib/CodeGen/TargetLoweringBase.cpp

#include "llvm/IR/Instruction.def"
   switch (static_cast<InstructionOpcodes>(Opcode)) {
   case Ret:            return 0;
   case Br:             return 0;
   case Switch:         return 0;
   case IndirectBr:     return 0;
   case Invoke:         return 0;
   case Resume:         return 0;
   case Unreachable:    return 0;
+  case CleanupEndPad:  return 0;
   case CleanupRet:     return 0;
   case CatchEndPad:  return 0;
   case CatchRet:       return 0;
   case CatchPad:     return 0;
   case TerminatePad: return 0;
   case CleanupPad:   return 0;
   case Add:            return ISD::ADD;
   case FAdd:           return ISD::FADD;

lib/CodeGen/WinEHPrepare.cpp

bool WinEHPrepare::runOnFunction(Function &Fn) {
   bool ForExplicitEH = false;
   for (BasicBlock &BB : Fn) {
     Instruction *First = BB.getFirstNonPHI();
     if (auto *LP = dyn_cast<LandingPadInst>(First)) {
       LPads.push_back(LP);
     } else if (First->isEHPad()) {
       if (!ForExplicitEH)
         EntryBlocks.push_back(&Fn.getEntryBlock());
-      if (!isa<CatchEndPadInst>(First))
+      if (!isa<CatchEndPadInst>(First) && !isa<CleanupEndPadInst>(First))
         EntryBlocks.push_back(&BB);
       ForExplicitEH = true;
     }
     if (auto *Resume = dyn_cast<ResumeInst>(BB.getTerminator()))
       Resumes.push_back(Resume);
   }
 
   if (ForExplicitEH)

 // to. If the unwind edge came from an invoke, return null.
 static const BasicBlock *getEHPadFromPredecessor(const BasicBlock *BB) {
   const TerminatorInst *TI = BB->getTerminator();
   if (isa<InvokeInst>(TI))
     return nullptr;
   if (isa<CatchPadInst>(TI) || isa<CatchEndPadInst>(TI) ||
       isa<TerminatePadInst>(TI))
     return BB;
+  if (auto *CEPI = dyn_cast<CleanupEndPadInst>(TI))
+    return CEPI->getCleanupPad()->getParent();
   return cast<CleanupReturnInst>(TI)->getCleanupPad()->getParent();
 }
 
 static void calculateExplicitStateNumbers(WinEHFuncInfo &FuncInfo,
                                           const BasicBlock &BB,
                                           int ParentState) {
   assert(BB.isEHPad());
   const Instruction *FirstNonPHI = BB.getFirstNonPHI();
void llvm::calculateWinCXXEHStateNumbers(const Function *ParentFn,
   // Return if it's already been done.
   if (!FuncInfo.EHPadStateMap.empty())
     return;
 
   bool IsExplicit = false;
   for (const BasicBlock &BB : *ParentFn) {
     if (!BB.isEHPad())
       continue;
+    const Instruction *FirstNonPHI = BB.getFirstNonPHI();
+    // Skip cleanupendpads; they are exits, not entries.
+    if (isa<CleanupEndPadInst>(FirstNonPHI))
+      continue;
     // Check if the EH Pad has no exceptional successors (i.e. it unwinds to
     // caller).  Cleanups are a little bit of a special case because their
     // control flow cannot be determined by looking at the pad but instead by
     // the pad's users.
     bool HasNoSuccessors = false;
-    const Instruction *FirstNonPHI = BB.getFirstNonPHI();
     if (FirstNonPHI->mayThrow()) {
       HasNoSuccessors = true;
     } else if (auto *CPI = dyn_cast<CleanupPadInst>(FirstNonPHI)) {
-      HasNoSuccessors =
-          CPI->use_empty() ||
-          cast<CleanupReturnInst>(CPI->user_back())->unwindsToCaller();
+      if (CPI->use_empty()) {
+        HasNoSuccessors = true;
+      } else {
+        const Instruction *User = CPI->user_back();
+        if (auto *CRI = dyn_cast<CleanupReturnInst>(User))
+          HasNoSuccessors = CRI->unwindsToCaller();
+        else
+          HasNoSuccessors = cast<CleanupEndPadInst>(User)->unwindsToCaller();
+      }
     }
 
     if (!HasNoSuccessors)
       continue;
     calculateExplicitStateNumbers(FuncInfo, BB, -1);
     IsExplicit = true;
   }
 
void WinEHPrepare::colorFunclets(Function &F,
 
   Worklist.push_back({EntryBlock, EntryBlock});
 
   while (!Worklist.empty()) {
     BasicBlock *Visiting;
     BasicBlock *Color;
     std::tie(Visiting, Color) = Worklist.pop_back_val();
     Instruction *VisitingHead = Visiting->getFirstNonPHI();
-    if (VisitingHead->isEHPad() && !isa<CatchEndPadInst>(VisitingHead)) {
+    if (VisitingHead->isEHPad() && !isa<CatchEndPadInst>(VisitingHead) &&
+        !isa<CleanupEndPadInst>(VisitingHead)) {
       // Mark this as a funclet head as a member of itself.
       FuncletBlocks[Visiting].insert(Visiting);
       // Queue exits with the parent color.
       for (User *Exit : VisitingHead->users()) {
         for (BasicBlock *Succ :
              successors(cast<Instruction>(Exit)->getParent())) {
           if (BlockColors[Succ].insert(Color).second) {
             Worklist.push_back({Succ, Color});
if (VisitingHead->isEHPad() && !isa<CatchEndPadInst>(VisitingHead) &&
       // visited with the color of the parent.
       if (!isa<TerminatePadInst>(VisitingHead))
         Color = Visiting;
     } else {
       // Note that this is a member of the given color.
       FuncletBlocks[Color].insert(Visiting);
       TerminatorInst *Terminator = Visiting->getTerminator();
       if (isa<CleanupReturnInst>(Terminator) ||
-          isa<CatchReturnInst>(Terminator)) {
+          isa<CatchReturnInst>(Terminator) ||
+          isa<CleanupEndPadInst>(Terminator)) {
         // These block's successors have already been queued with the parent
         // color.
         continue;
       }
     }
     for (BasicBlock *Succ : successors(Visiting)) {
       if (isa<CatchEndPadInst>(Succ->getFirstNonPHI())) {
         // The catchendpad needs to be visited with the parent's color, not
for (auto &Funclet : FuncletBlocks) {
     for (BasicBlock *BB : BlocksInFunclet) {
       TerminatorInst *TI = BB->getTerminator();
       // CatchPadInst and CleanupPadInst can't transfer control to a ReturnInst.
       bool IsUnreachableRet = isa<ReturnInst>(TI) && (CatchPad || CleanupPad);
       // The token consumed by a CatchReturnInst must match the funclet token.
       bool IsUnreachableCatchret = false;
       if (auto *CRI = dyn_cast<CatchReturnInst>(TI))
         IsUnreachableCatchret = CRI->getCatchPad() != CatchPad;
-      // The token consumed by a CleanupPadInst must match the funclet token.
+      // The token consumed by a CleanupReturnInst must match the funclet token.
       bool IsUnreachableCleanupret = false;
       if (auto *CRI = dyn_cast<CleanupReturnInst>(TI))
         IsUnreachableCleanupret = CRI->getCleanupPad() != CleanupPad;
-      if (IsUnreachableRet || IsUnreachableCatchret || IsUnreachableCleanupret) {
+      // The token consumed by a CleanupEndPadInst must match the funclet token.
+      bool IsUnreachableCleanupendpad = false;
+      if (auto *CEPI = dyn_cast<CleanupEndPadInst>(TI))
+        IsUnreachableCleanupendpad = CEPI->getCleanupPad() != CleanupPad;
+      if (IsUnreachableRet || IsUnreachableCatchret ||
+          IsUnreachableCleanupret || IsUnreachableCleanupendpad) {
         new UnreachableInst(BB->getContext(), TI);
         TI->eraseFromParent();
       }
     }
   }
 
   // Clean-up some of the mess we made by removing useles PHI nodes, trivial
   // branches, etc.
   for (Function::iterator FI = F.begin(), FE = F.end(); FI != FE;) {

lib/IR/AsmWriter.cpp

if (isa<BranchInst>(I) && cast<BranchInst>(I).isConditional()) {
     else
       Out << "to caller";
   } else if (const auto *CEPI = dyn_cast<CatchEndPadInst>(&I)) {
     Out << " unwind ";
     if (CEPI->hasUnwindDest())
       writeOperand(CEPI->getUnwindDest(), /*PrintType=*/true);
     else
       Out << "to caller";
+  } else if (const auto *CEPI = dyn_cast<CleanupEndPadInst>(&I)) {
+    Out << ' ';
+    writeOperand(CEPI->getCleanupPad(), /*PrintType=*/false);
+
+    Out << " unwind ";
+    if (CEPI->hasUnwindDest())
+      writeOperand(CEPI->getUnwindDest(), /*PrintType=*/true);
+    else
+      Out << "to caller";
   } else if (const CallInst *CI = dyn_cast<CallInst>(&I)) {
     // Print the calling convention being used.
     if (CI->getCallingConv() != CallingConv::C) {
       Out << " ";
       PrintCallingConv(CI->getCallingConv(), Out);
     }
 
     Operand = CI->getCalledValue();

lib/IR/Instruction.cpp

const char *Instruction::getOpcodeName(unsigned OpCode) {
   // Terminators
   case Ret:    return "ret";
   case Br:     return "br";
   case Switch: return "switch";
   case IndirectBr: return "indirectbr";
   case Invoke: return "invoke";
   case Resume: return "resume";
   case Unreachable: return "unreachable";
+  case CleanupEndPad: return "cleanupendpad";
   case CleanupRet: return "cleanupret";
   case CatchEndPad: return "catchendpad";
   case CatchRet: return "catchret";
   case CatchPad: return "catchpad";
   case TerminatePad: return "terminatepad";
 
   // Standard binary operators...
   case Add: return "add";
bool Instruction::isAtomic() const {
   }
 }
 
 bool Instruction::mayThrow() const {
   if (const CallInst *CI = dyn_cast<CallInst>(this))
     return !CI->doesNotThrow();
   if (const auto *CRI = dyn_cast<CleanupReturnInst>(this))
     return CRI->unwindsToCaller();
+  if (const auto *CEPI = dyn_cast<CleanupEndPadInst>(this))
+    return CEPI->unwindsToCaller();
   if (const auto *CEPI = dyn_cast<CatchEndPadInst>(this))
     return CEPI->unwindsToCaller();
   if (const auto *TPI = dyn_cast<TerminatePadInst>(this))
     return TPI->unwindsToCaller();
   return isa<ResumeInst>(this);
 }
 
 bool Instruction::mayReturn() const {

lib/IR/Instructions.cpp

void ResumeInst::setSuccessorV(unsigned idx, BasicBlock *NewSucc) {
   llvm_unreachable("ResumeInst has no successors!");
 }
 
 BasicBlock *ResumeInst::getSuccessorV(unsigned idx) const {
   llvm_unreachable("ResumeInst has no successors!");
 }
 
 //===----------------------------------------------------------------------===//
+//                        CleanupEndPadInst Implementation
+//===----------------------------------------------------------------------===//
+
+CleanupEndPadInst::CleanupEndPadInst(const CleanupEndPadInst &CEPI)
+    : TerminatorInst(CEPI.getType(), Instruction::CleanupEndPad,
+                     OperandTraits<CleanupEndPadInst>::op_end(this) -
+                         CEPI.getNumOperands(),
+                     CEPI.getNumOperands()) {
+  setInstructionSubclassData(CEPI.getSubclassDataFromInstruction());
+  setCleanupPad(CEPI.getCleanupPad());
+  if (BasicBlock *UnwindDest = CEPI.getUnwindDest())
+    setUnwindDest(UnwindDest);
+}
+
+void CleanupEndPadInst::init(CleanupPadInst *CleanupPad, BasicBlock *UnwindBB) {
+  setCleanupPad(CleanupPad);
+  if (UnwindBB) {
+    setInstructionSubclassData(getSubclassDataFromInstruction() | 1);
+    setUnwindDest(UnwindBB);
+  }
+}
+
+CleanupEndPadInst::CleanupEndPadInst(CleanupPadInst *CleanupPad,
+                                     BasicBlock *UnwindBB, unsigned Values,
+                                     Instruction *InsertBefore)
+    : TerminatorInst(Type::getVoidTy(CleanupPad->getContext()),
+                     Instruction::CleanupEndPad,
+                     OperandTraits<CleanupEndPadInst>::op_end(this) - Values,
+                     Values, InsertBefore) {
+  init(CleanupPad, UnwindBB);
+}
+
+CleanupEndPadInst::CleanupEndPadInst(CleanupPadInst *CleanupPad,
+                                     BasicBlock *UnwindBB, unsigned Values,
+                                     BasicBlock *InsertAtEnd)
+    : TerminatorInst(Type::getVoidTy(CleanupPad->getContext()),
+                     Instruction::CleanupEndPad,
+                     OperandTraits<CleanupEndPadInst>::op_end(this) - Values,
+                     Values, InsertAtEnd) {
+  init(CleanupPad, UnwindBB);
+}
+
+BasicBlock *CleanupEndPadInst::getSuccessorV(unsigned Idx) const {
+  assert(Idx == 0);
+  return getUnwindDest();
+}
+unsigned CleanupEndPadInst::getNumSuccessorsV() const {
+  return getNumSuccessors();
+}
+void CleanupEndPadInst::setSuccessorV(unsigned Idx, BasicBlock *B) {
+  assert(Idx == 0);
+  setUnwindDest(B);
+}
+
+//===----------------------------------------------------------------------===//
 //                        CleanupReturnInst Implementation
 //===----------------------------------------------------------------------===//
 
 CleanupReturnInst::CleanupReturnInst(const CleanupReturnInst &CRI)
     : TerminatorInst(CRI.getType(), Instruction::CleanupRet,
                      OperandTraits<CleanupReturnInst>::op_end(this) -
                          CRI.getNumOperands(),
                      CRI.getNumOperands()) {

 }
 
 InvokeInst *InvokeInst::cloneImpl() const {
   return new(getNumOperands()) InvokeInst(*this);
 }
 
 ResumeInst *ResumeInst::cloneImpl() const { return new (1) ResumeInst(*this); }
 
+CleanupEndPadInst *CleanupEndPadInst::cloneImpl() const {
+  return new (getNumOperands()) CleanupEndPadInst(*this);
+}
+
 CleanupReturnInst *CleanupReturnInst::cloneImpl() const {
   return new (getNumOperands()) CleanupReturnInst(*this);
 }
 
 CatchEndPadInst *CatchEndPadInst::cloneImpl() const {
   return new (getNumOperands()) CatchEndPadInst(*this);
 }
 

lib/IR/Verifier.cpp

#include "llvm/IR/Metadata.def"
   void visitAllocaInst(AllocaInst &AI);
   void visitExtractValueInst(ExtractValueInst &EVI);
   void visitInsertValueInst(InsertValueInst &IVI);
   void visitEHPadPredecessors(Instruction &I);
   void visitLandingPadInst(LandingPadInst &LPI);
   void visitCatchPadInst(CatchPadInst &CPI);
   void visitCatchEndPadInst(CatchEndPadInst &CEPI);
   void visitCleanupPadInst(CleanupPadInst &CPI);
+  void visitCleanupEndPadInst(CleanupEndPadInst &CEPI);
   void visitCleanupReturnInst(CleanupReturnInst &CRI);
   void visitTerminatePadInst(TerminatePadInst &TPI);
 
   void VerifyCallSite(CallSite CS);
   void verifyMustTailCall(CallInst &CI);
   bool PerformTypeCheck(Intrinsic::ID ID, Function *F, Type *Ty, int VT,
                         unsigned ArgNo, std::string &Suffix);
   bool VerifyIntrinsicType(Type *Ty, ArrayRef<Intrinsic::IITDescriptor> &Infos,
if (auto *II = dyn_cast<InvokeInst>(TI))
              "EH pad must be jumped to via an unwind edge", &I, II);
     else if (auto *CPI = dyn_cast<CatchPadInst>(TI))
       Assert(CPI->getUnwindDest() == BB && CPI->getNormalDest() != BB,
              "EH pad must be jumped to via an unwind edge", &I, CPI);
     else if (isa<CatchEndPadInst>(TI))
       ;
     else if (isa<CleanupReturnInst>(TI))
       ;
+    else if (isa<CleanupEndPadInst>(TI))
+      ;
     else if (isa<TerminatePadInst>(TI))
       ;
     else
       Assert(false, "EH pad must be jumped to via an unwind edge", &I, TI);
   }
 }
 
 void Verifier::visitLandingPadInst(LandingPadInst &LPI) {
Assert(F->hasPersonalityFn(),
          "CleanupPadInst needs to be in a function with a personality.", &CPI);
 
   // The cleanuppad instruction must be the first non-PHI instruction in the
   // block.
   Assert(BB->getFirstNonPHI() == &CPI,
          "CleanupPadInst not the first non-PHI instruction in the block.",
          &CPI);
 
-  CleanupReturnInst *FirstCRI = nullptr;
-  for (User *U : CPI.users())
+  User *FirstUser = nullptr;
+  BasicBlock *FirstUnwindDest = nullptr;
+  for (User *U : CPI.users()) {
+    BasicBlock *UnwindDest;
     if (CleanupReturnInst *CRI = dyn_cast<CleanupReturnInst>(U)) {
-      if (!FirstCRI)
-        FirstCRI = CRI;
-      else
-        Assert(CRI->getUnwindDest() == FirstCRI->getUnwindDest(),
-               "Cleanuprets from same cleanuppad have different exceptional "
-               "successors.",
-               FirstCRI, CRI);
-    }
+      UnwindDest = CRI->getUnwindDest();
+    } else {
+      UnwindDest = cast<CleanupEndPadInst>(U)->getUnwindDest();
+    }
+
+    if (!FirstUser) {
+      FirstUser = U;
+      FirstUnwindDest = UnwindDest;
+    } else {
+      Assert(UnwindDest == FirstUnwindDest,
+             "Cleanuprets/cleanupendpads from the same cleanuppad must "
+             "have the same unwind destination",
+             FirstUser, U);
+    }
 
   visitInstruction(CPI);
 }
 
+void Verifier::visitCleanupEndPadInst(CleanupEndPadInst &CEPI) {
+  visitEHPadPredecessors(CEPI);
+
+  BasicBlock *BB = CEPI.getParent();
+  Function *F = BB->getParent();
+  Assert(F->hasPersonalityFn(),
+         "CleanupEndPadInst needs to be in a function with a personality.",
+         &CEPI);
+
+  // The cleanupendpad instruction must be the first non-PHI instruction in the
+  // block.
+  Assert(BB->getFirstNonPHI() == &CEPI,
+         "CleanupEndPadInst not the first non-PHI instruction in the block.",
+         &CEPI);
+
+  if (BasicBlock *UnwindDest = CEPI.getUnwindDest()) {
+    Instruction *I = UnwindDest->getFirstNonPHI();
+    Assert(
+        I->isEHPad() && !isa<LandingPadInst>(I),
+        "CleanupEndPad must unwind to an EH block which is not a landingpad.",
+        &CEPI);
+  }
+
+  visitTerminatorInst(CEPI);
+}
+
 void Verifier::visitCleanupReturnInst(CleanupReturnInst &CRI) {
   if (BasicBlock *UnwindDest = CRI.getUnwindDest()) {
     Instruction *I = UnwindDest->getFirstNonPHI();
     Assert(I->isEHPad() && !isa<LandingPadInst>(I),
            "CleanupReturnInst must unwind to an EH block which is not a "
            "landingpad.",
            &CRI);
   }

lib/Transforms/Instrumentation/MemorySanitizer.cpp

void visitTerminatePad(TerminatePadInst &I) {
     // Nothing to do here.
   }
 
   void visitCatchEndPadInst(CatchEndPadInst &I) {
     DEBUG(dbgs() << "CatchEndPad: " << I << "\n");
     // Nothing to do here.
   }
 
+  void visitCleanupEndPadInst(CleanupEndPadInst &I) {
+    DEBUG(dbgs() << "CleanupEndPad: " << I << "\n");
+    // Nothing to do here.
+  }
+
   void visitGetElementPtrInst(GetElementPtrInst &I) {
     handleShadowOr(I);
   }
 
   void visitExtractValueInst(ExtractValueInst &I) {
     IRBuilder<> IRB(&I);
     Value *Agg = I.getAggregateOperand();
     DEBUG(dbgs() << "ExtractValue:  " << I << "\n");

lib/Transforms/Utils/InlineFunction.cpp

for (Function::iterator BB = FirstNewBlock, E = Caller->end(); BB != E;
     Instruction *I = BB->getFirstNonPHI();
     if (I->isEHPad()) {
       if (auto *CEPI = dyn_cast<CatchEndPadInst>(I)) {
         if (CEPI->unwindsToCaller()) {
           CatchEndPadInst::Create(CEPI->getContext(), UnwindDest, CEPI);
           CEPI->eraseFromParent();
           UpdatePHINodes(BB);
         }
+      } else if (auto *CEPI = dyn_cast<CleanupEndPadInst>(I)) {
+        if (CEPI->unwindsToCaller()) {
+          CleanupEndPadInst::Create(CEPI->getCleanupPad(), UnwindDest, CEPI);
+          CEPI->eraseFromParent();
+          UpdatePHINodes(BB);
+        }
       } else if (auto *TPI = dyn_cast<TerminatePadInst>(I)) {
         if (TPI->unwindsToCaller()) {
           SmallVector<Value *, 3> TerminatePadArgs;
           for (Value *Operand : TPI->operands())
             TerminatePadArgs.push_back(Operand);
           TerminatePadInst::Create(TPI->getContext(), UnwindDest, TPI);
           TPI->eraseFromParent();
           UpdatePHINodes(BB);

test/Assembler/invalid-OperatorConstraint.ll

 ; RUN: sed -e s/.T1:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK1 %s
 ; RUN: sed -e s/.T2:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK2 %s
 ; RUN: sed -e s/.T3:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK3 %s
 ; RUN: sed -e s/.T4:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK4 %s
 ; RUN: sed -e s/.T5:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK5 %s
 ; RUN: sed -e s/.T6:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK6 %s
+; RUN: sed -e s/.T7:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK7 %s
+; RUN: sed -e s/.T8:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK8 %s
+; RUN: sed -e s/.T9:// %s | not llvm-as -disable-output 2>&1 | FileCheck --check-prefix=CHECK9 %s
 
 ;T1: define void @f() {
 ;T1:   entry:
 ;T1:     ; operator constraint requires an operator
 ;T1:     catchret undef to label %entry
 ;T1:     ; CHECK1: [[@LINE-1]]:15: error: Catchpad value required in this position
 ;T1: }
 

 ;T6:   entry:
 ;T6:     ; cleanupret's first operand's operator must be cleanuppad
 ;T6:     ; (forward reference case)
 ;T6:     cleanupret %x unwind label %next
 ;T6:     ; CHECK6: [[@LINE-1]]:17: error: '%x' is not a cleanuppad
 ;T6:   next:
 ;T6:     %x = catchpad [] to label %entry unwind label %next
 ;T6: }
+
+;T7: define void @f() {
+;T7:   entry:
+;T7:     ; operator constraint requires an operator
+;T7:     cleanupendpad undef unwind to caller
+;T7:     ; CHECK7: [[@LINE-1]]:20: error: Cleanuppad value required in this position
+;T7: }
+
+;T8: define void @f() {
+;T8:   entry:
+;T8:     %x = catchpad []
+;T8:             to label %next unwind label %entry
+;T8:   next:
+;T8:     ; cleanupret first operand's operator must be cleanuppad
+;T8:     cleanupendpad %x unwind label next
+;T8:     ; CHECK8: [[@LINE-1]]:20: error: '%x' is not a cleanuppad
+;T8: }
+
+;T9: define void @f() {
+;T9:   entry:
+;T9:     ; cleanupret's first operand's operator must be cleanuppad
+;T9:     ; (forward reference case)
+;T9:     cleanupendpad %x unwind label %next
+;T9:     ; CHECK9: [[@LINE-1]]:20: error: '%x' is not a cleanuppad
+;T9:   next:
+;T9:     %x = catchpad [] to label %entry unwind label %next
+;T9: }

test/Feature/exception.ll

entry:
   br label %try.cont
 
 try.cont:
   invoke void @_Z3quxv() optsize
           to label %try.cont unwind label %bb
 bb:
   catchendpad unwind to caller
 }
+
+define void @cleanupendpad0() personality i32 (...)* @__gxx_personality_v0 {
+entry:
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+pad:
+  %cp = cleanuppad [i7 4]
+  invoke void @_Z3quxv() optsize
+          to label %stop unwind label %endpad
+stop:
+  unreachable
+endpad:
+  cleanupendpad %cp unwind label %pad
+exit:
+  ret void
+}
+
+; forward ref by name
+define void @cleanupendpad1() personality i32 (...)* @__gxx_personality_v0 {
+entry:
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+endpad:
+  cleanupendpad %cp unwind to caller
+pad:
+  %cp = cleanuppad []
+  invoke void @_Z3quxv() optsize
+          to label %stop unwind label %endpad
+stop:
+  unreachable
+exit:
+  ret void
+}
+
+; forward ref by ID
+define void @cleanupendpad2() personality i32 (...)* @__gxx_personality_v0 {
+entry:
+  invoke void @_Z3quxv() optsize
+          to label %exit unwind label %pad
+endpad:
+  cleanupendpad %0 unwind label %pad
+pad:
+  %0 = cleanuppad []
+  invoke void @_Z3quxv() optsize
+          to label %stop unwind label %endpad
+stop:
+  unreachable
+exit:
+  ret void
+}

tools/llvm-bcanalyzer/llvm-bcanalyzer.cpp

default: return nullptr;
       STRINGIFY_CODE(FUNC_CODE, INST_RET)
       STRINGIFY_CODE(FUNC_CODE, INST_BR)
       STRINGIFY_CODE(FUNC_CODE, INST_SWITCH)
       STRINGIFY_CODE(FUNC_CODE, INST_INVOKE)
       STRINGIFY_CODE(FUNC_CODE, INST_UNREACHABLE)
       STRINGIFY_CODE(FUNC_CODE, INST_CLEANUPRET)
       STRINGIFY_CODE(FUNC_CODE, INST_CATCHRET)
       STRINGIFY_CODE(FUNC_CODE, INST_CATCHPAD)
+      STRINGIFY_CODE(FUNC_CODE, INST_CLEANUPENDPAD)
       STRINGIFY_CODE(FUNC_CODE, INST_CATCHENDPAD)
       STRINGIFY_CODE(FUNC_CODE, INST_TERMINATEPAD)
       STRINGIFY_CODE(FUNC_CODE, INST_PHI)
       STRINGIFY_CODE(FUNC_CODE, INST_ALLOCA)
       STRINGIFY_CODE(FUNC_CODE, INST_LOAD)
       STRINGIFY_CODE(FUNC_CODE, INST_VAARG)
       STRINGIFY_CODE(FUNC_CODE, INST_STORE)
       STRINGIFY_CODE(FUNC_CODE, INST_EXTRACTVAL)