implement C++ dr388 for the Itanium C++ ABI: proper handling of catching exceptions by reference

[rsmith]

Summary

This patch fixes three issues with how Clang emits code to catch exceptions by reference:

1. We would sometimes bind the reference directly to the exception object when a temporary is required (when a pointer-to-member is converted by a qualification conversion but not adjusted).
2. We would sometimes bind the reference to a temporary when a direct binding is required (when catching a pointer-to-class-object where no derived-to-base or qualification conversion is required).
3. We would incorrectly allow a non-const (or volatile) reference to catch an exception of a different type that can be converted to the right type.

A single strategy is used to fix all three issues: where necessary, we extract the type_info of the exception object from the __cxa_exception header, and compare it directly against the reference's pointee type_info. For cases #1 and #2, this tells us whether to bind to a temporary object or to the exception object; for case #3, if the types don't match and the reference can't bind to a temporary, we rethrow the exception through the remaining catch clauses of the try statement.

This patch is insufficiently tested (no changes to the test suite, and no testing has been done whatsoever for the ARM64 non-unique type_info case), but I'm posting it now to gauge whether this is a direction we want to pursue for DR388 (as opposed to continuing to get this wrong until the Itanium C++ ABI has a chance to take an ABI break to properly handle catch-by-reference).

[Despite the different approach to determining catchability, the MS ABI appears to have exactly the same bugs (it too does not distinguish between catch by value and catch by reference), and I would assume the same approach would also work there, assuming the actual type of the exception object can be discerned, but I've not looked at all at implementing this in that ABI.]

[rsmith]

lib/CodeGen/ItaniumCXXABI.cpp

3878

There are a lot of other optimizations possible here. For example: if there are no later catches, and no enclosing EH scopes, we can just resume (before the __cxa_begin_catch) call. Or if the first such later catch will definitely catch the exception (eg, catch (int*&) {} catch (int*) {}), we could branch directly to it.

I'm inclined to think we should keep the frontend emission simpler and implement these cases as EHABI-aware middle-end optimization passes if we care enough about them.

[rnk]

lib/CodeGen/ItaniumCXXABI.cpp

3867

Based on our conversation, this fix feels a bit heroic. I think it would be better to find a way to put the const-ness in the LSDA and let new versions of the EH runtime deal with this.

[rjmccall]

This is definitely something that the personality function should handle. If we want to do heroic things in the absence of personality function support, we can, but the code should at least be written to be conditional on personality support.

If we can rev the personality function, we should also fix the longstanding problem with references to pointers to classes.

[rsmith]

In D42092#983892, @rjmccall wrote:

This is definitely something that the personality function should handle. If we want to do heroic things in the absence of personality function support, we can, but the code should at least be written to be conditional on personality support.

I suppose the question is, do we want to do such heroic things?

Also, we don't have a personality function that can get this right, nor even a concrete design proposed for one. It doesn't seem too hard -- we'd need to extend the EH table to indicate catch-by-reference -- but there's nothing for this to be made conditional on right now. I can easily add a CodeGenOption that turns this all off, if you'd like.

If we can rev the personality function, we should also fix the longstanding problem with references to pointers to classes.

Which issue is that? (And does this patch fail to address it? It should prevent catching Derived* by (non-const) Base *& while still allowing catch by Base *const&, if that's the issue you're referring to.)

[rjmccall]

In D42092#1057881, @rsmith wrote:

In D42092#983892, @rjmccall wrote:

This is definitely something that the personality function should handle. If we want to do heroic things in the absence of personality function support, we can, but the code should at least be written to be conditional on personality support.

I suppose the question is, do we want to do such heroic things?

Also, we don't have a personality function that can get this right, nor even a concrete design proposed for one. It doesn't seem too hard -- we'd need to extend the EH table to indicate catch-by-reference -- but there's nothing for this to be made conditional on right now. I can easily add a CodeGenOption that turns this all off, if you'd like.

Well, I don't have a use case for an option like that. If we're going to add the code to support this even without a cooperative personality, I think we can enable it unconditionally for now.

Issue #3 is tricky; it's arguably not okay to force a landing at a landing pad if we're not actually catching anything.

For what it's worth, I don't think the Itanium ABI needs any changes here. The existence of any particular personality function, and that function's expectations about the LSDA, are not official ABI. __gxx_personality_v0 is a sort of private ABI, a standard solution offered by libsupc++ / libc++abi for the convenience of compilers, but we could totally just write a new personality and add it to compiler-rt (or libc++abi, enabled by deployment-target-specific checks) if that was the best solution.

If we can rev the personality function, we should also fix the longstanding problem with references to pointers to classes.

Which issue is that? (And does this patch fail to address it? It should prevent catching Derived* by (non-const) Base *& while still allowing catch by Base *const&, if that's the issue you're referring to.)

Yes, that's the issue I'm referring to, and yeah, I see that you're addressing it here.

[rsmith]

In D42092#1058772, @rjmccall wrote:

Issue #3 is tricky; it's arguably not okay to force a landing at a landing pad if we're not actually catching anything.

I think the only way this is observable (at least within the bounds of well-defined C++ code) is whether -- and how much -- the stack is unwound before a call to terminate() in the case where an exception is not caught.

[except.handle]p9 says: "If no matching handler is found, the function std::terminate() is called; whether or not the stack is unwound before this call to std::terminate() is implementation-defined (18.5.1)."
[except.terminate]p2 says: " In the situation where no matching handler is found, it is implementation-defined whether or not the stack is unwound before std::terminate() is called. In the situation where the search for a handler (18.3) encounters the outermost block of a function with a non-throwing exception specification (18.4), it is implementation-defined whether the stack is unwound, unwound partially, or not unwound at all before std::terminate() is called. In all other situations, the stack shall not be unwound before std::terminate() is called."

So I think you're right; this solution to issue #3 is non-conforming, because it will result in the stack being partially unwound on the path to a call to std::terminate() for an unhandled exception. Even if we changed the rule for terminate to allow partial unwinding when no handler is found, this patch would still give a QoI regression for that case (because the call to the terminate_handler would happen after some unwinding had been performed).

I'm not particularly interested in pushing this patch further if it can't actually make catch-by-reference work properly. Let's drop this for now and encourage interested parties to work on an alternative personality function.

[rjmccall]

In D42092#1058841, @rsmith wrote:

In D42092#1058772, @rjmccall wrote:

Issue #3 is tricky; it's arguably not okay to force a landing at a landing pad if we're not actually catching anything.

I think the only way this is observable (at least within the bounds of well-defined C++ code) is whether -- and how much -- the stack is unwound before a call to terminate() in the case where an exception is not caught.

[except.handle]p9 says: "If no matching handler is found, the function std::terminate() is called; whether or not the stack is unwound before this call to std::terminate() is implementation-defined (18.5.1)."
[except.terminate]p2 says: " In the situation where no matching handler is found, it is implementation-defined whether or not the stack is unwound before std::terminate() is called. In the situation where the search for a handler (18.3) encounters the outermost block of a function with a non-throwing exception specification (18.4), it is implementation-defined whether the stack is unwound, unwound partially, or not unwound at all before std::terminate() is called. In all other situations, the stack shall not be unwound before std::terminate() is called."

So I think you're right; this solution to issue #3 is non-conforming, because it will result in the stack being partially unwound on the path to a call to std::terminate() for an unhandled exception. Even if we changed the rule for terminate to allow partial unwinding when no handler is found, this patch would still give a QoI regression for that case (because the call to the terminate_handler would happen after some unwinding had been performed).

I'm not particularly interested in pushing this patch further if it can't actually make catch-by-reference work properly. Let's drop this for now and encourage interested parties to work on an alternative personality function.

Agreed.

lib/CodeGen/CGCXXABI.h

 // of this implement code generation for specific C++ ABIs.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_CLANG_LIB_CODEGEN_CGCXXABI_H
 #define LLVM_CLANG_LIB_CODEGEN_CGCXXABI_H
 
 #include "CodeGenFunction.h"
+#include "CGCleanup.h"
 #include "clang/Basic/LLVM.h"
 
 namespace llvm {
 class Constant;
 class Type;
 class Value;
 class CallInst;
 }
public:
   virtual void emitThrow(CodeGenFunction &CGF, const CXXThrowExpr *E) = 0;
   virtual llvm::GlobalVariable *getThrowInfo(QualType T) { return nullptr; }
 
   /// \brief Determine whether it's possible to emit a vtable for \p RD, even
   /// though we do not know that the vtable has been marked as used by semantic
   /// analysis.
   virtual bool canSpeculativelyEmitVTable(const CXXRecordDecl *RD) const = 0;
 
-  virtual void emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C) = 0;
+  virtual void
+  emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C,
+                 ArrayRef<EHCatchScope::Handler> LaterHandlers) = 0;
 
   virtual llvm::CallInst *
   emitTerminateForUnexpectedException(CodeGenFunction &CGF,
                                       llvm::Value *Exn);
 
   virtual llvm::Constant *getAddrOfRTTIDescriptor(QualType Ty) = 0;
   virtual CatchTypeInfo
   getAddrOfCXXCatchHandlerType(QualType Ty, QualType CatchHandlerType) = 0;

lib/CodeGen/CGCleanup.h

public:
   // FIXME: it's better to always call clearHandlerBlocks in DTOR and have a
   // 'takeHandler' or some such function which removes ownership from the
   // EHCatchScope object if the handlers should live longer than EHCatchScope.
   void clearHandlerBlocks() {
     for (unsigned I = 0, N = getNumHandlers(); I != N; ++I)
       delete getHandler(I).Block;
   }
 
-  typedef const Handler *iterator;
-  iterator begin() const { return getHandlers(); }
-  iterator end() const { return getHandlers() + getNumHandlers(); }
+  typedef const Handler *const_iterator;
+  const_iterator begin() const { return getHandlers(); }
+  const_iterator end() const { return getHandlers() + getNumHandlers(); }
+
+  typedef Handler *iterator;
+  iterator begin() { return getHandlers(); }
+  iterator end() { return getHandlers() + getNumHandlers(); }
 
   static bool classof(const EHScope *Scope) {
     return Scope->getKind() == Catch;
   }
 };
 
 /// A cleanup scope which generates the cleanup blocks lazily.
 class LLVM_ALIGNAS(/*alignof(uint64_t)*/ 8) EHCleanupScope : public EHScope {

lib/CodeGen/CGException.cpp

void CodeGenFunction::EnterCXXTryStmt(const CXXTryStmt &S, bool IsFnTryBlock) {
   unsigned NumHandlers = S.getNumHandlers();
   EHCatchScope *CatchScope = EHStack.pushCatch(NumHandlers);
 
   for (unsigned I = 0; I != NumHandlers; ++I) {
     const CXXCatchStmt *C = S.getHandler(I);
 
     llvm::BasicBlock *Handler = createBasicBlock("catch");
     if (C->getExceptionDecl()) {
-      // FIXME: Dropping the reference type on the type into makes it
-      // impossible to correctly implement catch-by-reference
-      // semantics for pointers.  Unfortunately, this is what all
-      // existing compilers do, and it's not clear that the standard
-      // personality routine is capable of doing this right.  See C++ DR 388:
-      //   http://www.open-std.org/jtc1/sc22/wg21/docs/cwg_active.html#388
+      // Note that dropping the reference type may result in us catching
+      // exceptions we're not supposed to catch: for instance, a 'T&' handler
+      // cannot catch an exception that requires conversion to 'T', but a
+      // 'const T&' handler can; see C++ DR 388. The ABI-specific emission of
+      // catch handlers is expected to deal with this by rethrowing exceptions
+      // it doesn't want to catch.
       Qualifiers CaughtTypeQuals;
       QualType CaughtType = CGM.getContext().getUnqualifiedArrayType(
           C->getCaughtType().getNonReferenceType(), CaughtTypeQuals);
 
       CatchTypeInfo TypeInfo{nullptr, 0};
       if (CaughtType->isObjCObjectPointerType())
         TypeInfo.RTTI = CGM.getObjCRuntime().GetEHType(CaughtType);
       else
for (unsigned I = NumHandlers; I != 0; --I) {
 
     // Enter a cleanup scope, including the catch variable and the
     // end-catch.
     RunCleanupsScope CatchScope(*this);
 
     // Initialize the catch variable and set up the cleanups.
     SaveAndRestore<llvm::Instruction *> RestoreCurrentFuncletPad(
         CurrentFuncletPad);
-    CGM.getCXXABI().emitBeginCatch(*this, C);
+    CGM.getCXXABI().emitBeginCatch(
+        *this, C, llvm::makeArrayRef(Handlers.begin() + I, Handlers.end()));
 
     // Emit the PGO counter increment.
     incrementProfileCounter(C);
 
     // Perform the body of the catch.
     EmitStmt(C->getHandlerBlock());
 
     // [except.handle]p11:

lib/CodeGen/ItaniumCXXABI.cpp

public:
   CharUnits getAlignmentOfExnObject() {
     auto align = CGM.getContext().getTargetDefaultAlignForAttributeAligned();
     return CGM.getContext().toCharUnitsFromBits(align);
   }
 
   void emitRethrow(CodeGenFunction &CGF, bool isNoReturn) override;
   void emitThrow(CodeGenFunction &CGF, const CXXThrowExpr *E) override;
 
-  void emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C) override;
+  void InitCatchParam(CodeGenFunction &CGF, const VarDecl &CatchParam,
+                      Address ParamAddr, SourceLocation Loc,
+                      ArrayRef<EHCatchScope::Handler> LaterHandlers);
+  void emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C,
+                      ArrayRef<EHCatchScope::Handler> LaterHandlers) override;
 
   llvm::CallInst *
   emitTerminateForUnexpectedException(CodeGenFunction &CGF,
                                       llvm::Value *Exn) override;
 
   void EmitFundamentalRTTIDescriptor(QualType Type, bool DLLExport);
   void EmitFundamentalRTTIDescriptors(bool DLLExport);
   llvm::Constant *getAddrOfRTTIDescriptor(QualType Ty) override;
public:
   };
 
   /// BuildTypeInfo - Build the RTTI type info struct for the given type.
   ///
   /// \param Force - true to force the creation of this RTTI value
   /// \param DLLExport - true to mark the RTTI value as DLLExport
   llvm::Constant *BuildTypeInfo(QualType Ty, bool Force = false,
                                 bool DLLExport = false);
+
+  llvm::Value *EmitCompareTypeInfo(CodeGenFunction &CGF, QualType Ty,
+                                   llvm::Value *TypeInfo);
 };
 }
 
 llvm::GlobalVariable *ItaniumRTTIBuilder::GetAddrOfTypeName(
     QualType Ty, llvm::GlobalVariable::LinkageTypes Linkage) {
   SmallString<256> Name;
   llvm::raw_svector_ostream Out(Name);
   CGM.getCXXABI().getMangleContext().mangleCXXRTTIName(Ty, Out);
static llvm::Value *CallBeginCatch(CodeGenFunction &CGF,
   llvm::CallInst *call =
     CGF.EmitNounwindRuntimeCall(getBeginCatchFn(CGF.CGM), Exn);
 
   CGF.EHStack.pushCleanup<CallEndCatch>(NormalAndEHCleanup, EndMightThrow);
 
   return call;
 }
 
+llvm::Value *ItaniumRTTIBuilder::EmitCompareTypeInfo(CodeGenFunction &CGF,
+                                                     QualType Ty,
+                                                     llvm::Value *TypeInfo) {
+  // Start by just comparing the address of the type_info.
+  llvm::Value *Equal = CGF.Builder.CreateICmpEQ(BuildTypeInfo(Ty), TypeInfo);
+
+  // For types with unique type_info, the address comparison is sufficient.
+  llvm::GlobalVariable::LinkageTypes Linkage = getTypeInfoLinkage(CGM, Ty);
+  if (CXXABI.classifyRTTIUniqueness(Ty, Linkage) == ItaniumCXXABI::RUK_Unique)
+    return Equal;
+
+  // Otherwise, we need to compare the type name string.
+  llvm::BasicBlock *CurBlock = CGF.Builder.GetInsertBlock();
+  llvm::BasicBlock *CmpTypenameBlock = CGF.createBasicBlock("exc.typename.cmp");
+  llvm::BasicBlock *CmpDoneBlock = CGF.createBasicBlock("exc.typename.done");
+  CGF.Builder.CreateCondBr(Equal, CmpDoneBlock, CmpTypenameBlock);
+
+  CGF.EmitBlock(CmpTypenameBlock);
+
+  // The type name is the second pointer member, just after the vptr. Remove
+  // the high "requires string comparison" bit.
+  // FIXME: This is ARM64-specific.
+  llvm::Value *TypeNameField = CGF.Builder.CreateConstGEP1_32(
+      CGF.Builder.CreateBitCast(TypeInfo, CGF.CGM.Int8PtrPtrTy), 1);
+  TypeNameField = CGF.Builder.CreatePtrToInt(TypeNameField, CGM.Int64Ty);
+  llvm::Constant *Mask =
+      llvm::ConstantInt::get(CGM.Int64Ty, (uint64_t(1) << 63) - 1);
+  TypeNameField = CGF.Builder.CreateAnd(TypeNameField, Mask);
+  TypeNameField = CGF.Builder.CreateIntToPtr(TypeNameField, CGM.Int8PtrTy);
+
+  // Emit a strcmp between the type_info names.
+  Address TypeInfoStringAddr(TypeNameField, CGF.getPointerAlign());
+  llvm::Value *TypeInfoString = CGF.Builder.CreateLoad(TypeInfoStringAddr);
+  llvm::Value *Strcmp = CGM.CreateRuntimeFunction(
+      llvm::FunctionType::get(CGM.IntTy, {CGM.Int8PtrTy, CGM.Int8PtrTy},
+                              /*IsVarArgs=*/false),
+      "strcmp");
+  llvm::CallInst *StrcmpValue = CGF.EmitNounwindRuntimeCall(
+      Strcmp, {TypeInfoString, GetAddrOfTypeName(Ty, Linkage)});
+  llvm::Value *StrcmpResult = CGF.Builder.CreateICmpEQ(
+      StrcmpValue, llvm::ConstantInt::get(CGM.IntTy, 0));
+  CGF.EmitBranch(CmpDoneBlock);
+
+  CGF.EmitBlock(CmpDoneBlock);
+  llvm::PHINode *PHI = CGF.Builder.CreatePHI(Equal->getType(), 2);
+  PHI->addIncoming(Equal, CurBlock);
+  PHI->addIncoming(StrcmpResult, CmpTypenameBlock);
+  return PHI;
+}
+
 /// A "special initializer" callback for initializing a catch
 /// parameter during catch initialization.
-static void InitCatchParam(CodeGenFunction &CGF,
-                           const VarDecl &CatchParam,
-                           Address ParamAddr,
-                           SourceLocation Loc) {
+void ItaniumCXXABI::InitCatchParam(
+    CodeGenFunction &CGF, const VarDecl &CatchParam, Address ParamAddr,
+    SourceLocation Loc, ArrayRef<EHCatchScope::Handler> LaterHandlers) {
   // Load the exception from where the landing pad saved it.
   llvm::Value *Exn = CGF.getExceptionFromSlot();
 
   CanQualType CatchType =
     CGF.CGM.getContext().getCanonicalType(CatchParam.getType());
   llvm::Type *LLVMCatchTy = CGF.ConvertTypeForMem(CatchType);
 
   // If we're catching by reference, we can just cast the object
   // pointer to the appropriate pointer.
-  if (isa<ReferenceType>(CatchType)) {
-    QualType CaughtType = cast<ReferenceType>(CatchType)->getPointeeType();
+  if (auto *RT = dyn_cast<ReferenceType>(CatchType)) {
+    QualType CaughtType = RT->getPointeeType();
     bool EndCatchMightThrow = CaughtType->isRecordType();
+    bool CanBindToTemporary =
+        CaughtType.isConstQualified() && !CaughtType.isVolatileQualified();
+
+    // Determine whether a conversion was applied to the exception in
+    // forming the reference binding.
+    llvm::Value *MaybeTypeMatch = nullptr;
+    if (CaughtType->isPointerType() ||
+        (CaughtType->isMemberPointerType() && !CanBindToTemporary)) {
+      // Locate the type_info for the exception.
+      unsigned HeaderSize =
+          CGF.CGM.getTargetCodeGenInfo().getSizeOfCxaException();
+      llvm::Value *TypeInfoPtrPtr = CGF.Builder.CreateBitCast(
+          CGF.Builder.CreateConstGEP1_32(Exn, -HeaderSize),
+          CGF.CGM.VoidPtrPtrTy);
+      llvm::Value *TypeInfoPtr = CGF.Builder.CreateLoad(
+          Address(TypeInfoPtrPtr, CGF.getPointerAlign()), "exn.type_info");
+      MaybeTypeMatch = ItaniumRTTIBuilder(*this).EmitCompareTypeInfo(
+          CGF, CaughtType.getUnqualifiedType(), TypeInfoPtr);
+    }
 
     // __cxa_begin_catch returns the adjusted object pointer.
     llvm::Value *AdjustedExn = CallBeginCatch(CGF, Exn, EndCatchMightThrow);
 
-    // We have no way to tell the personality function that we're
-    // catching by reference, so if we're catching a pointer,
-    // __cxa_begin_catch will actually return that pointer by value.
-    if (const PointerType *PT = dyn_cast<PointerType>(CaughtType)) {
-      QualType PointeeType = PT->getPointeeType();
-
-      // When catching by reference, generally we should just ignore
-      // this by-value pointer and use the exception object instead.
-      if (!PointeeType->isRecordType()) {
-
-        // Exn points to the struct _Unwind_Exception header, which
-        // we have to skip past in order to reach the exception data.
-        unsigned HeaderSize =
+    // If the reference is to a pointer or pointer-to-member type and
+    // is not const (non-volatile) reference, and was type-converted, resume
+    // unwinding rather than re-entering this catch block. Note that we need
+    // subsequent catches in the same try statement to catch such a rethrow.
+    // See C++ DR 388.
+    if (!CanBindToTemporary && MaybeTypeMatch) {
+      llvm::BasicBlock *MismatchBlock = CGF.createBasicBlock("exn.mismatch");
+      llvm::BasicBlock *CatchBlock = CGF.createBasicBlock("exn.match");
+      CGF.Builder.CreateCondBr(MaybeTypeMatch, CatchBlock, MismatchBlock);
+
+      // Pass the exception off to the C++ ABI library to deal with.
+      CGF.EmitBlock(MismatchBlock);
+      if (LaterHandlers.empty()) {
+        emitRethrow(CGF, true);
+      } else {
+        // FIXME: We could filter out definitely-unreachable catch handlers.
+        EHCatchScope *LaterCatchScope =
+            CGF.EHStack.pushCatch(LaterHandlers.size());
+        std::uninitialized_copy(LaterHandlers.begin(), LaterHandlers.end(),
+                                LaterCatchScope->begin());
+        emitRethrow(CGF, true);
+        CGF.popCatchScope();
+      }
+
+      CGF.EmitBlock(CatchBlock);
+      MaybeTypeMatch = nullptr;
+    }
+
+    // __cxa_begin_catch returns pointers by value, so we need to find the
+    // actual exception object ourselves. Exn points to the struct
+    // _Unwind_Exception header, which we have to skip past in order to reach
+    // the exception data.
+    // We assume the ABI library returns a pointer to the exception object
+    // for non-pointer exceptions (if no conversion is applied).
+    llvm::Value *ExnObject;
+    if (CaughtType->isPointerType()) {
           CGF.CGM.getTargetCodeGenInfo().getSizeOfUnwindException();
-        AdjustedExn = CGF.Builder.CreateConstGEP1_32(Exn, HeaderSize);
-
-      // However, if we're catching a pointer-to-record type that won't
-      // work, because the personality function might have adjusted
-      // the pointer.  There's actually no way for us to fully satisfy
-      // the language/ABI contract here:  we can't use Exn because it
-      // might have the wrong adjustment, but we can't use the by-value
-      // pointer because it's off by a level of abstraction.
-      //
-      // The current solution is to dump the adjusted pointer into an
-      // alloca, which breaks language semantics (because changing the
-      // pointer doesn't change the exception) but at least works.
-      // The better solution would be to filter out non-exact matches
-      // and rethrow them, but this is tricky because the rethrow
-      // really needs to be catchable by other sites at this landing
-      // pad.  The best solution is to fix the personality function.
-      } else {
+      llvm::Value *ExnObj = CGF.Builder.CreateConstGEP1_32(Exn, HeaderSize);
+      ExnObject = CGF.Builder.CreateBitCast(ExnObj, LLVMCatchTy, "exn.orig");
+      ExnObject =
+          CGF.Builder.CreateBitCast(AdjustedExn, LLVMCatchTy, "exn.byref");
+    }
+
+    // If a conversion may have been performed, we need to materialize a local
+    // temporary to hold the converted value (even if the conversion didn't
+    // actually change the value, it must have a different address from that
+    // of the exception object).
+    if (MaybeTypeMatch) {
+      assert(CaughtType->isPointerType() || CaughtType->isMemberPointerType());
+      llvm::BasicBlock *PreBlock = CGF.Builder.GetInsertBlock();
+      llvm::BasicBlock *ConvBlock = CGF.createBasicBlock("exn.conv");
+      llvm::BasicBlock *ContBlock = CGF.createBasicBlock("exn.cont");
+      CGF.Builder.CreateCondBr(MaybeTypeMatch, ContBlock, ConvBlock);
+
+      // If the pointer was converted, the personality function might have
+      // adjusted it, and we need to materialize a separate temporary
+      // ourselves.
+      CGF.EmitBlock(ConvBlock);
+      llvm::Value *Temporary;
+      {
         // Pull the pointer for the reference type off.
         llvm::Type *PtrTy =
           cast<llvm::PointerType>(LLVMCatchTy)->getElementType();
 
         // Create the temporary and write the adjusted pointer into it.
-        Address ExnPtrTmp =
-          CGF.CreateTempAlloca(PtrTy, CGF.getPointerAlign(), "exn.byref.tmp");
-        llvm::Value *Casted = CGF.Builder.CreateBitCast(AdjustedExn, PtrTy);
-        CGF.Builder.CreateStore(Casted, ExnPtrTmp);
+        CharUnits Align = CGF.getNaturalTypeAlignment(CaughtType);
+        Address ExnPtrTmp = CGF.CreateTempAlloca(PtrTy, Align, "exn.byref.tmp");
+        llvm::Value *Value;
+        if (CaughtType->isPointerType())
+          Value = CGF.Builder.CreateBitCast(AdjustedExn, PtrTy);
+        else {
+          Address ExnAddr(AdjustedExn, Align);
+          ExnAddr = CGF.Builder.CreateElementBitCast(ExnAddr, PtrTy);
+          Value = CGF.EmitLoadOfScalar(ExnAddr, false, CaughtType, Loc);
+        }
+        CGF.EmitStoreOfScalar(Value, ExnPtrTmp, false, CaughtType);
 
         // Bind the reference to the temporary.
-        AdjustedExn = ExnPtrTmp.getPointer();
+        Temporary = ExnPtrTmp.getPointer();
+        CGF.EmitBranch(ContBlock);
       }
+
+      CGF.EmitBlock(ContBlock);
+      llvm::PHINode *Result = CGF.Builder.CreatePHI(LLVMCatchTy, 2);
+      Result->addIncoming(ExnObject, PreBlock);
+      Result->addIncoming(Temporary, ConvBlock);
+      ExnObject = Result;
     }
 
-    llvm::Value *ExnCast =
-      CGF.Builder.CreateBitCast(AdjustedExn, LLVMCatchTy, "exn.byref");
-    CGF.Builder.CreateStore(ExnCast, ParamAddr);
+    CGF.Builder.CreateStore(ExnObject, ParamAddr);
     return;
   }
 
   // Scalars and complexes.
   TypeEvaluationKind TEK = CGF.getEvaluationKind(CatchType);
   if (TEK != TEK_Aggregate) {
     llvm::Value *AdjustedExn = CallBeginCatch(CGF, Exn, false);
 
void ItaniumCXXABI::InitCatchParam(
   opaque.pop();
 
   // Finally we can call __cxa_begin_catch.
   CallBeginCatch(CGF, Exn, true);
 }
 
 /// Begins a catch statement by initializing the catch variable and
 /// calling __cxa_begin_catch.
-void ItaniumCXXABI::emitBeginCatch(CodeGenFunction &CGF,
-                                   const CXXCatchStmt *S) {
+void ItaniumCXXABI::emitBeginCatch(
+    CodeGenFunction &CGF, const CXXCatchStmt *S,
+    ArrayRef<EHCatchScope::Handler> LaterHandlers) {
   // We have to be very careful with the ordering of cleanups here:
   //   C++ [except.throw]p4:
   //     The destruction [of the exception temporary] occurs
   //     immediately after the destruction of the object declared in
   //     the exception-declaration in the handler.
   //
   // So the precise ordering is:
   //   1.  Construct catch variable.
void ItaniumCXXABI::emitBeginCatch(
   if (!CatchParam) {
     llvm::Value *Exn = CGF.getExceptionFromSlot();
     CallBeginCatch(CGF, Exn, true);
     return;
   }
 
   // Emit the local.
   CodeGenFunction::AutoVarEmission var = CGF.EmitAutoVarAlloca(*CatchParam);
-  InitCatchParam(CGF, *CatchParam, var.getObjectAddress(CGF), S->getLocStart());
+  InitCatchParam(CGF, *CatchParam, var.getObjectAddress(CGF), S->getLocStart(),
+                 LaterHandlers);
   CGF.EmitAutoVarCleanups(var);
 }
 
 /// Get or define the following function:
 ///   void @__clang_call_terminate(i8* %exn) nounwind noreturn
 /// This code is used only in C++.
 static llvm::Constant *getClangCallTerminateFn(CodeGenModule &CGM) {
   llvm::FunctionType *fnTy =

lib/CodeGen/MicrosoftCXXABI.cpp

public:
 
   void emitVirtualObjectDelete(CodeGenFunction &CGF, const CXXDeleteExpr *DE,
                                Address Ptr, QualType ElementType,
                                const CXXDestructorDecl *Dtor) override;
 
   void emitRethrow(CodeGenFunction &CGF, bool isNoReturn) override;
   void emitThrow(CodeGenFunction &CGF, const CXXThrowExpr *E) override;
 
-  void emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C) override;
+  void emitBeginCatch(CodeGenFunction &CGF, const CXXCatchStmt *C,
+                      ArrayRef<EHCatchScope::Handler> LaterHandlers) override;
 
   llvm::GlobalVariable *getMSCompleteObjectLocator(const CXXRecordDecl *RD,
                                                    const VPtrInfo &Info);
 
   llvm::Constant *getAddrOfRTTIDescriptor(QualType Ty) override;
   CatchTypeInfo
   getAddrOfCXXCatchHandlerType(QualType Ty, QualType CatchHandlerType) override;
 
struct CatchRetScope final : EHScopeStack::Cleanup {
   void Emit(CodeGenFunction &CGF, Flags flags) override {
     llvm::BasicBlock *BB = CGF.createBasicBlock("catchret.dest");
     CGF.Builder.CreateCatchRet(CPI, BB);
     CGF.EmitBlock(BB);
   }
 };
 }
 
-void MicrosoftCXXABI::emitBeginCatch(CodeGenFunction &CGF,
-                                     const CXXCatchStmt *S) {
+void MicrosoftCXXABI::emitBeginCatch(
+    CodeGenFunction &CGF, const CXXCatchStmt *S,
+    ArrayRef<EHCatchScope::Handler> LaterHandlers) {
   // In the MS ABI, the runtime handles the copy, and the catch handler is
   // responsible for destruction.
   VarDecl *CatchParam = S->getExceptionDecl();
   llvm::BasicBlock *CatchPadBB = CGF.Builder.GetInsertBlock();
   llvm::CatchPadInst *CPI =
       cast<llvm::CatchPadInst>(CatchPadBB->getFirstNonPHI());
   CGF.CurrentFuncletPad = CPI;
 

lib/CodeGen/TargetInfo.h

virtual void setTargetAttributes(const Decl *D, llvm::GlobalValue *GV,
                                    CodeGen::CodeGenModule &M,
                                    ForDefinition_t IsForDefinition) const {}
 
   /// emitTargetMD - Provides a convenient hook to handle extra
   /// target-specific metadata for the given global.
   virtual void emitTargetMD(const Decl *D, llvm::GlobalValue *GV,
                             CodeGen::CodeGenModule &M) const {}
 
+  /// Determines the size of struct __cxa_exception on this platform,
+  /// in 8-bit units, excluding the trailing _Unwind_Exception member.
+  /// The Itanium ABI defines the layout of this struct.
+  virtual unsigned getSizeOfCxaException() const;
+
   /// Determines the size of struct _Unwind_Exception on this platform,
   /// in 8-bit units.  The Itanium ABI defines this as:
   ///   struct _Unwind_Exception {
   ///     uint64 exception_class;
   ///     _Unwind_Exception_Cleanup_Fn exception_cleanup;
   ///     uint64 private_1;
   ///     uint64 private_2;
   ///   };

lib/CodeGen/TargetInfo.cpp

static Address emitMergePHI(CodeGenFunction &CGF,
   PHI->addIncoming(Addr1.getPointer(), Block1);
   PHI->addIncoming(Addr2.getPointer(), Block2);
   CharUnits Align = std::min(Addr1.getAlignment(), Addr2.getAlignment());
   return Address(PHI, Align);
 }
 
 TargetCodeGenInfo::~TargetCodeGenInfo() { delete Info; }
 
+unsigned TargetCodeGenInfo::getSizeOfCxaException() const {
+  // __cxa_exception comprises 5 pointers followed by 2 ints followed by 4
+  // more pointers, then the _Unwind_Exception object. We assume that no
+  // padding is introduced between these fields.
+  ASTContext &Ctx = Info->getContext();
+  return (9 * Ctx.getTypeSizeInChars(Ctx.VoidPtrTy) +
+          2 * Ctx.getTypeSizeInChars(Ctx.IntTy))
+      .getQuantity();
+}
+
 // If someone can figure out a general rule for this, that would be great.
 // It's probably just doomed to be platform-dependent, though.
 unsigned TargetCodeGenInfo::getSizeOfUnwindException() const {
   // Verified for:
   //   x86-64     FreeBSD, Linux, Darwin
   //   x86-32     FreeBSD, Linux, Darwin
   //   PowerPC    Linux, Darwin
   //   ARM        Darwin (*not* EABI)