[WebAssembly] Add WebAssemblyLateEHPrepare pass

[aheejin]

Summary

Add WebAssemblyLateEHPrepare pass that does several small jobs for
exception handling. This runs before CFGSort, and is different from
WasmEHPrepare pass that runs before ISel, even though the names are
similar.

[aheejin]

fixes

[aheejin]

Removed a blank line

[aheejin]

rebase

[aheejin]

removed setIsEHFuncletEntry

[aheejin]

• Deleted successors of catch-all terminatepads

[aheejin]

• removed an unnecessary call from a test case

[aheejin]

• debug location fix

[aheejin]

@majnemer Continuing the discussion from D44134:

So the background was:
__clang_call_terminate is called within a cleanuppad. But cleanuppad does not give you an exception object, which __clang_call_terminate takes as an argument. So even if it is a cleanuppad, we added a wasm catch instruction there to catch an exception object. But this cleanuppad should be entered even when we catch a foreign (e.g. JavaScript) exception, which we have no exception object to deal with. So in that case we just directly all std::terminate.

So, when the clang-generated code is like

terminate:
  %exn = catch(0)
  call void @__clang_call_terminate(%exn)
  unreachable

processTerminatePads() appends one more terminatepad right after this terminatepad, like:

terminate-catchall:
  catch_all
  call @std::terminate
  unreachable

So the intention was, if we catch a C++ exception we go to terminate, and if we catch a foreign exception we end up in terminate-catchall. Here I also assumed that the terminatepad is a single BB, but as you noted, it may not be. But because this pass runs even before ExceptinInfo (D44134), this does not have the exception grouping information here.

Do you think I can merge the terminatepad into a single BB in this pass, so this duplication of terminatepad process will be easier? I can't think of a situation in which a terminatepad can have complicated control flows other than a few straight-line BBs. (Actually I can't even imagine what kind of llvm pass is gonna divide a single terminate into two or more, but we can't guarantee that can't happen, so..)

Also, the other thing is, I merge terminate pads if there are more than two of them in mergeTerminatePads() here. This is in principle violation of EH scope membership, but this happens after all MachineFunctionPasses and only remaining wasm passes after this are CFGSort and CFGStackify, so I guess it should be OK.

[aheejin]

• Make wasm.get.exception/ehselector intrinsic take a token argument

[aheejin]

• Rebased onto D44090
• Not this pass deletes the following unreachable instruction after adding a rethrow instruction.

[aheejin]

Planning two changes:

1. Handling of the terminate pad
2. Removing unreachable BB after rethrow

[aheejin]

• Added a method to ensure a terminate pad is a single BB
• Using WasmEHInfo, now this adds RETHROW or RETHROW_TO_CALLER instruction depending on its unwind destination
• Ensured a rethrow instruction is in the same BB with __cxa_rethrow() call and deleted all unreachable children
• In hoistCatches, added handling of a case when catch instruction is not in the EH pad BB anymore
• Now uses several utility functions in WebAssemblyUtilities.cpp
• Added helper functions
• Misc. improvements here and there

[aheejin]

• Delete an unreachable after rethrow

[dschuff]

lib/Target/WebAssembly/WebAssemblyExceptionPrepare.cpp

27 ↗

(On Diff #151640)

I didn't think about the confusing-ness of this name when we put it in originally; maybe we could call it 'WebAssemblyMachineEHPrepare' or 'WebAssemblyLateEHPrepare'

110 ↗

(On Diff #151640)

So a function can rethrow but not have a personality function?

168 ↗

(On Diff #151640)

'instruction is'

212 ↗

(On Diff #151640)

Would it make more sense to just have the rethrow inside of cxa_rethrow (at least if the destination is the caller)? It might save some on code size? I guess we would just have to put an 'unreachable' after the call and it wouldn't matter; or is there some other reason?

[aheejin]

• WebAssemblyExceptionPrepare -> WebAssemblyLateEHPrepare
• Typo fix

[aheejin]

lib/Target/WebAssembly/WebAssemblyExceptionPrepare.cpp

27 ↗

(On Diff #151640)

Hmm, not to be confused with WasmEHPrepare? Yeah, that makes sense.. I'll go with WebAssemblyLateEHPrepare because 'Machine' sounds like it has its same or similar counterpart in LLVM IR passes. (like, LoopInfo vs MachineLoopInfo)

110 ↗

(On Diff #151640)

A function has its associated personality function when it has an EH pad (landingpad or catchpad/cleanuppad). Functions that call __cxa_rethrow() can rethrow to the caller, in which case the function doesn't have an EH pad. I didn't want to run all the functions below needlessly in that case.

212 ↗

(On Diff #151640)

That is the spec question. The current implementation is based on the first proposal, in which rethrow can only be placed within a catch block. We can change this later if that is shown to save code size.

I'll do some experiments on that once I'm able to run things end-to-end, but I actually doubt if that would be saving much, because we can save one rethrow instruction if we put that within a function, but that also means we have to pass a first-class exception argument to __cxa_rethrow(), and to put that argument on the wasm stack we have to consume one more instruction anyway. And also, if we use first-class exception objects, now every catch clause has to have that if_except ~ else ~ end instruction sequence too, which also contributes to the code size.

Anyway, I'm not saying I'm against it though. I'm saying just the current implementation is based on the first proposal :$

[dschuff]

OK, the pass looks good; don't know if @majnemer has any more feedback on this or D44134?

[aheejin]

@majnemer Do you think the problem we discussed in this CL and D44134 have been resolved?
The comment that started the discussion is here. So basically I tried to ensure a terminatepad is a single BB in this prepare pass.

llvm/trunk/lib/Target/WebAssembly/CMakeLists.txt

 add_public_tablegen_target(WebAssemblyCommonTableGen)
 
 add_llvm_target(WebAssemblyCodeGen
   WebAssemblyArgumentMove.cpp
   WebAssemblyAsmPrinter.cpp
   WebAssemblyCallIndirectFixup.cpp
   WebAssemblyCFGStackify.cpp
   WebAssemblyCFGSort.cpp
-  WebAssemblyExceptionPrepare.cpp
+  WebAssemblyLateEHPrepare.cpp
   WebAssemblyExplicitLocals.cpp
   WebAssemblyFastISel.cpp
   WebAssemblyFixIrreducibleControlFlow.cpp
   WebAssemblyFixFunctionBitcasts.cpp
   WebAssemblyFrameLowering.cpp
   WebAssemblyISelDAGToDAG.cpp
   WebAssemblyISelLowering.cpp
   WebAssemblyInstrInfo.cpp

llvm/trunk/lib/Target/WebAssembly/WebAssembly.h

 FunctionPass *createWebAssemblyReplacePhysRegs();
 FunctionPass *createWebAssemblyPrepareForLiveIntervals();
 FunctionPass *createWebAssemblyOptimizeLiveIntervals();
 FunctionPass *createWebAssemblyStoreResults();
 FunctionPass *createWebAssemblyRegStackify();
 FunctionPass *createWebAssemblyRegColoring();
 FunctionPass *createWebAssemblyExplicitLocals();
 FunctionPass *createWebAssemblyFixIrreducibleControlFlow();
-FunctionPass *createWebAssemblyExceptionPrepare();
+FunctionPass *createWebAssemblyLateEHPrepare();
 FunctionPass *createWebAssemblyCFGSort();
 FunctionPass *createWebAssemblyCFGStackify();
 FunctionPass *createWebAssemblyLowerBrUnless();
 FunctionPass *createWebAssemblyRegNumbering();
 FunctionPass *createWebAssemblyPeephole();
 FunctionPass *createWebAssemblyCallIndirectFixup();
 
 // PassRegistry initialization declarations.
 void initializeWebAssemblyLowerEmscriptenEHSjLjPass(PassRegistry &);
 void initializeLowerGlobalDtorsPass(PassRegistry &);
 void initializeFixFunctionBitcastsPass(PassRegistry &);
 void initializeOptimizeReturnedPass(PassRegistry &);
 void initializeWebAssemblyArgumentMovePass(PassRegistry &);
 void initializeWebAssemblySetP2AlignOperandsPass(PassRegistry &);
 void initializeWebAssemblyReplacePhysRegsPass(PassRegistry &);
 void initializeWebAssemblyPrepareForLiveIntervalsPass(PassRegistry &);
 void initializeWebAssemblyOptimizeLiveIntervalsPass(PassRegistry &);
 void initializeWebAssemblyStoreResultsPass(PassRegistry &);
 void initializeWebAssemblyRegStackifyPass(PassRegistry &);
 void initializeWebAssemblyRegColoringPass(PassRegistry &);
 void initializeWebAssemblyExplicitLocalsPass(PassRegistry &);
 void initializeWebAssemblyFixIrreducibleControlFlowPass(PassRegistry &);
-void initializeWebAssemblyExceptionPreparePass(PassRegistry &);
+void initializeWebAssemblyLateEHPreparePass(PassRegistry &);
 void initializeWebAssemblyCFGSortPass(PassRegistry &);
 void initializeWebAssemblyCFGStackifyPass(PassRegistry &);
 void initializeWebAssemblyLowerBrUnlessPass(PassRegistry &);
 void initializeWebAssemblyRegNumberingPass(PassRegistry &);
 void initializeWebAssemblyPeepholePass(PassRegistry &);
 void initializeWebAssemblyCallIndirectFixupPass(PassRegistry &);
 
 } // end namespace llvm
 
 #endif

llvm/trunk/lib/Target/WebAssembly/WebAssemblyExceptionPrepare.cpp

-//=== WebAssemblyExceptionPrepare.cpp - WebAssembly Exception Preparation -===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-///
-/// \file
-/// \brief Does various transformations for exception handling.
-///
-//===----------------------------------------------------------------------===//
-
-#include "MCTargetDesc/WebAssemblyMCTargetDesc.h"
-#include "WebAssembly.h"
-#include "WebAssemblySubtarget.h"
-#include "llvm/CodeGen/MachineInstrBuilder.h"
-using namespace llvm;
-
-#define DEBUG_TYPE "wasm-exception-prepare"
-
-namespace {
-class WebAssemblyExceptionPrepare final : public MachineFunctionPass {
-  StringRef getPassName() const override {
-    return "WebAssembly Prepare Exception";
-  }
-
-  bool runOnMachineFunction(MachineFunction &MF) override;
-
-  bool replaceFuncletReturns(MachineFunction &MF);
-
-public:
-  static char ID; // Pass identification, replacement for typeid
-  WebAssemblyExceptionPrepare() : MachineFunctionPass(ID) {}
-};
-} // end anonymous namespace
-
-char WebAssemblyExceptionPrepare::ID = 0;
-INITIALIZE_PASS(WebAssemblyExceptionPrepare, DEBUG_TYPE,
-                "WebAssembly Exception Preparation", false, false)
-
-FunctionPass *llvm::createWebAssemblyExceptionPrepare() {
-  return new WebAssemblyExceptionPrepare();
-}
-
-bool WebAssemblyExceptionPrepare::runOnMachineFunction(MachineFunction &MF) {
-  bool Changed = false;
-  if (!MF.getFunction().hasPersonalityFn())
-    return false;
-  Changed |= replaceFuncletReturns(MF);
-  // TODO More transformations will be added
-  return Changed;
-}
-
-bool WebAssemblyExceptionPrepare::replaceFuncletReturns(MachineFunction &MF) {
-  bool Changed = false;
-  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
-
-  for (auto &MBB : MF) {
-    auto Pos = MBB.getFirstTerminator();
-    if (Pos == MBB.end())
-      continue;
-    MachineInstr *TI = &*Pos;
-
-    switch (TI->getOpcode()) {
-    case WebAssembly::CATCHRET: {
-      // Replace a catchret with a branch
-      MachineBasicBlock *TBB = TI->getOperand(0).getMBB();
-      if (!MBB.isLayoutSuccessor(TBB))
-        BuildMI(MBB, TI, TI->getDebugLoc(), TII.get(WebAssembly::BR))
-            .addMBB(TBB);
-      TI->eraseFromParent();
-      Changed = true;
-      break;
-    }
-    case WebAssembly::CLEANUPRET: {
-      // Replace a cleanupret with a rethrow
-      BuildMI(MBB, TI, TI->getDebugLoc(), TII.get(WebAssembly::RETHROW))
-          .addImm(0);
-      TI->eraseFromParent();
-      Changed = true;
-      break;
-    }
-    }
-  }
-  return Changed;
-}

llvm/trunk/lib/Target/WebAssembly/WebAssemblyLateEHPrepare.cpp

+//=== WebAssemblyLateEHPrepare.cpp - WebAssembly Exception Preparation -===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+///
+/// \file
+/// \brief Does various transformations for exception handling.
+///
+//===----------------------------------------------------------------------===//
+
+#include "MCTargetDesc/WebAssemblyMCTargetDesc.h"
+#include "WebAssembly.h"
+#include "WebAssemblySubtarget.h"
+#include "WebAssemblyUtilities.h"
+#include "llvm/CodeGen/MachineInstrBuilder.h"
+#include "llvm/CodeGen/WasmEHFuncInfo.h"
+#include "llvm/MC/MCAsmInfo.h"
+using namespace llvm;
+
+#define DEBUG_TYPE "wasm-exception-prepare"
+
+namespace {
+class WebAssemblyLateEHPrepare final : public MachineFunctionPass {
+  StringRef getPassName() const override {
+    return "WebAssembly Prepare Exception";
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  bool replaceFuncletReturns(MachineFunction &MF);
+  bool hoistCatches(MachineFunction &MF);
+  bool addCatchAlls(MachineFunction &MF);
+  bool addRethrows(MachineFunction &MF);
+  bool ensureSingleBBTermPads(MachineFunction &MF);
+  bool mergeTerminatePads(MachineFunction &MF);
+  bool addCatchAllTerminatePads(MachineFunction &MF);
+
+public:
+  static char ID; // Pass identification, replacement for typeid
+  WebAssemblyLateEHPrepare() : MachineFunctionPass(ID) {}
+};
+} // end anonymous namespace
+
+char WebAssemblyLateEHPrepare::ID = 0;
+INITIALIZE_PASS(WebAssemblyLateEHPrepare, DEBUG_TYPE,
+                "WebAssembly Exception Preparation", false, false)
+
+FunctionPass *llvm::createWebAssemblyLateEHPrepare() {
+  return new WebAssemblyLateEHPrepare();
+}
+
+// Returns the nearest EH pad that dominates this instruction. This does not use
+// dominator analysis; it just does BFS on its predecessors until arriving at an
+// EH pad. This assumes valid EH scopes so the first EH pad it arrives in all
+// possible search paths should be the same.
+// Returns nullptr in case it does not find any EH pad in the search, or finds
+// multiple different EH pads.
+MachineBasicBlock *GetMatchingEHPad(MachineInstr *MI) {
+  MachineFunction *MF = MI->getParent()->getParent();
+  SmallVector<MachineBasicBlock *, 2> WL;
+  SmallPtrSet<MachineBasicBlock *, 2> Visited;
+  WL.push_back(MI->getParent());
+  MachineBasicBlock *EHPad = nullptr;
+  while (!WL.empty()) {
+    MachineBasicBlock *MBB = WL.pop_back_val();
+    if (Visited.count(MBB))
+      continue;
+    Visited.insert(MBB);
+    if (MBB->isEHPad()) {
+      if (EHPad && EHPad != MBB)
+        return nullptr;
+      EHPad = MBB;
+      continue;
+    }
+    if (MBB == &MF->front())
+      return nullptr;
+    WL.append(MBB->pred_begin(), MBB->pred_end());
+  }
+  return EHPad;
+}
+
+// Erases the given BB and all its children from the function. If other BBs have
+// this BB as a successor, the successor relationships will be deleted as well.
+static void EraseBBAndChildren(MachineBasicBlock *MBB) {
+  SmallVector<MachineBasicBlock *, 8> WL;
+  WL.push_back(MBB);
+  while (!WL.empty()) {
+    MachineBasicBlock *MBB = WL.pop_back_val();
+    for (auto *Pred : MBB->predecessors())
+      Pred->removeSuccessor(MBB);
+    for (auto *Succ : MBB->successors()) {
+      WL.push_back(Succ);
+      MBB->removeSuccessor(Succ);
+    }
+    MBB->eraseFromParent();
+  }
+}
+
+bool WebAssemblyLateEHPrepare::runOnMachineFunction(MachineFunction &MF) {
+  if (MF.getTarget().getMCAsmInfo()->getExceptionHandlingType() !=
+      ExceptionHandling::Wasm)
+    return false;
+
+  bool Changed = false;
+  Changed |= addRethrows(MF);
+  if (!MF.getFunction().hasPersonalityFn())
+    return Changed;
+  Changed |= replaceFuncletReturns(MF);
+  Changed |= hoistCatches(MF);
+  Changed |= addCatchAlls(MF);
+  Changed |= ensureSingleBBTermPads(MF);
+  Changed |= mergeTerminatePads(MF);
+  Changed |= addCatchAllTerminatePads(MF);
+  return Changed;
+}
+
+bool WebAssemblyLateEHPrepare::replaceFuncletReturns(MachineFunction &MF) {
+  bool Changed = false;
+  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
+  auto *EHInfo = MF.getWasmEHFuncInfo();
+
+  for (auto &MBB : MF) {
+    auto Pos = MBB.getFirstTerminator();
+    if (Pos == MBB.end())
+      continue;
+    MachineInstr *TI = &*Pos;
+
+    switch (TI->getOpcode()) {
+    case WebAssembly::CATCHRET: {
+      // Replace a catchret with a branch
+      MachineBasicBlock *TBB = TI->getOperand(0).getMBB();
+      if (!MBB.isLayoutSuccessor(TBB))
+        BuildMI(MBB, TI, TI->getDebugLoc(), TII.get(WebAssembly::BR))
+            .addMBB(TBB);
+      TI->eraseFromParent();
+      Changed = true;
+      break;
+    }
+    case WebAssembly::CLEANUPRET: {
+      // Replace a cleanupret with a rethrow
+      if (EHInfo->hasThrowUnwindDest(&MBB))
+        BuildMI(MBB, TI, TI->getDebugLoc(), TII.get(WebAssembly::RETHROW))
+            .addMBB(EHInfo->getThrowUnwindDest(&MBB));
+      else
+        BuildMI(MBB, TI, TI->getDebugLoc(),
+                TII.get(WebAssembly::RETHROW_TO_CALLER));
+
+      TI->eraseFromParent();
+      Changed = true;
+      break;
+    }
+    }
+  }
+  return Changed;
+}
+
+// Hoist catch instructions to the beginning of their matching EH pad BBs in
+// case,
+// (1) catch instruction is not the first instruction in EH pad.
+// ehpad:
+//   some_other_instruction
+//   ...
+//   %exn = catch 0
+// (2) catch instruction is in a non-EH pad BB. For example,
+// ehpad:
+//   br bb0
+// bb0:
+//   %exn = catch 0
+bool WebAssemblyLateEHPrepare::hoistCatches(MachineFunction &MF) {
+  bool Changed = false;
+  SmallVector<MachineInstr *, 16> Catches;
+  for (auto &MBB : MF)
+    for (auto &MI : MBB)
+      if (WebAssembly::isCatch(MI))
+        Catches.push_back(&MI);
+
+  for (auto *Catch : Catches) {
+    MachineBasicBlock *EHPad = GetMatchingEHPad(Catch);
+    assert(EHPad && "No matching EH pad for catch");
+    if (EHPad->begin() == Catch)
+      continue;
+    Changed = true;
+    EHPad->insert(EHPad->begin(), Catch->removeFromParent());
+  }
+  return Changed;
+}
+
+// Add catch_all to beginning of cleanup pads.
+bool WebAssemblyLateEHPrepare::addCatchAlls(MachineFunction &MF) {
+  bool Changed = false;
+  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
+
+  for (auto &MBB : MF) {
+    if (!MBB.isEHPad())
+      continue;
+    // This runs after hoistCatches(), so we assume that if there is a catch,
+    // that should be the first instruction in an EH pad.
+    if (!WebAssembly::isCatch(*MBB.begin())) {
+      Changed = true;
+      BuildMI(MBB, MBB.begin(), MBB.begin()->getDebugLoc(),
+              TII.get(WebAssembly::CATCH_ALL));
+    }
+  }
+  return Changed;
+}
+
+// Add a 'rethrow' instruction after __cxa_rethrow() call
+bool WebAssemblyLateEHPrepare::addRethrows(MachineFunction &MF) {
+  bool Changed = false;
+  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
+  auto *EHInfo = MF.getWasmEHFuncInfo();
+
+  for (auto &MBB : MF)
+    for (auto &MI : MBB) {
+      // Check if it is a call to __cxa_rethrow()
+      if (!MI.isCall())
+        continue;
+      MachineOperand &CalleeOp = MI.getOperand(0);
+      if (!CalleeOp.isGlobal() ||
+          CalleeOp.getGlobal()->getName() != WebAssembly::CxaRethrowFn)
+        continue;
+
+      // Now we have __cxa_rethrow() call
+      Changed = true;
+      auto InsertPt = std::next(MachineBasicBlock::iterator(MI));
+      while (InsertPt != MBB.end() && InsertPt->isLabel()) // Skip EH_LABELs
+        ++InsertPt;
+      MachineInstr *Rethrow = nullptr;
+      if (EHInfo->hasThrowUnwindDest(&MBB))
+        Rethrow = BuildMI(MBB, InsertPt, MI.getDebugLoc(),
+                          TII.get(WebAssembly::RETHROW))
+                      .addMBB(EHInfo->getThrowUnwindDest(&MBB));
+      else
+        Rethrow = BuildMI(MBB, InsertPt, MI.getDebugLoc(),
+                          TII.get(WebAssembly::RETHROW_TO_CALLER));
+
+      // Becasue __cxa_rethrow does not return, the instruction after the
+      // rethrow should be an unreachable or a branch to another BB that should
+      // eventually lead to an unreachable. Delete it because rethrow itself is
+      // a terminator, and also delete non-EH pad successors if any.
+      MBB.erase(std::next(MachineBasicBlock::iterator(Rethrow)), MBB.end());
+      for (auto *Succ : MBB.successors())
+        if (!Succ->isEHPad())
+          EraseBBAndChildren(Succ);
+    }
+  return Changed;
+}
+
+// Terminate pads are an single-BB EH pad in the form of
+// termpad:
+//   %exn = catch 0
+//   call @__clang_call_terminate(%exn)
+//   unreachable
+// (There can be set_local and get_locals before the call if we didn't run
+// RegStackify)
+// But code transformations can change or add more control flow, so the call to
+// __clang_call_terminate() function may not be in the original EH pad anymore.
+// This ensures every terminate pad is a single BB in the form illustrated
+// above.
+bool WebAssemblyLateEHPrepare::ensureSingleBBTermPads(MachineFunction &MF) {
+  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
+
+  // Find calls to __clang_call_terminate()
+  SmallVector<MachineInstr *, 8> ClangCallTerminateCalls;
+  for (auto &MBB : MF)
+    for (auto &MI : MBB)
+      if (MI.isCall()) {
+        const MachineOperand &CalleeOp = MI.getOperand(0);
+        if (CalleeOp.isGlobal() && CalleeOp.getGlobal()->getName() ==
+                                       WebAssembly::ClangCallTerminateFn)
+          ClangCallTerminateCalls.push_back(&MI);
+      }
+
+  bool Changed = false;
+  for (auto *Call : ClangCallTerminateCalls) {
+    MachineBasicBlock *EHPad = GetMatchingEHPad(Call);
+    assert(EHPad && "No matching EH pad for catch");
+
+    // If it is already the form we want, skip it
+    if (Call->getParent() == EHPad &&
+        Call->getNextNode()->getOpcode() == WebAssembly::UNREACHABLE)
+      continue;
+
+    // In case the __clang_call_terminate() call is not in its matching EH pad,
+    // move the call to the end of EH pad and add an unreachable instruction
+    // after that. Delete all successors and their children if any, because here
+    // the program terminates.
+    Changed = true;
+    MachineInstr *Catch = &*EHPad->begin();
+    // This runs after hoistCatches(), so catch instruction should be at the top
+    assert(WebAssembly::isCatch(*Catch));
+    // Takes the result register of the catch instruction as argument. There may
+    // have been some other set_local/get_locals in between, but at this point
+    // we don't care.
+    Call->getOperand(1).setReg(Catch->getOperand(0).getReg());
+    auto InsertPos = std::next(MachineBasicBlock::iterator(Catch));
+    EHPad->insert(InsertPos, Call->removeFromParent());
+    BuildMI(*EHPad, InsertPos, Call->getDebugLoc(),
+            TII.get(WebAssembly::UNREACHABLE));
+    EHPad->erase(InsertPos, EHPad->end());
+    for (auto *Succ : EHPad->successors())
+      EraseBBAndChildren(Succ);
+  }
+  return Changed;
+}
+
+// In case there are multiple terminate pads, merge them into one for code size.
+// This runs after ensureSingleBBTermPads() and assumes every terminate pad is a
+// single BB.
+// In principle this violates EH scope relationship because it can merge
+// multiple inner EH scopes, each of which is in different outer EH scope. But
+// getEHScopeMembership() function will not be called after this, so it is fine.
+bool WebAssemblyLateEHPrepare::mergeTerminatePads(MachineFunction &MF) {
+  SmallVector<MachineBasicBlock *, 8> TermPads;
+  for (auto &MBB : MF)
+    if (WebAssembly::isCatchTerminatePad(MBB))
+      TermPads.push_back(&MBB);
+  if (TermPads.empty())
+    return false;
+
+  MachineBasicBlock *UniqueTermPad = TermPads.front();
+  for (auto *TermPad :
+       llvm::make_range(std::next(TermPads.begin()), TermPads.end())) {
+    SmallVector<MachineBasicBlock *, 2> Preds(TermPad->pred_begin(),
+                                              TermPad->pred_end());
+    for (auto *Pred : Preds)
+      Pred->replaceSuccessor(TermPad, UniqueTermPad);
+    TermPad->eraseFromParent();
+  }
+  return true;
+}
+
+// Terminate pads are cleanup pads, so they should start with a 'catch_all'
+// instruction. But in the Itanium model, when we have a C++ exception object,
+// we pass them to __clang_call_terminate function, which calls __cxa_end_catch
+// with the passed exception pointer and then std::terminate. This is the reason
+// that terminate pads are generated with not a catch_all but a catch
+// instruction in clang and earlier llvm passes. Here we append a terminate pad
+// with a catch_all after each existing terminate pad so we can also catch
+// foreign exceptions. For every terminate pad:
+//   %exn = catch 0
+//   call @__clang_call_terminate(%exn)
+//   unreachable
+// We append this BB right after that:
+//   catch_all
+//   call @std::terminate()
+//   unreachable
+bool WebAssemblyLateEHPrepare::addCatchAllTerminatePads(MachineFunction &MF) {
+  const auto &TII = *MF.getSubtarget<WebAssemblySubtarget>().getInstrInfo();
+  SmallVector<MachineBasicBlock *, 8> TermPads;
+  for (auto &MBB : MF)
+    if (WebAssembly::isCatchTerminatePad(MBB))
+      TermPads.push_back(&MBB);
+  if (TermPads.empty())
+    return false;
+
+  Function *StdTerminateFn =
+      MF.getFunction().getParent()->getFunction(WebAssembly::StdTerminateFn);
+  assert(StdTerminateFn && "There is no std::terminate() function");
+  for (auto *CatchTermPad : TermPads) {
+    DebugLoc DL = CatchTermPad->findDebugLoc(CatchTermPad->begin());
+    auto *CatchAllTermPad = MF.CreateMachineBasicBlock();
+    MF.insert(std::next(MachineFunction::iterator(CatchTermPad)),
+              CatchAllTermPad);
+    CatchAllTermPad->setIsEHPad();
+    BuildMI(CatchAllTermPad, DL, TII.get(WebAssembly::CATCH_ALL));
+    BuildMI(CatchAllTermPad, DL, TII.get(WebAssembly::CALL_VOID))
+        .addGlobalAddress(StdTerminateFn);
+    BuildMI(CatchAllTermPad, DL, TII.get(WebAssembly::UNREACHABLE));
+
+    // Actually this CatchAllTermPad (new terminate pad with a catch_all) is not
+    // a successor of an existing terminate pad. CatchAllTermPad should have all
+    // predecessors CatchTermPad has instead. This is a hack to force
+    // CatchAllTermPad be always sorted right after CatchTermPad; the correct
+    // predecessor-successor relationships will be restored in CFGStackify pass.
+    CatchTermPad->addSuccessor(CatchAllTermPad);
+  }
+  return true;
+}

llvm/trunk/lib/Target/WebAssembly/WebAssemblyTargetMachine.cpp

extern "C" void LLVMInitializeWebAssemblyTarget() {
   initializeWebAssemblyReplacePhysRegsPass(PR);
   initializeWebAssemblyPrepareForLiveIntervalsPass(PR);
   initializeWebAssemblyOptimizeLiveIntervalsPass(PR);
   initializeWebAssemblyStoreResultsPass(PR);
   initializeWebAssemblyRegStackifyPass(PR);
   initializeWebAssemblyRegColoringPass(PR);
   initializeWebAssemblyExplicitLocalsPass(PR);
   initializeWebAssemblyFixIrreducibleControlFlowPass(PR);
-  initializeWebAssemblyExceptionPreparePass(PR);
+  initializeWebAssemblyLateEHPreparePass(PR);
   initializeWebAssemblyCFGSortPass(PR);
   initializeWebAssemblyCFGStackifyPass(PR);
   initializeWebAssemblyLowerBrUnlessPass(PR);
   initializeWebAssemblyRegNumberingPass(PR);
   initializeWebAssemblyPeepholePass(PR);
   initializeWebAssemblyCallIndirectFixupPass(PR);
 }
 
void WebAssemblyPassConfig::addPreEmitPass() {
   // and set_local operators because we create a new variable that we want
   // converted into a local.
   addPass(createWebAssemblyFixIrreducibleControlFlow());
 
   // Insert explicit get_local and set_local operators.
   addPass(createWebAssemblyExplicitLocals());
 
   // Do various transformations for exception handling
-  addPass(createWebAssemblyExceptionPrepare());
+  addPass(createWebAssemblyLateEHPrepare());
 
   // Sort the blocks of the CFG into topological order, a prerequisite for
   // BLOCK and LOOP markers.
   addPass(createWebAssemblyCFGSort());
 
   // Insert BLOCK and LOOP markers.
   addPass(createWebAssemblyCFGStackify());
 

llvm/trunk/test/CodeGen/WebAssembly/exception.ll

 ; CHECK-LABEL: test_throw:
 ; CHECK-NEXT: i32.const $push0=, 0
 ; CHECK-NEXT: throw 0, $pop0
 define void @test_throw() {
   call void @llvm.wasm.throw(i32 0, i8* null)
   ret void
 }
 
-; CHECK-LABEL: test_catch:
+; CHECK-LABEL: test_catch_rethrow:
 ; CHECK:   call      foo@FUNCTION
 ; CHECK:   i32.catch     $push{{.+}}=, 0
 ; CHECK-DAG:   i32.store  __wasm_lpad_context
 ; CHECK-DAG:   i32.store  __wasm_lpad_context+4
 ; CHECK:   i32.call  $push{{.+}}=, _Unwind_CallPersonality@FUNCTION
 ; CHECK:   i32.call  $push{{.+}}=, __cxa_begin_catch@FUNCTION
 ; CHECK:   call      __cxa_end_catch@FUNCTION
 ; CHECK:   call      __cxa_rethrow@FUNCTION
-define void @test_catch() personality i8* bitcast (i32 (...)* @__gxx_wasm_personality_v0 to i8*) {
+; CHECK-NEXT:   rethrow
+define void @test_catch_rethrow() personality i8* bitcast (i32 (...)* @__gxx_wasm_personality_v0 to i8*) {
 entry:
   invoke void @foo()
           to label %try.cont unwind label %catch.dispatch
 
 catch.dispatch:                                   ; preds = %entry
   %0 = catchswitch within none [label %catch.start] unwind to caller
 
 catch.start:                                      ; preds = %catch.dispatch
rethrow:                                          ; preds = %catch.start
   unreachable
 
 try.cont:                                         ; preds = %entry, %catch
   ret void
 }
 
 ; CHECK-LABEL: test_cleanup:
 ; CHECK:   call      foo@FUNCTION
-; CHECK:   return
+; CHECK:   catch_all
 ; CHECK:   i32.call  $push20=, _ZN7CleanupD1Ev@FUNCTION
-; CHECK:   rethrow   0
+; CHECK:   rethrow
 define void @test_cleanup() personality i8* bitcast (i32 (...)* @__gxx_wasm_personality_v0 to i8*) {
 entry:
   %c = alloca %struct.Cleanup, align 1
   invoke void @foo()
           to label %invoke.cont unwind label %ehcleanup
 
 invoke.cont:                                      ; preds = %entry
   %call = call %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c)
   ret void
 
 ehcleanup:                                        ; preds = %entry
   %0 = cleanuppad within none []
   %call1 = call %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c) [ "funclet"(token %0) ]
   cleanupret from %0 unwind to caller
 }
 
+; - Tests multple terminate pads are merged into one
+; - Tests a catch_all terminate pad is created after a catch terminate pad
+
+; CHECK-LABEL: test_terminatepad
+; CHECK:  i32.catch
+; CHECK:  call      __clang_call_terminate@FUNCTION
+; CHECK:  unreachable
+; CHECK:  catch_all
+; CHECK:  call      _ZSt9terminatev@FUNCTION
+; CHECK-NOT:  call      __clang_call_terminate@FUNCTION
+define hidden i32 @test_terminatepad() personality i8* bitcast (i32 (...)* @__gxx_wasm_personality_v0 to i8*) {
+entry:
+  %c = alloca %struct.Cleanup, align 1
+  %c1 = alloca %struct.Cleanup, align 1
+  invoke void @foo()
+          to label %invoke.cont unwind label %ehcleanup
+
+invoke.cont:                                      ; preds = %entry
+  %call = invoke %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c1)
+          to label %try.cont unwind label %catch.dispatch
+
+ehcleanup:                                        ; preds = %entry
+  %0 = cleanuppad within none []
+  %call4 = invoke %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c1) [ "funclet"(token %0) ]
+          to label %invoke.cont3 unwind label %terminate
+
+invoke.cont3:                                     ; preds = %ehcleanup
+  cleanupret from %0 unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %invoke.cont3, %invoke.cont
+  %1 = catchswitch within none [label %catch.start] unwind label %ehcleanup7
+
+catch.start:                                      ; preds = %catch.dispatch
+  %2 = catchpad within %1 [i8* null]
+  %3 = call i8* @llvm.wasm.get.exception(token %2)
+  %4 = call i32 @llvm.wasm.get.ehselector(token %2)
+  %5 = call i8* @__cxa_begin_catch(i8* %3) [ "funclet"(token %2) ]
+  invoke void @__cxa_end_catch() [ "funclet"(token %2) ]
+          to label %invoke.cont5 unwind label %ehcleanup7
+
+invoke.cont5:                                     ; preds = %catch.start
+  catchret from %2 to label %try.cont
+
+try.cont:                                         ; preds = %invoke.cont5, %invoke.cont
+  %call6 = call %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c)
+  ret i32 0
+
+ehcleanup7:                                       ; preds = %catch.start, %catch.dispatch
+  %6 = cleanuppad within none []
+  %call9 = invoke %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* %c) [ "funclet"(token %6) ]
+          to label %invoke.cont8 unwind label %terminate10
+
+invoke.cont8:                                     ; preds = %ehcleanup7
+  cleanupret from %6 unwind to caller
+
+terminate:                                        ; preds = %ehcleanup
+  %7 = cleanuppad within %0 []
+  %8 = call i8* @llvm.wasm.get.exception(token %7)
+  call void @__clang_call_terminate(i8* %8) [ "funclet"(token %7) ]
+  unreachable
+
+terminate10:                                      ; preds = %ehcleanup7
+  %9 = cleanuppad within %6 []
+  %10 = call i8* @llvm.wasm.get.exception(token %9)
+  call void @__clang_call_terminate(i8* %10) [ "funclet"(token %9) ]
+  unreachable
+}
+
 declare void @foo()
-declare void @func(i32)
 declare i32 @__gxx_wasm_personality_v0(...)
 declare i8* @llvm.wasm.get.exception(token)
 declare i32 @llvm.wasm.get.ehselector(token)
 declare i32 @llvm.eh.typeid.for(i8*)
 declare i8* @__cxa_begin_catch(i8*)
 declare void @__cxa_end_catch()
 declare void @__cxa_rethrow()
 declare void @__clang_call_terminate(i8*)
+declare void @_ZSt9terminatev()
 declare %struct.Cleanup* @_ZN7CleanupD1Ev(%struct.Cleanup* returned)