[ARM64] [Windows] Handle funclets

[efriedma]

Summary

This patch adds support for funclets in frame lowering and ISel lowering. Together with D50288 and D50166, it enables C++ exception handling.

[mgrang]

lib/Target/AArch64/AArch64FrameLowering.cpp

1201

default should be the first case.

lib/Target/AArch64/AArch64InstrInfo.cpp

1670

Can you move "MBB.getParent()-" to the next line so that it's more readable?

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

Check 80 char limits here.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

1201

Will change it in the updated patch.

lib/Target/AArch64/AArch64InstrInfo.cpp

1670

Will do in the updated patch.

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

Is the 80 char limit on the RUN line a rule? If so, many test cases don't follow this. A single RUN is not split into multiple lines from what I've seen.

[ssijaric]

A few formatting fixes.

[mgrang]

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

< many test cases don't follow this.
They should! 80 chars is what the LLVM coding standards recommend:
https://llvm.org/docs/CodingStandards.html#source-code-formatting

Write your code to fit within 80 columns of text. This helps those of us who like to print out code and look at your code in an xterm without resizing it.

Plus, horizontal scrolling is no fun :)

[rnk]

Neat. :)

lib/Target/AArch64/AArch64FrameLowering.cpp

645

"containing"

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1638

UnwidHelp -> UnwindHelp?

1639–1641

Should this be moved after the terminator check?

1675–1679

This is used to populate dispFrame in _s_HandlerType IIRC. IIUC, you will have issues with certain catch handlers without this properly populated.

[MatzeB]

I don't really know how exception handling on windows works or what a funclet is. So here's just a couple style comments and questions

lib/Target/AArch64/AArch64FrameLowering.cpp

909

Can be const MachineInstr&

934–935

Out of interest: How is getWinEHFuncletFrameSize() different from MachineFrameIndo::getStackSize()?

1182

Add a comment explaining why you need a frame pointer for funclet?
Knowing nothing about funclets myself I wonder why the normal rules for determining whether we need FP are not enough here...

1648

In case the previous loop ends with MBBI == MBB.end() all the following tests will be undefined behavior.

1652

My understanding is that all of this stuff is just compiler generated extra instructions? In that case I recommend using an empty debug location instead (I know other places of the code get this wrong too...)

1662

Use doxygen /// comments.

1668–1672

doesn't this change the behavior for non-funclets too? If yes this needs to go into a separate patch and needs some comments why.

lib/Target/AArch64/AArch64MCInstLower.cpp

256–268

Why is this in MCInstLower and not AsmPrinter.cpp? Maybe you can even specify it in tablegen (see for example ARMs LDMIA_RET and similar)

test/CodeGen/AArch64/wineh-funclets.ll

60 ↗

(On Diff #163556)

unused basic block?

76 ↗

(On Diff #163556)

You should be able to drop most of these attributes.

79–83 ↗

(On Diff #163556)

I'm pretty sure you can drop these lines from the test

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1675–1679

Ah, it could be that dispFrame is not needed for ARM64. In that case, WinException.cpp should not call getWinEHParentFrameOffset and the ParentFrameOffset should not be emitted as it will add padding between the HandlerType elements.

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1675–1679

Hmmm... So after a bunch of poking around with ARM64 CL.exe, I can confirm that they are allocating the dispFrame field and always fill it with zero!

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

645

Will fix in the updated test case.

909

Will change in the updated patch.

934–935

The stack size (getStackSize()) includes all the locals in its size calculation. We don't include these locals when computing the stack size of a funclet, as they are allocated in the parent's stack frame and accessed via the frame pointer from the funclet. Otherwise, we would be double allocating stack space in the funclet that we don't need. We only save the callee saved registers in the funclet, which are really the callee saved registers of the parent function, including the funclet. Perhaps, only the non-volatile registers that the funclet modifies should be saved/restored in the funclet prologue/epilogue, but we don't keep track of these separately.

I will document this.

1182

Will do in the updated patch.

1638

Will fix in the updated patch.

1639–1641

Yes, it's definitely a better place for it.

1652

Will fix in the updated patch.

1662

Will fix in the updated patch.

1675–1679

Thanks for double checking!! Will leave it as is.

lib/Target/AArch64/AArch64MCInstLower.cpp

256–268

X86 did the expansion in X86MCInstLower.cpp. I would keep this way in the current patch, and look into tablegen later. I'll add a comment.

test/CodeGen/AArch64/wineh-funclets.ll

60 ↗

(On Diff #163556)

The block is generated by clang, and SImplifyCFG removes it. I'll come up with a better test case.

76 ↗

(On Diff #163556)

Will update the test case.

79–83 ↗

(On Diff #163556)

Will update the test case.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

1668–1672

This is used to calculate displacement of objects from the stack pointer, such as the displacement of the unwind object when filling up the exception handling table on Windows. This is the only place where it is used for AArch64. All the frame indices belong to the parent function (e.g. never a funclet). I put it in this patch, because the unwind object above is created only if the function has funclets.

It may be hard to write a test case that checks the .s file, as it will crash with an assert in the WinException sanity check (since the displacement will be that from the frame pointer). I would like to keep in this patch, if at all possible. If there is a really strong objection, I can move it to a separate patch.

[ssijaric]

Address comments: spelling mistakes, better comments, a better test case.

[ssijaric]

lib/Target/AArch64/AArch64ISelLowering.cpp

1278

Move this into the assert; it will cause the release only build to fail.

[ssijaric]

Remove an unused variable. Fix a nasty bug that Eli just pointed out, where I didn't insert a PrologEnd in funclets in this patch. Previously, I relied on an RAII object to this for me when we would go out of scope. I removed that logic, but forgot to insert a prolog end marker for the funclet case.

With this patch, the following should now pass:

#include <stdio.h>
int
main() {
  int B[100];
  int x = 4;
  B[2] = 2;
  B[4] = 0;
  try {
    throw B[2];
  } catch (int i) {
    printf("Caught %d\n", i);
    B[4] = B[i] + i;
    try {
      throw 'c';
    } catch (char c) {
      printf("Caught char %c\n", c);
    }
  }
  return x + B[4];
}

with the correct print outs and a return value of 8.

[ssijaric]

Fix the check for IsFunclet in emitEpilogue.

[efriedma]

I'm not really an expert on funclets, but the code looks fine to me.

The test coverage here is a little thin... in particular, there isn't any coverage of the emitted unwind opcodes. I'll amend the testcase to check the llvm-readobj output. (Sanjin isn't available for the next few weeks, so I'm going to handle any further revisions to the patch.)

Reid, do you have an opinion on what test coverage is necessary here?

[rnk]

In D51524#1285854, @efriedma wrote:

Reid, do you have an opinion on what test coverage is necessary here?

It's probably important to handle each and every kind of prologue. I don't know if there are any special corners cases for arm64 prologues, but for x64, this ended up being:

• normal prologue
• normal with -fno-omit-frame-pointer
• stack realignment caused by a highly aligned alloca
• normal with dynamic alloca or VLA
• dynamic alloca with highly aligned static alloca

SEH and C++ EH have major differences, so test both. Alternatively, you can focus on one in one patch and defer work on the other.

SEH __except blocks in particular had to do some extra work to restore SP and register state when recovering from an exception, especially when the frame has dynamic SP updates.

I still need to go back and review, but hopefully that helps.

[rnk]

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831

We had bugs here with XMM registers, which I don't think we fixed: https://llvm.org/pr32507

How does the AArch64 prologue/epilogue save and restore registers? If you load/store to a fixed frame index stack object, it's not going to work well. If it does something analogous to x86 "PUSH" instructions, it'll work fine.

[efriedma]

This patch forces all functions that contain funclets to emit a frame pointer. (The existing testcase covers that.)

I'll do some testing with realignment and dynamic allocas. I don't think AArch64 prologue lowering has any other special cases.

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831

Spills of callee-saved registers should always use sp-relative loads/stores, as far as I know.

[efriedma]

Oh, I guess there is one special case for the prologue I didn't mention: varargs functions. I think the patch handles that correctly already, but I'll add a test.

For both dynamic allocas and stack realignment, the initial implementation of unwinding missed a significant rule: "r29 must point to the bottom of stack". That's probably straightforward to resolve, but I don't really want to do it in this patch (and I'm not sure what the practical effect is). Beyond that, there's probably more handling necessary to correctly handle a dynamic alloca inside a funclet (since we need to reset the frame pointer, but we still need access to the old frame pointer). And I don't really want to block this patch on solving that; it's obscure enough that it won't affect most code.

I have a couple small fixes for stack realignment, a fix for a MachineVerifier failure, and a testcase for realignment; I'll upload a new version.

[efriedma]

Misc fixes.

[efriedma]

The following testcase is also broken, for reasons I don't understand. I don't think it's a problem with any of the code in this patch, though; the throw out of the "catch" block somehow picks the wrong catch block, but this patch doesn't touch that code.

void a() { try { try { throw 1; } catch (...) { throw 1; } } catch (...) {}}

[efriedma]

Add a bit more test coverage, to cover cases with and without base pointers. Tweak base pointer emission so we emit and use a base pointer where appropriate (including some cases where it isn't strictly necessary, but allows more efficient addressing modes).

I think this covers all the interesting cases for C++ exceptions. (There will be a separate followup patch for SEH exceptions.)

[rnk]

In D51524#1286346, @efriedma wrote:

The following testcase is also broken, for reasons I don't understand. I don't think it's a problem with any of the code in this patch, though; the throw out of the "catch" block somehow picks the wrong catch block, but this patch doesn't touch that code.

void a() { try { try { throw 1; } catch (...) { throw 1; } } catch (...) {}}

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

[efriedma]

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

Looked a bit more; it looks like the problem is actually that the "throw" is the last instruction in the function, so unwinding gets confused. There's some code in X86TargetMachine::X86TargetMachine which turns on TrapUnreachable, which avoids the problem. It looks like MSVC does something similar (it inserts a nop instead of a trap, but same idea). I'll post a patch separately.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831

They should always be saved off the funclet's SP pointer. This is what cl does, if I remember correctly.

[ssijaric]

In D51524#1286331, @efriedma wrote:

Oh, I guess there is one special case for the prologue I didn't mention: varargs functions. I think the patch handles that correctly already, but I'll add a test.

For both dynamic allocas and stack realignment, the initial implementation of unwinding missed a significant rule: "r29 must point to the bottom of stack". That's probably straightforward to resolve, but I don't really want to do it in this patch (and I'm not sure what the practical effect is). Beyond that, there's probably more handling necessary to correctly handle a dynamic alloca inside a funclet (since we need to reset the frame pointer, but we still need access to the old frame pointer). And I don't really want to block this patch on solving that; it's obscure enough that it won't affect most code.

I have a couple small fixes for stack realignment, a fix for a MachineVerifier failure, and a testcase for realignment; I'll upload a new version.

Hi Eli,

Yin and I looked at the rule that says fp must be at the bottom if alloca is present. We tried a few (simple) test cases to see if this would be a problem. We didn't encounter any issues with our very limited testing, so decided not to implement it at this time. From my recollection, the fp (or the fp/lr pair) has to be separated from the code that saves the callee-saved registers, and handled on its own so that it ends up pointing to the bottom.

Thank you for taking over this!!! I may have some time time between flights to look at this a bit more, depending on the internet connection.

[thegameg]

I am afraid I don't know much about exceptions on Windows in general. A few comments below.

lib/Target/AArch64/AArch64FrameLowering.cpp

1495

Small typo: in the in -> in the

2057

Why is this needed?

2112

typo paren't -> parent's

lib/Target/AArch64/AArch64RegisterInfo.cpp

262

Isn't this already caught by the next nested ifs?

test/CodeGen/AArch64/wineh-try-catch-realign.ll

10 ↗

(On Diff #172668)

Is there a chance that we can test the PEI/FrameLowering part with more specific MIR tests? Especially the checks for what the base register should be. The fact that we don't serialize AArch64FunctionInfo may make this impossible for now.

[rnk]

In D51524#1288018, @efriedma wrote:

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

Looked a bit more; it looks like the problem is actually that the "throw" is the last instruction in the function, so unwinding gets confused. There's some code in X86TargetMachine::X86TargetMachine which turns on TrapUnreachable, which avoids the problem. It looks like MSVC does something similar (it inserts a nop instead of a trap, but same idea). I'll post a patch separately.

We came up with a different solution for this called SEH_Epilogue. It basically inserts a nop if the last real instruction was a call. It's slightly complicated because it looks backwards across empty basic blocks and things. You can see it here:
https://github.com/llvm-mirror/llvm/blob/master/lib/Target/X86/X86MCInstLower.cpp#L1800

I've been meaning to see if we can remove the code that enables TrapUnreachable on Windows.

[efriedma]

lib/Target/AArch64/AArch64FrameLowering.cpp

2057

It's not, as far as I can tell; I'll get rid of it.

lib/Target/AArch64/AArch64RegisterInfo.cpp

262

Yes, I refactored it and didn't realize it. Will fix.

test/CodeGen/AArch64/wineh-try-catch-realign.ll

10 ↗

(On Diff #172668)

Yes, I could write an MIR test using -run-pass=prologepilog, but I don't see the point. The CHECK lines would be essentially identical, just checking for MIR opcodes instead of assembly instructions. The MIR doesn't directly record whether we have a base register, or what it is; it's only used as part of frame lowering.

[efriedma]

Address review comments (no functional change from the previous version).

[efriedma]

Reid, are you okay with merging this? I'd prefer to get this merged sooner rather than later. (If you don't have to review carefully right now, I can address any minor issues post-commit.)

[rnk]

lgtm

Sorry, I lost track of this.

lib/Target/AArch64/AArch64InstrInfo.cpp

1681

Yep, makes sense, put the address of the target MBB in X0. Do you have any assurance that X0 isn't used by normal epilogue code? At least on x86, sometimes RAX gets used for large SP updates.

test/CodeGen/AArch64/wineh-try-catch.ll

14–16

Tests with comments about the correctness of numbers that we filecheck for below are always worrying, but I have no better suggestions. :)

[efriedma]

lib/Target/AArch64/AArch64InstrInfo.cpp

1681

The current AArch64 emitFrameOffset never does that. And even if it did in some cases, a funclet's stack can't be more than a few hundred bytes anyway.

lib/Target/AArch64/AArch64FrameLowering.h

public:
 
   /// Returns true if the target will correctly handle shrink wrapping.
   bool enableShrinkWrapping(const MachineFunction &MF) const override {
     return true;
   }
 
   bool enableStackSlotScavenging(const MachineFunction &MF) const override;
 
+  void processFunctionBeforeFrameFinalized(MachineFunction &MF,
+                                             RegScavenger *RS) const override;
+
+  unsigned getWinEHParentFrameOffset(const MachineFunction &MF) const override;
+
+  unsigned getWinEHFuncletFrameSize(const MachineFunction &MF) const;
+
+  int getFrameIndexReferencePreferSP(const MachineFunction &MF, int FI,
+                                     unsigned &FrameReg,
+                                     bool IgnoreSPUpdates) const override;
+
 private:
   bool shouldCombineCSRLocalStackBump(MachineFunction &MF,
                                       unsigned StackBumpBytes) const;
 };
 
 } // End llvm namespace
 
 #endif

lib/Target/AArch64/AArch64FrameLowering.cpp

bool AArch64FrameLowering::canUseRedZone(const MachineFunction &MF) const {
   return !(MFI.hasCalls() || hasFP(MF) || NumBytes > 128);
 }
 
 /// hasFP - Return true if the specified function should have a dedicated frame
 /// pointer register.
 bool AArch64FrameLowering::hasFP(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
+  // Win64 EH requires a frame pointer if funclets are present, as the locals
+  // are accessed off the frame pointer in both the parent function and the
+  // funclets.
+  if (MF.hasEHFunclets())
+    return true;
   // Retain behavior of always omitting the FP for leaf functions when possible.
   if (MFI.hasCalls() && MF.getTarget().Options.DisableFramePointerElim(MF))
     return true;
   if (MFI.hasVarSizedObjects() || MFI.isFrameAddressTaken() ||
       MFI.hasStackMap() || MFI.hasPatchPoint() ||
       RegInfo->needsStackRealignment(MF))
     return true;
   // With large callframes around we may need to use FP to access the scavenging
static MachineBasicBlock::iterator convertCalleeSaveRestoreToSPPrePostIncDec(
   case AArch64::LDRXui:
     NewOpc = AArch64::LDRXpost;
     break;
   case AArch64::LDRDui:
     NewOpc = AArch64::LDRDpost;
     break;
   case AArch64::LDRQui:
     NewOpc = AArch64::LDRQpost;
     break;
   }
   // Get rid of the SEH code associated with the old instruction.
   if (NeedsWinCFI) {
     auto SEH = std::next(MBBI);
     if (AArch64InstrInfo::isSEHInstruction(*SEH))
       SEH->eraseFromParent();
   }
 
if (!F.hasFnAttribute("sign-return-address-key"))
     return true;
 
   const StringRef Key =
       F.getFnAttribute("sign-return-address-key").getValueAsString();
   assert(Key.equals_lower("a_key") || Key.equals_lower("b_key"));
   return Key.equals_lower("a_key");
 }
 
+static bool needsWinCFI(const MachineFunction &MF) {
+  const Function &F = MF.getFunction();
+  return MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+         F.needsUnwindTableEntry();
+}
+
 void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.begin();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const Function &F = MF.getFunction();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   MachineModuleInfo &MMI = MF.getMMI();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   bool needsFrameMoves = (MMI.hasDebugInfo() || F.needsUnwindTableEntry()) &&
                          !MF.getTarget().getMCAsmInfo()->usesWindowsCFI();
   bool HasFP = hasFP(MF);
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     F.needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   MF.setHasWinCFI(NeedsWinCFI);
+  bool IsFunclet = MBB.isEHFuncletEntry();
+
   // At this point, we're going to decide whether or not the function uses a
   // redzone. In most cases, the function doesn't have a redzone so let's
   // assume that's false and set it to true in the case that there's a redzone.
   AFI->setHasRedZone(false);
 
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
 
   if (ShouldSignReturnAddress(MF)) {
     BuildMI(
         MBB, MBBI, DL,
         TII->get(ShouldSignWithAKey(MF) ? AArch64::PACIASP : AArch64::PACIBSP))
         .setMIFlag(MachineInstr::FrameSetup);
   }
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
-  int NumBytes = (int)MFI.getStackSize();
+  // getStackSize() includes all the locals in its size calculation. We don't
+  // include these locals when computing the stack size of a funclet, as they
+  // are allocated in the parent's stack frame and accessed via the frame
+  // pointer from the funclet.  We only save the callee saved registers in the
+  // funclet, which are really the callee saved registers of the parent
+  // function, including the funclet.
+  int NumBytes = IsFunclet ? (int)getWinEHFuncletFrameSize(MF)
+                           : (int)MFI.getStackSize();
   if (!AFI->hasStackFrame() && !windowsRequiresStackProbe(MF, NumBytes)) {
     assert(!HasFP && "unexpected function without stack frame but with FP");
     // All of the stack allocation is for locals.
     AFI->setLocalStackSize(NumBytes);
     if (!NumBytes)
       return;
     // REDZONE: If the stack size is less than 128 bytes, we don't need
     // to actually allocate.
if (NeedsWinCFI)
       BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_PrologEnd))
           .setMIFlag(MachineInstr::FrameSetup);
 
     return;
   }
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
-  unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
+  // Var args are accounted for in the containing function, so don't
+  // include them for funclets.
+  unsigned FixedObject = (IsWin64 && !IsFunclet) ?
+                         alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
 
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   // All of the remaining stack allocations are for locals.
   AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   if (CombineSPBump) {
     emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
                     MachineInstr::FrameSetup, false, NeedsWinCFI);
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   MachineBasicBlock::iterator End = MBB.end();
   while (MBBI != End && MBBI->getFlag(MachineInstr::FrameSetup)) {
     if (CombineSPBump)
       fixupCalleeSaveRestoreStackOffset(*MBBI, AFI->getLocalStackSize(),
                                         NeedsWinCFI);
     ++MBBI;
   }
 
+  // The code below is not applicable to funclets. We have emitted all the SEH
+  // opcodes that we needed to emit.  The FP and BP belong to the containing
+  // function.
+  if (IsFunclet) {
+    if (NeedsWinCFI)
+      BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_PrologEnd))
+          .setMIFlag(MachineInstr::FrameSetup);
+    return;
+  }
+
   if (HasFP) {
     // Only set up FP if we actually need to. Frame pointer is fp =
     // sp - fixedobject - 16.
     int FPOffset = AFI->getCalleeSavedStackSize() - 16;
     if (CombineSPBump)
       FPOffset += AFI->getLocalStackSize();
 
     // Issue    sub fp, sp, FPOffset or
     //          mov fp,sp          when FPOffset is zero.
     // Note: All stores of callee-saved registers are marked as "FrameSetup".
     // This code marks the instruction(s) that set the FP also.
     emitFrameOffset(MBB, MBBI, DL, AArch64::FP, AArch64::SP, FPOffset, TII,
                     MachineInstr::FrameSetup, false, NeedsWinCFI);
   }
 
   if (windowsRequiresStackProbe(MF, NumBytes)) {
     uint32_t NumWords = NumBytes >> 4;
     if (NeedsWinCFI) {
       // alloc_l can hold at most 256MB, so assume that NumBytes doesn't
       // exceed this amount.  We need to move at most 2^24 - 1 into x15.
       // This is at most two instructions, MOVZ follwed by MOVK.
       // TODO: Fix to use multiple stack alloc unwind codes for stacks
       // exceeding 256MB in size.
       if (NumBytes >= (1 << 28))
         report_fatal_error("Stack size cannot exceed 256MB for stack "
                             "unwinding purposes");
 
       uint32_t LowNumWords = NumWords & 0xFFFF;
       BuildMI(MBB, MBBI, DL, TII->get(AArch64::MOVZXi), AArch64::X15)
             .addImm(LowNumWords)
             .addImm(AArch64_AM::getShifterImm(AArch64_AM::LSL, 0))
             .setMIFlag(MachineInstr::FrameSetup);
       BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_Nop))
             .setMIFlag(MachineInstr::FrameSetup);
static void InsertReturnAddressAuth(MachineFunction &MF,
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
 
   MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
   DebugLoc DL;
   if (MBBI != MBB.end())
     DL = MBBI->getDebugLoc();
 
   // The AUTIASP instruction assembles to a hint instruction before v8.3a so
   // this instruction can safely used for any v8a architecture.
   // From v8.3a onwards there are optimised authenticate LR and return
   // instructions, namely RETA{A,B}, that can be used instead.
   if (Subtarget.hasV8_3aOps() && MBBI != MBB.end() &&
       MBBI->getOpcode() == AArch64::RET_ReallyLR) {
     BuildMI(MBB, MBBI, DL,
             TII->get(ShouldSignWithAKey(MF) ? AArch64::RETAA : AArch64::RETAB))
         .copyImplicitOps(*MBBI);
     MBB.erase(MBBI);
   } else {
     BuildMI(
         MBB, MBBI, DL,
         TII->get(ShouldSignWithAKey(MF) ? AArch64::AUTIASP : AArch64::AUTIBSP))
         .setMIFlag(MachineInstr::FrameDestroy);
   }
 }
 
+static bool isFuncletReturnInstr(const MachineInstr &MI) {
+  switch (MI.getOpcode()) {
+  default:
+    return false;
+  case AArch64::CATCHRET:
+  case AArch64::CLEANUPRET:
+    return true;
+  }
+}
+
 void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   DebugLoc DL;
   bool IsTailCallReturn = false;
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
+  bool IsFunclet = isFuncletReturnInstr(*MBBI);
 
   if (MBB.end() != MBBI) {
     DL = MBBI->getDebugLoc();
     unsigned RetOpcode = MBBI->getOpcode();
     IsTailCallReturn = RetOpcode == AArch64::TCRETURNdi ||
                        RetOpcode == AArch64::TCRETURNri ||
                        RetOpcode == AArch64::TCRETURNriBTI;
   }
 
-  int NumBytes = MFI.getStackSize();
+  int NumBytes = IsFunclet ? (int)getWinEHFuncletFrameSize(MF)
+                           : MFI.getStackSize();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   // Initial and residual are named for consistency with the prologue. Note that
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   // AArch64TargetLowering::LowerCall figures out ArgumentPopSize and keeps
   // it as the 2nd argument of AArch64ISD::TC_RETURN.
 
   auto Cleanup = make_scope_exit([&] { InsertReturnAddressAuth(MF, MBB); });
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
-
   uint64_t AfterCSRPopSize = ArgumentPopSize;
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
+  // Var args are accounted for in the containting function, so don't
+  // include them for funclets.
+  if (MF.hasEHFunclets())
+    AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   // Assume we can't combine the last pop with the sp restore.
 
   if (!CombineSPBump && PrologueSaveSize != 0) {
     MachineBasicBlock::iterator Pop = std::prev(MBB.getFirstTerminator());
     while (AArch64InstrInfo::isSEHInstruction(*Pop))
       Pop = std::prev(Pop);
     // Converting the last ldp to a post-index ldp is valid only if the last
if (isFixed) {
         // else we can use BP and FP, but the offset from FP won't fit.
         // That will make us scavenge registers which we can probably avoid by
         // using BP. If it won't fit for BP either, we'll scavenge anyway.
       } else if (FPOffset >= 0) {
         // Use SP or FP, whichever gives us the best chance of the offset
         // being in range for direct access. If the FPOffset is positive,
         // that'll always be best, as the SP will be even further away.
         UseFP = true;
+      } else if (MF.hasEHFunclets()) {
+        // Funclets access the locals contained in the in the parent's stack
+        // frame via the frame pointer, so we have to use the FP in the parent
+        // function.
+        assert(
+            Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv()) &&
+            "Funclets should only be present on Win64");
+        UseFP = true;
       } else {
         // We have the choice between FP and (SP or BP).
         if (FPOffsetFits && PreferFP) // If FP is the best fit, use it.
           UseFP = true;
       }
     }
   }
 

 static void computeCalleeSaveRegisterPairs(
     MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI, SmallVectorImpl<RegPairInfo> &RegPairs,
     bool &NeedShadowCallStackProlog) {
 
   if (CSI.empty())
     return;
 
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   CallingConv::ID CC = MF.getFunction().getCallingConv();
   unsigned Count = CSI.size();
   (void)CC;
   // MachO's compact unwind format relies on all registers being stored in
   // pairs.
   assert((!produceCompactUnwindFrame(MF) ||
if (i + 1 < Count) {
       case RegPairInfo::GPR:
         if (AArch64::GPR64RegClass.contains(NextReg) &&
             !invalidateWindowsRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI))
           RPI.Reg2 = NextReg;
         break;
       case RegPairInfo::FPR64:
         if (AArch64::FPR64RegClass.contains(NextReg) &&
             !invalidateWindowsRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI))
           RPI.Reg2 = NextReg;
         break;
       case RegPairInfo::FPR128:
         if (AArch64::FPR128RegClass.contains(NextReg))
           RPI.Reg2 = NextReg;
         break;
       }
     }
 
     // If either of the registers to be saved is the lr register, it means that
     // we also need to save lr in the shadow call stack.
     if ((RPI.Reg1 == AArch64::LR || RPI.Reg2 == AArch64::LR) &&
         MF.getFunction().hasFnAttribute(Attribute::ShadowCallStack)) {
       if (!MF.getSubtarget<AArch64Subtarget>().isXRegisterReserved(18))
         report_fatal_error("Must reserve x18 to use shadow call stack");
       NeedShadowCallStackProlog = true;
     }
 
     // GPRs and FPRs are saved in pairs of 64-bit regs. We expect the CSI
     // list to come in sorted by frame index so that we can issue the store
     // pair instructions directly. Assert if we see anything otherwise.
     //
     // The order of the registers in the list is controlled by
     // getCalleeSavedRegs(), so they will always be in-order, as well.
     assert((!RPI.isPaired() ||
             (CSI[i].getFrameIdx() + 1 == CSI[i + 1].getFrameIdx())) &&
            "Out of order callee saved regs!");
 
     // MachO's compact unwind format relies on all registers being stored in
     // adjacent register pairs.
     assert((!produceCompactUnwindFrame(MF) ||
             CC == CallingConv::PreserveMost ||
             (RPI.isPaired() &&
              ((RPI.Reg1 == AArch64::LR && RPI.Reg2 == AArch64::FP) ||
               RPI.Reg1 + 1 == RPI.Reg2))) &&
            "Callee-save registers not saved as adjacent register pair!");
 
     RPI.FrameIdx = CSI[i].getFrameIdx();
 
     int Scale = RPI.Type == RegPairInfo::FPR128 ? 16 : 8;
     Offset -= RPI.isPaired() ? 2 * Scale : Scale;
 
     // Round up size of non-pair to pair size if we need to pad the
     // callee-save area to ensure 16-byte alignment.
     if (AFI->hasCalleeSaveStackFreeSpace() && !FixupDone &&
         RPI.Type != RegPairInfo::FPR128 && !RPI.isPaired()) {
       FixupDone = true;
       Offset -= 8;
       assert(Offset % 16 == 0);
       assert(MFI.getObjectAlignment(RPI.FrameIdx) <= 16);

 }
 
 bool AArch64FrameLowering::spillCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
   const MachineRegisterInfo &MRI = MF.getRegInfo();
 

 bool AArch64FrameLowering::restoreCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
 
   if (MI != MBB.end())
     DL = MI->getDebugLoc();
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
 
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   AFI->setCalleeSaveStackHasFreeSpace(AlignedCSStackSize != CSStackSize);
 }
 
 bool AArch64FrameLowering::enableStackSlotScavenging(
     const MachineFunction &MF) const {
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   return AFI->hasCalleeSaveStackFreeSpace();
 }
+
+void AArch64FrameLowering::processFunctionBeforeFrameFinalized(
+    MachineFunction &MF, RegScavenger *RS) const {
+  // Mark the function as not having WinCFI. We will set it back to true in
+  // emitPrologue if unwinding info is needed.
+  MF.setHasWinCFI(false);
+
+  // If this function isn't doing Win64-style C++ EH, we don't need to do
+  // anything.
+  if (!MF.hasEHFunclets())
+    return;
+  const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  WinEHFuncInfo &EHInfo = *MF.getWinEHFuncInfo();
+
+  MachineBasicBlock &MBB = MF.front();
+  auto MBBI = MBB.begin();
+  while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup))
+    ++MBBI;
+
+  if (MBBI->isTerminator())
+    return;
+
+  // Create an UnwindHelp object.
+  int UnwindHelpFI =
+      MFI.CreateStackObject(/*size*/8, /*alignment*/16, false);
+  EHInfo.UnwindHelpFrameIdx = UnwindHelpFI;
+  // We need to store -2 into the UnwindHelp object at the start of the
+  // function.
+  DebugLoc DL;
+  RS->enterBasicBlock(MBB);
+  unsigned DstReg = RS->scavengeRegister(&AArch64::GPR64RegClass, MBBI, 0);
+  BuildMI(MBB, MBBI, DL, TII.get(AArch64::MOVi64imm), DstReg).addImm(-2);
+  BuildMI(MBB, MBBI, DL, TII.get(AArch64::STURXi))
+      .addReg(DstReg, getKillRegState(true))
+      .addFrameIndex(UnwindHelpFI)
+      .addImm(0);
+}
+
+/// For Win64 AArch64 EH, the offset to the Unwind object is from the SP before
+/// the update.  This is easily retrieved as it is exactly the offset that is set
+/// in processFunctionBeforeFrameFinalized.
+int AArch64FrameLowering::getFrameIndexReferencePreferSP(
+    const MachineFunction &MF, int FI, unsigned &FrameReg,
+    bool IgnoreSPUpdates) const {
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  LLVM_DEBUG(dbgs() << "Offset from the SP for " << FI << " is "
+                    << MFI.getObjectOffset(FI) << "\n");
+  FrameReg = AArch64::SP;
+  return MFI.getObjectOffset(FI);
+}
+
+/// The parent frame offset (aka dispFrame) is only used on X86_64 to retrieve
+/// the parent's frame pointer
+unsigned AArch64FrameLowering::getWinEHParentFrameOffset(
+    const MachineFunction &MF) const {
+  return 0;
+}
+
+/// Funclets only need to account for space for the callee saved registers,
+/// as the locals are account for in the paren't stack frame.
+unsigned AArch64FrameLowering::getWinEHFuncletFrameSize(
+    const MachineFunction &MF) const {
+  // This is the size of the pushed CSRs.
+  unsigned CSSize =
+      MF.getInfo<AArch64FunctionInfo>()->getCalleeSavedStackSize();
+  // This is the amount of stack a funclet needs to allocate.
+  return alignTo(CSSize + MF.getFrameInfo().getMaxCallFrameSize(),
+                 getStackAlignment());
+}

lib/Target/AArch64/AArch64ISelLowering.h

public:
   EVT getSetCCResultType(const DataLayout &DL, LLVMContext &Context,
                          EVT VT) const override;
 
   SDValue ReconstructShuffle(SDValue Op, SelectionDAG &DAG) const;
 
   MachineBasicBlock *EmitF128CSEL(MachineInstr &MI,
                                   MachineBasicBlock *BB) const;
 
+  MachineBasicBlock *EmitLoweredCatchRet(MachineInstr &MI,
+                                           MachineBasicBlock *BB) const;
+
+  MachineBasicBlock *EmitLoweredCatchPad(MachineInstr &MI,
+                                         MachineBasicBlock *BB) const;
+
   MachineBasicBlock *
   EmitInstrWithCustomInserter(MachineInstr &MI,
                               MachineBasicBlock *MBB) const override;
 
   bool getTgtMemIntrinsic(IntrinsicInfo &Info, const CallInst &I,
                           MachineFunction &MF,
                           unsigned Intrinsic) const override;
 
public:
   unsigned getNumInterleavedAccesses(VectorType *VecTy,
                                      const DataLayout &DL) const;
 
   MachineMemOperand::Flags getMMOFlags(const Instruction &I) const override;
 
   bool functionArgumentNeedsConsecutiveRegisters(Type *Ty,
                                                  CallingConv::ID CallConv,
                                                  bool isVarArg) const override;
+  /// Used for exception handling on Win64.
+  bool needsFixedCatchObjects() const override;
 private:
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   void addTypeForNEON(MVT VT, MVT PromotedBitwiseVT);

lib/Target/AArch64/AArch64ISelLowering.cpp

BuildMI(*EndBB, EndBB->begin(), DL, TII->get(AArch64::PHI), DestReg)
       .addMBB(TrueBB)
       .addReg(IfFalseReg)
       .addMBB(MBB);
 
   MI.eraseFromParent();
   return EndBB;
 }
 
+MachineBasicBlock *AArch64TargetLowering::EmitLoweredCatchRet(
+       MachineInstr &MI, MachineBasicBlock *BB) const {
+  assert(!isAsynchronousEHPersonality(classifyEHPersonality(
+             BB->getParent()->getFunction().getPersonalityFn())) &&
+         "SEH does not use catchret!");
+  return BB;
+}
+
+MachineBasicBlock *AArch64TargetLowering::EmitLoweredCatchPad(
+     MachineInstr &MI, MachineBasicBlock *BB) const {
+  MI.eraseFromParent();
+  return BB;
+}
+
 MachineBasicBlock *AArch64TargetLowering::EmitInstrWithCustomInserter(
     MachineInstr &MI, MachineBasicBlock *BB) const {
   switch (MI.getOpcode()) {
   default:
 #ifndef NDEBUG
     MI.dump();
 #endif
     llvm_unreachable("Unexpected instruction for custom inserter!");
 
   case AArch64::F128CSEL:
     return EmitF128CSEL(MI, BB);
 
   case TargetOpcode::STACKMAP:
   case TargetOpcode::PATCHPOINT:
     return emitPatchPoint(MI, BB);
+
+  case AArch64::CATCHRET:
+    return EmitLoweredCatchRet(MI, BB);
+  case AArch64::CATCHPAD:
+    return EmitLoweredCatchPad(MI, BB);
   }
 }
 
 //===----------------------------------------------------------------------===//
 // AArch64 Lowering private implementation.
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
AArch64TargetLowering::getVaListSizeInBits(const DataLayout &DL) const {
 
   return 3 * getPointerTy(DL).getSizeInBits() + 2 * 32;
 }
 
 void AArch64TargetLowering::finalizeLowering(MachineFunction &MF) const {
   MF.getFrameInfo().computeMaxCallFrameSize(MF);
   TargetLoweringBase::finalizeLowering(MF);
 }
+
+// Unlike X86, we let frame lowering assign offsets to all catch objects.
+bool AArch64TargetLowering::needsFixedCatchObjects() const {
+  return false;
+}

lib/Target/AArch64/AArch64InstrInfo.cpp

static cl::opt<unsigned> CBZDisplacementBits(
     "aarch64-cbz-offset-bits", cl::Hidden, cl::init(19),
     cl::desc("Restrict range of CB[N]Z instructions (DEBUG)"));
 
 static cl::opt<unsigned>
     BCCDisplacementBits("aarch64-bcc-offset-bits", cl::Hidden, cl::init(19),
                         cl::desc("Restrict range of Bcc instructions (DEBUG)"));
 
 AArch64InstrInfo::AArch64InstrInfo(const AArch64Subtarget &STI)
-    : AArch64GenInstrInfo(AArch64::ADJCALLSTACKDOWN, AArch64::ADJCALLSTACKUP),
+    : AArch64GenInstrInfo(AArch64::ADJCALLSTACKDOWN, AArch64::ADJCALLSTACKUP,
+                          AArch64::CATCHRET),
       RI(STI.getTargetTriple()), Subtarget(STI) {}
 
 /// GetInstSize - Return the number of bytes of code the specified
 /// instruction may be.  This returns the maximum number of bytes.
 unsigned AArch64InstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   const MachineBasicBlock &MBB = *MI.getParent();
   const MachineFunction *MF = MBB.getParent();
   const MCAsmInfo *MAI = MF->getTarget().getMCAsmInfo();
bool AArch64InstrInfo::substituteCmpToZero(
   bool succeeded = UpdateOperandRegClass(*MI);
   (void)succeeded;
   assert(succeeded && "Some operands reg class are incompatible!");
   MI->addRegisterDefined(AArch64::NZCV, TRI);
   return true;
 }
 
 bool AArch64InstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
-  if (MI.getOpcode() != TargetOpcode::LOAD_STACK_GUARD)
+  if (MI.getOpcode() != TargetOpcode::LOAD_STACK_GUARD &&
+      MI.getOpcode() != AArch64::CATCHRET)
     return false;
 
   MachineBasicBlock &MBB = *MI.getParent();
   DebugLoc DL = MI.getDebugLoc();
+
+  if (MI.getOpcode() == AArch64::CATCHRET) {
+    // Skip to the first instruction before the epilog.
+    const TargetInstrInfo *TII =
+      MBB.getParent()->getSubtarget().getInstrInfo();
+    MachineBasicBlock *TargetMBB = MI.getOperand(0).getMBB();
+    auto MBBI = MachineBasicBlock::iterator(MI);
+    MachineBasicBlock::iterator FirstEpilogSEH = std::prev(MBBI);
+    while (FirstEpilogSEH->getFlag(MachineInstr::FrameDestroy) &&
+           FirstEpilogSEH != MBB.begin())
+      FirstEpilogSEH = std::prev(FirstEpilogSEH);
+    if (FirstEpilogSEH != MBB.begin())
+      FirstEpilogSEH = std::next(FirstEpilogSEH);
+    BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADRP))
+        .addReg(AArch64::X0)
+        .addMBB(TargetMBB);
+    BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADDXri))
+        .addReg(AArch64::X0)
+        .addReg(AArch64::X0)
+        .addMBB(TargetMBB)
+        .addImm(0);
+    return true;
+  }
+
   unsigned Reg = MI.getOperand(0).getReg();
   const GlobalValue *GV =
       cast<GlobalValue>((*MI.memoperands_begin())->getValue());
   const TargetMachine &TM = MBB.getParent()->getTarget();
   unsigned char OpFlags = Subtarget.ClassifyGlobalReference(GV, TM);
   const unsigned char MO_NC = AArch64II::MO_NC;
 
   if ((OpFlags & AArch64II::MO_GOT) != 0) {

lib/Target/AArch64/AArch64InstrInfo.td

let isPseudo = 1 in {
   def SEH_SetFP : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_AddFP : Pseudo<(outs), (ins i32imm:$offs), []>, Sched<[]>;
   def SEH_Nop : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_PrologEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogStart : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
 }
 
+// Pseudo instructions for Windows EH
+//===----------------------------------------------------------------------===//
+let isTerminator = 1, hasSideEffects = 1, isBarrier = 1, hasCtrlDep = 1,
+    isCodeGenOnly = 1, isReturn = 1, isEHScopeReturn = 1, isPseudo = 1 in {
+   def CLEANUPRET : Pseudo<(outs), (ins), [(cleanupret)]>, Sched<[]>;
+   let usesCustomInserter = 1 in
+     def CATCHRET : Pseudo<(outs), (ins am_brcond:$dst, am_brcond:$src), [(catchret bb:$dst, bb:$src)]>,
+                    Sched<[]>;
+}
+
+let hasSideEffects = 1, hasCtrlDep = 1, isCodeGenOnly = 1,
+    usesCustomInserter = 1 in
+def CATCHPAD : Pseudo<(outs), (ins), [(catchpad)]>, Sched<[]>;
+
 //===----------------------------------------------------------------------===//
 // Floating point immediate move.
 //===----------------------------------------------------------------------===//
 
 let isReMaterializable = 1 in {
 defm FMOV : FPMoveImmediate<"fmov">;
 }
 

lib/Target/AArch64/AArch64MCInstLower.cpp

case MachineOperand::MO_MCSymbol:
     MCOp = LowerSymbolOperand(MO, MO.getMCSymbol());
     break;
   case MachineOperand::MO_JumpTableIndex:
     MCOp = LowerSymbolOperand(MO, Printer.GetJTISymbol(MO.getIndex()));
     break;
   case MachineOperand::MO_ConstantPoolIndex:
     MCOp = LowerSymbolOperand(MO, Printer.GetCPISymbol(MO.getIndex()));
     break;
   case MachineOperand::MO_BlockAddress:
     MCOp = LowerSymbolOperand(
         MO, Printer.GetBlockAddressSymbol(MO.getBlockAddress()));
     break;
   }
   return true;
 }
 
 void AArch64MCInstLower::Lower(const MachineInstr *MI, MCInst &OutMI) const {
   OutMI.setOpcode(MI->getOpcode());
 
   for (const MachineOperand &MO : MI->operands()) {
     MCOperand MCOp;
     if (lowerOperand(MO, MCOp))
       OutMI.addOperand(MCOp);
   }
+
+  switch (OutMI.getOpcode()) {
+  case AArch64::CATCHRET:
+    OutMI = MCInst();
+    OutMI.setOpcode(AArch64::RET);
+    OutMI.addOperand(MCOperand::createReg(AArch64::LR));
+    break;
+  case AArch64::CLEANUPRET:
+    OutMI = MCInst();
+    OutMI.setOpcode(AArch64::RET);
+    OutMI.addOperand(MCOperand::createReg(AArch64::LR));
+    break;
+  }
 }

lib/Target/AArch64/AArch64RegisterInfo.h

public:
   const TargetRegisterClass *
   getSubClassWithSubReg(const TargetRegisterClass *RC,
                         unsigned Idx) const override;
 
   // Calls involved in thread-local variable lookup save more registers than
   // normal calls, so they need a different mask to represent this.
   const uint32_t *getTLSCallPreservedMask() const;
 
+  // Funclets on ARM64 Windows don't preserve any registers.
+  const uint32_t *getNoPreservedMask() const override;
+
   /// getThisReturnPreservedMask - Returns a call preserved mask specific to the
   /// case that 'returned' is on an i64 first argument if the calling convention
   /// is one that can (partially) model this attribute with a preserved mask
   /// (i.e. it is a calling convention that uses the same register for the first
   /// i64 argument and an i64 return value)
   ///
   /// Should return NULL in the case that the calling convention does not have
   /// this property

lib/Target/AArch64/AArch64RegisterInfo.cpp

if (MF.getSubtarget<AArch64Subtarget>().isXRegCustomCalleeSaved(i)) {
         // register mask.
         UpdatedMask[*SubReg / 32] |= 1u << (*SubReg % 32);
       }
     }
   }
   *Mask = UpdatedMask;
 }
 
+const uint32_t *AArch64RegisterInfo::getNoPreservedMask() const {
+  return CSR_AArch64_NoRegs_RegMask;
+}
+
 const uint32_t *
 AArch64RegisterInfo::getThisReturnPreservedMask(const MachineFunction &MF,
                                                 CallingConv::ID CC) const {
   // This should return a register mask that is the same as that returned by
   // getCallPreservedMask but that additionally preserves the register used for
   // the first i64 argument (which must also be the register used to return a
   // single i64 return value)
   //

 
 bool AArch64RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
 
   // In the presence of variable sized objects, if the fixed stack size is
   // large enough that referencing from the FP won't result in things being
   // in range relatively often, we can use a base pointer to allow access
   // from the other direction like the SP normally works.
   // Furthermore, if both variable sized objects are present, and the
   // stack needs to be dynamically re-aligned, the base pointer is the only
   // reliable way to reference the locals.
   if (MFI.hasVarSizedObjects()) {
     if (needsStackRealignment(MF))
       return true;
     // Conservatively estimate whether the negative offset from the frame
     // pointer will be sufficient to reach. If a function has a smallish
     // frame, it's less likely to have lots of spills and callee saved

test/CodeGen/AArch64/wineh-try-catch.ll

+; RUN: llc -o - %s -mtriple=aarch64-windows | FileCheck %s
+
+; We test the following
+; 1) That the unwind help object is created and that its offset from the stack
+;    pointer on entry is patched into the table fed to __CxxFrameHandler3
+; 2) That the stack update for the catch funclet only includes the callee saved
+;    registers
+; 3) That the locals are accessed using the frame pointer in both the funclet
+;    and the parent function.
+
+; The following checks that the unwind help object has -2 stored into it at
+; fp - 400 - 256 = fp - 656, which is on-entry sp - 48 + 32 - 656 =
+; on-entry sp - 672.  We check this offset in the table later on.
+
+; CHECK-LABEL: "?func@@YAHXZ":
+; CHECK:       str     x28, [sp, #-48]!
+; CHECK:       stp     x19, x20, [sp, #16]
+; CHECK:       stp     x29, x30, [sp, #32]
+; CHECK:       add     x29, sp, #32
+; CHECK:       sub     sp, sp, #624
+; CHECK:       orr     x1, xzr, #0xfffffffffffffffe
+; CHECK:       sub     x8, x29, #400
+; CHECK:       stur    x1, [x8, #-256]
+
+; Now check that x is stored at fp - 20.  We check that this is the same
+; location accessed from the funclet to retrieve x.
+; CHECK:       orr     w8, wzr, #0x1
+; CHECK:       stur    w8, [x29, [[X_OFFSET:#-[1-9][0-9]+]]
+
+; Check the offset off the frame pointer at which B is located.
+; Check the same offset is used to pass the address of B to init2 in the
+; funclet.
+; CHECK:       sub     x0, x29, [[B_OFFSET:#[1-9][0-9]+]]
+; CHECK:       bl      "?init@@YAXPEAH@Z"
+
+; This is the label for the throw that is encoded in the ip2state.
+; We are inside the try block, where we make a call to func2
+; CHECK-LABEL: .Ltmp0:
+; CHECK:       bl      "?func2@@YAHXZ
+
+; CHECK:        [[CATCHRETDEST:.LBB0_[0-9]+]]:      ; %catchret.dest
+
+; Check the catch funclet.
+; CHECK-LABEL: "?catch$2@?0??func@@YAHXZ@4HA":
+
+; Check that the stack space is allocated only for the callee saved registers.
+; CHECK:       str     x28, [sp, #-48]!
+; CHECK:       stp     x19, x20, [sp, #16]
+; CHECK:       stp     x29, x30, [sp, #32]
+; CHECK:       sub     x19, x29, #636
+
+; Check that there are no further stack updates.
+; CHECK-NOT:   sub     sp, sp
+
+; Check that the stack address passed to init2 is off the frame pointer, and
+; that it matches the address of B in the parent function.
+; CHECK:       sub     x0, x29, [[B_OFFSET]]
+; CHECK:       bl      "?init2@@YAXPEAH@Z"
+
+; Check that are storing x back to the same location off the frame pointer as in
+; the parent function.
+; CHECK:       stur    w8, [x29, [[X_OFFSET]]]
+
+; Check that the funclet branches back to the catchret destination
+; CHECK:       adrp    x0, .LBB0_3
+; CHECK-NEXT:  add     x0, x0, [[CATCHRETDEST]]
+
+
+; Now check that the offset of the unwind help object from the stack pointer on
+; entry to func is encoded in cppxdata that is passed to __CxxFrameHandler3.  As
+; computed above, this comes to -672.
+; CHECK-LABEL:        "$cppxdata$?func@@YAHXZ":
+; CHECK-NEXT:         .word   429065506               ; MagicNumber
+; CHECK-NEXT:         .word   2                       ; MaxState
+; CHECK-NEXT:         .word   ("$stateUnwindMap$?func@@YAHXZ")@IMGREL ; UnwindMap
+; CHECK-NEXT:         .word   1                       ; NumTryBlocks
+; CHECK-NEXT:         .word   ("$tryMap$?func@@YAHXZ")@IMGREL ; TryBlockMap
+; CHECK-NEXT:         .word   4                       ; IPMapEntries
+; CHECK-NEXT:         .word   ("$ip2state$?func@@YAHXZ")@IMGREL ; IPToStateXData
+; CHECK-NEXT:         .word   -672                    ; UnwindHelp
+
+
+
+target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-windows-msvc19.11.0"
+
+%rtti.TypeDescriptor2 = type { i8**, i8*, [3 x i8] }
+%eh.CatchableType = type { i32, i32, i32, i32, i32, i32, i32 }
+%eh.CatchableTypeArray.1 = type { i32, [1 x i32] }
+%eh.ThrowInfo = type { i32, i32, i32, i32 }
+
+$"??_R0H@8" = comdat any
+
+$"_CT??_R0H@84" = comdat any
+
+$_CTA1H = comdat any
+
+$_TI1H = comdat any
+
+@"??_7type_info@@6B@" = external constant i8*
+@"??_R0H@8" = linkonce_odr global %rtti.TypeDescriptor2 { i8** @"??_7type_info@@6B@", i8* null, [3 x i8] c".H\00" }, comdat
+@__ImageBase = external dso_local constant i8
+@"_CT??_R0H@84" = linkonce_odr unnamed_addr constant %eh.CatchableType { i32 1, i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%rtti.TypeDescriptor2* @"??_R0H@8" to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32), i32 0, i32 -1, i32 0, i32 4, i32 0 }, section ".xdata", comdat
+@_CTA1H = linkonce_odr unnamed_addr constant %eh.CatchableTypeArray.1 { i32 1, [1 x i32] [i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%eh.CatchableType* @"_CT??_R0H@84" to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32)] }, section ".xdata", comdat
+@_TI1H = linkonce_odr unnamed_addr constant %eh.ThrowInfo { i32 0, i32 0, i32 0, i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%eh.CatchableTypeArray.1* @_CTA1H to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32) }, section ".xdata", comdat
+
+; Function Attrs: noinline optnone
+define dso_local i32 @"?func@@YAHXZ"() #0 personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %B = alloca [50 x i32], align 4
+  %x = alloca i32, align 4
+  %tmp = alloca i32, align 4
+  %i = alloca i32, align 4
+  %C = alloca [100 x i32], align 4
+  store i32 1, i32* %x, align 4
+  %arraydecay = getelementptr inbounds [50 x i32], [50 x i32]* %B, i32 0, i32 0
+  call void @"?init@@YAXPEAH@Z"(i32* %arraydecay)
+  %call = invoke i32 @"?func2@@YAHXZ"()
+          to label %invoke.cont unwind label %catch.dispatch
+
+invoke.cont:                                      ; preds = %entry
+  store i32 %call, i32* %tmp, align 4
+  %0 = bitcast i32* %tmp to i8*
+  invoke void @_CxxThrowException(i8* %0, %eh.ThrowInfo* @_TI1H) #2
+          to label %unreachable unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %invoke.cont, %entry
+  %1 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %2 = catchpad within %1 [%rtti.TypeDescriptor2* @"??_R0H@8", i32 0, i32* %i]
+  %arraydecay1 = getelementptr inbounds [100 x i32], [100 x i32]* %C, i32 0, i32 0
+  call void @"?init@@YAXPEAH@Z"(i32* %arraydecay1) [ "funclet"(token %2) ]
+  %arraydecay2 = getelementptr inbounds [50 x i32], [50 x i32]* %B, i32 0, i32 0
+  call void @"?init2@@YAXPEAH@Z"(i32* %arraydecay2) [ "funclet"(token %2) ]
+  %3 = load i32, i32* %i, align 4
+  %idxprom = sext i32 %3 to i64
+  %arrayidx = getelementptr inbounds [50 x i32], [50 x i32]* %B, i64 0, i64 %idxprom
+  %4 = load i32, i32* %arrayidx, align 4
+  %5 = load i32, i32* %i, align 4
+  %idxprom3 = sext i32 %5 to i64
+  %arrayidx4 = getelementptr inbounds [100 x i32], [100 x i32]* %C, i64 0, i64 %idxprom3
+  %6 = load i32, i32* %arrayidx4, align 4
+  %add = add nsw i32 %4, %6
+  %7 = load i32, i32* %i, align 4
+  %8 = load i32, i32* %i, align 4
+  %mul = mul nsw i32 %7, %8
+  %add5 = add nsw i32 %add, %mul
+  store i32 %add5, i32* %x, align 4
+  catchret from %2 to label %catchret.dest
+
+catchret.dest:                                    ; preds = %catch
+  br label %try.cont
+
+try.cont:                                         ; preds = %catchret.dest
+  %arrayidx6 = getelementptr inbounds [50 x i32], [50 x i32]* %B, i64 0, i64 2
+  %9 = load i32, i32* %arrayidx6, align 4
+  %10 = load i32, i32* %x, align 4
+  %add7 = add nsw i32 %9, %10
+  ret i32 %add7
+
+unreachable:                                      ; preds = %invoke.cont
+  unreachable
+}
+
+declare dso_local void @"?init@@YAXPEAH@Z"(i32*)
+
+declare dso_local i32 @"?func2@@YAHXZ"()
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+
+declare dllimport void @_CxxThrowException(i8*, %eh.ThrowInfo*)
+
+declare dso_local void @"?init2@@YAXPEAH@Z"(i32*)
+
+attributes #0 = { noinline optnone }
+attributes #2 = { noreturn }