[ARM64] [Windows] Handle funclets

[efriedma]

Summary

This patch adds support for funclets in frame lowering and ISel lowering. Together with D50288 and D50166, it enables C++ exception handling.

[mgrang]

lib/Target/AArch64/AArch64FrameLowering.cpp

910 ↗

(On Diff #163457)

default should be the first case.

lib/Target/AArch64/AArch64InstrInfo.cpp

1606 ↗

(On Diff #163457)

Can you move "MBB.getParent()-" to the next line so that it's more readable?

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

Check 80 char limits here.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

910 ↗

(On Diff #163457)

Will change it in the updated patch.

lib/Target/AArch64/AArch64InstrInfo.cpp

1606 ↗

(On Diff #163457)

Will do in the updated patch.

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

Is the 80 char limit on the RUN line a rule? If so, many test cases don't follow this. A single RUN is not split into multiple lines from what I've seen.

[ssijaric]

A few formatting fixes.

[mgrang]

test/CodeGen/AArch64/wineh-funclets.ll

1 ↗

(On Diff #163457)

< many test cases don't follow this.
They should! 80 chars is what the LLVM coding standards recommend:
https://llvm.org/docs/CodingStandards.html#source-code-formatting

Write your code to fit within 80 columns of text. This helps those of us who like to print out code and look at your code in an xterm without resizing it.

Plus, horizontal scrolling is no fun :)

[rnk]

Neat. :)

lib/Target/AArch64/AArch64FrameLowering.cpp

645 ↗

(On Diff #163556)

"containing"

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1638 ↗

(On Diff #163556)

UnwidHelp -> UnwindHelp?

1639–1641 ↗

(On Diff #163556)

Should this be moved after the terminator check?

1675–1679 ↗

(On Diff #163556)

This is used to populate dispFrame in _s_HandlerType IIRC. IIUC, you will have issues with certain catch handlers without this properly populated.

[MatzeB]

I don't really know how exception handling on windows works or what a funclet is. So here's just a couple style comments and questions

lib/Target/AArch64/AArch64FrameLowering.cpp

909 ↗

(On Diff #163556)

Can be const MachineInstr&

934–935 ↗

(On Diff #163556)

Out of interest: How is getWinEHFuncletFrameSize() different from MachineFrameIndo::getStackSize()?

1182 ↗

(On Diff #163556)

Add a comment explaining why you need a frame pointer for funclet?
Knowing nothing about funclets myself I wonder why the normal rules for determining whether we need FP are not enough here...

1648 ↗

(On Diff #163556)

In case the previous loop ends with MBBI == MBB.end() all the following tests will be undefined behavior.

1652 ↗

(On Diff #163556)

My understanding is that all of this stuff is just compiler generated extra instructions? In that case I recommend using an empty debug location instead (I know other places of the code get this wrong too...)

1662 ↗

(On Diff #163556)

Use doxygen /// comments.

1668–1672 ↗

(On Diff #163556)

doesn't this change the behavior for non-funclets too? If yes this needs to go into a separate patch and needs some comments why.

lib/Target/AArch64/AArch64MCInstLower.cpp

256–268 ↗

(On Diff #163556)

Why is this in MCInstLower and not AsmPrinter.cpp? Maybe you can even specify it in tablegen (see for example ARMs LDMIA_RET and similar)

test/CodeGen/AArch64/wineh-funclets.ll

60 ↗

(On Diff #163556)

unused basic block?

76 ↗

(On Diff #163556)

You should be able to drop most of these attributes.

79–83 ↗

(On Diff #163556)

I'm pretty sure you can drop these lines from the test

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1675–1679 ↗

(On Diff #163556)

Ah, it could be that dispFrame is not needed for ARM64. In that case, WinException.cpp should not call getWinEHParentFrameOffset and the ParentFrameOffset should not be emitted as it will add padding between the HandlerType elements.

[majnemer]

lib/Target/AArch64/AArch64FrameLowering.cpp

1675–1679 ↗

(On Diff #163556)

Hmmm... So after a bunch of poking around with ARM64 CL.exe, I can confirm that they are allocating the dispFrame field and always fill it with zero!

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

645 ↗

(On Diff #163556)

Will fix in the updated test case.

909 ↗

(On Diff #163556)

Will change in the updated patch.

934–935 ↗

(On Diff #163556)

The stack size (getStackSize()) includes all the locals in its size calculation. We don't include these locals when computing the stack size of a funclet, as they are allocated in the parent's stack frame and accessed via the frame pointer from the funclet. Otherwise, we would be double allocating stack space in the funclet that we don't need. We only save the callee saved registers in the funclet, which are really the callee saved registers of the parent function, including the funclet. Perhaps, only the non-volatile registers that the funclet modifies should be saved/restored in the funclet prologue/epilogue, but we don't keep track of these separately.

I will document this.

1182 ↗

(On Diff #163556)

Will do in the updated patch.

1638 ↗

(On Diff #163556)

Will fix in the updated patch.

1639–1641 ↗

(On Diff #163556)

Yes, it's definitely a better place for it.

1652 ↗

(On Diff #163556)

Will fix in the updated patch.

1662 ↗

(On Diff #163556)

Will fix in the updated patch.

1675–1679 ↗

(On Diff #163556)

Thanks for double checking!! Will leave it as is.

lib/Target/AArch64/AArch64MCInstLower.cpp

256–268 ↗

(On Diff #163556)

X86 did the expansion in X86MCInstLower.cpp. I would keep this way in the current patch, and look into tablegen later. I'll add a comment.

test/CodeGen/AArch64/wineh-funclets.ll

60 ↗

(On Diff #163556)

The block is generated by clang, and SImplifyCFG removes it. I'll come up with a better test case.

76 ↗

(On Diff #163556)

Will update the test case.

79–83 ↗

(On Diff #163556)

Will update the test case.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

1668–1672 ↗

(On Diff #163556)

This is used to calculate displacement of objects from the stack pointer, such as the displacement of the unwind object when filling up the exception handling table on Windows. This is the only place where it is used for AArch64. All the frame indices belong to the parent function (e.g. never a funclet). I put it in this patch, because the unwind object above is created only if the function has funclets.

It may be hard to write a test case that checks the .s file, as it will crash with an assert in the WinException sanity check (since the displacement will be that from the frame pointer). I would like to keep in this patch, if at all possible. If there is a really strong objection, I can move it to a separate patch.

[ssijaric]

Address comments: spelling mistakes, better comments, a better test case.

[ssijaric]

lib/Target/AArch64/AArch64ISelLowering.cpp

1278 ↗

(On Diff #172089)

Move this into the assert; it will cause the release only build to fail.

[ssijaric]

Remove an unused variable. Fix a nasty bug that Eli just pointed out, where I didn't insert a PrologEnd in funclets in this patch. Previously, I relied on an RAII object to this for me when we would go out of scope. I removed that logic, but forgot to insert a prolog end marker for the funclet case.

With this patch, the following should now pass:

#include <stdio.h>
int
main() {
  int B[100];
  int x = 4;
  B[2] = 2;
  B[4] = 0;
  try {
    throw B[2];
  } catch (int i) {
    printf("Caught %d\n", i);
    B[4] = B[i] + i;
    try {
      throw 'c';
    } catch (char c) {
      printf("Caught char %c\n", c);
    }
  }
  return x + B[4];
}

with the correct print outs and a return value of 8.

[ssijaric]

Fix the check for IsFunclet in emitEpilogue.

[efriedma]

I'm not really an expert on funclets, but the code looks fine to me.

The test coverage here is a little thin... in particular, there isn't any coverage of the emitted unwind opcodes. I'll amend the testcase to check the llvm-readobj output. (Sanjin isn't available for the next few weeks, so I'm going to handle any further revisions to the patch.)

Reid, do you have an opinion on what test coverage is necessary here?

[rnk]

In D51524#1285854, @efriedma wrote:

Reid, do you have an opinion on what test coverage is necessary here?

It's probably important to handle each and every kind of prologue. I don't know if there are any special corners cases for arm64 prologues, but for x64, this ended up being:

• normal prologue
• normal with -fno-omit-frame-pointer
• stack realignment caused by a highly aligned alloca
• normal with dynamic alloca or VLA
• dynamic alloca with highly aligned static alloca

SEH and C++ EH have major differences, so test both. Alternatively, you can focus on one in one patch and defer work on the other.

SEH __except blocks in particular had to do some extra work to restore SP and register state when recovering from an exception, especially when the frame has dynamic SP updates.

I still need to go back and review, but hopefully that helps.

[rnk]

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831 ↗

(On Diff #172305)

We had bugs here with XMM registers, which I don't think we fixed: https://llvm.org/pr32507

How does the AArch64 prologue/epilogue save and restore registers? If you load/store to a fixed frame index stack object, it's not going to work well. If it does something analogous to x86 "PUSH" instructions, it'll work fine.

[efriedma]

This patch forces all functions that contain funclets to emit a frame pointer. (The existing testcase covers that.)

I'll do some testing with realignment and dynamic allocas. I don't think AArch64 prologue lowering has any other special cases.

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831 ↗

(On Diff #172305)

Spills of callee-saved registers should always use sp-relative loads/stores, as far as I know.

[efriedma]

Oh, I guess there is one special case for the prologue I didn't mention: varargs functions. I think the patch handles that correctly already, but I'll add a test.

For both dynamic allocas and stack realignment, the initial implementation of unwinding missed a significant rule: "r29 must point to the bottom of stack". That's probably straightforward to resolve, but I don't really want to do it in this patch (and I'm not sure what the practical effect is). Beyond that, there's probably more handling necessary to correctly handle a dynamic alloca inside a funclet (since we need to reset the frame pointer, but we still need access to the old frame pointer). And I don't really want to block this patch on solving that; it's obscure enough that it won't affect most code.

I have a couple small fixes for stack realignment, a fix for a MachineVerifier failure, and a testcase for realignment; I'll upload a new version.

[efriedma]

Misc fixes.

[efriedma]

The following testcase is also broken, for reasons I don't understand. I don't think it's a problem with any of the code in this patch, though; the throw out of the "catch" block somehow picks the wrong catch block, but this patch doesn't touch that code.

void a() { try { try { throw 1; } catch (...) { throw 1; } } catch (...) {}}

[efriedma]

Add a bit more test coverage, to cover cases with and without base pointers. Tweak base pointer emission so we emit and use a base pointer where appropriate (including some cases where it isn't strictly necessary, but allows more efficient addressing modes).

I think this covers all the interesting cases for C++ exceptions. (There will be a separate followup patch for SEH exceptions.)

[rnk]

In D51524#1286346, @efriedma wrote:

The following testcase is also broken, for reasons I don't understand. I don't think it's a problem with any of the code in this patch, though; the throw out of the "catch" block somehow picks the wrong catch block, but this patch doesn't touch that code.

void a() { try { try { throw 1; } catch (...) { throw 1; } } catch (...) {}}

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

[efriedma]

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

Looked a bit more; it looks like the problem is actually that the "throw" is the last instruction in the function, so unwinding gets confused. There's some code in X86TargetMachine::X86TargetMachine which turns on TrapUnreachable, which avoids the problem. It looks like MSVC does something similar (it inserts a nop instead of a trap, but same idea). I'll post a patch separately.

[ssijaric]

lib/Target/AArch64/AArch64FrameLowering.cpp

829–831 ↗

(On Diff #172305)

They should always be saved off the funclet's SP pointer. This is what cl does, if I remember correctly.

[ssijaric]

In D51524#1286331, @efriedma wrote:

Oh, I guess there is one special case for the prologue I didn't mention: varargs functions. I think the patch handles that correctly already, but I'll add a test.

For both dynamic allocas and stack realignment, the initial implementation of unwinding missed a significant rule: "r29 must point to the bottom of stack". That's probably straightforward to resolve, but I don't really want to do it in this patch (and I'm not sure what the practical effect is). Beyond that, there's probably more handling necessary to correctly handle a dynamic alloca inside a funclet (since we need to reset the frame pointer, but we still need access to the old frame pointer). And I don't really want to block this patch on solving that; it's obscure enough that it won't affect most code.

I have a couple small fixes for stack realignment, a fix for a MachineVerifier failure, and a testcase for realignment; I'll upload a new version.

Hi Eli,

Yin and I looked at the rule that says fp must be at the bottom if alloca is present. We tried a few (simple) test cases to see if this would be a problem. We didn't encounter any issues with our very limited testing, so decided not to implement it at this time. From my recollection, the fp (or the fp/lr pair) has to be separated from the code that saves the callee-saved registers, and handled on its own so that it ends up pointing to the bottom.

Thank you for taking over this!!! I may have some time time between flights to look at this a bit more, depending on the internet connection.

[thegameg]

I am afraid I don't know much about exceptions on Windows in general. A few comments below.

lib/Target/AArch64/AArch64FrameLowering.cpp

1497 ↗

(On Diff #172668)

Small typo: in the in -> in the

2059 ↗

(On Diff #172668)

Why is this needed?

2114 ↗

(On Diff #172668)

typo paren't -> parent's

lib/Target/AArch64/AArch64RegisterInfo.cpp

260 ↗

(On Diff #172668)

Isn't this already caught by the next nested ifs?

test/CodeGen/AArch64/wineh-try-catch-realign.ll

10 ↗

(On Diff #172668)

Is there a chance that we can test the PEI/FrameLowering part with more specific MIR tests? Especially the checks for what the base register should be. The fact that we don't serialize AArch64FunctionInfo may make this impossible for now.

[rnk]

In D51524#1288018, @efriedma wrote:

It works for x64 and x86, so I guess it's arm-specific. The state table emission code is portable, so maybe the bug is in the arm ip2state mapping logic?

Looked a bit more; it looks like the problem is actually that the "throw" is the last instruction in the function, so unwinding gets confused. There's some code in X86TargetMachine::X86TargetMachine which turns on TrapUnreachable, which avoids the problem. It looks like MSVC does something similar (it inserts a nop instead of a trap, but same idea). I'll post a patch separately.

We came up with a different solution for this called SEH_Epilogue. It basically inserts a nop if the last real instruction was a call. It's slightly complicated because it looks backwards across empty basic blocks and things. You can see it here:
https://github.com/llvm-mirror/llvm/blob/master/lib/Target/X86/X86MCInstLower.cpp#L1800

I've been meaning to see if we can remove the code that enables TrapUnreachable on Windows.

[efriedma]

lib/Target/AArch64/AArch64FrameLowering.cpp

2059 ↗

(On Diff #172668)

It's not, as far as I can tell; I'll get rid of it.

lib/Target/AArch64/AArch64RegisterInfo.cpp

260 ↗

(On Diff #172668)

Yes, I refactored it and didn't realize it. Will fix.

test/CodeGen/AArch64/wineh-try-catch-realign.ll

10 ↗

(On Diff #172668)

Yes, I could write an MIR test using -run-pass=prologepilog, but I don't see the point. The CHECK lines would be essentially identical, just checking for MIR opcodes instead of assembly instructions. The MIR doesn't directly record whether we have a base register, or what it is; it's only used as part of frame lowering.

[efriedma]

Address review comments (no functional change from the previous version).

[efriedma]

Reid, are you okay with merging this? I'd prefer to get this merged sooner rather than later. (If you don't have to review carefully right now, I can address any minor issues post-commit.)

[rnk]

lgtm

Sorry, I lost track of this.

lib/Target/AArch64/AArch64InstrInfo.cpp

1681 ↗

(On Diff #173041)

Yep, makes sense, put the address of the target MBB in X0. Do you have any assurance that X0 isn't used by normal epilogue code? At least on x86, sometimes RAX gets used for large SP updates.

test/CodeGen/AArch64/wineh-try-catch.ll

13–15 ↗

(On Diff #173041)

Tests with comments about the correctness of numbers that we filecheck for below are always worrying, but I have no better suggestions. :)

[efriedma]

lib/Target/AArch64/AArch64InstrInfo.cpp

1681 ↗

(On Diff #173041)

The current AArch64 emitFrameOffset never does that. And even if it did in some cases, a funclet's stack can't be more than a few hundred bytes anyway.

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.h

public:
 
   /// Returns true if the target will correctly handle shrink wrapping.
   bool enableShrinkWrapping(const MachineFunction &MF) const override {
     return true;
   }
 
   bool enableStackSlotScavenging(const MachineFunction &MF) const override;
 
+  void processFunctionBeforeFrameFinalized(MachineFunction &MF,
+                                             RegScavenger *RS) const override;
+
+  unsigned getWinEHParentFrameOffset(const MachineFunction &MF) const override;
+
+  unsigned getWinEHFuncletFrameSize(const MachineFunction &MF) const;
+
+  int getFrameIndexReferencePreferSP(const MachineFunction &MF, int FI,
+                                     unsigned &FrameReg,
+                                     bool IgnoreSPUpdates) const override;
+
 private:
   bool shouldCombineCSRLocalStackBump(MachineFunction &MF,
                                       unsigned StackBumpBytes) const;
 };
 
 } // End llvm namespace
 
 #endif

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.cpp

bool AArch64FrameLowering::canUseRedZone(const MachineFunction &MF) const {
   return !(MFI.hasCalls() || hasFP(MF) || NumBytes > 128);
 }
 
 /// hasFP - Return true if the specified function should have a dedicated frame
 /// pointer register.
 bool AArch64FrameLowering::hasFP(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
+  // Win64 EH requires a frame pointer if funclets are present, as the locals
+  // are accessed off the frame pointer in both the parent function and the
+  // funclets.
+  if (MF.hasEHFunclets())
+    return true;
   // Retain behavior of always omitting the FP for leaf functions when possible.
   if (MFI.hasCalls() && MF.getTarget().Options.DisableFramePointerElim(MF))
     return true;
   if (MFI.hasVarSizedObjects() || MFI.isFrameAddressTaken() ||
       MFI.hasStackMap() || MFI.hasPatchPoint() ||
       RegInfo->needsStackRealignment(MF))
     return true;
   // With large callframes around we may need to use FP to access the scavenging
if (!F.hasFnAttribute("sign-return-address-key"))
     return true;
 
   const StringRef Key =
       F.getFnAttribute("sign-return-address-key").getValueAsString();
   assert(Key.equals_lower("a_key") || Key.equals_lower("b_key"));
   return Key.equals_lower("a_key");
 }
 
+static bool needsWinCFI(const MachineFunction &MF) {
+  const Function &F = MF.getFunction();
+  return MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+         F.needsUnwindTableEntry();
+}
+
 void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.begin();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const Function &F = MF.getFunction();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   MachineModuleInfo &MMI = MF.getMMI();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   bool needsFrameMoves = (MMI.hasDebugInfo() || F.needsUnwindTableEntry()) &&
                          !MF.getTarget().getMCAsmInfo()->usesWindowsCFI();
   bool HasFP = hasFP(MF);
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     F.needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   MF.setHasWinCFI(NeedsWinCFI);
+  bool IsFunclet = MBB.isEHFuncletEntry();
+
   // At this point, we're going to decide whether or not the function uses a
   // redzone. In most cases, the function doesn't have a redzone so let's
   // assume that's false and set it to true in the case that there's a redzone.
   AFI->setHasRedZone(false);
 
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
 
   if (ShouldSignReturnAddress(MF)) {
     BuildMI(
         MBB, MBBI, DL,
         TII->get(ShouldSignWithAKey(MF) ? AArch64::PACIASP : AArch64::PACIBSP))
         .setMIFlag(MachineInstr::FrameSetup);
   }
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
-  int NumBytes = (int)MFI.getStackSize();
+  // getStackSize() includes all the locals in its size calculation. We don't
+  // include these locals when computing the stack size of a funclet, as they
+  // are allocated in the parent's stack frame and accessed via the frame
+  // pointer from the funclet.  We only save the callee saved registers in the
+  // funclet, which are really the callee saved registers of the parent
+  // function, including the funclet.
+  int NumBytes = IsFunclet ? (int)getWinEHFuncletFrameSize(MF)
+                           : (int)MFI.getStackSize();
   if (!AFI->hasStackFrame() && !windowsRequiresStackProbe(MF, NumBytes)) {
     assert(!HasFP && "unexpected function without stack frame but with FP");
     // All of the stack allocation is for locals.
     AFI->setLocalStackSize(NumBytes);
     if (!NumBytes)
       return;
     // REDZONE: If the stack size is less than 128 bytes, we don't need
     // to actually allocate.
if (NeedsWinCFI)
       BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_PrologEnd))
           .setMIFlag(MachineInstr::FrameSetup);
 
     return;
   }
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
-  unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
+  // Var args are accounted for in the containing function, so don't
+  // include them for funclets.
+  unsigned FixedObject = (IsWin64 && !IsFunclet) ?
+                         alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
 
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   // All of the remaining stack allocations are for locals.
   AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   if (CombineSPBump) {
     emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
                     MachineInstr::FrameSetup, false, NeedsWinCFI);
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   MachineBasicBlock::iterator End = MBB.end();
   while (MBBI != End && MBBI->getFlag(MachineInstr::FrameSetup)) {
     if (CombineSPBump)
       fixupCalleeSaveRestoreStackOffset(*MBBI, AFI->getLocalStackSize(),
                                         NeedsWinCFI);
     ++MBBI;
   }
 
+  // The code below is not applicable to funclets. We have emitted all the SEH
+  // opcodes that we needed to emit.  The FP and BP belong to the containing
+  // function.
+  if (IsFunclet) {
+    if (NeedsWinCFI)
+      BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_PrologEnd))
+          .setMIFlag(MachineInstr::FrameSetup);
+    return;
+  }
+
   if (HasFP) {
     // Only set up FP if we actually need to. Frame pointer is fp =
     // sp - fixedobject - 16.
     int FPOffset = AFI->getCalleeSavedStackSize() - 16;
     if (CombineSPBump)
       FPOffset += AFI->getLocalStackSize();
 
     // Issue    sub fp, sp, FPOffset or
static void InsertReturnAddressAuth(MachineFunction &MF,
   } else {
     BuildMI(
         MBB, MBBI, DL,
         TII->get(ShouldSignWithAKey(MF) ? AArch64::AUTIASP : AArch64::AUTIBSP))
         .setMIFlag(MachineInstr::FrameDestroy);
   }
 }
 
+static bool isFuncletReturnInstr(const MachineInstr &MI) {
+  switch (MI.getOpcode()) {
+  default:
+    return false;
+  case AArch64::CATCHRET:
+  case AArch64::CLEANUPRET:
+    return true;
+  }
+}
+
 void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   DebugLoc DL;
   bool IsTailCallReturn = false;
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
+  bool IsFunclet = false;
 
   if (MBB.end() != MBBI) {
     DL = MBBI->getDebugLoc();
     unsigned RetOpcode = MBBI->getOpcode();
     IsTailCallReturn = RetOpcode == AArch64::TCRETURNdi ||
                        RetOpcode == AArch64::TCRETURNri ||
                        RetOpcode == AArch64::TCRETURNriBTI;
+    IsFunclet = isFuncletReturnInstr(*MBBI);
   }
 
-  int NumBytes = MFI.getStackSize();
+  int NumBytes = IsFunclet ? (int)getWinEHFuncletFrameSize(MF)
+                           : MFI.getStackSize();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   // Initial and residual are named for consistency with the prologue. Note that
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   auto Cleanup = make_scope_exit([&] { InsertReturnAddressAuth(MF, MBB); });
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
 
   uint64_t AfterCSRPopSize = ArgumentPopSize;
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
+  // Var args are accounted for in the containting function, so don't
+  // include them for funclets.
+  if (MF.hasEHFunclets())
+    AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   // Assume we can't combine the last pop with the sp restore.
 
   if (!CombineSPBump && PrologueSaveSize != 0) {
     MachineBasicBlock::iterator Pop = std::prev(MBB.getFirstTerminator());
     while (AArch64InstrInfo::isSEHInstruction(*Pop))
       Pop = std::prev(Pop);
     // Converting the last ldp to a post-index ldp is valid only if the last
if (!hasFP(MF)) {
 
     NumBytes = 0;
   }
 
   // Restore the original stack pointer.
   // FIXME: Rather than doing the math here, we should instead just use
   // non-post-indexed loads for the restores if we aren't actually going to
   // be able to save any instructions.
-  if (MFI.hasVarSizedObjects() || AFI->isStackRealigned())
+  if (!IsFunclet && (MFI.hasVarSizedObjects() || AFI->isStackRealigned()))
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::FP,
                     -AFI->getCalleeSavedStackSize() + 16, TII,
                     MachineInstr::FrameDestroy, false, NeedsWinCFI);
   else if (NumBytes)
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP, NumBytes, TII,
                     MachineInstr::FrameDestroy, false, NeedsWinCFI);
 
   // This must be placed after the callee-save restore code because that code
if (isFixed) {
         // else we can use BP and FP, but the offset from FP won't fit.
         // That will make us scavenge registers which we can probably avoid by
         // using BP. If it won't fit for BP either, we'll scavenge anyway.
       } else if (FPOffset >= 0) {
         // Use SP or FP, whichever gives us the best chance of the offset
         // being in range for direct access. If the FPOffset is positive,
         // that'll always be best, as the SP will be even further away.
         UseFP = true;
+      } else if (MF.hasEHFunclets() && !RegInfo->hasBasePointer(MF)) {
+        // Funclets access the locals contained in the parent's stack frame
+        // via the frame pointer, so we have to use the FP in the parent
+        // function.
+        assert(
+            Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv()) &&
+            "Funclets should only be present on Win64");
+        UseFP = true;
       } else {
         // We have the choice between FP and (SP or BP).
         if (FPOffsetFits && PreferFP) // If FP is the best fit, use it.
           UseFP = true;
       }
     }
   }
 

 static void computeCalleeSaveRegisterPairs(
     MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI, SmallVectorImpl<RegPairInfo> &RegPairs,
     bool &NeedShadowCallStackProlog) {
 
   if (CSI.empty())
     return;
 
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   CallingConv::ID CC = MF.getFunction().getCallingConv();
   unsigned Count = CSI.size();
   (void)CC;
   // MachO's compact unwind format relies on all registers being stored in
   // pairs.
   assert((!produceCompactUnwindFrame(MF) ||

 }
 
 bool AArch64FrameLowering::spillCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
   const MachineRegisterInfo &MRI = MF.getRegInfo();
 

 bool AArch64FrameLowering::restoreCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
-  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
-                     MF.getFunction().needsUnwindTableEntry();
+  bool NeedsWinCFI = needsWinCFI(MF);
 
   if (MI != MBB.end())
     DL = MI->getDebugLoc();
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
 
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   AFI->setCalleeSaveStackHasFreeSpace(AlignedCSStackSize != CSStackSize);
 }
 
 bool AArch64FrameLowering::enableStackSlotScavenging(
     const MachineFunction &MF) const {
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   return AFI->hasCalleeSaveStackFreeSpace();
 }
+
+void AArch64FrameLowering::processFunctionBeforeFrameFinalized(
+    MachineFunction &MF, RegScavenger *RS) const {
+  // If this function isn't doing Win64-style C++ EH, we don't need to do
+  // anything.
+  if (!MF.hasEHFunclets())
+    return;
+  const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  WinEHFuncInfo &EHInfo = *MF.getWinEHFuncInfo();
+
+  MachineBasicBlock &MBB = MF.front();
+  auto MBBI = MBB.begin();
+  while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup))
+    ++MBBI;
+
+  if (MBBI->isTerminator())
+    return;
+
+  // Create an UnwindHelp object.
+  int UnwindHelpFI =
+      MFI.CreateStackObject(/*size*/8, /*alignment*/16, false);
+  EHInfo.UnwindHelpFrameIdx = UnwindHelpFI;
+  // We need to store -2 into the UnwindHelp object at the start of the
+  // function.
+  DebugLoc DL;
+  RS->enterBasicBlock(MBB);
+  unsigned DstReg = RS->scavengeRegister(&AArch64::GPR64RegClass, MBBI, 0);
+  BuildMI(MBB, MBBI, DL, TII.get(AArch64::MOVi64imm), DstReg).addImm(-2);
+  BuildMI(MBB, MBBI, DL, TII.get(AArch64::STURXi))
+      .addReg(DstReg, getKillRegState(true))
+      .addFrameIndex(UnwindHelpFI)
+      .addImm(0);
+}
+
+/// For Win64 AArch64 EH, the offset to the Unwind object is from the SP before
+/// the update.  This is easily retrieved as it is exactly the offset that is set
+/// in processFunctionBeforeFrameFinalized.
+int AArch64FrameLowering::getFrameIndexReferencePreferSP(
+    const MachineFunction &MF, int FI, unsigned &FrameReg,
+    bool IgnoreSPUpdates) const {
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  LLVM_DEBUG(dbgs() << "Offset from the SP for " << FI << " is "
+                    << MFI.getObjectOffset(FI) << "\n");
+  FrameReg = AArch64::SP;
+  return MFI.getObjectOffset(FI);
+}
+
+/// The parent frame offset (aka dispFrame) is only used on X86_64 to retrieve
+/// the parent's frame pointer
+unsigned AArch64FrameLowering::getWinEHParentFrameOffset(
+    const MachineFunction &MF) const {
+  return 0;
+}
+
+/// Funclets only need to account for space for the callee saved registers,
+/// as the locals are accounted for in the parent's stack frame.
+unsigned AArch64FrameLowering::getWinEHFuncletFrameSize(
+    const MachineFunction &MF) const {
+  // This is the size of the pushed CSRs.
+  unsigned CSSize =
+      MF.getInfo<AArch64FunctionInfo>()->getCalleeSavedStackSize();
+  // This is the amount of stack a funclet needs to allocate.
+  return alignTo(CSSize + MF.getFrameInfo().getMaxCallFrameSize(),
+                 getStackAlignment());
+}

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.h

public:
   EVT getSetCCResultType(const DataLayout &DL, LLVMContext &Context,
                          EVT VT) const override;
 
   SDValue ReconstructShuffle(SDValue Op, SelectionDAG &DAG) const;
 
   MachineBasicBlock *EmitF128CSEL(MachineInstr &MI,
                                   MachineBasicBlock *BB) const;
 
+  MachineBasicBlock *EmitLoweredCatchRet(MachineInstr &MI,
+                                           MachineBasicBlock *BB) const;
+
+  MachineBasicBlock *EmitLoweredCatchPad(MachineInstr &MI,
+                                         MachineBasicBlock *BB) const;
+
   MachineBasicBlock *
   EmitInstrWithCustomInserter(MachineInstr &MI,
                               MachineBasicBlock *MBB) const override;
 
   bool getTgtMemIntrinsic(IntrinsicInfo &Info, const CallInst &I,
                           MachineFunction &MF,
                           unsigned Intrinsic) const override;
 
public:
   unsigned getNumInterleavedAccesses(VectorType *VecTy,
                                      const DataLayout &DL) const;
 
   MachineMemOperand::Flags getMMOFlags(const Instruction &I) const override;
 
   bool functionArgumentNeedsConsecutiveRegisters(Type *Ty,
                                                  CallingConv::ID CallConv,
                                                  bool isVarArg) const override;
+  /// Used for exception handling on Win64.
+  bool needsFixedCatchObjects() const override;
 private:
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   void addTypeForNEON(MVT VT, MVT PromotedBitwiseVT);

llvm/trunk/lib/Target/AArch64/AArch64ISelLowering.cpp

BuildMI(*EndBB, EndBB->begin(), DL, TII->get(AArch64::PHI), DestReg)
       .addMBB(TrueBB)
       .addReg(IfFalseReg)
       .addMBB(MBB);
 
   MI.eraseFromParent();
   return EndBB;
 }
 
+MachineBasicBlock *AArch64TargetLowering::EmitLoweredCatchRet(
+       MachineInstr &MI, MachineBasicBlock *BB) const {
+  assert(!isAsynchronousEHPersonality(classifyEHPersonality(
+             BB->getParent()->getFunction().getPersonalityFn())) &&
+         "SEH does not use catchret!");
+  return BB;
+}
+
+MachineBasicBlock *AArch64TargetLowering::EmitLoweredCatchPad(
+     MachineInstr &MI, MachineBasicBlock *BB) const {
+  MI.eraseFromParent();
+  return BB;
+}
+
 MachineBasicBlock *AArch64TargetLowering::EmitInstrWithCustomInserter(
     MachineInstr &MI, MachineBasicBlock *BB) const {
   switch (MI.getOpcode()) {
   default:
 #ifndef NDEBUG
     MI.dump();
 #endif
     llvm_unreachable("Unexpected instruction for custom inserter!");
 
   case AArch64::F128CSEL:
     return EmitF128CSEL(MI, BB);
 
   case TargetOpcode::STACKMAP:
   case TargetOpcode::PATCHPOINT:
     return emitPatchPoint(MI, BB);
+
+  case AArch64::CATCHRET:
+    return EmitLoweredCatchRet(MI, BB);
+  case AArch64::CATCHPAD:
+    return EmitLoweredCatchPad(MI, BB);
   }
 }
 
 //===----------------------------------------------------------------------===//
 // AArch64 Lowering private implementation.
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
AArch64TargetLowering::getVaListSizeInBits(const DataLayout &DL) const {
 
   return 3 * getPointerTy(DL).getSizeInBits() + 2 * 32;
 }
 
 void AArch64TargetLowering::finalizeLowering(MachineFunction &MF) const {
   MF.getFrameInfo().computeMaxCallFrameSize(MF);
   TargetLoweringBase::finalizeLowering(MF);
 }
+
+// Unlike X86, we let frame lowering assign offsets to all catch objects.
+bool AArch64TargetLowering::needsFixedCatchObjects() const {
+  return false;
+}

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.cpp

static cl::opt<unsigned> CBZDisplacementBits(
     "aarch64-cbz-offset-bits", cl::Hidden, cl::init(19),
     cl::desc("Restrict range of CB[N]Z instructions (DEBUG)"));
 
 static cl::opt<unsigned>
     BCCDisplacementBits("aarch64-bcc-offset-bits", cl::Hidden, cl::init(19),
                         cl::desc("Restrict range of Bcc instructions (DEBUG)"));
 
 AArch64InstrInfo::AArch64InstrInfo(const AArch64Subtarget &STI)
-    : AArch64GenInstrInfo(AArch64::ADJCALLSTACKDOWN, AArch64::ADJCALLSTACKUP),
+    : AArch64GenInstrInfo(AArch64::ADJCALLSTACKDOWN, AArch64::ADJCALLSTACKUP,
+                          AArch64::CATCHRET),
       RI(STI.getTargetTriple()), Subtarget(STI) {}
 
 /// GetInstSize - Return the number of bytes of code the specified
 /// instruction may be.  This returns the maximum number of bytes.
 unsigned AArch64InstrInfo::getInstSizeInBytes(const MachineInstr &MI) const {
   const MachineBasicBlock &MBB = *MI.getParent();
   const MachineFunction *MF = MBB.getParent();
   const MCAsmInfo *MAI = MF->getTarget().getMCAsmInfo();
bool AArch64InstrInfo::substituteCmpToZero(
   bool succeeded = UpdateOperandRegClass(*MI);
   (void)succeeded;
   assert(succeeded && "Some operands reg class are incompatible!");
   MI->addRegisterDefined(AArch64::NZCV, TRI);
   return true;
 }
 
 bool AArch64InstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
-  if (MI.getOpcode() != TargetOpcode::LOAD_STACK_GUARD)
+  if (MI.getOpcode() != TargetOpcode::LOAD_STACK_GUARD &&
+      MI.getOpcode() != AArch64::CATCHRET)
     return false;
 
   MachineBasicBlock &MBB = *MI.getParent();
   DebugLoc DL = MI.getDebugLoc();
+
+  if (MI.getOpcode() == AArch64::CATCHRET) {
+    // Skip to the first instruction before the epilog.
+    const TargetInstrInfo *TII =
+      MBB.getParent()->getSubtarget().getInstrInfo();
+    MachineBasicBlock *TargetMBB = MI.getOperand(0).getMBB();
+    auto MBBI = MachineBasicBlock::iterator(MI);
+    MachineBasicBlock::iterator FirstEpilogSEH = std::prev(MBBI);
+    while (FirstEpilogSEH->getFlag(MachineInstr::FrameDestroy) &&
+           FirstEpilogSEH != MBB.begin())
+      FirstEpilogSEH = std::prev(FirstEpilogSEH);
+    if (FirstEpilogSEH != MBB.begin())
+      FirstEpilogSEH = std::next(FirstEpilogSEH);
+    BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADRP))
+        .addReg(AArch64::X0, RegState::Define)
+        .addMBB(TargetMBB);
+    BuildMI(MBB, FirstEpilogSEH, DL, TII->get(AArch64::ADDXri))
+        .addReg(AArch64::X0, RegState::Define)
+        .addReg(AArch64::X0)
+        .addMBB(TargetMBB)
+        .addImm(0);
+    return true;
+  }
+
   unsigned Reg = MI.getOperand(0).getReg();
   const GlobalValue *GV =
       cast<GlobalValue>((*MI.memoperands_begin())->getValue());
   const TargetMachine &TM = MBB.getParent()->getTarget();
   unsigned char OpFlags = Subtarget.ClassifyGlobalReference(GV, TM);
   const unsigned char MO_NC = AArch64II::MO_NC;
 
   if ((OpFlags & AArch64II::MO_GOT) != 0) {

llvm/trunk/lib/Target/AArch64/AArch64InstrInfo.td

let isPseudo = 1 in {
   def SEH_SetFP : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_AddFP : Pseudo<(outs), (ins i32imm:$offs), []>, Sched<[]>;
   def SEH_Nop : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_PrologEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogStart : Pseudo<(outs), (ins), []>, Sched<[]>;
   def SEH_EpilogEnd : Pseudo<(outs), (ins), []>, Sched<[]>;
 }
 
+// Pseudo instructions for Windows EH
+//===----------------------------------------------------------------------===//
+let isTerminator = 1, hasSideEffects = 1, isBarrier = 1, hasCtrlDep = 1,
+    isCodeGenOnly = 1, isReturn = 1, isEHScopeReturn = 1, isPseudo = 1 in {
+   def CLEANUPRET : Pseudo<(outs), (ins), [(cleanupret)]>, Sched<[]>;
+   let usesCustomInserter = 1 in
+     def CATCHRET : Pseudo<(outs), (ins am_brcond:$dst, am_brcond:$src), [(catchret bb:$dst, bb:$src)]>,
+                    Sched<[]>;
+}
+
+let hasSideEffects = 1, hasCtrlDep = 1, isCodeGenOnly = 1,
+    usesCustomInserter = 1 in
+def CATCHPAD : Pseudo<(outs), (ins), [(catchpad)]>, Sched<[]>;
+
 //===----------------------------------------------------------------------===//
 // Floating point immediate move.
 //===----------------------------------------------------------------------===//
 
 let isReMaterializable = 1 in {
 defm FMOV : FPMoveImmediate<"fmov">;
 }
 

llvm/trunk/lib/Target/AArch64/AArch64MCInstLower.cpp

 void AArch64MCInstLower::Lower(const MachineInstr *MI, MCInst &OutMI) const {
   OutMI.setOpcode(MI->getOpcode());
 
   for (const MachineOperand &MO : MI->operands()) {
     MCOperand MCOp;
     if (lowerOperand(MO, MCOp))
       OutMI.addOperand(MCOp);
   }
+
+  switch (OutMI.getOpcode()) {
+  case AArch64::CATCHRET:
+    OutMI = MCInst();
+    OutMI.setOpcode(AArch64::RET);
+    OutMI.addOperand(MCOperand::createReg(AArch64::LR));
+    break;
+  case AArch64::CLEANUPRET:
+    OutMI = MCInst();
+    OutMI.setOpcode(AArch64::RET);
+    OutMI.addOperand(MCOperand::createReg(AArch64::LR));
+    break;
+  }
 }

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.h

public:
   const TargetRegisterClass *
   getSubClassWithSubReg(const TargetRegisterClass *RC,
                         unsigned Idx) const override;
 
   // Calls involved in thread-local variable lookup save more registers than
   // normal calls, so they need a different mask to represent this.
   const uint32_t *getTLSCallPreservedMask() const;
 
+  // Funclets on ARM64 Windows don't preserve any registers.
+  const uint32_t *getNoPreservedMask() const override;
+
   /// getThisReturnPreservedMask - Returns a call preserved mask specific to the
   /// case that 'returned' is on an i64 first argument if the calling convention
   /// is one that can (partially) model this attribute with a preserved mask
   /// (i.e. it is a calling convention that uses the same register for the first
   /// i64 argument and an i64 return value)
   ///
   /// Should return NULL in the case that the calling convention does not have
   /// this property

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.cpp

if (MF.getSubtarget<AArch64Subtarget>().isXRegCustomCalleeSaved(i)) {
         // register mask.
         UpdatedMask[*SubReg / 32] |= 1u << (*SubReg % 32);
       }
     }
   }
   *Mask = UpdatedMask;
 }
 
+const uint32_t *AArch64RegisterInfo::getNoPreservedMask() const {
+  return CSR_AArch64_NoRegs_RegMask;
+}
+
 const uint32_t *
 AArch64RegisterInfo::getThisReturnPreservedMask(const MachineFunction &MF,
                                                 CallingConv::ID CC) const {
   // This should return a register mask that is the same as that returned by
   // getCallPreservedMask but that additionally preserves the register used for
   // the first i64 argument (which must also be the register used to return a
   // single i64 return value)
   //
AArch64RegisterInfo::getCrossCopyRegClass(const TargetRegisterClass *RC) const {
   return RC;
 }
 
 unsigned AArch64RegisterInfo::getBaseRegister() const { return AArch64::X19; }
 
 bool AArch64RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
 
-  // In the presence of variable sized objects, if the fixed stack size is
-  // large enough that referencing from the FP won't result in things being
-  // in range relatively often, we can use a base pointer to allow access
+  // In the presence of variable sized objects or funclets, if the fixed stack
+  // size is large enough that referencing from the FP won't result in things
+  // being in range relatively often, we can use a base pointer to allow access
   // from the other direction like the SP normally works.
+  //
   // Furthermore, if both variable sized objects are present, and the
   // stack needs to be dynamically re-aligned, the base pointer is the only
   // reliable way to reference the locals.
-  if (MFI.hasVarSizedObjects()) {
+  if (MFI.hasVarSizedObjects() || MF.hasEHFunclets()) {
     if (needsStackRealignment(MF))
       return true;
     // Conservatively estimate whether the negative offset from the frame
     // pointer will be sufficient to reach. If a function has a smallish
     // frame, it's less likely to have lots of spills and callee saved
     // space, so it's all more likely to be within range of the frame pointer.
     // If it's wrong, we'll materialize the constant and still get to the
     // object; it's just suboptimal. Negative offsets use the unscaled

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch-nobase.ll

+; RUN: llc -o - %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s
+
+; Make sure we don't have a base pointer.
+; CHECK-LABEL: "?a@@YAXXZ":
+; CHECK-NOT: x19
+
+; Check that we compute the address relative to fp.
+; CHECK-LABEL: "?catch$2@?0??a@@YAXXZ@4HA":
+; CHECK:             stp     x29, x30, [sp, #-16]!   ; 16-byte Folded Spill
+; CHECK-NEXT:        sub     x0, x29, #16            ; =16
+; CHECK-NEXT:        mov     x1, xzr
+; CHECK-NEXT:        bl      "?bb@@YAXPEAHH@Z"
+; CHECK-NEXT:        adrp    x0, .LBB0_1
+; CHECK-NEXT:        add     x0, x0, .LBB0_1
+; CHECK-NEXT:        ldp     x29, x30, [sp], #16     ; 16-byte Folded Reload
+; CHECK-NEXT:        ret
+
+target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-windows-msvc19.11.0"
+
+define dso_local void @"?a@@YAXXZ"(i64 %p1) personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %a = alloca i32, align 16
+  %0 = bitcast i32* %a to i8*  
+  store i32 305419896, i32* %a, align 16
+  invoke void @"?bb@@YAXPEAHH@Z"(i32* nonnull %a, i32* null)
+          to label %try.cont unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %entry
+  %1 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %2 = catchpad within %1 [i8* null, i32 64, i8* null]
+  call void @"?bb@@YAXPEAHH@Z"(i32* nonnull %a, i32* null) [ "funclet"(token %2) ]
+  catchret from %2 to label %try.cont
+
+try.cont:                                         ; preds = %entry, %catch
+  call void @"?cc@@YAXXZ"()
+  ret void
+}
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1)
+
+declare dso_local void @"?bb@@YAXPEAHH@Z"(i32*, i32*)
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+
+declare dso_local void @"?cc@@YAXXZ"()
+

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch-realign.ll

+; RUN: llc -o - %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s
+
+; Make sure we have a base pointer.
+; CHECK-LABEL: "?a@@YAXXZ":
+; CHECK: and     sp, x9, #0xffffffffffffffc0
+; CHECK: mov     x19, sp
+
+; Make sure the funclet prologue/epilogue are correct: specifically,
+; it shouldn't access the parent's frame via sp, and the prologue and
+; epilogue should be symmetrical.
+; CHECK-LABEL: "?catch$2@?0??a@@YAXXZ@4HA":
+; CHECK:      str     x28, [sp, #-32]!
+; CHECK-NEXT: str     x19, [sp, #8]
+; CHECK-NEXT: stp     x29, x30, [sp, #16]
+; CHECK-NEXT: add     x0, x19, #64
+; CHECK-NEXT: mov     w1, wzr
+; CHECK-NEXT: bl      "?bb@@YAXPEAHH@Z"
+; CHECK-NEXT: adrp    x0, .LBB0_1
+; CHECK-NEXT: add     x0, x0, .LBB0_1
+; CHECK-NEXT: ldp     x29, x30, [sp, #16]
+; CHECK-NEXT: ldr     x19, [sp, #8]
+; CHECK-NEXT: ldr     x28, [sp], #32
+; CHECK-NEXT: ret
+
+
+target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-windows-msvc19.11.0"
+
+define dso_local void @"?a@@YAXXZ"() personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %a = alloca [100 x i32], align 64
+  %0 = bitcast [100 x i32]* %a to i8*  
+  call void @llvm.memset.p0i8.i64(i8* nonnull align 64 %0, i8 0, i64 400, i1 false)
+  %1 = getelementptr inbounds [100 x i32], [100 x i32]* %a, i64 0, i64 0
+  store i32 305419896, i32* %1, align 64
+  invoke void @"?bb@@YAXPEAHH@Z"(i32* nonnull %1, i32 1)
+          to label %try.cont unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %entry
+  %2 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %3 = catchpad within %2 [i8* null, i32 64, i8* null]
+  call void @"?bb@@YAXPEAHH@Z"(i32* nonnull %1, i32 0) [ "funclet"(token %3) ]
+  catchret from %3 to label %try.cont
+
+try.cont:                                         ; preds = %entry, %catch
+  call void @"?cc@@YAXXZ"()
+  ret void
+}
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1)
+
+declare dso_local void @"?bb@@YAXPEAHH@Z"(i32*, i32)
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+
+declare dso_local void @"?cc@@YAXXZ"()
+

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch-vla.ll

+; RUN: llc -o - %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s
+
+; Check we don't have a base pointer.
+; CHECK-LABEL: "?a@@YAXXZ":
+; CHECK-NOT: x19
+
+; Make sure the prologue and epilogue are sane. Make sure the
+; frame index is relative to the FP, since there is no base pointer.
+; (Funclets aren't allowed to contain dynamic allocas.)
+; CHECK-LABEL: "?catch$2@?0??a@@YAXXZ@4HA":
+; CHECK:      stp     x29, x30, [sp, #-16]!
+; CHECK-NEXT: ldur    x0, [x29, #-8]
+; CHECK-NEXT: mov     x1, x0
+; CHECK-NEXT: bl      "?bb@@YAXPEAHH@Z"
+; CHECK-NEXT: adrp    x0, .LBB0_1
+; CHECK-NEXT: add     x0, x0, .LBB0_1
+; CHECK-NEXT: ldp     x29, x30, [sp], #16
+; CHECK-NEXT: ret
+
+target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-windows-msvc19.11.0"
+
+define dso_local void @"?a@@YAXXZ"(i64 %p1) personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %a = alloca i32, i64 %p1, align 16
+  %0 = bitcast i32* %a to i8*  
+  call void @llvm.memset.p0i8.i64(i8* nonnull align 16 %0, i8 0, i64 400, i1 false)
+  store i32 305419896, i32* %a, align 16
+  invoke void @"?bb@@YAXPEAHH@Z"(i32* nonnull %a, i32* null)
+          to label %try.cont unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %entry
+  %1 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %2 = catchpad within %1 [i8* null, i32 64, i8* null]
+  call void @"?bb@@YAXPEAHH@Z"(i32* nonnull %a, i32* %a) [ "funclet"(token %2) ]
+  catchret from %2 to label %try.cont
+
+try.cont:                                         ; preds = %entry, %catch
+  call void @"?cc@@YAXXZ"()
+  ret void
+}
+
+declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1)
+
+declare dso_local void @"?bb@@YAXPEAHH@Z"(i32*, i32*)
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+
+declare dso_local void @"?cc@@YAXXZ"()
+

llvm/trunk/test/CodeGen/AArch64/wineh-try-catch.ll

+; RUN: llc -o - %s -mtriple=aarch64-windows -verify-machineinstrs | FileCheck %s
+; RUN: llc -o %t -filetype=obj %s -mtriple=aarch64-windows
+; RUN: llvm-readobj -unwind %t | FileCheck %s -check-prefix=UNWIND
+
+; We test the following
+; 1) That the unwind help object is created and that its offset from the stack
+;    pointer on entry is patched into the table fed to __CxxFrameHandler3
+; 2) That the stack update for the catch funclet only includes the callee saved
+;    registers
+; 3) That the locals are accessed using the frame pointer in both the funclet
+;    and the parent function.
+
+; The following checks that the unwind help object has -2 stored into it at
+; fp - 400 - 256 = fp - 656, which is on-entry sp - 48 + 32 - 656 =
+; on-entry sp - 672.  We check this offset in the table later on.
+
+; CHECK-LABEL: "?func@@YAHXZ":
+; CHECK:       str     x28, [sp, #-48]!
+; CHECK:       str     x21, [sp, #8]
+; CHECK:       stp     x19, x20, [sp, #16]
+; CHECK:       stp     x29, x30, [sp, #32]
+; CHECK:       add     x29, sp, #32
+; CHECK:       sub     sp, sp, #624
+; CHECK:       mov     x19, sp
+; CHECK:       orr     x1, xzr, #0xfffffffffffffffe
+; CHECK:       stur    x1, [x19]
+
+; Now check that x is stored at fp - 20.  We check that this is the same
+; location accessed from the funclet to retrieve x.
+; CHECK:       orr     w8, wzr, #0x1
+; CHECK:       stur    w8, [x29, [[X_OFFSET:#-[1-9][0-9]+]]
+
+; Check the offset off the frame pointer at which B is located.
+; Check the same offset is used to pass the address of B to init2 in the
+; funclet.
+; CHECK:       sub     x0, x29, [[B_OFFSET:#[1-9][0-9]+]]
+; CHECK:       bl      "?init@@YAXPEAH@Z"
+
+; This is the label for the throw that is encoded in the ip2state.
+; We are inside the try block, where we make a call to func2
+; CHECK-LABEL: .Ltmp0:
+; CHECK:       bl      "?func2@@YAHXZ
+
+; CHECK:        [[CATCHRETDEST:.LBB0_[0-9]+]]:      ; %catchret.dest
+
+; Check the catch funclet.
+; CHECK-LABEL: "?catch$2@?0??func@@YAHXZ@4HA":
+
+; Check that the stack space is allocated only for the callee saved registers.
+; CHECK:       str     x28, [sp, #-48]!
+; CHECK:       str     x21, [sp, #8]
+; CHECK:       stp     x19, x20, [sp, #16]
+; CHECK:       stp     x29, x30, [sp, #32]
+; CHECK:       add     x20, x19, #12
+
+; Check that there are no further stack updates.
+; CHECK-NOT:   sub     sp, sp
+
+; Check that the stack address passed to init2 is off the frame pointer, and
+; that it matches the address of B in the parent function.
+; CHECK:       sub     x0, x29, [[B_OFFSET]]
+; CHECK:       bl      "?init2@@YAXPEAH@Z"
+
+; Check that are storing x back to the same location off the frame pointer as in
+; the parent function.
+; CHECK:       stur    w8, [x29, [[X_OFFSET]]]
+
+; Check that the funclet branches back to the catchret destination
+; CHECK:       adrp    x0, .LBB0_3
+; CHECK-NEXT:  add     x0, x0, [[CATCHRETDEST]]
+
+
+; Now check that the offset of the unwind help object from the stack pointer on
+; entry to func is encoded in cppxdata that is passed to __CxxFrameHandler3.  As
+; computed above, this comes to -672.
+; CHECK-LABEL:        "$cppxdata$?func@@YAHXZ":
+; CHECK-NEXT:         .word   429065506               ; MagicNumber
+; CHECK-NEXT:         .word   2                       ; MaxState
+; CHECK-NEXT:         .word   ("$stateUnwindMap$?func@@YAHXZ")@IMGREL ; UnwindMap
+; CHECK-NEXT:         .word   1                       ; NumTryBlocks
+; CHECK-NEXT:         .word   ("$tryMap$?func@@YAHXZ")@IMGREL ; TryBlockMap
+; CHECK-NEXT:         .word   4                       ; IPMapEntries
+; CHECK-NEXT:         .word   ("$ip2state$?func@@YAHXZ")@IMGREL ; IPToStateXData
+; CHECK-NEXT:         .word   -672                    ; UnwindHelp
+
+; UNWIND: Function: ?func@@YAHXZ (0x0)
+; UNWIND: Prologue [
+; UNWIND-NEXT: ; nop
+; UNWIND-NEXT: ; sub sp, #624
+; UNWIND-NEXT: ; add fp, sp, #32
+; UNWIND-NEXT: ; stp x29, x30, [sp, #32]
+; UNWIND-NEXT: ; stp x19, x20, [sp, #16]
+; UNWIND-NEXT: ; str x21, [sp, #8]
+; UNWIND-NEXT: ; str x28, [sp, #48]!
+; UNWIND-NEXT: ; end
+; UNWIND: Function: ?catch$2@?0??func@@YAHXZ@4HA
+; UNWIND: Prologue [
+; UNWIND-NEXT: ; stp x29, x30, [sp, #32]
+; UNWIND-NEXT: ; stp x19, x20, [sp, #16]
+; UNWIND-NEXT: ; str x21, [sp, #8]
+; UNWIND-NEXT: ; str x28, [sp, #48]!
+; UNWIND-NEXT: ; end
+
+target datalayout = "e-m:w-p:64:64-i32:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-windows-msvc19.11.0"
+
+%rtti.TypeDescriptor2 = type { i8**, i8*, [3 x i8] }
+%eh.CatchableType = type { i32, i32, i32, i32, i32, i32, i32 }
+%eh.CatchableTypeArray.1 = type { i32, [1 x i32] }
+%eh.ThrowInfo = type { i32, i32, i32, i32 }
+
+$"??_R0H@8" = comdat any
+
+$"_CT??_R0H@84" = comdat any
+
+$_CTA1H = comdat any
+
+$_TI1H = comdat any
+
+@"??_7type_info@@6B@" = external constant i8*
+@"??_R0H@8" = linkonce_odr global %rtti.TypeDescriptor2 { i8** @"??_7type_info@@6B@", i8* null, [3 x i8] c".H\00" }, comdat
+@__ImageBase = external dso_local constant i8
+@"_CT??_R0H@84" = linkonce_odr unnamed_addr constant %eh.CatchableType { i32 1, i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%rtti.TypeDescriptor2* @"??_R0H@8" to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32), i32 0, i32 -1, i32 0, i32 4, i32 0 }, section ".xdata", comdat
+@_CTA1H = linkonce_odr unnamed_addr constant %eh.CatchableTypeArray.1 { i32 1, [1 x i32] [i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%eh.CatchableType* @"_CT??_R0H@84" to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32)] }, section ".xdata", comdat
+@_TI1H = linkonce_odr unnamed_addr constant %eh.ThrowInfo { i32 0, i32 0, i32 0, i32 trunc (i64 sub nuw nsw (i64 ptrtoint (%eh.CatchableTypeArray.1* @_CTA1H to i64), i64 ptrtoint (i8* @__ImageBase to i64)) to i32) }, section ".xdata", comdat
+
+; Function Attrs: noinline optnone
+define dso_local i32 @"?func@@YAHXZ"() #0 personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %B = alloca [50 x i32], align 4
+  %x = alloca i32, align 4
+  %tmp = alloca i32, align 4
+  %i = alloca i32, align 4
+  %C = alloca [100 x i32], align 4
+  store i32 1, i32* %x, align 4
+  %arraydecay = getelementptr inbounds [50 x i32], [50 x i32]* %B, i32 0, i32 0
+  call void @"?init@@YAXPEAH@Z"(i32* %arraydecay)
+  %call = invoke i32 @"?func2@@YAHXZ"()
+          to label %invoke.cont unwind label %catch.dispatch
+
+invoke.cont:                                      ; preds = %entry
+  store i32 %call, i32* %tmp, align 4
+  %0 = bitcast i32* %tmp to i8*
+  invoke void @_CxxThrowException(i8* %0, %eh.ThrowInfo* @_TI1H) #2
+          to label %unreachable unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %invoke.cont, %entry
+  %1 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %2 = catchpad within %1 [%rtti.TypeDescriptor2* @"??_R0H@8", i32 0, i32* %i]
+  %arraydecay1 = getelementptr inbounds [100 x i32], [100 x i32]* %C, i32 0, i32 0
+  call void @"?init@@YAXPEAH@Z"(i32* %arraydecay1) [ "funclet"(token %2) ]
+  %arraydecay2 = getelementptr inbounds [50 x i32], [50 x i32]* %B, i32 0, i32 0
+  call void @"?init2@@YAXPEAH@Z"(i32* %arraydecay2) [ "funclet"(token %2) ]
+  %3 = load i32, i32* %i, align 4
+  %idxprom = sext i32 %3 to i64
+  %arrayidx = getelementptr inbounds [50 x i32], [50 x i32]* %B, i64 0, i64 %idxprom
+  %4 = load i32, i32* %arrayidx, align 4
+  %5 = load i32, i32* %i, align 4
+  %idxprom3 = sext i32 %5 to i64
+  %arrayidx4 = getelementptr inbounds [100 x i32], [100 x i32]* %C, i64 0, i64 %idxprom3
+  %6 = load i32, i32* %arrayidx4, align 4
+  %add = add nsw i32 %4, %6
+  %7 = load i32, i32* %i, align 4
+  %8 = load i32, i32* %i, align 4
+  %mul = mul nsw i32 %7, %8
+  %add5 = add nsw i32 %add, %mul
+  store i32 %add5, i32* %x, align 4
+  catchret from %2 to label %catchret.dest
+
+catchret.dest:                                    ; preds = %catch
+  br label %try.cont
+
+try.cont:                                         ; preds = %catchret.dest
+  %arrayidx6 = getelementptr inbounds [50 x i32], [50 x i32]* %B, i64 0, i64 2
+  %9 = load i32, i32* %arrayidx6, align 4
+  %10 = load i32, i32* %x, align 4
+  %add7 = add nsw i32 %9, %10
+  ret i32 %add7
+
+unreachable:                                      ; preds = %invoke.cont
+  unreachable
+}
+
+declare dso_local void @"?init@@YAXPEAH@Z"(i32*)
+
+declare dso_local i32 @"?func2@@YAHXZ"()
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+
+declare dllimport void @_CxxThrowException(i8*, %eh.ThrowInfo*)
+
+declare dso_local void @"?init2@@YAXPEAH@Z"(i32*)
+
+attributes #0 = { noinline optnone }
+attributes #2 = { noreturn }