[COFF, ARM64] Implement support for SEH extensions __try/__except/__finally

[mgrang]

Summary

This patch supports MS SEH extensions try/except/__finally. The intrinsics localescape and localrecover are responsible for communicating escaped static allocas from the try block to the handler.

We need to preserve frame pointers for SEH. So we create a new function/property HasLocalEscape.

[rnk]

lib/Target/AArch64/AArch64ISelLowering.cpp

2721

It's not EBP for AArch64, surely. I would generalize the text to "FP" instead.

2728–2729

Yeah, this was a great hack. :)

[mstorsjo]

I think you got the titles swapped on this and D53541; this one adds the aarch64_seh_recoverfp intrinsic while D53541 uses it to support __try/__except.

[mgrang]

In D53540#1272057, @mstorsjo wrote:

I think you got the titles swapped on this and D53541; this one adds the aarch64_seh_recoverfp intrinsic while D53541 uses it to support __try/__except.

D53541 generates aarch64_seh_recoverfp while this patch lowers it. This patch also lowers the localescape/localrecover intrinsics. I have updated the title summary accordingly.

[mgrang]

lib/Target/AArch64/AArch64AsmPrinter.cpp

702

I see that X86 generates LEA to calculate the address of the .L label. AArch64 does not have a similar instruction. I tried to use ADRP + ADD but that requires either a GlobalSymbol or an ExternalSymbol type. However, the .L label is of type MCSymbol.

So I generate a MOVK x1, #.Lfoo. The problem is that MOVK takes an unsigned immediate. The .L label here represents offset from FP. This works if the offset is positive. But for negative offsets we should generate a MOVN. However, I am not sure how to detect negative immediate without first generating a MOV.

Any suggestions?

lib/Target/AArch64/AArch64ISelLowering.cpp

2728–2729

I am not really familiar with this piece of code. Could you please comment on whether this should work for AArch64?

lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

219 ↗

(On Diff #170698)

As I commented above, we assume a non-negative immediate.

[rnk]

lib/Target/AArch64/AArch64AsmPrinter.cpp

702

I would expect you would want a generic MOVMCSym instruction to be able to materialize any label, not just ones that are known during assembly. This might get selected if we needed to materialize, for example, the address of the LSDA, which is also a private label.

lib/Target/AArch64/AArch64ISelLowering.cpp

2726

I think this is suppoosed to be ".seh_setframe offset"

2728–2729

I think it should. Fundamentally, we are trying to transfer information about one function's frame layout into an independent filter function. The idea is that we can write assembly like this:

foo:
.Lfoo = 0x48
  ...

foo$filt:
  mov r0, .Lfoo

Hm, if possible, it might be nice to codegen this whole thing after SD ISel. I wonder if there is some way to pass it through as an intrinsic and lower it later. That will avoid the whole concern of having a generic MCSymbol materialization instruction selection pattern.

[ssijaric]

We need to have a test case that checks for access to local variables from the finally and except funclets. The parent's frame pointer is the second argument to both the filter functions and termination handlers, fro what I see. We need to make sure that locals variables in funclets are being accessed off x1, and not the stack pointer.

[mgrang]

Will push more comprehensive tests for ARM64 SEH soon.

[majnemer]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

Since you already did an isa, this can be cast<MCConstantExpr>(Expr).

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

I tried applying this patch (plus D53541, D51524, and D54248) and building Firefox, and I got a crash on null Expr here.

Let me know if you're interested in the repro, or if it's too early for me to be playing with this code, in which case sorry for the noise.

[mgrang]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

@dmajor Thanks for testing this out :) If you have a reduced test case to repro the failure that would be great. Although currently I am busy with an internal release I expect to get back to this in Dec.

[mgrang]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

@dmajor Also what patches (if any) to the Firefox code base are needed to build Firefox for arm64-windows. Are those patches publicly available?

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

I don't really recommend it right now, it's a bit of a hassle because you have to apply some hacks to the build system. We're working on making it easier (hopefully within a week or two). In the meantime I can try to get you a repro with creduce.

[TomTan]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

@dmajor, I hit the same null reference issue after cherry-pick changes you listed. Seems FrameAllocSym is not a set to SymContentsVariable here which is expected to happen in AArch64RegisterInfo::eliminateFrameIndex.

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6219

Here's the repro of the crash from creduce:

class a {
  void b();
  void c();
};
void a::b() {
  int f;
  __try {
    c();
  } __except (f) {
  }
}

[rnk]

Sorry, I didn't realize this was waiting on my review. It looks like this is mostly about localescape, so let's change the patch description and code to be about that.

include/llvm/CodeGen/MachineFunction.h

332

IMO this should really live on the AArch64-specific function info, since it's not general to x64/x86 yet.

Also, this isn't SEH specific, this is set to true only by localescape. Maybe you should rename it? localescape was intended to be orthogonal to SEH, potentially usable for non-externally visible lambdas, although the codegen right now is particularly bad.

[mgrang]

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

[TomTan]

In D53540#1326343, @mgrang wrote:

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

@mgrang, the SEH fix still causes ARM64 COFF file crash when linking with CFG enabled (/guard:cf). Seems functions are not marked as IMAGE_SYM_DTYPE_FUNCTION in ARM64 COFF which was done in both x86 and arm32 COFF (see below 2 links).

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/X86/X86AsmPrinter.cpp#L67

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/ARM/ARMAsmPrinter.cpp#L148

[mgrang]

In D53540#1326353, @TomTan wrote:

In D53540#1326343, @mgrang wrote:

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

@mgrang, the SEH fix still causes ARM64 COFF file crash when linking with CFG enabled (/guard:cf). Seems functions are not marked as IMAGE_SYM_DTYPE_FUNCTION in ARM64 COFF which was done in both x86 and arm32 COFF (see below 2 links).

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/X86/X86AsmPrinter.cpp#L67

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/ARM/ARMAsmPrinter.cpp#L148

Thanks @TomTan. I have a fix here: https://reviews.llvm.org/D55535

[rnk]

CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6166 ↗

(On Diff #177612)

It still seems true that this isn't about seh funclets, it's about localescape / localrecover, which is supposed to be orthogonal (for use in non-ABI visible lambdas), so I'd recommend renaming this boolean.

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

I think this assertion will fail if you use localrecover before localescape. I would make a test case for that. Is there really not some pseudo instruction we can use so that the materialization can be relaxed by the assembler?

Otherwise, there's no point to using this MCSymbol hack. We might as well have a simpler side table if we retain the requirement that localescape be code generated first.

[efriedma]

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

Is there really not some pseudo instruction we can use

The AArch64 assembler currently doesn't have any useful support for this sort of construct. Maybe it could be extended. (Is the value an arbitrary 64-bit integer, or is there some restriction on the possible values?)

[rnk]

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

These values are offsets from the FP (or SP) to a stack object, so they are typically small. If there are implementation limits on the size of an aarch64 stack frame, we can definitely use that to guide isel here.

[mgrang]

include/llvm/CodeGen/MachineFunction.h

332

I tried to move this to AArch64MachineFunctionInfo. I set it in AArch64RegisterInfo when I encounter LOCAL_ESCAPE and need to access it in AArch64FrameLowering. But it seems setting it in AArch64RegisterInfo is too late. So for now I have kept it on the MF.

lib/Target/AArch64/AArch64AsmPrinter.cpp

697

@rnk Yes, you are correct that the assert here will fail is the localrecover comes before localescape.
I have tried to model this similar to X86 but the problem is AArch64 cannot do an ADRP on an MCSymbol. So I wasn't sure how to materialize this in the assembler.
Could you give me some idea on how this can be done?

[rnk]

include/llvm/CodeGen/MachineFunction.h

332

I see, we could really only do that if localescape was lowered in target dependent code. I think this is fine the way it is now. It's clearer (to me, at least) after the rename.

lib/CodeGen/AsmPrinter/WinException.cpp

548–559

This can just be isAArch64, we already assigned that to a member boolean.

lib/Target/AArch64/AArch64AsmPrinter.cpp

697

I guess I've been assuming that you could assemble something like this in a .s file for aarch64, like you can for x86:

main:
  movl $.Lasdf, %eax
  ret
.Lasdf = 1234

I would assume that the aarch64 equivalent would be:

main:
  mov d0, #.Lasdf
  ret
.Lasdf = 1234

But that doesn't assemble. Does gnu as support that? I don't have an aarch64 binutils build. Maybe llvm-mc should allow mov/movk to accept a symbolic label operand.

I don't think we would want to use ADRP, IIRC that's for materializing some PC-relative address, but obviously this isn't that kind of symbol.

[mgrang]

lib/Target/AArch64/AArch64AsmPrinter.cpp

697

GNU does not seem to support it as well.

Error: operand 2 should be a SIMD vector element -- `mov d0,#.Lasdf'

[efriedma]

lib/Target/AArch64/AArch64AsmPrinter.cpp

697

Does gnu as support that?

"mov" is pseudo-instruction, but it will expand to at most one instruction (a movz, a movn, or an orr); this is defined in the AArch64 architecture reference. There is no variable-length pseudo-instruction; if you need a larger value, you have to write out a movz/movk sequence explicitly.

I guess the stack size is currently limited to 256MB on Windows because of the way we emit the prologue on Windows (that's the limit for the alloc_l unwind opcode). We could maybe teach LLVM to use multiple alloc_l operations, but it's probably reasonable to assume the stack for a function is less than 4GB, so we could emit just two instructions (movz+movk).

For value less than 2^32, gas supports something like the following:

movz x0, #:abs_g1:.Lasdf
movk x0, #:abs_g0_nc:.Lasdf
.Lasdf = 0x12345678

If you try this with LLVM, you currently get an error "no resolvable MOVZ/MOVK fixups supported yet", but I think that's straightforward to fix. (LLVM supports the syntax in general, just not the case where the value is defined in the same object file.)

[rnk]

lib/Target/AArch64/AArch64AsmPrinter.cpp

697

I saw your fixup patches, and this seems like the way to go. At least, this was the most principled way I could come up with to pass information about the stack frame layout of one machine function to another. I didn't want to use a side table and lock in the code generation order dependence.

[mgrang]

Addressed comments. The MCSymbol for the frame offset is now lowered in the assembler. D56037 needs to merge first.
I am now working on test cases. Will push them soon.

[efriedma]

lib/Target/AArch64/AArch64AsmPrinter.cpp

713

I think this "createImm" is wrong; should be 16. (It probably doesn't show up in the printed assembly, but I think it affects the encoding.)

lib/Target/AArch64/AArch64RegisterInfo.cpp

271

This can't be correct in general... in cases involving stack realignment, it's impossible to generate code without a base pointer.

[mgrang]

Added test cases for localescape and try-finally. Will add try-except test case soon.

[efriedma]

lib/Target/AArch64/AArch64ISelLowering.cpp

2756

Are you sure this does the right thing in general? The link between the choice of register here, and the register used for the getFrameIndexReference call in WinException::emitEHRegistrationOffsetLabel , seems fragile at best.

[rnk]

Thanks for pushing the MC work to support this forward!

@efriedma has one outstanding comment, but I don't see how to untangle the choice of frame register from AArch64FrameLowering::getIndexFrameReference, or the x86 version of the same, so I'm happy with this. Please wait for his response, though.

lib/Target/AArch64/AArch64ISelLowering.cpp

2756

It is fragile, but it's what was done for x86. Unfortunately, it's actually really hard to know which physical register is going to be used to address locals during ISel, so I just did this brittle thing and moved on. See these comments from AArch64FrameLowering::hasFP for some of the phase ordering problems:

// With large callframes around we may need to use FP to access the scavenging
// emergency spillslot.
//
// Unfortunately some calls to hasFP() like machine verifier ->
// getReservedReg() -> hasFP in the middle of global isel are too early
// to know the max call frame size. Hopefully conservatively returning "true"
// in those cases is fine.
// DefaultSafeSPDisplacement is fine as we only emergency spill GP regs.
if (!MFI.isMaxCallFrameSizeComputed() ||
    MFI.getMaxCallFrameSize() > DefaultSafeSPDisplacement)
  return true;

I think we are mostly safe because when using EH funclets, an FP is always needed, so it's just a decision between "base" and "frame" pointer registers. That is currently entirely determined by the existence of user allocas with alignments above the standard ABI stack alignment.

However, you can imagine that some later pass might try to create more highly aligned stack objects, which would require a base pointer. For example, the register allocator might spill vector registers that require more stack alignment than the ABI provides. Today, for x86 at least, LLVM punts on this problem, and simply uses unaligned instructions for the spill and reload. I don't know how to get into the analogous situation for AArch64 or what LLVM would do, though.

[smeenai]

lib/Target/AArch64/AArch64ISelLowering.cpp

2756

Is this at all related to https://bugs.llvm.org/show_bug.cgi?id=37169?

[rnk]

lib/Target/AArch64/AArch64ISelLowering.cpp

2756

I had forgotten about that issue, but that's exactly the case I was describing. I proposed this solution there:

I think localaddress should really lower down to a LEA of a zero-sized stack object. It's really trying to say, give me the address of the local variable area. The localrecover label assignments would instead compute the difference between this particular zero-sized stack object and their stack object.

For AArch64, LEA would really be ADC, or whatever is generated to materialize the address of an alloca. Does something like that make sense?

It occurs to me that this localescape system is pretty flawed, since not all local variables are static allocas. This source crashes clang today:

struct Foo {
  int x, y, z;
};
int filter(int);
void may_throw();
void seh_byval(Foo o) {
  __try {
    may_throw();
  } __except(filter(o.x)) {
  }
}

Oops. =/

[efriedma]

If you're okay with trying to resolve any remaining bugs as a followup, that's fine with me; LGTM.

lib/Target/AArch64/AArch64ISelLowering.cpp

2756

That is currently entirely determined by the existence of user allocas with alignments above the standard ABI stack alignment.

This is not true on AArch64. There's a second condition: whether the size of the local frame is greater than 256 bytes. This condition isn't strictly required, but avoids materializing offsets in some situations. (AArch64 addressing modes support very limited immediate negative offsets.)

That doesn't really have any impact on its own, but AArch64FrameLowering::getFrameIndexReference actually chooses between FP-relative and BP-relative addressing based on the offsets, in general. So we probably need to do something here, to force the frame index to be resolved the same way as localaddress.

I don't think we can hit an issue similar to bug 37169 on AArch64; the natural stack alignment is 16, and there currently isn't anything that requires higher alignment.

It occurs to me that this localescape system is pretty flawed, since not all local variables are static allocas.

Yes, we need special handling for byval/inalloca arguments at the IR level; in general, it's impossible to compute an offset to an argument relative to the base pointer.

[mgrang]

Added a default lowering for llvm.eh.recoverfp intrinsic.

[efriedma]

LGTM

[mgrang]

@rnk @efriedma Thanks a lot for all your valuable review comments. I'll commit this now :)

include/llvm/CodeGen/MachineFunction.h

class MachineFunction {
 
   /// CodeView label annotations.
   std::vector<std::pair<MCSymbol *, MDNode *>> CodeViewAnnotations;
 
   bool CallsEHReturn = false;
   bool CallsUnwindInit = false;
   bool HasEHScopes = false;
   bool HasEHFunclets = false;
+  bool HasLocalEscape = false;
 
   /// List of C++ TypeInfo used.
   std::vector<const GlobalValue *> TypeInfos;
 
   /// List of typeids encoding filters used.
   std::vector<unsigned> FilterIds;
 
   /// List of the indices in FilterIds corresponding to filter terminators.
public:
   void setCallsUnwindInit(bool b) { CallsUnwindInit = b; }
 
   bool hasEHScopes() const { return HasEHScopes; }
   void setHasEHScopes(bool V) { HasEHScopes = V; }
 
   bool hasEHFunclets() const { return HasEHFunclets; }
   void setHasEHFunclets(bool V) { HasEHFunclets = V; }
 
+  bool hasLocalEscape() const { return HasLocalEscape; }
+  void setHasLocalEscape(bool V) { HasLocalEscape = V; }
+
   /// Find or create an LandingPadInfo for the specified MachineBasicBlock.
   LandingPadInfo &getOrCreateLandingPadInfo(MachineBasicBlock *LandingPad);
 
   /// Remap landing pad labels and remove any deleted landing pads.
   void tidyLandingPads(DenseMap<MCSymbol *, uintptr_t> *LPMap = nullptr,
                        bool TidyIfNoBeginLabels = true);
 
   /// Return a reference to the landing pad info for the current function.

lib/CodeGen/AsmPrinter/WinException.cpp

void WinException::emitCSpecificHandlerTable(const MachineFunction *MF) {
   const WinEHFuncInfo &FuncInfo = *MF->getWinEHFuncInfo();
 
   bool VerboseAsm = OS.isVerboseAsm();
   auto AddComment = [&](const Twine &Comment) {
     if (VerboseAsm)
       OS.AddComment(Comment);
   };
 
+  if (!isAArch64) {
     // Emit a label assignment with the SEH frame offset so we can use it for
     // llvm.eh.recoverfp.
     StringRef FLinkageName =
         GlobalValue::dropLLVMManglingEscape(MF->getFunction().getName());
     MCSymbol *ParentFrameOffset =
         Ctx.getOrCreateParentFrameOffsetSymbol(FLinkageName);
     const MCExpr *MCOffset =
         MCConstantExpr::create(FuncInfo.SEHSetFrameOffset, Ctx);
     Asm->OutStreamer->EmitAssignment(ParentFrameOffset, MCOffset);
+  }
 
   // Use the assembler to compute the number of table entries through label
   // difference and division.
   MCSymbol *TableBegin =
       Ctx.createTempSymbol("lsda_begin", /*AlwaysAddSuffix=*/true);
   MCSymbol *TableEnd =
       Ctx.createTempSymbol("lsda_end", /*AlwaysAddSuffix=*/true);
   const MCExpr *LabelDiff = getOffset(TableEnd, TableBegin);
   const MCExpr *EntrySize = MCConstantExpr::create(16, Ctx);

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

for (unsigned Idx = 0, E = I.getNumArgOperands(); Idx < E; ++Idx) {
           MF.getMMI().getContext().getOrCreateFrameAllocSymbol(
               GlobalValue::dropLLVMManglingEscape(MF.getName()), Idx);
       BuildMI(*FuncInfo.MBB, FuncInfo.InsertPt, dl,
               TII->get(TargetOpcode::LOCAL_ESCAPE))
           .addSym(FrameAllocSym)
           .addFrameIndex(FI);
     }
 
+    MF.setHasLocalEscape(true);
+
     return nullptr;
   }
 
   case Intrinsic::localrecover: {
     // i8* @llvm.localrecover(i8* %fn, i8* %fp, i32 %idx)
     MachineFunction &MF = DAG.getMachineFunction();
     MVT PtrVT = TLI.getPointerTy(DAG.getDataLayout(), 0);
 
case Intrinsic::localrecover: {
     Value *FP = I.getArgOperand(1);
     SDValue FPVal = getValue(FP);
     SDValue Add = DAG.getNode(ISD::ADD, sdl, PtrVT, FPVal, OffsetVal);
     setValue(&I, Add);
 
     return nullptr;
   }
 
   case Intrinsic::eh_exceptionpointer:
   case Intrinsic::eh_exceptioncode: {
     // Get the exception pointer vreg, copy from it, and resize it to fit.
     const auto *CPI = cast<CatchPadInst>(I.getArgOperand(0));
     MVT PtrVT = TLI.getPointerTy(DAG.getDataLayout());
     const TargetRegisterClass *PtrRC = TLI.getRegClassFor(PtrVT);
     unsigned VReg = FuncInfo.getCatchPadExceptionPointerVReg(CPI, PtrRC);
     SDValue N =
         DAG.getCopyFromReg(DAG.getEntryNode(), getCurSDLoc(), VReg, PtrVT);

lib/Target/AArch64/AArch64AsmPrinter.cpp

void AArch64AsmPrinter::EmitInstruction(const MachineInstr *MI) {
   }
 
   AArch64TargetStreamer *TS =
     static_cast<AArch64TargetStreamer *>(OutStreamer->getTargetStreamer());
   // Do any manual lowerings.
   switch (MI->getOpcode()) {
   default:
     break;
+    case AArch64::MOVMCSym: {
+    unsigned DestReg = MI->getOperand(0).getReg();
+    const MachineOperand &MO_Sym = MI->getOperand(1);
+    MachineOperand Hi_MOSym(MO_Sym), Lo_MOSym(MO_Sym);
+    MCOperand Hi_MCSym, Lo_MCSym;
+
+    Hi_MOSym.setTargetFlags(AArch64II::MO_G1 | AArch64II::MO_S);
+    Lo_MOSym.setTargetFlags(AArch64II::MO_G0 | AArch64II::MO_NC);
+
+    MCInstLowering.lowerOperand(Hi_MOSym, Hi_MCSym);
+    MCInstLowering.lowerOperand(Lo_MOSym, Lo_MCSym);
+
+    MCInst MovZ;
+    MovZ.setOpcode(AArch64::MOVZXi);
+    MovZ.addOperand(MCOperand::createReg(DestReg));
+    MovZ.addOperand(Hi_MCSym);
+    MovZ.addOperand(MCOperand::createImm(16));
+    EmitToStreamer(*OutStreamer, MovZ);
+
+    MCInst MovK;
+    MovK.setOpcode(AArch64::MOVKXi);
+    MovK.addOperand(MCOperand::createReg(DestReg));
+    MovK.addOperand(MCOperand::createReg(DestReg));
+    MovK.addOperand(Lo_MCSym);
+    MovK.addOperand(MCOperand::createImm(0));
+    EmitToStreamer(*OutStreamer, MovK);
+    return;
+  }
   case AArch64::MOVIv2d_ns:
     // If the target has <rdar://problem/16473581>, lower this
     // instruction to movi.16b instead.
     if (STI->hasZeroCycleZeroingFPWorkaround() &&
         MI->getOperand(1).getImm() == 0) {
       MCInst TmpInst;
       TmpInst.setOpcode(AArch64::MOVIv16b_ns);
       TmpInst.addOperand(MCOperand::createReg(MI->getOperand(0).getReg()));

lib/Target/AArch64/AArch64FrameLowering.cpp

bool AArch64FrameLowering::hasFP(const MachineFunction &MF) const {
   // getReservedReg() -> hasFP in the middle of global isel are too early
   // to know the max call frame size. Hopefully conservatively returning "true"
   // in those cases is fine.
   // DefaultSafeSPDisplacement is fine as we only emergency spill GP regs.
   if (!MFI.isMaxCallFrameSizeComputed() ||
       MFI.getMaxCallFrameSize() > DefaultSafeSPDisplacement)
     return true;
 
+  // Win64 SEH requires frame pointer if funclets are present.
+  if (MF.hasLocalEscape())
+    return true;
+
   return false;
 }
 
 /// hasReservedCallFrame - Under normal circumstances, when a frame pointer is
 /// not required, we reserve argument space for call sites in the function
 /// immediately on entry to the current function.  This eliminates the need for
 /// add/sub sp brackets around call sites.  Returns true if the call frame is
 /// included as part of the stack frame.

lib/Target/AArch64/AArch64ISelLowering.cpp

SDValue AArch64TargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
   unsigned IntNo = cast<ConstantSDNode>(Op.getOperand(0))->getZExtValue();
   SDLoc dl(Op);
   switch (IntNo) {
   default: return SDValue();    // Don't custom lower most intrinsics.
   case Intrinsic::thread_pointer: {
     EVT PtrVT = getPointerTy(DAG.getDataLayout());
     return DAG.getNode(AArch64ISD::THREAD_POINTER, dl, PtrVT);
   }
   case Intrinsic::aarch64_neon_abs: {
     EVT Ty = Op.getValueType();
     if (Ty == MVT::i64) {
       SDValue Result = DAG.getNode(ISD::BITCAST, dl, MVT::v1i64,
                                    Op.getOperand(1));
       Result = DAG.getNode(ISD::ABS, dl, MVT::v1i64, Result);
       return DAG.getNode(ISD::BITCAST, dl, MVT::i64, Result);
     } else if (Ty.isVector() && Ty.isInteger() && isTypeLegal(Ty)) {
       return DAG.getNode(ISD::ABS, dl, Ty, Op.getOperand(1));
     } else {
       report_fatal_error("Unexpected type for AArch64 NEON intrinic");
     }
   }
   case Intrinsic::aarch64_neon_smax:
     return DAG.getNode(ISD::SMAX, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_umax:
     return DAG.getNode(ISD::UMAX, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_smin:
     return DAG.getNode(ISD::SMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_umin:
     return DAG.getNode(ISD::UMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
+
+  case Intrinsic::localaddress: {
+    // Returns one of the stack, base, or frame pointer registers, depending on
+    // which is used to reference local variables.
+    MachineFunction &MF = DAG.getMachineFunction();
+    const AArch64RegisterInfo *RegInfo = Subtarget->getRegisterInfo();
+    unsigned Reg;
+    if (RegInfo->hasBasePointer(MF))
+      Reg = RegInfo->getBaseRegister();
+    else // This function handles the SP or FP case.
+      Reg = RegInfo->getFrameRegister(MF);
+    return DAG.getCopyFromReg(DAG.getEntryNode(), dl, Reg,
+                              Op.getSimpleValueType());
+  }
+
+  case Intrinsic::eh_recoverfp: {
+    // FIXME: This needs to be implemented to correctly handle highly aligned
+    // stack objects. For now we simply return the incoming FP. Refer D53541
+    // for more details.
+    SDValue FnOp = Op.getOperand(1);
+    SDValue IncomingFPOp = Op.getOperand(2);
+    GlobalAddressSDNode *GSD = dyn_cast<GlobalAddressSDNode>(FnOp);
+    auto *Fn = dyn_cast_or_null<Function>(GSD ? GSD->getGlobal() : nullptr);
+    if (!Fn)
+      report_fatal_error(
+          "llvm.eh.recoverfp must take a function as the first argument");
+    return IncomingFPOp;
+  }
   }
 }
 
 // Custom lower trunc store for v4i8 vectors, since it is promoted to v4i16.
 static SDValue LowerTruncateVectorStore(SDLoc DL, StoreSDNode *ST,
                                         EVT VT, EVT MemVT,
                                         SelectionDAG &DAG) {
   assert(VT.isVector() && "VT should be a vector type");

lib/Target/AArch64/AArch64InstrInfo.td

 def IsBE             : Predicate<"!Subtarget->isLittleEndian()">;
 def UseAlternateSExtLoadCVTF32
     : Predicate<"Subtarget->useAlternateSExtLoadCVTF32Pattern()">;
 
 def UseNegativeImmediates
     : Predicate<"false">, AssemblerPredicate<"!FeatureNoNegativeImmediates",
                                              "NegativeImmediates">;
 
+def AArch64LocalRecover : SDNode<"ISD::LOCAL_RECOVER",
+                                  SDTypeProfile<1, 1, [SDTCisSameAs<0, 1>,
+                                                       SDTCisInt<1>]>>;
+
 
 //===----------------------------------------------------------------------===//
 // AArch64-specific DAG Nodes.
 //
 
 // SDTBinaryArithWithFlagsOut - RES1, FLAGS = op LHS, RHS
 def SDTBinaryArithWithFlagsOut : SDTypeProfile<2, 2,
                                               [SDTCisSameAs<0, 2>,

 def : Pat<(AArch64tcret rtcGPR64:$dst, (i32 timm:$FPDiff)),
           (TCRETURNriBTI rtcGPR64:$dst, imm:$FPDiff)>,
       Requires<[UseBTI]>;
 def : Pat<(AArch64tcret tglobaladdr:$dst, (i32 timm:$FPDiff)),
           (TCRETURNdi texternalsym:$dst, imm:$FPDiff)>;
 def : Pat<(AArch64tcret texternalsym:$dst, (i32 timm:$FPDiff)),
           (TCRETURNdi texternalsym:$dst, imm:$FPDiff)>;
 
+def MOVMCSym : Pseudo<(outs GPR64:$dst), (ins i64imm:$sym), []>, Sched<[]>;
+def : Pat<(i64 (AArch64LocalRecover mcsym:$sym)), (MOVMCSym mcsym:$sym)>;
+
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"

lib/Target/AArch64/AArch64RegisterInfo.cpp

bool AArch64RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   // size is large enough that referencing from the FP won't result in things
   // being in range relatively often, we can use a base pointer to allow access
   // from the other direction like the SP normally works.
   //
   // Furthermore, if both variable sized objects are present, and the
   // stack needs to be dynamically re-aligned, the base pointer is the only
   // reliable way to reference the locals.
   if (MFI.hasVarSizedObjects() || MF.hasEHFunclets()) {
     if (needsStackRealignment(MF))
       return true;
     // Conservatively estimate whether the negative offset from the frame
     // pointer will be sufficient to reach. If a function has a smallish
     // frame, it's less likely to have lots of spills and callee saved
     // space, so it's all more likely to be within range of the frame pointer.
     // If it's wrong, we'll materialize the constant and still get to the
     // object; it's just suboptimal. Negative offsets use the unscaled
     // load/store instructions, which have a 9-bit signed immediate.
if (MI.isDebugValue() || MI.getOpcode() == TargetOpcode::STACKMAP ||
     Offset += MI.getOperand(FIOperandNum + 1).getImm();
     MI.getOperand(FIOperandNum).ChangeToRegister(FrameReg, false /*isDef*/);
     MI.getOperand(FIOperandNum + 1).ChangeToImmediate(Offset);
     return;
   }
 
   // Modify MI as necessary to handle as much of 'Offset' as possible
   Offset = TFI->resolveFrameIndexReference(MF, FrameIndex, FrameReg);
+
+  if (MI.getOpcode() == TargetOpcode::LOCAL_ESCAPE) {
+    MachineOperand &FI = MI.getOperand(FIOperandNum);
+    FI.ChangeToImmediate(Offset);
+    return;
+  }
+
   if (rewriteAArch64FrameIndex(MI, FIOperandNum, FrameReg, Offset, TII))
     return;
 
   assert((!RS || !RS->isScavengingFrameIndex(FrameIndex)) &&
          "Emergency spill slot is out of reach");
 
   // If we get here, the immediate doesn't fit into the instruction.  We folded
   // as much as possible above.  Handle the rest, providing a register that is

test/CodeGen/AArch64/seh-finally.ll

+; RUN: llc -mtriple arm64-windows -o - %s | FileCheck %s
+
+; Function Attrs: noinline optnone uwtable
+define dso_local i32 @foo() {
+entry:
+; CHECK-LABEL: foo
+; CHECK: orr     w8, wzr, #0x1
+; CHECK: mov     w0, wzr
+; CHECK: mov     x1, x29
+; CHECK: .set .Lfoo$frame_escape_0, -4
+; CHECK: stur    w8, [x29, #-4]
+; CHECK: bl      "?fin$0@0@foo@@"
+; CHECK: ldur    w0, [x29, #-4]
+
+  %count = alloca i32, align 4
+  call void (...) @llvm.localescape(i32* %count)
+  store i32 0, i32* %count, align 4
+  %0 = load i32, i32* %count, align 4
+  %add = add nsw i32 %0, 1
+  store i32 %add, i32* %count, align 4
+  %1 = call i8* @llvm.localaddress()
+  call void @"?fin$0@0@foo@@"(i8 0, i8* %1)
+  %2 = load i32, i32* %count, align 4
+  ret i32 %2
+}
+
+define internal void @"?fin$0@0@foo@@"(i8 %abnormal_termination, i8* %frame_pointer) {
+entry:
+; CHECK-LABEL: @"?fin$0@0@foo@@"
+; CHECK: sub     sp, sp, #16
+; CHECK: str     x1, [sp, #8]
+; CHECK: strb    w0, [sp, #7]
+; CHECK: movz    x8, #:abs_g1_s:.Lfoo$frame_escape_0
+; CHECK: movk    x8, #:abs_g0_nc:.Lfoo$frame_escape_0
+; CHECK: add     x8, x1, x8
+; CHECK: ldr     w9, [x8]
+; CHECK: add     w9, w9, #1
+; CHECK: str     w9, [x8]
+
+  %frame_pointer.addr = alloca i8*, align 8
+  %abnormal_termination.addr = alloca i8, align 1
+  %0 = call i8* @llvm.localrecover(i8* bitcast (i32 ()* @foo to i8*), i8* %frame_pointer, i32 0)
+  %count = bitcast i8* %0 to i32*
+  store i8* %frame_pointer, i8** %frame_pointer.addr, align 8
+  store i8 %abnormal_termination, i8* %abnormal_termination.addr, align 1
+  %1 = zext i8 %abnormal_termination to i32
+  %cmp = icmp eq i32 %1, 0
+  br i1 %cmp, label %if.then, label %if.end
+
+if.then:                                          ; preds = %entry
+  %2 = load i32, i32* %count, align 4
+  %add = add nsw i32 %2, 1
+  store i32 %add, i32* %count, align 4
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %entry
+  ret void
+}
+
+; Function Attrs: nounwind readnone
+declare i8* @llvm.localrecover(i8*, i8*, i32)
+
+; Function Attrs: nounwind readnone
+declare i8* @llvm.localaddress()
+
+; Function Attrs: nounwind
+declare void @llvm.localescape(...)

test/CodeGen/AArch64/seh-localescape.ll

+; RUN: llc -mtriple arm64-windows %s -o - | FileCheck %s
+
+; Function Attrs: noinline nounwind optnone uwtable
+define dso_local i32 @foo() {
+entry:
+; CHECK-LABEL: foo
+; CHECK: .set .Lfoo$frame_escape_0, -4
+
+  %count = alloca i32, align 4
+  call void (...) @llvm.localescape(i32* %count)
+  ret i32 0
+}
+
+define internal i32 @"?filt$0@0@foo@@"(i8* %exception_pointers, i8* %frame_pointer) {
+entry:
+; CHECK-LABEL: @"?filt$0@0@foo@@"
+; CHECK: movz    x8, #:abs_g1_s:.Lfoo$frame_escape_0
+; CHECK: movk    x8, #:abs_g0_nc:.Lfoo$frame_escape_0
+
+  %0 = call i8* @llvm.localrecover(i8* bitcast (i32 ()* @foo to i8*), i8* %frame_pointer, i32 0)
+  %count = bitcast i8* %0 to i32*
+  %1 = load i32, i32* %count, align 4
+  ret i32 %1
+}
+
+; Function Attrs: nounwind readnone
+declare i8* @llvm.localrecover(i8*, i8*, i32) #2
+
+; Function Attrs: nounwind
+declare void @llvm.localescape(...) #3