[COFF, ARM64] Implement support for SEH extensions __try/__except/__finally

[mgrang]

Summary

This patch supports MS SEH extensions try/except/__finally. The intrinsics localescape and localrecover are responsible for communicating escaped static allocas from the try block to the handler.

We need to preserve frame pointers for SEH. So we create a new function/property HasLocalEscape.

[rnk]

lib/Target/AArch64/AArch64ISelLowering.cpp

2724

It's not EBP for AArch64, surely. I would generalize the text to "FP" instead.

2731–2732

Yeah, this was a great hack. :)

[mstorsjo]

I think you got the titles swapped on this and D53541; this one adds the aarch64_seh_recoverfp intrinsic while D53541 uses it to support __try/__except.

[mgrang]

In D53540#1272057, @mstorsjo wrote:

I think you got the titles swapped on this and D53541; this one adds the aarch64_seh_recoverfp intrinsic while D53541 uses it to support __try/__except.

D53541 generates aarch64_seh_recoverfp while this patch lowers it. This patch also lowers the localescape/localrecover intrinsics. I have updated the title summary accordingly.

[mgrang]

lib/Target/AArch64/AArch64AsmPrinter.cpp

571

I see that X86 generates LEA to calculate the address of the .L label. AArch64 does not have a similar instruction. I tried to use ADRP + ADD but that requires either a GlobalSymbol or an ExternalSymbol type. However, the .L label is of type MCSymbol.

So I generate a MOVK x1, #.Lfoo. The problem is that MOVK takes an unsigned immediate. The .L label here represents offset from FP. This works if the offset is positive. But for negative offsets we should generate a MOVN. However, I am not sure how to detect negative immediate without first generating a MOV.

Any suggestions?

lib/Target/AArch64/AArch64ISelLowering.cpp

2731–2732

I am not really familiar with this piece of code. Could you please comment on whether this should work for AArch64?

lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

219

As I commented above, we assume a non-negative immediate.

[rnk]

lib/Target/AArch64/AArch64AsmPrinter.cpp

571

I would expect you would want a generic MOVMCSym instruction to be able to materialize any label, not just ones that are known during assembly. This might get selected if we needed to materialize, for example, the address of the LSDA, which is also a private label.

lib/Target/AArch64/AArch64ISelLowering.cpp

2729

I think this is suppoosed to be ".seh_setframe offset"

2731–2732

I think it should. Fundamentally, we are trying to transfer information about one function's frame layout into an independent filter function. The idea is that we can write assembly like this:

foo:
.Lfoo = 0x48
  ...

foo$filt:
  mov r0, .Lfoo

Hm, if possible, it might be nice to codegen this whole thing after SD ISel. I wonder if there is some way to pass it through as an intrinsic and lower it later. That will avoid the whole concern of having a generic MCSymbol materialization instruction selection pattern.

[ssijaric]

We need to have a test case that checks for access to local variables from the finally and except funclets. The parent's frame pointer is the second argument to both the filter functions and termination handlers, fro what I see. We need to make sure that locals variables in funclets are being accessed off x1, and not the stack pointer.

[mgrang]

Will push more comprehensive tests for ARM64 SEH soon.

[majnemer]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

Since you already did an isa, this can be cast<MCConstantExpr>(Expr).

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

I tried applying this patch (plus D53541, D51524, and D54248) and building Firefox, and I got a crash on null Expr here.

Let me know if you're interested in the repro, or if it's too early for me to be playing with this code, in which case sorry for the noise.

[mgrang]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

@dmajor Thanks for testing this out :) If you have a reduced test case to repro the failure that would be great. Although currently I am busy with an internal release I expect to get back to this in Dec.

[mgrang]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

@dmajor Also what patches (if any) to the Firefox code base are needed to build Firefox for arm64-windows. Are those patches publicly available?

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

I don't really recommend it right now, it's a bit of a hassle because you have to apply some hacks to the build system. We're working on making it easier (hopefully within a week or two). In the meantime I can try to get you a repro with creduce.

[TomTan]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

@dmajor, I hit the same null reference issue after cherry-pick changes you listed. Seems FrameAllocSym is not a set to SymContentsVariable here which is expected to happen in AArch64RegisterInfo::eliminateFrameIndex.

[dmajor]

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6155 ↗

(On Diff #171996)

Here's the repro of the crash from creduce:

class a {
  void b();
  void c();
};
void a::b() {
  int f;
  __try {
    c();
  } __except (f) {
  }
}

[rnk]

Sorry, I didn't realize this was waiting on my review. It looks like this is mostly about localescape, so let's change the patch description and code to be about that.

include/llvm/CodeGen/MachineFunction.h

332 ↗

(On Diff #171996)

IMO this should really live on the AArch64-specific function info, since it's not general to x64/x86 yet.

Also, this isn't SEH specific, this is set to true only by localescape. Maybe you should rename it? localescape was intended to be orthogonal to SEH, potentially usable for non-externally visible lambdas, although the codegen right now is particularly bad.

[mgrang]

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

[TomTan]

In D53540#1326343, @mgrang wrote:

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

@mgrang, the SEH fix still causes ARM64 COFF file crash when linking with CFG enabled (/guard:cf). Seems functions are not marked as IMAGE_SYM_DTYPE_FUNCTION in ARM64 COFF which was done in both x86 and arm32 COFF (see below 2 links).

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/X86/X86AsmPrinter.cpp#L67

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/ARM/ARMAsmPrinter.cpp#L148

[mgrang]

In D53540#1326353, @TomTan wrote:

In D53540#1326343, @mgrang wrote:

@dmajor My updated change fixes the crash in your testcase.

Now, I am trying to build the SEH tests released by MS: https://github.com/Microsoft/windows_seh_tests. I started with XCPT4 tests. test5 is currently failing which I am triaging now.

@mgrang, the SEH fix still causes ARM64 COFF file crash when linking with CFG enabled (/guard:cf). Seems functions are not marked as IMAGE_SYM_DTYPE_FUNCTION in ARM64 COFF which was done in both x86 and arm32 COFF (see below 2 links).

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/X86/X86AsmPrinter.cpp#L67

https://github.com/llvm-mirror/llvm/blob/2cc0a7da876c1d8c32775b0119e1e15aaa759b9e/lib/Target/ARM/ARMAsmPrinter.cpp#L148

Thanks @TomTan. I have a fix here: https://reviews.llvm.org/D55535

[rnk]

CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

6166 ↗

(On Diff #177612)

It still seems true that this isn't about seh funclets, it's about localescape / localrecover, which is supposed to be orthogonal (for use in non-ABI visible lambdas), so I'd recommend renaming this boolean.

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

I think this assertion will fail if you use localrecover before localescape. I would make a test case for that. Is there really not some pseudo instruction we can use so that the materialization can be relaxed by the assembler?

Otherwise, there's no point to using this MCSymbol hack. We might as well have a simpler side table if we retain the requirement that localescape be code generated first.

[efriedma]

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

Is there really not some pseudo instruction we can use

The AArch64 assembler currently doesn't have any useful support for this sort of construct. Maybe it could be extended. (Is the value an arbitrary 64-bit integer, or is there some restriction on the possible values?)

[rnk]

Target/AArch64/AArch64AsmPrinter.cpp

680 ↗

(On Diff #177612)

These values are offsets from the FP (or SP) to a stack object, so they are typically small. If there are implementation limits on the size of an aarch64 stack frame, we can definitely use that to guide isel here.

[mgrang]

include/llvm/CodeGen/MachineFunction.h

332 ↗

(On Diff #171996)

I tried to move this to AArch64MachineFunctionInfo. I set it in AArch64RegisterInfo when I encounter LOCAL_ESCAPE and need to access it in AArch64FrameLowering. But it seems setting it in AArch64RegisterInfo is too late. So for now I have kept it on the MF.

lib/Target/AArch64/AArch64AsmPrinter.cpp

566

@rnk Yes, you are correct that the assert here will fail is the localrecover comes before localescape.
I have tried to model this similar to X86 but the problem is AArch64 cannot do an ADRP on an MCSymbol. So I wasn't sure how to materialize this in the assembler.
Could you give me some idea on how this can be done?

[rnk]

include/llvm/CodeGen/MachineFunction.h

332 ↗

(On Diff #171996)

I see, we could really only do that if localescape was lowered in target dependent code. I think this is fine the way it is now. It's clearer (to me, at least) after the rename.

lib/CodeGen/AsmPrinter/WinException.cpp

548–549 ↗

(On Diff #178775)

This can just be isAArch64, we already assigned that to a member boolean.

lib/Target/AArch64/AArch64AsmPrinter.cpp

566

I guess I've been assuming that you could assemble something like this in a .s file for aarch64, like you can for x86:

main:
  movl $.Lasdf, %eax
  ret
.Lasdf = 1234

I would assume that the aarch64 equivalent would be:

main:
  mov d0, #.Lasdf
  ret
.Lasdf = 1234

But that doesn't assemble. Does gnu as support that? I don't have an aarch64 binutils build. Maybe llvm-mc should allow mov/movk to accept a symbolic label operand.

I don't think we would want to use ADRP, IIRC that's for materializing some PC-relative address, but obviously this isn't that kind of symbol.

[mgrang]

lib/Target/AArch64/AArch64AsmPrinter.cpp

566

GNU does not seem to support it as well.

Error: operand 2 should be a SIMD vector element -- `mov d0,#.Lasdf'

[efriedma]

lib/Target/AArch64/AArch64AsmPrinter.cpp

566

Does gnu as support that?

"mov" is pseudo-instruction, but it will expand to at most one instruction (a movz, a movn, or an orr); this is defined in the AArch64 architecture reference. There is no variable-length pseudo-instruction; if you need a larger value, you have to write out a movz/movk sequence explicitly.

I guess the stack size is currently limited to 256MB on Windows because of the way we emit the prologue on Windows (that's the limit for the alloc_l unwind opcode). We could maybe teach LLVM to use multiple alloc_l operations, but it's probably reasonable to assume the stack for a function is less than 4GB, so we could emit just two instructions (movz+movk).

For value less than 2^32, gas supports something like the following:

movz x0, #:abs_g1:.Lasdf
movk x0, #:abs_g0_nc:.Lasdf
.Lasdf = 0x12345678

If you try this with LLVM, you currently get an error "no resolvable MOVZ/MOVK fixups supported yet", but I think that's straightforward to fix. (LLVM supports the syntax in general, just not the case where the value is defined in the same object file.)

[rnk]

lib/Target/AArch64/AArch64AsmPrinter.cpp

566

I saw your fixup patches, and this seems like the way to go. At least, this was the most principled way I could come up with to pass information about the stack frame layout of one machine function to another. I didn't want to use a side table and lock in the code generation order dependence.

[mgrang]

Addressed comments. The MCSymbol for the frame offset is now lowered in the assembler. D56037 needs to merge first.
I am now working on test cases. Will push them soon.

[efriedma]

lib/Target/AArch64/AArch64AsmPrinter.cpp

582

I think this "createImm" is wrong; should be 16. (It probably doesn't show up in the printed assembly, but I think it affects the encoding.)

lib/Target/AArch64/AArch64RegisterInfo.cpp

260

This can't be correct in general... in cases involving stack realignment, it's impossible to generate code without a base pointer.

[mgrang]

Added test cases for localescape and try-finally. Will add try-except test case soon.

[efriedma]

lib/Target/AArch64/AArch64ISelLowering.cpp

2775

Are you sure this does the right thing in general? The link between the choice of register here, and the register used for the getFrameIndexReference call in WinException::emitEHRegistrationOffsetLabel , seems fragile at best.

[rnk]

Thanks for pushing the MC work to support this forward!

@efriedma has one outstanding comment, but I don't see how to untangle the choice of frame register from AArch64FrameLowering::getIndexFrameReference, or the x86 version of the same, so I'm happy with this. Please wait for his response, though.

lib/Target/AArch64/AArch64ISelLowering.cpp

2775

It is fragile, but it's what was done for x86. Unfortunately, it's actually really hard to know which physical register is going to be used to address locals during ISel, so I just did this brittle thing and moved on. See these comments from AArch64FrameLowering::hasFP for some of the phase ordering problems:

// With large callframes around we may need to use FP to access the scavenging
// emergency spillslot.
//
// Unfortunately some calls to hasFP() like machine verifier ->
// getReservedReg() -> hasFP in the middle of global isel are too early
// to know the max call frame size. Hopefully conservatively returning "true"
// in those cases is fine.
// DefaultSafeSPDisplacement is fine as we only emergency spill GP regs.
if (!MFI.isMaxCallFrameSizeComputed() ||
    MFI.getMaxCallFrameSize() > DefaultSafeSPDisplacement)
  return true;

I think we are mostly safe because when using EH funclets, an FP is always needed, so it's just a decision between "base" and "frame" pointer registers. That is currently entirely determined by the existence of user allocas with alignments above the standard ABI stack alignment.

However, you can imagine that some later pass might try to create more highly aligned stack objects, which would require a base pointer. For example, the register allocator might spill vector registers that require more stack alignment than the ABI provides. Today, for x86 at least, LLVM punts on this problem, and simply uses unaligned instructions for the spill and reload. I don't know how to get into the analogous situation for AArch64 or what LLVM would do, though.

[smeenai]

lib/Target/AArch64/AArch64ISelLowering.cpp

2775

Is this at all related to https://bugs.llvm.org/show_bug.cgi?id=37169?

[rnk]

lib/Target/AArch64/AArch64ISelLowering.cpp

2775

I had forgotten about that issue, but that's exactly the case I was describing. I proposed this solution there:

I think localaddress should really lower down to a LEA of a zero-sized stack object. It's really trying to say, give me the address of the local variable area. The localrecover label assignments would instead compute the difference between this particular zero-sized stack object and their stack object.

For AArch64, LEA would really be ADC, or whatever is generated to materialize the address of an alloca. Does something like that make sense?

It occurs to me that this localescape system is pretty flawed, since not all local variables are static allocas. This source crashes clang today:

struct Foo {
  int x, y, z;
};
int filter(int);
void may_throw();
void seh_byval(Foo o) {
  __try {
    may_throw();
  } __except(filter(o.x)) {
  }
}

Oops. =/

[efriedma]

If you're okay with trying to resolve any remaining bugs as a followup, that's fine with me; LGTM.

lib/Target/AArch64/AArch64ISelLowering.cpp

2775

That is currently entirely determined by the existence of user allocas with alignments above the standard ABI stack alignment.

This is not true on AArch64. There's a second condition: whether the size of the local frame is greater than 256 bytes. This condition isn't strictly required, but avoids materializing offsets in some situations. (AArch64 addressing modes support very limited immediate negative offsets.)

That doesn't really have any impact on its own, but AArch64FrameLowering::getFrameIndexReference actually chooses between FP-relative and BP-relative addressing based on the offsets, in general. So we probably need to do something here, to force the frame index to be resolved the same way as localaddress.

I don't think we can hit an issue similar to bug 37169 on AArch64; the natural stack alignment is 16, and there currently isn't anything that requires higher alignment.

It occurs to me that this localescape system is pretty flawed, since not all local variables are static allocas.

Yes, we need special handling for byval/inalloca arguments at the IR level; in general, it's impossible to compute an offset to an argument relative to the base pointer.

[mgrang]

Added a default lowering for llvm.eh.recoverfp intrinsic.

[efriedma]

LGTM

[mgrang]

@rnk @efriedma Thanks a lot for all your valuable review comments. I'll commit this now :)

include/llvm/IR/IntrinsicsAArch64.td

 def int_aarch64_clrex : Intrinsic<[]>;
 
 def int_aarch64_sdiv : Intrinsic<[llvm_anyint_ty], [LLVMMatchType<0>,
                                 LLVMMatchType<0>], [IntrNoMem]>;
 def int_aarch64_udiv : Intrinsic<[llvm_anyint_ty], [LLVMMatchType<0>,
                                 LLVMMatchType<0>], [IntrNoMem]>;
 
 //===----------------------------------------------------------------------===//
+// SEH intrinsics for Windows
+// Given a pointer to the end of an EH registration object, returns the true
+// parent frame address that can be used with llvm.localrecover.
+def int_aarch64_seh_recoverfp : Intrinsic<[llvm_ptr_ty],
+                                          [llvm_ptr_ty, llvm_ptr_ty],
+                                          [IntrNoMem]>;
+
+//===----------------------------------------------------------------------===//
 // HINT
 
 def int_aarch64_hint : Intrinsic<[], [llvm_i32_ty]>;
 
 //===----------------------------------------------------------------------===//
 // Data Barrier Instructions
 
 def int_aarch64_dmb : GCCBuiltin<"__builtin_arm_dmb">, MSBuiltin<"__dmb">, Intrinsic<[], [llvm_i32_ty]>;

lib/Target/AArch64/AArch64AsmPrinter.cpp

if (AArch64FI->getLOHRelated().count(MI)) {
     LOHInstToLabel[MI] = LOHLabel;
     OutStreamer->EmitLabel(LOHLabel);
   }
 
   // Do any manual lowerings.
   switch (MI->getOpcode()) {
   default:
     break;
+  case AArch64::MOVMCSym: {
+    MCSymbol *Sym = MI->getOperand(1).getMCSymbol();
+    const MCExpr *Expr = MCSymbolRefExpr::create(Sym, OutContext);
+
+    MCInst TmpInst;
+    TmpInst.setOpcode(AArch64::MOVKXi);
+    TmpInst.addOperand(MCOperand::createReg(MI->getOperand(0).getReg()));
+    TmpInst.addOperand(MCOperand::createReg(MI->getOperand(0).getReg()));
+    TmpInst.addOperand(MCOperand::createExpr(Expr));
+    TmpInst.addOperand(MCOperand::createImm(0)); // unused shift.
+    EmitToStreamer(*OutStreamer, TmpInst);
+    return;
+  }
+
   case AArch64::MOVIv2d_ns:
     // If the target has <rdar://problem/16473581>, lower this
     // instruction to movi.16b instead.
     if (STI->hasZeroCycleZeroingFPWorkaround() &&
         MI->getOperand(1).getImm() == 0) {
       MCInst TmpInst;
       TmpInst.setOpcode(AArch64::MOVIv16b_ns);
       TmpInst.addOperand(MCOperand::createReg(MI->getOperand(0).getReg()));
       TmpInst.addOperand(MCOperand::createImm(MI->getOperand(1).getImm()));
       EmitToStreamer(*OutStreamer, TmpInst);
       return;

lib/Target/AArch64/AArch64ISelLowering.cpp

 #include "llvm/Analysis/VectorUtils.h"
 #include "llvm/CodeGen/CallingConvLower.h"
 #include "llvm/CodeGen/MachineBasicBlock.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstr.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
+#include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"
 #include "llvm/CodeGen/SelectionDAG.h"
 #include "llvm/CodeGen/SelectionDAGNodes.h"
 #include "llvm/CodeGen/TargetCallingConv.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/Attributes.h"
static SDValue LowerMULH(SDValue Op, SelectionDAG &DAG) {
   SDValue Mull2 = DAG.getNode(MullOpc, DL, MullVT, VMull2V0, VMull2V1);
 
   Mull  = DAG.getNode(ISD::BITCAST, DL, VT, Mull);
   Mull2 = DAG.getNode(ISD::BITCAST, DL, VT, Mull2);
 
   return DAG.getNode(AArch64ISD::UZP2, DL, VT, Mull, Mull2);
 }
 
+static SDValue recoverFramePointer(SelectionDAG &DAG, const Function *Fn,
+                                   SDValue EntryFP) {
+  MachineFunction &MF = DAG.getMachineFunction();
+  SDLoc dl;
+
+  const TargetLowering &TLI = DAG.getTargetLoweringInfo();
+  MVT PtrVT = TLI.getPointerTy(DAG.getDataLayout());
+
+  // It's possible that the parent function no longer has a personality function
+  // if the exceptional code was optimized away, in which case we just return
+  // the incoming FP.
+  if (!Fn->hasPersonalityFn())
+    return EntryFP;
+
+  // Get an MCSymbol that will ultimately resolve to the frame offset of the EH
+  // registration, or the .set_setframe offset.
+  MCSymbol *OffsetSym =
+      MF.getMMI().getContext().getOrCreateParentFrameOffsetSymbol(
+          GlobalValue::dropLLVMManglingEscape(Fn->getName()));
+  SDValue OffsetSymVal = DAG.getMCSymbol(OffsetSym, PtrVT);
+  SDValue ParentFrameOffset =
+      DAG.getNode(ISD::LOCAL_RECOVER, dl, PtrVT, OffsetSymVal);
+
+  return DAG.getNode(ISD::ADD, dl, PtrVT, EntryFP, ParentFrameOffset);
+}
+
 SDValue AArch64TargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
                                                      SelectionDAG &DAG) const {
   unsigned IntNo = cast<ConstantSDNode>(Op.getOperand(0))->getZExtValue();
   SDLoc dl(Op);
   switch (IntNo) {
   default: return SDValue();    // Don't custom lower most intrinsics.
   case Intrinsic::thread_pointer: {
     EVT PtrVT = getPointerTy(DAG.getDataLayout());
case Intrinsic::aarch64_neon_umax:
     return DAG.getNode(ISD::UMAX, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_smin:
     return DAG.getNode(ISD::SMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
   case Intrinsic::aarch64_neon_umin:
     return DAG.getNode(ISD::UMIN, dl, Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(2));
+
+  case Intrinsic::aarch64_seh_recoverfp: {
+    SDValue FnOp = Op.getOperand(1);
+    SDValue IncomingFPOp = Op.getOperand(2);
+    GlobalAddressSDNode *GSD = dyn_cast<GlobalAddressSDNode>(FnOp);
+    auto *Fn = dyn_cast_or_null<Function>(GSD ? GSD->getGlobal() : nullptr);
+    if (!Fn)
+      report_fatal_error(
+      "llvm.aarch64.seh.recoverfp must take a function as the first argument");
+    return recoverFramePointer(DAG, Fn, IncomingFPOp);
+  }
   }
 }
 
 // Custom lower trunc store for v4i8 vectors, since it is promoted to v4i16.
 static SDValue LowerTruncateVectorStore(SDLoc DL, StoreSDNode *ST,
                                         EVT VT, EVT MemVT,
                                         SelectionDAG &DAG) {
   assert(VT.isVector() && "VT should be a vector type");

lib/Target/AArch64/AArch64InstrInfo.td

 
 def AArch64saddv    : SDNode<"AArch64ISD::SADDV", SDT_AArch64UnaryVec>;
 def AArch64uaddv    : SDNode<"AArch64ISD::UADDV", SDT_AArch64UnaryVec>;
 def AArch64sminv    : SDNode<"AArch64ISD::SMINV", SDT_AArch64UnaryVec>;
 def AArch64uminv    : SDNode<"AArch64ISD::UMINV", SDT_AArch64UnaryVec>;
 def AArch64smaxv    : SDNode<"AArch64ISD::SMAXV", SDT_AArch64UnaryVec>;
 def AArch64umaxv    : SDNode<"AArch64ISD::UMAXV", SDT_AArch64UnaryVec>;
 
+def AArch64LocalRecover : SDNode<"ISD::LOCAL_RECOVER",
+                                  SDTypeProfile<1, 1, [SDTCisSameAs<0, 1>,
+                                                       SDTCisInt<1>]>>;
+
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
 
 // AArch64 Instruction Predicate Definitions.
 // We could compute these on a per-module basis but doing so requires accessing
 // the Function object through the <Target>Subtarget and objections were raised
 // to that (see post-commit review comments for r301750).

 def : Pat<(AArch64tcret rtcGPR64:$dst, (i32 timm:$FPDiff)),
           (TCRETURNriBTI rtcGPR64:$dst, imm:$FPDiff)>,
       Requires<[UseBTI]>;
 def : Pat<(AArch64tcret tglobaladdr:$dst, (i32 timm:$FPDiff)),
           (TCRETURNdi texternalsym:$dst, imm:$FPDiff)>;
 def : Pat<(AArch64tcret texternalsym:$dst, (i32 timm:$FPDiff)),
           (TCRETURNdi texternalsym:$dst, imm:$FPDiff)>;
 
+def MOVMCSym : Pseudo<(outs GPR64:$dst), (ins i64imm:$sym), []>, Sched<[]>;
+def : Pat<(i64 (AArch64LocalRecover mcsym:$sym)), (MOVMCSym mcsym:$sym)>;
+
 include "AArch64InstrAtomics.td"
 include "AArch64SVEInstrInfo.td"

lib/Target/AArch64/AArch64RegisterInfo.cpp

bool AArch64RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   // In the presence of variable sized objects, if the fixed stack size is
   // large enough that referencing from the FP won't result in things being
   // in range relatively often, we can use a base pointer to allow access
   // from the other direction like the SP normally works.
   // Furthermore, if both variable sized objects are present, and the
   // stack needs to be dynamically re-aligned, the base pointer is the only
   // reliable way to reference the locals.
   if (MFI.hasVarSizedObjects()) {
     if (needsStackRealignment(MF))
       return true;
     // Conservatively estimate whether the negative offset from the frame
     // pointer will be sufficient to reach. If a function has a smallish
     // frame, it's less likely to have lots of spills and callee saved
     // space, so it's all more likely to be within range of the frame pointer.
     // If it's wrong, we'll materialize the constant and still get to the
     // object; it's just suboptimal. Negative offsets use the unscaled
     // load/store instructions, which have a 9-bit signed immediate.
if (MI.isDebugValue() || MI.getOpcode() == TargetOpcode::STACKMAP ||
     Offset += MI.getOperand(FIOperandNum + 1).getImm();
     MI.getOperand(FIOperandNum).ChangeToRegister(FrameReg, false /*isDef*/);
     MI.getOperand(FIOperandNum + 1).ChangeToImmediate(Offset);
     return;
   }
 
   // Modify MI as necessary to handle as much of 'Offset' as possible
   Offset = TFI->resolveFrameIndexReference(MF, FrameIndex, FrameReg);
+
+  if (MI.getOpcode() == TargetOpcode::LOCAL_ESCAPE) {
+    MachineOperand &FI = MI.getOperand(FIOperandNum);
+    FI.ChangeToImmediate(Offset);
+    return;
+  }
+
   if (rewriteAArch64FrameIndex(MI, FIOperandNum, FrameReg, Offset, TII))
     return;
 
   assert((!RS || !RS->isScavengingFrameIndex(FrameIndex)) &&
          "Emergency spill slot is out of reach");
 
   // If we get here, the immediate doesn't fit into the instruction.  We folded
   // as much as possible above.  Handle the rest, providing a register that is

lib/Target/AArch64/MCTargetDesc/AArch64AsmBackend.cpp

if (TheTriple.isOSBinFormatCOFF() && !IsResolved)
       Value &= 0xfff;
     // Unsigned 12-bit immediate which gets multiplied by 16
     if (Value >= 0x10000)
       Ctx.reportError(Fixup.getLoc(), "fixup value out of range");
     if (Value & 0xf)
       Ctx.reportError(Fixup.getLoc(), "fixup must be 16-byte aligned");
     return Value >> 4;
   case AArch64::fixup_aarch64_movw:
-    Ctx.reportError(Fixup.getLoc(),
-                    "no resolvable MOVZ/MOVK fixups supported yet");
-    return Value;
+    // 16-bit immediate. Assumed unsigned.
+    return Value & 0xffff;
   case AArch64::fixup_aarch64_pcrel_branch14:
     // Signed 16-bit immediate
     if (SignedValue > 32767 || SignedValue < -32768)
       Ctx.reportError(Fixup.getLoc(), "fixup value out of range");
     // Low two bits are not encoded (4-byte alignment assumed).
     if (Value & 0x3)
       Ctx.reportError(Fixup.getLoc(), "fixup not sufficiently aligned");
     return (Value >> 2) & 0x3fff;

test/CodeGen/AArch64/seh-try-except.ll

+; RUN: llc -mtriple=arm64-windows -verify-machineinstrs < %s | FileCheck %s
+
+; Generated from C program:
+; void foo() {
+;   int x = 0;
+;   __try {}
+;   __except(x = 0) {}
+; }
+
+declare i8* @llvm.aarch64.seh.recoverfp(i8*, i8*)
+declare i8* @llvm.localrecover(i8*, i8*, i32)
+declare void @llvm.localescape(...)
+
+define dso_local void @foo() {
+entry:
+  %x = alloca i32, align 4
+  %__exception_code = alloca i32, align 4
+  call void (...) @llvm.localescape(i32* %x)
+  store i32 0, i32* %x, align 4
+  ret void
+}
+
+; CHECK-LABEL: foo
+; CHECK: .set .Lfoo$frame_escape_0, 12
+
+define internal i32 @"?filt$0@0@foo@@"(i8* %exception_pointers, i8* %frame_pointer) {
+entry:
+  %frame_pointer.addr = alloca i8*, align 8
+  %exception_pointers.addr = alloca i8*, align 8
+  %0 = call i8* @llvm.aarch64.seh.recoverfp(i8* bitcast (void ()* @foo to i8*), i8* %frame_pointer)
+  %1 = call i8* @llvm.localrecover(i8* bitcast (void ()* @foo to i8*), i8* %0, i32 0)
+  %x = bitcast i8* %1 to i32*
+  %__exception_code = alloca i32, align 4
+  store i8* %frame_pointer, i8** %frame_pointer.addr, align 8
+  store i8* %exception_pointers, i8** %exception_pointers.addr, align 8
+  %2 = bitcast i8* %exception_pointers to { i32*, i8* }*
+  %3 = getelementptr inbounds { i32*, i8* }, { i32*, i8* }* %2, i32 0, i32 0
+  %4 = load i32*, i32** %3, align 8
+  %5 = load i32, i32* %4, align 4
+  store i32 %5, i32* %__exception_code, align 4
+  store i32 0, i32* %x, align 4
+  ret i32 0
+}
+
+; CHECK-LABEL: "?filt$0@0@foo@@"
+; CHECK: movk x9, #.Lfoo$frame_escape_0