[ARM64] [Windows] Exception handling support in frame lowering

[ssijaric]

Summary

This is a follow-up patch to D50166. It modifies frame lowering to emit pseudo instructions corresponding to unwind codes (eg. SEH_SaveReg), that the patch in D50166 (MCLayer part) will emit into the .xdata section. This includes pairing of only consecutive registers in prologue and epilogue, and saving/restoring fp,lr instead of lr,fp on Windows when unwinding data needs to be emitted.

Changes for funclets and EH tables where they differ from X86 will be upstreamed in separate patches.

[ssijaric]

Remove a couple of unused functions from the initial patch. Fix formatting.

[thegameg]

lib/Target/AArch64/AArch64FrameLowering.cpp

185

From what I've seen in the way it's used, I'm not sure this class is actually useful. It seems to be always constructed at the end of a scope, where a BuildMI would have been much more readable.

409

Can you use MCRegisterInfo::getDwarfRegNum for this?

704

Can you move this into a separate function?

768

Typo (laset) and extra spaces here.

775

DL_CFI does not match the LLVM naming conventions.

832

This does not seem equivalent to what PrologueSaveSize was.

851

Extra spaces in the comments.

1167

This was already set in emitPrologue. Why is it needed here?

1170

Same here, it doesn't seem equivalent to PrologueSaveSize.

1477

Can we make this windows specific check a separate function and move the comment there?

[rnk]

The main points are:

1. Let's split funclet EH stuff out into a separate patch and just focus on CFI in this
2. I would prioritize implementing D50166 mostly in lib/Target/AArch64

lib/CodeGen/AsmPrinter/WinException.cpp

238–239 ↗

(On Diff #159158)

This variable is used frequently enough that I'd make it a member of WinException.

252 ↗

(On Diff #159158)

How is this different from the existing .seh_endproc directive emitted below?

636–637 ↗

(On Diff #159158)

Please abstract this into a generic getLabel method. It can internally check a member IsAArch64 boolean and add one or not.

882 ↗

(On Diff #159158)

None of the changes so far appear to be necessary for plain unwind info, no C++ exception handling. I'd prefer to split this patch into changes necessary for generating correct unwind info, and changes necessary to make C++ exception handling work.

lib/Target/AArch64/AArch64FrameLowering.cpp

586

Let's not have two boolean parameters with default values next to each other. It makes it easy to confuse their meaning at the call site.

1869

Hm, looks like I added this for X86. It seems more confusing than just defaulting MachineFunction::HasWinCFI to false. @hans, would you be OK if I made that Optional<bool> into a plain bool initialized to false? It seems intuitive enough that if emitPrologue isn't called (for naked functions), the function doesn't have CFI.

[hans]

lib/Target/AArch64/AArch64FrameLowering.cpp

1869

Sounds good to me.

[ssijaric]

lib/CodeGen/AsmPrinter/WinException.cpp

238–239 ↗

(On Diff #159158)

WinException is now part of the MCLayer patch (D50166) and includes this change. I moved WinException to D50166 to get the test case to work as I check for the validity of unwind codes (and other information) in the .xdata section.

252 ↗

(On Diff #159158)

This is now documented in the updated patch. The .seh_endproc below is not emitted until after the .xdata section is populated. We use the "FuncletOrFuncEnd" directive to calculate sizes of funclets and functions that .xdata requires.

636–637 ↗

(On Diff #159158)

WinException is now moved to the MCLayer patch, D50166, and this is addressed there.

882 ↗

(On Diff #159158)

I will update the MCLayer patch, which now includes WinException, to exclude this part.

lib/Target/AArch64/AArch64FrameLowering.cpp

185

I used it to avoid making sure that I inserted SEH_PrologEnd or SEH_EpilogEnd on all possible exits from emitEpilogue or emitPrologue. I will undo it.

409

Will do in the updated patch.

586

Will fix this in the updated patch.

704

Will do in the updated patch.

768

Will fix in the updated patch.

775

Will fix in the updated patch.

832

I ran into a bug where convertCalleeSaveRestoreToSPPrePostIncDec was being called when the CalleeSavedStackSize was zero, but FixedObject was of non-zero size (due to varargs on Windows). I cannot reproduce this any longer, so I will restore the original check. If I run into the problem again, I will post a separate patch.

851

Will fix in the updated patch.

1167

I will document this better. It was needed because it would get reset in the presence of funclets.

1170

Please see the reply to the previous comment.

1477

Yes, will fix in the updated patch.

[ssijaric]

Addressed the comments and reduced the patch to handle only the frame lowering portion. Changes for funclets and EH tables will go in separate patches.

[thegameg]

lib/Target/AArch64/AArch64FrameLowering.cpp

159

Should this be in AArch64InstrInfo?

459

How about:

case AArch64::STPDpost:
  Imm = -Imm;
  LLVM_FALLTHROUGH
case AArch64::STPDpre:
  [...]

Same applied to the other cases below.

466

Is Reg0 - AArch64::D0 the same as MRI->getDwarfRegNum?

Same applied to the other cases below.

483

Why the extra { / }?

1408

Indentation seems off.

1597

I would rather just special-case the whole block since it seems to be having a pretty different behaviour than the non-wincfi path, but it's just a personal preference.

1678

Same here.

[ssijaric]

Thanks for reviewing, Francis. I'll upload an updated patch.

lib/Target/AArch64/AArch64FrameLowering.cpp

159

Will move it in the updated patch.

459

Will change it in the updated patch.

466

getDwarfRegNum doesn't work for floating point registers, as the Dwarf register numbers start at 64 for those:

e.g.

def B0 : AArch64Reg<0, "b0">, DwarfRegNum<[64]>;

I looked around for an existing API to retrieve the number corresponding to the register, but didn't see one.

483

Will remove in the updated patch.

1408

Will fix in the updated patch.

1597

I'll special case the whole block in the updated patch.

[ssijaric]

Addressed Francis' review feedback, and ran clang-format through some parts of the changes in AArch64FrameLowering.cpp. Changed the title to better reflect the intent of the patch.

[thegameg]

lib/Target/AArch64/AArch64FrameLowering.cpp

460

We often use use LLVM_FALLTHROUGH; in this case.

[ssijaric]

Add test cases. Use getSEHReg to retrieve SEH register numbers. Clean up the logic in a few places.

[efriedma]

Test coverage seems reasonable to me.

lib/Target/AArch64/AArch64FrameLowering.cpp

567

llvm_unreachable.

868

MOVi64imm is a pseudo-instruction that expands to up to four instructions. (Here, probably not more than two, given that the unwind opcodes don't allow allocating more than 256MB of stack at a time anyway.) So I think you might be short one NOP in cases that involve allocating more than 1MB of stack.

Probably the simplest solution is to explicitly write out the relevant instructions in frame lowering, so you know how many nops you will need. (See AArch64ExpandPseudo::expandMOVImmSimple.)

1473

Not sure why you're deleting the comment here.

1511

FixupDone = NeedsWinCFI; is technically correct, I guess, but it seems simpler to write the equivalent FixupDone = true;.

It might be worth changing this logic further at some point, to rearrange spills to avoid ldp/stp operations that aren't 16-byte-aligned on Windows. (They're not wrong, but they might be slower.) But maybe it's rare enough that it doesn't matter.

[ssijaric]

Address Eli's feedback.

[efriedma]

LGTM

[rnk]

Looks good, thanks!

lib/Target/AArch64/AArch64FrameLowering.cpp

456

Please use llvm_unreachable instead of assert(false).

lib/Target/AArch64/AArch64CallingConvention.td

 // It would be better to model its preservation semantics properly (create a
 // vreg on entry, use it in RET & tail call generation; make that vreg def if we
 // end up saving LR as part of a call frame). Watch this space...
 def CSR_AArch64_AAPCS : CalleeSavedRegs<(add LR, FP, X19, X20, X21, X22,
                                            X23, X24, X25, X26, X27, X28,
                                            D8,  D9,  D10, D11,
                                            D12, D13, D14, D15)>;
 
+// Win64 has unwinding codes for an (FP,LR) pair, save_fplr and save_fplr_x.
+// We put FP before LR, so that frame lowering logic generates (FP,LR) pairs,
+// and not (LR,FP) pairs.
+def CSR_Win_AArch64_AAPCS : CalleeSavedRegs<(add FP, LR, X19, X20, X21, X22,
+                                               X23, X24, X25, X26, X27, X28,
+                                               D8, D9, D10, D11,
+                                               D12, D13, D14, D15)>;
+
 // Constructors and destructors return 'this' in the iOS 64-bit C++ ABI; since
 // 'this' and the pointer return value are both passed in X0 in these cases,
 // this can be partially modelled by treating X0 as a callee-saved register;
 // only the resulting RegMask is used; the SaveList is ignored
 //
 // (For generic ARM 64-bit ABI code, clang will not generate constructors or
 // destructors with 'this' returns, so this RegMask will not be used in that
 // case)

lib/Target/AArch64/AArch64FrameLowering.cpp

 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineModuleInfo.h"
 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/RegisterScavenging.h"
 #include "llvm/CodeGen/TargetInstrInfo.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/IR/Attributes.h"
 #include "llvm/IR/CallingConv.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/DebugLoc.h"
 #include "llvm/IR/Function.h"
+#include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCDwarf.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/raw_ostream.h"
 #include "llvm/Target/TargetMachine.h"
 #include "llvm/Target/TargetOptions.h"

 STATISTIC(NumRedZoneFunctions, "Number of functions using red zone");
 
 /// This is the biggest offset to the stack pointer we can encode in aarch64
 /// instructions (without using a separate calculation and a temp register).
 /// Note that the exception here are vector stores/loads which cannot encode any
 /// displacements (see estimateRSStackSizeLimit(), isAArch64FrameOffsetLegal()).
 static const unsigned DefaultSafeSPDisplacement = 255;
 
 /// Look at each instruction that references stack frames and return the stack
 /// size limit beyond which some of these instructions will require a scratch
 /// register during their expansion later.
 static unsigned estimateRSStackSizeLimit(MachineFunction &MF) {
   // FIXME: For now, just conservatively guestimate based on unscaled indexing
   // range. We'll end up allocating an unnecessary spill slot a lot, but
   // realistically that's not a big deal at this stage of the game.
   for (MachineBasicBlock &MBB : MF) {
     for (MachineInstr &MI : MBB) {
for (MachineInstr &MI : MBB) {
         int Offset = 0;
         if (isAArch64FrameOffsetLegal(MI, Offset, nullptr, nullptr, nullptr) ==
             AArch64FrameOffsetCannotUpdate)
           return 0;
       }
     }
   }
   return DefaultSafeSPDisplacement;
 }
 
 bool AArch64FrameLowering::canUseRedZone(const MachineFunction &MF) const {
   if (!EnableRedZone)
     return false;
   // Don't use the red zone if the function explicitly asks us not to.
   // This is typically used for kernel code.
   if (MF.getFunction().hasFnAttribute(Attribute::NoRedZone))
     return false;
static bool windowsRequiresStackProbe(MachineFunction &MF,
   if (F.hasFnAttribute("stack-probe-size"))
     F.getFnAttribute("stack-probe-size")
         .getValueAsString()
         .getAsInteger(0, StackProbeSize);
   return (StackSizeInBytes >= StackProbeSize) &&
          !F.hasFnAttribute("no-stack-arg-probe");
 }
 
 bool AArch64FrameLowering::shouldCombineCSRLocalStackBump(
     MachineFunction &MF, unsigned StackBumpBytes) const {
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
 
   if (AFI->getLocalStackSize() == 0)
     return false;
bool AArch64FrameLowering::shouldCombineCSRLocalStackBump(
   // current RedZone handling code assumes the SP is adjusted by the
   // callee-save save/restore code.
   if (canUseRedZone(MF))
     return false;
 
   return true;
 }
 
+// Given a load or a store instruction, generate an appropriate unwinding SEH
+// code on Windows.
+static MachineBasicBlock::iterator InsertSEH(MachineBasicBlock::iterator MBBI,
+                                             const TargetInstrInfo &TII,
+                                             MachineInstr::MIFlag Flag) {
+  unsigned Opc = MBBI->getOpcode();
+  MachineBasicBlock *MBB = MBBI->getParent();
+  MachineFunction &MF = *MBB->getParent();
+  const AArch64RegisterInfo *RegInfo = static_cast<const AArch64RegisterInfo *>(
+      MF.getSubtarget().getRegisterInfo());
+  DebugLoc DL = MBBI->getDebugLoc();
+  unsigned ImmIdx = MBBI->getNumOperands() - 1;
+  int Imm = MBBI->getOperand(ImmIdx).getImm();
+  // All offsets are scaled, so account for it.
+  Imm *= 8;
+  MachineInstrBuilder MIB;
+
+  switch (Opc) {
+  default:
+    assert(false && "No SEH Opcode for this instruction");
+  case AArch64::LDPDpost:
+    Imm = -Imm;
+  case AArch64::STPDpre: {
+    unsigned Reg0 = MBBI->getOperand(1).getReg() - AArch64::D0;
+    unsigned Reg1 = MBBI->getOperand(2).getReg() - AArch64::D0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFRegP_X))
+              .addImm(Reg0 - AArch64::D0)
+              .addImm(Reg1 - AArch64::D0)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::LDPXpost:
+    Imm = -Imm;
+  case AArch64::STPXpre: {
+    unsigned Reg0 = MBBI->getOperand(1).getReg();
+    unsigned Reg1 = MBBI->getOperand(2).getReg();
+    if (Reg0 == AArch64::FP && Reg1 == AArch64::LR)
+      MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFPLR_X))
+                .addImm(Imm)
+                .setMIFlag(Flag);
+    else
+      MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveRegP_X))
+                .addImm(Reg0 - AArch64::X0)
+                .addImm(Reg1 - AArch64::X0)
+                .addImm(Imm)
+                .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::LDRDpost:
+    Imm = -Imm;
+  case AArch64::STRDpre: {
+    unsigned Reg = MBBI->getOperand(1).getReg() - AArch64::D0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFReg_X))
+              .addImm(Reg)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::LDRXpost:
+    Imm = -Imm;
+  case AArch64::STRXpre: {
+    unsigned Reg = MBBI->getOperand(1).getReg() - AArch64::X0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveReg_X))
+              .addImm(Reg)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::STPDi:
+  case AArch64::LDPDi: {
+    unsigned Reg0 = MBBI->getOperand(0).getReg() - AArch64::D0;
+    unsigned Reg1 = MBBI->getOperand(1).getReg() - AArch64::D0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFRegP))
+              .addImm(Reg0)
+              .addImm(Reg1)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::STPXi:
+  case AArch64::LDPXi: {
+    unsigned Reg0 = MBBI->getOperand(0).getReg();
+    unsigned Reg1 = MBBI->getOperand(1).getReg();
+    if (MBBI->getOperand(0).getReg() == AArch64::FP &&
+        MBBI->getOperand(1).getReg() == AArch64::LR)
+      MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFPLR))
+                .addImm(Imm)
+                .setMIFlag(Flag);
+    else
+      MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveRegP))
+                .addImm(Reg0 - AArch64::X0)
+                .addImm(Reg1 - AArch64::X0)
+                .addImm(Imm)
+                .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::STRXui:
+  case AArch64::LDRXui: {
+    unsigned Reg = MBBI->getOperand(0).getReg() - AArch64::X0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveReg))
+              .addImm(Reg)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  case AArch64::STRDui:
+  case AArch64::LDRDui: {
+    unsigned Reg = MBBI->getOperand(0).getReg() - AArch64::D0;
+    MIB = BuildMI(MF, DL, TII.get(AArch64::SEH_SaveFReg))
+              .addImm(Reg)
+              .addImm(Imm)
+              .setMIFlag(Flag);
+    break;
+  }
+  }
+  auto I = MBB->insertAfter(MBBI, MIB);
+  return I;
+}
+
+// Fix up the SEH opcode associated with the save/restore instruction.
+static void fixupSEHOpcode(MachineBasicBlock::iterator MBBI,
+                           unsigned LocalStackSize) {
+  MachineOperand *ImmOpnd = nullptr;
+  unsigned ImmIdx = MBBI->getNumOperands() - 1;
+  switch (MBBI->getOpcode()) {
+  default:
+    assert(false && "Fix the offset in the SEH instruction");
+    break;
+  case AArch64::SEH_SaveFPLR:
+  case AArch64::SEH_SaveRegP:
+  case AArch64::SEH_SaveReg:
+  case AArch64::SEH_SaveFRegP:
+  case AArch64::SEH_SaveFReg:
+    ImmOpnd = &MBBI->getOperand(ImmIdx);
+    break;
+  }
+  if (ImmOpnd)
+    ImmOpnd->setImm(ImmOpnd->getImm() + LocalStackSize);
+}
+
 // Convert callee-save register save/restore instruction to do stack pointer
 // decrement/increment to allocate/deallocate the callee-save stack area by
 // converting store/load to use pre/post increment version.
 static MachineBasicBlock::iterator convertCalleeSaveRestoreToSPPrePostIncDec(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
-    const DebugLoc &DL, const TargetInstrInfo *TII, int CSStackSizeInc) {
+    const DebugLoc &DL, const TargetInstrInfo *TII, int CSStackSizeInc,
+    bool NeedsWinCFI, bool InProlog = true) {
   // Ignore instructions that do not operate on SP, i.e. shadow call stack
   // instructions.
   while (MBBI->getOpcode() == AArch64::STRXpost ||
          MBBI->getOpcode() == AArch64::LDRXpre) {
     assert(MBBI->getOperand(0).getReg() != AArch64::SP);
     ++MBBI;
   }
 
case AArch64::LDRXui:
     NewIsUnscaled = true;
     break;
   case AArch64::LDRDui:
     NewOpc = AArch64::LDRDpost;
     NewIsUnscaled = true;
     break;
   }
 
+  // Get rid of the SEH code associated with the old instruction.
+  if (NeedsWinCFI) {
+    auto SEH = std::next(MBBI);
+    if (AArch64InstrInfo::isSEHInstruction(*SEH))
+      SEH->eraseFromParent();
+  }
+
   MachineInstrBuilder MIB = BuildMI(MBB, MBBI, DL, TII->get(NewOpc));
   MIB.addReg(AArch64::SP, RegState::Define);
 
   // Copy all operands other than the immediate offset.
   unsigned OpndIdx = 0;
   for (unsigned OpndEnd = MBBI->getNumOperands() - 1; OpndIdx < OpndEnd;
        ++OpndIdx)
     MIB.add(MBBI->getOperand(OpndIdx));
 
   assert(MBBI->getOperand(OpndIdx).getImm() == 0 &&
          "Unexpected immediate offset in first/last callee-save save/restore "
          "instruction!");
   assert(MBBI->getOperand(OpndIdx - 1).getReg() == AArch64::SP &&
          "Unexpected base register in callee-save save/restore instruction!");
   // Last operand is immediate offset that needs fixing.
   assert(CSStackSizeInc % 8 == 0);
   int64_t CSStackSizeIncImm = CSStackSizeInc;
   if (!NewIsUnscaled)
     CSStackSizeIncImm /= 8;
   MIB.addImm(CSStackSizeIncImm);
 
   MIB.setMIFlags(MBBI->getFlags());
   MIB.setMemRefs(MBBI->memoperands());
 
+  // Generate a new SEH code that corresponds to the new instruction.
+  if (NeedsWinCFI)
+    InsertSEH(*MIB, *TII,
+              InProlog ? MachineInstr::FrameSetup : MachineInstr::FrameDestroy);
+
   return std::prev(MBB.erase(MBBI));
 }
 
 // Fixup callee-save register save/restore instructions to take into account
 // combined SP bump by adding the local stack size to the stack offsets.
 static void fixupCalleeSaveRestoreStackOffset(MachineInstr &MI,
-                                              unsigned LocalStackSize) {
+                                              unsigned LocalStackSize,
+                                              bool NeedsWinCFI) {
+  if (AArch64InstrInfo::isSEHInstruction(MI))
+    return;
+
   unsigned Opc = MI.getOpcode();
 
   // Ignore instructions that do not operate on SP, i.e. shadow call stack
   // instructions.
   if (Opc == AArch64::STRXpost || Opc == AArch64::LDRXpre) {
     assert(MI.getOperand(0).getReg() != AArch64::SP);
     return;
   }
 
   (void)Opc;
   assert((Opc == AArch64::STPXi || Opc == AArch64::STPDi ||
           Opc == AArch64::STRXui || Opc == AArch64::STRDui ||
           Opc == AArch64::LDPXi || Opc == AArch64::LDPDi ||
           Opc == AArch64::LDRXui || Opc == AArch64::LDRDui) &&
          "Unexpected callee-save save/restore opcode!");
 
   unsigned OffsetIdx = MI.getNumExplicitOperands() - 1;
   assert(MI.getOperand(OffsetIdx - 1).getReg() == AArch64::SP &&
          "Unexpected base register in callee-save save/restore instruction!");
   // Last operand is immediate offset that needs fixing.
   MachineOperand &OffsetOpnd = MI.getOperand(OffsetIdx);
   // All generated opcodes have scaled offsets.
   assert(LocalStackSize % 8 == 0);
   OffsetOpnd.setImm(OffsetOpnd.getImm() + LocalStackSize / 8);
+
+  if (NeedsWinCFI) {
+    auto MBBI = std::next(MachineBasicBlock::iterator(MI));
+    assert(MBBI != MI.getParent()->end() && "Expecting a valid instruction");
+    assert(AArch64InstrInfo::isSEHInstruction(*MBBI) &&
+           "Expecting a SEH instruction");
+    fixupSEHOpcode(MBBI, LocalStackSize);
+  }
 }
 
 static void adaptForLdStOpt(MachineBasicBlock &MBB,
                             MachineBasicBlock::iterator FirstSPPopI,
                             MachineBasicBlock::iterator LastPopI) {
   // Sometimes (when we restore in the same order as we save), we can end up
   // with code like this:
   //
void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
   const Function &F = MF.getFunction();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   MachineModuleInfo &MMI = MF.getMMI();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   bool needsFrameMoves = MMI.hasDebugInfo() || F.needsUnwindTableEntry();
   bool HasFP = hasFP(MF);
+  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+                     F.needsUnwindTableEntry();
+  MF.setHasWinCFI(NeedsWinCFI);
 
   // At this point, we're going to decide whether or not the function uses a
   // redzone. In most cases, the function doesn't have a redzone so let's
   // assume that's false and set it to true in the case that there's a redzone.
   AFI->setHasRedZone(false);
 
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
 
   if (ShouldSignReturnAddress(MF)) {
     BuildMI(MBB, MBBI, DL, TII->get(AArch64::PACIASP))
         .setMIFlag(MachineInstr::FrameSetup);
   }
 
+  // The very last FrameSetup instruction indicates the end of prologue. Emit a
+  // SEH opcode indicating the prologue end.
+  if (NeedsWinCFI) {
+    MachineBasicBlock::iterator LastFrameSetupI = MBB.begin();
+    while (LastFrameSetupI != MBB.end() &&
+           LastFrameSetupI->getFlag(MachineInstr::FrameSetup))
+      ++LastFrameSetupI;
+    DebugLoc NewDL =
+        (MBB.end() == LastFrameSetupI) ? DL : LastFrameSetupI->getDebugLoc();
+    BuildMI(MBB, LastFrameSetupI, DL, TII->get(AArch64::SEH_PrologEnd))
+        .setMIFlag(MachineInstr::FrameSetup);
+  }
+
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   int NumBytes = (int)MFI.getStackSize();
   if (!AFI->hasStackFrame() && !windowsRequiresStackProbe(MF, NumBytes)) {
     assert(!HasFP && "unexpected function without stack frame but with FP");
-
     // All of the stack allocation is for locals.
     AFI->setLocalStackSize(NumBytes);
 
     if (!NumBytes)
       return;
     // REDZONE: If the stack size is less than 128 bytes, we don't need
     // to actually allocate.
     if (canUseRedZone(MF)) {
       AFI->setHasRedZone(true);
       ++NumRedZoneFunctions;
     } else {
       emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
-                      MachineInstr::FrameSetup);
+                      MachineInstr::FrameSetup, false, NeedsWinCFI);
 
       // Label used to tie together the PROLOG_LABEL and the MachineMoves.
       MCSymbol *FrameLabel = MMI.getContext().createTempSymbol();
       // Encode the stack size of the leaf function.
       unsigned CFIIndex = MF.addFrameInst(
           MCCFIInstruction::createDefCfaOffset(FrameLabel, -NumBytes));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex)
           .setMIFlags(MachineInstr::FrameSetup);
     }
     return;
   }
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
 
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   // All of the remaining stack allocations are for locals.
   AFI->setLocalStackSize(NumBytes - PrologueSaveSize);
-
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   if (CombineSPBump) {
     emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
-                    MachineInstr::FrameSetup);
+                    MachineInstr::FrameSetup, false, NeedsWinCFI);
     NumBytes = 0;
   } else if (PrologueSaveSize != 0) {
-    MBBI = convertCalleeSaveRestoreToSPPrePostIncDec(MBB, MBBI, DL, TII,
-                                                     -PrologueSaveSize);
+    MBBI = convertCalleeSaveRestoreToSPPrePostIncDec(
+        MBB, MBBI, DL, TII, -PrologueSaveSize, NeedsWinCFI);
     NumBytes -= PrologueSaveSize;
   }
   assert(NumBytes >= 0 && "Negative stack allocation size!?");
 
   // Move past the saves of the callee-saved registers, fixing up the offsets
   // and pre-inc if we decided to combine the callee-save and local stack
   // pointer bump above.
   MachineBasicBlock::iterator End = MBB.end();
   while (MBBI != End && MBBI->getFlag(MachineInstr::FrameSetup)) {
     if (CombineSPBump)
-      fixupCalleeSaveRestoreStackOffset(*MBBI, AFI->getLocalStackSize());
+      fixupCalleeSaveRestoreStackOffset(*MBBI, AFI->getLocalStackSize(),
+                                        NeedsWinCFI);
     ++MBBI;
   }
+
   if (HasFP) {
     // Only set up FP if we actually need to. Frame pointer is fp =
     // sp - fixedobject - 16.
     int FPOffset = AFI->getCalleeSavedStackSize() - 16;
     if (CombineSPBump)
       FPOffset += AFI->getLocalStackSize();
 
     // Issue    sub fp, sp, FPOffset or
     //          mov fp,sp          when FPOffset is zero.
     // Note: All stores of callee-saved registers are marked as "FrameSetup".
     // This code marks the instruction(s) that set the FP also.
     emitFrameOffset(MBB, MBBI, DL, AArch64::FP, AArch64::SP, FPOffset, TII,
-                    MachineInstr::FrameSetup);
+                    MachineInstr::FrameSetup, false, NeedsWinCFI);
   }
 
   if (windowsRequiresStackProbe(MF, NumBytes)) {
     uint32_t NumWords = NumBytes >> 4;
 
     BuildMI(MBB, MBBI, DL, TII->get(AArch64::MOVi64imm), AArch64::X15)
         .addImm(NumWords)
         .setMIFlags(MachineInstr::FrameSetup);
 
     switch (MF.getTarget().getCodeModel()) {
     case CodeModel::Tiny:
     case CodeModel::Small:
     case CodeModel::Medium:
     case CodeModel::Kernel:
if (NumBytes) {
     }
 
     // If we're a leaf function, try using the red zone.
     if (!canUseRedZone(MF))
       // FIXME: in the case of dynamic re-alignment, NumBytes doesn't have
       // the correct value here, as NumBytes also includes padding bytes,
       // which shouldn't be counted here.
       emitFrameOffset(MBB, MBBI, DL, scratchSPReg, AArch64::SP, -NumBytes, TII,
-                      MachineInstr::FrameSetup);
+                      MachineInstr::FrameSetup, false, NeedsWinCFI);
 
     if (NeedsRealignment) {
       const unsigned Alignment = MFI.getMaxAlignment();
       const unsigned NrBitsToZero = countTrailingZeros(Alignment);
       assert(NrBitsToZero > 1);
       assert(scratchSPReg != AArch64::SP);
 
       // SUB X9, SP, NumBytes
       //   -- X9 is temporary register, so shouldn't contain any live data here,
       //   -- free to use. This is already produced by emitFrameOffset above.
       // AND SP, X9, 0b11111...0000
       // The logical immediates have a non-trivial encoding. The following
       // formula computes the encoded immediate with all ones but
       // NrBitsToZero zero bits as least significant bits.
       uint32_t andMaskEncoded = (1 << 12)                         // = N
                                 | ((64 - NrBitsToZero) << 6)      // immr
                                 | ((64 - NrBitsToZero - 1) << 0); // imms
 
       BuildMI(MBB, MBBI, DL, TII->get(AArch64::ANDXri), AArch64::SP)
           .addReg(scratchSPReg, RegState::Kill)
           .addImm(andMaskEncoded);
       AFI->setStackRealigned(true);
+      if (NeedsWinCFI)
+        BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_StackAlloc))
+            .addImm(NumBytes & andMaskEncoded)
+            .setMIFlag(MachineInstr::FrameSetup);
     }
   }
 
   // If we need a base pointer, set it up here. It's whatever the value of the
   // stack pointer is at this point. Any variable size objects will be allocated
   // after this, so we can still use the base pointer to reference locals.
   //
   // FIXME: Clarify FrameSetup flags here.
   // Note: Use emitFrameOffset() like above for FP if the FrameSetup flag is
   // needed.
   if (RegInfo->hasBasePointer(MF)) {
     TII->copyPhysReg(MBB, MBBI, DL, RegInfo->getBaseRegister(), AArch64::SP,
                      false);
+    if (NeedsWinCFI)
+      BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_Nop))
+          .setMIFlag(MachineInstr::FrameSetup);
   }
 
   if (needsFrameMoves) {
     const DataLayout &TD = MF.getDataLayout();
     const int StackGrowth = -TD.getPointerSize(0);
     unsigned FramePtr = RegInfo->getFrameRegister(MF);
     // An example of the prologue:
     //

 void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
   DebugLoc DL;
   bool IsTailCallReturn = false;
+  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+                     MF.getFunction().needsUnwindTableEntry();
+
   if (MBB.end() != MBBI) {
     DL = MBBI->getDebugLoc();
     unsigned RetOpcode = MBBI->getOpcode();
     IsTailCallReturn = RetOpcode == AArch64::TCRETURNdi ||
       RetOpcode == AArch64::TCRETURNri;
   }
+
+  if (NeedsWinCFI)
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_EpilogEnd))
+        .setMIFlag(MachineInstr::FrameDestroy);
+
   int NumBytes = MFI.getStackSize();
-  const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
+  AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction().getCallingConv() == CallingConv::GHC)
     return;
 
   // Initial and residual are named for consistency with the prologue. Note that
   // in the epilogue, the residual adjustment is executed first.
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   auto Cleanup = make_scope_exit([&] { InsertReturnAddressAuth(MF, MBB); });
 
   bool IsWin64 =
       Subtarget.isCallingConvWin64(MF.getFunction().getCallingConv());
   unsigned FixedObject = IsWin64 ? alignTo(AFI->getVarArgsGPRSize(), 16) : 0;
 
   uint64_t AfterCSRPopSize = ArgumentPopSize;
   auto PrologueSaveSize = AFI->getCalleeSavedStackSize() + FixedObject;
   bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
   // Assume we can't combine the last pop with the sp restore.
 
   if (!CombineSPBump && PrologueSaveSize != 0) {
     MachineBasicBlock::iterator Pop = std::prev(MBB.getFirstTerminator());
+    while (AArch64InstrInfo::isSEHInstruction(*Pop))
+      Pop = std::prev(Pop);
     // Converting the last ldp to a post-index ldp is valid only if the last
     // ldp's offset is 0.
     const MachineOperand &OffsetOp = Pop->getOperand(Pop->getNumOperands() - 1);
     // If the offset is 0, convert it to a post-index ldp.
     if (OffsetOp.getImm() == 0) {
-      convertCalleeSaveRestoreToSPPrePostIncDec(MBB, Pop, DL, TII,
-                                                PrologueSaveSize);
+      convertCalleeSaveRestoreToSPPrePostIncDec(
+          MBB, Pop, DL, TII, PrologueSaveSize, NeedsWinCFI, false);
     } else {
       // If not, make sure to emit an add after the last ldp.
       // We're doing this by transfering the size to be restored from the
       // adjustment *before* the CSR pops to the adjustment *after* the CSR
       // pops.
       AfterCSRPopSize += PrologueSaveSize;
     }
   }
 
   // Move past the restores of the callee-saved registers.
   // If we plan on combining the sp bump of the local stack size and the callee
   // save stack size, we might need to adjust the CSR save and restore offsets.
   MachineBasicBlock::iterator LastPopI = MBB.getFirstTerminator();
   MachineBasicBlock::iterator Begin = MBB.begin();
   while (LastPopI != Begin) {
     --LastPopI;
     if (!LastPopI->getFlag(MachineInstr::FrameDestroy)) {
       ++LastPopI;
       break;
-    } else if (CombineSPBump)
-      fixupCalleeSaveRestoreStackOffset(*LastPopI, AFI->getLocalStackSize());
+    } else if (CombineSPBump && !AArch64InstrInfo::isSEHInstruction(*LastPopI))
+      fixupCalleeSaveRestoreStackOffset(*LastPopI, AFI->getLocalStackSize(),
+                                        NeedsWinCFI);
   }
 
+  if (NeedsWinCFI)
+    BuildMI(MBB, LastPopI, DL, TII->get(AArch64::SEH_EpilogStart))
+        .setMIFlag(MachineInstr::FrameDestroy);
+
   // If there is a single SP update, insert it before the ret and we're done.
   if (CombineSPBump) {
     emitFrameOffset(MBB, MBB.getFirstTerminator(), DL, AArch64::SP, AArch64::SP,
-                    NumBytes + AfterCSRPopSize, TII,
-                    MachineInstr::FrameDestroy);
+                    NumBytes + AfterCSRPopSize, TII, MachineInstr::FrameDestroy,
+                    false, NeedsWinCFI);
     return;
   }
 
   NumBytes -= PrologueSaveSize;
   assert(NumBytes >= 0 && "Negative stack allocation size!?");
 
   if (!hasFP(MF)) {
     bool RedZone = canUseRedZone(MF);
if (!hasFP(MF)) {
     // then we're done.
     bool Done = NoCalleeSaveRestore || AfterCSRPopSize == 0;
 
     // If we're done after this, make sure to help the load store optimizer.
     if (Done)
       adaptForLdStOpt(MBB, MBB.getFirstTerminator(), LastPopI);
 
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP,
-                    StackRestoreBytes, TII, MachineInstr::FrameDestroy);
+                    StackRestoreBytes, TII, MachineInstr::FrameDestroy, false,
+                    NeedsWinCFI);
     if (Done)
       return;
 
     NumBytes = 0;
   }
 
   // Restore the original stack pointer.
   // FIXME: Rather than doing the math here, we should instead just use
   // non-post-indexed loads for the restores if we aren't actually going to
   // be able to save any instructions.
   if (MFI.hasVarSizedObjects() || AFI->isStackRealigned())
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::FP,
                     -AFI->getCalleeSavedStackSize() + 16, TII,
-                    MachineInstr::FrameDestroy);
+                    MachineInstr::FrameDestroy, false, NeedsWinCFI);
   else if (NumBytes)
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP, NumBytes, TII,
-                    MachineInstr::FrameDestroy);
+                    MachineInstr::FrameDestroy, false, NeedsWinCFI);
 
   // This must be placed after the callee-save restore code because that code
   // assumes the SP is at the same location as it was after the callee-save save
   // code in the prologue.
   if (AfterCSRPopSize) {
     // Find an insertion point for the first ldp so that it goes before the
     // shadow call stack epilog instruction. This ensures that the restore of
     // lr from x18 is placed after the restore from sp.
     auto FirstSPPopI = MBB.getFirstTerminator();
     while (FirstSPPopI != Begin) {
       auto Prev = std::prev(FirstSPPopI);
       if (Prev->getOpcode() != AArch64::LDRXpre ||
           Prev->getOperand(0).getReg() == AArch64::SP)
         break;
       FirstSPPopI = Prev;
     }
 
     adaptForLdStOpt(MBB, FirstSPPopI, LastPopI);
 
     emitFrameOffset(MBB, FirstSPPopI, DL, AArch64::SP, AArch64::SP,
-                    AfterCSRPopSize, TII, MachineInstr::FrameDestroy);
+                    AfterCSRPopSize, TII, MachineInstr::FrameDestroy, false,
+                    NeedsWinCFI);
   }
 }
 
 /// getFrameIndexReference - Provide a base+offset reference to an FI slot for
 /// debug info.  It's the same as what we use for resolving the code-gen
 /// references for now.  FIXME: This can go wrong when references are
 /// SP-relative and simple call frames aren't used.
 int AArch64FrameLowering::getFrameIndexReference(const MachineFunction &MF,

 static bool produceCompactUnwindFrame(MachineFunction &MF) {
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   AttributeList Attrs = MF.getFunction().getAttributes();
   return Subtarget.isTargetMachO() &&
          !(Subtarget.getTargetLowering()->supportSwiftError() &&
            Attrs.hasAttrSomewhere(Attribute::SwiftError));
 }
 
+static bool invalidateWindowsRegisterPairing(unsigned Reg1, unsigned Reg2,
+                                             bool NeedsWinCFI) {
+  // If we are generating register pairs for a Windows function that requires
+  // EH support, then pair consecutive registers only.  There are no unwind
+  // opcodes for saves/restores of non-consectuve register pairs.
+  // The unwind opcodes are save_regp, save_regp_x, save_fregp, save_frepg_x.
+  // https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling
+  if (!NeedsWinCFI)
+    return false;
+  if (Reg2 == Reg1 + 1)
+    return false;
+  return true;
+}
+
 namespace {
 
 struct RegPairInfo {
   unsigned Reg1 = AArch64::NoRegister;
   unsigned Reg2 = AArch64::NoRegister;
   int FrameIdx;
   int Offset;
   bool IsGPR;
 
   RegPairInfo() = default;
 
   bool isPaired() const { return Reg2 != AArch64::NoRegister; }
 };
 
 } // end anonymous namespace
 
 static void computeCalleeSaveRegisterPairs(
     MachineFunction &MF, const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI, SmallVectorImpl<RegPairInfo> &RegPairs,
     bool &NeedShadowCallStackProlog) {
 
   if (CSI.empty())
     return;
 
+  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+                     MF.getFunction().needsUnwindTableEntry();
   AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   CallingConv::ID CC = MF.getFunction().getCallingConv();
   unsigned Count = CSI.size();
   (void)CC;
   // MachO's compact unwind format relies on all registers being stored in
   // pairs.
   assert((!produceCompactUnwindFrame(MF) ||
           CC == CallingConv::PreserveMost ||
           (Count & 1) == 0) &&
          "Odd number of callee-saved regs to spill!");
   int Offset = AFI->getCalleeSavedStackSize();
 
+  // On Win64, we don't necessarily pair all registers or all registers except
+  // possibly one because the unwind opcodes for stack unwinding require that
+  // only consecutive registers are paired.  This flag makes sure that the
+  // padding below is done only once.
+  bool FixupDone = false;
   for (unsigned i = 0; i < Count; ++i) {
     RegPairInfo RPI;
     RPI.Reg1 = CSI[i].getReg();
 
     assert(AArch64::GPR64RegClass.contains(RPI.Reg1) ||
            AArch64::FPR64RegClass.contains(RPI.Reg1));
     RPI.IsGPR = AArch64::GPR64RegClass.contains(RPI.Reg1);
 
-    // Add the next reg to the pair if it is in the same register class.
     if (i + 1 < Count) {
       unsigned NextReg = CSI[i + 1].getReg();
-      if ((RPI.IsGPR && AArch64::GPR64RegClass.contains(NextReg)) ||
-          (!RPI.IsGPR && AArch64::FPR64RegClass.contains(NextReg)))
+      if (((RPI.IsGPR && AArch64::GPR64RegClass.contains(NextReg)) ||
+          (!RPI.IsGPR && AArch64::FPR64RegClass.contains(NextReg))) &&
+          !invalidateWindowsRegisterPairing(RPI.Reg1, NextReg, NeedsWinCFI))
         RPI.Reg2 = NextReg;
     }
 
     // If either of the registers to be saved is the lr register, it means that
     // we also need to save lr in the shadow call stack.
     if ((RPI.Reg1 == AArch64::LR || RPI.Reg2 == AArch64::LR) &&
         MF.getFunction().hasFnAttribute(Attribute::ShadowCallStack)) {
       if (!MF.getSubtarget<AArch64Subtarget>().isX18Reserved())
assert((!produceCompactUnwindFrame(MF) ||
             CC == CallingConv::PreserveMost ||
             (RPI.isPaired() &&
              ((RPI.Reg1 == AArch64::LR && RPI.Reg2 == AArch64::FP) ||
               RPI.Reg1 + 1 == RPI.Reg2))) &&
            "Callee-save registers not saved as adjacent register pair!");
 
     RPI.FrameIdx = CSI[i].getFrameIdx();
 
-    if (Count * 8 != AFI->getCalleeSavedStackSize() && !RPI.isPaired()) {
+    if (Count * 8 != AFI->getCalleeSavedStackSize() && !RPI.isPaired() &&
+        !FixupDone) {
       // Round up size of non-pair to pair size if we need to pad the
       // callee-save area to ensure 16-byte alignment.
+      FixupDone = NeedsWinCFI;
       Offset -= 16;
       assert(MFI.getObjectAlignment(RPI.FrameIdx) <= 16);
       MFI.setObjectAlignment(RPI.FrameIdx, 16);
       AFI->setCalleeSaveStackHasFreeSpace(true);
     } else
       Offset -= RPI.isPaired() ? 16 : 8;
     assert(Offset % 8 == 0);
     RPI.Offset = Offset / 8;
     assert((RPI.Offset >= -64 && RPI.Offset <= 63) &&
            "Offset out of bounds for LDP/STP immediate");
 
     RegPairs.push_back(RPI);
     if (RPI.isPaired())
       ++i;
   }
 }
 
 bool AArch64FrameLowering::spillCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
+  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+                     MF.getFunction().needsUnwindTableEntry();
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
   const MachineRegisterInfo &MRI = MF.getRegInfo();
 
for (auto RPII = RegPairs.rbegin(), RPIE = RegPairs.rend(); RPII != RPIE;
     else
       StrOpc = RPI.isPaired() ? AArch64::STPDi : AArch64::STRDui;
     LLVM_DEBUG(dbgs() << "CSR spill: (" << printReg(Reg1, TRI);
                if (RPI.isPaired()) dbgs() << ", " << printReg(Reg2, TRI);
                dbgs() << ") -> fi#(" << RPI.FrameIdx;
                if (RPI.isPaired()) dbgs() << ", " << RPI.FrameIdx + 1;
                dbgs() << ")\n");
 
+    assert((!NeedsWinCFI || !(Reg1 == AArch64::LR && Reg2 == AArch64::FP)) &&
+           "Windows unwdinding requires a consecutive (FP,LR) pair");
+
     MachineInstrBuilder MIB = BuildMI(MBB, MI, DL, TII.get(StrOpc));
     if (!MRI.isReserved(Reg1))
       MBB.addLiveIn(Reg1);
+
+    // Windows unwinding codes require that gprs be consecutive if they are paired.
+    if (NeedsWinCFI) {
+      MIB.addReg(Reg1, getPrologueDeath(MF, Reg1));
+      MIB.addMemOperand(MF.getMachineMemOperand(
+        MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
+        MachineMemOperand::MOStore, 8, 8));
+      if (RPI.isPaired()) {
+        if (!MRI.isReserved(Reg2))
+          MBB.addLiveIn(Reg2);
+        MIB.addReg(Reg2, getPrologueDeath(MF, Reg2));
+        MIB.addMemOperand(MF.getMachineMemOperand(
+          MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
+          MachineMemOperand::MOStore, 8, 8));
+      }
+      MIB.addReg(AArch64::SP)
+         .addImm(RPI.Offset) // [sp, #offset*8], where factor*8 is implicit
+         .setMIFlag(MachineInstr::FrameSetup);
+      InsertSEH(MIB, TII, MachineInstr::FrameSetup);
+    } else {
       if (RPI.isPaired()) {
         if (!MRI.isReserved(Reg2))
           MBB.addLiveIn(Reg2);
         MIB.addReg(Reg2, getPrologueDeath(MF, Reg2));
         MIB.addMemOperand(MF.getMachineMemOperand(
           MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
           MachineMemOperand::MOStore, 8, 8));
       }
       MIB.addReg(Reg1, getPrologueDeath(MF, Reg1))
           .addReg(AArch64::SP)
           .addImm(RPI.Offset) // [sp, #offset*8], where factor*8 is implicit
           .setMIFlag(MachineInstr::FrameSetup);
       MIB.addMemOperand(MF.getMachineMemOperand(
           MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
           MachineMemOperand::MOStore, 8, 8));
     }
+  }
   return true;
 }
 
 bool AArch64FrameLowering::restoreCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
+  bool NeedsWinCFI = MF.getTarget().getMCAsmInfo()->usesWindowsCFI() &&
+                     MF.getFunction().needsUnwindTableEntry();
 
   if (MI != MBB.end())
     DL = MI->getDebugLoc();
 
   bool NeedShadowCallStackProlog = false;
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs,
                                  NeedShadowCallStackProlog);
 
else
       LdrOpc = RPI.isPaired() ? AArch64::LDPDi : AArch64::LDRDui;
     LLVM_DEBUG(dbgs() << "CSR restore: (" << printReg(Reg1, TRI);
                if (RPI.isPaired()) dbgs() << ", " << printReg(Reg2, TRI);
                dbgs() << ") -> fi#(" << RPI.FrameIdx;
                if (RPI.isPaired()) dbgs() << ", " << RPI.FrameIdx + 1;
                dbgs() << ")\n");
 
     MachineInstrBuilder MIB = BuildMI(MBB, MI, DL, TII.get(LdrOpc));
+
+    if (NeedsWinCFI) {
+      MIB.addReg(Reg1, getDefRegState(true));
+      MIB.addMemOperand(MF.getMachineMemOperand(
+          MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
+          MachineMemOperand::MOLoad, 8, 8));
+      if (RPI.isPaired()) {
+        MIB.addReg(Reg2, getDefRegState(true));
+        MIB.addMemOperand(MF.getMachineMemOperand(
+            MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
+            MachineMemOperand::MOLoad, 8, 8));
+      }
+      MIB.addReg(AArch64::SP)
+         .addImm(RPI.Offset) // [sp, #offset*8] where the factor*8 is implicit
+         .setMIFlag(MachineInstr::FrameDestroy);
+      InsertSEH(MIB, TII, MachineInstr::FrameDestroy);
+    } else {
       if (RPI.isPaired()) {
         MIB.addReg(Reg2, getDefRegState(true));
         MIB.addMemOperand(MF.getMachineMemOperand(
             MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
             MachineMemOperand::MOLoad, 8, 8));
       }
       MIB.addReg(Reg1, getDefRegState(true))
          .addReg(AArch64::SP)
          .addImm(RPI.Offset) // [sp, #offset*8] where the factor*8 is implicit
          .setMIFlag(MachineInstr::FrameDestroy);
       MIB.addMemOperand(MF.getMachineMemOperand(
           MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
           MachineMemOperand::MOLoad, 8, 8));
+    }
   };
 
   if (ReverseCSRRestoreSeq)
     for (const RegPairInfo &RPI : reverse(RegPairs))
       EmitMI(RPI);
   else
     for (const RegPairInfo &RPI : RegPairs)
       EmitMI(RPI);
void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,
   // instructions.
   AFI->setCalleeSavedStackSize(alignTo(8 * NumRegsSpilled, 16));
 }
 
 bool AArch64FrameLowering::enableStackSlotScavenging(
     const MachineFunction &MF) const {
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   return AFI->hasCalleeSaveStackFreeSpace();
 }

lib/Target/AArch64/AArch64InstrInfo.h

public:
   /// executed more efficiently.
   bool isExynosResetFast(const MachineInstr &MI) const;
   /// Returns true if the instruction has a shift left that can be executed
   /// more efficiently.
   bool isExynosShiftLeftFast(const MachineInstr &MI) const;
   /// Returns true if the instruction has a shift by immediate that can be
   /// executed in one cycle less.
   bool isFalkorShiftExtFast(const MachineInstr &MI) const;
+  /// Return true if the instructions is a SEH instruciton used for unwinding
+  /// on Windows.
+  static bool isSEHInstruction(const MachineInstr &MI);
 
 private:
   /// Sets the offsets on outlined instructions in \p MBB which use SP
   /// so that they will be valid post-outlining.
   ///
   /// \param MBB A \p MachineBasicBlock in an outlined function.
   void fixupPostOutline(MachineBasicBlock &MBB) const;
 

 /// emitFrameOffset - Emit instructions as needed to set DestReg to SrcReg
 /// plus Offset.  This is intended to be used from within the prolog/epilog
 /// insertion (PEI) pass, where a virtual scratch register may be allocated
 /// if necessary, to be replaced by the scavenger at the end of PEI.
 void emitFrameOffset(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
                      const DebugLoc &DL, unsigned DestReg, unsigned SrcReg,
                      int Offset, const TargetInstrInfo *TII,
                      MachineInstr::MIFlag = MachineInstr::NoFlags,
-                     bool SetNZCV = false);
+                     bool SetNZCV = false, bool NeedsWinCFI = false);
 
 /// rewriteAArch64FrameIndex - Rewrite MI to access 'Offset' bytes from the
 /// FP. Return false if the offset could not be handled directly in MI, and
 /// return the left-over portion by reference.
 bool rewriteAArch64FrameIndex(MachineInstr &MI, unsigned FrameRegIdx,
                               unsigned FrameReg, int &Offset,
                               const AArch64InstrInfo *TII);
 

lib/Target/AArch64/AArch64InstrInfo.cpp

bool AArch64InstrInfo::isFalkorShiftExtFast(const MachineInstr &MI) const {
   case AArch64::STRXroW:
   case AArch64::STRXroX: {
     unsigned IsSigned = MI.getOperand(3).getImm();
     return !IsSigned;
   }
   }
 }
 
+bool AArch64InstrInfo::isSEHInstruction(const MachineInstr &MI) {
+  unsigned Opc = MI.getOpcode();
+  switch (Opc) {
+    default:
+      return false;
+    case AArch64::SEH_StackAlloc:
+    case AArch64::SEH_SaveFPLR:
+    case AArch64::SEH_SaveFPLR_X:
+    case AArch64::SEH_SaveReg:
+    case AArch64::SEH_SaveReg_X:
+    case AArch64::SEH_SaveRegP:
+    case AArch64::SEH_SaveRegP_X:
+    case AArch64::SEH_SaveFReg:
+    case AArch64::SEH_SaveFReg_X:
+    case AArch64::SEH_SaveFRegP:
+    case AArch64::SEH_SaveFRegP_X:
+    case AArch64::SEH_SetFP:
+    case AArch64::SEH_AddFP:
+    case AArch64::SEH_Nop:
+    case AArch64::SEH_PrologEnd:
+    case AArch64::SEH_EpilogStart:
+    case AArch64::SEH_EpilogEnd:
+      return true;
+  }
+}
+
 bool AArch64InstrInfo::isCoalescableExtInstr(const MachineInstr &MI,
                                              unsigned &SrcReg, unsigned &DstReg,
                                              unsigned &SubIdx) const {
   switch (MI.getOpcode()) {
   default:
     return false;
   case AArch64::SBFMXri: // aka sxtw
   case AArch64::UBFMXri: // aka uxtw
if (Offset)
     MI.addImm(0);
   MI.addMemOperand(MMO);
 }
 
 void llvm::emitFrameOffset(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI, const DebugLoc &DL,
                            unsigned DestReg, unsigned SrcReg, int Offset,
                            const TargetInstrInfo *TII,
-                           MachineInstr::MIFlag Flag, bool SetNZCV) {
+                           MachineInstr::MIFlag Flag, bool SetNZCV,
+                           bool NeedsWinCFI) {
   if (DestReg == SrcReg && Offset == 0)
     return;
 
   assert((DestReg != AArch64::SP || Offset % 16 == 0) &&
          "SP increment/decrement not 16-byte aligned");
 
   bool isSub = Offset < 0;
   if (isSub)
while (((unsigned)Offset) >= (1 << ShiftSize)) {
     if (Offset == 0)
       return;
   }
   BuildMI(MBB, MBBI, DL, TII->get(Opc), DestReg)
       .addReg(SrcReg)
       .addImm(Offset)
       .addImm(AArch64_AM::getShifterImm(AArch64_AM::LSL, 0))
       .setMIFlag(Flag);
+
+  if (NeedsWinCFI) {
+    if ((DestReg == AArch64::FP && SrcReg == AArch64::SP) ||
+        (SrcReg == AArch64::FP && DestReg == AArch64::SP)) {
+      if (Offset == 0)
+        BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_SetFP)).
+                setMIFlag(Flag);
+      else
+        BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_AddFP)).
+                addImm(Offset).setMIFlag(Flag);
+    } else if (DestReg == AArch64::SP) {
+      BuildMI(MBB, MBBI, DL, TII->get(AArch64::SEH_StackAlloc)).
+              addImm(Offset).setMIFlag(Flag);
+    }
+  }
 }
 
 MachineInstr *AArch64InstrInfo::foldMemoryOperandImpl(
     MachineFunction &MF, MachineInstr &MI, ArrayRef<unsigned> Ops,
     MachineBasicBlock::iterator InsertPt, int FrameIndex,
     LiveIntervals *LIS) const {
   // This is a bit of a hack. Consider this instruction:
   //

lib/Target/AArch64/AArch64RegisterInfo.cpp

 AArch64RegisterInfo::AArch64RegisterInfo(const Triple &TT)
     : AArch64GenRegisterInfo(AArch64::LR), TT(TT) {
   AArch64_MC::initLLVMToCVRegMapping(this);
 }
 
 const MCPhysReg *
 AArch64RegisterInfo::getCalleeSavedRegs(const MachineFunction *MF) const {
   assert(MF && "Invalid MachineFunction pointer.");
+  if (MF->getSubtarget<AArch64Subtarget>().isTargetWindows())
+    return CSR_Win_AArch64_AAPCS_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::GHC)
     // GHC set of callee saved regs is empty as all those regs are
     // used for passing STG regs around
     return CSR_AArch64_NoRegs_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::AnyReg)
     return CSR_AArch64_AllRegs_SaveList;
   if (MF->getFunction().getCallingConv() == CallingConv::CXX_FAST_TLS)
     return MF->getInfo<AArch64FunctionInfo>()->isSplitCSR() ?

test/CodeGen/AArch64/win64_vararg.ll

 declare void @llvm.lifetime.start.p0i8(i64, i8* nocapture) #1
 declare void @llvm.lifetime.end.p0i8(i64, i8* nocapture) #1
 
 declare i32 @__stdio_common_vsprintf(i64, i8*, i64, i8*, i8*, i8*) local_unnamed_addr #3
 declare i64* @__local_stdio_printf_options() local_unnamed_addr #4
 
 ; CHECK-LABEL: fp
 ; CHECK: str     x21, [sp, #-96]!
-; CHECK: stp     x20, x19, [sp, #16]
+; CHECK: stp     x19, x20, [sp, #16]
 ; CHECK: stp     x29, x30, [sp, #32]
 ; CHECK: add     x29, sp, #32
 ; CHECK: add     x8, x29, #24
 ; CHECK: mov     x19, x2
 ; CHECK: mov     x20, x1
 ; CHECK: mov     x21, x0
 ; CHECK: stp     x6, x7, [x29, #48]
 ; CHECK: stp     x4, x5, [x29, #32]
 ; CHECK: str     x3, [x29, #24]
 ; CHECK: str     x8, [sp, #8]
 ; CHECK: bl      __local_stdio_printf_options
 ; CHECK: ldr     x8, [x0]
 ; CHECK: add     x5, x29, #24
 ; CHECK: mov     x1, x21
 ; CHECK: mov     x2, x20
 ; CHECK: orr     x0, x8, #0x2
 ; CHECK: mov     x3, x19
 ; CHECK: mov     x4, xzr
 ; CHECK: bl      __stdio_common_vsprintf
 ; CHECK: ldp     x29, x30, [sp, #32]
-; CHECK: ldp     x20, x19, [sp, #16]
+; CHECK: ldp     x19, x20, [sp, #16]
 ; CHECK: cmp     w0, #0
 ; CHECK: csinv   w0, w0, wzr, ge
 ; CHECK: ldr     x21, [sp], #96
 ; CHECK: ret
 define i32 @fp(i8*, i64, i8*, ...) local_unnamed_addr #6 {
   %4 = alloca i8*, align 8
   %5 = bitcast i8** %4 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %5) #2
define i32 @fp(i8*, i64, i8*, ...) local_unnamed_addr #6 {
   call void @llvm.lifetime.end.p0i8(i64 8, i8* nonnull %5) #2
   ret i32 %12
 }
 
 attributes #6 = { "no-frame-pointer-elim"="true" }
 
 ; CHECK-LABEL: vla
 ; CHECK: str     x23, [sp, #-112]!
-; CHECK: stp     x22, x21, [sp, #16]
-; CHECK: stp     x20, x19, [sp, #32]
+; CHECK: stp     x21, x22, [sp, #16]
+; CHECK: stp     x19, x20, [sp, #32]
 ; CHECK: stp     x29, x30, [sp, #48]
 ; CHECK: add     x29, sp, #48
 ; CHECK: add     x8, x29, #16
 ; CHECK: stur    x8, [x29, #-40]
 ; CHECK: mov     w8, w0
 ; CHECK: add     x8, x8, #15
 ; CHECK: lsr     x15, x8, #4
 ; CHECK: mov     x19, x1

 ; CHECK: mov     x3, x19
 ; CHECK: orr     x0, x8, #0x2
 ; CHECK: mov     x4, xzr
 ; CHECK: mov     x5, [[REG3]]
 ; CHECK: bl      __stdio_common_vsprintf
 ; CHECK: mov     sp, [[REG2]]
 ; CHECK: sub     sp, x29, #48
 ; CHECK: ldp     x29, x30, [sp, #48]
-; CHECK: ldp     x20, x19, [sp, #32]
-; CHECK: ldp     x22, x21, [sp, #16]
+; CHECK: ldp     x19, x20, [sp, #32]
+; CHECK: ldp     x21, x22, [sp, #16]
 ; CHECK: ldr     x23, [sp], #112
 ; CHECK: ret
 define void @vla(i32, i8*, ...) local_unnamed_addr {
   %3 = alloca i8*, align 8
   %4 = bitcast i8** %3 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %4) #5
   call void @llvm.va_start(i8* nonnull %4)
   %5 = zext i32 %0 to i64
define void @vla(i32, i8*, ...) local_unnamed_addr {
   ret void
 }
 
 declare i8* @llvm.stacksave()
 declare void @llvm.stackrestore(i8*)
 
 ; CHECK-LABEL: snprintf
 ; CHECK: sub     sp,  sp, #96
-; CHECK: stp     x21, x20, [sp, #16]
-; CHECK: stp     x19, x30, [sp, #32]
+; CHECK: str     x21, [sp, #16]
+; CHECK: stp     x19, x20, [sp, #24]
+; CHECK: str     x30, [sp, #40]
 ; CHECK: add     x8, sp, #56
 ; CHECK: mov     x19, x2
 ; CHECK: mov     x20, x1
 ; CHECK: mov     x21, x0
 ; CHECK: stp     x6, x7, [sp, #80]
 ; CHECK: stp     x4, x5, [sp, #64]
 ; CHECK: str     x3, [sp, #56]
 ; CHECK: str     x8, [sp, #8]
 ; CHECK: bl      __local_stdio_printf_options
 ; CHECK: ldr     x8, [x0]
 ; CHECK: add     x5, sp, #56
 ; CHECK: mov     x1, x21
 ; CHECK: mov     x2, x20
 ; CHECK: orr     x0, x8, #0x2
 ; CHECK: mov     x3, x19
 ; CHECK: mov     x4, xzr
 ; CHECK: bl      __stdio_common_vsprintf
-; CHECK: ldp     x19, x30, [sp, #32]
-; CHECK: ldp     x21, x20, [sp, #16]
+; CHECK: ldr     x30, [sp, #40]
+; CHECK: ldp     x19, x20, [sp, #24]
+; CHECK: ldr     x21, [sp, #16]
 ; CHECK: cmp     w0, #0
 ; CHECK: csinv   w0, w0, wzr, ge
 ; CHECK: add     sp, sp, #96
 ; CHECK: ret
 define i32 @snprintf(i8*, i64, i8*, ...) local_unnamed_addr #5 {
   %4 = alloca i8*, align 8
   %5 = bitcast i8** %4 to i8*
   call void @llvm.lifetime.start.p0i8(i64 8, i8* nonnull %5) #2

test/CodeGen/AArch64/wineh-pei.mir

+# RUN: llc -o - %s -mtriple=aarch64-windows -start-before=prologepilog -stop-after=prologepilog | FileCheck %s
+
+# Test Win64 unwind opcodes generated by the PEI.
+
+# CHECK: frame-setup SEH_SaveRegP_X 27, 28, -80
+# CHECK: frame-setup SEH_SaveRegP 25, 26, 16
+# CHECK: frame-setup SEH_SaveRegP 23, 24, 32
+# CHECK: frame-setup SEH_SaveRegP 21, 22, 48
+# CHECK: frame-setup SEH_SaveRegP 19, 20, 64
+# CHECK: frame-setup SEH_PrologEnd
+# CHECK: frame-destroy SEH_EpilogStart
+# CHECK: frame-destroy SEH_SaveRegP 19, 20, 64
+# CHECK: frame-destroy SEH_SaveRegP 21, 22, 48
+# CHECK: frame-destroy SEH_SaveRegP 23, 24, 32
+# CHECK: frame-destroy SEH_SaveRegP 25, 26, 16
+# CHECK: frame-destroy SEH_SaveRegP_X 27, 28, -80
+# CHECK: frame-destroy SEH_EpilogEnd
+---
+name:            test
+alignment:       2
+tracksRegLiveness: true
+liveins:
+  - { reg: '$w0' }
+frameInfo:
+  maxCallFrameSize: 0
+  hasOpaqueSPAdjustment: true
+body:             |
+  bb.0.entry:
+    liveins: $x0, $x1
+
+    $x19 = ADDXrr $x0, killed $x1
+    $x20 = ADDXrr $x19, killed $x0
+    $x21 = ADDXrr $x20, killed $x19
+    $x22 = ADDXrr $x21, killed $x20
+    $x23 = ADDXrr $x22, killed $x21
+    $x24 = ADDXrr $x23, killed $x22
+    $x25 = ADDXrr $x24, killed $x23
+    $x26 = ADDXrr $x25, killed $x24
+    $x27 = ADDXrr $x26, killed $x25
+    $x28 = ADDXrr $x27, killed $x26
+    $x0 = COPY $x28
+    RET_ReallyLR implicit $x0
+...
+
+# CHECK: frame-setup SEH_SaveRegP_X 27, 28, -64
+# CHECK: frame-setup SEH_SaveRegP 25, 26, 16
+# CHECK: frame-setup SEH_SaveReg 23, 32
+# CHECK: frame-setup SEH_SaveRegP 21, 22, 40
+# CHECK: frame-setup SEH_SaveReg 19, 56
+# CHECK: frame-setup SEH_PrologEnd
+---
+name:            test2
+alignment:       2
+tracksRegLiveness: true
+liveins:
+  - { reg: '$w0' }
+frameInfo:
+  maxCallFrameSize: 0
+  hasOpaqueSPAdjustment: true
+body:             |
+  bb.0.entry:
+    liveins: $x0, $x1
+
+    $x19 = ADDXrr $x0, killed $x1
+    $x21 = ADDXrr $x19, $x19
+    $x22 = ADDXrr $x21, killed $x19
+    $x23 = ADDXrr $x22, killed $x21
+    $x25 = ADDXrr $x23, $x23
+    $x26 = ADDXrr $x25, killed $x23
+    $x27 = ADDXrr $x26, killed $x25
+    $x28 = ADDXrr $x27, killed $x26
+    $x0 = COPY $x28
+    RET_ReallyLR implicit $x0
+...
+