[AArch64] Add asm directives for the remaining SEH unwind codes

[mstorsjo]

Summary

Add support in llvm-readobj for displaying them and support in the asm parsser, AArch64TargetStreamer and MCWin64EH for emitting them.

The directives for the remaining basic opcodes have names that match the opcode in the documentation.

The directives for custom stack cases, that are named MSFT_OP_TRAP_FRAME, MSFT_OP_MACHINE_FRAME, MSFT_OP_CONTEXT and MSFT_OP_CLEAR_UNWOUND_TO_CALL, are given matching assembler directive names that fit into the rest of the opcode naming; .seh_trap_frame, .seh_context, .seh_clear_unwound_to_call

The opcode MSFT_OP_MACHINE_FRAME is mapped to the existing opecode enum UOP_PushMachFrame that is used on x86_64, and also uses the corresponding existing x86_64 directive name .seh_pushframe. TODO: Or should it be given a name that is consistent with the other ones, e.g. .seh_machine_frame?

[efriedma]

Adding assembler support for the basic opcodes we left out because the code generator doesn't use them seems fine. We could teach the assembler to optimize certain constructs: for example, turn save_regp_x into save_r19r20_x if the register matches and the offset is in range. Might be more confusing than helpful, though.

For the custom stack opcodes, do you have some intended use? If we just want to allow people to dump random bytes into the unwind opcode list, we could add a directive for that, without trying to come up with an appropriate name for opcodes that aren't really documented.

[mstorsjo]

In D86889#2248077, @efriedma wrote:

Adding assembler support for the basic opcodes we left out because the code generator doesn't use them seems fine. We could teach the assembler to optimize certain constructs: for example, turn save_regp_x into save_r19r20_x if the register matches and the offset is in range. Might be more confusing than helpful, though.

Yeah, I've thought about that kind of optimizations. One could also use the save_next opcode a bit to shrink out a few bytes here and there.

I'm a bit on the fence about what level to do it at, though. It'd be neat to do it e.g. at the MC/Win64EH.cpp level, but then you'd have a situation where assembler level constructs don't map 1:1 to what actually ends up emitted - I'm not sure if that's an issue in practice though.

But that's for later anyway.

For the custom stack opcodes, do you have some intended use?

Yes, I have an actual usecase for this. Within the wine implementation of ntdll, there's a case where unwinding needs to be diverted. If the wine ntdll is built as an ELF (or MachO), those unwinding rules can be done with dwarf escape codes, but if the wine ntdll is built as a PE, it needs one of these opcodes. The corresponding case for x86_64 was implemented like this recently: https://source.winehq.org/git/wine.git/commitdiff/c76dc32feffaeed260bf73499e43012b69bee1b4 This one used .seh_pushframe there (which is the only option there), while for aarch64 I guess it could either be done with MSFT_OP_CONTEXT or MSFT_OP_MACHINE_FRAME. My tentative implementation of the same uses MSFT_OP_CONTEXT.

If we just want to allow people to dump random bytes into the unwind opcode list, we could add a directive for that, without trying to come up with an appropriate name for opcodes that aren't really documented.

I guess that's a possibility as well. The opcodes here are documented in the sense that they are listed with a name, but their exact behaviour isn't described. Anybody actually using them needs to study their behaviour themselves (which can be done with e.g. RtlVirtualUnwind). I've documented the behaviour of two of them empirically by testcases in https://source.winehq.org/git/wine.git/commitdiff/bc3284f818f70d2de0bc76beaa69a3352f27fd34.

[mstorsjo]

@efriedma Any followup to the discussion above?

Regarding just allowing passing custom opcode bytes through vs defining names - it's only 4 odd ones, so a custom escape mechanism feels a bit overkill, and they at least have names in the docs.

[efriedma]

LGTM

Regarding just allowing passing custom opcode bytes through vs defining names - it's only 4 odd ones, so a custom escape mechanism feels a bit overkill, and they at least have names in the docs.

Fair enough.

llvm/include/llvm/Support/Win64EH.h

enum UnwindOpcodes {
   UOP_Epilog,
   UOP_SpareCode,
   UOP_SaveXMM128,
   UOP_SaveXMM128Big,
   UOP_PushMachFrame,
   // The following set of unwind opcodes is for ARM64.  They are documented at
   // https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling
   UOP_AllocMedium,
+  UOP_SaveR19R20X,
   UOP_SaveFPLRX,
   UOP_SaveFPLR,
   UOP_SaveReg,
   UOP_SaveRegX,
   UOP_SaveRegP,
   UOP_SaveRegPX,
+  UOP_SaveLRPair,
   UOP_SaveFReg,
   UOP_SaveFRegX,
   UOP_SaveFRegP,
   UOP_SaveFRegPX,
   UOP_SetFP,
   UOP_AddFP,
   UOP_Nop,
-  UOP_End
+  UOP_End,
+  UOP_SaveNext,
+  UOP_TrapFrame,
+  UOP_Context,
+  UOP_ClearUnwoundToCall
 };
 
 /// UnwindCode - This union describes a single operation in a function prolog,
 /// or part thereof.
 union UnwindCode {
   struct {
     uint8_t CodeOffset;
     uint8_t UnwindOpAndOpInfo;

llvm/lib/MC/MCWin64EH.cpp

case Win64EH::UOP_AllocSmall:
       Count += 1;
       break;
     case Win64EH::UOP_AllocMedium:
       Count += 2;
       break;
     case Win64EH::UOP_AllocLarge:
       Count += 4;
       break;
+    case Win64EH::UOP_SaveR19R20X:
+      Count += 1;
+      break;
     case Win64EH::UOP_SaveFPLRX:
       Count += 1;
       break;
     case Win64EH::UOP_SaveFPLR:
       Count += 1;
       break;
     case Win64EH::UOP_SaveReg:
       Count += 2;
       break;
     case Win64EH::UOP_SaveRegP:
       Count += 2;
       break;
     case Win64EH::UOP_SaveRegPX:
       Count += 2;
       break;
     case Win64EH::UOP_SaveRegX:
       Count += 2;
       break;
+    case Win64EH::UOP_SaveLRPair:
+      Count += 2;
+      break;
     case Win64EH::UOP_SaveFReg:
       Count += 2;
       break;
     case Win64EH::UOP_SaveFRegP:
       Count += 2;
       break;
     case Win64EH::UOP_SaveFRegX:
       Count += 2;
       break;
     case Win64EH::UOP_SaveFRegPX:
       Count += 2;
       break;
     case Win64EH::UOP_SetFP:
       Count += 1;
       break;
     case Win64EH::UOP_AddFP:
       Count += 2;
       break;
     case Win64EH::UOP_Nop:
       Count += 1;
       break;
     case Win64EH::UOP_End:
       Count += 1;
       break;
+    case Win64EH::UOP_SaveNext:
+      Count += 1;
+      break;
+    case Win64EH::UOP_TrapFrame:
+      Count += 1;
+      break;
+    case Win64EH::UOP_PushMachFrame:
+      Count += 1;
+      break;
+    case Win64EH::UOP_Context:
+      Count += 1;
+      break;
+    case Win64EH::UOP_ClearUnwoundToCall:
+      Count += 1;
+      break;
     }
   }
   return Count;
 }
 
 // Unwind opcode encodings and restrictions are documented at
 // https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling
 static void ARM64EmitUnwindCode(MCStreamer &streamer, const MCSymbol *begin,
case Win64EH::UOP_AddFP:
     streamer.emitInt8(b);
     b = (inst.Offset >> 3);
     streamer.emitInt8(b);
     break;
   case Win64EH::UOP_Nop:
     b = 0xE3;
     streamer.emitInt8(b);
     break;
+  case Win64EH::UOP_SaveR19R20X:
+    b = 0x20;
+    b |= (inst.Offset >> 3) & 0x1F;
+    streamer.emitInt8(b);
+    break;
   case Win64EH::UOP_SaveFPLRX:
     b = 0x80;
     b |= ((inst.Offset - 1) >> 3) & 0x3F;
     streamer.emitInt8(b);
     break;
   case Win64EH::UOP_SaveFPLR:
     b = 0x40;
     b |= (inst.Offset >> 3) & 0x3F;
static void ARM64EmitUnwindCode(MCStreamer &streamer, const MCSymbol *begin,
   case Win64EH::UOP_SaveRegPX:
     assert(inst.Register >= 19 && "Saved registers must be >= 19");
     reg = inst.Register - 19;
     b = 0xCC | ((reg & 0xC) >> 2);
     streamer.emitInt8(b);
     b = ((reg & 0x3) << 6) | ((inst.Offset >> 3) - 1);
     streamer.emitInt8(b);
     break;
+  case Win64EH::UOP_SaveLRPair:
+    assert(inst.Register >= 19 && "Saved reg must be >= 19");
+    reg = inst.Register - 19;
+    assert((reg % 2) == 0 && "Saved reg must be 19+2*X");
+    reg /= 2;
+    b = 0xD6 | ((reg & 0x7) >> 2);
+    streamer.emitInt8(b);
+    b = ((reg & 0x3) << 6) | (inst.Offset >> 3);
+    streamer.emitInt8(b);
+    break;
   case Win64EH::UOP_SaveFReg:
     assert(inst.Register >= 8 && "Saved dreg must be >= 8");
     reg = inst.Register - 8;
     b = 0xDC | ((reg & 0x4) >> 2);
     streamer.emitInt8(b);
     b = ((reg & 0x3) << 6) | (inst.Offset >> 3);
     streamer.emitInt8(b);
     break;
case Win64EH::UOP_SaveFRegPX:
     streamer.emitInt8(b);
     b = ((reg & 0x3) << 6) | ((inst.Offset >> 3) - 1);
     streamer.emitInt8(b);
     break;
   case Win64EH::UOP_End:
     b = 0xE4;
     streamer.emitInt8(b);
     break;
+  case Win64EH::UOP_SaveNext:
+    b = 0xE6;
+    streamer.emitInt8(b);
+    break;
+  case Win64EH::UOP_TrapFrame:
+    b = 0xE8;
+    streamer.emitInt8(b);
+    break;
+  case Win64EH::UOP_PushMachFrame:
+    b = 0xE9;
+    streamer.emitInt8(b);
+    break;
+  case Win64EH::UOP_Context:
+    b = 0xEA;
+    streamer.emitInt8(b);
+    break;
+  case Win64EH::UOP_ClearUnwoundToCall:
+    b = 0xEC;
+    streamer.emitInt8(b);
+    break;
   }
 }
 
 // Returns the epilog symbol of an epilog with the exact same unwind code
 // sequence, if it exists.  Otherwise, returns nulltpr.
 // EpilogInstrs - Unwind codes for the current epilog.
 // Epilogs - Epilogs that potentialy match the current epilog.
 static MCSymbol*

llvm/lib/Target/AArch64/AsmParser/AArch64AsmParser.cpp

private:
 
   bool parseDirectiveReq(StringRef Name, SMLoc L);
   bool parseDirectiveUnreq(SMLoc L);
   bool parseDirectiveCFINegateRAState();
   bool parseDirectiveCFIBKeyFrame();
 
   bool parseDirectiveSEHAllocStack(SMLoc L);
   bool parseDirectiveSEHPrologEnd(SMLoc L);
+  bool parseDirectiveSEHSaveR19R20X(SMLoc L);
   bool parseDirectiveSEHSaveFPLR(SMLoc L);
   bool parseDirectiveSEHSaveFPLRX(SMLoc L);
   bool parseDirectiveSEHSaveReg(SMLoc L);
   bool parseDirectiveSEHSaveRegX(SMLoc L);
   bool parseDirectiveSEHSaveRegP(SMLoc L);
   bool parseDirectiveSEHSaveRegPX(SMLoc L);
+  bool parseDirectiveSEHSaveLRPair(SMLoc L);
   bool parseDirectiveSEHSaveFReg(SMLoc L);
   bool parseDirectiveSEHSaveFRegX(SMLoc L);
   bool parseDirectiveSEHSaveFRegP(SMLoc L);
   bool parseDirectiveSEHSaveFRegPX(SMLoc L);
   bool parseDirectiveSEHSetFP(SMLoc L);
   bool parseDirectiveSEHAddFP(SMLoc L);
   bool parseDirectiveSEHNop(SMLoc L);
+  bool parseDirectiveSEHSaveNext(SMLoc L);
   bool parseDirectiveSEHEpilogStart(SMLoc L);
   bool parseDirectiveSEHEpilogEnd(SMLoc L);
+  bool parseDirectiveSEHTrapFrame(SMLoc L);
+  bool parseDirectiveSEHMachineFrame(SMLoc L);
+  bool parseDirectiveSEHContext(SMLoc L);
+  bool parseDirectiveSEHClearUnwoundToCall(SMLoc L);
 
   bool validateInstruction(MCInst &Inst, SMLoc &IDLoc,
                            SmallVectorImpl<SMLoc> &Loc);
   bool MatchAndEmitInstruction(SMLoc IDLoc, unsigned &Opcode,
                                OperandVector &Operands, MCStreamer &Out,
                                uint64_t &ErrorInfo,
                                bool MatchingInlineAsm) override;
 /// @name Auto-generated Match Functions
if (IDVal == MCLOHDirectiveName())
       parseDirectiveLOH(IDVal, Loc);
     else
       return true;
   } else if (IsCOFF) {
     if (IDVal == ".seh_stackalloc")
       parseDirectiveSEHAllocStack(Loc);
     else if (IDVal == ".seh_endprologue")
       parseDirectiveSEHPrologEnd(Loc);
+    else if (IDVal == ".seh_save_r19r20_x")
+      parseDirectiveSEHSaveR19R20X(Loc);
     else if (IDVal == ".seh_save_fplr")
       parseDirectiveSEHSaveFPLR(Loc);
     else if (IDVal == ".seh_save_fplr_x")
       parseDirectiveSEHSaveFPLRX(Loc);
     else if (IDVal == ".seh_save_reg")
       parseDirectiveSEHSaveReg(Loc);
     else if (IDVal == ".seh_save_reg_x")
       parseDirectiveSEHSaveRegX(Loc);
     else if (IDVal == ".seh_save_regp")
       parseDirectiveSEHSaveRegP(Loc);
     else if (IDVal == ".seh_save_regp_x")
       parseDirectiveSEHSaveRegPX(Loc);
+    else if (IDVal == ".seh_save_lrpair")
+      parseDirectiveSEHSaveLRPair(Loc);
     else if (IDVal == ".seh_save_freg")
       parseDirectiveSEHSaveFReg(Loc);
     else if (IDVal == ".seh_save_freg_x")
       parseDirectiveSEHSaveFRegX(Loc);
     else if (IDVal == ".seh_save_fregp")
       parseDirectiveSEHSaveFRegP(Loc);
     else if (IDVal == ".seh_save_fregp_x")
       parseDirectiveSEHSaveFRegPX(Loc);
     else if (IDVal == ".seh_set_fp")
       parseDirectiveSEHSetFP(Loc);
     else if (IDVal == ".seh_add_fp")
       parseDirectiveSEHAddFP(Loc);
     else if (IDVal == ".seh_nop")
       parseDirectiveSEHNop(Loc);
+    else if (IDVal == ".seh_save_next")
+      parseDirectiveSEHSaveNext(Loc);
     else if (IDVal == ".seh_startepilogue")
       parseDirectiveSEHEpilogStart(Loc);
     else if (IDVal == ".seh_endepilogue")
       parseDirectiveSEHEpilogEnd(Loc);
+    else if (IDVal == ".seh_trap_frame")
+      parseDirectiveSEHTrapFrame(Loc);
+    else if (IDVal == ".seh_pushframe")
+      parseDirectiveSEHMachineFrame(Loc);
+    else if (IDVal == ".seh_context")
+      parseDirectiveSEHContext(Loc);
+    else if (IDVal == ".seh_clear_unwound_to_call")
+      parseDirectiveSEHClearUnwoundToCall(Loc);
     else
       return true;
   } else
     return true;
   return false;
 }
 
 static void ExpandCryptoAEK(AArch64::ArchKind ArchKind,

 
 /// parseDirectiveSEHPrologEnd
 /// ::= .seh_endprologue
 bool AArch64AsmParser::parseDirectiveSEHPrologEnd(SMLoc L) {
   getTargetStreamer().EmitARM64WinCFIPrologEnd();
   return false;
 }
 
+/// parseDirectiveSEHSaveR19R20X
+/// ::= .seh_save_r19r20_x
+bool AArch64AsmParser::parseDirectiveSEHSaveR19R20X(SMLoc L) {
+  int64_t Offset;
+  if (parseImmExpr(Offset))
+    return true;
+  getTargetStreamer().EmitARM64WinCFISaveR19R20X(Offset);
+  return false;
+}
+
 /// parseDirectiveSEHSaveFPLR
 /// ::= .seh_save_fplr
 bool AArch64AsmParser::parseDirectiveSEHSaveFPLR(SMLoc L) {
   int64_t Offset;
   if (parseImmExpr(Offset))
     return true;
   getTargetStreamer().EmitARM64WinCFISaveFPLR(Offset);
   return false;
bool AArch64AsmParser::parseDirectiveSEHSaveRegPX(SMLoc L) {
   int64_t Offset;
   if (parseRegisterInRange(Reg, AArch64::X0, AArch64::X19, AArch64::X28) ||
       parseComma() || parseImmExpr(Offset))
     return true;
   getTargetStreamer().EmitARM64WinCFISaveRegPX(Reg, Offset);
   return false;
 }
 
+/// parseDirectiveSEHSaveLRPair
+/// ::= .seh_save_lrpair
+bool AArch64AsmParser::parseDirectiveSEHSaveLRPair(SMLoc L) {
+  unsigned Reg;
+  int64_t Offset;
+  L = getLoc();
+  if (parseRegisterInRange(Reg, AArch64::X0, AArch64::X19, AArch64::LR) ||
+      parseComma() || parseImmExpr(Offset))
+    return true;
+  if (check(((Reg - 19) % 2 != 0), L,
+            "expected register with even offset from x19"))
+    return true;
+  getTargetStreamer().EmitARM64WinCFISaveLRPair(Reg, Offset);
+  return false;
+}
+
 /// parseDirectiveSEHSaveFReg
 /// ::= .seh_save_freg
 bool AArch64AsmParser::parseDirectiveSEHSaveFReg(SMLoc L) {
   unsigned Reg;
   int64_t Offset;
   if (parseRegisterInRange(Reg, AArch64::D0, AArch64::D8, AArch64::D15) ||
       parseComma() || parseImmExpr(Offset))
     return true;

 
 /// parseDirectiveSEHNop
 /// ::= .seh_nop
 bool AArch64AsmParser::parseDirectiveSEHNop(SMLoc L) {
   getTargetStreamer().EmitARM64WinCFINop();
   return false;
 }
 
+/// parseDirectiveSEHSaveNext
+/// ::= .seh_save_next
+bool AArch64AsmParser::parseDirectiveSEHSaveNext(SMLoc L) {
+  getTargetStreamer().EmitARM64WinCFISaveNext();
+  return false;
+}
+
 /// parseDirectiveSEHEpilogStart
 /// ::= .seh_startepilogue
 bool AArch64AsmParser::parseDirectiveSEHEpilogStart(SMLoc L) {
   getTargetStreamer().EmitARM64WinCFIEpilogStart();
   return false;
 }
 
 /// parseDirectiveSEHEpilogEnd
 /// ::= .seh_endepilogue
 bool AArch64AsmParser::parseDirectiveSEHEpilogEnd(SMLoc L) {
   getTargetStreamer().EmitARM64WinCFIEpilogEnd();
   return false;
 }
 
+/// parseDirectiveSEHTrapFrame
+/// ::= .seh_trap_frame
+bool AArch64AsmParser::parseDirectiveSEHTrapFrame(SMLoc L) {
+  getTargetStreamer().EmitARM64WinCFITrapFrame();
+  return false;
+}
+
+/// parseDirectiveSEHMachineFrame
+/// ::= .seh_pushframe
+bool AArch64AsmParser::parseDirectiveSEHMachineFrame(SMLoc L) {
+  getTargetStreamer().EmitARM64WinCFIMachineFrame();
+  return false;
+}
+
+/// parseDirectiveSEHContext
+/// ::= .seh_context
+bool AArch64AsmParser::parseDirectiveSEHContext(SMLoc L) {
+  getTargetStreamer().EmitARM64WinCFIContext();
+  return false;
+}
+
+/// parseDirectiveSEHClearUnwoundToCall
+/// ::= .seh_clear_unwound_to_call
+bool AArch64AsmParser::parseDirectiveSEHClearUnwoundToCall(SMLoc L) {
+  getTargetStreamer().EmitARM64WinCFIClearUnwoundToCall();
+  return false;
+}
+
 bool
 AArch64AsmParser::classifySymbolRef(const MCExpr *Expr,
                                     AArch64MCExpr::VariantKind &ELFRefKind,
                                     MCSymbolRefExpr::VariantKind &DarwinRefKind,
                                     int64_t &Addend) {
   ELFRefKind = AArch64MCExpr::VK_INVALID;
   DarwinRefKind = MCSymbolRefExpr::VK_None;
   Addend = 0;

llvm/lib/Target/AArch64/MCTargetDesc/AArch64ELFStreamer.cpp

 class AArch64TargetAsmStreamer : public AArch64TargetStreamer {
   formatted_raw_ostream &OS;
 
   void emitInst(uint32_t Inst) override;
 
   void EmitARM64WinCFIAllocStack(unsigned Size) override {
     OS << "\t.seh_stackalloc " << Size << "\n";
   }
+  void EmitARM64WinCFISaveR19R20X(int Offset) override {
+    OS << "\t.seh_save_r19r20_x " << Offset << "\n";
+  }
   void EmitARM64WinCFISaveFPLR(int Offset) override {
     OS << "\t.seh_save_fplr " << Offset << "\n";
   }
   void EmitARM64WinCFISaveFPLRX(int Offset) override {
     OS << "\t.seh_save_fplr_x " << Offset << "\n";
   }
   void EmitARM64WinCFISaveReg(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_reg x" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveRegX(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_reg_x x" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveRegP(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_regp x" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveRegPX(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_regp_x x" << Reg << ", " << Offset << "\n";
   }
+  void EmitARM64WinCFISaveLRPair(unsigned Reg, int Offset) override {
+    OS << "\t.seh_save_lrpair x" << Reg << ", " << Offset << "\n";
+  }
   void EmitARM64WinCFISaveFReg(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_freg d" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveFRegX(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_freg_x d" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveFRegP(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_fregp d" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISaveFRegPX(unsigned Reg, int Offset) override {
     OS << "\t.seh_save_fregp_x d" << Reg << ", " << Offset << "\n";
   }
   void EmitARM64WinCFISetFP() override { OS << "\t.seh_set_fp\n"; }
   void EmitARM64WinCFIAddFP(unsigned Size) override {
     OS << "\t.seh_add_fp " << Size << "\n";
   }
   void EmitARM64WinCFINop() override { OS << "\t.seh_nop\n"; }
+  void EmitARM64WinCFISaveNext() override { OS << "\t.seh_save_next\n"; }
   void EmitARM64WinCFIPrologEnd() override { OS << "\t.seh_endprologue\n"; }
   void EmitARM64WinCFIEpilogStart() override { OS << "\t.seh_startepilogue\n"; }
   void EmitARM64WinCFIEpilogEnd() override { OS << "\t.seh_endepilogue\n"; }
+  void EmitARM64WinCFITrapFrame() override { OS << "\t.seh_trap_frame\n"; }
+  void EmitARM64WinCFIMachineFrame() override { OS << "\t.seh_pushframe\n"; }
+  void EmitARM64WinCFIContext() override { OS << "\t.seh_context\n"; }
+  void EmitARM64WinCFIClearUnwoundToCall() override {
+    OS << "\t.seh_clear_unwound_to_call\n";
+  }
 
 public:
   AArch64TargetAsmStreamer(MCStreamer &S, formatted_raw_ostream &OS);
 };
 
 AArch64TargetAsmStreamer::AArch64TargetAsmStreamer(MCStreamer &S,
                                                    formatted_raw_ostream &OS)
   : AArch64TargetStreamer(S), OS(OS) {}

llvm/lib/Target/AArch64/MCTargetDesc/AArch64TargetStreamer.h

public:
   /// Callback used to implemnt the .ltorg directive.
   /// Emit contents of constant pool for the current section.
   void emitCurrentConstantPool();
 
   /// Callback used to implement the .inst directive.
   virtual void emitInst(uint32_t Inst);
 
   virtual void EmitARM64WinCFIAllocStack(unsigned Size) {}
+  virtual void EmitARM64WinCFISaveR19R20X(int Offset) {}
   virtual void EmitARM64WinCFISaveFPLR(int Offset) {}
   virtual void EmitARM64WinCFISaveFPLRX(int Offset) {}
   virtual void EmitARM64WinCFISaveReg(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveRegX(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveRegP(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveRegPX(unsigned Reg, int Offset) {}
+  virtual void EmitARM64WinCFISaveLRPair(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveFReg(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveFRegX(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveFRegP(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISaveFRegPX(unsigned Reg, int Offset) {}
   virtual void EmitARM64WinCFISetFP() {}
   virtual void EmitARM64WinCFIAddFP(unsigned Size) {}
   virtual void EmitARM64WinCFINop() {}
+  virtual void EmitARM64WinCFISaveNext() {}
   virtual void EmitARM64WinCFIPrologEnd() {}
   virtual void EmitARM64WinCFIEpilogStart() {}
   virtual void EmitARM64WinCFIEpilogEnd() {}
+  virtual void EmitARM64WinCFITrapFrame() {}
+  virtual void EmitARM64WinCFIMachineFrame() {}
+  virtual void EmitARM64WinCFIContext() {}
+  virtual void EmitARM64WinCFIClearUnwoundToCall() {}
 
 private:
   std::unique_ptr<AssemblerConstantPools> ConstantPools;
 };
 
 class AArch64TargetELFStreamer : public AArch64TargetStreamer {
 private:
   AArch64ELFStreamer &getStreamer();
private:
   MCSymbol *CurrentEpilog = nullptr;
 public:
   AArch64TargetWinCOFFStreamer(llvm::MCStreamer &S)
     : AArch64TargetStreamer(S) {}
 
   // The unwind codes on ARM64 Windows are documented at
   // https://docs.microsoft.com/en-us/cpp/build/arm64-exception-handling
   void EmitARM64WinCFIAllocStack(unsigned Size) override;
+  void EmitARM64WinCFISaveR19R20X(int Offset) override;
   void EmitARM64WinCFISaveFPLR(int Offset) override;
   void EmitARM64WinCFISaveFPLRX(int Offset) override;
   void EmitARM64WinCFISaveReg(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveRegX(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveRegP(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveRegPX(unsigned Reg, int Offset) override;
+  void EmitARM64WinCFISaveLRPair(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveFReg(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveFRegX(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveFRegP(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISaveFRegPX(unsigned Reg, int Offset) override;
   void EmitARM64WinCFISetFP() override;
   void EmitARM64WinCFIAddFP(unsigned Size) override;
   void EmitARM64WinCFINop() override;
+  void EmitARM64WinCFISaveNext() override;
   void EmitARM64WinCFIPrologEnd() override;
   void EmitARM64WinCFIEpilogStart() override;
   void EmitARM64WinCFIEpilogEnd() override;
+  void EmitARM64WinCFITrapFrame() override;
+  void EmitARM64WinCFIMachineFrame() override;
+  void EmitARM64WinCFIContext() override;
+  void EmitARM64WinCFIClearUnwoundToCall() override;
+
 private:
   void EmitARM64WinUnwindCode(unsigned UnwindCode, int Reg, int Offset);
 };
 
 MCTargetStreamer *
 createAArch64ObjectTargetStreamer(MCStreamer &S, const MCSubtargetInfo &STI);
 
 } // end namespace llvm
 
 #endif

llvm/lib/Target/AArch64/MCTargetDesc/AArch64WinCOFFStreamer.cpp

void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIAllocStack(unsigned Size) {
   unsigned Op = Win64EH::UOP_AllocSmall;
   if (Size >= 16384)
     Op = Win64EH::UOP_AllocLarge;
   else if (Size >= 512)
     Op = Win64EH::UOP_AllocMedium;
   EmitARM64WinUnwindCode(Op, -1, Size);
 }
 
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveR19R20X(int Offset) {
+  EmitARM64WinUnwindCode(Win64EH::UOP_SaveR19R20X, -1, Offset);
+}
+
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveFPLR(int Offset) {
   EmitARM64WinUnwindCode(Win64EH::UOP_SaveFPLR, -1, Offset);
 }
 
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveFPLRX(int Offset) {
   EmitARM64WinUnwindCode(Win64EH::UOP_SaveFPLRX, -1, Offset);
 }
 
void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveRegP(unsigned Reg,
   EmitARM64WinUnwindCode(Win64EH::UOP_SaveRegP, Reg, Offset);
 }
 
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveRegPX(unsigned Reg,
                                                             int Offset) {
   EmitARM64WinUnwindCode(Win64EH::UOP_SaveRegPX, Reg, Offset);
 }
 
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveLRPair(unsigned Reg,
+                                                             int Offset) {
+  EmitARM64WinUnwindCode(Win64EH::UOP_SaveLRPair, Reg, Offset);
+}
+
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveFReg(unsigned Reg,
                                                            int Offset) {
   assert(Offset >= 0 && Offset <= 504 &&
         "Offset for save reg should be >= 0 && <= 504");
   EmitARM64WinUnwindCode(Win64EH::UOP_SaveFReg, Reg, Offset);
 }
 
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveFRegX(unsigned Reg,
void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIAddFP(unsigned Offset) {
   assert(Offset <= 2040 && "UOP_AddFP must have offset <= 2040");
   EmitARM64WinUnwindCode(Win64EH::UOP_AddFP, -1, Offset);
 }
 
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFINop() {
   EmitARM64WinUnwindCode(Win64EH::UOP_Nop, -1, 0);
 }
 
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFISaveNext() {
+  EmitARM64WinUnwindCode(Win64EH::UOP_SaveNext, -1, 0);
+}
+
 // The functions below handle opcodes that can end up in either a prolog or
 // an epilog, but not both.
 void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIPrologEnd() {
   auto &S = getStreamer();
   WinEH::FrameInfo *CurFrame = S.EnsureValidWinFrameInfo(SMLoc());
   if (!CurFrame)
     return;
 
void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIEpilogEnd() {
 
   InEpilogCFI = false;
   MCSymbol *Label = S.emitCFILabel();
   WinEH::Instruction Inst = WinEH::Instruction(Win64EH::UOP_End, Label, -1, 0);
   CurFrame->EpilogMap[CurrentEpilog].push_back(Inst);
   CurrentEpilog = nullptr;
 }
 
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFITrapFrame() {
+  EmitARM64WinUnwindCode(Win64EH::UOP_TrapFrame, -1, 0);
+}
+
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIMachineFrame() {
+  EmitARM64WinUnwindCode(Win64EH::UOP_PushMachFrame, -1, 0);
+}
+
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIContext() {
+  EmitARM64WinUnwindCode(Win64EH::UOP_Context, -1, 0);
+}
+
+void AArch64TargetWinCOFFStreamer::EmitARM64WinCFIClearUnwoundToCall() {
+  EmitARM64WinUnwindCode(Win64EH::UOP_ClearUnwoundToCall, -1, 0);
+}
+
 MCWinCOFFStreamer *createAArch64WinCOFFStreamer(
     MCContext &Context, std::unique_ptr<MCAsmBackend> MAB,
     std::unique_ptr<MCObjectWriter> OW, std::unique_ptr<MCCodeEmitter> Emitter,
     bool RelaxAll, bool IncrementalLinkerCompatible) {
   auto *S = new AArch64WinCOFFStreamer(Context, std::move(MAB),
                                        std::move(Emitter), std::move(OW));
   S->getAssembler().setIncrementalLinkerCompatible(IncrementalLinkerCompatible);
   return S;
 }
 
 } // end llvm namespace

llvm/test/MC/AArch64/seh.s

 // CHECK-NEXT:       ALIGN_4BYTES
 // CHECK-NEXT:       CNT_CODE
 // CHECK-NEXT:       MEM_EXECUTE
 // CHECK-NEXT:       MEM_READ
 // CHECK-NEXT:     ]
 // CHECK-NEXT:   }
 // CHECK:        Section {
 // CHECK:          Name: .xdata
-// CHECK:          RawDataSize: 48
+// CHECK:          RawDataSize: 56
 // CHECK:          RelocationCount: 1
 // CHECK:          Characteristics [
 // CHECK-NEXT:       ALIGN_4BYTES
 // CHECK-NEXT:       CNT_INITIALIZED_DATA
 // CHECK-NEXT:       MEM_READ
 // CHECK-NEXT:     ]
 // CHECK-NEXT:   }
 // CHECK:        Section {
 // CHECK:          Name: .pdata
 // CHECK:          RelocationCount: 2
 // CHECK:          Characteristics [
 // CHECK-NEXT:       ALIGN_4BYTES
 // CHECK-NEXT:       CNT_INITIALIZED_DATA
 // CHECK-NEXT:       MEM_READ
 // CHECK-NEXT:     ]
 // CHECK-NEXT:   }
 // CHECK-NEXT: ]
 
 // CHECK-NEXT: Relocations [
 // CHECK-NEXT:   Section (4) .xdata {
-// CHECK-NEXT:     0x24 IMAGE_REL_ARM64_ADDR32NB __C_specific_handler
+// CHECK-NEXT:     0x2C IMAGE_REL_ARM64_ADDR32NB __C_specific_handler
 // CHECK-NEXT:   }
 // CHECK-NEXT:   Section (5) .pdata {
 // CHECK-NEXT:     0x0 IMAGE_REL_ARM64_ADDR32NB func
 // CHECK-NEXT:     0x4 IMAGE_REL_ARM64_ADDR32NB .xdata
 // CHECK-NEXT:   }
 // CHECK-NEXT: ]
 
 // CHECK-NEXT: UnwindInformation [
 // CHECK-NEXT:   RuntimeFunction {
 // CHECK-NEXT:     Function: func
 // CHECK-NEXT:     ExceptionRecord: .xdata
 // CHECK-NEXT:     ExceptionData {
-// CHECK-NEXT:       FunctionLength: 72
+// CHECK-NEXT:       FunctionLength: 100
 // CHECK:            Prologue [
+// CHECK-NEXT:         0xec                ; clear unwound to call
+// CHECK-NEXT:         0xea                ; context
+// CHECK-NEXT:         0xe9                ; machine frame
+// CHECK-NEXT:         0xe8                ; trap frame
 // CHECK-NEXT:         0xe3                ; nop
 // CHECK-NEXT:         0xe202              ; add fp, sp, #16
 // CHECK-NEXT:         0xdd41              ; str d13, [sp, #8]
 // CHECK-NEXT:         0xde83              ; str d12, [sp, #-32]!
 // CHECK-NEXT:         0xd882              ; stp d10, d11, [sp, #16]
 // CHECK-NEXT:         0xda03              ; stp d8, d9, [sp, #-32]!
 // CHECK-NEXT:         0x83                ; stp x29, x30, [sp, #-32]!
 // CHECK-NEXT:         0x46                ; stp x29, x30, [sp, #48]
 // CHECK-NEXT:         0xd141              ; str x24, [sp, #8]
 // CHECK-NEXT:         0xd483              ; str x23, [sp, #-32]!
+// CHECK-NEXT:         0xe6                ; save next
 // CHECK-NEXT:         0xc882              ; stp x21, x22, [sp, #16]
+// CHECK-NEXT:         0xd6c2              ; stp x25, lr, [sp, #16]
+// CHECK-NEXT:         0x24                ; stp x19, x20, [sp, #-32]!
 // CHECK-NEXT:         0xcc03              ; stp x19, x20, [sp, #-32]!
 // CHECK-NEXT:         0x83                ; stp x29, x30, [sp, #-32]!
 // CHECK-NEXT:         0xe1                ; mov fp, sp
 // CHECK-NEXT:         0x01                ; sub sp, #16
 // CHECK-NEXT:         0xe4                ; end
 // CHECK-NEXT:       ]
 // CHECK-NEXT:       EpilogueScopes [
 // CHECK-NEXT:         EpilogueScope {
-// CHECK-NEXT:           StartOffset: 16
-// CHECK-NEXT:           EpilogueStartIndex: 25
+// CHECK-NEXT:           StartOffset: 23
+// CHECK-NEXT:           EpilogueStartIndex: 33
 // CHECK-NEXT:           Opcodes [
 // CHECK-NEXT:             0x01                ; add sp, #16
 // CHECK-NEXT:             0xe4                ; end
 // CHECK-NEXT:           ]
 // CHECK-NEXT:         }
 // CHECK-NEXT:       ]
 // CHECK-NEXT:       ExceptionHandler [
 // CHECK-NEXT:         Routine: __C_specific_handler (0x0)
func:
     sub sp, sp, #24
     .seh_stackalloc 24
     mov x29, sp
     .seh_set_fp
     stp x29, x30, [sp, #-32]!
     .seh_save_fplr_x 32
     stp x19, x20, [sp, #-32]!
     .seh_save_regp_x x19, 32
+    stp x19, x20, [sp, #-32]!
+    .seh_save_r19r20_x 32
+    stp x25, x30, [sp, #16]
+    .seh_save_lrpair x25, 16
     stp x21, x22, [sp, #16]
     .seh_save_regp x21, 16
+    stp x23, x24, [sp, #32]
+    .seh_save_next
     str x23, [sp, #-32]!
     .seh_save_reg_x x23, 32
     str x24, [sp, #8]
     .seh_save_reg x24, 8
     stp x29, x30, [sp, #48]
     .seh_save_fplr 48
     stp x29, x30, [sp, #-32]!
     .seh_save_fplr_x 32
     stp d8, d9, [sp, #-32]!
     .seh_save_fregp_x d8, 32
     stp d10, d11, [sp, #16]
     .seh_save_fregp d10, 16
     str d12, [sp, #-32]!
     .seh_save_freg_x d12, 32
     str d13, [sp, #8]
     .seh_save_freg d13, 8
     add x29, sp, #16
     .seh_add_fp 16
     nop
     .seh_nop
+    nop
+    .seh_trap_frame
+    nop
+    .seh_pushframe
+    nop
+    .seh_context
+    nop
+    .seh_clear_unwound_to_call
     .seh_endprologue
     nop
     .seh_startepilogue
     add sp, sp, #24
     .seh_stackalloc 24
     .seh_endepilogue
     ret
     .seh_handler __C_specific_handler, @except

llvm/tools/llvm-readobj/ARMWinEHPrinter.h

class Decoder {
   bool opcode_nop(const uint8_t *Opcodes, unsigned &Offset, unsigned Length,
                   bool Prologue);
   bool opcode_end(const uint8_t *Opcodes, unsigned &Offset, unsigned Length,
                   bool Prologue);
   bool opcode_end_c(const uint8_t *Opcodes, unsigned &Offset, unsigned Length,
                     bool Prologue);
   bool opcode_save_next(const uint8_t *Opcodes, unsigned &Offset,
                         unsigned Length, bool Prologue);
+  bool opcode_trap_frame(const uint8_t *Opcodes, unsigned &Offset,
+                         unsigned Length, bool Prologue);
+  bool opcode_machine_frame(const uint8_t *Opcodes, unsigned &Offset,
+                            unsigned Length, bool Prologue);
+  bool opcode_context(const uint8_t *Opcodes, unsigned &Offset, unsigned Length,
+                      bool Prologue);
+  bool opcode_clear_unwound_to_call(const uint8_t *Opcodes, unsigned &Offset,
+                                    unsigned Length, bool Prologue);
 
   void decodeOpcodes(ArrayRef<uint8_t> Opcodes, unsigned Offset,
                      bool Prologue);
 
   void printRegisters(const std::pair<uint16_t, uint32_t> &RegisterMask);
 
   ErrorOr<object::SectionRef>
   getSectionContaining(const object::COFFObjectFile &COFF, uint64_t Address);

llvm/tools/llvm-readobj/ARMWinEHPrinter.cpp

const Decoder::RingEntry Decoder::Ring64[] = {
   { 0xfe, 0xdc, 2, &Decoder::opcode_save_freg },
   { 0xff, 0xde, 2, &Decoder::opcode_save_freg_x },
   { 0xff, 0xe0, 4, &Decoder::opcode_alloc_l },
   { 0xff, 0xe1, 1, &Decoder::opcode_setfp },
   { 0xff, 0xe2, 2, &Decoder::opcode_addfp },
   { 0xff, 0xe3, 1, &Decoder::opcode_nop },
   { 0xff, 0xe4, 1, &Decoder::opcode_end },
   { 0xff, 0xe5, 1, &Decoder::opcode_end_c },
+  { 0xff, 0xe6, 1, &Decoder::opcode_save_next },
+  { 0xff, 0xe8, 1, &Decoder::opcode_trap_frame },
+  { 0xff, 0xe9, 1, &Decoder::opcode_machine_frame },
+  { 0xff, 0xea, 1, &Decoder::opcode_context },
+  { 0xff, 0xec, 1, &Decoder::opcode_clear_unwound_to_call },
 };
 
 void Decoder::printRegisters(const std::pair<uint16_t, uint32_t> &RegisterMask) {
   static const char * const GPRRegisterNames[16] = {
     "r0", "r1", "r2", "r3", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
     "r11", "ip", "sp", "lr", "pc",
   };
 

 
 bool Decoder::opcode_end_c(const uint8_t *OC, unsigned &Offset, unsigned Length,
                            bool Prologue) {
   SW.startLine() << format("0x%02x                ; end_c\n", OC[Offset]);
   ++Offset;
   return true;
 }
 
+bool Decoder::opcode_save_next(const uint8_t *OC, unsigned &Offset,
+                               unsigned Length, bool Prologue) {
+  if (Prologue)
+    SW.startLine() << format("0x%02x                ; save next\n", OC[Offset]);
+  else
+    SW.startLine() << format("0x%02x                ; restore next\n",
+                             OC[Offset]);
+  ++Offset;
+  return false;
+}
+
+bool Decoder::opcode_trap_frame(const uint8_t *OC, unsigned &Offset,
+                                unsigned Length, bool Prologue) {
+  SW.startLine() << format("0x%02x                ; trap frame\n", OC[Offset]);
+  ++Offset;
+  return false;
+}
+
+bool Decoder::opcode_machine_frame(const uint8_t *OC, unsigned &Offset,
+                                   unsigned Length, bool Prologue) {
+  SW.startLine() << format("0x%02x                ; machine frame\n",
+                           OC[Offset]);
+  ++Offset;
+  return false;
+}
+
+bool Decoder::opcode_context(const uint8_t *OC, unsigned &Offset,
+                             unsigned Length, bool Prologue) {
+  SW.startLine() << format("0x%02x                ; context\n", OC[Offset]);
+  ++Offset;
+  return false;
+}
+
+bool Decoder::opcode_clear_unwound_to_call(const uint8_t *OC, unsigned &Offset,
+                                           unsigned Length, bool Prologue) {
+  SW.startLine() << format("0x%02x                ; clear unwound to call\n",
+                           OC[Offset]);
+  ++Offset;
+  return false;
+}
+
 void Decoder::decodeOpcodes(ArrayRef<uint8_t> Opcodes, unsigned Offset,
                             bool Prologue) {
   assert((!Prologue || Offset == 0) && "prologue should always use offset 0");
   const RingEntry* DecodeRing = isAArch64 ? Ring64 : Ring;
   bool Terminated = false;
   for (unsigned OI = Offset, OE = Opcodes.size(); !Terminated && OI < OE; ) {
     for (unsigned DI = 0;; ++DI) {
       if ((isAArch64 && (DI >= array_lengthof(Ring64))) ||