[WinEH] Allocate space in funclets stack to save XMM CSRs

[pengfei]

Summary

This is an alternate approach to D57970.

Currently funclets reuse the same stack slots that are used in the parent function for saving callee-saved xmm registers. If the parent function modifies a callee-saved xmm register before an excpetion is thrown, the catch handler will overwrite the original saved value.

This patch allocates space in funclets stack for saving callee-saved xmm registers and uses RSP instead RBP to access memory.

[rnk]

llvm/lib/Target/X86/X86FrameLowering.cpp

1435

This math relies on too many fragile undocumented invariants:

• frame indices are consecutive
• frame indices are fixed (or non-fixed), one is negative, the other not

Is it possible to adjust from the assigned fixed offset in the parent frame to the funclet frame using some other MachineFrameInfo values? This MOVAPS instruction replacement seems like the correct fix, I just want to make it more robust.

1446

This code should execute for funclets as well, I believe, so we get .seh_savexmm directives for funclets.

2266

This is not the right place to do this, I would expect to see it in emitepilog.

[pengfei]

Refine code as Reid's proposal.
Separate the XMM slot caculating so that we can get a consecutive layout in stack.
Also fix an alignment bug due to it affecting this patch.

[pengfei]

llvm/lib/Target/X86/X86FrameLowering.cpp

1435

Thanks for the advice.

• I separate the slot allocation of XMM, so it should be consecutive now
• XMM frame indices are always fixed for now

[rnk]

I haven't found time to review this, so I added @craig.topper and @RKSimon. They work on the x86 backend and should be able to provide guidance on structuring the frame lowering code.

[craig.topper]

llvm/lib/Target/X86/X86FrameLowering.cpp

1418

Should we use a VEX instruction here under AVX?

1424

Put else on the line above

2245

Any reason we're using FR64 here instead of VR128?

2247

VEX instruction under AVX?

[pengfei]

Refine code as Craig's proposal

[pengfei]

Thanks for the review

llvm/lib/Target/X86/X86FrameLowering.cpp

2245

VR128 should be more accurate. Thanks!

[craig.topper]

llvm/lib/Target/X86/X86FrameLowering.cpp

3274

If its always going to be XMM0-15. Which I think it is since you only checked VR128, not VR128X. Then you can ignore VLX and AVX512 I think.

[craig.topper]

llvm/test/CodeGen/X86/x86-interrupt_cc.ll

1

Why is a test case that isn't testing Windows affected by this change?

[pengfei]

We don't need to consider the upper registers [X,Y,Z]MM16~[X,Y,Z]MM31. So remove the VLX and AVX512 check.

[pengfei]

llvm/test/CodeGen/X86/x86-interrupt_cc.ll

1

There's a bug in the former slot reservation function. Because I changed the behavior on reserving XMM, I fixed it in the same patch, too.

[majnemer]

llvm/lib/Target/X86/X86FrameLowering.cpp

1417

Seems easier to read as alignDown(NumBytes, Align)

2051

Missing a period at the end of this sentence.

2251–2252

These should be on the same line.

[pengfei]

Refine code as David's proposal

[pengfei]

Thanks for the review.

[pengfei]

Ping.

[craig.topper]

LGTM

[rnk]

This broke this C++ code:

int getcsr();
void usecsrs(double, int, int, int, int, int, int, int, int);
double getxmm();
struct HasDtor {
  int x = 42;
  ~HasDtor();
};
int foo() {
  int v1 = getcsr();
  int v2 = getcsr();
  int v3 = getcsr();
  int v4 = getcsr();
  int v5 = getcsr();
  int v6 = getcsr();
  int v7 = getcsr();
  int v8 = getcsr();
  double f1 = getxmm();
  HasDtor o;
  usecsrs(f1, v1, v2, v3, v4, v5, v6, v7, v8);
  usecsrs(f1, v1, v2, v3, v4, v5, v6, v7, v8);
  usecsrs(f1, v1, v2, v3, v4, v5, v6, v7, v8);
  usecsrs(f1, v1, v2, v3, v4, v5, v6, v7, v8);
  return v1 + v2 + v3 + v4 + v5 + v6 + v7 + v8;
}

The movaps instruction in the funclet epilogue does not match the prologue:

$ ninja -j900 clang && clang -S t.cpp  -o - -O1 | grep 'movaps.*rsp\|dtor'
ninja: no work to do.
        .def     "?dtor$5@?0??foo@@YAHXZ@4HA";
"?dtor$5@?0??foo@@YAHXZ@4HA":
.seh_proc "?dtor$5@?0??foo@@YAHXZ@4HA"
        movaps  %xmm6, 64(%rsp)
        movaps  72(%rsp), %xmm6
        .long   "?dtor$5@?0??foo@@YAHXZ@4HA"@IMGREL # Action

The code crashes in the epilogue.

I fixed a different issue in rL368631, but I think this time I will revert this.

[pengfei]

This comment was removed by pengfei.

[pengfei]

Add downward alignment for XMM storing in epilogue. This fix the crush when MaxCallFrameSize is not 16 bytes aligned.

[pengfei]

Update test case to verify XMM registers correctly when MaxCallFrameSize is not 16 bytes aligned.

[rnk]

My suggestion for drastically simplifying this is to modify X86RegisterInfo::eliminateFrameIndex to check if it's lowering an XMM CSR slot. In that case, it would call a new method, X86FrameLowering::getSaveXMMFrameIndexRef, which would contain the logic to check if the current block is a funclet prologue or epilogue. If the parent BB of the current instruction is in a funclet, then XMM slots would be resolved relative to RSP using the funclet frame size. If not, it would delegate to the standard getFrameIndexReference logic, yielding the correct register and offset for XMM slots in non-funclet prologues and epilogues. The prologue code that currently inserts savexmm markers would need to call this new helper as well instead of the standard getFrameIndexReference method. Does that plan seem reasonable?

llvm/lib/Target/X86/X86FrameLowering.cpp

1193–1194

I think you should include this offset in the return value from getWinEHFuncletFrameSize. With your change, getWinEHFuncletFrameSize no longer returns the true funclet frame size, and it would be a bug to have a call site that forgets to include the XMM size offset. In fact, I found and fixed this exact bug in the catch funclet establishing frame offset code. We should fix it by design before relanding.

2244–2253

For a multi-bb funclet, won't this be false? You should reuse the isFuncletReturnInstr check from above to know if this is a funclet epilogue.

2246

I don't like this repeated math that has to be kept in sync between the prologue and epilogue. If we could factor it out, I would find it easier to read and have more confidence in the correctness.

[pengfei]

In D63396#1640095, @rnk wrote:

My suggestion for drastically simplifying this is to modify X86RegisterInfo::eliminateFrameIndex to check if it's lowering an XMM CSR slot. In that case, it would call a new method, X86FrameLowering::getSaveXMMFrameIndexRef, which would contain the logic to check if the current block is a funclet prologue or epilogue. If the parent BB of the current instruction is in a funclet, then XMM slots would be resolved relative to RSP using the funclet frame size. If not, it would delegate to the standard getFrameIndexReference logic, yielding the correct register and offset for XMM slots in non-funclet prologues and epilogues. The prologue code that currently inserts savexmm markers would need to call this new helper as well instead of the standard getFrameIndexReference method. Does that plan seem reasonable?

It's a great idea that simplifies the implementation. I commit a new patch on D66596 that follows your suggestions. Thanks a lot for the review and suggestions.

llvm/lib/Target/X86/X86FrameLowering.cpp

[else]
       .seh_stackalloc NNN
 
   ; Spill XMMs
   ; Note, that while only Windows 64 ABI specifies XMMs as callee-preserved,
   ; they may get spilled on any platform, if the current function
   ; calls @llvm.eh.unwind.init
   [if needs FP]
       [for all callee-saved XMM registers]
+          [if funclet]
+              movaps  %<xmm reg>, -MMM(%rsp)
+          [else]
               movaps  %<xmm reg>, -MMM(%rbp)
       [for all callee-saved XMM registers]
           .seh_savexmm %<xmm reg>, (-MMM + SEHFrameOffset)
               ; i.e. the offset relative to (%rbp - SEHFrameOffset)
   [else]
       [for all callee-saved XMM registers]
           movaps  %<xmm reg>, KKK(%rsp)
       [for all callee-saved XMM registers]
           .seh_savexmm %<xmm reg>, KKK
 
   .seh_endprologue
 
   [if needs base pointer]
       mov  %rsp, %rbx
       [if needs to restore base pointer]
           mov %rsp, -MMM(%rbp)
 
   ; Emit CFI info
   [if needs FP]
       [for all callee-saved registers]
+          [if funclet]
+              movaps -MMM(%rsp), %<xmm reg>
+          [else]
               .cfi_offset %<reg>, (offset from %rbp)
   [else]
        .cfi_def_cfa_offset (offset from RETADDR)
       [for all callee-saved registers]
           .cfi_offset %<reg>, (offset from %rsp)
 
   Notes:
   - .seh directives are emitted only for Windows 64 ABI
   - .cv_fpo directives are emitted on win32 when emitting CodeView
void X86FrameLowering::emitPrologue(MachineFunction &MF,
   // from ESP to VFRAME relative local variable offsets.
   if (!IsFunclet) {
     if (HasFP && TRI->needsStackRealignment(MF))
       MFI.setOffsetAdjustment(-NumBytes);
     else
       MFI.setOffsetAdjustment(-StackSize);
   }
 
-  // For EH funclets, only allocate enough space for outgoing calls. Save the
-  // NumBytes value that we would've used for the parent frame.
+  // For EH funclets, only allocate enough space for outgoing calls and callee
+  // saved XMM registers on Windows 64 bits. Save the NumBytes value that we
+  // would've used for the parent frame.
+  int XMMFrameSlotOrigin;
   unsigned ParentFrameNumBytes = NumBytes;
-  if (IsFunclet)
+  if (IsFunclet) {
     NumBytes = getWinEHFuncletFrameSize(MF);
+    if (IsWin64Prologue)
+      NumBytes += X86FI->getCalleeSavedXMMFrameInfo(XMMFrameSlotOrigin);
+  }
 
   // Skip the callee-saved push instructions.
   bool PushedRegs = false;
   int StackOffset = 2 * stackGrowth;
 
   while (MBBI != MBB.end() &&
          MBBI->getFlag(MachineInstr::FrameSetup) &&
          (MBBI->getOpcode() == X86::PUSH32r ||
if (!MBB.isCleanupFuncletEntry()) {
       // ESP is the first field, so no extra displacement is needed.
       addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV32mr)), FrameReg,
                    false, EHRegOffset)
           .addReg(X86::ESP);
     }
   }
 
   while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup)) {
-    const MachineInstr &FrameInstr = *MBBI;
+    auto FrameInstr = MBBI;
     ++MBBI;
 
     if (NeedsWinCFI) {
       int FI;
-      if (unsigned Reg = TII.isStoreToStackSlot(FrameInstr, FI)) {
+      if (unsigned Reg = TII.isStoreToStackSlot(*FrameInstr, FI)) {
         if (X86::FR64RegClass.contains(Reg)) {
-          unsigned IgnoredFrameReg;
-          int Offset = getFrameIndexReference(MF, FI, IgnoredFrameReg);
-          Offset += SEHFrameOffset;
-
+          int Offset = 0;
           HasWinCFI = true;
+          if (IsFunclet) {
+            assert(IsWin64Prologue && "Only valid on Windows 64bit");
+            unsigned Size = TRI->getSpillSize(X86::VR128RegClass);
+            unsigned Align = TRI->getSpillAlignment(X86::VR128RegClass);
+            Offset = (FI - XMMFrameSlotOrigin - 1) * Size +
+                     (NumBytes & ~(Align - 1));
+            addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(X86::MOVAPSmr)),
+                         StackPtr, true, Offset)
+                .addReg(Reg)
+                .setMIFlag(MachineInstr::FrameSetup);
+            MBB.erase(FrameInstr);
+          }
+          else {
             assert(!NeedsWinFPO && "SEH_SaveXMM incompatible with FPO data");
+            unsigned IgnoredFrameReg;
+            Offset = getFrameIndexReference(MF, FI, IgnoredFrameReg) +
+                     SEHFrameOffset;
+          }
           BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_SaveXMM))
               .addImm(Reg)
               .addImm(Offset)
               .setMIFlag(MachineInstr::FrameSetup);
         }
       }
     }
   }
 
   if (NeedsWinCFI && HasWinCFI)
     BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_EndPrologue))
         .setMIFlag(MachineInstr::FrameSetup);
 
   if (FnHasClrFunclet && !IsFunclet) {
     // Save the so-called Initial-SP (i.e. the value of the stack pointer
     // immediately after the prolog)  into the PSPSlot so that funclets
     // and the GC can recover it.
     unsigned PSPSlotOffset = getPSPSlotOffsetFromSP(MF);
     auto PSPInfo = MachinePointerInfo::getFixedStack(
         MF, MF.getWinEHFuncInfo()->PSPSymFrameIdx);
     addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV64mr)), StackPtr, false,
                  PSPSlotOffset)
         .addReg(StackPtr)
         .addMemOperand(MF.getMachineMemOperand(
             PSPInfo, MachineMemOperand::MOStore | MachineMemOperand::MOVolatile,
void X86FrameLowering::emitEpilogue(MachineFunction &MF,
   bool NeedsDwarfCFI =
       (!MF.getTarget().getTargetTriple().isOSDarwin() &&
        !MF.getTarget().getTargetTriple().isOSWindows()) &&
       (MF.getMMI().hasDebugInfo() || MF.getFunction().needsUnwindTableEntry());
 
   if (IsFunclet) {
     assert(HasFP && "EH funclets without FP not yet implemented");
     NumBytes = getWinEHFuncletFrameSize(MF);
+    int Ignore;
+    if (IsWin64Prologue)
+      NumBytes += X86FI->getCalleeSavedXMMFrameInfo(Ignore);
   } else if (HasFP) {
     // Calculate required stack adjustment.
     uint64_t FrameSize = StackSize - SlotSize;
     NumBytes = FrameSize - CSSize;
 
     // Callee-saved registers were pushed on stack before the stack was
     // realigned.
     if (TRI->needsStackRealignment(MF) && !IsWin64Prologue)

 
 bool X86FrameLowering::assignCalleeSavedSpillSlots(
     MachineFunction &MF, const TargetRegisterInfo *TRI,
     std::vector<CalleeSavedInfo> &CSI) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
   X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
 
   unsigned CalleeSavedFrameSize = 0;
+  unsigned CalleeSavedXMMFrameSize = 0;
+  int CalleeSavedXMMSlotOrigin = 0;
   int SpillSlotOffset = getOffsetOfLocalArea() + X86FI->getTCReturnAddrDelta();
 
   int64_t TailCallReturnAddrDelta = X86FI->getTCReturnAddrDelta();
 
   if (TailCallReturnAddrDelta < 0) {
     // create RETURNADDR area
     //   arg
     //   arg
for (unsigned i = CSI.size(); i != 0; --i) {
     int SlotIndex = MFI.CreateFixedSpillStackObject(SlotSize, SpillSlotOffset);
     CSI[i - 1].setFrameIdx(SlotIndex);
   }
 
   X86FI->setCalleeSavedFrameSize(CalleeSavedFrameSize);
   MFI.setCVBytesOfCalleeSavedRegisters(CalleeSavedFrameSize);
 
   // Assign slots for XMMs.
+  for (unsigned i = CSI.size(), Size = 0; i != 0; --i) {
+    unsigned Reg = CSI[i - 1].getReg();
+    // According to Microsoft "x64 software conventions", only XMM registers
+    // are nonvolatile except the GPR.
+    if (!X86::VR128RegClass.contains(Reg))
+      continue;
+    // Since all registers have the same size, we just initialize once
+    if (Size == 0) {
+      unsigned Align = TRI->getSpillAlignment(X86::VR128RegClass);
+      // ensure alignment
+      int Remainder = SpillSlotOffset % Align;
+      if (Remainder < 0)
+        SpillSlotOffset -= Align + Remainder;
+      else
+        SpillSlotOffset -= Remainder;
+      MFI.ensureMaxAlignment(Align);
+      Size = TRI->getSpillSize(X86::VR128RegClass);
+    }
+    // spill into slot
+    SpillSlotOffset -= Size;
+    int SlotIndex = MFI.CreateFixedSpillStackObject(Size, SpillSlotOffset);
+    CSI[i - 1].setFrameIdx(SlotIndex);
+    // Since we allocate XMM slot consecutively in stack, we just need to
+    // record the first one for the funclet use
+    if (CalleeSavedXMMFrameSize == 0) {
+      CalleeSavedXMMSlotOrigin = SlotIndex;
+    }
+    CalleeSavedXMMFrameSize += Size;
+  }
+
+  X86FI->setCalleeSavedXMMFrameInfo(CalleeSavedXMMFrameSize,
+                                    CalleeSavedXMMSlotOrigin);
+
+  // Assign slots for others.
   for (unsigned i = CSI.size(); i != 0; --i) {
     unsigned Reg = CSI[i - 1].getReg();
-    if (X86::GR64RegClass.contains(Reg) || X86::GR32RegClass.contains(Reg))
+    if (X86::GR64RegClass.contains(Reg) || X86::GR32RegClass.contains(Reg) ||
+        X86::VR128RegClass.contains(Reg))
       continue;
 
     // If this is k-register make sure we lookup via the largest legal type.
     MVT VT = MVT::Other;
     if (X86::VK16RegClass.contains(Reg))
       VT = STI.hasBWI() ? MVT::v64i1 : MVT::v16i1;
 
     const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg, VT);
     unsigned Size = TRI->getSpillSize(*RC);
     unsigned Align = TRI->getSpillAlignment(*RC);
     // ensure alignment
-    SpillSlotOffset -= std::abs(SpillSlotOffset) % Align;
+    int Remainder = SpillSlotOffset % Align;
+    if (Remainder < 0)
+      SpillSlotOffset -= Align + Remainder;
+    else
+      SpillSlotOffset -= Remainder;
     // spill into slot
     SpillSlotOffset -= Size;
     int SlotIndex = MFI.CreateFixedSpillStackObject(Size, SpillSlotOffset);
     CSI[i - 1].setFrameIdx(SlotIndex);
     MFI.ensureMaxAlignment(Align);
   }
 
   return true;
if (MI->getOpcode() == X86::CATCHRET) {
       if (IsSEH)
         return true;
     }
   }
 
   DebugLoc DL = MBB.findDebugLoc(MI);
 
   // Reload XMMs from stack frame.
+  MachineFunction &MF = *MBB.getParent();
+  X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
+  int XMMFrameSlotOrigin;
+  int SEHFrameOffset = X86FI->getCalleeSavedXMMFrameInfo(XMMFrameSlotOrigin) +
+                       MF.getFrameInfo().getMaxCallFrameSize();
   for (unsigned i = 0, e = CSI.size(); i != e; ++i) {
     unsigned Reg = CSI[i].getReg();
-    if (X86::GR64RegClass.contains(Reg) ||
-        X86::GR32RegClass.contains(Reg))
+    if (MBB.isEHFuncletEntry() && STI.is64Bit()) {
+      if (X86::FR64RegClass.contains(Reg)) {
+        int Offset = (CSI[i].getFrameIdx() - XMMFrameSlotOrigin - 1) * 16;
+        addRegOffset(BuildMI(MBB, MI, DL, TII.get(X86::MOVAPSrm), Reg),
+                     X86::RSP, true, SEHFrameOffset + Offset);
+      }
+    }
+    else {
+      if (X86::GR64RegClass.contains(Reg) || X86::GR32RegClass.contains(Reg))
         continue;
 
       // If this is k-register make sure we lookup via the largest legal type.
       MVT VT = MVT::Other;
       if (X86::VK16RegClass.contains(Reg))
         VT = STI.hasBWI() ? MVT::v64i1 : MVT::v16i1;
 
       const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg, VT);
       TII.loadRegFromStackSlot(MBB, MI, Reg, CSI[i].getFrameIdx(), RC, TRI);
     }
+  }
 
   // POP GPRs.
   unsigned Opc = STI.is64Bit() ? X86::POP64r : X86::POP32r;
   for (unsigned i = 0, e = CSI.size(); i != e; ++i) {
     unsigned Reg = CSI[i].getReg();
     if (!X86::GR64RegClass.contains(Reg) &&
         !X86::GR32RegClass.contains(Reg))
       continue;
 
     BuildMI(MBB, MI, DL, TII.get(Opc), Reg)
         .setMIFlag(MachineInstr::FrameDestroy);
void X86FrameLowering::processFunctionBeforeFrameFinalized(
   auto MBBI = MBB.begin();
   while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup))
     ++MBBI;
 
   DebugLoc DL = MBB.findDebugLoc(MBBI);
   addFrameReference(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV64mi32)),
                     UnwindHelpFI)
       .addImm(-2);
 }

llvm/lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   /// displacement from the frame pointer to a slot where the base pointer
   /// is stashed.
   signed char RestoreBasePointerOffset = 0;
 
   /// CalleeSavedFrameSize - Size of the callee-saved register portion of the
   /// stack frame in bytes.
   unsigned CalleeSavedFrameSize = 0;
 
+  /// CalleeSavedXMMFrameSize - Size of the callee-saved XMM register portion
+  /// of the stack frame in bytes.
+  unsigned CalleeSavedXMMFrameSize = 0;
+
+  /// CalleeSavedXMMFrameOrigin - Origin slot of the callee-saved XMM register
+  /// portion of the stack frame.
+  int CalleeSavedXMMFrameOrigin = 0;
+
   /// BytesToPopOnReturn - Number of bytes function pops on return (in addition
   /// to the space used by the return address).
   /// Used on windows platform for stdcall & fastcall name decoration
   unsigned BytesToPopOnReturn = 0;
 
   /// ReturnAddrIndex - FrameIndex for return slot.
   int ReturnAddrIndex = 0;
 
public:
 
   bool getRestoreBasePointer() const { return RestoreBasePointerOffset!=0; }
   void setRestoreBasePointer(const MachineFunction *MF);
   int getRestoreBasePointerOffset() const {return RestoreBasePointerOffset; }
 
   unsigned getCalleeSavedFrameSize() const { return CalleeSavedFrameSize; }
   void setCalleeSavedFrameSize(unsigned bytes) { CalleeSavedFrameSize = bytes; }
 
+  unsigned getCalleeSavedXMMFrameInfo(int &origin) const
+           { origin = CalleeSavedXMMFrameOrigin; return CalleeSavedXMMFrameSize; }
+  void setCalleeSavedXMMFrameInfo(unsigned size, int origin)
+       { CalleeSavedXMMFrameSize = size; CalleeSavedXMMFrameOrigin = origin; }
+
   unsigned getBytesToPopOnReturn() const { return BytesToPopOnReturn; }
   void setBytesToPopOnReturn (unsigned bytes) { BytesToPopOnReturn = bytes;}
 
   int getRAIndex() const { return ReturnAddrIndex; }
   void setRAIndex(int Index) { ReturnAddrIndex = Index; }
 
   int getFAIndex() const { return FrameAddrIndex; }
   void setFAIndex(int Index) { FrameAddrIndex = Index; }

llvm/test/CodeGen/X86/avx512-intel-ocl.ll

 ; WIN32-LABEL: test_prolog_epilog:
 ; WIN32:       # %bb.0:
 ; WIN32-NEXT:    calll _func_float16
 ; WIN32-NEXT:    retl
 ;
 ; WIN64-KNL-LABEL: test_prolog_epilog:
 ; WIN64-KNL:       # %bb.0:
 ; WIN64-KNL-NEXT:    pushq %rbp
-; WIN64-KNL-NEXT:    subq $1328, %rsp # imm = 0x530
+; WIN64-KNL-NEXT:    subq $1264, %rsp # imm = 0x4F0
 ; WIN64-KNL-NEXT:    leaq {{[0-9]+}}(%rsp), %rbp
-; WIN64-KNL-NEXT:    kmovw %k7, 1198(%rbp) # 2-byte Spill
-; WIN64-KNL-NEXT:    kmovw %k6, 1196(%rbp) # 2-byte Spill
-; WIN64-KNL-NEXT:    kmovw %k5, 1194(%rbp) # 2-byte Spill
-; WIN64-KNL-NEXT:    kmovw %k4, 1192(%rbp) # 2-byte Spill
-; WIN64-KNL-NEXT:    vmovaps %zmm21, 1104(%rbp) # 64-byte Spill
-; WIN64-KNL-NEXT:    vmovaps %zmm20, 992(%rbp) # 64-byte Spill
+; WIN64-KNL-NEXT:    kmovw %k7, 1134(%rbp) # 2-byte Spill
+; WIN64-KNL-NEXT:    kmovw %k6, 1132(%rbp) # 2-byte Spill
+; WIN64-KNL-NEXT:    kmovw %k5, 1130(%rbp) # 2-byte Spill
+; WIN64-KNL-NEXT:    kmovw %k4, 1128(%rbp) # 2-byte Spill
+; WIN64-KNL-NEXT:    vmovaps %zmm21, 1024(%rbp) # 64-byte Spill
+; WIN64-KNL-NEXT:    vmovaps %zmm20, 960(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm19, 896(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm18, 832(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm17, 768(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm16, 704(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm15, 640(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm14, 576(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm13, 512(%rbp) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm12, 448(%rbp) # 64-byte Spill

 ; WIN64-KNL-NEXT:    vmovaps 448(%rbp), %zmm12 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 512(%rbp), %zmm13 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 576(%rbp), %zmm14 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 640(%rbp), %zmm15 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 704(%rbp), %zmm16 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 768(%rbp), %zmm17 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 832(%rbp), %zmm18 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps 896(%rbp), %zmm19 # 64-byte Reload
-; WIN64-KNL-NEXT:    vmovaps 992(%rbp), %zmm20 # 64-byte Reload
-; WIN64-KNL-NEXT:    vmovaps 1104(%rbp), %zmm21 # 64-byte Reload
-; WIN64-KNL-NEXT:    kmovw 1192(%rbp), %k4 # 2-byte Reload
-; WIN64-KNL-NEXT:    kmovw 1194(%rbp), %k5 # 2-byte Reload
-; WIN64-KNL-NEXT:    kmovw 1196(%rbp), %k6 # 2-byte Reload
-; WIN64-KNL-NEXT:    kmovw 1198(%rbp), %k7 # 2-byte Reload
-; WIN64-KNL-NEXT:    leaq 1200(%rbp), %rsp
+; WIN64-KNL-NEXT:    vmovaps 960(%rbp), %zmm20 # 64-byte Reload
+; WIN64-KNL-NEXT:    vmovaps 1024(%rbp), %zmm21 # 64-byte Reload
+; WIN64-KNL-NEXT:    kmovw 1128(%rbp), %k4 # 2-byte Reload
+; WIN64-KNL-NEXT:    kmovw 1130(%rbp), %k5 # 2-byte Reload
+; WIN64-KNL-NEXT:    kmovw 1132(%rbp), %k6 # 2-byte Reload
+; WIN64-KNL-NEXT:    kmovw 1134(%rbp), %k7 # 2-byte Reload
+; WIN64-KNL-NEXT:    leaq 1136(%rbp), %rsp
 ; WIN64-KNL-NEXT:    popq %rbp
 ; WIN64-KNL-NEXT:    retq
 ;
 ; WIN64-SKX-LABEL: test_prolog_epilog:
 ; WIN64-SKX:       # %bb.0:
 ; WIN64-SKX-NEXT:    pushq %rbp
-; WIN64-SKX-NEXT:    subq $1328, %rsp # imm = 0x530
+; WIN64-SKX-NEXT:    subq $1264, %rsp # imm = 0x4F0
 ; WIN64-SKX-NEXT:    leaq {{[0-9]+}}(%rsp), %rbp
-; WIN64-SKX-NEXT:    kmovq %k7, 1192(%rbp) # 8-byte Spill
-; WIN64-SKX-NEXT:    kmovq %k6, 1184(%rbp) # 8-byte Spill
-; WIN64-SKX-NEXT:    kmovq %k5, 1176(%rbp) # 8-byte Spill
-; WIN64-SKX-NEXT:    kmovq %k4, 1168(%rbp) # 8-byte Spill
-; WIN64-SKX-NEXT:    vmovaps %zmm21, 1056(%rbp) # 64-byte Spill
+; WIN64-SKX-NEXT:    kmovq %k7, 1128(%rbp) # 8-byte Spill
+; WIN64-SKX-NEXT:    kmovq %k6, 1120(%rbp) # 8-byte Spill
+; WIN64-SKX-NEXT:    kmovq %k5, 1112(%rbp) # 8-byte Spill
+; WIN64-SKX-NEXT:    kmovq %k4, 1104(%rbp) # 8-byte Spill
+; WIN64-SKX-NEXT:    vmovaps %zmm21, 1024(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm20, 960(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm19, 896(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm18, 832(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm17, 768(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm16, 704(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm15, 640(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm14, 576(%rbp) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm13, 512(%rbp) # 64-byte Spill

 ; WIN64-SKX-NEXT:    vmovaps 512(%rbp), %zmm13 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 576(%rbp), %zmm14 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 640(%rbp), %zmm15 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 704(%rbp), %zmm16 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 768(%rbp), %zmm17 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 832(%rbp), %zmm18 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 896(%rbp), %zmm19 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps 960(%rbp), %zmm20 # 64-byte Reload
-; WIN64-SKX-NEXT:    vmovaps 1056(%rbp), %zmm21 # 64-byte Reload
-; WIN64-SKX-NEXT:    kmovq 1168(%rbp), %k4 # 8-byte Reload
-; WIN64-SKX-NEXT:    kmovq 1176(%rbp), %k5 # 8-byte Reload
-; WIN64-SKX-NEXT:    kmovq 1184(%rbp), %k6 # 8-byte Reload
-; WIN64-SKX-NEXT:    kmovq 1192(%rbp), %k7 # 8-byte Reload
-; WIN64-SKX-NEXT:    leaq 1200(%rbp), %rsp
+; WIN64-SKX-NEXT:    vmovaps 1024(%rbp), %zmm21 # 64-byte Reload
+; WIN64-SKX-NEXT:    kmovq 1104(%rbp), %k4 # 8-byte Reload
+; WIN64-SKX-NEXT:    kmovq 1112(%rbp), %k5 # 8-byte Reload
+; WIN64-SKX-NEXT:    kmovq 1120(%rbp), %k6 # 8-byte Reload
+; WIN64-SKX-NEXT:    kmovq 1128(%rbp), %k7 # 8-byte Reload
+; WIN64-SKX-NEXT:    leaq 1136(%rbp), %rsp
 ; WIN64-SKX-NEXT:    popq %rbp
 ; WIN64-SKX-NEXT:    retq
 ;
 ; X64-KNL-LABEL: test_prolog_epilog:
 ; X64-KNL:       ## %bb.0:
 ; X64-KNL-NEXT:    pushq %rsi
 ; X64-KNL-NEXT:    pushq %rdi
 ; X64-KNL-NEXT:    subq $1064, %rsp ## imm = 0x428

 ; X64-KNL-NEXT:    popq %rdi
 ; X64-KNL-NEXT:    popq %rsi
 ; X64-KNL-NEXT:    retq
 ;
 ; X64-SKX-LABEL: test_prolog_epilog:
 ; X64-SKX:       ## %bb.0:
 ; X64-SKX-NEXT:    pushq %rsi
 ; X64-SKX-NEXT:    pushq %rdi
-; X64-SKX-NEXT:    subq $1192, %rsp ## imm = 0x4A8
+; X64-SKX-NEXT:    subq $1064, %rsp ## imm = 0x428
 ; X64-SKX-NEXT:    kmovq %k7, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k6, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k5, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k4, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm31, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm30, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm29, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm28, {{[0-9]+}}(%rsp) ## 64-byte Spill

 ; X64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm28 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm29 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm30 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm31 ## 64-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k4 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k5 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k6 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k7 ## 8-byte Reload
-; X64-SKX-NEXT:    addq $1192, %rsp ## imm = 0x4A8
+; X64-SKX-NEXT:    addq $1064, %rsp ## imm = 0x428
 ; X64-SKX-NEXT:    popq %rdi
 ; X64-SKX-NEXT:    popq %rsi
 ; X64-SKX-NEXT:    retq
    %c = call <16 x float> @func_float16(<16 x float> %a, <16 x float> %b)
    ret <16 x float> %c
 }
 
 

llvm/test/CodeGen/X86/catchpad-realign-savexmm.ll

 ; CHECK: callq   g
 ; CHECK: .Ltmp{{.*}}
 ; CHECK: .LBB{{.*}} # Block address taken
 ; CHECK: movaps  -16(%rbp), %xmm6
 ; CHECK: addq    $64, %rsp
 ; CHECK: popq    %rbp
 ; CHECK: retq
 ; CHECK: .seh_handlerdata
+; CHECK: # %catch
+; CHECK: movq    %rdx, 16(%rsp)
+; CHECK: pushq   %rbp
+; CHECK: .seh_pushreg 5
+; CHECK: subq    $48, %rsp
+; CHECK: .seh_stackalloc 48
+; CHECK: leaq    64(%rdx), %rbp
+; CHECK: movapd  %xmm6, 32(%rsp)
+; CHECK: .seh_savexmm 6, 32
+; CHECK: .seh_endprologue
+; CHECK: movapd  32(%rsp), %xmm6
+; CHECK: leaq    .LBB0_1(%rip), %rax
+; CHECK: addq    $48, %rsp
+; CHECK: popq    %rbp
+; CHECK: retq # CATCHRET

llvm/test/CodeGen/X86/win64-funclet-savexmm.ll

+; RUN: llc -mtriple=x86_64-pc-windows-msvc -mattr=+avx < %s | FileCheck %s
+
+; void foo(void)
+; {
+;   __asm("nop" ::: "bx", "cx", "xmm5", "xmm6", "ymm7");
+;   try {
+;     throw;
+;   }
+;   catch (int x) {
+;   }
+; }
+
+%rtti.TypeDescriptor2 = type { i8**, i8*, [3 x i8] }
+%eh.ThrowInfo = type { i32, i8*, i8*, i8* }
+
+$"??_R0H@8" = comdat any
+
+@"??_7type_info@@6B@" = external constant i8*
+@"??_R0H@8" = linkonce_odr global %rtti.TypeDescriptor2 { i8** @"??_7type_info@@6B@", i8* null, [3 x i8] c".H\00" }, comdat
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+declare dso_local x86_stdcallcc void @_CxxThrowException(i8*, %eh.ThrowInfo*)
+
+define dso_local void @"?foo@@YAXXZ"() personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %x = alloca i32, align 4
+  call void asm sideeffect "nop", "~{bx},~{cx},~{xmm5},~{xmm6},~{ymm7}"()
+  invoke void @_CxxThrowException(i8* null, %eh.ThrowInfo* null)
+          to label %unreachable unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %entry
+  %0 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %1 = catchpad within %0 [%rtti.TypeDescriptor2* @"??_R0H@8", i32 0, i32* %x]
+  catchret from %1 to label %catchret.dest
+
+catchret.dest:                                    ; preds = %catch
+  br label %try.cont
+
+try.cont:                                         ; preds = %catchret.dest
+  ret void
+
+unreachable:                                      ; preds = %entry
+  unreachable
+}
+
+; CHECK: # %catch
+; CHECK: movq    %rdx, 16(%rsp)
+; CHECK: pushq   %rbp
+; CHECK: .seh_pushreg 5
+; CHECK: pushq   %rbx
+; CHECK: .seh_pushreg 3
+; CHECK: subq    $72, %rsp
+; CHECK: .seh_stackalloc 72
+; CHECK: leaq    80(%rdx), %rbp
+; CHECK: movaps  %xmm7, 48(%rsp)
+; CHECK: .seh_savexmm 7, 48
+; CHECK: movaps  %xmm6, 32(%rsp)
+; CHECK: .seh_savexmm 6, 32
+; CHECK: .seh_endprologue
+; CHECK: movaps  32(%rsp), %xmm6
+; CHECK: movaps  48(%rsp), %xmm7
+; CHECK: leaq    .LBB0_3(%rip), %rax
+; CHECK: addq    $72, %rsp
+; CHECK: popq    %rbx
+; CHECK: popq    %rbp
+; CHECK: retq # CATCHRET

llvm/test/CodeGen/X86/x86-interrupt_cc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=x86_64-apple-macosx -show-mc-encoding -mcpu=knl < %s | FileCheck %s -check-prefix=CHECK64 -check-prefix=CHECK64-KNL
 ; RUN: llc -verify-machineinstrs -mtriple=x86_64-apple-macosx -show-mc-encoding -mcpu=skx < %s | FileCheck %s -check-prefix=CHECK64 -check-prefix=CHECK64-SKX
 ; RUN: llc -verify-machineinstrs -mtriple=i386-apple-macosx -show-mc-encoding -mcpu=knl < %s | FileCheck %s -check-prefix=CHECK32 -check-prefix=CHECK32-KNL
 ; RUN: llc -verify-machineinstrs -mtriple=i386-apple-macosx -show-mc-encoding -mcpu=skx < %s | FileCheck %s -check-prefix=CHECK32 -check-prefix=CHECK32-SKX
 
 ; Make sure we spill the high numbered zmm registers and K registers with the right encoding.
 
 define x86_intrcc void @foo(i8* %frame) {

 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x9c,0x24,0x48,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k2, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x94,0x24,0x40,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k1, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x8c,0x24,0x38,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k0, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x84,0x24,0x30,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    vmovups %zmm31, {{[0-9]+}}(%rsp) ## 64-byte Spill
-; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0xbc,0x24,0xe0,0x07,0x00,0x00]
+; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x7c,0x24,0x1f]
 ; CHECK64-SKX-NEXT:    vmovups %zmm30, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x74,0x24,0x1e]
 ; CHECK64-SKX-NEXT:    vmovups %zmm29, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x6c,0x24,0x1d]
 ; CHECK64-SKX-NEXT:    vmovups %zmm28, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x64,0x24,0x1c]
 ; CHECK64-SKX-NEXT:    vmovups %zmm27, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x5c,0x24,0x1b]

 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm23, -768
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm24, -704
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm25, -640
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm26, -576
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm27, -512
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm28, -448
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm29, -384
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm30, -320
-; CHECK64-SKX-NEXT:    .cfi_offset %xmm31, -224
+; CHECK64-SKX-NEXT:    .cfi_offset %xmm31, -256
 ; CHECK64-SKX-NEXT:    .cfi_offset %k0, -144
 ; CHECK64-SKX-NEXT:    .cfi_offset %k1, -136
 ; CHECK64-SKX-NEXT:    .cfi_offset %k2, -128
 ; CHECK64-SKX-NEXT:    .cfi_offset %k3, -120
 ; CHECK64-SKX-NEXT:    .cfi_offset %k4, -112
 ; CHECK64-SKX-NEXT:    .cfi_offset %k5, -104
 ; CHECK64-SKX-NEXT:    .cfi_offset %k6, -96
 ; CHECK64-SKX-NEXT:    .cfi_offset %k7, -88

 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x5c,0x24,0x1b]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm28 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x64,0x24,0x1c]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm29 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x6c,0x24,0x1d]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm30 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x74,0x24,0x1e]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm31 ## 64-byte Reload
-; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0xbc,0x24,0xe0,0x07,0x00,0x00]
+; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x7c,0x24,0x1f]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k0 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x84,0x24,0x30,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k1 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x8c,0x24,0x38,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k2 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x94,0x24,0x40,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k3 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x9c,0x24,0x48,0x08,0x00,0x00]

 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x9c,0x24,0x48,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k2, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x94,0x24,0x40,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k1, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x8c,0x24,0x38,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k0, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x84,0x24,0x30,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    vmovups %zmm7, {{[0-9]+}}(%esp) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xbc,0x24,0xe0,0x01,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x7c,0x24,0x07]
 ; CHECK32-SKX-NEXT:    vmovups %zmm6, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x74,0x24,0x06]
 ; CHECK32-SKX-NEXT:    vmovups %zmm5, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x6c,0x24,0x05]
 ; CHECK32-SKX-NEXT:    vmovups %zmm4, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x64,0x24,0x04]
 ; CHECK32-SKX-NEXT:    vmovups %zmm3, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x5c,0x24,0x03]

 ; CHECK32-SKX-NEXT:    .cfi_offset %edx, -8
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm0, -640
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm1, -576
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm2, -512
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm3, -448
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm4, -384
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm5, -320
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm6, -256
-; CHECK32-SKX-NEXT:    .cfi_offset %xmm7, -160
+; CHECK32-SKX-NEXT:    .cfi_offset %xmm7, -192
 ; CHECK32-SKX-NEXT:    .cfi_offset %k0, -80
 ; CHECK32-SKX-NEXT:    .cfi_offset %k1, -72
 ; CHECK32-SKX-NEXT:    .cfi_offset %k2, -64
 ; CHECK32-SKX-NEXT:    .cfi_offset %k3, -56
 ; CHECK32-SKX-NEXT:    .cfi_offset %k4, -48
 ; CHECK32-SKX-NEXT:    .cfi_offset %k5, -40
 ; CHECK32-SKX-NEXT:    .cfi_offset %k6, -32
 ; CHECK32-SKX-NEXT:    .cfi_offset %k7, -24

 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x5c,0x24,0x03]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm4 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x64,0x24,0x04]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm5 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x6c,0x24,0x05]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm6 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x74,0x24,0x06]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm7 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xbc,0x24,0xe0,0x01,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x7c,0x24,0x07]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k0 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x84,0x24,0x30,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k1 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x8c,0x24,0x38,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k2 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x94,0x24,0x40,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k3 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x9c,0x24,0x48,0x02,0x00,0x00]