[WinEH] Allocate space in funclets stack to save XMM CSRs

[pengfei]

Summary

This is an alternate approach to D63396

Currently funclets reuse the same stack slots that are used in the
parent function for saving callee-saved xmm registers. If the parent
function modifies a callee-saved xmm register before an excpetion is
thrown, the catch handler will overwrite the original saved value.

This patch allocates space in funclets stack for saving callee-saved xmm
registers and uses RSP instead RBP to access memory.

Signed-off-by: Pengfei Wang <[e-mail]>

[rnk]

Thanks, this seems more straightforward. :)

llvm/lib/Target/X86/X86FrameLowering.cpp

1397–1398 ↗

(On Diff #216627)

I think this else can be simplified by using getFrameIndexReferenceSP. The .seh_savexmm directive expects an offset relative to SP-after-prologue, but getFrameIndexReference may use an RBP offset, and the SEHFrameOffset is the difference between them at this point. Making the preference for RSP offsets explicit seems clearer.

At first I wanted to find a way to eliminate the need for the if/else here, but I see it's not possible without a "prefer SP" variant to the new XMM frame index ref helper.

1862 ↗

(On Diff #216627)

This calculation is only correct in funclets. I think what I had in mind was that we'd pass in IsFunclet to this method, since all the callers have to calculate it anyway. That way we avoid one more conditional at the call site.

Is 32 always correct? Shouldn't it be MFI.getMaxCallFrameSize()?

2042 ↗

(On Diff #216627)

What is the desired change in behavior here? It looks like this can be simplified. SpillSlotOffset is always negative and Align is always positive, so I would expect Remainder to always be zero or negative.

Would this expression calculate the same value?

SpillSlotOffset = -alignTo(-SpillSlotOffset, Align);

2054 ↗

(On Diff #216627)

"for funclets"

llvm/lib/Target/X86/X86MachineFunctionInfo.h

41 ↗

(On Diff #216627)

Perhaps initialize it to INT32_MIN for safety?

llvm/lib/Target/X86/X86RegisterInfo.cpp

713 ↗

(On Diff #216627)

I would expect that for a multi-bb cleanup funclet, MBB.isEHFuncletEntry() wouldn't be sufficient, and you'd need to check if the terminator is a funclet return.

[pengfei]

Address review comments.

[pengfei]

Thanks for the patient review and valuable suggestions.

llvm/lib/Target/X86/X86FrameLowering.cpp

1397–1398 ↗

(On Diff #216627)

The return value of getFrameIndexReferenceSP(MF, FI, IgnoredFrameReg, SEHFrameOffset) has 8 bytes gap as against the former math. It may need more code to eliminate the gap. So it's better to keep it.

1862 ↗

(On Diff #216627)

Because we always need condition IsFunclet to select this method in caller emitPrologue.
I changed the function name to getWin64EHFrameIndexRef, which may be more appropriate.

Right, MFI.getMaxCallFrameSize() is more robust.

2042 ↗

(On Diff #216627)

The former code doesn't align it in some cases, e.g. SpillSlotOffset = -8 and Align = 64.

Yes, it can be simplified to this. Thanks!

llvm/lib/Target/X86/X86MachineFunctionInfo.h

41 ↗

(On Diff #216627)

I have used a FI->Slot map to replace this. It's more robust and flexible.

llvm/lib/Target/X86/X86RegisterInfo.cpp

713 ↗

(On Diff #216627)

You are correct. Thanks for the reminding!

[pengfei]

Fix the error in calculating funclet stack pointer.

[rnk]

lgtm, thanks!

llvm/lib/Target/X86/X86RegisterInfo.cpp

713 ↗

(On Diff #216989)

I confirmed this is O(#terminators) instead of O(#instrs), so this isn't O(n^2) in BB size. :)

llvm/trunk/lib/Target/X86/X86FrameLowering.h

public:
   bool hasFP(const MachineFunction &MF) const override;
   bool hasReservedCallFrame(const MachineFunction &MF) const override;
   bool canSimplifyCallFramePseudos(const MachineFunction &MF) const override;
   bool needsFrameIndexResolution(const MachineFunction &MF) const override;
 
   int getFrameIndexReference(const MachineFunction &MF, int FI,
                              unsigned &FrameReg) const override;
 
+  int getWin64EHFrameIndexRef(const MachineFunction &MF,
+                              int FI, unsigned &SPReg) const;
   int getFrameIndexReferenceSP(const MachineFunction &MF,
                                int FI, unsigned &SPReg, int Adjustment) const;
   int getFrameIndexReferencePreferSP(const MachineFunction &MF, int FI,
                                      unsigned &FrameReg,
                                      bool IgnoreSPUpdates) const override;
 
   MachineBasicBlock::iterator
   eliminateCallFramePseudoInstr(MachineFunction &MF, MachineBasicBlock &MBB,

llvm/trunk/lib/Target/X86/X86FrameLowering.cpp

void X86FrameLowering::emitPrologue(MachineFunction &MF,
   while (MBBI != MBB.end() && MBBI->getFlag(MachineInstr::FrameSetup)) {
     const MachineInstr &FrameInstr = *MBBI;
     ++MBBI;
 
     if (NeedsWinCFI) {
       int FI;
       if (unsigned Reg = TII.isStoreToStackSlot(FrameInstr, FI)) {
         if (X86::FR64RegClass.contains(Reg)) {
+          int Offset;
           unsigned IgnoredFrameReg;
-          int Offset = getFrameIndexReference(MF, FI, IgnoredFrameReg);
-          Offset += SEHFrameOffset;
+          if (IsWin64Prologue && IsFunclet)
+            Offset = getWin64EHFrameIndexRef(MF, FI, IgnoredFrameReg);
+          else
+            Offset = getFrameIndexReference(MF, FI, IgnoredFrameReg) +
+                     SEHFrameOffset;
 
           HasWinCFI = true;
           assert(!NeedsWinFPO && "SEH_SaveXMM incompatible with FPO data");
           BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_SaveXMM))
               .addImm(Reg)
               .addImm(Offset)
               .setMIFlag(MachineInstr::FrameSetup);
         }
X86FrameLowering::getPSPSlotOffsetFromSP(const MachineFunction &MF) const {
   int Offset = getFrameIndexReferencePreferSP(MF, Info.PSPSymFrameIdx, SPReg,
                                               /*IgnoreSPUpdates*/ true);
   assert(Offset >= 0 && SPReg == TRI->getStackRegister());
   return static_cast<unsigned>(Offset);
 }
 
 unsigned
 X86FrameLowering::getWinEHFuncletFrameSize(const MachineFunction &MF) const {
+  const X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
   // This is the size of the pushed CSRs.
-  unsigned CSSize =
-      MF.getInfo<X86MachineFunctionInfo>()->getCalleeSavedFrameSize();
+  unsigned CSSize = X86FI->getCalleeSavedFrameSize();
+  // This is the size of callee saved XMMs.
+  const auto& WinEHXMMSlotInfo = X86FI->getWinEHXMMSlotInfo();
+  unsigned XMMSize = WinEHXMMSlotInfo.size() *
+                     TRI->getSpillSize(X86::VR128RegClass);
   // This is the amount of stack a funclet needs to allocate.
   unsigned UsedSize;
   EHPersonality Personality =
       classifyEHPersonality(MF.getFunction().getPersonalityFn());
   if (Personality == EHPersonality::CoreCLR) {
     // CLR funclets need to hold enough space to include the PSPSym, at the
     // same offset from the stack pointer (immediately after the prolog) as it
     // resides at in the main function.
     UsedSize = getPSPSlotOffsetFromSP(MF) + SlotSize;
   } else {
     // Other funclets just need enough stack for outgoing call arguments.
     UsedSize = MF.getFrameInfo().getMaxCallFrameSize();
   }
   // RBP is not included in the callee saved register block. After pushing RBP,
   // everything is 16 byte aligned. Everything we allocate before an outgoing
   // call must also be 16 byte aligned.
   unsigned FrameSizeMinusRBP = alignTo(CSSize + UsedSize, getStackAlignment());
   // Subtract out the size of the callee saved registers. This is how much stack
   // each funclet will allocate.
-  return FrameSizeMinusRBP - CSSize;
+  return FrameSizeMinusRBP + XMMSize - CSSize;
 }
 
 static bool isTailCallOpcode(unsigned Opc) {
     return Opc == X86::TCRETURNri || Opc == X86::TCRETURNdi ||
         Opc == X86::TCRETURNmi ||
         Opc == X86::TCRETURNri64 || Opc == X86::TCRETURNdi64 ||
         Opc == X86::TCRETURNmi64;
 }
if (TRI->hasBasePointer(MF)) {
     int TailCallReturnAddrDelta = X86FI->getTCReturnAddrDelta();
     if (TailCallReturnAddrDelta < 0)
       Offset -= TailCallReturnAddrDelta;
   }
 
   return Offset + FPDelta;
 }
 
+int X86FrameLowering::getWin64EHFrameIndexRef(const MachineFunction &MF,
+                                              int FI, unsigned &FrameReg) const {
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
+  const auto& WinEHXMMSlotInfo = X86FI->getWinEHXMMSlotInfo();
+  const auto it = WinEHXMMSlotInfo.find(FI);
+
+  if (it == WinEHXMMSlotInfo.end())
+    return getFrameIndexReference(MF, FI, FrameReg);
+
+  FrameReg = TRI->getStackRegister();
+  return alignTo(MFI.getMaxCallFrameSize(), getStackAlignment()) + it->second;
+}
+
 int X86FrameLowering::getFrameIndexReferenceSP(const MachineFunction &MF,
                                                int FI, unsigned &FrameReg,
                                                int Adjustment) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   FrameReg = TRI->getStackRegister();
   return MFI.getObjectOffset(FI) - getOffsetOfLocalArea() + Adjustment;
 }
 

 
 bool X86FrameLowering::assignCalleeSavedSpillSlots(
     MachineFunction &MF, const TargetRegisterInfo *TRI,
     std::vector<CalleeSavedInfo> &CSI) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
   X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
 
   unsigned CalleeSavedFrameSize = 0;
+  unsigned XMMCalleeSavedFrameSize = 0;
+  auto &WinEHXMMSlotInfo = X86FI->getWinEHXMMSlotInfo();
   int SpillSlotOffset = getOffsetOfLocalArea() + X86FI->getTCReturnAddrDelta();
 
   int64_t TailCallReturnAddrDelta = X86FI->getTCReturnAddrDelta();
 
   if (TailCallReturnAddrDelta < 0) {
     // create RETURNADDR area
     //   arg
     //   arg
for (unsigned i = CSI.size(); i != 0; --i) {
     MVT VT = MVT::Other;
     if (X86::VK16RegClass.contains(Reg))
       VT = STI.hasBWI() ? MVT::v64i1 : MVT::v16i1;
 
     const TargetRegisterClass *RC = TRI->getMinimalPhysRegClass(Reg, VT);
     unsigned Size = TRI->getSpillSize(*RC);
     unsigned Align = TRI->getSpillAlignment(*RC);
     // ensure alignment
-    SpillSlotOffset -= std::abs(SpillSlotOffset) % Align;
+    assert(SpillSlotOffset < 0 && "SpillSlotOffset should always < 0 on X86");
+    SpillSlotOffset = -alignTo(-SpillSlotOffset, Align);
+
     // spill into slot
     SpillSlotOffset -= Size;
     int SlotIndex = MFI.CreateFixedSpillStackObject(Size, SpillSlotOffset);
     CSI[i - 1].setFrameIdx(SlotIndex);
     MFI.ensureMaxAlignment(Align);
+
+    // Save the start offset and size of XMM in stack frame for funclets.
+    if (X86::VR128RegClass.contains(Reg)) {
+      WinEHXMMSlotInfo[SlotIndex] = XMMCalleeSavedFrameSize;
+      XMMCalleeSavedFrameSize += Size;
+    }
   }
 
   return true;
 }
 
 bool X86FrameLowering::spillCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     const std::vector<CalleeSavedInfo> &CSI,

llvm/trunk/lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   bool ForceFramePointer = false;
 
   /// RestoreBasePointerOffset - Non-zero if the function has base pointer
   /// and makes call to llvm.eh.sjlj.setjmp. When non-zero, the value is a
   /// displacement from the frame pointer to a slot where the base pointer
   /// is stashed.
   signed char RestoreBasePointerOffset = 0;
 
+  /// WinEHXMMSlotInfo - Slot information of XMM registers in the stack frame
+  /// in bytes.
+  DenseMap<int, unsigned> WinEHXMMSlotInfo;
+
   /// CalleeSavedFrameSize - Size of the callee-saved register portion of the
   /// stack frame in bytes.
   unsigned CalleeSavedFrameSize = 0;
 
   /// BytesToPopOnReturn - Number of bytes function pops on return (in addition
   /// to the space used by the return address).
   /// Used on windows platform for stdcall & fastcall name decoration
   unsigned BytesToPopOnReturn = 0;
public:
 
   bool getHasPushSequences() const { return HasPushSequences; }
   void setHasPushSequences(bool HasPush) { HasPushSequences = HasPush; }
 
   bool getRestoreBasePointer() const { return RestoreBasePointerOffset!=0; }
   void setRestoreBasePointer(const MachineFunction *MF);
   int getRestoreBasePointerOffset() const {return RestoreBasePointerOffset; }
 
+  DenseMap<int, unsigned>& getWinEHXMMSlotInfo() { return WinEHXMMSlotInfo; }
+  const DenseMap<int, unsigned>& getWinEHXMMSlotInfo() const {
+    return WinEHXMMSlotInfo; }
+
   unsigned getCalleeSavedFrameSize() const { return CalleeSavedFrameSize; }
   void setCalleeSavedFrameSize(unsigned bytes) { CalleeSavedFrameSize = bytes; }
 
   unsigned getBytesToPopOnReturn() const { return BytesToPopOnReturn; }
   void setBytesToPopOnReturn (unsigned bytes) { BytesToPopOnReturn = bytes;}
 
   int getRAIndex() const { return ReturnAddrIndex; }
   void setRAIndex(int Index) { ReturnAddrIndex = Index; }

llvm/trunk/lib/Target/X86/X86RegisterInfo.cpp

static bool tryOptimizeLEAtoMOV(MachineBasicBlock::iterator II) {
   const X86InstrInfo *TII =
       MI.getParent()->getParent()->getSubtarget<X86Subtarget>().getInstrInfo();
   TII->copyPhysReg(*MI.getParent(), II, MI.getDebugLoc(), NewDestReg, BasePtr,
                    MI.getOperand(1).isKill());
   MI.eraseFromParent();
   return true;
 }
 
+static bool isFuncletReturnInstr(MachineInstr &MI) {
+  switch (MI.getOpcode()) {
+  case X86::CATCHRET:
+  case X86::CLEANUPRET:
+    return true;
+  default:
+    return false;
+  }
+  llvm_unreachable("impossible");
+}
+
 void
 X86RegisterInfo::eliminateFrameIndex(MachineBasicBlock::iterator II,
                                      int SPAdj, unsigned FIOperandNum,
                                      RegScavenger *RS) const {
   MachineInstr &MI = *II;
-  MachineFunction &MF = *MI.getParent()->getParent();
+  MachineBasicBlock &MBB = *MI.getParent();
+  MachineFunction &MF = *MBB.getParent();
+  MachineBasicBlock::iterator MBBI = MBB.getFirstTerminator();
+  bool IsEHFuncletEpilogue = MBBI == MBB.end() ? false
+                                               : isFuncletReturnInstr(*MBBI);
   const X86FrameLowering *TFI = getFrameLowering(MF);
   int FrameIndex = MI.getOperand(FIOperandNum).getIndex();
 
   // Determine base register and offset.
   int FIOffset;
   unsigned BasePtr;
   if (MI.isReturn()) {
     assert((!needsStackRealignment(MF) ||
            MF.getFrameInfo().isFixedObjectIndex(FrameIndex)) &&
            "Return instruction can only reference SP relative frame objects");
     FIOffset = TFI->getFrameIndexReferenceSP(MF, FrameIndex, BasePtr, 0);
+  } else if (TFI->Is64Bit && (MBB.isEHFuncletEntry() || IsEHFuncletEpilogue)) {
+    FIOffset = TFI->getWin64EHFrameIndexRef(MF, FrameIndex, BasePtr);
   } else {
     FIOffset = TFI->getFrameIndexReference(MF, FrameIndex, BasePtr);
   }
 
   // LOCAL_ESCAPE uses a single offset, with no register. It only works in the
   // simple FP case, and doesn't work with stack realignment. On 32-bit, the
   // offset is from the traditional base pointer location.  On 64-bit, the
   // offset is from the SP at the end of the prologue, not the FP location. This

llvm/trunk/test/CodeGen/X86/avx512-intel-ocl.ll

 ; WIN32-LABEL: test_prolog_epilog:
 ; WIN32:       # %bb.0:
 ; WIN32-NEXT:    calll _func_float16
 ; WIN32-NEXT:    retl
 ;
 ; WIN64-KNL-LABEL: test_prolog_epilog:
 ; WIN64-KNL:       # %bb.0:
 ; WIN64-KNL-NEXT:    pushq %rbp
-; WIN64-KNL-NEXT:    subq $1328, %rsp # imm = 0x530
+; WIN64-KNL-NEXT:    subq $1264, %rsp # imm = 0x4F0
 ; WIN64-KNL-NEXT:    leaq {{[0-9]+}}(%rsp), %rbp
 ; WIN64-KNL-NEXT:    kmovw %k7, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
 ; WIN64-KNL-NEXT:    kmovw %k6, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
 ; WIN64-KNL-NEXT:    kmovw %k5, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
 ; WIN64-KNL-NEXT:    kmovw %k4, {{[-0-9]+}}(%r{{[sb]}}p) # 2-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm21, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm20, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
 ; WIN64-KNL-NEXT:    vmovaps %zmm19, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill

 ; WIN64-KNL-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm18 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm19 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm20 # 64-byte Reload
 ; WIN64-KNL-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm21 # 64-byte Reload
 ; WIN64-KNL-NEXT:    kmovw {{[-0-9]+}}(%r{{[sb]}}p), %k4 # 2-byte Reload
 ; WIN64-KNL-NEXT:    kmovw {{[-0-9]+}}(%r{{[sb]}}p), %k5 # 2-byte Reload
 ; WIN64-KNL-NEXT:    kmovw {{[-0-9]+}}(%r{{[sb]}}p), %k6 # 2-byte Reload
 ; WIN64-KNL-NEXT:    kmovw {{[-0-9]+}}(%r{{[sb]}}p), %k7 # 2-byte Reload
-; WIN64-KNL-NEXT:    leaq 1200(%rbp), %rsp
+; WIN64-KNL-NEXT:    leaq 1136(%rbp), %rsp
 ; WIN64-KNL-NEXT:    popq %rbp
 ; WIN64-KNL-NEXT:    retq
 ;
 ; WIN64-SKX-LABEL: test_prolog_epilog:
 ; WIN64-SKX:       # %bb.0:
 ; WIN64-SKX-NEXT:    pushq %rbp
-; WIN64-SKX-NEXT:    subq $1328, %rsp # imm = 0x530
+; WIN64-SKX-NEXT:    subq $1264, %rsp # imm = 0x4F0
 ; WIN64-SKX-NEXT:    leaq {{[0-9]+}}(%rsp), %rbp
 ; WIN64-SKX-NEXT:    kmovq %k7, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; WIN64-SKX-NEXT:    kmovq %k6, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; WIN64-SKX-NEXT:    kmovq %k5, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; WIN64-SKX-NEXT:    kmovq %k4, {{[-0-9]+}}(%r{{[sb]}}p) # 8-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm21, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm20, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill
 ; WIN64-SKX-NEXT:    vmovaps %zmm19, {{[-0-9]+}}(%r{{[sb]}}p) # 64-byte Spill

 ; WIN64-SKX-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm18 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm19 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm20 # 64-byte Reload
 ; WIN64-SKX-NEXT:    vmovaps {{[-0-9]+}}(%r{{[sb]}}p), %zmm21 # 64-byte Reload
 ; WIN64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k4 # 8-byte Reload
 ; WIN64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k5 # 8-byte Reload
 ; WIN64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k6 # 8-byte Reload
 ; WIN64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k7 # 8-byte Reload
-; WIN64-SKX-NEXT:    leaq 1200(%rbp), %rsp
+; WIN64-SKX-NEXT:    leaq 1136(%rbp), %rsp
 ; WIN64-SKX-NEXT:    popq %rbp
 ; WIN64-SKX-NEXT:    retq
 ;
 ; X64-KNL-LABEL: test_prolog_epilog:
 ; X64-KNL:       ## %bb.0:
 ; X64-KNL-NEXT:    pushq %rsi
 ; X64-KNL-NEXT:    pushq %rdi
 ; X64-KNL-NEXT:    subq $1064, %rsp ## imm = 0x428

 ; X64-KNL-NEXT:    popq %rdi
 ; X64-KNL-NEXT:    popq %rsi
 ; X64-KNL-NEXT:    retq
 ;
 ; X64-SKX-LABEL: test_prolog_epilog:
 ; X64-SKX:       ## %bb.0:
 ; X64-SKX-NEXT:    pushq %rsi
 ; X64-SKX-NEXT:    pushq %rdi
-; X64-SKX-NEXT:    subq $1192, %rsp ## imm = 0x4A8
+; X64-SKX-NEXT:    subq $1064, %rsp ## imm = 0x428
 ; X64-SKX-NEXT:    kmovq %k7, {{[-0-9]+}}(%r{{[sb]}}p) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k6, {{[-0-9]+}}(%r{{[sb]}}p) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k5, {{[-0-9]+}}(%r{{[sb]}}p) ## 8-byte Spill
 ; X64-SKX-NEXT:    kmovq %k4, {{[-0-9]+}}(%r{{[sb]}}p) ## 8-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm31, {{[-0-9]+}}(%r{{[sb]}}p) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm30, {{[-0-9]+}}(%r{{[sb]}}p) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm29, {{[-0-9]+}}(%r{{[sb]}}p) ## 64-byte Spill
 ; X64-SKX-NEXT:    vmovups %zmm28, {{[-0-9]+}}(%r{{[sb]}}p) ## 64-byte Spill

 ; X64-SKX-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %zmm28 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %zmm29 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %zmm30 ## 64-byte Reload
 ; X64-SKX-NEXT:    vmovups {{[-0-9]+}}(%r{{[sb]}}p), %zmm31 ## 64-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k4 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k5 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k6 ## 8-byte Reload
 ; X64-SKX-NEXT:    kmovq {{[-0-9]+}}(%r{{[sb]}}p), %k7 ## 8-byte Reload
-; X64-SKX-NEXT:    addq $1192, %rsp ## imm = 0x4A8
+; X64-SKX-NEXT:    addq $1064, %rsp ## imm = 0x428
 ; X64-SKX-NEXT:    popq %rdi
 ; X64-SKX-NEXT:    popq %rsi
 ; X64-SKX-NEXT:    retq
    %c = call <16 x float> @func_float16(<16 x float> %a, <16 x float> %b)
    ret <16 x float> %c
 }
 
 

llvm/trunk/test/CodeGen/X86/catchpad-realign-savexmm.ll

 ; CHECK: callq   g
 ; CHECK: .Ltmp{{.*}}
 ; CHECK: .LBB{{.*}} # Block address taken
 ; CHECK: movaps  -16(%rbp), %xmm6
 ; CHECK: addq    $64, %rsp
 ; CHECK: popq    %rbp
 ; CHECK: retq
 ; CHECK: .seh_handlerdata
+; CHECK: # %catch
+; CHECK: movq    %rdx, 16(%rsp)
+; CHECK: pushq   %rbp
+; CHECK: .seh_pushreg 5
+; CHECK: subq    $48, %rsp
+; CHECK: .seh_stackalloc 48
+; CHECK: leaq    64(%rdx), %rbp
+; CHECK: movapd  %xmm6, 32(%rsp)
+; CHECK: .seh_savexmm 6, 32
+; CHECK: .seh_endprologue
+; CHECK: movapd  32(%rsp), %xmm6
+; CHECK: leaq    .LBB0_1(%rip), %rax
+; CHECK: addq    $48, %rsp
+; CHECK: popq    %rbp
+; CHECK: retq # CATCHRET

llvm/trunk/test/CodeGen/X86/win64-funclet-savexmm.ll

+; RUN: llc -mtriple=x86_64-pc-windows-msvc -mattr=+avx < %s | FileCheck %s
+
+; void bar(int a, int b, int c, int d, int e);
+; void baz(int x);
+; 
+; void foo(int a, int b, int c, int d, int e)
+; {
+;   __asm("nop" ::: "bx", "cx", "xmm5", "xmm6", "ymm7");
+;   try {
+;     bar(a, b, c, d, e);
+;   }
+;   catch (...) {
+;     baz(a);
+;     if (a)
+;       __asm("nop" ::: "xmm8");
+;   }
+; }
+
+%rtti.TypeDescriptor2 = type { i8**, i8*, [3 x i8] }
+
+$"??_R0H@8" = comdat any
+
+@"??_7type_info@@6B@" = external constant i8*
+@"??_R0H@8" = linkonce_odr global %rtti.TypeDescriptor2 { i8** @"??_7type_info@@6B@", i8* null, [3 x i8] c".H\00" }, comdat
+
+declare dso_local i32 @__CxxFrameHandler3(...)
+declare dso_local void @"?bar@@YAXHHHHH@Z"(i32, i32, i32, i32, i32)
+declare dso_local void @"?baz@@YAXH@Z"(i32)
+
+define dso_local void @"?foo@@YAXHHHHH@Z"(i32 %a, i32 %b, i32 %c, i32 %d, i32 %e) personality i8* bitcast (i32 (...)* @__CxxFrameHandler3 to i8*) {
+entry:
+  %e.addr = alloca i32, align 4
+  %d.addr = alloca i32, align 4
+  %c.addr = alloca i32, align 4
+  %b.addr = alloca i32, align 4
+  %a.addr = alloca i32, align 4
+  store i32 %e, i32* %e.addr, align 4
+  store i32 %d, i32* %d.addr, align 4
+  store i32 %c, i32* %c.addr, align 4
+  store i32 %b, i32* %b.addr, align 4
+  store i32 %a, i32* %a.addr, align 4
+  call void asm sideeffect "nop", "~{bx},~{cx},~{xmm5},~{xmm6},~{ymm7}"()
+  %0 = load i32, i32* %e.addr, align 4
+  %1 = load i32, i32* %d.addr, align 4
+  %2 = load i32, i32* %c.addr, align 4
+  %3 = load i32, i32* %b.addr, align 4
+  %4 = load i32, i32* %a.addr, align 4
+  invoke void @"?bar@@YAXHHHHH@Z"(i32 %4, i32 %3, i32 %2, i32 %1, i32 %0)
+          to label %invoke.cont unwind label %catch.dispatch
+
+catch.dispatch:                                   ; preds = %entry
+  %5 = catchswitch within none [label %catch] unwind to caller
+
+catch:                                            ; preds = %catch.dispatch
+  %6 = catchpad within %5 [i8* null, i32 64, i8* null]
+  %7 = load i32, i32* %a.addr, align 4
+  call void @"?baz@@YAXH@Z"(i32 %7) [ "funclet"(token %6) ]
+  %8 = load i32, i32* %a.addr, align 4
+  %tobool = icmp ne i32 %8, 0
+  br i1 %tobool, label %if.then, label %if.end
+
+if.then:                                          ; preds = %catch
+  call void asm sideeffect "nop", "~{xmm8}"() [ "funclet"(token %6) ]
+  br label %if.end
+
+invoke.cont:                                      ; preds = %entry
+  br label %try.cont
+
+if.end:                                           ; preds = %if.then, %catch
+  catchret from %6 to label %catchret.dest
+
+catchret.dest:                                    ; preds = %if.end
+  br label %try.cont
+
+try.cont:                                         ; preds = %catchret.dest, %invoke.cont
+  ret void
+}
+
+; CHECK: # %catch
+; CHECK: movq    %rdx, 16(%rsp)
+; CHECK: pushq   %rbp
+; CHECK: .seh_pushreg 5
+; CHECK: pushq   %rbx
+; CHECK: .seh_pushreg 3
+; CHECK: subq    $88, %rsp
+; CHECK: .seh_stackalloc 88
+; CHECK: leaq    112(%rdx), %rbp
+; CHECK: vmovaps %xmm8, 48(%rsp)
+; CHECK: .seh_savexmm 8, 48
+; CHECK: vmovaps %xmm7, 64(%rsp)
+; CHECK: .seh_savexmm 7, 64
+; CHECK: vmovaps %xmm6, 80(%rsp)
+; CHECK: .seh_savexmm 6, 80
+; CHECK: .seh_endprologue
+; CHECK: movl   -{{[0-9]+}}(%rbp), %ecx
+; CHECK: vmovaps 80(%rsp), %xmm6
+; CHECK: vmovaps 64(%rsp), %xmm7
+; CHECK: vmovaps 48(%rsp), %xmm8
+; CHECK: leaq    .LBB0_1(%rip), %rax
+; CHECK: addq    $88, %rsp
+; CHECK: popq    %rbx
+; CHECK: popq    %rbp
+; CHECK: retq # CATCHRET
+
+; CHECK-LABEL: "$handlerMap$0$?foo@@YAXHHHHH@Z":
+; CHECK-NEXT: .long   64                      # Adjectives
+; CHECK-NEXT: .long   0                       # Type
+; CHECK-NEXT: .long   0                       # CatchObjOffset
+; CHECK-NEXT: .long   "?catch$2@?0??foo@@YAXHHHHH@Z@4HA"@IMGREL # Handler
+; Sum of:
+;   16 RDX store offset
+;   16 two pushes
+;   72 stack alloc
+; CHECK-NEXT: .long   120                     # ParentFrameOffset
+

llvm/trunk/test/CodeGen/X86/x86-interrupt_cc.ll

 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x9c,0x24,0x48,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k2, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x94,0x24,0x40,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k1, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x8c,0x24,0x38,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq %k0, {{[0-9]+}}(%rsp) ## 8-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x84,0x24,0x30,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    vmovups %zmm31, {{[0-9]+}}(%rsp) ## 64-byte Spill
-; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0xbc,0x24,0xe0,0x07,0x00,0x00]
+; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x7c,0x24,0x1f]
 ; CHECK64-SKX-NEXT:    vmovups %zmm30, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x74,0x24,0x1e]
 ; CHECK64-SKX-NEXT:    vmovups %zmm29, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x6c,0x24,0x1d]
 ; CHECK64-SKX-NEXT:    vmovups %zmm28, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x64,0x24,0x1c]
 ; CHECK64-SKX-NEXT:    vmovups %zmm27, {{[0-9]+}}(%rsp) ## 64-byte Spill
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x11,0x5c,0x24,0x1b]

 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm23, -768
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm24, -704
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm25, -640
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm26, -576
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm27, -512
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm28, -448
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm29, -384
 ; CHECK64-SKX-NEXT:    .cfi_offset %xmm30, -320
-; CHECK64-SKX-NEXT:    .cfi_offset %xmm31, -224
+; CHECK64-SKX-NEXT:    .cfi_offset %xmm31, -256
 ; CHECK64-SKX-NEXT:    .cfi_offset %k0, -144
 ; CHECK64-SKX-NEXT:    .cfi_offset %k1, -136
 ; CHECK64-SKX-NEXT:    .cfi_offset %k2, -128
 ; CHECK64-SKX-NEXT:    .cfi_offset %k3, -120
 ; CHECK64-SKX-NEXT:    .cfi_offset %k4, -112
 ; CHECK64-SKX-NEXT:    .cfi_offset %k5, -104
 ; CHECK64-SKX-NEXT:    .cfi_offset %k6, -96
 ; CHECK64-SKX-NEXT:    .cfi_offset %k7, -88

 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x5c,0x24,0x1b]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm28 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x64,0x24,0x1c]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm29 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x6c,0x24,0x1d]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm30 ## 64-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x74,0x24,0x1e]
 ; CHECK64-SKX-NEXT:    vmovups {{[0-9]+}}(%rsp), %zmm31 ## 64-byte Reload
-; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0xbc,0x24,0xe0,0x07,0x00,0x00]
+; CHECK64-SKX-NEXT:    ## encoding: [0x62,0x61,0x7c,0x48,0x10,0x7c,0x24,0x1f]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k0 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x84,0x24,0x30,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k1 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x8c,0x24,0x38,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k2 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x94,0x24,0x40,0x08,0x00,0x00]
 ; CHECK64-SKX-NEXT:    kmovq {{[0-9]+}}(%rsp), %k3 ## 8-byte Reload
 ; CHECK64-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x9c,0x24,0x48,0x08,0x00,0x00]

 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x9c,0x24,0x48,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k2, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x94,0x24,0x40,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k1, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x8c,0x24,0x38,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq %k0, {{[0-9]+}}(%esp) ## 8-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x84,0x24,0x30,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    vmovups %zmm7, {{[0-9]+}}(%esp) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xbc,0x24,0xe0,0x01,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x7c,0x24,0x07]
 ; CHECK32-SKX-NEXT:    vmovups %zmm6, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x74,0x24,0x06]
 ; CHECK32-SKX-NEXT:    vmovups %zmm5, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x6c,0x24,0x05]
 ; CHECK32-SKX-NEXT:    vmovups %zmm4, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x64,0x24,0x04]
 ; CHECK32-SKX-NEXT:    vmovups %zmm3, {{[0-9]+}}(%esp) ## 64-byte Spill
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x5c,0x24,0x03]

 ; CHECK32-SKX-NEXT:    .cfi_offset %edx, -8
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm0, -640
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm1, -576
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm2, -512
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm3, -448
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm4, -384
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm5, -320
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm6, -256
-; CHECK32-SKX-NEXT:    .cfi_offset %xmm7, -160
+; CHECK32-SKX-NEXT:    .cfi_offset %xmm7, -192
 ; CHECK32-SKX-NEXT:    .cfi_offset %k0, -80
 ; CHECK32-SKX-NEXT:    .cfi_offset %k1, -72
 ; CHECK32-SKX-NEXT:    .cfi_offset %k2, -64
 ; CHECK32-SKX-NEXT:    .cfi_offset %k3, -56
 ; CHECK32-SKX-NEXT:    .cfi_offset %k4, -48
 ; CHECK32-SKX-NEXT:    .cfi_offset %k5, -40
 ; CHECK32-SKX-NEXT:    .cfi_offset %k6, -32
 ; CHECK32-SKX-NEXT:    .cfi_offset %k7, -24

 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x5c,0x24,0x03]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm4 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x64,0x24,0x04]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm5 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x6c,0x24,0x05]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm6 ## 64-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x74,0x24,0x06]
 ; CHECK32-SKX-NEXT:    vmovups {{[0-9]+}}(%esp), %zmm7 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xbc,0x24,0xe0,0x01,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x7c,0x24,0x07]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k0 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x84,0x24,0x30,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k1 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x8c,0x24,0x38,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k2 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x94,0x24,0x40,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    kmovq {{[0-9]+}}(%esp), %k3 ## 8-byte Reload
 ; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x9c,0x24,0x48,0x02,0x00,0x00]