[X86] Align stack to 16-bytes on 32-bit with X86_INTR call convention

[antangelo]

Summary

Adds a dynamic stack alignment to functions under the interrupt call
convention on x86-32. This fixes the issue where the stack can be
misaligned on entry, since x86-32 makes no guarantees about the stack
pointer position when the interrupt service routine is called.

The alignment is done by overriding X86RegisterInfo::shouldRealignStack,
and by setting the correct alignment in X86FrameLowering::calculateMaxStackAlign.
This forces the interrupt handler to be dynamically aligned, generating
the appropriate and instruction in the prologue and lea in the
epilogue. The no-realign-stack attribute can be used as an opt-out.

Fixes #26851

[RKSimon]

Please can you rebase on rG56cdeac194b1926605def60376baeb5f001a53e8

llvm/test/CodeGen/X86/x86-32-intrcc.ll

52–54

Update all these descriptions

[pengfei]

llvm/lib/Target/X86/X86FrameLowering.cpp

1248

I think max is enough. We don't have no-power-of-2 alginment.

1248

Where's the 16 request from, ABI?

llvm/test/CodeGen/X86/x86-32-intrcc.ll

11–13

This seems conflict with the intention here.

51–54

ditto.

146–173

This should not be affected.

[antangelo]

Update test descriptions, use max instead of lcm for alignment

[antangelo]

llvm/lib/Target/X86/X86FrameLowering.cpp

1248

The 16 byte alignment is from SysV ABI. I believe the latest revision is here https://gitlab.com/x86-psABIs/i386-ABI/-/tree/hjl/x86/master

The end of the input argument area shall be aligned on a 16 (32 or 64, if __m256 or __m512 is passed on stack) byte boundary

[pengfei]

llvm/lib/Target/X86/X86FrameLowering.cpp

1248

My understanding is psABI just defines the default calling conversion. I didn't see description about interrupt in the ABI doc.
What's the GCC behavior? Is this to match with it?

[antangelo]

llvm/lib/Target/X86/X86FrameLowering.cpp

1248

GCC aligns the stack to 16 bytes if the interrupt handler makes calls to functions outside of the current compilation unit (I believe this is where the ABI alignment comes into play unless I'm misunderstanding). The goal is to align with GCC's behavior.

[pengfei]

LGTM except for one nit.

llvm/lib/Target/X86/X86FrameLowering.cpp

1248

Got it, thanks!

llvm/test/CodeGen/X86/x86-32-intrcc.ll

392–393

Add nounwind to remove .cfi*?

[antangelo]

Added nounwind to new x86-32 interrupt tests to remove .cfi*

[antangelo]

I don't have commit access to commit the changes myself, if someone can commit on my behalf. My name and email are:

Name: Antonio Abbatangelo
Email: [e-mail]

llvm/lib/Target/X86/X86FrameLowering.cpp

 // If we're forcing a stack realignment we can't rely on just the frame
 // info, we need to know the ABI stack alignment as well in case we
 // have a call out.  Otherwise just make sure we have some alignment - we'll
 // go with the minimum SlotSize.
 uint64_t X86FrameLowering::calculateMaxStackAlign(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   Align MaxAlign = MFI.getMaxAlign(); // Desired stack alignment.
   Align StackAlign = getStackAlign();
-  if (MF.getFunction().hasFnAttribute("stackrealign")) {
+  bool HasRealign = MF.getFunction().hasFnAttribute("stackrealign");
+  if (HasRealign) {
     if (MFI.hasCalls())
       MaxAlign = (StackAlign > MaxAlign) ? StackAlign : MaxAlign;
     else if (MaxAlign < SlotSize)
       MaxAlign = Align(SlotSize);
   }
+
+  if (!Is64Bit && MF.getFunction().getCallingConv() == CallingConv::X86_INTR) {
+    if (HasRealign)
+      MaxAlign = (MaxAlign > 16) ? MaxAlign : Align(16);
+    else
+      MaxAlign = Align(16);
+  }
   return MaxAlign.value();
 }
 
 void X86FrameLowering::BuildStackAlignAND(MachineBasicBlock &MBB,
                                           MachineBasicBlock::iterator MBBI,
                                           const DebugLoc &DL, unsigned Reg,
                                           uint64_t MaxAlign) const {
   uint64_t Val = -MaxAlign;

llvm/lib/Target/X86/X86RegisterInfo.h

bool isFixedRegister(const MachineFunction &MF,
                        MCRegister PhysReg) const override;
 
   void adjustStackMapLiveOutMask(uint32_t *Mask) const override;
 
   bool hasBasePointer(const MachineFunction &MF) const;
 
   bool canRealignStack(const MachineFunction &MF) const override;
 
+  bool shouldRealignStack(const MachineFunction &MF) const override;
+
   void eliminateFrameIndex(MachineBasicBlock::iterator II,
                            unsigned FIOperandNum, Register BaseReg,
                            int FIOffset) const;
 
   bool eliminateFrameIndex(MachineBasicBlock::iterator MI,
                            int SPAdj, unsigned FIOperandNum,
                            RegScavenger *RS = nullptr) const override;
 

llvm/lib/Target/X86/X86RegisterInfo.cpp

bool X86RegisterInfo::canRealignStack(const MachineFunction &MF) const {
 
   // If a base pointer is necessary.  Check that it isn't too late to reserve
   // it.
   if (CantUseSP(MFI))
     return MRI->canReserveReg(BasePtr);
   return true;
 }
 
+bool X86RegisterInfo::shouldRealignStack(const MachineFunction &MF) const {
+  if (TargetRegisterInfo::shouldRealignStack(MF))
+    return true;
+
+  return !Is64Bit && MF.getFunction().getCallingConv() == CallingConv::X86_INTR;
+}
+
 // tryOptimizeLEAtoMOV - helper function that tries to replace a LEA instruction
 // of the form 'lea (%esp), %ebx' --> 'mov %esp, %ebx'.
 // TODO: In this case we should be really trying first to entirely eliminate
 // this instruction which is a plain copy.
 static bool tryOptimizeLEAtoMOV(MachineBasicBlock::iterator II) {
   MachineInstr &MI = *II;
   unsigned Opc = II->getOpcode();
   // Check if this is a LEA of the form 'lea (%esp), %ebx'

llvm/test/CodeGen/X86/x86-32-intrcc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp --version 2
 ; RUN: llc -mtriple=i686-unknown-unknown < %s | FileCheck %s
 ; RUN: llc -mtriple=i686-unknown-unknown -O0 < %s | FileCheck %s -check-prefix=CHECK0
 
 %struct.interrupt_frame = type { i32, i32, i32, i32, i32 }
 
 @sink_address = global ptr null
 @sink_i32 = global i32 0
 
 
 ; Spills eax, putting original esp at +4.
-; No stack adjustment if declared with no error code
+; Stack is dyamically realigned to 16 bytes, and then reloaded to ebp - 4
+; With no error code, the stack is not incremented by 4 bytes before returning
 define x86_intrcc void @test_isr_no_ecode(ptr byval(%struct.interrupt_frame) %frame) nounwind {
 ; CHECK-LABEL: test_isr_no_ecode:
 ; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
-; CHECK-NEXT:    movl 12(%esp), %eax
+; CHECK-NEXT:    movl 12(%ebp), %eax
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    leal -4(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
+; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_isr_no_ecode:
 ; CHECK0:       # %bb.0:
+; CHECK0-NEXT:    pushl %ebp
+; CHECK0-NEXT:    movl %esp, %ebp
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
-; CHECK0-NEXT:    leal 4(%esp), %eax
+; CHECK0-NEXT:    leal 4(%ebp), %eax
 ; CHECK0-NEXT:    movl 8(%eax), %eax
 ; CHECK0-NEXT:    #APP
 ; CHECK0-NEXT:    #NO_APP
+; CHECK0-NEXT:    leal -4(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
+; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    iretl
+; CHECK-NEXT;    movl %esp, %ebp
   %pflags = getelementptr inbounds %struct.interrupt_frame, ptr %frame, i32 0, i32 2
   %flags = load i32, ptr %pflags, align 4
   call void asm sideeffect "", "r"(i32 %flags)
   ret void
 }
 
-; Spills eax and ecx, putting original esp at +8. Stack is adjusted up another 4 bytes
-; before return, popping the error code.
+; Spills eax and ecx, putting original esp at +8.
+; Stack is dynamically realigned to 16 bytes, and then reloaded to ebp - 8
+; Error code is popped from the stack with an increment of 4 before returning
 define x86_intrcc void @test_isr_ecode(ptr byval(%struct.interrupt_frame) %frame, i32 %ecode) nounwind {
 ; CHECK-LABEL: test_isr_ecode:
 ; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    pushl %ecx
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
-; CHECK-NEXT:    movl 8(%esp), %eax
-; CHECK-NEXT:    movl 20(%esp), %ecx
+; CHECK-NEXT:    movl 4(%ebp), %eax
+; CHECK-NEXT:    movl 16(%ebp), %ecx
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    leal -8(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    popl %ecx
+; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    addl $4, %esp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_isr_ecode:
 ; CHECK0:       # %bb.0:
+; CHECK0-NEXT:    pushl %ebp
+; CHECK0-NEXT:    movl %esp, %ebp
 ; CHECK0-NEXT:    pushl %ecx
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
-; CHECK0-NEXT:    movl 8(%esp), %ecx
-; CHECK0-NEXT:    leal 12(%esp), %eax
+; CHECK0-NEXT:    movl 4(%ebp), %ecx
+; CHECK0-NEXT:    leal 8(%ebp), %eax
 ; CHECK0-NEXT:    movl 8(%eax), %eax
 ; CHECK0-NEXT:    #APP
 ; CHECK0-NEXT:    #NO_APP
+; CHECK0-NEXT:    leal -8(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
 ; CHECK0-NEXT:    popl %ecx
+; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    addl $4, %esp
 ; CHECK0-NEXT:    iretl
   %pflags = getelementptr inbounds %struct.interrupt_frame, ptr %frame, i32 0, i32 2
   %flags = load i32, ptr %pflags, align 4
   call x86_fastcallcc void asm sideeffect "", "r,r"(i32 %flags, i32 %ecode)
   ret void
 }
 
 ; All clobbered registers must be saved
 define x86_intrcc void @test_isr_clobbers(ptr byval(%struct.interrupt_frame) %frame, i32 %ecode) nounwind {
 ; CHECK-LABEL: test_isr_clobbers:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    pushl %ecx
 ; CHECK-NEXT:    pushl %ebx
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    leal -12(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    popl %ebx
+; CHECK-NEXT:    popl %ecx
 ; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    addl $4, %esp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_isr_clobbers:
 ; CHECK0:       # %bb.0:
 ; CHECK0-NEXT:    pushl %ebp
+; CHECK0-NEXT:    movl %esp, %ebp
+; CHECK0-NEXT:    pushl %ecx
 ; CHECK0-NEXT:    pushl %ebx
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
 ; CHECK0-NEXT:    #APP
 ; CHECK0-NEXT:    #NO_APP
+; CHECK0-NEXT:    leal -12(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
 ; CHECK0-NEXT:    popl %ebx
+; CHECK0-NEXT:    popl %ecx
 ; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    addl $4, %esp
 ; CHECK0-NEXT:    iretl
-  call void asm sideeffect "", "~{eax},~{ebx},~{ebp}"()
+  call void asm sideeffect "", "~{eax},~{ebx},~{ecx},~{ebp}"()
   ret void
 }
 
 @f80 = common global x86_fp80 0xK00000000000000000000, align 4
 
 ; Test that the presence of x87 does not crash the FP stackifier
 define x86_intrcc void @test_isr_x87(ptr byval(%struct.interrupt_frame) %frame) nounwind {
 ; CHECK-LABEL: test_isr_x87:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
 ; CHECK-NEXT:    fldt f80
 ; CHECK-NEXT:    fld1
 ; CHECK-NEXT:    faddp %st, %st(1)
 ; CHECK-NEXT:    fstpt f80
+; CHECK-NEXT:    movl %ebp, %esp
+; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_isr_x87:
 ; CHECK0:       # %bb.0: # %entry
+; CHECK0-NEXT:    pushl %ebp
+; CHECK0-NEXT:    movl %esp, %ebp
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
 ; CHECK0-NEXT:    fldt f80
 ; CHECK0-NEXT:    fld1
 ; CHECK0-NEXT:    faddp %st, %st(1)
 ; CHECK0-NEXT:    fstpt f80
+; CHECK0-NEXT:    movl %ebp, %esp
+; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    iretl
 entry:
   %ld = load x86_fp80, ptr @f80, align 4
   %add = fadd x86_fp80 %ld, 0xK3FFF8000000000000000
   store x86_fp80 %add, ptr @f80, align 4
   ret void
 }
 
-; Use a frame pointer to check the offsets. No return address, arguments start
-; at EBP+4.
+; Use the interrupt_frame pointer to check the offsets.
+; No return address, arguments start at EBP+4.
 define dso_local x86_intrcc void @test_fp_1(ptr byval(%struct.interrupt_frame) %p) #0 {
 ; CHECK-LABEL: test_fp_1:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    pushl %ecx
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
 ; CHECK-NEXT:    leal 20(%ebp), %eax
 ; CHECK-NEXT:    leal 4(%ebp), %ecx
 ; CHECK-NEXT:    movl %ecx, sink_address
 ; CHECK-NEXT:    movl %eax, sink_address
+; CHECK-NEXT:    leal -8(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    popl %ecx
 ; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_fp_1:
 ; CHECK0:       # %bb.0: # %entry
 ; CHECK0-NEXT:    pushl %ebp
 ; CHECK0-NEXT:    movl %esp, %ebp
 ; CHECK0-NEXT:    pushl %ecx
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
 ; CHECK0-NEXT:    leal 4(%ebp), %ecx
 ; CHECK0-NEXT:    movl %ecx, %eax
 ; CHECK0-NEXT:    addl $16, %eax
 ; CHECK0-NEXT:    movl %ecx, sink_address
 ; CHECK0-NEXT:    movl %eax, sink_address
+; CHECK0-NEXT:    leal -8(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
 ; CHECK0-NEXT:    popl %ecx
 ; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    iretl
 entry:
   %arrayidx2 = getelementptr inbounds %struct.interrupt_frame, ptr %p, i32 0, i32 4
   store volatile ptr %p, ptr @sink_address
   store volatile ptr %arrayidx2, ptr @sink_address
   ret void
 }
 
 ; The error code is between EBP and the interrupt_frame.
 define dso_local x86_intrcc void @test_fp_2(ptr byval(%struct.interrupt_frame) %p, i32 %err) #0 {
 ; CHECK-LABEL: test_fp_2:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    pushl %edx
 ; CHECK-NEXT:    pushl %ecx
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
 ; CHECK-NEXT:    movl 4(%ebp), %eax
 ; CHECK-NEXT:    leal 24(%ebp), %ecx
 ; CHECK-NEXT:    leal 8(%ebp), %edx
 ; CHECK-NEXT:    movl %edx, sink_address
 ; CHECK-NEXT:    movl %ecx, sink_address
 ; CHECK-NEXT:    movl %eax, sink_i32
+; CHECK-NEXT:    leal -12(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    popl %ecx
 ; CHECK-NEXT:    popl %edx
 ; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    addl $4, %esp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_fp_2:
 ; CHECK0:       # %bb.0: # %entry
 ; CHECK0-NEXT:    pushl %ebp
 ; CHECK0-NEXT:    movl %esp, %ebp
 ; CHECK0-NEXT:    pushl %edx
 ; CHECK0-NEXT:    pushl %ecx
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
 ; CHECK0-NEXT:    movl 4(%ebp), %eax
 ; CHECK0-NEXT:    leal 8(%ebp), %edx
 ; CHECK0-NEXT:    movl %edx, %ecx
 ; CHECK0-NEXT:    addl $16, %ecx
 ; CHECK0-NEXT:    movl %edx, sink_address
 ; CHECK0-NEXT:    movl %ecx, sink_address
 ; CHECK0-NEXT:    movl %eax, sink_i32
+; CHECK0-NEXT:    leal -12(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
 ; CHECK0-NEXT:    popl %ecx
 ; CHECK0-NEXT:    popl %edx
 ; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    addl $4, %esp
 ; CHECK0-NEXT:    iretl
 entry:
   %arrayidx2 = getelementptr inbounds %struct.interrupt_frame, ptr %p, i32 0, i32 4
   store volatile ptr %p, ptr @sink_address
   store volatile ptr %arrayidx2, ptr @sink_address
   store volatile i32 %err, ptr @sink_i32
   ret void
 }
 
 ; Test argument copy elision when copied to a local alloca.
 define x86_intrcc void @test_copy_elide(ptr byval(%struct.interrupt_frame) %frame, i32 %err) #0 {
 ; CHECK-LABEL: test_copy_elide:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    pushl %ebp
 ; CHECK-NEXT:    movl %esp, %ebp
 ; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-16, %esp
 ; CHECK-NEXT:    cld
 ; CHECK-NEXT:    leal 4(%ebp), %eax
 ; CHECK-NEXT:    movl %eax, sink_address
+; CHECK-NEXT:    leal -4(%ebp), %esp
 ; CHECK-NEXT:    popl %eax
 ; CHECK-NEXT:    popl %ebp
 ; CHECK-NEXT:    addl $4, %esp
 ; CHECK-NEXT:    iretl
 ;
 ; CHECK0-LABEL: test_copy_elide:
 ; CHECK0:       # %bb.0: # %entry
 ; CHECK0-NEXT:    pushl %ebp
 ; CHECK0-NEXT:    movl %esp, %ebp
 ; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-16, %esp
 ; CHECK0-NEXT:    cld
 ; CHECK0-NEXT:    movl 4(%ebp), %eax
 ; CHECK0-NEXT:    leal 4(%ebp), %eax
 ; CHECK0-NEXT:    movl %eax, sink_address
+; CHECK0-NEXT:    leal -4(%ebp), %esp
 ; CHECK0-NEXT:    popl %eax
 ; CHECK0-NEXT:    popl %ebp
 ; CHECK0-NEXT:    addl $4, %esp
 ; CHECK0-NEXT:    iretl
 entry:
   %err.addr = alloca i32, align 4
   store i32 %err, ptr %err.addr, align 4
   store volatile ptr %err.addr, ptr @sink_address
   ret void
 }
 
+; Disabling dynamic realignment with attributes should work
+define x86_intrcc void @test_isr_no_realign(ptr byval(%struct.interrupt_frame) %frame) #1 {
+; CHECK-LABEL: test_isr_no_realign:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    cld
+; CHECK-NEXT:    movl 12(%esp), %eax
+; CHECK-NEXT:    #APP
+; CHECK-NEXT:    #NO_APP
+; CHECK-NEXT:    popl %eax
+; CHECK-NEXT:    iretl
+;
+; CHECK0-LABEL: test_isr_no_realign:
+; CHECK0:       # %bb.0:
+; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    cld
+; CHECK0-NEXT:    leal 4(%esp), %eax
+; CHECK0-NEXT:    movl 8(%eax), %eax
+; CHECK0-NEXT:    #APP
+; CHECK0-NEXT:    #NO_APP
+; CHECK0-NEXT:    popl %eax
+; CHECK0-NEXT:    iretl
+  %pflags = getelementptr inbounds %struct.interrupt_frame, ptr %frame, i32 0, i32 2
+  %flags = load i32, ptr %pflags, align 4
+  call void asm sideeffect "", "r"(i32 %flags)
+  ret void
+}
+
+; The stackrealign attribute should work, and the function's alignment
+; should be respected over the default 16-byte alignment required by the calling
+; convention.
+define x86_intrcc void @test_isr_realign(ptr byval(%struct.interrupt_frame) %frame, i32 %ecode) #2 {
+; CHECK-LABEL: test_isr_realign:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushl %ebp
+; CHECK-NEXT:    movl %esp, %ebp
+; CHECK-NEXT:    pushl %eax
+; CHECK-NEXT:    andl $-32, %esp
+; CHECK-NEXT:    subl $32, %esp
+; CHECK-NEXT:    cld
+; CHECK-NEXT:    movl 4(%ebp), %eax
+; CHECK-NEXT:    movl %eax, (%esp)
+; CHECK-NEXT:    leal -4(%ebp), %esp
+; CHECK-NEXT:    popl %eax
+; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    addl $4, %esp
+; CHECK-NEXT:    iretl
+;
+; CHECK0-LABEL: test_isr_realign:
+; CHECK0:       # %bb.0:
+; CHECK0-NEXT:    pushl %ebp
+; CHECK0-NEXT:    movl %esp, %ebp
+; CHECK0-NEXT:    pushl %eax
+; CHECK0-NEXT:    andl $-32, %esp
+; CHECK0-NEXT:    subl $32, %esp
+; CHECK0-NEXT:    cld
+; CHECK0-NEXT:    movl 4(%ebp), %eax
+; CHECK0-NEXT:    movl %eax, (%esp)
+; CHECK0-NEXT:    leal -4(%ebp), %esp
+; CHECK0-NEXT:    popl %eax
+; CHECK0-NEXT:    popl %ebp
+; CHECK0-NEXT:    addl $4, %esp
+; CHECK0-NEXT:    iretl
+  %ecode.stack = alloca i32, align 32
+  store i32 %ecode, ptr %ecode.stack
+  ret void
+}
+
+
 attributes #0 = { nounwind "frame-pointer"="all" }
+attributes #1 = { nounwind "no-realign-stack" }
+attributes #2 = { nounwind "stackrealign" }

llvm/test/CodeGen/X86/x86-interrupt_cc.ll

 ; CHECK64-SKX-NEXT:    popq %r9 ## encoding: [0x41,0x59]
 ; CHECK64-SKX-NEXT:    popq %r10 ## encoding: [0x41,0x5a]
 ; CHECK64-SKX-NEXT:    popq %r11 ## encoding: [0x41,0x5b]
 ; CHECK64-SKX-NEXT:    popq %rax ## encoding: [0x58]
 ; CHECK64-SKX-NEXT:    iretq ## encoding: [0x48,0xcf]
 ;
 ; CHECK32-KNL-LABEL: foo:
 ; CHECK32-KNL:       ## %bb.0:
-; CHECK32-KNL-NEXT:    pushl %edx ## encoding: [0x52]
+; CHECK32-KNL-NEXT:    pushl %ebp ## encoding: [0x55]
 ; CHECK32-KNL-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-KNL-NEXT:    .cfi_offset %ebp, -8
+; CHECK32-KNL-NEXT:    movl %esp, %ebp ## encoding: [0x89,0xe5]
+; CHECK32-KNL-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK32-KNL-NEXT:    pushl %edx ## encoding: [0x52]
 ; CHECK32-KNL-NEXT:    pushl %ecx ## encoding: [0x51]
-; CHECK32-KNL-NEXT:    .cfi_def_cfa_offset 12
 ; CHECK32-KNL-NEXT:    pushl %eax ## encoding: [0x50]
-; CHECK32-KNL-NEXT:    .cfi_def_cfa_offset 16
+; CHECK32-KNL-NEXT:    andl $-16, %esp ## encoding: [0x83,0xe4,0xf0]
 ; CHECK32-KNL-NEXT:    subl $560, %esp ## encoding: [0x81,0xec,0x30,0x02,0x00,0x00]
 ; CHECK32-KNL-NEXT:    ## imm = 0x230
 ; CHECK32-KNL-NEXT:    kmovw %k7, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0xbc,0x24,0x2e,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x7d,0xf2]
 ; CHECK32-KNL-NEXT:    kmovw %k6, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0xb4,0x24,0x2c,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x75,0xf0]
 ; CHECK32-KNL-NEXT:    kmovw %k5, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0xac,0x24,0x2a,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x6d,0xee]
 ; CHECK32-KNL-NEXT:    kmovw %k4, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0xa4,0x24,0x28,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x65,0xec]
 ; CHECK32-KNL-NEXT:    kmovw %k3, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x9c,0x24,0x26,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x5d,0xea]
 ; CHECK32-KNL-NEXT:    kmovw %k2, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x94,0x24,0x24,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x55,0xe8]
 ; CHECK32-KNL-NEXT:    kmovw %k1, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x8c,0x24,0x22,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x4d,0xe6]
 ; CHECK32-KNL-NEXT:    kmovw %k0, {{[-0-9]+}}(%e{{[sb]}}p) ## 2-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x84,0x24,0x20,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x91,0x45,0xe4]
 ; CHECK32-KNL-NEXT:    vmovups %zmm7, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x7c,0x24,0x07]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xbd,0x88,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm6, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x74,0x24,0x06]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xb5,0x48,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm5, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x6c,0x24,0x05]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xad,0x08,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm4, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x64,0x24,0x04]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xa5,0xc8,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm3, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x5c,0x24,0x03]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x9d,0x88,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm2, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x54,0x24,0x02]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x95,0x48,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups %zmm1, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x4c,0x24,0x01]
-; CHECK32-KNL-NEXT:    vmovups %zmm0, (%esp) ## 64-byte Spill
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x04,0x24]
-; CHECK32-KNL-NEXT:    .cfi_def_cfa_offset 576
-; CHECK32-KNL-NEXT:    .cfi_offset %eax, -16
-; CHECK32-KNL-NEXT:    .cfi_offset %ecx, -12
-; CHECK32-KNL-NEXT:    .cfi_offset %edx, -8
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x8d,0x08,0xfe,0xff,0xff]
+; CHECK32-KNL-NEXT:    vmovups %zmm0, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x85,0xc8,0xfd,0xff,0xff]
+; CHECK32-KNL-NEXT:    .cfi_offset %eax, -20
+; CHECK32-KNL-NEXT:    .cfi_offset %ecx, -16
+; CHECK32-KNL-NEXT:    .cfi_offset %edx, -12
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm0, -576
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm1, -512
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm2, -448
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm3, -384
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm4, -320
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm5, -256
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm6, -192
 ; CHECK32-KNL-NEXT:    .cfi_offset %xmm7, -128
-; CHECK32-KNL-NEXT:    .cfi_offset %k0, -32
-; CHECK32-KNL-NEXT:    .cfi_offset %k1, -30
-; CHECK32-KNL-NEXT:    .cfi_offset %k2, -28
-; CHECK32-KNL-NEXT:    .cfi_offset %k3, -26
-; CHECK32-KNL-NEXT:    .cfi_offset %k4, -24
-; CHECK32-KNL-NEXT:    .cfi_offset %k5, -22
-; CHECK32-KNL-NEXT:    .cfi_offset %k6, -20
-; CHECK32-KNL-NEXT:    .cfi_offset %k7, -18
+; CHECK32-KNL-NEXT:    .cfi_offset %k0, -36
+; CHECK32-KNL-NEXT:    .cfi_offset %k1, -34
+; CHECK32-KNL-NEXT:    .cfi_offset %k2, -32
+; CHECK32-KNL-NEXT:    .cfi_offset %k3, -30
+; CHECK32-KNL-NEXT:    .cfi_offset %k4, -28
+; CHECK32-KNL-NEXT:    .cfi_offset %k5, -26
+; CHECK32-KNL-NEXT:    .cfi_offset %k6, -24
+; CHECK32-KNL-NEXT:    .cfi_offset %k7, -22
 ; CHECK32-KNL-NEXT:    cld ## encoding: [0xfc]
 ; CHECK32-KNL-NEXT:    calll _bar ## encoding: [0xe8,A,A,A,A]
 ; CHECK32-KNL-NEXT:    ## fixup A - offset: 1, value: _bar-4, kind: FK_PCRel_4
-; CHECK32-KNL-NEXT:    vmovups (%esp), %zmm0 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x04,0x24]
+; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm0 ## 64-byte Reload
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x85,0xc8,0xfd,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm1 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x4c,0x24,0x01]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x8d,0x08,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm2 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x54,0x24,0x02]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x95,0x48,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm3 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x5c,0x24,0x03]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x9d,0x88,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm4 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x64,0x24,0x04]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xa5,0xc8,0xfe,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm5 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x6c,0x24,0x05]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xad,0x08,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm6 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x74,0x24,0x06]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xb5,0x48,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm7 ## 64-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x7c,0x24,0x07]
+; CHECK32-KNL-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xbd,0x88,0xff,0xff,0xff]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k0 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x84,0x24,0x20,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x45,0xe4]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k1 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x8c,0x24,0x22,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x4d,0xe6]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k2 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x94,0x24,0x24,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x55,0xe8]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k3 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x9c,0x24,0x26,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x5d,0xea]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k4 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0xa4,0x24,0x28,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x65,0xec]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k5 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0xac,0x24,0x2a,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x6d,0xee]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k6 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0xb4,0x24,0x2c,0x02,0x00,0x00]
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x75,0xf0]
 ; CHECK32-KNL-NEXT:    kmovw {{[-0-9]+}}(%e{{[sb]}}p), %k7 ## 2-byte Reload
-; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0xbc,0x24,0x2e,0x02,0x00,0x00]
-; CHECK32-KNL-NEXT:    addl $560, %esp ## encoding: [0x81,0xc4,0x30,0x02,0x00,0x00]
-; CHECK32-KNL-NEXT:    ## imm = 0x230
+; CHECK32-KNL-NEXT:    ## encoding: [0xc5,0xf8,0x90,0x7d,0xf2]
+; CHECK32-KNL-NEXT:    leal -12(%ebp), %esp ## encoding: [0x8d,0x65,0xf4]
 ; CHECK32-KNL-NEXT:    popl %eax ## encoding: [0x58]
 ; CHECK32-KNL-NEXT:    popl %ecx ## encoding: [0x59]
 ; CHECK32-KNL-NEXT:    popl %edx ## encoding: [0x5a]
+; CHECK32-KNL-NEXT:    popl %ebp ## encoding: [0x5d]
 ; CHECK32-KNL-NEXT:    iretl ## encoding: [0xcf]
 ;
 ; CHECK32-SKX-LABEL: foo:
 ; CHECK32-SKX:       ## %bb.0:
-; CHECK32-SKX-NEXT:    pushl %edx ## encoding: [0x52]
+; CHECK32-SKX-NEXT:    pushl %ebp ## encoding: [0x55]
 ; CHECK32-SKX-NEXT:    .cfi_def_cfa_offset 8
+; CHECK32-SKX-NEXT:    .cfi_offset %ebp, -8
+; CHECK32-SKX-NEXT:    movl %esp, %ebp ## encoding: [0x89,0xe5]
+; CHECK32-SKX-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK32-SKX-NEXT:    pushl %edx ## encoding: [0x52]
 ; CHECK32-SKX-NEXT:    pushl %ecx ## encoding: [0x51]
-; CHECK32-SKX-NEXT:    .cfi_def_cfa_offset 12
 ; CHECK32-SKX-NEXT:    pushl %eax ## encoding: [0x50]
-; CHECK32-SKX-NEXT:    .cfi_def_cfa_offset 16
+; CHECK32-SKX-NEXT:    andl $-16, %esp ## encoding: [0x83,0xe4,0xf0]
 ; CHECK32-SKX-NEXT:    subl $624, %esp ## encoding: [0x81,0xec,0x70,0x02,0x00,0x00]
 ; CHECK32-SKX-NEXT:    ## imm = 0x270
 ; CHECK32-SKX-NEXT:    kmovq %k7, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0xbc,0x24,0x68,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x7d,0xe8]
 ; CHECK32-SKX-NEXT:    kmovq %k6, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0xb4,0x24,0x60,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x75,0xe0]
 ; CHECK32-SKX-NEXT:    kmovq %k5, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0xac,0x24,0x58,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x6d,0xd8]
 ; CHECK32-SKX-NEXT:    kmovq %k4, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0xa4,0x24,0x50,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x65,0xd0]
 ; CHECK32-SKX-NEXT:    kmovq %k3, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x9c,0x24,0x48,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x5d,0xc8]
 ; CHECK32-SKX-NEXT:    kmovq %k2, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x94,0x24,0x40,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x55,0xc0]
 ; CHECK32-SKX-NEXT:    kmovq %k1, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x8c,0x24,0x38,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x4d,0xb8]
 ; CHECK32-SKX-NEXT:    kmovq %k0, {{[-0-9]+}}(%e{{[sb]}}p) ## 8-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x84,0x24,0x30,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x91,0x45,0xb0]
 ; CHECK32-SKX-NEXT:    vmovups %zmm7, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x7c,0x24,0x07]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xbd,0x48,0xff,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm6, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x74,0x24,0x06]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xb5,0x08,0xff,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm5, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x6c,0x24,0x05]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xad,0xc8,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm4, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x64,0x24,0x04]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0xa5,0x88,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm3, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x5c,0x24,0x03]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x9d,0x48,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm2, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x54,0x24,0x02]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x95,0x08,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups %zmm1, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x4c,0x24,0x01]
-; CHECK32-SKX-NEXT:    vmovups %zmm0, (%esp) ## 64-byte Spill
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x04,0x24]
-; CHECK32-SKX-NEXT:    .cfi_def_cfa_offset 640
-; CHECK32-SKX-NEXT:    .cfi_offset %eax, -16
-; CHECK32-SKX-NEXT:    .cfi_offset %ecx, -12
-; CHECK32-SKX-NEXT:    .cfi_offset %edx, -8
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x8d,0xc8,0xfd,0xff,0xff]
+; CHECK32-SKX-NEXT:    vmovups %zmm0, {{[-0-9]+}}(%e{{[sb]}}p) ## 64-byte Spill
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x11,0x85,0x88,0xfd,0xff,0xff]
+; CHECK32-SKX-NEXT:    .cfi_offset %eax, -20
+; CHECK32-SKX-NEXT:    .cfi_offset %ecx, -16
+; CHECK32-SKX-NEXT:    .cfi_offset %edx, -12
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm0, -640
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm1, -576
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm2, -512
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm3, -448
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm4, -384
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm5, -320
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm6, -256
 ; CHECK32-SKX-NEXT:    .cfi_offset %xmm7, -192
-; CHECK32-SKX-NEXT:    .cfi_offset %k0, -80
-; CHECK32-SKX-NEXT:    .cfi_offset %k1, -72
-; CHECK32-SKX-NEXT:    .cfi_offset %k2, -64
-; CHECK32-SKX-NEXT:    .cfi_offset %k3, -56
-; CHECK32-SKX-NEXT:    .cfi_offset %k4, -48
-; CHECK32-SKX-NEXT:    .cfi_offset %k5, -40
-; CHECK32-SKX-NEXT:    .cfi_offset %k6, -32
-; CHECK32-SKX-NEXT:    .cfi_offset %k7, -24
+; CHECK32-SKX-NEXT:    .cfi_offset %k0, -88
+; CHECK32-SKX-NEXT:    .cfi_offset %k1, -80
+; CHECK32-SKX-NEXT:    .cfi_offset %k2, -72
+; CHECK32-SKX-NEXT:    .cfi_offset %k3, -64
+; CHECK32-SKX-NEXT:    .cfi_offset %k4, -56
+; CHECK32-SKX-NEXT:    .cfi_offset %k5, -48
+; CHECK32-SKX-NEXT:    .cfi_offset %k6, -40
+; CHECK32-SKX-NEXT:    .cfi_offset %k7, -32
 ; CHECK32-SKX-NEXT:    cld ## encoding: [0xfc]
 ; CHECK32-SKX-NEXT:    vzeroupper ## encoding: [0xc5,0xf8,0x77]
 ; CHECK32-SKX-NEXT:    calll _bar ## encoding: [0xe8,A,A,A,A]
 ; CHECK32-SKX-NEXT:    ## fixup A - offset: 1, value: _bar-4, kind: FK_PCRel_4
-; CHECK32-SKX-NEXT:    vmovups (%esp), %zmm0 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x04,0x24]
+; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm0 ## 64-byte Reload
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x85,0x88,0xfd,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm1 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x4c,0x24,0x01]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x8d,0xc8,0xfd,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm2 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x54,0x24,0x02]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x95,0x08,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm3 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x5c,0x24,0x03]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x9d,0x48,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm4 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x64,0x24,0x04]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xa5,0x88,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm5 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x6c,0x24,0x05]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xad,0xc8,0xfe,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm6 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x74,0x24,0x06]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xb5,0x08,0xff,0xff,0xff]
 ; CHECK32-SKX-NEXT:    vmovups {{[-0-9]+}}(%e{{[sb]}}p), %zmm7 ## 64-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0x7c,0x24,0x07]
+; CHECK32-SKX-NEXT:    ## encoding: [0x62,0xf1,0x7c,0x48,0x10,0xbd,0x48,0xff,0xff,0xff]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k0 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x84,0x24,0x30,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x45,0xb0]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k1 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x8c,0x24,0x38,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x4d,0xb8]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k2 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x94,0x24,0x40,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x55,0xc0]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k3 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x9c,0x24,0x48,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x5d,0xc8]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k4 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0xa4,0x24,0x50,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x65,0xd0]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k5 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0xac,0x24,0x58,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x6d,0xd8]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k6 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0xb4,0x24,0x60,0x02,0x00,0x00]
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x75,0xe0]
 ; CHECK32-SKX-NEXT:    kmovq {{[-0-9]+}}(%e{{[sb]}}p), %k7 ## 8-byte Reload
-; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0xbc,0x24,0x68,0x02,0x00,0x00]
-; CHECK32-SKX-NEXT:    addl $624, %esp ## encoding: [0x81,0xc4,0x70,0x02,0x00,0x00]
-; CHECK32-SKX-NEXT:    ## imm = 0x270
+; CHECK32-SKX-NEXT:    ## encoding: [0xc4,0xe1,0xf8,0x90,0x7d,0xe8]
+; CHECK32-SKX-NEXT:    leal -12(%ebp), %esp ## encoding: [0x8d,0x65,0xf4]
 ; CHECK32-SKX-NEXT:    popl %eax ## encoding: [0x58]
 ; CHECK32-SKX-NEXT:    popl %ecx ## encoding: [0x59]
 ; CHECK32-SKX-NEXT:    popl %edx ## encoding: [0x5a]
+; CHECK32-SKX-NEXT:    popl %ebp ## encoding: [0x5d]
 ; CHECK32-SKX-NEXT:    iretl ## encoding: [0xcf]
   call void @bar()
   ret void
 }
 
 declare void @bar()