[X86] Improve lowering of the unrolled inline-asm probing

[nagisa]

Summary

This new implementation emits instructions such as these:

movb $0, -4096(%rsp)

which is both faster and smaller than pairs of

sub $4096, %rsp
movq $0, (%rsp)

This implementation also trivially preserves the preciseness of the
uwtables during the preamble by not modifying the stack pointer in the
first place.

Testing the generated code for stacks of 0x4000 bytes (4 probes) llvm-mca reports
(over 100 iterations):

test case	mcpu	cycles	IPC	RThroughput
old	znver2	603	1.66	2.5
new	znver2	204	2.94	1.5
old	skylake	603	1.66	4.0
new	skylake	403	1.49	4.0
old	bdver1	603	1.66	6.0
new	bdver1	403	1.49	4.0
old	haswell	603	1.66	4.0
new	haswell	403	1.49	4.0

So overall in terms of throughput its either the same or
an improvement.

Depends on D98909

[nagisa]

update tests to not scrub away stack pointers

[nagisa]

rebase on top of tests in D98909

[nagisa]

restore the code that handled overaligned allocs

[serge-sans-paille]

Thanks for proposing this optimization. There's a reason (which is debatable) why we explicitly sub before moving :

Consider the following:

0000000000400400 <main>:
  400400:       c7 84 24 fc bf ff ff    movl   $0x1,-0x4004(%rsp)
  400407:       01 00 00 00 
  40040b:       c3                      retq

the mov accesses « unallocated » stack, which may be spotted as illegal by tools verifying memory accesses. When running valgrind on the above, I indeed get

==1335611== Invalid write of size 4
==1335611==    at 0x400400: main (in /tmp/a.out)
==1335611==  Address 0x1ffeffb534 is not stack'd, malloc'd or (recently) free'd

[nagisa]

Ah, I see!

Well, this isn't strictly just an optimization. This sequence came to me as I was thinking how to make CFI metadata correct with least amount of work on my end. It seems that in presence of tools such as valgrind we don't really have any other choice than to carefully emit a -cfi_def_cfa_offset for each of the subs. Which is fine I guess, but leaves me not super happy about the code we emit.

Besides the accurate uwtables, there's another conflicting use-case here – signal handlers. If we sub and then probe, we may receive a signal (and allocate a stack slot for it) in a potentially invalid stack area. Whereas if we probe and then sub, any signal handlers that occur during probing would still execute on what is significantly more likely to be a valid stack.

So, I guess, my question here is this: Is the codegen responsible for generating code that's palatable to analysis tools, or are the analysis tools responsible for comprehending the code that they inspect?

[andreadb]

Be careful about referencing stack locations below the red-zone.

The x86-64 ABI mandates a 128 bytes red zone. Your new instruction would effectively write a zero at a location below the red-zone.
That region of memory is not reserved, and should be considered volatile. For example: signals and interrupt handlers are allowed to modify it.

That's the reason why, for leaf functions, compilers always emit a SUB of RSP in the case of too big negative offsets (w.r.t. RSP).

[nagisa]

In D98906#2641418, @andreadb wrote:

Be careful about referencing stack locations below the red-zone.

The x86-64 ABI mandates a 128 bytes red zone. Your new instruction would effectively write a zero at a location below the red-zone.
That region of memory is not reserved, and should be considered volatile. For example: signals and interrupt handlers are allowed to modify it.

Isn't the volatility unimportant for the purposes of stack probing? The write exists only to poke the page (lightly) to ensure there's no page that is not rw, probing does not particularly care if the data at the relevant addresses are overwritten, or even that the byte is written to the address in the first place (as long as it triggers page permission checks). The only concern could be that some signal handler or an interrupt put some data there, but when these are running the probing itself is suspended, isn't it?

[andreadb]

In D98906#2641469, @nagisa wrote:

In D98906#2641418, @andreadb wrote:

Be careful about referencing stack locations below the red-zone.

The x86-64 ABI mandates a 128 bytes red zone. Your new instruction would effectively write a zero at a location below the red-zone.
That region of memory is not reserved, and should be considered volatile. For example: signals and interrupt handlers are allowed to modify it.

Isn't the volatility unimportant for the purposes of stack probing? The write exists only to poke the page (lightly) to ensure there's no page that is not rw, probing does not particularly care if the data at the relevant addresses are overwritten, or even that the byte is written to the address in the first place (as long as it triggers page permission checks). The only concern could be that some signal handler or an interrupt put some data there, but when these are running the probing itself is suspended, isn't it?

I agree. If the goal is just doing stack probing then it should be fine.

Mine was more of a generic "be careful about the red-zone". Just to make sure that it was considered in this design. I didn't know about your particular use case scenario though.

[serge-sans-paille]

Besides the accurate uwtables, there's another conflicting use-case here – signal handlers. If we sub and then probe, we may receive a signal (and allocate a stack slot for it) in a potentially invalid stack area. Whereas if we probe and then sub, any signal handlers that occur during probing would still execute on what is significantly more likely to be a valid stack.

If my understanding is correct, (and in the case where an alternate stack is not used) the signal handler begins with a push rbp so it should touch the stack and triggers the PAGE_GUARD... mmmh unless we're right at the end of the page guard after the sub, so maybe we should subq $4088 ?

llvm/lib/Target/X86/X86FrameLowering.cpp

 void X86FrameLowering::emitStackProbeInlineGenericBlock(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MBBI, const DebugLoc &DL, uint64_t Offset,
     uint64_t AlignOffset) const {
 
   const X86Subtarget &STI = MF.getSubtarget<X86Subtarget>();
   const X86TargetLowering &TLI = *STI.getTargetLowering();
   const unsigned Opc = getSUBriOpcode(Uses64BitFramePtr, Offset);
-  const unsigned MovMIOpc = Is64Bit ? X86::MOV64mi32 : X86::MOV32mi;
   const uint64_t StackProbeSize = TLI.getStackProbeSize(MF);
 
   uint64_t CurrentOffset = 0;
 
   assert(AlignOffset < StackProbeSize);
 
-  // If the offset is so small it fits within a page, there's nothing to do.
-  if (StackProbeSize < Offset + AlignOffset) {
-
-    MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
-                           .addReg(StackPtr)
-                           .addImm(StackProbeSize - AlignOffset)
-                           .setMIFlag(MachineInstr::FrameSetup);
-    MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
-
-    addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(MovMIOpc))
-                     .setMIFlag(MachineInstr::FrameSetup),
-                 StackPtr, false, 0)
-        .addImm(0)
-        .setMIFlag(MachineInstr::FrameSetup);
-    NumFrameExtraProbe++;
-    CurrentOffset = StackProbeSize - AlignOffset;
-  }
-
-  // For the next N - 1 pages, just probe. I tried to take advantage of
-  // natural probes but it implies much more logic and there was very few
-  // interesting natural probes to interleave.
+  // For the first N - 1 pages, probe.
+  //
+  // We emit the most basic `movb $0, -offset(%rsp)` instruction which is good
+  // for offsets of up-to 2GB. This is also most throughput and space efficient
+  // encoding that I (nagisa) could come up.
+  //
+  // It also naturally doesn't need any special handling for precise uwtables.
   while (CurrentOffset + StackProbeSize < Offset) {
-    MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
-                           .addReg(StackPtr)
-                           .addImm(StackProbeSize)
-                           .setMIFlag(MachineInstr::FrameSetup);
-    MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
-
-
-    addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(MovMIOpc))
+    NumFrameExtraProbe++;
+    CurrentOffset += StackProbeSize;
+    addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV8mi))
                      .setMIFlag(MachineInstr::FrameSetup),
-                 StackPtr, false, 0)
+                 StackPtr, false, -CurrentOffset)
         .addImm(0)
         .setMIFlag(MachineInstr::FrameSetup);
-    NumFrameExtraProbe++;
-    CurrentOffset += StackProbeSize;
   }
 
   // No need to probe the tail, it is smaller than a Page.
-  uint64_t ChunkSize = Offset - CurrentOffset;
   MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
                          .addReg(StackPtr)
-                         .addImm(ChunkSize)
+                         .addImm(Offset)
                          .setMIFlag(MachineInstr::FrameSetup);
   MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
 }
 
 void X86FrameLowering::emitStackProbeInlineGenericLoop(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MBBI, const DebugLoc &DL, uint64_t Offset,
     uint64_t AlignOffset) const {

llvm/test/CodeGen/X86/stack-clash-dynamic-alloca.ll

-; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s 
-; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s 
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
+; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s
+; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s
 
 define i32 @foo(i32 %n) local_unnamed_addr #0 {
+; CHECK-X86-64-LABEL: foo:
+; CHECK-X86-64:       # %bb.0:
+; CHECK-X86-64-NEXT:    pushq %rbp
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-X86-64-NEXT:    .cfi_offset %rbp, -16
+; CHECK-X86-64-NEXT:    movq %rsp, %rbp
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-X86-64-NEXT:    movq %rsp, %rax
+; CHECK-X86-64-NEXT:    movl %edi, %ecx
+; CHECK-X86-64-NEXT:    leaq 15(,%rcx,4), %rcx
+; CHECK-X86-64-NEXT:    andq $-16, %rcx
+; CHECK-X86-64-NEXT:    subq %rcx, %rax
+; CHECK-X86-64-NEXT:    cmpq %rsp, %rax
+; CHECK-X86-64-NEXT:    jge .LBB0_3
+; CHECK-X86-64-NEXT:  .LBB0_2: # =>This Inner Loop Header: Depth=1
+; CHECK-X86-64-NEXT:    xorq $0, (%rsp)
+; CHECK-X86-64-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-X86-64-NEXT:    cmpq %rsp, %rax
+; CHECK-X86-64-NEXT:    jl .LBB0_2
+; CHECK-X86-64-NEXT:  .LBB0_3:
+; CHECK-X86-64-NEXT:    movq %rax, %rsp
+; CHECK-X86-64-NEXT:    movl $1, 4792(%rax)
+; CHECK-X86-64-NEXT:    movl (%rax), %eax
+; CHECK-X86-64-NEXT:    movq %rbp, %rsp
+; CHECK-X86-64-NEXT:    popq %rbp
+; CHECK-X86-64-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-X86-64-NEXT:    retq
+;
+; CHECK-X86-32-LABEL: foo:
+; CHECK-X86-32:       # %bb.0:
+; CHECK-X86-32-NEXT:    pushl %ebp
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-X86-32-NEXT:    .cfi_offset %ebp, -8
+; CHECK-X86-32-NEXT:    movl %esp, %ebp
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK-X86-32-NEXT:    subl $8, %esp
+; CHECK-X86-32-NEXT:    movl 8(%ebp), %ecx
+; CHECK-X86-32-NEXT:    movl %esp, %eax
+; CHECK-X86-32-NEXT:    leal 15(,%ecx,4), %ecx
+; CHECK-X86-32-NEXT:    andl $-16, %ecx
+; CHECK-X86-32-NEXT:    subl %ecx, %eax
+; CHECK-X86-32-NEXT:    cmpl %esp, %eax
+; CHECK-X86-32-NEXT:    jge .LBB0_3
+; CHECK-X86-32-NEXT:  .LBB0_2: # =>This Inner Loop Header: Depth=1
+; CHECK-X86-32-NEXT:    xorl $0, (%esp)
+; CHECK-X86-32-NEXT:    subl $4096, %esp # imm = 0x1000
+; CHECK-X86-32-NEXT:    cmpl %esp, %eax
+; CHECK-X86-32-NEXT:    jl .LBB0_2
+; CHECK-X86-32-NEXT:  .LBB0_3:
+; CHECK-X86-32-NEXT:    movl %eax, %esp
+; CHECK-X86-32-NEXT:    movl $1, 4792(%eax)
+; CHECK-X86-32-NEXT:    movl (%eax), %eax
+; CHECK-X86-32-NEXT:    movl %ebp, %esp
+; CHECK-X86-32-NEXT:    popl %ebp
+; CHECK-X86-32-NEXT:    .cfi_def_cfa %esp, 4
+; CHECK-X86-32-NEXT:    retl
   %a = alloca i32, i32 %n, align 16
   %b = getelementptr inbounds i32, i32* %a, i64 1198
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-large-large-align.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo_noprotect() local_unnamed_addr {
 ; CHECK-LABEL: foo_noprotect:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT:   .cfi_def_cfa_offset 16
-; CHECK-NEXT:   .cfi_offset %rbp, -16
-; CHECK-NEXT:   movq  %rsp, %rbp
-; CHECK-NEXT:   .cfi_def_cfa_register %rbp
-; CHECK-NEXT:   andq  $-4096, %rsp                    # imm = 0xF000
-; CHECK-NEXT:   subq  $73728, %rsp                    # imm = 0x12000
-; CHECK-NEXT:   movl  $1, 392(%rsp)
-; CHECK-NEXT:   movl  $1, 28792(%rsp)
-; CHECK-NEXT:   movl  (%rsp), %eax
-; CHECK-NEXT:   movq  %rbp, %rsp
-; CHECK-NEXT:   popq  %rbp
-; CHECK-NEXT:   .cfi_def_cfa %rsp, 8
-; CHECK-NEXT:   retq
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    andq $-4096, %rsp # imm = 0xF000
+; CHECK-NEXT:    subq $73728, %rsp # imm = 0x12000
+; CHECK-NEXT:    movl $1, 392(%rsp)
+; CHECK-NEXT:    movl $1, 28792(%rsp)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
 
   %a = alloca i32, i64 18000, align 4096
   %b0 = getelementptr inbounds i32, i32* %a, i64 98
   %b1 = getelementptr inbounds i32, i32* %a, i64 7198
   store volatile i32 1, i32* %b0
   store volatile i32 1, i32* %b1
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 define i32 @foo_protect() local_unnamed_addr #0 {
 ; CHECK-LABEL: foo_protect:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT: movq  %rsp, %r11
-; CHECK-NEXT: andq  $-4096, %r11                    # imm = 0xF000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: je  .LBB1_4
-; CHECK-NEXT:# %bb.1:
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: jb  .LBB1_3
-; CHECK-NEXT:.LBB1_2:                                # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: jb  .LBB1_2
-; CHECK-NEXT:.LBB1_3:
-; CHECK-NEXT: movq  %r11, %rsp
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT:.LBB1_4:
-; CHECK-NEXT: movq  %rsp, %r11
-; CHECK-NEXT: subq  $73728, %r11                    # imm = 0x12000
-; CHECK-NEXT:.LBB1_5:                                # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT: cmpq  %r11, %rsp
-; CHECK-NEXT: jne .LBB1_5
-; CHECK-NEXT:# %bb.6:
-; CHECK-NEXT: movl  $1, 392(%rsp)
-; CHECK-NEXT: movl  $1, 28792(%rsp)
-; CHECK-NEXT: movl  (%rsp), %eax
-; CHECK-NEXT: movq  %rbp, %rsp
-; CHECK-NEXT: popq  %rbp
-; CHECK-NEXT: .cfi_def_cfa %rsp, 8
-; CHECK-NEXT: retq
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    movq %rsp, %r11
+; CHECK-NEXT:    andq $-4096, %r11 # imm = 0xF000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    je .LBB1_4
+; CHECK-NEXT:  # %bb.1:
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    jb .LBB1_3
+; CHECK-NEXT:  .LBB1_2: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    jb .LBB1_2
+; CHECK-NEXT:  .LBB1_3:
+; CHECK-NEXT:    movq %r11, %rsp
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:  .LBB1_4:
+; CHECK-NEXT:    movq %rsp, %r11
+; CHECK-NEXT:    subq $73728, %r11 # imm = 0x12000
+; CHECK-NEXT:  .LBB1_5: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:    cmpq %r11, %rsp
+; CHECK-NEXT:    jne .LBB1_5
+; CHECK-NEXT:  # %bb.6:
+; CHECK-NEXT:    movl $1, 392(%rsp)
+; CHECK-NEXT:    movl $1, 28792(%rsp)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
 
 
 
   %a = alloca i32, i64 18000, align 4096
   %b0 = getelementptr inbounds i32, i32* %a, i64 98
   %b1 = getelementptr inbounds i32, i32* %a, i64 7198
   store volatile i32 1, i32* %b0
   store volatile i32 1, i32* %b1
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-large.ll

-; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s 
-; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s 
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
+; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s
+; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s
 
 define i32 @foo() local_unnamed_addr #0 {
-
+; CHECK-X86-64-LABEL: foo:
+; CHECK-X86-64:       # %bb.0:
+; CHECK-X86-64-NEXT:    movq %rsp, %r11
+; CHECK-X86-64-NEXT:    subq $69632, %r11 # imm = 0x11000
+; CHECK-X86-64-NEXT:  .LBB0_1: # =>This Inner Loop Header: Depth=1
+; CHECK-X86-64-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-X86-64-NEXT:    movq $0, (%rsp)
+; CHECK-X86-64-NEXT:    cmpq %r11, %rsp
+; CHECK-X86-64-NEXT:    jne .LBB0_1
+; CHECK-X86-64-NEXT:  # %bb.2:
+; CHECK-X86-64-NEXT:    subq $2248, %rsp # imm = 0x8C8
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_offset 71888
+; CHECK-X86-64-NEXT:    movl $1, 264(%rsp)
+; CHECK-X86-64-NEXT:    movl $1, 28664(%rsp)
+; CHECK-X86-64-NEXT:    movl -128(%rsp), %eax
+; CHECK-X86-64-NEXT:    addq $71880, %rsp # imm = 0x118C8
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-X86-64-NEXT:    retq
+;
+; CHECK-X86-32-LABEL: foo:
+; CHECK-X86-32:       # %bb.0:
+; CHECK-X86-32-NEXT:    movl %esp, %r11d
+; CHECK-X86-32-NEXT:    subl $69632, %r11d # imm = 0x11000
+; CHECK-X86-32-NEXT:  .LBB0_1: # =>This Inner Loop Header: Depth=1
+; CHECK-X86-32-NEXT:    subl $4096, %esp # imm = 0x1000
+; CHECK-X86-32-NEXT:    movl $0, (%esp)
+; CHECK-X86-32-NEXT:    cmpl %r11d, %esp
+; CHECK-X86-32-NEXT:    jne .LBB0_1
+; CHECK-X86-32-NEXT:  # %bb.2:
+; CHECK-X86-32-NEXT:    subl $2380, %esp # imm = 0x94C
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 72016
+; CHECK-X86-32-NEXT:    movl $1, 392(%esp)
+; CHECK-X86-32-NEXT:    movl $1, 28792(%esp)
+; CHECK-X86-32-NEXT:    movl (%esp), %eax
+; CHECK-X86-32-NEXT:    addl $72012, %esp # imm = 0x1194C
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK-X86-32-NEXT:    retl
   %a = alloca i32, i64 18000, align 16
   %b0 = getelementptr inbounds i32, i32* %a, i64 98
   %b1 = getelementptr inbounds i32, i32* %a, i64 7198
   store volatile i32 1, i32* %b0
   store volatile i32 1, i32* %b1
   %c = load volatile i32, i32* %a
   ret i32 %c
 }

llvm/test/CodeGen/X86/stack-clash-medium-natural-probes-mutliple-objects.ll

-; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo() local_unnamed_addr #0 {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
-; CHECK-NEXT:    movq $0, (%rsp)
-; CHECK-NEXT:    subq $1784, %rsp # imm = 0x6F8
+; CHECK-NEXT:    movb $0, -4096(%rsp)
+; CHECK-NEXT:    subq $5880, %rsp # imm = 0x16F8
 ; CHECK-NEXT:    .cfi_def_cfa_offset 5888
-; CHECK-NEXT:    movl $1, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movl $2, {{[0-9]+}}(%rsp)
-; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %eax
+; CHECK-NEXT:    movl $1, 3872(%rsp)
+; CHECK-NEXT:    movl $2, 672(%rsp)
+; CHECK-NEXT:    movl 1872(%rsp), %eax
 ; CHECK-NEXT:    addq $5880, %rsp # imm = 0x16F8
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8
 ; CHECK-NEXT:    retq
   %a = alloca i32, i64 1000, align 16
   %b = alloca i32, i64 500, align 16
   %a0 = getelementptr inbounds i32, i32* %a, i64 500
   %b0 = getelementptr inbounds i32, i32* %b, i64 200
   store volatile i32 1, i32* %a0
   store volatile i32 2, i32* %b0
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-medium-natural-probes.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo() local_unnamed_addr #0 {
-
 ; CHECK-LABEL: foo:
-; CHECK:         # %bb.0:
-; CHECK-NEXT: subq  $4096, %rsp             # imm = 0x1000
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT: subq  $3784, %rsp             # imm = 0xEC8
-; CHECK-NEXT: .cfi_def_cfa_offset 7888
-; CHECK-NEXT: movl  $1, 264(%rsp)
-; CHECK-NEXT: movl  $1, 4664(%rsp)
-; CHECK-NEXT: movl  -128(%rsp), %eax
-; CHECK-NEXT: addq  $7880, %rsp             # imm = 0x1EC8
-; CHECK-NEXT: .cfi_def_cfa_offset 8
-; CHECK-NEXT: retq
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movb $0, -4096(%rsp)
+; CHECK-NEXT:    subq $7880, %rsp # imm = 0x1EC8
+; CHECK-NEXT:    .cfi_def_cfa_offset 7888
+; CHECK-NEXT:    movl $1, 264(%rsp)
+; CHECK-NEXT:    movl $1, 4664(%rsp)
+; CHECK-NEXT:    movl -128(%rsp), %eax
+; CHECK-NEXT:    addq $7880, %rsp # imm = 0x1EC8
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
 
 
 
   %a = alloca i32, i64 2000, align 16
   %b0 = getelementptr inbounds i32, i32* %a, i64 98
   %b1 = getelementptr inbounds i32, i32* %a, i64 1198
   store i32 1, i32* %b0
   store i32 1, i32* %b1
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-medium.ll

-; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s 
-; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s 
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
+; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s
+; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s
 
 define i32 @foo() local_unnamed_addr #0 {
+; CHECK-X86-64-LABEL: foo:
+; CHECK-X86-64:       # %bb.0:
+; CHECK-X86-64-NEXT:    movb $0, -4096(%rsp)
+; CHECK-X86-64-NEXT:    subq $7880, %rsp # imm = 0x1EC8
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_offset 7888
+; CHECK-X86-64-NEXT:    movl $1, 672(%rsp)
+; CHECK-X86-64-NEXT:    movl -128(%rsp), %eax
+; CHECK-X86-64-NEXT:    addq $7880, %rsp # imm = 0x1EC8
+; CHECK-X86-64-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-X86-64-NEXT:    retq
+;
+; CHECK-X86-32-LABEL: foo:
+; CHECK-X86-32:       # %bb.0:
+; CHECK-X86-32-NEXT:    movb $0, -4096(%esp)
+; CHECK-X86-32-NEXT:    subl $8012, %esp # imm = 0x1F4C
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 8016
+; CHECK-X86-32-NEXT:    movl $1, 800(%esp)
+; CHECK-X86-32-NEXT:    movl (%esp), %eax
+; CHECK-X86-32-NEXT:    addl $8012, %esp # imm = 0x1F4C
+; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 4
+; CHECK-X86-32-NEXT:    retl
   %a = alloca i32, i64 2000, align 16
   %b = getelementptr inbounds i32, i32* %a, i64 200
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-small-alloc-medium-align.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 ; | case1 | alloca + align < probe_size
 define i32 @foo1(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LABEL: foo1:
-; CHECK:        # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT:   andq    $-64, %rsp
-; CHECK-NEXT:   subq    $832, %rsp                      # imm = 0x340
-; CHECK-NEXT:   movl    $1, (%rsp,%rdi,4)
-; CHECK-NEXT:   movl    (%rsp), %eax
-; CHECK-NEXT:   movq    %rbp, %rsp
-; CHECK-NEXT:   popq    %rbp
-; CHECK-NEXT: .cfi_def_cfa %rsp, 8
-; CHECK-NEXT: retq
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    andq $-64, %rsp
+; CHECK-NEXT:    subq $832, %rsp # imm = 0x340
+; CHECK-NEXT:    movl $1, (%rsp,%rdi,4)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
   %a = alloca i32, i32 200, align 64
   %b = getelementptr inbounds i32, i32* %a, i64 %i
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 ; | case2 | alloca > probe_size, align > probe_size
 define i32 @foo2(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LABEL: foo2:
-; CHECK:        # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT:   andq    $-2048, %rsp                    # imm = 0xF800
-; CHECK-NEXT:   subq    $2048, %rsp                     # imm = 0x800
-; CHECK-NEXT:   movq    $0, (%rsp)
-; CHECK-NEXT:   subq    $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT:   movq    $0, (%rsp)
-; CHECK-NEXT:   subq    $2048, %rsp                     # imm = 0x800
-; CHECK-NEXT:   movl    $1, (%rsp,%rdi,4)
-; CHECK-NEXT:   movl    (%rsp), %eax
-; CHECK-NEXT:   movq    %rbp, %rsp
-; CHECK-NEXT:   popq    %rbp
-; CHECK-NEXT:   .cfi_def_cfa %rsp, 8
-; CHECK-NEXT:   retq
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    andq $-2048, %rsp # imm = 0xF800
+; CHECK-NEXT:    movb $0, -4096(%rsp)
+; CHECK-NEXT:    subq $8192, %rsp # imm = 0x2000
+; CHECK-NEXT:    movl $1, (%rsp,%rdi,4)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
   %a = alloca i32, i32 2000, align 2048
   %b = getelementptr inbounds i32, i32* %a, i64 %i
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 ; | case3 | alloca < probe_size, align < probe_size, alloca + align > probe_size
 define i32 @foo3(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LABEL: foo3:
-; CHECK:        # %bb.0:
-; CHECK-NEXT:   pushq   %rbp
-; CHECK-NEXT:   .cfi_def_cfa_offset 16
-; CHECK-NEXT:   .cfi_offset %rbp, -16
-; CHECK-NEXT:   movq    %rsp, %rbp
-; CHECK-NEXT:   .cfi_def_cfa_register %rbp
-; CHECK-NEXT:   andq    $-1024, %rsp                    # imm = 0xFC00
-; CHECK-NEXT:   subq    $3072, %rsp                     # imm = 0xC00
-; CHECK-NEXT:   movq    $0, (%rsp)
-; CHECK-NEXT:   subq    $1024, %rsp                     # imm = 0x400
-; CHECK-NEXT:   movl    $1, (%rsp,%rdi,4)
-; CHECK-NEXT:   movl    (%rsp), %eax
-; CHECK-NEXT:   movq    %rbp, %rsp
-; CHECK-NEXT:   popq    %rbp
-; CHECK-NEXT:   .cfi_def_cfa %rsp, 8
-; CHECK-NEXT:   retq
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    andq $-1024, %rsp # imm = 0xFC00
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    movl $1, (%rsp,%rdi,4)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
 
   %a = alloca i32, i32 1000, align 1024
   %b = getelementptr inbounds i32, i32* %a, i64 %i
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 ; | case4 | alloca + probe_size < probe_size, followed by dynamic alloca
 define i32 @foo4(i64 %i) local_unnamed_addr #0 {
 ; CHECK-LABEL: foo4:
-; CHECK:        # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT: pushq %rbx
-; CHECK-NEXT: andq  $-64, %rsp
-; CHECK-NEXT: subq  $896, %rsp                      # imm = 0x380
-; CHECK-NEXT: movq  %rsp, %rbx
-; CHECK-NEXT: .cfi_offset %rbx, -24
-; CHECK-NEXT: movl  $1, (%rbx,%rdi,4)
-; CHECK-NEXT: movl  (%rbx), %ecx
-; CHECK-NEXT: movq  %rsp, %rax
-; CHECK-NEXT: leaq  15(,%rcx,4), %rcx
-; CHECK-NEXT: andq  $-16, %rcx
-; CHECK-NEXT: subq  %rcx, %rax
-; CHECK-NEXT: cmpq  %rsp, %rax
-; CHECK-NEXT: jge .LBB3_3
-; CHECK-NEXT:.LBB3_2:                                # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: xorq  $0, (%rsp)
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: cmpq  %rsp, %rax
-; CHECK-NEXT: jl  .LBB3_2
-; CHECK-NEXT:.LBB3_3:
-; CHECK-NEXT: andq  $-64, %rax
-; CHECK-NEXT: movq  %rax, %rsp
-; CHECK-NEXT: movl  (%rax), %eax
-; CHECK-NEXT: leaq  -8(%rbp), %rsp
-; CHECK-NEXT: popq  %rbx
-; CHECK-NEXT: popq  %rbp
-; CHECK-NEXT: .cfi_def_cfa %rsp, 8
-; CHECK-NEXT: retq
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    pushq %rbx
+; CHECK-NEXT:    andq $-64, %rsp
+; CHECK-NEXT:    subq $896, %rsp # imm = 0x380
+; CHECK-NEXT:    movq %rsp, %rbx
+; CHECK-NEXT:    .cfi_offset %rbx, -24
+; CHECK-NEXT:    movl $1, (%rbx,%rdi,4)
+; CHECK-NEXT:    movl (%rbx), %ecx
+; CHECK-NEXT:    movq %rsp, %rax
+; CHECK-NEXT:    leaq 15(,%rcx,4), %rcx
+; CHECK-NEXT:    andq $-16, %rcx
+; CHECK-NEXT:    subq %rcx, %rax
+; CHECK-NEXT:    cmpq %rsp, %rax
+; CHECK-NEXT:    jge .LBB3_3
+; CHECK-NEXT:  .LBB3_2: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    xorq $0, (%rsp)
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    cmpq %rsp, %rax
+; CHECK-NEXT:    jl .LBB3_2
+; CHECK-NEXT:  .LBB3_3:
+; CHECK-NEXT:    andq $-64, %rax
+; CHECK-NEXT:    movq %rax, %rsp
+; CHECK-NEXT:    movl (%rax), %eax
+; CHECK-NEXT:    leaq -8(%rbp), %rsp
+; CHECK-NEXT:    popq %rbx
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
   %a = alloca i32, i32 200, align 64
   %b = getelementptr inbounds i32, i32* %a, i64 %i
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   %d = alloca i32, i32 %c, align 64
   %e = load volatile i32, i32* %d
   ret i32 %e
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}
 

llvm/test/CodeGen/X86/stack-clash-small-large-align.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo_noprotect() local_unnamed_addr {
 ; CHECK-LABEL: foo_noprotect:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT: andq  $-65536, %rsp
-; CHECK-NEXT: subq  $65536, %rsp
-; CHECK-NEXT: movl  $1, 392(%rsp)
-; CHECK-NEXT: movl  (%rsp), %eax
-; CHECK-NEXT: movq  %rbp, %rsp
-; CHECK-NEXT: popq  %rbp
-; CHECK-NEXT: .cfi_def_cfa %rsp, 8
-; CHECK-NEXT: retq
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    andq $-65536, %rsp # imm = 0xFFFF0000
+; CHECK-NEXT:    subq $65536, %rsp # imm = 0x10000
+; CHECK-NEXT:    movl $1, 392(%rsp)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
 
 
   %a = alloca i32, i64 100, align 65536
   %b = getelementptr inbounds i32, i32* %a, i64 98
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 define i32 @foo_protect() local_unnamed_addr #0 {
 ; CHECK-LABEL: foo_protect:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT: pushq %rbp
-; CHECK-NEXT: .cfi_def_cfa_offset 16
-; CHECK-NEXT: .cfi_offset %rbp, -16
-; CHECK-NEXT: movq  %rsp, %rbp
-; CHECK-NEXT: .cfi_def_cfa_register %rbp
-; CHECK-NEXT: movq  %rsp, %r11
-; CHECK-NEXT: andq  $-65536, %r11                   # imm = 0xFFFF0000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: je  .LBB1_4
-; CHECK-NEXT:# %bb.1:
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: jb  .LBB1_3
-; CHECK-NEXT:.LBB1_2:                                # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: cmpq  %rsp, %r11
-; CHECK-NEXT: jb  .LBB1_2
-; CHECK-NEXT:.LBB1_3:
-; CHECK-NEXT: movq  %r11, %rsp
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT:.LBB1_4:
-; CHECK-NEXT: movq  %rsp, %r11
-; CHECK-NEXT: subq  $65536, %r11                    # imm = 0x10000
-; CHECK-NEXT:.LBB1_5:                                # =>This Inner Loop Header: Depth=1
-; CHECK-NEXT: subq  $4096, %rsp                     # imm = 0x1000
-; CHECK-NEXT: movq  $0, (%rsp)
-; CHECK-NEXT: cmpq  %r11, %rsp
-; CHECK-NEXT: jne .LBB1_5
-; CHECK-NEXT:# %bb.6:
-; CHECK-NEXT: movl  $1, 392(%rsp)
-; CHECK-NEXT: movl  (%rsp), %eax
-; CHECK-NEXT: movq  %rbp, %rsp
-; CHECK-NEXT: popq  %rbp
-; CHECK-NEXT: .cfi_def_cfa %rsp, 8
-; CHECK-NEXT: retq
+; CHECK-NEXT:    pushq %rbp
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    .cfi_offset %rbp, -16
+; CHECK-NEXT:    movq %rsp, %rbp
+; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    movq %rsp, %r11
+; CHECK-NEXT:    andq $-65536, %r11 # imm = 0xFFFF0000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    je .LBB1_4
+; CHECK-NEXT:  # %bb.1:
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    jb .LBB1_3
+; CHECK-NEXT:  .LBB1_2: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    cmpq %rsp, %r11
+; CHECK-NEXT:    jb .LBB1_2
+; CHECK-NEXT:  .LBB1_3:
+; CHECK-NEXT:    movq %r11, %rsp
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:  .LBB1_4:
+; CHECK-NEXT:    movq %rsp, %r11
+; CHECK-NEXT:    subq $65536, %r11 # imm = 0x10000
+; CHECK-NEXT:  .LBB1_5: # =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    movq $0, (%rsp)
+; CHECK-NEXT:    cmpq %r11, %rsp
+; CHECK-NEXT:    jne .LBB1_5
+; CHECK-NEXT:  # %bb.6:
+; CHECK-NEXT:    movl $1, 392(%rsp)
+; CHECK-NEXT:    movl (%rsp), %eax
+; CHECK-NEXT:    movq %rbp, %rsp
+; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
+; CHECK-NEXT:    retq
 
 
 
 
   %a = alloca i32, i64 100, align 65536
   %b = getelementptr inbounds i32, i32* %a, i64 98
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-small.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo() local_unnamed_addr #0 {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:  subq $280, %rsp # imm = 0x118
-; CHECK-NEXT:  .cfi_def_cfa_offset 288
-; CHECK-NEXT:  movl $1, 264(%rsp)
-; CHECK-NEXT:  movl -128(%rsp), %eax
-; CHECK-NEXT:  addq $280, %rsp # imm = 0x118
-; CHECK-NEXT:  .cfi_def_cfa_offset 8
-; CHECK-NEXT:  retq
+; CHECK-NEXT:    subq $280, %rsp # imm = 0x118
+; CHECK-NEXT:    .cfi_def_cfa_offset 288
+; CHECK-NEXT:    movl $1, 264(%rsp)
+; CHECK-NEXT:    movl -128(%rsp), %eax
+; CHECK-NEXT:    addq $280, %rsp # imm = 0x118
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
 
   %a = alloca i32, i64 100, align 16
   %b = getelementptr inbounds i32, i32* %a, i64 98
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}

llvm/test/CodeGen/X86/stack-clash-unknown-call.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py UTC_ARGS: --no_x86_scrub_sp
 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg);
 
 define void @foo() local_unnamed_addr #0 {
-
-;CHECK-LABEL: foo:
-;CHECK:         # %bb.0:
-;CHECK-NEXT:  subq  $4096, %rsp # imm = 0x1000
-; it's important that we don't use the call as a probe here
-;CHECK-NEXT:  movq  $0, (%rsp)
-;CHECK-NEXT:  subq  $3912, %rsp # imm = 0xF48
-;CHECK-NEXT:  .cfi_def_cfa_offset 8016
-;CHECK-NEXT:  movq  %rsp, %rdi
-;CHECK-NEXT:  movl  $8000, %edx # imm = 0x1F40
-;CHECK-NEXT:  xorl  %esi, %esi
-;CHECK-NEXT:  callq memset
-;CHECK-NEXT:  addq  $8008, %rsp # imm = 0x1F48
-;CHECK-NEXT:  .cfi_def_cfa_offset 8
-;CHECK-NEXT:  retq
+; CHECK-LABEL: foo:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    movb $0, -4096(%rsp)
+; CHECK-NEXT:    subq $8008, %rsp # imm = 0x1F48
+; CHECK-NEXT:    .cfi_def_cfa_offset 8016
+; CHECK-NEXT:    movq %rsp, %rdi
+; CHECK-NEXT:    movl $8000, %edx # imm = 0x1F40
+; CHECK-NEXT:    xorl %esi, %esi
+; CHECK-NEXT:    callq memset@PLT
+; CHECK-NEXT:    addq $8008, %rsp # imm = 0x1F48
+; CHECK-NEXT:    .cfi_def_cfa_offset 8
+; CHECK-NEXT:    retq
+; it's important that we don't use the call as a probe here
 
   %a = alloca i8, i64 8000, align 16
   call void @llvm.memset.p0i8.i64(i8* align 16 %a, i8 0, i64 8000, i1 false)
   ret void
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}