[PEI] add dwarf information for stack probe

[YangKeao]

Summary

While probing stack, the stack register is moved without
dwarf information, which could cause panic if unwind the
backtrace at that point. This commit add dwarf information
for these operation, and use r11 (instead of rsp) to iterate
over pages to probe.

[YangKeao]

There has been a bug report for this on bugzilla. A more "downstream" context for this feature is discussed in rust#83139.

[nagisa]

llvm/test/CodeGen/X86/stack-clash-unknown-call.ll

18

The def_cfa_offset is superfluous here now. The last cfi directive should be either the adjust or def.

[efriedma]

Generally, unlike the formats used on Windows etc., DWARF unwind isn't accurate in the prologue. I mean, you could add separate unwind info for each relevant instruction, but that would be a lot of data, and we currently don't have any option to do that.

Given that, I'm not sure what you're trying to accomplish here. I can't see how adding more .cfi_adjust_cfa_offset directives does anything useful.

[efriedma]

Err, sorry, please pretend I didn't write that. I have no idea what I just wrote.

[efriedma]

To clarify, I've done some more reading now, and figured out where I went wrong. For a long time, LLVM did not emit accurate unwind info to describe the prologue/epilogue (and still doesn't on some targets), so I was under the impression it wasn't possible. Clearly, it is, and it's implemented on x86.

---

The change to use r11 isn't implemented correctly: we can't adjust the stack pointer until *after* we've probed the relevant pages. It'll appear to work, but it won't actually provide complete protection if a signal handler triggers at the wrong time.

[YangKeao]

In D98789#2632702, @efriedma wrote:

To clarify, I've done some more reading now, and figured out where I went wrong. For a long time, LLVM did not emit accurate unwind info to describe the prologue/epilogue (and still doesn't on some targets), so I was under the impression it wasn't possible. Clearly, it is, and it's implemented on x86.

---

The change to use r11 isn't implemented correctly: we can't adjust the stack pointer until *after* we've probed the relevant pages. It'll appear to work, but it won't actually provide complete protection if a signal handler triggers at the wrong time.

Make sense. I tried to use r11 + offset to represent CFA temporarily. However, r11d cannot be used as a dwarf register on x86_32. Can I use another register (like di) here?

[YangKeao]

• Use rdi to represent the stack bound and CFA
• Remove extra tailing offset adjust

[YangKeao]

llvm/test/CodeGen/X86/stack-clash-unknown-call.ll

18

The def_cfa_offset is superfluous here now. The last cfi directive should be either the adjust or def.

18

Fixed by removing the last adjust

[YangKeao]

reformat the code

[nagisa]

Make sense. I tried to use r11 + offset to represent CFA temporarily. However, r11d cannot be used as a dwarf register on x86_32.

What registers can be used? I did a quick search and couldn't find anything.

Can I use another register (like di) here?

Any register that isn't callee-preserved could be used (or any register in general if its spilled first), assuming it isn't used for something else already. I did a quick search in an effort to figure out which registers are callee-saved on x86, but couldn't find anything definitive :(

This makes me wonder, though: why not leave selection of the register to use here to regalloc?

llvm/test/CodeGen/X86/stack-clash-large.ll

23

I… think this wants to be a def_cfa_offset? def_cfa_register does not reset the offset so its not at all obvious what this is offsetting from.

Alternatively there's a form that combines both setting the new register and the offset into a single directive:

.cfi_def_cfa %rdi, 69632

[YangKeao]

What registers can be used? I did a quick search and couldn't find anything.

I found the available dwarf registers under the x86 register table in LLVM (X86RegisterInfo.td).

Any register that isn't callee-preserved could be used (or any register in general if its spilled first), assuming it isn't used for something else already. I did a quick search in an effort to figure out which registers are callee-saved on x86, but couldn't find anything definitive :(

It seems that different platform will have different set of callee-preserved register. I also found them in the list in LLVM (which is refered in the function getCalleeSavedRegs of llvm/lib/Target/X86/X86RegisterInfo.cpp). Sadly, it seems like RDI is more likely to be a callee-saved register.

Only considering the SYSV and Windows, all callee-saved registers are:

X86::RBX, X86::R12, X86::R13, X86::R14, X86::R15, X86::RBP
X86::RBX, X86::RBP, X86::RDI, X86::RSI, X86::R12, X86::R13, X86::R14, X86::R15, X86::XMM6, X86::XMM7, X86::XMM8, X86::XMM9, X86::XMM10, X86::XMM11, X86::XMM12, X86::XMM13, X86::XMM14, X86::XMM15

And all available dwarf registers are:

32bit:
EAX, EDX, ECX, EBX, ESI, EDI, EBP, ESP, EIP
64bit, X86-64:
RAX, RDX, RCX, RBX, RSI, RDI, RBP, RSP, R8-R15, RIP

For 64bit, RAX, RCX, RDX, R8-R11 could be a good choice, and for 32 bit, it could only be choosen from EAX, ECX, EDX

This makes me wonder, though: why not leave selection of the register to use here to regalloc?

Sounds like a good idea. But I'm not familiar with LLVM codes. Is there any example on how to use the regalloc?

(I don't know whether it will make this patch too complicate. Allocating stack probe register with regalloc and fixing the dwarf sound like two things and I'd like to do it in a separated patch.)

llvm/test/CodeGen/X86/stack-clash-large.ll

23

The problem is that I don't know the accurate offset. When the callee-saved registers are pushed to the stack, there will be an offset before probing the stack (if I understand the prolog part correctly, please tell me if I'm wrong).

[YangKeao]

Use RDX/EDX instead of RDI/EDI, as RDI/EDI is callee saved on Win64 calling convension

[nagisa]

In D98789#2634751, @YangKeao wrote:

Any register that isn't callee-preserved could be used (or any register in general if its spilled first), assuming it isn't used for something else already. I did a quick search in an effort to figure out which registers are callee-saved on x86, but couldn't find anything definitive :(

It seems that different platform will have different set of callee-preserved register. I also found them in the list in LLVM (which is refered in the function getCalleeSavedRegs of llvm/lib/Target/X86/X86RegisterInfo.cpp). Sadly, it seems like RDI is more likely to be a callee-saved register.

Only considering the SYSV and Windows, all callee-saved registers are:

X86::RBX, X86::R12, X86::R13, X86::R14, X86::R15, X86::RBP
X86::RBX, X86::RBP, X86::RDI, X86::RSI, X86::R12, X86::R13, X86::R14, X86::R15, X86::XMM6, X86::XMM7, X86::XMM8, X86::XMM9, X86::XMM10, X86::XMM11, X86::XMM12, X86::XMM13, X86::XMM14, X86::XMM15

And all available dwarf registers are:

32bit:
EAX, EDX, ECX, EBX, ESI, EDI, EBP, ESP, EIP
64bit, X86-64:
RAX, RDX, RCX, RBX, RSI, RDI, RBP, RSP, R8-R15, RIP

For 64bit, RAX, RCX, RDX, R8-R11 could be a good choice, and for 32 bit, it could only be choosen from EAX, ECX, EDX

A care must be taken to not overwrite the arguments as well. For instance on SysV x86_64 ABI rdi, rsi, rdx, rcx, r8, r9 are used to pass in integer arguments. For functions with a small number of arguments one of these could be reused, but if a function happens to use all of them, unconditional use of rdx would clobber the argument.

In D98789#2634751, @YangKeao wrote:

Sounds like a good idea. But I'm not familiar with LLVM codes. Is there any example on how to use the regalloc?

(I don't know whether it will make this patch too complicate. Allocating stack probe register with regalloc and fixing the dwarf sound like two things and I'd like to do it in a separated patch.)

You would probably have to split the patches up into distinct parts, yes (first one adjusting the backend to allocate virtual registers and the next being this one).

I'll get back to you with advice on how to specify a virtual register a little bit later –

[nagisa]

See createVirtualRegister.

[YangKeao]

Use RAX/EAX as the iterate register, as RDX/EDX is used as arguments under systemv

[YangKeao]

A care must be taken to not overwrite the arguments as well. For instance on SysV x86_64 ABI rdi, rsi, rdx, rcx, r8, r9 are used to pass in integer arguments. For functions with a small number of arguments one of these could be reused, but if a function happens to use all of them, unconditional use of rdx would clobber the argument.

Nice catch! After eliminating RCX, RDX from the selections, it seems that the only possible "always correct" choice is RAX (being used as the return value doesn't bother).

See createVirtualRegister.

Thanks. I have seen some usage of this function. It seems that there isn't any suitable RegClass for this situation ("dwarf suitable registers class", is there an equivalent one?). Should I create one by myself?

[nagisa]

I investigated this a little bit, and it seems like using createVirtualRegister won't work here, after all. It seems that this code does in fact run after regalloc happens. I'm out of ideas, but I also am not that familiar with this area to give any useful advice or guidance here.

[efriedma]

Normally, I'd expect some register is naturally free in the prologue, but you could get into weird situations. On 32-bit specifically, consider compiling with -mregparm=3; I think there are no registers which are unconditionally safe in that case. One possibility is to always use EAX, and just save/restore it if necessary. See isEAXAlive in X86FrameLowering::emitPrologue.

Alternatively, you could ensure that some callee-save GPR is spilled, and explicitly use that register. This is taking advantage of the fact this is part of the prologue: there can't be any other uses of callee-save registers at that point. (In theory, it might be possibly for an exotic calling convention to have no callee-save registers, but I don't think there are any in practice.)

Outside the prologue, the allocation should be represented by some instruction; that instruction should clobber some register, and regalloc will ensure that register is free.

[efriedma]

llvm/test/CodeGen/X86/stack-clash-large.ll

38

BTW, this is completely broken; r11d doesn't exist on 32-bit x86.

[YangKeao]

llvm/test/CodeGen/X86/stack-clash-large.ll

38

Wow, surprising discovery. (I think) a "bad register name" should be given when compiling this codes?

Is there any pass which will omit this problem? Run clang -fstack-clash-protection -m32 -fomit-frame-pointer -S will generate codes containing r11d, which is bad. However, clang -fstack-clash-protection -m32 -fomit-frame-pointer -c and disassemble the output, the register used here will become ebx.

[YangKeao]

Normally, I'd expect some register is naturally free in the prologue, but you could get into weird situations. On 32-bit specifically, consider compiling with -mregparm=3; I think there are no registers which are unconditionally safe in that case. One possibility is to always use EAX, and just save/restore it if necessary. See isEAXAlive in X86FrameLowering::emitPrologue.

Alternatively, you could ensure that some callee-save GPR is spilled, and explicitly use that register. This is taking advantage of the fact this is part of the prologue: there can't be any other uses of callee-save registers at that point. (In theory, it might be possibly for an exotic calling convention to have no callee-save registers, but I don't think there are any in practice.)

Both solutions seems to be either complicate or with extra cost. And surprisingly found that the original implementation of stack probe is wrong on 32bit. Given that, I preferred to only provide DWARF information in 64bit situation (and left a comment), so that r11 can be used and solve this problem easily.

[YangKeao]

Remove dwarf information for 32bit and use R11 as the iterate bound / dwarf register

[YangKeao]

left comments about 32bit

[nagisa]

btw I prototyped a D98906: [X86] Improve lowering of the unrolled inline-asm probing yesterday as an alternative approach towards improving the unrolled case.

[YangKeao]

In D98789#2636879, @nagisa wrote:

btw I prototyped a D98906: [X86] Improve lowering of the unrolled inline-asm probing yesterday as an alternative approach towards improving the unrolled case.

Great! It seems better. I will rebase on it.

[nagisa]

Looks like my alternative has caveats of its own. Can you please split your change into two parts? One that affects only the unrolled case (which I believe should be good to land) and the other which affects the loop case (potentially subject to further discussion).

[nagisa]

@YangKeao Will you be pursuing this further? Should I take over this for you?

[YangKeao]

In D98789#2657046, @nagisa wrote:

@YangKeao Will you be pursuing this further? Should I take over this for you?

Oops. Sorry, I missed the former comment "Looks like my alternative has caveats of its own.". I will split this patch into two parts right now. Thanks.

[YangKeao]

Stack probing with loop should be discussed further in D99585.

llvm/lib/Target/X86/X86FrameLowering.h

public:
 
   Register getInitialCFARegister(const MachineFunction &MF) const override;
 
   /// Return true if the function has a redzone (accessible bytes past the
   /// frame of the top of stack function) as part of it's ABI.
   bool has128ByteRedZone(const MachineFunction& MF) const;
 
 private:
+  bool isWin64Prologue(const MachineFunction &MF) const;
+
+  bool needsDwarfCFI(const MachineFunction &MF) const;
+
   uint64_t calculateMaxStackAlign(const MachineFunction &MF) const;
 
   /// Emit target stack probe as a call to a helper function
   void emitStackProbeCall(MachineFunction &MF, MachineBasicBlock &MBB,
                           MachineBasicBlock::iterator MBBI, const DebugLoc &DL,
                           bool InProlog) const;
 
   /// Emit target stack probe as an inline sequence.

llvm/lib/Target/X86/X86FrameLowering.cpp

void X86FrameLowering::emitStackProbeInlineGeneric(
   }
 }
 
 void X86FrameLowering::emitStackProbeInlineGenericBlock(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MBBI, const DebugLoc &DL, uint64_t Offset,
     uint64_t AlignOffset) const {
 
+  const bool NeedsDwarfCFI = needsDwarfCFI(MF);
+  const bool HasFP = hasFP(MF);
   const X86Subtarget &STI = MF.getSubtarget<X86Subtarget>();
   const X86TargetLowering &TLI = *STI.getTargetLowering();
   const unsigned Opc = getSUBriOpcode(Uses64BitFramePtr, Offset);
   const unsigned MovMIOpc = Is64Bit ? X86::MOV64mi32 : X86::MOV32mi;
   const uint64_t StackProbeSize = TLI.getStackProbeSize(MF);
 
   uint64_t CurrentOffset = 0;
 
   assert(AlignOffset < StackProbeSize);
 
   // If the offset is so small it fits within a page, there's nothing to do.
   if (StackProbeSize < Offset + AlignOffset) {
 
     MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
                            .addReg(StackPtr)
                            .addImm(StackProbeSize - AlignOffset)
                            .setMIFlag(MachineInstr::FrameSetup);
     MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
 
+    if (!HasFP && NeedsDwarfCFI) {
+      BuildCFI(MBB, MBBI, DL,
+               MCCFIInstruction::createAdjustCfaOffset(
+                   nullptr, StackProbeSize - AlignOffset));
+    }
+
     addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(MovMIOpc))
                      .setMIFlag(MachineInstr::FrameSetup),
                  StackPtr, false, 0)
         .addImm(0)
         .setMIFlag(MachineInstr::FrameSetup);
     NumFrameExtraProbe++;
     CurrentOffset = StackProbeSize - AlignOffset;
   }
 
   // For the next N - 1 pages, just probe. I tried to take advantage of
   // natural probes but it implies much more logic and there was very few
   // interesting natural probes to interleave.
   while (CurrentOffset + StackProbeSize < Offset) {
     MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
                            .addReg(StackPtr)
                            .addImm(StackProbeSize)
                            .setMIFlag(MachineInstr::FrameSetup);
     MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
 
+    if (!HasFP && NeedsDwarfCFI) {
+      BuildCFI(
+          MBB, MBBI, DL,
+          MCCFIInstruction::createAdjustCfaOffset(nullptr, StackProbeSize));
+    }
 
     addRegOffset(BuildMI(MBB, MBBI, DL, TII.get(MovMIOpc))
                      .setMIFlag(MachineInstr::FrameSetup),
                  StackPtr, false, 0)
         .addImm(0)
         .setMIFlag(MachineInstr::FrameSetup);
     NumFrameExtraProbe++;
     CurrentOffset += StackProbeSize;
   }
 
   // No need to probe the tail, it is smaller than a Page.
   uint64_t ChunkSize = Offset - CurrentOffset;
   MachineInstr *MI = BuildMI(MBB, MBBI, DL, TII.get(Opc), StackPtr)
                          .addReg(StackPtr)
                          .addImm(ChunkSize)
                          .setMIFlag(MachineInstr::FrameSetup);
+  // No need to adjust Dwarf CFA offset here, the last position of the stack has
+  // been defined
   MI->getOperand(3).setIsDead(); // The EFLAGS implicit def is dead.
 }
 
 void X86FrameLowering::emitStackProbeInlineGenericLoop(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MBBI, const DebugLoc &DL, uint64_t Offset,
     uint64_t AlignOffset) const {
   assert(Offset && "null offset");
 
+  const bool NeedsDwarfCFI = needsDwarfCFI(MF);
+  const bool HasFP = hasFP(MF);
   const X86Subtarget &STI = MF.getSubtarget<X86Subtarget>();
   const X86TargetLowering &TLI = *STI.getTargetLowering();
   const unsigned MovMIOpc = Is64Bit ? X86::MOV64mi32 : X86::MOV32mi;
   const uint64_t StackProbeSize = TLI.getStackProbeSize(MF);
 
   if (AlignOffset) {
     if (AlignOffset < StackProbeSize) {
       // Perform a first smaller allocation followed by a probe.
void X86FrameLowering::emitStackProbeInlineGenericLoop(
 
   Register FinalStackProbed = Uses64BitFramePtr ? X86::R11 : X86::R11D;
   BuildMI(MBB, MBBI, DL, TII.get(TargetOpcode::COPY), FinalStackProbed)
       .addReg(StackPtr)
       .setMIFlag(MachineInstr::FrameSetup);
 
   // save loop bound
   {
+    const unsigned BoundOffset = Offset / StackProbeSize * StackProbeSize;
     const unsigned SUBOpc = getSUBriOpcode(Uses64BitFramePtr, Offset);
     BuildMI(MBB, MBBI, DL, TII.get(SUBOpc), FinalStackProbed)
         .addReg(FinalStackProbed)
-        .addImm(Offset / StackProbeSize * StackProbeSize)
+        .addImm(BoundOffset)
         .setMIFlag(MachineInstr::FrameSetup);
+
+    // TODO: generate DWARF information for 32bit architecture
+    // didn't generate dwarf information on 32bit, because R11D is not usable
+    // and decide a safe register could be complicate or with extra cost
+    if (!HasFP && NeedsDwarfCFI && Uses64BitFramePtr) {
+      BuildCFI(MBB, MBBI, DL,
+               MCCFIInstruction::createDefCfaRegister(
+                   nullptr, TRI->getDwarfRegNum(FinalStackProbed, true)));
+      BuildCFI(MBB, MBBI, DL,
+               MCCFIInstruction::createAdjustCfaOffset(nullptr, BoundOffset));
+    }
   }
 
   // allocate a page
   {
     const unsigned SUBOpc = getSUBriOpcode(Uses64BitFramePtr, StackProbeSize);
     BuildMI(testMBB, DL, TII.get(SUBOpc), StackPtr)
         .addReg(StackPtr)
         .addImm(StackProbeSize)
void X86FrameLowering::emitStackProbeInlineGenericLoop(
 
   // BB management
   tailMBB->splice(tailMBB->end(), &MBB, MBBI, MBB.end());
   tailMBB->transferSuccessorsAndUpdatePHIs(&MBB);
   MBB.addSuccessor(testMBB);
 
   // handle tail
   unsigned TailOffset = Offset % StackProbeSize;
+  MachineBasicBlock::iterator TailMBBIter = tailMBB->begin();
   if (TailOffset) {
     const unsigned Opc = getSUBriOpcode(Uses64BitFramePtr, TailOffset);
-    BuildMI(*tailMBB, tailMBB->begin(), DL, TII.get(Opc), StackPtr)
+    BuildMI(*tailMBB, TailMBBIter, DL, TII.get(Opc), StackPtr)
         .addReg(StackPtr)
         .addImm(TailOffset)
         .setMIFlag(MachineInstr::FrameSetup);
   }
 
+  // TODO: generate DWARF information for 32bit architecture
+  // didn't generate dwarf information on 32bit, because R11D is not usable
+  // and decide a safe register could be complicate or with extra cost
+  if (!HasFP && NeedsDwarfCFI && Uses64BitFramePtr) {
+    BuildCFI(*tailMBB, TailMBBIter, DL,
+             MCCFIInstruction::createDefCfaRegister(
+                 nullptr, TRI->getDwarfRegNum(StackPtr, true)));
+  }
+
   // Update Live In information
   recomputeLiveIns(*testMBB);
   recomputeLiveIns(*tailMBB);
 }
 
 void X86FrameLowering::emitStackProbeInlineWindowsCoreCLR64(
     MachineFunction &MF, MachineBasicBlock &MBB,
     MachineBasicBlock::iterator MBBI, const DebugLoc &DL, bool InProlog) const {
bool X86FrameLowering::has128ByteRedZone(const MachineFunction& MF) const {
   // clobbered by any interrupt handler.
   assert(&STI == &MF.getSubtarget<X86Subtarget>() &&
          "MF used frame lowering for wrong subtarget");
   const Function &Fn = MF.getFunction();
   const bool IsWin64CC = STI.isCallingConvWin64(Fn.getCallingConv());
   return Is64Bit && !IsWin64CC && !Fn.hasFnAttribute(Attribute::NoRedZone);
 }
 
+bool X86FrameLowering::isWin64Prologue(const MachineFunction &MF) const {
+  return MF.getTarget().getMCAsmInfo()->usesWindowsCFI();
+}
+
+bool X86FrameLowering::needsDwarfCFI(const MachineFunction &MF) const {
+  return !isWin64Prologue(MF) && MF.needsFrameMoves();
+}
 
 /// emitPrologue - Push callee-saved registers onto the stack, which
 /// automatically adjust the stack pointer. Adjust the stack pointer to allocate
 /// space for local variables. Also emit labels used by the exception handler to
 /// generate the exception handling frames.
 
 /*
   Here's a gist of what gets emitted:
void X86FrameLowering::emitPrologue(MachineFunction &MF,
   bool IsFunclet = MBB.isEHFuncletEntry();
   EHPersonality Personality = EHPersonality::Unknown;
   if (Fn.hasPersonalityFn())
     Personality = classifyEHPersonality(Fn.getPersonalityFn());
   bool FnHasClrFunclet =
       MF.hasEHFunclets() && Personality == EHPersonality::CoreCLR;
   bool IsClrFunclet = IsFunclet && FnHasClrFunclet;
   bool HasFP = hasFP(MF);
-  bool IsWin64Prologue = MF.getTarget().getMCAsmInfo()->usesWindowsCFI();
+  bool IsWin64Prologue = isWin64Prologue(MF);
   bool NeedsWin64CFI = IsWin64Prologue && Fn.needsUnwindTableEntry();
   // FIXME: Emit FPO data for EH funclets.
   bool NeedsWinFPO =
       !IsFunclet && STI.isTargetWin32() && MMI.getModule()->getCodeViewFlag();
   bool NeedsWinCFI = NeedsWin64CFI || NeedsWinFPO;
-  bool NeedsDwarfCFI = !IsWin64Prologue && MF.needsFrameMoves();
+  bool NeedsDwarfCFI = needsDwarfCFI(MF);
   Register FramePtr = TRI->getFrameRegister(MF);
   const Register MachineFramePtr =
       STI.isTarget64BitILP32()
           ? Register(getX86SubSuperRegister(FramePtr, 64)) : FramePtr;
   Register BasePtr = TRI->getBaseRegister();
   bool HasWinCFI = false;
 
   // Debug location must be unknown since the first debug location is used

llvm/test/CodeGen/X86/stack-clash-large.ll

 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}
 
 ; CHECK-X86-64-LABEL: foo:
 ; CHECK-X86-64:        # %bb.0:
 ; CHECK-X86-64-NEXT:  movq  %rsp, %r11
 ; CHECK-X86-64-NEXT:  subq  $69632, %r11 # imm = 0x11000
+; CHECK-X86-64-NEXT:  .cfi_def_cfa_register %r11
+; CHECK-X86-64-NEXT:  .cfi_adjust_cfa_offset 69632
 ; CHECK-X86-64-NEXT:   .LBB0_1:
 ; CHECK-X86-64-NEXT:  subq  $4096, %rsp # imm = 0x1000
 ; CHECK-X86-64-NEXT:  movq  $0, (%rsp)
 ; CHECK-X86-64-NEXT:  cmpq  %r11, %rsp
 ; CHECK-X86-64-NEXT:  jne .LBB0_1
 ; CHECK-X86-64-NEXT:# %bb.2:
-; CHECK-X86-64-NEXT:  subq    $2248, %rsp
+; CHECK-X86-64-NEXT:  subq  $2248, %rsp
+; CHECK-X86-64-NEXT:  .cfi_def_cfa_register %rsp
 ; CHECK-X86-64-NEXT:  .cfi_def_cfa_offset 71888
 ; CHECK-X86-64-NEXT:  movl  $1, 264(%rsp)
 ; CHECK-X86-64-NEXT:  movl  $1, 28664(%rsp)
 ; CHECK-X86-64-NEXT:  movl  -128(%rsp), %eax
 ; CHECK-X86-64-NEXT:  addq  $71880, %rsp # imm = 0x118C8
 ; CHECK-X86-64-NEXT:  .cfi_def_cfa_offset 8
 ; CHECK-X86-64-NEXT:  retq
 
 ; CHECK-X86-32-LABEL: foo:
 ; CHECK-X86-32:      # %bb.0:
 ; CHECK-X86-32-NEXT:    movl    %esp, %r11d
 ; CHECK-X86-32-NEXT:    subl    $69632, %r11d # imm = 0x11000
 ; CHECK-X86-32-NEXT:    .LBB0_1: # =>This Inner Loop Header: Depth=1
 ; CHECK-X86-32-NEXT:    subl    $4096, %esp # imm = 0x1000
 ; CHECK-X86-32-NEXT:    movl    $0, (%esp)
 ; CHECK-X86-32-NEXT:    cmpl    %r11d, %esp
 ; CHECK-X86-32-NEXT:    jne  .LBB0_1
 ; CHECK-X86-32-NEXT:# %bb.2:
 ; CHECK-X86-32-NEXT:    subl    $2380, %esp
 ; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 72016
 ; CHECK-X86-32-NEXT:    movl    $1, 392(%esp)
 ; CHECK-X86-32-NEXT:    movl    $1, 28792(%esp)
 ; CHECK-X86-32-NEXT:    movl    (%esp), %eax
 ; CHECK-X86-32-NEXT:    addl    $72012, %esp # imm = 0x1194C
 ; CHECK-X86-32-NEXT:    .cfi_def_cfa_offset 4
 ; CHECK-X86-32-NEXT:    retl

llvm/test/CodeGen/X86/stack-clash-medium-natural-probes-mutliple-objects.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo() local_unnamed_addr #0 {
 ; CHECK-LABEL: foo:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    subq $4096, %rsp # imm = 0x1000
+; CHECK-NEXT:    .cfi_adjust_cfa_offset 4096
 ; CHECK-NEXT:    movq $0, (%rsp)
 ; CHECK-NEXT:    subq $1784, %rsp # imm = 0x6F8
 ; CHECK-NEXT:    .cfi_def_cfa_offset 5888
 ; CHECK-NEXT:    movl $1, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movl $2, {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    movl {{[0-9]+}}(%rsp), %eax
 ; CHECK-NEXT:    addq $5880, %rsp # imm = 0x16F8
 ; CHECK-NEXT:    .cfi_def_cfa_offset 8

llvm/test/CodeGen/X86/stack-clash-medium-natural-probes.ll

 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i32 @foo() local_unnamed_addr #0 {
 
 ; CHECK-LABEL: foo:
 ; CHECK:         # %bb.0:
 ; CHECK-NEXT: subq  $4096, %rsp             # imm = 0x1000
+; CHECK-NEXT: .cfi_adjust_cfa_offset 4096
 ; CHECK-NEXT: movq  $0, (%rsp)
 ; CHECK-NEXT: subq  $3784, %rsp             # imm = 0xEC8
 ; CHECK-NEXT: .cfi_def_cfa_offset 7888
 ; CHECK-NEXT: movl  $1, 264(%rsp)
 ; CHECK-NEXT: movl  $1, 4664(%rsp)
 ; CHECK-NEXT: movl  -128(%rsp), %eax
 ; CHECK-NEXT: addq  $7880, %rsp             # imm = 0x1EC8
 ; CHECK-NEXT: .cfi_def_cfa_offset 8

llvm/test/CodeGen/X86/stack-clash-medium.ll

 ; RUN: llc -mtriple=x86_64-linux-android < %s | FileCheck -check-prefix=CHECK-X86-64 %s 
 ; RUN: llc -mtriple=i686-linux-android < %s | FileCheck -check-prefix=CHECK-X86-32 %s 
 
 define i32 @foo() local_unnamed_addr #0 {
   %a = alloca i32, i64 2000, align 16
   %b = getelementptr inbounds i32, i32* %a, i64 200
   store volatile i32 1, i32* %b
   %c = load volatile i32, i32* %a
   ret i32 %c
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}
 
 ; CHECK-X86-64-LABEL: foo:
 ; CHECK-X86-64:      # %bb.0:
 ; CHECK-X86-64-NEXT: subq $4096, %rsp             # imm = 0x1000
+; CHECK-X86-64-NEXT: .cfi_adjust_cfa_offset 4096
 ; CHECK-X86-64-NEXT: movq $0, (%rsp)
 ; CHECK-X86-64-NEXT: subq $3784, %rsp             # imm = 0xEC8
 ; CHECK-X86-64-NEXT: .cfi_def_cfa_offset 7888
 ; CHECK-X86-64-NEXT: movl $1, 672(%rsp)
 ; CHECK-X86-64-NEXT: movl -128(%rsp), %eax
 ; CHECK-X86-64-NEXT: addq $7880, %rsp             # imm = 0x1EC8
 ; CHECK-X86-64-NEXT: .cfi_def_cfa_offset 8
 ; CHECK-X86-64-NEXT: retq
 
 
 ; CHECK-X86-32-LABEL: foo:
 ; CHECK-X86-32:      # %bb.0:
 ; CHECK-X86-32-NEXT: subl $4096, %esp # imm = 0x1000
+; CHECK-X86-32-NEXT: .cfi_adjust_cfa_offset 4096
 ; CHECK-X86-32-NEXT: movl $0, (%esp)
 ; CHECK-X86-32-NEXT: subl $3916, %esp # imm = 0xF4C
 ; CHECK-X86-32-NEXT: .cfi_def_cfa_offset 8016
 ; CHECK-X86-32-NEXT: movl $1, 800(%esp)
 ; CHECK-X86-32-NEXT: movl (%esp), %eax
 ; CHECK-X86-32-NEXT: addl $8012, %esp # imm = 0x1F4C
 ; CHECK-X86-32-NEXT: .cfi_def_cfa_offset 4
 ; CHECK-X86-32-NEXT: retl

llvm/test/CodeGen/X86/stack-clash-unknown-call.ll

 ; RUN: llc < %s | FileCheck %s
 
 
 target datalayout = "e-m:e-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-unknown-linux-gnu"
 
 declare void @llvm.memset.p0i8.i64(i8* nocapture writeonly, i8, i64, i1 immarg);
 
 define void @foo() local_unnamed_addr #0 {
 
 ;CHECK-LABEL: foo:
 ;CHECK:         # %bb.0:
 ;CHECK-NEXT:  subq  $4096, %rsp # imm = 0x1000
+;CHECK-NEXT:  .cfi_adjust_cfa_offset 4096
 ; it's important that we don't use the call as a probe here
 ;CHECK-NEXT:  movq  $0, (%rsp)
 ;CHECK-NEXT:  subq  $3912, %rsp # imm = 0xF48
 ;CHECK-NEXT:  .cfi_def_cfa_offset 8016
 ;CHECK-NEXT:  movq  %rsp, %rdi
 ;CHECK-NEXT:  movl  $8000, %edx # imm = 0x1F40
 ;CHECK-NEXT:  xorl  %esi, %esi
 ;CHECK-NEXT:  callq memset
 ;CHECK-NEXT:  addq  $8008, %rsp # imm = 0x1F48
 ;CHECK-NEXT:  .cfi_def_cfa_offset 8
 ;CHECK-NEXT:  retq
 
   %a = alloca i8, i64 8000, align 16
   call void @llvm.memset.p0i8.i64(i8* align 16 %a, i8 0, i64 8000, i1 false)
   ret void
 }
 
 attributes #0 =  {"probe-stack"="inline-asm"}