[X86] Create extra prolog/epilog for stack realignment

[LuoYuanke]

Summary

The base pointer register is reserved by compiler when there is
dynamic size alloca and stack realign in a function. However the
base pointer register is not defined in X86 ABI, so user can use
this register in inline assembly. The inline assembly would
clobber base pointer register without being awared by user. This
patch is to create extra prolog to save the stack pointer to a
scratch register and use this register to reference argument from
stack. For some calling convention (e.g. regcall), there may be
few scratch register.
Below is the example code for such case.

extern int bar(void *p);
long long foo(size_t size, char c, int id) {
  __attribute__((__aligned__(64))) int a;
  char *p = (char *)alloca(size);
  asm volatile ("nop"::"S"(405):);
  asm volatile ("movl %0, %1"::"r"(id), "m"(a):);
  p[2] = 8;
  memset(p, c, size);
  return bar(p);
}

And below prolog/epilog will be emit for this case.

leal    4(%esp), %ebx
.cfi_def_cfa %ebx, 0
andl    $-128, %esp
pushl   -4(%ebx)
...
leal    4(%ebx), %esp
.cfi_def_cfa %esp, 4

[LuoYuanke]

Add unwind attibute to test cfi instruction emit.

[LuoYuanke]

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

80

About 60 lit test fails and need update if we create a new register class for it.

[pengfei]

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

2

Update name

77

Are we replacing RBX with RBX on 64-bits?

80

Why R10? Do you want R11, according to ABI

%r10 temporary register, used for passing a function’s static chain pointer No
%r11 temporary register                                                     No

llvm/lib/Target/X86/X86FrameLowering.cpp

566–567

Remove

llvm/lib/Target/X86/X86RegisterInfo.td

436–439

Should remove it?

llvm/test/CodeGen/X86/swifttail-realign.ll

9

No sure if it's correct. I assume the old code used rsp to access %ptr but it is intentionally omitted.
It must be wrong if the new code still use rsp because it is not aligned to 32 now.

[LuoYuanke]

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

77

Good catch, we may have to need virtual register since inline assembly may clobber any physical register.

80

Both R10 and R11 should be OK, but we should use virtual register.

llvm/test/CodeGen/X86/swifttail-realign.ll

9

There is alignment instruction andq $-32, %rax in both the old code and new code.

movq    %rsp, %rax
addq    $15, %rdi
andq    $-16, %rdi
subq    %rdi, %rax
andq    $-32, %rax
movq    %rax, %rsp
movq    var@GOTPCREL(%rip), %rcx
movq    %rax, (%rcx)

[LuoYuanke]

Address some of Phoebe's comments. Need another patch
to create new register class.

[LuoYuanke]

Address Phoebe's comments.
Use virtual register for argument stack slot rebase.

[LuoYuanke]

Update test cases.

[LuoYuanke]

Update test cases.

[LuoYuanke]

1. Use scratch register for argument base pointer.
2. Fix CFI instruction
3. Enhance replacement of frame pointer with argument base pointer.
4. c++ test case (https://reviews.llvm.org/D145926) can pass with -g.

[LuoYuanke]

llvm/test/CodeGen/MIR/X86/inline-asm-registers.mir

31 ↗

(On Diff #504599)

GR64 register class ID is changed from 67 to 68. Since register class is encoded in the flag, this test case need to be updated.

[LuoYuanke]

The patch is ready for review.

[LuoYuanke]

llvm/test/CodeGen/X86/i386-baseptr.ll

52

The argument base register is saved to a normal stack slot.

71

The argument base register is restored from a normal stack

[LuoYuanke]

llvm/test/CodeGen/X86/i386-baseptr.ll

51–53

Stack is aligned with 128 byte. (244 + 4 (ret addr) + 4 (ebp) +4 (esi) = 256)

[RKSimon]

llvm/lib/Target/X86/CMakeLists.txt

92

(style) keep this sorted if you can

[LuoYuanke]

Address Simon's comments.

[pengfei]

We had an offline discussion on this. I generally got the idea of the design. Let's land it first to see if there're unconsidered cases.

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

98

Should assert rather than break?

146

Should assert rather than return?

llvm/lib/Target/X86/X86FrameLowering.cpp

4014

This can be auto.

4018

Use explicit type.

llvm/lib/Target/X86/X86RegisterInfo.cpp

819–820

The format looks odd, maybe clang-format?

llvm/lib/Target/X86/X86RegisterInfo.td

437

last .

443

ditto.

llvm/test/CodeGen/X86/i386-baseptr.ll

51–53

Nit, it seems a bit waste in space.

llvm/test/CodeGen/X86/swifttail-realign.ll

9

That looks good, thanks!

[LuoYuanke]

Address Phoebe's comments.

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

98

If we use break it can return NoReg, so that nothing would be done in this pass and the extra prolog/epilog won't be inserted.

146

My original idea is to leave it as it is if we can't handle it.

llvm/test/CodeGen/X86/i386-baseptr.ll

51–53

Yes, but it is original code logic that is to align the stack address to 128.

[nikic]

It looks like this broke the expensive checks build (x86-64-baseptr.ll and i386-baseptr.ll fail).

[LuoYuanke]

In D145650#4209068, @nikic wrote:

It looks like this broke the expensive checks build (x86-64-baseptr.ll and i386-baseptr.ll fail).

I'll check it with "-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON". Thanks for notifying.

[LuoYuanke]

There are live range issue for the physical register. Let me revert the patch first.

[rupprecht]

This relanded in e4ceb5a7bb9b8f6d730530345649286370dd3ff8, and we're seeing runtime errors in Halide when building with asan.

I don't have a standalone reproducer I can share yet, but the issue we see is when attempting to return from this method: https://github.com/halide/Halide/blob/main/src/runtime/x86_cpu_features.cpp. If I had to guess, it's a bad interaction with the cpuid implementation here: https://github.com/halide/Halide/blob/2a51f71a0add536ab986c675d1239db860dbc655/src/runtime/x86.ll#L145

The problem is $rsp is not pointing to the right place, and so ret jumps to a bogus location, so the next instruction faults:

... break on start address of x86_cpu_features ...
(lldb) x/2 $rsp
0x7fffffffb818: 0x5806dc35 0x00005555
(lldb) c
...
(lldb) di -l

repro_test`Halide::Runtime::Internal::halide_get_cpu_features:
    0x55555806e23e <+1246>: movq   -0x48(%rbp), %r10
    0x55555806e242 <+1250>: leaq   -0x28(%rbp), %rsp
    0x55555806e246 <+1254>: popq   %rbx
    0x55555806e247 <+1255>: popq   %r12
    0x55555806e249 <+1257>: popq   %r13
    0x55555806e24b <+1259>: popq   %r14
    0x55555806e24d <+1261>: popq   %r15
    0x55555806e24f <+1263>: popq   %rbp
    0x55555806e250 <+1264>: leaq   -0x10(%r10), %rsp
->  0x55555806e254 <+1268>: retq
(lldb) x/2 $rsp
0x7fffffffb810: 0xffffb890 0x00007fff
(lldb) x/2 0x7fffffffb818
0x7fffffffb818: 0x5806dc35 0x00005555

[LuoYuanke]

repro_test`Halide::Runtime::Internal::halide_get_cpu_features:

0x55555806e23e <+1246>: movq   -0x48(%rbp), %r10
0x55555806e242 <+1250>: leaq   -0x28(%rbp), %rsp
0x55555806e246 <+1254>: popq   %rbx
0x55555806e247 <+1255>: popq   %r12
0x55555806e249 <+1257>: popq   %r13
0x55555806e24b <+1259>: popq   %r14
0x55555806e24d <+1261>: popq   %r15
0x55555806e24f <+1263>: popq   %rbp
0x55555806e250 <+1264>: leaq   -0x10(%r10), %rsp

-> 0x55555806e254 <+1268>: retq
(lldb) x/2 $rsp
0x7fffffffb810: 0xffffb890 0x00007fff
(lldb) x/2 0x7fffffffb818
0x7fffffffb818: 0x5806dc35 0x00005555

Thank you very much for the indicator. It seems the offset is incorrect for leaq -0x10(%r10), %rsp. It should be leaq -0x8(%r10), %rsp. Sorry for inconvenience. I'll fix it as soon as possible and I'll develop more runtime test case for it.

[LuoYuanke]

@rupprecht, could you help to check if https://reviews.llvm.org/D146862 can fix your issue?

llvm/lib/Target/X86/CMakeLists.txt

set(sources
   X86SpeculativeExecutionSideEffectSuppression.cpp
   X86Subtarget.cpp
   X86TargetMachine.cpp
   X86TargetObjectFile.cpp
   X86TargetTransformInfo.cpp
   X86VZeroUpper.cpp
   X86WinEHState.cpp
   X86InsertWait.cpp
+  X86ArgumentStackSlotRebase.cpp
   )
 
 add_llvm_target(X86CodeGen ${sources}
   LINK_COMPONENTS
   Analysis
   AsmPrinter
   CodeGen
   Core

llvm/lib/Target/X86/X86.h

 InstructionSelector *createX86InstructionSelector(const X86TargetMachine &TM,
                                                   X86Subtarget &,
                                                   X86RegisterBankInfo &);
 
 FunctionPass *createX86LoadValueInjectionLoadHardeningPass();
 FunctionPass *createX86LoadValueInjectionRetHardeningPass();
 FunctionPass *createX86SpeculativeLoadHardeningPass();
 FunctionPass *createX86SpeculativeExecutionSideEffectSuppression();
+FunctionPass *createX86ArgumentStackSlotPass();
 
 void initializeEvexToVexInstPassPass(PassRegistry &);
 void initializeFPSPass(PassRegistry &);
 void initializeFixupBWInstPassPass(PassRegistry &);
 void initializeFixupLEAPassPass(PassRegistry &);
 void initializeX86FixupInstTuningPassPass(PassRegistry &);
 void initializeWinEHStatePassPass(PassRegistry &);
 void initializeX86AvoidSFBPassPass(PassRegistry &);

 void initializeX86OptimizeLEAPassPass(PassRegistry &);
 void initializeX86PartialReductionPass(PassRegistry &);
 void initializeX86PreAMXConfigPassPass(PassRegistry &);
 void initializeX86PreTileConfigPass(PassRegistry &);
 void initializeX86ReturnThunksPass(PassRegistry &);
 void initializeX86SpeculativeExecutionSideEffectSuppressionPass(PassRegistry &);
 void initializeX86SpeculativeLoadHardeningPassPass(PassRegistry &);
 void initializeX86TileConfigPass(PassRegistry &);
+void initializeX86ArgumentStackSlotPassPass(PassRegistry &);
 
 namespace X86AS {
 enum : unsigned {
   GS = 256,
   FS = 257,
   SS = 258,
   PTR32_SPTR = 270,
   PTR32_UPTR = 271,
   PTR64 = 272
 };
 } // End X86AS namespace
 
 } // End llvm namespace
 
 #endif

llvm/lib/Target/X86/X86ArgumentStackSlotRebase.cpp

+//===---- X86ArgumentStackSlotRebase.cpp - rebase argument stack slot -----===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This pass replace the frame register with a GPR virtual register and set
+// the stack offset for each instruction which reference argument from stack.
+//
+//===----------------------------------------------------------------------===//
+
+#include "X86.h"
+#include "X86MachineFunctionInfo.h"
+#include "X86RegisterInfo.h"
+#include "X86Subtarget.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/CodeGen/MachineFrameInfo.h"
+#include "llvm/CodeGen/MachineFunction.h"
+#include "llvm/CodeGen/MachineFunctionPass.h"
+#include "llvm/CodeGen/MachineInstr.h"
+#include "llvm/CodeGen/MachineOperand.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
+#include "llvm/CodeGen/TargetOpcodes.h"
+#include "llvm/CodeGen/TargetRegisterInfo.h"
+#include "llvm/CodeGen/TargetSubtargetInfo.h"
+#include "llvm/IR/Attributes.h"
+#include "llvm/IR/Function.h"
+#include "llvm/InitializePasses.h"
+#include "llvm/Pass.h"
+
+using namespace llvm;
+
+#define DEBUG_TYPE "x86argumentstackrebase"
+
+namespace {
+
+class X86ArgumentStackSlotPass : public MachineFunctionPass {
+
+public:
+  static char ID; // Pass identification, replacement for typeid
+
+  explicit X86ArgumentStackSlotPass() : MachineFunctionPass(ID) {
+    initializeX86ArgumentStackSlotPassPass(*PassRegistry::getPassRegistry());
+  }
+
+  bool runOnMachineFunction(MachineFunction &MF) override;
+
+  void getAnalysisUsage(AnalysisUsage &AU) const override {
+    AU.setPreservesCFG();
+    MachineFunctionPass::getAnalysisUsage(AU);
+  }
+};
+
+} // end anonymous namespace
+
+char X86ArgumentStackSlotPass::ID = 0;
+
+INITIALIZE_PASS(X86ArgumentStackSlotPass, DEBUG_TYPE, "Argument Stack Rebase",
+                false, false)
+
+FunctionPass *llvm::createX86ArgumentStackSlotPass() {
+  return new X86ArgumentStackSlotPass();
+}
+
+static Register getArgBaseReg(MachineFunction &MF) {
+  MachineRegisterInfo &MRI = MF.getRegInfo();
+  const X86Subtarget &STI = MF.getSubtarget<X86Subtarget>();
+  const Function &F = MF.getFunction();
+  CallingConv::ID CC = F.getCallingConv();
+  switch (CC) {
+  // Ww need a virtual register in case there is inline assembly
+  // clobber ebx/rbx.
+  case CallingConv::X86_RegCall: {
+    const TargetRegisterClass *RC = STI.is64Bit() ? &X86::GR64_RC_NoArgRegClass
+                                                  : &X86::GR32_RC_NoArgRegClass;
+    return MRI.createVirtualRegister(RC);
+  }
+  // TODO: Refine register class for each calling convention.
+  default: {
+    const TargetRegisterClass *RC =
+        STI.is64Bit() ? &X86::GR64_NoArgRegClass : &X86::GR32_BSIRegClass;
+    return MRI.createVirtualRegister(RC);
+  }
+  }
+}
+
+bool X86ArgumentStackSlotPass::runOnMachineFunction(MachineFunction &MF) {
+  const Function &F = MF.getFunction();
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  const X86Subtarget &STI = MF.getSubtarget<X86Subtarget>();
+  const X86RegisterInfo *TRI = STI.getRegisterInfo();
+  const X86InstrInfo *TII = STI.getInstrInfo();
+  X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
+  bool Changed = false;
+
+  if (F.hasFnAttribute(Attribute::Naked))
+    return false;
+  // Only support Linux
+  if (!STI.isTargetLinux())
+    return false;
+  if (!TRI->hasBasePointer(MF))
+    return false;
+
+  Register BasePtr = TRI->getBaseRegister();
+  auto IsBaseRegisterClobbered = [&]() {
+    for (MachineBasicBlock &MBB : MF) {
+      for (MachineInstr &MI : MBB) {
+        if (!MI.isInlineAsm())
+          continue;
+        for (MachineOperand &MO : MI.operands()) {
+          if (!MO.isReg())
+            continue;
+          Register Reg = MO.getReg();
+          if (!Register::isPhysicalRegister(Reg))
+            continue;
+          if (TRI->isSuperOrSubRegisterEq(BasePtr, Reg))
+            return true;
+        }
+      }
+    }
+    return false;
+  };
+  if (!IsBaseRegisterClobbered())
+    return false;
+
+  Register ArgBaseReg = getArgBaseReg(MF);
+  // leal    4(%esp), %reg
+  // FIXME: will the instruction be duplicated or eliminated? Should
+  // define a pseudo instruction for it?
+  MachineBasicBlock &MBB = MF.front();
+  MachineBasicBlock::iterator MBBI = MBB.begin();
+  DebugLoc DL;
+  // Emit instruction to copy get stack pointer to a virtual register
+  // and save the instruction to x86 machine functon info. We can get
+  // physical register of ArgBaseReg after register allocation.
+  MachineInstr *LEA;
+  if (STI.is64Bit())
+    LEA = BuildMI(MBB, MBBI, DL, TII->get(X86::LEA64r), ArgBaseReg)
+              .addUse(X86::RSP)
+              .addImm(1)
+              .addUse(X86::NoRegister)
+              .addImm(8)
+              .addUse(X86::NoRegister)
+              .setMIFlag(MachineInstr::FrameSetup);
+  else
+    LEA = BuildMI(MBB, MBBI, DL, TII->get(X86::LEA32r), ArgBaseReg)
+              .addUse(X86::ESP)
+              .addImm(1)
+              .addUse(X86::NoRegister)
+              .addImm(4)
+              .addUse(X86::NoRegister)
+              .setMIFlag(MachineInstr::FrameSetup);
+  X86FI->setStackPtrSaveMI(LEA);
+
+  for (MachineBasicBlock &MBB : MF) {
+    for (MachineInstr &MI : MBB) {
+      int I = 0;
+      for (MachineOperand &MO : MI.operands()) {
+        if (MO.isFI()) {
+          int Idx = MO.getIndex();
+          if (!MFI.isFixedObjectIndex(Idx))
+            continue;
+          int64_t Offset = MFI.getObjectOffset(Idx);
+          if (Offset < 0)
+            continue;
+          // Replace frame register with general virtual register.
+          MO.ChangeToRegister(ArgBaseReg, false);
+          // Fill the offset of the stack slot.
+          MI.getOperand(I + 3).setImm(Offset);
+          Changed = true;
+        }
+        ++I;
+      }
+    }
+  }
+
+  return Changed;
+}

llvm/lib/Target/X86/X86FrameLowering.cpp

 #include "llvm/CodeGen/WinEHFuncInfo.h"
 #include "llvm/IR/DataLayout.h"
 #include "llvm/IR/EHPersonalities.h"
 #include "llvm/IR/Function.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCObjectFileInfo.h"
 #include "llvm/MC/MCSymbol.h"
 #include "llvm/Support/Debug.h"
+#include "llvm/Support/LEB128.h"
 #include "llvm/Target/TargetOptions.h"
 #include <cstdlib>
 
 #define DEBUG_TYPE "x86-fl"
 
 STATISTIC(NumFrameLoopProbe, "Number of loop stack probes used in prologue");
 STATISTIC(NumFrameExtraProbe,
           "Number of extra stack probes generated in prologue");

 
 void X86FrameLowering::emitCalleeSavedFrameMoves(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI,
     const DebugLoc &DL, bool IsPrologue) const {
   MachineFunction &MF = *MBB.getParent();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   MachineModuleInfo &MMI = MF.getMMI();
   const MCRegisterInfo *MRI = MMI.getContext().getRegisterInfo();
+  X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
 
   // Add callee saved registers to move list.
   const std::vector<CalleeSavedInfo> &CSI = MFI.getCalleeSavedInfo();
 
   // Calculate offsets.
   for (const CalleeSavedInfo &I : CSI) {
     int64_t Offset = MFI.getObjectOffset(I.getFrameIdx());
     Register Reg = I.getReg();
     unsigned DwarfReg = MRI->getDwarfRegNum(Reg, true);
 
     if (IsPrologue) {
+      if (X86FI->getStackPtrSaveMI()) {
+        // +2*SlotSize because there is return address and ebp at the bottom
+        // of the stack.
+        // | retaddr |
+        // | ebp     |
+        // |         |<--ebp
+        Offset += 2 * SlotSize;
+        SmallString<64> CfaExpr;
+        CfaExpr.push_back(dwarf::DW_CFA_expression);
+        uint8_t buffer[16];
+        CfaExpr.append(buffer, buffer + encodeULEB128(DwarfReg, buffer));
+        CfaExpr.push_back(2);
+        Register FramePtr = TRI->getFrameRegister(MF);
+        const Register MachineFramePtr =
+            STI.isTarget64BitILP32()
+                ? Register(getX86SubSuperRegister(FramePtr, 64))
+                : FramePtr;
+        unsigned DwarfFramePtr = MRI->getDwarfRegNum(MachineFramePtr, true);
+        CfaExpr.push_back((uint8_t)(dwarf::DW_OP_breg0 + DwarfFramePtr));
+        CfaExpr.append(buffer, buffer + encodeSLEB128(Offset, buffer));
+        BuildCFI(MBB, MBBI, DL,
+                 MCCFIInstruction::createEscape(nullptr, CfaExpr.str()),
+                 MachineInstr::FrameSetup);
+      } else {
         BuildCFI(MBB, MBBI, DL,
                  MCCFIInstruction::createOffset(nullptr, DwarfReg, Offset));
+      }
     } else {
       BuildCFI(MBB, MBBI, DL,
                MCCFIInstruction::createRestore(nullptr, DwarfReg));
     }
   }
 }
 
 void X86FrameLowering::emitZeroCallUsedRegs(BitVector RegsToZero,
if (TRI->isGeneralPurposeRegister(MF, Reg)) {
       GPRsToZero.set(getX86SubSuperRegister(Reg, 32));
       RegsToZero.reset(Reg);
     }
 
   for (MCRegister Reg : GPRsToZero.set_bits())
     BuildMI(MBB, MBBI, DL, TII.get(X86::XOR32rr), Reg)
         .addReg(Reg, RegState::Undef)
         .addReg(Reg, RegState::Undef);
 
   // Zero out registers.
   for (MCRegister Reg : RegsToZero.set_bits()) {
     if (ST.hasMMX() && X86::VR64RegClass.contains(Reg))
       // FIXME: Ignore MMX registers?
       continue;
 
     unsigned XorOp;
     if (X86::VR128RegClass.contains(Reg)) {
       // XMM#
const Register MachineFramePtr =
       STI.isTarget64BitILP32()
           ? Register(getX86SubSuperRegister(FramePtr, 64)) : FramePtr;
   Register BasePtr = TRI->getBaseRegister();
   bool HasWinCFI = false;
 
   // Debug location must be unknown since the first debug location is used
   // to determine the end of the prologue.
   DebugLoc DL;
+  Register ArgBaseReg;
+
+  // Emit extra prolog for argument stack slot reference.
+  if (auto *MI = X86FI->getStackPtrSaveMI()) {
+    // MI is lea instruction that created in X86ArgumentStackSlotPass.
+    // Crreat extra prolog for stack realignment.
+    // leal    4(%esp), %basereg
+    // .cfi_def_cfa %basereg, 0
+    // andl    $-128, %esp
+    // pushl   -4(%basereg)
+    ArgBaseReg = MI->getOperand(0).getReg();
+    if (Is64Bit) {
+      // leaq    8(%rsp), %reg
+      BuildMI(MBB, MBBI, DL, TII.get(X86::LEA64r), ArgBaseReg)
+          .addUse(X86::RSP)
+          .addImm(1)
+          .addUse(X86::NoRegister)
+          .addImm(8)
+          .addUse(X86::NoRegister)
+          .setMIFlag(MachineInstr::FrameSetup);
+    } else {
+      // leal    4(%esp), %reg
+      BuildMI(MBB, MBBI, DL, TII.get(X86::LEA32r), ArgBaseReg)
+          .addUse(X86::ESP)
+          .addImm(1)
+          .addUse(X86::NoRegister)
+          .addImm(4)
+          .addUse(X86::NoRegister)
+          .setMIFlag(MachineInstr::FrameSetup);
+    }
+    if (NeedsDwarfCFI) {
+      // .cfi_def_cfa %basereg, 0
+      unsigned DwarfStackPtr = TRI->getDwarfRegNum(ArgBaseReg, true);
+      BuildCFI(MBB, MBBI, DL,
+               MCCFIInstruction::cfiDefCfa(nullptr, DwarfStackPtr, 0),
+               MachineInstr::FrameSetup);
+    }
+    BuildStackAlignAND(MBB, MBBI, DL, StackPtr, MaxAlign);
+    int64_t Offset = Is64Bit ? -8 : -4;
+    BuildMI(MBB, MBBI, DL, TII.get(Is64Bit ? X86::PUSH64rmm: X86::PUSH32rmm))
+        .addReg(ArgBaseReg)
+        .addImm(1)
+        .addReg(X86::NoRegister)
+        .addImm(Offset)
+        .addReg(X86::NoRegister)
+        .setMIFlag(MachineInstr::FrameSetup);
+  }
 
   // Space reserved for stack-based arguments when making a (ABI-guaranteed)
   // tail call.
   unsigned TailCallArgReserveSize = -X86FI->getTCReturnAddrDelta();
   if (TailCallArgReserveSize  && IsWin64Prologue)
     report_fatal_error("Can't handle guaranteed tail call under win64 yet");
 
   const bool EmitStackProbeCall =
if (HasFP) {
     if (TRI->hasStackRealignment(MF) && !IsWin64Prologue)
       NumBytes = alignTo(NumBytes, MaxAlign);
 
     // Save EBP/RBP into the appropriate stack slot.
     BuildMI(MBB, MBBI, DL, TII.get(Is64Bit ? X86::PUSH64r : X86::PUSH32r))
       .addReg(MachineFramePtr, RegState::Kill)
       .setMIFlag(MachineInstr::FrameSetup);
 
-    if (NeedsDwarfCFI) {
+    if (NeedsDwarfCFI && !ArgBaseReg.isValid()) {
       // Mark the place where EBP/RBP was saved.
       // Define the current CFA rule to use the provided offset.
       assert(StackSize);
       BuildCFI(MBB, MBBI, DL,
                MCCFIInstruction::cfiDefCfaOffset(
                    nullptr, -2 * stackGrowth + (int)TailCallArgReserveSize),
                MachineInstr::FrameSetup);
 
if (!IsFunclet) {
         if (!X86FI->hasSwiftAsyncContext())
           BuildMI(MBB, MBBI, DL,
                   TII.get(Uses64BitFramePtr ? X86::MOV64rr : X86::MOV32rr),
                   FramePtr)
               .addReg(StackPtr)
               .setMIFlag(MachineInstr::FrameSetup);
 
         if (NeedsDwarfCFI) {
+          if (ArgBaseReg.isValid()) {
+            SmallString<64> CfaExpr;
+            CfaExpr.push_back(dwarf::DW_CFA_expression);
+            uint8_t buffer[16];
+            unsigned DwarfReg = TRI->getDwarfRegNum(MachineFramePtr, true);
+            CfaExpr.append(buffer, buffer + encodeULEB128(DwarfReg, buffer));
+            CfaExpr.push_back(2);
+            CfaExpr.push_back((uint8_t)(dwarf::DW_OP_breg0 + DwarfReg));
+            CfaExpr.push_back(0);
+            BuildCFI(MBB, MBBI, DL,
+                     MCCFIInstruction::createEscape(nullptr, CfaExpr.str()),
+                     MachineInstr::FrameSetup);
+          } else {
             // Mark effective beginning of when frame pointer becomes valid.
             // Define the current CFA to use the EBP/RBP register.
             unsigned DwarfFramePtr = TRI->getDwarfRegNum(MachineFramePtr, true);
             BuildCFI(
                 MBB, MBBI, DL,
                 MCCFIInstruction::createDefCfaRegister(nullptr, DwarfFramePtr),
                 MachineInstr::FrameSetup);
           }
+        }
 
         if (NeedsWinFPO) {
           // .cv_fpo_setframe $FramePtr
           HasWinCFI = true;
           BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_SetFrame))
               .addImm(FramePtr)
               .addImm(0)
               .setMIFlag(MachineInstr::FrameSetup);
if (NeedsWinCFI) {
           .addImm(Reg)
           .setMIFlag(MachineInstr::FrameSetup);
     }
   }
 
   // Realign stack after we pushed callee-saved registers (so that we'll be
   // able to calculate their offsets from the frame pointer).
   // Don't do this for Win64, it needs to realign the stack after the prologue.
-  if (!IsWin64Prologue && !IsFunclet && TRI->hasStackRealignment(MF)) {
+  if (!IsWin64Prologue && !IsFunclet && TRI->hasStackRealignment(MF) &&
+      !ArgBaseReg.isValid()) {
     assert(HasFP && "There should be a frame pointer if stack is realigned.");
     BuildStackAlignAND(MBB, MBBI, DL, StackPtr, MaxAlign);
 
     if (NeedsWinCFI) {
       HasWinCFI = true;
       BuildMI(MBB, MBBI, DL, TII.get(X86::SEH_StackAlign))
           .addImm(MaxAlign)
           .setMIFlag(MachineInstr::FrameSetup);
void X86FrameLowering::emitEpilogue(MachineFunction &MF,
   unsigned TailCallArgReserveSize = -X86FI->getTCReturnAddrDelta();
   bool HasFP = hasFP(MF);
   uint64_t NumBytes = 0;
 
   bool NeedsDwarfCFI = (!MF.getTarget().getTargetTriple().isOSDarwin() &&
                         !MF.getTarget().getTargetTriple().isOSWindows()) &&
                        MF.needsFrameMoves();
 
+  Register ArgBaseReg;
+  if (auto *MI = X86FI->getStackPtrSaveMI()) {
+    unsigned Opc = X86::LEA32r;
+    Register StackReg = X86::ESP;
+    ArgBaseReg = MI->getOperand(0).getReg();
+    if (STI.is64Bit()) {
+      Opc = X86::LEA64r;
+      StackReg = X86::RSP;
+    }
+    // leal    -4(%basereg), %esp
+    // .cfi_def_cfa %esp, 4
+    BuildMI(MBB, MBBI, DL, TII.get(Opc), StackReg)
+        .addUse(ArgBaseReg)
+        .addImm(1)
+        .addUse(X86::NoRegister)
+        .addImm(SlotSize)
+        .addUse(X86::NoRegister)
+        .setMIFlag(MachineInstr::FrameDestroy);
+    if (NeedsDwarfCFI) {
+      unsigned DwarfStackPtr = TRI->getDwarfRegNum(StackReg, true);
+      BuildCFI(MBB, MBBI, DL,
+               MCCFIInstruction::cfiDefCfa(nullptr, DwarfStackPtr, SlotSize),
+               MachineInstr::FrameDestroy);
+      --MBBI;
+    }
+    --MBBI;
+    // Don't need stack pointer saving instruction any more.
+    MI->eraseFromParent();
+    X86FI->setStackPtrSaveMI(nullptr);
+  }
+
   if (IsFunclet) {
     assert(HasFP && "EH funclets without FP not yet implemented");
     NumBytes = getWinEHFuncletFrameSize(MF);
   } else if (HasFP) {
     // Calculate required stack adjustment.
     uint64_t FrameSize = StackSize - SlotSize;
     NumBytes = FrameSize - CSSize - TailCallArgReserveSize;
 
if (X86FI->hasSwiftAsyncContext()) {
       BuildMI(MBB, MBBI, DL, TII.get(X86::BTR64ri8),
               MachineFramePtr)
           .addUse(MachineFramePtr)
           .addImm(60)
           .setMIFlag(MachineInstr::FrameDestroy);
     }
 
     if (NeedsDwarfCFI) {
+      if (!ArgBaseReg.isValid()) {
         unsigned DwarfStackPtr =
             TRI->getDwarfRegNum(Is64Bit ? X86::RSP : X86::ESP, true);
         BuildCFI(MBB, MBBI, DL,
                  MCCFIInstruction::cfiDefCfa(nullptr, DwarfStackPtr, SlotSize),
                  MachineInstr::FrameDestroy);
+      }
       if (!MBB.succ_empty() && !MBB.isReturnBlock()) {
         unsigned DwarfFramePtr = TRI->getDwarfRegNum(MachineFramePtr, true);
         BuildCFI(MBB, AfterPop, DL,
                  MCCFIInstruction::createRestore(nullptr, DwarfFramePtr),
                  MachineInstr::FrameDestroy);
         --MBBI;
         --AfterPop;
       }
void X86FrameLowering::adjustFrameForMsvcCxxEh(MachineFunction &MF) const {
   DebugLoc DL = MBB.findDebugLoc(MBBI);
   addFrameReference(BuildMI(MBB, MBBI, DL, TII.get(X86::MOV64mi32)),
                     UnwindHelpFI)
       .addImm(-2);
 }
 
 void X86FrameLowering::processFunctionBeforeFrameIndicesReplaced(
     MachineFunction &MF, RegScavenger *RS) const {
   if (STI.is32Bit() && MF.hasEHFunclets())
     restoreWinEHStackPointersInParent(MF);
 }
 
 void X86FrameLowering::restoreWinEHStackPointersInParent(
     MachineFunction &MF) const {
   // 32-bit functions have to restore stack pointers when control is transferred
   // back to the parent function. These blocks are identified as eh pads that
   // are not funclet entries.
   bool IsSEH = isAsynchronousEHPersonality(
       classifyEHPersonality(MF.getFunction().getPersonalityFn()));
   for (MachineBasicBlock &MBB : MF) {
     bool NeedsRestore = MBB.isEHPad() && !MBB.isEHFuncletEntry();
     if (NeedsRestore)
       restoreWin32EHStackPointers(MBB, MBB.begin(), DebugLoc(),
                                   /*RestoreSP=*/IsSEH);
   }
 }

llvm/lib/Target/X86/X86MachineFunctionInfo.h

class X86MachineFunctionInfo : public MachineFunctionInfo {
   /// addr]. If so, bit 60 of the in-memory frame pointer will be 1 to enable
   /// other tools to detect the extended record.
   bool HasSwiftAsyncContext = false;
 
   /// True if this function has tile virtual register. This is used to
   /// determine if we should insert tilerelease in frame lowering.
   bool HasVirtualTileReg = false;
 
+  MachineInstr *StackPtrSaveMI = nullptr;
+
   std::optional<int> SwiftAsyncContextFrameIdx;
 
   // Preallocated fields are only used during isel.
   // FIXME: Can we find somewhere else to store these?
   DenseMap<const Value *, size_t> PreallocatedIds;
   SmallVector<size_t, 0> PreallocatedStackSizes;
   SmallVector<SmallVector<size_t, 4>, 0> PreallocatedArgOffsets;
 
public:
   void setHasPreallocatedCall(bool v) { HasPreallocatedCall = v; }
 
   bool hasSwiftAsyncContext() const { return HasSwiftAsyncContext; }
   void setHasSwiftAsyncContext(bool v) { HasSwiftAsyncContext = v; }
 
   bool hasVirtualTileReg() const { return HasVirtualTileReg; }
   void setHasVirtualTileReg(bool v) { HasVirtualTileReg = v; }
 
+  void setStackPtrSaveMI(MachineInstr *MI) { StackPtrSaveMI = MI; }
+  MachineInstr *getStackPtrSaveMI() const { return StackPtrSaveMI; }
+
   std::optional<int> getSwiftAsyncContextFrameIdx() const {
     return SwiftAsyncContextFrameIdx;
   }
   void setSwiftAsyncContextFrameIdx(int v) { SwiftAsyncContextFrameIdx = v; }
 
   size_t getPreallocatedIdForCallSite(const Value *CS) {
     auto Insert = PreallocatedIds.insert({CS, PreallocatedIds.size()});
     if (Insert.second) {

llvm/lib/Target/X86/X86RegisterInfo.cpp

 //===----------------------------------------------------------------------===//
 
 static bool CantUseSP(const MachineFrameInfo &MFI) {
   return MFI.hasVarSizedObjects() || MFI.hasOpaqueSPAdjustment();
 }
 
 bool X86RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   const X86MachineFunctionInfo *X86FI = MF.getInfo<X86MachineFunctionInfo>();
+  // We have a virtual register to reference argument, and don't need base
+  // pointer.
+  if (X86FI->getStackPtrSaveMI() != nullptr)
+    return false;
+
   if (X86FI->hasPreallocatedCall())
     return true;
 
   const MachineFrameInfo &MFI = MF.getFrameInfo();
 
   if (!EnableBasePointer)
     return false;
 
if (MI.isReturn()) {
     FIOffset = TFI->getFrameIndexReference(MF, FrameIndex, BasePtr).getFixed();
   }
 
   // LOCAL_ESCAPE uses a single offset, with no register. It only works in the
   // simple FP case, and doesn't work with stack realignment. On 32-bit, the
   // offset is from the traditional base pointer location.  On 64-bit, the
   // offset is from the SP at the end of the prologue, not the FP location. This
   // matches the behavior of llvm.frameaddress.
   unsigned Opc = MI.getOpcode();
   if (Opc == TargetOpcode::LOCAL_ESCAPE) {
     MachineOperand &FI = MI.getOperand(FIOperandNum);
     FI.ChangeToImmediate(FIOffset);
     return false;
   }
 
   // For LEA64_32r when BasePtr is 32-bits (X32) we can use full-size 64-bit
   // register as source operand, semantic is the same and destination is
   // 32-bits. It saves one byte per lea in code since 0x67 prefix is avoided.

llvm/lib/Target/X86/X86RegisterInfo.td

def GR64 : RegisterClass<"X86", [i64], 64,
                               RBX, R14, R15, R12, R13, RBP, RSP, RIP)>;
 
 // GR64PLTSafe - 64-bit GPRs without R10, R11, RSP and RIP. Could be used when
 // emitting code for intrinsics, which use implict input registers.
 def GR64PLTSafe : RegisterClass<"X86", [i64], 64,
                               (add RAX, RCX, RDX, RSI, RDI, R8, R9,
                                    RBX, R14, R15, R12, R13, RBP)>;
 
+// It includes the intersection of win64 and linux64 registers that are
+// not used to pass/return argument. R10 and R11 are scratch register.
+def GR64_NoArg: RegisterClass<"X86", [i64], 64, (add R10, R11, RBX,
+                                                     R12, R13, R14, R15)> {
+  let GeneratePressureSet = 0;
+}
+// It includes the linux64 registers that are not used to pass/return argument
+// in regcall.
+def GR64_RC_NoArg: RegisterClass<"X86", [i64], 64, (add RBX, R10)> {
+  let GeneratePressureSet = 0;
+}
+// It includes the linux32 registers that are not used to pass/return argument
+// in regcall.
+def GR32_RC_NoArg: RegisterClass<"X86", [i32], 32, (add EBX)> {
+  let GeneratePressureSet = 0;
+}
+
 // Segment registers for use by MOV instructions (and others) that have a
 //   segment register as one operand.  Always contain a 16-bit segment
 //   descriptor.
 def SEGMENT_REG : RegisterClass<"X86", [i16], 16, (add CS, DS, SS, ES, FS, GS)>;
 
 // Debug registers.
 def DEBUG_REG : RegisterClass<"X86", [i32], 32, (sequence "DR%u", 0, 15)>;
 

llvm/lib/Target/X86/X86TargetMachine.cpp

extern "C" LLVM_EXTERNAL_VISIBILITY void LLVMInitializeX86Target() {
   initializeX86FlagsCopyLoweringPassPass(PR);
   initializeX86LoadValueInjectionLoadHardeningPassPass(PR);
   initializeX86LoadValueInjectionRetHardeningPassPass(PR);
   initializeX86OptimizeLEAPassPass(PR);
   initializeX86PartialReductionPass(PR);
   initializePseudoProbeInserterPass(PR);
   initializeX86ReturnThunksPass(PR);
   initializeX86DAGToDAGISelPass(PR);
+  initializeX86ArgumentStackSlotPassPass(PR);
 }
 
 static std::unique_ptr<TargetLoweringObjectFile> createTLOF(const Triple &TT) {
   if (TT.isOSBinFormatMachO()) {
     if (TT.getArch() == Triple::x86_64)
       return std::make_unique<X86_64MachoTargetObjectFile>();
     return std::make_unique<TargetLoweringObjectFileMachO>();
   }
bool X86PassConfig::addPreISel() {
   // Only add this pass for 32-bit x86 Windows.
   const Triple &TT = TM->getTargetTriple();
   if (TT.isOSWindows() && TT.getArch() == Triple::x86)
     addPass(createX86WinEHStatePass());
   return true;
 }
 
 void X86PassConfig::addPreRegAlloc() {
+  addPass(createX86ArgumentStackSlotPass());
   if (getOptLevel() != CodeGenOpt::None) {
     addPass(&LiveRangeShrinkID);
     addPass(createX86FixupSetCC());
     addPass(createX86OptimizeLEAs());
     addPass(createX86CallFrameOptimization());
     addPass(createX86AvoidStoreForwardingBlocks());
   }
 

llvm/test/CodeGen/X86/O0-pipeline.ll

 ; CHECK-NEXT:       Safe Stack instrumentation pass
 ; CHECK-NEXT:       Insert stack protectors
 ; CHECK-NEXT:       Module Verifier
 ; CHECK-NEXT:       Assignment Tracking Analysis
 ; CHECK-NEXT:       X86 DAG->DAG Instruction Selection
 ; CHECK-NEXT:       X86 PIC Global Base Reg Initialization
 ; CHECK-NEXT:       Finalize ISel and expand pseudo-instructions
 ; CHECK-NEXT:       Local Stack Slot Allocation
+; CHECK-NEXT:       Argument Stack Rebase
 ; CHECK-NEXT:       X86 speculative load hardening
 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       X86 EFLAGS copy lowering
 ; CHECK-NEXT:       X86 DynAlloca Expander
 ; CHECK-NEXT:       Fast Tile Register Preconfigure
 ; CHECK-NEXT:       Eliminate PHI nodes for register allocation
 ; CHECK-NEXT:       Two-Address instruction pass
 ; CHECK-NEXT:       Fast Register Allocator

llvm/test/CodeGen/X86/i386-baseptr.ll

entry:
   %a = alloca i32, i32 %k, align 4
   store i32 0, ptr %a, align 4
   ret void
 }
 
 define void @clobber_base() #0 {
 ; CHECK-LABEL: clobber_base:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    leal {{[0-9]+}}(%esp), %ebx
+; CHECK-NEXT:    .cfi_def_cfa %ebx, 0
+; CHECK-NEXT:    andl $-128, %esp
+; CHECK-NEXT:    pushl -4(%ebx)
 ; CHECK-NEXT:    pushl %ebp
-; CHECK-NEXT:    .cfi_def_cfa_offset 8
-; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
-; CHECK-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x05, 0x02, 0x75, 0x00 #
+; CHECK-NEXT:    pushl %ebx
 ; CHECK-NEXT:    pushl %esi
-; CHECK-NEXT:    andl $-128, %esp
-; CHECK-NEXT:    subl $128, %esp
-; CHECK-NEXT:    movl %esp, %esi
-; CHECK-NEXT:    .cfi_offset %esi, -12
+; CHECK-NEXT:    subl $112, %esp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x06, 0x02, 0x75, 0x78 #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x75, 0x7c #
 ; CHECK-NEXT:    calll helper@PLT
 ; CHECK-NEXT:    movl %esp, %ecx
 ; CHECK-NEXT:    leal 31(,%eax,4), %eax
 ; CHECK-NEXT:    andl $-32, %eax
 ; CHECK-NEXT:    movl %ecx, %edx
 ; CHECK-NEXT:    subl %eax, %edx
 ; CHECK-NEXT:    movl %edx, %esp
 ; CHECK-NEXT:    negl %eax
 ; CHECK-NEXT:    movl $405, %esi # imm = 0x195
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $8, %edx
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    movl %edx, (%esi)
+; CHECK-NEXT:    movl %edx, -120(%ebp)
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $0, (%ecx,%eax)
-; CHECK-NEXT:    leal -4(%ebp), %esp
+; CHECK-NEXT:    leal -8(%ebp), %esp
 ; CHECK-NEXT:    popl %esi
+; CHECK-NEXT:    popl %ebx
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    leal 4(%ebx), %esp
 ; CHECK-NEXT:    .cfi_def_cfa %esp, 4
 ; CHECK-NEXT:    retl
 entry:
   %k = call i32 @helper()
   %a = alloca i32, align 128
   %b = alloca i32, i32 %k, align 4
   ; clobber base pointer register
   tail call void asm sideeffect "nop", "{si}"(i32 405)
   call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
   store i32 0, ptr %b, align 4
   ret void
 }
 
 define x86_regcallcc void @clobber_baseptr_argptr(i32 %param1, i32 %param2, i32 %param3, i32 %param4, i32 %param5, i32 %param6) #0 {
 ; CHECK-LABEL: clobber_baseptr_argptr:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    leal {{[0-9]+}}(%esp), %ebx
+; CHECK-NEXT:    .cfi_def_cfa %ebx, 0
+; CHECK-NEXT:    andl $-128, %esp
+; CHECK-NEXT:    pushl -4(%ebx)
 ; CHECK-NEXT:    pushl %ebp
-; CHECK-NEXT:    .cfi_def_cfa_offset 8
-; CHECK-NEXT:    .cfi_offset %ebp, -8
 ; CHECK-NEXT:    movl %esp, %ebp
-; CHECK-NEXT:    .cfi_def_cfa_register %ebp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x05, 0x02, 0x75, 0x00 #
 ; CHECK-NEXT:    pushl %ebx
-; CHECK-NEXT:    andl $-128, %esp
-; CHECK-NEXT:    subl $128, %esp
-; CHECK-NEXT:    movl %esp, %esi
-; CHECK-NEXT:    .cfi_offset %ebx, -12
-; CHECK-NEXT:    movl 8(%ebp), %edi
+; CHECK-NEXT:    subl $116, %esp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x75, 0x7c #
+; CHECK-NEXT:    movl (%ebx), %edi
 ; CHECK-NEXT:    calll helper@PLT
 ; CHECK-NEXT:    movl %esp, %ecx
 ; CHECK-NEXT:    leal 31(,%eax,4), %eax
 ; CHECK-NEXT:    andl $-32, %eax
 ; CHECK-NEXT:    movl %ecx, %edx
 ; CHECK-NEXT:    subl %eax, %edx
 ; CHECK-NEXT:    movl %edx, %esp
 ; CHECK-NEXT:    negl %eax
 ; CHECK-NEXT:    movl $405, %esi # imm = 0x195
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $405, %ebx # imm = 0x195
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $8, %edx
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    movl %edx, (%esi)
+; CHECK-NEXT:    movl %edx, -120(%ebp)
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl %edi, (%ecx,%eax)
 ; CHECK-NEXT:    leal -4(%ebp), %esp
 ; CHECK-NEXT:    popl %ebx
 ; CHECK-NEXT:    popl %ebp
+; CHECK-NEXT:    leal 4(%ebx), %esp
 ; CHECK-NEXT:    .cfi_def_cfa %esp, 4
 ; CHECK-NEXT:    retl
 entry:
   %k = call i32 @helper()
   %a = alloca i32, align 128
   %b = alloca i32, i32 %k, align 4
   ; clobber base pointer register
   tail call void asm sideeffect "nop", "{si}"(i32 405)
   ; clobber argument pointer register
   tail call void asm sideeffect "nop", "{bx}"(i32 405)
   call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
   store i32 %param6, ptr %b, align 4
   ret void
 }
 
-attributes #0 = { "frame-pointer"="all"}
+attributes #0 = {"frame-pointer"="all"}
 !llvm.module.flags = !{!0}
 !0 = !{i32 2, !"override-stack-alignment", i32 32}

llvm/test/CodeGen/X86/opt-pipeline.ll

 ; CHECK-NEXT:       MachineDominator Tree Construction
 ; CHECK-NEXT:       Machine Block Frequency Analysis
 ; CHECK-NEXT:       Machine Common Subexpression Elimination
 ; CHECK-NEXT:       MachinePostDominator Tree Construction
 ; CHECK-NEXT:       Machine Cycle Info Analysis
 ; CHECK-NEXT:       Machine code sinking
 ; CHECK-NEXT:       Peephole Optimizations
 ; CHECK-NEXT:       Remove dead machine instructions
+; CHECK-NEXT:       Argument Stack Rebase
 ; CHECK-NEXT:       Live Range Shrink
 ; CHECK-NEXT:       X86 Fixup SetCC
 ; CHECK-NEXT:       Lazy Machine Block Frequency Analysis
 ; CHECK-NEXT:       X86 LEA Optimize
 ; CHECK-NEXT:       X86 Optimize Call Frame
 ; CHECK-NEXT:       X86 Avoid Store Forwarding Block
 ; CHECK-NEXT:       X86 speculative load hardening
 ; CHECK-NEXT:       MachineDominator Tree Construction

llvm/test/CodeGen/X86/swifttail-realign.ll

 ; RUN: llc -mtriple=x86_64-linux-gnu %s -o - | FileCheck %s
 
 declare swifttailcc void @callee([6 x i64], i64, i64)
 
 @var = external global ptr
 
 define swifttailcc void @caller(i64 %n) {
 ; CHECK-LABEL: caller:
+; CHECK: andq $-32, %rsp
 ; CHECK: subq $16, %rsp
 ; CHECK: pushq %rbp
 ; CHECK: movq %rsp, %rbp
 ; CHECK: pushq %rbx
-; CHECK: andq $-32, %rsp
 ; [... don't really care what happens to rsp to allocate %ptr ...]
 ; CHECK: movq 24(%rbp), [[RETADDR:%.*]]
 ; CHECK: movq [[RETADDR]], 8(%rbp)
 ; CHECK: movq $42, 16(%rbp)
 ; CHECK: movq $0, 24(%rbp)
 ; CHECK: leaq -8(%rbp), %rsp
 ; CHECK: popq %rbx
 ; CHECK: popq %rbp
 ; CHECK: jmp callee
 
   call void asm sideeffect "", "~{rbx}"()
   %ptr = alloca i8, i64 %n, align 32
   store ptr %ptr, ptr @var
   tail call swifttailcc void @callee([6 x i64] undef, i64 42, i64 0)
   ret void
 }

llvm/test/CodeGen/X86/x86-64-baseptr.ll

entry:
   %a = alloca i32, i32 %k, align 4
   store i32 0, ptr %a, align 4
   ret void
 }
 
 define void @clobber_base() #0 {
 ; CHECK-LABEL: clobber_base:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    leaq {{[0-9]+}}(%rsp), %r10
+; CHECK-NEXT:    .cfi_def_cfa %r10, 0
+; CHECK-NEXT:    andq $-128, %rsp
+; CHECK-NEXT:    pushq -8(%r10)
 ; CHECK-NEXT:    pushq %rbp
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
-; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x06, 0x02, 0x76, 0x00 #
 ; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    andq $-128, %rsp
-; CHECK-NEXT:    subq $128, %rsp
-; CHECK-NEXT:    movq %rsp, %rbx
-; CHECK-NEXT:    .cfi_offset %rbx, -24
+; CHECK-NEXT:    subq $104, %rsp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x76, 0x78 #
 ; CHECK-NEXT:    callq helper@PLT
 ; CHECK-NEXT:    movq %rsp, %rcx
 ; CHECK-NEXT:    movl %eax, %eax
 ; CHECK-NEXT:    leaq 31(,%rax,4), %rax
 ; CHECK-NEXT:    andq $-32, %rax
 ; CHECK-NEXT:    movq %rcx, %rdx
 ; CHECK-NEXT:    subq %rax, %rdx
 ; CHECK-NEXT:    movq %rdx, %rsp
 ; CHECK-NEXT:    negq %rax
 ; CHECK-NEXT:    movl $405, %ebx # imm = 0x195
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $8, %edx
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    movl %edx, (%rbx)
+; CHECK-NEXT:    movl %edx, -112(%rbp)
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $0, (%rcx,%rax)
 ; CHECK-NEXT:    leaq -8(%rbp), %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    leaq 8(%r10), %rsp
 ; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 ;
 ; X32ABI-LABEL: clobber_base:
 ; X32ABI:       # %bb.0: # %entry
+; X32ABI-NEXT:    leaq {{[0-9]+}}(%rsp), %r10
+; X32ABI-NEXT:    .cfi_def_cfa %r10, 0
+; X32ABI-NEXT:    andl $-128, %esp
+; X32ABI-NEXT:    pushq -8(%r10)
 ; X32ABI-NEXT:    pushq %rbp
-; X32ABI-NEXT:    .cfi_def_cfa_offset 16
-; X32ABI-NEXT:    .cfi_offset %rbp, -16
 ; X32ABI-NEXT:    movl %esp, %ebp
-; X32ABI-NEXT:    .cfi_def_cfa_register %rbp
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x06, 0x02, 0x76, 0x00 #
 ; X32ABI-NEXT:    pushq %rbx
-; X32ABI-NEXT:    andl $-128, %esp
-; X32ABI-NEXT:    subl $128, %esp
-; X32ABI-NEXT:    movl %esp, %ebx
-; X32ABI-NEXT:    .cfi_offset %rbx, -24
+; X32ABI-NEXT:    subl $104, %esp
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x76, 0x78 #
 ; X32ABI-NEXT:    callq helper@PLT
 ; X32ABI-NEXT:    # kill: def $eax killed $eax def $rax
 ; X32ABI-NEXT:    leal 31(,%rax,4), %eax
 ; X32ABI-NEXT:    andl $-32, %eax
 ; X32ABI-NEXT:    movl %esp, %ecx
 ; X32ABI-NEXT:    movl %ecx, %edx
 ; X32ABI-NEXT:    subl %eax, %edx
 ; X32ABI-NEXT:    negl %eax
 ; X32ABI-NEXT:    movl %edx, %esp
 ; X32ABI-NEXT:    movl $405, %ebx # imm = 0x195
 ; X32ABI-NEXT:    #APP
 ; X32ABI-NEXT:    nop
 ; X32ABI-NEXT:    #NO_APP
 ; X32ABI-NEXT:    movl $8, %edx
 ; X32ABI-NEXT:    #APP
-; X32ABI-NEXT:    movl %edx, (%ebx)
+; X32ABI-NEXT:    movl %edx, -112(%ebp)
 ; X32ABI-NEXT:    #NO_APP
 ; X32ABI-NEXT:    movl $0, (%ecx,%eax)
 ; X32ABI-NEXT:    leal -8(%ebp), %esp
 ; X32ABI-NEXT:    popq %rbx
 ; X32ABI-NEXT:    popq %rbp
+; X32ABI-NEXT:    leaq 8(%r10), %rsp
 ; X32ABI-NEXT:    .cfi_def_cfa %rsp, 8
 ; X32ABI-NEXT:    retq
 entry:
   %k = call i32 @helper()
   %a = alloca i32, align 128
   %b = alloca i32, i32 %k, align 4
   ; clobber base pointer register
   tail call void asm sideeffect "nop", "{bx}"(i32 405)
   call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
   store i32 0, ptr %b, align 4
   ret void
 }
 
 define x86_regcallcc void @clobber_baseptr_argptr(i32 %param1, i32 %param2, i32 %param3, i32 %param4, i32 %param5, i32 %param6, i32 %param7, i32 %param8, i32 %param9, i32 %param10, i32 %param11, i32 %param12) #0 {
 ; CHECK-LABEL: clobber_baseptr_argptr:
 ; CHECK:       # %bb.0: # %entry
+; CHECK-NEXT:    leaq {{[0-9]+}}(%rsp), %r10
+; CHECK-NEXT:    .cfi_def_cfa %r10, 0
+; CHECK-NEXT:    andq $-128, %rsp
+; CHECK-NEXT:    pushq -8(%r10)
 ; CHECK-NEXT:    pushq %rbp
-; CHECK-NEXT:    .cfi_def_cfa_offset 16
-; CHECK-NEXT:    .cfi_offset %rbp, -16
 ; CHECK-NEXT:    movq %rsp, %rbp
-; CHECK-NEXT:    .cfi_def_cfa_register %rbp
+; CHECK-NEXT:    .cfi_escape 0x10, 0x06, 0x02, 0x76, 0x00 #
 ; CHECK-NEXT:    pushq %rbx
-; CHECK-NEXT:    andq $-128, %rsp
-; CHECK-NEXT:    subq $256, %rsp # imm = 0x100
+; CHECK-NEXT:    subq $232, %rsp
 ; CHECK-NEXT:    movaps %xmm15, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm14, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm13, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm12, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm11, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm10, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm9, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
 ; CHECK-NEXT:    movaps %xmm8, {{[-0-9]+}}(%r{{[sb]}}p) # 16-byte Spill
-; CHECK-NEXT:    movq %rsp, %rbx
-; CHECK-NEXT:    .cfi_offset %rbx, -24
-; CHECK-NEXT:    .cfi_offset %xmm8, -160
-; CHECK-NEXT:    .cfi_offset %xmm9, -144
-; CHECK-NEXT:    .cfi_offset %xmm10, -128
-; CHECK-NEXT:    .cfi_offset %xmm11, -112
-; CHECK-NEXT:    .cfi_offset %xmm12, -96
-; CHECK-NEXT:    .cfi_offset %xmm13, -80
-; CHECK-NEXT:    .cfi_offset %xmm14, -64
-; CHECK-NEXT:    .cfi_offset %xmm15, -48
-; CHECK-NEXT:    movl 16(%rbp), %r14d
+; CHECK-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x76, 0x78 #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x19, 0x02, 0x76, 0xf0, 0x7e #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1a, 0x02, 0x76, 0x80, 0x7f #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1b, 0x02, 0x76, 0x90, 0x7f #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1c, 0x02, 0x76, 0xa0, 0x7f #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1d, 0x02, 0x76, 0xb0, 0x7f #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1e, 0x02, 0x76, 0x40 #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x1f, 0x02, 0x76, 0x50 #
+; CHECK-NEXT:    .cfi_escape 0x10, 0x20, 0x02, 0x76, 0x60 #
+; CHECK-NEXT:    movl (%r10), %r14d
 ; CHECK-NEXT:    callq helper@PLT
 ; CHECK-NEXT:    movq %rsp, %rcx
 ; CHECK-NEXT:    movl %eax, %eax
 ; CHECK-NEXT:    leaq 31(,%rax,4), %rax
 ; CHECK-NEXT:    andq $-32, %rax
 ; CHECK-NEXT:    movq %rcx, %rdx
 ; CHECK-NEXT:    subq %rax, %rdx
 ; CHECK-NEXT:    movq %rdx, %rsp
 ; CHECK-NEXT:    negq %rax
 ; CHECK-NEXT:    movl $405, %ebx # imm = 0x195
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    #APP
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl $8, %edx
 ; CHECK-NEXT:    #APP
-; CHECK-NEXT:    movl %edx, (%rbx)
+; CHECK-NEXT:    movl %edx, -240(%rbp)
 ; CHECK-NEXT:    #NO_APP
 ; CHECK-NEXT:    movl %r14d, (%rcx,%rax)
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm8 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm9 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm10 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm11 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm12 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm13 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm14 # 16-byte Reload
 ; CHECK-NEXT:    movaps {{[-0-9]+}}(%r{{[sb]}}p), %xmm15 # 16-byte Reload
 ; CHECK-NEXT:    leaq -8(%rbp), %rsp
 ; CHECK-NEXT:    popq %rbx
 ; CHECK-NEXT:    popq %rbp
+; CHECK-NEXT:    leaq 8(%r10), %rsp
 ; CHECK-NEXT:    .cfi_def_cfa %rsp, 8
 ; CHECK-NEXT:    retq
 ;
 ; X32ABI-LABEL: clobber_baseptr_argptr:
 ; X32ABI:       # %bb.0: # %entry
+; X32ABI-NEXT:    leaq {{[0-9]+}}(%rsp), %r10
+; X32ABI-NEXT:    .cfi_def_cfa %r10, 0
+; X32ABI-NEXT:    andl $-128, %esp
+; X32ABI-NEXT:    pushq -8(%r10)
 ; X32ABI-NEXT:    pushq %rbp
-; X32ABI-NEXT:    .cfi_def_cfa_offset 16
-; X32ABI-NEXT:    .cfi_offset %rbp, -16
 ; X32ABI-NEXT:    movl %esp, %ebp
-; X32ABI-NEXT:    .cfi_def_cfa_register %rbp
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x06, 0x02, 0x76, 0x00 #
 ; X32ABI-NEXT:    pushq %rbx
-; X32ABI-NEXT:    andl $-128, %esp
-; X32ABI-NEXT:    subl $256, %esp # imm = 0x100
+; X32ABI-NEXT:    subl $232, %esp
 ; X32ABI-NEXT:    movaps %xmm15, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm14, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm13, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm12, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm11, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm10, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm9, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
 ; X32ABI-NEXT:    movaps %xmm8, {{[-0-9]+}}(%e{{[sb]}}p) # 16-byte Spill
-; X32ABI-NEXT:    movl %esp, %ebx
-; X32ABI-NEXT:    .cfi_offset %rbx, -24
-; X32ABI-NEXT:    .cfi_offset %xmm8, -160
-; X32ABI-NEXT:    .cfi_offset %xmm9, -144
-; X32ABI-NEXT:    .cfi_offset %xmm10, -128
-; X32ABI-NEXT:    .cfi_offset %xmm11, -112
-; X32ABI-NEXT:    .cfi_offset %xmm12, -96
-; X32ABI-NEXT:    .cfi_offset %xmm13, -80
-; X32ABI-NEXT:    .cfi_offset %xmm14, -64
-; X32ABI-NEXT:    .cfi_offset %xmm15, -48
-; X32ABI-NEXT:    movl 16(%ebp), %r14d
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x03, 0x02, 0x76, 0x78 #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x19, 0x02, 0x76, 0xf0, 0x7e #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1a, 0x02, 0x76, 0x80, 0x7f #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1b, 0x02, 0x76, 0x90, 0x7f #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1c, 0x02, 0x76, 0xa0, 0x7f #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1d, 0x02, 0x76, 0xb0, 0x7f #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1e, 0x02, 0x76, 0x40 #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x1f, 0x02, 0x76, 0x50 #
+; X32ABI-NEXT:    .cfi_escape 0x10, 0x20, 0x02, 0x76, 0x60 #
+; X32ABI-NEXT:    movl (%r10), %r14d
 ; X32ABI-NEXT:    callq helper@PLT
 ; X32ABI-NEXT:    # kill: def $eax killed $eax def $rax
 ; X32ABI-NEXT:    leal 31(,%rax,4), %eax
 ; X32ABI-NEXT:    andl $-32, %eax
 ; X32ABI-NEXT:    movl %esp, %ecx
 ; X32ABI-NEXT:    movl %ecx, %edx
 ; X32ABI-NEXT:    subl %eax, %edx
 ; X32ABI-NEXT:    negl %eax
 ; X32ABI-NEXT:    movl %edx, %esp
 ; X32ABI-NEXT:    movl $405, %ebx # imm = 0x195
 ; X32ABI-NEXT:    #APP
 ; X32ABI-NEXT:    nop
 ; X32ABI-NEXT:    #NO_APP
 ; X32ABI-NEXT:    #APP
 ; X32ABI-NEXT:    nop
 ; X32ABI-NEXT:    #NO_APP
 ; X32ABI-NEXT:    movl $8, %edx
 ; X32ABI-NEXT:    #APP
-; X32ABI-NEXT:    movl %edx, (%ebx)
+; X32ABI-NEXT:    movl %edx, -240(%ebp)
 ; X32ABI-NEXT:    #NO_APP
 ; X32ABI-NEXT:    movl %r14d, (%ecx,%eax)
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm8 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm9 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm10 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm11 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm12 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm13 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm14 # 16-byte Reload
 ; X32ABI-NEXT:    movaps {{[-0-9]+}}(%e{{[sb]}}p), %xmm15 # 16-byte Reload
 ; X32ABI-NEXT:    leal -8(%ebp), %esp
 ; X32ABI-NEXT:    popq %rbx
 ; X32ABI-NEXT:    popq %rbp
+; X32ABI-NEXT:    leaq 8(%r10), %rsp
 ; X32ABI-NEXT:    .cfi_def_cfa %rsp, 8
 ; X32ABI-NEXT:    retq
 entry:
   %k = call i32 @helper()
   %a = alloca i32, align 128
   %b = alloca i32, i32 %k, align 4
   ; clobber base pointer register
   tail call void asm sideeffect "nop", "{bx}"(i32 405)
   ; clobber argument pointer register
   tail call void asm sideeffect "nop", "~{bx},~{r10},~{r11}"()
   call void asm sideeffect "movl $0, $1", "r,*m"(i32 8, ptr elementtype(i32) %a)
   store i32 %param12, ptr %b, align 4
   ret void
 }
 
-attributes #0 = { "frame-pointer"="all"}
+attributes #0 = {"frame-pointer"="all"}
 !llvm.module.flags = !{!0}
 !0 = !{i32 2, !"override-stack-alignment", i32 32}