[mips] Add support for dynamic stack realignment.

[vkalintiris]

Summary

With this change we are able to realign the stack dynamically, whenever it
contains objects with alignment requirements that are larger than the
alignment specified from the given ABI.

We have to use the $fp register as the frame pointer when we perform
dynamic stack realignment. In complex stack frames, with variably-sized
objects, we reserve additionally the callee-saved register $s7 as the
base pointer in order to reference locals.

[vkalintiris]

Fix register usage for functions without dynamic stack allocation.

In functions without dynamic stack allocation we were using $fp for
referencing locals. However, in a function's body we have to use $fp only
for referencing inbound arguments and $sp for everything else. I added
some tests that would have caught this case.

[dsanders]

I haven't finished looking at func_03 - func_10 in the test case yet but here's my comments so far.

Could you add some tests that check the debug info for variables affected by this? I'd like to be sure that the location information is correct.

lib/Target/Mips/MipsFrameLowering.cpp

92–96

Nit? MFI->isFrameAddressTaken() doesn't seem to come under any of these conditions.

lib/Target/Mips/MipsRegisterInfo.cpp

185–186

Nit: This is the same as hasBP() we should use that if we can to avoid code duplication.

290–302

Nit: We should explain these conditions.

313–328

I think I may be missing something here. The code seems to be saying that if we need realignment but can't do achieve it, then we don't really need it. Shouldn't that at least emit a warning?

I see other targets are silently doing the same thing, so this may be intended but it seems odd.

lib/Target/Mips/MipsSERegisterInfo.cpp

143

It's not necessary in this patch but we should consider adding inlined members to MipsRegisterInfo rather than duplicating the selection between REG and REG_64 all the time (e.g. getZeroReg(), getFPReg(), getSPReg(), etc.).

144

Why 'FrameIndex < 0' and not 'MFI->isFixedObjectIndex(FrameIndex)'?

test/CodeGen/Mips/dynamic-stack-realignment.ll

18

Nit: Could you make it a bit clearer what cases each test covers? I was going to comment that we should check N32/N64's output too but I see that func_02 is covering the N64 case.

23

I don't think -1024 is correct (but the program will work as far as I can tell). The frame will contain the 8-bytes for $ra/$fp, up to 504-bytes padding, 4-bytes for the alloca'd i32, 4-bytes for the 5th argument to helper_01, and 16-bytes for the reserved space, giving a total of between 32 and 536 bytes for the frame.

At the moment, we seem to be allocating 1024 bytes as well as up to 504 bytes padding. It looks like we're rounding the size of each region of the frame up to the maximum alignment.

We're discussing off-list whether the base pointer and locals with large alignments should be where they currently are. Some of the above issues may be related to the current position of this pointer and region.

49

We should check the N32 case too.

[dsanders]

lib/Target/Mips/MipsSERegisterInfo.cpp

143

Most of this has been added to MipsABIInfo

test/CodeGen/Mips/dynamic-stack-realignment.ll

23

We reached the conclusion that we can do better but it's not actually wrong. It's functional but not optimal. Given that the alignment of vectors isn't honoured at the moment (and this breaks pocl), we should proceed with this layout and refine it later.

234–260

Shouldn't these two cases be errors? We need to align to a large alignment but we can't because we've been told not to realign the stack.

[vkalintiris]

Addressed the first set of comments in the original review request.

[vkalintiris]

lib/Target/Mips/MipsRegisterInfo.cpp

185–186

We need access to a MipsFrameLowering object in order to use hasBP(). I'm not sure but I don't think we can get a reference to this object from inside MipsRegisterInfo.

313–328

I agree with you on that. IMHO, a better name would be shouldRealignStack() or something similar. I kept this name because I wanted to be consistent with the other targets.

[vkalintiris]

Clarify what happens when we get a request for dynamic stack realignment
if the function has the attribute "no-realign-stack" too.

[vkalintiris]

In reply to comments for lines 233–[e-mail] (Diff #22948)

I agree with you. However, reporting something back to the user is a little bit more difficult than it seems. The alignment of the objects inside such a function is clamped to that of the stack's alignment based on the given ABI. I moved the check for MF.getFunction()->hasFnAttribute("no-realign-stack") from canRealign() to needsStackRealignment(), in order to make the check more visible and added a comment that explains the whole situation.

[dsanders]

LGTM with the nits fixed and a warning for the 'needs realignment but can't realign' case.

We'll need to follow up on the overallocation issue but suboptimal is better than non-functional.

lib/Target/Mips/MipsFrameLowering.cpp

92–96

Done

lib/Target/Mips/MipsRegisterInfo.cpp

185–186

In that case, could you add a comment indicating it should be the same as hasBP()

290–302

Done

313–328

Both spellings have the same issue. If the answer to 'Are we able to realign the stack?' is 'no' then needsStackRealignment()/shouldRealignStack() can return false even though the answer to 'Do we need stack realignment?' is yes.

We should emit a warning when this happens since the users code is very likely to break in this situation.

319–325

A couple typos:
Unfotunately -> Unfortunately
attirbute -> attribute

lib/Target/Mips/MipsSERegisterInfo.cpp

143

Nit: Instead of:

ABI.IsN64() ? Mips::S7_64 : Mips::S7;

add a MipsABIInfo::GetBasePtr() and use that.

144

Done

test/CodeGen/Mips/dynamic-stack-realignment.ll

18

Done

31

Nit: Please document the overallocation with a FIXME. This particular case needs a frame of up to between 16 and 512-bytes but currently allocates between 1024 and 1536 bytes.

Likewise for the other examples below.

49

Done

234–260

Done (MipsRegisterInfo::needsStackRealignment() has a comment explaining why they aren't errors).

lib/Target/Mips/MipsFrameLowering.h

protected:
   const MipsSubtarget &STI;
 
 public:
   explicit MipsFrameLowering(const MipsSubtarget &sti, unsigned Alignment)
     : TargetFrameLowering(StackGrowsDown, Alignment, 0, Alignment), STI(sti) {}
 
   static const MipsFrameLowering *create(const MipsSubtarget &ST);
 
-  bool hasFP(const MachineFunction &MF) const override;
-
   void
   eliminateCallFramePseudoInstr(MachineFunction &MF,
                                 MachineBasicBlock &MBB,
                                 MachineBasicBlock::iterator I) const override;
 
+
+  bool hasFP(const MachineFunction &MF) const override;
+
+  bool hasBP(const MachineFunction &MF) const;
+
 protected:
   uint64_t estimateStackSize(const MachineFunction &MF) const;
 };
 
 /// Create MipsFrameLowering objects.
 const MipsFrameLowering *createMips16FrameLowering(const MipsSubtarget &ST);
 const MipsFrameLowering *createMipsSEFrameLowering(const MipsSubtarget &ST);
 
 } // End llvm namespace
 
 #endif

lib/Target/Mips/MipsFrameLowering.cpp

 
 const MipsFrameLowering *MipsFrameLowering::create(const MipsSubtarget &ST) {
   if (ST.inMips16Mode())
     return llvm::createMips16FrameLowering(ST);
 
   return llvm::createMipsSEFrameLowering(ST);
 }
 
 // hasFP - Return true if the specified function should have a dedicated frame
-// pointer register.  This is true if the function has variable sized allocas or
-// if frame pointer elimination is disabled.
+// pointer register.  This is true if the function has variable sized allocas,
+// if it needs dynamic stack realignment, if frame pointer elimination is
+// disabled, or if the frame address is taken.
 bool MipsFrameLowering::hasFP(const MachineFunction &MF) const {
   const MachineFrameInfo *MFI = MF.getFrameInfo();
+  const TargetRegisterInfo *TRI = STI.getRegisterInfo();
+
   return MF.getTarget().Options.DisableFramePointerElim(MF) ||
-      MFI->hasVarSizedObjects() || MFI->isFrameAddressTaken();
+      MFI->hasVarSizedObjects() || MFI->isFrameAddressTaken() ||
+      TRI->needsStackRealignment(MF);
+}
+
+bool MipsFrameLowering::hasBP(const MachineFunction &MF) const {
+  const MachineFrameInfo *MFI = MF.getFrameInfo();
+  const TargetRegisterInfo *TRI = STI.getRegisterInfo();
+
+  return MFI->hasVarSizedObjects() && TRI->needsStackRealignment(MF);
 }
 
 uint64_t MipsFrameLowering::estimateStackSize(const MachineFunction &MF) const {
   const MachineFrameInfo *MFI = MF.getFrameInfo();
   const TargetRegisterInfo &TRI = *STI.getRegisterInfo();
 
   int64_t Offset = 0;
 

lib/Target/Mips/MipsRegisterInfo.h

public:
   /// Stack Frame Processing Methods
   void eliminateFrameIndex(MachineBasicBlock::iterator II,
                            int SPAdj, unsigned FIOperandNum,
                            RegScavenger *RS = nullptr) const override;
 
   void processFunctionBeforeFrameFinalized(MachineFunction &MF,
                                        RegScavenger *RS = nullptr) const;
 
+  // Stack realignment queries.
+  bool canRealignStack(const MachineFunction &MF) const;
+
+  bool needsStackRealignment(const MachineFunction &MF) const override;
+
   /// Debug information queries.
   unsigned getFrameRegister(const MachineFunction &MF) const override;
 
   /// \brief Return GPR register class.
   virtual const TargetRegisterClass *intRegClass(unsigned Size) const = 0;
 
 private:
   virtual void eliminateFI(MachineBasicBlock::iterator II, unsigned OpNo,
                            int FrameIndex, uint64_t StackSize,
                            int64_t SPOffset) const = 0;
 };
 
 } // end namespace llvm
 
 #endif

lib/Target/Mips/MipsRegisterInfo.cpp

 #include "MipsAnalyzeImmediate.h"
 #include "MipsInstrInfo.h"
 #include "MipsMachineFunction.h"
 #include "MipsSubtarget.h"
 #include "MipsTargetMachine.h"
 #include "llvm/ADT/BitVector.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
+#include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/MachineFunction.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/IR/Constants.h"
 #include "llvm/IR/DebugInfo.h"
 #include "llvm/IR/Function.h"
 #include "llvm/IR/Type.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
getReservedRegs(const MachineFunction &MF) const {
   }
   // Reserve FP if this function should have a dedicated frame pointer register.
   if (Subtarget.getFrameLowering()->hasFP(MF)) {
     if (Subtarget.inMips16Mode())
       Reserved.set(Mips::S0);
     else {
       Reserved.set(Mips::FP);
       Reserved.set(Mips::FP_64);
+
+      // Reserve the base register if we need to both realign the stack and
+      // allocate variable-sized objects at runtime.
+      if (needsStackRealignment(MF) &&
+          MF.getFrameInfo()->hasVarSizedObjects()) {
+        Reserved.set(Mips::S7);
+        Reserved.set(Mips::S7_64);
+      }
     }
   }
 
   // Reserve hardware registers.
   Reserved.set(Mips::HWR29);
 
   // Reserve DSP control register.
   Reserved.set(Mips::DSPPos);
getFrameRegister(const MachineFunction &MF) const {
   bool IsN64 =
       static_cast<const MipsTargetMachine &>(MF.getTarget()).getABI().IsN64();
 
   if (Subtarget.inMips16Mode())
     return TFI->hasFP(MF) ? Mips::S0 : Mips::SP;
   else
     return TFI->hasFP(MF) ? (IsN64 ? Mips::FP_64 : Mips::FP) :
                             (IsN64 ? Mips::SP_64 : Mips::SP);
+}
+
+bool MipsRegisterInfo::canRealignStack(const MachineFunction &MF) const {
+  const MipsSubtarget &Subtarget = MF.getSubtarget<MipsSubtarget>();
+  unsigned FP = Subtarget.isGP32bit() ? Mips::FP : Mips::FP_64;
+  unsigned BP = Subtarget.isGP32bit() ? Mips::S7 : Mips::S7_64;
+
+  // Support dynamic stack realignment only for targets with standard encoding.
+  if (!Subtarget.hasStandardEncoding())
+    return false;
+
+  // Avoid realigning functions that explicitly do not want to be realigned.
+  if (MF.getFunction()->hasFnAttribute("no-realign-stack"))
+    return false;
+
+  // We can't perform dynamic stack realignment if we can't reserve the
+  // frame pointer register.
+  if (!MF.getRegInfo().canReserveReg(FP))
+    return false;
+
+  // We can realign the stack if we know the maximum call frame size and we
+  // don't have variable sized objects.
+  if (Subtarget.getFrameLowering()->hasReservedCallFrame(MF))
+    return true;
 
+  // We have to reserve the base pointer register in the presence of variable
+  // sized objects.
+  return MF.getRegInfo().canReserveReg(BP);
 }
 
+bool MipsRegisterInfo::needsStackRealignment(const MachineFunction &MF) const {
+  const MipsSubtarget &Subtarget = MF.getSubtarget<MipsSubtarget>();
+  const MachineFrameInfo *MFI = MF.getFrameInfo();
+
+  bool CanRealign = canRealignStack(MF);
+
+  const Function *F = MF.getFunction();
+  if (F->hasFnAttribute(Attribute::StackAlignment))
+    return CanRealign;
+
+  unsigned StackAlignment = Subtarget.getFrameLowering()->getStackAlignment();
+  if (MFI->getMaxAlignment() > StackAlignment)
+    return CanRealign;
+
+  return false;
+}

lib/Target/Mips/MipsSEFrameLowering.cpp

void MipsSEFrameLowering::emitPrologue(MachineFunction &MF) const {
 
   MachineBasicBlock::iterator MBBI = MBB.begin();
   DebugLoc dl = MBBI != MBB.end() ? MBBI->getDebugLoc() : DebugLoc();
   MipsABIInfo ABI = STI.getABI();
   unsigned SP = ABI.GetStackPtr();
   unsigned FP = ABI.GetFramePtr();
   unsigned ZERO = ABI.GetNullPtr();
   unsigned ADDu = ABI.GetPtrAdduOp();
+  unsigned ADDiu = ABI.GetPtrAddiuOp();
+  unsigned AND = ABI.IsN64() ? Mips::AND64 : Mips::AND;
+
+  const TargetRegisterClass *RC = ABI.ArePtrs64bit() ?
+        &Mips::GPR64RegClass : &Mips::GPR32RegClass;
 
   // First, compute final stack size.
   uint64_t StackSize = MFI->getStackSize();
 
   // No need to allocate space on the stack.
   if (StackSize == 0 && !MFI->adjustsStack()) return;
 
   MachineModuleInfo &MMI = MF.getMMI();
for (std::vector<CalleeSavedInfo>::const_iterator I = CSI.begin(),
             nullptr, MRI->getDwarfRegNum(Reg, 1), Offset));
         BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
             .addCFIIndex(CFIIndex);
       }
     }
   }
 
   if (MipsFI->callsEhReturn()) {
-    const TargetRegisterClass *PtrRC =
-        ABI.ArePtrs64bit() ? &Mips::GPR64RegClass : &Mips::GPR32RegClass;
-
     // Insert instructions that spill eh data registers.
     for (int I = 0; I < 4; ++I) {
       if (!MBB.isLiveIn(ABI.GetEhDataReg(I)))
         MBB.addLiveIn(ABI.GetEhDataReg(I));
       TII.storeRegToStackSlot(MBB, MBBI, ABI.GetEhDataReg(I), false,
-                              MipsFI->getEhDataRegFI(I), PtrRC, &RegInfo);
+                              MipsFI->getEhDataRegFI(I), RC, &RegInfo);
     }
 
     // Emit .cfi_offset directives for eh data registers.
     for (int I = 0; I < 4; ++I) {
       int64_t Offset = MFI->getObjectOffset(MipsFI->getEhDataRegFI(I));
       unsigned Reg = MRI->getDwarfRegNum(ABI.GetEhDataReg(I), true);
       unsigned CFIIndex = MMI.addFrameInst(
           MCCFIInstruction::createOffset(nullptr, Reg, Offset));
       BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex);
     }
   }
 
   // if framepointer enabled, set it to point to the stack pointer.
   if (hasFP(MF)) {
     // Insert instruction "move $fp, $sp" at this location.
     BuildMI(MBB, MBBI, dl, TII.get(ADDu), FP).addReg(SP).addReg(ZERO)
       .setMIFlag(MachineInstr::FrameSetup);
 
     // emit ".cfi_def_cfa_register $fp"
     unsigned CFIIndex = MMI.addFrameInst(MCCFIInstruction::createDefCfaRegister(
         nullptr, MRI->getDwarfRegNum(FP, true)));
     BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
         .addCFIIndex(CFIIndex);
+
+    if (RegInfo.needsStackRealignment(MF)) {
+      // addiu $Reg, $zero, -MaxAlignment
+      // andi $sp, $sp, $Reg
+      unsigned VR = MF.getRegInfo().createVirtualRegister(RC);
+      assert(isInt<16>(MFI->getMaxAlignment()) &&
+             "Function's alignment size requirement is not supported.");
+      int MaxAlign = - (signed) MFI->getMaxAlignment();
+
+      BuildMI(MBB, MBBI, dl, TII.get(ADDiu), VR).addReg(ZERO) .addImm(MaxAlign);
+      BuildMI(MBB, MBBI, dl, TII.get(AND), SP).addReg(SP).addReg(VR);
+
+      if (hasBP(MF)) {
+        // move $s7, $sp
+        unsigned BP = STI.isABI_N64() ? Mips::S7_64 : Mips::S7;
+        BuildMI(MBB, MBBI, dl, TII.get(ADDu), BP)
+          .addReg(SP)
+          .addReg(ZERO);
+      }
+    }
   }
 }
 
 void MipsSEFrameLowering::emitEpilogue(MachineFunction &MF,
                                        MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.getLastNonDebugInstr();
   MachineFrameInfo *MFI            = MF.getFrameInfo();
   MipsFunctionInfo *MipsFI = MF.getInfo<MipsFunctionInfo>();

 
 void MipsSEFrameLowering::
 processFunctionBeforeCalleeSavedScan(MachineFunction &MF,
                                      RegScavenger *RS) const {
   MachineRegisterInfo &MRI = MF.getRegInfo();
   MipsFunctionInfo *MipsFI = MF.getInfo<MipsFunctionInfo>();
   MipsABIInfo ABI = STI.getABI();
   unsigned FP = ABI.GetFramePtr();
+  unsigned BP = ABI.IsN64() ? Mips::S7_64 : Mips::S7;
 
   // Mark $fp as used if function has dedicated frame pointer.
   if (hasFP(MF))
     MRI.setPhysRegUsed(FP);
+  // Mark $s7 as used if function has dedicated base pointer.
+  if (hasBP(MF))
+    MRI.setPhysRegUsed(BP);
 
   // Create spill slots for eh data registers if function calls eh_return.
   if (MipsFI->callsEhReturn())
     MipsFI->createEhDataRegsFI();
 
   // Expand pseudo instructions which load, store or copy accumulators.
   // Add an emergency spill slot if a pseudo was expanded.
   if (ExpandPseudo(MF).expand()) {

lib/Target/Mips/MipsSERegisterInfo.cpp

 void MipsSERegisterInfo::eliminateFI(MachineBasicBlock::iterator II,
                                      unsigned OpNo, int FrameIndex,
                                      uint64_t StackSize,
                                      int64_t SPOffset) const {
   MachineInstr &MI = *II;
   MachineFunction &MF = *MI.getParent()->getParent();
   MachineFrameInfo *MFI = MF.getFrameInfo();
   MipsFunctionInfo *MipsFI = MF.getInfo<MipsFunctionInfo>();
+
   MipsABIInfo ABI =
       static_cast<const MipsTargetMachine &>(MF.getTarget()).getABI();
+  const MipsRegisterInfo *RegInfo =
+    static_cast<const MipsRegisterInfo *>(MF.getSubtarget().getRegisterInfo());
 
   const std::vector<CalleeSavedInfo> &CSI = MFI->getCalleeSavedInfo();
   int MinCSFI = 0;
   int MaxCSFI = -1;
 
   if (CSI.size()) {
     MinCSFI = CSI[0].getFrameIdx();
     MaxCSFI = CSI[CSI.size() - 1].getFrameIdx();
   }
 
   bool EhDataRegFI = MipsFI->isEhDataRegFI(FrameIndex);
 
   // The following stack frame objects are always referenced relative to $sp:
   //  1. Outgoing arguments.
   //  2. Pointer to dynamically allocated stack space.
   //  3. Locations for callee-saved registers.
   //  4. Locations for eh data registers.
   // Everything else is referenced relative to whatever register
   // getFrameRegister() returns.
   unsigned FrameReg;
 
   if ((FrameIndex >= MinCSFI && FrameIndex <= MaxCSFI) || EhDataRegFI)
     FrameReg = ABI.GetStackPtr();
+  else if (RegInfo->needsStackRealignment(MF)) {
+    if (MFI->hasVarSizedObjects() && !MFI->isFixedObjectIndex(FrameIndex))
+      FrameReg = ABI.IsN64() ? Mips::S7_64 : Mips::S7;
+    else if (MFI->isFixedObjectIndex(FrameIndex))
+      FrameReg = getFrameRegister(MF);
     else
+      FrameReg = ABI.IsN64() ? Mips::SP_64 : Mips::SP;
+  } else
     FrameReg = getFrameRegister(MF);
 
   // Calculate final offset.
   // - There is no need to change the offset if the frame object is one of the
   //   following: an outgoing argument, pointer to a dynamically allocated
   //   stack space or a $gp restore location,
   // - If the frame object is any of the following, its offset must be adjusted
   //   by adding the size of the stack:

test/CodeGen/Mips/dynamic-stack-realignment.ll

+; RUN: llc < %s -march=mips -mcpu=mips2 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP32
+; RUN: llc < %s -march=mips -mcpu=mips32 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP32
+; RUN: llc < %s -march=mips -mcpu=mips32r6 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP32
+; RUN: llc < %s -march=mips64 -mcpu=mips3 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N64
+; RUN: llc < %s -march=mips64 -mcpu=mips64 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N64
+; RUN: llc < %s -march=mips64 -mcpu=mips64r6 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N64
+; RUN: llc < %s -march=mips64 -mcpu=mips3 -target-abi n32 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N32
+; RUN: llc < %s -march=mips64 -mcpu=mips64 -target-abi n32 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N32
+; RUN: llc < %s -march=mips64 -mcpu=mips64r6 -target-abi n32 | FileCheck %s \
+; RUN:    --check-prefix=ALL --check-prefix=GP64 -check-prefix=N32
+
+; Check dynamic stack realignment in functions without variable-sized objects.
+
+declare void @helper_01(i32, i32, i32, i32, i32*)
+
+; O32 ABI
+define void @func_01() {
+entry:
+; GP32-LABEL: func_01:
+
+  ; prologue
+  ; GP32:       addiu   $sp, $sp, -1024
+  ; GP32:       sw      $ra, 1020($sp)
+  ; GP32:       sw      $fp, 1016($sp)
+  ;
+  ; GP32:       move    $fp, $sp
+  ; GP32:       addiu   $[[T0:[0-9]+|ra|gp]], $zero, -512
+  ; GP32-NEXT:  and     $sp, $sp, $[[T0]]
+
+  ; body
+  ; GP32:       addiu   $[[T1:[0-9]+]], $sp, 512
+  ; GP32:       sw      $[[T1]], 16($sp)
+
+  ; epilogue
+  ; GP32:       move    $sp, $fp
+  ; GP32:       lw      $fp, 1016($sp)
+  ; GP32:       lw      $ra, 1020($sp)
+  ; GP32:       addiu   $sp, $sp, 1024
+
+  %a = alloca i32, align 512
+  call void @helper_01(i32 0, i32 0, i32 0, i32 0, i32* %a)
+  ret void
+}
+
+declare void @helper_02(i32, i32, i32, i32,
+                        i32, i32, i32, i32, i32*)
+
+; N32/N64 ABIs
+define void @func_02() {
+entry:
+; GP64-LABEL: func_02:
+
+  ; prologue
+  ; N32:        addiu   $sp, $sp, -1024
+  ; N64:        daddiu  $sp, $sp, -1024
+  ; GP64:       sd      $ra, 1016($sp)
+  ; GP64:       sd      $fp, 1008($sp)
+  ; N32:        sd      $gp, 1000($sp)
+  ;
+  ; GP64:       move    $fp, $sp
+  ; N32:        addiu   $[[T0:[0-9]+|ra]], $zero, -512
+  ; N64:        daddiu  $[[T0:[0-9]+|ra]], $zero, -512
+  ; GP64-NEXT:  and     $sp, $sp, $[[T0]]
+
+  ; body
+  ; N32:        addiu   $[[T1:[0-9]+]], $sp, 512
+  ; N64:        daddiu  $[[T1:[0-9]+]], $sp, 512
+  ; GP64:       sd      $[[T1]], 0($sp)
+
+  ; epilogue
+  ; GP64:       move    $sp, $fp
+  ; N32:        ld      $gp, 1000($sp)
+  ; GP64:       ld      $fp, 1008($sp)
+  ; GP64:       ld      $ra, 1016($sp)
+  ; N32:        addiu   $sp, $sp, 1024
+  ; N64:        daddiu  $sp, $sp, 1024
+
+  %a = alloca i32, align 512
+  call void @helper_02(i32 0, i32 0, i32 0, i32 0,
+                       i32 0, i32 0, i32 0, i32 0, i32* %a)
+  ret void
+}
+
+; Verify that we use $fp for referencing incoming arguments.
+
+declare void @helper_03(i32, i32, i32, i32, i32*, i32*)
+
+; O32 ABI
+define void @func_03(i32 %p0, i32 %p1, i32 %p2, i32 %p3, i32* %b) {
+entry:
+; GP32-LABEL: func_03:
+
+  ; body
+  ; GP32-DAG:   addiu   $[[T0:[0-9]+]], $sp, 512
+  ; GP32-DAG:   sw      $[[T0]], 16($sp)
+  ; GP32-DAG:   lw      $[[T1:[0-9]+]], 1040($fp)
+  ; GP32-DAG:   sw      $[[T1]], 20($sp)
+
+  %a = alloca i32, align 512
+  call void @helper_03(i32 0, i32 0, i32 0, i32 0, i32* %a, i32* %b)
+  ret void
+}
+
+declare void @helper_04(i32, i32, i32, i32,
+                        i32, i32, i32, i32, i32*, i32*)
+
+; N32/N64 ABIs
+define void @func_04(i32 %p0, i32 %p1, i32 %p2, i32 %p3,
+                     i32 %p4, i32 %p5, i32 %p6, i32 %p7,
+                     i32* %b) {
+entry:
+; GP64-LABEL: func_04:
+
+  ; body
+  ; N32-DAG:    addiu   $[[T0:[0-9]+]], $sp, 512
+  ; N64-DAG:    daddiu  $[[T0:[0-9]+]], $sp, 512
+  ; GP64-DAG:   sd      $[[T0]], 0($sp)
+  ; GP64-DAG:   ld      $[[T1:[0-9]+]], 1024($fp)
+  ; GP64-DAG:   sd      $[[T1]], 8($sp)
+
+  %a = alloca i32, align 512
+  call void @helper_04(i32 0, i32 0, i32 0, i32 0,
+                       i32 0, i32 0, i32 0, i32 0, i32* %a, i32* %b)
+  ret void
+}
+
+; Check dynamic stack realignment in functions with variable-sized objects.
+
+; O32 ABI
+define void @func_05(i32 %sz) {
+entry:
+; GP32-LABEL: func_05:
+
+  ; prologue
+  ; GP32:       addiu   $sp, $sp, -1024
+  ; GP32:       sw      $fp, 1020($sp)
+  ; GP32:       sw      $23, 1016($sp)
+  ;
+  ; GP32:       move    $fp, $sp
+  ; GP32:       addiu   $[[T0:[0-9]+|gp]], $zero, -512
+  ; GP32-NEXT:  and     $sp, $sp, $[[T0]]
+  ; GP32-NEXT:  move    $23, $sp
+
+  ; body
+  ; GP32:       addiu   $[[T1:[0-9]+]], $zero, 222
+  ; GP32:       sw      $[[T1]], 508($23)
+
+  ; epilogue
+  ; GP32:       move    $sp, $fp
+  ; GP32:       lw      $23, 1016($sp)
+  ; GP32:       lw      $fp, 1020($sp)
+  ; GP32:       addiu   $sp, $sp, 1024
+
+  %a0 = alloca i32, i32 %sz, align 512
+  %a1 = alloca i32, align 4
+
+  store volatile i32 111, i32* %a0, align 512
+  store volatile i32 222, i32* %a1, align 4
+
+  ret void
+}
+
+; N32/N64 ABIs
+define void @func_06(i32 %sz) {
+entry:
+; GP64-LABEL: func_06:
+
+  ; prologue
+  ; N32:        addiu   $sp, $sp, -1024
+  ; N64:        daddiu  $sp, $sp, -1024
+  ; GP64:       sd      $fp, 1016($sp)
+  ; GP64:       sd      $23, 1008($sp)
+  ;
+  ; GP64:       move    $fp, $sp
+  ; GP64:       addiu   $[[T0:[0-9]+|gp]], $zero, -512
+  ; GP64-NEXT:  and     $sp, $sp, $[[T0]]
+  ; GP64-NEXT:  move    $23, $sp
+
+  ; body
+  ; GP64:       addiu   $[[T1:[0-9]+]], $zero, 222
+  ; GP64:       sw      $[[T1]], 508($23)
+
+  ; epilogue
+  ; GP64:       move    $sp, $fp
+  ; GP64:       ld      $23, 1008($sp)
+  ; GP64:       ld      $fp, 1016($sp)
+  ; N32:        addiu   $sp, $sp, 1024
+  ; N64:        daddiu  $sp, $sp, 1024
+
+  %a0 = alloca i32, i32 %sz, align 512
+  %a1 = alloca i32, align 4
+
+  store volatile i32 111, i32* %a0, align 512
+  store volatile i32 222, i32* %a1, align 4
+
+  ret void
+}
+
+; Verify that we use $fp for referencing incoming arguments and $sp for
+; building outbound arguments for nested function calls.
+
+; O32 ABI
+define void @func_07(i32 %p0, i32 %p1, i32 %p2, i32 %p3, i32 %sz) {
+entry:
+; GP32-LABEL: func_07:
+
+  ; body
+  ; GP32-DAG:       lw      $[[T0:[0-9]+]], 1040($fp)
+  ;
+  ; GP32-DAG:       addiu   $[[T1:[0-9]+]], $zero, 222
+  ; GP32-DAG:       sw      $[[T1]], 508($23)
+  ;
+  ; GP32-DAG:       sw      $[[T2:[0-9]+]], 16($sp)
+
+  %a0 = alloca i32, i32 %sz, align 512
+  %a1 = alloca i32, align 4
+
+  store volatile i32 111, i32* %a0, align 512
+  store volatile i32 222, i32* %a1, align 4
+
+  call void @helper_01(i32 0, i32 0, i32 0, i32 0, i32* %a1)
+
+  ret void
+}
+
+; N32/N64 ABIs
+define void @func_08(i32 %p0, i32 %p1, i32 %p2, i32 %p3,
+                     i32 %p4, i32 %p5, i32 %p6, i32 %p7,
+                     i32 %sz) {
+entry:
+; GP64-LABEL: func_08:
+
+  ; body
+  ; N32-DAG:        lw      $[[T0:[0-9]+]], 1028($fp)
+  ; N64-DAG:        lwu     $[[T0:[0-9]+]], 1028($fp)
+  ;
+  ; GP64-DAG:       addiu   $[[T1:[0-9]+]], $zero, 222
+  ; GP64-DAG:       sw      $[[T1]], 508($23)
+  ;
+  ; GP64-DAG:       sd      $[[T2:[0-9]+]], 0($sp)
+
+  %a0 = alloca i32, i32 %sz, align 512
+  %a1 = alloca i32, align 4
+
+  store volatile i32 111, i32* %a0, align 512
+  store volatile i32 222, i32* %a1, align 4
+
+  call void @helper_02(i32 0, i32 0, i32 0, i32 0,
+                       i32 0, i32 0, i32 0, i32 0, i32* %a1)
+  ret void
+}
+
+; Check that we do not perform dynamic stack realignment in the presence of
+; the "no-realign-stack" function attribute.
+define void @func_09() "no-realign-stack" {
+entry:
+; ALL-LABEL: func_09:
+
+  ; ALL-NOT:  and     $sp, $sp, $[[T0:[0-9]+|ra|gp]]
+
+  %a = alloca i32, align 512
+  call void @helper_01(i32 0, i32 0, i32 0, i32 0, i32* %a)
+  ret void
+}
+
+define void @func_10(i32 %sz) "no-realign-stack" {
+entry:
+; ALL-LABEL: func_10:
+
+  ; ALL-NOT:  and     $sp, $sp, $[[T0:[0-9]+|ra|gp]]
+
+  %a0 = alloca i32, i32 %sz, align 512
+  %a1 = alloca i32, align 4
+
+  store volatile i32 111, i32* %a0, align 512
+  store volatile i32 222, i32* %a1, align 4
+
+  ret void
+}