[AArch64] Add support for dynamically aligning stack objects

[kristof.beyls]

Summary

This patch adds support for dynamically aligning stack objects for AArch64 targets.

[t.p.northover]

Hi Kristof,

Thanks for working on this, it's a good feature to have. Just a couple of suggestions:

lib/Target/AArch64/AArch64FrameLowering.cpp

367 ↗

(On Diff #23355)

What do we do if NumBytes == 0 but we need realignment? I'm OK with not handling that case (Clang won't produce an "alignstack=BIG" attribute for AArch64), but we should probably assert that it can't happen somewhere.

374 ↗

(On Diff #23355)

Any particular reason for using BFI here? I'd have thought AND would be the more natural choice: it only has one register dependency (even if the XZR is a false dependency here) and can directly write to SP (avoiding the copy).

Slightly more difficult to encode, of course.

Either way, could we assert that ScratchSPReg is not SP here: neither instruction permits SP as an input, but refactoring could easily mess up that invariant later on.

[t.p.northover]

Oh, and particularly thanks for those comments at the top! Very useful.

[kristof.beyls]

Thanks for the quick review and nice catches.
It's indeed better to use AND instead of BFI; the patch now does this. It also makes some of the logic a bit simpler - but doesn't reduce the number of instructions needed when a base pointer is also needed, which is when there are both VLAs and dynamically aligned local variables.
I've also added the asserts you indicated - indeed they are good to have.

[t.p.northover]

Thanks for the updated patch Kristof. I think this is fine now.

Tim.

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.h

 #include "llvm/Target/TargetFrameLowering.h"
 
 namespace llvm {
 
 class AArch64FrameLowering : public TargetFrameLowering {
 public:
   explicit AArch64FrameLowering()
       : TargetFrameLowering(StackGrowsDown, 16, 0, 16,
-                            false /*StackRealignable*/) {}
+                            true /*StackRealignable*/) {}
 
   void emitCalleeSavedFrameMoves(MachineBasicBlock &MBB,
                                  MachineBasicBlock::iterator MBBI,
                                  unsigned FramePtr) const;
 
   void eliminateCallFramePseudoInstr(MachineFunction &MF,
                                   MachineBasicBlock &MBB,
                                   MachineBasicBlock::iterator I) const override;

llvm/trunk/lib/Target/AArch64/AArch64FrameLowering.cpp

 //===- AArch64FrameLowering.cpp - AArch64 Frame Lowering -------*- C++ -*-====//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 // This file contains the AArch64 implementation of TargetFrameLowering class.
 //
+// On AArch64, stack frames are structured as follows:
+//
+// The stack grows downward.
+//
+// All of the individual frame areas on the frame below are optional, i.e. it's
+// possible to create a function so that the particular area isn't present
+// in the frame.
+//
+// At function entry, the "frame" looks as follows:
+//
+// |                                   | Higher address
+// |-----------------------------------|
+// |                                   |
+// | arguments passed on the stack     |
+// |                                   |
+// |-----------------------------------| <- sp
+// |                                   | Lower address
+//
+//
+// After the prologue has run, the frame has the following general structure.
+// Note that this doesn't depict the case where a red-zone is used. Also,
+// technically the last frame area (VLAs) doesn't get created until in the
+// main function body, after the prologue is run. However, it's depicted here
+// for completeness.
+//
+// |                                   | Higher address
+// |-----------------------------------|
+// |                                   |
+// | arguments passed on the stack     |
+// |                                   |
+// |-----------------------------------|
+// |                                   |
+// | prev_fp, prev_lr                  |
+// | (a.k.a. "frame record")           |
+// |-----------------------------------| <- fp(=x29)
+// |                                   |
+// | other callee-saved registers      |
+// |                                   |
+// |-----------------------------------|
+// |.empty.space.to.make.part.below....|
+// |.aligned.in.case.it.needs.more.than| (size of this area is unknown at
+// |.the.standard.16-byte.alignment....|  compile time; if present)
+// |-----------------------------------|
+// |                                   |
+// | local variables of fixed size     |
+// | including spill slots             |
+// |-----------------------------------| <- bp(not defined by ABI,
+// |.variable-sized.local.variables....|       LLVM chooses X19)
+// |.(VLAs)............................| (size of this area is unknown at
+// |...................................|  compile time)
+// |-----------------------------------| <- sp
+// |                                   | Lower address
+//
+//
+// To access the data in a frame, at-compile time, a constant offset must be
+// computable from one of the pointers (fp, bp, sp) to access it. The size
+// of the areas with a dotted background cannot be computed at compile-time
+// if they are present, making it required to have all three of fp, bp and
+// sp to be set up to be able to access all contents in the frame areas,
+// assuming all of the frame areas are non-empty.
+//
+// For most functions, some of the frame areas are empty. For those functions,
+// it may not be necessary to set up fp or bp:
+// * A base pointer is definitly needed when there are both VLAs and local
+//   variables with more-than-default alignment requirements.
+// * A frame pointer is definitly needed when there are local variables with
+//   more-than-default alignment requirements.
+//
+// In some cases when a base pointer is not strictly needed, it is generated
+// anyway when offsets from the frame pointer to access local variables become
+// so large that the offset can't be encoded in the immediate fields of loads
+// or stores.
+//
+// FIXME: also explain the redzone concept.
+// FIXME: also explain the concept of reserved call frames.
+//
 //===----------------------------------------------------------------------===//
 
 #include "AArch64FrameLowering.h"
 #include "AArch64InstrInfo.h"
 #include "AArch64MachineFunctionInfo.h"
 #include "AArch64Subtarget.h"
 #include "AArch64TargetMachine.h"
 #include "llvm/ADT/Statistic.h"

 #define DEBUG_TYPE "frame-info"
 
 static cl::opt<bool> EnableRedZone("aarch64-redzone",
                                    cl::desc("enable use of redzone on AArch64"),
                                    cl::init(false), cl::Hidden);
 
 STATISTIC(NumRedZoneFunctions, "Number of functions using red zone");
 
-static unsigned estimateStackSize(MachineFunction &MF) {
-  const MachineFrameInfo *FFI = MF.getFrameInfo();
-  int Offset = 0;
-  for (int i = FFI->getObjectIndexBegin(); i != 0; ++i) {
-    int FixedOff = -FFI->getObjectOffset(i);
-    if (FixedOff > Offset)
-      Offset = FixedOff;
-  }
-  for (unsigned i = 0, e = FFI->getObjectIndexEnd(); i != e; ++i) {
-    if (FFI->isDeadObjectIndex(i))
-      continue;
-    Offset += FFI->getObjectSize(i);
-    unsigned Align = FFI->getObjectAlignment(i);
-    // Adjust to alignment boundary
-    Offset = (Offset + Align - 1) / Align * Align;
-  }
-  // This does not include the 16 bytes used for fp and lr.
-  return (unsigned)Offset;
-}
-
 bool AArch64FrameLowering::canUseRedZone(const MachineFunction &MF) const {
   if (!EnableRedZone)
     return false;
   // Don't use the red zone if the function explicitly asks us not to.
   // This is typically used for kernel code.
   if (MF.getFunction()->hasFnAttribute(Attribute::NoRedZone))
     return false;
 
   const MachineFrameInfo *MFI = MF.getFrameInfo();
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   unsigned NumBytes = AFI->getLocalStackSize();
 
   // Note: currently hasFP() is always true for hasCalls(), but that's an
   // implementation detail of the current code, not a strict requirement,
   // so stay safe here and check both.
   if (MFI->hasCalls() || hasFP(MF) || NumBytes > 128)
     return false;
   return true;
 }
 
 /// hasFP - Return true if the specified function should have a dedicated frame
 /// pointer register.
 bool AArch64FrameLowering::hasFP(const MachineFunction &MF) const {
   const MachineFrameInfo *MFI = MF.getFrameInfo();
-
-#ifndef NDEBUG
   const TargetRegisterInfo *RegInfo = MF.getSubtarget().getRegisterInfo();
-  assert(!RegInfo->needsStackRealignment(MF) &&
-         "No stack realignment on AArch64!");
-#endif
-
   return (MFI->hasCalls() || MFI->hasVarSizedObjects() ||
           MFI->isFrameAddressTaken() || MFI->hasStackMap() ||
-          MFI->hasPatchPoint());
+          MFI->hasPatchPoint() || RegInfo->needsStackRealignment(MF));
 }
 
 /// hasReservedCallFrame - Under normal circumstances, when a frame pointer is
 /// not required, we reserve argument space for call sites in the function
 /// immediately on entry to the current function.  This eliminates the need for
 /// add/sub sp brackets around call sites.  Returns true if the call frame is
 /// included as part of the stack frame.
 bool
if (HasFP) {
     emitFrameOffset(MBB, MBBI, DL, AArch64::FP, AArch64::SP, FPOffset, TII,
                     MachineInstr::FrameSetup);
   }
 
   // All of the remaining stack allocations are for locals.
   AFI->setLocalStackSize(NumBytes);
 
   // Allocate space for the rest of the frame.
-  if (NumBytes) {
-    // If we're a leaf function, try using the red zone.
-    if (!canUseRedZone(MF))
-      emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
-                      MachineInstr::FrameSetup);
+
+  const unsigned Alignment = MFI->getMaxAlignment();
+  const bool NeedsRealignment = (Alignment > 16);
+  unsigned scratchSPReg = AArch64::SP;
+  if (NeedsRealignment) {
+    // Use the first callee-saved register as a scratch register
+    assert(MF.getRegInfo().isPhysRegUsed(AArch64::X9) &&
+           "No scratch register to align SP!");
+    scratchSPReg = AArch64::X9;
+  }
+
+  if (NumBytes && !canUseRedZone(MF))
+    // FIXME: in the case of dynamic re-alignment, NumBytes doesn't have
+    // the correct value here, as NumBytes also includes padding bytes,
+    // which shouldn't be counted here.
+    emitFrameOffset(MBB, MBBI, DL, scratchSPReg, AArch64::SP, -NumBytes, TII,
+
+  assert(!(NeedsRealignment && NumBytes==0) &&
+         "NumBytes should never be 0 when realignment is needed");
+
+  if (NumBytes && NeedsRealignment) {
+    const unsigned NrBitsToZero = countTrailingZeros(Alignment);
+    assert(NrBitsToZero > 1);
+    assert(scratchSPReg != AArch64::SP);
+
+    // SUB X9, SP, NumBytes
+    //   -- X9 is temporary register, so shouldn't contain any live data here,
+    //   -- free to use. This is already produced by emitFrameOffset above.
+    // AND SP, X9, 0b11111...0000
+    // The logical immediates have a non-trivial encoding. The following
+    // formula computes the encoded immediate with all ones but
+    // NrBitsToZero zero bits as least significant bits.
+    uint32_t andMaskEncoded =
+        (1                   <<12) // = N
+      | ((64-NrBitsToZero)   << 6) // immr
+      | ((64-NrBitsToZero-1) << 0) // imms
+      ;
+    BuildMI(MBB, MBBI, DL, TII->get(AArch64::ANDXri), AArch64::SP)
+      .addReg(scratchSPReg, RegState::Kill)
+      .addImm(andMaskEncoded);
   }
 
   // If we need a base pointer, set it up here. It's whatever the value of the
   // stack pointer is at this point. Any variable size objects will be allocated
   // after this, so we can still use the base pointer to reference locals.
   //
   // FIXME: Clarify FrameSetup flags here.
   // Note: Use emitFrameOffset() like above for FP if the FrameSetup flag is
   // needed.
-  //
-  if (RegInfo->hasBasePointer(MF))
-    TII->copyPhysReg(MBB, MBBI, DL, AArch64::X19, AArch64::SP, false);
+  if (RegInfo->hasBasePointer(MF)) {
+    TII->copyPhysReg(MBB, MBBI, DL, RegInfo->getBaseRegister(), AArch64::SP,
+                     false);
+  }
 
   if (needsFrameMoves) {
     const DataLayout *TD = MF.getTarget().getDataLayout();
     const int StackGrowth = -TD->getPointerSize(0);
     unsigned FramePtr = RegInfo->getFrameRegister(MF);
-
     // An example of the prologue:
     //
     //     .globl __foo
     //     .align 2
     //  __foo:
     // Ltmp0:
     //     .cfi_startproc
     //     .cfi_personality 155, ___gxx_personality_v0
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   int NumBytes = MFI->getStackSize();
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
 
   // All calls are tail calls in GHC calling conv, and functions have no
   // prologue/epilogue.
   if (MF.getFunction()->getCallingConv() == CallingConv::GHC)
     return;
 
-  // Initial and residual are named for consitency with the prologue. Note that
+  // Initial and residual are named for consistency with the prologue. Note that
   // in the epilogue, the residual adjustment is executed first.
   uint64_t ArgumentPopSize = 0;
   if (RetOpcode == AArch64::TCRETURNdi || RetOpcode == AArch64::TCRETURNri) {
     MachineOperand &StackAdjust = MBBI->getOperand(1);
 
     // For a tail-call in a callee-pops-arguments environment, some or all of
     // the stack may actually be in use for the call's arguments, this is
     // calculated during LowerCall and consumed here...
int AArch64FrameLowering::resolveFrameIndexReference(const MachineFunction &MF,
   const AArch64RegisterInfo *RegInfo = static_cast<const AArch64RegisterInfo *>(
       MF.getSubtarget().getRegisterInfo());
   const AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
   int FPOffset = MFI->getObjectOffset(FI) + 16;
   int Offset = MFI->getObjectOffset(FI) + MFI->getStackSize();
   bool isFixed = MFI->isFixedObjectIndex(FI);
 
   // Use frame pointer to reference fixed objects. Use it for locals if
-  // there are VLAs (and thus the SP isn't reliable as a base).
-  // Make sure useFPForScavengingIndex() does the right thing for the emergency
-  // spill slot.
+  // there are VLAs or a dynamically realigned SP (and thus the SP isn't
+  // reliable as a base). Make sure useFPForScavengingIndex() does the
+  // right thing for the emergency spill slot.
   bool UseFP = false;
   if (AFI->hasStackFrame()) {
     // Note: Keeping the following as multiple 'if' statements rather than
     // merging to a single expression for readability.
     //
     // Argument access should always use the FP.
     if (isFixed) {
       UseFP = hasFP(MF);
-    } else if (hasFP(MF) && !RegInfo->hasBasePointer(MF)) {
+    } else if (hasFP(MF) && !RegInfo->hasBasePointer(MF) &&
+               !RegInfo->needsStackRealignment(MF)) {
       // Use SP or FP, whichever gives us the best chance of the offset
       // being in range for direct access. If the FPOffset is positive,
       // that'll always be best, as the SP will be even further away.
       // If the FPOffset is negative, we have to keep in mind that the
       // available offset range for negative offsets is smaller than for
       // positive ones. If we have variable sized objects, we're stuck with
       // using the FP regardless, though, as the SP offset is unknown
       // and we don't have a base pointer available. If an offset is
       // available via the FP and the SP, use whichever is closest.
       if (PreferFP || MFI->hasVarSizedObjects() || FPOffset >= 0 ||
           (FPOffset >= -256 && Offset > -FPOffset))
         UseFP = true;
     }
   }
 
+  assert((isFixed || !RegInfo->needsStackRealignment(MF) || !UseFP) &&
+         "In the presence of dynamic stack pointer realignment, "
+         "non-argument objects cannot be accessed through the frame pointer");
+
   if (UseFP) {
     FrameReg = RegInfo->getFrameRegister(MF);
     return FPOffset;
   }
 
   // Use the base pointer if we have one.
   if (RegInfo->hasBasePointer(MF))
     FrameReg = RegInfo->getBaseRegister();
if (hasFP(MF)) {
     MRI->setPhysRegUsed(AArch64::LR);
   }
 
   // Spill the BasePtr if it's used. Do this first thing so that the
   // getCalleeSavedRegs() below will get the right answer.
   if (RegInfo->hasBasePointer(MF))
     MRI->setPhysRegUsed(RegInfo->getBaseRegister());
 
+  if (RegInfo->needsStackRealignment(MF) && !RegInfo->hasBasePointer(MF))
+    MRI->setPhysRegUsed(AArch64::X9);
+
   // If any callee-saved registers are used, the frame cannot be eliminated.
   unsigned NumGPRSpilled = 0;
   unsigned NumFPRSpilled = 0;
   bool ExtraCSSpill = false;
   bool CanEliminateFrame = true;
   DEBUG(dbgs() << "*** processFunctionBeforeCalleeSavedScan\nUsed CSRs:");
   const MCPhysReg *CSRegs = RegInfo->getCalleeSavedRegs(&MF);
 
void AArch64FrameLowering::processFunctionBeforeCalleeSavedScan(
 
   // FIXME: Set BigStack if any stack slot references may be out of range.
   // For now, just conservatively guestimate based on unscaled indexing
   // range. We'll end up allocating an unnecessary spill slot a lot, but
   // realistically that's not a big deal at this stage of the game.
   // The CSR spill slots have not been allocated yet, so estimateStackSize
   // won't include them.
   MachineFrameInfo *MFI = MF.getFrameInfo();
-  unsigned CFSize = estimateStackSize(MF) + 8 * (NumGPRSpilled + NumFPRSpilled);
+  unsigned CFSize =
+      MFI->estimateStackSize(MF) + 8 * (NumGPRSpilled + NumFPRSpilled);
   DEBUG(dbgs() << "Estimated stack frame size: " << CFSize << " bytes.\n");
   bool BigStack = (CFSize >= 256);
   if (BigStack || !CanEliminateFrame || RegInfo->cannotEliminateFrame(MF))
     AFI->setHasStackFrame(true);
 
   // Estimate if we might need to scavenge a register at some point in order
   // to materialize a stack offset. If so, either spill one additional
   // callee-saved register or reserve a special spill slot to facilitate

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.h

public:
   bool hasBasePointer(const MachineFunction &MF) const;
   unsigned getBaseRegister() const;
 
   // Debug information queries.
   unsigned getFrameRegister(const MachineFunction &MF) const override;
 
   unsigned getRegPressureLimit(const TargetRegisterClass *RC,
                                MachineFunction &MF) const override;
+  // Base pointer (stack realignment) support.
+  bool canRealignStack(const MachineFunction &MF) const;
+  bool needsStackRealignment(const MachineFunction &MF) const override;
 };
 
 } // end namespace llvm
 
 #endif

llvm/trunk/lib/Target/AArch64/AArch64RegisterInfo.cpp

 
 bool AArch64RegisterInfo::hasBasePointer(const MachineFunction &MF) const {
   const MachineFrameInfo *MFI = MF.getFrameInfo();
 
   // In the presence of variable sized objects, if the fixed stack size is
   // large enough that referencing from the FP won't result in things being
   // in range relatively often, we can use a base pointer to allow access
   // from the other direction like the SP normally works.
+  // Furthermore, if both variable sized objects are present, and the
+  // stack needs to be dynamically re-aligned, the base pointer is the only
+  // reliable way to reference the locals.
   if (MFI->hasVarSizedObjects()) {
+    if (needsStackRealignment(MF))
+      return true;
     // Conservatively estimate whether the negative offset from the frame
     // pointer will be sufficient to reach. If a function has a smallish
     // frame, it's less likely to have lots of spills and callee saved
     // space, so it's all more likely to be within range of the frame pointer.
     // If it's wrong, we'll materialize the constant and still get to the
     // object; it's just suboptimal. Negative offsets use the unscaled
     // load/store instructions, which have a 9-bit signed immediate.
     if (MFI->getLocalFrameSize() < 256)
       return false;
     return true;
   }
 
   return false;
 }
 
+bool AArch64RegisterInfo::canRealignStack(const MachineFunction &MF) const {
+
+  if (MF.getFunction()->hasFnAttribute("no-realign-stack"))
+    return false;
+
+  return true;
+}
+
+// FIXME: share this with other backends with identical implementation?
+bool
+AArch64RegisterInfo::needsStackRealignment(const MachineFunction &MF) const {
+  const MachineFrameInfo *MFI = MF.getFrameInfo();
+  const Function *F = MF.getFunction();
+  unsigned StackAlign = MF.getTarget()
+                            .getSubtargetImpl(*MF.getFunction())
+                            ->getFrameLowering()
+                            ->getStackAlignment();
+  bool requiresRealignment =
+      ((MFI->getMaxAlignment() > StackAlign) ||
+       F->getAttributes().hasAttribute(AttributeSet::FunctionIndex,
+                                       Attribute::StackAlignment));
+
+  return requiresRealignment && canRealignStack(MF);
+}
+
 unsigned
 AArch64RegisterInfo::getFrameRegister(const MachineFunction &MF) const {
   const TargetFrameLowering *TFI = MF.getSubtarget().getFrameLowering();
 
   return TFI->hasFP(MF) ? AArch64::FP : AArch64::SP;
 }
 
 bool AArch64RegisterInfo::requiresRegisterScavenging(

llvm/trunk/test/CodeGen/AArch64/aarch64-dynamic-stack-layout.ll

+; RUN: llc -verify-machineinstrs -mtriple=aarch64-none-linux-gnu < %s | FileCheck %s
+
+; This test aims to check basic correctness of frame layout &
+; frame access code. There are 8 functions in this test file,
+; each function implements one element in the cartesian product
+; of:
+; . a function having a VLA/noVLA
+; . a function with dynamic stack realignment/no dynamic stack realignment.
+; . a function needing a frame pionter/no frame pointer,
+; since the presence/absence of these has influence on the frame
+; layout and which pointer to use to access various part of the
+; frame (bp,sp,fp).
+;
+; Furthermore: in every test function:
+; . there is always one integer and 1 floating point argument to be able
+;   to check those are accessed correctly.
+; . there is always one local variable to check that is accessed
+;   correctly
+;
+; The LLVM-IR below was produced by clang on the following C++ code:
+;extern "C" int g();
+;extern "C" int novla_nodynamicrealign_call(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                             double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  volatile int l1;
+;  return i10 + (int)d10 + l1 + g();
+;}
+;extern "C" int novla_nodynamicrealign_nocall(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                             double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  volatile int l1;
+;  return i10 + (int)d10 + l1;
+;}
+;extern "C" int novla_dynamicrealign_call(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                         double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  alignas(128) volatile int l1;
+;  return i10 + (int)d10 + l1 + g();
+;}
+;extern "C" int novla_dynamicrealign_nocall(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                           double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  alignas(128) volatile int l1;
+;  return i10 + (int)d10 + l1;
+;}
+;
+;extern "C" int vla_nodynamicrealign_call(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                         double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  volatile int l1;
+;  volatile int vla[i1];
+;  return i10 + (int)d10 + l1 + g() + vla[0];
+;}
+;extern "C" int vla_nodynamicrealign_nocall(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                           double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  volatile int l1;
+;  volatile int vla[i1];
+;  return i10 + (int)d10 + l1 + vla[0];
+;}
+;extern "C" int vla_dynamicrealign_call(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                       double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  alignas(128) volatile int l1;
+;  volatile int vla[i1];
+;  return i10 + (int)d10 + l1 + g() + vla[0];
+;}
+;extern "C" int vla_dynamicrealign_nocall(int i1, int i2, int i3, int i4, int i5, int i6, int i7, int i8, int i9, int i10,
+;                                         double d1, double d2, double d3, double d4, double d5, double d6, double d7, double d8, double d9, double d10)
+;{
+;  // use an argument passed on the stack.
+;  alignas(128) volatile int l1;
+;  volatile int vla[i1];
+;  return i10 + (int)d10 + l1 + vla[0];
+;}
+
+
+
+define i32 @novla_nodynamicrealign_call(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #0 {
+entry:
+  %l1 = alloca i32, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 4
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %call = tail call i32 @g()
+  %add2 = add nsw i32 %add1, %call
+  ret i32 %add2
+}
+; CHECK-LABEL: novla_nodynamicrealign_call
+; CHECK: .cfi_startproc
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x20, x19, [sp, #-32]!
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #16]
+; CHECK: add  x29, sp, #16
+;   Check correctness of cfi pseudo-instructions
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w19, -24
+; CHECK: .cfi_offset w20, -32
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+;   Check correct access to local variable on the stack, through stack pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [sp, #12]
+;   Check epilogue:
+; CHECK: ldp  x29, x30, [sp, #16]
+; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: ret
+; CHECK: .cfi_endproc
+
+
+declare i32 @g() #0
+
+; Function Attrs: nounwind
+define i32 @novla_nodynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
+entry:
+  %l1 = alloca i32, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 4
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  ret i32 %add1
+}
+; CHECK-LABEL: novla_nodynamicrealign_nocall
+;   Check that space is reserved for one local variable on the stack.
+; CHECK:  sub sp, sp, #16             // =16
+;   Check correct access to arguments passed on the stack, through stack pointer
+; CHECK: ldr  d[[DARG:[0-9]+]], [sp, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [sp, #24]
+;   Check correct access to local variable on the stack, through stack pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [sp, #12]
+;   Check epilogue:
+; CHECK: add  sp, sp, #16             // =16
+; CHECK: ret
+
+
+define i32 @novla_dynamicrealign_call(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #0 {
+entry:
+  %l1 = alloca i32, align 128
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 128
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %call = tail call i32 @g()
+  %add2 = add nsw i32 %add1, %call
+  ret i32 %add2
+}
+
+; CHECK-LABEL: novla_dynamicrealign_call
+; CHECK: .cfi_startproc
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x20, x19, [sp, #-32]!
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #16]
+; CHECK: add  x29, sp, #16
+;   Check the dynamic realignment of the stack pointer to a 128-byte boundary
+; CHECK: sub  x9, sp, #96
+; CHECK: and  sp, x9, #0xffffffffffffff80
+;   Check correctness of cfi pseudo-instructions
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w19, -24
+; CHECK: .cfi_offset w20, -32
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+;   Check correct access to local variable on the stack, through re-aligned stack pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [sp]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: sub  sp, x29, #16            // =16
+; CHECK: ldp  x29, x30, [sp, #16]
+; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: ret
+; CHECK: .cfi_endproc
+
+
+; Function Attrs: nounwind
+define i32 @novla_dynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
+entry:
+  %l1 = alloca i32, align 128
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 128
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  ret i32 %add1
+}
+
+; CHECK-LABEL: novla_dynamicrealign_nocall
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #-16]!
+; CHECK: mov  x29, sp
+;   Check the dynamic realignment of the stack pointer to a 128-byte boundary
+; CHECK: sub  x9, sp, #112
+; CHECK: and  sp, x9, #0xffffffffffffff80
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+;   Check correct access to local variable on the stack, through re-aligned stack pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [sp]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: mov  sp, x29
+; CHECK: ldp  x29, x30, [sp], #16
+; CHECK: ret
+
+
+define i32 @vla_nodynamicrealign_call(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #0 {
+entry:
+  %l1 = alloca i32, align 4
+  %0 = zext i32 %i1 to i64
+  %vla = alloca i32, i64 %0, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 4
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %call = tail call i32 @g()
+  %add2 = add nsw i32 %add1, %call
+  %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
+  %add3 = add nsw i32 %add2, %1
+  ret i32 %add3
+}
+
+; CHECK-LABEL: vla_nodynamicrealign_call
+; CHECK: .cfi_startproc
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x20, x19, [sp, #-32]!
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #16]
+; CHECK: add  x29, sp, #16
+;   Check that space is reserved on the stack for the local variable,
+;   rounded up to a multiple of 16 to keep the stack pointer 16-byte aligned.
+; CHECK: sub  sp, sp, #16
+;   Check correctness of cfi pseudo-instructions
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w19, -24
+; CHECK: .cfi_offset w20, -32
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
+; CHECK: ubfx x9, x0, #0, #32
+; CHECK: lsl  x9, x9, #2
+; CHECK: add  x9, x9, #15
+; CHECK: and  x9, x9, #0xfffffffffffffff0
+; CHECK: mov   x10, sp
+; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
+; CHECK: mov   sp, x[[VLASPTMP]]
+;   Check correct access to local variable, through frame pointer
+; CHECK: ldur w[[ILOC:[0-9]+]], [x29, #-20]
+;   Check correct accessing of the VLA variable through the base pointer
+; CHECK: ldr  w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: sub  sp, x29, #16            // =16
+; CHECK: ldp  x29, x30, [sp, #16]
+; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: ret
+; CHECK: .cfi_endproc
+
+
+; Function Attrs: nounwind
+define i32 @vla_nodynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
+entry:
+  %l1 = alloca i32, align 4
+  %0 = zext i32 %i1 to i64
+  %vla = alloca i32, i64 %0, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 4
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
+  %add2 = add nsw i32 %add1, %1
+  ret i32 %add2
+}
+
+; CHECK-LABEL: vla_nodynamicrealign_nocall
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #-16]!
+; CHECK: mov  x29, sp
+;   Check that space is reserved on the stack for the local variable,
+;   rounded up to a multiple of 16 to keep the stack pointer 16-byte aligned.
+; CHECK: sub  sp, sp, #16
+;   Check correctness of cfi pseudo-instructions
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
+; CHECK: ubfx x9, x0, #0, #32
+; CHECK: lsl  x9, x9, #2
+; CHECK: add  x9, x9, #15
+; CHECK: and  x9, x9, #0xfffffffffffffff0
+; CHECK: mov   x10, sp
+; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
+; CHECK: mov   sp, x[[VLASPTMP]]
+;   Check correct access to local variable, through frame pointer
+; CHECK: ldur w[[ILOC:[0-9]+]], [x29, #-4]
+;   Check correct accessing of the VLA variable through the base pointer
+; CHECK: ldr  w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: mov    sp, x29
+; CHECK: ldp  x29, x30, [sp], #16
+; CHECK: ret
+
+
+define i32 @vla_dynamicrealign_call(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #0 {
+entry:
+  %l1 = alloca i32, align 128
+  %0 = zext i32 %i1 to i64
+  %vla = alloca i32, i64 %0, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 128
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %call = tail call i32 @g()
+  %add2 = add nsw i32 %add1, %call
+  %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
+  %add3 = add nsw i32 %add2, %1
+  ret i32 %add3
+}
+
+; CHECK-LABEL: vla_dynamicrealign_call
+; CHECK: .cfi_startproc
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x22, x21, [sp, #-48]!
+; CHECK: stp  x20, x19, [sp, #16]
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #32]
+; CHECK: add  x29, sp, #32
+;   Check that the stack pointer gets re-aligned to 128
+;   bytes & the base pointer (x19) gets initialized to
+;   this 128-byte aligned area for local variables &
+;   spill slots
+; CHECK: sub  x9, sp, #80            // =80
+; CHECK: and  sp, x9, #0xffffffffffffff80
+; CHECK: mov    x19, sp
+;   Check correctness of cfi pseudo-instructions
+; CHECK: .cfi_def_cfa w29, 16
+; CHECK: .cfi_offset w30, -8
+; CHECK: .cfi_offset w29, -16
+; CHECK: .cfi_offset w19, -24
+; CHECK: .cfi_offset w20, -32
+; CHECK: .cfi_offset w21, -40
+; CHECK: .cfi_offset w22, -48
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
+;   and set-up of base pointer (x19).
+; CHECK: ubfx x9, x0, #0, #32
+; CHECK: lsl  x9, x9, #2
+; CHECK: add  x9, x9, #15
+; CHECK: and  x9, x9, #0xfffffffffffffff0
+; CHECK: mov   x10, sp
+; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
+; CHECK: mov   sp, x[[VLASPTMP]]
+;   Check correct access to local variable, through base pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
+; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: sub  sp, x29, #32
+; CHECK: ldp  x29, x30, [sp, #32]
+; CHECK: ldp  x20, x19, [sp, #16]
+; CHECK: ldp  x22, x21, [sp], #48
+; CHECK: ret
+; CHECK: .cfi_endproc
+
+
+; Function Attrs: nounwind
+define i32 @vla_dynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
+entry:
+  %l1 = alloca i32, align 128
+  %0 = zext i32 %i1 to i64
+  %vla = alloca i32, i64 %0, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 128
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
+  %add2 = add nsw i32 %add1, %1
+  ret i32 %add2
+}
+
+; CHECK-LABEL: vla_dynamicrealign_nocall
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x20, x19, [sp, #-32]!
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #16]
+; CHECK: add  x29, sp, #16
+;   Check that the stack pointer gets re-aligned to 128
+;   bytes & the base pointer (x19) gets initialized to
+;   this 128-byte aligned area for local variables &
+;   spill slots
+; CHECK: sub  x9, sp, #96
+; CHECK: and  sp, x9, #0xffffffffffffff80
+; CHECK: mov    x19, sp
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
+;   and set-up of base pointer (x19).
+; CHECK: ubfx x9, x0, #0, #32
+; CHECK: lsl  x9, x9, #2
+; CHECK: add  x9, x9, #15
+; CHECK: and  x9, x9, #0xfffffffffffffff0
+; CHECK: mov   x10, sp
+; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
+; CHECK: mov   sp, x[[VLASPTMP]]
+;   Check correct access to local variable, through base pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
+; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: sub  sp, x29, #16
+; CHECK: ldp  x29, x30, [sp, #16]
+; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: ret
+
+
+; Function Attrs: nounwind
+define i32 @vla_dynamicrealign_nocall_large_align(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {
+entry:
+  %l1 = alloca i32, align 32768
+  %0 = zext i32 %i1 to i64
+  %vla = alloca i32, i64 %0, align 4
+  %conv = fptosi double %d10 to i32
+  %add = add nsw i32 %conv, %i10
+  %l1.0.l1.0. = load volatile i32, i32* %l1, align 32768
+  %add1 = add nsw i32 %add, %l1.0.l1.0.
+  %1 = load volatile i32, i32* %vla, align 4, !tbaa !1
+  %add2 = add nsw i32 %add1, %1
+  ret i32 %add2
+}
+
+; CHECK-LABEL: vla_dynamicrealign_nocall_large_align
+;   Check that used callee-saved registers are saved
+; CHECK: stp  x20, x19, [sp, #-32]!
+;   Check that the frame pointer is created:
+; CHECK: stp  x29, x30, [sp, #16]
+; CHECK: add  x29, sp, #16
+;   Check that the stack pointer gets re-aligned to 128
+;   bytes & the base pointer (x19) gets initialized to
+;   this 128-byte aligned area for local variables &
+;   spill slots
+; CHECK: sub  x9, sp, #7, lsl #12
+; CHECK: and  sp, x9, #0xffffffffffff8000
+; CHECK: mov    x19, sp
+;   Check correct access to arguments passed on the stack, through frame pointer
+; CHECK: ldr  w[[IARG:[0-9]+]], [x29, #24]
+; CHECK: ldr  d[[DARG:[0-9]+]], [x29, #40]
+;   Check correct reservation of 16-byte aligned VLA (size in w0) on stack
+;   and set-up of base pointer (x19).
+; CHECK: ubfx x9, x0, #0, #32
+; CHECK: lsl  x9, x9, #2
+; CHECK: add  x9, x9, #15
+; CHECK: and  x9, x9, #0xfffffffffffffff0
+; CHECK: mov   x10, sp
+; CHECK: sub   x[[VLASPTMP:[0-9]+]], x10, x9
+; CHECK: mov   sp, x[[VLASPTMP]]
+;   Check correct access to local variable, through base pointer
+; CHECK: ldr  w[[ILOC:[0-9]+]], [x19]
+; CHECK: ldr   w[[VLA:[0-9]+]], [x[[VLASPTMP]]]
+;   Check epilogue:
+;     Check that stack pointer get restored from frame pointer.
+; CHECK: sub  sp, x29, #16
+; CHECK: ldp  x29, x30, [sp, #16]
+; CHECK: ldp  x20, x19, [sp], #32
+; CHECK: ret
+
+attributes #0 = { "less-precise-fpmad"="false" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "stack-protector-buffer-size"="8" "unsafe-fp-math"="false" "use-soft-float"="false" }
+attributes #1 = { nounwind "less-precise-fpmad"="false" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "stack-protector-buffer-size"="8" "unsafe-fp-math"="false" "use-soft-float"="false" }
+
+!1 = !{!2, !2, i64 0}
+!2 = !{!"int", !3, i64 0}
+!3 = !{!"omnipotent char", !4, i64 0}
+!4 = !{!"Simple C/C++ TBAA"}