[AArch64] Combine callee-save and local stack SP adjustment instructions.

[gberry]

Summary

If a function needs to allocate both callee-save stack memory and local
stack memory, we currently decrement/increment the SP in two steps:
first for the callee-save area, and then for the local stack area. This
changes the code to allocate them both at once at the very beginning/end
of the function. This has a few benefits:

1. there is one fewer sub/add micro-op in the prologue/epilogue

2. stack usage is reduced in cases where both the callee-save area and

local stack area size are equal to 8 mod 16

3. the stack adjustment instructions act as a scheduling barrier, so

moving them to the very beginning/end of the function increases post-RA
scheduler's ability to move instructions (that only depend on argument
registers) before any of the callee-save stores

This change can cause an increase in instructions if the original local
stack SP decrement could be folded into the first store to the stack.
This occurs when the first local stack store is to stack offset 0. In
this case we are trading off one more sub instruction for one fewer sub
micro-op (along with benefits (2) and (3) above).

[gberry]

Ping?

[rengolin]

Hi Geoff,

I'm not familiar with this part of the code, so I'll leave the full review for Tim, but I think the idea is a good one, and the prologue/epilogue changes in the test look mostly good. Some inline comments, too.

cheers,
--renato

lib/Target/AArch64/AArch64FrameLowering.cpp

839

Can't you just make:

if (ForceSP)
  UseFP = false;
else
  ...

?

965–966

Would it be possible to get here and (size % 16 != 0)? If so, keeping the assert would be good.

[gberry]

lib/Target/AArch64/AArch64FrameLowering.cpp

839

I don't think that works in the case where isFixed is true.

965–966

These asserts have just moved to the BumpSP == true part of spillCalleeSavedRegisters/restoreCalleeSavedRegisters

[gberry]

Rebased

[rengolin]

lib/Target/AArch64/AArch64FrameLowering.cpp

839

Right, than maybe moving the assert inside "if (isFixed)" would guide people when it does hit?

965–966

Indeed.

[gberry]

Ping (#2)?

[gberry]

Ping (#3)?

[t.p.northover]

Hi Geoff,

Sorry it took so long to get back to you, I needed to do some serious
thinking about this one. Mostly it's OK, but I think this is
problematic:

unsigned EstNonCSStackSize = MFI->estimateStackSize(MF);
unsigned StackSize = EstNonCSStackSize + 8 * NumRegsSpilled;
if (EstNonCSStackSize != 0 &&

 // 512 is the maximum immediate for stp/ldp that will be used for
// callee-save save/restores
 StackSize < 512 &&

Using an estimate here leaves us open for a change later on, which is
catastrophic if it pushes the amount over the 512-byte limit. It's
rare that the estimate differs from the actual amount by more than 8
(the correctness threshold since the STP offset is actually where you
want the first register to go) but it can happen.

I've attached an example where I've tweaked this discrepancy into a
real failure.

Tim.

• simple.ll1 KBDownload

[gberry]

Hi Tim,

Thanks for the review, it was worth the wait :)
I recall thinking about this issue and convincing myself that it wasn't a problem, but I clearly don't know all the dark corners of this code. Do you think it is reasonable to try to enumerate the cases where this estimation might go wrong (even conservatively) to avoid this problem?

[gberry]

Hi Tim,

I believe the below change fixes the problem you identified. I'll upload a new version with this fix. I'm still looking into your suggestion to refactor the code to avoid the stack size estimation in determinCalleeSaves() by doing the stack bump combining in emitPrologue/emitEpilogue instead to isolate the logic a bit more.

• a/lib/Target/AArch64/AArch64FrameLowering.cpp

+++ b/lib/Target/AArch64/AArch64FrameLowering.cpp
@@ -1116,7 +1116,8 @@ void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,

// Check to see if we can combine the callee-save and local stack pointer
// adjustment into a single decrement/increment.
unsigned EstNonCSStackSize = MFI->estimateStackSize(MF);

• unsigned StackSize = EstNonCSStackSize + 8 * NumRegsSpilled;

+ unsigned StackSize =
+ alignTo(EstNonCSStackSize + 8 * NumRegsSpilled, getStackAlignment());

if (EstNonCSStackSize != 0 &&
    // 512 is the maximum immediate for stp/ldp that will be used for
    // callee-save save/restores

[gberry]

Update to fix bug Tim noticed in review

[t.p.northover]

I don't think that fixes it. I'm seeing discrepancies of up to 32 in the estimate even with that fix in place. Unfortunately this one's not playing well with llc so I don't have a reduced example, but I can help you reproduce it if you want (it's also from povray in the test-suite-externals).

Really, I think this shows just how misguided it is to try and use estimateStackSize in this way: neither of us really understands what it's doing and even if *we* did, future readers won't.

Cheers.

Tim.

[gberry]

Tim,

Thanks. Don't worry about debugging that failure further, I mostly just wanted to capture that fix in case the alternate strategy of doing the SP bump combining in emitPrologue/emitEpilogue didn't work out. If that turns out to be the case, we can revisit, but hopefully it will work out and be an easier to understand change.

[gberry]

Update change to take the approach suggested by Tim: wait until
emitPrologue/emitEpilogue to combine SP stack bumps since we know the
final stack size then.

[t.p.northover]

lib/Target/AArch64/AArch64FrameLowering.cpp

336

I think we're duplicating effort here. If the spill/restore functions always produce non-incremented versions we have fewer cases to cover here and those functions will be simpler too.

412

According to AAPCS, SP must equal 0 (mod 16) whenever it's used as the base of a load/store. Given the risk that there are adjacent loads/stores, LocalStackSize should probably satisfy that too. There seems to be no benefit in allowing 0x...8 since a following ADD/SUB would be needed anyway.

[gberry]

Add back assert to emitFrameOffset that I accidentally removed from last update.

[gberry]

lib/Target/AArch64/AArch64FrameLowering.cpp

336

Yes, that's true with the following caveat. We would be relying on AArch64LoadStoreOptimizer to combine the first/last store/load and sp decrement/increment in the case where we decide not to combine the SP bumps (e.g. if the local stack size is 0). I think that is generally okay, though when I have tried simplifying the code in this way in the past I did run into one phase ordering issue. If I recall correctly, occasionally tail duplication of the return block would no longer get done because it appears to the tail merger that the ret block had 3 instructions (add, ldp, ret) instead of 2 (ldp post inc, ret), which caused it to make a different decision. This was due to the fact that tail duplication was being done before AArch64LoadStoreOptimizer. Do you think this simplifcation of the code is worth it in light of this extra complication? I will also say I didn't see any performance degradations because of the tail duplication difference, I just noticed it looking over code diffs.

412

I agree, though I'm not sure this is the place to check for that. I mainly added this assert because of the (LocalStackSize / 8) on the next line.
I can add asserts that the SP bumps are always 16-byte aligned when they are generated. Does that seem reasonable to you?

[t.p.northover]

lib/Target/AArch64/AArch64FrameLowering.cpp

336

I'm not opposed to using the load/store optimizer for that if it works (and not particularly worried about the tail-duplication change -- if it's ever important we can tweak its heuristic).

But unless I'm missing something performing that transformation here would still be simpler than the proposed patch. We'd just have a makeMemOpPrePost(MBBI, TotalOffset) call if emitPrologue/emitEpilogue decide it would be useful, wouldn't we?

412

Yep, that sounds fine.

[gberry]

lib/Target/AArch64/AArch64FrameLowering.cpp

336

Ah, I think I misunderstood your original comment. What you are proposing is:

• have spillCalleeSavedRegisters/restoreCalleeSavedRegisters emit stores/loads without any pre/post decrement/increment
• have emitPrologue/emitEpilogue either insert an add/sub or change the first callee-save store/load into a pre/post decrement/increment depending on which is better

If that is correct, I agree that is cleaner. If not, let's try again :)

412

Great, I'll do that.

[t.p.northover]

lib/Target/AArch64/AArch64FrameLowering.cpp

336

Yep, that's the one. Sorry about the confusion.

[gberry]

Update to simplify code based on Tim's feedback

[t.p.northover]

lib/Target/AArch64/AArch64FrameLowering.cpp

324

Perhaps use MBBI->isStore()? No obvious way to infer NewIsUnscaled, unfortunately.

395–397

I think this function could just be

static void fixupCalleeSaveRestoreStackOffsetIncAndDec(MachineInstr *MI, unsigned LocalStackSize) {
  MachineOperand &MO = MI->getOperand(MI->getNumExplicitOperands() - 1);
  assert(LocalStackSize % 8 == 0);
  MO.setImm(MO.getImm() + LocalStackSize/8);
}

But you probably made the right call in convertCalleeSaveRestoreToSPPrePostIncDec: the pre/post versions have extra operands that would get messy.

lib/Target/AArch64/AArch64RegisterInfo.cpp

391–394 ↗

(On Diff #56203)

Is this left-over from when you tried to convert spill/restoreCalleeSavedRegisters to use "addFrameIndex"? I can't see the need for it in the present scheme (no FrameSetup/FrameDestroy instructions seem to be created with a FrameIndex operand).

[gberry]

Tim,

Thanks, we're almost there :)

lib/Target/AArch64/AArch64FrameLowering.cpp

324

Yeah, that seems better.

395–397

Yep, this was just my lack of grep skills/knowledge of the MI API. I looked for a setOperand to do just what you are suggesting, without realizing I could just change the immediate value.

lib/Target/AArch64/AArch64RegisterInfo.cpp

391–394 ↗

(On Diff #56203)

Yep, I was right to get rid of it the first time :)

[gberry]

Update to address Tim's latest comments

[mcrosier]

lib/Target/AArch64/AArch64FrameLowering.cpp

354

The default case goes at the top of the switch.

401

default at top.

[t.p.northover]

Hi Geoff,

I think this looks basically good now (just trivia left), thanks for all the updates and feel free to commit without uploading another version here.

Tim.

lib/Target/AArch64/AArch64FrameLowering.cpp

402

llvm_unreachable is probably better here. Though I'd probably write the whole thing as an assertion (it's using up a lot of vertical space for something quite trivial).

lib/Target/AArch64/AArch64FrameLowering.h

public:
 
   void determineCalleeSaves(MachineFunction &MF, BitVector &SavedRegs,
                             RegScavenger *RS) const override;
 
   /// Returns true if the target will correctly handle shrink wrapping.
   bool enableShrinkWrapping(const MachineFunction &MF) const override {
     return true;
   }
+
+private:
+  bool shouldCombineCSRLocalStackBump(MachineFunction &MF,
+                                      unsigned StackBumpBytes) const;
 };
 
 } // End llvm namespace
 
 #endif

lib/Target/AArch64/AArch64FrameLowering.cpp

bool AArch64FrameLowering::canUseAsPrologue(
   // Don't need a scratch register if we're not going to re-align the stack.
   if (!RegInfo->needsStackRealignment(*MF))
     return true;
   // Otherwise, we can use any block as long as it has a scratch register
   // available.
   return findScratchNonCalleeSaveRegister(TmpMBB) != AArch64::NoRegister;
 }
 
+bool AArch64FrameLowering::shouldCombineCSRLocalStackBump(
+    MachineFunction &MF, unsigned StackBumpBytes) const {
+  AArch64FunctionInfo *AFI = MF.getInfo<AArch64FunctionInfo>();
+  const MachineFrameInfo *MFI = MF.getFrameInfo();
+  const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
+  const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
+
+  if (AFI->getLocalStackSize() == 0)
+    return false;
+
+  // 512 is the maximum immediate for stp/ldp that will be used for
+  // callee-save save/restores
+  if (StackBumpBytes >= 512)
+    return false;
+
+  if (MFI->hasVarSizedObjects())
+    return false;
+
+  if (RegInfo->needsStackRealignment(MF))
+    return false;
+
+  // This isn't strictly necessary, but it simplifies things a bit since the
+  // current RedZone handling code assumes the SP is adjusted by the
+  // callee-save save/restore code.
+  if (canUseRedZone(MF))
+    return false;
+
+  return true;
+}
+
+// Convert callee-save register save/restore instruction to do stack pointer
+// decrement/increment to allocate/deallocate the callee-save stack area by
+// converting store/load to use pre/post increment version.
+static MachineBasicBlock::iterator convertCalleeSaveRestoreToSPPrePostIncDec(
+    MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, DebugLoc DL,
+    const TargetInstrInfo *TII, int CSStackSizeInc) {
+
+  unsigned NewOpc;
+  bool NewIsUnscaled = false;
+  switch (MBBI->getOpcode()) {
+  case AArch64::STPXi:
+    NewOpc = AArch64::STPXpre;
+    break;
+  case AArch64::STPDi:
+    NewOpc = AArch64::STPDpre;
+    break;
+  case AArch64::STRXui:
+    NewOpc = AArch64::STRXpre;
+    NewIsUnscaled = true;
+    break;
+  case AArch64::STRDui:
+    NewOpc = AArch64::STRDpre;
+    NewIsUnscaled = true;
+    break;
+  case AArch64::LDPXi:
+    NewOpc = AArch64::LDPXpost;
+    break;
+  case AArch64::LDPDi:
+    NewOpc = AArch64::LDPDpost;
+    break;
+  case AArch64::LDRXui:
+    NewOpc = AArch64::LDRXpost;
+    NewIsUnscaled = true;
+    break;
+  case AArch64::LDRDui:
+    NewOpc = AArch64::LDRDpost;
+    NewIsUnscaled = true;
+    break;
+  default:
+    assert(!"Unexpected callee-save save/restore opcode!");
+    break;
+  }
+
+  MachineInstrBuilder MIB = BuildMI(MBB, MBBI, DL, TII->get(NewOpc));
+  MIB.addReg(AArch64::SP, RegState::Define);
+
+  // Copy all operands other than the immediate offset.
+  unsigned OpndIdx = 0;
+  for (unsigned OpndEnd = MBBI->getNumOperands() - 1; OpndIdx < OpndEnd;
+       ++OpndIdx)
+    MIB.addOperand(MBBI->getOperand(OpndIdx));
+
+  assert(MBBI->getOperand(OpndIdx).getImm() == 0 &&
+         "Unexpected immediate offset in first/last callee-save save/restore "
+         "instruction!");
+  assert(MBBI->getOperand(OpndIdx - 1).getReg() == AArch64::SP &&
+         "Unexpected base register in callee-save save/restore instruction!");
+  // Last operand is immediate offset that needs fixing.
+  assert(CSStackSizeInc % 8 == 0);
+  int64_t CSStackSizeIncImm = CSStackSizeInc;
+  if (!NewIsUnscaled)
+    CSStackSizeIncImm /= 8;
+  MIB.addImm(CSStackSizeIncImm);
+
+  MIB.setMIFlags(MBBI->getFlags());
+  MIB.setMemRefs(MBBI->memoperands_begin(), MBBI->memoperands_end());
+
+  return std::prev(MBB.erase(MBBI));
+}
+
+// Fixup callee-save register save/restore instructions to take into account
+// combined SP bump by adding the local stack size to the stack offsets.
+static void fixupCalleeSaveRestoreStackOffset(MachineInstr *MI,
+                                              unsigned LocalStackSize) {
+
+  switch (MI->getOpcode()) {
+  case AArch64::STPXi:
+  case AArch64::STPDi:
+  case AArch64::STRXui:
+  case AArch64::STRDui:
+  case AArch64::LDPXi:
+  case AArch64::LDPDi:
+  case AArch64::LDRXui:
+  case AArch64::LDRDui:
+    break;
+  default:
+    assert(!"Unexpected callee-save save/restore opcode!");
+    break;
+  }
+
+  unsigned OffsetIdx = MI->getNumExplicitOperands() - 1;
+  assert(MI->getOperand(OffsetIdx - 1).getReg() == AArch64::SP &&
+         "Unexpected base register in callee-save save/restore instruction!");
+  // Last operand is immediate offset that needs fixing.
+  MachineOperand &OffsetOpnd = MI->getOperand(OffsetIdx);
+  // All generated opcodes have scaled offsets.
+  assert(LocalStackSize % 8 == 0);
+  OffsetOpnd.setImm(OffsetOpnd.getImm() + LocalStackSize / 8);
+}
+
 void AArch64FrameLowering::emitPrologue(MachineFunction &MF,
                                         MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.begin();
   const MachineFrameInfo *MFI = MF.getFrameInfo();
   const Function *Fn = MF.getFunction();
   const AArch64Subtarget &Subtarget = MF.getSubtarget<AArch64Subtarget>();
   const AArch64RegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   const TargetInstrInfo *TII = Subtarget.getInstrInfo();
else {
           MCCFIInstruction::createDefCfaOffset(FrameLabel, -NumBytes));
       BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIIndex)
           .setMIFlags(MachineInstr::FrameSetup);
     }
     return;
   }
 
-  NumBytes -= AFI->getCalleeSavedStackSize();
-  assert(NumBytes >= 0 && "Negative stack allocation size!?");
+  auto CSStackSize = AFI->getCalleeSavedStackSize();
   // All of the remaining stack allocations are for locals.
-  AFI->setLocalStackSize(NumBytes);
+  AFI->setLocalStackSize(NumBytes - CSStackSize);
 
-  // Move past the saves of the callee-saved registers.
+  bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
+  if (CombineSPBump) {
+    emitFrameOffset(MBB, MBBI, DL, AArch64::SP, AArch64::SP, -NumBytes, TII,
+                    MachineInstr::FrameSetup);
+    NumBytes = 0;
+  } else if (CSStackSize != 0) {
+    MBBI = convertCalleeSaveRestoreToSPPrePostIncDec(MBB, MBBI, DL, TII,
+                                                     -CSStackSize);
+    NumBytes -= CSStackSize;
+  }
+  assert(NumBytes >= 0 && "Negative stack allocation size!?");
+
+  // Move past the saves of the callee-saved registers, fixing up the offsets
+  // and pre-inc if we decided to combine the callee-save and local stack
+  // pointer bump above.
   MachineBasicBlock::iterator End = MBB.end();
-  while (MBBI != End && MBBI->getFlag(MachineInstr::FrameSetup))
+  while (MBBI != End && MBBI->getFlag(MachineInstr::FrameSetup)) {
+    if (CombineSPBump)
+      fixupCalleeSaveRestoreStackOffset(MBBI, AFI->getLocalStackSize());
     ++MBBI;
+  }
   if (HasFP) {
     // Only set up FP if we actually need to. Frame pointer is fp = sp - 16.
-    int FPOffset = AFI->getCalleeSavedStackSize() - 16;
+    int FPOffset = CSStackSize - 16;
+    if (CombineSPBump)
+      FPOffset += AFI->getLocalStackSize();
 
     // Issue    sub fp, sp, FPOffset or
     //          mov fp,sp          when FPOffset is zero.
     // Note: All stores of callee-saved registers are marked as "FrameSetup".
     // This code marks the instruction(s) that set the FP also.
     emitFrameOffset(MBB, MBBI, DL, AArch64::FP, AArch64::SP, FPOffset, TII,
                     MachineInstr::FrameSetup);
   }
void AArch64FrameLowering::emitEpilogue(MachineFunction &MF,
   //      ----------------------        ---          ---
   //
   // So NumBytes = StackSize + BytesInStackArgArea - CalleeArgStackSize
   //             = StackSize + ArgumentPopSize
   //
   // AArch64TargetLowering::LowerCall figures out ArgumentPopSize and keeps
   // it as the 2nd argument of AArch64ISD::TC_RETURN.
 
+  auto CSStackSize = AFI->getCalleeSavedStackSize();
+  bool CombineSPBump = shouldCombineCSRLocalStackBump(MF, NumBytes);
+
+  if (!CombineSPBump && CSStackSize != 0)
+    convertCalleeSaveRestoreToSPPrePostIncDec(
+        MBB, std::prev(MBB.getFirstTerminator()), DL, TII, CSStackSize);
+
   // Move past the restores of the callee-saved registers.
   MachineBasicBlock::iterator LastPopI = MBB.getFirstTerminator();
   MachineBasicBlock::iterator Begin = MBB.begin();
   while (LastPopI != Begin) {
     --LastPopI;
     if (!LastPopI->getFlag(MachineInstr::FrameDestroy)) {
       ++LastPopI;
       break;
-    }
+    } else if (CombineSPBump)
+      fixupCalleeSaveRestoreStackOffset(LastPopI, AFI->getLocalStackSize());
+
+  // If there is a single SP update, insert it before the ret and we're done.
+  if (CombineSPBump) {
+    emitFrameOffset(MBB, MBB.getFirstTerminator(), DL, AArch64::SP, AArch64::SP,
+                    NumBytes + ArgumentPopSize, TII,
+                    MachineInstr::FrameDestroy);
+    return;
   }
-  NumBytes -= AFI->getCalleeSavedStackSize();
+
+  NumBytes -= CSStackSize;
   assert(NumBytes >= 0 && "Negative stack allocation size!?");
 
   if (!hasFP(MF)) {
     bool RedZone = canUseRedZone(MF);
     // If this was a redzone leaf function, we don't need to restore the
     // stack pointer (but we may need to pop stack args for fastcc).
     if (RedZone && ArgumentPopSize == 0)
       return;
 
-    bool NoCalleeSaveRestore = AFI->getCalleeSavedStackSize() == 0;
+    bool NoCalleeSaveRestore = CSStackSize == 0;
     int StackRestoreBytes = RedZone ? 0 : NumBytes;
     if (NoCalleeSaveRestore)
       StackRestoreBytes += ArgumentPopSize;
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP,
                     StackRestoreBytes, TII, MachineInstr::FrameDestroy);
     // If we were able to combine the local stack pop with the argument pop,
     // then we're done.
     if (NoCalleeSaveRestore || ArgumentPopSize == 0)
       return;
     NumBytes = 0;
   }
 
   // Restore the original stack pointer.
   // FIXME: Rather than doing the math here, we should instead just use
   // non-post-indexed loads for the restores if we aren't actually going to
   // be able to save any instructions.
   if (MFI->hasVarSizedObjects() || AFI->isStackRealigned())
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::FP,
-                    -AFI->getCalleeSavedStackSize() + 16, TII,
-                    MachineInstr::FrameDestroy);
+                    -CSStackSize + 16, TII, MachineInstr::FrameDestroy);
   else if (NumBytes)
     emitFrameOffset(MBB, LastPopI, DL, AArch64::SP, AArch64::SP, NumBytes, TII,
                     MachineInstr::FrameDestroy);
 
   // This must be placed after the callee-save restore code because that code
   // assumes the SP is at the same location as it was after the callee-save save
   // code in the prologue.
   if (ArgumentPopSize)
if (isFixed) {
       // and we don't have a base pointer available. If an offset is
       // available via the FP and the SP, use whichever is closest.
       if (PreferFP || MFI->hasVarSizedObjects() || FPOffset >= 0 ||
           (FPOffset >= -256 && Offset > -FPOffset))
         UseFP = true;
     }
   }
 
   assert((isFixed || !RegInfo->needsStackRealignment(MF) || !UseFP) &&
          "In the presence of dynamic stack pointer realignment, "
          "non-argument objects cannot be accessed through the frame pointer");
 
   if (UseFP) {
     FrameReg = RegInfo->getFrameRegister(MF);
     return FPOffset;
   }
 
for (unsigned i = 0; i < Count; ++i) {
     assert(Offset % 8 == 0);
     RPI.Offset = Offset / 8;
     assert((RPI.Offset >= -64 && RPI.Offset <= 63) &&
            "Offset out of bounds for LDP/STP immediate");
 
     RegPairs.push_back(RPI);
     if (RPI.isPaired())
       ++i;
   }
-
-  // Align first offset to even 16-byte boundary to avoid additional SP
-  // adjustment instructions.
-  // Last pair offset is size of whole callee-save region for SP
-  // pre-dec/post-inc.
-  RegPairInfo &LastPair = RegPairs.back();
-  assert(AFI->getCalleeSavedStackSize() % 8 == 0);
-  LastPair.Offset = AFI->getCalleeSavedStackSize() / 8;
 }
 
 bool AArch64FrameLowering::spillCalleeSavedRegisters(
     MachineBasicBlock &MBB, MachineBasicBlock::iterator MI,
     const std::vector<CalleeSavedInfo> &CSI,
     const TargetRegisterInfo *TRI) const {
   MachineFunction &MF = *MBB.getParent();
   const TargetInstrInfo &TII = *MF.getSubtarget().getInstrInfo();
   DebugLoc DL;
   SmallVector<RegPairInfo, 8> RegPairs;
 
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs);
 
   for (auto RPII = RegPairs.rbegin(), RPIE = RegPairs.rend(); RPII != RPIE;
        ++RPII) {
     RegPairInfo RPI = *RPII;
     unsigned Reg1 = RPI.Reg1;
     unsigned Reg2 = RPI.Reg2;
     unsigned StrOpc;
 
-    // Issue sequence of non-sp increment and pi sp spills for cs regs. The
-    // first spill is a pre-increment that allocates the stack.
+    // Issue sequence of spills for cs regs.  The first spill may be converted
+    // to a pre-decrement store later by emitPrologue if the callee-save stack
+    // area allocation can't be combined with the local stack area allocation.
     // For example:
-    //    stp     x22, x21, [sp, #-48]!   // addImm(-6)
+    //    stp     x22, x21, [sp, #0]     // addImm(+0)
     //    stp     x20, x19, [sp, #16]    // addImm(+2)
     //    stp     fp, lr, [sp, #32]      // addImm(+4)
     // Rationale: This sequence saves uop updates compared to a sequence of
     // pre-increment spills like stp xi,xj,[sp,#-16]!
     // Note: Similar rationale and sequence for restores in epilog.
-    bool BumpSP = RPII == RegPairs.rbegin();
-    if (RPI.IsGPR) {
-      // For first spill use pre-increment store.
-      if (BumpSP)
-        StrOpc = RPI.isPaired() ? AArch64::STPXpre : AArch64::STRXpre;
-      else
-        StrOpc = RPI.isPaired() ? AArch64::STPXi : AArch64::STRXui;
-    } else {
-      // For first spill use pre-increment store.
-      if (BumpSP)
-        StrOpc = RPI.isPaired() ? AArch64::STPDpre : AArch64::STRDpre;
-      else
-        StrOpc = RPI.isPaired() ? AArch64::STPDi : AArch64::STRDui;
-    }
+    if (RPI.IsGPR)
     DEBUG(dbgs() << "CSR spill: (" << TRI->getName(Reg1);
           if (RPI.isPaired())
             dbgs() << ", " << TRI->getName(Reg2);
           dbgs() << ") -> fi#(" << RPI.FrameIdx;
           if (RPI.isPaired())
             dbgs() << ", " << RPI.FrameIdx+1;
           dbgs() << ")\n");
 
-    const int Offset = BumpSP ? -RPI.Offset : RPI.Offset;
     MachineInstrBuilder MIB = BuildMI(MBB, MI, DL, TII.get(StrOpc));
-    if (BumpSP)
-      MIB.addReg(AArch64::SP, RegState::Define);
-
-    if (RPI.isPaired()) {
-      MBB.addLiveIn(Reg1);
+    if (RPI.isPaired()) {
       MBB.addLiveIn(Reg2);
-      MIB.addReg(Reg2, getPrologueDeath(MF, Reg2))
-        .addReg(Reg1, getPrologueDeath(MF, Reg1))
-        .addReg(AArch64::SP)
-        .addImm(Offset) // [sp, #offset * 8], where factor * 8 is implicit
-        .setMIFlag(MachineInstr::FrameSetup);
+      MIB.addReg(Reg2, getPrologueDeath(MF, Reg2));
       MIB.addMemOperand(MF.getMachineMemOperand(
           MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
           MachineMemOperand::MOStore, 8, 8));
-    } else {
-      MBB.addLiveIn(Reg1);
-      MIB.addReg(Reg1, getPrologueDeath(MF, Reg1))
+    }
         .addReg(AArch64::SP)
-        .addImm(BumpSP ? Offset * 8 : Offset) // pre-inc version is unscaled
+        .addImm(RPI.Offset) // [sp, #offset*8], where factor*8 is implicit
         .setMIFlag(MachineInstr::FrameSetup);
-    }
     MIB.addMemOperand(MF.getMachineMemOperand(
         MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
         MachineMemOperand::MOStore, 8, 8));
   }
   return true;
 }
 
 bool AArch64FrameLowering::restoreCalleeSavedRegisters(
bool AArch64FrameLowering::restoreCalleeSavedRegisters(
   computeCalleeSaveRegisterPairs(MF, CSI, TRI, RegPairs);
 
   for (auto RPII = RegPairs.begin(), RPIE = RegPairs.end(); RPII != RPIE;
        ++RPII) {
     RegPairInfo RPI = *RPII;
     unsigned Reg1 = RPI.Reg1;
     unsigned Reg2 = RPI.Reg2;
 
-    // Issue sequence of non-sp increment and sp-pi restores for cs regs. Only
-    // the last load is sp-pi post-increment and de-allocates the stack:
+    // Issue sequence of restores for cs regs. The last restore may be converted
+    // to a post-increment load later by emitEpilogue if the callee-save stack
+    // area allocation can't be combined with the local stack area allocation.
     // For example:
     //    ldp     fp, lr, [sp, #32]       // addImm(+4)
     //    ldp     x20, x19, [sp, #16]     // addImm(+2)
-    //    ldp     x22, x21, [sp], #48     // addImm(+6)
+    //    ldp     x22, x21, [sp, #0]      // addImm(+0)
     // Note: see comment in spillCalleeSavedRegisters()
     unsigned LdrOpc;
-    bool BumpSP = RPII == std::prev(RegPairs.end());
-    if (RPI.IsGPR) {
-      if (BumpSP)
-        LdrOpc = RPI.isPaired() ? AArch64::LDPXpost : AArch64::LDRXpost;
-      else
-        LdrOpc = RPI.isPaired() ? AArch64::LDPXi : AArch64::LDRXui;
-    } else {
-      if (BumpSP)
-        LdrOpc = RPI.isPaired() ? AArch64::LDPDpost : AArch64::LDRDpost;
-      else
-        LdrOpc = RPI.isPaired() ? AArch64::LDPDi : AArch64::LDRDui;
-    }
+    if (RPI.IsGPR)
     DEBUG(dbgs() << "CSR restore: (" << TRI->getName(Reg1);
           if (RPI.isPaired())
             dbgs() << ", " << TRI->getName(Reg2);
           dbgs() << ") -> fi#(" << RPI.FrameIdx;
           if (RPI.isPaired())
             dbgs() << ", " << RPI.FrameIdx+1;
           dbgs() << ")\n");
 
-    const int Offset = RPI.Offset;
     MachineInstrBuilder MIB = BuildMI(MBB, MI, DL, TII.get(LdrOpc));
-    if (BumpSP)
-      MIB.addReg(AArch64::SP, RegState::Define);
-
     if (RPI.isPaired()) {
-      MIB.addReg(Reg2, getDefRegState(true))
-        .addReg(Reg1, getDefRegState(true))
-        .addReg(AArch64::SP)
-        .addImm(Offset) // [sp], #offset * 8  or [sp, #offset * 8]
-                        // where the factor * 8 is implicit
-        .setMIFlag(MachineInstr::FrameDestroy);
+      MIB.addReg(Reg2, getDefRegState(true));
       MIB.addMemOperand(MF.getMachineMemOperand(
           MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx + 1),
           MachineMemOperand::MOLoad, 8, 8));
-    } else {
-      MIB.addReg(Reg1, getDefRegState(true))
+    }
         .addReg(AArch64::SP)
-        .addImm(BumpSP ? Offset * 8 : Offset) // post-dec version is unscaled
+        .addImm(RPI.Offset) // [sp, #offset*8] where the factor*8 is implicit
         .setMIFlag(MachineInstr::FrameDestroy);
-    }
     MIB.addMemOperand(MF.getMachineMemOperand(
         MachinePointerInfo::getFixedStack(MF, RPI.FrameIdx),
         MachineMemOperand::MOLoad, 8, 8));
   }
   return true;
 }
 
 void AArch64FrameLowering::determineCalleeSaves(MachineFunction &MF,

lib/Target/AArch64/AArch64InstrInfo.cpp

 void llvm::emitFrameOffset(MachineBasicBlock &MBB,
                            MachineBasicBlock::iterator MBBI, DebugLoc DL,
                            unsigned DestReg, unsigned SrcReg, int Offset,
                            const TargetInstrInfo *TII,
                            MachineInstr::MIFlag Flag, bool SetNZCV) {
   if (DestReg == SrcReg && Offset == 0)
     return;
 
+  assert((DestReg != AArch64::SP || Offset % 16 == 0) &&
+         "SP increment/decrement not 16-byte aligned");
+
   bool isSub = Offset < 0;
   if (isSub)
     Offset = -Offset;
 
   // FIXME: If the offset won't fit in 24-bits, compute the offset into a
   // scratch register.  If DestReg is a virtual register, use it as the
   // scratch register; otherwise, create a new virtual register (to be
   // replaced by the scavenger at the end of PEI).  That case can be optimized

test/CodeGen/AArch64/aarch64-dynamic-stack-layout.ll

entry:
   %add1 = add nsw i32 %add, %l1.0.l1.0.
   %call = tail call i32 @g()
   %add2 = add nsw i32 %add1, %call
   ret i32 %add2
 }
 ; CHECK-LABEL: novla_nodynamicrealign_call
 ; CHECK: .cfi_startproc
 ;   Check that used callee-saved registers are saved
-; CHECK: stp  x19, x30, [sp, #-16]!
-; CHECK: sub  sp, sp, #16
+; CHECK: sub  sp, sp, #32
+; CHECK: stp  x19, x30, [sp, #16]
 ;   Check correctness of cfi pseudo-instructions
 ; CHECK: .cfi_def_cfa_offset 32
 ; CHECK: .cfi_offset w30, -8
 ; CHECK: .cfi_offset w19, -16
 ;   Check correct access to arguments passed on the stack, through stack pointer
 ; CHECK: ldr  d[[DARG:[0-9]+]], [sp, #56]
 ; CHECK: ldr  w[[IARG:[0-9]+]], [sp, #40]
 ;   Check correct access to local variable on the stack, through stack pointer
 ; CHECK: ldr  w[[ILOC:[0-9]+]], [sp, #12]
 ;   Check epilogue:
-; CHECK: ldp  x19, x30, [sp], #16
+; CHECK: ldp  x19, x30, [sp, #16]
 ; CHECK: ret
 ; CHECK: .cfi_endproc
 
 ; CHECK-MACHO-LABEL: _novla_nodynamicrealign_call:
 ; CHECK-MACHO: .cfi_startproc
 ;   Check that used callee-saved registers are saved
-; CHECK-MACHO: stp  x20, x19, [sp, #-32]!
+; CHECK-MACHO: sub  sp, sp, #48
+; CHECK-MACHO: stp  x20, x19, [sp, #16]
 ;   Check that the frame pointer is created:
-; CHECK-MACHO: stp  x29, x30, [sp, #16]
-; CHECK-MACHO: add  x29, sp, #16
+; CHECK-MACHO: stp  x29, x30, [sp, #32]
+; CHECK-MACHO: add  x29, sp, #32
 ;   Check correctness of cfi pseudo-instructions
 ; CHECK-MACHO: .cfi_def_cfa w29, 16
 ; CHECK-MACHO: .cfi_offset w30, -8
 ; CHECK-MACHO: .cfi_offset w29, -16
 ; CHECK-MACHO: .cfi_offset w19, -24
 ; CHECK-MACHO: .cfi_offset w20, -32
 ;   Check correct access to arguments passed on the stack, through frame pointer
 ; CHECK-MACHO: ldr  d[[DARG:[0-9]+]], [x29, #32]
 ; CHECK-MACHO: ldr  w[[IARG:[0-9]+]], [x29, #20]
 ;   Check correct access to local variable on the stack, through stack pointer
 ; CHECK-MACHO: ldr  w[[ILOC:[0-9]+]], [sp, #12]
 ;   Check epilogue:
-; CHECK-MACHO: ldp  x29, x30, [sp, #16]
-; CHECK-MACHO: ldp  x20, x19, [sp], #32
+; CHECK-MACHO: ldp  x29, x30, [sp, #32]
+; CHECK-MACHO: ldp  x20, x19, [sp, #16]
 ; CHECK-MACHO: ret
 ; CHECK-MACHO: .cfi_endproc
 
 
 declare i32 @g() #0
 
 ; Function Attrs: nounwind
 define i32 @novla_nodynamicrealign_nocall(i32 %i1, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6, i32 %i7, i32 %i8, i32 %i9, i32 %i10, double %d1, double %d2, double %d3, double %d4, double %d5, double %d6, double %d7, double %d8, double %d9, double %d10) #1 {

test/CodeGen/AArch64/arm64-aapcs-be.ll

 ; CHECK-LABEL: test_block_addr:
 ; CHECK: ldr s0, [sp]
   %val = extractvalue [1 x float] %in, 0
   ret float %val
 }
 
 define void @test_block_addr_callee() {
 ; CHECK-LABEL: test_block_addr_callee:
-; CHECK: str {{[a-z0-9]+}}, [sp, #-16]!
+; CHECK: sub sp, sp, #32
+; CHECK: str {{[a-z0-9]+}}, [sp, #16]
 ; CHECK: bl test_block_addr
   %val = insertvalue [1 x float] undef, float 0.0, 0
   call float @test_block_addr([8 x float] undef, [1 x float] %val)
   ret void
 }

test/CodeGen/AArch64/arm64-abi.ll

 
 ; rdar://12648441
 ; Check alignment on stack for v64, f64, i64, f32, i32.
 define double @test3(<2 x i32>* nocapture %in) nounwind {
 entry:
 ; CHECK-LABEL: test3
 ; CHECK: str [[REG_1:d[0-9]+]], [sp, #8]
 ; FAST-LABEL: test3
-; FAST: sub sp, sp, #32
+; FAST: sub sp, sp, #48
 ; FAST: mov x[[ADDR:[0-9]+]], sp
 ; FAST: str [[REG_1:d[0-9]+]], [x[[ADDR]], #8]
   %0 = load <2 x i32>, <2 x i32>* %in, align 8
   %call = tail call double @args_vec_2i(double 3.000000e+00, <2 x i32> %0,
           <2 x i32> %0, <2 x i32> %0, <2 x i32> %0, <2 x i32> %0, <2 x i32> %0,
           <2 x i32> %0, float 3.000000e+00, <2 x i32> %0, i8 signext 3)
   ret double %call
 }

test/CodeGen/AArch64/arm64-abi_align.ll

 ; CHECK: str {{x[0-9]+}}, [sp, #16]
 ; CHECK: str {{q[0-9]+}}, [sp]
 ; CHECK: add x1, sp, #32
 ; CHECK: mov x2, sp
 ; Space for s1 is allocated at sp+32
 ; Space for s2 is allocated at sp
 
 ; FAST-LABEL: caller42
-; FAST: sub sp, sp, #96
+; FAST: sub sp, sp, #112
 ; Space for s1 is allocated at fp-24 = sp+72
 ; Space for s2 is allocated at sp+48
 ; FAST: sub x[[A:[0-9]+]], x29, #24
 ; FAST: add x[[A:[0-9]+]], sp, #48
 ; Call memcpy with size = 24 (0x18)
 ; FAST: orr {{x[0-9]+}}, xzr, #0x18
   %tmp = alloca %struct.s42, align 4
   %tmp1 = alloca %struct.s42, align 4

 
 declare i32 @f42_stack(i32 %i, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6,
                        i32 %i7, i32 %i8, i32 %i9, %struct.s42* nocapture %s1,
                        %struct.s42* nocapture %s2) #2
 
 define i32 @caller42_stack() #3 {
 entry:
 ; CHECK-LABEL: caller42_stack
-; CHECK: mov x29, sp
-; CHECK: sub sp, sp, #96
+; CHECK: sub sp, sp, #112
+; CHECK: add x29, sp, #96
 ; CHECK: stur {{x[0-9]+}}, [x29, #-16]
 ; CHECK: stur {{q[0-9]+}}, [x29, #-32]
 ; CHECK: str {{x[0-9]+}}, [sp, #48]
 ; CHECK: str {{q[0-9]+}}, [sp, #32]
 ; Space for s1 is allocated at x29-32 = sp+64
 ; Space for s2 is allocated at sp+32
 ; CHECK: add x[[B:[0-9]+]], sp, #32
 ; CHECK: str x[[B]], [sp, #16]

 ; CHECK: str {{q[0-9]+}}, [sp, #16]
 ; CHECK: str {{q[0-9]+}}, [sp]
 ; CHECK: add x1, sp, #32
 ; CHECK: mov x2, sp
 ; Space for s1 is allocated at sp+32
 ; Space for s2 is allocated at sp
 
 ; FAST-LABEL: caller43
-; FAST: mov x29, sp
+; FAST: add x29, sp, #64
 ; Space for s1 is allocated at sp+32
 ; Space for s2 is allocated at sp
 ; FAST: add x1, sp, #32
 ; FAST: mov x2, sp
 ; FAST: str {{x[0-9]+}}, [sp, #32]
 ; FAST: str {{x[0-9]+}}, [sp, #40]
 ; FAST: str {{x[0-9]+}}, [sp, #48]
 ; FAST: str {{x[0-9]+}}, [sp, #56]

 
 declare i32 @f43_stack(i32 %i, i32 %i2, i32 %i3, i32 %i4, i32 %i5, i32 %i6,
                        i32 %i7, i32 %i8, i32 %i9, %struct.s43* nocapture %s1,
                        %struct.s43* nocapture %s2) #2
 
 define i32 @caller43_stack() #3 {
 entry:
 ; CHECK-LABEL: caller43_stack
-; CHECK: mov x29, sp
-; CHECK: sub sp, sp, #96
+; CHECK: sub sp, sp, #112
+; CHECK: add x29, sp, #96
 ; CHECK: stur {{q[0-9]+}}, [x29, #-16]
 ; CHECK: stur {{q[0-9]+}}, [x29, #-32]
 ; CHECK: str {{q[0-9]+}}, [sp, #48]
 ; CHECK: str {{q[0-9]+}}, [sp, #32]
 ; Space for s1 is allocated at x29-32 = sp+64
 ; Space for s2 is allocated at sp+32
 ; CHECK: add x[[B:[0-9]+]], sp, #32
 ; CHECK: str x[[B]], [sp, #16]
 ; CHECK: sub x[[A:[0-9]+]], x29, #32
 ; Address of s1 is passed on stack at sp+8
 ; CHECK: str x[[A]], [sp, #8]
 ; CHECK: movz w[[C:[0-9]+]], #0x9
 ; CHECK: str w[[C]], [sp]
 
 ; FAST-LABEL: caller43_stack
-; FAST: sub sp, sp, #96
+; FAST: sub sp, sp, #112
 ; Space for s1 is allocated at fp-32 = sp+64
 ; Space for s2 is allocated at sp+32
 ; FAST: sub x[[A:[0-9]+]], x29, #32
 ; FAST: add x[[B:[0-9]+]], sp, #32
 ; FAST: stur {{x[0-9]+}}, [x29, #-32]
 ; FAST: stur {{x[0-9]+}}, [x29, #-24]
 ; FAST: stur {{x[0-9]+}}, [x29, #-16]
 ; FAST: stur {{x[0-9]+}}, [x29, #-8]
declare i32 @callee_i64(i32 %i, i32 %i2, i32 %i3, i32 %i4, i32 %i5,
                                i32 %i6, i32 %i7, i64 %s1, i32 %i8)
 
 define i32 @i64_split() {
 entry:
 ; CHECK-LABEL: i64_split
 ; "i64 %0" should be in register x7.
 ; "i32 8" should be on stack at [sp].
 ; CHECK: ldr x7, [{{x[0-9]+}}]
-; CHECK: str {{w[0-9]+}}, [sp, #-16]!
+; CHECK: str {{w[0-9]+}}, [sp]
 ; FAST-LABEL: i64_split
 ; FAST: ldr x7, [{{x[0-9]+}}]
 ; FAST: mov x[[R0:[0-9]+]], sp
 ; FAST: orr w[[R1:[0-9]+]], wzr, #0x8
 ; FAST: str w[[R1]], {{\[}}x[[R0]]{{\]}}
   %0 = load i64, i64* bitcast (%struct.s41* @g41 to i64*), align 16
   %call = tail call i32 @callee_i64(i32 1, i32 2, i32 3, i32 4, i32 5,
                                     i32 6, i32 7, i64 %0, i32 8) #5

test/CodeGen/AArch64/arm64-fast-isel-alloca.ll

 ; This test should cause the TargetMaterializeAlloca to be invoked
 ; RUN: llc -O0 -fast-isel-abort=1 -verify-machineinstrs -mtriple=arm64-apple-darwin -disable-fp-elim < %s | FileCheck %s
 
 %struct.S1Ty = type { i64 }
 %struct.S2Ty = type { %struct.S1Ty, %struct.S1Ty }
 
 define void @takeS1(%struct.S1Ty* %V) nounwind {
 entry:
   %V.addr = alloca %struct.S1Ty*, align 8
   store %struct.S1Ty* %V, %struct.S1Ty** %V.addr, align 8
   ret void
 }
 
 define void @main() nounwind {
 entry:
 ; CHECK: main
-; CHECK: mov x29, sp
+; CHECK: add x29, sp, #16
 ; CHECK: mov [[REG:x[0-9]+]], sp
 ; CHECK-NEXT: add x0, [[REG]], #8
   %E = alloca %struct.S2Ty, align 4
   %B = getelementptr inbounds %struct.S2Ty, %struct.S2Ty* %E, i32 0, i32 1
   call void @takeS1(%struct.S1Ty* %B)
   ret void
 }

test/CodeGen/AArch64/arm64-hello.ll

 ; RUN: llc < %s -mtriple=arm64-apple-ios7.0 -disable-post-ra -disable-fp-elim | FileCheck %s
 ; RUN: llc < %s -mtriple=arm64-linux-gnu -disable-post-ra | FileCheck %s --check-prefix=CHECK-LINUX
 
 ; CHECK-LABEL: main:
-; CHECK:  stp x29, x30, [sp, #-16]!
-; CHECK-NEXT: mov x29, sp
-; CHECK-NEXT: sub sp, sp, #16
+; CHECK:  sub sp, sp, #32
+; CHECK-NEXT: stp x29, x30, [sp, #16]
+; CHECK-NEXT: add x29, sp, #16
 ; CHECK-NEXT: stur  wzr, [x29, #-4]
 ; CHECK:  adrp  x0, L_.str@PAGE
 ; CHECK:  add x0, x0, L_.str@PAGEOFF
 ; CHECK-NEXT: bl  _puts
-; CHECK-NEXT: add sp, sp, #16
-; CHECK-NEXT: ldp x29, x30, [sp], #16
+; CHECK-NEXT: ldp x29, x30, [sp, #16]
+; CHECK-NEXT: add sp, sp, #32
 ; CHECK-NEXT: ret
 
 ; CHECK-LINUX-LABEL: main:
-; CHECK-LINUX:  str x30, [sp, #-16]!
-; CHECK-LINUX-NEXT: sub sp, sp, #16
+; CHECK-LINUX:  sub sp, sp, #32
+; CHECK-LINUX-NEXT: str x30, [sp, #16]
 ; CHECK-LINUX-NEXT: str wzr, [sp, #12]
 ; CHECK-LINUX:  adrp  x0, .L.str
 ; CHECK-LINUX:  add x0, x0, :lo12:.L.str
 ; CHECK-LINUX-NEXT: bl  puts
-; CHECK-LINUX-NEXT: add sp, sp, #16
-; CHECK-LINUX-NEXT: ldr x30, [sp], #16
+; CHECK-LINUX-NEXT: ldr x30, [sp, #16]
+; CHECK-LINUX-NEXT: add sp, sp, #32
 ; CHECK-LINUX-NEXT: ret
 
 @.str = private unnamed_addr constant [7 x i8] c"hello\0A\00"
 
 define i32 @main() nounwind ssp {
 entry:
   %retval = alloca i32, align 4
   store i32 0, i32* %retval
   %call = call i32 @puts(i8* getelementptr inbounds ([7 x i8], [7 x i8]* @.str, i32 0, i32 0))
   ret i32 %call
 }
 
 declare i32 @puts(i8*)

test/CodeGen/AArch64/arm64-join-reserved.ll

 ; RUN: llc < %s -verify-machineinstrs | FileCheck %s
 target triple = "arm64-apple-macosx10"
 
 ; Make sure that a store to [sp] addresses off sp directly.
 ; A move isn't necessary.
 ; <rdar://problem/11492712>
 ; CHECK-LABEL: g:
-; CHECK: str xzr, [sp, #-16]!
+; CHECK: str xzr, [sp]
 ; CHECK: bl
 ; CHECK: ret
 define void @g() nounwind ssp {
 entry:
   tail call void (i32, ...) @f(i32 0, i32 0) nounwind
   ret void
 }
 
 declare void @f(i32, ...)

test/CodeGen/AArch64/arm64-patchpoint-webkit_jscc.ll

 ; RUN: llc -mtriple=arm64-apple-darwin -enable-misched=0 -mcpu=cyclone            < %s | FileCheck %s
 ; RUN: llc -mtriple=arm64-apple-darwin -enable-misched=0 -mcpu=cyclone -fast-isel < %s | FileCheck %s --check-prefix=FAST
 
 ; One argument will be passed in register, the other will be pushed on the stack.
 ; Return value in x0.
 define void @jscall_patchpoint_codegen(i64 %p1, i64 %p2, i64 %p3, i64 %p4) {
 entry:
 ; CHECK-LABEL: jscall_patchpoint_codegen:
 ; CHECK:       Ltmp
-; CHECK:       str x{{.+}}, [sp, #-16]!
+; CHECK:       str x{{.+}}, [sp]
 ; CHECK-NEXT:  mov  x0, x{{.+}}
 ; CHECK:       Ltmp
 ; CHECK-NEXT:  movz  x16, #0xffff, lsl #32
 ; CHECK-NEXT:  movk  x16, #0xdead, lsl #16
 ; CHECK-NEXT:  movk  x16, #0xbeef
 ; CHECK-NEXT:  blr x16
 ; FAST-LABEL:  jscall_patchpoint_codegen:
 ; FAST:        Ltmp
-; FAST:        str x{{.+}}, [sp, #-16]!
+; FAST:        str x{{.+}}, [sp]
 ; FAST:        Ltmp
 ; FAST-NEXT:   movz  x16, #0xffff, lsl #32
 ; FAST-NEXT:   movk  x16, #0xdead, lsl #16
 ; FAST-NEXT:   movk  x16, #0xbeef
 ; FAST-NEXT:   blr x16
   %resolveCall2 = inttoptr i64 281474417671919 to i8*
   %result = tail call webkit_jscc i64 (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.i64(i64 5, i32 20, i8* %resolveCall2, i32 2, i64 %p4, i64 %p2)
   %resolveCall3 = inttoptr i64 244837814038255 to i8*

 ; CHECK-NEXT:  movk  x16, #0xdead, lsl #16
 ; CHECK-NEXT:  movk  x16, #0xbeef
 ; CHECK-NEXT:  blr x16
 ; FAST-LABEL:  jscall_patchpoint_codegen2:
 ; FAST:        Ltmp
 ; FAST:        orr [[REG1:x[0-9]+]], xzr, #0x2
 ; FAST-NEXT:   orr [[REG2:w[0-9]+]], wzr, #0x4
 ; FAST-NEXT:   orr [[REG3:x[0-9]+]], xzr, #0x6
-; FAST-NEXT:   str [[REG1]], [sp, #-32]!
+; FAST-NEXT:   str [[REG1]], [sp]
 ; FAST-NEXT:   str [[REG2]], [sp, #16]
 ; FAST-NEXT:   str [[REG3]], [sp, #24]
 ; FAST:        Ltmp
 ; FAST-NEXT:   movz  x16, #0xffff, lsl #32
 ; FAST-NEXT:   movk  x16, #0xdead, lsl #16
 ; FAST-NEXT:   movk  x16, #0xbeef
 ; FAST-NEXT:   blr x16
   %call = inttoptr i64 281474417671919 to i8*

 ; CHECK-NEXT:  blr x16
 ; FAST-LABEL:  jscall_patchpoint_codegen3:
 ; FAST:        Ltmp
 ; FAST:        orr [[REG1:x[0-9]+]], xzr, #0x2
 ; FAST-NEXT:   orr [[REG2:w[0-9]+]], wzr, #0x4
 ; FAST-NEXT:   orr [[REG3:x[0-9]+]], xzr, #0x6
 ; FAST-NEXT:   orr [[REG4:w[0-9]+]], wzr, #0x8
 ; FAST-NEXT:   movz [[REG5:x[0-9]+]], #0xa
-; FAST-NEXT:   str [[REG1]], [sp, #-64]!
+; FAST-NEXT:   str [[REG1]], [sp]
 ; FAST-NEXT:   str [[REG2]], [sp, #16]
 ; FAST-NEXT:   str [[REG3]], [sp, #24]
 ; FAST-NEXT:   str [[REG4]], [sp, #36]
 ; FAST-NEXT:   str [[REG5]], [sp, #48]
 ; FAST:        Ltmp
 ; FAST-NEXT:   movz  x16, #0xffff, lsl #32
 ; FAST-NEXT:   movk  x16, #0xdead, lsl #16
 ; FAST-NEXT:   movk  x16, #0xbeef

test/CodeGen/AArch64/arm64-patchpoint.ll

; CHECK:       ret
   tail call void (i64, i32, i8*, i32, ...) @llvm.experimental.patchpoint.void(i64 3, i32 20, i8* %resolveCall3, i32 2, i64 %p1, i64 %result)
   ret i64 %result
 }
 
 ; Caller frame metadata with stackmaps. This should not be optimized
 ; as a leaf function.
 ;
 ; CHECK-LABEL: caller_meta_leaf
-; CHECK:       mov x29, sp
-; CHECK-NEXT:  sub sp, sp, #32
+; CHECK:       sub sp, sp, #48
+; CHECK-NEXT:  stp x29, x30, [sp, #32]
+; CHECK-NEXT:  add x29, sp, #32
 ; CHECK:       Ltmp
-; CHECK:       add sp, sp, #32
+; CHECK:       add sp, sp, #48
 ; CHECK:       ret
 
 define void @caller_meta_leaf() {
 entry:
   %metadata = alloca i64, i32 3, align 8
   store i64 11, i64* %metadata
   store i64 12, i64* %metadata
   store i64 13, i64* %metadata

test/CodeGen/AArch64/arm64-shrink-wrapping.ll

 ; RUN: llc %s -o - -enable-shrink-wrap=true -disable-post-ra -disable-fp-elim | FileCheck %s --check-prefix=CHECK --check-prefix=ENABLE
 ; RUN: llc %s -o - -enable-shrink-wrap=false -disable-post-ra -disable-fp-elim | FileCheck %s --check-prefix=CHECK --check-prefix=DISABLE
 target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
 target triple = "arm64-apple-ios"
 
 
 ; Initial motivating example: Simple diamond with a call just on one side.
 ; CHECK-LABEL: foo:
 ;
 ; Compare the arguments and jump to exit.
 ; No prologue needed.
 ; ENABLE: cmp w0, w1
 ; ENABLE-NEXT: b.ge [[EXIT_LABEL:LBB[0-9_]+]]
 ;
 ; Prologue code.
-; CHECK: stp [[SAVE_SP:x[0-9]+]], [[CSR:x[0-9]+]], [sp, #-16]!
-; CHECK-NEXT: mov [[SAVE_SP]], sp
-; CHECK-NEXT: sub sp, sp, #16
+; CHECK: sub sp, sp, #32
+; CHECK-NEXT: stp [[SAVE_SP:x[0-9]+]], [[CSR:x[0-9]+]], [sp, #16]
+; CHECK-NEXT: add [[SAVE_SP]], sp, #16
 ;
 ; Compare the arguments and jump to exit.
 ; After the prologue is set.
 ; DISABLE: cmp w0, w1
 ; DISABLE-NEXT: b.ge [[EXIT_LABEL:LBB[0-9_]+]]
 ;
 ; Store %a in the alloca.
 ; CHECK: stur w0, {{\[}}[[SAVE_SP]], #-4]
 ; Set the alloca address in the second argument.
 ; CHECK-NEXT: sub x1, [[SAVE_SP]], #4
 ; Set the first argument to zero.
 ; CHECK-NEXT: mov w0, wzr
 ; CHECK-NEXT: bl _doSomething
 ;
 ; Without shrink-wrapping, epilogue is in the exit block.
 ; DISABLE: [[EXIT_LABEL]]:
 ; Epilogue code.
-; CHECK-NEXT: add sp, sp, #16
-; CHECK-NEXT: ldp x{{[0-9]+}}, [[CSR]], [sp], #16
+; CHECK-NEXT: ldp x{{[0-9]+}}, [[CSR]], [sp, #16]
+; CHECK-NEXT: add sp, sp, #32
 ;
 ; With shrink-wrapping, exit block is a simple return.
 ; ENABLE: [[EXIT_LABEL]]:
 ; CHECK-NEXT: ret
 define i32 @foo(i32 %a, i32 %b) {
   %tmp = alloca i32, align 4
   %tmp2 = icmp slt i32 %a, %b
   br i1 %tmp2, label %true, label %false

 }
 
 ; Check that we handle calls to variadic functions correctly.
 ; CHECK-LABEL: callVariadicFunc:
 ;
 ; ENABLE: cbz w0, [[ELSE_LABEL:LBB[0-9_]+]]
 ;
 ; Prologue code.
-; CHECK: stp [[CSR1:x[0-9]+]], [[CSR2:x[0-9]+]], [sp, #-16]!
-; CHECK-NEXT: mov [[NEW_SP:x[0-9]+]], sp
-; CHECK-NEXT: sub sp, sp, #48
+; CHECK: sub sp, sp, #64
+; CHECK-NEXT: stp [[CSR1:x[0-9]+]], [[CSR2:x[0-9]+]], [sp, #48]
+; CHECK-NEXT: add [[NEW_SP:x[0-9]+]], sp, #48
 ;
 ; DISABLE: cbz w0, [[ELSE_LABEL:LBB[0-9_]+]]
 ; Setup of the varags.
 ; CHECK: stp x1, x1, [sp, #32]
 ; CHECK-NEXT: stp x1, x1, [sp, #16]
 ; CHECK-NEXT: stp x1, x1, [sp]
 ; CHECK-NEXT: mov w0, w1
 ; CHECK-NEXT: bl _someVariadicFunc
 ; CHECK-NEXT: lsl w0, w0, #3
 ;
 ; DISABLE: b [[IFEND_LABEL:LBB[0-9_]+]]
 ; DISABLE: [[ELSE_LABEL]]: ; %if.else
 ; DISABLE-NEXT: lsl w0, w1, #1
 ; DISABLE: [[IFEND_LABEL]]: ; %if.end
 ;
 ; Epilogue code.
-; CHECK: add sp, sp, #48
-; CHECK-NEXT: ldp [[CSR1]], [[CSR2]], [sp], #16
+; CHECK: ldp [[CSR1]], [[CSR2]], [sp, #48]
+; CHECK-NEXT: add sp, sp, #64
 ; CHECK-NEXT: ret
 ;
 ; ENABLE: [[ELSE_LABEL]]: ; %if.else
 ; ENABLE-NEXT: lsl w0, w1, #1
 ; ENABLE-NEXT: ret
 define i32 @callVariadicFunc(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0

test/CodeGen/AArch64/fastcc.ll

 ; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu -disable-fp-elim -tailcallopt | FileCheck %s -check-prefix CHECK-TAIL
 ; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu -disable-fp-elim | FileCheck %s
 ; RUN: llc -verify-machineinstrs < %s -mtriple=aarch64-none-linux-gnu -disable-fp-elim -tailcallopt -aarch64-redzone | FileCheck %s -check-prefix CHECK-TAIL-RZ
 
 ; Without tailcallopt fastcc still means the caller cleans up the
 ; stack, so try to make sure this is respected.
 
 define fastcc void @func_stack0() {
 ; CHECK-LABEL: func_stack0:
-; CHECK: mov x29, sp
-; CHECK: str w{{[0-9]+}}, [sp, #-32]!
+; CHECK: sub sp, sp, #48
+; CHECK: add x29, sp, #32
+; CHECK: str w{{[0-9]+}}, [sp]
 
 ; CHECK-TAIL-LABEL: func_stack0:
-; CHECK-TAIL: stp x29, x30, [sp, #-16]!
-; CHECK-TAIL-NEXT: mov x29, sp
-; CHECK-TAIL: str w{{[0-9]+}}, [sp, #-32]!
+; CHECK-TAIL: sub sp, sp, #48
+; CHECK-TAIL-NEXT: stp x29, x30, [sp, #32]
+; CHECK-TAIL-NEXT: add x29, sp, #32
+; CHECK-TAIL: str w{{[0-9]+}}, [sp]
 
 
   call fastcc void @func_stack8([8 x i32] undef, i32 42)
 ; CHECK:  bl func_stack8
 ; CHECK-NOT: sub sp, sp,
 
 ; CHECK-TAIL: bl func_stack8
 ; CHECK-TAIL: sub sp, sp, #16

 ; CHECK: bl func_stack0
 ; CHECK-NOT: sub sp, sp
 
 
 ; CHECK-TAIL: bl func_stack0
 ; CHECK-TAIL-NOT: sub sp, sp
 
   ret void
-; CHECK: add sp, sp, #32
-; CHECK-NEXT: ldp     x29, x30, [sp], #16
+; CHECK: ldp     x29, x30, [sp, #32]
+; CHECK-NEXT: add sp, sp, #48
 ; CHECK-NEXT: ret
 
 
-; CHECK-TAIL: add sp, sp, #32
-; CHECK-TAIL-NEXT: ldp     x29, x30, [sp], #16
+; CHECK-TAIL: ldp     x29, x30, [sp, #32]
+; CHECK-TAIL-NEXT: add sp, sp, #48
 ; CHECK-TAIL-NEXT: ret
 }
 
 define fastcc void @func_stack8([8 x i32], i32 %stacked) {
 ; CHECK-LABEL: func_stack8:
-; CHECK: stp x29, x30, [sp, #-16]!
-; CHECK: mov x29, sp
-; CHECK: str w{{[0-9]+}}, [sp, #-32]!
+; CHECK: sub sp, sp, #48
+; CHECK: stp x29, x30, [sp, #32]
+; CHECK: add x29, sp, #32
+; CHECK: str w{{[0-9]+}}, [sp]
 
 
 ; CHECK-TAIL-LABEL: func_stack8:
-; CHECK-TAIL: stp x29, x30, [sp, #-16]!
-; CHECK-TAIL: mov x29, sp
-; CHECK-TAIL: str w{{[0-9]+}}, [sp, #-32]!
+; CHECK-TAIL: sub sp, sp, #48
+; CHECK-TAIL: stp x29, x30, [sp, #32]
+; CHECK-TAIL: add x29, sp, #32
+; CHECK-TAIL: str w{{[0-9]+}}, [sp]
 
 
   call fastcc void @func_stack8([8 x i32] undef, i32 42)
 ; CHECK:  bl func_stack8
 ; CHECK-NOT: sub sp, sp,
 
 
 ; CHECK-TAIL: bl func_stack8
; CHECK-TAIL: sub sp, sp, #32
   call fastcc void @func_stack0()
 ; CHECK: bl func_stack0
 ; CHECK-NOT: sub sp, sp
 
 ; CHECK-TAIL: bl func_stack0
 ; CHECK-TAIL-NOT: sub sp, sp
 
   ret void
-; CHECK: add sp, sp, #32
-; CHECK-NEXT: ldp     x29, x30, [sp], #16
+; CHECK-NEXT: ldp     x29, x30, [sp, #32]
+; CHECK: add sp, sp, #48
 ; CHECK-NEXT: ret
 
 
-; CHECK-TAIL: add sp, sp, #32
-; CHECK-TAIL-NEXT: ldp     x29, x30, [sp], #16
-; CHECK-TAIL-NEXT: add     sp, sp, #16
+; CHECK-TAIL: ldp     x29, x30, [sp, #32]
+; CHECK-TAIL-NEXT: add     sp, sp, #64
 ; CHECK-TAIL-NEXT: ret
 }
 
 define fastcc void @func_stack32([8 x i32], i128 %stacked0, i128 %stacked1) {
 ; CHECK-LABEL: func_stack32:
-; CHECK: mov x29, sp
+; CHECK: add x29, sp, #32
 
 ; CHECK-TAIL-LABEL: func_stack32:
-; CHECK-TAIL: mov x29, sp
+; CHECK-TAIL: add x29, sp, #32
 
 
   call fastcc void @func_stack8([8 x i32] undef, i32 42)
 ; CHECK:  bl func_stack8
 ; CHECK-NOT: sub sp, sp,
 
 ; CHECK-TAIL: bl func_stack8
 ; CHECK-TAIL: sub sp, sp, #16

 ; CHECK: bl func_stack0
 ; CHECK-NOT: sub sp, sp
 
 
 ; CHECK-TAIL: bl func_stack0
 ; CHECK-TAIL-NOT: sub sp, sp
 
   ret void
-; CHECK: add sp, sp, #32
-; CHECK-NEXT: ldp     x29, x30, [sp], #16
+; CHECK: ldp     x29, x30, [sp, #32]
+; CHECK-NEXT: add sp, sp, #48
 ; CHECK-NEXT: ret
 
-; CHECK-TAIL: add sp, sp, #32
-; CHECK-TAIL-NEXT: ldp     x29, x30, [sp], #16
-; CHECK-TAIL-NEXT: add     sp, sp, #32
+; CHECK-TAIL: ldp     x29, x30, [sp, #32]
+; CHECK-TAIL-NEXT: add     sp, sp, #80
 ; CHECK-TAIL-NEXT: ret
 }
 
 ; Check that arg stack pop is done after callee-save restore when no frame pointer is used.
 define fastcc void @func_stack32_leaf([8 x i32], i128 %stacked0, i128 %stacked1) {
 ; CHECK-LABEL: func_stack32_leaf:
 ; CHECK: str     x20, [sp, #-16]!
 ; CHECK: nop
; CHECK-TAIL-RZ-NEXT: ret
   ; Make sure there is a callee-save register to save/restore.
   call void asm sideeffect "nop", "~{x20}"() nounwind
   ret void
 }
 
 ; Check that arg stack pop is done after callee-save restore when no frame pointer is used.
 define fastcc void @func_stack32_leaf_local([8 x i32], i128 %stacked0, i128 %stacked1) {
 ; CHECK-LABEL: func_stack32_leaf_local:
-; CHECK: str     x20, [sp, #-16]!
-; CHECK-NEXT: sub     sp, sp, #16
+; CHECK: sub     sp, sp, #32
+; CHECK-NEXT: str     x20, [sp, #16]
 ; CHECK: nop
 ; CHECK-NEXT: //NO_APP
-; CHECK-NEXT: add     sp, sp, #16
-; CHECK-NEXT: ldr     x20, [sp], #16
+; CHECK-NEXT: ldr     x20, [sp, #16]
+; CHECK-NEXT: add     sp, sp, #32
 ; CHECK-NEXT: ret
 
 ; CHECK-TAIL-LABEL: func_stack32_leaf_local:
-; CHECK-TAIL: str     x20, [sp, #-16]!
-; CHECK-TAIL-NEXT: sub     sp, sp, #16
+; CHECK-TAIL: sub     sp, sp, #32
+; CHECK-TAIL-NEXT: str     x20, [sp, #16]
 ; CHECK-TAIL: nop
 ; CHECK-TAIL-NEXT: //NO_APP
-; CHECK-TAIL-NEXT: add     sp, sp, #16
-; CHECK-TAIL-NEXT: ldr     x20, [sp], #16
-; CHECK-TAIL-NEXT: add     sp, sp, #32
+; CHECK-TAIL-NEXT: ldr     x20, [sp, #16]
+; CHECK-TAIL-NEXT: add     sp, sp, #64
 ; CHECK-TAIL-NEXT: ret
 
 ; CHECK-TAIL-RZ-LABEL: func_stack32_leaf_local:
 ; CHECK-TAIL-RZ: str     x20, [sp, #-16]!
 ; CHECK-TAIL-RZ-NOT: sub     sp, sp
 ; CHECK-TAIL-RZ: nop
 ; CHECK-TAIL-RZ-NEXT: //NO_APP
 ; CHECK-TAIL-RZ-NEXT: ldr     x20, [sp], #16

test/CodeGen/AArch64/func-calls.ll

 ; CHECK-LABEL: check_stack_args:
   call i32 @struct_on_stack(i8 0, i16 12, i32 42, i64 99, i128 1,
                             i32* @var32, %myStruct* byval @varstruct,
                             i32 999, double 1.0)
   ; Want to check that the final double is passed in registers and
   ; that varstruct is passed on the stack. Rather dependent on how a
   ; memcpy gets created, but the following works for now.
 
-; CHECK-DAG: str {{q[0-9]+}}, [sp, #-16]
+; CHECK-DAG: str {{q[0-9]+}}, [sp]
 ; CHECK-DAG: fmov d[[FINAL_DOUBLE:[0-9]+]], #1.0
 ; CHECK: mov v0.16b, v[[FINAL_DOUBLE]].16b
 
-; CHECK-NONEON-DAG: str {{q[0-9]+}}, [sp, #-16]!
+; CHECK-NONEON-DAG: str {{q[0-9]+}}, [sp]
 ; CHECK-NONEON-DAG: fmov d[[FINAL_DOUBLE:[0-9]+]], #1.0
 ; CHECK-NONEON: fmov d0, d[[FINAL_DOUBLE]]
 
 ; CHECK: bl struct_on_stack
 ; CHECK-NOFP-NOT: fmov
 
   call void @stacked_fpu(float -1.0, double 1.0, float 4.0, float 2.0,
                          float -2.0, float -8.0, float 16.0, float 1.0,

test/CodeGen/AArch64/tailcall-implicit-sret.ll

-; RUN: llc < %s -mtriple arm64-apple-darwin -aarch64-load-store-opt=false -asm-verbose=false | FileCheck %s
+; RUN: llc < %s -mtriple arm64-apple-darwin -aarch64-load-store-opt=false -disable-post-ra -asm-verbose=false | FileCheck %s
 ; Disable the load/store optimizer to avoid having LDP/STPs and simplify checks.
 
 target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
 
 ; Check that we don't try to tail-call with an sret-demoted return.
 
 declare i1024 @test_sret() #0
 

test/DebugInfo/AArch64/prologue_end.ll

 ; RUN: llc -disable-fp-elim -O0 %s -mtriple aarch64-apple-darwin -o - | FileCheck %s
 
 ; int func(void);
 ; void prologue_end_test() {
 ;   func();
 ;   func();
 ; }
 
 define void @prologue_end_test() nounwind uwtable !dbg !4 {
   ; CHECK: prologue_end_test:
   ; CHECK: .cfi_startproc
-  ; CHECK: stp x29, x30
-  ; CHECK: mov x29, sp
   ; CHECK: sub sp, sp
+  ; CHECK: stp x29, x30
+  ; CHECK: add x29, sp
   ; CHECK: .loc 1 3 3 prologue_end
   ; CHECK: bl _func
   ; CHECK: bl _func
 entry:
   %call = call i32 @func(), !dbg !11
   %call1 = call i32 @func(), !dbg !12
   ret void, !dbg !13
 }