[PowerPC] Try to move the stack pointer update instruction later in the prologue and earlier in the epilogue (Version 2)

[stefanp]

Summary

This is the second attempt to move the stdu instruction in the prologue and epilogue.
The first attempt was D41737 but that contained a bug that was exposed by "make test" in libvpx.

In order to fix that issue the transformation is turned off for functions that require frame index scavenging. In order to determine that information the requiresFrameIndexScavenging had to be implemented for PowerPC.

[lei]

lib/Target/PowerPC/PPCRegisterInfo.cpp

406

maybe an early exit here instead...
if (FrIdx >=0) continue;

[syzaara]

lib/Target/PowerPC/PPCFrameLowering.cpp

971

form -> from

981

Can use a range based for loop.

1527

Can use range based for loop here as well.

[stefanp]

Fixed the issues mentioned in previous comments.

[nemanjai]

lib/Target/PowerPC/PPCRegisterInfo.cpp

403

We use spaces around binary/assignment operators. Maybe just clang-format-diff the patch.

410

I'm not a fan of this solution. It provides yet another place we check for the register class for a physical register without a clear explanation for *why* we care about the register class.

I'd much prefer a unified solution between StoreRegToStackSlot(), isStoreToStackSlot() and requiresFrameIndexScavenging(). What I'm thinking is something along the lines of:

static const unsigned OpcodesForSpills[] = { PPC::STD, PPC::STW, ... };
PPCInstrInfo::getOpcodeForSpill(unsigned Reg, const TargetRegisterClass *RC = nullptr);

That way we'd have a single definitive list of opcodes that are used for spilling registers and wouldn't have to keep this delicate dance of keeping multiple functions in sync.

• isStoreToStackSlot() would just check the array to see if its opcode is in there
• StoreRegToStackSlot() and requiresFrameIndexScavenging() would use getOpcodeForSpill() with the register class or physical register respectively
• getOpcodeForSpill() would just compute the index into the array based on the register class and target features and return the respective element

Of course, it doesn't have to be done that way, but any solution that would unify this would be good.

[nemanjai]

lib/Target/PowerPC/PPCFrameLowering.cpp

986

I imagine this is impossible, but it may not be a terrible idea to assert that you haven't somehow iterated past a terminator.

1518

There are a few places where there are extra spaces in comments. But I'm sure that it'll all get fixed up when you run clang-format-diff.

1529

This probably applies both here and above, but doesn't the condition FrIdx > 0 actually mean we should not do this at all? I'm not sure if it's possible, but what if you had frame indices that are say { negative, negative, positive, negative } as you iterate? Wouldn't you want to leave the function alone then?

Also, what is it about FrIdx == 0 that prevents you from moving the stack pointer update up past it? For that matter, there should be a comment why we require the condition at all.

Finally, please add a comment as to why it is not necessary to manually update the offsets for the CSR spills and restores - we just move the stack ptr update and something else figures out the offsets.

[stefanp]

After some code cleanup that was required for requiresFrameIndexScavenging here is the new version of this patch.
It should take into consideration all of the reviewer's comments.

[nemanjai]

Overall I'm very happy with how this patch looks now. The cleanup really allowed this patch to flow much more clearly. However, I think you still haven't addressed the comment https://reviews.llvm.org/D42590#inline-374441

lib/Target/PowerPC/PPCRegisterInfo.cpp

338

Indices.

[inouehrs]

lib/Target/PowerPC/PPCFrameLowering.cpp

1688

If the latency of MTLR is the critical path in the epilogue, can we move this load before stack pointer update (with adjusted offset) to hide the latency further? (But this can be a separate patch.)

[stefanp]

lib/Target/PowerPC/PPCFrameLowering.cpp

1688

That's a good point.
When I did the original performance tests I had moved the the mtlr past the callee restores and not all the way past the stack pointer update.
I'm going to update this patch without that change and then I'll put in another patch with that change alone.

[stefanp]

I've updated the negative frame indices to use isFixedObjectIndex which is cleaner.

[lei]

lib/Target/PowerPC/PPCFrameLowering.cpp

984

This if can either be merged into the one below... if (FrIdx <0 && MFI.isFixedObjectIndex ...)
or do an early exit of this loop iteration if (FrIdx >=0) continue;

[lei]

lib/Target/PowerPC/PPCFrameLowering.cpp

1518

I don't think you addressed this issue with the extra spaces in comments. From what I can see we don't put an extra space after // when continuing a comment from the previous line.

1529

merge this if statement with the nested if

[stefanp]

Address comments from previous review.

[inouehrs]

lib/Target/PowerPC/PPCFrameLowering.cpp

1359

Since we have only one MovingStackUpdateDown, all CSIs are modified back even only a part of them were update above.

[nemanjai]

lib/Target/PowerPC/PPCFrameLowering.cpp

1359

@stefanp Can you respond to this comment? Is this a problem? Can we get into a situation where we end up with incorrect offsets if not all of the callee-saved spills/restores have been moved/updated?

[stefanp]

I'm sorry it took so long for me to look at this review.

I think you are correct Hiroshi in that what I was doing is not safe. I've added a check where the operation is aborted if not all of the Callee Saved Info is updated at the same time.
Also added a couple of missing lines to getStoreOpcodeForSpill and getLoadOpcodeForSpill that was missed when the Signal Processing Engine work was done. This was exposed by this patch.

[nemanjai]

I think this is close to ready but there are a few comments that have to be addressed. Also, @hfinkel and @inouehrs do you see anything that needs to be handled in addition to what I commented on?

lib/Target/PowerPC/PPCFrameLowering.cpp

975

No need to make the reader work it out. Add something like this to the comment:

(i.e. a frame size of 32768 bytes satisfies isLargeFrame here,
but not in the epilogue inserter).

976

Why do we need this? How come we can't just use RegInfo defined above? It should be the same object should it not?

984

This needs a comment explaining why we need this check and also why failing this check with one of the Callee Saved Register spills doesn't require us to abort the operation the way failing the subsequent check does.

1524

The condition here looks like a repeat of the one in the prologue inserter. Can we actually extract this into a function that can be queried by both? Then this can be changed to something like:
if (stackUpdateCanBeMoved()) or something equivalently simple.

1529

Same comment as a similar check in the prologue inserter.

lib/Target/PowerPC/PPCRegisterInfo.cpp

327

"saved info" doesn't really mean anything in this context. The CalleeSavedInfo type is a structure containing information about Callee Saved Registers.

331

Same as above.

334

Formatting.

347

I am not sure I really follow when we need FI scavenging...

I was under the impression that we might need it if:

• The frame size is too large so we need to use an X-Form store/load for the spill/restore
• The alignment of the spill/restore is lower than what is required by the D-Form (4 for DS-Form, 16 for DQ-Form)
• The only opcode for the spill/restore is an X-Form

This if statement certainly seems to accomplish the last of those but I don't see anything that accomplishes the other two. The first can certainly be an early exit before the loop.

[hfinkel]

lib/Target/PowerPC/PPCFrameLowering.cpp

975

I think this is restricted to frames where the relevant saves (i.e., whatever is stored before the stack-pointer update is performed) must be less than the red-zone size, which at least under ELF v2 is 288 bytes (see ABI spec: 2.2.2.4 Protected Zone). Maybe this is handled below somehow, but so long as we have a comment here explaining everything, we should explain how this factors in as well.

It is, however, not clear to me where you are checking this below. If you're not, I'm pretty sure that's a significant problem. If we're saving more than just GPRs, we can easily exceed 288 bytes. An explicit test case this would also be a very-good idea.

[nemanjai]

lib/Target/PowerPC/PPCFrameLowering.cpp

975

We are not actually spilling into the Red Zone here (or at least not if we wouldn't in the original code) but into the current stack frame. This just moves the stack pointer update instruction around. However, the idea is to save into the same stack frame/slot by adjusting the offset in each spill to account for how much the stack pointer will be updated by.
Namely, change something like (the actual values have nothing to do with reality, just an illustration):

stdu 1, -48(1)
std 29, 16(1)
std 30, 8(1)

to something like

std 29, -32(1)
std 30, -40(1)
stdu 1, -48(1)

That way all the stores can be dispatched in parallel rather than the CSR spills for R29 and R30 waiting for the update to R1 to actually complete. But we are still saving everything to the exact same memory location.

But I agree that we should add further testing to clearly illustrate such a change in behaviour.

[hfinkel]

lib/Target/PowerPC/PPCFrameLowering.cpp

975

We are not actually spilling into the Red Zone here (or at least not if we wouldn't in the original code) but into the current stack frame.

But we are, as you've illustrated with your example...

stdu 1, -48(1)
std 29, 16(1)
std 30, 8(1)

here we adjust the stack pointer first, and all stores are to addresses above the stack pointer.

std 29, -32(1)
std 30, -40(1)
stdu 1, -48(1)

here we adjust it afterward, and so the first two stores are below the current stack pointer. That's into the red zone. It becomes not the red zone only after you adjust the stack pointer. But, if we have:

std 29, -332(1)
std 30, -340(1)
; interrupt handler runs here!
stdu 1, -348(1)

(I made the numbers bigger to illustrate the issue)

then we have a potential problem. The FP save area can be up to 256 bytes, and then comes the GPR area. Then the vector area (which can be up to 512 bytes). Thus, we can certainly go beyond 288 bytes.

[stefanp]

Addressed reviewer comments.
Limited optimization to cases where the fixed called saved regs fit in the red zone of 288 bytes.

[jsji]

Some quick notes.

lib/Target/PowerPC/PPCFrameLowering.cpp

786

Can we use getRedZoneSize instead of hardcoded 288 here?

790

We should check ABI before checking framesize? As the red zone size is different for different ABIs.
eg: DarwinABI has a 224-byte red zone. PPC32 SVR4ABI(Non-DarwinABI) has no red zone and PPC64 SVR4ABI has a 288-byte red zone.

[stefanp]

Addressed comments from Jinsong.

I've removed the magic number and I'm using getRedZoneSize() now.
I've also moved the isPPC64() check up as well as adding an isELFv2ABI() check.

[nemanjai]

Please add the following test cases:

1. A stack frame that is just a bit larger than required
2. A stack frame with a CSR save that is not a fixed object
3. A stack frame where we have to use scavenging
4. A test case where everything fits and it saves multiple register types (GPR, FPR, VSR)

I would recommend that you produce the CHECK directives using the script in utils/ and commit it first as an NFC patch so that this patch clearly shows the differences in the code emitted.

lib/Target/PowerPC/PPCFrameLowering.cpp

515

This seems to duplicate code. Can we not implement the update version in terms of this function? Seems that the only thing that slightly complicates things is the computation of maxCallFrameSize so the update form should probably have that as a pointer output parameter with a nullptr default.

777

Combine the subtarget feature check early exits.

801

s/mode/move

809

It is not clear why? I presume because the scavenger can add further spills? In any case, the comment should mention why.
Also:
return !RegInfo->requiresFrameIndexScavenging(MF);

973

Won't we be broken if we do this to some of the CSR's and then we bail because one of them doesn't satisfy this condition below? It seems that we need to either defer any offset updates until we are sure we're moving the stack ptr update or keep track of any we updated so we can backtrack if we need to.

lib/Target/PowerPC/PPCFrameLowering.h

78

callee saved instructions sounds odd. Perhaps CSR save/restore code?

lib/Target/PowerPC/PPCRegisterInfo.cpp

335

I think a constant such as 0x7FFF communicates intent more clearly. Furthermore, a condition such as this may clearly illustrate that we can't have any higher bits set:
if (FrameSize & ~0x7FFF)

[stefanp]

Committed revision 353994 with the test cases as NFC.
Will update the patch for the remaining comments soon...

[stefanp]

Fixed according to review comments.

[nemanjai]

LGTM with the minor comments addressed on the commit.

Looking at the test case has actually made me think of something else that might be worthwhile following up on...
Namely, I think there would be general benefit of breaking the dependence of the link register restore from the stack pointer update when we are not moving the stack pointer update.

For example, the following change might be a significant improvement in the epilogue:

CSR restores
addi 1, 1, <Value>
ld 0, 16(1)
mtlr 0

to

CSR restores
ld 0, <Value> + 16(1)
mtlr 0
addi 1, 1, <Value>

This would allow the scheduler to schedule the ld->mtlr sequence for restoring the link register in a way that the CSR restores hide the latency of the sequence. Furthermore, that should be a safe transformation regardless of the size of the stack frame since any interrupt handlers that run during the epilogue of a function must restore the LR when resuming control.

lib/Target/PowerPC/PPCFrameLowering.cpp

784

s/until the stack pointer is moved/until the stack pointer is updated

797

s/mode/move

Also, a comment that quite literally just says in words what the code clearly says isn't useful. It is more useful to say why we need the check. Perhaps something like:

// Calls to fast_cc functions use different rules for passing parameters on
// the stack from the ABI and using PIC base in the function imposes
// similar restrictions to using the base pointer. It is not generally safe
// to move the stack pointer updatein these situations.

It might be reasonable to also combine these into a single condition, but this is completely a matter of personal preference.

1528

s/stack update pointer/update of the stack pointer

1535

// Abort the operation as we can't update all CSR restores.

lib/Target/PowerPC/PPCFrameLowering.h

class PPCFrameLowering: public TargetFrameLowering {
 
   /**
    * Create branch instruction for PPC::TCRETURN* (tail call return)
    *
    * \param[in] MBB that is terminated by PPC::TCRETURN*
    */
   void createTailCallBranchInstr(MachineBasicBlock &MBB) const;
 
+  /**
+    * Check if the conditions are correct to allow for the stack update
+    * to be moved past the callee saved instructions.
+    */
+  bool stackUpdateCanBeMoved(MachineFunction &MF) const;
+
 public:
   PPCFrameLowering(const PPCSubtarget &STI);
 
-  unsigned determineFrameLayout(MachineFunction &MF,
-                                bool UpdateMF = true,
+  /**
+   * Determine the frame layout and update the machine function.
+   */
+  unsigned determineFrameLayoutAndUpdate(MachineFunction &MF,
+                                         bool UseEstimate = false) const;
+
+  /**
+   * Determine the frame layout but do not update the machine function.
+   * The MachineFunction object can be const in this case as it is not
+   * modified.
+   */
+  unsigned determineFrameLayout(const MachineFunction &MF,
                                 bool UseEstimate = false) const;
 
   /// emitProlog/emitEpilog - These methods insert prolog and epilog code into
   /// the function.
   void emitPrologue(MachineFunction &MF, MachineBasicBlock &MBB) const override;
   void emitEpilogue(MachineFunction &MF, MachineBasicBlock &MBB) const override;
 
   bool hasFP(const MachineFunction &MF) const override;

lib/Target/PowerPC/PPCFrameLowering.cpp

static bool MustSaveLR(const MachineFunction &MF, unsigned LR) {
   // We need a save/restore of LR if there is any def of LR (which is
   // defined by calls, including the PIC setup sequence), or if there is
   // some use of the LR stack slot (e.g. for builtin_return_address).
   // (LR comes in 32 and 64 bit versions.)
   MachineRegisterInfo::def_iterator RI = MF.getRegInfo().def_begin(LR);
   return RI !=MF.getRegInfo().def_end() || MFI->isLRStoreRequired();
 }
 
-/// determineFrameLayout - Determine the size of the frame and maximum call
-/// frame size.
-unsigned PPCFrameLowering::determineFrameLayout(MachineFunction &MF,
-                                                bool UpdateMF,
+/// determineFrameLayoutAndUpdate - Determine the size of the frame and maximum
+/// call frame size. Update the MachineFunction object with the stack size.
+unsigned
+PPCFrameLowering::determineFrameLayoutAndUpdate(MachineFunction &MF,
                                                 bool UseEstimate) const {
   MachineFrameInfo &MFI = MF.getFrameInfo();
 
   // Get the number of bytes to allocate from the FrameInfo
   unsigned FrameSize =
     UseEstimate ? MFI.estimateStackSize(MF) : MFI.getStackSize();
 
   // Get stack alignments. The frame must be aligned to the greatest of these:
PPCFrameLowering::determineFrameLayoutAndUpdate(MachineFunction &MF,
   // Note: for PPC32 SVR4ABI (Non-DarwinABI), we can still generate stackless
   // code if all local vars are reg-allocated.
   bool FitsInRedZone = FrameSize <= Subtarget.getRedZoneSize();
 
   // Check whether we can skip adjusting the stack pointer (by using red zone)
   if (!DisableRedZone && CanUseRedZone && FitsInRedZone) {
     NumNoNeedForFrame++;
     // No need for frame
-    if (UpdateMF)
-      MFI.setStackSize(0);
     return 0;
   }
 
   // Get the maximum call frame size of all the calls.
   unsigned maxCallFrameSize = MFI.getMaxCallFrameSize();
 
   // Maximum call frame needs to be at least big enough for linkage area.
   unsigned minCallFrameSize = getLinkageSize();
   maxCallFrameSize = std::max(maxCallFrameSize, minCallFrameSize);
 
   // If we have dynamic alloca then maxCallFrameSize needs to be aligned so
   // that allocations will be aligned.
   if (MFI.hasVarSizedObjects())
     maxCallFrameSize = (maxCallFrameSize + AlignMask) & ~AlignMask;
 
   // Update maximum call frame size.
-  if (UpdateMF)
-    MFI.setMaxCallFrameSize(maxCallFrameSize);
 
   // Include call frame size in total.
   FrameSize += maxCallFrameSize;
 
   // Make sure the frame is aligned.
   FrameSize = (FrameSize + AlignMask) & ~AlignMask;
 
   // Update frame info.
-  if (UpdateMF)
-    MFI.setStackSize(FrameSize);
 
   return FrameSize;
 }
 
+/// determineFrameLayout - Determine the size of the frame and maximum call
+/// frame size.
+unsigned PPCFrameLowering::determineFrameLayout(const MachineFunction &MF,
+                                                bool UseEstimate) const {
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+
+  // Get the number of bytes to allocate from the FrameInfo
+  unsigned FrameSize =
+    UseEstimate ? MFI.estimateStackSize(MF) : MFI.getStackSize();
+
+  // Get stack alignments. The frame must be aligned to the greatest of these:
+  unsigned TargetAlign = getStackAlignment(); // alignment required per the ABI
+  unsigned MaxAlign = MFI.getMaxAlignment(); // algmt required by data in frame
+  unsigned AlignMask = std::max(MaxAlign, TargetAlign) - 1;
+
+  const PPCRegisterInfo *RegInfo = Subtarget.getRegisterInfo();
+
+  unsigned LR = RegInfo->getRARegister();
+  bool DisableRedZone = MF.getFunction().hasFnAttribute(Attribute::NoRedZone);
+  bool CanUseRedZone = !MFI.hasVarSizedObjects() && // No dynamic alloca.
+                       !MFI.adjustsStack() &&       // No calls.
+                       !MustSaveLR(MF, LR) &&       // No need to save LR.
+                       !RegInfo->hasBasePointer(MF); // No special alignment.
+
+  // Note: for PPC32 SVR4ABI (Non-DarwinABI), we can still generate stackless
+  // code if all local vars are reg-allocated.
+  bool FitsInRedZone = FrameSize <= Subtarget.getRedZoneSize();
+
+  // Check whether we can skip adjusting the stack pointer (by using red zone)
+  if (!DisableRedZone && CanUseRedZone && FitsInRedZone) {
+    // No need for frame
+    return 0;
+  }
+
+  // Get the maximum call frame size of all the calls.
+  unsigned maxCallFrameSize = MFI.getMaxCallFrameSize();
+
+  // Maximum call frame needs to be at least big enough for linkage area.
+  unsigned minCallFrameSize = getLinkageSize();
+  maxCallFrameSize = std::max(maxCallFrameSize, minCallFrameSize);
+
+  // If we have dynamic alloca then maxCallFrameSize needs to be aligned so
+  // that allocations will be aligned.
+  if (MFI.hasVarSizedObjects())
+    maxCallFrameSize = (maxCallFrameSize + AlignMask) & ~AlignMask;
+
+  // Include call frame size in total.
+  FrameSize += maxCallFrameSize;
+
+  // Make sure the frame is aligned.
+  FrameSize = (FrameSize + AlignMask) & ~AlignMask;
+
+  return FrameSize;
+}
+
 // hasFP - Return true if the specified function actually has a dedicated frame
 // pointer register.
 bool PPCFrameLowering::hasFP(const MachineFunction &MF) const {
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   // FIXME: This is pretty much broken by design: hasFP() might be called really
   // early, before the stack layout was calculated and thus hasFP() might return
   // true or false here depending on the time of call.
   return (MFI.getStackSize()) && needsFP(MF);

 // register is available, we can adjust for that by not overlapping the spill
 // code. However, if we need to realign the stack (i.e. have a base pointer)
 // and the stack frame is large, we need two scratch registers.
 bool
 PPCFrameLowering::twoUniqueScratchRegsRequired(MachineBasicBlock *MBB) const {
   const PPCRegisterInfo *RegInfo = Subtarget.getRegisterInfo();
   MachineFunction &MF = *(MBB->getParent());
   bool HasBP = RegInfo->hasBasePointer(MF);
-  unsigned FrameSize = determineFrameLayout(MF, false);
+  unsigned FrameSize = determineFrameLayout(MF);
   int NegFrameSize = -FrameSize;
   bool IsLargeFrame = !isInt<16>(NegFrameSize);
   MachineFrameInfo &MFI = MF.getFrameInfo();
   unsigned MaxAlign = MFI.getMaxAlignment();
   bool HasRedZone = Subtarget.isPPC64() || !Subtarget.isSVR4ABI();
 
   return (IsLargeFrame || !HasRedZone) && HasBP && MaxAlign > 1;
 }
 
 bool PPCFrameLowering::canUseAsPrologue(const MachineBasicBlock &MBB) const {
   MachineBasicBlock *TmpMBB = const_cast<MachineBasicBlock *>(&MBB);
 
   return findScratchRegister(TmpMBB, false,
                              twoUniqueScratchRegsRequired(TmpMBB));
 }
 
 bool PPCFrameLowering::canUseAsEpilogue(const MachineBasicBlock &MBB) const {
   MachineBasicBlock *TmpMBB = const_cast<MachineBasicBlock *>(&MBB);
 
   return findScratchRegister(TmpMBB, true);
 }
 
+bool PPCFrameLowering::stackUpdateCanBeMoved(MachineFunction &MF) const {
+  const PPCRegisterInfo *RegInfo = Subtarget.getRegisterInfo();
+  PPCFunctionInfo *FI = MF.getInfo<PPCFunctionInfo>();
+
+  // Abort if there is no register info or function info.
+  if (!RegInfo || !FI)
+    return false;
+
+  // Only try to move the stack update on PPC64.
+  if (!Subtarget.isPPC64())
+    return false;
+
+  // Only move the stack update on ELFv2 ABI.
+  if (!Subtarget.isELFv2ABI())
+    return false;
+
+  // Check the frame size first and return false if it does not fit the
+  // requirements.
+  // We need a non-zero frame size as well as a frame that will fit in the red
+  // zone. This is because by moving the stack pointer update we are now storing
+  // to the red zone until the stack pointer is moved. If we get an interrupt
+  // inside the prologue but before the stack update we now have a number of
+  // stores to the red zone and those stores must all fit.
+  MachineFrameInfo &MFI = MF.getFrameInfo();
+  unsigned FrameSize = MFI.getStackSize();
+  if (!FrameSize || FrameSize > Subtarget.getRedZoneSize())
+    return false;
+
+  // Frame pointers and base pointers complicate matters so don't do anything
+  // if we have them.
+  if (hasFP(MF) || RegInfo->hasBasePointer(MF))
+    return false;
+
+  // Cannot mode the stack pointer if we have a fast call or we use the PIC
+  // base.
+  if (FI->hasFastCall() || FI->usesPICBase())
+    return false;
+
+  bool RequiresScavenging = RegInfo->requiresFrameIndexScavenging(MF);
+
+  // Finally we can move the stack update if we do not require regiser
+  // scavenging.
+  return !RequiresScavenging;
+}
+
 void PPCFrameLowering::emitPrologue(MachineFunction &MF,
                                     MachineBasicBlock &MBB) const {
   MachineBasicBlock::iterator MBBI = MBB.begin();
   MachineFrameInfo &MFI = MF.getFrameInfo();
   const PPCInstrInfo &TII = *Subtarget.getInstrInfo();
   const PPCRegisterInfo *RegInfo = Subtarget.getRegisterInfo();
 
   MachineModuleInfo &MMI = MF.getMMI();
for (unsigned i = 0; MBBI != MBB.end(); ++i, ++MBBI) {
         break;
       }
     }
 
   // Move MBBI back to the beginning of the prologue block.
   MBBI = MBB.begin();
 
   // Work out frame sizes.
-  unsigned FrameSize = determineFrameLayout(MF);
+  unsigned FrameSize = determineFrameLayoutAndUpdate(MF);
   int NegFrameSize = -FrameSize;
   if (!isInt<32>(NegFrameSize))
     llvm_unreachable("Unhandled stack size!");
 
   if (MFI.isFrameAddressTaken())
     replaceFPWithRealFP(MF);
 
   // Check if the link register (LR) must be saved.
void PPCFrameLowering::emitPrologue(MachineFunction &MF,
 
   // Frames of 32KB & larger require special handling because they cannot be
   // indexed into with a simple STDU/STWU/STD/STW immediate offset operand.
   bool isLargeFrame = !isInt<16>(NegFrameSize);
 
   assert((isPPC64 || !MustSaveCR) &&
          "Prologue CR saving supported only in 64-bit mode");
 
+  // Check if we can move the stack update instruction (stdu) down the prologue
+  // past the callee saves. Hopefully this will avoid the situation where the
+  // saves are waiting for the update on the store with update to complete.
+  MachineBasicBlock::iterator StackUpdateLoc = MBBI;
+  bool MovingStackUpdateDown = false;
+
+  // Check if we can move the stack update.
+  if (stackUpdateCanBeMoved(MF)) {
+    const std::vector<CalleeSavedInfo> &Info = MFI.getCalleeSavedInfo();
+    for (CalleeSavedInfo CSI : Info) {
+      int FrIdx = CSI.getFrameIdx();
+      // If the frame index is not negative the callee saved info belongs to a
+      // stack object that is not a fixed stack object. We ignore non-fixed
+      // stack objects because we won't move the stack update pointer past them.
+      if (FrIdx >= 0)
+        continue;
+
+      if (MFI.isFixedObjectIndex(FrIdx) && MFI.getObjectOffset(FrIdx) < 0) {
+        MFI.setObjectOffset(FrIdx, MFI.getObjectOffset(FrIdx) + NegFrameSize);
+        StackUpdateLoc++;
+        MovingStackUpdateDown = true;
+      } else {
+        // We need all of the Frame Indices to meet these conditions.
+        // If they do not, abort the whole operation.
+        StackUpdateLoc = MBBI;
+        MovingStackUpdateDown = false;
+        break;
+      }
+    }
+  }
+
   // If we need to spill the CR and the LR but we don't have two separate
   // registers available, we must spill them one at a time
   if (MustSaveCR && SingleScratchReg && MustSaveLR) {
     // In the ELFv2 ABI, we are not required to save all CR fields.
     // If only one or two CR fields are clobbered, it is more efficient to use
     // mfocrf to selectively save just those fields, because mfocrf has short
     // latency compares to mfcr.
     unsigned MfcrOpcode = PPC::MFCR8;
     unsigned CrState = RegState::ImplicitKill;
if (HasRedZone) {
     if (HasBP)
       BuildMI(MBB, MBBI, dl, StoreInst)
         .addReg(BPReg)
         .addImm(BPOffset)
         .addReg(SPReg);
   }
 
   if (MustSaveLR)
-    BuildMI(MBB, MBBI, dl, StoreInst)
+    BuildMI(MBB, StackUpdateLoc, dl, StoreInst)
       .addReg(ScratchReg, getKillRegState(true))
       .addImm(LROffset)
       .addReg(SPReg);
 
   if (MustSaveCR &&
       !(SingleScratchReg && MustSaveLR)) { // will only occur for PPC64
     assert(HasRedZone && "A red zone is always available on PPC64");
     BuildMI(MBB, MBBI, dl, TII.get(PPC::STW8))
if (HasBP && MaxAlign > 1) {
 
     BuildMI(MBB, MBBI, dl, StoreUpdtIdxInst, SPReg)
       .addReg(SPReg, RegState::Kill)
       .addReg(SPReg)
       .addReg(ScratchReg);
     HasSTUX = true;
 
   } else if (!isLargeFrame) {
-    BuildMI(MBB, MBBI, dl, StoreUpdtInst, SPReg)
+    BuildMI(MBB, StackUpdateLoc, dl, StoreUpdtInst, SPReg)
       .addReg(SPReg)
       .addImm(NegFrameSize)
       .addReg(SPReg);
 
   } else {
     BuildMI(MBB, MBBI, dl, LoadImmShiftedInst, ScratchReg)
       .addImm(NegFrameSize >> 16);
     BuildMI(MBB, MBBI, dl, OrImmInst, ScratchReg)
for (unsigned I = 0, E = CSI.size(); I != E; ++I) {
         BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
             .addCFIIndex(CFIIndex);
         continue;
       }
 
       if (CSI[I].isSpilledToReg()) {
         unsigned SpilledReg = CSI[I].getDstReg();
         unsigned CFIRegister = MF.addFrameInst(MCCFIInstruction::createRegister(
             nullptr, MRI->getDwarfRegNum(Reg, true),
             MRI->getDwarfRegNum(SpilledReg, true)));
         BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
           .addCFIIndex(CFIRegister);
       } else {
         int Offset = MFI.getObjectOffset(CSI[I].getFrameIdx());
+        // We have changed the object offset above but we do not want to change
+        // the actual offsets in the CFI instruction so we have to undo the
+        // offset change here.
+        if (MovingStackUpdateDown)
+          Offset -= NegFrameSize;
+
         unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::createOffset(
             nullptr, MRI->getDwarfRegNum(Reg, true), Offset));
         BuildMI(MBB, MBBI, dl, TII.get(TargetOpcode::CFI_INSTRUCTION))
             .addCFIIndex(CFIIndex);
       }
     }
   }
 }
void PPCFrameLowering::emitEpilogue(MachineFunction &MF,
   // them. In such case, the final update of SP will be to add the frame
   // size to it.
   // To simplify the code, set RBReg to the base register used to restore
   // values from the stack, and set SPAdd to the value that needs to be added
   // to the SP at the end. The default values are as if red zone was present.
   unsigned RBReg = SPReg;
   unsigned SPAdd = 0;
 
+  // Check if we can move the stack update instruction up the epilogue
+  // past the callee saves. This will allow the move to LR instruction
+  // to be executed before the restores of the callee saves which means
+  // that the callee saves can hide the latency from the MTLR instrcution.
+  MachineBasicBlock::iterator StackUpdateLoc = MBBI;
+  if (stackUpdateCanBeMoved(MF)) {
+    const std::vector<CalleeSavedInfo> & Info = MFI.getCalleeSavedInfo();
+    for (CalleeSavedInfo CSI : Info) {
+      int FrIdx = CSI.getFrameIdx();
+      // If the frame index is not negative the callee saved info belongs to a
+      // stack object that is not a fixed stack object. We ignore non-fixed
+      // stack objects because we won't move the stack update pointer past them.
+      if (FrIdx >= 0)
+        continue;
+
+      if (MFI.isFixedObjectIndex(FrIdx) && MFI.getObjectOffset(FrIdx) < 0)
+        StackUpdateLoc--;
+      else {
+        // Abort
+        StackUpdateLoc = MBBI;
+        break;
+      }
+    }
+  }
+
   if (FrameSize) {
     // In the prologue, the loaded (or persistent) stack pointer value is
     // offset by the STDU/STDUX/STWU/STWUX instruction. For targets with red
     // zone add this offset back now.
 
     // If this function contained a fastcc call and GuaranteedTailCallOpt is
     // enabled (=> hasFastCall()==true) the fastcc call might contain a tail
     // call which invalidates the stack pointer value in SP(0). So we use the
if (FI->hasFastCall()) {
           .addImm(FrameSize & 0xFFFF);
         BuildMI(MBB, MBBI, dl, AddInst)
           .addReg(RBReg)
           .addReg(FPReg)
           .addReg(ScratchReg);
       }
     } else if (!isLargeFrame && !HasBP && !MFI.hasVarSizedObjects()) {
       if (HasRedZone) {
-        BuildMI(MBB, MBBI, dl, AddImmInst, SPReg)
+        BuildMI(MBB, StackUpdateLoc, dl, AddImmInst, SPReg)
           .addReg(SPReg)
           .addImm(FrameSize);
       } else {
         // Make sure that adding FrameSize will not overflow the max offset
         // size.
         assert(FPOffset <= 0 && BPOffset <= 0 && PBPOffset <= 0 &&
                "Local offsets should be negative");
         SPAdd = FrameSize;
         FPOffset += FrameSize;
         BPOffset += FrameSize;
         PBPOffset += FrameSize;
       }
     } else {
       // We don't want to use ScratchReg as a base register, because it
       // could happen to be R0. Use FP instead, but make sure to preserve it.
       if (!HasRedZone) {
         // If FP is not saved, copy it to ScratchReg.
         if (!HasFP)
           BuildMI(MBB, MBBI, dl, OrInst, ScratchReg)
             .addReg(FPReg)
             .addReg(FPReg);
         RBReg = FPReg;
       }
-      BuildMI(MBB, MBBI, dl, LoadInst, RBReg)
+      BuildMI(MBB, StackUpdateLoc, dl, LoadInst, RBReg)
         .addImm(0)
         .addReg(SPReg);
     }
   }
   assert(RBReg != ScratchReg && "Should have avoided ScratchReg");
   // If there is no red zone, ScratchReg may be needed for holding a useful
   // value (although not the base register). Make sure it is not overwritten
   // too early.
void PPCFrameLowering::emitEpilogue(MachineFunction &MF,
   }
 
   // Delay restoring of the LR if ScratchReg is needed. This is ok, since
   // LR is stored in the caller's stack frame. ScratchReg will be needed
   // if RBReg is anything other than SP. We shouldn't use ScratchReg as
   // a base register anyway, because it may happen to be R0.
   bool LoadedLR = false;
   if (MustSaveLR && RBReg == SPReg && isInt<16>(LROffset+SPAdd)) {
-    BuildMI(MBB, MBBI, dl, LoadInst, ScratchReg)
+    BuildMI(MBB, StackUpdateLoc, dl, LoadInst, ScratchReg)
       .addImm(LROffset+SPAdd)
       .addReg(RBReg);
     LoadedLR = true;
   }
 
   if (MustSaveCR && !(SingleScratchReg && MustSaveLR)) {
     // This will only occur for PPC64.
     assert(isPPC64 && "Expecting 64-bit mode");
if (RBReg != SPReg || SPAdd != 0) {
     assert(RBReg != ScratchReg && "Should be using FP or SP as base register");
     if (RBReg == FPReg)
       BuildMI(MBB, MBBI, dl, OrInst, FPReg)
         .addReg(ScratchReg)
         .addReg(ScratchReg);
 
     // Now load the LR from the caller's stack frame.
     if (MustSaveLR && !LoadedLR)
       BuildMI(MBB, MBBI, dl, LoadInst, ScratchReg)
         .addImm(LROffset)
         .addReg(SPReg);
   }
 
   if (MustSaveCR &&
       !(SingleScratchReg && MustSaveLR)) // will only occur for PPC64
     for (unsigned i = 0, e = MustSaveCRs.size(); i != e; ++i)
       BuildMI(MBB, MBBI, dl, TII.get(PPC::MTOCRF8), MustSaveCRs[i])
         .addReg(TempReg, getKillRegState(i == e-1));
 
   if (MustSaveLR)
-    BuildMI(MBB, MBBI, dl, MTLRInst).addReg(ScratchReg);
+    BuildMI(MBB, StackUpdateLoc, dl, MTLRInst).addReg(ScratchReg);
 
   // Callee pop calling convention. Pop parameter/linkage area. Used for tail
   // call optimization
   if (IsReturnBlock) {
     unsigned RetOpcode = MBBI->getOpcode();
     if (MF.getTarget().Options.GuaranteedTailCallOpt &&
         (RetOpcode == PPC::BLR || RetOpcode == PPC::BLR8) &&
         MF.getFunction().getCallingConv() == CallingConv::Fast) {
PPCFrameLowering::addScavengingSpillSlot(MachineFunction &MF,
   // large. In case there is no free register for large-offset addressing,
   // this slot is used for the necessary emergency spill. Also, we need the
   // slot for dynamic stack allocations.
 
   // The scavenger might be invoked if the frame offset does not fit into
   // the 16-bit immediate. We don't know the complete frame size here
   // because we've not yet computed callee-saved register spills or the
   // needed alignment padding.
-  unsigned StackSize = determineFrameLayout(MF, false, true);
+  unsigned StackSize = determineFrameLayout(MF, true);
   MachineFrameInfo &MFI = MF.getFrameInfo();
   if (MFI.hasVarSizedObjects() || spillsCR(MF) || spillsVRSAVE(MF) ||
       hasNonRISpills(MF) || (hasSpills(MF) && !isInt<16>(StackSize))) {
     const TargetRegisterClass &GPRC = PPC::GPRCRegClass;
     const TargetRegisterClass &G8RC = PPC::G8RCRegClass;
     const TargetRegisterClass &RC = Subtarget.isPPC64() ? G8RC : GPRC;
     const TargetRegisterInfo &TRI = *Subtarget.getRegisterInfo();
     unsigned Size = TRI.getSpillSize(RC);

lib/Target/PowerPC/PPCInstrInfo.cpp

if (PPC::GPRCRegClass.contains(Reg) ||
       OpcodeIndex = SOK_Int4Spill;
     } else if (PPC::G8RCRegClass.contains(Reg) ||
                PPC::G8RC_NOX0RegClass.contains(Reg)) {
       OpcodeIndex = SOK_Int8Spill;
     } else if (PPC::F8RCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_Float8Spill;
     } else if (PPC::F4RCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_Float4Spill;
+    } else if (PPC::SPERCRegClass.contains(Reg)) {
+      OpcodeIndex = SOK_SPESpill;
+    } else if (PPC::SPE4RCRegClass.contains(Reg)) {
+      OpcodeIndex = SOK_SPE4Spill;
     } else if (PPC::CRRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_CRSpill;
     } else if (PPC::CRBITRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_CRBitSpill;
     } else if (PPC::VRRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_VRVectorSpill;
     } else if (PPC::VSRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_VSXVectorSpill;
if (PPC::GPRCRegClass.contains(Reg) ||
       OpcodeIndex = SOK_Int4Spill;
     } else if (PPC::G8RCRegClass.contains(Reg) ||
                PPC::G8RC_NOX0RegClass.contains(Reg)) {
       OpcodeIndex = SOK_Int8Spill;
     } else if (PPC::F8RCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_Float8Spill;
     } else if (PPC::F4RCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_Float4Spill;
+    } else if (PPC::SPERCRegClass.contains(Reg)) {
+      OpcodeIndex = SOK_SPESpill;
+    } else if (PPC::SPE4RCRegClass.contains(Reg)) {
+      OpcodeIndex = SOK_SPE4Spill;
     } else if (PPC::CRRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_CRSpill;
     } else if (PPC::CRBITRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_CRBitSpill;
     } else if (PPC::VRRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_VRVectorSpill;
     } else if (PPC::VSRCRegClass.contains(Reg)) {
       OpcodeIndex = SOK_VSXVectorSpill;

lib/Target/PowerPC/PPCRegisterInfo.h

public:
   BitVector getReservedRegs(const MachineFunction &MF) const override;
   bool isCallerPreservedPhysReg(unsigned PhysReg, const MachineFunction &MF) const override;
 
   /// We require the register scavenger.
   bool requiresRegisterScavenging(const MachineFunction &MF) const override {
     return true;
   }
 
-  bool requiresFrameIndexScavenging(const MachineFunction &MF) const override {
-    return true;
-  }
+  bool requiresFrameIndexScavenging(const MachineFunction &MF) const override;
 
   bool trackLivenessAfterRegAlloc(const MachineFunction &MF) const override {
     return true;
   }
 
   bool requiresVirtualBaseRegisters(const MachineFunction &MF) const override {
     return true;
   }

lib/Target/PowerPC/PPCRegisterInfo.cpp

EnableGPRToVecSpills("ppc-enable-gpr-to-vsr-spills", cl::Hidden, cl::init(false),
          cl::desc("Enable spills from gpr to vsr rather than stack"));
 
 static cl::opt<bool>
 StackPtrConst("ppc-stack-ptr-caller-preserved",
                 cl::desc("Consider R1 caller preserved so stack saves of "
                          "caller preserved registers can be LICM candidates"),
                 cl::init(true), cl::Hidden);
 
+static unsigned offsetMinAlignForOpcode(unsigned OpC);
+
 PPCRegisterInfo::PPCRegisterInfo(const PPCTargetMachine &TM)
   : PPCGenRegisterInfo(TM.isPPC64() ? PPC::LR8 : PPC::LR,
                        TM.isPPC64() ? 0 : 1,
                        TM.isPPC64() ? 0 : 1),
     TM(TM) {
   ImmToIdxMap[PPC::LD]   = PPC::LDX;    ImmToIdxMap[PPC::STD]  = PPC::STDX;
   ImmToIdxMap[PPC::LBZ]  = PPC::LBZX;   ImmToIdxMap[PPC::STB]  = PPC::STBX;
   ImmToIdxMap[PPC::LHZ]  = PPC::LHZX;   ImmToIdxMap[PPC::LHA]  = PPC::LHAX;
if (!Subtarget.hasAltivec())
     for (TargetRegisterClass::iterator I = PPC::VRRCRegClass.begin(),
          IE = PPC::VRRCRegClass.end(); I != IE; ++I)
       markSuperRegs(Reserved, *I);
 
   assert(checkAllSuperRegsMarked(Reserved));
   return Reserved;
 }
 
+bool PPCRegisterInfo::requiresFrameIndexScavenging(const MachineFunction &MF) const {
+  const PPCSubtarget &Subtarget = MF.getSubtarget<PPCSubtarget>();
+  const PPCInstrInfo *InstrInfo =  Subtarget.getInstrInfo();
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  const std::vector<CalleeSavedInfo> &Info = MFI.getCalleeSavedInfo();
+
+  // If the callee saved info is invalid we have to default to true for safety.
+  if (!MFI.isCalleeSavedInfoValid())
+    return true;
+
+  // We will require the use of X-Forms because the frame is larger than what
+  // can be represented in signed 16 bits that fit in the immediate of a D-Form.
+  // If we need an X-Form then we need a register to store the address offset.
+  unsigned FrameSize = MFI.getStackSize();
+  if (FrameSize > 32767)
+    return true;
+
+  // The callee saved info is valid so it can be traversed.
+  // Checking for registers that need saving that do not have load or store
+  // forms where the address offset is an immediate.
+  for (unsigned i = 0; i < Info.size(); i++) {
+    int FrIdx = Info[i].getFrameIdx();
+    unsigned Reg = Info[i].getReg();
+
+    unsigned Opcode = InstrInfo->getStoreOpcodeForSpill(Reg);
+    if (!MFI.isFixedObjectIndex(FrIdx)) {
+      // This is not a fixed object. If it requires alignment then we may still
+      // need to use the XForm.
+      if (offsetMinAlignForOpcode(Opcode) > 1)
+        return true;
+    }
+
+    // This is eiher:
+    // 1) A fixed frame index object which we know are aligned so
+    // as long as we have a valid DForm/DSForm/DQForm (non XForm) we don't
+    // need to consider the alignement here.
+    // 2) A not fixed object but in that case we now know that the min required
+    // alignment is no more than 1 based on the previous check.
+    if (InstrInfo->isXFormMemOp(Opcode))
+      return true;
+  }
+  return false;
+}
+
 bool PPCRegisterInfo::isCallerPreservedPhysReg(unsigned PhysReg,
                                                const MachineFunction &MF) const {
   assert(TargetRegisterInfo::isPhysicalRegister(PhysReg));
   const PPCSubtarget &Subtarget = MF.getSubtarget<PPCSubtarget>();
   const MachineFrameInfo &MFI = MF.getFrameInfo();
   if (!TM.isPPC64())
     return false;
 
unsigned PPCRegisterInfo::getRegPressureLimit(const TargetRegisterClass *RC,
   switch (RC->getID()) {
   default:
     return 0;
   case PPC::G8RC_NOX0RegClassID:
   case PPC::GPRC_NOR0RegClassID:
   case PPC::SPERCRegClassID:
   case PPC::SPE4RCRegClassID:
   case PPC::G8RCRegClassID:
   case PPC::GPRCRegClassID: {
     unsigned FP = TFI->hasFP(MF) ? 1 : 0;
     return 32 - FP - DefaultSafety;
   }
   case PPC::F8RCRegClassID:
   case PPC::F4RCRegClassID:
   case PPC::QFRCRegClassID:
   case PPC::QSRCRegClassID:
   case PPC::QBRCRegClassID:
   case PPC::VRRCRegClassID:
   case PPC::VFRCRegClassID:
   case PPC::VSLRCRegClassID:
     return 32 - DefaultSafety;
   case PPC::VSRCRegClassID:
   case PPC::VSFRCRegClassID:
   case PPC::VSSRCRegClassID:
else {
       FrameIdx = FI->getCRSpillFrameIndex();
     }
     return true;
   }
   return false;
 }
 
 // If the offset must be a multiple of some value, return what that value is.
-static unsigned offsetMinAlign(const MachineInstr &MI) {
-  unsigned OpC = MI.getOpcode();
-
+static unsigned offsetMinAlignForOpcode(unsigned OpC) {
   switch (OpC) {
   default:
     return 1;
   case PPC::LWA:
   case PPC::LWA_32:
   case PPC::LD:
   case PPC::LDU:
   case PPC::STD:
   case PPC::STDU:
   case PPC::DFLOADf32:
   case PPC::DFLOADf64:
   case PPC::DFSTOREf32:
   case PPC::DFSTOREf64:
   case PPC::LXSD:
   case PPC::LXSSP:
   case PPC::STXSD:
   case PPC::STXSSP:
     return 4;
   case PPC::LXV:
   case PPC::STXV:
     return 16;
   }
 }
 
+// If the offset must be a multiple of some value, return what that value is.
+static unsigned offsetMinAlign(const MachineInstr &MI) {
+  unsigned OpC = MI.getOpcode();
+  return offsetMinAlignForOpcode(OpC);
+}
+
 // Return the OffsetOperandNo given the FIOperandNum (and the instruction).
 static unsigned getOffsetONFromFION(const MachineInstr &MI,
                                     unsigned FIOperandNum) {
   // Take into account whether it's an add or mem instruction
   unsigned OffsetOperandNo = (FIOperandNum == 2) ? 1 : 2;
   if (MI.isInlineAsm())
     OffsetOperandNo = FIOperandNum - 1;
   else if (MI.getOpcode() == TargetOpcode::STACKMAP ||
needsFrameBaseReg(MachineInstr *MI, int64_t Offset) const {
   // Don't generate a new virtual base register just to add zero to it.
   if ((OpC == PPC::ADDI || OpC == PPC::ADDI8) &&
       MI->getOperand(2).getImm() == 0)
     return false;
 
   MachineBasicBlock &MBB = *MI->getParent();
   MachineFunction &MF = *MBB.getParent();
   const PPCFrameLowering *TFI = getFrameLowering(MF);
-  unsigned StackEst = TFI->determineFrameLayout(MF, false, true);
+  unsigned StackEst = TFI->determineFrameLayout(MF, true);
 
   // If we likely don't need a stack frame, then we probably don't need a
   // virtual base register either.
   if (!StackEst)
     return false;
 
   // Estimate an offset from the stack pointer.
   // The incoming offset is relating to the SP at the start of the function,

test/CodeGen/PowerPC/MCSE-caller-preserved-reg.ll

 %class.CC = type { %struct.SS }
 %struct.SS = type { void ()* }
 
 @_ZN2CC2ccE = external thread_local global %"struct.CC::TT", align 8
 
 define noalias i8* @_ZN2CC3funEv(%class.CC* %this) {
 ; CHECK-LABEL: _ZN2CC3funEv:
 ; CHECK:    mflr 0
-; CHECK-NEXT:    std 0, 16(1)
-; CHECK-NEXT:    stdu 1, -48(1)
 ; CHECK-NEXT:    .cfi_def_cfa_offset 48
 ; CHECK-NEXT:    .cfi_offset lr, 16
 ; CHECK-NEXT:    .cfi_offset r30, -16
+; CHECK-NEXT:    std 30, -16(1)
+; CHECK-NEXT:    std 0, 16(1)
+; CHECK-NEXT:    stdu 1, -48(1)
 ; CHECK-NEXT:    ld 12, 0(3)
-; CHECK-NEXT:    std 30, 32(1)
 ; CHECK-NEXT:    mr 30, 3
 ; CHECK-NEXT:    std 2, 24(1)
 ; CHECK-NEXT:    mtctr 12
 ; CHECK-NEXT:    bctrl
 ; CHECK-NEXT:    ld 2, 24(1)
 ; CHECK-NEXT:    addis 3, 2, _ZN2CC2ccE@got@tlsgd@ha
 ; CHECK-NEXT:    addi 3, 3, _ZN2CC2ccE@got@tlsgd@l
 ; CHECK-NEXT:    bl __tls_get_addr(_ZN2CC2ccE@tlsgd)
 ; CHECK-NEXT:    nop
 ; CHECK-NEXT:    ld 4, 0(3)
 ; CHECK-NEXT:    cmpldi 4, 0
 ; CHECK-NEXT:    beq 0, .LBB0_2
 ; CHECK:    addi 4, 3, 8
 ; CHECK-NEXT:    mr 3, 30
 ; CHECK-NEXT:    bl _ZN2CC3barEPi
 ; CHECK-NEXT:    nop
-; CHECK:    ld 30, 32(1)
-; CHECK-NEXT:    li 3, 0
+; CHECK:    li 3, 0
 ; CHECK-NEXT:    addi 1, 1, 48
 ; CHECK-NEXT:    ld 0, 16(1)
 ; CHECK-NEXT:    mtlr 0
+; CHECK:    ld 30, -16(1)
 ; CHECK-NEXT:    blr
 entry:
   %foo = getelementptr inbounds %class.CC, %class.CC* %this, i64 0, i32 0, i32 0
   %0 = load void ()*, void ()** %foo, align 8
   tail call void %0()
   %1 = load i64, i64* getelementptr inbounds (%"struct.CC::TT", %"struct.CC::TT"* @_ZN2CC2ccE, i64 0, i32 0)
   %tobool = icmp eq i64 %1, 0
   br i1 %tobool, label %if.end, label %if.then

test/CodeGen/PowerPC/ppc-shrink-wrapping.ll

 ;
 ; DISABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; DISABLE: slwi 3, 4, 1
 ; DISABLE: .[[EPILOG_BB]]: # %if.end
 ;
 ; Epilogue code.
 ; CHECK: mtlr {{[0-9]+}}
-; CHECK-NEXT: blr
+; CHECK: blr
 ;
 ; ENABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; ENABLE: slwi 3, 4, 1
 ; ENABLE-NEXT: blr
 define i32 @freqSaveAndRestoreOutsideLoop(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0

 ; CHECK-DAG: addi [[IV]], [[IV]], -1
 ; CHECK-DAG: add [[SUM]], 3, [[SUM]] 
 ; CHECK-NEXT: cmplwi [[IV]], 0
 ; CHECK-NEXT: bne 0, .[[LOOP]]
 ;
 ; Next BB
 ; CHECK: %for.exit
 ; CHECK: mtlr {{[0-9]+}}
-; CHECK-NEXT: blr
+; CHECK: blr
 define i32 @freqSaveAndRestoreOutsideLoop2(i32 %cond) {
 entry:
   br label %for.preheader
 
 for.preheader:
   tail call void asm "nop", ""()
   br label %for.body
 

 ;
 ; ENABLE: cmplwi 0, 3, 0
 ; ENABLE-NEXT: beq 0, .[[ELSE_LABEL:LBB[0-9_]+]]
 ;
 ; Prologue code.
 ; Make sure we save the link register 
 ; CHECK: mflr {{[0-9]+}}
 ;
-; DISABLE: cmplwi 0, 3, 0
-; DISABLE-NEXT: std
+; DISABLE: std
 ; DISABLE-NEXT: std
+; DISABLE: cmplwi 0, 3, 0
 ; DISABLE-NEXT: beq 0, .[[ELSE_LABEL:LBB[0-9_]+]]
 ;
 ; Loop preheader
 ; CHECK-DAG: li [[SUM:[0-9]+]], 0
 ; CHECK-DAG: li [[IV:[0-9]+]], 10
 ; 
 ; Loop body
 ; CHECK: .[[LOOP:LBB[0-9_]+]]: # %for.body

 ;
 ; DISABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; DISABLE: slwi 3, 4, 1
 ;
 ; DISABLE: .[[EPILOG_BB]]: # %if.end
 ; Epilog code
 ; CHECK: mtlr {{[0-9]+}}
-; CHECK-NEXT: blr
+; CHECK: blr
 ; 
 ; ENABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; ENABLE: slwi 3, 4, 1
 ; ENABLE-NEXT: blr
 define i32 @loopInfoSaveOutsideLoop(i32 %cond, i32 %N) {
 entry:
   %tobool = icmp eq i32 %cond, 0

 ;
 ; ENABLE: cmplwi 0, 3, 0
 ; ENABLE-NEXT: beq 0, .[[ELSE_LABEL:LBB[0-9_]+]]
 ;
 ; Prologue code.
 ; Make sure we save the link register
 ; CHECK: mflr {{[0-9]+}}
 ;
-; DISABLE: cmplwi 0, 3, 0
-; DISABLE-NEXT: std
+; DISABLE: std
 ; DISABLE-NEXT: std
+; DISABLE: cmplwi 0, 3, 0
 ; DISABLE-NEXT: beq 0, .[[ELSE_LABEL:LBB[0-9_]+]]
 ;
 ; CHECK: bl somethingElse
 ;
 ; Loop preheader
 ; CHECK-DAG: li [[SUM:[0-9]+]], 0
 ; CHECK-DAG: li [[IV:[0-9]+]], 10
 ; 

 ;
 ; DISABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; DISABLE: slwi 3, 4, 1
 ; DISABLE: .[[EPILOG_BB]]: # %if.end
 ;
 ; Epilogue code.
 ; CHECK: mtlr {{[0-9]+}}
-; CHECK-NEXT: blr
+; CHECK: blr
 ;
 ; ENABLE: .[[ELSE_LABEL]]: # %if.else
 ; Shift second argument by one and store into returned register.
 ; ENABLE: slwi 3, 4, 1
 ; ENABLE-NEXT: blr
 define i32 @loopInfoRestoreOutsideLoop(i32 %cond, i32 %N) nounwind {
 entry:
   %tobool = icmp eq i32 %cond, 0

test/CodeGen/PowerPC/tls_get_addr_clobbers.ll

 ; RUN: llc -verify-machineinstrs -mtriple="powerpc64le-unknown-linux-gnu" -relocation-model=pic < %s | FileCheck %s
 
 @a = thread_local global i32* null, align 8
 
 define void @test_foo(i32* nocapture %x01, i32* nocapture %x02, i32* nocapture %x03, i32* nocapture %x04, i32* nocapture %x05, i32* nocapture %x06, i32* nocapture %x07, i32* nocapture %x08) #0 {
 entry:
 
 ; CHECK-LABEL: test_foo:
-; CHECK: stdu 1, {{-?[0-9]+}}(1)
+; CHECK-DAG: stdu 1, {{-?[0-9]+}}(1)
 ; CHECK-DAG: mr [[BACKUP_3:[0-9]+]], 3
 ; CHECK-DAG: mr [[BACKUP_4:[0-9]+]], 4
 ; CHECK-DAG: mr [[BACKUP_5:[0-9]+]], 5
 ; CHECK-DAG: mr [[BACKUP_6:[0-9]+]], 6
 ; CHECK-DAG: mr [[BACKUP_7:[0-9]+]], 7
 ; CHECK-DAG: mr [[BACKUP_8:[0-9]+]], 8
 ; CHECK-DAG: mr [[BACKUP_9:[0-9]+]], 9
 ; CHECK-DAG: mr [[BACKUP_10:[0-9]+]], 10
-; CHECK-DAG: std [[BACKUP_3]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_4]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_5]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_6]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_7]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_8]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_9]], {{[0-9]+}}(1)
-; CHECK-DAG: std [[BACKUP_10]], {{[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_3]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_4]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_5]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_6]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_7]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_8]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_9]], {{-?[0-9]+}}(1)
+; CHECK-DAG: std [[BACKUP_10]], {{-?[0-9]+}}(1)
 ; CHECK: bl __tls_get_addr
 ; CHECK-DAG: stw 3, 0([[BACKUP_3]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_4]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_5]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_6]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_7]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_8]])
 ; CHECK-DAG: stw 3, 0([[BACKUP_9]])