Add functions to MachineLICM to hoist invariant stores.

[syzaara]

Summary

This patch adds functions to allow MachineLICM to hoist invariant stores. Currently, MachineLICM does not hoist any store instructions, however when storing the same value to a constant spot on the stack, the store instruction should be considered invariant and be hoisted.
Such stores are common with indirect calls on PowerPC where we save the TOC pointer in R2 to a constant spot on the stack. R2 is constant throughout the function.

The function isInvariantStore iterates each operand of the store instruction and checks that each register operand satisfies isCallerPreservedPhysReg.

These store instructions are fed by a copy:
%vreg12<def> = COPY %X2; G8RC:%vreg12
STD %vreg12, 24, %X1; mem:ST8[Stack+24] G8RC:%vreg12

When hoisting the store, the copy should be hoisted as well. This is handled with isCopyFeedingInvariantStore.

For more details, please see RFC: http://lists.llvm.org/pipermail/llvm-dev/2017-November/119116.html

[nemanjai]

Hoisting these invariant stores is certainly something we want on PPC. It makes sense that MachineLICM is the place to do it. I've added a few people that I think are familiar with the pass to help with the review.

lib/CodeGen/MachineLICM.cpp

717 ↗

(On Diff #123409)

Looks like this is just

// If we have hoisted an instruction that may store, it can only be a
// constant store.
else if (MI->mayStore())
  NumStoreConst++;

lib/CodeGen/TargetInstrInfo.cpp

890 ↗

(On Diff #123409)

It should be safe to not have this condition (and perhaps just turn it into an assert).

891 ↗

(On Diff #123409)

Why not for (MachineInstr &UseMI : use_instructions(CopyDstReg))?

[qcolombet]

include/llvm/CodeGen/TargetInstrInfo.h

1133 ↗

(On Diff #123409)

Instead of patching every single optimization with this new hook, could you teach isSafeToMove this concept of invariant store?

1136 ↗

(On Diff #123409)

Why does this need to be a TII hook?

[MatzeB]

Didn't we already add TargetRegisterInfo::isCallerPreservedPhysReg() specifically for this PPC issue?

include/llvm/CodeGen/TargetInstrInfo.h

1133 ↗

(On Diff #123409)

This should also have more detailed explanations! Can you really move the store anywhere in the function? (As an example: Would in front of the prologue or before a return be still okay? I suspect not.)

[syzaara]

include/llvm/CodeGen/TargetInstrInfo.h

1133 ↗

(On Diff #123409)

There are other optimizations that use isSafeToMove where it does not make sense to teach it about invariant stores. For example, DeadMachineInstructionElim.cpp will end up deleting the store instruction if isSafeToMove was modified. We want to restrict what we do with these invariant stores to MachineLICM.

[syzaara]

In D40196#939508, @MatzeB wrote:

Didn't we already add TargetRegisterInfo::isCallerPreservedPhysReg() specifically for this PPC issue?

isCallerPreservedPhysReg was added to allow hoisting of load instructions whose operands were preserved physical regs.
In this case, we are trying to hoist a store instruction, which currently is not supported by MachineLICM. MachineLICM considers all stores unsafe to hoist.
This teaches MachineLICM to allow hoisting of invariant stores when all the operands of the store satisfy isCallerPreservedPhysReg.

[lei]

lib/CodeGen/MachineLICM.cpp

892 ↗

(On Diff #123409)

nit: since HoistConstStores is a boolean controlled by option, I'd put this as the first ck in this && cond.

1112 ↗

(On Diff #123409)

I don't think you need to ck for MI.isCopy() here since isCopyFeedingInvariantStore() checks for this and does an early exit if it's false.

lib/Target/PowerPC/PPCInstrInfo.cpp

302 ↗

(On Diff #123409)

Please add documentation for the following 2 functions.

330 ↗

(On Diff #123409)

why not use: for (const MachineOperand &MO : MI.operands())

lib/Target/PowerPC/PPCRegisterInfo.cpp

304 ↗

(On Diff #123409)

Need more explanation as to why X1 is caller preserved.

[hfinkel]

include/llvm/CodeGen/TargetInstrInfo.h

1135 ↗

(On Diff #125601)

Document here whether or not it is legal to call this on instructions that aren't stores.

lib/CodeGen/MachineLICM.cpp

890 ↗

(On Diff #125601)

If we decide to move the mayStore check into isInvariantStore, then remove it here.

1098 ↗

(On Diff #125601)

This really seems to represent the way that the PPC implementation of isInvariantStore works. What if other targets look through things that aren't pure copies? Your PPC implementation of isInvariantStore could be target independent. I'd just move it here to be next to this implementation. (This won't affect any other targets anyway, because only PPC currently implements isCallerPreservedPhysReg).

lib/Target/PowerPC/PPCInstrInfo.cpp

308 ↗

(On Diff #125601)

This is going to be added to PPCInstInfo in D31852, don't add it twice :-)

335 ↗

(On Diff #125601)

Add something here to validate that we've been called on an instruction that might store?

assert(MI->mayStore() && "isInvariantStore not called on a store instruction");

or put the check here:

if (!MI->mayStore())
  return false;

depending on the contract we want.

lib/Target/PowerPC/PPCRegisterInfo.cpp

69 ↗

(On Diff #125601)

I don't see any test cases for this. Can you add one?

296 ↗

(On Diff #125601)

This applies to ELFv1 too, please don't restrict this to ELFv2.

[syzaara]

lib/Target/PowerPC/PPCRegisterInfo.cpp

69 ↗

(On Diff #125601)

The testcase added test/CodeGen/PowerPC/loop-hoist-toc-save.ll tests for -ppc-stack-ptr-caller-preserved in combination with hoist-const-stores.

[hfinkel]

LGTM

[syzaara]

@echristo Hi Eric, do you have any comments regarding this patch?

[echristo]

A few comments inline.

include/llvm/CodeGen/TargetInstrInfo.h

303 ↗

(On Diff #128028)

This API is described a little awkwardly and you can see it from the comment - "... for a Machine instruction" and the API doesn't take an MI, but just a register and MRI.

Maybe instead it should be on TRI?

lib/CodeGen/MachineLICM.cpp

894–897 ↗

(On Diff #128028)

I understand the desire to make these functions static, but you should probably either pass in the information you need into them or make them class functions. There's no need to look up all of this information in a loop for sure when we have it cached in the class.

926–929 ↗

(On Diff #128028)

Should possibly just be a class function or take the variables it needs as arguments rather than grabbing them.

lib/CodeGen/TargetInstrInfo.cpp

301 ↗

(On Diff #128028)

const?

[syzaara]

@echristo Hi Eric, are you okay to approve this now?

[echristo]

In D40196#1005679, @syzaara wrote:

@echristo Hi Eric, are you okay to approve this now?

You didn't answer any of my questions?

[syzaara]

In D40196#1005685, @echristo wrote:

In D40196#1005679, @syzaara wrote:

@echristo Hi Eric, are you okay to approve this now?

You didn't answer any of my questions?

Sorry I missed the first point. I agree, I think it makes more sense to move lookThruCopyLike to TRI rather than TII.

[echristo]

Thanks!

-eric

llvm/trunk/include/llvm/CodeGen/TargetRegisterInfo.h

public:
   /// Returns true if Reg contains RegUnit.
   bool hasRegUnit(unsigned Reg, unsigned RegUnit) const {
     for (MCRegUnitIterator Units(Reg, this); Units.isValid(); ++Units)
       if (*Units == RegUnit)
         return true;
     return false;
   }
 
+  /// Returns the original SrcReg unless it is the target of a copy-like
+  /// operation, in which case we chain backwards through all such operations
+  /// to the ultimate source register.  If a physical register is encountered,
+  /// we stop the search.
+  virtual unsigned lookThruCopyLike(unsigned SrcReg,
+                                    const MachineRegisterInfo *MRI) const;
+
   /// Return a null-terminated list of all of the callee-saved registers on
   /// this target. The register should be in the order of desired callee-save
   /// stack frame offset. The first register is closest to the incoming stack
   /// pointer if stack grows down, and vice versa.
   /// Notice: This function does not take into account disabled CSRs.
   ///         In most cases you will want to use instead the function 
   ///         getCalleeSavedRegs that is implemented in MachineRegisterInfo.
   virtual const MCPhysReg*

llvm/trunk/lib/CodeGen/MachineLICM.cpp

HoistCheapInsts("hoist-cheap-insts",
                 cl::desc("MachineLICM should hoist even cheap instructions"),
                 cl::init(false), cl::Hidden);
 
 static cl::opt<bool>
 SinkInstsToAvoidSpills("sink-insts-to-avoid-spills",
                        cl::desc("MachineLICM should sink instructions into "
                                 "loops to avoid register spills"),
                        cl::init(false), cl::Hidden);
+static cl::opt<bool>
+HoistConstStores("hoist-const-stores",
+                 cl::desc("Hoist invariant stores"),
+                 cl::init(true), cl::Hidden);
 
 STATISTIC(NumHoisted,
           "Number of machine instructions hoisted out of loops");
 STATISTIC(NumLowRP,
           "Number of instructions hoisted in low reg pressure situation");
 STATISTIC(NumHighLatency,
           "Number of high latency instructions hoisted");
 STATISTIC(NumCSEed,
           "Number of hoisted machine instructions CSEed");
 STATISTIC(NumPostRAHoisted,
           "Number of machine instructions hoisted out of loops post regalloc");
+STATISTIC(NumStoreConst,
+          "Number of stores of const phys reg hoisted out of loops");
 
 namespace {
 
   class MachineLICMBase : public MachineFunctionPass {
     const TargetInstrInfo *TII;
     const TargetLoweringBase *TLI;
     const TargetRegisterInfo *TRI;
     const MachineFrameInfo *MFI;
for (MachineDomTreeNode *Node : Scopes) {
     // Process the block
     SpeculationState = SpeculateUnknown;
     for (MachineBasicBlock::iterator
          MII = MBB->begin(), E = MBB->end(); MII != E; ) {
       MachineBasicBlock::iterator NextMII = MII; ++NextMII;
       MachineInstr *MI = &*MII;
       if (!Hoist(MI, Preheader))
         UpdateRegPressure(MI);
+      // If we have hoisted an instruction that may store, it can only be a
+      // constant store.
+      else if (MI->mayStore())
+        NumStoreConst++;
       MII = NextMII;
     }
 
     // If it's a leaf node, it's done. Traverse upwards to pop ancestors.
     ExitScopeIfDone(Node, OpenChildren, ParentMap);
   }
 }
 
static bool mayLoadFromGOTOrConstantPool(MachineInstr &MI) {
   for (MachineMemOperand *MemOp : MI.memoperands())
     if (const PseudoSourceValue *PSV = MemOp->getPseudoValue())
       if (PSV->isGOT() || PSV->isConstantPool())
         return true;
 
   return false;
 }
 
+// This function iterates through all the operands of the input store MI and
+// checks that each register operand statisfies isCallerPreservedPhysReg.
+// This means, the value being stored and the address where it is being stored
+// is constant throughout the body of the function (not including prologue and
+// epilogue). When called with an MI that isn't a store, it returns false.
+static bool isInvariantStore(const MachineInstr &MI,
+                             const TargetRegisterInfo *TRI,
+                             const MachineRegisterInfo *MRI) {
+
+  if (!MI.mayStore() || MI.hasUnmodeledSideEffects() ||
+      (MI.getNumOperands() == 0))
+    return false;
+
+  // Check that all register operands are caller-preserved physical registers.
+  for (const MachineOperand &MO : MI.operands()) {
+    if (MO.isReg()) {
+      unsigned Reg = MO.getReg();
+      // If operand is a virtual register, check if it comes from a copy of a
+      // physical register.
+      if (TargetRegisterInfo::isVirtualRegister(Reg))
+        Reg = TRI->lookThruCopyLike(MO.getReg(), MRI);
+      if (TargetRegisterInfo::isVirtualRegister(Reg))
+        return false;
+      if (!TRI->isCallerPreservedPhysReg(Reg, *MI.getMF()))
+        return false;
+    }
+  }
+  return true;
+}
+
+// Return true if the input MI is a copy instruction that feeds an invariant
+// store instruction. This means that the src of the copy has to satisfy
+// isCallerPreservedPhysReg and atleast one of it's users should satisfy
+// isInvariantStore.
+static bool isCopyFeedingInvariantStore(const MachineInstr &MI,
+                                        const MachineRegisterInfo *MRI,
+                                        const TargetRegisterInfo *TRI) {
+
+  // FIXME: If targets would like to look through instructions that aren't
+  // pure copies, this can be updated to a query.
+  if (!MI.isCopy())
+    return false;
+
+  const MachineFunction *MF = MI.getMF();
+  // Check that we are copying a constant physical register.
+  unsigned CopySrcReg = MI.getOperand(1).getReg();
+  if (TargetRegisterInfo::isVirtualRegister(CopySrcReg))
+    return false;
+
+  if (!TRI->isCallerPreservedPhysReg(CopySrcReg, *MF))
+    return false;
+
+  unsigned CopyDstReg = MI.getOperand(0).getReg();
+  // Check if any of the uses of the copy are invariant stores.
+  assert (TargetRegisterInfo::isVirtualRegister(CopyDstReg) &&
+          "copy dst is not a virtual reg");
+
+  for (MachineInstr &UseMI : MRI->use_instructions(CopyDstReg)) {
+    if (UseMI.mayStore() && isInvariantStore(UseMI, TRI, MRI))
+      return true;
+  }
+  return false;
+}
+
 /// Returns true if the instruction may be a suitable candidate for LICM.
 /// e.g. If the instruction is a call, then it's obviously not safe to hoist it.
 bool MachineLICMBase::IsLICMCandidate(MachineInstr &I) {
   // Check if it's safe to move the instruction.
   bool DontMoveAcrossStore = true;
-  if (!I.isSafeToMove(AA, DontMoveAcrossStore))
+  if ((!I.isSafeToMove(AA, DontMoveAcrossStore)) &&
+      !(HoistConstStores && isInvariantStore(I, TRI, MRI))) {
     return false;
+  }
 
   // If it is load then check if it is guaranteed to execute by making sure that
   // it dominates all exiting blocks. If it doesn't, then there is a path out of
   // the loop which does not execute this load, so we can't hoist it. Loads
   // from constant memory are not safe to speculate all the time, for example
   // indexed load from a jump table.
   // Stores and side effects are already checked by isSafeToMove.
   if (I.mayLoad() && !mayLoadFromGOTOrConstantPool(I) &&
bool MachineLICMBase::IsProfitableToHoist(MachineInstr &MI) {
   //   loop. This increases register pressure in the loop.
   //
   // - If the value is used by a PHI in the loop, a copy will be required for
   //   lowering the PHI after extending the live range.
   //
   // - When hoisting the last use of a value in the loop, that value no longer
   //   needs to be live in the loop. This lowers register pressure in the loop.
 
+  if (HoistConstStores &&  isCopyFeedingInvariantStore(MI, MRI, TRI))
+    return true;
+
   bool CheapInstr = IsCheapInstruction(MI);
   bool CreatesCopy = HasLoopPHIUse(&MI);
 
   // Don't hoist a cheap instruction if it would create a copy in the loop.
   if (CheapInstr && CreatesCopy) {
     DEBUG(dbgs() << "Won't hoist cheap instr with loop PHI use: " << MI);
     return false;
   }

llvm/trunk/lib/CodeGen/TargetRegisterInfo.cpp

if (RegSize)
       return RegSize;
     // Since Reg is not a generic register, it must have a register class.
     RC = MRI.getRegClass(Reg);
   }
   assert(RC && "Unable to deduce the register class");
   return getRegSizeInBits(*RC);
 }
 
+unsigned
+TargetRegisterInfo::lookThruCopyLike(unsigned SrcReg,
+                                     const MachineRegisterInfo *MRI) const {
+  while (true) {
+    const MachineInstr *MI = MRI->getVRegDef(SrcReg);
+    if (!MI->isCopyLike())
+      return SrcReg;
+
+    unsigned CopySrcReg;
+    if (MI->isCopy())
+      CopySrcReg = MI->getOperand(1).getReg();
+    else {
+      assert(MI->isSubregToReg() && "Bad opcode for lookThruCopyLike");
+      CopySrcReg = MI->getOperand(2).getReg();
+    }
+
+    if (!isVirtualRegister(CopySrcReg))
+      return CopySrcReg;
+
+    SrcReg = CopySrcReg;
+  }
+}
+
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
 LLVM_DUMP_METHOD
 void TargetRegisterInfo::dumpReg(unsigned Reg, unsigned SubRegIndex,
                                  const TargetRegisterInfo *TRI) {
   dbgs() << printReg(Reg, TRI, SubRegIndex) << "\n";
 }
 #endif

llvm/trunk/lib/Target/PowerPC/PPCInstrInfo.h

public:
   bool isZeroExtended(const MachineInstr &MI, const unsigned depth = 0) const {
    return isSignOrZeroExtended(MI, false, depth);
   }
 
   bool convertToImmediateForm(MachineInstr &MI,
                               MachineInstr **KilledDef = nullptr) const;
   void replaceInstrWithLI(MachineInstr &MI, const LoadImmediateInfo &LII) const;
 
-  // This is used to find the "true" source register for n
-  // Machine instruction. Returns the original SrcReg unless it is the target
-  // of a copy-like operation, in which case we chain backwards through all
-  // such operations to the ultimate source register.  If a
-  // physical register is encountered, we stop the search.
-  static unsigned lookThruCopyLike(unsigned SrcReg,
-                                   const MachineRegisterInfo *MRI);
   bool instrHasImmForm(const MachineInstr &MI, ImmInstrInfo &III) const;
 };
 
 }
 
 #endif

llvm/trunk/lib/Target/PowerPC/PPCInstrInfo.cpp

case PPC::CFENCE8: {
     MI.setDesc(get(PPC::ISYNC));
     MI.RemoveOperand(0);
     return true;
   }
   }
   return false;
 }
 
-unsigned PPCInstrInfo::lookThruCopyLike(unsigned SrcReg,
-                                        const MachineRegisterInfo *MRI) {
-  while (true) {
-    MachineInstr *MI = MRI->getVRegDef(SrcReg);
-    if (!MI->isCopyLike())
-      return SrcReg;
-
-    unsigned CopySrcReg;
-    if (MI->isCopy())
-      CopySrcReg = MI->getOperand(1).getReg();
-    else {
-      assert(MI->isSubregToReg() && "Bad opcode for lookThruCopyLike");
-      CopySrcReg = MI->getOperand(2).getReg();
-    }
-
-    if (!TargetRegisterInfo::isVirtualRegister(CopySrcReg))
-      return CopySrcReg;
-
-    SrcReg = CopySrcReg;
-  }
-}
-
 // Essentially a compile-time implementation of a compare->isel sequence.
 // It takes two constants to compare, along with the true/false registers
 // and the comparison type (as a subreg to a CR field) and returns one
 // of the true/false registers, depending on the comparison results.
 static unsigned selectReg(int64_t Imm1, int64_t Imm2, unsigned CompareOpc,
                           unsigned TrueReg, unsigned FalseReg,
                           unsigned CRSubReg) {
   // Signed comparisons. The immediates are assumed to be sign-extended.

 }
 
 MachineInstr *PPCInstrInfo::getConstantDefMI(MachineInstr &MI,
                                              unsigned &ConstOp,
                                              bool &SeenIntermediateUse) const {
   ConstOp = ~0U;
   MachineInstr *DefMI = nullptr;
   MachineRegisterInfo *MRI = &MI.getParent()->getParent()->getRegInfo();
+  const TargetRegisterInfo *TRI = &getRegisterInfo();
   // If we'ere in SSA, get the defs through the MRI. Otherwise, only look
   // within the basic block to see if the register is defined using an LI/LI8.
   if (MRI->isSSA()) {
     for (int i = 1, e = MI.getNumOperands(); i < e; i++) {
       if (!MI.getOperand(i).isReg())
         continue;
       unsigned Reg = MI.getOperand(i).getReg();
       if (!TargetRegisterInfo::isVirtualRegister(Reg))
         continue;
-      unsigned TrueReg = lookThruCopyLike(Reg, MRI);
+      unsigned TrueReg = TRI->lookThruCopyLike(Reg, MRI);
       if (TargetRegisterInfo::isVirtualRegister(TrueReg)) {
         DefMI = MRI->getVRegDef(TrueReg);
         if (DefMI->getOpcode() == PPC::LI || DefMI->getOpcode() == PPC::LI8) {
           ConstOp = i;
           break;
         }
       }
     }

llvm/trunk/lib/Target/PowerPC/PPCMIPeephole.cpp

void PPCMIPeephole::UpdateTOCSaves(
   TOCSaves[MI] = Keep;
 }
 
 // Perform peephole optimizations.
 bool PPCMIPeephole::simplifyCode(void) {
   bool Simplified = false;
   MachineInstr* ToErase = nullptr;
   std::map<MachineInstr *, bool> TOCSaves;
-
+  const TargetRegisterInfo *TRI = &TII->getRegisterInfo();
   NumFunctionsEnteredInMIPeephole++;
   if (ConvertRegReg) {
     // Fixed-point conversion of reg/reg instructions fed by load-immediate
     // into reg/imm instructions. FIXME: This is expensive, control it with
     // an option.
     bool SomethingChanged = false;
     do {
       NumFixedPointIterations++;
for (MachineInstr &MI : MBB) {
 
           // For each of these simplifications, we need the two source
           // regs to match.  Unfortunately, MachineCSE ignores COPY and
           // SUBREG_TO_REG, so for example we can see
           //   XXPERMDI t, SUBREG_TO_REG(s), SUBREG_TO_REG(s), immed.
           // We have to look through chains of COPY and SUBREG_TO_REG
           // to find the real source values for comparison.
           unsigned TrueReg1 =
-            TII->lookThruCopyLike(MI.getOperand(1).getReg(), MRI);
+            TRI->lookThruCopyLike(MI.getOperand(1).getReg(), MRI);
           unsigned TrueReg2 =
-            TII->lookThruCopyLike(MI.getOperand(2).getReg(), MRI);
+            TRI->lookThruCopyLike(MI.getOperand(2).getReg(), MRI);
 
           if (TrueReg1 == TrueReg2
               && TargetRegisterInfo::isVirtualRegister(TrueReg1)) {
             MachineInstr *DefMI = MRI->getVRegDef(TrueReg1);
             unsigned DefOpc = DefMI ? DefMI->getOpcode() : 0;
 
             // If this is a splat fed by a splatting load, the splat is
             // redundant. Replace with a copy. This doesn't happen directly due
             // to code in PPCDAGToDAGISel.cpp, but it can happen when converting
             // a load of a double to a vector of 64-bit integers.
             auto isConversionOfLoadAndSplat = [=]() -> bool {
               if (DefOpc != PPC::XVCVDPSXDS && DefOpc != PPC::XVCVDPUXDS)
                 return false;
               unsigned DefReg =
-                TII->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
+                TRI->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
               if (TargetRegisterInfo::isVirtualRegister(DefReg)) {
                 MachineInstr *LoadMI = MRI->getVRegDef(DefReg);
                 if (LoadMI && LoadMI->getOpcode() == PPC::LXVDSX)
                   return true;
               }
               return false;
             };
             if (DefMI && (Immed == 0 || Immed == 3)) {
for (MachineInstr &MI : MBB) {
               }
             }
 
             // If this is a splat or a swap fed by another splat, we
             // can replace it with a copy.
             if (DefOpc == PPC::XXPERMDI) {
               unsigned FeedImmed = DefMI->getOperand(3).getImm();
               unsigned FeedReg1 =
-                TII->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
+                TRI->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
               unsigned FeedReg2 =
-                TII->lookThruCopyLike(DefMI->getOperand(2).getReg(), MRI);
+                TRI->lookThruCopyLike(DefMI->getOperand(2).getReg(), MRI);
 
               if ((FeedImmed == 0 || FeedImmed == 3) && FeedReg1 == FeedReg2) {
                 DEBUG(dbgs()
                       << "Optimizing splat/swap or splat/splat "
                       "to splat/copy: ");
                 DEBUG(MI.dump());
                 BuildMI(MBB, &MI, MI.getDebugLoc(), TII->get(PPC::COPY),
                         MI.getOperand(0).getReg())
for (MachineInstr &MI : MBB) {
         break;
       }
       case PPC::VSPLTB:
       case PPC::VSPLTH:
       case PPC::XXSPLTW: {
         unsigned MyOpcode = MI.getOpcode();
         unsigned OpNo = MyOpcode == PPC::XXSPLTW ? 1 : 2;
         unsigned TrueReg =
-          TII->lookThruCopyLike(MI.getOperand(OpNo).getReg(), MRI);
+          TRI->lookThruCopyLike(MI.getOperand(OpNo).getReg(), MRI);
         if (!TargetRegisterInfo::isVirtualRegister(TrueReg))
           break;
         MachineInstr *DefMI = MRI->getVRegDef(TrueReg);
         if (!DefMI)
           break;
         unsigned DefOpcode = DefMI->getOpcode();
         auto isConvertOfSplat = [=]() -> bool {
           if (DefOpcode != PPC::XVCVSPSXWS && DefOpcode != PPC::XVCVSPUXWS)
for (MachineInstr &MI : MBB) {
             MI.getOperand(2).setImm(NewElem);
           }
         }
         break;
       }
       case PPC::XVCVDPSP: {
         // If this is a DP->SP conversion fed by an FRSP, the FRSP is redundant.
         unsigned TrueReg =
-          TII->lookThruCopyLike(MI.getOperand(1).getReg(), MRI);
+          TRI->lookThruCopyLike(MI.getOperand(1).getReg(), MRI);
         if (!TargetRegisterInfo::isVirtualRegister(TrueReg))
           break;
         MachineInstr *DefMI = MRI->getVRegDef(TrueReg);
 
         // This can occur when building a vector of single precision or integer
         // values.
         if (DefMI && DefMI->getOpcode() == PPC::XXPERMDI) {
           unsigned DefsReg1 =
-            TII->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
+            TRI->lookThruCopyLike(DefMI->getOperand(1).getReg(), MRI);
           unsigned DefsReg2 =
-            TII->lookThruCopyLike(DefMI->getOperand(2).getReg(), MRI);
+            TRI->lookThruCopyLike(DefMI->getOperand(2).getReg(), MRI);
           if (!TargetRegisterInfo::isVirtualRegister(DefsReg1) ||
               !TargetRegisterInfo::isVirtualRegister(DefsReg2))
             break;
           MachineInstr *P1 = MRI->getVRegDef(DefsReg1);
           MachineInstr *P2 = MRI->getVRegDef(DefsReg2);
 
           if (!P1 || !P2)
             break;

llvm/trunk/lib/Target/PowerPC/PPCRegisterInfo.cpp

 static cl::opt<bool>
 AlwaysBasePointer("ppc-always-use-base-pointer", cl::Hidden, cl::init(false),
          cl::desc("Force the use of a base pointer in every function"));
 
 static cl::opt<bool>
 EnableGPRToVecSpills("ppc-enable-gpr-to-vsr-spills", cl::Hidden, cl::init(false),
          cl::desc("Enable spills from gpr to vsr rather than stack"));
 
+static cl::opt<bool>
+StackPtrConst("ppc-stack-ptr-caller-preserved",
+                cl::desc("Consider R1 caller preserved so stack saves of "
+                         "caller preserved registers can be LICM candidates"),
+                cl::init(true), cl::Hidden);
+
 PPCRegisterInfo::PPCRegisterInfo(const PPCTargetMachine &TM)
   : PPCGenRegisterInfo(TM.isPPC64() ? PPC::LR8 : PPC::LR,
                        TM.isPPC64() ? 0 : 1,
                        TM.isPPC64() ? 0 : 1),
     TM(TM) {
   ImmToIdxMap[PPC::LD]   = PPC::LDX;    ImmToIdxMap[PPC::STD]  = PPC::STDX;
   ImmToIdxMap[PPC::LBZ]  = PPC::LBZX;   ImmToIdxMap[PPC::STB]  = PPC::STBX;
   ImmToIdxMap[PPC::LHZ]  = PPC::LHZX;   ImmToIdxMap[PPC::LHA]  = PPC::LHAX;
BitVector PPCRegisterInfo::getReservedRegs(const MachineFunction &MF) const {
 
   assert(checkAllSuperRegsMarked(Reserved));
   return Reserved;
 }
 
 bool PPCRegisterInfo::isCallerPreservedPhysReg(unsigned PhysReg,
                                                const MachineFunction &MF) const {
   assert(TargetRegisterInfo::isPhysicalRegister(PhysReg));
-  if (TM.isELFv2ABI() && PhysReg == PPC::X2) {
+  const PPCSubtarget &Subtarget = MF.getSubtarget<PPCSubtarget>();
+  const MachineFrameInfo &MFI = MF.getFrameInfo();
+  if (!TM.isPPC64())
+    return false;
+
+  if (!Subtarget.isSVR4ABI())
+    return false;
+  if (PhysReg == PPC::X2)
     // X2 is guaranteed to be preserved within a function if it is reserved.
     // The reason it's reserved is that it's the TOC pointer (and the function
     // uses the TOC). In functions where it isn't reserved (i.e. leaf functions
     // with no TOC access), we can't claim that it is preserved.
     return (getReservedRegs(MF).test(PPC::X2));
-  } else {
-    return false;
-  }
-}
+  if (StackPtrConst && (PhysReg == PPC::X1) && !MFI.hasVarSizedObjects()
+      && !MFI.hasOpaqueSPAdjustment())
+    // The value of the stack pointer does not change within a function after
+    // the prologue and before the epilogue if there are no dynamic allocations
+    // and no inline asm which clobbers X1.
+    return true;
 
 unsigned PPCRegisterInfo::getRegPressureLimit(const TargetRegisterClass *RC,
                                               MachineFunction &MF) const {
   const PPCFrameLowering *TFI = getFrameLowering(MF);
   const unsigned DefaultSafety = 1;
 
   switch (RC->getID()) {
   default:

llvm/trunk/test/CodeGen/PowerPC/extra-toc-reg-deps.ll

 ; CHECK: addis [[REG1:[0-9]+]], 2, .LC0@toc@ha
 ; CHECK: std 2, 40(1)
 ; CHECK: ld {{[0-9]+}}, .LC0@toc@l([[REG1]])
 ; CHECK: {{mr|ld}} 2,
 ; CHECK: mtctr
 ; CHECK: bctrl
 ; CHECK: ld 2, 40(1)
 
-; CHECK: addis [[REG1:[0-9]+]], 2, .LC0@toc@ha
 ; CHECK: std 2, 40(1)
-; CHECK: ld {{[0-9]+}}, .LC0@toc@l([[REG1]])
 ; CHECK: {{mr|ld}} 2,
 ; CHECK: mtctr
 ; CHECK: bctrl
 ; CHECK: ld 2, 40(1)
 
 for.cond.cleanup:                                 ; preds = %entry
   br i1 undef, label %if.then.i, label %if.else.i
 

llvm/trunk/test/CodeGen/PowerPC/loop-hoist-toc-save.ll

+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -hoist-const-stores -ppc-stack-ptr-caller-preserved < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu -hoist-const-stores -ppc-stack-ptr-caller-preserved < %s | FileCheck %s -check-prefix=CHECKBE
+
+; Test hoist out of single loop
+define signext i32 @test1(i32 signext %lim, i32 (i32)* nocapture %Func) {
+entry:
+; CHECK-LABEL: test1
+; CHECK: for.body.preheader
+; CHECK: std 2, 24(1)
+; CHECK: for.body
+; CHECK-NOT: std 2, 24(1)
+; CHECKBE-LABEL: test1
+; CHECKBE: for.body.preheader
+; CHECKBE: std 2, 40(1)
+; CHECKBE: for.body
+; CHECKBE-NOT: std 2, 40(1)
+
+  %cmp6 = icmp sgt i32 %lim, 0
+  br i1 %cmp6, label %for.body.preheader, label %for.cond.cleanup
+
+for.body.preheader:                               ; preds = %entry
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %for.body, %entry
+  %Sum.0.lcssa = phi i32 [ 0, %entry ], [ %add, %for.body ]
+  ret i32 %Sum.0.lcssa
+
+for.body:                                         ; preds = %for.body.preheader, %for.body
+  %i.08 = phi i32 [ %inc, %for.body ], [ 0, %for.body.preheader ]
+  %Sum.07 = phi i32 [ %add, %for.body ], [ 0, %for.body.preheader ]
+  %call = tail call signext i32 %Func(i32 signext %i.08)
+  %add = add nsw i32 %call, %Sum.07
+  %inc = add nuw nsw i32 %i.08, 1
+  %exitcond = icmp eq i32 %inc, %lim
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+; Test hoist of nested loop goes to outter loop preheader
+define signext i32 @test2(i32 signext %lim, i32 (i32)* nocapture %Func) {
+entry:
+; CHECK-LABEL: test2
+; CHECK: for.body4.lr.ph.preheader
+; CHECK: std 2, 24(1)
+; CHECK: for.body4.lr.ph
+; CHECK-NOT: std 2, 24(1)
+; CHECKBE-LABEL: test2
+; CHECKBE: for.body4.lr.ph.preheader
+; CHECKBE: std 2, 40(1)
+; CHECKBE: for.body4.lr.ph
+; CHECKBE-NOT: std 2, 40(1)
+
+  %cmp20 = icmp sgt i32 %lim, 0
+  br i1 %cmp20, label %for.body4.lr.ph.preheader, label %for.cond.cleanup
+
+for.body4.lr.ph.preheader:                        ; preds = %entry
+  br label %for.body4.lr.ph
+
+for.cond.cleanup:                                 ; preds = %for.cond.cleanup3, %entry
+  %Sum.0.lcssa = phi i32 [ 0, %entry ], [ %add, %for.cond.cleanup3 ]
+  ret i32 %Sum.0.lcssa
+
+for.body4.lr.ph:                                  ; preds = %for.body4.lr.ph.preheader, %for.cond.cleanup3
+  %j.022 = phi i32 [ %inc6, %for.cond.cleanup3 ], [ 0, %for.body4.lr.ph.preheader ]
+  %Sum.021 = phi i32 [ %add, %for.cond.cleanup3 ], [ 0, %for.body4.lr.ph.preheader ]
+  br label %for.body4
+
+for.cond.cleanup3:                                ; preds = %for.body4
+  %inc6 = add nuw nsw i32 %j.022, 1
+  %exitcond24 = icmp eq i32 %inc6, %lim
+  br i1 %exitcond24, label %for.cond.cleanup, label %for.body4.lr.ph
+
+for.body4:                                        ; preds = %for.body4, %for.body4.lr.ph
+  %i.019 = phi i32 [ %j.022, %for.body4.lr.ph ], [ %inc, %for.body4 ]
+  %Sum.118 = phi i32 [ %Sum.021, %for.body4.lr.ph ], [ %add, %for.body4 ]
+  %call = tail call signext i32 %Func(i32 signext %i.019)
+  %add = add nsw i32 %call, %Sum.118
+  %inc = add nuw nsw i32 %i.019, 1
+  %exitcond = icmp eq i32 %inc, %lim
+  br i1 %exitcond, label %for.cond.cleanup3, label %for.body4
+}
+
+; Test hoist out of if statement with low branch probability
+; FIXME: we shouldn't hoist in such cases as it could increase the number
+; of stores after hoisting.
+define signext i32 @test3(i32 signext %lim, i32 (i32)* nocapture %Func) {
+entry:
+; CHECK-LABEL: test3
+; CHECK: %for.body.lr.ph
+; CHECK: std 2, 24(1)
+; CHECK: %for.body
+; CHECK-NOT: std 2, 24(1)
+; CHECKBE-LABEL: test3
+; CHECKBE: %for.body.lr.ph
+; CHECKBE: std 2, 40(1)
+; CHECKBE: %for.body
+; CHECKBE-NOT: std 2, 40(1)
+
+  %cmp13 = icmp sgt i32 %lim, 0
+  br i1 %cmp13, label %for.body.lr.ph, label %for.cond.cleanup
+
+for.body.lr.ph:                                   ; preds = %entry
+  %sub = add nsw i32 %lim, -1
+  br label %for.body
+
+for.cond.cleanup:                                 ; preds = %if.end, %entry
+  %Sum.0.lcssa = phi i32 [ 0, %entry ], [ %add3, %if.end ]
+  ret i32 %Sum.0.lcssa
+
+for.body:                                         ; preds = %if.end, %for.body.lr.ph
+  %i.015 = phi i32 [ 0, %for.body.lr.ph ], [ %inc, %if.end ]
+  %Sum.014 = phi i32 [ 0, %for.body.lr.ph ], [ %add3, %if.end ]
+  %cmp1 = icmp eq i32 %i.015, %sub
+  br i1 %cmp1, label %if.then, label %if.end
+
+if.then:                                          ; preds = %for.body
+  %call = tail call signext i32 %Func(i32 signext %sub)
+  %add = add nsw i32 %call, %Sum.014
+  br label %if.end
+
+if.end:                                           ; preds = %if.then, %for.body
+  %Sum.1 = phi i32 [ %add, %if.then ], [ %Sum.014, %for.body ]
+  %call2 = tail call signext i32 @func(i32 signext %i.015)
+  %add3 = add nsw i32 %call2, %Sum.1
+  %inc = add nuw nsw i32 %i.015, 1
+  %exitcond = icmp eq i32 %inc, %lim
+  br i1 %exitcond, label %for.cond.cleanup, label %for.body
+}
+
+declare signext i32 @func(i32 signext)

llvm/trunk/test/CodeGen/PowerPC/ppc64-func-desc-hoist.ll

entry:
   br label %for.body
 
 ; INVFUNCDESC-LABEL: @bar
 ; INVFUNCDESC-DAG: ld [[REG1:[0-9]+]], 8(3)
 ; INVFUNCDESC-DAG: ld [[REG2:[0-9]+]], 16(3)
 ; INVFUNCDESC-DAG: ld [[REG3:[0-9]+]], 0(3)
 
 ; INVFUNCDESC: %for.body
-; INVFUNCDESC: std 2, 40(1)
 ; INVFUNCDESC-DAG: mtctr [[REG3]]
 ; INVFUNCDESC-DAG: mr 11, [[REG2]]
 ; INVFUNCDESC-DAG: mr 2, [[REG1]]
 ; INVFUNCDESC: bctrl
 ; INVFUNCDESC-NEXT: ld 2, 40(1)
 
 ; NONINVFUNCDESC-LABEL: @bar
 ; NONINVFUNCDESC: %for.body
-; NONINVFUNCDESC: std 2, 40(1)
 ; NONINVFUNCDESC-DAG: ld 3, 0(30)
 ; NONINVFUNCDESC-DAG: ld 11, 16(30)
 ; NONINVFUNCDESC-DAG: ld 2, 8(30)
 ; NONINVFUNCDESC: mtctr 3
 ; NONINVFUNCDESC: bctrl
 ; NONINVFUNCDESC-NEXT: ld 2, 40(1)
 
 for.body:                                         ; preds = %for.body, %entry