[PowerPC] Add a peephole post RA to transform the inst that fed by add

[steven.zhang]

Summary

If the arch is P8, we will select XFLOAD to load the floating point, and then, expand it to vsx and non-vsx X-form instruction post RA. This patch is trying to convert the X-form to D-form if it meets the requirement that one operand of the x-form inst is the special Zero register, and another operand fed by add inst. i.e.
y = add imm, reg
LFDX. 0, y
-->
LFD imm(reg)

[nemanjai]

This needs to be beefed up to account for the two unsafe situations I outlined in the comment.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2438

I just realized that this is not safe. There needs to be a check that the register being forwarded to the user isn't clobbered between DefMI and MI.

Example:

addis r5, r2, .LCPI0_0@toc@ha
addi r4, r5, .LCPI0_0@toc@l
lfdx f3, 0, r4

is safe to transform to

addis r5, r2, .LCPI0_0@toc@ha
lfd f3, .LCPI0_0@toc@l(r5)

However,

addis r6, r2, .LCPI0_1@toc@ha
addis r5, r2, .LCPI0_0@toc@ha
addi r4, r5, .LCPI0_0@toc@l
addi r5, r6, .LCPI0_1@toc@l
lfdx f3, 0, r4

is not safe to transform to

addis r6, r2, .LCPI0_1@toc@ha
addis r5, r2, .LCPI0_0@toc@ha
addi r5, r6, .LCPI0_1@toc@l
lfd f3, .LCPI0_0@toc@l(r5)

because r5 that's being forwarded is already clobbered.
Furthermore, we can't even transform

addis r4, r2, .LCPI0_0@toc@ha
addi r4, r4, .LCPI0_0@toc@l
lfdx f3, 0, r4

into

addis r4, r2, .LCPI0_0@toc@ha
addi r4, r4, .LCPI0_0@toc@l
lfd f3, .LCPI0_0@toc@l(r4)

(i.e. we can only transform this case if we're sure that addi r4, r4, .LCPI0_0@toc@l will be deleted).

[steven.zhang]

Yes, you are right.
y = ADDI x, imm.
w = LFDX 0, y

We have the code to do the check for "y", but miss to do the check for "x". And this also caused some failure inside llvm test suite. (ie. Benchmarks/Stanford/FloatMM)
I will try to clean up all the failure first and then, continue the code review. Sorry for this, I should have addressed this issue early before.

[steven.zhang]

Clean up all the failures from llvm test-suite. We miss to check the RegMO operand for the ADDI.

[steven.zhang]

I have updated the patch and we can continue the review now.

[nemanjai]

I think that this is semantically correct now. There are just a few other stylistic comments.
Also, since this is a rather high-risk patch, can you describe on the patch the functional testing you have done with the patch (list of applications, bootstrap, etc.)?

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3044

s/describe/described

3126

s/reversely/in reverse

3133

I think the reader has to do a fair bit of thinking to convince themselves that this loop does the right thing. I certainly had to.
Would it not be easier to just:

• Exit immediately if a def is found that is not the DefMI
• Break out of the loop if we get to DefMI since we don't care what happens prior to it

The idea would be that we simply cannot transform if we found a register def prior to DefMI in the reverse walk. Also, I don't really understand why we care about any other uses of the register being forwarded? If we have shown that the only visible def of the register is DefMI, we don't care about any uses - we just need to make sure that if DefMI->readsRegister(Reg,...), we can only transform if KilledDefMI.
So the simplified sequence would be:

for (; It != E; ++It) {
  if (It->modifiesRegister(...) && (&*It) != DefMI)
    return false;
  // Made it to DefMI without encountering a clobber.
  if ((&*It) == DefMI)
    break;
}

// If DefMI also uses the register to be forwarded, we can only forward it if
// DefMI is being erased.
if (DefMI->readsRegister(Reg, ...))
  return KillDefMI;
return true;

Or did I miss something and that code doesn't sufficiently specify the constraints?

[steven.zhang]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3133

You are right. If the DefMI could be erased, there won't any use between the DefMI and MI. We don't need to check the use here.

[steven.zhang]

I have done the following tests with this patch:

1. LLVM check-all uniitests
2. LLVM test-suites
3. bootstrap build the clang with the clang including my fix, and then, run the llvm check-all and test-suites, and it passed.

The patch passed all these tests.

[nemanjai]

In D49007#1203739, @steven.zhang wrote:

I have done the following tests with this patch:

1. LLVM check-all uniitests
2. LLVM test-suites
3. bootstrap build the clang with the clang including my fix, and then, run the llvm check-all and test-suites, and it passed.

The patch passed all these tests.

Thank you for the update to the patch as well as the update regarding testing that was done.
LGTM.

llvm/lib/Target/PowerPC/PPCInstrInfo.h

struct ImmInstrInfo {
   // Is R0/X0 treated specially by the original r+r instruction?
   // If so, in which operand?
   uint64_t ZeroIsSpecialOrig : 3;
   // Is R0/X0 treated specially by the new r+i instruction?
   // If so, in which operand?
   uint64_t ZeroIsSpecialNew : 3;
   // Is the operation commutative?
   uint64_t IsCommutative : 1;
-  // The operand number to check for load immediate.
-  uint64_t ConstantOpNo : 3;
+  // The operand number to check for add-immediate def.
+  uint64_t OpNoForForwarding : 3;
   // The operand number for the immediate.
   uint64_t ImmOpNo : 3;
   // The opcode of the new instruction.
   uint64_t ImmOpcode : 16;
   // The size of the immediate.
   uint64_t ImmWidth : 5;
   // The immediate should be truncated to N bits.
   uint64_t TruncateImmTo : 5;
+  // Is the instruction summing the operand
+  uint64_t IsSummingOperands : 1;
 };
 
 // Information required to convert an instruction to just a materialized
 // immediate.
 struct LoadImmediateInfo {
   unsigned Imm : 16;
   unsigned Is64Bit : 1;
   unsigned SetCR : 1;
 };
 
 class PPCSubtarget;
 class PPCInstrInfo : public PPCGenInstrInfo {
   PPCSubtarget &Subtarget;
   const PPCRegisterInfo RI;
 
   void StoreRegToStackSlot(MachineFunction &MF, unsigned SrcReg, bool isKill,
                            int FrameIdx, const TargetRegisterClass *RC,
                            SmallVectorImpl<MachineInstr *> &NewMIs) const;
   void LoadRegFromStackSlot(MachineFunction &MF, const DebugLoc &DL,
                             unsigned DestReg, int FrameIdx,
                             const TargetRegisterClass *RC,
                             SmallVectorImpl<MachineInstr *> &NewMIs) const;
-  bool transformToImmForm(MachineInstr &MI, const ImmInstrInfo &III,
+
+  // If the inst has imm-form and one of its operand is produced by a LI,
+  // put the imm into the inst directly and remove the LI if possible.
+  bool transformToImmFormFedByLI(MachineInstr &MI, const ImmInstrInfo &III,
                                  unsigned ConstantOpNo, int64_t Imm) const;
-  MachineInstr *getConstantDefMI(MachineInstr &MI, unsigned &ConstOp,
-                                 bool &SeenIntermediateUse) const;
+  // If the inst has imm-form and one of its operand is produced by an
+  // add-immediate, try to transform it when possible.
+  bool transformToImmFormFedByAdd(MachineInstr &MI, const ImmInstrInfo &III,
+                                  unsigned ConstantOpNo,
+                                  MachineInstr &DefMI,
+                                  bool KillDefMI) const;
+  // Try to find that, if the instruction 'MI' contains any operand that 
+  // could be forwarded from some inst that feeds it. If yes, return the
+  // Def of that operand. And OpNoForForwarding is the operand index in
+  // the 'MI' for that 'Def'. 
+  MachineInstr *getForwardingDefMI(MachineInstr &MI,
+                                   unsigned &OpNoForForwarding) const;
+  // Reverse search the MI in the range of [From, To) to get the last Def and
+  // Use of the register "Reg". The result is stored in the DefMI and UseMI.
+  bool getLastDefAndUseMIInBetween(unsigned Reg,
+                                   const MachineInstr &From,
+                                   const MachineInstr &To,
+                                   const MachineInstr *&DefMI,
+                                   const MachineInstr *&UseMI) const;
+
+  // Can the user MI have it's source at index \p OpNoForForwarding
+  // forwarded from an add-immediate that feeds it?
+  bool isUseMIElgibleForForwarding(MachineInstr &MI, const ImmInstrInfo &III,
+                                   unsigned OpNoForForwarding) const;
+  bool isDefMIElgibleForForwarding(MachineInstr &DefMI,
+                                   const ImmInstrInfo &III,
+                                   MachineOperand *&ImmMO,
+                                   MachineOperand *&RegMO) const;
+  bool isImmElgibleForForwarding(const MachineOperand &ImmMO,
+                                 const MachineInstr &DefMI,
+                                 const ImmInstrInfo &III,
+                                 int64_t &Imm) const;
+  bool isRegElgibleForForwarding(const MachineOperand &RegMO,
+                                 const MachineInstr &DefMI,
+                                 const MachineInstr &MI,
+                                 bool KillDefMI) const;
+  bool isForwardingDefMIKilled(const MachineInstr &DefMI,
+                               const MachineInstr &MI,
+                               unsigned OpNoForForwarding) const;
   const unsigned *getStoreOpcodesForSpillArray() const;
   const unsigned *getLoadOpcodesForSpillArray() const;
   virtual void anchor();
 
 protected:
   /// Commutes the operands in the given instruction.
   /// The commutable operands are specified by their indices OpIdx1 and OpIdx2.
   ///

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

if ((TargetReg >= PPC::F0 && TargetReg <= PPC::F31) ||
         (TargetReg >= PPC::VSL0 && TargetReg <= PPC::VSL31))
       Opcode = LowerOpcode;
     else
       Opcode = UpperOpcode;
     MI.setDesc(get(Opcode));
     return true;
 }
 
-#ifndef NDEBUG
 static bool isAnImmediateOperand(const MachineOperand &MO) {
   return MO.isCPI() || MO.isGlobal() || MO.isImm();
 }
-#endif
 
 bool PPCInstrInfo::expandPostRAPseudo(MachineInstr &MI) const {
   auto &MBB = *MI.getParent();
   auto DL = MI.getDebugLoc();
 
   switch (MI.getOpcode()) {
   case TargetOpcode::LOAD_STACK_GUARD: {
     assert(Subtarget.isTargetLinux() &&
void PPCInstrInfo::replaceInstrWithLI(MachineInstr &MI,
   else
     MI.setDesc(get(LII.Is64Bit ? PPC::LI8 : PPC::LI));
 
   // Set the immediate.
   MachineInstrBuilder(*MI.getParent()->getParent(), MI)
       .addImm(LII.Imm);
 }
 
-MachineInstr *PPCInstrInfo::getConstantDefMI(MachineInstr &MI,
-                                             unsigned &ConstOp,
-                                             bool &SeenIntermediateUse) const {
-  ConstOp = ~0U;
+bool PPCInstrInfo::getLastDefAndUseMIInBetween(unsigned Reg,
+                                               const MachineInstr &From,
+                                               const MachineInstr &To,
+                                               const MachineInstr *&DefMI,
+                                               const MachineInstr *&UseMI
+                                               ) const {
+  const MachineRegisterInfo &MRI = From.getParent()->getParent()->getRegInfo();
+  if (MRI.isSSA())
+    return false;
+
+  assert(From.getParent() == To.getParent() &&
+         "From and To must belong to the same basic block");
+
+  // MachineInstr::readsRegister only returns true if the machine
+  // instruction reads the exact register or its super-register. It
+  // does not consider uses of sub-registers which seems like strange
+  // behaviour. Nonetheless, if we end up with a 64-bit register here,
+  // get the corresponding 32-bit register to check.
+  if (PPC::G8RCRegClass.contains(Reg))
+    Reg = Reg - PPC::X0 + PPC::R0;
+
+  DefMI = nullptr;
+  UseMI = nullptr;
+  MachineBasicBlock::const_reverse_iterator It = To;
+  It++;
+  for (;It != From.getParent()->rend(); ++It) {
+    if (!DefMI && It->modifiesRegister(Reg, &getRegisterInfo()))
+      DefMI = &*It;
+    if (!UseMI && It->readsRegister(Reg, &getRegisterInfo()))
+      UseMI = &*It;
+    if ((UseMI && DefMI) || (It == &From))
+      break;
+  }
+
+  return true;
+}
+
+MachineInstr *PPCInstrInfo::getForwardingDefMI(MachineInstr &MI,
+                                               unsigned &OpNoForForwarding
+                                               ) const {
+  OpNoForForwarding = ~0U;
   MachineInstr *DefMI = nullptr;
   MachineRegisterInfo *MRI = &MI.getParent()->getParent()->getRegInfo();
   const TargetRegisterInfo *TRI = &getRegisterInfo();
   // If we're in SSA, get the defs through the MRI. Otherwise, only look
   // within the basic block to see if the register is defined using an LI/LI8.
   if (MRI->isSSA()) {
     for (int i = 1, e = MI.getNumOperands(); i < e; i++) {
       if (!MI.getOperand(i).isReg())
         continue;
       unsigned Reg = MI.getOperand(i).getReg();
       if (!TargetRegisterInfo::isVirtualRegister(Reg))
         continue;
       unsigned TrueReg = TRI->lookThruCopyLike(Reg, MRI);
       if (TargetRegisterInfo::isVirtualRegister(TrueReg)) {
         DefMI = MRI->getVRegDef(TrueReg);
         if (DefMI->getOpcode() == PPC::LI || DefMI->getOpcode() == PPC::LI8) {
-          ConstOp = i;
+          OpNoForForwarding = i;
           break;
         }
       }
     }
   } else {
     // Looking back through the definition for each operand could be expensive,
     // so exit early if this isn't an instruction that either has an immediate
     // form or is already an immediate form that we can handle.
if (!instrHasImmForm(MI, III) && !ConvertibleImmForm)
       return nullptr;
 
     // Don't convert or %X, %Y, %Y since that's just a register move.
     if ((Opc == PPC::OR || Opc == PPC::OR8) &&
         MI.getOperand(1).getReg() == MI.getOperand(2).getReg())
       return nullptr;
     for (int i = 1, e = MI.getNumOperands(); i < e; i++) {
       MachineOperand &MO = MI.getOperand(i);
-      SeenIntermediateUse = false;
       if (MO.isReg() && MO.isUse() && !MO.isImplicit()) {
         MachineBasicBlock::reverse_iterator E = MI.getParent()->rend(), It = MI;
         It++;
         unsigned Reg = MI.getOperand(i).getReg();
         // MachineInstr::readsRegister only returns true if the machine
         // instruction reads the exact register or its super-register. It
         // does not consider uses of sub-registers which seems like strange
         // behaviour. Nonetheless, if we end up with a 64-bit register here,
         // get the corresponding 32-bit register to check.
         if (PPC::G8RCRegClass.contains(Reg))
           Reg = Reg - PPC::X0 + PPC::R0;
 
-        // Is this register defined by a load-immediate in this block?
+        // Is this register defined by some form of add-immediate (including
+        // load-immediate) within this basic block?
         for ( ; It != E; ++It) {
           if (It->modifiesRegister(Reg, &getRegisterInfo())) {
-            if (It->getOpcode() == PPC::LI || It->getOpcode() == PPC::LI8) {
-              ConstOp = i;
+            switch (It->getOpcode()) {
+            default: break;
+            case PPC::LI:
+            case PPC::LI8:
+            case PPC::ADDItocL:
+            case PPC::ADDI:
+            case PPC::ADDI8:
+              OpNoForForwarding = i;
               return &*It;
-            } else
-              break;
-          } else if (It->readsRegister(Reg, &getRegisterInfo()))
-            // If we see another use of this reg between the def and the MI,
-            // we want to flat it so the def isn't deleted.
-            SeenIntermediateUse = true;
+            }
+
           }
         }
       }
     }
-  return ConstOp == ~0U ? nullptr : DefMI;
+  }
+  return OpNoForForwarding == ~0U ? nullptr : DefMI;
 }
 
 const unsigned *PPCInstrInfo::getStoreOpcodesForSpillArray() const {
   static const unsigned OpcodesForSpill[2][SOK_LastOpcodeSpill] = {
       // Power 8
       {PPC::STW, PPC::STD, PPC::STFD, PPC::STFS, PPC::SPILL_CR,
        PPC::SPILL_CRBIT, PPC::STVX, PPC::STXVD2X, PPC::STXSDX, PPC::STXSSPX,
        PPC::SPILL_VRSAVE, PPC::QVSTFDX, PPC::QVSTFSXs, PPC::QVSTFDXb,
static const unsigned OpcodesForSpill[2][SOK_LastOpcodeSpill] = {
        PPC::RESTORE_CRBIT, PPC::LVX, PPC::LXV, PPC::DFLOADf64, PPC::DFLOADf32,
        PPC::RESTORE_VRSAVE, PPC::QVLFDX, PPC::QVLFSXs, PPC::QVLFDXb,
        PPC::SPILLTOVSR_LD}};
 
   return OpcodesForSpill[(Subtarget.hasP9Vector()) ? 1 : 0];
 }
 
 // If this instruction has an immediate form and one of its operands is a
-// result of a load-immediate, convert it to the immediate form if the constant
-// is in range.
+// result of a load-immediate or an add-immediate, convert it to
+// the immediate form if the constant is in range.
 bool PPCInstrInfo::convertToImmediateForm(MachineInstr &MI,
                                           MachineInstr **KilledDef) const {
   MachineFunction *MF = MI.getParent()->getParent();
   MachineRegisterInfo *MRI = &MF->getRegInfo();
   bool PostRA = !MRI->isSSA();
-  bool SeenIntermediateUse = true;
-  unsigned ConstantOperand = ~0U;
-  MachineInstr *DefMI = getConstantDefMI(MI, ConstantOperand,
-                                         SeenIntermediateUse);
-  if (!DefMI || !DefMI->getOperand(1).isImm())
+  unsigned ForwardingOperand = ~0U;
+  MachineInstr *DefMI = getForwardingDefMI(MI, ForwardingOperand);
+  if (!DefMI)
+    return false;
+  assert(ForwardingOperand < MI.getNumOperands() &&
+         "The forwarding operand needs to be valid at this point");
+  bool KillFwdDefMI = isForwardingDefMIKilled(*DefMI, MI, ForwardingOperand);
+  if (KilledDef && KillFwdDefMI)
+    *KilledDef = DefMI;
+
+  ImmInstrInfo III;
+  bool HasImmForm = instrHasImmForm(MI, III);
+  // If this is a reg+reg instruction that has a reg+imm form,
+  // and one of the operands is produced by an add-immediate,
+  // try to convert it.
+  if (HasImmForm && transformToImmFormFedByAdd(MI, III, ForwardingOperand,
+                                               *DefMI, KillFwdDefMI))
+    return true;
+
+  if ((DefMI->getOpcode() != PPC::LI && DefMI->getOpcode() != PPC::LI8) ||
+      !DefMI->getOperand(1).isImm())
     return false;
-  assert(ConstantOperand < MI.getNumOperands() &&
-         "The constant operand needs to be valid at this point");
 
   int64_t Immediate = DefMI->getOperand(1).getImm();
   // Sign-extend to 64-bits.
   int64_t SExtImm = ((uint64_t)Immediate & ~0x7FFFuLL) != 0 ?
     (Immediate | 0xFFFFFFFFFFFF0000) : Immediate;
 
-  if (KilledDef && MI.getOperand(ConstantOperand).isKill() &&
-      !SeenIntermediateUse)
-    *KilledDef = DefMI;
-
-  // If this is a reg+reg instruction that has a reg+imm form, convert it now.
-  ImmInstrInfo III;
-  if (instrHasImmForm(MI, III))
-    return transformToImmForm(MI, III, ConstantOperand, SExtImm);
+  // If this is a reg+reg instruction that has a reg+imm form,
+  // and one of the operands is produced by LI, convert it now.
+  if (HasImmForm)
+    return transformToImmFormFedByLI(MI, III, ForwardingOperand, SExtImm);
 
   bool ReplaceWithLI = false;
   bool Is64BitLI = false;
   int64_t NewImm = 0;
   bool SetCR = false;
   unsigned Opc = MI.getOpcode();
   switch (Opc) {
   default: return false;

 bool PPCInstrInfo::instrHasImmForm(const MachineInstr &MI,
                                    ImmInstrInfo &III) const {
   unsigned Opc = MI.getOpcode();
   // The vast majority of the instructions would need their operand 2 replaced
   // with an immediate when switching to the reg+imm form. A marked exception
   // are the update form loads/stores for which a constant operand 2 would need
   // to turn into a displacement and move operand 1 to the operand 2 position.
   III.ImmOpNo = 2;
-  III.ConstantOpNo = 2;
+  III.OpNoForForwarding = 2;
   III.ImmWidth = 16;
   III.ImmMustBeMultipleOf = 1;
   III.TruncateImmTo = 0;
+  III.IsSummingOperands = false;
   switch (Opc) {
   default: return false;
   case PPC::ADD4:
   case PPC::ADD8:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 0;
     III.ZeroIsSpecialNew = 1;
     III.IsCommutative = true;
+    III.IsSummingOperands = true;
     III.ImmOpcode = Opc == PPC::ADD4 ? PPC::ADDI : PPC::ADDI8;
     break;
   case PPC::ADDC:
   case PPC::ADDC8:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 0;
     III.ZeroIsSpecialNew = 0;
     III.IsCommutative = true;
+    III.IsSummingOperands = true;
     III.ImmOpcode = Opc == PPC::ADDC ? PPC::ADDIC : PPC::ADDIC8;
     break;
   case PPC::ADDCo:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 0;
     III.ZeroIsSpecialNew = 0;
     III.IsCommutative = true;
+    III.IsSummingOperands = true;
     III.ImmOpcode = PPC::ADDICo;
     break;
   case PPC::SUBFC:
   case PPC::SUBFC8:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 0;
     III.ZeroIsSpecialNew = 0;
     III.IsCommutative = false;
bool PPCInstrInfo::instrHasImmForm(const MachineInstr &MI,
   case PPC::STWX8:
   case PPC::STDX:
   case PPC::STFSX:
   case PPC::STFDX:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 1;
     III.ZeroIsSpecialNew = 2;
     III.IsCommutative = true;
+    III.IsSummingOperands = true;
     III.ImmOpNo = 1;
-    III.ConstantOpNo = 2;
+    III.OpNoForForwarding = 2;
     switch(Opc) {
     default: llvm_unreachable("Unknown opcode");
     case PPC::LBZX: III.ImmOpcode = PPC::LBZ; break;
     case PPC::LBZX8: III.ImmOpcode = PPC::LBZ8; break;
     case PPC::LHZX: III.ImmOpcode = PPC::LHZ; break;
     case PPC::LHZX8: III.ImmOpcode = PPC::LHZ8; break;
     case PPC::LHAX: III.ImmOpcode = PPC::LHA; break;
     case PPC::LHAX8: III.ImmOpcode = PPC::LHA8; break;
bool PPCInstrInfo::instrHasImmForm(const MachineInstr &MI,
   case PPC::STWUX8:
   case PPC::STDUX:
   case PPC::STFSUX:
   case PPC::STFDUX:
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 2;
     III.ZeroIsSpecialNew = 3;
     III.IsCommutative = false;
+    III.IsSummingOperands = true;
     III.ImmOpNo = 2;
-    III.ConstantOpNo = 3;
+    III.OpNoForForwarding = 3;
     switch(Opc) {
     default: llvm_unreachable("Unknown opcode");
     case PPC::LBZUX: III.ImmOpcode = PPC::LBZU; break;
     case PPC::LBZUX8: III.ImmOpcode = PPC::LBZU8; break;
     case PPC::LHZUX: III.ImmOpcode = PPC::LHZU; break;
     case PPC::LHZUX8: III.ImmOpcode = PPC::LHZU8; break;
     case PPC::LHAUX: III.ImmOpcode = PPC::LHAU; break;
     case PPC::LHAUX8: III.ImmOpcode = PPC::LHAU8; break;
bool PPCInstrInfo::instrHasImmForm(const MachineInstr &MI,
   case PPC::STXSSPX:
   case PPC::STXSDX:
     if (!Subtarget.hasP9Vector())
       return false;
     III.SignedImm = true;
     III.ZeroIsSpecialOrig = 1;
     III.ZeroIsSpecialNew = 2;
     III.IsCommutative = true;
+    III.IsSummingOperands = true;
     III.ImmOpNo = 1;
-    III.ConstantOpNo = 2;
+    III.OpNoForForwarding = 2;
     switch(Opc) {
     default: llvm_unreachable("Unknown opcode");
     case PPC::LXVX:
       III.ImmOpcode = PPC::LXV;
       III.ImmMustBeMultipleOf = 16;
       break;
     case PPC::LXSSPX:
       III.ImmOpcode = PPC::LXSSP;
for (unsigned i = MI.getNumOperands(); i < TotalOps; i++) {
       else {
         MI.addOperand(MOps.back());
         MOps.pop_back();
       }
     }
   }
 }
 
-bool PPCInstrInfo::transformToImmForm(MachineInstr &MI, const ImmInstrInfo &III,
+// Check if the 'MI' that has the index OpNoForForwarding 
+// meets the requirement describe in the ImmInstrInfo.
+bool PPCInstrInfo::isUseMIElgibleForForwarding(MachineInstr &MI,
+                                               const ImmInstrInfo &III,
+                                               unsigned OpNoForForwarding
+                                               ) const {
+  // As the algorithm of checking for PPC::ZERO/PPC::ZERO8
+  // would not work pre-RA, we can only do the check post RA.
+  MachineRegisterInfo &MRI = MI.getParent()->getParent()->getRegInfo();
+  if (MRI.isSSA())
+    return false;
+
+  // Cannot do the transform if MI isn't summing the operands.
+  if (!III.IsSummingOperands)
+    return false;
+
+  // The instruction we are trying to replace must have the ZeroIsSpecialOrig set.
+  if (!III.ZeroIsSpecialOrig)
+    return false;
+
+  // We cannot do the transform if the operand we are trying to replace
+  // isn't the same as the operand the instruction allows.
+  if (OpNoForForwarding != III.OpNoForForwarding)
+    return false;
+
+  // Check if the instruction we are trying to transform really has
+  // the special zero register as its operand.
+  if (MI.getOperand(III.ZeroIsSpecialOrig).getReg() != PPC::ZERO &&
+      MI.getOperand(III.ZeroIsSpecialOrig).getReg() != PPC::ZERO8)
+    return false;
+
+  // This machine instruction is convertible if it is,
+  // 1. summing the operands.
+  // 2. one of the operands is special zero register.
+  // 3. the operand we are trying to replace is allowed by the MI.
+  return true;
+}
+
+// Check if the DefMI is the add inst and set the ImmMO and RegMO
+// accordingly.
+bool PPCInstrInfo::isDefMIElgibleForForwarding(MachineInstr &DefMI,
+                                               const ImmInstrInfo &III,
+                                               MachineOperand *&ImmMO,
+                                               MachineOperand *&RegMO) const {
+  unsigned Opc = DefMI.getOpcode();
+  if (Opc != PPC::ADDItocL && Opc != PPC::ADDI && Opc != PPC::ADDI8)
+    return false; 
+
+  assert(DefMI.getNumOperands() >= 3 &&
+         "Add inst must have at least three operands");
+  RegMO = &DefMI.getOperand(1);
+  ImmMO = &DefMI.getOperand(2);
+
+  // This DefMI is elgible for forwarding if it is:
+  // 1. add inst
+  // 2. one of the operands is Imm/CPI/Global.
+  return isAnImmediateOperand(*ImmMO);
+}
+
+bool PPCInstrInfo::isRegElgibleForForwarding(const MachineOperand &RegMO,
+                                             const MachineInstr &DefMI,
+                                             const MachineInstr &MI,
+                                             bool KillDefMI
+                                             ) const {
+  // x = addi y, imm
+  // ...
+  // z = lfdx 0, x   -> z = lfd imm(y)
+  // The Reg "y" can be forwarded to the MI(z) only when there is no DEF
+  // of "y" between the DEF of "x" and "z".
+  const MachineInstr *LastDefMI = nullptr;
+  const MachineInstr *LastUseMI = nullptr;
+  if (!getLastDefAndUseMIInBetween(RegMO.getReg(), DefMI, MI,
+                                   LastDefMI, LastUseMI))
+    return false;
+
+  if (LastDefMI) {
+    // Cannot forward the "y" if see another DEF and it is not DefMI.
+    if (LastDefMI != &DefMI)
+      return false;
+
+    // This is the pattern here:
+    // x = addi x, imm
+    // ...
+    // z = lfdx 0, x
+    // If the DefMI isn't killed, we cannot forward the "x" as its value
+    // is updated. 
+    assert(LastUseMI && "No one is using RegMO");
+    return KillDefMI && (LastUseMI == &DefMI);
+  }
+
+  return true;
+}
+
+bool PPCInstrInfo::isImmElgibleForForwarding(const MachineOperand &ImmMO,
+                                             const MachineInstr &DefMI,
+                                             const ImmInstrInfo &III,
+                                             int64_t &Imm) const {
+  assert(isAnImmediateOperand(ImmMO) && "ImmMO is NOT an immediate");
+  if (DefMI.getOpcode() == PPC::ADDItocL) {
+    // The operand for ADDItocL is CPI, which isn't imm at compiling time,
+    // However, we know that, it is 16-bit width, and has the alignment of 4.
+    // Check if the instruction met the requirement.
+    if (III.ImmMustBeMultipleOf > 4 ||
+       III.TruncateImmTo || III.ImmWidth != 16)
+      return false;
+
+    return true;
+  }
+
+  if (ImmMO.isImm()) {
+    // It is Imm, we need to check if the Imm fit the range.
+    int64_t Immediate = ImmMO.getImm();
+    // Sign-extend to 64-bits.
+    Imm = ((uint64_t)Immediate & ~0x7FFFuLL) != 0 ?
+      (Immediate | 0xFFFFFFFFFFFF0000) : Immediate;
+
+    if (Imm % III.ImmMustBeMultipleOf)
+      return false;
+    if (III.TruncateImmTo)
+      Imm &= ((1 << III.TruncateImmTo) - 1);
+    if (III.SignedImm) {
+      APInt ActualValue(64, Imm, true);
+      if (!ActualValue.isSignedIntN(III.ImmWidth))
+        return false;
+    } else {
+      uint64_t UnsignedMax = (1 << III.ImmWidth) - 1;
+      if ((uint64_t)Imm > UnsignedMax)
+        return false;
+    }
+  }
+  else
+    return false;
+
+  // This ImmMO is forwarded if it meets the requriement describle
+  // in ImmInstrInfo
+  return true;
+}
+
+bool PPCInstrInfo::isForwardingDefMIKilled(const MachineInstr &DefMI,
+                                           const MachineInstr &MI,
+                                           unsigned OpNoForForwarding) const {
+  // Determine if the DefMI is safe to kill. First, get the last def and
+  // use MI for this Reg in the range of [DefMI, MI). 
+  const MachineInstr *LastDefMI = nullptr;
+  const MachineInstr *LastUseMI = nullptr;
+  const MachineOperand &MO = MI.getOperand(OpNoForForwarding);
+  if (!getLastDefAndUseMIInBetween(MO.getReg(),
+                                   DefMI, MI, LastDefMI, LastUseMI))
+    return false;
+  // Because MI is fed by DefMI, it is expected that, the nearest DEF is the
+  // DefMI itself.
+  assert(LastDefMI && LastDefMI == &DefMI &&
+         "The DefMI isn't the nearest def for operand OpNoForForwarding.");
+
+  // Kill the DefMI if it isn't used between (DefMI, MI) and after MI.
+  return (LastUseMI == nullptr || LastUseMI == &DefMI) && MO.isKill();
+}
+
+// If an X-Form instruction is fed by an add-immediate and one of its operands
+// is the literal zero, attempt to forward the source of the add-immediate to
+// the corresponding D-Form instruction with the displacement coming from
+// the immediate being added.
+bool PPCInstrInfo::transformToImmFormFedByAdd(MachineInstr &MI,
+                                              const ImmInstrInfo &III,
+                                              unsigned OpNoForForwarding,
+                                              MachineInstr &DefMI,
+                                              bool KillDefMI) const {
+  //         RegMO ImmMO
+  //           |    |
+  // x = addi reg, imm  <----- DefMI
+  // y = op    0 ,  x   <----- MI
+  //                |
+  //         OpNoForForwarding
+  // Check if the MI meet the requirement described in the III.
+  if (!isUseMIElgibleForForwarding(MI, III, OpNoForForwarding))
+    return false;
+
+  // Check if the DefMI meet the requirement
+  // described in the III. If yes, set the ImmMO and RegMO accordingly.
+  MachineOperand *ImmMO = nullptr;
+  MachineOperand *RegMO = nullptr;
+  if (!isDefMIElgibleForForwarding(DefMI, III, ImmMO, RegMO))
+    return false;
+  assert(ImmMO && RegMO && "Imm and Reg operand must have been set");
+
+  // As we get the Imm operand now, we need to check if the ImmMO meet
+  // the requirement described in the III. If yes set the Imm.
+  int64_t Imm = 0;
+  if (!isImmElgibleForForwarding(*ImmMO, DefMI, III, Imm))
+    return false;
+
+  // Check if the RegMO can be forwarded to MI.
+  if (!isRegElgibleForForwarding(*RegMO, DefMI, MI, KillDefMI))
+    return false;
+
+  // We know that, the MI and DefMI both meet the pattern, and
+  // the Imm also meet the requirement with the new Imm-form.
+  // It is safe to do the transformation now.
+  LLVM_DEBUG(dbgs() << "Replacing instruction:\n");
+  LLVM_DEBUG(MI.dump());
+  LLVM_DEBUG(dbgs() << "Fed by:\n");
+  LLVM_DEBUG(DefMI.dump());
+
+  // Update the base reg first.
+  MI.getOperand(III.OpNoForForwarding).ChangeToRegister(RegMO->getReg(),
+                                                        false, false,
+                                                        RegMO->isKill());
+
+  // Then, update the imm.
+  if (ImmMO->isImm()) {
+    // If the ImmMO is Imm, change the operand that has ZERO to that Imm
+    // directly.
+    MI.getOperand(III.ZeroIsSpecialOrig).ChangeToImmediate(Imm);
+  }
+  else {
+    // Otherwise, it is Constant Pool Index(CPI) or Global,
+    // which is relocation in fact. We need to replace the special zero
+    // register with ImmMO.
+    // Before that, we need to fixup the target flags for imm. 
+    // For some reason, we miss to set the flag for the ImmMO if it is CPI.
+    if (DefMI.getOpcode() == PPC::ADDItocL)
+      ImmMO->setTargetFlags(PPCII::MO_TOC_LO);
+
+    // MI didn't have the interface such as MI.setOperand(i) though
+    // it has MI.getOperand(i). To repalce the ZERO MachineOperand with
+    // ImmMO, we need to remove ZERO operand and all the operands behind it,
+    // and, add the ImmMO, then, move back all the operands behind ZERO.
+    SmallVector<MachineOperand, 2> MOps;
+    for (unsigned i = MI.getNumOperands() - 1; i >= III.ZeroIsSpecialOrig; i--) {
+      MOps.push_back(MI.getOperand(i));
+      MI.RemoveOperand(i);
+    }
+
+    // Remove the last MO in the list, which is ZERO operand in fact.
+    MOps.pop_back();
+    // Add the imm operand.
+    MI.addOperand(*ImmMO);
+    // Now add the rest back.
+    for (auto &MO : MOps)
+      MI.addOperand(MO);
+  }
+
+  // Update the opcode.
+  MI.setDesc(get(III.ImmOpcode));
+
+  LLVM_DEBUG(dbgs() << "With:\n");
+  LLVM_DEBUG(MI.dump());
+
+  return true;
+}
+
+bool PPCInstrInfo::transformToImmFormFedByLI(MachineInstr &MI,
+                                             const ImmInstrInfo &III,
                                              unsigned ConstantOpNo,
                                              int64_t Imm) const {
   MachineRegisterInfo &MRI = MI.getParent()->getParent()->getRegInfo();
   bool PostRA = !MRI.isSSA();
   // Exit early if we can't convert this.
-  if ((ConstantOpNo != III.ConstantOpNo) && !III.IsCommutative)
+  if ((ConstantOpNo != III.OpNoForForwarding) && !III.IsCommutative)
     return false;
   if (Imm % III.ImmMustBeMultipleOf)
     return false;
   if (III.TruncateImmTo)
     Imm &= ((1 << III.TruncateImmTo) - 1);
   if (III.SignedImm) {
     APInt ActualValue(64, Imm, true);
     if (!ActualValue.isSignedIntN(III.ImmWidth))
bool PPCInstrInfo::transformToImmFormFedByLI(MachineInstr &MI,
   bool SpecialShift64 =
     Opc == PPC::SLD || Opc == PPC::SLDo || Opc == PPC::SRD || Opc == PPC::SRDo;
   bool SetCR = Opc == PPC::SLWo || Opc == PPC::SRWo ||
     Opc == PPC::SLDo || Opc == PPC::SRDo;
   bool RightShift =
     Opc == PPC::SRW || Opc == PPC::SRWo || Opc == PPC::SRD || Opc == PPC::SRDo;
 
   MI.setDesc(get(III.ImmOpcode));
-  if (ConstantOpNo == III.ConstantOpNo) {
+  if (ConstantOpNo == III.OpNoForForwarding) {
     // Converting shifts to immediate form is a bit tricky since they may do
     // one of three things:
     // 1. If the shift amount is between OpSize and 2*OpSize, the result is zero
     // 2. If the shift amount is zero, the result is unchanged (save for maybe
     //    setting CR0)
     // 3. If the shift amount is in [1, OpSize), it's just a shift
     if (SpecialShift32 || SpecialShift64) {
       LoadImmediateInfo LII;
if (SpecialShift32 || SpecialShift64) {
         MI.setDesc(get(PPC::COPY));
       } else {
         // The 32 bit and 64 bit instructions are quite different.
         if (SpecialShift32) {
           // Left shifts use (N, 0, 31-N), right shifts use (32-N, N, 31).
           uint64_t SH = RightShift ? 32 - ShAmt : ShAmt;
           uint64_t MB = RightShift ? ShAmt : 0;
           uint64_t ME = RightShift ? 31 : 31 - ShAmt;
-          MI.getOperand(III.ConstantOpNo).ChangeToImmediate(SH);
+          MI.getOperand(III.OpNoForForwarding).ChangeToImmediate(SH);
           MachineInstrBuilder(*MI.getParent()->getParent(), MI).addImm(MB)
             .addImm(ME);
         } else {
           // Left shifts use (N, 63-N), right shifts use (64-N, N).
           uint64_t SH = RightShift ? 64 - ShAmt : ShAmt;
           uint64_t ME = RightShift ? ShAmt : 63 - ShAmt;
-          MI.getOperand(III.ConstantOpNo).ChangeToImmediate(SH);
+          MI.getOperand(III.OpNoForForwarding).ChangeToImmediate(SH);
           MachineInstrBuilder(*MI.getParent()->getParent(), MI).addImm(ME);
         }
       }
     } else
       MI.getOperand(ConstantOpNo).ChangeToImmediate(Imm);
   }
   // Convert commutative instructions (switch the operands and convert the
   // desired one to an immediate.
   else if (III.IsCommutative) {
     MI.getOperand(ConstantOpNo).ChangeToImmediate(Imm);
-    swapMIOperands(MI, ConstantOpNo, III.ConstantOpNo);
+    swapMIOperands(MI, ConstantOpNo, III.OpNoForForwarding);
   } else
     llvm_unreachable("Should have exited early!");
 
   // For instructions for which the constant register replaces a different
   // operand than where the immediate goes, we need to swap them.
-  if (III.ConstantOpNo != III.ImmOpNo)
-    swapMIOperands(MI, III.ConstantOpNo, III.ImmOpNo);
+  if (III.OpNoForForwarding != III.ImmOpNo)
+    swapMIOperands(MI, III.OpNoForForwarding, III.ImmOpNo);
 
   // If the R0/X0 register is special for the original instruction and not for
   // the new instruction (or vice versa), we need to fix up the register class.
   if (!PostRA && III.ZeroIsSpecialOrig != III.ZeroIsSpecialNew) {
     if (!III.ZeroIsSpecialOrig) {
       unsigned RegToModify = MI.getOperand(III.ZeroIsSpecialNew).getReg();
       const TargetRegisterClass *NewRC =
         MRI.getRegClass(RegToModify)->hasSuperClassEq(&PPC::GPRCRegClass) ?

llvm/test/CodeGen/PowerPC/bitcasts-direct-move.ll

 ; CHECK: xxsldwi [[SHIFTREG:[0-9]+]], [[CONVREG]], [[CONVREG]], 3
 ; CHECK: mfvsrwz 3, [[SHIFTREG]]
 }
 
 define i64 @f64toi64(double %a) {
 entry:
   %0 = bitcast double %a to i64
   ret i64 %0
-; CHECK-P7: stfdx 1,
+; CHECK-P7: stfd 1,
 ; CHECK-P7: ld 3,
 ; CHECK: mffprd 3, 1
 }
 
 define float @i32tof32(i32 signext %a) {
 entry:
   %0 = bitcast i32 %a to float
   ret float %0
 ; CHECK-P7: stw 3,
 ; CHECK-P7: lfs 1,
 ; CHECK: mtvsrd [[MOVEREG:[0-9]+]], 3
 ; CHECK: xxsldwi [[SHIFTREG:[0-9]+]], [[MOVEREG]], [[MOVEREG]], 1
 ; CHECK: xscvspdpn 1, [[SHIFTREG]]
 }
 
 define double @i64tof64(i64 %a) {
 entry:
   %0 = bitcast i64 %a to double
   ret double %0
 ; CHECK-P7: std 3,
-; CHECK-P7: lfdx 1,
+; CHECK-P7: lfd 1,
 ; CHECK: mtvsrd 1, 3
 }
 
 define zeroext i32 @f32toi32u(float %a) {
 entry:
   %0 = bitcast float %a to i32
   ret i32 %0
 ; CHECK-P7: stfs 1,
 ; CHECK-P7: lwz 3,
 ; CHECK: xscvdpspn [[CONVREG:[0-9]+]], 1
 ; CHECK: xxsldwi [[SHIFTREG:[0-9]+]], [[CONVREG]], [[CONVREG]], 3
 ; CHECK: mfvsrwz 3, [[SHIFTREG]]
 }
 
 define i64 @f64toi64u(double %a) {
 entry:
   %0 = bitcast double %a to i64
   ret i64 %0
-; CHECK-P7: stfdx 1,
+; CHECK-P7: stfd 1,
 ; CHECK-P7: ld 3,
 ; CHECK: mffprd 3, 1
 }
 
 define float @i32utof32(i32 zeroext %a) {
 entry:
   %0 = bitcast i32 %a to float
   ret float %0
 ; CHECK-P7: stw 3,
 ; CHECK-P7: lfs 1,
 ; CHECK: mtvsrd [[MOVEREG:[0-9]+]], 3
 ; CHECK: xxsldwi [[SHIFTREG:[0-9]+]], [[MOVEREG]], [[MOVEREG]], 1
 ; CHECK: xscvspdpn 1, [[SHIFTREG]]
 }
 
 define double @i64utof64(i64 %a) {
 entry:
   %0 = bitcast i64 %a to double
   ret double %0
 ; CHECK-P7: std 3,
-; CHECK-P7: lfdx 1,
+; CHECK-P7: lfd 1,
 ; CHECK: mtvsrd 1, 3
 }

llvm/test/CodeGen/PowerPC/branch_coalesce.ll

 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -verify-machineinstrs -enable-ppc-branch-coalesce < %s | FileCheck %s
 ; RUN: llc -mcpu=pwr8 -mtriple=powerpc64-unknown-linux-gnu -verify-machineinstrs -enable-ppc-branch-coalesce < %s | FileCheck %s
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -verify-machineinstrs < %s | FileCheck --check-prefix=CHECK-NOCOALESCE %s
 ; RUN: llc -mcpu=pwr8 -mtriple=powerpc64-unknown-linux-gnu -verify-machineinstrs < %s | FileCheck --check-prefix=CHECK-NOCOALESCE %s
 
 ; Function Attrs: nounwind
 define double @testBranchCoal(double %a, double %b, double %c, i32 %x) {
 
 ; CHECK-LABEL: @testBranchCoal
 ; CHECK: cmplwi [[CMPR:[0-7]+]], 6, 0
 ; CHECK: beq [[CMPR]], .LBB[[LAB1:[0-9_]+]]
 ; CHECK-DAG: addis [[LD1REG:[0-9]+]], 2, .LCPI0_0@toc@ha
 ; CHECK-DAG: addis [[LD2REG:[0-9]+]], 2, .LCPI0_1@toc@ha
 ; CHECK-DAG: xxlxor 2, 2, 2
 ; CHECK-NOT: beq
-; CHECK-DAG: addi [[LD1BASE:[0-9]+]], [[LD1REG]]
-; CHECK-DAG: addi [[LD2BASE:[0-9]+]], [[LD2REG]]
-; CHECK-DAG: lfdx 1, 0, [[LD1BASE]]
-; CHECK-DAG: lfdx 3, 0, [[LD2BASE]]
+; CHECK-DAG: lfd 1, .LCPI0_0@toc@l([[LD1REG]])
+; CHECK-DAG: lfd 3, .LCPI0_1@toc@l([[LD2REG]])
 ; CHECK: .LBB[[LAB1]]
 ; CHECK: xsadddp 0, 1, 2
 ; CHECK: xsadddp 1, 0, 3
 ; CHECK: blr
 
 ; CHECK-NOCOALESCE-LABEL: testBranchCoal:
 ; CHECK-NOCOALESCE:       # %bb.0: # %entry
 ; CHECK-NOCOALESCE-NEXT:    cmplwi 0, 6, 0
 ; CHECK-NOCOALESCE-NEXT:    bne 0, .LBB0_5
 ; CHECK-NOCOALESCE-NEXT:  # %bb.1: # %entry
 ; CHECK-NOCOALESCE-NEXT:    bne 0, .LBB0_6
 ; CHECK-NOCOALESCE-NEXT:  .LBB0_2: # %entry
 ; CHECK-NOCOALESCE-NEXT:    beq 0, .LBB0_4
 ; CHECK-NOCOALESCE-NEXT:  .LBB0_3: # %entry
 ; CHECK-NOCOALESCE-NEXT:    addis 3, 2, .LCPI0_1@toc@ha
-; CHECK-NOCOALESCE-NEXT:    addi 3, 3, .LCPI0_1@toc@l
-; CHECK-NOCOALESCE-NEXT:    lfdx 3, 0, 3
+; CHECK-NOCOALESCE-NEXT:    lfd 3, .LCPI0_1@toc@l(3)
 ; CHECK-NOCOALESCE-NEXT:  .LBB0_4: # %entry
 ; CHECK-NOCOALESCE-NEXT:    xsadddp 0, 1, 2
 ; CHECK-NOCOALESCE-NEXT:    xsadddp 1, 0, 3
 ; CHECK-NOCOALESCE-NEXT:    blr
 ; CHECK-NOCOALESCE-NEXT:  .LBB0_5: # %entry
 ; CHECK-NOCOALESCE-NEXT:    addis 3, 2, .LCPI0_0@toc@ha
-; CHECK-NOCOALESCE-NEXT:    addi 3, 3, .LCPI0_0@toc@l
-; CHECK-NOCOALESCE-NEXT:    lfdx 1, 0, 3
+; CHECK-NOCOALESCE-NEXT:    lfd 1, .LCPI0_0@toc@l(3)
 ; CHECK-NOCOALESCE-NEXT:    beq 0, .LBB0_2
 ; CHECK-NOCOALESCE-NEXT:  .LBB0_6: # %entry
 ; CHECK-NOCOALESCE-NEXT:    xxlxor 2, 2, 2
 ; CHECK-NOCOALESCE-NEXT:    bne 0, .LBB0_3
 ; CHECK-NOCOALESCE-NEXT:    b .LBB0_4
   entry:
 
   %test = icmp eq i32 %x, 0
   %tmp1 = select i1 %test, double %a, double 2.000000e-03
   %tmp2 = select i1 %test, double %b, double 0.000000e+00
   %tmp3 = select i1 %test, double %c, double 5.000000e-03
 
   %res1 = fadd double %tmp1, %tmp2
   %result = fadd double %res1, %tmp3
   ret double %result
 }

llvm/test/CodeGen/PowerPC/fast-isel-load-store-vsx.ll

-; RUN: llc < %s -O0 -fast-isel -mattr=+vsx -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 | FileCheck %s --check-prefix=ELF64VSX
+; RUN: llc < %s -O0 -fast-isel -mattr=+vsx -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 -ppc-late-peephole=false | FileCheck %s --check-prefix=ELF64VSX
 
 ;; The semantics of VSX stores for when R0 is used is different depending on
 ;; whether it is used as base or offset. If used as base, the effective
 ;; address computation will use zero regardless of the content of R0. If used as
 ;; an offset the content will be used in the effective address. We observed that
 ;; for some constructors, the initialization values were being stored without
 ;; an offset register being specified which was causing R0 to be used as offset
 ;; in regions where it contained the value in the link register. This test

llvm/test/CodeGen/PowerPC/float-to-int.ll

 ; CHECK: @foo
 ; CHECK: fctidz [[REG:[0-9]+]], 1
 ; CHECK: stfd [[REG]],
 ; CHECK: ld 3,
 ; CHECK: blr
 
 ; CHECK-VSX: @foo
 ; CHECK-VSX: xscvdpsxds [[REG:[0-9]+]], 1
-; CHECK-VSX: stfdx [[REG]],
+; CHECK-VSX: stfd [[REG]],
 ; CHECK-VSX: ld 3,
 ; CHECK-VSX: blr
 
 ; CHECK-LABEL-P9: @foo
 ; CHECK-P9: xscvdpsxds [[REG:[0-9]+]], 1
 ; CHECK-P9: stfd [[REG]],
 ; CHECK-P9: ld 3,
 ; CHECK-P9: blr
 }
 
 define i64 @foo2(double %a) nounwind {
   %x = fptosi double %a to i64
   ret i64 %x
 
 ; CHECK: @foo2
 ; CHECK: fctidz [[REG:[0-9]+]], 1
 ; CHECK: stfd [[REG]],
 ; CHECK: ld 3,
 ; CHECK: blr
 
 ; CHECK-VSX: @foo2
 ; CHECK-VSX: xscvdpsxds [[REG:[0-9]+]], 1
-; CHECK-VSX: stfdx [[REG]],
+; CHECK-VSX: stfd [[REG]],
 ; CHECK-VSX: ld 3,
 ; CHECK-VSX: blr
 
 ; CHECK-LABEL-P9: @foo2
 ; CHECK-P9: xscvdpsxds [[REG:[0-9]+]], 1
 ; CHECK-P9: stfd [[REG]],
 ; CHECK-P9: ld 3,
 ; CHECK-P9: blr
 }
 
 define i64 @foo3(float %a) nounwind {
   %x = fptoui float %a to i64
   ret i64 %x
 
 ; CHECK: @foo3
 ; CHECK: fctiduz [[REG:[0-9]+]], 1
 ; CHECK: stfd [[REG]],
 ; CHECK: ld 3,
 ; CHECK: blr
 
 ; CHECK-VSX: @foo3
 ; CHECK-VSX: xscvdpuxds [[REG:[0-9]+]], 1
-; CHECK-VSX: stfdx [[REG]],
+; CHECK-VSX: stfd [[REG]],
 ; CHECK-VSX: ld 3,
 ; CHECK-VSX: blr
 
 ; CHECK-LABEL-P9: @foo3
 ; CHECK-P9: xscvdpuxds [[REG:[0-9]+]], 1
 ; CHECK-P9: stfd [[REG]],
 ; CHECK-P9: ld 3,
 ; CHECK-P9: blr
 }
 
 define i64 @foo4(double %a) nounwind {
   %x = fptoui double %a to i64
   ret i64 %x
 
 ; CHECK: @foo4
 ; CHECK: fctiduz [[REG:[0-9]+]], 1
 ; CHECK: stfd [[REG]],
 ; CHECK: ld 3,
 ; CHECK: blr
 
 ; CHECK-VSX: @foo4
 ; CHECK-VSX: xscvdpuxds [[REG:[0-9]+]], 1
-; CHECK-VSX: stfdx [[REG]],
+; CHECK-VSX: stfd [[REG]],
 ; CHECK-VSX: ld 3,
 ; CHECK-VSX: blr
 
 ; CHECK-LABEL-P9: @foo4
 ; CHECK-P9: xscvdpuxds [[REG:[0-9]+]], 1
 ; CHECK-P9: stfd [[REG]],
 ; CHECK-P9: ld 3,
 ; CHECK-P9: blr

llvm/test/CodeGen/PowerPC/fmf-propagation.ll

 ; GLOBALDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_reassoc1:'
 ; GLOBALDEBUG:         fmul reassoc {{t[0-9]+}}
 ; GLOBALDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_reassoc1:'
 
 define float @fmul_fma_reassoc1(float %x) {
 ; FMF-LABEL: fmul_fma_reassoc1:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    addis 3, 2, .LCPI6_0@toc@ha
-; FMF-NEXT:    addi 3, 3, .LCPI6_0@toc@l
-; FMF-NEXT:    lfsx 0, 0, 3
+; FMF-NEXT:    lfs 0, .LCPI6_0@toc@l(3)
 ; FMF-NEXT:    xsmulsp 1, 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: fmul_fma_reassoc1:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI6_0@toc@ha
-; GLOBAL-NEXT:    addi 3, 3, .LCPI6_0@toc@l
-; GLOBAL-NEXT:    lfsx 0, 0, 3
+; GLOBAL-NEXT:    lfs 0, .LCPI6_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmulsp 1, 1, 0
 ; GLOBAL-NEXT:    blr
   %mul = fmul float %x, 42.0
   %fma = call reassoc float @llvm.fma.f32(float %x, float 7.0, float %mul)
   ret float %fma
 }
 
 ; This shouldn't change anything - the intermediate fmul result is now also flagged.
 
 ; FMFDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_reassoc2:'
 ; FMFDEBUG:         fmul reassoc {{t[0-9]+}}
 ; FMFDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_reassoc2:'
 
 ; GLOBALDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_reassoc2:'
 ; GLOBALDEBUG:         fmul reassoc {{t[0-9]+}}
 ; GLOBALDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_reassoc2:'
 
 define float @fmul_fma_reassoc2(float %x) {
 ; FMF-LABEL: fmul_fma_reassoc2:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    addis 3, 2, .LCPI7_0@toc@ha
-; FMF-NEXT:    addi 3, 3, .LCPI7_0@toc@l
-; FMF-NEXT:    lfsx 0, 0, 3
+; FMF-NEXT:    lfs 0, .LCPI7_0@toc@l(3)
 ; FMF-NEXT:    xsmulsp 1, 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: fmul_fma_reassoc2:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI7_0@toc@ha
-; GLOBAL-NEXT:    addi 3, 3, .LCPI7_0@toc@l
-; GLOBAL-NEXT:    lfsx 0, 0, 3
+; GLOBAL-NEXT:    lfs 0, .LCPI7_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmulsp 1, 1, 0
 ; GLOBAL-NEXT:    blr
   %mul = fmul reassoc float %x, 42.0
   %fma = call reassoc float @llvm.fma.f32(float %x, float 7.0, float %mul)
   ret float %fma
 }
 
 ; The FMA is now fully 'fast'. This implies that reassociation is allowed.
 
 ; FMFDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_fast1:'
 ; FMFDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; FMFDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_fast1:'
 
 ; GLOBALDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_fast1:'
 ; GLOBALDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; GLOBALDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_fast1:'
 
 define float @fmul_fma_fast1(float %x) {
 ; FMF-LABEL: fmul_fma_fast1:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    addis 3, 2, .LCPI8_0@toc@ha
-; FMF-NEXT:    addi 3, 3, .LCPI8_0@toc@l
-; FMF-NEXT:    lfsx 0, 0, 3
+; FMF-NEXT:    lfs 0, .LCPI8_0@toc@l(3)
 ; FMF-NEXT:    xsmulsp 1, 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: fmul_fma_fast1:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI8_0@toc@ha
-; GLOBAL-NEXT:    addi 3, 3, .LCPI8_0@toc@l
-; GLOBAL-NEXT:    lfsx 0, 0, 3
+; GLOBAL-NEXT:    lfs 0, .LCPI8_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmulsp 1, 1, 0
 ; GLOBAL-NEXT:    blr
   %mul = fmul float %x, 42.0
   %fma = call fast float @llvm.fma.f32(float %x, float 7.0, float %mul)
   ret float %fma
 }
 
 ; This shouldn't change anything - the intermediate fmul result is now also flagged.
 
 ; FMFDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_fast2:'
 ; FMFDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; FMFDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_fast2:'
 
 ; GLOBALDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'fmul_fma_fast2:'
 ; GLOBALDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; GLOBALDEBUG:       Type-legalized selection DAG: %bb.0 'fmul_fma_fast2:'
 
 define float @fmul_fma_fast2(float %x) {
 ; FMF-LABEL: fmul_fma_fast2:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    addis 3, 2, .LCPI9_0@toc@ha
-; FMF-NEXT:    addi 3, 3, .LCPI9_0@toc@l
-; FMF-NEXT:    lfsx 0, 0, 3
+; FMF-NEXT:    lfs 0, .LCPI9_0@toc@l(3)
 ; FMF-NEXT:    xsmulsp 1, 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: fmul_fma_fast2:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI9_0@toc@ha
-; GLOBAL-NEXT:    addi 3, 3, .LCPI9_0@toc@l
-; GLOBAL-NEXT:    lfsx 0, 0, 3
+; GLOBAL-NEXT:    lfs 0, .LCPI9_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmulsp 1, 1, 0
 ; GLOBAL-NEXT:    blr
   %mul = fmul fast float %x, 42.0
   %fma = call fast float @llvm.fma.f32(float %x, float 7.0, float %mul)
   ret float %fma
 }
 
 ; Reduced precision for sqrt is allowed - should use estimate and NR iterations.

 ; FMF-LABEL: sqrt_afn:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    xxlxor 0, 0, 0
 ; FMF-NEXT:    fcmpu 0, 1, 0
 ; FMF-NEXT:    beq 0, .LBB10_2
 ; FMF-NEXT:  # %bb.1:
 ; FMF-NEXT:    addis 3, 2, .LCPI10_0@toc@ha
 ; FMF-NEXT:    xsrsqrtesp 3, 1
-; FMF-NEXT:    addi 3, 3, .LCPI10_0@toc@l
-; FMF-NEXT:    lfsx 0, 0, 3
+; FMF-NEXT:    lfs 0, .LCPI10_0@toc@l(3)
 ; FMF-NEXT:    xsmulsp 2, 1, 0
 ; FMF-NEXT:    xsmulsp 4, 3, 3
 ; FMF-NEXT:    xssubsp 2, 2, 1
 ; FMF-NEXT:    xsmulsp 2, 2, 4
 ; FMF-NEXT:    xssubsp 0, 0, 2
 ; FMF-NEXT:    xsmulsp 0, 3, 0
 ; FMF-NEXT:    xsmulsp 0, 0, 1
 ; FMF-NEXT:  .LBB10_2:
 ; FMF-NEXT:    fmr 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: sqrt_afn:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    xxlxor 0, 0, 0
 ; GLOBAL-NEXT:    fcmpu 0, 1, 0
 ; GLOBAL-NEXT:    beq 0, .LBB10_2
 ; GLOBAL-NEXT:  # %bb.1:
 ; GLOBAL-NEXT:    xsrsqrtesp 2, 1
 ; GLOBAL-NEXT:    fneg 0, 1
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI10_0@toc@ha
 ; GLOBAL-NEXT:    fmr 4, 1
-; GLOBAL-NEXT:    addi 3, 3, .LCPI10_0@toc@l
-; GLOBAL-NEXT:    lfsx 3, 0, 3
+; GLOBAL-NEXT:    lfs 3, .LCPI10_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmaddasp 4, 0, 3
 ; GLOBAL-NEXT:    xsmulsp 0, 2, 2
 ; GLOBAL-NEXT:    xsmaddasp 3, 4, 0
 ; GLOBAL-NEXT:    xsmulsp 0, 2, 3
 ; GLOBAL-NEXT:    xsmulsp 0, 0, 1
 ; GLOBAL-NEXT:  .LBB10_2:
 ; GLOBAL-NEXT:    fmr 1, 0
 ; GLOBAL-NEXT:    blr

 ; FMF-NEXT:    xxlxor 0, 0, 0
 ; FMF-NEXT:    fcmpu 0, 1, 0
 ; FMF-NEXT:    beq 0, .LBB11_2
 ; FMF-NEXT:  # %bb.1:
 ; FMF-NEXT:    xsrsqrtesp 2, 1
 ; FMF-NEXT:    fneg 0, 1
 ; FMF-NEXT:    addis 3, 2, .LCPI11_0@toc@ha
 ; FMF-NEXT:    fmr 4, 1
-; FMF-NEXT:    addi 3, 3, .LCPI11_0@toc@l
-; FMF-NEXT:    lfsx 3, 0, 3
+; FMF-NEXT:    lfs 3, .LCPI11_0@toc@l(3)
 ; FMF-NEXT:    xsmaddasp 4, 0, 3
 ; FMF-NEXT:    xsmulsp 0, 2, 2
 ; FMF-NEXT:    xsmaddasp 3, 4, 0
 ; FMF-NEXT:    xsmulsp 0, 2, 3
 ; FMF-NEXT:    xsmulsp 0, 0, 1
 ; FMF-NEXT:  .LBB11_2:
 ; FMF-NEXT:    fmr 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: sqrt_fast:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    xxlxor 0, 0, 0
 ; GLOBAL-NEXT:    fcmpu 0, 1, 0
 ; GLOBAL-NEXT:    beq 0, .LBB11_2
 ; GLOBAL-NEXT:  # %bb.1:
 ; GLOBAL-NEXT:    xsrsqrtesp 2, 1
 ; GLOBAL-NEXT:    fneg 0, 1
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI11_0@toc@ha
 ; GLOBAL-NEXT:    fmr 4, 1
-; GLOBAL-NEXT:    addi 3, 3, .LCPI11_0@toc@l
-; GLOBAL-NEXT:    lfsx 3, 0, 3
+; GLOBAL-NEXT:    lfs 3, .LCPI11_0@toc@l(3)
 ; GLOBAL-NEXT:    xsmaddasp 4, 0, 3
 ; GLOBAL-NEXT:    xsmulsp 0, 2, 2
 ; GLOBAL-NEXT:    xsmaddasp 3, 4, 0
 ; GLOBAL-NEXT:    xsmulsp 0, 2, 3
 ; GLOBAL-NEXT:    xsmulsp 0, 0, 1
 ; GLOBAL-NEXT:  .LBB11_2:
 ; GLOBAL-NEXT:    fmr 1, 0
 ; GLOBAL-NEXT:    blr

llvm/test/CodeGen/PowerPC/fp128-bitcast-after-operation.ll

 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=PPC64-P8
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr7 < %s | FileCheck %s -check-prefix=PPC64
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=PPC64-P8
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 < %s | FileCheck %s -check-prefix=PPC64
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc-unknown-linux-gnu < %s | FileCheck %s -check-prefix=PPC32
 
 define i128 @test_abs(ppc_fp128 %x) nounwind  {
 entry:
 ; PPC64-LABEL: test_abs:
-; PPC64-DAG: stfdx 2, 0, [[ADDR_HI:[0-9]+]]
-; PPC64-DAG: stfdx 1, 0, [[ADDR_LO:[0-9]+]]
-; PPC64-DAG: addi [[ADDR_HI]], [[SP:[0-9]+]], [[OFFSET_HI:-?[0-9]+]]
-; PPC64-DAG: addi [[ADDR_LO]], [[SP]], [[OFFSET_LO:-?[0-9]+]]
+; PPC64-DAG: stfd 2, [[OFFSET_HI:-?[0-9]+]]([[SP:[0-9]+]])
+; PPC64-DAG: stfd 1, [[OFFSET_LO:-?[0-9]+]]([[SP]]) 
 ; PPC64-DAG: ld [[HI:[0-9]+]], [[OFFSET_LO]]([[SP]])
 ; PPC64-DAG: ld [[LO:[0-9]+]], [[OFFSET_HI]]([[SP]])
 ; PPC64-DAG: rldicr [[FLIP_BIT:[0-9]+]], [[HI]], 0, 0
 ; PPC64-DAG: xor 3, [[HI]], [[FLIP_BIT]]
 ; PPC64-DAG: xor 4, [[LO]], [[FLIP_BIT]]
 ; PPC64: blr
 
 ; PPC64-P8-LABEL: test_abs:
; PPC32: blr
   %0 = tail call ppc_fp128 @llvm.fabs.ppcf128(ppc_fp128 %x)
   %1 = bitcast ppc_fp128 %0 to i128
   ret i128 %1
 }
 
 define i128 @test_neg(ppc_fp128 %x) nounwind  {
 entry:
 ; PPC64-LABEL: test_neg:
-; PPC64-DAG: stfdx 2, 0, [[ADDR_HI:[0-9]+]]
-; PPC64-DAG: stfdx 1, 0, [[ADDR_LO:[0-9]+]]
-; PPC64-DAG: addi [[ADDR_HI]], [[SP:[0-9]+]], [[OFFSET_HI:-?[0-9]+]]
-; PPC64-DAG: addi [[ADDR_LO]], [[SP]], [[OFFSET_LO:-?[0-9]+]]
+; PPC64-DAG: stfd 2, [[OFFSET_HI:-?[0-9]+]]([[SP:[0-9]+]])
+; PPC64-DAG: stfd 1, [[OFFSET_LO:-?[0-9]+]]([[SP]]) 
 ; PPC64-DAG: li [[FLIP_BIT:[0-9]+]], 1
 ; PPC64-DAG: sldi [[FLIP_BIT]], [[FLIP_BIT]], 63
 ; PPC64-DAG: ld [[HI:[0-9]+]], [[OFFSET_LO]]([[SP]])
 ; PPC64-DAG: ld [[LO:[0-9]+]], [[OFFSET_HI]]([[SP]])
 ; PPC64-NOT: BARRIER
 ; PPC64-DAG: xor 3, [[HI]], [[FLIP_BIT]]
 ; PPC64-DAG: xor 4, [[LO]], [[FLIP_BIT]]
 ; PPC64: blr
; PPC32: blr
   %0 = fsub ppc_fp128 0xM80000000000000000000000000000000, %x
   %1 = bitcast ppc_fp128 %0 to i128
   ret i128 %1
 }
 
 define i128 @test_copysign(ppc_fp128 %x) nounwind  {
 entry:
 ; PPC64-LABEL: test_copysign:
-; PPC64-DAG: stfdx 1, 0, [[ADDR_REG:[0-9]+]]
-; PPC64-DAG: addi [[ADDR_REG]], 1, [[OFFSET:-?[0-9]+]]
+; PPC64-DAG: stfd 1, [[OFFSET:-?[0-9]+]](1)
 ; PPC64-DAG: li [[HI_TMP:[0-9]+]], 16399
 ; PPC64-DAG: li [[LO_TMP:[0-9]+]], 3019
 ; PPC64-NOT: BARRIER
 ; PPC64-DAG: sldi [[CST_HI:[0-9]+]], [[HI_TMP]], 48
 ; PPC64-DAG: sldi [[CST_LO:[0-9]+]], [[LO_TMP]], 52
 ; PPC64-DAG: ld [[X_HI:[0-9]+]], [[OFFSET]](1)
 ; PPC64-DAG: rldicr [[NEW_HI_TMP:[0-9]+]], [[X_HI]], 0, 0
 ; PPC64-DAG: or 3, [[NEW_HI_TMP]], [[CST_HI]]

llvm/test/CodeGen/PowerPC/i64-to-float.ll

 ; CHECK: @foo
 ; CHECK: std 3,
 ; CHECK: lfd [[REG:[0-9]+]],
 ; CHECK: fcfids 1, [[REG]]
 ; CHECK: blr
 
 ; CHECK-VSX: @foo
 ; CHECK-VSX: std 3,
-; CHECK-VSX: lfdx [[REG:[0-9]+]],
+; CHECK-VSX: lfd [[REG:[0-9]+]],
 ; CHECK-VSX: fcfids 1, [[REG]]
 ; CHECK-VSX: blr
 
 ; CHECK-P9: @foo
 ; CHECK-P9: std 3,
 ; CHECK-P9: lfd [[REG:[0-9]+]],
 ; CHECK-P9: xscvsxdsp 1, [[REG]]
 ; CHECK-P9: blr
 }
 
 define double @goo(i64 %a) nounwind {
 entry:
   %x = sitofp i64 %a to double
   ret double %x
 
 ; CHECK: @goo
 ; CHECK: std 3,
 ; CHECK: lfd [[REG:[0-9]+]],
 ; CHECK: fcfid 1, [[REG]]
 ; CHECK: blr
 
 ; CHECK-VSX: @goo
 ; CHECK-VSX: std 3,
-; CHECK-VSX: lfdx [[REG:[0-9]+]],
+; CHECK-VSX: lfd [[REG:[0-9]+]],
 ; CHECK-VSX: xscvsxddp 1, [[REG]]
 ; CHECK-VSX: blr
 
 ; CHECK-P9: @goo
 ; CHECK-P9: std 3,
 ; CHECK-P9: lfd [[REG:[0-9]+]],
 ; CHECK-P9: xscvsxddp 1, [[REG]]
 ; CHECK-P9: blr
 }
 
 define float @foou(i64 %a) nounwind {
 entry:
   %x = uitofp i64 %a to float
   ret float %x
 
 ; CHECK: @foou
 ; CHECK: std 3,
 ; CHECK: lfd [[REG:[0-9]+]],
 ; CHECK: fcfidus 1, [[REG]]
 ; CHECK: blr
 
 ; CHECK-VSX: @foou
 ; CHECK-VSX: std 3,
-; CHECK-VSX: lfdx [[REG:[0-9]+]],
+; CHECK-VSX: lfd [[REG:[0-9]+]],
 ; CHECK-VSX: fcfidus 1, [[REG]]
 ; CHECK-VSX: blr
 
 ; CHECK-P9: @foou
 ; CHECK-P9: std 3,
 ; CHECK-P9: lfd [[REG:[0-9]+]],
 ; CHECK-P9: xscvuxdsp 1, [[REG]]
 ; CHECK-P9: blr
 }
 
 define double @goou(i64 %a) nounwind {
 entry:
   %x = uitofp i64 %a to double
   ret double %x
 
 ; CHECK: @goou
 ; CHECK: std 3,
 ; CHECK: lfd [[REG:[0-9]+]],
 ; CHECK: fcfidu 1, [[REG]]
 ; CHECK: blr
 
 ; CHECK-VSX: @goou
 ; CHECK-VSX: std 3,
-; CHECK-VSX: lfdx [[REG:[0-9]+]],
+; CHECK-VSX: lfd [[REG:[0-9]+]],
 ; CHECK-VSX: xscvuxddp 1, [[REG]]
 ; CHECK-VSX: blr
 
 ; CHECK-P9: @goou
 ; CHECK-P9: std 3,
 ; CHECK-P9: lfd [[REG:[0-9]+]],
 ; CHECK-P9: xscvuxddp 1, [[REG]]
 ; CHECK-P9: blr
 }
 

llvm/test/CodeGen/PowerPC/lxv-aligned-stack-slots.ll

-; RUN: llc -O3 -o - %s | FileCheck %s
+; RUN: llc -O3 -ppc-late-peephole=false -o - %s | FileCheck %s
 target datalayout = "e-m:e-i64:64-n32:64"
 target triple = "powerpc64le-unknown-linux-gnu"
 
 %class1 = type { %union1 }
 %union1 = type { i64, [24 x i8] }
 %class2 = type { %class3 }
 %class3 = type { %class4 }
 %class4 = type { %class5, i64, %union.anon }

llvm/test/CodeGen/PowerPC/mcm-12.ll

 ; CHECK-LABEL: test_double_const:
 ; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR]]@toc@ha
 ; CHECK: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
 
 ; CHECK-VSX: [[VAR:[a-z0-9A-Z_.]+]]:
 ; CHECK-VSX: .quad 4562098671269285104
 ; CHECK-VSX-LABEL: test_double_const:
 ; CHECK-VSX: addis [[REG1:[0-9]+]], 2, [[VAR]]@toc@ha
-; CHECK-VSX: addi [[REG1]], {{[0-9]+}}, [[VAR]]@toc@l
-; CHECK-VSX: lfdx {{[0-9]+}}, 0, [[REG1]]
+; CHECK-VSX: lfd {{[0-9]+}}, [[VAR]]@toc@l({{[0-9]+}}) 
 
 ; CHECK-P9: [[VAR:[a-z0-9A-Z_.]+]]:
 ; CHECK-P9: .quad 4562098671269285104
 ; CHECK-P9-LABEL: test_double_const:
 ; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR]]@toc@ha
 ; CHECK-P9: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])

llvm/test/CodeGen/PowerPC/mcm-4.ll

 ; MEDIUM: addis [[REG1:[0-9]+]], 2, [[VAR]]@toc@ha
 ; MEDIUM: addi [[REG2:[0-9]+]], [[REG1]], [[VAR]]@toc@l
 ; MEDIUM: lfd {{[0-9]+}}, 0([[REG2]])
 
 ; MEDIUM-VSX: [[VAR:[a-z0-9A-Z_.]+]]:
 ; MEDIUM-VSX: .quad 4562098671269285104
 ; MEDIUM-VSX-LABEL: test_double_const:
 ; MEDIUM-VSX: addis [[REG1:[0-9]+]], 2, [[VAR]]@toc@ha
-; MEDIUM-VSX: addi [[REG2:[0-9]+]], [[REG1]], [[VAR]]@toc@l
-; MEDIUM-VSX: lfdx {{[0-9]+}}, 0, [[REG2]]
+; MEDIUM-VSX: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
 
 ; LARGE: [[VAR:[a-z0-9A-Z_.]+]]:
 ; LARGE: .quad 4562098671269285104
 ; LARGE-LABEL: test_double_const:
 ; LARGE: addis [[REG1:[0-9]+]], 2, [[VAR2:[a-z0-9A-Z_.]+]]@toc@ha
 ; LARGE: ld [[REG2:[0-9]+]], [[VAR2]]@toc@l([[REG1]])
 ; LARGE: lfd {{[0-9]+}}, 0([[REG2]])
 

llvm/test/CodeGen/PowerPC/ppc64-align-long-double.ll

 ; FIXMECHECK: lfd 2, 72(1)
 
 ; CHECK-VSX-DAG: std 6, 72(1)
 ; CHECK-VSX-DAG: std 5, 64(1)
 ; CHECK-VSX-DAG: std 4, 56(1)
 ; CHECK-VSX-DAG: std 3, 48(1)
 ; CHECK-VSX-DAG: std 5, -16(1)
 ; CHECK-VSX-DAG: std 6, -8(1)
-; CHECK-VSX-DAG: addi [[REG1:[0-9]+]], 1, -16
-; CHECK-VSX-DAG: addi 3, 1, -8
-; CHECK-VSX: lfdx 1, 0, [[REG1]]
-; CHECK-VSX: lfdx 2, 0, 3
+; CHECK-VSX: lfd 1, -16(1)
+; CHECK-VSX: lfd 2, -8(1)
 
 ; FIXME-VSX: addi 4, 1, 48
 ; FIXME-VSX: lxsdx 1, 4, 3
 ; FIXME-VSX: li 3, 24
 ; FIXME-VSX: lxsdx 2, 4, 3
 
 ; CHECK-P9-DAG: std 6, 72(1)
 ; CHECK-P9-DAG: std 5, 64(1)
 ; CHECK-P9-DAG: std 4, 56(1)
 ; CHECK-P9-DAG: std 3, 48(1)
 ; CHECK-P9-DAG: mtvsrd 1, 5
 ; CHECK-P9-DAG: mtvsrd 2, 6

llvm/test/CodeGen/PowerPC/ppc64le-smallarg.ll

 
 declare void @test1(%struct.small_arg* sret, %struct.large_arg* byval, %struct.small_arg* byval)
 
 define float @callee2(float %pad1, float %pad2, float %pad3, float %pad4, float %pad5, float %pad6, float %pad7, float %pad8, float %pad9, float %pad10, float %pad11, float %pad12, float %pad13, float %x) {
 entry:
   ret float %x
 }
 ; CHECK: @callee2
-; CHECK: addi [[TOCREG:[0-9]+]], 1, 136
-; CHECK: lfsx {{[0-9]+}}, {{[0-9]+}}, [[TOCREG]]
+; CHECK: lfs {{[0-9]+}}, 136(1) 
 ; CHECK: blr
 
 define void @caller2() {
 entry:
   %0 = load float, float* @gf, align 4
   %call = tail call float @test2(float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float 0.000000e+00, float %0)
   ret void
 }
 ; CHECK: @caller2
-; CHECK: addi [[TOCOFF:[0-9]+]], {{[0-9]+}}, 136
-; CHECK: stfsx {{[0-9]+}}, 0, [[TOCOFF]]
+; CHECK: stfs {{[0-9]+}}, 136({{[0-9]+}})
 ; CHECK: bl test2
 
 declare float @test2(float, float, float, float, float, float, float, float, float, float, float, float, float, float)
 

llvm/test/CodeGen/PowerPC/pr25157-peephole.ll

L.LB38_2452:
   %0 = load float, float* bitcast (i8* getelementptr inbounds (%struct.BSS38.51.4488.9911.14348.16813.20264.24701.28152.31603.35054.39491.44914.45407.46393.46886.47872.49351.49844.50830.51323.52309.53295.53788.54281.55267.55760.59211.61625, %struct.BSS38.51.4488.9911.14348.16813.20264.24701.28152.31603.35054.39491.44914.45407.46393.46886.47872.49351.49844.50830.51323.52309.53295.53788.54281.55267.55760.59211.61625* @.BSS38, i64 0, i32 0, i64 16) to float*), align 16
   %1 = fpext float %0 to double
   %2 = insertelement <2 x double> undef, double %1, i32 1
   store <2 x double> %2, <2 x double>* bitcast (i8* getelementptr inbounds (%struct_main1_2_.491.4928.10351.14788.17253.20704.25141.28592.32043.35494.39931.45354.45847.46833.47326.48312.49791.50284.51270.51763.52749.53735.54228.54721.55707.56200.59651.61626, %struct_main1_2_.491.4928.10351.14788.17253.20704.25141.28592.32043.35494.39931.45354.45847.46833.47326.48312.49791.50284.51270.51763.52749.53735.54228.54721.55707.56200.59651.61626* @_main1_2_, i64 0, i32 0, i64 32) to <2 x double>*), align 16
   unreachable
 }
 
 ; CHECK-LABEL: @aercalc_
-; CHECK: lfsx
+; CHECK: lfs
 ; CHECK: xxspltd
 ; CHECK: stxvd2x
 ; CHECK-NOT: xxswapd
 
 ; CHECK-P9-LABEL: @aercalc_
 ; CHECK-P9: lfs
 ; CHECK-P9: xxspltd
 ; CHECK-P9: stxv
 ; CHECK-P9-NOT: xxswapd

llvm/test/CodeGen/PowerPC/pr25157.ll

L.LB38_2452:
   %0 = load float, float* bitcast (i8* getelementptr inbounds (%struct.BSS38.51.4488.9911.14348.16813.20264.24701.28152.31603.35054.39491.44914.45407.46393.46886.47872.49351.49844.50830.51323.52309.53295.53788.54281.55267.55760.59211.61625, %struct.BSS38.51.4488.9911.14348.16813.20264.24701.28152.31603.35054.39491.44914.45407.46393.46886.47872.49351.49844.50830.51323.52309.53295.53788.54281.55267.55760.59211.61625* @.BSS38, i64 0, i32 0, i64 16) to float*), align 16
   %1 = fpext float %0 to double
   %2 = insertelement <2 x double> undef, double %1, i32 1
   store <2 x double> %2, <2 x double>* bitcast (i8* getelementptr inbounds (%struct_main1_2_.491.4928.10351.14788.17253.20704.25141.28592.32043.35494.39931.45354.45847.46833.47326.48312.49791.50284.51270.51763.52749.53735.54228.54721.55707.56200.59651.61626, %struct_main1_2_.491.4928.10351.14788.17253.20704.25141.28592.32043.35494.39931.45354.45847.46833.47326.48312.49791.50284.51270.51763.52749.53735.54228.54721.55707.56200.59651.61626* @_main1_2_, i64 0, i32 0, i64 32) to <2 x double>*), align 16
   unreachable
 }
 
 ; CHECK-LABEL: @aercalc_
-; CHECK: lfsx
+; CHECK: lfs
 ; CHECK-P9-LABEL: @aercalc_
 ; CHECK-P9: lfs

llvm/test/CodeGen/PowerPC/select_const.ll

 
 define double @sel_constants_frem_constant(i1 %cond) {
 ; ALL-LABEL: sel_constants_frem_constant:
 ; ALL:       # %bb.0:
 ; ALL-NEXT:    andi. 3, 3, 1
 ; ALL-NEXT:    bc 12, 1, .LBB48_2
 ; ALL-NEXT:  # %bb.1:
 ; ALL-NEXT:    addis 3, 2, .LCPI48_0@toc@ha
-; ALL-NEXT:    addi 3, 3, .LCPI48_0@toc@l
-; ALL-NEXT:    lfdx 1, 0, 3
+; ALL-NEXT:    lfd 1, .LCPI48_0@toc@l(3)
 ; ALL-NEXT:    blr
 ; ALL-NEXT:  .LBB48_2:
 ; ALL-NEXT:    addis 3, 2, .LCPI48_1@toc@ha
-; ALL-NEXT:    addi 3, 3, .LCPI48_1@toc@l
-; ALL-NEXT:    lfsx 1, 0, 3
+; ALL-NEXT:    lfs 1, .LCPI48_1@toc@l(3)
 ; ALL-NEXT:    blr
   %sel = select i1 %cond, double -4.0, double 23.3
   %bo = frem double %sel, 5.1
   ret double %bo
 }
 
 define double @frem_constant_sel_constants(i1 %cond) {
 ; ISEL-LABEL: frem_constant_sel_constants:

llvm/test/CodeGen/PowerPC/toc-float.ll

-; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 <%s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 <%s | FileCheck -check-prefix=CHECK-P9 %s
+; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 <%s | FileCheck -check-prefix=CHECK-P8 %s
 
 ; As the constant could be represented as float, a float is
 ; loaded from constant pool.
 define double @doubleConstant1() {
   ret double 1.400000e+01
-}
 
 ; CHECK-LABEL: doubleConstant1:
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
-; CHECK: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P9: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+}
 
 ; As the constant couldn't be represented as float, a double is
 ; loaded from constant pool.
 define double @doubleConstant2() {
   ret double 2.408904e+01
-}
 
 ; CHECK-LABEL: doubleConstant2:
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
-; CHECK: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P9: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: lfd {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+}
 
 @FArr = hidden local_unnamed_addr global [10 x float] zeroinitializer, align 4
 
 define float @floatConstantArray() local_unnamed_addr  {
   %1 = load float, float* getelementptr inbounds ([10 x float], [10 x float]* @FArr, i64 0, i64 3), align 4
   %2 = fadd float %1, 0x400B333340000000
   ret float %2
-}
 
 ; CHECK-LABEL: floatConstantArray 
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha+[[REG2:[0-9]+]]
-; CHECK: lfs {{[0-9]+}}, [[VAR]]@toc@l+[[REG2]]([[REG1]])
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha+[[REG2:[0-9]+]]
+; CHECK-P9: lfs {{[0-9]+}}, [[VAR]]@toc@l+[[REG2]]([[REG1]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: addi [[REG2:[0-9]+]], [[REG1]], [[VAR]]@toc@l
+; CHECK-P8: lfs {{[0-9]+}}, 12([[REG2]])
+}
 
 define float @floatConstant() {
   ret float 0x400470A3E0000000
-}
 
 ; CHECK-LABEL: floatConstant:
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
-; CHECK: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P9: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: lfs {{[0-9]+}}, [[VAR]]@toc@l([[REG1]])
+}
 
 ; llvm put the hidden globals into the TOC table.
 ; TODO - do some analysis and decide which globals could be put into TOC.
 @d = hidden local_unnamed_addr global [200 x double] zeroinitializer, align 8
 
 define double @doubleConstantArray()  {
   %1 = load double, double* getelementptr inbounds ([200 x double], [200 x double]* @d, i64 0, i64 3), align 8
   %2 = fadd double %1, 6.880000e+00
   ret double %2
-}
 
 ; CHECK-LABEL: doubleConstantArray
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha+[[REG2:[0-9]+]]
-; CHECK: lfd {{[0-9]+}}, [[VAR]]@toc@l+[[REG2]]([[REG1]])
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha+[[REG2:[0-9]+]]
+; CHECK-P9: lfd {{[0-9]+}}, [[VAR]]@toc@l+[[REG2]]([[REG1]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: addi [[REG2:[0-9]+]], [[REG1]], [[VAR]]@toc@l
+; CHECK-P8: lfd {{[0-9]+}}, 24([[REG2]])
+}
 
 @arr = hidden local_unnamed_addr global [20000 x double] zeroinitializer, align 8
 
 define double @doubleLargeConstantArray()  {
   %1 = load double, double* getelementptr inbounds ([20000 x double], [20000 x double]* @arr, i64 0, i64 4096), align 8
   %2 = fadd double %1, 6.880000e+00
   ret double %2
-}
 
-; access element that out of range
+; Access an element with an offset that doesn't fit in the displacement field of LFD. 
 ; CHECK-LABEL: doubleLargeConstantArray
-; CHECK: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
-; CHECK: li [[REG2:[0-9]+]], 0 
-; CHECK: addi [[REG3:[0-9]+]], [[REG1]], [[VAR:[a-z0-9A-Z_.]+]]@toc@l
-; CHECK: ori [[REG4:[0-9]+]], [[REG2]], 32768 
-; CHECK: lfdx {{[0-9]+}}, [[REG3]], [[REG4]] 
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P9: li [[REG2:[0-9]+]], 0 
+; CHECK-P9: addi [[REG3:[0-9]+]], [[REG1]], [[VAR:[a-z0-9A-Z_.]+]]@toc@l
+; CHECK-P9: ori [[REG4:[0-9]+]], [[REG2]], 32768 
+; CHECK-P9: lfdx {{[0-9]+}}, [[REG3]], [[REG4]] 
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: li [[REG2:[0-9]+]], 0 
+; CHECK-P8: addi [[REG3:[0-9]+]], [[REG1]], [[VAR:[a-z0-9A-Z_.]+]]@toc@l
+; CHECK-P8: ori [[REG4:[0-9]+]], [[REG2]], 32768 
+; CHECK-P8: lfdx {{[0-9]+}}, [[REG3]], [[REG4]] 
+}
+
+@vec_arr = global [10 x <4 x i32>] zeroinitializer, align 16
+
+define <4 x i32> @vectorArray() #0 {
+entry:
+  %0 = load <4 x i32>, <4 x i32>* getelementptr inbounds ([10 x <4 x i32>], [10 x <4 x i32>]* @vec_arr, i64 0, i64 2), align 16
+  ret <4 x i32> %0
+
+; CHECK-LABEL: vectorArray
+; CHECK-P9: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P9: ld [[REG2:[0-9]+]], [[VAR]]@toc@l([[REG1]])
+; CHECK-P9: lxv {{[0-9]+}}, 32([[REG2]])
+; CHECK-P8: addis [[REG1:[0-9]+]], 2, [[VAR:[a-z0-9A-Z_.]+]]@toc@ha
+; CHECK-P8: ld [[REG2:[0-9]+]], [[VAR]]@toc@l([[REG1]])
+; CHECK-P8: addi [[REG3:[0-9]+]], [[REG2]], 32
+; CHECK-P8: lvx {{[0-9]+}}, 0, [[REG3]]
+}

llvm/test/CodeGen/PowerPC/vsx_scalar_ld_st.ll

 entry:
   %ff = alloca float, align 4
   %0 = load double, double* @d, align 8
   %conv = fptrunc double %0 to float
   store volatile float %conv, float* %ff, align 4
   ret void
 ; CHECK-LABEL: @dblToFloat
 ; CHECK: lfdx [[REGLD5:[0-9]+]],
-; CHECK: stfsx [[REGLD5]],
+; CHECK: stfs [[REGLD5]],
 ; CHECK-P9-LABEL: @dblToFloat
 ; CHECK-P9: lfd [[REGLD5:[0-9]+]],
 ; CHECK-P9: stfs [[REGLD5]],
 }
 
 ; Function Attrs: nounwind
 define void @floatToDbl() #0 {
 entry:
   %dd = alloca double, align 8
   %0 = load float, float* @f, align 4
   %conv = fpext float %0 to double
   store volatile double %conv, double* %dd, align 8
   ret void
 ; CHECK-LABEL: @floatToDbl
 ; CHECK: lfsx [[REGLD5:[0-9]+]],
-; CHECK: stfdx [[REGLD5]],
+; CHECK: stfd [[REGLD5]],
 ; CHECK-P9-LABEL: @floatToDbl
 ; CHECK-P9: lfs [[REGLD5:[0-9]+]],
 ; CHECK-P9: stfd [[REGLD5]],
 }