[PowerPC] Optimize compare by using record form in post-RA.

[Esme]

Summary

We currently optimize the comparison only in SSA, therefore we will miss some optimization opportunities where the input of comparison is lowered from COPY in post-RA.
Ie. ExpandPostRA::LowerCopy is called after PPCInstrInfo::optimizeCompareInstr.

This patch optimizes the comparison in post-RA and only the cases that compare against zero can be handled.
D131374 converts the comparison and its user to a compare against zero with the appropriate predicate on the branch, which creates additional opportunities for this patch.

[shchenz]

This patch misses MIR cases for this transformation. See example in llvm/test/CodeGen/PowerPC/fold-frame-offset-using-rr.mir

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2778

Better to add comments to explicitly specify the condition for optimizing, eg:
1: this should be a compare instruction between a reg and an imm
2: the imm should be 0

2779

nit: isVirtual() is not needed since this is for post RA.

2782

Add a MIR case for this?

2788

getDefMIPostRA() would not return a def in another block, so is SrcMI->getParent() != CmpMI.getParent() needed?

SrcReg = op1
xx1 = op2 SrcReg
xx2= cmp SrcReg

Intermediate use of SrcReg should not stop the compare optimization?

2801

If there is any use of cr0 between SrcMI and CmpMI, we should also bail out? We will define the cr0 in the new record form or SrcMI?

2805

I can not image a case where OldOpc == NewOpC, could you please give an example here?

2808

I don't understand the logic here, and what result will MRI->use_instructions() produce after RA?

2823

RegMO is the def of CmpMI, it will not have kill flag. And CR0 will be be killed between SrcMI and CmpMI, otherwise we can not do the transformation. No use of CR0 should exist between SrcMI and CmpMI.

Here I think we should handle the killed flag for SrcReg if it is killed in CmpMI and there is other use between SrcMI and CmpMI?

[Esme]

Addressed Zheng's comments for the code part and the MIR case will added soon.

[Esme]

Added mir test cases.

[lkail]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2792

nit: OtherUse_SrcReg -> SrcRegHasOtherUse

2804

This check can be moved after

if (CmpMI.hasImplicitDef())
  return false;

2808

nit: OtherUse_CRReg -> SeenUseOfCRReg.

2823

Here I think we should handle the killed flag for SrcReg if it is killed in CmpMI and there is other use between SrcMI and CmpMI?

Look no need. We are folding

L1: a = op1 b, c
...
L2: b = op2 a, 0

->

L1: a, b = op3 b, c
...

The constraint here is b can't be def/use between [L1,L2). Use of a between [L1, L2] is still the version defined at L1. And call of getDefMIPostRA should guarantee L1 is the nearest point defined a.

[lkail]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2823

edit:

L1: a = op1 b, c
...
L2: d = op2 a, 0

->

L1: a, d = op3 b, c
...

The constraint here is d can't be def/use between [L1,L2).

[shchenz]

Look almost good to me now. Thanks for adding this folding.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2797

Have you seen any perf impact due to this pattern? If so, there are many other record form rotate and shift instructions on PowerPC target, like RLDICL/RLDICR(not complete.)

2828

Can we add a MIR case for this?

[Esme]

Apologize for the late update due to some compiler-rt errors occurred. I'm working on fixing it.

[Esme]

The bootstrap test is clean.

[lkail]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2791

Is

if (Opc != PPC::CMPWI && Opc != PPC::CMPDI)
  return false;

// <Explain why CMPWI doesn't apply on PPC64>
if (Subtarget.isPPC64() && Opc == PPC::CMPWI)
  return false;

more clear?

[Esme]

Update comments.

[lkail]

LGTM thanks.

[shchenz]

Compared with previous versions, to me, seems some valid transformations are gone now because of the new bail out case Subtarget.isPPC64() && Opc == PPC::CMPWI. Maybe we can have a further check why PPC64 generates CMPWI 0 for some cases. For these cases, I guess CMPDI 0 should also be ok, so that there will be more record form opportunities here. But we can do that in another patch.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2779

We don't need the check for isVirtual()? we are post-ra here.

2791

Maybe we don't need Subtarget.isPPC64() for PPC::CMPLDI because this should be impossible.

2839

hmm, another caller of fixupIsDeadOrKill(), we refactor this in D133103. FYI @nemanjai

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2791

I meant isPPC32 && PPC::CMPLDI is impossible, so that we don't need check Subtarget.isPPC64().

[Esme]

In D131873#3881526, @shchenz wrote:

Compared with previous versions, to me, seems some valid transformations are gone now because of the new bail out case Subtarget.isPPC64() && Opc == PPC::CMPWI. Maybe we can have a further check why PPC64 generates CMPWI 0 for some cases. For these cases, I guess CMPDI 0 should also be ok, so that there will be more record form opportunities here. But we can do that in another patch.

Currently PPC64 generates CMPWI 0 for int32 operands, which is reasonable.
CMPDI 0 will be ok if the use of the comparison is an equality check, however it's not easy to find the reliable use in Post-RA.
More investigation is required to exploit other opportunities for optimization comparison.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

2839

D133103 seems in progress, so this patch will not be rebase on it. Maybe D133103 can refactor this after this patch lands.

[shchenz]

More investigation is required to exploit other opportunities for optimization comparison.

Yes, I agree, this is definitely worth further investigation.

This patch LGTM too. Thanks for adding this.

llvm/lib/Target/PowerPC/PPCInstrInfo.h

class PPCInstrInfo : public PPCGenInstrInfo {
   bool isImmElgibleForForwarding(const MachineOperand &ImmMO,
                                  const MachineInstr &DefMI,
                                  const ImmInstrInfo &III,
                                  int64_t &Imm,
                                  int64_t BaseImm = 0) const;
   bool isRegElgibleForForwarding(const MachineOperand &RegMO,
                                  const MachineInstr &DefMI,
                                  const MachineInstr &MI, bool KillDefMI,
-                                 bool &IsFwdFeederRegKilled) const;
+                                 bool &IsFwdFeederRegKilled,
+                                 bool &SeenIntermediateUse) const;
   unsigned getSpillTarget() const;
   const unsigned *getStoreOpcodesForSpillArray() const;
   const unsigned *getLoadOpcodesForSpillArray() const;
   unsigned getSpillIndex(const TargetRegisterClass *RC) const;
   int16_t getFMAOpIdxInfo(unsigned Opcode) const;
   void reassociateFMA(MachineInstr &Root, MachineCombinerPattern Pattern,
                       SmallVectorImpl<MachineInstr *> &InsInstrs,
                       SmallVectorImpl<MachineInstr *> &DelInstrs,
#endif
   /// Return true if get the base operand, byte offset of an instruction and
   /// the memory width. Width is the size of memory that is being
   /// loaded/stored (e.g. 1, 2, 4, 8).
   bool getMemOperandWithOffsetWidth(const MachineInstr &LdSt,
                                     const MachineOperand *&BaseOp,
                                     int64_t &Offset, unsigned &Width,
                                     const TargetRegisterInfo *TRI) const;
 
+  bool optimizeCmpPostRA(MachineInstr &MI) const;
+
   /// Get the base operand and byte offset of an instruction that reads/writes
   /// memory.
   bool getMemOperandsWithOffsetWidth(
       const MachineInstr &LdSt,
       SmallVectorImpl<const MachineOperand *> &BaseOps, int64_t &Offset,
       bool &OffsetIsScalable, unsigned &Width,
       const TargetRegisterInfo *TRI) const override;
 

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

for (unsigned i = 0, e = PredsToUpdate.size(); i < e; i++)
     PredsToUpdate[i].first->setImm(PredsToUpdate[i].second);
 
   for (unsigned i = 0, e = SubRegsToUpdate.size(); i < e; i++)
     SubRegsToUpdate[i].first->setSubReg(SubRegsToUpdate[i].second);
 
   return true;
 }
 
+bool PPCInstrInfo::optimizeCmpPostRA(MachineInstr &CmpMI) const {
+  MachineRegisterInfo *MRI = &CmpMI.getParent()->getParent()->getRegInfo();
+  if (MRI->isSSA())
+    return false;
+
+  Register SrcReg, SrcReg2;
+  int64_t CmpMask, CmpValue;
+  if (!analyzeCompare(CmpMI, SrcReg, SrcReg2, CmpMask, CmpValue))
+    return false;
+
+  // Try to optimize the comparison against 0.
+  if (CmpValue || !CmpMask || SrcReg2)
+    return false;
+
+  // The record forms set the condition register based on a signed comparison
+  // with zero (see comments in optimizeCompareInstr). Since we can't do the
+  // equality checks in post-RA, we are more restricted on a unsigned
+  // comparison.
+  unsigned Opc = CmpMI.getOpcode();
+  if (Opc == PPC::CMPLWI || Opc == PPC::CMPLDI)
+    return false;
+
+  // The record forms are always based on a 64-bit comparison on PPC64
+  // (similary, a 32-bit comparison on PPC32), while the CMPWI is a 32-bit
+  // comparison. Since we can't do the equality checks in post-RA, we bail out
+  // the case.
+  if (Subtarget.isPPC64() && Opc == PPC::CMPWI)
+    return false;
+
+  // CmpMI can't be deleted if it has implicit def.
+  if (CmpMI.hasImplicitDef())
+    return false;
+
+  bool SrcRegHasOtherUse = false;
+  MachineInstr *SrcMI = getDefMIPostRA(SrcReg, CmpMI, SrcRegHasOtherUse);
+  if (!SrcMI || !SrcMI->definesRegister(SrcReg))
+    return false;
+
+  MachineOperand RegMO = CmpMI.getOperand(0);
+  Register CRReg = RegMO.getReg();
+  if (CRReg != PPC::CR0)
+    return false;
+
+  // Make sure there is no def/use of CRReg between SrcMI and CmpMI.
+  bool SeenUseOfCRReg = false;
+  bool IsCRRegKilled = false;
+  if (!isRegElgibleForForwarding(RegMO, *SrcMI, CmpMI, false, IsCRRegKilled,
+                                 SeenUseOfCRReg) ||
+      SrcMI->definesRegister(CRReg) || SeenUseOfCRReg)
+    return false;
+
+  int SrcMIOpc = SrcMI->getOpcode();
+  int NewOpC = PPC::getRecordFormOpcode(SrcMIOpc);
+  if (NewOpC == -1)
+    return false;
+
+  LLVM_DEBUG(dbgs() << "Replace Instr: ");
+  LLVM_DEBUG(SrcMI->dump());
+
+  const MCInstrDesc &NewDesc = get(NewOpC);
+  SrcMI->setDesc(NewDesc);
+  MachineInstrBuilder(*SrcMI->getParent()->getParent(), SrcMI)
+      .addReg(CRReg, RegState::ImplicitDefine);
+  SrcMI->clearRegisterDeads(CRReg);
+
+  // Fix up killed/dead flag for SrcReg after transformation.
+  if (SrcRegHasOtherUse || CmpMI.getOperand(1).isKill())
+    fixupIsDeadOrKill(SrcMI, &CmpMI, SrcReg);
+
+  assert(SrcMI->definesRegister(PPC::CR0) &&
+         "Record-form instruction does not define cr0?");
+
+  LLVM_DEBUG(dbgs() << "with: ");
+  LLVM_DEBUG(SrcMI->dump());
+  LLVM_DEBUG(dbgs() << "Delete dead instruction: ");
+  LLVM_DEBUG(CmpMI.dump());
+  return true;
+}
+
 bool PPCInstrInfo::getMemOperandsWithOffsetWidth(
     const MachineInstr &LdSt, SmallVectorImpl<const MachineOperand *> &BaseOps,
     int64_t &Offset, bool &OffsetIsScalable, unsigned &Width,
     const TargetRegisterInfo *TRI) const {
   const MachineOperand *BaseOp;
   OffsetIsScalable = false;
   if (!getMemOperandWithOffsetWidth(LdSt, BaseOp, Offset, Width, TRI))
     return false;
bool PPCInstrInfo::isDefMIElgibleForForwarding(MachineInstr &DefMI,
   // 1. add inst
   // 2. one of the operands is Imm/CPI/Global.
   return isAnImmediateOperand(*ImmMO);
 }
 
 bool PPCInstrInfo::isRegElgibleForForwarding(
     const MachineOperand &RegMO, const MachineInstr &DefMI,
     const MachineInstr &MI, bool KillDefMI,
-    bool &IsFwdFeederRegKilled) const {
+    bool &IsFwdFeederRegKilled, bool &SeenIntermediateUse) const {
   // x = addi y, imm
   // ...
   // z = lfdx 0, x   -> z = lfd imm(y)
   // The Reg "y" can be forwarded to the MI(z) only when there is no DEF
   // of "y" between the DEF of "x" and "z".
   // The query is only valid post RA.
   const MachineRegisterInfo &MRI = MI.getParent()->getParent()->getRegInfo();
   if (MRI.isSSA())
     return false;
 
   Register Reg = RegMO.getReg();
 
   // Walking the inst in reverse(MI-->DefMI) to get the last DEF of the Reg.
   MachineBasicBlock::const_reverse_iterator It = MI;
   MachineBasicBlock::const_reverse_iterator E = MI.getParent()->rend();
   It++;
   for (; It != E; ++It) {
     if (It->modifiesRegister(Reg, &getRegisterInfo()) && (&*It) != &DefMI)
       return false;
     else if (It->killsRegister(Reg, &getRegisterInfo()) && (&*It) != &DefMI)
       IsFwdFeederRegKilled = true;
+    if (It->readsRegister(Reg, &getRegisterInfo()) && (&*It) != &DefMI)
+      SeenIntermediateUse = true;
     // Made it to DefMI without encountering a clobber.
     if ((&*It) == &DefMI)
       break;
   }
   assert((&*It) == &DefMI && "DefMI is missing");
 
   // If DefMI also defines the register to be forwarded, we can only forward it
   // if DefMI is being erased.
bool PPCInstrInfo::transformToImmFormFedByAdd(
 
   // As we get the Imm operand now, we need to check if the ImmMO meet
   // the requirement described in the III. If yes set the Imm.
   int64_t Imm = 0;
   if (!isImmElgibleForForwarding(*ImmMO, DefMI, III, Imm))
     return false;
 
   bool IsFwdFeederRegKilled = false;
+  bool SeenIntermediateUse = false;
   // Check if the RegMO can be forwarded to MI.
   if (!isRegElgibleForForwarding(*RegMO, DefMI, MI, KillDefMI,
-                                 IsFwdFeederRegKilled))
+                                 IsFwdFeederRegKilled, SeenIntermediateUse))
     return false;
 
   // Get killed info in case fixup needed after transformation.
   unsigned ForwardKilledOperandReg = ~0U;
   MachineRegisterInfo &MRI = MI.getParent()->getParent()->getRegInfo();
   bool PostRA = !MRI.isSSA();
   if (PostRA && MI.getOperand(OpNoForForwarding).isKill())
     ForwardKilledOperandReg = MI.getOperand(OpNoForForwarding).getReg();

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

 STATISTIC(NumRRConvertedInPreEmit,
           "Number of r+r instructions converted to r+i in pre-emit peephole");
 STATISTIC(NumRemovedInPreEmit,
           "Number of instructions deleted in pre-emit peephole");
 STATISTIC(NumberOfSelfCopies,
           "Number of self copy instructions eliminated");
 STATISTIC(NumFrameOffFoldInPreEmit,
           "Number of folding frame offset by using r+r in pre-emit peephole");
+STATISTIC(NumCmpsInPreEmit,
+          "Number of compares eliminated in pre-emit peephole");
 
 static cl::opt<bool>
 EnablePCRelLinkerOpt("ppc-pcrel-linker-opt", cl::Hidden, cl::init(true),
                      cl::desc("enable PC Relative linker optimization"));
 
 static cl::opt<bool>
 RunPreEmitPeephole("ppc-late-peephole", cl::Hidden, cl::init(true),
                    cl::desc("Run pre-emit peephole optimizations."));
bool runOnMachineFunction(MachineFunction &MF) override {
             }
           }
           if (TII->foldFrameOffset(MI)) {
             Changed = true;
             NumFrameOffFoldInPreEmit++;
             LLVM_DEBUG(dbgs() << "Frame offset folding by using index form: ");
             LLVM_DEBUG(MI.dump());
           }
+          if (TII->optimizeCmpPostRA(MI)) {
+            Changed = true;
+            NumCmpsInPreEmit++;
+            LLVM_DEBUG(dbgs() << "Optimize compare by using record form: ");
+            LLVM_DEBUG(MI.dump());
+            InstrsToErase.push_back(&MI);
+          }
         }
 
         // Eliminate conditional branch based on a constant CR bit by
         // CRSET or CRUNSET. We eliminate the conditional branch or
         // convert it into an unconditional branch. Also, if the CR bit
         // is not used by other instructions, we eliminate CRSET as well.
         auto I = MBB.getFirstInstrTerminator();
         if (I == MBB.instr_end())

llvm/test/CodeGen/PowerPC/opt-cmp-rec-postra.mir

+# RUN: llc -mtriple=powerpc64le-linux-gnu -stop-after ppc-pre-emit-peephole %s -o - -verify-machineinstrs | FileCheck %s
+
+---
+name: test1
+# The cmp instr is optimized with the record form.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4
+    renamable $x3 = OR8 killed renamable $x3, killed renamable $x4
+    renamable $cr0 = CMPDI renamable $x3, 0, implicit killed $x3
+    ; CHECK-LABEL: name: test1
+    ; CHECK: renamable $x3 = OR8_rec renamable $x3, killed renamable $x4, implicit-def $cr0
+    ; CHECK-NOT: CMPDI
+    BCC 68, killed renamable $cr0, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...
+
+---
+name: test2
+# The imm of the comparison instr isn't 0.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4
+    renamable $x3 = OR8 killed renamable $x3, killed renamable $x4
+    renamable $cr0 = CMPDI renamable $x3, 2, implicit killed $x3
+    ; CHECK-LABEL: name: test2
+    ; CHECK: CMPDI
+    BCC 68, killed renamable $cr0, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...
+
+---
+name: test3
+# The comparison instr has a implicit def.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4
+    renamable $x3 = OR8 killed renamable $x3, killed renamable $x4
+    renamable $cr0 = CMPDI renamable $x3, 0, implicit-def $x3
+    ; CHECK-LABEL: name: test3
+    ; CHECK: CMPDI
+    BCC 68, killed renamable $cr0, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...
+
+---
+name: test4
+# There is another use for cr0 between OR8 instr and CMPWI instr.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4, $cr0
+    renamable $x3 = OR8 killed renamable $x3, killed renamable $x4
+    renamable $cr1 = MCRF killed $cr0, implicit $x3
+    renamable $cr0 = CMPDI renamable $x3, 0, implicit killed $x3, implicit $cr1
+    ; CHECK-LABEL: name: test4
+    ; CHECK: CMPDI
+    BCC 68, killed renamable $cr0, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...
+
+---
+name: test5
+# There is another def for cr0 between OR8 instr and CMPWI instr.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4
+    renamable $x3 = OR8 killed renamable $x3, renamable $x4
+    renamable $cr1 = CMPD renamable $x3, renamable $x4, implicit-def $cr0
+    renamable $cr0 = CMPDI renamable $x3, 0, implicit killed $x3, implicit $cr1
+    ; CHECK-LABEL: name: test5
+    ; CHECK: CMPDI
+    BCC 68, killed renamable $cr0, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...
+
+---
+name: test6
+# The SrcReg isn't CR0.
+tracksRegLiveness: true
+body:             |
+  bb.0.entry:
+    successors: %bb.1(0x30000000), %bb.2(0x50000000)
+    liveins: $x3, $x4
+    renamable $x3 = OR8 killed renamable $x3, killed renamable $x4
+    renamable $cr1 = CMPDI renamable $x3, 0, implicit killed $x3
+    ; CHECK-LABEL: name: test6
+    ; CHECK: CMPDI
+    BCC 68, killed renamable $cr1, %bb.2
+
+  bb.1:
+    $x3 = LI8 102
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+
+  bb.2:
+    $x3 = LI8 116
+    BLR8 implicit $lr8, implicit $rm, implicit $x3
+...

llvm/test/CodeGen/PowerPC/ppc64-rop-protection.ll

 ; LE-P8-NEXT:    blr
 ;
 ; LE-P10-O0-LABEL: shrinkwrap:
 ; LE-P10-O0:       # %bb.0: # %entry
 ; LE-P10-O0-NEXT:    mflr r0
 ; LE-P10-O0-NEXT:    std r0, 16(r1)
 ; LE-P10-O0-NEXT:    hashst r0, -8(r1)
 ; LE-P10-O0-NEXT:    stdu r1, -64(r1)
-; LE-P10-O0-NEXT:    mr r4, r3
+; LE-P10-O0-NEXT:    mr. r4, r3
 ; LE-P10-O0-NEXT:    std r4, 40(r1) # 8-byte Folded Spill
 ; LE-P10-O0-NEXT:    li r3, 0
-; LE-P10-O0-NEXT:    cmpdi r4, 0
 ; LE-P10-O0-NEXT:    stw r3, 48(r1) # 4-byte Folded Spill
 ; LE-P10-O0-NEXT:    beq cr0, .LBB2_2
 ; LE-P10-O0-NEXT:  # %bb.1: # %if.end
 ; LE-P10-O0-NEXT:    ld r3, 40(r1) # 8-byte Folded Reload
 ; LE-P10-O0-NEXT:    lwz r3, 12(r3)
 ; LE-P10-O0-NEXT:    stw r3, 52(r1)
 ; LE-P10-O0-NEXT:    addi r3, r1, 52
 ; LE-P10-O0-NEXT:    bl callee2@notoc

 ; LE-P10-O0-NEXT:    blr
 ;
 ; LE-P9-O0-LABEL: shrinkwrap:
 ; LE-P9-O0:       # %bb.0: # %entry
 ; LE-P9-O0-NEXT:    mflr r0
 ; LE-P9-O0-NEXT:    std r0, 16(r1)
 ; LE-P9-O0-NEXT:    hashst r0, -8(r1)
 ; LE-P9-O0-NEXT:    stdu r1, -128(r1)
-; LE-P9-O0-NEXT:    mr r4, r3
+; LE-P9-O0-NEXT:    mr. r4, r3
 ; LE-P9-O0-NEXT:    std r4, 104(r1) # 8-byte Folded Spill
 ; LE-P9-O0-NEXT:    li r3, 0
-; LE-P9-O0-NEXT:    cmpdi r4, 0
 ; LE-P9-O0-NEXT:    stw r3, 112(r1) # 4-byte Folded Spill
 ; LE-P9-O0-NEXT:    beq cr0, .LBB2_2
 ; LE-P9-O0-NEXT:  # %bb.1: # %if.end
 ; LE-P9-O0-NEXT:    ld r3, 104(r1) # 8-byte Folded Reload
 ; LE-P9-O0-NEXT:    lwz r3, 12(r3)
 ; LE-P9-O0-NEXT:    stw r3, 116(r1)
 ; LE-P9-O0-NEXT:    addi r3, r1, 116
 ; LE-P9-O0-NEXT:    bl callee2

 ; LE-P9-O0-NEXT:    blr
 ;
 ; LE-P8-O0-LABEL: shrinkwrap:
 ; LE-P8-O0:       # %bb.0: # %entry
 ; LE-P8-O0-NEXT:    mflr r0
 ; LE-P8-O0-NEXT:    std r0, 16(r1)
 ; LE-P8-O0-NEXT:    hashst r0, -8(r1)
 ; LE-P8-O0-NEXT:    stdu r1, -128(r1)
-; LE-P8-O0-NEXT:    mr r4, r3
+; LE-P8-O0-NEXT:    mr. r4, r3
 ; LE-P8-O0-NEXT:    std r4, 104(r1) # 8-byte Folded Spill
 ; LE-P8-O0-NEXT:    li r3, 0
-; LE-P8-O0-NEXT:    cmpdi r4, 0
 ; LE-P8-O0-NEXT:    stw r3, 112(r1) # 4-byte Folded Spill
 ; LE-P8-O0-NEXT:    beq cr0, .LBB2_2
 ; LE-P8-O0-NEXT:  # %bb.1: # %if.end
 ; LE-P8-O0-NEXT:    ld r3, 104(r1) # 8-byte Folded Reload
 ; LE-P8-O0-NEXT:    lwz r3, 12(r3)
 ; LE-P8-O0-NEXT:    stw r3, 116(r1)
 ; LE-P8-O0-NEXT:    addi r3, r1, 116
 ; LE-P8-O0-NEXT:    bl callee2