[PowerPC] Extend folding RLWINM + RLWINM to post-RA.

[Esme]

Summary

This patch depends on D89846.
We have the patterns to fold 2 RLWINMs in ppc-mi-peephole, while some RLWINM will be generated after RA, for example rGc4690b007743. If the RLWINM generated after RA followed by another RLWINM, we expect to perform the optimization after RA, too.

[shchenz]

It would be great if we could split this patch into two parts:
1: extend rlwinm + rlwinm folding to post-ra and add some post-ra testcases, MIR cases are better.
2: add new simplication for rlwinm + andi_rec and add some post-ra and pre-ra testcases.

[steven.zhang]

Some nits.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3893–3918

Comments need to be updated.

3897

How about use the MachineInstr *&ToErase to avoid the null check and default arguments ?

3921

It is preferred to initialize the stack variable.

3922

Remove such kind of comments as the code is self explained.

3927

Common line 3209 and 3217 then, add a check in 3220

3939

A better code style to avoid the duplicate check of the SrcOpCode.

switch (SrcOpCode) {
case RLWINM8
case RLWINM8_rec:
    Is64Bit = true;
    break;
case ...
}

4080

null check?

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

41

Change the name as it is extended to all rotate combining.

519

No {} if there is only one statement.

[Esme]

Updated as Steven's comments.

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3893–3918

The strange character before "We"

3898

Can we initial Is64Bit to true or false and then we only need to handle 32-bit or 64-bit opcodes in following switch?

3931

Any special reason that we don't do the folding after RA when there are multiple uses for SrcMI? Before RA, we do such kind of transformation as it is still benefit. We eliminate the dependancy between SrcMI and MI.

3938

What case will have rlwinm input is rlwinm8 or rlwinm8 input is rlwinm? Just curious.

4079

Same as above, after RA, even SrcMI has multiple uses, we still should do the transform?

[Esme]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3938

There is a case in llvm/test/CodeGen/PowerPC/popcnt-zext.ll

declare i16 @llvm.ctpop.i16(i16) nounwind readnone
define i64 @popa_i16_i64(i16 %x) {
  %pop = call i16 @llvm.ctpop.i16(i16 %x)
  %z = zext i16 %pop to i64 ; SimplifyDemandedBits may turn zext (or sext) into aext
  %a = and i64 %z, 16
  ret i64 %a
}

llc -verify-machineinstrs -mtriple=powerpc64-- -mattr=+slow-popcntd < cnt16.ll -debug
We will see such transformation:
From

368B	  %22:gprc = RLWINM %21:gprc, 8, 24, 31
384B	  undef %23.sub_32:g8rc = COPY %22:gprc
400B	  %25:g8rc = RLWINM8 %23:g8rc, 0, 27, 27
416B	  $x3 = COPY %25:g8rc

To

undef %23.sub_32:g8rc = RLWINM %21:gprc, 8, 24, 31
%25:g8rc = RLWINM8 %23:g8rc, 0, 27, 27
$x3 = COPY %25:g8rc

After RA

renamable $r3 = RLWINM killed renamable $r3, 8, 24, 31, implicit-def $x3
renamable $x3 = RLWINM8 killed renamable $x3, 0, 27, 27

We will treat them as DefMI and MI after RA, but they shouldn't be folded. Please correct me if I'm wrong.

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3928

This is redundant or can be changed to an assert?

3938

Thanks, I see, this is just a post-ra issue. Before RA, there are always register copy instructions between RLWINM and RLWINM8.

After Post-RA,
1: for RLWINM8(RLWINM), it should be ok to fold it to single RLWINM8,
2: for RLWINM(RLWINM8), it should be ok to fold it to a single RLWINM.

[Esme]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3938

Thanks! I will mark this as a follow-up work after this patch submitted.

[steven.zhang]

LGTM overall with one minor comment.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

4078

Can we set the NoUse for pre-ra also，so that just check the NoUse here？

[shchenz]

LGTM too. Please remember to rebase this patch after you commit NFC patch D89846

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3938

sounds good to me.

[Esme]

Thanks for your reviews, and I apologize for not responding in time. I will commit this patch after all tests done.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

4078

For the scenario of pre-ra, we have to check MRI->use_nodbg_empty(FoldingReg) here instead of setting NoUse before folding as post-ra did, since MRI->use_nodbg_empty(FoldingReg) is false before folding MI and SrcMI.

[Esme]

In post-RA, if SrcMI also defines the register to be forwarded, we can only do the folding if SrcMI is going to be erased.
For example, SrcMI and MI can't not be folded in the scenario:

$r3 = RLWINM_rec $r3, 27, 5, 10, implicit-def $cr0
dead renamable $r3 = RLWINM_rec killed renamable $r3, 8, 5, 10, implicit-def $cr0
BLR8 implicit $lr8, implicit $rm, implicit killed $cr0

I've run the bootstrap-test as well as SEPC.

[shchenz]

I saw some testcases are removed compared with your committed version. Is there any reason?

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3983

Can these lines be moved after line 3221? So we will bail out early.

4079

Can we move CanErase assignment for SSA to the place where we assign it for post SSA?

[Esme]

In D89855#2398960, @shchenz wrote:

I saw some testcases are removed compared with your committed version. Is there any reason?

Every specific folding patterns with different masks have been tested in llvm/test/CodeGen/PowerPC/fold-rlwinm.mir
This patch didn't change the patterns, so I supposed it would be clearer to remove some redundant test cases.
Do you think they should be added back？

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3983

It's safe to call SrcMI->getOperand(1).getReg() after filtering the opcode of SrcMI.

4079

The condition MRI->use_nodbg_empty(FoldingReg) has to be checked after the folding of MI and SrcMI.

[shchenz]

In D89855#2399063, @Esme wrote:

In D89855#2398960, @shchenz wrote:

I saw some testcases are removed compared with your committed version. Is there any reason?

Every specific folding patterns with different masks have been tested in llvm/test/CodeGen/PowerPC/fold-rlwinm.mir
This patch didn't change the patterns, so I supposed it would be clearer to remove some redundant test cases.
Do you think they should be added back？

Understanding your concerns here. I prefer to keep that testcases as well. If new added post ra peephole in pre-emit-peephole pass changes the case, the pre-ra cases will not be aware of? This is up to you.^^

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3983

Is it safe if we add SrcMI->getOperand(1).isReg()?. We can bail out early if the operand 1 of SrcMI is not a register.

4079

Is it ok if we change use_nodbg_empty to hasOneNonDBGUse()?

[Esme]

Updated as Zheng's suggestions.

[shchenz]

LGTM. Thanks for the new fix.

[lkail]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3928

I suggest checking Register::isPhysicalRegister explicitly, since Reg might also be a stackslot.

3951

Suggest using modifiesRegister or it'll miss case like $x3 = rlwinm $r3, ...

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3928

Could you please explain more here? this is after RA, if FoldingReg is a stackslot, we should get r1/x1? I don't understand how Register::isPhysicalRegister would change the semantic here？ Thanks.

3951

That would not happen, for RLWINM, we get input and output are both r3, for RLWINM8 we get input and output are both x3.

[lkail]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3928

If it's the case, an assertion here would be better.

[Esme]

Check if the register used can be forward.

[nemanjai]

I have added a number of comments to code that isn't part of this review which may seem odd. However, you are adding a fair amount of code to a function that is already very long/complex and rather than doing so, I'd prefer that you refactor the original function so the new code integrates more seamlessly.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

3938

This is a bit of a bizarre set up that makes it hard to read. How about skipping these switch statements and just doing what you're after:

1. Find the source MI
2. Check if they're both 32-bit or both 64-bit instructions

For 2. you can do something like:

static bool sameWidthRLWINM(MachineInstr *SrcMI, MachineInstr *UseMI,
                            bool &Is64Bit) {
  unsigned SrcOpc = SrcMI->getOpcode();
  unsigned UseOpc = UseMI->getOpcode();
  if ((SrcOpc == PPC::RLWINM8 || SrcOpc == PPC::RLWINM8_rec) &&
      (UseOpc == PPC::RLWINM8 || UseOpc == PPC::RLWINM8_rec)) {
    Is64Bit = true;
    return true;
  }
  if ((SrcOpc == PPC::RLWINM || SrcOpc == PPC::RLWINM_rec) &&
      (UseOpc == PPC::RLWINM || UseOpc == PPC::RLWINM_rec))
    return true;
  Is64Bit = false;
  return false;
}

3999

Please describe this "special case" since it appears to be two special cases:
The first part means all bits in a 64-bit register and the second part means the low 32 bits in a 64-bit register.

4007–4008

I am not sure what the intended meaning of lowerest is here, but please use something like least significant bit.

4018–4019

Why can't this whole block be replaced by a call to replaceInstrWithLI()?

4043

I find it unlikely that clang-format produced this formatting.

4053–4054

No need, but is it incorrect to update them? Namely, won't the NewMB/NewME be the same as MBMI/MEMI respectively? If so, I think it is preferable to avoid the condition and leave it to the reader to figure out.

4059–4060

In general, I find that manually messing with kill flags when changing the instructions to be an error-prone thing that should be avoided if possible? Is it possible to re-run the pass that computes kill flags after the peephole?

For example I think that here, if MI.getOperand(1).isKill() before the change, we will not update the kill flag on the previous use of that register so it will be considered live after MI whereas before it wasn't. This is of course not semantically incorrect, but may hinder some optimizations.

llvm/test/CodeGen/PowerPC/vsx_builtins.ll

167

It is not really related to this patch, but what is this test case supposed to do? Why would we produce a 4-bit field in a CR, move it to a GPR and then right shift by 4 (clearly shifting out all the bits)?

[Esme]

Addressed nemanjai's comments.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

4053–4054

It's incorrect to update MBMI/MEMI with NewMB/NewME if the SrcMI mask is full, since isRunOfOnes(FinalMask) may not hold true in such scenario. Well might it be more clear to replace no need with don't?

4059–4060

Good catch! However I have no idea which pass can solve the problem. It seems that LiveInternals Pass will computes kill flags, but it can't handle the case you mentioned. Do you have any idea about this?

llvm/test/CodeGen/PowerPC/vsx_builtins.ll

167

Yes. you're right. lol... This test case was designed to show the folding patter, rlwinm + rlwinm -> li 0 in post-ra.

[Esme]

Rebase.
LIT, LNT and Bootstrap are clean.

[Esme]

gentle ping

[Esme]

Migrated to Github PR https://github.com/llvm/llvm-project/pull/67293

llvm/lib/Target/PowerPC/PPCInstrInfo.h

#endif
   bool isZeroExtended(const unsigned Reg,
                       const MachineRegisterInfo *MRI) const {
     return isSignOrZeroExtended(Reg, 0, MRI).second;
   }
 
   bool convertToImmediateForm(MachineInstr &MI,
                               MachineInstr **KilledDef = nullptr) const;
   bool foldFrameOffset(MachineInstr &MI) const;
-  bool combineRLWINM(MachineInstr &MI, MachineInstr **ToErase = nullptr) const;
+  bool simplifyRotateAndMaskInstr(MachineInstr &MI,
+                                  MachineInstr *&ToErase) const;
   bool isADDIInstrEligibleForFolding(MachineInstr &ADDIMI, int64_t &Imm) const;
   bool isADDInstrEligibleForFolding(MachineInstr &ADDMI) const;
   bool isImmInstrEligibleForFolding(MachineInstr &MI, unsigned &BaseReg,
                                     unsigned &XFormOpcode,
                                     int64_t &OffsetOfImmInstr,
                                     ImmInstrInfo &III) const;
   bool isValidToBeChangedReg(MachineInstr *ADDMI, unsigned Index,
                              MachineInstr *&ADDIMI, int64_t &OffsetAddi,

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

bool PPCInstrInfo::convertToImmediateForm(MachineInstr &MI,
   // If this is not a reg+reg, but the DefMI is LI/LI8, check if its user MI
   // can be simpified to LI.
   if (!HasImmForm && simplifyToLI(MI, *DefMI, ForwardingOperand, KilledDef))
     return true;
 
   return false;
 }
 
-bool PPCInstrInfo::combineRLWINM(MachineInstr &MI,
-                                 MachineInstr **ToErase) const {
+// Return true if SrcMI and MI are both 32-bit or both 64-bit instructions.
+static bool sameWidthMIs(MachineInstr *SrcMI, MachineInstr *MI, bool &Is64Bit) {
+  unsigned Opc = MI->getOpcode();
+  unsigned SrcOpc = SrcMI->getOpcode();
+  if ((SrcOpc == PPC::RLWINM8 || SrcOpc == PPC::RLWINM8_rec) &&
+      (Opc == PPC::RLWINM8 || Opc == PPC::RLWINM8_rec)) {
+    Is64Bit = true;
+    return true;
+  }
+  if ((SrcOpc == PPC::RLWINM || SrcOpc == PPC::RLWINM_rec) &&
+      (Opc == PPC::RLWINM || Opc == PPC::RLWINM_rec))
+    return true;
+  return false;
+}
+
+// This function tries to combine two RLWINMs. We not only perform such
+// optimization in SSA, but also after RA, since some RLWINM is generated after
+// RA.
+bool PPCInstrInfo::simplifyRotateAndMaskInstr(MachineInstr &MI,
+                                              MachineInstr *&ToErase) const {
+  unsigned UseOpc = MI.getOpcode();
+  if (UseOpc != PPC::RLWINM && UseOpc != PPC::RLWINM_rec &&
+      UseOpc != PPC::RLWINM8 && UseOpc != PPC::RLWINM8_rec)
+    return false;
+
+  // Find the source MI.
   MachineRegisterInfo *MRI = &MI.getParent()->getParent()->getRegInfo();
   Register FoldingReg = MI.getOperand(1).getReg();
+  MachineInstr *SrcMI = nullptr;
+  bool CanErase = false;
+  bool OtherIntermediateUse = true;
+  if (MRI->isSSA()) {
     if (!FoldingReg.isVirtual())
       return false;
-  MachineInstr *SrcMI = MRI->getVRegDef(FoldingReg);
-  if (SrcMI->getOpcode() != PPC::RLWINM &&
-      SrcMI->getOpcode() != PPC::RLWINM_rec &&
-      SrcMI->getOpcode() != PPC::RLWINM8 &&
-      SrcMI->getOpcode() != PPC::RLWINM8_rec)
+    SrcMI = MRI->getVRegDef(FoldingReg);
+  } else {
+    if (!Register::isPhysicalRegister(FoldingReg))
+      return false;
+    SrcMI = getDefMIPostRA(FoldingReg, MI, OtherIntermediateUse);
+  }
+  if (!SrcMI)
+    return false;
+
+  // Check if MI and SrcMI are both 32-bit or both 64-bit instructions.
+  // TODO: The pairs of RLWINM8(RLWINM) or RLWINM(RLWINM8) never occur before
+  // RA, but after RA. Even they are not in the same bit-width, we can do the
+  // foldings for RLWINM8(RLWINM)->RLWINM8, or RLWINM(RLWINM8)->RLWINM.
+  bool Is64Bit = false;
+  if (!sameWidthMIs(SrcMI, &MI, Is64Bit))
+    return false;
+
+  // Check if the registers(def and use) meet the requirements for folding.
+  MachineOperand ForwardRegOp = SrcMI->getOperand(1);
+  Register ForwardReg = ForwardRegOp.getReg();
+  bool IsFwdFeederRegKilled = false;
+  bool SeenIntermediateUse = false;
+  bool IsMIUseRegKilled = MI.getOperand(1).isKill();
+  if (MRI->isSSA()) {
+    CanErase = !SrcMI->hasImplicitDef() && MRI->hasOneNonDBGUse(FoldingReg);
+  } else {
+    bool KillFwdDefMI = !OtherIntermediateUse && IsMIUseRegKilled;
+    CanErase = KillFwdDefMI && !SrcMI->hasImplicitDef();
+    // In post-RA, if SrcMI also defines the register to be forwarded, we can
+    // only do the folding if SrcMI is going to be erased.
+    if (!CanErase && SrcMI->definesRegister(ForwardReg))
+      return false;
+    // Check if the SrcReg can be forwarded to MI.
+    if (!isRegElgibleForForwarding(ForwardRegOp, *SrcMI, MI, KillFwdDefMI,
+                                   IsFwdFeederRegKilled, SeenIntermediateUse))
       return false;
+  }
+
   assert((MI.getOperand(2).isImm() && MI.getOperand(3).isImm() &&
           MI.getOperand(4).isImm() && SrcMI->getOperand(2).isImm() &&
           SrcMI->getOperand(3).isImm() && SrcMI->getOperand(4).isImm()) &&
          "Invalid PPC::RLWINM Instruction!");
   uint64_t SHSrc = SrcMI->getOperand(2).getImm();
   uint64_t SHMI = MI.getOperand(2).getImm();
   uint64_t MBSrc = SrcMI->getOperand(3).getImm();
   uint64_t MBMI = MI.getOperand(3).getImm();
   uint64_t MESrc = SrcMI->getOperand(4).getImm();
   uint64_t MEMI = MI.getOperand(4).getImm();
-
   assert((MEMI < 32 && MESrc < 32 && MBMI < 32 && MBSrc < 32) &&
          "Invalid PPC::RLWINM Instruction!");
   // If MBMI is bigger than MEMI, we always can not get run of ones.
   // RotatedSrcMask non-wrap:
   //                 0........31|32........63
   // RotatedSrcMask:   B---E        B---E
   // MaskMI:         -----------|--E  B------
   // Result:           -----          ---      (Bad candidate)
   //
   // RotatedSrcMask wrap:
   //                 0........31|32........63
   // RotatedSrcMask: --E   B----|--E    B----
   // MaskMI:         -----------|--E  B------
   // Result:         ---   -----|---    -----  (Bad candidate)
   //
   // One special case is RotatedSrcMask is a full set mask.
   // RotatedSrcMask full:
   //                 0........31|32........63
   // RotatedSrcMask: ------EB---|-------EB---
   // MaskMI:         -----------|--E  B------
   // Result:         -----------|---  -------  (Good candidate)
 
-  // Mark special case.
+  // Mark the special cases of all bits in a 64-bit register or the low 32 bits
+  // in a 64-bit register.
   bool SrcMaskFull = (MBSrc - MESrc == 1) || (MBSrc == 0 && MESrc == 31);
 
   // For other MBMI > MEMI cases, just return.
   if ((MBMI > MEMI) && !SrcMaskFull)
     return false;
 
   // Handle MBMI <= MEMI cases.
   APInt MaskMI = APInt::getBitsSetWithWrap(32, 32 - MEMI - 1, 32 - MBMI);
   // In MI, we only need low 32 bits of SrcMI, just consider about low 32
-  // bit of SrcMI mask. Note that in APInt, lowerest bit is at index 0,
-  // while in PowerPC ISA, lowerest bit is at index 63.
+  // bit of SrcMI mask. Note that in APInt, the least significant bit is at
+  // index 0, while in PowerPC ISA, the least significant bit is at index 63.
   APInt MaskSrc = APInt::getBitsSetWithWrap(32, 32 - MESrc - 1, 32 - MBSrc);
 
   APInt RotatedSrcMask = MaskSrc.rotl(SHMI);
   APInt FinalMask = RotatedSrcMask & MaskMI;
   uint32_t NewMB, NewME;
   bool Simplified = false;
 
-  // If final mask is 0, MI result should be 0 too.
+  // If final mask is 0, replace MI with LI/LI8 0 or ANDI_rec/ANDI8_rec 0.
   if (FinalMask.isZero()) {
-    bool Is64Bit =
-        (MI.getOpcode() == PPC::RLWINM8 || MI.getOpcode() == PPC::RLWINM8_rec);
     Simplified = true;
     LLVM_DEBUG(dbgs() << "Replace Instr: ");
     LLVM_DEBUG(MI.dump());
 
-    if (MI.getOpcode() == PPC::RLWINM || MI.getOpcode() == PPC::RLWINM8) {
-      // Replace MI with "LI 0"
-      MI.removeOperand(4);
-      MI.removeOperand(3);
-      MI.removeOperand(2);
-      MI.getOperand(1).ChangeToImmediate(0);
-      MI.setDesc(get(Is64Bit ? PPC::LI8 : PPC::LI));
-    } else {
-      // Replace MI with "ANDI_rec reg, 0"
-      MI.removeOperand(4);
-      MI.removeOperand(3);
-      MI.getOperand(2).setImm(0);
-      MI.setDesc(get(Is64Bit ? PPC::ANDI8_rec : PPC::ANDI_rec));
-      MI.getOperand(1).setReg(SrcMI->getOperand(1).getReg());
-      if (SrcMI->getOperand(1).isKill()) {
+    LoadImmediateInfo LII;
+    LII.Imm = 0;
+    LII.Is64Bit = Is64Bit;
+    LII.SetCR = (UseOpc == PPC::RLWINM_rec || UseOpc == PPC::RLWINM8_rec);
+    replaceInstrWithLI(MI, LII);
+    if (LII.SetCR) {
+      MI.getOperand(1).setReg(ForwardReg);
+      // FIXME: If the register used by MI is `killed` before change, we need
+      // update the kill flag on the previous use of that register. Here we only
+      // considered the kill flag of the register used by SrcMI.
+      if (ForwardRegOp.isKill()) {
         MI.getOperand(1).setIsKill(true);
         SrcMI->getOperand(1).setIsKill(false);
       } else
         // About to replace MI.getOperand(1), clear its kill flag.
         MI.getOperand(1).setIsKill(false);
     }
 
     LLVM_DEBUG(dbgs() << "With: ");
     LLVM_DEBUG(MI.dump());
-
   } else if ((isRunOfOnes((unsigned)(FinalMask.getZExtValue()), NewMB, NewME) &&
               NewMB <= NewME) ||
              SrcMaskFull) {
     // Here we only handle MBMI <= MEMI case, so NewMB must be no bigger
     // than NewME. Otherwise we get a 64 bit value after folding, but MI
     // return a 32 bit value.
     Simplified = true;
     LLVM_DEBUG(dbgs() << "Converting Instr: ");
     LLVM_DEBUG(MI.dump());
 
-    uint16_t NewSH = (SHSrc + SHMI) % 32;
-    MI.getOperand(2).setImm(NewSH);
-    // If SrcMI mask is full, no need to update MBMI and MEMI.
+    MI.getOperand(2).setImm((SHSrc + SHMI) % 32);
+    // If SrcMI mask is full, do not update MBMI and MEMI.
     if (!SrcMaskFull) {
       MI.getOperand(3).setImm(NewMB);
       MI.getOperand(4).setImm(NewME);
     }
-    MI.getOperand(1).setReg(SrcMI->getOperand(1).getReg());
-    if (SrcMI->getOperand(1).isKill()) {
+    MI.getOperand(1).setReg(ForwardReg);
+    if (ForwardRegOp.isKill()) {
       MI.getOperand(1).setIsKill(true);
       SrcMI->getOperand(1).setIsKill(false);
     } else
       // About to replace MI.getOperand(1), clear its kill flag.
       MI.getOperand(1).setIsKill(false);
 
     LLVM_DEBUG(dbgs() << "To: ");
     LLVM_DEBUG(MI.dump());
   }
-  if (Simplified & MRI->use_nodbg_empty(FoldingReg) &&
-      !SrcMI->hasImplicitDef()) {
-    // If FoldingReg has no non-debug use and it has no implicit def (it
-    // is not RLWINMO or RLWINM8o), it's safe to delete its def SrcMI.
-    // Otherwise keep it.
-    *ToErase = SrcMI;
+  if (Simplified && CanErase) {
+    // If SrcMI has no implicit def, and FoldingReg has no non-debug use or
+    // its flag is "killed", it's safe to delete SrcMI. Otherwise keep it.
+    ToErase = SrcMI;
     LLVM_DEBUG(dbgs() << "Delete dead instruction: ");
     LLVM_DEBUG(SrcMI->dump());
   }
   return Simplified;
 }
 
 bool PPCInstrInfo::instrHasImmForm(unsigned Opc, bool IsVFReg,
                                    ImmInstrInfo &III, bool PostRA) const {
   // The vast majority of the instructions would need their operand 2 replaced
   // with an immediate when switching to the reg+imm form. A marked exception
   // are the update form loads/stores for which a constant operand 2 would need
   // to turn into a displacement and move operand 1 to the operand 2 position.
   III.ImmOpNo = 2;
   III.OpNoForForwarding = 2;
   III.ImmWidth = 16;

llvm/lib/Target/PowerPC/PPCMIPeephole.cpp

for (MachineInstr &MI : MBB) {
         Simplified |= emitRLDICWhenLoweringJumpTables(MI) ||
                       combineSEXTAndSHL(MI, ToErase);
         break;
       }
       case PPC::RLWINM:
       case PPC::RLWINM_rec:
       case PPC::RLWINM8:
       case PPC::RLWINM8_rec: {
-        Simplified = TII->combineRLWINM(MI, &ToErase);
+        Simplified = TII->simplifyRotateAndMaskInstr(MI, ToErase);
         if (Simplified)
           ++NumRotatesCollapsed;
         break;
       }
       // We will replace TD/TW/TDI/TWI with an unconditional trap if it will
       // always trap, we will delete the node if it will never trap.
       case PPC::TDI:
       case PPC::TWI:

llvm/lib/Target/PowerPC/PPCPreEmitPeephole.cpp

 STATISTIC(NumRRConvertedInPreEmit,
           "Number of r+r instructions converted to r+i in pre-emit peephole");
 STATISTIC(NumRemovedInPreEmit,
           "Number of instructions deleted in pre-emit peephole");
 STATISTIC(NumberOfSelfCopies,
           "Number of self copy instructions eliminated");
 STATISTIC(NumFrameOffFoldInPreEmit,
           "Number of folding frame offset by using r+r in pre-emit peephole");
 STATISTIC(NumCmpsInPreEmit,
           "Number of compares eliminated in pre-emit peephole");
+STATISTIC(NumRotateInstrFoldInPreEmit,
+          "Number of folding Rotate instructions in pre-emit peephole");
 
 static cl::opt<bool>
 EnablePCRelLinkerOpt("ppc-pcrel-linker-opt", cl::Hidden, cl::init(true),
                      cl::desc("enable PC Relative linker optimization"));
 
 static cl::opt<bool>
 RunPreEmitPeephole("ppc-late-peephole", cl::Hidden, cl::init(true),
                    cl::desc("Run pre-emit peephole optimizations."));
bool runOnMachineFunction(MachineFunction &MF) override {
             NumFrameOffFoldInPreEmit++;
             LLVM_DEBUG(dbgs() << "Frame offset folding by using index form: ");
             LLVM_DEBUG(MI.dump());
           }
           if (TII->optimizeCmpPostRA(MI)) {
             Changed = true;
             NumCmpsInPreEmit++;
             LLVM_DEBUG(dbgs() << "Optimize compare by using record form: ");
             LLVM_DEBUG(MI.dump());
             InstrsToErase.push_back(&MI);
           }
+          MachineInstr *ToErase = nullptr;
+          if (TII->simplifyRotateAndMaskInstr(MI, ToErase)) {
+            Changed = true;
+            NumRotateInstrFoldInPreEmit++;
+            if (ToErase)
+              InstrsToErase.push_back(ToErase);
+          }
         }
 
         // Eliminate conditional branch based on a constant CR bit by
         // CRSET or CRUNSET. We eliminate the conditional branch or
         // convert it into an unconditional branch. Also, if the CR bit
         // is not used by other instructions, we eliminate CRSET as well.
         auto I = MBB.getFirstInstrTerminator();
         if (I == MBB.instr_end())

llvm/test/CodeGen/PowerPC/fold-rlwinm-after-ra.mir

+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py
+# RUN: llc -mcpu=pwr9 -mtriple=powerpc64le-unknown-unknown -stop-after \
+# RUN:   ppc-pre-emit-peephole %s -o - | FileCheck %s
+
+---
+name: testFoldRLWINM
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINM
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 14, 0, 12, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 27, 5, 31
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 0, 12, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMSrcFullMask1
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMSrcFullMask1
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 14, 0, 12, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 27, 0, 31
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 0, 12, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMSrcFullMask2
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r2, $r3
+    ; CHECK-LABEL: name: testFoldRLWINMSrcFullMask2
+    ; CHECK: liveins: $r2, $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $r3 = RLWINM $r2, 14, 10, 1, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r2, 27, 10, 9
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 10, 1, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMSrcWrapped
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMSrcWrapped
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 14, 11, 12, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 27, 30, 10
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 0, 12, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMUserWrapped
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMUserWrapped
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $r3 = RLWINM killed $r3, 10, 5, 31
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 10, 30, 5, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 10, 5, 31
+    dead renamable $r3 = RLWINM killed renamable $r3, 10, 30, 5, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMResultWrapped
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMResultWrapped
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $r3 = RLWINM killed $r3, 10, 20, 10
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 10, 0, 31, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 10, 20, 10
+    dead renamable $r3 = RLWINM killed renamable $r3, 10, 0, 31, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINMToZero
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMToZero
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: renamable $r3 = LI 0
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 27, 5, 10
+    dead renamable $r3 = RLWINM killed renamable $r3, 8, 5, 10, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINM_recToZero
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINM_recToZero
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: dead renamable $r3 = ANDI_rec killed renamable $r3, 0, implicit-def $cr0
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+    $r3 = RLWINM killed $r3, 27, 5, 10
+    dead renamable $r3 = RLWINM_rec killed renamable $r3, 8, 5, 10, implicit-def $cr0
+    BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+...
+---
+name: testFoldRLWINMInvalidMask
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testFoldRLWINMInvalidMask
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $r3 = RLWINM killed $r3, 20, 5, 31
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 19, 10, 20, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+    $r3 = RLWINM killed $r3, 20, 5, 31
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 10, 20, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x3
+...
+---
+name: testFoldRLWINCanNotBeDeleted
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r2, $r3
+    ; CHECK-LABEL: name: testFoldRLWINCanNotBeDeleted
+    ; CHECK: liveins: $r2, $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $r3 = RLWINM_rec $r2, 27, 5, 10, implicit-def dead $cr0
+    ; CHECK-NEXT: dead renamable $r3 = ANDI_rec $r2, 0, implicit-def $cr0
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+    $r3 = RLWINM_rec $r2, 27, 5, 10, implicit-def $cr0
+    dead renamable $r3 = RLWINM_rec killed renamable $r3, 8, 5, 10, implicit-def $cr0
+    BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+...
+---
+name: testCanNotFoldRLWINM
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r3
+    ; CHECK-LABEL: name: testCanNotFoldRLWINM
+    ; CHECK: liveins: $r3
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: $r3 = RLWINM_rec killed $r3, 27, 5, 10, implicit-def dead $cr0
+    ; CHECK-NEXT: dead renamable $r3 = RLWINM_rec killed renamable $r3, 8, 5, 10, implicit-def $cr0
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+    $r3 = RLWINM_rec $r3, 27, 5, 10, implicit-def $cr0
+    dead renamable $r3 = RLWINM_rec killed renamable $r3, 8, 5, 10, implicit-def $cr0
+    BLR8 implicit $lr8, implicit $rm, implicit killed $cr0
+...
+---
+name: testCanNotFoldRLWINM2
+tracksRegLiveness: true
+body: |
+  bb.0.entry:
+    liveins: $r2, $r3
+    ; CHECK-LABEL: name: testCanNotFoldRLWINM2
+    ; CHECK: liveins: $r2, $r3, $x2
+    ; CHECK-NEXT: {{  $}}
+    ; CHECK-NEXT: STD $x2, -8, $x1 :: (store (s64) into %stack.0)
+    ; CHECK-NEXT: $r3 = RLWINM killed $r2, 4, 28, 31
+    ; CHECK-NEXT: $r2 = LI 0, implicit-def $x2
+    ; CHECK-NEXT: $x2 = LD -8, $x1 :: (load (s64) from %stack.0)
+    ; CHECK-NEXT: renamable $r3 = RLWINM killed renamable $r3, 19, 0, 12, implicit-def $x3
+    ; CHECK-NEXT: BLR8 implicit $lr8, implicit $rm, implicit $x2, implicit killed $x3
+    $r3 = RLWINM killed $r2, 4, 28, 31
+    $r2 = LI 0, implicit-def $x2
+    dead renamable $r3 = RLWINM killed renamable $r3, 19, 0, 12, implicit-def $x3
+    BLR8 implicit $lr8, implicit $rm, implicit killed $x2, implicit killed $x3
+...

llvm/test/CodeGen/PowerPC/vsx_builtins.ll

entry:
     ret i32 %retval.0
 }
 
 define i32 @xvtdivdp_shift(<2 x double> %a, <2 x double> %b) {
 ; CHECK-LABEL: xvtdivdp_shift:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    xvtdivdp cr0, v2, v3
 ; CHECK-NEXT:    mfocrf r3, 128
-; CHECK-NEXT:    srwi r3, r3, 28
-; CHECK-NEXT:    rlwinm r3, r3, 28, 31, 31
+; CHECK-NEXT:    li r3, 0
 ; CHECK-NEXT:    blr
 entry:
   %0 = tail call i32 @llvm.ppc.vsx.xvtdivdp(<2 x double> %a, <2 x double> %b)
   %1 = lshr i32 %0, 4
   %.lobit = and i32 %1, 1
   ret i32 %.lobit
 }
 
 ; Function Attrs: nounwind readnone
 define <2 x double> @test_lxvd2x(ptr %a) {
 ; CHECK-P9UP-LABEL: test_lxvd2x:
 ; CHECK-P9UP:       # %bb.0: # %entry