[PowerPC] support register pressure reduction in machine combiner.

[shchenz]

Summary

This patch tries to transform following patterns:

// Pattern 1:
//   A = FSUB  X,    Y        (Leaf)
//   C = FMA   A31,  M31,  A  (Root)
// M31 is const -->
//   A = FMA   A31,  Y,  -M31
//   C = FMA   A,    X,  M31
//
// Pattern 2:
//   A = FSUB  X,    Y        (Leaf)
//   C = FMA   A31,  A,  M32  (Root)
// M32 is const -->
//   A = FMA   A31,  Y,  -M32
//   C = FMA   A,    X,  M32
//

On PowerPC target, fma instructions are destructive, its def is always assigned with the same physical register with one of its operands. We could use this characteristic to generate more fma instructions to generate friendly code for register allocation.

For example, for the below case:

T = A * B + Const1 * (C - D) + Const2 * (E - F)

Without this patch, llvm generates:

t0 = mul A, B
t1 = sub C, D
t2 = sub E, F
....
t3 = FMA t0, Const1, t1
T = FMA t3, Const2, t2

t0 & t1 & t2 must be assigned with different registers.
With this patch, we get

t0 = mul A, B
t1 = FMA t0, Const1, C
t2 = FMA t1, -Const1, D
t3 = FMA t2, Const2, E
T = FMA t3, -Const2, F

Now, t0 & t1 & t2 & t3 & T must be assigned with same physical register.

We only do this transformation when the register is high as the transformation will reduce ILP.
We saw some obvious improvement for some cpu2017 benchmarks.

[shchenz]

1: don't require LiveIntervals analysis pass to estimate register pressure.

[shchenz]

1: update according to parent patch https://reviews.llvm.org/D92068 changes

[jsji]

llvm/include/llvm/CodeGen/MachineCombinerPattern.h

34

REASSOC_XY_BCA,
REASSOC_XY_BAC,

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

80

"ppc-fma-rp-factor"

Since we multiple this value, this is more a factor than a threshold, then the default value should be slightly more than 1.0?

83

EnableFMARegPressureReduction

84

"ppc-fma-rp-reduction"

350

A = FSUB X, Y (Leaf)
D = FMA B, C, A (Root)
->
A = FMA B, Y, -C
C = FMA A, X, C

355

This is the same pattern as above, as we can commute the operands of FMA?

A = FSUB X, Y (Leaf)
D = FMA B, A, C (Root)
->
A = FMA B, Y, -C
C = FMA A, X, C

442

Do all candidate checks in isRegisterPressureReduceCandidate, so that we have simple logic like:

if (DoRegPressureReduce && isRegisterPressureReduceCandidate(Root, MULInstrL,MULInstrR )) {
     if (isLoadFromConstantPool(MULInstrL) &&
         IsReassociableAddOrSub(*MULInstrR, InfoArrayIdxFSubInst)) {
          LLVM_DEBUG(dbgs() << "add pattern REASSOC_XY_BCA\n");
          Patterns.push_back(MachineCombinerPattern::REASSOC_XY_BCA);
          return true;
     }
     if (isLoadFromConstantPool(MULInstrR) &&
         IsReassociableAddOrSub(*MULInstrL, InfoArrayIdxFSubInst)) {
          LLVM_DEBUG(dbgs() << "add pattern REASSOC_XY_BCA\n");
          Patterns.push_back(MachineCombinerPattern::REASSOC_XY_BCA);
          return true;
     }
}

445

This check should be in isRegisterPressureReduceCandidate

447

Uninitialized variable.

449

This should be in isRegisterPressureReduceCandidate as well.

550

Why this can not be inlined into next line of ConstantFP::get?

569

What if we are in non-assert version and we can't find Placeholder ?

602

Split this out into a lamda or function , just call something like

currMBBPresure = getMBBPressure (MBB...);
VSSRCRPSetLimit= ...

return currMBBPresure >= VSSRCRPSetLimit * RegPressureFactor

637

This should check the Root, and look through copies, isVirtualRegister.
Only return true when all the requirements are met.

705

Add document about function and argument assumptions.

787

I think a switch table here would be clearer.

switch (Pattern){
case: MachineCombinerPattern::REASSOC_XY_AMM_BMM:
case :MachineCombinerPattern::REASSOC_XY_AMM_BMM:
     IsILPReassociate = true;
    Prev = MRI.getUniqueVRegDef(Root.getOperand(AddOpIdx).getReg());
    Leaf = MRI.getUniqueVRegDef(Prev->getOperand(AddOpIdx).getReg());
    IntersectedFlags = Root.getFlags() & Prev->getFlags() & Leaf->getFlags();
break;
case: MachineCombinerPattern::REASSOC_XY_BAC:
...
break;
}

803

Uninitialized variable.

882

Switch table please as well.

llvm/lib/Target/PowerPC/PPCInstrInfo.h

255

isRPReductionCandidate

353

DoRPReduction

362

/// Return true when ...

368

Fixup the placeholders we put in genAlternativeCodeSequence() for MachineCombiner.

llvm/test/CodeGen/PowerPC/register-pressure-reduction.ll

9

Can we add test to see whether we will be able to reuse const?
eg: If there is already negative const in the const pool.

45

If we have 2 patterns, we should test both patterns..

[shchenz]

update according to @jsji comments

Thanks very much for your careful review. I updated or replied accordingly.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

80

The logic here is if the calculated register pressure(RPTracker.getPressure()) exceeds the limits(get`RegPressureSetLimit`), we should do the reassociation. So I used 1.0. Does changing the compare from >= to > make sense to you?

550

return type of F1.changeSign(); is void

569

if we are in non-assert version, we should get SEGV, we use Placeholder like Placeholder->setReg(LoadNewConst);. If we come here, Placeholder must be not null, we only handle register reduction patterns here.

Do we need to add if(!Placeholder) return;? Seems we do not do this kind of protection in current llvm code base, for example:

ICmpInst::Predicate Loop::LoopBounds::getCanonicalPredicate() const {
  BasicBlock *Latch = L.getLoopLatch();
  assert(Latch && "Expecting valid latch");

  BranchInst *BI = dyn_cast_or_null<BranchInst>(Latch->getTerminator());

637

new IsRPReductionCandidate already does more than this.

llvm/test/CodeGen/PowerPC/register-pressure-reduction.ll

45

I tried this before, seems in ISel, constant operand is put as the second mul operand in FMA instruction. I also can not create a MIR case, as there are const pool inputs, met some syntax errors.

[shchenz]

fix lint warnings

[shchenz]

gentle ping

[jsji]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

80

Yes, I know it is exceed the limit, but I don't think we should invoke this optimization immediately when we exceed the limit, shouldn't we do such reg pressure reduction when the reg pressure is really high enough?

[shchenz]

1: set the register pressure factor to 1.5

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

80

Make sense. I changed the factor to 1.5 in the updated patch.

[jsji]

LGTM. Thanks.

[shchenz]

Appreciate your review @jsji . Could you please help to have a look at this patch's parent https://reviews.llvm.org/D92069. Thanks again.

[tpopp]

I'm going to roll this back as it's causing build bot failures (in progress example here: http://45.33.8.238/win/31506/). Sanitizers show the following issues:

==9097==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5628a1bd3d10 in llvm::PPCInstrInfo::getFMAPatterns(llvm::MachineInstr&, llvm::SmallVectorImpl<llvm::MachineCombinerPattern>&, bool) const third_party/llvm/llvm-project/llvm/lib/Target/PowerPC/PPCInstrIn
fo.cpp:474:27                                                                                            
    #1 0x5628a1bd7e6c in llvm::PPCInstrInfo::getMachineCombinerPatterns(llvm::MachineInstr&, llvm::SmallVectorImpl<llvm::MachineCombinerPattern>&, bool) const third_party/llvm/llvm-project/llvm/lib/Target/PowerP
C/PPCInstrInfo.cpp:751:7                            
    #2 0x5628a3291840 in combineInstructions third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineCombiner.cpp:593:15
    #3 0x5628a3291840 in (anonymous namespace)::MachineCombiner::runOnMachineFunction(llvm::MachineFunction&) third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineCombiner.cpp:736:16                            
    #4 0x5628a32f91a2 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13                                                    
    #5 0x5628a5e0dce7 in llvm::FPPassManager::runOnFunction(llvm::Function&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1439:27                                                               
    #6 0x5628a5e22f60 in llvm::FPPassManager::runOnModule(llvm::Module&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1485:16                                                                   
    #7 0x5628a5e0f1d5 in runOnModule third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1554:27
    #8 0x5628a5e0f1d5 in llvm::legacy::PassManagerImpl::run(llvm::Module&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:541:44                                                                  
    #9 0x56289f8dcb66 in compileModule(char**, llvm::LLVMContext&) third_party/llvm/llvm-project/llvm/tools/llc/llc.cpp:658:8

and

==1709==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7f155364a060 at pc 0x555a067755c4 bp 0x7fff9b909050 sp 0x7fff9b909048
READ of size 8 at 0x7f155364a060 thread T0
    #0 0x555a067755c3 in operator[] third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/vector:1550:18
    #1 0x555a067755c3 in llvm::PPCInstrInfo::shouldReduceRegisterPressure(llvm::MachineBasicBlock*, llvm::RegisterClassInfo*) const third_party/llvm/llvm-project/llvm/lib/Target/PowerPC/PPCInstrInfo.cpp:655:10
    #2 0x555a07df5618 in combineInstructions third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineCombiner.cpp:561:12
    #3 0x555a07df5618 in (anonymous namespace)::MachineCombiner::runOnMachineFunction(llvm::MachineFunction&) third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineCombiner.cpp:736:16
    #4 0x555a07e54479 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) third_party/llvm/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
    #5 0x555a0b360710 in llvm::FPPassManager::runOnFunction(llvm::Function&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1439:27
    #6 0x555a0b374470 in llvm::FPPassManager::runOnModule(llvm::Module&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1485:16
    #7 0x555a0b3616e0 in runOnModule third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1554:27
    #8 0x555a0b3616e0 in llvm::legacy::PassManagerImpl::run(llvm::Module&) third_party/llvm/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:541:44
    #9 0x555a0461db8b in compileModule(char**, llvm::LLVMContext&) third_party/llvm/llvm-project/llvm/tools/llc/llc.cpp:658:8
    #10 0x555a046183bf in main third_party/llvm/llvm-project/llvm/tools/llc/llc.cpp:363:22               
                                                    
Address 0x7f155364a060 is located in stack of thread T0 at offset 96 in frame                                                                                                                                      
    #0 0x555a067742ff in llvm::PPCInstrInfo::shouldReduceRegisterPressure(llvm::MachineBasicBlock*, llvm::RegisterClassInfo*) const third_party/llvm/llvm-project/llvm/lib/Target/PowerPC/PPCInstrInfo.cpp:598
                                                    
  This frame has 7 object(s):                                                                            
    [32, 40) '__x.i.i'                                                                                                                                                                                             
    [64, 72) 'retval.i.i'                                                                                                                                                                                          
    [96, 424) 'Pressure.i' (line 624) <== Memory access at offset 96 is inside this variable                                                                                                                       
    [496, 848) 'RPTracker.i' (line 625)                                                                                                                                                                            
    [912, 920) 'MII.i' (line 631)                                                                                                                                                                                  
    [944, 952) 'MIE.i' (line 631)                                                                                                                                                                                  
    [976, 1408) 'RegOpers.i' (line 637)                                                                                                                                                                            
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork                                                                                                   
      (longjmp and C++ exceptions *are* supported)

[shchenz]

1: fix sanitizer warning

[shchenz]

Hi @tpopp , thanks for reverting. Do you happen to know how to run the Sanitizers tests on X86-64 target? I don't have an X86 testing environment.

I addressed one issue related to stack-use-after-scope, but I can not find out the reason for failure use-of-uninitialized-value. Maybe the use-of-uninitialized-value failure is also the same issue as there is one variable be assigned with returning value of function shouldReduceRegisterPressure(This is where failure stack-use-after-scope happens.). But I am not sure.

If you can rerun the checks, could you please help to have a verify with the updated patch? Thanks.

[shchenz]

full context

[shchenz]

Confirmed that use-of-uninitialized-value can be reproduced on the PowerPC target and it can be fixed by the new patch.

[tpopp]

Thanks for addressing the issues shchenz! I'm glad you figured out the use-of-uninitialized-value because that one confused me. I'm finishing my workday for the day, so I can rerun the patch tomorrow. Alternatively, all I know is that I had errors in my environment and that's what sanitizers showed, so you reproducing and fixing the issues is plenty I think.

[shchenz]

update according to api change in D92069

[foad]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

347

Typo "floating".

362

Is it true that these patterns only improve register pressure if X and Y are "live out" of the pattern? Otherwise A could be allocated the same register as X or Y and there would be no increase.

365

Typo "attribute".

[shchenz]

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

347

Thanks for pointing this out. NFC patch b0869a7d72f121b77d48d6496c6c3f00dd4731da fix this.

362

One condition for this transformation is FMA is the only user of FSUB, which means we can delete the original FSUB to save its definition, the original A. After the transformation, A must be assigned with the same register with B and D

365

fixed in commit b0869a7d72f121b77d48d6496c6c3f00dd4731da

llvm/include/llvm/CodeGen/MachineCombinerPattern.h

enum class MachineCombinerPattern {
   REASSOC_AX_YB,
   REASSOC_XA_BY,
   REASSOC_XA_YB,
 
   // These are patterns matched by the PowerPC to reassociate FMA chains.
   REASSOC_XY_AMM_BMM,
   REASSOC_XMM_AMM_BMM,
 
+  // These are patterns matched by the PowerPC to reassociate FMA and FSUB to
+  // reduce register pressure.
+  REASSOC_FMA_M1C_FSUB,
+  REASSOC_FMA_M2C_FSUB,
+
   // These are multiply-add patterns matched by the AArch64 machine combiner.
   MULADDW_OP1,
   MULADDW_OP2,
   MULSUBW_OP1,
   MULSUBW_OP2,
   MULADDWI_OP1,
   MULSUBWI_OP1,
   MULADDX_OP1,

llvm/lib/CodeGen/MachineCombiner.cpp

static CombinerObjective getCombinerObjective(MachineCombinerPattern P) {
   switch (P) {
   case MachineCombinerPattern::REASSOC_AX_BY:
   case MachineCombinerPattern::REASSOC_AX_YB:
   case MachineCombinerPattern::REASSOC_XA_BY:
   case MachineCombinerPattern::REASSOC_XA_YB:
   case MachineCombinerPattern::REASSOC_XY_AMM_BMM:
   case MachineCombinerPattern::REASSOC_XMM_AMM_BMM:
     return CombinerObjective::MustReduceDepth;
+  case MachineCombinerPattern::REASSOC_FMA_M1C_FSUB:
+  case MachineCombinerPattern::REASSOC_FMA_M2C_FSUB:
+    return CombinerObjective::MustReduceRegisterPressure;
   default:
     return CombinerObjective::Default;
   }
 }
 
 /// Estimate the latency of the new and original instruction sequence by summing
 /// up the latencies of the inserted and deleted instructions. This assumes
 /// that the inserted and deleted instructions are dependent instruction chains,

llvm/lib/Target/PowerPC/PPCInstrInfo.h

class PPCInstrInfo : public PPCGenInstrInfo {
   const unsigned *getStoreOpcodesForSpillArray() const;
   const unsigned *getLoadOpcodesForSpillArray() const;
   unsigned getSpillIndex(const TargetRegisterClass *RC) const;
   int16_t getFMAOpIdxInfo(unsigned Opcode) const;
   void reassociateFMA(MachineInstr &Root, MachineCombinerPattern Pattern,
                       SmallVectorImpl<MachineInstr *> &InsInstrs,
                       SmallVectorImpl<MachineInstr *> &DelInstrs,
                       DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const;
+  bool isRegisterPressureReduceCandidate(MachineInstr &Root) const;
+  bool isLoadFromConstantPool(MachineInstr *I) const;
+  Register
+  generateLoadForNewConst(unsigned Idx, MachineInstr *MI, Type *Ty,
+                          SmallVectorImpl<MachineInstr *> &InsInstrs) const;
+  const Constant *getConstantFromConstantPool(MachineInstr *I) const;
   virtual void anchor();
 
 protected:
   /// Commutes the operands in the given instruction.
   /// The commutable operands are specified by their indices OpIdx1 and OpIdx2.
   ///
   /// Do not call this method for a non-commutable instruction or for
   /// non-commutable pair of operand indices OpIdx1 and OpIdx2.
void genAlternativeCodeSequence(
       SmallVectorImpl<MachineInstr *> &InsInstrs,
       SmallVectorImpl<MachineInstr *> &DelInstrs,
       DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const override;
 
   /// Return true when there is potentially a faster code sequence for a fma
   /// chain ending in \p Root. All potential patterns are output in the \p
   /// P array.
   bool getFMAPatterns(MachineInstr &Root,
-                      SmallVectorImpl<MachineCombinerPattern> &P) const;
+                      SmallVectorImpl<MachineCombinerPattern> &P,
+                      bool DoRegPressureReduce) const;
 
   /// Return true when there is potentially a faster code sequence
   /// for an instruction chain ending in <Root>. All potential patterns are
   /// output in the <Pattern> array.
   bool getMachineCombinerPatterns(MachineInstr &Root,
                                   SmallVectorImpl<MachineCombinerPattern> &P,
                                   bool DoRegPressureReduce) const override;
 
+  /// On PowerPC, we leverage machine combiner pass to reduce register pressure
+  /// when the register pressure is high for one BB.
+  bool
+  shouldReduceRegisterPressure(MachineBasicBlock *MBB,
+                               RegisterClassInfo *RegClassInfo) const override;
+
+  /// Fixup the placeholders we put in genAlternativeCodeSequence().
+  void
+  finalizeInsInstrs(MachineInstr &Root, MachineCombinerPattern &P,
+                    SmallVectorImpl<MachineInstr *> &InsInstrs) const override;
+
   bool isAssociativeAndCommutative(const MachineInstr &Inst) const override;
 
   /// On PowerPC, we try to reassociate FMA chain which will increase
   /// instruction size. Set extension resource length limit to 1 for edge case.
   /// Resource Length is calculated by scaled resource usage in getCycles().
   /// Because of the division in getCycles(), it returns different cycles due to
   /// legacy scaled resource usage. So new resource length may be same with
   /// legacy or 1 bigger than legacy.

llvm/lib/Target/PowerPC/PPCInstrInfo.cpp

 #include "PPCHazardRecognizers.h"
 #include "PPCInstrBuilder.h"
 #include "PPCMachineFunctionInfo.h"
 #include "PPCTargetMachine.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Analysis/AliasAnalysis.h"
 #include "llvm/CodeGen/LiveIntervals.h"
+#include "llvm/CodeGen/MachineConstantPool.h"
 #include "llvm/CodeGen/MachineFrameInfo.h"
 #include "llvm/CodeGen/MachineFunctionPass.h"
 #include "llvm/CodeGen/MachineInstrBuilder.h"
 #include "llvm/CodeGen/MachineMemOperand.h"
 #include "llvm/CodeGen/MachineRegisterInfo.h"
 #include "llvm/CodeGen/PseudoSourceValue.h"
+#include "llvm/CodeGen/RegisterClassInfo.h"
+#include "llvm/CodeGen/RegisterPressure.h"
 #include "llvm/CodeGen/ScheduleDAG.h"
 #include "llvm/CodeGen/SlotIndexes.h"
 #include "llvm/CodeGen/StackMaps.h"
 #include "llvm/MC/MCAsmInfo.h"
 #include "llvm/MC/MCInst.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"

 static cl::opt<bool> VSXSelfCopyCrash("crash-on-ppc-vsx-self-copy",
 cl::desc("Causes the backend to crash instead of generating a nop VSX copy"),
 cl::Hidden);
 
 static cl::opt<bool>
 UseOldLatencyCalc("ppc-old-latency-calc", cl::Hidden,
   cl::desc("Use the old (incorrect) instruction latency calculation"));
 
+static cl::opt<float> RegPressureThreshold(
+    "reg-pressure-threshold", cl::Hidden, cl::init(1.0),
+    cl::desc("register pressure threshold for the transformations."));
+
+static cl::opt<bool> EnableRegPressureReduce(
+    "enable-regpressure-reduce", cl::Hidden, cl::init(true),
+    cl::desc("enable register pressure reduce in machine combiner pass."));
+
 // Pin the vtable to this file.
 void PPCInstrInfo::anchor() {}
 
 PPCInstrInfo::PPCInstrInfo(PPCSubtarget &STI)
     : PPCGenInstrInfo(PPC::ADJCALLSTACKDOWN, PPC::ADJCALLSTACKUP,
                       /* CatchRetOpcode */ -1,
                       STI.isPPC64() ? PPC::BLR8 : PPC::BLR),
       Subtarget(STI), RI(STI.getTargetMachine()) {}
bool PPCInstrInfo::isAssociativeAndCommutative(const MachineInstr &Inst) const {
   }
 }
 
 #define InfoArrayIdxFMAInst 0
 #define InfoArrayIdxFAddInst 1
 #define InfoArrayIdxFMULInst 2
 #define InfoArrayIdxAddOpIdx 3
 #define InfoArrayIdxMULOpIdx 4
+#define InfoArrayIdxFSubInst 5
 // Array keeps info for FMA instructions:
 // Index 0(InfoArrayIdxFMAInst): FMA instruction;
-// Index 1(InfoArrayIdxFAddInst): ADD instruction assoaicted with FMA;
-// Index 2(InfoArrayIdxFMULInst): MUL instruction assoaicted with FMA;
+// Index 1(InfoArrayIdxFAddInst): ADD instruction associated with FMA;
+// Index 2(InfoArrayIdxFMULInst): MUL instruction associated with FMA;
 // Index 3(InfoArrayIdxAddOpIdx): ADD operand index in FMA operands;
 // Index 4(InfoArrayIdxMULOpIdx): first MUL operand index in FMA operands;
-//                                second MUL operand index is plus 1.
-static const uint16_t FMAOpIdxInfo[][5] = {
+//                                second MUL operand index is plus 1;
+// Index 5(InfoArrayIdxFSubInst): SUB instruction associated with FMA.
+static const uint16_t FMAOpIdxInfo[][6] = {
     // FIXME: Add more FMA instructions like XSNMADDADP and so on.
-    {PPC::XSMADDADP, PPC::XSADDDP, PPC::XSMULDP, 1, 2},
-    {PPC::XSMADDASP, PPC::XSADDSP, PPC::XSMULSP, 1, 2},
-    {PPC::XVMADDADP, PPC::XVADDDP, PPC::XVMULDP, 1, 2},
-    {PPC::XVMADDASP, PPC::XVADDSP, PPC::XVMULSP, 1, 2},
-    {PPC::FMADD, PPC::FADD, PPC::FMUL, 3, 1},
-    {PPC::FMADDS, PPC::FADDS, PPC::FMULS, 3, 1}};
+    {PPC::XSMADDADP, PPC::XSADDDP, PPC::XSMULDP, 1, 2, PPC::XSSUBDP},
+    {PPC::XSMADDASP, PPC::XSADDSP, PPC::XSMULSP, 1, 2, PPC::XSSUBSP},
+    {PPC::XVMADDADP, PPC::XVADDDP, PPC::XVMULDP, 1, 2, PPC::XVSUBDP},
+    {PPC::XVMADDASP, PPC::XVADDSP, PPC::XVMULSP, 1, 2, PPC::XVSUBSP},
+    {PPC::FMADD, PPC::FADD, PPC::FMUL, 3, 1, PPC::FSUB},
+    {PPC::FMADDS, PPC::FADDS, PPC::FMULS, 3, 1, PPC::FSUBS}};
 
 // Check if an opcode is a FMA instruction. If it is, return the index in array
 // FMAOpIdxInfo. Otherwise, return -1.
 int16_t PPCInstrInfo::getFMAOpIdxInfo(unsigned Opcode) const {
   for (unsigned I = 0; I < array_lengthof(FMAOpIdxInfo); I++)
     if (FMAOpIdxInfo[I][InfoArrayIdxFMAInst] == Opcode)
       return I;
   return -1;
 }
 
+// On PowerPC target, we have two kinds of patterns related to FMA:
+// 1: Improve ILP.
 // Try to reassociate FMA chains like below:
 //
 // Pattern 1:
 //   A =  FADD X,  Y          (Leaf)
 //   B =  FMA  A,  M21,  M22  (Prev)
 //   C =  FMA  B,  M31,  M32  (Root)
 // -->
 //   A =  FMA  X,  M21,  M22
 //   B =  FMA  Y,  M31,  M32
 //   C =  FADD A,  B
 //
 // Pattern 2:
 //   A =  FMA  X,  M11,  M12  (Leaf)
 //   B =  FMA  A,  M21,  M22  (Prev)
 //   C =  FMA  B,  M31,  M32  (Root)
 // -->
 //   A =  FMUL M11,  M12
 //   B =  FMA  X,  M21,  M22
 //   D =  FMA  A,  M31,  M32
 //   C =  FADD B,  D
 //
 // breaking the dependency between A and B, allowing FMA to be executed in
 // parallel (or back-to-back in a pipeline) instead of depending on each other.
+//
+// 2: Reduce register pressure.
+// Try to reassociate FMA with FSUB and a constant like below:
+//
+// Pattern 1:
+//   A = FSUB  X,    Y        (Leaf)
+//   C = FMA   A31,  M31,  A  (Root)
+// M31 is const -->
+//   A = FMA   A31,  Y,  -M31
+//   C = FMA   A,    X,  M31
+//
+// Pattern 2:
+//   A = FSUB  X,    Y        (Leaf)
+//   C = FMA   A31,  A,  M32  (Root)
+// M32 is const -->
+//   A = FMA   A31,  Y,  -M32
+//   C = FMA   A,    X,  M32
+//
+//  Before the transformation, A must be assigned with different hardware
+//  register with C. After the transformation, A and C must be assigned with
+//  same hardware register due to TIE attricute of FMA instructions.
+//
 bool PPCInstrInfo::getFMAPatterns(
-    MachineInstr &Root,
-    SmallVectorImpl<MachineCombinerPattern> &Patterns) const {
+    MachineInstr &Root, SmallVectorImpl<MachineCombinerPattern> &Patterns,
+    bool DoRegPressureReduce) const {
   MachineBasicBlock *MBB = Root.getParent();
-  const MachineRegisterInfo &MRI = MBB->getParent()->getRegInfo();
+  const MachineRegisterInfo *MRI = &MBB->getParent()->getRegInfo();
+  const TargetRegisterInfo *TRI = &getRegisterInfo();
 
   auto IsAllOpsVirtualReg = [](const MachineInstr &Instr) {
     for (const auto &MO : Instr.explicit_operands())
       if (!(MO.isReg() && Register::isVirtualRegister(MO.getReg())))
         return false;
     return true;
   };
 
-  auto IsReassociableAdd = [&](const MachineInstr &Instr) {
+  auto IsReassociableAddOrSub = [&](const MachineInstr &Instr,
+                                    unsigned OpType) {
     if (Instr.getOpcode() !=
-        FMAOpIdxInfo[getFMAOpIdxInfo(Root.getOpcode())][InfoArrayIdxFAddInst])
+        FMAOpIdxInfo[getFMAOpIdxInfo(Root.getOpcode())][OpType])
       return false;
 
     // Instruction can be reassociated.
     // fast math flags may prohibit reassociation.
     if (!(Instr.getFlag(MachineInstr::MIFlag::FmReassoc) &&
           Instr.getFlag(MachineInstr::MIFlag::FmNsz)))
       return false;
 
     // Instruction operands are virtual registers for reassociation.
     if (!IsAllOpsVirtualReg(Instr))
       return false;
 
+    // For register pressure reassociation, the FSub must have only one use as
+    // we want to delete the sub to save its def.
+    if (OpType == InfoArrayIdxFSubInst &&
+        !MRI->hasOneNonDBGUse(Instr.getOperand(0).getReg()))
+      return false;
+
     return true;
   };
 
   auto IsReassociableFMA = [&](const MachineInstr &Instr, int16_t &AddOpIdx,
-                               bool IsLeaf) {
+                               int16_t &MulOpIdx, bool IsLeaf) {
     int16_t Idx = getFMAOpIdxInfo(Instr.getOpcode());
     if (Idx < 0)
       return false;
 
     // Instruction can be reassociated.
     // fast math flags may prohibit reassociation.
     if (!(Instr.getFlag(MachineInstr::MIFlag::FmReassoc) &&
           Instr.getFlag(MachineInstr::MIFlag::FmNsz)))
       return false;
 
     // Instruction operands are virtual registers for reassociation.
     if (!IsAllOpsVirtualReg(Instr))
       return false;
 
+    MulOpIdx = FMAOpIdxInfo[Idx][InfoArrayIdxMULOpIdx];
     if (IsLeaf)
       return true;
 
     AddOpIdx = FMAOpIdxInfo[Idx][InfoArrayIdxAddOpIdx];
 
     const MachineOperand &OpAdd = Instr.getOperand(AddOpIdx);
-    MachineInstr *MIAdd = MRI.getUniqueVRegDef(OpAdd.getReg());
+    MachineInstr *MIAdd = MRI->getUniqueVRegDef(OpAdd.getReg());
     // If 'add' operand's def is not in current block, don't do ILP related opt.
     if (!MIAdd || MIAdd->getParent() != MBB)
       return false;
 
     // If this is not Leaf FMA Instr, its 'add' operand should only have one use
     // as this fma will be changed later.
-    return IsLeaf ? true : MRI.hasOneNonDBGUse(OpAdd.getReg());
+    return IsLeaf ? true : MRI->hasOneNonDBGUse(OpAdd.getReg());
   };
 
   int16_t AddOpIdx = -1;
+  int16_t MulOpIdx = -1;
+
+  // Register pressure fma reassociation patterns.
+  if (DoRegPressureReduce && isRegisterPressureReduceCandidate(Root)) {
     // Root must be a valid FMA like instruction.
-  if (!IsReassociableFMA(Root, AddOpIdx, false))
+    // Treat it as leaf as we don't care its add operand.
+    if (IsReassociableFMA(Root, AddOpIdx, MulOpIdx, true)) {
+      assert((MulOpIdx >= 0) && "mul operand index not right!");
+      bool IsUsedOnceL;
+      bool IsUsedOnceR;
+      Register MULRegL = TRI->lookThruCopyLike(
+          Root.getOperand(MulOpIdx).getReg(), MRI, &IsUsedOnceL);
+      Register MULRegR = TRI->lookThruCopyLike(
+          Root.getOperand(MulOpIdx + 1).getReg(), MRI, &IsUsedOnceR);
+      if (Register::isVirtualRegister(MULRegL) &&
+          Register::isVirtualRegister(MULRegR)) {
+        MachineInstr *MULInstrL = MRI->getVRegDef(MULRegL);
+        MachineInstr *MULInstrR = MRI->getVRegDef(MULRegR);
+
+        // Register pressure pattern 1
+        if (isLoadFromConstantPool(MULInstrL) && IsUsedOnceR &&
+            IsReassociableAddOrSub(*MULInstrR, InfoArrayIdxFSubInst)) {
+          LLVM_DEBUG(dbgs() << "add pattern REASSOC_FMA_M1C_FSUB\n");
+          Patterns.push_back(MachineCombinerPattern::REASSOC_FMA_M1C_FSUB);
+          return true;
+        }
+
+        // Register pressure pattern 2
+        if ((isLoadFromConstantPool(MULInstrR) && IsUsedOnceL &&
+             IsReassociableAddOrSub(*MULInstrL, InfoArrayIdxFSubInst))) {
+          LLVM_DEBUG(dbgs() << "add pattern REASSOC_FMA_M2C_FSUB\n");
+          Patterns.push_back(MachineCombinerPattern::REASSOC_FMA_M2C_FSUB);
+          return true;
+        }
+      }
+    }
+  }
+
+  // ILP fma reassociation patterns.
+  // Root must be a valid FMA like instruction.
+  if (!IsReassociableFMA(Root, AddOpIdx, MulOpIdx, false))
     return false;
 
   assert((AddOpIdx >= 0) && "add operand index not right!");
 
   Register RegB = Root.getOperand(AddOpIdx).getReg();
-  MachineInstr *Prev = MRI.getUniqueVRegDef(RegB);
+  MachineInstr *Prev = MRI->getUniqueVRegDef(RegB);
 
   // Prev must be a valid FMA like instruction.
   AddOpIdx = -1;
-  if (!IsReassociableFMA(*Prev, AddOpIdx, false))
+  if (!IsReassociableFMA(*Prev, AddOpIdx, MulOpIdx, false))
     return false;
 
   assert((AddOpIdx >= 0) && "add operand index not right!");
 
   Register RegA = Prev->getOperand(AddOpIdx).getReg();
-  MachineInstr *Leaf = MRI.getUniqueVRegDef(RegA);
+  MachineInstr *Leaf = MRI->getUniqueVRegDef(RegA);
   AddOpIdx = -1;
-  if (IsReassociableFMA(*Leaf, AddOpIdx, true)) {
+  if (IsReassociableFMA(*Leaf, AddOpIdx, MulOpIdx, true)) {
     Patterns.push_back(MachineCombinerPattern::REASSOC_XMM_AMM_BMM);
+    LLVM_DEBUG(dbgs() << "add pattern REASSOC_XMM_AMM_BMM\n");
     return true;
   }
-  if (IsReassociableAdd(*Leaf)) {
+  if (IsReassociableAddOrSub(*Leaf, InfoArrayIdxFAddInst)) {
     Patterns.push_back(MachineCombinerPattern::REASSOC_XY_AMM_BMM);
+    LLVM_DEBUG(dbgs() << "add pattern REASSOC_XY_AMM_BMM\n");
     return true;
   }
   return false;
 }
 
+void PPCInstrInfo::finalizeInsInstrs(
+    MachineInstr &Root, MachineCombinerPattern &P,
+    SmallVectorImpl<MachineInstr *> &InsInstrs) const {
+  assert(!InsInstrs.empty() && "Instructions set to be inserted is empty!");
+
+  MachineFunction *MF = Root.getMF();
+  MachineRegisterInfo *MRI = &MF->getRegInfo();
+  const TargetRegisterInfo *TRI = &getRegisterInfo();
+  MachineConstantPool *MCP = MF->getConstantPool();
+
+  int16_t Idx = getFMAOpIdxInfo(Root.getOpcode());
+  if (Idx < 0)
+    return;
+
+  uint16_t FirstMulOpIdx = FMAOpIdxInfo[Idx][InfoArrayIdxMULOpIdx];
+
+  // For now we only need to fix up placeholder for register pressure reduce
+  // patterns.
+  Register ConstReg = 0;
+  switch (P) {
+  case MachineCombinerPattern::REASSOC_FMA_M1C_FSUB:
+    ConstReg =
+        TRI->lookThruCopyLike(Root.getOperand(FirstMulOpIdx).getReg(), MRI);
+    break;
+  case MachineCombinerPattern::REASSOC_FMA_M2C_FSUB:
+    ConstReg =
+        TRI->lookThruCopyLike(Root.getOperand(FirstMulOpIdx + 1).getReg(), MRI);
+    break;
+  default:
+    // Not register pressure reduce patterns.
+    return;
+  }
+
+  MachineInstr *ConstDefInstr = MRI->getVRegDef(ConstReg);
+  // Get const value from const pool.
+  const Constant *C = getConstantFromConstantPool(ConstDefInstr);
+  assert(isa<llvm::ConstantFP>(C) && "not a valid constant!");
+
+  // Get negative fp const.
+  APFloat F1((dyn_cast<ConstantFP>(C))->getValueAPF());
+  F1.changeSign();
+  Constant *NegC = ConstantFP::get(dyn_cast<ConstantFP>(C)->getContext(), F1);
+  Align Alignment = MF->getDataLayout().getPrefTypeAlign(C->getType());
+
+  // Put negative fp const into constant pool.
+  unsigned ConstPoolIdx = MCP->getConstantPoolIndex(NegC, Alignment);
+
+  MachineOperand *Placeholder = nullptr;
+  // Record the placeholder PPC::ZERO8 we add in reassociateFMA.
+  for (auto *Inst : InsInstrs) {
+    for (MachineOperand &Operand : Inst->explicit_operands()) {
+      assert(Operand.isReg() && "Invalid instruction in InsInstrs!");
+      if (Operand.getReg() == PPC::ZERO8) {
+        Placeholder = &Operand;
+        break;
+      }
+    }
+  }
+
+  assert(Placeholder && "Placeholder does not exist!");
+
+  // Generate instructions to load the const fp from constant pool.
+  // We only support PPC64 and medium code model.
+  Register LoadNewConst =
+      generateLoadForNewConst(ConstPoolIdx, &Root, C->getType(), InsInstrs);
+
+  // Fill the placeholder with the new load from constant pool.
+  Placeholder->setReg(LoadNewConst);
+}
+
+bool PPCInstrInfo::shouldReduceRegisterPressure(
+    MachineBasicBlock *MBB, RegisterClassInfo *RegClassInfo) const {
+
+  if (!EnableRegPressureReduce)
+    return false;
+
+  // Currently, we only enable register pressure reducing in machine combiner
+  // for: 1: PPC64; 2: Code Model is Medium; 3: Power9 which also has vector
+  // support.
+  //
+  // So we need following instructions to access a TOC entry:
+  //
+  // %6:g8rc_and_g8rc_nox0 = ADDIStocHA8 $x2, %const.0
+  // %7:vssrc = DFLOADf32 target-flags(ppc-toc-lo) %const.0,
+  //   killed %6:g8rc_and_g8rc_nox0, implicit $x2 :: (load 4 from constant-pool)
+  //
+  // FIXME: add more supported targets, like Small and Large code model, PPC32,
+  // AIX.
+  if (!(Subtarget.isPPC64() && Subtarget.hasP9Vector() &&
+        Subtarget.getTargetMachine().getCodeModel() == CodeModel::Medium))
+    return false;
+
+  RegionPressure Pressure;
+  RegPressureTracker RPTracker(Pressure);
+  const TargetRegisterInfo *TRI = &getRegisterInfo();
+  MachineFunction *MF = MBB->getParent();
+  MachineRegisterInfo *MRI = &MF->getRegInfo();
+
+  // Initialize the register pressure tracker.
+  RPTracker.init(MBB->getParent(), RegClassInfo, nullptr, MBB, MBB->end(),
+                 /*TrackLaneMasks*/ false, /*TrackUntiedDefs=*/true);
+
+  for (MachineBasicBlock::iterator MII = MBB->instr_end(),
+                                   MIE = MBB->instr_begin();
+       MII != MIE; --MII) {
+    MachineInstr &MI = *std::prev(MII);
+    if (MI.isDebugValue() || MI.isDebugLabel())
+      continue;
+    RegisterOperands RegOpers;
+    RegOpers.collect(MI, *TRI, *MRI, false, false);
+    RPTracker.recedeSkipDebugValues();
+    assert(&*RPTracker.getPos() == &MI && "RPTracker sync error!");
+    RPTracker.recede(RegOpers);
+  }
+
+  // Close the RPTracker to finalize live ins.
+  RPTracker.closeRegion();
+
+  // For now we only care about float and double type fma.
+  // Only reduce register pressure when pressure is high.
+  return RPTracker.getPressure()
+             .MaxSetPressure[PPC::RegisterPressureSets::VSSRC] >=
+         ((float)TRI->getRegPressureSetLimit(*MBB->getParent(),
+                                             PPC::RegisterPressureSets::VSSRC) *
+          RegPressureThreshold);
+}
+
+bool PPCInstrInfo::isRegisterPressureReduceCandidate(MachineInstr &Root) const {
+  // Currently, we only support float and double.
+  // FIXME: add support for other types.
+  unsigned Opcode = Root.getOpcode();
+  return (Opcode == PPC::XSMADDASP || Opcode == PPC::XSMADDADP);
+}
+
+bool PPCInstrInfo::isLoadFromConstantPool(MachineInstr *I) const {
+  // I has only one memory operand which is load from constant pool.
+  if (!I->hasOneMemOperand())
+    return false;
+
+  MachineMemOperand *Op = I->memoperands()[0];
+  return Op->isLoad() && Op->getPseudoValue() &&
+         Op->getPseudoValue()->kind() == PseudoSourceValue::ConstantPool;
+}
+
+Register PPCInstrInfo::generateLoadForNewConst(
+    unsigned Idx, MachineInstr *MI, Type *Ty,
+    SmallVectorImpl<MachineInstr *> &InsInstrs) const {
+  // Now we only support PPC64, Medium code model and P9 with vector.
+  // We have immutable pattern to access const pool. See function
+  // shouldReduceRegisterPressure.
+  assert((Subtarget.isPPC64() && Subtarget.hasP9Vector() &&
+          Subtarget.getTargetMachine().getCodeModel() == CodeModel::Medium) &&
+         "Target not supported!\n");
+
+  MachineFunction *MF = MI->getMF();
+  MachineRegisterInfo *MRI = &MF->getRegInfo();
+
+  // Generate ADDIStocHA8
+  Register VReg1 = MRI->createVirtualRegister(&PPC::G8RC_and_G8RC_NOX0RegClass);
+  MachineInstrBuilder TOCOffset =
+      BuildMI(*MF, MI->getDebugLoc(), get(PPC::ADDIStocHA8), VReg1)
+          .addReg(PPC::X2)
+          .addConstantPoolIndex(Idx);
+
+  assert((Ty->isFloatTy() || Ty->isDoubleTy()) &&
+         "Only float and double are supported!");
+
+  unsigned LoadOpcode;
+  // Should be float type or double type.
+  if (Ty->isFloatTy())
+    LoadOpcode = PPC::DFLOADf32;
+  else
+    LoadOpcode = PPC::DFLOADf64;
+
+  const TargetRegisterClass *RC = MRI->getRegClass(MI->getOperand(0).getReg());
+  Register VReg2 = MRI->createVirtualRegister(RC);
+  MachineMemOperand *MMO = MF->getMachineMemOperand(
+      MachinePointerInfo::getConstantPool(*MF), MachineMemOperand::MOLoad,
+      Ty->getScalarSizeInBits() / 8, MF->getDataLayout().getPrefTypeAlign(Ty));
+
+  // Generate Load from constant pool.
+  MachineInstrBuilder Load =
+      BuildMI(*MF, MI->getDebugLoc(), get(LoadOpcode), VReg2)
+          .addConstantPoolIndex(Idx)
+          .addReg(VReg1, getKillRegState(true))
+          .addMemOperand(MMO);
+
+  Load->getOperand(1).setTargetFlags(PPCII::MO_TOC_LO);
+
+  // Insert the toc load instructions into InsInstrs.
+  InsInstrs.insert(InsInstrs.begin(), Load);
+  InsInstrs.insert(InsInstrs.begin(), TOCOffset);
+  return VReg2;
+}
+
+const Constant *
+PPCInstrInfo::getConstantFromConstantPool(MachineInstr *I) const {
+  MachineFunction *MF = I->getMF();
+  MachineRegisterInfo *MRI = &MF->getRegInfo();
+  MachineConstantPool *MCP = MF->getConstantPool();
+  assert(I->mayLoad() && "Should be a load instruction.\n");
+  for (auto MO : I->uses()) {
+    if (!MO.isReg())
+      continue;
+    Register Reg = MO.getReg();
+    if (Reg == 0 || !Register::isVirtualRegister(Reg))
+      continue;
+    // Find the toc address.
+    MachineInstr *DefMI = MRI->getVRegDef(Reg);
+    for (auto MO2 : DefMI->uses())
+      if (MO2.isCPI())
+        return (MCP->getConstants())[MO2.getIndex()].Val.ConstVal;
+  }
+  return nullptr;
+}
+
 bool PPCInstrInfo::getMachineCombinerPatterns(
     MachineInstr &Root, SmallVectorImpl<MachineCombinerPattern> &Patterns,
     bool DoRegPressureReduce) const {
   // Using the machine combiner in this way is potentially expensive, so
   // restrict to when aggressive optimizations are desired.
   if (Subtarget.getTargetMachine().getOptLevel() != CodeGenOpt::Aggressive)
     return false;
 
-  if (getFMAPatterns(Root, Patterns))
+  if (getFMAPatterns(Root, Patterns, DoRegPressureReduce))
     return true;
 
   return TargetInstrInfo::getMachineCombinerPatterns(Root, Patterns,
                                                      DoRegPressureReduce);
 }
 
 void PPCInstrInfo::genAlternativeCodeSequence(
     MachineInstr &Root, MachineCombinerPattern Pattern,
     SmallVectorImpl<MachineInstr *> &InsInstrs,
     SmallVectorImpl<MachineInstr *> &DelInstrs,
     DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const {
   switch (Pattern) {
   case MachineCombinerPattern::REASSOC_XY_AMM_BMM:
   case MachineCombinerPattern::REASSOC_XMM_AMM_BMM:
+  case MachineCombinerPattern::REASSOC_FMA_M1C_FSUB:
+  case MachineCombinerPattern::REASSOC_FMA_M2C_FSUB:
     reassociateFMA(Root, Pattern, InsInstrs, DelInstrs, InstrIdxForVirtReg);
     break;
   default:
     // Reassociate default patterns.
     TargetInstrInfo::genAlternativeCodeSequence(Root, Pattern, InsInstrs,
                                                 DelInstrs, InstrIdxForVirtReg);
     break;
   }
 }
 
-// Currently, only handle two patterns REASSOC_XY_AMM_BMM and
-// REASSOC_XMM_AMM_BMM. See comments for getFMAPatterns.
 void PPCInstrInfo::reassociateFMA(
     MachineInstr &Root, MachineCombinerPattern Pattern,
     SmallVectorImpl<MachineInstr *> &InsInstrs,
     SmallVectorImpl<MachineInstr *> &DelInstrs,
     DenseMap<unsigned, unsigned> &InstrIdxForVirtReg) const {
   MachineFunction *MF = Root.getMF();
   MachineRegisterInfo &MRI = MF->getRegInfo();
+  const TargetRegisterInfo *TRI = &getRegisterInfo();
   MachineOperand &OpC = Root.getOperand(0);
   Register RegC = OpC.getReg();
   const TargetRegisterClass *RC = MRI.getRegClass(RegC);
   MRI.constrainRegClass(RegC, RC);
 
   unsigned FmaOp = Root.getOpcode();
   int16_t Idx = getFMAOpIdxInfo(FmaOp);
   assert(Idx >= 0 && "Root must be a FMA instruction");
 
+  bool IsILPReassociate =
+      (Pattern == MachineCombinerPattern::REASSOC_XY_AMM_BMM) ||
+      (Pattern == MachineCombinerPattern::REASSOC_XMM_AMM_BMM);
+
   uint16_t AddOpIdx = FMAOpIdxInfo[Idx][InfoArrayIdxAddOpIdx];
   uint16_t FirstMulOpIdx = FMAOpIdxInfo[Idx][InfoArrayIdxMULOpIdx];
-  MachineInstr *Prev = MRI.getUniqueVRegDef(Root.getOperand(AddOpIdx).getReg());
-  MachineInstr *Leaf =
-      MRI.getUniqueVRegDef(Prev->getOperand(AddOpIdx).getReg());
-  uint16_t IntersectedFlags =
-      Root.getFlags() & Prev->getFlags() & Leaf->getFlags();
+
+  MachineInstr *Prev = nullptr;
+  MachineInstr *Leaf = nullptr;
+  if (IsILPReassociate) {
+    Prev = MRI.getUniqueVRegDef(Root.getOperand(AddOpIdx).getReg());
+    Leaf = MRI.getUniqueVRegDef(Prev->getOperand(AddOpIdx).getReg());
+  } else {
+    if (Pattern == MachineCombinerPattern::REASSOC_FMA_M2C_FSUB) {
+      Register MULReg =
+          TRI->lookThruCopyLike(Root.getOperand(FirstMulOpIdx).getReg(), &MRI);
+      Leaf = MRI.getVRegDef(MULReg);
+    } else if (Pattern == MachineCombinerPattern::REASSOC_FMA_M1C_FSUB) {
+      Register MULReg = TRI->lookThruCopyLike(
+          Root.getOperand(FirstMulOpIdx + 1).getReg(), &MRI);
+      Leaf = MRI.getVRegDef(MULReg);
+    } else
+      llvm_unreachable("not recognized pattern!");
+  }
+
+  uint16_t IntersectedFlags;
+  if (IsILPReassociate)
+    IntersectedFlags = Root.getFlags() & Prev->getFlags() & Leaf->getFlags();
+  else
+    IntersectedFlags = Root.getFlags() & Leaf->getFlags();
 
   auto GetOperandInfo = [&](const MachineOperand &Operand, Register &Reg,
                             bool &KillFlag) {
     Reg = Operand.getReg();
     MRI.constrainRegClass(Reg, RC);
     KillFlag = Operand.isKill();
   };
 
   auto GetFMAInstrInfo = [&](const MachineInstr &Instr, Register &MulOp1,
-                             Register &MulOp2, bool &MulOp1KillFlag,
-                             bool &MulOp2KillFlag) {
+                             Register &MulOp2, Register &AddOp,
+                             bool &MulOp1KillFlag, bool &MulOp2KillFlag,
+                             bool &AddOpKillFlag) {
     GetOperandInfo(Instr.getOperand(FirstMulOpIdx), MulOp1, MulOp1KillFlag);
     GetOperandInfo(Instr.getOperand(FirstMulOpIdx + 1), MulOp2, MulOp2KillFlag);
+    GetOperandInfo(Instr.getOperand(AddOpIdx), AddOp, AddOpKillFlag);
   };
 
-  Register RegM11, RegM12, RegX, RegY, RegM21, RegM22, RegM31, RegM32;
+  Register RegM11, RegM12, RegX, RegY, RegM21, RegM22, RegM31, RegM32, RegA11,
+      RegA21, RegA31;
   bool KillX = false, KillY = false, KillM11 = false, KillM12 = false,
-       KillM21 = false, KillM22 = false, KillM31 = false, KillM32 = false;
+       KillM21 = false, KillM22 = false, KillM31 = false, KillM32 = false,
+       KillA11 = false, KillA21 = false, KillA31 = false;
+
+  GetFMAInstrInfo(Root, RegM31, RegM32, RegA31, KillM31, KillM32, KillA31);
 
-  GetFMAInstrInfo(Root, RegM31, RegM32, KillM31, KillM32);
-  GetFMAInstrInfo(*Prev, RegM21, RegM22, KillM21, KillM22);
+  if (IsILPReassociate)
+    GetFMAInstrInfo(*Prev, RegM21, RegM22, RegA21, KillM21, KillM22, KillA21);
 
   if (Pattern == MachineCombinerPattern::REASSOC_XMM_AMM_BMM) {
-    GetFMAInstrInfo(*Leaf, RegM11, RegM12, KillM11, KillM12);
+    GetFMAInstrInfo(*Leaf, RegM11, RegM12, RegA11, KillM11, KillM12, KillA11);
     GetOperandInfo(Leaf->getOperand(AddOpIdx), RegX, KillX);
   } else if (Pattern == MachineCombinerPattern::REASSOC_XY_AMM_BMM) {
     GetOperandInfo(Leaf->getOperand(1), RegX, KillX);
     GetOperandInfo(Leaf->getOperand(2), RegY, KillY);
+  } else {
+    // Get FSUB instruction info.
+    GetOperandInfo(Leaf->getOperand(1), RegX, KillX);
+    GetOperandInfo(Leaf->getOperand(2), RegY, KillY);
   }
 
   // Create new virtual registers for the new results instead of
   // recycling legacy ones because the MachineCombiner's computation of the
   // critical path requires a new register definition rather than an existing
   // one.
+  // For register pressure reassociation, we only need create one virtual
+  // register for the new fma.
   Register NewVRA = MRI.createVirtualRegister(RC);
   InstrIdxForVirtReg.insert(std::make_pair(NewVRA, 0));
 
-  Register NewVRB = MRI.createVirtualRegister(RC);
+  Register NewVRB = 0;
+  if (IsILPReassociate) {
+    NewVRB = MRI.createVirtualRegister(RC);
     InstrIdxForVirtReg.insert(std::make_pair(NewVRB, 1));
+  }
 
   Register NewVRD = 0;
   if (Pattern == MachineCombinerPattern::REASSOC_XMM_AMM_BMM) {
     NewVRD = MRI.createVirtualRegister(RC);
     InstrIdxForVirtReg.insert(std::make_pair(NewVRD, 2));
   }
 
   auto AdjustOperandOrder = [&](MachineInstr *MI, Register RegAdd, bool KillAdd,
                                 Register RegMul1, bool KillRegMul1,
                                 Register RegMul2, bool KillRegMul2) {
     MI->getOperand(AddOpIdx).setReg(RegAdd);
     MI->getOperand(AddOpIdx).setIsKill(KillAdd);
     MI->getOperand(FirstMulOpIdx).setReg(RegMul1);
     MI->getOperand(FirstMulOpIdx).setIsKill(KillRegMul1);
     MI->getOperand(FirstMulOpIdx + 1).setReg(RegMul2);
     MI->getOperand(FirstMulOpIdx + 1).setIsKill(KillRegMul2);
   };
 
+  MachineInstrBuilder NewARegPressure, NewCRegPressure;
+
   if (Pattern == MachineCombinerPattern::REASSOC_XY_AMM_BMM) {
     // Create new instructions for insertion.
     MachineInstrBuilder MINewB =
         BuildMI(*MF, Prev->getDebugLoc(), get(FmaOp), NewVRB)
             .addReg(RegX, getKillRegState(KillX))
             .addReg(RegM21, getKillRegState(KillM21))
             .addReg(RegM22, getKillRegState(KillM22));
     MachineInstrBuilder MINewA =
         BuildMI(*MF, Root.getDebugLoc(), get(FmaOp), NewVRA)
if (Pattern == MachineCombinerPattern::REASSOC_XY_AMM_BMM) {
     setSpecialOperandAttr(*MINewD, IntersectedFlags);
     setSpecialOperandAttr(*MINewC, IntersectedFlags);
 
     // Record new instructions for insertion.
     InsInstrs.push_back(MINewA);
     InsInstrs.push_back(MINewB);
     InsInstrs.push_back(MINewD);
     InsInstrs.push_back(MINewC);
+  } else {
+    Register VarReg;
+    bool KillVarReg = false;
+    assert((Pattern == MachineCombinerPattern::REASSOC_FMA_M1C_FSUB ||
+            Pattern == MachineCombinerPattern::REASSOC_FMA_M2C_FSUB) &&
+           "Not recognize pattern!");
+    if (Pattern == MachineCombinerPattern::REASSOC_FMA_M1C_FSUB) {
+      VarReg = RegM31;
+      KillVarReg = KillM31;
+    } else {
+      VarReg = RegM32;
+      KillVarReg = KillM32;
+    }
+    // We don't want to get negative const from memory pool too early, as the
+    // created entry will not be deleted even if it has no users. Since all
+    // operand of Leaf and Root are virtual register, we use zero register
+    // here as a placeholder. When the InsInstrs is selected in
+    // MachineCombiner, we call finalizeInsInstrs to replace the zero register
+    // with a virtual register which is a load from constant pool.
+    NewARegPressure = BuildMI(*MF, Root.getDebugLoc(), get(FmaOp), NewVRA)
+                          .addReg(RegA31, getKillRegState(RegA31))
+                          .addReg(RegY, getKillRegState(KillY))
+                          .addReg(PPC::ZERO8);
+    NewCRegPressure = BuildMI(*MF, Root.getDebugLoc(), get(FmaOp), RegC)
+                          .addReg(NewVRA, getKillRegState(true))
+                          .addReg(RegX, getKillRegState(KillX))
+                          .addReg(VarReg, getKillRegState(KillVarReg));
+    // For now, we only support xsmaddadp/xsmaddasp, their add operand are
+    // both at index 1, no need to adjust.
+    // FIXME: when add more fma instructions support, like fma/fmas, adjust
+    // the operand index here.
+  }
+
+  if (!IsILPReassociate) {
+    setSpecialOperandAttr(*NewARegPressure, IntersectedFlags);
+    setSpecialOperandAttr(*NewCRegPressure, IntersectedFlags);
+
+    InsInstrs.push_back(NewARegPressure);
+    InsInstrs.push_back(NewCRegPressure);
   }
 
   assert(!InsInstrs.empty() &&
          "Insertion instructions set should not be empty!");
 
   // Record old instructions for deletion.
   DelInstrs.push_back(Leaf);
+  if (IsILPReassociate)
     DelInstrs.push_back(Prev);
   DelInstrs.push_back(&Root);
 }
 
 // Detect 32 -> 64-bit extensions where we may reuse the low sub-register.
 bool PPCInstrInfo::isCoalescableExtInstr(const MachineInstr &MI,
                                          Register &SrcReg, Register &DstReg,
                                          unsigned &SubIdx) const {
   switch (MI.getOpcode()) {

llvm/test/CodeGen/PowerPC/register-pressure-reduction.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -verify-machineinstrs -ppc-asm-full-reg-names -O3 < %s \
+; RUN:   -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 | FileCheck %s
+; RUN: llc -verify-machineinstrs -ppc-asm-full-reg-names -O3 < %s \
+; RUN:   -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 | FileCheck %s --check-prefix=CHECK-P8
+; RUN: llc -verify-machineinstrs -ppc-asm-full-reg-names -reg-pressure-threshold=0.0 -O3 < %s \
+; RUN:   -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr9 | FileCheck %s --check-prefix=CHECK-FMA
+
+define float @foo_float(float %0, float %1, float %2, float %3) {
+; CHECK-LABEL: foo_float:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    addis r3, r2, .LCPI0_0@toc@ha
+; CHECK-NEXT:    xsmulsp f1, f2, f1
+; CHECK-NEXT:    xssubsp f0, f3, f4
+; CHECK-NEXT:    lfs f2, .LCPI0_0@toc@l(r3)
+; CHECK-NEXT:    xsmaddasp f1, f0, f2
+; CHECK-NEXT:    blr
+;
+; CHECK-P8-LABEL: foo_float:
+; CHECK-P8:       # %bb.0:
+; CHECK-P8-NEXT:    xsmulsp f1, f2, f1
+; CHECK-P8-NEXT:    addis r3, r2, .LCPI0_0@toc@ha
+; CHECK-P8-NEXT:    xssubsp f0, f3, f4
+; CHECK-P8-NEXT:    lfs f2, .LCPI0_0@toc@l(r3)
+; CHECK-P8-NEXT:    xsmaddasp f1, f0, f2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-FMA-LABEL: foo_float:
+; CHECK-FMA:       # %bb.0:
+; CHECK-FMA-NEXT:    addis r3, r2, .LCPI0_0@toc@ha
+; CHECK-FMA-NEXT:    xsmulsp f1, f2, f1
+; CHECK-FMA-NEXT:    lfs f0, .LCPI0_0@toc@l(r3)
+; CHECK-FMA-NEXT:    addis r3, r2, .LCPI0_1@toc@ha
+; CHECK-FMA-NEXT:    lfs f2, .LCPI0_1@toc@l(r3)
+; CHECK-FMA-NEXT:    xsmaddasp f1, f4, f2
+; CHECK-FMA-NEXT:    xsmaddasp f1, f3, f0
+; CHECK-FMA-NEXT:    blr
+  %5 = fmul reassoc nsz float %1, %0
+  %6 = fsub reassoc nsz float %2, %3
+  %7 = fmul reassoc nsz float %6, 0x3DB2533FE0000000
+  %8 = fadd reassoc nsz float %7, %5
+  ret float %8
+}
+
+define double @foo_double(double %0, double %1, double %2, double %3) {
+; CHECK-LABEL: foo_double:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    xsmuldp f1, f2, f1
+; CHECK-NEXT:    xssubdp f0, f3, f4
+; CHECK-NEXT:    addis r3, r2, .LCPI1_0@toc@ha
+; CHECK-NEXT:    lfd f2, .LCPI1_0@toc@l(r3)
+; CHECK-NEXT:    xsmaddadp f1, f0, f2
+; CHECK-NEXT:    blr
+;
+; CHECK-P8-LABEL: foo_double:
+; CHECK-P8:       # %bb.0:
+; CHECK-P8-NEXT:    xsmuldp f1, f2, f1
+; CHECK-P8-NEXT:    addis r3, r2, .LCPI1_0@toc@ha
+; CHECK-P8-NEXT:    xssubdp f0, f3, f4
+; CHECK-P8-NEXT:    lfd f2, .LCPI1_0@toc@l(r3)
+; CHECK-P8-NEXT:    xsmaddadp f1, f0, f2
+; CHECK-P8-NEXT:    blr
+;
+; CHECK-FMA-LABEL: foo_double:
+; CHECK-FMA:       # %bb.0:
+; CHECK-FMA-NEXT:    addis r3, r2, .LCPI1_0@toc@ha
+; CHECK-FMA-NEXT:    xsmuldp f1, f2, f1
+; CHECK-FMA-NEXT:    lfd f0, .LCPI1_0@toc@l(r3)
+; CHECK-FMA-NEXT:    addis r3, r2, .LCPI1_1@toc@ha
+; CHECK-FMA-NEXT:    lfd f2, .LCPI1_1@toc@l(r3)
+; CHECK-FMA-NEXT:    xsmaddadp f1, f4, f2
+; CHECK-FMA-NEXT:    xsmaddadp f1, f3, f0
+; CHECK-FMA-NEXT:    blr
+  %5 = fmul reassoc nsz double %1, %0
+  %6 = fsub reassoc nsz double %2, %3
+  %7 = fmul reassoc nsz double %6, 0x3DB2533FE68CADDE
+  %8 = fadd reassoc nsz double %7, %5
+  ret double %8
+}
+