[PowerPC] Require NSZ flag for c-a*b to FNMSUB

[qiucf]

Summary

On PowerPC, FNMSUB (including VSX and non-VSX version) means -(a*b-c). But the backend now generates these instructions regardless whether nsz flag exists or not.

If a*b-c==0, such transformation changes sign of zero. This patch fixes this issue.

[qiucf]

llvm/test/CodeGen/PowerPC/fma-negate.ll

290

This should be a regression. And the reason is SplitVectorRes did not pass SDNode flags to legalized ones. If that's truly a bug, I will cook a patch for it.

[spatel]

llvm/test/CodeGen/PowerPC/fma-negate.ll

290

This should not be a regression, right? If we dropped the flags during legalization, that is a bug.

[qiucf]

llvm/test/CodeGen/PowerPC/fma-negate.ll

290

I posted the fix at D76832 :)

[spatel]

llvm/test/CodeGen/PowerPC/fma-negate.ll

290

Thanks! Please rebase this patch after that the other one is pushed.
PPC experts can judge whether we want to proceed with this one or wait for possible improvements in getNegatibleCost().
We have known bugs with that logic (D76638, D76439), but I'm not sure yet what the best solution is. @qshanz may have a better idea.

[jsji]

@steven.zhang Can you please *Disable* or *Delete* the deprecated account @qshanz to avoid confusion. Thanks.

[qiucf]

Rebase to reflect changes after D76832

[amyk]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

1485

Make the return statement inline with the rest of the return statements.

15654

This could be a silly question but I'm curious and wanting to learn the transformation you implemented. When I read the comment "choose the cheaper one to negate," it looks like in the first condition that N1 is the cheaper one but N0 is being negated. Could you elaborate on this a little more?

16008

Maybe good to add a comment to this function to briefly describe the transformation you are trying to do.

16020

Did you mean, we don't need to care about fnmadd?

16021

s/transform/transformation

[steven.zhang]

In D76585#1943627, @jsji wrote:

@steven.zhang Can you please *Disable* or *Delete* the deprecated account @qshanz to avoid confusion. Thanks.

I have updated the profile. Thank you for reminder.

[qiucf]

Address comments

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

1485

Thanks. clang-format always changes this line...

15654

Yes. It's a little bit confusing since NegatibleCost defines Expensive=0; Neutral=1; Cheaper=2. And code in DAG combiner also follows such convention.

16020

I think at least for now we don't really need fnmadd since dag combiner already handles it ((fneg (fma ...))). But if we drop code for fnmsub here, we would get worse code sequence.

[qiucf]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

15654

This should be fixed after D77993.

[qiucf]

Rebase after the getNegatibleCost refactoring patch: D77319

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

15617

You should check the value of NegN2 instead of the cost though their semantics might be the same.

15633

Please refer to how we did for FMA. i.e. Your implementation has problems when A=neutral, B=neutral, and C=cheaper

[qiucf]

Address comments from Steven

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

15633

Thanks, I was not aware of (A=neutral && B=neutral && C=cheaper) => cheaper.

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

15630

As we already know that, N0Cost <= N1Cost, so, what we need to do is as:
Cost = std::min(N0Cost, N2Cost);
I will update the logic in DAGCombine there as it has similar issue.

[qiucf]

Simplify a cost calculation

[RKSimon]

This will need refactoring after D77319

[steven.zhang]

@spatel @RKSimon Do you know that why we didn't have the generic node for FNMSUB/FMSUB as I see the duplicate logic between X86 and PowerPC ?

[spatel]

In D76585#2009259, @steven.zhang wrote:

@spatel @RKSimon Do you know that why we didn't have the generic node for FNMSUB/FMSUB as I see the duplicate logic between X86 and PowerPC ?

I can't say for sure, but my guess is that target-specific logic could be added without incurring the overhead of general discussion/approval:
rG3cb3b0045c62bc6cc

There's also a question of deciding what name corresponds to each math operation. The x86 instructions don't match PPC. For example:
[PPC] fnmsub = -(A * B - C)
[x86] vfnmsub132sd = -(A * B) - C

It would be nice to have common nodes/optimizations if we can find enough code to share. If we do that, we also need to see if other targets can benefit (AArch64 has fnmsub machine IR).

[steven.zhang]

In D76585#2009908, @spatel wrote:

In D76585#2009259, @steven.zhang wrote:

@spatel @RKSimon Do you know that why we didn't have the generic node for FNMSUB/FMSUB as I see the duplicate logic between X86 and PowerPC ?

I can't say for sure, but my guess is that target-specific logic could be added without incurring the overhead of general discussion/approval:
rG3cb3b0045c62bc6cc

There's also a question of deciding what name corresponds to each math operation. The x86 instructions don't match PPC. For example:
[PPC] fnmsub = -(A * B - C)
[x86] vfnmsub132sd = -(A * B) - C

It would be nice to have common nodes/optimizations if we can find enough code to share. If we do that, we also need to see if other targets can benefit (AArch64 has fnmsub machine IR).

Thank you for the information. Maybe,we can add fmsub(A*B-C), fnmsub(-A*B-C), fnmadd(-A*B+C). But yes, we need to see if other targets can benefit from this.

[qiucf]

Fix test failure after several td/case changes

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

16242

Can we move all these logic into getNegatedExpression so that more opportunity will be found？

[qiucf]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

16242

Removing main part of these logic into getNegatedExpression makes code cleaner. But FNMSUB nodes will never try negate unless we combine it?

[steven.zhang]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

16242

ok. I think, we still need to combine it as FNMSUB as what you did now to reduce FNEG. But we miss to do the negation between FMA and FNMSUB. But it could be a follow up patch.

16252

Do we need this transformation ? I didn't see the benefit of doing this as it didn't reduces the FNEG.

[qiucf]

Address Steven's comments. Fixed a crash due to type legalizing.

[steven.zhang]

LGTM now overall with some minor comments. As you are adding the pattern for VNMSUBFP, I would suggest you do it in another patch.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

15818

Remove the InvertedOpc but use FMA directly as we know that it is FNMSUB now. And move the invertFMAOpcode as lambda of the getNegatedExpression.

15833

It will be great if you can add more wording on how it changes the sign of zeroes here.

[qiucf]

Move vnmsubfp exploitation to another patch

[steven.zhang]

LGTM now as far as you update the comments in getNegatedExpression about why we are changing the sign of zero. But please hold on several days in case others have more comments,

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
     /// are used, where n is log2(sizeof(element) * 8). See sld/slw, etc.
     /// for exact behaviors.
     ///
     /// For vector types, only the last n bits are used. See vsld.
     SRL,
     SRA,
     SHL,
 
+    /// FNMSUB - Negated multiply-subtract instruction.
+    FNMSUB,
+
     /// EXTSWSLI = The PPC extswsli instruction, which does an extend-sign
     /// word and shift left immediate.
     EXTSWSLI,
 
     /// The combination of sra[wd]i and addze used to implemented signed
     /// integer division by a power of 2. The first operand is the dividend,
     /// and the second is the constant shift amount (representing the
     /// divisor).
public:
     }
 
     void initializeSplitCSR(MachineBasicBlock *Entry) const override;
 
     void insertCopiesSplitCSR(
       MachineBasicBlock *Entry,
       const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
 
+    NegatibleCost getNegatibleCost(SDValue Op, SelectionDAG &DAG,
+                                   bool LegalOperations, bool ForCodeSize,
+                                   unsigned Depth = 0) const override;
+
+    SDValue getNegatedExpression(SDValue Op, SelectionDAG &DAG,
+                                 bool LegalOperations, bool ForCodeSize,
+                                 unsigned Depth = 0) const override;
+
     /// getSetCCResultType - Return the ISD::SETCC ValueType
     EVT getSetCCResultType(const DataLayout &DL, LLVMContext &Context,
                            EVT VT) const override;
 
     /// Return true if target always beneficiates from combining into FMA for a
     /// given value type. This must typically return false on targets where FMA
     /// takes more cycles to execute than FADD.
     bool enableAggressiveFMAFusion(EVT VT) const override;
private:
     SDValue DAGCombineTruncBoolExt(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineStoreFPToInt(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineFPToIntToFP(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSHL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSRA(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSRL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineMUL(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineADD(SDNode *N, DAGCombinerInfo &DCI) const;
+    SDValue combineFMALike(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineTRUNCATE(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineSetCC(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineABS(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVSelect(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineVReverseMemOP(ShuffleVectorSDNode *SVN, LSBaseSDNode *LSBase,
                                  DAGCombinerInfo &DCI) const;
 
     /// ConvertSETCCToSubtract - looks at SETCC that compares ints. It replaces

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

   setStackPointerRegisterToSaveRestore(isPPC64 ? PPC::X1 : PPC::R1);
 
   // We have target-specific dag combine patterns for the following nodes:
   setTargetDAGCombine(ISD::ADD);
   setTargetDAGCombine(ISD::SHL);
   setTargetDAGCombine(ISD::SRA);
   setTargetDAGCombine(ISD::SRL);
   setTargetDAGCombine(ISD::MUL);
+  setTargetDAGCombine(ISD::FMA);
   setTargetDAGCombine(ISD::SINT_TO_FP);
   setTargetDAGCombine(ISD::BUILD_VECTOR);
   if (Subtarget.hasFPCVT())
     setTargetDAGCombine(ISD::UINT_TO_FP);
   setTargetDAGCombine(ISD::LOAD);
   setTargetDAGCombine(ISD::STORE);
   setTargetDAGCombine(ISD::BR_CC);
   if (Subtarget.useCRBits())
const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::QVLFSb:          return "PPCISD::QVLFSb";
   case PPCISD::BUILD_FP128:     return "PPCISD::BUILD_FP128";
   case PPCISD::BUILD_SPE64:     return "PPCISD::BUILD_SPE64";
   case PPCISD::EXTRACT_SPE:     return "PPCISD::EXTRACT_SPE";
   case PPCISD::EXTSWSLI:        return "PPCISD::EXTSWSLI";
   case PPCISD::LD_VSX_LH:       return "PPCISD::LD_VSX_LH";
   case PPCISD::FP_EXTEND_HALF:  return "PPCISD::FP_EXTEND_HALF";
   case PPCISD::LD_SPLAT:        return "PPCISD::LD_SPLAT";
+  case PPCISD::FNMSUB:          return "PPCISD::FNMSUB";
   }
   return nullptr;
 }
 
 EVT PPCTargetLowering::getSetCCResultType(const DataLayout &DL, LLVMContext &C,
                                           EVT VT) const {
   if (!VT.isVector())
     return Subtarget.useCRBits() ? MVT::i1 : MVT::i32;
 
SDValue PPCTargetLowering::PerformDAGCombine(SDNode *N,
   case ISD::SHL:
     return combineSHL(N, DCI);
   case ISD::SRA:
     return combineSRA(N, DCI);
   case ISD::SRL:
     return combineSRL(N, DCI);
   case ISD::MUL:
     return combineMUL(N, DCI);
+  case ISD::FMA:
+  case PPCISD::FNMSUB:
+    return combineFMALike(N, DCI);
   case PPCISD::SHL:
     if (isNullConstant(N->getOperand(0))) // 0 << V -> 0.
         return N->getOperand(0);
     break;
   case PPCISD::SRL:
     if (isNullConstant(N->getOperand(0))) // 0 >>u V -> 0.
         return N->getOperand(0);
     break;
for (const MCPhysReg *I = IStart; *I; ++I) {
     // Insert the copy-back instructions right before the terminator.
     for (auto *Exit : Exits)
       BuildMI(*Exit, Exit->getFirstTerminator(), DebugLoc(),
               TII->get(TargetOpcode::COPY), *I)
         .addReg(NewVR);
   }
 }
 
+TargetLowering::NegatibleCost
+PPCTargetLowering::getNegatibleCost(SDValue Op, SelectionDAG &DAG,
+                                    bool LegalOperations, bool ForCodeSize,
+                                    unsigned Depth) const {
+  unsigned Opc = Op.getOpcode();
+  EVT VT = Op.getValueType();
+
+  // All negation to FMA in PPC should be at least neutral, but keeping it
+  // simpler would help target independent combiner to do more foldings.
+  switch (Opc) {
+  case PPCISD::FNMSUB:
+    if (!Op.hasOneUse() || !isTypeLegal(VT))
+      break;
+
+    NegatibleCost N2Cost = getNegatibleCost(
+        Op.getOperand(2), DAG, LegalOperations, ForCodeSize, Depth + 1);
+    // The last operand will always be negated, so its cost matters.
+    if (N2Cost == NegatibleCost::Expensive)
+      return NegatibleCost::Expensive;
+
+    NegatibleCost N0Cost = getNegatibleCost(
+        Op.getOperand(0), DAG, LegalOperations, ForCodeSize, Depth + 1);
+    NegatibleCost N1Cost = getNegatibleCost(
+        Op.getOperand(1), DAG, LegalOperations, ForCodeSize, Depth + 1);
+
+    return std::max(N0Cost, N1Cost);
+  }
+
+  return TargetLowering::getNegatibleCost(Op, DAG, LegalOperations, ForCodeSize,
+                                          Depth);
+}
+
+SDValue PPCTargetLowering::getNegatedExpression(SDValue Op, SelectionDAG &DAG,
+                                                bool LegalOperations,
+                                                bool ForCodeSize,
+                                                unsigned Depth) const {
+  unsigned Opc = Op.getOpcode();
+  EVT VT = Op.getValueType();
+  SDNodeFlags Flags = Op.getNode()->getFlags();
+
+  // Base class already defined negation for FMA
+  switch (Opc) {
+  case PPCISD::FNMSUB:
+    if (!Op.hasOneUse() || !isTypeLegal(VT))
+      break;
+
+    SDValue N0 = Op.getOperand(0);
+    SDValue N1 = Op.getOperand(1);
+
+    // (fneg (fnmsub a b c)) => (fnmsub (fneg a) b (fneg c))
+    // (fneg (fnmsub a b c)) => (fnmsub a (fneg b) (fneg c))
+    SDValue NegN2 = getNegatedExpression(Op.getOperand(2), DAG, LegalOperations,
+                                         ForCodeSize, Depth + 1);
+    NegatibleCost N0Cost =
+        getNegatibleCost(N0, DAG, LegalOperations, ForCodeSize, Depth + 1);
+    NegatibleCost N1Cost =
+        getNegatibleCost(N1, DAG, LegalOperations, ForCodeSize, Depth + 1);
+
+    // Choose the cheaper one to negate.
+    if (N0Cost > N1Cost) {
+      SDValue NegN0 = getNegatedExpression(N0, DAG, LegalOperations,
+                                           ForCodeSize, Depth + 1);
+      return DAG.getNode(Opc, SDLoc(Op), VT, NegN0, N1, NegN2, Flags);
+    } else {
+      SDValue NegN1 = getNegatedExpression(N1, DAG, LegalOperations,
+                                           ForCodeSize, Depth + 1);
+      return DAG.getNode(Opc, SDLoc(Op), VT, N0, NegN1, NegN2, Flags);
+    }
+  }
+
+  return TargetLowering::getNegatedExpression(Op, DAG, LegalOperations,
+                                              ForCodeSize, Depth);
+}
+
 // Override to enable LOAD_STACK_GUARD lowering on Linux.
 bool PPCTargetLowering::useLoadStackGuardNode() const {
   if (!Subtarget.isTargetLinux())
     return TargetLowering::useLoadStackGuardNode();
   return true;
 }
 
 // Override to disable global variable loading on Linux.
static SDValue combineADDToADDZE(SDNode *N, SelectionDAG &DAG,
   bool LHSHasPattern = isZextOfCompareWithConstant(LHS);
   bool RHSHasPattern = isZextOfCompareWithConstant(RHS);
 
   // If there is a pattern, canonicalize a zext operand to the RHS.
   if (LHSHasPattern && !RHSHasPattern)
     std::swap(LHS, RHS);
   else if (!LHSHasPattern && !RHSHasPattern)
     return SDValue();
 
   SDLoc DL(N);
   SDVTList VTs = DAG.getVTList(MVT::i64, MVT::Glue);
   SDValue Cmp = RHS.getOperand(0);
   SDValue Z = Cmp.getOperand(0);
   auto *Constant = dyn_cast<ConstantSDNode>(Cmp.getOperand(1));
 
   assert(Constant && "Constant Should not be a null pointer.");
   int64_t NegConstant = 0 - Constant->getSExtValue();
 
   switch(cast<CondCodeSDNode>(Cmp.getOperand(2))->get()) {
   default: break;
   case ISD::SETNE: {
     //                                 when C == 0
     //                             --> addze X, (addic Z, -1).carry
     //                            /
     // add X, (zext(setne Z, C))--
     //                            \    when -32768 <= -C <= 32767 && C != 0
     //                             --> addze X, (addic (addi Z, -C), -1).carry
     SDValue Add = DAG.getNode(ISD::ADD, DL, MVT::i64, Z,
                               DAG.getConstant(NegConstant, DL, MVT::i64));
     SDValue AddOrZ = NegConstant != 0 ? Add : Z;
     SDValue Addc = DAG.getNode(ISD::ADDC, DL, DAG.getVTList(MVT::i64, MVT::Glue),
                                AddOrZ, DAG.getConstant(-1ULL, DL, MVT::i64));
if ((MulAmtAbs - 1).isPowerOf2()) {
     else
       return DAG.getNode(ISD::SUB, DL, VT, Op0, Op1);
 
   } else {
     return SDValue();
   }
 }
 
+// Combine fma-like op (like fnmsub) with fnegs to appropriate op. Do this
+// in combiner since we need to check SD flags and other subtarget features.
+SDValue PPCTargetLowering::combineFMALike(SDNode *N,
+                                          DAGCombinerInfo &DCI) const {
+  SDValue N0 = N->getOperand(0);
+  SDValue N1 = N->getOperand(1);
+  SDValue N2 = N->getOperand(2);
+  SDNodeFlags Flags = N->getFlags();
+  EVT VT = N->getValueType(0);
+  SelectionDAG &DAG = DCI.DAG;
+  const TargetOptions &Options = getTargetMachine().Options;
+  unsigned Opc = N->getOpcode();
+
+  // Check nsz flag before generating fnmsub instructions, getNegatibleCost
+  // already checked before negating FMA, so we don't to care about fnmadd.
+  // FNMSUB means (fneg (fma a b (fneg c))), allowing this transformation may
+  // change sign of zero when a*b-c=0 since (fnmsub a b c)=-0 while c-a*b=+0.
+  if (!Flags.hasNoSignedZeros() && !Options.NoSignedZerosFPMath)
+    return SDValue();
+
+  // Vector version can't be selected if VSX disabled.
+  if (VT.isVector() && !Subtarget.hasVSX())
+    return SDValue();
+
+  bool CodeSize = DAG.getMachineFunction().getFunction().hasOptSize();
+  bool LegalOperations = !DCI.isBeforeLegalizeOps();
+  SDLoc Loc(N);
+
+  // Try 'invert' this op to reduce negs. They're symmetric.
+  auto InvertOpCode = [](unsigned Opc) -> unsigned {
+    if (Opc == ISD::FMA)
+      return PPCISD::FNMSUB;
+    else if (Opc == PPCISD::FNMSUB)
+      return ISD::FMA;
+    else
+      llvm_unreachable("Unexpected FMA opcode");
+  };
+
+  // (fma (fneg a) b c) => (fnmsub a b c)
+  // (fnmsub (fneg a) b c) => (fma a b c)
+  if (getNegatibleCost(N0, DAG, LegalOperations, CodeSize) ==
+      NegatibleCost::Cheaper)
+    return DAG.getNode(InvertOpCode(Opc), Loc, VT,
+                       getNegatedExpression(N0, DAG, LegalOperations, CodeSize),
+                       N1, N2, Flags);
+
+  // (fma a (fneg b) c) => (fnmsub a b c)
+  // (fnmsub a (fneg b) c) => (fma a b c)
+  if (getNegatibleCost(N1, DAG, LegalOperations, CodeSize) ==
+      NegatibleCost::Cheaper)
+    return DAG.getNode(InvertOpCode(Opc), Loc, VT, N0,
+                       getNegatedExpression(N1, DAG, LegalOperations, CodeSize),
+                       N2, Flags);
+
+  // (fma a b (fneg c)) => (fneg (fnmsub a b c))
+  // (fnmsub a b (fneg c)) => (fneg (fma a b c))
+  if (getNegatibleCost(N2, DAG, LegalOperations, CodeSize) ==
+      NegatibleCost::Cheaper)
+    return DAG.getNode(
+        ISD::FNEG, Loc, VT,
+        DAG.getNode(InvertOpCode(Opc), Loc, VT, N0, N1,
+                    getNegatedExpression(N2, DAG, LegalOperations, CodeSize),
+                    Flags),
+        Flags);
+
+  return SDValue();
+}
+
 bool PPCTargetLowering::mayBeEmittedAsTailCall(const CallInst *CI) const {
   // Only duplicate to increase tail-calls for the 64bit SysV ABIs.
   if (!Subtarget.is64BitELFABI())
     return false;
 
   // If not a tail call then no need to proceed.
   if (!CI->isTailCall())
     return false;
if (TrueOpnd.getOperand(0) == CmpOpnd1 &&
       FalseOpnd.getOperand(0) == CmpOpnd2 &&
       FalseOpnd.getOperand(1) == CmpOpnd1) {
     return DAG.getNode(PPCISD::VABSD, dl, N->getOperand(1).getValueType(),
                        CmpOpnd1, CmpOpnd2,
                        DAG.getTargetConstant(0, dl, MVT::i32));
   }
 
   return SDValue();
 }

llvm/lib/Target/PowerPC/PPCInstrInfo.td

 def PPCcmpb     : SDNode<"PPCISD::CMPB", SDTIntBinOp, []>;
 
 // These nodes represent the 32-bit PPC shifts that operate on 6-bit shift
 // amounts.  These nodes are generated by the multi-precision shift code.
 def PPCsrl        : SDNode<"PPCISD::SRL"       , SDTIntShiftOp>;
 def PPCsra        : SDNode<"PPCISD::SRA"       , SDTIntShiftOp>;
 def PPCshl        : SDNode<"PPCISD::SHL"       , SDTIntShiftOp>;
 
+def PPCfnmsub     : SDNode<"PPCISD::FNMSUB"    , SDTFPTernaryOp>;
+
 def PPCextswsli : SDNode<"PPCISD::EXTSWSLI" , SDT_PPCextswsli>;
 
 // Move 2 i64 values into a VSX register
 def PPCbuild_fp128: SDNode<"PPCISD::BUILD_FP128",
                            SDTypeProfile<1, 2,
                              [SDTCisFP<0>, SDTCisSameSizeAs<1,2>,
                               SDTCisSameAs<1,2>]>,
                            []>;

 // The rule for seq_cst is duplicated to work with both 64 bits and 32 bits
 // versions of Power.
 def : Pat<(atomic_fence (i64 7), (timm)), (SYNC 0)>, Requires<[HasSYNC]>;
 def : Pat<(atomic_fence (i32 7), (timm)), (SYNC 0)>, Requires<[HasSYNC]>;
 def : Pat<(atomic_fence (timm), (timm)), (SYNC 1)>, Requires<[HasSYNC]>;
 def : Pat<(atomic_fence (timm), (timm)), (MSYNC)>, Requires<[HasOnlyMSYNC]>;
 
 let Predicates = [HasFPU] in {
-// Additional FNMSUB patterns: -a*c + b == -(a*c - b)
-def : Pat<(fma (fneg f64:$A), f64:$C, f64:$B),
-          (FNMSUB $A, $C, $B)>;
-def : Pat<(fma f64:$A, (fneg f64:$C), f64:$B),
-          (FNMSUB $A, $C, $B)>;
-def : Pat<(fma (fneg f32:$A), f32:$C, f32:$B),
-          (FNMSUBS $A, $C, $B)>;
-def : Pat<(fma f32:$A, (fneg f32:$C), f32:$B),
-          (FNMSUBS $A, $C, $B)>;
+// Additional fnmsub patterns
+def : Pat<(PPCfnmsub f64:$A, f64:$B, f64:$C),
+          (FNMSUB $A, $B, $C)>;
+def : Pat<(PPCfnmsub f32:$A, f32:$B, f32:$C),
+          (FNMSUBS $A, $B, $C)>;
+def : Pat<(fneg (PPCfnmsub f64:$A, f64:$B, f64:$C)),
+          (FMSUB $A, $B, $C)>;
+def : Pat<(fneg (PPCfnmsub f32:$A, f32:$B, f32:$C)),
+          (FMSUBS $A, $B, $C)>;
 
 // FCOPYSIGN's operand types need not agree.
 def : Pat<(fcopysign f64:$frB, f32:$frA),
           (FCPSGND (COPY_TO_REGCLASS $frA, F8RC), $frB)>;
 def : Pat<(fcopysign f32:$frB, f64:$frA),
           (FCPSGNS (COPY_TO_REGCLASS $frA, F4RC), $frB)>;
 }
 

llvm/lib/Target/PowerPC/PPCInstrVSX.td

def : Pat<(v2f64 (scalar_to_vector f64:$A)),
                            (SUBREG_TO_REG (i64 1), $A, sub_64), 0))>;
 
 def : Pat<(f64 (extractelt v2f64:$S, 0)),
           (f64 (EXTRACT_SUBREG (XXPERMDI $S, $S, 2), sub_64))>;
 def : Pat<(f64 (extractelt v2f64:$S, 1)),
           (f64 (EXTRACT_SUBREG $S, sub_64))>;
 }
 
-// Additional fnmsub patterns: -a*b + c == -(a*b - c)
-def : Pat<(fma (fneg f64:$A), f64:$B, f64:$C),
-          (XSNMSUBADP $C, $A, $B)>;
-def : Pat<(fma f64:$A, (fneg f64:$B), f64:$C),
+// Additional fnmsub pattern for PPC specific ISD opcode
+def : Pat<(PPCfnmsub f64:$A, f64:$B, f64:$C),
           (XSNMSUBADP $C, $A, $B)>;
+def : Pat<(fneg (PPCfnmsub f64:$A, f64:$B, f64:$C)),
+          (XSMSUBADP $C, $A, $B)>;
 
-def : Pat<(fma (fneg v2f64:$A), v2f64:$B, v2f64:$C),
-          (XVNMSUBADP $C, $A, $B)>;
-def : Pat<(fma v2f64:$A, (fneg v2f64:$B), v2f64:$C),
+def : Pat<(PPCfnmsub v2f64:$A, v2f64:$B, v2f64:$C),
           (XVNMSUBADP $C, $A, $B)>;
+def : Pat<(fneg (PPCfnmsub v2f64:$A, v2f64:$B, v2f64:$C)),
+          (XVMSUBADP $C, $A, $B)>;
 
-def : Pat<(fma (fneg v4f32:$A), v4f32:$B, v4f32:$C),
-          (XVNMSUBASP $C, $A, $B)>;
-def : Pat<(fma v4f32:$A, (fneg v4f32:$B), v4f32:$C),
+def : Pat<(PPCfnmsub v4f32:$A, v4f32:$B, v4f32:$C),
           (XVNMSUBASP $C, $A, $B)>;
+def : Pat<(fneg (PPCfnmsub v4f32:$A, v4f32:$B, v4f32:$C)),
+          (XVMSUBASP $C, $A, $B)>;
 
 def : Pat<(v2f64 (bitconvert v4f32:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v4i32:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v8i16:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v16i8:$A)),
let AddedComplexity = 400 in { // Prefer VSX patterns over non-VSX patterns.
   def XSNMSUBMSP : XX3Form<60, 153,
                           (outs vssrc:$XT),
                           (ins vssrc:$XTi, vssrc:$XA, vssrc:$XB),
                           "xsnmsubmsp $XT, $XA, $XB", IIC_VecFP, []>,
                           RegConstraint<"$XTi = $XT">, NoEncode<"$XTi">,
                           AltVSXFMARel;
   }
 
-  // Additional xsnmsubasp patterns: -a*b + c == -(a*b - c)
-  def : Pat<(fma (fneg f32:$A), f32:$B, f32:$C),
-            (XSNMSUBASP $C, $A, $B)>;
-  def : Pat<(fma f32:$A, (fneg f32:$B), f32:$C),
+  // Additional fnmsub pattern for PPC specific ISD opcode
+  def : Pat<(PPCfnmsub f32:$A, f32:$B, f32:$C),
             (XSNMSUBASP $C, $A, $B)>;
 
   // Single Precision Conversions (FP <-> INT)
   def XSCVSXDSP : XX2Form<60, 312,
                       (outs vssrc:$XT), (ins vsfrc:$XB),
                       "xscvsxdsp $XT, $XB", IIC_VecFP,
                       [(set f32:$XT, (PPCfcfids f64:$XB))]>;
   def XSCVUXDSP : XX2Form<60, 296,

llvm/test/CodeGen/PowerPC/combine-fneg.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=powerpc64le-unknown-unknown | FileCheck %s
 
 ; Infinite loop identified in D62963.
 define <4 x double> @fneg_fdiv_splat(double %a0, <4 x double> %a1) {
 ; CHECK-LABEL: fneg_fdiv_splat:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    addis 3, 2, .LCPI0_0@toc@ha
 ; CHECK-NEXT:    # kill: def $f1 killed $f1 def $vsl1
 ; CHECK-NEXT:    xxspltd 0, 1, 0
 ; CHECK-NEXT:    addi 3, 3, .LCPI0_0@toc@l
 ; CHECK-NEXT:    lxvd2x 1, 0, 3
 ; CHECK-NEXT:    xvredp 2, 0
 ; CHECK-NEXT:    xxswapd 1, 1
 ; CHECK-NEXT:    xxlor 3, 1, 1
-; CHECK-NEXT:    xvmaddadp 3, 0, 2
-; CHECK-NEXT:    xvnmsubadp 2, 2, 3
-; CHECK-NEXT:    xvmaddadp 1, 0, 2
-; CHECK-NEXT:    xvmsubadp 2, 2, 1
+; CHECK-NEXT:    xvnmsubadp 3, 0, 2
+; CHECK-NEXT:    xvmaddadp 2, 2, 3
+; CHECK-NEXT:    xvnmsubadp 1, 0, 2
+; CHECK-NEXT:    xvnmaddadp 2, 2, 1
 ; CHECK-NEXT:    xvmuldp 34, 34, 2
 ; CHECK-NEXT:    xvmuldp 35, 35, 2
 ; CHECK-NEXT:    blr
 entry:
   %splat.splatinsert = insertelement <4 x double> undef, double %a0, i32 0
   %splat.splat = shufflevector <4 x double> %splat.splatinsert, <4 x double> undef, <4 x i32> zeroinitializer
   %div = fdiv fast <4 x double> %a1, %splat.splat
   %sub = fsub fast <4 x double> <double 0.000000e+00, double 0.000000e+00, double 0.000000e+00, double 0.000000e+00>, %div
   ret <4 x double> %sub
 }

llvm/test/CodeGen/PowerPC/fma-assoc.ll

; CHECK-VSX-NEXT:    blr
   %I = fpext float %H to double   ; <double> [#uses=1]
   %J = fsub double %I, %E         ; <double> [#uses=1]
   ret double %J
 }
 
 define double @test_FMSUB_ASSOC_EXT3(float %A, float %B, double %C,
 ; CHECK-LABEL: test_FMSUB_ASSOC_EXT3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    fnmsub 0, 1, 2, 5
-; CHECK-NEXT:    fnmsub 1, 3, 4, 0
+; CHECK-NEXT:    fneg 0, 1
+; CHECK-NEXT:    fmadd 0, 0, 2, 5
+; CHECK-NEXT:    fneg 1, 3
+; CHECK-NEXT:    fmadd 1, 1, 4, 0
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-VSX-LABEL: test_FMSUB_ASSOC_EXT3:
 ; CHECK-VSX:       # %bb.0:
-; CHECK-VSX-NEXT:    xsnmsubmdp 1, 2, 5
-; CHECK-VSX-NEXT:    xsnmsubadp 1, 3, 4
+; CHECK-VSX-NEXT:    xsnegdp 1, 1
+; CHECK-VSX-NEXT:    xsnegdp 0, 3
+; CHECK-VSX-NEXT:    xsmaddmdp 1, 2, 5
+; CHECK-VSX-NEXT:    xsmaddadp 1, 0, 4
 ; CHECK-VSX-NEXT:    blr
                                  double %D, double %E) {
   %F = fmul float %A, %B          ; <float> [#uses=1]
   %G = fpext float %F to double   ; <double> [#uses=1]
   %H = fmul double %C, %D         ; <double> [#uses=1]
   %I = fadd double %H, %G         ; <double> [#uses=1]
   %J = fsub double %E, %I         ; <double> [#uses=1]
   ret double %J
 }
 
 define double @test_FMSUB_ASSOC_EXT4(float %A, float %B, float %C,
 ; CHECK-LABEL: test_FMSUB_ASSOC_EXT4:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    fnmsub 0, 3, 4, 5
-; CHECK-NEXT:    fnmsub 1, 1, 2, 0
+; CHECK-NEXT:    fneg 0, 3
+; CHECK-NEXT:    fmadd 0, 0, 4, 5
+; CHECK-NEXT:    fneg 1, 1
+; CHECK-NEXT:    fmadd 1, 1, 2, 0
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-VSX-LABEL: test_FMSUB_ASSOC_EXT4:
 ; CHECK-VSX:       # %bb.0:
-; CHECK-VSX-NEXT:    xsnmsubmdp 3, 4, 5
-; CHECK-VSX-NEXT:    xsnmsubadp 3, 1, 2
-; CHECK-VSX-NEXT:    fmr 1, 3
+; CHECK-VSX-NEXT:    xsnegdp 0, 3
+; CHECK-VSX-NEXT:    xsnegdp 1, 1
+; CHECK-VSX-NEXT:    xsmaddmdp 0, 4, 5
+; CHECK-VSX-NEXT:    xsmaddadp 0, 1, 2
+; CHECK-VSX-NEXT:    fmr 1, 0
 ; CHECK-VSX-NEXT:    blr
                                  float %D, double %E) {
   %F = fmul float %A, %B          ; <float> [#uses=1]
   %G = fmul float %C, %D          ; <float> [#uses=1]
   %H = fadd float %F, %G          ; <float> [#uses=1]
   %I = fpext float %H to double   ; <double> [#uses=1]
   %J = fsub double %E, %I         ; <double> [#uses=1]
   ret double %J
; CHECK-VSX-NEXT:    blr
   %I = fpext float %H to double   ; <double> [#uses=1]
   %J = fsub reassoc double %I, %E         ; <double> [#uses=1]
   ret double %J
 }
 
 define double @test_reassoc_FMSUB_ASSOC_EXT3(float %A, float %B, double %C,
 ; CHECK-LABEL: test_reassoc_FMSUB_ASSOC_EXT3:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    fnmsub 0, 1, 2, 5
-; CHECK-NEXT:    fnmsub 1, 3, 4, 0
+; CHECK-NEXT:    fneg 0, 1
+; CHECK-NEXT:    fmadd 0, 0, 2, 5
+; CHECK-NEXT:    fneg 1, 3
+; CHECK-NEXT:    fmadd 1, 1, 4, 0
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-VSX-LABEL: test_reassoc_FMSUB_ASSOC_EXT3:
 ; CHECK-VSX:       # %bb.0:
-; CHECK-VSX-NEXT:    xsnmsubmdp 1, 2, 5
-; CHECK-VSX-NEXT:    xsnmsubadp 1, 3, 4
+; CHECK-VSX-NEXT:    xsnegdp 1, 1
+; CHECK-VSX-NEXT:    xsnegdp 0, 3
+; CHECK-VSX-NEXT:    xsmaddmdp 1, 2, 5
+; CHECK-VSX-NEXT:    xsmaddadp 1, 0, 4
 ; CHECK-VSX-NEXT:    blr
                                  double %D, double %E) {
   %F = fmul reassoc float %A, %B          ; <float> [#uses=1]
   %G = fpext float %F to double   ; <double> [#uses=1]
   %H = fmul reassoc double %C, %D         ; <double> [#uses=1]
   %I = fadd reassoc double %H, %G         ; <double> [#uses=1]
   %J = fsub reassoc double %E, %I         ; <double> [#uses=1]
   ret double %J
 }
 
+; fnmsub/xsnmsubadp may affect the sign of zero, we need nsz flag
+; to ensure generating them
+define double @test_fast_FMSUB_ASSOC_EXT3(float %A, float %B, double %C,
+; CHECK-LABEL: test_fast_FMSUB_ASSOC_EXT3:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fnmsub 0, 1, 2, 5
+; CHECK-NEXT:    fnmsub 1, 3, 4, 0
+; CHECK-NEXT:    blr
+;
+; CHECK-VSX-LABEL: test_fast_FMSUB_ASSOC_EXT3:
+; CHECK-VSX:       # %bb.0:
+; CHECK-VSX-NEXT:    xsnmsubmdp 1, 2, 5
+; CHECK-VSX-NEXT:    xsnmsubadp 1, 3, 4
+; CHECK-VSX-NEXT:    blr
+                                             double %D, double %E) {
+  %F = fmul fast float %A, %B          ; <float> [#uses=1]
+  %G = fpext float %F to double   ; <double> [#uses=1]
+  %H = fmul fast double %C, %D         ; <double> [#uses=1]
+  %I = fadd fast double %H, %G         ; <double> [#uses=1]
+  %J = fsub fast double %E, %I         ; <double> [#uses=1]
+  ret double %J
+}
+
 define double @test_reassoc_FMSUB_ASSOC_EXT4(float %A, float %B, float %C,
 ; CHECK-LABEL: test_reassoc_FMSUB_ASSOC_EXT4:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    fnmsub 0, 3, 4, 5
-; CHECK-NEXT:    fnmsub 1, 1, 2, 0
+; CHECK-NEXT:    fneg 0, 3
+; CHECK-NEXT:    fmadd 0, 0, 4, 5
+; CHECK-NEXT:    fneg 1, 1
+; CHECK-NEXT:    fmadd 1, 1, 2, 0
 ; CHECK-NEXT:    blr
 ;
 ; CHECK-VSX-LABEL: test_reassoc_FMSUB_ASSOC_EXT4:
 ; CHECK-VSX:       # %bb.0:
-; CHECK-VSX-NEXT:    xsnmsubmdp 3, 4, 5
-; CHECK-VSX-NEXT:    xsnmsubadp 3, 1, 2
-; CHECK-VSX-NEXT:    fmr 1, 3
+; CHECK-VSX-NEXT:    xsnegdp 0, 3
+; CHECK-VSX-NEXT:    xsnegdp 1, 1
+; CHECK-VSX-NEXT:    xsmaddmdp 0, 4, 5
+; CHECK-VSX-NEXT:    xsmaddadp 0, 1, 2
+; CHECK-VSX-NEXT:    fmr 1, 0
 ; CHECK-VSX-NEXT:    blr
                                  float %D, double %E) {
   %F = fmul reassoc float %A, %B          ; <float> [#uses=1]
   %G = fmul reassoc float %C, %D          ; <float> [#uses=1]
   %H = fadd reassoc float %F, %G          ; <float> [#uses=1]
   %I = fpext float %H to double   ; <double> [#uses=1]
   %J = fsub reassoc double %E, %I         ; <double> [#uses=1]
   ret double %J
 }
+
+define double @test_fast_FMSUB_ASSOC_EXT4(float %A, float %B, float %C,
+; CHECK-LABEL: test_fast_FMSUB_ASSOC_EXT4:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    fnmsub 0, 3, 4, 5
+; CHECK-NEXT:    fnmsub 1, 1, 2, 0
+; CHECK-NEXT:    blr
+;
+; CHECK-VSX-LABEL: test_fast_FMSUB_ASSOC_EXT4:
+; CHECK-VSX:       # %bb.0:
+; CHECK-VSX-NEXT:    xsnmsubmdp 3, 4, 5
+; CHECK-VSX-NEXT:    xsnmsubadp 3, 1, 2
+; CHECK-VSX-NEXT:    fmr 1, 3
+; CHECK-VSX-NEXT:    blr
+                                          float %D, double %E) {
+  %F = fmul fast float %A, %B          ; <float> [#uses=1]
+  %G = fmul fast float %C, %D          ; <float> [#uses=1]
+  %H = fadd fast float %F, %G          ; <float> [#uses=1]
+  %I = fpext float %H to double   ; <double> [#uses=1]
+  %J = fsub fast double %E, %I         ; <double> [#uses=1]
+  ret double %J
+}

llvm/test/CodeGen/PowerPC/fma-ext.ll

 }
 
 define double @test_FMSUB_EXT2(float %A, float %B, double %C) {
     %D = fmul float %A, %B          ; <float> [#uses=1]
     %E = fpext float %D to double   ; <double> [#uses=1]
     %F = fsub double %C, %E         ; <double> [#uses=1]
     ret double %F
 ; CHECK-LABEL: test_FMSUB_EXT2:
-; CHECK: fnmsub
+; CHECK: fneg
+; CHECK-NEXT: fmadd
 ; CHECK-NEXT: blr
                                 
 ; CHECK-VSX-LABEL: test_FMSUB_EXT2:
+; CHECK-VSX: xsnegdp
+; CHECK-VSX-NEXT: xsmaddmdp
+; CHECK-VSX-NEXT: blr
+}
+
+; need nsz flag to generate fnmsub since it may affect sign of zero
+define double @test_FMSUB_EXT2_NSZ(float %A, float %B, double %C) {
+    %D = fmul nsz float %A, %B      ; <float> [#uses=1]
+    %E = fpext float %D to double   ; <double> [#uses=1]
+    %F = fsub nsz double %C, %E     ; <double> [#uses=1]
+    ret double %F
+; CHECK-LABEL: test_FMSUB_EXT2_NSZ:
+; CHECK: fnmsub
+; CHECK-NEXT: blr
+
+; CHECK-VSX-LABEL: test_FMSUB_EXT2_NSZ:
 ; CHECK-VSX: xsnmsubmdp
 ; CHECK-VSX-NEXT: blr
 }
 
 define double @test_FMSUB_EXT3(float %A, float %B, double %C) {
     %D = fmul float %A, %B          ; <float> [#uses=1]
     %E = fsub float -0.000000e+00, %D ;    <float> [#uses=1]
     %F = fpext float %E to double   ; <double> [#uses=1]

llvm/test/CodeGen/PowerPC/fma-negate.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple powerpc64le -verify-machineinstrs \
 ; RUN:   | FileCheck -check-prefix=VSX %s
 ; RUN: llc < %s -mtriple powerpc64le -verify-machineinstrs -mattr=-vsx \
 ; RUN:   | FileCheck -check-prefix=NO-VSX %s
 
 define double @test_mul_sub_f64(double %a, double %b, double %c) {
 ; VSX-LABEL: test_mul_sub_f64:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xsnmsubadp 1, 2, 3
+; VSX-NEXT:    xsnegdp 0, 2
+; VSX-NEXT:    xsmaddadp 1, 0, 3
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_mul_sub_f64:
 ; NO-VSX:       # %bb.0: # %entry
-; NO-VSX-NEXT:    fnmsub 1, 2, 3, 1
+; NO-VSX-NEXT:    fneg 0, 2
+; NO-VSX-NEXT:    fmadd 1, 0, 3, 1
 ; NO-VSX-NEXT:    blr
 entry:
   %0 = fmul contract reassoc double %b, %c
   %1 = fsub contract reassoc double %a, %0
   ret double %1
 }
 
 define double @test_2mul_sub_f64(double %a, double %b, double %c, double %d) {
entry:
   %1 = fmul contract reassoc double %c, %d
   %2 = fsub contract reassoc double %0, %1
   ret double %2
 }
 
 define double @test_neg_fma_f64(double %a, double %b, double %c) {
 ; VSX-LABEL: test_neg_fma_f64:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xsnmsubadp 3, 1, 2
+; VSX-NEXT:    xsnegdp 0, 1
+; VSX-NEXT:    xsmaddadp 3, 0, 2
 ; VSX-NEXT:    fmr 1, 3
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_neg_fma_f64:
 ; NO-VSX:       # %bb.0: # %entry
-; NO-VSX-NEXT:    fnmsub 1, 1, 2, 3
+; NO-VSX-NEXT:    fneg 0, 1
+; NO-VSX-NEXT:    fmadd 1, 0, 2, 3
 ; NO-VSX-NEXT:    blr
 entry:
   %0 = fsub contract reassoc double -0.0, %a
   %1 = call contract reassoc double @llvm.fma.f64(double %0, double %b,
                                                   double %c)
   ret double %1
 }
 
 define float @test_mul_sub_f32(float %a, float %b, float %c) {
 ; VSX-LABEL: test_mul_sub_f32:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xsnmsubasp 1, 2, 3
+; VSX-NEXT:    fneg 0, 2
+; VSX-NEXT:    xsmaddasp 1, 0, 3
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_mul_sub_f32:
 ; NO-VSX:       # %bb.0: # %entry
-; NO-VSX-NEXT:    fnmsubs 1, 2, 3, 1
+; NO-VSX-NEXT:    fneg 0, 2
+; NO-VSX-NEXT:    fmadds 1, 0, 3, 1
 ; NO-VSX-NEXT:    blr
 entry:
   %0 = fmul contract reassoc float %b, %c
   %1 = fsub contract reassoc float %a, %0
   ret float %1
 }
 
 define float @test_2mul_sub_f32(float %a, float %b, float %c, float %d) {
entry:
   %1 = fmul contract reassoc float %c, %d
   %2 = fsub contract reassoc float %0, %1
   ret float %2
 }
 
 define float @test_neg_fma_f32(float %a, float %b, float %c) {
 ; VSX-LABEL: test_neg_fma_f32:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xsnmsubasp 3, 1, 2
+; VSX-NEXT:    fneg 0, 1
+; VSX-NEXT:    xsmaddasp 3, 0, 2
 ; VSX-NEXT:    fmr 1, 3
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_neg_fma_f32:
 ; NO-VSX:       # %bb.0: # %entry
-; NO-VSX-NEXT:    fnmsubs 1, 1, 2, 3
+; NO-VSX-NEXT:    fneg 0, 1
+; NO-VSX-NEXT:    fmadds 1, 0, 2, 3
 ; NO-VSX-NEXT:    blr
 entry:
   %0 = fsub contract reassoc float -0.0, %a
   %1 = call contract reassoc float @llvm.fma.f32(float %0, float %b, float %c)
   ret float %1
 }
 
 define <2 x double> @test_neg_fma_v2f64(<2 x double> %a, <2 x double> %b,
 ; VSX-LABEL: test_neg_fma_v2f64:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xvnmsubadp 36, 34, 35
+; VSX-NEXT:    xvnegdp 0, 34
+; VSX-NEXT:    xvmaddadp 36, 0, 35
 ; VSX-NEXT:    vmr 2, 4
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_neg_fma_v2f64:
 ; NO-VSX:       # %bb.0: # %entry
-; NO-VSX-NEXT:    fnmsub 1, 1, 3, 5
-; NO-VSX-NEXT:    fnmsub 2, 2, 4, 6
+; NO-VSX-NEXT:    fneg 0, 2
+; NO-VSX-NEXT:    fneg 1, 1
+; NO-VSX-NEXT:    fmadd 1, 1, 3, 5
+; NO-VSX-NEXT:    fmadd 2, 0, 4, 6
 ; NO-VSX-NEXT:    blr
                                         <2 x double> %c) {
 entry:
   %0 = fsub contract reassoc <2 x double> <double -0.0, double -0.0>, %a
   %1 = call contract reassoc <2 x double> @llvm.fma.v2f64(<2 x double> %0,
                                                           <2 x double> %b,
                                                           <2 x double> %c)
   ret <2 x double> %1
 }
 
 define <4 x float> @test_neg_fma_v4f32(<4 x float> %a, <4 x float> %b,
 ; VSX-LABEL: test_neg_fma_v4f32:
 ; VSX:       # %bb.0: # %entry
-; VSX-NEXT:    xvnmsubasp 36, 34, 35
+; VSX-NEXT:    xvnegsp 0, 34
+; VSX-NEXT:    xvmaddasp 36, 0, 35
 ; VSX-NEXT:    vmr 2, 4
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_neg_fma_v4f32:
 ; NO-VSX:       # %bb.0: # %entry
 ; NO-VSX-NEXT:    vspltisb 5, -1
 ; NO-VSX-NEXT:    vslw 5, 5, 5
 ; NO-VSX-NEXT:    vsubfp 2, 5, 2
entry:
   %1 = fsub fast float %a, %0
   ret float %1
 }
 
 define float @test_fast_2mul_sub_f32(float %a, float %b, float %c, float %d) {
 ; VSX-LABEL: test_fast_2mul_sub_f32:
 ; VSX:       # %bb.0: # %entry
 ; VSX-NEXT:    xsmulsp 0, 3, 4
-; VSX-NEXT:    xsmsubasp 0, 1, 2
-; VSX-NEXT:    fmr 1, 0
+; VSX-NEXT:    fmsubs 1, 1, 2, 0
 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_fast_2mul_sub_f32:
 ; NO-VSX:       # %bb.0: # %entry
 ; NO-VSX-NEXT:    fmuls 0, 3, 4
 ; NO-VSX-NEXT:    fmsubs 1, 1, 2, 0
 ; NO-VSX-NEXT:    blr
 entry:

 ; VSX-NEXT:    blr
 ;
 ; NO-VSX-LABEL: test_fast_neg_fma_v2f64:
 ; NO-VSX:       # %bb.0: # %entry
 ; NO-VSX-NEXT:    fnmsub 1, 1, 3, 5
 ; NO-VSX-NEXT:    fnmsub 2, 2, 4, 6
 ; NO-VSX-NEXT:    blr
                                              <2 x double> %c) {
 entry:
   %0 = fsub fast <2 x double> <double -0.0, double -0.0>, %a
   %1 = call fast <2 x double> @llvm.fma.v2f64(<2 x double> %0, <2 x double> %b,
                                               <2 x double> %c)
   ret <2 x double> %1
 }
 
 define <4 x float> @test_fast_neg_fma_v4f32(<4 x float> %a, <4 x float> %b,
 ; VSX-LABEL: test_fast_neg_fma_v4f32:

llvm/test/CodeGen/PowerPC/fma.ll

 ; CHECK-VSX-NEXT: blr
 }
 
 define double @test_FNMSUB1(double %A, double %B, double %C) {
   %D = fmul double %A, %B   ; <double> [#uses=1]
   %E = fsub double %C, %D   ; <double> [#uses=1]
   ret double %E
 ; CHECK-LABEL: test_FNMSUB1:
-; CHECK: fnmsub
+; CHECK: fneg
+; CHECK-NEXT: fmadd
 ; CHECK-NEXT: blr
 
 ; CHECK-VSX-LABEL: test_FNMSUB1:
+; CHECK-VSX: xsnegdp
+; CHECK-VSX-NEXT: xsmaddmdp
+}
+
+; need nsz flag to generate fnmsub since it may affect sign of zero
+define double @test_FNMSUB1_NSZ(double %A, double %B, double %C) {
+  %D = fmul nsz double %A, %B    ; <double> [#uses=1]
+  %E = fsub nsz double %C, %D    ; <double> [#uses=1]
+  ret double %E
+; CHECK-LABEL: test_FNMSUB1_NSZ:
+; CHECK: fnmsub
+; CHECK-NEXT: blr
+
+; CHECK-VSX-LABEL: test_FNMSUB1_NSZ:
 ; CHECK-VSX: xsnmsubmdp
 }
 
 define double @test_FNMSUB2(double %A, double %B, double %C) {
   %D = fmul double %A, %B   ; <double> [#uses=1]
   %E = fsub double %D, %C   ; <double> [#uses=1]
   %F = fsub double -0.000000e+00, %E    ; <double> [#uses=1]
   ret double %F

llvm/test/CodeGen/PowerPC/repeated-fp-divisors.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs -mtriple=powerpc64le-- < %s | FileCheck %s
 
 define <4 x float> @repeated_fp_divisor(float %a, <4 x float> %b) {
 ; CHECK-LABEL: repeated_fp_divisor:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    xscvdpspn 0, 1
 ; CHECK-NEXT:    addis 3, 2, .LCPI0_0@toc@ha
 ; CHECK-NEXT:    addi 3, 3, .LCPI0_0@toc@l
 ; CHECK-NEXT:    lvx 3, 0, 3
 ; CHECK-NEXT:    addis 3, 2, .LCPI0_1@toc@ha
 ; CHECK-NEXT:    addi 3, 3, .LCPI0_1@toc@l
 ; CHECK-NEXT:    lvx 4, 0, 3
 ; CHECK-NEXT:    xxspltw 0, 0, 0
 ; CHECK-NEXT:    xvresp 1, 0
-; CHECK-NEXT:    xvmaddasp 35, 0, 1
+; CHECK-NEXT:    xvnmsubasp 35, 0, 1
 ; CHECK-NEXT:    xvmulsp 0, 34, 36
-; CHECK-NEXT:    xvnmsubasp 1, 1, 35
+; CHECK-NEXT:    xvmaddasp 1, 1, 35
 ; CHECK-NEXT:    xvmulsp 34, 0, 1
 ; CHECK-NEXT:    blr
   %ins = insertelement <4 x float> undef, float %a, i32 0
   %splat = shufflevector <4 x float> %ins, <4 x float> undef, <4 x i32> zeroinitializer
   %t1 = fmul fast <4 x float> %b, <float 1.000000e+00, float 1.000000e+00, float 1.000000e+00, float 0x3FF028F5C0000000>
   %mul = fdiv fast <4 x float> %t1, %splat
   ret <4 x float> %mul
 }