[X86] Combine constant vector inputs for FMA

[e-kud]

Summary

Inspired by https://discourse.llvm.org/t/folding-memory-into-fma/69217

[e-kud]

Limit to constant splat vectors.

[e-kud]

Rebased

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Why just limit to splat constant?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Also - why the AVX2 and hasOneUse limits?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

[goldstein.w.n]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

That should be commented explicitly as these subtle checks to prevent infinite loops can easily be removed accidentally.

Also why AVX2 still?

54037

Also - why the AVX2 and hasOneUse limits?

54037

Why just limit to splat constant?

Should be match on ImmConstant no?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Also - why the AVX2 and hasOneUse limits?

Why just limit to splat constant?

This is to limit to broadcasts, probably it is better to work with broadcast explicitly.

I tried more general approach but it seems not much profitable. E.g.

@@ -1,7 +1,7 @@
-; FMA3-LABEL: negated_constant_v4f64:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; FMA3-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; FMA3-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; FMA3-NEXT:    vmovapd %ymm2, %ymm3
+; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; FMA3-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; FMA3-NEXT:    retq

What do you think?

54037

Should be match on ImmConstant no?

AFAIK, m_* matching works with IR. Here we work with DAG. Am I missing something?

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

Yes. The problem appears when each constant initially has several users, then we start ping-ponging between them. To solve it thoroughly we need to check whether a constant can be eliminated completely: take two constant vectors and check that all users are fmas. It seems out of combiner context to me, doesn't it? However, I see some code that checks instruction's users during combining, maybe, it is the right way.

[e-kud]

Addressed the comments.

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

53984–53985

Do we need to check it again given it's the only condition checked in line 54073?

53986

This doesn't explain why we need to iterate all its uses?

53988

== is intended? Shouldn't it be covered by User->getOpcode() != ISD::FMA?

53996

Maybe use cast?

53999–54000

Any case can fall into here?

54008–54010

Is the comment for line 54015? Why checking the use again?

54012

ditto.

llvm/test/CodeGen/X86/fma-fneg-combine-2.ll

132–133

We can make the elements different now, right?

[e-kud]

Addressed comments.
Added comments.
Fixed negative value search in case of undefs.
Updated tests.

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

53984–53985

Do we need to check it again given it's the only condition checked in line 54073?

Dropped it.

53986

This doesn't explain why we need to iterate all its uses?

I added comments describing the approach. Generally we don't want to optimize anything if there are other non-FMA consumers of constant BUILD_VECTOR. Because it will be loaded anyway. So, we want to understand which one (inverted or original) can be eliminated. If both can be eliminated, we need to fix a particular vector because we don't have a context to understand which one is original and which one is inverted even among a single instruction operands.

If there is a concern about complexity of this check, it can be replaced with simplier hasOneUse. It limits the scope though.

53996

Maybe use cast?

53999–54000

Maybe use cast?

Any case can fall into here?

This is completely OK to have a constant vector with undef values, e.g <2 x double> <double undef, double undef>. There are some tests with such vectors. So, we can't use cast here. I assumed that during optimization we may obtain such the constant.

54008–54010

I've described approach above. Here we check users of an inverted vector.

[pengfei]

Thanks @e-kud, LGTM.

[e-kud]

Rebase

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

Couldn't both V and NV have negative values now that they aren't splats?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

It seems to me that they can't. If V is <x0, x1, ..., xn> then NV is <-x0, -x1, ..., -xn>. So, V and NV always have elements of different sign. The corner case is V consisting only of undefs. But original and negated versions of such vector are the same. So, everything should be handled well.

[goldstein.w.n]

llvm/lib/Target/X86/X86ISelLowering.cpp

54173

persistently

54178

But couldn't both V and NV contain some negative and some positive values?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

The idea here is to prefer a vector that has the first negative operand. We don't care about other operands. A negative value can be only in V or NV not in both.

But here is a pitfall that instead of FP value we may meet undef, That's why we iterate over operands until the first non-undef value is met. V and NV have undefs on same places.

So, V and NV may contain both negative and positive values but we consider only the first non-undef operand.

[e-kud]

Rebase. Spelling.

[pengfei]

@e-kud do you want me to commit if for you? Or you are waiting for another approval?

[e-kud]

In D146494#4339563, @pengfei wrote:

@e-kud do you want me to commit if for you? Or you are waiting for another approval?

@pengfei yes, I'll appreciate it! I wanted additionally check on SPEC: there are minor binary size reductions and several cases where this combine applies -- eliminated broadcasts as well as using a register operand instead of memory.

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

54487

Can we use llvm::any_of or !llvm::all_of with a lambda?

54495

Capitalize op

54510

Can we use llvm::any_of or !llvm::all_of with a lambda?

54514

"consistently" is probably a better word than "peristently"

54517

llvm::any_of

[e-kud]

Addressed the comments.

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54514

Indeed :)

54517

I don't think that any_of is much better here. We need to stop traversing after the first non-undef value (this is controlled by return value of the lambda) and return true or false (using a variable from context). Then we generally ignore return result of any_of and use only the variable with the result. It looks artificial to me. Probably I'm missing something.

Added an extra comment here.

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

54517

Oops. I think I missed the break statement.

llvm/lib/Target/X86/X86ISelLowering.cpp

static SDValue combineX86INT_TO_FP(SDNode *N, SelectionDAG &DAG,
   EVT VT = N->getValueType(0);
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
 
   APInt DemandedElts = APInt::getAllOnes(VT.getVectorNumElements());
   if (TLI.SimplifyDemandedVectorElts(SDValue(N, 0), DemandedElts, DCI))
     return SDValue(N, 0);
 
   // Convert a full vector load into vzload when not all bits are needed.
   SDValue In = N->getOperand(0);
   MVT InVT = In.getSimpleValueType();
   if (VT.getVectorNumElements() < InVT.getVectorNumElements() &&
       ISD::isNormalLoad(In.getNode()) && In.hasOneUse()) {
     assert(InVT.is128BitVector() && "Expected 128-bit input vector");
     LoadSDNode *LN = cast<LoadSDNode>(N->getOperand(0));
     unsigned NumBits = InVT.getScalarSizeInBits() * VT.getVectorNumElements();
     MVT MemVT = MVT::getIntegerVT(NumBits);
     MVT LoadVT = MVT::getVectorVT(MemVT, 128 / NumBits);
     if (SDValue VZLoad = narrowLoadToVZLoad(LN, MemVT, LoadVT, DAG)) {
       SDLoc dl(N);
       SDValue Convert = DAG.getNode(N->getOpcode(), dl, VT,
                                     DAG.getBitcast(InVT, VZLoad));
       DCI.CombineTo(N, Convert);
       DAG.ReplaceAllUsesOfValueWith(SDValue(LN, 1), VZLoad.getValue(1));
       DCI.recursivelyDeleteUnusedNodes(LN);
       return SDValue(N, 0);
     }
   }
 
   return SDValue();
 }
 
 static SDValue combineCVTP2I_CVTTP2I(SDNode *N, SelectionDAG &DAG,
                                      TargetLowering::DAGCombinerInfo &DCI) {
   bool IsStrict = N->isTargetStrictFPOpcode();
   EVT VT = N->getValueType(0);
 
   // Convert a full vector load into vzload when not all bits are needed.
   SDValue In = N->getOperand(IsStrict ? 1 : 0);
   MVT InVT = In.getSimpleValueType();
   if (VT.getVectorNumElements() < InVT.getVectorNumElements() &&
       ISD::isNormalLoad(In.getNode()) && In.hasOneUse()) {
     assert(InVT.is128BitVector() && "Expected 128-bit input vector");
     LoadSDNode *LN = cast<LoadSDNode>(In);
     unsigned NumBits = InVT.getScalarSizeInBits() * VT.getVectorNumElements();
     MVT MemVT = MVT::getFloatingPointVT(NumBits);
     MVT LoadVT = MVT::getVectorVT(MemVT, 128 / NumBits);
     if (SDValue VZLoad = narrowLoadToVZLoad(LN, MemVT, LoadVT, DAG)) {
       SDLoc dl(N);
       if (IsStrict) {
         SDValue Convert =
             DAG.getNode(N->getOpcode(), dl, {VT, MVT::Other},
                         {N->getOperand(0), DAG.getBitcast(InVT, VZLoad)});
         DCI.CombineTo(N, Convert, Convert.getValue(1));
       } else {
         SDValue Convert =
             DAG.getNode(N->getOpcode(), dl, VT, DAG.getBitcast(InVT, VZLoad));
         DCI.CombineTo(N, Convert);
       }
       DAG.ReplaceAllUsesOfValueWith(SDValue(LN, 1), VZLoad.getValue(1));
       DCI.recursivelyDeleteUnusedNodes(LN);
       return SDValue(N, 0);
     }
   }
 
   return SDValue();
 }
 
 /// Do target-specific dag combines on X86ISD::ANDNP nodes.
 static SDValue combineAndnp(SDNode *N, SelectionDAG &DAG,
                             TargetLowering::DAGCombinerInfo &DCI,
static SDValue combineCVTPH2PS(SDNode *N, SelectionDAG &DAG,
   bool IsStrict = N->getOpcode() == X86ISD::STRICT_CVTPH2PS;
   SDValue Src = N->getOperand(IsStrict ? 1 : 0);
 
   if (N->getValueType(0) == MVT::v4f32 && Src.getValueType() == MVT::v8i16) {
     const TargetLowering &TLI = DAG.getTargetLoweringInfo();
     APInt DemandedElts = APInt::getLowBitsSet(8, 4);
     if (TLI.SimplifyDemandedVectorElts(Src, DemandedElts, DCI)) {
       if (N->getOpcode() != ISD::DELETED_NODE)
         DCI.AddToWorklist(N);
       return SDValue(N, 0);
     }
 
     // Convert a full vector load into vzload when not all bits are needed.
     if (ISD::isNormalLoad(Src.getNode()) && Src.hasOneUse()) {
       LoadSDNode *LN = cast<LoadSDNode>(N->getOperand(IsStrict ? 1 : 0));
       if (SDValue VZLoad = narrowLoadToVZLoad(LN, MVT::i64, MVT::v2i64, DAG)) {
         SDLoc dl(N);
         if (IsStrict) {
           SDValue Convert = DAG.getNode(
               N->getOpcode(), dl, {MVT::v4f32, MVT::Other},
               {N->getOperand(0), DAG.getBitcast(MVT::v8i16, VZLoad)});
           DCI.CombineTo(N, Convert, Convert.getValue(1));
static SDValue combineSext(SDNode *N, SelectionDAG &DAG,
   EVT VT = N->getValueType(0);
   SDLoc DL(N);
 
   // (i32 (sext (i8 (x86isd::setcc_carry)))) -> (i32 (x86isd::setcc_carry))
   if (!DCI.isBeforeLegalizeOps() &&
       N0.getOpcode() == X86ISD::SETCC_CARRY) {
     SDValue Setcc = DAG.getNode(X86ISD::SETCC_CARRY, DL, VT, N0->getOperand(0),
                                  N0->getOperand(1));
     bool ReplaceOtherUses = !N0.hasOneUse();
     DCI.CombineTo(N, Setcc);
     // Replace other uses with a truncate of the widened setcc_carry.
     if (ReplaceOtherUses) {
       SDValue Trunc = DAG.getNode(ISD::TRUNCATE, SDLoc(N0),
                                   N0.getValueType(), Setcc);
       DCI.CombineTo(N0.getNode(), Trunc);
     }
 
     return SDValue(N, 0);
   }
 
   if (SDValue NewCMov = combineToExtendCMOV(N, DAG))
     return NewCMov;
 
   if (!DCI.isBeforeLegalizeOps())
     return SDValue();
 
   if (SDValue V = combineExtSetcc(N, DAG, Subtarget))
     return V;
 
   if (SDValue V = combineToExtendBoolVectorInReg(N->getOpcode(), DL, VT, N0,
                                                  DAG, DCI, Subtarget))
     return V;
 
   if (VT.isVector()) {
     if (SDValue R = PromoteMaskArithmetic(N, DAG, Subtarget))
       return R;
 
     if (N0.getOpcode() == ISD::SIGN_EXTEND_VECTOR_INREG)
       return DAG.getNode(N0.getOpcode(), DL, VT, N0.getOperand(0));
   }
 
   if (SDValue NewAdd = promoteExtBeforeAdd(N, DAG, Subtarget))
     return NewAdd;
 
   return SDValue();
 }
 
+// Inverting a constant vector is profitable if it can be eliminated and the
+// inverted vector is already present in DAG. Otherwise, it will be loaded
+// anyway.
+//
+// We determine which of the values can be completely eliminated and invert it.
+// If both are eliminable, select a vector with the first negative element.
+static SDValue getInvertedVectorForFMA(SDValue V, SelectionDAG &DAG) {
+  assert(ISD::isBuildVectorOfConstantFPSDNodes(V.getNode()) &&
+         "ConstantFP build vector expected");
+  // Check if we can eliminate V. We assume if a value is only used in FMAs, we
+  // can eliminate it. Since this function is invoked for each FMA with this
+  // vector.
+  auto IsNotFMA = [](SDNode *Use) {
+    return Use->getOpcode() != ISD::FMA && Use->getOpcode() != ISD::STRICT_FMA;
+  };
+  if (llvm::any_of(V->uses(), IsNotFMA))
+    return SDValue();
+
+  SmallVector<SDValue, 8> Ops;
+  EVT VT = V.getValueType();
+  EVT EltVT = VT.getVectorElementType();
+  for (auto Op : V->op_values()) {
+    if (auto *Cst = dyn_cast<ConstantFPSDNode>(Op)) {
+      Ops.push_back(DAG.getConstantFP(-Cst->getValueAPF(), SDLoc(Op), EltVT));
+    } else {
+      assert(Op.isUndef());
+      Ops.push_back(DAG.getUNDEF(EltVT));
+    }
+  }
+
+  SDNode *NV = DAG.getNodeIfExists(ISD::BUILD_VECTOR, DAG.getVTList(VT), Ops);
+  if (!NV)
+    return SDValue();
+
+  // If an inverted version cannot be eliminated, choose it instead of the
+  // original version.
+  if (llvm::any_of(NV->uses(), IsNotFMA))
+    return SDValue(NV, 0);
+
+  // If the inverted version also can be eliminated, we have to consistently
+  // prefer one of the values. We prefer a constant with a negative value on
+  // the first place.
+  // N.B. We need to skip undefs that may precede a value.
+  for (auto op : V->op_values()) {
+    if (auto *Cst = dyn_cast<ConstantFPSDNode>(op)) {
+      if (Cst->isNegative())
+        return SDValue();
+      break;
+    }
+  }
+  return SDValue(NV, 0);
+}
+
 static SDValue combineFMA(SDNode *N, SelectionDAG &DAG,
                           TargetLowering::DAGCombinerInfo &DCI,
                           const X86Subtarget &Subtarget) {
   SDLoc dl(N);
   EVT VT = N->getValueType(0);
   bool IsStrict = N->isStrictFPOpcode() || N->isTargetStrictFPOpcode();
 
   // Let legalize expand this if it isn't a legal type yet.
if (V.getOpcode() == ISD::EXTRACT_VECTOR_ELT &&
       SDValue Vec = V.getOperand(0);
       if (SDValue NegV = TLI.getCheaperNegatedExpression(
               Vec, DAG, LegalOperations, CodeSize)) {
         V = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, SDLoc(V), V.getValueType(),
                         NegV, V.getOperand(1));
         return true;
       }
     }
-
+    // Lookup if there is an inverted version of constant vector V in DAG.
+    if (ISD::isBuildVectorOfConstantFPSDNodes(V.getNode())) {
+      if (SDValue NegV = getInvertedVectorForFMA(V, DAG)) {
+        V = NegV;
+        return true;
+      }
+    }
     return false;
   };
 
   // Do not convert the passthru input of scalar intrinsics.
   // FIXME: We could allow negations of the lower element only.
   bool NegA = invertIfNegative(A);
   bool NegB = invertIfNegative(B);
   bool NegC = invertIfNegative(C);

llvm/test/CodeGen/X86/avx2-fma-fneg-combine.ll

; X64-NEXT:    retq
   %t3 = tail call nsz <8 x float> @llvm.fma.v8f32(<8 x float> %t2, <8 x float> %b, <8 x float> %c)
   ret <8 x float> %t3
 }
 
 define <4 x double> @test9(<4 x double> %a) {
 ; X32-LABEL: test9:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X32-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm1
+; X32-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test9:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X64-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm1
+; X64-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; X64-NEXT:    retq
   %t = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   ret <4 x double> %t
 }
 
 define <4 x double> @test10(<4 x double> %a, <4 x double> %b) {
 ; X32-LABEL: test10:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vmovapd {{.*#+}} ymm2 = <-9.5E+0,u,-5.5E+0,-2.5E+0>
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; X32-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; X32-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; X32-NEXT:    vmovapd %ymm2, %ymm3
+; X32-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; X32-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; X32-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test10:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vmovapd {{.*#+}} ymm2 = <-9.5E+0,u,-5.5E+0,-2.5E+0>
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; X64-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; X64-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; X64-NEXT:    vmovapd %ymm2, %ymm3
+; X64-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; X64-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; X64-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; X64-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double -95.00000e-01, double undef, double -55.00000e-01, double -25.00000e-01>, <4 x double> %b)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 95.00000e-01, double undef, double 55.00000e-01, double 25.00000e-01>, <4 x double> %b)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }
 
 define <4 x double> @test11(<4 x double> %a) {
 ; X32-LABEL: test11:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vbroadcastf128 {{.*#+}} ymm1 = [5.0E-1,2.5E+0,5.0E-1,2.5E+0]
 ; X32-NEXT:    # ymm1 = mem[0,1,0,1]
 ; X32-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; X32-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test11:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vbroadcastf128 {{.*#+}} ymm1 = [5.0E-1,2.5E+0,5.0E-1,2.5E+0]
 ; X64-NEXT:    # ymm1 = mem[0,1,0,1]
 ; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; X64-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; X64-NEXT:    retq
   %t0 = fadd <4 x double> %a, <double 5.000000e-01, double 25.00000e-01, double 5.000000e-01, double 25.00000e-01>
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %t0, <4 x double> <double 5.000000e-01, double 25.00000e-01, double 5.000000e-01, double 25.00000e-01>, <4 x double> <double -5.000000e-01, double -25.00000e-01, double -5.000000e-01, double -25.00000e-01>)
   ret <4 x double> %t1
 }
 
 define <4 x double> @test12(<4 x double> %a, <4 x double> %b) {
 ; X32-LABEL: test12:
 ; X32:       # %bb.0:
-; X32-NEXT:    vmovapd {{.*#+}} ymm2 = [7.5E+0,2.5E+0,5.5E+0,9.5E+0]
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + mem
-; X32-NEXT:    vmovapd {{.*#+}} ymm0 = <u,2.5E+0,5.5E+0,9.5E+0>
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
-; X32-NEXT:    vaddpd %ymm0, %ymm2, %ymm0
+; X32-NEXT:    vmovapd {{.*#+}} ymm2 = [-7.5E+0,-2.5E+0,-5.5E+0,-9.5E+0]
+; X32-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm2 * ymm0) + mem
+; X32-NEXT:    vfmadd132pd {{.*#+}} ymm1 = (ymm1 * mem) + ymm2
+; X32-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test12:
 ; X64:       # %bb.0:
-; X64-NEXT:    vmovapd {{.*#+}} ymm2 = [7.5E+0,2.5E+0,5.5E+0,9.5E+0]
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + mem
-; X64-NEXT:    vmovapd {{.*#+}} ymm0 = <u,2.5E+0,5.5E+0,9.5E+0>
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
-; X64-NEXT:    vaddpd %ymm0, %ymm2, %ymm0
+; X64-NEXT:    vmovapd {{.*#+}} ymm2 = [-7.5E+0,-2.5E+0,-5.5E+0,-9.5E+0]
+; X64-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm2 * ymm0) + mem
+; X64-NEXT:    vfmadd132pd {{.*#+}} ymm1 = (ymm1 * mem) + ymm2
+; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; X64-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 75.00000e-01, double 25.00000e-01, double 55.00000e-01, double 95.00000e-01>, <4 x double> <double -75.00000e-01, double undef, double -55.00000e-01, double -95.00000e-01>)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %b, <4 x double> <double undef, double 25.00000e-01, double 55.00000e-01, double 95.00000e-01>, <4 x double> <double -75.00000e-01, double -25.00000e-01, double -55.00000e-01, double -95.00000e-01>)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }

llvm/test/CodeGen/X86/fma-fneg-combine-2.ll

; FMA4-NEXT:    retq
   %fma = call nsz float @llvm.fma.f32(float %x, float -42.0, float %m)
   %nfma = fneg float %fma
   ret float %nfma
 }
 
 define <4 x double> @negated_constant_v4f64(<4 x double> %a) {
 ; FMA3-LABEL: negated_constant_v4f64:
 ; FMA3:       # %bb.0:
-; FMA3-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,2.5E-1,1.25E-1,6.25E-2]
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; FMA3-NEXT:    vmovapd {{.*#+}} ymm1 = [-5.0E-1,-2.5E-1,-1.25E-1,-6.25E-2]
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64:
 ; FMA4:       # %bb.0:
-; FMA4-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,2.5E-1,1.25E-1,6.25E-2]
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * ymm1) + mem
+; FMA4-NEXT:    vmovapd {{.*#+}} ymm1 = [-5.0E-1,-2.5E-1,-1.25E-1,-6.25E-2]
+; FMA4-NEXT:    vfnmaddpd {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
 ; FMA4-NEXT:    retq
   %t = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double 2.5000000e-01, double 1.25000000e-01, double 0.62500000e-01>, <4 x double> <double -5.000000e-01, double -2.5000000e-01, double -1.25000000e-01, double -0.62500000e-01>)
   ret <4 x double> %t
 }
 
 define <4 x double> @negated_constant_v4f64_2fmas(<4 x double> %a, <4 x double> %b) {
 ; FMA3-LABEL: negated_constant_v4f64_2fmas:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = <-5.0E-1,u,-2.5E+0,-4.5E+0>
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; FMA3-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; FMA3-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; FMA3-NEXT:    vmovapd %ymm2, %ymm3
+; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; FMA3-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64_2fmas:
 ; FMA4:       # %bb.0:
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm2 = (ymm0 * mem) + ymm1
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * mem) + ymm1
-; FMA4-NEXT:    vaddpd %ymm0, %ymm2, %ymm0
+; FMA4-NEXT:    vmovapd {{.*#+}} ymm2 = <-5.0E-1,u,-2.5E+0,-4.5E+0>
+; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm3 = (ymm0 * ymm2) + ymm1
+; FMA4-NEXT:    vfnmaddpd {{.*#+}} ymm0 = -(ymm0 * ymm2) + ymm1
+; FMA4-NEXT:    vaddpd %ymm0, %ymm3, %ymm0
 ; FMA4-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double -5.000000e-01, double undef, double -25.000000e-01, double -45.000000e-01>, <4 x double> %b)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double undef, double 25.000000e-01, double 45.000000e-01>, <4 x double> %b)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }
 
 define <4 x double> @negated_constant_v4f64_fadd(<4 x double> %a) {
 ; FMA3-LABEL: negated_constant_v4f64_fadd:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vbroadcastf128 {{.*#+}} ymm1 = [1.5E+0,1.25E-1,1.5E+0,1.25E-1]
 ; FMA3-NEXT:    # ymm1 = mem[0,1,0,1]
 ; FMA3-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; FMA3-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64_fadd:
 ; FMA4:       # %bb.0:
 ; FMA4-NEXT:    vbroadcastf128 {{.*#+}} ymm1 = [1.5E+0,1.25E-1,1.5E+0,1.25E-1]
 ; FMA4-NEXT:    # ymm1 = mem[0,1,0,1]
 ; FMA4-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * ymm1) + mem
+; FMA4-NEXT:    vfmsubpd {{.*#+}} ymm0 = (ymm0 * ymm1) - ymm1
 ; FMA4-NEXT:    retq
   %t0 = fadd <4 x double> %a, <double 15.000000e-01, double 1.25000000e-01, double 15.000000e-01, double 1.25000000e-01>
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %t0, <4 x double> <double 15.000000e-01, double 1.25000000e-01, double 15.000000e-01, double 1.25000000e-01>, <4 x double> <double -15.000000e-01, double -1.25000000e-01, double -15.000000e-01, double -1.25000000e-01>)
   ret <4 x double> %t1
 }
 
 define <4 x double> @negated_constant_v4f64_2fma_undefs(<4 x double> %a, <4 x double> %b) {
 ; FMA3-LABEL: negated_constant_v4f64_2fma_undefs:
 ; FMA3:       # %bb.0:
-; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + mem
-; FMA3-NEXT:    vmovapd {{.*#+}} ymm0 = <u,5.0E-1,5.0E-1,5.0E-1>
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
-; FMA3-NEXT:    vaddpd %ymm0, %ymm2, %ymm0
+; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm2 * ymm0) + mem
+; FMA3-NEXT:    vfmadd132pd {{.*#+}} ymm1 = (ymm1 * mem) + ymm2
+; FMA3-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64_2fma_undefs:
 ; FMA4:       # %bb.0:
-; FMA4-NEXT:    vmovapd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * ymm2) + mem
-; FMA4-NEXT:    vmovapd {{.*#+}} ymm2 = <u,5.0E-1,5.0E-1,5.0E-1>
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm1 = (ymm1 * ymm2) + mem
+; FMA4-NEXT:    vmovapd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
+; FMA4-NEXT:    vfnmaddpd {{.*#+}} ymm0 = -(ymm0 * ymm2) + mem
+; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm1 = (ymm1 * mem) + ymm2
 ; FMA4-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
 ; FMA4-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double undef, double -5.000000e-01, double -5.000000e-01>)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %b, <4 x double> <double undef, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }
 
 declare float @llvm.fma.f32(float, float, float)
 declare <4 x float> @llvm.fma.v4f32(<4 x float>, <4 x float>, <4 x float>)
 declare <4 x double> @llvm.fma.v4f64(<4 x double>, <4 x double>, <4 x double>)