[X86] Combine constant vector inputs for FMA

[e-kud]

Summary

Inspired by https://discourse.llvm.org/t/folding-memory-into-fma/69217

[e-kud]

Limit to constant splat vectors.

[e-kud]

Rebased

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Why just limit to splat constant?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Also - why the AVX2 and hasOneUse limits?

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

[goldstein.w.n]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

That should be commented explicitly as these subtle checks to prevent infinite loops can easily be removed accidentally.

Also why AVX2 still?

54037

Also - why the AVX2 and hasOneUse limits?

54037

Why just limit to splat constant?

Should be match on ImmConstant no?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54037

Also - why the AVX2 and hasOneUse limits?

Why just limit to splat constant?

This is to limit to broadcasts, probably it is better to work with broadcast explicitly.

I tried more general approach but it seems not much profitable. E.g.

@@ -1,7 +1,7 @@
-; FMA3-LABEL: negated_constant_v4f64:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; FMA3-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; FMA3-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; FMA3-NEXT:    vmovapd %ymm2, %ymm3
+; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; FMA3-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; FMA3-NEXT:    retq

What do you think?

54037

Should be match on ImmConstant no?

AFAIK, m_* matching works with IR. Here we work with DAG. Am I missing something?

54037

Sorry - the hasOneUse is probably necessary to stop ping-ponging between negated constants

Yes. The problem appears when each constant initially has several users, then we start ping-ponging between them. To solve it thoroughly we need to check whether a constant can be eliminated completely: take two constant vectors and check that all users are fmas. It seems out of combiner context to me, doesn't it? However, I see some code that checks instruction's users during combining, maybe, it is the right way.

[e-kud]

Addressed the comments.

[pengfei]

llvm/lib/Target/X86/X86ISelLowering.cpp

53984–53985

Do we need to check it again given it's the only condition checked in line 54073?

53986

This doesn't explain why we need to iterate all its uses?

53988

== is intended? Shouldn't it be covered by User->getOpcode() != ISD::FMA?

53996

Maybe use cast?

53999–54000

Any case can fall into here?

54008–54010

Is the comment for line 54015? Why checking the use again?

54012

ditto.

llvm/test/CodeGen/X86/fma-fneg-combine-2.ll

132

We can make the elements different now, right?

[e-kud]

Addressed comments.
Added comments.
Fixed negative value search in case of undefs.
Updated tests.

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

53984–53985

Do we need to check it again given it's the only condition checked in line 54073?

Dropped it.

53986

This doesn't explain why we need to iterate all its uses?

I added comments describing the approach. Generally we don't want to optimize anything if there are other non-FMA consumers of constant BUILD_VECTOR. Because it will be loaded anyway. So, we want to understand which one (inverted or original) can be eliminated. If both can be eliminated, we need to fix a particular vector because we don't have a context to understand which one is original and which one is inverted even among a single instruction operands.

If there is a concern about complexity of this check, it can be replaced with simplier hasOneUse. It limits the scope though.

53996

Maybe use cast?

53999–54000

Maybe use cast?

Any case can fall into here?

This is completely OK to have a constant vector with undef values, e.g <2 x double> <double undef, double undef>. There are some tests with such vectors. So, we can't use cast here. I assumed that during optimization we may obtain such the constant.

54008–54010

I've described approach above. Here we check users of an inverted vector.

[pengfei]

Thanks @e-kud, LGTM.

[e-kud]

Rebase

[RKSimon]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

Couldn't both V and NV have negative values now that they aren't splats?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

It seems to me that they can't. If V is <x0, x1, ..., xn> then NV is <-x0, -x1, ..., -xn>. So, V and NV always have elements of different sign. The corner case is V consisting only of undefs. But original and negated versions of such vector are the same. So, everything should be handled well.

[goldstein.w.n]

llvm/lib/Target/X86/X86ISelLowering.cpp

54173

persistently

54178

But couldn't both V and NV contain some negative and some positive values?

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54178

The idea here is to prefer a vector that has the first negative operand. We don't care about other operands. A negative value can be only in V or NV not in both.

But here is a pitfall that instead of FP value we may meet undef, That's why we iterate over operands until the first non-undef value is met. V and NV have undefs on same places.

So, V and NV may contain both negative and positive values but we consider only the first non-undef operand.

[e-kud]

Rebase. Spelling.

[pengfei]

@e-kud do you want me to commit if for you? Or you are waiting for another approval?

[e-kud]

In D146494#4339563, @pengfei wrote:

@e-kud do you want me to commit if for you? Or you are waiting for another approval?

@pengfei yes, I'll appreciate it! I wanted additionally check on SPEC: there are minor binary size reductions and several cases where this combine applies -- eliminated broadcasts as well as using a register operand instead of memory.

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

54487

Can we use llvm::any_of or !llvm::all_of with a lambda?

54495

Capitalize op

54510

Can we use llvm::any_of or !llvm::all_of with a lambda?

54514

"consistently" is probably a better word than "peristently"

54517

llvm::any_of

[e-kud]

Addressed the comments.

[e-kud]

llvm/lib/Target/X86/X86ISelLowering.cpp

54514

Indeed :)

54517

I don't think that any_of is much better here. We need to stop traversing after the first non-undef value (this is controlled by return value of the lambda) and return true or false (using a variable from context). Then we generally ignore return result of any_of and use only the variable with the result. It looks artificial to me. Probably I'm missing something.

Added an extra comment here.

[craig.topper]

llvm/lib/Target/X86/X86ISelLowering.cpp

54517

Oops. I think I missed the break statement.

llvm/lib/Target/X86/X86ISelLowering.cpp

   }
 
   if (SDValue NewAdd = promoteExtBeforeAdd(N, DAG, Subtarget))
     return NewAdd;
 
   return SDValue();
 }
 
+static SDValue getInvertedVectorForFMA(SDValue V, SelectionDAG &DAG) {
+  assert(ISD::isBuildVectorOfConstantFPSDNodes(V.getNode()) &&
+         "ConstantFP build vector expected");
+  // Check if we can eliminate a constant completely
+  for (const SDNode *User : V->uses()) {
+    if (User->getOpcode() != ISD::FMA || User->getOpcode() == ISD::STRICT_FMA)
+      return SDValue();
+  }
+  // Form an inverted vector
+  SmallVector<SDValue, 8> Ops;
+  EVT VT = V.getValueType();
+  EVT EltVT = VT.getVectorElementType();
+  for (auto op : V->op_values()) {
+    if (auto *Cst = dyn_cast<ConstantFPSDNode>(op)) {
+      Ops.push_back(DAG.getConstantFP(-Cst->getValueAPF(), SDLoc(op), EltVT));
+    } else {
+      assert(op.isUndef());
+      Ops.push_back(DAG.getUNDEF(EltVT));
+    }
+  }
+
+  SDNode *NV = DAG.getNodeIfExists(ISD::BUILD_VECTOR, DAG.getVTList(VT), Ops);
+  if (!NV)
+    return SDValue();
+
+  // If the inverted value also can be eliminated, we have to persistancy
+  // prefer one of the values. We prefer a constant with negative value on the
+  // first element.
+  for (const SDNode *User : NV->uses())
+    if (User->getOpcode() != ISD::FMA || User->getOpcode() == ISD::STRICT_FMA)
+      return SDValue(NV, 0);
+
+  if (cast<ConstantFPSDNode>(V->getOperand(0))->isNegative())
+    return SDValue();
+
+  return SDValue(NV, 0);
+}
+
 static SDValue combineFMA(SDNode *N, SelectionDAG &DAG,
                           TargetLowering::DAGCombinerInfo &DCI,
                           const X86Subtarget &Subtarget) {
   SDLoc dl(N);
   EVT VT = N->getValueType(0);
   bool IsStrict = N->isStrictFPOpcode() || N->isTargetStrictFPOpcode();
 
   // Let legalize expand this if it isn't a legal type yet.
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   if (!TLI.isTypeLegal(VT))
     return SDValue();
 
   SDValue A = N->getOperand(IsStrict ? 1 : 0);
   SDValue B = N->getOperand(IsStrict ? 2 : 1);
   SDValue C = N->getOperand(IsStrict ? 3 : 2);
 
   // If the operation allows fast-math and the target does not support FMA,
   // split this into mul+add to avoid libcall(s).
   SDNodeFlags Flags = N->getFlags();
   if (!IsStrict && Flags.hasAllowReassociation() &&
       TLI.isOperationExpand(ISD::FMA, VT)) {
     SDValue Fmul = DAG.getNode(ISD::FMUL, dl, VT, A, B, Flags);
     return DAG.getNode(ISD::FADD, dl, VT, Fmul, C, Flags);
   }
 

       SDValue Vec = V.getOperand(0);
       if (SDValue NegV = TLI.getCheaperNegatedExpression(
               Vec, DAG, LegalOperations, CodeSize)) {
         V = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, SDLoc(V), V.getValueType(),
                         NegV, V.getOperand(1));
         return true;
       }
     }
-
+    // Lookup if there is an inverted version of constant vector V in DAG.
+    if (ISD::isBuildVectorOfConstantFPSDNodes(V.getNode())) {
+      if (SDValue NegV = getInvertedVectorForFMA(V, DAG)) {
+        V = NegV;
+        return true;
+      }
+    }
     return false;
   };
 
   // Do not convert the passthru input of scalar intrinsics.
   // FIXME: We could allow negations of the lower element only.
   bool NegA = invertIfNegative(A);
   bool NegB = invertIfNegative(B);
   bool NegC = invertIfNegative(C);

                                                  DAG, DCI, Subtarget))
     return V;
 
   if (VT.isVector())
     if (SDValue R = PromoteMaskArithmetic(N, DAG, Subtarget))
       return R;
 
   if (SDValue NewAdd = promoteExtBeforeAdd(N, DAG, Subtarget))
     return NewAdd;
 
   if (SDValue R = combineOrCmpEqZeroToCtlzSrl(N, DAG, DCI, Subtarget))
     return R;
 
   // TODO: Combine with any target/faux shuffle.
   if (N0.getOpcode() == X86ISD::PACKUS && N0.getValueSizeInBits() == 128 &&
       VT.getScalarSizeInBits() == N0.getOperand(0).getScalarValueSizeInBits()) {
     SDValue N00 = N0.getOperand(0);
     SDValue N01 = N0.getOperand(1);
     unsigned NumSrcEltBits = N00.getScalarValueSizeInBits();
     APInt ZeroMask = APInt::getHighBitsSet(NumSrcEltBits, NumSrcEltBits / 2);
     if ((N00.isUndef() || DAG.MaskedValueIsZero(N00, ZeroMask)) &&
         (N01.isUndef() || DAG.MaskedValueIsZero(N01, ZeroMask))) {

   // Fold movmsk(icmp_sgt(x,-1)) -> not(movmsk(x)) to improve folding of movmsk
   // results with scalar comparisons.
   if (Src.getOpcode() == X86ISD::PCMPGT &&
       ISD::isBuildVectorAllOnes(Src.getOperand(1).getNode())) {
     SDLoc DL(N);
     APInt NotMask = APInt::getLowBitsSet(NumBits, NumElts);
     return DAG.getNode(ISD::XOR, DL, VT,
                        DAG.getNode(X86ISD::MOVMSK, DL, VT, Src.getOperand(0)),
                        DAG.getConstant(NotMask, DL, VT));
   }
 
   // Fold movmsk(icmp_eq(and(x,c1),0)) -> movmsk(not(shl(x,c2)))
   // iff pow2splat(c1).
   // Use KnownBits to determine if only a single bit is non-zero
   // in each element (pow2 or zero), and shift that bit to the msb.
   if (Src.getOpcode() == X86ISD::PCMPEQ &&
       ISD::isBuildVectorAllZeros(Src.getOperand(1).getNode())) {
     KnownBits KnownSrc = DAG.computeKnownBits(Src.getOperand(0));
     if (KnownSrc.countMaxPopulation() == 1) {
       SDLoc DL(N);
       MVT ShiftVT = SrcVT;
       SDValue ShiftSrc = Src.getOperand(0);
       if (ShiftVT.getScalarType() == MVT::i8) {
         // vXi8 shifts - we only care about the signbit so can use PSLLW.
         ShiftVT = MVT::getVectorVT(MVT::i16, NumElts / 2);
         ShiftSrc = DAG.getBitcast(ShiftVT, ShiftSrc);
       }
       unsigned ShiftAmt = KnownSrc.countMinLeadingZeros();
       ShiftSrc = getTargetVShiftByConstNode(X86ISD::VSHLI, DL, ShiftVT,
                                             ShiftSrc, ShiftAmt, DAG);
       ShiftSrc = DAG.getNOT(DL, DAG.getBitcast(SrcVT, ShiftSrc), SrcVT);
       return DAG.getNode(X86ISD::MOVMSK, DL, VT, ShiftSrc);
     }
   }
 
   // Simplify the inputs.
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   APInt DemandedMask(APInt::getAllOnes(NumBits));
   if (TLI.SimplifyDemandedBits(SDValue(N, 0), DemandedMask, DCI))
     return SDValue(N, 0);
 
   return SDValue();
 }
 
 static SDValue combineX86GatherScatter(SDNode *N, SelectionDAG &DAG,
                                        TargetLowering::DAGCombinerInfo &DCI,
                                        const X86Subtarget &Subtarget) {

llvm/test/CodeGen/X86/avx2-fma-fneg-combine.ll

; X64-NEXT:    retq
   %t3 = tail call nsz <8 x float> @llvm.fma.v8f32(<8 x float> %t2, <8 x float> %b, <8 x float> %c)
   ret <8 x float> %t3
 }
 
 define <4 x double> @test9(<4 x double> %a) {
 ; X32-LABEL: test9:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X32-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm1
+; X32-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test9:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X64-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm1
+; X64-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; X64-NEXT:    retq
   %t = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   ret <4 x double> %t
 }
 
 define <4 x double> @test10(<4 x double> %a, <4 x double> %b) {
 ; X32-LABEL: test10:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X32-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; X32-NEXT:    vbroadcastsd {{.*#+}} ymm3 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
+; X32-NEXT:    vmovapd %ymm2, %ymm3
 ; X32-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
-; X32-NEXT:    vaddpd %ymm3, %ymm2, %ymm0
+; X32-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; X32-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test10:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vbroadcastsd {{.*#+}} ymm2 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X64-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; X64-NEXT:    vbroadcastsd {{.*#+}} ymm3 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
+; X64-NEXT:    vmovapd %ymm2, %ymm3
 ; X64-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
-; X64-NEXT:    vaddpd %ymm3, %ymm2, %ymm0
+; X64-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; X64-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; X64-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double -5.000000e-01, double undef, double -5.000000e-01, double -5.000000e-01>, <4 x double> %b)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double undef, double 5.000000e-01, double 5.000000e-01>, <4 x double> %b)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }
 
 define <4 x double> @test11(<4 x double> %a) {
 ; X32-LABEL: test11:
 ; X32:       # %bb.0:
 ; X32-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X32-NEXT:    vaddpd %ymm1, %ymm0, %ymm2
-; X32-NEXT:    vbroadcastsd {{.*#+}} ymm0 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X32-NEXT:    vfmadd231pd {{.*#+}} ymm0 = (ymm1 * ymm2) + ymm0
+; X32-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
+; X32-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; X32-NEXT:    retl
 ;
 ; X64-LABEL: test11:
 ; X64:       # %bb.0:
 ; X64-NEXT:    vbroadcastsd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm2
-; X64-NEXT:    vbroadcastsd {{.*#+}} ymm0 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
-; X64-NEXT:    vfmadd231pd {{.*#+}} ymm0 = (ymm1 * ymm2) + ymm0
+; X64-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
+; X64-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; X64-NEXT:    retq
   %t0 = fadd <4 x double> %a, <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %t0, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   ret <4 x double> %t1
 }
 
 define <4 x double> @test12(<4 x double> %a) {
 ; X32-LABEL: test12:

llvm/test/CodeGen/X86/fma-fneg-combine-2.ll

; FMA4-NEXT:    retq
   %fma = call nsz float @llvm.fma.f32(float %x, float -42.0, float %m)
   %nfma = fneg float %fma
   ret float %nfma
 }
 
 define <4 x double> @negated_constant_v4f64(<4 x double> %a) {
 ; FMA3-LABEL: negated_constant_v4f64:
 ; FMA3:       # %bb.0:
-; FMA3-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; FMA3-NEXT:    vmovapd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm1
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64:
 ; FMA4:       # %bb.0:
-; FMA4-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * ymm1) + mem
+; FMA4-NEXT:    vmovapd {{.*#+}} ymm1 = [-5.0E-1,-5.0E-1,-5.0E-1,-5.0E-1]
+; FMA4-NEXT:    vfnmaddpd {{.*#+}} ymm0 = -(ymm0 * ymm1) + ymm1
 ; FMA4-NEXT:    retq
   %t = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   ret <4 x double> %t
 }
 
 define <4 x double> @negated_constant_v4f64_2fmas(<4 x double> %a, <4 x double> %b) {
 ; FMA3-LABEL: negated_constant_v4f64_2fmas:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vmovapd {{.*#+}} ymm2 = <-5.0E-1,u,-5.0E-1,-5.0E-1>
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm2 = (ymm0 * ymm2) + ymm1
-; FMA3-NEXT:    vfmadd231pd {{.*#+}} ymm1 = (ymm0 * mem) + ymm1
-; FMA3-NEXT:    vaddpd %ymm1, %ymm2, %ymm0
+; FMA3-NEXT:    vmovapd %ymm2, %ymm3
+; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm3 = (ymm0 * ymm3) + ymm1
+; FMA3-NEXT:    vfnmadd213pd {{.*#+}} ymm2 = -(ymm0 * ymm2) + ymm1
+; FMA3-NEXT:    vaddpd %ymm2, %ymm3, %ymm0
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64_2fmas:
 ; FMA4:       # %bb.0:
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm2 = (ymm0 * mem) + ymm1
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * mem) + ymm1
-; FMA4-NEXT:    vaddpd %ymm0, %ymm2, %ymm0
+; FMA4-NEXT:    vmovapd {{.*#+}} ymm2 = <-5.0E-1,u,-5.0E-1,-5.0E-1>
+; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm3 = (ymm0 * ymm2) + ymm1
+; FMA4-NEXT:    vfnmaddpd {{.*#+}} ymm0 = -(ymm0 * ymm2) + ymm1
+; FMA4-NEXT:    vaddpd %ymm0, %ymm3, %ymm0
 ; FMA4-NEXT:    retq
   %t0 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double -5.000000e-01, double undef, double -5.000000e-01, double -5.000000e-01>, <4 x double> %b)
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %a, <4 x double> <double 5.000000e-01, double undef, double 5.000000e-01, double 5.000000e-01>, <4 x double> %b)
   %t2 = fadd <4 x double> %t0, %t1
   ret <4 x double> %t2
 }
 
 define <4 x double> @negated_constant_v4f64_fadd(<4 x double> %a) {
 ; FMA3-LABEL: negated_constant_v4f64_fadd:
 ; FMA3:       # %bb.0:
 ; FMA3-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
 ; FMA3-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; FMA3-NEXT:    vfmadd213pd {{.*#+}} ymm0 = (ymm1 * ymm0) + mem
+; FMA3-NEXT:    vfmsub213pd {{.*#+}} ymm0 = (ymm1 * ymm0) - ymm1
 ; FMA3-NEXT:    retq
 ;
 ; FMA4-LABEL: negated_constant_v4f64_fadd:
 ; FMA4:       # %bb.0:
 ; FMA4-NEXT:    vmovapd {{.*#+}} ymm1 = [5.0E-1,5.0E-1,5.0E-1,5.0E-1]
 ; FMA4-NEXT:    vaddpd %ymm1, %ymm0, %ymm0
-; FMA4-NEXT:    vfmaddpd {{.*#+}} ymm0 = (ymm0 * ymm1) + mem
+; FMA4-NEXT:    vfmsubpd {{.*#+}} ymm0 = (ymm0 * ymm1) - ymm1
 ; FMA4-NEXT:    retq
   %t0 = fadd <4 x double> %a, <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>
   %t1 = tail call <4 x double> @llvm.fma.v4f64(<4 x double> %t0, <4 x double> <double 5.000000e-01, double 5.000000e-01, double 5.000000e-01, double 5.000000e-01>, <4 x double> <double -5.000000e-01, double -5.000000e-01, double -5.000000e-01, double -5.000000e-01>)
   ret <4 x double> %t1
 }
 
 define <4 x double> @negated_constant_v4f64_2fadd(<4 x double> %a) {
 ; FMA3-LABEL: negated_constant_v4f64_2fadd: