X86: FMA intrinsic + FNEG - sequence optimization

[delena]

Summary

Optimized FMA intrinsic + FNEG , like

-(a*b+c)

and FNEG + FMA, like

a*b-c or (-a)*b+c

Legalization of ISD::FNEG is delayed to give a chance to combine it with X86ISD::FMADD/MSUB/NMADD/NMSUB.

7 out of 8 test cases are optimized. The last test case requires additional investigation and, probably, changes in the clang header.

Fixed https://llvm.org/bugs/show_bug.cgi?id=28892.

[RKSimon]

../test/CodeGen/X86/fma-fneg-combine.ll

72 ↗

(On Diff #67319)

Is this case fixed by D23108?

[delena]

The patch https://reviews.llvm.org/D23108 will remove redundant "vmovaps" after the FMA.

[RKSimon]

Instead of adding the forceLegalization approach why can't we just expand combineFMA to support the negation of inputs for all X86ISD FMA/AVX512 opcodes? A helper function might be necessary (to detect FNEG or FXOR signbit-flipping) but that should be all.

[delena]

In D23313#511156, @RKSimon wrote:

Instead of adding the forceLegalization approach why can't we just expand combineFMA to support the negation of inputs for all X86ISD FMA/AVX512 opcodes? A helper function might be necessary (to detect FNEG or FXOR signbit-flipping) but that should be all.

When we receive X86ISD FMA/AVX512 opcodes, FNEG does not exist any more. It is lowered to FXOR and this is the problem. Analysis of FXOR requires visiting load and visiting constant pool and checking the value of the integer constant under the FP load.
That's why I want to delay FNEG legalization.

[delena]

Ping*

[spatel]

In D23313#511163, @delena wrote:

In D23313#511156, @RKSimon wrote:

Instead of adding the forceLegalization approach why can't we just expand combineFMA to support the negation of inputs for all X86ISD FMA/AVX512 opcodes? A helper function might be necessary (to detect FNEG or FXOR signbit-flipping) but that should be all.

When we receive X86ISD FMA/AVX512 opcodes, FNEG does not exist any more. It is lowered to FXOR and this is the problem. Analysis of FXOR requires visiting load and visiting constant pool and checking the value of the integer constant under the FP load.
That's why I want to delay FNEG legalization.

The premature lowering to load from constant pool is the real problem isn't it? If we didn't do that, then you'd just match the expected X86ISD::FXOR with -1 pattern.

There's a FIXME in LowerFABSorFNEG() about making this better with optsize. I think we should fix this now rather than adding a work-around for something that's broken.

I didn't check if the failures due to this change are just due to load folding diffs or other problems, but the basic patch is just delete the bogus code in LowerFABSorFNEG():

Index: lib/Target/X86/X86ISelLowering.cpp

• lib/Target/X86/X86ISelLowering.cpp (revision 278598)

+++ lib/Target/X86/X86ISelLowering.cpp (working copy)
@@ -14519,7 +14519,7 @@

// decide if we should generate a 16-byte constant mask when we only need 4 or
// 8 bytes for the scalar case.

• MVT LogicVT;

+ EVT LogicVT;

MVT EltVT;
unsigned NumElts;

@@ -14543,18 +14543,10 @@

}
 
unsigned EltBits = EltVT.getSizeInBits();

• LLVMContext *Context = DAG.getContext(); // For FABS, mask is 0x7f...; for FNEG, mask is 0x80... APInt MaskElt = IsFABS ? APInt::getSignedMaxValue(EltBits) : APInt::getSignBit(EltBits);
• Constant *C = ConstantInt::get(*Context, MaskElt);
• C = ConstantVector::getSplat(NumElts, C);
• const TargetLowering &TLI = DAG.getTargetLoweringInfo();
• SDValue CPIdx = DAG.getConstantPool(C, TLI.getPointerTy(DAG.getDataLayout()));
• unsigned Alignment = cast<ConstantPoolSDNode>(CPIdx)->getAlignment();
• SDValue Mask = DAG.getLoad(
• LogicVT, dl, DAG.getEntryNode(), CPIdx,
• MachinePointerInfo::getConstantPool(DAG.getMachineFunction()), Alignment);

+ SDValue Mask = DAG.getConstant(MaskElt, dl, LogicVT.changeVectorElementTypeToInteger());

[RKSimon]

We're already doing some constant pool extraction with getTargetShuffleMaskIndices/getTargetShuffleMaskConstant - what if we generalized that?

[spatel]

In D23313#516440, @RKSimon wrote:

We're already doing some constant pool extraction with getTargetShuffleMaskIndices/getTargetShuffleMaskConstant - what if we generalized that?

rL279958 fixed the constant loading, so this patch should be updated (and should become simpler now).

[delena]

Agree, I'll proceed.

[delena]

Changed code according to the comments bellow. I also added more tests, not all of the test cases are optimized.

[spatel]

../lib/Target/X86/X86ISelLowering.cpp

4869 ↗

(On Diff #69739)

It's good to generalize this, but you should also change the variable names from Mask* to something generic since it's not about shuffles anymore. Please make this change ahead of the fneg changes to reduce the size of the patch.

30361–30373 ↗

(On Diff #69739)

Are the *RND changes independent of the fneg bug fix? If yes, can you separate them from this patch?

30402 ↗

(On Diff #69739)

returns -> Returns
Add period to end of sentence.

30411 ↗

(On Diff #69739)

Sing -> Sign

[delena]

../lib/Target/X86/X86ISelLowering.cpp

30361–30373 ↗

(On Diff #69739)

The goal of this patch is to combine FNEG with FMA intrinsics.
*_RND nodes are generated from intrinsics only.
Here we are combining the following patterns:
FNEG ( avx512.mask.fm* a, b ,c) and
FNEG ( avx512.mask.fm* a, b ,c, Rounding)

[delena]

Updated comments inside the code and variable names. NFC since the previous patch.

[spatel]

Should there be some tests for scalar double and packed double as well?

../lib/Target/X86/X86ISelLowering.cpp

4872 ↗

(On Diff #69800)

MaskLoad -> Load ?

4881 ↗

(On Diff #69800)

MaskCP -> ConstantPoolNode ?

[delena]

Added more tests for "double" scalar and vector intrinsics.

[spatel]

LGTM.

llvm/trunk/lib/Target/X86/X86ISelLowering.cpp

else if (auto *CFN = dyn_cast<ConstantFPSDNode>(Op.getNode()))
       SplitElementToMask(CFN->getValueAPF().bitcastToAPInt());
     else
       return false;
   }
 
   return true;
 }
 
-static const Constant *getTargetShuffleMaskConstant(SDValue MaskNode) {
-  MaskNode = peekThroughBitcasts(MaskNode);
+static const Constant *getTargetConstantFromNode(SDValue Op) {
+  Op = peekThroughBitcasts(Op);
 
-  auto *MaskLoad = dyn_cast<LoadSDNode>(MaskNode);
-  if (!MaskLoad)
+  auto *Load = dyn_cast<LoadSDNode>(Op);
+  if (!Load)
     return nullptr;
 
-  SDValue Ptr = MaskLoad->getBasePtr();
+  SDValue Ptr = Load->getBasePtr();
   if (Ptr->getOpcode() == X86ISD::Wrapper ||
       Ptr->getOpcode() == X86ISD::WrapperRIP)
     Ptr = Ptr->getOperand(0);
 
-  auto *MaskCP = dyn_cast<ConstantPoolSDNode>(Ptr);
-  if (!MaskCP || MaskCP->isMachineConstantPoolEntry())
+  auto *CNode = dyn_cast<ConstantPoolSDNode>(Ptr);
+  if (!CNode || CNode->isMachineConstantPoolEntry())
     return nullptr;
 
-  return dyn_cast<Constant>(MaskCP->getConstVal());
+  return dyn_cast<Constant>(CNode->getConstVal());
 }
 
 /// Calculates the shuffle mask corresponding to the target-specific opcode.
 /// If the mask could be calculated, returns it in \p Mask, returns the shuffle
 /// operands in \p Ops, and returns true.
 /// Sets \p IsUnary to true if only one source is used. Note that this will set
 /// IsUnary for shuffles which use a single input multiple times, and in those
 /// cases it will adjust the mask to only have indices within that single input.
case X86ISD::VPERMILPV: {
     IsUnary = true;
     SDValue MaskNode = N->getOperand(1);
     unsigned MaskEltSize = VT.getScalarSizeInBits();
     SmallVector<uint64_t, 32> RawMask;
     if (getTargetShuffleMaskIndices(MaskNode, MaskEltSize, RawMask)) {
       DecodeVPERMILPMask(VT, RawMask, Mask);
       break;
     }
-    if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+    if (auto *C = getTargetConstantFromNode(MaskNode)) {
       DecodeVPERMILPMask(C, MaskEltSize, Mask);
       break;
     }
     return false;
   }
   case X86ISD::PSHUFB: {
     IsUnary = true;
     SDValue MaskNode = N->getOperand(1);
     SmallVector<uint64_t, 32> RawMask;
     if (getTargetShuffleMaskIndices(MaskNode, 8, RawMask)) {
       DecodePSHUFBMask(RawMask, Mask);
       break;
     }
-    if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+    if (auto *C = getTargetConstantFromNode(MaskNode)) {
       DecodePSHUFBMask(C, Mask);
       break;
     }
     return false;
   }
   case X86ISD::VPERMI:
     ImmN = N->getOperand(N->getNumOperands()-1);
     DecodeVPERMMask(VT, cast<ConstantSDNode>(ImmN)->getZExtValue(), Mask);
case X86ISD::VPERMIL2: {
     SDValue CtrlNode = N->getOperand(3);
     if (ConstantSDNode *CtrlOp = dyn_cast<ConstantSDNode>(CtrlNode)) {
       unsigned CtrlImm = CtrlOp->getZExtValue();
       SmallVector<uint64_t, 32> RawMask;
       if (getTargetShuffleMaskIndices(MaskNode, MaskEltSize, RawMask)) {
         DecodeVPERMIL2PMask(VT, CtrlImm, RawMask, Mask);
         break;
       }
-      if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+      if (auto *C = getTargetConstantFromNode(MaskNode)) {
         DecodeVPERMIL2PMask(C, CtrlImm, MaskEltSize, Mask);
         break;
       }
     }
     return false;
   }
   case X86ISD::VPPERM: {
     IsUnary = IsFakeUnary = N->getOperand(0) == N->getOperand(1);
     SDValue MaskNode = N->getOperand(2);
     SmallVector<uint64_t, 32> RawMask;
     if (getTargetShuffleMaskIndices(MaskNode, 8, RawMask)) {
       DecodeVPPERMMask(RawMask, Mask);
       break;
     }
-    if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+    if (auto *C = getTargetConstantFromNode(MaskNode)) {
       DecodeVPPERMMask(C, Mask);
       break;
     }
     return false;
   }
   case X86ISD::VPERMV: {
     IsUnary = true;
     // Unlike most shuffle nodes, VPERMV's mask operand is operand 0.
     Ops.push_back(N->getOperand(1));
     SDValue MaskNode = N->getOperand(0);
     SmallVector<uint64_t, 32> RawMask;
     unsigned MaskEltSize = VT.getScalarSizeInBits();
     if (getTargetShuffleMaskIndices(MaskNode, MaskEltSize, RawMask)) {
       DecodeVPERMVMask(RawMask, Mask);
       break;
     }
-    if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+    if (auto *C = getTargetConstantFromNode(MaskNode)) {
       DecodeVPERMVMask(C, VT, Mask);
       break;
     }
     return false;
   }
   case X86ISD::VPERMV3: {
     IsUnary = IsFakeUnary = N->getOperand(0) == N->getOperand(2);
     // Unlike most shuffle nodes, VPERMV3's mask operand is the middle one.
     Ops.push_back(N->getOperand(0));
     Ops.push_back(N->getOperand(2));
     SDValue MaskNode = N->getOperand(1);
-    if (auto *C = getTargetShuffleMaskConstant(MaskNode)) {
+    if (auto *C = getTargetConstantFromNode(MaskNode)) {
       DecodeVPERMV3Mask(C, VT, Mask);
       break;
     }
     return false;
   }
   default: llvm_unreachable("unknown target shuffle node");
   }
 
case X86ISD::FMSUB:
       return DAG.getNode(X86ISD::FNMADD, DL, VT, Arg.getOperand(0),
                          Arg.getOperand(1), Arg.getOperand(2));
     case X86ISD::FNMADD:
       return DAG.getNode(X86ISD::FMSUB, DL, VT, Arg.getOperand(0),
                          Arg.getOperand(1), Arg.getOperand(2));
     case X86ISD::FNMSUB:
       return DAG.getNode(X86ISD::FMADD, DL, VT, Arg.getOperand(0),
                          Arg.getOperand(1), Arg.getOperand(2));
+    case X86ISD::FMADD_RND:
+      return DAG.getNode(X86ISD::FNMSUB_RND, DL, VT, Arg.getOperand(0),
+                         Arg.getOperand(1), Arg.getOperand(2), Arg.getOperand(3));
+    case X86ISD::FMSUB_RND:
+      return DAG.getNode(X86ISD::FNMADD_RND, DL, VT, Arg.getOperand(0),
+                         Arg.getOperand(1), Arg.getOperand(2), Arg.getOperand(3));
+    case X86ISD::FNMADD_RND:
+      return DAG.getNode(X86ISD::FMSUB_RND, DL, VT, Arg.getOperand(0),
+                         Arg.getOperand(1), Arg.getOperand(2), Arg.getOperand(3));
+    case X86ISD::FNMSUB_RND:
+      return DAG.getNode(X86ISD::FMADD_RND, DL, VT, Arg.getOperand(0),
+                         Arg.getOperand(1), Arg.getOperand(2), Arg.getOperand(3));
     }
   }
   return SDValue();
 }
 
 static SDValue lowerX86FPLogicOp(SDNode *N, SelectionDAG &DAG,
                               const X86Subtarget &Subtarget) {
   EVT VT = N->getValueType(0);
switch (N->getOpcode()) {
       case X86ISD::FAND: IntOpcode = ISD::AND; break;
       case X86ISD::FANDN: IntOpcode = X86ISD::ANDNP; break;
     }
     SDValue IntOp = DAG.getNode(IntOpcode, dl, MVT::v8i64, Op0, Op1);
     return DAG.getBitcast(VT, IntOp);
   }
   return SDValue();
 }
+
+/// Returns true if the node \p N is FNEG(x) or FXOR (x, 0x80000000).
+bool isFNEG(const SDNode *N) {
+  if (N->getOpcode() == ISD::FNEG)
+    return true;
+
+  if (N->getOpcode() == X86ISD::FXOR) {
+    unsigned EltBits = N->getSimpleValueType(0).getScalarSizeInBits();
+    SDValue Op1 = N->getOperand(1);
+
+    auto isSignBitValue = [&](const ConstantFP *C) {
+      return C->getValueAPF().bitcastToAPInt() == APInt::getSignBit(EltBits);
+    };
+
+    // There is more than one way to represent the same constant on
+    // the different X86 targets. The type of the node may also depend on size.
+    //  - load scalar value and broadcast
+    //  - BUILD_VECTOR node
+    //  - load from a constant pool.
+    // We check all variants here.
+    if (Op1.getOpcode() == X86ISD::VBROADCAST) {
+      if (auto *C = getTargetConstantFromNode(Op1.getOperand(0)))
+        return isSignBitValue(cast<ConstantFP>(C));
+
+    } else if (BuildVectorSDNode *BV = dyn_cast<BuildVectorSDNode>(Op1)) {
+      if (ConstantFPSDNode *CN = BV->getConstantFPSplatNode())
+        return isSignBitValue(CN->getConstantFPValue());
+
+    } else if (auto *C = getTargetConstantFromNode(Op1)) {
+      if (C->getType()->isVectorTy()) {
+        if (auto *SplatV = C->getSplatValue())
+          return isSignBitValue(cast<ConstantFP>(SplatV));
+      } else if (auto *FPConst = dyn_cast<ConstantFP>(C))
+        return isSignBitValue(FPConst);
+    }
+  }
+  return false;
+}
+
 /// Do target-specific dag combines on X86ISD::FOR and X86ISD::FXOR nodes.
 static SDValue combineFOr(SDNode *N, SelectionDAG &DAG,
                           const X86Subtarget &Subtarget) {
   assert(N->getOpcode() == X86ISD::FOR || N->getOpcode() == X86ISD::FXOR);
 
   // F[X]OR(0.0, x) -> x
   if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(N->getOperand(0)))
     if (C->getValueAPF().isPosZero())
       return N->getOperand(1);
 
   // F[X]OR(x, 0.0) -> x
   if (ConstantFPSDNode *C = dyn_cast<ConstantFPSDNode>(N->getOperand(1)))
     if (C->getValueAPF().isPosZero())
       return N->getOperand(0);
 
+  if (isFNEG(N))
+    if (SDValue NewVal = combineFneg(N, DAG, Subtarget))
+      return NewVal;
   return lowerX86FPLogicOp(N, DAG, Subtarget);
 }
 
 /// Do target-specific dag combines on X86ISD::FMIN and X86ISD::FMAX nodes.
 static SDValue combineFMinFMax(SDNode *N, SelectionDAG &DAG) {
   assert(N->getOpcode() == X86ISD::FMIN || N->getOpcode() == X86ISD::FMAX);
 
   // Only perform optimizations if UnsafeMath is used.
static SDValue combineFMA(SDNode *N, SelectionDAG &DAG,
   EVT ScalarVT = VT.getScalarType();
   if ((ScalarVT != MVT::f32 && ScalarVT != MVT::f64) || !Subtarget.hasAnyFMA())
     return SDValue();
 
   SDValue A = N->getOperand(0);
   SDValue B = N->getOperand(1);
   SDValue C = N->getOperand(2);
 
-  bool NegA = (A.getOpcode() == ISD::FNEG);
-  bool NegB = (B.getOpcode() == ISD::FNEG);
-  bool NegC = (C.getOpcode() == ISD::FNEG);
+  bool NegA = isFNEG(A.getNode());
+  bool NegB = isFNEG(B.getNode());
+  bool NegC = isFNEG(C.getNode());
 
   // Negative multiplication when NegA xor NegB
   bool NegMul = (NegA != NegB);
   if (NegA)
     A = A.getOperand(0);
   if (NegB)
     B = B.getOperand(0);
   if (NegC)
     C = C.getOperand(0);
 
-  unsigned Opcode;
+  unsigned NewOpcode;
   if (!NegMul)
-    Opcode = (!NegC) ? X86ISD::FMADD : X86ISD::FMSUB;
+    NewOpcode = (!NegC) ? X86ISD::FMADD : X86ISD::FMSUB;
   else
-    Opcode = (!NegC) ? X86ISD::FNMADD : X86ISD::FNMSUB;
+    NewOpcode = (!NegC) ? X86ISD::FNMADD : X86ISD::FNMSUB;
 
-  return DAG.getNode(Opcode, dl, VT, A, B, C);
+  if (N->getOpcode() == X86ISD::FMADD_RND) {
+    switch (NewOpcode) {
+      case X86ISD::FMADD:  NewOpcode = X86ISD::FMADD_RND; break;
+      case X86ISD::FMSUB:  NewOpcode = X86ISD::FMSUB_RND; break;
+      case X86ISD::FNMADD: NewOpcode = X86ISD::FNMADD_RND; break;
+      case X86ISD::FNMSUB: NewOpcode = X86ISD::FNMSUB_RND; break;
+    }
+    return DAG.getNode(NewOpcode, dl, VT, A, B, C, N->getOperand(3));
+  }
+  return DAG.getNode(NewOpcode, dl, VT, A, B, C);
 }
 
 static SDValue combineZext(SDNode *N, SelectionDAG &DAG,
                            TargetLowering::DAGCombinerInfo &DCI,
                            const X86Subtarget &Subtarget) {
   // (i32 zext (and (i8  x86isd::setcc_carry), 1)) ->
   //           (and (i32 x86isd::setcc_carry), 1)
   // This eliminates the zext. This transformation is necessary because
SDValue X86TargetLowering::PerformDAGCombine(SDNode *N,
   case X86ISD::VPERMV:
   case X86ISD::VPERMV3:
   case X86ISD::VPERMIL2:
   case X86ISD::VPERMILPI:
   case X86ISD::VPERMILPV:
   case X86ISD::VPERM2X128:
   case X86ISD::VZEXT_MOVL:
   case ISD::VECTOR_SHUFFLE: return combineShuffle(N, DAG, DCI,Subtarget);
+  case X86ISD::FMADD:
+  case X86ISD::FMADD_RND:
   case ISD::FMA:            return combineFMA(N, DAG, Subtarget);
   case ISD::MGATHER:
   case ISD::MSCATTER:       return combineGatherScatter(N, DAG);
   case X86ISD::LSUB:        return combineLockSub(N, DAG, Subtarget);
   case X86ISD::TESTM:       return combineTestM(N, DAG);
   case X86ISD::PCMPEQ:
   case X86ISD::PCMPGT:      return combineVectorCompare(N, DAG, Subtarget);
   }

llvm/trunk/test/CodeGen/X86/avx2-fma-fneg-combine.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx2 -mattr=+fma  | FileCheck %s
+
+; This test checks combinations of FNEG and FMA intrinsics
+
+define <8 x float> @test1(<8 x float> %a, <8 x float> %b, <8 x float> %c)  {
+; CHECK-LABEL: test1:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfmsub213ps %ymm2, %ymm1, %ymm0
+; CHECK-NEXT:    retq
+entry:
+  %sub.i = fsub <8 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %c
+  %0 = tail call <8 x float> @llvm.x86.fma.vfmadd.ps.256(<8 x float> %a, <8 x float> %b, <8 x float> %sub.i) #2
+  ret <8 x float> %0
+}
+
+declare <8 x float> @llvm.x86.fma.vfmadd.ps.256(<8 x float>, <8 x float>, <8 x float>)
+
+define <4 x float> @test2(<4 x float> %a, <4 x float> %b, <4 x float> %c) {
+; CHECK-LABEL: test2:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmsub213ps %xmm2, %xmm1, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <4 x float> @llvm.x86.fma.vfmadd.ps(<4 x float> %a, <4 x float> %b, <4 x float> %c) #2
+  %sub.i = fsub <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
+  ret <4 x float> %sub.i
+}
+
+declare <4 x float> @llvm.x86.fma.vfmadd.ps(<4 x float> %a, <4 x float> %b, <4 x float> %c)
+
+define <4 x float> @test3(<4 x float> %a, <4 x float> %b, <4 x float> %c)  {
+; CHECK-LABEL: test3:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmadd213ss %xmm2, %xmm1, %xmm0
+; CHECK-NEXT:    vbroadcastss {{.*}}(%rip), %xmm1
+; CHECK-NEXT:    vxorps %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <4 x float> @llvm.x86.fma.vfnmadd.ss(<4 x float> %a, <4 x float> %b, <4 x float> %c) #2
+  %sub.i = fsub <4 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
+  ret <4 x float> %sub.i
+}
+
+declare <4 x float> @llvm.x86.fma.vfnmadd.ss(<4 x float> %a, <4 x float> %b, <4 x float> %c)
+
+define <8 x float> @test4(<8 x float> %a, <8 x float> %b, <8 x float> %c) {
+; CHECK-LABEL: test4:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmadd213ps %ymm2, %ymm1, %ymm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float> %a, <8 x float> %b, <8 x float> %c) #2
+  %sub.i = fsub <8 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
+  ret <8 x float> %sub.i
+}
+
+define <8 x float> @test5(<8 x float> %a, <8 x float> %b, <8 x float> %c) {
+; CHECK-LABEL: test5:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vbroadcastss {{.*}}(%rip), %ymm3
+; CHECK-NEXT:    vxorps %ymm3, %ymm2, %ymm2
+; CHECK-NEXT:    vfmsub213ps %ymm2, %ymm1, %ymm0
+; CHECK-NEXT:    retq
+entry:
+  %sub.c = fsub <8 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %c
+  %0 = tail call <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float> %a, <8 x float> %b, <8 x float> %sub.c) #2
+  ret <8 x float> %0
+}
+
+declare <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float>, <8 x float>, <8 x float>)
+
+
+define <2 x double> @test6(<2 x double> %a, <2 x double> %b, <2 x double> %c) {
+; CHECK-LABEL: test6:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmsub213pd %xmm2, %xmm1, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <2 x double> @llvm.x86.fma.vfmadd.pd(<2 x double> %a, <2 x double> %b, <2 x double> %c) #2
+  %sub.i = fsub <2 x double> <double -0.000000e+00, double -0.000000e+00>, %0
+  ret <2 x double> %sub.i
+}
+
+declare <2 x double> @llvm.x86.fma.vfmadd.pd(<2 x double> %a, <2 x double> %b, <2 x double> %c)
+

llvm/trunk/test/CodeGen/X86/fma-fneg-combine.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown-linux-gnu -mattr=+avx512bw -mattr=+avx512vl -mattr=+avx512dq  | FileCheck %s
 
 ; This test checks combinations of FNEG and FMA intrinsics on AVX-512 target
 ; PR28892
 
 define <16 x float> @test1(<16 x float> %a, <16 x float> %b, <16 x float> %c)  {
 ; CHECK-LABEL: test1:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm2, %zmm2
-; CHECK-NEXT:    vfmadd213ps %zmm2, %zmm1, %zmm0
+; CHECK-NEXT:    vfmsub213ps %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %c
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfmadd.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %sub.i, i16 -1, i32 4) #2
   ret <16 x float> %0
 }
 
 declare <16 x float> @llvm.x86.avx512.mask.vfmadd.ps.512(<16 x float>, <16 x float>, <16 x float>, i16, i32)
 declare <16 x float> @llvm.x86.avx512.mask.vfnmadd.ps.512(<16 x float>, <16 x float>, <16 x float>, i16, i32)
 declare <16 x float> @llvm.x86.avx512.mask.vfnmsub.ps.512(<16 x float>, <16 x float>, <16 x float>, i16, i32)
 
 
 define <16 x float> @test2(<16 x float> %a, <16 x float> %b, <16 x float> %c) {
 ; CHECK-LABEL: test2:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vfmadd213ps %zmm2, %zmm1, %zmm0
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm0, %zmm0
+; CHECK-NEXT:    vfnmsub213ps %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfmadd.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %c, i16 -1, i32 4) #2
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
   ret <16 x float> %sub.i
 }
 
 define <16 x float> @test3(<16 x float> %a, <16 x float> %b, <16 x float> %c)  {
 ; CHECK-LABEL: test3:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vfnmadd213ps %zmm2, %zmm1, %zmm0
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm0, %zmm0
+; CHECK-NEXT:    vfmsub213ps %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfnmadd.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %c, i16 -1, i32 4) #2
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
   ret <16 x float> %sub.i
 }
 
 define <16 x float> @test4(<16 x float> %a, <16 x float> %b, <16 x float> %c) {
 ; CHECK-LABEL: test4:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vfnmsub213ps %zmm2, %zmm1, %zmm0
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm0, %zmm0
+; CHECK-NEXT:    vfmadd213ps %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfnmsub.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %c, i16 -1, i32 4) #2
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
   ret <16 x float> %sub.i
 }
 
 define <16 x float> @test5(<16 x float> %a, <16 x float> %b, <16 x float> %c) {
 ; CHECK-LABEL: test5:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm2, %zmm2
-; CHECK-NEXT:    vfmadd213ps {ru-sae}, %zmm2, %zmm1, %zmm0
+; CHECK-NEXT:    vfmsub213ps {ru-sae}, %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %c
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfmadd.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %sub.i, i16 -1, i32 2) #2
   ret <16 x float> %0
 }
 
 define <16 x float> @test6(<16 x float> %a, <16 x float> %b, <16 x float> %c) {
 ; CHECK-LABEL: test6:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vfnmsub213ps {ru-sae}, %zmm2, %zmm1, %zmm0
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to16}, %zmm0, %zmm0
+; CHECK-NEXT:    vfmadd213ps {ru-sae}, %zmm2, %zmm1, %zmm0
 ; CHECK-NEXT:    retq
 entry:
   %0 = tail call <16 x float> @llvm.x86.avx512.mask.vfnmsub.ps.512(<16 x float> %a, <16 x float> %b, <16 x float> %c, i16 -1, i32 2) #2
   %sub.i = fsub <16 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
   ret <16 x float> %sub.i
 }
 
 
 define <8 x float> @test7(<8 x float> %a, <8 x float> %b, <8 x float> %c) {
 ; CHECK-LABEL: test7:
 ; CHECK:       # BB#0: # %entry
-; CHECK-NEXT:    vfmsub213ps %ymm2, %ymm1, %ymm0
-; CHECK-NEXT:    vxorps {{.*}}(%rip){1to8}, %ymm0, %ymm0
+; CHECK-NEXT:    vfnmadd213ps %ymm2, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %0 = tail call <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float> %a, <8 x float> %b, <8 x float> %c) #2
   %sub.i = fsub <8 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %0
   ret <8 x float> %sub.i
 }
 
 define <8 x float> @test8(<8 x float> %a, <8 x float> %b, <8 x float> %c) {
 ; CHECK-LABEL: test8:
 ; CHECK:       # BB#0: # %entry
 ; CHECK-NEXT:    vxorps {{.*}}(%rip){1to8}, %ymm2, %ymm2
 ; CHECK-NEXT:    vfmsub213ps %ymm2, %ymm1, %ymm0
 ; CHECK-NEXT:    retq
 entry:
   %sub.c = fsub <8 x float> <float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00, float -0.000000e+00>, %c
   %0 = tail call <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float> %a, <8 x float> %b, <8 x float> %sub.c) #2
   ret <8 x float> %0
 }
 
 declare <8 x float> @llvm.x86.fma.vfmsub.ps.256(<8 x float>, <8 x float>, <8 x float>)
 
+
+define <8 x double> @test9(<8 x double> %a, <8 x double> %b, <8 x double> %c) {
+; CHECK-LABEL: test9:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmsub213pd %zmm2, %zmm1, %zmm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <8 x double> @llvm.x86.avx512.mask.vfmadd.pd.512(<8 x double> %a, <8 x double> %b, <8 x double> %c, i8 -1, i32 4) #2
+  %sub.i = fsub <8 x double> <double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00>, %0
+  ret <8 x double> %sub.i
+}
+
+declare <8 x double> @llvm.x86.avx512.mask.vfmadd.pd.512(<8 x double> %a, <8 x double> %b, <8 x double> %c, i8, i32)
+
+define <4 x double> @test10(<4 x double> %a, <4 x double> %b, <4 x double> %c) {
+; CHECK-LABEL: test10:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmsub213pd %ymm2, %ymm1, %ymm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <4 x double> @llvm.x86.avx512.mask.vfmadd.pd.256(<4 x double> %a, <4 x double> %b, <4 x double> %c, i8 -1) #2
+  %sub.i = fsub <4 x double> <double -0.000000e+00, double -0.000000e+00, double -0.000000e+00, double -0.000000e+00>, %0
+  ret <4 x double> %sub.i
+}
+
+declare <4 x double> @llvm.x86.avx512.mask.vfmadd.pd.256(<4 x double> %a, <4 x double> %b, <4 x double> %c, i8)
+
+define <2 x double> @test11(<2 x double> %a, <2 x double> %b, <2 x double> %c) {
+; CHECK-LABEL: test11:
+; CHECK:       # BB#0: # %entry
+; CHECK-NEXT:    vfnmsub213sd %xmm2, %xmm0, %xmm1
+; CHECK-NEXT:    vmovaps %xmm1, %xmm0
+; CHECK-NEXT:    retq
+entry:
+  %0 = tail call <2 x double> @llvm.x86.avx512.mask.vfmadd.sd(<2 x double> %a, <2 x double> %b, <2 x double> %c, i8 -1, i32 4) #2
+  %sub.i = fsub <2 x double> <double -0.000000e+00, double -0.000000e+00>, %0
+  ret <2 x double> %sub.i
+}
+
+declare <2 x double> @llvm.x86.avx512.mask.vfmadd.sd(<2 x double> %a, <2 x double> %b, <2 x double> %c, i8, i32)
+