[DAGCombiner][VP] Add DAGCombine for merging VP_FADD and VP_FMUL to VP_FMA.

[fakepaper56]

Summary

The patch does two DAGcombines:
fold (vp_fadd a, (vp_mul b, c)) to (vp_fma b, c, a)
fold (vp_fadd (vp_mul a, b), c) to (vp_fma a, b, c)

[fakepaper56]

Using clang-format.

[craig.topper]

llvm/test/CodeGen/RISCV/rvv/fold-fadd-and-fmul.ll

1 ↗

(On Diff #431586)

Test should probably have -vp- in it's name

[liaolucy]

When I test tsvc.

The IR is:

@llvm.fmuladd.nxv2f32(<vscale x 2 x float>.....

Not

@llvm.riscv.vfmul.nxv2f32.nxv2f32(<vscale x 2 x float> ......
@llvm.riscv.vfadd.nxv2f32.nxv2f32(<vscale x 2 x float> ......

So I am not sure, we need merge vp.fmul and vp.fadd to vp.fma. maybe vp.fma is enough?

[fakepaper56]

Rename fold-fadd-and-fmul.ll to fold-vp-fadd-and-vp-fmul.ll.

[fakepaper56]

In D126273#3533580, @liaolucy wrote:

When I test tsvc.

The IR is:

@llvm.fmuladd.nxv2f32(<vscale x 2 x float>.....

Not

@llvm.riscv.vfmul.nxv2f32.nxv2f32(<vscale x 2 x float> ......
@llvm.riscv.vfadd.nxv2f32.nxv2f32(<vscale x 2 x float> ......

So I am not sure, we need merge vp.fmul and vp.fadd to vp.fma. maybe vp.fma is enough?

I am not sure what is tscv. Could you tell its full name?
But I think the case may happen when loop vectorizer generates vector prediction intrinsics.

[liaolucy]

In D126273#3534004, @fakepaper56 wrote:

In D126273#3533580, @liaolucy wrote:

When I test tsvc.

The IR is:

@llvm.fmuladd.nxv2f32(<vscale x 2 x float>.....

Not

@llvm.riscv.vfmul.nxv2f32.nxv2f32(<vscale x 2 x float> ......
@llvm.riscv.vfadd.nxv2f32.nxv2f32(<vscale x 2 x float> ......

So I am not sure, we need merge vp.fmul and vp.fadd to vp.fma. maybe vp.fma is enough?

I am not sure what is tscv. Could you tell its full name?
But I think the case may happen when loop vectorizer generates vector prediction intrinsics.

tsvc:
https://github.com/llvm/llvm-test-suite/blob/main/MultiSource/Benchmarks/TSVC/tsc.inc

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

23061

It'd be very useful if we pulled stuff like this out and shared it between all the various FMA generating combines.

[simoll]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

23061

Yes. It is possible to lift the existing DAGCombiner Patterns to work on VP SDNodes as well as on regular SDNodes.
I've implemented this in the LLVM stack for SX-Aurora under the LLVM license, ie it could be upstreamed:

https://github.com/sx-aurora-dev/llvm-project/blob/feature/generalized_pattern_rewriting/llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp#L13859

You could rephrase this patch using the generalized pattern-rewriting technique. I'd happy to help with that!

The same applies to https://reviews.llvm.org/D121187

[simoll]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

23061

Some more info on the generalized pattern-matching thing:

The visitFADDForFMACombine function is templatized. The template parameter abstracts away the actual matching and SDNode creation. The flow of the code is the same (SDNode matching and creation is re-directed through the matcher class, that's all).

The templated function is instantiated twice, once with the EmptyMatchContext for regular SDNodes and once with the VPMatchContext for VP-SDNodes.

[craig.topper]

In D126273#3533580, @liaolucy wrote:

When I test tsvc.

The IR is:

@llvm.fmuladd.nxv2f32(<vscale x 2 x float>.....

Not

@llvm.riscv.vfmul.nxv2f32.nxv2f32(<vscale x 2 x float> ......
@llvm.riscv.vfadd.nxv2f32.nxv2f32(<vscale x 2 x float> ......

So I am not sure, we need merge vp.fmul and vp.fadd to vp.fma. maybe vp.fma is enough?

fmuladd is use with -ffp-contract=on(the default). fmul+fadd is used with -ffast-math or -ffp-contract=fast or -Ofast.

[liaolucy]

In D126273#3534429, @craig.topper wrote:

In D126273#3533580, @liaolucy wrote:

When I test tsvc.

The IR is:

@llvm.fmuladd.nxv2f32(<vscale x 2 x float>.....

Not

@llvm.riscv.vfmul.nxv2f32.nxv2f32(<vscale x 2 x float> ......
@llvm.riscv.vfadd.nxv2f32.nxv2f32(<vscale x 2 x float> ......

So I am not sure, we need merge vp.fmul and vp.fadd to vp.fma. maybe vp.fma is enough?

fmuladd is use with -ffp-contract=on(the default). fmul+fadd is used with -ffast-math or -ffp-contract=fast or -Ofast.

Got it. Thank you very much.

[craig.topper]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

23055

What about these early exits from the original function

// If the addition is not contractable, do not combine.                        
if (!AllowFusionGlobally && !N->getFlags().hasAllowContract())                 
  return SDValue();                                                            
                                                                               
if (TLI.generateFMAsInMachineCombiner(VT, OptLevel))                           
  return SDValue();

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue visitFP16_TO_FP(SDNode *N);
     SDValue visitVECREDUCE(SDNode *N);
     SDValue visitVPOp(SDNode *N);
 
     SDValue visitFADDForFMACombine(SDNode *N);
     SDValue visitFSUBForFMACombine(SDNode *N);
     SDValue visitFMULForFMADistributiveCombine(SDNode *N);
 
+    SDValue visitVPFADDForVPFMACombine(SDNode *N);
+
     SDValue XformToShuffleWithZero(SDNode *N);
     bool reassociationCanBreakAddressingModePattern(unsigned Opc,
                                                     const SDLoc &DL,
                                                     SDNode *N,
                                                     SDValue N0,
                                                     SDValue N1);
     SDValue reassociateOpsCommutative(unsigned Opc, const SDLoc &DL, SDValue N0,
                                       SDValue N1);
if ((Opcode == ISD::VECREDUCE_OR &&
         (Opcode == ISD::VECREDUCE_AND &&
          (N0.getOperand(0).isUndef() || isAllOnesOrAllOnesSplat(Vec))))
       return DAG.getNode(Opcode, SDLoc(N), N->getValueType(0), Subvec);
   }
 
   return SDValue();
 }
 
+/// Try to perform VP_FMA combining on a given VP_FADD node.
+SDValue DAGCombiner::visitVPFADDForVPFMACombine(SDNode *N) {
+  SDValue N0 = N->getOperand(0);
+  SDValue N1 = N->getOperand(1);
+  SDValue Mask = N->getOperand(2);
+  SDValue VL = N->getOperand(3);
+  EVT VT = N->getValueType(0);
+  SDLoc SL(N);
+
+  const TargetOptions &Options = DAG.getTarget().Options;
+
+  bool HasFMA =
+      TLI.isFMAFasterThanFMulAndFAdd(DAG.getMachineFunction(), VT) &&
+      (!LegalOperations || TLI.isOperationLegalOrCustom(ISD::VP_FMA, VT));
+
+  if (!HasFMA)
+    return SDValue();
+
+  bool AllowFusionGlobally =
+      (Options.AllowFPOpFusion == FPOpFusion::Fast || Options.UnsafeFPMath);
+
+  // Is the node an VP_FMUL and contractable either due to global flags or
+  // SDNodeFlags.
+  auto isContractableVPFMUL = [AllowFusionGlobally](SDValue N) {
+    if (N.getOpcode() != ISD::VP_FMUL)
+      return false;
+    return AllowFusionGlobally || N->getFlags().hasAllowContract();
+  };
+
+  // fold (vp_fadd (vp_fmul x, y), z) -> (vp_fma x, y, z)
+  if (isContractableVPFMUL(N0) && N0->hasOneUse())
+    return DAG.getNode(ISD::VP_FMA, SL, VT, N0.getOperand(0), N0.getOperand(1),
+                       N1, Mask, VL);
+
+  // fold (vp_fadd x, (vp_fmul y, z)) -> (vp_fma y, z, x)
+  // Note: Commutes VP_FADD operands.
+  if (isContractableVPFMUL(N1) && N1->hasOneUse())
+    return DAG.getNode(ISD::VP_FMA, SL, VT, N1.getOperand(0), N1.getOperand(1),
+                       N0, Mask, VL);
+
+  return SDValue();
+}
+
 SDValue DAGCombiner::visitVPOp(SDNode *N) {
   // VP operations in which all vector elements are disabled - either by
   // determining that the mask is all false or that the EVL is 0 - can be
   // eliminated.
   bool AreAllEltsDisabled = false;
   if (auto EVLIdx = ISD::getVPExplicitVectorLengthIdx(N->getOpcode()))
     AreAllEltsDisabled |= isNullConstant(N->getOperand(*EVLIdx));
   if (auto MaskIdx = ISD::getVPMaskIdx(N->getOpcode()))
     AreAllEltsDisabled |=
         ISD::isConstantSplatVectorAllZeros(N->getOperand(*MaskIdx).getNode());
 
   // This is the only generic VP combine we support for now.
-  if (!AreAllEltsDisabled)
+  if (!AreAllEltsDisabled) {
+    switch (N->getOpcode()) {
+    case ISD::VP_FADD:
+      return visitVPFADDForVPFMACombine(N);
+    }
     return SDValue();
+  }
 
   // Binary operations can be replaced by UNDEF.
   if (ISD::isVPBinaryOp(N->getOpcode()))
     return DAG.getUNDEF(N->getValueType(0));
 
   // VP Memory operations can be replaced by either the chain (stores) or the
   // chain + undef (loads).
   if (const auto *MemSD = dyn_cast<MemSDNode>(N)) {

llvm/test/CodeGen/RISCV/rvv/fold-vp-fadd-and-vp-fmul.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=riscv64 -mattr=+v -target-abi=lp64d -verify-machineinstrs < %s | FileCheck %s
+
+declare <vscale x 1 x double> @llvm.vp.fmul.nxv1f64(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x i1> %m, i32 %vl)
+declare <vscale x 1 x double> @llvm.vp.fadd.nxv1f64(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x i1> %m, i32 %vl)
+
+define <vscale x 1 x double> @test1(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x double> %z, <vscale x 1 x i1> %m, i32 %vl) {
+; CHECK-LABEL: test1:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    slli a0, a0, 32
+; CHECK-NEXT:    srli a0, a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e64, m1, tu, mu
+; CHECK-NEXT:    vfmadd.vv v9, v8, v10, v0.t
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %1 = call fast <vscale x 1 x double> @llvm.vp.fmul.nxv1f64(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x i1> %m, i32 %vl)
+  %2 = call fast <vscale x 1 x double> @llvm.vp.fadd.nxv1f64(<vscale x 1 x double> %1, <vscale x 1 x double> %z, <vscale x 1 x i1> %m, i32 %vl)
+  ret <vscale x 1 x double> %2
+}
+
+define <vscale x 1 x double> @test2(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x double> %z, <vscale x 1 x i1> %m, i32 %vl) {
+; CHECK-LABEL: test2:
+; CHECK:       # %bb.0:
+; CHECK-NEXT:    slli a0, a0, 32
+; CHECK-NEXT:    srli a0, a0, 32
+; CHECK-NEXT:    vsetvli zero, a0, e64, m1, tu, mu
+; CHECK-NEXT:    vfmadd.vv v9, v8, v10, v0.t
+; CHECK-NEXT:    vmv1r.v v8, v9
+; CHECK-NEXT:    ret
+  %1 = call fast <vscale x 1 x double> @llvm.vp.fmul.nxv1f64(<vscale x 1 x double> %x, <vscale x 1 x double> %y, <vscale x 1 x i1> %m, i32 %vl)
+  %2 = call fast <vscale x 1 x double> @llvm.vp.fadd.nxv1f64(<vscale x 1 x double> %z, <vscale x 1 x double> %1, <vscale x 1 x i1> %m, i32 %vl)
+  ret <vscale x 1 x double> %2
+}