SelectionDAG: Prefer to combine multiplication with less uses for fma

[wengxt]

Summary

For example:

s6 = s0*s5;
s2 = s6*s6 + s6;
...
s4 = s6*s3;

We notice that it is possible for s2 is folded to fma (s0, s5, fmul (s6 s6)).
This only happens when Aggressive is true, otherwise hasOneUse() check
already prevents from folding the multiplication with more uses.

Test Plan: test/CodeGen/NVPTX/fma-assoc.ll

Patch by Xuetian Weng

[arsenm]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

7429–7430

Grammar fixes: "If we have two choices trying to fold (fadd (fmul u, v), (fmul x, y)), prefer to fold the multiply with fewer uses"

7434–7435

Just use std::swap without using

[jingyue]

LGTM

test/CodeGen/NVPTX/fma-assoc.ll

27

s/choice/choices

#0 is unnecessary

[wengxt]

Address comments

[jingyue]

test/CodeGen/NVPTX/fma-assoc.ll

27

Actually, the metadata is unnecessary too. Without nvvm.annotations, all functions are treated as device functions by llc.

[jingyue]

+Olivier

[jholewinski]

test/CodeGen/NVPTX/fma-assoc.ll

27

Hrm, this should give an error during compilation. Entry functions are not allowed to return values.

[wengxt]

Address jingyue's comment

[arsenm]

LGTM

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitFADDForFMACombine(SDNode *N) {
   if (!HasFMAD && !HasFMA)
     return SDValue();
 
   // Always prefer FMAD to FMA for precision.
   unsigned int PreferredFusedOpcode = HasFMAD ? ISD::FMAD : ISD::FMA;
   bool Aggressive = TLI.enableAggressiveFMAFusion(VT);
   bool LookThroughFPExt = TLI.isFPExtFree(VT);
 
+  // If we have two choices trying to fold (fadd (fmul u, v), (fmul x, y)),
+  // prefer to fold the multiply with fewer uses.
+  if (Aggressive && N0.getOpcode() == ISD::FMUL &&
+      N1.getOpcode() == ISD::FMUL) {
+    if (N0.getNode()->use_size() > N1.getNode()->use_size())
+      std::swap(N0, N1);
+  }
+
   // fold (fadd (fmul x, y), z) -> (fma x, y, z)
   if (N0.getOpcode() == ISD::FMUL &&
       (Aggressive || N0->hasOneUse())) {
     return DAG.getNode(PreferredFusedOpcode, SL, VT,
                        N0.getOperand(0), N0.getOperand(1), N1);
   }
 
   // fold (fadd x, (fmul y, z)) -> (fma y, z, x)

test/CodeGen/NVPTX/fma-assoc.ll

 ; CHECK: fma.rn.f64 %fd{{[0-9]+}}, %fd{{[0-9]+}}, %fd{{[0-9]+}}, %fd{{[0-9]+}};
 ; CHECK: ret;
   %a = fmul double %x, %y
   %b = fmul double %u, %v
   %c = fadd double %a, %b
   %d = fadd double %c, %z
   ret double %d
 }
+
+define double @two_choices(double %val1, double %val2) {
+; CHECK-LABEL: .visible .entry two_choices
+; CHECK: mul.f64
+; CHECK-NOT: mul.f64
+; CHECK: fma.rn.f64
+  %1 = fmul double %val1, %val2
+  %2 = fmul double %1, %1
+  %3 = fadd double %1, %2
+
+  ret double %3
+}
+
+!nvvm.annotations = !{!0}
+
+!0 = !{double (double, double)* @two_choices, !"kernel", i32 1}