[DAGCombine] Allow FMA combine with both FMA and FMAD

[critson]

Summary

Without this change only the preferred fusion opcode is tested
when attempting to combine FMA operations.
If both FMA and FMAD are available then FMA ops formed prior to
legalization will not be merged post legalization as FMAD becomes
the preferred fusion opcode.

[foad]

s/FMAC/FMAD/g in the commit message, since that's what SelectionDAG calls it.

[foad]

will not be merged post legalization as FMAC becomes
the preferred fusion opcode

I'm not sure wha you mean about FMAD "becoming the preferred fusion opcode". Is that just an AMDGPU quirk? Or is there something in generic code that prefers FMA before legalization and FMAD after??

llvm/test/CodeGen/AMDGPU/dagcombine-fma-fmad.ll

5

Could use GCN-COUNT-13: v_fma.

But maybe it would be better to generate the checks for this file? It's not immediately obvious (to me) why 13 is the optimal number of fmas. There are more than 13 fmuls and fadds and fsubs in the IR.

[critson]

In D108619#2962032, @foad wrote:

will not be merged post legalization as FMAC becomes
the preferred fusion opcode

I'm not sure wha you mean about FMAD "becoming the preferred fusion opcode". Is that just an AMDGPU quirk? Or is there something in generic code that prefers FMA before legalization and FMAD after??

bool HasFMAD = (LegalOperations && TLI.isFMADLegal(DAG, N));

bool HasFMA =
    TLI.isFMAFasterThanFMulAndFAdd(DAG.getMachineFunction(), VT) &&
    (!LegalOperations || TLI.isOperationLegalOrCustom(ISD::FMA, VT));

unsigned PreferredFusedOpcode = HasFMAD ? ISD::FMAD : ISD::FMA;

This is most probably an AMDGPU quirk.
FMAD is only available when LegalOperations is true, i.e. post legalize vector ops. Where as FMA is always legal if backend sets it to be Legal or Custom. I don't know the origin of this behaviour; however, the only backend that marks ISD::FMAD legal appears to be AMDGPU (SI and R600).
So preferred fusion opcode will only be FMAD on AMDGPU.

llvm/test/CodeGen/AMDGPU/dagcombine-fma-fmad.ll

5

I'll generate the checks for this file and precommit.
This example was specifically reduced to take the code paths changed by this patch.
With the patch the number of instructions falls, number of FMAs goes from 14 to 13.

[critson]

• Rebase onto pre-committed test

[foad]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13056

Why do you need the HasFMA and HasFMAD checks here? I can see that you don't want to create new FMA instructions if FMA is not legal, but I can't see why you wouldn't want to combine existing FMAs.

In other words can this just be return Opcode == ISD::FMA || Opcode == ISD::FMAD;?

[critson]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13056

Sure, this yields some test changes (instruction count reductions).

[critson]

• Incorporate reviewer comments

[foad]

Looks fine to me, with or without the tweak suggested inline.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13107

Could use FMA.getOpcode() instead of PreferredFusedOpcode here, to try to preserve the original opcode of the A*B+something part of the expression. But that's getting pretty subtle and I'm not sure if it will make any practical difference.

The same goes for any other parts of combines which preserve an fma(d) instead of creating a new one from scratch.

[critson]

• Tweak to preserve original fusion opcode when forming new FMAs

[foad]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13107

What I meant was, use FMA.getOpcode() for A*B+something because it's preserving an existing fma(d) from the input, but still use PreferredFusedOpcode for C*D+something because it is a new fma(d) that we are creating by fusing an fmul and and fadd from the input.

I realise this is pretty subtle and I'm not sure it will make any practical difference.

[foad]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13108

On second thoughts, perhaps there's no point trying to preserve the original opcode for the A*B+something part, because the something is different, so it's impossible to retain the (fused vs unfused) rounding behaviour of the original opcode.

So perhaps your original approach of using DAG.getNode(PreferredFusedOpcode, ...) everywhere is the best we can do. Sorry for the noise.

[critson]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

13108

No worries, I have spent a while thinking about it too.
I tested both versions so far on image output tests and could not detect and meaningful difference.
I also implemented the mixed preservation intended with your comment.
However, let's just go with the original.

[critson]

• Revert opcode preservation changes

[foad]

Still LGTM, thanks!

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitFADDForFMACombine(SDNode *N) {
 
   if (TLI.generateFMAsInMachineCombiner(VT, OptLevel))
     return SDValue();
 
   // Always prefer FMAD to FMA for precision.
   unsigned PreferredFusedOpcode = HasFMAD ? ISD::FMAD : ISD::FMA;
   bool Aggressive = TLI.enableAggressiveFMAFusion(VT);
 
+  auto isFusedOp = [&](SDValue N) {
+    unsigned Opcode = N.getOpcode();
+    return Opcode == ISD::FMA || Opcode == ISD::FMAD;
+  };
+
   // Is the node an FMUL and contractable either due to global flags or
   // SDNodeFlags.
   auto isContractableFMUL = [AllowFusionGlobally](SDValue N) {
     if (N.getOpcode() != ISD::FMUL)
       return false;
     return AllowFusionGlobally || N->getFlags().hasAllowContract();
   };
   // If we have two choices trying to fold (fadd (fmul u, v), (fmul x, y)),
if (isContractableFMUL(N1) && (Aggressive || N1->hasOneUse())) {
     return DAG.getNode(PreferredFusedOpcode, SL, VT, N1.getOperand(0),
                        N1.getOperand(1), N0);
   }
 
   // fadd (fma A, B, (fmul C, D)), E --> fma A, B, (fma C, D, E)
   // fadd E, (fma A, B, (fmul C, D)) --> fma A, B, (fma C, D, E)
   // This requires reassociation because it changes the order of operations.
   SDValue FMA, E;
-  if (CanReassociate && N0.getOpcode() == PreferredFusedOpcode &&
+  if (CanReassociate && isFusedOp(N0) &&
       N0.getOperand(2).getOpcode() == ISD::FMUL && N0.hasOneUse() &&
       N0.getOperand(2).hasOneUse()) {
     FMA = N0;
     E = N1;
-  } else if (CanReassociate && N1.getOpcode() == PreferredFusedOpcode &&
+  } else if (CanReassociate && isFusedOp(N1) &&
              N1.getOperand(2).getOpcode() == ISD::FMUL && N1.hasOneUse() &&
              N1.getOperand(2).hasOneUse()) {
     FMA = N1;
     E = N0;
   }
   if (FMA && E) {
     SDValue A = FMA.getOperand(0);
     SDValue B = FMA.getOperand(1);
     SDValue C = FMA.getOperand(2).getOperand(0);
     SDValue D = FMA.getOperand(2).getOperand(1);
     SDValue CDE = DAG.getNode(PreferredFusedOpcode, SL, VT, C, D, E);
     return DAG.getNode(PreferredFusedOpcode, SL, VT, A, B, CDE);
   }
 
   // Look through FP_EXTEND nodes to do more combining.
 
   // fold (fadd (fpext (fmul x, y)), z) -> (fma (fpext x), (fpext y), z)
   if (N0.getOpcode() == ISD::FP_EXTEND) {
     SDValue N00 = N0.getOperand(0);
     if (isContractableFMUL(N00) &&
         TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
if (Aggressive) {
     auto FoldFAddFMAFPExtFMul = [&](SDValue X, SDValue Y, SDValue U, SDValue V,
                                     SDValue Z) {
       return DAG.getNode(PreferredFusedOpcode, SL, VT, X, Y,
                          DAG.getNode(PreferredFusedOpcode, SL, VT,
                                      DAG.getNode(ISD::FP_EXTEND, SL, VT, U),
                                      DAG.getNode(ISD::FP_EXTEND, SL, VT, V),
                                      Z));
     };
-    if (N0.getOpcode() == PreferredFusedOpcode) {
+    if (isFusedOp(N0)) {
       SDValue N02 = N0.getOperand(2);
       if (N02.getOpcode() == ISD::FP_EXTEND) {
         SDValue N020 = N02.getOperand(0);
         if (isContractableFMUL(N020) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N020.getValueType())) {
           return FoldFAddFMAFPExtFMul(N0.getOperand(0), N0.getOperand(1),
                                       N020.getOperand(0), N020.getOperand(1),
auto FoldFAddFPExtFMAFMul = [&](SDValue X, SDValue Y, SDValue U, SDValue V,
           PreferredFusedOpcode, SL, VT, DAG.getNode(ISD::FP_EXTEND, SL, VT, X),
           DAG.getNode(ISD::FP_EXTEND, SL, VT, Y),
           DAG.getNode(PreferredFusedOpcode, SL, VT,
                       DAG.getNode(ISD::FP_EXTEND, SL, VT, U),
                       DAG.getNode(ISD::FP_EXTEND, SL, VT, V), Z));
     };
     if (N0.getOpcode() == ISD::FP_EXTEND) {
       SDValue N00 = N0.getOperand(0);
-      if (N00.getOpcode() == PreferredFusedOpcode) {
+      if (isFusedOp(N00)) {
         SDValue N002 = N00.getOperand(2);
         if (isContractableFMUL(N002) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N00.getValueType())) {
           return FoldFAddFPExtFMAFMul(N00.getOperand(0), N00.getOperand(1),
                                       N002.getOperand(0), N002.getOperand(1),
                                       N1);
         }
       }
     }
 
     // fold (fadd x, (fma y, z, (fpext (fmul u, v)))
     //   -> (fma y, z, (fma (fpext u), (fpext v), x))
-    if (N1.getOpcode() == PreferredFusedOpcode) {
+    if (isFusedOp(N1)) {
       SDValue N12 = N1.getOperand(2);
       if (N12.getOpcode() == ISD::FP_EXTEND) {
         SDValue N120 = N12.getOperand(0);
         if (isContractableFMUL(N120) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N120.getValueType())) {
           return FoldFAddFMAFPExtFMul(N1.getOperand(0), N1.getOperand(1),
                                       N120.getOperand(0), N120.getOperand(1),
                                       N0);
         }
       }
     }
 
     // fold (fadd x, (fpext (fma y, z, (fmul u, v)))
     //   -> (fma (fpext y), (fpext z), (fma (fpext u), (fpext v), x))
     // FIXME: This turns two single-precision and one double-precision
     // operation into two double-precision operations, which might not be
     // interesting for all targets, especially GPUs.
     if (N1.getOpcode() == ISD::FP_EXTEND) {
       SDValue N10 = N1.getOperand(0);
-      if (N10.getOpcode() == PreferredFusedOpcode) {
+      if (isFusedOp(N10)) {
         SDValue N102 = N10.getOperand(2);
         if (isContractableFMUL(N102) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N10.getValueType())) {
           return FoldFAddFPExtFMAFMul(N10.getOperand(0), N10.getOperand(1),
                                       N102.getOperand(0), N102.getOperand(1),
                                       N0);
         }
auto isReassociable = [Options](SDNode *N) {
     return Options.UnsafeFPMath || N->getFlags().hasAllowReassociation();
   };
 
   auto isContractableAndReassociableFMUL = [isContractableFMUL,
                                             isReassociable](SDValue N) {
     return isContractableFMUL(N) && isReassociable(N.getNode());
   };
 
+  auto isFusedOp = [&](SDValue N) {
+    unsigned Opcode = N.getOpcode();
+    return Opcode == ISD::FMA || Opcode == ISD::FMAD;
+  };
+
   // More folding opportunities when target permits.
   if (Aggressive && isReassociable(N)) {
     bool CanFuse = Options.UnsafeFPMath || N->getFlags().hasAllowContract();
     // fold (fsub (fma x, y, (fmul u, v)), z)
     //   -> (fma x, y (fma u, v, (fneg z)))
-    if (CanFuse && N0.getOpcode() == PreferredFusedOpcode &&
+    if (CanFuse && isFusedOp(N0) &&
         isContractableAndReassociableFMUL(N0.getOperand(2)) &&
         N0->hasOneUse() && N0.getOperand(2)->hasOneUse()) {
       return DAG.getNode(PreferredFusedOpcode, SL, VT, N0.getOperand(0),
                          N0.getOperand(1),
                          DAG.getNode(PreferredFusedOpcode, SL, VT,
                                      N0.getOperand(2).getOperand(0),
                                      N0.getOperand(2).getOperand(1),
                                      DAG.getNode(ISD::FNEG, SL, VT, N1)));
     }
 
     // fold (fsub x, (fma y, z, (fmul u, v)))
     //   -> (fma (fneg y), z, (fma (fneg u), v, x))
-    if (CanFuse && N1.getOpcode() == PreferredFusedOpcode &&
+    if (CanFuse && isFusedOp(N1) &&
         isContractableAndReassociableFMUL(N1.getOperand(2)) &&
         N1->hasOneUse() && NoSignedZero) {
       SDValue N20 = N1.getOperand(2).getOperand(0);
       SDValue N21 = N1.getOperand(2).getOperand(1);
       return DAG.getNode(
           PreferredFusedOpcode, SL, VT,
           DAG.getNode(ISD::FNEG, SL, VT, N1.getOperand(0)), N1.getOperand(1),
           DAG.getNode(PreferredFusedOpcode, SL, VT,
                       DAG.getNode(ISD::FNEG, SL, VT, N20), N21, N0));
     }
 
     // fold (fsub (fma x, y, (fpext (fmul u, v))), z)
     //   -> (fma x, y (fma (fpext u), (fpext v), (fneg z)))
-    if (N0.getOpcode() == PreferredFusedOpcode &&
-        N0->hasOneUse()) {
+    if (isFusedOp(N0) && N0->hasOneUse()) {
       SDValue N02 = N0.getOperand(2);
       if (N02.getOpcode() == ISD::FP_EXTEND) {
         SDValue N020 = N02.getOperand(0);
         if (isContractableAndReassociableFMUL(N020) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N020.getValueType())) {
           return DAG.getNode(
               PreferredFusedOpcode, SL, VT, N0.getOperand(0), N0.getOperand(1),
if (Aggressive && isReassociable(N)) {
     // fold (fsub (fpext (fma x, y, (fmul u, v))), z)
     //   -> (fma (fpext x), (fpext y),
     //           (fma (fpext u), (fpext v), (fneg z)))
     // FIXME: This turns two single-precision and one double-precision
     // operation into two double-precision operations, which might not be
     // interesting for all targets, especially GPUs.
     if (N0.getOpcode() == ISD::FP_EXTEND) {
       SDValue N00 = N0.getOperand(0);
-      if (N00.getOpcode() == PreferredFusedOpcode) {
+      if (isFusedOp(N00)) {
         SDValue N002 = N00.getOperand(2);
         if (isContractableAndReassociableFMUL(N002) &&
             TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                                 N00.getValueType())) {
           return DAG.getNode(
               PreferredFusedOpcode, SL, VT,
               DAG.getNode(ISD::FP_EXTEND, SL, VT, N00.getOperand(0)),
               DAG.getNode(ISD::FP_EXTEND, SL, VT, N00.getOperand(1)),
               DAG.getNode(
                   PreferredFusedOpcode, SL, VT,
                   DAG.getNode(ISD::FP_EXTEND, SL, VT, N002.getOperand(0)),
                   DAG.getNode(ISD::FP_EXTEND, SL, VT, N002.getOperand(1)),
                   DAG.getNode(ISD::FNEG, SL, VT, N1)));
         }
       }
     }
 
     // fold (fsub x, (fma y, z, (fpext (fmul u, v))))
     //   -> (fma (fneg y), z, (fma (fneg (fpext u)), (fpext v), x))
-    if (N1.getOpcode() == PreferredFusedOpcode &&
-        N1.getOperand(2).getOpcode() == ISD::FP_EXTEND &&
+    if (isFusedOp(N1) && N1.getOperand(2).getOpcode() == ISD::FP_EXTEND &&
         N1->hasOneUse()) {
       SDValue N120 = N1.getOperand(2).getOperand(0);
       if (isContractableAndReassociableFMUL(N120) &&
           TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                               N120.getValueType())) {
         SDValue N1200 = N120.getOperand(0);
         SDValue N1201 = N120.getOperand(1);
         return DAG.getNode(
             PreferredFusedOpcode, SL, VT,
             DAG.getNode(ISD::FNEG, SL, VT, N1.getOperand(0)), N1.getOperand(1),
             DAG.getNode(PreferredFusedOpcode, SL, VT,
                         DAG.getNode(ISD::FNEG, SL, VT,
                                     DAG.getNode(ISD::FP_EXTEND, SL, VT, N1200)),
                         DAG.getNode(ISD::FP_EXTEND, SL, VT, N1201), N0));
       }
     }
 
     // fold (fsub x, (fpext (fma y, z, (fmul u, v))))
     //   -> (fma (fneg (fpext y)), (fpext z),
     //           (fma (fneg (fpext u)), (fpext v), x))
     // FIXME: This turns two single-precision and one double-precision
     // operation into two double-precision operations, which might not be
     // interesting for all targets, especially GPUs.
-    if (N1.getOpcode() == ISD::FP_EXTEND &&
-        N1.getOperand(0).getOpcode() == PreferredFusedOpcode) {
+    if (N1.getOpcode() == ISD::FP_EXTEND && isFusedOp(N1.getOperand(0))) {
       SDValue CvtSrc = N1.getOperand(0);
       SDValue N100 = CvtSrc.getOperand(0);
       SDValue N101 = CvtSrc.getOperand(1);
       SDValue N102 = CvtSrc.getOperand(2);
       if (isContractableAndReassociableFMUL(N102) &&
           TLI.isFPExtFoldable(DAG, PreferredFusedOpcode, VT,
                               CvtSrc.getValueType())) {
         SDValue N1020 = N102.getOperand(0);

llvm/test/CodeGen/AMDGPU/dagcombine-fma-fmad.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -march=amdgcn -mcpu=gfx1010 -verify-machineinstrs < %s | FileCheck -enable-var-scope -check-prefix=GCN %s
 
 define amdgpu_ps float @_amdgpu_ps_main() #0 {
 ; GCN-LABEL: _amdgpu_ps_main:
 ; GCN:       ; %bb.0: ; %.entry
 ; GCN-NEXT:    s_mov_b32 s0, 0
 ; GCN-NEXT:    s_mov_b32 s1, s0
 ; GCN-NEXT:    s_mov_b32 s2, s0
 ; GCN-NEXT:    s_mov_b32 s3, s0
 ; GCN-NEXT:    s_mov_b32 s4, s0
 ; GCN-NEXT:    s_mov_b32 s5, s0
 ; GCN-NEXT:    s_mov_b32 s6, s0
 ; GCN-NEXT:    s_mov_b32 s7, s0
 ; GCN-NEXT:    image_sample v[0:1], v[0:1], s[0:7], s[0:3] dmask:0x3 dim:SQ_RSRC_IMG_2D
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    s_clause 0x2
 ; GCN-NEXT:    image_sample v2, v[0:1], s[0:7], s[0:3] dmask:0x4 dim:SQ_RSRC_IMG_2D
 ; GCN-NEXT:    image_sample v3, v[0:1], s[0:7], s[0:3] dmask:0x1 dim:SQ_RSRC_IMG_2D
 ; GCN-NEXT:    image_load v4, v[0:1], s[0:7] dmask:0x4 dim:SQ_RSRC_IMG_2D unorm
 ; GCN-NEXT:    s_clause 0x3
 ; GCN-NEXT:    s_buffer_load_dword s24, s[0:3], 0x5c
 ; GCN-NEXT:    s_buffer_load_dword s28, s[0:3], 0x7c
 ; GCN-NEXT:    s_buffer_load_dword s29, s[0:3], 0xc0
 ; GCN-NEXT:    s_waitcnt_depctr 0xffe3
 ; GCN-NEXT:    s_nop 0
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[0:3], s[0:3], 0x40
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    v_sub_f32_e64 v5, s24, s28
-; GCN-NEXT:    v_add_f32_e64 v7, s29, -1.0
 ; GCN-NEXT:    s_clause 0x1
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[4:7], s[0:3], 0x50
 ; GCN-NEXT:    s_nop 0
 ; GCN-NEXT:    s_buffer_load_dword s0, s[0:3], 0x2c
 ; GCN-NEXT:    v_fma_f32 v1, v1, v5, s28
+; GCN-NEXT:    v_add_f32_e64 v5, s29, -1.0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_clause 0x3
+; GCN-NEXT:    s_clause 0x4
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[8:11], s[0:3], 0x60
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[12:15], s[0:3], 0x20
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[16:19], s[0:3], 0x0
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[20:23], s[0:3], 0x70
-; GCN-NEXT:    v_max_f32_e64 v6, s0, s0 clamp
 ; GCN-NEXT:    s_buffer_load_dwordx4 s[24:27], s[0:3], 0x10
-; GCN-NEXT:    v_sub_f32_e32 v9, s0, v1
+; GCN-NEXT:    v_max_f32_e64 v6, s0, s0 clamp
+; GCN-NEXT:    v_sub_f32_e32 v8, s0, v1
 ; GCN-NEXT:    s_mov_b32 s0, 0x3c23d70a
-; GCN-NEXT:    v_mul_f32_e32 v5, s2, v6
-; GCN-NEXT:    v_fma_f32 v8, -s2, v6, s6
-; GCN-NEXT:    v_fmac_f32_e32 v1, v6, v9
-; GCN-NEXT:    v_fma_f32 v7, v6, v7, 1.0
-; GCN-NEXT:    v_fmac_f32_e32 v5, v8, v6
+; GCN-NEXT:    v_fma_f32 v7, -s2, v6, s6
+; GCN-NEXT:    v_fmac_f32_e32 v1, v6, v8
+; GCN-NEXT:    v_fma_f32 v5, v6, v5, 1.0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mul_f32_e32 v8, s10, v0
+; GCN-NEXT:    v_mul_f32_e32 v9, s10, v0
 ; GCN-NEXT:    v_fma_f32 v0, -v0, s10, s14
-; GCN-NEXT:    v_fmac_f32_e32 v8, v0, v6
-; GCN-NEXT:    v_sub_f32_e32 v0, v1, v7
-; GCN-NEXT:    v_fmac_f32_e32 v7, v0, v6
+; GCN-NEXT:    v_fmac_f32_e32 v9, v0, v6
+; GCN-NEXT:    v_sub_f32_e32 v0, v1, v5
+; GCN-NEXT:    v_fmac_f32_e32 v5, v0, v6
 ; GCN-NEXT:    s_waitcnt vmcnt(2)
-; GCN-NEXT:    v_mul_f32_e32 v9, s18, v2
+; GCN-NEXT:    v_mad_f32 v10, s2, v6, v2
+; GCN-NEXT:    v_mul_f32_e32 v8, s18, v2
 ; GCN-NEXT:    s_waitcnt vmcnt(1)
 ; GCN-NEXT:    v_mul_f32_e32 v3, s22, v3
-; GCN-NEXT:    v_add_f32_e32 v5, v2, v5
-; GCN-NEXT:    v_mul_f32_e32 v1, v9, v6
-; GCN-NEXT:    v_mul_f32_e32 v9, v6, v3
-; GCN-NEXT:    v_fmac_f32_e64 v8, -v6, v3
+; GCN-NEXT:    v_mac_f32_e32 v10, v7, v6
+; GCN-NEXT:    v_mul_f32_e32 v1, v8, v6
+; GCN-NEXT:    v_mul_f32_e32 v7, v6, v3
+; GCN-NEXT:    v_fmac_f32_e64 v9, -v6, v3
 ; GCN-NEXT:    s_waitcnt vmcnt(0)
-; GCN-NEXT:    v_add_f32_e32 v4, v4, v5
+; GCN-NEXT:    v_add_f32_e32 v3, v4, v10
 ; GCN-NEXT:    v_fma_f32 v0, v2, s26, -v1
-; GCN-NEXT:    v_fmac_f32_e32 v9, v8, v6
-; GCN-NEXT:    v_mul_f32_e32 v3, v4, v6
-; GCN-NEXT:    v_fma_f32 v4, v7, s0, 0x3ca3d70a
+; GCN-NEXT:    v_fma_f32 v4, v5, s0, 0x3ca3d70a
+; GCN-NEXT:    v_fmac_f32_e32 v7, v9, v6
+; GCN-NEXT:    v_mul_f32_e32 v3, v3, v6
 ; GCN-NEXT:    v_fmac_f32_e32 v1, v0, v6
 ; GCN-NEXT:    v_mul_f32_e32 v0, v2, v6
-; GCN-NEXT:    v_mul_f32_e32 v2, v9, v4
+; GCN-NEXT:    v_mul_f32_e32 v2, v7, v4
 ; GCN-NEXT:    v_mul_f32_e32 v1, v3, v1
 ; GCN-NEXT:    v_fmac_f32_e32 v1, v2, v0
 ; GCN-NEXT:    v_max_f32_e32 v0, 0, v1
 ; GCN-NEXT:    ; return to shader part epilog
 .entry:
   %0 = call <3 x float> @llvm.amdgcn.image.sample.2d.v3f32.f32(i32 7, float undef, float undef, <8 x i32> undef, <4 x i32> undef, i1 false, i32 0, i32 0)
   %.i2243 = extractelement <3 x float> %0, i32 2
   %1 = call <3 x i32> @llvm.amdgcn.s.buffer.load.v3i32(<4 x i32> undef, i32 0, i32 0)

llvm/test/CodeGen/AMDGPU/fpext-free.ll

entry:
   ret float %add
 }
 
 ; fold (fadd (fma x, y, (fpext (fmul u, v))), z)
 ;   -> (fma x, y, (fma (fpext u), (fpext v), z))
 
 ; GCN-LABEL: {{^}}fadd_fma_fpext_fmul_f16_to_f32:
 ; GCN: s_waitcnt
-; GFX89: v_mul_f16
-; GFX89: v_cvt_f32_f16
-; GFX89: v_fma_f32
-; GFX89: v_add_f32
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v2, v2, v3, v4 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: v_mac_f32_e32 v2, v0, v1
+; GFX9-F32FLUSH-NEXT: v_mov_b32_e32 v0, v2
+; GFX9-F32FLUSH-NEXT: s_setpc_b64
+
+; GFX9-F32DENORM-NEXT: v_mul_f16_e32 v2, v2, v3
+; GFX9-F32DENORM-NEXT: v_cvt_f32_f16_e32 v2, v2
+; GFX9-F32DENORM-NEXT: v_fma_f32 v0, v0, v1, v2
+; GFX9-F32DENORM-NEXT: v_add_f32_e32 v0, v0, v4
+; GFX9-F32DENORM-NEXT: s_setpc_b64
 define float @fadd_fma_fpext_fmul_f16_to_f32(float %x, float %y, half %u, half %v, float %z) #0 {
 entry:
   %mul = fmul contract half %u, %v
   %mul.ext = fpext half %mul to float
   %fma = call float @llvm.fma.f32(float %x, float %y, float %mul.ext)
   %add = fadd float %fma, %z
   ret float %add
 }
 
 ; GCN-LABEL: {{^}}fadd_fma_fpext_fmul_f16_to_f32_commute:
 ; GCN: s_waitcnt
-; GFX89: v_mul_f16
-; GFX89: v_cvt_f32_f16
-; GFX89: v_fma_f32
-; GFX89: v_add_f32
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v2, v2, v3, v4 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: v_mac_f32_e32 v2, v0, v1
+; GFX9-F32FLUSH-NEXT: v_mov_b32_e32 v0, v2
+; GFX9-F32FLUSH-NEXT: s_setpc_b64
+
+; GFX9-F32DENORM-NEXT: v_mul_f16_e32 v2, v2, v3
+; GFX9-F32DENORM-NEXT: v_cvt_f32_f16_e32 v2, v2
+; GFX9-F32DENORM-NEXT: v_fma_f32 v0, v0, v1, v2
+; GFX9-F32DENORM-NEXT: v_add_f32_e32 v0, v4, v0
+; GFX9-F32DENORM-NEXT: s_setpc_b64
 define float @fadd_fma_fpext_fmul_f16_to_f32_commute(float %x, float %y, half %u, half %v, float %z) #0 {
 entry:
   %mul = fmul contract half %u, %v
   %mul.ext = fpext half %mul to float
   %fma = call float @llvm.fma.f32(float %x, float %y, float %mul.ext)
   %add = fadd float %z, %fma
   ret float %add
 }
 
 ; fold (fadd x, (fpext (fma y, z, (fmul u, v)))
 ;   -> (fma (fpext y), (fpext z), (fma (fpext u), (fpext v), x))
 
 ; GCN-LABEL: {{^}}fadd_fpext_fmuladd_f16_to_f32:
-; GFX9: v_mul_f16
-; GFX9: v_fma_f16
-; GFX9: v_cvt_f32_f16
-; GFX9: v_add_f32_e32
+; GCN: s_waitcnt
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v3, v4, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v1, v2, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: s_setpc_b64
+
+; GFX9-F32DENORM-NEXT: v_mul_f16
+; GFX9-F32DENORM-NEXT: v_fma_f16
+; GFX9-F32DENORM-NEXT: v_cvt_f32_f16
+; GFX9-F32DENORM-NEXT: v_add_f32
+; GFX9-F32DENORM-NEXT: s_setpc_b64
 define float @fadd_fpext_fmuladd_f16_to_f32(float %x, half %y, half %z, half %u, half %v) #0 {
 entry:
   %mul = fmul contract half %u, %v
   %fma = call half @llvm.fmuladd.f16(half %y, half %z, half %mul)
   %ext.fma = fpext half %fma to float
   %add = fadd float %x, %ext.fma
   ret float %add
 }
 
 ; GCN-LABEL: {{^}}fadd_fpext_fma_f16_to_f32:
-; GFX9: v_mul_f16
-; GFX9: v_fma_f16
-; GFX9: v_cvt_f32_f16
-; GFX9: v_add_f32_e32
+; GCN: s_waitcnt
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v3, v4, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v1, v2, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: s_setpc_b64
+
+; GFX9-F32DENORM-NEXT: v_mul_f16
+; GFX9-F32DENORM-NEXT: v_fma_f16
+; GFX9-F32DENORM-NEXT: v_cvt_f32_f16
+; GFX9-F32DENORM-NEXT: v_add_f32
+; GFX9-F32DENORM-NEXT: s_setpc_b64
 define float @fadd_fpext_fma_f16_to_f32(float %x, half %y, half %z, half %u, half %v) #0 {
 entry:
   %mul = fmul contract half %u, %v
   %fma = call half @llvm.fma.f16(half %y, half %z, half %mul)
   %ext.fma = fpext half %fma to float
   %add = fadd float %x, %ext.fma
   ret float %add
 }
 
 ; GCN-LABEL: {{^}}fadd_fpext_fma_f16_to_f32_commute:
-; GFX9: v_mul_f16
-; GFX9: v_fma_f16
-; GFX9: v_cvt_f32_f16
-; GFX9: v_add_f32_e32
+; GCN: s_waitcnt
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v3, v4, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: v_mad_mix_f32 v0, v1, v2, v0 op_sel_hi:[1,1,0]
+; GFX9-F32FLUSH-NEXT: s_setpc_b64
+
+; GFX9-F32DENORM-NEXT: v_mul_f16
+; GFX9-F32DENORM-NEXT: v_fma_f16
+; GFX9-F32DENORM-NEXT: v_cvt_f32_f16
+; GFX9-F32DENORM-NEXT: v_add_f32_e32
+; GFX9-F32DENORM-NEXT: s_setpc_b64
 define float @fadd_fpext_fma_f16_to_f32_commute(float %x, half %y, half %z, half %u, half %v) #0 {
 entry:
   %mul = fmul contract half %u, %v
   %fma = call half @llvm.fma.f16(half %y, half %z, half %mul)
   %ext.fma = fpext half %fma to float
   %add = fadd float %ext.fma, %x
   ret float %add
 }

llvm/test/CodeGen/AMDGPU/mad-combine.ll

 
 ; FUNC-LABEL: {{^}}aggressive_combine_to_mad_fsub_0_f32:
 ; SI-DAG: buffer_load_dword [[A:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64 glc{{$}}
 ; SI-DAG: buffer_load_dword [[B:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:4 glc{{$}}
 ; SI-DAG: buffer_load_dword [[C:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:8 glc{{$}}
 ; SI-DAG: buffer_load_dword [[D:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:12 glc{{$}}
 ; SI-DAG: buffer_load_dword [[E:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:16 glc{{$}}
 
-; SI-STD: v_mul_f32_e32 [[TMP0:v[0-9]+]], [[D]], [[E]]
-; SI-STD: v_fma_f32 [[TMP1:v[0-9]+]], [[A]], [[B]], [[TMP0]]
-; SI-STD: v_sub_f32_e32 [[RESULT:v[0-9]+]], [[TMP1]], [[C]]
+; SI-STD-SAFE: v_mul_f32_e32 [[TMP0:v[0-9]+]], [[D]], [[E]]
+; SI-STD-SAFE: v_fma_f32 [[TMP1:v[0-9]+]], [[A]], [[B]], [[TMP0]]
+; SI-STD-SAFE: v_sub_f32_e32 [[RESULT:v[0-9]+]], [[TMP1]], [[C]]
+
+; SI-STD-UNSAFE: v_mad_f32 [[RESULT:v[0-9]+]], [[D]], [[E]], -[[C]]
+; SI-STD-UNSAFE: v_mac_f32_e32 [[RESULT]], [[A]], [[B]]
 
 ; SI-DENORM: v_mul_f32_e32 [[TMP0:v[0-9]+]], [[D]], [[E]]
 ; SI-DENORM: v_fma_f32 [[TMP1:v[0-9]+]], [[A]], [[B]], [[TMP0]]
 ; SI-DENORM: v_sub_f32_e32 [[RESULT:v[0-9]+]], [[TMP1]], [[C]]
 
 ; SI: buffer_store_dword [[RESULT]], v{{\[[0-9]+:[0-9]+\]}}, s{{\[[0-9]+:[0-9]+\]}}, 0 addr64{{$}}
 define amdgpu_kernel void @aggressive_combine_to_mad_fsub_0_f32(float addrspace(1)* noalias %out, float addrspace(1)* noalias %in) #1 {
   %tid = tail call i32 @llvm.amdgcn.workitem.id.x() #0