[AArch64] Change order of candidate FMLS patterns

[mssimpso]

Summary

A few floating-point SPEC benchmarks are regressed on Falkor after rL319980. That patch added new patterns to the machine combiner for transforming (fsub (fmul x y) z) into (fmla (fneg z) x y). That is, fsub's where the first source operand is an fmul are transformed. We previously only matched the case where the second source operand of an fsub was an fmul, transforming (fsub z (fmul x y)) into (fmls z x y). Now, if we have an fsub where both source operands are fmuls, both of the above patterns are applicable.

However, the order in which we add the patterns to the list of candidates determines the transformation that takes place, since only the first pattern that matches will be used. This patch changes the order these two patterns are added to the list of candidates such that we prefer the case where the second source operand is an fmul (the fmls case), rather than the other one (the fmla/fneg case). When both source operands are fmuls, this ordering results in fewer instructions. Please refer to the added test cases for examples.

[fhahn]

Thanks for spotting this!
The reordering looks good to me, give the current behavior of the MachineCombiner it makes sense to move up the more profitable patterns.

However, the order in which we add the patterns to the list of candidates determines the transformation that takes place, since only the first pattern that matches will be used.

This behavior is not ideal though, especially as some order of patterns might be good for one micro architecture, whereas a different order is better for a different micro architecture.

I'll try to look into this. Maybe it's worth to change the MachineCombiner to try all available patterns.

[mssimpso]

In D41587#964226, @fhahn wrote:

Thanks for spotting this!
The reordering looks good to me, give the current behavior of the MachineCombiner it makes sense to move up the more profitable patterns.

However, the order in which we add the patterns to the list of candidates determines the transformation that takes place, since only the first pattern that matches will be used.

This behavior is not ideal though, especially as some order of patterns might be good for one micro architecture, whereas a different order is better for a different micro architecture.

I'll try to look into this. Maybe it's worth to change the MachineCombiner to try all available patterns.

Thanks Florian! Yes, I agree with your comment. There's actually a note about the limitations of the current behavior in MachineCombiner::combineInstructions.

lib/Target/AArch64/AArch64InstrInfo.cpp

if (canCombineWithFMUL(MBB, Root.getOperand(2), AArch64::FMULDrr)) {
       Found = true;
     }
     if (canCombineWithFMUL(MBB, Root.getOperand(1), AArch64::FNMULDrr)) {
       Patterns.push_back(MachineCombinerPattern::FNMULSUBD_OP1);
       Found = true;
     }
     break;
   case AArch64::FSUBv2f32:
-    if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                           AArch64::FMULv2i32_indexed)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv2i32_indexed_OP1);
-      Found = true;
-    } else if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                                  AArch64::FMULv2f32)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv2f32_OP1);
-      Found = true;
-    }
     if (canCombineWithFMUL(MBB, Root.getOperand(2),
                            AArch64::FMULv2i32_indexed)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv2i32_indexed_OP2);
       Found = true;
     } else if (canCombineWithFMUL(MBB, Root.getOperand(2),
                                   AArch64::FMULv2f32)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv2f32_OP2);
       Found = true;
     }
-    break;
-  case AArch64::FSUBv2f64:
     if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                           AArch64::FMULv2i64_indexed)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv2i64_indexed_OP1);
+                           AArch64::FMULv2i32_indexed)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv2i32_indexed_OP1);
       Found = true;
     } else if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                                  AArch64::FMULv2f64)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv2f64_OP1);
+                                  AArch64::FMULv2f32)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv2f32_OP1);
       Found = true;
     }
+    break;
+  case AArch64::FSUBv2f64:
     if (canCombineWithFMUL(MBB, Root.getOperand(2),
                            AArch64::FMULv2i64_indexed)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv2i64_indexed_OP2);
       Found = true;
     } else if (canCombineWithFMUL(MBB, Root.getOperand(2),
                                   AArch64::FMULv2f64)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv2f64_OP2);
       Found = true;
     }
-    break;
-  case AArch64::FSUBv4f32:
     if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                           AArch64::FMULv4i32_indexed)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv4i32_indexed_OP1);
+                           AArch64::FMULv2i64_indexed)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv2i64_indexed_OP1);
       Found = true;
     } else if (canCombineWithFMUL(MBB, Root.getOperand(1),
-                                  AArch64::FMULv4f32)) {
-      Patterns.push_back(MachineCombinerPattern::FMLSv4f32_OP1);
+                                  AArch64::FMULv2f64)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv2f64_OP1);
       Found = true;
     }
+    break;
+  case AArch64::FSUBv4f32:
     if (canCombineWithFMUL(MBB, Root.getOperand(2),
                            AArch64::FMULv4i32_indexed)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv4i32_indexed_OP2);
       Found = true;
     } else if (canCombineWithFMUL(MBB, Root.getOperand(2),
                                   AArch64::FMULv4f32)) {
       Patterns.push_back(MachineCombinerPattern::FMLSv4f32_OP2);
       Found = true;
     }
+    if (canCombineWithFMUL(MBB, Root.getOperand(1),
+                           AArch64::FMULv4i32_indexed)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv4i32_indexed_OP1);
+      Found = true;
+    } else if (canCombineWithFMUL(MBB, Root.getOperand(1),
+                                  AArch64::FMULv4f32)) {
+      Patterns.push_back(MachineCombinerPattern::FMLSv4f32_OP1);
+      Found = true;
+    }
     break;
   }
   return Found;
 }
 
 /// Return true when a code sequence can improve throughput. It
 /// should be called only for instructions in loops.
 /// \param Pattern - combiner pattern
MachineBasicBlock::iterator AArch64InstrInfo::insertOutlinedCall(
   MachineInstr *LDRXpost = BuildMI(MF, DebugLoc(), get(AArch64::LDRXpost))
                                .addReg(AArch64::SP, RegState::Define)
                                .addReg(AArch64::LR, RegState::Define)
                                .addReg(AArch64::SP)
                                .addImm(16);
   It = MBB.insert(It, LDRXpost);
 
   return It;
-}
+}
 

test/CodeGen/AArch64/aarch64-combine-fmul-fsub.mir

-# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=cortex-a57 -enable-unsafe-fp-math %s | FileCheck --check-prefix=UNPROFITABLE %s
-# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=falkor -enable-unsafe-fp-math %s | FileCheck --check-prefix=PROFITABLE %s
-# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=exynosm1 -enable-unsafe-fp-math %s | FileCheck --check-prefix=PROFITABLE %s
-# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=thunderx2t99 -enable-unsafe-fp-math %s | FileCheck --check-prefix=PROFITABLE %s
+# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=cortex-a57 -enable-unsafe-fp-math %s | FileCheck --check-prefixes=UNPROFITABLE,ALL %s
+# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=falkor -enable-unsafe-fp-math %s | FileCheck --check-prefixes=PROFITABLE,ALL %s
+# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=exynosm1 -enable-unsafe-fp-math %s | FileCheck --check-prefixes=PROFITABLE,ALL %s
+# RUN: llc -run-pass=machine-combiner -o - -mtriple=aarch64-unknown-linux -mcpu=thunderx2t99 -enable-unsafe-fp-math %s | FileCheck --check-prefixes=PROFITABLE,ALL %s
 #
 name:            f1_2s
 registers:
   - { id: 0, class: fpr64 }
   - { id: 1, class: fpr64 }
   - { id: 2, class: fpr64 }
   - { id: 3, class: fpr64 }
   - { id: 4, class: fpr64 }

 ...
 # UNPROFITABLE-LABEL: name: f1_2d
 # UNPROFITABLE: %3:fpr128 = FMULv2f64 %0, %1
 # UNPROFITABLE-NEXT: FSUBv2f64 killed %3, %2
 #
 # PROFITABLE-LABEL: name: f1_2d
 # PROFITABLE: %5:fpr128 = FNEGv2f64 %2
 # PROFITABLE-NEXT: FMLAv2f64 killed %5, %0, %1
+---
+name:            f1_both_fmul_2s
+registers:
+  - { id: 0, class: fpr64 }
+  - { id: 1, class: fpr64 }
+  - { id: 2, class: fpr64 }
+  - { id: 3, class: fpr64 }
+  - { id: 4, class: fpr64 }
+  - { id: 5, class: fpr64 }
+  - { id: 6, class: fpr64 }
+body:             |
+  bb.0.entry:
+    %3:fpr64 = COPY %q3
+    %2:fpr64 = COPY %q2
+    %1:fpr64 = COPY %q1
+    %0:fpr64 = COPY %q0
+    %4:fpr64 = FMULv2f32 %0, %1
+    %5:fpr64 = FMULv2f32 %2, %3
+    %6:fpr64 = FSUBv2f32 killed %4, %5
+    %q0 = COPY %6
+    RET_ReallyLR implicit %q0
+
+...
+# ALL-LABEL: name: f1_both_fmul_2s
+# ALL: %4:fpr64 = FMULv2f32 %0, %1
+# ALL-NEXT: FMLSv2f32 killed %4, %2, %3
+---
+name:            f1_both_fmul_4s
+registers:
+  - { id: 0, class: fpr128 }
+  - { id: 1, class: fpr128 }
+  - { id: 2, class: fpr128 }
+  - { id: 3, class: fpr128 }
+  - { id: 4, class: fpr128 }
+  - { id: 5, class: fpr128 }
+  - { id: 6, class: fpr128 }
+body:             |
+  bb.0.entry:
+    %3:fpr128 = COPY %q3
+    %2:fpr128 = COPY %q2
+    %1:fpr128 = COPY %q1
+    %0:fpr128 = COPY %q0
+    %4:fpr128 = FMULv4f32 %0, %1
+    %5:fpr128 = FMULv4f32 %2, %3
+    %6:fpr128 = FSUBv4f32 killed %4, %5
+    %q0 = COPY %6
+    RET_ReallyLR implicit %q0
+
+...
+# ALL-LABEL: name: f1_both_fmul_4s
+# ALL: %4:fpr128 = FMULv4f32 %0, %1
+# ALL-NEXT: FMLSv4f32 killed %4, %2, %3
+---
+name:            f1_both_fmul_2d
+registers:
+  - { id: 0, class: fpr128 }
+  - { id: 1, class: fpr128 }
+  - { id: 2, class: fpr128 }
+  - { id: 3, class: fpr128 }
+  - { id: 4, class: fpr128 }
+  - { id: 5, class: fpr128 }
+  - { id: 6, class: fpr128 }
+body:             |
+  bb.0.entry:
+    %3:fpr128 = COPY %q3
+    %2:fpr128 = COPY %q2
+    %1:fpr128 = COPY %q1
+    %0:fpr128 = COPY %q0
+    %4:fpr128 = FMULv2f64 %0, %1
+    %5:fpr128 = FMULv2f64 %2, %3
+    %6:fpr128 = FSUBv2f64 killed %4, %5
+    %q0 = COPY %6
+    RET_ReallyLR implicit %q0
+
+...
+# ALL-LABEL: name: f1_both_fmul_2d
+# ALL: %4:fpr128 = FMULv2f64 %0, %1
+# ALL-NEXT: FMLSv2f64 killed %4, %2, %3
+