[Clang] Allow fmuladd for option -ffast-math

[Allen]

Summary

Discussion on https://discourse.llvm.org/t/fp-contraction-fma-on-by-default/64975

• The behavior controlled by -ffp-contract=on is compliant with the C standard, double fma(double a, double b) { return a * b + 5; } can emit an fusedMultiplyAdd operation.
• The behavior controlled by -ffp-contract=fast enables non-C-standard-compliant behavior that ignores the #pragma and will create contractions after other optimizations like inlining, and across expressions.

Fixes https://github.com/llvm/llvm-project/issues/64244

[aaron.ballman]

Adding some more reviewers.

[jyknight]

This is permitted...but LLVM should be able to convert to FMA during IR optimization when in -ffp-contract=fast mode. So the frontend shouldn't need to do so. It'd be good to figure out why that's not happening as expected in the referenced example code.

[arsenm]

I think isFMAFasterThanFMulAndFAdd is just wrong for some targets. There have been a few patches and posts I've seen all trying to work around this while treating it like a representational issue to work around instead of fixing the backend making the wrong decision

clang/test/CodeGen/ffp-contract-fast-option.cpp

1

Switch to generated checks should be committed separately

5

I believe instcombine will just turn this right back into fmul+fadd with contract set (the flags for this are just stricter than they need to be, see D152166)

[Allen]

So do you think the right direction to fix this issue is something like following ?

• convert it to llvm.fmuladd at the front end of clang if possiable
• then the instcombine turn right back into fmul+fadd with contract set
• then the backend fold them into fmuladd if isFMAFasterThanFMulAndFAdd return true.

[arsenm]

In D156653#4575335, @Allen wrote:

So do you think the right direction to fix this issue is something like following ?

• convert it to llvm.fmuladd at the front end of clang if possiable

No. fmul contract+fadd contract is the right representation. If it produces slower code there's a backend issue to solve

• then the instcombine turn right back into fmul+fadd with contract set

This makes any change to clang pointless here. We already do this, just in overly constrained conditions (only full fast flags)

• then the backend fold them into fmuladd if isFMAFasterThanFMulAndFAdd return true.

Either some other combine went wrong and that broke the fusion, or isFMAFasterThanFMulAndFAdd was wrong for some reason. I don't think there's a representational issue here, just a bad transform somewhere

[arsenm]

I think contract on everything is a better representation for this, if it ends up slower there's a backend issue

clang/lib/CodeGen/CGExprScalar.cpp

                          const CodeGenFunction &CGF, CGBuilderTy &Builder,
                          bool isSub=false) {
 
   assert((op.Opcode == BO_Add || op.Opcode == BO_AddAssign ||
           op.Opcode == BO_Sub || op.Opcode == BO_SubAssign) &&
          "Only fadd/fsub can be the root of an fmuladd.");
 
   // Check whether this op is marked as fusable.
-  if (!op.FPFeatures.allowFPContractWithinStatement())
+  if (!op.FPFeatures.allowFPContractWithinStatement() &&
+      !op.FPFeatures.allowFPContractAcrossStatement())
     return nullptr;
 
   Value *LHS = op.LHS;
   Value *RHS = op.RHS;
 
   // Peek through fneg to look for fmul. Make sure fneg has no users, and that
   // it is the only use of its operand.
   bool NegLHS = false;

clang/test/CodeGen/ffp-contract-fast-option.cpp

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
 // RUN: %clang_cc1 -O3 -ffp-contract=fast -triple %itanium_abi_triple -emit-llvm -o - %s | FileCheck %s
 
+// CHECK-LABEL: @_Z13fp_contract_1fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[C:%.*]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 float fp_contract_1(float a, float b, float c) {
-  // CHECK-LABEL: fp_contract_1fff(
-  // CHECK: fmul contract float
-  // CHECK: fadd contract float
   return a * b + c;
 }
 
+// CHECK-LABEL: @_Z13fp_contract_2fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[NEG:%.*]] = fneg contract float [[C:%.*]]
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[NEG]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 float fp_contract_2(float a, float b, float c) {
-  // CHECK-LABEL: fp_contract_2fff(
-  // CHECK: fmul contract float
-  // CHECK: fsub contract float
   return a * b - c;
 }
 
+// CHECK-LABEL: @_Z13fp_contract_3Pfff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A:%.*]], align 4, !tbaa [[TBAA2:![0-9]+]]
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[B:%.*]], float [[C:%.*]], float [[TMP0]])
+// CHECK-NEXT:    store float [[TMP1]], ptr [[A]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
 void fp_contract_3(float *a, float b, float c) {
-  // CHECK-LABEL: fp_contract_3Pfff(
-  // CHECK: fmul contract float
-  // CHECK: fadd contract float
   a[0] += b * c;
 }
 
+// CHECK-LABEL: @_Z13fp_contract_4Pfff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = load float, ptr [[A:%.*]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    [[NEG:%.*]] = fneg contract float [[B:%.*]]
+// CHECK-NEXT:    [[TMP1:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[NEG]], float [[C:%.*]], float [[TMP0]])
+// CHECK-NEXT:    store float [[TMP1]], ptr [[A]], align 4, !tbaa [[TBAA2]]
+// CHECK-NEXT:    ret void
+//
 void fp_contract_4(float *a, float b, float c) {
-  // CHECK-LABEL: fp_contract_4Pfff(
-  // CHECK: fmul contract float
-  // CHECK: fsub contract float
   a[0] -= b * c;
 }

clang/test/CodeGen/ffp-contract-option.c

float mymuladd(float x, float y, float z) {
 
   // CHECK-ON: load float, ptr
   // CHECK-ON: load float, ptr
   // CHECK-ON: load float, ptr
   // CHECK-ON: call float @llvm.fmuladd.f32(float {{.*}}, float {{.*}}, float {{.*}})
 
   // CHECK-CONTRACTFAST: load float, ptr
   // CHECK-CONTRACTFAST: load float, ptr
-  // CHECK-CONTRACTFAST: fmul contract float
   // CHECK-CONTRACTFAST: load float, ptr
-  // CHECK-CONTRACTFAST: fadd contract float
+  // CHECK-CONTRACTFAST: call contract float @llvm.fmuladd.f32
 
   // CHECK-CONTRACTOFF: load float, ptr
   // CHECK-CONTRACTOFF: load float, ptr
   // CHECK-CONTRACTOFF: fmul reassoc nnan ninf nsz arcp afn float
   // CHECK-CONTRACTOFF: load float, ptr
   // CHECK-CONTRACTOFF: fadd reassoc nnan ninf nsz arcp afn float {{.*}}, {{.*}}
 
   // CHECK-ONFAST: load float, ptr
   // CHECK-ONFAST: load float, ptr
   // CHECK-ONFAST: load float, ptr
   // CHECK-ONFAST: call reassoc nnan ninf nsz arcp afn float @llvm.fmuladd.f32(float {{.*}}, float {{.*}}, float {{.*}})
 
   // CHECK-FASTFAST: load float, ptr
   // CHECK-FASTFAST: load float, ptr
-  // CHECK-FASTFAST: fmul fast float
   // CHECK-FASTFAST: load float, ptr
-  // CHECK-FASTFAST: fadd fast float {{.*}}, {{.*}}
+  // CHECK-FASTFAST: call fast float @llvm.fmuladd.f32
 
   // CHECK-NOFAST: load float, ptr
   // CHECK-NOFAST: load float, ptr
   // CHECK-NOFAST: fmul float {{.*}}, {{.*}}
   // CHECK-NOFAST: load float, ptr
   // CHECK-NOFAST: fadd float {{.*}}, {{.*}}
 
   // CHECK-FPC-ON: load float, ptr

clang/test/CodeGen/ffp-model.c

 
 // RUN: %clang -S -emit-llvm -ffp-model=precise -ffast-math \
 // RUN: %s -o - | FileCheck %s --check-prefixes CHECK,CHECK-FAST1
 
 float mymuladd(float x, float y, float z) {
   // CHECK: define{{.*}} float @mymuladd
   return x * y + z;
 
-  // CHECK-FAST: fmul fast float
   // CHECK-FAST: load float, ptr
-  // CHECK-FAST: fadd fast float
+  // CHECK-FAST: call fast float @llvm.fmuladd.f32
 
   // CHECK-PRECISE: load float, ptr
   // CHECK-PRECISE: load float, ptr
   // CHECK-PRECISE: load float, ptr
   // CHECK-PRECISE: call float @llvm.fmuladd.f32(float {{.*}}, float {{.*}}, float {{.*}})
 
   // CHECK-STRICT: load float, ptr
   // CHECK-STRICT: load float, ptr
   // CHECK-STRICT: call float @llvm.experimental.constrained.fmul.f32(float {{.*}}, float {{.*}}, {{.*}})
   // CHECK-STRICT: load float, ptr
   // CHECK-STRICT: call float @llvm.experimental.constrained.fadd.f32(float {{.*}}, float {{.*}}, {{.*}})
 
   // CHECK-STRICT-FAST: load float, ptr
   // CHECK-STRICT-FAST: load float, ptr
-  // CHECK-STRICT-FAST: fmul fast float {{.*}}, {{.*}}
   // CHECK-STRICT-FAST: load float, ptr
-  // CHECK-STRICT-FAST: fadd fast float {{.*}}, {{.*}}
+  // CHECK-STRICT-FAST: call fast float @llvm.fmuladd.f32
 
   // CHECK-FAST1: load float, ptr
   // CHECK-FAST1: load float, ptr
-  // CHECK-FAST1: fmul fast float {{.*}}, {{.*}}
   // CHECK-FAST1: load float, ptr {{.*}}
-  // CHECK-FAST1: fadd fast float {{.*}}, {{.*}}
+  // CHECK-FAST1: call fast float @llvm.fmuladd.f32
 }

clang/test/CodeGen/fp-contract-fast-pragma.cpp

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
 // RUN: %clang_cc1 -O3 -triple %itanium_abi_triple -emit-llvm -o - %s | FileCheck %s
 
 // Is FP_CONTRACT honored in a simple case?
+// CHECK-LABEL: @_Z13fp_contract_1fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[C:%.*]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 float fp_contract_1(float a, float b, float c) {
-// CHECK: _Z13fp_contract_1fff
-// CHECK: %[[M:.+]] = fmul contract float %a, %b
-// CHECK-NEXT: fadd contract float %[[M]], %c
 #pragma clang fp contract(fast)
   return a * b + c;
 }
 
 // Is FP_CONTRACT state cleared on exiting compound statements?
+// CHECK-LABEL: @_Z13fp_contract_2fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[MUL:%.*]] = fmul float [[A:%.*]], [[B:%.*]]
+// CHECK-NEXT:    [[ADD:%.*]] = fadd float [[MUL]], [[C:%.*]]
+// CHECK-NEXT:    ret float [[ADD]]
+//
 float fp_contract_2(float a, float b, float c) {
-  // CHECK: _Z13fp_contract_2fff
-  // CHECK: %[[M:.+]] = fmul float %a, %b
-  // CHECK-NEXT: fadd float %[[M]], %c
   {
 #pragma clang fp contract(fast)
   }
   return a * b + c;
 }
 
 // Does FP_CONTRACT survive template instantiation?
 class Foo {};
 Foo operator+(Foo, Foo);
 
 template <typename T>
 T template_muladd(T a, T b, T c) {
 #pragma clang fp contract(fast)
   return a * b + c;
 }
 
+// CHECK-LABEL: @_Z13fp_contract_3fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[C:%.*]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 float fp_contract_3(float a, float b, float c) {
-  // CHECK: _Z13fp_contract_3fff
-  // CHECK: %[[M:.+]] = fmul contract float %a, %b
-  // CHECK-NEXT: fadd contract float %[[M]], %c
   return template_muladd<float>(a, b, c);
 }
 
 template <typename T>
 class fp_contract_4 {
   float method(float a, float b, float c) {
 #pragma clang fp contract(fast)
     return a * b + c;
   }
 };
 
+// CHECK-LABEL: @_ZN13fp_contract_4IiE6methodEfff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[C:%.*]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 template class fp_contract_4<int>;
-// CHECK: _ZN13fp_contract_4IiE6methodEfff
-// CHECK: %[[M:.+]] = fmul contract float %a, %b
-// CHECK-NEXT: fadd contract float %[[M]], %c
 
 // Check file-scoped FP_CONTRACT
 #pragma clang fp contract(fast)
+// CHECK-LABEL: @_Z13fp_contract_5fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = tail call contract float @llvm.fmuladd.f32(float [[A:%.*]], float [[B:%.*]], float [[C:%.*]])
+// CHECK-NEXT:    ret float [[TMP0]]
+//
 float fp_contract_5(float a, float b, float c) {
-  // CHECK: _Z13fp_contract_5fff
-  // CHECK: %[[M:.+]] = fmul contract float %a, %b
-  // CHECK-NEXT: fadd contract float %[[M]], %c
   return a * b + c;
 }
 
 // Verify that we can handle multiple flags on the same pragma
 #pragma clang fp contract(fast) contract(off)
+// CHECK-LABEL: @_Z13fp_contract_6fff(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[MUL:%.*]] = fmul float [[A:%.*]], [[B:%.*]]
+// CHECK-NEXT:    [[ADD:%.*]] = fadd float [[MUL]], [[C:%.*]]
+// CHECK-NEXT:    ret float [[ADD]]
+//
 float fp_contract_6(float a, float b, float c) {
-  // CHECK: _Z13fp_contract_6fff
-  // CHECK: %[[M:.+]] = fmul float %a, %b
-  // CHECK-NEXT: fadd float %[[M]], %c
   return a * b + c;
 }

clang/test/CodeGen/fp-floatcontrol-stack.cpp

 #if EBSTRICT
 // Note that backend wants constrained intrinsics used
 // throughout the function if they are needed anywhere in the function.
 // In that case, operations are built with constrained intrinsics operator
 // but using default settings for exception behavior and rounding mode.
 //CHECK-DEBSTRICT: llvm.experimental.constrained.fmul{{.*}}tonearest{{.*}}strict
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(push)
 #ifndef FAST
 // Rule: precise must be enabled
 #pragma float_control(except, on)
 #endif
     // CHECK-FAST: Function Attrs: mustprogress noinline nounwind optnone{{$$}}
     // CHECK-DDEFAULT: Function Attrs: mustprogress noinline nounwind optnone strictfp{{$$}}
     // CHECK-DEBSTRICT: Function Attrs: mustprogress noinline nounwind optnone strictfp{{$$}}
     // CHECK-NOHONOR: Function Attrs: mustprogress noinline nounwind optnone strictfp{{$$}}
     float exc_on FUN(2)
 //CHECK-LABEL: define {{.*}} @_Z6exc_onf{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: llvm.experimental.constrained.fmul{{.*}}
 #endif
 #if EBSTRICT
 //CHECK-DEBSTRICT: llvm.experimental.constrained.fmuladd{{.*}}tonearest{{.*}}strict
 #endif
 #if NOHONOR
 //CHECK-NOHONOR: nnan ninf float {{.*}}llvm.experimental.constrained.fmuladd{{.*}}tonearest{{.*}}strict
 #endif
 #if FAST
 //Not possible to enable float_control(except) in FAST mode.
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(pop)
     // CHECK-DDEFAULT: Function Attrs: mustprogress noinline nounwind optnone{{$$}}
     // CHECK-DEBSTRICT: Function Attrs: mustprogress noinline nounwind optnone strictfp{{$$}}
     // CHECK-FAST: Function Attrs: mustprogress noinline nounwind optnone{{$$}}
     // CHECK-NOHONOR: Function Attrs: mustprogress noinline nounwind optnone{{$$}}
     float exc_pop FUN(5)
 //CHECK-LABEL: define {{.*}} @_Z7exc_popf{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: call float @llvm.fmuladd{{.*}}
 #endif
 #if EBSTRICT
 //CHECK-DEBSTRICT: llvm.experimental.constrained.fmuladd{{.*}}tonearest{{.*}}strict
 #endif
 #if NOHONOR
 //CHECK-NOHONOR: call nnan ninf float @llvm.fmuladd{{.*}}
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(except, off)
         float exc_off FUN(5)
 //CHECK-LABEL: define {{.*}} @_Z7exc_offf{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: call float @llvm.fmuladd{{.*}}
 #endif
 #if EBSTRICT
 //CHECK-DEBSTRICT: call float @llvm.fmuladd{{.*}}
 #endif
 #if NOHONOR
 //CHECK-NOHONOR: call nnan ninf float @llvm.fmuladd{{.*}}
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(precise, on, push)
             float precise_on FUN(3)
 //CHECK-LABEL: define {{.*}} @_Z10precise_onf{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: float {{.*}}llvm.fmuladd{{.*}}
 #endif

 #endif
 #if EBSTRICT
 //CHECK-DEBSTRICT: float {{.*}}llvm.fmuladd{{.*}}
 #endif
 #if NOHONOR
 //CHECK-NOHONOR: call nnan ninf float @llvm.fmuladd{{.*}}
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 #pragma float_control(precise, off)
                     float precise_off FUN(4)
 //CHECK-LABEL: define {{.*}} @_Z11precise_offf{{.*}}
 #if DEFAULT
 // Note: precise_off enables fp_contract=fast and the instructions
 // generated do not include the contract flag, although it was enabled
 // in IRBuilder.
-//CHECK-DDEFAULT: fmul fast float
-//CHECK-DDEFAULT: fadd fast float
+//CHECK-DDEFAULT: call fast float @llvm.fmuladd.f32
 #endif
 #if EBSTRICT
-//CHECK-DEBSTRICT: fmul fast float
-//CHECK-DEBSTRICT: fadd fast float
+//CHECK-DEBSTRICT: call fast float @llvm.fmuladd.f32
 #endif
 #if NOHONOR
 // fast math should be enabled, and contract should be fast
-//CHECK-NOHONOR: fmul fast float
-//CHECK-NOHONOR: fadd fast float
+//CHECK-NOHONOR: call fast float @llvm.fmuladd.f32
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(precise, on)
                         float precise_on2 FUN(3)
 //CHECK-LABEL: define {{.*}} @_Z11precise_on2f{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: llvm.fmuladd{{.*}}
 #endif

 #if FAST
 //CHECK-FAST: float {{.*}}llvm.fmuladd{{.*}}
 #endif
 
 #pragma float_control(precise, off)
                                 float precise_off2 FUN(4)
 //CHECK-LABEL: define {{.*}} @_Z12precise_off2f{{.*}}
 #if DEFAULT
-//CHECK-DDEFAULT: fmul fast float
-//CHECK-DDEFAULT: fadd fast float
+//CHECK-DDEFAULT: call fast float @llvm.fmuladd.f32
 #endif
 #if EBSTRICT
-//CHECK-DEBSTRICT: fmul fast float
-//CHECK-DEBSTRICT: fadd fast float
+//CHECK-DEBSTRICT: call fast float @llvm.fmuladd.f32
 #endif
 #if NOHONOR
 // fast math settings since precise is off
-//CHECK-NOHONOR: fmul fast float
-//CHECK-NOHONOR: fadd fast float
+//CHECK-NOHONOR: call fast float @llvm.fmuladd.f32
 #endif
 #if FAST
-//CHECK-FAST: fmul fast float
-//CHECK-FAST: fadd fast float
+//CHECK-FAST: call fast float @llvm.fmuladd.f32
 #endif
 
 #pragma float_control(pop)
                                     float precise_pop2 FUN(3)
 //CHECK-LABEL: define {{.*}} @_Z12precise_pop2f{{.*}}
 #if DEFAULT
 //CHECK-DDEFAULT: llvm.fmuladd{{.*}}
 #endif