[IR] Add AllowContract to FastMathFlags

[anemet]

Summary

-ffp-contract=fast does not currently work with LTO because it's passed as a
TargetOption to the backend rather than in the IR. This adds it to
FastMathFlags.

This is toward fixing PR25721

[arsenm]

Needs assembler and bitcode compatibility tests

docs/LangRef.rst

2197–2200

Should we also have an unsafe algebra that preserves signed zeros/nans etc?

[mehdi_amini]

docs/LangRef.rst

2197–2200

@arsenm : isn't it the default when there is no nsz or nnan ?

2199

Is it the intent to allow this as well : x + x + x -> 3 * x ?

[arsenm]

docs/LangRef.rst

2197–2200

No, that is normal safe math. I mean unsafe algebraic transformations similar to contraction which will preserve nan/signed zeros/inf behavior but are still not safe.

[anemet]

docs/LangRef.rst

2199

I don't think so. We do the general case (any number of terms) under unsafe-math, in FAddCombine::simplify in InstCombine.

[anemet]

docs/LangRef.rst

2197–2200

Maybe but we may want that separate from contraction where the result is more exact.

[mehdi_amini]

docs/LangRef.rst

2197–2200

Do you have an example of such transformations?

[mehdi_amini]

docs/LangRef.rst

2199

I know we do this under unsafe-fp-math, I was wondering specifically about how fat the "contract" mode allows to go.

[anemet]

docs/LangRef.rst

2199

Ah ok, sorry, I misunderstood. The C standard frames FP_CONTRACT as a permission to omit rounding errors while contracting, so potentially this may be included too.

[spatel]

docs/LangRef.rst

2197–2200

If I'm understanding Matt's question, a mode that allows reassociation but is independent of the other relaxation bits could be handled by not having "fast" imply all of the other bits. This came up here:
http://lists.llvm.org/pipermail/llvm-dev/2016-November/107104.html

I think we need that mode, but it's separate from this set of patches.

[scanon]

docs/LangRef.rst

2199

Fun fact, x + x + x -> 3*x is always legal (both expressions only have a single rounding). x + x + x + x -> 4*x requires the assumption of default rounding, however.

[anemet]

docs/LangRef.rst

2197–2200

@spatel, thanks for the context. I agree that is a different topic from this set of patches. Here the goal is to represent -ffp-contract=fast in the IR without changing semantics or its relation to -ffast-math ('fast' implies contraction).

[anemet]

Address Matt's comments

[anemet]

Ping

[spatel]

include/llvm/IR/Operator.h

177–178

This comment isn't right - FMF is using "SubclassOptionalData" not "SubclassData".

Not sure if it's worth mentioning as a code comment, but we only have room for one more FMF bit if we keep using SubclassOptionalData; it only has 7 bits total.

209–211

If we want to add a bool param, should we do that for all of the flags as a follow-up change? Seems odd to make setAllowContract() different than everything else if there's no change to use a 'false' setter in this patch.

319–320

I know it's copying the format of the other comments, but when do we use these single-letter abbreviations of the FMF? If never, just remove these comments?

[anemet]

include/llvm/IR/Operator.h

177–178

Yes, that is why I added the comment. You're right I mixed up the name of the field.

209–211

Setting it with false is used in the clang patch in https://reviews.llvm.org/D31168.

But yes, I am planning to change all of these as a follow-on. I was going to put it in this patchset but the change is actually larger than I expected.

[anemet]

Address Sanjay's comments

[spatel]

Thanks! LGTM.

docs/LangRef.rst

 ``nsz``
    No Signed Zeros - Allow optimizations to treat the sign of a zero
    argument or result as insignificant.
 
 ``arcp``
    Allow Reciprocal - Allow optimizations to use the reciprocal of an
    argument rather than perform division.
 
+``contract``
+   Allow floating-point contraction (e.g. fusing a multiply followed by an
+   addition into a fused multiply-and-add).
+
 ``fast``
    Fast - Allow algebraically equivalent transformations that may
    dramatically change results in floating point (e.g. reassociate). This
    flag implies all the others.
 
 .. _uselistorder:
 
 Use-list Order Directives

include/llvm/IR/Instruction.h

public:
   bool hasNoInfs() const;
 
   /// Determine whether the no-signed-zeros flag is set.
   bool hasNoSignedZeros() const;
 
   /// Determine whether the allow-reciprocal flag is set.
   bool hasAllowReciprocal() const;
 
+  /// Determine whether the allow-contract flag is set.
+  bool hasAllowContract() const;
+
   /// Convenience function for getting all the fast-math flags, which must be an
   /// operator which supports these flags. See LangRef.html for the meaning of
   /// these flags.
   FastMathFlags getFastMathFlags() const;
 
   /// Copy I's fast-math flags
   void copyFastMathFlags(const Instruction *I);
 

include/llvm/IR/Operator.h

 private:
   friend class FPMathOperator;
 
   unsigned Flags = 0;
 
   FastMathFlags(unsigned F) : Flags(F) { }
 
 public:
+  /// This is how the bits are used in Value::SubclassData so they should fit
+  /// there too.
   enum {
     UnsafeAlgebra   = (1 << 0),
     NoNaNs          = (1 << 1),
     NoInfs          = (1 << 2),
     NoSignedZeros   = (1 << 3),
-    AllowReciprocal = (1 << 4)
+    AllowReciprocal = (1 << 4),
+    AllowContract   = (1 << 5)
   };
 
   FastMathFlags() = default;
 
   /// Whether any flag is set
   bool any() const { return Flags != 0; }
 
   /// Set all the flags to false
   void clear() { Flags = 0; }
 
   /// Flag queries
   bool noNaNs() const          { return 0 != (Flags & NoNaNs); }
   bool noInfs() const          { return 0 != (Flags & NoInfs); }
   bool noSignedZeros() const   { return 0 != (Flags & NoSignedZeros); }
   bool allowReciprocal() const { return 0 != (Flags & AllowReciprocal); }
+  bool allowContract() const { return 0 != (Flags & AllowContract); }
   bool unsafeAlgebra() const   { return 0 != (Flags & UnsafeAlgebra); }
 
   /// Flag setters
   void setNoNaNs()          { Flags |= NoNaNs; }
   void setNoInfs()          { Flags |= NoInfs; }
   void setNoSignedZeros()   { Flags |= NoSignedZeros; }
   void setAllowReciprocal() { Flags |= AllowReciprocal; }
+  void setAllowContract(bool B) {
+    Flags = (Flags & ~AllowContract) | B * AllowContract;
+  }
   void setUnsafeAlgebra() {
     Flags |= UnsafeAlgebra;
     setNoNaNs();
     setNoInfs();
     setNoSignedZeros();
     setAllowReciprocal();
+    setAllowContract(true);
   }
 
   void operator&=(const FastMathFlags &OtherFlags) {
     Flags &= OtherFlags.Flags;
   }
 };
 
 /// Utility class for floating point operations which can have
private:
   }
 
   void setHasAllowReciprocal(bool B) {
     SubclassOptionalData =
       (SubclassOptionalData & ~FastMathFlags::AllowReciprocal) |
       (B * FastMathFlags::AllowReciprocal);
   }
 
+  void setHasAllowContract(bool B) {
+    SubclassOptionalData =
+        (SubclassOptionalData & ~FastMathFlags::AllowContract) |
+        (B * FastMathFlags::AllowContract);
+  }
+
   /// Convenience function for setting multiple fast-math flags.
   /// FMF is a mask of the bits to set.
   void setFastMathFlags(FastMathFlags FMF) {
     SubclassOptionalData |= FMF.Flags;
   }
 
   /// Convenience function for copying all fast-math flags.
   /// All values in FMF are transferred to this operator.
public:
   }
 
   /// Test whether this operation is permitted to use
   /// reciprocal instead of division, aka the 'R' fast-math property.
   bool hasAllowReciprocal() const {
     return (SubclassOptionalData & FastMathFlags::AllowReciprocal) != 0;
   }
 
+  /// Test whether this operation is permitted to
+  /// be floating-point contracted, aka the 'C' fast-math property.
+  bool hasAllowContract() const {
+    return (SubclassOptionalData & FastMathFlags::AllowContract) != 0;
+  }
+
   /// Convenience function for getting all the fast-math flags
   FastMathFlags getFastMathFlags() const {
     return FastMathFlags(SubclassOptionalData);
   }
 
   /// Get the maximum error permitted by this operation in ULPs. An accuracy of
   /// 0.0 means that the operation should be performed with the default
   /// precision.

lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(acq_rel);
   KEYWORD(seq_cst);
   KEYWORD(singlethread);
 
   KEYWORD(nnan);
   KEYWORD(ninf);
   KEYWORD(nsz);
   KEYWORD(arcp);
+  KEYWORD(contract);
   KEYWORD(fast);
   KEYWORD(nuw);
   KEYWORD(nsw);
   KEYWORD(exact);
   KEYWORD(inbounds);
   KEYWORD(inrange);
   KEYWORD(align);
   KEYWORD(addrspace);

lib/AsmParser/LLParser.h

FastMathFlags EatFastMathFlagsIfPresent() {
       FastMathFlags FMF;
       while (true)
         switch (Lex.getKind()) {
         case lltok::kw_fast: FMF.setUnsafeAlgebra();   Lex.Lex(); continue;
         case lltok::kw_nnan: FMF.setNoNaNs();          Lex.Lex(); continue;
         case lltok::kw_ninf: FMF.setNoInfs();          Lex.Lex(); continue;
         case lltok::kw_nsz:  FMF.setNoSignedZeros();   Lex.Lex(); continue;
         case lltok::kw_arcp: FMF.setAllowReciprocal(); Lex.Lex(); continue;
+        case lltok::kw_contract:
+          FMF.setAllowContract(true);
+          Lex.Lex();
+          continue;
         default: return FMF;
         }
       return FMF;
     }
 
     bool ParseOptionalToken(lltok::Kind T, bool &Present,
                             LocTy *Loc = nullptr) {
       if (Lex.getKind() != T) {

lib/AsmParser/LLToken.h

enum Kind {
   kw_release,
   kw_acq_rel,
   kw_seq_cst,
   kw_singlethread,
   kw_nnan,
   kw_ninf,
   kw_nsz,
   kw_arcp,
+  kw_contract,
   kw_fast,
   kw_nuw,
   kw_nsw,
   kw_exact,
   kw_inbounds,
   kw_inrange,
   kw_align,
   kw_addrspace,

lib/Bitcode/Reader/BitcodeReader.cpp

static FastMathFlags getDecodedFastMathFlags(unsigned Val) {
   if (0 != (Val & FastMathFlags::NoNaNs))
     FMF.setNoNaNs();
   if (0 != (Val & FastMathFlags::NoInfs))
     FMF.setNoInfs();
   if (0 != (Val & FastMathFlags::NoSignedZeros))
     FMF.setNoSignedZeros();
   if (0 != (Val & FastMathFlags::AllowReciprocal))
     FMF.setAllowReciprocal();
+  if (0 != (Val & FastMathFlags::AllowContract))
+    FMF.setAllowContract(true);
   return FMF;
 }
 
 static void upgradeDLLImportExportLinkage(GlobalValue *GV, unsigned Val) {
   switch (Val) {
   case 5: GV->setDLLStorageClass(GlobalValue::DLLImportStorageClass); break;
   case 6: GV->setDLLStorageClass(GlobalValue::DLLExportStorageClass); break;
   }

lib/Bitcode/Writer/BitcodeWriter.cpp

if (const auto *OBO = dyn_cast<OverflowingBinaryOperator>(V)) {
     if (FPMO->hasNoNaNs())
       Flags |= FastMathFlags::NoNaNs;
     if (FPMO->hasNoInfs())
       Flags |= FastMathFlags::NoInfs;
     if (FPMO->hasNoSignedZeros())
       Flags |= FastMathFlags::NoSignedZeros;
     if (FPMO->hasAllowReciprocal())
       Flags |= FastMathFlags::AllowReciprocal;
+    if (FPMO->hasAllowContract())
+      Flags |= FastMathFlags::AllowContract;
   }
 
   return Flags;
 }
 
 void ModuleBitcodeWriter::writeValueAsMetadata(
     const ValueAsMetadata *MD, SmallVectorImpl<uint64_t> &Record) {
   // Mimic an MDNode with a value as one operand.

lib/IR/AsmWriter.cpp

else {
       if (FPO->hasNoNaNs())
         Out << " nnan";
       if (FPO->hasNoInfs())
         Out << " ninf";
       if (FPO->hasNoSignedZeros())
         Out << " nsz";
       if (FPO->hasAllowReciprocal())
         Out << " arcp";
+      if (FPO->hasAllowContract())
+        Out << " contract";
     }
   }
 
   if (const OverflowingBinaryOperator *OBO =
         dyn_cast<OverflowingBinaryOperator>(U)) {
     if (OBO->hasNoUnsignedWrap())
       Out << " nuw";
     if (OBO->hasNoSignedWrap())

lib/IR/Instruction.cpp

bool Instruction::hasNoSignedZeros() const {
   return cast<FPMathOperator>(this)->hasNoSignedZeros();
 }
 
 bool Instruction::hasAllowReciprocal() const {
   assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
   return cast<FPMathOperator>(this)->hasAllowReciprocal();
 }
 
+bool Instruction::hasAllowContract() const {
+  assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
+  return cast<FPMathOperator>(this)->hasAllowContract();
+}
+
 FastMathFlags Instruction::getFastMathFlags() const {
   assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
   return cast<FPMathOperator>(this)->getFastMathFlags();
 }
 
 void Instruction::copyFastMathFlags(const Instruction *I) {
   copyFastMathFlags(I->getFastMathFlags());
 }

test/Assembler/fast-math-flags.ll

 ; CHECK:  %e = frem nnan  float %x, %y
   %e = frem nnan  float %x, %y
 ; CHECK:  %e_vec = frem nnan  <3 x float> %vec, %vec
   %e_vec = frem nnan  <3 x float> %vec, %vec
 ; CHECK:  ret float %e
   ret float %e
 }
 
+; CHECK: @contract(
+define float @contract(float %x, float %y) {
+entry:
+; CHECK: %a = fsub contract float %x, %y
+  %a = fsub contract float %x, %y
+; CHECK: %b = fadd contract float %x, %y
+  %b = fadd contract float %x, %y
+; CHECK: %c = fmul contract float %a, %b
+  %c = fmul contract float %a, %b
+  ret float %c
+}
+
 ; CHECK: no_nan_inf
 define float @no_nan_inf(float %x, float %y) {
 entry:
 ; CHECK:  %vec = load <3 x float>, <3 x float>* @vec
   %vec    = load <3 x float>, <3 x float>* @vec
 ; CHECK:  %select = load i1, i1* @select
   %select = load i1, i1* @select
 ; CHECK:  %arr = load [3 x float], [3 x float]* @arr

test/Bitcode/compatibility.ll

define void @fastmathflags(float %op1, float %op2) {
   %f.nnan = fadd nnan float %op1, %op2
   ; CHECK: %f.nnan = fadd nnan float %op1, %op2
   %f.ninf = fadd ninf float %op1, %op2
   ; CHECK: %f.ninf = fadd ninf float %op1, %op2
   %f.nsz = fadd nsz float %op1, %op2
   ; CHECK: %f.nsz = fadd nsz float %op1, %op2
   %f.arcp = fadd arcp float %op1, %op2
   ; CHECK: %f.arcp = fadd arcp float %op1, %op2
+  %f.contract = fadd contract float %op1, %op2
+  ; CHECK: %f.contract = fadd contract float %op1, %op2
   %f.fast = fadd fast float %op1, %op2
   ; CHECK: %f.fast = fadd fast float %op1, %op2
   ret void
 }
 
 ; Check various fast math flags and floating-point types on calls.
 
 declare float @fmf1()

unittests/IR/IRBuilderTest.cpp

TEST_F(IRBuilderTest, FastMathFlags) {
   FC = Builder.CreateFCmpOEQ(F, F);
   EXPECT_TRUE(Builder.getFastMathFlags().any());
   EXPECT_TRUE(Builder.getFastMathFlags().AllowReciprocal);
   ASSERT_TRUE(isa<Instruction>(FC));
   FCmp = cast<Instruction>(FC);
   EXPECT_TRUE(FCmp->hasAllowReciprocal());
 
   Builder.clearFastMathFlags();
- 
+  // Test FP-contract
+  FC = Builder.CreateFAdd(F, F);
+  ASSERT_TRUE(isa<Instruction>(FC));
+  FAdd = cast<Instruction>(FC);
+  EXPECT_FALSE(FAdd->hasAllowContract());
+
+  FMF.clear();
+  FMF.setAllowContract(true);
+  Builder.setFastMathFlags(FMF);
+
+  FC = Builder.CreateFAdd(F, F);
+  EXPECT_TRUE(Builder.getFastMathFlags().any());
+  EXPECT_TRUE(Builder.getFastMathFlags().AllowContract);
+  ASSERT_TRUE(isa<Instruction>(FC));
+  FAdd = cast<Instruction>(FC);
+  EXPECT_TRUE(FAdd->hasAllowContract());
+
+  Builder.clearFastMathFlags();
+
   // Test a call with FMF.
   auto CalleeTy = FunctionType::get(Type::getFloatTy(Ctx),
                                     /*isVarArg=*/false);
   auto Callee =
       Function::Create(CalleeTy, Function::ExternalLinkage, "", M.get());
 
   FCall = Builder.CreateCall(Callee, None);
   EXPECT_FALSE(FCall->hasNoNaNs());