[FPEnv Core 02/14] Add FPEnv access flags to fast-math flags

[sdmitrouk]

Summary

Will be used to disable optimizations related to floating-point
operations which would affect runtime semantics (e.g. raising
floating-point exceptions).

[scanon]

docs/LangRef.rst

1988

Let's avoid "thrown", which tends to conjure an entirely different notion of "exception". The IEEE-754 nomenclature is "raised".

1995

It should explicitly imply that kexc and kround are *not set*.

[majnemer]

I'd like to propose a mildly different design; I'd phrase the two flags using a positive perspective:

• nexc: Assume that floating-point exceptions are not relevant.
• nrnd: Assume that the rounding-mode is round-to-nearest (ties even).

This has the nice property that fast is the union of all flags and, in general, keeps things consistent with all the other fast math flags. The general pattern is that adding flags permits further optimization; places where this is violated have been a wellspring of bugs (namely volatile).

Older bitcode would have to be auto-upgraded to add those flags onto older instructions.

[scanon]

In D14067#275220, @majnemer wrote:

I'd like to propose a mildly different design; I'd phrase the two flags using a positive perspective:

• nexc: Assume that floating-point exceptions are not relevant.
• nrnd: Assume that the rounding-mode is round-to-nearest (ties even).

This has the nice property that fast is the union of all flags and, in general, keeps things consistent with all the other fast math flags. The general pattern is that adding flags permits further optimization; places where this is violated have been a wellspring of bugs (namely volatile).

I like this ("observed" instead of "relevant", perhaps).

[sdmitrouk]

In D14067#275220, @majnemer wrote:

I'd like to propose a mildly different design; I'd phrase the two flags using a positive perspective:

• nexc: Assume that floating-point exceptions are not relevant.
• nrnd: Assume that the rounding-mode is round-to-nearest (ties even).

This has the nice property that fast is the union of all flags and, in general, keeps things consistent with all the other fast math flags. The general pattern is that adding flags permits further optimization; places where this is violated have been a wellspring of bugs (namely volatile).

Older bitcode would have to be auto-upgraded to add those flags onto older instructions.

There is a compatibility issue. kexc and kround can be added when they are needed leaving regular behaviour for cases where these flags are not specified, they also have property of having all fast-math flags unset by default. Combination of nexc and nrnd flags corresponds to current behaviour and all new code that wants folding will need to include these flags explicitly. Or is this what you mean by saying Older bitcode would have to be auto-upgraded to add those flags onto older instructions.?

[sdmitrouk]

In D14067#277179, @sdmitrouk wrote:

In D14067#275220, @majnemer wrote:

Older bitcode would have to be auto-upgraded to add those flags onto older instructions.

There is a compatibility issue. kexc and kround can be added when they are needed leaving regular behaviour for cases where these flags are not specified, they also have property of having all fast-math flags unset by default. Combination of nexc and nrnd flags corresponds to current behaviour and all new code that wants folding will need to include these flags explicitly. Or is this what you mean by saying Older bitcode would have to be auto-upgraded to add those flags onto older instructions.?

@majnemer, I guess you were talking about AutoUpgrade.cpp rather than the issue I mentioned. I don't see a good way of handling this change, existing .ll files with FP instructions will change their meaning if not updated to include nexc and nrnd. I think such change isn't worth it. Is it? Or just keep kexc and kround (maybe rename it to krnd then)?

[sdmitrouk]

Ping.

[sdmitrouk]

Ping.

[sdmitrouk]

Ping.

[mehdi_amini]

We don't provide any guarantee for .ll compatibility across version of llvm (best effort).
So it is only when reading a .bc that does not contain the flags that you need to set the default value correctly.

[sdmitrouk]

In D14067#296684, @joker.eph wrote:

We don't provide any guarantee for .ll compatibility across version of llvm (best effort).
So it is only when reading a .bc that does not contain the flags that you need to set the default value correctly.

Thank you, Mehdi.

[mehdi_amini]

Something that also probably needs to be thought about is what will be the default behavior for clang and how to control it?

[scanon]

Something that also probably needs to be thought about is what will be the default behavior for clang and how to control it?

The default behavior should be assume-default-rounding and dont-care-about-flags. These should be controlled jointly by #pragma STDC FENV_ACCESS [ON|OFF], and possibly individually by some other means.

[mehdi_amini]

Sounds reasonable to me, but it is unfortunate that the "default" textual IR will be clobbered with these two flags everywhere.

[sdmitrouk]

Inverted meaning of flags, updated text of comments.

[scanon]

Thanks!

[mehdi_amini]

lib/Bitcode/Reader/BitcodeReader.cpp

4152

Could we avoid the code duplication by not doing it an else block here?

[mehdi_amini]

LGTM, thanks.

[mehdi_amini]

(but please try to remove the code duplication before committing if possible)

[sdmitrouk]

Removed code duplication and moved several tests from patch#4 here. Will wait until patch#4 is accepted before committing, otherwise new flags will become visible in IR, but won't be used by any code, which can confuse.

docs/LangRef.rst

 ``nsz``
    No Signed Zeros - Allow optimizations to treat the sign of a zero
    argument or result as insignificant.
 
 ``arcp``
    Allow Reciprocal - Allow optimizations to use the reciprocal of an
    argument rather than perform division.
 
+``nexc``
+   No Exceptions - Assume that floating-point exceptions are not observed.
+
+``nrnd``
+   No Rounding - Assume that the rounding-mode is round-to-nearest (ties even).
+
 ``fast``
    Fast - Allow algebraically equivalent transformations that may
    dramatically change results in floating point (e.g. reassociate). This
    flag implies all the others.
 
 .. _uselistorder:
 
 Use-list Order Directives
 -------------------------
 
 Use-list directives encode the in-memory order of each use-list, allowing the
 order to be recreated. ``<order-indexes>`` is a comma-separated list of

include/llvm/IR/Instruction.h

public:
   /// this flag.
   void setHasNoSignedZeros(bool B);
 
   /// Set or clear the allow-reciprocal flag on this instruction, which must be
   /// an operator which supports this flag. See LangRef.html for the meaning of
   /// this flag.
   void setHasAllowReciprocal(bool B);
 
+  /// Set or clear the no-exceptions flag on this instruction, which must be an
+  /// operator which supports this flag. See LangRef.html for the meaning of
+  /// this flag.
+  void setHasNoExceptions(bool B);
+
+  /// Set or clear the no-rounding flag on this instruction, which must be an
+  /// operator which supports this flag. See LangRef.html for the meaning of
+  /// this flag.
+  void setHasNoRounding(bool B);
+
   /// Convenience function for setting multiple fast-math flags on this
   /// instruction, which must be an operator which supports these flags. See
   /// LangRef.html for the meaning of these flags.
   void setFastMathFlags(FastMathFlags FMF);
 
   /// Convenience function for transferring all fast-math flag values to this
   /// instruction, which must be an operator which supports these flags. See
   /// LangRef.html for the meaning of these flags.
public:
   bool hasNoInfs() const;
 
   /// Determine whether the no-signed-zeros flag is set.
   bool hasNoSignedZeros() const;
 
   /// Determine whether the allow-reciprocal flag is set.
   bool hasAllowReciprocal() const;
 
+  /// Determine whether the no-exceptions flag is set.
+  bool hasNoExceptions() const;
+
+  /// Determine whether the no-rounding flag is set.
+  bool hasNoRounding() const;
+
   /// Convenience function for getting all the fast-math flags, which must be an
   /// operator which supports these flags. See LangRef.html for the meaning of
   /// these flags.
   FastMathFlags getFastMathFlags() const;
 
   /// Copy I's fast-math flags
   void copyFastMathFlags(const Instruction *I);
 

include/llvm/IR/Operator.h

private:
   FastMathFlags(unsigned F) : Flags(F) { }
 
 public:
   enum {
     UnsafeAlgebra   = (1 << 0),
     NoNaNs          = (1 << 1),
     NoInfs          = (1 << 2),
     NoSignedZeros   = (1 << 3),
-    AllowReciprocal = (1 << 4)
+    AllowReciprocal = (1 << 4),
+    NoExceptions    = (1 << 5),
+    NoRounding      = (1 << 6)
   };
 
   FastMathFlags() : Flags(0)
   { }
 
   /// Whether any flag is set
   bool any() const { return Flags != 0; }
 
   /// Set all the flags to false
   void clear() { Flags = 0; }
 
   /// Flag queries
   bool noNaNs() const          { return 0 != (Flags & NoNaNs); }
   bool noInfs() const          { return 0 != (Flags & NoInfs); }
   bool noSignedZeros() const   { return 0 != (Flags & NoSignedZeros); }
   bool allowReciprocal() const { return 0 != (Flags & AllowReciprocal); }
   bool unsafeAlgebra() const   { return 0 != (Flags & UnsafeAlgebra); }
+  bool noExceptions() const    { return 0 != (Flags & NoExceptions); }
+  bool noRounding() const      { return 0 != (Flags & NoRounding); }
 
   /// Flag setters
   void setNoNaNs()          { Flags |= NoNaNs; }
   void setNoInfs()          { Flags |= NoInfs; }
   void setNoSignedZeros()   { Flags |= NoSignedZeros; }
   void setAllowReciprocal() { Flags |= AllowReciprocal; }
+  void setNoExceptions()    { Flags |= NoExceptions; }
+  void setNoRounding()      { Flags |= NoRounding; }
   void setUnsafeAlgebra() {
     Flags |= UnsafeAlgebra;
     setNoNaNs();
     setNoInfs();
     setNoSignedZeros();
     setAllowReciprocal();
+    setNoExceptions();
+    setNoRounding();
   }
 
   void operator&=(const FastMathFlags &OtherFlags) {
     Flags &= OtherFlags.Flags;
   }
 };
 
 
SubclassOptionalData =
       (SubclassOptionalData & ~FastMathFlags::NoSignedZeros) |
       (B * FastMathFlags::NoSignedZeros);
   }
   void setHasAllowReciprocal(bool B) {
     SubclassOptionalData =
       (SubclassOptionalData & ~FastMathFlags::AllowReciprocal) |
       (B * FastMathFlags::AllowReciprocal);
   }
+  void setHasNoExceptions(bool B) {
+    SubclassOptionalData =
+      (SubclassOptionalData & ~FastMathFlags::NoExceptions) |
+      (B * FastMathFlags::NoExceptions);
+  }
+  void setHasNoRounding(bool B) {
+    SubclassOptionalData =
+      (SubclassOptionalData & ~FastMathFlags::NoRounding) |
+      (B * FastMathFlags::NoRounding);
+  }
 
   /// Convenience function for setting multiple fast-math flags.
   /// FMF is a mask of the bits to set.
   void setFastMathFlags(FastMathFlags FMF) {
     SubclassOptionalData |= FMF.Flags;
   }
 
   /// Convenience function for copying all fast-math flags.
public:
   }
 
   /// Test whether this operation is permitted to use
   /// reciprocal instead of division, aka the 'R' fast-math property.
   bool hasAllowReciprocal() const {
     return (SubclassOptionalData & FastMathFlags::AllowReciprocal) != 0;
   }
 
+  /// Test whether this operation can be optimized even if its runtime
+  /// semantics regarding floating-point exceptions won't be preserved.
+  bool hasNoExceptions() const {
+    return (SubclassOptionalData & FastMathFlags::NoExceptions) != 0;
+  }
+
+  /// Test whether this operation can be optimized assuming use of
+  /// round-to-nearest (ties even) rounding mode at run-time.
+  bool hasNoRounding() const {
+    return (SubclassOptionalData & FastMathFlags::NoRounding) != 0;
+  }
+
   /// Convenience function for getting all the fast-math flags
   FastMathFlags getFastMathFlags() const {
     return FastMathFlags(SubclassOptionalData);
   }
 
   /// \brief Get the maximum error permitted by this operation in ULPs.  An
   /// accuracy of 0.0 means that the operation should be performed with the
   /// default precision.

lib/AsmParser/LLLexer.cpp

#define KEYWORD(STR)                                                           \
   KEYWORD(acq_rel);
   KEYWORD(seq_cst);
   KEYWORD(singlethread);
 
   KEYWORD(nnan);
   KEYWORD(ninf);
   KEYWORD(nsz);
   KEYWORD(arcp);
+  KEYWORD(nexc);
+  KEYWORD(nrnd);
   KEYWORD(fast);
   KEYWORD(nuw);
   KEYWORD(nsw);
   KEYWORD(exact);
   KEYWORD(inbounds);
   KEYWORD(align);
   KEYWORD(addrspace);
   KEYWORD(section);

lib/AsmParser/LLParser.h

FastMathFlags EatFastMathFlagsIfPresent() {
       FastMathFlags FMF;
       while (true)
         switch (Lex.getKind()) {
         case lltok::kw_fast: FMF.setUnsafeAlgebra();   Lex.Lex(); continue;
         case lltok::kw_nnan: FMF.setNoNaNs();          Lex.Lex(); continue;
         case lltok::kw_ninf: FMF.setNoInfs();          Lex.Lex(); continue;
         case lltok::kw_nsz:  FMF.setNoSignedZeros();   Lex.Lex(); continue;
         case lltok::kw_arcp: FMF.setAllowReciprocal(); Lex.Lex(); continue;
+        case lltok::kw_nexc: FMF.setNoExceptions();    Lex.Lex(); continue;
+        case lltok::kw_nrnd: FMF.setNoRounding();      Lex.Lex(); continue;
         default: return FMF;
         }
       return FMF;
     }
 
     bool ParseOptionalToken(lltok::Kind T, bool &Present,
                             LocTy *Loc = nullptr) {
       if (Lex.getKind() != T) {

lib/AsmParser/LLToken.h

enum Kind {
     kw_volatile,
     kw_atomic,
     kw_unordered, kw_monotonic, kw_acquire, kw_release, kw_acq_rel, kw_seq_cst,
     kw_singlethread,
     kw_nnan,
     kw_ninf,
     kw_nsz,
     kw_arcp,
+    kw_nexc,
+    kw_nrnd,
     kw_fast,
     kw_nuw,
     kw_nsw,
     kw_exact,
     kw_inbounds,
     kw_align,
     kw_addrspace,
     kw_section,

lib/Bitcode/Reader/BitcodeReader.cpp

class BitcodeReader : public GVMaterializer {
   /// Indicates that we are using a new encoding for instruction operands where
   /// most operands in the current FUNCTION_BLOCK are encoded relative to the
   /// instruction number, for a more compact encoding.  Some instruction
   /// operands are not relative to the instruction ID: basic block numbers, and
   /// types. Once the old style function blocks have been phased out, we would
   /// not need this flag.
   bool UseRelativeIDs = false;
 
+  /// Indicates that floating-point environment flags must be set to values
+  /// that correspond to behaviour before addition of nexc and nrnd fast-math
+  /// flags (i.e., enable aggressive optimizations of operations on
+  /// floating-point values).
+  bool SetNoFEnvFMF = false;
+
   /// True if all functions will be materialized, negating the need to process
   /// (e.g.) blockaddress forward references.
   bool WillMaterializeAllForwardRefs = false;
 
   /// Functions that have block addresses taken.  This is usually empty.
   SmallPtrSet<const Function *, 4> BlockAddressesTaken;
 
   /// True if any Metadata block has been materialized.
static FastMathFlags getDecodedFastMathFlags(unsigned Val) {
   if (0 != (Val & FastMathFlags::NoNaNs))
     FMF.setNoNaNs();
   if (0 != (Val & FastMathFlags::NoInfs))
     FMF.setNoInfs();
   if (0 != (Val & FastMathFlags::NoSignedZeros))
     FMF.setNoSignedZeros();
   if (0 != (Val & FastMathFlags::AllowReciprocal))
     FMF.setAllowReciprocal();
+  if (0 != (Val & FastMathFlags::NoExceptions))
+    FMF.setNoExceptions();
+  if (0 != (Val & FastMathFlags::NoRounding))
+    FMF.setNoRounding();
   return FMF;
 }
 
 static void upgradeDLLImportExportLinkage(llvm::GlobalValue *GV, unsigned Val) {
   switch (Val) {
   case 5: GV->setDLLStorageClass(GlobalValue::DLLImportStorageClass); break;
   case 6: GV->setDLLStorageClass(GlobalValue::DLLExportStorageClass); break;
   }
while (1) {
 
     // Read a record.
     auto BitCode = Stream.readRecord(Entry.ID, Record);
     switch (BitCode) {
     default: break;  // Default behavior, ignore unknown content.
     case bitc::MODULE_CODE_VERSION: {  // VERSION: [version#]
       if (Record.size() < 1)
         return error("Invalid record");
-      // Only version #0 and #1 are supported so far.
+      // Only version #0 -- #2 are supported so far.
       unsigned module_version = Record[0];
       switch (module_version) {
         default:
           return error("Invalid value");
         case 0:
           UseRelativeIDs = false;
+          SetNoFEnvFMF = true;
           break;
         case 1:
           UseRelativeIDs = true;
+          SetNoFEnvFMF = true;
+          break;
+        case 2:
+          UseRelativeIDs = true;
+          SetNoFEnvFMF = false;
           break;
       }
       break;
     }
     case bitc::MODULE_CODE_TRIPLE: {  // TRIPLE: [strchr x N]
       std::string S;
       if (convertToString(Record, 0, S))
         return error("Invalid record");
case bitc::FUNC_CODE_INST_BINOP: {    // BINOP: [opval, ty, opval, opcode]
         } else if (Opc == Instruction::SDiv ||
                    Opc == Instruction::UDiv ||
                    Opc == Instruction::LShr ||
                    Opc == Instruction::AShr) {
           if (Record[OpNum] & (1 << bitc::PEO_EXACT))
             cast<BinaryOperator>(I)->setIsExact(true);
         } else if (isa<FPMathOperator>(I)) {
           FastMathFlags FMF = getDecodedFastMathFlags(Record[OpNum]);
+          if (SetNoFEnvFMF) {
+            // Enable optimizations for older bitcode, when these flags were not
+            // available.
+            FMF.setNoExceptions();
+            FMF.setNoRounding();
+          }
           if (FMF.any())
             I->setFastMathFlags(FMF);
         }
-
+      } else if (SetNoFEnvFMF && isa<FPMathOperator>(I)) {
+        FastMathFlags FMF;
+        // Enable optimizations for older bitcode, when these flags were not
+        // available.
+        FMF.setNoExceptions();
+        FMF.setNoRounding();
+        I->setFastMathFlags(FMF);
       }
       break;
     }
     case bitc::FUNC_CODE_INST_CAST: {    // CAST: [opval, opty, destty, castopc]
       unsigned OpNum = 0;
       Value *Op;
       if (getValueTypePair(Record, OpNum, NextValueNo, Op) ||
           OpNum+2 != Record.size())

lib/Bitcode/Writer/BitcodeWriter.cpp

if (const auto *OBO = dyn_cast<OverflowingBinaryOperator>(V)) {
     if (FPMO->hasNoNaNs())
       Flags |= FastMathFlags::NoNaNs;
     if (FPMO->hasNoInfs())
       Flags |= FastMathFlags::NoInfs;
     if (FPMO->hasNoSignedZeros())
       Flags |= FastMathFlags::NoSignedZeros;
     if (FPMO->hasAllowReciprocal())
       Flags |= FastMathFlags::AllowReciprocal;
+    if (FPMO->hasNoExceptions())
+      Flags |= FastMathFlags::NoExceptions;
+    if (FPMO->hasNoRounding())
+      Flags |= FastMathFlags::NoRounding;
   }
 
   return Flags;
 }
 
 static void WriteValueAsMetadata(const ValueAsMetadata *MD,
                                  const ValueEnumerator &VE,
                                  BitstreamWriter &Stream,

 
 /// WriteModule - Emit the specified module to the bitstream.
 static void WriteModule(const Module *M, BitstreamWriter &Stream,
                         bool ShouldPreserveUseListOrder,
                         uint64_t BitcodeStartBit, bool EmitFunctionSummary) {
   Stream.EnterSubblock(bitc::MODULE_BLOCK_ID, 3);
 
   SmallVector<unsigned, 1> Vals;
-  unsigned CurVersion = 1;
+  unsigned CurVersion = 2;
   Vals.push_back(CurVersion);
   Stream.EmitRecord(bitc::MODULE_CODE_VERSION, Vals);
 
   // Analyze the module, enumerating globals, functions, etc.
   ValueEnumerator VE(*M, ShouldPreserveUseListOrder);
 
   // Emit blockinfo, which defines the standard abbreviations etc.
   WriteBlockInfo(VE, Stream);
void llvm::WriteFunctionSummaryToFile(const FunctionInfoIndex &Index,
   BitstreamWriter Stream(Buffer);
 
   // Emit the bitcode header.
   WriteBitcodeHeader(Stream);
 
   Stream.EnterSubblock(bitc::MODULE_BLOCK_ID, 3);
 
   SmallVector<unsigned, 1> Vals;
-  unsigned CurVersion = 1;
+  unsigned CurVersion = 2;
   Vals.push_back(CurVersion);
   Stream.EmitRecord(bitc::MODULE_CODE_VERSION, Vals);
 
   // Write the module paths in the combined index.
   WriteModStrings(Index, Stream);
 
   // Write the function summary combined index records.
   WriteCombinedFunctionSummary(Index, Stream);

lib/IR/AsmWriter.cpp

else {
       if (FPO->hasNoNaNs())
         Out << " nnan";
       if (FPO->hasNoInfs())
         Out << " ninf";
       if (FPO->hasNoSignedZeros())
         Out << " nsz";
       if (FPO->hasAllowReciprocal())
         Out << " arcp";
+      if (FPO->hasNoExceptions())
+        Out << " nexc";
+      if (FPO->hasNoRounding())
+        Out << " nrnd";
     }
   }
 
   if (const OverflowingBinaryOperator *OBO =
         dyn_cast<OverflowingBinaryOperator>(U)) {
     if (OBO->hasNoUnsignedWrap())
       Out << " nuw";
     if (OBO->hasNoSignedWrap())

lib/IR/Instruction.cpp

 /// Set or clear the allow-reciprocal flag on this instruction, which must be an
 /// operator which supports this flag. See LangRef.html for the meaning of this
 /// flag.
 void Instruction::setHasAllowReciprocal(bool B) {
   assert(isa<FPMathOperator>(this) && "setting fast-math flag on invalid op");
   cast<FPMathOperator>(this)->setHasAllowReciprocal(B);
 }
 
+/// Set or clear the no-exceptions flag on this instruction, which must be an
+/// operator which supports this flag. See LangRef.html for the meaning of this
+/// flag.
+void Instruction::setHasNoExceptions(bool B) {
+  assert(isa<FPMathOperator>(this) && "setting fast-math flag on invalid op");
+  cast<FPMathOperator>(this)->setHasNoExceptions(B);
+}
+
+/// Set or clear the no-rounding flag on this instruction, which must be an
+/// operator which supports this flag. See LangRef.html for the meaning of this
+/// flag.
+void Instruction::setHasNoRounding(bool B) {
+  assert(isa<FPMathOperator>(this) && "setting fast-math flag on invalid op");
+  cast<FPMathOperator>(this)->setHasNoRounding(B);
+}
+
 /// Convenience function for setting all the fast-math flags on this
 /// instruction, which must be an operator which supports these flags. See
 /// LangRef.html for the meaning of these flats.
 void Instruction::setFastMathFlags(FastMathFlags FMF) {
   assert(isa<FPMathOperator>(this) && "setting fast-math flag on invalid op");
   cast<FPMathOperator>(this)->setFastMathFlags(FMF);
 }
 

 }
 
 /// Determine whether the allow-reciprocal flag is set.
 bool Instruction::hasAllowReciprocal() const {
   assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
   return cast<FPMathOperator>(this)->hasAllowReciprocal();
 }
 
+/// Determine whether the no-exceptions flag is set.
+bool Instruction::hasNoExceptions() const {
+  assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
+  return cast<FPMathOperator>(this)->hasNoExceptions();
+}
+
+/// Determine whether the no-rounding flag is set.
+bool Instruction::hasNoRounding() const {
+  assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
+  return cast<FPMathOperator>(this)->hasNoRounding();
+}
+
 /// Convenience function for getting all the fast-math flags, which must be an
 /// operator which supports these flags. See LangRef.html for the meaning of
 /// these flags.
 FastMathFlags Instruction::getFastMathFlags() const {
   assert(isa<FPMathOperator>(this) && "getting fast-math flag on invalid op");
   return cast<FPMathOperator>(this)->getFastMathFlags();
 }
 

test/Assembler/fast-math-flags.ll

 ; CHECK:  %d = fdiv nnan ninf arcp float %x, %y
   %d = fdiv arcp ninf nnan float %x, %y
 ; CHECK:  %d_vec = fdiv fast <3 x float> %vec, %vec
   %d_vec = fdiv fast nnan arcp <3 x float> %vec, %vec
 ; CHECK:  %e = frem nnan nsz float %x, %y
   %e = frem nnan nsz float %x, %y
 ; CHECK:  %e_vec = frem nnan <3 x float> %vec, %vec
   %e_vec = frem nnan <3 x float> %vec, %vec
+; CHECK:  %f = fmul fast float %x, %y
+  %f = fmul nexc nrnd fast float %x, %y
+; CHECK:  %f_vec = frem fast <3 x float> %vec, %vec
+  %f_vec = frem nexc nrnd fast <3 x float> %vec, %vec
+; CHECK:  %g = fmul nexc nrnd float %x, %y
+  %g = fmul nexc nrnd float %x, %y
+; CHECK:  %g_vec = frem nexc nrnd <3 x float> %vec, %vec
+  %g_vec = frem nexc nrnd <3 x float> %vec, %vec
 ; CHECK:  ret  float %e
   ret  float %e
 }