add fast-math-flags to 'call' instructions (PR21290)

[spatel]

Summary

This patch adds optional fast-math-flags (the same that apply to fmul/fadd/fsub/fdiv/frem/fcmp) to call instructions in IR. Follow-up patches would use these flags in LibCallSimplifier, add support to clang, and extend FMF to the DAG for calls.

A motivating example is something like this:

%y = fmul fast float %x, %x
%z = tail call float @sqrtf(float %y)

We'd like to be able to optimize sqrt(x*x) into fabs(x). We do this today using a function-wide attribute for unsafe-math, but we really want to trigger on the instructions themselves:

%z = tail call fast float @sqrtf(float %y)

The code changes and tests are based on the recent commits that added "notail":
http://reviews.llvm.org/rL252368

and added FMF to fcmp:
http://reviews.llvm.org/rL241901

[majnemer]

Why is http://llvm.org/docs/doxygen/html/SimplifyLibCalls_8cpp_source.html#l00114 insufficient for this purpose?

[spatel]

In D14707#290108, @majnemer wrote:

Why is http://llvm.org/docs/doxygen/html/SimplifyLibCalls_8cpp_source.html#l00114 insufficient for this purpose?

Function-level attributes fall apart when code gets mixed by LTO; please see PR21290 for more details:
https://llvm.org/bugs/show_bug.cgi?id=21290

See also comments here:
http://reviews.llvm.org/D7257

[spatel]

In D14707#290137, @spatel wrote:

See also comments here:
http://reviews.llvm.org/D7257

Disregard that one - that was just about fcmp.

[ahatanak]

lib/AsmParser/LLParser.cpp

5647

Is it possible to commit the 'notail' change in a separate patch (and thank you for catching this)?

5678

Are there cases where you want to optimize calls to functions that take floating point arguments but don't have floating point return types?

[spatel]

lib/AsmParser/LLParser.cpp

5647

Sure - will do that first and update this patch.

5678

I thought about that too, however, if we use this definition, then we do not have to redefine "isa<FPMathOperator>" in this patch.

I don't know if that is a reasonable justification by itself, but I think we can expand the definition of FPMathOperator in a separate patch if we do decide that FMF should apply to more types of calls.

[majnemer]

My 2 ¢:

It seems more natural to me to phrase these as normal string attributes at the point of the call and construct an FMF from a CallSite but I can see arguments for and against both designs.

[spatel]

In D14707#292002, @majnemer wrote:

My 2 ¢:

It seems more natural to me to phrase these as normal string attributes at the point of the call and construct an FMF from a CallSite but I can see arguments for and against both designs.

I thought about it some more, looked into the implementation, and convinced myself that the attribute solution isn't as good.

1. We already have some of the function-level FMF attributes here: http://reviews.llvm.org/diffusion/L/browse/llvm/trunk/include/llvm/IR/Attributes.td but they are 'StrBoolAttr' rather than the 'EnumAttr' that I think we want. The names don't match. There would be some duplication/ugliness here at the least.
2. This is the big one: we'd need to update independent chunks of code any time FMF changes. This scares me because I'm still hoping to get advanced reciprocal estimate functionality ( http://reviews.llvm.org/rL239001 , http://reviews.llvm.org/rL239536 ) into instruction-level FMF. Those settings don't exist at all in IR currently, so in an LTO build, they simply disappear. I think the same thing happens with FMAs. The likelihood that FMF internals require changes seems high, so having a unified FMF interface for all instructions is the better solution.

[spatel]

Patch updated:
As suggested by Akira, I added the 'notail' comment and updated the error message in ParseCall() in r253580 since they are independent from this patch.

[spatel]

In D14707#293138, @spatel wrote:

The likelihood that FMF internals require changes seems high, so having a unified FMF interface for all instructions is the better solution.

As evidence that FMF is evolving, there's currently a patch under review that would add more flags to support observing the FENV:
http://reviews.llvm.org/D14067

[spatel]

Ping.

[spatel]

Ping * 2.

[hfinkel]

A few minor things, and with these changes, LGTM.

docs/LangRef.rst

8413

I find this confusing; we should just use [fast-math flags]* instead of [fmf] as we do elsewhere.

8470

Adjust for [fmf] -> [fast-math flags]*

8472

We must document here that these flags are valid only for calls with a floating-point scalar or vector return type.

lib/AsmParser/LLParser.cpp

5679

I think it would be better to say how the call was invalid (and, minus the fmfs, the call itself may be fine). How about:

return Error(CallLoc, "fast-math-flags specified for call without floating-point scalar or vector return type");

lib/Bitcode/Reader/BitcodeReader.cpp

5056

Match this error text to the text in lib/AsmParser/LLParser.cpp.

[spatel]

Patch updated based on Hal's feedback and rebased (one more existing test for SimplifyLibCalls now has the 'fast' flag).

[mehdi_amini]

I wonder about the design of this, especially since Chris (and Chandler) rejected it in the past: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20121217/159446.html ?

[mehdi_amini]

(note there might be a fundamental difference I didn't see skimming through this)

docs/LangRef.rst

 '``call``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <result> = [tail | musttail | notail ] call [cconv] [ret attrs] <ty> [<fnty>*] <fnptrval>(<function args>) [fn attrs]
+      <result> = [tail | musttail | notail ] call [fmf] [cconv] [ret attrs] <ty> [<fnty>*] <fnptrval>(<function args>) [fn attrs]
                    [ operand bundles ]
 
 Overview:
 """""""""
 
 The '``call``' instruction represents a simple function call.
 
 Arguments:
-  Option ``-tailcallopt`` is enabled, or
       ``llvm::GuaranteedTailCallOpt`` is ``true``.
    -  `Platform-specific constraints are
       met. <CodeGenerator.html#tailcallopt>`_
 
 #. The optional ``notail`` marker indicates that the optimizers should not add
    ``tail`` or ``musttail`` markers to the call. It is used to prevent tail
    call optimization from being performed on the call.
 
+#. The optional ``fmf`` marker indicates that the call has one or more 
+   :ref:`fast-math flags <fastmath>`, which are optimization hints to enable
+   otherwise unsafe floating-point optimizations.
+
 #. The optional "cconv" marker indicates which :ref:`calling
    convention <callingconv>` the call should use. If none is
    specified, the call defaults to using C calling conventions. The
    calling convention of the call must match the calling convention of
    the target function, or else the behavior is undefined.
 #. The optional :ref:`Parameter Attributes <paramattrs>` list for return
    values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
    are valid here.

include/llvm/Bitcode/LLVMBitCodes.h

enum { BITCODE_CURRENT_EPOCH = 0 };
   };
 
   /// Markers and flags for call instruction.
   enum CallMarkersFlags {
     CALL_TAIL = 0,
     CALL_CCONV = 1,
     CALL_MUSTTAIL = 14,
     CALL_EXPLICIT_TYPE = 15,
-    CALL_NOTAIL = 16
+    CALL_NOTAIL = 16,
+    CALL_FMF = 17  // Call has optional fast-math-flags.
   };
 
   // The function body block (FUNCTION_BLOCK_ID) describes function bodies.  It
   // can contain a constant block (CONSTANTS_BLOCK_ID).
   enum FunctionCodes {
     FUNC_CODE_DECLAREBLOCKS    =  1, // DECLAREBLOCKS: [n]
 
     FUNC_CODE_INST_BINOP       =  2, // BINOP:      [opcode, ty, opval, opval]

include/llvm/IR/IRBuilder.h

public:
   //===--------------------------------------------------------------------===//
 
   PHINode *CreatePHI(Type *Ty, unsigned NumReservedValues,
                      const Twine &Name = "") {
     return Insert(PHINode::Create(Ty, NumReservedValues), Name);
   }
 
   CallInst *CreateCall(Value *Callee, ArrayRef<Value *> Args = None,
-                       const Twine &Name = "") {
-    return Insert(CallInst::Create(Callee, Args), Name);
+                       const Twine &Name = "", MDNode *FPMathTag = nullptr) {
+    PointerType *PTy = cast<PointerType>(Callee->getType());
+    FunctionType *FTy = cast<FunctionType>(PTy->getElementType());
+    return CreateCall(FTy, Callee, Args, Name, FPMathTag);
   }
 
   CallInst *CreateCall(llvm::FunctionType *FTy, Value *Callee,
-                       ArrayRef<Value *> Args, const Twine &Name = "") {
-    return Insert(CallInst::Create(FTy, Callee, Args), Name);
+                       ArrayRef<Value *> Args, const Twine &Name = "",
+                       MDNode *FPMathTag = nullptr) {
+    CallInst *CI = CallInst::Create(FTy, Callee, Args);
+    if (isa<FPMathOperator>(CI))
+      CI = cast<CallInst>(AddFPMathAttributes(CI, FPMathTag, FMF));
+    return Insert(CI, Name);
   }
 
   CallInst *CreateCall(Function *Callee, ArrayRef<Value *> Args,
-                       const Twine &Name = "") {
-    return CreateCall(Callee->getFunctionType(), Callee, Args, Name);
+                       const Twine &Name = "", MDNode *FPMathTag = nullptr) {
+    return CreateCall(Callee->getFunctionType(), Callee, Args, Name, FPMathTag);
   }
 
   Value *CreateSelect(Value *C, Value *True, Value *False,
                       const Twine &Name = "") {
     if (Constant *CC = dyn_cast<Constant>(C))
       if (Constant *TC = dyn_cast<Constant>(True))
         if (Constant *FC = dyn_cast<Constant>(False))
           return Insert(Folder.CreateSelect(CC, TC, FC), Name);

lib/AsmParser/LLParser.cpp

while (Lex.getKind() == lltok::kw_catch || Lex.getKind() == lltok::kw_filter){
     LP->addClause(CV);
   }
 
   Inst = LP.release();
   return false;
 }
 
 /// ParseCall
-///   ::= 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
-///   ::= 'tail' 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
-///   ::= 'musttail' 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
+///   ::= 'call' OptionalFastMathFlags OptionalCallingConv
+///       OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'tail' 'call' OptionalFastMathFlags OptionalCallingConv
+///       OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'musttail' 'call' OptionalFastMathFlags OptionalCallingConv
+///       OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'notail' 'call' OptionalFastMathFlags OptionalCallingConv
+///       OptionalAttrs Type Value ParameterList OptionalAttrs
 bool LLParser::ParseCall(Instruction *&Inst, PerFunctionState &PFS,
                          CallInst::TailCallKind TCK) {
   AttrBuilder RetAttrs, FnAttrs;
   std::vector<unsigned> FwdRefAttrGrps;
   LocTy BuiltinLoc;
   unsigned CC;
   Type *RetType = nullptr;
   LocTy RetTypeLoc;
   ValID CalleeID;
   SmallVector<ParamInfo, 16> ArgList;
   SmallVector<OperandBundleDef, 2> BundleList;
   LocTy CallLoc = Lex.getLoc();
 
-  if ((TCK != CallInst::TCK_None &&
-       ParseToken(lltok::kw_call, "expected 'tail call'")) ||
-      ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+  if (TCK != CallInst::TCK_None &&
+      ParseToken(lltok::kw_call,
+                 "expected 'tail call', 'musttail call', or 'notail call'"))
+    return true;
+
+  FastMathFlags FMF = EatFastMathFlagsIfPresent();
+
+  if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
       ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
       ParseValID(CalleeID) ||
       ParseParameterList(ArgList, PFS, TCK == CallInst::TCK_MustTail,
                          PFS.getFunction().isVarArg()) ||
       ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false, BuiltinLoc) ||
       ParseOptionalOperandBundles(BundleList, PFS))
     return true;
 
+  if (FMF.any() && !RetType->isFPOrFPVectorTy())
+    return Error(CallLoc, "fast-math-flags specified for invalid call");
+
   // If RetType is a non-function pointer type, then this is the short syntax
   // for the call, which means that RetType is just the return type.  Infer the
   // rest of the function argument types from the arguments that are present.
   FunctionType *Ty = dyn_cast<FunctionType>(RetType);
   if (!Ty) {
     // Pull out the types of all of the arguments...
     std::vector<Type*> ParamTypes;
     for (unsigned i = 0, e = ArgList.size(); i != e; ++i)
bool LLParser::ParseCall(Instruction *&Inst, PerFunctionState &PFS,
   }
 
   // Finish off the Attribute and check them
   AttributeSet PAL = AttributeSet::get(Context, Attrs);
 
   CallInst *CI = CallInst::Create(Ty, Callee, Args, BundleList);
   CI->setTailCallKind(TCK);
   CI->setCallingConv(CC);
+  if (FMF.any())
+    CI->setFastMathFlags(FMF);
   CI->setAttributes(PAL);
   ForwardRefAttrGroups[CI] = FwdRefAttrGrps;
   Inst = CI;
   return false;
 }
 
 //===----------------------------------------------------------------------===//
 // Memory Instructions.

lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_FENCE: { // FENCE:[ordering, synchscope]
           Ordering == Monotonic)
         return error("Invalid record");
       SynchronizationScope SynchScope = getDecodedSynchScope(Record[1]);
       I = new FenceInst(Context, Ordering, SynchScope);
       InstructionList.push_back(I);
       break;
     }
     case bitc::FUNC_CODE_INST_CALL: {
-      // CALL: [paramattrs, cc, fnty, fnid, arg0, arg1...]
+      // CALL: [paramattrs, cc, fmf, fnty, fnid, arg0, arg1...]
       if (Record.size() < 3)
         return error("Invalid record");
 
       unsigned OpNum = 0;
       AttributeSet PAL = getAttributes(Record[OpNum++]);
       unsigned CCInfo = Record[OpNum++];
 
+      FastMathFlags FMF;
+      if ((CCInfo >> bitc::CALL_FMF) & 1) {
+        FMF = getDecodedFastMathFlags(Record[OpNum++]);
+        if (!FMF.any())
+          return error("Fast math flags indicator set for call with no FMF");
+      }
+
       FunctionType *FTy = nullptr;
       if (CCInfo >> bitc::CALL_EXPLICIT_TYPE & 1 &&
           !(FTy = dyn_cast<FunctionType>(getTypeByID(Record[OpNum++]))))
         return error("Explicit call type is not a function type");
 
       Value *Callee;
       if (getValueTypePair(Record, OpNum, NextValueNo, Callee))
         return error("Invalid record");
case bitc::FUNC_CODE_INST_CALL: {
       if (CCInfo & 1 << bitc::CALL_TAIL)
         TCK = CallInst::TCK_Tail;
       if (CCInfo & (1 << bitc::CALL_MUSTTAIL))
         TCK = CallInst::TCK_MustTail;
       if (CCInfo & (1 << bitc::CALL_NOTAIL))
         TCK = CallInst::TCK_NoTail;
       cast<CallInst>(I)->setTailCallKind(TCK);
       cast<CallInst>(I)->setAttributes(PAL);
+      if (FMF.any()) {
+        if (!isa<FPMathOperator>(I))
+          return error("Fast math flags set for non-floating-point call");
+        I->setFastMathFlags(FMF);
+      }
       break;
     }
     case bitc::FUNC_CODE_INST_VAARG: { // VAARG: [valistty, valist, instty]
       if (Record.size() < 3)
         return error("Invalid record");
       Type *OpTy = getTypeByID(Record[0]);
       Value *Op = getValue(Record, 1, NextValueNo, OpTy);
       Type *ResTy = getTypeByID(Record[2]);

lib/Bitcode/Writer/BitcodeWriter.cpp

case Instruction::Call: {
     FunctionType *FTy = CI.getFunctionType();
 
     if (CI.hasOperandBundles())
       WriteOperandBundles(Stream, &CI, InstID, VE);
 
     Code = bitc::FUNC_CODE_INST_CALL;
 
     Vals.push_back(VE.getAttributeID(CI.getAttributes()));
+
+    unsigned Flags = GetOptimizationFlags(&I);
     Vals.push_back(CI.getCallingConv() << bitc::CALL_CCONV |
                    unsigned(CI.isTailCall()) << bitc::CALL_TAIL |
                    unsigned(CI.isMustTailCall()) << bitc::CALL_MUSTTAIL |
                    1 << bitc::CALL_EXPLICIT_TYPE |
-                   unsigned(CI.isNoTailCall()) << bitc::CALL_NOTAIL);
+                   unsigned(CI.isNoTailCall()) << bitc::CALL_NOTAIL |
+                   unsigned(Flags != 0) << bitc::CALL_FMF);
+    if (Flags != 0)
+      Vals.push_back(Flags);
+
     Vals.push_back(VE.getTypeID(FTy));
     PushValueAndType(CI.getCalledValue(), InstID, Vals, VE);  // Callee
 
     // Emit value #'s for the fixed parameters.
     for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i) {
       // Check for labels (can happen with asm labels).
       if (FTy->getParamType(i)->isLabelTy())
         Vals.push_back(VE.getValueID(CI.getArgOperand(i)));

test/Bitcode/compatibility.ll

define void @fastmathflags(float %op1, float %op2) {
   ; CHECK: %f.nsz = fadd nsz float %op1, %op2
   %f.arcp = fadd arcp float %op1, %op2
   ; CHECK: %f.arcp = fadd arcp float %op1, %op2
   %f.fast = fadd fast float %op1, %op2
   ; CHECK: %f.fast = fadd fast float %op1, %op2
   ret void
 }
 
+; Check various fast math flags and floating-point types on calls.
+
+declare float @fmf1()
+declare double @fmf2()
+declare <4 x double> @fmf3()
+
+; CHECK-LABEL: fastMathFlagsForCalls(
+define void @fastMathFlagsForCalls(float %f, double %d1, <4 x double> %d2) {
+  %call.fast = call fast float @fmf1()
+  ; CHECK: %call.fast = call fast float @fmf1()
+
+  ; Throw in some other attributes to make sure those stay in the right places.
+
+  %call.nsz.arcp = notail call nsz arcp double @fmf2()
+  ; CHECK: %call.nsz.arcp = notail call nsz arcp double @fmf2()
+
+  %call.nnan.ninf = tail call nnan ninf fastcc <4 x double> @fmf3()
+  ; CHECK: %call.nnan.ninf = tail call nnan ninf fastcc <4 x double> @fmf3()
+
+  ret void
+}
+
 ;; Type System
 %opaquety = type opaque
 define void @typesystem() {
   %p0 = bitcast i8* null to i32 (i32)*
   ; CHECK: %p0 = bitcast i8* null to i32 (i32)*
   %p1 = bitcast i8* null to void (i8*)*
   ; CHECK: %p1 = bitcast i8* null to void (i8*)*
   %p2 = bitcast i8* null to i32 (i8*, ...)*

test/Transforms/InstCombine/fast-math.ll

 declare double @llvm.sqrt.f64(double)
 
 define double @sqrt_intrinsic_arg_squared(double %x) #0 {
   %mul = fmul fast double %x, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_squared(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
 ; CHECK-NEXT: ret double %fabs
 }
 
 ; Check all 6 combinations of a 3-way multiplication tree where
 ; one factor is repeated.
 
 define double @sqrt_intrinsic_three_args1(double %x, double %y) #0 {
   %mul = fmul fast double %y, %x
   %mul2 = fmul fast double %mul, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args1(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args2(double %x, double %y) #0 {
   %mul = fmul fast double %x, %y
   %mul2 = fmul fast double %mul, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args2(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args3(double %x, double %y) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %y
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args3(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args4(double %x, double %y) #0 {
   %mul = fmul fast double %y, %x
   %mul2 = fmul fast double %x, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args4(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args5(double %x, double %y) #0 {
   %mul = fmul fast double %x, %y
   %mul2 = fmul fast double %x, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args5(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args6(double %x, double %y) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %y, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args6(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_arg_4th(double %x) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_4th(
 ; CHECK-NEXT: %mul = fmul fast double %x, %x
 ; CHECK-NEXT: ret double %mul
 }
 
 define double @sqrt_intrinsic_arg_5th(double %x) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %x
   %mul3 = fmul fast double %mul2, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul3)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_5th(
 ; CHECK-NEXT: %mul = fmul fast double %x, %x
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %x)
 ; CHECK-NEXT: %1 = fmul fast double %mul, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 ; Check that square root calls have the same behavior.
 
 declare float @sqrtf(float)
 declare double @sqrt(double)
 declare fp128 @sqrtl(fp128)
 
 define float @sqrt_call_squared_f32(float %x) #0 {
   %mul = fmul fast float %x, %x
   %sqrt = call float @sqrtf(float %mul)
   ret float %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f32(
-; CHECK-NEXT: %fabs = call float @llvm.fabs.f32(float %x)
+; CHECK-NEXT: %fabs = call fast float @llvm.fabs.f32(float %x)
 ; CHECK-NEXT: ret float %fabs
 }
 
 define double @sqrt_call_squared_f64(double %x) #0 {
   %mul = fmul fast double %x, %x
   %sqrt = call double @sqrt(double %mul)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f64(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
 ; CHECK-NEXT: ret double %fabs
 }
 
 define fp128 @sqrt_call_squared_f128(fp128 %x) #0 {
   %mul = fmul fast fp128 %x, %x
   %sqrt = call fp128 @sqrtl(fp128 %mul)
   ret fp128 %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f128(
-; CHECK-NEXT: %fabs = call fp128 @llvm.fabs.f128(fp128 %x)
+; CHECK-NEXT: %fabs = call fast fp128 @llvm.fabs.f128(fp128 %x)
 ; CHECK-NEXT: ret fp128 %fabs
 }
 
 ; =========================================================================
 ;
 ;   Test-cases for fmin / fmax
 ;
 ; =========================================================================

test/Transforms/InstCombine/inline-intrinsic-assert.ll

 ; RUN: opt < %s -inline -instcombine -S | FileCheck %s
 
 ; PR22857: http://llvm.org/bugs/show_bug.cgi?id=22857
 ; The inliner should not add an edge to an intrinsic and
 ; then assert that it did not add an edge to an intrinsic!
 
 define float @foo(float %f1) #0 {
   %call = call float @bar(float %f1)
   ret float %call
 
 ; CHECK-LABEL: @foo(
-; CHECK-NEXT: call float @llvm.fabs.f32
+; CHECK-NEXT: call fast float @llvm.fabs.f32
 ; CHECK-NEXT: ret float
 }
 
 define float @bar(float %f1) #0 {
   %call = call float @sqr(float %f1)
   %call1 = call float @sqrtf(float %call) #0
   ret float %call1
 }

test/Transforms/InstCombine/no_cgscc_assert.ll

 ; RUN: opt < %s -inline -instcombine -S | FileCheck %s
 
 ; PR21403: http://llvm.org/bugs/show_bug.cgi?id=21403
 ; When the call to sqrtf is replaced by an intrinsic call to fabs,
 ; it should not cause a problem in CGSCC. 
 
 define float @bar(float %f) #0 {
   %mul = fmul fast float %f, %f
   %call1 = call float @sqrtf(float %mul) #0
   ret float %call1
 
 ; CHECK-LABEL: @bar(
-; CHECK-NEXT: call float @llvm.fabs.f32
+; CHECK-NEXT: call fast float @llvm.fabs.f32
 ; CHECK-NEXT: ret float
 }
 
 declare float @sqrtf(float) #0
 
 attributes #0 = { readnone "unsafe-fp-math"="true" }

test/Transforms/InstCombine/pow-exp.ll

 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define double @mypow(double %x, double %y) #0 {
 entry:
   %call = call double @exp(double %x)
   %pow = call double @llvm.pow.f64(double %call, double %y)
   ret double %pow
 }
 
 ; CHECK-LABEL: define double @mypow(
 ; CHECK:   %mul = fmul fast double %x, %y
-; CHECK:   %exp = call double @exp(double %mul) #0
+; CHECK:   %exp = call fast double @exp(double %mul) #0
 ; CHECK:   ret double %exp
 ; CHECK: }
 
 declare double @exp(double) #1
 declare double @llvm.pow.f64(double, double)
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { "unsafe-fp-math"="true" }

test/Transforms/InstCombine/pow-exp2.ll

 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define double @mypow(double %x, double %y) #0 {
 entry:
   %call = call double @exp2(double %x)
   %pow = call double @llvm.pow.f64(double %call, double %y)
   ret double %pow
 }
 
 ; CHECK-LABEL: define double @mypow(
 ; CHECK:   %mul = fmul fast double %x, %y
-; CHECK:   %exp2 = call double @exp2(double %mul) #0
+; CHECK:   %exp2 = call fast double @exp2(double %mul) #0
 ; CHECK:   ret double %exp2
 ; CHECK: }
 
 declare double @exp2(double) #1
 declare double @llvm.pow.f64(double, double)
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { "unsafe-fp-math"="true" }

unittests/IR/IRBuilderTest.cpp

TEST_F(IRBuilderTest, GetIntTy) {
   unsigned IntPtrBitSize =  DL->getPointerSizeInBits(0);
   EXPECT_EQ(IntPtrTy, IntegerType::get(Ctx, IntPtrBitSize));
   delete DL;
 }
 
 TEST_F(IRBuilderTest, FastMathFlags) {
   IRBuilder<> Builder(BB);
   Value *F, *FC;
-  Instruction *FDiv, *FAdd, *FCmp;
+  Instruction *FDiv, *FAdd, *FCmp, *FCall;
 
   F = Builder.CreateLoad(GV);
   F = Builder.CreateFAdd(F, F);
 
   EXPECT_FALSE(Builder.getFastMathFlags().any());
   ASSERT_TRUE(isa<Instruction>(F));
   FAdd = cast<Instruction>(F);
   EXPECT_FALSE(FAdd->hasNoNaNs());
TEST_F(IRBuilderTest, FastMathFlags) {
   FC = Builder.CreateFCmpOEQ(F, F);
   EXPECT_TRUE(Builder.getFastMathFlags().any());
   EXPECT_TRUE(Builder.getFastMathFlags().AllowReciprocal);
   ASSERT_TRUE(isa<Instruction>(FC));
   FCmp = cast<Instruction>(FC);
   EXPECT_TRUE(FCmp->hasAllowReciprocal());
 
   Builder.clearFastMathFlags();
  
+  // Test a call with FMF.
+  auto CalleeTy = FunctionType::get(Type::getFloatTy(Ctx),
+                                    /*isVarArg=*/false);
+  auto Callee =
+      Function::Create(CalleeTy, Function::ExternalLinkage, "", M.get());
+
+  FCall = Builder.CreateCall(Callee, None);
+  EXPECT_FALSE(FCall->hasNoNaNs());
+
+  FMF.clear();
+  FMF.setNoNaNs();
+  Builder.SetFastMathFlags(FMF);
+
+  FCall = Builder.CreateCall(Callee, None);
+  EXPECT_TRUE(Builder.getFastMathFlags().any());
+  EXPECT_TRUE(Builder.getFastMathFlags().NoNaNs);
+  EXPECT_TRUE(FCall->hasNoNaNs());
+
+  Builder.clearFastMathFlags();
+
   // To test a copy, make sure that a '0' and a '1' change state. 
   F = Builder.CreateFDiv(F, F);
   ASSERT_TRUE(isa<Instruction>(F));
   FDiv = cast<Instruction>(F);
   EXPECT_FALSE(FDiv->getFastMathFlags().any());
   FDiv->setHasAllowReciprocal(true);
   FAdd->setHasAllowReciprocal(false);
   FDiv->copyFastMathFlags(FAdd);