add fast-math-flags to 'call' instructions (PR21290)

[spatel]

Summary

This patch adds optional fast-math-flags (the same that apply to fmul/fadd/fsub/fdiv/frem/fcmp) to call instructions in IR. Follow-up patches would use these flags in LibCallSimplifier, add support to clang, and extend FMF to the DAG for calls.

A motivating example is something like this:

%y = fmul fast float %x, %x
%z = tail call float @sqrtf(float %y)

We'd like to be able to optimize sqrt(x*x) into fabs(x). We do this today using a function-wide attribute for unsafe-math, but we really want to trigger on the instructions themselves:

%z = tail call fast float @sqrtf(float %y)

The code changes and tests are based on the recent commits that added "notail":
http://reviews.llvm.org/rL252368

and added FMF to fcmp:
http://reviews.llvm.org/rL241901

[majnemer]

Why is http://llvm.org/docs/doxygen/html/SimplifyLibCalls_8cpp_source.html#l00114 insufficient for this purpose?

[spatel]

In D14707#290108, @majnemer wrote:

Why is http://llvm.org/docs/doxygen/html/SimplifyLibCalls_8cpp_source.html#l00114 insufficient for this purpose?

Function-level attributes fall apart when code gets mixed by LTO; please see PR21290 for more details:
https://llvm.org/bugs/show_bug.cgi?id=21290

See also comments here:
http://reviews.llvm.org/D7257

[spatel]

In D14707#290137, @spatel wrote:

See also comments here:
http://reviews.llvm.org/D7257

Disregard that one - that was just about fcmp.

[ahatanak]

lib/AsmParser/LLParser.cpp

5647 ↗

(On Diff #40288)

Is it possible to commit the 'notail' change in a separate patch (and thank you for catching this)?

5678 ↗

(On Diff #40288)

Are there cases where you want to optimize calls to functions that take floating point arguments but don't have floating point return types?

[spatel]

lib/AsmParser/LLParser.cpp

5647 ↗

(On Diff #40288)

Sure - will do that first and update this patch.

5678 ↗

(On Diff #40288)

I thought about that too, however, if we use this definition, then we do not have to redefine "isa<FPMathOperator>" in this patch.

I don't know if that is a reasonable justification by itself, but I think we can expand the definition of FPMathOperator in a separate patch if we do decide that FMF should apply to more types of calls.

[majnemer]

My 2 ¢:

It seems more natural to me to phrase these as normal string attributes at the point of the call and construct an FMF from a CallSite but I can see arguments for and against both designs.

[spatel]

In D14707#292002, @majnemer wrote:

My 2 ¢:

It seems more natural to me to phrase these as normal string attributes at the point of the call and construct an FMF from a CallSite but I can see arguments for and against both designs.

I thought about it some more, looked into the implementation, and convinced myself that the attribute solution isn't as good.

1. We already have some of the function-level FMF attributes here: http://reviews.llvm.org/diffusion/L/browse/llvm/trunk/include/llvm/IR/Attributes.td but they are 'StrBoolAttr' rather than the 'EnumAttr' that I think we want. The names don't match. There would be some duplication/ugliness here at the least.
2. This is the big one: we'd need to update independent chunks of code any time FMF changes. This scares me because I'm still hoping to get advanced reciprocal estimate functionality ( http://reviews.llvm.org/rL239001 , http://reviews.llvm.org/rL239536 ) into instruction-level FMF. Those settings don't exist at all in IR currently, so in an LTO build, they simply disappear. I think the same thing happens with FMAs. The likelihood that FMF internals require changes seems high, so having a unified FMF interface for all instructions is the better solution.

[spatel]

Patch updated:
As suggested by Akira, I added the 'notail' comment and updated the error message in ParseCall() in r253580 since they are independent from this patch.

[spatel]

In D14707#293138, @spatel wrote:

The likelihood that FMF internals require changes seems high, so having a unified FMF interface for all instructions is the better solution.

As evidence that FMF is evolving, there's currently a patch under review that would add more flags to support observing the FENV:
http://reviews.llvm.org/D14067

[spatel]

Ping.

[spatel]

Ping * 2.

[hfinkel]

A few minor things, and with these changes, LGTM.

docs/LangRef.rst

8453 ↗

(On Diff #40686)

I find this confusing; we should just use [fast-math flags]* instead of [fmf] as we do elsewhere.

8510 ↗

(On Diff #40686)

Adjust for [fmf] -> [fast-math flags]*

8512 ↗

(On Diff #40686)

We must document here that these flags are valid only for calls with a floating-point scalar or vector return type.

lib/AsmParser/LLParser.cpp

5677 ↗

(On Diff #40686)

I think it would be better to say how the call was invalid (and, minus the fmfs, the call itself may be fine). How about:

return Error(CallLoc, "fast-math-flags specified for call without floating-point scalar or vector return type");

lib/Bitcode/Reader/BitcodeReader.cpp

5108 ↗

(On Diff #40686)

Match this error text to the text in lib/AsmParser/LLParser.cpp.

[spatel]

Patch updated based on Hal's feedback and rebased (one more existing test for SimplifyLibCalls now has the 'fast' flag).

[mehdi_amini]

I wonder about the design of this, especially since Chris (and Chandler) rejected it in the past: http://lists.llvm.org/pipermail/llvm-commits/Week-of-Mon-20121217/159446.html ?

[mehdi_amini]

(note there might be a fundamental difference I didn't see skimming through this)

llvm/trunk/docs/LangRef.rst

 '``call``' Instruction
 ^^^^^^^^^^^^^^^^^^^^^^
 
 Syntax:
 """""""
 
 ::
 
-      <result> = [tail | musttail | notail ] call [cconv] [ret attrs] <ty> [<fnty>*] <fnptrval>(<function args>) [fn attrs]
+      <result> = [tail | musttail | notail ] call [fast-math flags] [cconv] [ret attrs] <ty> [<fnty>*] <fnptrval>(<function args>) [fn attrs]
                    [ operand bundles ]
 
 Overview:
 """""""""
 
 The '``call``' instruction represents a simple function call.
 
 Arguments:
-  Option ``-tailcallopt`` is enabled, or
       ``llvm::GuaranteedTailCallOpt`` is ``true``.
    -  `Platform-specific constraints are
       met. <CodeGenerator.html#tailcallopt>`_
 
 #. The optional ``notail`` marker indicates that the optimizers should not add
    ``tail`` or ``musttail`` markers to the call. It is used to prevent tail
    call optimization from being performed on the call.
 
+#. The optional ``fast-math flags`` marker indicates that the call has one or more 
+   :ref:`fast-math flags <fastmath>`, which are optimization hints to enable
+   otherwise unsafe floating-point optimizations. Fast-math flags are only valid
+   for calls that return a floating-point scalar or vector type.
+
 #. The optional "cconv" marker indicates which :ref:`calling
    convention <callingconv>` the call should use. If none is
    specified, the call defaults to using C calling conventions. The
    calling convention of the call must match the calling convention of
    the target function, or else the behavior is undefined.
 #. The optional :ref:`Parameter Attributes <paramattrs>` list for return
    values. Only '``zeroext``', '``signext``', and '``inreg``' attributes
    are valid here.

llvm/trunk/include/llvm/Bitcode/LLVMBitCodes.h

enum { BITCODE_CURRENT_EPOCH = 0 };
   };
 
   /// Markers and flags for call instruction.
   enum CallMarkersFlags {
     CALL_TAIL = 0,
     CALL_CCONV = 1,
     CALL_MUSTTAIL = 14,
     CALL_EXPLICIT_TYPE = 15,
-    CALL_NOTAIL = 16
+    CALL_NOTAIL = 16,
+    CALL_FMF = 17  // Call has optional fast-math-flags.
   };
 
   // The function body block (FUNCTION_BLOCK_ID) describes function bodies.  It
   // can contain a constant block (CONSTANTS_BLOCK_ID).
   enum FunctionCodes {
     FUNC_CODE_DECLAREBLOCKS    =  1, // DECLAREBLOCKS: [n]
 
     FUNC_CODE_INST_BINOP       =  2, // BINOP:      [opcode, ty, opval, opval]

llvm/trunk/include/llvm/IR/IRBuilder.h

PHINode *CreatePHI(Type *Ty, unsigned NumReservedValues,
     return Insert(PHINode::Create(Ty, NumReservedValues), Name);
   }
 
   CallInst *CreateCall(Value *Callee, ArrayRef<Value *> Args = None,
                        ArrayRef<OperandBundleDef> OpBundles = None,
                        const Twine &Name = "") {
     return Insert(CallInst::Create(Callee, Args, OpBundles), Name);
   }
+
   CallInst *CreateCall(Value *Callee, ArrayRef<Value *> Args,
-                       const Twine &Name) {
-    return Insert(CallInst::Create(Callee, Args), Name);
+                       const Twine &Name, MDNode *FPMathTag = nullptr) {
+    PointerType *PTy = cast<PointerType>(Callee->getType());
+    FunctionType *FTy = cast<FunctionType>(PTy->getElementType());
+    return CreateCall(FTy, Callee, Args, Name, FPMathTag);
   }
 
   CallInst *CreateCall(llvm::FunctionType *FTy, Value *Callee,
-                       ArrayRef<Value *> Args, const Twine &Name = "") {
-    return Insert(CallInst::Create(FTy, Callee, Args), Name);
+                       ArrayRef<Value *> Args, const Twine &Name = "",
+                       MDNode *FPMathTag = nullptr) {
+    CallInst *CI = CallInst::Create(FTy, Callee, Args);
+    if (isa<FPMathOperator>(CI))
+      CI = cast<CallInst>(AddFPMathAttributes(CI, FPMathTag, FMF));
+    return Insert(CI, Name);
   }
 
   CallInst *CreateCall(Function *Callee, ArrayRef<Value *> Args,
-                       const Twine &Name = "") {
-    return CreateCall(Callee->getFunctionType(), Callee, Args, Name);
+                       const Twine &Name = "", MDNode *FPMathTag = nullptr) {
+    return CreateCall(Callee->getFunctionType(), Callee, Args, Name, FPMathTag);
   }
 
   Value *CreateSelect(Value *C, Value *True, Value *False,
                       const Twine &Name = "") {
     if (Constant *CC = dyn_cast<Constant>(C))
       if (Constant *TC = dyn_cast<Constant>(True))
         if (Constant *FC = dyn_cast<Constant>(False))
           return Insert(Folder.CreateSelect(CC, TC, FC), Name);

llvm/trunk/lib/AsmParser/LLParser.cpp

while (Lex.getKind() == lltok::kw_catch || Lex.getKind() == lltok::kw_filter){
     LP->addClause(CV);
   }
 
   Inst = LP.release();
   return false;
 }
 
 /// ParseCall
-///   ::= 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
-///   ::= 'tail' 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
-///   ::= 'musttail' 'call' OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
-///   ::= 'notail' 'call'  OptionalCallingConv OptionalAttrs Type Value
-///       ParameterList OptionalAttrs
+///   ::= 'call' OptionalFastMathFlags OptionalCallingConv
+///           OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'tail' 'call' OptionalFastMathFlags OptionalCallingConv
+///           OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'musttail' 'call' OptionalFastMathFlags OptionalCallingConv
+///           OptionalAttrs Type Value ParameterList OptionalAttrs
+///   ::= 'notail' 'call'  OptionalFastMathFlags OptionalCallingConv
+///           OptionalAttrs Type Value ParameterList OptionalAttrs
 bool LLParser::ParseCall(Instruction *&Inst, PerFunctionState &PFS,
                          CallInst::TailCallKind TCK) {
   AttrBuilder RetAttrs, FnAttrs;
   std::vector<unsigned> FwdRefAttrGrps;
   LocTy BuiltinLoc;
   unsigned CC;
   Type *RetType = nullptr;
   LocTy RetTypeLoc;
   ValID CalleeID;
   SmallVector<ParamInfo, 16> ArgList;
   SmallVector<OperandBundleDef, 2> BundleList;
   LocTy CallLoc = Lex.getLoc();
 
-  if ((TCK != CallInst::TCK_None &&
-       ParseToken(lltok::kw_call,
-                  "expected 'tail call', 'musttail call', or 'notail call'")) ||
-      ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
+  if (TCK != CallInst::TCK_None &&
+                 "expected 'tail call', 'musttail call', or 'notail call'"))
+    return true;
+
+  FastMathFlags FMF = EatFastMathFlagsIfPresent();
+
+  if (ParseOptionalCallingConv(CC) || ParseOptionalReturnAttrs(RetAttrs) ||
       ParseType(RetType, RetTypeLoc, true /*void allowed*/) ||
       ParseValID(CalleeID) ||
       ParseParameterList(ArgList, PFS, TCK == CallInst::TCK_MustTail,
                          PFS.getFunction().isVarArg()) ||
       ParseFnAttributeValuePairs(FnAttrs, FwdRefAttrGrps, false, BuiltinLoc) ||
       ParseOptionalOperandBundles(BundleList, PFS))
     return true;
 
+  if (FMF.any() && !RetType->isFPOrFPVectorTy())
+    return Error(CallLoc, "fast-math-flags specified for call without "
+                          "floating-point scalar or vector return type");
+
   // If RetType is a non-function pointer type, then this is the short syntax
   // for the call, which means that RetType is just the return type.  Infer the
   // rest of the function argument types from the arguments that are present.
   FunctionType *Ty = dyn_cast<FunctionType>(RetType);
   if (!Ty) {
     // Pull out the types of all of the arguments...
     std::vector<Type*> ParamTypes;
     for (unsigned i = 0, e = ArgList.size(); i != e; ++i)
bool LLParser::ParseCall(Instruction *&Inst, PerFunctionState &PFS,
   }
 
   // Finish off the Attribute and check them
   AttributeSet PAL = AttributeSet::get(Context, Attrs);
 
   CallInst *CI = CallInst::Create(Ty, Callee, Args, BundleList);
   CI->setTailCallKind(TCK);
   CI->setCallingConv(CC);
+  if (FMF.any())
+    CI->setFastMathFlags(FMF);
   CI->setAttributes(PAL);
   ForwardRefAttrGroups[CI] = FwdRefAttrGrps;
   Inst = CI;
   return false;
 }
 
 //===----------------------------------------------------------------------===//
 // Memory Instructions.

llvm/trunk/lib/Bitcode/Reader/BitcodeReader.cpp

case bitc::FUNC_CODE_INST_FENCE: { // FENCE:[ordering, synchscope]
           Ordering == Monotonic)
         return error("Invalid record");
       SynchronizationScope SynchScope = getDecodedSynchScope(Record[1]);
       I = new FenceInst(Context, Ordering, SynchScope);
       InstructionList.push_back(I);
       break;
     }
     case bitc::FUNC_CODE_INST_CALL: {
-      // CALL: [paramattrs, cc, fnty, fnid, arg0, arg1...]
+      // CALL: [paramattrs, cc, fmf, fnty, fnid, arg0, arg1...]
       if (Record.size() < 3)
         return error("Invalid record");
 
       unsigned OpNum = 0;
       AttributeSet PAL = getAttributes(Record[OpNum++]);
       unsigned CCInfo = Record[OpNum++];
 
+      FastMathFlags FMF;
+      if ((CCInfo >> bitc::CALL_FMF) & 1) {
+        FMF = getDecodedFastMathFlags(Record[OpNum++]);
+        if (!FMF.any())
+          return error("Fast math flags indicator set for call with no FMF");
+      }
+
       FunctionType *FTy = nullptr;
       if (CCInfo >> bitc::CALL_EXPLICIT_TYPE & 1 &&
           !(FTy = dyn_cast<FunctionType>(getTypeByID(Record[OpNum++]))))
         return error("Explicit call type is not a function type");
 
       Value *Callee;
       if (getValueTypePair(Record, OpNum, NextValueNo, Callee))
         return error("Invalid record");
case bitc::FUNC_CODE_INST_CALL: {
       if (CCInfo & 1 << bitc::CALL_TAIL)
         TCK = CallInst::TCK_Tail;
       if (CCInfo & (1 << bitc::CALL_MUSTTAIL))
         TCK = CallInst::TCK_MustTail;
       if (CCInfo & (1 << bitc::CALL_NOTAIL))
         TCK = CallInst::TCK_NoTail;
       cast<CallInst>(I)->setTailCallKind(TCK);
       cast<CallInst>(I)->setAttributes(PAL);
+      if (FMF.any()) {
+        if (!isa<FPMathOperator>(I))
+          return error("Fast-math-flags specified for call without "
+                       "floating-point scalar or vector return type");
+        I->setFastMathFlags(FMF);
+      }
       break;
     }
     case bitc::FUNC_CODE_INST_VAARG: { // VAARG: [valistty, valist, instty]
       if (Record.size() < 3)
         return error("Invalid record");
       Type *OpTy = getTypeByID(Record[0]);
       Value *Op = getValue(Record, 1, NextValueNo, OpTy);
       Type *ResTy = getTypeByID(Record[2]);

llvm/trunk/lib/Bitcode/Writer/BitcodeWriter.cpp

case Instruction::Call: {
     FunctionType *FTy = CI.getFunctionType();
 
     if (CI.hasOperandBundles())
       WriteOperandBundles(Stream, &CI, InstID, VE);
 
     Code = bitc::FUNC_CODE_INST_CALL;
 
     Vals.push_back(VE.getAttributeID(CI.getAttributes()));
+
+    unsigned Flags = GetOptimizationFlags(&I);
     Vals.push_back(CI.getCallingConv() << bitc::CALL_CCONV |
                    unsigned(CI.isTailCall()) << bitc::CALL_TAIL |
                    unsigned(CI.isMustTailCall()) << bitc::CALL_MUSTTAIL |
                    1 << bitc::CALL_EXPLICIT_TYPE |
-                   unsigned(CI.isNoTailCall()) << bitc::CALL_NOTAIL);
+                   unsigned(CI.isNoTailCall()) << bitc::CALL_NOTAIL |
+                   unsigned(Flags != 0) << bitc::CALL_FMF);
+    if (Flags != 0)
+      Vals.push_back(Flags);
+
     Vals.push_back(VE.getTypeID(FTy));
     PushValueAndType(CI.getCalledValue(), InstID, Vals, VE);  // Callee
 
     // Emit value #'s for the fixed parameters.
     for (unsigned i = 0, e = FTy->getNumParams(); i != e; ++i) {
       // Check for labels (can happen with asm labels).
       if (FTy->getParamType(i)->isLabelTy())
         Vals.push_back(VE.getValueID(CI.getArgOperand(i)));

llvm/trunk/test/Bitcode/compatibility.ll

define void @fastmathflags(float %op1, float %op2) {
   ; CHECK: %f.nsz = fadd nsz float %op1, %op2
   %f.arcp = fadd arcp float %op1, %op2
   ; CHECK: %f.arcp = fadd arcp float %op1, %op2
   %f.fast = fadd fast float %op1, %op2
   ; CHECK: %f.fast = fadd fast float %op1, %op2
   ret void
 }
 
+; Check various fast math flags and floating-point types on calls.
+
+declare float @fmf1()
+declare double @fmf2()
+declare <4 x double> @fmf3()
+
+; CHECK-LABEL: fastMathFlagsForCalls(
+define void @fastMathFlagsForCalls(float %f, double %d1, <4 x double> %d2) {
+  %call.fast = call fast float @fmf1()
+  ; CHECK: %call.fast = call fast float @fmf1()
+
+  ; Throw in some other attributes to make sure those stay in the right places.
+
+  %call.nsz.arcp = notail call nsz arcp double @fmf2()
+  ; CHECK: %call.nsz.arcp = notail call nsz arcp double @fmf2()
+
+  %call.nnan.ninf = tail call nnan ninf fastcc <4 x double> @fmf3()
+  ; CHECK: %call.nnan.ninf = tail call nnan ninf fastcc <4 x double> @fmf3()
+
+  ret void
+}
+
 ;; Type System
 %opaquety = type opaque
 define void @typesystem() {
   %p0 = bitcast i8* null to i32 (i32)*
   ; CHECK: %p0 = bitcast i8* null to i32 (i32)*
   %p1 = bitcast i8* null to void (i8*)*
   ; CHECK: %p1 = bitcast i8* null to void (i8*)*
   %p2 = bitcast i8* null to i32 (i8*, ...)*

llvm/trunk/test/Transforms/InstCombine/fast-math.ll

 declare double @llvm.sqrt.f64(double)
 
 define double @sqrt_intrinsic_arg_squared(double %x) #0 {
   %mul = fmul fast double %x, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_squared(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
 ; CHECK-NEXT: ret double %fabs
 }
 
 ; Check all 6 combinations of a 3-way multiplication tree where
 ; one factor is repeated.
 
 define double @sqrt_intrinsic_three_args1(double %x, double %y) #0 {
   %mul = fmul fast double %y, %x
   %mul2 = fmul fast double %mul, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args1(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args2(double %x, double %y) #0 {
   %mul = fmul fast double %x, %y
   %mul2 = fmul fast double %mul, %x
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args2(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args3(double %x, double %y) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %y
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args3(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args4(double %x, double %y) #0 {
   %mul = fmul fast double %y, %x
   %mul2 = fmul fast double %x, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args4(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args5(double %x, double %y) #0 {
   %mul = fmul fast double %x, %y
   %mul2 = fmul fast double %x, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args5(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_three_args6(double %x, double %y) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %y, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_three_args6(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %y)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %y)
 ; CHECK-NEXT: %1 = fmul fast double %fabs, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 define double @sqrt_intrinsic_arg_4th(double %x) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul2)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_4th(
 ; CHECK-NEXT: %mul = fmul fast double %x, %x
 ; CHECK-NEXT: ret double %mul
 }
 
 define double @sqrt_intrinsic_arg_5th(double %x) #0 {
   %mul = fmul fast double %x, %x
   %mul2 = fmul fast double %mul, %x
   %mul3 = fmul fast double %mul2, %mul
   %sqrt = call double @llvm.sqrt.f64(double %mul3)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_intrinsic_arg_5th(
 ; CHECK-NEXT: %mul = fmul fast double %x, %x
-; CHECK-NEXT: %sqrt1 = call double @llvm.sqrt.f64(double %x)
+; CHECK-NEXT: %sqrt1 = call fast double @llvm.sqrt.f64(double %x)
 ; CHECK-NEXT: %1 = fmul fast double %mul, %sqrt1
 ; CHECK-NEXT: ret double %1
 }
 
 ; Check that square root calls have the same behavior.
 
 declare float @sqrtf(float)
 declare double @sqrt(double)
 declare fp128 @sqrtl(fp128)
 
 define float @sqrt_call_squared_f32(float %x) #0 {
   %mul = fmul fast float %x, %x
   %sqrt = call float @sqrtf(float %mul)
   ret float %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f32(
-; CHECK-NEXT: %fabs = call float @llvm.fabs.f32(float %x)
+; CHECK-NEXT: %fabs = call fast float @llvm.fabs.f32(float %x)
 ; CHECK-NEXT: ret float %fabs
 }
 
 define double @sqrt_call_squared_f64(double %x) #0 {
   %mul = fmul fast double %x, %x
   %sqrt = call double @sqrt(double %mul)
   ret double %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f64(
-; CHECK-NEXT: %fabs = call double @llvm.fabs.f64(double %x)
+; CHECK-NEXT: %fabs = call fast double @llvm.fabs.f64(double %x)
 ; CHECK-NEXT: ret double %fabs
 }
 
 define fp128 @sqrt_call_squared_f128(fp128 %x) #0 {
   %mul = fmul fast fp128 %x, %x
   %sqrt = call fp128 @sqrtl(fp128 %mul)
   ret fp128 %sqrt
 
 ; CHECK-LABEL: sqrt_call_squared_f128(
-; CHECK-NEXT: %fabs = call fp128 @llvm.fabs.f128(fp128 %x)
+; CHECK-NEXT: %fabs = call fast fp128 @llvm.fabs.f128(fp128 %x)
 ; CHECK-NEXT: ret fp128 %fabs
 }
 
 ; =========================================================================
 ;
 ;   Test-cases for fmin / fmax
 ;
 ; =========================================================================

llvm/trunk/test/Transforms/InstCombine/inline-intrinsic-assert.ll

 ; RUN: opt < %s -inline -instcombine -S | FileCheck %s
 
 ; PR22857: http://llvm.org/bugs/show_bug.cgi?id=22857
 ; The inliner should not add an edge to an intrinsic and
 ; then assert that it did not add an edge to an intrinsic!
 
 define float @foo(float %f1) #0 {
   %call = call float @bar(float %f1)
   ret float %call
 
 ; CHECK-LABEL: @foo(
-; CHECK-NEXT: call float @llvm.fabs.f32
+; CHECK-NEXT: call fast float @llvm.fabs.f32
 ; CHECK-NEXT: ret float
 }
 
 define float @bar(float %f1) #0 {
   %call = call float @sqr(float %f1)
   %call1 = call float @sqrtf(float %call) #0
   ret float %call1
 }

llvm/trunk/test/Transforms/InstCombine/log-pow.ll

 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define double @mylog(double %x, double %y) #0 {
 entry:
   %pow = call double @llvm.pow.f64(double %x, double %y)
   %call = call double @log(double %pow) #0
   ret double %call
 }
 
 ; CHECK-LABEL: define double @mylog(
-; CHECK:   %log = call double @log(double %x) #0
+; CHECK:   %log = call fast double @log(double %x) #0
 ; CHECK:   %mul = fmul fast double %log, %y
 ; CHECK:   ret double %mul
 ; CHECK: }
 
 define double @test2(double ()* %fptr, double %p1) #0 {
   %call1 = call double %fptr()
   %pow = call double @log(double %call1)
   ret double %pow

llvm/trunk/test/Transforms/InstCombine/no_cgscc_assert.ll

 ; RUN: opt < %s -inline -instcombine -S | FileCheck %s
 
 ; PR21403: http://llvm.org/bugs/show_bug.cgi?id=21403
 ; When the call to sqrtf is replaced by an intrinsic call to fabs,
 ; it should not cause a problem in CGSCC. 
 
 define float @bar(float %f) #0 {
   %mul = fmul fast float %f, %f
   %call1 = call float @sqrtf(float %mul) #0
   ret float %call1
 
 ; CHECK-LABEL: @bar(
-; CHECK-NEXT: call float @llvm.fabs.f32
+; CHECK-NEXT: call fast float @llvm.fabs.f32
 ; CHECK-NEXT: ret float
 }
 
 declare float @sqrtf(float) #0
 
 attributes #0 = { readnone "unsafe-fp-math"="true" }

llvm/trunk/test/Transforms/InstCombine/pow-exp.ll

 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define double @mypow(double %x, double %y) #0 {
 entry:
   %call = call double @exp(double %x)
   %pow = call double @llvm.pow.f64(double %call, double %y)
   ret double %pow
 }
 
 ; CHECK-LABEL: define double @mypow(
 ; CHECK:   %mul = fmul fast double %x, %y
-; CHECK:   %exp = call double @exp(double %mul) #0
+; CHECK:   %exp = call fast double @exp(double %mul) #0
 ; CHECK:   ret double %exp
 ; CHECK: }
 
 define double @test2(double ()* %fptr, double %p1) #0 {
   %call1 = call double %fptr()
   %pow = call double @llvm.pow.f64(double %call1, double %p1)
   ret double %pow
 }
 
 ; CHECK-LABEL: @test2
 ; CHECK: llvm.pow.f64
 
 declare double @exp(double) #1
 declare double @llvm.pow.f64(double, double)
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { "unsafe-fp-math"="true" }

llvm/trunk/test/Transforms/InstCombine/pow-exp2.ll

 ; RUN: opt < %s -instcombine -S | FileCheck %s
 
 define double @mypow(double %x, double %y) #0 {
 entry:
   %call = call double @exp2(double %x)
   %pow = call double @llvm.pow.f64(double %call, double %y)
   ret double %pow
 }
 
 ; CHECK-LABEL: define double @mypow(
 ; CHECK:   %mul = fmul fast double %x, %y
-; CHECK:   %exp2 = call double @exp2(double %mul) #0
+; CHECK:   %exp2 = call fast double @exp2(double %mul) #0
 ; CHECK:   ret double %exp2
 ; CHECK: }
 
 declare double @exp2(double) #1
 declare double @llvm.pow.f64(double, double)
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { "unsafe-fp-math"="true" }

llvm/trunk/unittests/IR/IRBuilderTest.cpp

TEST_F(IRBuilderTest, GetIntTy) {
   unsigned IntPtrBitSize =  DL->getPointerSizeInBits(0);
   EXPECT_EQ(IntPtrTy, IntegerType::get(Ctx, IntPtrBitSize));
   delete DL;
 }
 
 TEST_F(IRBuilderTest, FastMathFlags) {
   IRBuilder<> Builder(BB);
   Value *F, *FC;
-  Instruction *FDiv, *FAdd, *FCmp;
+  Instruction *FDiv, *FAdd, *FCmp, *FCall;
 
   F = Builder.CreateLoad(GV);
   F = Builder.CreateFAdd(F, F);
 
   EXPECT_FALSE(Builder.getFastMathFlags().any());
   ASSERT_TRUE(isa<Instruction>(F));
   FAdd = cast<Instruction>(F);
   EXPECT_FALSE(FAdd->hasNoNaNs());
TEST_F(IRBuilderTest, FastMathFlags) {
   FC = Builder.CreateFCmpOEQ(F, F);
   EXPECT_TRUE(Builder.getFastMathFlags().any());
   EXPECT_TRUE(Builder.getFastMathFlags().AllowReciprocal);
   ASSERT_TRUE(isa<Instruction>(FC));
   FCmp = cast<Instruction>(FC);
   EXPECT_TRUE(FCmp->hasAllowReciprocal());
 
   Builder.clearFastMathFlags();
  
+  // Test a call with FMF.
+  auto CalleeTy = FunctionType::get(Type::getFloatTy(Ctx),
+                                    /*isVarArg=*/false);
+  auto Callee =
+      Function::Create(CalleeTy, Function::ExternalLinkage, "", M.get());
+
+  FCall = Builder.CreateCall(Callee, None);
+  EXPECT_FALSE(FCall->hasNoNaNs());
+
+  FMF.clear();
+  FMF.setNoNaNs();
+  Builder.SetFastMathFlags(FMF);
+
+  FCall = Builder.CreateCall(Callee, None);
+  EXPECT_TRUE(Builder.getFastMathFlags().any());
+  EXPECT_TRUE(Builder.getFastMathFlags().NoNaNs);
+  EXPECT_TRUE(FCall->hasNoNaNs());
+
+  Builder.clearFastMathFlags();
+
   // To test a copy, make sure that a '0' and a '1' change state. 
   F = Builder.CreateFDiv(F, F);
   ASSERT_TRUE(isa<Instruction>(F));
   FDiv = cast<Instruction>(F);
   EXPECT_FALSE(FDiv->getFastMathFlags().any());
   FDiv->setHasAllowReciprocal(true);
   FAdd->setHasAllowReciprocal(false);
   FDiv->copyFastMathFlags(FAdd);