Utilize new SDNode flag functionality to expand current support model

[mcberg2017]

Summary

This review contains binary operations and is a natural progression with a small and manageable set of changes.

[arsenm]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

686

Are we almost to the point where we could seed the fast math flags from the attributes during initial DAG creation to avoid doing this everywhere, or do too many places still not preserve them?

[mcberg2017]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

686

Many places do not preserve the flags as of yet. Even when we finish here there will still be work to be done. Also, because we propagate in these scenarios to expressions we make, that seems less likely as the context will become more divergent over time at the sub flag level.

[mcberg2017]

So if I remove FP_ROUND from this patch, this entire patch becomes extension of optimization through guard testing and is no longer in any propagation context, given that, would this patch be ok?

[mcberg2017]

I will have to update this again once D46854 is checked in.

[spatel]

In D46562#1098836, @mcberg2017 wrote:

I will have to update this again once D46854 is checked in.

That part is committed, so we should be propagating from IR almost completely now. So in the larger scheme:

1. DAG flags match IR FMF (D45710)
2. IR FMF is propagated for all FP math ops (D46854) (although there are some corner cases like libcalls where it's not done properly yet)
3. We can update transforms to use and propagate the node-level flags (this patch)

We have very few codegen tests that include IR-level FMF, so there's no way to remove the global unsafe predicate checks with confidence IMO.
My preference is to split this patch up and add tests along the way. So:

1. Remove the global 'unsafe' check from some transform.
2. Get a regression test failure.
3. Transform that test to a sibling test that has IR-level FMF, but no global/function-level unsafe-ness and commit it.
4. Update the code to use node-level flags *and* the global unsafe (show the change in the new test).

When that's done, every transform will be predicated by node level flags and globals. At that point, it's safe to remove all the global checks and tests.
If we remove the global unsafe predicate usage as we go along, we run the risk of regressing perf for multi-fold scenarios that are not covered by any tests. No idea how common that would be, but I don't see much upside to taking that shortcut.

[mcberg2017]

As per the discussion, I have split the review. The subset is for fadd,fsub,fdiv and fmul with tests created by removal of the unsafe flag and augmented for fmf.

[mcberg2017]

Mind that the addition of the remaining components, post review, will cause changes to some of these tests.

[spatel]

I don't know about the other reviewers, but I'm still overwhelmed by this patch, and I see existing bugs expanding in scope.

Can you split only the first hunk in SelectionDAG.cpp into its own patch with tests for each opcode?

case ISD::FADD:
case ISD::FSUB:
case ISD::FMUL:
case ISD::FDIV:
case ISD::FREM:
  if (getTarget().Options.UnsafeFPMath || Flags.isFast()) {

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

4806–4807

This is just wrong. Let me try to clean this up before we compound the problem.

[mcberg2017]

With code moved to D46968 and the balance here. Undef is deferred.

[mcberg2017]

In process are updates to this patch for sub flag usage, on a case by case basis.

[mcberg2017]

This working model uses fast math sub flags to guide optimization and currently passes all the lit tests. A number of lit tests were expanded to dually test the new patterns.

[mcberg2017]

Also float arithmetic getNode optimizations have been cleaned up as all the equivalent optimizations now happen in the DAG combiner.

[mcberg2017]

test/CodeGen/AMDGPU/fdiv.f16.ll

221

Note for Matt, the numbers now used reflect 16 bit fp values.

test/CodeGen/X86/fadd-combines.ll

225

I trimmed this down to the required essentials required to test both paths.

[mcberg2017]

Cleanup.

[mcberg2017]

Still looking for some discussion on these changes...

test/CodeGen/AMDGPU/ffloor.f64.ll

38

some cleanup here, this file will disappear shortly from this review (no changes)

[spatel]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

698–700

This 1st transform shows why I'm not comfortable with patches that are trying to make multiple changes at once. Which test(s) correspond to this code change?

I think the existing predicate is bogus, and the new ones are wrong too. Why does NaN/Inf make a difference here?

Is this safe other than with -0.0?

Test program to check that part:

#include <stdio.h>
  
int main() {
  float x,y;
  x = -0.0;
  y = -0.0;
  printf("-(%f + %f) = %f\n", x, y, -(x + y));
  printf("(-%f) - %f = %f\n", x, y, (-x) - y);
  x = +0.0;
  y = -0.0;
  printf("-(%f + %f) = %f\n", x, y, -(x + y));
  printf("(-%f) - %f = %f\n", x, y, (-x) - y);
  x = -0.0;
  y = +0.0;
  printf("-(%f + %f) = %f\n", x, y, -(x + y));
  printf("(-%f) - %f = %f\n", x, y, (-x) - y);
  x = +0.0;
  y = +0.0;
  printf("-(%f + %f) = %f\n", x, y, -(x + y));
  printf("(-%f) - %f = %f\n", x, y, (-x) - y);
  return 0;
}

[mcberg2017]

This review is now covered in other checkins.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

static char isNegatibleForFree(SDValue Op, bool LegalOperations,
                                const TargetLowering &TLI,
                                const TargetOptions *Options,
                                unsigned Depth = 0) {
   // fneg is removable even if it has multiple uses.
   if (Op.getOpcode() == ISD::FNEG) return 2;
 
   // Don't allow anything with multiple uses unless we know it is free.
   EVT VT = Op.getValueType();
+  const SDNodeFlags Flags = Op->getFlags();
   if (!Op.hasOneUse())
     if (!(Op.getOpcode() == ISD::FP_EXTEND &&
           TLI.isFPExtFree(VT, Op.getOperand(0).getValueType())))
       return 0;
 
   // Don't recurse exponentially.
   if (Depth > 6) return 0;
 
   switch (Op.getOpcode()) {
   default: return false;
   case ISD::ConstantFP: {
     if (!LegalOperations)
       return 1;
 
     // Don't invert constant FP values after legalization unless the target says
     // the negated constant is legal.
     return TLI.isOperationLegal(ISD::ConstantFP, VT) ||
       TLI.isFPImmLegal(neg(cast<ConstantFPSDNode>(Op)->getValueAPF()), VT);
   }
   case ISD::FADD:
-    // FIXME: determine better conditions for this xform.
-    if (!Options->UnsafeFPMath) return 0;
+    if (!Options->UnsafeFPMath && 
+        !Flags.hasNoNaNs() && !Flags.hasNoInfs()) return 0;
 
     // After operation legalization, it might not be legal to create new FSUBs.
     if (LegalOperations && !TLI.isOperationLegalOrCustom(ISD::FSUB, VT))
       return 0;
 
     // fold (fneg (fadd A, B)) -> (fsub (fneg A), B)
     if (char V = isNegatibleForFree(Op.getOperand(0), LegalOperations, TLI,
                                     Options, Depth + 1))
       return V;
+
     // fold (fneg (fadd A, B)) -> (fsub (fneg B), A)
     return isNegatibleForFree(Op.getOperand(1), LegalOperations, TLI, Options,
                               Depth + 1);
   case ISD::FSUB:
     // We can't turn -(A-B) into B-A when we honor signed zeros.
     if (!Options->NoSignedZerosFPMath &&
-        !Op.getNode()->getFlags().hasNoSignedZeros())
+        !Flags.hasNoSignedZeros())
       return 0;
 
     // fold (fneg (fsub A, B)) -> (fsub B, A)
     return 1;
 
   case ISD::FMUL:
   case ISD::FDIV:
-    if (Options->HonorSignDependentRoundingFPMath()) return 0;
+    if (Options->HonorSignDependentRoundingFPMathOption && 
+        !Options->UnsafeFPMath && !Flags.hasNoNaNs()) return 0;
 
     // fold (fneg (fmul X, Y)) -> (fmul (fneg X), Y) or (fmul X, (fneg Y))
     if (char V = isNegatibleForFree(Op.getOperand(0), LegalOperations, TLI,
                                     Options, Depth + 1))
       return V;
 
     return isNegatibleForFree(Op.getOperand(1), LegalOperations, TLI, Options,
                               Depth + 1);
static SDValue GetNegatedExpression(SDValue Op, SelectionDAG &DAG,
   switch (Op.getOpcode()) {
   default: llvm_unreachable("Unknown code");
   case ISD::ConstantFP: {
     APFloat V = cast<ConstantFPSDNode>(Op)->getValueAPF();
     V.changeSign();
     return DAG.getConstantFP(V, SDLoc(Op), Op.getValueType());
   }
   case ISD::FADD:
-    // FIXME: determine better conditions for this xform.
-    assert(Options.UnsafeFPMath);
+    assert(Options.UnsafeFPMath || 
+           (Flags.hasNoNaNs() && Flags.hasNoInfs()));
 
     // fold (fneg (fadd A, B)) -> (fsub (fneg A), B)
     if (isNegatibleForFree(Op.getOperand(0), LegalOperations,
                            DAG.getTargetLoweringInfo(), &Options, Depth+1))
       return DAG.getNode(ISD::FSUB, SDLoc(Op), Op.getValueType(),
                          GetNegatedExpression(Op.getOperand(0), DAG,
                                               LegalOperations, Depth+1),
                          Op.getOperand(1), Flags);
if (ConstantFPSDNode *N0CFP = dyn_cast<ConstantFPSDNode>(Op.getOperand(0)))
         return Op.getOperand(1);
 
     // fold (fneg (fsub A, B)) -> (fsub B, A)
     return DAG.getNode(ISD::FSUB, SDLoc(Op), Op.getValueType(),
                        Op.getOperand(1), Op.getOperand(0), Flags);
 
   case ISD::FMUL:
   case ISD::FDIV:
-    assert(!Options.HonorSignDependentRoundingFPMath());
+    assert(!Options.HonorSignDependentRoundingFPMath() || Flags.hasNoNaNs());
 
     // fold (fneg (fmul X, Y)) -> (fmul (fneg X), Y)
     if (isNegatibleForFree(Op.getOperand(0), LegalOperations,
                            DAG.getTargetLoweringInfo(), &Options, Depth+1))
       return DAG.getNode(Op.getOpcode(), SDLoc(Op), Op.getValueType(),
                          GetNegatedExpression(Op.getOperand(0), DAG,
                                               LegalOperations, Depth+1),
                          Op.getOperand(1), Flags);
SDValue DAGCombiner::visitFADD(SDNode *N) {
   if ((isFMulNegTwo(N0) && N0.hasOneUse()) ||
       (isFMulNegTwo(N1) && N1.hasOneUse())) {
     bool N1IsFMul = isFMulNegTwo(N1);
     SDValue AddOp = N1IsFMul ? N1.getOperand(0) : N0.getOperand(0);
     SDValue Add = DAG.getNode(ISD::FADD, DL, VT, AddOp, AddOp, Flags);
     return DAG.getNode(ISD::FSUB, DL, VT, N1IsFMul ? N0 : N1, Add, Flags);
   }
 
-  // FIXME: Auto-upgrade the target/function-level option.
-  if (Options.NoSignedZerosFPMath || N->getFlags().hasNoSignedZeros()) {
+  ConstantFPSDNode *N1C = isConstOrConstSplatFP(N1);
+  if (N1C && N1C->isZero()) {
+    if (N1C->isNegative() || Options.UnsafeFPMath ||
+        Flags.hasNoSignedZeros()) {
       // fold (fadd A, 0) -> A
-    if (ConstantFPSDNode *N1C = isConstOrConstSplatFP(N1))
-      if (N1C->isZero())
-        return N0;
     }
+  }
 
   // If 'unsafe math' is enabled, fold lots of things.
-  if (Options.UnsafeFPMath) {
+  bool AllowNewConst = (Level < AfterLegalizeDAG);
+  if (Options.UnsafeFPMath || Flags.hasNoNaNs()) {
+    // If allowed, fold (fadd (fneg x), x) -> 0.0
+    if (AllowNewConst && N0.getOpcode() == ISD::FNEG && N0.getOperand(0) == N1)
+      return DAG.getConstantFP(0.0, DL, VT);
+
+    // If allowed, fold (fadd x, (fneg x)) -> 0.0
+    if (AllowNewConst && N1.getOpcode() == ISD::FNEG && N1.getOperand(0) == N0)
+      return DAG.getConstantFP(0.0, DL, VT);
+  }
+
+  // If 'unsafe math' is enabled, fold lots of things.
+  if (Options.UnsafeFPMath || Flags.hasAllowReassociation()) {
     // No FP constant should be created after legalization as Instruction
     // Selection pass has a hard time dealing with FP constants.
-    bool AllowNewConst = (Level < AfterLegalizeDAG);
 
     // fold (fadd (fadd x, c1), c2) -> (fadd x, (fadd c1, c2))
     if (N1CFP && N0.getOpcode() == ISD::FADD && N0.getNode()->hasOneUse() &&
         isConstantFPBuildVectorOrConstantFP(N0.getOperand(1)))
       return DAG.getNode(ISD::FADD, DL, VT, N0.getOperand(0),
                          DAG.getNode(ISD::FADD, DL, VT, N0.getOperand(1), N1,
                                      Flags),
                          Flags);
 
-    // If allowed, fold (fadd (fneg x), x) -> 0.0
-    if (AllowNewConst && N0.getOpcode() == ISD::FNEG && N0.getOperand(0) == N1)
-      return DAG.getConstantFP(0.0, DL, VT);
-
-    // If allowed, fold (fadd x, (fneg x)) -> 0.0
-    if (AllowNewConst && N1.getOpcode() == ISD::FNEG && N1.getOperand(0) == N0)
-      return DAG.getConstantFP(0.0, DL, VT);
-
     // We can fold chains of FADD's of the same value into multiplications.
     // This transform is not safe in general because we are reducing the number
     // of rounding steps.
     if (TLI.isOperationLegalOrCustom(ISD::FMUL, VT) && !N0CFP && !N1CFP) {
       if (N0.getOpcode() == ISD::FMUL) {
         bool CFP00 = isConstantFPBuildVectorOrConstantFP(N0.getOperand(0));
         bool CFP01 = isConstantFPBuildVectorOrConstantFP(N0.getOperand(1));
 
if (N0CFP && N0CFP->isZero()) {
         return GetNegatedExpression(N1, DAG, LegalOperations);
       if (!LegalOperations || TLI.isOperationLegal(ISD::FNEG, VT))
         return DAG.getNode(ISD::FNEG, DL, VT, N1, Flags);
     }
   }
 
   // If 'unsafe math' is enabled, fold lots of things.
   if (Options.UnsafeFPMath) {
-    // (fsub A, 0) -> A
-    if (N1CFP && N1CFP->isZero())
-      return N0;
-
     // (fsub x, x) -> 0.0
     if (N0 == N1)
       return DAG.getConstantFP(0.0f, DL, VT);
 
     // (fsub x, (fadd x, y)) -> (fneg y)
     // (fsub x, (fadd y, x)) -> (fneg y)
     if (N1.getOpcode() == ISD::FADD) {
       SDValue N10 = N1->getOperand(0);
       SDValue N11 = N1->getOperand(1);
 
       if (N10 == N0 && isNegatibleForFree(N11, LegalOperations, TLI, &Options))
         return GetNegatedExpression(N11, DAG, LegalOperations);
 
       if (N11 == N0 && isNegatibleForFree(N10, LegalOperations, TLI, &Options))
         return GetNegatedExpression(N10, DAG, LegalOperations);
     }
+  } else {
+    // (fsub x, x) -> 0.0
+    if (N0 == N1 && Flags.hasNoNaNs()) {
+      return DAG.getConstantFP(0.0f, DL, VT);
+    }
+  }
+
+  // (fsub A, 0) -> A
+  if (N1CFP && N1CFP->isZero()) {
+    if (!N1CFP->isNegative() || Options.UnsafeFPMath ||
+        Flags.hasNoSignedZeros()) {
+      return N0;
+    }
   }
 
   // FSUB -> FMA combines:
   if (SDValue Fused = visitFSUBForFMACombine(N)) {
     AddToWorklist(Fused.getNode());
     return Fused;
   }
 
SDValue DAGCombiner::visitFMUL(SDNode *N) {
 
   // fold (fmul A, 1.0) -> A
   if (N1CFP && N1CFP->isExactlyValue(1.0))
     return N0;
 
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
-  if (Options.UnsafeFPMath) {
+  if (Options.UnsafeFPMath || 
+      (Flags.hasNoNaNs() && Flags.hasNoSignedZeros())) {
     // fold (fmul A, 0) -> 0
     if (N1CFP && N1CFP->isZero())
       return N1;
+  } 
 
+  if (Options.UnsafeFPMath || Flags.hasAllowReassociation()) {
     // fold (fmul (fmul x, c1), c2) -> (fmul x, (fmul c1, c2))
     if (N0.getOpcode() == ISD::FMUL) {
       // Fold scalars or any vector constants (not just splats).
       // This fold is done in general by InstCombine, but extra fmul insts
       // may have been generated during lowering.
       SDValue N00 = N0.getOperand(0);
       SDValue N01 = N0.getOperand(1);
       auto *BV1 = dyn_cast<BuildVectorSDNode>(N1);
SDValue DAGCombiner::visitFDIV(SDNode *N) {
 
   // fold (fdiv c1, c2) -> c1/c2
   if (N0CFP && N1CFP)
     return DAG.getNode(ISD::FDIV, SDLoc(N), VT, N0, N1, Flags);
 
   if (SDValue NewSel = foldBinOpIntoSelect(N))
     return NewSel;
 
-  if (Options.UnsafeFPMath) {
+  if (Options.UnsafeFPMath || Flags.hasAllowReciprocal()) {
     // fold (fdiv X, c2) -> fmul X, 1/c2 if losing precision is acceptable.
     if (N1CFP) {
       // Compute the reciprocal 1.0 / c2.
       const APFloat &N1APF = N1CFP->getValueAPF();
       APFloat Recip(N1APF.getSemantics(), 1); // 1.0
       APFloat::opStatus st = Recip.divide(N1APF, APFloat::rmNearestTiesToEven);
       // Only do the transform if the reciprocal is a legal fp immediate that
       // isn't too nasty (eg NaN, denormal, ...).
SDValue DAGCombiner::visitFP_TO_UINT(SDNode *N) {
   return FoldIntToFPToInt(N, DAG);
 }
 
 SDValue DAGCombiner::visitFP_ROUND(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   SDValue N1 = N->getOperand(1);
   ConstantFPSDNode *N0CFP = dyn_cast<ConstantFPSDNode>(N0);
   EVT VT = N->getValueType(0);
+  SDNodeFlags Flags = N->getFlags();
 
   // fold (fp_round c1fp) -> c1fp
   if (N0CFP)
     return DAG.getNode(ISD::FP_ROUND, SDLoc(N), VT, N0, N1);
 
   // fold (fp_round (fp_extend x)) -> x
   if (N0.getOpcode() == ISD::FP_EXTEND && VT == N0.getOperand(0).getValueType())
     return N0.getOperand(0);
if (N0.getOpcode() == ISD::FP_ROUND) {
     if (N0.getOperand(0).getValueType() == MVT::f80 && VT == MVT::f16)
       return SDValue();
 
     // If the first fp_round isn't a value preserving truncation, it might
     // introduce a tie in the second fp_round, that wouldn't occur in the
     // single-step fp_round we want to fold to.
     // In other words, double rounding isn't the same as rounding.
     // Also, this is a value preserving truncation iff both fp_round's are.
-    if (DAG.getTarget().Options.UnsafeFPMath || N0IsTrunc) {
+    if (DAG.getTarget().Options.UnsafeFPMath || 
+        Flags.hasAllowReassociation() || N0IsTrunc) {
       SDLoc DL(N);
       return DAG.getNode(ISD::FP_ROUND, DL, VT, N0.getOperand(0),
                          DAG.getIntPtrConstant(NIsTrunc && N0IsTrunc, DL));
     }
   }
 
   // fold (fp_round (copysign X, Y)) -> (copysign (fp_round X), Y)
   if (N0.getOpcode() == ISD::FCOPYSIGN && N0.getNode()->hasOneUse()) {

lib/CodeGen/SelectionDAG/SelectionDAG.cpp

case ISD::UMAX:
     assert(N1.getValueType() == N2.getValueType() &&
            N1.getValueType() == VT && "Binary operator types must match!");
     break;
   case ISD::FADD:
   case ISD::FSUB:
   case ISD::FMUL:
   case ISD::FDIV:
   case ISD::FREM:
-    if (getTarget().Options.UnsafeFPMath) {
-      if (Opcode == ISD::FADD) {
-        // x+0 --> x
-        if (N2CFP && N2CFP->getValueAPF().isZero())
-          return N1;
-      } else if (Opcode == ISD::FSUB) {
-        // x-0 --> x
-        if (N2CFP && N2CFP->getValueAPF().isZero())
-          return N1;
-      } else if (Opcode == ISD::FMUL) {
-        // x*0 --> 0
-        if (N2CFP && N2CFP->isZero())
-          return N2;
-        // x*1 --> x
-        if (N2CFP && N2CFP->isExactlyValue(1.0))
-          return N1;
-      }
-    }
     assert(VT.isFloatingPoint() && "This operator only applies to FP types!");
     assert(N1.getValueType() == N2.getValueType() &&
            N1.getValueType() == VT && "Binary operator types must match!");
     break;
   case ISD::FCOPYSIGN:   // N1 and result must match.  N1/N2 need not match.
     assert(N1.getValueType() == VT &&
            N1.getValueType().isFloatingPoint() &&
            N2.getValueType().isFloatingPoint() &&
if (N2.isUndef()) {
     case ISD::SUB:
     case ISD::UDIV:
     case ISD::SDIV:
     case ISD::UREM:
     case ISD::SREM:
     case ISD::SRA:
     case ISD::SRL:
     case ISD::SHL:
       return getUNDEF(VT);       // fold op(arg1, undef) -> undef
     case ISD::MUL:
     case ISD::AND:
       return getConstant(0, DL, VT);  // fold op(arg1, undef) -> 0
     case ISD::OR:
       return getAllOnesConstant(DL, VT);
     }
   }
 
   // Memoize this node if possible.

test/CodeGen/AArch64/fdiv-combine.ll

-; RUN: llc -mtriple=aarch64-unknown-unknown < %s | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-unknown-unknown                        | FileCheck %s
+; RUN: llc < %s -mtriple=aarch64-unknown-unknown -enable-unsafe-fp-math | FileCheck %s
 
 ; Following test cases check:
 ;   a / D; b / D; c / D;
 ;                =>
 ;   recip = 1.0 / D; a * recip; b * recip; c * recip;
 define void @three_fdiv_float(float %D, float %a, float %b, float %c) #0 {
 ; CHECK-LABEL: three_fdiv_float:
 ; CHECK: fdiv s
 ; CHECK-NOT: fdiv
 ; CHECK: fmul
 ; CHECK: fmul
 ; CHECK: fmul
-  %div = fdiv float %a, %D
-  %div1 = fdiv float %b, %D
-  %div2 = fdiv float %c, %D
+  %div = fdiv fast float %a, %D
+  %div1 = fdiv fast float %b, %D
+  %div2 = fdiv fast float %c, %D
   tail call void @foo_3f(float %div, float %div1, float %div2)
   ret void
 }
 
 define void @three_fdiv_double(double %D, double %a, double %b, double %c) #0 {
 ; CHECK-LABEL: three_fdiv_double:
 ; CHECK: fdiv d
 ; CHECK-NOT: fdiv
 ; CHECK: fmul
 ; CHECK: fmul
 ; CHECK: fmul
-  %div = fdiv double %a, %D
-  %div1 = fdiv double %b, %D
-  %div2 = fdiv double %c, %D
+  %div = fdiv fast double %a, %D
+  %div1 = fdiv fast double %b, %D
+  %div2 = fdiv fast double %c, %D
   tail call void @foo_3d(double %div, double %div1, double %div2)
   ret void
 }
 
 define void @three_fdiv_4xfloat(<4 x float> %D, <4 x float> %a, <4 x float> %b, <4 x float> %c) #0 {
 ; CHECK-LABEL: three_fdiv_4xfloat:
 ; CHECK: fdiv v
 ; CHECK-NOT: fdiv
 ; CHECK: fmul
 ; CHECK: fmul
 ; CHECK: fmul
-  %div = fdiv <4 x float> %a, %D
-  %div1 = fdiv <4 x float> %b, %D
-  %div2 = fdiv <4 x float> %c, %D
+  %div = fdiv fast <4 x float> %a, %D
+  %div1 = fdiv fast <4 x float> %b, %D
+  %div2 = fdiv fast <4 x float> %c, %D
   tail call void @foo_3_4xf(<4 x float> %div, <4 x float> %div1, <4 x float> %div2)
   ret void
 }
 
 define void @three_fdiv_2xdouble(<2 x double> %D, <2 x double> %a, <2 x double> %b, <2 x double> %c) #0 {
 ; CHECK-LABEL: three_fdiv_2xdouble:
 ; CHECK: fdiv v
 ; CHECK-NOT: fdiv
 ; CHECK: fmul
 ; CHECK: fmul
 ; CHECK: fmul
-  %div = fdiv <2 x double> %a, %D
-  %div1 = fdiv <2 x double> %b, %D
-  %div2 = fdiv <2 x double> %c, %D
+  %div = fdiv fast <2 x double> %a, %D
+  %div1 = fdiv fast <2 x double> %b, %D
+  %div2 = fdiv fast <2 x double> %c, %D
   tail call void @foo_3_2xd(<2 x double> %div, <2 x double> %div1, <2 x double> %div2)
   ret void
 }
 
 ; Following test cases check we never combine two FDIVs if neither of them
 ; calculates a reciprocal.
 define void @two_fdiv_float(float %D, float %a, float %b) #0 {
 ; CHECK-LABEL: two_fdiv_float:
 ; CHECK: fdiv s
 ; CHECK: fdiv s
 ; CHECK-NOT: fmul
-  %div = fdiv float %a, %D
-  %div1 = fdiv float %b, %D
+  %div = fdiv fast float %a, %D
+  %div1 = fdiv fast float %b, %D
   tail call void @foo_2f(float %div, float %div1)
   ret void
 }
 
 define void @two_fdiv_double(double %D, double %a, double %b) #0 {
 ; CHECK-LABEL: two_fdiv_double:
 ; CHECK: fdiv d
 ; CHECK: fdiv d
 ; CHECK-NOT: fmul
-  %div = fdiv double %a, %D
-  %div1 = fdiv double %b, %D
+  %div = fdiv fast double %a, %D
+  %div1 = fdiv fast double %b, %D
   tail call void @foo_2d(double %div, double %div1)
   ret void
 }
 
 declare void @foo_3f(float, float, float)
 declare void @foo_3d(double, double, double)
 declare void @foo_3_4xf(<4 x float>, <4 x float>, <4 x float>)
 declare void @foo_3_2xd(<2 x double>, <2 x double>, <2 x double>)
 declare void @foo_2f(float, float)
 declare void @foo_2d(double, double)
-
-attributes #0 = { "unsafe-fp-math"="true" }

test/CodeGen/AMDGPU/fadd.ll

define amdgpu_kernel void @fadd_v8f32(<8 x float> addrspace(1)* %out, <8 x float> %a, <8 x float> %b) #0 {
   %add = fadd <8 x float> %a, %b
   store <8 x float> %add, <8 x float> addrspace(1)* %out, align 32
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fadd_0_nsz_attr_f32:
 ; SI-NOT: v_add_f32
 define amdgpu_kernel void @fadd_0_nsz_attr_f32(float addrspace(1)* %out, float %a) #1 {
-   %add = fadd float %a, 0.0
+   %add = fadd nsz float %a, 0.0
    store float %add, float addrspace(1)* %out, align 4
    ret void
 }
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind "no-signed-zeros-fp-math"="true" }

test/CodeGen/AMDGPU/fdiv.f16.ll

 define amdgpu_kernel void @div_arcp_2_x_pat_f16(half addrspace(1)* %out) #0 {
   %x = load half, half addrspace(1)* undef
   %rcp = fdiv arcp half %x, 2.0
   store half %rcp, half addrspace(1)* %out, align 4
   ret void
 }
 
 ; FUNC-LABEL: {{^}}div_arcp_k_x_pat_f16:
-; SI: v_mul_f32_e32 v{{[0-9]+}}, 0x3dcccccd, v{{[0-9]+}}
+; SI: v_mul_f32_e32 v{{[0-9]+}}, 0x3dccc000, v{{[0-9]+}}
 
 ; GFX8_9: v_mul_f16_e32 [[MUL:v[0-9]+]], 0x2e66, v{{[0-9]+}}
 ; GFX8_9: buffer_store_short [[MUL]]
 define amdgpu_kernel void @div_arcp_k_x_pat_f16(half addrspace(1)* %out) #0 {
   %x = load half, half addrspace(1)* undef
   %rcp = fdiv arcp half %x, 10.0
   store half %rcp, half addrspace(1)* %out, align 4
   ret void
 }
 
 ; FUNC-LABEL: {{^}}div_arcp_neg_k_x_pat_f16:
-; SI: v_mul_f32_e32 v{{[0-9]+}}, 0xbdcccccd, v{{[0-9]+}}
+; SI: v_mul_f32_e32 v{{[0-9]+}}, 0xbdccc000, v{{[0-9]+}}
 
 ; GFX8_9: v_mul_f16_e32 [[MUL:v[0-9]+]], 0xae66, v{{[0-9]+}}
 ; GFX8_9: buffer_store_short [[MUL]]
 define amdgpu_kernel void @div_arcp_neg_k_x_pat_f16(half addrspace(1)* %out) #0 {
   %x = load half, half addrspace(1)* undef
   %rcp = fdiv arcp half %x, -10.0
   store half %rcp, half addrspace(1)* %out, align 4
   ret void

test/CodeGen/AMDGPU/ffloor.f64.ll

 ; SI: v_fract_f64_e64 {{v[[0-9]+:[0-9]+]}}, -[[INPUT:s[[0-9]+:[0-9]+]]]
 ; SI-DAG: v_min_f64
 ; SI-DAG: v_cmp_class_f64_e64 vcc
 ; SI: v_cndmask_b32_e32
 ; SI: v_cndmask_b32_e32
 ; SI: v_add_f64 {{v[[0-9]+:[0-9]+]}}, -[[INPUT]]
 ; SI: s_endpgm
 define amdgpu_kernel void @ffloor_f64_neg(double addrspace(1)* %out, double %x) {
-  %neg = fsub double 0.0, %x
+  %neg = fsub nsz double 0.0, %x
   %y = call double @llvm.floor.f64(double %neg) nounwind readnone
   store double %y, double addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}ffloor_f64_neg_abs:
 ; CI: v_floor_f64_e64
 ; SI: v_fract_f64_e64 {{v[[0-9]+:[0-9]+]}}, -|[[INPUT:s[[0-9]+:[0-9]+]]]|
 ; SI-DAG: v_min_f64
 ; SI-DAG: v_cmp_class_f64_e64 vcc
 ; SI: v_cndmask_b32_e32
 ; SI: v_cndmask_b32_e32
 ; SI: v_add_f64 {{v[[0-9]+:[0-9]+]}}, -|[[INPUT]]|
 ; SI: s_endpgm
 define amdgpu_kernel void @ffloor_f64_neg_abs(double addrspace(1)* %out, double %x) {
   %abs = call double @llvm.fabs.f64(double %x)
-  %neg = fsub double 0.0, %abs
+  %neg = fsub nsz double 0.0, %abs
   %y = call double @llvm.floor.f64(double %neg) nounwind readnone
   store double %y, double addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}ffloor_v2f64:
 ; CI: v_floor_f64_e32
 ; CI: v_floor_f64_e32

test/CodeGen/PowerPC/fdiv-combine.ll

-; RUN: llc -verify-machineinstrs -mcpu=ppc64 < %s | FileCheck %s
+; RUN: llc -verify-machineinstrs -mcpu=ppc64 < %s                        | FileCheck %s
+; RUN: llc -verify-machineinstrs -mcpu=ppc64 -enable-unsafe-fp-math < %s | FileCheck %s
 target datalayout = "E-m:e-i64:64-n32:64"
 target triple = "powerpc64-unknown-linux-gnu"
 
 ; Following test case checks:
 ;   a / D; b / D; c / D;
 ;                =>
 ;   recip = 1.0 / D; a * recip; b * recip; c * recip;
 
 define void @three_fdiv_double(double %D, double %a, double %b, double %c) #0 {
 ; CHECK-LABEL: three_fdiv_double:
 ; CHECK: fdiv {{[0-9]}}
 ; CHECK-NOT: fdiv
 ; CHECK: fmul
 ; CHECK: fmul
 ; CHECK: fmul
-  %div = fdiv double %a, %D
-  %div1 = fdiv double %b, %D
-  %div2 = fdiv double %c, %D
+  %div = fdiv fast double %a, %D
+  %div1 = fdiv fast double %b, %D
+  %div2 = fdiv fast double %c, %D
   tail call void @foo_3d(double %div, double %div1, double %div2)
   ret void
 }
 
 define void @two_fdiv_double(double %D, double %a, double %b) #0 {
 ; CHECK-LABEL: two_fdiv_double:
 ; CHECK: fdiv {{[0-9]}}
 ; CHECK: fdiv {{[0-9]}}
 ; CHECK-NOT: fmul
-  %div = fdiv double %a, %D
-  %div1 = fdiv double %b, %D
+  %div = fdiv fast double %a, %D
+  %div1 = fdiv fast double %b, %D
   tail call void @foo_2d(double %div, double %div1)
   ret void
 }
 
 declare void @foo_3d(double, double, double)
 declare void @foo_3_2xd(<2 x double>, <2 x double>, <2 x double>)
 declare void @foo_2d(double, double)
-
-attributes #0 = { "unsafe-fp-math"="true" }

test/CodeGen/PowerPC/fmf-math.ll

+; RUN: llc -verify-machineinstrs < %s -mattr=-vsx -mtriple=ppc32-- | grep fmul | count 1
+
+define double @foo(double %X) nounwind {
+        %tmp1 = fmul fast double %X, 1.23
+        %tmp2 = fmul fast double %tmp1, 4.124
+        ret double %tmp2
+}
+

test/CodeGen/X86/change-ir-fp-math.ll

+; Check that we can enable/disable fast IR flag attributes.  
+
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown \
+; RUN:   | FileCheck %s --check-prefix=CHECK
+
+; The div in these functions should be converted to a mul when unsafe-fp-math
+; is enabled.
+
+; CHECK-LABEL: fast_fp_math:
+define double @fast_fp_math(double %x) {
+; CHECK:    mulsd
+  %div = fdiv fast double %x, 2.0
+  ret double %div
+}
+
+; CHECK-LABEL: noflags_fp_math:
+define double @noflags_fp_math(double %x) {
+; CHECK:      divsd
+  %div = fdiv double %x, 2.0
+  ret double %div
+}
+

test/CodeGen/X86/fadd-combines.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=x86_64-unknown-unknown < %s | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown < %s                        | FileCheck %s
+; RUN: llc -mtriple=x86_64-unknown-unknown < %s -enable-unsafe-fp-math | FileCheck %s
 
 define float @fadd_zero_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_zero_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, 0.0
+  %y = fadd fast float %x, 0.0
   ret float %y
 }
 
 define <4 x float> @fadd_zero_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_zero_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, zeroinitializer
+  %y = fadd fast <4 x float> %x, zeroinitializer
   ret <4 x float> %y
 }
 
 ; CHECK: float 3
 define float @fadd_2const_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_2const_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, 1.0
-  %z = fadd float %y, 2.0
+  %y = fadd fast float %x, 1.0
+  %z = fadd fast float %y, 2.0
   ret float %z
 }
 
 ; CHECK: float 5
 ; CHECK: float 5
 ; CHECK: float 5
 ; CHECK: float 5
 define <4 x float> @fadd_2const_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_2const_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    addps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
-  %z = fadd <4 x float> %y, <float 4.0, float 3.0, float 2.0, float 1.0>
+  %y = fadd fast <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
+  %z = fadd fast <4 x float> %y, <float 4.0, float 3.0, float 2.0, float 1.0>
   ret <4 x float> %z
 }
 
 ; CHECK: float 3
 define float @fadd_x_fmul_x_c_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_x_fmul_x_c_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fmul float %x, 2.0
-  %z = fadd float %x, %y
+  %y = fmul fast float %x, 2.0
+  %z = fadd fast float %x, %y
   ret float %z
 }
 
 ; CHECK: float 2
 ; CHECK: float 3
 ; CHECK: float 4
 ; CHECK: float 5
 define <4 x float> @fadd_x_fmul_x_c_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_x_fmul_x_c_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fmul <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
-  %z = fadd <4 x float> %x, %y
+  %y = fmul fast <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
+  %z = fadd fast <4 x float> %x, %y
   ret <4 x float> %z
 }
 
 ; CHECK: float 3
 define float @fadd_fmul_x_c_x_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_fmul_x_c_x_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fmul float %x, 2.0
-  %z = fadd float %y, %x
+  %y = fmul fast float %x, 2.0
+  %z = fadd fast float %y, %x
   ret float %z
 }
 
 ; CHECK: float 2
 ; CHECK: float 3
 ; CHECK: float 4
 ; CHECK: float 5
 define <4 x float> @fadd_fmul_x_c_x_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_fmul_x_c_x_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fmul <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
-  %z = fadd <4 x float> %y, %x
+  %y = fmul fast <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
+  %z = fadd fast <4 x float> %y, %x
   ret <4 x float> %z
 }
 
 ; CHECK: float 4
 define float @fadd_fadd_x_x_fmul_x_c_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_fmul_x_c_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, %x
-  %z = fmul float %x, 2.0
-  %w = fadd float %y, %z
+  %y = fadd fast float %x, %x
+  %z = fmul fast float %x, 2.0
+  %w = fadd fast float %y, %z
   ret float %w
 }
 
 ; CHECK: float 3
 ; CHECK: float 4
 ; CHECK: float 5
 ; CHECK: float 6
 define <4 x float> @fadd_fadd_x_x_fmul_x_c_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_fmul_x_c_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, %x
-  %z = fmul <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
-  %w = fadd <4 x float> %y, %z
+  %y = fadd fast <4 x float> %x, %x
+  %z = fmul fast <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
+  %w = fadd fast <4 x float> %y, %z
   ret <4 x float> %w
 }
 
 ; CHECK: float 4
 define float @fadd_fmul_x_c_fadd_x_x_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_fmul_x_c_fadd_x_x_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, %x
-  %z = fmul float %x, 2.0
-  %w = fadd float %z, %y
+  %y = fadd fast float %x, %x
+  %z = fmul fast float %x, 2.0
+  %w = fadd fast float %z, %y
   ret float %w
 }
 
 ; CHECK: float 3
 ; CHECK: float 4
 ; CHECK: float 5
 ; CHECK: float 6
 define <4 x float> @fadd_fmul_x_c_fadd_x_x_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_fmul_x_c_fadd_x_x_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, %x
-  %z = fmul <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
-  %w = fadd <4 x float> %z, %y
+  %y = fadd fast <4 x float> %x, %x
+  %z = fmul fast <4 x float> %x, <float 1.0, float 2.0, float 3.0, float 4.0>
+  %w = fadd fast <4 x float> %z, %y
   ret <4 x float> %w
 }
 
 ; CHECK: float 3
 define float @fadd_x_fadd_x_x_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_x_fadd_x_x_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, %x
-  %z = fadd float %x, %y
+  %y = fadd fast float %x, %x
+  %z = fadd fast float %x, %y
   ret float %z
 }
 
 ; CHECK: float 3
 ; CHECK: float 3
 ; CHECK: float 3
 ; CHECK: float 3
 define <4 x float> @fadd_x_fadd_x_x_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_x_fadd_x_x_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, %x
-  %z = fadd <4 x float> %x, %y
+  %y = fadd fast <4 x float> %x, %x
+  %z = fadd fast <4 x float> %x, %y
   ret <4 x float> %z
 }
 
 ; CHECK: float 3
 define float @fadd_fadd_x_x_x_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_x_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, %x
-  %z = fadd float %y, %x
+  %y = fadd fast float %x, %x
+  %z = fadd fast float %y, %x
   ret float %z
 }
 
 ; CHECK: float 3
 ; CHECK: float 3
 ; CHECK: float 3
 ; CHECK: float 3
 define <4 x float> @fadd_fadd_x_x_x_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_x_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, %x
-  %z = fadd <4 x float> %y, %x
+  %y = fadd fast <4 x float> %x, %x
+  %z = fadd fast <4 x float> %y, %x
   ret <4 x float> %z
 }
 
 ; CHECK: float 4
 define float @fadd_fadd_x_x_fadd_x_x_f32(float %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_fadd_x_x_f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd float %x, %x
-  %z = fadd float %y, %y
+  %y = fadd fast float %x, %x
+  %z = fadd fast float %y, %y
   ret float %z
 }
 
 ; CHECK: float 4
 ; CHECK: float 4
 ; CHECK: float 4
 ; CHECK: float 4
 define <4 x float> @fadd_fadd_x_x_fadd_x_x_4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: fadd_fadd_x_x_fadd_x_x_4f32:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulps {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %y = fadd <4 x float> %x, %x
-  %z = fadd <4 x float> %y, %y
+  %y = fadd fast <4 x float> %x, %x
+  %z = fadd fast <4 x float> %y, %y
   ret <4 x float> %z
 }
 
-attributes #0 = { "less-precise-fpmad"="true" "no-infs-fp-math"="true" "no-nans-fp-math"="true" "unsafe-fp-math"="true" "no-signed-zeros-fp-math"="true" }
+attributes #0 = { "less-precise-fpmad"="true" }

test/CodeGen/X86/fdiv-combine.ll

; CHECK-NEXT:    retq
   ret float %div1
 }
 
 ; All math instructions are 'arcp', so optimize.
 
 define float @div2_arcp_all(float %x, float %y, float %z) {
 ; CHECK-LABEL: div2_arcp_all:
 ; CHECK:       # %bb.0:
-; CHECK-NEXT:    movss {{.*#+}} xmm3 = mem[0],zero,zero,zero
+; CHECK-NEXT:    movss {{.*}}(%rip), %xmm3
 ; CHECK-NEXT:    divss %xmm2, %xmm3
 ; CHECK-NEXT:    mulss %xmm3, %xmm0
 ; CHECK-NEXT:    mulss %xmm1, %xmm0
 ; CHECK-NEXT:    mulss %xmm3, %xmm0
 ; CHECK-NEXT:    retq
   %div1 = fdiv arcp float %x, %z
   %mul = fmul arcp float %div1, %y
   %div2 = fdiv arcp float %mul, %z

test/CodeGen/X86/fdiv.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown                        | FileCheck %s
 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -enable-unsafe-fp-math | FileCheck %s
 
 define double @exact(double %x) {
 ; Exact division by a constant converted to multiplication.
 ; CHECK-LABEL: exact:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulsd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %div = fdiv double %x, 2.0
+  %div = fdiv fast double %x, 2.0
   ret double %div
 }
 
 define double @inexact(double %x) {
 ; Inexact division by a constant converted to multiplication.
 ; CHECK-LABEL: inexact:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    mulsd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %div = fdiv double %x, 0x41DFFFFFFFC00000
+  %div = fdiv fast double %x, 0x41DFFFFFFFC00000
   ret double %div
 }
 
 define double @funky(double %x) {
 ; No conversion to multiplication if too funky.
 ; CHECK-LABEL: funky:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    xorpd %xmm1, %xmm1
 ; CHECK-NEXT:    divsd %xmm1, %xmm0
 ; CHECK-NEXT:    retq
-  %div = fdiv double %x, 0.0
+  %div = fdiv fast double %x, 0.0
   ret double %div
 }
 
 define double @denormal1(double %x) {
 ; Don't generate multiplication by a denormal.
 ; CHECK-LABEL: denormal1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    divsd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %div = fdiv double %x, 0x7FD0000000000001
+  %div = fdiv fast double %x, 0x7FD0000000000001
   ret double %div
 }
 
 define double @denormal2(double %x) {
 ; Don't generate multiplication by a denormal.
 ; CHECK-LABEL: denormal2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    divsd {{.*}}(%rip), %xmm0
 ; CHECK-NEXT:    retq
-  %div = fdiv double %x, 0x7FEFFFFFFFFFFFFF
+  %div = fdiv fast double %x, 0x7FEFFFFFFFFFFFFF
   ret double %div
 }
 
 ; Deleting the negates does not require unsafe-fp-math.
 
 define float @double_negative(float %x, float %y) #0 {
 ; CHECK-LABEL: double_negative:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    divss %xmm1, %xmm0
 ; CHECK-NEXT:    retq
-  %neg1 = fsub float -0.0, %x
-  %neg2 = fsub float -0.0, %y
-  %div = fdiv float %neg1, %neg2
+  %neg1 = fsub fast float -0.0, %x
+  %neg2 = fsub fast float -0.0, %y
+  %div = fdiv fast float %neg1, %neg2
   ret float %div
 }
-
-attributes #0 = { "unsafe-fp-math"="false" }
-

test/CodeGen/X86/fmf-flags.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-unknown | FileCheck %s -check-prefix=X64
 ; RUN: llc < %s -mtriple=i686-unknown   | FileCheck %s -check-prefix=X86
 
 declare float @llvm.sqrt.f32(float %x);
 
 define float @fast_recip_sqrt(float %x) {
 ; X64-LABEL: fast_recip_sqrt:
 ; X64:       # %bb.0:
-; X64-NEXT:    sqrtss %xmm0, %xmm1
-; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; X64-NEXT:    divss %xmm1, %xmm0
+; X64-NEXT:    rsqrtss %xmm0, %xmm1
+; X64-NEXT:    mulss %xmm1, %xmm0
+; X64-NEXT:    mulss %xmm1, %xmm0
+; X64-NEXT:    addss {{.*}}(%rip), %xmm0
+; X64-NEXT:    mulss {{.*}}(%rip), %xmm1
+; X64-NEXT:    mulss %xmm1, %xmm0
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: fast_recip_sqrt:
 ; X86:       # %bb.0:
 ; X86-NEXT:    flds {{[0-9]+}}(%esp)
 ; X86-NEXT:    fsqrt
 ; X86-NEXT:    fld1
 ; X86-NEXT:    fdivp %st(1)
 ; X86-NEXT:    retl
   %y = call fast float @llvm.sqrt.f32(float %x)
   %z = fdiv fast float 1.0,  %y
   ret float %z
 }
 
 declare float @llvm.fmuladd.f32(float %a, float %b, float %c);
 
 define float @fast_fmuladd_opts(float %a , float %b , float %c) {
 ; X64-LABEL: fast_fmuladd_opts:
 ; X64:       # %bb.0:
-; X64-NEXT:    movaps %xmm0, %xmm1
-; X64-NEXT:    addss %xmm0, %xmm1
-; X64-NEXT:    addss %xmm0, %xmm1
-; X64-NEXT:    movaps %xmm1, %xmm0
+; X64-NEXT:    mulss {{.*}}(%rip), %xmm0
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: fast_fmuladd_opts:
 ; X86:       # %bb.0:
 ; X86-NEXT:    flds {{[0-9]+}}(%esp)
-; X86-NEXT:    fld %st(0)
-; X86-NEXT:    fadd %st(1)
-; X86-NEXT:    faddp %st(1)
+; X86-NEXT:    fmuls {{.*}}
 ; X86-NEXT:    retl
   %res = call fast float @llvm.fmuladd.f32(float %a, float 2.0, float %a)
   ret float %res
 }
 
 ; The multiply is strict.
 
 @mul1 = common global double 0.000000e+00, align 4
 
 define double @not_so_fast_mul_add(double %x) {
 ; X64-LABEL: not_so_fast_mul_add:
 ; X64:       # %bb.0:
-; X64-NEXT:    movsd {{.*#+}} xmm1 = mem[0],zero
+; X64-NEXT:    movsd {{.*}}(%rip), %xmm1
 ; X64-NEXT:    mulsd %xmm0, %xmm1
-; X64-NEXT:    addsd %xmm1, %xmm0
+; X64-NEXT:    mulsd {{.*}}(%rip), %xmm0
 ; X64-NEXT:    movsd %xmm1, {{.*}}(%rip)
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: not_so_fast_mul_add:
 ; X86:       # %bb.0:
 ; X86-NEXT:    fldl {{[0-9]+}}(%esp)
 ; X86-NEXT:    fld %st(0)
 ; X86-NEXT:    fmull {{\.LCPI.*}}
-; X86-NEXT:    fadd %st(0), %st(1)
+; X86-NEXT:    fxch %st(1)
+; X86-NEXT:    fmull {{\.LCPI.*}}
+; X86-NEXT:    fxch %st(1)
 ; X86-NEXT:    fstpl mul1
 ; X86-NEXT:    retl
   %m = fmul double %x, 4.2
   %a = fadd fast double %m, %x
   store double %m, double* @mul1, align 4
   ret double %a
 }
 
 ; The sqrt is strict.
 
 @sqrt1 = common global float 0.000000e+00, align 4
 
 define float @not_so_fast_recip_sqrt(float %x) {
 ; X64-LABEL: not_so_fast_recip_sqrt:
 ; X64:       # %bb.0:
-; X64-NEXT:    sqrtss %xmm0, %xmm1
-; X64-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
-; X64-NEXT:    divss %xmm1, %xmm0
-; X64-NEXT:    movss %xmm1, {{.*}}(%rip)
+; X64-NEXT:    rsqrtss %xmm0, %xmm1
+; X64-NEXT:    sqrtss %xmm0, %xmm2
+; X64-NEXT:    mulss %xmm1, %xmm0
+; X64-NEXT:    mulss %xmm1, %xmm0
+; X64-NEXT:    addss {{.*}}(%rip), %xmm0
+; X64-NEXT:    mulss {{.*}}(%rip), %xmm1
+; X64-NEXT:    mulss %xmm1, %xmm0
+; X64-NEXT:    movss %xmm2, {{.*}}(%rip)
 ; X64-NEXT:    retq
 ;
 ; X86-LABEL: not_so_fast_recip_sqrt:
 ; X86:       # %bb.0:
 ; X86-NEXT:    flds {{[0-9]+}}(%esp)
 ; X86-NEXT:    fsqrt
 ; X86-NEXT:    fld1
 ; X86-NEXT:    fdiv %st(1)

test/CodeGen/X86/fp-fast.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=avx < %s                        | FileCheck %s
 ; RUN: llc -mtriple=x86_64-unknown-unknown -mattr=avx -enable-unsafe-fp-math < %s | FileCheck %s
 
 define float @test1(float %a) {
 ; CHECK-LABEL: test1:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fadd float %a, %a
-  %r = fadd float %t1, %t1
+  %t1 = fadd fast float %a, %a
+  %r = fadd fast float %t1, %t1
   ret float %r
 }
 
 define float @test2(float %a) {
 ; CHECK-LABEL: test2:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fmul float 4.0, %a
-  %t2 = fadd float %a, %a
-  %r = fadd float %t1, %t2
+  %t1 = fmul fast float 4.0, %a
+  %t2 = fadd fast float %a, %a
+  %r = fadd fast float %t1, %t2
   ret float %r
 }
 
 define float @test3(float %a) {
 ; CHECK-LABEL: test3:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fmul float %a, 4.0
-  %t2 = fadd float %a, %a
-  %r = fadd float %t1, %t2
+  %t1 = fmul fast float %a, 4.0
+  %t2 = fadd fast float %a, %a
+  %r = fadd fast float %t1, %t2
   ret float %r
 }
 
 define float @test4(float %a) {
 ; CHECK-LABEL: test4:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fadd float %a, %a
-  %t2 = fmul float 4.0, %a
-  %r = fadd float %t1, %t2
+  %t1 = fadd fast float %a, %a
+  %t2 = fmul fast float 4.0, %a
+  %r = fadd fast float %t1, %t2
   ret float %r
 }
 
 define float @test5(float %a) {
 ; CHECK-LABEL: test5:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fadd float %a, %a
-  %t2 = fmul float %a, 4.0
-  %r = fadd float %t1, %t2
+  %t1 = fadd fast float %a, %a
+  %t2 = fmul fast float %a, 4.0
+  %r = fadd fast float %t1, %t2
   ret float %r
 }
 
 define float @test6(float %a) {
 ; CHECK-LABEL: test6:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fmul float 2.0, %a
-  %t2 = fadd float %a, %a
-  %r = fsub float %t1, %t2
+  %t1 = fmul fast float 2.0, %a
+  %t2 = fadd fast float %a, %a
+  %r = fsub fast float %t1, %t2
   ret float %r
 }
 
 define float @test7(float %a) {
 ; CHECK-LABEL: test7:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fmul float %a, 2.0
-  %t2 = fadd float %a, %a
-  %r = fsub float %t1, %t2
+  %t1 = fmul fast float %a, 2.0
+  %t2 = fadd fast float %a, %a
+  %r = fsub fast float %t1, %t2
   ret float %r
 }
 
 define float @test8(float %a) {
 ; CHECK-LABEL: test8:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    retq
-  %t1 = fmul float %a, 0.0
-  %t2 = fadd float %a, %t1
+  %t1 = fmul fast float %a, 0.0
+  %t2 = fadd fast float %a, %t1
   ret float %t2
 }
 
 define float @test9(float %a) {
 ; CHECK-LABEL: test9:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    retq
-  %t1 = fmul float 0.0, %a
-  %t2 = fadd float %t1, %a
+  %t1 = fmul fast float 0.0, %a
+  %t2 = fadd fast float %t1, %a
   ret float %t2
 }
 
 define float @test10(float %a) {
 ; CHECK-LABEL: test10:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fsub float -0.0, %a
-  %t2 = fadd float %a, %t1
+  %t1 = fsub fast float -0.0, %a
+  %t2 = fadd fast float %a, %t1
   ret float %t2
 }
 
 define float @test11(float %a) {
 ; CHECK-LABEL: test11:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    vxorps %xmm0, %xmm0, %xmm0
 ; CHECK-NEXT:    retq
-  %t1 = fsub float -0.0, %a
-  %t2 = fadd float %a, %t1
+  %t1 = fsub fast float -0.0, %a
+  %t2 = fadd fast float %a, %t1
   ret float %t2
 }
 

test/CodeGen/X86/fp-fold.ll

 ; UNSAFE-LABEL: fadd_zero:
 ; UNSAFE:       # %bb.0:
 ; UNSAFE-NEXT:    retq
   %r = fadd float %x, 0.0
   ret float %r
 }
 
 define float @fadd_negzero(float %x) {
-; STRICT-LABEL: fadd_negzero:
-; STRICT:       # %bb.0:
-; STRICT-NEXT:    addss {{.*}}(%rip), %xmm0
-; STRICT-NEXT:    retq
-;
-; UNSAFE-LABEL: fadd_negzero:
-; UNSAFE:       # %bb.0:
-; UNSAFE-NEXT:    retq
+; ANY-LABEL: fadd_negzero:
+; ANY:       # %bb.0:
+; ANY-NEXT:    retq
   %r = fadd float %x, -0.0
   ret float %r
 }
 
 define float @fadd_zero_nsz(float %x) {
 ; ANY-LABEL: fadd_zero_nsz:
 ; ANY:       # %bb.0:
 ; ANY-NEXT:    retq
   %r = fadd nsz float %x, 0.0
   ret float %r
 }
 
 define float @fadd_negzero_nsz(float %x) {
 ; ANY-LABEL: fadd_negzero_nsz:
 ; ANY:       # %bb.0:
 ; ANY-NEXT:    retq
   %r = fadd nsz float %x, -0.0
   ret float %r
 }
 
 define float @fsub_zero(float %x) {
-; STRICT-LABEL: fsub_zero:
-; STRICT:       # %bb.0:
-; STRICT-NEXT:    addss {{.*}}(%rip), %xmm0
-; STRICT-NEXT:    retq
-;
-; UNSAFE-LABEL: fsub_zero:
-; UNSAFE:       # %bb.0:
-; UNSAFE-NEXT:    retq
+; ANY-LABEL: fsub_zero:
+; ANY:       # %bb.0:
+; ANY-NEXT:    retq
   %r = fsub float %x, 0.0
   ret float %r
 }
 
 define float @fsub_negzero(float %x) {
 ; STRICT-LABEL: fsub_negzero:
 ; STRICT:       # %bb.0:
 ; STRICT-NEXT:    xorps %xmm1, %xmm1

 ; ANY:       # %bb.0:
 ; ANY-NEXT:    retq
   %r = fsub nsz float %x, -0.0
   ret float %r
 }
 
 ; TODO: handle x*0 for fast flags the same as unsafe
 define float @fmul_zero(float %x) {
-; STRICT-LABEL: fmul_zero:
-; STRICT:       # %bb.0:
-; STRICT-NEXT:    xorps %xmm1, %xmm1
-; STRICT-NEXT:    mulss %xmm1, %xmm0
-; STRICT-NEXT:    retq
-;
-; UNSAFE-LABEL: fmul_zero:
-; UNSAFE:       # %bb.0:
-; UNSAFE-NEXT:    xorps %xmm0, %xmm0
-; UNSAFE-NEXT:    retq
+; ANY-LABEL: fmul_zero:
+; ANY:       # %bb.0:
+; ANY-NEXT:    xorps %xmm0, %xmm0
+; ANY-NEXT:    retq
   %r = fmul nnan nsz float %x, 0.0
   ret float %r
 }
 
 define float @fmul_one(float %x) {
 ; ANY-LABEL: fmul_one:
 ; ANY:       # %bb.0:
 ; ANY-NEXT:    retq
   %r = fmul float %x, 1.0
   ret float %r
 }