AMDGPU: Enhancement on FDIV lowering in AMDGPUCodeGenPrepare

[cfang]

Summary

This is to enhance the work in https://reviews.llvm.org/D71293.

1. We change the fpmath accuracy threshold for rcp from 2.5 to 1.0 ULP. This is cased on the accuracy that rcp can achieve in the hardware.
2. We do fdiv re-association only with unsafe-math or reciprocal-math (fast-math implies reciprocal-math).

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

714

fdiv.fast doesn't' care about the reassociation

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

714

You are right. This is just the optimization priority issue.

If we can reassociate fdiv, x/y -> x * rcp(y) is faster than fdiv.fast so we don't do fdiv.fast.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

714

The comment and variable name are misleading, as no reassociate is going on here. This needs an explanation here

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

714

I am going to write an explanation here.
But I am confused about fdiv.fast intrinsic:
1.0/x -> fdiv.fast (1.0, x) when denormals are supported. Because I think does not support fdiv.fast.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

714

I'm not sure what the question is. fdiv.fast is used depending on whether the denormal mode needs to be switched or not, and is separate from rcp. If we can use rcp, it's preferable to fdiv.fast

[cfang]

update based on comment.

1. 2.5ulp threshold for fdiv.fast, 1.0 ulp for rcp
2. introduce a function lowerUsingFDivFast to use fdiv.fast, this function should be called after lowerUsingRcp because rcp is the prefedrence

This update should clear some confusion.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

This has nothing to do with reassociation

614

This should not be referred to ass lowering

624

We aren't fdiv here. We're handling an fdiv, and not splitting it into a multiple and rcp

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

Division re-association: a/b -> a * rcp(b), and one special case is 1.0/b => 1.0*rcp(b) = rcp(b).
This is how 1.0/x -> rcp(x) associated with "re-association".

614

I am thinking of a different name. Do you have a meaningful name for the function in mind?

624

As explained in a previous comment, 1.0/x -> 1.0*rcp(x) = rcp(x) is a special case of re-association.
As a result, if the options specify re-association, we can do 1.0/x -> rcp(x).

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

This isn't reassocation. This is just special handling of 1.0/b. Nothing algebraic changes here. There's no multiply introduced here

614

combineRcp?

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

Ok, it seems we have a different understanding here.

I think this is still just a naming issue. Originally the name is something like UnsafeMath?
But I do think arcp also gives the permission to do 1.0/x -> rcp(x) even though no multitply is explicitly generated
(1.0 * rcp(x) = rcp(x)).

Maybe we should go back to use the original name as HasFastUnsafeOptions to clear your confusion?

614

Better to be somethingUseRcp and somethingUseFastFDiv. I am still not sure what something should be here.
optimizeFDivUsingRcp?

[cfang]

Rename a few functions and variables:

1. optimizeWithRcp and optimizeWithFDivFast
2. bool AllowInaccurateRcp = unsafe-fp-math || arcp
   • Specified or implied by the options or function attributes, rcp is allowed to be used even though it is not accurate.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

It's also the comment and flag being checked. There's no implicit or explicit multiply here, this is just a reciprocal. This pass is not responsible for doing the reassociating allowed by arcp. This should be a check for the approximate function math flag. allowRecriprocal is not relevant here

715

You don't need this anymore with getFPAccuracy

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

7434

Again, I thought we decided that arcp does not mean this can be less accurate. This should be checking for approximate function

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

So we only check unsafe-fast-math or afn here, not arcp? Thanks.

715

Do you think we will no longer need fpmath metadata after this point?
in fdiv.fast intrinsic or the fdiv itself which may not be changed (especially in the vector cases).

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

7434

Check only afn right? not arcp && afn? Thanks.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

611

afn, and the closest match for the attribute is unsafe-fast-math, and not arcp.

715

getFPAccuracy already did the only check you needed for it, so this should be a dead variable

[cfang]

Updated based on feedback:

1. replace arcp with afn
   • it is the ApproxFunc fast math bit that matters to use inaccurate rcp.
2. drop fpmath metadata since it will no longer be used after we visit fdiv here.

[arsenm]

LGTM

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

   BO.replaceAllUsesWith(NewSelect);
   BO.eraseFromParent();
   if (CastOp)
     CastOp->eraseFromParent();
   Sel->eraseFromParent();
   return true;
 }
 
-// Perform RCP optimizations:
+// lowerUsingRcp:
 //
-// 1/x -> rcp(x) when fast unsafe rcp is legal or fpmath >= 2.5ULP with
-//                                                denormals flushed.
+// 1/x -> rcp(x) when fdiv is allowed to be re-associated or rcp is accurate.
 //
-// a/b -> a*rcp(b) when fast unsafe rcp is legal.
-static Value *performRCPOpt(Value *Num, Value *Den, bool FastUnsafeRcpLegal,
-                            IRBuilder<> Builder, MDNode *FPMath, Module *Mod,
-                            bool HasDenormals, bool NeedHighAccuracy) {
+// a/b -> a*rcp(b) when fdiv is allowed to be re-associated.
+static Value *lowerUsingRcp (Value *Num, Value *Den, bool CanReassociateFDiv,
+                             bool RcpIsAccurate, IRBuilder<> Builder,
+                             MDNode *FPMath, Module *Mod) {
 
-  Type *Ty = Den->getType();
-  if (!FastUnsafeRcpLegal && Ty->isFloatTy() &&
-                             (HasDenormals || NeedHighAccuracy))
+  if (!CanReassociateFDiv && !RcpIsAccurate)
     return nullptr;
 
+  Type *Ty = Den->getType();
   Function *Decl = Intrinsic::getDeclaration(Mod, Intrinsic::amdgcn_rcp, Ty);
   if (const ConstantFP *CLHS = dyn_cast<ConstantFP>(Num)) {
-    if (FastUnsafeRcpLegal || Ty->isFloatTy() || Ty->isHalfTy()) {
+    if (CanReassociateFDiv || RcpIsAccurate) {
       if (CLHS->isExactlyValue(1.0)) {
         // v_rcp_f32 and v_rsq_f32 do not support denormals, and according to
         // the CI documentation has a worst case error of 1 ulp.
         // OpenCL requires <= 2.5 ulp for 1.0 / x, so it should always be OK to
         // use it as long as we aren't trying to use denormals.
         //
         // v_rcp_f16 and v_rsq_f16 DO support denormals.
 
         // NOTE: v_sqrt and v_rcp will be combined to v_rsq later. So we don't
         //       insert rsq intrinsic here.
 
         // 1.0 / x -> rcp(x)
         return Builder.CreateCall(Decl, { Den });
        }
 
        // Same as for 1.0, but expand the sign out of the constant.
        if (CLHS->isExactlyValue(-1.0)) {
          // -1.0 / x -> rcp (fneg x)
          Value *FNeg = Builder.CreateFNeg(Den);
          return Builder.CreateCall(Decl, { FNeg });
        }
     }
   }
 
-  if (FastUnsafeRcpLegal) {
+  if (CanReassociateFDiv) {
     // Turn into multiply by the reciprocal.
     // x / y -> x * (1.0 / y)
     Value *Recip = Builder.CreateCall(Decl, { Den });
     return Builder.CreateFMul(Num, Recip, "", FPMath);
   }
   return nullptr;
 }
 
-static bool shouldKeepFDivF32(Value *Num, bool FastUnsafeRcpLegal,
-                              bool HasDenormals) {
-  const ConstantFP *CNum = dyn_cast<ConstantFP>(Num);
-  if (!CNum)
-    return HasDenormals;
+// lowerUsingFDivFast:
+//
+// a/b -> fdiv.fast(a, b) when !fpmath >= 2.5ulp with denormals flushed.
+//
+// 1/x -> fdiv.fast(1,x)  when !fpmath >= 2.5ulp.
+//
+// NOTE: lowerUsingRcp should be tried first because rcp is the preference.
+static Value *lowerUsingFDivFast(Value *Num, Value *Den, float ReqdAccuracy,
+                                 bool HasDenormals, IRBuilder<> Builder,
+                                 MDNode *FPMath, Module *Mod) {
+  // fdiv.fast can achieve 2.5 ULP accuracy.
+  if (ReqdAccuracy < 2.5f)
+    return nullptr;
 
-  if (FastUnsafeRcpLegal)
-    return true;
+  // Only have fdiv.fast for f32.
+  Type *Ty = Den->getType();
+  if (!Ty->isFloatTy())
+    return nullptr;
 
-  bool IsOne = CNum->isExactlyValue(+1.0) || CNum->isExactlyValue(-1.0);
+  bool NumIsOne = false;
+  if (const ConstantFP *CNum = dyn_cast<ConstantFP>(Num)) {
+    if (CNum->isExactlyValue(+1.0) || CNum->isExactlyValue(-1.0))
+      NumIsOne = true;
+  }
 
-  // Reciprocal f32 is handled separately without denormals.
-  return HasDenormals ^ IsOne;
-}
+  // fdiv does not support denormals. But 1.0/x is always fine to use it.
+  if (HasDenormals && !NumIsOne)
+    return nullptr;
 
+  Function *Decl = Intrinsic::getDeclaration(Mod, Intrinsic::amdgcn_fdiv_fast);
+  return Builder.CreateCall(Decl, { Num, Den }, "", FPMath);
+}
 
 // Optimizations is performed based on fpmath, fast math flags as wells as
 // denormals to lower fdiv using either rcp or fdiv.fast.
 //
-// FastUnsafeRcpLegal: We determine whether it is legal to use rcp based on
-//                     unsafe-fp-math, fast math flags, denormals and fpmath
-//                     accuracy request.
+// Use rcp:
+//   1/x -> rcp(x) when fdiv is allowed to be re-associated or rcp is
+//                 sufficiently accurate.
 //
-// RCP Optimizations:
-//   1/x -> rcp(x) when fast unsafe rcp is legal or fpmath >= 2.5ULP with
-//                                                  denormals flushed.
-//   a/b -> a*rcp(b) when fast unsafe rcp is legal.
+//   a/b -> a*rcp(b) fdiv is allowed to be re-associated.
 //
 // Use fdiv.fast:
-//   a/b -> fdiv.fast(a, b) when RCP optimization is not performed and
-//                          fpmath >= 2.5ULP with denormals flushed.
+//   a/b -> fdiv.fast(a, b) when !fpmath >= 2.5ulp with denormals flushed.
+//
+//   1/x -> fdiv.fast(1,x)  when !fpmath >= 2.5ulp.
 //
-//   1/x -> fdiv.fast(1,x)  when RCP optimization is not performed and
-//                          fpmath >= 2.5ULP with denormals.
+// Using rcp is the preference.
 bool AMDGPUCodeGenPrepare::visitFDiv(BinaryOperator &FDiv) {
 
   Type *Ty = FDiv.getType()->getScalarType();
 
   // No intrinsic for fdiv16 if target does not support f16.
   if (Ty->isHalfTy() && !ST->has16BitInsts())
     return false;
 
   const FPMathOperator *FPOp = cast<const FPMathOperator>(&FDiv);
   MDNode *FPMath = FDiv.getMetadata(LLVMContext::MD_fpmath);
-  const bool NeedHighAccuracy = !FPMath || FPOp->getFPAccuracy() < 2.5f;
+  const float ReqdAccuracy =  FPOp->getFPAccuracy();
 
   FastMathFlags FMF = FPOp->getFastMathFlags();
-  // Determine whether it is ok to use rcp based on unsafe-fp-math,
-  // fast math flags, denormals and accuracy request.
-  const bool FastUnsafeRcpLegal = HasUnsafeFPMath || FMF.isFast() ||
-          (FMF.allowReciprocal() && ((!HasFP32Denormals && !NeedHighAccuracy)
-                                     || FMF.approxFunc()));
+  const bool CanReassociateFDiv = HasUnsafeFPMath || FMF.allowReciprocal();
 
-  // Use fdiv.fast for only f32, fpmath >= 2.5ULP and rcp is not used.
-  const bool UseFDivFast = Ty->isFloatTy() && !NeedHighAccuracy &&
-                           !FastUnsafeRcpLegal;
+  // rcp_f16 is accurate for !fpmath >= 1.0ulp.
+  // rcp_f32 is accurate for !fpmath >= 1.0ulp and denormals are flushed.
+  // rcp_f64 is never accurate.
+  const bool RcpIsAccurate = (Ty->isHalfTy() && ReqdAccuracy >= 1.0f) ||
+            (Ty->isFloatTy() && !HasFP32Denormals && ReqdAccuracy >= 1.0f);
 
   IRBuilder<> Builder(FDiv.getParent(), std::next(FDiv.getIterator()));
   Builder.setFastMathFlags(FMF);
   Builder.SetCurrentDebugLocation(FDiv.getDebugLoc());
 
   Value *Num = FDiv.getOperand(0);
   Value *Den = FDiv.getOperand(1);
 
   Value *NewFDiv = nullptr;
   if (VectorType *VT = dyn_cast<VectorType>(FDiv.getType())) {
     NewFDiv = UndefValue::get(VT);
 
     // FIXME: Doesn't do the right thing for cases where the vector is partially
     // constant. This works when the scalarizer pass is run first.
     for (unsigned I = 0, E = VT->getNumElements(); I != E; ++I) {
       Value *NumEltI = Builder.CreateExtractElement(Num, I);
       Value *DenEltI = Builder.CreateExtractElement(Den, I);
-      Value *NewElt = nullptr;
-      if (UseFDivFast && !shouldKeepFDivF32(NumEltI, FastUnsafeRcpLegal,
-                                           HasFP32Denormals)) {
-        Function *Decl =
-                 Intrinsic::getDeclaration(Mod, Intrinsic::amdgcn_fdiv_fast);
-        NewElt = Builder.CreateCall(Decl, { NumEltI, DenEltI }, "", FPMath);
-      }
-      if (!NewElt) // Try rcp.
-        NewElt = performRCPOpt(NumEltI, DenEltI, FastUnsafeRcpLegal, Builder,
-                               FPMath, Mod, HasFP32Denormals, NeedHighAccuracy);
-      if (!NewElt)
+      // Try rcp first.
+      Value *NewElt = lowerUsingRcp(NumEltI, DenEltI, CanReassociateFDiv,
+                                    RcpIsAccurate, Builder, FPMath, Mod);
+      if (!NewElt) // Try fdiv.fast.
+        NewElt = lowerUsingFDivFast(NumEltI, DenEltI, ReqdAccuracy,
+                                    HasFP32Denormals, Builder, FPMath, Mod);
+      if (!NewElt) // Keep the original.
         NewElt = Builder.CreateFDiv(NumEltI, DenEltI, "", FPMath);
 
       NewFDiv = Builder.CreateInsertElement(NewFDiv, NewElt, I);
     }
-  } else { // Scalar.
-    if (UseFDivFast && !shouldKeepFDivF32(Num, FastUnsafeRcpLegal,
-                                          HasFP32Denormals)) {
-      Function *Decl =
-               Intrinsic::getDeclaration(Mod, Intrinsic::amdgcn_fdiv_fast);
-      NewFDiv = Builder.CreateCall(Decl, { Num, Den }, "", FPMath);
-    }
-    if (!NewFDiv) { // Try rcp.
-      NewFDiv = performRCPOpt(Num, Den, FastUnsafeRcpLegal, Builder, FPMath,
-                              Mod, HasFP32Denormals, NeedHighAccuracy);
+  } else { // Scalar FDiv.
+    // Try rcp first.
+    NewFDiv = lowerUsingRcp(Num, Den, CanReassociateFDiv, RcpIsAccurate,
+                            Builder, FPMath, Mod);
+    if (!NewFDiv) { // Try fdiv.fast.
+      NewFDiv = lowerUsingFDivFast(Num, Den, ReqdAccuracy, HasFP32Denormals,
+                                   Builder, FPMath, Mod);
     }
   }
 
   if (NewFDiv) {
     FDiv.replaceAllUsesWith(NewFDiv);
     NewFDiv->takeName(&FDiv);
     FDiv.eraseFromParent();
   }

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

 SDValue SITargetLowering::lowerFastUnsafeFDIV(SDValue Op,
                                               SelectionDAG &DAG) const {
   SDLoc SL(Op);
   SDValue LHS = Op.getOperand(0);
   SDValue RHS = Op.getOperand(1);
   EVT VT = Op.getValueType();
   const SDNodeFlags Flags = Op->getFlags();
 
-  bool FastUnsafeRcpLegal = DAG.getTarget().Options.UnsafeFPMath ||
-         (Flags.hasAllowReciprocal() &&
-          ((VT == MVT::f32 && hasFP32Denormals(DAG.getMachineFunction())) ||
-            VT == MVT::f16 ||
-            Flags.hasApproximateFuncs()));
-
-  // Do rcp optimization only when fast unsafe rcp is legal here.
-  // NOTE: We already performed RCP optimization to insert intrinsics in
-  // AMDGPUCodeGenPrepare. Ideally there should have no opportunity here to
-  // rcp optimization.
-  //   However, there are cases like FREM, which is expended into a sequence
-  // of instructions including FDIV, which may expose new opportunities.
-  if (!FastUnsafeRcpLegal)
+  bool CanReassociateFDiv = DAG.getTarget().Options.UnsafeFPMath ||
+                            Flags.hasAllowReciprocal();
+
+  // Without !fpmath accuracy information, we can't do more because we don't
+  // know exactly whether rcp is accurate enough to meet !fpmath requirement.
+  if (!CanReassociateFDiv)
     return SDValue();
 
   if (const ConstantFPSDNode *CLHS = dyn_cast<ConstantFPSDNode>(LHS)) {
     if (CLHS->isExactlyValue(1.0)) {
       // v_rcp_f32 and v_rsq_f32 do not support denormals, and according to
       // the CI documentation has a worst case error of 1 ulp.
       // OpenCL requires <= 2.5 ulp for 1.0 / x, so it should always be OK to
       // use it as long as we aren't trying to use denormals.

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fdiv.ll

 
   ret void
 }
 
 ; CHECK-LABEL: @rcp_fdiv_fpmath(
 ; CHECK: %no.md = fdiv float 1.000000e+00, %x{{$}}
 ; CHECK: %md.25ulp = call float @llvm.amdgcn.rcp.f32(float %x)
 ; CHECK: %md.half.ulp = fdiv float 1.000000e+00, %x, !fpmath !1
-; CHECK: %arcp.no.md = fdiv arcp float 1.000000e+00, %x
+; CHECK: %arcp.no.md = call arcp float @llvm.amdgcn.rcp.f32(float %x)
 ; CHECK: %arcp.25ulp = call arcp float @llvm.amdgcn.rcp.f32(float %x)
 ; CHECK: %fast.no.md = call fast float @llvm.amdgcn.rcp.f32(float %x)
 ; CHECK: %fast.25ulp = call fast float @llvm.amdgcn.rcp.f32(float %x)
 define amdgpu_kernel void @rcp_fdiv_fpmath(float addrspace(1)* %out, float %x) #1 {
   %no.md = fdiv float 1.0, %x
   store volatile float %no.md, float addrspace(1)* %out
 
   %md.25ulp = fdiv float 1.0, %x, !fpmath !0

   store volatile float %fast.no.md, float addrspace(1)* %out
 
   %fast.25ulp = fdiv fast float 1.0, %x, !fpmath !0
   store volatile float %fast.25ulp, float addrspace(1)* %out
 
   ret void
 }
 
-; CHECK-LABEL: @rcp_fdiv_arcp_denormal(
-; CHECK: %arcp.low.accuracy = call arcp float @llvm.amdgcn.fdiv.fast(float 1.000000e+00, float %x), !fpmath !0
-; CHECK: %arcp.high.accuracy = fdiv arcp float 1.000000e+00, %x, !fpmath !2
-; CHECK: %arcp.low.afn = call arcp afn float @llvm.amdgcn.rcp.f32(float %x)
-; CHECK: %arcp.high.afn = call arcp afn float @llvm.amdgcn.rcp.f32(float %x)
-define amdgpu_kernel void @rcp_fdiv_arcp_denormal(float addrspace(1)* %out, float %x) #2 {
-
-  %arcp.low.accuracy = fdiv arcp float 1.0, %x, !fpmath !0
-  store volatile float %arcp.low.accuracy, float addrspace(1)* %out
-
-  %arcp.high.accuracy = fdiv arcp float 1.0, %x, !fpmath !2
-  store volatile float %arcp.high.accuracy, float addrspace(1)* %out
-
-  %arcp.low.afn = fdiv arcp afn float 1.0, %x, !fpmath !0
-  store volatile float %arcp.low.afn, float addrspace(1)* %out
-
-  %arcp.high.afn = fdiv arcp afn float 1.0, %x, !fpmath !2
-  store volatile float %arcp.high.afn, float addrspace(1)* %out
-
-  ret void
-}
-
 ; CHECK-LABEL: @fdiv_fpmath_vector(
 ; CHECK: %[[NO_A0:[0-9]+]] = extractelement <2 x float> %a, i64 0
 ; CHECK: %[[NO_B0:[0-9]+]] = extractelement <2 x float> %b, i64 0
 ; CHECK: %[[NO_FDIV0:[0-9]+]] = fdiv float %[[NO_A0]], %[[NO_B0]]
 ; CHECK: %[[NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[NO_FDIV0]], i64 0
 ; CHECK: %[[NO_A1:[0-9]+]] = extractelement <2 x float> %a, i64 1
 ; CHECK: %[[NO_B1:[0-9]+]] = extractelement <2 x float> %b, i64 1
 ; CHECK: %[[NO_FDIV1:[0-9]+]] = fdiv float %[[NO_A1]], %[[NO_B1]]

 ; CHECK: %[[HALF_FDIV0:[0-9]+]] = fdiv float 1.000000e+00, %[[HALF0]], !fpmath !1
 ; CHECK: %[[HALF_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[HALF_FDIV0]], i64 0
 ; CHECK: %[[HALF1:[0-9]+]] = extractelement <2 x float> %x, i64 1
 ; CHECK: %[[HALF_FDIV1:[0-9]+]] =  fdiv float 1.000000e+00, %[[HALF1]], !fpmath !1
 ; CHECK: %md.half.ulp = insertelement <2 x float> %[[HALF_INS0]], float %[[HALF_FDIV1]], i64 1
 ; CHECK: store volatile <2 x float> %md.half.ulp, <2 x float> addrspace(1)* %out
 
 ; CHECK: %[[ARCP_NO0:[0-9]+]] =  extractelement <2 x float> %x, i64 0
-; CHECK: %[[ARCP_NO_FDIV0:[0-9]+]] = fdiv arcp float 1.000000e+00, %[[ARCP_NO0]]
+; CHECK: %[[ARCP_NO_FDIV0:[0-9]+]] = call arcp float @llvm.amdgcn.rcp.f32(float %[[ARCP_NO0]])
 ; CHECK: %[[ARCP_NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[ARCP_NO_FDIV0]], i64 0
 ; CHECK: %[[ARCP_NO1:[0-9]+]] = extractelement <2 x float> %x, i64 1
-; CHECK: %[[ARCP_NO_FDIV1:[0-9]+]] =  fdiv arcp float 1.000000e+00, %[[ARCP_NO1]]
+; CHECK: %[[ARCP_NO_FDIV1:[0-9]+]] =  call arcp float @llvm.amdgcn.rcp.f32(float %[[ARCP_NO1]])
 ; CHECK: %arcp.no.md = insertelement <2 x float> %[[ARCP_NO_INS0]], float %[[ARCP_NO_FDIV1]], i64 1
 ; CHECK: store volatile <2 x float> %arcp.no.md, <2 x float> addrspace(1)* %out
 
 ; CHECK: %[[FAST_NO0:[0-9]+]] =  extractelement <2 x float> %x, i64 0
 ; CHECK: %[[FAST_NO_RCP0:[0-9]+]] = call fast float @llvm.amdgcn.rcp.f32(float %[[FAST_NO0]])
 ; CHECK: %[[FAST_NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[FAST_NO_RCP0]], i64 0
 ; CHECK: %[[FAST_NO1:[0-9]+]] = extractelement <2 x float> %x, i64 1
 ; CHECK: %[[FAST_NO_RCP1:[0-9]+]] =  call fast float @llvm.amdgcn.rcp.f32(float %[[FAST_NO1]])

 ; CHECK: %[[NO_FDIV0:[0-9]+]] = fdiv float 1.000000e+00, %[[NO0]]
 ; CHECK: %[[NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[NO_FDIV0]], i64 0
 ; CHECK: %[[NO1:[0-9]+]] = extractelement <2 x float> %x, i64 1
 ; CHECK: %[[NO_FDIV1:[0-9]+]] = fdiv float 2.000000e+00, %[[NO1]]
 ; CHECK: %no.md = insertelement <2 x float> %[[NO_INS0]], float %[[NO_FDIV1]], i64 1
 ; CHECK: store volatile <2 x float> %no.md, <2 x float> addrspace(1)* %out
 
 ; CHECK: %[[ARCP_NO0:[0-9]+]] =  extractelement <2 x float> %x, i64 0
-; CHECK: %[[ARCP_NO_FDIV0:[0-9]+]] = fdiv arcp float 1.000000e+00, %[[ARCP_NO0]]
+; CHECK: %[[ARCP_NO_FDIV0:[0-9]+]] = call arcp float @llvm.amdgcn.rcp.f32(float %[[ARCP_NO0]])
 ; CHECK: %[[ARCP_NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[ARCP_NO_FDIV0]], i64 0
 ; CHECK: %[[ARCP_NO1:[0-9]+]] = extractelement <2 x float> %x, i64 1
-; CHECK: %[[ARCP_NO_FDIV1:[0-9]+]] =  fdiv arcp float 2.000000e+00, %[[ARCP_NO1]]
-; CHECK: %arcp.no.md = insertelement <2 x float> %[[ARCP_NO_INS0]], float %[[ARCP_NO_FDIV1]], i64 1
+; CHECK: %[[ARCP_NO_FDIV1:[0-9]+]] =  call arcp float @llvm.amdgcn.rcp.f32(float %[[ARCP_NO1]])
+; CHECK: %[[ARCP_NO_MUL1:[0-9]+]] = fmul arcp float 2.000000e+00, %[[ARCP_NO_FDIV1]]
+; CHECK: %arcp.no.md = insertelement <2 x float> %[[ARCP_NO_INS0]], float %[[ARCP_NO_MUL1]], i64 1
 ; CHECK: store volatile <2 x float> %arcp.no.md, <2 x float> addrspace(1)* %out
 
 ; CHECK: %[[FAST_NO0:[0-9]+]] =  extractelement <2 x float> %x, i64 0
 ; CHECK: %[[FAST_NO_RCP0:[0-9]+]] = call fast float @llvm.amdgcn.rcp.f32(float %[[FAST_NO0]])
 ; CHECK: %[[FAST_NO_INS0:[0-9]+]] = insertelement <2 x float> undef, float %[[FAST_NO_RCP0]], i64 0
 ; CHECK: %[[FAST_NO1:[0-9]+]] = extractelement <2 x float> %x, i64 1
 ; CHECK: %[[FAST_NO_RCP1:[0-9]+]] =  call fast float @llvm.amdgcn.rcp.f32(float %[[FAST_NO1]])
 ; CHECK: %[[FAST_NO_MUL1:[0-9]+]] = fmul fast float 2.000000e+00, %[[FAST_NO_RCP1]]

 ; CHECK-LABEL: @fdiv_fpmath_f32_denormals(
 ; CHECK: %no.md = fdiv float %a, %b{{$}}
 ; CHECK: %md.half.ulp = fdiv float %a, %b, !fpmath !1
 ; CHECK: %md.1ulp = fdiv float %a, %b, !fpmath !2
 ; CHECK: %md.25ulp = fdiv float %a, %b, !fpmath !0
 ; CHECK: %md.3ulp = fdiv float %a, %b, !fpmath !3
 ; CHECK: %[[RCP_FAST:[0-9]+]] = call fast float @llvm.amdgcn.rcp.f32(float %b)
 ; CHECK: %fast.md.25ulp = fmul fast float %a, %[[RCP_FAST]], !fpmath !0
-; CHECK: %arcp.md.25ulp  = fdiv arcp float %a, %b, !fpmath !0
+; CHECK: %[[RCP_ARCP:[0-9]+]] = call arcp float @llvm.amdgcn.rcp.f32(float %b)
+; CHECK: %arcp.md.25ulp  = fmul arcp float %a, %[[RCP_ARCP]], !fpmath !0
 define amdgpu_kernel void @fdiv_fpmath_f32_denormals(float addrspace(1)* %out, float %a, float %b) #2 {
   %no.md = fdiv float %a, %b
   store volatile float %no.md, float addrspace(1)* %out
 
   %md.half.ulp = fdiv float %a, %b, !fpmath !1
   store volatile float %md.half.ulp, float addrspace(1)* %out
 
   %md.1ulp = fdiv float %a, %b, !fpmath !2

llvm/test/CodeGen/AMDGPU/fdiv.f16.ll

 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rcp_f16(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
-  %r.val = fdiv half 1.0, %b.val
+  %r.val = fdiv half 1.0, %b.val, !fpmath !0
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_rcp_f16_abs:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[VAL:v[0-9]+]]
 ; GFX8_9_10-NOT: [[VAL]]
 ; GFX8_9_10: v_rcp_f16_e64 [[RESULT:v[0-9]+]], |[[VAL]]|
 ; GFX8_9_10-NOT: [RESULT]]
 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rcp_f16_abs(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
   %b.abs = call half @llvm.fabs.f16(half %b.val)
-  %r.val = fdiv half 1.0, %b.abs
+  %r.val = fdiv half 1.0, %b.abs, !fpmath !0
+  store half %r.val, half addrspace(1)* %gep.r
+  ret void
+}
+
+; We could not do 1/b -> rcp_f16(b) under !fpmath < 1ulp.
+
+; GCN-LABEL: {{^}}reciprocal_f16_rounded:
+; GFX8_9_10: {{flat|global}}_load_ushort [[VAL16:v[0-9]+]], v{{\[[0-9]+:[0-9]+\]}}
+; GFX8_9_10: v_cvt_f32_f16_e32 [[CVT_TO32:v[0-9]+]], [[VAL16]]
+; GFX8_9_10: v_rcp_f32_e32 [[RCP32:v[0-9]+]], [[CVT_TO32]]
+; GFX8_9_10: v_cvt_f16_f32_e32 [[CVT_BACK16:v[0-9]+]], [[RCP32]]
+; GFX8_9_10: v_div_fixup_f16 [[RESULT:v[0-9]+]], [[CVT_BACK16]], [[VAL16]], 1.0
+; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
+define amdgpu_kernel void @reciprocal_f16_rounded(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
+entry:
+  %tid = call i32 @llvm.amdgcn.workitem.id.x()
+  %tid.ext = sext i32 %tid to i64
+  %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
+  %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
+  %b.val = load volatile half, half addrspace(1)* %gep.b
+  %r.val = fdiv half 1.0, %b.val
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_rcp_f16_arcp:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[VAL:v[0-9]+]]
 ; GFX8_9_10-NOT: [[VAL]]
 ; GFX8_9_10: v_rcp_f16_e32 [[RESULT:v[0-9]+]], [[VAL]]
 ; GFX8_9_10-NOT: [[RESULT]]
 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rcp_f16_arcp(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
-  %r.val = fdiv arcp half 1.0, %b.val
+  %r.val = fdiv arcp half 1.0, %b.val, !fpmath !0
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_rcp_f16_neg:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[VAL:v[0-9]+]]
 ; GFX8_9_10-NOT: [[VAL]]
 ; GFX8_9_10: v_rcp_f16_e64 [[RESULT:v[0-9]+]], -[[VAL]]
 ; GFX8_9_10-NOT: [RESULT]]
 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rcp_f16_neg(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
-  %r.val = fdiv half -1.0, %b.val
+  %r.val = fdiv half -1.0, %b.val, !fpmath !0
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_rsq_f16:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[VAL:v[0-9]+]]
 ; GFX8_9_10-NOT: [[VAL]]
 ; GFX8_9_10: v_rsq_f16_e32 [[RESULT:v[0-9]+]], [[VAL]]
 ; GFX8_9_10-NOT: [RESULT]]
 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rsq_f16(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
   %b.sqrt = call half @llvm.sqrt.f16(half %b.val)
-  %r.val = fdiv half 1.0, %b.sqrt
+  %r.val = fdiv half 1.0, %b.sqrt, !fpmath !0
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_rsq_f16_neg:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[VAL:v[0-9]+]]
 ; GFX8_9_10-NOT: [[VAL]]
 ; GFX8_9_10: v_sqrt_f16_e32 [[SQRT:v[0-9]+]], [[VAL]]
 ; GFX8_9_10-NEXT: v_rcp_f16_e64 [[RESULT:v[0-9]+]], -[[SQRT]]
 ; GFX8_9_10-NOT: [RESULT]]
 ; GFX8_9_10: {{flat|global}}_store_short v{{\[[0-9]+:[0-9]+\]}}, [[RESULT]]
 define amdgpu_kernel void @v_rsq_f16_neg(half addrspace(1)* %r, half addrspace(1)* %b) #0 {
 entry:
   %tid = call i32 @llvm.amdgcn.workitem.id.x()
   %tid.ext = sext i32 %tid to i64
   %gep.b = getelementptr inbounds half, half addrspace(1)* %b, i64 %tid.ext
   %gep.r = getelementptr inbounds half, half addrspace(1)* %r, i64 %tid.ext
   %b.val = load volatile half, half addrspace(1)* %gep.b
   %b.sqrt = call half @llvm.sqrt.f16(half %b.val)
-  %r.val = fdiv half -1.0, %b.sqrt
+  %r.val = fdiv half -1.0, %b.sqrt, !fpmath !0
   store half %r.val, half addrspace(1)* %gep.r
   ret void
 }
 
 ; GCN-LABEL: {{^}}v_fdiv_f16_arcp:
 ; GFX8_9_10: {{flat|global}}_load_ushort [[LHS:v[0-9]+]]
 ; GFX8_9_10: {{flat|global}}_load_ushort [[RHS:v[0-9]+]]
 

 
 declare i32 @llvm.amdgcn.workitem.id.x() #1
 declare half @llvm.sqrt.f16(half) #1
 declare half @llvm.fabs.f16(half) #1
 
 attributes #0 = { nounwind }
 attributes #1 = { nounwind readnone }
 attributes #2 = { nounwind "unsafe-fp-math"="true" }
+
+!0 = !{float 2.500000e+00}