AMDGPU: Generate the correct sequence of code for FDIV32 when correctly-rounded-divide-sqrt is set

[cfang]

Summary

As the name suggests, correctly-rounded-divide-sqrt specifies the result of divion/sqrt to be rounded, and
thus we need to generate the correct sequence of code even when we flush the denormals.

[b-sumner]

This looks OK to me, although tuning on correctly rounded division any time denorms are enabled is not actually required by OpenCL.

[arsenm]

The attribute should not de directly checked (we probably shouldn’t even be putting it on the function). The proper thing to check is the fpmath metadata on the individual instruction. This isn’t propagated into the DAG, so AMDGPUCodeGenPrepare inserts intrinsic calls which isn’t ideal

[cfang]

In D71293#1778867, @arsenm wrote:

The attribute should not de directly checked (we probably shouldn’t even be putting it on the function). The proper thing to check is the fpmath metadata on the individual instruction. This isn’t propagated into the DAG, so AMDGPUCodeGenPrepare inserts intrinsic calls which isn’t ideal

:
So what's your suggestion here? The current logic in AMDGPUCodeGenPrepare is to find cases that we can insert the intrinsic to generate "Faster 2.5 ULP division that does not support denormals."
Otherwise SIISelLowering will lower FDIV32 UnsafeMath and Demorm support.

Do you want to change this logic to insert new intrinsics to generate the expected sequence of code for fdiv32?

[cfang]

Introduce an intrinsic in AMDGPUCodeGenPrepare to generate correctly rounded fdiv32.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

568–569

The attribute should not be considered at all. Only the fpmath metadata matters. If -cl-fp32-correctly-rounded-divide-sqrt is specified, the regular fdiv instruction should behave correctly.

571–575

An intrinsic should only be introduced when the fdiv differs from the default FP environment. Here you are doing the opposite, and not even considering the denormal mode. You should be inhibiting the insertion of the fdiv.fast if denormals are enabled, not introducing a new intrinsic. You can also consider the afn fast flag and use that to ignore the denormal mode

628–631

There's no need to check the attribute

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fdiv.ll

245

The attribute should be removed

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

568–569

Do you mean that in AMDGPUCodeGenPrepare, we should check the fpmath metadata to keep regular fdiv (instead of an intrinsic) when -cl-fp32-correctly-rounded-divide-sqrt is specified?

The issue is, when -cl-fp32-correctly-rounded-divide-sqrt is specified, a simple v_rcp is generated for a fdiv. Apparently the codegen produces the wrong sequence of code for a "regular" fdiv.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

568–569

Yes. We shouldn't even have an IR attribute for this flag. The interpretation of the flag is entirely represented in the use of the !fpmath metadata.

[cfang]

Implement rcp optimization for fdiv in AMGGPUCodegenPrepare to insert amdgcn_rcp intrinsic. For f32 type fdiv,
if fpmath metadata is unavailable, we could not do rcp optimization unless fast unsafe math is specified.

[arsenm]

The GlobalISel path should also be fixed, but that can be a follow up patch

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

585

Needs comment explaining the interaction between !fpmath requirements and denormals. This could use a chart of different fast math options, FP math and denormal handling and the expected lowering

603

I don't think just allow reciprocal is sufficient without either checking FPMath or afn. I think this needs to be something more like
UnsafeFP || isFast || (allowReciprocal && (denormal hasLowAccuracy || approximateFunction))

612–614

It would be clearer to do something like
bool NeedHighAccuracy = !FPMath || FPMath->getFPAccuracy() < 2.5

617

Typo metadat

618

It would be clearer to invert this, instead of the logic below relying on the double negative

619

FPMath should be checked once, and in relation to it's value only. Checking for the lack of metadata here is imprecise

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

7540

Based on the original problem, Flags.hasAllowReciprocal() isn't sufficient here. Without knowledge of !fpmath, this also needs approximate function

7542

Needs comment explaining why

8726

Braces

[arsenm]

For GlobalISel, I'm not sure this should reproduce the same fix. We can more plausibly preserve the !fpmath in the gMIR and handle it the right way, instead of hacking around it in AMDGPUCodeGenPrepare. I think a few asserts and the verifier would need to be updated, but it should be possible to allow arbitrary MDNode operands on an instruction, similar to how implicit registers can be added. I think we should disallow implicit register operands on G_* instructions, and instead only allow implicit metadata arguments. The fdiv lowering can then do the right thing with the original !fpmath information

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

612–614

Is < 2.5 ulp the limiting factor that we can not do 1/x -> rcp(x) ?

619

Do you mean here we should check like this:

(Ty->isFloatTy() && (HasFP32Denormals || NeedHighAccuracy));

where NeedHighAccuracy is checked like a previous comment?

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

603

Can you explain what is exactly "denormal hasLowAccuracy" here?

[cfang]

Update based on feedback from the reviewer.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

87–89

This should be redundant with the below logic

96–98

This should be fully captured by the logic above

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

600

UnsafeDiv is too imprecise here. This should explain in concrete terms why we need to insert the intrinsics and not just refer to the variable names. We need fdiv.fast when we only need 2.5 ULP and denormals are flushed

624

I think this should maybe be rephrased into RcpLegal and UseFDivFast

[cfang]

Update based on the comments.

1. Rewrite the comments of the function visitFDiv;
2. Rename a few variables.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

616

You can just initialize this below with the logical value instead of setting the value conditionally

640

I think this still isn't quite right.

I think this should be (FMF.allowReciprocal() && ((!HasFP32Denormals && !NeedHighAccuracy) || FMF.approxFunc())).

As is, this will allow reciprocal when denormals are flushed, but the higher fdiv precision is required, which was the case you were trying to fix in the first place

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

7542

This still needs the denormal and type checks

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fdiv.ll

88

This should not have produced rcp since denormals are enabled and it doesn't have afn.

91–92

The name says high accuracy, but 5 ulp is lower accuracy. This didn't form rcp, but I think for the wrong reason

94–98

These two I think are OK because of afn

[cfang]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

616

Thanks, Will do like that.

640

How could we handle fp16 and fp64? I think HasFP32Denormals only matter for fp32.

Also, the issue I am working on seems not related to FMF.allowReciprocal() at all unless arcp is default.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

640

Yes, this also needs to account for FP32denormals. RCP for f16 doesn't' care about the fp16 denormal mode

[cfang]

update based on feedback.

1. Using arcp && (( no denormals && fpmath>=2.5) || afn)
2. update arcp related LIT tests.

[arsenm]

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

122

Set UseFDivFast once based on the logical expression below and never mutate it. UseFDivFast should be const

llvm/test/CodeGen/AMDGPU/fdiv.f16.ll

253 ↗

(On Diff #239724)

I don't know what ulp the f16 rcp instruction provides. This test change looks incomplete if there isn't already a case without !fpmath

[arsenm]

llvm/test/CodeGen/AMDGPU/fdiv.f16.ll

253 ↗

(On Diff #239724)

I found a document stating this provides "~0.5ulp", so I guess check that value for f16?

[cfang]

llvm/test/CodeGen/AMDGPU/fdiv.f16.ll

253 ↗

(On Diff #239724)

Currently the logic in DAG lowering does "1/x -> rcp(x)" for fp16 without checking fpmath accuracy.
Actually it always does "1/x -> rcp(x)" for fp16 because v_rcp_f16 supports denormals.

We need to revisit that logic in DAG lowering. But I would rather to do that in a follow-up patch.

[cfang]

Update based on feedback:

1. const for UseFDivFast variable
2. Remove the added "!fpmath !0" for an arcp f16 test, because the current logic in DAG loweing generates the same code with/without !fpmath.

TODO (in an follow up patch maybe): Change the accuracy threshold and apply the threshold to all types. Also need to re-visit
the rcp logic in DAG Lowering as long as the work in AMDGPUCodegenPrepare is done.

[arsenm]

LGTM

[cfang]

commit 2531535984ad989ce88aeee23cb92a827da6686e
Author: Changpeng Fang <[e-mail]>
Date: Thu Jan 23 16:57:43 2020 -0800

llvm/include/llvm/IR/IntrinsicsAMDGPU.td

 def int_amdgcn_unreachable : Intrinsic<[], [], [IntrConvergent]>;
 
 // Emit 2.5 ulp, no denormal division. Should only be inserted by
 // pass based on !fpmath metadata.
 def int_amdgcn_fdiv_fast : Intrinsic<
   [llvm_float_ty], [llvm_float_ty, llvm_float_ty],
   [IntrNoMem, IntrSpeculatable]
 >;
+
+// Emit correctly rounded fp32 divide and sqrt.
+def int_amdgcn_fdiv_rounded : Intrinsic<
+  [llvm_float_ty], [llvm_float_ty, llvm_float_ty],
+  [IntrNoMem, IntrSpeculatable]
+>;
 }

llvm/lib/Target/AMDGPU/AMDGPUCodeGenPrepare.cpp

 class AMDGPUCodeGenPrepare : public FunctionPass,
                              public InstVisitor<AMDGPUCodeGenPrepare, bool> {
   const GCNSubtarget *ST = nullptr;
   AssumptionCache *AC = nullptr;
   LegacyDivergenceAnalysis *DA = nullptr;
   Module *Mod = nullptr;
   const DataLayout *DL = nullptr;
   bool HasUnsafeFPMath = false;
+  bool HasCorrectlyRoundedDivideSqrt = false;
   bool HasFP32Denormals = false;
 
   /// Copies exact/nsw/nuw flags (if any) from binary operation \p I to
   /// binary operation \p V.
   ///
   /// \returns Binary operation \p V.
   /// \returns \p T's base element bit width.
   unsigned getBaseElementBitWidth(const Type *T) const;
 
   /// \returns Equivalent 32 bit integer type for given type \p T. For example,
   /// if \p T is i7, then i32 is returned; if \p T is <3 x i12>, then <3 x i32>
   /// is returned.
   Type *getI32Ty(IRBuilder<> &B, const Type *T) const;
 
   /// \returns True if binary operation \p I is a signed binary operation, false
   /// otherwise.
   bool isSigned(const BinaryOperator &I) const;
 
   /// \returns True if the condition of 'select' operation \p I comes from a
   /// signed 'icmp' operation, false otherwise.

 
   // Reciprocal f32 is handled separately without denormals.
   return HasDenormals ^ IsOne;
 }
 
 // Insert an intrinsic for fast fdiv for safe math situations where we can
 // reduce precision. Leave fdiv for situations where the generic node is
 // expected to be optimized.
+//
+// Also, insert an intrinsic for safe fdiv when
+// -cl-fp32-correctly-rounded-divide-sqrt is enabled.
 bool AMDGPUCodeGenPrepare::visitFDiv(BinaryOperator &FDiv) {
   Type *Ty = FDiv.getType();
 
   if (!Ty->getScalarType()->isFloatTy())
     return false;
 
-  MDNode *FPMath = FDiv.getMetadata(LLVMContext::MD_fpmath);
-  if (!FPMath)
-    return false;
-
-  const FPMathOperator *FPOp = cast<const FPMathOperator>(&FDiv);
-  float ULP = FPOp->getFPAccuracy();
-  if (ULP < 2.5f)
-    return false;
-
+   const FPMathOperator *FPOp = cast<const FPMathOperator>(&FDiv);
   FastMathFlags FMF = FPOp->getFastMathFlags();
   bool UnsafeDiv = HasUnsafeFPMath || FMF.isFast() ||
                                       FMF.allowReciprocal();
-
   // With UnsafeDiv node will be optimized to just rcp and mul.
   if (UnsafeDiv)
     return false;
 
+  bool SafeFast = true;
+  MDNode *FPMath = FDiv.getMetadata(LLVMContext::MD_fpmath);
+  if (!FPMath) {
+    // Insert an intrinsic when -cl-fp32-correctly-rounded-divide-sqrt
+    // is enabled.
+    if (!HasCorrectlyRoundedDivideSqrt)
+      return false;
+    SafeFast = false;
+  } else {
+    float ULP = FPOp->getFPAccuracy();
+    if (ULP < 2.5f)
+      return false;
+  }
+
   IRBuilder<> Builder(FDiv.getParent(), std::next(FDiv.getIterator()), FPMath);
   Builder.setFastMathFlags(FMF);
   Builder.SetCurrentDebugLocation(FDiv.getDebugLoc());
 
-  Function *Decl = Intrinsic::getDeclaration(Mod, Intrinsic::amdgcn_fdiv_fast);
+  unsigned IntrinsicOpc = SafeFast ? Intrinsic::amdgcn_fdiv_fast 
+                                   : Intrinsic::amdgcn_fdiv_rounded;
+
+  Function *Decl = Intrinsic::getDeclaration(Mod, IntrinsicOpc);
 
   Value *Num = FDiv.getOperand(0);
   Value *Den = FDiv.getOperand(1);
 
   Value *NewFDiv = nullptr;
 
   if (VectorType *VT = dyn_cast<VectorType>(Ty)) {
     NewFDiv = UndefValue::get(VT);
 
     // FIXME: Doesn't do the right thing for cases where the vector is partially
     // constant. This works when the scalarizer pass is run first.
     for (unsigned I = 0, E = VT->getNumElements(); I != E; ++I) {
       Value *NumEltI = Builder.CreateExtractElement(Num, I);
       Value *DenEltI = Builder.CreateExtractElement(Den, I);
       Value *NewElt;
 
-      if (shouldKeepFDivF32(NumEltI, UnsafeDiv, HasFP32Denormals)) {
+      if (SafeFast && shouldKeepFDivF32(NumEltI, UnsafeDiv, HasFP32Denormals)) {
         NewElt = Builder.CreateFDiv(NumEltI, DenEltI);
       } else {
         NewElt = Builder.CreateCall(Decl, { NumEltI, DenEltI });
       }
 
       NewFDiv = Builder.CreateInsertElement(NewFDiv, NewElt, I);
     }
   } else {
-    if (!shouldKeepFDivF32(Num, UnsafeDiv, HasFP32Denormals))
+    if (!SafeFast || !shouldKeepFDivF32(Num, UnsafeDiv, HasFP32Denormals))
       NewFDiv = Builder.CreateCall(Decl, { Num, Den });
   }
 
   if (NewFDiv) {
     FDiv.replaceAllUsesWith(NewFDiv);
     NewFDiv->takeName(&FDiv);
     FDiv.eraseFromParent();
   }
 
   return !!NewFDiv;
 }
 
 static bool hasUnsafeFPMath(const Function &F) {
   Attribute Attr = F.getFnAttribute("unsafe-fp-math");
   return Attr.getValueAsString() == "true";
 }
 
+static bool hasCorrectlyRoundedDivideSqrt(const Function &F) {
+  Attribute Attr = F.getFnAttribute("correctly-rounded-divide-sqrt-fp-math");
+  return (Attr.getValueAsString() == "true");
+}
+
 static std::pair<Value*, Value*> getMul64(IRBuilder<> &Builder,
                                           Value *LHS, Value *RHS) {
   Type *I32Ty = Builder.getInt32Ty();
   Type *I64Ty = Builder.getInt64Ty();
 
   Value *LHS_EXT64 = Builder.CreateZExt(LHS, I64Ty);
   Value *RHS_EXT64 = Builder.CreateZExt(RHS, I64Ty);
   Value *MUL64 = Builder.CreateMul(LHS_EXT64, RHS_EXT64);
   Value *Lo = Builder.CreateTrunc(MUL64, I32Ty);
   Value *Hi = Builder.CreateLShr(MUL64, Builder.getInt64(32));
   Hi = Builder.CreateTrunc(Hi, I32Ty);
   return std::make_pair(Lo, Hi);
 }
 
 static Value* getMulHu(IRBuilder<> &Builder, Value *LHS, Value *RHS) {
   return getMul64(Builder, LHS, RHS).second;

   if (!TPC)
     return false;
 
   const AMDGPUTargetMachine &TM = TPC->getTM<AMDGPUTargetMachine>();
   ST = &TM.getSubtarget<GCNSubtarget>(F);
   AC = &getAnalysis<AssumptionCacheTracker>().getAssumptionCache(F);
   DA = &getAnalysis<LegacyDivergenceAnalysis>();
   HasUnsafeFPMath = hasUnsafeFPMath(F);
+  HasCorrectlyRoundedDivideSqrt = hasCorrectlyRoundedDivideSqrt(F);
   HasFP32Denormals = ST->hasFP32Denormals(F);
 
   bool MadeChange = false;
 
   for (BasicBlock &BB : F) {
     BasicBlock::iterator Next;
     for (BasicBlock::iterator I = BB.begin(), E = BB.end(); I != E; I = Next) {
       Next = std::next(I);

llvm/lib/Target/AMDGPU/SIISelLowering.h

   std::pair<SDValue, SDValue> splitBufferOffsets(SDValue Offset,
                                                  SelectionDAG &DAG) const;
 
   SDValue widenLoad(LoadSDNode *Ld, DAGCombinerInfo &DCI) const;
   SDValue LowerLOAD(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSELECT(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerFastUnsafeFDIV(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerFDIV_FAST(SDValue Op, SelectionDAG &DAG) const;
+  SDValue lowerFDIV_ROUNDED(SDValue Op, SelectionDAG &DAG) const;
+  SDValue lowerFDIV_SAFE(SDValue LHS, SDValue RHS, SelectionDAG &DAG) const;
   SDValue LowerFDIV16(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV32(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV64(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFDIV(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINT_TO_FP(SDValue Op, SelectionDAG &DAG, bool Signed) const;
   SDValue LowerSTORE(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerTrig(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerATOMIC_CMP_SWAP(SDValue Op, SelectionDAG &DAG) const;

llvm/lib/Target/AMDGPU/SIISelLowering.cpp

     if (!parseCachePolicy(Op.getOperand(3), DAG, &GLC, nullptr,
                           IsGFX10 ? &DLC : nullptr))
       return Op;
     return lowerSBuffer(VT, DL, Op.getOperand(1), Op.getOperand(2), GLC, DLC,
                         DAG);
   }
   case Intrinsic::amdgcn_fdiv_fast:
     return lowerFDIV_FAST(Op, DAG);
+  case Intrinsic::amdgcn_fdiv_rounded:
+    return lowerFDIV_ROUNDED(Op, DAG);
   case Intrinsic::amdgcn_interp_p1_f16: {
     SDValue ToM0 = DAG.getCopyToReg(DAG.getEntryNode(), DL, AMDGPU::M0,
                                     Op.getOperand(5), SDValue());
     if (getSubtarget()->getLDSBankCount() == 16) {
       // 16 bank LDS
 
       // FIXME: This implicitly will insert a second CopyToReg to M0.
       SDValue S = DAG.getNode(
         ISD::INTRINSIC_WO_CHAIN, DL, MVT::f32,
         DAG.getTargetConstant(Intrinsic::amdgcn_interp_mov, DL, MVT::i32),
         DAG.getConstant(2, DL, MVT::i32), // P0
         Op.getOperand(2),  // Attrchan
         Op.getOperand(3),  // Attr
         Op.getOperand(5)); // m0
 
       SDValue Ops[] = {
         Op.getOperand(1), // Src0
         Op.getOperand(2), // Attrchan
         Op.getOperand(3), // Attr
         DAG.getTargetConstant(0, DL, MVT::i32), // $src0_modifiers

   int DPDenormModeDefault = hasFP64FP16Denormals(DAG.getMachineFunction())
                                 ? FP_DENORM_FLUSH_NONE
                                 : FP_DENORM_FLUSH_IN_FLUSH_OUT;
 
   int Mode = SPDenormMode | (DPDenormModeDefault << 2);
   return DAG.getTargetConstant(Mode, SL, MVT::i32);
 }
 
-SDValue SITargetLowering::LowerFDIV32(SDValue Op, SelectionDAG &DAG) const {
-  if (SDValue FastLowered = lowerFastUnsafeFDIV(Op, DAG))
-    return FastLowered;
 
-  SDLoc SL(Op);
-  SDValue LHS = Op.getOperand(0);
-  SDValue RHS = Op.getOperand(1);
+SDValue SITargetLowering::lowerFDIV_SAFE(SDValue LHS, SDValue RHS, SelectionDAG &DAG) const {
+  SDLoc SL(RHS);
 
   const SDValue One = DAG.getConstantFP(1.0, SL, MVT::f32);
 
   SDVTList ScaleVT = DAG.getVTList(MVT::f32, MVT::i1);
 
   SDValue DenominatorScaled = DAG.getNode(AMDGPUISD::DIV_SCALE, SL, ScaleVT,
                                           RHS, RHS, LHS);
   SDValue NumeratorScaled = DAG.getNode(AMDGPUISD::DIV_SCALE, SL, ScaleVT,

 
   SDValue Scale = NumeratorScaled.getValue(1);
   SDValue Fmas = DAG.getNode(AMDGPUISD::DIV_FMAS, SL, MVT::f32,
                              Fma4, Fma1, Fma3, Scale);
 
   return DAG.getNode(AMDGPUISD::DIV_FIXUP, SL, MVT::f32, Fmas, RHS, LHS);
 }
 
+SDValue SITargetLowering::lowerFDIV_ROUNDED(SDValue Op, SelectionDAG &DAG) const {
+  SDValue LHS = Op.getOperand(1);
+  SDValue RHS = Op.getOperand(2);
+  return lowerFDIV_SAFE(LHS, RHS, DAG);
+}
+
+SDValue SITargetLowering::LowerFDIV32(SDValue Op, SelectionDAG &DAG) const {
+  if (SDValue FastLowered = lowerFastUnsafeFDIV(Op, DAG))
+    return FastLowered;
+
+  SDValue LHS = Op.getOperand(0);
+  SDValue RHS = Op.getOperand(1);
+  return lowerFDIV_SAFE(LHS, RHS, DAG);
+}
+
 SDValue SITargetLowering::LowerFDIV64(SDValue Op, SelectionDAG &DAG) const {
   if (DAG.getTarget().Options.UnsafeFPMath)
     return lowerFastUnsafeFDIV(Op, DAG);
 
   SDLoc SL(Op);
   SDValue X = Op.getOperand(0);
   SDValue Y = Op.getOperand(1);
 
   const SDValue One = DAG.getConstantFP(1.0, SL, MVT::f64);
 
   SDVTList ScaleVT = DAG.getVTList(MVT::f64, MVT::i1);
 
   SDValue DivScale0 = DAG.getNode(AMDGPUISD::DIV_SCALE, SL, ScaleVT, Y, Y, X);
 
   SDValue NegDivScale0 = DAG.getNode(ISD::FNEG, SL, MVT::f64, DivScale0);
 
   SDValue Rcp = DAG.getNode(AMDGPUISD::RCP, SL, MVT::f64, DivScale0);
 
   SDValue Fma0 = DAG.getNode(ISD::FMA, SL, MVT::f64, NegDivScale0, Rcp, One);
 

llvm/test/CodeGen/AMDGPU/amdgpu-codegenprepare-fdiv.ll

   store volatile float %fast.md.25ulp, float addrspace(1)* %out
 
   %arcp.md.25ulp = fdiv arcp float %a, %b, !fpmath !0
   store volatile float %arcp.md.25ulp, float addrspace(1)* %out
 
   ret void
 }
 
+; CHECK-LABEL: @fdiv_rounded(
+; CHECK: %no.md = call float @llvm.amdgcn.fdiv.rounded(float %a, float %b)
+define amdgpu_kernel void @fdiv_rounded(float addrspace(1)* %out, float %a, float %b) #3 {
+  %no.md = fdiv float %a, %b
+  store volatile float %no.md, float addrspace(1)* %out
+
+  ret void
+}
+
+; CHECK-LABEL: @fdiv_rounded_vector(
+; CHECK: %[[Anomd0:[0-9]+]] = extractelement <2 x float> %a, i64 0
+; CHECK: %[[Bnomd0:[0-9]+]] = extractelement <2 x float> %b, i64 0
+; CHECK: %[[FDIVnomd0:[0-9]+]] = call float @llvm.amdgcn.fdiv.rounded(float %[[Anomd0]], float %[[Bnomd0]])
+; CHECK: %[[INSnomd0:[0-9]+]] = insertelement <2 x float> undef, float %[[FDIVnomd0]], i64 0
+; CHECK: %[[Anomd1:[0-9]+]] = extractelement <2 x float> %a, i64 1
+; CHECK: %[[Bnomd1:[0-9]+]] = extractelement <2 x float> %b, i64 1
+; CHECK: %[[FDIVnomd1:[0-9]+]] = call float @llvm.amdgcn.fdiv.rounded(float %[[Anomd1]], float %[[Bnomd1]])
+; CHECK: %no.md = insertelement <2 x float> %[[INSnomd0]], float %[[FDIVnomd1]], i64 1
+define amdgpu_kernel void @fdiv_rounded_vector(<2 x float> addrspace(1)* %out, <2 x float> %a, <2 x float> %b) #3 {
+  %no.md = fdiv <2 x float> %a, %b
+  store volatile <2 x float> %no.md, <2 x float> addrspace(1)* %out
+
+  ret void
+}
+
+
 attributes #0 = { nounwind optnone noinline }
 attributes #1 = { nounwind }
 attributes #2 = { nounwind "target-features"="+fp32-denormals" }
+attributes #3 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="true" }
 
 ; CHECK: !0 = !{float 2.500000e+00}
 ; CHECK: !1 = !{float 5.000000e-01}
 ; CHECK: !2 = !{float 1.000000e+00}
 ; CHECK: !3 = !{float 3.000000e+00}
 
 !0 = !{float 2.500000e+00}
 !1 = !{float 5.000000e-01}
 !2 = !{float 1.000000e+00}
 !3 = !{float 3.000000e+00}

llvm/test/CodeGen/AMDGPU/fdiv.ll

 ; GCN: v_div_fixup_f32 v{{[0-9]+}}, [[FMAS]],
 define amdgpu_kernel void @fdiv_f32_denormals(float addrspace(1)* %out, float %a, float %b) #2 {
 entry:
   %fdiv = fdiv float %a, %b
   store float %fdiv, float addrspace(1)* %out
   ret void
 }
 
+; FUNC-LABEL: {{^}}fdiv_f32_correctly_rounded_divide_sqrt:
+
+; GCN: v_div_scale_f32 [[NUM_SCALE:v[0-9]+]]
+; GCN-DAG: v_div_scale_f32 [[DEN_SCALE:v[0-9]+]]
+; GCN-DAG: v_rcp_f32_e32 [[NUM_RCP:v[0-9]+]], [[NUM_SCALE]]
+
+; PREGFX10: s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 3
+; GFX10: s_denorm_mode 15
+; GCN: v_fma_f32 [[A:v[0-9]+]], -[[NUM_SCALE]], [[NUM_RCP]], 1.0
+; GCN: v_fma_f32 [[B:v[0-9]+]], [[A]], [[NUM_RCP]], [[NUM_RCP]]
+; GCN: v_mul_f32_e32 [[C:v[0-9]+]], [[DEN_SCALE]], [[B]]
+; GCN: v_fma_f32 [[D:v[0-9]+]], -[[NUM_SCALE]], [[C]], [[DEN_SCALE]]
+; GCN: v_fma_f32 [[E:v[0-9]+]], [[D]], [[B]], [[C]]
+; GCN: v_fma_f32 [[F:v[0-9]+]], -[[NUM_SCALE]], [[E]], [[DEN_SCALE]]
+; PREGFX10: s_setreg_imm32_b32 hwreg(HW_REG_MODE, 4, 2), 0
+; GFX10: s_denorm_mode 12
+; GCN: v_div_fmas_f32 [[FMAS:v[0-9]+]], [[F]], [[B]], [[E]]
+; GCN: v_div_fixup_f32 v{{[0-9]+}}, [[FMAS]],
+
+define amdgpu_kernel void @fdiv_f32_correctly_rounded_divide_sqrt(float addrspace(1)* %out, float %a) #3 {
+entry:
+  %fdiv = fdiv float 1.000000e+00, %a
+  store float %fdiv, float addrspace(1)* %out
+  ret void
+}
+
+
+; FUNC-LABEL: {{^}}fdiv_f32_denorms_correctly_rounded_divide_sqrt:
+
+; GCN: v_div_scale_f32 [[NUM_SCALE:v[0-9]+]]
+; GCN-DAG: v_rcp_f32_e32 [[NUM_RCP:v[0-9]+]], [[NUM_SCALE]]
+
+; PREGFX10-DAG: v_div_scale_f32 [[DEN_SCALE:v[0-9]+]]
+; PREGFX10-NOT: s_setreg
+; PREGFX10: v_fma_f32 [[A:v[0-9]+]], -[[NUM_SCALE]], [[NUM_RCP]], 1.0
+; PREGFX10: v_fma_f32 [[B:v[0-9]+]], [[A]], [[NUM_RCP]], [[NUM_RCP]]
+; PREGFX10: v_mul_f32_e32 [[C:v[0-9]+]], [[DEN_SCALE]], [[B]]
+; PREGFX10: v_fma_f32 [[D:v[0-9]+]], -[[NUM_SCALE]], [[C]], [[DEN_SCALE]]
+; PREGFX10: v_fma_f32 [[E:v[0-9]+]], [[D]], [[B]], [[C]]
+; PREGFX10: v_fma_f32 [[F:v[0-9]+]], -[[NUM_SCALE]], [[E]], [[DEN_SCALE]]
+; PREGFX10-NOT: s_setreg
+
+; GFX10-NOT: s_denorm_mode
+; GFX10: v_fma_f32 [[A:v[0-9]+]], -[[NUM_SCALE]], [[NUM_RCP]], 1.0
+; GFX10: v_fmac_f32_e32 [[B:v[0-9]+]], [[A]], [[NUM_RCP]]
+; GFX10: v_div_scale_f32 [[DEN_SCALE:v[0-9]+]]
+; GFX10: v_mul_f32_e32 [[C:v[0-9]+]], [[DEN_SCALE]], [[B]]
+; GFX10: v_fma_f32 [[D:v[0-9]+]], [[C]], -[[NUM_SCALE]], [[DEN_SCALE]]
+; GFX10: v_fmac_f32_e32 [[E:v[0-9]+]], [[D]], [[B]]
+; GFX10: v_fmac_f32_e64 [[F:v[0-9]+]], -[[NUM_SCALE]], [[E]]
+; GFX10-NOT: s_denorm_mode
+
+; GCN: v_div_fmas_f32 [[FMAS:v[0-9]+]], [[F]], [[B]], [[E]]
+; GCN: v_div_fixup_f32 v{{[0-9]+}}, [[FMAS]],
+define amdgpu_kernel void @fdiv_f32_denorms_correctly_rounded_divide_sqrt(float addrspace(1)* %out, float %a) #4 {
+entry:
+  %fdiv = fdiv float 1.000000e+00, %a
+  store float %fdiv, float addrspace(1)* %out
+  ret void
+}
+
+
 ; FUNC-LABEL: {{^}}fdiv_25ulp_f32:
 ; GCN: v_cndmask_b32
 ; GCN: v_mul_f32
 ; GCN: v_rcp_f32
 ; GCN: v_mul_f32
 ; GCN: v_mul_f32
 define amdgpu_kernel void @fdiv_25ulp_f32(float addrspace(1)* %out, float %a, float %b) #0 {
 entry:

   %result = fdiv arcp <4 x float> %a, %b
   store <4 x float> %result, <4 x float> addrspace(1)* %out
   ret void
 }
 
 attributes #0 = { nounwind "enable-unsafe-fp-math"="false" "target-features"="-fp32-denormals,+fp64-fp16-denormals,-flat-for-global" }
 attributes #1 = { nounwind "enable-unsafe-fp-math"="true" "target-features"="-fp32-denormals,-flat-for-global" }
 attributes #2 = { nounwind "enable-unsafe-fp-math"="false" "target-features"="+fp32-denormals,-flat-for-global" }
+attributes #3 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="true" "target-features"="-fp32-denormals,-flat-for-global" }
+attributes #4 = { nounwind "correctly-rounded-divide-sqrt-fp-math"="true" "target-features"="+fp32-denormals,-flat-for-global" }
+
 
 !0 = !{float 2.500000e+00}