[TLI] Rework interface for approximate (r)sqrt and reciprocal.

[jlebar]

Summary

Previously this had a major gotcha: The interface expected the return value of getSqrtEstimate to be a reciprocal sqrt (even if !Reciprocal), unless we were performing zero iterations of Newton's method, in which case it expected a regular sqrt.

Not only was this confusing, but it meant that basically none of our backends were correct if you passed "reciprocal-estimates"="sqrt:0".

This patch reworks the API to separate out three questions which previously were all squished into getSqrtEstimate:

• Should we do an estimate at all?
• If so, how many iterations of Newton's method should we perform to refine our estimate?
• What's our first approximation of the rsqrt?

Now we have one function which answers the first two questions, and a separate function which answers the third.

We also now have a separate function which we call in the special case when we're doing zero iterations of Newton's method and calculating a plain sqrt, and NVPTX hooks into this.

We made the same changes to the reciprocal estimate functions.

In addition, we reworked the parsing of the reciprocal-estimates metadata flag, which needed some love, and otherwise wasn't compatible with this new scheme.

llvm/include/llvm/Target/TargetLowering.h

 ///
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TARGET_TARGETLOWERING_H
 #define LLVM_TARGET_TARGETLOWERING_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
-#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/CodeGen/DAGCombine.h"
 #include "llvm/CodeGen/ISDOpcodes.h"
 #include "llvm/CodeGen/MachineValueType.h"
 #include "llvm/CodeGen/RuntimeLibcalls.h"
 #include "llvm/CodeGen/SelectionDAGNodes.h"
 #include "llvm/CodeGen/ValueTypes.h"
 #include "llvm/IR/Attributes.h"
public:
   }
 
   /// Return true if SQRT(X) shouldn't be replaced with X*RSQRT(X).
   virtual bool isFsqrtCheap(SDValue X, SelectionDAG &DAG) const {
     // Default behavior is to replace SQRT(X) with X*RSQRT(X).
     return false;
   }
 
-  /// Reciprocal estimate status values used by the functions below.
-  enum ReciprocalEstimate : int {
-    Unspecified = -1,
-    Disabled = 0,
-    Enabled = 1
+  /// Default number of steps of Newton's method to perform when refining an
+  /// approximate (r)sqrt.  Overridden by the reciprocal-estimates attr.
+  ///
+  /// There are actually two pieces of information we need to provide here:
+  ///
+  ///  - Whether to emit an approximate (r)sqrt at all if reciprocal-estimates
+  ///    says nothing about the op in question (e.g. it's empty or equal to
+  ///    "default").
+  ///
+  ///  - The default number of steps of Newton's method to run when the
+  ///    reciprocal-estimates attribute doesn't give an explicit number of steps
+  ///    (e.g. it's "sqrt", rather than "sqrt:2").
+  ///
+  /// If you want to enable approximate (r)sqrts on your target, you'll need to
+  /// overload both getSqrtRefinementDefaults and getRsqrtEstimate.
+  ///
+  /// The recip version of this function is analogous, for the operation 1/x.
+  struct RefinementDefaults {
+    bool EnabledByDefault;
+    unsigned DefaultSteps;
   };
+  virtual RefinementDefaults getSqrtRefinementDefaults(EVT) const {
+    return {/*EnabledByDefault=*/false, /*DefaultSteps=*/0};
+  }
+  virtual RefinementDefaults getRecipRefinementDefaults(EVT) const {
+    return {/*EnabledByDefault=*/false, /*DefaultSteps=*/0};
+  }
+
+  /// If true, we use a Newton-Raphson algorithm with a single constant to
+  /// refine sqrt estimates.  Otherwise, we use an algorithm with two
+  /// constants.
+  virtual bool useOneConstNRForSqrtEstimate(EVT) const { return false; }
+
+  /// Get the number of refinement steps we should perform on an approximate
+  /// (r)sqrt operation with the given type and in the given function.
+  ///
+  /// A target has a default number of (r)sqrt refinement steps, returned by
+  /// getDefaultSqrtRefinementSteps(), and a function can override this with the
+  /// reciprocal-estimates attr.
+  ///
+  /// If this returns None, we don't use an approximate (r)sqrt here.
+  ///
+  /// The recip version of this function is analogous, for the operation 1/x.
+  virtual Optional<unsigned>
+  getSqrtEstimateRefinementSteps(EVT VT, const MachineFunction &MF) const;
+
+  virtual Optional<unsigned>
+  getRecipEstimateRefinementSteps(EVT VT, const MachineFunction &MF) const;
+
+  /// Get a first approximate to the reciprocal square root of Operand. This
+  /// will be refined by getSqrtEstimateRefinementSteps() steps of Newton's
+  /// method.
+  virtual SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const {
+    return SDValue();
+  }
 
-  /// Return a ReciprocalEstimate enum value for a square root of the given type
-  /// based on the function's attributes. If the operation is not overridden by
-  /// the function's attributes, "Unspecified" is returned and target defaults
-  /// are expected to be used for instruction selection.
-  int getRecipEstimateSqrtEnabled(EVT VT, MachineFunction &MF) const;
-
-  /// Return a ReciprocalEstimate enum value for a division of the given type
-  /// based on the function's attributes. If the operation is not overridden by
-  /// the function's attributes, "Unspecified" is returned and target defaults
-  /// are expected to be used for instruction selection.
-  int getRecipEstimateDivEnabled(EVT VT, MachineFunction &MF) const;
-
-  /// Return the refinement step count for a square root of the given type based
-  /// on the function's attributes. If the operation is not overridden by
-  /// the function's attributes, "Unspecified" is returned and target defaults
-  /// are expected to be used for instruction selection.
-  int getSqrtRefinementSteps(EVT VT, MachineFunction &MF) const;
-
-  /// Return the refinement step count for a division of the given type based
-  /// on the function's attributes. If the operation is not overridden by
-  /// the function's attributes, "Unspecified" is returned and target defaults
-  /// are expected to be used for instruction selection.
-  int getDivRefinementSteps(EVT VT, MachineFunction &MF) const;
+  /// Get a first approximation of the square root of Operand.
+  ///
+  /// This is never fed into Newton's method -- for that, we always use the
+  /// rsqrt estimate.  It's only used when getSqrtEstimateRefinementSteps() is
+  /// 0.  If this returns an empty SDValue, callers should try to compute an
+  /// approximate sqrt using getRsqrtEstimate().
+  virtual SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG) const {
+    return SDValue();
+  }
+
+  /// Get a first approximation of the reciprocal of Operand.  This will be
+  /// refined by getRecipEstimateRefinementSteps() iterations of Newton's
+  /// method.
+  virtual SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG) const {
+    return SDValue();
+  }
 
   /// Returns true if target has indicated at least one type should be bypassed.
   bool isSlowDivBypassed() const { return !BypassSlowDivWidths.empty(); }
 
   /// Returns map of slow types for division or remainder with corresponding
   /// fast types
   const DenseMap<unsigned int, unsigned int> &getBypassSlowDivWidths() const {
     return BypassSlowDivWidths;
public:
   /// Indicate whether this target prefers to combine FDIVs with the same
   /// divisor. If the transform should never be done, return zero. If the
   /// transform should be done, return the minimum number of divisor uses
   /// that must exist.
   virtual unsigned combineRepeatedFPDivisors() const {
     return 0;
   }
 
-  /// Hooks for building estimates in place of slower divisions and square
-  /// roots.
-
-  /// Return either a square root or its reciprocal estimate value for the input
-  /// operand.
-  /// \p Enabled is a ReciprocalEstimate enum with value either 'Unspecified' or
-  /// 'Enabled' as set by a potential default override attribute.
-  /// If \p RefinementSteps is 'Unspecified', the number of Newton-Raphson
-  /// refinement iterations required to generate a sufficient (though not
-  /// necessarily IEEE-754 compliant) estimate is returned in that parameter.
-  /// The boolean UseOneConstNR output is used to select a Newton-Raphson
-  /// algorithm implementation that uses either one or two constants.
-  /// The boolean Reciprocal is used to select whether the estimate is for the
-  /// square root of the input operand or the reciprocal of its square root.
-  /// A target may choose to implement its own refinement within this function.
-  /// If that's true, then return '0' as the number of RefinementSteps to avoid
-  /// any further refinement of the estimate.
-  /// An empty SDValue return means no estimate sequence can be created.
-  virtual SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG,
-                                  int Enabled, int &RefinementSteps,
-                                  bool &UseOneConstNR, bool Reciprocal) const {
-    return SDValue();
-  }
-
-  /// Return a reciprocal estimate value for the input operand.
-  /// \p Enabled is a ReciprocalEstimate enum with value either 'Unspecified' or
-  /// 'Enabled' as set by a potential default override attribute.
-  /// If \p RefinementSteps is 'Unspecified', the number of Newton-Raphson
-  /// refinement iterations required to generate a sufficient (though not
-  /// necessarily IEEE-754 compliant) estimate is returned in that parameter.
-  /// A target may choose to implement its own refinement within this function.
-  /// If that's true, then return '0' as the number of RefinementSteps to avoid
-  /// any further refinement of the estimate.
-  /// An empty SDValue return means no estimate sequence can be created.
-  virtual SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG,
-                                   int Enabled, int &RefinementSteps) const {
-    return SDValue();
-  }
-
   //===--------------------------------------------------------------------===//
   // Legalization utility functions
   //
 
   /// Expand a MUL or [US]MUL_LOHI of n-bit values into two or four nodes,
   /// respectively, each computing an n/2-bit part of the result.
   /// \param Result A vector that will be filled with the parts of the result
   ///        in little-endian order.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::BuildReciprocalEstimate(SDValue Op, SDNodeFlags *Flags) {
 
   // TODO: Handle half and/or extended types?
   EVT VT = Op.getValueType();
   if (VT.getScalarType() != MVT::f32 && VT.getScalarType() != MVT::f64)
     return SDValue();
 
   // If estimates are explicitly disabled for this function, we're done.
   MachineFunction &MF = DAG.getMachineFunction();
-  int Enabled = TLI.getRecipEstimateDivEnabled(VT, MF);
-  if (Enabled == TLI.ReciprocalEstimate::Disabled)
+  Optional<unsigned> Iterations = TLI.getRecipEstimateRefinementSteps(VT, MF);
+  if (!Iterations)
     return SDValue();
 
-  // Estimates may be explicitly enabled for this type with a custom number of
-  // refinement steps.
-  int Iterations = TLI.getDivRefinementSteps(VT, MF);
-  if (SDValue Est = TLI.getRecipEstimate(Op, DAG, Enabled, Iterations)) {
+  if (SDValue Est = TLI.getRecipEstimate(Op, DAG)) {
     AddToWorklist(Est.getNode());
 
     if (Iterations) {
       EVT VT = Op.getValueType();
       SDLoc DL(Op);
       SDValue FPOne = DAG.getConstantFP(1.0, DL, VT);
 
       // Newton iterations: Est = Est + Est (1 - Arg * Est)
-      for (int i = 0; i < Iterations; ++i) {
+      for (unsigned i = 0; i < *Iterations; ++i) {
         SDValue NewEst = DAG.getNode(ISD::FMUL, DL, VT, Op, Est, Flags);
         AddToWorklist(NewEst.getNode());
 
         NewEst = DAG.getNode(ISD::FSUB, DL, VT, FPOne, NewEst, Flags);
         AddToWorklist(NewEst.getNode());
 
         NewEst = DAG.getNode(ISD::FMUL, DL, VT, Est, NewEst, Flags);
         AddToWorklist(NewEst.getNode());
SDValue DAGCombiner::buildSqrtEstimateImpl(SDValue Op, SDNodeFlags *Flags,
 
   // TODO: Handle half and/or extended types?
   EVT VT = Op.getValueType();
   if (VT.getScalarType() != MVT::f32 && VT.getScalarType() != MVT::f64)
     return SDValue();
 
   // If estimates are explicitly disabled for this function, we're done.
   MachineFunction &MF = DAG.getMachineFunction();
-  int Enabled = TLI.getRecipEstimateSqrtEnabled(VT, MF);
-  if (Enabled == TLI.ReciprocalEstimate::Disabled)
+  Optional<unsigned> Iterations = TLI.getSqrtEstimateRefinementSteps(VT, MF);
+  if (!Iterations)
     return SDValue();
 
-  // Estimates may be explicitly enabled for this type with a custom number of
-  // refinement steps.
-  int Iterations = TLI.getSqrtRefinementSteps(VT, MF);
-
-  bool UseOneConstNR = false;
-  if (SDValue Est =
-      TLI.getSqrtEstimate(Op, DAG, Enabled, Iterations, UseOneConstNR,
-                          Reciprocal)) {
+  // Special case: No iterations of Newton's method, and computing plain sqrt.
+  // In this case, return TLI.getSqrtEstimate if possible.  If not, we'll
+  // compute the sqrt from the rsqrt estimate.
+  if (*Iterations == 0 && !Reciprocal) {
+    if (SDValue Est = TLI.getSqrtEstimate(Op, DAG)) {
       AddToWorklist(Est.getNode());
+      return Est;
+    }
+  }
 
-    if (Iterations) {
-      Est = UseOneConstNR
-            ? buildSqrtNROneConst(Op, Est, Iterations, Flags, Reciprocal)
-            : buildSqrtNRTwoConst(Op, Est, Iterations, Flags, Reciprocal);
+  if (SDValue Est = TLI.getRsqrtEstimate(Op, DAG)) {
+    AddToWorklist(Est.getNode());
+
+    if (*Iterations > 0)
+      Est = TLI.useOneConstNRForSqrtEstimate(VT)
+                ? buildSqrtNROneConst(Op, Est, *Iterations, Flags, Reciprocal)
+                : buildSqrtNRTwoConst(Op, Est, *Iterations, Flags, Reciprocal);
 
-      if (!Reciprocal) {
-        // Unfortunately, Est is now NaN if the input was exactly 0.0.
-        // Select out this case and force the answer to 0.0.
-        EVT VT = Op.getValueType();
-        SDLoc DL(Op);
 
-        SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
-        EVT CCVT = getSetCCResultType(VT);
-        SDValue ZeroCmp = DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
-        AddToWorklist(ZeroCmp.getNode());
 
-        Est = DAG.getNode(VT.isVector() ? ISD::VSELECT : ISD::SELECT, DL, VT,
-                          ZeroCmp, FPZero, Est);
-        AddToWorklist(Est.getNode());
-      }
-    }
     return Est;
   }
 
   return SDValue();
 }
 
 SDValue DAGCombiner::buildRsqrtEstimate(SDValue Op, SDNodeFlags *Flags) {
   return buildSqrtEstimateImpl(Op, Flags, true);

llvm/lib/CodeGen/TargetLoweringBase.cpp

 void TargetLoweringBase::setMaximumJumpTableSize(unsigned Val) {
   MaximumJumpTableSize = Val;
 }
 
 //===----------------------------------------------------------------------===//
 //  Reciprocal Estimates
 //===----------------------------------------------------------------------===//
 
-/// Get the reciprocal estimate attribute string for a function that will
-/// override the target defaults.
-static StringRef getRecipEstimateForFunc(MachineFunction &MF) {
-  const Function *F = MF.getFunction();
-  return F->getFnAttribute("reciprocal-estimates").getValueAsString();
-}
-
-/// Construct a string for the given reciprocal operation of the given type.
-/// This string should match the corresponding option to the front-end's
-/// "-mrecip" flag assuming those strings have been passed through in an
-/// attribute string. For example, "vec-divf" for a division of a vXf32.
-static std::string getReciprocalOpName(bool IsSqrt, EVT VT) {
-  std::string Name = VT.isVector() ? "vec-" : "";
-
-  Name += IsSqrt ? "sqrt" : "div";
-
-  // TODO: Handle "half" or other float types?
-  if (VT.getScalarType() == MVT::f64) {
-    Name += "d";
-  } else {
-    assert(VT.getScalarType() == MVT::f32 &&
-           "Unexpected FP type for reciprocal estimate");
-    Name += "f";
-  }
-
-  return Name;
-}
-
-/// Return the character position and value (a single numeric character) of a
-/// customized refinement operation in the input string if it exists. Return
-/// false if there is no customized refinement step count.
-static bool parseRefinementStep(StringRef In, size_t &Position,
-                                uint8_t &Value) {
-  const char RefStepToken = ':';
-  Position = In.find(RefStepToken);
-  if (Position == StringRef::npos)
-    return false;
+/// Gets the number of refinement steps we should perform on an approx (r)sqrt
+/// or div of the given type.
+///
+/// Returns None if we shouldn't perform an approx (r)sqrt/div at all.
+static Optional<unsigned>
+getOpRefinementStepsFromAttr(bool IsSqrt, EVT VT, const MachineFunction &MF,
+                             TargetLowering::RefinementDefaults Defaults) {
+  StringRef Attr = MF.getFunction()
+                       ->getFnAttribute("reciprocal-estimates")
+                       .getValueAsString();
 
-  StringRef RefStepString = In.substr(Position + 1);
-  // Allow exactly one numeric character for the additional refinement
-  // step parameter.
-  if (RefStepString.size() == 1) {
-    char RefStepChar = RefStepString[0];
-    if (RefStepChar >= '0' && RefStepChar <= '9') {
-      Value = RefStepChar - '0';
-      return true;
-    }
+  // Attr is a comma-separated list of tokens.  Valid tokens are of the form
+  // "fn", "fn:N", "!fn", or "!fn:N", where N is an integer and "fn" is one of:
+  //
+  //  - none, all, default
+  //  - div, divf, divd
+  //  - vec-div, vec-divf, vec-divd,
+  //  - sqrt, sqrtf, sqrtd
+  //  - vec-sqrt, vec-sqrtf, vec-sqrtd
+  //
+
+  SmallString<16> UntypedOpName;
+  UntypedOpName += VT.isVector() ? "vec-" : "";
+  UntypedOpName += IsSqrt ? "sqrt" : "div";
+
+  SmallString<16> TypedOpName = UntypedOpName;
+  if (VT.getScalarType() == MVT::f64)
+    TypedOpName += "d";
+  else if (VT.getScalarType() == MVT::f32)
+    TypedOpName += "f";
+  else
+    llvm_unreachable("Unexpected type; not f64 or f32.");
+
+  Optional<unsigned> Steps;
+  if (Defaults.EnabledByDefault)
+    Steps = Defaults.DefaultSteps;
+
+  SmallVector<StringRef, 4> SplitAttr;
+  SplitString(Attr, SplitAttr, ",");
+  for (StringRef Tok : SplitAttr) {
+    StringRef OpName, OpStepsStr;
+    std::tie(OpName, OpStepsStr) = Tok.split(':');
+
+    if (OpName == "none") {
+      Steps = None;
+      continue;
     }
-  report_fatal_error("Invalid refinement step for -recip.");
+
+    if (OpName.startswith("!")) {
+      OpName = OpName.drop_front(1);
+      if (OpName == UntypedOpName || OpName == TypedOpName)
+        Steps = None;
+      continue;
     }
 
-/// For the input attribute string, return one of the ReciprocalEstimate enum
-/// status values (enabled, disabled, or not specified) for this operation on
-/// the specified data type.
-static int getOpEnabled(bool IsSqrt, EVT VT, StringRef Override) {
-  if (Override.empty())
-    return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-
-  SmallVector<StringRef, 4> OverrideVector;
-  SplitString(Override, OverrideVector, ",");
-  unsigned NumArgs = OverrideVector.size();
-
-  // Check if "all", "none", or "default" was specified.
-  if (NumArgs == 1) {
-    // Look for an optional setting of the number of refinement steps needed
-    // for this type of reciprocal operation.
-    size_t RefPos;
-    uint8_t RefSteps;
-    if (parseRefinementStep(Override, RefPos, RefSteps)) {
-      // Split the string for further processing.
-      Override = Override.substr(0, RefPos);
-    }
-
-    // All reciprocal types are enabled.
-    if (Override == "all")
-      return TargetLoweringBase::ReciprocalEstimate::Enabled;
-
-    // All reciprocal types are disabled.
-    if (Override == "none")
-      return TargetLoweringBase::ReciprocalEstimate::Disabled;
-
-    // Target defaults for enablement are used.
-    if (Override == "default")
-      return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-  }
-
-  // The attribute string may omit the size suffix ('f'/'d').
-  std::string VTName = getReciprocalOpName(IsSqrt, VT);
-  std::string VTNameNoSize = VTName;
-  VTNameNoSize.pop_back();
-  static const char DisabledPrefix = '!';
-
-  for (StringRef RecipType : OverrideVector) {
-    size_t RefPos;
-    uint8_t RefSteps;
-    if (parseRefinementStep(RecipType, RefPos, RefSteps))
-      RecipType = RecipType.substr(0, RefPos);
-
-    // Ignore the disablement token for string matching.
-    bool IsDisabled = RecipType[0] == DisabledPrefix;
-    if (IsDisabled)
-      RecipType = RecipType.substr(1);
-
-    if (RecipType.equals(VTName) || RecipType.equals(VTNameNoSize))
-      return IsDisabled ? TargetLoweringBase::ReciprocalEstimate::Disabled
-                        : TargetLoweringBase::ReciprocalEstimate::Enabled;
-  }
-
-  return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-}
-
-/// For the input attribute string, return the customized refinement step count
-/// for this operation on the specified data type. If the step count does not
-/// exist, return the ReciprocalEstimate enum value for unspecified.
-static int getOpRefinementSteps(bool IsSqrt, EVT VT, StringRef Override) {
-  if (Override.empty())
-    return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-
-  SmallVector<StringRef, 4> OverrideVector;
-  SplitString(Override, OverrideVector, ",");
-  unsigned NumArgs = OverrideVector.size();
-
-  // Check if "all", "default", or "none" was specified.
-  if (NumArgs == 1) {
-    // Look for an optional setting of the number of refinement steps needed
-    // for this type of reciprocal operation.
-    size_t RefPos;
-    uint8_t RefSteps;
-    if (!parseRefinementStep(Override, RefPos, RefSteps))
-      return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-
-    // Split the string for further processing.
-    Override = Override.substr(0, RefPos);
-    assert(Override != "none" &&
-           "Disabled reciprocals, but specifed refinement steps?");
-
-    // If this is a general override, return the specified number of steps.
-    if (Override == "all" || Override == "default")
-      return RefSteps;
-  }
-
-  // The attribute string may omit the size suffix ('f'/'d').
-  std::string VTName = getReciprocalOpName(IsSqrt, VT);
-  std::string VTNameNoSize = VTName;
-  VTNameNoSize.pop_back();
-
-  for (StringRef RecipType : OverrideVector) {
-    size_t RefPos;
-    uint8_t RefSteps;
-    if (!parseRefinementStep(RecipType, RefPos, RefSteps))
+    if (OpName != "all" && OpName != "default" && OpName != UntypedOpName &&
+        OpName != TypedOpName)
       continue;
 
-    RecipType = RecipType.substr(0, RefPos);
-    if (RecipType.equals(VTName) || RecipType.equals(VTNameNoSize))
-      return RefSteps;
+    Optional<unsigned> OpSteps;
+    if (!OpStepsStr.empty()) {
+      unsigned ParsedSteps;
+      if (OpStepsStr.getAsInteger(/*Radix=*/10, ParsedSteps))
+        continue;
+      OpSteps = ParsedSteps;
     }
 
-  return TargetLoweringBase::ReciprocalEstimate::Unspecified;
-}
+    // Apply target's default number of steps if the token didn't contain a
+    // number of steps.
+    if (!OpSteps)
+      OpSteps = Defaults.DefaultSteps;
 
-int TargetLoweringBase::getRecipEstimateSqrtEnabled(EVT VT,
-                                                    MachineFunction &MF) const {
-  return getOpEnabled(true, VT, getRecipEstimateForFunc(MF));
+    Steps = OpSteps;
   }
 
-int TargetLoweringBase::getRecipEstimateDivEnabled(EVT VT,
-                                                   MachineFunction &MF) const {
-  return getOpEnabled(false, VT, getRecipEstimateForFunc(MF));
+  return Steps;
 }
 
-int TargetLoweringBase::getSqrtRefinementSteps(EVT VT,
-                                               MachineFunction &MF) const {
-  return getOpRefinementSteps(true, VT, getRecipEstimateForFunc(MF));
+Optional<unsigned> TargetLoweringBase::getSqrtEstimateRefinementSteps(
+    EVT VT, const MachineFunction &MF) const {
+  return getOpRefinementStepsFromAttr(/*IsSqrt=*/true, VT, MF,
+                                      getSqrtRefinementDefaults(VT));
 }
 
-int TargetLoweringBase::getDivRefinementSteps(EVT VT,
-                                              MachineFunction &MF) const {
-  return getOpRefinementSteps(false, VT, getRecipEstimateForFunc(MF));
+Optional<unsigned> TargetLoweringBase::getRecipEstimateRefinementSteps(
+    EVT VT, const MachineFunction &MF) const {
+  return getOpRefinementStepsFromAttr(/*IsSqrt=*/false, VT, MF,
+                                      getRecipRefinementDefaults(VT));
 }

llvm/lib/Target/AArch64/AArch64ISelLowering.h

public:
   void insertCopiesSplitCSR(
       MachineBasicBlock *Entry,
       const SmallVectorImpl<MachineBasicBlock *> &Exits) const override;
 
   bool supportSwiftError() const override {
     return true;
   }
 
+  Optional<unsigned>
+  getSqrtEstimateRefinementSteps(EVT, const MachineFunction &) const override;
+  Optional<unsigned>
+  getRecipEstimateRefinementSteps(EVT, const MachineFunction &) const override;
+
 private:
   bool isExtFreeImpl(const Instruction *Ext) const override;
 
   /// Keep a pointer to the AArch64Subtarget around so that we can
   /// make the right decision when generating code for different targets.
   const AArch64Subtarget *Subtarget;
 
   void addTypeForNEON(MVT VT, MVT PromotedBitwiseVT);
private:
   SDValue LowerINT_TO_FP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorAND(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCONCAT_VECTORS(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFSINCOS(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                         std::vector<SDNode *> *Created) const override;
-  SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                          int &ExtraSteps, bool &UseOneConst,
-                          bool Reciprocal) const override;
-  SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                           int &ExtraSteps) const override;
+
+  RefinementDefaults getSqrtRefinementDefaults(EVT) const override;
+  RefinementDefaults getRecipRefinementDefaults(EVT) const override;
+
+  SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+  SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+  SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG) const override;
   unsigned combineRepeatedFPDivisors() const override;
 
   ConstraintType getConstraintType(StringRef Constraint) const override;
   unsigned getRegisterByName(const char* RegName, EVT VT,
                              SelectionDAG &DAG) const override;
 
   /// Examine constraint string and operand type and determine a weight value.
   /// The operand object must already have been set up with the operand type.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

else if (VT == MVT::f32)
     return AArch64_AM::getFP32Imm(Imm) != -1;
   return false;
 }
 
 //===----------------------------------------------------------------------===//
 //                          AArch64 Optimization Hooks
 //===----------------------------------------------------------------------===//
 
-static SDValue getEstimate(const AArch64Subtarget *ST, unsigned Opcode,
-                           SDValue Operand, SelectionDAG &DAG,
-                           int &ExtraSteps) {
-  EVT VT = Operand.getValueType();
-  if (ST->hasNEON() &&
-      (VT == MVT::f64 || VT == MVT::v1f64 || VT == MVT::v2f64 ||
-       VT == MVT::f32 || VT == MVT::v1f32 ||
-       VT == MVT::v2f32 || VT == MVT::v4f32)) {
-    if (ExtraSteps == TargetLoweringBase::ReciprocalEstimate::Unspecified)
-      // For the reciprocal estimates, convergence is quadratic, so the number
-      // of digits is doubled after each iteration.  In ARMv8, the accuracy of
-      // the initial estimate is 2^-8.  Thus the number of extra steps to refine
-      // the result for float (23 mantissa bits) is 2 and for double (52
-      // mantissa bits) is 3.
-      ExtraSteps = VT == MVT::f64 ? 3 : 2;
+// We do our own custom (r)sqrt and recip refinement instead of relying on the
+// generic logic in DAGCombine, so we have to abuse the TLI interface a bit.
+//
+// The return value of getSqrtEstimateRefinementSteps is essentially a bool --
+// it's always either 0 or None.  This controls whether approx (r)sqrts are
+// enabled at all.  If they're enabled (getSqrtEstimateRefinementSteps returns
+// 0), we end up in getRsqrtEstimate, which then calls
+// TargetLowering::getSqrtEstimateRefinementSteps to find out how many
+// refinement steps it *actually* should do.
+//
+// If we didn't override getSqrtEstimateRefinementSteps to return 0, then when
+// presented with e.g. "reciprocal-estimates"="sqrt:3", we'd do three custom
+// refinements of the sqrt here, followed by *another* three refinements in
+// DAGCombine.
+//
+// TargetLowering::getSqrtEstimateRefinementSteps may call back into
+// getDefaultSqrtRefinementSteps, if e.g. you specify
+// "reciprocal-estimates"="sqrt" (meaning, "turn on reciprocal estimates for
+// sqrts, using whatever number of refinements you want").
+//
+// The recip versions of these functions are analogous for the op 1/x.
 
-    return DAG.getNode(Opcode, SDLoc(Operand), VT, Operand);
-  }
+static bool canDoApproxSqrtOrRecipOnType(EVT VT) {
+  return VT == MVT::f64 || VT == MVT::v1f64 || VT == MVT::v2f64 ||
+         VT == MVT::f32 || VT == MVT::v1f32 || VT == MVT::v2f32 ||
+         VT == MVT::v4f32;
 
-  return SDValue();
+static Optional<unsigned> getRefinementStepsForType(EVT VT) {
+  if (canDoApproxSqrtOrRecipOnType(VT)) {
+    // For the reciprocal estimates, convergence is quadratic, so the number of
+    // digits is doubled after each iteration.  In ARMv8, the accuracy of the
+    // initial estimate is 2^-8.  Thus the number of extra steps to refine the
+    // result for float (23 mantissa bits) is 2 and for double (52 mantissa
+    // bits) is 3.
+    //
+    // FIXME: Do we mean VT.getScalarType() == MVT::f64?
+    return VT == MVT::f64 ? 3 : 2;
+  }
+  return None;
 }
 
-SDValue AArch64TargetLowering::getSqrtEstimate(SDValue Operand,
-                                               SelectionDAG &DAG, int Enabled,
-                                               int &ExtraSteps,
-                                               bool &UseOneConst,
-                                               bool Reciprocal) const {
-  if (Enabled == ReciprocalEstimate::Enabled ||
-      (Enabled == ReciprocalEstimate::Unspecified && Subtarget->useRSqrt()))
-    if (SDValue Estimate = getEstimate(Subtarget, AArch64ISD::FRSQRTE, Operand,
-                                       DAG, ExtraSteps)) {
-      SDLoc DL(Operand);
-      EVT VT = Operand.getValueType();
+TargetLowering::RefinementDefaults
+AArch64TargetLowering::getSqrtRefinementDefaults(EVT VT) const {
+  Optional<unsigned> Steps = getRefinementStepsForType(VT);
+  if (!Steps) {
+    // The value we choose for DefaultSteps doesn't matter here, because
+    // getRsqrtEstimate will return SDValue() if getRefinementStepsForType()
+    // returns None.
+    return {/*EnabledByDefault=*/false, /*DefaultSteps=*/0};
+  }
+
+  return {/*EnabledByDefault=*/Subtarget->useRSqrt(), /*DefaultSteps=*/*Steps};
+}
+
+TargetLowering::RefinementDefaults
+AArch64TargetLowering::getRecipRefinementDefaults(EVT VT) const {
+  Optional<unsigned> Steps = getRefinementStepsForType(VT);
+  if (!Steps) {
+    // The value we choose for DefaultSteps doesn't matter here, because
+    // getRecipEstimate will return SDValue() if getRefinementStepsForType()
+    // returns None.
+    return {/*EnabledByDefault=*/false, /*DefaultSteps=*/0};
+  }
+
+  // Div estimates are disabled unless explicitly asked for via the
+  // reciprocal-estimates function attribute.
+  return {/*EnabledByDefault=*/false, /*DefaultSteps=*/*Steps};
+}
+
+Optional<unsigned> AArch64TargetLowering::getSqrtEstimateRefinementSteps(
+    EVT VT, const MachineFunction &MF) const {
+  Optional<unsigned> Steps =
+      TargetLowering::getSqrtEstimateRefinementSteps(VT, MF);
+  if (Steps)
+    return 0;
+  return None;
+}
+
+Optional<unsigned> AArch64TargetLowering::getRecipEstimateRefinementSteps(
+    EVT VT, const MachineFunction &MF) const {
+  Optional<unsigned> Steps =
+      TargetLowering::getRecipEstimateRefinementSteps(VT, MF);
+  if (Steps)
+    return 0;
+  return None;
+}
+
+SDValue AArch64TargetLowering::getRsqrtEstimate(SDValue Operand,
+                                                SelectionDAG &DAG) const {
+  if (!Subtarget->hasNEON() || !canDoApproxSqrtOrRecipOnType(VT))
+    return SDValue();
+
+  Optional<unsigned> RefinementSteps =
+      Subtarget->getTargetLowering()
+          ->TargetLowering::getSqrtEstimateRefinementSteps(
+              VT, DAG.getMachineFunction());
+  if (!RefinementSteps)
+    return SDValue();
+
+  SDLoc DL(Operand);
+  SDValue Estimate = DAG.getNode(AArch64ISD::FRSQRTE, DL, VT, Operand);
 
-      SDNodeFlags Flags;
-      Flags.setUnsafeAlgebra(true);
 
-      // Newton reciprocal square root iteration: E * 0.5 * (3 - X * E^2)
-      // AArch64 reciprocal square root iteration instruction: 0.5 * (3 - M * N)
-      for (int i = ExtraSteps; i > 0; --i) {
-        SDValue Step = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Estimate,
-                                   &Flags);
-        Step = DAG.getNode(AArch64ISD::FRSQRTS, DL, VT, Operand, Step, &Flags);
-        Estimate = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Step, &Flags);
-      }
+  for (int i = *RefinementSteps; i > 0; --i) {
+    SDValue Step = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Estimate, &Flags);
+  return Estimate;
+}
+
+SDValue AArch64TargetLowering::getSqrtEstimate(SDValue Operand,
+                                               SelectionDAG &DAG) const {
+  SDValue Estimate = getRsqrtEstimate(Operand, DAG);
+  if (!Estimate)
+    return SDValue();
+
+  SDLoc DL(Operand);
+  EVT VT = Operand.getValueType();
+
+  SDNodeFlags Flags;
+  Flags.setUnsafeAlgebra(true);
 
-      if (!Reciprocal) {
-        EVT CCVT = getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(),
-                                      VT);
-        SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
-        SDValue Eq = DAG.getSetCC(DL, CCVT, Operand, FPZero, ISD::SETEQ);
+  EVT CCVT = getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
 
-        Estimate = DAG.getNode(ISD::FMUL, DL, VT, Operand, Estimate, &Flags);
-        // Correct the result if the operand is 0.0.
-        Estimate = DAG.getNode(VT.isVector() ? ISD::VSELECT : ISD::SELECT, DL,
-                               VT, Eq, Operand, Estimate);
-      }
-
-      ExtraSteps = 0;
-      return Estimate;
-    }
+  Estimate = DAG.getNode(VT.isVector() ? ISD::VSELECT : ISD::SELECT, DL, VT, Eq,
+                         Operand, Estimate);
 
+SDValue AArch64TargetLowering::getRecipEstimate(SDValue Operand,
+                                                SelectionDAG &DAG) const {
+  EVT VT = Operand.getValueType();
+  if (!Subtarget->hasNEON() || !canDoApproxSqrtOrRecipOnType(VT))
+    return SDValue();
+
+  Optional<unsigned> RefinementSteps =
+      Subtarget->getTargetLowering()
+          ->TargetLowering::getRecipEstimateRefinementSteps(
+              VT, DAG.getMachineFunction());
+  if (!RefinementSteps)
     return SDValue();
-}
 
-SDValue AArch64TargetLowering::getRecipEstimate(SDValue Operand,
-                                                SelectionDAG &DAG, int Enabled,
-                                                int &ExtraSteps) const {
-  if (Enabled == ReciprocalEstimate::Enabled)
-    if (SDValue Estimate = getEstimate(Subtarget, AArch64ISD::FRECPE, Operand,
-                                       DAG, ExtraSteps)) {
-      SDLoc DL(Operand);
-      EVT VT = Operand.getValueType();
+  SDValue Estimate = DAG.getNode(AArch64ISD::FRECPE, DL, VT, Operand);
 
-      SDNodeFlags Flags;
-      Flags.setUnsafeAlgebra(true);
 
-      // Newton reciprocal iteration: E * (2 - X * E)
-      // AArch64 reciprocal iteration instruction: (2 - M * N)
-      for (int i = ExtraSteps; i > 0; --i) {
-        SDValue Step = DAG.getNode(AArch64ISD::FRECPS, DL, VT, Operand,
-                                   Estimate, &Flags);
-        Estimate = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Step, &Flags);
-      }
+  for (int i = *RefinementSteps; i > 0; --i) {
+    SDValue Step =
+        DAG.getNode(AArch64ISD::FRECPS, DL, VT, Operand, Estimate, &Flags);
 
-      ExtraSteps = 0;
-      return Estimate;
-    }
 
-  return SDValue();
-}
-
 //===----------------------------------------------------------------------===//
 //                          AArch64 Inline Assembly Support
 //===----------------------------------------------------------------------===//
 
 // Table of Constraints
 // TODO: This is the current set of constraints supported by ARM for the
 // compiler, not all of them may make sense, e.g. S may be difficult to support.
 //

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.h

SDValue CombineFMinMaxLegacy(const SDLoc &DL, EVT VT, SDValue LHS,
                                SDValue RHS, SDValue True, SDValue False,
                                SDValue CC, DAGCombinerInfo &DCI) const;
 
   const char* getTargetNodeName(unsigned Opcode) const override;
 
   bool isFsqrtCheap(SDValue Operand, SelectionDAG &DAG) const override {
     return true;
   }
-  SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                           int &RefinementSteps, bool &UseOneConstNR,
-                           bool Reciprocal) const override;
-  SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                           int &RefinementSteps) const override;
+
+  RefinementDefaults getSqrtRefinementDefaults(EVT) const override;
+  RefinementDefaults getRecipRefinementDefaults(EVT) const override;
+  SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+  SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG) const override;
 
   virtual SDNode *PostISelFolding(MachineSDNode *N,
                                   SelectionDAG &DAG) const = 0;
 
   /// \brief Determine which of the bits specified in \p Mask are known to be
   /// either zero or one and return them in the \p KnownZero and \p KnownOne
   /// bitsets.
   void computeKnownBitsForTargetNode(const SDValue Op,

llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp

const char* AMDGPUTargetLowering::getTargetNodeName(unsigned Opcode) const {
   NODE_NAME_CASE(ATOMIC_DEC)
   NODE_NAME_CASE(BUFFER_LOAD)
   NODE_NAME_CASE(BUFFER_LOAD_FORMAT)
   case AMDGPUISD::LAST_AMDGPU_ISD_NUMBER: break;
   }
   return nullptr;
 }
 
-SDValue AMDGPUTargetLowering::getSqrtEstimate(SDValue Operand,
-                                              SelectionDAG &DAG, int Enabled,
-                                              int &RefinementSteps,
-                                              bool &UseOneConstNR,
-                                              bool Reciprocal) const {
-  EVT VT = Operand.getValueType();
+TargetLowering::RefinementDefaults
+AMDGPUTargetLowering::getSqrtRefinementDefaults(EVT VT) const {
+  return {/*EnabledByDefault=*/VT == MVT::f32, /*DefaultSteps=*/0};
+}
 
-  if (VT == MVT::f32) {
-    RefinementSteps = 0;
-    return DAG.getNode(AMDGPUISD::RSQ, SDLoc(Operand), VT, Operand);
-  }
+TargetLowering::RefinementDefaults
+AMDGPUTargetLowering::getRecipRefinementDefaults(EVT VT) const {
+  return {/*EnabledByDefault=*/VT == MVT::f32, /*DefaultSteps=*/0};
 
+SDValue AMDGPUTargetLowering::getRsqrtEstimate(SDValue Operand,
+                                               SelectionDAG &DAG) const {
+  EVT VT = Operand.getValueType();
+  if (VT == MVT::f32)
+    return DAG.getNode(AMDGPUISD::RSQ, SDLoc(Operand), VT, Operand);
+
   // TODO: There is also f64 rsq instruction, but the documentation is less
   // clear on its precision.
 
   return SDValue();
 }
 
 SDValue AMDGPUTargetLowering::getRecipEstimate(SDValue Operand,
-                                               SelectionDAG &DAG, int Enabled,
-                                               int &RefinementSteps) const {
+                                               SelectionDAG &DAG) const {
   EVT VT = Operand.getValueType();
-
   if (VT == MVT::f32) {
     // Reciprocal, < 1 ulp error.
     //
     // This reciprocal approximation converges to < 0.5 ulp error with one
     // newton rhapson performed with two fused multiple adds (FMAs).
-
-    RefinementSteps = 0;
     return DAG.getNode(AMDGPUISD::RCP, SDLoc(Operand), VT, Operand);
   }
 
   // TODO: There is also f64 rcp instruction, but the documentation is less
   // clear on its precision.
 
   return SDValue();
 }

llvm/lib/Target/NVPTX/NVPTXISelLowering.h

public:
   // Get whether we should use a precise or approximate 32-bit floating point
   // sqrt instruction.
   bool usePrecSqrtF32() const;
 
   // Get whether we should use instructions that flush floating-point denormals
   // to sign-preserving zero.
   bool useF32FTZ(const MachineFunction &MF) const;
 
-  SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                          int &ExtraSteps, bool &UseOneConst,
-                          bool Reciprocal) const override;
+  RefinementDefaults getSqrtRefinementDefaults(EVT) const override;
+  SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+  SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
 
   bool allowFMA(MachineFunction &MF, CodeGenOpt::Level OptLevel) const;
   bool allowUnsafeFPMath(MachineFunction &MF) const;
 
   bool isFMAFasterThanFMulAndFAdd(EVT) const override { return true; }
 
   bool enableAggressiveFMAFusion(EVT VT) const override { return true; }
 

llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp

 TargetLoweringBase::LegalizeTypeAction
 NVPTXTargetLowering::getPreferredVectorAction(EVT VT) const {
   if (VT.getVectorNumElements() != 1 && VT.getScalarType() == MVT::i1)
     return TypeSplitVector;
 
   return TargetLoweringBase::getPreferredVectorAction(VT);
 }
 
-SDValue NVPTXTargetLowering::getSqrtEstimate(SDValue Operand, SelectionDAG &DAG,
-                                             int Enabled, int &ExtraSteps,
-                                             bool &UseOneConst,
-                                             bool Reciprocal) const {
-  if (!(Enabled == ReciprocalEstimate::Enabled ||
-        (Enabled == ReciprocalEstimate::Unspecified && !usePrecSqrtF32())))
-    return SDValue();
-
-  if (ExtraSteps == ReciprocalEstimate::Unspecified)
-    ExtraSteps = 0;
+TargetLowering::RefinementDefaults
+NVPTXTargetLowering::getSqrtRefinementDefaults(EVT) const {
+  return {/*EnabledByDefault=*/!usePrecSqrtF32(), /*DefaultSteps=*/0};
+}
 
+SDValue NVPTXTargetLowering::getSqrtEstimate(SDValue Operand,
+                                             SelectionDAG &DAG) const {
   SDLoc DL(Operand);
   EVT VT = Operand.getValueType();
   bool Ftz = useF32FTZ(DAG.getMachineFunction());
 
   auto MakeIntrinsicCall = [&](Intrinsic::ID IID) {
     return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
                        DAG.getConstant(IID, DL, MVT::i32), Operand);
   };
 
-  // The sqrt and rsqrt refinement processes assume we always start out with an
-  // approximation of the rsqrt.  Therefore, if we're going to do any refinement
-  // (i.e. ExtraSteps > 0), we must return an rsqrt.  But if we're *not* doing
-  // any refinement, we must return a regular sqrt.
-  if (Reciprocal || ExtraSteps > 0) {
-    if (VT == MVT::f32)
-      return MakeIntrinsicCall(Ftz ? Intrinsic::nvvm_rsqrt_approx_ftz_f
-                                   : Intrinsic::nvvm_rsqrt_approx_f);
-    else if (VT == MVT::f64)
-      return MakeIntrinsicCall(Intrinsic::nvvm_rsqrt_approx_d);
-    else
-      return SDValue();
-  } else {
-    if (VT == MVT::f32)
-      return MakeIntrinsicCall(Ftz ? Intrinsic::nvvm_sqrt_approx_ftz_f
-                                   : Intrinsic::nvvm_sqrt_approx_f);
-    else {
-      // There's no sqrt.approx.f64 instruction, so we emit
-      // reciprocal(rsqrt(x)).  This is faster than
-      // select(x == 0, 0, x * rsqrt(x)).  (In fact, it's faster than plain
-      // x * rsqrt(x).)
-      return DAG.getNode(
-          ISD::INTRINSIC_WO_CHAIN, DL, VT,
-          DAG.getConstant(Intrinsic::nvvm_rcp_approx_ftz_d, DL, MVT::i32),
-          MakeIntrinsicCall(Intrinsic::nvvm_rsqrt_approx_d));
-    }
-  }
+
+  if (VT == MVT::f64) {
+
+  return SDValue();
+
+SDValue NVPTXTargetLowering::getRsqrtEstimate(SDValue Operand,
+                                              SelectionDAG &DAG) const {
+  SDLoc DL(Operand);
+  EVT VT = Operand.getValueType();
+  bool Ftz = useF32FTZ(DAG.getMachineFunction());
+
+  Intrinsic::ID IID;
+  if (VT == MVT::f32)
+    IID = Ftz ? Intrinsic::nvvm_rsqrt_approx_ftz_f
+              : Intrinsic::nvvm_rsqrt_approx_f;
+  else if (VT == MVT::f64)
+    IID = Intrinsic::nvvm_rsqrt_approx_d;
+  else
+    return SDValue();
+
+  return DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
+                     DAG.getConstant(IID, DL, MVT::i32), Operand);
 }
 
 SDValue
 NVPTXTargetLowering::LowerGlobalAddress(SDValue Op, SelectionDAG &DAG) const {
   SDLoc dl(Op);
   const GlobalValue *GV = cast<GlobalAddressSDNode>(Op)->getGlobal();
   auto PtrVT = getPointerTy(DAG.getDataLayout());
   Op = DAG.getTargetGlobalAddress(GV, dl, PtrVT);

llvm/lib/Target/PowerPC/PPCISelLowering.h

private:
     SDValue DAGCombineTruncBoolExt(SDNode *N, DAGCombinerInfo &DCI) const;
     SDValue combineFPToIntToFP(SDNode *N, DAGCombinerInfo &DCI) const;
 
     /// ConvertSETCCToSubtract - looks at SETCC that compares ints. It replaces
     /// SETCC with integer subtraction when (1) there is a legal way of doing it
     /// (2) keeping the result of comparison in GPR has performance benefit.
     SDValue ConvertSETCCToSubtract(SDNode *N, DAGCombinerInfo &DCI) const;
 
-    SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                            int &RefinementSteps, bool &UseOneConstNR,
-                            bool Reciprocal) const override;
-    SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                             int &RefinementSteps) const override;
+    RefinementDefaults getSqrtRefinementDefaults(EVT) const override;
+    RefinementDefaults getRecipRefinementDefaults(EVT) const override;
+    bool useOneConstNRForSqrtEstimate(EVT) const override { return true; }
+
+    SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+    SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG) const override;
+
     unsigned combineRepeatedFPDivisors() const override;
 
     CCAssignFn *useFastISelCCs(unsigned Flag) const;
 
     SDValue
     combineElementTruncationToVectorTruncation(SDNode *N,
                                                DAGCombinerInfo &DCI) const;
   };

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

PPCTargetLowering::EmitInstrWithCustomInserter(MachineInstr &MI,
   MI.eraseFromParent(); // The pseudo instruction is gone now.
   return BB;
 }
 
 //===----------------------------------------------------------------------===//
 // Target Optimization Hooks
 //===----------------------------------------------------------------------===//
 
-static int getEstimateRefinementSteps(EVT VT, const PPCSubtarget &Subtarget) {
+static unsigned getEstimateRefinementSteps(EVT VT,
+                                           const PPCSubtarget &Subtarget) {
   // For the estimates, convergence is quadratic, so we essentially double the
   // number of digits correct after every iteration. For both FRE and FRSQRTE,
   // the minimum architected relative accuracy is 2^-5. When hasRecipPrec(),
   // this is 2^-14. IEEE float has 23 digits and double has 52 digits.
   int RefinementSteps = Subtarget.hasRecipPrec() ? 1 : 3;
   if (VT.getScalarType() == MVT::f64)
     RefinementSteps++;
   return RefinementSteps;
 }
 
-SDValue PPCTargetLowering::getSqrtEstimate(SDValue Operand, SelectionDAG &DAG,
-                                           int Enabled, int &RefinementSteps,
-                                           bool &UseOneConstNR,
-                                           bool Reciprocal) const {
+TargetLowering::RefinementDefaults
+PPCTargetLowering::getSqrtRefinementDefaults(EVT VT) const {
+  return {/*EnabledByDefault=*/true, getEstimateRefinementSteps(VT, Subtarget)};
+}
+
+TargetLowering::RefinementDefaults
+PPCTargetLowering::getRecipRefinementDefaults(EVT VT) const {
+  return {/*EnabledByDefault=*/true, getEstimateRefinementSteps(VT, Subtarget)};
+}
+
+SDValue PPCTargetLowering::getRsqrtEstimate(SDValue Operand,
+                                            SelectionDAG &DAG) const {
   EVT VT = Operand.getValueType();
   if ((VT == MVT::f32 && Subtarget.hasFRSQRTES()) ||
       (VT == MVT::f64 && Subtarget.hasFRSQRTE()) ||
       (VT == MVT::v4f32 && Subtarget.hasAltivec()) ||
       (VT == MVT::v2f64 && Subtarget.hasVSX()) ||
       (VT == MVT::v4f32 && Subtarget.hasQPX()) ||
-      (VT == MVT::v4f64 && Subtarget.hasQPX())) {
-    if (RefinementSteps == ReciprocalEstimate::Unspecified)
-      RefinementSteps = getEstimateRefinementSteps(VT, Subtarget);
-
-    UseOneConstNR = true;
+      (VT == MVT::v4f64 && Subtarget.hasQPX()))
     return DAG.getNode(PPCISD::FRSQRTE, SDLoc(Operand), VT, Operand);
-  }
+
   return SDValue();
 }
 
-SDValue PPCTargetLowering::getRecipEstimate(SDValue Operand, SelectionDAG &DAG,
-                                            int Enabled,
-                                            int &RefinementSteps) const {
+SDValue PPCTargetLowering::getRecipEstimate(SDValue Operand,
+                                            SelectionDAG &DAG) const {
   EVT VT = Operand.getValueType();
   if ((VT == MVT::f32 && Subtarget.hasFRES()) ||
       (VT == MVT::f64 && Subtarget.hasFRE()) ||
       (VT == MVT::v4f32 && Subtarget.hasAltivec()) ||
       (VT == MVT::v2f64 && Subtarget.hasVSX()) ||
       (VT == MVT::v4f32 && Subtarget.hasQPX()) ||
-      (VT == MVT::v4f64 && Subtarget.hasQPX())) {
-    if (RefinementSteps == ReciprocalEstimate::Unspecified)
-      RefinementSteps = getEstimateRefinementSteps(VT, Subtarget);
+      (VT == MVT::v4f64 && Subtarget.hasQPX()))
     return DAG.getNode(PPCISD::FRE, SDLoc(Operand), VT, Operand);
-  }
+
   return SDValue();
 }
 
 unsigned PPCTargetLowering::combineRepeatedFPDivisors() const {
   // Note: This functionality is used only when unsafe-fp-math is enabled, and
   // on cores with reciprocal estimates (which are used when unsafe-fp-math is
   // enabled for division), this functionality is redundant with the default
   // combiner logic (once the division -> reciprocal/multiply transformation

llvm/lib/Target/X86/X86ISelLowering.h

private:
 
     /// Convert a comparison if required by the subtarget.
     SDValue ConvertCmpIfNecessary(SDValue Cmp, SelectionDAG &DAG) const;
 
     /// Check if replacement of SQRT with RSQRT should be disabled.
     bool isFsqrtCheap(SDValue Operand, SelectionDAG &DAG) const override;
 
     /// Use rsqrt* to speed up sqrt calculations.
-    SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                            int &RefinementSteps, bool &UseOneConstNR,
-                            bool Reciprocal) const override;
+    bool useOneConstNRForSqrtEstimate(EVT) const override { return false; }
+    RefinementDefaults getSqrtRefinementDefaults(EVT) const override;
+    SDValue getRsqrtEstimate(SDValue Operand, SelectionDAG &DAG) const override;
 
     /// Use rcp* to speed up fdiv calculations.
-    SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
-                             int &RefinementSteps) const override;
+    RefinementDefaults getRecipRefinementDefaults(EVT) const override;
+    SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG) const override;
 
     /// Reassociate floating point divisions into multiply by reciprocal.
     unsigned combineRepeatedFPDivisors() const override;
   };
 
   namespace X86 {
     FastISel *createFastISel(FunctionLoweringInfo &funcInfo,
                              const TargetLibraryInfo *libInfo);

llvm/lib/Target/X86/X86ISelLowering.cpp

bool X86TargetLowering::isFsqrtCheap(SDValue Op, SelectionDAG &DAG) const {
   if (DAG.getNodeIfExists(X86ISD::FRSQRT, DAG.getVTList(VT), Op))
     return false;
 
   if (VT.isVector())
     return Subtarget.hasFastVectorFSQRT();
   return Subtarget.hasFastScalarFSQRT();
 }
 
+TargetLowering::RefinementDefaults
+X86TargetLowering::getSqrtRefinementDefaults(EVT VT) const {
+  return {/*EnabledByDefault=*/true, /*DefaultSteps=*/1};
+}
+
 /// The minimum architected relative accuracy is 2^-12. We need one
 /// Newton-Raphson step to have a good float result (24 bits of precision).
-SDValue X86TargetLowering::getSqrtEstimate(SDValue Op,
-                                           SelectionDAG &DAG, int Enabled,
-                                           int &RefinementSteps,
-                                           bool &UseOneConstNR,
-                                           bool Reciprocal) const {
+SDValue X86TargetLowering::getRsqrtEstimate(SDValue Op,
+                                            SelectionDAG &DAG) const {
   EVT VT = Op.getValueType();
 
   // SSE1 has rsqrtss and rsqrtps. AVX adds a 256-bit variant for rsqrtps.
   // TODO: Add support for AVX512 (v16f32).
   // It is likely not profitable to do this for f64 because a double-precision
   // rsqrt estimate with refinement on x86 prior to FMA requires at least 16
   // instructions: convert to single, rsqrtss, convert back to double, refine
   // (3 steps = at least 13 insts). If an 'rsqrtsd' variant was added to the ISA
   // along with FMA, this could be a throughput win.
   if ((VT == MVT::f32 && Subtarget.hasSSE1()) ||
       (VT == MVT::v4f32 && Subtarget.hasSSE1()) ||
-      (VT == MVT::v8f32 && Subtarget.hasAVX())) {
-    if (RefinementSteps == ReciprocalEstimate::Unspecified)
-      RefinementSteps = 1;
-
-    UseOneConstNR = false;
+      (VT == MVT::v8f32 && Subtarget.hasAVX()))
     return DAG.getNode(X86ISD::FRSQRT, SDLoc(Op), VT, Op);
-  }
+
   return SDValue();
 }
 
+TargetLowering::RefinementDefaults
+X86TargetLowering::getRecipRefinementDefaults(EVT VT) const {
+  // Enable estimate codegen with 1 refinement step for vector division.
+  // Scalar division estimates are disabled because they break too much
+  // real-world code. These defaults are intended to match GCC behavior.
+  return {/*EnabledByDefault=*/VT.isVector(), /*DefaultSteps=*/1};
+}
+
 /// The minimum architected relative accuracy is 2^-12. We need one
 /// Newton-Raphson step to have a good float result (24 bits of precision).
-SDValue X86TargetLowering::getRecipEstimate(SDValue Op, SelectionDAG &DAG,
-                                            int Enabled,
-                                            int &RefinementSteps) const {
-  EVT VT = Op.getValueType();
-
+SDValue X86TargetLowering::getRecipEstimate(SDValue Op,
+                                            SelectionDAG &DAG) const {
   // SSE1 has rcpss and rcpps. AVX adds a 256-bit variant for rcpps.
   // TODO: Add support for AVX512 (v16f32).
   // It is likely not profitable to do this for f64 because a double-precision
   // reciprocal estimate with refinement on x86 prior to FMA requires
   // 15 instructions: convert to single, rcpss, convert back to double, refine
   // (3 steps = 12 insts). If an 'rcpsd' variant was added to the ISA
   // along with FMA, this could be a throughput win.
 
+  EVT VT = Op.getValueType();
   if ((VT == MVT::f32 && Subtarget.hasSSE1()) ||
       (VT == MVT::v4f32 && Subtarget.hasSSE1()) ||
-      (VT == MVT::v8f32 && Subtarget.hasAVX())) {
-    // Enable estimate codegen with 1 refinement step for vector division.
-    // Scalar division estimates are disabled because they break too much
-    // real-world code. These defaults are intended to match GCC behavior.
-    if (VT == MVT::f32 && Enabled == ReciprocalEstimate::Unspecified)
-      return SDValue();
-
-    if (RefinementSteps == ReciprocalEstimate::Unspecified)
-      RefinementSteps = 1;
-
+      (VT == MVT::v8f32 && Subtarget.hasAVX()))
     return DAG.getNode(X86ISD::FRCP, SDLoc(Op), VT, Op);
-  }
+
   return SDValue();
 }
 
 /// If we have at least two divisions that use the same divisor, convert to
 /// multplication by a reciprocal. This may need to be adjusted for a given
 /// CPU if a division's cost is not at least twice the cost of a multiplication.
 /// This is because we still need one division to calculate the reciprocal and
 /// then we need two multiplies by that reciprocal as replacements for the