[NFC] [DAGCombine] Correct the result for sqrt even the iteration is zero

[steven.zhang]

Summary

For now, we correct the result for sqrt if iteration > 0. This doesn't make sense as they are not strict relative. Besides, some target(i.e. AArch64 and PowerPC downstream) want to have custom sqrt soft expansion instead of the default OneConstantNR/TwoConstantNR and set the iteration as zero. We need to correct the result for them also. As AArch64 don't want to test the input with IEEE, so, I move the test implementation into the hook.

[steven.zhang]

Gentle ping...

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

5873

(style) drop the else

  if (Mode.Input == DenormalMode::IEEE) {
  ...
  }

  // Test = X == 0.0
  return DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
}

[steven.zhang]

Address comments.

[spatel]

llvm/test/CodeGen/AArch64/sqrt-fastmath.ll

75–76

The bif/bic difference seems logically fine and probably better overall since we're using less registers, but someone who knows AArch should comment.

Also, it would be interesting to know why the code changed because we are seemingly producing the same set of SDNode ops? There might be another combine/lowering opportunity.

[steven.zhang]

The bif/bic difference seems logically fine and probably better overall since we're using less registers, but someone who knows AArch should comment.
Also, it would be interesting to know why the code changed because we are seemingly producing the same set of SDNode ops? There might be another combine/lowering opportunity.

This is a good example to show how important the code review is and thank you for pointing out this. I should have realized it as it should be a NFC patch. The reason why you see the test change is that, AArch64 don't want the zero if it is denormal input while the default implementation return zero if it is denormal input. I miss to override it.

[dmgreen]

but someone who knows AArch should comment.

Sorry, I had no idea what this was and what it was doing. I see now, and this does indeed look like an NFC.

It turns out that on AArch64 it is controlled by a subtarget feature that is never enabled, so won't be used without being forced. Not handling IEEE denomal inputs might be a mistake, I'm not sure. This patch is an NFC in that regards though.

LGTM

llvm/include/llvm/CodeGen/TargetLowering.h

virtual SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG,
     return SDValue();
   }
 
   /// Return a target-dependent comparison result if the input operand is
   /// suitable for use with a square root estimate calculation. For example, the
   /// comparison may check if the operand is NAN, INF, zero, normal, etc. The
   /// result should be used as the condition operand for a select or branch.
   virtual SDValue getSqrtInputTest(SDValue Operand, SelectionDAG &DAG,
-                                   const DenormalMode &Mode) const {
-    return SDValue();
-  }
+                                   const DenormalMode &Mode) const;
 
   /// Return a target-dependent result if the input operand is not suitable for
   /// use with a square root estimate calculation.
   virtual SDValue getSqrtResultForDenormInput(SDValue Operand,
                                               SelectionDAG &DAG) const {
     return DAG.getConstantFP(0.0, SDLoc(Operand), Operand.getValueType());
   }
 

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::buildSqrtEstimateImpl(SDValue Op, SDNodeFlags Flags,
   int Iterations = TLI.getSqrtRefinementSteps(VT, MF);
 
   bool UseOneConstNR = false;
   if (SDValue Est =
       TLI.getSqrtEstimate(Op, DAG, Enabled, Iterations, UseOneConstNR,
                           Reciprocal)) {
     AddToWorklist(Est.getNode());
 
-    if (Iterations) {
+    if (Iterations)
       Est = UseOneConstNR
             ? buildSqrtNROneConst(Op, Est, Iterations, Flags, Reciprocal)
             : buildSqrtNRTwoConst(Op, Est, Iterations, Flags, Reciprocal);
-
-      if (!Reciprocal) {
-        SDLoc DL(Op);
-        EVT CCVT = getSetCCResultType(VT);
-        SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
-        DenormalMode DenormMode = DAG.getDenormalMode(VT);
-        // Try the target specific test first.
-        SDValue Test = TLI.getSqrtInputTest(Op, DAG, DenormMode);
-        if (!Test) {
-          // If no test provided by target, testing it with denormal inputs to
-          // avoid wrong estimate.
-          if (DenormMode.Input == DenormalMode::IEEE) {
-            // This is specifically a check for the handling of denormal inputs,
-            // not the result.
-
-            // Test = fabs(X) < SmallestNormal
-            const fltSemantics &FltSem = DAG.EVTToAPFloatSemantics(VT);
-            APFloat SmallestNorm = APFloat::getSmallestNormalized(FltSem);
-            SDValue NormC = DAG.getConstantFP(SmallestNorm, DL, VT);
-            SDValue Fabs = DAG.getNode(ISD::FABS, DL, VT, Op);
-            Test = DAG.getSetCC(DL, CCVT, Fabs, NormC, ISD::SETLT);
-          } else
-            // Test = X == 0.0
-            Test = DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
-        }
+      SDValue Test = TLI.getSqrtInputTest(Op, DAG, DAG.getDenormalMode(VT));
 
-        // The estimate is now completely wrong if the input was exactly 0.0 or
-        // possibly a denormal. Force the answer to 0.0 or value provided by
-        // target for those cases.
-        Est = DAG.getNode(
-            Test.getValueType().isVector() ? ISD::VSELECT : ISD::SELECT, DL, VT,
-            Test, TLI.getSqrtResultForDenormInput(Op, DAG), Est);
-      }
-    }
     return Est;
   }
 
   return SDValue();
 }
 
 SDValue DAGCombiner::buildRsqrtEstimate(SDValue Op, SDNodeFlags Flags) {
   return buildSqrtEstimateImpl(Op, Flags, true);

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

if (!isa<ConstantSDNode>(Op.getOperand(0))) {
     DAG.getContext()->emitError("argument to '__builtin_return_address' must "
                                 "be a constant integer");
     return true;
   }
 
   return false;
 }
 
+SDValue TargetLowering::getSqrtInputTest(SDValue Op, SelectionDAG &DAG,
+                                         const DenormalMode &Mode) const {
+  SDLoc DL(Op);
+  EVT VT = Op.getValueType();
+  EVT CCVT = getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
+  SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
+  // Testing it with denormal inputs to avoid wrong estimate.
+  if (Mode.Input == DenormalMode::IEEE) {
+    // This is specifically a check for the handling of denormal inputs,
+    // not the result.
+
+    // Test = fabs(X) < SmallestNormal
+    const fltSemantics &FltSem = DAG.EVTToAPFloatSemantics(VT);
+    APFloat SmallestNorm = APFloat::getSmallestNormalized(FltSem);
+    SDValue NormC = DAG.getConstantFP(SmallestNorm, DL, VT);
+    SDValue Fabs = DAG.getNode(ISD::FABS, DL, VT, Op);
+    return DAG.getSetCC(DL, CCVT, Fabs, NormC, ISD::SETLT);
+  }
+  // Test = X == 0.0
+  return DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
+}
+
 SDValue TargetLowering::getNegatedExpression(SDValue Op, SelectionDAG &DAG,
                                              bool LegalOps, bool OptForSize,
                                              NegatibleCost &Cost,
                                              unsigned Depth) const {
   // fneg is removable even if it has multiple uses.
   if (Op.getOpcode() == ISD::FNEG) {
     Cost = NegatibleCost::Cheaper;
     return Op.getOperand(0);

llvm/lib/Target/AArch64/AArch64ISelLowering.h

private:
 
   SDValue BuildSDIVPow2(SDNode *N, const APInt &Divisor, SelectionDAG &DAG,
                         SmallVectorImpl<SDNode *> &Created) const override;
   SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                           int &ExtraSteps, bool &UseOneConst,
                           bool Reciprocal) const override;
   SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                            int &ExtraSteps) const override;
+  SDValue getSqrtInputTest(SDValue Operand, SelectionDAG &DAG,
+                           const DenormalMode &Mode) const override;
   unsigned combineRepeatedFPDivisors() const override;
 
   ConstraintType getConstraintType(StringRef Constraint) const override;
   Register getRegisterByName(const char* RegName, LLT VT,
                              const MachineFunction &MF) const override;
 
   /// Examine constraint string and operand type and determine a weight value.
   /// The operand object must already have been set up with the operand type.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

if (ExtraSteps == TargetLoweringBase::ReciprocalEstimate::Unspecified)
       ExtraSteps = VT.getScalarType() == MVT::f64 ? 3 : 2;
 
     return DAG.getNode(Opcode, SDLoc(Operand), VT, Operand);
   }
 
   return SDValue();
 }
 
+SDValue
+AArch64TargetLowering::getSqrtInputTest(SDValue Op, SelectionDAG &DAG,
+                                        const DenormalMode &Mode) const {
+  SDLoc DL(Op);
+  EVT VT = Op.getValueType();
+  EVT CCVT = getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT);
+  SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
+  return DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
+}
+
 SDValue AArch64TargetLowering::getSqrtEstimate(SDValue Operand,
                                                SelectionDAG &DAG, int Enabled,
                                                int &ExtraSteps,
                                                bool &UseOneConst,
                                                bool Reciprocal) const {
   if (Enabled == ReciprocalEstimate::Enabled ||
       (Enabled == ReciprocalEstimate::Unspecified && Subtarget->useRSqrt()))
     if (SDValue Estimate = getEstimate(Subtarget, AArch64ISD::FRSQRTE, Operand,
                                        DAG, ExtraSteps)) {
       SDLoc DL(Operand);
       EVT VT = Operand.getValueType();
 
       SDNodeFlags Flags;
       Flags.setAllowReassociation(true);
 
       // Newton reciprocal square root iteration: E * 0.5 * (3 - X * E^2)
       // AArch64 reciprocal square root iteration instruction: 0.5 * (3 - M * N)
       for (int i = ExtraSteps; i > 0; --i) {
         SDValue Step = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Estimate,
                                    Flags);
         Step = DAG.getNode(AArch64ISD::FRSQRTS, DL, VT, Operand, Step, Flags);
         Estimate = DAG.getNode(ISD::FMUL, DL, VT, Estimate, Step, Flags);
       }
-      if (!Reciprocal) {
-        EVT CCVT = getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(),
-                                      VT);
-        SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
-        SDValue Eq = DAG.getSetCC(DL, CCVT, Operand, FPZero, ISD::SETEQ);
-
+      if (!Reciprocal)
         Estimate = DAG.getNode(ISD::FMUL, DL, VT, Operand, Estimate, Flags);
-        // Correct the result if the operand is 0.0.
-        Estimate = DAG.getNode(VT.isVector() ? ISD::VSELECT : ISD::SELECT, DL,
-                               VT, Eq, Operand, Estimate);
-      }
 
       ExtraSteps = 0;
       return Estimate;
     }
 
   return SDValue();
 }
 

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

 
 SDValue PPCTargetLowering::getSqrtInputTest(SDValue Op, SelectionDAG &DAG,
                                             const DenormalMode &Mode) const {
   // We only have VSX Vector Test for software Square Root.
   EVT VT = Op.getValueType();
   if (!isTypeLegal(MVT::i1) ||
       (VT != MVT::f64 &&
        ((VT != MVT::v2f64 && VT != MVT::v4f32) || !Subtarget.hasVSX())))
-    return SDValue();
+    return TargetLowering::getSqrtInputTest(Op, DAG, Mode);
 
   SDLoc DL(Op);
   // The output register of FTSQRT is CR field.
   SDValue FTSQRT = DAG.getNode(PPCISD::FTSQRT, DL, MVT::i32, Op);
   // ftsqrt BF,FRB
   // Let e_b be the unbiased exponent of the double-precision
   // floating-point operand in register FRB.
   // fe_flag is set to 1 if either of the following conditions occurs.

llvm/test/CodeGen/AArch64/sqrt-fastmath.ll

 ; CHECK-NEXT:    frsqrte v1.2s, v0.2s
 ; CHECK-NEXT:    fmul v2.2s, v1.2s, v1.2s
 ; CHECK-NEXT:    frsqrts v2.2s, v0.2s, v2.2s
 ; CHECK-NEXT:    fmul v1.2s, v1.2s, v2.2s
 ; CHECK-NEXT:    fmul v2.2s, v1.2s, v1.2s
 ; CHECK-NEXT:    frsqrts v2.2s, v0.2s, v2.2s
 ; CHECK-NEXT:    fmul v2.2s, v2.2s, v0.2s
 ; CHECK-NEXT:    fmul v1.2s, v1.2s, v2.2s
-; CHECK-NEXT:    fcmeq v2.2s, v0.2s, #0.0
-; CHECK-NEXT:    bif v0.8b, v1.8b, v2.8b
+; CHECK-NEXT:    fcmeq v0.2s, v0.2s, #0.0
+; CHECK-NEXT:    bic v0.8b, v1.8b, v0.8b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <2 x float> @llvm.sqrt.v2f32(<2 x float> %a)
   ret <2 x float> %1
 }
 
 define <4 x float> @f4sqrt(<4 x float> %a) #0 {
 ; FAULT-LABEL: f4sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.4s, v0.4s
 ; FAULT-NEXT:    ret
 ;
 ; CHECK-LABEL: f4sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v1.4s, v0.4s
 ; CHECK-NEXT:    fmul v2.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    frsqrts v2.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    fmul v1.4s, v1.4s, v2.4s
 ; CHECK-NEXT:    fmul v2.4s, v1.4s, v1.4s
 ; CHECK-NEXT:    frsqrts v2.4s, v0.4s, v2.4s
 ; CHECK-NEXT:    fmul v2.4s, v2.4s, v0.4s
 ; CHECK-NEXT:    fmul v1.4s, v1.4s, v2.4s
-; CHECK-NEXT:    fcmeq v2.4s, v0.4s, #0.0
-; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; CHECK-NEXT:    bic v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <4 x float> @llvm.sqrt.v4f32(<4 x float> %a)
   ret <4 x float> %1
 }
 
 define <8 x float> @f8sqrt(<8 x float> %a) #0 {
 ; FAULT-LABEL: f8sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.4s, v0.4s
 ; FAULT-NEXT:    fsqrt v1.4s, v1.4s
 ; FAULT-NEXT:    ret
 ;
 ; CHECK-LABEL: f8sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v2.4s, v0.4s
-; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
-; CHECK-NEXT:    frsqrts v3.4s, v0.4s, v3.4s
-; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
-; CHECK-NEXT:    frsqrts v3.4s, v0.4s, v3.4s
-; CHECK-NEXT:    fmul v3.4s, v3.4s, v0.4s
-; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    fcmeq v3.4s, v0.4s, #0.0
-; CHECK-NEXT:    bif v0.16b, v2.16b, v3.16b
-; CHECK-NEXT:    frsqrte v2.4s, v1.4s
-; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
-; CHECK-NEXT:    frsqrts v3.4s, v1.4s, v3.4s
-; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    fmul v3.4s, v2.4s, v2.4s
-; CHECK-NEXT:    frsqrts v3.4s, v1.4s, v3.4s
-; CHECK-NEXT:    fmul v3.4s, v3.4s, v1.4s
-; CHECK-NEXT:    fmul v2.4s, v2.4s, v3.4s
-; CHECK-NEXT:    fcmeq v3.4s, v1.4s, #0.0
-; CHECK-NEXT:    bif v1.16b, v2.16b, v3.16b
+; CHECK-NEXT:    fmul v4.4s, v2.4s, v2.4s
+; CHECK-NEXT:    frsqrte v3.4s, v1.4s
+; CHECK-NEXT:    frsqrts v4.4s, v0.4s, v4.4s
+; CHECK-NEXT:    fmul v2.4s, v2.4s, v4.4s
+; CHECK-NEXT:    fmul v4.4s, v3.4s, v3.4s
+; CHECK-NEXT:    frsqrts v4.4s, v1.4s, v4.4s
+; CHECK-NEXT:    fmul v3.4s, v3.4s, v4.4s
+; CHECK-NEXT:    fmul v4.4s, v2.4s, v2.4s
+; CHECK-NEXT:    frsqrts v4.4s, v0.4s, v4.4s
+; CHECK-NEXT:    fmul v4.4s, v4.4s, v0.4s
+; CHECK-NEXT:    fmul v2.4s, v2.4s, v4.4s
+; CHECK-NEXT:    fmul v4.4s, v3.4s, v3.4s
+; CHECK-NEXT:    frsqrts v4.4s, v1.4s, v4.4s
+; CHECK-NEXT:    fmul v4.4s, v4.4s, v1.4s
+; CHECK-NEXT:    fmul v3.4s, v3.4s, v4.4s
+; CHECK-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; CHECK-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; CHECK-NEXT:    bic v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    bic v1.16b, v3.16b, v1.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <8 x float> @llvm.sqrt.v8f32(<8 x float> %a)
   ret <8 x float> %1
 }
 
 define double @dsqrt(double %a) #0 {
 ; FAULT-LABEL: dsqrt:
 ; FAULT:       // %bb.0:

 ; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v1.2d, v1.2d
 ; CHECK-NEXT:    frsqrts v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v1.2d, v1.2d
 ; CHECK-NEXT:    frsqrts v2.2d, v0.2d, v2.2d
 ; CHECK-NEXT:    fmul v2.2d, v2.2d, v0.2d
 ; CHECK-NEXT:    fmul v1.2d, v1.2d, v2.2d
-; CHECK-NEXT:    fcmeq v2.2d, v0.2d, #0.0
-; CHECK-NEXT:    bif v0.16b, v1.16b, v2.16b
+; CHECK-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; CHECK-NEXT:    bic v0.16b, v1.16b, v0.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <2 x double> @llvm.sqrt.v2f64(<2 x double> %a)
   ret <2 x double> %1
 }
 
 define <4 x double> @d4sqrt(<4 x double> %a) #0 {
 ; FAULT-LABEL: d4sqrt:
 ; FAULT:       // %bb.0:
 ; FAULT-NEXT:    fsqrt v0.2d, v0.2d
 ; FAULT-NEXT:    fsqrt v1.2d, v1.2d
 ; FAULT-NEXT:    ret
 ;
 ; CHECK-LABEL: d4sqrt:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte v2.2d, v0.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v0.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v3.2d, v0.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fcmeq v3.2d, v0.2d, #0.0
-; CHECK-NEXT:    bif v0.16b, v2.16b, v3.16b
-; CHECK-NEXT:    frsqrte v2.2d, v1.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v2.2d, v2.2d
-; CHECK-NEXT:    frsqrts v3.2d, v1.2d, v3.2d
-; CHECK-NEXT:    fmul v3.2d, v3.2d, v1.2d
-; CHECK-NEXT:    fmul v2.2d, v2.2d, v3.2d
-; CHECK-NEXT:    fcmeq v3.2d, v1.2d, #0.0
-; CHECK-NEXT:    bif v1.16b, v2.16b, v3.16b
+; CHECK-NEXT:    fmul v4.2d, v2.2d, v2.2d
+; CHECK-NEXT:    frsqrte v3.2d, v1.2d
+; CHECK-NEXT:    frsqrts v4.2d, v0.2d, v4.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v3.2d, v3.2d
+; CHECK-NEXT:    frsqrts v4.2d, v1.2d, v4.2d
+; CHECK-NEXT:    fmul v3.2d, v3.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v2.2d, v2.2d
+; CHECK-NEXT:    frsqrts v4.2d, v0.2d, v4.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v3.2d, v3.2d
+; CHECK-NEXT:    frsqrts v4.2d, v1.2d, v4.2d
+; CHECK-NEXT:    fmul v3.2d, v3.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v2.2d, v2.2d
+; CHECK-NEXT:    frsqrts v4.2d, v0.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v4.2d, v0.2d
+; CHECK-NEXT:    fmul v2.2d, v2.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v3.2d, v3.2d
+; CHECK-NEXT:    frsqrts v4.2d, v1.2d, v4.2d
+; CHECK-NEXT:    fmul v4.2d, v4.2d, v1.2d
+; CHECK-NEXT:    fmul v3.2d, v3.2d, v4.2d
+; CHECK-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; CHECK-NEXT:    fcmeq v1.2d, v1.2d, #0.0
+; CHECK-NEXT:    bic v0.16b, v2.16b, v0.16b
+; CHECK-NEXT:    bic v1.16b, v3.16b, v1.16b
 ; CHECK-NEXT:    ret
   %1 = tail call fast <4 x double> @llvm.sqrt.v4f64(<4 x double> %a)
   ret <4 x double> %1
 }
 
 define float @frsqrt(float %a) #0 {
 ; FAULT-LABEL: frsqrt:
 ; FAULT:       // %bb.0:

 ; CHECK-NEXT:    fmul d1, d1, d2
 ; CHECK-NEXT:    fmul d2, d1, d1
 ; CHECK-NEXT:    frsqrts d2, d0, d2
 ; CHECK-NEXT:    fmul d1, d1, d2
 ; CHECK-NEXT:    fmul d2, d1, d1
 ; CHECK-NEXT:    frsqrts d2, d0, d2
 ; CHECK-NEXT:    fmul d1, d1, d2
 ; CHECK-NEXT:    fcmp d0, #0.0
-; CHECK-NEXT:    fmul d1, d0, d1
-; CHECK-NEXT:    fcsel d0, d0, d1, eq
-; CHECK-NEXT:    str d0, [x0]
-; CHECK-NEXT:    mov v0.16b, v1.16b
+; CHECK-NEXT:    fmul d0, d0, d1
+; CHECK-NEXT:    fmov d1, xzr
+; CHECK-NEXT:    fcsel d1, d1, d0, eq
+; CHECK-NEXT:    str d1, [x0]
 ; CHECK-NEXT:    ret
   %sqrt = call fast double @llvm.sqrt.f64(double %x)
   store double %sqrt, double* %p
   %r = fdiv fast double %x, %sqrt
   ret double %r
 }
 
 define double @sqrt_simplify_before_recip_3_uses(double %x, double* %p1, double* %p2) nounwind {

 ; FAULT-NEXT:    str d1, [x0]
 ; FAULT-NEXT:    str d2, [x1]
 ; FAULT-NEXT:    str d3, [x2]
 ; FAULT-NEXT:    ret
 ;
 ; CHECK-LABEL: sqrt_simplify_before_recip_4_uses:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    frsqrte d1, d0
-; CHECK-NEXT:    fmul d3, d1, d1
-; CHECK-NEXT:    frsqrts d3, d0, d3
-; CHECK-NEXT:    fmul d1, d1, d3
-; CHECK-NEXT:    fmul d3, d1, d1
-; CHECK-NEXT:    frsqrts d3, d0, d3
-; CHECK-NEXT:    fmul d1, d1, d3
+; CHECK-NEXT:    fmul d4, d1, d1
+; CHECK-NEXT:    frsqrts d4, d0, d4
+; CHECK-NEXT:    fmul d1, d1, d4
+; CHECK-NEXT:    fmul d4, d1, d1
+; CHECK-NEXT:    frsqrts d4, d0, d4
 ; CHECK-NEXT:    mov x8, #4631107791820423168
-; CHECK-NEXT:    fmul d3, d1, d1
-; CHECK-NEXT:    fmov d2, x8
+; CHECK-NEXT:    fmul d1, d1, d4
+; CHECK-NEXT:    fmov d3, x8
 ; CHECK-NEXT:    mov x8, #140737488355328
-; CHECK-NEXT:    frsqrts d3, d0, d3
+; CHECK-NEXT:    fmul d4, d1, d1
 ; CHECK-NEXT:    movk x8, #16453, lsl #48
-; CHECK-NEXT:    fmul d1, d1, d3
-; CHECK-NEXT:    fcmp d0, #0.0
+; CHECK-NEXT:    frsqrts d4, d0, d4
+; CHECK-NEXT:    fmul d1, d1, d4
 ; CHECK-NEXT:    fmov d4, x8
-; CHECK-NEXT:    fmul d3, d0, d1
-; CHECK-NEXT:    fmul d2, d1, d2
+; CHECK-NEXT:    fcmp d0, #0.0
+; CHECK-NEXT:    fmov d2, xzr
+; CHECK-NEXT:    fmul d3, d1, d3
 ; CHECK-NEXT:    fmul d4, d1, d4
 ; CHECK-NEXT:    str d1, [x0]
-; CHECK-NEXT:    fcsel d1, d0, d3, eq
+; CHECK-NEXT:    fmul d1, d0, d1
+; CHECK-NEXT:    fcsel d1, d2, d1, eq
 ; CHECK-NEXT:    fdiv d0, d0, d1
-; CHECK-NEXT:    str d2, [x1]
+; CHECK-NEXT:    str d3, [x1]
 ; CHECK-NEXT:    str d4, [x2]
 ; CHECK-NEXT:    ret
   %sqrt = tail call fast double @llvm.sqrt.f64(double %x)
   %rsqrt = fdiv fast double 1.0, %sqrt
   %r1 = fdiv fast double 42.0, %sqrt
   %r2 = fdiv fast double 43.0, %sqrt
   %sqrt_fast = fdiv fast double %x, %sqrt
   store double %rsqrt, double* %p1, align 8
   store double %r1, double* %p2, align 8
   store double %r2, double* %p3, align 8
   ret double %sqrt_fast
 }
 
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { "unsafe-fp-math"="true" "denormal-fp-math"="ieee" }