[DAGCombine] Adding a hook to improve the precision of fsqrt if the input is denormal

[steven.zhang]

Summary

For now, we will hardcode the result as 0.0 if the input is denormal or 0. That will have the impact the precision. As the fsqrt added belong to the cold path of the cmp+branch, it won't impact the performance for normal inputs for PowerPC. Besides, it removes the xxlxor of the hot path.

clang without this patch

sqrt(2.2250738585072014e-308) = 1.4916681462400413e-154
sqrt(2.2250738585072009e-308) = 0
sqrt(4.9406564584124654e-324) = 0

With this patch:

sqrt(2.2250738585072014e-308) = 1.4916681462400413e-154
sqrt(2.2250738585072009e-308) = 1.4916681462400412e-154
sqrt(4.9406564584124654e-324) = 2.2227587494850775e-162

[spatel]

Not sure about other targets, but x86 is mostly expected to flush denorms (FTZ/DAZ) when using fast-math, so this should not matter even if we decide to override the default setting.

llvm/include/llvm/CodeGen/TargetLowering.h

4213

Can we have this default to:

return DAG.getConstantFP(0.0, SDLoc(Operand), Operand.getValueType());

and save some code in the calling function?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

21403

"value that provided" -> "value provided"

[steven.zhang]

Address comments.

[spatel]

As with the other patch, this LGTM in general, so if someone can verify that the PPC changes are as expected, we should be good.

[shchenz]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

12413

Target independent code supports vector type and we also have vector sqrt instruction on Powerpc target. Can we use them for vector types like v4f32 or v2f64?

[qiucf]

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

12413

Yes. Supported types:

• VSX off: f32, f64
• VSX on: f32, f64, f128, v4f32, v2f64

[steven.zhang]

Address comments.

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

12413

Yes, I will post another patch for the vector type as what I have mentioned in the comments of D80706.

[shchenz]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

21416

On Powerpc target, we use 0.0 as denormal float point sqrt result for a long time. Changing the result to hardware sqrt instructions will improve the precision for sure, but it also degrades the runtime performance. Is it possible to do it like: if we concern about performance, we use 0.0, if we concern about precision, we use hardware sqrt instruction. Maybe -Ofast is an indicatation for this?

[shchenz]

oops, accept by mistake...

[steven.zhang]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

21416

It won't deg the runtime performance if the input is not denormal which is the usual case, as the select is expanded as cmp + branch later and hw will predict to the normal code path. And it indeed slows down the performance if the input is denormal, but it is expected as the precision is improved. All the optimization is done under -Ofast.

Considering that we only have impact on the denormal input code path, which usually cares about the precision, not the performance, I tend to keep it this way. Does it make sense ?

[shchenz]

LGTM

llvm/include/llvm/CodeGen/TargetLowering.h

   /// suitable for use with a square root estimate calculation. For example, the
   /// comparison may check if the operand is NAN, INF, zero, normal, etc. The
   /// result should be used as the condition operand for a select or branch.
   virtual SDValue getSqrtInputTest(SDValue Operand, SelectionDAG &DAG,
                                    const DenormalMode &Mode) const {
     return SDValue();
   }
 
+  /// Return a target-dependent result if the input operand is not suitable for
+  /// use with a square root estimate calculation.
+  virtual SDValue getSqrtResultForDenormInput(SDValue Operand,
+                                              SelectionDAG &DAG) const {
+    return SDValue();
+  }
+
   //===--------------------------------------------------------------------===//
   // Legalization utility functions
   //
 
   /// Expand a MUL or [US]MUL_LOHI of n-bit values into two or four nodes,
   /// respectively, each computing an n/2-bit part of the result.
   /// \param Result A vector that will be filled with the parts of the result
   ///        in little-endian order.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

if (SDValue Est =
     AddToWorklist(Est.getNode());
 
     if (Iterations) {
       Est = UseOneConstNR
             ? buildSqrtNROneConst(Op, Est, Iterations, Flags, Reciprocal)
             : buildSqrtNRTwoConst(Op, Est, Iterations, Flags, Reciprocal);
 
       if (!Reciprocal) {
-        // The estimate is now completely wrong if the input was exactly 0.0 or
-        // possibly a denormal. Force the answer to 0.0 for those cases.
         SDLoc DL(Op);
         EVT CCVT = getSetCCResultType(VT);
         SDValue FPZero = DAG.getConstantFP(0.0, DL, VT);
         DenormalMode DenormMode = DAG.getDenormalMode(VT);
         // Try the target specific test first.
         SDValue Test = TLI.getSqrtInputTest(Op, DAG, DenormMode);
         if (!Test) {
           // If no test provided by target, testing it with denormal inputs to
           // avoid wrong estimate.
           if (DenormMode.Input == DenormalMode::IEEE) {
             // This is specifically a check for the handling of denormal inputs,
             // not the result.
 
             // Test = fabs(X) < SmallestNormal
             const fltSemantics &FltSem = DAG.EVTToAPFloatSemantics(VT);
             APFloat SmallestNorm = APFloat::getSmallestNormalized(FltSem);
             SDValue NormC = DAG.getConstantFP(SmallestNorm, DL, VT);
             SDValue Fabs = DAG.getNode(ISD::FABS, DL, VT, Op);
             Test = DAG.getSetCC(DL, CCVT, Fabs, NormC, ISD::SETLT);
           } else
             // Test = X == 0.0
             Test = DAG.getSetCC(DL, CCVT, Op, FPZero, ISD::SETEQ);
         }
-        // Test ? 0.0 : Est
+
+        // The estimate is now completely wrong if the input was exactly 0.0 or
+        // possibly a denormal. Force the answer to 0.0 or value that provided
+        // by target for those cases.
+        SDValue ResultForDenorm = TLI.getSqrtResultForDenormInput(Op, DAG);
+        if (!ResultForDenorm)
+          ResultForDenorm = FPZero;
+
+        // Test ? ResultForDenorm : Est
         Est = DAG.getNode(VT.isVector() ? ISD::VSELECT : ISD::SELECT, DL, VT,
-                          Test, FPZero, Est);
+                          Test, ResultForDenorm, Est);
       }
     }
     return Est;
   }
 
   return SDValue();
 }
 
 SDValue DAGCombiner::buildRsqrtEstimate(SDValue Op, SDNodeFlags Flags) {
   return buildSqrtEstimateImpl(Op, Flags, true);
 }
 
 SDValue DAGCombiner::buildSqrtEstimate(SDValue Op, SDNodeFlags Flags) {

llvm/lib/Target/PowerPC/PPCISelLowering.h

enum NodeType : unsigned {
 
     /// Reciprocal estimate instructions (unary FP ops).
     FRE,
     FRSQRTE,
 
     /// Test instruction for software square root.
     FTSQRT,
 
+    /// Square root instruction.
+    FSQRT,
+
     // VMADDFP, VNMSUBFP - The VMADDFP and VNMSUBFP instructions, taking
     // three v4f32 operands and producing a v4f32 result.
     VMADDFP,
     VNMSUBFP,
 
     /// VPERM - The PPC VPERM Instruction.
     ///
     VPERM,
private:
 
     SDValue getSqrtEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                             int &RefinementSteps, bool &UseOneConstNR,
                             bool Reciprocal) const override;
     SDValue getRecipEstimate(SDValue Operand, SelectionDAG &DAG, int Enabled,
                              int &RefinementSteps) const override;
     SDValue getSqrtInputTest(SDValue Operand, SelectionDAG &DAG,
                              const DenormalMode &Mode) const override;
+    SDValue getSqrtResultForDenormInput(SDValue Operand,
+                                        SelectionDAG &DAG) const override;
     unsigned combineRepeatedFPDivisors() const override;
 
     SDValue
     combineElementTruncationToVectorTruncation(SDNode *N,
                                                DAGCombinerInfo &DCI) const;
 
     /// lowerToVINSERTH - Return the SDValue if this VECTOR_SHUFFLE can be
     /// handled by the VINSERTH instruction introduced in ISA 3.0. This is

llvm/lib/Target/PowerPC/PPCISelLowering.cpp

const char *PPCTargetLowering::getTargetNodeName(unsigned Opcode) const {
   case PPCISD::FP_TO_UINT_IN_VSR:
                                 return "PPCISD::FP_TO_UINT_IN_VSR,";
   case PPCISD::FP_TO_SINT_IN_VSR:
                                 return "PPCISD::FP_TO_SINT_IN_VSR";
   case PPCISD::FRE:             return "PPCISD::FRE";
   case PPCISD::FRSQRTE:         return "PPCISD::FRSQRTE";
   case PPCISD::FTSQRT:
     return "PPCISD::FTSQRT";
+  case PPCISD::FSQRT:
+    return "PPCISD::FSQRT";
   case PPCISD::STFIWX:          return "PPCISD::STFIWX";
   case PPCISD::VMADDFP:         return "PPCISD::VMADDFP";
   case PPCISD::VNMSUBFP:        return "PPCISD::VNMSUBFP";
   case PPCISD::VPERM:           return "PPCISD::VPERM";
   case PPCISD::XXSPLT:          return "PPCISD::XXSPLT";
   case PPCISD::VECINSERT:       return "PPCISD::VECINSERT";
   case PPCISD::XXPERMDI:        return "PPCISD::XXPERMDI";
   case PPCISD::VECSHL:          return "PPCISD::VECSHL";
SDValue PPCTargetLowering::getSqrtInputTest(SDValue Op, SelectionDAG &DAG,
   //   - e_b is less than or equal to -970.
   // Otherwise fe_flag is set to 0.
   SDValue SRIdxVal = DAG.getTargetConstant(PPC::sub_eq, DL, MVT::i32);
   return SDValue(DAG.getMachineNode(TargetOpcode::EXTRACT_SUBREG, DL, MVT::i1,
                                     FTSQRT, SRIdxVal),
                  0);
 }
 
+SDValue
+PPCTargetLowering::getSqrtResultForDenormInput(SDValue Op,
+                                               SelectionDAG &DAG) const {
+  EVT VT = Op.getValueType();
+  if (VT != MVT::f64)
+    return SDValue();
+
+  return DAG.getNode(PPCISD::FSQRT, SDLoc(Op), VT, Op);
+}
+
 SDValue PPCTargetLowering::getSqrtEstimate(SDValue Operand, SelectionDAG &DAG,
                                            int Enabled, int &RefinementSteps,
                                            bool &UseOneConstNR,
                                            bool Reciprocal) const {
   EVT VT = Operand.getValueType();
   if ((VT == MVT::f32 && Subtarget.hasFRSQRTES()) ||
       (VT == MVT::f64 && Subtarget.hasFRSQRTE()) ||
       (VT == MVT::v4f32 && Subtarget.hasAltivec()) ||

llvm/lib/Target/PowerPC/PPCInstrInfo.td

 ]>;
 
 //===----------------------------------------------------------------------===//
 // PowerPC specific DAG Nodes.
 //
 
 def PPCfre    : SDNode<"PPCISD::FRE",     SDTFPUnaryOp, []>;
 def PPCfrsqrte: SDNode<"PPCISD::FRSQRTE", SDTFPUnaryOp, []>;
+def PPCfsqrt  : SDNode<"PPCISD::FSQRT",   SDTFPUnaryOp, []>;
 def PPCftsqrt : SDNode<"PPCISD::FTSQRT",  SDT_PPCFtsqrt,[]>;
 
 def PPCfcfid  : SDNode<"PPCISD::FCFID",   SDTFPUnaryOp, []>;
 def PPCfcfidu : SDNode<"PPCISD::FCFIDU",  SDTFPUnaryOp, []>;
 def PPCfcfids : SDNode<"PPCISD::FCFIDS",  SDTFPRoundOp, []>;
 def PPCfcfidus: SDNode<"PPCISD::FCFIDUS", SDTFPRoundOp, []>;
 def PPCfctidz : SDNode<"PPCISD::FCTIDZ", SDTFPUnaryOp, []>;
 def PPCfctiwz : SDNode<"PPCISD::FCTIWZ", SDTFPUnaryOp, []>;
defm FSQRT  : XForm_26r<63, 22, (outs f8rc:$frD), (ins f8rc:$frB),
                           [(set f64:$frD, (any_fsqrt f64:$frB))]>;
   defm FSQRTS : XForm_26r<59, 22, (outs f4rc:$frD), (ins f4rc:$frB),
                           "fsqrts", "$frD, $frB", IIC_FPSqrtS,
                           [(set f32:$frD, (any_fsqrt f32:$frB))]>;
   }
   }
 }
 
+def : Pat<(PPCfsqrt f64:$frA), (FSQRT $frA)>;
+
 /// Note that FMR is defined as pseudo-ops on the PPC970 because they are
 /// often coalesced away and we don't want the dispatch group builder to think
 /// that they will fill slots (which could cause the load of a LSU reject to
 /// sneak into a d-group with a store).
 let hasSideEffects = 0, Predicates = [HasFPU] in
 defm FMR   : XForm_26r<63, 72, (outs f4rc:$frD), (ins f4rc:$frB),
                        "fmr", "$frD, $frB", IIC_FPGeneral,
                        []>,  // (set f32:$frD, f32:$frB)

llvm/lib/Target/PowerPC/PPCInstrVSX.td

 def : Pat<(fma v2f64:$A, (fneg v2f64:$B), v2f64:$C),
           (XVNMSUBADP $C, $A, $B)>;
 
 def : Pat<(fma (fneg v4f32:$A), v4f32:$B, v4f32:$C),
           (XVNMSUBASP $C, $A, $B)>;
 def : Pat<(fma v4f32:$A, (fneg v4f32:$B), v4f32:$C),
           (XVNMSUBASP $C, $A, $B)>;
 
+def : Pat<(PPCfsqrt f64:$frA), (XSSQRTDP $frA)>;
+
 def : Pat<(v2f64 (bitconvert v4f32:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v4i32:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v8i16:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;
 def : Pat<(v2f64 (bitconvert v16i8:$A)),
           (COPY_TO_REGCLASS $A, VSRC)>;

llvm/test/CodeGen/PowerPC/fma-mutate.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 -mattr=+vsx -disable-ppc-vsx-fma-mutation=false | FileCheck %s
 
 declare double @llvm.sqrt.f64(double)
 
 ; Test several VSX FMA mutation opportunities.
 
 ; This is reasonable transformation since it eliminates extra register copy.
 define double @foo3_fmf(double %a) nounwind {
 ; CHECK-LABEL: foo3_fmf:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    xstsqrtdp 0, 1
-; CHECK-NEXT:    xxlxor 0, 0, 0
 ; CHECK-NEXT:    bc 12, 2, .LBB0_2
 ; CHECK-NEXT:  # %bb.1:
 ; CHECK-NEXT:    xsrsqrtedp 0, 1
 ; CHECK-NEXT:    addis 3, 2, .LCPI0_0@toc@ha
 ; CHECK-NEXT:    lfs 3, .LCPI0_0@toc@l(3)
 ; CHECK-NEXT:    addis 3, 2, .LCPI0_1@toc@ha
 ; CHECK-NEXT:    lfs 4, .LCPI0_1@toc@l(3)
 ; CHECK-NEXT:    xsmuldp 2, 1, 0
 ; CHECK-NEXT:    xsmaddmdp 2, 0, 3
 ; CHECK-NEXT:    xsmuldp 0, 0, 4
 ; CHECK-NEXT:    xsmuldp 0, 0, 2
 ; CHECK-NEXT:    xsmuldp 1, 1, 0
 ; CHECK-NEXT:    xsmaddadp 3, 1, 0
 ; CHECK-NEXT:    xsmuldp 0, 1, 4
-; CHECK-NEXT:    xsmuldp 0, 0, 3
+; CHECK-NEXT:    xsmuldp 1, 0, 3
+; CHECK-NEXT:    blr
 ; CHECK-NEXT:  .LBB0_2:
-; CHECK-NEXT:    fmr 1, 0
+; CHECK-NEXT:    xssqrtdp 1, 1
 ; CHECK-NEXT:    blr
   %r = call reassoc afn ninf double @llvm.sqrt.f64(double %a)
   ret double %r
 }
 
 define double @foo3_safe(double %a) nounwind {
 ; CHECK-LABEL: foo3_safe:
 ; CHECK:       # %bb.0:
 ; CHECK-NEXT:    xssqrtdp 1, 1
 ; CHECK-NEXT:    blr
   %r = call double @llvm.sqrt.f64(double %a)
   ret double %r
 }
 

llvm/test/CodeGen/PowerPC/recipest.ll

 ; CHECK-P7-NEXT:    fmul 0, 0, 4
 ; CHECK-P7-NEXT:    fmul 0, 0, 2
 ; CHECK-P7-NEXT:    fmul 1, 1, 0
 ; CHECK-P7-NEXT:    fmadd 0, 1, 0, 3
 ; CHECK-P7-NEXT:    fmul 1, 1, 4
 ; CHECK-P7-NEXT:    fmul 1, 1, 0
 ; CHECK-P7-NEXT:    blr
 ; CHECK-P7-NEXT:  .LBB20_2:
-; CHECK-P7-NEXT:    addis 3, 2, .LCPI20_2@toc@ha
-; CHECK-P7-NEXT:    lfs 1, .LCPI20_2@toc@l(3)
+; CHECK-P7-NEXT:    fsqrt 1, 1
 ; CHECK-P7-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: foo3_fmf:
 ; CHECK-P8:       # %bb.0:
 ; CHECK-P8-NEXT:    xstsqrtdp 0, 1
-; CHECK-P8-NEXT:    xxlxor 0, 0, 0
 ; CHECK-P8-NEXT:    bc 12, 2, .LBB20_2
 ; CHECK-P8-NEXT:  # %bb.1:
 ; CHECK-P8-NEXT:    xsrsqrtedp 0, 1
 ; CHECK-P8-NEXT:    addis 3, 2, .LCPI20_0@toc@ha
 ; CHECK-P8-NEXT:    lfs 3, .LCPI20_0@toc@l(3)
 ; CHECK-P8-NEXT:    addis 3, 2, .LCPI20_1@toc@ha
 ; CHECK-P8-NEXT:    lfs 4, .LCPI20_1@toc@l(3)
 ; CHECK-P8-NEXT:    fmr 5, 3
 ; CHECK-P8-NEXT:    xsmuldp 2, 1, 0
 ; CHECK-P8-NEXT:    xsmaddadp 5, 2, 0
 ; CHECK-P8-NEXT:    xsmuldp 0, 0, 4
 ; CHECK-P8-NEXT:    xsmuldp 0, 0, 5
 ; CHECK-P8-NEXT:    xsmuldp 1, 1, 0
 ; CHECK-P8-NEXT:    xsmaddadp 3, 1, 0
 ; CHECK-P8-NEXT:    xsmuldp 0, 1, 4
-; CHECK-P8-NEXT:    xsmuldp 0, 0, 3
+; CHECK-P8-NEXT:    xsmuldp 1, 0, 3
+; CHECK-P8-NEXT:    blr
 ; CHECK-P8-NEXT:  .LBB20_2:
-; CHECK-P8-NEXT:    fmr 1, 0
+; CHECK-P8-NEXT:    xssqrtdp 1, 1
 ; CHECK-P8-NEXT:    blr
 ;
 ; CHECK-P9-LABEL: foo3_fmf:
 ; CHECK-P9:       # %bb.0:
 ; CHECK-P9-NEXT:    xstsqrtdp 0, 1
-; CHECK-P9-NEXT:    xxlxor 0, 0, 0
 ; CHECK-P9-NEXT:    bc 12, 2, .LBB20_2
 ; CHECK-P9-NEXT:  # %bb.1:
 ; CHECK-P9-NEXT:    xsrsqrtedp 0, 1
 ; CHECK-P9-NEXT:    addis 3, 2, .LCPI20_0@toc@ha
 ; CHECK-P9-NEXT:    lfs 3, .LCPI20_0@toc@l(3)
 ; CHECK-P9-NEXT:    addis 3, 2, .LCPI20_1@toc@ha
 ; CHECK-P9-NEXT:    xsmuldp 2, 1, 0
 ; CHECK-P9-NEXT:    fmr 4, 3
 ; CHECK-P9-NEXT:    xsmaddadp 4, 2, 0
 ; CHECK-P9-NEXT:    lfs 2, .LCPI20_1@toc@l(3)
 ; CHECK-P9-NEXT:    xsmuldp 0, 0, 2
 ; CHECK-P9-NEXT:    xsmuldp 0, 0, 4
 ; CHECK-P9-NEXT:    xsmuldp 1, 1, 0
 ; CHECK-P9-NEXT:    xsmaddadp 3, 1, 0
 ; CHECK-P9-NEXT:    xsmuldp 0, 1, 2
-; CHECK-P9-NEXT:    xsmuldp 0, 0, 3
+; CHECK-P9-NEXT:    xsmuldp 1, 0, 3
+; CHECK-P9-NEXT:    blr
 ; CHECK-P9-NEXT:  .LBB20_2:
-; CHECK-P9-NEXT:    fmr 1, 0
+; CHECK-P9-NEXT:    xssqrtdp 1, 1
 ; CHECK-P9-NEXT:    blr
   %r = call reassoc ninf afn double @llvm.sqrt.f64(double %a)
   ret double %r
 }
 
 define double @foo3_safe(double %a) nounwind {
 ; CHECK-P7-LABEL: foo3_safe:
 ; CHECK-P7:       # %bb.0:

 ; CHECK-P9-NEXT:    blr
   %r = call <4 x float> @llvm.sqrt.v4f32(<4 x float> %a)
   ret <4 x float> %r
 }
 
 define <2 x double> @hoo4_fmf(<2 x double> %a) #1 {
 ; CHECK-P7-LABEL: hoo4_fmf:
 ; CHECK-P7:       # %bb.0:
-; CHECK-P7-NEXT:    addis 3, 2, .LCPI26_2@toc@ha
 ; CHECK-P7-NEXT:    ftsqrt 0, 1
-; CHECK-P7-NEXT:    fmr 3, 1
-; CHECK-P7-NEXT:    addis 4, 2, .LCPI26_0@toc@ha
-; CHECK-P7-NEXT:    lfs 0, .LCPI26_2@toc@l(3)
-; CHECK-P7-NEXT:    addis 3, 2, .LCPI26_1@toc@ha
-; CHECK-P7-NEXT:    lfs 5, .LCPI26_0@toc@l(4)
-; CHECK-P7-NEXT:    lfs 4, .LCPI26_1@toc@l(3)
-; CHECK-P7-NEXT:    fmr 1, 0
-; CHECK-P7-NEXT:    bc 4, 2, .LBB26_3
+; CHECK-P7-NEXT:    addis 3, 2, .LCPI26_0@toc@ha
+; CHECK-P7-NEXT:    addis 4, 2, .LCPI26_1@toc@ha
+; CHECK-P7-NEXT:    lfs 3, .LCPI26_0@toc@l(3)
+; CHECK-P7-NEXT:    lfs 0, .LCPI26_1@toc@l(4)
+; CHECK-P7-NEXT:    bc 12, 2, .LBB26_3
 ; CHECK-P7-NEXT:  # %bb.1:
+; CHECK-P7-NEXT:    frsqrte 4, 1
+; CHECK-P7-NEXT:    fmul 5, 1, 4
+; CHECK-P7-NEXT:    fmadd 5, 5, 4, 3
+; CHECK-P7-NEXT:    fmul 4, 4, 0
+; CHECK-P7-NEXT:    fmul 4, 4, 5
+; CHECK-P7-NEXT:    fmul 1, 1, 4
+; CHECK-P7-NEXT:    fmadd 4, 1, 4, 3
+; CHECK-P7-NEXT:    fmul 1, 1, 0
+; CHECK-P7-NEXT:    fmul 1, 1, 4
 ; CHECK-P7-NEXT:    ftsqrt 0, 2
 ; CHECK-P7-NEXT:    bc 4, 2, .LBB26_4
 ; CHECK-P7-NEXT:  .LBB26_2:
-; CHECK-P7-NEXT:    fmr 2, 0
+; CHECK-P7-NEXT:    fsqrt 2, 2
 ; CHECK-P7-NEXT:    blr
 ; CHECK-P7-NEXT:  .LBB26_3:
-; CHECK-P7-NEXT:    frsqrte 1, 3
-; CHECK-P7-NEXT:    fmul 6, 3, 1
-; CHECK-P7-NEXT:    fmadd 6, 6, 1, 5
-; CHECK-P7-NEXT:    fmul 1, 1, 4
-; CHECK-P7-NEXT:    fmul 1, 1, 6
-; CHECK-P7-NEXT:    fmul 3, 3, 1
-; CHECK-P7-NEXT:    fmadd 1, 3, 1, 5
-; CHECK-P7-NEXT:    fmul 3, 3, 4
-; CHECK-P7-NEXT:    fmul 1, 3, 1
+; CHECK-P7-NEXT:    fsqrt 1, 1
 ; CHECK-P7-NEXT:    ftsqrt 0, 2
 ; CHECK-P7-NEXT:    bc 12, 2, .LBB26_2
 ; CHECK-P7-NEXT:  .LBB26_4:
-; CHECK-P7-NEXT:    frsqrte 0, 2
-; CHECK-P7-NEXT:    fmul 3, 2, 0
-; CHECK-P7-NEXT:    fmadd 3, 3, 0, 5
-; CHECK-P7-NEXT:    fmul 0, 0, 4
-; CHECK-P7-NEXT:    fmul 0, 0, 3
-; CHECK-P7-NEXT:    fmul 2, 2, 0
-; CHECK-P7-NEXT:    fmadd 0, 2, 0, 5
+; CHECK-P7-NEXT:    frsqrte 4, 2
+; CHECK-P7-NEXT:    fmul 5, 2, 4
+; CHECK-P7-NEXT:    fmadd 5, 5, 4, 3
+; CHECK-P7-NEXT:    fmul 4, 4, 0
+; CHECK-P7-NEXT:    fmul 4, 4, 5
 ; CHECK-P7-NEXT:    fmul 2, 2, 4
+; CHECK-P7-NEXT:    fmadd 3, 2, 4, 3
 ; CHECK-P7-NEXT:    fmul 0, 2, 0
-; CHECK-P7-NEXT:    fmr 2, 0
+; CHECK-P7-NEXT:    fmul 2, 0, 3
 ; CHECK-P7-NEXT:    blr
 ;
 ; CHECK-P8-LABEL: hoo4_fmf:
 ; CHECK-P8:       # %bb.0:
 ; CHECK-P8-NEXT:    xvrsqrtedp 0, 34
 ; CHECK-P8-NEXT:    addis 3, 2, .LCPI26_0@toc@ha
 ; CHECK-P8-NEXT:    addi 3, 3, .LCPI26_0@toc@l
 ; CHECK-P8-NEXT:    lxvd2x 1, 0, 3