[PowerPC] Use the two-constant NR algorithm for refining estimates

[nemanjai]

Summary

The single-constant algorithm produces infinities on a lot of denormal values. The precision of the two-constant algorithm is actually sufficient across the range of denormals. We will switch to that algorithm for now to avoid the infinities on denormals. In the future, we will re-evaluate the algorithm to find the optimal one for PowerPC.

Example:

$ cat a.c 
#include <stdio.h>
#include <math.h>
float __attribute__((noinline)) test(float f) { return sqrtf(f); }
int main(void) {
  return printf("sqrt(0.49e-43): %g\n", test(0.49e-43));
}

$ clang -Ofast a.c
$ ./a.out 
sqrt(0.49e-43): -inf

Desired output (and output with this patch applied):

$ ./a.out 
sqrt(0.49e-43): -inf

We have also run this through a reasonable approximation of the gamut of tests (1,000,000 tests per exponent over the full single-precision range vs. the precise HW instruction). Here are the results from this test (courtesy of @renenkel):

 0 ulps:  72 %
  1 ulps:  27
  2 ulps:  0.032
  3 ulps:  0
 >3 ulps:  0.35

max error = 2 ulps over full range
except returns NaN for +Inf

[stefanp]

I have a couple of nits. The main one being the test case for fma-mutate.ll.
I'm not sure if there is a way to use one const NR for just that one test.

lib/Target/PowerPC/PPC.td

140 ↗

(On Diff #192994)

nit:
I think it is "Newton-Raphson" not "Newton-Rhapson".

lib/Target/PowerPC/PPCISelLowering.cpp

10988 ↗

(On Diff #192994)

Same, nit:
Newton-Raphson

test/CodeGen/PowerPC/fma-mutate.ll

17 ↗

(On Diff #192994)

With this new patch this test no longer really tests all of the things that the author wanted to test.

As the author of the test mentions, the first transformation is not reasonable. However, the potential opportunity for this first transformation is no longer there (the fmr instruction for it is gone) with two const Newton-Raphson. We can at least detect the second legal transformation with the CHECK-NOT.

; CHECK: @foo3
; CHECK-NOT: fmr
; CHECK: xsmaddmdp
; CHECK: xsmaddadp

[nemanjai]

Fixed the typos, added a check for no register move.

[stefanp]

I only have one comment here and I think it can be fixed when the patch is committed.
LGTM.

lib/Target/PowerPC/PPCSubtarget.h

101 ↗

(On Diff #197763)

I think that this variable needs to be initialized to false in void PPCSubtarget::initializeEnvironment().

[nemanjai]

lib/Target/PowerPC/PPC.td

140 ↗

(On Diff #192994)

Thanks. I'll fix it.

lib/Target/PowerPC/PPCSubtarget.h

101 ↗

(On Diff #197763)

Ah, good catch. I forgot about that.

llvm/trunk/lib/Target/PowerPC/PPC.td

 def FeaturePPC6xx    : SubtargetFeature<"ppc6xx", "IsPPC6xx", "true",
                                         "Enable PPC 6xx instructions">;
 def FeatureQPX       : SubtargetFeature<"qpx","HasQPX", "true",
                                         "Enable QPX instructions",
                                         [FeatureFPU]>;
 def FeatureVSX       : SubtargetFeature<"vsx","HasVSX", "true",
                                         "Enable VSX instructions",
                                         [FeatureAltivec]>;
+def FeatureTwoConstNR :
+  SubtargetFeature<"two-const-nr", "NeedsTwoConstNR", "true",
+                   "Requires two constant Newton-Raphson computation">;
 def FeatureP8Altivec : SubtargetFeature<"power8-altivec", "HasP8Altivec", "true",
                                         "Enable POWER8 Altivec instructions",
                                         [FeatureAltivec]>;
 def FeatureP8Crypto : SubtargetFeature<"crypto", "HasP8Crypto", "true",
                                        "Enable POWER8 Crypto instructions",
                                        [FeatureP8Altivec]>;
 def FeatureP8Vector  : SubtargetFeature<"power8-vector", "HasP8Vector", "true",
                                         "Enable POWER8 vector instructions",
list<SubtargetFeature> Power7FeatureList =
       [DirectivePwr7, FeatureAltivec, FeatureVSX,
        FeatureMFOCRF, FeatureFCPSGN, FeatureFSqrt, FeatureFRE,
        FeatureFRES, FeatureFRSQRTE, FeatureFRSQRTES,
        FeatureRecipPrec, FeatureSTFIWX, FeatureLFIWAX,
        FeatureFPRND, FeatureFPCVT, FeatureISEL,
        FeaturePOPCNTD, FeatureCMPB, FeatureLDBRX,
        Feature64Bit /*, Feature64BitRegs */,
        FeatureBPERMD, FeatureExtDiv,
-       FeatureMFTB, DeprecatedDST];
+       FeatureMFTB, DeprecatedDST, FeatureTwoConstNR];
   list<SubtargetFeature> Power8SpecificFeatures =
       [DirectivePwr8, FeatureP8Altivec, FeatureP8Vector, FeatureP8Crypto,
        FeatureHTM, FeatureDirectMove, FeatureICBT, FeaturePartwordAtomic,
        FeatureFusion];
   list<SubtargetFeature> Power8FeatureList =
       !listconcat(Power7FeatureList, Power8SpecificFeatures);
   list<SubtargetFeature> Power9SpecificFeatures =
       [DirectivePwr9, FeatureP9Altivec, FeatureP9Vector, FeatureISA3_0,

llvm/trunk/lib/Target/PowerPC/PPCISelLowering.cpp

if ((VT == MVT::f32 && Subtarget.hasFRSQRTES()) ||
       (VT == MVT::f64 && Subtarget.hasFRSQRTE()) ||
       (VT == MVT::v4f32 && Subtarget.hasAltivec()) ||
       (VT == MVT::v2f64 && Subtarget.hasVSX()) ||
       (VT == MVT::v4f32 && Subtarget.hasQPX()) ||
       (VT == MVT::v4f64 && Subtarget.hasQPX())) {
     if (RefinementSteps == ReciprocalEstimate::Unspecified)
       RefinementSteps = getEstimateRefinementSteps(VT, Subtarget);
 
-    UseOneConstNR = true;
+    // The Newton-Raphson computation with a single constant does not provide
+    // enough accuracy on some CPUs.
+    UseOneConstNR = !Subtarget.needsTwoConstNR();
     return DAG.getNode(PPCISD::FRSQRTE, SDLoc(Operand), VT, Operand);
   }
   return SDValue();
 }
 
 SDValue PPCTargetLowering::getRecipEstimate(SDValue Operand, SelectionDAG &DAG,
                                             int Enabled,
                                             int &RefinementSteps) const {

llvm/trunk/lib/Target/PowerPC/PPCSubtarget.h

protected:
   bool UseCRBits;
   bool HasHardFloat;
   bool IsPPC64;
   bool HasAltivec;
   bool HasFPU;
   bool HasSPE;
   bool HasQPX;
   bool HasVSX;
+  bool NeedsTwoConstNR;
   bool HasP8Vector;
   bool HasP8Altivec;
   bool HasP8Crypto;
   bool HasP9Vector;
   bool HasP9Altivec;
   bool HasFCPSGN;
   bool HasFSQRT;
   bool HasFRE, HasFRES, HasFRSQRTE, HasFRSQRTES;
public:
   bool hasLFIWAX() const { return HasLFIWAX; }
   bool hasFPRND() const { return HasFPRND; }
   bool hasFPCVT() const { return HasFPCVT; }
   bool hasAltivec() const { return HasAltivec; }
   bool hasSPE() const { return HasSPE; }
   bool hasFPU() const { return HasFPU; }
   bool hasQPX() const { return HasQPX; }
   bool hasVSX() const { return HasVSX; }
+  bool needsTwoConstNR() const { return NeedsTwoConstNR; }
   bool hasP8Vector() const { return HasP8Vector; }
   bool hasP8Altivec() const { return HasP8Altivec; }
   bool hasP8Crypto() const { return HasP8Crypto; }
   bool hasP9Vector() const { return HasP9Vector; }
   bool hasP9Altivec() const { return HasP9Altivec; }
   bool hasMFOCRF() const { return HasMFOCRF; }
   bool hasISEL() const { return HasISEL; }
   bool hasBPERMD() const { return HasBPERMD; }

llvm/trunk/lib/Target/PowerPC/PPCSubtarget.cpp

void PPCSubtarget::initializeEnvironment() {
   Use64BitRegs = false;
   UseCRBits = false;
   HasHardFloat = false;
   HasAltivec = false;
   HasSPE = false;
   HasFPU = false;
   HasQPX = false;
   HasVSX = false;
+  NeedsTwoConstNR = false;
   HasP8Vector = false;
   HasP8Altivec = false;
   HasP8Crypto = false;
   HasP9Vector = false;
   HasP9Altivec = false;
   HasFCPSGN = false;
   HasFSQRT = false;
   HasFRE = false;

llvm/trunk/test/CodeGen/PowerPC/fma-mutate.ll

 ; Test several VSX FMA mutation opportunities.  The first one isn't a
 ; reasonable transformation because the killed product register is the
 ; same as the FMA target register.  The second one is legal.  The third
 ; one doesn't fit the feeding-copy pattern.
 
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 -enable-unsafe-fp-math -mattr=+vsx -disable-ppc-vsx-fma-mutation=false | FileCheck %s
 target datalayout = "E-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-f128:128:128-v128:128:128-n32:64"
 target triple = "powerpc64-unknown-linux-gnu"
 
 declare double @llvm.sqrt.f64(double)
 
 define double @foo3(double %a) nounwind {
   %r = call double @llvm.sqrt.f64(double %a)
   ret double %r
 
 ; CHECK: @foo3
-; CHECK: fmr [[REG:[0-9]+]], [[REG2:[0-9]+]]
-; CHECK: xsnmsubadp [[REG]], {{[0-9]+}}, [[REG2]]
+; CHECK-NOT: fmr
 ; CHECK: xsmaddmdp
 ; CHECK: xsmaddadp
 }
 

llvm/trunk/test/CodeGen/PowerPC/fmf-propagation.ll

 
 define float @sqrt_afn(float %x) {
 ; FMF-LABEL: sqrt_afn:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    xxlxor 0, 0, 0
 ; FMF-NEXT:    fcmpu 0, 1, 0
 ; FMF-NEXT:    beq 0, .LBB10_2
 ; FMF-NEXT:  # %bb.1:
+; FMF-NEXT:    xsrsqrtesp 0, 1
 ; FMF-NEXT:    addis 3, 2, .LCPI10_0@toc@ha
-; FMF-NEXT:    xsrsqrtesp 3, 1
-; FMF-NEXT:    lfs 0, .LCPI10_0@toc@l(3)
-; FMF-NEXT:    xsmulsp 2, 1, 0
-; FMF-NEXT:    xsmulsp 4, 3, 3
-; FMF-NEXT:    xssubsp 2, 2, 1
-; FMF-NEXT:    xsmulsp 2, 2, 4
-; FMF-NEXT:    xssubsp 0, 0, 2
-; FMF-NEXT:    xsmulsp 0, 3, 0
-; FMF-NEXT:    xsmulsp 0, 0, 1
+; FMF-NEXT:    addis 4, 2, .LCPI10_1@toc@ha
+; FMF-NEXT:    lfs 2, .LCPI10_0@toc@l(3)
+; FMF-NEXT:    lfs 3, .LCPI10_1@toc@l(4)
+; FMF-NEXT:    xsmulsp 1, 1, 0
+; FMF-NEXT:    xsmulsp 0, 1, 0
+; FMF-NEXT:    xsmulsp 1, 1, 2
+; FMF-NEXT:    xsaddsp 0, 0, 3
+; FMF-NEXT:    xsmulsp 0, 1, 0
 ; FMF-NEXT:  .LBB10_2:
 ; FMF-NEXT:    fmr 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: sqrt_afn:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    xxlxor 0, 0, 0
 ; GLOBAL-NEXT:    fcmpu 0, 1, 0
 ; GLOBAL-NEXT:    beq 0, .LBB10_2
 ; GLOBAL-NEXT:  # %bb.1:
-; GLOBAL-NEXT:    xsrsqrtesp 2, 1
-; GLOBAL-NEXT:    fneg 0, 1
+; GLOBAL-NEXT:    xsrsqrtesp 0, 1
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI10_0@toc@ha
-; GLOBAL-NEXT:    fmr 4, 1
-; GLOBAL-NEXT:    lfs 3, .LCPI10_0@toc@l(3)
-; GLOBAL-NEXT:    xsmaddasp 4, 0, 3
-; GLOBAL-NEXT:    xsmulsp 0, 2, 2
-; GLOBAL-NEXT:    xsmaddasp 3, 4, 0
-; GLOBAL-NEXT:    xsmulsp 0, 2, 3
-; GLOBAL-NEXT:    xsmulsp 0, 0, 1
+; GLOBAL-NEXT:    addis 4, 2, .LCPI10_1@toc@ha
+; GLOBAL-NEXT:    lfs 2, .LCPI10_0@toc@l(3)
+; GLOBAL-NEXT:    lfs 3, .LCPI10_1@toc@l(4)
+; GLOBAL-NEXT:    xsmulsp 1, 1, 0
+; GLOBAL-NEXT:    xsmaddasp 2, 1, 0
+; GLOBAL-NEXT:    xsmulsp 0, 1, 3
+; GLOBAL-NEXT:    xsmulsp 0, 0, 2
 ; GLOBAL-NEXT:  .LBB10_2:
 ; GLOBAL-NEXT:    fmr 1, 0
 ; GLOBAL-NEXT:    blr
   %rt = call afn float @llvm.sqrt.f32(float %x)
   ret float %rt
 }
 
 ; The call is now fully 'fast'. This implies that approximation is allowed.
 
 ; FMFDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'sqrt_fast:'
 ; FMFDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; FMFDEBUG:       Type-legalized selection DAG: %bb.0 'sqrt_fast:'
 
 ; GLOBALDEBUG-LABEL: Optimized lowered selection DAG: %bb.0 'sqrt_fast:'
 ; GLOBALDEBUG:         fmul nnan ninf nsz arcp contract afn reassoc {{t[0-9]+}}
 ; GLOBALDEBUG:       Type-legalized selection DAG: %bb.0 'sqrt_fast:'
 
 define float @sqrt_fast(float %x) {
 ; FMF-LABEL: sqrt_fast:
 ; FMF:       # %bb.0:
 ; FMF-NEXT:    xxlxor 0, 0, 0
 ; FMF-NEXT:    fcmpu 0, 1, 0
 ; FMF-NEXT:    beq 0, .LBB11_2
 ; FMF-NEXT:  # %bb.1:
-; FMF-NEXT:    xsrsqrtesp 2, 1
-; FMF-NEXT:    fneg 0, 1
+; FMF-NEXT:    xsrsqrtesp 0, 1
 ; FMF-NEXT:    addis 3, 2, .LCPI11_0@toc@ha
-; FMF-NEXT:    fmr 4, 1
-; FMF-NEXT:    lfs 3, .LCPI11_0@toc@l(3)
-; FMF-NEXT:    xsmaddasp 4, 0, 3
-; FMF-NEXT:    xsmulsp 0, 2, 2
-; FMF-NEXT:    xsmaddasp 3, 4, 0
-; FMF-NEXT:    xsmulsp 0, 2, 3
-; FMF-NEXT:    xsmulsp 0, 0, 1
+; FMF-NEXT:    addis 4, 2, .LCPI11_1@toc@ha
+; FMF-NEXT:    lfs 2, .LCPI11_0@toc@l(3)
+; FMF-NEXT:    lfs 3, .LCPI11_1@toc@l(4)
+; FMF-NEXT:    xsmulsp 1, 1, 0
+; FMF-NEXT:    xsmaddasp 2, 1, 0
+; FMF-NEXT:    xsmulsp 0, 1, 3
+; FMF-NEXT:    xsmulsp 0, 0, 2
 ; FMF-NEXT:  .LBB11_2:
 ; FMF-NEXT:    fmr 1, 0
 ; FMF-NEXT:    blr
 ;
 ; GLOBAL-LABEL: sqrt_fast:
 ; GLOBAL:       # %bb.0:
 ; GLOBAL-NEXT:    xxlxor 0, 0, 0
 ; GLOBAL-NEXT:    fcmpu 0, 1, 0
 ; GLOBAL-NEXT:    beq 0, .LBB11_2
 ; GLOBAL-NEXT:  # %bb.1:
-; GLOBAL-NEXT:    xsrsqrtesp 2, 1
-; GLOBAL-NEXT:    fneg 0, 1
+; GLOBAL-NEXT:    xsrsqrtesp 0, 1
 ; GLOBAL-NEXT:    addis 3, 2, .LCPI11_0@toc@ha
-; GLOBAL-NEXT:    fmr 4, 1
-; GLOBAL-NEXT:    lfs 3, .LCPI11_0@toc@l(3)
-; GLOBAL-NEXT:    xsmaddasp 4, 0, 3
-; GLOBAL-NEXT:    xsmulsp 0, 2, 2
-; GLOBAL-NEXT:    xsmaddasp 3, 4, 0
-; GLOBAL-NEXT:    xsmulsp 0, 2, 3
-; GLOBAL-NEXT:    xsmulsp 0, 0, 1
+; GLOBAL-NEXT:    addis 4, 2, .LCPI11_1@toc@ha
+; GLOBAL-NEXT:    lfs 2, .LCPI11_0@toc@l(3)
+; GLOBAL-NEXT:    lfs 3, .LCPI11_1@toc@l(4)
+; GLOBAL-NEXT:    xsmulsp 1, 1, 0
+; GLOBAL-NEXT:    xsmaddasp 2, 1, 0
+; GLOBAL-NEXT:    xsmulsp 0, 1, 3
+; GLOBAL-NEXT:    xsmulsp 0, 0, 2
 ; GLOBAL-NEXT:  .LBB11_2:
 ; GLOBAL-NEXT:    fmr 1, 0
 ; GLOBAL-NEXT:    blr
   %rt = call fast float @llvm.sqrt.f32(float %x)
   ret float %rt
 }
 
 ; fcmp can have fast-math-flags.

llvm/trunk/test/CodeGen/PowerPC/recipest.ll

 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 -enable-unsafe-fp-math -mattr=-vsx | FileCheck %s
 ; RUN: llc -verify-machineinstrs < %s -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr7 -mattr=-vsx | FileCheck -check-prefix=CHECK-SAFE %s
 
 target datalayout = "E-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-f128:128:128-v128:128:128-n32:64"
 target triple = "powerpc64-unknown-linux-gnu"
 
 declare double @llvm.sqrt.f64(double)
 declare float @llvm.sqrt.f32(float)
 declare <4 x float> @llvm.sqrt.v4f32(<4 x float>)
 
 define double @foo(double %a, double %b) nounwind {
   %x = call double @llvm.sqrt.f64(double %b)
   %r = fdiv double %a, %x
   ret double %r
 
 ; CHECK: @foo
-; CHECK-DAG: frsqrte
-; CHECK-DAG: fnmsub
+; CHECK: frsqrte
 ; CHECK: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmul
+; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmul
+; CHECK-NEXT: fmul
 ; CHECK: blr
 
 ; CHECK-SAFE: @foo
 ; CHECK-SAFE: fsqrt
 ; CHECK-SAFE: fdiv
 ; CHECK-SAFE: blr
 }
 

 define double @foof(double %a, float %b) nounwind {
   %x = call float @llvm.sqrt.f32(float %b)
   %y = fpext float %x to double
   %r = fdiv double %a, %y
   ret double %r
 
 ; CHECK: @foof
 ; CHECK-DAG: frsqrtes
-; CHECK-DAG: fnmsubs
 ; CHECK: fmuls
 ; CHECK-NEXT: fmadds
 ; CHECK-NEXT: fmuls
+; CHECK-NEXT: fmuls
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: blr
 
 ; CHECK-SAFE: @foof
 ; CHECK-SAFE: fsqrts
 ; CHECK-SAFE: fdiv
 ; CHECK-SAFE: blr
 }
 
 define float @food(float %a, double %b) nounwind {
   %x = call double @llvm.sqrt.f64(double %b)
   %y = fptrunc double %x to float
   %r = fdiv float %a, %y
   ret float %r
 
 ; CHECK: @foo
 ; CHECK-DAG: frsqrte
-; CHECK-DAG: fnmsub
 ; CHECK: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmul
+; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
+; CHECK-NEXT: fmul
 ; CHECK-NEXT: frsp
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: blr
 
 ; CHECK-SAFE: @foo
 ; CHECK-SAFE: fsqrt
 ; CHECK-SAFE: fdivs
 ; CHECK-SAFE: blr
 }
 
 define float @goo(float %a, float %b) nounwind {
   %x = call float @llvm.sqrt.f32(float %b)
   %r = fdiv float %a, %x
   ret float %r
 
 ; CHECK: @goo
 ; CHECK-DAG: frsqrtes
-; CHECK-DAG: fnmsubs
 ; CHECK: fmuls
 ; CHECK-NEXT: fmadds
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: fmuls
+; CHECK-NEXT: fmuls
 ; CHECK-NEXT: blr
 
 ; CHECK-SAFE: @goo
 ; CHECK-SAFE: fsqrts
 ; CHECK-SAFE: fdivs
 ; CHECK-SAFE: blr
 }
 
define float @rsqrt_fmul(float %a, float %b, float %c) {
   %z = fdiv float %c, %y
   ret float %z
 
 ; CHECK: @rsqrt_fmul
 ; CHECK-DAG: frsqrtes
 ; CHECK-DAG: fres
 ; CHECK-DAG: fnmsubs
 ; CHECK-DAG: fmuls
-; CHECK-DAG: fnmsubs
 ; CHECK-DAG: fmadds
 ; CHECK-DAG: fmadds
 ; CHECK: fmuls
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: blr
 
 ; CHECK-SAFE: @rsqrt_fmul

 
 define double @foo3(double %a) nounwind {
   %r = call double @llvm.sqrt.f64(double %a)
   ret double %r
 
 ; CHECK: @foo3
 ; CHECK: fcmpu
 ; CHECK-DAG: frsqrte
-; CHECK-DAG: fnmsub
 ; CHECK: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmul
+; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmadd
 ; CHECK-NEXT: fmul
 ; CHECK-NEXT: fmul
 ; CHECK: blr
 
 ; CHECK-SAFE: @foo3
 ; CHECK-SAFE: fsqrt
 ; CHECK-SAFE: blr
 }
 
 define float @goo3(float %a) nounwind {
   %r = call float @llvm.sqrt.f32(float %a)
   ret float %r
 
 ; CHECK: @goo3
 ; CHECK: fcmpu
 ; CHECK-DAG: frsqrtes
-; CHECK-DAG: fnmsubs
 ; CHECK: fmuls
 ; CHECK-NEXT: fmadds
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: fmuls
 ; CHECK: blr
 
 ; CHECK-SAFE: @goo3
 ; CHECK-SAFE: fsqrts

llvm/trunk/test/CodeGen/PowerPC/vsx-fma-mutate-trivial-copy.ll

 ; RUN: llc -verify-machineinstrs < %s | FileCheck %s
 target datalayout = "E-m:e-i64:64-n32:64"
 target triple = "powerpc64-unknown-linux-gnu"
 
 ; Function Attrs: nounwind
 define void @LSH_recall_init(float %d_min, float %W) #0 {
 entry:
   br i1 undef, label %for.body.lr.ph, label %for.end
 
 ; CHECK-LABEL: @LSH_recall_init
-; CHECK: xsnmsubadp
+; CHECK: xsmaddadp
 
 for.body.lr.ph:                                   ; preds = %entry
   %conv3 = fpext float %W to double
   br label %for.body
 
 for.body:                                         ; preds = %for.body, %for.body.lr.ph
   %div = fdiv fast float 0.000000e+00, 0.000000e+00
   %add = fadd fast float %div, %d_min