Improve division estimation of floating points.

[qiucf]

Summary

Current implementation of _fast_ division (A/B) is to:

1. Get an initial estimation of reciprocal of B
2. Use Newton's iteration method to improve the reciprocal
3. Multiply the estimation with A

Compared with GCC, this loses some precision since multiplication is done after all iterations.

This patch is to do multiplication before the last iteration to make the result more accurate. It won't add/change any existing nodes/instructions except reordering calculation.

[lebedev.ri]

Missing test coverage; performance/precision overview - how much of performance do we loose for how much extra precision, comparison with native division?

[xbolva00]

ICC behaviour?

[nemanjai]

I suppose this is essentially an RFC to gauge the community's interest in improving the algorithm to gain more precision.
I think that in order to pull the trigger on a change such as this we would need the following to happen:

1. The patch needs full context to be reviewable
2. The formatting needs to be fixed up (some lines too long, etc.)
3. We need testing (I imagine this makes a bunch of LIT tests fail which need to be updated)
4. b) It might not be a bad idea to add a test to test-suite that will do some fast division vs. non-fast division to make sure the accuracy isn't too bad
5. We should do some thorough analysis of the accuracy of the algorithm vs. the HW implementation on a wide range of values on a couple of targets (as was suggested above). I know this was done on PPC, so please share those results and try to replicate on another easily available target (such as X86).

[qiucf]

In D66050#1624254, @lebedev.ri wrote:

Missing test coverage; performance/precision overview - how much of performance do we loose for how much extra precision, comparison with native division?

The patch won't affect the performance by either comparing running time or analyzing it in theory, since actually it just does a re-order for them, producing no more DAG nodes/instructions.

Compared with native division (tested with our own math library on PPC), the maximum error from estimation by GCC is 0.5ulp, while that of Clang is >4ulp.

In D66050#1624257, @xbolva00 wrote:

ICC behaviour?

Do you mean the behavior is the same as ICC? I've not tested it, but the patch is to make result as precise as GCC.

[qiucf]

In D66050#1624955, @nemanjai wrote:

1. The patch needs full context to be reviewable
2. The formatting needs to be fixed up (some lines too long, etc.)
3. We need testing (I imagine this makes a bunch of LIT tests fail which need to be updated)

I'm updating the patch :) Broken LIT tests aren't many.

3. It might not be a bad idea to add a test to test-suite that will do some fast division vs. non-fast division to make sure the accuracy isn't too bad

I'll add a portable case can be compiled and run at all platforms. That can be merged into our test-suite.

4. We should do some thorough analysis of the accuracy of the algorithm vs. the HW implementation on a wide range of values on a couple of targets (as was suggested above). I know this was done on PPC, so please share those results and try to replicate on another easily available target (such as X86).

Yes. But actually only a few platforms have support for such estimation. Clang doesn't do it on X86 but GCC does. I'll add the author to reviewer list.

[qiucf]

Update patch format.

[spatel]

x86 is moving away from reciprocal estimate code because recent hardware implements a real fdiv at about the same speed as reciprocal estimate+refinement. So x86 perf probably isn't that big of a concern, but it would be good to see the regression test diffs (use the auto-generation script to update those files).

I don't have a good sense of the accuracy gains/trade-offs from pulling the numerator into the estimate calc. Ping @scanon for better insight/data on that.

[qiucf]

Fix broken LIT tests.

[xbolva00]

(Regressions)

llvm/test/CodeGen/PowerPC/combine-fneg.ll

19 ↗

(On Diff #215542)

Regression

llvm/test/CodeGen/X86/recip-fastmath2.ll

602 ↗

(On Diff #215542)

Extra instruction

[spatel]

llvm/test/CodeGen/X86/recip-fastmath2.ll

602 ↗

(On Diff #215542)

It is an extra instruction, but it's not an extra uop. We're loading the constant numerator into a register because it is used twice now.

My guess is that fdiv with a constant isn't the usual case. We should have tests with 2 variable operands (fdiv x, y).

That might show that we save an instruction/uop with this patch because we don't have to load the "1.0" constant.

[spatel]

@qiucf - please rebase and update for test changes after rL369106.

[spatel]

In D66050#1633020, @spatel wrote:

@qiucf - please rebase and update for test changes after rL369106.

Oops - make that rebase for after rL369107.

[qiucf]

Rebase to fix regression.

[qiucf]

I posted a simple test for fp division on GitHub.
This patch helps on the accuracy especially when compared with GCC. :)
@nemanjai @lebedev.ri

[hfinkel]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

20080 ↗

(On Diff #215779)

"last time of iteration" -> "last iteration"

"try taking numerator into consideration" -> "also multiply by the numerator"

llvm/test/CodeGen/PowerPC/combine-fneg.ll

19 ↗

(On Diff #215542)

There's still an extra arithmetic instruction here?

[qiucf]

llvm/test/CodeGen/PowerPC/combine-fneg.ll

19 ↗

(On Diff #215542)

It's as expected. Since visitFDIV calls combineRepeatedFPDivisors to transform the vector divisions before BuildDivEstimate.

But the extra instructions are really redundant here. If visitFMA folds (fma (fneg a) (fneg b) c) into (fma a b c) like what visitFMUL does, the extra instructions will get eliminated. This can be done in future patch.

[qiucf]

Fix typo and rebase.

[spatel]

llvm/test/CodeGen/PowerPC/combine-fneg.ll

19 ↗

(On Diff #215542)

Did rL370071 solve that?

[qiucf]

llvm/test/CodeGen/PowerPC/combine-fneg.ll

19 ↗

(On Diff #215542)

Yes, there're no more instructions now.

[qiucf]

Update test to reflect changes introduced in rL370071.

[spatel]

LGTM

[lebedev.ri]

In D66050#1624955, @nemanjai wrote:

3. b) It might not be a bad idea to add a test to test-suite that will do some fast division vs. non-fast division to make sure the accuracy isn't too bad
4. We should do some thorough analysis of the accuracy of the algorithm vs. the HW implementation on a wide range of values on a couple of targets (as was suggested above). I know this was done on PPC, so please share those results and try to replicate on another easily available target (such as X86).

I think these two points weren't addressed.
I'd like to see at least some publicly-stated numbers on accuracy,
just so we all know this is going in the right direction for all inputs.

[spatel]

In D66050#1654733, @lebedev.ri wrote:

In D66050#1624955, @nemanjai wrote:

3. b) It might not be a bad idea to add a test to test-suite that will do some fast division vs. non-fast division to make sure the accuracy isn't too bad
4. We should do some thorough analysis of the accuracy of the algorithm vs. the HW implementation on a wide range of values on a couple of targets (as was suggested above). I know this was done on PPC, so please share those results and try to replicate on another easily available target (such as X86).

I think these two points weren't addressed.
I'd like to see at least some publicly-stated numbers on accuracy,
just so we all know this is going in the right direction for all inputs.

Changing my 'accepted' until this is answered.

The test at:
https://github.com/ecnelises/fp-division-test/
...seems to do a small random sampling.

The original transform was tested on x86 using brute force for all possible floats (1.0f/x) and is attached here:
https://bugs.llvm.org/show_bug.cgi?id=21385

I'm not sure how to prove this, but by distributing the multiplication into the last step of the estimate, I think we are always trading better accuracy around the numerator value with potentially overflowing to infinity for extremely different numerator/denominator. That's a good trade-off IMO and within the loosely-defined behavior enabled by 'arcp' in LLVM and '-mrecip' with Clang.

[qiucf]

In D66050#1654762, @spatel wrote:

In D66050#1654733, @lebedev.ri wrote:

I think these two points weren't addressed.
I'd like to see at least some publicly-stated numbers on accuracy,
just so we all know this is going in the right direction for all inputs.

Changing my 'accepted' until this is answered.

The test at:
https://github.com/ecnelises/fp-division-test/
...seems to do a small random sampling.

The original transform was tested on x86 using brute force for all possible floats (1.0f/x) and is attached here:
https://bugs.llvm.org/show_bug.cgi?id=21385

I'm not sure how to prove this, but by distributing the multiplication into the last step of the estimate, I think we are always trading better accuracy around the numerator value with potentially overflowing to infinity for extremely different numerator/denominator. That's a good trade-off IMO and within the loosely-defined behavior enabled by 'arcp' in LLVM and '-mrecip' with Clang.

Thanks for test case in PR21385. I'll write tests on a wider range of numbers. We, from my point of view, need two kind of tests:

• A compiler-independent program showing _distributing the multiplication into the last step of estimation_ is really more accurate. It shoule be just like the case you showed.
• A program with functions optimized at different level (e.g. -Ofast and -O3) comparing results of them with real divisions. This can originate from my previous fp-division-test. I think this is suitable for test suites.

Result of test should include:

• Accuracy (< 2ulp?) rate compared with real divisions.
• Accuracy rate compared with current implementation.
• Accuracy rate compared with other implementations, such as GCC.

A problem here: iterate from 0x00800000 to 0x7E800000 is acceptable for testing reciprocals, but not for testing divisions (n^2). I'm not sure changing iteration step from 1 to 10, 100 or larger to reduce running time is okay.

[qiucf]

I updated a test for testing this new way of division estimations. It's posted at https://github.com/ecnelises/fp-division-test/blob/master/algorithm_test.c so people can do test by their own. Here are my accuracy results:

(Here OLD means do iteration once and multiply)

• OLD: 79.47% (7600728294/9563620427), NEW: 99.92% (9555976854/9563620427) with enumeration steps 21277, 21961
• OLD: 79.51% (17982192012/22615809720), NEW: 99.92% (22597759561/22615809720) with enumeration steps 11587, 17053
• OLD: 79.52% (72788895710/91532258037), NEW: 99.92% (73098710/91532258037) with enumeration steps 6037, 8087
• OLD: 79.50% (175651525048/220935684644), NEW: 99.92% (220759033938/220935684644), with enumeration steps 5471, 3697
• OLD: 79.53% (1807487453401/2272767717744), NEW: 99.92% (2270951449072/2272767717744), with enumeration steps 1523, 1291
• OLD: 79.52% (4700786248799/5911547429304), NEW: 99.92% (5906822137336/5911547429304), with enumeration steps 983, 769

Inaccurate stands for that error is larger than 0.5ulp. Also I have some other sources in that repository which can be used to test division accuracy after Ofast. I think they can be good test suite candidate.

Results here are quite stable to show that the new way has better accuracy.

[qiucf]

Update patch to fix check regressions from recent commits.

[lebedev.ri]

In D66050#1658902, @qiucf wrote:

I updated a test for testing this new way of division estimations. It's posted at https://github.com/ecnelises/fp-division-test/blob/master/algorithm_test.c so people can do test by their own. Here are my accuracy results:

(Here OLD means do iteration once and multiply)

• OLD: 79.47% (7600728294/9563620427), NEW: 99.92% (9555976854/9563620427) with enumeration steps 21277, 21961
• OLD: 79.51% (17982192012/22615809720), NEW: 99.92% (22597759561/22615809720) with enumeration steps 11587, 17053
• OLD: 79.52% (72788895710/91532258037), NEW: 99.92% (73098710/91532258037) with enumeration steps 6037, 8087
• OLD: 79.50% (175651525048/220935684644), NEW: 99.92% (220759033938/220935684644), with enumeration steps 5471, 3697
• OLD: 79.53% (1807487453401/2272767717744), NEW: 99.92% (2270951449072/2272767717744), with enumeration steps 1523, 1291
• OLD: 79.52% (4700786248799/5911547429304), NEW: 99.92% (5906822137336/5911547429304), with enumeration steps 983, 769

Inaccurate stands for that error is larger than 0.5ulp. Also I have some other sources in that repository which can be used to test division accuracy after Ofast. I think they can be good test suite candidate.

Results here are quite stable to show that the new way has better accuracy.

Thank you for posting these.
I don't know if the step is fine-grained enough, but these numbers look good.

[spatel]

LGTM for x86. I tried the test program on Haswell and see the expected improvements (better results with FMA as also expected).
cc'ing some potential AMDGPU reviewers.

[rampitec]

AMDGPU test changes looks good.

[jsji]

Please include *Full Context* in newer diff . Thanks.

[qiucf]

Upload patch with full context.

[qiucf]

Remove unexpected changed file caused by newline characters.

[nemanjai]

It would appear that we are converging here. Both @jsji and @spatel are presumably in the "Must Review" for this. Would you guys mind taking another look as your approval is necessary for this to proceed.

[spatel]

LGTM

[jsji]

LGTM.

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

private:
     SDValue CombineZExtLogicopShiftLoad(SDNode *N);
     SDValue combineRepeatedFPDivisors(SDNode *N);
     SDValue combineInsertEltToShuffle(SDNode *N, unsigned InsIndex);
     SDValue ConstantFoldBITCASTofBUILD_VECTOR(SDNode *, EVT);
     SDValue BuildSDIV(SDNode *N);
     SDValue BuildSDIVPow2(SDNode *N);
     SDValue BuildUDIV(SDNode *N);
     SDValue BuildLogBase2(SDValue V, const SDLoc &DL);
-    SDValue BuildReciprocalEstimate(SDValue Op, SDNodeFlags Flags);
+    SDValue BuildDivEstimate(SDValue N, SDValue Op, SDNodeFlags Flags);
     SDValue buildRsqrtEstimate(SDValue Op, SDNodeFlags Flags);
     SDValue buildSqrtEstimate(SDValue Op, SDNodeFlags Flags);
     SDValue buildSqrtEstimateImpl(SDValue Op, SDNodeFlags Flags, bool Recip);
     SDValue buildSqrtNROneConst(SDValue Arg, SDValue Est, unsigned Iterations,
                                 SDNodeFlags Flags, bool Reciprocal);
     SDValue buildSqrtNRTwoConst(SDValue Arg, SDValue Est, unsigned Iterations,
                                 SDNodeFlags Flags, bool Reciprocal);
     SDValue MatchBSwapHWordLow(SDNode *N, SDValue N0, SDValue N1,
if (N1.getOpcode() == ISD::FSQRT) {
           RV = DAG.getNode(ISD::FDIV, SDLoc(N1), VT, RV, OtherOp, Flags);
           AddToWorklist(RV.getNode());
           return DAG.getNode(ISD::FMUL, DL, VT, N0, RV, Flags);
         }
       }
     }
 
     // Fold into a reciprocal estimate and multiply instead of a real divide.
-    if (SDValue RV = BuildReciprocalEstimate(N1, Flags)) {
-      AddToWorklist(RV.getNode());
-      return DAG.getNode(ISD::FMUL, DL, VT, N0, RV, Flags);
-    }
+    if (SDValue RV = BuildDivEstimate(N0, N1, Flags))
+      return RV;
   }
 
   // (fdiv (fneg X), (fneg Y)) -> (fdiv X, Y)
   if (char LHSNeg = isNegatibleForFree(N0, LegalOperations, TLI, &Options,
                                        ForCodeSize)) {
     if (char RHSNeg = isNegatibleForFree(N1, LegalOperations, TLI, &Options,
                                          ForCodeSize)) {
       // Both can be negated for free, check to see if at least one is cheaper

 }
 
 /// Newton iteration for a function: F(X) is X_{i+1} = X_i - F(X_i)/F'(X_i)
 /// For the reciprocal, we need to find the zero of the function:
 ///   F(X) = A X - 1 [which has a zero at X = 1/A]
 ///     =>
 ///   X_{i+1} = X_i (2 - A X_i) = X_i + X_i (1 - A X_i) [this second form
 ///     does not require additional intermediate precision]
-SDValue DAGCombiner::BuildReciprocalEstimate(SDValue Op, SDNodeFlags Flags) {
+/// For the last iteration, put numerator N into it to gain more precision:
+///   Result = N X_i + X_i (N - N A X_i)
+SDValue DAGCombiner::BuildDivEstimate(SDValue N, SDValue Op,
+                                      SDNodeFlags Flags) {
   if (Level >= AfterLegalizeDAG)
     return SDValue();
 
   // TODO: Handle half and/or extended types?
   EVT VT = Op.getValueType();
   if (VT.getScalarType() != MVT::f32 && VT.getScalarType() != MVT::f64)
     return SDValue();
 
   // If estimates are explicitly disabled for this function, we're done.
   MachineFunction &MF = DAG.getMachineFunction();
   int Enabled = TLI.getRecipEstimateDivEnabled(VT, MF);
   if (Enabled == TLI.ReciprocalEstimate::Disabled)
     return SDValue();
 
   // Estimates may be explicitly enabled for this type with a custom number of
   // refinement steps.
   int Iterations = TLI.getDivRefinementSteps(VT, MF);
   if (SDValue Est = TLI.getRecipEstimate(Op, DAG, Enabled, Iterations)) {
     AddToWorklist(Est.getNode());
 
-    if (Iterations) {
-      SDLoc DL(Op);
+    if (Iterations) {
       SDValue FPOne = DAG.getConstantFP(1.0, DL, VT);
 
-      // Newton iterations: Est = Est + Est (1 - Arg * Est)
+      // Newton iterations: Est = Est + Est (N - Arg * Est)
+      // If this is the last iteration, also multiply by the numerator.
       for (int i = 0; i < Iterations; ++i) {
-        SDValue NewEst = DAG.getNode(ISD::FMUL, DL, VT, Op, Est, Flags);
-        NewEst = DAG.getNode(ISD::FSUB, DL, VT, FPOne, NewEst, Flags);
+        SDValue MulEst = Est;
+
+        if (i == Iterations - 1) {
+          MulEst = DAG.getNode(ISD::FMUL, DL, VT, N, Est, Flags);
+          AddToWorklist(MulEst.getNode());
+        }
+
+        SDValue NewEst = DAG.getNode(ISD::FMUL, DL, VT, Op, MulEst, Flags);
+        AddToWorklist(NewEst.getNode());
+
+        NewEst = DAG.getNode(ISD::FSUB, DL, VT,
+                             (i == Iterations - 1 ? N : FPOne), NewEst, Flags);
+        AddToWorklist(NewEst.getNode());
+
         NewEst = DAG.getNode(ISD::FMUL, DL, VT, Est, NewEst, Flags);
-        Est = DAG.getNode(ISD::FADD, DL, VT, Est, NewEst, Flags);
+        AddToWorklist(NewEst.getNode());
+
+        Est = DAG.getNode(ISD::FADD, DL, VT, MulEst, NewEst, Flags);
+        AddToWorklist(Est.getNode());
       }
+    } else {
+      // If no iterations are available, multiply with N.
+      Est = DAG.getNode(ISD::FMUL, DL, VT, Est, N, Flags);
+      AddToWorklist(Est.getNode());
     }
+
     return Est;
   }
 
   return SDValue();
 }
 
 /// Newton iteration for a function: F(X) is X_{i+1} = X_i - F(X_i)/F'(X_i)
 /// For the reciprocal sqrt, we need to find the zero of the function:

llvm/trunk/test/CodeGen/AMDGPU/fdiv.ll

 entry:
   %fdiv = fdiv fast float %a, %b
   store float %fdiv, float addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_f32_fast_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[2].W
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[2].Z, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[2].Z,
 
 ; GCN: v_rcp_f32_e32 [[RCP:v[0-9]+]], s{{[0-9]+}}
 ; GCN: v_mul_f32_e32 [[RESULT:v[0-9]+]], s{{[0-9]+}}, [[RCP]]
 ; GCN-NOT: [[RESULT]]
 ; GCN: buffer_store_dword [[RESULT]]
 define amdgpu_kernel void @fdiv_f32_fast_math(float addrspace(1)* %out, float %a, float %b) #0 {
 entry:
   %fdiv = fdiv fast float %a, %b
   store float %fdiv, float addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_ulp25_f32_fast_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[2].W
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[2].Z, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[2].Z,
 
 ; GCN: v_rcp_f32_e32 [[RCP:v[0-9]+]], s{{[0-9]+}}
 ; GCN: v_mul_f32_e32 [[RESULT:v[0-9]+]], s{{[0-9]+}}, [[RCP]]
 ; GCN-NOT: [[RESULT]]
 ; GCN: buffer_store_dword [[RESULT]]
 define amdgpu_kernel void @fdiv_ulp25_f32_fast_math(float addrspace(1)* %out, float %a, float %b) #0 {
 entry:
   %fdiv = fdiv fast float %a, %b, !fpmath !0
   store float %fdiv, float addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_f32_arcp_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[2].W
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[2].Z, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[2].Z,
 
 ; GCN: v_rcp_f32_e32 [[RCP:v[0-9]+]], s{{[0-9]+}}
 ; GCN: v_mul_f32_e32 [[RESULT:v[0-9]+]], s{{[0-9]+}}, [[RCP]]
 ; GCN-NOT: [[RESULT]]
 ; GCN: buffer_store_dword [[RESULT]]
 define amdgpu_kernel void @fdiv_f32_arcp_math(float addrspace(1)* %out, float %a, float %b) #0 {
 entry:
   %fdiv = fdiv arcp float %a, %b
entry:
   %fdiv = fdiv arcp <2 x float> %a, %b, !fpmath !0
   store <2 x float> %fdiv, <2 x float> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_v2f32_fast_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[3].Z
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[3].Y
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[3].X, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[2].W, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[3].X,
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[2].W,
 
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 define amdgpu_kernel void @fdiv_v2f32_fast_math(<2 x float> addrspace(1)* %out, <2 x float> %a, <2 x float> %b) #0 {
 entry:
   %fdiv = fdiv fast <2 x float> %a, %b
   store <2 x float> %fdiv, <2 x float> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_v2f32_arcp_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[3].Z
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW]}}, KC0[3].Y
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[3].X, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, KC0[2].W, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[3].X,
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW]}}, PS, KC0[2].W,
 
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 define amdgpu_kernel void @fdiv_v2f32_arcp_math(<2 x float> addrspace(1)* %out, <2 x float> %a, <2 x float> %b) #0 {
 entry:
   %fdiv = fdiv arcp <2 x float> %a, %b
   store <2 x float> %fdiv, <2 x float> addrspace(1)* %out
   ret void
define amdgpu_kernel void @fdiv_v4f32(<4 x float> addrspace(1)* %out, <4 x float> addrspace(1)* %in) #0 {
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_v4f32_fast_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
 
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 define amdgpu_kernel void @fdiv_v4f32_fast_math(<4 x float> addrspace(1)* %out, <4 x float> addrspace(1)* %in) #0 {
   %b_ptr = getelementptr <4 x float>, <4 x float> addrspace(1)* %in, i32 1
   %a = load <4 x float>, <4 x float> addrspace(1) * %in
   %b = load <4 x float>, <4 x float> addrspace(1) * %b_ptr
   %result = fdiv fast <4 x float> %a, %b
   store <4 x float> %result, <4 x float> addrspace(1)* %out
   ret void
 }
 
 ; FUNC-LABEL: {{^}}fdiv_v4f32_arcp_math:
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
 ; R600-DAG: RECIP_IEEE * T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
-; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], T[0-9]+\.[XYZW]}}, PS
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
+; R600-DAG: MUL_IEEE {{\** *}}T{{[0-9]+\.[XYZW], PS, T[0-9]+\.[XYZW]}},
 
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 ; GCN: v_rcp_f32
 define amdgpu_kernel void @fdiv_v4f32_arcp_math(<4 x float> addrspace(1)* %out, <4 x float> addrspace(1)* %in) #0 {
   %b_ptr = getelementptr <4 x float>, <4 x float> addrspace(1)* %in, i32 1
   %a = load <4 x float>, <4 x float> addrspace(1) * %in

llvm/trunk/test/CodeGen/AMDGPU/rsq.ll

 
 ; SI-LABEL: @rsqrt_fmul
 ; SI-DAG: buffer_load_dword [[A:v[0-9]+]], {{v\[[0-9]+:[0-9]+\]}}, {{s\[[0-9]+:[0-9]+\]}}, 0 addr64{{$}}
 ; SI-DAG: buffer_load_dword [[B:v[0-9]+]], {{v\[[0-9]+:[0-9]+\]}}, {{s\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:4
 ; SI-DAG: buffer_load_dword [[C:v[0-9]+]], {{v\[[0-9]+:[0-9]+\]}}, {{s\[[0-9]+:[0-9]+\]}}, 0 addr64 offset:8
 
 ; SI-UNSAFE-DAG: v_rsq_f32_e32 [[RSQA:v[0-9]+]], [[A]]
 ; SI-UNSAFE-DAG: v_rcp_f32_e32 [[RCPB:v[0-9]+]], [[B]]
-; SI-UNSAFE-DAG: v_mul_f32_e32 [[TMP:v[0-9]+]], [[RSQA]], [[RCPB]]
+; SI-UNSAFE-DAG: v_mul_f32_e32 [[TMP:v[0-9]+]], [[RCPB]], [[RSQA]]
 ; SI-UNSAFE: v_mul_f32_e32 [[RESULT:v[0-9]+]], [[C]], [[TMP]]
 ; SI-UNSAFE: buffer_store_dword [[RESULT]]
 
 ; SI-SAFE-NOT: v_rsq_f32
 
 ; SI: s_endpgm
 define amdgpu_kernel void @rsqrt_fmul(float addrspace(1)* %out, float addrspace(1)* %in) {
   %tid = call i32 @llvm.amdgcn.workitem.id.x() nounwind readnone

llvm/trunk/test/CodeGen/PowerPC/qpx-recipest.ll

 
 define <4 x double> @foo2_fmf(<4 x double> %a, <4 x double> %b) nounwind {
 ; CHECK-LABEL: foo2_fmf:
 ; CHECK:       # %bb.0: # %entry
 ; CHECK-NEXT:    addis 3, 2, .LCPI8_0@toc@ha
 ; CHECK-NEXT:    qvfre 3, 2
 ; CHECK-NEXT:    addi 3, 3, .LCPI8_0@toc@l
 ; CHECK-NEXT:    qvlfdx 0, 0, 3
-; CHECK-NEXT:    qvfnmsub 4, 2, 3, 0
-; CHECK-NEXT:    qvfmadd 3, 3, 4, 3
 ; CHECK-NEXT:    qvfnmsub 0, 2, 3, 0
 ; CHECK-NEXT:    qvfmadd 0, 3, 0, 3
-; CHECK-NEXT:    qvfmul 1, 1, 0
+; CHECK-NEXT:    qvfmul 3, 1, 0
+; CHECK-NEXT:    qvfnmsub 1, 2, 3, 1
+; CHECK-NEXT:    qvfmadd 1, 0, 1, 3
 ; CHECK-NEXT:    blr
 entry:
   %r = fdiv fast <4 x double> %a, %b
   ret <4 x double> %r
 }
 
 define <4 x double> @foo2_safe(<4 x double> %a, <4 x double> %b) nounwind {
 ; CHECK-LABEL: foo2_safe:

 ; CHECK-NEXT:    blr
   %r = fdiv <4 x double> %a, %b
   ret <4 x double> %r
 }
 
 define <4 x float> @goo2_fmf(<4 x float> %a, <4 x float> %b) nounwind {
 ; CHECK-LABEL: goo2_fmf:
 ; CHECK:       # %bb.0: # %entry
-; CHECK-NEXT:    addis 3, 2, .LCPI10_0@toc@ha
-; CHECK-NEXT:    qvfres 3, 2
-; CHECK-NEXT:    addi 3, 3, .LCPI10_0@toc@l
-; CHECK-NEXT:    qvlfsx 0, 0, 3
-; CHECK-NEXT:    qvfnmsubs 0, 2, 3, 0
-; CHECK-NEXT:    qvfmadds 0, 3, 0, 3
-; CHECK-NEXT:    qvfmuls 1, 1, 0
+; CHECK-NEXT:    qvfres 0, 2
+; CHECK-NEXT:    qvfmuls 3, 1, 0
+; CHECK-NEXT:    qvfnmsubs 1, 2, 3, 1
+; CHECK-NEXT:    qvfmadds 1, 0, 1, 3
 ; CHECK-NEXT:    blr
 entry:
   %r = fdiv fast <4 x float> %a, %b
   ret <4 x float> %r
 }
 
 define <4 x float> @goo2_safe(<4 x float> %a, <4 x float> %b) nounwind {
 ; CHECK-LABEL: goo2_safe:

llvm/trunk/test/CodeGen/PowerPC/recipest.ll

; CHECK: blr
   ret float %r
 }
 
 ; Recognize that this is rsqrt(a) * rcp(b) * c, 
 ; not 1 / ( 1 / sqrt(a)) * rcp(b) * c.
 define float @rsqrt_fmul_fmf(float %a, float %b, float %c) {
 ; CHECK: @rsqrt_fmul_fmf
 ; CHECK-DAG: frsqrtes
-; CHECK-DAG: fres
-; CHECK-DAG: fnmsubs
-; CHECK-DAG: fmuls
-; CHECK-DAG: fmadds
-; CHECK-DAG: fmadds
 ; CHECK: fmuls
+; CHECK-NEXT: fmadds
 ; CHECK-NEXT: fmuls
+; CHECK-DAG: fres
+; CHECK-COUNT-3: fmuls
+; CHECK-NEXT: fmsubs
+; CHECK-NEXT: fmadds
 ; CHECK-NEXT: fmuls
 ; CHECK-NEXT: blr
   %x = call fast float @llvm.sqrt.f32(float %a)
   %y = fmul fast float %x, %b 
   %z = fdiv fast float %c, %y
   ret float %z
 }
 
; CHECK: blr
   ret <4 x float> %r
 }
 
 define double @foo2_fmf(double %a, double %b) nounwind {
 ; CHECK: @foo2_fmf
 ; CHECK-DAG: fre
 ; CHECK-DAG: fnmsub
 ; CHECK: fmadd
+; CHECK-NEXT: fmul
 ; CHECK-NEXT: fnmsub
 ; CHECK-NEXT: fmadd
-; CHECK-NEXT: fmul
 ; CHECK-NEXT: blr
   %r = fdiv fast double %a, %b
   ret double %r
 }
 
 define double @foo2_safe(double %a, double %b) nounwind {
 ; CHECK: @foo2_safe
 ; CHECK: fdiv
 ; CHECK: blr
   %r = fdiv double %a, %b
   ret double %r
 }
 
 define float @goo2_fmf(float %a, float %b) nounwind {
 ; CHECK: @goo2_fmf
 ; CHECK-DAG: fres
+; CHECK-NEXT: fmuls
 ; CHECK-DAG: fnmsubs
 ; CHECK: fmadds
-; CHECK-NEXT: fmuls
 ; CHECK-NEXT: blr
   %r = fdiv fast float %a, %b
   ret float %r
 }
 
 define float @goo2_safe(float %a, float %b) nounwind {
 ; CHECK: @goo2_safe
 ; CHECK: fdivs

llvm/trunk/test/CodeGen/X86/recip-fastmath.ll

; AVX512-NEXT:    retq
   %div = fdiv fast float 1.0, %x
   ret float %div
 }
 
 define float @f32_one_step_variables(float %x, float %y) #1 {
 ; SSE-LABEL: f32_one_step_variables:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpss %xmm1, %xmm2
-; SSE-NEXT:    mulss %xmm2, %xmm1
-; SSE-NEXT:    movss {{.*#+}} xmm3 = mem[0],zero,zero,zero
-; SSE-NEXT:    subss %xmm1, %xmm3
+; SSE-NEXT:    movaps %xmm0, %xmm3
 ; SSE-NEXT:    mulss %xmm2, %xmm3
-; SSE-NEXT:    addss %xmm2, %xmm3
-; SSE-NEXT:    mulss %xmm3, %xmm0
+; SSE-NEXT:    mulss %xmm3, %xmm1
+; SSE-NEXT:    subss %xmm1, %xmm0
+; SSE-NEXT:    mulss %xmm2, %xmm0
+; SSE-NEXT:    addss %xmm3, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: f32_one_step_variables:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; AVX-RECIP-NEXT:    vmulss %xmm2, %xmm1, %xmm1
-; AVX-RECIP-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
-; AVX-RECIP-NEXT:    vsubss %xmm1, %xmm3, %xmm1
-; AVX-RECIP-NEXT:    vmulss %xmm1, %xmm2, %xmm1
-; AVX-RECIP-NEXT:    vaddss %xmm1, %xmm2, %xmm1
-; AVX-RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; AVX-RECIP-NEXT:    vmulss %xmm3, %xmm1, %xmm1
+; AVX-RECIP-NEXT:    vsubss %xmm1, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vmulss %xmm0, %xmm2, %xmm0
+; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: f32_one_step_variables:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm2 * xmm1) + mem
-; FMA-RECIP-NEXT:    vfmadd132ss {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; FMA-RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; FMA-RECIP-NEXT:    vfmadd213ss {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; FMA-RECIP-NEXT:    vmovaps %xmm2, %xmm0
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: f32_one_step_variables:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; BDVER2-NEXT:    vfnmaddss {{.*}}(%rip), %xmm2, %xmm1, %xmm1
-; BDVER2-NEXT:    vfmaddss %xmm2, %xmm1, %xmm2, %xmm1
-; BDVER2-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; BDVER2-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; BDVER2-NEXT:    vfnmaddss %xmm0, %xmm3, %xmm1, %xmm0
+; BDVER2-NEXT:    vfmaddss %xmm3, %xmm0, %xmm2, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: f32_one_step_variables:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; BTVER2-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; BTVER2-NEXT:    vmulss %xmm2, %xmm1, %xmm1
-; BTVER2-NEXT:    vsubss %xmm1, %xmm3, %xmm1
-; BTVER2-NEXT:    vmulss %xmm1, %xmm2, %xmm1
-; BTVER2-NEXT:    vaddss %xmm1, %xmm2, %xmm1
-; BTVER2-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; BTVER2-NEXT:    vmulss %xmm3, %xmm1, %xmm1
+; BTVER2-NEXT:    vsubss %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulss %xmm0, %xmm2, %xmm0
+; BTVER2-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: f32_one_step_variables:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; SANDY-NEXT:    vmulss %xmm2, %xmm1, %xmm1
-; SANDY-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
-; SANDY-NEXT:    vsubss %xmm1, %xmm3, %xmm1
-; SANDY-NEXT:    vmulss %xmm1, %xmm2, %xmm1
-; SANDY-NEXT:    vaddss %xmm1, %xmm2, %xmm1
-; SANDY-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; SANDY-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; SANDY-NEXT:    vmulss %xmm3, %xmm1, %xmm1
+; SANDY-NEXT:    vsubss %xmm1, %xmm0, %xmm0
+; SANDY-NEXT:    vmulss %xmm0, %xmm2, %xmm0
+; SANDY-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: f32_one_step_variables:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm2 * xmm1) + mem
-; HASWELL-NEXT:    vfmadd132ss {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; HASWELL-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; HASWELL-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; HASWELL-NEXT:    vfmadd213ss {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; HASWELL-NEXT:    vmovaps %xmm2, %xmm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: f32_one_step_variables:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm1, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
-; HASWELL-NO-FMA-NEXT:    vsubss %xmm1, %xmm3, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulss %xmm1, %xmm2, %xmm1
-; HASWELL-NO-FMA-NEXT:    vaddss %xmm1, %xmm2, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm3, %xmm1, %xmm1
+; HASWELL-NO-FMA-NEXT:    vsubss %xmm1, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm0, %xmm2, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: f32_one_step_variables:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpss %xmm1, %xmm1, %xmm2
-; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm2 * xmm1) + mem
-; AVX512-NEXT:    vfmadd132ss {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; AVX512-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vmulss %xmm2, %xmm0, %xmm3
+; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; AVX512-NEXT:    vfmadd213ss {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; AVX512-NEXT:    vmovaps %xmm2, %xmm0
 ; AVX512-NEXT:    retq
   %div = fdiv fast float %x, %y
   ret float %div
 }
 
 define float @f32_two_step(float %x) #2 {
 ; SSE-LABEL: f32_two_step:
 ; SSE:       # %bb.0:
; SKX-NEXT:    retq
   %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %x
   ret <4 x float> %div
 }
 
 define <4 x float> @v4f32_one_step_variables(<4 x float> %x, <4 x float> %y) #1 {
 ; SSE-LABEL: v4f32_one_step_variables:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm1, %xmm2
-; SSE-NEXT:    mulps %xmm2, %xmm1
-; SSE-NEXT:    movaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    subps %xmm1, %xmm3
+; SSE-NEXT:    movaps %xmm0, %xmm3
 ; SSE-NEXT:    mulps %xmm2, %xmm3
-; SSE-NEXT:    addps %xmm2, %xmm3
-; SSE-NEXT:    mulps %xmm3, %xmm0
+; SSE-NEXT:    mulps %xmm3, %xmm1
+; SSE-NEXT:    subps %xmm1, %xmm0
+; SSE-NEXT:    mulps %xmm2, %xmm0
+; SSE-NEXT:    addps %xmm3, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v4f32_one_step_variables:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpps %xmm1, %xmm2
-; AVX-RECIP-NEXT:    vmulps %xmm2, %xmm1, %xmm1
-; AVX-RECIP-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; AVX-RECIP-NEXT:    vsubps %xmm1, %xmm3, %xmm1
-; AVX-RECIP-NEXT:    vmulps %xmm1, %xmm2, %xmm1
-; AVX-RECIP-NEXT:    vaddps %xmm1, %xmm2, %xmm1
-; AVX-RECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; AVX-RECIP-NEXT:    vmulps %xmm3, %xmm1, %xmm1
+; AVX-RECIP-NEXT:    vsubps %xmm1, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vmulps %xmm0, %xmm2, %xmm0
+; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v4f32_one_step_variables:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %xmm1, %xmm2
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm2 * xmm1) + mem
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; FMA-RECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; FMA-RECIP-NEXT:    vmovaps %xmm2, %xmm0
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v4f32_one_step_variables:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %xmm1, %xmm2
-; BDVER2-NEXT:    vfnmaddps {{.*}}(%rip), %xmm2, %xmm1, %xmm1
-; BDVER2-NEXT:    vfmaddps %xmm2, %xmm1, %xmm2, %xmm1
-; BDVER2-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; BDVER2-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; BDVER2-NEXT:    vfnmaddps %xmm0, %xmm3, %xmm1, %xmm0
+; BDVER2-NEXT:    vfmaddps %xmm3, %xmm0, %xmm2, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v4f32_one_step_variables:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BTVER2-NEXT:    vrcpps %xmm1, %xmm2
-; BTVER2-NEXT:    vmulps %xmm2, %xmm1, %xmm1
-; BTVER2-NEXT:    vsubps %xmm1, %xmm3, %xmm1
-; BTVER2-NEXT:    vmulps %xmm1, %xmm2, %xmm1
-; BTVER2-NEXT:    vaddps %xmm1, %xmm2, %xmm1
-; BTVER2-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; BTVER2-NEXT:    vmulps %xmm3, %xmm1, %xmm1
+; BTVER2-NEXT:    vsubps %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulps %xmm0, %xmm2, %xmm0
+; BTVER2-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v4f32_one_step_variables:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %xmm1, %xmm2
-; SANDY-NEXT:    vmulps %xmm2, %xmm1, %xmm1
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vsubps %xmm1, %xmm3, %xmm1
-; SANDY-NEXT:    vmulps %xmm1, %xmm2, %xmm1
-; SANDY-NEXT:    vaddps %xmm1, %xmm2, %xmm1
-; SANDY-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; SANDY-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; SANDY-NEXT:    vmulps %xmm3, %xmm1, %xmm1
+; SANDY-NEXT:    vsubps %xmm1, %xmm0, %xmm0
+; SANDY-NEXT:    vmulps %xmm0, %xmm2, %xmm0
+; SANDY-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_one_step_variables:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %xmm1, %xmm2
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm2 * xmm1) + xmm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; HASWELL-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; HASWELL-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; HASWELL-NEXT:    vmovaps %xmm2, %xmm0
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_one_step_variables:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm1, %xmm2
-; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm1, %xmm1
-; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NO-FMA-NEXT:    vsubps %xmm1, %xmm3, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm2, %xmm1
-; HASWELL-NO-FMA-NEXT:    vaddps %xmm1, %xmm2, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm3, %xmm1, %xmm1
+; HASWELL-NO-FMA-NEXT:    vsubps %xmm1, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm2, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
-; KNL-LABEL: v4f32_one_step_variables:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vrcpps %xmm1, %xmm2
-; KNL-NEXT:    vbroadcastss {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm2 * xmm1) + xmm3
-; KNL-NEXT:    vfmadd132ps {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; KNL-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; KNL-NEXT:    retq
-;
-; SKX-LABEL: v4f32_one_step_variables:
-; SKX:       # %bb.0:
-; SKX-NEXT:    vrcpps %xmm1, %xmm2
-; SKX-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm2 * xmm1) + mem
-; SKX-NEXT:    vfmadd132ps {{.*#+}} xmm1 = (xmm1 * xmm2) + xmm2
-; SKX-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; SKX-NEXT:    retq
+; AVX512-LABEL: v4f32_one_step_variables:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vrcpps %xmm1, %xmm2
+; AVX512-NEXT:    vmulps %xmm2, %xmm0, %xmm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm1 = -(xmm3 * xmm1) + xmm0
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} xmm2 = (xmm1 * xmm2) + xmm3
+; AVX512-NEXT:    vmovaps %xmm2, %xmm0
+; AVX512-NEXT:    retq
   %div = fdiv fast <4 x float> %x, %y
   ret <4 x float> %div
 }
 
 define <4 x float> @v4f32_two_step(<4 x float> %x) #2 {
 ; SSE-LABEL: v4f32_two_step:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm2

llvm/trunk/test/CodeGen/X86/recip-fastmath2.ll

; AVX-NEXT:    retq
   %div = fdiv fast float 1234.0, %x
   ret float %div
 }
 
 define float @f32_one_step_2(float %x) #1 {
 ; SSE-LABEL: f32_one_step_2:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpss %xmm0, %xmm2
-; SSE-NEXT:    mulss %xmm2, %xmm0
 ; SSE-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    movaps %xmm2, %xmm3
+; SSE-NEXT:    mulss %xmm1, %xmm3
+; SSE-NEXT:    mulss %xmm3, %xmm0
 ; SSE-NEXT:    subss %xmm0, %xmm1
 ; SSE-NEXT:    mulss %xmm2, %xmm1
-; SSE-NEXT:    addss %xmm2, %xmm1
-; SSE-NEXT:    mulss {{.*}}(%rip), %xmm1
+; SSE-NEXT:    addss %xmm3, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: f32_one_step_2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; AVX-RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm0
 ; AVX-RECIP-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; AVX-RECIP-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; AVX-RECIP-NEXT:    vmulss %xmm3, %xmm0, %xmm0
 ; AVX-RECIP-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: f32_one_step_2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm1 * xmm0) + mem
-; FMA-RECIP-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; FMA-RECIP-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; FMA-RECIP-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; FMA-RECIP-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: f32_one_step_2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; BDVER2-NEXT:    vfnmaddss {{.*}}(%rip), %xmm1, %xmm0, %xmm0
-; BDVER2-NEXT:    vfmaddss %xmm1, %xmm0, %xmm1, %xmm0
-; BDVER2-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; BDVER2-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; BDVER2-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; BDVER2-NEXT:    vfnmaddss %xmm2, %xmm3, %xmm0, %xmm0
+; BDVER2-NEXT:    vfmaddss %xmm3, %xmm0, %xmm1, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: f32_one_step_2:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
 ; BTVER2-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; BTVER2-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; BTVER2-NEXT:    vmulss %xmm3, %xmm0, %xmm0
 ; BTVER2-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; BTVER2-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; BTVER2-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: f32_one_step_2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; SANDY-NEXT:    vmulss %xmm1, %xmm0, %xmm0
 ; SANDY-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; SANDY-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; SANDY-NEXT:    vmulss %xmm3, %xmm0, %xmm0
 ; SANDY-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; SANDY-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: f32_one_step_2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm1 * xmm0) + mem
-; HASWELL-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; HASWELL-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; HASWELL-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; HASWELL-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: f32_one_step_2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulss %xmm1, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm3, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: f32_one_step_2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm1 * xmm0) + mem
-; AVX512-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; AVX512-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; AVX512-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; AVX512-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast float 3456.0, %x
   ret float %div
 }
 
 define float @f32_one_step_2_divs(float %x) #1 {
 ; SSE-LABEL: f32_one_step_2_divs:
 ; SSE:       # %bb.0:
; AVX512-NEXT:    retq
   %div = fdiv fast float 3456.0, %x
   %div2 = fdiv fast float %div, %x
   ret float %div2
 }
 
 define float @f32_two_step_2(float %x) #2 {
 ; SSE-LABEL: f32_two_step_2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rcpss %xmm0, %xmm2
-; SSE-NEXT:    movaps %xmm0, %xmm3
-; SSE-NEXT:    mulss %xmm2, %xmm3
+; SSE-NEXT:    rcpss %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm0, %xmm2
+; SSE-NEXT:    mulss %xmm1, %xmm2
+; SSE-NEXT:    movss {{.*#+}} xmm3 = mem[0],zero,zero,zero
+; SSE-NEXT:    subss %xmm2, %xmm3
+; SSE-NEXT:    mulss %xmm1, %xmm3
+; SSE-NEXT:    addss %xmm1, %xmm3
 ; SSE-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
-; SSE-NEXT:    movaps %xmm1, %xmm4
-; SSE-NEXT:    subss %xmm3, %xmm4
-; SSE-NEXT:    mulss %xmm2, %xmm4
-; SSE-NEXT:    addss %xmm2, %xmm4
-; SSE-NEXT:    mulss %xmm4, %xmm0
+; SSE-NEXT:    movaps %xmm3, %xmm2
+; SSE-NEXT:    mulss %xmm1, %xmm2
+; SSE-NEXT:    mulss %xmm2, %xmm0
 ; SSE-NEXT:    subss %xmm0, %xmm1
-; SSE-NEXT:    mulss %xmm4, %xmm1
-; SSE-NEXT:    addss %xmm4, %xmm1
-; SSE-NEXT:    mulss {{.*}}(%rip), %xmm1
+; SSE-NEXT:    mulss %xmm3, %xmm1
+; SSE-NEXT:    addss %xmm2, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: f32_two_step_2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; AVX-RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm2
 ; AVX-RECIP-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; AVX-RECIP-NEXT:    vsubss %xmm2, %xmm3, %xmm2
 ; AVX-RECIP-NEXT:    vmulss %xmm2, %xmm1, %xmm2
 ; AVX-RECIP-NEXT:    vaddss %xmm2, %xmm1, %xmm1
-; AVX-RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm0
-; AVX-RECIP-NEXT:    vsubss %xmm0, %xmm3, %xmm0
+; AVX-RECIP-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; AVX-RECIP-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; AVX-RECIP-NEXT:    vmulss %xmm3, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: f32_two_step_2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; FMA-RECIP-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; FMA-RECIP-NEXT:    vmovaps %xmm1, %xmm3
-; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; FMA-RECIP-NEXT:    vfmadd132ss {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; FMA-RECIP-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; FMA-RECIP-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vfnmadd231ss {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; FMA-RECIP-NEXT:    vfmadd132ss {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; FMA-RECIP-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; FMA-RECIP-NEXT:    vmulss %xmm1, %xmm2, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; FMA-RECIP-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: f32_two_step_2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
-; BDVER2-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; BDVER2-NEXT:    vfnmaddss %xmm2, %xmm1, %xmm0, %xmm3
-; BDVER2-NEXT:    vfmaddss %xmm1, %xmm3, %xmm1, %xmm1
-; BDVER2-NEXT:    vfnmaddss %xmm2, %xmm1, %xmm0, %xmm0
-; BDVER2-NEXT:    vfmaddss %xmm1, %xmm0, %xmm1, %xmm0
-; BDVER2-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; BDVER2-NEXT:    vfnmaddss {{.*}}(%rip), %xmm1, %xmm0, %xmm2
+; BDVER2-NEXT:    vmovss {{.*#+}} xmm4 = mem[0],zero,zero,zero
+; BDVER2-NEXT:    vfmaddss %xmm1, %xmm2, %xmm1, %xmm1
+; BDVER2-NEXT:    vmulss %xmm4, %xmm1, %xmm3
+; BDVER2-NEXT:    vfnmaddss %xmm4, %xmm3, %xmm0, %xmm0
+; BDVER2-NEXT:    vfmaddss %xmm3, %xmm0, %xmm1, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: f32_two_step_2:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; BTVER2-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
+; BTVER2-NEXT:    vmovss {{.*#+}} xmm4 = mem[0],zero,zero,zero
 ; BTVER2-NEXT:    vmulss %xmm1, %xmm0, %xmm2
 ; BTVER2-NEXT:    vsubss %xmm2, %xmm3, %xmm2
 ; BTVER2-NEXT:    vmulss %xmm2, %xmm1, %xmm2
 ; BTVER2-NEXT:    vaddss %xmm2, %xmm1, %xmm1
-; BTVER2-NEXT:    vmulss %xmm1, %xmm0, %xmm0
-; BTVER2-NEXT:    vsubss %xmm0, %xmm3, %xmm0
+; BTVER2-NEXT:    vmulss %xmm4, %xmm1, %xmm3
+; BTVER2-NEXT:    vmulss %xmm3, %xmm0, %xmm0
+; BTVER2-NEXT:    vsubss %xmm0, %xmm4, %xmm0
 ; BTVER2-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; BTVER2-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: f32_two_step_2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; SANDY-NEXT:    vmulss %xmm1, %xmm0, %xmm2
 ; SANDY-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; SANDY-NEXT:    vsubss %xmm2, %xmm3, %xmm2
 ; SANDY-NEXT:    vmulss %xmm2, %xmm1, %xmm2
 ; SANDY-NEXT:    vaddss %xmm2, %xmm1, %xmm1
-; SANDY-NEXT:    vmulss %xmm1, %xmm0, %xmm0
-; SANDY-NEXT:    vsubss %xmm0, %xmm3, %xmm0
+; SANDY-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; SANDY-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; SANDY-NEXT:    vmulss %xmm3, %xmm0, %xmm0
+; SANDY-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; SANDY-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: f32_two_step_2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; HASWELL-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; HASWELL-NEXT:    vmovaps %xmm1, %xmm3
-; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; HASWELL-NEXT:    vfmadd132ss {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; HASWELL-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; HASWELL-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NEXT:    vfnmadd231ss {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; HASWELL-NEXT:    vfmadd132ss {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; HASWELL-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; HASWELL-NEXT:    vmulss %xmm1, %xmm2, %xmm3
+; HASWELL-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; HASWELL-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: f32_two_step_2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm1, %xmm0, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
 ; HASWELL-NO-FMA-NEXT:    vsubss %xmm2, %xmm3, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm1, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vaddss %xmm2, %xmm1, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulss %xmm1, %xmm0, %xmm0
-; HASWELL-NO-FMA-NEXT:    vsubss %xmm0, %xmm3, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm2, %xmm1, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulss %xmm3, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vsubss %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulss %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddss %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: f32_two_step_2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
 ; AVX512-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
-; AVX512-NEXT:    vmovaps %xmm1, %xmm3
-; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; AVX512-NEXT:    vfmadd132ss {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; AVX512-NEXT:    vfmadd132ss {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; AVX512-NEXT:    vmulss {{.*}}(%rip), %xmm0, %xmm0
+; AVX512-NEXT:    vfnmadd231ss {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; AVX512-NEXT:    vfmadd132ss {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; AVX512-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmulss %xmm1, %xmm2, %xmm3
+; AVX512-NEXT:    vfnmadd213ss {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; AVX512-NEXT:    vfmadd213ss {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast float 6789.0, %x
   ret float %div
 }
 
 define <4 x float> @v4f32_one_step2(<4 x float> %x) #1 {
 ; SSE-LABEL: v4f32_one_step2:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm2
-; SSE-NEXT:    mulps %xmm2, %xmm0
-; SSE-NEXT:    movaps {{.*#+}} xmm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SSE-NEXT:    movaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm2, %xmm3
+; SSE-NEXT:    mulps %xmm1, %xmm3
+; SSE-NEXT:    mulps %xmm3, %xmm0
 ; SSE-NEXT:    subps %xmm0, %xmm1
 ; SSE-NEXT:    mulps %xmm2, %xmm1
-; SSE-NEXT:    addps %xmm2, %xmm1
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm1
+; SSE-NEXT:    addps %xmm3, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v4f32_one_step2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpps %xmm0, %xmm1
-; AVX-RECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; AVX-RECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; AVX-RECIP-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; AVX-RECIP-NEXT:    vmulps %xmm3, %xmm0, %xmm0
 ; AVX-RECIP-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v4f32_one_step2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %xmm0, %xmm1
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm1 * xmm0) + mem
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; FMA-RECIP-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v4f32_one_step2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %xmm0, %xmm1
-; BDVER2-NEXT:    vfnmaddps {{.*}}(%rip), %xmm1, %xmm0, %xmm0
-; BDVER2-NEXT:    vfmaddps %xmm1, %xmm0, %xmm1, %xmm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; BDVER2-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; BDVER2-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; BDVER2-NEXT:    vfnmaddps %xmm2, %xmm3, %xmm0, %xmm0
+; BDVER2-NEXT:    vfmaddps %xmm3, %xmm0, %xmm1, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v4f32_one_step2:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; BTVER2-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
 ; BTVER2-NEXT:    vrcpps %xmm0, %xmm1
-; BTVER2-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; BTVER2-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; BTVER2-NEXT:    vmulps %xmm3, %xmm0, %xmm0
 ; BTVER2-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; BTVER2-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; BTVER2-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v4f32_one_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %xmm0, %xmm1
-; SANDY-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SANDY-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; SANDY-NEXT:    vmulps %xmm3, %xmm0, %xmm0
 ; SANDY-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; SANDY-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_one_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %xmm0, %xmm1
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm1 * xmm0) + xmm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; HASWELL-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_one_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm0, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm3, %xmm0, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
-; KNL-LABEL: v4f32_one_step2:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vrcpps %xmm0, %xmm1
-; KNL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm1 * xmm0) + xmm2
-; KNL-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; KNL-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
-; KNL-NEXT:    retq
-;
-; SKX-LABEL: v4f32_one_step2:
-; SKX:       # %bb.0:
-; SKX-NEXT:    vrcpps %xmm0, %xmm1
-; SKX-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm1 * xmm0) + mem
-; SKX-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm1) + xmm1
-; SKX-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
-; SKX-NEXT:    retq
+; AVX512-LABEL: v4f32_one_step2:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vrcpps %xmm0, %xmm1
+; AVX512-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; AVX512-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm1 * xmm0) + xmm3
+; AVX512-NEXT:    retq
   %div = fdiv fast <4 x float> <float 1.0, float 2.0, float 3.0, float 4.0>, %x
   ret <4 x float> %div
 }
 
 define <4 x float> @v4f32_one_step_2_divs(<4 x float> %x) #1 {
 ; SSE-LABEL: v4f32_one_step_2_divs:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm1
; SKX-NEXT:    retq
   %div = fdiv fast <4 x float> <float 1.0, float 2.0, float 3.0, float 4.0>, %x
   %div2 = fdiv fast <4 x float> %div, %x
   ret <4 x float> %div2
 }
 
 define <4 x float> @v4f32_two_step2(<4 x float> %x) #2 {
 ; SSE-LABEL: v4f32_two_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rcpps %xmm0, %xmm2
-; SSE-NEXT:    movaps %xmm0, %xmm3
-; SSE-NEXT:    mulps %xmm2, %xmm3
-; SSE-NEXT:    movaps {{.*#+}} xmm1 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    movaps %xmm1, %xmm4
-; SSE-NEXT:    subps %xmm3, %xmm4
-; SSE-NEXT:    mulps %xmm2, %xmm4
-; SSE-NEXT:    addps %xmm2, %xmm4
-; SSE-NEXT:    mulps %xmm4, %xmm0
+; SSE-NEXT:    rcpps %xmm0, %xmm1
+; SSE-NEXT:    movaps %xmm0, %xmm2
+; SSE-NEXT:    mulps %xmm1, %xmm2
+; SSE-NEXT:    movaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SSE-NEXT:    subps %xmm2, %xmm3
+; SSE-NEXT:    mulps %xmm1, %xmm3
+; SSE-NEXT:    addps %xmm1, %xmm3
+; SSE-NEXT:    movaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm3, %xmm2
+; SSE-NEXT:    mulps %xmm1, %xmm2
+; SSE-NEXT:    mulps %xmm2, %xmm0
 ; SSE-NEXT:    subps %xmm0, %xmm1
-; SSE-NEXT:    mulps %xmm4, %xmm1
-; SSE-NEXT:    addps %xmm4, %xmm1
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm1
+; SSE-NEXT:    mulps %xmm3, %xmm1
+; SSE-NEXT:    addps %xmm2, %xmm1
 ; SSE-NEXT:    movaps %xmm1, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v4f32_two_step2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpps %xmm0, %xmm1
 ; AVX-RECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm2
 ; AVX-RECIP-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; AVX-RECIP-NEXT:    vsubps %xmm2, %xmm3, %xmm2
 ; AVX-RECIP-NEXT:    vmulps %xmm2, %xmm1, %xmm2
 ; AVX-RECIP-NEXT:    vaddps %xmm2, %xmm1, %xmm1
-; AVX-RECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; AVX-RECIP-NEXT:    vsubps %xmm0, %xmm3, %xmm0
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; AVX-RECIP-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; AVX-RECIP-NEXT:    vmulps %xmm3, %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; AVX-RECIP-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; AVX-RECIP-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v4f32_two_step2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %xmm0, %xmm1
 ; FMA-RECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; FMA-RECIP-NEXT:    vmovaps %xmm1, %xmm3
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; FMA-RECIP-NEXT:    vfnmadd231ps {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; FMA-RECIP-NEXT:    vmulps %xmm1, %xmm2, %xmm3
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v4f32_two_step2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %xmm0, %xmm1
-; BDVER2-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BDVER2-NEXT:    vfnmaddps %xmm2, %xmm1, %xmm0, %xmm3
-; BDVER2-NEXT:    vfmaddps %xmm1, %xmm3, %xmm1, %xmm1
-; BDVER2-NEXT:    vfnmaddps %xmm2, %xmm1, %xmm0, %xmm0
-; BDVER2-NEXT:    vfmaddps %xmm1, %xmm0, %xmm1, %xmm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; BDVER2-NEXT:    vfnmaddps {{.*}}(%rip), %xmm1, %xmm0, %xmm2
+; BDVER2-NEXT:    vmovaps {{.*#+}} xmm4 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; BDVER2-NEXT:    vfmaddps %xmm1, %xmm2, %xmm1, %xmm1
+; BDVER2-NEXT:    vmulps %xmm4, %xmm1, %xmm3
+; BDVER2-NEXT:    vfnmaddps %xmm4, %xmm3, %xmm0, %xmm0
+; BDVER2-NEXT:    vfmaddps %xmm3, %xmm0, %xmm1, %xmm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v4f32_two_step2:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BTVER2-NEXT:    vrcpps %xmm0, %xmm1
+; BTVER2-NEXT:    vmovaps {{.*#+}} xmm4 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
 ; BTVER2-NEXT:    vmulps %xmm1, %xmm0, %xmm2
 ; BTVER2-NEXT:    vsubps %xmm2, %xmm3, %xmm2
 ; BTVER2-NEXT:    vmulps %xmm2, %xmm1, %xmm2
 ; BTVER2-NEXT:    vaddps %xmm2, %xmm1, %xmm1
-; BTVER2-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; BTVER2-NEXT:    vsubps %xmm0, %xmm3, %xmm0
+; BTVER2-NEXT:    vmulps %xmm4, %xmm1, %xmm3
+; BTVER2-NEXT:    vmulps %xmm3, %xmm0, %xmm0
+; BTVER2-NEXT:    vsubps %xmm0, %xmm4, %xmm0
 ; BTVER2-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; BTVER2-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v4f32_two_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %xmm0, %xmm1
 ; SANDY-NEXT:    vmulps %xmm1, %xmm0, %xmm2
 ; SANDY-NEXT:    vmovaps {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %xmm2, %xmm3, %xmm2
 ; SANDY-NEXT:    vmulps %xmm2, %xmm1, %xmm2
 ; SANDY-NEXT:    vaddps %xmm2, %xmm1, %xmm1
-; SANDY-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; SANDY-NEXT:    vsubps %xmm0, %xmm3, %xmm0
+; SANDY-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SANDY-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; SANDY-NEXT:    vmulps %xmm3, %xmm0, %xmm0
+; SANDY-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; SANDY-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; SANDY-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v4f32_two_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %xmm0, %xmm1
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vmovaps %xmm1, %xmm3
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NEXT:    vfnmadd231ps {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; HASWELL-NEXT:    vfmadd132ps {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; HASWELL-NEXT:    vmovaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; HASWELL-NEXT:    vmulps %xmm1, %xmm2, %xmm3
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_two_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %xmm0, %xmm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} xmm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %xmm2, %xmm3, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm1, %xmm2
 ; HASWELL-NO-FMA-NEXT:    vaddps %xmm2, %xmm1, %xmm1
-; HASWELL-NO-FMA-NEXT:    vmulps %xmm1, %xmm0, %xmm0
-; HASWELL-NO-FMA-NEXT:    vsubps %xmm0, %xmm3, %xmm0
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm2, %xmm1, %xmm3
+; HASWELL-NO-FMA-NEXT:    vmulps %xmm3, %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vsubps %xmm0, %xmm2, %xmm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm1, %xmm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; HASWELL-NO-FMA-NEXT:    vaddps %xmm0, %xmm3, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v4f32_two_step2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpps %xmm0, %xmm1
 ; AVX512-NEXT:    vbroadcastss {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; AVX512-NEXT:    vmovaps %xmm1, %xmm3
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm3 = -(xmm0 * xmm3) + xmm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} xmm3 = (xmm3 * xmm1) + xmm1
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} xmm0 = (xmm0 * xmm3) + xmm3
-; AVX512-NEXT:    vmulps {{.*}}(%rip), %xmm0, %xmm0
+; AVX512-NEXT:    vfnmadd231ps {{.*#+}} xmm2 = -(xmm0 * xmm1) + xmm2
+; AVX512-NEXT:    vfmadd132ps {{.*#+}} xmm2 = (xmm2 * xmm1) + xmm1
+; AVX512-NEXT:    vmovaps {{.*#+}} xmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; AVX512-NEXT:    vmulps %xmm1, %xmm2, %xmm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} xmm0 = -(xmm3 * xmm0) + xmm1
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} xmm0 = (xmm2 * xmm0) + xmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast <4 x float> <float 1.0, float 2.0, float 3.0, float 4.0>, %x
   ret <4 x float> %div
 }
 
 define <8 x float> @v8f32_one_step2(<8 x float> %x) #1 {
 ; SSE-LABEL: v8f32_one_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rcpps %xmm1, %xmm4
+; SSE-NEXT:    rcpps %xmm0, %xmm3
+; SSE-NEXT:    movaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm3, %xmm4
+; SSE-NEXT:    mulps %xmm2, %xmm4
+; SSE-NEXT:    mulps %xmm4, %xmm0
+; SSE-NEXT:    subps %xmm0, %xmm2
+; SSE-NEXT:    mulps %xmm3, %xmm2
+; SSE-NEXT:    addps %xmm4, %xmm2
+; SSE-NEXT:    rcpps %xmm1, %xmm0
+; SSE-NEXT:    movaps {{.*#+}} xmm3 = [5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SSE-NEXT:    movaps %xmm0, %xmm4
+; SSE-NEXT:    mulps %xmm3, %xmm4
 ; SSE-NEXT:    mulps %xmm4, %xmm1
-; SSE-NEXT:    movaps {{.*#+}} xmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    movaps %xmm2, %xmm3
 ; SSE-NEXT:    subps %xmm1, %xmm3
-; SSE-NEXT:    mulps %xmm4, %xmm3
+; SSE-NEXT:    mulps %xmm0, %xmm3
 ; SSE-NEXT:    addps %xmm4, %xmm3
-; SSE-NEXT:    rcpps %xmm0, %xmm1
-; SSE-NEXT:    mulps %xmm1, %xmm0
-; SSE-NEXT:    subps %xmm0, %xmm2
-; SSE-NEXT:    mulps %xmm1, %xmm2
-; SSE-NEXT:    addps %xmm1, %xmm2
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm2
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm3
 ; SSE-NEXT:    movaps %xmm2, %xmm0
 ; SSE-NEXT:    movaps %xmm3, %xmm1
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v8f32_one_step2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm1
-; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; AVX-RECIP-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v8f32_one_step2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm1
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm1 * ymm0) + mem
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm1) + ymm1
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; FMA-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm1 * ymm0) + ymm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v8f32_one_step2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %ymm0, %ymm1
-; BDVER2-NEXT:    vfnmaddps {{.*}}(%rip), %ymm1, %ymm0, %ymm0
-; BDVER2-NEXT:    vfmaddps %ymm1, %ymm0, %ymm1, %ymm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BDVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; BDVER2-NEXT:    vfnmaddps %ymm2, %ymm3, %ymm0, %ymm0
+; BDVER2-NEXT:    vfmaddps %ymm3, %ymm0, %ymm1, %ymm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v8f32_one_step2:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; BTVER2-NEXT:    vrcpps %ymm0, %ymm1
-; BTVER2-NEXT:    vmulps %ymm1, %ymm0, %ymm0
+; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; BTVER2-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; BTVER2-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; BTVER2-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BTVER2-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_one_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; SANDY-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; SANDY-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_one_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm1
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm1) + ymm1
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm1 * ymm0) + ymm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v8f32_one_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm1
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm0, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
-; KNL-LABEL: v8f32_one_step2:
-; KNL:       # %bb.0:
-; KNL-NEXT:    vrcpps %ymm0, %ymm1
-; KNL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; KNL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm1 * ymm0) + ymm2
-; KNL-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm1) + ymm1
-; KNL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; KNL-NEXT:    retq
-;
-; SKX-LABEL: v8f32_one_step2:
-; SKX:       # %bb.0:
-; SKX-NEXT:    vrcpps %ymm0, %ymm1
-; SKX-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm1 * ymm0) + mem
-; SKX-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm1) + ymm1
-; SKX-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; SKX-NEXT:    retq
+; AVX512-LABEL: v8f32_one_step2:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vrcpps %ymm0, %ymm1
+; AVX512-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX512-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm1 * ymm0) + ymm3
+; AVX512-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, %x
   ret <8 x float> %div
 }
 
 define <8 x float> @v8f32_one_step_2_divs(<8 x float> %x) #1 {
 ; SSE-LABEL: v8f32_one_step_2_divs:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm2
; SKX-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, %x
   %div2 = fdiv fast <8 x float> %div, %x
   ret <8 x float> %div2
 }
 
 define <8 x float> @v8f32_two_step2(<8 x float> %x) #2 {
 ; SSE-LABEL: v8f32_two_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    movaps %xmm0, %xmm2
-; SSE-NEXT:    rcpps %xmm1, %xmm3
-; SSE-NEXT:    movaps %xmm1, %xmm4
-; SSE-NEXT:    mulps %xmm3, %xmm4
-; SSE-NEXT:    movaps {{.*#+}} xmm0 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    movaps %xmm0, %xmm5
-; SSE-NEXT:    subps %xmm4, %xmm5
-; SSE-NEXT:    mulps %xmm3, %xmm5
-; SSE-NEXT:    addps %xmm3, %xmm5
-; SSE-NEXT:    mulps %xmm5, %xmm1
+; SSE-NEXT:    rcpps %xmm0, %xmm2
 ; SSE-NEXT:    movaps %xmm0, %xmm3
-; SSE-NEXT:    subps %xmm1, %xmm3
-; SSE-NEXT:    mulps %xmm5, %xmm3
-; SSE-NEXT:    addps %xmm5, %xmm3
-; SSE-NEXT:    rcpps %xmm2, %xmm1
-; SSE-NEXT:    movaps %xmm2, %xmm4
-; SSE-NEXT:    mulps %xmm1, %xmm4
-; SSE-NEXT:    movaps %xmm0, %xmm5
-; SSE-NEXT:    subps %xmm4, %xmm5
-; SSE-NEXT:    mulps %xmm1, %xmm5
-; SSE-NEXT:    addps %xmm1, %xmm5
+; SSE-NEXT:    mulps %xmm2, %xmm3
+; SSE-NEXT:    movaps {{.*#+}} xmm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SSE-NEXT:    movaps %xmm4, %xmm5
+; SSE-NEXT:    subps %xmm3, %xmm5
+; SSE-NEXT:    mulps %xmm2, %xmm5
+; SSE-NEXT:    addps %xmm2, %xmm5
+; SSE-NEXT:    movaps {{.*#+}} xmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm5, %xmm3
+; SSE-NEXT:    mulps %xmm2, %xmm3
+; SSE-NEXT:    mulps %xmm3, %xmm0
+; SSE-NEXT:    subps %xmm0, %xmm2
 ; SSE-NEXT:    mulps %xmm5, %xmm2
-; SSE-NEXT:    subps %xmm2, %xmm0
-; SSE-NEXT:    mulps %xmm5, %xmm0
-; SSE-NEXT:    addps %xmm5, %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm3
+; SSE-NEXT:    addps %xmm3, %xmm2
+; SSE-NEXT:    rcpps %xmm1, %xmm0
+; SSE-NEXT:    movaps %xmm1, %xmm3
+; SSE-NEXT:    mulps %xmm0, %xmm3
+; SSE-NEXT:    subps %xmm3, %xmm4
+; SSE-NEXT:    mulps %xmm0, %xmm4
+; SSE-NEXT:    addps %xmm0, %xmm4
+; SSE-NEXT:    movaps {{.*#+}} xmm3 = [5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SSE-NEXT:    movaps %xmm4, %xmm0
+; SSE-NEXT:    mulps %xmm3, %xmm0
+; SSE-NEXT:    mulps %xmm0, %xmm1
+; SSE-NEXT:    subps %xmm1, %xmm3
+; SSE-NEXT:    mulps %xmm4, %xmm3
+; SSE-NEXT:    addps %xmm0, %xmm3
+; SSE-NEXT:    movaps %xmm2, %xmm0
 ; SSE-NEXT:    movaps %xmm3, %xmm1
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v8f32_two_step2:
 ; AVX-RECIP:       # %bb.0:
 ; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm1
 ; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm0, %ymm2
 ; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; AVX-RECIP-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; AVX-RECIP-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; AVX-RECIP-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v8f32_two_step2:
 ; FMA-RECIP:       # %bb.0:
 ; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm1
 ; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; FMA-RECIP-NEXT:    vmovaps %ymm1, %ymm3
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm3 = -(ymm0 * ymm3) + ymm2
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm3 = (ymm3 * ymm1) + ymm1
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm3) + ymm3
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; FMA-RECIP-NEXT:    vfnmadd231ps {{.*#+}} ymm2 = -(ymm0 * ymm1) + ymm2
+; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm2 = (ymm2 * ymm1) + ymm1
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; FMA-RECIP-NEXT:    vmulps %ymm1, %ymm2, %ymm3
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm1
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm3
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v8f32_two_step2:
 ; BDVER2:       # %bb.0:
 ; BDVER2-NEXT:    vrcpps %ymm0, %ymm1
-; BDVER2-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BDVER2-NEXT:    vfnmaddps %ymm2, %ymm1, %ymm0, %ymm3
-; BDVER2-NEXT:    vfmaddps %ymm1, %ymm3, %ymm1, %ymm1
-; BDVER2-NEXT:    vfnmaddps %ymm2, %ymm1, %ymm0, %ymm0
-; BDVER2-NEXT:    vfmaddps %ymm1, %ymm0, %ymm1, %ymm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BDVER2-NEXT:    vfnmaddps {{.*}}(%rip), %ymm1, %ymm0, %ymm2
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BDVER2-NEXT:    vfmaddps %ymm1, %ymm2, %ymm1, %ymm1
+; BDVER2-NEXT:    vmulps %ymm4, %ymm1, %ymm3
+; BDVER2-NEXT:    vfnmaddps %ymm4, %ymm3, %ymm0, %ymm0
+; BDVER2-NEXT:    vfmaddps %ymm3, %ymm0, %ymm1, %ymm0
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v8f32_two_step2:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BTVER2-NEXT:    vrcpps %ymm0, %ymm1
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
 ; BTVER2-NEXT:    vmulps %ymm1, %ymm0, %ymm2
 ; BTVER2-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; BTVER2-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; BTVER2-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; BTVER2-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; BTVER2-NEXT:    vmulps %ymm4, %ymm1, %ymm3
+; BTVER2-NEXT:    vmulps %ymm3, %ymm0, %ymm0
+; BTVER2-NEXT:    vsubps %ymm0, %ymm4, %ymm0
 ; BTVER2-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; BTVER2-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BTVER2-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v8f32_two_step2:
 ; SANDY:       # %bb.0:
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm1
 ; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm2
 ; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; SANDY-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; SANDY-NEXT:    vmulps %ymm3, %ymm0, %ymm0
+; SANDY-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; SANDY-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; SANDY-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v8f32_two_step2:
 ; HASWELL:       # %bb.0:
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vmovaps %ymm1, %ymm3
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm3 = -(ymm0 * ymm3) + ymm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm3 = (ymm3 * ymm1) + ymm1
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm3) + ymm3
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; HASWELL-NEXT:    vfnmadd231ps {{.*#+}} ymm2 = -(ymm0 * ymm1) + ymm2
+; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm2 = (ymm2 * ymm1) + ymm1
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NEXT:    vmulps %ymm1, %ymm2, %ymm3
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm1
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm3
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v8f32_two_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm0, %ymm2
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm2, %ymm3, %ymm2
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm2
 ; HASWELL-NO-FMA-NEXT:    vaddps %ymm2, %ymm1, %ymm1
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm2, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm1, %ymm0
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm1, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm3, %ymm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v8f32_two_step2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcpps %ymm0, %ymm1
 ; AVX512-NEXT:    vbroadcastss {{.*#+}} ymm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; AVX512-NEXT:    vmovaps %ymm1, %ymm3
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} ymm3 = -(ymm0 * ymm3) + ymm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} ymm3 = (ymm3 * ymm1) + ymm1
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm3) + ymm3
-; AVX512-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; AVX512-NEXT:    vfnmadd231ps {{.*#+}} ymm2 = -(ymm0 * ymm1) + ymm2
+; AVX512-NEXT:    vfmadd132ps {{.*#+}} ymm2 = (ymm2 * ymm1) + ymm1
+; AVX512-NEXT:    vmovaps {{.*#+}} ymm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX512-NEXT:    vmulps %ymm1, %ymm2, %ymm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm3 * ymm0) + ymm1
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, %x
   ret <8 x float> %div
 }
 
 define <8 x float> @v8f32_no_step(<8 x float> %x) #3 {
 ; SSE-LABEL: v8f32_no_step:
 ; SSE:       # %bb.0:
 ; SSE-NEXT:    rcpps %xmm0, %xmm0
 ; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: v8f32_no_step:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vrcpps %ymm0, %ymm0
 ; AVX-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
   ret <8 x float> %div
 }
 
 define <8 x float> @v8f32_no_step2(<8 x float> %x) #3 {
 ; SSE-LABEL: v8f32_no_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    rcpps %xmm0, %xmm0
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm0
+; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm1
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: v8f32_no_step2:
 ; AVX:       # %bb.0:
 ; AVX-NEXT:    vrcpps %ymm0, %ymm0
 ; AVX-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; AVX-NEXT:    retq
   %div = fdiv fast <8 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0>, %x
   ret <8 x float> %div
 }
 
 define <16 x float> @v16f32_one_step2(<16 x float> %x) #1 {
 ; SSE-LABEL: v16f32_one_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    movaps %xmm3, %xmm4
-; SSE-NEXT:    movaps %xmm2, %xmm5
-; SSE-NEXT:    movaps %xmm0, %xmm6
-; SSE-NEXT:    rcpps %xmm3, %xmm2
-; SSE-NEXT:    mulps %xmm2, %xmm4
-; SSE-NEXT:    movaps {{.*#+}} xmm0 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    movaps %xmm0, %xmm3
-; SSE-NEXT:    subps %xmm4, %xmm3
-; SSE-NEXT:    mulps %xmm2, %xmm3
-; SSE-NEXT:    addps %xmm2, %xmm3
-; SSE-NEXT:    rcpps %xmm5, %xmm4
-; SSE-NEXT:    mulps %xmm4, %xmm5
-; SSE-NEXT:    movaps %xmm0, %xmm2
-; SSE-NEXT:    subps %xmm5, %xmm2
-; SSE-NEXT:    mulps %xmm4, %xmm2
-; SSE-NEXT:    addps %xmm4, %xmm2
-; SSE-NEXT:    rcpps %xmm1, %xmm5
+; SSE-NEXT:    movaps %xmm1, %xmm4
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    rcpps %xmm0, %xmm5
+; SSE-NEXT:    movaps {{.*#+}} xmm0 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm0, %xmm6
+; SSE-NEXT:    mulps %xmm6, %xmm1
+; SSE-NEXT:    subps %xmm1, %xmm0
+; SSE-NEXT:    mulps %xmm5, %xmm0
+; SSE-NEXT:    addps %xmm6, %xmm0
+; SSE-NEXT:    rcpps %xmm4, %xmm5
+; SSE-NEXT:    movaps {{.*#+}} xmm1 = [5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SSE-NEXT:    movaps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm1, %xmm6
+; SSE-NEXT:    mulps %xmm6, %xmm4
+; SSE-NEXT:    subps %xmm4, %xmm1
 ; SSE-NEXT:    mulps %xmm5, %xmm1
-; SSE-NEXT:    movaps %xmm0, %xmm4
-; SSE-NEXT:    subps %xmm1, %xmm4
+; SSE-NEXT:    addps %xmm6, %xmm1
+; SSE-NEXT:    rcpps %xmm2, %xmm5
+; SSE-NEXT:    movaps {{.*#+}} xmm4 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1]
+; SSE-NEXT:    movaps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm4, %xmm6
+; SSE-NEXT:    mulps %xmm6, %xmm2
+; SSE-NEXT:    subps %xmm2, %xmm4
 ; SSE-NEXT:    mulps %xmm5, %xmm4
-; SSE-NEXT:    addps %xmm5, %xmm4
-; SSE-NEXT:    rcpps %xmm6, %xmm1
-; SSE-NEXT:    mulps %xmm1, %xmm6
-; SSE-NEXT:    subps %xmm6, %xmm0
-; SSE-NEXT:    mulps %xmm1, %xmm0
-; SSE-NEXT:    addps %xmm1, %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm4
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm2
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm3
-; SSE-NEXT:    movaps %xmm4, %xmm1
+; SSE-NEXT:    addps %xmm6, %xmm4
+; SSE-NEXT:    rcpps %xmm3, %xmm2
+; SSE-NEXT:    movaps {{.*#+}} xmm5 = [1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SSE-NEXT:    movaps %xmm2, %xmm6
+; SSE-NEXT:    mulps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm6, %xmm3
+; SSE-NEXT:    subps %xmm3, %xmm5
+; SSE-NEXT:    mulps %xmm2, %xmm5
+; SSE-NEXT:    addps %xmm6, %xmm5
+; SSE-NEXT:    movaps %xmm4, %xmm2
+; SSE-NEXT:    movaps %xmm5, %xmm3
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v16f32_one_step2:
 ; AVX-RECIP:       # %bb.0:
-; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; AVX-RECIP-NEXT:    vsubps %ymm1, %ymm3, %ymm1
-; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; AVX-RECIP-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; AVX-RECIP-NEXT:    vmulps %ymm4, %ymm0, %ymm0
 ; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm3, %ymm0
 ; AVX-RECIP-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm4, %ymm0
+; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm2
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; AVX-RECIP-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; AVX-RECIP-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; AVX-RECIP-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v16f32_one_step2:
 ; FMA-RECIP:       # %bb.0:
-; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm2
-; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm2 * ymm1) + ymm3
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm1 = (ymm1 * ymm2) + ymm2
 ; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm2
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm2 * ymm0) + ymm3
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm2) + ymm2
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; FMA-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm4 * ymm0) + ymm3
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm4
+; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm2
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; FMA-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm3
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm1 = (ymm2 * ymm1) + ymm4
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v16f32_one_step2:
 ; BDVER2:       # %bb.0:
-; BDVER2-NEXT:    vrcpps %ymm1, %ymm2
-; BDVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BDVER2-NEXT:    vrcpps %ymm0, %ymm4
-; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm1, %ymm1
+; BDVER2-NEXT:    vrcpps %ymm0, %ymm2
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BDVER2-NEXT:    vrcpps %ymm1, %ymm5
+; BDVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm4
 ; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm4, %ymm0, %ymm0
-; BDVER2-NEXT:    vfmaddps %ymm2, %ymm1, %ymm2, %ymm1
-; BDVER2-NEXT:    vfmaddps %ymm4, %ymm0, %ymm4, %ymm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; BDVER2-NEXT:    vfmaddps %ymm4, %ymm0, %ymm2, %ymm0
+; BDVER2-NEXT:    vmulps %ymm3, %ymm5, %ymm4
+; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm4, %ymm1, %ymm1
+; BDVER2-NEXT:    vfmaddps %ymm4, %ymm1, %ymm5, %ymm1
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v16f32_one_step2:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BTVER2-NEXT:    vrcpps %ymm1, %ymm2
-; BTVER2-NEXT:    vrcpps %ymm0, %ymm4
-; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm1
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BTVER2-NEXT:    vrcpps %ymm0, %ymm2
+; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm4
 ; BTVER2-NEXT:    vmulps %ymm4, %ymm0, %ymm0
-; BTVER2-NEXT:    vsubps %ymm1, %ymm3, %ymm1
 ; BTVER2-NEXT:    vsubps %ymm0, %ymm3, %ymm0
-; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; BTVER2-NEXT:    vmulps %ymm0, %ymm4, %ymm0
-; BTVER2-NEXT:    vaddps %ymm1, %ymm2, %ymm1
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; BTVER2-NEXT:    vmulps %ymm0, %ymm2, %ymm0
+; BTVER2-NEXT:    vrcpps %ymm1, %ymm2
+; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm5
 ; BTVER2-NEXT:    vaddps %ymm0, %ymm4, %ymm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; BTVER2-NEXT:    vmulps %ymm5, %ymm1, %ymm1
+; BTVER2-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; BTVER2-NEXT:    vaddps %ymm1, %ymm5, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_one_step2:
 ; SANDY:       # %bb.0:
-; SANDY-NEXT:    vrcpps %ymm1, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; SANDY-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; SANDY-NEXT:    vmulps %ymm4, %ymm0, %ymm0
 ; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; SANDY-NEXT:    vrcpps %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; SANDY-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; SANDY-NEXT:    vaddps %ymm0, %ymm4, %ymm0
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SANDY-NEXT:    vmulps %ymm2, %ymm3, %ymm4
+; SANDY-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; SANDY-NEXT:    vsubps %ymm1, %ymm2, %ymm1
+; SANDY-NEXT:    vmulps %ymm1, %ymm3, %ymm1
+; SANDY-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_one_step2:
 ; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %ymm1, %ymm2
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vrcpps %ymm0, %ymm4
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm2 * ymm1) + ymm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm1 = (ymm1 * ymm2) + ymm2
+; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NEXT:    vmulps %ymm3, %ymm2, %ymm4
 ; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm4 * ymm0) + ymm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm4) + ymm4
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm2 * ymm0) + ymm4
+; HASWELL-NEXT:    vrcpps %ymm1, %ymm2
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; HASWELL-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm3
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm1 = (ymm2 * ymm1) + ymm4
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v16f32_one_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
-; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm2
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NO-FMA-NEXT:    vsubps %ymm1, %ymm3, %ymm1
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm2
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm4, %ymm0, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm3
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm4, %ymm0
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm3, %ymm4
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; HASWELL-NO-FMA-NEXT:    vsubps %ymm1, %ymm2, %ymm1
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm3, %ymm1
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v16f32_one_step2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcp14ps %zmm0, %zmm1
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} zmm0 = -(zmm1 * zmm0) + mem
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} zmm0 = (zmm0 * zmm1) + zmm1
-; AVX512-NEXT:    vmulps {{.*}}(%rip), %zmm0, %zmm0
+; AVX512-NEXT:    vmovaps {{.*#+}} zmm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0,9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; AVX512-NEXT:    vmulps %zmm2, %zmm1, %zmm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} zmm0 = -(zmm3 * zmm0) + zmm2
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} zmm0 = (zmm1 * zmm0) + zmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast <16 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0, float 9.0, float 10.0, float 11.0, float 12.0, float 13.0, float 14.0, float 15.0, float 16.0>, %x
   ret <16 x float> %div
 }
 
 define <16 x float> @v16f32_one_step_2_divs(<16 x float> %x) #1 {
 ; SSE-LABEL: v16f32_one_step_2_divs:
 ; SSE:       # %bb.0:
; AVX512-NEXT:    retq
   %div = fdiv fast <16 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0, float 9.0, float 10.0, float 11.0, float 12.0, float 13.0, float 14.0, float 15.0, float 16.0>, %x
   %div2 = fdiv fast <16 x float> %div, %x
   ret <16 x float> %div2
 }
 
 define <16 x float> @v16f32_two_step2(<16 x float> %x) #2 {
 ; SSE-LABEL: v16f32_two_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    movaps %xmm3, %xmm6
-; SSE-NEXT:    movaps %xmm2, %xmm5
-; SSE-NEXT:    movaps %xmm0, %xmm4
-; SSE-NEXT:    rcpps %xmm3, %xmm2
-; SSE-NEXT:    mulps %xmm2, %xmm3
-; SSE-NEXT:    movaps {{.*#+}} xmm0 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SSE-NEXT:    movaps %xmm0, %xmm7
-; SSE-NEXT:    subps %xmm3, %xmm7
-; SSE-NEXT:    mulps %xmm2, %xmm7
-; SSE-NEXT:    addps %xmm2, %xmm7
-; SSE-NEXT:    mulps %xmm7, %xmm6
-; SSE-NEXT:    movaps %xmm0, %xmm3
-; SSE-NEXT:    subps %xmm6, %xmm3
-; SSE-NEXT:    mulps %xmm7, %xmm3
-; SSE-NEXT:    addps %xmm7, %xmm3
-; SSE-NEXT:    rcpps %xmm5, %xmm2
-; SSE-NEXT:    movaps %xmm5, %xmm6
-; SSE-NEXT:    mulps %xmm2, %xmm6
-; SSE-NEXT:    movaps %xmm0, %xmm7
-; SSE-NEXT:    subps %xmm6, %xmm7
-; SSE-NEXT:    mulps %xmm2, %xmm7
-; SSE-NEXT:    addps %xmm2, %xmm7
-; SSE-NEXT:    mulps %xmm7, %xmm5
-; SSE-NEXT:    movaps %xmm0, %xmm2
-; SSE-NEXT:    subps %xmm5, %xmm2
-; SSE-NEXT:    mulps %xmm7, %xmm2
-; SSE-NEXT:    addps %xmm7, %xmm2
-; SSE-NEXT:    rcpps %xmm1, %xmm5
-; SSE-NEXT:    movaps %xmm1, %xmm6
-; SSE-NEXT:    mulps %xmm5, %xmm6
-; SSE-NEXT:    movaps %xmm0, %xmm7
-; SSE-NEXT:    subps %xmm6, %xmm7
-; SSE-NEXT:    mulps %xmm5, %xmm7
-; SSE-NEXT:    addps %xmm5, %xmm7
-; SSE-NEXT:    mulps %xmm7, %xmm1
-; SSE-NEXT:    movaps %xmm0, %xmm5
-; SSE-NEXT:    subps %xmm1, %xmm5
-; SSE-NEXT:    mulps %xmm7, %xmm5
-; SSE-NEXT:    addps %xmm7, %xmm5
+; SSE-NEXT:    movaps %xmm1, %xmm4
+; SSE-NEXT:    movaps %xmm0, %xmm1
+; SSE-NEXT:    rcpps %xmm0, %xmm0
+; SSE-NEXT:    movaps %xmm1, %xmm5
+; SSE-NEXT:    mulps %xmm0, %xmm5
+; SSE-NEXT:    movaps {{.*#+}} xmm6 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0]
+; SSE-NEXT:    movaps %xmm6, %xmm7
+; SSE-NEXT:    subps %xmm5, %xmm7
+; SSE-NEXT:    mulps %xmm0, %xmm7
+; SSE-NEXT:    addps %xmm0, %xmm7
+; SSE-NEXT:    movaps {{.*#+}} xmm0 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0]
+; SSE-NEXT:    movaps %xmm7, %xmm5
+; SSE-NEXT:    mulps %xmm0, %xmm5
+; SSE-NEXT:    mulps %xmm5, %xmm1
+; SSE-NEXT:    subps %xmm1, %xmm0
+; SSE-NEXT:    mulps %xmm7, %xmm0
+; SSE-NEXT:    addps %xmm5, %xmm0
 ; SSE-NEXT:    rcpps %xmm4, %xmm1
-; SSE-NEXT:    movaps %xmm4, %xmm6
-; SSE-NEXT:    mulps %xmm1, %xmm6
-; SSE-NEXT:    movaps %xmm0, %xmm7
-; SSE-NEXT:    subps %xmm6, %xmm7
+; SSE-NEXT:    movaps %xmm4, %xmm5
+; SSE-NEXT:    mulps %xmm1, %xmm5
+; SSE-NEXT:    movaps %xmm6, %xmm7
+; SSE-NEXT:    subps %xmm5, %xmm7
 ; SSE-NEXT:    mulps %xmm1, %xmm7
 ; SSE-NEXT:    addps %xmm1, %xmm7
+; SSE-NEXT:    movaps {{.*#+}} xmm1 = [5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SSE-NEXT:    movaps %xmm7, %xmm5
+; SSE-NEXT:    mulps %xmm1, %xmm5
+; SSE-NEXT:    mulps %xmm5, %xmm4
+; SSE-NEXT:    subps %xmm4, %xmm1
+; SSE-NEXT:    mulps %xmm7, %xmm1
+; SSE-NEXT:    addps %xmm5, %xmm1
+; SSE-NEXT:    rcpps %xmm2, %xmm4
+; SSE-NEXT:    movaps %xmm2, %xmm5
+; SSE-NEXT:    mulps %xmm4, %xmm5
+; SSE-NEXT:    movaps %xmm6, %xmm7
+; SSE-NEXT:    subps %xmm5, %xmm7
+; SSE-NEXT:    mulps %xmm4, %xmm7
+; SSE-NEXT:    addps %xmm4, %xmm7
+; SSE-NEXT:    movaps {{.*#+}} xmm4 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1]
+; SSE-NEXT:    movaps %xmm7, %xmm5
+; SSE-NEXT:    mulps %xmm4, %xmm5
+; SSE-NEXT:    mulps %xmm5, %xmm2
+; SSE-NEXT:    subps %xmm2, %xmm4
 ; SSE-NEXT:    mulps %xmm7, %xmm4
-; SSE-NEXT:    subps %xmm4, %xmm0
-; SSE-NEXT:    mulps %xmm7, %xmm0
-; SSE-NEXT:    addps %xmm7, %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm0
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm5
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm2
-; SSE-NEXT:    mulps {{.*}}(%rip), %xmm3
-; SSE-NEXT:    movaps %xmm5, %xmm1
+; SSE-NEXT:    addps %xmm5, %xmm4
+; SSE-NEXT:    rcpps %xmm3, %xmm2
+; SSE-NEXT:    movaps %xmm3, %xmm5
+; SSE-NEXT:    mulps %xmm2, %xmm5
+; SSE-NEXT:    subps %xmm5, %xmm6
+; SSE-NEXT:    mulps %xmm2, %xmm6
+; SSE-NEXT:    addps %xmm2, %xmm6
+; SSE-NEXT:    movaps {{.*#+}} xmm5 = [1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SSE-NEXT:    movaps %xmm6, %xmm2
+; SSE-NEXT:    mulps %xmm5, %xmm2
+; SSE-NEXT:    mulps %xmm2, %xmm3
+; SSE-NEXT:    subps %xmm3, %xmm5
+; SSE-NEXT:    mulps %xmm6, %xmm5
+; SSE-NEXT:    addps %xmm2, %xmm5
+; SSE-NEXT:    movaps %xmm4, %xmm2
+; SSE-NEXT:    movaps %xmm5, %xmm3
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v16f32_two_step2:
 ; AVX-RECIP:       # %bb.0:
-; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm2
+; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm0, %ymm3
 ; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; AVX-RECIP-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; AVX-RECIP-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; AVX-RECIP-NEXT:    vsubps %ymm1, %ymm4, %ymm1
-; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; AVX-RECIP-NEXT:    vaddps %ymm1, %ymm2, %ymm1
-; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm0, %ymm3
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm5
+; AVX-RECIP-NEXT:    vmulps %ymm5, %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; AVX-RECIP-NEXT:    vmulps %ymm0, %ymm2, %ymm0
+; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm5, %ymm0
+; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm2
+; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm1, %ymm3
 ; AVX-RECIP-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; AVX-RECIP-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; AVX-RECIP-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vsubps %ymm0, %ymm4, %ymm0
-; AVX-RECIP-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; AVX-RECIP-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; AVX-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; AVX-RECIP-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; AVX-RECIP-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; AVX-RECIP-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; AVX-RECIP-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; AVX-RECIP-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v16f32_two_step2:
 ; FMA-RECIP:       # %bb.0:
-; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm2
-; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; FMA-RECIP-NEXT:    vmovaps %ymm2, %ymm4
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm4 = -(ymm1 * ymm4) + ymm3
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm4 = (ymm4 * ymm2) + ymm2
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm3
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm1 = (ymm1 * ymm4) + ymm4
 ; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm2
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; FMA-RECIP-NEXT:    vmovaps %ymm2, %ymm4
 ; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm4 = -(ymm0 * ymm4) + ymm3
 ; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm4 = (ymm4 * ymm2) + ymm2
-; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm4 * ymm0) + ymm3
-; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm4) + ymm4
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; FMA-RECIP-NEXT:    vmulps %ymm2, %ymm4, %ymm5
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm5 * ymm0) + ymm2
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm4 * ymm0) + ymm5
+; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm2
+; FMA-RECIP-NEXT:    vfnmadd231ps {{.*#+}} ymm3 = -(ymm1 * ymm2) + ymm3
+; FMA-RECIP-NEXT:    vfmadd132ps {{.*#+}} ymm3 = (ymm3 * ymm2) + ymm2
+; FMA-RECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; FMA-RECIP-NEXT:    vmulps %ymm2, %ymm3, %ymm4
+; FMA-RECIP-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm2
+; FMA-RECIP-NEXT:    vfmadd213ps {{.*#+}} ymm1 = (ymm3 * ymm1) + ymm4
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v16f32_two_step2:
 ; BDVER2:       # %bb.0:
-; BDVER2-NEXT:    vrcpps %ymm1, %ymm2
-; BDVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm1, %ymm4
-; BDVER2-NEXT:    vfmaddps %ymm2, %ymm4, %ymm2, %ymm2
-; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm1, %ymm1
-; BDVER2-NEXT:    vfmaddps %ymm2, %ymm1, %ymm2, %ymm1
 ; BDVER2-NEXT:    vrcpps %ymm0, %ymm2
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm0, %ymm4
 ; BDVER2-NEXT:    vfmaddps %ymm2, %ymm4, %ymm2, %ymm2
-; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm0, %ymm0
-; BDVER2-NEXT:    vfmaddps %ymm2, %ymm0, %ymm2, %ymm0
-; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BDVER2-NEXT:    vmulps %ymm4, %ymm2, %ymm5
+; BDVER2-NEXT:    vfnmaddps %ymm4, %ymm5, %ymm0, %ymm0
+; BDVER2-NEXT:    vfmaddps %ymm5, %ymm0, %ymm2, %ymm0
+; BDVER2-NEXT:    vrcpps %ymm1, %ymm2
+; BDVER2-NEXT:    vmovaps {{.*#+}} ymm5 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; BDVER2-NEXT:    vfnmaddps %ymm3, %ymm2, %ymm1, %ymm3
+; BDVER2-NEXT:    vfmaddps %ymm2, %ymm3, %ymm2, %ymm2
+; BDVER2-NEXT:    vmulps %ymm5, %ymm2, %ymm4
+; BDVER2-NEXT:    vfnmaddps %ymm5, %ymm4, %ymm1, %ymm1
+; BDVER2-NEXT:    vfmaddps %ymm4, %ymm1, %ymm2, %ymm1
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v16f32_two_step2:
 ; BTVER2:       # %bb.0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; BTVER2-NEXT:    vrcpps %ymm1, %ymm2
-; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm3
-; BTVER2-NEXT:    vsubps %ymm3, %ymm4, %ymm3
-; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm3
-; BTVER2-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; BTVER2-NEXT:    vsubps %ymm1, %ymm4, %ymm1
-; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; BTVER2-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; BTVER2-NEXT:    vrcpps %ymm0, %ymm2
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; BTVER2-NEXT:    vmulps %ymm2, %ymm0, %ymm3
 ; BTVER2-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; BTVER2-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; BTVER2-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; BTVER2-NEXT:    vsubps %ymm0, %ymm4, %ymm0
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm5
+; BTVER2-NEXT:    vmulps %ymm5, %ymm0, %ymm0
+; BTVER2-NEXT:    vsubps %ymm0, %ymm3, %ymm0
 ; BTVER2-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; BTVER2-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; BTVER2-NEXT:    vrcpps %ymm1, %ymm2
+; BTVER2-NEXT:    vmulps %ymm2, %ymm1, %ymm3
+; BTVER2-NEXT:    vaddps %ymm0, %ymm5, %ymm0
+; BTVER2-NEXT:    vmovaps {{.*#+}} ymm5 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; BTVER2-NEXT:    vsubps %ymm3, %ymm4, %ymm3
+; BTVER2-NEXT:    vmulps %ymm3, %ymm2, %ymm3
+; BTVER2-NEXT:    vaddps %ymm3, %ymm2, %ymm2
+; BTVER2-NEXT:    vmulps %ymm5, %ymm2, %ymm4
+; BTVER2-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; BTVER2-NEXT:    vsubps %ymm1, %ymm5, %ymm1
+; BTVER2-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; BTVER2-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_two_step2:
 ; SANDY:       # %bb.0:
-; SANDY-NEXT:    vrcpps %ymm1, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm3
-; SANDY-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; SANDY-NEXT:    vsubps %ymm3, %ymm4, %ymm3
-; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm3
-; SANDY-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; SANDY-NEXT:    vsubps %ymm1, %ymm4, %ymm1
-; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; SANDY-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm2
 ; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm3
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; SANDY-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; SANDY-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; SANDY-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; SANDY-NEXT:    vsubps %ymm0, %ymm4, %ymm0
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm5
+; SANDY-NEXT:    vmulps %ymm5, %ymm0, %ymm0
+; SANDY-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; SANDY-NEXT:    vrcpps %ymm1, %ymm3
 ; SANDY-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; SANDY-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; SANDY-NEXT:    vaddps %ymm0, %ymm5, %ymm0
+; SANDY-NEXT:    vmulps %ymm3, %ymm1, %ymm2
+; SANDY-NEXT:    vsubps %ymm2, %ymm4, %ymm2
+; SANDY-NEXT:    vmulps %ymm2, %ymm3, %ymm2
+; SANDY-NEXT:    vaddps %ymm2, %ymm3, %ymm2
+; SANDY-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; SANDY-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; SANDY-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; SANDY-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; SANDY-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; SANDY-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_two_step2:
 ; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %ymm1, %ymm2
-; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NEXT:    vmovaps %ymm2, %ymm4
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm4 = -(ymm1 * ymm4) + ymm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm4 = (ymm4 * ymm2) + ymm2
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm1 = (ymm1 * ymm4) + ymm4
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm2
+; HASWELL-NEXT:    vbroadcastss {{.*#+}} ymm3 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NEXT:    vmovaps %ymm2, %ymm4
 ; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm4 = -(ymm0 * ymm4) + ymm3
 ; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm4 = (ymm4 * ymm2) + ymm2
-; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm4 * ymm0) + ymm3
-; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm0 = (ymm0 * ymm4) + ymm4
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm2 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NEXT:    vmulps %ymm2, %ymm4, %ymm5
+; HASWELL-NEXT:    vrcpps %ymm1, %ymm6
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm0 = -(ymm5 * ymm0) + ymm2
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm0 = (ymm4 * ymm0) + ymm5
+; HASWELL-NEXT:    vfnmadd231ps {{.*#+}} ymm3 = -(ymm1 * ymm6) + ymm3
+; HASWELL-NEXT:    vfmadd132ps {{.*#+}} ymm3 = (ymm3 * ymm6) + ymm6
+; HASWELL-NEXT:    vmovaps {{.*#+}} ymm2 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; HASWELL-NEXT:    vmulps %ymm2, %ymm3, %ymm4
+; HASWELL-NEXT:    vfnmadd213ps {{.*#+}} ymm1 = -(ymm4 * ymm1) + ymm2
+; HASWELL-NEXT:    vfmadd213ps {{.*#+}} ymm1 = (ymm3 * ymm1) + ymm4
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v16f32_two_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
-; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm2
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm3
-; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; HASWELL-NO-FMA-NEXT:    vsubps %ymm3, %ymm4, %ymm3
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm2, %ymm3
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm1, %ymm1
-; HASWELL-NO-FMA-NEXT:    vsubps %ymm1, %ymm4, %ymm1
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm2, %ymm1
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm1, %ymm2, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm2
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm0, %ymm3
+; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*#+}} ymm4 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
 ; HASWELL-NO-FMA-NEXT:    vsubps %ymm3, %ymm4, %ymm3
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm2, %ymm3
 ; HASWELL-NO-FMA-NEXT:    vaddps %ymm3, %ymm2, %ymm2
-; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm4, %ymm0
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm3 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm2, %ymm5
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm5, %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vsubps %ymm0, %ymm3, %ymm0
+; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm3
 ; HASWELL-NO-FMA-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm2, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
-; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm0, %ymm5, %ymm0
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm1, %ymm2
+; HASWELL-NO-FMA-NEXT:    vsubps %ymm2, %ymm4, %ymm2
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm2, %ymm3, %ymm2
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm2, %ymm3, %ymm2
+; HASWELL-NO-FMA-NEXT:    vmovaps {{.*#+}} ymm3 = [9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm3, %ymm2, %ymm4
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm4, %ymm1, %ymm1
+; HASWELL-NO-FMA-NEXT:    vsubps %ymm1, %ymm3, %ymm1
+; HASWELL-NO-FMA-NEXT:    vmulps %ymm1, %ymm2, %ymm1
+; HASWELL-NO-FMA-NEXT:    vaddps %ymm1, %ymm4, %ymm1
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v16f32_two_step2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcp14ps %zmm0, %zmm1
 ; AVX512-NEXT:    vbroadcastss {{.*#+}} zmm2 = [1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0,1.0E+0]
-; AVX512-NEXT:    vmovaps %zmm1, %zmm3
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} zmm3 = -(zmm0 * zmm3) + zmm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} zmm3 = (zmm3 * zmm1) + zmm1
-; AVX512-NEXT:    vfnmadd213ps {{.*#+}} zmm0 = -(zmm3 * zmm0) + zmm2
-; AVX512-NEXT:    vfmadd132ps {{.*#+}} zmm0 = (zmm0 * zmm3) + zmm3
-; AVX512-NEXT:    vmulps {{.*}}(%rip), %zmm0, %zmm0
+; AVX512-NEXT:    vfnmadd231ps {{.*#+}} zmm2 = -(zmm0 * zmm1) + zmm2
+; AVX512-NEXT:    vfmadd132ps {{.*#+}} zmm2 = (zmm2 * zmm1) + zmm1
+; AVX512-NEXT:    vmovaps {{.*#+}} zmm1 = [1.0E+0,2.0E+0,3.0E+0,4.0E+0,5.0E+0,6.0E+0,7.0E+0,8.0E+0,9.0E+0,1.0E+1,1.1E+1,1.2E+1,1.3E+1,1.4E+1,1.5E+1,1.6E+1]
+; AVX512-NEXT:    vmulps %zmm1, %zmm2, %zmm3
+; AVX512-NEXT:    vfnmadd213ps {{.*#+}} zmm0 = -(zmm3 * zmm0) + zmm1
+; AVX512-NEXT:    vfmadd213ps {{.*#+}} zmm0 = (zmm2 * zmm0) + zmm3
 ; AVX512-NEXT:    retq
   %div = fdiv fast <16 x float> <float 1.0, float 2.0, float 3.0, float 4.0, float 5.0, float 6.0, float 7.0, float 8.0, float 9.0, float 10.0, float 11.0, float 12.0, float 13.0, float 14.0, float 15.0, float 16.0>, %x
   ret <16 x float> %div
 }
 
 define <16 x float> @v16f32_no_step(<16 x float> %x) #3 {
 ; SSE-LABEL: v16f32_no_step:
 ; SSE:       # %bb.0:

 ; AVX512-NEXT:    retq
   %div = fdiv fast <16 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
   ret <16 x float> %div
 }
 
 define <16 x float> @v16f32_no_step2(<16 x float> %x) #3 {
 ; SSE-LABEL: v16f32_no_step2:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    rcpps %xmm3, %xmm3
-; SSE-NEXT:    rcpps %xmm2, %xmm2
-; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    rcpps %xmm0, %xmm0
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm0
+; SSE-NEXT:    rcpps %xmm1, %xmm1
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm1
+; SSE-NEXT:    rcpps %xmm2, %xmm2
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm2
+; SSE-NEXT:    rcpps %xmm3, %xmm3
 ; SSE-NEXT:    mulps {{.*}}(%rip), %xmm3
 ; SSE-NEXT:    retq
 ;
 ; AVX-RECIP-LABEL: v16f32_no_step2:
 ; AVX-RECIP:       # %bb.0:
-; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm1
 ; AVX-RECIP-NEXT:    vrcpps %ymm0, %ymm0
 ; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; AVX-RECIP-NEXT:    vrcpps %ymm1, %ymm1
 ; AVX-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; AVX-RECIP-NEXT:    retq
 ;
 ; FMA-RECIP-LABEL: v16f32_no_step2:
 ; FMA-RECIP:       # %bb.0:
-; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm1
 ; FMA-RECIP-NEXT:    vrcpps %ymm0, %ymm0
 ; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
+; FMA-RECIP-NEXT:    vrcpps %ymm1, %ymm1
 ; FMA-RECIP-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; FMA-RECIP-NEXT:    retq
 ;
 ; BDVER2-LABEL: v16f32_no_step2:
 ; BDVER2:       # %bb.0:
-; BDVER2-NEXT:    vrcpps %ymm1, %ymm1
 ; BDVER2-NEXT:    vrcpps %ymm0, %ymm0
+; BDVER2-NEXT:    vrcpps %ymm1, %ymm1
 ; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; BDVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; BDVER2-NEXT:    retq
 ;
 ; BTVER2-LABEL: v16f32_no_step2:
 ; BTVER2:       # %bb.0:
-; BTVER2-NEXT:    vrcpps %ymm1, %ymm1
 ; BTVER2-NEXT:    vrcpps %ymm0, %ymm0
+; BTVER2-NEXT:    vrcpps %ymm1, %ymm1
 ; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; BTVER2-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; BTVER2-NEXT:    retq
 ;
 ; SANDY-LABEL: v16f32_no_step2:
 ; SANDY:       # %bb.0:
-; SANDY-NEXT:    vrcpps %ymm1, %ymm1
 ; SANDY-NEXT:    vrcpps %ymm0, %ymm0
+; SANDY-NEXT:    vrcpps %ymm1, %ymm1
 ; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; SANDY-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; SANDY-NEXT:    retq
 ;
 ; HASWELL-LABEL: v16f32_no_step2:
 ; HASWELL:       # %bb.0:
-; HASWELL-NEXT:    vrcpps %ymm1, %ymm1
 ; HASWELL-NEXT:    vrcpps %ymm0, %ymm0
+; HASWELL-NEXT:    vrcpps %ymm1, %ymm1
 ; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; HASWELL-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; HASWELL-NEXT:    retq
 ;
 ; HASWELL-NO-FMA-LABEL: v16f32_no_step2:
 ; HASWELL-NO-FMA:       # %bb.0:
-; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vrcpps %ymm0, %ymm0
+; HASWELL-NO-FMA-NEXT:    vrcpps %ymm1, %ymm1
 ; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm0, %ymm0
 ; HASWELL-NO-FMA-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v16f32_no_step2:
 ; AVX512:       # %bb.0:
 ; AVX512-NEXT:    vrcp14ps %zmm0, %zmm0
 ; AVX512-NEXT:    vmulps {{.*}}(%rip), %zmm0, %zmm0