make reciprocal estimate code generation more flexible by adding command-line options

[spatel]

Summary

We need separation of scalar and vector reciprocal codegen to handle an -mrecip clang flag that provides equal functionality to gcc's:
https://gcc.gnu.org/onlinedocs/gcc-4.9.2/gcc/i386-and-x86-64-Options.html#index-mrecip_003dopt-1627

This patch adds a Target class for processing all of the recip codegen possibilities. The x86 backend is updated to use the new functionality.

[hfinkel]

Whether or not we're allowed to use estimates for operations that are normally required to be exact, div and sqrt, are not really target features. They're much closer, conceptually, to UnsafeFPMath and friends. Plus, we need these same options for other backends (PowerPC, for example). I'd like to see flags for these added to include/llvm/Target/TargetOptions.h (along with UnsafeFPMath) so that we can handle them in a uniform way.

[spatel]

Hi Hal -

Thanks for the quick feedback. I'm looking at the PPC variations that gcc supports. I'm thinking that we'll need an enum with bitmasks to cover all the potential variations. Just making sure that I'm not going into the weeds...

Also (if maintaining compatibility with gcc is a goal), we'll still need to accept and process different flags per arch.

[hfinkel]

In D8982#155207, @spatel wrote:

Hi Hal -

Thanks for the quick feedback. I'm looking at the PPC variations that gcc supports. I'm thinking that we'll need an enum with bitmasks to cover all the potential variations. Just making sure that I'm not going into the weeds...

I think that sounds right.

Also (if maintaining compatibility with gcc is a goal), we'll still need to accept and process different flags per arch.

Not exactly in that sense. If we accept a superset of the options that gcc supports because we apply them to targets/variants that gcc does not, that's fine. This seems pretty target independent, and I'd rather handle it in a uniform way if possible.

[spatel]

After thinking it over, I decided that a simple enum of reciprocal operation bools wasn't going to do the job. In addition to wanting to enable specific ops on specific data types, we have users asking to customize the number of N-R refinements calculated...and they want different counts based on the type of op.

The best way to achieve this level of flexibility (and further customization requests that are sure to follow) is with a 'TargetRecip' class. So that's what I'm proposing to add in this updated patch.

From the llc command-line, you can now do something like this:
$ llc -recip=vec-divf,sqrtd.2

That translates to: allow vector float division and scalar double square root codegen. Use the target default setting for N-R refinement steps for the div but override the refinement steps for the sqrt to be '2'.

To minimize change in the updated x86 backend, I've defaulted to disabling all recip codegen, but for enabled operations, we use 1 N-R step.

This means only users that were targeting AMD Jaguar will see a functional change from this patch (that chip had defaulted both sqrt and div codegen for scalars and vectors on). We can easily change the x86 target defaults in a follow-on patch. We'll also want to update the PowerPC and R600 backends to use this new command-line functionality.

I wanted to add more testing for the llc parameter parsing, but I'm not finding a way to do it. If there's a way to do that with a generic target, please let me know.

I have a dependent clang driver / front-end patch to pass similar command-line params through to the backend in progress.

[spatel]

Patch updated: minor cleanup to spacing, includes, interface, comments.

[spatel]

Ping.

[spatel]

Ping * 2.

[spatel]

Ping * 3.

[spatel]

Ping * 4.

[hfinkel]

include/llvm/Target/TargetOptions.h

240 ↗

(On Diff #23815)

reciprocal estimate -> reciprocal-estimate

include/llvm/Target/TargetRecip.h

34 ↗

(On Diff #23815)

To reduce confusion, I think it would be better to have a naming prefix on these. TailCallKind, for example, uses TCK_. Let's stick RO_ on these (including the INVALID one).

34 ↗

(On Diff #23815)

If you're using INVALID for the "number of", please name it NUM_RECIP_OPS (or similar).

48 ↗

(On Diff #23815)

Let's not use INVALID for "all". You can add an All to the enum (it can even have the same value as INVALID).

lib/Target/TargetRecip.cpp

58 ↗

(On Diff #23815)

These strings are user-provided, we can't assert on invalid inputs.

148 ↗

(On Diff #23815)

Hrmm, there could be duplicates. Just parse them in order (users may provide duplicates).

[spatel]

include/llvm/Target/TargetRecip.h

34 ↗

(On Diff #23815)

With a named enum, I was expecting that users always have to use the prefix "RecipOps::" like I did in the x86 code.

But I see that's not done with TailCallKind...

[spatel]

include/llvm/Target/TargetOptions.h

240 ↗

(On Diff #23815)

Fixed.

include/llvm/Target/TargetRecip.h

34 ↗

(On Diff #23815)

Fixed.

34 ↗

(On Diff #23815)

Fixed.

48 ↗

(On Diff #23815)

Fixed - added an RO_All that is equal to RO_NUM_RECIP_OPS.

lib/Target/TargetRecip.cpp

58 ↗

(On Diff #23815)

Fixed - replaced with 'report_fatal_error'.

148 ↗

(On Diff #23815)

Fixed. Still not sure if there's a decent way to test option parsing for llc, so there are very likely malformed input parsing bugs here.

[spatel]

Patch updated based on feedback from Hal - thanks!

[spatel]

Patch updated:

1. Use '!' to indicate disabled (match change in D8989)
2. Use array of strings as keys to a map instead of enum + loosely-coupled array of recip parameters.

[hfinkel]

include/llvm/Target/TargetRecip.h

66 ↗

(On Diff #26013)

function names should start with a lower-case letter.

lib/Target/TargetRecip.cpp

49 ↗

(On Diff #26013)

Function names start with a lower-case letter.

112 ↗

(On Diff #26013)

DisabledPrefix (this is not a macro, don't name it like one)

132 ↗

(On Diff #26013)

Extra space?

136 ↗

(On Diff #26013)

Please also try with the 'd' suffix.

[spatel]

include/llvm/Target/TargetRecip.h

66 ↗

(On Diff #26013)

Fixed.

lib/Target/TargetRecip.cpp

49 ↗

(On Diff #26013)

Fixed.

112 ↗

(On Diff #26013)

Fixed.

132 ↗

(On Diff #26013)

Fixed. Nice catch. :)

136 ↗

(On Diff #26013)

If we matched a 'd' entry but failed 'f', that would be an assertion failure given the logic below. Let me know if you were thinking of something else.

[spatel]

Patch updated based on Hal's feedback:

1. Fixed function names to start with lowercase
2. Fixed variable names to not be all caps
3. Removed extra space
4. Added assert for 'f' and 'd' suffix matching

[hfinkel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

Why false? Do you want a target feature here?

[spatel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

We had target features to control these, but I think it would be better to behave like gcc unless we have reason to diverge.

That said, this does not match gcc behavior yet; that would be my next patch. For x86 at least, we would turn the following on by default when using -ffast-math:
sqrt
vec-sqrt
vec-div

I didn't set these defaults in this patch because it would change -ffast-math codegen for all CPUs other than btver2 (which had the recip codegen enabled for all eligible x86 recip types via target features).

[hfinkel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

I thought that getRsqrtEstimate, etc. were only called when fast-math is on?

[spatel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

That's correct; everything is gated by -ffast-math or an unsafe algebra equivalent. Recip-est codegen should only be active after that check.

With gcc, once you have -ffast-math, you also get -mrecip=sqrt,vec-sqrt,vec-div by default. AFAIK, that is independent of arch or CPU subtarget.

The lone exception is 'div' (scalar division). Estimating scalar div on x86 (no FMA until recently) breaks a lot of real world code, so I want to keep that off by default (and again match gcc default behavior).

[hfinkel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

But you turn them all off here, right? So that does not match gcc. Please add a comment here explaining what is going on.

[spatel]

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

Right - my intent was to separate the structural change from the functional change as much as possible.

We don't currently generate any reciprocal estimates for x86 with -ffast-math except when targeting btver2. So all x86 codegen should be identical after this patch except in the case where someone has specified -mcpu=btver2 -ffast-math.

I'll send the patch to change this default behavior to match GCC defaults as soon as possible, but I wanted to keep that separate in case anyone does not agree that we should change that default behavior. PPC and ARM recip patches would also be independent but similar (assuming they want to match GCC too).

[hfinkel]

LGTM.

lib/Target/X86/X86TargetMachine.cpp

108 ↗

(On Diff #26185)

Okay, but please add a comment explaining the current state of things, with a TODO, here (maybe the follow-up with happen soon, but in case there's trouble, the code will be clear in the mean time).

[spatel]

Patch updated:
Added TODO comment to explain why x86 reciprocal estimate is defaulted to 'off' and the reason to change that default soon.

lib/Target/X86/X86.td

def FeatureCallRegIndirect : SubtargetFeature<"call-reg-indirect",
                                      "CallRegIndirect", "true",
                                      "Call register indirect">;
 def FeatureLEAUsesAG : SubtargetFeature<"lea-uses-ag", "LEAUsesAG", "true",
                                    "LEA instruction needs inputs at AG stage">;
 def FeatureSlowLEA : SubtargetFeature<"slow-lea", "SlowLEA", "true",
                                    "LEA instruction with certain arguments is slow">;
 def FeatureSlowIncDec : SubtargetFeature<"slow-incdec", "SlowIncDec", "true",
                                    "INC and DEC instructions are slower than ADD and SUB">;
-def FeatureUseSqrtEst : SubtargetFeature<"use-sqrt-est", "UseSqrtEst", "true",
-                            "Use RSQRT* to optimize square root calculations">;
-def FeatureUseRecipEst : SubtargetFeature<"use-recip-est", "UseReciprocalEst",
-                          "true", "Use RCP* to optimize division calculations">;
+def FeatureRecip : SubtargetFeature<"recip-div", "UseRecip", "true",
+                        "Allow scalar reciprocal estimate code generation">;
+def FeatureVecRecip : SubtargetFeature<"recip-vec-div", "UseVecRecip", "true",
+                        "Allow vector reciprocal estimate code generation">;
+def FeatureRsqrt : SubtargetFeature<"recip-sqrt", "UseRsqrt", "true",
+                        "Allow scalar reciprocal square root code generation">;
+def FeatureVecRsqrt : SubtargetFeature<"recip-vec-sqrt", "UseVecRsqrt", "true",
+                        "Allow vector reciprocal square root code generation">;
 
 //===----------------------------------------------------------------------===//
 // X86 processors supported.
 //===----------------------------------------------------------------------===//
 
 include "X86Schedule.td"
 
 def ProcIntelAtom : SubtargetFeature<"atom", "X86ProcFamily", "IntelAtom",
def : Proc<"btver1",          [FeatureSSSE3, FeatureSSE4A, FeatureCMPXCHG16B,
                                FeatureSlowSHLD]>;
 
 // Jaguar
 def : ProcessorModel<"btver2", BtVer2Model,
                      [FeatureAVX, FeatureSSE4A, FeatureCMPXCHG16B,
                       FeaturePRFCHW, FeatureAES, FeaturePCLMUL,
                       FeatureBMI, FeatureF16C, FeatureMOVBE,
                       FeatureLZCNT, FeaturePOPCNT, FeatureFastUAMem,
-                      FeatureSlowSHLD, FeatureUseSqrtEst, FeatureUseRecipEst]>;
+                      FeatureSlowSHLD, FeatureRecip, FeatureVecRecip,
+                      FeatureRsqrt, FeatureVecRsqrt]>;
 
 // TODO: We should probably add 'FeatureFastUAMem' to all of the AMD chips.
 
 // Bulldozer
 def : Proc<"bdver1",          [FeatureXOP, FeatureFMA4, FeatureCMPXCHG16B,
                                FeatureAES, FeaturePRFCHW, FeaturePCLMUL,
                                FeatureAVX, FeatureSSE4A, FeatureLZCNT,
                                FeaturePOPCNT, FeatureSlowSHLD]>;

lib/Target/X86/X86ISelLowering.cpp

 }
 
 /// The minimum architected relative accuracy is 2^-12. We need one
 /// Newton-Raphson step to have a good float result (24 bits of precision).
 SDValue X86TargetLowering::getRsqrtEstimate(SDValue Op,
                                             DAGCombinerInfo &DCI,
                                             unsigned &RefinementSteps,
                                             bool &UseOneConstNR) const {
-  // FIXME: We should use instruction latency models to calculate the cost of
-  // each potential sequence, but this is very hard to do reliably because
-  // at least Intel's Core* chips have variable timing based on the number of
-  // significant digits in the divisor and/or sqrt operand.
-  if (!Subtarget->useSqrtEst())
-    return SDValue();
-
   EVT VT = Op.getValueType();
 
   // SSE1 has rsqrtss and rsqrtps.
   // TODO: Add support for AVX512 (v16f32).
   // It is likely not profitable to do this for f64 because a double-precision
   // rsqrt estimate with refinement on x86 prior to FMA requires at least 16
   // instructions: convert to single, rsqrtss, convert back to double, refine
   // (3 steps = at least 13 insts). If an 'rsqrtsd' variant was added to the ISA
   // along with FMA, this could be a throughput win.
-  if ((Subtarget->hasSSE1() && (VT == MVT::f32 || VT == MVT::v4f32)) ||
-      (Subtarget->hasAVX() && VT == MVT::v8f32)) {
+  bool UseRSQRTSS =
+    (Subtarget->hasSSE1() && Subtarget->useRsqrtEst() && VT == MVT::f32);
+  bool UseRSQRTPS =
+    (Subtarget->hasSSE1() && Subtarget->useVecRsqrtEst() && VT == MVT::v4f32);
+  bool UseVRSQRTPS =
+    (Subtarget->hasAVX() && Subtarget->useVecRsqrtEst() && VT == MVT::v8f32);
+  
+  if (UseRSQRTSS || UseRSQRTPS || UseVRSQRTPS) {
     RefinementSteps = 1;
     UseOneConstNR = false;
     return DCI.DAG.getNode(X86ISD::FRSQRT, SDLoc(Op), VT, Op);
   }
   return SDValue();
 }
 
 /// The minimum architected relative accuracy is 2^-12. We need one
 /// Newton-Raphson step to have a good float result (24 bits of precision).
 SDValue X86TargetLowering::getRecipEstimate(SDValue Op,
                                             DAGCombinerInfo &DCI,
                                             unsigned &RefinementSteps) const {
-  // FIXME: We should use instruction latency models to calculate the cost of
-  // each potential sequence, but this is very hard to do reliably because
-  // at least Intel's Core* chips have variable timing based on the number of
-  // significant digits in the divisor.
-  if (!Subtarget->useReciprocalEst())
-    return SDValue();
-
   EVT VT = Op.getValueType();
 
   // SSE1 has rcpss and rcpps. AVX adds a 256-bit variant for rcpps.
   // TODO: Add support for AVX512 (v16f32).
   // It is likely not profitable to do this for f64 because a double-precision
   // reciprocal estimate with refinement on x86 prior to FMA requires
   // 15 instructions: convert to single, rcpss, convert back to double, refine
   // (3 steps = 12 insts). If an 'rcpsd' variant was added to the ISA
   // along with FMA, this could be a throughput win.
-  if ((Subtarget->hasSSE1() && (VT == MVT::f32 || VT == MVT::v4f32)) ||
-      (Subtarget->hasAVX() && VT == MVT::v8f32)) {
+  bool UseRCPSS =
+    (Subtarget->hasSSE1() && Subtarget->useRecipEst() && VT == MVT::f32);
+  bool UseRCPPS =
+    (Subtarget->hasSSE1() && Subtarget->useVecRecipEst() && VT == MVT::v4f32);
+  bool UseVRCPPS =
+    (Subtarget->hasAVX() && Subtarget->useVecRecipEst() && VT == MVT::v8f32);
+
+  if (UseRCPSS || UseRCPPS || UseVRCPPS) {
     RefinementSteps = ReciprocalEstimateRefinementSteps;
     return DCI.DAG.getNode(X86ISD::FRCP, SDLoc(Op), VT, Op);
   }
   return SDValue();
 }
 
 static bool isAllOnes(SDValue V) {
   ConstantSDNode *C = dyn_cast<ConstantSDNode>(V);

lib/Target/X86/X86Subtarget.h

protected:
   bool LEAUsesAG;
 
   /// True if the LEA instruction with certain arguments is slow
   bool SlowLEA;
 
   /// True if INC and DEC instructions are slow when writing to flags
   bool SlowIncDec;
 
-  /// Use the RSQRT* instructions to optimize square root calculations.
-  /// For this to be profitable, the cost of FSQRT and FDIV must be
-  /// substantially higher than normal FP ops like FADD and FMUL.
-  bool UseSqrtEst;
-
-  /// Use the RCP* instructions to optimize FP division calculations.
-  /// For this to be profitable, the cost of FDIV must be
-  /// substantially higher than normal FP ops like FADD and FMUL.
-  bool UseReciprocalEst;
+  /// True if estimate instructions (rsqrtss, rsqrtps, rcpss, rcpps) may be
+  /// used in place of full-precision floating point math.
+  bool UseRsqrt;
+  bool UseVecRsqrt;
+  bool UseRecip;
+  bool UseVecRecip;
   
   /// Processor has AVX-512 PreFetch Instructions
   bool HasPFI;
 
   /// Processor has AVX-512 Exponential and Reciprocal Instructions
   bool HasERI;
 
   /// Processor has AVX-512 Conflict Detection Instructions
   bool HasCDI;
public:
   bool useLeaForSP() const { return UseLeaForSP; }
   bool hasSlowDivide32() const { return HasSlowDivide32; }
   bool hasSlowDivide64() const { return HasSlowDivide64; }
   bool padShortFunctions() const { return PadShortFunctions; }
   bool callRegIndirect() const { return CallRegIndirect; }
   bool LEAusesAG() const { return LEAUsesAG; }
   bool slowLEA() const { return SlowLEA; }
   bool slowIncDec() const { return SlowIncDec; }
-  bool useSqrtEst() const { return UseSqrtEst; }
-  bool useReciprocalEst() const { return UseReciprocalEst; }
+  bool useRsqrtEst() const { return UseRsqrt; }
+  bool useVecRsqrtEst() const { return UseVecRsqrt; }
+  bool useRecipEst() const { return UseRecip; }
+  bool useVecRecipEst() const { return UseVecRecip; }
   bool hasCDI() const { return HasCDI; }
   bool hasPFI() const { return HasPFI; }
   bool hasERI() const { return HasERI; }
   bool hasDQI() const { return HasDQI; }
   bool hasBWI() const { return HasBWI; }
   bool hasVLX() const { return HasVLX; }
 
   bool isAtom() const { return X86ProcFamily == IntelAtom; }

lib/Target/X86/X86Subtarget.cpp

void X86Subtarget::initializeEnvironment() {
   UseLeaForSP = false;
   HasSlowDivide32 = false;
   HasSlowDivide64 = false;
   PadShortFunctions = false;
   CallRegIndirect = false;
   LEAUsesAG = false;
   SlowLEA = false;
   SlowIncDec = false;
-  UseSqrtEst = false;
-  UseReciprocalEst = false;
+  UseRsqrt = false;
+  UseVecRsqrt = false;
+  UseRecip = false;
+  UseVecRecip = false;
   stackAlignment = 4;
   // FIXME: this is a known good value for Yonah. How about others?
   MaxInlineSizeThreshold = 128;
 }
 
 X86Subtarget &X86Subtarget::initializeSubtargetDependencies(StringRef CPU,
                                                             StringRef FS) {
   initializeEnvironment();

test/CodeGen/X86/recip-fastmath.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=sse2 | FileCheck %s
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,use-recip-est | FileCheck %s --check-prefix=RECIP
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,use-recip-est -x86-recip-refinement-steps=2 | FileCheck %s --check-prefix=REFINE
-
-; If the target's divss/divps instructions are substantially
-; slower than rcpss/rcpps with a Newton-Raphson refinement,
-; we should generate the estimate sequence.
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,recip-div | FileCheck %s --check-prefix=RECIP
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,recip-vec-div | FileCheck %s --check-prefix=VECRECIP
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,recip-div -x86-recip-refinement-steps=2 | FileCheck %s --check-prefix=REFINE
 
 ; See PR21385 ( http://llvm.org/bugs/show_bug.cgi?id=21385 )
 ; for details about the accuracy, speed, and implementation
 ; differences of x86 reciprocal estimates.
 
 define float @reciprocal_estimate(float %x) #0 {
-  %div = fdiv fast float 1.0, %x
-  ret float %div
-
 ; CHECK-LABEL: reciprocal_estimate:
-; CHECK: movss
-; CHECK-NEXT: divss
-; CHECK-NEXT: movaps
-; CHECK-NEXT: retq
-
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; CHECK-NEXT:    divss %xmm0, %xmm1
+; CHECK-NEXT:    movaps %xmm1, %xmm0
+; CHECK-NEXT:    retq
+;
 ; RECIP-LABEL: reciprocal_estimate:
-; RECIP: vrcpss
-; RECIP: vmulss
-; RECIP: vsubss
-; RECIP: vmulss
-; RECIP: vaddss
-; RECIP-NEXT: retq
-
+; RECIP:       # BB#0:
+; RECIP-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
+; RECIP-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; RECIP-NEXT:    vmovss {{.*#+}} xmm2 = mem[0],zero,zero,zero
+; RECIP-NEXT:    vsubss %xmm0, %xmm2, %xmm0
+; RECIP-NEXT:    vmulss %xmm0, %xmm1, %xmm0
+; RECIP-NEXT:    vaddss %xmm0, %xmm1, %xmm0
+; RECIP-NEXT:    retq
+;
+; VECRECIP-LABEL: reciprocal_estimate:
+; VECRECIP:       # BB#0:
+; VECRECIP-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; VECRECIP-NEXT:    vdivss %xmm0, %xmm1, %xmm0
+; VECRECIP-NEXT:    retq
+;
 ; REFINE-LABEL: reciprocal_estimate:
-; REFINE: vrcpss
-; REFINE: vmulss
-; REFINE: vsubss
-; REFINE: vmulss
-; REFINE: vaddss
-; REFINE: vmulss
-; REFINE: vsubss
-; REFINE: vmulss
-; REFINE: vaddss
-; REFINE-NEXT: retq
+; REFINE:       # BB#0:
+; REFINE-NEXT:    vrcpss %xmm0, %xmm0, %xmm1
+; REFINE-NEXT:    vmulss %xmm1, %xmm0, %xmm2
+; REFINE-NEXT:    vmovss {{.*#+}} xmm3 = mem[0],zero,zero,zero
+; REFINE-NEXT:    vsubss %xmm2, %xmm3, %xmm2
+; REFINE-NEXT:    vmulss %xmm2, %xmm1, %xmm2
+; REFINE-NEXT:    vaddss %xmm2, %xmm1, %xmm1
+; REFINE-NEXT:    vmulss %xmm1, %xmm0, %xmm0
+; REFINE-NEXT:    vsubss %xmm0, %xmm3, %xmm0
+; REFINE-NEXT:    vmulss %xmm0, %xmm1, %xmm0
+; REFINE-NEXT:    vaddss %xmm0, %xmm1, %xmm0
+; REFINE-NEXT:    retq
+  %div = fdiv fast float 1.0, %x
+  ret float %div
 }
 
 define <4 x float> @reciprocal_estimate_v4f32(<4 x float> %x) #0 {
-  %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %x
-  ret <4 x float> %div
-
 ; CHECK-LABEL: reciprocal_estimate_v4f32:
-; CHECK: movaps
-; CHECK-NEXT: divps
-; CHECK-NEXT: movaps
-; CHECK-NEXT: retq
-
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; CHECK-NEXT:    divps %xmm0, %xmm1
+; CHECK-NEXT:    movaps %xmm1, %xmm0
+; CHECK-NEXT:    retq
+;
 ; RECIP-LABEL: reciprocal_estimate_v4f32:
-; RECIP: vrcpps
-; RECIP: vmulps
-; RECIP: vsubps
-; RECIP: vmulps
-; RECIP: vaddps
-; RECIP-NEXT: retq
-
+; RECIP:       # BB#0:
+; RECIP-NEXT:    vmovaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; RECIP-NEXT:    vdivps %xmm0, %xmm1, %xmm0
+; RECIP-NEXT:    retq
+;
+; VECRECIP-LABEL: reciprocal_estimate_v4f32:
+; VECRECIP:       # BB#0:
+; VECRECIP-NEXT:    vrcpps %xmm0, %xmm1
+; VECRECIP-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; VECRECIP-NEXT:    vmovaps {{.*#+}} xmm2 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; VECRECIP-NEXT:    vsubps %xmm0, %xmm2, %xmm0
+; VECRECIP-NEXT:    vmulps %xmm0, %xmm1, %xmm0
+; VECRECIP-NEXT:    vaddps %xmm0, %xmm1, %xmm0
+; VECRECIP-NEXT:    retq
+;
 ; REFINE-LABEL: reciprocal_estimate_v4f32:
-; REFINE: vrcpps
-; REFINE: vmulps
-; REFINE: vsubps
-; REFINE: vmulps
-; REFINE: vaddps
-; REFINE: vmulps
-; REFINE: vsubps
-; REFINE: vmulps
-; REFINE: vaddps
-; REFINE-NEXT: retq
+; REFINE:       # BB#0:
+; REFINE-NEXT:    vmovaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; REFINE-NEXT:    vdivps %xmm0, %xmm1, %xmm0
+; REFINE-NEXT:    retq
+  %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %x
+  ret <4 x float> %div
 }
 
 define <8 x float> @reciprocal_estimate_v8f32(<8 x float> %x) #0 {
-  %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
-  ret <8 x float> %div
-
 ; CHECK-LABEL: reciprocal_estimate_v8f32:
-; CHECK: movaps
-; CHECK: movaps
-; CHECK-NEXT: divps
-; CHECK-NEXT: divps
-; CHECK-NEXT: movaps
-; CHECK-NEXT: movaps
-; CHECK-NEXT: retq
-
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movaps {{.*#+}} xmm2 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; CHECK-NEXT:    movaps %xmm2, %xmm3
+; CHECK-NEXT:    divps %xmm0, %xmm3
+; CHECK-NEXT:    divps %xmm1, %xmm2
+; CHECK-NEXT:    movaps %xmm3, %xmm0
+; CHECK-NEXT:    movaps %xmm2, %xmm1
+; CHECK-NEXT:    retq
+;
 ; RECIP-LABEL: reciprocal_estimate_v8f32:
-; RECIP: vrcpps
-; RECIP: vmulps
-; RECIP: vsubps
-; RECIP: vmulps
-; RECIP: vaddps
-; RECIP-NEXT: retq
-
+; RECIP:       # BB#0:
+; RECIP-NEXT:    vmovaps {{.*#+}} ymm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; RECIP-NEXT:    vdivps %ymm0, %ymm1, %ymm0
+; RECIP-NEXT:    retq
+;
+; VECRECIP-LABEL: reciprocal_estimate_v8f32:
+; VECRECIP:       # BB#0:
+; VECRECIP-NEXT:    vrcpps %ymm0, %ymm1
+; VECRECIP-NEXT:    vmulps %ymm1, %ymm0, %ymm0
+; VECRECIP-NEXT:    vmovaps {{.*#+}} ymm2 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; VECRECIP-NEXT:    vsubps %ymm0, %ymm2, %ymm0
+; VECRECIP-NEXT:    vmulps %ymm0, %ymm1, %ymm0
+; VECRECIP-NEXT:    vaddps %ymm0, %ymm1, %ymm0
+; VECRECIP-NEXT:    retq
+;
 ; REFINE-LABEL: reciprocal_estimate_v8f32:
-; REFINE: vrcpps
-; REFINE: vmulps
-; REFINE: vsubps
-; REFINE: vmulps
-; REFINE: vaddps
-; REFINE: vmulps
-; REFINE: vsubps
-; REFINE: vmulps
-; REFINE: vaddps
-; REFINE-NEXT: retq
+; REFINE:       # BB#0:
+; REFINE-NEXT:    vmovaps {{.*#+}} ymm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; REFINE-NEXT:    vdivps %ymm0, %ymm1, %ymm0
+; REFINE-NEXT:    retq
+  %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
+  ret <8 x float> %div
 }
 
 attributes #0 = { "unsafe-fp-math"="true" }

test/CodeGen/X86/sqrt-fastmath.ll

 ; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=sse2 | FileCheck %s
-; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,use-sqrt-est | FileCheck %s --check-prefix=ESTIMATE
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,recip-sqrt | FileCheck %s --check-prefix=ESTIMATE
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=avx,recip-vec-sqrt | FileCheck %s --check-prefix=VECEST
 
 declare double @__sqrt_finite(double) #0
 declare float @__sqrtf_finite(float) #0
 declare x86_fp80 @__sqrtl_finite(x86_fp80) #0
 declare float @llvm.sqrt.f32(float) #0
 declare <4 x float> @llvm.sqrt.v4f32(<4 x float>) #0
 declare <8 x float> @llvm.sqrt.v8f32(<8 x float>) #0
 
 
 define double @fd(double %d) #0 {
 ; CHECK-LABEL: fd:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    sqrtsd %xmm0, %xmm0
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: fd:
 ; ESTIMATE:       # BB#0:
 ; ESTIMATE-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: fd:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0
+; VECEST-NEXT:    retq
   %call = tail call double @__sqrt_finite(double %d) #1
   ret double %call
 }
 
 
 define float @ff(float %f) #0 {
 ; CHECK-LABEL: ff:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    sqrtss %xmm0, %xmm0
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: ff:
 ; ESTIMATE:       # BB#0:
 ; ESTIMATE-NEXT:    vrsqrtss %xmm0, %xmm0, %xmm1
 ; ESTIMATE-NEXT:    vmulss {{.*}}(%rip), %xmm1, %xmm2
 ; ESTIMATE-NEXT:    vmulss %xmm1, %xmm1, %xmm1
 ; ESTIMATE-NEXT:    vmulss %xmm0, %xmm1, %xmm1
 ; ESTIMATE-NEXT:    vaddss {{.*}}(%rip), %xmm1, %xmm1
 ; ESTIMATE-NEXT:    vmulss %xmm2, %xmm1, %xmm1
 ; ESTIMATE-NEXT:    vmulss %xmm1, %xmm0, %xmm1
 ; ESTIMATE-NEXT:    vxorps %xmm2, %xmm2, %xmm2
 ; ESTIMATE-NEXT:    vcmpeqss %xmm2, %xmm0, %xmm0
 ; ESTIMATE-NEXT:    vandnps %xmm1, %xmm0, %xmm0
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: ff:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0
+; VECEST-NEXT:    retq
   %call = tail call float @__sqrtf_finite(float %f) #1
   ret float %call
 }
 
 
 define x86_fp80 @fld(x86_fp80 %ld) #0 {
 ; CHECK-LABEL: fld:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; CHECK-NEXT:    fsqrt
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: fld:
 ; ESTIMATE:       # BB#0:
 ; ESTIMATE-NEXT:    fldt {{[0-9]+}}(%rsp)
 ; ESTIMATE-NEXT:    fsqrt
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: fld:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    fldt {{[0-9]+}}(%rsp)
+; VECEST-NEXT:    fsqrt
+; VECEST-NEXT:    retq
   %call = tail call x86_fp80 @__sqrtl_finite(x86_fp80 %ld) #1
   ret x86_fp80 %call
 }
 
 
 
 define float @reciprocal_square_root(float %x) #0 {
 ; CHECK-LABEL: reciprocal_square_root:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    sqrtss %xmm0, %xmm1
 ; CHECK-NEXT:    movss {{.*#+}} xmm0 = mem[0],zero,zero,zero
 ; CHECK-NEXT:    divss %xmm1, %xmm0
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: reciprocal_square_root:
 ; ESTIMATE:       # BB#0:
 ; ESTIMATE-NEXT:    vrsqrtss %xmm0, %xmm0, %xmm1
 ; ESTIMATE-NEXT:    vmulss {{.*}}(%rip), %xmm1, %xmm2
 ; ESTIMATE-NEXT:    vmulss %xmm1, %xmm1, %xmm1
 ; ESTIMATE-NEXT:    vmulss %xmm0, %xmm1, %xmm0
 ; ESTIMATE-NEXT:    vaddss {{.*}}(%rip), %xmm0, %xmm0
 ; ESTIMATE-NEXT:    vmulss %xmm2, %xmm0, %xmm0
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: reciprocal_square_root:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0
+; VECEST-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; VECEST-NEXT:    vdivss %xmm0, %xmm1, %xmm0
+; VECEST-NEXT:    retq
   %sqrt = tail call float @llvm.sqrt.f32(float %x)
   %div = fdiv fast float 1.0, %sqrt
   ret float %div
 }
 
 define <4 x float> @reciprocal_square_root_v4f32(<4 x float> %x) #0 {
 ; CHECK-LABEL: reciprocal_square_root_v4f32:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    sqrtps %xmm0, %xmm1
 ; CHECK-NEXT:    movaps {{.*#+}} xmm0 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
 ; CHECK-NEXT:    divps %xmm1, %xmm0
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: reciprocal_square_root_v4f32:
 ; ESTIMATE:       # BB#0:
-; ESTIMATE-NEXT:    vrsqrtps %xmm0, %xmm1
-; ESTIMATE-NEXT:    vmulps %xmm1, %xmm1, %xmm2
-; ESTIMATE-NEXT:    vmulps %xmm0, %xmm2, %xmm0
-; ESTIMATE-NEXT:    vaddps {{.*}}(%rip), %xmm0, %xmm0
-; ESTIMATE-NEXT:    vmulps {{.*}}(%rip), %xmm1, %xmm1
-; ESTIMATE-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; ESTIMATE-NEXT:    vsqrtps %xmm0, %xmm0
+; ESTIMATE-NEXT:    vmovaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; ESTIMATE-NEXT:    vdivps %xmm0, %xmm1, %xmm0
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: reciprocal_square_root_v4f32:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    vrsqrtps %xmm0, %xmm1
+; VECEST-NEXT:    vmulps %xmm1, %xmm1, %xmm2
+; VECEST-NEXT:    vmulps %xmm0, %xmm2, %xmm0
+; VECEST-NEXT:    vaddps {{.*}}(%rip), %xmm0, %xmm0
+; VECEST-NEXT:    vmulps {{.*}}(%rip), %xmm1, %xmm1
+; VECEST-NEXT:    vmulps %xmm1, %xmm0, %xmm0
+; VECEST-NEXT:    retq
   %sqrt = tail call <4 x float> @llvm.sqrt.v4f32(<4 x float> %x)
   %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %sqrt
   ret <4 x float> %div
 }
 
 define <8 x float> @reciprocal_square_root_v8f32(<8 x float> %x) #0 {
 ; CHECK-LABEL: reciprocal_square_root_v8f32:
 ; CHECK:       # BB#0:
 ; CHECK-NEXT:    sqrtps %xmm1, %xmm2
 ; CHECK-NEXT:    sqrtps %xmm0, %xmm3
 ; CHECK-NEXT:    movaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
 ; CHECK-NEXT:    movaps %xmm1, %xmm0
 ; CHECK-NEXT:    divps %xmm3, %xmm0
 ; CHECK-NEXT:    divps %xmm2, %xmm1
 ; CHECK-NEXT:    retq
 ;
 ; ESTIMATE-LABEL: reciprocal_square_root_v8f32:
 ; ESTIMATE:       # BB#0:
-; ESTIMATE-NEXT:    vrsqrtps %ymm0, %ymm1
-; ESTIMATE-NEXT:    vmulps %ymm1, %ymm1, %ymm2
-; ESTIMATE-NEXT:    vmulps %ymm0, %ymm2, %ymm0
-; ESTIMATE-NEXT:    vaddps {{.*}}(%rip), %ymm0, %ymm0
-; ESTIMATE-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
-; ESTIMATE-NEXT:    vmulps %ymm1, %ymm0, %ymm0
+; ESTIMATE-NEXT:    vsqrtps %ymm0, %ymm0
+; ESTIMATE-NEXT:    vmovaps {{.*#+}} ymm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00]
+; ESTIMATE-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; ESTIMATE-NEXT:    retq
+;
+; VECEST-LABEL: reciprocal_square_root_v8f32:
+; VECEST:       # BB#0:
+; VECEST-NEXT:    vrsqrtps %ymm0, %ymm1
+; VECEST-NEXT:    vmulps %ymm1, %ymm1, %ymm2
+; VECEST-NEXT:    vmulps %ymm0, %ymm2, %ymm0
+; VECEST-NEXT:    vaddps {{.*}}(%rip), %ymm0, %ymm0
+; VECEST-NEXT:    vmulps {{.*}}(%rip), %ymm1, %ymm1
+; VECEST-NEXT:    vmulps %ymm1, %ymm0, %ymm0
+; VECEST-NEXT:    retq
   %sqrt = tail call <8 x float> @llvm.sqrt.v8f32(<8 x float> %x)
   %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %sqrt
   ret <8 x float> %div
 }
 
 
 attributes #0 = { "unsafe-fp-math"="true" }
 attributes #1 = { nounwind readnone }