[X86][SSE] Improve DIV/SQRT throughput estimates for SB/HW schedule models

[RKSimon]

Summary

The current DIV/SQRT throughput estimates for SB/HW schedule models use the default 1cy value, which is highly unrealistic.

I've updated the values with estimates based on the latencies which is typically about right for DIV/SQRT units, its also in the ballpark of what Agner suggests - if anyone has even more accurate values that would be great but these alone should be a major improvement to scheduling.

[avt77]

What are your plans here? I've just checked (with help of "-print-schedule=true") IMUL and LEA for Jaguar: they are completely wrong if we compare with numbers from http://www.agner.org/optimize/instruction_tables.pdf. Are we going to change all these things step-by-step?

[RKSimon]

In D32219#733371, @avt77 wrote:

What are your plans here? I've just checked (with help of "-print-schedule=true") IMUL and LEA for Jaguar: they are completely wrong if we compare with numbers from http://www.agner.org/optimize/instruction_tables.pdf. Are we going to change all these things step-by-step?

The basic process will be: add thorough tests, identify issues, fix issues (either direct commit or reviewed patch if it warrants discussion). I'm intending to initially focus on the SSE/AVX instructions so if you want to add scheduler tests for the mul/imul/lea/etc. instructions then I say go for it.

[gadi.haber]

lib/Target/X86/X86SchedHaswell.td

139

let NumMicroOps = 1;

143

let NumMicroOps = 2;

148

let NumMicroOps = 1;

152

let NumMicroOps = 2;

lib/Target/X86/X86SchedSandyBridge.td

126

let NumMicroOps = 1;

130

let NumMicroOps = 2;

135

let NumMicroOps = 1;

139

let NumMicroOps = 2;

[RKSimon]

Add NumMicroOps and regenerate (adds 256-bit vector cases which were added recently).

[gadi.haber]

lib/Target/X86/X86SchedHaswell.td

140

instruction latency of X87 FDIV in Haswell is actually higher and takes 20 cycles

141

I believe ResourceCycles here should be 1.

145

latency of FDIVLd in Haswell is 24

146

ResourceCycles for FDIVLd is [1, 1]

151

latency of FSqrt in Haswell is 23

156

I don't have the exact latency for Haswell but is larger than 23

157

ResourceCycles is [1, 1]

1922

HWPort15 should actually be changed to HWPort015 in Haswell

1925

ResourceCycles should be [2, 1]

ResourceCycles lists the number of times where HW port was used in the instruction.
In this case HWPort0 is used twice (by uOp1 and uOp2) and HWPort015 is used only once (by uOp3)

[RKSimon]

lib/Target/X86/X86SchedHaswell.td

140

Despite its name this scheduling class is also used by the SSE/AVX float double division (just the xmm variants here as the ymm are overridden). Given that we barely use x87 these days aren't we better off using the value just for SSE/AVX?

145

Then why is load latency in HWWriteResPair just 4 cycles?

151

Please can you cite the source of these numbers? I've been careful not to change the current latency values (as shown in the diffs in the tests below) and am just trying to add more realistic throughput values.

1925

I don't think I agree. ResourceCycles is an analogue for throughput here - the number of cycles that the op consumes this resource for in that stage. It should be 12 (ish) cycles to indicate that HWPort0 won't accept instructions for 12 cycles while it completes the division.

[RKSimon]

D33897 is moving SNB/HW scheduler table to auto-gen

lib/Target/X86/X86SchedHaswell.td

 def : WriteRes<WriteIDivLd, [HWPort23, HWPort0, HWDivider]> {
   let Latency = 29;
   let ResourceCycles = [1, 1, 10];
 }
 
 // Scalar and vector floating point.
 defm : HWWriteResPair<WriteFAdd,   HWPort1, 3>;
 defm : HWWriteResPair<WriteFMul,   HWPort0, 5>;
-defm : HWWriteResPair<WriteFDiv,   HWPort0, 12>; // 10-14 cycles.
 defm : HWWriteResPair<WriteFRcp,   HWPort0, 5>;
 defm : HWWriteResPair<WriteFRsqrt, HWPort0, 5>;
-defm : HWWriteResPair<WriteFSqrt,  HWPort0, 15>;
 defm : HWWriteResPair<WriteCvtF2I, HWPort1, 3>;
 defm : HWWriteResPair<WriteCvtI2F, HWPort1, 4>;
 defm : HWWriteResPair<WriteCvtF2F, HWPort1, 3>;
 defm : HWWriteResPair<WriteFShuffle,  HWPort5,  1>;
 defm : HWWriteResPair<WriteFBlend,  HWPort015,  1>;
 defm : HWWriteResPair<WriteFShuffle256,  HWPort5,  3>;
 
+def : WriteRes<WriteFDiv, [HWPort0]> {
+  let Latency = 12; // 10-14 cycles.
+  let ResourceCycles = [12];
+}
+def : WriteRes<WriteFDivLd, [HWPort23, HWPort0]> {
+  let Latency = 16; // load + 10-14 cycles.
+  let ResourceCycles = [1, 12];
+}
+
+def : WriteRes<WriteFSqrt, [HWPort0]> {
+  let Latency = 15;
+  let ResourceCycles = [15];
+}
+def : WriteRes<WriteFSqrtLd, [HWPort23, HWPort0]> {
+  let Latency = 19;
+  let ResourceCycles = [1, 15];
+}
+
 def : WriteRes<WriteFVarBlend, [HWPort5]> {
   let Latency = 2;
   let ResourceCycles = [2];
 }
 def : WriteRes<WriteFVarBlendLd, [HWPort5, HWPort23]> {
   let Latency = 6;
   let ResourceCycles = [2, 1];
 }
 
def WriteMULm : SchedWriteRes<[HWPort01, HWPort23]> {
   let Latency = 9;
   let NumMicroOps = 2;
   let ResourceCycles = [1, 1];
 }
 def : InstRW<[WriteMULm], (instregex "(V?)MUL(P|S)(S|D)rm")>;
 
 // VDIVPS.
 // y,y,y.
 def WriteVDIVPSYrr : SchedWriteRes<[HWPort0, HWPort15]> {
   let Latency = 19; // 18-21 cycles.
   let NumMicroOps = 3;
-  let ResourceCycles = [2, 1];
+  let ResourceCycles = [2, 19];
 }
 def : InstRW<[WriteVDIVPSYrr], (instregex "VDIVPSYrr")>;
 
 // y,y,m256.
 def WriteVDIVPSYrm : SchedWriteRes<[HWPort0, HWPort15, HWPort23]> {
   let Latency = 23; // 18-21 + 4 cycles.
   let NumMicroOps = 4;
-  let ResourceCycles = [2, 1, 1];
+  let ResourceCycles = [2, 1, 19];
 }
 def : InstRW<[WriteVDIVPSYrm, ReadAfterLd], (instregex "VDIVPSYrm")>;
 
 // VDIVPD.
 // y,y,y.
 def WriteVDIVPDYrr : SchedWriteRes<[HWPort0, HWPort15]> {
   let Latency = 27; // 19-35 cycles.
   let NumMicroOps = 3;
-  let ResourceCycles = [2, 1];
+  let ResourceCycles = [2, 27];
 }
 def : InstRW<[WriteVDIVPDYrr], (instregex "VDIVPDYrr")>;
 
 // y,y,m256.
 def WriteVDIVPDYrm : SchedWriteRes<[HWPort0, HWPort15, HWPort23]> {
   let Latency = 31; // 19-35 + 4 cycles.
   let NumMicroOps = 4;
-  let ResourceCycles = [2, 1, 1];
+  let ResourceCycles = [2, 1, 27];
 }
 def : InstRW<[WriteVDIVPDYrm, ReadAfterLd], (instregex "VDIVPDYrm")>;
 
 // VRCPPS.
 // y,y.
 def WriteVRCPPSr : SchedWriteRes<[HWPort0, HWPort15]> {
   let Latency = 7;
   let NumMicroOps = 3;

 
 //-- Math instructions --//
 
 // VSQRTPS.
 // y,y.
 def WriteVSQRTPSYr : SchedWriteRes<[HWPort0, HWPort15]> {
   let Latency = 19;
   let NumMicroOps = 3;
-  let ResourceCycles = [2, 1];
+  let ResourceCycles = [2, 19];
 }
 def : InstRW<[WriteVSQRTPSYr], (instregex "VSQRTPSYr")>;
 
 // y,m256.
 def WriteVSQRTPSYm : SchedWriteRes<[HWPort0, HWPort15, HWPort23]> {
   let Latency = 23;
   let NumMicroOps = 4;
-  let ResourceCycles = [2, 1, 1];
+  let ResourceCycles = [2, 1, 19];
 }
 def : InstRW<[WriteVSQRTPSYm], (instregex "VSQRTPSYm")>;
 
 // VSQRTPD.
 // y,y.
 def WriteVSQRTPDYr : SchedWriteRes<[HWPort0, HWPort15]> {
   let Latency = 28;
   let NumMicroOps = 3;
-  let ResourceCycles = [2, 1];
+  let ResourceCycles = [2, 28];
 }
 def : InstRW<[WriteVSQRTPDYr], (instregex "VSQRTPDYr")>;
 
 // y,m256.
 def WriteVSQRTPDYm : SchedWriteRes<[HWPort0, HWPort15, HWPort23]> {
   let Latency = 32;
   let NumMicroOps = 4;
-  let ResourceCycles = [2, 1, 1];
+  let ResourceCycles = [2, 1, 32];
 }
 def : InstRW<[WriteVSQRTPDYm], (instregex "VSQRTPDYm")>;
 
 // RSQRT SS/PS.
 // x,x.
 def WriteRSQRTr : SchedWriteRes<[HWPort0]> {
   let Latency = 5;
 }

lib/Target/X86/X86SchedSandyBridge.td

 def : WriteRes<WriteIDivLd, [SBPort23, SBPort0, SBDivider]> {
   let Latency = 29;
   let ResourceCycles = [1, 1, 10];
 }
 
 // Scalar and vector floating point.
 defm : SBWriteResPair<WriteFAdd,   SBPort1, 3>;
 defm : SBWriteResPair<WriteFMul,   SBPort0, 5>;
-defm : SBWriteResPair<WriteFDiv,   SBPort0, 12>; // 10-14 cycles.
 defm : SBWriteResPair<WriteFRcp,   SBPort0, 5>;
 defm : SBWriteResPair<WriteFRsqrt, SBPort0, 5>;
-defm : SBWriteResPair<WriteFSqrt,  SBPort0, 15>;
 defm : SBWriteResPair<WriteCvtF2I, SBPort1, 3>;
 defm : SBWriteResPair<WriteCvtI2F, SBPort1, 4>;
 defm : SBWriteResPair<WriteCvtF2F, SBPort1, 3>;
 defm : SBWriteResPair<WriteFShuffle,  SBPort5,  1>;
 defm : SBWriteResPair<WriteFBlend,  SBPort05,  1>;
+
+def : WriteRes<WriteFDiv, [SBPort0]> {
+  let Latency = 12; // 10-14 cycles.
+  let ResourceCycles = [12];
+}
+def : WriteRes<WriteFDivLd, [SBPort23, SBPort0]> {
+  let Latency = 16; // load + 10-14 cycles.
+  let ResourceCycles = [1, 12];
+}
+
+def : WriteRes<WriteFSqrt, [SBPort0]> {
+  let Latency = 15;
+  let ResourceCycles = [15];
+}
+def : WriteRes<WriteFSqrtLd, [SBPort23, SBPort0]> {
+  let Latency = 19;
+  let ResourceCycles = [1, 15];
+}
+
 def : WriteRes<WriteFVarBlend, [SBPort0, SBPort5]> {
   let Latency = 2;
   let ResourceCycles = [1, 1];
 }
 def : WriteRes<WriteFVarBlendLd, [SBPort0, SBPort5, SBPort23]> {
   let Latency = 6;
   let ResourceCycles = [1, 1, 1];
 }

test/CodeGen/X86/recip-fastmath.ll

 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero sched: [5:1.00]
 ; BTVER2-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: f32_no_estimate:
 ; SANDY:       # BB#0:
 ; SANDY-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero sched: [4:0.50]
-; SANDY-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; SANDY-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: f32_no_estimate:
 ; HASWELL:       # BB#0:
 ; HASWELL-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero sched: [4:0.50]
-; HASWELL-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; HASWELL-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; HASWELL-NO-FMA-LABEL: f32_no_estimate:
 ; HASWELL-NO-FMA:       # BB#0:
 ; HASWELL-NO-FMA-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero
 ; HASWELL-NO-FMA-NEXT:    vdivss %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: f32_no_estimate:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vmovss {{.*#+}} xmm1 = mem[0],zero,zero,zero sched: [4:0.50]
-; AVX512-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; AVX512-NEXT:    vdivss %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; AVX512-NEXT:    retq # sched: [1:1.00]
   %div = fdiv fast float 1.0, %x
   ret float %div
 }
 
 define float @f32_one_step(float %x) #1 {
 ; SSE-LABEL: f32_one_step:
 ; SSE:       # BB#0:

 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00] sched: [5:1.00]
 ; BTVER2-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: v4f32_no_estimate:
 ; SANDY:       # BB#0:
 ; SANDY-NEXT:    vmovaps {{.*#+}} xmm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00] sched: [4:0.50]
-; SANDY-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; SANDY-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: v4f32_no_estimate:
 ; HASWELL:       # BB#0:
 ; HASWELL-NEXT:    vbroadcastss {{.*}}(%rip), %xmm1 # sched: [4:0.50]
-; HASWELL-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; HASWELL-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; HASWELL-NO-FMA-LABEL: v4f32_no_estimate:
 ; HASWELL-NO-FMA:       # BB#0:
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*}}(%rip), %xmm1
 ; HASWELL-NO-FMA-NEXT:    vdivps %xmm0, %xmm1, %xmm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v4f32_no_estimate:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vbroadcastss {{.*}}(%rip), %xmm1 # sched: [4:0.50]
-; AVX512-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:1.00]
+; AVX512-NEXT:    vdivps %xmm0, %xmm1, %xmm0 # sched: [12:12.00]
 ; AVX512-NEXT:    retq # sched: [1:1.00]
   %div = fdiv fast <4 x float> <float 1.0, float 1.0, float 1.0, float 1.0>, %x
   ret <4 x float> %div
 }
 
 define <4 x float> @v4f32_one_step(<4 x float> %x) #1 {
 ; SSE-LABEL: v4f32_one_step:
 ; SSE:       # BB#0:

 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vmovaps {{.*#+}} ymm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00] sched: [5:1.00]
 ; BTVER2-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: v8f32_no_estimate:
 ; SANDY:       # BB#0:
 ; SANDY-NEXT:    vmovaps {{.*#+}} ymm1 = [1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00,1.000000e+00] sched: [4:0.50]
-; SANDY-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [12:1.00]
+; SANDY-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [12:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: v8f32_no_estimate:
 ; HASWELL:       # BB#0:
 ; HASWELL-NEXT:    vbroadcastss {{.*}}(%rip), %ymm1 # sched: [5:1.00]
-; HASWELL-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [19:2.00]
+; HASWELL-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [19:9.50]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; HASWELL-NO-FMA-LABEL: v8f32_no_estimate:
 ; HASWELL-NO-FMA:       # BB#0:
 ; HASWELL-NO-FMA-NEXT:    vbroadcastss {{.*}}(%rip), %ymm1
 ; HASWELL-NO-FMA-NEXT:    vdivps %ymm0, %ymm1, %ymm0
 ; HASWELL-NO-FMA-NEXT:    retq
 ;
 ; AVX512-LABEL: v8f32_no_estimate:
 ; AVX512:       # BB#0:
 ; AVX512-NEXT:    vbroadcastss {{.*}}(%rip), %ymm1 # sched: [5:1.00]
-; AVX512-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [19:2.00]
+; AVX512-NEXT:    vdivps %ymm0, %ymm1, %ymm0 # sched: [19:9.50]
 ; AVX512-NEXT:    retq # sched: [1:1.00]
   %div = fdiv fast <8 x float> <float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0, float 1.0>, %x
   ret <8 x float> %div
 }
 
 define <8 x float> @v8f32_one_step(<8 x float> %x) #1 {
 ; SSE-LABEL: v8f32_one_step:
 ; SSE:       # BB#0:

test/CodeGen/X86/sse-schedule.ll

 ; SLM-LABEL: test_divps:
 ; SLM:       # BB#0:
 ; SLM-NEXT:    divps %xmm1, %xmm0 # sched: [34:34.00]
 ; SLM-NEXT:    divps (%rdi), %xmm0 # sched: [37:34.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_divps:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vdivps %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; SANDY-NEXT:    vdivps (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; SANDY-NEXT:    vdivps %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; SANDY-NEXT:    vdivps (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_divps:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vdivps %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; HASWELL-NEXT:    vdivps (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; HASWELL-NEXT:    vdivps %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; HASWELL-NEXT:    vdivps (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_divps:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vdivps %xmm1, %xmm0, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    vdivps (%rdi), %xmm0, %xmm0 # sched: [24:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
   %1 = fdiv <4 x float> %a0, %a1

 ; SLM-LABEL: test_divss:
 ; SLM:       # BB#0:
 ; SLM-NEXT:    divss %xmm1, %xmm0 # sched: [34:34.00]
 ; SLM-NEXT:    divss (%rdi), %xmm0 # sched: [37:34.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_divss:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vdivss %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; SANDY-NEXT:    vdivss (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; SANDY-NEXT:    vdivss %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; SANDY-NEXT:    vdivss (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_divss:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vdivss %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; HASWELL-NEXT:    vdivss (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; HASWELL-NEXT:    vdivss %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; HASWELL-NEXT:    vdivss (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_divss:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vdivss %xmm1, %xmm0, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    vdivss (%rdi), %xmm0, %xmm0 # sched: [24:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
   %1 = fdiv float %a0, %a1

 ; SLM-NEXT:    sqrtps (%rdi), %xmm1 # sched: [18:1.00]
 ; SLM-NEXT:    sqrtps %xmm0, %xmm0 # sched: [15:1.00]
 ; SLM-NEXT:    addps %xmm0, %xmm1 # sched: [3:1.00]
 ; SLM-NEXT:    movaps %xmm1, %xmm0 # sched: [1:1.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_sqrtps:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vsqrtps %xmm0, %xmm0 # sched: [15:1.00]
-; SANDY-NEXT:    vsqrtps (%rdi), %xmm1 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtps %xmm0, %xmm0 # sched: [15:15.00]
+; SANDY-NEXT:    vsqrtps (%rdi), %xmm1 # sched: [19:15.00]
 ; SANDY-NEXT:    vaddps %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_sqrtps:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vsqrtps %xmm0, %xmm0 # sched: [15:1.00]
-; HASWELL-NEXT:    vsqrtps (%rdi), %xmm1 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtps %xmm0, %xmm0 # sched: [15:15.00]
+; HASWELL-NEXT:    vsqrtps (%rdi), %xmm1 # sched: [19:15.00]
 ; HASWELL-NEXT:    vaddps %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_sqrtps:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vsqrtps (%rdi), %xmm1 # sched: [26:21.00]
 ; BTVER2-NEXT:    vsqrtps %xmm0, %xmm0 # sched: [21:21.00]
 ; BTVER2-NEXT:    vaddps %xmm1, %xmm0, %xmm0 # sched: [3:1.00]

 ; SLM-NEXT:    movaps (%rdi), %xmm1 # sched: [3:1.00]
 ; SLM-NEXT:    sqrtss %xmm0, %xmm0 # sched: [18:1.00]
 ; SLM-NEXT:    sqrtss %xmm1, %xmm1 # sched: [18:1.00]
 ; SLM-NEXT:    addps %xmm1, %xmm0 # sched: [3:1.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_sqrtss:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0 # sched: [19:15.00]
 ; SANDY-NEXT:    vmovaps (%rdi), %xmm1 # sched: [4:0.50]
-; SANDY-NEXT:    vsqrtss %xmm1, %xmm1, %xmm1 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtss %xmm1, %xmm1, %xmm1 # sched: [19:15.00]
 ; SANDY-NEXT:    vaddps %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_sqrtss:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0 # sched: [19:15.00]
 ; HASWELL-NEXT:    vmovaps (%rdi), %xmm1 # sched: [4:0.50]
-; HASWELL-NEXT:    vsqrtss %xmm1, %xmm1, %xmm1 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtss %xmm1, %xmm1, %xmm1 # sched: [19:15.00]
 ; HASWELL-NEXT:    vaddps %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_sqrtss:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vmovaps (%rdi), %xmm1 # sched: [5:1.00]
 ; BTVER2-NEXT:    vsqrtss %xmm0, %xmm0, %xmm0 # sched: [26:21.00]
 ; BTVER2-NEXT:    vsqrtss %xmm1, %xmm1, %xmm1 # sched: [26:21.00]

test/CodeGen/X86/sse2-schedule.ll

 ; SLM-LABEL: test_divpd:
 ; SLM:       # BB#0:
 ; SLM-NEXT:    divpd %xmm1, %xmm0 # sched: [34:34.00]
 ; SLM-NEXT:    divpd (%rdi), %xmm0 # sched: [37:34.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_divpd:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vdivpd %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; SANDY-NEXT:    vdivpd (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; SANDY-NEXT:    vdivpd %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; SANDY-NEXT:    vdivpd (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_divpd:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vdivpd %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; HASWELL-NEXT:    vdivpd (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; HASWELL-NEXT:    vdivpd %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; HASWELL-NEXT:    vdivpd (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_divpd:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vdivpd %xmm1, %xmm0, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    vdivpd (%rdi), %xmm0, %xmm0 # sched: [24:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
   %1 = fdiv <2 x double> %a0, %a1

 ; SLM-LABEL: test_divsd:
 ; SLM:       # BB#0:
 ; SLM-NEXT:    divsd %xmm1, %xmm0 # sched: [34:34.00]
 ; SLM-NEXT:    divsd (%rdi), %xmm0 # sched: [37:34.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_divsd:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vdivsd %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; SANDY-NEXT:    vdivsd (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; SANDY-NEXT:    vdivsd %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; SANDY-NEXT:    vdivsd (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_divsd:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vdivsd %xmm1, %xmm0, %xmm0 # sched: [12:1.00]
-; HASWELL-NEXT:    vdivsd (%rdi), %xmm0, %xmm0 # sched: [16:1.00]
+; HASWELL-NEXT:    vdivsd %xmm1, %xmm0, %xmm0 # sched: [12:12.00]
+; HASWELL-NEXT:    vdivsd (%rdi), %xmm0, %xmm0 # sched: [16:12.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_divsd:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vdivsd %xmm1, %xmm0, %xmm0 # sched: [19:19.00]
 ; BTVER2-NEXT:    vdivsd (%rdi), %xmm0, %xmm0 # sched: [24:19.00]
 ; BTVER2-NEXT:    retq # sched: [4:1.00]
   %1 = fdiv double %a0, %a1

 ; SLM-NEXT:    sqrtpd (%rdi), %xmm1 # sched: [18:1.00]
 ; SLM-NEXT:    sqrtpd %xmm0, %xmm0 # sched: [15:1.00]
 ; SLM-NEXT:    addpd %xmm0, %xmm1 # sched: [3:1.00]
 ; SLM-NEXT:    movapd %xmm1, %xmm0 # sched: [1:1.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_sqrtpd:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vsqrtpd %xmm0, %xmm0 # sched: [15:1.00]
-; SANDY-NEXT:    vsqrtpd (%rdi), %xmm1 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtpd %xmm0, %xmm0 # sched: [15:15.00]
+; SANDY-NEXT:    vsqrtpd (%rdi), %xmm1 # sched: [19:15.00]
 ; SANDY-NEXT:    vaddpd %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_sqrtpd:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vsqrtpd %xmm0, %xmm0 # sched: [15:1.00]
-; HASWELL-NEXT:    vsqrtpd (%rdi), %xmm1 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtpd %xmm0, %xmm0 # sched: [15:15.00]
+; HASWELL-NEXT:    vsqrtpd (%rdi), %xmm1 # sched: [19:15.00]
 ; HASWELL-NEXT:    vaddpd %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_sqrtpd:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vsqrtpd (%rdi), %xmm1 # sched: [26:21.00]
 ; BTVER2-NEXT:    vsqrtpd %xmm0, %xmm0 # sched: [21:21.00]
 ; BTVER2-NEXT:    vaddpd %xmm1, %xmm0, %xmm0 # sched: [3:1.00]

 ; SLM-NEXT:    movapd (%rdi), %xmm1 # sched: [3:1.00]
 ; SLM-NEXT:    sqrtsd %xmm0, %xmm0 # sched: [18:1.00]
 ; SLM-NEXT:    sqrtsd %xmm1, %xmm1 # sched: [18:1.00]
 ; SLM-NEXT:    addpd %xmm1, %xmm0 # sched: [3:1.00]
 ; SLM-NEXT:    retq # sched: [4:1.00]
 ;
 ; SANDY-LABEL: test_sqrtsd:
 ; SANDY:       # BB#0:
-; SANDY-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0 # sched: [19:15.00]
 ; SANDY-NEXT:    vmovapd (%rdi), %xmm1 # sched: [4:0.50]
-; SANDY-NEXT:    vsqrtsd %xmm1, %xmm1, %xmm1 # sched: [19:1.00]
+; SANDY-NEXT:    vsqrtsd %xmm1, %xmm1, %xmm1 # sched: [19:15.00]
 ; SANDY-NEXT:    vaddpd %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; SANDY-NEXT:    retq # sched: [5:1.00]
 ;
 ; HASWELL-LABEL: test_sqrtsd:
 ; HASWELL:       # BB#0:
-; HASWELL-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0 # sched: [19:15.00]
 ; HASWELL-NEXT:    vmovapd (%rdi), %xmm1 # sched: [4:0.50]
-; HASWELL-NEXT:    vsqrtsd %xmm1, %xmm1, %xmm1 # sched: [19:1.00]
+; HASWELL-NEXT:    vsqrtsd %xmm1, %xmm1, %xmm1 # sched: [19:15.00]
 ; HASWELL-NEXT:    vaddpd %xmm1, %xmm0, %xmm0 # sched: [3:1.00]
 ; HASWELL-NEXT:    retq # sched: [1:1.00]
 ;
 ; BTVER2-LABEL: test_sqrtsd:
 ; BTVER2:       # BB#0:
 ; BTVER2-NEXT:    vmovapd (%rdi), %xmm1 # sched: [5:1.00]
 ; BTVER2-NEXT:    vsqrtsd %xmm0, %xmm0, %xmm0 # sched: [26:21.00]
 ; BTVER2-NEXT:    vsqrtsd %xmm1, %xmm1, %xmm1 # sched: [26:21.00]