Improved cost model for FDIV and FSQRT

[avt77]

Summary

There is a bug describing poor cost model for floating point operations: Bug 29083 - [X86][SSE] Improve costs for floating point operations. This patch is the second one in series of patches dealing with cost model.

[avt77]

I updated cost numbers corresponding to Simon requirements

[avt77]

Cost model numbers related to Pentium and Nehalem were updated.

[RKSimon]

lib/Target/X86/X86TargetTransformInfo.cpp

273

AVXCustomCostTable was recently added - you can probably merge these into that table and avoid the extra lookup.

406

Put these above the SDIV/UDIV costs so it doesn't look like they are under the same 'avoid vectorize division' comment, which is purely an integer division thing.

459

if (ST->hasSSE())

1072

Move SSE1FloatCostTable below SSE2CostTbl to keep to 'descending subtarget' convention. You can probably rename it SSE1CostTble as well.

1073

Worth adding a SSE41CostTbl for Core2 era costs?

1160

if (ST->hasSSE())

[mkuper]

lib/Target/X86/X86TargetTransformInfo.cpp

269

A YMM fdiv being 3 times as expensive as a XMM fdiv seems slightly odd.

I'd expect "2, maybe a bit more", and Agner seems to agree - e.g. for Sandybridge he gives a range of 10-14 cycles for XMM DIVPS, and 20-28 cycles for YMM VDIVPS. Were your IACA checks accounting for additional instructions? Or is this an inconsistency between IACA and Anger's tests?

(Note that these numbers are supposed to represent reciprocal throughput, but Agner's data for latency also has factor of ~2)

[avt77]

lib/Target/X86/X86TargetTransformInfo.cpp

269

I use the following numbers:
atischenko@ip-172-31-21-62:~/iaca-lin64/bin$ ./test-arith-fdiv.sh
*********SandyBridge*********
Intel(R) Architecture Code Analyzer Version - 2.1
Analyzed File - arith-fdiv.o
Binary Format - 64Bit
Architecture - SNB
Analysis Type - Throughput

Throughput Analysis Report

Block Throughput: 14.00 Cycles Throughput Bottleneck: Divider

Port Binding In Cycles Per Iteration:

| Port | 0 - DV | 1 | 2 - D | 3 - D | 4 | 5 |

| Cycles | 1.0 14.0 | 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 | 0.0 |

N - port number or number of cycles resource conflict caused delay, DV - Divider pipe (on port 0)
D - Data fetch pipe (on ports 2 and 3), CP - on a critical path
F - Macro Fusion with the previous instruction occurred

• - instruction micro-ops not bound to a port

^ - Micro Fusion happened

- ESP Tracking sync uop was issued

@ - SSE instruction followed an AVX256 instruction, dozens of cycles penalty is expected
! - instruction not supported, was not accounted in Analysis

Num Of	Ports pressure in cycles
Uops	0 - DV	1	2 - D	3 - D	4	5

---

1	1.0 14.0						CP	vdivps xmm0, xmm0, xmm0

Total Num Of Uops: 1

atischenko@ip-172-31-21-62:~/iaca-lin64/bin$
./test-arith-fdiv.sh
*********SandyBridge*********
Intel(R) Architecture Code Analyzer Version - 2.1
Analyzed File - arith-fdiv.o
Binary Format - 64Bit
Architecture - SNB
Analysis Type - Throughput

Throughput Analysis Report

Block Throughput: 41.00 Cycles Throughput Bottleneck: InterIteration

Port Binding In Cycles Per Iteration:

| Port | 0 - DV | 1 | 2 - D | 3 - D | 4 | 5 |

| Cycles | 2.0 28.0 | 0.0 | 0.0 0.0 | 0.0 0.0 | 0.0 | 1.0 |

N - port number or number of cycles resource conflict caused delay, DV - Divider pipe (on port 0)
D - Data fetch pipe (on ports 2 and 3), CP - on a critical path
F - Macro Fusion with the previous instruction occurred

• - instruction micro-ops not bound to a port

^ - Micro Fusion happened

- ESP Tracking sync uop was issued

@ - SSE instruction followed an AVX256 instruction, dozens of cycles penalty is expected
! - instruction not supported, was not accounted in Analysis

Num Of	Ports pressure in cycles
Uops	0 - DV	1	2 - D	3 - D	4	5

---

3	2.0 28.0					1.0	CP	vdivps ymm0, ymm0, ymm0

Total Num Of Uops: 3

If we use DV value then it's about 2 times. But I used "Block Throughput" value above. As you see my block is exactly one instruction that's why I decided to use "Block Throughput". Maybe I'm wrong? Can anyone suggest me the right decision?

[avt77]

I changed everything except the issue with ymm FDIV. As I wrote in my answer on the comment I'm using Block Throughput from IACA tool. If I'm wrong please say me about and I'll recollect all numbers.

[RKSimon]

lib/Target/X86/X86TargetTransformInfo.cpp

1073

Please add Nehalem costs (from Agner) - they're notably better than the P4 default:

FSQRT f32/4f32 : 18 f64/2f64 : 32

[RKSimon]

I'm swaying toward using Agner's numbers for SandyBridge FDIV / FSQRT costs - would anyone have any objections?

[mkuper]

In D25722#578473, @RKSimon wrote:

I'm swaying toward using Agner's numbers for SandyBridge FDIV / FSQRT costs - would anyone have any objections?

I'd really like to understand the discrepancy between Agner's numbers and the IACA numbers - especially since IACA does give the expected number for the latency.
Andrew, any chance you could ping the IACA people at Intel and ask them about this?

Meanwhile, I think I'd also prefer this to go in with Agner's numbers - and change it to the IACA numbers later if it turns out it's accounting for something real.

[RKSimon]

In D25722#578664, @mkuper wrote:

I'd really like to understand the discrepancy between Agner's numbers and the IACA numbers - especially since IACA does give the expected number for the latency.
Andrew, any chance you could ping the IACA people at Intel and ask them about this?

IACA doesn't have that great support these days - the website says development has been suspended, although they have announced they will do a BDW/SKL bugfix release at some point. Similar comments about dodgy costs don't seem to be answered.

[mkuper]

In D25722#578712, @RKSimon wrote:

In D25722#578664, @mkuper wrote:

I'd really like to understand the discrepancy between Agner's numbers and the IACA numbers - especially since IACA does give the expected number for the latency.
Andrew, any chance you could ping the IACA people at Intel and ask them about this?

IACA doesn't have that great support these days - the website says development has been suspended, although they have announced they will do a BDW/SKL bugfix release at some point. Similar comments about dodgy costs don't seem to be answered.

To the best of my knowledge, Andrew works for Intel, so he may have a better chance of getting an answer. :-)
As to support - the website has a comment from July that states that "[they] are resuming support for Intel(R) Architecture Code Analyzer with BDW and SKL support probably before end of 2016".

So, one can hope...

[avt77]

If I understood correctly I should replace all IACA numbers with Agner's numbers, right? OK, I'll do it.
JFYI, I'm not working in Intel since July but of course I know a lot of guys from Intel and I'll try to ask them about IACA future.

[avt77]

lib/Target/X86/X86TargetTransformInfo.cpp

269

Maybe I understood the difference between Anger's and IACA numbers (for SandyBridge at least): Anger's table has numbers for ymm only :
VDIVPS y,y,y 3 3 2 1 21-29 20-28 AVX
VDIVPS y,y,m256 4 3 2 1 1+ 20-28 AVX

but when I played with IACA it showed different types of operands: xmm0, ... and ymm0, ....

269

And another comment: for f32/v4f32 IACA shows 14 for vdivss/vdivps but Agner's table shows the same 10-14 for divss/divps and 20-28 for vdivss/vdivps. Clang generates vdivss/vdivps (for SNB target) that's why I should select 28 as a cost value is we're using Agner's numbers. Is it OK?

269

As an intermediate decision I did the following for SNB numbers: xmm operands use the first number as a cost while ymm operands use the second number. For example:

{ ISD::FDIV, MVT::f32,   20 }, // SNB from http://www.agner.org/ (IACA: 14)
{ ISD::FDIV, MVT::v4f32, 20 }, // SNB from http://www.agner.org/ (IACA: 14)
{ ISD::FDIV, MVT::v8f32, 28 }, // SNB from http://www.agner.org/ (IACA: 41)

Is it acceptable? In fact we need here some extension of our current Cost tables: it is not enough to know the instruction itself and the type of its operands. We need some target dependent info as well (e.g. if we're speaking about X86 targets only (exactly our case) then we could add the size of the target registers or something similar, e.g. ISA: SNB/HSW could use both divps and vdivps and usage of 2 divps could be better then usage of one vdivps but our current Cost Model cannot help to decide what's better).

269

BTW, I've just realized that Haswell IACA numbers are really closed to the expected ones:

f32 13 vdivss
v4f32 13 vdivps xmm0...
v8f32 26 vdivps ymm0...
f64 20 vdivsd
v2f64 20 vdivpd
v4f62 47 vdivpd ymm0...

It means we have problems with IACA numbers for SNB only (too big difference for xmm and ymm). Maybe new CPUs simply resolved this issue?

1073

JFYI, I got the same numbers for Nehalem with IACA

[avt77]

All numbers from IACA were replaced with Agner's numbers

[avt77]

The wrong SNB numbers were fixed (tnx to Simon Pilgrim)

[RKSimon]

I think this is almost done now - please can you add the AVX2/Haswell costs:

Haswell
FDIV f32/4f32 = 7, 8f32 = 14, f64/2f64 = 14, 4f64 = 28
FSQRT f32/4f32 = 7, 8f32 = 14, f64/2f64 = 14, 4f64 = 28

Other than that does anyone have any additional feedback?

[mkuper]

In D25722#582086, @RKSimon wrote:

I think this is almost done now - please can you add the AVX2/Haswell costs:

Haswell
FDIV f32/4f32 = 7, 8f32 = 14, f64/2f64 = 14, 4f64 = 28
FSQRT f32/4f32 = 7, 8f32 = 14, f64/2f64 = 14, 4f64 = 28

Other than that does anyone have any additional feedback?

No, LGTM (modulo what Simon said about Haswell)

Thanks for fixing this, Andrew.

[avt77]

Haswell numbers added for AVX2

[RKSimon]

LGTM with one minor

lib/Target/X86/X86TargetTransformInfo.cpp

1112

Better to use the Pentium III costs F32 = 28, VF432 = 56

[avt77]

FSQRT changes: SSE1 Cost table updated with Pentium III numbers; SSE42 cost table added with Nehalem numbers

lib/Target/X86/X86TargetTransformInfo.cpp

int X86TTIImpl::getArithmeticInstrCost(
 
   // Look for AVX2 lowering tricks for custom cases.
   if (ST->hasAVX2()) {
     if (const auto *Entry = CostTableLookup(AVX2CustomCostTable, ISD,
                                             LT.second))
       return LT.first * Entry->Cost;
   }
 
+  static const CostTblEntry AVXFloatCostTable[] = {
+    { ISD::FDIV,  MVT::f32, 13 },   // IACA value for Haswell arch
+    { ISD::FDIV,  MVT::v4f32, 13 }, // IACA value for Haswell arch
+    { ISD::FDIV,  MVT::v8f32, 26 }, // IACA value for Haswell arch
+    { ISD::FDIV,  MVT::f64,   20 }, // IACA value for Haswell arch
+    { ISD::FDIV,  MVT::v2f64, 20 }, // IACA value for Haswell arch
+    { ISD::FDIV,  MVT::v4f64, 47 }, // IACA value for Haswell arch
+  };
+
+  if (ST->hasAVX()) {
+    if (const auto *Entry = CostTableLookup(AVXFloatCostTable, ISD,
+                                            LT.second))
+      return LT.first * Entry->Cost;
+  }
+
   static const CostTblEntry
   SSE2UniformConstCostTable[] = {
     // Constant splats are cheaper for the following instructions.
     { ISD::SDIV, MVT::v8i16,  6 }, // pmulhw sequence
     { ISD::UDIV, MVT::v8i16,  6 }, // pmulhuw sequence
     { ISD::SDIV, MVT::v4i32, 19 }, // pmuludq sequence
     { ISD::UDIV, MVT::v4i32, 15 }, // pmuludq sequence
   };
static const CostTblEntry SSE2CostTable[] = {
     { ISD::SDIV,  MVT::v16i8,  16*20 },
     { ISD::SDIV,  MVT::v8i16,  8*20 },
     { ISD::SDIV,  MVT::v4i32,  4*20 },
     { ISD::SDIV,  MVT::v2i64,  2*20 },
     { ISD::UDIV,  MVT::v16i8,  16*20 },
     { ISD::UDIV,  MVT::v8i16,  8*20 },
     { ISD::UDIV,  MVT::v4i32,  4*20 },
     { ISD::UDIV,  MVT::v2i64,  2*20 },
+    { ISD::FDIV, MVT::f64,   21 }, // IACA value for Nehalem arch
+    { ISD::FDIV, MVT::v2f64, 21 }, // IACA value for Nehalem arch
   };
 
   if (ST->hasSSE2()) {
     if (const auto *Entry = CostTableLookup(SSE2CostTable, ISD, LT.second))
       return LT.first * Entry->Cost;
   }
 
   static const CostTblEntry AVX1CostTable[] = {
     // We don't have to scalarize unsupported ops. We can issue two half-sized
     // operations and we only need to extract the upper YMM half.
if (const auto *Entry = CostTableLookup(CustomLowered, ISD, LT.second))
     return LT.first * Entry->Cost;
 
   // Special lowering of v4i32 mul on sse2, sse3: Lower v4i32 mul as 2x shuffle,
   // 2x pmuludq, 2x shuffle.
   if (ISD == ISD::MUL && LT.second == MVT::v4i32 && ST->hasSSE2() &&
       !ST->hasSSE41())
     return LT.first * 6;
 
+  static const CostTblEntry SSE1FloatCostTable[] = {
+    { ISD::FDIV, MVT::f32,   14 }, // IACA value for Nehalem arch
+    { ISD::FDIV, MVT::v4f32, 14 }, // IACA value for Nehalem arch
+  };
+
+  if (const auto *Entry = CostTableLookup(SSE1FloatCostTable, ISD,
+                                          LT.second))
+    return LT.first * Entry->Cost;
   // Fallback to the default implementation.
   return BaseT::getArithmeticInstrCost(Opcode, Ty, Op1Info, Op2Info);
 }
 
 int X86TTIImpl::getShuffleCost(TTI::ShuffleKind Kind, Type *Tp, int Index,
                                Type *SubTp) {
   // We only estimate the cost of reverse and alternate shuffles.
   if (Kind != TTI::SK_Reverse && Kind != TTI::SK_Alternate)
static const CostTblEntry AVX1CostTbl[] = {
     { ISD::CTPOP,      MVT::v4i64,  14 },
     { ISD::CTPOP,      MVT::v8i32,  22 },
     { ISD::CTPOP,      MVT::v16i16, 18 },
     { ISD::CTPOP,      MVT::v32i8,  12 },
     { ISD::CTTZ,       MVT::v4i64,  20 },
     { ISD::CTTZ,       MVT::v8i32,  28 },
     { ISD::CTTZ,       MVT::v16i16, 24 },
     { ISD::CTTZ,       MVT::v32i8,  18 },
+    { ISD::FSQRT,      MVT::f32,    13 }, // IACA value for Haswell arch
+    { ISD::FSQRT,      MVT::v4f32,  13 }, // IACA value for Haswell arch
+    { ISD::FSQRT,      MVT::v8f32,  26 }, // IACA value for Haswell arch
+    { ISD::FSQRT,      MVT::f64,    20 }, // IACA value for Haswell arch
+    { ISD::FSQRT,      MVT::v2f64,  20 }, // IACA value for Haswell arch
+    { ISD::FSQRT,      MVT::v4f64,  47 }, // IACA value for Haswell arch
+  };
+  static const CostTblEntry SSE1FloatCostTable[] = {
+    { ISD::FSQRT, MVT::f32,   18 },
+    { ISD::FSQRT, MVT::v4f32, 18 },
   };
   static const CostTblEntry SSSE3CostTbl[] = {
     { ISD::BITREVERSE, MVT::v2i64,   5 },
     { ISD::BITREVERSE, MVT::v4i32,   5 },
     { ISD::BITREVERSE, MVT::v8i16,   5 },
     { ISD::BITREVERSE, MVT::v16i8,   5 },
     { ISD::BSWAP,      MVT::v2i64,   1 },
     { ISD::BSWAP,      MVT::v4i32,   1 },
     { ISD::BSWAP,      MVT::v8i16,   1 },
     { ISD::CTLZ,       MVT::v2i64,  23 },
static const CostTblEntry SSE2CostTbl[] = {
     /* ISD::CTLZ - currently scalarized pre-SSSE3 */
     { ISD::CTPOP,      MVT::v2i64,  12 },
     { ISD::CTPOP,      MVT::v4i32,  15 },
     { ISD::CTPOP,      MVT::v8i16,  13 },
     { ISD::CTPOP,      MVT::v16i8,  10 },
     { ISD::CTTZ,       MVT::v2i64,  14 },
     { ISD::CTTZ,       MVT::v4i32,  18 },
     { ISD::CTTZ,       MVT::v8i16,  16 },
-    { ISD::CTTZ,       MVT::v16i8,  13 }
+    { ISD::CTTZ,       MVT::v16i8,  13 },
+    { ISD::FSQRT,      MVT::f64,    32 },
+    { ISD::FSQRT,      MVT::v2f64,  32 },
   };
 
   unsigned ISD = ISD::DELETED_NODE;
   switch (IID) {
   default:
     break;
   case Intrinsic::bitreverse:
     ISD = ISD::BITREVERSE;
     break;
   case Intrinsic::bswap:
     ISD = ISD::BSWAP;
     break;
   case Intrinsic::ctlz:
     ISD = ISD::CTLZ;
     break;
   case Intrinsic::ctpop:
     ISD = ISD::CTPOP;
     break;
   case Intrinsic::cttz:
     ISD = ISD::CTTZ;
     break;
+  case Intrinsic::sqrt:
+    ISD = ISD::FSQRT;
+    break;
   }
 
   // Legalize the type.
   std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, RetTy);
   MVT MTy = LT.second;
 
   // Attempt to lookup cost.
   if (ST->hasXOP())
int X86TTIImpl::getIntrinsicInstrCost(Intrinsic::ID IID, Type *RetTy,
   if (ST->hasSSSE3())
     if (const auto *Entry = CostTableLookup(SSSE3CostTbl, ISD, MTy))
       return LT.first * Entry->Cost;
 
   if (ST->hasSSE2())
     if (const auto *Entry = CostTableLookup(SSE2CostTbl, ISD, MTy))
       return LT.first * Entry->Cost;
 
+  if (const auto *Entry = CostTableLookup(SSE1FloatCostTable, ISD, MTy))
+    return LT.first * Entry->Cost;
+    
   return BaseT::getIntrinsicInstrCost(IID, RetTy, Tys, FMF);
 }
 
 int X86TTIImpl::getIntrinsicInstrCost(Intrinsic::ID IID, Type *RetTy,
                                       ArrayRef<Value *> Args, FastMathFlags FMF) {
   return BaseT::getIntrinsicInstrCost(IID, RetTy, Args, FMF);
 }
 

test/Analysis/CostModel/X86/arith-fp.ll

-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+sse2 | FileCheck %s --check-prefix=CHECK --check-prefix=SSE2
-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+sse4.2 | FileCheck %s --check-prefix=CHECK --check-prefix=SSE42
-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx,+fma | FileCheck %s --check-prefix=CHECK --check-prefix=AVX
-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx2,+fma | FileCheck %s --check-prefix=CHECK --check-prefix=AVX2
-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx512f | FileCheck %s --check-prefix=CHECK --check-prefix=AVX512 --check-prefix=AVX512F
-; RUN: opt < %s  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx512f,+avx512bw | FileCheck %s --check-prefix=CHECK --check-prefix=AVX512 --check-prefix=AVX512BW
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+sse2 | FileCheck %s --check-prefix=CHECK --check-prefix=SSE2
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+sse4.2 | FileCheck %s --check-prefix=CHECK --check-prefix=SSE42
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx,+fma | FileCheck %s --check-prefix=CHECK --check-prefix=AVX
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx2,+fma | FileCheck %s --check-prefix=CHECK --check-prefix=AVX2
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx512f | FileCheck %s --check-prefix=CHECK --check-prefix=AVX512 --check-prefix=AVX512F
+; RUN: opt < %s -enable-no-nans-fp-math  -cost-model -analyze -mtriple=x86_64-apple-macosx10.8.0 -mattr=+avx512f,+avx512bw | FileCheck %s --check-prefix=CHECK --check-prefix=AVX512 --check-prefix=AVX512BW
 
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.8.0"
 
 ; CHECK-LABEL: 'fadd'
 define i32 @fadd(i32 %arg) {
   ; SSE2: cost of 2 {{.*}} %F32 = fadd
   ; SSE42: cost of 2 {{.*}} %F32 = fadd
define i32 @fmul(i32 %arg) {
   ; AVX512: cost of 2 {{.*}} %V8F64 = fmul
   %V8F64 = fmul <8 x double> undef, undef
 
   ret i32 undef
 }
 
 ; CHECK-LABEL: 'fdiv'
 define i32 @fdiv(i32 %arg) {
-  ; SSE2: cost of 2 {{.*}} %F32 = fdiv
-  ; SSE42: cost of 2 {{.*}} %F32 = fdiv
-  ; AVX: cost of 2 {{.*}} %F32 = fdiv
-  ; AVX2: cost of 2 {{.*}} %F32 = fdiv
-  ; AVX512: cost of 2 {{.*}} %F32 = fdiv
+  ; SSE2: cost of 14 {{.*}} %F32 = fdiv
+  ; SSE42: cost of 14 {{.*}} %F32 = fdiv
+  ; AVX: cost of 13 {{.*}} %F32 = fdiv
+  ; AVX2: cost of 13 {{.*}} %F32 = fdiv
+  ; AVX512: cost of 13 {{.*}} %F32 = fdiv
   %F32 = fdiv float undef, undef
-  ; SSE2: cost of 2 {{.*}} %V4F32 = fdiv
-  ; SSE42: cost of 2 {{.*}} %V4F32 = fdiv
-  ; AVX: cost of 2 {{.*}} %V4F32 = fdiv
-  ; AVX2: cost of 2 {{.*}} %V4F32 = fdiv
-  ; AVX512: cost of 2 {{.*}} %V4F32 = fdiv
+  ; SSE2: cost of 14 {{.*}} %V4F32 = fdiv
+  ; SSE42: cost of 14 {{.*}} %V4F32 = fdiv
+  ; AVX: cost of 13 {{.*}} %V4F32 = fdiv
+  ; AVX2: cost of 13 {{.*}} %V4F32 = fdiv
+  ; AVX512: cost of 13 {{.*}} %V4F32 = fdiv
   %V4F32 = fdiv <4 x float> undef, undef
-  ; SSE2: cost of 4 {{.*}} %V8F32 = fdiv
-  ; SSE42: cost of 4 {{.*}} %V8F32 = fdiv
-  ; AVX: cost of 2 {{.*}} %V8F32 = fdiv
-  ; AVX2: cost of 2 {{.*}} %V8F32 = fdiv
-  ; AVX512: cost of 2 {{.*}} %V8F32 = fdiv
+  ; SSE2: cost of 28 {{.*}} %V8F32 = fdiv
+  ; SSE42: cost of 28 {{.*}} %V8F32 = fdiv
+  ; AVX: cost of 26 {{.*}} %V8F32 = fdiv
+  ; AVX2: cost of 26 {{.*}} %V8F32 = fdiv
+  ; AVX512: cost of 26 {{.*}} %V8F32 = fdiv
   %V8F32 = fdiv <8 x float> undef, undef
-  ; SSE2: cost of 8 {{.*}} %V16F32 = fdiv
-  ; SSE42: cost of 8 {{.*}} %V16F32 = fdiv
-  ; AVX: cost of 4 {{.*}} %V16F32 = fdiv
-  ; AVX2: cost of 4 {{.*}} %V16F32 = fdiv
+  ; SSE2: cost of 56 {{.*}} %V16F32 = fdiv
+  ; SSE42: cost of 56 {{.*}} %V16F32 = fdiv
+  ; AVX: cost of 52 {{.*}} %V16F32 = fdiv
+  ; AVX2: cost of 52 {{.*}} %V16F32 = fdiv
   ; AVX512: cost of 2 {{.*}} %V16F32 = fdiv
   %V16F32 = fdiv <16 x float> undef, undef
 
-  ; SSE2: cost of 2 {{.*}} %F64 = fdiv
-  ; SSE42: cost of 2 {{.*}} %F64 = fdiv
-  ; AVX: cost of 2 {{.*}} %F64 = fdiv
-  ; AVX2: cost of 2 {{.*}} %F64 = fdiv
-  ; AVX512: cost of 2 {{.*}} %F64 = fdiv
+  ; SSE2: cost of 21 {{.*}} %F64 = fdiv
+  ; SSE42: cost of 21 {{.*}} %F64 = fdiv
+  ; AVX: cost of 20 {{.*}} %F64 = fdiv
+  ; AVX2: cost of 20 {{.*}} %F64 = fdiv
+  ; AVX512: cost of 20 {{.*}} %F64 = fdiv
   %F64 = fdiv double undef, undef
-  ; SSE2: cost of 2 {{.*}} %V2F64 = fdiv
-  ; SSE42: cost of 2 {{.*}} %V2F64 = fdiv
-  ; AVX: cost of 2 {{.*}} %V2F64 = fdiv
-  ; AVX2: cost of 2 {{.*}} %V2F64 = fdiv
-  ; AVX512: cost of 2 {{.*}} %V2F64 = fdiv
+  ; SSE2: cost of 21 {{.*}} %V2F64 = fdiv
+  ; SSE42: cost of 21 {{.*}} %V2F64 = fdiv
+  ; AVX: cost of 20 {{.*}} %V2F64 = fdiv
+  ; AVX2: cost of 20 {{.*}} %V2F64 = fdiv
+  ; AVX512: cost of 20 {{.*}} %V2F64 = fdiv
   %V2F64 = fdiv <2 x double> undef, undef
-  ; SSE2: cost of 4 {{.*}} %V4F64 = fdiv
-  ; SSE42: cost of 4 {{.*}} %V4F64 = fdiv
-  ; AVX: cost of 2 {{.*}} %V4F64 = fdiv
-  ; AVX2: cost of 2 {{.*}} %V4F64 = fdiv
-  ; AVX512: cost of 2 {{.*}} %V4F64 = fdiv
+  ; SSE2: cost of 42 {{.*}} %V4F64 = fdiv
+  ; SSE42: cost of 42 {{.*}} %V4F64 = fdiv
+  ; AVX: cost of 47 {{.*}} %V4F64 = fdiv
+  ; AVX2: cost of 47 {{.*}} %V4F64 = fdiv
+  ; AVX512: cost of 47 {{.*}} %V4F64 = fdiv
   %V4F64 = fdiv <4 x double> undef, undef
-  ; SSE2: cost of 8 {{.*}} %V8F64 = fdiv
-  ; SSE42: cost of 8 {{.*}} %V8F64 = fdiv
-  ; AVX: cost of 4 {{.*}} %V8F64 = fdiv
-  ; AVX2: cost of 4 {{.*}} %V8F64 = fdiv
+  ; SSE2: cost of 84 {{.*}} %V8F64 = fdiv
+  ; SSE42: cost of 84 {{.*}} %V8F64 = fdiv
+  ; AVX: cost of 94 {{.*}} %V8F64 = fdiv
+  ; AVX2: cost of 94 {{.*}} %V8F64 = fdiv
   ; AVX512: cost of 2 {{.*}} %V8F64 = fdiv
   %V8F64 = fdiv <8 x double> undef, undef
 
   ret i32 undef
 }
 
 ; CHECK-LABEL: 'frem'
 define i32 @frem(i32 %arg) {
define i32 @frem(i32 %arg) {
   ; AVX512: cost of 30 {{.*}} %V8F64 = frem
   %V8F64 = frem <8 x double> undef, undef
 
   ret i32 undef
 }
 
 ; CHECK-LABEL: 'fsqrt'
 define i32 @fsqrt(i32 %arg) {
-  ; SSE2: cost of 1 {{.*}} %F32 = call float @llvm.sqrt.f32
-  ; SSE42: cost of 1 {{.*}} %F32 = call float @llvm.sqrt.f32
-  ; AVX: cost of 1 {{.*}} %F32 = call float @llvm.sqrt.f32
-  ; AVX2: cost of 1 {{.*}} %F32 = call float @llvm.sqrt.f32
-  ; AVX512: cost of 1 {{.*}} %F32 = call float @llvm.sqrt.f32
+  ; SSE2: cost of 18 {{.*}} %F32 = call float @llvm.sqrt.f32
+  ; SSE42: cost of 18 {{.*}} %F32 = call float @llvm.sqrt.f32
+  ; AVX: cost of 13 {{.*}} %F32 = call float @llvm.sqrt.f32
+  ; AVX2: cost of 13 {{.*}} %F32 = call float @llvm.sqrt.f32
+  ; AVX512: cost of 13 {{.*}} %F32 = call float @llvm.sqrt.f32
   %F32 = call float @llvm.sqrt.f32(float undef)
-  ; SSE2: cost of 1 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
-  ; SSE42: cost of 1 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
-  ; AVX: cost of 1 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
-  ; AVX2: cost of 1 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
-  ; AVX512: cost of 1 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
+  ; SSE2: cost of 18 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
+  ; SSE42: cost of 18 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
+  ; AVX: cost of 13 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
+  ; AVX2: cost of 13 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
+  ; AVX512: cost of 13 {{.*}} %V4F32 = call <4 x float> @llvm.sqrt.v4f32
   %V4F32 = call <4 x float> @llvm.sqrt.v4f32(<4 x float> undef)
-  ; SSE2: cost of 4 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
-  ; SSE42: cost of 4 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
-  ; AVX: cost of 1 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
-  ; AVX2: cost of 1 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
-  ; AVX512: cost of 1 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
+  ; SSE2: cost of 36 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
+  ; SSE42: cost of 36 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
+  ; AVX: cost of 26 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
+  ; AVX2: cost of 26 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
+  ; AVX512: cost of 26 {{.*}} %V8F32 = call <8 x float> @llvm.sqrt.v8f32
   %V8F32 = call <8 x float> @llvm.sqrt.v8f32(<8 x float> undef)
-  ; SSE2: cost of 8 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
-  ; SSE42: cost of 8 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
-  ; AVX: cost of 4 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
-  ; AVX2: cost of 4 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
+  ; SSE2: cost of 72 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
+  ; SSE42: cost of 72 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
+  ; AVX: cost of 52 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
+  ; AVX2: cost of 52 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
   ; AVX512: cost of 1 {{.*}} %V16F32 = call <16 x float> @llvm.sqrt.v16f32
   %V16F32 = call <16 x float> @llvm.sqrt.v16f32(<16 x float> undef)
 
-  ; SSE2: cost of 1 {{.*}} %F64 = call double @llvm.sqrt.f64
-  ; SSE42: cost of 1 {{.*}} %F64 = call double @llvm.sqrt.f64
-  ; AVX: cost of 1 {{.*}} %F64 = call double @llvm.sqrt.f64
-  ; AVX2: cost of 1 {{.*}} %F64 = call double @llvm.sqrt.f64
-  ; AVX512: cost of 1 {{.*}} %F64 = call double @llvm.sqrt.f64
+  ; SSE2: cost of 32 {{.*}} %F64 = call double @llvm.sqrt.f64
+  ; SSE42: cost of 32 {{.*}} %F64 = call double @llvm.sqrt.f64
+  ; AVX: cost of 20 {{.*}} %F64 = call double @llvm.sqrt.f64
+  ; AVX2: cost of 20 {{.*}} %F64 = call double @llvm.sqrt.f64
+  ; AVX512: cost of 20 {{.*}} %F64 = call double @llvm.sqrt.f64
   %F64 = call double @llvm.sqrt.f64(double undef)
-  ; SSE2: cost of 1 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
-  ; SSE42: cost of 1 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
-  ; AVX: cost of 1 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
-  ; AVX2: cost of 1 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
-  ; AVX512: cost of 1 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
+  ; SSE2: cost of 32 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
+  ; SSE42: cost of 32 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
+  ; AVX: cost of 20 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
+  ; AVX2: cost of 20 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
+  ; AVX512: cost of 20 {{.*}} %V2F64 = call <2 x double> @llvm.sqrt.v2f64
   %V2F64 = call <2 x double> @llvm.sqrt.v2f64(<2 x double> undef)
-  ; SSE2: cost of 4 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
-  ; SSE42: cost of 4 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
-  ; AVX: cost of 1 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
-  ; AVX2: cost of 1 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
-  ; AVX512: cost of 1 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
+  ; SSE2: cost of 64 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
+  ; SSE42: cost of 64 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
+  ; AVX: cost of 47 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
+  ; AVX2: cost of 47 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
+  ; AVX512: cost of 47 {{.*}} %V4F64 = call <4 x double> @llvm.sqrt.v4f64
   %V4F64 = call <4 x double> @llvm.sqrt.v4f64(<4 x double> undef)
-  ; SSE2: cost of 8 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
-  ; SSE42: cost of 8 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
-  ; AVX: cost of 4 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
-  ; AVX2: cost of 4 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
+  ; SSE2: cost of 128 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
+  ; SSE42: cost of 128 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
+  ; AVX: cost of 94 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
+  ; AVX2: cost of 94 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
   ; AVX512: cost of 1 {{.*}} %V8F64 = call <8 x double> @llvm.sqrt.v8f64
   %V8F64 = call <8 x double> @llvm.sqrt.v8f64(<8 x double> undef)
 
   ret i32 undef
 }
 
 ; CHECK-LABEL: 'fabs'
 define i32 @fabs(i32 %arg) {