This is an archive of the discontinued LLVM Phabricator instance.

Differential D111060

[X86][Costmodel] Load/store i32/f32 Stride=4 VF=2 interleaving costs
ClosedPublic

Authored by lebedev.ri on Oct 4 2021, 8:04 AM.

Details

Reviewers
RKSimon
Commits
rGb6234c1edffc: [X86][Costmodel] Load/store i32/f32 Stride=4 VF=2 interleaving costs
Summary

Finally, we are getting to the heavy-hitter stuff!

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/7crGWoar6 - for intels Block RThroughput: =4.0; for ryzens, Block RThroughput: <=2.0
So could pick cost of 4.

For store we have:
https://godbolt.org/z/T8aq3MszM - for intels Block RThroughput: =5.0; for ryzens, Block RThroughput: <=2.0
So we could pick cost of 5.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

Diff Detail

Revision Contents

Diff 377207

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

Show First 20 Lines • Show All 5,122 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedLoadTbl[] = {
{4, MVT::v32i8, 56}, // (load 128i8 and) deinterleave into 4 x 32i8 {4, MVT::v32i8, 56}, // (load 128i8 and) deinterleave into 4 x 32i8
{4, MVT::v2i16, 6}, // (load 8i16 and) deinterleave into 4 x 2i16 {4, MVT::v2i16, 6}, // (load 8i16 and) deinterleave into 4 x 2i16
{4, MVT::v4i16, 17}, // (load 16i16 and) deinterleave into 4 x 4i16 {4, MVT::v4i16, 17}, // (load 16i16 and) deinterleave into 4 x 4i16
{4, MVT::v8i16, 33}, // (load 32i16 and) deinterleave into 4 x 8i16 {4, MVT::v8i16, 33}, // (load 32i16 and) deinterleave into 4 x 8i16
{4, MVT::v16i16, 75}, // (load 64i16 and) deinterleave into 4 x 16i16 {4, MVT::v16i16, 75}, // (load 64i16 and) deinterleave into 4 x 16i16
{4, MVT::v32i16, 150}, // (load 128i16 and) deinterleave into 4 x 32i16 {4, MVT::v32i16, 150}, // (load 128i16 and) deinterleave into 4 x 32i16
{4, MVT::v2i32, 4}, // (load 8i32 and) deinterleave into 4 x 2i32
{6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8 {6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8
{6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8 {6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8
{6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8 {6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8
{6, MVT::v16i8, 43}, // (load 96i8 and) deinterleave into 6 x 16i8 {6, MVT::v16i8, 43}, // (load 96i8 and) deinterleave into 6 x 16i8
{6, MVT::v32i8, 82}, // (load 192i8 and) deinterleave into 6 x 32i8 {6, MVT::v32i8, 82}, // (load 192i8 and) deinterleave into 6 x 32i8
{6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16 {6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16
{6, MVT::v4i16, 9}, // (load 24i16 and) deinterleave into 6 x 4i16 {6, MVT::v4i16, 9}, // (load 24i16 and) deinterleave into 6 x 4i16
▲ Show 20 Lines • Show All 56 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedStoreTbl[] = {
{4, MVT::v32i8, 12}, // interleave 4 x 32i8 into 128i8 (and store) {4, MVT::v32i8, 12}, // interleave 4 x 32i8 into 128i8 (and store)
{4, MVT::v2i16, 2}, // interleave 4 x 2i16 into 8i16 (and store) {4, MVT::v2i16, 2}, // interleave 4 x 2i16 into 8i16 (and store)
{4, MVT::v4i16, 6}, // interleave 4 x 4i16 into 16i16 (and store) {4, MVT::v4i16, 6}, // interleave 4 x 4i16 into 16i16 (and store)
{4, MVT::v8i16, 10}, // interleave 4 x 8i16 into 32i16 (and store) {4, MVT::v8i16, 10}, // interleave 4 x 8i16 into 32i16 (and store)
{4, MVT::v16i16, 32}, // interleave 4 x 16i16 into 64i16 (and store) {4, MVT::v16i16, 32}, // interleave 4 x 16i16 into 64i16 (and store)
{4, MVT::v32i16, 64}, // interleave 4 x 32i16 into 128i16 (and store) {4, MVT::v32i16, 64}, // interleave 4 x 32i16 into 128i16 (and store)
{4, MVT::v2i32, 5}, // interleave 4 x 2i32 into 8i32 (and store)
{6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store) {6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store)
{6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store) {6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store)
{6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store) {6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store)
{6, MVT::v16i8, 27}, // interleave 6 x 16i8 into 96i8 (and store) {6, MVT::v16i8, 27}, // interleave 6 x 16i8 into 96i8 (and store)
{6, MVT::v32i8, 90}, // interleave 6 x 32i8 into 192i8 (and store) {6, MVT::v32i8, 90}, // interleave 6 x 32i8 into 192i8 (and store)
{6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store) {6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store)
{6, MVT::v4i16, 15}, // interleave 6 x 4i16 into 24i16 (and store) {6, MVT::v4i16, 15}, // interleave 6 x 4i16 into 24i16 (and store)
▲ Show 20 Lines • Show All 174 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-f32-stride-4.ll

Show All 18 Lines
; ;
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 15 for VF 2 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 15 for VF 2 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 34 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 34 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 76 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 76 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 152 for VF 16 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 152 for VF 16 For instruction: %v0 = load float, float* %in0, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 15 for VF 2 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 34 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 34 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 76 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 76 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 152 for VF 16 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 152 for VF 16 For instruction: %v0 = load float, float* %in0, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
▲ Show 20 Lines • Show All 43 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-i32-stride-4.ll

Show All 18 Lines
; ;
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 21 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 21 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 42 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 42 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 92 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 92 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 184 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 184 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 21 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 42 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 42 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 92 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 92 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 184 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 184 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
▲ Show 20 Lines • Show All 43 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-f32-stride-4.ll

Show All 18 Lines
; ;
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4
; AVX1: LV: Found an estimated cost of 13 for VF 2 For instruction: store float %v3, float* %out3, align 4 ; AVX1: LV: Found an estimated cost of 13 for VF 2 For instruction: store float %v3, float* %out3, align 4
; AVX1: LV: Found an estimated cost of 30 for VF 4 For instruction: store float %v3, float* %out3, align 4 ; AVX1: LV: Found an estimated cost of 30 for VF 4 For instruction: store float %v3, float* %out3, align 4
; AVX1: LV: Found an estimated cost of 76 for VF 8 For instruction: store float %v3, float* %out3, align 4 ; AVX1: LV: Found an estimated cost of 76 for VF 8 For instruction: store float %v3, float* %out3, align 4
; AVX1: LV: Found an estimated cost of 152 for VF 16 For instruction: store float %v3, float* %out3, align 4 ; AVX1: LV: Found an estimated cost of 152 for VF 16 For instruction: store float %v3, float* %out3, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4
; AVX2: LV: Found an estimated cost of 13 for VF 2 For instruction: store float %v3, float* %out3, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction: store float %v3, float* %out3, align 4
; AVX2: LV: Found an estimated cost of 30 for VF 4 For instruction: store float %v3, float* %out3, align 4 ; AVX2: LV: Found an estimated cost of 30 for VF 4 For instruction: store float %v3, float* %out3, align 4
; AVX2: LV: Found an estimated cost of 76 for VF 8 For instruction: store float %v3, float* %out3, align 4 ; AVX2: LV: Found an estimated cost of 76 for VF 8 For instruction: store float %v3, float* %out3, align 4
; AVX2: LV: Found an estimated cost of 152 for VF 16 For instruction: store float %v3, float* %out3, align 4 ; AVX2: LV: Found an estimated cost of 152 for VF 16 For instruction: store float %v3, float* %out3, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v3, float* %out3, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store float %v3, float* %out3, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store float %v3, float* %out3, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: store float %v3, float* %out3, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: store float %v3, float* %out3, align 4
; AVX512: LV: Found an estimated cost of 11 for VF 8 For instruction: store float %v3, float* %out3, align 4 ; AVX512: LV: Found an estimated cost of 11 for VF 8 For instruction: store float %v3, float* %out3, align 4
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-i32-stride-4.ll

Show All 18 Lines
; ;
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4
; AVX1: LV: Found an estimated cost of 19 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX1: LV: Found an estimated cost of 19 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4
; AVX1: LV: Found an estimated cost of 38 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX1: LV: Found an estimated cost of 38 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4
; AVX1: LV: Found an estimated cost of 92 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX1: LV: Found an estimated cost of 92 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4
; AVX1: LV: Found an estimated cost of 184 for VF 16 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX1: LV: Found an estimated cost of 184 for VF 16 For instruction: store i32 %v3, i32* %out3, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4
; AVX2: LV: Found an estimated cost of 19 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4
; AVX2: LV: Found an estimated cost of 38 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX2: LV: Found an estimated cost of 38 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4
; AVX2: LV: Found an estimated cost of 92 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX2: LV: Found an estimated cost of 92 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4
; AVX2: LV: Found an estimated cost of 184 for VF 16 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX2: LV: Found an estimated cost of 184 for VF 16 For instruction: store i32 %v3, i32* %out3, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v3, i32* %out3, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store i32 %v3, i32* %out3, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 4 For instruction: store i32 %v3, i32* %out3, align 4
; AVX512: LV: Found an estimated cost of 11 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4 ; AVX512: LV: Found an estimated cost of 11 for VF 8 For instruction: store i32 %v3, i32* %out3, align 4
▲ Show 20 Lines • Show All 45 Lines • Show Last 20 Lines