[X86][Costmodel] Load/store i32 Stride=3 VF=32 interleaving costs

[lebedev.ri]

Summary

A few more tuples are being queried after D111546. Might be good to model them,
They all require a lot of manual assembly surgery.

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/s5b6E6jsP - for intels Block RThroughput: <=32.0; for ryzens, Block RThroughput: <=24.0
So could pick cost of 32

For store we have:
https://godbolt.org/z/efh99d93b - for intels Block RThroughput: <=48.0; for ryzens, Block RThroughput: <=32.0
So we could pick cost of 48.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

[lebedev.ri]

Actually upload the right patch.

[RKSimon]

@lebedev.ri I've created D111960 to see if we can reduce these costs a little - given how manual their testing is, it'd be great to ensure the codegen is reasonable before you do any more work on them.

[lebedev.ri]

In D111942#3068918, @RKSimon wrote:

@lebedev.ri I've created D111960 to see if we can reduce these costs a little - given how manual their testing is, it'd be great to ensure the codegen is reasonable before you do any more work on them.

Sure, but for these strides these patches only finish the few missing pieces,
while after codegen improves we may optionally have to go back and update the costs
and a few more items to update won't really affect things.
The next big batch (stride=5/7/8) can indeed wait a bit for better codegen.

After these, hopefully D111460 lands, and then i think i'd like to decide on discounting not-fully-interleaved loads before finishing with the missing strides.

[RKSimon]

LGTM

[lebedev.ri]

In D111942#3068957, @RKSimon wrote:

LGTM

Thank you for the reviews!

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

static const CostTblEntry AVX2InterleavedLoadTbl[] = {
       {3, MVT::v8i16, 9},   // (load 24i16 and) deinterleave into 3 x 8i16
       {3, MVT::v16i16, 28}, // (load 48i16 and) deinterleave into 3 x 16i16
       {3, MVT::v32i16, 56}, // (load 96i16 and) deinterleave into 3 x 32i16
 
       {3, MVT::v2i32, 3},   // (load 6i32 and) deinterleave into 3 x 2i32
       {3, MVT::v4i32, 3},   // (load 12i32 and) deinterleave into 3 x 4i32
       {3, MVT::v8i32, 7},   // (load 24i32 and) deinterleave into 3 x 8i32
       {3, MVT::v16i32, 14}, // (load 48i32 and) deinterleave into 3 x 16i32
+      {3, MVT::v32i32, 32}, // (load 96i32 and) deinterleave into 3 x 32i32
 
       {3, MVT::v2i64, 1},   // (load 6i64 and) deinterleave into 3 x 2i64
       {3, MVT::v4i64, 5},   // (load 12i64 and) deinterleave into 3 x 4i64
       {3, MVT::v8i64, 10},  // (load 24i64 and) deinterleave into 3 x 8i64
       {3, MVT::v16i64, 20}, // (load 48i64 and) deinterleave into 3 x 16i64
 
       {4, MVT::v2i8, 4},   // (load 8i8 and) deinterleave into 4 x 2i8
       {4, MVT::v4i8, 4},   // (load 16i8 and) deinterleave into 4 x 4i8
static const CostTblEntry AVX2InterleavedStoreTbl[] = {
       {3, MVT::v8i16, 12},  // interleave 3 x 8i16 into 24i16 (and store)
       {3, MVT::v16i16, 27}, // interleave 3 x 16i16 into 48i16 (and store)
       {3, MVT::v32i16, 54}, // interleave 3 x 32i16 into 96i16 (and store)
 
       {3, MVT::v2i32, 4},   // interleave 3 x 2i32 into 6i32 (and store)
       {3, MVT::v4i32, 5},   // interleave 3 x 4i32 into 12i32 (and store)
       {3, MVT::v8i32, 11},  // interleave 3 x 8i32 into 24i32 (and store)
       {3, MVT::v16i32, 22}, // interleave 3 x 16i32 into 48i32 (and store)
+      {3, MVT::v32i32, 48}, // interleave 3 x 32i32 into 96i32 (and store)
 
       {3, MVT::v2i64, 4},   // interleave 3 x 2i64 into 6i64 (and store)
       {3, MVT::v4i64, 6},   // interleave 3 x 4i64 into 12i64 (and store)
       {3, MVT::v8i64, 12},  // interleave 3 x 8i64 into 24i64 (and store)
       {3, MVT::v16i64, 24}, // interleave 3 x 16i64 into 48i64 (and store)
 
       {4, MVT::v2i8, 4},   // interleave 4 x 2i8 into 8i8 (and store)
       {4, MVT::v4i8, 4},   // interleave 4 x 4i8 into 16i8 (and store)

llvm/test/Analysis/CostModel/X86/interleaved-load-f32-stride-3.ll

 ; AVX1: LV: Found an estimated cost of 114 for VF 16 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX1: LV: Found an estimated cost of 228 for VF 32 For instruction:   %v0 = load float, float* %in0, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 5 for VF 4 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 10 for VF 8 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 20 for VF 16 For instruction:   %v0 = load float, float* %in0, align 4
-; AVX2: LV: Found an estimated cost of 228 for VF 32 For instruction:   %v0 = load float, float* %in0, align 4
+; AVX2: LV: Found an estimated cost of 44 for VF 32 For instruction:   %v0 = load float, float* %in0, align 4
 
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 2 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 4 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 6 for VF 8 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 12 for VF 16 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 51 for VF 32 For instruction:   %v0 = load float, float* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 210 for VF 64 For instruction:   %v0 = load float, float* %in0, align 4

llvm/test/Analysis/CostModel/X86/interleaved-load-i32-stride-3-indices-01u.ll

 ; AVX1: LV: Found an estimated cost of 94 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX1: LV: Found an estimated cost of 188 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 5 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 10 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 20 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
-; AVX2: LV: Found an estimated cost of 188 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
+; AVX2: LV: Found an estimated cost of 44 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 5 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 9 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 36 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 144 for VF 64 For instruction:   %v0 = load i32, i32* %in0, align 4

llvm/test/Analysis/CostModel/X86/interleaved-load-i32-stride-3-indices-0uu.ll

 ; AVX1: LV: Found an estimated cost of 50 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX1: LV: Found an estimated cost of 100 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 5 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 10 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 20 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
-; AVX2: LV: Found an estimated cost of 100 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
+; AVX2: LV: Found an estimated cost of 44 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 1 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 1 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 2 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 3 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 21 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 78 for VF 64 For instruction:   %v0 = load i32, i32* %in0, align 4

llvm/test/Analysis/CostModel/X86/interleaved-load-i32-stride-3.ll

 ; AVX1: LV: Found an estimated cost of 138 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX1: LV: Found an estimated cost of 276 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 6 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 5 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 10 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX2: LV: Found an estimated cost of 20 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
-; AVX2: LV: Found an estimated cost of 276 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
+; AVX2: LV: Found an estimated cost of 44 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 2 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 4 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 6 for VF 8 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 12 for VF 16 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 51 for VF 32 For instruction:   %v0 = load i32, i32* %in0, align 4
 ; AVX512: LV: Found an estimated cost of 210 for VF 64 For instruction:   %v0 = load i32, i32* %in0, align 4

llvm/test/Analysis/CostModel/X86/interleaved-store-f32-stride-3.ll

 ; AVX1: LV: Found an estimated cost of 114 for VF 16 For instruction:   store float %v2, float* %out2, align 4
 ; AVX1: LV: Found an estimated cost of 228 for VF 32 For instruction:   store float %v2, float* %out2, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   store float %v2, float* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 7 for VF 2 For instruction:   store float %v2, float* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 7 for VF 4 For instruction:   store float %v2, float* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 14 for VF 8 For instruction:   store float %v2, float* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 28 for VF 16 For instruction:   store float %v2, float* %out2, align 4
-; AVX2: LV: Found an estimated cost of 228 for VF 32 For instruction:   store float %v2, float* %out2, align 4
+; AVX2: LV: Found an estimated cost of 60 for VF 32 For instruction:   store float %v2, float* %out2, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 2 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 4 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 12 for VF 16 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 24 for VF 32 For instruction:   store float %v2, float* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 48 for VF 64 For instruction:   store float %v2, float* %out2, align 4

llvm/test/Analysis/CostModel/X86/interleaved-store-i32-stride-3.ll

 ; AVX1: LV: Found an estimated cost of 138 for VF 16 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX1: LV: Found an estimated cost of 276 for VF 32 For instruction:   store i32 %v2, i32* %out2, align 4
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 7 for VF 2 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 7 for VF 4 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 14 for VF 8 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX2: LV: Found an estimated cost of 28 for VF 16 For instruction:   store i32 %v2, i32* %out2, align 4
-; AVX2: LV: Found an estimated cost of 276 for VF 32 For instruction:   store i32 %v2, i32* %out2, align 4
+; AVX2: LV: Found an estimated cost of 60 for VF 32 For instruction:   store i32 %v2, i32* %out2, align 4
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 2 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 4 for VF 4 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 8 for VF 8 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 12 for VF 16 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 24 for VF 32 For instruction:   store i32 %v2, i32* %out2, align 4
 ; AVX512: LV: Found an estimated cost of 48 for VF 64 For instruction:   store i32 %v2, i32* %out2, align 4