[X86][Costmodel] Load/store i8 Stride=2 VF=32 interleaving costs

[lebedev.ri]

Summary

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/xz6x7c35P - for intels Block RThroughput: =6.0; for ryzens, Block RThroughput: <=2.5
So pick cost of 6.

For store we have:
https://godbolt.org/z/xz6x7c35P - for intels Block RThroughput: =4.0; for ryzens, Block RThroughput: <=2.0
So pick cost of 4.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

[RKSimon]

LGTM

[lebedev.ri]

In D110709#3030657, @RKSimon wrote:

LGTM

Thank you for the reviews!
Are there issues with VF=16 (D110708)?

After this, i will need to check double check but i believe, i will look into stride=2 for i32 and i64.
(i'm not just mass-adding these, at least so far, but only those that are queried in real code, and are missing)

[RKSimon]

In D110709#3031066, @lebedev.ri wrote:

In D110709#3030657, @RKSimon wrote:

LGTM

Thank you for the reviews!
Are there issues with VF=16 (D110708)?

Sorry I missed that one!

After this, i will need to check double check but i believe, i will look into stride=2 for i32 and i64.
(i'm not just mass-adding these, at least so far, but only those that are queried in real code, and are missing)

Improving i32/i64 + f32/f64 test + costs coverage would be great.

Do you happen to know the relationship between these cost files and the interleaved test files in codegen\x86? They also have poor target coverage.

[lebedev.ri]

In D110709#3031248, @RKSimon wrote:

In D110709#3031066, @lebedev.ri wrote:

In D110709#3030657, @RKSimon wrote:

LGTM

Thank you for the reviews!
Are there issues with VF=16 (D110708)?

Sorry I missed that one!

After this, i will need to check double check but i believe, i will look into stride=2 for i32 and i64.
(i'm not just mass-adding these, at least so far, but only those that are queried in real code, and are missing)

Improving i32/i64 + f32/f64 test + costs coverage would be great.

Do you happen to know the relationship between these cost files and the interleaved test files in codegen\x86? They also have poor target coverage.

Define relationship? I just add new consistent test coverage (both codegen and costmodel) following a consistent naming model,
so we already have that coverage elsewhere, it should be removed in favor of the new coverage.
As for codegen tests, yes, there i only test AVX2 currently. We could extend it to test other ISA's.

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

InstructionCost X86TTIImpl::getInterleavedMemoryOpCostAVX2(
   // Factor (stride) and VectorType=VFxiN.
   // The Cost accounts only for the shuffle sequence;
   // The cost of the loads/stores is accounted for separately.
   //
   static const CostTblEntry AVX2InterleavedLoadTbl[] = {
       {2, MVT::v2i8, 2},  // (load 4i8 and) deinterleave into 2 x 2i8
       {2, MVT::v4i8, 2},  // (load 8i8 and) deinterleave into 2 x 4i8
       {2, MVT::v8i8, 2},  // (load 16i8 and) deinterleave into 2 x 8i8
       {2, MVT::v16i8, 4},  // (load 32i8 and) deinterleave into 2 x 16i8
+      {2, MVT::v32i8, 6},  // (load 64i8 and) deinterleave into 2 x 32i8
 
       {2, MVT::v2i16, 2}, // (load 4i16 and) deinterleave into 2 x 2i16
       {2, MVT::v4i16, 2}, // (load 8i16 and) deinterleave into 2 x 4i16
       {2, MVT::v8i16, 6}, // (load 16i16 and) deinterleave into 2 x 8i16
       {2, MVT::v16i16, 9}, // (load 32i16 and) deinterleave into 2 x 16i16
       {2, MVT::v32i16, 18}, // (load 64i16 and) deinterleave into 2 x 32i16
 
       {2, MVT::v4i64, 6}, // (load 8i64 and) deinterleave into 2 x 4i64
 
       {3, MVT::v2i8, 10},  // (load 6i8 and) deinterleave into 3 x 2i8
       {3, MVT::v4i8, 4},   // (load 12i8 and) deinterleave into 3 x 4i8
       {3, MVT::v8i8, 9},   // (load 24i8 and) deinterleave into 3 x 8i8
       {3, MVT::v16i8, 11}, // (load 48i8 and) deinterleave into 3 x 16i8
       {3, MVT::v32i8, 13}, // (load 96i8 and) deinterleave into 3 x 32i8
 
       {3, MVT::v8i32, 17}, // (load 24i32 and) deinterleave into 3 x 8i32
 
       {4, MVT::v2i8, 12},  // (load 8i8 and) deinterleave into 4 x 2i8
       {4, MVT::v4i8, 4},   // (load 16i8 and) deinterleave into 4 x 4i8
       {4, MVT::v8i8, 20},  // (load 32i8 and) deinterleave into 4 x 8i8
       {4, MVT::v16i8, 39}, // (load 64i8 and) deinterleave into 4 x 16i8
       {4, MVT::v32i8, 80}, // (load 128i8 and) deinterleave into 4 x 32i8
 
       {4, MVT::v2i16, 6}, // (load 8i16 and) deinterleave into 4 x 2i16
       {4, MVT::v4i16, 17}, // (load 16i16 and) deinterleave into 4 x 4i16
       {4, MVT::v8i16, 33}, // (load 32i16 and) deinterleave into 4 x 8i16
       {4, MVT::v16i16, 75}, // (load 64i16 and) deinterleave into 4 x 16i16
       {4, MVT::v32i16, 150}, // (load 128i16 and) deinterleave into 4 x 32i16
 
       {6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16
       {6, MVT::v4i16, 9}, // (load 24i16 and) deinterleave into 6 x 4i16
       {6, MVT::v8i16, 39}, // (load 48i16 and) deinterleave into 6 x 8i16
       {6, MVT::v16i16, 106}, // (load 96i16 and) deinterleave into 6 x 16i16
 
       {8, MVT::v8i32, 40} // (load 64i32 and) deinterleave into 8 x 8i32
   };
 
   static const CostTblEntry AVX2InterleavedStoreTbl[] = {
       {2, MVT::v2i8, 1}, // interleave 2 x 2i8 into 4i8 (and store)
       {2, MVT::v4i8, 1}, // interleave 2 x 4i8 into 8i8 (and store)
       {2, MVT::v8i8, 1}, // interleave 2 x 8i8 into 16i8 (and store)
       {2, MVT::v16i8, 3}, // interleave 2 x 16i8 into 32i8 (and store)
+      {2, MVT::v32i8, 4}, // interleave 2 x 32i8 into 64i8 (and store)
 
       {2, MVT::v2i16, 1}, // interleave 2 x 2i16 into 4i16 (and store)
       {2, MVT::v4i16, 1}, // interleave 2 x 4i16 into 8i16 (and store)
       {2, MVT::v8i16, 3}, // interleave 2 x 8i16 into 16i16 (and store)
       {2, MVT::v16i16, 4}, // interleave 2 x 16i16 into 32i16 (and store)
       {2, MVT::v32i16, 8}, // interleave 2 x 32i16 into 64i16 (and store)
 
       {2, MVT::v4i64, 6}, // interleave 2 x 4i64 into 8i64 (and store)
 
       {3, MVT::v2i8, 7},   // interleave 3 x 2i8 into 6i8 (and store)
       {3, MVT::v4i8, 8},   // interleave 3 x 4i8 into 12i8 (and store)
       {3, MVT::v8i8, 11},  // interleave 3 x 8i8 into 24i8 (and store)
       {3, MVT::v16i8, 11}, // interleave 3 x 16i8 into 48i8 (and store)
       {3, MVT::v32i8, 13}, // interleave 3 x 32i8 into 96i8 (and store)
 
       {4, MVT::v2i8, 12},  // interleave 4 x 2i8 into 8i8 (and store)
       {4, MVT::v4i8, 9},   // interleave 4 x 4i8 into 16i8 (and store)
       {4, MVT::v8i8, 10},  // interleave 4 x 8i8 into 32i8 (and store)
       {4, MVT::v16i8, 10}, // interleave 4 x 16i8 into 64i8 (and store)
       {4, MVT::v32i8, 12}, // interleave 4 x 32i8 into 128i8 (and store)
 
       {4, MVT::v2i16, 2},  // interleave 4 x 2i16 into 8i16 (and store)
       {4, MVT::v4i16, 6},  // interleave 4 x 4i16 into 16i16 (and store)
       {4, MVT::v8i16, 10},  // interleave 4 x 8i16 into 32i16 (and store)
       {4, MVT::v16i16, 32},  // interleave 4 x 16i16 into 64i16 (and store)
       {4, MVT::v32i16, 64},  // interleave 4 x 32i16 into 128i16 (and store)
 
       {6, MVT::v2i16, 10},  // interleave 6 x 2i16 into 12i16 (and store)
       {6, MVT::v4i16, 15},  // interleave 6 x 4i16 into 24i16 (and store)
       {6, MVT::v8i16, 21},  // interleave 6 x 8i16 into 48i16 (and store)
       {6, MVT::v16i16, 58},  // interleave 6 x 16i16 into 96i16 (and store)
   };
 
   if (Opcode == Instruction::Load) {
     if (const auto *Entry =
             CostTableLookup(AVX2InterleavedLoadTbl, Factor, ETy.getSimpleVT()))
       return MemOpCosts + Entry->Cost;
   } else {
     assert(Opcode == Instruction::Store &&

llvm/test/Analysis/CostModel/X86/interleaved-load-i8-stride-2.ll

 ; AVX1: LV: Found an estimated cost of 81 for VF 16 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX1: LV: Found an estimated cost of 226 for VF 32 For instruction:   %v0 = load i8, i8* %in0, align 1
 ;
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX2: LV: Found an estimated cost of 3 for VF 2 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX2: LV: Found an estimated cost of 3 for VF 8 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX2: LV: Found an estimated cost of 5 for VF 16 For instruction:   %v0 = load i8, i8* %in0, align 1
-; AVX2: LV: Found an estimated cost of 226 for VF 32 For instruction:   %v0 = load i8, i8* %in0, align 1
+; AVX2: LV: Found an estimated cost of 8 for VF 32 For instruction:   %v0 = load i8, i8* %in0, align 1
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 3 for VF 8 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 9 for VF 16 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 17 for VF 32 For instruction:   %v0 = load i8, i8* %in0, align 1
 ; AVX512: LV: Found an estimated cost of 41 for VF 64 For instruction:   %v0 = load i8, i8* %in0, align 1

llvm/test/Analysis/CostModel/X86/interleaved-store-i8-stride-2.ll

 ; AVX1: LV: Found an estimated cost of 97 for VF 16 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX1: LV: Found an estimated cost of 226 for VF 32 For instruction:   store i8 %v1, i8* %out1, align 1
 
 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX2: LV: Found an estimated cost of 2 for VF 2 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX2: LV: Found an estimated cost of 2 for VF 4 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX2: LV: Found an estimated cost of 2 for VF 8 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX2: LV: Found an estimated cost of 4 for VF 16 For instruction:   store i8 %v1, i8* %out1, align 1
-; AVX2: LV: Found an estimated cost of 226 for VF 32 For instruction:   store i8 %v1, i8* %out1, align 1
+; AVX2: LV: Found an estimated cost of 6 for VF 32 For instruction:   store i8 %v1, i8* %out1, align 1
 ;
 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 4 for VF 2 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 4 for VF 4 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 4 for VF 8 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 8 for VF 16 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 20 for VF 32 For instruction:   store i8 %v1, i8* %out1, align 1
 ; AVX512: LV: Found an estimated cost of 41 for VF 64 For instruction:   store i8 %v1, i8* %out1, align 1