This is an archive of the discontinued LLVM Phabricator instance.

Differential D111012

[X86][Costmodel] Load/store i8 Stride=6 VF=16 interleaving costs
ClosedPublic

Authored by lebedev.ri on Oct 3 2021, 5:11 AM.

Details

Reviewers
RKSimon
Commits
rGbd5ba437fd8f: [X86][Costmodel] Load/store i8 Stride=6 VF=16 interleaving costs
Summary

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/Gz8hhqfTM - for intels Block RThroughput: <=43.0; for ryzens, Block RThroughput: <=14.0
So pick cost of 43.

For store we have:
https://godbolt.org/z/9vrdssYa8 - for intels Block RThroughput: <=27.0; for ryzens, Block RThroughput: <=12.0
So pick cost of 27.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

Diff Detail

Revision Contents

PathSize
llvm/
lib/
Target/
X86/
2 lines
test/
Analysis/
CostModel/
X86/
2 lines
2 lines

Diff 376782

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

Show First 20 Lines • Show All 5,111 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedLoadTbl[] = {
{4, MVT::v4i16, 17}, // (load 16i16 and) deinterleave into 4 x 4i16 {4, MVT::v4i16, 17}, // (load 16i16 and) deinterleave into 4 x 4i16
{4, MVT::v8i16, 33}, // (load 32i16 and) deinterleave into 4 x 8i16 {4, MVT::v8i16, 33}, // (load 32i16 and) deinterleave into 4 x 8i16
{4, MVT::v16i16, 75}, // (load 64i16 and) deinterleave into 4 x 16i16 {4, MVT::v16i16, 75}, // (load 64i16 and) deinterleave into 4 x 16i16
{4, MVT::v32i16, 150}, // (load 128i16 and) deinterleave into 4 x 32i16 {4, MVT::v32i16, 150}, // (load 128i16 and) deinterleave into 4 x 32i16
{6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8 {6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8
{6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8 {6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8
{6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8 {6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8
{6, MVT::v16i8, 43}, // (load 96i8 and) deinterleave into 6 x 16i8
{6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16 {6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16
{6, MVT::v4i16, 9}, // (load 24i16 and) deinterleave into 6 x 4i16 {6, MVT::v4i16, 9}, // (load 24i16 and) deinterleave into 6 x 4i16
{6, MVT::v8i16, 39}, // (load 48i16 and) deinterleave into 6 x 8i16 {6, MVT::v8i16, 39}, // (load 48i16 and) deinterleave into 6 x 8i16
{6, MVT::v16i16, 106}, // (load 96i16 and) deinterleave into 6 x 16i16 {6, MVT::v16i16, 106}, // (load 96i16 and) deinterleave into 6 x 16i16
{8, MVT::v8i32, 40} // (load 64i32 and) deinterleave into 8 x 8i32 {8, MVT::v8i32, 40} // (load 64i32 and) deinterleave into 8 x 8i32
}; };
Show All 38 Lines static const CostTblEntry AVX2InterleavedStoreTbl[] = {
{4, MVT::v4i16, 6}, // interleave 4 x 4i16 into 16i16 (and store) {4, MVT::v4i16, 6}, // interleave 4 x 4i16 into 16i16 (and store)
{4, MVT::v8i16, 10}, // interleave 4 x 8i16 into 32i16 (and store) {4, MVT::v8i16, 10}, // interleave 4 x 8i16 into 32i16 (and store)
{4, MVT::v16i16, 32}, // interleave 4 x 16i16 into 64i16 (and store) {4, MVT::v16i16, 32}, // interleave 4 x 16i16 into 64i16 (and store)
{4, MVT::v32i16, 64}, // interleave 4 x 32i16 into 128i16 (and store) {4, MVT::v32i16, 64}, // interleave 4 x 32i16 into 128i16 (and store)
{6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store) {6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store)
{6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store) {6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store)
{6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store) {6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store)
{6, MVT::v16i8, 27}, // interleave 6 x 16i8 into 96i8 (and store)
{6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store) {6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store)
{6, MVT::v4i16, 15}, // interleave 6 x 4i16 into 24i16 (and store) {6, MVT::v4i16, 15}, // interleave 6 x 4i16 into 24i16 (and store)
{6, MVT::v8i16, 21}, // interleave 6 x 8i16 into 48i16 (and store) {6, MVT::v8i16, 21}, // interleave 6 x 8i16 into 48i16 (and store)
{6, MVT::v16i16, 58}, // interleave 6 x 16i16 into 96i16 (and store) {6, MVT::v16i16, 58}, // interleave 6 x 16i16 into 96i16 (and store)
}; };
if (Opcode == Instruction::Load) { if (Opcode == Instruction::Load) {
▲ Show 20 Lines • Show All 169 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-i8-stride-6.ll

Show All 23 Lines
; AVX1: LV: Found an estimated cost of 114 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX1: LV: Found an estimated cost of 114 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX1: LV: Found an estimated cost of 243 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX1: LV: Found an estimated cost of 243 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX1: LV: Found an estimated cost of 498 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX1: LV: Found an estimated cost of 498 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX2: LV: Found an estimated cost of 9 for VF 2 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 9 for VF 2 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX2: LV: Found an estimated cost of 17 for VF 4 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 17 for VF 4 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX2: LV: Found an estimated cost of 20 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 20 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX2: LV: Found an estimated cost of 243 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 46 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX2: LV: Found an estimated cost of 498 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX2: LV: Found an estimated cost of 498 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX512: LV: Found an estimated cost of 7 for VF 2 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 7 for VF 2 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX512: LV: Found an estimated cost of 25 for VF 4 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 25 for VF 4 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX512: LV: Found an estimated cost of 49 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 49 for VF 8 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX512: LV: Found an estimated cost of 119 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 119 for VF 16 For instruction: %v0 = load i8, i8* %in0, align 1
; AVX512: LV: Found an estimated cost of 237 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1 ; AVX512: LV: Found an estimated cost of 237 for VF 32 For instruction: %v0 = load i8, i8* %in0, align 1
▲ Show 20 Lines • Show All 48 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-i8-stride-6.ll

Show All 23 Lines
; AVX1: LV: Found an estimated cost of 101 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX1: LV: Found an estimated cost of 101 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1
; AVX1: LV: Found an estimated cost of 201 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX1: LV: Found an estimated cost of 201 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1
; AVX1: LV: Found an estimated cost of 498 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX1: LV: Found an estimated cost of 498 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i8 %v5, i8* %out5, align 1
; AVX2: LV: Found an estimated cost of 10 for VF 2 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 10 for VF 2 For instruction: store i8 %v5, i8* %out5, align 1
; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store i8 %v5, i8* %out5, align 1
; AVX2: LV: Found an estimated cost of 18 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 18 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1
; AVX2: LV: Found an estimated cost of 201 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 30 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1
; AVX2: LV: Found an estimated cost of 498 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX2: LV: Found an estimated cost of 498 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i8 %v5, i8* %out5, align 1
; AVX512: LV: Found an estimated cost of 18 for VF 2 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 18 for VF 2 For instruction: store i8 %v5, i8* %out5, align 1
; AVX512: LV: Found an estimated cost of 38 for VF 4 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 38 for VF 4 For instruction: store i8 %v5, i8* %out5, align 1
; AVX512: LV: Found an estimated cost of 98 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 98 for VF 8 For instruction: store i8 %v5, i8* %out5, align 1
; AVX512: LV: Found an estimated cost of 197 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 197 for VF 16 For instruction: store i8 %v5, i8* %out5, align 1
; AVX512: LV: Found an estimated cost of 295 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1 ; AVX512: LV: Found an estimated cost of 295 for VF 32 For instruction: store i8 %v5, i8* %out5, align 1
▲ Show 20 Lines • Show All 49 Lines • Show Last 20 Lines