This is an archive of the discontinued LLVM Phabricator instance.

Differential D111076

[X86][Costmodel] Load/store i64/f64 Stride=4 VF=8 interleaving costs
ClosedPublic

Authored by lebedev.ri on Oct 4 2021, 9:58 AM.

Details

Reviewers
RKSimon
Commits
rGc800119c46fb: [X86][Costmodel] Load/store i64/f64 Stride=4 VF=8 interleaving costs
Summary

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/3M3hbq7n8 - for intels Block RThroughput: =20.0; for ryzens, Block RThroughput: =8.0
So could pick cost of 20.

For store we have:
https://godbolt.org/z/zvnPYWTx7 - for intels Block RThroughput: =20.0; for ryzens, Block RThroughput: =8.0
So we could pick cost of 20.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

Diff Detail

Event Timeline

lebedev.ri created this revision.Oct 4 2021, 9:58 AM
Herald added subscribers: pengfei, hiraditya. · View Herald TranscriptOct 4 2021, 9:58 AM
lebedev.ri requested review of this revision.Oct 4 2021, 9:58 AM
lebedev.ri updated this revision to Diff 376933.
lebedev.ri added a parent revision: D111075: [X86][Costmodel] Load/store i64/f64 Stride=4 VF=4 interleaving costs.

Actually upload the right patch.

Harbormaster completed remote builds in B126851: Diff 376933.Oct 4 2021, 12:03 PM
RKSimon accepted this revision.Oct 4 2021, 12:05 PM

LGTM

This revision is now accepted and ready to land.Oct 4 2021, 12:05 PM
This revision was landed with ongoing or failed builds.Oct 5 2021, 7:00 AM
Closed by commit rGc800119c46fb: [X86][Costmodel] Load/store i64/f64 Stride=4 VF=8 interleaving costs (authored by lebedev.ri). · Explain Why
This revision was automatically updated to reflect the committed changes.
lebedev.ri added a commit: rGc800119c46fb: [X86][Costmodel] Load/store i64/f64 Stride=4 VF=8 interleaving costs.

Revision Contents

Diff 377213

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

Show First 20 Lines • Show All 5,129 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedLoadTbl[] = {
{4, MVT::v2i32, 4}, // (load 8i32 and) deinterleave into 4 x 2i32 {4, MVT::v2i32, 4}, // (load 8i32 and) deinterleave into 4 x 2i32
{4, MVT::v4i32, 8}, // (load 16i32 and) deinterleave into 4 x 4i32 {4, MVT::v4i32, 8}, // (load 16i32 and) deinterleave into 4 x 4i32
{4, MVT::v8i32, 16}, // (load 32i32 and) deinterleave into 4 x 8i32 {4, MVT::v8i32, 16}, // (load 32i32 and) deinterleave into 4 x 8i32
{4, MVT::v16i32, 32}, // (load 64i32 and) deinterleave into 4 x 16i32 {4, MVT::v16i32, 32}, // (load 64i32 and) deinterleave into 4 x 16i32
{4, MVT::v2i64, 6}, // (load 8i64 and) deinterleave into 4 x 2i64 {4, MVT::v2i64, 6}, // (load 8i64 and) deinterleave into 4 x 2i64
{4, MVT::v4i64, 8}, // (load 16i64 and) deinterleave into 4 x 4i64 {4, MVT::v4i64, 8}, // (load 16i64 and) deinterleave into 4 x 4i64
{4, MVT::v8i64, 20}, // (load 32i64 and) deinterleave into 4 x 8i64
{6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8 {6, MVT::v2i8, 6}, // (load 12i8 and) deinterleave into 6 x 2i8
{6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8 {6, MVT::v4i8, 14}, // (load 24i8 and) deinterleave into 6 x 4i8
{6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8 {6, MVT::v8i8, 18}, // (load 48i8 and) deinterleave into 6 x 8i8
{6, MVT::v16i8, 43}, // (load 96i8 and) deinterleave into 6 x 16i8 {6, MVT::v16i8, 43}, // (load 96i8 and) deinterleave into 6 x 16i8
{6, MVT::v32i8, 82}, // (load 192i8 and) deinterleave into 6 x 32i8 {6, MVT::v32i8, 82}, // (load 192i8 and) deinterleave into 6 x 32i8
{6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16 {6, MVT::v2i16, 13}, // (load 12i16 and) deinterleave into 6 x 2i16
▲ Show 20 Lines • Show All 64 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedStoreTbl[] = {
{4, MVT::v2i32, 5}, // interleave 4 x 2i32 into 8i32 (and store) {4, MVT::v2i32, 5}, // interleave 4 x 2i32 into 8i32 (and store)
{4, MVT::v4i32, 6}, // interleave 4 x 4i32 into 16i32 (and store) {4, MVT::v4i32, 6}, // interleave 4 x 4i32 into 16i32 (and store)
{4, MVT::v8i32, 16}, // interleave 4 x 8i32 into 32i32 (and store) {4, MVT::v8i32, 16}, // interleave 4 x 8i32 into 32i32 (and store)
{4, MVT::v16i32, 32}, // interleave 4 x 16i32 into 64i32 (and store) {4, MVT::v16i32, 32}, // interleave 4 x 16i32 into 64i32 (and store)
{4, MVT::v2i64, 6}, // interleave 4 x 2i64 into 8i64 (and store) {4, MVT::v2i64, 6}, // interleave 4 x 2i64 into 8i64 (and store)
{4, MVT::v4i64, 8}, // interleave 4 x 4i64 into 16i64 (and store) {4, MVT::v4i64, 8}, // interleave 4 x 4i64 into 16i64 (and store)
{4, MVT::v8i64, 20}, // interleave 4 x 8i64 into 32i64 (and store)
{6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store) {6, MVT::v2i8, 7}, // interleave 6 x 2i8 into 12i8 (and store)
{6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store) {6, MVT::v4i8, 9}, // interleave 6 x 4i8 into 24i8 (and store)
{6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store) {6, MVT::v8i8, 16}, // interleave 6 x 8i8 into 48i8 (and store)
{6, MVT::v16i8, 27}, // interleave 6 x 16i8 into 96i8 (and store) {6, MVT::v16i8, 27}, // interleave 6 x 16i8 into 96i8 (and store)
{6, MVT::v32i8, 90}, // interleave 6 x 32i8 into 192i8 (and store) {6, MVT::v32i8, 90}, // interleave 6 x 32i8 into 192i8 (and store)
{6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store) {6, MVT::v2i16, 10}, // interleave 6 x 2i16 into 12i16 (and store)
▲ Show 20 Lines • Show All 175 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-f64-stride-4.ll

Show All 18 Lines
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8
; AVX1: LV: Found an estimated cost of 14 for VF 2 For instruction: %v0 = load double, double* %in0, align 8 ; AVX1: LV: Found an estimated cost of 14 for VF 2 For instruction: %v0 = load double, double* %in0, align 8
; AVX1: LV: Found an estimated cost of 32 for VF 4 For instruction: %v0 = load double, double* %in0, align 8 ; AVX1: LV: Found an estimated cost of 32 for VF 4 For instruction: %v0 = load double, double* %in0, align 8
; AVX1: LV: Found an estimated cost of 64 for VF 8 For instruction: %v0 = load double, double* %in0, align 8 ; AVX1: LV: Found an estimated cost of 64 for VF 8 For instruction: %v0 = load double, double* %in0, align 8
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8
; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: %v0 = load double, double* %in0, align 8 ; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: %v0 = load double, double* %in0, align 8
; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: %v0 = load double, double* %in0, align 8 ; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: %v0 = load double, double* %in0, align 8
; AVX2: LV: Found an estimated cost of 64 for VF 8 For instruction: %v0 = load double, double* %in0, align 8 ; AVX2: LV: Found an estimated cost of 28 for VF 8 For instruction: %v0 = load double, double* %in0, align 8
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load double, double* %in0, align 8
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load double, double* %in0, align 8
; AVX512: LV: Found an estimated cost of 8 for VF 4 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 8 for VF 4 For instruction: %v0 = load double, double* %in0, align 8
; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: %v0 = load double, double* %in0, align 8
; AVX512: LV: Found an estimated cost of 80 for VF 16 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 80 for VF 16 For instruction: %v0 = load double, double* %in0, align 8
; AVX512: LV: Found an estimated cost of 160 for VF 32 For instruction: %v0 = load double, double* %in0, align 8 ; AVX512: LV: Found an estimated cost of 160 for VF 32 For instruction: %v0 = load double, double* %in0, align 8
; ;
Show All 40 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-i64-stride-4.ll

Show All 18 Lines
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX1: LV: Found an estimated cost of 22 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX1: LV: Found an estimated cost of 22 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX1: LV: Found an estimated cost of 52 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX1: LV: Found an estimated cost of 52 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX1: LV: Found an estimated cost of 104 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX1: LV: Found an estimated cost of 104 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX2: LV: Found an estimated cost of 104 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX2: LV: Found an estimated cost of 28 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX512: LV: Found an estimated cost of 8 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 8 for VF 4 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX512: LV: Found an estimated cost of 80 for VF 16 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 80 for VF 16 For instruction: %v0 = load i64, i64* %in0, align 8
; AVX512: LV: Found an estimated cost of 160 for VF 32 For instruction: %v0 = load i64, i64* %in0, align 8 ; AVX512: LV: Found an estimated cost of 160 for VF 32 For instruction: %v0 = load i64, i64* %in0, align 8
; ;
Show All 40 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-f64-stride-4.ll

Show All 18 Lines
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8
; AVX1: LV: Found an estimated cost of 12 for VF 2 For instruction: store double %v3, double* %out3, align 8 ; AVX1: LV: Found an estimated cost of 12 for VF 2 For instruction: store double %v3, double* %out3, align 8
; AVX1: LV: Found an estimated cost of 32 for VF 4 For instruction: store double %v3, double* %out3, align 8 ; AVX1: LV: Found an estimated cost of 32 for VF 4 For instruction: store double %v3, double* %out3, align 8
; AVX1: LV: Found an estimated cost of 64 for VF 8 For instruction: store double %v3, double* %out3, align 8 ; AVX1: LV: Found an estimated cost of 64 for VF 8 For instruction: store double %v3, double* %out3, align 8
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8
; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: store double %v3, double* %out3, align 8 ; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: store double %v3, double* %out3, align 8
; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store double %v3, double* %out3, align 8 ; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store double %v3, double* %out3, align 8
; AVX2: LV: Found an estimated cost of 64 for VF 8 For instruction: store double %v3, double* %out3, align 8 ; AVX2: LV: Found an estimated cost of 28 for VF 8 For instruction: store double %v3, double* %out3, align 8
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store double %v3, double* %out3, align 8
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store double %v3, double* %out3, align 8
; AVX512: LV: Found an estimated cost of 11 for VF 4 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 11 for VF 4 For instruction: store double %v3, double* %out3, align 8
; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: store double %v3, double* %out3, align 8
; AVX512: LV: Found an estimated cost of 44 for VF 16 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 44 for VF 16 For instruction: store double %v3, double* %out3, align 8
; AVX512: LV: Found an estimated cost of 88 for VF 32 For instruction: store double %v3, double* %out3, align 8 ; AVX512: LV: Found an estimated cost of 88 for VF 32 For instruction: store double %v3, double* %out3, align 8
; ;
▲ Show 20 Lines • Show All 41 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-i64-stride-4.ll

Show All 18 Lines
; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX1: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8
; AVX1: LV: Found an estimated cost of 22 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX1: LV: Found an estimated cost of 22 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8
; AVX1: LV: Found an estimated cost of 52 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX1: LV: Found an estimated cost of 52 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8
; AVX1: LV: Found an estimated cost of 104 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX1: LV: Found an estimated cost of 104 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8
; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX2: LV: Found an estimated cost of 8 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8
; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX2: LV: Found an estimated cost of 12 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8
; AVX2: LV: Found an estimated cost of 104 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX2: LV: Found an estimated cost of 28 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i64 %v3, i64* %out3, align 8
; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 5 for VF 2 For instruction: store i64 %v3, i64* %out3, align 8
; AVX512: LV: Found an estimated cost of 11 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 11 for VF 4 For instruction: store i64 %v3, i64* %out3, align 8
; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 22 for VF 8 For instruction: store i64 %v3, i64* %out3, align 8
; AVX512: LV: Found an estimated cost of 44 for VF 16 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 44 for VF 16 For instruction: store i64 %v3, i64* %out3, align 8
; AVX512: LV: Found an estimated cost of 88 for VF 32 For instruction: store i64 %v3, i64* %out3, align 8 ; AVX512: LV: Found an estimated cost of 88 for VF 32 For instruction: store i64 %v3, i64* %out3, align 8
; ;
▲ Show 20 Lines • Show All 41 Lines • Show Last 20 Lines