This is an archive of the discontinued LLVM Phabricator instance.

Differential D110755

[X86][Costmodel] Load/store i32/f32 Stride=2 VF=8 interleaving costs
ClosedPublic

Authored by lebedev.ri on Sep 29 2021, 12:36 PM.

Details

Reviewers
RKSimon
Commits
rG3a0643e9c225: [X86][Costmodel] Load/store i32/f32 Stride=2 VF=8 interleaving costs
Summary

The only sched models that for cpu's that support avx2
but not avx512 are: haswell, broadwell, skylake, zen1-3

For load we have:
https://godbolt.org/z/n8aMKeo4E - for intels Block RThroughput: =4.0; for ryzens, Block RThroughput: <=2.0
So pick cost of 4.

For store we have:
https://godbolt.org/z/n8aMKeo4E - for intels Block RThroughput: =4.0; for ryzens, Block RThroughput: =2.0
So pick cost of 4.

I'm directly using the shuffling asm the llc produced,
without any manual fixups that may be needed
to ensure sequential execution.

Diff Detail

Event Timeline

lebedev.ri created this revision.Sep 29 2021, 12:36 PM
Herald added subscribers: pengfei, hiraditya. · View Herald TranscriptSep 29 2021, 12:36 PM
lebedev.ri requested review of this revision.Sep 29 2021, 12:36 PM
lebedev.ri added a parent revision: D110754: [X86][Costmodel] Load/store i32/f32 Stride=2 VF=4 interleaving costs.
lebedev.ri added a child revision: D110756: [X86][Costmodel] Load/store i32/f32 Stride=2 VF=16 interleaving costs.Sep 29 2021, 12:49 PM
Harbormaster completed remote builds in B126408: Diff 375993.Sep 29 2021, 12:58 PM
lebedev.ri updated this revision to Diff 376161.Sep 30 2021, 4:33 AM
lebedev.ri updated this revision to Diff 376162.
Harbormaster completed remote builds in B126527: Diff 376162.Sep 30 2021, 4:34 AM
lebedev.ri updated this revision to Diff 376164.Sep 30 2021, 4:34 AM
Harbormaster completed remote builds in B126530: Diff 376164.Sep 30 2021, 5:34 AM
RKSimon accepted this revision.Oct 1 2021, 6:08 AM

LGTM

This revision is now accepted and ready to land.Oct 1 2021, 6:08 AM
This revision was landed with ongoing or failed builds.Oct 1 2021, 7:49 AM
Closed by commit rG3a0643e9c225: [X86][Costmodel] Load/store i32/f32 Stride=2 VF=8 interleaving costs (authored by lebedev.ri). · Explain Why
This revision was automatically updated to reflect the committed changes.
lebedev.ri added a commit: rG3a0643e9c225: [X86][Costmodel] Load/store i32/f32 Stride=2 VF=8 interleaving costs.

Revision Contents

PathSize
llvm/
lib/
Target/
X86/
2 lines
test/
Analysis/
CostModel/
X86/
2 lines
2 lines
2 lines
2 lines
Transforms/
LoopVectorize/
X86/
70 lines

Diff 376532

llvm/lib/Target/X86/X86TargetTransformInfo.cpp

Show First 20 Lines • Show All 5,071 Lines • ▼ Show 20 Lines static const CostTblEntry AVX2InterleavedLoadTbl[] = {
{2, MVT::v2i16, 2}, // (load 4i16 and) deinterleave into 2 x 2i16 {2, MVT::v2i16, 2}, // (load 4i16 and) deinterleave into 2 x 2i16
{2, MVT::v4i16, 2}, // (load 8i16 and) deinterleave into 2 x 4i16 {2, MVT::v4i16, 2}, // (load 8i16 and) deinterleave into 2 x 4i16
{2, MVT::v8i16, 6}, // (load 16i16 and) deinterleave into 2 x 8i16 {2, MVT::v8i16, 6}, // (load 16i16 and) deinterleave into 2 x 8i16
{2, MVT::v16i16, 9}, // (load 32i16 and) deinterleave into 2 x 16i16 {2, MVT::v16i16, 9}, // (load 32i16 and) deinterleave into 2 x 16i16
{2, MVT::v32i16, 18}, // (load 64i16 and) deinterleave into 2 x 32i16 {2, MVT::v32i16, 18}, // (load 64i16 and) deinterleave into 2 x 32i16
{2, MVT::v2i32, 2}, // (load 4i32 and) deinterleave into 2 x 2i32 {2, MVT::v2i32, 2}, // (load 4i32 and) deinterleave into 2 x 2i32
{2, MVT::v4i32, 2}, // (load 8i32 and) deinterleave into 2 x 4i32 {2, MVT::v4i32, 2}, // (load 8i32 and) deinterleave into 2 x 4i32
{2, MVT::v8i32, 4}, // (load 16i32 and) deinterleave into 2 x 8i32
{2, MVT::v4i64, 6}, // (load 8i64 and) deinterleave into 2 x 4i64 {2, MVT::v4i64, 6}, // (load 8i64 and) deinterleave into 2 x 4i64
{3, MVT::v2i8, 10}, // (load 6i8 and) deinterleave into 3 x 2i8 {3, MVT::v2i8, 10}, // (load 6i8 and) deinterleave into 3 x 2i8
{3, MVT::v4i8, 4}, // (load 12i8 and) deinterleave into 3 x 4i8 {3, MVT::v4i8, 4}, // (load 12i8 and) deinterleave into 3 x 4i8
{3, MVT::v8i8, 9}, // (load 24i8 and) deinterleave into 3 x 8i8 {3, MVT::v8i8, 9}, // (load 24i8 and) deinterleave into 3 x 8i8
{3, MVT::v16i8, 11}, // (load 48i8 and) deinterleave into 3 x 16i8 {3, MVT::v16i8, 11}, // (load 48i8 and) deinterleave into 3 x 16i8
{3, MVT::v32i8, 13}, // (load 96i8 and) deinterleave into 3 x 32i8 {3, MVT::v32i8, 13}, // (load 96i8 and) deinterleave into 3 x 32i8
Show All 30 Lines static const CostTblEntry AVX2InterleavedStoreTbl[] = {
{2, MVT::v2i16, 1}, // interleave 2 x 2i16 into 4i16 (and store) {2, MVT::v2i16, 1}, // interleave 2 x 2i16 into 4i16 (and store)
{2, MVT::v4i16, 1}, // interleave 2 x 4i16 into 8i16 (and store) {2, MVT::v4i16, 1}, // interleave 2 x 4i16 into 8i16 (and store)
{2, MVT::v8i16, 3}, // interleave 2 x 8i16 into 16i16 (and store) {2, MVT::v8i16, 3}, // interleave 2 x 8i16 into 16i16 (and store)
{2, MVT::v16i16, 4}, // interleave 2 x 16i16 into 32i16 (and store) {2, MVT::v16i16, 4}, // interleave 2 x 16i16 into 32i16 (and store)
{2, MVT::v32i16, 8}, // interleave 2 x 32i16 into 64i16 (and store) {2, MVT::v32i16, 8}, // interleave 2 x 32i16 into 64i16 (and store)
{2, MVT::v2i32, 1}, // interleave 2 x 2i32 into 4i32 (and store) {2, MVT::v2i32, 1}, // interleave 2 x 2i32 into 4i32 (and store)
{2, MVT::v4i32, 2}, // interleave 2 x 4i32 into 8i32 (and store) {2, MVT::v4i32, 2}, // interleave 2 x 4i32 into 8i32 (and store)
{2, MVT::v8i32, 4}, // interleave 2 x 8i32 into 16i32 (and store)
{2, MVT::v4i64, 6}, // interleave 2 x 4i64 into 8i64 (and store) {2, MVT::v4i64, 6}, // interleave 2 x 4i64 into 8i64 (and store)
{3, MVT::v2i8, 7}, // interleave 3 x 2i8 into 6i8 (and store) {3, MVT::v2i8, 7}, // interleave 3 x 2i8 into 6i8 (and store)
{3, MVT::v4i8, 8}, // interleave 3 x 4i8 into 12i8 (and store) {3, MVT::v4i8, 8}, // interleave 3 x 4i8 into 12i8 (and store)
{3, MVT::v8i8, 11}, // interleave 3 x 8i8 into 24i8 (and store) {3, MVT::v8i8, 11}, // interleave 3 x 8i8 into 24i8 (and store)
{3, MVT::v16i8, 11}, // interleave 3 x 16i8 into 48i8 (and store) {3, MVT::v16i8, 11}, // interleave 3 x 16i8 into 48i8 (and store)
{3, MVT::v32i8, 13}, // interleave 3 x 32i8 into 96i8 (and store) {3, MVT::v32i8, 13}, // interleave 3 x 32i8 into 96i8 (and store)
▲ Show 20 Lines • Show All 189 Lines • Show Last 20 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-f32-stride-2.ll

Show All 22 Lines
; AVX1: LV: Found an estimated cost of 17 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 17 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 38 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 38 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 76 for VF 16 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 76 for VF 16 For instruction: %v0 = load float, float* %in0, align 4
; AVX1: LV: Found an estimated cost of 152 for VF 32 For instruction: %v0 = load float, float* %in0, align 4 ; AVX1: LV: Found an estimated cost of 152 for VF 32 For instruction: %v0 = load float, float* %in0, align 4
;; ;;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 38 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 76 for VF 16 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 76 for VF 16 For instruction: %v0 = load float, float* %in0, align 4
; AVX2: LV: Found an estimated cost of 152 for VF 32 For instruction: %v0 = load float, float* %in0, align 4 ; AVX2: LV: Found an estimated cost of 152 for VF 32 For instruction: %v0 = load float, float* %in0, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 8 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 8 For instruction: %v0 = load float, float* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: %v0 = load float, float* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: %v0 = load float, float* %in0, align 4
Show All 35 Lines

llvm/test/Analysis/CostModel/X86/interleaved-load-i32-stride-2.ll

Show All 22 Lines
; AVX1: LV: Found an estimated cost of 21 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 21 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 46 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 46 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 92 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 92 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX1: LV: Found an estimated cost of 184 for VF 32 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX1: LV: Found an estimated cost of 184 for VF 32 For instruction: %v0 = load i32, i32* %in0, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 46 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 92 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 92 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX2: LV: Found an estimated cost of 184 for VF 32 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX2: LV: Found an estimated cost of 184 for VF 32 For instruction: %v0 = load i32, i32* %in0, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 2 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 4 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 3 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 3 for VF 8 For instruction: %v0 = load i32, i32* %in0, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: %v0 = load i32, i32* %in0, align 4
Show All 35 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-f32-stride-2.ll

Show All 22 Lines
; AVX1: LV: Found an estimated cost of 15 for VF 4 For instruction: store float %v1, float* %out1, align 4 ; AVX1: LV: Found an estimated cost of 15 for VF 4 For instruction: store float %v1, float* %out1, align 4
; AVX1: LV: Found an estimated cost of 38 for VF 8 For instruction: store float %v1, float* %out1, align 4 ; AVX1: LV: Found an estimated cost of 38 for VF 8 For instruction: store float %v1, float* %out1, align 4
; AVX1: LV: Found an estimated cost of 76 for VF 16 For instruction: store float %v1, float* %out1, align 4 ; AVX1: LV: Found an estimated cost of 76 for VF 16 For instruction: store float %v1, float* %out1, align 4
; AVX1: LV: Found an estimated cost of 152 for VF 32 For instruction: store float %v1, float* %out1, align 4 ; AVX1: LV: Found an estimated cost of 152 for VF 32 For instruction: store float %v1, float* %out1, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v1, float* %out1, align 4
; AVX2: LV: Found an estimated cost of 2 for VF 2 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 2 for VF 2 For instruction: store float %v1, float* %out1, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: store float %v1, float* %out1, align 4
; AVX2: LV: Found an estimated cost of 38 for VF 8 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 8 For instruction: store float %v1, float* %out1, align 4
; AVX2: LV: Found an estimated cost of 76 for VF 16 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 76 for VF 16 For instruction: store float %v1, float* %out1, align 4
; AVX2: LV: Found an estimated cost of 152 for VF 32 For instruction: store float %v1, float* %out1, align 4 ; AVX2: LV: Found an estimated cost of 152 for VF 32 For instruction: store float %v1, float* %out1, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v1, float* %out1, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store float %v1, float* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 2 For instruction: store float %v1, float* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 2 For instruction: store float %v1, float* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 4 For instruction: store float %v1, float* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 4 For instruction: store float %v1, float* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 8 For instruction: store float %v1, float* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 8 For instruction: store float %v1, float* %out1, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: store float %v1, float* %out1, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: store float %v1, float* %out1, align 4
Show All 36 Lines

llvm/test/Analysis/CostModel/X86/interleaved-store-i32-stride-2.ll

Show All 22 Lines
; AVX1: LV: Found an estimated cost of 19 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX1: LV: Found an estimated cost of 19 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4
; AVX1: LV: Found an estimated cost of 46 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX1: LV: Found an estimated cost of 46 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4
; AVX1: LV: Found an estimated cost of 92 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX1: LV: Found an estimated cost of 92 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4
; AVX1: LV: Found an estimated cost of 184 for VF 32 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX1: LV: Found an estimated cost of 184 for VF 32 For instruction: store i32 %v1, i32* %out1, align 4
; ;
; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v1, i32* %out1, align 4
; AVX2: LV: Found an estimated cost of 2 for VF 2 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 2 for VF 2 For instruction: store i32 %v1, i32* %out1, align 4
; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 3 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4
; AVX2: LV: Found an estimated cost of 46 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 6 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4
; AVX2: LV: Found an estimated cost of 92 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 92 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4
; AVX2: LV: Found an estimated cost of 184 for VF 32 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX2: LV: Found an estimated cost of 184 for VF 32 For instruction: store i32 %v1, i32* %out1, align 4
; ;
; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX512: LV: Found an estimated cost of 1 for VF 1 For instruction: store i32 %v1, i32* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 2 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 2 For instruction: store i32 %v1, i32* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 4 For instruction: store i32 %v1, i32* %out1, align 4
; AVX512: LV: Found an estimated cost of 2 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX512: LV: Found an estimated cost of 2 for VF 8 For instruction: store i32 %v1, i32* %out1, align 4
; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4 ; AVX512: LV: Found an estimated cost of 5 for VF 16 For instruction: store i32 %v1, i32* %out1, align 4
Show All 36 Lines

llvm/test/Transforms/LoopVectorize/X86/interleaving.ll

Show First 20 Lines • Show All 58 Lines • ▼ Show 20 Lines
; AVX2-NEXT: entry: ; AVX2-NEXT: entry:
; AVX2-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]] ; AVX2-NEXT: br i1 false, label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
; AVX2: vector.ph: ; AVX2: vector.ph:
; AVX2-NEXT: br label [[VECTOR_BODY:%.*]] ; AVX2-NEXT: br label [[VECTOR_BODY:%.*]]
; AVX2: vector.body: ; AVX2: vector.body:
; AVX2-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ] ; AVX2-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
; AVX2-NEXT: [[TMP0:%.*]] = shl nsw i64 [[INDEX]], 1 ; AVX2-NEXT: [[TMP0:%.*]] = shl nsw i64 [[INDEX]], 1
; AVX2-NEXT: [[TMP1:%.*]] = shl i64 [[INDEX]], 1 ; AVX2-NEXT: [[TMP1:%.*]] = shl i64 [[INDEX]], 1
; AVX2-NEXT: [[TMP2:%.*]] = or i64 [[TMP1]], 8 ; AVX2-NEXT: [[TMP2:%.*]] = or i64 [[TMP1]], 16
; AVX2-NEXT: [[TMP3:%.*]] = shl i64 [[INDEX]], 1 ; AVX2-NEXT: [[TMP3:%.*]] = shl i64 [[INDEX]], 1
; AVX2-NEXT: [[TMP4:%.*]] = or i64 [[TMP3]], 16 ; AVX2-NEXT: [[TMP4:%.*]] = or i64 [[TMP3]], 32
; AVX2-NEXT: [[TMP5:%.*]] = shl i64 [[INDEX]], 1 ; AVX2-NEXT: [[TMP5:%.*]] = shl i64 [[INDEX]], 1
; AVX2-NEXT: [[TMP6:%.*]] = or i64 [[TMP5]], 24 ; AVX2-NEXT: [[TMP6:%.*]] = or i64 [[TMP5]], 48
; AVX2-NEXT: [[TMP7:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[TMP0]] ; AVX2-NEXT: [[TMP7:%.*]] = getelementptr inbounds i32, i32* [[B:%.*]], i64 [[TMP0]]
; AVX2-NEXT: [[TMP8:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP2]] ; AVX2-NEXT: [[TMP8:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP2]]
; AVX2-NEXT: [[TMP9:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP4]] ; AVX2-NEXT: [[TMP9:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP4]]
; AVX2-NEXT: [[TMP10:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP6]] ; AVX2-NEXT: [[TMP10:%.*]] = getelementptr inbounds i32, i32* [[B]], i64 [[TMP6]]
; AVX2-NEXT: [[TMP11:%.*]] = bitcast i32* [[TMP7]] to <8 x i32>* ; AVX2-NEXT: [[TMP11:%.*]] = bitcast i32* [[TMP7]] to <16 x i32>*
; AVX2-NEXT: [[TMP12:%.*]] = bitcast i32* [[TMP8]] to <8 x i32>* ; AVX2-NEXT: [[TMP12:%.*]] = bitcast i32* [[TMP8]] to <16 x i32>*
; AVX2-NEXT: [[TMP13:%.*]] = bitcast i32* [[TMP9]] to <8 x i32>* ; AVX2-NEXT: [[TMP13:%.*]] = bitcast i32* [[TMP9]] to <16 x i32>*
; AVX2-NEXT: [[TMP14:%.*]] = bitcast i32* [[TMP10]] to <8 x i32>* ; AVX2-NEXT: [[TMP14:%.*]] = bitcast i32* [[TMP10]] to <16 x i32>*
; AVX2-NEXT: [[WIDE_VEC:%.*]] = load <8 x i32>, <8 x i32>* [[TMP11]], align 4 ; AVX2-NEXT: [[WIDE_VEC:%.*]] = load <16 x i32>, <16 x i32>* [[TMP11]], align 4
; AVX2-NEXT: [[WIDE_VEC1:%.*]] = load <8 x i32>, <8 x i32>* [[TMP12]], align 4 ; AVX2-NEXT: [[WIDE_VEC1:%.*]] = load <16 x i32>, <16 x i32>* [[TMP12]], align 4
; AVX2-NEXT: [[WIDE_VEC2:%.*]] = load <8 x i32>, <8 x i32>* [[TMP13]], align 4 ; AVX2-NEXT: [[WIDE_VEC2:%.*]] = load <16 x i32>, <16 x i32>* [[TMP13]], align 4
; AVX2-NEXT: [[WIDE_VEC3:%.*]] = load <8 x i32>, <8 x i32>* [[TMP14]], align 4 ; AVX2-NEXT: [[WIDE_VEC3:%.*]] = load <16 x i32>, <16 x i32>* [[TMP14]], align 4
; AVX2-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6> ; AVX2-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <16 x i32> [[WIDE_VEC]], <16 x i32> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
; AVX2-NEXT: [[STRIDED_VEC4:%.*]] = shufflevector <8 x i32> [[WIDE_VEC1]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6> ; AVX2-NEXT: [[STRIDED_VEC4:%.*]] = shufflevector <16 x i32> [[WIDE_VEC1]], <16 x i32> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
; AVX2-NEXT: [[STRIDED_VEC5:%.*]] = shufflevector <8 x i32> [[WIDE_VEC2]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6> ; AVX2-NEXT: [[STRIDED_VEC5:%.*]] = shufflevector <16 x i32> [[WIDE_VEC2]], <16 x i32> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
; AVX2-NEXT: [[STRIDED_VEC6:%.*]] = shufflevector <8 x i32> [[WIDE_VEC3]], <8 x i32> poison, <4 x i32> <i32 0, i32 2, i32 4, i32 6> ; AVX2-NEXT: [[STRIDED_VEC6:%.*]] = shufflevector <16 x i32> [[WIDE_VEC3]], <16 x i32> poison, <8 x i32> <i32 0, i32 2, i32 4, i32 6, i32 8, i32 10, i32 12, i32 14>
; AVX2-NEXT: [[STRIDED_VEC7:%.*]] = shufflevector <8 x i32> [[WIDE_VEC]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7> ; AVX2-NEXT: [[STRIDED_VEC7:%.*]] = shufflevector <16 x i32> [[WIDE_VEC]], <16 x i32> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
; AVX2-NEXT: [[STRIDED_VEC8:%.*]] = shufflevector <8 x i32> [[WIDE_VEC1]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7> ; AVX2-NEXT: [[STRIDED_VEC8:%.*]] = shufflevector <16 x i32> [[WIDE_VEC1]], <16 x i32> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
; AVX2-NEXT: [[STRIDED_VEC9:%.*]] = shufflevector <8 x i32> [[WIDE_VEC2]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7> ; AVX2-NEXT: [[STRIDED_VEC9:%.*]] = shufflevector <16 x i32> [[WIDE_VEC2]], <16 x i32> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
; AVX2-NEXT: [[STRIDED_VEC10:%.*]] = shufflevector <8 x i32> [[WIDE_VEC3]], <8 x i32> poison, <4 x i32> <i32 1, i32 3, i32 5, i32 7> ; AVX2-NEXT: [[STRIDED_VEC10:%.*]] = shufflevector <16 x i32> [[WIDE_VEC3]], <16 x i32> poison, <8 x i32> <i32 1, i32 3, i32 5, i32 7, i32 9, i32 11, i32 13, i32 15>
; AVX2-NEXT: [[TMP15:%.*]] = add nsw <4 x i32> [[STRIDED_VEC7]], [[STRIDED_VEC]] ; AVX2-NEXT: [[TMP15:%.*]] = add nsw <8 x i32> [[STRIDED_VEC7]], [[STRIDED_VEC]]
; AVX2-NEXT: [[TMP16:%.*]] = add nsw <4 x i32> [[STRIDED_VEC8]], [[STRIDED_VEC4]] ; AVX2-NEXT: [[TMP16:%.*]] = add nsw <8 x i32> [[STRIDED_VEC8]], [[STRIDED_VEC4]]
; AVX2-NEXT: [[TMP17:%.*]] = add nsw <4 x i32> [[STRIDED_VEC9]], [[STRIDED_VEC5]] ; AVX2-NEXT: [[TMP17:%.*]] = add nsw <8 x i32> [[STRIDED_VEC9]], [[STRIDED_VEC5]]
; AVX2-NEXT: [[TMP18:%.*]] = add nsw <4 x i32> [[STRIDED_VEC10]], [[STRIDED_VEC6]] ; AVX2-NEXT: [[TMP18:%.*]] = add nsw <8 x i32> [[STRIDED_VEC10]], [[STRIDED_VEC6]]
; AVX2-NEXT: [[TMP19:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]] ; AVX2-NEXT: [[TMP19:%.*]] = getelementptr inbounds i32, i32* [[A:%.*]], i64 [[INDEX]]
; AVX2-NEXT: [[TMP20:%.*]] = bitcast i32* [[TMP19]] to <4 x i32>* ; AVX2-NEXT: [[TMP20:%.*]] = bitcast i32* [[TMP19]] to <8 x i32>*
; AVX2-NEXT: store <4 x i32> [[TMP15]], <4 x i32>* [[TMP20]], align 4 ; AVX2-NEXT: store <8 x i32> [[TMP15]], <8 x i32>* [[TMP20]], align 4
; AVX2-NEXT: [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 4 ; AVX2-NEXT: [[TMP21:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 8
; AVX2-NEXT: [[TMP22:%.*]] = bitcast i32* [[TMP21]] to <4 x i32>* ; AVX2-NEXT: [[TMP22:%.*]] = bitcast i32* [[TMP21]] to <8 x i32>*
; AVX2-NEXT: store <4 x i32> [[TMP16]], <4 x i32>* [[TMP22]], align 4 ; AVX2-NEXT: store <8 x i32> [[TMP16]], <8 x i32>* [[TMP22]], align 4
; AVX2-NEXT: [[TMP23:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 8 ; AVX2-NEXT: [[TMP23:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 16
; AVX2-NEXT: [[TMP24:%.*]] = bitcast i32* [[TMP23]] to <4 x i32>* ; AVX2-NEXT: [[TMP24:%.*]] = bitcast i32* [[TMP23]] to <8 x i32>*
; AVX2-NEXT: store <4 x i32> [[TMP17]], <4 x i32>* [[TMP24]], align 4 ; AVX2-NEXT: store <8 x i32> [[TMP17]], <8 x i32>* [[TMP24]], align 4
; AVX2-NEXT: [[TMP25:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 12 ; AVX2-NEXT: [[TMP25:%.*]] = getelementptr inbounds i32, i32* [[TMP19]], i64 24
; AVX2-NEXT: [[TMP26:%.*]] = bitcast i32* [[TMP25]] to <4 x i32>* ; AVX2-NEXT: [[TMP26:%.*]] = bitcast i32* [[TMP25]] to <8 x i32>*
; AVX2-NEXT: store <4 x i32> [[TMP18]], <4 x i32>* [[TMP26]], align 4 ; AVX2-NEXT: store <8 x i32> [[TMP18]], <8 x i32>* [[TMP26]], align 4
; AVX2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 16 ; AVX2-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 32
; AVX2-NEXT: [[TMP27:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024 ; AVX2-NEXT: [[TMP27:%.*]] = icmp eq i64 [[INDEX_NEXT]], 1024
; AVX2-NEXT: br i1 [[TMP27]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]] ; AVX2-NEXT: br i1 [[TMP27]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
; AVX2: middle.block: ; AVX2: middle.block:
; AVX2-NEXT: br i1 true, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]] ; AVX2-NEXT: br i1 true, label [[FOR_COND_CLEANUP:%.*]], label [[SCALAR_PH]]
; AVX2: scalar.ph: ; AVX2: scalar.ph:
; AVX2-NEXT: br label [[FOR_BODY:%.*]] ; AVX2-NEXT: br label [[FOR_BODY:%.*]]
; AVX2: for.cond.cleanup: ; AVX2: for.cond.cleanup:
; AVX2-NEXT: ret void ; AVX2-NEXT: ret void
Show All 24 Lines