Page MenuHomePhabricator
Differential D110978

[SLP]Improve gathering of the scalars used in the graph.
Needs ReviewPublic
Actions

Authored by ABataev on Oct 1 2021, 4:10 PM.

Details

Reviewers
RKSimon
spatel
dtemirbulatov
anton-afanasyev
Summary

Currently we emit gathers for scalars being vectorized in the tre as
a pair of extractelement/insertelement instructions. Instead we can try
to find all required vectors and emit shuffle vector instructions
directly, improving the code and reducing compile time.

Diff Detail

Unit TestsFailed
View All

TimeTest
140 msx64 debian > Flang.Driver::code-gen-rv64.f90
Script: -- : 'RUN: at line 5'; rm -f /var/lib/buildkite-agent/builds/llvm-project/build/tools/flang/test/Driver/Output/code-gen-rv64.f90.tmp.o

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ABataev updated this revision to Diff 390702.Nov 30 2021, 6:49 AM

Rebase

Harbormaster completed remote builds in B136694: Diff 390702.Nov 30 2021, 8:08 AM
ABataev updated this revision to Diff 390783.Nov 30 2021, 12:26 PM

Rebase

Harbormaster completed remote builds in B136747: Diff 390783.Nov 30 2021, 1:09 PM
ABataev updated this revision to Diff 392842.Dec 8 2021, 11:21 AM

Rebase

Harbormaster completed remote builds in B138215: Diff 392842.Dec 8 2021, 12:09 PM
ABataev updated this revision to Diff 394249.Dec 14 2021, 7:47 AM

Rebase

RKSimon added inline comments.Dec 14 2021, 8:04 AM
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
6813

Wshadow warning vs Idx @ Line 4688?

6846

Wshadow warning vs Idx @ Line 4688?

ABataev updated this revision to Diff 394269.Dec 14 2021, 8:33 AM

Address comments

Harbormaster completed remote builds in B139236: Diff 394269.Dec 14 2021, 9:48 AM
ABataev updated this revision to Diff 396715.Dec 30 2021, 1:33 PM

Rebase

Harbormaster completed remote builds in B141051: Diff 396715.Dec 30 2021, 2:15 PM
ABataev mentioned this in D123587: [SLP] Generate shuffles if we can reorder an existing node.Apr 12 2022, 12:05 PM
ABataev updated this revision to Diff 455910.Aug 26 2022, 7:51 AM

Rebase

Herald added a project: Restricted Project. · View Herald TranscriptAug 26 2022, 7:51 AM
Herald added subscribers: pcwang-thead, nlopes, kosarev. · View Herald Transcript
nlopes added inline comments.Aug 26 2022, 7:54 AM
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
9873

Please use PoisonValue whenever possible. It seems this is just a placeholder, so it can be switched.
Thank you!

ABataev added inline comments.Aug 26 2022, 8:08 AM
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
9873

Sure, thanks!

ABataev updated this revision to Diff 455933.Aug 26 2022, 8:52 AM

Address comments

Harbormaster completed remote builds in B183623: Diff 455933.Aug 26 2022, 10:50 AM
ABataev updated this revision to Diff 459790.Sep 13 2022, 10:00 AM

Rebase

Harbormaster completed remote builds in B186399: Diff 459790.Sep 13 2022, 11:19 AM
ABataev mentioned this in rG796af0c02728: [SLP] Move getInsertIndex function, NFC..Sep 14 2022, 6:24 AM
ABataev mentioned this in rGd647312e3f57: [SLP][NFC]Extract getLastInstructionInBundle function for better.Sep 14 2022, 8:44 AM
ABataev updated this revision to Diff 468668.Oct 18 2022, 12:43 PM

Rebase

Harbormaster completed remote builds in B192832: Diff 468668.Oct 18 2022, 1:42 PM
nhaehnle removed a subscriber: nhaehnle.Oct 19 2022, 2:00 AM
ABataev updated this revision to Diff 480583.Dec 6 2022, 12:45 PM

Large update.
Includes:

  1. Unifies all shuffle builders and shuffle demission operands.
  2. Generalizes emission and cost model estimation of the buildvectors/gathers.

Will be splitted into several smaller patches eventually.

Harbormaster completed remote builds in B201460: Diff 480583.Dec 6 2022, 9:34 PM
ABataev mentioned this in D139718: [SLP][NFC]Inital redesign of ShuffleInstructionBuilder, NFC..Dec 9 2022, 7:50 AM
ABataev mentioned this in rGecac8192dbf6: [SLP][NFC]Initial redesign of ShuffleInstructionBuilder, NFC..Dec 13 2022, 9:54 AM
ABataev updated this revision to Diff 482594.Dec 13 2022, 12:36 PM

Rebase

Harbormaster completed remote builds in B202927: Diff 482594.Dec 13 2022, 1:17 PM
ABataev updated this revision to Diff 482619.Dec 13 2022, 1:32 PM

Restore accidentally removed code.

Harbormaster completed remote builds in B202945: Diff 482619.Dec 13 2022, 2:43 PM
ABataev updated this revision to Diff 484571.Dec 21 2022, 7:02 AM

Rebase

Harbormaster completed remote builds in B204383: Diff 484571.Dec 21 2022, 7:50 AM
ABataev mentioned this in D140499: [SLP]Use ShuffleInstructionBuilder for vector shrinking..Dec 21 2022, 1:54 PM
khchen added a subscriber: khchen.Dec 22 2022, 8:35 AM
ABataev mentioned this in rGac01ae71f0c4: [SLP]Use ShuffleInstructionBuilder for vector shrinking..Dec 28 2022, 6:11 AM
ABataev updated this revision to Diff 486895.Jan 6 2023, 8:42 AM

Rebase

Harbormaster completed remote builds in B206131: Diff 486895.Jan 6 2023, 10:07 AM
ABataev updated this revision to Diff 487485.Jan 9 2023, 9:43 AM

Rebase

Herald added a subscriber: StephenFan. · View Herald TranscriptJan 9 2023, 9:43 AM
Harbormaster completed remote builds in B206577: Diff 487485.Jan 9 2023, 10:30 AM
ABataev mentioned this in D141512: [SLP]Improve isGatherShuffledEntry by looking deeper through the reused scalars..Jan 11 2023, 8:33 AM
ABataev mentioned this in D141940: [SLP]Add shuffling of extractelements to avoid extra costs/data movement..Jan 17 2023, 8:01 AM
ABataev mentioned this in rG9bdcf8778a5c: [SLP]Improve isGatherShuffledEntry by looking deeper through the reused scalars..Jan 19 2023, 1:50 PM
ABataev mentioned this in rG708eb1b96d9a: [SLP]Add shuffling of extractelements to avoid extra costs/data movement..Feb 20 2023, 6:16 AM
ABataev mentioned this in D144958: [SLP]Initial support for reshuffling of non-starting buildvector/gather nodes..Tue, Feb 28, 5:21 AM
ABataev mentioned this in rGa611b3f3059e: [SLP]Initial support for reshuffling of non-starting buildvector/gather nodes..Tue, Mar 7, 12:47 PM
ABataev updated this revision to Diff 503493.Wed, Mar 8, 1:19 PM

Rebase

ABataev updated this revision to Diff 503510.Wed, Mar 8, 1:56 PM

Restore deleted code/update test

Harbormaster completed remote builds in B218206: Diff 503510.Wed, Mar 8, 2:48 PM
ABataev mentioned this in D145732: [SLP][NFC]Initial merge of gather/buildvector code in the createBuildVector function..Thu, Mar 9, 2:20 PM
hans mentioned this in rG3b3a4c270bcb: Revert "[SLP]Initial support for reshuffling of non-starting buildvector/gather….Fri, Mar 10, 5:40 AM
ABataev mentioned this in rG93a9be0cea0a: [SLP]Initial support for reshuffling of non-starting buildvector/gather nodes..Fri, Mar 10, 1:22 PM
ABataev mentioned this in rGf3a68ac10c84: [SLP][NFC]Initial merge of gather/buildvector code in the createBuildVector….Mon, Mar 13, 6:27 AM
ABataev updated this revision to Diff 504816.Mon, Mar 13, 12:49 PM

Rebase

RKSimon added inline comments.Mon, Mar 13, 2:27 PM
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
6731

Any chance that we can use ShuffleVectorInst::isIdentityMask ?

7180

auto *

7182

auto *

ABataev added inline comments.Mon, Mar 13, 2:42 PM
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
6731

Sure, will do it later

7182

Both these cases are the existing code, just the diff is not quite correct because of the big differences.

ABataev updated this revision to Diff 504861.Mon, Mar 13, 3:10 PM

Restore accidentally removed lines, address comments

Harbormaster completed remote builds in B219182: Diff 504861.Mon, Mar 13, 5:18 PM
ABataev updated this revision to Diff 505461.Wed, Mar 15, 6:11 AM

Rebase

ABataev updated this revision to Diff 505467.Wed, Mar 15, 6:16 AM

Restore some deleted code

Harbormaster completed remote builds in B219617: Diff 505467.Wed, Mar 15, 7:08 AM
ABataev mentioned this in D146167: [SLP]Introduce shuffle of the nodes + gather/vectorbuild of the remaining scalars..Wed, Mar 15, 2:14 PM
ABataev mentioned this in rG0ad87ffdcc23: [SLP]Introduce shuffle of the nodes + gather/vectorbuild of the remaining….Fri, Mar 17, 11:21 AM
ABataev updated this revision to Diff 506162.Fri, Mar 17, 11:54 AM

Rebase

Harbormaster completed remote builds in B220124: Diff 506162.Fri, Mar 17, 12:55 PM
ABataev mentioned this in D146564: [SLP]Find reused scalars in buildvector sequences, if any..Tue, Mar 21, 2:11 PM

Revision Contents
Changeset List

PathSize
llvm/
lib/
Transforms/
Vectorize/
1252 lines
test/
DebugInfo/
Generic/
assignment-tracking/
slp-vectorizer/
7 lines
Transforms/
SLPVectorizer/
AArch64/
36 lines
634 lines
23 lines
X86/
20 lines
20 lines
38 lines
38 lines
29 lines
8 lines
2 lines
2 lines
46 lines
21 lines
27 lines
24 lines
20 lines
2 lines
2 lines
13 lines

Diff 506162

View Options

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

Loading...
View Options

llvm/test/DebugInfo/Generic/assignment-tracking/slp-vectorizer/merge-scalars.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/AArch64/extractelements-to-shuffle.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/AArch64/loadorder.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/AArch64/tsc-s116.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/alternate-int-inseltpoison.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/alternate-int.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/arith-fp-inseltpoison.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/arith-fp.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/commutativity.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/crash_lencod.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/crash_netbsd_decompress.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/entries-different-vf.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/lookahead.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/matched-shuffled-entries.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/pr47623.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/reduced-gathered-vectorized.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/redux-feed-buildvector.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/remark_extract_broadcast.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/scatter-vectorize-reorder.ll

Loading...
View Options

llvm/test/Transforms/SLPVectorizer/X86/shrink_after_reorder.ll

Loading...