[SLP]Generalize cost model.

[ABataev]

Summary

Generalized the cost model estimation. Improved cost model estimation
for repeated scalars (no need to count their cost anymore), improved

cost model for extractelement instructions.

cpu2017

511.povray_r             0.57
520.omnetpp_r           -0.98
521.wrf_r               -0.01
525.x264_r               3.59 <+
526.blender_r           -0.12
531.deepsjeng_r         -0.07
538.imagick_r           -1.42

Geometric mean: 0.21

[ABataev]

Rebase

[ABataev]

Rebase

[ABataev]

Rebase

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5893–5896

I->getOperand(OpIdx) == Op0;

5900–5907

auto *Op = I->getOperand(OpIdx);

5902

true?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5902

Here is my idea. Generally speaking, here we can have only undef/poison or just a pointer (as part of getelementpr node) currently (maybe, need to add an assert). I thought that because of that we can treat its (virtual) operand as power-of-2, since it can be any value. Thoughts?

[ABataev]

Address comments

[ABataev]

Rebase

[ABataev]

Ping!

[ABataev]

Rebase. Ping!

[ABataev]

Ping!

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

Is this purely compile time saving?

6259

Can't we directly initialize Operands from PHI-> incoming_values()? Or even create a ArrayRef<> from it?

6307

We're going to lose the recent scalarization cost improvement here :-(

6545

Would isa<UnaryOperator> ? 0 : 1 be better?

6595

Why do you avoid allowing uniform as well? Can't you just use getOperandInfo(VL, 0)..getNoProps() below ?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

Replaced by 0, since it does not matter.

6259

operand_value returns llvm::Use, Operands list requires Value *. Did not find better way to do it.

6595

Just missed this change, will fix.

[ABataev]

Rebase, address comments

[ABataev]

Ping!

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5902

Still not sure about this - this undef case is fine, but the GEP I can imagine might not work in all special cases.

How good is return isa<UndefValue>(V); instead?

5916

return isa<UndefValue>(V)?

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6504

Why didn't we ever use getGEPCost?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6504

It originally used add cost for GEPs, most probably there were getGEPCost. Plus, I don't see target-specific getGEPCost implementations, looks like we have only base one.
We could try to switch to getGEPCost but it must be in a separate patch.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5902

GetElementPtr may have just a pointer, which is not a GEP instructions.
Say, we have this:

void @foo(ptr %v) {
...
load %v
%n = getelementptr i32, ptr v, 1
load %n
...
}

and we may have a ПУЗ node which includes {%v, %n}. Here %v is not a GEP though still part of the GEP node and we consider as gentlementptr ptr %v, 0.

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5902

OK - so for GEPs you're saying that either the operands will be constant or a phi of constants?

6262

I'm sorry but I had to revert the ScalarizationOverheadBuilder code - this needs reverting back to the older version

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5902

No (or I missed something). I mean, that except for GEPs we may have just a non-ПУЗ pointer value and we just treat it as getelementptr, 0. But we return true if we have just a value (which is GEP with offset 0) and all other GEPs are power-of-2 constant ints.

[ABataev]

Rebase

[ABataev]

Rebase

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6438

So CommonCost is only applied if (!MinBWs.count(VL0) || VecTy != SrcVecTy) ?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6438

Yeah, looks like should not ignore it here, since it includes the shuffle cost.

[ABataev]

Address comments

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

Do you have a specific reason to keep this as lambda(lambda(...),lambda(...)) design?
I mean this would look much better if you make one more step and outline this into a separate class (for example CostDifferentiator) with dedicated methods instead of stuffing code with tons of lambdas.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

It will still have lambdas for each particular opcode

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

Why? It is not prohibited to have dedicated method in the class for each particular opcode.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

It still will be a switch with functions calls, which have the pretty similar loops and other processing. This what exactly I want to avoid - duplicate of the code.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

I can't agree with statement that it will be a duplication of code. Organized differently - yes.
IMO it's always worth investing time in developing a better interface.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

I don't quite understand what you're proposing. Provide at least a small example.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6221

Just a sketch:

sample.txt6 KBDownload

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

I'm not sure if we need to pull them out into separate methods if they're just being called once. But it would help a little for readability if the lambdas weren't embedded in the GetCostDiff calls themselves.

So:

auto ScalarEltCost = [](unsigned Idx)  {...};
auto VectorCost = [](InstructionCost CommonCost)  {...};
return GetCostDiff(ScalarEltCost, VectorCost);

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

I prefer this approach. I don't want to introduce virtual member functions. Without virtual functions the only approach - generic lambda, which increases codesize.

[ABataev]

Address comments, formatting

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

What problem you see with virtual class here?
Using named lambdas instead of unnamed ones is definitely better. But essentially it anyway remains a C-style programming, just using C++ syntax. You are also making it very difficult to customize this code downstream. If that is the goal - okay, got it.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

It usually increases compile time.
Why it is hard to customize it?

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

It usually increases compile time.

Use of C++ has its costs. I'm pretty sure the increase will be unmeasurable.
I believe that if invest a bit more time into this something better can be found.

Why it is hard to customize it?

because this will require replacing a lambda rather than just make a descendant with custom functionality that might fully or partially reuse its base. I'm not saying it is impossible to do that with lambdas. It will just be significantly more ugly: create another lambda with the original one in capture list...

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

I though about templated lambda instead of virtual class, but it will implicitly copied across all cases.

[RKSimon]

LGTM

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

6231

I don't want to stop this conversation as I'd dearly like to see the SLP code be easier to understand as its complexity/density puts off a lot of people from contributing - but we need somewhere better to discuss this than on a patch review thread.

6247–6254

It looks like theres a clang-format change across this entire switch statement that ideally needs pre-committing to reduce the patch whitespace/indentation diff?

[fmayer]

This seems to have an MSAN error: https://lab.llvm.org/buildbot/#/builders/74/builds/14338/steps/13/logs/stdio

FAIL: LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll (57172 of 68894)
******************** TEST 'LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt < /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll -slp-vectorizer -dce -S -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7 | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
--
Exit Code: 2
Command Output (stderr):
--
==3579288==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x56270dbbe918 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19
    #1 0x56270dbbe918 in llvm::InstructionCost llvm::function_ref<llvm::InstructionCost (unsigned int)>::callback_fn<llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_63>(long, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #2 0x56270dafc992 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #3 0x56270dafc992 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_58::operator()(llvm::function_ref<llvm::InstructionCost (unsigned int)>, llvm::function_ref<llvm::InstructionCost (llvm::InstructionCost)>) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6336:29
    #4 0x56270daeeb36 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
    #5 0x56270db033d8 in llvm::slpvectorizer::BoUpSLP::getTreeCost(llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:7273:25
    #6 0x56270db4bb38 in llvm::SLPVectorizerPass::vectorizeStoreChain(llvm::ArrayRef<llvm::Value*>, llvm::slpvectorizer::BoUpSLP&, unsigned int, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10379:28
    #7 0x56270db4eadb in llvm::SLPVectorizerPass::vectorizeStores(llvm::ArrayRef<llvm::StoreInst*>, llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10521:13
    #8 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12807:18
    #9 0x56270db4279c in callback_fn<(lambda at /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12806:9)> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #10 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #11 0x56270db4279c in tryToVectorizeSequence<llvm::StoreInst> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12192:9
    #12 0x56270db4279c in llvm::SLPVectorizerPass::vectorizeStoreChains(llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12804:16
    #13 0x56270db3f9f7 in llvm::SLPVectorizerPass::runImpl(llvm::Function&, llvm::ScalarEvolution*, llvm::TargetTransformInfo*, llvm::TargetLibraryInfo*, llvm::AAResults*, llvm::LoopInfo*, llvm::DominatorTree*, llvm::AssumptionCache*, llvm::DemandedBits*, llvm::OptimizationRemarkEmitter*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10332:18
    #14 0x56270db3e0b0 in llvm::SLPVectorizerPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10267:18
    #15 0x56270ddc2a21 in llvm::detail::PassModel<llvm::Function, llvm::SLPVectorizerPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #16 0x56270bf43ae0 in llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #17 0x56270779e201 in llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #18 0x56270bf4fc31 in llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/IR/PassManager.cpp:124:38
    #19 0x56270779db71 in llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #20 0x56270bf40fb0 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #21 0x562706b90baf in llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/NewPMDriver.cpp:497:7
    #22 0x562706bc0377 in main /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/opt.cpp:784:12
    #23 0x7ff7be184d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #24 0x7ff7be184e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #25 0x562706af7664 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt+0x386a664)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19 in operator()
Exiting
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll

[ABataev]

In D115757#3866816, @fmayer wrote:

This seems to have an MSAN error: https://lab.llvm.org/buildbot/#/builders/74/builds/14338/steps/13/logs/stdio

FAIL: LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll (57172 of 68894)
******************** TEST 'LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt < /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll -slp-vectorizer -dce -S -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7 | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
--
Exit Code: 2
Command Output (stderr):
--
==3579288==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x56270dbbe918 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19
    #1 0x56270dbbe918 in llvm::InstructionCost llvm::function_ref<llvm::InstructionCost (unsigned int)>::callback_fn<llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_63>(long, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #2 0x56270dafc992 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #3 0x56270dafc992 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_58::operator()(llvm::function_ref<llvm::InstructionCost (unsigned int)>, llvm::function_ref<llvm::InstructionCost (llvm::InstructionCost)>) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6336:29
    #4 0x56270daeeb36 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
    #5 0x56270db033d8 in llvm::slpvectorizer::BoUpSLP::getTreeCost(llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:7273:25
    #6 0x56270db4bb38 in llvm::SLPVectorizerPass::vectorizeStoreChain(llvm::ArrayRef<llvm::Value*>, llvm::slpvectorizer::BoUpSLP&, unsigned int, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10379:28
    #7 0x56270db4eadb in llvm::SLPVectorizerPass::vectorizeStores(llvm::ArrayRef<llvm::StoreInst*>, llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10521:13
    #8 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12807:18
    #9 0x56270db4279c in callback_fn<(lambda at /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12806:9)> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #10 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #11 0x56270db4279c in tryToVectorizeSequence<llvm::StoreInst> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12192:9
    #12 0x56270db4279c in llvm::SLPVectorizerPass::vectorizeStoreChains(llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12804:16
    #13 0x56270db3f9f7 in llvm::SLPVectorizerPass::runImpl(llvm::Function&, llvm::ScalarEvolution*, llvm::TargetTransformInfo*, llvm::TargetLibraryInfo*, llvm::AAResults*, llvm::LoopInfo*, llvm::DominatorTree*, llvm::AssumptionCache*, llvm::DemandedBits*, llvm::OptimizationRemarkEmitter*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10332:18
    #14 0x56270db3e0b0 in llvm::SLPVectorizerPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10267:18
    #15 0x56270ddc2a21 in llvm::detail::PassModel<llvm::Function, llvm::SLPVectorizerPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #16 0x56270bf43ae0 in llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #17 0x56270779e201 in llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #18 0x56270bf4fc31 in llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/IR/PassManager.cpp:124:38
    #19 0x56270779db71 in llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #20 0x56270bf40fb0 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #21 0x562706b90baf in llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/NewPMDriver.cpp:497:7
    #22 0x562706bc0377 in main /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/opt.cpp:784:12
    #23 0x7ff7be184d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #24 0x7ff7be184e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #25 0x562706af7664 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt+0x386a664)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19 in operator()
Exiting
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll

It was already reverted and then committed back with a fix.

[fmayer]

In D115757#3866833, @ABataev wrote:

It was already reverted and then committed back with a fix.

Sorry about that, I missed the revert line, I only skimmed the comments. Thanks!

[LuoYuanke]

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

[ABataev]

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

[LuoYuanke]

In D115757#3958922, @ABataev wrote:

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

Do you have any plan to fix it?

[ABataev]

In D115757#3959221, @LuoYuanke wrote:

In D115757#3958922, @ABataev wrote:

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

Do you have any plan to fix it?

A plan of improving SLP vectorizer, not this particular regression. It will be fixed eventually

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (auto *CI0 = dyn_cast<CmpInst>(MainOp)) {
     return AltP0 == CurrentPred || AltP0Swapped == CurrentPred;
   }
   return I->getOpcode() == AltOp->getOpcode();
 }
 
 TTI::OperandValueInfo BoUpSLP::getOperandInfo(ArrayRef<Value *> VL,
                                               unsigned OpIdx) {
   assert(!VL.empty());
-  const auto *Op0 = cast<Instruction>(VL.front())->getOperand(OpIdx);
+  const auto *I0 = cast<Instruction>(*find_if(VL, Instruction::classof));
+  const auto *Op0 = I0->getOperand(OpIdx);
 
   const bool IsConstant = all_of(VL, [&](Value *V) {
     // TODO: We should allow undef elements here
-    auto *Op = cast<Instruction>(V)->getOperand(OpIdx);
+    const auto *I = dyn_cast<Instruction>(V);
+    if (!I)
+      return true;
+    auto *Op = I->getOperand(OpIdx);
     return isConstant(Op) && !isa<UndefValue>(Op);
   });
   const bool IsUniform = all_of(VL, [&](Value *V) {
     // TODO: We should allow undef elements here
-    return cast<Instruction>(V)->getOperand(OpIdx) == Op0;
+    const auto *I = dyn_cast<Instruction>(V);
+    if (!I)
+      return false;
+    return I->getOperand(OpIdx) == Op0;
   });
   const bool IsPowerOfTwo = all_of(VL, [&](Value *V) {
     // TODO: We should allow undef elements here
-    auto *Op = cast<Instruction>(V)->getOperand(OpIdx);
+    const auto *I = dyn_cast<Instruction>(V);
+    if (!I) {
+      assert((isa<UndefValue>(V) ||
+              I0->getOpcode() == Instruction::GetElementPtr) &&
+             "Expected undef or GEP.");
+      return true;
+    }
+    auto *Op = I->getOperand(OpIdx);
     if (auto *CI = dyn_cast<ConstantInt>(Op))
       return CI->getValue().isPowerOf2();
     return false;
   });
 
   TTI::OperandValueKind VK = TTI::OK_AnyValue;
   if (IsConstant && IsUniform)
     VK = TTI::OK_UniformConstantValue;
   else if (IsConstant)
     VK = TTI::OK_NonUniformConstantValue;
   else if (IsUniform)
     VK = TTI::OK_UniformValue;
 
   const TTI::OperandValueProperties VP =
     IsPowerOfTwo ? TTI::OP_PowerOf2 : TTI::OP_None;
   return {VK, VP};
 }
InstructionCost BoUpSLP::getEntryCost(const TreeEntry *E,
   assert(E->getOpcode() &&
          ((allSameType(VL) && allSameBlock(VL)) ||
           (E->getOpcode() == Instruction::GetElementPtr &&
            E->getMainOp()->getType()->isPointerTy())) &&
          "Invalid VL");
   Instruction *VL0 = E->getMainOp();
   unsigned ShuffleOrOp =
       E->isAltShuffle() ? (unsigned)Instruction::ShuffleVector : E->getOpcode();
+  auto &&GetCostDiff =
+      [this, E, VL, CommonCost,
+       VL0](function_ref<InstructionCost(unsigned)> ScalarEltCost,
+            function_ref<InstructionCost(InstructionCost)> VectorCost) {
+        // Calculate the cost of this instruction.
+        InstructionCost ScalarCost = 0;
+        if (isa<CastInst, CmpInst, SelectInst, CallInst>(VL0)) {
+          // For some of the instructions no need to calculate cost for each
+          // particular instruction, we can use the cost of the single
+          // instruction x total number of scalar instructions.
+          ScalarCost = VL.size() *
+                       ScalarEltCost(std::distance(VL.begin(), find(VL, VL0)));
+        } else {
+          for (unsigned I = 0, Sz = VL.size(); I < Sz; ++I)
+            ScalarCost += ScalarEltCost(I);
+        }
+
+        InstructionCost VecCost = VectorCost(CommonCost);
+        LLVM_DEBUG(
+            dumpTreeCosts(E, CommonCost, VecCost - CommonCost, ScalarCost));
+        // Disable warnings for `this` and `E` are unused. Required for
+        // `dumpTreeCosts`.
+        (void)this;
+        (void)E;
+        return VecCost - ScalarCost;
+      };
   switch (ShuffleOrOp) {
-    case Instruction::PHI:
-      return 0;
+  case Instruction::PHI: {
+    // Count reused scalars.
+    InstructionCost ScalarCost = 0;
+    SmallPtrSet<const TreeEntry *, 4> CountedOps;
+    for (Value *V : VL) {
+      auto *PHI = dyn_cast<PHINode>(V);
+      if (!PHI)
+        continue;
 
-    case Instruction::ExtractValue:
-    case Instruction::ExtractElement: {
-      // The common cost of removal ExtractElement/ExtractValue instructions +
-      // the cost of shuffles, if required to resuffle the original vector.
-      if (NeedToShuffleReuses) {
-        unsigned Idx = 0;
-        for (unsigned I : E->ReuseShuffleIndices) {
-          if (ShuffleOrOp == Instruction::ExtractElement) {
-            auto *EE = cast<ExtractElementInst>(VL[I]);
-            CommonCost -= TTI->getVectorInstrCost(
-                *EE, EE->getVectorOperandType(), *getExtractIndex(EE));
-          } else {
-            CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
-            ++Idx;
-          }
-        }
-        Idx = EntryVF;
-        for (Value *V : VL) {
+      ValueList Operands(PHI->getNumIncomingValues(), nullptr);
+      for (unsigned I = 0, N = PHI->getNumIncomingValues(); I < N; ++I) {
+        Value *Op = PHI->getIncomingValue(I);
+        Operands[I] = Op;
+      if (const TreeEntry *OpTE = getTreeEntry(Operands.front()))
+        if (OpTE->isSame(Operands) && CountedOps.insert(OpTE).second)
+          if (!OpTE->ReuseShuffleIndices.empty())
+            ScalarCost += TTI::TCC_Basic * (OpTE->ReuseShuffleIndices.size() -
+                                            OpTE->Scalars.size());
+
+    return CommonCost - ScalarCost;
+  }
+    case Instruction::ExtractValue:
+    case Instruction::ExtractElement:
+      return GetCostDiff(
+          [this, ShuffleOrOp, VL, ScalarTy, CostKind](unsigned Idx) {
+            auto *I = cast<Instruction>(VL[Idx]);
+            VectorType *SrcVecTy;
             if (ShuffleOrOp == Instruction::ExtractElement) {
-            auto *EE = cast<ExtractElementInst>(V);
-            CommonCost += TTI->getVectorInstrCost(
-                *EE, EE->getVectorOperandType(), *getExtractIndex(EE));
+              auto *EE = cast<ExtractElementInst>(I);
+              SrcVecTy = EE->getVectorOperandType();
             } else {
-            --Idx;
-            CommonCost += TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
-          }
-        }
+              auto *EV = cast<ExtractValueInst>(I);
+              Type *AggregateTy = EV->getAggregateOperand()->getType();
+              unsigned NumElts;
+              if (auto *ATy = dyn_cast<ArrayType>(AggregateTy))
+                NumElts = ATy->getNumElements();
+              else
+                NumElts = AggregateTy->getStructNumElements();
+              SrcVecTy = FixedVectorType::get(ScalarTy, NumElts);
             }
-      if (ShuffleOrOp == Instruction::ExtractValue) {
-        for (unsigned I = 0, E = VL.size(); I < E; ++I) {
-          auto *EI = cast<Instruction>(VL[I]);
-          // Take credit for instruction that will become dead.
-          if (EI->hasOneUse()) {
-            Instruction *Ext = EI->user_back();
-            if (isa<SExtInst, ZExtInst>(Ext) &&
+            if (I->hasOneUse()) {
+              Instruction *Ext = I->user_back();
+              if ((isa<SExtInst>(Ext) || isa<ZExtInst>(Ext)) &&
                   all_of(Ext->users(),
                          [](User *U) { return isa<GetElementPtrInst>(U); })) {
                 // Use getExtractWithExtendCost() to calculate the cost of
                 // extractelement/ext pair.
-            CommonCost -= TTI->getExtractWithExtendCost(
-                Ext->getOpcode(), Ext->getType(), VecTy, I);
-            // Add back the cost of s|zext which is subtracted separately.
-            CommonCost += TTI->getCastInstrCost(
-                Ext->getOpcode(), Ext->getType(), EI->getType(),
+                InstructionCost Cost = TTI->getExtractWithExtendCost(
+                    Ext->getOpcode(), Ext->getType(), SrcVecTy,
+                    *getExtractIndex(I));
+                // Subtract the cost of s|zext which is subtracted separately.
+                Cost -= TTI->getCastInstrCost(
+                    Ext->getOpcode(), Ext->getType(), I->getType(),
                     TTI::getCastContextHint(Ext), CostKind, Ext);
-            continue;
-            }
-          }
-          CommonCost -=
-              TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, I);
-        }
-      } else {
-        AdjustExtractsCost(CommonCost);
+                return Cost;
               }
-      return CommonCost;
             }
+            return TTI->getVectorInstrCost(Instruction::ExtractElement,
+                                           SrcVecTy, *getExtractIndex(I));
+          },
+          [](InstructionCost CommonCost) { return CommonCost; });
     case Instruction::InsertElement: {
       assert(E->ReuseShuffleIndices.empty() &&
              "Unique insertelements only are expected.");
       auto *SrcVecTy = cast<FixedVectorType>(VL0->getType());
       unsigned const NumElts = SrcVecTy->getNumElements();
       unsigned const NumScalars = VL.size();
 
       unsigned NumOfParts = TTI->getNumberOfParts(SrcVecTy);
case Instruction::PHI: {
     case Instruction::FPToSI:
     case Instruction::FPExt:
     case Instruction::PtrToInt:
     case Instruction::IntToPtr:
     case Instruction::SIToFP:
     case Instruction::UIToFP:
     case Instruction::Trunc:
     case Instruction::FPTrunc:
-    case Instruction::BitCast: {
+    case Instruction::BitCast:
+      return GetCostDiff(
+          [this, E, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
+            return TTI->getCastInstrCost(
+                E->getOpcode(), ScalarTy, VI->getOperand(0)->getType(),
+                TTI::getCastContextHint(VI), CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
             Type *SrcTy = VL0->getOperand(0)->getType();
-      InstructionCost ScalarEltCost =
-          TTI->getCastInstrCost(E->getOpcode(), ScalarTy, SrcTy,
-                                TTI::getCastContextHint(VL0), CostKind, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-
-      // Calculate the cost of this instruction.
-      InstructionCost ScalarCost = VL.size() * ScalarEltCost;
-
             auto *SrcVecTy = FixedVectorType::get(SrcTy, VL.size());
             InstructionCost VecCost = 0;
             // Check if the values are candidates to demote.
             if (!MinBWs.count(VL0) || VecTy != SrcVecTy) {
-        VecCost = CommonCost + TTI->getCastInstrCost(
-                                   E->getOpcode(), VecTy, SrcVecTy,
-                                   TTI::getCastContextHint(VL0), CostKind, VL0);
-      }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return VecCost - ScalarCost;
+              VecCost = CommonCost +
+                        TTI->getCastInstrCost(E->getOpcode(), VecTy, SrcVecTy,
+                                              TTI::getCastContextHint(VL0),
+                                              CostKind, VL0);
             }
+            return VecCost;
+          });
     case Instruction::FCmp:
     case Instruction::ICmp:
     case Instruction::Select: {
-      // Calculate the cost of this instruction.
-      InstructionCost ScalarEltCost =
-          TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy, Builder.getInt1Ty(),
-                                  CmpInst::BAD_ICMP_PREDICATE, CostKind, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-
-      // Check if all entries in VL are either compares or selects with compares
-      // as condition that have the same predicates.
-      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE;
-      bool First = true;
-      for (auto *V : VL) {
+      CmpInst::Predicate VecPred, SwappedVecPred;
+      auto MatchCmp = m_Cmp(VecPred, m_Value(), m_Value());
+      if (match(VL0, m_Select(MatchCmp, m_Value(), m_Value())) ||
+          match(VL0, MatchCmp))
+        SwappedVecPred = CmpInst::getSwappedPredicate(VecPred);
+      else
+        SwappedVecPred = VecPred = CmpInst::BAD_ICMP_PREDICATE;
+      return GetCostDiff(
+          [this, E, ScalarTy, VL, CostKind, &VecPred,
+           &SwappedVecPred](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
             CmpInst::Predicate CurrentPred;
             auto MatchCmp = m_Cmp(CurrentPred, m_Value(), m_Value());
-        if ((!match(V, m_Select(MatchCmp, m_Value(), m_Value())) &&
-             !match(V, MatchCmp)) ||
-            (!First && VecPred != CurrentPred)) {
-          VecPred = CmpInst::BAD_ICMP_PREDICATE;
-          break;
-        }
-        First = false;
-        VecPred = CurrentPred;
-      }
+            if ((!match(VI, m_Select(MatchCmp, m_Value(), m_Value())) &&
+                 !match(VI, MatchCmp)) ||
+                (CurrentPred != VecPred && CurrentPred != SwappedVecPred))
+              VecPred = SwappedVecPred = CmpInst::BAD_ICMP_PREDICATE;
+
+            return TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy,
+                                           Builder.getInt1Ty(), CurrentPred,
+                                           CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind,
+           &VecPred](InstructionCost CommonCost) {
+            auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
 
             InstructionCost VecCost = TTI->getCmpSelInstrCost(
                 E->getOpcode(), VecTy, MaskTy, VecPred, CostKind, VL0);
-      // Check if it is possible and profitable to use min/max for selects in
-      // VL.
+            // Check if it is possible and profitable to use min/max for selects
+            // in VL.
             //
             auto IntrinsicAndUse = canConvertToMinOrMaxIntrinsic(VL);
             if (IntrinsicAndUse.first != Intrinsic::not_intrinsic) {
               IntrinsicCostAttributes CostAttrs(IntrinsicAndUse.first, VecTy,
                                                 {VecTy, VecTy});
               InstructionCost IntrinsicCost =
                   TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
-        // If the selects are the only uses of the compares, they will be dead
-        // and we can adjust the cost by removing their cost.
+              // If the selects are the only uses of the compares, they will be
+              // dead and we can adjust the cost by removing their cost.
               if (IntrinsicAndUse.second)
-          IntrinsicCost -= TTI->getCmpSelInstrCost(Instruction::ICmp, VecTy,
-                                                   MaskTy, VecPred, CostKind);
+                IntrinsicCost -= TTI->getCmpSelInstrCost(
+                    Instruction::ICmp, VecTy, MaskTy, VecPred, CostKind);
               VecCost = std::min(VecCost, IntrinsicCost);
             }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
+            return VecCost + CommonCost;
+          });
     }
     case Instruction::FNeg:
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
     case Instruction::UDiv:
     case Instruction::SDiv:
     case Instruction::FDiv:
     case Instruction::URem:
     case Instruction::SRem:
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
-    case Instruction::Xor: {
+    case Instruction::Xor:
+    case Instruction::GetElementPtr: {
+      unsigned Opcode = ShuffleOrOp == Instruction::GetElementPtr
+                            ? Instruction::Add
+                            : ShuffleOrOp;
+      // Certain instructions can be cheaper to vectorize if they have a
+      // constant second vector operand.
       TTI::OperandValueInfo Op1Info = {TTI::OK_AnyValue, TTI::OP_None};
 
       // Certain instructions can be cheaper to vectorize if they have a
       // constant second vector operand.
-      const unsigned OpIdx = isa<BinaryOperator>(VL0) ? 1 : 0;
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind, &Op1Info, Opcode](unsigned Idx) {
+            auto *VI = dyn_cast<Instruction>(VL[Idx]);
+            // GEPs may contain just addresses without instructions, consider
+            // their cost 0.
+            if (!VI)
+              return InstructionCost();
+            unsigned OpIdx = isa<BinaryOperator, GetElementPtrInst>(VI) ? 1 : 0;
+            auto Op2Info = getOperandInfo(VI, OpIdx);
+            if (!Op2Info.isConstant() && isCommutative(VI))
+              Op2Info = getOperandInfo(VI, 0);
+            SmallVector<const Value *> Operands(VI->operand_values());
+            return TTI->getArithmeticInstrCost(Opcode, ScalarTy, CostKind,
+                                               Op1Info, Op2Info, Operands, VI);
+          },
+          [this, VecTy, CostKind, VL0, VL, &Op1Info,
+           Opcode](InstructionCost CommonCost) {
+            unsigned OpIdx =
+                isa<BinaryOperator, GetElementPtrInst>(VL0) ? 1 : 0;
             auto Op2Info = getOperandInfo(VL, OpIdx);
-
-      SmallVector<const Value *, 4> Operands(VL0->operand_values());
-      InstructionCost ScalarEltCost =
-          TTI->getArithmeticInstrCost(E->getOpcode(), ScalarTy, CostKind,
-                                      Op1Info, Op2Info,
-                                      Operands, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      for (unsigned I = 0, Num = VL0->getNumOperands(); I < Num; ++I) {
-        if (all_of(VL, [I](Value *V) {
-              return isConstant(cast<Instruction>(V)->getOperand(I));
-            }))
-          Operands[I] = ConstantVector::getNullValue(VecTy);
-      }
-      InstructionCost VecCost =
-          TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind,
-                                      Op1Info, Op2Info,
-                                      Operands, VL0);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
-    }
-    case Instruction::GetElementPtr: {
-      TargetTransformInfo::OperandValueKind Op1VK =
-          TargetTransformInfo::OK_AnyValue;
-      TargetTransformInfo::OperandValueKind Op2VK =
-          any_of(VL,
-                 [](Value *V) {
-                   return isa<GetElementPtrInst>(V) &&
-                          !isConstant(
-                              cast<GetElementPtrInst>(V)->getOperand(1));
-                 })
-              ? TargetTransformInfo::OK_AnyValue
-              : TargetTransformInfo::OK_UniformConstantValue;
-
-      InstructionCost ScalarEltCost = TTI->getArithmeticInstrCost(
-          Instruction::Add, ScalarTy, CostKind,
-          {Op1VK, TargetTransformInfo::OP_None},
-          {Op2VK, TargetTransformInfo::OP_None});
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      InstructionCost VecCost = TTI->getArithmeticInstrCost(
-          Instruction::Add, VecTy, CostKind,
-          {Op1VK, TargetTransformInfo::OP_None},
-          {Op2VK, TargetTransformInfo::OP_None});
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
-    }
-    case Instruction::Load: {
-      // Cost of wide load - cost of scalar loads.
-      Align Alignment = cast<LoadInst>(VL0)->getAlign();
-      InstructionCost ScalarEltCost =
-          TTI->getMemoryOpCost(Instruction::Load, ScalarTy, Alignment, 0,
-                               CostKind, {TTI::OK_AnyValue, TTI::OP_None}, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      InstructionCost ScalarLdCost = VecTy->getNumElements() * ScalarEltCost;
+            return TTI->getArithmeticInstrCost(Opcode, VecTy, CostKind, Op1Info,
+                                               Op2Info) +
+                   CommonCost;
+          });
+    case Instruction::Load:
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<LoadInst>(VL[Idx]);
+            return TTI->getMemoryOpCost(Instruction::Load, ScalarTy,
+                                        VI->getAlign(),
+                                        VI->getPointerAddressSpace(), CostKind,
+                                        {TTI::OK_AnyValue, TTI::OP_None}, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
+            auto *LI0 = cast<LoadInst>(VL0);
             InstructionCost VecLdCost;
             if (E->State == TreeEntry::Vectorize) {
-        VecLdCost = TTI->getMemoryOpCost(Instruction::Load, VecTy, Alignment, 0,
-                                         CostKind, {TTI::OK_AnyValue, TTI::OP_None}, VL0);
-      } else {
-        assert(E->State == TreeEntry::ScatterVectorize && "Unknown EntryState");
-        Align CommonAlignment = Alignment;
+              VecLdCost = TTI->getMemoryOpCost(
+                  Instruction::Load, VecTy, LI0->getAlign(),
+                  LI0->getPointerAddressSpace(), CostKind,
+                  {TTI::OK_AnyValue, TTI::OP_None});
+            } else {
+              assert(E->State == TreeEntry::ScatterVectorize &&
+                     "Unknown EntryState");
+              Align CommonAlignment = LI0->getAlign();
               for (Value *V : VL)
                 CommonAlignment =
                     std::min(CommonAlignment, cast<LoadInst>(V)->getAlign());
               VecLdCost = TTI->getGatherScatterOpCost(
-            Instruction::Load, VecTy, cast<LoadInst>(VL0)->getPointerOperand(),
-            /*VariableMask=*/false, CommonAlignment, CostKind, VL0);
-      }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecLdCost, ScalarLdCost));
-      return CommonCost + VecLdCost - ScalarLdCost;
+                  Instruction::Load, VecTy, LI0->getPointerOperand(),
+                  /*VariableMask=*/false, CommonAlignment, CostKind);
             }
-    case Instruction::Store: {
+            return VecLdCost + CommonCost;
+          });
+    case Instruction::Store:
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<StoreInst>(VL[Idx]);
+            TTI::OperandValueInfo OpInfo = getOperandInfo(VI, 0);
+            return TTI->getMemoryOpCost(
+                Instruction::Store, ScalarTy, VI->getAlign(),
+                VI->getPointerAddressSpace(), CostKind, OpInfo, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
             // We know that we can merge the stores. Calculate the cost.
             bool IsReorder = !E->ReorderIndices.empty();
-      auto *SI =
-          cast<StoreInst>(IsReorder ? VL[E->ReorderIndices.front()] : VL0);
-      Align Alignment = SI->getAlign();
-      TTI::OperandValueInfo OpInfo = getOperandInfo(VL, 0);
-      InstructionCost ScalarEltCost = TTI->getMemoryOpCost(
-          Instruction::Store, ScalarTy, Alignment, 0, CostKind, OpInfo, VL0);
-      InstructionCost ScalarStCost = VecTy->getNumElements() * ScalarEltCost;
+            auto *SI = cast<StoreInst>(IsReorder ? VL[E->ReorderIndices.front()]
+                                                 : VL0);
             TTI::OperandValueKind OpVK = TTI::OK_AnyValue;
-      if (OpInfo.isConstant())
+            if (getOperandInfo(VL, 0).isConstant())
               OpVK = TTI::OK_NonUniformConstantValue;
-      InstructionCost VecStCost = TTI->getMemoryOpCost(
-          Instruction::Store, VecTy, Alignment, 0, CostKind,
-          {OpVK, TTI::OP_None}, VL0);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecStCost, ScalarStCost));
-      return CommonCost + VecStCost - ScalarStCost;
-    }
+            return TTI->getMemoryOpCost(Instruction::Store, VecTy,
+                                        SI->getAlign(),
+                                        SI->getPointerAddressSpace(), CostKind,
+                                        {OpVK, TTI::OP_None}) +
+                   CommonCost;
+          });
     case Instruction::Call: {
-      CallInst *CI = cast<CallInst>(VL0);
+      return GetCostDiff(
+          [this, VL, CostKind](unsigned Idx) {
+            auto *CI = cast<CallInst>(VL[Idx]);
             Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
-
-      // Calculate the cost of the scalar and vector calls.
+            if (ID != Intrinsic::not_intrinsic) {
               IntrinsicCostAttributes CostAttrs(ID, *CI, 1);
-      InstructionCost ScalarEltCost =
-          TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
+              return TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
             }
-      InstructionCost ScalarCallCost = VecTy->getNumElements() * ScalarEltCost;
-
+            return TTI->getCallInstrCost(
+                CI->getCalledFunction(), CI->getFunctionType()->getReturnType(),
+                CI->getFunctionType()->params(), CostKind);
+          },
+          [this, VL0, VecTy](InstructionCost CommonCost) {
+            auto *CI = cast<CallInst>(VL0);
             auto VecCallCosts = getVectorCallCosts(CI, VecTy, TTI, TLI);
-      InstructionCost VecCallCost =
-          std::min(VecCallCosts.first, VecCallCosts.second);
-
-      LLVM_DEBUG(dbgs() << "SLP: Call cost " << VecCallCost - ScalarCallCost
-                        << " (" << VecCallCost << "-" << ScalarCallCost << ")"
-                        << " for " << *CI << "\n");
-
-      return CommonCost + VecCallCost - ScalarCallCost;
+            return std::min(VecCallCosts.first, VecCallCosts.second) +
+                   CommonCost;
+          });
     }
     case Instruction::ShuffleVector: {
       assert(E->isAltShuffle() &&
              ((Instruction::isBinaryOp(E->getOpcode()) &&
                Instruction::isBinaryOp(E->getAltOpcode())) ||
               (Instruction::isCast(E->getOpcode()) &&
                Instruction::isCast(E->getAltOpcode())) ||
               (isa<CmpInst>(VL0) && isa<CmpInst>(E->getAltOp()))) &&
              "Invalid Shuffle Vector Operand");
-      InstructionCost ScalarCost = 0;
-      if (NeedToShuffleReuses) {
-        for (unsigned Idx : E->ReuseShuffleIndices) {
-          Instruction *I = cast<Instruction>(VL[Idx]);
-          CommonCost -= TTI->getInstructionCost(I, CostKind);
-        }
-        for (Value *V : VL) {
-          Instruction *I = cast<Instruction>(V);
-          CommonCost += TTI->getInstructionCost(I, CostKind);
-        }
-      }
-      for (Value *V : VL) {
-        Instruction *I = cast<Instruction>(V);
-        assert(E->isOpcodeOrAlt(I) && "Unexpected main/alternate opcode");
-        ScalarCost += TTI->getInstructionCost(I, CostKind);
-      }
-      // VecCost is equal to sum of the cost of creating 2 vectors
-      // and the cost of creating shuffle.
-      InstructionCost VecCost = 0;
       // Try to find the previous shuffle node with the same operands and same
       // main/alternate ops.
       auto &&TryFindNodeWithEqualOperands = [this, E]() {
         for (const std::unique_ptr<TreeEntry> &TE : VectorizableTree) {
           if (TE.get() == E)
             break;
           if (TE->isAltShuffle() &&
               ((TE->getOpcode() == E->getOpcode() &&
                 TE->getAltOpcode() == E->getAltOpcode()) ||
                (TE->getOpcode() == E->getAltOpcode() &&
                 TE->getAltOpcode() == E->getOpcode())) &&
               TE->hasEqualOperands(*E))
             return true;
         }
         return false;
       };
+      // Need to clear CommonCost since the final shuffle cost is included into
+      // vector cost.
+      return GetCostDiff(
+          [this, VL, CostKind, E](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
+            assert(E->isOpcodeOrAlt(VI) && "Unexpected main/alternate opcode");
+            (void)E;
+            return TTI->getInstructionCost(VI, CostKind);
+          },
+          [this, VecTy, E, CostKind, VL, FinalVecTy,
+           &TryFindNodeWithEqualOperands, VL0, ScalarTy](InstructionCost) {
+            // VecCost is equal to sum of the cost of creating 2 vectors
+            // and the cost of creating shuffle.
+            InstructionCost VecCost = 0;
             if (TryFindNodeWithEqualOperands()) {
               LLVM_DEBUG({
                 dbgs() << "SLP: diamond match for alternate node found.\n";
                 E->dump();
               });
               // No need to add new vector costs here since we're going to reuse
               // same main/alternate vector ops, just do different shuffling.
             } else if (Instruction::isBinaryOp(E->getOpcode())) {
-        VecCost = TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind);
+              VecCost =
+                  TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind);
               VecCost += TTI->getArithmeticInstrCost(E->getAltOpcode(), VecTy,
                                                      CostKind);
             } else if (auto *CI0 = dyn_cast<CmpInst>(VL0)) {
-        VecCost = TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy,
-                                          Builder.getInt1Ty(),
-                                          CI0->getPredicate(), CostKind, VL0);
+              VecCost = TTI->getCmpSelInstrCost(
+                  E->getOpcode(), ScalarTy, Builder.getInt1Ty(),
               VecCost += TTI->getCmpSelInstrCost(
                   E->getOpcode(), ScalarTy, Builder.getInt1Ty(),
                   cast<CmpInst>(E->getAltOp())->getPredicate(), CostKind,
                   E->getAltOp());
             } else {
               Type *Src0SclTy = E->getMainOp()->getOperand(0)->getType();
               Type *Src1SclTy = E->getAltOp()->getOperand(0)->getType();
               auto *Src0Ty = FixedVectorType::get(Src0SclTy, VL.size());
               auto *Src1Ty = FixedVectorType::get(Src1SclTy, VL.size());
-        VecCost = TTI->getCastInstrCost(E->getOpcode(), VecTy, Src0Ty,
+              VecCost =
+                  TTI->getCastInstrCost(E->getOpcode(), VecTy, Src0Ty,
                                         TTI::CastContextHint::None, CostKind);
-        VecCost += TTI->getCastInstrCost(E->getAltOpcode(), VecTy, Src1Ty,
-                                         TTI::CastContextHint::None, CostKind);
+              VecCost +=
+                  TTI->getCastInstrCost(E->getAltOpcode(), VecTy, Src1Ty,
             }
-
+            SmallVector<int> Mask;
             if (E->ReuseShuffleIndices.empty()) {
-        CommonCost =
-            TTI->getShuffleCost(TargetTransformInfo::SK_Select, FinalVecTy);
+              VecCost += TTI->getShuffleCost(TargetTransformInfo::SK_Select,
+                                               FinalVecTy);
             } else {
               SmallVector<int> Mask;
               buildShuffleEntryMask(
                   E->Scalars, E->ReorderIndices, E->ReuseShuffleIndices,
                   [E](Instruction *I) {
-              assert(E->isOpcodeOrAlt(I) && "Unexpected main/alternate opcode");
+                    assert(E->isOpcodeOrAlt(I) &&
+                           "Unexpected main/alternate opcode");
                     return I->getOpcode() == E->getAltOpcode();
                   },
                   Mask);
-        CommonCost = TTI->getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc,
-                                         FinalVecTy, Mask);
+              VecCost += TTI->getShuffleCost(
+                  TargetTransformInfo::SK_PermuteTwoSrc, FinalVecTy, Mask);
             }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
+            return VecCost;
+          });
     }
     default:
       llvm_unreachable("Unknown instruction");
   }
 }
 
 bool BoUpSLP::isFullyVectorizableTinyTree(bool ForReduction) const {
   LLVM_DEBUG(dbgs() << "SLP: Check whether the tree with height "
for (ExternalUser &EU : ExternalUses) {
     // removed as well).
     if (EphValues.count(EU.User))
       continue;
 
     // No extract cost for vector "scalar"
     if (isa<FixedVectorType>(EU.Scalar->getType()))
       continue;
 
-    // Already counted the cost for external uses when tried to adjust the cost
-    // for extractelements, no need to add it again.
-    if (isa<ExtractElementInst>(EU.Scalar))
-      continue;
-
     // If found user is an insertelement, do not calculate extract cost but try
     // to detect it as a final shuffled/identity match.
     if (auto *VU = dyn_cast_or_null<InsertElementInst>(EU.User)) {
       if (auto *FTy = dyn_cast<FixedVectorType>(VU->getType())) {
         Optional<unsigned> InsertIdx = getInsertIndex(VU);
         if (InsertIdx) {
           const TreeEntry *ScalarTE = getTreeEntry(EU.Scalar);
           auto *It = find_if(

llvm/test/Transforms/SLPVectorizer/AArch64/horizontal.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -slp-vectorizer -slp-threshold=-5 -S -pass-remarks-output=%t < %s | FileCheck %s
+; RUN: opt -slp-vectorizer -slp-threshold=-3 -S -pass-remarks-output=%t < %s | FileCheck %s
 ; RUN: cat %t | FileCheck -check-prefix=YAML %s
 
 
 ; FIXME: The threshold is changed to keep this test case a bit smaller.
 ; The AArch64 cost model should not give such high costs to select statements.
 
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64--linux"

llvm/test/Transforms/SLPVectorizer/AArch64/vectorizable-selects-min-max.ll

entry:
   store i32 %s.3, i32* %gep.3, align 4
 
   ret void
 }
 
 define void @select_ule_ugt_mix_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_ule_ugt_mix_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ult i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ugt i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR:%.*]] to <4 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <4 x i32>, <4 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <4 x i32> <i32 poison, i32 poison, i32 poison, i32 16383>, <4 x i32> [[TMP1]], <4 x i32> <i32 4, i32 5, i32 6, i32 3>
+; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <4 x i32> <i32 16383, i32 16383, i32 16383, i32 poison>, <4 x i32> [[TMP1]], <4 x i32> <i32 0, i32 1, i32 2, i32 7>
+; CHECK-NEXT:    [[TMP4:%.*]] = icmp ult <4 x i32> [[TMP2]], [[TMP3]]
+; CHECK-NEXT:    [[TMP5:%.*]] = icmp ugt <4 x i32> [[TMP2]], [[TMP3]]
+; CHECK-NEXT:    [[TMP6:%.*]] = shufflevector <4 x i1> [[TMP4]], <4 x i1> [[TMP5]], <4 x i32> <i32 0, i32 5, i32 2, i32 3>
+; CHECK-NEXT:    [[TMP7:%.*]] = select <4 x i1> [[TMP6]], <4 x i32> [[TMP1]], <4 x i32> <i32 16383, i32 16383, i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i32* [[PTR]] to <4 x i32>*
+; CHECK-NEXT:    store <4 x i32> [[TMP7]], <4 x i32>* [[TMP8]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp ult i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 

llvm/test/Transforms/SLPVectorizer/X86/PR32086.ll

;
   store i64 %t0, i64* %arrayidx3, align 8
   store i64 %t1, i64* %arrayidx4, align 8
   store i64 %t0, i64* %arrayidx5, align 8
   ret void
 }
 
 define void @i64_simplifiedi_extract(i64* noalias %st, i64* noalias %ld) {
 ; CHECK-LABEL: @i64_simplifiedi_extract(
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i64* [[LD:%.*]] to <2 x i64>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i64>, <2 x i64>* [[TMP1]], align 8
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i64> [[TMP2]], <2 x i64> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i64* [[ST:%.*]] to <4 x i64>*
-; CHECK-NEXT:    store <4 x i64> [[SHUFFLE]], <4 x i64>* [[TMP3]], align 8
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <4 x i64> [[SHUFFLE]], i32 3
-; CHECK-NEXT:    store i64 [[TMP4]], i64* [[LD]], align 8
+; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i64, i64* [[LD:%.*]], i64 1
+; CHECK-NEXT:    [[T0:%.*]] = load i64, i64* [[LD]], align 8
+; CHECK-NEXT:    [[T1:%.*]] = load i64, i64* [[ARRAYIDX1]], align 8
+; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds i64, i64* [[ST:%.*]], i64 1
+; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i64, i64* [[ST]], i64 2
+; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds i64, i64* [[ST]], i64 3
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ST]], align 8
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ARRAYIDX3]], align 8
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ARRAYIDX4]], align 8
+; CHECK-NEXT:    store i64 [[T1]], i64* [[ARRAYIDX5]], align 8
+; CHECK-NEXT:    store i64 [[T1]], i64* [[LD]], align 8
 ; CHECK-NEXT:    ret void
 ;
   %arrayidx1 = getelementptr inbounds i64, i64* %ld, i64 1
 
   %t0 = load i64, i64* %ld, align 8
   %t1 = load i64, i64* %arrayidx1, align 8
 
   %arrayidx3 = getelementptr inbounds i64, i64* %st, i64 1

llvm/test/Transforms/SLPVectorizer/X86/reduction-same-vals.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -S -slp-vectorizer < %s | FileCheck %s
 
 define i64 @test() {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  bb1:
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       bb2:
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb3:
-; CHECK-NEXT:    [[TMP0:%.*]] = phi <2 x i32> [ poison, [[BB2:%.*]] ], [ zeroinitializer, [[BB1:%.*]] ]
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
-; CHECK-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <8 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP2:%.*]] = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> [[TMP1]])
-; CHECK-NEXT:    [[TMP3:%.*]] = call i32 @llvm.vector.reduce.mul.v4i32(<4 x i32> [[SHUFFLE]])
-; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP2]], [[TMP3]]
-; CHECK-NEXT:    [[TMP65:%.*]] = sext i32 [[OP_RDX]] to i64
+; CHECK-NEXT:    [[TMP:%.*]] = phi i32 [ 0, [[BB2:%.*]] ], [ 0, [[BB1:%.*]] ]
+; CHECK-NEXT:    [[TMP4:%.*]] = phi i32 [ 0, [[BB2]] ], [ 0, [[BB1]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x i32> poison, i32 [[TMP4]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <8 x i32> [[TMP0]], i32 [[TMP4]], i32 1
+; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i32> [[TMP1]], i32 [[TMP4]], i32 2
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i32> [[TMP2]], i32 [[TMP4]], i32 3
+; CHECK-NEXT:    [[TMP44:%.*]] = insertelement <8 x i32> [[TMP3]], i32 [[TMP4]], i32 4
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x i32> [[TMP44]], i32 [[TMP4]], i32 5
+; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x i32> [[TMP5]], i32 [[TMP4]], i32 6
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x i32> [[TMP6]], i32 [[TMP4]], i32 7
+; CHECK-NEXT:    [[TMP8:%.*]] = call i32 @llvm.vector.reduce.mul.v8i32(<8 x i32> [[TMP7]])
+; CHECK-NEXT:    [[OP_RDX:%.*]] = mul i32 [[TMP8]], [[TMP4]]
+; CHECK-NEXT:    [[OP_RDX1:%.*]] = mul i32 [[TMP4]], [[TMP4]]
+; CHECK-NEXT:    [[OP_RDX2:%.*]] = mul i32 [[OP_RDX]], [[OP_RDX1]]
+; CHECK-NEXT:    [[OP_RDX3:%.*]] = mul i32 [[OP_RDX2]], [[TMP]]
+; CHECK-NEXT:    [[TMP65:%.*]] = sext i32 [[OP_RDX3]] to i64
 ; CHECK-NEXT:    ret i64 [[TMP65]]
 ;
 bb1:
   br label %bb3
 
 bb2:
   br label %bb3
 

llvm/test/Transforms/SLPVectorizer/X86/vectorize-reorder-reuse.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -slp-vectorizer -S -mtriple=x86_64-unknown-linux-gnu -mcpu=bdver2 < %s | FileCheck %s
 
 define i32 @foo(i32* nocapture readonly %arr, i32 %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6, i32 %a7, i32 %a8) {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[ARR:%.*]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x i32>, <2 x i32>* [[TMP0]], align 4
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <8 x i32> <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 0, i32 0>
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i32> poison, i32 [[A2:%.*]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i32> [[TMP2]], i32 [[A1:%.*]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x i32> [[TMP3]], i32 [[A3:%.*]], i32 2
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x i32> [[TMP4]], i32 [[A4:%.*]], i32 3
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x i32> [[TMP5]], i32 [[A5:%.*]], i32 4
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x i32> [[TMP6]], i32 [[A6:%.*]], i32 5
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x i32> [[TMP7]], i32 [[A7:%.*]], i32 6
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <8 x i32> [[TMP8]], i32 [[A8:%.*]], i32 7
-; CHECK-NEXT:    [[TMP10:%.*]] = add <8 x i32> [[SHUFFLE]], [[TMP9]]
-; CHECK-NEXT:    [[TMP11:%.*]] = call i32 @llvm.vector.reduce.umin.v8i32(<8 x i32> [[TMP10]])
-; CHECK-NEXT:    ret i32 [[TMP11]]
+; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[ARR:%.*]], i64 1
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[TMP0]], [[A1:%.*]]
+; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[TMP0]], [[A2:%.*]]
+; CHECK-NEXT:    [[ADD4:%.*]] = add i32 [[TMP0]], [[A3:%.*]]
+; CHECK-NEXT:    [[ADD6:%.*]] = add i32 [[TMP0]], [[A4:%.*]]
+; CHECK-NEXT:    [[ADD8:%.*]] = add i32 [[TMP0]], [[A5:%.*]]
+; CHECK-NEXT:    [[ADD10:%.*]] = add i32 [[TMP0]], [[A6:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[ARR]], align 4
+; CHECK-NEXT:    [[ADD12:%.*]] = add i32 [[TMP1]], [[A7:%.*]]
+; CHECK-NEXT:    [[ADD14:%.*]] = add i32 [[TMP1]], [[A8:%.*]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[ADD]], [[ADD2]]
+; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i32 [[ADD]], i32 [[ADD2]]
+; CHECK-NEXT:    [[CMP15:%.*]] = icmp ult i32 [[COND]], [[ADD4]]
+; CHECK-NEXT:    [[COND19:%.*]] = select i1 [[CMP15]], i32 [[COND]], i32 [[ADD4]]
+; CHECK-NEXT:    [[CMP20:%.*]] = icmp ult i32 [[COND19]], [[ADD6]]
+; CHECK-NEXT:    [[COND24:%.*]] = select i1 [[CMP20]], i32 [[COND19]], i32 [[ADD6]]
+; CHECK-NEXT:    [[CMP25:%.*]] = icmp ult i32 [[COND24]], [[ADD8]]
+; CHECK-NEXT:    [[COND29:%.*]] = select i1 [[CMP25]], i32 [[COND24]], i32 [[ADD8]]
+; CHECK-NEXT:    [[CMP30:%.*]] = icmp ult i32 [[COND29]], [[ADD10]]
+; CHECK-NEXT:    [[COND34:%.*]] = select i1 [[CMP30]], i32 [[COND29]], i32 [[ADD10]]
+; CHECK-NEXT:    [[CMP35:%.*]] = icmp ult i32 [[COND34]], [[ADD12]]
+; CHECK-NEXT:    [[COND39:%.*]] = select i1 [[CMP35]], i32 [[COND34]], i32 [[ADD12]]
+; CHECK-NEXT:    [[CMP40:%.*]] = icmp ult i32 [[COND39]], [[ADD14]]
+; CHECK-NEXT:    [[COND44:%.*]] = select i1 [[CMP40]], i32 [[COND39]], i32 [[ADD14]]
+; CHECK-NEXT:    ret i32 [[COND44]]
 ;
 entry:
   %arrayidx = getelementptr inbounds i32, i32* %arr, i64 1
   %0 = load i32, i32* %arrayidx, align 4
   %add = add i32 %0, %a1
   %add2 = add i32 %0, %a2
   %add4 = add i32 %0, %a3
   %add6 = add i32 %0, %a4