[SLP]Generalize cost model.

[ABataev]

Summary

Generalized the cost model estimation. Improved cost model estimation
for repeated scalars (no need to count their cost anymore), improved

cost model for extractelement instructions.

cpu2017

511.povray_r             0.57
520.omnetpp_r           -0.98
521.wrf_r               -0.01
525.x264_r               3.59 <+
526.blender_r           -0.12
531.deepsjeng_r         -0.07
538.imagick_r           -1.42

Geometric mean: 0.21

[ABataev]

Rebase

[ABataev]

Rebase

[ABataev]

Rebase

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4566–4567

I->getOperand(OpIdx) == Op0;

4566–4567

auto *Op = I->getOperand(OpIdx);

4569

true?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4569

Here is my idea. Generally speaking, here we can have only undef/poison or just a pointer (as part of getelementpr node) currently (maybe, need to add an assert). I thought that because of that we can treat its (virtual) operand as power-of-2, since it can be any value. Thoughts?

[ABataev]

Address comments

[ABataev]

Rebase

[ABataev]

Ping!

[ABataev]

Rebase. Ping!

[ABataev]

Ping!

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

Is this purely compile time saving?

4889

Can't we directly initialize Operands from PHI-> incoming_values()? Or even create a ArrayRef<> from it?

4938

We're going to lose the recent scalarization cost improvement here :-(

5138

Would isa<UnaryOperator> ? 0 : 1 be better?

5205

Why do you avoid allowing uniform as well? Can't you just use getOperandInfo(VL, 0)..getNoProps() below ?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

Replaced by 0, since it does not matter.

4889

operand_value returns llvm::Use, Operands list requires Value *. Did not find better way to do it.

5205

Just missed this change, will fix.

[ABataev]

Rebase, address comments

[ABataev]

Ping!

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4569

Still not sure about this - this undef case is fine, but the GEP I can imagine might not work in all special cases.

How good is return isa<UndefValue>(V); instead?

4571

return isa<UndefValue>(V)?

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5134

Why didn't we ever use getGEPCost?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5134

It originally used add cost for GEPs, most probably there were getGEPCost. Plus, I don't see target-specific getGEPCost implementations, looks like we have only base one.
We could try to switch to getGEPCost but it must be in a separate patch.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4569

GetElementPtr may have just a pointer, which is not a GEP instructions.
Say, we have this:

void @foo(ptr %v) {
...
load %v
%n = getelementptr i32, ptr v, 1
load %n
...
}

and we may have a ПУЗ node which includes {%v, %n}. Here %v is not a GEP though still part of the GEP node and we consider as gentlementptr ptr %v, 0.

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4569

OK - so for GEPs you're saying that either the operands will be constant or a phi of constants?

4893

I'm sorry but I had to revert the ScalarizationOverheadBuilder code - this needs reverting back to the older version

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4569

No (or I missed something). I mean, that except for GEPs we may have just a non-ПУЗ pointer value and we just treat it as getelementptr, 0. But we return true if we have just a value (which is GEP with offset 0) and all other GEPs are power-of-2 constant ints.

[ABataev]

Rebase

[ABataev]

Rebase

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5029

So CommonCost is only applied if (!MinBWs.count(VL0) || VecTy != SrcVecTy) ?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5029

Yeah, looks like should not ignore it here, since it includes the shuffle cost.

[ABataev]

Address comments

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

Do you have a specific reason to keep this as lambda(lambda(...),lambda(...)) design?
I mean this would look much better if you make one more step and outline this into a separate class (for example CostDifferentiator) with dedicated methods instead of stuffing code with tons of lambdas.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

It will still have lambdas for each particular opcode

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

Why? It is not prohibited to have dedicated method in the class for each particular opcode.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

It still will be a switch with functions calls, which have the pretty similar loops and other processing. This what exactly I want to avoid - duplicate of the code.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

I can't agree with statement that it will be a duplication of code. Organized differently - yes.
IMO it's always worth investing time in developing a better interface.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

I don't quite understand what you're proposing. Provide at least a small example.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4852

Just a sketch:

sample.txt6 KBDownload

[RKSimon]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

I'm not sure if we need to pull them out into separate methods if they're just being called once. But it would help a little for readability if the lambdas weren't embedded in the GetCostDiff calls themselves.

So:

auto ScalarEltCost = [](unsigned Idx)  {...};
auto VectorCost = [](InstructionCost CommonCost)  {...};
return GetCostDiff(ScalarEltCost, VectorCost);

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

I prefer this approach. I don't want to introduce virtual member functions. Without virtual functions the only approach - generic lambda, which increases codesize.

[ABataev]

Address comments, formatting

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

What problem you see with virtual class here?
Using named lambdas instead of unnamed ones is definitely better. But essentially it anyway remains a C-style programming, just using C++ syntax. You are also making it very difficult to customize this code downstream. If that is the goal - okay, got it.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

It usually increases compile time.
Why it is hard to customize it?

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

It usually increases compile time.

Use of C++ has its costs. I'm pretty sure the increase will be unmeasurable.
I believe that if invest a bit more time into this something better can be found.

Why it is hard to customize it?

because this will require replacing a lambda rather than just make a descendant with custom functionality that might fully or partially reuse its base. I'm not saying it is impossible to do that with lambdas. It will just be significantly more ugly: create another lambda with the original one in capture list...

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

I though about templated lambda instead of virtual class, but it will implicitly copied across all cases.

[RKSimon]

LGTM

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

4862

I don't want to stop this conversation as I'd dearly like to see the SLP code be easier to understand as its complexity/density puts off a lot of people from contributing - but we need somewhere better to discuss this than on a patch review thread.

4878–4885

It looks like theres a clang-format change across this entire switch statement that ideally needs pre-committing to reduce the patch whitespace/indentation diff?

[fmayer]

This seems to have an MSAN error: https://lab.llvm.org/buildbot/#/builders/74/builds/14338/steps/13/logs/stdio

FAIL: LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll (57172 of 68894)
******************** TEST 'LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt < /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll -slp-vectorizer -dce -S -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7 | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
--
Exit Code: 2
Command Output (stderr):
--
==3579288==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x56270dbbe918 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19
    #1 0x56270dbbe918 in llvm::InstructionCost llvm::function_ref<llvm::InstructionCost (unsigned int)>::callback_fn<llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_63>(long, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #2 0x56270dafc992 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #3 0x56270dafc992 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_58::operator()(llvm::function_ref<llvm::InstructionCost (unsigned int)>, llvm::function_ref<llvm::InstructionCost (llvm::InstructionCost)>) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6336:29
    #4 0x56270daeeb36 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
    #5 0x56270db033d8 in llvm::slpvectorizer::BoUpSLP::getTreeCost(llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:7273:25
    #6 0x56270db4bb38 in llvm::SLPVectorizerPass::vectorizeStoreChain(llvm::ArrayRef<llvm::Value*>, llvm::slpvectorizer::BoUpSLP&, unsigned int, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10379:28
    #7 0x56270db4eadb in llvm::SLPVectorizerPass::vectorizeStores(llvm::ArrayRef<llvm::StoreInst*>, llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10521:13
    #8 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12807:18
    #9 0x56270db4279c in callback_fn<(lambda at /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12806:9)> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #10 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #11 0x56270db4279c in tryToVectorizeSequence<llvm::StoreInst> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12192:9
    #12 0x56270db4279c in llvm::SLPVectorizerPass::vectorizeStoreChains(llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12804:16
    #13 0x56270db3f9f7 in llvm::SLPVectorizerPass::runImpl(llvm::Function&, llvm::ScalarEvolution*, llvm::TargetTransformInfo*, llvm::TargetLibraryInfo*, llvm::AAResults*, llvm::LoopInfo*, llvm::DominatorTree*, llvm::AssumptionCache*, llvm::DemandedBits*, llvm::OptimizationRemarkEmitter*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10332:18
    #14 0x56270db3e0b0 in llvm::SLPVectorizerPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10267:18
    #15 0x56270ddc2a21 in llvm::detail::PassModel<llvm::Function, llvm::SLPVectorizerPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #16 0x56270bf43ae0 in llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #17 0x56270779e201 in llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #18 0x56270bf4fc31 in llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/IR/PassManager.cpp:124:38
    #19 0x56270779db71 in llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #20 0x56270bf40fb0 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #21 0x562706b90baf in llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/NewPMDriver.cpp:497:7
    #22 0x562706bc0377 in main /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/opt.cpp:784:12
    #23 0x7ff7be184d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #24 0x7ff7be184e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #25 0x562706af7664 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt+0x386a664)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19 in operator()
Exiting
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll

[ABataev]

In D115757#3866816, @fmayer wrote:

This seems to have an MSAN error: https://lab.llvm.org/buildbot/#/builders/74/builds/14338/steps/13/logs/stdio

FAIL: LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll (57172 of 68894)
******************** TEST 'LLVM :: Transforms/SLPVectorizer/X86/crash_7zip.ll' FAILED ********************
Script:
--
: 'RUN: at line 2';   /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt < /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll -slp-vectorizer -dce -S -mtriple=x86_64-apple-macosx10.8.0 -mcpu=corei7 | /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll
--
Exit Code: 2
Command Output (stderr):
--
==3579288==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x56270dbbe918 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19
    #1 0x56270dbbe918 in llvm::InstructionCost llvm::function_ref<llvm::InstructionCost (unsigned int)>::callback_fn<llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_63>(long, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #2 0x56270dafc992 in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #3 0x56270dafc992 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>)::$_58::operator()(llvm::function_ref<llvm::InstructionCost (unsigned int)>, llvm::function_ref<llvm::InstructionCost (llvm::InstructionCost)>) const /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6336:29
    #4 0x56270daeeb36 in llvm::slpvectorizer::BoUpSLP::getEntryCost(llvm::slpvectorizer::BoUpSLP::TreeEntry const*, llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
    #5 0x56270db033d8 in llvm::slpvectorizer::BoUpSLP::getTreeCost(llvm::ArrayRef<llvm::Value*>) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:7273:25
    #6 0x56270db4bb38 in llvm::SLPVectorizerPass::vectorizeStoreChain(llvm::ArrayRef<llvm::Value*>, llvm::slpvectorizer::BoUpSLP&, unsigned int, unsigned int) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10379:28
    #7 0x56270db4eadb in llvm::SLPVectorizerPass::vectorizeStores(llvm::ArrayRef<llvm::StoreInst*>, llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10521:13
    #8 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12807:18
    #9 0x56270db4279c in callback_fn<(lambda at /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12806:9)> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
    #10 0x56270db4279c in operator() /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
    #11 0x56270db4279c in tryToVectorizeSequence<llvm::StoreInst> /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12192:9
    #12 0x56270db4279c in llvm::SLPVectorizerPass::vectorizeStoreChains(llvm::slpvectorizer::BoUpSLP&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:12804:16
    #13 0x56270db3f9f7 in llvm::SLPVectorizerPass::runImpl(llvm::Function&, llvm::ScalarEvolution*, llvm::TargetTransformInfo*, llvm::TargetLibraryInfo*, llvm::AAResults*, llvm::LoopInfo*, llvm::DominatorTree*, llvm::AssumptionCache*, llvm::DemandedBits*, llvm::OptimizationRemarkEmitter*) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10332:18
    #14 0x56270db3e0b0 in llvm::SLPVectorizerPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:10267:18
    #15 0x56270ddc2a21 in llvm::detail::PassModel<llvm::Function, llvm::SLPVectorizerPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #16 0x56270bf43ae0 in llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #17 0x56270779e201 in llvm::detail::PassModel<llvm::Function, llvm::PassManager<llvm::Function, llvm::AnalysisManager<llvm::Function>>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Function>>::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #18 0x56270bf4fc31 in llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/IR/PassManager.cpp:124:38
    #19 0x56270779db71 in llvm::detail::PassModel<llvm::Module, llvm::ModuleToFunctionPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManagerInternal.h:88:17
    #20 0x56270bf40fb0 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/include/llvm/IR/PassManager.h:517:40
    #21 0x562706b90baf in llvm::runPassPipeline(llvm::StringRef, llvm::Module&, llvm::TargetMachine*, llvm::TargetLibraryInfoImpl*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::ToolOutputFile*, llvm::StringRef, llvm::ArrayRef<llvm::StringRef>, llvm::ArrayRef<llvm::PassPlugin>, llvm::opt_tool::OutputKind, llvm::opt_tool::VerifierKind, bool, bool, bool, bool, bool, bool) /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/NewPMDriver.cpp:497:7
    #22 0x562706bc0377 in main /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/tools/opt/opt.cpp:784:12
    #23 0x7ff7be184d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #24 0x7ff7be184e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 69389d485a9793dbe873f0ea2c93e02efaa9aa3d)
    #25 0x562706af7664 in _start (/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/opt+0x386a664)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp:6569:19 in operator()
Exiting
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build_msan/bin/FileCheck /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm/test/Transforms/SLPVectorizer/X86/crash_7zip.ll

It was already reverted and then committed back with a fix.

[fmayer]

In D115757#3866833, @ABataev wrote:

It was already reverted and then committed back with a fix.

Sorry about that, I missed the revert line, I only skimmed the comments. Thanks!

[LuoYuanke]

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

[ABataev]

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

[LuoYuanke]

In D115757#3958922, @ABataev wrote:

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

Do you have any plan to fix it?

[ABataev]

In D115757#3959221, @LuoYuanke wrote:

In D115757#3958922, @ABataev wrote:

In D115757#3958787, @LuoYuanke wrote:

It seems there is regression for 538 in your data, and we observed the regression for cpu2017/638 too. Could you fix the regression for them?

538.imagick_r -1.42

Not as an immediate fix, hope future changes will fix it.

Do you have any plan to fix it?

A plan of improving SLP vectorizer, not this particular regression. It will be fixed eventually

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

buildSuffleEntryMask(ArrayRef<Value *> VL, ArrayRef<unsigned> ReorderIndices,
   if (!ReusesIndices.empty()) {
     SmallVector<int> NewMask(ReusesIndices.size(), UndefMaskElem);
     transform(ReusesIndices, NewMask.begin(), [&Mask](int Idx) {
       return Idx != UndefMaskElem ? Mask[Idx] : UndefMaskElem;
     });
     Mask.swap(NewMask);
   }
 }
 
 InstructionCost BoUpSLP::getEntryCost(const TreeEntry *E,
                                       ArrayRef<Value *> VectorizedVals) {
   ArrayRef<Value*> VL = E->Scalars;
 
   Type *ScalarTy = VL[0]->getType();
   if (StoreInst *SI = dyn_cast<StoreInst>(VL[0]))
     ScalarTy = SI->getValueOperand()->getType();
   else if (CmpInst *CI = dyn_cast<CmpInst>(VL[0]))
     ScalarTy = CI->getOperand(0)->getType();
   else if (auto *IE = dyn_cast<InsertElementInst>(VL[0]))
     ScalarTy = IE->getOperand(1)->getType();
   auto *VecTy = FixedVectorType::get(ScalarTy, VL.size());
   TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput;
CommonCost =
         TTI->getShuffleCost(TTI::SK_PermuteSingleSrc, FinalVecTy, Mask);
   assert((E->State == TreeEntry::Vectorize ||
           E->State == TreeEntry::ScatterVectorize) &&
          "Unhandled state");
   assert(E->getOpcode() && allSameType(VL) && allSameBlock(VL) && "Invalid VL");
   Instruction *VL0 = E->getMainOp();
   unsigned ShuffleOrOp =
       E->isAltShuffle() ? (unsigned)Instruction::ShuffleVector : E->getOpcode();
+  auto &&GetCostDiff =
+      [this, E, VL, CommonCost,
+       VL0](function_ref<InstructionCost(unsigned)> ScalarEltCost,
+            function_ref<InstructionCost(InstructionCost)> VectorCost) {
+        // Calculate the cost of this instruction.
+        InstructionCost ScalarCost = 0;
+        if (isa<CastInst, CmpInst, SelectInst, CallInst>(VL0)) {
+          // For some of the instructions no need to calculate cost for each
+          // particular instruction, we can use the cost of the single
+          // instruction x total number of scalar instructions.
+          ScalarCost = VL.size() *
+                       ScalarEltCost(std::distance(VL.begin(), find(VL, VL0)));
+        } else {
+          for (unsigned I = 0, Sz = VL.size(); I < Sz; ++I)
+            ScalarCost += ScalarEltCost(I);
+        }
+
+        InstructionCost VecCost = VectorCost(CommonCost);
+        LLVM_DEBUG(
+            dumpTreeCosts(E, CommonCost, VecCost - CommonCost, ScalarCost));
+        // Disable warnings for `this` and `E` are unused. Required for
+        // `dumpTreeCosts`.
+        (void)this;
+        (void)E;
+        return VecCost - ScalarCost;
+      };
   switch (ShuffleOrOp) {
-    case Instruction::PHI:
-      return 0;
+  case Instruction::PHI: {
+    // Count reused scalars.
+    InstructionCost ScalarCost = 0;
+    SmallPtrSet<const TreeEntry *, 4> CountedOps;
+    for (Value *V : VL) {
+      auto *PHI = dyn_cast<PHINode>(V);
+      if (!PHI)
+        continue;
 
-    case Instruction::ExtractValue:
-    case Instruction::ExtractElement: {
-      // The common cost of removal ExtractElement/ExtractValue instructions +
-      // the cost of shuffles, if required to resuffle the original vector.
-      if (NeedToShuffleReuses) {
-        unsigned Idx = 0;
-        for (unsigned I : E->ReuseShuffleIndices) {
-          if (ShuffleOrOp == Instruction::ExtractElement) {
-            auto *EE = cast<ExtractElementInst>(VL[I]);
-            CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  EE->getVectorOperandType(),
-                                                  *getExtractIndex(EE));
-          } else {
-            CommonCost -= TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
-            ++Idx;
-          }
-        }
-        Idx = EntryVF;
-        for (Value *V : VL) {
+      ValueList Operands(PHI->getNumIncomingValues(), nullptr);
+      for (unsigned I = 0, N = PHI->getNumIncomingValues(); I < N; ++I) {
+        Value *Op = PHI->getIncomingValue(I);
+        Operands[I] = Op;
+      }
+      if (const TreeEntry *OpTE = getTreeEntry(Operands.front()))
+        if (OpTE->isSame(Operands) && CountedOps.insert(OpTE).second)
+          if (!OpTE->ReuseShuffleIndices.empty())
+            ScalarCost += TTI::TCC_Basic * (OpTE->ReuseShuffleIndices.size() -
+                                            OpTE->Scalars.size());
+
+    return CommonCost - ScalarCost;
+    case Instruction::ExtractValue:
+    case Instruction::ExtractElement:
+      return GetCostDiff(
+          [this, ShuffleOrOp, VL, ScalarTy, CostKind](unsigned Idx) {
+            auto *I = cast<Instruction>(VL[Idx]);
+            VectorType *SrcVecTy;
             if (ShuffleOrOp == Instruction::ExtractElement) {
-            auto *EE = cast<ExtractElementInst>(V);
-            CommonCost += TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  EE->getVectorOperandType(),
-                                                  *getExtractIndex(EE));
+              auto *EE = cast<ExtractElementInst>(I);
+              SrcVecTy = EE->getVectorOperandType();
             } else {
-            --Idx;
-            CommonCost += TTI->getVectorInstrCost(Instruction::ExtractElement,
-                                                  VecTy, Idx);
-          }
-        }
+              auto *EV = cast<ExtractValueInst>(I);
+              Type *AggregateTy = EV->getAggregateOperand()->getType();
+              unsigned NumElts;
+              if (auto *ATy = dyn_cast<ArrayType>(AggregateTy))
+                NumElts = ATy->getNumElements();
+              else
+                NumElts = AggregateTy->getStructNumElements();
+              SrcVecTy = FixedVectorType::get(ScalarTy, NumElts);
             }
-      if (ShuffleOrOp == Instruction::ExtractValue) {
-        for (unsigned I = 0, E = VL.size(); I < E; ++I) {
-          auto *EI = cast<Instruction>(VL[I]);
-          // Take credit for instruction that will become dead.
-          if (EI->hasOneUse()) {
-            Instruction *Ext = EI->user_back();
+            if (I->hasOneUse()) {
+              Instruction *Ext = I->user_back();
               if ((isa<SExtInst>(Ext) || isa<ZExtInst>(Ext)) &&
                   all_of(Ext->users(),
                          [](User *U) { return isa<GetElementPtrInst>(U); })) {
                 // Use getExtractWithExtendCost() to calculate the cost of
                 // extractelement/ext pair.
-              CommonCost -= TTI->getExtractWithExtendCost(
-                  Ext->getOpcode(), Ext->getType(), VecTy, I);
-              // Add back the cost of s|zext which is subtracted separately.
-              CommonCost += TTI->getCastInstrCost(
-                  Ext->getOpcode(), Ext->getType(), EI->getType(),
+                InstructionCost Cost = TTI->getExtractWithExtendCost(
+                    Ext->getOpcode(), Ext->getType(), SrcVecTy,
+                    *getExtractIndex(I));
+                // Subtract the cost of s|zext which is subtracted separately.
+                Cost -= TTI->getCastInstrCost(
+                    Ext->getOpcode(), Ext->getType(), I->getType(),
                     TTI::getCastContextHint(Ext), CostKind, Ext);
-              continue;
-            }
-          }
-          CommonCost -=
-              TTI->getVectorInstrCost(Instruction::ExtractElement, VecTy, I);
-        }
-      } else {
-        AdjustExtractsCost(CommonCost);
+                return Cost;
               }
-      return CommonCost;
             }
+            return TTI->getVectorInstrCost(Instruction::ExtractElement,
+                                           SrcVecTy, *getExtractIndex(I));
+          },
+          [](InstructionCost CommonCost) { return CommonCost; });
     case Instruction::InsertElement: {
       assert(E->ReuseShuffleIndices.empty() &&
              "Unique insertelements only are expected.");
       auto *SrcVecTy = cast<FixedVectorType>(VL0->getType());
 
       unsigned const NumElts = SrcVecTy->getNumElements();
       unsigned const NumScalars = VL.size();
       APInt DemandedElts = APInt::getZero(NumElts);
case Instruction::InsertElement: {
           Cost += TTI->getShuffleCost(TTI::SK_PermuteSingleSrc, SrcVecTy, Mask);
         } else {
           SmallVector<int> InsertMask(NumElts);
           std::iota(InsertMask.begin(), InsertMask.end(), 0);
           for (unsigned I = 0; I < NumElts; I++) {
             if (Mask[I] != UndefMaskElem)
               InsertMask[Offset + I] = NumElts + I;
           }
           Cost +=
               TTI->getShuffleCost(TTI::SK_PermuteTwoSrc, SrcVecTy, InsertMask);
         }
       }
 
       return Cost;
     }
     case Instruction::ZExt:
     case Instruction::SExt:
     case Instruction::FPToUI:
     case Instruction::FPToSI:
     case Instruction::FPExt:
     case Instruction::PtrToInt:
     case Instruction::IntToPtr:
     case Instruction::SIToFP:
     case Instruction::UIToFP:
     case Instruction::Trunc:
     case Instruction::FPTrunc:
-    case Instruction::BitCast: {
+    case Instruction::BitCast:
+      return GetCostDiff(
+          [this, E, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
+            return TTI->getCastInstrCost(
+                E->getOpcode(), ScalarTy, VI->getOperand(0)->getType(),
+                TTI::getCastContextHint(VI), CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
             Type *SrcTy = VL0->getOperand(0)->getType();
-      InstructionCost ScalarEltCost =
-          TTI->getCastInstrCost(E->getOpcode(), ScalarTy, SrcTy,
-                                TTI::getCastContextHint(VL0), CostKind, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-
-      // Calculate the cost of this instruction.
-      InstructionCost ScalarCost = VL.size() * ScalarEltCost;
-
             auto *SrcVecTy = FixedVectorType::get(SrcTy, VL.size());
             InstructionCost VecCost = 0;
             // Check if the values are candidates to demote.
             if (!MinBWs.count(VL0) || VecTy != SrcVecTy) {
-        VecCost = CommonCost + TTI->getCastInstrCost(
-                                   E->getOpcode(), VecTy, SrcVecTy,
-                                   TTI::getCastContextHint(VL0), CostKind, VL0);
-      }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return VecCost - ScalarCost;
+              VecCost = CommonCost +
+                        TTI->getCastInstrCost(E->getOpcode(), VecTy, SrcVecTy,
+                                              TTI::getCastContextHint(VL0),
+                                              CostKind, VL0);
             }
+            return VecCost;
+          });
     case Instruction::FCmp:
     case Instruction::ICmp:
     case Instruction::Select: {
-      // Calculate the cost of this instruction.
-      InstructionCost ScalarEltCost =
-          TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy, Builder.getInt1Ty(),
-                                  CmpInst::BAD_ICMP_PREDICATE, CostKind, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-
-      // Check if all entries in VL are either compares or selects with compares
-      // as condition that have the same predicates.
-      CmpInst::Predicate VecPred = CmpInst::BAD_ICMP_PREDICATE;
-      bool First = true;
-      for (auto *V : VL) {
+      CmpInst::Predicate VecPred, SwappedVecPred;
+      auto MatchCmp = m_Cmp(VecPred, m_Value(), m_Value());
+      if (match(VL0, m_Select(MatchCmp, m_Value(), m_Value())) ||
+          match(VL0, MatchCmp))
+        SwappedVecPred = CmpInst::getSwappedPredicate(VecPred);
+      else
+        SwappedVecPred = VecPred = CmpInst::BAD_ICMP_PREDICATE;
+      return GetCostDiff(
+          [this, E, ScalarTy, VL, CostKind, &VecPred,
+           &SwappedVecPred](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
             CmpInst::Predicate CurrentPred;
             auto MatchCmp = m_Cmp(CurrentPred, m_Value(), m_Value());
-        if ((!match(V, m_Select(MatchCmp, m_Value(), m_Value())) &&
-             !match(V, MatchCmp)) ||
-            (!First && VecPred != CurrentPred)) {
-          VecPred = CmpInst::BAD_ICMP_PREDICATE;
-          break;
-        }
-        First = false;
-        VecPred = CurrentPred;
-      }
+            if ((!match(VI, m_Select(MatchCmp, m_Value(), m_Value())) &&
+                 !match(VI, MatchCmp)) ||
+                (CurrentPred != VecPred && CurrentPred != SwappedVecPred))
+              VecPred = SwappedVecPred = CmpInst::BAD_ICMP_PREDICATE;
+
+            return TTI->getCmpSelInstrCost(E->getOpcode(), ScalarTy,
+                                           Builder.getInt1Ty(), CurrentPred,
+                                           CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind,
+           &VecPred](InstructionCost CommonCost) {
+            auto *MaskTy = FixedVectorType::get(Builder.getInt1Ty(), VL.size());
 
             InstructionCost VecCost = TTI->getCmpSelInstrCost(
                 E->getOpcode(), VecTy, MaskTy, VecPred, CostKind, VL0);
-      // Check if it is possible and profitable to use min/max for selects in
-      // VL.
+            // Check if it is possible and profitable to use min/max for selects
+            // in VL.
             //
             auto IntrinsicAndUse = canConvertToMinOrMaxIntrinsic(VL);
             if (IntrinsicAndUse.first != Intrinsic::not_intrinsic) {
               IntrinsicCostAttributes CostAttrs(IntrinsicAndUse.first, VecTy,
                                                 {VecTy, VecTy});
               InstructionCost IntrinsicCost =
                   TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
-        // If the selects are the only uses of the compares, they will be dead
-        // and we can adjust the cost by removing their cost.
+              // If the selects are the only uses of the compares, they will be
+              // dead and we can adjust the cost by removing their cost.
               if (IntrinsicAndUse.second)
-          IntrinsicCost -=
-              TTI->getCmpSelInstrCost(Instruction::ICmp, VecTy, MaskTy,
-                                      CmpInst::BAD_ICMP_PREDICATE, CostKind);
+                IntrinsicCost -= TTI->getCmpSelInstrCost(
+                    Instruction::ICmp, VecTy, MaskTy,
               VecCost = std::min(VecCost, IntrinsicCost);
             }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
+            return VecCost + CommonCost;
+          });
     }
     case Instruction::FNeg:
     case Instruction::Add:
     case Instruction::FAdd:
     case Instruction::Sub:
     case Instruction::FSub:
     case Instruction::Mul:
     case Instruction::FMul:
     case Instruction::UDiv:
     case Instruction::SDiv:
     case Instruction::FDiv:
     case Instruction::URem:
     case Instruction::SRem:
     case Instruction::FRem:
     case Instruction::Shl:
     case Instruction::LShr:
     case Instruction::AShr:
     case Instruction::And:
     case Instruction::Or:
-    case Instruction::Xor: {
+    case Instruction::Xor:
+    case Instruction::GetElementPtr: {
+      unsigned Opcode = ShuffleOrOp == Instruction::GetElementPtr
+                            ? Instruction::Add
+                            : ShuffleOrOp;
       // Certain instructions can be cheaper to vectorize if they have a
       // constant second vector operand.
-      TargetTransformInfo::OperandValueKind Op1VK =
-          TargetTransformInfo::OK_AnyValue;
       TargetTransformInfo::OperandValueKind Op2VK =
           TargetTransformInfo::OK_UniformConstantValue;
-      TargetTransformInfo::OperandValueProperties Op1VP =
-          TargetTransformInfo::OP_None;
       TargetTransformInfo::OperandValueProperties Op2VP =
           TargetTransformInfo::OP_PowerOf2;
-
-      // If all operands are exactly the same ConstantInt then set the
-      // operand kind to OK_UniformConstantValue.
-      // If instead not all operands are constants, then set the operand kind
-      // to OK_AnyValue. If all operands are constants but not the same,
-      // then set the operand kind to OK_NonUniformConstantValue.
       ConstantInt *CInt0 = nullptr;
-      for (unsigned i = 0, e = VL.size(); i < e; ++i) {
-        const Instruction *I = cast<Instruction>(VL[i]);
-        unsigned OpIdx = isa<BinaryOperator>(I) ? 1 : 0;
-        ConstantInt *CInt = dyn_cast<ConstantInt>(I->getOperand(OpIdx));
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind, &Op2VK, &Op2VP, &CInt0,
+           Opcode](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
+            unsigned OpIdx = isa<BinaryOperator, GetElementPtrInst>(VI) ? 1 : 0;
+            TargetTransformInfo::OperandValueKind ScalarOp2VK =
+                TargetTransformInfo::OK_UniformConstantValue;
+            TargetTransformInfo::OperandValueProperties ScalarOp2VP =
+                TargetTransformInfo::OP_PowerOf2;
+            ConstantInt *CInt = dyn_cast<ConstantInt>(VI->getOperand(OpIdx));
             if (!CInt) {
-          Op2VK = TargetTransformInfo::OK_AnyValue;
-          Op2VP = TargetTransformInfo::OP_None;
-          break;
-        }
-        if (Op2VP == TargetTransformInfo::OP_PowerOf2 &&
-            !CInt->getValue().isPowerOf2())
-          Op2VP = TargetTransformInfo::OP_None;
-        if (i == 0) {
+              ScalarOp2VK = Op2VK = TargetTransformInfo::OK_AnyValue;
+              ScalarOp2VP = Op2VP = TargetTransformInfo::OP_None;
+            } else {
+              if (!CInt->getValue().isPowerOf2())
+                ScalarOp2VP = Op2VP = TargetTransformInfo::OP_None;
+              if (Op2VK == TargetTransformInfo::OK_UniformConstantValue) {
+                if (!CInt0)
                   CInt0 = CInt;
-          continue;
-        }
-        if (CInt0 != CInt)
+                else if (CInt0 != CInt)
                   Op2VK = TargetTransformInfo::OK_NonUniformConstantValue;
               }
-
-      SmallVector<const Value *, 4> Operands(VL0->operand_values());
-      InstructionCost ScalarEltCost =
-          TTI->getArithmeticInstrCost(E->getOpcode(), ScalarTy, CostKind, Op1VK,
-                                      Op2VK, Op1VP, Op2VP, Operands, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      InstructionCost VecCost =
-          TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind, Op1VK,
-                                      Op2VK, Op1VP, Op2VP, Operands, VL0);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
-    }
-    case Instruction::GetElementPtr: {
-      TargetTransformInfo::OperandValueKind Op1VK =
-          TargetTransformInfo::OK_AnyValue;
-      TargetTransformInfo::OperandValueKind Op2VK =
-          TargetTransformInfo::OK_UniformConstantValue;
-
-      InstructionCost ScalarEltCost = TTI->getArithmeticInstrCost(
-          Instruction::Add, ScalarTy, CostKind, Op1VK, Op2VK);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
             }
-      InstructionCost ScalarCost = VecTy->getNumElements() * ScalarEltCost;
-      InstructionCost VecCost = TTI->getArithmeticInstrCost(
-          Instruction::Add, VecTy, CostKind, Op1VK, Op2VK);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
-    }
-    case Instruction::Load: {
-      // Cost of wide load - cost of scalar loads.
-      Align Alignment = cast<LoadInst>(VL0)->getAlign();
-      InstructionCost ScalarEltCost = TTI->getMemoryOpCost(
-          Instruction::Load, ScalarTy, Alignment, 0, CostKind, VL0);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
-      }
-      InstructionCost ScalarLdCost = VecTy->getNumElements() * ScalarEltCost;
+            SmallVector<const Value *> Operands(VI->operand_values());
+            return TTI->getArithmeticInstrCost(
+                Opcode, ScalarTy, CostKind, TargetTransformInfo::OK_AnyValue,
+                ScalarOp2VK, TargetTransformInfo::OP_None, ScalarOp2VP,
+                Operands, VI);
+          },
+          [this, VecTy, CostKind, E, VL0, &Op2VK, &Op2VP,
+           Opcode](InstructionCost CommonCost) {
+            if (Op2VK == TargetTransformInfo::OK_AnyValue &&
+                isSplat(E->getOperand(
+                    isa<BinaryOperator, GetElementPtrInst>(VL0) ? 1 : 0)))
+              Op2VK = TargetTransformInfo::OK_UniformValue;
+            return TTI->getArithmeticInstrCost(
+                       Opcode, VecTy, CostKind,
+                       TargetTransformInfo::OK_AnyValue, Op2VK,
+                       TargetTransformInfo::OP_None, Op2VP) +
+                   CommonCost;
+          });
+    case Instruction::Load:
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<LoadInst>(VL[Idx]);
+            return TTI->getMemoryOpCost(
+                Instruction::Load, ScalarTy, VI->getAlign(),
+                VI->getPointerAddressSpace(), CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
+            auto *LI0 = cast<LoadInst>(VL0);
             InstructionCost VecLdCost;
             if (E->State == TreeEntry::Vectorize) {
-        VecLdCost = TTI->getMemoryOpCost(Instruction::Load, VecTy, Alignment, 0,
-                                         CostKind, VL0);
-      } else {
-        assert(E->State == TreeEntry::ScatterVectorize && "Unknown EntryState");
-        Align CommonAlignment = Alignment;
+              VecLdCost = TTI->getMemoryOpCost(
+                  Instruction::Load, VecTy, LI0->getAlign(),
+                  LI0->getPointerAddressSpace(), CostKind);
+            } else {
+              assert(E->State == TreeEntry::ScatterVectorize &&
+                     "Unknown EntryState");
+              Align CommonAlignment = LI0->getAlign();
               for (Value *V : VL)
-          CommonAlignment =
-              commonAlignment(CommonAlignment, cast<LoadInst>(V)->getAlign());
+                CommonAlignment = commonAlignment(
+                    CommonAlignment, cast<LoadInst>(V)->getAlign());
               VecLdCost = TTI->getGatherScatterOpCost(
-            Instruction::Load, VecTy, cast<LoadInst>(VL0)->getPointerOperand(),
-            /*VariableMask=*/false, CommonAlignment, CostKind, VL0);
+                  Instruction::Load, VecTy, LI0->getPointerOperand(),
+                  /*VariableMask=*/false, CommonAlignment, CostKind);
             }
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecLdCost, ScalarLdCost));
-      return CommonCost + VecLdCost - ScalarLdCost;
-    }
-    case Instruction::Store: {
+            return VecLdCost + CommonCost;
+          });
+    case Instruction::Store:
+      return GetCostDiff(
+          [this, ScalarTy, VL, CostKind](unsigned Idx) {
+            auto *VI = cast<StoreInst>(VL[Idx]);
+            return TTI->getMemoryOpCost(
+                Instruction::Store, ScalarTy, VI->getAlign(),
+                VI->getPointerAddressSpace(), CostKind, VI);
+          },
+          [this, VL, VL0, VecTy, E, CostKind](InstructionCost CommonCost) {
             // We know that we can merge the stores. Calculate the cost.
             bool IsReorder = !E->ReorderIndices.empty();
-      auto *SI =
-          cast<StoreInst>(IsReorder ? VL[E->ReorderIndices.front()] : VL0);
-      Align Alignment = SI->getAlign();
-      InstructionCost ScalarEltCost = TTI->getMemoryOpCost(
-          Instruction::Store, ScalarTy, Alignment, 0, CostKind, VL0);
-      InstructionCost ScalarStCost = VecTy->getNumElements() * ScalarEltCost;
-      InstructionCost VecStCost = TTI->getMemoryOpCost(
-          Instruction::Store, VecTy, Alignment, 0, CostKind, VL0);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecStCost, ScalarStCost));
-      return CommonCost + VecStCost - ScalarStCost;
-    }
+            auto *SI = cast<StoreInst>(IsReorder ? VL[E->ReorderIndices.front()]
+                                                 : VL0);
+            return TTI->getMemoryOpCost(
+                       Instruction::Store, VecTy, SI->getAlign(),
+                       SI->getPointerAddressSpace(), CostKind) +
+                   CommonCost;
+          });
     case Instruction::Call: {
-      CallInst *CI = cast<CallInst>(VL0);
+      return GetCostDiff(
+          [this, VL, CostKind](unsigned Idx) {
+            auto *CI = cast<CallInst>(VL[Idx]);
             Intrinsic::ID ID = getVectorIntrinsicIDForCall(CI, TLI);
-
-      // Calculate the cost of the scalar and vector calls.
+            if (ID != Intrinsic::not_intrinsic) {
               IntrinsicCostAttributes CostAttrs(ID, *CI, 1);
-      InstructionCost ScalarEltCost =
-          TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
-      if (NeedToShuffleReuses) {
-        CommonCost -= (EntryVF - VL.size()) * ScalarEltCost;
+              return TTI->getIntrinsicInstrCost(CostAttrs, CostKind);
             }
-      InstructionCost ScalarCallCost = VecTy->getNumElements() * ScalarEltCost;
-
+            return TTI->getCallInstrCost(
+                CI->getCalledFunction(), CI->getFunctionType()->getReturnType(),
+                CI->getFunctionType()->params(), CostKind);
+          },
+          [this, VL0, VecTy](InstructionCost CommonCost) {
+            auto *CI = cast<CallInst>(VL0);
             auto VecCallCosts = getVectorCallCosts(CI, VecTy, TTI, TLI);
-      InstructionCost VecCallCost =
-          std::min(VecCallCosts.first, VecCallCosts.second);
-
-      LLVM_DEBUG(dbgs() << "SLP: Call cost " << VecCallCost - ScalarCallCost
-                        << " (" << VecCallCost << "-" << ScalarCallCost << ")"
-                        << " for " << *CI << "\n");
-
-      return CommonCost + VecCallCost - ScalarCallCost;
+            return std::min(VecCallCosts.first, VecCallCosts.second) +
+                   CommonCost;
+          });
     }
     case Instruction::ShuffleVector: {
       assert(E->isAltShuffle() &&
              ((Instruction::isBinaryOp(E->getOpcode()) &&
                Instruction::isBinaryOp(E->getAltOpcode())) ||
               (Instruction::isCast(E->getOpcode()) &&
                Instruction::isCast(E->getAltOpcode()))) &&
              "Invalid Shuffle Vector Operand");
-      InstructionCost ScalarCost = 0;
-      if (NeedToShuffleReuses) {
-        for (unsigned Idx : E->ReuseShuffleIndices) {
-          Instruction *I = cast<Instruction>(VL[Idx]);
-          CommonCost -= TTI->getInstructionCost(I, CostKind);
-        }
-        for (Value *V : VL) {
-          Instruction *I = cast<Instruction>(V);
-          CommonCost += TTI->getInstructionCost(I, CostKind);
-        }
-      }
-      for (Value *V : VL) {
-        Instruction *I = cast<Instruction>(V);
-        assert(E->isOpcodeOrAlt(I) && "Unexpected main/alternate opcode");
-        ScalarCost += TTI->getInstructionCost(I, CostKind);
-      }
-      // VecCost is equal to sum of the cost of creating 2 vectors
-      // and the cost of creating shuffle.
-      InstructionCost VecCost = 0;
       // Try to find the previous shuffle node with the same operands and same
       // main/alternate ops.
       auto &&TryFindNodeWithEqualOperands = [this, E]() {
         for (const std::unique_ptr<TreeEntry> &TE : VectorizableTree) {
           if (TE.get() == E)
             break;
           if (TE->isAltShuffle() &&
               ((TE->getOpcode() == E->getOpcode() &&
                 TE->getAltOpcode() == E->getAltOpcode()) ||
                (TE->getOpcode() == E->getAltOpcode() &&
                 TE->getAltOpcode() == E->getOpcode())) &&
               TE->hasEqualOperands(*E))
             return true;
         }
         return false;
       };
+      // Need to clear CommonCost since the final shuffle cost is included into
+      // vector cost.
+      return GetCostDiff(
+          [this, VL, CostKind, E](unsigned Idx) {
+            auto *VI = cast<Instruction>(VL[Idx]);
+            assert(E->isOpcodeOrAlt(VI) && "Unexpected main/alternate opcode");
+            (void)E;
+            return TTI->getInstructionCost(VI, CostKind);
+          },
+          [this, VecTy, E, CostKind, VL, FinalVecTy,
+           &TryFindNodeWithEqualOperands](InstructionCost) {
+            // VecCost is equal to sum of the cost of creating 2 vectors
+            // and the cost of creating shuffle.
+            InstructionCost VecCost = 0;
             if (TryFindNodeWithEqualOperands()) {
               LLVM_DEBUG({
                 dbgs() << "SLP: diamond match for alternate node found.\n";
                 E->dump();
               });
               // No need to add new vector costs here since we're going to reuse
               // same main/alternate vector ops, just do different shuffling.
             } else if (Instruction::isBinaryOp(E->getOpcode())) {
-        VecCost = TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind);
+              VecCost =
+                  TTI->getArithmeticInstrCost(E->getOpcode(), VecTy, CostKind);
               VecCost += TTI->getArithmeticInstrCost(E->getAltOpcode(), VecTy,
                                                      CostKind);
             } else {
               Type *Src0SclTy = E->getMainOp()->getOperand(0)->getType();
               Type *Src1SclTy = E->getAltOp()->getOperand(0)->getType();
               auto *Src0Ty = FixedVectorType::get(Src0SclTy, VL.size());
               auto *Src1Ty = FixedVectorType::get(Src1SclTy, VL.size());
-        VecCost = TTI->getCastInstrCost(E->getOpcode(), VecTy, Src0Ty,
+              VecCost =
+                  TTI->getCastInstrCost(E->getOpcode(), VecTy, Src0Ty,
                                         TTI::CastContextHint::None, CostKind);
-        VecCost += TTI->getCastInstrCost(E->getAltOpcode(), VecTy, Src1Ty,
-                                         TTI::CastContextHint::None, CostKind);
+              VecCost +=
+                  TTI->getCastInstrCost(E->getAltOpcode(), VecTy, Src1Ty,
             }
-
             SmallVector<int> Mask;
             buildSuffleEntryMask(
                 E->Scalars, E->ReorderIndices, E->ReuseShuffleIndices,
                 [E](Instruction *I) {
-            assert(E->isOpcodeOrAlt(I) && "Unexpected main/alternate opcode");
+                  assert(E->isOpcodeOrAlt(I) &&
+                         "Unexpected main/alternate opcode");
                   return I->getOpcode() == E->getAltOpcode();
                 },
                 Mask);
-      CommonCost =
-          TTI->getShuffleCost(TargetTransformInfo::SK_Select, FinalVecTy, Mask);
-      LLVM_DEBUG(dumpTreeCosts(E, CommonCost, VecCost, ScalarCost));
-      return CommonCost + VecCost - ScalarCost;
+            VecCost += TTI->getShuffleCost(TargetTransformInfo::SK_Select,
+                                           FinalVecTy, Mask);
+            return VecCost;
+          });
     }
     default:
       llvm_unreachable("Unknown instruction");
   }
 }
 
 bool BoUpSLP::isFullyVectorizableTinyTree(bool ForReduction) const {
   LLVM_DEBUG(dbgs() << "SLP: Check whether the tree with height "
for (ExternalUser &EU : ExternalUses) {
     // removed as well).
     if (EphValues.count(EU.User))
       continue;
 
     // No extract cost for vector "scalar"
     if (isa<FixedVectorType>(EU.Scalar->getType()))
       continue;
 
-    // Already counted the cost for external uses when tried to adjust the cost
-    // for extractelements, no need to add it again.
-    if (isa<ExtractElementInst>(EU.Scalar))
-      continue;
-
     // If found user is an insertelement, do not calculate extract cost but try
     // to detect it as a final shuffled/identity match.
     if (auto *VU = dyn_cast_or_null<InsertElementInst>(EU.User)) {
       if (auto *FTy = dyn_cast<FixedVectorType>(VU->getType())) {
         Optional<int> InsertIdx = getInsertIndex(VU, 0);
         if (!InsertIdx || *InsertIdx == UndefMaskElem)
           continue;
         auto *It = find_if(FirstUsers, [VU](Value *V) {

llvm/test/Transforms/SLPVectorizer/AArch64/horizontal.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -slp-vectorizer -slp-threshold=-5 -S -pass-remarks-output=%t < %s | FileCheck %s
+; RUN: opt -slp-vectorizer -slp-threshold=-3 -S -pass-remarks-output=%t < %s | FileCheck %s
 ; RUN: cat %t | FileCheck -check-prefix=YAML %s
 
 
 ; FIXME: The threshold is changed to keep this test case a bit smaller.
 ; The AArch64 cost model should not give such high costs to select statements.
 
 target datalayout = "e-m:e-i64:64-i128:128-n32:64-S128"
 target triple = "aarch64--linux"

llvm/test/Transforms/SLPVectorizer/AArch64/vectorizable-selects-min-max.ll

entry:
   store i32 %s.3, i32* %gep.3, align 4
 
   ret void
 }
 
 define void @select_ule_ugt_mix_4xi32(i32* %ptr, i32 %x) {
 ; CHECK-LABEL: @select_ule_ugt_mix_4xi32(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[L_0:%.*]] = load i32, i32* [[PTR:%.*]], align 4
-; CHECK-NEXT:    [[CMP_0:%.*]] = icmp ult i32 [[L_0]], 16383
-; CHECK-NEXT:    [[S_0:%.*]] = select i1 [[CMP_0]], i32 [[L_0]], i32 16383
-; CHECK-NEXT:    store i32 [[S_0]], i32* [[PTR]], align 4
-; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 1
-; CHECK-NEXT:    [[L_1:%.*]] = load i32, i32* [[GEP_1]], align 4
-; CHECK-NEXT:    [[CMP_1:%.*]] = icmp ugt i32 [[L_1]], 16383
-; CHECK-NEXT:    [[S_1:%.*]] = select i1 [[CMP_1]], i32 [[L_1]], i32 16383
-; CHECK-NEXT:    store i32 [[S_1]], i32* [[GEP_1]], align 4
+; CHECK-NEXT:    [[GEP_1:%.*]] = getelementptr inbounds i32, i32* [[PTR:%.*]], i32 1
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[PTR]] to <2 x i32>*
+; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x i32>, <2 x i32>* [[TMP0]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <2 x i32> [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x i32> <i32 poison, i32 16383>, i32 [[TMP2]], i32 0
+; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x i32> [[TMP1]], i32 1
+; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x i32> <i32 16383, i32 poison>, i32 [[TMP4]], i32 1
+; CHECK-NEXT:    [[TMP6:%.*]] = icmp ult <2 x i32> [[TMP3]], [[TMP5]]
+; CHECK-NEXT:    [[TMP7:%.*]] = select <2 x i1> [[TMP6]], <2 x i32> [[TMP1]], <2 x i32> <i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP8:%.*]] = bitcast i32* [[PTR]] to <2 x i32>*
+; CHECK-NEXT:    store <2 x i32> [[TMP7]], <2 x i32>* [[TMP8]], align 4
 ; CHECK-NEXT:    [[GEP_2:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 2
-; CHECK-NEXT:    [[L_2:%.*]] = load i32, i32* [[GEP_2]], align 4
-; CHECK-NEXT:    [[CMP_2:%.*]] = icmp ult i32 [[L_2]], 16383
-; CHECK-NEXT:    [[S_2:%.*]] = select i1 [[CMP_2]], i32 [[L_2]], i32 16383
-; CHECK-NEXT:    store i32 [[S_2]], i32* [[GEP_2]], align 4
 ; CHECK-NEXT:    [[GEP_3:%.*]] = getelementptr inbounds i32, i32* [[PTR]], i32 3
-; CHECK-NEXT:    [[L_3:%.*]] = load i32, i32* [[GEP_3]], align 4
-; CHECK-NEXT:    [[CMP_3:%.*]] = icmp ugt i32 [[L_3]], 16383
-; CHECK-NEXT:    [[S_3:%.*]] = select i1 [[CMP_3]], i32 [[L_3]], i32 16383
-; CHECK-NEXT:    store i32 [[S_3]], i32* [[GEP_3]], align 4
+; CHECK-NEXT:    [[TMP9:%.*]] = bitcast i32* [[GEP_2]] to <2 x i32>*
+; CHECK-NEXT:    [[TMP10:%.*]] = load <2 x i32>, <2 x i32>* [[TMP9]], align 4
+; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <2 x i32> [[TMP10]], i32 0
+; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <2 x i32> <i32 poison, i32 16383>, i32 [[TMP11]], i32 0
+; CHECK-NEXT:    [[TMP13:%.*]] = extractelement <2 x i32> [[TMP10]], i32 1
+; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <2 x i32> <i32 16383, i32 poison>, i32 [[TMP13]], i32 1
+; CHECK-NEXT:    [[TMP15:%.*]] = icmp ult <2 x i32> [[TMP12]], [[TMP14]]
+; CHECK-NEXT:    [[TMP16:%.*]] = select <2 x i1> [[TMP15]], <2 x i32> [[TMP10]], <2 x i32> <i32 16383, i32 16383>
+; CHECK-NEXT:    [[TMP17:%.*]] = bitcast i32* [[GEP_2]] to <2 x i32>*
+; CHECK-NEXT:    store <2 x i32> [[TMP16]], <2 x i32>* [[TMP17]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %l.0 = load i32, i32* %ptr
   %cmp.0 = icmp ult i32 %l.0, 16383
   %s.0 = select i1 %cmp.0, i32 %l.0, i32 16383
   store i32 %s.0, i32* %ptr, align 4
 

llvm/test/Transforms/SLPVectorizer/X86/PR32086.ll

;
   store i64 %t1, i64* %arrayidx4, align 8
   store i64 %t0, i64* %arrayidx5, align 8
   ret void
 }
 
 define void @i64_simplifiedi_extract(i64* noalias %st, i64* noalias %ld) {
 ; CHECK-LABEL: @i64_simplifiedi_extract(
 ; CHECK-NEXT:    [[ARRAYIDX1:%.*]] = getelementptr inbounds i64, i64* [[LD:%.*]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i64* [[LD]] to <2 x i64>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i64>, <2 x i64>* [[TMP1]], align 8
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i64> [[TMP2]], <2 x i64> poison, <4 x i32> <i32 0, i32 0, i32 0, i32 1>
+; CHECK-NEXT:    [[T0:%.*]] = load i64, i64* [[LD]], align 8
+; CHECK-NEXT:    [[T1:%.*]] = load i64, i64* [[ARRAYIDX1]], align 8
 ; CHECK-NEXT:    [[ARRAYIDX3:%.*]] = getelementptr inbounds i64, i64* [[ST:%.*]], i64 1
 ; CHECK-NEXT:    [[ARRAYIDX4:%.*]] = getelementptr inbounds i64, i64* [[ST]], i64 2
 ; CHECK-NEXT:    [[ARRAYIDX5:%.*]] = getelementptr inbounds i64, i64* [[ST]], i64 3
-; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i64* [[ST]] to <4 x i64>*
-; CHECK-NEXT:    store <4 x i64> [[SHUFFLE]], <4 x i64>* [[TMP3]], align 8
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <4 x i64> [[SHUFFLE]], i32 3
-; CHECK-NEXT:    store i64 [[TMP4]], i64* [[LD]], align 8
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ST]], align 8
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ARRAYIDX3]], align 8
+; CHECK-NEXT:    store i64 [[T0]], i64* [[ARRAYIDX4]], align 8
+; CHECK-NEXT:    store i64 [[T1]], i64* [[ARRAYIDX5]], align 8
+; CHECK-NEXT:    store i64 [[T1]], i64* [[LD]], align 8
 ; CHECK-NEXT:    ret void
 ;
   %arrayidx1 = getelementptr inbounds i64, i64* %ld, i64 1
 
   %t0 = load i64, i64* %ld, align 8
   %t1 = load i64, i64* %arrayidx1, align 8
 
   %arrayidx3 = getelementptr inbounds i64, i64* %st, i64 1

llvm/test/Transforms/SLPVectorizer/X86/PR39774.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt -slp-vectorizer -S < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake -slp-threshold=-6 | FileCheck %s --check-prefix=CHECK
+; RUN: opt -slp-vectorizer -S < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake -slp-threshold=-5 | FileCheck %s --check-prefix=CHECK
 ; RUN: opt -slp-vectorizer -S < %s -mtriple=x86_64-unknown-linux-gnu -mcpu=skylake -slp-threshold=-7 -slp-min-tree-size=6 | FileCheck %s --check-prefix=FORCE_REDUCTION
 
 define void @Test(i32) {
 ; CHECK-LABEL: @Test(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP10:%.*]], [[LOOP]] ], [ zeroinitializer, [[ENTRY:%.*]] ]

llvm/test/Transforms/SLPVectorizer/X86/vectorize-reorder-reuse.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -slp-vectorizer -S -mtriple=x86_64-unknown-linux-gnu -mcpu=bdver2 < %s | FileCheck %s
 
 define i32 @foo(i32* nocapture readonly %arr, i32 %a1, i32 %a2, i32 %a3, i32 %a4, i32 %a5, i32 %a6, i32 %a7, i32 %a8) {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[ARRAYIDX:%.*]] = getelementptr inbounds i32, i32* [[ARR:%.*]], i64 1
-; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i32* [[ARR]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x i32>, <2 x i32>* [[TMP0]], align 4
-; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <2 x i32> [[TMP1]], <2 x i32> poison, <8 x i32> <i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 0, i32 0>
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <8 x i32> poison, i32 [[A1:%.*]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <8 x i32> [[TMP2]], i32 [[A2:%.*]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <8 x i32> [[TMP3]], i32 [[A3:%.*]], i32 2
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <8 x i32> [[TMP4]], i32 [[A4:%.*]], i32 3
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <8 x i32> [[TMP5]], i32 [[A5:%.*]], i32 4
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <8 x i32> [[TMP6]], i32 [[A6:%.*]], i32 5
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <8 x i32> [[TMP7]], i32 [[A7:%.*]], i32 6
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <8 x i32> [[TMP8]], i32 [[A8:%.*]], i32 7
-; CHECK-NEXT:    [[TMP10:%.*]] = add <8 x i32> [[SHUFFLE]], [[TMP9]]
-; CHECK-NEXT:    [[TMP11:%.*]] = call i32 @llvm.vector.reduce.umin.v8i32(<8 x i32> [[TMP10]])
-; CHECK-NEXT:    ret i32 [[TMP11]]
+; CHECK-NEXT:    [[TMP0:%.*]] = load i32, i32* [[ARRAYIDX]], align 4
+; CHECK-NEXT:    [[ADD:%.*]] = add i32 [[TMP0]], [[A1:%.*]]
+; CHECK-NEXT:    [[ADD2:%.*]] = add i32 [[TMP0]], [[A2:%.*]]
+; CHECK-NEXT:    [[ADD4:%.*]] = add i32 [[TMP0]], [[A3:%.*]]
+; CHECK-NEXT:    [[ADD6:%.*]] = add i32 [[TMP0]], [[A4:%.*]]
+; CHECK-NEXT:    [[ADD8:%.*]] = add i32 [[TMP0]], [[A5:%.*]]
+; CHECK-NEXT:    [[ADD10:%.*]] = add i32 [[TMP0]], [[A6:%.*]]
+; CHECK-NEXT:    [[TMP1:%.*]] = load i32, i32* [[ARR]], align 4
+; CHECK-NEXT:    [[ADD12:%.*]] = add i32 [[TMP1]], [[A7:%.*]]
+; CHECK-NEXT:    [[ADD14:%.*]] = add i32 [[TMP1]], [[A8:%.*]]
+; CHECK-NEXT:    [[CMP:%.*]] = icmp ult i32 [[ADD]], [[ADD2]]
+; CHECK-NEXT:    [[COND:%.*]] = select i1 [[CMP]], i32 [[ADD]], i32 [[ADD2]]
+; CHECK-NEXT:    [[CMP15:%.*]] = icmp ult i32 [[COND]], [[ADD4]]
+; CHECK-NEXT:    [[COND19:%.*]] = select i1 [[CMP15]], i32 [[COND]], i32 [[ADD4]]
+; CHECK-NEXT:    [[CMP20:%.*]] = icmp ult i32 [[COND19]], [[ADD6]]
+; CHECK-NEXT:    [[COND24:%.*]] = select i1 [[CMP20]], i32 [[COND19]], i32 [[ADD6]]
+; CHECK-NEXT:    [[CMP25:%.*]] = icmp ult i32 [[COND24]], [[ADD8]]
+; CHECK-NEXT:    [[COND29:%.*]] = select i1 [[CMP25]], i32 [[COND24]], i32 [[ADD8]]
+; CHECK-NEXT:    [[CMP30:%.*]] = icmp ult i32 [[COND29]], [[ADD10]]
+; CHECK-NEXT:    [[COND34:%.*]] = select i1 [[CMP30]], i32 [[COND29]], i32 [[ADD10]]
+; CHECK-NEXT:    [[CMP35:%.*]] = icmp ult i32 [[COND34]], [[ADD12]]
+; CHECK-NEXT:    [[COND39:%.*]] = select i1 [[CMP35]], i32 [[COND34]], i32 [[ADD12]]
+; CHECK-NEXT:    [[CMP40:%.*]] = icmp ult i32 [[COND39]], [[ADD14]]
+; CHECK-NEXT:    [[COND44:%.*]] = select i1 [[CMP40]], i32 [[COND39]], i32 [[ADD14]]
+; CHECK-NEXT:    ret i32 [[COND44]]
 ;
 entry:
   %arrayidx = getelementptr inbounds i32, i32* %arr, i64 1
   %0 = load i32, i32* %arrayidx, align 4
   %add = add i32 %0, %a1
   %add2 = add i32 %0, %a2
   %add4 = add i32 %0, %a3
   %add6 = add i32 %0, %a4