[SLP] Try to match reductions before trying to vectorize a vector build sequence.

[vdmitrie]

Summary

This patch changes order of searching for reductions vs other vectorization possibilities.
The idea is if we do not match a reduction it won't be harmful for further attempts to
find vectorizable operations on a vector build sequences. But doing it in the opposite
order we have good chance to ruin opportunity to match a reduction later.
We also don't want to try vectorizing binary operations too early as 2-way vectorization
may effectively prohibit wider ones leading to producing less effective code.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11710–11713

Can you put this and corresponding changes to the separate NFC patch?

12031–12032

Formatting

12044–12048

Outline this code to the separate function? Same code is used in 2 places.

[vdmitrie]

Address review comments.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11710–11713

Split into https://reviews.llvm.org/D132603

[vdmitrie]

update due to nfc split

[vdmitrie]

update

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11830

Why did you decide to remove these functions and embed their code instead?

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11830

Both methods are only called (and make sense) in context of vector build sequence and that is already ensured by skipping them if findBuildAggregate did not detect a sequence in the loop where they used. If we keep the call inside the methods it will be redundant. If remove that call from both of the methods they would become non self sufficient , i.e. would rely on assumption that they are called within certain context. But assumptions are dangerous things.
As all that context is contained within just a small loop I decided that for better clarity it is be better to submerge these methods into the loop rather than keep them with assumptions (or add an ugly assertions to show that assumption).

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

12011–12015

I think you're doing it too early, need to do it after the vectorizeHorReduction, which should start with I, otherwise we may miss single insertelement instruction which has reduction.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

12011–12015

You probably meant to make another call to vectorizeHorReduction if we did not match a vector build. I'll try to reproduce the situation you described in a test case.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

12011–12015

I mean for horizontal reductions you don't need to perform this, you can do it after the horizontal reduction matching. Plus continue is too early, if findBuildAggregate fails.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

12011–12015

The whole point of the patch was to make sure to visit every vector build operand early. In order to do that we need the call. But I also see what you mean.

[vdmitrie]

Address comments + added coverage test case.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

Why do you want still call this after findBuildAggregate?

llvm/test/Transforms/SLPVectorizer/X86/redux-feed-buildvector.ll

117

Please, precommit the test

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

In order to match hor reduction on build vector operands. Can you offer other approach to do the same?

llvm/test/Transforms/SLPVectorizer/X86/redux-feed-buildvector.ll

117

The vectorizer does not change its behavior on this test with the patch . I can commit it separately if you want but that won't be a pre-commit.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

But vectorizeRootInstruction already does this, why do you want to do it twice, here and in vectorizeRootInstruction? It increases compile time.
Can we just call vectorizeRootInstruction(nullptr, I, BB, R, TTI); and vectorizeBuildVector after? And remove this loop and final OpsChanged |= tryToVectorize(PostponedInsts, R); call?

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

But vectorizeRootInstruction already does this,

Nope. This is exactly what does not happen after your commit https://reviews.llvm.org/rG7ea03f0b4e4e
Before the commit we traversed through operands of insertelements when collected instructions for work stack and were able to locate all reductions at once.

Can we just call vectorizeRootInstruction(nullptr, I, BB, R, TTI); and vectorizeBuildVector after?

Nope. vectorizeRootInstruction may too early create 2-way vectorizations. We only want to call it after trying on with wider VFs with tryToVectorizeList.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

Then you don't need to call vectorizeRootInstruction. Also, why do you need to find operands of the buildvector sequence? Just call vectorizeHorReduction for each insert instruction in the loop and in the second loop call findBuildAggregate and all other stuff for buildvector and postponed insts.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

Just call vectorizeHorReduction for each insert ...

I don't like this approach. I'd rather enable vectorizeHorReduction to traverse through insertelement operands. BTW you did not explain why you suppressed that.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

To save compile time, to avoid analysis of the same instructions several times.
Why you don't like it?

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

To save compile time, to avoid analysis of the same instructions several times.
Why you don't like it?

You are probably talking about https://reviews.llvm.org/D114171 when saying "compile time". But I'm about the different change. Note that the patch that was last reviewed in https://reviews.llvm.org/D114171
does not match what was actually committed (with the reference to differential revision). I'm specifically concerned about this piece of code:

// Try to vectorize operands.
// Continue analysis for the instruction from the same basic block only to
// save compile time.
if (++Level < RecursionMaxDepth)
  for (auto *Op : Inst->operand_values())
    if (VisitedInstrs.insert(Op).second)
      if (auto *I = dyn_cast<Instruction>(Op))
        // Do not try to vectorize CmpInst operands,  this is done
        // separately.
        if (!isa<PHINode>(I) && !isa<CmpInst>(I) && !R.isDeleted(I) &&
            I->getParent() == BB)
          Stack.emplace(I, Level);

where you changed to avoid traversing through insertelement operands:

if (!isa<PHINode, CmpInst, InsertElementInst, InsertValueInst>(I) &&

I'm not convinced that this specific change worth extra compile time.

I can agree that sticking to buildvector is not the best approach, but not traversing the operands early does not save compile time. It only leads to postponing the action for another visit but with much lower chances for vectorizing it the right way.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

It was changed after several comments regarding compile time.
There are 2 problems with the original code.

1. Compile time in case of unsuccessful attempts. We repeat analysis of same instructions at least twice, if not more.
2. It does not preseve the analysis order for postponable instructions, some of them gets analyzed too early. It breaks internal logic, makes it harder for the perf abd bug analysis. And may lead to a too early vectorization.

The problem that insertelement is also the operand and it maybe a part of the another build vector sequence. In case of too early vectorization attempt we may end up with 2 x vectorization of the operand of the insertelement ibstruction instead of possible wider buildvector sequence.

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

In case of too early vectorization attempt we may end up with 2 x vectorization of the operand of the insertelement ibstruction instead of possible wider buildvector sequence.

Now vectorizeHorRedcution is decoupled from 2-way vectorization. we just need to call vectorizeHorRedcution instead of vectorizeRootInstruction in order to avoid unwanted 2-way vectors early.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

And it is good. But then there is a call for vectorizeRootInstruction, which just repeats almost same stuff again. If you want to keep current implementation, I suggest to remove the call for vectorizeRootInstruction and call it if findBuildAggregate fails only, like

if (!findBuildAggregate(I, TTI, BuildVectorOpds, BuildVectorInsts))
    return vectorizeRootInstruction(...);

[vdmitrie]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

11846–11849

Yep. I was thinking about that too.
I also tried to add an extra loop over instructions just targeting reductions in vectorizeSimpleInstructions:

 // pass1 - try to vectorize reductions only
 for (auto *I : reverse(Instructions)) {
   if (R.isDeleted(I))
     continue;
   if (isa<CmpInst>(I)) {
     PostponedCmps.push_back(I);
    continue;
   }
   OpsChanged |= vectorizeHorReduction(nullptr, I, BB, R, TTI, PostponedInsts);
 }
 // pass2 - try to match and vectorize a buildvector sequence.
 for (auto *I : reverse(Instructions)) {
  if (R.isDeleted(I) || isa<CmpInst>(I))
     continue;
  if (auto *LastInsertValue = dyn_cast<InsertValueInst>(I)) {
    OpsChanged |= vectorizeInsertValueInst(LastInsertValue, BB, R);
  } else if (auto *LastInsertElem = dyn_cast<InsertElementInst>(I)) {
    OpsChanged |= vectorizeInsertElementInst(LastInsertElem, BB, R);
   }
}
 // Now try to vectorize postponed instructions.
 OpsChanged |= tryToVectorize(PostponedInsts, R);

Here we don't stick to vector build for reductions and have no interface changes.
What's your opinion in this approach?

[ABataev]

Yep, that's what I proposed earlier. Can you run perf testing with for these changes?

[vdmitrie]

In D132590#3751406, @ABataev wrote:

Yep, that's what I proposed earlier. Can you run perf testing with for these changes?

It's running. There are a couple of things to note. I'm only able to run perf testing on a downstream compiler with quite limited number of benchmarks.
Thus evaluation of performance impact will be based on that and in theory might differ for llvm-project.
And there will be no performance numbers - just can say whether looses/gains are expected.

[vdmitrie]

The performance run did not reveal any regression while there were couple of gains (13% and 18% - two tests in different benchmarks, one of them was targeted). On cpu2017 the numbers are nearly flat.
Tested with benchmarks SPEC CPU2017, Coremark Pro, SPEC HPC 2021, and a few more.

[ABataev]

LG, thanks!

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (match(I, m_Intrinsic<Intrinsic::smin>(m_Value(V0), m_Value(V1))))
     return true;
   if (match(I, m_Intrinsic<Intrinsic::umax>(m_Value(V0), m_Value(V1))))
     return true;
   if (match(I, m_Intrinsic<Intrinsic::umin>(m_Value(V0), m_Value(V1))))
     return true;
   return false;
 }
 
 bool SLPVectorizerPass::vectorizeHorReduction(
     PHINode *P, Value *V, BasicBlock *BB, BoUpSLP &R, TargetTransformInfo *TTI,
     SmallVectorImpl<WeakTrackingVH> &PostponedInsts) {
   if (!ShouldVectorizeHor)
     return false;
 
   auto *Root = dyn_cast_or_null<Instruction>(V);
   if (!Root)
     return false;
 
   if (!isa<BinaryOperator>(Root))
     P = nullptr;
bool SLPVectorizerPass::tryToVectorize(ArrayRef<WeakTrackingVH> Insts,
                                        BoUpSLP &R) {
   bool Res = false;
   for (Value *V : Insts)
     if (auto *Inst = dyn_cast<Instruction>(V); Inst && !R.isDeleted(Inst))
       Res |= tryToVectorize(Inst, R);
   return Res;
 }
 
 bool SLPVectorizerPass::vectorizeInsertValueInst(InsertValueInst *IVI,
                                                  BasicBlock *BB, BoUpSLP &R) {
   const DataLayout &DL = BB->getModule()->getDataLayout();
   if (!R.canMapToVector(IVI->getType(), DL))
     return false;
 
   SmallVector<Value *, 16> BuildVectorOpds;
   SmallVector<Value *, 16> BuildVectorInsts;
   if (!findBuildAggregate(IVI, TTI, BuildVectorOpds, BuildVectorInsts))
     return false;
 
   LLVM_DEBUG(dbgs() << "SLP: array mappable to vector: " << *IVI << "\n");
   // Aggregate value is unlikely to be processed in vector register.
   return tryToVectorizeList(BuildVectorOpds, R);
 }
 
 bool SLPVectorizerPass::vectorizeInsertElementInst(InsertElementInst *IEI,
                                                    BasicBlock *BB, BoUpSLP &R) {
   SmallVector<Value *, 16> BuildVectorInsts;
   SmallVector<Value *, 16> BuildVectorOpds;
   SmallVector<int> Mask;
   if (!findBuildAggregate(IEI, TTI, BuildVectorOpds, BuildVectorInsts) ||
       (llvm::all_of(
            BuildVectorOpds,
            [](Value *V) { return isa<ExtractElementInst, UndefValue>(V); }) &&
        isFixedVectorShuffle(BuildVectorOpds, Mask)))
     return false;
 
static bool compareCmp(Value *V, Value *V2,
   return IsCompatibility;
 }
 
 bool SLPVectorizerPass::vectorizeSimpleInstructions(InstSetVector &Instructions,
                                                     BasicBlock *BB, BoUpSLP &R,
                                                     bool AtTerminator) {
   bool OpsChanged = false;
   SmallVector<Instruction *, 4> PostponedCmps;
+  SmallVector<WeakTrackingVH> PostponedInsts;
+  // pass1 - try to vectorize reductions only
   for (auto *I : reverse(Instructions)) {
     if (R.isDeleted(I))
       continue;
+    if (isa<CmpInst>(I)) {
+      PostponedCmps.push_back(I);
+      continue;
+    }
+    OpsChanged |= vectorizeHorReduction(nullptr, I, BB, R, TTI, PostponedInsts);
+  }
+  // pass2 - try to match and vectorize a buildvector sequence.
+  for (auto *I : reverse(Instructions)) {
+    if (R.isDeleted(I) || isa<CmpInst>(I))
+      continue;
     if (auto *LastInsertValue = dyn_cast<InsertValueInst>(I)) {
       OpsChanged |= vectorizeInsertValueInst(LastInsertValue, BB, R);
     } else if (auto *LastInsertElem = dyn_cast<InsertElementInst>(I)) {
       OpsChanged |= vectorizeInsertElementInst(LastInsertElem, BB, R);
-    } else if (isa<CmpInst>(I)) {
-      PostponedCmps.push_back(I);
-      continue;
     }
-    // Try to find reductions in buildvector sequnces.
-    OpsChanged |= vectorizeRootInstruction(nullptr, I, BB, R, TTI);
   }
+  // Now try to vectorize postponed instructions.
+  OpsChanged |= tryToVectorize(PostponedInsts, R);
+
   if (AtTerminator) {
     // Try to find reductions first.
     for (Instruction *I : PostponedCmps) {
       if (R.isDeleted(I))
         continue;
       for (Value *Op : I->operands())
         OpsChanged |= vectorizeRootInstruction(nullptr, Op, BB, R, TTI);
     }
     // Try to vectorize operands as vector bundles.
     for (Instruction *I : PostponedCmps) {
       if (R.isDeleted(I))
         continue;
       OpsChanged |= tryToVectorize(I, R);
     }
     // Try to vectorize list of compares.
     // Sort by type, compare predicate, etc.
     auto &&CompareSorter = [&R](Value *V, Value *V2) {
       return compareCmp<false>(V, V2,
                                [&R](Instruction *I) { return R.isDeleted(I); });
     };
 
     auto &&AreCompatibleCompares = [&R](Value *V1, Value *V2) {
       if (V1 == V2)
         return true;
       return compareCmp<true>(V1, V2,
                               [&R](Instruction *I) { return R.isDeleted(I); });
     };
     auto Limit = [&R](Value *V) {
       unsigned EltSize = R.getVectorElementSize(V);
       return std::max(2U, R.getMaxVecRegSize() / EltSize);
     };
 
     SmallVector<Value *> Vals(PostponedCmps.begin(), PostponedCmps.end());
     OpsChanged |= tryToVectorizeSequence<Value>(
         Vals, Limit, CompareSorter, AreCompatibleCompares,
         [this, &R](ArrayRef<Value *> Candidates, bool LimitForRegisterSize) {
           // Exclude possible reductions from other blocks.
           bool ArePossiblyReducedInOtherBlock =
               any_of(Candidates, [](Value *V) {
                 return any_of(V->users(), [V](User *U) {
                   return isa<SelectInst>(U) &&

llvm/test/Transforms/SLPVectorizer/X86/redux-feed-buildvector.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -mtriple=x86_64 -slp-vectorizer -S -mcpu=skylake-avx512 | FileCheck %s
 
 ; The test represents the case with multiple vectorization possibilities
 ; but the most effective way to vectorize it is to match both 8-way reductions
 ; feeding the insertelement vector build sequence.
 
 declare void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double>, <2 x double*>, i32 immarg, <2 x i1>)
 
 define void @test(double* nocapture readonly %arg, double* nocapture readonly %arg1, double* nocapture %arg2) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[GEP1_0:%.*]] = getelementptr inbounds double, double* [[ARG:%.*]], i64 1
-; CHECK-NEXT:    [[LD1_0:%.*]] = load double, double* [[GEP1_0]], align 8
+; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <8 x double*> poison, double* [[ARG:%.*]], i32 0
+; CHECK-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x double*> [[TMP0]], <8 x double*> poison, <8 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP1:%.*]] = getelementptr double, <8 x double*> [[SHUFFLE]], <8 x i64> <i64 1, i64 3, i64 5, i64 7, i64 9, i64 11, i64 13, i64 15>
 ; CHECK-NEXT:    [[GEP2_0:%.*]] = getelementptr inbounds double, double* [[ARG1:%.*]], i64 16
-; CHECK-NEXT:    [[GEP1_1:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 3
-; CHECK-NEXT:    [[LD1_1:%.*]] = load double, double* [[GEP1_1]], align 8
-; CHECK-NEXT:    [[GEP0_1:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 1
-; CHECK-NEXT:    [[GEP2_1:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 17
-; CHECK-NEXT:    [[GEP1_2:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 5
-; CHECK-NEXT:    [[LD1_2:%.*]] = load double, double* [[GEP1_2]], align 8
-; CHECK-NEXT:    [[GEP0_2:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 2
-; CHECK-NEXT:    [[GEP2_2:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 18
-; CHECK-NEXT:    [[GEP1_3:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 7
-; CHECK-NEXT:    [[LD1_3:%.*]] = load double, double* [[GEP1_3]], align 8
-; CHECK-NEXT:    [[GEP0_3:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 3
-; CHECK-NEXT:    [[GEP2_3:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 19
-; CHECK-NEXT:    [[GEP1_4:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 9
-; CHECK-NEXT:    [[LD1_4:%.*]] = load double, double* [[GEP1_4]], align 8
-; CHECK-NEXT:    [[GEP0_4:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 4
-; CHECK-NEXT:    [[GEP2_4:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 20
-; CHECK-NEXT:    [[GEP1_5:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 11
-; CHECK-NEXT:    [[LD1_5:%.*]] = load double, double* [[GEP1_5]], align 8
-; CHECK-NEXT:    [[GEP0_5:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 5
-; CHECK-NEXT:    [[GEP2_5:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 21
-; CHECK-NEXT:    [[GEP1_6:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 13
-; CHECK-NEXT:    [[LD1_6:%.*]] = load double, double* [[GEP1_6]], align 8
-; CHECK-NEXT:    [[GEP0_6:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 6
-; CHECK-NEXT:    [[GEP2_6:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 22
-; CHECK-NEXT:    [[GEP1_7:%.*]] = getelementptr inbounds double, double* [[ARG]], i64 15
-; CHECK-NEXT:    [[LD1_7:%.*]] = load double, double* [[GEP1_7]], align 8
-; CHECK-NEXT:    [[GEP0_7:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 7
-; CHECK-NEXT:    [[GEP2_7:%.*]] = getelementptr inbounds double, double* [[ARG1]], i64 23
-; CHECK-NEXT:    [[LD0_0:%.*]] = load double, double* [[ARG1]], align 8
-; CHECK-NEXT:    [[LD2_0:%.*]] = load double, double* [[GEP2_0]], align 8
-; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x double> poison, double [[LD0_0]], i32 0
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x double> [[TMP0]], double [[LD2_0]], i32 1
-; CHECK-NEXT:    [[TMP2:%.*]] = insertelement <2 x double> poison, double [[LD1_0]], i32 0
-; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x double> [[TMP2]], double [[LD1_0]], i32 1
-; CHECK-NEXT:    [[TMP4:%.*]] = fmul fast <2 x double> [[TMP1]], [[TMP3]]
-; CHECK-NEXT:    [[LD0_1:%.*]] = load double, double* [[GEP0_1]], align 8
-; CHECK-NEXT:    [[LD2_1:%.*]] = load double, double* [[GEP2_1]], align 8
-; CHECK-NEXT:    [[TMP5:%.*]] = insertelement <2 x double> poison, double [[LD0_1]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = insertelement <2 x double> [[TMP5]], double [[LD2_1]], i32 1
-; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x double> poison, double [[LD1_1]], i32 0
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x double> [[TMP7]], double [[LD1_1]], i32 1
-; CHECK-NEXT:    [[TMP9:%.*]] = fmul fast <2 x double> [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TMP10:%.*]] = fadd fast <2 x double> [[TMP4]], [[TMP9]]
-; CHECK-NEXT:    [[LD0_2:%.*]] = load double, double* [[GEP0_2]], align 8
-; CHECK-NEXT:    [[LD2_2:%.*]] = load double, double* [[GEP2_2]], align 8
-; CHECK-NEXT:    [[TMP11:%.*]] = insertelement <2 x double> poison, double [[LD0_2]], i32 0
-; CHECK-NEXT:    [[TMP12:%.*]] = insertelement <2 x double> [[TMP11]], double [[LD2_2]], i32 1
-; CHECK-NEXT:    [[TMP13:%.*]] = insertelement <2 x double> poison, double [[LD1_2]], i32 0
-; CHECK-NEXT:    [[TMP14:%.*]] = insertelement <2 x double> [[TMP13]], double [[LD1_2]], i32 1
-; CHECK-NEXT:    [[TMP15:%.*]] = fmul fast <2 x double> [[TMP12]], [[TMP14]]
-; CHECK-NEXT:    [[TMP16:%.*]] = fadd fast <2 x double> [[TMP10]], [[TMP15]]
-; CHECK-NEXT:    [[LD0_3:%.*]] = load double, double* [[GEP0_3]], align 8
-; CHECK-NEXT:    [[LD2_3:%.*]] = load double, double* [[GEP2_3]], align 8
-; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <2 x double> poison, double [[LD0_3]], i32 0
-; CHECK-NEXT:    [[TMP18:%.*]] = insertelement <2 x double> [[TMP17]], double [[LD2_3]], i32 1
-; CHECK-NEXT:    [[TMP19:%.*]] = insertelement <2 x double> poison, double [[LD1_3]], i32 0
-; CHECK-NEXT:    [[TMP20:%.*]] = insertelement <2 x double> [[TMP19]], double [[LD1_3]], i32 1
-; CHECK-NEXT:    [[TMP21:%.*]] = fmul fast <2 x double> [[TMP18]], [[TMP20]]
-; CHECK-NEXT:    [[TMP22:%.*]] = fadd fast <2 x double> [[TMP16]], [[TMP21]]
-; CHECK-NEXT:    [[LD0_4:%.*]] = load double, double* [[GEP0_4]], align 8
-; CHECK-NEXT:    [[LD2_4:%.*]] = load double, double* [[GEP2_4]], align 8
-; CHECK-NEXT:    [[TMP23:%.*]] = insertelement <2 x double> poison, double [[LD0_4]], i32 0
-; CHECK-NEXT:    [[TMP24:%.*]] = insertelement <2 x double> [[TMP23]], double [[LD2_4]], i32 1
-; CHECK-NEXT:    [[TMP25:%.*]] = insertelement <2 x double> poison, double [[LD1_4]], i32 0
-; CHECK-NEXT:    [[TMP26:%.*]] = insertelement <2 x double> [[TMP25]], double [[LD1_4]], i32 1
-; CHECK-NEXT:    [[TMP27:%.*]] = fmul fast <2 x double> [[TMP24]], [[TMP26]]
-; CHECK-NEXT:    [[TMP28:%.*]] = fadd fast <2 x double> [[TMP22]], [[TMP27]]
-; CHECK-NEXT:    [[LD0_5:%.*]] = load double, double* [[GEP0_5]], align 8
-; CHECK-NEXT:    [[LD2_5:%.*]] = load double, double* [[GEP2_5]], align 8
-; CHECK-NEXT:    [[TMP29:%.*]] = insertelement <2 x double> poison, double [[LD0_5]], i32 0
-; CHECK-NEXT:    [[TMP30:%.*]] = insertelement <2 x double> [[TMP29]], double [[LD2_5]], i32 1
-; CHECK-NEXT:    [[TMP31:%.*]] = insertelement <2 x double> poison, double [[LD1_5]], i32 0
-; CHECK-NEXT:    [[TMP32:%.*]] = insertelement <2 x double> [[TMP31]], double [[LD1_5]], i32 1
-; CHECK-NEXT:    [[TMP33:%.*]] = fmul fast <2 x double> [[TMP30]], [[TMP32]]
-; CHECK-NEXT:    [[TMP34:%.*]] = fadd fast <2 x double> [[TMP28]], [[TMP33]]
-; CHECK-NEXT:    [[LD0_6:%.*]] = load double, double* [[GEP0_6]], align 8
-; CHECK-NEXT:    [[LD2_6:%.*]] = load double, double* [[GEP2_6]], align 8
-; CHECK-NEXT:    [[TMP35:%.*]] = insertelement <2 x double> poison, double [[LD0_6]], i32 0
-; CHECK-NEXT:    [[TMP36:%.*]] = insertelement <2 x double> [[TMP35]], double [[LD2_6]], i32 1
-; CHECK-NEXT:    [[TMP37:%.*]] = insertelement <2 x double> poison, double [[LD1_6]], i32 0
-; CHECK-NEXT:    [[TMP38:%.*]] = insertelement <2 x double> [[TMP37]], double [[LD1_6]], i32 1
-; CHECK-NEXT:    [[TMP39:%.*]] = fmul fast <2 x double> [[TMP36]], [[TMP38]]
-; CHECK-NEXT:    [[TMP40:%.*]] = fadd fast <2 x double> [[TMP34]], [[TMP39]]
-; CHECK-NEXT:    [[LD0_7:%.*]] = load double, double* [[GEP0_7]], align 8
-; CHECK-NEXT:    [[LD2_7:%.*]] = load double, double* [[GEP2_7]], align 8
-; CHECK-NEXT:    [[TMP41:%.*]] = insertelement <2 x double> poison, double [[LD0_7]], i32 0
-; CHECK-NEXT:    [[TMP42:%.*]] = insertelement <2 x double> [[TMP41]], double [[LD2_7]], i32 1
-; CHECK-NEXT:    [[TMP43:%.*]] = insertelement <2 x double> poison, double [[LD1_7]], i32 0
-; CHECK-NEXT:    [[TMP44:%.*]] = insertelement <2 x double> [[TMP43]], double [[LD1_7]], i32 1
-; CHECK-NEXT:    [[TMP45:%.*]] = fmul fast <2 x double> [[TMP42]], [[TMP44]]
-; CHECK-NEXT:    [[TMP46:%.*]] = fadd fast <2 x double> [[TMP40]], [[TMP45]]
+; CHECK-NEXT:    [[TMP2:%.*]] = call <8 x double> @llvm.masked.gather.v8f64.v8p0f64(<8 x double*> [[TMP1]], i32 8, <8 x i1> <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, <8 x double> undef)
+; CHECK-NEXT:    [[TMP3:%.*]] = bitcast double* [[GEP2_0]] to <8 x double>*
+; CHECK-NEXT:    [[TMP4:%.*]] = load <8 x double>, <8 x double>* [[TMP3]], align 8
+; CHECK-NEXT:    [[TMP5:%.*]] = fmul fast <8 x double> [[TMP4]], [[TMP2]]
+; CHECK-NEXT:    [[TMP6:%.*]] = bitcast double* [[ARG1]] to <8 x double>*
+; CHECK-NEXT:    [[TMP7:%.*]] = load <8 x double>, <8 x double>* [[TMP6]], align 8
+; CHECK-NEXT:    [[TMP8:%.*]] = fmul fast <8 x double> [[TMP7]], [[TMP2]]
+; CHECK-NEXT:    [[TMP9:%.*]] = call fast double @llvm.vector.reduce.fadd.v8f64(double -0.000000e+00, <8 x double> [[TMP8]])
+; CHECK-NEXT:    [[TMP10:%.*]] = call fast double @llvm.vector.reduce.fadd.v8f64(double -0.000000e+00, <8 x double> [[TMP5]])
+; CHECK-NEXT:    [[I142:%.*]] = insertelement <2 x double> poison, double [[TMP9]], i64 0
+; CHECK-NEXT:    [[I143:%.*]] = insertelement <2 x double> [[I142]], double [[TMP10]], i64 1
 ; CHECK-NEXT:    [[P:%.*]] = getelementptr inbounds double, double* [[ARG2:%.*]], <2 x i64> <i64 0, i64 16>
-; CHECK-NEXT:    call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> [[TMP46]], <2 x double*> [[P]], i32 8, <2 x i1> <i1 true, i1 true>)
+; CHECK-NEXT:    call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> [[I143]], <2 x double*> [[P]], i32 8, <2 x i1> <i1 true, i1 true>)
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %gep1.0 = getelementptr inbounds double, double* %arg, i64 1
   %ld1.0 = load double, double* %gep1.0, align 8
   %ld0.0 = load double, double* %arg1, align 8
   %mul1.0 = fmul fast double %ld0.0, %ld1.0
   %gep2.0 = getelementptr inbounds double, double* %arg1, i64 16
entry:
   %ld2.7 = load double, double* %gep2.7, align 8
   %mul2.7 = fmul fast double %ld2.7, %ld1.7
   %rdx2 = fadd fast double %rdx2.5, %mul2.7
   %i142 = insertelement <2 x double> poison, double %rdx1, i64 0
   %i143 = insertelement <2 x double> %i142, double %rdx2, i64 1
   %p = getelementptr inbounds double, double* %arg2, <2 x i64> <i64 0, i64 16>
   call void @llvm.masked.scatter.v2f64.v2p0f64(<2 x double> %i143, <2 x double*> %p, i32 8, <2 x i1> <i1 true, i1 true>)
   ret void
 }