[SLP] Simplify `tryToVectorizeSequence()`

[vporpo]

Summary

The original code attempted to vectorize the seed vector without first
separating the entries by Type, which complicated the code.
It would also try to slice the seed vector, but this seems to be redundant as
both vectorizeStores() and tryToVectorizeList() already do that internally.

With this patch the logic is quite simple:

1. First separate the seeds based on their Type, then
2. Sort the seeds of each Type using the Comparator
3. And finally go over each Type and try to vectorize it.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Removal of this parameter causes regressions in the vectorization, better to restore it.

14329–14331

It increases memory consumption.

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Do you have some examples where this is needed?

14329–14331

The previous version was using a Candidates vector, this one is using a map, so the memory consumption should be fairly similar.
Also this is not a global data structure, it is being freed right away, so memory consumption shouldn't matter much in this context.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

No, I don't, just from my experience when I added this parameter. Testing showed regressions in LTO mode with it, IIRC.

14329–14331

It is not quite map, it is mapvector, which is vector + map

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

IIRC, test/Transforms/SLPVectorizer/slp-max-phi-size.ll is one of the tests

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Is this in SPEC?

14329–14331

OK, so if I use an std::map would you be fine with this?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Yes

14329–14331

DenseMap should be good here, yes, since the order of the types does not matter

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

OK, I will run SPEC lto to check.

IIRC, test/Transforms/SLPVectorizer/slp-max-phi-size.ll is one of the tests

Doesn't slp-max-phi-size.ll look OK though? As far as I understand the purpose of this test is to make sure that we don't vectorize wider than specified by the -slp-max-vf option.

14329–14331

Hmm is iterating over a DenseMap deterministic though?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Probably. If I do remember correctly, it caused regresssions because of too early optimization of phi nodes and some other instructions.

14329–14331

Yes, you cannot iterate through it, so here you need to use MapVector.

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

I checked the C/C++ benchmarks of CPU2017 and I don't see any regressions. I also checked with our internal testing and there are no issues. So I think it is pretty safe.
In the unlikely case that it causes a regression, we can look into it again and add any missing tests. Wdyt?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Did you try LTO?

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Ah let me check. I guess you mean -flto right?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Yes. There were regressions because of too early phi nodes vectorization at compile time, which prevents some later optimizations at link time. We could fix this by checking if LTO is enabled and the stage is linking. In this case we can set the flag to true for phis and other nodes. It would be good if you could make a patch for this!

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

I checked an LTO build with and I didn't see any performance difference. It's all within the noise margin.
This is the list of test I tried:

500.perlbench_r
502.gcc_r
505.mcf_r
508.namd_r
511.povray_r
519.lbm_r
523.xalancbmk_r
525.x264_r
531.deepsjeng_r
538.imagick_r
544.nab_r
557.xz_r

However, my build configuration may be different than yours. Could you try out the patch with your configuration and point me to the right benchmark/build options?

Changing the functionality depending on whether we are in the linking stage, just to make a SPEC test run faster in some specific configuration does not sound a very good solution to me.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

The solution is the same, just at link time we can ignore limits and do our best to vectorize even small sequences, which may cause regression at compile time.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Also, try to compile everything with -march=native on something like Skylake

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Yeah I used native for the builds, and I ran it on a Xeon(R) Gold 6154.

just at link time we can ignore limits and do our best to vectorize even small sequences

So do you mean like setting MaxVFOnly to true only for the linking stage of LTO ?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Yeah I used native for the builds, and I ran it on a Xeon(R) Gold 6154.

Then check with the default target also (SSE2). Check D103638, D108740, D92059 for more context about the changes.

just at link time we can ignore limits and do our best to vectorize even small sequences

So do you mean like setting MaxVFOnly to true only for the linking stage of LTO ?

Yes, something like this. Because currently we have to limit some vectorization to avoid regressions with LTO.

llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll

96 ↗

(On Diff #521059)

Regression?

llvm/test/Transforms/SLPVectorizer/X86/rgb_phi.ll

26

Regression?

[vporpo]

llvm/test/Transforms/SLPVectorizer/X86/reduction-logical.ll

96 ↗

(On Diff #521059)

These regressions are because we are not slicing the Seed vector into smaller sections. I can add this in.

[vporpo]

If vectorizing the full Seeds vector fails we are also trying to vectorize Slices of it. This should remove the test regressions.

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

Yes, something like this. Because currently we have to limit some vectorization to avoid regressions with LTO.

But how am I going to test how well this will work? I am pretty sure this won't be the only phase-ordering issue in LTO builds if SLP is enabled in pre-link.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14361–14365

Does it make sense to drop this check?

[aeubanks]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

https://reviews.llvm.org/D148010 sounds like the proper solution for this sort of issue

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14361–14365

Isn't this already checked within TryToVectorizeHelper? If I am not mistaken both vectorizeStores() and tryToVectorizeList() already check for minimum VF.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14327

I rather doubt it would be good solution for SLP. I tried something like this before, saw lots of regressions. Better to run some optimizations (at full register, for example) at compile time and the remaining ones at link time, otherwise there might be not enough resources. SLP vectorizer has some internal limits and optimizing at link time only may exhaust it too fast.

14339

Seeds.size() <= 1

14342

What I'm afraid is that it may prevent some full-register optimizations. You will early optimize part register tress and miss some full-register trees (probably, with alternate opcodes)

14361–14365

It just early skips not profitable cases, saves some compile time, checks that vector can be fed with the elements, otherwise it does not worth trying vectorization, it will be scalarized.

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14342

So would you prefer iterating over the SeedMap twice?
Once with MaxVFOnly = true and then once with MaxVFOnly = false?

14361–14365

It might save a bit of compilation time, but it is just duplicate code. I think it is probably better to move the checks in vectorizeStores() and tryToVectorizeList() as early as possible. Let me check if this is possible.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14342

Yes, that's the idea

14361–14365

Sure, I'm fine with this too

[vporpo]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14342

The problem is that neither is clearly the better strategy. You can come up with examples where vectorizing for MaxVF first across all seeds leads to worse vectorization than trying to vectorize each seed multiple times starting from MaxVF all the way to MinVF. For example when you end up with a short wide tree versus a deep narrower tree.
Also if we are to follow a strategy MaxVF first, then it would probably make more sense if we visit seeds based on some specific order, e.g. from largest to smallest, and even then there would still be cases when the strategy won't work.

In such cases I am in favor of keeping the code as simple as possible because a more complicated solution won't necessarily pay off.

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

14342

I prefer to keep original implementation for the 2 reasons.

1. I plan to add a new node entry, which may improve vectorization of nodes with the different operands.
2. Need to try to improve the whole process by trying vectorization not only for tail values, but for other too.

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

if (!findBuildAggregate(IEI, TTI, BuildVectorOpds, BuildVectorInsts) ||
     return false;
 
   LLVM_DEBUG(dbgs() << "SLP: array mappable to vector: " << *IEI << "\n");
   return tryToVectorizeList(BuildVectorInsts, R);
 }
 
 template <typename T>
 static bool tryToVectorizeSequence(
-    SmallVectorImpl<T *> &Incoming, function_ref<bool(T *, T *)> Comparator,
+    SmallVectorImpl<T *> &UnsortedSeeds,
+    function_ref<bool(T *, T *)> Comparator,
     function_ref<bool(T *, T *)> AreCompatible,
-    function_ref<bool(ArrayRef<T *>, bool)> TryToVectorizeHelper,
-    bool MaxVFOnly, BoUpSLP &R) {
+    function_ref<bool(ArrayRef<T *>, bool)> TryToVectorizeHelper, BoUpSLP &R) {
+  // We separate the seeds into groups based on their types.
+  MapVector<Type *, SmallVector<T *>> SeedsMap;
+  for (T *Seed : UnsortedSeeds)
+    SeedsMap[Seed->getType()].push_back(Seed);
+  // Sort the seeds that correspond to each type.
+  for (auto &Pair : SeedsMap)
+    stable_sort(Pair.second, Comparator);
+  // For all types try to vectorize the seed vector.
   bool Changed = false;
-  // Sort by type, parent, operands.
-  stable_sort(Incoming, Comparator);
-
-  // Try to vectorize elements base on their type.
-  SmallVector<T *> Candidates;
-  for (auto *IncIt = Incoming.begin(), *E = Incoming.end(); IncIt != E;) {
-    // Look for the next elements with the same type, parent and operand
-    // kinds.
-    auto *SameTypeIt = IncIt;
-    while (SameTypeIt != E && AreCompatible(*SameTypeIt, *IncIt))
-      ++SameTypeIt;
-
-    // Try to vectorize them.
-    unsigned NumElts = (SameTypeIt - IncIt);
-    LLVM_DEBUG(dbgs() << "SLP: Trying to vectorize starting at nodes ("
-                      << NumElts << ")\n");
-    // The vectorization is a 3-state attempt:
-    // 1. Try to vectorize instructions with the same/alternate opcodes with the
-    // size of maximal register at first.
-    // 2. Try to vectorize remaining instructions with the same type, if
-    // possible. This may result in the better vectorization results rather than
-    // if we try just to vectorize instructions with the same/alternate opcodes.
-    // 3. Final attempt to try to vectorize all instructions with the
-    // same/alternate ops only, this may result in some extra final
-    // vectorization.
-    if (NumElts > 1 &&
-        TryToVectorizeHelper(ArrayRef(IncIt, NumElts), MaxVFOnly)) {
-      // Success start over because instructions might have been changed.
-      Changed = true;
-    } else {
-      /// \Returns the minimum number of elements that we will attempt to
-      /// vectorize.
-      auto GetMinNumElements = [&R](Value *V) {
-        unsigned EltSize = R.getVectorElementSize(V);
-        return std::max(2U, R.getMaxVecRegSize() / EltSize);
-      };
-      if (NumElts < GetMinNumElements(*IncIt) &&
-          (Candidates.empty() ||
-           Candidates.front()->getType() == (*IncIt)->getType())) {
-        Candidates.append(IncIt, std::next(IncIt, NumElts));
-      }
-    }
-    // Final attempt to vectorize instructions with the same types.
-    if (Candidates.size() > 1 &&
-        (SameTypeIt == E || (*SameTypeIt)->getType() != (*IncIt)->getType())) {
-      if (TryToVectorizeHelper(Candidates, /*MaxVFOnly=*/false)) {
-        // Success start over because instructions might have been changed.
-        Changed = true;
-      } else if (MaxVFOnly) {
-        // Try to vectorize using small vectors.
-        for (auto *It = Candidates.begin(), *End = Candidates.end();
-             It != End;) {
-          auto *SameTypeIt = It;
-          while (SameTypeIt != End && AreCompatible(*SameTypeIt, *It))
-            ++SameTypeIt;
-          unsigned NumElts = (SameTypeIt - It);
-          if (NumElts > 1 && TryToVectorizeHelper(ArrayRef(It, NumElts),
-                                                  /*MaxVFOnly=*/false))
-            Changed = true;
-          It = SameTypeIt;
-        }
-      }
-      Candidates.clear();
+  for (auto &Pair : SeedsMap) {
+    auto &Seeds = Pair.second;
+    if (Seeds.empty())
+      continue;
+    // Try the full Seeds vector first.
+    if (TryToVectorizeHelper(Seeds, /*MaxVFOnly=*/false)) {
+      continue;
+    // If the full vector failed, slice the Seeds according to `AreCompatible()`
+    uint32_t SliceSz;
+    for (auto SliceBegin = Seeds.begin(), ItE = Seeds.end(); SliceBegin != ItE;
+         SliceBegin += SliceSz) {
+      auto SliceEnd = std::next(SliceBegin);
+      while (SliceEnd != ItE && AreCompatible(*SliceEnd, *std::prev(SliceEnd)))
+        ++SliceEnd;
+      SliceSz = SliceEnd - SliceBegin;
+      if (SliceSz <= 1)
+        continue;
+      ArrayRef<T *> SeedsSlice(SliceBegin, SliceEnd);
+      if (TryToVectorizeHelper(SeedsSlice, /*MaxVFOnly=*/false))
+        Changed = true;
     }
-
-    // Start over at the next instruction of a different type (or the end).
-    IncIt = SameTypeIt;
   }
   return Changed;
 }
 
 /// Compare two cmp instructions. If IsCompatibility is true, function returns
 /// true if 2 cmps have same/swapped predicates and mos compatible corresponding
 /// operands. If IsCompatibility is false, function implements strict weak
 /// ordering relation between two cmp instructions, returning true if the first
Changed |= tryToVectorizeSequence<Value>(
             return Select &&
                    Select->getParent() != cast<Instruction>(V)->getParent();
           });
         });
         if (ArePossiblyReducedInOtherBlock)
           return false;
         return tryToVectorizeList(Candidates, R, MaxVFOnly);
       },
-      /*MaxVFOnly=*/true, R);
+      R);
   return Changed;
 }
 
 bool SLPVectorizerPass::vectorizeInserts(InstSetVector &Instructions,
                                          BasicBlock *BB, BoUpSLP &R) {
   assert(all_of(Instructions,
                 [](auto *I) {
                   return isa<InsertElementInst, InsertValueInst>(I);
for (Value *V : Incoming) {
       }
     }
 
     HaveVectorizedPhiNodes = tryToVectorizeSequence<Value>(
         Incoming, PHICompare, AreCompatiblePHIs,
         [this, &R](ArrayRef<Value *> Candidates, bool MaxVFOnly) {
           return tryToVectorizeList(Candidates, R, MaxVFOnly);
         },
-        /*MaxVFOnly=*/true, R);
+        R);
     Changed |= HaveVectorizedPhiNodes;
     VisitedInstrs.insert(Incoming.begin(), Incoming.end());
   } while (HaveVectorizedPhiNodes);
 
   VisitedInstrs.clear();
 
   InstSetVector PostProcessInserts;
   SmallSetVector<CmpInst *, 8> PostProcessCmps;
for (auto &Pair : Stores) {
     if (!isValidElementType(Pair.second.front()->getValueOperand()->getType()))
       continue;
 
     Changed |= tryToVectorizeSequence<StoreInst>(
         Pair.second, StoreSorter, AreCompatibleStores,
         [this, &R](ArrayRef<StoreInst *> Candidates, bool) {
           return vectorizeStores(Candidates, R);
         },
-        /*MaxVFOnly=*/false, R);
+        R);
   }
   return Changed;
 }

llvm/test/Transforms/SLPVectorizer/AArch64/extractelements-to-shuffle.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -passes=slp-vectorizer -S < %s -mtriple=aarch64 -aarch64-insert-extract-base-cost=3 | FileCheck %s
 
 define void @test(<2 x i64> %0, <2 x i64> %1, <2 x i64> %2) {
 ; CHECK-LABEL: @test(
-; CHECK-NEXT:    [[TMP4:%.*]] = extractelement <2 x i64> [[TMP1:%.*]], i64 0
-; CHECK-NEXT:    [[TMP5:%.*]] = or i64 [[TMP4]], 0
-; CHECK-NEXT:    [[TMP6:%.*]] = trunc i64 [[TMP5]] to i32
-; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <2 x i64> [[TMP0:%.*]], i64 0
-; CHECK-NEXT:    [[TMP8:%.*]] = or i64 [[TMP7]], 0
-; CHECK-NEXT:    [[TMP9:%.*]] = trunc i64 [[TMP8]] to i32
-; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x i64> [[TMP2:%.*]], i64 0
-; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <2 x i64> [[TMP2]], i64 1
-; CHECK-NEXT:    [[TMP12:%.*]] = or i64 [[TMP10]], [[TMP11]]
-; CHECK-NEXT:    [[TMP13:%.*]] = trunc i64 [[TMP12]] to i32
-; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <2 x i64> [[TMP0]], i64 0
-; CHECK-NEXT:    [[TMP15:%.*]] = or i64 [[TMP14]], 0
-; CHECK-NEXT:    [[TMP16:%.*]] = trunc i64 [[TMP15]] to i32
-; CHECK-NEXT:    br label [[TMP17:%.*]]
-; CHECK:       17:
-; CHECK-NEXT:    [[TMP18:%.*]] = phi i32 [ [[TMP22:%.*]], [[TMP17]] ], [ [[TMP6]], [[TMP3:%.*]] ]
-; CHECK-NEXT:    [[TMP19:%.*]] = phi i32 [ 0, [[TMP17]] ], [ [[TMP9]], [[TMP3]] ]
-; CHECK-NEXT:    [[TMP20:%.*]] = phi i32 [ 0, [[TMP17]] ], [ [[TMP13]], [[TMP3]] ]
-; CHECK-NEXT:    [[TMP21:%.*]] = phi i32 [ 0, [[TMP17]] ], [ [[TMP16]], [[TMP3]] ]
-; CHECK-NEXT:    [[TMP22]] = or i32 [[TMP18]], 0
-; CHECK-NEXT:    br label [[TMP17]]
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i64> [[TMP1:%.*]], <2 x i64> [[TMP0:%.*]], <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[TMP5:%.*]] = or <2 x i64> [[TMP4]], zeroinitializer
+; CHECK-NEXT:    [[TMP6:%.*]] = trunc <2 x i64> [[TMP5]] to <2 x i32>
+; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x i64> [[TMP2:%.*]], <2 x i64> [[TMP0]], <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[TMP8:%.*]] = shufflevector <2 x i64> [[TMP2]], <2 x i64> <i64 poison, i64 0>, <2 x i32> <i32 1, i32 3>
+; CHECK-NEXT:    [[TMP9:%.*]] = or <2 x i64> [[TMP7]], [[TMP8]]
+; CHECK-NEXT:    [[TMP10:%.*]] = trunc <2 x i64> [[TMP9]] to <2 x i32>
+; CHECK-NEXT:    br label [[TMP11:%.*]]
+; CHECK:       11:
+; CHECK-NEXT:    [[TMP12:%.*]] = phi <2 x i32> [ [[TMP17:%.*]], [[TMP11]] ], [ [[TMP6]], [[TMP3:%.*]] ]
+; CHECK-NEXT:    [[TMP13:%.*]] = phi <2 x i32> [ zeroinitializer, [[TMP11]] ], [ [[TMP10]], [[TMP3]] ]
+; CHECK-NEXT:    [[TMP14:%.*]] = shufflevector <2 x i32> [[TMP12]], <2 x i32> <i32 poison, i32 0>, <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP15:%.*]] = or <2 x i32> zeroinitializer, [[TMP14]]
+; CHECK-NEXT:    [[TMP16:%.*]] = add <2 x i32> zeroinitializer, [[TMP14]]
+; CHECK-NEXT:    [[TMP17]] = shufflevector <2 x i32> [[TMP15]], <2 x i32> [[TMP16]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    br label [[TMP11]]
 ;
   %4 = extractelement <2 x i64> %1, i64 0
   %5 = or i64 %4, 0
   %6 = trunc i64 %5 to i32
   %7 = extractelement <2 x i64> %0, i64 0
   %8 = or i64 %7, 0
   %9 = trunc i64 %8 to i32
   %10 = extractelement <2 x i64> %2, i64 0

llvm/test/Transforms/SLPVectorizer/X86/landing_pad.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -passes=slp-vectorizer,verify -slp-threshold=-99999 -S | FileCheck %s
 
 target triple = "x86_64-unknown-linux-gnu"
 
 define void @foo() personality ptr @bar {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:  bb1:
 ; CHECK-NEXT:    br label [[BB3:%.*]]
 ; CHECK:       bb2.loopexit:
 ; CHECK-NEXT:    br label [[BB2:%.*]]
 ; CHECK:       bb2:
-; CHECK-NEXT:    [[TMP0:%.*]] = phi <4 x i32> [ [[TMP8:%.*]], [[BB9:%.*]] ], [ poison, [[BB2_LOOPEXIT:%.*]] ]
+; CHECK-NEXT:    [[TMP0:%.*]] = phi <4 x i32> [ [[TMP7:%.*]], [[BB9:%.*]] ], [ poison, [[BB2_LOOPEXIT:%.*]] ]
 ; CHECK-NEXT:    ret void
 ; CHECK:       bb3:
 ; CHECK-NEXT:    [[TMP1:%.*]] = phi <2 x i32> [ [[TMP3:%.*]], [[BB6:%.*]] ], [ poison, [[BB1:%.*]] ]
 ; CHECK-NEXT:    [[TMP2:%.*]] = invoke i32 poison(ptr addrspace(1) nonnull poison, i32 0, i32 0, i32 poison) [ "deopt"() ]
 ; CHECK-NEXT:    to label [[BB4:%.*]] unwind label [[BB10:%.*]]
 ; CHECK:       bb4:
 ; CHECK-NEXT:    br i1 poison, label [[BB11:%.*]], label [[BB5:%.*]]
 ; CHECK:       bb5:
 ; CHECK-NEXT:    br label [[BB7:%.*]]
 ; CHECK:       bb6:
 ; CHECK-NEXT:    [[TMP3]] = phi <2 x i32> [ <i32 0, i32 poison>, [[BB8:%.*]] ]
 ; CHECK-NEXT:    br label [[BB3]]
 ; CHECK:       bb7:
 ; CHECK-NEXT:    [[LOCAL_5_84111:%.*]] = phi i32 [ poison, [[BB8]] ], [ poison, [[BB5]] ]
 ; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i32> poison, i32 [[LOCAL_5_84111]], i32 0
 ; CHECK-NEXT:    [[TMP5:%.*]] = invoke i32 poison(ptr addrspace(1) nonnull poison, i32 poison, i32 poison, i32 poison) [ "deopt"() ]
 ; CHECK-NEXT:    to label [[BB8]] unwind label [[BB12:%.*]]
 ; CHECK:       bb8:
 ; CHECK-NEXT:    br i1 poison, label [[BB7]], label [[BB6]]
 ; CHECK:       bb9:
 ; CHECK-NEXT:    [[INDVARS_IV528799:%.*]] = phi i64 [ poison, [[BB10]] ], [ poison, [[BB12]] ]
-; CHECK-NEXT:    [[TMP6:%.*]] = phi <2 x i32> [ [[TMP9:%.*]], [[BB10]] ], [ [[TMP10:%.*]], [[BB12]] ]
-; CHECK-NEXT:    [[TMP7:%.*]] = shufflevector <2 x i32> [[TMP6]], <2 x i32> poison, <4 x i32> <i32 1, i32 0, i32 poison, i32 poison>
-; CHECK-NEXT:    [[TMP8]] = shufflevector <4 x i32> [[TMP7]], <4 x i32> poison, <4 x i32> <i32 poison, i32 poison, i32 0, i32 1>
+; CHECK-NEXT:    [[TMP6:%.*]] = phi <2 x i32> [ [[TMP8:%.*]], [[BB10]] ], [ [[TMP9:%.*]], [[BB12]] ]
+; CHECK-NEXT:    [[TMP7]] = shufflevector <2 x i32> [[TMP6]], <2 x i32> poison, <4 x i32> <i32 poison, i32 poison, i32 1, i32 0>
 ; CHECK-NEXT:    br label [[BB2]]
 ; CHECK:       bb10:
-; CHECK-NEXT:    [[TMP9]] = phi <2 x i32> [ [[TMP1]], [[BB3]] ]
+; CHECK-NEXT:    [[TMP8]] = phi <2 x i32> [ [[TMP1]], [[BB3]] ]
 ; CHECK-NEXT:    [[LANDING_PAD68:%.*]] = landingpad { ptr, i32 }
 ; CHECK-NEXT:    cleanup
 ; CHECK-NEXT:    br label [[BB9]]
 ; CHECK:       bb11:
 ; CHECK-NEXT:    ret void
 ; CHECK:       bb12:
-; CHECK-NEXT:    [[TMP10]] = phi <2 x i32> [ [[TMP4]], [[BB7]] ]
+; CHECK-NEXT:    [[TMP9]] = phi <2 x i32> [ [[TMP4]], [[BB7]] ]
 ; CHECK-NEXT:    [[LANDING_PAD149:%.*]] = landingpad { ptr, i32 }
 ; CHECK-NEXT:    cleanup
 ; CHECK-NEXT:    br label [[BB9]]
 ;
 bb1:
   br label %bb3
 
 bb2.loopexit:

llvm/test/Transforms/SLPVectorizer/X86/multi-nodes-to-shuffle.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt -passes=slp-vectorizer -S < %s -mtriple=x86_64-unknown-linux -slp-threshold=-107 | FileCheck %s
 
 define void @test(i64 %p0, i64 %p1, i64 %p2, i64 %p3) {
 ; CHECK-LABEL: @test(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    [[TMP0:%.*]] = insertelement <2 x i64> poison, i64 [[P0:%.*]], i32 0
 ; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <2 x i64> [[TMP0]], i64 [[P1:%.*]], i32 1
 ; CHECK-NEXT:    [[TMP2:%.*]] = add <2 x i64> [[TMP1]], [[TMP1]]
-; CHECK-NEXT:    [[A2:%.*]] = add i64 [[P2:%.*]], [[P2]]
-; CHECK-NEXT:    [[A3:%.*]] = add i64 [[P3:%.*]], [[P3]]
-; CHECK-NEXT:    [[TMP3:%.*]] = mul <2 x i64> [[TMP1]], [[TMP1]]
-; CHECK-NEXT:    [[M2:%.*]] = mul i64 [[P2]], [[P2]]
-; CHECK-NEXT:    [[M3:%.*]] = mul i64 [[P3]], [[P3]]
-; CHECK-NEXT:    [[TMP4:%.*]] = sdiv <2 x i64> [[TMP1]], [[TMP1]]
-; CHECK-NEXT:    [[D2:%.*]] = sdiv i64 [[P2]], [[P2]]
-; CHECK-NEXT:    [[D3:%.*]] = sdiv i64 [[P3]], [[P3]]
-; CHECK-NEXT:    [[TMP5:%.*]] = extractelement <2 x i64> [[TMP4]], i32 0
-; CHECK-NEXT:    [[TMP6:%.*]] = extractelement <2 x i64> [[TMP3]], i32 0
-; CHECK-NEXT:    [[S0:%.*]] = sub i64 [[TMP6]], [[TMP5]]
-; CHECK-NEXT:    [[TMP7:%.*]] = extractelement <2 x i64> [[TMP4]], i32 1
-; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x i64> [[TMP3]], i32 1
-; CHECK-NEXT:    [[S1:%.*]] = sub i64 [[TMP8]], [[TMP7]]
-; CHECK-NEXT:    [[S2:%.*]] = sub i64 [[M2]], [[D2]]
-; CHECK-NEXT:    [[S3:%.*]] = sub i64 [[M3]], [[D3]]
-; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x i64> [[TMP2]], i32 0
-; CHECK-NEXT:    [[SHL1:%.*]] = shl i64 [[TMP9]], [[S0]]
-; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x i64> [[TMP2]], i32 1
-; CHECK-NEXT:    [[SHL2:%.*]] = shl i64 [[TMP10]], [[S1]]
-; CHECK-NEXT:    [[SHL3:%.*]] = shl i64 [[A2]], [[S2]]
-; CHECK-NEXT:    [[SHL4:%.*]] = shl i64 [[A3]], [[S3]]
-; CHECK-NEXT:    [[O0:%.*]] = or i64 [[TMP9]], [[TMP10]]
+; CHECK-NEXT:    [[TMP3:%.*]] = insertelement <2 x i64> poison, i64 [[P2:%.*]], i32 0
+; CHECK-NEXT:    [[TMP4:%.*]] = insertelement <2 x i64> [[TMP3]], i64 [[P3:%.*]], i32 1
+; CHECK-NEXT:    [[TMP5:%.*]] = add <2 x i64> [[TMP4]], [[TMP4]]
+; CHECK-NEXT:    [[TMP6:%.*]] = mul <2 x i64> [[TMP1]], [[TMP1]]
+; CHECK-NEXT:    [[TMP7:%.*]] = mul <2 x i64> [[TMP4]], [[TMP4]]
+; CHECK-NEXT:    [[D0:%.*]] = sdiv i64 [[P0]], [[P0]]
+; CHECK-NEXT:    [[D1:%.*]] = sdiv i64 [[P1]], [[P1]]
+; CHECK-NEXT:    [[TMP8:%.*]] = extractelement <2 x i64> [[TMP6]], i32 0
+; CHECK-NEXT:    [[S0:%.*]] = sub i64 [[TMP8]], [[D0]]
+; CHECK-NEXT:    [[TMP9:%.*]] = extractelement <2 x i64> [[TMP6]], i32 1
+; CHECK-NEXT:    [[S1:%.*]] = sub i64 [[TMP9]], [[D1]]
+; CHECK-NEXT:    [[TMP10:%.*]] = extractelement <2 x i64> [[TMP2]], i32 0
+; CHECK-NEXT:    [[SHL1:%.*]] = shl i64 [[TMP10]], [[S0]]
+; CHECK-NEXT:    [[TMP11:%.*]] = extractelement <2 x i64> [[TMP2]], i32 1
+; CHECK-NEXT:    [[SHL2:%.*]] = shl i64 [[TMP11]], [[S1]]
+; CHECK-NEXT:    [[O0:%.*]] = or i64 [[TMP10]], [[TMP11]]
 ; CHECK-NEXT:    [[TT0:%.*]] = trunc i64 [[O0]] to i32
-; CHECK-NEXT:    [[O1:%.*]] = or i64 [[TMP6]], [[TMP8]]
+; CHECK-NEXT:    [[O1:%.*]] = or i64 [[TMP8]], [[TMP9]]
 ; CHECK-NEXT:    [[TT1:%.*]] = trunc i64 [[O1]] to i32
-; CHECK-NEXT:    [[O2:%.*]] = or i64 [[TMP5]], [[TMP7]]
-; CHECK-NEXT:    [[TT2:%.*]] = trunc i64 [[O2]] to i32
-; CHECK-NEXT:    [[O3:%.*]] = or i64 [[TMP6]], [[TMP8]]
-; CHECK-NEXT:    [[TT3:%.*]] = trunc i64 [[O3]] to i32
+; CHECK-NEXT:    [[TMP12:%.*]] = sdiv <2 x i64> [[TMP4]], [[TMP4]]
+; CHECK-NEXT:    [[TMP13:%.*]] = sub <2 x i64> [[TMP7]], [[TMP12]]
+; CHECK-NEXT:    [[TMP14:%.*]] = shl <2 x i64> [[TMP5]], [[TMP13]]
+; CHECK-NEXT:    [[TMP15:%.*]] = insertelement <2 x i64> poison, i64 [[D0]], i32 0
+; CHECK-NEXT:    [[TMP16:%.*]] = shufflevector <2 x i64> [[TMP15]], <2 x i64> [[TMP6]], <2 x i32> <i32 0, i32 2>
+; CHECK-NEXT:    [[TMP17:%.*]] = insertelement <2 x i64> poison, i64 [[D1]], i32 0
+; CHECK-NEXT:    [[TMP18:%.*]] = shufflevector <2 x i64> [[TMP17]], <2 x i64> [[TMP6]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[TMP19:%.*]] = or <2 x i64> [[TMP16]], [[TMP18]]
+; CHECK-NEXT:    [[TMP20:%.*]] = trunc <2 x i64> [[TMP19]] to <2 x i32>
 ; CHECK-NEXT:    br label [[BB:%.*]]
 ; CHECK:       bb:
 ; CHECK-NEXT:    [[PHI0:%.*]] = phi i32 [ [[T1:%.*]], [[BB]] ], [ [[TT0]], [[ENTRY:%.*]] ]
 ; CHECK-NEXT:    [[PHI1:%.*]] = phi i32 [ [[T2:%.*]], [[BB]] ], [ [[TT1]], [[ENTRY]] ]
-; CHECK-NEXT:    [[PHI2:%.*]] = phi i32 [ [[T3:%.*]], [[BB]] ], [ [[TT2]], [[ENTRY]] ]
-; CHECK-NEXT:    [[PHI3:%.*]] = phi i32 [ [[T4:%.*]], [[BB]] ], [ [[TT3]], [[ENTRY]] ]
+; CHECK-NEXT:    [[TMP21:%.*]] = phi <2 x i32> [ [[TMP22:%.*]], [[BB]] ], [ [[TMP20]], [[ENTRY]] ]
 ; CHECK-NEXT:    [[T1]] = trunc i64 [[SHL1]] to i32
 ; CHECK-NEXT:    [[T2]] = trunc i64 [[SHL2]] to i32
-; CHECK-NEXT:    [[T3]] = trunc i64 [[SHL3]] to i32
-; CHECK-NEXT:    [[T4]] = trunc i64 [[SHL4]] to i32
+; CHECK-NEXT:    [[TMP22]] = trunc <2 x i64> [[TMP14]] to <2 x i32>
 ; CHECK-NEXT:    br label [[BB]]
 ;
 entry:
   %a0 = add i64 %p0, %p0
   %a1 = add i64 %p1, %p1
   %a2 = add i64 %p2, %p2
   %a3 = add i64 %p3, %p3
   %m0 = mul i64 %p0, %p0

llvm/test/Transforms/SLPVectorizer/X86/rgb_phi.ll

 ;   }
 ;
 ;   return R+G+B;
 ; }
 
 define float @foo(ptr nocapture readonly %A) {
 ; CHECK-LABEL: @foo(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP1:%.*]] = load <2 x float>, ptr [[A:%.*]], align 4
+; CHECK-NEXT:    [[TMP0:%.*]] = load <2 x float>, ptr [[A:%.*]], align 4
 ; CHECK-NEXT:    [[ARRAYIDX2:%.*]] = getelementptr inbounds float, ptr [[A]], i64 2
-; CHECK-NEXT:    [[TMP2:%.*]] = load float, ptr [[ARRAYIDX2]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = extractelement <2 x float> [[TMP1]], i32 0
+; CHECK-NEXT:    [[TMP1:%.*]] = load float, ptr [[ARRAYIDX2]], align 4
+; CHECK-NEXT:    [[TMP2:%.*]] = extractelement <2 x float> [[TMP0]], i32 0
 ; CHECK-NEXT:    br label [[FOR_BODY:%.*]]
 ; CHECK:       for.body:
-; CHECK-NEXT:    [[TMP4:%.*]] = phi float [ [[TMP3]], [[ENTRY:%.*]] ], [ [[DOTPRE:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE:%.*]] ]
+; CHECK-NEXT:    [[TMP3:%.*]] = phi float [ [[TMP2]], [[ENTRY:%.*]] ], [ [[DOTPRE:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE:%.*]] ]
 ; CHECK-NEXT:    [[INDVARS_IV:%.*]] = phi i64 [ 0, [[ENTRY]] ], [ [[INDVARS_IV_NEXT:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE]] ]
-; CHECK-NEXT:    [[B_032:%.*]] = phi float [ [[TMP2]], [[ENTRY]] ], [ [[ADD14:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE]] ]
-; CHECK-NEXT:    [[TMP5:%.*]] = phi <2 x float> [ [[TMP1]], [[ENTRY]] ], [ [[TMP11:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE]] ]
-; CHECK-NEXT:    [[TMP6:%.*]] = add nsw i64 [[INDVARS_IV]], 1
-; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP6]]
-; CHECK-NEXT:    [[TMP7:%.*]] = load float, ptr [[ARRAYIDX7]], align 4
-; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x float> poison, float [[TMP4]], i32 0
-; CHECK-NEXT:    [[TMP9:%.*]] = insertelement <2 x float> [[TMP8]], float [[TMP7]], i32 1
-; CHECK-NEXT:    [[TMP10:%.*]] = fmul <2 x float> [[TMP9]], <float 7.000000e+00, float 8.000000e+00>
-; CHECK-NEXT:    [[TMP11]] = fadd <2 x float> [[TMP5]], [[TMP10]]
-; CHECK-NEXT:    [[TMP12:%.*]] = add nsw i64 [[INDVARS_IV]], 2
-; CHECK-NEXT:    [[ARRAYIDX12:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP12]]
-; CHECK-NEXT:    [[TMP13:%.*]] = load float, ptr [[ARRAYIDX12]], align 4
-; CHECK-NEXT:    [[MUL13:%.*]] = fmul float [[TMP13]], 9.000000e+00
+; CHECK-NEXT:    [[B_032:%.*]] = phi float [ [[TMP1]], [[ENTRY]] ], [ [[ADD14:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE]] ]
+; CHECK-NEXT:    [[TMP4:%.*]] = phi <2 x float> [ [[TMP0]], [[ENTRY]] ], [ [[TMP10:%.*]], [[FOR_BODY_FOR_BODY_CRIT_EDGE]] ]
+; CHECK-NEXT:    [[TMP5:%.*]] = add nsw i64 [[INDVARS_IV]], 1
+; CHECK-NEXT:    [[ARRAYIDX7:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP5]]
+; CHECK-NEXT:    [[TMP6:%.*]] = load float, ptr [[ARRAYIDX7]], align 4
+; CHECK-NEXT:    [[TMP7:%.*]] = insertelement <2 x float> poison, float [[TMP3]], i32 0
+; CHECK-NEXT:    [[TMP8:%.*]] = insertelement <2 x float> [[TMP7]], float [[TMP6]], i32 1
+; CHECK-NEXT:    [[TMP9:%.*]] = fmul <2 x float> [[TMP8]], <float 7.000000e+00, float 8.000000e+00>
+; CHECK-NEXT:    [[TMP10]] = fadd <2 x float> [[TMP4]], [[TMP9]]
+; CHECK-NEXT:    [[TMP11:%.*]] = add nsw i64 [[INDVARS_IV]], 2
+; CHECK-NEXT:    [[ARRAYIDX12:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[TMP11]]
+; CHECK-NEXT:    [[TMP12:%.*]] = load float, ptr [[ARRAYIDX12]], align 4
+; CHECK-NEXT:    [[MUL13:%.*]] = fmul float [[TMP12]], 9.000000e+00
 ; CHECK-NEXT:    [[ADD14]] = fadd float [[B_032]], [[MUL13]]
 ; CHECK-NEXT:    [[INDVARS_IV_NEXT]] = add i64 [[INDVARS_IV]], 3
-; CHECK-NEXT:    [[TMP14:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
-; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP14]], 121
+; CHECK-NEXT:    [[TMP13:%.*]] = trunc i64 [[INDVARS_IV_NEXT]] to i32
+; CHECK-NEXT:    [[CMP:%.*]] = icmp slt i32 [[TMP13]], 121
 ; CHECK-NEXT:    br i1 [[CMP]], label [[FOR_BODY_FOR_BODY_CRIT_EDGE]], label [[FOR_END:%.*]]
 ; CHECK:       for.body.for.body_crit_edge:
 ; CHECK-NEXT:    [[ARRAYIDX3_PHI_TRANS_INSERT:%.*]] = getelementptr inbounds float, ptr [[A]], i64 [[INDVARS_IV_NEXT]]
 ; CHECK-NEXT:    [[DOTPRE]] = load float, ptr [[ARRAYIDX3_PHI_TRANS_INSERT]], align 4
 ; CHECK-NEXT:    br label [[FOR_BODY]]
 ; CHECK:       for.end:
-; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP11]], i32 0
-; CHECK-NEXT:    [[TMP16:%.*]] = extractelement <2 x float> [[TMP11]], i32 1
-; CHECK-NEXT:    [[ADD16:%.*]] = fadd float [[TMP15]], [[TMP16]]
+; CHECK-NEXT:    [[TMP14:%.*]] = extractelement <2 x float> [[TMP10]], i32 0
+; CHECK-NEXT:    [[TMP15:%.*]] = extractelement <2 x float> [[TMP10]], i32 1
+; CHECK-NEXT:    [[ADD16:%.*]] = fadd float [[TMP14]], [[TMP15]]
 ; CHECK-NEXT:    [[ADD17:%.*]] = fadd float [[ADD16]], [[ADD14]]
 ; CHECK-NEXT:    ret float [[ADD17]]
 ;
 entry:
   %0 = load float, ptr %A, align 4
   %arrayidx1 = getelementptr inbounds float, ptr %A, i64 1
   %1 = load float, ptr %arrayidx1, align 4
   %arrayidx2 = getelementptr inbounds float, ptr %A, i64 2

llvm/test/Transforms/SLPVectorizer/slp-max-phi-size.ll

 ; MAX256-NEXT:  bb:
 ; MAX256-NEXT:    br label [[BB1:%.*]]
 ; MAX256:       bb1:
 ; MAX256-NEXT:    [[I:%.*]] = fpext half [[HVAL:%.*]] to float
 ; MAX256-NEXT:    [[I3:%.*]] = fpext half [[HVAL]] to float
 ; MAX256-NEXT:    [[I6:%.*]] = fpext half [[HVAL]] to float
 ; MAX256-NEXT:    [[I9:%.*]] = fpext half [[HVAL]] to float
 ; MAX256-NEXT:    [[TMP0:%.*]] = insertelement <8 x float> poison, float [[I]], i32 0
-; MAX256-NEXT:    [[SHUFFLE11:%.*]] = shufflevector <8 x float> [[TMP0]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX256-NEXT:    [[TMP1:%.*]] = insertelement <8 x float> poison, float [[FVAL:%.*]], i32 0
-; MAX256-NEXT:    [[SHUFFLE12:%.*]] = shufflevector <8 x float> [[TMP1]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX256-NEXT:    [[TMP2:%.*]] = fmul <8 x float> [[SHUFFLE11]], [[SHUFFLE12]]
-; MAX256-NEXT:    [[TMP3:%.*]] = fadd <8 x float> zeroinitializer, [[TMP2]]
-; MAX256-NEXT:    [[TMP4:%.*]] = insertelement <8 x float> poison, float [[I3]], i32 0
-; MAX256-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x float> [[TMP4]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX256-NEXT:    [[TMP5:%.*]] = fmul <8 x float> [[SHUFFLE]], [[SHUFFLE12]]
-; MAX256-NEXT:    [[TMP6:%.*]] = fadd <8 x float> zeroinitializer, [[TMP5]]
-; MAX256-NEXT:    [[TMP7:%.*]] = insertelement <8 x float> poison, float [[I6]], i32 0
-; MAX256-NEXT:    [[SHUFFLE5:%.*]] = shufflevector <8 x float> [[TMP7]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX256-NEXT:    [[TMP8:%.*]] = fmul <8 x float> [[SHUFFLE5]], [[SHUFFLE12]]
+; MAX256-NEXT:    [[TMP1:%.*]] = shufflevector <8 x float> [[TMP0]], <8 x float> poison, <8 x i32> zeroinitializer
+; MAX256-NEXT:    [[TMP2:%.*]] = insertelement <8 x float> poison, float [[FVAL:%.*]], i32 0
+; MAX256-NEXT:    [[TMP3:%.*]] = shufflevector <8 x float> [[TMP2]], <8 x float> poison, <8 x i32> zeroinitializer
+; MAX256-NEXT:    [[TMP4:%.*]] = fmul <8 x float> [[TMP1]], [[TMP3]]
+; MAX256-NEXT:    [[TMP5:%.*]] = fadd <8 x float> zeroinitializer, [[TMP4]]
+; MAX256-NEXT:    [[TMP6:%.*]] = insertelement <8 x float> poison, float [[I3]], i32 0
+; MAX256-NEXT:    [[TMP7:%.*]] = shufflevector <8 x float> [[TMP6]], <8 x float> poison, <8 x i32> zeroinitializer
+; MAX256-NEXT:    [[TMP8:%.*]] = fmul <8 x float> [[TMP7]], [[TMP3]]
 ; MAX256-NEXT:    [[TMP9:%.*]] = fadd <8 x float> zeroinitializer, [[TMP8]]
-; MAX256-NEXT:    [[TMP10:%.*]] = insertelement <8 x float> poison, float [[I9]], i32 0
-; MAX256-NEXT:    [[SHUFFLE8:%.*]] = shufflevector <8 x float> [[TMP10]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX256-NEXT:    [[TMP11:%.*]] = fmul <8 x float> [[SHUFFLE8]], [[SHUFFLE12]]
-; MAX256-NEXT:    [[TMP12:%.*]] = fadd <8 x float> zeroinitializer, [[TMP11]]
+; MAX256-NEXT:    [[TMP10:%.*]] = insertelement <8 x float> poison, float [[I6]], i32 0
+; MAX256-NEXT:    [[TMP11:%.*]] = shufflevector <8 x float> [[TMP10]], <8 x float> poison, <8 x i32> zeroinitializer
+; MAX256-NEXT:    [[TMP12:%.*]] = fmul <8 x float> [[TMP11]], [[TMP3]]
+; MAX256-NEXT:    [[TMP13:%.*]] = fadd <8 x float> zeroinitializer, [[TMP12]]
+; MAX256-NEXT:    [[TMP14:%.*]] = insertelement <8 x float> poison, float [[I9]], i32 0
+; MAX256-NEXT:    [[TMP15:%.*]] = shufflevector <8 x float> [[TMP14]], <8 x float> poison, <8 x i32> zeroinitializer
+; MAX256-NEXT:    [[TMP16:%.*]] = fmul <8 x float> [[TMP15]], [[TMP3]]
+; MAX256-NEXT:    [[TMP17:%.*]] = fadd <8 x float> zeroinitializer, [[TMP16]]
 ; MAX256-NEXT:    switch i32 undef, label [[BB5:%.*]] [
 ; MAX256-NEXT:    i32 0, label [[BB2:%.*]]
 ; MAX256-NEXT:    i32 1, label [[BB3:%.*]]
 ; MAX256-NEXT:    i32 2, label [[BB4:%.*]]
 ; MAX256-NEXT:    ]
 ; MAX256:       bb3:
 ; MAX256-NEXT:    br label [[BB2]]
 ; MAX256:       bb4:
 ; MAX256-NEXT:    br label [[BB2]]
 ; MAX256:       bb5:
 ; MAX256-NEXT:    br label [[BB2]]
 ; MAX256:       bb2:
-; MAX256-NEXT:    [[TMP13:%.*]] = phi <8 x float> [ [[TMP6]], [[BB3]] ], [ [[SHUFFLE12]], [[BB4]] ], [ [[SHUFFLE12]], [[BB5]] ], [ [[SHUFFLE12]], [[BB1]] ]
-; MAX256-NEXT:    [[TMP14:%.*]] = phi <8 x float> [ [[TMP9]], [[BB3]] ], [ [[SHUFFLE12]], [[BB4]] ], [ [[TMP9]], [[BB5]] ], [ [[TMP9]], [[BB1]] ]
-; MAX256-NEXT:    [[TMP15:%.*]] = phi <8 x float> [ [[TMP12]], [[BB3]] ], [ [[TMP12]], [[BB4]] ], [ [[SHUFFLE12]], [[BB5]] ], [ [[TMP12]], [[BB1]] ]
-; MAX256-NEXT:    [[TMP16:%.*]] = phi <8 x float> [ [[TMP3]], [[BB3]] ], [ [[TMP3]], [[BB4]] ], [ [[TMP3]], [[BB5]] ], [ [[SHUFFLE12]], [[BB1]] ]
-; MAX256-NEXT:    [[TMP17:%.*]] = extractelement <8 x float> [[TMP14]], i32 7
-; MAX256-NEXT:    store float [[TMP17]], ptr undef, align 4
+; MAX256-NEXT:    [[TMP18:%.*]] = phi <8 x float> [ [[TMP9]], [[BB3]] ], [ [[TMP3]], [[BB4]] ], [ [[TMP3]], [[BB5]] ], [ [[TMP3]], [[BB1]] ]
+; MAX256-NEXT:    [[TMP19:%.*]] = phi <8 x float> [ [[TMP13]], [[BB3]] ], [ [[TMP3]], [[BB4]] ], [ [[TMP13]], [[BB5]] ], [ [[TMP13]], [[BB1]] ]
+; MAX256-NEXT:    [[TMP20:%.*]] = phi <8 x float> [ [[TMP17]], [[BB3]] ], [ [[TMP17]], [[BB4]] ], [ [[TMP3]], [[BB5]] ], [ [[TMP17]], [[BB1]] ]
+; MAX256-NEXT:    [[TMP21:%.*]] = phi <8 x float> [ [[TMP5]], [[BB3]] ], [ [[TMP5]], [[BB4]] ], [ [[TMP5]], [[BB5]] ], [ [[TMP3]], [[BB1]] ]
+; MAX256-NEXT:    [[TMP22:%.*]] = extractelement <8 x float> [[TMP19]], i32 7
+; MAX256-NEXT:    store float [[TMP22]], ptr undef, align 4
 ; MAX256-NEXT:    ret void
 ;
 ; MAX1024-LABEL: @phi_float32(
 ; MAX1024-NEXT:  bb:
 ; MAX1024-NEXT:    br label [[BB1:%.*]]
 ; MAX1024:       bb1:
-; MAX1024-NEXT:    [[I:%.*]] = fpext half [[HVAL:%.*]] to float
-; MAX1024-NEXT:    [[I3:%.*]] = fpext half [[HVAL]] to float
-; MAX1024-NEXT:    [[I6:%.*]] = fpext half [[HVAL]] to float
-; MAX1024-NEXT:    [[I9:%.*]] = fpext half [[HVAL]] to float
-; MAX1024-NEXT:    [[TMP0:%.*]] = insertelement <8 x float> poison, float [[I]], i32 0
-; MAX1024-NEXT:    [[SHUFFLE11:%.*]] = shufflevector <8 x float> [[TMP0]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX1024-NEXT:    [[TMP1:%.*]] = insertelement <8 x float> poison, float [[FVAL:%.*]], i32 0
-; MAX1024-NEXT:    [[SHUFFLE12:%.*]] = shufflevector <8 x float> [[TMP1]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX1024-NEXT:    [[TMP2:%.*]] = fmul <8 x float> [[SHUFFLE11]], [[SHUFFLE12]]
-; MAX1024-NEXT:    [[TMP3:%.*]] = fadd <8 x float> zeroinitializer, [[TMP2]]
-; MAX1024-NEXT:    [[TMP4:%.*]] = insertelement <8 x float> poison, float [[I3]], i32 0
-; MAX1024-NEXT:    [[SHUFFLE:%.*]] = shufflevector <8 x float> [[TMP4]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX1024-NEXT:    [[TMP5:%.*]] = fmul <8 x float> [[SHUFFLE]], [[SHUFFLE12]]
-; MAX1024-NEXT:    [[TMP6:%.*]] = fadd <8 x float> zeroinitializer, [[TMP5]]
-; MAX1024-NEXT:    [[TMP7:%.*]] = insertelement <8 x float> poison, float [[I6]], i32 0
-; MAX1024-NEXT:    [[SHUFFLE5:%.*]] = shufflevector <8 x float> [[TMP7]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX1024-NEXT:    [[TMP8:%.*]] = fmul <8 x float> [[SHUFFLE5]], [[SHUFFLE12]]
-; MAX1024-NEXT:    [[TMP9:%.*]] = fadd <8 x float> zeroinitializer, [[TMP8]]
-; MAX1024-NEXT:    [[TMP10:%.*]] = insertelement <8 x float> poison, float [[I9]], i32 0
-; MAX1024-NEXT:    [[SHUFFLE8:%.*]] = shufflevector <8 x float> [[TMP10]], <8 x float> poison, <8 x i32> zeroinitializer
-; MAX1024-NEXT:    [[TMP11:%.*]] = fmul <8 x float> [[SHUFFLE8]], [[SHUFFLE12]]
-; MAX1024-NEXT:    [[TMP12:%.*]] = fadd <8 x float> zeroinitializer, [[TMP11]]
+; MAX1024-NEXT:    [[TMP0:%.*]] = insertelement <4 x half> poison, half [[HVAL:%.*]], i32 0
+; MAX1024-NEXT:    [[TMP1:%.*]] = shufflevector <4 x half> [[TMP0]], <4 x half> poison, <4 x i32> zeroinitializer
+; MAX1024-NEXT:    [[TMP2:%.*]] = fpext <4 x half> [[TMP1]] to <4 x float>
+; MAX1024-NEXT:    [[TMP3:%.*]] = shufflevector <4 x float> [[TMP2]], <4 x float> poison, <32 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 1, i32 2, i32 2, i32 2, i32 2, i32 2, i32 2, i32 2, i32 2, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3, i32 3>
+; MAX1024-NEXT:    [[TMP4:%.*]] = insertelement <32 x float> poison, float [[FVAL:%.*]], i32 0
+; MAX1024-NEXT:    [[TMP5:%.*]] = shufflevector <32 x float> [[TMP4]], <32 x float> poison, <32 x i32> zeroinitializer
+; MAX1024-NEXT:    [[TMP6:%.*]] = fmul <32 x float> [[TMP3]], [[TMP5]]
+; MAX1024-NEXT:    [[TMP7:%.*]] = fadd <32 x float> zeroinitializer, [[TMP6]]
+; MAX1024-NEXT:    [[TMP8:%.*]] = shufflevector <32 x float> [[TMP4]], <32 x float> [[TMP7]], <32 x i32> <i32 0, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison, i32 poison>
+; MAX1024-NEXT:    [[TMP9:%.*]] = shufflevector <32 x float> [[TMP8]], <32 x float> poison, <32 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>
 ; MAX1024-NEXT:    switch i32 undef, label [[BB5:%.*]] [
 ; MAX1024-NEXT:    i32 0, label [[BB2:%.*]]
 ; MAX1024-NEXT:    i32 1, label [[BB3:%.*]]
 ; MAX1024-NEXT:    i32 2, label [[BB4:%.*]]
 ; MAX1024-NEXT:    ]
 ; MAX1024:       bb3:
 ; MAX1024-NEXT:    br label [[BB2]]
 ; MAX1024:       bb4:
+; MAX1024-NEXT:    [[TMP10:%.*]] = shufflevector <32 x float> [[TMP8]], <32 x float> [[TMP7]], <32 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63>
 ; MAX1024-NEXT:    br label [[BB2]]
 ; MAX1024:       bb5:
+; MAX1024-NEXT:    [[TMP11:%.*]] = shufflevector <32 x float> [[TMP8]], <32 x float> [[TMP7]], <32 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63>
 ; MAX1024-NEXT:    br label [[BB2]]
 ; MAX1024:       bb2:
-; MAX1024-NEXT:    [[TMP13:%.*]] = phi <8 x float> [ [[TMP6]], [[BB3]] ], [ [[SHUFFLE12]], [[BB4]] ], [ [[SHUFFLE12]], [[BB5]] ], [ [[SHUFFLE12]], [[BB1]] ]
-; MAX1024-NEXT:    [[TMP14:%.*]] = phi <8 x float> [ [[TMP9]], [[BB3]] ], [ [[SHUFFLE12]], [[BB4]] ], [ [[TMP9]], [[BB5]] ], [ [[TMP9]], [[BB1]] ]
-; MAX1024-NEXT:    [[TMP15:%.*]] = phi <8 x float> [ [[TMP12]], [[BB3]] ], [ [[TMP12]], [[BB4]] ], [ [[SHUFFLE12]], [[BB5]] ], [ [[TMP12]], [[BB1]] ]
-; MAX1024-NEXT:    [[TMP16:%.*]] = phi <8 x float> [ [[TMP3]], [[BB3]] ], [ [[TMP3]], [[BB4]] ], [ [[TMP3]], [[BB5]] ], [ [[SHUFFLE12]], [[BB1]] ]
-; MAX1024-NEXT:    [[TMP17:%.*]] = extractelement <8 x float> [[TMP14]], i32 7
-; MAX1024-NEXT:    store float [[TMP17]], ptr undef, align 4
+; MAX1024-NEXT:    [[TMP12:%.*]] = phi <32 x float> [ [[TMP7]], [[BB3]] ], [ [[TMP10]], [[BB4]] ], [ [[TMP11]], [[BB5]] ], [ [[TMP9]], [[BB1]] ]
+; MAX1024-NEXT:    [[TMP13:%.*]] = extractelement <32 x float> [[TMP12]], i32 15
+; MAX1024-NEXT:    store float [[TMP13]], ptr undef, align 4
 ; MAX1024-NEXT:    ret void
 ;
 bb:
   br label %bb1
 
 bb1:
   %i = fpext half %hval to float
   %i1 = fmul float %i, %fval