[SLP] Extract intermediate insertelement for external use

[anton-afanasyev]

Summary

If insertelement, extracted from vectorized bundle, is intermediate,
i.e. not the last one, then shuffle vectorized value with original
vector to get correct value for external use.

Fixes PR52275 (#51617)

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

731–732

Make it a separate NFC patch

[anton-afanasyev]

Make a separate NFC patch for default parameter of getInsertIndex

[anton-afanasyev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

731–732

Done

[ABataev]

Hmm, looks like the problem appears only if we have insertelements to be vectorized not from vectorizeInsertElementInst, where we have a proper check. Maybe just add a check for insertelements in other functions and just expect them from vectorizeInsertElementInst?

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

779

Drop llvm:

787

std::next(Mask.begin(), NumElts) -> Mask.end()

787

Would be good to fill unused elements with UndefMaskElem instead, which may help to improve cost in the future for large vectors.

791

What if InsertIdx is UndefMaskElem?

5925–5947

What if the same insertelement is used in several users? Would not we miss some cost here?

[anton-afanasyev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

779

Done

787

std::next(Mask.begin(), NumElts) -> Mask.end()

Sure, thanks.

Would be good to fill unused elements with UndefMaskElem instead, which may help to improve cost in the future for large vectors.

What do you mean by unused elements? We have to copy whole vector (with several replaced elements).

791

I forgot about it, thanks. But it may be reasonable to get rid of non-constant and undef indices at all, at buildTree() stage: https://reviews.llvm.org/D119623

[anton-afanasyev]

Address some comments, rebase against D119623

[anton-afanasyev]

In D119536#3314640, @ABataev wrote:

Hmm, looks like the problem appears only if we have insertelements to be vectorized not from vectorizeInsertElementInst, where we have a proper check. Maybe just add a check for insertelements in other functions and just expect them from vectorizeInsertElementInst?

Well, that would definitely solve the problem. There is also another way: on the contrary, remove hasOneUse() check from vectorizeInsertElementInst. This patch allows it.

[anton-afanasyev]

Fix rebase

[anton-afanasyev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5925–5947

Why? See line 5924 above, "We only add extract cost once for the same scalar", comment states.

[ABataev]

In D119536#3316636, @anton-afanasyev wrote:

In D119536#3314640, @ABataev wrote:

Hmm, looks like the problem appears only if we have insertelements to be vectorized not from vectorizeInsertElementInst, where we have a proper check. Maybe just add a check for insertelements in other functions and just expect them from vectorizeInsertElementInst?

Well, that would definitely solve the problem. There is also another way: on the contrary, remove hasOneUse() check from vectorizeInsertElementInst. This patch allows it.

Yes, sure, but it will require some extra stuff and some time to polish and fix all corner cases. If we’d like to fix it in 14.0, better to start with a simple fix and then try to implement more complex feature.

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5925–5947

Currently it is not quite correct for scalars (too optimistic) but will be fixed in future, once improved reductions support is committed. But for insertelements it is not correct for sure. If one instruction is a part of several build vector sequences, you need to calculate the shuffles for each of them independently. Same applies to the final vector emission.

[anton-afanasyev]

Well, that would definitely solve the problem. There is also another way: on the contrary, remove hasOneUse() check from vectorizeInsertElementInst. This patch allows it.

Yes, sure, but it will require some extra stuff and some time to polish and fix all corner cases. If we’d like to fix it in 14.0, better to start with a simple fix and then try to implement more complex feature.

Sounds reasonable, thanks! Please take a look at the simple fix: D119679 .

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5925–5947

If one insertelement is a part of several build vector sequences, it is enough to calculate the only one shuffle for any of them, isn't it? Or do you mean issue with scheduling of new shuffles?

[ABataev]

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

5925–5947

I think you need to calculate the cost for each buildvector sequence, this insertelement is part of. You need to shuffle the resulting vector with n other vectors.

[anton-afanasyev]

Abandoning this. PR52275 was solved by D119679 and following patches.

llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp

static void inversePermutation(ArrayRef<unsigned> Indices,
   const unsigned E = Indices.size();
   Mask.resize(E, UndefMaskElem);
   for (unsigned I = 0; I < E; ++I)
     Mask[Indices[I]] = I;
 }
 
 /// \returns inserting index of InsertElement or InsertValue instruction,
 /// using Offset as base offset for index.
 static Optional<unsigned> getInsertIndex(Value *InsertInst,
                                          unsigned Offset = 0) {
   int Index = Offset;
   if (auto *IE = dyn_cast<InsertElementInst>(InsertInst)) {
     if (auto *CI = dyn_cast<ConstantInt>(IE->getOperand(2))) {
       auto *VT = cast<FixedVectorType>(IE->getType());
       if (CI->getValue().uge(VT->getNumElements()))
         return None;
       Index *= VT->getNumElements();
       Index += CI->getZExtValue();
static void reorderScalars(SmallVectorImpl<Value *> &Scalars,
   SmallVector<Value *> Prev(Scalars.size(),
                             UndefValue::get(Scalars.front()->getType()));
   Prev.swap(Scalars);
   for (unsigned I = 0, E = Prev.size(); I < E; ++I)
     if (Mask[I] != UndefMaskElem)
       Scalars[Mask[I]] = Prev[I];
 }
 
+/// \returns shuffle mask if \p Insert is not the last of \p Inserts
+static Optional<SmallVector<int>>
+getMaskIfIntermediateInsert(Value *Insert, ArrayRef<Value *> Inserts) {
+  auto *LastInsert = cast<Instruction>(*find_if(Inserts, [Inserts](Value *V) {
+    return all_of(Inserts, [V](Value *VV) {
+      return V != cast<Instruction>(VV)->getOperand(0);
+    });
+  }));
+  if (Insert != LastInsert) {
+    const unsigned NumElts =
+        cast<FixedVectorType>(Insert->getType())->getNumElements();
+    SmallVector<int> Mask(NumElts);
+    std::iota(Mask.begin(), Mask.end(), 0);
+
+    Value *CurrentInsert = LastInsert;
+    while (CurrentInsert != Insert) {
+      Optional<int> InsertIdx = getInsertIndex(CurrentInsert);
+      Mask[*InsertIdx] = *InsertIdx + NumElts;
+      CurrentInsert = cast<Instruction>(CurrentInsert)->getOperand(0);
+    }
+    return Mask;
+  }
+  return None;
+}
+
 namespace slpvectorizer {
 
 /// Bottom Up SLP Vectorizer.
 class BoUpSLP {
   struct TreeEntry;
   struct ScheduleData;
 
 public:
InstructionCost BoUpSLP::getTreeCost(ArrayRef<Value *> VectorizedVals) {
   SmallPtrSet<Value *, 16> ExtractCostCalculated;
   InstructionCost ExtractCost = 0;
   SmallVector<unsigned> VF;
   SmallVector<SmallVector<int>> ShuffleMask;
   SmallVector<Value *> FirstUsers;
   SmallVector<APInt> DemandedElts;
   for (ExternalUser &EU : ExternalUses) {
     // We only add extract cost once for the same scalar.
     if (!isa_and_nonnull<InsertElementInst>(EU.User) &&
         !ExtractCostCalculated.insert(EU.Scalar).second)
       continue;
 
     // Uses by ephemeral values are free (because the ephemeral value will be
     // removed prior to code generation, and so the extraction will be
     // removed as well).
     if (EphValues.count(EU.User))
       continue;
 
-    // No extract cost for vector "scalar"
-    if (isa<FixedVectorType>(EU.Scalar->getType()))
+    // Add extract cost for vector "scalar", if it's not the last insert
+    if (isa<FixedVectorType>(EU.Scalar->getType())) {
+      TreeEntry *E = getTreeEntry(EU.Scalar);
+      assert(E && "Invalid scalar");
+      if (auto Mask = getMaskIfIntermediateInsert(EU.Scalar, E->Scalars)) {
+        InstructionCost C = TTI->getShuffleCost(
+            TTI::SK_PermuteTwoSrc, cast<FixedVectorType>(EU.Scalar->getType()),
+            *Mask);
+        ExtractCost += C;
+      }
       continue;
+    }
 
     // Already counted the cost for external uses when tried to adjust the cost
     // for extractelements, no need to add it again.
     if (isa<ExtractElementInst>(EU.Scalar))
       continue;
 
     // If found user is an insertelement, do not calculate extract cost but try
     // to detect it as a final shuffled/identity match.
     if (auto *VU = dyn_cast_or_null<InsertElementInst>(EU.User)) {
auto ExtractAndExtendIfNeeded = [&](Value *Vec) {
           return Ex;
         if (MinBWs[ScalarRoot].second)
           return Builder.CreateSExt(Ex, Scalar->getType());
         return Builder.CreateZExt(Ex, Scalar->getType());
       }
       assert(isa<FixedVectorType>(Scalar->getType()) &&
              isa<InsertElementInst>(Scalar) &&
              "In-tree scalar of vector type is not insertelement?");
+      // For insertelements, if Scalar is not the last insert,
+      // then shuffle vectorized value with original vector
+      if (auto Mask = getMaskIfIntermediateInsert(Scalar, E->Scalars)) {
+        auto *FirstInsert =
+            cast<Instruction>(*find_if(E->Scalars, [E](Value *V) {
+              return !is_contained(E->Scalars,
+                                   cast<Instruction>(V)->getOperand(0));
+            }));
+        Vec =
+            Builder.CreateShuffleVector(Vec, FirstInsert->getOperand(0), *Mask);
+      }
+
       return Vec;
     };
     // If User == nullptr, the Scalar is used as extra arg. Generate
     // ExtractElement instruction and update the record for this scalar in
     // ExternallyUsedValues.
     if (!User) {
       assert(ExternallyUsedValues.count(Scalar) &&
              "Scalar with nullptr as an external user must be registered in "

llvm/test/Transforms/SLPVectorizer/X86/pr52275.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
 ; RUN: opt < %s -slp-vectorizer -S -mcpu=core-i7 | FileCheck %s
 ; RUN: opt < %s -slp-vectorizer -slp-threshold=-100 -S -mcpu=core-i7 | FileCheck %s --check-prefix=FORCE_SLP
 
 define  <4 x i8> @test(<4 x i8> %v, i8* %x) {
 ; CHECK-LABEL: @test(
-; CHECK-NEXT:    [[G1:%.*]] = getelementptr inbounds i8, i8* [[X:%.*]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i8* [[X]] to <2 x i8>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i8>, <2 x i8>* [[TMP1]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i8> [[TMP2]], <2 x i8> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; CHECK-NEXT:    [[V11:%.*]] = shufflevector <4 x i8> [[V:%.*]], <4 x i8> [[TMP3]], <4 x i32> <i32 4, i32 5, i32 2, i32 3>
-; CHECK-NEXT:    [[V2:%.*]] = add <4 x i8> [[V11]], [[V11]]
+; CHECK-NEXT:    [[X0:%.*]] = load i8, i8* [[X:%.*]], align 4
+; CHECK-NEXT:    [[G1:%.*]] = getelementptr inbounds i8, i8* [[X]], i64 1
+; CHECK-NEXT:    [[X1:%.*]] = load i8, i8* [[G1]], align 4
+; CHECK-NEXT:    [[V0:%.*]] = insertelement <4 x i8> [[V:%.*]], i8 [[X0]], i64 0
+; CHECK-NEXT:    [[V1:%.*]] = insertelement <4 x i8> [[V0]], i8 [[X1]], i64 1
+; CHECK-NEXT:    [[V2:%.*]] = add <4 x i8> [[V0]], [[V1]]
 ; CHECK-NEXT:    ret <4 x i8> [[V2]]
 ;
 ; FORCE_SLP-LABEL: @test(
 ; FORCE_SLP-NEXT:    [[G1:%.*]] = getelementptr inbounds i8, i8* [[X:%.*]], i64 1
 ; FORCE_SLP-NEXT:    [[TMP1:%.*]] = bitcast i8* [[X]] to <2 x i8>*
 ; FORCE_SLP-NEXT:    [[TMP2:%.*]] = load <2 x i8>, <2 x i8>* [[TMP1]], align 4
 ; FORCE_SLP-NEXT:    [[TMP3:%.*]] = shufflevector <2 x i8> [[TMP2]], <2 x i8> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; FORCE_SLP-NEXT:    [[V11:%.*]] = shufflevector <4 x i8> [[V:%.*]], <4 x i8> [[TMP3]], <4 x i32> <i32 4, i32 5, i32 2, i32 3>
-; FORCE_SLP-NEXT:    [[V2:%.*]] = add <4 x i8> [[V11]], [[V11]]
+; FORCE_SLP-NEXT:    [[TMP4:%.*]] = shufflevector <4 x i8> [[V11]], <4 x i8> [[V]], <4 x i32> <i32 0, i32 5, i32 2, i32 3>
+; FORCE_SLP-NEXT:    [[V2:%.*]] = add <4 x i8> [[TMP4]], [[V11]]
 ; FORCE_SLP-NEXT:    ret <4 x i8> [[V2]]
 ;
   %x0 = load i8, i8* %x, align 4
   %g1 = getelementptr inbounds i8, i8* %x, i64 1
   %x1 = load i8, i8* %g1, align 4
   %v0 = insertelement <4 x i8> %v, i8 %x0, i64 0
   %v1 = insertelement <4 x i8> %v0, i8 %x1, i64 1
   %v2 = add <4 x i8> %v0, %v1
   ret <4 x i8> %v2
 }
 
 define  <2 x i8> @test2(<2 x i8> %t6, i32* %t1) {
 ; CHECK-LABEL: @test2(
 ; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
-; CHECK-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP3]], [[TMP3]]
+; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> [[T6:%.*]], <2 x i32> <i32 0, i32 3>
+; CHECK-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP3]], [[TMP4]]
 ; CHECK-NEXT:    ret <2 x i8> [[T11]]
 ;
 ; FORCE_SLP-LABEL: @test2(
 ; FORCE_SLP-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; FORCE_SLP-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; FORCE_SLP-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; FORCE_SLP-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
-; FORCE_SLP-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP3]], [[TMP3]]
+; FORCE_SLP-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> [[T6:%.*]], <2 x i32> <i32 0, i32 3>
+; FORCE_SLP-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP3]], [[TMP4]]
 ; FORCE_SLP-NEXT:    ret <2 x i8> [[T11]]
 ;
   %t3 = load i32, i32* %t1, align 4
   %t4 = getelementptr inbounds i32, i32* %t1, i64 1
   %t5 = load i32, i32* %t4, align 4
   %t7 = trunc i32 %t3 to i8
   %t8 = insertelement <2 x i8> %t6, i8 %t7, i64 0
   %t9 = trunc i32 %t5 to i8
   %t10 = insertelement <2 x i8> %t8, i8 %t9, i64 1
   %t11 = add <2 x i8> %t10, %t8
   ret <2 x i8> %t11
 }
 
 define  <2 x i8> @test_reorder(<2 x i8> %t6, i32* %t1) {
 ; CHECK-LABEL: @test_reorder(
-; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <2 x i32> <i32 1, i32 0>
-; CHECK-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP4]], [[TMP4]]
+; CHECK-NEXT:    [[T3:%.*]] = load i32, i32* [[T1:%.*]], align 4
+; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1]], i64 1
+; CHECK-NEXT:    [[T5:%.*]] = load i32, i32* [[T4]], align 4
+; CHECK-NEXT:    [[T7:%.*]] = trunc i32 [[T3]] to i8
+; CHECK-NEXT:    [[T8:%.*]] = insertelement <2 x i8> [[T6:%.*]], i8 [[T7]], i64 1
+; CHECK-NEXT:    [[T9:%.*]] = trunc i32 [[T5]] to i8
+; CHECK-NEXT:    [[T10:%.*]] = insertelement <2 x i8> [[T8]], i8 [[T9]], i64 0
+; CHECK-NEXT:    [[T11:%.*]] = add <2 x i8> [[T10]], [[T8]]
 ; CHECK-NEXT:    ret <2 x i8> [[T11]]
 ;
 ; FORCE_SLP-LABEL: @test_reorder(
 ; FORCE_SLP-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; FORCE_SLP-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; FORCE_SLP-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; FORCE_SLP-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
 ; FORCE_SLP-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <2 x i32> <i32 1, i32 0>
-; FORCE_SLP-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP4]], [[TMP4]]
+; FORCE_SLP-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i8> [[TMP4]], <2 x i8> [[T6:%.*]], <2 x i32> <i32 2, i32 1>
+; FORCE_SLP-NEXT:    [[T11:%.*]] = add <2 x i8> [[TMP4]], [[TMP5]]
 ; FORCE_SLP-NEXT:    ret <2 x i8> [[T11]]
 ;
   %t3 = load i32, i32* %t1, align 4
   %t4 = getelementptr inbounds i32, i32* %t1, i64 1
   %t5 = load i32, i32* %t4, align 4
   %t7 = trunc i32 %t3 to i8
   %t8 = insertelement <2 x i8> %t6, i8 %t7, i64 1
   %t9 = trunc i32 %t5 to i8
   %t10 = insertelement <2 x i8> %t8, i8 %t9, i64 0
   %t11 = add <2 x i8> %t10, %t8
   ret <2 x i8> %t11
 }
 
 define  <4 x i8> @test_subvector(<4 x i8> %t6, i32* %t1) {
 ; CHECK-LABEL: @test_subvector(
 ; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; CHECK-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
 ; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; CHECK-NEXT:    [[T101:%.*]] = shufflevector <4 x i8> [[T6:%.*]], <4 x i8> [[TMP4]], <4 x i32> <i32 4, i32 5, i32 2, i32 3>
-; CHECK-NEXT:    [[T11:%.*]] = add <4 x i8> [[T101]], [[T101]]
+; CHECK-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i8> [[T101]], <4 x i8> [[T6]], <4 x i32> <i32 0, i32 5, i32 2, i32 3>
+; CHECK-NEXT:    [[T11:%.*]] = add <4 x i8> [[T101]], [[TMP5]]
 ; CHECK-NEXT:    ret <4 x i8> [[T11]]
 ;
 ; FORCE_SLP-LABEL: @test_subvector(
 ; FORCE_SLP-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; FORCE_SLP-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; FORCE_SLP-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; FORCE_SLP-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
 ; FORCE_SLP-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
 ; FORCE_SLP-NEXT:    [[T101:%.*]] = shufflevector <4 x i8> [[T6:%.*]], <4 x i8> [[TMP4]], <4 x i32> <i32 4, i32 5, i32 2, i32 3>
-; FORCE_SLP-NEXT:    [[T11:%.*]] = add <4 x i8> [[T101]], [[T101]]
+; FORCE_SLP-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i8> [[T101]], <4 x i8> [[T6]], <4 x i32> <i32 0, i32 5, i32 2, i32 3>
+; FORCE_SLP-NEXT:    [[T11:%.*]] = add <4 x i8> [[T101]], [[TMP5]]
 ; FORCE_SLP-NEXT:    ret <4 x i8> [[T11]]
 ;
   %t3 = load i32, i32* %t1, align 4
   %t4 = getelementptr inbounds i32, i32* %t1, i64 1
   %t5 = load i32, i32* %t4, align 4
   %t7 = trunc i32 %t3 to i8
   %t8 = insertelement <4 x i8> %t6, i8 %t7, i64 0
   %t9 = trunc i32 %t5 to i8
   %t10 = insertelement <4 x i8> %t8, i8 %t9, i64 1
   %t11 = add <4 x i8> %t10, %t8
   ret <4 x i8> %t11
 }
 
 define  <4 x i8> @test_subvector_reorder(<4 x i8> %t6, i32* %t1) {
 ; CHECK-LABEL: @test_subvector_reorder(
-; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
-; CHECK-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
-; CHECK-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
-; CHECK-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <4 x i32> <i32 1, i32 0, i32 undef, i32 undef>
-; CHECK-NEXT:    [[T81:%.*]] = shufflevector <4 x i8> [[T6:%.*]], <4 x i8> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
-; CHECK-NEXT:    [[T11:%.*]] = add <4 x i8> [[T81]], [[T81]]
+; CHECK-NEXT:    [[T3:%.*]] = load i32, i32* [[T1:%.*]], align 4
+; CHECK-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1]], i64 1
+; CHECK-NEXT:    [[T5:%.*]] = load i32, i32* [[T4]], align 4
+; CHECK-NEXT:    [[T7:%.*]] = trunc i32 [[T3]] to i8
+; CHECK-NEXT:    [[T8:%.*]] = insertelement <4 x i8> [[T6:%.*]], i8 [[T7]], i64 3
+; CHECK-NEXT:    [[T9:%.*]] = trunc i32 [[T5]] to i8
+; CHECK-NEXT:    [[T10:%.*]] = insertelement <4 x i8> [[T8]], i8 [[T9]], i64 2
+; CHECK-NEXT:    [[T11:%.*]] = add <4 x i8> [[T10]], [[T8]]
 ; CHECK-NEXT:    ret <4 x i8> [[T11]]
 ;
 ; FORCE_SLP-LABEL: @test_subvector_reorder(
 ; FORCE_SLP-NEXT:    [[T4:%.*]] = getelementptr inbounds i32, i32* [[T1:%.*]], i64 1
 ; FORCE_SLP-NEXT:    [[TMP1:%.*]] = bitcast i32* [[T1]] to <2 x i32>*
 ; FORCE_SLP-NEXT:    [[TMP2:%.*]] = load <2 x i32>, <2 x i32>* [[TMP1]], align 4
 ; FORCE_SLP-NEXT:    [[TMP3:%.*]] = trunc <2 x i32> [[TMP2]] to <2 x i8>
 ; FORCE_SLP-NEXT:    [[TMP4:%.*]] = shufflevector <2 x i8> [[TMP3]], <2 x i8> poison, <4 x i32> <i32 1, i32 0, i32 undef, i32 undef>
 ; FORCE_SLP-NEXT:    [[T81:%.*]] = shufflevector <4 x i8> [[T6:%.*]], <4 x i8> [[TMP4]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
-; FORCE_SLP-NEXT:    [[T11:%.*]] = add <4 x i8> [[T81]], [[T81]]
+; FORCE_SLP-NEXT:    [[TMP5:%.*]] = shufflevector <4 x i8> [[T81]], <4 x i8> [[T6]], <4 x i32> <i32 0, i32 1, i32 6, i32 3>
+; FORCE_SLP-NEXT:    [[T11:%.*]] = add <4 x i8> [[T81]], [[TMP5]]
 ; FORCE_SLP-NEXT:    ret <4 x i8> [[T11]]
 ;
   %t3 = load i32, i32* %t1, align 4
   %t4 = getelementptr inbounds i32, i32* %t1, i64 1
   %t5 = load i32, i32* %t4, align 4
   %t7 = trunc i32 %t3 to i8
   %t8 = insertelement <4 x i8> %t6, i8 %t7, i64 3
   %t9 = trunc i32 %t5 to i8
   %t10 = insertelement <4 x i8> %t8, i8 %t9, i64 2
   %t11 = add <4 x i8> %t10, %t8
   ret <4 x i8> %t11
 }