[InstCombine] canonicalize trunc + insert as bitcast + shuffle, part 1

[spatel]

Summary

This is the main patch for converting a truncated scalar that is inserted into a vector to bitcast+shuffle. We could go either way on patterns like this, but this direction will allow collapsing a pair of these sequences on the motivating example from issue #17113.

The patch is split into 3 parts to make it easier to see the progression of tests diffs. We allow inserting/shuffling into a different size vector for flexibility, so there are several test variations. The length-changing is handled by shortening/padding the shuffle mask with undef elements.

In part 1, handle the basic pattern:

inselt undef, (trunc T), IndexC --> shuffle (bitcast T), IdentityMask

Proof for the endian-dependency behaving as expected:
https://alive2.llvm.org/ce/z/BsA7yC

The TODO items for handling shifts and insert into an arbitrary base vector value are implemented as follow-ups.

[RKSimon]

LGTM with one (very) minor

llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp

1518

Add a head comment describing the fold.

[lebedev.ri]

Why is a trunc, and an one-use at that, a requirement?
It seems like looking through trunc should either be a follow-up for this fold, or for shuffle combining.

llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp

1525

Why is a trunc, and an one-use at that, a requirement?
Please at least add a FIXME.

[lebedev.ri]

Looking into follow-ups, it really seems like there should be part 0 without trunc handling.
If not, could you please explain why we must see a trunc?

[lebedev.ri]

Looking into follow-ups, it really seems like there should be part 0 without trunc handling.
If not, could you please explain why we must see a trunc?

[RKSimon]

@lebedev.ri Do you mean this pattern?

define <8 x i16> @src(i64 %a0) {
  %v = insertelement <2 x i64> poison, i64 %a0, i32 0
  %r = bitcast <2 x i64> %v to <8 x i16>
  ret <8 x i16> %r
}
define <8 x i16> @tgt(i64 %a0) {
  %v = bitcast i64 %a0 to <4 x i16>
  %r = shufflevector <4 x i16> %v, <4 x i16> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef>
  ret <8 x i16> %r
}

[lebedev.ri]

In D138872#3960253, @RKSimon wrote:

@lebedev.ri Do you mean this pattern?

define <8 x i16> @src(i64 %a0) {
  %v = insertelement <2 x i64> poison, i64 %a0, i32 0
  %r = bitcast <2 x i64> %v to <8 x i16>
  ret <8 x i16> %r
}
define <8 x i16> @tgt(i64 %a0) {
  %v = bitcast i64 %a0 to <4 x i16>
  %r = shufflevector <4 x i16> %v, <4 x i16> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef>
  ret <8 x i16> %r
}

I guess?

[spatel]

In D138872#3960319, @lebedev.ri wrote:

In D138872#3960253, @RKSimon wrote:

@lebedev.ri Do you mean this pattern?

define <8 x i16> @src(i64 %a0) {
  %v = insertelement <2 x i64> poison, i64 %a0, i32 0
  %r = bitcast <2 x i64> %v to <8 x i16>
  ret <8 x i16> %r
}
define <8 x i16> @tgt(i64 %a0) {
  %v = bitcast i64 %a0 to <4 x i16>
  %r = shufflevector <4 x i16> %v, <4 x i16> poison, <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 undef, i32 undef, i32 undef, i32 undef>
  ret <8 x i16> %r
}

I guess?

That's another related potentially missing canonicalization, but the pattern doesn't end at the insertelement. Without the trunc, there's just a single insertelement inst, so we can't do anything without a trunc?

llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp

1525

The one-use is there because the fold is minimally creating 2 instructions: bitcast + shuffle.
It gets more complicated when we match the shift in the next patch part.
In the final patch, we can reduce instructions in some cases, but others will gain an instruction. If we don't have the one-use limit, that could mean we'd end up +2 instructions. I don't want to risk a regression from that possibility.

[spatel]

Patch updated:

1. Added documentation comment for the general transform.
2. Added TODO for one-use restriction.

[lebedev.ri]

In D138872#3960604, @spatel wrote:

Patch updated:

1. Added documentation comment for the general transform.
2. Added TODO for one-use restriction.

Actually, i guess i was thinking about canonicalizing all insertelements @ constant idx into shuffles
(+the shuffle to convert the scalar into an appropriately-sized vector),
but i guess that is almost borderline bad?
So LG i suppose.

[spatel]

In D138872#3960754, @lebedev.ri wrote:

In D138872#3960604, @spatel wrote:

Patch updated:

1. Added documentation comment for the general transform.
2. Added TODO for one-use restriction.

Actually, i guess i was thinking about canonicalizing all insertelements @ constant idx into shuffles
(+the shuffle to convert the scalar into an appropriately-sized vector),
but i guess that is almost borderline bad?
So LG i suppose.

Ah, I see. Yes, we could turn all insert-to-constant-index into shuffles after bitcasting a scalar to a <1 x N> type. The insertelt is the direct representation though, so that's a tough sell. TBH even with a trunc, it's a borderline canonicalization; it could go either way. I think the backend is fine either way too, but I'll watch for fallout.

[DavidSpickett]

FYI, I tracked an SVE miscompile down to this change https://github.com/llvm/llvm-project/issues/59357. However, so far it looks like this change just made a pre-existing problem more obvious.

We'll follow up with you if that's not the case.

llvm/lib/Transforms/InstCombine/InstCombineVectorOps.cpp

 #include "llvm/IR/User.h"
 #include "llvm/IR/Value.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Transforms/InstCombine/InstCombiner.h"
 #include <cassert>
 #include <cstdint>
 #include <iterator>
+#include <numeric>
 #include <utility>
 
 #define DEBUG_TYPE "instcombine"
 
 using namespace llvm;
 using namespace PatternMatch;
 
 STATISTIC(NumAggregateReconstructionsSimplified,
static Instruction *narrowInsElt(InsertElementInst &InsElt,
   if (X->getType()->getScalarType() != Y->getType())
     return nullptr;
 
   // inselt (ext X), (ext Y), Index --> ext (inselt X, Y, Index)
   Value *NewInsElt = Builder.CreateInsertElement(X, Y, InsElt.getOperand(2));
   return CastInst::Create(CastOpcode, NewInsElt, InsElt.getType());
 }
 
+/// Try to convert scalar extraction ops (shift+trunc) with insertelt to
+/// bitcast and shuffle:
+/// inselt V, (lshr (trunc X)), IndexC --> shuffle (bitcast X), V, Mask
+static Instruction *foldTruncInsElt(InsertElementInst &InsElt, bool IsBigEndian,
+                                    InstCombiner::BuilderTy &Builder) {
+  // inselt undef, (trunc T), IndexC
+  // TODO: Allow any base vector value.
+  // TODO: The one-use limitation could be removed for some cases (eg, no
+  //       extra shuffle is needed and a shift is eliminated).
+  auto *VTy = dyn_cast<FixedVectorType>(InsElt.getType());
+  Value *T, *V = InsElt.getOperand(0);
+  uint64_t IndexC;
+  if (!VTy || !match(InsElt.getOperand(1), m_OneUse(m_Trunc(m_Value(T)))) ||
+      !match(InsElt.getOperand(2), m_ConstantInt(IndexC)) ||
+      !match(V, m_Undef()))
+    return nullptr;
+
+  Type *SrcTy = T->getType();
+  unsigned ScalarWidth = SrcTy->getScalarSizeInBits();
+  unsigned VecEltWidth = VTy->getScalarSizeInBits();
+  if (ScalarWidth % VecEltWidth != 0)
+    return nullptr;
+
+  unsigned NumEltsInScalar = ScalarWidth / VecEltWidth;
+  Value *X = T;
+  if ((IsBigEndian && IndexC == NumEltsInScalar - 1) ||
+      (!IsBigEndian && IndexC == 0)) {
+    // The insert is to the LSB end of the vector (depends on endian).
+    // That's all we need.
+  } else {
+    // TODO: Look through a shift-right and translate the insert index.
+    return nullptr;
+  }
+
+  // Bitcast the scalar to a vector type with the destination element type.
+  Type *CastTy = FixedVectorType::get(VTy->getElementType(), NumEltsInScalar);
+  Value *VecX = Builder.CreateBitCast(X, CastTy, "vec." + X->getName());
+
+  unsigned NumElts = VTy->getNumElements();
+  if (NumElts > NumEltsInScalar) {
+    // Pad the source vector with undef elements, so it matches the dest type.
+    SmallVector<int> IdentityPaddedMask(NumElts, UndefMaskElem);
+    for (unsigned i = 0; i != NumEltsInScalar; ++i)
+      IdentityPaddedMask[i] = i;
+    VecX = Builder.CreateShuffleVector(VecX, IdentityPaddedMask);
+  } else if (NumElts < NumEltsInScalar) {
+    // Narrow the source vector, so it matches the dest type.
+    SmallVector<int> IdentityExtractMask(NumElts);
+    std::iota(IdentityExtractMask.begin(), IdentityExtractMask.end(), 0);
+    VecX = Builder.CreateShuffleVector(VecX, IdentityExtractMask);
+  }
+
+  // Insert the truncated element using a select-shuffle. All lanes but one are
+  // from the base vector V.
+  SmallVector<int> SelectMask(NumElts);
+  std::iota(SelectMask.begin(), SelectMask.end(), 0);
+  SelectMask[IndexC] = (int)IndexC + NumElts;
+  return new ShuffleVectorInst(V, VecX, SelectMask);
+}
+
 Instruction *InstCombinerImpl::visitInsertElementInst(InsertElementInst &IE) {
   Value *VecOp    = IE.getOperand(0);
   Value *ScalarOp = IE.getOperand(1);
   Value *IdxOp    = IE.getOperand(2);
 
   if (auto *V = simplifyInsertElementInst(
           VecOp, ScalarOp, IdxOp, SQ.getWithInstruction(&IE)))
     return replaceInstUsesWith(IE, V);
if (Instruction *Splat = foldInsEltIntoSplat(IE))
     return Splat;
 
   if (Instruction *IdentityShuf = foldInsEltIntoIdentityShuffle(IE))
     return IdentityShuf;
 
   if (Instruction *Ext = narrowInsElt(IE, Builder))
     return Ext;
 
+  if (Instruction *Shuf = foldTruncInsElt(IE, DL.isBigEndian(), Builder))
+    return Shuf;
+
   return nullptr;
 }
 
 /// Return true if we can evaluate the specified expression tree if the vector
 /// elements were shuffled in a different order.
 static bool canEvaluateShuffled(Value *V, ArrayRef<int> Mask,
                                 unsigned Depth = 5) {
   // We can always reorder the elements of a constant.

llvm/test/Transforms/InstCombine/insert-trunc.ll

 ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=instcombine -S -data-layout="E" | FileCheck %s --check-prefixes=ALL
-; RUN: opt < %s -passes=instcombine -S -data-layout="e" | FileCheck %s --check-prefixes=ALL
+; RUN: opt < %s -passes=instcombine -S -data-layout="E" | FileCheck %s --check-prefixes=ALL,BE
+; RUN: opt < %s -passes=instcombine -S -data-layout="e" | FileCheck %s --check-prefixes=ALL,LE
 
 declare void @use(i8)
 declare void @use64(i64)
 
 define <4 x i16> @low_index_same_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @low_index_same_length_poison_basevec(
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 0
-; ALL-NEXT:    ret <4 x i16> [[R]]
+; BE-LABEL: @low_index_same_length_poison_basevec(
+; BE-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
+; BE-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 0
+; BE-NEXT:    ret <4 x i16> [[R]]
+;
+; LE-LABEL: @low_index_same_length_poison_basevec(
+; LE-NEXT:    [[VEC_X:%.*]] = bitcast i64 [[X:%.*]] to <4 x i16>
+; LE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_X]], <4 x i16> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; LE-NEXT:    ret <4 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 0
   ret <4 x i16> %r
 }
 
 define <4 x i16> @high_index_same_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @high_index_same_length_poison_basevec(
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 3
-; ALL-NEXT:    ret <4 x i16> [[R]]
+; BE-LABEL: @high_index_same_length_poison_basevec(
+; BE-NEXT:    [[VEC_X:%.*]] = bitcast i64 [[X:%.*]] to <4 x i16>
+; BE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_X]], <4 x i16> poison, <4 x i32> <i32 undef, i32 undef, i32 undef, i32 3>
+; BE-NEXT:    ret <4 x i16> [[R]]
+;
+; LE-LABEL: @high_index_same_length_poison_basevec(
+; LE-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
+; LE-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 3
+; LE-NEXT:    ret <4 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 3
   ret <4 x i16> %r
 }
 
 define <4 x i16> @wrong_index_same_length_poison_basevec(i64 %x) {
 ; ALL-LABEL: @wrong_index_same_length_poison_basevec(
 ; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
 ; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 1
 ; ALL-NEXT:    ret <4 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 1
   ret <4 x i16> %r
 }
 
 define <8 x i16> @low_index_longer_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @low_index_longer_length_poison_basevec(
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 0
-; ALL-NEXT:    ret <8 x i16> [[R]]
+; BE-LABEL: @low_index_longer_length_poison_basevec(
+; BE-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
+; BE-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 0
+; BE-NEXT:    ret <8 x i16> [[R]]
+;
+; LE-LABEL: @low_index_longer_length_poison_basevec(
+; LE-NEXT:    [[VEC_X:%.*]] = bitcast i64 [[X:%.*]] to <4 x i16>
+; LE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_X]], <4 x i16> poison, <8 x i32> <i32 0, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
+; LE-NEXT:    ret <8 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <8 x i16> poison, i16 %t, i64 0
   ret <8 x i16> %r
 }
 
 define <8 x i16> @high_index_longer_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @high_index_longer_length_poison_basevec(
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 3
-; ALL-NEXT:    ret <8 x i16> [[R]]
+; BE-LABEL: @high_index_longer_length_poison_basevec(
+; BE-NEXT:    [[VEC_X:%.*]] = bitcast i64 [[X:%.*]] to <4 x i16>
+; BE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_X]], <4 x i16> poison, <8 x i32> <i32 undef, i32 undef, i32 undef, i32 3, i32 undef, i32 undef, i32 undef, i32 undef>
+; BE-NEXT:    ret <8 x i16> [[R]]
+;
+; LE-LABEL: @high_index_longer_length_poison_basevec(
+; LE-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
+; LE-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 3
+; LE-NEXT:    ret <8 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <8 x i16> poison, i16 %t, i64 3
   ret <8 x i16> %r
 }
 
 define <8 x i16> @wrong_index_longer_length_poison_basevec(i64 %x) {
 ; ALL-LABEL: @wrong_index_longer_length_poison_basevec(
 ; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
 ; ALL-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 7
 ; ALL-NEXT:    ret <8 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <8 x i16> poison, i16 %t, i64 7
   ret <8 x i16> %r
 }
 
 define <2 x i16> @low_index_shorter_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @low_index_shorter_length_poison_basevec(
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <2 x i16> poison, i16 [[T]], i64 0
-; ALL-NEXT:    ret <2 x i16> [[R]]
+; BE-LABEL: @low_index_shorter_length_poison_basevec(
+; BE-NEXT:    [[T:%.*]] = trunc i64 [[X:%.*]] to i16
+; BE-NEXT:    [[R:%.*]] = insertelement <2 x i16> poison, i16 [[T]], i64 0
+; BE-NEXT:    ret <2 x i16> [[R]]
+;
+; LE-LABEL: @low_index_shorter_length_poison_basevec(
+; LE-NEXT:    [[VEC_X:%.*]] = bitcast i64 [[X:%.*]] to <4 x i16>
+; LE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_X]], <4 x i16> poison, <2 x i32> <i32 0, i32 undef>
+; LE-NEXT:    ret <2 x i16> [[R]]
 ;
   %t = trunc i64 %x to i16
   %r = insertelement <2 x i16> poison, i16 %t, i64 0
   ret <2 x i16> %r
 }
 
 define <4 x i8> @wrong_index_shorter_length_poison_basevec(i64 %x) {
 ; ALL-LABEL: @wrong_index_shorter_length_poison_basevec(

 ;
   %s = lshr i64 %x, 32
   %t = trunc i64 %s to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 1
   ret <4 x i16> %r
 }
 
 define <4 x i16> @lshr_same_length_poison_basevec_both_endian(i64 %x) {
-; ALL-LABEL: @lshr_same_length_poison_basevec_both_endian(
-; ALL-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 0
-; ALL-NEXT:    ret <4 x i16> [[R]]
+; BE-LABEL: @lshr_same_length_poison_basevec_both_endian(
+; BE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
+; BE-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i16
+; BE-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 0
+; BE-NEXT:    ret <4 x i16> [[R]]
+;
+; LE-LABEL: @lshr_same_length_poison_basevec_both_endian(
+; LE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
+; LE-NEXT:    [[VEC_S:%.*]] = bitcast i64 [[S]] to <4 x i16>
+; LE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_S]], <4 x i16> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; LE-NEXT:    ret <4 x i16> [[R]]
 ;
   %s = lshr i64 %x, 48
   %t = trunc i64 %s to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 0
   ret <4 x i16> %r
 }
 
 define <4 x i16> @lshr_wrong_index_same_length_poison_basevec(i64 %x) {
 ; ALL-LABEL: @lshr_wrong_index_same_length_poison_basevec(
 ; ALL-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
 ; ALL-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i16
 ; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i16> poison, i16 [[T]], i64 1
 ; ALL-NEXT:    ret <4 x i16> [[R]]
 ;
   %s = lshr i64 %x, 48
   %t = trunc i64 %s to i16
   %r = insertelement <4 x i16> poison, i16 %t, i64 1
   ret <4 x i16> %r
 }
 
 define <8 x i16> @lshr_longer_length_poison_basevec_le(i64 %x) {
-; ALL-LABEL: @lshr_longer_length_poison_basevec_le(
-; ALL-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i16
-; ALL-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 3
-; ALL-NEXT:    ret <8 x i16> [[R]]
+; BE-LABEL: @lshr_longer_length_poison_basevec_le(
+; BE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
+; BE-NEXT:    [[VEC_S:%.*]] = bitcast i64 [[S]] to <4 x i16>
+; BE-NEXT:    [[R:%.*]] = shufflevector <4 x i16> [[VEC_S]], <4 x i16> poison, <8 x i32> <i32 undef, i32 undef, i32 undef, i32 3, i32 undef, i32 undef, i32 undef, i32 undef>
+; BE-NEXT:    ret <8 x i16> [[R]]
+;
+; LE-LABEL: @lshr_longer_length_poison_basevec_le(
+; LE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 48
+; LE-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i16
+; LE-NEXT:    [[R:%.*]] = insertelement <8 x i16> poison, i16 [[T]], i64 3
+; LE-NEXT:    ret <8 x i16> [[R]]
 ;
   %s = lshr i64 %x, 48
   %t = trunc i64 %s to i16
   %r = insertelement <8 x i16> poison, i16 %t, i64 3
   ret <8 x i16> %r
 }
 
 define <8 x i16> @lshr_longer_length_poison_basevec_be(i64 %x) {

 ;
   %s = lshr i64 %x, 40
   %t = trunc i64 %s to i8
   %r = insertelement <4 x i8> poison, i8 %t, i64 3
   ret <4 x i8> %r
 }
 
 define <4 x i8> @lshr_wrong_shift_shorter_length_poison_basevec(i64 %x) {
-; ALL-LABEL: @lshr_wrong_shift_shorter_length_poison_basevec(
-; ALL-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 57
-; ALL-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i8
-; ALL-NEXT:    [[R:%.*]] = insertelement <4 x i8> poison, i8 [[T]], i64 0
-; ALL-NEXT:    ret <4 x i8> [[R]]
+; BE-LABEL: @lshr_wrong_shift_shorter_length_poison_basevec(
+; BE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 57
+; BE-NEXT:    [[T:%.*]] = trunc i64 [[S]] to i8
+; BE-NEXT:    [[R:%.*]] = insertelement <4 x i8> poison, i8 [[T]], i64 0
+; BE-NEXT:    ret <4 x i8> [[R]]
+;
+; LE-LABEL: @lshr_wrong_shift_shorter_length_poison_basevec(
+; LE-NEXT:    [[S:%.*]] = lshr i64 [[X:%.*]], 57
+; LE-NEXT:    [[VEC_S:%.*]] = bitcast i64 [[S]] to <8 x i8>
+; LE-NEXT:    [[R:%.*]] = shufflevector <8 x i8> [[VEC_S]], <8 x i8> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; LE-NEXT:    ret <4 x i8> [[R]]
 ;
   %s = lshr i64 %x, 57
   %t = trunc i64 %s to i8
   %r = insertelement <4 x i8> poison, i8 %t, i64 0
   ret <4 x i8> %r
 }
 
 define <4 x i8> @lshr_shorter_length_poison_basevec_be_extra_use(i64 %x) {

llvm/test/Transforms/InstCombine/vec_phi_extract-inseltpoison.ll

 
 ret:
   ret void
 }
 
 define void @nocopy(i64 %val, i32  %limit, ptr %ptr) {
 ; CHECK-LABEL: @nocopy(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = trunc i64 [[VAL:%.*]] to i32
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <16 x i32> undef, i32 [[TMP0]], i64 0
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <16 x i32> [[TMP1]], <16 x i32> poison, <16 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = add <16 x i32> [[TMP2]], <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[VEC_VAL:%.*]] = bitcast i64 [[VAL:%.*]] to <2 x i32>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <2 x i32> [[VEC_VAL]], <2 x i32> poison, <16 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP1:%.*]] = add <16 x i32> [[TMP0]], <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[TMP4:%.*]] = phi <16 x i32> [ [[TMP3]], [[ENTRY:%.*]] ], [ [[INC:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[ELT:%.*]] = extractelement <16 x i32> [[TMP4]], i64 0
-; CHECK-NEXT:    [[ELTCOPY:%.*]] = extractelement <16 x i32> [[TMP4]], i64 1
+; CHECK-NEXT:    [[TMP2:%.*]] = phi <16 x i32> [ [[TMP1]], [[ENTRY:%.*]] ], [ [[INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[ELT:%.*]] = extractelement <16 x i32> [[TMP2]], i64 0
+; CHECK-NEXT:    [[ELTCOPY:%.*]] = extractelement <16 x i32> [[TMP2]], i64 1
 ; CHECK-NEXT:    [[END:%.*]] = icmp ult i32 [[ELT]], [[LIMIT:%.*]]
-; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[ELTCOPY]], 10
-; CHECK-NEXT:    [[TMP6:%.*]] = sext i32 [[ELT]] to i64
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i32, ptr [[PTR:%.*]], i64 [[TMP6]]
-; CHECK-NEXT:    store i32 [[TMP5]], ptr [[TMP7]], align 4
-; CHECK-NEXT:    [[INC]] = add <16 x i32> [[TMP4]], <i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
+; CHECK-NEXT:    [[TMP3:%.*]] = add i32 [[ELTCOPY]], 10
+; CHECK-NEXT:    [[TMP4:%.*]] = sext i32 [[ELT]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i32, ptr [[PTR:%.*]], i64 [[TMP4]]
+; CHECK-NEXT:    store i32 [[TMP3]], ptr [[TMP5]], align 4
+; CHECK-NEXT:    [[INC]] = add <16 x i32> [[TMP2]], <i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
 ; CHECK-NEXT:    br i1 [[END]], label [[LOOP]], label [[RET:%.*]]
 ; CHECK:       ret:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %tempvector = insertelement <16 x i64> poison, i64 %val, i32 0
   %vector = shufflevector <16 x i64> %tempvector, <16 x i64> poison, <16 x i32> zeroinitializer
   %0 = add <16 x i64> %vector, <i64 0, i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8, i64 9, i64 10, i64 11, i64 12, i64 13, i64 14, i64 15>

llvm/test/Transforms/InstCombine/vec_phi_extract.ll

 
 ret:
   ret void
 }
 
 define void @nocopy(i64 %val, i32  %limit, ptr %ptr) {
 ; CHECK-LABEL: @nocopy(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = trunc i64 [[VAL:%.*]] to i32
-; CHECK-NEXT:    [[TMP1:%.*]] = insertelement <16 x i32> undef, i32 [[TMP0]], i64 0
-; CHECK-NEXT:    [[TMP2:%.*]] = shufflevector <16 x i32> [[TMP1]], <16 x i32> poison, <16 x i32> zeroinitializer
-; CHECK-NEXT:    [[TMP3:%.*]] = add <16 x i32> [[TMP2]], <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+; CHECK-NEXT:    [[VEC_VAL:%.*]] = bitcast i64 [[VAL:%.*]] to <2 x i32>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <2 x i32> [[VEC_VAL]], <2 x i32> poison, <16 x i32> zeroinitializer
+; CHECK-NEXT:    [[TMP1:%.*]] = add <16 x i32> [[TMP0]], <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
-; CHECK-NEXT:    [[TMP4:%.*]] = phi <16 x i32> [ [[TMP3]], [[ENTRY:%.*]] ], [ [[INC:%.*]], [[LOOP]] ]
-; CHECK-NEXT:    [[ELT:%.*]] = extractelement <16 x i32> [[TMP4]], i64 0
-; CHECK-NEXT:    [[ELTCOPY:%.*]] = extractelement <16 x i32> [[TMP4]], i64 1
+; CHECK-NEXT:    [[TMP2:%.*]] = phi <16 x i32> [ [[TMP1]], [[ENTRY:%.*]] ], [ [[INC:%.*]], [[LOOP]] ]
+; CHECK-NEXT:    [[ELT:%.*]] = extractelement <16 x i32> [[TMP2]], i64 0
+; CHECK-NEXT:    [[ELTCOPY:%.*]] = extractelement <16 x i32> [[TMP2]], i64 1
 ; CHECK-NEXT:    [[END:%.*]] = icmp ult i32 [[ELT]], [[LIMIT:%.*]]
-; CHECK-NEXT:    [[TMP5:%.*]] = add i32 [[ELTCOPY]], 10
-; CHECK-NEXT:    [[TMP6:%.*]] = sext i32 [[ELT]] to i64
-; CHECK-NEXT:    [[TMP7:%.*]] = getelementptr i32, ptr [[PTR:%.*]], i64 [[TMP6]]
-; CHECK-NEXT:    store i32 [[TMP5]], ptr [[TMP7]], align 4
-; CHECK-NEXT:    [[INC]] = add <16 x i32> [[TMP4]], <i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
+; CHECK-NEXT:    [[TMP3:%.*]] = add i32 [[ELTCOPY]], 10
+; CHECK-NEXT:    [[TMP4:%.*]] = sext i32 [[ELT]] to i64
+; CHECK-NEXT:    [[TMP5:%.*]] = getelementptr i32, ptr [[PTR:%.*]], i64 [[TMP4]]
+; CHECK-NEXT:    store i32 [[TMP3]], ptr [[TMP5]], align 4
+; CHECK-NEXT:    [[INC]] = add <16 x i32> [[TMP2]], <i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
 ; CHECK-NEXT:    br i1 [[END]], label [[LOOP]], label [[RET:%.*]]
 ; CHECK:       ret:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %tempvector = insertelement <16 x i64> undef, i64 %val, i32 0
   %vector = shufflevector <16 x i64> %tempvector, <16 x i64> undef, <16 x i32> zeroinitializer
   %0 = add <16 x i64> %vector, <i64 0, i64 1, i64 2, i64 3, i64 4, i64 5, i64 6, i64 7, i64 8, i64 9, i64 10, i64 11, i64 12, i64 13, i64 14, i64 15>

llvm/test/Transforms/LoopVectorize/ARM/pointer_iv.ll

for.body:
   br i1 %exitcond, label %end, label %for.body
 end:
   ret void
 }
 
 define hidden void @pointer_phi_v8i16_add1(i16* noalias nocapture readonly %A, i16* noalias nocapture %B, i32 %y) {
 ; CHECK-LABEL: @pointer_phi_v8i16_add1(
 ; CHECK-NEXT:  entry:
-; CHECK-NEXT:    [[TMP0:%.*]] = trunc i32 [[Y:%.*]] to i16
-; CHECK-NEXT:    [[BROADCAST_SPLATINSERT:%.*]] = insertelement <8 x i16> poison, i16 [[TMP0]], i64 0
-; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <8 x i16> [[BROADCAST_SPLATINSERT]], <8 x i16> poison, <8 x i32> zeroinitializer
+; CHECK-NEXT:    [[VEC_Y:%.*]] = bitcast i32 [[Y:%.*]] to <2 x i16>
+; CHECK-NEXT:    [[BROADCAST_SPLAT:%.*]] = shufflevector <2 x i16> [[VEC_Y]], <2 x i16> poison, <8 x i32> zeroinitializer
 ; CHECK-NEXT:    br label [[VECTOR_BODY:%.*]]
 ; CHECK:       vector.body:
 ; CHECK-NEXT:    [[INDEX:%.*]] = phi i32 [ 0, [[ENTRY:%.*]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
 ; CHECK-NEXT:    [[NEXT_GEP:%.*]] = getelementptr i16, i16* [[A:%.*]], i32 [[INDEX]]
 ; CHECK-NEXT:    [[NEXT_GEP4:%.*]] = getelementptr i16, i16* [[B:%.*]], i32 [[INDEX]]
-; CHECK-NEXT:    [[TMP1:%.*]] = bitcast i16* [[NEXT_GEP]] to <8 x i16>*
-; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <8 x i16>, <8 x i16>* [[TMP1]], align 2
-; CHECK-NEXT:    [[TMP2:%.*]] = add <8 x i16> [[WIDE_LOAD]], [[BROADCAST_SPLAT]]
-; CHECK-NEXT:    [[TMP3:%.*]] = bitcast i16* [[NEXT_GEP4]] to <8 x i16>*
-; CHECK-NEXT:    store <8 x i16> [[TMP2]], <8 x i16>* [[TMP3]], align 2
+; CHECK-NEXT:    [[TMP0:%.*]] = bitcast i16* [[NEXT_GEP]] to <8 x i16>*
+; CHECK-NEXT:    [[WIDE_LOAD:%.*]] = load <8 x i16>, <8 x i16>* [[TMP0]], align 2
+; CHECK-NEXT:    [[TMP1:%.*]] = add <8 x i16> [[WIDE_LOAD]], [[BROADCAST_SPLAT]]
+; CHECK-NEXT:    [[TMP2:%.*]] = bitcast i16* [[NEXT_GEP4]] to <8 x i16>*
+; CHECK-NEXT:    store <8 x i16> [[TMP1]], <8 x i16>* [[TMP2]], align 2
 ; CHECK-NEXT:    [[INDEX_NEXT]] = add nuw i32 [[INDEX]], 8
-; CHECK-NEXT:    [[TMP4:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1000
-; CHECK-NEXT:    br i1 [[TMP4]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
+; CHECK-NEXT:    [[TMP3:%.*]] = icmp eq i32 [[INDEX_NEXT]], 1000
+; CHECK-NEXT:    br i1 [[TMP3]], label [[END:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
 ; CHECK:       end:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   %0 = trunc i32 %y to i16
   br label %for.body
 for.body:                                         ; preds = %for.body, %for.body.lr.ph
   %A.addr.011 = phi i16* [ %A, %entry ], [ %add.ptr, %for.body ]

llvm/test/Transforms/PhaseOrdering/X86/vec-load-combine.ll

 }
 
 define noundef <4 x float> @ConvertVectors_ByVal(ptr noundef nonnull align 16 dereferenceable(16) %V) #0 {
 ; SSE-LABEL: @ConvertVectors_ByVal(
 ; SSE-NEXT:  entry:
 ; SSE-NEXT:    [[V_VAL20:%.*]] = load i64, ptr [[V:%.*]], align 16
 ; SSE-NEXT:    [[TMP0:%.*]] = getelementptr i8, ptr [[V]], i64 8
 ; SSE-NEXT:    [[V_VAL421:%.*]] = load i64, ptr [[TMP0]], align 8
-; SSE-NEXT:    [[TMP1:%.*]] = lshr i64 [[V_VAL20]], 32
-; SSE-NEXT:    [[TMP2:%.*]] = insertelement <2 x i64> poison, i64 [[V_VAL20]], i64 0
-; SSE-NEXT:    [[TMP3:%.*]] = insertelement <2 x i64> [[TMP2]], i64 [[TMP1]], i64 1
-; SSE-NEXT:    [[TMP4:%.*]] = trunc <2 x i64> [[TMP3]] to <2 x i32>
-; SSE-NEXT:    [[TMP5:%.*]] = shufflevector <2 x i32> [[TMP4]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 undef, i32 undef>
-; SSE-NEXT:    [[TMP6:%.*]] = trunc i64 [[V_VAL421]] to i32
-; SSE-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[TMP6]], i64 2
-; SSE-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32> [[TMP7]], i32 [[TMP6]], i64 3
-; SSE-NEXT:    [[VECINIT16:%.*]] = bitcast <4 x i32> [[TMP8]] to <4 x float>
+; SSE-NEXT:    [[VEC_V_VAL20:%.*]] = bitcast i64 [[V_VAL20]] to <2 x i32>
+; SSE-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i32> [[VEC_V_VAL20]], <2 x i32> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; SSE-NEXT:    [[TMP2:%.*]] = lshr i64 [[V_VAL20]], 32
+; SSE-NEXT:    [[TMP3:%.*]] = trunc i64 [[TMP2]] to i32
+; SSE-NEXT:    [[TMP4:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[TMP3]], i64 1
+; SSE-NEXT:    [[TMP5:%.*]] = trunc i64 [[V_VAL421]] to i32
+; SSE-NEXT:    [[TMP6:%.*]] = insertelement <4 x i32> [[TMP4]], i32 [[TMP5]], i64 2
+; SSE-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[TMP5]], i64 3
+; SSE-NEXT:    [[VECINIT16:%.*]] = bitcast <4 x i32> [[TMP7]] to <4 x float>
 ; SSE-NEXT:    ret <4 x float> [[VECINIT16]]
 ;
 ; AVX-LABEL: @ConvertVectors_ByVal(
 ; AVX-NEXT:  entry:
 ; AVX-NEXT:    [[V_VAL20:%.*]] = load i64, ptr [[V:%.*]], align 16
 ; AVX-NEXT:    [[TMP0:%.*]] = getelementptr i8, ptr [[V]], i64 8
 ; AVX-NEXT:    [[V_VAL421:%.*]] = load i64, ptr [[TMP0]], align 8
-; AVX-NEXT:    [[TMP1:%.*]] = trunc i64 [[V_VAL20]] to i32
-; AVX-NEXT:    [[TMP2:%.*]] = insertelement <4 x i32> undef, i32 [[TMP1]], i64 0
-; AVX-NEXT:    [[TMP3:%.*]] = lshr i64 [[V_VAL20]], 32
-; AVX-NEXT:    [[TMP4:%.*]] = trunc i64 [[TMP3]] to i32
-; AVX-NEXT:    [[TMP5:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[TMP4]], i64 1
-; AVX-NEXT:    [[TMP6:%.*]] = trunc i64 [[V_VAL421]] to i32
-; AVX-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[TMP6]], i64 2
-; AVX-NEXT:    [[TMP8:%.*]] = insertelement <4 x i32> [[TMP7]], i32 [[TMP6]], i64 3
-; AVX-NEXT:    [[VECINIT16:%.*]] = bitcast <4 x i32> [[TMP8]] to <4 x float>
+; AVX-NEXT:    [[VEC_V_VAL20:%.*]] = bitcast i64 [[V_VAL20]] to <2 x i32>
+; AVX-NEXT:    [[TMP1:%.*]] = shufflevector <2 x i32> [[VEC_V_VAL20]], <2 x i32> poison, <4 x i32> <i32 0, i32 undef, i32 undef, i32 undef>
+; AVX-NEXT:    [[TMP2:%.*]] = lshr i64 [[V_VAL20]], 32
+; AVX-NEXT:    [[TMP3:%.*]] = trunc i64 [[TMP2]] to i32
+; AVX-NEXT:    [[TMP4:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[TMP3]], i64 1
+; AVX-NEXT:    [[TMP5:%.*]] = trunc i64 [[V_VAL421]] to i32
+; AVX-NEXT:    [[TMP6:%.*]] = insertelement <4 x i32> [[TMP4]], i32 [[TMP5]], i64 2
+; AVX-NEXT:    [[TMP7:%.*]] = insertelement <4 x i32> [[TMP6]], i32 [[TMP5]], i64 3
+; AVX-NEXT:    [[VECINIT16:%.*]] = bitcast <4 x i32> [[TMP7]] to <4 x float>
 ; AVX-NEXT:    ret <4 x float> [[VECINIT16]]
 ;
 entry:
   %V.addr = alloca ptr, align 8
   %.compoundliteral = alloca <4 x float>, align 16
   %ref.tmp = alloca %union.ElementWiseAccess, align 16
   %ref.tmp2 = alloca %union.ElementWiseAccess, align 16
   %ref.tmp7 = alloca %union.ElementWiseAccess, align 16