[DAGCombiner] reduce buildvec of zexted extracted element to shuffle

[spatel]

Summary

The motivating case for this is shown in the first regression test. We are transferring to scalar and back rather than just zero-extending with 'vpmovzxdq'.

That's a special-case for a more general pattern as shown here. In all tests, we're avoiding the vector-scalar-vector moves in favor of vector ops.

I suspect that we aren't producing optimal shuffle code in some cases though, so we may want to limit this patch and/or account for those patterns first. But I figured it was worth posting the larger test diffs, so we can see what's happening and make sure the logic is correct.

If we want to limit this patch but still get that 1st motivating case, I see 2 possibilities:

1. Don't handle patterns where we require translating the source element to a different location in the result.
2. Don't handle patterns with zero elements in the build vector (only deal with undefs there).

[spatel]

Patch updated:
I reduced the reach of this patch to only handle a build vector of undefs + 1-non-undef. If x86 isn't lowering the other cases optimally, then other targets may not be either even if there's no regression test evidence of that.

So now, these should be mostly clear wins. Not sure about some of the SSE2-only shuffling.

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16222

Should ANY_EXTEND be handled as well? SimplifyDemandedBits can reduce ZERO_EXTEND -> ANY_EXTEND more aggressively these days.

test/CodeGen/X86/buildvec-extract.ll

424

Please can you investigate what's happening here? The xmm0[6,7] at the end seems really weird....

[RKSimon]

I also wonder if we have much crossover with PR39975 (truncate(extract()) -> extract(bitcast())) to handle TRUNCATE as well as *_EXTEND

[spatel]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16222

That seems ok, and I can update with a draft of that change. But any idea how to make a test that would provide coverage for that pattern? I'm not showing any existing test diffs. Alternatively, I can add a TODO enhancement comment while trying to find a way to make that happen.

test/CodeGen/X86/buildvec-extract.ll

424

Hmm - I didn't notice that before.

There's some shuffle lowering madness that I haven't stepped through yet that creates several nodes before this becomes the single pshufb.

I filed PR40306 to track this:
https://bugs.llvm.org/show_bug.cgi?id=40306

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16222

TODO is fine

[spatel]

Patch updated:
Added TODO comment for handling ISD::ANY_EXTEND.

[spatel]

Patch updated:
No code changes, but rebased after rL351103 so we get more vector zext'ing codegen.

[RKSimon]

LGTM (with one pcg bug request).

test/CodeGen/X86/buildvec-extract.ll

412

Please can you raise a bug on this - we should do better for this shuffle.

458

Add this shuffle to the same bug - I think its the same culprit.

[spatel]

test/CodeGen/X86/buildvec-extract.ll

412

https://bugs.llvm.org/show_bug.cgi?id=40318

[SjoerdMeijer]

Hello. I am investigating a crash and this assertion failure on AArch64:

lib/CodeGen/SelectionDAG/SelectionDAG.cpp:1548: llvm::SDValue llvm::SelectionDAG::getVectorShuffle(llvm::EVT, const llvm::SDLoc&, llvm::SDValue, llvm::SDValue, llvm::ArrayRef<int>): Assertion `VT.getVectorNumElements() == Mask.size() && "Must have the same number of vector elements as mask elements!"' failed.

Bisecting is blaming this commit. I've only just started debugging this (and it is hitting this new function), but just wanted to share this in case you have some ideas and thoughts on this. This is my reproducer:

target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
target triple = "aarch64-unknown-linux-gnu"

@a = common dso_local local_unnamed_addr global <4 x i16> zeroinitializer, align 8
@b = common dso_local local_unnamed_addr global i32 0, align 4

; Function Attrs: nounwind
define dso_local i32 @vqmovns_s32() local_unnamed_addr {
entry:
  %0 = load <4 x i16>, <4 x i16>* @a, align 8
  %1 = extractelement <4 x i16> %0, i32 2
  %vgetq_lane = zext i16 %1 to i32
  %2 = insertelement <4 x i32> undef, i32 %vgetq_lane, i64 0
  %vqmovns_s32 = tail call <4 x i16> @llvm.aarch64.neon.sqxtn.v4i16(<4 x i32> %2)
  %3 = extractelement <4 x i16> %vqmovns_s32, i64 0
  %conv = sext i16 %3 to i32
  store i32 %conv, i32* @b, align 4
  ret i32 undef
}

; Function Attrs: nounwind readnone
declare <4 x i16> @llvm.aarch64.neon.sqxtn.v4i16(<4 x i32>)

[spatel]

In D56281#1365397, @SjoerdMeijer wrote:

Hello. I am investigating a crash and this assertion failure on AArch64:

lib/CodeGen/SelectionDAG/SelectionDAG.cpp:1548: llvm::SDValue llvm::SelectionDAG::getVectorShuffle(llvm::EVT, const llvm::SDLoc&, llvm::SDValue, llvm::SDValue, llvm::ArrayRef<int>): Assertion `VT.getVectorNumElements() == Mask.size() && "Must have the same number of vector elements as mask elements!"' failed.

I see the problem now: I forgot to verify that the build vector and the source vector of the extract element are the same size. Should have a fix committed soon.

[spatel]

In D56281#1365476, @spatel wrote:

I see the problem now: I forgot to verify that the build vector and the source vector of the extract element are the same size. Should have a fix committed soon.

Please see:
rL351753
rL351754

[spatel]

Filed a blocking bug for the 8.0 release:
https://bugs.llvm.org/show_bug.cgi?id=40394

[SjoerdMeijer]

Many thanks for your speedy responses and for fixing this!

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::createBuildVecShuffle(const SDLoc &DL, SDNode *N,
 
   SDValue Shuffle = DAG.getVectorShuffle(InVT1, DL, VecIn1, VecIn2, Mask);
   if (ShuffleNumElems > NumElems)
     Shuffle = DAG.getNode(ISD::EXTRACT_SUBVECTOR, DL, VT, Shuffle, ZeroIdx);
 
   return Shuffle;
 }
 
+static SDValue reduceBuildVecToShuffleWithZero(SDNode *BV, SelectionDAG &DAG) {
+  assert(BV->getOpcode() == ISD::BUILD_VECTOR && "Expected build vector");
+
+  // First, determine where the build vector is not undef.
+  // TODO: We could extend this to handle zero elements as well as undefs.
+  int NumBVOps = BV->getNumOperands();
+  int ZextElt = -1;
+  for (int i = 0; i != NumBVOps; ++i) {
+    SDValue Op = BV->getOperand(i);
+    if (Op.isUndef())
+      continue;
+    if (ZextElt == -1)
+      ZextElt = i;
+    else
+      return SDValue();
+  }
+  // Bail out if there's no non-undef element.
+  if (ZextElt == -1)
+    return SDValue();
+
+  // The build vector contains some number of undef elements and exactly
+  // one other element. That other element must be a zero-extended scalar
+  // extracted from a vector at a constant index to turn this into a shuffle.
+  // TODO: This could be enhanced to allow ANY_EXTEND as well as ZERO_EXTEND.
+  SDValue Zext = BV->getOperand(ZextElt);
+  if (Zext.getOpcode() != ISD::ZERO_EXTEND || !Zext.hasOneUse() ||
+      Zext.getOperand(0).getOpcode() != ISD::EXTRACT_VECTOR_ELT ||
+      !isa<ConstantSDNode>(Zext.getOperand(0).getOperand(1)))
+    return SDValue();
+
+  // The zero-extend must be a multiple of the source size.
+  SDValue Extract = Zext.getOperand(0);
+  unsigned DestSize = Zext.getValueSizeInBits();
+  unsigned SrcSize = Extract.getValueSizeInBits();
+  if (DestSize % SrcSize != 0)
+    return SDValue();
+
+  // Create a shuffle mask that will combine the extracted element with zeros
+  // and undefs.
+  int ZextRatio =  DestSize / SrcSize;
+  int NumMaskElts = NumBVOps * ZextRatio;
+  SmallVector<int, 32> ShufMask(NumMaskElts, -1);
+  for (int i = 0; i != NumMaskElts; ++i) {
+    if (i / ZextRatio == ZextElt) {
+      // The low bits of the (potentially translated) extracted element map to
+      // the source vector. The high bits map to zero. We will use a zero vector
+      // as the 2nd source operand of the shuffle, so use the 1st element of
+      // that vector (mask value is number-of-elements) for the high bits.
+      if (i % ZextRatio == 0)
+        ShufMask[i] = Extract.getConstantOperandVal(1);
+      else
+        ShufMask[i] = NumMaskElts;
+    }
+
+    // Undef elements of the build vector remain undef because we initialize
+    // the shuffle mask with -1.
+  }
+
+  // Turn this into a shuffle with zero if that's legal.
+  EVT VecVT = Extract.getOperand(0).getValueType();
+  if (!DAG.getTargetLoweringInfo().isShuffleMaskLegal(ShufMask, VecVT))
+    return SDValue();
+
+  // buildvec undef, ..., (zext (extractelt V, IndexC)), undef... -->
+  // bitcast (shuffle V, ZeroVec, VectorMask)
+  SDLoc DL(BV);
+  SDValue ZeroVec = DAG.getConstant(0, DL, VecVT);
+  SDValue Shuf = DAG.getVectorShuffle(VecVT, DL, Extract.getOperand(0), ZeroVec,
+                                      ShufMask);
+  return DAG.getBitcast(BV->getValueType(0), Shuf);
+}
+
 // Check to see if this is a BUILD_VECTOR of a bunch of EXTRACT_VECTOR_ELT
 // operations. If the types of the vectors we're extracting from allow it,
 // turn this into a vector_shuffle node.
 SDValue DAGCombiner::reduceBuildVecToShuffle(SDNode *N) {
   SDLoc DL(N);
   EVT VT = N->getValueType(0);
 
   // Only type-legal BUILD_VECTOR nodes are converted to shuffle nodes.
   if (!isTypeLegal(VT))
     return SDValue();
 
+  if (SDValue V = reduceBuildVecToShuffleWithZero(N, DAG))
+    return V;
+
   // May only combine to shuffle after legalize if shuffle is legal.
   if (LegalOperations && !TLI.isOperationLegal(ISD::VECTOR_SHUFFLE, VT))
     return SDValue();
 
   bool UsesZeroVector = false;
   unsigned NumElems = N->getNumOperands();
 
   // Record, for each element of the newly built vector, which input vector

test/CodeGen/X86/buildvec-extract.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+sse2   | FileCheck %s --check-prefixes=ANY,SSE,SSE2
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+sse4.1 | FileCheck %s --check-prefixes=ANY,SSE,SSE41
 ; RUN: llc < %s -mtriple=x86_64-- -mattr=+avx    | FileCheck %s --check-prefixes=ANY,AVX
 
 define <2 x i64> @extract0_i32_zext_insert0_i64_undef(<4 x i32> %x) {
-; SSE-LABEL: extract0_i32_zext_insert0_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    movd %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract0_i32_zext_insert0_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    xorps %xmm1, %xmm1
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract0_i32_zext_insert0_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pmovzxdq {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract0_i32_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vmovd %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpmovzxdq {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 0
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i32_zext_insert0_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 0
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i32_zext_insert0_i64_undef(<4 x i32> %x) {
-; SSE2-LABEL: extract1_i32_zext_insert0_i64_undef:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE41-LABEL: extract1_i32_zext_insert0_i64_undef:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $1, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    retq
+; SSE-LABEL: extract1_i32_zext_insert0_i64_undef:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psrlq $32, %xmm0
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: extract1_i32_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $1, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpsrlq $32, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 1
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i32_zext_insert0_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 1
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i32_zext_insert0_i64_undef(<4 x i32> %x) {
-; SSE2-LABEL: extract2_i32_zext_insert0_i64_undef:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE41-LABEL: extract2_i32_zext_insert0_i64_undef:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $2, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    retq
+; SSE-LABEL: extract2_i32_zext_insert0_i64_undef:
+; SSE:       # %bb.0:
+; SSE-NEXT:    xorps %xmm1, %xmm1
+; SSE-NEXT:    unpckhps {{.*#+}} xmm0 = xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: extract2_i32_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $2, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vunpckhps {{.*#+}} xmm0 = xmm0[2],xmm1[2],xmm0[3],xmm1[3]
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 2
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i32_zext_insert0_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 2
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i32_zext_insert0_i64_undef(<4 x i32> %x) {
-; SSE2-LABEL: extract3_i32_zext_insert0_i64_undef:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[3,1,2,3]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSE41-LABEL: extract3_i32_zext_insert0_i64_undef:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $3, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    retq
+; SSE-LABEL: extract3_i32_zext_insert0_i64_undef:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psrldq {{.*#+}} xmm0 = xmm0[12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: extract3_i32_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $3, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpsrldq {{.*#+}} xmm0 = xmm0[12,13,14,15],zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero,zero
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 3
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i32_zext_insert0_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 3
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i32_zext_insert1_i64_undef(<4 x i32> %x) {
-; SSE-LABEL: extract0_i32_zext_insert1_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    movd %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract0_i32_zext_insert1_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,0,1,1]
+; SSE2-NEXT:    pxor %xmm1, %xmm1
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract0_i32_zext_insert1_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[0,1,0,1]
+; SSE41-NEXT:    pxor %xmm0, %xmm0
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm1[0,1,2,3,4,5],xmm0[6,7]
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract0_i32_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vmovd %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpermilps {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vblendps {{.*#+}} xmm0 = xmm0[0,1,2],xmm1[3]
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 0
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i32_zext_insert1_i64_zero(<4 x i32> %x) {
; AVX-NEXT:    retq
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i32_zext_insert1_i64_undef(<4 x i32> %x) {
 ; SSE2-LABEL: extract1_i32_zext_insert1_i64_undef:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE2-NEXT:    xorps %xmm1, %xmm1
+; SSE2-NEXT:    unpcklps {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: extract1_i32_zext_insert1_i64_undef:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $1, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE41-NEXT:    pmovzxdq {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract1_i32_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $1, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpmovzxdq {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 1
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i32_zext_insert1_i64_zero(<4 x i32> %x) {
; AVX-NEXT:    retq
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i32_zext_insert1_i64_undef(<4 x i32> %x) {
 ; SSE2-LABEL: extract2_i32_zext_insert1_i64_undef:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE2-NEXT:    andps {{.*}}(%rip), %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSE41-LABEL: extract2_i32_zext_insert1_i64_undef:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $2, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE41-NEXT:    xorps %xmm1, %xmm1
+; SSE41-NEXT:    blendps {{.*#+}} xmm0 = xmm0[0,1,2],xmm1[3]
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract2_i32_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $2, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vxorps %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vblendps {{.*#+}} xmm0 = xmm0[0,1,2],xmm1[3]
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 2
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i32_zext_insert1_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 2
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i32_zext_insert1_i64_undef(<4 x i32> %x) {
-; SSE2-LABEL: extract3_i32_zext_insert1_i64_undef:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[3,1,2,3]
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    movq %rax, %xmm0
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE2-NEXT:    retq
-;
-; SSE41-LABEL: extract3_i32_zext_insert1_i64_undef:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    extractps $3, %xmm0, %eax
-; SSE41-NEXT:    movq %rax, %xmm0
-; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE41-NEXT:    retq
+; SSE-LABEL: extract3_i32_zext_insert1_i64_undef:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psrlq $32, %xmm0
+; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: extract3_i32_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vextractps $3, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpsrlq $32, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 3
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i32_zext_insert1_i64_zero(<4 x i32> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <4 x i32> %x, i32 3
   %z = zext i32 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i16_zext_insert0_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract0_i16_zext_insert0_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $0, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract0_i16_zext_insert0_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pxor %xmm1, %xmm1
+; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract0_i16_zext_insert0_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract0_i16_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $0, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 0
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i16_zext_insert0_i64_zero(<8 x i16> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 0
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i16_zext_insert0_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract1_i16_zext_insert0_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $1, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract1_i16_zext_insert0_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pxor %xmm1, %xmm1
+; SSE2-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
+; SSE2-NEXT:    pshufhw {{.*#+}} xmm0 = xmm1[0,1,2,3,4,6,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,1,2,3]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[3,2,0,1,4,5,6,7]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract1_i16_zext_insert0_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    psrld $16, %xmm0
+; SSE41-NEXT:    pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract1_i16_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $1, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpsrld $16, %xmm0, %xmm0
+; AVX-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 1
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i16_zext_insert0_i64_zero(<8 x i16> %x) {
 ; SSE-LABEL: extract1_i16_zext_insert0_i64_zero:

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 1
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i16_zext_insert0_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract2_i16_zext_insert0_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $2, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract2_i16_zext_insert0_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pxor %xmm1, %xmm1
+; SSE2-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm1[3,1,2,0]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[3,1,2,0,4,5,6,7]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract2_i16_zext_insert0_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE41-NEXT:    pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract2_i16_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $2, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; AVX-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 2
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i16_zext_insert0_i64_zero(<8 x i16> %x) {
; AVX-NEXT:    retq
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i16_zext_insert0_i64_undef(<8 x i16> %x) {
 ; SSE-LABEL: extract3_i16_zext_insert0_i64_undef:
 ; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $3, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
+; SSE-NEXT:    psrlq $48, %xmm0
 ; SSE-NEXT:    retq
 ;
 ; AVX-LABEL: extract3_i16_zext_insert0_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $3, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
+; AVX-NEXT:    vpsrlq $48, %xmm0, %xmm0
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 3
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i16_zext_insert0_i64_zero(<8 x i16> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 3
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 0
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i16_zext_insert1_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract0_i16_zext_insert1_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $0, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract0_i16_zext_insert1_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract0_i16_zext_insert1_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pshufd {{.*#+}} xmm1 = xmm0[0,1,0,1]
+; SSE41-NEXT:    pxor %xmm0, %xmm0
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0,1,2,3],xmm1[4],xmm0[5,6,7]
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract0_i16_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $0, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
 ; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm1[0,1,2,3],xmm0[4],xmm1[5,6,7]
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 0
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract0_i16_zext_insert1_i64_zero(<8 x i16> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 0
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i16_zext_insert1_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract1_i16_zext_insert1_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $1, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract1_i16_zext_insert1_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pxor %xmm1, %xmm1
+; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1]
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract1_i16_zext_insert1_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract1_i16_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $1, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 1
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract1_i16_zext_insert1_i64_zero(<8 x i16> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 1
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i16_zext_insert1_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract2_i16_zext_insert1_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $2, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract2_i16_zext_insert1_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,1,3]
+; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract2_i16_zext_insert1_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pmovzxdq {{.*#+}} xmm1 = xmm0[0],zero,xmm0[1],zero
+; SSE41-NEXT:    pxor %xmm0, %xmm0
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0,1,2,3],xmm1[4],xmm0[5,6,7]
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract2_i16_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $2, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpmovzxdq {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm1[0,1,2,3],xmm0[4],xmm1[5,6,7]
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 2
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract2_i16_zext_insert1_i64_zero(<8 x i16> %x) {

 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 2
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> zeroinitializer, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i16_zext_insert1_i64_undef(<8 x i16> %x) {
-; SSE-LABEL: extract3_i16_zext_insert1_i64_undef:
-; SSE:       # %bb.0:
-; SSE-NEXT:    pextrw $3, %xmm0, %eax
-; SSE-NEXT:    movq %rax, %xmm0
-; SSE-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
-; SSE-NEXT:    retq
+; SSE2-LABEL: extract3_i16_zext_insert1_i64_undef:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pslldq {{.*#+}} xmm0 = zero,zero,xmm0[0,1,2,3,4,5,6,7,8,9,10,11,12,13]
+; SSE2-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE2-NEXT:    retq
+;
+; SSE41-LABEL: extract3_i16_zext_insert1_i64_undef:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pslldq {{.*#+}} xmm0 = zero,zero,xmm0[0,1,2,3,4,5,6,7,8,9,10,11,12,13]
+; SSE41-NEXT:    pxor %xmm1, %xmm1
+; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm1[0,1,2,3],xmm0[4],xmm1[5,6,7]
+; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: extract3_i16_zext_insert1_i64_undef:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $3, %xmm0, %eax
-; AVX-NEXT:    vmovq %rax, %xmm0
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[0,1,0,1]
+; AVX-NEXT:    vpslldq {{.*#+}} xmm0 = zero,zero,xmm0[0,1,2,3,4,5,6,7,8,9,10,11,12,13]
+; AVX-NEXT:    vpxor %xmm1, %xmm1, %xmm1
+; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm1[0,1,2,3],xmm0[4],xmm1[5,6,7]
 ; AVX-NEXT:    retq
   %e = extractelement <8 x i16> %x, i32 3
   %z = zext i16 %e to i64
   %r = insertelement <2 x i64> undef, i64 %z, i32 1
   ret <2 x i64> %r
 }
 
 define <2 x i64> @extract3_i16_zext_insert1_i64_zero(<8 x i16> %x) {