[DAGCombiner] (insert_vector_elt (vector_shuffle X, Y), (extract_vector_elt X, N), IdxC) -> (vector_shuffle X, Y)

[deadalnix]

Summary

This is beneficial when the shuffle is only used once and end up being generated in a few places when some node is combined into a shuffle.

[lebedev.ri]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16432–16433 ↗

(On Diff #217050)

Do you check that both vector_shuffle and extract_vector_elt have the same first argument?

16434–16436 ↗

(On Diff #217050)

Here we have two instructions, and only require that the first one is single-use.
Observation: we only create a single new instruction.
Do we want to either require both of them be single-use, the first one, either one of them, or none of them?

Another observation - you end up keeping (using) the instruction you just checked for extra uses.
Did you mean to check the other one instead?

16448–16450 ↗

(On Diff #217050)

So NewMask is simply inited as a copy of SVN->getMask()?
Would something like SmallVector<int, 16> NewMask(SVN->getMask()); just work?

16452 ↗

(On Diff #217050)

You either need to check that the cast succeeded, or drop dyn_ prefix

16453 ↗

(On Diff #217050)

This could use a comment

test/CodeGen/X86/madd.ll

1901–1903 ↗

(On Diff #217050)

If i'm reading this correctly this does showcase the need to handle this in backend - it can be introduced there?

[deadalnix]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16432–16433 ↗

(On Diff #217050)

This is checked bellow. It is because we can match either the first or the second argument, as vector_shuffle does in fact shuffle two vectors together.

16434–16436 ↗

(On Diff #217050)

Even if InsertVal is reused, this transform reduces data dependencies, but indeed do not save anything on the instruction count from. Removing data dependencies is beneficial on its own.

16448–16450 ↗

(On Diff #217050)

I asked mys elf the same question. This doesn't seem to work.

16452 ↗

(On Diff #217050)

The cast must succeed, as we check for it in the if guarding this code, so I'll remove dyn_

16453 ↗

(On Diff #217050)

I'll add a few comments.

[craig.topper]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16448–16450 ↗

(On Diff #217050)

Yeah SmallVector doesn't take an ArrayRef, but you can do

SmallVector<int, 16> NewMask(Mask.begin(), Mask.end());

[lebedev.ri]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16434–16436 ↗

(On Diff #217050)

So, do we need that one-use restriction here then?

16448–16450 ↗

(On Diff #217050)

SmallVector<int, 16> NewMask(SVN->getMask().begin(), SVN->getMask().end()); then.

[deadalnix]

• Add comments
• dyn_cast -> cast

[deadalnix]

Build NewMask using iterators

[lebedev.ri]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16448–16450 ↗

(On Diff #217050)

Please mark comments as resolved as appropriate.

[deadalnix]

Comments amrked as done.

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16451 ↗

(On Diff #217064)

You already have this at line 16438:

ArrayRef<int> Mask = SVN->getMask()

16456 ↗

(On Diff #217064)

You need to check NewMask is legal: if (!TLI.isShuffleMaskLegal(NewMask, Vec.getValueType()))

[deadalnix]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16456 ↗

(On Diff #217064)

There are a few places where isShuffleMaskLegal is used. Some try to flip operands inc ase it isn't legal. Should that be an idiom?

[lebedev.ri]

Looks like the link was incorrect, this didn't auto-close?

[deadalnix]

In D66718#1645956, @lebedev.ri wrote:

Looks like the link was incorrect, this didn't auto-close?

Why should this auto close?

It wasn't landed yet. I was wondering if the creation of vector shuffles should be normalized. I think it may be worth doing.

[lebedev.ri]

In D66718#1646859, @deadalnix wrote:

In D66718#1645956, @lebedev.ri wrote:

Looks like the link was incorrect, this didn't auto-close?

Why should this auto close?

It wasn't landed yet. I was wondering if the creation of vector shuffles should be normalized. I think it may be worth doing.

Sorry, i got confused by "commits" tab

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16456 ↗

(On Diff #217064)

I've no objection to you creating a static helper that tries isShuffleMaskLegal with original and commuted masks, and call DAG.getVectorShuffle on success.

[deadalnix]

Rebase on top of D66804

[lebedev.ri]

This looks good to me but i find naming choice really confusing here.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16420 ↗

(On Diff #217409)

Hm, please rename this to ExtractVal instead.

16423 ↗

(On Diff #217409)

Comment is wrong i think, should be
s/IdxC/InsIndex/

16447 ↗

(On Diff #217409)

ExtrIndex

[lebedev.ri]

Actually, i have one concern.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16421 ↗

(On Diff #217409)

OldShuffle

16431–16432 ↗

(On Diff #217409)

Just x/y like in comments?

16437 ↗

(On Diff #217409)

Maybe XOffset?

16441 ↗

(On Diff #217409)

Actually, this is not correct.
While both shuffle inputs must have same types,
the shuffle mask can have different dimensions.
So this should be closer to Offset = X.getType().getVectorNumElements().

https://llvm.org/docs/LangRef.html#shufflevector-instruction

[RKSimon]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16421 ↗

(On Diff #217409)

There's no need to rename - tbh we don't know its a shuffle at this stage!

16441 ↗

(On Diff #217409)

That is only true for IR - in the DAG, the result must have the same type as the shuffle operands - so the number of mask elts must match the number of result/operand elts - see SelectionDAG::getVectorShuffle

16447 ↗

(On Diff #217409)

Please can you assert that IndexX's value (and InsIndex come to that) are both in bounds (0 <= idx < Mask.size())?

[deadalnix]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16421 ↗

(On Diff #217409)

That is true, and this is why the names are what they are. But I have to agree with @lebedev.ri here, the name are very confusing. I'll try to see if I can come up with better names.

[lebedev.ri]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16441 ↗

(On Diff #217409)

Oh, forgot about that, sorry. An assert later on will be fine then.

[deadalnix]

IndexC => ExtrIndex

[lebedev.ri]

I still find naming choices to be actively misleading.
LG otherwise.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

16435 ↗

(On Diff #217666)

There's no need to rename - tbh we don't know its a shuffle at this stage!

But we will only proceed *if* it will be a shuffle. This is a common pattern everywhere else in the code.

llvm/trunk/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::splitMergedValStore(StoreSDNode *ST) {
   SDValue St1 =
       DAG.getStore(St0, DL, Hi, Ptr,
                    ST->getPointerInfo().getWithOffset(HalfValBitSize / 8),
                    Alignment / 2, MMOFlags, AAInfo);
   return St1;
 }
 
 /// Convert a disguised subvector insertion into a shuffle:
-/// insert_vector_elt V, (bitcast X from vector type), IdxC -->
-/// bitcast(shuffle (bitcast V), (extended X), Mask)
-/// Note: We do not use an insert_subvector node because that requires a legal
-/// subvector type.
 SDValue DAGCombiner::combineInsertEltToShuffle(SDNode *N, unsigned InsIndex) {
   SDValue InsertVal = N->getOperand(1);
+  SDValue Vec = N->getOperand(0);
+
+  // (insert_vector_elt (vector_shuffle X, Y), (extract_vector_elt X, N), InsIndex)
+  //   --> (vector_shuffle X, Y)
+  if (Vec.getOpcode() == ISD::VECTOR_SHUFFLE && Vec.hasOneUse() &&
+      InsertVal.getOpcode() == ISD::EXTRACT_VECTOR_ELT &&
+      isa<ConstantSDNode>(InsertVal.getOperand(1))) {
+    ShuffleVectorSDNode *SVN = cast<ShuffleVectorSDNode>(Vec.getNode());
+    ArrayRef<int> Mask = SVN->getMask();
+
+    SDValue X = Vec.getOperand(0);
+    SDValue Y = Vec.getOperand(1);
+
+    // Vec's operand 0 is using indices from 0 to N-1 and
+    // operand 1 from N to 2N - 1, where N is the number of
+    // elements in the vectors.
+    int XOffset = -1;
+    if (InsertVal.getOperand(0) == X) {
+      XOffset = 0;
+    } else if (InsertVal.getOperand(0) == Y) {
+      XOffset = X.getValueType().getVectorNumElements();
+    }
+
+    if (XOffset != -1) {
+      SmallVector<int, 16> NewMask(Mask.begin(), Mask.end());
+
+      auto *ExtrIndex = cast<ConstantSDNode>(InsertVal.getOperand(1));
+      NewMask[InsIndex] = XOffset + ExtrIndex->getZExtValue();
+      assert(NewMask[InsIndex] < 2 * Vec.getValueType().getVectorNumElements() &&
+             NewMask[InsIndex] >= 0 && "NewMask[InsIndex] is out of bound");
+
+      SDValue LegalShuffle =
+              TLI.buildLegalVectorShuffle(Vec.getValueType(), SDLoc(N), X,
+                                          Y, NewMask, DAG);
+      if (LegalShuffle)
+        return LegalShuffle;
+    }
+  }
+
+  // insert_vector_elt V, (bitcast X from vector type), IdxC -->
+  // bitcast(shuffle (bitcast V), (extended X), Mask)
+  // Note: We do not use an insert_subvector node because that requires a
+  // legal subvector type.
   if (InsertVal.getOpcode() != ISD::BITCAST || !InsertVal.hasOneUse() ||
       !InsertVal.getOperand(0).getValueType().isVector())
     return SDValue();
 
   SDValue SubVec = InsertVal.getOperand(0);
   SDValue DestVec = N->getOperand(0);
   EVT SubVecVT = SubVec.getValueType();
   EVT VT = DestVec.getValueType();

llvm/trunk/test/CodeGen/X86/madd.ll

 ; AVX512-NEXT:    vpmulld %zmm1, %zmm0, %zmm0
 ; AVX512-NEXT:    vpextrd $2, %xmm0, %eax
 ; AVX512-NEXT:    vpinsrd $1, %eax, %xmm0, %xmm1
 ; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm2
 ; AVX512-NEXT:    vmovd %xmm2, %eax
 ; AVX512-NEXT:    vpinsrd $2, %eax, %xmm1, %xmm1
 ; AVX512-NEXT:    vpextrd $2, %xmm2, %eax
 ; AVX512-NEXT:    vpinsrd $3, %eax, %xmm1, %xmm1
-; AVX512-NEXT:    vpextrd $3, %xmm0, %eax
-; AVX512-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
-; AVX512-NEXT:    vpinsrd $1, %eax, %xmm0, %xmm0
+; AVX512-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,3,2,3]
 ; AVX512-NEXT:    vpextrd $1, %xmm2, %eax
 ; AVX512-NEXT:    vpinsrd $2, %eax, %xmm0, %xmm0
 ; AVX512-NEXT:    vpextrd $3, %xmm2, %eax
 ; AVX512-NEXT:    vpinsrd $3, %eax, %xmm0, %xmm0
 ; AVX512-NEXT:    vpaddd %xmm0, %xmm1, %xmm0
 ; AVX512-NEXT:    vzeroupper
 ; AVX512-NEXT:    retq
    %a = sext <16 x i16> %A to <16 x i32>

llvm/trunk/test/CodeGen/X86/vector-shuffle-combining.ll

; AVX-NEXT:    retq
   %12 = or <4 x i32> %5, %9
   %13 = or <4 x i32> %12, %10
   %14 = or <4 x i32> %13, %11
   %15 = add <4 x i32> %a0, %14
   ret <4 x i32> %15
 }
 
 define <8 x i16> @shuffle_extract_insert(<8 x i16> %a) {
-; SSE2-LABEL: shuffle_extract_insert:
-; SSE2:       # %bb.0:
-; SSE2-NEXT:    pextrw $3, %xmm0, %eax
-; SSE2-NEXT:    pextrw $4, %xmm0, %r8d
-; SSE2-NEXT:    pextrw $5, %xmm0, %edx
-; SSE2-NEXT:    pextrw $6, %xmm0, %esi
-; SSE2-NEXT:    movd %xmm0, %edi
-; SSE2-NEXT:    pextrw $7, %xmm0, %ecx
-; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[2,1,2,3,4,5,6,7]
-; SSE2-NEXT:    pinsrw $2, %edi, %xmm0
-; SSE2-NEXT:    pinsrw $3, %eax, %xmm0
-; SSE2-NEXT:    pinsrw $4, %esi, %xmm0
-; SSE2-NEXT:    pinsrw $5, %edx, %xmm0
-; SSE2-NEXT:    pinsrw $6, %r8d, %xmm0
-; SSE2-NEXT:    pinsrw $7, %ecx, %xmm0
-; SSE2-NEXT:    retq
-;
-; SSSE3-LABEL: shuffle_extract_insert:
-; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    pextrw $3, %xmm0, %eax
-; SSSE3-NEXT:    pextrw $4, %xmm0, %r8d
-; SSSE3-NEXT:    pextrw $5, %xmm0, %edx
-; SSSE3-NEXT:    pextrw $6, %xmm0, %esi
-; SSSE3-NEXT:    movd %xmm0, %edi
-; SSSE3-NEXT:    pextrw $7, %xmm0, %ecx
-; SSSE3-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[2,1,2,3,4,5,6,7]
-; SSSE3-NEXT:    pinsrw $2, %edi, %xmm0
-; SSSE3-NEXT:    pinsrw $3, %eax, %xmm0
-; SSSE3-NEXT:    pinsrw $4, %esi, %xmm0
-; SSSE3-NEXT:    pinsrw $5, %edx, %xmm0
-; SSSE3-NEXT:    pinsrw $6, %r8d, %xmm0
-; SSSE3-NEXT:    pinsrw $7, %ecx, %xmm0
-; SSSE3-NEXT:    retq
-;
-; SSE41-LABEL: shuffle_extract_insert:
-; SSE41:       # %bb.0:
-; SSE41-NEXT:    pextrw $4, %xmm0, %eax
-; SSE41-NEXT:    pextrw $6, %xmm0, %ecx
-; SSE41-NEXT:    movd %xmm0, %edx
-; SSE41-NEXT:    pshuflw {{.*#+}} xmm1 = xmm0[2,1,2,3,4,5,6,7]
-; SSE41-NEXT:    pinsrw $2, %edx, %xmm1
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1,2],xmm0[3],xmm1[4,5,6,7]
-; SSE41-NEXT:    pinsrw $4, %ecx, %xmm1
-; SSE41-NEXT:    pblendw {{.*#+}} xmm1 = xmm1[0,1,2,3,4],xmm0[5],xmm1[6,7]
-; SSE41-NEXT:    pinsrw $6, %eax, %xmm1
-; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm1[0,1,2,3,4,5,6],xmm0[7]
-; SSE41-NEXT:    retq
-;
-; AVX-LABEL: shuffle_extract_insert:
-; AVX:       # %bb.0:
-; AVX-NEXT:    vpextrw $4, %xmm0, %eax
-; AVX-NEXT:    vpextrw $6, %xmm0, %ecx
-; AVX-NEXT:    vmovd %xmm0, %edx
-; AVX-NEXT:    vpshuflw {{.*#+}} xmm1 = xmm0[2,1,2,3,4,5,6,7]
-; AVX-NEXT:    vpinsrw $2, %edx, %xmm1, %xmm1
-; AVX-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0,1,2],xmm0[3],xmm1[4,5,6,7]
-; AVX-NEXT:    vpinsrw $4, %ecx, %xmm1, %xmm1
-; AVX-NEXT:    vpblendw {{.*#+}} xmm1 = xmm1[0,1,2,3,4],xmm0[5],xmm1[6,7]
-; AVX-NEXT:    vpinsrw $6, %eax, %xmm1, %xmm1
-; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm1[0,1,2,3,4,5,6],xmm0[7]
-; AVX-NEXT:    retq
+; SSE-LABEL: shuffle_extract_insert:
+; SSE:       # %bb.0:
+; SSE-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[2,1,0,3,4,5,6,7]
+; SSE-NEXT:    pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,6,5,4,7]
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: shuffle_extract_insert:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[2,1,0,3,4,5,6,7]
+; AVX1-NEXT:    vpshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,6,5,4,7]
+; AVX1-NEXT:    retq
+;
+; AVX2-SLOW-LABEL: shuffle_extract_insert:
+; AVX2-SLOW:       # %bb.0:
+; AVX2-SLOW-NEXT:    vpshuflw {{.*#+}} xmm0 = xmm0[2,1,0,3,4,5,6,7]
+; AVX2-SLOW-NEXT:    vpshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,6,5,4,7]
+; AVX2-SLOW-NEXT:    retq
+;
+; AVX2-FAST-LABEL: shuffle_extract_insert:
+; AVX2-FAST:       # %bb.0:
+; AVX2-FAST-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,2,3,0,1,6,7,12,13,10,11,8,9,14,15]
+; AVX2-FAST-NEXT:    retq
   %a0 = extractelement <8 x i16> %a, i32 0
   %a1 = extractelement <8 x i16> %a, i32 1
   %a3 = extractelement <8 x i16> %a, i32 3
   %a4 = extractelement <8 x i16> %a, i32 4
   %a5 = extractelement <8 x i16> %a, i32 5
   %a6 = extractelement <8 x i16> %a, i32 6
   %a7 = extractelement <8 x i16> %a, i32 7
   %1 = shufflevector <8 x i16> %a, <8 x i16> %a, <8 x i32> <i32 2, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   %2 = insertelement <8 x i16> %1, i16 %a1, i32 1
   %3 = insertelement <8 x i16> %2, i16 %a0, i32 2
   %4 = insertelement <8 x i16> %3, i16 %a3, i32 3
   %5 = insertelement <8 x i16> %4, i16 %a6, i32 4
   %6 = insertelement <8 x i16> %5, i16 %a5, i32 5
   %7 = insertelement <8 x i16> %6, i16 %a4, i32 6
   %8 = insertelement <8 x i16> %7, i16 %a7, i32 7
   ret <8 x i16> %8
 }
 
 define <8 x i16> @shuffle_extract_insert_double(<8 x i16> %a, <8 x i16> %b) {
 ; SSE2-LABEL: shuffle_extract_insert_double:
 ; SSE2:       # %bb.0:
-; SSE2-NEXT:    movd %xmm0, %eax
-; SSE2-NEXT:    pextrw $4, %xmm0, %r8d
-; SSE2-NEXT:    pextrw $6, %xmm0, %edx
-; SSE2-NEXT:    pextrw $3, %xmm1, %esi
-; SSE2-NEXT:    pextrw $5, %xmm1, %edi
-; SSE2-NEXT:    pextrw $7, %xmm1, %ecx
-; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,3,2,3,4,5,6,7]
+; SSE2-NEXT:    pshufhw {{.*#+}} xmm1 = xmm1[0,1,2,3,7,5,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[0,2,2,3]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm1 = xmm1[0,1,3,2,4,5,6,7]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[0,2,2,3,4,5,6,7]
+; SSE2-NEXT:    pshufhw {{.*#+}} xmm0 = xmm0[0,1,2,3,4,6,6,7]
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[0,2,2,3]
+; SSE2-NEXT:    pshuflw {{.*#+}} xmm0 = xmm0[1,0,3,2,4,5,6,7]
 ; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
-; SSE2-NEXT:    pinsrw $2, %eax, %xmm0
-; SSE2-NEXT:    pinsrw $3, %esi, %xmm0
-; SSE2-NEXT:    pinsrw $4, %edx, %xmm0
-; SSE2-NEXT:    pinsrw $5, %edi, %xmm0
-; SSE2-NEXT:    pinsrw $6, %r8d, %xmm0
-; SSE2-NEXT:    pinsrw $7, %ecx, %xmm0
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: shuffle_extract_insert_double:
 ; SSSE3:       # %bb.0:
-; SSSE3-NEXT:    movd %xmm0, %eax
-; SSSE3-NEXT:    pextrw $4, %xmm0, %r8d
-; SSSE3-NEXT:    pextrw $6, %xmm0, %edx
-; SSSE3-NEXT:    pextrw $3, %xmm1, %esi
-; SSSE3-NEXT:    pextrw $5, %xmm1, %edi
-; SSSE3-NEXT:    pextrw $7, %xmm1, %ecx
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSSE3-NEXT:    pshufb {{.*#+}} xmm1 = xmm1[0,1,6,7,10,11,14,15,14,15,10,11,12,13,14,15]
+; SSSE3-NEXT:    pshufb {{.*#+}} xmm0 = xmm0[4,5,0,1,12,13,8,9,8,9,12,13,12,13,14,15]
 ; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
-; SSSE3-NEXT:    pinsrw $2, %eax, %xmm0
-; SSSE3-NEXT:    pinsrw $3, %esi, %xmm0
-; SSSE3-NEXT:    pinsrw $4, %edx, %xmm0
-; SSSE3-NEXT:    pinsrw $5, %edi, %xmm0
-; SSSE3-NEXT:    pinsrw $6, %r8d, %xmm0
-; SSSE3-NEXT:    pinsrw $7, %ecx, %xmm0
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: shuffle_extract_insert_double:
 ; SSE41:       # %bb.0:
-; SSE41-NEXT:    movd %xmm0, %eax
-; SSE41-NEXT:    pextrw $4, %xmm0, %ecx
-; SSE41-NEXT:    pextrw $6, %xmm0, %edx
-; SSE41-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE41-NEXT:    pshufb {{.*#+}} xmm1 = xmm1[0,1,6,7,10,11,14,15,14,15,10,11,12,13,14,15]
+; SSE41-NEXT:    pshufb {{.*#+}} xmm0 = xmm0[4,5,0,1,12,13,8,9,8,9,12,13,12,13,14,15]
 ; SSE41-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
-; SSE41-NEXT:    pinsrw $2, %eax, %xmm0
-; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0,1,2],xmm1[3],xmm0[4,5,6,7]
-; SSE41-NEXT:    pinsrw $4, %edx, %xmm0
-; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0,1,2,3,4],xmm1[5],xmm0[6,7]
-; SSE41-NEXT:    pinsrw $6, %ecx, %xmm0
-; SSE41-NEXT:    pblendw {{.*#+}} xmm0 = xmm0[0,1,2,3,4,5,6],xmm1[7]
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: shuffle_extract_insert_double:
 ; AVX:       # %bb.0:
-; AVX-NEXT:    vmovd %xmm0, %eax
-; AVX-NEXT:    vpextrw $4, %xmm0, %ecx
-; AVX-NEXT:    vpextrw $6, %xmm0, %edx
-; AVX-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; AVX-NEXT:    vpshufb {{.*#+}} xmm1 = xmm1[0,1,6,7,10,11,14,15,14,15,10,11,12,13,14,15]
+; AVX-NEXT:    vpshufb {{.*#+}} xmm0 = xmm0[4,5,0,1,12,13,8,9,8,9,12,13,12,13,14,15]
 ; AVX-NEXT:    vpunpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
-; AVX-NEXT:    vpinsrw $2, %eax, %xmm0, %xmm0
-; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1,2],xmm1[3],xmm0[4,5,6,7]
-; AVX-NEXT:    vpinsrw $4, %edx, %xmm0, %xmm0
-; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1,2,3,4],xmm1[5],xmm0[6,7]
-; AVX-NEXT:    vpinsrw $6, %ecx, %xmm0, %xmm0
-; AVX-NEXT:    vpblendw {{.*#+}} xmm0 = xmm0[0,1,2,3,4,5,6],xmm1[7]
 ; AVX-NEXT:    retq
   %a0 = extractelement <8 x i16> %a, i32 0
   %a4 = extractelement <8 x i16> %a, i32 4
   %a6 = extractelement <8 x i16> %a, i32 6
   %b11 = extractelement <8 x i16> %b, i32 3
   %b13 = extractelement <8 x i16> %b, i32 5
   %b15 = extractelement <8 x i16> %b, i32 7
   %1 = shufflevector <8 x i16> %a, <8 x i16> %b, <8 x i32> <i32 2, i32 8, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef, i32 undef>
   %2 = insertelement <8 x i16> %1, i16 %a0, i32 2
   %3 = insertelement <8 x i16> %2, i16 %b11, i32 3
   %4 = insertelement <8 x i16> %3, i16 %a6, i32 4
   %5 = insertelement <8 x i16> %4, i16 %b13, i32 5
   %6 = insertelement <8 x i16> %5, i16 %a4, i32 6
   %7 = insertelement <8 x i16> %6, i16 %b15, i32 7
   ret <8 x i16> %7
 }