[CodeGen] Add hooks/combine to form vector extloads, and enable it on X86.

[ab]

Summary

Moved from the ML:

As the last change in my extload series, here are 3 squashed (WIP) patches
to actually form extloads on vector types.
They used to be disabled, because "None of the supported targets knows
how to perform load and sign extend on vectors in one instruction."

The first part enables the combine on legal vectors, but hides it
behind a profitability callback.
For instance, on ARM, several instructions have folded extload forms,
so it's not always beneficial to create an extload node (and trying to
match extloads is a whole 'nother can of worms).

The second part adds a combine to fold s/zextloads of illegal
(splittable) vector types, to replace it directly by multiple smaller
extloads. I'm not a big fan of this kind of pseudo-legalization in
combines. However, I tried the alternative: form illegal extloads, and
later try to split them up, but then, you sometimes generate extloads
that can't be split up, but have a valid ext+load expansion. At
vector-op legalization time, it's too late to generate this kind of
thing, so you end up forced to scalarize. It's better to just avoid
creating egregiously illegal nodes.

Finally, the last part enables this all, unconditionally, on X86.

Note that the splitting combine is happy with "custom" extloads. As
is, this bypasses the actual custom lowering, and just unrolls the
extload. But from what I've seen, this is still much better than the
current custom lowering, which does some kind of unrolling at the end
anyway (see for instance load_sext_4i8_to_4i64 on SSE2, and the added
FIXME).

Also note that there's a regression in widen_load-2.ll, where
we can no longer fold the load. I'll look into that later.

Anyway: as can be seen from the nice testcase cleanups, there's
something to be done here. The combines feel a bit dirty, but I don't
see a better alternative. Finally, I didn't see changes on the
testsuite (SSE2 X86-64, I'll try SSE4.1 and AVX2 as well.)

P.S.: I squashed all three because I don't think it makes it harder to
review, just longer. The three changes are nicely isolated, and will
be committed separately. I'll split into 3 threads if desired.

Depends on D6533.

Thanks,
-Ahmed

[arsenm]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5334–5335

I don't think this will work correctly for 3 vectors, but it probably doesn't matter

5336

This should use .getStoreSize()

5671–5682

This looks like the same in the sextload handling. Should probably be moved to a new function

5688

This should also use getStoreSize()

5694–5698

It might be helpful to add a new version of getExtLoad based on the version that takes the MMO and an offset rather than having to pass all of the separate arguments

lib/Target/X86/X86ISelLowering.cpp

20109

Missing override

[chandlerc]

Wow this looks amazing. I'm glad you were able to make it all the way through to fixing this properly (I wasn't when I looked here, I ran away screaming).

Some comments inline...

include/llvm/Target/TargetLowering.h

1512

Do we you "ExtLd" in APIs elsewhere? this kind of extreme abbreviation, especially if deviating from other abbreviations, makes the APIs really hard to understand and find.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5660

This code is extremely similar to the sext case. Can it be factored out with an appropriate predicate or input instruction tag or something?

5665–5671

Even if you can't put this into a common function with sext, at least pull both into their own functions so that you can use early exit for all of this rather than boolean variables.

test/CodeGen/X86/widen_load-2.ll

76–79 ↗

(On Diff #17942)

This is both weird and unfortunate... Do understand why this happens?

[ab]

include/llvm/Target/TargetLowering.h

1512

I agree, and originally wanted to use the closer-to-ubiquitous LoadExt. However, there's enableExtLdPromotion, which is the most similar API so should be similarly named.

I guess I could rename both to LoadExt instead, how does that sound? That would stay consistent with ISD::LoadExtType, and the various *LoadExt* legality functions.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5334–5335

You're right, I added a check for isPow2VectorType.

5660

It should be trivial to factor it out. Other similar combines as well (I'll take care of it separately.)

5694–5698

I thought so as well, but two things would make it confusing IMO: 1) the offset operand is for indexed loads only, which is a concept different from an ADDed base pointer. Also, less importantly, 2) we're not sure we can reuse the previous MMO, as the types may differ.
Or maybe you meant something else?

test/CodeGen/X86/widen_load-2.ll

76–79 ↗

(On Diff #17942)

This was actually a rebasing mistake, and doesn't happen on a clean apply of the patch. Sorry for the noise.

[chandlerc]

include/llvm/Target/TargetLowering.h

1512

We're so inconsistent here that I don't really care about what the old code happened to name things provided we're consistent going forward and converge on that. Given that, I'd prefer the minimal amount of abbreviation that is sane, and I think spelling out "load" but abbreviating "ext" is a fine compromise there.

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5660

Please don't add duplicated code and then factor it afterward.

Either do the minimal amount of factoring in this change so that we don't have to review 2x the code, or do the factoring first, and then the functional change.

[ab]

Ping!

-Ahmed

[hfinkel]

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

5227

Please make this comment more verbose. We might generate multiple loads and then combine them together, and it is not clear here whether this combining happens to the results of the sext or the sext's operand.

5274

Is this alignment correct? Aren't these loads smaller than the original one?

[chandlerc]

My only comments beyond what Hal has already mentioned are real nits:

s/dl/DL/

Can you use DL everywhere? It's odd that in one place you use SDLoc(N0). It may be correct, but if so, it's likely not obvious to the reader why we need to do something different here.

Finally, I'd find the indentation more consistent with clang-format. But now we're waaay down into minor details. This is generally looking much better.

[ab]

• Use smaller alignment for the split loads.
• Improve formatting, names.
• Don't accept ANYEXTs, since it's not testable or supported yet.

[hfinkel]

LGTM.

include/llvm/Target/TargetLowering.h

     return isZExtFree(Val.getValueType(), VT2);
   }
 
+  /// Return true if folding a vector load into ExtVal (a sign, zero, or any
+  /// extend node) is profitable.
+  virtual bool isVectorLoadExtDesirable(SDValue ExtVal) const {
+    return false;
+  }
+
   /// Return true if an fneg operation is free to the point where it is never
   /// worthwhile to replace it with a bitwise operation.
   virtual bool isFNegFree(EVT VT) const {

lib/CodeGen/SelectionDAG/DAGCombiner.cpp

     SDValue SimplifyNodeWithTwoResults(SDNode *N, unsigned LoOp,
                                          unsigned HiOp);
     SDValue CombineConsecutiveLoads(SDNode *N, EVT VT);
+    SDValue CombineExtLoad(SDNode *N);
     SDValue ConstantFoldBITCASTofBUILD_VECTOR(SDNode *, EVT);
     SDValue BuildSDIV(SDNode *N);
     SDValue BuildSDIVPow2(SDNode *N);

   }
 }
 
+// FIXME: Bring more similar combines here, common to sext/zext (and aext).
+SDValue DAGCombiner::CombineExtLoad(SDNode *N) {
+  SDValue N0 = N->getOperand(0);
+  EVT DstVT = N->getValueType(0);
+  EVT SrcVT = N0.getValueType();
+
+  assert((N->getOpcode() == ISD::SIGN_EXTEND ||
+          N->getOpcode() == ISD::ZERO_EXTEND ||
+          N->getOpcode() == ISD::ANY_EXTEND) &&
+         "Unexpected node type (not an extend)!");
+
+  // fold (sext (load x)) -> (sext (truncate (sextload x)))
+  // Only on illegal but splittable vectors.
+  // All legal types, and illegal non-vector types, are handled elsewhere.
+  // This combine is controlled by TargetLowering::isVectorLoadExtDesirable.
+  if (N0->getOpcode() != ISD::LOAD)
+    return SDValue();
+
+  LoadSDNode *LN0 = cast<LoadSDNode>(N0);
+
+  if (!ISD::isNON_EXTLoad(LN0) || !ISD::isUNINDEXEDLoad(LN0) ||
+      !N0.hasOneUse() || LN0->isVolatile() ||
+      !DstVT.isVector() || !DstVT.isPow2VectorType() ||
+      !TLI.isVectorLoadExtDesirable(SDValue(N, 0)))
+    return SDValue();
+
+  SmallVector<SDNode *, 4> SetCCs;
+  if (!ExtendUsesToFormExtLoad(N, N0, N->getOpcode(), SetCCs, TLI))
+    return SDValue();
+
+  ISD::LoadExtType ExtType =
+      N->getOpcode() == ISD::SIGN_EXTEND ? ISD::SEXTLOAD : ISD::ZEXTLOAD;
+
+  // Try to split the vector types to get down to legal types.
+  EVT SplitSrcVT = SrcVT;
+  EVT SplitDstVT = DstVT;
+  while (!TLI.isLoadExtLegalOrCustom(ExtType, SplitDstVT, SplitSrcVT) &&
+         SplitSrcVT.getVectorNumElements() > 1) {
+    SplitDstVT = DAG.GetSplitDestVTs(SplitDstVT).first;
+    SplitSrcVT = DAG.GetSplitDestVTs(SplitSrcVT).first;
+  }
+
+  if (!TLI.isLoadExtLegalOrCustom(ExtType, SplitDstVT, SplitSrcVT))
+    return SDValue();
+
+  SDLoc dl(N);
+  unsigned NumSplits =
+    DstVT.getVectorNumElements() / SplitDstVT.getVectorNumElements();
+  unsigned Stride = SplitSrcVT.getStoreSize();
+  SmallVector<SDValue, 4> SplitDsts;
+  SmallVector<SDValue, 4> LoadChains;
+
+  SDValue BasePTR = LN0->getBasePtr();
+  for (unsigned Idx = 0; Idx < NumSplits; Idx++) {
+    SDValue SplitLoad = DAG.getExtLoad(
+      ExtType, dl, SplitDstVT, LN0->getChain(), BasePTR,
+      LN0->getPointerInfo().getWithOffset(Idx * Stride), SplitSrcVT,
+      LN0->isVolatile(), LN0->isNonTemporal(), LN0->isInvariant(),
+      LN0->getAlignment(), LN0->getAAInfo());
+
+    BasePTR =
+      DAG.getNode(ISD::ADD, dl, BasePTR.getValueType(), BasePTR,
+                  DAG.getConstant(Stride, BasePTR.getValueType()));
+
+    SplitDsts.push_back(SplitLoad.getValue(0));
+    LoadChains.push_back(SplitLoad.getValue(1));
+  }
+  SDValue NewChain =
+    DAG.getNode(ISD::TokenFactor, dl, MVT::Other, LoadChains);
+  SDValue Value = DAG.getNode(ISD::CONCAT_VECTORS, dl, DstVT, SplitDsts);
+
+  CombineTo(N, Value);
+  SDValue Trunc =
+    DAG.getNode(ISD::TRUNCATE, SDLoc(N0), N0.getValueType(), Value);
+  CombineTo(N0.getNode(), Trunc, NewChain);
+  ExtendSetCCUses(SetCCs, Trunc, Value, SDLoc(N),
+                  (ISD::NodeType)N->getOpcode());
+  return SDValue(N, 0); // Return N so it doesn't get rechecked!
+}
+
 SDValue DAGCombiner::visitSIGN_EXTEND(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);

   }
 
   // fold (sext (load x)) -> (sext (truncate (sextload x)))
-  // None of the supported targets knows how to perform load and sign extend
-  // on vectors in one instruction.  We only perform this transformation on
-  // scalars.
-  if (ISD::isNON_EXTLoad(N0.getNode()) && !VT.isVector() &&
+  if (ISD::isNON_EXTLoad(N0.getNode()) &&
       ISD::isUNINDEXEDLoad(N0.getNode()) &&
-      ((!LegalOperations && !cast<LoadSDNode>(N0)->isVolatile()) ||
+      ((!LegalOperations && !VT.isVector() &&
+        !cast<LoadSDNode>(N0)->isVolatile()) ||
        TLI.isLoadExtLegal(ISD::SEXTLOAD, VT, N0.getValueType()))) {
     bool DoXform = true;
     SmallVector<SDNode*, 4> SetCCs;
     if (!N0.hasOneUse())
       DoXform = ExtendUsesToFormExtLoad(N, N0, ISD::SIGN_EXTEND, SetCCs, TLI);
+    if (VT.isVector())
+      DoXform &= TLI.isVectorLoadExtDesirable(SDValue(N, 0));
     if (DoXform) {
       LoadSDNode *LN0 = cast<LoadSDNode>(N0);
       SDValue ExtLoad = DAG.getExtLoad(ISD::SEXTLOAD, SDLoc(N), VT,

     }
   }
 
+  // fold (sext (load x)) -> (sext (truncate (sextload x)))
+  // Only for illegal but splittable vectors.
+  if (SDValue ExtLoad = CombineExtLoad(N))
+    return ExtLoad;
+
   // fold (sext (sextload x)) -> (sext (truncate (sextload x)))
   // fold (sext ( extload x)) -> (sext (truncate (sextload x)))
   if ((ISD::isSEXTLoad(N0.getNode()) || ISD::isEXTLoad(N0.getNode())) &&

   }
 
   // fold (zext (load x)) -> (zext (truncate (zextload x)))
-  // None of the supported targets knows how to perform load and vector_zext
-  // on vectors in one instruction.  We only perform this transformation on
-  // scalars.
-  if (ISD::isNON_EXTLoad(N0.getNode()) && !VT.isVector() &&
-      ISD::isUNINDEXEDLoad(N0.getNode()) &&
-      ((!LegalOperations && !cast<LoadSDNode>(N0)->isVolatile()) ||
+  if (ISD::isNON_EXTLoad(N0.getNode()) && ISD::isUNINDEXEDLoad(N0.getNode()) &&
+      ((!LegalOperations && !VT.isVector() &&
+        !cast<LoadSDNode>(N0)->isVolatile()) ||
        TLI.isLoadExtLegal(ISD::ZEXTLOAD, VT, N0.getValueType()))) {
     bool DoXform = true;
     SmallVector<SDNode*, 4> SetCCs;
     if (!N0.hasOneUse())
       DoXform = ExtendUsesToFormExtLoad(N, N0, ISD::ZERO_EXTEND, SetCCs, TLI);
+    if (VT.isVector())
+      DoXform &= TLI.isVectorLoadExtDesirable(SDValue(N, 0));
     if (DoXform) {
       LoadSDNode *LN0 = cast<LoadSDNode>(N0);
       SDValue ExtLoad = DAG.getExtLoad(ISD::ZEXTLOAD, SDLoc(N), VT,

     }
   }
 
+  // fold (zext (load x)) -> (zext (truncate (zextload x)))
+  // Only on illegal but splittable vectors.
+  if (SDValue ExtLoad = CombineExtLoad(N))
+    return ExtLoad;
+
   // fold (zext (and/or/xor (load x), cst)) ->
   //      (and/or/xor (zextload x), (zext cst))
   if ((N0.getOpcode() == ISD::AND || N0.getOpcode() == ISD::OR ||

lib/Target/X86/X86ISelLowering.h

     bool isZExtFree(EVT VT1, EVT VT2) const override;
     bool isZExtFree(SDValue Val, EVT VT2) const override;
 
+    /// Return true if folding a vector load into ExtVal (a sign, zero, or any
+    /// extend node) is profitable.
+    bool isVectorLoadExtDesirable(SDValue) const override;
+
     /// Return true if an FMA operation is faster than a pair of fmul and fadd
     /// instructions. fmuladd intrinsics will be expanded to FMAs when this
     /// method returns true, otherwise fmuladd is expanded to fmul + fadd.

lib/Target/X86/X86ISelLowering.cpp

 // may emit an illegal shuffle but the expansion is still better than scalar
 // code. We generate X86ISD::VSEXT for SEXTLOADs if it's available, otherwise
 // we'll emit a shuffle and a arithmetic shift.
+// FIXME: Is the expansion actually better than scalar code? It doesn't seem so.
 // TODO: It is possible to support ZExt by zeroing the undef values during
 // the shuffle phase or after the shuffle.
 static SDValue LowerExtendedLoad(SDValue Op, const X86Subtarget *Subtarget,

   return false;
 }
 
+bool X86TargetLowering::isVectorLoadExtDesirable(SDValue) const {
+  return true;
+}
+
 bool
 X86TargetLowering::isFMAFasterThanFMulAndFAdd(EVT VT) const {
   if (!(Subtarget->hasFMA() || Subtarget->hasFMA4()))

test/CodeGen/X86/vector-sext.ll

 define <16 x i16> @sext_16i8_to_16i16(<16 x i8> *%ptr) {
 ; SSE2-LABEL: sext_16i8_to_16i16:
 ; SSE2:       # BB#0: # %entry
-; SSE2-NEXT:    movdqa (%rdi), %xmm1
-; SSE2-NEXT:    movdqa %xmm1, %xmm0
-; SSE2-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
-; SSE2-NEXT:    psllw $8, %xmm0
+; SSE2-NEXT:    movq (%rdi), %xmm0
+; SSE2-NEXT:    punpcklbw %xmm0, %xmm0    # xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
 ; SSE2-NEXT:    psraw $8, %xmm0
-; SSE2-NEXT:    punpckhbw {{.*#+}} xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
-; SSE2-NEXT:    psllw $8, %xmm1
+; SSE2-NEXT:    movq 8(%rdi), %xmm1
+; SSE2-NEXT:    punpcklbw %xmm1, %xmm1    # xmm1 = xmm1[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
 ; SSE2-NEXT:    psraw $8, %xmm1
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: sext_16i8_to_16i16:
 ; SSSE3:       # BB#0: # %entry
-; SSSE3-NEXT:    movdqa (%rdi), %xmm1
-; SSSE3-NEXT:    movdqa %xmm1, %xmm0
-; SSSE3-NEXT:    punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
-; SSSE3-NEXT:    psllw $8, %xmm0
+; SSSE3-NEXT:    movq (%rdi), %xmm0
+; SSSE3-NEXT:    punpcklbw %xmm0, %xmm0    # xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
 ; SSSE3-NEXT:    psraw $8, %xmm0
-; SSSE3-NEXT:    punpckhbw {{.*#+}} xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
-; SSSE3-NEXT:    psllw $8, %xmm1
+; SSSE3-NEXT:    movq 8(%rdi), %xmm1
+; SSSE3-NEXT:    punpcklbw %xmm1, %xmm1    # xmm1 = xmm1[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
 ; SSSE3-NEXT:    psraw $8, %xmm1
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: sext_16i8_to_16i16:
 ; SSE41:       # BB#0: # %entry
-; SSE41-NEXT:    movdqa (%rdi), %xmm1
-; SSE41-NEXT:    pmovzxbw %xmm1, %xmm0
-; SSE41-NEXT:    psllw $8, %xmm0
-; SSE41-NEXT:    psraw $8, %xmm0
-; SSE41-NEXT:    punpckhbw {{.*#+}} xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
-; SSE41-NEXT:    psllw $8, %xmm1
-; SSE41-NEXT:    psraw $8, %xmm1
+; SSE41-NEXT:    pmovsxbw (%rdi), %xmm0
+; SSE41-NEXT:    pmovsxbw 8(%rdi), %xmm1
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: sext_16i8_to_16i16:
 ; AVX1:       # BB#0: # %entry
-; AVX1-NEXT:    vmovdqa (%rdi), %xmm0
-; AVX1-NEXT:    vpmovsxbw %xmm0, %xmm1
-; AVX1-NEXT:    vpshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
-; AVX1-NEXT:    vpmovsxbw %xmm0, %xmm0
-; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
+; AVX1-NEXT:    vpmovsxbw (%rdi), %xmm0
+; AVX1-NEXT:    vpmovsxbw 8(%rdi), %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: sext_16i8_to_16i16:

 ; X32-SSE41-LABEL: sext_16i8_to_16i16:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE41-NEXT:    movdqa (%eax), %xmm1
-; X32-SSE41-NEXT:    pmovzxbw %xmm1, %xmm0
-; X32-SSE41-NEXT:    psllw $8, %xmm0
-; X32-SSE41-NEXT:    psraw $8, %xmm0
-; X32-SSE41-NEXT:    punpckhbw {{.*#+}} xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
-; X32-SSE41-NEXT:    psllw $8, %xmm1
-; X32-SSE41-NEXT:    psraw $8, %xmm1
+; X32-SSE41-NEXT:    pmovsxbw (%eax), %xmm0
+; X32-SSE41-NEXT:    pmovsxbw 8(%eax), %xmm1
 ; X32-SSE41-NEXT:    retl
 entry:
  %X = load <16 x i8>* %ptr

 define <4 x i64> @load_sext_4i8_to_4i64(<4 x i8> *%ptr) {
 ; SSE2-LABEL: load_sext_4i8_to_4i64:
 ; SSE2:       # BB#0: # %entry
-; SSE2-NEXT:    movd (%rdi), %xmm1
-; SSE2-NEXT:    punpcklbw {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3],xmm1[4],xmm0[4],xmm1[5],xmm0[5],xmm1[6],xmm0[6],xmm1[7],xmm0[7]
-; SSE2-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[0,1,1,3]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movsbq %al, %rax
+; SSE2-NEXT:    movsbq 1(%rdi), %rax
+; SSE2-NEXT:    movd %rax, %xmm1
+; SSE2-NEXT:    movsbq (%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm0
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movsbq %al, %rax
+; SSE2-NEXT:    punpcklqdq %xmm1, %xmm0    # xmm0 = xmm0[0],xmm1[0]
+; SSE2-NEXT:    movsbq 3(%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm2
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movsbq %al, %rax
+; SSE2-NEXT:    movsbq 2(%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm1
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movsbq %al, %rax
-; SSE2-NEXT:    movd %rax, %xmm2
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSE2-NEXT:    punpcklqdq %xmm2, %xmm1    # xmm1 = xmm1[0],xmm2[0]
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: load_sext_4i8_to_4i64:
 ; SSSE3:       # BB#0: # %entry
-; SSSE3-NEXT:    movd (%rdi), %xmm1
-; SSSE3-NEXT:    punpcklbw {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3],xmm1[4],xmm0[4],xmm1[5],xmm0[5],xmm1[6],xmm0[6],xmm1[7],xmm0[7]
-; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[0,1,1,3]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movsbq %al, %rax
+; SSSE3-NEXT:    movsbq 1(%rdi), %rax
+; SSSE3-NEXT:    movd %rax, %xmm1
+; SSSE3-NEXT:    movsbq (%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm0
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movsbq %al, %rax
+; SSSE3-NEXT:    punpcklqdq %xmm1, %xmm0    # xmm0 = xmm0[0],xmm1[0]
+; SSSE3-NEXT:    movsbq 3(%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm2
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movsbq %al, %rax
+; SSSE3-NEXT:    movsbq 2(%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm1
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movsbq %al, %rax
-; SSSE3-NEXT:    movd %rax, %xmm2
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSSE3-NEXT:    punpcklqdq %xmm2, %xmm1    # xmm1 = xmm1[0],xmm2[0]
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: load_sext_4i8_to_4i64:
 ; SSE41:       # BB#0: # %entry
-; SSE41-NEXT:    pmovzxbd (%rdi), %xmm1
-; SSE41-NEXT:    pmovzxdq %xmm1, %xmm0
-; SSE41-NEXT:    pextrq $1, %xmm0, %rax
-; SSE41-NEXT:    movsbq %al, %rax
-; SSE41-NEXT:    movd %rax, %xmm2
-; SSE41-NEXT:    movd %xmm0, %rax
-; SSE41-NEXT:    movsbq %al, %rax
-; SSE41-NEXT:    movd %rax, %xmm0
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE41-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,2,3,3]
-; SSE41-NEXT:    pextrq $1, %xmm1, %rax
-; SSE41-NEXT:    movsbq %al, %rax
-; SSE41-NEXT:    movd %rax, %xmm2
-; SSE41-NEXT:    movd %xmm1, %rax
-; SSE41-NEXT:    movsbq %al, %rax
-; SSE41-NEXT:    movd %rax, %xmm1
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSE41-NEXT:    pmovsxbq (%rdi), %xmm0
+; SSE41-NEXT:    pmovsxbq 2(%rdi), %xmm1
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: load_sext_4i8_to_4i64:

 ; X32-SSE41-LABEL: load_sext_4i8_to_4i64:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE41-NEXT:    movd (%eax), %xmm0
-; X32-SSE41-NEXT:    pmovzxbd %xmm0, %xmm1
-; X32-SSE41-NEXT:    pmovzxbq %xmm0, %xmm2
-; X32-SSE41-NEXT:    movd %xmm2, %eax
-; X32-SSE41-NEXT:    movsbl %al, %eax
-; X32-SSE41-NEXT:    movd %eax, %xmm0
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $1, %eax, %xmm0
-; X32-SSE41-NEXT:    pextrd $2, %xmm2, %eax
-; X32-SSE41-NEXT:    movsbl %al, %eax
-; X32-SSE41-NEXT:    pinsrd $2, %eax, %xmm0
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $3, %eax, %xmm0
-; X32-SSE41-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; X32-SSE41-NEXT:    movd %xmm2, %eax
-; X32-SSE41-NEXT:    movsbl %al, %eax
-; X32-SSE41-NEXT:    movd %eax, %xmm1
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $1, %eax, %xmm1
-; X32-SSE41-NEXT:    pextrd $2, %xmm2, %eax
-; X32-SSE41-NEXT:    movsbl %al, %eax
-; X32-SSE41-NEXT:    pinsrd $2, %eax, %xmm1
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $3, %eax, %xmm1
+; X32-SSE41-NEXT:    pmovsxbq (%eax), %xmm0
+; X32-SSE41-NEXT:    pmovsxbq 2(%eax), %xmm1
 ; X32-SSE41-NEXT:    retl
 entry:
  %X = load <4 x i8>* %ptr

 define <4 x i64> @load_sext_4i16_to_4i64(<4 x i16> *%ptr) {
 ; SSE2-LABEL: load_sext_4i16_to_4i64:
 ; SSE2:       # BB#0: # %entry
-; SSE2-NEXT:    movq (%rdi), %xmm1
-; SSE2-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[0,1,1,3]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movswq %ax, %rax
+; SSE2-NEXT:    movswq 2(%rdi), %rax
+; SSE2-NEXT:    movd %rax, %xmm1
+; SSE2-NEXT:    movswq (%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm0
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movswq %ax, %rax
+; SSE2-NEXT:    punpcklqdq %xmm1, %xmm0    # xmm0 = xmm0[0],xmm1[0]
+; SSE2-NEXT:    movswq 6(%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm2
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movswq %ax, %rax
+; SSE2-NEXT:    movswq 4(%rdi), %rax
 ; SSE2-NEXT:    movd %rax, %xmm1
-; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSE2-NEXT:    movd %xmm2, %rax
-; SSE2-NEXT:    movswq %ax, %rax
-; SSE2-NEXT:    movd %rax, %xmm2
-; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSE2-NEXT:    punpcklqdq %xmm2, %xmm1    # xmm1 = xmm1[0],xmm2[0]
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: load_sext_4i16_to_4i64:
 ; SSSE3:       # BB#0: # %entry
-; SSSE3-NEXT:    movq (%rdi), %xmm1
-; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm1 = xmm1[0],xmm0[0],xmm1[1],xmm0[1],xmm1[2],xmm0[2],xmm1[3],xmm0[3]
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[0,1,1,3]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movswq %ax, %rax
+; SSSE3-NEXT:    movswq 2(%rdi), %rax
+; SSSE3-NEXT:    movd %rax, %xmm1
+; SSSE3-NEXT:    movswq (%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm0
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movswq %ax, %rax
+; SSSE3-NEXT:    punpcklqdq %xmm1, %xmm0    # xmm0 = xmm0[0],xmm1[0]
+; SSSE3-NEXT:    movswq 6(%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm2
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movswq %ax, %rax
+; SSSE3-NEXT:    movswq 4(%rdi), %rax
 ; SSSE3-NEXT:    movd %rax, %xmm1
-; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm2[2,3,0,1]
-; SSSE3-NEXT:    movd %xmm2, %rax
-; SSSE3-NEXT:    movswq %ax, %rax
-; SSSE3-NEXT:    movd %rax, %xmm2
-; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSSE3-NEXT:    punpcklqdq %xmm2, %xmm1    # xmm1 = xmm1[0],xmm2[0]
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: load_sext_4i16_to_4i64:
 ; SSE41:       # BB#0: # %entry
-; SSE41-NEXT:    movq (%rdi), %xmm0
-; SSE41-NEXT:    pmovzxwd %xmm0, %xmm1
-; SSE41-NEXT:    pmovzxwq %xmm0, %xmm0
-; SSE41-NEXT:    pextrq $1, %xmm0, %rax
-; SSE41-NEXT:    movswq %ax, %rax
-; SSE41-NEXT:    movd %rax, %xmm2
-; SSE41-NEXT:    movd %xmm0, %rax
-; SSE41-NEXT:    movswq %ax, %rax
-; SSE41-NEXT:    movd %rax, %xmm0
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
-; SSE41-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,2,3,3]
-; SSE41-NEXT:    pextrq $1, %xmm1, %rax
-; SSE41-NEXT:    movswq %ax, %rax
-; SSE41-NEXT:    movd %rax, %xmm2
-; SSE41-NEXT:    movd %xmm1, %rax
-; SSE41-NEXT:    movswq %ax, %rax
-; SSE41-NEXT:    movd %rax, %xmm1
-; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm1 = xmm1[0],xmm2[0]
+; SSE41-NEXT:    pmovsxwq (%rdi), %xmm0
+; SSE41-NEXT:    pmovsxwq 4(%rdi), %xmm1
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: load_sext_4i16_to_4i64:

 ; X32-SSE41-LABEL: load_sext_4i16_to_4i64:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE41-NEXT:    movsd (%eax), %xmm0
-; X32-SSE41-NEXT:    pmovzxwd %xmm0, %xmm1
-; X32-SSE41-NEXT:    pmovzxwq %xmm0, %xmm2
-; X32-SSE41-NEXT:    movd %xmm2, %eax
-; X32-SSE41-NEXT:    cwtl
-; X32-SSE41-NEXT:    movd %eax, %xmm0
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $1, %eax, %xmm0
-; X32-SSE41-NEXT:    pextrd $2, %xmm2, %eax
-; X32-SSE41-NEXT:    cwtl
-; X32-SSE41-NEXT:    pinsrd $2, %eax, %xmm0
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $3, %eax, %xmm0
-; X32-SSE41-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[2,2,3,3]
-; X32-SSE41-NEXT:    movd %xmm2, %eax
-; X32-SSE41-NEXT:    cwtl
-; X32-SSE41-NEXT:    movd %eax, %xmm1
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $1, %eax, %xmm1
-; X32-SSE41-NEXT:    pextrd $2, %xmm2, %eax
-; X32-SSE41-NEXT:    cwtl
-; X32-SSE41-NEXT:    pinsrd $2, %eax, %xmm1
-; X32-SSE41-NEXT:    sarl $31, %eax
-; X32-SSE41-NEXT:    pinsrd $3, %eax, %xmm1
+; X32-SSE41-NEXT:    pmovsxwq (%eax), %xmm0
+; X32-SSE41-NEXT:    pmovsxwq 4(%eax), %xmm1
 ; X32-SSE41-NEXT:    retl
 entry:
  %X = load <4 x i16>* %ptr

test/CodeGen/X86/vector-zext.ll

 
 ; SSE41-LABEL: load_zext_16i8_to_16i16:
 ; SSE41:        # BB#0: # %entry
-; SSE41-NEXT:   movdqa  (%rdi), %xmm1
-; SSE41-NEXT:   pmovzxbw  %xmm1, %xmm0
-; SSE41-NEXT:   movdqa  {{.*#+}} xmm2 = [255,255,255,255,255,255,255,255]
-; SSE41-NEXT:   pand  %xmm2, %xmm0
-; SSE41-NEXT:   punpckhbw %xmm1, %xmm1    # xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
-; SSE41-NEXT:   pand  %xmm2, %xmm1
+; SSE41-NEXT:   pmovzxbw (%rdi), %xmm0
+; SSE41-NEXT:   pmovzxbw 8(%rdi), %xmm1
 ; SSE41-NEXT:   retq
 
 ; AVX1-LABEL: load_zext_16i8_to_16i16:
 ; AVX1:        # BB#0: # %entry
-; AVX1-NEXT:  vmovdqa (%rdi), %xmm0
-; AVX1-NEXT:  vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:  vpunpckhbw  %xmm1, %xmm0, %xmm1 # xmm1 = xmm0[8],xmm1[8],xmm0[9],xmm1[9],xmm0[10],xmm1[10],xmm0[11],xmm1[11],xmm0[12],xmm1[12],xmm0[13],xmm1[13],xmm0[14],xmm1[14],xmm0[15],xmm1[15]
-; AVX1-NEXT:  vpmovzxbw %xmm0, %xmm0
-; AVX1-NEXT:  vinsertf128 $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:  vpmovzxbw (%rdi), %xmm0
+; AVX1-NEXT:  vpmovzxbw 8(%rdi), %xmm1
+; AVX1-NEXT:  vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX1-NEXT:  retq
 
 ; AVX2-LABEL: load_zext_16i8_to_16i16:

 
 ; SSE41-LABEL: load_zext_8i16_to_8i32:
 ; SSE41:        # BB#0: # %entry
-; SSE41-NEXT:   movdqa  (%rdi), %xmm1
-; SSE41-NEXT:   pmovzxwd  %xmm1, %xmm0
-; SSE41-NEXT:   movdqa  {{.*#+}} xmm2 = [65535,65535,65535,65535]
-; SSE41-NEXT:   pand  %xmm2, %xmm0
-; SSE41-NEXT:   punpckhwd %xmm1, %xmm1    # xmm1 = xmm1[4,4,5,5,6,6,7,7]
-; SSE41-NEXT:   pand  %xmm2, %xmm1
+; SSE41-NEXT:   pmovzxwd (%rdi), %xmm0
+; SSE41-NEXT:   pmovzxwd 8(%rdi), %xmm1
 ; SSE41-NEXT:   retq
 
 ; AVX1-LABEL: load_zext_8i16_to_8i32:
 ; AVX1:        # BB#0: # %entry
-; AVX1-NEXT:    vmovdqa       (%rdi), %xmm0
-; AVX1-NEXT:    vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:    vpunpckhwd    %xmm1, %xmm0, %xmm1 # xmm1 = xmm0[4],xmm1[4],xmm0[5],xmm1[5],xmm0[6],xmm1[6],xmm0[7],xmm1[7]
-; AVX1-NEXT:    vpmovzxwd     %xmm0, %xmm0
-; AVX1-NEXT:    vinsertf128   $1, %xmm1, %ymm0, %ymm0
+; AVX1-NEXT:    vpmovzxwd (%rdi), %xmm0
+; AVX1-NEXT:    vpmovzxwd 8(%rdi), %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX1-NEXT:    retq
 
 ; AVX2-LABEL: load_zext_8i16_to_8i32:

 
 ; SSE41-LABEL: load_zext_4i32_to_4i64:
 ; SSE41:        # BB#0: # %entry
-; SSE41-NEXT:   movdqa  (%rdi), %xmm1
-; SSE41-NEXT:   pmovzxdq  %xmm1, %xmm0
-; SSE41-NEXT:   movdqa  {{.*#+}} xmm2 = [4294967295,4294967295]
-; SSE41-NEXT:   pand  %xmm2, %xmm0
-; SSE41-NEXT:   pshufd  $-6, %xmm1, %xmm1       # xmm1 = xmm1[2,2,3,3]
-; SSE41-NEXT:   pand  %xmm2, %xmm1
+; SSE41-NEXT:   pmovzxdq (%rdi), %xmm0
+; SSE41-NEXT:   pmovzxdq 8(%rdi), %xmm1
 ; SSE41-NEXT:   retq
 
 ; AVX1-LABEL: load_zext_4i32_to_4i64:
 ; AVX1:        # BB#0: # %entry
-; AVX1-NEXT:  vmovdqa (%rdi), %xmm0
-; AVX1-NEXT:  vpxor %xmm1, %xmm1, %xmm1
-; AVX1-NEXT:  vpunpckhdq  %xmm1, %xmm0, %xmm1 # xmm1 = xmm0[2],xmm1[2],xmm0[3],xmm1[3]
-; AVX1-NEXT:  vpmovzxdq %xmm0, %xmm0
+; AVX1-NEXT:  vpmovzxdq (%rdi), %xmm0
+; AVX1-NEXT:  vpmovzxdq 8(%rdi), %xmm1
 ; AVX1-NEXT:  vinsertf128 $1, %xmm1, %ymm0, %ymm0
 ; AVX1-NEXT:  retq