[PATCH, PR24373] Combine shifts for x86

[evstupac]

Summary

The patch folds

(ashr (shl, a, [56,48,32,24,16]), SarConst)

into

(shl, (sext (a), [56,48,32,24,16] - SarConst))

or into

(lshr, (sext (a), SarConst - [56,48,32,24,16]))

depending on sign of (SarConst - [56,48,32,24,16])

sexts in X86 are MOVs. The MOVs have the same code size as above SHIFTs (only SHIFT on 1 has lower code size).
However the MOVs have 2 advantages to SHIFTs on x86:

1. MOVs can write to a register that differs from source
2. MOVs accept memory operands

[RKSimon]

Do you have any before/after perf timings?

test/CodeGen/X86/sar_fold.ll

3

Load tests?

Possibly regenerate this using update_llc_test_checks.py?

test/CodeGen/X86/sar_fold64.ll

2

Load tests?

Possibly regenerate this using update_llc_test_checks.py?

[evstupac]

In D13161#255982, @RKSimon wrote:

Do you have any before/after perf timings?

Yes. Spec2000 performance is almost flat.
Unit performance tests get up to 40% gain.
The patch fixes regression in PR24373.

[evstupac]

test/CodeGen/X86/sar_fold.ll

3

The test checks if "(a<<16)>>17" is folded to movswl and any of possible variant of "<<1". It could be "add %eax, %eax", "shl %eax" or even "lea". The test is to check only folding to movswl. Regenerating the test using update_llc_test_checks.py will make it less flexible.

test/CodeGen/X86/sar_fold64.ll

2

Yes.

[RKSimon]

I do wonder if this could be beneficial for other targets - possibly moving this to DAGCombiner and using a a test against isExtFree() or similar?

Also, please can you add tests against the load-execute versions movs*?

[evstupac]

I do wonder if this could be beneficial for other targets - possibly moving this to DAGCombiner and using a a test against isExtFree() or similar?

That could be, however that is not obvious. I know that for Arm shifts could go in addition to the logic instructions. That they it could be better to leave shifts. Anyway keeping IR target independent is better. Note that at some point IR expand sext to pair of shifts (this, I think, simplify IR for further optimizations):

InstCombineCasts.cpp, visitSExt function// We need to emit a shl + ashr to do the sign extend.

Also, please can you add tests against the load-execute versions movs*?

Do you mean segment moves? If so there are no sar/shl pair that could be folded to such mov.

[RKSimon]

In D13161#259762, @evstupac wrote:

Also, please can you add tests against the load-execute versions movs*?

Do you mean segment moves? If so there are no sar/shl pair that could be folded to such mov.

I meant something that tested that a load + shl + ashr pattern gets combined to a folded movs** instruction

[evstupac]

In D13161#260607, @RKSimon wrote:

In D13161#259762, @evstupac wrote:

Also, please can you add tests against the load-execute versions movs*?

Do you mean segment moves? If so there are no sar/shl pair that could be folded to such mov.

I meant something that tested that a load + shl + ashr pattern gets combined to a folded movs** instruction

Ok. Good point. Actually that is what "test/CodeGen/X86/sar_fold.ll" test now, as in 32 bit mode parameter "i32 %a" goes from stack. Let's add CHECK-NEXT after BB0:
; CHECK: # BB#0:
; CHECK-NEXT: movswl {{[0-9]+}}(%esp), %eax
to be sure that there is no simple "movl" from stack.

[evstupac]

Add check for the tests.

[RKSimon]

LGTM

[evstupac]

During final check 2 new tests failed on x86:

1. sar_fold64.ll failed on Windows. Fixed by using regular expression for parameter register:

"movswq %{{[cd][xi]}}, %rax" as parameter could be "cx" or "di"
and
"movsbq %{{[cdi]*l}}, %rax" as parameter could be "cl" or "dil"

2. recently modified "vector-sext.ll" also failed as contains appropriate for folding shr/shl combinations.

This case is less obvious as for some reason the folding influence on scheduling pass.
I'm not happy with such dramatic change on scheduler pass, however I believe that should be fixed by another patch, not related to instructions combine. I've also checked the performance of changed function "load_sext_16i1_to_16i16" and it is the same with and without the patch.
I'll submit corresponding bug report after commit.

Is the patch still ok?

[andreadb]

In D13161#266882, @evstupac wrote:

During final check 2 new tests failed on x86:

1. sar_fold64.ll failed on Windows. Fixed by using regular expression for parameter register:

"movswq %{{[cd][xi]}}, %rax" as parameter could be "cx" or "di"
and
"movsbq %{{[cdi]*l}}, %rax" as parameter could be "cl" or "dil"

The easier way to fix those problems is to force a target triple to the tests.
What if you explicitly force -mtriple=x86_64-unknown-unknown to test sar_fold64.ll and -mtriple=i686-unknown-unknown to sar_fold.ll?

About the vector-sext.ll failures,
are those failures only related to the last RUN line in the file (the i686 run line)? Does the problem disappear if you replace -mcpu=i686 with -mcpu=generic ?

[evstupac]

The easier way to fix those problems is to force a target triple to the tests.
What if you explicitly force -mtriple=x86_64-unknown-unknown to test sar_fold64.ll and -mtriple=i686-unknown-unknown to sar_fold.ll?

Yes. This also works.

About the vector-sext.ll failures,
are those failures only related to the last RUN line in the file (the i686 run line)? Does the problem disappear if you replace -mcpu=i686 with -mcpu=generic ?

No. The are related to AVX and AVX2 x86-64 lines. That way replace -mcpu=i686 by -mcpu=generic does not help.

[RKSimon]

OK - it looks like some additional changes are required - comments below. I agree that the scheduling issue isn't directly tied to this patch, but you need to at least create a bugzilla with a minimal repro.

lib/Target/X86/X86ISelLowering.cpp

24400

I think you will need to add a test for N0.hasOneUse() as well here - otherwise there is a likely chance that the SHL will still need to be performed.

test/CodeGen/X86/sar_fold.ll

4

Please can you replace the march with a mtriple?

test/CodeGen/X86/sar_fold64.ll

3

Please can you replace the march with a mtriple?

test/CodeGen/X86/vector-sext.ll

1615

Annotating with nounwind readnone should help here.

[evstupac]

replace march by mtriple in tests
add "nounwind readnone" to "@load_sext_16i1_to_16i16" function in vector-sext.ll test
add "!N0.hasOneUse()" to early exit from the folding

[evstupac]

PING.

[RKSimon]

My concern is the massive increase in register pressure in load_sext_16i1_to_16i16.

We can certainly improve vXi1 -> vXiY sign extension lowering (it should be vectorizable using a broadcast + variable shl/mul + immediate sra) but I'm worried that there will be other similar cases that we just don't see in the tests.

test/CodeGen/X86/sar_fold.ll

2

Remove the -O2

test/CodeGen/X86/sar_fold64.ll

2

Remove the -O2

test/CodeGen/X86/vector-sext.ll

1615

Turns out it didn't help ;-(

[evstupac]

"-O2" removed from sar_fold* tests.
vector-sext updated without "nounwind" attribute

[evstupac]

PING.

[evstupac]

PING.

[nadav]

Did you measure the performance impact of this patch on the llvm test suite (or SPEC or other test suite?). Is this a win?

[evstupac]

Yes. Spec2000 performance is almost flat.
Unit performance tests get up to 40% gain.
The patch fixes regression in PR24373.

[evstupac]

PING.

[nadav]

I did not get a chance to review this patch carefully. Andrea, Simon, David, Elena, Sanjay, did you get a chance to review the patch? Does it look okay? I did not see a LGTM in the thread.

[evstupac]

Simon accepted this patch Oct 11 2015, 7:02 AM with LGTM.
But during the review a test requiring changes was added to LLVM tests. I've fixed it and now waiting for new approve.

[andreadb]

The change looks good to me.

Thanks for measuring the performances after this change.
I agree with you that the poor codegen caused by a suboptimal scheduling of instructions in test 'load_sext_16i1_to_16i16' can be addressed by a later patch.
However, please file a bug for it so that we don't lose track of that problem.

lib/Target/X86/X86ISelLowering.cpp

24427

You can remove this else after return.

[RKSimon]

Can you review the tests please - extra vXi1 sextload tests have been added to vector-sext.ll recently

[evstupac]

In D13161#289205, @RKSimon wrote:

Can you review the tests please - extra vXi1 sextload tests have been added to vector-sext.ll recently

Yes there is valuable change in the test.
I'll update the it before commit.
The issue with scheduling could be hided by adding "-enable-misched=0" option.
Anyway I'm going to file a bug after commit.

[evstupac]

Patch with updated "vector-sext.ll" that will be committed,

[evstupac]

miss new "test/CodeGen/X86/sar_fold.ll" and "test/CodeGen/X86/sar_fold64.ll" while updating patch.

[RKSimon]

LGTM

BTW I have a patch to improve the vXi1 sext codegen, I'll be putting it up for review soon.

lib/Target/X86/X86ISelLowering.cpp

if (auto *N1SplatC = N1BV->getConstantSplatNode()) {
       // of two values.
       if (N1SplatC->getAPIntValue() == 1)
         return DAG.getNode(ISD::ADD, SDLoc(N), VT, N0, N0);
     }
 
   return SDValue();
 }
 
+static SDValue PerformSRACombine(SDNode *N, SelectionDAG &DAG) {
+  SDValue N0 = N->getOperand(0);
+  SDValue N1 = N->getOperand(1);
+  EVT VT = N0.getValueType();
+  unsigned Size = VT.getSizeInBits();
+
+  // fold (ashr (shl, a, [56,48,32,24,16]), SarConst)
+  // into (shl, (sext (a), [56,48,32,24,16] - SarConst)) or
+  // into (lshr, (sext (a), SarConst - [56,48,32,24,16]))
+  // depending on sign of (SarConst - [56,48,32,24,16])
+
+  // sexts in X86 are MOVs. The MOVs have the same code size
+  // as above SHIFTs (only SHIFT on 1 has lower code size).
+  // However the MOVs have 2 advantages to a SHIFT:
+  // 1. MOVs can write to a register that differs from source
+  // 2. MOVs accept memory operands
+
+  if (!VT.isInteger() || VT.isVector() || N1.getOpcode() != ISD::Constant ||
+      N0.getOpcode() != ISD::SHL || !N0.hasOneUse() ||
+      N0.getOperand(1).getOpcode() != ISD::Constant)
+    return SDValue();
+
+  SDValue N00 = N0.getOperand(0);
+  SDValue N01 = N0.getOperand(1);
+  APInt ShlConst = (cast<ConstantSDNode>(N01))->getAPIntValue();
+  APInt SarConst = (cast<ConstantSDNode>(N1))->getAPIntValue();
+  EVT CVT = N1.getValueType();
+
+  if (SarConst.isNegative())
+    return SDValue();
+
+  for (MVT SVT : MVT::integer_valuetypes()) {
+    unsigned ShiftSize = SVT.getSizeInBits();
+    // skipping types without corresponding sext/zext and
+    // ShlConst that is not one of [56,48,32,24,16]
+    if (ShiftSize < 8 || ShiftSize > 64 || ShlConst != Size - ShiftSize)
+      continue;
+    SDLoc DL(N);
+    SDValue NN =
+        DAG.getNode(ISD::SIGN_EXTEND_INREG, DL, VT, N00, DAG.getValueType(SVT));
+    SarConst = SarConst - (Size - ShiftSize);
+    if (SarConst == 0)
+      return NN;
+    else if (SarConst.isNegative())
+      return DAG.getNode(ISD::SHL, DL, VT, NN,
+                         DAG.getConstant(-SarConst, DL, CVT));
+    else
+      return DAG.getNode(ISD::SRA, DL, VT, NN,
+                         DAG.getConstant(SarConst, DL, CVT));
+  }
+  return SDValue();
+}
+
 /// \brief Returns a vector of 0s if the node in input is a vector logical
 /// shift by a constant amount which is known to be bigger than or equal
 /// to the vector element size in bits.
 static SDValue performShiftToAllZeros(SDNode *N, SelectionDAG &DAG,
                                       const X86Subtarget *Subtarget) {
   EVT VT = N->getValueType(0);
 
   if (VT != MVT::v2i64 && VT != MVT::v4i32 && VT != MVT::v8i16 &&

 /// PerformShiftCombine - Combine shifts.
 static SDValue PerformShiftCombine(SDNode* N, SelectionDAG &DAG,
                                    TargetLowering::DAGCombinerInfo &DCI,
                                    const X86Subtarget *Subtarget) {
   if (N->getOpcode() == ISD::SHL)
     if (SDValue V = PerformSHLCombine(N, DAG))
       return V;
 
+  if (N->getOpcode() == ISD::SRA)
+    if (SDValue V = PerformSRACombine(N, DAG))
+      return V;
+
   // Try to fold this logical shift into a zero vector.
   if (N->getOpcode() != ISD::SRA)
     if (SDValue V = performShiftToAllZeros(N, DAG, Subtarget))
       return V;
 
   return SDValue();
 }
 

test/CodeGen/X86/2009-05-23-dagcombine-shifts.ll

 ; RUN: llc < %s | FileCheck %s
 
 ; Check that the shr(shl X, 56), 48) is not mistakenly turned into
 ; a shr (X, -8) that gets subsequently "optimized away" as undef
 ; PR4254
 
+; after fixing PR24373
+; shlq $56, %rdi
+; sarq $48, %rdi
+; folds into
+; movsbq %dil, %rax
+; shlq $8, %rax
+; which is better for x86
+
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-f80:128:128"
 target triple = "x86_64-unknown-linux-gnu"
 
 define i64 @foo(i64 %b) nounwind readnone {
 entry:
 ; CHECK-LABEL: foo:
-; CHECK: shlq $56, %rdi
-; CHECK: sarq $48, %rdi
-; CHECK: leaq 1(%rdi), %rax
+; CHECK: movsbq %dil, %rax
+; CHECK: shlq $8, %rax
+; CHECK: orq $1, %rax
   %shl = shl i64 %b, 56   ; <i64> [#uses=1]
   %shr = ashr i64 %shl, 48    ; <i64> [#uses=1]
   %add5 = or i64 %shr, 1    ; <i64> [#uses=1]
   ret i64 %add5
 }

test/CodeGen/X86/sar_fold.ll

+; RUN: llc < %s -mtriple=i686-unknown-unknown | FileCheck %s
+
+define i32 @shl16sar15(i32 %a) #0 {
+; CHECK-LABEL: shl16sar15:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movswl {{[0-9]+}}(%esp), %eax
+  %1 = shl i32 %a, 16
+  %2 = ashr exact i32 %1, 15
+  ret i32 %2
+}
+
+define i32 @shl16sar17(i32 %a) #0 {
+; CHECK-LABEL: shl16sar17:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movswl {{[0-9]+}}(%esp), %eax
+  %1 = shl i32 %a, 16
+  %2 = ashr exact i32 %1, 17
+  ret i32 %2
+}
+
+define i32 @shl24sar23(i32 %a) #0 {
+; CHECK-LABEL: shl24sar23:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movsbl {{[0-9]+}}(%esp), %eax
+  %1 = shl i32 %a, 24
+  %2 = ashr exact i32 %1, 23
+  ret i32 %2
+}
+
+define i32 @shl24sar25(i32 %a) #0 {
+; CHECK-LABEL: shl24sar25:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movsbl {{[0-9]+}}(%esp), %eax
+  %1 = shl i32 %a, 24
+  %2 = ashr exact i32 %1, 25
+  ret i32 %2
+}

test/CodeGen/X86/sar_fold64.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-unknown | FileCheck %s
+
+define i32 @shl48sar47(i64 %a) #0 {
+; CHECK-LABEL: shl48sar47:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movswq %di, %rax
+  %1 = shl i64 %a, 48
+  %2 = ashr exact i64 %1, 47
+  %3 = trunc i64 %2 to i32
+  ret i32 %3
+}
+
+define i32 @shl48sar49(i64 %a) #0 {
+; CHECK-LABEL: shl48sar49:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movswq %di, %rax
+  %1 = shl i64 %a, 48
+  %2 = ashr exact i64 %1, 49
+  %3 = trunc i64 %2 to i32
+  ret i32 %3
+}
+
+define i32 @shl56sar55(i64 %a) #0 {
+; CHECK-LABEL: shl56sar55:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movsbq %dil, %rax
+  %1 = shl i64 %a, 56
+  %2 = ashr exact i64 %1, 55
+  %3 = trunc i64 %2 to i32
+  ret i32 %3
+}
+
+define i32 @shl56sar57(i64 %a) #0 {
+; CHECK-LABEL: shl56sar57:
+; CHECK:       # BB#0:
+; CHECK-NEXT:    movsbq %dil, %rax
+  %1 = shl i64 %a, 56
+  %2 = ashr exact i64 %1, 57
+  %3 = trunc i64 %2 to i32
+  ret i32 %3
+}
+
+attributes #0 = { nounwind }

test/CodeGen/X86/vector-sext.ll

  %X = load <4 x i8>, <4 x i8>* %ptr
  %Y = sext <4 x i8> %X to <4 x i64>
  ret <4 x i64> %Y
 }
 
 define <8 x i16> @load_sext_8i1_to_8i16(<8 x i1> *%ptr) {
 ; SSE2-LABEL: load_sext_8i1_to_8i16:
 ; SSE2:       # BB#0: # %entry
-; SSE2-NEXT:    movzbl (%rdi), %eax
+; SSE2-NEXT:    movsbq (%rdi), %rax
 ; SSE2-NEXT:    movq %rax, %rcx
-; SSE2-NEXT:    shlq $56, %rcx
-; SSE2-NEXT:    sarq $63, %rcx
+; SSE2-NEXT:    shrq $7, %rcx
 ; SSE2-NEXT:    movd %ecx, %xmm0
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shlq $60, %rcx
 ; SSE2-NEXT:    sarq $63, %rcx
 ; SSE2-NEXT:    movd %ecx, %xmm2
 ; SSE2-NEXT:    punpcklwd {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1],xmm2[2],xmm0[2],xmm2[3],xmm0[3]
 ; SSE2-NEXT:    movq %rax, %rcx
 ; SSE2-NEXT:    shlq $58, %rcx

 ; SSE2-NEXT:    movd %eax, %xmm0
 ; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1],xmm0[2],xmm2[2],xmm0[3],xmm2[3]
 ; SSE2-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
 ; SSE2-NEXT:    retq
 ;
 ; SSSE3-LABEL: load_sext_8i1_to_8i16:
 ; SSSE3:       # BB#0: # %entry
-; SSSE3-NEXT:    movzbl (%rdi), %eax
+; SSSE3-NEXT:    movsbq (%rdi), %rax
 ; SSSE3-NEXT:    movq %rax, %rcx
-; SSSE3-NEXT:    shlq $56, %rcx
-; SSSE3-NEXT:    sarq $63, %rcx
+; SSSE3-NEXT:    shrq $7, %rcx
 ; SSSE3-NEXT:    movd %ecx, %xmm0
 ; SSSE3-NEXT:    movq %rax, %rcx
 ; SSSE3-NEXT:    shlq $60, %rcx
 ; SSSE3-NEXT:    sarq $63, %rcx
 ; SSSE3-NEXT:    movd %ecx, %xmm2
 ; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1],xmm2[2],xmm0[2],xmm2[3],xmm0[3]
 ; SSSE3-NEXT:    movq %rax, %rcx
 ; SSSE3-NEXT:    shlq $58, %rcx

 ; SSSE3-NEXT:    movd %eax, %xmm0
 ; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm3[0],xmm0[1],xmm3[1],xmm0[2],xmm3[2],xmm0[3],xmm3[3]
 ; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm2[0],xmm0[1],xmm2[1],xmm0[2],xmm2[2],xmm0[3],xmm2[3]
 ; SSSE3-NEXT:    punpcklwd {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3]
 ; SSSE3-NEXT:    retq
 ;
 ; SSE41-LABEL: load_sext_8i1_to_8i16:
 ; SSE41:       # BB#0: # %entry
-; SSE41-NEXT:    movzbl (%rdi), %eax
+; SSE41-NEXT:    movsbq (%rdi), %rax
 ; SSE41-NEXT:    movq %rax, %rcx
 ; SSE41-NEXT:    shlq $62, %rcx
 ; SSE41-NEXT:    sarq $63, %rcx
 ; SSE41-NEXT:    movq %rax, %rdx
 ; SSE41-NEXT:    shlq $63, %rdx
 ; SSE41-NEXT:    sarq $63, %rdx
 ; SSE41-NEXT:    movd %edx, %xmm0
 ; SSE41-NEXT:    pinsrw $1, %ecx, %xmm0

 ; SSE41-NEXT:    movq %rax, %rcx
 ; SSE41-NEXT:    shlq $58, %rcx
 ; SSE41-NEXT:    sarq $63, %rcx
 ; SSE41-NEXT:    pinsrw $5, %ecx, %xmm0
 ; SSE41-NEXT:    movq %rax, %rcx
 ; SSE41-NEXT:    shlq $57, %rcx
 ; SSE41-NEXT:    sarq $63, %rcx
 ; SSE41-NEXT:    pinsrw $6, %ecx, %xmm0
-; SSE41-NEXT:    shlq $56, %rax
-; SSE41-NEXT:    sarq $63, %rax
+; SSE41-NEXT:    shrq $7, %rax
 ; SSE41-NEXT:    pinsrw $7, %eax, %xmm0
 ; SSE41-NEXT:    retq
 ;
 ; AVX-LABEL: load_sext_8i1_to_8i16:
 ; AVX:       # BB#0: # %entry
-; AVX-NEXT:    movzbl (%rdi), %eax
+; AVX-NEXT:    movsbq (%rdi), %rax
 ; AVX-NEXT:    movq %rax, %rcx
 ; AVX-NEXT:    shlq $62, %rcx
 ; AVX-NEXT:    sarq $63, %rcx
 ; AVX-NEXT:    movq %rax, %rdx
 ; AVX-NEXT:    shlq $63, %rdx
 ; AVX-NEXT:    sarq $63, %rdx
 ; AVX-NEXT:    vmovd %edx, %xmm0
 ; AVX-NEXT:    vpinsrw $1, %ecx, %xmm0, %xmm0

 ; AVX-NEXT:    movq %rax, %rcx
 ; AVX-NEXT:    shlq $58, %rcx
 ; AVX-NEXT:    sarq $63, %rcx
 ; AVX-NEXT:    vpinsrw $5, %ecx, %xmm0, %xmm0
 ; AVX-NEXT:    movq %rax, %rcx
 ; AVX-NEXT:    shlq $57, %rcx
 ; AVX-NEXT:    sarq $63, %rcx
 ; AVX-NEXT:    vpinsrw $6, %ecx, %xmm0, %xmm0
-; AVX-NEXT:    shlq $56, %rax
-; AVX-NEXT:    sarq $63, %rax
+; AVX-NEXT:    shrq $7, %rax
 ; AVX-NEXT:    vpinsrw $7, %eax, %xmm0, %xmm0
 ; AVX-NEXT:    retq
 ;
 ; X32-SSE41-LABEL: load_sext_8i1_to_8i16:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
-; X32-SSE41-NEXT:    movl (%eax), %eax
+; X32-SSE41-NEXT:    movsbl (%eax), %eax
 ; X32-SSE41-NEXT:    movl %eax, %ecx
 ; X32-SSE41-NEXT:    shll $30, %ecx
 ; X32-SSE41-NEXT:    sarl $31, %ecx
 ; X32-SSE41-NEXT:    movl %eax, %edx
 ; X32-SSE41-NEXT:    shll $31, %edx
 ; X32-SSE41-NEXT:    sarl $31, %edx
 ; X32-SSE41-NEXT:    movd %edx, %xmm0
 ; X32-SSE41-NEXT:    pinsrw $1, %ecx, %xmm0

 ; X32-SSE41-NEXT:    movl %eax, %ecx
 ; X32-SSE41-NEXT:    shll $26, %ecx
 ; X32-SSE41-NEXT:    sarl $31, %ecx
 ; X32-SSE41-NEXT:    pinsrw $5, %ecx, %xmm0
 ; X32-SSE41-NEXT:    movl %eax, %ecx
 ; X32-SSE41-NEXT:    shll $25, %ecx
 ; X32-SSE41-NEXT:    sarl $31, %ecx
 ; X32-SSE41-NEXT:    pinsrw $6, %ecx, %xmm0
-; X32-SSE41-NEXT:    shll $24, %eax
-; X32-SSE41-NEXT:    sarl $31, %eax
+; X32-SSE41-NEXT:    shrl $7, %eax
 ; X32-SSE41-NEXT:    pinsrw $7, %eax, %xmm0
 ; X32-SSE41-NEXT:    retl
 entry:
  %X = load <8 x i1>, <8 x i1>* %ptr
  %Y = sext <8 x i1> %X to <8 x i16>
  ret <8 x i16> %Y
 }
 

 ; SSE41-NEXT:    psrad $31, %xmm0
 ; SSE41-NEXT:    punpckhwd {{.*#+}} xmm1 = xmm1[4,4,5,5,6,6,7,7]
 ; SSE41-NEXT:    pslld $31, %xmm1
 ; SSE41-NEXT:    psrad $31, %xmm1
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: load_sext_8i1_to_8i32:
 ; AVX1:       # BB#0: # %entry
-; AVX1-NEXT:    movzbl (%rdi), %eax
+; AVX1-NEXT:    movsbq (%rdi), %rax
 ; AVX1-NEXT:    movq %rax, %rcx
 ; AVX1-NEXT:    shlq $58, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    movq %rax, %rdx
 ; AVX1-NEXT:    shlq $59, %rdx
 ; AVX1-NEXT:    sarq $63, %rdx
 ; AVX1-NEXT:    vmovd %edx, %xmm0
 ; AVX1-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
 ; AVX1-NEXT:    movq %rax, %rcx
 ; AVX1-NEXT:    shlq $57, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    vpinsrd $2, %ecx, %xmm0, %xmm0
 ; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $56, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
+; AVX1-NEXT:    shrq $7, %rcx
 ; AVX1-NEXT:    vpinsrd $3, %ecx, %xmm0, %xmm0
 ; AVX1-NEXT:    movq %rax, %rcx
 ; AVX1-NEXT:    shlq $62, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    movq %rax, %rdx
 ; AVX1-NEXT:    shlq $63, %rdx
 ; AVX1-NEXT:    sarq $63, %rdx
 ; AVX1-NEXT:    vmovd %edx, %xmm1
 ; AVX1-NEXT:    vpinsrd $1, %ecx, %xmm1, %xmm1
 ; AVX1-NEXT:    movq %rax, %rcx
 ; AVX1-NEXT:    shlq $61, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    vpinsrd $2, %ecx, %xmm1, %xmm1
 ; AVX1-NEXT:    shlq $60, %rax
 ; AVX1-NEXT:    sarq $63, %rax
 ; AVX1-NEXT:    vpinsrd $3, %eax, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: load_sext_8i1_to_8i32:
 ; AVX2:       # BB#0: # %entry
-; AVX2-NEXT:    movzbl (%rdi), %eax
+; AVX2-NEXT:    movsbq (%rdi), %rax
 ; AVX2-NEXT:    movq %rax, %rcx
 ; AVX2-NEXT:    shlq $58, %rcx
 ; AVX2-NEXT:    sarq $63, %rcx
 ; AVX2-NEXT:    movq %rax, %rdx
 ; AVX2-NEXT:    shlq $59, %rdx
 ; AVX2-NEXT:    sarq $63, %rdx
 ; AVX2-NEXT:    vmovd %edx, %xmm0
 ; AVX2-NEXT:    vpinsrd $1, %ecx, %xmm0, %xmm0
 ; AVX2-NEXT:    movq %rax, %rcx
 ; AVX2-NEXT:    shlq $57, %rcx
 ; AVX2-NEXT:    sarq $63, %rcx
 ; AVX2-NEXT:    vpinsrd $2, %ecx, %xmm0, %xmm0
 ; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $56, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
+; AVX2-NEXT:    shrq $7, %rcx
 ; AVX2-NEXT:    vpinsrd $3, %ecx, %xmm0, %xmm0
 ; AVX2-NEXT:    movq %rax, %rcx
 ; AVX2-NEXT:    shlq $62, %rcx
 ; AVX2-NEXT:    sarq $63, %rcx
 ; AVX2-NEXT:    movq %rax, %rdx
 ; AVX2-NEXT:    shlq $63, %rdx
 ; AVX2-NEXT:    sarq $63, %rdx
 ; AVX2-NEXT:    vmovd %edx, %xmm1

 ; X32-SSE41-NEXT:    pmovsxbd 4(%eax), %xmm1
 ; X32-SSE41-NEXT:    retl
 entry:
  %X = load <8 x i8>, <8 x i8>* %ptr
  %Y = sext <8 x i8> %X to <8 x i32>
  ret <8 x i32> %Y
 }
 
 define <16 x i16> @load_sext_16i1_to_16i16(<16 x i1> *%ptr) {
 ; SSE2-LABEL: load_sext_16i1_to_16i16:
 ; SSE2:       # BB#0: # %entry
 ; SSE2-NEXT:    movzwl (%rdi), %eax
 ; SSE2-NEXT:    movl %eax, %ecx
 ; SSE2-NEXT:    shrl $14, %ecx
 ; SSE2-NEXT:    andl $1, %ecx
 ; SSE2-NEXT:    movd %ecx, %xmm0
 ; SSE2-NEXT:    movl %eax, %ecx

 ; SSE41-NEXT:    psraw $15, %xmm0
 ; SSE41-NEXT:    punpckhbw {{.*#+}} xmm1 = xmm1[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
 ; SSE41-NEXT:    psllw $15, %xmm1
 ; SSE41-NEXT:    psraw $15, %xmm1
 ; SSE41-NEXT:    retq
 ;
 ; AVX1-LABEL: load_sext_16i1_to_16i16:
 ; AVX1:       # BB#0: # %entry
-; AVX1-NEXT:    movzwl (%rdi), %eax
+; AVX1-NEXT:    pushq %rbp
+; AVX1-NEXT:  .Ltmp0:
+; AVX1-NEXT:    .cfi_def_cfa_offset 16
+; AVX1-NEXT:    pushq %r15
+; AVX1-NEXT:  .Ltmp1:
+; AVX1-NEXT:    .cfi_def_cfa_offset 24
+; AVX1-NEXT:    pushq %r14
+; AVX1-NEXT:  .Ltmp2:
+; AVX1-NEXT:    .cfi_def_cfa_offset 32
+; AVX1-NEXT:    pushq %r13
+; AVX1-NEXT:  .Ltmp3:
+; AVX1-NEXT:    .cfi_def_cfa_offset 40
+; AVX1-NEXT:    pushq %r12
+; AVX1-NEXT:  .Ltmp4:
+; AVX1-NEXT:    .cfi_def_cfa_offset 48
+; AVX1-NEXT:    pushq %rbx
+; AVX1-NEXT:  .Ltmp5:
+; AVX1-NEXT:    .cfi_def_cfa_offset 56
+; AVX1-NEXT:  .Ltmp6:
+; AVX1-NEXT:    .cfi_offset %rbx, -56
+; AVX1-NEXT:  .Ltmp7:
+; AVX1-NEXT:    .cfi_offset %r12, -48
+; AVX1-NEXT:  .Ltmp8:
+; AVX1-NEXT:    .cfi_offset %r13, -40
+; AVX1-NEXT:  .Ltmp9:
+; AVX1-NEXT:    .cfi_offset %r14, -32
+; AVX1-NEXT:  .Ltmp10:
+; AVX1-NEXT:    .cfi_offset %r15, -24
+; AVX1-NEXT:  .Ltmp11:
+; AVX1-NEXT:    .cfi_offset %rbp, -16
+; AVX1-NEXT:    movswq (%rdi), %rax
 ; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $54, %rcx
+; AVX1-NEXT:    shlq $55, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    movq %rax, %rdx
-; AVX1-NEXT:    shlq $55, %rdx
-; AVX1-NEXT:    sarq $63, %rdx
-; AVX1-NEXT:    vmovd %edx, %xmm0
-; AVX1-NEXT:    vpinsrw $1, %ecx, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %ecx, %xmm0
+; AVX1-NEXT:    movq %rax, %r8
+; AVX1-NEXT:    movq %rax, %r10
+; AVX1-NEXT:    movq %rax, %r11
+; AVX1-NEXT:    movq %rax, %r14
+; AVX1-NEXT:    movq %rax, %r15
+; AVX1-NEXT:    movq %rax, %r9
+; AVX1-NEXT:    movq %rax, %r12
+; AVX1-NEXT:    movq %rax, %r13
+; AVX1-NEXT:    movq %rax, %rbx
+; AVX1-NEXT:    movq %rax, %rdi
 ; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $53, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $2, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $52, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $3, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $51, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $4, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $50, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $5, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $49, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $6, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $48, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $7, %ecx, %xmm0, %xmm0
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $62, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    movq %rax, %rdx
-; AVX1-NEXT:    shlq $63, %rdx
-; AVX1-NEXT:    sarq $63, %rdx
-; AVX1-NEXT:    vmovd %edx, %xmm1
-; AVX1-NEXT:    vpinsrw $1, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $61, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $2, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $60, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $3, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    movq %rax, %rcx
+; AVX1-NEXT:    movq %rax, %rsi
+; AVX1-NEXT:    movsbq %al, %rbp
+; AVX1-NEXT:    shlq $54, %rax
+; AVX1-NEXT:    sarq $63, %rax
+; AVX1-NEXT:    vpinsrw $1, %eax, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $53, %r8
+; AVX1-NEXT:    sarq $63, %r8
+; AVX1-NEXT:    vpinsrw $2, %r8d, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $52, %r10
+; AVX1-NEXT:    sarq $63, %r10
+; AVX1-NEXT:    vpinsrw $3, %r10d, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $51, %r11
+; AVX1-NEXT:    sarq $63, %r11
+; AVX1-NEXT:    vpinsrw $4, %r11d, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $50, %r14
+; AVX1-NEXT:    sarq $63, %r14
+; AVX1-NEXT:    vpinsrw $5, %r14d, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $49, %r15
+; AVX1-NEXT:    sarq $63, %r15
+; AVX1-NEXT:    vpinsrw $6, %r15d, %xmm0, %xmm0
+; AVX1-NEXT:    shrq $15, %r9
+; AVX1-NEXT:    vpinsrw $7, %r9d, %xmm0, %xmm0
+; AVX1-NEXT:    shlq $63, %r13
+; AVX1-NEXT:    sarq $63, %r13
+; AVX1-NEXT:    vmovd %r13d, %xmm1
+; AVX1-NEXT:    shlq $62, %r12
+; AVX1-NEXT:    sarq $63, %r12
+; AVX1-NEXT:    vpinsrw $1, %r12d, %xmm1, %xmm1
+; AVX1-NEXT:    shlq $61, %rbx
+; AVX1-NEXT:    sarq $63, %rbx
+; AVX1-NEXT:    vpinsrw $2, %ebx, %xmm1, %xmm1
+; AVX1-NEXT:    shlq $60, %rdi
+; AVX1-NEXT:    sarq $63, %rdi
+; AVX1-NEXT:    vpinsrw $3, %edi, %xmm1, %xmm1
 ; AVX1-NEXT:    shlq $59, %rcx
 ; AVX1-NEXT:    sarq $63, %rcx
 ; AVX1-NEXT:    vpinsrw $4, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $58, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $5, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    movq %rax, %rcx
-; AVX1-NEXT:    shlq $57, %rcx
-; AVX1-NEXT:    sarq $63, %rcx
-; AVX1-NEXT:    vpinsrw $6, %ecx, %xmm1, %xmm1
-; AVX1-NEXT:    shlq $56, %rax
-; AVX1-NEXT:    sarq $63, %rax
-; AVX1-NEXT:    vpinsrw $7, %eax, %xmm1, %xmm1
+; AVX1-NEXT:    shlq $58, %rdx
+; AVX1-NEXT:    sarq $63, %rdx
+; AVX1-NEXT:    vpinsrw $5, %edx, %xmm1, %xmm1
+; AVX1-NEXT:    shlq $57, %rsi
+; AVX1-NEXT:    sarq $63, %rsi
+; AVX1-NEXT:    vpinsrw $6, %esi, %xmm1, %xmm1
+; AVX1-NEXT:    shrq $7, %rbp
+; AVX1-NEXT:    vpinsrw $7, %ebp, %xmm1, %xmm1
 ; AVX1-NEXT:    vinsertf128 $1, %xmm0, %ymm1, %ymm0
+; AVX1-NEXT:    popq %rbx
+; AVX1-NEXT:    popq %r12
+; AVX1-NEXT:    popq %r13
+; AVX1-NEXT:    popq %r14
+; AVX1-NEXT:    popq %r15
+; AVX1-NEXT:    popq %rbp
 ; AVX1-NEXT:    retq
 ;
 ; AVX2-LABEL: load_sext_16i1_to_16i16:
 ; AVX2:       # BB#0: # %entry
-; AVX2-NEXT:    movzwl (%rdi), %eax
+; AVX2-NEXT:    pushq %rbp
+; AVX2-NEXT:  .Ltmp0:
+; AVX2-NEXT:    .cfi_def_cfa_offset 16
+; AVX2-NEXT:    pushq %r15
+; AVX2-NEXT:  .Ltmp1:
+; AVX2-NEXT:    .cfi_def_cfa_offset 24
+; AVX2-NEXT:    pushq %r14
+; AVX2-NEXT:  .Ltmp2:
+; AVX2-NEXT:    .cfi_def_cfa_offset 32
+; AVX2-NEXT:    pushq %r13
+; AVX2-NEXT:  .Ltmp3:
+; AVX2-NEXT:    .cfi_def_cfa_offset 40
+; AVX2-NEXT:    pushq %r12
+; AVX2-NEXT:  .Ltmp4:
+; AVX2-NEXT:    .cfi_def_cfa_offset 48
+; AVX2-NEXT:    pushq %rbx
+; AVX2-NEXT:  .Ltmp5:
+; AVX2-NEXT:    .cfi_def_cfa_offset 56
+; AVX2-NEXT:  .Ltmp6:
+; AVX2-NEXT:    .cfi_offset %rbx, -56
+; AVX2-NEXT:  .Ltmp7:
+; AVX2-NEXT:    .cfi_offset %r12, -48
+; AVX2-NEXT:  .Ltmp8:
+; AVX2-NEXT:    .cfi_offset %r13, -40
+; AVX2-NEXT:  .Ltmp9:
+; AVX2-NEXT:    .cfi_offset %r14, -32
+; AVX2-NEXT:  .Ltmp10:
+; AVX2-NEXT:    .cfi_offset %r15, -24
+; AVX2-NEXT:  .Ltmp11:
+; AVX2-NEXT:    .cfi_offset %rbp, -16
+; AVX2-NEXT:    movswq (%rdi), %rax
+; AVX2-NEXT:    movq %rax, %rcx
+; AVX2-NEXT:    shlq $55, %rcx
+; AVX2-NEXT:    sarq $63, %rcx
+; AVX2-NEXT:    vmovd %ecx, %xmm0
+; AVX2-NEXT:    movq %rax, %r8
+; AVX2-NEXT:    movq %rax, %r10
+; AVX2-NEXT:    movq %rax, %r11
+; AVX2-NEXT:    movq %rax, %r14
+; AVX2-NEXT:    movq %rax, %r15
+; AVX2-NEXT:    movq %rax, %r9
+; AVX2-NEXT:    movq %rax, %r12
+; AVX2-NEXT:    movq %rax, %r13
+; AVX2-NEXT:    movq %rax, %rbx
+; AVX2-NEXT:    movq %rax, %rdi
 ; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $54, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
 ; AVX2-NEXT:    movq %rax, %rdx
-; AVX2-NEXT:    shlq $55, %rdx
-; AVX2-NEXT:    sarq $63, %rdx
-; AVX2-NEXT:    vmovd %edx, %xmm0
-; AVX2-NEXT:    vpinsrw $1, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $53, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $2, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $52, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $3, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $51, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $4, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $50, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $5, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $49, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $6, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $48, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $7, %ecx, %xmm0, %xmm0
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $62, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    movq %rax, %rdx
-; AVX2-NEXT:    shlq $63, %rdx
-; AVX2-NEXT:    sarq $63, %rdx
-; AVX2-NEXT:    vmovd %edx, %xmm1
-; AVX2-NEXT:    vpinsrw $1, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $61, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $2, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $60, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $3, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    movq %rax, %rcx
+; AVX2-NEXT:    movq %rax, %rsi
+; AVX2-NEXT:    movsbq %al, %rbp
+; AVX2-NEXT:    shlq $54, %rax
+; AVX2-NEXT:    sarq $63, %rax
+; AVX2-NEXT:    vpinsrw $1, %eax, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $53, %r8
+; AVX2-NEXT:    sarq $63, %r8
+; AVX2-NEXT:    vpinsrw $2, %r8d, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $52, %r10
+; AVX2-NEXT:    sarq $63, %r10
+; AVX2-NEXT:    vpinsrw $3, %r10d, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $51, %r11
+; AVX2-NEXT:    sarq $63, %r11
+; AVX2-NEXT:    vpinsrw $4, %r11d, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $50, %r14
+; AVX2-NEXT:    sarq $63, %r14
+; AVX2-NEXT:    vpinsrw $5, %r14d, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $49, %r15
+; AVX2-NEXT:    sarq $63, %r15
+; AVX2-NEXT:    vpinsrw $6, %r15d, %xmm0, %xmm0
+; AVX2-NEXT:    shrq $15, %r9
+; AVX2-NEXT:    vpinsrw $7, %r9d, %xmm0, %xmm0
+; AVX2-NEXT:    shlq $63, %r13
+; AVX2-NEXT:    sarq $63, %r13
+; AVX2-NEXT:    vmovd %r13d, %xmm1
+; AVX2-NEXT:    shlq $62, %r12
+; AVX2-NEXT:    sarq $63, %r12
+; AVX2-NEXT:    vpinsrw $1, %r12d, %xmm1, %xmm1
+; AVX2-NEXT:    shlq $61, %rbx
+; AVX2-NEXT:    sarq $63, %rbx
+; AVX2-NEXT:    vpinsrw $2, %ebx, %xmm1, %xmm1
+; AVX2-NEXT:    shlq $60, %rdi
+; AVX2-NEXT:    sarq $63, %rdi
+; AVX2-NEXT:    vpinsrw $3, %edi, %xmm1, %xmm1
 ; AVX2-NEXT:    shlq $59, %rcx
 ; AVX2-NEXT:    sarq $63, %rcx
 ; AVX2-NEXT:    vpinsrw $4, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $58, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $5, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    movq %rax, %rcx
-; AVX2-NEXT:    shlq $57, %rcx
-; AVX2-NEXT:    sarq $63, %rcx
-; AVX2-NEXT:    vpinsrw $6, %ecx, %xmm1, %xmm1
-; AVX2-NEXT:    shlq $56, %rax
-; AVX2-NEXT:    sarq $63, %rax
-; AVX2-NEXT:    vpinsrw $7, %eax, %xmm1, %xmm1
+; AVX2-NEXT:    shlq $58, %rdx
+; AVX2-NEXT:    sarq $63, %rdx
+; AVX2-NEXT:    vpinsrw $5, %edx, %xmm1, %xmm1
+; AVX2-NEXT:    shlq $57, %rsi
+; AVX2-NEXT:    sarq $63, %rsi
+; AVX2-NEXT:    vpinsrw $6, %esi, %xmm1, %xmm1
+; AVX2-NEXT:    shrq $7, %rbp
+; AVX2-NEXT:    vpinsrw $7, %ebp, %xmm1, %xmm1
 ; AVX2-NEXT:    vinserti128 $1, %xmm0, %ymm1, %ymm0
+; AVX2-NEXT:    popq %rbx
+; AVX2-NEXT:    popq %r12
+; AVX2-NEXT:    popq %r13
+; AVX2-NEXT:    popq %r14
+; AVX2-NEXT:    popq %r15
+; AVX2-NEXT:    popq %rbp
 ; AVX2-NEXT:    retq
 ;
 ; X32-SSE41-LABEL: load_sext_16i1_to_16i16:
 ; X32-SSE41:       # BB#0: # %entry
 ; X32-SSE41-NEXT:    movl {{[0-9]+}}(%esp), %eax
 ; X32-SSE41-NEXT:    movzwl (%eax), %eax
 ; X32-SSE41-NEXT:    movl %eax, %ecx
 ; X32-SSE41-NEXT:    shrl %ecx