[X86][SSE] Vectorized i8 and i16 shift operators

[RKSimon]

Summary

This patch ensures that SHL/SRL/SRA shifts for i8 and i16 vectors avoid scalarization. It builds on the existing i8 SHL vectorized implementation of moving the shift bits up to the sign bit position and separating the 4, 2 & 1 bit shifts with several improvements:

1 - SSE41 targets can use (v)pblendvb directly with the sign bit instead of performing a comparison to feed into a VSELECT node.
2 - pre-SSE41 targets were masking + comparing with an 0x80 constant - we avoid this by using the fact that a set sign bit means a negative integer which can be compared against zero to then feed into VSELECT, avoiding the need for a constant mask (zero generation is much cheaper).
3 - SRA i8 needs to be unpacked to the upper byte of a i16 so that the i16 psraw instruction can be correctly used for sign extension - we have to do more work than for SHL/SRL but perf tests indicate that this is still beneficial.

The i16 implementation is similar but simpler than for i8 - we have to do 8, 4, 2 & 1 bit shifts but less shift masking is involved. SSE41 use of (v)pblendvb requires that the i16 shift amount is splatted to both bytes however.

I've updated the vectorization costs as best as I understand them. I'd like to know whether I should be providing SSE41 and AVX1 cost tables as well as these both deviate from the SSE2 version (use of (v)pblendvb and vex-encoded independent destination registers) - there appears to be very little testing of anything other than SSE2 so some advice would be helpful here.

Tested on SSE2, SSE41 and AVX machines.

[RKSimon]

ping

[RKSimon]

ping^2

[delena]

test/CodeGen/X86/avx2-vector-shifts.ll

541

You can extend to 2 vectors of <8 x i32> and use variable shift left VPSLLVD.

test/CodeGen/X86/vec_shift8.ll

202

extend to 2 vectors of <4 x i32> and use PSRAD instruction

[RKSimon]

Thanks Elena, I'll get a fix for the AVX2 shifts done but can you explain the PSRAD approach please?

test/CodeGen/X86/avx2-vector-shifts.ll

541

No problem - I'll adapt the existing 8i16 AVX2 shift code.

test/CodeGen/X86/vec_shift8.ll

202

Can you explain this? I'm not sure what PSRAD will give us.

[escha]

Is it possible to do an i8 variable shift as follows? pseudocode (in0 = value being shifted, in1 = shift amounts):

vec8_multiplies: [1, 2, 4, 8, 16, 32, 64, 128]

pshufb tmp1, [vec8_multiplies], in1
pmovzxbw tmp2, in0
pmovzxbw tmp3, tmp1
pmullw tmp2, tmp3

or something like that? would it be faster, and could something of the sort maybe be doable for right shifts (multiply, then shift back down?) i16 might also be doable with appropriate hacks...

[RKSimon]

Hi Fiona,

I did look at implementing more of the shifts as multiplies, but we end up using a lot of instructions (+ registers) for the extension/truncation stages. It is easier with SSE41 instructions, but at that point we can use PBLENDVB which gives a much tighter solution. It might still be worthwhile for constant shift amounts but that would be a later patch.

[RKSimon]

I've added AVX2 optimizations for v16i16 shifts - I investigated using extend/shift/truncation but the unpack/shift/pack approach I went with required fewer instructions.

[delena]

Hi,

I think you can commit all shifts v8i16 and v16i16 for AVX2.
I don't have time to check all other sequences.

• Elena

[RKSimon]

Thanks Elena, I've committed the AVX2 i16 shift lowering in rL238149. Updated this patch with just the pre-AVX2 lowering patterns.

[RKSimon]

ping

[RKSimon]

Ping - rebased + updated to use the new SelectionDAG::getBitcast helper function

[RKSimon]

Updated patch now that we have the scalarized tests as a reference.

[RKSimon]

Fixed tests to check that scalar shifts are no longer happening.

[RKSimon]

ping*3

[andreadb]

Hi Simon,

Thanks for the patch and sorry it took a long time for me to review this patch.
The change looks good to me.

Thanks!
Andrea

lib/Target/X86/X86ISelLowering.cpp

16852–16872

Okay, this is because VPBLENDVB on v32i8 is only available on AVX2.

[RKSimon]

Thanks Andrea

lib/Target/X86/X86ISelLowering.cpp

if (CanBeSimplified && isa<ConstantSDNode>(Amt1) &&
       SDValue BitCast1 = DAG.getNode(ISD::BITCAST, dl, CastVT, Shift1);
       SDValue BitCast2 = DAG.getNode(ISD::BITCAST, dl, CastVT, Shift2);
       SDValue Result = getTargetShuffleNode(TargetOpcode, dl, CastVT, BitCast2,
                                             BitCast1, DAG);
       return DAG.getNode(ISD::BITCAST, dl, VT, Result);
     }
   }
 
-  if (VT == MVT::v16i8 && Op->getOpcode() == ISD::SHL) {
-    // Turn 'a' into a mask suitable for VSELECT: a = a << 5;
-    Op = DAG.getNode(ISD::SHL, dl, VT, Amt, DAG.getConstant(5, dl, VT));
+  if (VT == MVT::v16i8 || (VT == MVT::v32i8 && Subtarget->hasInt256())) {
+    MVT ExtVT = MVT::getVectorVT(MVT::i16, VT.getVectorNumElements() / 2);
+    unsigned ShiftOpcode = Op->getOpcode();
+
+    auto SignBitSelect = [&](MVT SelVT, SDValue Sel, SDValue V0, SDValue V1) {
+      // On SSE41 targets we make use of the fact that VSELECT lowers
+      // to PBLENDVB which selects bytes based just on the sign bit.
+      if (Subtarget->hasSSE41()) {
+        V0 = DAG.getNode(ISD::BITCAST, dl, VT, V0);
+        V1 = DAG.getNode(ISD::BITCAST, dl, VT, V1);
+        Sel = DAG.getNode(ISD::BITCAST, dl, VT, Sel);
+        return DAG.getNode(ISD::BITCAST, dl, SelVT,
+                           DAG.getNode(ISD::VSELECT, dl, VT, Sel, V0, V1));
+      }
+      // On pre-SSE41 targets we test for the sign bit by comparing to
+      // zero - a negative value will set all bits of the lanes to true
+      // and VSELECT uses that in its OR(AND(V0,C),AND(V1,~C)) lowering.
+      SDValue Z = getZeroVector(SelVT, Subtarget, DAG, dl);
+      SDValue C = DAG.getNode(X86ISD::PCMPGT, dl, SelVT, Z, Sel);
+      return DAG.getNode(ISD::VSELECT, dl, SelVT, C, V0, V1);
+    };
 
-    SDValue VSelM = DAG.getConstant(0x80, dl, VT);
-    SDValue OpVSel = DAG.getNode(ISD::AND, dl, VT, VSelM, Op);
-    OpVSel = DAG.getNode(X86ISD::PCMPEQ, dl, VT, OpVSel, VSelM);
-
-    // r = VSELECT(r, shl(r, 4), a);
-    SDValue M = DAG.getNode(ISD::SHL, dl, VT, R, DAG.getConstant(4, dl, VT));
-    R = DAG.getNode(ISD::VSELECT, dl, VT, OpVSel, M, R);
+    // Turn 'a' into a mask suitable for VSELECT: a = a << 5;
+    // We can safely do this using i16 shifts as we're only interested in
+    // the 3 lower bits of each byte.
+    Amt = DAG.getNode(ISD::BITCAST, dl, ExtVT, Amt);
+    Amt = DAG.getNode(ISD::SHL, dl, ExtVT, Amt, DAG.getConstant(5, dl, ExtVT));
+    Amt = DAG.getNode(ISD::BITCAST, dl, VT, Amt);
+
+    if (Op->getOpcode() == ISD::SHL || Op->getOpcode() == ISD::SRL) {
+      // r = VSELECT(r, shift(r, 4), a);
+      SDValue M =
+          DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(4, dl, VT));
+      R = SignBitSelect(VT, Amt, M, R);
 
       // a += a
-    Op = DAG.getNode(ISD::ADD, dl, VT, Op, Op);
-    OpVSel = DAG.getNode(ISD::AND, dl, VT, VSelM, Op);
-    OpVSel = DAG.getNode(X86ISD::PCMPEQ, dl, VT, OpVSel, VSelM);
-
-    // r = VSELECT(r, shl(r, 2), a);
-    M = DAG.getNode(ISD::SHL, dl, VT, R, DAG.getConstant(2, dl, VT));
-    R = DAG.getNode(ISD::VSELECT, dl, VT, OpVSel, M, R);
+      Amt = DAG.getNode(ISD::ADD, dl, VT, Amt, Amt);
+
+      // r = VSELECT(r, shift(r, 2), a);
+      M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(2, dl, VT));
+      R = SignBitSelect(VT, Amt, M, R);
 
       // a += a
-    Op = DAG.getNode(ISD::ADD, dl, VT, Op, Op);
-    OpVSel = DAG.getNode(ISD::AND, dl, VT, VSelM, Op);
-    OpVSel = DAG.getNode(X86ISD::PCMPEQ, dl, VT, OpVSel, VSelM);
-
-    // return VSELECT(r, r+r, a);
-    R = DAG.getNode(ISD::VSELECT, dl, VT, OpVSel,
-                    DAG.getNode(ISD::ADD, dl, VT, R, R), R);
+      Amt = DAG.getNode(ISD::ADD, dl, VT, Amt, Amt);
+
+      // return VSELECT(r, shift(r, 1), a);
+      M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(1, dl, VT));
+      R = SignBitSelect(VT, Amt, M, R);
       return R;
     }
 
+    if (Op->getOpcode() == ISD::SRA) {
+      // For SRA we need to unpack each byte to the higher byte of a i16 vector
+      // so we can correctly sign extend. We don't care what happens to the
+      // lower byte.
+      SDValue ALo = DAG.getNode(X86ISD::UNPCKL, dl, VT, DAG.getUNDEF(VT), Amt);
+      SDValue AHi = DAG.getNode(X86ISD::UNPCKH, dl, VT, DAG.getUNDEF(VT), Amt);
+      SDValue RLo = DAG.getNode(X86ISD::UNPCKL, dl, VT, DAG.getUNDEF(VT), R);
+      SDValue RHi = DAG.getNode(X86ISD::UNPCKH, dl, VT, DAG.getUNDEF(VT), R);
+      ALo = DAG.getNode(ISD::BITCAST, dl, ExtVT, ALo);
+      AHi = DAG.getNode(ISD::BITCAST, dl, ExtVT, AHi);
+      RLo = DAG.getNode(ISD::BITCAST, dl, ExtVT, RLo);
+      RHi = DAG.getNode(ISD::BITCAST, dl, ExtVT, RHi);
+
+      // r = VSELECT(r, shift(r, 4), a);
+      SDValue MLo = DAG.getNode(ShiftOpcode, dl, ExtVT, RLo,
+                                DAG.getConstant(4, dl, ExtVT));
+      SDValue MHi = DAG.getNode(ShiftOpcode, dl, ExtVT, RHi,
+                                DAG.getConstant(4, dl, ExtVT));
+      RLo = SignBitSelect(ExtVT, ALo, MLo, RLo);
+      RHi = SignBitSelect(ExtVT, AHi, MHi, RHi);
+
+      // a += a
+      ALo = DAG.getNode(ISD::ADD, dl, ExtVT, ALo, ALo);
+      AHi = DAG.getNode(ISD::ADD, dl, ExtVT, AHi, AHi);
+
+      // r = VSELECT(r, shift(r, 2), a);
+      MLo = DAG.getNode(ShiftOpcode, dl, ExtVT, RLo,
+                        DAG.getConstant(2, dl, ExtVT));
+      MHi = DAG.getNode(ShiftOpcode, dl, ExtVT, RHi,
+                        DAG.getConstant(2, dl, ExtVT));
+      RLo = SignBitSelect(ExtVT, ALo, MLo, RLo);
+      RHi = SignBitSelect(ExtVT, AHi, MHi, RHi);
+
+      // a += a
+      ALo = DAG.getNode(ISD::ADD, dl, ExtVT, ALo, ALo);
+      AHi = DAG.getNode(ISD::ADD, dl, ExtVT, AHi, AHi);
+
+      // r = VSELECT(r, shift(r, 1), a);
+      MLo = DAG.getNode(ShiftOpcode, dl, ExtVT, RLo,
+                        DAG.getConstant(1, dl, ExtVT));
+      MHi = DAG.getNode(ShiftOpcode, dl, ExtVT, RHi,
+                        DAG.getConstant(1, dl, ExtVT));
+      RLo = SignBitSelect(ExtVT, ALo, MLo, RLo);
+      RHi = SignBitSelect(ExtVT, AHi, MHi, RHi);
+
+      // Logical shift the result back to the lower byte, leaving a zero upper
+      // byte
+      // meaning that we can safely pack with PACKUSWB.
+      RLo =
+          DAG.getNode(ISD::SRL, dl, ExtVT, RLo, DAG.getConstant(8, dl, ExtVT));
+      RHi =
+          DAG.getNode(ISD::SRL, dl, ExtVT, RHi, DAG.getConstant(8, dl, ExtVT));
+      return DAG.getNode(X86ISD::PACKUS, dl, VT, RLo, RHi);
+    }
+  }
+
   // It's worth extending once and using the v8i32 shifts for 16-bit types, but
   // the extra overheads to get from v16i8 to v8i32 make the existing SSE
   // solution better.
   if (Subtarget->hasInt256() && VT == MVT::v8i16) {
     MVT ExtVT = MVT::v8i32;
     unsigned ExtOpc =
         Op.getOpcode() == ISD::SRA ? ISD::SIGN_EXTEND : ISD::ZERO_EXTEND;
     R = DAG.getNode(ExtOpc, dl, ExtVT, R);
if (Subtarget->hasInt256() && VT == MVT::v16i16) {
     RHi = DAG.getNode(ISD::BITCAST, dl, ExtVT, RHi);
     SDValue Lo = DAG.getNode(Op.getOpcode(), dl, ExtVT, RLo, ALo);
     SDValue Hi = DAG.getNode(Op.getOpcode(), dl, ExtVT, RHi, AHi);
     Lo = DAG.getNode(ISD::SRL, dl, ExtVT, Lo, DAG.getConstant(16, dl, ExtVT));
     Hi = DAG.getNode(ISD::SRL, dl, ExtVT, Hi, DAG.getConstant(16, dl, ExtVT));
     return DAG.getNode(X86ISD::PACKUS, dl, VT, Lo, Hi);
   }
 
+  if (VT == MVT::v8i16) {
+    unsigned ShiftOpcode = Op->getOpcode();
+
+    auto SignBitSelect = [&](SDValue Sel, SDValue V0, SDValue V1) {
+      // On SSE41 targets we make use of the fact that VSELECT lowers
+      // to PBLENDVB which selects bytes based just on the sign bit.
+      if (Subtarget->hasSSE41()) {
+        MVT ExtVT = MVT::getVectorVT(MVT::i8, VT.getVectorNumElements() * 2);
+        V0 = DAG.getNode(ISD::BITCAST, dl, ExtVT, V0);
+        V1 = DAG.getNode(ISD::BITCAST, dl, ExtVT, V1);
+        Sel = DAG.getNode(ISD::BITCAST, dl, ExtVT, Sel);
+        return DAG.getNode(ISD::BITCAST, dl, VT,
+                           DAG.getNode(ISD::VSELECT, dl, ExtVT, Sel, V0, V1));
+      }
+      // On pre-SSE41 targets we splat the sign bit - a negative value will
+      // set all bits of the lanes to true and VSELECT uses that in
+      // its OR(AND(V0,C),AND(V1,~C)) lowering.
+      SDValue C = DAG.getNode(ISD::SRA, dl, VT, Sel,
+                              DAG.getConstant(15, dl, VT));
+      return DAG.getNode(ISD::VSELECT, dl, VT, C, V0, V1);
+    };
+
+    // Turn 'a' into a mask suitable for VSELECT: a = a << 12;
+    if (Subtarget->hasSSE41()) {
+      // On SSE41 targets we need to replicate the shift mask in both
+      // bytes for PBLENDVB.
+      Amt = DAG.getNode(
+          ISD::OR, dl, VT,
+          DAG.getNode(ISD::SHL, dl, VT, Amt, DAG.getConstant(4, dl, VT)),
+          DAG.getNode(ISD::SHL, dl, VT, Amt, DAG.getConstant(12, dl, VT)));
+    } else {
+      Amt = DAG.getNode(ISD::SHL, dl, VT, Amt, DAG.getConstant(12, dl, VT));
+    }
+
+    // r = VSELECT(r, shift(r, 8), a);
+    SDValue M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(8, dl, VT));
+    R = SignBitSelect(Amt, M, R);
+
+    // a += a
+    Amt = DAG.getNode(ISD::ADD, dl, VT, Amt, Amt);
+
+    // r = VSELECT(r, shift(r, 4), a);
+    M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(4, dl, VT));
+    R = SignBitSelect(Amt, M, R);
+
+    // a += a
+    Amt = DAG.getNode(ISD::ADD, dl, VT, Amt, Amt);
+
+    // r = VSELECT(r, shift(r, 2), a);
+    M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(2, dl, VT));
+    R = SignBitSelect(Amt, M, R);
+
+    // a += a
+    Amt = DAG.getNode(ISD::ADD, dl, VT, Amt, Amt);
+
+    // return VSELECT(r, shift(r, 1), a);
+    M = DAG.getNode(ShiftOpcode, dl, VT, R, DAG.getConstant(1, dl, VT));
+    R = SignBitSelect(Amt, M, R);
+    return R;
+  }
+
   // Decompose 256-bit shifts into smaller 128-bit shifts.
   if (VT.is256BitVector()) {
     unsigned NumElems = VT.getVectorNumElements();
     MVT EltVT = VT.getVectorElementType();
     EVT NewVT = MVT::getVectorVT(EltVT, NumElems/2);
 
     // Extract the two vectors
     SDValue V1 = Extract128BitVector(R, 0, DAG, dl);

lib/Target/X86/X86TargetTransformInfo.cpp

static const CostTblEntry<MVT::SimpleValueType> AVX2CostTable[] = {
     { ISD::SHL,     MVT::v8i32,    1 },
     { ISD::SRL,     MVT::v8i32,    1 },
     { ISD::SRA,     MVT::v8i32,    1 },
     { ISD::SHL,     MVT::v2i64,    1 },
     { ISD::SRL,     MVT::v2i64,    1 },
     { ISD::SHL,     MVT::v4i64,    1 },
     { ISD::SRL,     MVT::v4i64,    1 },
 
-    { ISD::SHL,  MVT::v32i8,      42 }, // cmpeqb sequence.
+    { ISD::SHL,  MVT::v32i8,      11 }, // vpblendvb sequence.
     { ISD::SHL,  MVT::v16i16,     10 }, // extend/vpsrlvd/pack sequence.
 
-    { ISD::SRL,  MVT::v32i8,   32*10 }, // Scalarized.
+    { ISD::SRL,  MVT::v32i8,      11 }, // vpblendvb sequence.
     { ISD::SRL,  MVT::v16i16,     10 }, // extend/vpsrlvd/pack sequence.
 
-    { ISD::SRA,  MVT::v32i8,   32*10 }, // Scalarized.
+    { ISD::SRA,  MVT::v32i8,      24 }, // vpblendvb sequence.
     { ISD::SRA,  MVT::v16i16,     10 }, // extend/vpsravd/pack sequence.
     { ISD::SRA,  MVT::v4i64,    4*10 }, // Scalarized.
 
     // Vectorizing division is a bad idea. See the SSE2 table for more comments.
     { ISD::SDIV,  MVT::v32i8,  32*20 },
     { ISD::SDIV,  MVT::v16i16, 16*20 },
     { ISD::SDIV,  MVT::v8i32,  8*20 },
     { ISD::SDIV,  MVT::v4i64,  4*20 },
static const CostTblEntry<MVT::SimpleValueType> SSE2CostTable[] = {
     // We don't correctly identify costs of casts because they are marked as
     // custom.
     // For some cases, where the shift amount is a scalar we would be able
     // to generate better code. Unfortunately, when this is the case the value
     // (the splat) will get hoisted out of the loop, thereby making it invisible
     // to ISel. The cost model must return worst case assumptions because it is
     // used for vectorization and we don't want to make vectorized code worse
     // than scalar code.
-    { ISD::SHL,  MVT::v16i8,  30 }, // cmpeqb sequence.
-    { ISD::SHL,  MVT::v8i16,  8*10 }, // Scalarized.
+    { ISD::SHL,  MVT::v16i8,    26 }, // cmpgtb sequence.
+    { ISD::SHL,  MVT::v8i16,    32 }, // cmpgtb sequence.
     { ISD::SHL,  MVT::v4i32,  2*5 }, // We optimized this using mul.
     { ISD::SHL,  MVT::v2i64,  2*10 }, // Scalarized.
     { ISD::SHL,  MVT::v4i64,  4*10 }, // Scalarized.
 
-    { ISD::SRL,  MVT::v16i8,  16*10 }, // Scalarized.
-    { ISD::SRL,  MVT::v8i16,  8*10 }, // Scalarized.
+    { ISD::SRL,  MVT::v16i8,    26 }, // cmpgtb sequence.
+    { ISD::SRL,  MVT::v8i16,    32 }, // cmpgtb sequence.
     { ISD::SRL,  MVT::v4i32,  4*10 }, // Scalarized.
     { ISD::SRL,  MVT::v2i64,  2*10 }, // Scalarized.
 
-    { ISD::SRA,  MVT::v16i8,  16*10 }, // Scalarized.
-    { ISD::SRA,  MVT::v8i16,  8*10 }, // Scalarized.
+    { ISD::SRA,  MVT::v16i8,    54 }, // unpacked cmpgtb sequence.
+    { ISD::SRA,  MVT::v8i16,    32 }, // cmpgtb sequence.
     { ISD::SRA,  MVT::v4i32,  4*10 }, // Scalarized.
     { ISD::SRA,  MVT::v2i64,  2*10 }, // Scalarized.
 
     // It is not a good idea to vectorize division. We have to scalarize it and
     // in the process we will often end up having to spilling regular
     // registers. The overhead of division is going to dominate most kernels
     // anyways so try hard to prevent vectorization of division - it is
     // generally a bad idea. Assume somewhat arbitrarily that we have to be able
if ((Idx < 4) || (Imm.getBitWidth() <= 64 && isInt<64>(Imm.getSExtValue())))
       return TTI::TCC_Free;
     break;
   }
   return X86TTIImpl::getIntImmCost(Imm, Ty);
 }
 
 bool X86TTIImpl::isLegalMaskedLoad(Type *DataTy, int Consecutive) {
   int DataWidth = DataTy->getPrimitiveSizeInBits();
-  
   // Todo: AVX512 allows gather/scatter, works with strided and random as well
   if ((DataWidth < 32) || (Consecutive == 0))
     return false;
-  if (ST->hasAVX512() || ST->hasAVX2()) 
+  if (ST->hasAVX512() || ST->hasAVX2())
     return true;
   return false;
 }
 
 bool X86TTIImpl::isLegalMaskedStore(Type *DataType, int Consecutive) {
   return isLegalMaskedLoad(DataType, Consecutive);
 }
 

test/Analysis/CostModel/X86/testshiftashr.ll

entry:
   %0 = ashr %shifttype4i16 %a , %b
   ret %shifttype4i16 %0
 }
 
 %shifttype8i16 = type <8 x i16>
 define %shifttype8i16 @shift8i16(%shifttype8i16 %a, %shifttype8i16 %b) {
 entry:
   ; SSE2: shift8i16
-  ; SSE2: cost of 80 {{.*}} ashr
+  ; SSE2: cost of 32 {{.*}} ashr
   ; SSE2-CODEGEN: shift8i16
-  ; SSE2-CODEGEN: sarw %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype8i16 %a , %b
   ret %shifttype8i16 %0
 }
 
 %shifttype16i16 = type <16 x i16>
 define %shifttype16i16 @shift16i16(%shifttype16i16 %a, %shifttype16i16 %b) {
 entry:
   ; SSE2: shift16i16
-  ; SSE2: cost of 160 {{.*}} ashr
+  ; SSE2: cost of 64 {{.*}} ashr
   ; SSE2-CODEGEN: shift16i16
-  ; SSE2-CODEGEN: sarw %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype16i16 %a , %b
   ret %shifttype16i16 %0
 }
 
 %shifttype32i16 = type <32 x i16>
 define %shifttype32i16 @shift32i16(%shifttype32i16 %a, %shifttype32i16 %b) {
 entry:
   ; SSE2: shift32i16
-  ; SSE2: cost of 320 {{.*}} ashr
+  ; SSE2: cost of 128 {{.*}} ashr
   ; SSE2-CODEGEN: shift32i16
-  ; SSE2-CODEGEN: sarw %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype32i16 %a , %b
   ret %shifttype32i16 %0
 }
 
 %shifttype2i32 = type <2 x i32>
 define %shifttype2i32 @shift2i32(%shifttype2i32 %a, %shifttype2i32 %b) {
 entry:
entry:
   %0 = ashr %shifttype4i8 %a , %b
   ret %shifttype4i8 %0
 }
 
 %shifttype8i8 = type <8 x i8>
 define %shifttype8i8 @shift8i8(%shifttype8i8 %a, %shifttype8i8 %b) {
 entry:
   ; SSE2: shift8i8
-  ; SSE2: cost of 80 {{.*}} ashr
+  ; SSE2: cost of 32 {{.*}} ashr
   ; SSE2-CODEGEN: shift8i8
-  ; SSE2-CODEGEN: sarw %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype8i8 %a , %b
   ret %shifttype8i8 %0
 }
 
 %shifttype16i8 = type <16 x i8>
 define %shifttype16i8 @shift16i8(%shifttype16i8 %a, %shifttype16i8 %b) {
 entry:
   ; SSE2: shift16i8
-  ; SSE2: cost of 160 {{.*}} ashr
+  ; SSE2: cost of 54 {{.*}} ashr
   ; SSE2-CODEGEN: shift16i8
-  ; SSE2-CODEGEN: sarb %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype16i8 %a , %b
   ret %shifttype16i8 %0
 }
 
 %shifttype32i8 = type <32 x i8>
 define %shifttype32i8 @shift32i8(%shifttype32i8 %a, %shifttype32i8 %b) {
 entry:
   ; SSE2: shift32i8
-  ; SSE2: cost of 320 {{.*}} ashr
+  ; SSE2: cost of 108 {{.*}} ashr
   ; SSE2-CODEGEN: shift32i8
-  ; SSE2-CODEGEN: sarb %cl
+  ; SSE2-CODEGEN: psraw
 
   %0 = ashr %shifttype32i8 %a , %b
   ret %shifttype32i8 %0
 }
 
 ; Test shift by a constant a value.
 
 %shifttypec = type <2 x i16>

test/Analysis/CostModel/X86/testshiftlshr.ll

entry:
   %0 = lshr %shifttype4i16 %a , %b
   ret %shifttype4i16 %0
 }
 
 %shifttype8i16 = type <8 x i16>
 define %shifttype8i16 @shift8i16(%shifttype8i16 %a, %shifttype8i16 %b) {
 entry:
   ; SSE2: shift8i16
-  ; SSE2: cost of 80 {{.*}} lshr
+  ; SSE2: cost of 32 {{.*}} lshr
   ; SSE2-CODEGEN: shift8i16
-  ; SSE2-CODEGEN: shrl %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype8i16 %a , %b
   ret %shifttype8i16 %0
 }
 
 %shifttype16i16 = type <16 x i16>
 define %shifttype16i16 @shift16i16(%shifttype16i16 %a, %shifttype16i16 %b) {
 entry:
   ; SSE2: shift16i16
-  ; SSE2: cost of 160 {{.*}} lshr
+  ; SSE2: cost of 64 {{.*}} lshr
   ; SSE2-CODEGEN: shift16i16
-  ; SSE2-CODEGEN: shrl %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype16i16 %a , %b
   ret %shifttype16i16 %0
 }
 
 %shifttype32i16 = type <32 x i16>
 define %shifttype32i16 @shift32i16(%shifttype32i16 %a, %shifttype32i16 %b) {
 entry:
   ; SSE2: shift32i16
-  ; SSE2: cost of 320 {{.*}} lshr
+  ; SSE2: cost of 128 {{.*}} lshr
   ; SSE2-CODEGEN: shift32i16
-  ; SSE2-CODEGEN: shrl %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype32i16 %a , %b
   ret %shifttype32i16 %0
 }
 
 %shifttype2i32 = type <2 x i32>
 define %shifttype2i32 @shift2i32(%shifttype2i32 %a, %shifttype2i32 %b) {
 entry:
entry:
   %0 = lshr %shifttype4i8 %a , %b
   ret %shifttype4i8 %0
 }
 
 %shifttype8i8 = type <8 x i8>
 define %shifttype8i8 @shift8i8(%shifttype8i8 %a, %shifttype8i8 %b) {
 entry:
   ; SSE2: shift8i8
-  ; SSE2: cost of 80 {{.*}} lshr
+  ; SSE2: cost of 32 {{.*}} lshr
   ; SSE2-CODEGEN: shift8i8
-  ; SSE2-CODEGEN: shrl %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype8i8 %a , %b
   ret %shifttype8i8 %0
 }
 
 %shifttype16i8 = type <16 x i8>
 define %shifttype16i8 @shift16i8(%shifttype16i8 %a, %shifttype16i8 %b) {
 entry:
   ; SSE2: shift16i8
-  ; SSE2: cost of 160 {{.*}} lshr
+  ; SSE2: cost of 26 {{.*}} lshr
   ; SSE2-CODEGEN: shift16i8
-  ; SSE2-CODEGEN: shrb %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype16i8 %a , %b
   ret %shifttype16i8 %0
 }
 
 %shifttype32i8 = type <32 x i8>
 define %shifttype32i8 @shift32i8(%shifttype32i8 %a, %shifttype32i8 %b) {
 entry:
   ; SSE2: shift32i8
-  ; SSE2: cost of 320 {{.*}} lshr
+  ; SSE2: cost of 52 {{.*}} lshr
   ; SSE2-CODEGEN: shift32i8
-  ; SSE2-CODEGEN: shrb %cl
+  ; SSE2-CODEGEN: psrlw
 
   %0 = lshr %shifttype32i8 %a , %b
   ret %shifttype32i8 %0
 }
 
 ; Test shift by a constant vector.
 
 %shifttypec = type <2 x i16>

test/Analysis/CostModel/X86/testshiftshl.ll

entry:
   %0 = shl %shifttype4i16 %a , %b
   ret %shifttype4i16 %0
 }
 
 %shifttype8i16 = type <8 x i16>
 define %shifttype8i16 @shift8i16(%shifttype8i16 %a, %shifttype8i16 %b) {
 entry:
   ; SSE2: shift8i16
-  ; SSE2: cost of 80 {{.*}} shl
+  ; SSE2: cost of 32 {{.*}} shl
   ; SSE2-CODEGEN: shift8i16
-  ; SSE2-CODEGEN: shll %cl
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype8i16 %a , %b
   ret %shifttype8i16 %0
 }
 
 %shifttype16i16 = type <16 x i16>
 define %shifttype16i16 @shift16i16(%shifttype16i16 %a, %shifttype16i16 %b) {
 entry:
   ; SSE2: shift16i16
-  ; SSE2: cost of 160 {{.*}} shl
+  ; SSE2: cost of 64 {{.*}} shl
   ; SSE2-CODEGEN: shift16i16
-  ; SSE2-CODEGEN: shll %cl
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype16i16 %a , %b
   ret %shifttype16i16 %0
 }
 
 %shifttype32i16 = type <32 x i16>
 define %shifttype32i16 @shift32i16(%shifttype32i16 %a, %shifttype32i16 %b) {
 entry:
   ; SSE2: shift32i16
-  ; SSE2: cost of 320 {{.*}} shl
+  ; SSE2: cost of 128 {{.*}} shl
   ; SSE2-CODEGEN: shift32i16
-  ; SSE2-CODEGEN: shll %cl
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype32i16 %a , %b
   ret %shifttype32i16 %0
 }
 
 %shifttype2i32 = type <2 x i32>
 define %shifttype2i32 @shift2i32(%shifttype2i32 %a, %shifttype2i32 %b) {
 entry:
entry:
   %0 = shl %shifttype4i8 %a , %b
   ret %shifttype4i8 %0
 }
 
 %shifttype8i8 = type <8 x i8>
 define %shifttype8i8 @shift8i8(%shifttype8i8 %a, %shifttype8i8 %b) {
 entry:
   ; SSE2: shift8i8
-  ; SSE2: cost of 80 {{.*}} shl
+  ; SSE2: cost of 32 {{.*}} shl
   ; SSE2-CODEGEN: shift8i8
-  ; SSE2-CODEGEN: shll
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype8i8 %a , %b
   ret %shifttype8i8 %0
 }
 
 %shifttype16i8 = type <16 x i8>
 define %shifttype16i8 @shift16i8(%shifttype16i8 %a, %shifttype16i8 %b) {
 entry:
   ; SSE2: shift16i8
-  ; SSE2: cost of 30 {{.*}} shl
+  ; SSE2: cost of 26 {{.*}} shl
   ; SSE2-CODEGEN: shift16i8
-  ; SSE2-CODEGEN: cmpeqb
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype16i8 %a , %b
   ret %shifttype16i8 %0
 }
 
 %shifttype32i8 = type <32 x i8>
 define %shifttype32i8 @shift32i8(%shifttype32i8 %a, %shifttype32i8 %b) {
 entry:
   ; SSE2: shift32i8
-  ; SSE2: cost of 60 {{.*}} shl
+  ; SSE2: cost of 52 {{.*}} shl
   ; SSE2-CODEGEN: shift32i8
-  ; SSE2-CODEGEN: cmpeqb
+  ; SSE2-CODEGEN: psllw
 
   %0 = shl %shifttype32i8 %a , %b
   ret %shifttype32i8 %0
 }
 
 ; Test shift by a constant vector.
 
 %shifttypec = type <2 x i16>

test/CodeGen/X86/2011-12-15-vec_shift.ll

 ; RUN: llc -march=x86-64 -mattr=+sse4.1 -mcpu=penryn < %s | FileCheck %s -check-prefix=CHECK-W-SSE4
 ; RUN: llc -march=x86-64 -mattr=-sse4.1 -mcpu=penryn < %s | FileCheck %s -check-prefix=CHECK-WO-SSE4
 ; Test case for r146671
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"
 target triple = "x86_64-apple-macosx10.7"
 
 define <16 x i8> @shift(<16 x i8> %a, <16 x i8> %b) nounwind {
   ; Make sure operands to pblend are in the right order.
   ; CHECK-W-SSE4: psllw $4, [[REG1:%xmm.]]
   ; CHECK-W-SSE4: pblendvb [[REG1]],{{ %xmm.}}
   ; CHECK-W-SSE4: psllw $2
 
   ; Make sure we're masking and pcmp'ing the VSELECT conditon vector.
   ; CHECK-WO-SSE4: psllw $5, [[REG1:%xmm.]]
-  ; CHECK-WO-SSE4: pand [[REG1]], [[REG2:%xmm.]]
-  ; CHECK-WO-SSE4: pcmpeqb {{%xmm., }}[[REG2]]
+  ; CHECK-WO-SSE4: pxor [[REG2:%xmm.]], [[REG2:%xmm.]]
+  ; CHECK-WO-SSE4: pcmpgtb {{%xmm., }}[[REG2]]
   %1 = shl <16 x i8> %a, %b
   ret <16 x i8> %1
 }

test/CodeGen/X86/avx2-vector-shifts.ll

 ; CHECK-NEXT:   vpsllvd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:   vpsrld $16, %ymm0, %ymm0
 ; CHECK-NEXT:   vpackusdw %ymm3, %ymm0, %ymm0
 ; CHECK-NEXT:   retq
   %shl = shl <16 x i16> %r, %a
   ret <16 x i16> %shl
 }
 
+define <32 x i8> @shl_32i8(<32 x i8> %r, <32 x i8> %a) nounwind {
+; CHECK-LABEL:  shl_32i8
+; CHECK:        vpsllw    $5, %ymm1, %ymm1
+; CHECK-NEXT:   vpsllw    $4, %ymm0, %ymm2
+; CHECK-NEXT:   vpand     {{.*}}(%rip), %ymm2, %ymm2
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   vpsllw    $2, %ymm0, %ymm2
+; CHECK-NEXT:   vpand     {{.*}}(%rip), %ymm2, %ymm2
+; CHECK-NEXT:   vpaddb    %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   vpaddb    %ymm0, %ymm0, %ymm2
+; CHECK-NEXT:   vpaddb    %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   retq
+  %shl = shl <32 x i8> %r, %a
+  ret <32 x i8> %shl
+}
+
 define <8 x i16> @ashr_8i16(<8 x i16> %r, <8 x i16> %a) nounwind {
 ; CHECK-LABEL:  ashr_8i16
 ; CHECK:        vpmovzxwd {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
 ; CHECK-NEXT:   vpmovsxwd %xmm0, %ymm0
 ; CHECK-NEXT:   vpsravd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:   vpshufb {{.*#+}} ymm0 = ymm0[0,1,4,5,8,9,12,13],zero,zero,zero,zero,zero,zero,zero,zero,ymm0[16,17,20,21,24,25,28,29],zero,zero,zero,zero,zero,zero,zero,zero
 ; CHECK-NEXT:   vpermq {{.*#+}} ymm0 = ymm0[0,2,2,3]
 ; CHECK:        retq

 ; CHECK-NEXT:   vpsravd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:   vpsrld $16, %ymm0, %ymm0
 ; CHECK-NEXT:   vpackusdw %ymm3, %ymm0, %ymm0
 ; CHECK-NEXT:   retq
   %ashr = ashr <16 x i16> %r, %a
   ret <16 x i16> %ashr
 }
 
+define <32 x i8> @ashr_32i8(<32 x i8> %r, <32 x i8> %a) nounwind {
+; CHECK-LABEL:  ashr_32i8
+; CHECK:        vpsllw     $5, %ymm1, %ymm1
+; CHECK-NEXT:   vpunpckhbw {{.*#+}} ymm2 = ymm0[8],ymm1[8],ymm0[9],ymm1[9],ymm0[10],ymm1[10],ymm0[11],ymm1[11],ymm0[12],ymm1[12],ymm0[13],ymm1[13],ymm0[14],ymm1[14],ymm0[15],ymm1[15],ymm0[24],ymm1[24],ymm0[25],ymm1[25],ymm0[26],ymm1[26],ymm0[27],ymm1[27],ymm0[28],ymm1[28],ymm0[29],ymm1[29],ymm0[30],ymm1[30],ymm0[31],ymm1[31]
+; CHECK-NEXT:   vpunpckhbw {{.*#+}} ymm3 = ymm0[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15,24,24,25,25,26,26,27,27,28,28,29,29,30,30,31,31]
+; CHECK-NEXT:   vpsraw     $4, %ymm3, %ymm4
+; CHECK-NEXT:   vpblendvb  %ymm2, %ymm4, %ymm3, %ymm3
+; CHECK-NEXT:   vpsraw     $2, %ymm3, %ymm4
+; CHECK-NEXT:   vpaddw     %ymm2, %ymm2, %ymm2
+; CHECK-NEXT:   vpblendvb  %ymm2, %ymm4, %ymm3, %ymm3
+; CHECK-NEXT:   vpsraw     $1, %ymm3, %ymm4
+; CHECK-NEXT:   vpaddw     %ymm2, %ymm2, %ymm2
+; CHECK-NEXT:   vpblendvb  %ymm2, %ymm4, %ymm3, %ymm2
+; CHECK-NEXT:   vpsrlw     $8, %ymm2, %ymm2
+; CHECK-NEXT:   vpunpcklbw {{.*#+}} ymm1 = ymm0[0],ymm1[0],ymm0[1],ymm1[1],ymm0[2],ymm1[2],ymm0[3],ymm1[3],ymm0[4],ymm1[4],ymm0[5],ymm1[5],ymm0[6],ymm1[6],ymm0[7],ymm1[7],ymm0[16],ymm1[16],ymm0[17],ymm1[17],ymm0[18],ymm1[18],ymm0[19],ymm1[19],ymm0[20],ymm1[20],ymm0[21],ymm1[21],ymm0[22],ymm1[22],ymm0[23],ymm1[23]
+; CHECK-NEXT:   vpunpcklbw {{.*#+}} ymm0 = ymm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,16,16,17,17,18,18,19,19,20,20,21,21,22,22,23,23]
+; CHECK-NEXT:   vpsraw     $4, %ymm0, %ymm3
+; CHECK-NEXT:   vpblendvb  %ymm1, %ymm3, %ymm0, %ymm0
+; CHECK-NEXT:   vpsraw     $2, %ymm0, %ymm3
+; CHECK-NEXT:   vpaddw     %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb  %ymm1, %ymm3, %ymm0, %ymm0
+; CHECK-NEXT:   vpsraw     $1, %ymm0, %ymm3
+; CHECK-NEXT:   vpaddw     %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb  %ymm1, %ymm3, %ymm0, %ymm0
+; CHECK-NEXT:   vpsrlw     $8, %ymm0, %ymm0
+; CHECK-NEXT:   vpackuswb  %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   retq
+  %ashr = ashr <32 x i8> %r, %a
+  ret <32 x i8> %ashr
+}
+
 define <8 x i16> @lshr_8i16(<8 x i16> %r, <8 x i16> %a) nounwind {
 ; CHECK-LABEL:  lshr_8i16
 ; CHECK:        vpmovzxwd {{.*#+}} ymm1 = xmm1[0],zero,xmm1[1],zero,xmm1[2],zero,xmm1[3],zero,xmm1[4],zero,xmm1[5],zero,xmm1[6],zero,xmm1[7],zero
 ; CHECK-NEXT:   vpmovzxwd {{.*#+}} ymm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
 ; CHECK-NEXT:   vpsrlvd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:   vpshufb {{.*#+}} ymm0 = ymm0[0,1,4,5,8,9,12,13],zero,zero,zero,zero,zero,zero,zero,zero,ymm0[16,17,20,21,24,25,28,29],zero,zero,zero,zero,zero,zero,zero,zero
 ; CHECK-NEXT:   vpermq {{.*#+}} ymm0 = ymm0[0,2,2,3]
 ; CHECK:        retq

 ; CHECK-NEXT:   vpunpcklwd {{.*#+}} ymm0 = ymm0[0,0,1,1,2,2,3,3,8,8,9,9,10,10,11,11]
 ; CHECK-NEXT:   vpsrlvd %ymm1, %ymm0, %ymm0
 ; CHECK-NEXT:   vpsrld $16, %ymm0, %ymm0
 ; CHECK-NEXT:   vpackusdw %ymm3, %ymm0, %ymm0
 ; CHECK-NEXT:   retq
   %lshr = lshr <16 x i16> %r, %a
   ret <16 x i16> %lshr
 }
+
+define <32 x i8> @lshr_32i8(<32 x i8> %r, <32 x i8> %a) nounwind {
+; CHECK-LABEL:  lshr_32i8
+; CHECK:        vpsllw    $5, %ymm1, %ymm1
+; CHECK-NEXT:   vpsrlw    $4, %ymm0, %ymm2
+; CHECK-NEXT:   vpand     {{.*}}(%rip), %ymm2, %ymm2
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   vpsrlw    $2, %ymm0, %ymm2
+; CHECK-NEXT:   vpand     {{.*}}(%rip), %ymm2, %ymm2
+; CHECK-NEXT:   vpaddb    %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   vpsrlw    $1, %ymm0, %ymm2
+; CHECK-NEXT:   vpand     {{.*}}(%rip), %ymm2, %ymm2
+; CHECK-NEXT:   vpaddb    %ymm1, %ymm1, %ymm1
+; CHECK-NEXT:   vpblendvb %ymm1, %ymm2, %ymm0, %ymm0
+; CHECK-NEXT:   retq
+  %lshr = lshr <32 x i8> %r, %a
+  ret <32 x i8> %lshr
+}

test/CodeGen/X86/vec_shift8.ll

+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefix=SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.1 | FileCheck %s --check-prefix=SSE41
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefix=AVX
+
+;
+; Vectorized integer shifts
+;
+
+define <2 x i64> @shl_8i16(<8 x i16> %r, <8 x i16> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: shlw
+;
+; SSE2:       psllw   $12, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psllw   $8, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psllw   $4, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psllw   $2, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  psraw   $15, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  pandn   %xmm0, %xmm2
+; SSE2-NEXT:  psllw   $1, %xmm0
+; SSE2-NEXT:  pand    %xmm1, %xmm0
+; SSE2-NEXT:  por     %xmm2, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa   %xmm0, %xmm2
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: psllw    $12, %xmm0
+; SSE41-NEXT: psllw    $4, %xmm1
+; SSE41-NEXT: por      %xmm0, %xmm1
+; SSE41-NEXT: movdqa   %xmm1, %xmm3
+; SSE41-NEXT: paddw    %xmm3, %xmm3
+; SSE41-NEXT: movdqa   %xmm2, %xmm4
+; SSE41-NEXT: psllw    $8, %xmm4
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm4, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm1
+; SSE41-NEXT: psllw    $4, %xmm1
+; SSE41-NEXT: movdqa   %xmm3, %xmm0
+; SSE41-NEXT: pblendvb %xmm1, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm1
+; SSE41-NEXT: psllw    $2, %xmm1
+; SSE41-NEXT: paddw    %xmm3, %xmm3
+; SSE41-NEXT: movdqa   %xmm3, %xmm0
+; SSE41-NEXT: pblendvb %xmm1, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm1
+; SSE41-NEXT: psllw    $1, %xmm1
+; SSE41-NEXT: paddw    %xmm3, %xmm3
+; SSE41-NEXT: movdqa   %xmm3, %xmm0
+; SSE41-NEXT: pblendvb %xmm1, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw    $12, %xmm1, %xmm2
+; AVX-NEXT:   vpsllw    $4, %xmm1, %xmm1
+; AVX-NEXT:   vpor      %xmm2, %xmm1, %xmm1
+; AVX-NEXT:   vpaddw    %xmm1, %xmm1, %xmm2
+; AVX-NEXT:   vpsllw    $8, %xmm0, %xmm3
+; AVX-NEXT:   vpblendvb %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsllw    $4, %xmm0, %xmm1
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsllw    $2, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsllw    $1, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %shl = shl <8 x i16> %r, %a
+  %tmp2 = bitcast <8 x i16> %shl to <2 x i64>
+  ret <2 x i64> %tmp2
+}
+
+define <2 x i64> @shl_16i8(<16 x i8> %r, <16 x i8> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: shlb
+;
+; SSE2:       psllw   $5, %xmm1
+; SSE2-NEXT:  pxor    %xmm2, %xmm2
+; SSE2-NEXT:  pxor    %xmm3, %xmm3
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm3
+; SSE2-NEXT:  movdqa  %xmm3, %xmm4
+; SSE2-NEXT:  pandn   %xmm0, %xmm4
+; SSE2-NEXT:  psllw   $4, %xmm0
+; SSE2-NEXT:  pand    {{.*}}(%rip), %xmm0
+; SSE2-NEXT:  pand    %xmm3, %xmm0
+; SSE2-NEXT:  por     %xmm4, %xmm0
+; SSE2-NEXT:  paddb   %xmm1, %xmm1
+; SSE2-NEXT:  pxor    %xmm3, %xmm3
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm3
+; SSE2-NEXT:  movdqa  %xmm3, %xmm4
+; SSE2-NEXT:  pandn   %xmm0, %xmm4
+; SSE2-NEXT:  psllw   $2, %xmm0
+; SSE2-NEXT:  pand    {{.*}}(%rip), %xmm0
+; SSE2-NEXT:  pand    %xmm3, %xmm0
+; SSE2-NEXT:  por     %xmm4, %xmm0
+; SSE2-NEXT:  paddb   %xmm1, %xmm1
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm1
+; SSE2-NEXT:  pandn   %xmm0, %xmm1
+; SSE2-NEXT:  paddb   %xmm0, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm1, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa   %xmm0, %xmm2
+; SSE41-NEXT: psllw    $5, %xmm1
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: psllw    $4, %xmm3
+; SSE41-NEXT: pand     {{.*}}(%rip), %xmm3
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: psllw    $2, %xmm3
+; SSE41-NEXT: pand     {{.*}}(%rip), %xmm3
+; SSE41-NEXT: paddb    %xmm1, %xmm1
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: paddb    %xmm3, %xmm3
+; SSE41-NEXT: paddb    %xmm1, %xmm1
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw    $5, %xmm1, %xmm1
+; AVX-NEXT:   vpsllw    $4, %xmm0, %xmm2
+; AVX-NEXT:   vpand     {{.*}}(%rip), %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   vpsllw    $2, %xmm0, %xmm2
+; AVX-NEXT:   vpand     {{.*}}(%rip), %xmm2, %xmm2
+; AVX-NEXT:   vpaddb    %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   vpaddb    %xmm0, %xmm0, %xmm2
+; AVX-NEXT:   vpaddb    %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %shl = shl <16 x i8> %r, %a
+  %tmp2 = bitcast <16 x i8> %shl to <2 x i64>
+  ret <2 x i64> %tmp2
+}
+
+define <2 x i64> @ashr_8i16(<8 x i16> %r, <8 x i16> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: sarb
+;
+; SSE2:       psllw   $12, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psraw   $8, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psraw   $4, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psraw   $2, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  psraw   $15, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  pandn   %xmm0, %xmm2
+; SSE2-NEXT:  psraw   $1, %xmm0
+; SSE2-NEXT:  pand    %xmm1, %xmm0
+; SSE2-NEXT:  por     %xmm2, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa    %xmm0, %xmm2
+; SSE41-NEXT: movdqa    %xmm1, %xmm0
+; SSE41-NEXT: psllw     $12, %xmm0
+; SSE41-NEXT: psllw     $4, %xmm1
+; SSE41-NEXT: por       %xmm0, %xmm1
+; SSE41-NEXT: movdqa    %xmm1, %xmm3
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm2, %xmm4
+; SSE41-NEXT: psraw     $8, %xmm4
+; SSE41-NEXT: movdqa    %xmm1, %xmm0
+; SSE41-NEXT: pblendvb  %xmm4, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psraw     $4, %xmm1
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psraw     $2, %xmm1
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psraw     $1, %xmm1
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw    $12, %xmm1, %xmm2
+; AVX-NEXT:   vpsllw    $4, %xmm1, %xmm1
+; AVX-NEXT:   vpor      %xmm2, %xmm1, %xmm1
+; AVX-NEXT:   vpaddw    %xmm1, %xmm1, %xmm2
+; AVX-NEXT:   vpsraw    $8, %xmm0, %xmm3
+; AVX-NEXT:   vpblendvb %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsraw    $4, %xmm0, %xmm1
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsraw    $2, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsraw    $1, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %ashr = ashr <8 x i16> %r, %a
+  %tmp2 = bitcast <8 x i16> %ashr to <2 x i64>
+  ret <2 x i64> %tmp2
+}
+
+define <2 x i64> @ashr_16i8(<16 x i8> %r, <16 x i8> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: sarb
+;
+; SSE2:       punpckhbw {{.*#+}} xmm2 = xmm2[8],xmm0[8],xmm2[9],xmm0[9],xmm2[10],xmm0[10],xmm2[11],xmm0[11],xmm2[12],xmm0[12],xmm2[13],xmm0[13],xmm2[14],xmm0[14],xmm2[15],xmm0[15]
+; SSE2-NEXT:  psllw     $5, %xmm1
+; SSE2-NEXT:  punpckhbw {{.*#+}} xmm4 = xmm4[8],xmm1[8],xmm4[9],xmm1[9],xmm4[10],xmm1[10],xmm4[11],xmm1[11],xmm4[12],xmm1[12],xmm4[13],xmm1[13],xmm4[14],xmm1[14],xmm4[15],xmm1[15]
+; SSE2-NEXT:  pxor      %xmm3, %xmm3
+; SSE2-NEXT:  pxor      %xmm5, %xmm5
+; SSE2-NEXT:  pcmpgtw   %xmm4, %xmm5
+; SSE2-NEXT:  movdqa    %xmm5, %xmm6
+; SSE2-NEXT:  pandn     %xmm2, %xmm6
+; SSE2-NEXT:  psraw     $4, %xmm2
+; SSE2-NEXT:  pand      %xmm5, %xmm2
+; SSE2-NEXT:  por       %xmm6, %xmm2
+; SSE2-NEXT:  paddw     %xmm4, %xmm4
+; SSE2-NEXT:  pxor      %xmm5, %xmm5
+; SSE2-NEXT:  pcmpgtw   %xmm4, %xmm5
+; SSE2-NEXT:  movdqa    %xmm5, %xmm6
+; SSE2-NEXT:  pandn     %xmm2, %xmm6
+; SSE2-NEXT:  psraw     $2, %xmm2
+; SSE2-NEXT:  pand      %xmm5, %xmm2
+; SSE2-NEXT:  por       %xmm6, %xmm2
+; SSE2-NEXT:  paddw     %xmm4, %xmm4
+; SSE2-NEXT:  pxor      %xmm5, %xmm5
+; SSE2-NEXT:  pcmpgtw   %xmm4, %xmm5
+; SSE2-NEXT:  movdqa    %xmm5, %xmm4
+; SSE2-NEXT:  pandn     %xmm2, %xmm4
+; SSE2-NEXT:  psraw     $1, %xmm2
+; SSE2-NEXT:  pand      %xmm5, %xmm2
+; SSE2-NEXT:  por       %xmm4, %xmm2
+; SSE2-NEXT:  psrlw     $8, %xmm2
+; SSE2-NEXT:  punpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:  punpcklbw {{.*#+}} xmm1 = xmm1[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; SSE2-NEXT:  pxor      %xmm4, %xmm4
+; SSE2-NEXT:  pcmpgtw   %xmm1, %xmm4
+; SSE2-NEXT:  movdqa    %xmm4, %xmm5
+; SSE2-NEXT:  pandn     %xmm0, %xmm5
+; SSE2-NEXT:  psraw     $4, %xmm0
+; SSE2-NEXT:  pand      %xmm4, %xmm0
+; SSE2-NEXT:  por       %xmm5, %xmm0
+; SSE2-NEXT:  paddw     %xmm1, %xmm1
+; SSE2-NEXT:  pxor      %xmm4, %xmm4
+; SSE2-NEXT:  pcmpgtw   %xmm1, %xmm4
+; SSE2-NEXT:  movdqa    %xmm4, %xmm5
+; SSE2-NEXT:  pandn     %xmm0, %xmm5
+; SSE2-NEXT:  psraw     $2, %xmm0
+; SSE2-NEXT:  pand      %xmm4, %xmm0
+; SSE2-NEXT:  por       %xmm5, %xmm0
+; SSE2-NEXT:  paddw     %xmm1, %xmm1
+; SSE2-NEXT:  pcmpgtw   %xmm1, %xmm3
+; SSE2-NEXT:  movdqa    %xmm3, %xmm1
+; SSE2-NEXT:  pandn     %xmm0, %xmm1
+; SSE2-NEXT:  psraw     $1, %xmm0
+; SSE2-NEXT:  pand      %xmm3, %xmm0
+; SSE2-NEXT:  por       %xmm1, %xmm0
+; SSE2-NEXT:  psrlw     $8, %xmm0
+; SSE2-NEXT:  packuswb  %xmm2, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa    %xmm0, %xmm2
+; SSE41-NEXT: psllw     $5, %xmm1
+; SSE41-NEXT: punpckhbw {{.*#+}} xmm0 = xmm0[8],xmm1[8],xmm0[9],xmm1[9],xmm0[10],xmm1[10],xmm0[11],xmm1[11],xmm0[12],xmm1[12],xmm0[13],xmm1[13],xmm0[14],xmm1[14],xmm0[15],xmm1[15]
+; SSE41-NEXT: punpckhbw {{.*#+}} xmm3 = xmm3[8],xmm2[8],xmm3[9],xmm2[9],xmm3[10],xmm2[10],xmm3[11],xmm2[11],xmm3[12],xmm2[12],xmm3[13],xmm2[13],xmm3[14],xmm2[14],xmm3[15],xmm2[15]
+; SSE41-NEXT: movdqa    %xmm3, %xmm4
+; SSE41-NEXT: psraw     $4, %xmm4
+; SSE41-NEXT: pblendvb  %xmm4, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm4
+; SSE41-NEXT: psraw     $2, %xmm4
+; SSE41-NEXT: paddw     %xmm0, %xmm0
+; SSE41-NEXT: pblendvb  %xmm4, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm4
+; SSE41-NEXT: psraw     $1, %xmm4
+; SSE41-NEXT: paddw     %xmm0, %xmm0
+; SSE41-NEXT: pblendvb  %xmm4, %xmm3
+; SSE41-NEXT: psrlw     $8, %xmm3
+; SSE41-NEXT: punpcklbw {{.*#+}} xmm0 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3],xmm0[4],xmm1[4],xmm0[5],xmm1[5],xmm0[6],xmm1[6],xmm0[7],xmm1[7]
+; SSE41-NEXT: punpcklbw {{.*#+}} xmm1 = xmm1[0],xmm2[0],xmm1[1],xmm2[1],xmm1[2],xmm2[2],xmm1[3],xmm2[3],xmm1[4],xmm2[4],xmm1[5],xmm2[5],xmm1[6],xmm2[6],xmm1[7],xmm2[7]
+; SSE41-NEXT: movdqa    %xmm1, %xmm2
+; SSE41-NEXT: psraw     $4, %xmm2
+; SSE41-NEXT: pblendvb  %xmm2, %xmm1
+; SSE41-NEXT: movdqa    %xmm1, %xmm2
+; SSE41-NEXT: psraw     $2, %xmm2
+; SSE41-NEXT: paddw     %xmm0, %xmm0
+; SSE41-NEXT: pblendvb  %xmm2, %xmm1
+; SSE41-NEXT: movdqa    %xmm1, %xmm2
+; SSE41-NEXT: psraw     $1, %xmm2
+; SSE41-NEXT: paddw     %xmm0, %xmm0
+; SSE41-NEXT: pblendvb  %xmm2, %xmm1
+; SSE41-NEXT: psrlw     $8, %xmm1
+; SSE41-NEXT: packuswb  %xmm3, %xmm1
+; SSE41-NEXT: movdqa    %xmm1, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw     $5, %xmm1, %xmm1
+; AVX-NEXT:   vpunpckhbw {{.*#+}} xmm2 = xmm0[8],xmm1[8],xmm0[9],xmm1[9],xmm0[10],xmm1[10],xmm0[11],xmm1[11],xmm0[12],xmm1[12],xmm0[13],xmm1[13],xmm0[14],xmm1[14],xmm0[15],xmm1[15]
+; AVX-NEXT:   vpunpckhbw {{.*#+}} xmm3 = xmm0[8,8,9,9,10,10,11,11,12,12,13,13,14,14,15,15]
+; AVX-NEXT:   vpsraw     $4, %xmm3, %xmm4
+; AVX-NEXT:   vpblendvb  %xmm2, %xmm4, %xmm3, %xmm3
+; AVX-NEXT:   vpsraw     $2, %xmm3, %xmm4
+; AVX-NEXT:   vpaddw     %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb  %xmm2, %xmm4, %xmm3, %xmm3
+; AVX-NEXT:   vpsraw     $1, %xmm3, %xmm4
+; AVX-NEXT:   vpaddw     %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb  %xmm2, %xmm4, %xmm3, %xmm2
+; AVX-NEXT:   vpsrlw     $8, %xmm2, %xmm2
+; AVX-NEXT:   vpunpcklbw {{.*#+}} xmm1 = xmm0[0],xmm1[0],xmm0[1],xmm1[1],xmm0[2],xmm1[2],xmm0[3],xmm1[3],xmm0[4],xmm1[4],xmm0[5],xmm1[5],xmm0[6],xmm1[6],xmm0[7],xmm1[7]
+; AVX-NEXT:   vpunpcklbw {{.*#+}} xmm0 = xmm0[0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7]
+; AVX-NEXT:   vpsraw     $4, %xmm0, %xmm3
+; AVX-NEXT:   vpblendvb  %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsraw     $2, %xmm0, %xmm3
+; AVX-NEXT:   vpaddw     %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb  %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsraw     $1, %xmm0, %xmm3
+; AVX-NEXT:   vpaddw     %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb  %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw     $8, %xmm0, %xmm0
+; AVX-NEXT:   vpackuswb  %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %ashr = ashr <16 x i8> %r, %a
+  %tmp2 = bitcast <16 x i8> %ashr to <2 x i64>
+  ret <2 x i64> %tmp2
+}
+
+define <2 x i64> @lshr_8i16(<8 x i16> %r, <8 x i16> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: shrb
+;
+; SSE2:       psllw   $12, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psrlw   $8, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psrlw   $4, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  psraw   $15, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm3
+; SSE2-NEXT:  pandn   %xmm0, %xmm3
+; SSE2-NEXT:  psrlw   $2, %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm3, %xmm0
+; SSE2-NEXT:  paddw   %xmm1, %xmm1
+; SSE2-NEXT:  psraw   $15, %xmm1
+; SSE2-NEXT:  movdqa  %xmm1, %xmm2
+; SSE2-NEXT:  pandn   %xmm0, %xmm2
+; SSE2-NEXT:  psrlw   $1, %xmm0
+; SSE2-NEXT:  pand    %xmm1, %xmm0
+; SSE2-NEXT:  por     %xmm2, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa    %xmm0, %xmm2
+; SSE41-NEXT: movdqa    %xmm1, %xmm0
+; SSE41-NEXT: psllw     $12, %xmm0
+; SSE41-NEXT: psllw     $4, %xmm1
+; SSE41-NEXT: por       %xmm0, %xmm1
+; SSE41-NEXT: movdqa    %xmm1, %xmm3
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm2, %xmm4
+; SSE41-NEXT: psrlw     $8, %xmm4
+; SSE41-NEXT: movdqa    %xmm1, %xmm0
+; SSE41-NEXT: pblendvb  %xmm4, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psrlw     $4, %xmm1
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psrlw     $2, %xmm1
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm1
+; SSE41-NEXT: psrlw     $1, %xmm1
+; SSE41-NEXT: paddw     %xmm3, %xmm3
+; SSE41-NEXT: movdqa    %xmm3, %xmm0
+; SSE41-NEXT: pblendvb  %xmm1, %xmm2
+; SSE41-NEXT: movdqa    %xmm2, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw    $12, %xmm1, %xmm2
+; AVX-NEXT:   vpsllw    $4, %xmm1, %xmm1
+; AVX-NEXT:   vpor      %xmm2, %xmm1, %xmm1
+; AVX-NEXT:   vpaddw    %xmm1, %xmm1, %xmm2
+; AVX-NEXT:   vpsrlw    $8, %xmm0, %xmm3
+; AVX-NEXT:   vpblendvb %xmm1, %xmm3, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw    $4, %xmm0, %xmm1
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw    $2, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw    $1, %xmm0, %xmm1
+; AVX-NEXT:   vpaddw    %xmm2, %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm2, %xmm1, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %lshr = lshr <8 x i16> %r, %a
+  %tmp2 = bitcast <8 x i16> %lshr to <2 x i64>
+  ret <2 x i64> %tmp2
+}
+
+define <2 x i64> @lshr_16i8(<16 x i8> %r, <16 x i8> %a) nounwind readnone ssp {
+entry:
+; ALL-NOT: shrb
+;
+; SSE2:       psllw   $5, %xmm1
+; SSE2-NEXT:  pxor    %xmm2, %xmm2
+; SSE2-NEXT:  pxor    %xmm3, %xmm3
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm3
+; SSE2-NEXT:  movdqa  %xmm3, %xmm4
+; SSE2-NEXT:  pandn   %xmm0, %xmm4
+; SSE2-NEXT:  psrlw   $4, %xmm0
+; SSE2-NEXT:  pand    {{.*}}(%rip), %xmm0
+; SSE2-NEXT:  pand    %xmm3, %xmm0
+; SSE2-NEXT:  por     %xmm4, %xmm0
+; SSE2-NEXT:  paddb   %xmm1, %xmm1
+; SSE2-NEXT:  pxor    %xmm3, %xmm3
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm3
+; SSE2-NEXT:  movdqa  %xmm3, %xmm4
+; SSE2-NEXT:  pandn   %xmm0, %xmm4
+; SSE2-NEXT:  psrlw   $2, %xmm0
+; SSE2-NEXT:  pand    {{.*}}(%rip), %xmm0
+; SSE2-NEXT:  pand    %xmm3, %xmm0
+; SSE2-NEXT:  por     %xmm4, %xmm0
+; SSE2-NEXT:  paddb   %xmm1, %xmm1
+; SSE2-NEXT:  pcmpgtb %xmm1, %xmm2
+; SSE2-NEXT:  movdqa  %xmm2, %xmm1
+; SSE2-NEXT:  pandn   %xmm0, %xmm1
+; SSE2-NEXT:  psrlw   $1, %xmm0
+; SSE2-NEXT:  pand    {{.*}}(%rip), %xmm0
+; SSE2-NEXT:  pand    %xmm2, %xmm0
+; SSE2-NEXT:  por     %xmm1, %xmm0
+; SSE2-NEXT:  retq
+;
+; SSE41:      movdqa   %xmm0, %xmm2
+; SSE41-NEXT: psllw    $5, %xmm1
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: psrlw    $4, %xmm3
+; SSE41-NEXT: pand     {{.*}}(%rip), %xmm3
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: psrlw    $2, %xmm3
+; SSE41-NEXT: pand     {{.*}}(%rip), %xmm3
+; SSE41-NEXT: paddb    %xmm1, %xmm1
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm3
+; SSE41-NEXT: psrlw    $1, %xmm3
+; SSE41-NEXT: pand     {{.*}}(%rip), %xmm3
+; SSE41-NEXT: paddb    %xmm1, %xmm1
+; SSE41-NEXT: movdqa   %xmm1, %xmm0
+; SSE41-NEXT: pblendvb %xmm3, %xmm2
+; SSE41-NEXT: movdqa   %xmm2, %xmm0
+; SSE41-NEXT: retq
+;
+; AVX:        vpsllw    $5, %xmm1, %xmm1
+; AVX-NEXT:   vpsrlw    $4, %xmm0, %xmm2
+; AVX-NEXT:   vpand     {{.*}}(%rip), %xmm2, %xmm2
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw    $2, %xmm0, %xmm2
+; AVX-NEXT:   vpand     {{.*}}(%rip), %xmm2, %xmm2
+; AVX-NEXT:   vpaddb    %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   vpsrlw    $1, %xmm0, %xmm2
+; AVX-NEXT:   vpand     {{.*}}(%rip), %xmm2, %xmm2
+; AVX-NEXT:   vpaddb    %xmm1, %xmm1, %xmm1
+; AVX-NEXT:   vpblendvb %xmm1, %xmm2, %xmm0, %xmm0
+; AVX-NEXT:   retq
+  %lshr = lshr <16 x i8> %r, %a
+  %tmp2 = bitcast <16 x i8> %lshr to <2 x i64>
+  ret <2 x i64> %tmp2
+}