[ARM] MVE Vector Shifts

[dmgreen]

Summary

This is basic shifts for MVE. There are many shifts in MVE, but the ones handled here are:

• VSHL (imm)
• VSHRu (imm)
• VSHRs (imm)
• VSHL (vector)
• VSHL (register)

MVE, like NEON before it, doesn't have shift right by a vector (or register). We instead have to negate the amount and shift in the opposite direction. This means we have to convert any SHR's into a from of SHL (that is still signed or unsigned) with a negated condition and selecting from there. MVE still does have shifting by an immediate for SHL, ASR and LSR.

I've added lowering for these and for register forms, which work well for shift lefts but may require an extra fold of neg(vdup(x)) -> vdup(neg(x)) to optimally for right shifts.

[SjoerdMeijer]

Just a general question first, I was wondering about this:

NEON was previously lowering the shifts into intrinsics and selecting on the intrinsic. I've converted that to a node shared between NEON and MVE which shifts left whilst still storing sign (ARMISD::VSHLs and ARMISD:VSHLu)

Would it perhaps be useful to do this in 2 steps? First the refactoring, the NEON part and the introduction of the nodes, which should be a NFC, and then in a 2nd follow up patch the MVE changes?

[dmgreen]

Sure, I can do that.

[SjoerdMeijer]

Cheers, makes the patches smaller and easier to review too :-)

[dmgreen]

Now sits on top of D64426, this just handles the MVE side of things.

[SjoerdMeijer]

llvm/test/CodeGen/Thumb2/mve-shifts.ll

109 ↗

(On Diff #208737)

Function 'isVShiftLImm' is checking that all constants are the same. Do we have already a test somewhere that checks what happens if e.g. one of the constants is different?

[dmgreen]

Added extra tests

[SjoerdMeijer]

LGTM

llvm/trunk/lib/Target/ARM/ARMISelLowering.cpp

void ARMTargetLowering::addMVEVectorTypes(bool HasMVEFP) {
   const MVT IntTypes[] = { MVT::v16i8, MVT::v8i16, MVT::v4i32 };
 
   for (auto VT : IntTypes) {
     addRegisterClass(VT, &ARM::QPRRegClass);
     setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);
     setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
     setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
+    setOperationAction(ISD::SHL, VT, Custom);
+    setOperationAction(ISD::SRA, VT, Custom);
+    setOperationAction(ISD::SRL, VT, Custom);
     setOperationAction(ISD::SMIN, VT, Legal);
     setOperationAction(ISD::SMAX, VT, Legal);
     setOperationAction(ISD::UMIN, VT, Legal);
     setOperationAction(ISD::UMAX, VT, Legal);
     setOperationAction(ISD::ABS, VT, Legal);
 
     // No native support for these.
     setOperationAction(ISD::UDIV, VT, Expand);
static SDValue LowerShift(SDNode *N, SelectionDAG &DAG,
   EVT VT = N->getValueType(0);
   SDLoc dl(N);
   int64_t Cnt;
 
   if (!VT.isVector())
     return SDValue();
 
   // We essentially have two forms here. Shift by an immediate and shift by a
-  // vector register. We cannot easily match shift by an immediate in tablegen
-  // so we do that here and generate a VSHLIMM/VSHRsIMM/VSHRuIMM.  For shifting
-  // by a vector, we don't have VSHR, only VSHL (which can be signed or
-  // unsigned, and a negative shift indicates a shift right).
+  // vector register (there are also shift by a gpr, but that is just handled
+  // with a tablegen pattern). We cannot easily match shift by an immediate in
+  // tablegen so we do that here and generate a VSHLIMM/VSHRsIMM/VSHRuIMM.
+  // For shifting by a vector, we don't have VSHR, only VSHL (which can be
+  // signed or unsigned, and a negative shift indicates a shift right).
   if (N->getOpcode() == ISD::SHL) {
     if (isVShiftLImm(N->getOperand(1), VT, false, Cnt))
       return DAG.getNode(ARMISD::VSHLIMM, dl, VT, N->getOperand(0),
                          DAG.getConstant(Cnt, dl, MVT::i32));
     return DAG.getNode(ARMISD::VSHLu, dl, VT, N->getOperand(0),
                        N->getOperand(1));
   }
 
if (ST->isThumb1Only() && N->getOpcode() == ISD::SHL && VT == MVT::i32 &&
     }
   }
 
   // Nothing to be done for scalar shifts.
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   if (!VT.isVector() || !TLI.isTypeLegal(VT))
     return SDValue();
 
-  assert(ST->hasNEON() && "unexpected vector shift");
   int64_t Cnt;
 
   switch (N->getOpcode()) {
   default: llvm_unreachable("unexpected shift opcode");
 
   case ISD::SHL:
     if (isVShiftLImm(N->getOperand(1), VT, false, Cnt)) {
       SDLoc dl(N);

llvm/trunk/lib/Target/ARM/ARMInstrInfo.td

 def ARMvgetlaneu : SDNode<"ARMISD::VGETLANEu", SDTARMVGETLN>;
 def ARMvgetlanes : SDNode<"ARMISD::VGETLANEs", SDTARMVGETLN>;
 
 def SDTARMVMOVIMM : SDTypeProfile<1, 1, [SDTCisVec<0>, SDTCisVT<1, i32>]>;
 def ARMvmovImm   : SDNode<"ARMISD::VMOVIMM", SDTARMVMOVIMM>;
 def ARMvmvnImm   : SDNode<"ARMISD::VMVNIMM", SDTARMVMOVIMM>;
 def ARMvmovFPImm : SDNode<"ARMISD::VMOVFPIMM", SDTARMVMOVIMM>;
 
+
+def SDTARMVSHIMM : SDTypeProfile<1, 2, [SDTCisInt<0>, SDTCisSameAs<0, 1>,
+                                        SDTCisVT<2, i32>]>;
+def SDTARMVSH : SDTypeProfile<1, 2, [SDTCisInt<0>, SDTCisSameAs<0, 1>,
+                                     SDTCisSameAs<0, 2>,]>;
+def ARMvshlImm   : SDNode<"ARMISD::VSHLIMM", SDTARMVSHIMM>;
+def ARMvshrsImm  : SDNode<"ARMISD::VSHRsIMM", SDTARMVSHIMM>;
+def ARMvshruImm  : SDNode<"ARMISD::VSHRuIMM", SDTARMVSHIMM>;
+def ARMvshls     : SDNode<"ARMISD::VSHLs", SDTARMVSH>;
+def ARMvshlu     : SDNode<"ARMISD::VSHLu", SDTARMVSH>;
+
 def ARMWLS : SDNode<"ARMISD::WLS", SDT_ARMWhileLoop,
                     [SDNPHasChain]>;
 
 //===----------------------------------------------------------------------===//
 // ARM Flag Definitions.
 
 class RegConstraint<string C> {
   string Constraints = C;

llvm/trunk/lib/Target/ARM/ARMInstrMVE.td

multiclass mve_shift_by_vec_multi<string iname, bit bit_4, bit bit_8> {
   def u32 : MVE_shift_by_vec<iname, "u32", 0b1, 0b10, bit_4, bit_8>;
 }
 
 defm MVE_VSHL_by_vec   : mve_shift_by_vec_multi<"vshl",   0b0, 0b0>;
 defm MVE_VQSHL_by_vec  : mve_shift_by_vec_multi<"vqshl",  0b1, 0b0>;
 defm MVE_VQRSHL_by_vec : mve_shift_by_vec_multi<"vqrshl", 0b1, 0b1>;
 defm MVE_VRSHL_by_vec  : mve_shift_by_vec_multi<"vrshl",  0b0, 0b1>;
 
+let Predicates = [HasMVEInt] in {
+  def : Pat<(v4i32 (ARMvshlu (v4i32 MQPR:$Qm), (v4i32 MQPR:$Qn))),
+            (v4i32 (MVE_VSHL_by_vecu32 (v4i32 MQPR:$Qm), (v4i32 MQPR:$Qn)))>;
+  def : Pat<(v8i16 (ARMvshlu (v8i16 MQPR:$Qm), (v8i16 MQPR:$Qn))),
+            (v8i16 (MVE_VSHL_by_vecu16 (v8i16 MQPR:$Qm), (v8i16 MQPR:$Qn)))>;
+  def : Pat<(v16i8 (ARMvshlu (v16i8 MQPR:$Qm), (v16i8 MQPR:$Qn))),
+            (v16i8 (MVE_VSHL_by_vecu8 (v16i8 MQPR:$Qm), (v16i8 MQPR:$Qn)))>;
+
+  def : Pat<(v4i32 (ARMvshls (v4i32 MQPR:$Qm), (v4i32 MQPR:$Qn))),
+            (v4i32 (MVE_VSHL_by_vecs32 (v4i32 MQPR:$Qm), (v4i32 MQPR:$Qn)))>;
+  def : Pat<(v8i16 (ARMvshls (v8i16 MQPR:$Qm), (v8i16 MQPR:$Qn))),
+            (v8i16 (MVE_VSHL_by_vecs16 (v8i16 MQPR:$Qm), (v8i16 MQPR:$Qn)))>;
+  def : Pat<(v16i8 (ARMvshls (v16i8 MQPR:$Qm), (v16i8 MQPR:$Qn))),
+            (v16i8 (MVE_VSHL_by_vecs8 (v16i8 MQPR:$Qm), (v16i8 MQPR:$Qn)))>;
+}
+
 class MVE_shift_with_imm<string iname, string suffix, dag oops, dag iops,
                          string ops, vpred_ops vpred, string cstr,
                          list<dag> pattern=[]>
   : MVE_p<oops, iops, NoItinerary, iname, suffix, ops, vpred, cstr, pattern> {
   bits<4> Qd;
   bits<4> Qm;
 
   let Inst{23} = 0b1;

 def MVE_VSHL_immi16 : MVE_VSHL_imm<"i16", (ins imm0_15:$imm)> {
   let Inst{21-20} = 0b01;
 }
 
 def MVE_VSHL_immi32 : MVE_VSHL_imm<"i32", (ins imm0_31:$imm)> {
   let Inst{21} = 0b1;
 }
 
+let Predicates = [HasMVEInt] in {
+  def : Pat<(v4i32 (ARMvshlImm (v4i32 MQPR:$src), imm0_31:$imm)),
+            (v4i32 (MVE_VSHL_immi32 (v4i32 MQPR:$src), imm0_31:$imm))>;
+  def : Pat<(v8i16 (ARMvshlImm (v8i16 MQPR:$src), imm0_15:$imm)),
+            (v8i16 (MVE_VSHL_immi16 (v8i16 MQPR:$src), imm0_15:$imm))>;
+  def : Pat<(v16i8 (ARMvshlImm (v16i8 MQPR:$src), imm0_7:$imm)),
+            (v16i8 (MVE_VSHL_immi8 (v16i8 MQPR:$src), imm0_7:$imm))>;
+
+  def : Pat<(v4i32 (ARMvshruImm (v4i32 MQPR:$src), imm0_31:$imm)),
+            (v4i32 (MVE_VSHR_immu32 (v4i32 MQPR:$src), imm0_31:$imm))>;
+  def : Pat<(v8i16 (ARMvshruImm (v8i16 MQPR:$src), imm0_15:$imm)),
+            (v8i16 (MVE_VSHR_immu16 (v8i16 MQPR:$src), imm0_15:$imm))>;
+  def : Pat<(v16i8 (ARMvshruImm (v16i8 MQPR:$src), imm0_7:$imm)),
+            (v16i8 (MVE_VSHR_immu8 (v16i8 MQPR:$src), imm0_7:$imm))>;
+
+  def : Pat<(v4i32 (ARMvshrsImm (v4i32 MQPR:$src), imm0_31:$imm)),
+            (v4i32 (MVE_VSHR_imms32 (v4i32 MQPR:$src), imm0_31:$imm))>;
+  def : Pat<(v8i16 (ARMvshrsImm (v8i16 MQPR:$src), imm0_15:$imm)),
+            (v8i16 (MVE_VSHR_imms16 (v8i16 MQPR:$src), imm0_15:$imm))>;
+  def : Pat<(v16i8 (ARMvshrsImm (v16i8 MQPR:$src), imm0_7:$imm)),
+            (v16i8 (MVE_VSHR_imms8 (v16i8 MQPR:$src), imm0_7:$imm))>;
+}
+
 // end of mve_shift instructions
 
 // start of MVE Floating Point instructions
 
 class MVE_float<string iname, string suffix, dag oops, dag iops, string ops,
                 vpred_ops vpred, string cstr, list<dag> pattern=[]>
   : MVE_f<oops, iops, NoItinerary, iname, suffix, ops, vpred, cstr, pattern> {
   bits<4> Qm;
multiclass MVE_VxSHL_qr_types<string iname, bit bit_7, bit bit_17> {
   def u32 : MVE_VxSHL_qr<iname, "u32", 0b1, 0b10, bit_7, bit_17>;
 }
 
 defm MVE_VSHL_qr   : MVE_VxSHL_qr_types<"vshl",   0b0, 0b0>;
 defm MVE_VRSHL_qr  : MVE_VxSHL_qr_types<"vrshl",  0b0, 0b1>;
 defm MVE_VQSHL_qr  : MVE_VxSHL_qr_types<"vqshl",  0b1, 0b0>;
 defm MVE_VQRSHL_qr : MVE_VxSHL_qr_types<"vqrshl", 0b1, 0b1>;
 
+let Predicates = [HasMVEInt] in {
+  def : Pat<(v4i32 (ARMvshlu (v4i32 MQPR:$Qm), (v4i32 (ARMvdup GPR:$Rm)))),
+            (v4i32 (MVE_VSHL_qru32 (v4i32 MQPR:$Qm), GPR:$Rm))>;
+  def : Pat<(v8i16 (ARMvshlu (v8i16 MQPR:$Qm), (v8i16 (ARMvdup GPR:$Rm)))),
+            (v8i16 (MVE_VSHL_qru16 (v8i16 MQPR:$Qm), GPR:$Rm))>;
+  def : Pat<(v16i8 (ARMvshlu (v16i8 MQPR:$Qm), (v16i8 (ARMvdup GPR:$Rm)))),
+            (v16i8 (MVE_VSHL_qru8 (v16i8 MQPR:$Qm), GPR:$Rm))>;
+
+  def : Pat<(v4i32 (ARMvshls (v4i32 MQPR:$Qm), (v4i32 (ARMvdup GPR:$Rm)))),
+            (v4i32 (MVE_VSHL_qrs32 (v4i32 MQPR:$Qm), GPR:$Rm))>;
+  def : Pat<(v8i16 (ARMvshls (v8i16 MQPR:$Qm), (v8i16 (ARMvdup GPR:$Rm)))),
+            (v8i16 (MVE_VSHL_qrs16 (v8i16 MQPR:$Qm), GPR:$Rm))>;
+  def : Pat<(v16i8 (ARMvshls (v16i8 MQPR:$Qm), (v16i8 (ARMvdup GPR:$Rm)))),
+            (v16i8 (MVE_VSHL_qrs8 (v16i8 MQPR:$Qm), GPR:$Rm))>;
+}
+
 class MVE_VBRSR<string iname, string suffix, bits<2> size, list<dag> pattern=[]>
   : MVE_qDest_rSrc<iname, suffix, pattern> {
 
   let Inst{28} = 0b1;
   let Inst{21-20} = size;
   let Inst{16} = 0b1;
   let Inst{12} = 0b1;
   let Inst{8} = 0b0;

llvm/trunk/lib/Target/ARM/ARMInstrNEON.td

 def NEONvclez     : SDNode<"ARMISD::VCLEZ", SDTARMVCMPZ>;
 def NEONvcgeu     : SDNode<"ARMISD::VCGEU", SDTARMVCMP>;
 def NEONvcgt      : SDNode<"ARMISD::VCGT", SDTARMVCMP>;
 def NEONvcgtz     : SDNode<"ARMISD::VCGTZ", SDTARMVCMPZ>;
 def NEONvcltz     : SDNode<"ARMISD::VCLTZ", SDTARMVCMPZ>;
 def NEONvcgtu     : SDNode<"ARMISD::VCGTU", SDTARMVCMP>;
 def NEONvtst      : SDNode<"ARMISD::VTST", SDTARMVCMP>;
 
-// Vector Shifts
-def SDTARMVSH : SDTypeProfile<1, 2, [SDTCisInt<0>, SDTCisSameAs<0, 1>,
-                                     SDTCisSameAs<0, 2>,]>;
-
-def NEONvshls        : SDNode<"ARMISD::VSHLs", SDTARMVSH>;
-def NEONvshlu        : SDNode<"ARMISD::VSHLu", SDTARMVSH>;
-
 // Types for vector shift by immediates.  The "SHX" version is for long and
 // narrow operations where the source and destination vectors have different
 // types.  The "SHINS" version is for shift and insert operations.
-def SDTARMVSHIMM     : SDTypeProfile<1, 2, [SDTCisInt<0>, SDTCisSameAs<0, 1>,
-                                            SDTCisVT<2, i32>]>;
 def SDTARMVSHXIMM    : SDTypeProfile<1, 2, [SDTCisInt<0>, SDTCisInt<1>,
                                             SDTCisVT<2, i32>]>;
 def SDTARMVSHINSIMM  : SDTypeProfile<1, 3, [SDTCisInt<0>, SDTCisSameAs<0, 1>,
                                             SDTCisSameAs<0, 2>, SDTCisVT<3, i32>]>;
 
-def NEONvshlImm      : SDNode<"ARMISD::VSHLIMM", SDTARMVSHIMM>;
-def NEONvshrsImm     : SDNode<"ARMISD::VSHRsIMM", SDTARMVSHIMM>;
-def NEONvshruImm     : SDNode<"ARMISD::VSHRuIMM", SDTARMVSHIMM>;
 def NEONvshrnImm     : SDNode<"ARMISD::VSHRNIMM", SDTARMVSHXIMM>;
 
 def NEONvrshrsImm    : SDNode<"ARMISD::VRSHRsIMM", SDTARMVSHIMM>;
 def NEONvrshruImm    : SDNode<"ARMISD::VRSHRuIMM", SDTARMVSHIMM>;
 def NEONvrshrnImm    : SDNode<"ARMISD::VRSHRNIMM", SDTARMVSHXIMM>;
 
 def NEONvqshlsImm    : SDNode<"ARMISD::VQSHLsIMM", SDTARMVSHIMM>;
 def NEONvqshluImm    : SDNode<"ARMISD::VQSHLuIMM", SDTARMVSHIMM>;
defm VQADDu   : N3VInt_QHSD<1, 0, 0b0000, 1, N3RegFrm,
                             "vqadd", "u", int_arm_neon_vqaddu, 1>;
 //   VADDHN   : Vector Add and Narrow Returning High Half (D = Q + Q)
 defm VADDHN   : N3VNInt_HSD<0,1,0b0100,0, "vaddhn", "i", null_frag, 1>;
 //   VRADDHN  : Vector Rounding Add and Narrow Returning High Half (D = Q + Q)
 defm VRADDHN  : N3VNInt_HSD<1,1,0b0100,0, "vraddhn", "i",
                             int_arm_neon_vraddhn, 1>;
 
 let Predicates = [HasNEON] in {
-def : Pat<(v8i8  (trunc (NEONvshruImm (add (v8i16 QPR:$Vn), QPR:$Vm), 8))),
+def : Pat<(v8i8  (trunc (ARMvshruImm (add (v8i16 QPR:$Vn), QPR:$Vm), 8))),
           (VADDHNv8i8 QPR:$Vn, QPR:$Vm)>;
-def : Pat<(v4i16 (trunc (NEONvshruImm (add (v4i32 QPR:$Vn), QPR:$Vm), 16))),
+def : Pat<(v4i16 (trunc (ARMvshruImm (add (v4i32 QPR:$Vn), QPR:$Vm), 16))),
           (VADDHNv4i16 QPR:$Vn, QPR:$Vm)>;
-def : Pat<(v2i32 (trunc (NEONvshruImm (add (v2i64 QPR:$Vn), QPR:$Vm), 32))),
+def : Pat<(v2i32 (trunc (ARMvshruImm (add (v2i64 QPR:$Vn), QPR:$Vm), 32))),
           (VADDHNv2i32 QPR:$Vn, QPR:$Vm)>;
 }
 
 // Vector Multiply Operations.
 
 //   VMUL     : Vector Multiply (integer, polynomial and floating-point)
 defm VMUL     : N3V_QHS<0, 0, 0b1001, 1, IIC_VMULi16D, IIC_VMULi32D,
                         IIC_VMULi16Q, IIC_VMULi32Q, "vmul", "i", mul, 1>;
defm VQSUBu   : N3VInt_QHSD<1, 0, 0b0010, 1, N3RegFrm,
                             "vqsub", "u", int_arm_neon_vqsubu, 0>;
 //   VSUBHN   : Vector Subtract and Narrow Returning High Half (D = Q - Q)
 defm VSUBHN   : N3VNInt_HSD<0,1,0b0110,0, "vsubhn", "i", null_frag, 0>;
 //   VRSUBHN  : Vector Rounding Subtract and Narrow Returning High Half (D=Q-Q)
 defm VRSUBHN  : N3VNInt_HSD<1,1,0b0110,0, "vrsubhn", "i",
                             int_arm_neon_vrsubhn, 0>;
 
 let Predicates = [HasNEON] in {
-def : Pat<(v8i8  (trunc (NEONvshruImm (sub (v8i16 QPR:$Vn), QPR:$Vm), 8))),
+def : Pat<(v8i8  (trunc (ARMvshruImm (sub (v8i16 QPR:$Vn), QPR:$Vm), 8))),
           (VSUBHNv8i8 QPR:$Vn, QPR:$Vm)>;
-def : Pat<(v4i16 (trunc (NEONvshruImm (sub (v4i32 QPR:$Vn), QPR:$Vm), 16))),
+def : Pat<(v4i16 (trunc (ARMvshruImm (sub (v4i32 QPR:$Vn), QPR:$Vm), 16))),
           (VSUBHNv4i16 QPR:$Vn, QPR:$Vm)>;
-def : Pat<(v2i32 (trunc (NEONvshruImm (sub (v2i64 QPR:$Vn), QPR:$Vm), 32))),
+def : Pat<(v2i32 (trunc (ARMvshruImm (sub (v2i64 QPR:$Vn), QPR:$Vm), 32))),
           (VSUBHNv2i32 QPR:$Vn, QPR:$Vm)>;
 }
 
 // Vector Comparisons.
 
 //   VCEQ     : Vector Compare Equal
 defm VCEQ     : N3V_QHS<1, 0, 0b1000, 1, IIC_VSUBi4D, IIC_VSUBi4D, IIC_VSUBi4Q,
                         IIC_VSUBi4Q, "vceq", "i", NEONvceq, 1>;
def : Pat<(v4i32 (abs (sub (zext (v4i16 DPR:$opA)), (zext (v4i16 DPR:$opB))))),
           (VABDLuv4i32 DPR:$opA, DPR:$opB)>;
 }
 
 // ISD::ABS is not legal for v2i64, so VABDL needs to be matched from the
 // shift/xor pattern for ABS.
 
 def abd_shr :
     PatFrag<(ops node:$in1, node:$in2, node:$shift),
-            (NEONvshrsImm (sub (zext node:$in1),
+            (ARMvshrsImm (sub (zext node:$in1),
                             (zext node:$in2)), (i32 $shift))>;
 
 let Predicates = [HasNEON] in {
 def : Pat<(xor (v4i32 (bitconvert (v2i64 (abd_shr (v2i32 DPR:$opA), (v2i32 DPR:$opB), 63)))),
                (v4i32 (bitconvert (v2i64 (add (sub (zext (v2i32 DPR:$opA)),
                                                    (zext (v2i32 DPR:$opB))),
                                          (abd_shr (v2i32 DPR:$opA), (v2i32 DPR:$opB), 63)))))),
           (VABDLuv2i64 DPR:$opA, DPR:$opB)>;

 defm VSHLs    : N3VInt_QHSDSh<0, 0, 0b0100, 0, N3RegVShFrm,
                             IIC_VSHLiD, IIC_VSHLiD, IIC_VSHLiQ, IIC_VSHLiQ,
                             "vshl", "s", int_arm_neon_vshifts>;
 defm VSHLu    : N3VInt_QHSDSh<1, 0, 0b0100, 0, N3RegVShFrm,
                             IIC_VSHLiD, IIC_VSHLiD, IIC_VSHLiQ, IIC_VSHLiQ,
                             "vshl", "u", int_arm_neon_vshiftu>;
 
 let Predicates = [HasNEON] in {
-def : Pat<(v8i8 (NEONvshls (v8i8 DPR:$Dn), (v8i8 DPR:$Dm))),
+def : Pat<(v8i8 (ARMvshls (v8i8 DPR:$Dn), (v8i8 DPR:$Dm))),
           (VSHLsv8i8 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v4i16 (NEONvshls (v4i16 DPR:$Dn), (v4i16 DPR:$Dm))),
+def : Pat<(v4i16 (ARMvshls (v4i16 DPR:$Dn), (v4i16 DPR:$Dm))),
           (VSHLsv4i16 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v2i32 (NEONvshls (v2i32 DPR:$Dn), (v2i32 DPR:$Dm))),
+def : Pat<(v2i32 (ARMvshls (v2i32 DPR:$Dn), (v2i32 DPR:$Dm))),
           (VSHLsv2i32 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v1i64 (NEONvshls (v1i64 DPR:$Dn), (v1i64 DPR:$Dm))),
+def : Pat<(v1i64 (ARMvshls (v1i64 DPR:$Dn), (v1i64 DPR:$Dm))),
           (VSHLsv1i64 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v16i8 (NEONvshls (v16i8 QPR:$Dn), (v16i8 QPR:$Dm))),
+def : Pat<(v16i8 (ARMvshls (v16i8 QPR:$Dn), (v16i8 QPR:$Dm))),
           (VSHLsv16i8 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v8i16 (NEONvshls (v8i16 QPR:$Dn), (v8i16 QPR:$Dm))),
+def : Pat<(v8i16 (ARMvshls (v8i16 QPR:$Dn), (v8i16 QPR:$Dm))),
           (VSHLsv8i16 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v4i32 (NEONvshls (v4i32 QPR:$Dn), (v4i32 QPR:$Dm))),
+def : Pat<(v4i32 (ARMvshls (v4i32 QPR:$Dn), (v4i32 QPR:$Dm))),
           (VSHLsv4i32 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v2i64 (NEONvshls (v2i64 QPR:$Dn), (v2i64 QPR:$Dm))),
+def : Pat<(v2i64 (ARMvshls (v2i64 QPR:$Dn), (v2i64 QPR:$Dm))),
           (VSHLsv2i64 QPR:$Dn, QPR:$Dm)>;
 
-def : Pat<(v8i8 (NEONvshlu (v8i8 DPR:$Dn), (v8i8 DPR:$Dm))),
+def : Pat<(v8i8 (ARMvshlu (v8i8 DPR:$Dn), (v8i8 DPR:$Dm))),
           (VSHLuv8i8 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v4i16 (NEONvshlu (v4i16 DPR:$Dn), (v4i16 DPR:$Dm))),
+def : Pat<(v4i16 (ARMvshlu (v4i16 DPR:$Dn), (v4i16 DPR:$Dm))),
           (VSHLuv4i16 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v2i32 (NEONvshlu (v2i32 DPR:$Dn), (v2i32 DPR:$Dm))),
+def : Pat<(v2i32 (ARMvshlu (v2i32 DPR:$Dn), (v2i32 DPR:$Dm))),
           (VSHLuv2i32 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v1i64 (NEONvshlu (v1i64 DPR:$Dn), (v1i64 DPR:$Dm))),
+def : Pat<(v1i64 (ARMvshlu (v1i64 DPR:$Dn), (v1i64 DPR:$Dm))),
           (VSHLuv1i64 DPR:$Dn, DPR:$Dm)>;
-def : Pat<(v16i8 (NEONvshlu (v16i8 QPR:$Dn), (v16i8 QPR:$Dm))),
+def : Pat<(v16i8 (ARMvshlu (v16i8 QPR:$Dn), (v16i8 QPR:$Dm))),
           (VSHLuv16i8 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v8i16 (NEONvshlu (v8i16 QPR:$Dn), (v8i16 QPR:$Dm))),
+def : Pat<(v8i16 (ARMvshlu (v8i16 QPR:$Dn), (v8i16 QPR:$Dm))),
           (VSHLuv8i16 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v4i32 (NEONvshlu (v4i32 QPR:$Dn), (v4i32 QPR:$Dm))),
+def : Pat<(v4i32 (ARMvshlu (v4i32 QPR:$Dn), (v4i32 QPR:$Dm))),
           (VSHLuv4i32 QPR:$Dn, QPR:$Dm)>;
-def : Pat<(v2i64 (NEONvshlu (v2i64 QPR:$Dn), (v2i64 QPR:$Dm))),
+def : Pat<(v2i64 (ARMvshlu (v2i64 QPR:$Dn), (v2i64 QPR:$Dm))),
           (VSHLuv2i64 QPR:$Dn, QPR:$Dm)>;
 
 }
 
 //   VSHL     : Vector Shift Left (Immediate)
-defm VSHLi    : N2VShL_QHSD<0, 1, 0b0101, 1, IIC_VSHLiD, "vshl", "i", NEONvshlImm>;
+defm VSHLi    : N2VShL_QHSD<0, 1, 0b0101, 1, IIC_VSHLiD, "vshl", "i", ARMvshlImm>;
 
 //   VSHR     : Vector Shift Right (Immediate)
 defm VSHRs    : N2VShR_QHSD<0, 1, 0b0000, 1, IIC_VSHLiD, "vshr", "s", "VSHRs",
-                            NEONvshrsImm>;
+                            ARMvshrsImm>;
 defm VSHRu    : N2VShR_QHSD<1, 1, 0b0000, 1, IIC_VSHLiD, "vshr", "u", "VSHRu",
-                            NEONvshruImm>;
+                            ARMvshruImm>;
 
 //   VSHLL    : Vector Shift Left Long
 defm VSHLLs   : N2VLSh_QHS<0, 1, 0b1010, 0, 0, 1, "vshll", "s",
-  PatFrag<(ops node:$LHS, node:$RHS), (NEONvshlImm (sext node:$LHS), node:$RHS)>>;
+  PatFrag<(ops node:$LHS, node:$RHS), (ARMvshlImm (sext node:$LHS), node:$RHS)>>;
 defm VSHLLu   : N2VLSh_QHS<1, 1, 0b1010, 0, 0, 1, "vshll", "u",
-  PatFrag<(ops node:$LHS, node:$RHS), (NEONvshlImm (zext node:$LHS), node:$RHS)>>;
+  PatFrag<(ops node:$LHS, node:$RHS), (ARMvshlImm (zext node:$LHS), node:$RHS)>>;
 
 //   VSHLL    : Vector Shift Left Long (with maximum shift count)
 class N2VLShMax<bit op24, bit op23, bits<6> op21_16, bits<4> op11_8, bit op7,
                 bit op6, bit op4, string OpcodeStr, string Dt, ValueType ResTy,
                 ValueType OpTy, Operand ImmTy>
   : N2VLSh<op24, op23, op11_8, op7, op6, op4, OpcodeStr, Dt,
            ResTy, OpTy, ImmTy, null_frag> {
   let Inst{21-16} = op21_16;
   let DecoderMethod = "DecodeVSHLMaxInstruction";
 }
 def  VSHLLi8  : N2VLShMax<1, 1, 0b110010, 0b0011, 0, 0, 0, "vshll", "i8",
                           v8i16, v8i8, imm8>;
 def  VSHLLi16 : N2VLShMax<1, 1, 0b110110, 0b0011, 0, 0, 0, "vshll", "i16",
                           v4i32, v4i16, imm16>;
 def  VSHLLi32 : N2VLShMax<1, 1, 0b111010, 0b0011, 0, 0, 0, "vshll", "i32",
                           v2i64, v2i32, imm32>;
 
 let Predicates = [HasNEON] in {
-def : Pat<(v8i16 (NEONvshlImm (zext (v8i8 DPR:$Rn)), (i32 8))),
+def : Pat<(v8i16 (ARMvshlImm (zext (v8i8 DPR:$Rn)), (i32 8))),
           (VSHLLi8 DPR:$Rn, 8)>;
-def : Pat<(v4i32 (NEONvshlImm (zext (v4i16 DPR:$Rn)), (i32 16))),
+def : Pat<(v4i32 (ARMvshlImm (zext (v4i16 DPR:$Rn)), (i32 16))),
           (VSHLLi16 DPR:$Rn, 16)>;
-def : Pat<(v2i64 (NEONvshlImm (zext (v2i32 DPR:$Rn)), (i32 32))),
+def : Pat<(v2i64 (ARMvshlImm (zext (v2i32 DPR:$Rn)), (i32 32))),
           (VSHLLi32 DPR:$Rn, 32)>;
-def : Pat<(v8i16 (NEONvshlImm (sext (v8i8 DPR:$Rn)), (i32 8))),
+def : Pat<(v8i16 (ARMvshlImm (sext (v8i8 DPR:$Rn)), (i32 8))),
           (VSHLLi8 DPR:$Rn, 8)>;
-def : Pat<(v4i32 (NEONvshlImm (sext (v4i16 DPR:$Rn)), (i32 16))),
+def : Pat<(v4i32 (ARMvshlImm (sext (v4i16 DPR:$Rn)), (i32 16))),
           (VSHLLi16 DPR:$Rn, 16)>;
-def : Pat<(v2i64 (NEONvshlImm (sext (v2i32 DPR:$Rn)), (i32 32))),
+def : Pat<(v2i64 (ARMvshlImm (sext (v2i32 DPR:$Rn)), (i32 32))),
           (VSHLLi32 DPR:$Rn, 32)>;
-def : Pat<(v8i16 (NEONvshlImm (anyext (v8i8 DPR:$Rn)), (i32 8))),
+def : Pat<(v8i16 (ARMvshlImm (anyext (v8i8 DPR:$Rn)), (i32 8))),
           (VSHLLi8 DPR:$Rn, 8)>;
-def : Pat<(v4i32 (NEONvshlImm (anyext (v4i16 DPR:$Rn)), (i32 16))),
+def : Pat<(v4i32 (ARMvshlImm (anyext (v4i16 DPR:$Rn)), (i32 16))),
           (VSHLLi16 DPR:$Rn, 16)>;
-def : Pat<(v2i64 (NEONvshlImm (anyext (v2i32 DPR:$Rn)), (i32 32))),
+def : Pat<(v2i64 (ARMvshlImm (anyext (v2i32 DPR:$Rn)), (i32 32))),
           (VSHLLi32 DPR:$Rn, 32)>;
 }
 
 //   VSHRN    : Vector Shift Right and Narrow
 defm VSHRN    : N2VNSh_HSD<0,1,0b1000,0,0,1, IIC_VSHLiD, "vshrn", "i",
                            PatFrag<(ops node:$Rn, node:$amt),
-                                   (trunc (NEONvshrsImm node:$Rn, node:$amt))>>;
+                                   (trunc (ARMvshrsImm node:$Rn, node:$amt))>>;
 
 let Predicates = [HasNEON] in {
-def : Pat<(v8i8 (trunc (NEONvshruImm (v8i16 QPR:$Vn), shr_imm8:$amt))),
+def : Pat<(v8i8 (trunc (ARMvshruImm (v8i16 QPR:$Vn), shr_imm8:$amt))),
           (VSHRNv8i8 QPR:$Vn, shr_imm8:$amt)>;
-def : Pat<(v4i16 (trunc (NEONvshruImm (v4i32 QPR:$Vn), shr_imm16:$amt))),
+def : Pat<(v4i16 (trunc (ARMvshruImm (v4i32 QPR:$Vn), shr_imm16:$amt))),
           (VSHRNv4i16 QPR:$Vn, shr_imm16:$amt)>;
-def : Pat<(v2i32 (trunc (NEONvshruImm (v2i64 QPR:$Vn), shr_imm32:$amt))),
+def : Pat<(v2i32 (trunc (ARMvshruImm (v2i64 QPR:$Vn), shr_imm32:$amt))),
           (VSHRNv2i32 QPR:$Vn, shr_imm32:$amt)>;
 }
 
 //   VRSHL    : Vector Rounding Shift
 defm VRSHLs   : N3VInt_QHSDSh<0, 0, 0b0101, 0, N3RegVShFrm,
                             IIC_VSHLi4D, IIC_VSHLi4D, IIC_VSHLi4Q, IIC_VSHLi4Q,
                             "vrshl", "s", int_arm_neon_vrshifts>;
 defm VRSHLu   : N3VInt_QHSDSh<1, 0, 0b0101, 0, N3RegVShFrm,

 defm VQRSHRNu : N2VNSh_HSD<1, 1, 0b1001, 0, 1, 1, IIC_VSHLi4D, "vqrshrn", "u",
                            NEONvqrshrnuImm>;
 
 //   VQRSHRUN : Vector Saturating Rounding Shift Right and Narrow (Unsigned)
 defm VQRSHRUN : N2VNSh_HSD<1, 1, 0b1000, 0, 1, 1, IIC_VSHLi4D, "vqrshrun", "s",
                            NEONvqrshrnsuImm>;
 
 //   VSRA     : Vector Shift Right and Accumulate
-defm VSRAs    : N2VShAdd_QHSD<0, 1, 0b0001, 1, "vsra", "s", NEONvshrsImm>;
-defm VSRAu    : N2VShAdd_QHSD<1, 1, 0b0001, 1, "vsra", "u", NEONvshruImm>;
+defm VSRAs    : N2VShAdd_QHSD<0, 1, 0b0001, 1, "vsra", "s", ARMvshrsImm>;
+defm VSRAu    : N2VShAdd_QHSD<1, 1, 0b0001, 1, "vsra", "u", ARMvshruImm>;
 //   VRSRA    : Vector Rounding Shift Right and Accumulate
 defm VRSRAs   : N2VShAdd_QHSD<0, 1, 0b0011, 1, "vrsra", "s", NEONvrshrsImm>;
 defm VRSRAu   : N2VShAdd_QHSD<1, 1, 0b0011, 1, "vrsra", "u", NEONvrshruImm>;
 
 //   VSLI     : Vector Shift Left and Insert
 defm VSLI     : N2VShInsL_QHSD<1, 1, 0b0101, 1, "vsli">;
 
 //   VSRI     : Vector Shift Right and Insert

llvm/trunk/test/CodeGen/Thumb2/mve-shifts.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -O3 -mtriple=thumbv8.1m.main-arm-none-eabi -mattr=+mve %s -o - | FileCheck %s
+
+define arm_aapcs_vfpcc <16 x i8> @shl_qq_int8_t(<16 x i8> %src1, <16 x i8> %src2) {
+; CHECK-LABEL: shl_qq_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <16 x i8> %src1, %src2
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shl_qq_int16_t(<8 x i16> %src1, <8 x i16> %src2) {
+; CHECK-LABEL: shl_qq_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <8 x i16> %src1, %src2
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shl_qq_int32_t(<4 x i32> %src1, <4 x i32> %src2) {
+; CHECK-LABEL: shl_qq_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <4 x i32> %src1, %src2
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shru_qq_int8_t(<16 x i8> %src1, <16 x i8> %src2) {
+; CHECK-LABEL: shru_qq_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s8 q1, q1
+; CHECK-NEXT:    vshl.u8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <16 x i8> %src1, %src2
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shru_qq_int16_t(<8 x i16> %src1, <8 x i16> %src2) {
+; CHECK-LABEL: shru_qq_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s16 q1, q1
+; CHECK-NEXT:    vshl.u16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <8 x i16> %src1, %src2
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shru_qq_int32_t(<4 x i32> %src1, <4 x i32> %src2) {
+; CHECK-LABEL: shru_qq_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s32 q1, q1
+; CHECK-NEXT:    vshl.u32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <4 x i32> %src1, %src2
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shrs_qq_int8_t(<16 x i8> %src1, <16 x i8> %src2) {
+; CHECK-LABEL: shrs_qq_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s8 q1, q1
+; CHECK-NEXT:    vshl.s8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <16 x i8> %src1, %src2
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shrs_qq_int16_t(<8 x i16> %src1, <8 x i16> %src2) {
+; CHECK-LABEL: shrs_qq_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s16 q1, q1
+; CHECK-NEXT:    vshl.s16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <8 x i16> %src1, %src2
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shrs_qq_int32_t(<4 x i32> %src1, <4 x i32> %src2) {
+; CHECK-LABEL: shrs_qq_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vneg.s32 q1, q1
+; CHECK-NEXT:    vshl.s32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <4 x i32> %src1, %src2
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shl_qi_int8_t(<16 x i8> %src1) {
+; CHECK-LABEL: shl_qi_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.i8 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <16 x i8> %src1, <i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4>
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shl_qi_int16_t(<8 x i16> %src1) {
+; CHECK-LABEL: shl_qi_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.i16 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <8 x i16> %src1, <i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4>
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shl_qi_int32_t(<4 x i32> %src1) {
+; CHECK-LABEL: shl_qi_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.i32 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = shl <4 x i32> %src1, <i32 4, i32 4, i32 4, i32 4>
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shru_qi_int8_t(<16 x i8> %src1) {
+; CHECK-LABEL: shru_qi_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.u8 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <16 x i8> %src1, <i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4>
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shru_qi_int16_t(<8 x i16> %src1) {
+; CHECK-LABEL: shru_qi_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.u16 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <8 x i16> %src1, <i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4>
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shru_qi_int32_t(<4 x i32> %src1) {
+; CHECK-LABEL: shru_qi_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.u32 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = lshr <4 x i32> %src1, <i32 4, i32 4, i32 4, i32 4>
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shrs_qi_int8_t(<16 x i8> %src1) {
+; CHECK-LABEL: shrs_qi_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.s8 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <16 x i8> %src1, <i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4, i8 4>
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shrs_qi_int16_t(<8 x i16> %src1) {
+; CHECK-LABEL: shrs_qi_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.s16 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <8 x i16> %src1, <i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4, i16 4>
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shrs_qi_int32_t(<4 x i32> %src1) {
+; CHECK-LABEL: shrs_qi_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshr.s32 q0, q0, #4
+; CHECK-NEXT:    bx lr
+entry:
+  %0 = ashr <4 x i32> %src1, <i32 4, i32 4, i32 4, i32 4>
+  ret <4 x i32> %0
+}
+
+
+
+define arm_aapcs_vfpcc <16 x i8> @shl_qr_int8_t(<16 x i8> %src1, i8 %src2) {
+; CHECK-LABEL: shl_qr_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u8 q0, r0
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <16 x i8> undef, i8 %src2, i32 0
+  %s = shufflevector <16 x i8> %i, <16 x i8> undef, <16 x i32> zeroinitializer
+  %0 = shl <16 x i8> %src1, %s
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shl_qr_int16_t(<8 x i16> %src1, i16 %src2) {
+; CHECK-LABEL: shl_qr_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u16 q0, r0
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <8 x i16> undef, i16 %src2, i32 0
+  %s = shufflevector <8 x i16> %i, <8 x i16> undef, <8 x i32> zeroinitializer
+  %0 = shl <8 x i16> %src1, %s
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shl_qr_int32_t(<4 x i32> %src1, i32 %src2) {
+; CHECK-LABEL: shl_qr_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vshl.u32 q0, r0
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <4 x i32> undef, i32 %src2, i32 0
+  %s = shufflevector <4 x i32> %i, <4 x i32> undef, <4 x i32> zeroinitializer
+  %0 = shl <4 x i32> %src1, %s
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shru_qr_int8_t(<16 x i8> %src1, i8 %src2) {
+; CHECK-LABEL: shru_qr_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.8 q1, r0
+; CHECK-NEXT:    vneg.s8 q1, q1
+; CHECK-NEXT:    vshl.u8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <16 x i8> undef, i8 %src2, i32 0
+  %s = shufflevector <16 x i8> %i, <16 x i8> undef, <16 x i32> zeroinitializer
+  %0 = lshr <16 x i8> %src1, %s
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shru_qr_int16_t(<8 x i16> %src1, i16 %src2) {
+; CHECK-LABEL: shru_qr_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.16 q1, r0
+; CHECK-NEXT:    vneg.s16 q1, q1
+; CHECK-NEXT:    vshl.u16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <8 x i16> undef, i16 %src2, i32 0
+  %s = shufflevector <8 x i16> %i, <8 x i16> undef, <8 x i32> zeroinitializer
+  %0 = lshr <8 x i16> %src1, %s
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shru_qr_int32_t(<4 x i32> %src1, i32 %src2) {
+; CHECK-LABEL: shru_qr_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.32 q1, r0
+; CHECK-NEXT:    vneg.s32 q1, q1
+; CHECK-NEXT:    vshl.u32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <4 x i32> undef, i32 %src2, i32 0
+  %s = shufflevector <4 x i32> %i, <4 x i32> undef, <4 x i32> zeroinitializer
+  %0 = lshr <4 x i32> %src1, %s
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shrs_qr_int8_t(<16 x i8> %src1, i8 %src2) {
+; CHECK-LABEL: shrs_qr_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.8 q1, r0
+; CHECK-NEXT:    vneg.s8 q1, q1
+; CHECK-NEXT:    vshl.s8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <16 x i8> undef, i8 %src2, i32 0
+  %s = shufflevector <16 x i8> %i, <16 x i8> undef, <16 x i32> zeroinitializer
+  %0 = ashr <16 x i8> %src1, %s
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shrs_qr_int16_t(<8 x i16> %src1, i16 %src2) {
+; CHECK-LABEL: shrs_qr_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.16 q1, r0
+; CHECK-NEXT:    vneg.s16 q1, q1
+; CHECK-NEXT:    vshl.s16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <8 x i16> undef, i16 %src2, i32 0
+  %s = shufflevector <8 x i16> %i, <8 x i16> undef, <8 x i32> zeroinitializer
+  %0 = ashr <8 x i16> %src1, %s
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shrs_qr_int32_t(<4 x i32> %src1, i32 %src2) {
+; CHECK-LABEL: shrs_qr_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    vdup.32 q1, r0
+; CHECK-NEXT:    vneg.s32 q1, q1
+; CHECK-NEXT:    vshl.s32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+entry:
+  %i = insertelement <4 x i32> undef, i32 %src2, i32 0
+  %s = shufflevector <4 x i32> %i, <4 x i32> undef, <4 x i32> zeroinitializer
+  %0 = ashr <4 x i32> %src1, %s
+  ret <4 x i32> %0
+}
+
+
+define arm_aapcs_vfpcc <16 x i8> @shl_qiv_int8_t(<16 x i8> %src1) {
+; CHECK-LABEL: shl_qiv_int8_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    adr r0, .LCPI27_0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vshl.u8 q0, q0, q1
+; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    .p2align 4
+; CHECK-NEXT:  @ %bb.1:
+; CHECK-NEXT:  .LCPI27_0:
+; CHECK-NEXT:    .byte 1 @ 0x1
+; CHECK-NEXT:    .byte 2 @ 0x2
+; CHECK-NEXT:    .byte 3 @ 0x3
+; CHECK-NEXT:    .byte 4 @ 0x4
+; CHECK-NEXT:    .byte 1 @ 0x1
+; CHECK-NEXT:    .byte 2 @ 0x2
+; CHECK-NEXT:    .byte 3 @ 0x3
+; CHECK-NEXT:    .byte 4 @ 0x4
+; CHECK-NEXT:    .byte 1 @ 0x1
+; CHECK-NEXT:    .byte 2 @ 0x2
+; CHECK-NEXT:    .byte 3 @ 0x3
+; CHECK-NEXT:    .byte 4 @ 0x4
+; CHECK-NEXT:    .byte 1 @ 0x1
+; CHECK-NEXT:    .byte 2 @ 0x2
+; CHECK-NEXT:    .byte 3 @ 0x3
+; CHECK-NEXT:    .byte 4 @ 0x4
+entry:
+  %0 = shl <16 x i8> %src1, <i8 1, i8 2, i8 3, i8 4, i8 1, i8 2, i8 3, i8 4, i8 1, i8 2, i8 3, i8 4, i8 1, i8 2, i8 3, i8 4>
+  ret <16 x i8> %0
+}
+
+define arm_aapcs_vfpcc <8 x i16> @shl_qiv_int16_t(<8 x i16> %src1) {
+; CHECK-LABEL: shl_qiv_int16_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    adr r0, .LCPI28_0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vshl.u16 q0, q0, q1
+; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    .p2align 4
+; CHECK-NEXT:  @ %bb.1:
+; CHECK-NEXT:  .LCPI28_0:
+; CHECK-NEXT:    .short 1 @ 0x1
+; CHECK-NEXT:    .short 2 @ 0x2
+; CHECK-NEXT:    .short 3 @ 0x3
+; CHECK-NEXT:    .short 4 @ 0x4
+; CHECK-NEXT:    .short 1 @ 0x1
+; CHECK-NEXT:    .short 2 @ 0x2
+; CHECK-NEXT:    .short 3 @ 0x3
+; CHECK-NEXT:    .short 4 @ 0x4
+entry:
+  %0 = shl <8 x i16> %src1, <i16 1, i16 2, i16 3, i16 4, i16 1, i16 2, i16 3, i16 4>
+  ret <8 x i16> %0
+}
+
+define arm_aapcs_vfpcc <4 x i32> @shl_qiv_int32_t(<4 x i32> %src1) {
+; CHECK-LABEL: shl_qiv_int32_t:
+; CHECK:       @ %bb.0: @ %entry
+; CHECK-NEXT:    adr r0, .LCPI29_0
+; CHECK-NEXT:    vldrw.u32 q1, [r0]
+; CHECK-NEXT:    vshl.u32 q0, q0, q1
+; CHECK-NEXT:    bx lr
+; CHECK-NEXT:    .p2align 4
+; CHECK-NEXT:  @ %bb.1:
+; CHECK-NEXT:  .LCPI29_0:
+; CHECK-NEXT:    .long 1 @ 0x1
+; CHECK-NEXT:    .long 2 @ 0x2
+; CHECK-NEXT:    .long 3 @ 0x3
+; CHECK-NEXT:    .long 4 @ 0x4
+entry:
+  %0 = shl <4 x i32> %src1, <i32 1, i32 2, i32 3, i32 4>
+  ret <4 x i32> %0
+}