[DAGCombine][ARM] Custom lower smaller-than-legal MULH/AVG/ABD

[dmgreen]

Summary

MVE only has 128bit legal vectors, no 64bit vectors. There are a number of combines for nodes (MULH/AVG/ABD) that are beneficial for these smaller-than-legal vectors, and often created by the vectorizers, but are not currently transformed. There is no way to tell the target independent dag-combiner that it should, allowing the ARM backend to legalize them.

This changes the legality check in those nodes from TLI.isOperationLegalOrCustom(Opc, VT) (which inherently checks isTypeLegal(VT)) to TLI.isOperationLegal(Opc, VT) || TLI.isOperationCustom(Opc, VT), which allows the backend to mark the nodes as Custom for illegal types, legalising the nodes as it requires. The actual legalisation on MVE uses any_extends and vector casts to perform the operation on a legal vector, truncating the result back to the original type.

There may be other ways to do this - let me know if there is a better way. We can do the same for AArch64 for small types but I've not done that in this same patch.

[RKSimon]

Why not handle this generically - accept AVG patterns of any type pre-legalization and expand back during legalization if necessary?

[dmgreen]

In D119075#3299246, @RKSimon wrote:

Why not handle this generically - accept AVG patterns of any type pre-legalization and expand back during legalization if necessary?

Hmm. Won't that be quite inefficient? Creating nodes for architectures that don't even expect to use them? A mulh for example - I wouldn't expect the generic DAG combiner to create i8 mulh on Arm targets - it's not something it will ever need, and I feel it would just get in the way of the optimisations it is trying to make.

[RKSimon]

Fair enough - I was mainly thinking in terms of the AVG opcodes, but I can understand that it might cause other issues.

[efriedma]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8852–8855

I think the predicate you actually need here is something like !TLI.isOperationLegalOrCustom(MulhOpcode, NarrowVT) && (LegalTypes || !TLI.isOperationCustom(MulhOpcode, NarrowVT)). i.e. before type legalization, allow custom lowering. After type legalization, only allow custom lowering if the type is legal.

Constructing an operator with an illegal type after type legalization is likely to crash the compiler.

[dmgreen]

In D119075#3301157, @RKSimon wrote:

Fair enough - I was mainly thinking in terms of the AVG opcodes, but I can understand that it might cause other issues.

I often find myself fighting against DAG combines more than I would like. I am a fan of more control from the target.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

8852–8855

Thanks for the suggestion.

[efriedma]

I'd like to avoid target-specific custom lowering here, i.e. copy-pasting LowerBinopWithBitcast into every target. A few possible alternatives.

1. Make the combine detect whatever pattern shows up after type legalization, and use the legal nodes. Not sure how hard this is; I guess type legalization splits the nodes in ways that make it hard to detect the relevant pattern?
2. Shove something equivalent to LowerBinopWithBitcast directly into the relevant DAGCombines. I guess the difficulty here is mostly that there isn't any target-independent equivalent to VECTOR_REG_CAST?
3. Teach type legalization to legalize these nodes, then make DAGCombine produce illegal nodes if it predicts the legalized result is cheap. (Not sure this approach is best, though; "predicting" is sort of hard, and we don't really want to produce these nodes if they aren't going to lower to a single instruction.)

[dmgreen]

Rebase the current patch. I would like to try and get something like this in if we can.

Unfortunately you seem to have given some pretty good reasons why the suggestions you made will not easily work. The lowering is target specific to uses the NVCast. I was attempting to make use of larger (potentially legal) hadds in combineShiftToAVG, but x86 wants to widen, not promote, and really wants to use the existing custom lowering it has. It would be better for Arm to use extend(hadd) than hadd(extend, extend), but as far as I understand that needs NVCast to be correct for BE.

Any suggestions on how to proceed?

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

SDValue DAGCombiner::visitSHL(SDNode *N) {
   return SDValue();
 }
 
 // Transform a right shift of a multiply into a multiply-high.
 // Examples:
 // (srl (mul (zext i32:$a to i64), (zext i32:$a to i64)), 32) -> (mulhu $a, $b)
 // (sra (mul (sext i32:$a to i64), (sext i32:$a to i64)), 32) -> (mulhs $a, $b)
 static SDValue combineShiftToMULH(SDNode *N, SelectionDAG &DAG,
-                                  const TargetLowering &TLI) {
+                                  const TargetLowering &TLI, bool LegalTypes) {
   assert((N->getOpcode() == ISD::SRL || N->getOpcode() == ISD::SRA) &&
          "SRL or SRA node is required here!");
 
   // Check the shift amount. Proceed with the transformation if the shift
   // amount is constant.
   ConstantSDNode *ShiftAmtSrc = isConstOrConstSplat(N->getOperand(1));
   if (!ShiftAmtSrc)
     return SDValue();
static SDValue combineShiftToMULH(SDNode *N, SelectionDAG &DAG,
   unsigned ShiftAmt = ShiftAmtSrc->getZExtValue();
   if (ShiftAmt != NarrowVTSize)
     return SDValue();
 
   // If the operation feeding into the MUL is a sign extend (sext),
   // we use mulhs. Othewise, zero extends (zext) use mulhu.
   unsigned MulhOpcode = IsSignExt ? ISD::MULHS : ISD::MULHU;
 
   // Combine to mulh if mulh is legal/custom for the narrow type on the target.
-  if (!TLI.isOperationLegalOrCustom(MulhOpcode, NarrowVT))
+  if (!TLI.isOperationLegalOrCustom(MulhOpcode, NarrowVT) &&
+      (LegalTypes || !TLI.isOperationCustom(MulhOpcode, NarrowVT)))
     return SDValue();
 
   SDValue Result =
       DAG.getNode(MulhOpcode, DL, NarrowVT, LeftOp.getOperand(0), MulhRightOp);
   return (N->getOpcode() == ISD::SRA ? DAG.getSExtOrTrunc(Result, DL, WideVT)
                                      : DAG.getZExtOrTrunc(Result, DL, WideVT));
 }
 
 SDValue DAGCombiner::visitSRA(SDNode *N) {
if (DAG.SignBitIsZero(N0))
     return DAG.getNode(ISD::SRL, SDLoc(N), VT, N0, N1);
 
   if (N1C && !N1C->isOpaque())
     if (SDValue NewSRA = visitShiftByConstant(N))
       return NewSRA;
 
   // Try to transform this shift into a multiply-high if
   // it matches the appropriate pattern detected in combineShiftToMULH.
-  if (SDValue MULH = combineShiftToMULH(N, DAG, TLI))
+  if (SDValue MULH = combineShiftToMULH(N, DAG, TLI, LegalTypes))
     return MULH;
 
   // Attempt to convert a sra of a load into a narrower sign-extending load.
   if (SDValue NarrowLoad = reduceLoadWidth(N))
     return NarrowLoad;
 
   return SDValue();
 }
else if (Use->getOpcode() == ISD::TRUNCATE && Use->hasOneUse()) {
       Use = *Use->use_begin();
       if (Use->getOpcode() == ISD::BRCOND)
         AddToWorklist(Use);
     }
   }
 
   // Try to transform this shift into a multiply-high if
   // it matches the appropriate pattern detected in combineShiftToMULH.
-  if (SDValue MULH = combineShiftToMULH(N, DAG, TLI))
+  if (SDValue MULH = combineShiftToMULH(N, DAG, TLI, LegalTypes))
     return MULH;
 
   return SDValue();
 }
 
 SDValue DAGCombiner::visitFunnelShift(SDNode *N) {
   EVT VT = N->getValueType(0);
   SDValue N0 = N->getOperand(0);
SDValue DAGCombiner::visitSHLSAT(SDNode *N) {
 
   return SDValue();
 }
 
 // Given a ABS node, detect the following pattern:
 // (ABS (SUB (EXTEND a), (EXTEND b))).
 // Generates UABD/SABD instruction.
 static SDValue combineABSToABD(SDNode *N, SelectionDAG &DAG,
-                               const TargetLowering &TLI) {
+                               const TargetLowering &TLI, bool LegalTypes) {
   SDValue AbsOp1 = N->getOperand(0);
   SDValue Op0, Op1;
 
   if (AbsOp1.getOpcode() != ISD::SUB)
     return SDValue();
 
   Op0 = AbsOp1.getOperand(0);
   Op1 = AbsOp1.getOperand(1);
 
   unsigned Opc0 = Op0.getOpcode();
   // Check if the operands of the sub are (zero|sign)-extended.
   if (Opc0 != Op1.getOpcode() ||
       (Opc0 != ISD::ZERO_EXTEND && Opc0 != ISD::SIGN_EXTEND))
     return SDValue();
 
   EVT VT1 = Op0.getOperand(0).getValueType();
   EVT VT2 = Op1.getOperand(0).getValueType();
   // Check if the operands are of same type and valid size.
   unsigned ABDOpcode = (Opc0 == ISD::SIGN_EXTEND) ? ISD::ABDS : ISD::ABDU;
-  if (VT1 != VT2 || !TLI.isOperationLegalOrCustom(ABDOpcode, VT1))
+  if (VT1 != VT2 || (!TLI.isOperationLegalOrCustom(ABDOpcode, VT1) &&
+                     (LegalTypes || !TLI.isOperationCustom(ABDOpcode, VT1))))
     return SDValue();
 
   Op0 = Op0.getOperand(0);
   Op1 = Op1.getOperand(0);
   SDValue ABD =
       DAG.getNode(ABDOpcode, SDLoc(N), Op0->getValueType(0), Op0, Op1);
   return DAG.getNode(ISD::ZERO_EXTEND, SDLoc(N), N->getValueType(0), ABD);
 }
 
 SDValue DAGCombiner::visitABS(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);
 
   // fold (abs c1) -> c2
   if (DAG.isConstantIntBuildVectorOrConstantInt(N0))
     return DAG.getNode(ISD::ABS, SDLoc(N), VT, N0);
   // fold (abs (abs x)) -> (abs x)
   if (N0.getOpcode() == ISD::ABS)
     return N0;
   // fold (abs x) -> x iff not-negative
   if (DAG.SignBitIsZero(N0))
     return N0;
 
-  if (SDValue ABD = combineABSToABD(N, DAG, TLI))
+  if (SDValue ABD = combineABSToABD(N, DAG, TLI, LegalTypes))
     return ABD;
 
   return SDValue();
 }
 
 SDValue DAGCombiner::visitBSWAP(SDNode *N) {
   SDValue N0 = N->getOperand(0);
   EVT VT = N->getValueType(0);

llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp

SDValue TargetLowering::SimplifyMultipleUseDemandedVectorElts(
     unsigned Depth) const {
   APInt DemandedBits = APInt::getAllOnes(Op.getScalarValueSizeInBits());
   return SimplifyMultipleUseDemandedBits(Op, DemandedBits, DemandedElts, DAG,
                                          Depth);
 }
 
 // Attempt to form ext(avgfloor(A, B)) from shr(add(ext(A), ext(B)), 1).
 //      or to form ext(avgceil(A, B)) from shr(add(ext(A), ext(B), 1), 1).
-static SDValue combineShiftToAVG(SDValue Op, SelectionDAG &DAG,
+static SDValue combineShiftToAVG(SDValue Op,
+                                 TargetLowering::TargetLoweringOpt &TLO,
                                  const TargetLowering &TLI,
                                  const APInt &DemandedBits,
-                                 const APInt &DemandedElts,
-                                 unsigned Depth) {
+                                 const APInt &DemandedElts, unsigned Depth) {
+  SelectionDAG &DAG = TLO.DAG;
   assert((Op.getOpcode() == ISD::SRL || Op.getOpcode() == ISD::SRA) &&
          "SRL or SRA node is required here!");
   // Is the right shift using an immediate value of 1?
   ConstantSDNode *N1C = isConstOrConstSplat(Op.getOperand(1), DemandedElts);
   if (!N1C || !N1C->isOne())
     return SDValue();
 
   // We are looking for an avgfloor
static SDValue combineShiftToAVG(SDValue Op,
   // operation, given the original type size and the number of known sign/zero
   // bits.
   EVT VT = Op.getValueType();
   unsigned MinWidth =
       std::max<unsigned>(VT.getScalarSizeInBits() - KnownBits, 8);
   EVT NVT = EVT::getIntegerVT(*DAG.getContext(), PowerOf2Ceil(MinWidth));
   if (VT.isVector())
     NVT = EVT::getVectorVT(*DAG.getContext(), NVT, VT.getVectorElementCount());
-  if (!TLI.isOperationLegalOrCustom(AVGOpc, NVT))
+  if (!TLI.isOperationLegalOrCustom(AVGOpc, NVT) &&
+      (TLO.LegalTypes() || !TLI.isOperationCustom(AVGOpc, NVT)))
     return SDValue();
 
   SDLoc DL(Op);
   SDValue ResultAVG =
       DAG.getNode(AVGOpc, DL, NVT, DAG.getNode(ISD::TRUNCATE, DL, NVT, ExtOpA),
                   DAG.getNode(ISD::TRUNCATE, DL, NVT, ExtOpB));
   return DAG.getNode(IsSigned ? ISD::SIGN_EXTEND : ISD::ZERO_EXTEND, DL, VT,
                      ResultAVG);
case ISD::SHL: {
     break;
   }
   case ISD::SRL: {
     SDValue Op0 = Op.getOperand(0);
     SDValue Op1 = Op.getOperand(1);
     EVT ShiftVT = Op1.getValueType();
 
     // Try to match AVG patterns.
-    if (SDValue AVG = combineShiftToAVG(Op, TLO.DAG, *this, DemandedBits,
+    if (SDValue AVG = combineShiftToAVG(Op, TLO, *this, DemandedBits,
                                         DemandedElts, Depth + 1))
       return TLO.CombineTo(Op, AVG);
 
     if (const APInt *SA =
             TLO.DAG.getValidShiftAmountConstant(Op, DemandedElts)) {
       unsigned ShAmt = SA->getZExtValue();
       if (ShAmt == 0)
         return TLO.CombineTo(Op, Op0);
case ISD::SRA: {
     // If this is an arithmetic shift right and only the low-bit is set, we can
     // always convert this into a logical shr, even if the shift amount is
     // variable.  The low bit of the shift cannot be an input sign bit unless
     // the shift amount is >= the size of the datatype, which is undefined.
     if (DemandedBits.isOne())
       return TLO.CombineTo(Op, TLO.DAG.getNode(ISD::SRL, dl, VT, Op0, Op1));
 
     // Try to match AVG patterns.
-    if (SDValue AVG = combineShiftToAVG(Op, TLO.DAG, *this, DemandedBits,
+    if (SDValue AVG = combineShiftToAVG(Op, TLO, *this, DemandedBits,
                                         DemandedElts, Depth + 1))
       return TLO.CombineTo(Op, AVG);
 
     if (const APInt *SA =
             TLO.DAG.getValidShiftAmountConstant(Op, DemandedElts)) {
       unsigned ShAmt = SA->getZExtValue();
       if (ShAmt == 0)
         return TLO.CombineTo(Op, Op0);

llvm/lib/Target/ARM/ARMISelLowering.cpp

for (unsigned im = (unsigned)ISD::PRE_INC;
     for (auto VT : {MVT::v8i8, MVT::v4i8, MVT::v4i16}) {
       setIndexedLoadAction(im, VT, Legal);
       setIndexedStoreAction(im, VT, Legal);
       setIndexedMaskedLoadAction(im, VT, Legal);
       setIndexedMaskedStoreAction(im, VT, Legal);
     }
   }
 
+  // Custom extend some nodes so that the generic combines fire on smaller than
+  // legal types.
+  for (auto VT : {MVT::v8i8, MVT::v4i8, MVT::v4i16}) {
+    setOperationAction(ISD::MULHS, VT, Custom);
+    setOperationAction(ISD::MULHU, VT, Custom);
+    setOperationAction(ISD::AVGFLOORS, VT, Custom);
+    setOperationAction(ISD::AVGFLOORU, VT, Custom);
+    setOperationAction(ISD::AVGCEILS, VT, Custom);
+    setOperationAction(ISD::AVGCEILU, VT, Custom);
+    setOperationAction(ISD::ABDS, VT, Custom);
+    setOperationAction(ISD::ABDU, VT, Custom);
+  }
+
   // Predicate types
   const MVT pTypes[] = {MVT::v16i1, MVT::v8i1, MVT::v4i1, MVT::v2i1};
   for (auto VT : pTypes) {
     addRegisterClass(VT, &ARM::VCCRRegClass);
     setOperationAction(ISD::BUILD_VECTOR, VT, Custom);
     setOperationAction(ISD::VECTOR_SHUFFLE, VT, Custom);
     setOperationAction(ISD::EXTRACT_SUBVECTOR, VT, Custom);
     setOperationAction(ISD::CONCAT_VECTORS, VT, Custom);
static SDValue LowerADDSUBSAT(SDValue Op, SelectionDAG &DAG,
   SDLoc dl(Op);
   SDValue Add =
       DAG.getNode(NewOpcode, dl, MVT::i32,
                   DAG.getSExtOrTrunc(Op->getOperand(0), dl, MVT::i32),
                   DAG.getSExtOrTrunc(Op->getOperand(1), dl, MVT::i32));
   return DAG.getNode(ISD::TRUNCATE, dl, VT, Add);
 }
 
+// Custom lower MULH, ABD, HADD and RHADD nodes that are smaller than legal,
+// using a bitcast to a larger legal type upon which we perform the bitcast.
+// This allow DAG combine to recognize the nodes where it usually would not.
+static SDValue LowerBinopWithBitcast(SDNode *N, SelectionDAG &DAG) {
+  EVT VT = N->getValueType(0);
+
+  assert((VT == MVT::v4i8 || VT == MVT::v8i8 || VT == MVT::v4i16) &&
+         "Expected smaller than legal type!");
+
+  EVT ExtVT = VT.getVectorNumElements() == 4 ? MVT::v4i32 : MVT::v8i16;
+  EVT BinOpVT = VT.getScalarType() == MVT::i8 ? MVT::v16i8 : MVT::v8i16;
+
+  SDLoc DL(N);
+  SDValue Ext0 = DAG.getNode(ISD::ANY_EXTEND, DL, ExtVT, N->getOperand(0));
+  SDValue Ext1 = DAG.getNode(ISD::ANY_EXTEND, DL, ExtVT, N->getOperand(1));
+  SDValue BC0 = DAG.getNode(ARMISD::VECTOR_REG_CAST, DL, BinOpVT, Ext0);
+  SDValue BC1 = DAG.getNode(ARMISD::VECTOR_REG_CAST, DL, BinOpVT, Ext1);
+  SDValue BinOp = DAG.getNode(N->getOpcode(), DL, BinOpVT, BC0, BC1);
+  SDValue BC2 = DAG.getNode(ARMISD::VECTOR_REG_CAST, DL, ExtVT, BinOp);
+  return DAG.getNode(ISD::TRUNCATE, DL, VT, BC2);
+}
+
 SDValue ARMTargetLowering::LowerSELECT(SDValue Op, SelectionDAG &DAG) const {
   SDValue Cond = Op.getOperand(0);
   SDValue SelectTrue = Op.getOperand(1);
   SDValue SelectFalse = Op.getOperand(2);
   SDLoc dl(Op);
   unsigned Opc = Cond.getOpcode();
 
   if (Cond.getResNo() == 1 &&
case ISD::UDIVREM:
     Results.push_back(Res.getValue(1));
     return;
   case ISD::SADDSAT:
   case ISD::SSUBSAT:
   case ISD::UADDSAT:
   case ISD::USUBSAT:
     Res = LowerADDSUBSAT(SDValue(N, 0), DAG, Subtarget);
     break;
+  case ISD::MULHS:
+  case ISD::MULHU:
+  case ISD::ABDS:
+  case ISD::ABDU:
+  case ISD::AVGFLOORS:
+  case ISD::AVGFLOORU:
+  case ISD::AVGCEILS:
+  case ISD::AVGCEILU:
+    Res = LowerBinopWithBitcast(N, DAG);
+    break;
   case ISD::READCYCLECOUNTER:
     ReplaceREADCYCLECOUNTER(N, Results, DAG, Subtarget);
     return;
   case ISD::UDIV:
   case ISD::SDIV:
     assert(Subtarget->isTargetWindows() && "can only expand DIV on Windows");
     return ExpandDIV_Windows(SDValue(N, 0), DAG, N->getOpcode() == ISD::SDIV,
                              Results);

llvm/test/CodeGen/Thumb2/mve-vabdus.ll

; CHECK-NEXT:    bx lr
   %s = select <16 x i1> %c, <16 x i16> %add1, <16 x i16> %add2
   %result = trunc <16 x i16> %s to <16 x i8>
   ret <16 x i8> %result
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vabd_v8s8(<8 x i8> %src1, <8 x i8> %src2) {
 ; CHECK-LABEL: vabd_v8s8:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmovlb.s8 q1, q1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vsub.i16 q0, q0, q1
-; CHECK-NEXT:    vabs.s16 q0, q0
+; CHECK-NEXT:    vabd.s8 q0, q0, q1
+; CHECK-NEXT:    vmovlb.u8 q0, q0
 ; CHECK-NEXT:    bx lr
   %sextsrc1 = sext <8 x i8> %src1 to <8 x i16>
   %sextsrc2 = sext <8 x i8> %src2 to <8 x i16>
   %add1 = sub <8 x i16> %sextsrc1, %sextsrc2
   %add2 = sub <8 x i16> zeroinitializer, %add1
   %c = icmp sge <8 x i16> %add1, zeroinitializer
   %s = select <8 x i1> %c, <8 x i16> %add1, <8 x i16> %add2
   %result = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %result
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vabd_v4s8(<4 x i8> %src1, <4 x i8> %src2) {
 ; CHECK-LABEL: vabd_v4s8:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmovlb.s8 q1, q1
-; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q1, q1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vsub.i32 q0, q0, q1
-; CHECK-NEXT:    vabs.s32 q0, q0
+; CHECK-NEXT:    vabd.s8 q0, q0, q1
+; CHECK-NEXT:    vmov.i32 q1, #0xff
+; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
   %sextsrc1 = sext <4 x i8> %src1 to <4 x i16>
   %sextsrc2 = sext <4 x i8> %src2 to <4 x i16>
   %add1 = sub <4 x i16> %sextsrc1, %sextsrc2
   %add2 = sub <4 x i16> zeroinitializer, %add1
   %c = icmp sge <4 x i16> %add1, zeroinitializer
   %s = select <4 x i1> %c, <4 x i16> %add1, <4 x i16> %add2
   %result = trunc <4 x i16> %s to <4 x i8>
; CHECK-NEXT:    bx lr
   %s = select <8 x i1> %c, <8 x i32> %add1, <8 x i32> %add2
   %result = trunc <8 x i32> %s to <8 x i16>
   ret <8 x i16> %result
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vabd_v4s16(<4 x i16> %src1, <4 x i16> %src2) {
 ; CHECK-LABEL: vabd_v4s16:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmovlb.s16 q1, q1
-; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vsub.i32 q0, q0, q1
-; CHECK-NEXT:    vabs.s32 q0, q0
+; CHECK-NEXT:    vabd.s16 q0, q0, q1
+; CHECK-NEXT:    vmovlb.u16 q0, q0
 ; CHECK-NEXT:    bx lr
   %sextsrc1 = sext <4 x i16> %src1 to <4 x i32>
   %sextsrc2 = sext <4 x i16> %src2 to <4 x i32>
   %add1 = sub <4 x i32> %sextsrc1, %sextsrc2
   %add2 = sub <4 x i32> zeroinitializer, %add1
   %c = icmp sge <4 x i32> %add1, zeroinitializer
   %s = select <4 x i1> %c, <4 x i32> %add1, <4 x i32> %add2
   %result = trunc <4 x i32> %s to <4 x i16>
; CHECK-NEXT:    bx lr
   %s = select <16 x i1> %c, <16 x i16> %add1, <16 x i16> %add2
   %result = trunc <16 x i16> %s to <16 x i8>
   ret <16 x i8> %result
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vabd_v8u8(<8 x i8> %src1, <8 x i8> %src2) {
 ; CHECK-LABEL: vabd_v8u8:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmovlb.u8 q1, q1
+; CHECK-NEXT:    vabd.u8 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    vsub.i16 q0, q0, q1
-; CHECK-NEXT:    vabs.s16 q0, q0
 ; CHECK-NEXT:    bx lr
   %zextsrc1 = zext <8 x i8> %src1 to <8 x i16>
   %zextsrc2 = zext <8 x i8> %src2 to <8 x i16>
   %add1 = sub <8 x i16> %zextsrc1, %zextsrc2
   %add2 = sub <8 x i16> zeroinitializer, %add1
   %c = icmp sge <8 x i16> %add1, zeroinitializer
   %s = select <8 x i1> %c, <8 x i16> %add1, <8 x i16> %add2
   %result = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %result
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vabd_v4u8(<4 x i8> %src1, <4 x i8> %src2) {
 ; CHECK-LABEL: vabd_v4u8:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vand q1, q1, q2
-; CHECK-NEXT:    vand q0, q0, q2
-; CHECK-NEXT:    vsub.i32 q0, q0, q1
-; CHECK-NEXT:    vabs.s32 q0, q0
+; CHECK-NEXT:    vabd.u8 q0, q0, q1
+; CHECK-NEXT:    vmov.i32 q1, #0xff
+; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
   %zextsrc1 = zext <4 x i8> %src1 to <4 x i16>
   %zextsrc2 = zext <4 x i8> %src2 to <4 x i16>
   %add1 = sub <4 x i16> %zextsrc1, %zextsrc2
   %add2 = sub <4 x i16> zeroinitializer, %add1
   %c = icmp sge <4 x i16> %add1, zeroinitializer
   %s = select <4 x i1> %c, <4 x i16> %add1, <4 x i16> %add2
   %result = trunc <4 x i16> %s to <4 x i8>
; CHECK-NEXT:    bx lr
   %s = select <8 x i1> %c, <8 x i32> %add1, <8 x i32> %add2
   %result = trunc <8 x i32> %s to <8 x i16>
   ret <8 x i16> %result
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vabd_v4u16(<4 x i16> %src1, <4 x i16> %src2) {
 ; CHECK-LABEL: vabd_v4u16:
 ; CHECK:       @ %bb.0:
-; CHECK-NEXT:    vmovlb.u16 q1, q1
+; CHECK-NEXT:    vabd.u16 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u16 q0, q0
-; CHECK-NEXT:    vsub.i32 q0, q0, q1
-; CHECK-NEXT:    vabs.s32 q0, q0
 ; CHECK-NEXT:    bx lr
   %zextsrc1 = zext <4 x i16> %src1 to <4 x i32>
   %zextsrc2 = zext <4 x i16> %src2 to <4 x i32>
   %add1 = sub <4 x i32> %zextsrc1, %zextsrc2
   %add2 = sub <4 x i32> zeroinitializer, %add1
   %c = icmp sge <4 x i32> %add1, zeroinitializer
   %s = select <4 x i1> %c, <4 x i32> %add1, <4 x i32> %add2
   %result = trunc <4 x i32> %s to <4 x i16>

llvm/test/CodeGen/Thumb2/mve-vhadd.ll

entry:
   %s = lshr <4 x i32> %m, <i32 1, i32 1, i32 1, i32 1>
   %s2 = trunc <4 x i32> %s to <4 x i16>
   ret <4 x i16> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vhaddu_v4i16(<4 x i16> %s0, <4 x i16> %s1) {
 ; CHECK-LABEL: vhaddu_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmovlb.u16 q1, q1
+; CHECK-NEXT:    vhadd.u16 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u16 q0, q0
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vshr.u32 q0, q0, #1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i16> %s0 to <4 x i32>
   %s1s = zext <4 x i16> %s1 to <4 x i32>
   %m = add <4 x i32> %s0s, %s1s
   %s = lshr <4 x i32> %m, <i32 1, i32 1, i32 1, i32 1>
   %s2 = trunc <4 x i32> %s to <4 x i16>
   ret <4 x i16> %s2
entry:
   %s = lshr <4 x i16> %m, <i16 1, i16 1, i16 1, i16 1>
   %s2 = trunc <4 x i16> %s to <4 x i8>
   ret <4 x i8> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vhaddu_v4i8(<4 x i8> %s0, <4 x i8> %s1) {
 ; CHECK-LABEL: vhaddu_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vand q1, q1, q2
-; CHECK-NEXT:    vand q0, q0, q2
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vshr.u32 q0, q0, #1
+; CHECK-NEXT:    vhadd.u8 q0, q0, q1
+; CHECK-NEXT:    vmov.i32 q1, #0xff
+; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i8> %s0 to <4 x i16>
   %s1s = zext <4 x i8> %s1 to <4 x i16>
   %m = add <4 x i16> %s0s, %s1s
   %s = lshr <4 x i16> %m, <i16 1, i16 1, i16 1, i16 1>
   %s2 = trunc <4 x i16> %s to <4 x i8>
   ret <4 x i8> %s2
entry:
   %s = lshr <8 x i16> %m, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
   %s2 = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %s2
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vhaddu_v8i8(<8 x i8> %s0, <8 x i8> %s1) {
 ; CHECK-LABEL: vhaddu_v8i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmovlb.u8 q1, q1
+; CHECK-NEXT:    vhadd.u8 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    vadd.i16 q0, q0, q1
-; CHECK-NEXT:    vshr.u16 q0, q0, #1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <8 x i8> %s0 to <8 x i16>
   %s1s = zext <8 x i8> %s1 to <8 x i16>
   %m = add <8 x i16> %s0s, %s1s
   %s = lshr <8 x i16> %m, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
   %s2 = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %s2
entry:
   %s = lshr <4 x i32> %add2, <i32 1, i32 1, i32 1, i32 1>
   %result = trunc <4 x i32> %s to <4 x i16>
   ret <4 x i16> %result
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vrhaddu_v4i16(<4 x i16> %s0, <4 x i16> %s1) {
 ; CHECK-LABEL: vrhaddu_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmovlb.u16 q1, q1
+; CHECK-NEXT:    vrhadd.u16 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u16 q0, q0
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    movs r0, #1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vshr.u32 q0, q0, #1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i16> %s0 to <4 x i32>
   %s1s = zext <4 x i16> %s1 to <4 x i32>
   %add = add <4 x i32> %s0s, %s1s
   %add2 = add <4 x i32> %add, <i32 1, i32 1, i32 1, i32 1>
   %s = lshr <4 x i32> %add2, <i32 1, i32 1, i32 1, i32 1>
   %result = trunc <4 x i32> %s to <4 x i16>
entry:
   %s = lshr <4 x i16> %add2, <i16 1, i16 1, i16 1, i16 1>
   %result = trunc <4 x i16> %s to <4 x i8>
   ret <4 x i8> %result
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vrhaddu_v4i8(<4 x i8> %s0, <4 x i8> %s1) {
 ; CHECK-LABEL: vrhaddu_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    movs r0, #1
-; CHECK-NEXT:    vand q1, q1, q2
-; CHECK-NEXT:    vand q0, q0, q2
-; CHECK-NEXT:    vadd.i32 q0, q0, q1
-; CHECK-NEXT:    vadd.i32 q0, q0, r0
-; CHECK-NEXT:    vshr.u32 q0, q0, #1
+; CHECK-NEXT:    vrhadd.u8 q0, q0, q1
+; CHECK-NEXT:    vmov.i32 q1, #0xff
+; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i8> %s0 to <4 x i16>
   %s1s = zext <4 x i8> %s1 to <4 x i16>
   %add = add <4 x i16> %s0s, %s1s
   %add2 = add <4 x i16> %add, <i16 1, i16 1, i16 1, i16 1>
   %s = lshr <4 x i16> %add2, <i16 1, i16 1, i16 1, i16 1>
   %result = trunc <4 x i16> %s to <4 x i8>
entry:
   %s = lshr <8 x i16> %add2, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
   %result = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %result
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vrhaddu_v8i8(<8 x i8> %s0, <8 x i8> %s1) {
 ; CHECK-LABEL: vrhaddu_v8i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmovlb.u8 q1, q1
+; CHECK-NEXT:    vrhadd.u8 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.u8 q0, q0
-; CHECK-NEXT:    vadd.i16 q0, q0, q1
-; CHECK-NEXT:    movs r0, #1
-; CHECK-NEXT:    vadd.i16 q0, q0, r0
-; CHECK-NEXT:    vshr.u16 q0, q0, #1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <8 x i8> %s0 to <8 x i16>
   %s1s = zext <8 x i8> %s1 to <8 x i16>
   %add = add <8 x i16> %s0s, %s1s
   %add2 = add <8 x i16> %add, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
   %s = lshr <8 x i16> %add2, <i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1, i16 1>
   %result = trunc <8 x i16> %s to <8 x i8>

llvm/test/CodeGen/Thumb2/mve-vmulh.ll

entry:
   %s = lshr <4 x i64> %m, <i64 32, i64 32, i64 32, i64 32>
   %s2 = trunc <4 x i64> %s to <4 x i32>
   ret <4 x i32> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vmulhs_v4i16(<4 x i16> %s0, <4 x i16> %s1) {
 ; CHECK-LABEL: vmulhs_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmullb.s16 q0, q0, q1
-; CHECK-NEXT:    vshr.s32 q0, q0, #16
+; CHECK-NEXT:    vmulh.s16 q0, q0, q1
+; CHECK-NEXT:    vmovlb.s16 q0, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = sext <4 x i16> %s0 to <4 x i32>
   %s1s = sext <4 x i16> %s1 to <4 x i32>
   %m = mul <4 x i32> %s0s, %s1s
   %s = ashr <4 x i32> %m, <i32 16, i32 16, i32 16, i32 16>
   %s2 = trunc <4 x i32> %s to <4 x i16>
   ret <4 x i16> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i16> @vmulhu_v4i16(<4 x i16> %s0, <4 x i16> %s1) {
 ; CHECK-LABEL: vmulhu_v4i16:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmullb.u16 q0, q0, q1
-; CHECK-NEXT:    vshr.u32 q0, q0, #16
+; CHECK-NEXT:    vmulh.u16 q0, q0, q1
+; CHECK-NEXT:    vmovlb.u16 q0, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i16> %s0 to <4 x i32>
   %s1s = zext <4 x i16> %s1 to <4 x i32>
   %m = mul <4 x i32> %s0s, %s1s
   %s = lshr <4 x i32> %m, <i32 16, i32 16, i32 16, i32 16>
   %s2 = trunc <4 x i32> %s to <4 x i16>
   ret <4 x i16> %s2
entry:
   %s = lshr <8 x i32> %m, <i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
   %s2 = trunc <8 x i32> %s to <8 x i16>
   ret <8 x i16> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vmulhs_v4i8(<4 x i8> %s0, <4 x i8> %s1) {
 ; CHECK-LABEL: vmulhs_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmovlb.s8 q1, q1
+; CHECK-NEXT:    vmulh.s8 q0, q0, q1
 ; CHECK-NEXT:    vmovlb.s8 q0, q0
-; CHECK-NEXT:    vmovlb.s16 q1, q1
 ; CHECK-NEXT:    vmovlb.s16 q0, q0
-; CHECK-NEXT:    vmul.i32 q0, q0, q1
-; CHECK-NEXT:    vshr.s32 q0, q0, #8
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = sext <4 x i8> %s0 to <4 x i16>
   %s1s = sext <4 x i8> %s1 to <4 x i16>
   %m = mul <4 x i16> %s0s, %s1s
   %s = ashr <4 x i16> %m, <i16 8, i16 8, i16 8, i16 8>
   %s2 = trunc <4 x i16> %s to <4 x i8>
   ret <4 x i8> %s2
 }
 
 define arm_aapcs_vfpcc <4 x i8> @vmulhu_v4i8(<4 x i8> %s0, <4 x i8> %s1) {
 ; CHECK-LABEL: vmulhu_v4i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmov.i32 q2, #0xff
-; CHECK-NEXT:    vand q1, q1, q2
-; CHECK-NEXT:    vand q0, q0, q2
-; CHECK-NEXT:    vmul.i32 q0, q0, q1
-; CHECK-NEXT:    vshr.u32 q0, q0, #8
+; CHECK-NEXT:    vmulh.u8 q0, q0, q1
+; CHECK-NEXT:    vmov.i32 q1, #0xff
+; CHECK-NEXT:    vand q0, q0, q1
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <4 x i8> %s0 to <4 x i16>
   %s1s = zext <4 x i8> %s1 to <4 x i16>
   %m = mul <4 x i16> %s0s, %s1s
   %s = lshr <4 x i16> %m, <i16 8, i16 8, i16 8, i16 8>
   %s2 = trunc <4 x i16> %s to <4 x i8>
   ret <4 x i8> %s2
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vmulhs_v8i8(<8 x i8> %s0, <8 x i8> %s1) {
 ; CHECK-LABEL: vmulhs_v8i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmullb.s8 q0, q0, q1
-; CHECK-NEXT:    vshr.s16 q0, q0, #8
+; CHECK-NEXT:    vmulh.s8 q0, q0, q1
+; CHECK-NEXT:    vmovlb.s8 q0, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = sext <8 x i8> %s0 to <8 x i16>
   %s1s = sext <8 x i8> %s1 to <8 x i16>
   %m = mul <8 x i16> %s0s, %s1s
   %s = ashr <8 x i16> %m, <i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8>
   %s2 = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %s2
 }
 
 define arm_aapcs_vfpcc <8 x i8> @vmulhu_v8i8(<8 x i8> %s0, <8 x i8> %s1) {
 ; CHECK-LABEL: vmulhu_v8i8:
 ; CHECK:       @ %bb.0: @ %entry
-; CHECK-NEXT:    vmullb.u8 q0, q0, q1
-; CHECK-NEXT:    vshr.u16 q0, q0, #8
+; CHECK-NEXT:    vmulh.u8 q0, q0, q1
+; CHECK-NEXT:    vmovlb.u8 q0, q0
 ; CHECK-NEXT:    bx lr
 entry:
   %s0s = zext <8 x i8> %s0 to <8 x i16>
   %s1s = zext <8 x i8> %s1 to <8 x i16>
   %m = mul <8 x i16> %s0s, %s1s
   %s = lshr <8 x i16> %m, <i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8, i16 8>
   %s2 = trunc <8 x i16> %s to <8 x i8>
   ret <8 x i8> %s2