[ARM] Add lowering of uadd_sat to uq{add|sub}8 and uq{add|sub}16

[therealprof]

Summary

This follow the lead of https://reviews.llvm.org/D68974 to add lowering
of unsigned saturated addition/subtraction.

Signed-off-by: Daniel Egger <[e-mail]>

[dmgreen]

Hello. Looks like a good patch to me.

llvm/lib/Target/ARM/ARMISelLowering.cpp

4988

This is very similar to the code in LowerSADDSUBSAT. Can we combine the functions into one?

[therealprof]

In D105413#2858000, @dmgreen wrote:

Hello. Looks like a good patch to me.

This is very similar to the code in LowerSADDSUBSAT. Can we combine the functions into one?

The opcode selection would be vastly more complex but I can certainly take on a rework of that function if desirable.

[dmgreen]

The opcode selection would be vastly more complex but I can certainly take on a rework of that function if desirable.

Yeah, it sounds like it should share more code than it complicates finding the new opcode.

[therealprof]

In D105413#2859219, @dmgreen wrote:

The opcode selection would be vastly more complex but I can certainly take on a rework of that function if desirable.

Yeah, it sounds like it should share more code than it complicates finding the new opcode.

Sorry I don't follow. Are you asking me to unify the functions or saying it's fine as is? 😅

[dmgreen]

In D105413#2859221, @therealprof wrote:

In D105413#2859219, @dmgreen wrote:

The opcode selection would be vastly more complex but I can certainly take on a rework of that function if desirable.

Yeah, it sounds like it should share more code than it complicates finding the new opcode.

Sorry I don't follow. Are you asking me to unify the functions or saying it's fine as is? 😅

Yep, unifying them sounds better to me. It looks like its should be possible to keep the complexity of the opcode check down, and then it's simpler overall with a single function.

[therealprof]

Refactor into a single lowering function

[therealprof]

In D105413#2859261, @dmgreen wrote:

Yep, unifying them sounds better to me. It looks like its should be possible to keep the complexity of the opcode check down, and then it's simpler overall with a single function.

Done.

[dmgreen]

Thanks. LGTM

[therealprof]

Anything I can do to expedite the application? I do have more changes planned which are depending on/conflicting with this one...

[dmgreen]

Oh, sorry. Yes this is good to go. Either you can request commit access, via https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access, or I am happy to commit it for you. If it is your first patch, then it may require someone with existing access to commit it, I'm not very sure how the policy on getting access usually goes.

If you are happy for me to commit it, then I just need a way to attribute the patch. Something like "Daniel Egger <[e-mail]>".

[therealprof]

In D105413#2869046, @dmgreen wrote:

Oh, sorry. Yes this is good to go. Either you can request commit access, via https://llvm.org/docs/DeveloperPolicy.html#obtaining-commit-access, or I am happy to commit it for you. If it is your first patch, then it may require someone with existing access to commit it, I'm not very sure how the policy on getting access usually goes.

If you are happy for me to commit it, then I just need a way to attribute the patch. Something like "Daniel Egger <[e-mail]>".

Please go ahead and commit it.

There's a Signed-off-by: Daniel Egger <[e-mail]> within the commit message of the patch but feel free to modify as needed.

[dmgreen]

Ah that'll work great, thanks. I will probably commit it tomorrow morning when the buildbots are less red.

Thanks for the patch. I look forward to seeing what else you have.

[therealprof]

In D105413#2869061, @dmgreen wrote:

Thanks for the patch. I look forward to seeing what else you have.

At the moment I'm struggling to add v4i8 and v2i16 support. 😅 Is there any chat channel or something where someone might help me laying the foundation for future work?

[dmgreen]

We usually don't treat the dsp instructions as llvm vector operations directly. They don't cover a full enough set of operations to make them worthwhile, and nothing will generate them from llvm/clang.

For the cases we do recognize, it's from scalar code either through ISel or the ARMParallelDSPPass.

If you have control of the code, the best bet may be to just use the @llvm.arm.qadd8 intrinsics directly.

[therealprof]

In D105413#2869935, @dmgreen wrote:

We usually don't treat the dsp instructions as llvm vector operations directly. They don't cover a full enough set of operations to make them worthwhile, and nothing will generate them from llvm/clang.

DSP covers quite a lot of cases but for integer operations only. What makes DSP so interesting is that quite a lot of chips support them, starting from Cortex-M4 microcontrollers up to every CPU manufactured in the last decade or so. It's quite curious that there seems to be little interest in picking up this super low-hanging fruit.

You're right that at the moment I don't know of any compiler generating v4i8 and v2i16 types, but with the current state of code lowering it's quite understandable that noone does since this would result in quite a bit of overhead over scalar code, however if the code generation would be better I could totally see compilers making use of autovectorization and/or exposing SIMD types directly. My focus is very much on MCUs and if LLVM can generate good code for DSP I'd work on the Rust side of code generation.

For the cases we do recognize, it's from scalar code either through ISel or the ARMParallelDSPPass.

I am aware. ;)

If you have control of the code, the best bet may be to just use the @llvm.arm.qadd8 intrinsics directly.

Intrinsics are pretty much always a lackluster workaround for me. I want compilers to generate ideal code instead of users having to write it over and over again.

llvm/lib/Target/ARM/ARMISelLowering.h

 //===- ARMISelLowering.h - ARM DAG Lowering Interface -----------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines the interfaces that ARM uses to lower LLVM code into a
enum NodeType : unsigned {
     WLS, // Low-overhead loops, While Loop Start branch. See t2WhileLoopStart
     WLSSETUP, // Setup for the iteration count of a WLS. See t2WhileLoopSetup.
     LOOP_DEC, // Really a part of LE, performs the sub
     LE,       // Low-overhead loops, Loop End
 
     PREDICATE_CAST,  // Predicate cast for MVE i1 types
     VECTOR_REG_CAST, // Reinterpret the current contents of a vector register
 
     MVETRUNC,     // Legalization aid for truncating two vectors into one.
 
     VCMP,  // Vector compare.
     VCMPZ, // Vector compare to zero.
     VTST,  // Vector test bits.
 
     // Vector shift by vector
     VSHLs, // ...left/right by signed
     VSHLu, // ...left/right by unsigned
enum NodeType : unsigned {
     SMMLSR,  // Signed multiply long, subtract and round
 
     // Single Lane QADD8 and QADD16. Only the bottom lane. That's what the b
     // stands for.
     QADD8b,
     QSUB8b,
     QADD16b,
     QSUB16b,
+    UQADD8b,
+    UQSUB8b,
+    UQADD16b,
+    UQSUB16b,
 
     // Operands of the standard BUILD_VECTOR node are not legalized, which
     // is fine if BUILD_VECTORs are always lowered to shuffles or other
     // operations, but for ARM some BUILD_VECTORs are legal as-is and their
     // operands need to be legalized.  Define an ARM-specific version of
     // BUILD_VECTOR for this purpose.
     BUILD_VECTOR,
 

llvm/lib/Target/ARM/ARMISelLowering.cpp

ARMTargetLowering::ARMTargetLowering(const TargetMachine &TM,
 
   setOperationAction(ISD::ADDCARRY, MVT::i32, Custom);
   setOperationAction(ISD::SUBCARRY, MVT::i32, Custom);
   if (Subtarget->hasDSP()) {
     setOperationAction(ISD::SADDSAT, MVT::i8, Custom);
     setOperationAction(ISD::SSUBSAT, MVT::i8, Custom);
     setOperationAction(ISD::SADDSAT, MVT::i16, Custom);
     setOperationAction(ISD::SSUBSAT, MVT::i16, Custom);
+    setOperationAction(ISD::UADDSAT, MVT::i8, Custom);
+    setOperationAction(ISD::USUBSAT, MVT::i8, Custom);
+    setOperationAction(ISD::UADDSAT, MVT::i16, Custom);
+    setOperationAction(ISD::USUBSAT, MVT::i16, Custom);
   }
   if (Subtarget->hasBaseDSP()) {
     setOperationAction(ISD::SADDSAT, MVT::i32, Legal);
     setOperationAction(ISD::SSUBSAT, MVT::i32, Legal);
   }
 
   // i64 operation support.
   setOperationAction(ISD::MUL,     MVT::i64, Expand);
case ARMISD::FIRST_NUMBER:
     MAKE_CASE(ARMISD::SMLSLD)
     MAKE_CASE(ARMISD::SMLSLDX)
     MAKE_CASE(ARMISD::SMMLAR)
     MAKE_CASE(ARMISD::SMMLSR)
     MAKE_CASE(ARMISD::QADD16b)
     MAKE_CASE(ARMISD::QSUB16b)
     MAKE_CASE(ARMISD::QADD8b)
     MAKE_CASE(ARMISD::QSUB8b)
+    MAKE_CASE(ARMISD::UQADD16b)
+    MAKE_CASE(ARMISD::UQSUB16b)
+    MAKE_CASE(ARMISD::UQADD8b)
+    MAKE_CASE(ARMISD::UQSUB8b)
     MAKE_CASE(ARMISD::BUILD_VECTOR)
     MAKE_CASE(ARMISD::BFI)
     MAKE_CASE(ARMISD::VORRIMM)
     MAKE_CASE(ARMISD::VBICIMM)
     MAKE_CASE(ARMISD::VBSP)
     MAKE_CASE(ARMISD::MEMCPY)
     MAKE_CASE(ARMISD::VLD1DUP)
     MAKE_CASE(ARMISD::VLD2DUP)
static SDValue LowerSADDSUBSAT(SDValue Op, SelectionDAG &DAG,
   SDLoc dl(Op);
   SDValue Add =
       DAG.getNode(NewOpcode, dl, MVT::i32,
                   DAG.getSExtOrTrunc(Op->getOperand(0), dl, MVT::i32),
                   DAG.getSExtOrTrunc(Op->getOperand(1), dl, MVT::i32));
   return DAG.getNode(ISD::TRUNCATE, dl, VT, Add);
 }
 
+static SDValue LowerUADDSUBSAT(SDValue Op, SelectionDAG &DAG,
+                               const ARMSubtarget *Subtarget) {
+  EVT VT = Op.getValueType();
+  if (!Subtarget->hasV6Ops() || !Subtarget->hasDSP())
+    return SDValue();
+  if (!VT.isSimple())
+    return SDValue();
+
+  unsigned NewOpcode;
+  bool IsAdd = Op->getOpcode() == ISD::UADDSAT;
+  switch (VT.getSimpleVT().SimpleTy) {
+  default:
+    return SDValue();
+  case MVT::i8:
+    NewOpcode = IsAdd ? ARMISD::UQADD8b : ARMISD::UQSUB8b;
+    break;
+  case MVT::i16:
+    NewOpcode = IsAdd ? ARMISD::UQADD16b : ARMISD::UQSUB16b;
+    break;
+  }
+
+  SDLoc dl(Op);
+  SDValue Add =
+      DAG.getNode(NewOpcode, dl, MVT::i32,
+                  DAG.getSExtOrTrunc(Op->getOperand(0), dl, MVT::i32),
+                  DAG.getSExtOrTrunc(Op->getOperand(1), dl, MVT::i32));
+  return DAG.getNode(ISD::TRUNCATE, dl, VT, Add);
+}
+
 SDValue ARMTargetLowering::LowerSELECT(SDValue Op, SelectionDAG &DAG) const {
   SDValue Cond = Op.getOperand(0);
   SDValue SelectTrue = Op.getOperand(1);
   SDValue SelectFalse = Op.getOperand(2);
   SDLoc dl(Op);
   unsigned Opc = Cond.getOpcode();
 
   if (Cond.getResNo() == 1 &&
SDValue ARMTargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::SSUBO:
     return LowerSignedALUO(Op, DAG);
   case ISD::UADDO:
   case ISD::USUBO:
     return LowerUnsignedALUO(Op, DAG);
   case ISD::SADDSAT:
   case ISD::SSUBSAT:
     return LowerSADDSUBSAT(Op, DAG, Subtarget);
+  case ISD::UADDSAT:
+  case ISD::USUBSAT:
+    return LowerUADDSUBSAT(Op, DAG, Subtarget);
   case ISD::LOAD:
     return LowerPredicateLoad(Op, DAG);
   case ISD::STORE:
     return LowerSTORE(Op, DAG, Subtarget);
   case ISD::MLOAD:
     return LowerMLOAD(Op, DAG);
   case ISD::VECREDUCE_MUL:
   case ISD::VECREDUCE_AND:
case ISD::UDIVREM:
     assert(Res.getNumOperands() == 2 && "DivRem needs two values");
     Results.push_back(Res.getValue(0));
     Results.push_back(Res.getValue(1));
     return;
   case ISD::SADDSAT:
   case ISD::SSUBSAT:
     Res = LowerSADDSUBSAT(SDValue(N, 0), DAG, Subtarget);
     break;
+  case ISD::UADDSAT:
+  case ISD::USUBSAT:
+    Res = LowerUADDSUBSAT(SDValue(N, 0), DAG, Subtarget);
+    break;
   case ISD::READCYCLECOUNTER:
     ReplaceREADCYCLECOUNTER(N, Results, DAG, Subtarget);
     return;
   case ISD::UDIV:
   case ISD::SDIV:
     assert(Subtarget->isTargetWindows() && "can only expand DIV on Windows");
     return ExpandDIV_Windows(SDValue(N, 0), DAG, N->getOpcode() == ISD::SDIV,
                              Results);
case ARMISD::SMULWT: {
     unsigned BitWidth = N->getValueType(0).getSizeInBits();
     APInt DemandedMask = APInt::getHighBitsSet(BitWidth, 16);
     if (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI))
       return SDValue();
     break;
   }
   case ARMISD::SMLALBB:
   case ARMISD::QADD16b:
-  case ARMISD::QSUB16b: {
+  case ARMISD::QSUB16b:
+  case ARMISD::UQADD16b:
+  case ARMISD::UQSUB16b: {
     unsigned BitWidth = N->getValueType(0).getSizeInBits();
     APInt DemandedMask = APInt::getLowBitsSet(BitWidth, 16);
     if ((SimplifyDemandedBits(N->getOperand(0), DemandedMask, DCI)) ||
         (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI)))
       return SDValue();
     break;
   }
   case ARMISD::SMLALBT: {
case ARMISD::SMLALTT: {
     unsigned BitWidth = N->getValueType(0).getSizeInBits();
     APInt DemandedMask = APInt::getHighBitsSet(BitWidth, 16);
     if ((SimplifyDemandedBits(N->getOperand(0), DemandedMask, DCI)) ||
         (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI)))
       return SDValue();
     break;
   }
   case ARMISD::QADD8b:
-  case ARMISD::QSUB8b: {
+  case ARMISD::QSUB8b:
+  case ARMISD::UQADD8b:
+  case ARMISD::UQSUB8b: {
     unsigned BitWidth = N->getValueType(0).getSizeInBits();
     APInt DemandedMask = APInt::getLowBitsSet(BitWidth, 8);
     if ((SimplifyDemandedBits(N->getOperand(0), DemandedMask, DCI)) ||
         (SimplifyDemandedBits(N->getOperand(1), DemandedMask, DCI)))
       return SDValue();
     break;
   }
   case ISD::INTRINSIC_VOID:

llvm/lib/Target/ARM/ARMInstrInfo.td

 def ARMsmlaltb      : SDNode<"ARMISD::SMLALTB", SDT_LongMac, []>;
 def ARMsmlaltt      : SDNode<"ARMISD::SMLALTT", SDT_LongMac, []>;
 
 def ARMqadd8b       : SDNode<"ARMISD::QADD8b", SDT_ARMAnd, []>;
 def ARMqsub8b       : SDNode<"ARMISD::QSUB8b", SDT_ARMAnd, []>;
 def ARMqadd16b      : SDNode<"ARMISD::QADD16b", SDT_ARMAnd, []>;
 def ARMqsub16b      : SDNode<"ARMISD::QSUB16b", SDT_ARMAnd, []>;
 
+def ARMuqadd8b       : SDNode<"ARMISD::UQADD8b", SDT_ARMAnd, []>;
+def ARMuqsub8b       : SDNode<"ARMISD::UQSUB8b", SDT_ARMAnd, []>;
+def ARMuqadd16b      : SDNode<"ARMISD::UQADD16b", SDT_ARMAnd, []>;
+def ARMuqsub16b      : SDNode<"ARMISD::UQSUB16b", SDT_ARMAnd, []>;
+
 def SDT_ARMldrd     : SDTypeProfile<2, 1, [SDTCisVT<0, i32>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 def ARMldrd         : SDNode<"ARMISD::LDRD", SDT_ARMldrd, [SDNPHasChain, SDNPMayLoad, SDNPMemOperand]>;
 
 def SDT_ARMstrd     : SDTypeProfile<0, 3, [SDTCisVT<0, i32>, SDTCisSameAs<0, 1>, SDTCisPtrTy<2>]>;
 def ARMstrd         : SDNode<"ARMISD::STRD", SDT_ARMstrd, [SDNPHasChain, SDNPMayStore, SDNPMemOperand]>;
 
 // Vector operations shared between NEON and MVE
 

 def : ARMV5TEPat<(saddsat GPR:$a, GPR:$b),
                  (QADD GPR:$a, GPR:$b)>;
 def : ARMV5TEPat<(ssubsat GPR:$a, GPR:$b),
                  (QSUB GPR:$a, GPR:$b)>;
 def : ARMV5TEPat<(saddsat rGPR:$Rm, (saddsat rGPR:$Rn, rGPR:$Rn)),
                  (QDADD rGPR:$Rm, rGPR:$Rn)>;
 def : ARMV5TEPat<(ssubsat rGPR:$Rm, (saddsat rGPR:$Rn, rGPR:$Rn)),
                  (QDSUB rGPR:$Rm, rGPR:$Rn)>;
+
 def : ARMV6Pat<(ARMqadd8b rGPR:$Rm, rGPR:$Rn),
                (QADD8 rGPR:$Rm, rGPR:$Rn)>;
 def : ARMV6Pat<(ARMqsub8b rGPR:$Rm, rGPR:$Rn),
                (QSUB8 rGPR:$Rm, rGPR:$Rn)>;
 def : ARMV6Pat<(ARMqadd16b rGPR:$Rm, rGPR:$Rn),
                (QADD16 rGPR:$Rm, rGPR:$Rn)>;
 def : ARMV6Pat<(ARMqsub16b rGPR:$Rm, rGPR:$Rn),
                (QSUB16 rGPR:$Rm, rGPR:$Rn)>;
 
 def UQADD16 : AAIIntrinsic<0b01100110, 0b11110001, "uqadd16", int_arm_uqadd16>;
 def UQADD8  : AAIIntrinsic<0b01100110, 0b11111001, "uqadd8", int_arm_uqadd8>;
 def UQSUB16 : AAIIntrinsic<0b01100110, 0b11110111, "uqsub16", int_arm_uqsub16>;
 def UQSUB8  : AAIIntrinsic<0b01100110, 0b11111111, "uqsub8", int_arm_uqsub8>;
 def QASX    : AAIIntrinsic<0b01100010, 0b11110011, "qasx", int_arm_qasx>;
 def QSAX    : AAIIntrinsic<0b01100010, 0b11110101, "qsax", int_arm_qsax>;
 def UQASX   : AAIIntrinsic<0b01100110, 0b11110011, "uqasx", int_arm_uqasx>;
 def UQSAX   : AAIIntrinsic<0b01100110, 0b11110101, "uqsax", int_arm_uqsax>;
 
+def : ARMV6Pat<(ARMuqadd8b rGPR:$Rm, rGPR:$Rn),
+               (UQADD8 rGPR:$Rm, rGPR:$Rn)>;
+def : ARMV6Pat<(ARMuqsub8b rGPR:$Rm, rGPR:$Rn),
+               (UQSUB8 rGPR:$Rm, rGPR:$Rn)>;
+def : ARMV6Pat<(ARMuqadd16b rGPR:$Rm, rGPR:$Rn),
+               (UQADD16 rGPR:$Rm, rGPR:$Rn)>;
+def : ARMV6Pat<(ARMuqsub16b rGPR:$Rm, rGPR:$Rn),
+               (UQSUB16 rGPR:$Rm, rGPR:$Rn)>;
+
+
 // Signed/Unsigned add/subtract
 
 def SASX   : AAIIntrinsic<0b01100001, 0b11110011, "sasx", int_arm_sasx>;
 def SADD16 : AAIIntrinsic<0b01100001, 0b11110001, "sadd16", int_arm_sadd16>;
 def SADD8  : AAIIntrinsic<0b01100001, 0b11111001, "sadd8", int_arm_sadd8>;
 def SSAX   : AAIIntrinsic<0b01100001, 0b11110101, "ssax", int_arm_ssax>;
 def SSUB16 : AAIIntrinsic<0b01100001, 0b11110111, "ssub16", int_arm_ssub16>;
 def SSUB8  : AAIIntrinsic<0b01100001, 0b11111111, "ssub8", int_arm_ssub8>;

llvm/lib/Target/ARM/ARMInstrThumb2.td

 def : Thumb2DSPPat<(saddsat rGPR:$Rm, rGPR:$Rn),
                    (t2QADD rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(ssubsat rGPR:$Rm, rGPR:$Rn),
                    (t2QSUB rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(saddsat rGPR:$Rm, (saddsat rGPR:$Rn, rGPR:$Rn)),
                    (t2QDADD rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(ssubsat rGPR:$Rm, (saddsat rGPR:$Rn, rGPR:$Rn)),
                    (t2QDSUB rGPR:$Rm, rGPR:$Rn)>;
+
 def : Thumb2DSPPat<(ARMqadd8b rGPR:$Rm, rGPR:$Rn),
                    (t2QADD8 rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(ARMqsub8b rGPR:$Rm, rGPR:$Rn),
                    (t2QSUB8 rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(ARMqadd16b rGPR:$Rm, rGPR:$Rn),
                    (t2QADD16 rGPR:$Rm, rGPR:$Rn)>;
 def : Thumb2DSPPat<(ARMqsub16b rGPR:$Rm, rGPR:$Rn),
                    (t2QSUB16 rGPR:$Rm, rGPR:$Rn)>;
 
+def : Thumb2DSPPat<(ARMuqadd8b rGPR:$Rm, rGPR:$Rn),
+                   (t2UQADD8 rGPR:$Rm, rGPR:$Rn)>;
+def : Thumb2DSPPat<(ARMuqsub8b rGPR:$Rm, rGPR:$Rn),
+                   (t2UQSUB8 rGPR:$Rm, rGPR:$Rn)>;
+def : Thumb2DSPPat<(ARMuqadd16b rGPR:$Rm, rGPR:$Rn),
+                   (t2UQADD16 rGPR:$Rm, rGPR:$Rn)>;
+def : Thumb2DSPPat<(ARMuqsub16b rGPR:$Rm, rGPR:$Rn),
+                   (t2UQSUB16 rGPR:$Rm, rGPR:$Rn)>;
+
 // Signed/Unsigned add/subtract
 
 def t2SASX    : T2I_pam_intrinsics<0b010, 0b0000, "sasx", int_arm_sasx>;
 def t2SADD16  : T2I_pam_intrinsics<0b001, 0b0000, "sadd16", int_arm_sadd16>;
 def t2SADD8   : T2I_pam_intrinsics<0b000, 0b0000, "sadd8", int_arm_sadd8>;
 def t2SSAX    : T2I_pam_intrinsics<0b110, 0b0000, "ssax", int_arm_ssax>;
 def t2SSUB16  : T2I_pam_intrinsics<0b101, 0b0000, "ssub16", int_arm_ssub16>;
 def t2SSUB8   : T2I_pam_intrinsics<0b100, 0b0000, "ssub8", int_arm_ssub8>;

llvm/test/CodeGen/ARM/uadd_sat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=thumbv6m-none-eabi | FileCheck %s --check-prefix=CHECK-T1
-; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2
-; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2
+; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2NODSP
+; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2DSP
 ; RUN: llc < %s -mtriple=armv8a-none-eabi | FileCheck %s --check-prefix=CHECK-ARM
 
 declare i4 @llvm.uadd.sat.i4(i4, i4)
 declare i8 @llvm.uadd.sat.i8(i8, i8)
 declare i16 @llvm.uadd.sat.i16(i16, i16)
 declare i32 @llvm.uadd.sat.i32(i32, i32)
 declare i64 @llvm.uadd.sat.i64(i64, i64)
 

 ; CHECK-T1-NEXT:    mov r0, r1
 ; CHECK-T1-NEXT:  .LBB2_2:
 ; CHECK-T1-NEXT:    bx lr
 ; CHECK-T1-NEXT:    .p2align 2
 ; CHECK-T1-NEXT:  @ %bb.3:
 ; CHECK-T1-NEXT:  .LCPI2_0:
 ; CHECK-T1-NEXT:    .long 65535 @ 0xffff
 ;
-; CHECK-T2-LABEL: func16:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    add r1, r0
-; CHECK-T2-NEXT:    movw r0, #65535
-; CHECK-T2-NEXT:    cmp r1, r0
-; CHECK-T2-NEXT:    it lo
-; CHECK-T2-NEXT:    movlo r0, r1
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func16:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    add r1, r0
+; CHECK-T2NODSP-NEXT:    movw r0, #65535
+; CHECK-T2NODSP-NEXT:    cmp r1, r0
+; CHECK-T2NODSP-NEXT:    it lo
+; CHECK-T2NODSP-NEXT:    movlo r0, r1
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func16:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    uqadd16 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxth r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func16:
 ; CHECK-ARM:       @ %bb.0:
-; CHECK-ARM-NEXT:    add r1, r0, r1
-; CHECK-ARM-NEXT:    movw r0, #65535
-; CHECK-ARM-NEXT:    cmp r1, r0
-; CHECK-ARM-NEXT:    movlo r0, r1
+; CHECK-ARM-NEXT:    uqadd16 r0, r0, r1
+; CHECK-ARM-NEXT:    uxth r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %tmp = call i16 @llvm.uadd.sat.i16(i16 %x, i16 %y)
   ret i16 %tmp
 }
 
 define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y) nounwind {
 ; CHECK-T1-LABEL: func8:
 ; CHECK-T1:       @ %bb.0:
 ; CHECK-T1-NEXT:    adds r0, r0, r1
 ; CHECK-T1-NEXT:    cmp r0, #255
 ; CHECK-T1-NEXT:    blo .LBB3_2
 ; CHECK-T1-NEXT:  @ %bb.1:
 ; CHECK-T1-NEXT:    movs r0, #255
 ; CHECK-T1-NEXT:  .LBB3_2:
 ; CHECK-T1-NEXT:    bx lr
 ;
-; CHECK-T2-LABEL: func8:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    add r0, r1
-; CHECK-T2-NEXT:    cmp r0, #255
-; CHECK-T2-NEXT:    it hs
-; CHECK-T2-NEXT:    movhs r0, #255
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func8:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    add r0, r1
+; CHECK-T2NODSP-NEXT:    cmp r0, #255
+; CHECK-T2NODSP-NEXT:    it hs
+; CHECK-T2NODSP-NEXT:    movhs r0, #255
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func8:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    uqadd8 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxtb r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func8:
 ; CHECK-ARM:       @ %bb.0:
-; CHECK-ARM-NEXT:    add r0, r0, r1
-; CHECK-ARM-NEXT:    cmp r0, #255
-; CHECK-ARM-NEXT:    movhs r0, #255
+; CHECK-ARM-NEXT:    uqadd8 r0, r0, r1
+; CHECK-ARM-NEXT:    uxtb r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %tmp = call i8 @llvm.uadd.sat.i8(i8 %x, i8 %y)
   ret i8 %tmp
 }
 
 define zeroext i4 @func3(i4 zeroext %x, i4 zeroext %y) nounwind {
 ; CHECK-T1-LABEL: func3:
 ; CHECK-T1:       @ %bb.0:

llvm/test/CodeGen/ARM/uadd_sat_plus.ll

 ; CHECK-T2NODSP-NEXT:    cmp r1, r0
 ; CHECK-T2NODSP-NEXT:    it lo
 ; CHECK-T2NODSP-NEXT:    movlo r0, r1
 ; CHECK-T2NODSP-NEXT:    bx lr
 ;
 ; CHECK-T2DSP-LABEL: func16:
 ; CHECK-T2DSP:       @ %bb.0:
 ; CHECK-T2DSP-NEXT:    muls r1, r2, r1
-; CHECK-T2DSP-NEXT:    uxtah r1, r0, r1
-; CHECK-T2DSP-NEXT:    movw r0, #65535
-; CHECK-T2DSP-NEXT:    cmp r1, r0
-; CHECK-T2DSP-NEXT:    it lo
-; CHECK-T2DSP-NEXT:    movlo r0, r1
+; CHECK-T2DSP-NEXT:    uqadd16 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxth r0, r0
 ; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func16:
 ; CHECK-ARM:       @ %bb.0:
 ; CHECK-ARM-NEXT:    mul r1, r1, r2
-; CHECK-ARM-NEXT:    uxtah r1, r0, r1
-; CHECK-ARM-NEXT:    movw r0, #65535
-; CHECK-ARM-NEXT:    cmp r1, r0
-; CHECK-ARM-NEXT:    movlo r0, r1
+; CHECK-ARM-NEXT:    uqadd16 r0, r0, r1
+; CHECK-ARM-NEXT:    uxth r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %a = mul i16 %y, %z
   %tmp = call i16 @llvm.uadd.sat.i16(i16 %x, i16 %a)
   ret i16 %tmp
 }
 
 define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y, i8 zeroext %z) nounwind {
 ; CHECK-T1-LABEL: func8:

 ; CHECK-T2NODSP-NEXT:    cmp r0, #255
 ; CHECK-T2NODSP-NEXT:    it hs
 ; CHECK-T2NODSP-NEXT:    movhs r0, #255
 ; CHECK-T2NODSP-NEXT:    bx lr
 ;
 ; CHECK-T2DSP-LABEL: func8:
 ; CHECK-T2DSP:       @ %bb.0:
 ; CHECK-T2DSP-NEXT:    muls r1, r2, r1
-; CHECK-T2DSP-NEXT:    uxtab r0, r0, r1
-; CHECK-T2DSP-NEXT:    cmp r0, #255
-; CHECK-T2DSP-NEXT:    it hs
-; CHECK-T2DSP-NEXT:    movhs r0, #255
+; CHECK-T2DSP-NEXT:    uqadd8 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxtb r0, r0
 ; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func8:
 ; CHECK-ARM:       @ %bb.0:
 ; CHECK-ARM-NEXT:    smulbb r1, r1, r2
-; CHECK-ARM-NEXT:    uxtab r0, r0, r1
-; CHECK-ARM-NEXT:    cmp r0, #255
-; CHECK-ARM-NEXT:    movhs r0, #255
+; CHECK-ARM-NEXT:    uqadd8 r0, r0, r1
+; CHECK-ARM-NEXT:    uxtb r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %a = mul i8 %y, %z
   %tmp = call i8 @llvm.uadd.sat.i8(i8 %x, i8 %a)
   ret i8 %tmp
 }
 
 define zeroext i4 @func4(i4 zeroext %x, i4 zeroext %y, i4 zeroext %z) nounwind {
 ; CHECK-T1-LABEL: func4:

llvm/test/CodeGen/ARM/usub_sat.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=thumbv6m-none-eabi | FileCheck %s --check-prefix=CHECK-T1
-; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2
-; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2
+; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2NODSP
+; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2DSP
 ; RUN: llc < %s -mtriple=armv8a-none-eabi | FileCheck %s --check-prefix=CHECK-ARM
 
 declare i4 @llvm.usub.sat.i4(i4, i4)
 declare i8 @llvm.usub.sat.i8(i8, i8)
 declare i16 @llvm.usub.sat.i16(i16, i16)
 declare i32 @llvm.usub.sat.i32(i32, i32)
 declare i64 @llvm.usub.sat.i64(i64, i64)
 

 ; CHECK-T1:       @ %bb.0:
 ; CHECK-T1-NEXT:    subs r0, r0, r1
 ; CHECK-T1-NEXT:    bhs .LBB2_2
 ; CHECK-T1-NEXT:  @ %bb.1:
 ; CHECK-T1-NEXT:    movs r0, #0
 ; CHECK-T1-NEXT:  .LBB2_2:
 ; CHECK-T1-NEXT:    bx lr
 ;
-; CHECK-T2-LABEL: func16:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    subs r0, r0, r1
-; CHECK-T2-NEXT:    it lo
-; CHECK-T2-NEXT:    movlo r0, #0
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func16:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    subs r0, r0, r1
+; CHECK-T2NODSP-NEXT:    it lo
+; CHECK-T2NODSP-NEXT:    movlo r0, #0
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func16:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    uqsub16 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxth r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func16:
 ; CHECK-ARM:       @ %bb.0:
-; CHECK-ARM-NEXT:    subs r0, r0, r1
-; CHECK-ARM-NEXT:    movlo r0, #0
+; CHECK-ARM-NEXT:    uqsub16 r0, r0, r1
+; CHECK-ARM-NEXT:    uxth r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %tmp = call i16 @llvm.usub.sat.i16(i16 %x, i16 %y)
   ret i16 %tmp
 }
 
 define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y) nounwind {
 ; CHECK-T1-LABEL: func8:
 ; CHECK-T1:       @ %bb.0:
 ; CHECK-T1-NEXT:    subs r0, r0, r1
 ; CHECK-T1-NEXT:    bhs .LBB3_2
 ; CHECK-T1-NEXT:  @ %bb.1:
 ; CHECK-T1-NEXT:    movs r0, #0
 ; CHECK-T1-NEXT:  .LBB3_2:
 ; CHECK-T1-NEXT:    bx lr
 ;
-; CHECK-T2-LABEL: func8:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    subs r0, r0, r1
-; CHECK-T2-NEXT:    it lo
-; CHECK-T2-NEXT:    movlo r0, #0
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func8:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    subs r0, r0, r1
+; CHECK-T2NODSP-NEXT:    it lo
+; CHECK-T2NODSP-NEXT:    movlo r0, #0
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func8:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    uqsub8 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxtb r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func8:
 ; CHECK-ARM:       @ %bb.0:
-; CHECK-ARM-NEXT:    subs r0, r0, r1
-; CHECK-ARM-NEXT:    movlo r0, #0
+; CHECK-ARM-NEXT:    uqsub8 r0, r0, r1
+; CHECK-ARM-NEXT:    uxtb r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %tmp = call i8 @llvm.usub.sat.i8(i8 %x, i8 %y)
   ret i8 %tmp
 }
 
 define zeroext i4 @func3(i4 zeroext %x, i4 zeroext %y) nounwind {
 ; CHECK-T1-LABEL: func3:
 ; CHECK-T1:       @ %bb.0:

llvm/test/CodeGen/ARM/usub_sat_plus.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s -mtriple=thumbv6m-none-eabi | FileCheck %s --check-prefix=CHECK-T1
-; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2
-; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2
+; RUN: llc < %s -mtriple=thumbv7m-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2NODSP
+; RUN: llc < %s -mtriple=thumbv7em-none-eabi | FileCheck %s --check-prefix=CHECK-T2 --check-prefix=CHECK-T2DSP
 ; RUN: llc < %s -mtriple=armv8a-none-eabi | FileCheck %s --check-prefix=CHECK-ARM
 
 declare i4 @llvm.usub.sat.i4(i4, i4)
 declare i8 @llvm.usub.sat.i8(i8, i8)
 declare i16 @llvm.usub.sat.i16(i16, i16)
 declare i32 @llvm.usub.sat.i32(i32, i32)
 declare i64 @llvm.usub.sat.i64(i64, i64)
 

 ; CHECK-T1-NEXT:    uxth r1, r1
 ; CHECK-T1-NEXT:    subs r0, r0, r1
 ; CHECK-T1-NEXT:    bhs .LBB2_2
 ; CHECK-T1-NEXT:  @ %bb.1:
 ; CHECK-T1-NEXT:    movs r0, #0
 ; CHECK-T1-NEXT:  .LBB2_2:
 ; CHECK-T1-NEXT:    bx lr
 ;
-; CHECK-T2-LABEL: func16:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    muls r1, r2, r1
-; CHECK-T2-NEXT:    uxth r1, r1
-; CHECK-T2-NEXT:    subs r0, r0, r1
-; CHECK-T2-NEXT:    it lo
-; CHECK-T2-NEXT:    movlo r0, #0
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func16:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    muls r1, r2, r1
+; CHECK-T2NODSP-NEXT:    uxth r1, r1
+; CHECK-T2NODSP-NEXT:    subs r0, r0, r1
+; CHECK-T2NODSP-NEXT:    it lo
+; CHECK-T2NODSP-NEXT:    movlo r0, #0
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func16:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    muls r1, r2, r1
+; CHECK-T2DSP-NEXT:    uqsub16 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxth r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func16:
 ; CHECK-ARM:       @ %bb.0:
 ; CHECK-ARM-NEXT:    mul r1, r1, r2
-; CHECK-ARM-NEXT:    uxth r1, r1
-; CHECK-ARM-NEXT:    subs r0, r0, r1
-; CHECK-ARM-NEXT:    movlo r0, #0
+; CHECK-ARM-NEXT:    uqsub16 r0, r0, r1
+; CHECK-ARM-NEXT:    uxth r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %a = mul i16 %y, %z
   %tmp = call i16 @llvm.usub.sat.i16(i16 %x, i16 %a)
   ret i16 %tmp
 }
 
 define zeroext i8 @func8(i8 zeroext %x, i8 zeroext %y, i8 zeroext %z) nounwind {
 ; CHECK-T1-LABEL: func8:
 ; CHECK-T1:       @ %bb.0:
 ; CHECK-T1-NEXT:    muls r1, r2, r1
 ; CHECK-T1-NEXT:    uxtb r1, r1
 ; CHECK-T1-NEXT:    subs r0, r0, r1
 ; CHECK-T1-NEXT:    bhs .LBB3_2
 ; CHECK-T1-NEXT:  @ %bb.1:
 ; CHECK-T1-NEXT:    movs r0, #0
 ; CHECK-T1-NEXT:  .LBB3_2:
 ; CHECK-T1-NEXT:    bx lr
 ;
-; CHECK-T2-LABEL: func8:
-; CHECK-T2:       @ %bb.0:
-; CHECK-T2-NEXT:    muls r1, r2, r1
-; CHECK-T2-NEXT:    uxtb r1, r1
-; CHECK-T2-NEXT:    subs r0, r0, r1
-; CHECK-T2-NEXT:    it lo
-; CHECK-T2-NEXT:    movlo r0, #0
-; CHECK-T2-NEXT:    bx lr
+; CHECK-T2NODSP-LABEL: func8:
+; CHECK-T2NODSP:       @ %bb.0:
+; CHECK-T2NODSP-NEXT:    muls r1, r2, r1
+; CHECK-T2NODSP-NEXT:    uxtb r1, r1
+; CHECK-T2NODSP-NEXT:    subs r0, r0, r1
+; CHECK-T2NODSP-NEXT:    it lo
+; CHECK-T2NODSP-NEXT:    movlo r0, #0
+; CHECK-T2NODSP-NEXT:    bx lr
+;
+; CHECK-T2DSP-LABEL: func8:
+; CHECK-T2DSP:       @ %bb.0:
+; CHECK-T2DSP-NEXT:    muls r1, r2, r1
+; CHECK-T2DSP-NEXT:    uqsub8 r0, r0, r1
+; CHECK-T2DSP-NEXT:    uxtb r0, r0
+; CHECK-T2DSP-NEXT:    bx lr
 ;
 ; CHECK-ARM-LABEL: func8:
 ; CHECK-ARM:       @ %bb.0:
 ; CHECK-ARM-NEXT:    smulbb r1, r1, r2
-; CHECK-ARM-NEXT:    uxtb r1, r1
-; CHECK-ARM-NEXT:    subs r0, r0, r1
-; CHECK-ARM-NEXT:    movlo r0, #0
+; CHECK-ARM-NEXT:    uqsub8 r0, r0, r1
+; CHECK-ARM-NEXT:    uxtb r0, r0
 ; CHECK-ARM-NEXT:    bx lr
   %a = mul i8 %y, %z
   %tmp = call i8 @llvm.usub.sat.i8(i8 %x, i8 %a)
   ret i8 %tmp
 }
 
 define zeroext i4 @func4(i4 zeroext %x, i4 zeroext %y, i4 zeroext %z) nounwind {
 ; CHECK-T1-LABEL: func4: