[X86] Use UADDSAT/USUBSAT instead of ADDUS/SUBUS

[nikic]

Summary

Part of https://bugs.llvm.org/show_bug.cgi?id=40056.

This removes the ADDUS/SUBUS X86ISD opcodes in favor of the generic UADDSAT/USUBSAT opcodes, without touching anything else.

I've marked a bunch of types like v8i8 as custom legalized, so they are widened (as now) rather than promoted (as default). Is there any more elegant way to do this, by reusing the generic vector widening code, rather than the custom code in X86 ReplaceNodeResults?

[RKSimon]

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

605

Op1.getScalarValueSizeInBits();

lib/Target/X86/X86ISelLowering.cpp

26301–26302

X86ISD::AVG/VPMADDWD or ISD::UADDSAT/USUBSAT

[nikic]

Use getScalarValueSizeInBits(), update comment.

[craig.topper]

LGTM. I'll rebase D55780 after this goes in.

[RKSimon]

lib/Target/X86/X86ISelLowering.cpp

1217

These need to use the HasInt256 ? Legal : Custom and you need to add a split256IntArith lowering for AVX1 targets (see LowerADD_SUB).

[craig.topper]

You’ll need to use an intrinsic to test the changes Simon asked for. The existing combines for X86 do their own split for AVX1.

[RKSimon]

In D55787#1333569, @craig.topper wrote:

You’ll need to use an intrinsic to test the changes Simon asked for. The existing combines for X86 do their own split for AVX1.

If that is going to be tricky you can just add a Subtarget->hasInt256() limit and a TODO comment.

[nikic]

In D55787#1333621, @RKSimon wrote:

In D55787#1333569, @craig.topper wrote:

You’ll need to use an intrinsic to test the changes Simon asked for. The existing combines for X86 do their own split for AVX1.

If that is going to be tricky you can just add a Subtarget->hasInt256() limit and a TODO comment.

I plan to add tests for llvm.uadd.sat/llvm.usub.sat with vectors, which are also going to exercise this case. We need those tests anyway :)

[nikic]

Handle 256-bit vector split, add vector tests for llvm.uadd.sub, llvm.usub.sat.

[nikic]

test/CodeGen/X86/uadd_sat_vec.ll

615

This expansion is pretty crazy...

[craig.topper]

test/CodeGen/X86/uadd_sat_vec.ll

615

Yeah I wouldn't expect vXi1 to be good, but this is even worse than I imagined. Its definitely being scalarized and then promoted. But the scalarization looks weird. But I'm not sure how much we care.

We can pretty easily replace with vXi1 ADDUSAT with OR X,Y and SUBUSAT with X & ~Y if we care.

620

Can you add nounwind to the test case to drop the .cfi directives

[RKSimon]

test/CodeGen/X86/uadd_sat_vec.ll

615

You can either add vXi1 handling to LowerUADDSAT_USUBSAT in a similar way to LowerADD_SUB or you can add a DAGCombine transform just to convert them - I don't think we're losing any info converting to bool logic ops?

test/CodeGen/X86/usub_sat_vec.ll

3

Might as well use --check-prefixes

--check-prefixes=SSE,SSE2

[nikic]

Custom lower vNi1, use --check-prefixes.

[nikic]

test/CodeGen/X86/uadd_sat_vec.ll

615

I've handled this in LowerUADDSAT_USUBSAT for now, with the suggested OR and AND+NOT expansions. If/when we move the X86 ADDUS/SUBUS combines into generic DAGCombine we can move this one as well.

[RKSimon]

Thanks - please add the new test files to trunk with current codegen so the patch shows the diff.

[nikic]

Thanks - please add the new test files to trunk with current codegen so the patch shows the diff.

I can't commit these as-is, because the fixes for UADDSAT/USUBSAT legalization at least are needed (there will be assertion failures otherwise). Should I commit the tests together with those changes?

[lebedev.ri]

In D55787#1334280, @nikic wrote:

Thanks - please add the new test files to trunk with current codegen so the patch shows the diff.

I can't commit these as-is, because the fixes for UADDSAT/USUBSAT legalization at least are needed (there will be assertion failures otherwise).

Phab has this feature of being able to specify dependencies between the differentials. It is sometimes useful to do that.

Should I commit the tests together with those changes?

You could at least split the baseline tests into a separate 'parent' differential, and base this differential on that new one.

[RKSimon]

In D55787#1334280, @nikic wrote:

Thanks - please add the new test files to trunk with current codegen so the patch shows the diff.

I can't commit these as-is, because the fixes for UADDSAT/USUBSAT legalization at least are needed (there will be assertion failures otherwise). Should I commit the tests together with those changes?

LGTM - yes please. The legalization + (current codegen) tests, followed by the improved x86 support and tests codegen.

[RKSimon]

@nikic I forgot to ask - are you intending to do the signed add/sub equivalents as well?

[nikic]

@RKSimon Yeah, I plan to migrate the signed ones as well.

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

SDValue DAGTypeLegalizer::PromoteIntRes_ADDSUBSAT(SDNode *N) {
   // For promoting iN -> iM, this can be expanded by
   // 1. ANY_EXTEND iN to iM
   // 2. SHL by M-N
   // 3. [US][ADD|SUB]SAT
   // 4. L/ASHR by M-N
   SDLoc dl(N);
   SDValue Op1 = N->getOperand(0);
   SDValue Op2 = N->getOperand(1);
-  unsigned OldBits = Op1.getValueSizeInBits();
+  unsigned OldBits = Op1.getScalarValueSizeInBits();
 
   unsigned Opcode = N->getOpcode();
   unsigned ShiftOp;
   switch (Opcode) {
   case ISD::SADDSAT:
   case ISD::SSUBSAT:
     ShiftOp = ISD::SRA;
     break;
   case ISD::UADDSAT:
   case ISD::USUBSAT:
     ShiftOp = ISD::SRL;
     break;
   default:
     llvm_unreachable("Expected opcode to be signed or unsigned saturation "
                      "addition or subtraction");
   }
 
   SDValue Op1Promoted = GetPromotedInteger(Op1);
   SDValue Op2Promoted = GetPromotedInteger(Op2);
 
   EVT PromotedType = Op1Promoted.getValueType();
-  unsigned NewBits = Op1Promoted.getValueSizeInBits();
+  unsigned NewBits = PromotedType.getScalarSizeInBits();
   unsigned SHLAmount = NewBits - OldBits;
   EVT SHVT = TLI.getShiftAmountTy(PromotedType, DAG.getDataLayout());
   SDValue ShiftAmount = DAG.getConstant(SHLAmount, dl, SHVT);
   Op1Promoted =
       DAG.getNode(ISD::SHL, dl, PromotedType, Op1Promoted, ShiftAmount);
   Op2Promoted =
       DAG.getNode(ISD::SHL, dl, PromotedType, Op2Promoted, ShiftAmount);
 

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::FMINNUM:
   case ISD::FMAXNUM:
   case ISD::FMINIMUM:
   case ISD::FMAXIMUM:
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
+  case ISD::UADDSAT:
+  case ISD::SADDSAT:
+  case ISD::USUBSAT:
+  case ISD::SSUBSAT:
     Res = WidenVecRes_Binary(N);
     break;
 
   case ISD::FADD:
   case ISD::FMUL:
   case ISD::FPOW:
   case ISD::FSUB:
   case ISD::FDIV:

lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
       // FP vector get exponent.
       FGETEXP_RND, FGETEXPS_RND,
       // Extract Normalized Mantissas.
       VGETMANT, VGETMANT_RND, VGETMANTS, VGETMANTS_RND,
       // FP Scale.
       SCALEF,
       SCALEFS,
 
-      // Integer add/sub with unsigned saturation.
-      ADDUS,
-      SUBUS,
-
       // Integer add/sub with signed saturation.
       ADDS,
       SUBS,
 
       // Unsigned Integer average.
       AVG,
 
       /// Integer horizontal add/sub.

lib/Target/X86/X86ISelLowering.cpp

if (!Subtarget.useSoftFloat() && Subtarget.hasSSE2()) {
 
     for (auto VT : { MVT::v16i8, MVT::v8i16, MVT::v4i32, MVT::v2i64 }) {
       setOperationAction(ISD::SMAX, VT, VT == MVT::v8i16 ? Legal : Custom);
       setOperationAction(ISD::SMIN, VT, VT == MVT::v8i16 ? Legal : Custom);
       setOperationAction(ISD::UMAX, VT, VT == MVT::v16i8 ? Legal : Custom);
       setOperationAction(ISD::UMIN, VT, VT == MVT::v16i8 ? Legal : Custom);
     }
 
+    setOperationAction(ISD::UADDSAT,            MVT::v16i8, Legal);
+    setOperationAction(ISD::UADDSAT,            MVT::v8i16, Legal);
+    setOperationAction(ISD::USUBSAT,            MVT::v16i8, Legal);
+    setOperationAction(ISD::USUBSAT,            MVT::v8i16, Legal);
+    // Use widening instead of promotion.
+    for (auto VT : { MVT::v8i8, MVT::v4i8, MVT::v2i8,
+                     MVT::v4i16, MVT::v2i16 }) {
+      setOperationAction(ISD::UADDSAT,          VT,  Custom);
+      setOperationAction(ISD::USUBSAT,          VT,  Custom);
+    }
+
     setOperationAction(ISD::INSERT_VECTOR_ELT,  MVT::v8i16, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT,  MVT::v4i32, Custom);
     setOperationAction(ISD::INSERT_VECTOR_ELT,  MVT::v4f32, Custom);
 
     // Provide custom widening for v2f32 setcc. This is really for VLX when
     // setcc result type returns v2i1/v4i1 vector for v2f32/v4f32 leading to
     // type legalization changing the result type to v4i1 during widening.
     // It works fine for SSE2 and is probably faster so no need to qualify with
if (!Subtarget.useSoftFloat() && Subtarget.hasAVX()) {
     setOperationAction(ISD::MULHU,     MVT::v32i8,  Custom);
     setOperationAction(ISD::MULHS,     MVT::v32i8,  Custom);
 
     setOperationAction(ISD::SMAX,      MVT::v4i64,  Custom);
     setOperationAction(ISD::UMAX,      MVT::v4i64,  Custom);
     setOperationAction(ISD::SMIN,      MVT::v4i64,  Custom);
     setOperationAction(ISD::UMIN,      MVT::v4i64,  Custom);
 
+    setOperationAction(ISD::UADDSAT,   MVT::v32i8,  HasInt256 ? Legal : Custom);
+    setOperationAction(ISD::UADDSAT,   MVT::v16i16, HasInt256 ? Legal : Custom);
+    setOperationAction(ISD::USUBSAT,   MVT::v32i8,  HasInt256 ? Legal : Custom);
+    setOperationAction(ISD::USUBSAT,   MVT::v16i16, HasInt256 ? Legal : Custom);
+
     for (auto VT : { MVT::v32i8, MVT::v16i16, MVT::v8i32 }) {
       setOperationAction(ISD::ABS,  VT, HasInt256 ? Legal : Custom);
       setOperationAction(ISD::SMAX, VT, HasInt256 ? Legal : Custom);
       setOperationAction(ISD::UMAX, VT, HasInt256 ? Legal : Custom);
       setOperationAction(ISD::SMIN, VT, HasInt256 ? Legal : Custom);
       setOperationAction(ISD::UMIN, VT, HasInt256 ? Legal : Custom);
     }
 
if (!Subtarget.useSoftFloat() && Subtarget.hasAVX512()) {
 
     for (auto VT : { MVT::v2i1, MVT::v4i1, MVT::v8i1, MVT::v16i1 }) {
       setOperationAction(ISD::ADD,              VT, Custom);
       setOperationAction(ISD::SUB,              VT, Custom);
       setOperationAction(ISD::MUL,              VT, Custom);
       setOperationAction(ISD::SETCC,            VT, Custom);
       setOperationAction(ISD::SELECT,           VT, Custom);
       setOperationAction(ISD::TRUNCATE,         VT, Custom);
+      setOperationAction(ISD::UADDSAT,          VT, Custom);
+      setOperationAction(ISD::USUBSAT,          VT, Custom);
 
       setOperationAction(ISD::BUILD_VECTOR,     VT, Custom);
       setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
       setOperationAction(ISD::INSERT_VECTOR_ELT, VT, Custom);
       setOperationAction(ISD::VECTOR_SHUFFLE,   VT,  Custom);
       setOperationAction(ISD::VSELECT,          VT,  Expand);
     }
 
if (!Subtarget.useSoftFloat() && Subtarget.hasBWI()) {
     addRegisterClass(MVT::v32i1,  &X86::VK32RegClass);
     addRegisterClass(MVT::v64i1,  &X86::VK64RegClass);
 
     for (auto VT : { MVT::v32i1, MVT::v64i1 }) {
       setOperationAction(ISD::ADD,                VT, Custom);
       setOperationAction(ISD::SUB,                VT, Custom);
       setOperationAction(ISD::MUL,                VT, Custom);
       setOperationAction(ISD::VSELECT,            VT, Expand);
+      setOperationAction(ISD::UADDSAT,            VT, Custom);
+      setOperationAction(ISD::USUBSAT,            VT, Custom);
 
       setOperationAction(ISD::TRUNCATE,           VT, Custom);
       setOperationAction(ISD::SETCC,              VT, Custom);
       setOperationAction(ISD::EXTRACT_VECTOR_ELT, VT, Custom);
       setOperationAction(ISD::INSERT_VECTOR_ELT,  VT, Custom);
       setOperationAction(ISD::SELECT,             VT, Custom);
       setOperationAction(ISD::BUILD_VECTOR,       VT, Custom);
       setOperationAction(ISD::VECTOR_SHUFFLE,     VT, Custom);
for (auto VT : { MVT::v64i8, MVT::v32i16 }) {
       setOperationAction(ISD::MSTORE,       VT, Legal);
       setOperationAction(ISD::CTPOP,        VT, Custom);
       setOperationAction(ISD::CTLZ,         VT, Custom);
       setOperationAction(ISD::SMAX,         VT, Legal);
       setOperationAction(ISD::UMAX,         VT, Legal);
       setOperationAction(ISD::SMIN,         VT, Legal);
       setOperationAction(ISD::UMIN,         VT, Legal);
       setOperationAction(ISD::SETCC,        VT, Custom);
+      setOperationAction(ISD::UADDSAT,      VT, Legal);
+      setOperationAction(ISD::USUBSAT,      VT, Legal);
 
       // The condition codes aren't legal in SSE/AVX and under AVX512 we use
       // setcc all the way to isel and prefer SETGT in some isel patterns.
       setCondCodeAction(ISD::SETLT, VT, Custom);
       setCondCodeAction(ISD::SETLE, VT, Custom);
     }
 
     for (auto ExtType : {ISD::ZEXTLOAD, ISD::SEXTLOAD}) {
static SDValue LowerVSETCCWithSUBUS(SDValue Op0, SDValue Op1, MVT VT,
   // Psubus is better than flip-sign because it requires no inversion.
   case ISD::SETUGE:
     std::swap(Op0, Op1);
     break;
   case ISD::SETULE:
     break;
   }
 
-  SDValue Result = DAG.getNode(X86ISD::SUBUS, dl, VT, Op0, Op1);
+  SDValue Result = DAG.getNode(ISD::USUBSAT, dl, VT, Op0, Op1);
   return DAG.getNode(X86ISD::PCMPEQ, dl, VT, Result,
                      DAG.getConstant(0, dl, VT));
 }
 
 static SDValue LowerVSETCC(SDValue Op, const X86Subtarget &Subtarget,
                            SelectionDAG &DAG) {
   SDValue Op0 = Op.getOperand(0);
   SDValue Op1 = Op.getOperand(1);
if (VT.getScalarType() == MVT::i1)
     return DAG.getNode(ISD::XOR, SDLoc(Op), VT,
                        Op.getOperand(0), Op.getOperand(1));
   assert(Op.getSimpleValueType().is256BitVector() &&
          Op.getSimpleValueType().isInteger() &&
          "Only handle AVX 256-bit vector integer operation");
   return split256IntArith(Op, DAG);
 }
 
+static SDValue LowerUADDSAT_USUBSAT(SDValue Op, SelectionDAG &DAG) {
+  MVT VT = Op.getSimpleValueType();
+  if (VT.getScalarType() == MVT::i1) {
+    SDLoc dl(Op);
+    switch (Op.getOpcode()) {
+    default: llvm_unreachable("Expected saturated arithmetic opcode");
+    case ISD::UADDSAT:
+      return DAG.getNode(ISD::OR, dl, VT, Op.getOperand(0), Op.getOperand(1));
+    case ISD::USUBSAT:
+      return DAG.getNode(ISD::AND, dl, VT, Op.getOperand(0),
+                         DAG.getNOT(dl, Op.getOperand(1), VT));
+    }
+  }
+
+  assert(Op.getSimpleValueType().is256BitVector() &&
+         Op.getSimpleValueType().isInteger() &&
+         "Only handle AVX 256-bit vector integer operation");
+  return split256IntArith(Op, DAG);
+}
+
 static SDValue LowerABS(SDValue Op, SelectionDAG &DAG) {
   MVT VT = Op.getSimpleValueType();
   if (VT == MVT::i16 || VT == MVT::i32 || VT == MVT::i64) {
     // Since X86 does not have CMOV for 8-bit integer, we don't convert
     // 8-bit integer abs to NEG and CMOV.
     SDLoc DL(Op);
     SDValue N0 = Op.getOperand(0);
     SDValue Neg = DAG.getNode(X86ISD::SUB, DL, DAG.getVTList(VT, MVT::i32),
SDValue X86TargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::SMULO:
   case ISD::UMULO:              return LowerXALUO(Op, DAG);
   case ISD::READCYCLECOUNTER:   return LowerREADCYCLECOUNTER(Op, Subtarget,DAG);
   case ISD::BITCAST:            return LowerBITCAST(Op, Subtarget, DAG);
   case ISD::ADDCARRY:
   case ISD::SUBCARRY:           return LowerADDSUBCARRY(Op, DAG);
   case ISD::ADD:
   case ISD::SUB:                return LowerADD_SUB(Op, DAG);
+  case ISD::UADDSAT:
+  case ISD::USUBSAT:            return LowerUADDSAT_USUBSAT(Op, DAG);
   case ISD::SMAX:
   case ISD::SMIN:
   case ISD::UMAX:
   case ISD::UMIN:               return LowerMINMAX(Op, DAG);
   case ISD::ABS:                return LowerABS(Op, DAG);
   case ISD::FSINCOS:            return LowerFSINCOS(Op, Subtarget, DAG);
   case ISD::MLOAD:              return LowerMLOAD(Op, Subtarget, DAG);
   case ISD::MSTORE:             return LowerMSTORE(Op, Subtarget, DAG);
if (getTypeAction(*DAG.getContext(), VT) == TypePromoteInteger &&
       unsigned NumConcats = 16 / VT.getVectorNumElements();
       SmallVector<SDValue, 8> ConcatOps(NumConcats, DAG.getUNDEF(VT));
       ConcatOps[0] = Res;
       Res = DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v16i8, ConcatOps);
       Results.push_back(Res);
     }
     return;
   }
+  case ISD::UADDSAT:
+  case ISD::USUBSAT:
   case X86ISD::VPMADDWD:
-  case X86ISD::ADDUS:
-  case X86ISD::SUBUS:
   case X86ISD::AVG: {
-    // Legalize types for X86ISD::AVG/ADDUS/SUBUS/VPMADDWD by widening.
+    // Legalize types for ISD::UADDSAT/USUBSAT and X86ISD::AVG/VPMADDWD
+    // by widening.
     assert(Subtarget.hasSSE2() && "Requires at least SSE2!");
 
     EVT VT = N->getValueType(0);
     EVT InVT = N->getOperand(0).getValueType();
     assert(VT.getSizeInBits() < 128 && 128 % VT.getSizeInBits() == 0 &&
            "Expected a VT that divides into 128 bits.");
     unsigned NumConcat = 128 / InVT.getSizeInBits();
 
const char *X86TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case X86ISD::PEXTRW:             return "X86ISD::PEXTRW";
   case X86ISD::INSERTPS:           return "X86ISD::INSERTPS";
   case X86ISD::PINSRB:             return "X86ISD::PINSRB";
   case X86ISD::PINSRW:             return "X86ISD::PINSRW";
   case X86ISD::PSHUFB:             return "X86ISD::PSHUFB";
   case X86ISD::ANDNP:              return "X86ISD::ANDNP";
   case X86ISD::BLENDI:             return "X86ISD::BLENDI";
   case X86ISD::SHRUNKBLEND:        return "X86ISD::SHRUNKBLEND";
-  case X86ISD::ADDUS:              return "X86ISD::ADDUS";
-  case X86ISD::SUBUS:              return "X86ISD::SUBUS";
   case X86ISD::HADD:               return "X86ISD::HADD";
   case X86ISD::HSUB:               return "X86ISD::HSUB";
   case X86ISD::FHADD:              return "X86ISD::FHADD";
   case X86ISD::FHSUB:              return "X86ISD::FHSUB";
   case X86ISD::CONFLICT:           return "X86ISD::CONFLICT";
   case X86ISD::FMAX:               return "X86ISD::FMAX";
   case X86ISD::FMAXS:              return "X86ISD::FMAXS";
   case X86ISD::FMAX_RND:           return "X86ISD::FMAX_RND";
if (ISD::isBuildVectorAllZeros(LHS.getNode())) {
       Other = LHS;
     }
 
     if (Other.getNode() && Other->getNumOperands() == 2 &&
         Other->getOperand(0) == Cond.getOperand(0)) {
       SDValue OpLHS = Other->getOperand(0), OpRHS = Other->getOperand(1);
       SDValue CondRHS = Cond->getOperand(1);
 
-      auto SUBUSBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
+      auto USUBSATBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
                                ArrayRef<SDValue> Ops) {
-        return DAG.getNode(X86ISD::SUBUS, DL, Ops[0].getValueType(), Ops);
+        return DAG.getNode(ISD::USUBSAT, DL, Ops[0].getValueType(), Ops);
       };
 
       // Look for a general sub with unsigned saturation first.
       // x >= y ? x-y : 0 --> subus x, y
       // x >  y ? x-y : 0 --> subus x, y
       if ((CC == ISD::SETUGE || CC == ISD::SETUGT) &&
           Other->getOpcode() == ISD::SUB && OpRHS == CondRHS)
         return SplitOpsAndApply(DAG, Subtarget, DL, VT, { OpLHS, OpRHS },
-                                SUBUSBuilder);
+                                USUBSATBuilder);
 
       if (auto *OpRHSBV = dyn_cast<BuildVectorSDNode>(OpRHS)) {
         if (isa<BuildVectorSDNode>(CondRHS)) {
           // If the RHS is a constant we have to reverse the const
           // canonicalization.
           // x > C-1 ? x+-C : 0 --> subus x, C
-          auto MatchSUBUS = [](ConstantSDNode *Op, ConstantSDNode *Cond) {
+          auto MatchUSUBSAT = [](ConstantSDNode *Op, ConstantSDNode *Cond) {
             return Cond->getAPIntValue() == (-Op->getAPIntValue() - 1);
           };
           if (CC == ISD::SETUGT && Other->getOpcode() == ISD::ADD &&
-              ISD::matchBinaryPredicate(OpRHS, CondRHS, MatchSUBUS)) {
+              ISD::matchBinaryPredicate(OpRHS, CondRHS, MatchUSUBSAT)) {
             OpRHS = DAG.getNode(ISD::SUB, DL, VT,
                                 DAG.getConstant(0, DL, VT), OpRHS);
             return SplitOpsAndApply(DAG, Subtarget, DL, VT, { OpLHS, OpRHS },
-                                    SUBUSBuilder);
+                                    USUBSATBuilder);
           }
 
           // Another special case: If C was a sign bit, the sub has been
           // canonicalized into a xor.
           // FIXME: Would it be better to use computeKnownBits to determine
           //        whether it's safe to decanonicalize the xor?
           // x s< 0 ? x^C : 0 --> subus x, C
           if (auto *OpRHSConst = OpRHSBV->getConstantSplatNode()) {
             if (CC == ISD::SETLT && Other.getOpcode() == ISD::XOR &&
                 ISD::isBuildVectorAllZeros(CondRHS.getNode()) &&
                 OpRHSConst->getAPIntValue().isSignMask()) {
               OpRHS = DAG.getConstant(OpRHSConst->getAPIntValue(), DL, VT);
               // Note that we have to rebuild the RHS constant here to ensure we
               // don't rely on particular values of undef lanes.
               return SplitOpsAndApply(DAG, Subtarget, DL, VT, { OpLHS, OpRHS },
-                                      SUBUSBuilder);
+                                      USUBSATBuilder);
             }
           }
         }
       }
     }
   }
 
   // Match VSELECTs into add with unsigned saturation.
if (ISD::isBuildVectorAllOnes(LHS.getNode())) {
       CC = ISD::getSetCCInverse(CC, true);
     } else if (ISD::isBuildVectorAllOnes(RHS.getNode())) {
       Other = LHS;
     }
 
     if (Other.getNode() && Other.getOpcode() == ISD::ADD) {
       SDValue OpLHS = Other.getOperand(0), OpRHS = Other.getOperand(1);
 
-      auto ADDUSBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
+      auto UADDSATBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
                                ArrayRef<SDValue> Ops) {
-        return DAG.getNode(X86ISD::ADDUS, DL, Ops[0].getValueType(), Ops);
+        return DAG.getNode(ISD::UADDSAT, DL, Ops[0].getValueType(), Ops);
       };
 
       // Canonicalize condition operands.
       if (CC == ISD::SETUGE) {
         std::swap(CondLHS, CondRHS);
         CC = ISD::SETULE;
       }
 
       // We can test against either of the addition operands.
       // x <= x+y ? x+y : ~0 --> addus x, y
       // x+y >= x ? x+y : ~0 --> addus x, y
       if (CC == ISD::SETULE && Other == CondRHS &&
           (OpLHS == CondLHS || OpRHS == CondLHS))
         return SplitOpsAndApply(DAG, Subtarget, DL, VT, { OpLHS, OpRHS },
-                                ADDUSBuilder);
+                                UADDSATBuilder);
 
       if (isa<BuildVectorSDNode>(OpRHS) && isa<BuildVectorSDNode>(CondRHS) &&
           CondLHS == OpLHS) {
         // If the RHS is a constant we have to reverse the const
         // canonicalization.
         // x > ~C ? x+C : ~0 --> addus x, C
-        auto MatchADDUS = [](ConstantSDNode *Op, ConstantSDNode *Cond) {
+        auto MatchUADDSAT = [](ConstantSDNode *Op, ConstantSDNode *Cond) {
           return Cond->getAPIntValue() == ~Op->getAPIntValue();
         };
         if (CC == ISD::SETULE &&
-            ISD::matchBinaryPredicate(OpRHS, CondRHS, MatchADDUS))
+            ISD::matchBinaryPredicate(OpRHS, CondRHS, MatchUADDSAT))
           return SplitOpsAndApply(DAG, Subtarget, DL, VT, { OpLHS, OpRHS },
-                                  ADDUSBuilder);
+                                  UADDSATBuilder);
       }
     }
   }
 
   // Early exit check
   if (!TLI.isTypeLegal(VT))
     return SDValue();
 
if (MinLHS == Op0)
       SubusRHS = MinRHS;
     else if (MinRHS == Op0)
       SubusRHS = MinLHS;
     else
       return SDValue();
   } else
     return SDValue();
 
-  auto SUBUSBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
+  auto USUBSATBuilder = [](SelectionDAG &DAG, const SDLoc &DL,
                            ArrayRef<SDValue> Ops) {
-    return DAG.getNode(X86ISD::SUBUS, DL, Ops[0].getValueType(), Ops);
+    return DAG.getNode(ISD::USUBSAT, DL, Ops[0].getValueType(), Ops);
   };
 
   // PSUBUS doesn't support v8i32/v8i64/v16i32, but it can be enabled with
   // special preprocessing in some cases.
   if (VT != MVT::v8i32 && VT != MVT::v16i32 && VT != MVT::v8i64)
     return SplitOpsAndApply(DAG, Subtarget, SDLoc(N), VT,
-                            { SubusLHS, SubusRHS }, SUBUSBuilder);
+                            { SubusLHS, SubusRHS }, USUBSATBuilder);
 
   // Special preprocessing case can be only applied
   // if the value was zero extended from 16 bit,
   // so we require first 16 bits to be zeros for 32 bit
   // values, or first 48 bits for 64 bit values.
   KnownBits Known;
   DAG.computeKnownBits(SubusLHS, Known);
   unsigned NumZeros = Known.countMinLeadingZeros();
SDValue SaturationConst =
                       SDLoc(SubusLHS), ExtType);
   SDValue UMin = DAG.getNode(ISD::UMIN, SDLoc(SubusLHS), ExtType, SubusRHS,
                              SaturationConst);
   SDValue NewSubusLHS =
       DAG.getZExtOrTrunc(SubusLHS, SDLoc(SubusLHS), ShrinkedType);
   SDValue NewSubusRHS = DAG.getZExtOrTrunc(UMin, SDLoc(SubusRHS), ShrinkedType);
   SDValue Psubus =
       SplitOpsAndApply(DAG, Subtarget, SDLoc(N), ShrinkedType,
-                       { NewSubusLHS, NewSubusRHS }, SUBUSBuilder);
+                       { NewSubusLHS, NewSubusRHS }, USUBSATBuilder);
   // Zero extend the result, it may be used somewhere as 32 bit,
   // if not zext and following trunc will shrink.
   return DAG.getZExtOrTrunc(Psubus, SDLoc(N), ExtType);
 }
 
 static SDValue combineSub(SDNode *N, SelectionDAG &DAG,
                           const X86Subtarget &Subtarget) {
   SDValue Op0 = N->getOperand(0);

lib/Target/X86/X86InstrAVX512.td

 defm VPADD : avx512_binop_rm_vl_all<0xFC, 0xFD, 0xFE, 0xD4, "vpadd", add,
                                     SchedWriteVecALU, 1>;
 defm VPSUB : avx512_binop_rm_vl_all<0xF8, 0xF9, 0xFA, 0xFB, "vpsub", sub,
                                     SchedWriteVecALU, 0>;
 defm VPADDS : avx512_binop_rm_vl_bw<0xEC, 0xED, "vpadds", X86adds,
                                     SchedWriteVecALU, HasBWI, 1>;
 defm VPSUBS : avx512_binop_rm_vl_bw<0xE8, 0xE9, "vpsubs", X86subs,
                                     SchedWriteVecALU, HasBWI, 0>;
-defm VPADDUS : avx512_binop_rm_vl_bw<0xDC, 0xDD, "vpaddus", X86addus,
+defm VPADDUS : avx512_binop_rm_vl_bw<0xDC, 0xDD, "vpaddus", uaddsat,
                                      SchedWriteVecALU, HasBWI, 1>;
-defm VPSUBUS : avx512_binop_rm_vl_bw<0xD8, 0xD9, "vpsubus", X86subus,
+defm VPSUBUS : avx512_binop_rm_vl_bw<0xD8, 0xD9, "vpsubus", usubsat,
                                      SchedWriteVecALU, HasBWI, 0>;
 defm VPMULLD : avx512_binop_rm_vl_d<0x40, "vpmulld", mul,
                                     SchedWritePMULLD, HasAVX512, 1>, T8PD;
 defm VPMULLW : avx512_binop_rm_vl_w<0xD5, "vpmullw", mul,
                                     SchedWriteVecIMul, HasBWI, 1>;
 defm VPMULLQ : avx512_binop_rm_vl_q<0x40, "vpmullq", mul,
                                     SchedWriteVecIMul, HasDQI, 1>, T8PD,
                                     NotEVEX2VEXConvertible;

lib/Target/X86/X86InstrFragmentsSIMD.td

 def X86vpperm : SDNode<"X86ISD::VPPERM",
                         SDTypeProfile<1, 3, [SDTCisVT<0, v16i8>, SDTCisSameAs<0,1>,
                                              SDTCisSameAs<0,2>, SDTCisSameAs<0, 3>]>>;
 
 def SDTX86CmpPTest : SDTypeProfile<1, 2, [SDTCisVT<0, i32>,
                                           SDTCisVec<1>,
                                           SDTCisSameAs<2, 1>]>;
 
-def X86addus   : SDNode<"X86ISD::ADDUS", SDTIntBinOp, [SDNPCommutative]>;
-def X86subus   : SDNode<"X86ISD::SUBUS", SDTIntBinOp>;
 def X86adds    : SDNode<"X86ISD::ADDS", SDTIntBinOp, [SDNPCommutative]>;
 def X86subs    : SDNode<"X86ISD::SUBS", SDTIntBinOp>;
 def X86mulhrs  : SDNode<"X86ISD::MULHRS", SDTIntBinOp, [SDNPCommutative]>;
 def X86avg     : SDNode<"X86ISD::AVG" , SDTIntBinOp, [SDNPCommutative]>;
 def X86ptest   : SDNode<"X86ISD::PTEST", SDTX86CmpPTest>;
 def X86testp   : SDNode<"X86ISD::TESTP", SDTX86CmpPTest>;
 def X86kortest : SDNode<"X86ISD::KORTEST", SDTX86CmpPTest>;
 def X86ktest   : SDNode<"X86ISD::KTEST", SDTX86CmpPTest>;

lib/Target/X86/X86InstrSSE.td

 defm PADDD   : PDI_binop_all<0xFE, "paddd", add, v4i32, v8i32,
                              SchedWriteVecALU, 1, NoVLX>;
 defm PADDQ   : PDI_binop_all<0xD4, "paddq", add, v2i64, v4i64,
                              SchedWriteVecALU, 1, NoVLX>;
 defm PADDSB  : PDI_binop_all<0xEC, "paddsb", X86adds, v16i8, v32i8,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
 defm PADDSW  : PDI_binop_all<0xED, "paddsw", X86adds, v8i16, v16i16,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
-defm PADDUSB : PDI_binop_all<0xDC, "paddusb", X86addus, v16i8, v32i8,
+defm PADDUSB : PDI_binop_all<0xDC, "paddusb", uaddsat, v16i8, v32i8,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
-defm PADDUSW : PDI_binop_all<0xDD, "paddusw", X86addus, v8i16, v16i16,
+defm PADDUSW : PDI_binop_all<0xDD, "paddusw", uaddsat, v8i16, v16i16,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
 defm PMULLW  : PDI_binop_all<0xD5, "pmullw", mul, v8i16, v16i16,
                              SchedWriteVecIMul, 1, NoVLX_Or_NoBWI>;
 defm PMULHUW : PDI_binop_all<0xE4, "pmulhuw", mulhu, v8i16, v16i16,
                              SchedWriteVecIMul, 1, NoVLX_Or_NoBWI>;
 defm PMULHW  : PDI_binop_all<0xE5, "pmulhw", mulhs, v8i16, v16i16,
                              SchedWriteVecIMul, 1, NoVLX_Or_NoBWI>;
 defm PSUBB   : PDI_binop_all<0xF8, "psubb", sub, v16i8, v32i8,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
 defm PSUBW   : PDI_binop_all<0xF9, "psubw", sub, v8i16, v16i16,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
 defm PSUBD   : PDI_binop_all<0xFA, "psubd", sub, v4i32, v8i32,
                              SchedWriteVecALU, 0, NoVLX>;
 defm PSUBQ   : PDI_binop_all<0xFB, "psubq", sub, v2i64, v4i64,
                              SchedWriteVecALU, 0, NoVLX>;
 defm PSUBSB  : PDI_binop_all<0xE8, "psubsb", X86subs, v16i8, v32i8,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
 defm PSUBSW  : PDI_binop_all<0xE9, "psubsw", X86subs, v8i16, v16i16,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
-defm PSUBUSB : PDI_binop_all<0xD8, "psubusb", X86subus, v16i8, v32i8,
+defm PSUBUSB : PDI_binop_all<0xD8, "psubusb", usubsat, v16i8, v32i8,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
-defm PSUBUSW : PDI_binop_all<0xD9, "psubusw", X86subus, v8i16, v16i16,
+defm PSUBUSW : PDI_binop_all<0xD9, "psubusw", usubsat, v8i16, v16i16,
                              SchedWriteVecALU, 0, NoVLX_Or_NoBWI>;
 defm PMINUB  : PDI_binop_all<0xDA, "pminub", umin, v16i8, v32i8,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
 defm PMINSW  : PDI_binop_all<0xEA, "pminsw", smin, v8i16, v16i16,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
 defm PMAXUB  : PDI_binop_all<0xDE, "pmaxub", umax, v16i8, v32i8,
                              SchedWriteVecALU, 1, NoVLX_Or_NoBWI>;
 defm PMAXSW  : PDI_binop_all<0xEE, "pmaxsw", smax, v8i16, v16i16,

test/CodeGen/X86/uadd_sat_vec.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefixes=SSE,SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+ssse3 | FileCheck %s --check-prefixes=SSE,SSSE3
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.1 | FileCheck %s --check-prefixes=SSE,SSE41
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefixes=AVX,AVX1
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw,+avx512vl,+fast-variable-shuffle | FileCheck %s --check-prefixes=AVX,AVX512
+
+declare <1 x i8> @llvm.uadd.sat.v1i8(<1 x i8>, <1 x i8>)
+declare <2 x i8> @llvm.uadd.sat.v2i8(<2 x i8>, <2 x i8>)
+declare <4 x i8> @llvm.uadd.sat.v4i8(<4 x i8>, <4 x i8>)
+declare <8 x i8> @llvm.uadd.sat.v8i8(<8 x i8>, <8 x i8>)
+declare <12 x i8> @llvm.uadd.sat.v12i8(<12 x i8>, <12 x i8>)
+declare <16 x i8> @llvm.uadd.sat.v16i8(<16 x i8>, <16 x i8>)
+declare <32 x i8> @llvm.uadd.sat.v32i8(<32 x i8>, <32 x i8>)
+declare <64 x i8> @llvm.uadd.sat.v64i8(<64 x i8>, <64 x i8>)
+
+declare <1 x i16> @llvm.uadd.sat.v1i16(<1 x i16>, <1 x i16>)
+declare <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16>, <2 x i16>)
+declare <4 x i16> @llvm.uadd.sat.v4i16(<4 x i16>, <4 x i16>)
+declare <8 x i16> @llvm.uadd.sat.v8i16(<8 x i16>, <8 x i16>)
+declare <12 x i16> @llvm.uadd.sat.v12i16(<12 x i16>, <12 x i16>)
+declare <16 x i16> @llvm.uadd.sat.v16i16(<16 x i16>, <16 x i16>)
+declare <32 x i16> @llvm.uadd.sat.v32i16(<32 x i16>, <32 x i16>)
+
+declare <16 x i1> @llvm.uadd.sat.v16i1(<16 x i1>, <16 x i1>)
+declare <16 x i4> @llvm.uadd.sat.v16i4(<16 x i4>, <16 x i4>)
+
+declare <4 x i32> @llvm.uadd.sat.v4i32(<4 x i32>, <4 x i32>)
+declare <2 x i32> @llvm.uadd.sat.v2i32(<2 x i32>, <2 x i32>)
+declare <4 x i24> @llvm.uadd.sat.v4i24(<4 x i24>, <4 x i24>)
+declare <2 x i128> @llvm.uadd.sat.v2i128(<2 x i128>, <2 x i128>)
+
+; Legal types, depending on architecture.
+
+define <16 x i8> @v16i8(<16 x i8> %x, <16 x i8> %y) {
+; SSE-LABEL: v16i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusb %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v16i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <16 x i8> @llvm.uadd.sat.v16i8(<16 x i8> %x, <16 x i8> %y)
+  ret <16 x i8> %z
+}
+
+define <32 x i8> @v32i8(<32 x i8> %x, <32 x i8> %y) {
+; SSE-LABEL: v32i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusb %xmm2, %xmm0
+; SSE-NEXT:    paddusb %xmm3, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v32i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpaddusb %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v32i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpaddusb %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v32i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpaddusb %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    retq
+  %z = call <32 x i8> @llvm.uadd.sat.v32i8(<32 x i8> %x, <32 x i8> %y)
+  ret <32 x i8> %z
+}
+
+define <64 x i8> @v64i8(<64 x i8> %x, <64 x i8> %y) {
+; SSE-LABEL: v64i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusb %xmm4, %xmm0
+; SSE-NEXT:    paddusb %xmm5, %xmm1
+; SSE-NEXT:    paddusb %xmm6, %xmm2
+; SSE-NEXT:    paddusb %xmm7, %xmm3
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v64i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
+; AVX1-NEXT:    vpaddusb %xmm4, %xmm5, %xmm4
+; AVX1-NEXT:    vpaddusb %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm3, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
+; AVX1-NEXT:    vpaddusb %xmm2, %xmm4, %xmm2
+; AVX1-NEXT:    vpaddusb %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v64i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpaddusb %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vpaddusb %ymm3, %ymm1, %ymm1
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v64i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpaddusb %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    retq
+  %z = call <64 x i8> @llvm.uadd.sat.v64i8(<64 x i8> %x, <64 x i8> %y)
+  ret <64 x i8> %z
+}
+
+define <8 x i16> @v8i16(<8 x i16> %x, <8 x i16> %y) {
+; SSE-LABEL: v8i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusw %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v8i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <8 x i16> @llvm.uadd.sat.v8i16(<8 x i16> %x, <8 x i16> %y)
+  ret <8 x i16> %z
+}
+
+define <16 x i16> @v16i16(<16 x i16> %x, <16 x i16> %y) {
+; SSE-LABEL: v16i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusw %xmm2, %xmm0
+; SSE-NEXT:    paddusw %xmm3, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v16i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpaddusw %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v16i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpaddusw %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v16i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpaddusw %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    retq
+  %z = call <16 x i16> @llvm.uadd.sat.v16i16(<16 x i16> %x, <16 x i16> %y)
+  ret <16 x i16> %z
+}
+
+define <32 x i16> @v32i16(<32 x i16> %x, <32 x i16> %y) {
+; SSE-LABEL: v32i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusw %xmm4, %xmm0
+; SSE-NEXT:    paddusw %xmm5, %xmm1
+; SSE-NEXT:    paddusw %xmm6, %xmm2
+; SSE-NEXT:    paddusw %xmm7, %xmm3
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v32i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
+; AVX1-NEXT:    vpaddusw %xmm4, %xmm5, %xmm4
+; AVX1-NEXT:    vpaddusw %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm3, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
+; AVX1-NEXT:    vpaddusw %xmm2, %xmm4, %xmm2
+; AVX1-NEXT:    vpaddusw %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v32i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpaddusw %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vpaddusw %ymm3, %ymm1, %ymm1
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v32i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpaddusw %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    retq
+  %z = call <32 x i16> @llvm.uadd.sat.v32i16(<32 x i16> %x, <32 x i16> %y)
+  ret <32 x i16> %z
+}
+
+; Too narrow vectors, legalized by widening.
+
+define void @v8i8(<8 x i8>* %px, <8 x i8>* %py, <8 x i8>* %pz) {
+; SSE-LABEL: v8i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero
+; SSE-NEXT:    paddusb %xmm0, %xmm1
+; SSE-NEXT:    movq %xmm1, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v8i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX1-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX1-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovq %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v8i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX2-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovq %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v8i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX512-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX512-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbw {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
+; AVX512-NEXT:    vpmovwb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <8 x i8>, <8 x i8>* %px
+  %y = load <8 x i8>, <8 x i8>* %py
+  %z = call <8 x i8> @llvm.uadd.sat.v8i8(<8 x i8> %x, <8 x i8> %y)
+  store <8 x i8> %z, <8 x i8>* %pz
+  ret void
+}
+
+define void @v4i8(<4 x i8>* %px, <4 x i8>* %py, <4 x i8>* %pz) {
+; SSE-LABEL: v4i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    paddusb %xmm0, %xmm1
+; SSE-NEXT:    movd %xmm1, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v4i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v4i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovd %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v4i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbd {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero
+; AVX512-NEXT:    vpmovdb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <4 x i8>, <4 x i8>* %px
+  %y = load <4 x i8>, <4 x i8>* %py
+  %z = call <4 x i8> @llvm.uadd.sat.v4i8(<4 x i8> %x, <4 x i8> %y)
+  store <4 x i8> %z, <4 x i8>* %pz
+  ret void
+}
+
+define void @v2i8(<2 x i8>* %px, <2 x i8>* %py, <2 x i8>* %pz) {
+; SSE2-LABEL: v2i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzwl (%rdi), %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    movzwl (%rsi), %eax
+; SSE2-NEXT:    movd %eax, %xmm1
+; SSE2-NEXT:    paddusb %xmm0, %xmm1
+; SSE2-NEXT:    movd %xmm1, %eax
+; SSE2-NEXT:    movw %ax, (%rdx)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v2i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzwl (%rdi), %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    movzwl (%rsi), %eax
+; SSSE3-NEXT:    movd %eax, %xmm1
+; SSSE3-NEXT:    paddusb %xmm0, %xmm1
+; SSSE3-NEXT:    movd %xmm1, %eax
+; SSSE3-NEXT:    movw %ax, (%rdx)
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v2i8:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movzwl (%rdi), %eax
+; SSE41-NEXT:    movd %eax, %xmm0
+; SSE41-NEXT:    movzwl (%rsi), %eax
+; SSE41-NEXT:    movd %eax, %xmm1
+; SSE41-NEXT:    paddusb %xmm0, %xmm1
+; SSE41-NEXT:    pextrw $0, %xmm1, (%rdx)
+; SSE41-NEXT:    retq
+;
+; AVX1-LABEL: v2i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzwl (%rdi), %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    movzwl (%rsi), %eax
+; AVX1-NEXT:    vmovd %eax, %xmm1
+; AVX1-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpextrw $0, %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v2i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzwl (%rdi), %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    movzwl (%rsi), %eax
+; AVX2-NEXT:    vmovd %eax, %xmm1
+; AVX2-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpextrw $0, %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v2i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movzwl (%rdi), %eax
+; AVX512-NEXT:    vmovd %eax, %xmm0
+; AVX512-NEXT:    movzwl (%rsi), %eax
+; AVX512-NEXT:    vmovd %eax, %xmm1
+; AVX512-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,zero,zero,zero,zero,xmm0[1],zero,zero,zero,zero,zero,zero,zero
+; AVX512-NEXT:    vpmovqb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <2 x i8>, <2 x i8>* %px
+  %y = load <2 x i8>, <2 x i8>* %py
+  %z = call <2 x i8> @llvm.uadd.sat.v2i8(<2 x i8> %x, <2 x i8> %y)
+  store <2 x i8> %z, <2 x i8>* %pz
+  ret void
+}
+
+define void @v4i16(<4 x i16>* %px, <4 x i16>* %py, <4 x i16>* %pz) {
+; SSE-LABEL: v4i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero
+; SSE-NEXT:    paddusw %xmm0, %xmm1
+; SSE-NEXT:    movq %xmm1, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v4i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX1-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX1-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovq %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v4i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX2-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovq %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v4i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX512-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX512-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxwd {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
+; AVX512-NEXT:    vpmovdw %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <4 x i16>, <4 x i16>* %px
+  %y = load <4 x i16>, <4 x i16>* %py
+  %z = call <4 x i16> @llvm.uadd.sat.v4i16(<4 x i16> %x, <4 x i16> %y)
+  store <4 x i16> %z, <4 x i16>* %pz
+  ret void
+}
+
+define void @v2i16(<2 x i16>* %px, <2 x i16>* %py, <2 x i16>* %pz) {
+; SSE-LABEL: v2i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    paddusw %xmm0, %xmm1
+; SSE-NEXT:    movd %xmm1, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v2i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v2i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovd %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v2i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vpaddusw %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; AVX512-NEXT:    vpmovqw %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <2 x i16>, <2 x i16>* %px
+  %y = load <2 x i16>, <2 x i16>* %py
+  %z = call <2 x i16> @llvm.uadd.sat.v2i16(<2 x i16> %x, <2 x i16> %y)
+  store <2 x i16> %z, <2 x i16>* %pz
+  ret void
+}
+
+define <12 x i8> @v12i8(<12 x i8> %x, <12 x i8> %y) {
+; SSE-LABEL: v12i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    paddusb %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v12i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <12 x i8> @llvm.uadd.sat.v12i8(<12 x i8> %x, <12 x i8> %y)
+  ret <12 x i8> %z
+}
+
+define void @v12i16(<12 x i16>* %px, <12 x i16>* %py, <12 x i16>* %pz) {
+; SSE-LABEL: v12i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa (%rdi), %xmm0
+; SSE-NEXT:    movdqa 16(%rdi), %xmm1
+; SSE-NEXT:    paddusw (%rsi), %xmm0
+; SSE-NEXT:    paddusw 16(%rsi), %xmm1
+; SSE-NEXT:    movq %xmm1, 16(%rdx)
+; SSE-NEXT:    movdqa %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v12i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa 16(%rdi), %xmm1
+; AVX1-NEXT:    vpaddusw (%rsi), %xmm0, %xmm0
+; AVX1-NEXT:    vpaddusw 16(%rsi), %xmm1, %xmm1
+; AVX1-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v12i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovdqa (%rdi), %ymm0
+; AVX2-NEXT:    vpaddusw (%rsi), %ymm0, %ymm0
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX2-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v12i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovdqa (%rdi), %ymm0
+; AVX512-NEXT:    vpaddusw (%rsi), %ymm0, %ymm0
+; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; AVX512-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX512-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  %x = load <12 x i16>, <12 x i16>* %px
+  %y = load <12 x i16>, <12 x i16>* %py
+  %z = call <12 x i16> @llvm.uadd.sat.v12i16(<12 x i16> %x, <12 x i16> %y)
+  store <12 x i16> %z, <12 x i16>* %pz
+  ret void
+}
+
+; Scalarization
+
+define void @v1i8(<1 x i8>* %px, <1 x i8>* %py, <1 x i8>* %pz) {
+; SSE-LABEL: v1i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movb (%rdi), %al
+; SSE-NEXT:    addb (%rsi), %al
+; SSE-NEXT:    movb $-1, %cl
+; SSE-NEXT:    jb .LBB13_2
+; SSE-NEXT:  # %bb.1:
+; SSE-NEXT:    movl %eax, %ecx
+; SSE-NEXT:  .LBB13_2:
+; SSE-NEXT:    movb %cl, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v1i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movb (%rdi), %al
+; AVX-NEXT:    addb (%rsi), %al
+; AVX-NEXT:    movb $-1, %cl
+; AVX-NEXT:    jb .LBB13_2
+; AVX-NEXT:  # %bb.1:
+; AVX-NEXT:    movl %eax, %ecx
+; AVX-NEXT:  .LBB13_2:
+; AVX-NEXT:    movb %cl, (%rdx)
+; AVX-NEXT:    retq
+  %x = load <1 x i8>, <1 x i8>* %px
+  %y = load <1 x i8>, <1 x i8>* %py
+  %z = call <1 x i8> @llvm.uadd.sat.v1i8(<1 x i8> %x, <1 x i8> %y)
+  store <1 x i8> %z, <1 x i8>* %pz
+  ret void
+}
+
+define void @v1i16(<1 x i16>* %px, <1 x i16>* %py, <1 x i16>* %pz) {
+; SSE-LABEL: v1i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movzwl (%rdi), %eax
+; SSE-NEXT:    addw (%rsi), %ax
+; SSE-NEXT:    movl $65535, %ecx # imm = 0xFFFF
+; SSE-NEXT:    cmovael %eax, %ecx
+; SSE-NEXT:    movw %cx, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v1i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movzwl (%rdi), %eax
+; AVX-NEXT:    addw (%rsi), %ax
+; AVX-NEXT:    movl $65535, %ecx # imm = 0xFFFF
+; AVX-NEXT:    cmovael %eax, %ecx
+; AVX-NEXT:    movw %cx, (%rdx)
+; AVX-NEXT:    retq
+  %x = load <1 x i16>, <1 x i16>* %px
+  %y = load <1 x i16>, <1 x i16>* %py
+  %z = call <1 x i16> @llvm.uadd.sat.v1i16(<1 x i16> %x, <1 x i16> %y)
+  store <1 x i16> %z, <1 x i16>* %pz
+  ret void
+}
+
+; Promotion
+
+define <16 x i4> @v16i4(<16 x i4> %x, <16 x i4> %y) {
+; SSE-LABEL: v16i4:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psllw $4, %xmm1
+; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [240,240,240,240,240,240,240,240,240,240,240,240,240,240,240,240]
+; SSE-NEXT:    pand %xmm2, %xmm1
+; SSE-NEXT:    psllw $4, %xmm0
+; SSE-NEXT:    pand %xmm2, %xmm0
+; SSE-NEXT:    paddusb %xmm1, %xmm0
+; SSE-NEXT:    psrlw $4, %xmm0
+; SSE-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v16i4:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsllw $4, %xmm1, %xmm1
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm2 = [240,240,240,240,240,240,240,240,240,240,240,240,240,240,240,240]
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX-NEXT:    vpsllw $4, %xmm0, %xmm0
+; AVX-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsrlw $4, %xmm0, %xmm0
+; AVX-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <16 x i4> @llvm.uadd.sat.v16i4(<16 x i4> %x, <16 x i4> %y)
+  ret <16 x i4> %z
+}
+
+define <16 x i1> @v16i1(<16 x i1> %x, <16 x i1> %y) {
+; SSE-LABEL: v16i1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psllw $7, %xmm1
+; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; SSE-NEXT:    pand %xmm2, %xmm1
+; SSE-NEXT:    psllw $7, %xmm0
+; SSE-NEXT:    pand %xmm2, %xmm0
+; SSE-NEXT:    paddusb %xmm1, %xmm0
+; SSE-NEXT:    psrlw $7, %xmm0
+; SSE-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v16i1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vpsllw $7, %xmm1, %xmm1
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX1-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsrlw $7, %xmm0, %xmm0
+; AVX1-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v16i1:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsllw $7, %xmm1, %xmm1
+; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX2-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX2-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX2-NEXT:    vpaddusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpsrlw $7, %xmm0, %xmm0
+; AVX2-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v16i1:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsllw $7, %xmm1, %xmm1
+; AVX512-NEXT:    vpmovb2m %xmm1, %k0
+; AVX512-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovb2m %xmm0, %k1
+; AVX512-NEXT:    korw %k0, %k1, %k0
+; AVX512-NEXT:    vpmovm2b %k0, %xmm0
+; AVX512-NEXT:    retq
+  %z = call <16 x i1> @llvm.uadd.sat.v16i1(<16 x i1> %x, <16 x i1> %y)
+  ret <16 x i1> %z
+}
+
+; Expanded
+
+define <4 x i32> @v4i32(<4 x i32> %x, <4 x i32> %y) {
+; SSE2-LABEL: v4i32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %ecx
+; SSE2-NEXT:    addl %eax, %ecx
+; SSE2-NEXT:    movl $-1, %eax
+; SSE2-NEXT:    cmovbl %eax, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %ecx
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm3
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSE2-NEXT:    movd %xmm1, %ecx
+; SSE2-NEXT:    movd %xmm0, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSE2-NEXT:    movd %xmm1, %ecx
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE2-NEXT:    movd %xmm0, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm0
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v4i32:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %ecx
+; SSSE3-NEXT:    addl %eax, %ecx
+; SSSE3-NEXT:    movl $-1, %eax
+; SSSE3-NEXT:    cmovbl %eax, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %ecx
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm3
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSSE3-NEXT:    movd %xmm1, %ecx
+; SSSE3-NEXT:    movd %xmm0, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm1, %ecx
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm0, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm0
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v4i32:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pextrd $1, %xmm1, %eax
+; SSE41-NEXT:    pextrd $1, %xmm0, %ecx
+; SSE41-NEXT:    addl %eax, %ecx
+; SSE41-NEXT:    movl $-1, %eax
+; SSE41-NEXT:    cmovbl %eax, %ecx
+; SSE41-NEXT:    movd %xmm1, %edx
+; SSE41-NEXT:    movd %xmm0, %esi
+; SSE41-NEXT:    addl %edx, %esi
+; SSE41-NEXT:    cmovbl %eax, %esi
+; SSE41-NEXT:    movd %esi, %xmm2
+; SSE41-NEXT:    pinsrd $1, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $2, %xmm1, %ecx
+; SSE41-NEXT:    pextrd $2, %xmm0, %edx
+; SSE41-NEXT:    addl %ecx, %edx
+; SSE41-NEXT:    cmovbl %eax, %edx
+; SSE41-NEXT:    pinsrd $2, %edx, %xmm2
+; SSE41-NEXT:    pextrd $3, %xmm1, %ecx
+; SSE41-NEXT:    pextrd $3, %xmm0, %edx
+; SSE41-NEXT:    addl %ecx, %edx
+; SSE41-NEXT:    cmovbl %eax, %edx
+; SSE41-NEXT:    pinsrd $3, %edx, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v4i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpextrd $1, %xmm1, %eax
+; AVX-NEXT:    vpextrd $1, %xmm0, %ecx
+; AVX-NEXT:    addl %eax, %ecx
+; AVX-NEXT:    movl $-1, %eax
+; AVX-NEXT:    cmovbl %eax, %ecx
+; AVX-NEXT:    vmovd %xmm1, %edx
+; AVX-NEXT:    vmovd %xmm0, %esi
+; AVX-NEXT:    addl %edx, %esi
+; AVX-NEXT:    cmovbl %eax, %esi
+; AVX-NEXT:    vmovd %esi, %xmm2
+; AVX-NEXT:    vpinsrd $1, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $2, %xmm1, %ecx
+; AVX-NEXT:    vpextrd $2, %xmm0, %edx
+; AVX-NEXT:    addl %ecx, %edx
+; AVX-NEXT:    cmovbl %eax, %edx
+; AVX-NEXT:    vpinsrd $2, %edx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $3, %xmm1, %ecx
+; AVX-NEXT:    vpextrd $3, %xmm0, %edx
+; AVX-NEXT:    addl %ecx, %edx
+; AVX-NEXT:    cmovbl %eax, %edx
+; AVX-NEXT:    vpinsrd $3, %edx, %xmm2, %xmm0
+; AVX-NEXT:    retq
+  %z = call <4 x i32> @llvm.uadd.sat.v4i32(<4 x i32> %x, <4 x i32> %y)
+  ret <4 x i32> %z
+}
+
+define <2 x i32> @v2i32(<2 x i32> %x, <2 x i32> %y) {
+; SSE2-LABEL: v2i32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    psllq $32, %xmm1
+; SSE2-NEXT:    movq %xmm1, %rax
+; SSE2-NEXT:    psllq $32, %xmm0
+; SSE2-NEXT:    movq %xmm0, %rcx
+; SSE2-NEXT:    addq %rax, %rcx
+; SSE2-NEXT:    movq $-1, %rax
+; SSE2-NEXT:    cmovbq %rax, %rcx
+; SSE2-NEXT:    movq %rcx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movq %xmm1, %rcx
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movq %xmm0, %rdx
+; SSE2-NEXT:    addq %rcx, %rdx
+; SSE2-NEXT:    cmovbq %rax, %rdx
+; SSE2-NEXT:    movq %rdx, %xmm0
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
+; SSE2-NEXT:    psrlq $32, %xmm2
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v2i32:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    psllq $32, %xmm1
+; SSSE3-NEXT:    movq %xmm1, %rax
+; SSSE3-NEXT:    psllq $32, %xmm0
+; SSSE3-NEXT:    movq %xmm0, %rcx
+; SSSE3-NEXT:    addq %rax, %rcx
+; SSSE3-NEXT:    movq $-1, %rax
+; SSSE3-NEXT:    cmovbq %rax, %rcx
+; SSSE3-NEXT:    movq %rcx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movq %xmm1, %rcx
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movq %xmm0, %rdx
+; SSSE3-NEXT:    addq %rcx, %rdx
+; SSSE3-NEXT:    cmovbq %rax, %rdx
+; SSSE3-NEXT:    movq %rdx, %xmm0
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
+; SSSE3-NEXT:    psrlq $32, %xmm2
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v2i32:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    psllq $32, %xmm1
+; SSE41-NEXT:    pextrq $1, %xmm1, %rax
+; SSE41-NEXT:    psllq $32, %xmm0
+; SSE41-NEXT:    pextrq $1, %xmm0, %rcx
+; SSE41-NEXT:    addq %rax, %rcx
+; SSE41-NEXT:    movq $-1, %rax
+; SSE41-NEXT:    cmovbq %rax, %rcx
+; SSE41-NEXT:    movq %rcx, %xmm2
+; SSE41-NEXT:    movq %xmm1, %rcx
+; SSE41-NEXT:    movq %xmm0, %rdx
+; SSE41-NEXT:    addq %rcx, %rdx
+; SSE41-NEXT:    cmovbq %rax, %rdx
+; SSE41-NEXT:    movq %rdx, %xmm0
+; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE41-NEXT:    psrlq $32, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v2i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsllq $32, %xmm1, %xmm1
+; AVX-NEXT:    vpextrq $1, %xmm1, %rax
+; AVX-NEXT:    vpsllq $32, %xmm0, %xmm0
+; AVX-NEXT:    vpextrq $1, %xmm0, %rcx
+; AVX-NEXT:    addq %rax, %rcx
+; AVX-NEXT:    movq $-1, %rax
+; AVX-NEXT:    cmovbq %rax, %rcx
+; AVX-NEXT:    vmovq %rcx, %xmm2
+; AVX-NEXT:    vmovq %xmm1, %rcx
+; AVX-NEXT:    vmovq %xmm0, %rdx
+; AVX-NEXT:    addq %rcx, %rdx
+; AVX-NEXT:    cmovbq %rax, %rdx
+; AVX-NEXT:    vmovq %rdx, %xmm0
+; AVX-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; AVX-NEXT:    vpsrlq $32, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <2 x i32> @llvm.uadd.sat.v2i32(<2 x i32> %x, <2 x i32> %y)
+  ret <2 x i32> %z
+}
+
+define <4 x i24> @v4i24(<4 x i24> %x, <4 x i24> %y) {
+; SSE2-LABEL: v4i24:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pslld $8, %xmm1
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %eax
+; SSE2-NEXT:    pslld $8, %xmm0
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %ecx
+; SSE2-NEXT:    addl %eax, %ecx
+; SSE2-NEXT:    movl $-1, %eax
+; SSE2-NEXT:    cmovbl %eax, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %ecx
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm3
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSE2-NEXT:    movd %xmm1, %ecx
+; SSE2-NEXT:    movd %xmm0, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSE2-NEXT:    movd %xmm1, %ecx
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE2-NEXT:    movd %xmm0, %edx
+; SSE2-NEXT:    addl %ecx, %edx
+; SSE2-NEXT:    cmovbl %eax, %edx
+; SSE2-NEXT:    movd %edx, %xmm0
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSE2-NEXT:    psrld $8, %xmm2
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v4i24:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    pslld $8, %xmm1
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %eax
+; SSSE3-NEXT:    pslld $8, %xmm0
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %ecx
+; SSSE3-NEXT:    addl %eax, %ecx
+; SSSE3-NEXT:    movl $-1, %eax
+; SSSE3-NEXT:    cmovbl %eax, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %ecx
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm3
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSSE3-NEXT:    movd %xmm1, %ecx
+; SSSE3-NEXT:    movd %xmm0, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm1, %ecx
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm0, %edx
+; SSSE3-NEXT:    addl %ecx, %edx
+; SSSE3-NEXT:    cmovbl %eax, %edx
+; SSSE3-NEXT:    movd %edx, %xmm0
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSSE3-NEXT:    psrld $8, %xmm2
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v4i24:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pslld $8, %xmm1
+; SSE41-NEXT:    pextrd $1, %xmm1, %eax
+; SSE41-NEXT:    pslld $8, %xmm0
+; SSE41-NEXT:    pextrd $1, %xmm0, %ecx
+; SSE41-NEXT:    addl %eax, %ecx
+; SSE41-NEXT:    movl $-1, %eax
+; SSE41-NEXT:    cmovbl %eax, %ecx
+; SSE41-NEXT:    movd %xmm1, %edx
+; SSE41-NEXT:    movd %xmm0, %esi
+; SSE41-NEXT:    addl %edx, %esi
+; SSE41-NEXT:    cmovbl %eax, %esi
+; SSE41-NEXT:    movd %esi, %xmm2
+; SSE41-NEXT:    pinsrd $1, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $2, %xmm1, %ecx
+; SSE41-NEXT:    pextrd $2, %xmm0, %edx
+; SSE41-NEXT:    addl %ecx, %edx
+; SSE41-NEXT:    cmovbl %eax, %edx
+; SSE41-NEXT:    pinsrd $2, %edx, %xmm2
+; SSE41-NEXT:    pextrd $3, %xmm1, %ecx
+; SSE41-NEXT:    pextrd $3, %xmm0, %edx
+; SSE41-NEXT:    addl %ecx, %edx
+; SSE41-NEXT:    cmovbl %eax, %edx
+; SSE41-NEXT:    pinsrd $3, %edx, %xmm2
+; SSE41-NEXT:    psrld $8, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v4i24:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpslld $8, %xmm1, %xmm1
+; AVX-NEXT:    vpextrd $1, %xmm1, %eax
+; AVX-NEXT:    vpslld $8, %xmm0, %xmm0
+; AVX-NEXT:    vpextrd $1, %xmm0, %ecx
+; AVX-NEXT:    addl %eax, %ecx
+; AVX-NEXT:    movl $-1, %eax
+; AVX-NEXT:    cmovbl %eax, %ecx
+; AVX-NEXT:    vmovd %xmm1, %edx
+; AVX-NEXT:    vmovd %xmm0, %esi
+; AVX-NEXT:    addl %edx, %esi
+; AVX-NEXT:    cmovbl %eax, %esi
+; AVX-NEXT:    vmovd %esi, %xmm2
+; AVX-NEXT:    vpinsrd $1, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $2, %xmm1, %ecx
+; AVX-NEXT:    vpextrd $2, %xmm0, %edx
+; AVX-NEXT:    addl %ecx, %edx
+; AVX-NEXT:    cmovbl %eax, %edx
+; AVX-NEXT:    vpinsrd $2, %edx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $3, %xmm1, %ecx
+; AVX-NEXT:    vpextrd $3, %xmm0, %edx
+; AVX-NEXT:    addl %ecx, %edx
+; AVX-NEXT:    cmovbl %eax, %edx
+; AVX-NEXT:    vpinsrd $3, %edx, %xmm2, %xmm0
+; AVX-NEXT:    vpsrld $8, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <4 x i24> @llvm.uadd.sat.v4i24(<4 x i24> %x, <4 x i24> %y)
+  ret <4 x i24> %z
+}
+
+define <2 x i128> @v2i128(<2 x i128> %x, <2 x i128> %y) {
+; SSE-LABEL: v2i128:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq %rdi, %rax
+; SSE-NEXT:    addq %r9, %rsi
+; SSE-NEXT:    adcq {{[0-9]+}}(%rsp), %rdx
+; SSE-NEXT:    movq $-1, %rdi
+; SSE-NEXT:    cmovbq %rdi, %rsi
+; SSE-NEXT:    cmovbq %rdi, %rdx
+; SSE-NEXT:    addq {{[0-9]+}}(%rsp), %rcx
+; SSE-NEXT:    adcq {{[0-9]+}}(%rsp), %r8
+; SSE-NEXT:    cmovbq %rdi, %r8
+; SSE-NEXT:    cmovbq %rdi, %rcx
+; SSE-NEXT:    movq %r8, 24(%rax)
+; SSE-NEXT:    movq %rcx, 16(%rax)
+; SSE-NEXT:    movq %rdx, 8(%rax)
+; SSE-NEXT:    movq %rsi, (%rax)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v2i128:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movq %rdi, %rax
+; AVX-NEXT:    addq %r9, %rsi
+; AVX-NEXT:    adcq {{[0-9]+}}(%rsp), %rdx
+; AVX-NEXT:    movq $-1, %rdi
+; AVX-NEXT:    cmovbq %rdi, %rsi
+; AVX-NEXT:    cmovbq %rdi, %rdx
+; AVX-NEXT:    addq {{[0-9]+}}(%rsp), %rcx
+; AVX-NEXT:    adcq {{[0-9]+}}(%rsp), %r8
+; AVX-NEXT:    cmovbq %rdi, %r8
+; AVX-NEXT:    cmovbq %rdi, %rcx
+; AVX-NEXT:    movq %r8, 24(%rax)
+; AVX-NEXT:    movq %rcx, 16(%rax)
+; AVX-NEXT:    movq %rdx, 8(%rax)
+; AVX-NEXT:    movq %rsi, (%rax)
+; AVX-NEXT:    retq
+  %z = call <2 x i128> @llvm.uadd.sat.v2i128(<2 x i128> %x, <2 x i128> %y)
+  ret <2 x i128> %z
+}

test/CodeGen/X86/usub_sat_vec.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse2 | FileCheck %s --check-prefixes=SSE,SSE2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+ssse3 | FileCheck %s --check-prefixes=SSE,SSSE3
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+sse4.1 | FileCheck %s --check-prefixes=SSE,SSE41
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx | FileCheck %s --check-prefixes=AVX,AVX1
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx2 | FileCheck %s --check-prefixes=AVX,AVX2
+; RUN: llc < %s -mtriple=x86_64-unknown-unknown -mattr=+avx512bw,+avx512vl,+fast-variable-shuffle | FileCheck %s --check-prefixes=AVX,AVX512
+
+declare <1 x i8> @llvm.usub.sat.v1i8(<1 x i8>, <1 x i8>)
+declare <2 x i8> @llvm.usub.sat.v2i8(<2 x i8>, <2 x i8>)
+declare <4 x i8> @llvm.usub.sat.v4i8(<4 x i8>, <4 x i8>)
+declare <8 x i8> @llvm.usub.sat.v8i8(<8 x i8>, <8 x i8>)
+declare <12 x i8> @llvm.usub.sat.v12i8(<12 x i8>, <12 x i8>)
+declare <16 x i8> @llvm.usub.sat.v16i8(<16 x i8>, <16 x i8>)
+declare <32 x i8> @llvm.usub.sat.v32i8(<32 x i8>, <32 x i8>)
+declare <64 x i8> @llvm.usub.sat.v64i8(<64 x i8>, <64 x i8>)
+
+declare <1 x i16> @llvm.usub.sat.v1i16(<1 x i16>, <1 x i16>)
+declare <2 x i16> @llvm.usub.sat.v2i16(<2 x i16>, <2 x i16>)
+declare <4 x i16> @llvm.usub.sat.v4i16(<4 x i16>, <4 x i16>)
+declare <8 x i16> @llvm.usub.sat.v8i16(<8 x i16>, <8 x i16>)
+declare <12 x i16> @llvm.usub.sat.v12i16(<12 x i16>, <12 x i16>)
+declare <16 x i16> @llvm.usub.sat.v16i16(<16 x i16>, <16 x i16>)
+declare <32 x i16> @llvm.usub.sat.v32i16(<32 x i16>, <32 x i16>)
+
+declare <16 x i1> @llvm.usub.sat.v16i1(<16 x i1>, <16 x i1>)
+declare <16 x i4> @llvm.usub.sat.v16i4(<16 x i4>, <16 x i4>)
+
+declare <4 x i32> @llvm.usub.sat.v4i32(<4 x i32>, <4 x i32>)
+declare <2 x i32> @llvm.usub.sat.v2i32(<2 x i32>, <2 x i32>)
+declare <4 x i24> @llvm.usub.sat.v4i24(<4 x i24>, <4 x i24>)
+declare <2 x i128> @llvm.usub.sat.v2i128(<2 x i128>, <2 x i128>)
+
+; Legal types, depending on architecture.
+
+define <16 x i8> @v16i8(<16 x i8> %x, <16 x i8> %y) {
+; SSE-LABEL: v16i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v16i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <16 x i8> @llvm.usub.sat.v16i8(<16 x i8> %x, <16 x i8> %y)
+  ret <16 x i8> %z
+}
+
+define <32 x i8> @v32i8(<32 x i8> %x, <32 x i8> %y) {
+; SSE-LABEL: v32i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusb %xmm2, %xmm0
+; SSE-NEXT:    psubusb %xmm3, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v32i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpsubusb %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v32i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsubusb %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v32i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsubusb %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    retq
+  %z = call <32 x i8> @llvm.usub.sat.v32i8(<32 x i8> %x, <32 x i8> %y)
+  ret <32 x i8> %z
+}
+
+define <64 x i8> @v64i8(<64 x i8> %x, <64 x i8> %y) {
+; SSE-LABEL: v64i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusb %xmm4, %xmm0
+; SSE-NEXT:    psubusb %xmm5, %xmm1
+; SSE-NEXT:    psubusb %xmm6, %xmm2
+; SSE-NEXT:    psubusb %xmm7, %xmm3
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v64i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
+; AVX1-NEXT:    vpsubusb %xmm4, %xmm5, %xmm4
+; AVX1-NEXT:    vpsubusb %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm3, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
+; AVX1-NEXT:    vpsubusb %xmm2, %xmm4, %xmm2
+; AVX1-NEXT:    vpsubusb %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v64i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsubusb %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubusb %ymm3, %ymm1, %ymm1
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v64i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsubusb %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    retq
+  %z = call <64 x i8> @llvm.usub.sat.v64i8(<64 x i8> %x, <64 x i8> %y)
+  ret <64 x i8> %z
+}
+
+define <8 x i16> @v8i16(<8 x i16> %x, <8 x i16> %y) {
+; SSE-LABEL: v8i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusw %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v8i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <8 x i16> @llvm.usub.sat.v8i16(<8 x i16> %x, <8 x i16> %y)
+  ret <8 x i16> %z
+}
+
+define <16 x i16> @v16i16(<16 x i16> %x, <16 x i16> %y) {
+; SSE-LABEL: v16i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusw %xmm2, %xmm0
+; SSE-NEXT:    psubusw %xmm3, %xmm1
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v16i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm3
+; AVX1-NEXT:    vpsubusw %xmm2, %xmm3, %xmm2
+; AVX1-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm0, %ymm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v16i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsubusw %ymm1, %ymm0, %ymm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v16i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsubusw %ymm1, %ymm0, %ymm0
+; AVX512-NEXT:    retq
+  %z = call <16 x i16> @llvm.usub.sat.v16i16(<16 x i16> %x, <16 x i16> %y)
+  ret <16 x i16> %z
+}
+
+define <32 x i16> @v32i16(<32 x i16> %x, <32 x i16> %y) {
+; SSE-LABEL: v32i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusw %xmm4, %xmm0
+; SSE-NEXT:    psubusw %xmm5, %xmm1
+; SSE-NEXT:    psubusw %xmm6, %xmm2
+; SSE-NEXT:    psubusw %xmm7, %xmm3
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v32i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vextractf128 $1, %ymm2, %xmm4
+; AVX1-NEXT:    vextractf128 $1, %ymm0, %xmm5
+; AVX1-NEXT:    vpsubusw %xmm4, %xmm5, %xmm4
+; AVX1-NEXT:    vpsubusw %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vinsertf128 $1, %xmm4, %ymm0, %ymm0
+; AVX1-NEXT:    vextractf128 $1, %ymm3, %xmm2
+; AVX1-NEXT:    vextractf128 $1, %ymm1, %xmm4
+; AVX1-NEXT:    vpsubusw %xmm2, %xmm4, %xmm2
+; AVX1-NEXT:    vpsubusw %xmm3, %xmm1, %xmm1
+; AVX1-NEXT:    vinsertf128 $1, %xmm2, %ymm1, %ymm1
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v32i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsubusw %ymm2, %ymm0, %ymm0
+; AVX2-NEXT:    vpsubusw %ymm3, %ymm1, %ymm1
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v32i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsubusw %zmm1, %zmm0, %zmm0
+; AVX512-NEXT:    retq
+  %z = call <32 x i16> @llvm.usub.sat.v32i16(<32 x i16> %x, <32 x i16> %y)
+  ret <32 x i16> %z
+}
+
+; Too narrow vectors, legalized by widening.
+
+define void @v8i8(<8 x i8>* %px, <8 x i8>* %py, <8 x i8>* %pz) {
+; SSE-LABEL: v8i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    movq %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v8i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX1-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX1-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovq %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v8i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX2-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovq %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v8i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX512-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX512-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbw {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero,xmm0[4],zero,xmm0[5],zero,xmm0[6],zero,xmm0[7],zero
+; AVX512-NEXT:    vpmovwb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <8 x i8>, <8 x i8>* %px
+  %y = load <8 x i8>, <8 x i8>* %py
+  %z = call <8 x i8> @llvm.usub.sat.v8i8(<8 x i8> %x, <8 x i8> %y)
+  store <8 x i8> %z, <8 x i8>* %pz
+  ret void
+}
+
+define void @v4i8(<4 x i8>* %px, <4 x i8>* %py, <4 x i8>* %pz) {
+; SSE-LABEL: v4i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    movd %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v4i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v4i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovd %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v4i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbd {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero,xmm0[2],zero,zero,zero,xmm0[3],zero,zero,zero
+; AVX512-NEXT:    vpmovdb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <4 x i8>, <4 x i8>* %px
+  %y = load <4 x i8>, <4 x i8>* %py
+  %z = call <4 x i8> @llvm.usub.sat.v4i8(<4 x i8> %x, <4 x i8> %y)
+  store <4 x i8> %z, <4 x i8>* %pz
+  ret void
+}
+
+define void @v2i8(<2 x i8>* %px, <2 x i8>* %py, <2 x i8>* %pz) {
+; SSE2-LABEL: v2i8:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    movzwl (%rdi), %eax
+; SSE2-NEXT:    movd %eax, %xmm0
+; SSE2-NEXT:    movzwl (%rsi), %eax
+; SSE2-NEXT:    movd %eax, %xmm1
+; SSE2-NEXT:    psubusb %xmm1, %xmm0
+; SSE2-NEXT:    movd %xmm0, %eax
+; SSE2-NEXT:    movw %ax, (%rdx)
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v2i8:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    movzwl (%rdi), %eax
+; SSSE3-NEXT:    movd %eax, %xmm0
+; SSSE3-NEXT:    movzwl (%rsi), %eax
+; SSSE3-NEXT:    movd %eax, %xmm1
+; SSSE3-NEXT:    psubusb %xmm1, %xmm0
+; SSSE3-NEXT:    movd %xmm0, %eax
+; SSSE3-NEXT:    movw %ax, (%rdx)
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v2i8:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    movzwl (%rdi), %eax
+; SSE41-NEXT:    movd %eax, %xmm0
+; SSE41-NEXT:    movzwl (%rsi), %eax
+; SSE41-NEXT:    movd %eax, %xmm1
+; SSE41-NEXT:    psubusb %xmm1, %xmm0
+; SSE41-NEXT:    pextrw $0, %xmm0, (%rdx)
+; SSE41-NEXT:    retq
+;
+; AVX1-LABEL: v2i8:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    movzwl (%rdi), %eax
+; AVX1-NEXT:    vmovd %eax, %xmm0
+; AVX1-NEXT:    movzwl (%rsi), %eax
+; AVX1-NEXT:    vmovd %eax, %xmm1
+; AVX1-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpextrw $0, %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v2i8:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    movzwl (%rdi), %eax
+; AVX2-NEXT:    vmovd %eax, %xmm0
+; AVX2-NEXT:    movzwl (%rsi), %eax
+; AVX2-NEXT:    vmovd %eax, %xmm1
+; AVX2-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpextrw $0, %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v2i8:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    movzwl (%rdi), %eax
+; AVX512-NEXT:    vmovd %eax, %xmm0
+; AVX512-NEXT:    movzwl (%rsi), %eax
+; AVX512-NEXT:    vmovd %eax, %xmm1
+; AVX512-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxbq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,zero,zero,zero,zero,xmm0[1],zero,zero,zero,zero,zero,zero,zero
+; AVX512-NEXT:    vpmovqb %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <2 x i8>, <2 x i8>* %px
+  %y = load <2 x i8>, <2 x i8>* %py
+  %z = call <2 x i8> @llvm.usub.sat.v2i8(<2 x i8> %x, <2 x i8> %y)
+  store <2 x i8> %z, <2 x i8>* %pz
+  ret void
+}
+
+define void @v4i16(<4 x i16>* %px, <4 x i16>* %py, <4 x i16>* %pz) {
+; SSE-LABEL: v4i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq {{.*#+}} xmm0 = mem[0],zero
+; SSE-NEXT:    movq {{.*#+}} xmm1 = mem[0],zero
+; SSE-NEXT:    psubusw %xmm1, %xmm0
+; SSE-NEXT:    movq %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v4i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX1-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX1-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovq %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v4i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX2-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX2-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovq %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v4i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovq {{.*#+}} xmm0 = mem[0],zero
+; AVX512-NEXT:    vmovq {{.*#+}} xmm1 = mem[0],zero
+; AVX512-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxwd {{.*#+}} xmm0 = xmm0[0],zero,xmm0[1],zero,xmm0[2],zero,xmm0[3],zero
+; AVX512-NEXT:    vpmovdw %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <4 x i16>, <4 x i16>* %px
+  %y = load <4 x i16>, <4 x i16>* %py
+  %z = call <4 x i16> @llvm.usub.sat.v4i16(<4 x i16> %x, <4 x i16> %y)
+  store <4 x i16> %z, <4 x i16>* %pz
+  ret void
+}
+
+define void @v2i16(<2 x i16>* %px, <2 x i16>* %py, <2 x i16>* %pz) {
+; SSE-LABEL: v2i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; SSE-NEXT:    movd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; SSE-NEXT:    psubusw %xmm1, %xmm0
+; SSE-NEXT:    movd %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v2i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX1-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vmovd %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v2i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX2-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vmovd %xmm0, (%rdx)
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v2i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovd {{.*#+}} xmm0 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vmovd {{.*#+}} xmm1 = mem[0],zero,zero,zero
+; AVX512-NEXT:    vpsubusw %xmm1, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovzxwq {{.*#+}} xmm0 = xmm0[0],zero,zero,zero,xmm0[1],zero,zero,zero
+; AVX512-NEXT:    vpmovqw %xmm0, (%rdx)
+; AVX512-NEXT:    retq
+  %x = load <2 x i16>, <2 x i16>* %px
+  %y = load <2 x i16>, <2 x i16>* %py
+  %z = call <2 x i16> @llvm.usub.sat.v2i16(<2 x i16> %x, <2 x i16> %y)
+  store <2 x i16> %z, <2 x i16>* %pz
+  ret void
+}
+
+define <12 x i8> @v12i8(<12 x i8> %x, <12 x i8> %y) {
+; SSE-LABEL: v12i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v12i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <12 x i8> @llvm.usub.sat.v12i8(<12 x i8> %x, <12 x i8> %y)
+  ret <12 x i8> %z
+}
+
+define void @v12i16(<12 x i16>* %px, <12 x i16>* %py, <12 x i16>* %pz) {
+; SSE-LABEL: v12i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movdqa (%rdi), %xmm0
+; SSE-NEXT:    movdqa 16(%rdi), %xmm1
+; SSE-NEXT:    psubusw (%rsi), %xmm0
+; SSE-NEXT:    psubusw 16(%rsi), %xmm1
+; SSE-NEXT:    movq %xmm1, 16(%rdx)
+; SSE-NEXT:    movdqa %xmm0, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v12i16:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vmovdqa (%rdi), %xmm0
+; AVX1-NEXT:    vmovdqa 16(%rdi), %xmm1
+; AVX1-NEXT:    vpsubusw (%rsi), %xmm0, %xmm0
+; AVX1-NEXT:    vpsubusw 16(%rsi), %xmm1, %xmm1
+; AVX1-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX1-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v12i16:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vmovdqa (%rdi), %ymm0
+; AVX2-NEXT:    vpsubusw (%rsi), %ymm0, %ymm0
+; AVX2-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; AVX2-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX2-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX2-NEXT:    vzeroupper
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v12i16:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vmovdqa (%rdi), %ymm0
+; AVX512-NEXT:    vpsubusw (%rsi), %ymm0, %ymm0
+; AVX512-NEXT:    vextracti128 $1, %ymm0, %xmm1
+; AVX512-NEXT:    vmovq %xmm1, 16(%rdx)
+; AVX512-NEXT:    vmovdqa %xmm0, (%rdx)
+; AVX512-NEXT:    vzeroupper
+; AVX512-NEXT:    retq
+  %x = load <12 x i16>, <12 x i16>* %px
+  %y = load <12 x i16>, <12 x i16>* %py
+  %z = call <12 x i16> @llvm.usub.sat.v12i16(<12 x i16> %x, <12 x i16> %y)
+  store <12 x i16> %z, <12 x i16>* %pz
+  ret void
+}
+
+; Scalarization
+
+define void @v1i8(<1 x i8>* %px, <1 x i8>* %py, <1 x i8>* %pz) {
+; SSE-LABEL: v1i8:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movb (%rdi), %al
+; SSE-NEXT:    subb (%rsi), %al
+; SSE-NEXT:    jae .LBB13_2
+; SSE-NEXT:  # %bb.1:
+; SSE-NEXT:    xorl %eax, %eax
+; SSE-NEXT:  .LBB13_2:
+; SSE-NEXT:    movb %al, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v1i8:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movb (%rdi), %al
+; AVX-NEXT:    subb (%rsi), %al
+; AVX-NEXT:    jae .LBB13_2
+; AVX-NEXT:  # %bb.1:
+; AVX-NEXT:    xorl %eax, %eax
+; AVX-NEXT:  .LBB13_2:
+; AVX-NEXT:    movb %al, (%rdx)
+; AVX-NEXT:    retq
+  %x = load <1 x i8>, <1 x i8>* %px
+  %y = load <1 x i8>, <1 x i8>* %py
+  %z = call <1 x i8> @llvm.usub.sat.v1i8(<1 x i8> %x, <1 x i8> %y)
+  store <1 x i8> %z, <1 x i8>* %pz
+  ret void
+}
+
+define void @v1i16(<1 x i16>* %px, <1 x i16>* %py, <1 x i16>* %pz) {
+; SSE-LABEL: v1i16:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movzwl (%rdi), %eax
+; SSE-NEXT:    xorl %ecx, %ecx
+; SSE-NEXT:    subw (%rsi), %ax
+; SSE-NEXT:    cmovbl %ecx, %eax
+; SSE-NEXT:    movw %ax, (%rdx)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v1i16:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movzwl (%rdi), %eax
+; AVX-NEXT:    xorl %ecx, %ecx
+; AVX-NEXT:    subw (%rsi), %ax
+; AVX-NEXT:    cmovbl %ecx, %eax
+; AVX-NEXT:    movw %ax, (%rdx)
+; AVX-NEXT:    retq
+  %x = load <1 x i16>, <1 x i16>* %px
+  %y = load <1 x i16>, <1 x i16>* %py
+  %z = call <1 x i16> @llvm.usub.sat.v1i16(<1 x i16> %x, <1 x i16> %y)
+  store <1 x i16> %z, <1 x i16>* %pz
+  ret void
+}
+
+; Promotion
+
+define <16 x i4> @v16i4(<16 x i4> %x, <16 x i4> %y) {
+; SSE-LABEL: v16i4:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psllw $4, %xmm1
+; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [240,240,240,240,240,240,240,240,240,240,240,240,240,240,240,240]
+; SSE-NEXT:    pand %xmm2, %xmm1
+; SSE-NEXT:    psllw $4, %xmm0
+; SSE-NEXT:    pand %xmm2, %xmm0
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    psrlw $4, %xmm0
+; SSE-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v16i4:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsllw $4, %xmm1, %xmm1
+; AVX-NEXT:    vmovdqa {{.*#+}} xmm2 = [240,240,240,240,240,240,240,240,240,240,240,240,240,240,240,240]
+; AVX-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX-NEXT:    vpsllw $4, %xmm0, %xmm0
+; AVX-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX-NEXT:    vpsrlw $4, %xmm0, %xmm0
+; AVX-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <16 x i4> @llvm.usub.sat.v16i4(<16 x i4> %x, <16 x i4> %y)
+  ret <16 x i4> %z
+}
+
+define <16 x i1> @v16i1(<16 x i1> %x, <16 x i1> %y) {
+; SSE-LABEL: v16i1:
+; SSE:       # %bb.0:
+; SSE-NEXT:    psllw $7, %xmm1
+; SSE-NEXT:    movdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; SSE-NEXT:    pand %xmm2, %xmm1
+; SSE-NEXT:    psllw $7, %xmm0
+; SSE-NEXT:    pand %xmm2, %xmm0
+; SSE-NEXT:    psubusb %xmm1, %xmm0
+; SSE-NEXT:    psrlw $7, %xmm0
+; SSE-NEXT:    pand {{.*}}(%rip), %xmm0
+; SSE-NEXT:    retq
+;
+; AVX1-LABEL: v16i1:
+; AVX1:       # %bb.0:
+; AVX1-NEXT:    vpsllw $7, %xmm1, %xmm1
+; AVX1-NEXT:    vmovdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; AVX1-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX1-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX1-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX1-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX1-NEXT:    vpsrlw $7, %xmm0, %xmm0
+; AVX1-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX1-NEXT:    retq
+;
+; AVX2-LABEL: v16i1:
+; AVX2:       # %bb.0:
+; AVX2-NEXT:    vpsllw $7, %xmm1, %xmm1
+; AVX2-NEXT:    vmovdqa {{.*#+}} xmm2 = [128,128,128,128,128,128,128,128,128,128,128,128,128,128,128,128]
+; AVX2-NEXT:    vpand %xmm2, %xmm1, %xmm1
+; AVX2-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX2-NEXT:    vpand %xmm2, %xmm0, %xmm0
+; AVX2-NEXT:    vpsubusb %xmm1, %xmm0, %xmm0
+; AVX2-NEXT:    vpsrlw $7, %xmm0, %xmm0
+; AVX2-NEXT:    vpand {{.*}}(%rip), %xmm0, %xmm0
+; AVX2-NEXT:    retq
+;
+; AVX512-LABEL: v16i1:
+; AVX512:       # %bb.0:
+; AVX512-NEXT:    vpsllw $7, %xmm0, %xmm0
+; AVX512-NEXT:    vpmovb2m %xmm0, %k0
+; AVX512-NEXT:    vpsllw $7, %xmm1, %xmm0
+; AVX512-NEXT:    vpmovb2m %xmm0, %k1
+; AVX512-NEXT:    kandnw %k0, %k1, %k0
+; AVX512-NEXT:    vpmovm2b %k0, %xmm0
+; AVX512-NEXT:    retq
+  %z = call <16 x i1> @llvm.usub.sat.v16i1(<16 x i1> %x, <16 x i1> %y)
+  ret <16 x i1> %z
+}
+
+; Expanded
+
+define <4 x i32> @v4i32(<4 x i32> %x, <4 x i32> %y) {
+; SSE2-LABEL: v4i32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %ecx
+; SSE2-NEXT:    xorl %edx, %edx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm3
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSE2-NEXT:    movd %xmm1, %eax
+; SSE2-NEXT:    movd %xmm0, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSE2-NEXT:    movd %xmm1, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE2-NEXT:    movd %xmm0, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm0
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v4i32:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %ecx
+; SSSE3-NEXT:    xorl %edx, %edx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm3
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSSE3-NEXT:    movd %xmm1, %eax
+; SSSE3-NEXT:    movd %xmm0, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm1, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm0, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm0
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v4i32:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pextrd $1, %xmm1, %eax
+; SSE41-NEXT:    pextrd $1, %xmm0, %ecx
+; SSE41-NEXT:    xorl %edx, %edx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    movd %xmm1, %eax
+; SSE41-NEXT:    movd %xmm0, %esi
+; SSE41-NEXT:    subl %eax, %esi
+; SSE41-NEXT:    cmovbl %edx, %esi
+; SSE41-NEXT:    movd %esi, %xmm2
+; SSE41-NEXT:    pinsrd $1, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $2, %xmm1, %eax
+; SSE41-NEXT:    pextrd $2, %xmm0, %ecx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    pinsrd $2, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $3, %xmm1, %eax
+; SSE41-NEXT:    pextrd $3, %xmm0, %ecx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    pinsrd $3, %ecx, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v4i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpextrd $1, %xmm1, %eax
+; AVX-NEXT:    vpextrd $1, %xmm0, %ecx
+; AVX-NEXT:    xorl %edx, %edx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vmovd %xmm1, %eax
+; AVX-NEXT:    vmovd %xmm0, %esi
+; AVX-NEXT:    subl %eax, %esi
+; AVX-NEXT:    cmovbl %edx, %esi
+; AVX-NEXT:    vmovd %esi, %xmm2
+; AVX-NEXT:    vpinsrd $1, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $2, %xmm1, %eax
+; AVX-NEXT:    vpextrd $2, %xmm0, %ecx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vpinsrd $2, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $3, %xmm1, %eax
+; AVX-NEXT:    vpextrd $3, %xmm0, %ecx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vpinsrd $3, %ecx, %xmm2, %xmm0
+; AVX-NEXT:    retq
+  %z = call <4 x i32> @llvm.usub.sat.v4i32(<4 x i32> %x, <4 x i32> %y)
+  ret <4 x i32> %z
+}
+
+define <2 x i32> @v2i32(<2 x i32> %x, <2 x i32> %y) {
+; SSE2-LABEL: v2i32:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    psllq $32, %xmm1
+; SSE2-NEXT:    movq %xmm1, %rax
+; SSE2-NEXT:    psllq $32, %xmm0
+; SSE2-NEXT:    movq %xmm0, %rcx
+; SSE2-NEXT:    xorl %edx, %edx
+; SSE2-NEXT:    subq %rax, %rcx
+; SSE2-NEXT:    cmovbq %rdx, %rcx
+; SSE2-NEXT:    movq %rcx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movq %xmm1, %rax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movq %xmm0, %rcx
+; SSE2-NEXT:    subq %rax, %rcx
+; SSE2-NEXT:    cmovbq %rdx, %rcx
+; SSE2-NEXT:    movq %rcx, %xmm0
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
+; SSE2-NEXT:    psrlq $32, %xmm2
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v2i32:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    psllq $32, %xmm1
+; SSSE3-NEXT:    movq %xmm1, %rax
+; SSSE3-NEXT:    psllq $32, %xmm0
+; SSSE3-NEXT:    movq %xmm0, %rcx
+; SSSE3-NEXT:    xorl %edx, %edx
+; SSSE3-NEXT:    subq %rax, %rcx
+; SSSE3-NEXT:    cmovbq %rdx, %rcx
+; SSSE3-NEXT:    movq %rcx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movq %xmm1, %rax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movq %xmm0, %rcx
+; SSSE3-NEXT:    subq %rax, %rcx
+; SSSE3-NEXT:    cmovbq %rdx, %rcx
+; SSSE3-NEXT:    movq %rcx, %xmm0
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm0[0]
+; SSSE3-NEXT:    psrlq $32, %xmm2
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v2i32:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    psllq $32, %xmm1
+; SSE41-NEXT:    pextrq $1, %xmm1, %rax
+; SSE41-NEXT:    psllq $32, %xmm0
+; SSE41-NEXT:    pextrq $1, %xmm0, %rcx
+; SSE41-NEXT:    xorl %edx, %edx
+; SSE41-NEXT:    subq %rax, %rcx
+; SSE41-NEXT:    cmovbq %rdx, %rcx
+; SSE41-NEXT:    movq %rcx, %xmm2
+; SSE41-NEXT:    movq %xmm1, %rax
+; SSE41-NEXT:    movq %xmm0, %rcx
+; SSE41-NEXT:    subq %rax, %rcx
+; SSE41-NEXT:    cmovbq %rdx, %rcx
+; SSE41-NEXT:    movq %rcx, %xmm0
+; SSE41-NEXT:    punpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; SSE41-NEXT:    psrlq $32, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v2i32:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpsllq $32, %xmm1, %xmm1
+; AVX-NEXT:    vpextrq $1, %xmm1, %rax
+; AVX-NEXT:    vpsllq $32, %xmm0, %xmm0
+; AVX-NEXT:    vpextrq $1, %xmm0, %rcx
+; AVX-NEXT:    xorl %edx, %edx
+; AVX-NEXT:    subq %rax, %rcx
+; AVX-NEXT:    cmovbq %rdx, %rcx
+; AVX-NEXT:    vmovq %rcx, %xmm2
+; AVX-NEXT:    vmovq %xmm1, %rax
+; AVX-NEXT:    vmovq %xmm0, %rcx
+; AVX-NEXT:    subq %rax, %rcx
+; AVX-NEXT:    cmovbq %rdx, %rcx
+; AVX-NEXT:    vmovq %rcx, %xmm0
+; AVX-NEXT:    vpunpcklqdq {{.*#+}} xmm0 = xmm0[0],xmm2[0]
+; AVX-NEXT:    vpsrlq $32, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <2 x i32> @llvm.usub.sat.v2i32(<2 x i32> %x, <2 x i32> %y)
+  ret <2 x i32> %z
+}
+
+define <4 x i24> @v4i24(<4 x i24> %x, <4 x i24> %y) {
+; SSE2-LABEL: v4i24:
+; SSE2:       # %bb.0:
+; SSE2-NEXT:    pslld $8, %xmm1
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %eax
+; SSE2-NEXT:    pslld $8, %xmm0
+; SSE2-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSE2-NEXT:    movd %xmm2, %ecx
+; SSE2-NEXT:    xorl %edx, %edx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSE2-NEXT:    movd %xmm3, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm3
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSE2-NEXT:    movd %xmm1, %eax
+; SSE2-NEXT:    movd %xmm0, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm2
+; SSE2-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSE2-NEXT:    movd %xmm1, %eax
+; SSE2-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSE2-NEXT:    movd %xmm0, %ecx
+; SSE2-NEXT:    subl %eax, %ecx
+; SSE2-NEXT:    cmovbl %edx, %ecx
+; SSE2-NEXT:    movd %ecx, %xmm0
+; SSE2-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSE2-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSE2-NEXT:    psrld $8, %xmm2
+; SSE2-NEXT:    movdqa %xmm2, %xmm0
+; SSE2-NEXT:    retq
+;
+; SSSE3-LABEL: v4i24:
+; SSSE3:       # %bb.0:
+; SSSE3-NEXT:    pslld $8, %xmm1
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm1[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %eax
+; SSSE3-NEXT:    pslld $8, %xmm0
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm2 = xmm0[3,1,2,3]
+; SSSE3-NEXT:    movd %xmm2, %ecx
+; SSSE3-NEXT:    xorl %edx, %edx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm1[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm3 = xmm0[2,3,0,1]
+; SSSE3-NEXT:    movd %xmm3, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm3
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm3 = xmm3[0],xmm2[0],xmm3[1],xmm2[1]
+; SSSE3-NEXT:    movd %xmm1, %eax
+; SSSE3-NEXT:    movd %xmm0, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm2
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm1 = xmm1[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm1, %eax
+; SSSE3-NEXT:    pshufd {{.*#+}} xmm0 = xmm0[1,1,2,3]
+; SSSE3-NEXT:    movd %xmm0, %ecx
+; SSSE3-NEXT:    subl %eax, %ecx
+; SSSE3-NEXT:    cmovbl %edx, %ecx
+; SSSE3-NEXT:    movd %ecx, %xmm0
+; SSSE3-NEXT:    punpckldq {{.*#+}} xmm2 = xmm2[0],xmm0[0],xmm2[1],xmm0[1]
+; SSSE3-NEXT:    punpcklqdq {{.*#+}} xmm2 = xmm2[0],xmm3[0]
+; SSSE3-NEXT:    psrld $8, %xmm2
+; SSSE3-NEXT:    movdqa %xmm2, %xmm0
+; SSSE3-NEXT:    retq
+;
+; SSE41-LABEL: v4i24:
+; SSE41:       # %bb.0:
+; SSE41-NEXT:    pslld $8, %xmm1
+; SSE41-NEXT:    pextrd $1, %xmm1, %eax
+; SSE41-NEXT:    pslld $8, %xmm0
+; SSE41-NEXT:    pextrd $1, %xmm0, %ecx
+; SSE41-NEXT:    xorl %edx, %edx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    movd %xmm1, %eax
+; SSE41-NEXT:    movd %xmm0, %esi
+; SSE41-NEXT:    subl %eax, %esi
+; SSE41-NEXT:    cmovbl %edx, %esi
+; SSE41-NEXT:    movd %esi, %xmm2
+; SSE41-NEXT:    pinsrd $1, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $2, %xmm1, %eax
+; SSE41-NEXT:    pextrd $2, %xmm0, %ecx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    pinsrd $2, %ecx, %xmm2
+; SSE41-NEXT:    pextrd $3, %xmm1, %eax
+; SSE41-NEXT:    pextrd $3, %xmm0, %ecx
+; SSE41-NEXT:    subl %eax, %ecx
+; SSE41-NEXT:    cmovbl %edx, %ecx
+; SSE41-NEXT:    pinsrd $3, %ecx, %xmm2
+; SSE41-NEXT:    psrld $8, %xmm2
+; SSE41-NEXT:    movdqa %xmm2, %xmm0
+; SSE41-NEXT:    retq
+;
+; AVX-LABEL: v4i24:
+; AVX:       # %bb.0:
+; AVX-NEXT:    vpslld $8, %xmm1, %xmm1
+; AVX-NEXT:    vpextrd $1, %xmm1, %eax
+; AVX-NEXT:    vpslld $8, %xmm0, %xmm0
+; AVX-NEXT:    vpextrd $1, %xmm0, %ecx
+; AVX-NEXT:    xorl %edx, %edx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vmovd %xmm1, %eax
+; AVX-NEXT:    vmovd %xmm0, %esi
+; AVX-NEXT:    subl %eax, %esi
+; AVX-NEXT:    cmovbl %edx, %esi
+; AVX-NEXT:    vmovd %esi, %xmm2
+; AVX-NEXT:    vpinsrd $1, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $2, %xmm1, %eax
+; AVX-NEXT:    vpextrd $2, %xmm0, %ecx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vpinsrd $2, %ecx, %xmm2, %xmm2
+; AVX-NEXT:    vpextrd $3, %xmm1, %eax
+; AVX-NEXT:    vpextrd $3, %xmm0, %ecx
+; AVX-NEXT:    subl %eax, %ecx
+; AVX-NEXT:    cmovbl %edx, %ecx
+; AVX-NEXT:    vpinsrd $3, %ecx, %xmm2, %xmm0
+; AVX-NEXT:    vpsrld $8, %xmm0, %xmm0
+; AVX-NEXT:    retq
+  %z = call <4 x i24> @llvm.usub.sat.v4i24(<4 x i24> %x, <4 x i24> %y)
+  ret <4 x i24> %z
+}
+
+define <2 x i128> @v2i128(<2 x i128> %x, <2 x i128> %y) {
+; SSE-LABEL: v2i128:
+; SSE:       # %bb.0:
+; SSE-NEXT:    movq %rdi, %rax
+; SSE-NEXT:    xorl %edi, %edi
+; SSE-NEXT:    subq %r9, %rsi
+; SSE-NEXT:    sbbq {{[0-9]+}}(%rsp), %rdx
+; SSE-NEXT:    cmovbq %rdi, %rsi
+; SSE-NEXT:    cmovbq %rdi, %rdx
+; SSE-NEXT:    subq {{[0-9]+}}(%rsp), %rcx
+; SSE-NEXT:    sbbq {{[0-9]+}}(%rsp), %r8
+; SSE-NEXT:    cmovbq %rdi, %r8
+; SSE-NEXT:    cmovbq %rdi, %rcx
+; SSE-NEXT:    movq %r8, 24(%rax)
+; SSE-NEXT:    movq %rcx, 16(%rax)
+; SSE-NEXT:    movq %rdx, 8(%rax)
+; SSE-NEXT:    movq %rsi, (%rax)
+; SSE-NEXT:    retq
+;
+; AVX-LABEL: v2i128:
+; AVX:       # %bb.0:
+; AVX-NEXT:    movq %rdi, %rax
+; AVX-NEXT:    xorl %edi, %edi
+; AVX-NEXT:    subq %r9, %rsi
+; AVX-NEXT:    sbbq {{[0-9]+}}(%rsp), %rdx
+; AVX-NEXT:    cmovbq %rdi, %rsi
+; AVX-NEXT:    cmovbq %rdi, %rdx
+; AVX-NEXT:    subq {{[0-9]+}}(%rsp), %rcx
+; AVX-NEXT:    sbbq {{[0-9]+}}(%rsp), %r8
+; AVX-NEXT:    cmovbq %rdi, %r8
+; AVX-NEXT:    cmovbq %rdi, %rcx
+; AVX-NEXT:    movq %r8, 24(%rax)
+; AVX-NEXT:    movq %rcx, 16(%rax)
+; AVX-NEXT:    movq %rdx, 8(%rax)
+; AVX-NEXT:    movq %rsi, (%rax)
+; AVX-NEXT:    retq
+  %z = call <2 x i128> @llvm.usub.sat.v2i128(<2 x i128> %x, <2 x i128> %y)
+  ret <2 x i128> %z
+}