Add intrinsics and SDNodes for signed/unsigned absolute difference (absdiff) and horizontal add (hadd).

[rengolin]

Summary

This adds new intrinsics "*absdiff" and "*hadd" to facilitate the efficient code generation for "sum of absolute differences" operation.

The patch also contains the introduction of corresponding SDNodes and basic legalization support

An RFC discussion could be found at below link
https://groups.google.com/forum/#!topic/llvm-dev/S3P1Sh3H--Q

[jmolloy]

Hi,

Thanks for working on this!

I have a bunch of comments, and this is just a first round review. Also, this needs to be split into three separate patches:

1. Addition of sabsdiff/uabsdiff
2. Addition of hadd
3. X86 changes

Each should have testcases. The first two don't currently; you're only testing the X86 backend's handling of the intrinsics which may be fine, but you do need to ensure you're checking the generic EXPAND behaviour as well as the specific X86 behaviour you've added in patch (3).

Cheers,

James

docs/LangRef.rst

9562

I don't think floating point datatypes should be bundled in with integer datatypes in these intrinsics. Signedness isn't a concept for floating point datatypes, so it would be weird to have something operate on floats with a defined signedness. Also, it would mean two intrinsics were identical for floating point types which raises a canonicalisation problem.

I'd be happy if the floating point support for these intrinsics was removed for the time being, if you don't want to add another intrinsic right now.

9586

What does this note mean, in terms of action required by the reader? Do you mean that users should aim not to write these intrinsics? If so, what should they do instead?

9619

Because addition in wrapping 2's complement arithmetic is sign independent, you shouldn't need a signed and unsigned version of this. Just "llvm.hadd" should do.

include/llvm/CodeGen/ISDOpcodes.h

335

I'd prefer "SABSDIFF" here to match the intrinsic better and be more self documenting.

include/llvm/Target/TargetSelectionDAG.td

388

I'd prefer "sabsdiff" here to match the intrinsic name better, and be more self documenting.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2888

What happens if the "if" condition fails? Tmp1 is a dangling pointer at the moment.

2893

Please write how you expect the expansion to be improved.

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

160

I'm not sure about this - I think you shouldn't need to care about the high bits of a promoted value (that's the responsibility of PromoteIntOp of the user), so you can just use PromoteIntRes_SimpleBinOp here instead.

908

Why are you not handling ISD::SABD and friends here?

1243

You don't need this. Because addition is agnostic to the high bits, you don't need to extend anything. You can just use PromoteIntOp_SimpleIntBinOp here.

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

229

Indentation error.

[• ashahid]

Hi James,

Thanks for your comments.

I am on vacation now. I will look into it accordingly afterwards.

Regards,
Shahid

[• ashahid]

docs/LangRef.rst

9562

Thanks, I would prefer to defer floating point intrinsics for time being.

9586

In fact I meant this.
"These intrinsics are primarily used during code generation stage of compilation.They are generated by the compiler passes such as Loop/SLP vectorizer."

9619

Even I was not sure about this. However, as ARM has both signed / unsigned horizontal add instruction, how would you decide to to emit the proper one?

include/llvm/CodeGen/ISDOpcodes.h

335

Will handle it in follow up patches.

include/llvm/Target/TargetSelectionDAG.td

388

Will handle it in follow up patches.

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2888

Will be handled in follow up patches.

2893

Oops, comment slipped

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

160

Will be handled in follow up patches.

908

Will be handled accordingly.

[jmolloy]

[Phab spring clean] This patch has been usurped by D10964 and D10967.

[rengolin]

Shall we abandon it, then?

@ashahid?

[• ashahid]

Hi,

Yes, this can be abandoned.

Sorry for the delayed response as I am not keeping well these days.

Regards,
Shahid

docs/LangRef.rst

 
 Examples:
 """""""""
 
 .. code-block:: llvm
 
       %r2 = call float @llvm.fmuladd.f32(float %a, float %b, float %c) ; yields float:r2 = (a * b) + c
 
+
+'``llvm.uabsdiff.*``' and '``llvm.sabsdiff.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic. The loaded data is a vector of any integer or floating point data type.
+
+.. code-block:: llvm
+
+      declare <4 x integer> @llvm.uabsdiff.v4i32(<4 x integer> %a, <4 x integer> %b)
+      declare <4 x float> @llvm.uabsdiff.v4f32(<4 x float> %a, <4 x float> %b)
+
+
+Overview:
+"""""""""
+
+The ``llvm.uabsdiff`` intrinsic returns a vector result of the absolute difference of the two operands,
+treating them both as unsigned integers or floats.
+
+The ``llvm.sabsdiff`` intrinsic returns  a vector result of the absolute difference of the two operands,
+treating them both as signed integers or float.
+
+.. note::
+
+    These intrinsics are primarily used during optimization phase.
+
+Arguments:
+""""""""""
+
+Both intrinsics take two integer or float arguments of the same bitwidth.
+
+Semantics:
+""""""""""
+
+The expression::
+
+    call <4 x i32> @llvm.uabsdiff.v4i32(<4 x i32> %a, <4 x i32> %b)
+
+is equivalent to::
+
+    %sub = sub <4 x i32> %a, %b
+    %ispos = icmp ugt <4 x i32> %sub, <i32 -1, i32 -1, i32 -1, i32 -1>
+    %neg = sub <4 x i32> zeroinitializer, %sub
+    %1 = select <4 x i1> %ispos, <4 x i32> %sub, <4 x i32> %neg
+
+Similarly the expression::
+
+    call <4 x i32> @llvm.sabsdiff.v4i32(<4 x i32> %a, <4 x i32> %b)
+
+is equivalent to::
+
+    %sub = sub nsw <4 x i32> %a, %b
+    %ispos = icmp sgt <4 x i32> %sub, <i32 -1, i32 -1, i32 -1, i32 -1>
+    %neg = sub nsw <4 x i32> zeroinitializer, %sub
+    %1 = select <4 x i1> %ispos, <4 x i32> %sub, <4 x i32> %neg
+
+
+'``llvm.uhadd.*``' and '``llvm.shadd.*``'Intrinsic
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic. The loaded data is a vector of any integer or floating point data type.
+
+.. code-block:: llvm
+
+      declare <4 x integer> @llvm.uhadd.v4i32(<4 x integer> %a)
+      declare <4 x float> @llvm.shadd.v4f32(<4 x float> %a)
+
+Overview:
+"""""""""
+
+The ``llvm.uhadd`` intrinsic returns the result of the horizontal or reduction sum of the elements of the
+vector operand, treating it as unsigned integers or floats.
+
+The ``llvm.shadd`` intrinsic returns  a vector result of the horizontal or reduction sum of the elements
+of the vector operand, treating it as signed integers or floats.
+
+.. note::
+
+    These intrinsics are primarily used during optimization phase.
+
+Arguments:
+""""""""""
+
+Both intrinsics take single integer or float argument of the same bitwidth.
+
+Semantics:
+""""""""""
+
+The expression::
+
+    call i32 @llvm.uhadd.v4i32(<4 x i32> %a)
+
+is equivalent to::
+
+    %1 = shufflevector <4 x i32> %a, <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
+    %2 = add <4 x i32> %a, %1
+    %3 = shufflevector <4 x i32> %2, <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
+    %4 = add <4 x i32> %2, %3
+    %5 = extractelement <4 x i32> %4, i32 0
+
+Similarly the expression::
+
+    call <4 x i32> @llvm.shadd.v4i32(<4 x i32> %a)
+
+is equivalent to::
+
+    %1 = shufflevector <4 x i32> %a, <4 x i32> undef, <4 x i32> <i32 2, i32 3, i32 undef, i32 undef>
+    %2 = add nsw <4 x i32> %a, %1
+    %3 = shufflevector <4 x i32> %2, <4 x i32> undef, <4 x i32> <i32 1, i32 undef, i32 undef, i32 undef>
+    %4 = add <4 x i32> %2, %3
+    %5 = extractelement <4 x i32> %4, i32 0
+
+
 Half Precision Floating Point Intrinsics
 ----------------------------------------
 
 For most target platforms, half precision floating point is a
 storage-only format. This means that it is a dense encoding (in memory)
 but does not support computation in the format.
 
 This means that code must first load the half-precision floating point

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// When the 1st operand is a vector, the shift amount must be in the same
     /// type. (TLI.getShiftAmountTy() will return the same type when the input
     /// type is a vector.)
     SHL, SRA, SRL, ROTL, ROTR,
 
     /// Byte Swap and Counting operators.
     BSWAP, CTTZ, CTLZ, CTPOP,
 
+    /// [SU]ABSD - Signed/Unsigned absolute difference of two input integer or
+    /// float vector.
+    /// These nodes are generated from llvm.*absdiff* intrinsics.
+    SABSD, UABSD,
+
+    /// [SU]HADD - Signed/Unsigned horizontal sum across the elements of sole
+    /// integer or float input vector.
+    /// These nodes are generated from llvm.*hadd* intrinsics.
+    UHADD, SHADD,
+
     /// Bit counting operators with an undefined result for zero inputs.
     CTTZ_ZERO_UNDEF, CTLZ_ZERO_UNDEF,
 
     /// Select(COND, TRUEVAL, FALSEVAL).  If the type of the boolean COND is not
     /// i1 then the high bits must conform to getBooleanContents.
     SELECT,
 
     /// Select with a vector condition (op #0) and two vector operands (ops #1

include/llvm/IR/Intrinsics.td

 def int_convertuu  : Intrinsic<[llvm_anyint_ty],
                                [llvm_anyint_ty, llvm_i32_ty, llvm_i32_ty]>;
 
 // Clear cache intrinsic, default to ignore (ie. emit nothing)
 // maps to void __clear_cache() on supporting platforms
 def int_clear_cache : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],
                                 [], "llvm.clear_cache">;
 
+// Calculate the Absolute Differences of the two input vectors.
+def int_sabsdiff : Intrinsic<[llvm_anyvector_ty],
+                        [ llvm_anyvector_ty, llvm_anyvector_ty ], [IntrNoMem]>;
+def int_uabsdiff : Intrinsic<[llvm_anyvector_ty],
+                        [ llvm_anyvector_ty, llvm_anyvector_ty ], [IntrNoMem]>;
+
+// Calculate the horizontal/reduction sum across the elements of input vector.
+def int_uhadd : Intrinsic<[llvm_anyint_ty], [llvm_anyvector_ty], [IntrNoMem]>;
+def int_shadd : Intrinsic<[llvm_anyint_ty], [llvm_anyvector_ty], [IntrNoMem]>;
 //===-------------------------- Masked Intrinsics -------------------------===//
 //
 def int_masked_store : Intrinsic<[], [llvm_anyvector_ty, LLVMPointerTo<0>,
                                       llvm_i32_ty,
                                       LLVMVectorSameWidth<0, llvm_i1_ty>],
                                  [IntrReadWriteArgMem]>;
 
 def int_masked_load  : Intrinsic<[llvm_anyvector_ty],

include/llvm/Target/TargetSelectionDAG.td

def subc       : SDNode<"ISD::SUBC"      , SDTIntBinOp,
                         [SDNPOutGlue]>;
 def sube       : SDNode<"ISD::SUBE"      , SDTIntBinOp,
                         [SDNPOutGlue, SDNPInGlue]>;
 def smin       : SDNode<"ISD::SMIN"      , SDTIntBinOp>;
 def smax       : SDNode<"ISD::SMAX"      , SDTIntBinOp>;
 def umin       : SDNode<"ISD::UMIN"      , SDTIntBinOp>;
 def umax       : SDNode<"ISD::UMAX"      , SDTIntBinOp>;
 
+def sabsd       : SDNode<"ISD::SABSD"      , SDTIntBinOp>;
+def shadd       : SDNode<"ISD::SHADD"      , SDTIntUnaryOp>;
+def uabsd       : SDNode<"ISD::UABSD"      , SDTIntBinOp>;
+def uhadd       : SDNode<"ISD::UHADD"      , SDTIntUnaryOp>;
 def sext_inreg : SDNode<"ISD::SIGN_EXTEND_INREG", SDTExtInreg>;
 def bswap      : SDNode<"ISD::BSWAP"      , SDTIntUnaryOp>;
 def ctlz       : SDNode<"ISD::CTLZ"       , SDTIntUnaryOp>;
 def cttz       : SDNode<"ISD::CTTZ"       , SDTIntUnaryOp>;
 def ctpop      : SDNode<"ISD::CTPOP"      , SDTIntUnaryOp>;
 def ctlz_zero_undef : SDNode<"ISD::CTLZ_ZERO_UNDEF", SDTIntUnaryOp>;
 def cttz_zero_undef : SDNode<"ISD::CTTZ_ZERO_UNDEF", SDTIntUnaryOp>;
 def sext       : SDNode<"ISD::SIGN_EXTEND", SDTIntExtendOp>;

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

void SelectionDAGLegalize::ExpandNode(SDNode *Node) {
   case ISD::CTTZ:
   case ISD::CTTZ_ZERO_UNDEF:
     Tmp1 = ExpandBitCount(Node->getOpcode(), Node->getOperand(0), dl);
     Results.push_back(Tmp1);
     break;
   case ISD::BSWAP:
     Results.push_back(ExpandBSWAP(Node->getOperand(0), dl));
     break;
+  case ISD::UABSD:
+  case ISD::SABSD: {
+    EVT VT = Node->getValueType(0);
+    unsigned int SubOpc = ISD::SUB;
+    Tmp2 = Node->getOperand(0);
+    Tmp3 = Node->getOperand(1);
+    if (TLI.isOperationLegalOrCustom(ISD::VSELECT, VT)) {
+      Tmp1 = DAG.getNode(SubOpc, dl, VT, Tmp2, Tmp3);
+      Tmp2 = DAG.getNode(ISD::SUB, dl, VT, DAG.getConstant(0, dl, VT), Tmp1);
+      Tmp4 =
+          DAG.getNode(ISD::SETCC, dl, getSetCCResultType(VT), Tmp2,
+                      DAG.getConstant(0, dl, VT), DAG.getCondCode(ISD::SETLT));
+      Tmp1 = DAG.getNode(ISD::VSELECT, dl, VT, Tmp4, Tmp1, Tmp2);
+    }
+    Results.push_back(Tmp1);
+    break;
+  }
+  case ISD::UHADD:
+  case ISD::SHADD: {
+    // FIXME: Improve the expansion
+    SDValue OpVal = Node->getOperand(0);
+    EVT VT = OpVal.getValueType();
+    unsigned NumElems = VT.getVectorNumElements();
+    EVT EltVT = VT.getVectorElementType();
+
+    SmallVector<int, 16> ShuffleMask(NumElems, -1);
+    for (unsigned i = NumElems; i != 1; i >>= 1) {
+      // Move the upper half of the vector to the lower half.
+      for (unsigned j = 0; j != i / 2; ++j)
+        ShuffleMask[j] = (i / 2 + j);
+      SDValue Shuffle = DAG.getVectorShuffle(VT, dl, OpVal, DAG.getUNDEF(VT),
+                                             &ShuffleMask[0]);
+      OpVal = DAG.getNode(ISD::ADD, dl, VT, OpVal, Shuffle);
+    }
+    Tmp1 = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, EltVT, OpVal,
+                       DAG.getConstant(0, dl, TLI.getVectorIdxTy()));
+    Results.push_back(Tmp1);
+    break;
+  }
   case ISD::FRAMEADDR:
   case ISD::RETURNADDR:
   case ISD::FRAME_TO_ARGS_OFFSET:
     Results.push_back(DAG.getConstant(0, dl, Node->getValueType(0)));
     break;
   case ISD::FLT_ROUNDS_:
     Results.push_back(DAG.getConstant(1, dl, Node->getValueType(0)));
     break;

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
   case ISD::ATOMIC_LOAD_UMAX:
   case ISD::ATOMIC_SWAP:
     Res = PromoteIntRes_Atomic1(cast<AtomicSDNode>(N)); break;
 
   case ISD::ATOMIC_CMP_SWAP:
   case ISD::ATOMIC_CMP_SWAP_WITH_SUCCESS:
     Res = PromoteIntRes_AtomicCmpSwap(cast<AtomicSDNode>(N), ResNo);
     break;
+  case ISD::UABSD:
+  case ISD::SABSD:
+    Res = PromoteIntRes_AbsoluteDiff(N);
+    break;
   }
 
   // If the result is null then the sub-method took care of registering it.
   if (Res.getNode())
     SetPromotedInteger(SDValue(N, ResNo), Res);
 }
 
+SDValue DAGTypeLegalizer::PromoteIntRes_AbsoluteDiff(SDNode *N) {
+  SDValue LHS, RHS;
+  if (N->getOpcode() == ISD::SABSD) {
+    LHS = SExtPromotedInteger(N->getOperand(0));
+    RHS = SExtPromotedInteger(N->getOperand(1));
+  } else {
+    LHS = ZExtPromotedInteger(N->getOperand(0));
+    RHS = ZExtPromotedInteger(N->getOperand(1));
+  }
+  return DAG.getNode(N->getOpcode(), SDLoc(N),
+                     LHS.getValueType(), LHS, RHS);
+}
+
+SDValue DAGTypeLegalizer::PromoteIntRes_HADD(SDNode *N) {
+  SDValue LHS, RHS;
+  if (N->getOpcode() == ISD::SHADD) {
+    LHS = SExtPromotedInteger(N->getOperand(0));
+    RHS = SExtPromotedInteger(N->getOperand(1));
+  } else {
+    LHS = ZExtPromotedInteger(N->getOperand(0));
+    RHS = ZExtPromotedInteger(N->getOperand(1));
+  }
+  return DAG.getNode(N->getOpcode(), SDLoc(N),
+                     LHS.getValueType(), LHS, RHS);
+}
+
 SDValue DAGTypeLegalizer::PromoteIntRes_MERGE_VALUES(SDNode *N,
                                                      unsigned ResNo) {
   SDValue Op = DisintegrateMERGE_VALUES(N, ResNo);
   return GetPromotedInteger(Op);
 }
 
 SDValue DAGTypeLegalizer::PromoteIntRes_AssertSext(SDNode *N) {
   // Sign-extend the new bits, and continue the assertion.
bool DAGTypeLegalizer::PromoteIntegerOperand(SDNode *N, unsigned OpNo) {
   case ISD::ZERO_EXTEND:  Res = PromoteIntOp_ZERO_EXTEND(N); break;
   case ISD::EXTRACT_SUBVECTOR: Res = PromoteIntOp_EXTRACT_SUBVECTOR(N); break;
 
   case ISD::SHL:
   case ISD::SRA:
   case ISD::SRL:
   case ISD::ROTL:
   case ISD::ROTR: Res = PromoteIntOp_Shift(N); break;
+  case ISD::UHADD:
+  case ISD::SHADD: Res = PromoteIntOp_HADD(N); break;
   }
 
   // If the result is null, the sub-method took care of registering results etc.
   if (!Res.getNode()) return false;
 
   // If the result is N, the sub-method updated N in place.  Tell the legalizer
   // core about this.
   if (Res.getNode() == N)

 SDValue DAGTypeLegalizer::PromoteIntOp_ZERO_EXTEND(SDNode *N) {
   SDLoc dl(N);
   SDValue Op = GetPromotedInteger(N->getOperand(0));
   Op = DAG.getNode(ISD::ANY_EXTEND, dl, N->getValueType(0), Op);
   return DAG.getZeroExtendInReg(Op, dl,
                                 N->getOperand(0).getValueType().getScalarType());
 }
 
+SDValue DAGTypeLegalizer::PromoteIntOp_HADD(SDNode *N) {
+  SDValue Op;
+  if (N->getOpcode() == ISD::SHADD) {
+    Op = SExtPromotedInteger(N->getOperand(0));
+  } else {
+    Op = ZExtPromotedInteger(N->getOperand(0));
+  }
+  return DAG.getNode(N->getOpcode(), SDLoc(N),
+                     N->getValueType(0), Op);
+}
 
 //===----------------------------------------------------------------------===//
 //  Integer Result Expansion
 //===----------------------------------------------------------------------===//
 
 /// ExpandIntegerResult - This method is called when the specified result of the
 /// specified node is found to need expansion.  At this point, the node may also
 /// have invalid operands or may have other results that need promotion, we just

lib/CodeGen/SelectionDAG/LegalizeTypes.h

private:
   SDValue PromoteIntRes_SRA(SDNode *N);
   SDValue PromoteIntRes_SRL(SDNode *N);
   SDValue PromoteIntRes_TRUNCATE(SDNode *N);
   SDValue PromoteIntRes_UADDSUBO(SDNode *N, unsigned ResNo);
   SDValue PromoteIntRes_UDIV(SDNode *N);
   SDValue PromoteIntRes_UNDEF(SDNode *N);
   SDValue PromoteIntRes_VAARG(SDNode *N);
   SDValue PromoteIntRes_XMULO(SDNode *N, unsigned ResNo);
+  SDValue PromoteIntRes_AbsoluteDiff(SDNode *N);
+  SDValue PromoteIntRes_HADD(SDNode *N);
 
   // Integer Operand Promotion.
   bool PromoteIntegerOperand(SDNode *N, unsigned OperandNo);
   SDValue PromoteIntOp_ANY_EXTEND(SDNode *N);
   SDValue PromoteIntOp_ATOMIC_STORE(AtomicSDNode *N);
   SDValue PromoteIntOp_BITCAST(SDNode *N);
   SDValue PromoteIntOp_BUILD_PAIR(SDNode *N);
   SDValue PromoteIntOp_BR_CC(SDNode *N, unsigned OpNo);
private:
   SDValue PromoteIntOp_SIGN_EXTEND(SDNode *N);
   SDValue PromoteIntOp_SINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_STORE(StoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_TRUNCATE(SDNode *N);
   SDValue PromoteIntOp_UINT_TO_FP(SDNode *N);
   SDValue PromoteIntOp_ZERO_EXTEND(SDNode *N);
   SDValue PromoteIntOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);
   SDValue PromoteIntOp_MLOAD(MaskedLoadSDNode *N, unsigned OpNo);
+  SDValue PromoteIntOp_HADD(SDNode *N);
 
   void PromoteSetCCOperands(SDValue &LHS,SDValue &RHS, ISD::CondCode Code);
 
   //===--------------------------------------------------------------------===//
   // Integer Expansion Support: LegalizeIntegerTypes.cpp
   //===--------------------------------------------------------------------===//
 
   /// GetExpandedInteger - Given a processed operand Op which was expanded into
private:
   void SplitVecRes_VECTOR_SHUFFLE(ShuffleVectorSDNode *N, SDValue &Lo,
                                   SDValue &Hi);
 
   // Vector Operand Splitting: <128 x ty> -> 2 x <64 x ty>.
   bool SplitVectorOperand(SDNode *N, unsigned OpNo);
   SDValue SplitVecOp_VSELECT(SDNode *N, unsigned OpNo);
   SDValue SplitVecOp_UnaryOp(SDNode *N);
   SDValue SplitVecOp_TruncateHelper(SDNode *N);
+  SDValue SplitVecOp_HADD(SDNode *N);
 
   SDValue SplitVecOp_BITCAST(SDNode *N);
   SDValue SplitVecOp_EXTRACT_SUBVECTOR(SDNode *N);
   SDValue SplitVecOp_EXTRACT_VECTOR_ELT(SDNode *N);
   SDValue SplitVecOp_STORE(StoreSDNode *N, unsigned OpNo);
   SDValue SplitVecOp_MSTORE(MaskedStoreSDNode *N, unsigned OpNo);
   SDValue SplitVecOp_MSCATTER(MaskedScatterSDNode *N, unsigned OpNo);
   SDValue SplitVecOp_MGATHER(MaskedGatherSDNode *N, unsigned OpNo);

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

class VectorLegalizer {
   SDValue LegalizeOp(SDValue Op);
 
   /// \brief Assuming the node is legal, "legalize" the results.
   SDValue TranslateLegalizeResults(SDValue Op, SDValue Result);
 
   /// \brief Implements unrolling a VSETCC.
   SDValue UnrollVSETCC(SDValue Op);
 
+  /// \brief Implements unrolling a [SU]HADD.
+  SDValue UnrollHADD(SDValue Op);
+
   /// \brief Implement expand-based legalization of vector operations.
   ///
   /// This is just a high-level routine to dispatch to specific code paths for
   /// operations to legalize them.
   SDValue Expand(SDValue Op);
 
   /// \brief Implements expansion for FNEG; falls back to UnrollVectorOp if
   /// FSUB isn't legal.
SDValue VectorLegalizer::Expand(SDValue Op) {
   case ISD::SELECT:
     return ExpandSELECT(Op);
   case ISD::UINT_TO_FP:
     return ExpandUINT_TO_FLOAT(Op);
   case ISD::FNEG:
     return ExpandFNEG(Op);
   case ISD::SETCC:
     return UnrollVSETCC(Op);
+  case ISD::UHADD:
+  case ISD::SHADD:
+    return UnrollHADD(Op);
   default:
     return DAG.UnrollVectorOp(Op.getNode());
   }
 }
 
 SDValue VectorLegalizer::ExpandSELECT(SDValue Op) {
   // Lower a select instruction where the condition is a scalar and the
   // operands are vectors. Lower this select to VSELECT and implement it
for (unsigned i = 0; i < NumElems; ++i) {
     Ops[i] = DAG.getSelect(dl, EltVT, Ops[i],
                            DAG.getConstant(APInt::getAllOnesValue
                                            (EltVT.getSizeInBits()), dl, EltVT),
                            DAG.getConstant(0, dl, EltVT));
   }
   return DAG.getNode(ISD::BUILD_VECTOR, dl, VT, Ops);
 }
 
+SDValue VectorLegalizer::UnrollHADD(SDValue Op) {
+  EVT VT = Op.getValueType();
+  unsigned NumElems = VT.getVectorNumElements();
+  EVT EltVT = VT.getVectorElementType();
+  SDValue OpVal = Op.getOperand(0);
+  EVT TmpEltVT = OpVal.getValueType().getVectorElementType();
+  SDLoc dl(Op);
+
+  SDValue LHSElem = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, OpVal,
+                                DAG.getConstant(0, dl, TLI.getVectorIdxTy()));
+  SDValue RHSElem = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, OpVal,
+                                DAG.getConstant(1, dl, TLI.getVectorIdxTy()));
+  SDValue Ops = DAG.getNode(ISD::ADD, dl, VT, LHSElem, RHSElem);
+
+  for (unsigned i = 2; i < NumElems; ++i) {
+    LHSElem = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, dl, TmpEltVT, OpVal,
+                          DAG.getConstant(i, dl, TLI.getVectorIdxTy()));
+    Ops = DAG.getNode(ISD::ADD, dl, VT, LHSElem, Ops);
+  }
+  return Ops;
+}
 }
 
 bool SelectionDAG::LegalizeVectors() {
   return VectorLegalizer(*this).Run();
 }

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::FRINT:
   case ISD::FROUND:
   case ISD::FSIN:
   case ISD::FSQRT:
   case ISD::FTRUNC:
   case ISD::SINT_TO_FP:
   case ISD::TRUNCATE:
   case ISD::UINT_TO_FP:
+  case ISD::UHADD:
+  case ISD::SHADD:
     SplitVecRes_UnaryOp(N, Lo, Hi);
     break;
 
   case ISD::ANY_EXTEND:
   case ISD::SIGN_EXTEND:
   case ISD::ZERO_EXTEND:
     SplitVecRes_ExtendOp(N, Lo, Hi);
     break;
#endif
   case ISD::SRL:
   case ISD::UREM:
   case ISD::SREM:
   case ISD::FREM:
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
+  case ISD::UABSD:
+  case ISD::SABSD:
     SplitVecRes_BinOp(N, Lo, Hi);
     break;
   case ISD::FMA:
     SplitVecRes_TernaryOp(N, Lo, Hi);
     break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.
#endif
     case ISD::CTPOP:
     case ISD::FP_EXTEND:
     case ISD::SIGN_EXTEND:
     case ISD::ZERO_EXTEND:
     case ISD::ANY_EXTEND:
     case ISD::FTRUNC:
       Res = SplitVecOp_UnaryOp(N);
       break;
+    case ISD::UHADD:
+    case ISD::SHADD:
+      Res = SplitVecOp_HADD(N);
+      break;
     }
   }
 
   // If the result is null, the sub-method took care of registering results etc.
   if (!Res.getNode()) return false;
 
   // If the result is N, the sub-method updated N in place.  Tell the legalizer
   // core about this.
SDValue DAGTypeLegalizer::SplitVecOp_BITCAST(SDNode *N) {
 
   if (TLI.isBigEndian())
     std::swap(Lo, Hi);
 
   return DAG.getNode(ISD::BITCAST, SDLoc(N), N->getValueType(0),
                      JoinIntegers(Lo, Hi));
 }
 
+SDValue DAGTypeLegalizer::SplitVecOp_HADD(SDNode *N) {
+  SDValue Lo, Hi;
+  EVT SubVT = N->getValueType(0);
+  SDValue OpVal = N->getOperand(0);
+  SDLoc dl(N);
+  GetSplitVector(OpVal, Lo, Hi);
+  Lo = DAG.getNode(N->getOpcode(), dl, SubVT, Lo);
+  Hi = DAG.getNode(N->getOpcode(), dl, SubVT, Hi);
+
+  return DAG.getNode(ISD::ADD, dl, SubVT, Lo, Hi);
+}
+
 SDValue DAGTypeLegalizer::SplitVecOp_EXTRACT_SUBVECTOR(SDNode *N) {
   // We know that the extracted result type is legal.
   EVT SubVT = N->getValueType(0);
   SDValue Idx = N->getOperand(1);
   SDLoc dl(N);
   SDValue Lo, Hi;
   GetSplitVector(N->getOperand(0), Lo, Hi);
 

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::readcyclecounter: {
     DAG.setRoot(Res.getValue(1));
     return nullptr;
   }
   case Intrinsic::bswap:
     setValue(&I, DAG.getNode(ISD::BSWAP, sdl,
                              getValue(I.getArgOperand(0)).getValueType(),
                              getValue(I.getArgOperand(0))));
     return nullptr;
+  case Intrinsic::uabsdiff:
+    setValue(&I, DAG.getNode(ISD::UABSD, sdl,
+                             getValue(I.getArgOperand(0)).getValueType(),
+                             getValue(I.getArgOperand(0)),
+                             getValue(I.getArgOperand(1))));
+    return nullptr;
+  case Intrinsic::sabsdiff:
+    setValue(&I, DAG.getNode(ISD::SABSD, sdl,
+                             getValue(I.getArgOperand(0)).getValueType(),
+                             getValue(I.getArgOperand(0)),
+                             getValue(I.getArgOperand(1))));
+    return nullptr;
+  case Intrinsic::uhadd:
+    setValue(&I, DAG.getNode(ISD::UHADD, sdl,
+                             TLI.getValueType(I.getType()),
+                             getValue(I.getArgOperand(0))));
+    return nullptr;
+  case Intrinsic::shadd:
+    setValue(&I, DAG.getNode(ISD::SHADD, sdl,
+                             TLI.getValueType(I.getType()),
+                             getValue(I.getArgOperand(0))));
+    return nullptr;
   case Intrinsic::cttz: {
     SDValue Arg = getValue(I.getArgOperand(0));
     ConstantInt *CI = cast<ConstantInt>(I.getArgOperand(1));
     EVT Ty = Arg.getValueType();
     setValue(&I, DAG.getNode(CI->isZero() ? ISD::CTTZ : ISD::CTTZ_ZERO_UNDEF,
                              sdl, Ty, Arg));
     return nullptr;
   }

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::FRINT:                      return "frint";
   case ISD::FNEARBYINT:                 return "fnearbyint";
   case ISD::FROUND:                     return "fround";
   case ISD::FEXP:                       return "fexp";
   case ISD::FEXP2:                      return "fexp2";
   case ISD::FLOG:                       return "flog";
   case ISD::FLOG2:                      return "flog2";
   case ISD::FLOG10:                     return "flog10";
+  case ISD::UHADD:                      return "uhadd";
+  case ISD::SHADD:                      return "shadd";
 
   // Binary operators
   case ISD::ADD:                        return "add";
   case ISD::SUB:                        return "sub";
   case ISD::MUL:                        return "mul";
   case ISD::MULHU:                      return "mulhu";
   case ISD::MULHS:                      return "mulhs";
   case ISD::SDIV:                       return "sdiv";
#endif
   case ISD::USUBO:                      return "usubo";
   case ISD::SMULO:                      return "smulo";
   case ISD::UMULO:                      return "umulo";
   case ISD::SUBC:                       return "subc";
   case ISD::SUBE:                       return "sube";
   case ISD::SHL_PARTS:                  return "shl_parts";
   case ISD::SRA_PARTS:                  return "sra_parts";
   case ISD::SRL_PARTS:                  return "srl_parts";
+  case ISD::UABSD:                       return "uabsd";
+  case ISD::SABSD:                       return "sabsd";
 
   // Conversion operators.
   case ISD::SIGN_EXTEND:                return "sign_extend";
   case ISD::ZERO_EXTEND:                return "zero_extend";
   case ISD::ANY_EXTEND:                 return "any_extend";
   case ISD::SIGN_EXTEND_INREG:          return "sign_extend_inreg";
   case ISD::ANY_EXTEND_VECTOR_INREG:    return "any_extend_vector_inreg";
   case ISD::SIGN_EXTEND_VECTOR_INREG:   return "sign_extend_vector_inreg";

lib/CodeGen/TargetLoweringBase.cpp

for (MVT VT : MVT::all_valuetypes()) {
 
     // Overflow operations default to expand
     setOperationAction(ISD::SADDO, VT, Expand);
     setOperationAction(ISD::SSUBO, VT, Expand);
     setOperationAction(ISD::UADDO, VT, Expand);
     setOperationAction(ISD::USUBO, VT, Expand);
     setOperationAction(ISD::SMULO, VT, Expand);
     setOperationAction(ISD::UMULO, VT, Expand);
+    setOperationAction(ISD::UABSD, VT, Expand);
+    setOperationAction(ISD::SABSD, VT, Expand);
+    setOperationAction(ISD::UHADD, VT, Expand);
+    setOperationAction(ISD::SHADD, VT, Expand);
 
     // These library functions default to expand.
     setOperationAction(ISD::FROUND, VT, Expand);
 
     // These operations default to expand for vector types.
     if (VT.isVector()) {
       setOperationAction(ISD::FCOPYSIGN, VT, Expand);
       setOperationAction(ISD::ANY_EXTEND_VECTOR_INREG, VT, Expand);

lib/Target/X86/X86ISelDAGToDAG.cpp

SDNode *X86DAGToDAGISel::Select(SDNode *Node) {
   if (Node->isMachineOpcode()) {
     DEBUG(dbgs() << "== ";  Node->dump(CurDAG); dbgs() << '\n');
     Node->setNodeId(-1);
     return nullptr;   // Already selected.
   }
 
   switch (Opcode) {
   default: break;
+  case X86ISD::USAD: {
+    SDNode *New;
+    SDValue Ops[] = {Node->getOperand(0), Node->getOperand(1)};
+
+    if (Subtarget->hasAVX() || Subtarget->hasSSE1() || Subtarget->hasSSE2())
+      New =
+          CurDAG->getMachineNode(X86::PSADBWrr, dl, Node->getValueType(0), Ops);
+    else
+      New = Node;
+    return New;
+  }
   case ISD::INTRINSIC_W_CHAIN: {
     unsigned IntNo = cast<ConstantSDNode>(Node->getOperand(1))->getZExtValue();
     switch (IntNo) {
     default: break;
     case Intrinsic::x86_avx2_gather_d_pd:
     case Intrinsic::x86_avx2_gather_d_pd_256:
     case Intrinsic::x86_avx2_gather_q_pd:
     case Intrinsic::x86_avx2_gather_q_pd_256:

lib/Target/X86/X86ISelLowering.h

enum NodeType : unsigned {
       // Start the numbering where the builtin ops leave off.
       FIRST_NUMBER = ISD::BUILTIN_OP_END,
 
       /// Bit scan forward.
       BSF,
       /// Bit scan reverse.
       BSR,
 
+      /// Sum of absolute differences
+      USAD,
+
       /// Double shift instructions. These correspond to
       /// X86::SHLDxx and X86::SHRDxx instructions.
       SHLD,
       SHRD,
 
       /// Bitwise logical AND of floating point values. This corresponds
       /// to X86::ANDPS or X86::ANDPD.
       FAND,

lib/Target/X86/X86ISelLowering.cpp

X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
   // SETOEQ and SETUNE require checking two conditions.
   setCondCodeAction(ISD::SETOEQ, MVT::f32, Expand);
   setCondCodeAction(ISD::SETOEQ, MVT::f64, Expand);
   setCondCodeAction(ISD::SETOEQ, MVT::f80, Expand);
   setCondCodeAction(ISD::SETUNE, MVT::f32, Expand);
   setCondCodeAction(ISD::SETUNE, MVT::f64, Expand);
   setCondCodeAction(ISD::SETUNE, MVT::f80, Expand);
 
+  setOperationAction(ISD::UHADD       , MVT::i8   , Promote);
+  setOperationAction(ISD::SHADD       , MVT::i8   , Promote);
+
   // Promote all UINT_TO_FP to larger SINT_TO_FP's, as X86 doesn't have this
   // operation.
   setOperationAction(ISD::UINT_TO_FP       , MVT::i1   , Promote);
   setOperationAction(ISD::UINT_TO_FP       , MVT::i8   , Promote);
   setOperationAction(ISD::UINT_TO_FP       , MVT::i16  , Promote);
 
   if (Subtarget->is64Bit()) {
     setOperationAction(ISD::UINT_TO_FP     , MVT::i32  , Promote);
for (MVT VT : MVT::vector_valuetypes()) {
     setOperationAction(ISD::CTLZ, VT, Expand);
     setOperationAction(ISD::CTLZ_ZERO_UNDEF, VT, Expand);
     setOperationAction(ISD::SHL, VT, Expand);
     setOperationAction(ISD::SRA, VT, Expand);
     setOperationAction(ISD::SRL, VT, Expand);
     setOperationAction(ISD::ROTL, VT, Expand);
     setOperationAction(ISD::ROTR, VT, Expand);
     setOperationAction(ISD::BSWAP, VT, Expand);
+    setOperationAction(ISD::UABSD, VT, Expand);
+    setOperationAction(ISD::SABSD, VT, Expand);
+    setOperationAction(ISD::UHADD, VT, Expand);
+    setOperationAction(ISD::SHADD, VT, Expand);
     setOperationAction(ISD::SETCC, VT, Expand);
     setOperationAction(ISD::FLOG, VT, Expand);
     setOperationAction(ISD::FLOG2, VT, Expand);
     setOperationAction(ISD::FLOG10, VT, Expand);
     setOperationAction(ISD::FEXP, VT, Expand);
     setOperationAction(ISD::FEXP2, VT, Expand);
     setOperationAction(ISD::FP_TO_UINT, VT, Expand);
     setOperationAction(ISD::FP_TO_SINT, VT, Expand);
if (Subtarget->isTargetWin64()) {
     setOperationAction(ISD::UDIV, MVT::i128, Custom);
     setOperationAction(ISD::SREM, MVT::i128, Custom);
     setOperationAction(ISD::UREM, MVT::i128, Custom);
     setOperationAction(ISD::SDIVREM, MVT::i128, Custom);
     setOperationAction(ISD::UDIVREM, MVT::i128, Custom);
   }
 
   // We have target-specific dag combine patterns for the following nodes:
+  setTargetDAGCombine(ISD::UHADD);
+  setTargetDAGCombine(ISD::SHADD);
   setTargetDAGCombine(ISD::VECTOR_SHUFFLE);
   setTargetDAGCombine(ISD::EXTRACT_VECTOR_ELT);
   setTargetDAGCombine(ISD::BITCAST);
   setTargetDAGCombine(ISD::VSELECT);
   setTargetDAGCombine(ISD::SELECT);
   setTargetDAGCombine(ISD::SHL);
   setTargetDAGCombine(ISD::SRA);
   setTargetDAGCombine(ISD::SRL);
SDValue X86TargetLowering::LowerVSELECT(SDValue Op, SelectionDAG &DAG) const {
   case MVT::v8i16:
   case MVT::v16i16:
     // AVX-512 BWI and VLX features support VSELECT with i16 elements.
     if (Subtarget->hasBWI() && Subtarget->hasVLX())
       return Op;
 
     // FIXME: We should custom lower this by fixing the condition and using i8
     // blends.
+    if(Op.getSimpleValueType().SimpleTy == MVT::v8i16) {
+      SDValue V0, V1, V2;
+      SDLoc DL(Op);
+      V0 = Op.getOperand(0);
+      V1 = Op.getOperand(1);
+      V2 = Op.getOperand(2);
+      V0 = DAG.getNode(ISD::BITCAST, DL, MVT::v16i8, V0);
+      V1 = DAG.getNode(ISD::BITCAST, DL, MVT::v16i8, V1);
+      V2 = DAG.getNode(ISD::BITCAST, DL, MVT::v16i8, V2);
+      return DAG.getNode(ISD::VSELECT, DL, MVT::v16i8, V0, V1, V2);
+    }
     return SDValue();
   }
 }
 
 static SDValue LowerEXTRACT_VECTOR_ELT_SSE4(SDValue Op, SelectionDAG &DAG) {
   MVT VT = Op.getSimpleValueType();
   SDLoc dl(Op);
 
const char *X86TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case X86ISD::EXP2:               return "X86ISD::EXP2";
   case X86ISD::RSQRT28:            return "X86ISD::RSQRT28";
   case X86ISD::FADD_RND:           return "X86ISD::FADD_RND";
   case X86ISD::FSUB_RND:           return "X86ISD::FSUB_RND";
   case X86ISD::FMUL_RND:           return "X86ISD::FMUL_RND";
   case X86ISD::FDIV_RND:           return "X86ISD::FDIV_RND";
   case X86ISD::ADDS:               return "X86ISD::ADDS";
   case X86ISD::SUBS:               return "X86ISD::SUBS";
+  case X86ISD::USAD:               return "X86ISD::USAD";
   }
   return nullptr;
 }
 
 // isLegalAddressingMode - Return true if the addressing mode represented
 // by AM is legal for this target, for a load/store of the specified type.
 bool X86TargetLowering::isLegalAddressingMode(const AddrMode &AM,
                                               Type *Ty) const {
if (SetCC0.getOpcode() != X86ISD::SETCC ||
     return false;
 
   CC0 = (X86::CondCode)SetCC0->getConstantOperandVal(0);
   CC1 = (X86::CondCode)SetCC1->getConstantOperandVal(0);
   Flags = SetCC0->getOperand(1);
   return true;
 }
 
+static SDValue PerformHADDCombine(SDNode *N, SelectionDAG &DAG,
+                                  const X86Subtarget *Subtarget) {
+  SDLoc DL(N);
+  EVT DestVT = N->getValueType(0);
+  SDValue absd = N->getOperand(0);
+
+  if (absd->getOpcode() != ISD::UABSD)
+    return SDValue();
+
+  SDValue arg1 = absd->getOperand(0);
+  SDValue arg2 = absd->getOperand(1);
+  SDValue result;
+
+  if ((arg1.getValueType() != MVT::v8i8) && (arg1.getValueType() != MVT::v16i8))
+    return SDValue();
+
+  if (Subtarget->hasSSE1() || Subtarget->hasSSE2() || Subtarget->hasAVX()) {
+    if (arg1.getValueType() == MVT::v16i8) {
+      SDValue sad = DAG.getNode(X86ISD::USAD, DL, MVT::v2i64, arg1, arg2);
+      EVT VecIdxTy = DAG.getTargetLoweringInfo().getVectorIdxTy();
+      SDValue BottomHalf = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i64,
+                                       sad, DAG.getConstant(0, DL, VecIdxTy));
+      SDValue TopHalf = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i64, sad,
+                                    DAG.getConstant(1, DL, VecIdxTy));
+      BottomHalf = DAG.getNode(ISD::TRUNCATE, DL, DestVT, BottomHalf);
+      TopHalf = DAG.getNode(ISD::TRUNCATE, DL, DestVT, TopHalf);
+      result = DAG.getNode(ISD::ADD, DL, DestVT, TopHalf, BottomHalf);
+      return result;
+     } else if (arg1.getValueType() == MVT::v8i8) {
+      EVT VecIdxTy = DAG.getTargetLoweringInfo().getVectorIdxTy();
+      SDValue V0 = DAG.getUNDEF(MVT::v8i8);
+      SDValue Op0 = DAG.getNode(ISD::CONCAT_VECTORS, DL, MVT::v16i8, V0, arg1);
+      SDValue Op1 = DAG.getNode(ISD::CONCAT_VECTORS, DL, MVT::v16i8, V0, arg2);
+      SDValue Ops[] = {Op0, Op1};
+      SDValue SAD = DAG.getNode(X86ISD::USAD, DL, MVT::v2i64, Ops);
+      SDValue BottomHalf = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL, MVT::i64,
+                                       SAD, DAG.getConstant(0, DL, VecIdxTy));
+      result = DAG.getNode(ISD::TRUNCATE, DL, DestVT, BottomHalf);
+      return result;
+    }
+  }
+  return SDValue();
+}
+
 /// Optimize X86ISD::CMOV [LHS, RHS, CONDCODE (e.g. X86::COND_NE), CONDVAL]
 static SDValue PerformCMOVCombine(SDNode *N, SelectionDAG &DAG,
                                   TargetLowering::DAGCombinerInfo &DCI,
                                   const X86Subtarget *Subtarget) {
   SDLoc DL(N);
 
   // If the flag operand isn't dead, don't touch this CMOV.
   if (N->getNumValues() == 2 && !SDValue(N, 1).use_empty())
SDValue X86TargetLowering::PerformDAGCombine(SDNode *N,
   default: break;
   case ISD::EXTRACT_VECTOR_ELT:
     return PerformEXTRACT_VECTOR_ELTCombine(N, DAG, DCI);
   case ISD::VSELECT:
   case ISD::SELECT:
   case X86ISD::SHRUNKBLEND:
     return PerformSELECTCombine(N, DAG, DCI, Subtarget);
   case ISD::BITCAST:        return PerformBITCASTCombine(N, DAG);
+  case ISD::UHADD:
+    return PerformHADDCombine(N, DAG, Subtarget);
   case X86ISD::CMOV:        return PerformCMOVCombine(N, DAG, DCI, Subtarget);
   case ISD::ADD:            return PerformAddCombine(N, DAG, Subtarget);
   case ISD::SUB:            return PerformSubCombine(N, DAG, Subtarget);
   case X86ISD::ADC:         return PerformADCCombine(N, DAG, DCI);
   case ISD::MUL:            return PerformMulCombine(N, DAG, DCI);
   case ISD::SHL:
   case ISD::SRA:
   case ISD::SRL:            return PerformShiftCombine(N, DAG, DCI, Subtarget);

test/CodeGen/X86/absdiff_intrinsic.ll

+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  -mattr=+avx < %s | FileCheck %s -check-prefix=CHECK
+
+declare <4 x i8> @llvm.uabsdiff.v4i8.v4i8.v4i8(<4 x i8>, <4 x i8>)
+
+define <4 x i8> @test_uabsdiff_v4i8_expand(<4 x i8> %a1, <4 x i8> %a2) {
+; CHECK-LABEL: test_uabsdiff_v4i8_expand
+; CHECK:             vmovdqa .LCPI0_0(%rip), %xmm2   # xmm2 = [255,255,255,255]
+; CHECK-NEXT:        vpand   %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vpand   %xmm2, %xmm0, %xmm0
+; CHECK-NEXT:        vpsubd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT:        vpsubd  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT:        vpcmpgtd        %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vblendvps       %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT:        retq
+
+  %1 = call <4 x i8> @llvm.uabsdiff.v4i8.v4i8.v4i8(<4 x i8> %a1, <4 x i8> %a2)
+  ret <4 x i8> %1
+}
+
+declare <4 x i8> @llvm.sabsdiff.v4i8.v4i8.v4i8(<4 x i8>, <4 x i8>)
+
+define <4 x i8> @test_sabsdiff_v4i8_expand(<4 x i8> %a1, <4 x i8> %a2) {
+; CHECK-LABEL: test_sabsdiff_v4i8_expand
+; CHECK:      vpslld  $24, %xmm1, %xmm1
+; CHECK-NEXT: vpsrad  $24, %xmm1, %xmm1
+; CHECK-NEXT: vpslld  $24, %xmm0, %xmm0
+; CHECK-NEXT: vpsrad  $24, %xmm0, %xmm0
+; CHECK-NEXT: vpsubd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT: vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT: vpsubd  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT: vpcmpgtd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT: vblendvps  %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT: retq
+
+  %1 = call <4 x i8> @llvm.sabsdiff.v4i8.v4i8.v4i8(<4 x i8> %a1, <4 x i8> %a2)
+  ret <4 x i8> %1
+}
+
+declare <4 x i32> @llvm.sabsdiff.v4i32.v4i8.v4i8(<4 x i8>, <4 x i8>)
+
+define <4 x i32> @test_sabsdiff_promote_intres(<4 x i8> %a1, <4 x i8> %a2) {
+; CHECK-LABEL: test_sabsdiff_promote_intres
+; CHECK:      vpslld  $24, %xmm1, %xmm1
+; CHECK-NEXT: vpsrad  $24, %xmm1, %xmm1
+; CHECK-NEXT: vpslld  $24, %xmm0, %xmm0
+; CHECK-NEXT: vpsrad  $24, %xmm0, %xmm0
+; CHECK-NEXT: vpsubd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT: vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT: vpsubd  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT: vpcmpgtd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT: vblendvps  %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT: retq
+  %1 = call <4 x i32> @llvm.sabsdiff.v4i32.v4i8.v4i8(<4 x i8> %a1, <4 x i8> %a2)
+  ret <4 x i32> %1
+}
+
+declare <8 x i8> @llvm.sabsdiff.v8i8.v8i8.v8i8(<8 x i8>, <8 x i8>)
+
+define <8 x i8> @test_sabsdiff_v8i8_expand(<8 x i8> %a1, <8 x i8> %a2) {
+; CHECK-LABEL: test_sabsdiff_v8i8_expand
+; CHECK:      vpsllw  $8, %xmm1, %xmm1
+; CHECK-NEXT: vpsraw  $8, %xmm1, %xmm1
+; CHECK-NEXT: vpsllw  $8, %xmm0, %xmm0
+; CHECK-NEXT: vpsraw  $8, %xmm0, %xmm0
+; CHECK-NEXT: vpsubw  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT: vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT: vpsubw  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT: vpcmpgtw        %xmm2, %xmm1, %xmm1
+; CHECK-NEXT: vpblendvb       %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT: retq
+  %1 = call <8 x i8> @llvm.sabsdiff.v8i8.v8i8.v8i8(<8 x i8> %a1, <8 x i8> %a2)
+  ret <8 x i8> %1
+}
+
+declare <16 x i8> @llvm.uabsdiff.v16i8.v16i8.v16i8(<16 x i8>, <16 x i8>)
+declare i32 @llvm.uhadd.i32.v16i8(<16 x i8>)
+
+define i32 @test_v16i8_uhadd_combine(<16 x i8> %a1, <16 x i8> %a2) {
+; CHECK-LABEL: test_v16i8_uhadd_combine
+; CHECK:      psadbw  %xmm1, %xmm0
+; CHECK-NEXT: vmovq   %xmm0, %rcx
+; CHECK-NEXT: vpextrq $1, %xmm0, %rax
+; CHECK-NEXT: addl    %ecx, %eax
+; CHECK: retq
+  %1 = call <16 x i8> @llvm.uabsdiff.v16i8.v16i8.v16i8(<16 x i8> %a1, <16 x i8> %a2)
+  %2 = call i32 @llvm.uhadd.i32.v16i8(<16 x i8> %1)
+  ret i32 %2
+}
+
+declare <4 x i32> @llvm.sabsdiff.v4i32.v2i32.v2i32(<2 x i32>, <2 x i32>)
+
+define <4 x i32> @test_sabsdiff_v2i32_expand(<2 x i32> %a1, <2 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v2i32_expand
+; CHECK:        vpextrq $1, %xmm1, %rax
+; CHECK-NEXT:   cltq
+; CHECK-NEXT:   vmovq   %rax, %xmm2
+; CHECK-NEXT:   vmovq   %xmm1, %rax
+; CHECK-NEXT:   cltq
+; CHECK-NEXT:   vmovq   %rax, %xmm1
+; CHECK-NEXT:   vpunpcklqdq     %xmm2, %xmm1, %xmm1 # xmm1 = xmm1[0],xmm2[0]
+; CHECK-NEXT:   vpextrq $1, %xmm0, %rax
+; CHECK-NEXT:   cltq
+; CHECK-NEXT:   vmovq   %rax, %xmm2
+; CHECK-NEXT:   vmovq   %xmm0, %rax
+; CHECK-NEXT:   cltq
+; CHECK-NEXT:   vmovq   %rax, %xmm0
+; CHECK-NEXT:   vpunpcklqdq     %xmm2, %xmm0, %xmm0 # xmm0 = xmm0[0],xmm2[0]
+; CHECK-NEXT:   vpsubq  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:   vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT:   vpsubq  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT:   vpcmpgtq        %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:   vblendvpd       %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT:   vpshufd $232, %xmm0, %xmm0      # xmm0 = xmm0[0,2,2,3]
+; CHECK-NEXT:   retq
+  %1 = call <4 x i32> @llvm.sabsdiff.v4i32.v2i32.v2i32(<2 x i32> %a1, <2 x i32> %a2)
+  ret <4 x i32> %1
+}
+
+declare <16 x i32> @llvm.sabsdiff.v16i32.v16i32.v16i32(<16 x i32>, <16 x i32>)
+
+define <16 x i32> @test_sabsdiff_v16i32_expand(<16 x i32> %a1, <16 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v16i32_expand
+; CHECK:             vextractf128    $1, %ymm2, %xmm4
+; CHECK-NEXT:        vextractf128    $1, %ymm0, %xmm5
+; CHECK-NEXT:        vpsubd  %xmm4, %xmm5, %xmm4
+; CHECK-NEXT:        vpsubd  %xmm2, %xmm0, %xmm0
+; CHECK-NEXT:        vinsertf128     $1, %xmm4, %ymm0, %ymm2
+; CHECK-NEXT:        vpxor   %xmm5, %xmm5, %xmm5
+; CHECK-NEXT:        vpsubd  %xmm4, %xmm5, %xmm4
+; CHECK-NEXT:        vpsubd  %xmm0, %xmm5, %xmm0
+; CHECK-NEXT:        vinsertf128     $1, %xmm4, %ymm0, %ymm6
+; CHECK-NEXT:        vpcmpgtd        %xmm4, %xmm5, %xmm4
+; CHECK-NEXT:        vpcmpgtd        %xmm0, %xmm5, %xmm0
+; CHECK-NEXT:        vinsertf128     $1, %xmm4, %ymm0, %ymm0
+; CHECK-NEXT:        vblendvps       %ymm0, %ymm2, %ymm6, %ymm0
+; CHECK-NEXT:        vextractf128    $1, %ymm3, %xmm2
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm4
+; CHECK-NEXT:        vpsubd  %xmm2, %xmm4, %xmm2
+; CHECK-NEXT:        vpsubd  %xmm3, %xmm1, %xmm1
+; CHECK-NEXT:        vinsertf128     $1, %xmm2, %ymm1, %ymm3
+; CHECK-NEXT:        vpsubd  %xmm2, %xmm5, %xmm2
+; CHECK-NEXT:        vpsubd  %xmm1, %xmm5, %xmm1
+; CHECK-NEXT:        vinsertf128     $1, %xmm2, %ymm1, %ymm4
+; CHECK-NEXT:        vpcmpgtd        %xmm2, %xmm5, %xmm2
+; CHECK-NEXT:        vpcmpgtd        %xmm1, %xmm5, %xmm1
+; CHECK-NEXT:        vinsertf128     $1, %xmm2, %ymm1, %ymm1
+; CHECK-NEXT:        vblendvps       %ymm1, %ymm3, %ymm4, %ymm1
+; CHECK-NEXT:        retq
+  %1 = call <16 x i32> @llvm.sabsdiff.v16i32.v16i32.v16i32(<16 x i32> %a1, <16 x i32> %a2)
+  ret <16 x i32> %1
+}
+
+declare <4 x i32> @llvm.sabsdiff.v4i32.v4i32.v4i32(<4 x i32>, <4 x i32>)
+
+define <4 x i32> @test_sabsdiff_v4i32_expand(<4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v4i32_expand
+; CHECK:             vpsubd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT:        vpsubd  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT:        vpcmpgtd        %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vblendvps       %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <4 x i32> @llvm.sabsdiff.v4i32.v4i32.v4i32(<4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %1
+}
+
+declare <4 x i32> @llvm.uabsdiff.v4i32.v4i32.v4i32(<4 x i32>, <4 x i32>)
+
+define <4 x i32> @test_uabsdiff_v4i32_expand(<4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: test_uabsdiff_v4i32_expand
+; CHECK:             vpsubd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpxor   %xmm1, %xmm1, %xmm1
+; CHECK-NEXT:        vpsubd  %xmm0, %xmm1, %xmm2
+; CHECK-NEXT:        vpcmpgtd        %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vblendvps       %xmm1, %xmm0, %xmm2, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <4 x i32> @llvm.uabsdiff.v4i32.v4i32.v4i32(<4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %1
+}
+
+;define <4 x i32> @test_sabsdiff_expand(<4 x i32> %a1, <4 x i32> %a2) {
+;  %sub = sub nsw <4 x i32> %a1, %a2
+;  %ispos = icmp sgt <4 x i32> %sub, <i32 -1, i32 -1, i32 -1, i32 -1>
+;  %neg = sub nsw <4 x i32> zeroinitializer, %sub
+;  %1 = select <4 x i1> %ispos, <4 x i32> %sub, <4 x i32> %neg
+;  ret <4 x i32> %1
+;}
+

test/CodeGen/X86/hadd_intrinsic.ll

+; RUN: llc -mtriple=x86_64-unknown-linux-gnu -mattr=+avx < %s | FileCheck %s -check-prefix=CHECK
+
+declare i32 @llvm.uhadd.i32.v4i8(<4 x i8>)
+
+define i32 @test1_uhadd_intrinsic(<4 x i8> %a1) {
+; CHECK-LABEL: test1_uhadd_intrinsic
+; CHECK:             vpand    .LCPI0_0(%rip), %xmm0
+; CHECK-NEXT:        vpshufd  $78, %xmm0, %xmm1       # xmm1 = xmm0[2,3,0,1]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpshufd  $237, %xmm0, %xmm1      # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd    %xmm0, %eax
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.uhadd.i32.v4i8(<4 x i8> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.shadd.i32.v4i8(<4 x i8>)
+
+define i32 @test1_shadd_intrinsic(<4 x i8> %a1) {
+; CHECK-LABEL: test1_shadd_intrinsic
+; CHECK:             vpslld  $24, %xmm0, %xmm0
+; CHECK-NEXT:        vpsrad   $24, %xmm0, %xmm0
+; CHECK-NEXT:        vpshufd  $78, %xmm0, %xmm1       # xmm1 = xmm0[2,3,0,1]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpshufd  $237, %xmm0, %xmm1      # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd    %xmm0, %eax
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.shadd.i32.v4i8(<4 x i8> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.uhadd.i32.v4i32(<4 x i32>)
+
+define i32 @test2_uhadd_intrinsic(<4 x i32> %a1) {
+; CHECK-LABEL: test2_uhadd_intrinsic
+; CHECK:             vpshufd  $78, %xmm0, %xmm1       # xmm1 = xmm0[2,3,0,1]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpshufd  $237, %xmm0, %xmm1      # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd    %xmm0, %eax
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.uhadd.i32.v4i32(<4 x i32> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.shadd.i32.v4i32(<4 x i32>)
+
+define i32 @test2_shadd_intrinsic(<4 x i32> %a1) {
+; CHECK-LABEL: test2_shadd_intrinsic
+; CHECK:             vpshufd  $78, %xmm0, %xmm1       # xmm1 = xmm0[2,3,0,1]
+; CHECK-NEXT:        vpaddd   %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vpshufd  $237, %xmm0, %xmm1      # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd    %xmm0, %eax
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.shadd.i32.v4i32(<4 x i32> %a1)
+  ret i32 %1
+}
+
+
+declare i32 @llvm.uhadd.i32.v8i32(<8 x i32>)
+
+define i32 @test3_uhadd_intrinsic(<8 x i32> %a1) {
+; CHECK-LABEL: test3_uhadd_intrinsic
+; CHECK:             vperm2f128      $1, %ymm0, %ymm0, %ymm1 # ymm1 = ymm0[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm2
+; CHECK-NEXT:        vextractf128    $1, %ymm0, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm3, %xmm2
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vunpckhpd       %xmm2, %xmm0, %xmm1 # xmm1 = xmm0[1],xmm2[1]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm0, %xmm1 # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd   %xmm0, %eax
+; CHECK-NEXT:        vzeroupper
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.uhadd.i32.v8i32(<8 x i32> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.shadd.i32.v8i32(<8 x i32>)
+
+define i32 @test3_shadd_intrinsic(<8 x i32> %a1) {
+; CHECK-LABEL: test3_shadd_intrinsic
+; CHECK:             vperm2f128      $1, %ymm0, %ymm0, %ymm1 # ymm1 = ymm0[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm2
+; CHECK-NEXT:        vextractf128    $1, %ymm0, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm3, %xmm2
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vunpckhpd       %xmm2, %xmm0, %xmm1 # xmm1 = xmm0[1],xmm2[1]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm0, %xmm1 # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd   %xmm0, %eax
+; CHECK-NEXT:        vzeroupper
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.shadd.i32.v8i32(<8 x i32> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.uhadd.i32.v16i32(<16 x i32>)
+
+define i32 @test4_uhadd_intrinsic(<16 x i32> %a1) {
+; CHECK-LABEL: test4_uhadd_intrinsic
+; CHECK:             vperm2f128      $1, %ymm0, %ymm1, %ymm2 # ymm2 = ymm1[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm2, %xmm3
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm4
+; CHECK-NEXT:        vpaddd  %xmm3, %xmm4, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vunpckhpd       %xmm3, %xmm1, %xmm2 # xmm2 = xmm1[1],xmm3[1]
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm1, %xmm2 # xmm2 = xmm1[1,3],xmm0[2,3]
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vmovd   %xmm1, %ecx
+; CHECK-NEXT:        vperm2f128      $1, %ymm0, %ymm0, %ymm1 # ymm1 = ymm0[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm2
+; CHECK-NEXT:        vextractf128    $1, %ymm0, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm3, %xmm2
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vunpckhpd       %xmm2, %xmm0, %xmm1 # xmm1 = xmm0[1],xmm2[1]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm0, %xmm1 # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd   %xmm0, %eax
+; CHECK-NEXT:        addl    %ecx, %eax
+; CHECK-NEXT:        vzeroupper
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.uhadd.i32.v16i32(<16 x i32> %a1)
+  ret i32 %1
+}
+
+declare i32 @llvm.shadd.i32.v16i32(<16 x i32>)
+
+define i32 @test4_shadd_intrinsic(<16 x i32> %a1) {
+; CHECK-LABEL: test4_shadd_intrinsic
+; CHECK:             vperm2f128      $1, %ymm0, %ymm1, %ymm2 # ymm2 = ymm1[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm2, %xmm3
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm4
+; CHECK-NEXT:        vpaddd  %xmm3, %xmm4, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vunpckhpd       %xmm3, %xmm1, %xmm2 # xmm2 = xmm1[1],xmm3[1]
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm1, %xmm2 # xmm2 = xmm1[1,3],xmm0[2,3]
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm1, %xmm1
+; CHECK-NEXT:        vmovd   %xmm1, %ecx
+; CHECK-NEXT:        vperm2f128      $1, %ymm0, %ymm0, %ymm1 # ymm1 = ymm0[2,3,0,1]
+; CHECK-NEXT:        vextractf128    $1, %ymm1, %xmm2
+; CHECK-NEXT:        vextractf128    $1, %ymm0, %xmm3
+; CHECK-NEXT:        vpaddd  %xmm2, %xmm3, %xmm2
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vunpckhpd       %xmm2, %xmm0, %xmm1 # xmm1 = xmm0[1],xmm2[1]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vshufps $237, %xmm0, %xmm0, %xmm1 # xmm1 = xmm0[1,3,2,3]
+; CHECK-NEXT:        vpaddd  %xmm1, %xmm0, %xmm0
+; CHECK-NEXT:        vmovd   %xmm0, %eax
+; CHECK-NEXT:        addl    %ecx, %eax
+; CHECK-NEXT:        vzeroupper
+; CHECK-NEXT:        retq
+  %1 = call i32 @llvm.shadd.i32.v16i32(<16 x i32> %a1)
+  ret i32 %1
+}