[Codegen] Added intrinsics 'absdiff' and corresponding SDNodes for absolute difference operation

[• ashahid]

Summary

This adds new intrinsics "*absdiff" for absolute difference ops to facilitate efficient code generation for "sum of absolute differences" operation.
The patch also contains the introduction of corresponding SDNodes and basic legalization support.Sanity of the generated code is tested on X86.

This is 1st of the three patches.

[jmolloy]

Apart from the comments I've given, this looks fine by me but please can you get the OK of someone like Hal or Chandler before committing, as this changes LangRef?

docs/LangRef.rst

10355

Slight re-Englishing: "... used during *the* code generation...", "They are generated by compiler passes such as the Loop and SLP vectorizers."

I'd also suggest adding an explanation of what that means for users, something like "The expectation is that frontends should not need to generate these intrinsics themselves."

include/llvm/IR/Intrinsics.td

610

These should be "LLVMMatchType<0>", not llvm_anyvector_ty. This enforces all arguments are of the same type as the result.

[• ashahid]

Hi James,

Thanks for your comments, will update the patch accordingly.

Hal / Chandler,
Please have a look, awaiting your comment if any.

Regards,
Shahid

[bruno]

Hi,

Very nice. I wonder what are your plans regarding the ISD node for the horizontal sum part? It would be great if it's done in a way that we can also use it in the expansion of vector population count, as discussed in http://lists.cs.uiuc.edu/pipermail/llvm-commits/Week-of-Mon-20150601/279536.html

Thanks.

[• ashahid]

Hi Bruno,

Pls find the link to "horizontal sum" patch below.

http://reviews.llvm.org/D10964

Regards,
Shahid

[hfinkel]

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2699 ↗

(On Diff #28854)

Why are you adding this in this patch?

2935 ↗

(On Diff #28854)

Both ISD::UABSDIFF and ISD::SABSDIFF are the same?

[• ashahid]

Hi Hal,

Thanks for the comments, response Inlined.

Regards,
Shahid

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2699 ↗

(On Diff #28854)

In absence of this, instruction selection was hitting "LLVM ERROR: Cannot select: vselect".

2935 ↗

(On Diff #28854)

Correct me if I am wrong, I think with respect to expansion here both are same as the signedness of 'absolute diff' operation depends on the 1st 'SUB' operation.

[hfinkel]

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2699 ↗

(On Diff #28854)

If we're just plain missing code to expand VSELECT, I'd think you should be able to trigger that with IR for some target, and test it, directly. I'd prefer this be a separate commit if possible.

2935 ↗

(On Diff #28854)

Ah, I see what you mean. The difference at the IR level is that SABSDIFF has NSW on the SUBs, and the unsigned version does not. The nice thing is that we can now do this at the SDAG level too..

SDNodeFlags Flags;
Flags.setNoSignedWrap(true);

Tmp1 = DAG.getNode(ISD::SUB, dl, VT, Tmp2, Tmp3, &Flags);

(and similar for the other nodes)

[jmolloy]

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2935 ↗

(On Diff #28854)

I think ISD::SETLT should also change to ISD::SETULT depending on signedness.

[• ashahid]

lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

2699 ↗

(On Diff #28854)

ExpandVselect is present in LegalizeVecctorOps, so triggering just VSELECT works fine.However our case is in fact "Expansion of VSELECT during expansion of ABSDIFF" which is why this error is being reported.

2935 ↗

(On Diff #28854)

Oh nice, so NSW and frnds are diving deep :)

[• ashahid]

The updated revision has slight document change.Another change is to propagate NSW flag to appropriate SDAG.
It also has a change for intrinsic declaration.

[• ashahid]

The expansion of *ABSDIFF node has been moved to LegalizeVectorOps from LegalizeDAG as this is a vector operation.

[jmolloy]

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

725

This is unnecessarily complex. You've duplicated most of the code, whereas the only things that need changing are:

Flags.setNoSignedWrap(Op->getOpcode() == ISD::SABSDIFF);

and

DAG.getCondCode(Op->getOpcode() == ISD::SABSDIFF ? ISD::SETLT : ISD::SETULT);

[• ashahid]

Refactored *ABSDIFF* expansion code.

[jmolloy]

LGTM! Please wait for Hal's OK too.

[hfinkel]

LGTM too.

[• ashahid]

Hi Hal/James,

Since I don't have commit access yet, can somebody commit it on my behalf.

Regards,
Shahid

[• ashahid]

Rebased and resolved the error.

[jmolloy]

This got committed.

docs/LangRef.rst

 
 Examples:
 """""""""
 
 .. code-block:: llvm
 
       %r2 = call float @llvm.fmuladd.f32(float %a, float %b, float %c) ; yields float:r2 = (a * b) + c
 
+
+'``llvm.uabsdiff.*``' and '``llvm.sabsdiff.*``' Intrinsics
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Syntax:
+"""""""
+This is an overloaded intrinsic. The loaded data is a vector of any integer bit width.
+
+.. code-block:: llvm
+
+      declare <4 x integer> @llvm.uabsdiff.v4i32(<4 x integer> %a, <4 x integer> %b)
+
+
+Overview:
+"""""""""
+
+The ``llvm.uabsdiff`` intrinsic returns a vector result of the absolute difference of the two operands,
+treating them both as unsigned integers.
+
+The ``llvm.sabsdiff`` intrinsic returns  a vector result of the absolute difference of the two operands,
+treating them both as signed integers.
+
+.. note::
+
+    These intrinsics are primarily used during the code generation stage of compilation.
+    They are generated by compiler passes such as the Loop and SLP vectorizers.it is not
+    recommended for users to create them manually.
+
+Arguments:
+""""""""""
+
+Both intrinsics take two integer of the same bitwidth.
+
+Semantics:
+""""""""""
+
+The expression::
+
+    call <4 x i32> @llvm.uabsdiff.v4i32(<4 x i32> %a, <4 x i32> %b)
+
+is equivalent to::
+
+    %sub = sub <4 x i32> %a, %b
+    %ispos = icmp ugt <4 x i32> %sub, <i32 -1, i32 -1, i32 -1, i32 -1>
+    %neg = sub <4 x i32> zeroinitializer, %sub
+    %1 = select <4 x i1> %ispos, <4 x i32> %sub, <4 x i32> %neg
+
+Similarly the expression::
+
+    call <4 x i32> @llvm.sabsdiff.v4i32(<4 x i32> %a, <4 x i32> %b)
+
+is equivalent to::
+
+    %sub = sub nsw <4 x i32> %a, %b
+    %ispos = icmp sgt <4 x i32> %sub, <i32 -1, i32 -1, i32 -1, i32 -1>
+    %neg = sub nsw <4 x i32> zeroinitializer, %sub
+    %1 = select <4 x i1> %ispos, <4 x i32> %sub, <4 x i32> %neg
+
+
 Half Precision Floating Point Intrinsics
 ----------------------------------------
 
 For most target platforms, half precision floating point is a
 storage-only format. This means that it is a dense encoding (in memory)
 but does not support computation in the format.
 
 This means that code must first load the half-precision floating point

include/llvm/CodeGen/ISDOpcodes.h

enum NodeType {
     /// When the 1st operand is a vector, the shift amount must be in the same
     /// type. (TLI.getShiftAmountTy() will return the same type when the input
     /// type is a vector.)
     SHL, SRA, SRL, ROTL, ROTR,
 
     /// Byte Swap and Counting operators.
     BSWAP, CTTZ, CTLZ, CTPOP,
 
+    /// [SU]ABSDIFF - Signed/Unsigned absolute difference of two input integer
+    /// vector. These nodes are generated from llvm.*absdiff* intrinsics.
+    SABSDIFF, UABSDIFF,
+
     /// Bit counting operators with an undefined result for zero inputs.
     CTTZ_ZERO_UNDEF, CTLZ_ZERO_UNDEF,
 
     /// Select(COND, TRUEVAL, FALSEVAL).  If the type of the boolean COND is not
     /// i1 then the high bits must conform to getBooleanContents.
     SELECT,
 
     /// Select with a vector condition (op #0) and two vector operands (ops #1

include/llvm/IR/Intrinsics.td

 def int_convertuu  : Intrinsic<[llvm_anyint_ty],
                                [llvm_anyint_ty, llvm_i32_ty, llvm_i32_ty]>;
 
 // Clear cache intrinsic, default to ignore (ie. emit nothing)
 // maps to void __clear_cache() on supporting platforms
 def int_clear_cache : Intrinsic<[], [llvm_ptr_ty, llvm_ptr_ty],
                                 [], "llvm.clear_cache">;
 
+// Calculate the Absolute Differences of the two input vectors.
+def int_sabsdiff : Intrinsic<[llvm_anyvector_ty],
+                        [ LLVMMatchType<0>, LLVMMatchType<0> ], [IntrNoMem]>;
+def int_uabsdiff : Intrinsic<[llvm_anyvector_ty],
+                        [ LLVMMatchType<0>, LLVMMatchType<0> ], [IntrNoMem]>;
+
 //===-------------------------- Masked Intrinsics -------------------------===//
 //
 def int_masked_store : Intrinsic<[], [llvm_anyvector_ty, LLVMPointerTo<0>,
                                       llvm_i32_ty,
                                       LLVMVectorSameWidth<0, llvm_i1_ty>],
                                  [IntrReadWriteArgMem]>;
 
 def int_masked_load  : Intrinsic<[llvm_anyvector_ty],

include/llvm/Target/TargetSelectionDAG.td

def subc       : SDNode<"ISD::SUBC"      , SDTIntBinOp,
                         [SDNPOutGlue]>;
 def sube       : SDNode<"ISD::SUBE"      , SDTIntBinOp,
                         [SDNPOutGlue, SDNPInGlue]>;
 def smin       : SDNode<"ISD::SMIN"      , SDTIntBinOp>;
 def smax       : SDNode<"ISD::SMAX"      , SDTIntBinOp>;
 def umin       : SDNode<"ISD::UMIN"      , SDTIntBinOp>;
 def umax       : SDNode<"ISD::UMAX"      , SDTIntBinOp>;
 
+def sabsdiff   : SDNode<"ISD::SABSDIFF"   , SDTIntBinOp>;
+def uabsdiff   : SDNode<"ISD::UABSDIFF"   , SDTIntBinOp>;
 def sext_inreg : SDNode<"ISD::SIGN_EXTEND_INREG", SDTExtInreg>;
 def bswap      : SDNode<"ISD::BSWAP"      , SDTIntUnaryOp>;
 def ctlz       : SDNode<"ISD::CTLZ"       , SDTIntUnaryOp>;
 def cttz       : SDNode<"ISD::CTTZ"       , SDTIntUnaryOp>;
 def ctpop      : SDNode<"ISD::CTPOP"      , SDTIntUnaryOp>;
 def ctlz_zero_undef : SDNode<"ISD::CTLZ_ZERO_UNDEF", SDTIntUnaryOp>;
 def cttz_zero_undef : SDNode<"ISD::CTTZ_ZERO_UNDEF", SDTIntUnaryOp>;
 def sext       : SDNode<"ISD::SIGN_EXTEND", SDTIntExtendOp>;

lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

#endif
   case ISD::ATOMIC_LOAD_UMAX:
   case ISD::ATOMIC_SWAP:
     Res = PromoteIntRes_Atomic1(cast<AtomicSDNode>(N)); break;
 
   case ISD::ATOMIC_CMP_SWAP:
   case ISD::ATOMIC_CMP_SWAP_WITH_SUCCESS:
     Res = PromoteIntRes_AtomicCmpSwap(cast<AtomicSDNode>(N), ResNo);
     break;
+  case ISD::UABSDIFF:
+  case ISD::SABSDIFF:
+    Res = PromoteIntRes_SimpleIntBinOp(N);
+    break;
   }
 
   // If the result is null then the sub-method took care of registering it.
   if (Res.getNode())
     SetPromotedInteger(SDValue(N, ResNo), Res);
 }
 
 SDValue DAGTypeLegalizer::PromoteIntRes_MERGE_VALUES(SDNode *N,

lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

class VectorLegalizer {
 
   /// \brief Implement vselect in terms of XOR, AND, OR when blend is not
   /// supported by the target.
   SDValue ExpandVSELECT(SDValue Op);
   SDValue ExpandSELECT(SDValue Op);
   SDValue ExpandLoad(SDValue Op);
   SDValue ExpandStore(SDValue Op);
   SDValue ExpandFNEG(SDValue Op);
+  SDValue ExpandABSDIFF(SDValue Op);
 
   /// \brief Implements vector promotion.
   ///
   /// This is essentially just bitcasting the operands to a different type and
   /// bitcasting the result back to the original type.
   SDValue Promote(SDValue Op);
 
   /// \brief Implements [SU]INT_TO_FP vector promotion.
SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
   case ISD::SIGN_EXTEND_INREG:
   case ISD::ANY_EXTEND_VECTOR_INREG:
   case ISD::SIGN_EXTEND_VECTOR_INREG:
   case ISD::ZERO_EXTEND_VECTOR_INREG:
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
+  case ISD::UABSDIFF:
+  case ISD::SABSDIFF:
     QueryType = Node->getValueType(0);
     break;
   case ISD::FP_ROUND_INREG:
     QueryType = cast<VTSDNode>(Node->getOperand(1))->getVT();
     break;
   case ISD::SINT_TO_FP:
   case ISD::UINT_TO_FP:
     QueryType = Node->getOperand(0).getValueType();
SDValue VectorLegalizer::Expand(SDValue Op) {
   case ISD::SELECT:
     return ExpandSELECT(Op);
   case ISD::UINT_TO_FP:
     return ExpandUINT_TO_FLOAT(Op);
   case ISD::FNEG:
     return ExpandFNEG(Op);
   case ISD::SETCC:
     return UnrollVSETCC(Op);
+  case ISD::UABSDIFF:
+  case ISD::SABSDIFF:
+    return ExpandABSDIFF(Op);
   default:
     return DAG.UnrollVectorOp(Op.getNode());
   }
 }
 
+SDValue VectorLegalizer::ExpandABSDIFF(SDValue Op) {
+  SDLoc dl(Op);
+  SDValue Tmp1, Tmp2, Tmp3, Tmp4;
+  EVT VT = Op.getValueType();
+  SDNodeFlags Flags;
+  Flags.setNoSignedWrap(Op->getOpcode() == ISD::SABSDIFF);
+
+  Tmp2 = Op.getOperand(0);
+  Tmp3 = Op.getOperand(1);
+  Tmp1 = DAG.getNode(ISD::SUB, dl, VT, Tmp2, Tmp3, &Flags);
+  Tmp2 =
+      DAG.getNode(ISD::SUB, dl, VT, DAG.getConstant(0, dl, VT), Tmp1, &Flags);
+  Tmp4 = DAG.getNode(
+      ISD::SETCC, dl,
+      TLI.getSetCCResultType(DAG.getDataLayout(), *DAG.getContext(), VT), Tmp2,
+      DAG.getConstant(0, dl, VT),
+      DAG.getCondCode(Op->getOpcode() == ISD::SABSDIFF ? ISD::SETLT
+                                                       : ISD::SETULT));
+  Tmp1 = DAG.getNode(ISD::VSELECT, dl, VT, Tmp4, Tmp1, Tmp2);
+  return Tmp1;
+}
+
 SDValue VectorLegalizer::ExpandSELECT(SDValue Op) {
   // Lower a select instruction where the condition is a scalar and the
   // operands are vectors. Lower this select to VSELECT and implement it
   // using XOR AND OR. The selector bit is broadcasted.
   EVT VT = Op.getValueType();
   SDLoc DL(Op);
 
   SDValue Mask = Op.getOperand(0);

lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp

#endif
   case ISD::SRL:
   case ISD::UREM:
   case ISD::SREM:
   case ISD::FREM:
   case ISD::SMIN:
   case ISD::SMAX:
   case ISD::UMIN:
   case ISD::UMAX:
+  case ISD::UABSDIFF:
+  case ISD::SABSDIFF:
     SplitVecRes_BinOp(N, Lo, Hi);
     break;
   case ISD::FMA:
     SplitVecRes_TernaryOp(N, Lo, Hi);
     break;
   }
 
   // If Lo/Hi is null, the sub-method took care of registering results etc.

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

case Intrinsic::readcyclecounter: {
     DAG.setRoot(Res.getValue(1));
     return nullptr;
   }
   case Intrinsic::bswap:
     setValue(&I, DAG.getNode(ISD::BSWAP, sdl,
                              getValue(I.getArgOperand(0)).getValueType(),
                              getValue(I.getArgOperand(0))));
     return nullptr;
+  case Intrinsic::uabsdiff:
+    setValue(&I, DAG.getNode(ISD::UABSDIFF, sdl,
+                             getValue(I.getArgOperand(0)).getValueType(),
+                             getValue(I.getArgOperand(0)),
+                             getValue(I.getArgOperand(1))));
+    return nullptr;
+  case Intrinsic::sabsdiff:
+    setValue(&I, DAG.getNode(ISD::SABSDIFF, sdl,
+                             getValue(I.getArgOperand(0)).getValueType(),
+                             getValue(I.getArgOperand(0)),
+                             getValue(I.getArgOperand(1))));
+    return nullptr;
   case Intrinsic::cttz: {
     SDValue Arg = getValue(I.getArgOperand(0));
     ConstantInt *CI = cast<ConstantInt>(I.getArgOperand(1));
     EVT Ty = Arg.getValueType();
     setValue(&I, DAG.getNode(CI->isZero() ? ISD::CTTZ : ISD::CTTZ_ZERO_UNDEF,
                              sdl, Ty, Arg));
     return nullptr;
   }

lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp

#endif
   case ISD::USUBO:                      return "usubo";
   case ISD::SMULO:                      return "smulo";
   case ISD::UMULO:                      return "umulo";
   case ISD::SUBC:                       return "subc";
   case ISD::SUBE:                       return "sube";
   case ISD::SHL_PARTS:                  return "shl_parts";
   case ISD::SRA_PARTS:                  return "sra_parts";
   case ISD::SRL_PARTS:                  return "srl_parts";
+  case ISD::UABSDIFF:                   return "uabsdiff";
+  case ISD::SABSDIFF:                   return "sabsdiff";
 
   // Conversion operators.
   case ISD::SIGN_EXTEND:                return "sign_extend";
   case ISD::ZERO_EXTEND:                return "zero_extend";
   case ISD::ANY_EXTEND:                 return "any_extend";
   case ISD::SIGN_EXTEND_INREG:          return "sign_extend_inreg";
   case ISD::ANY_EXTEND_VECTOR_INREG:    return "any_extend_vector_inreg";
   case ISD::SIGN_EXTEND_VECTOR_INREG:   return "sign_extend_vector_inreg";

lib/CodeGen/TargetLoweringBase.cpp

for (MVT VT : MVT::all_valuetypes()) {
 
     // Overflow operations default to expand
     setOperationAction(ISD::SADDO, VT, Expand);
     setOperationAction(ISD::SSUBO, VT, Expand);
     setOperationAction(ISD::UADDO, VT, Expand);
     setOperationAction(ISD::USUBO, VT, Expand);
     setOperationAction(ISD::SMULO, VT, Expand);
     setOperationAction(ISD::UMULO, VT, Expand);
+    setOperationAction(ISD::UABSDIFF, VT, Expand);
+    setOperationAction(ISD::SABSDIFF, VT, Expand);
 
     // These library functions default to expand.
     setOperationAction(ISD::FROUND, VT, Expand);
 
     // These operations default to expand for vector types.
     if (VT.isVector()) {
       setOperationAction(ISD::FCOPYSIGN, VT, Expand);
       setOperationAction(ISD::ANY_EXTEND_VECTOR_INREG, VT, Expand);

test/CodeGen/X86/absdiff_expand.ll

+; RUN: llc -mtriple=x86_64-unknown-linux-gnu  < %s | FileCheck %s -check-prefix=CHECK
+
+declare <4 x i8> @llvm.uabsdiff.v4i8(<4 x i8>, <4 x i8>)
+
+define <4 x i8> @test_uabsdiff_v4i8_expand(<4 x i8> %a1, <4 x i8> %a2) {
+; CHECK-LABEL: test_uabsdiff_v4i8_expand
+; CHECK:             psubd  %xmm1, %xmm0
+; CHECK-NEXT:        pxor   %xmm1, %xmm1
+; CHECK-NEXT:        psubd  %xmm0, %xmm1
+; CHECK-NEXT:        movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:        movdqa  %xmm1, %xmm3
+; CHECK-NEXT:        pxor   %xmm2, %xmm3
+; CHECK-NEXT:        pcmpgtd        %xmm3, %xmm2
+; CHECK-NEXT:        pand    %xmm2, %xmm0
+; CHECK-NEXT:        pandn   %xmm1, %xmm2
+; CHECK-NEXT:        por     %xmm2, %xmm0
+; CHECK-NEXT:        retq
+
+  %1 = call <4 x i8> @llvm.uabsdiff.v4i8(<4 x i8> %a1, <4 x i8> %a2)
+  ret <4 x i8> %1
+}
+
+declare <4 x i8> @llvm.sabsdiff.v4i8(<4 x i8>, <4 x i8>)
+
+define <4 x i8> @test_sabsdiff_v4i8_expand(<4 x i8> %a1, <4 x i8> %a2) {
+; CHECK-LABEL: test_sabsdiff_v4i8_expand
+; CHECK:      psubd  %xmm1, %xmm0
+; CHECK-NEXT: pxor   %xmm1, %xmm1
+; CHECK-NEXT: pxor    %xmm2, %xmm2
+; CHECK-NEXT: psubd  %xmm0, %xmm2
+; CHECK-NEXT: pcmpgtd  %xmm2, %xmm1
+; CHECK-NEXT: pand    %xmm1, %xmm0
+; CHECK-NEXT: pandn   %xmm2, %xmm1
+; CHECK-NEXT: por     %xmm1, %xmm0
+; CHECK-NEXT: retq
+
+  %1 = call <4 x i8> @llvm.sabsdiff.v4i8(<4 x i8> %a1, <4 x i8> %a2)
+  ret <4 x i8> %1
+}
+
+
+declare <8 x i8> @llvm.sabsdiff.v8i8(<8 x i8>, <8 x i8>)
+
+define <8 x i8> @test_sabsdiff_v8i8_expand(<8 x i8> %a1, <8 x i8> %a2) {
+; CHECK-LABEL: test_sabsdiff_v8i8_expand
+; CHECK:      psubw  %xmm1, %xmm0
+; CHECK-NEXT: pxor   %xmm1, %xmm1
+; CHECK-NEXT: pxor   %xmm2, %xmm2
+; CHECK-NEXT: psubw  %xmm0, %xmm2
+; CHECK-NEXT: pcmpgtw        %xmm2, %xmm1
+; CHECK-NEXT: pand  %xmm1, %xmm0
+; CHECK-NEXT: pandn %xmm2, %xmm1
+; CHECK-NEXT: por  %xmm1, %xmm0
+; CHECK-NEXT: retq
+  %1 = call <8 x i8> @llvm.sabsdiff.v8i8(<8 x i8> %a1, <8 x i8> %a2)
+  ret <8 x i8> %1
+}
+
+declare <16 x i8> @llvm.uabsdiff.v16i8(<16 x i8>, <16 x i8>)
+
+define <16 x i8> @test_uabsdiff_v16i8_expand(<16 x i8> %a1, <16 x i8> %a2) {
+; CHECK-LABEL: test_uabsdiff_v16i8_expand
+; CHECK:             psubb  %xmm1, %xmm0
+; CHECK-NEXT:        pxor   %xmm1, %xmm1
+; CHECK-NEXT:        psubb  %xmm0, %xmm1
+; CHECK-NEXT:        movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:        movdqa  %xmm1, %xmm3
+; CHECK-NEXT:        pxor   %xmm2, %xmm3
+; CHECK-NEXT:        pcmpgtb        %xmm3, %xmm2
+; CHECK-NEXT:        pand    %xmm2, %xmm0
+; CHECK-NEXT:        pandn   %xmm1, %xmm2
+; CHECK-NEXT:        por     %xmm2, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <16 x i8> @llvm.uabsdiff.v16i8(<16 x i8> %a1, <16 x i8> %a2)
+  ret <16 x i8> %1
+}
+
+declare <8 x i16> @llvm.uabsdiff.v8i16(<8 x i16>, <8 x i16>)
+
+define <8 x i16> @test_uabsdiff_v8i16_expand(<8 x i16> %a1, <8 x i16> %a2) {
+; CHECK-LABEL: test_uabsdiff_v8i16_expand
+; CHECK:             psubw  %xmm1, %xmm0
+; CHECK-NEXT:        pxor   %xmm1, %xmm1
+; CHECK-NEXT:        psubw  %xmm0, %xmm1
+; CHECK-NEXT:        movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:        movdqa  %xmm1, %xmm3
+; CHECK-NEXT:        pxor   %xmm2, %xmm3
+; CHECK-NEXT:        pcmpgtw        %xmm3, %xmm2
+; CHECK-NEXT:        pand    %xmm2, %xmm0
+; CHECK-NEXT:        pandn   %xmm1, %xmm2
+; CHECK-NEXT:        por     %xmm2, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <8 x i16> @llvm.uabsdiff.v8i16(<8 x i16> %a1, <8 x i16> %a2)
+  ret <8 x i16> %1
+}
+
+declare <8 x i16> @llvm.sabsdiff.v8i16(<8 x i16>, <8 x i16>)
+
+define <8 x i16> @test_sabsdiff_v8i16_expand(<8 x i16> %a1, <8 x i16> %a2) {
+; CHECK-LABEL: test_sabsdiff_v8i16_expand
+; CHECK:      psubw  %xmm1, %xmm0
+; CHECK-NEXT: pxor   %xmm1, %xmm1
+; CHECK-NEXT: pxor   %xmm2, %xmm2
+; CHECK-NEXT: psubw  %xmm0, %xmm2
+; CHECK-NEXT: pcmpgtw        %xmm2, %xmm1
+; CHECK-NEXT: pand  %xmm1, %xmm0
+; CHECK-NEXT: pandn %xmm2, %xmm1
+; CHECK-NEXT: por  %xmm1, %xmm0
+; CHECK-NEXT: retq
+  %1 = call <8 x i16> @llvm.sabsdiff.v8i16(<8 x i16> %a1, <8 x i16> %a2)
+  ret <8 x i16> %1
+}
+
+declare <4 x i32> @llvm.sabsdiff.v4i32(<4 x i32>, <4 x i32>)
+
+define <4 x i32> @test_sabsdiff_v4i32_expand(<4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v4i32_expand
+; CHECK:             psubd  %xmm1, %xmm0
+; CHECK-NEXT:        pxor  %xmm1, %xmm1
+; CHECK-NEXT:        pxor  %xmm2, %xmm2
+; CHECK-NEXT:        psubd  %xmm0, %xmm2
+; CHECK-NEXT:        pcmpgtd        %xmm2, %xmm1
+; CHECK-NEXT:        pand    %xmm1, %xmm0
+; CHECK-NEXT:        pandn   %xmm2, %xmm1
+; CHECK-NEXT:        por    %xmm1, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <4 x i32> @llvm.sabsdiff.v4i32(<4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %1
+}
+
+declare <4 x i32> @llvm.uabsdiff.v4i32(<4 x i32>, <4 x i32>)
+
+define <4 x i32> @test_uabsdiff_v4i32_expand(<4 x i32> %a1, <4 x i32> %a2) {
+; CHECK-LABEL: test_uabsdiff_v4i32_expand
+; CHECK:             psubd  %xmm1, %xmm0
+; CHECK-NEXT:        pxor   %xmm1, %xmm1
+; CHECK-NEXT:        psubd  %xmm0, %xmm1
+; CHECK-NEXT:        movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:        movdqa  %xmm1, %xmm3
+; CHECK-NEXT:        pxor   %xmm2, %xmm3
+; CHECK-NEXT:        pcmpgtd        %xmm3, %xmm2
+; CHECK-NEXT:        pand    %xmm2, %xmm0
+; CHECK-NEXT:        pandn   %xmm1, %xmm2
+; CHECK-NEXT:        por     %xmm2, %xmm0
+; CHECK-NEXT:        retq
+  %1 = call <4 x i32> @llvm.uabsdiff.v4i32(<4 x i32> %a1, <4 x i32> %a2)
+  ret <4 x i32> %1
+}
+
+declare <2 x i32> @llvm.sabsdiff.v2i32(<2 x i32>, <2 x i32>)
+
+define <2 x i32> @test_sabsdiff_v2i32_expand(<2 x i32> %a1, <2 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v2i32_expand
+; CHECK:        psubq   %xmm1, %xmm0
+; CHECK-NEXT:   pxor    %xmm1, %xmm1
+; CHECK-NEXT:   psubq   %xmm0, %xmm1
+; CHECK-NEXT:   movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:   movdqa  %xmm1, %xmm3
+; CHECK-NEXT:   pxor    %xmm2, %xmm3
+; CHECK-NEXT:   movdqa  %xmm2, %xmm4
+; CHECK-NEXT:   pcmpgtd %xmm3, %xmm4
+; CHECK-NEXT:   pshufd  $160, %xmm4, %xmm5      # xmm5 = xmm4[0,0,2,2]
+; CHECK-NEXT:   pcmpeqd %xmm2, %xmm3
+; CHECK-NEXT:   pshufd  $245, %xmm3, %xmm2      # xmm2 = xmm3[1,1,3,3]
+; CHECK-NEXT:   pand    %xmm5, %xmm2
+; CHECK-NEXT:   pshufd  $245, %xmm4, %xmm3      # xmm3 = xmm4[1,1,3,3]
+; CHECK-NEXT:   por     %xmm2, %xmm3
+; CHECK-NEXT:   pand    %xmm3, %xmm0
+; CHECK-NEXT:   pandn   %xmm1, %xmm3
+; CHECK-NEXT:   por     %xmm3, %xmm0
+; CHECK-NEXT:   retq
+  %1 = call <2 x i32> @llvm.sabsdiff.v2i32(<2 x i32> %a1, <2 x i32> %a2)
+  ret <2 x i32> %1
+}
+
+declare <2 x i64> @llvm.sabsdiff.v2i64(<2 x i64>, <2 x i64>)
+
+define <2 x i64> @test_sabsdiff_v2i64_expand(<2 x i64> %a1, <2 x i64> %a2) {
+; CHECK-LABEL: test_sabsdiff_v2i64_expand
+; CHECK:        psubq   %xmm1, %xmm0
+; CHECK-NEXT:   pxor    %xmm1, %xmm1
+; CHECK-NEXT:   psubq   %xmm0, %xmm1
+; CHECK-NEXT:   movdqa  .LCPI{{[0-9_]*}}
+; CHECK-NEXT:   movdqa  %xmm1, %xmm3
+; CHECK-NEXT:   pxor    %xmm2, %xmm3
+; CHECK-NEXT:   movdqa  %xmm2, %xmm4
+; CHECK-NEXT:   pcmpgtd %xmm3, %xmm4
+; CHECK-NEXT:   pshufd  $160, %xmm4, %xmm5      # xmm5 = xmm4[0,0,2,2]
+; CHECK-NEXT:   pcmpeqd %xmm2, %xmm3
+; CHECK-NEXT:   pshufd  $245, %xmm3, %xmm2      # xmm2 = xmm3[1,1,3,3]
+; CHECK-NEXT:   pand    %xmm5, %xmm2
+; CHECK-NEXT:   pshufd  $245, %xmm4, %xmm3      # xmm3 = xmm4[1,1,3,3]
+; CHECK-NEXT:   por     %xmm2, %xmm3
+; CHECK-NEXT:   pand    %xmm3, %xmm0
+; CHECK-NEXT:   pandn   %xmm1, %xmm3
+; CHECK-NEXT:   por     %xmm3, %xmm0
+; CHECK-NEXT:   retq
+  %1 = call <2 x i64> @llvm.sabsdiff.v2i64(<2 x i64> %a1, <2 x i64> %a2)
+  ret <2 x i64> %1
+}
+
+declare <16 x i32> @llvm.sabsdiff.v16i32(<16 x i32>, <16 x i32>)
+
+define <16 x i32> @test_sabsdiff_v16i32_expand(<16 x i32> %a1, <16 x i32> %a2) {
+; CHECK-LABEL: test_sabsdiff_v16i32_expand
+; CHECK:             psubd  %xmm4, %xmm0
+; CHECK-NEXT:        pxor    %xmm8, %xmm8
+; CHECK-NEXT:        pxor    %xmm9, %xmm9
+; CHECK-NEXT:        psubd   %xmm0, %xmm9
+; CHECK-NEXT:        pxor    %xmm4, %xmm4
+; CHECK-NEXT:        pcmpgtd %xmm9, %xmm4
+; CHECK-NEXT:        pand    %xmm4, %xmm0
+; CHECK-NEXT:        pandn   %xmm9, %xmm4
+; CHECK-NEXT:        por     %xmm4, %xmm0
+; CHECK-NEXT:        psubd   %xmm5, %xmm1
+; CHECK-NEXT:        pxor    %xmm4, %xmm4
+; CHECK-NEXT:        psubd   %xmm1, %xmm4
+; CHECK-NEXT:        pxor    %xmm5, %xmm5
+; CHECK-NEXT:        pcmpgtd %xmm4, %xmm5
+; CHECK-NEXT:        pand    %xmm5, %xmm1
+; CHECK-NEXT:        pandn   %xmm4, %xmm5
+; CHECK-NEXT:        por     %xmm5, %xmm1
+; CHECK-NEXT:        psubd   %xmm6, %xmm2
+; CHECK-NEXT:        pxor    %xmm4, %xmm4
+; CHECK-NEXT:        psubd   %xmm2, %xmm4
+; CHECK-NEXT:        pxor    %xmm5, %xmm5
+; CHECK-NEXT:        pcmpgtd %xmm4, %xmm5
+; CHECK-NEXT:        pand    %xmm5, %xmm2
+; CHECK-NEXT:        pandn   %xmm4, %xmm5
+; CHECK-NEXT:        por     %xmm5, %xmm2
+; CHECK-NEXT:        psubd   %xmm7, %xmm3
+; CHECK-NEXT:        pxor    %xmm4, %xmm4
+; CHECK-NEXT:        psubd   %xmm3, %xmm4
+; CHECK-NEXT:        pcmpgtd %xmm4, %xmm8
+; CHECK-NEXT:        pand    %xmm8, %xmm3
+; CHECK-NEXT:        pandn   %xmm4, %xmm8
+; CHECK-NEXT:        por     %xmm8, %xmm3
+; CHECK-NEXT:        retq
+  %1 = call <16 x i32> @llvm.sabsdiff.v16i32(<16 x i32> %a1, <16 x i32> %a2)
+  ret <16 x i32> %1
+}
+