[PPC64] Add vector quadword add/sub instructions for POWER8

[kbarton]

Summary

This patch adds support for the vector quadword add/sub instructions introduced in POWER8:

• vadduqm
• vaddeuqm
• vaddcuq
• vaddecuq
• vsubuqm
• vsubeuqm
• vsubcuq
• vsubecuq

In addition to adding the instructions themselves, it also adds support for the v1i128 type for intrinsics (Intrinsics.td, Function.cpp, and IntrinsicEmitter.cpp).

[wschmidt]

A couple of inline notes.

lib/Target/PowerPC/PPCCallingConv.td

65 ↗

(On Diff #23941)

Looks like a line-too-long.

lib/Target/PowerPC/PPCISelLowering.cpp

4925

This is a lot more complicated than that. You've just asserted you can pass a 128-bit value in a single 64-bit register, which will bring about wailing and gnashing of teeth.

An i128 needs to be passed in two GPRs, and furthermore you may need to first skip a GPR to ensure that the image of the i128 in the parameter save area is aligned on a 16-byte boundary. (If the parameter's address is taken, it will be forced to its home location in memory, and that must be properly aligned.) I've verified that in gcc we have rs6000_function_arg_advance_1 which certainly makes this adjustment, as does rs6000_function_arg.

I believe you can borrow some of the necessarily logic from the varargs handling of vector types, below; it has to pass varargs parameters in consecutive GPRs as well. However, I'm bothered that I don't see anything there for alignment. The ABI document is wishy-washy about this for varargs: "Alignment requirements such as those for vector types may require the va_list pointer to first be aligned before accessing a value." Note the "may." But GCC's varargs handling uses rs6000_function_arg_advance_1 as well, so I believe to be compatible we have to have the same alignment done here.

CCing Uli for his thoughts, since he spent a lot of his life in this ABI code. At one time this passed the gcc ABI compatibility tests, so maybe I'm missing something.

[wschmidt]

One more general comment -- you don't have any testing for the ABI changes. Also, the ABI changes should probably be a separate patch from the quadword instruction patch....sorry. :/

[nemanjai]

include/llvm/IR/IntrinsicsPowerPC.td

552

Do we not want these guarded for Power8 Altivec feature? It would seem that we don't want these available in the back end for CPU's that don't have the feature.

test/CodeGen/PowerPC/vec_add_sub_quadword.ll

10

I don't think all the registers are ABI-required registers. Probably ok to use a regex for the other(s). Just making the same comment I received previously and I think it works well. A regex can be something like {{[0-9]+}}.

It may or may not make sense, but you can check that the value being added is loaded into a register and that that register is used for the add (at least for the immediate). Although I don't know if that's a good idea for commutative operations. Maybe someone more knowledgeable can comment on that.

16

1. Why the star at the end?
2. I think that in a large test case, it helps to have CHECK-LABEL for every function so that if something fails, it is easier to find where the failure is. The tool won't go past a check if the previous one did not match.

72

Would it make sense to check that the right parameter is in the right register?

[kbarton]

In D9081#157856, @wschmidt wrote:

One more general comment -- you don't have any testing for the ABI changes. Also, the ABI changes should probably be a separate patch from the quadword instruction patch....sorry. :/

Agreed. I will refactor these changes and take a closer look at your comments in PPCISelLowering.cpp and submit a new patch for review.

[kbarton]

include/llvm/IR/IntrinsicsPowerPC.td

552

We don't have any other guards in this file, as far as I can tell.
I believe this is fine to define these intrinsics as is, and let other diagnostic messages determine whether they are valid for the target architecture.

test/CodeGen/PowerPC/vec_add_sub_quadword.ll

10

Agreed. Will change.

16

The star is a typo.
I've also added the check label for the first several functions. Good call.

72

I don't know how this would be done, short of actually running the test case and making sure the output is correct.

[kbarton]

test/CodeGen/PowerPC/vec_add_sub_quadword.ll

10

Actually, on second thought I think it's better to leave these tests as is.
The ABI specifies which registers the parameters to the function are passed in and given these are extremely short functions there is no reason to not reuse the same registers for the specific instructions being tested. So, by using the same registers as the parameters, we are testing to ensure no unnecessary register moves are being added to the code.

If these were more complicated test cases, then we would need to use regular expressions because we couldn't guarantee in all cases the same registers would be used.

[kbarton]

The changes related to the ABI have been refactored and put into a different patch.
This patch now contains only the 8 new instructions to be added, and the necessary support for the v1i128 type in the intrinsics code.

[wschmidt]

LGTM. Sorry I missed this with the email server problem!

[kbarton]

Committed revision 238144.

include/llvm/IR/Intrinsics.td

 def llvm_v8i32_ty      : LLVMType<v8i32>;    //  8 x i32
 def llvm_v16i32_ty     : LLVMType<v16i32>;   // 16 x i32
 def llvm_v1i64_ty      : LLVMType<v1i64>;    //  1 x i64
 def llvm_v2i64_ty      : LLVMType<v2i64>;    //  2 x i64
 def llvm_v4i64_ty      : LLVMType<v4i64>;    //  4 x i64
 def llvm_v8i64_ty      : LLVMType<v8i64>;    //  8 x i64
 def llvm_v16i64_ty     : LLVMType<v16i64>;   // 16 x i64
 
+def llvm_v1i128_ty     : LLVMType<v1i128>;   //  1 x i128
+
 def llvm_v2f16_ty      : LLVMType<v2f16>;    //  2 x half (__fp16)
 def llvm_v4f16_ty      : LLVMType<v4f16>;    //  4 x half (__fp16)
 def llvm_v8f16_ty      : LLVMType<v8f16>;    //  8 x half (__fp16)
 def llvm_v1f32_ty      : LLVMType<v1f32>;    //  1 x float
 def llvm_v2f32_ty      : LLVMType<v2f32>;    //  2 x float
 def llvm_v4f32_ty      : LLVMType<v4f32>;    //  4 x float
 def llvm_v8f32_ty      : LLVMType<v8f32>;    //  8 x float
 def llvm_v16f32_ty     : LLVMType<v16f32>;   // 16 x float

include/llvm/IR/IntrinsicsPowerPC.td

 
 /// PowerPC_Vec_DDD_Intrinsic - A PowerPC intrinsic that takes two v2i64
 /// vectors and returns one.  These intrinsics have no side effects.
 class PowerPC_Vec_DDD_Intrinsic<string GCCIntSuffix>
   : PowerPC_Vec_Intrinsic<GCCIntSuffix,
                           [llvm_v2i64_ty], [llvm_v2i64_ty, llvm_v2i64_ty],
                           [IntrNoMem]>;
 
+/// PowerPC_Vec_QQQ_Intrinsic - A PowerPC intrinsic that takes two v1i128
+/// vectors and returns one. These intrinsics have no side effects.
+class PowerPC_Vec_QQQ_Intrinsic<string GCCIntSuffix>
+  : PowerPC_Vec_Intrinsic<GCCIntSuffix,
+                         [llvm_v1i128_ty], [llvm_v1i128_ty, llvm_v1i128_ty],
+                         [IntrNoMem]>;
+
 //===----------------------------------------------------------------------===//
 // PowerPC VSX Intrinsic Class Definitions.
 //
 
 /// PowerPC_VSX_Vec_DDD_Intrinsic - A PowerPC intrinsic that takes two v2f64
 /// vectors and returns one.  These intrinsics have no side effects.
 class PowerPC_VSX_Vec_DDD_Intrinsic<string GCCIntSuffix>
   : PowerPC_VSX_Intrinsic<GCCIntSuffix,

 // Saturating adds.
 def int_ppc_altivec_vaddubs : PowerPC_Vec_BBB_Intrinsic<"vaddubs">;
 def int_ppc_altivec_vaddsbs : PowerPC_Vec_BBB_Intrinsic<"vaddsbs">;
 def int_ppc_altivec_vadduhs : PowerPC_Vec_HHH_Intrinsic<"vadduhs">;
 def int_ppc_altivec_vaddshs : PowerPC_Vec_HHH_Intrinsic<"vaddshs">;
 def int_ppc_altivec_vadduws : PowerPC_Vec_WWW_Intrinsic<"vadduws">;
 def int_ppc_altivec_vaddsws : PowerPC_Vec_WWW_Intrinsic<"vaddsws">;
 def int_ppc_altivec_vaddcuw : PowerPC_Vec_WWW_Intrinsic<"vaddcuw">;
+def int_ppc_altivec_vaddcuq : PowerPC_Vec_QQQ_Intrinsic<"vaddcuq">;
 
 // Saturating subs.
 def int_ppc_altivec_vsububs : PowerPC_Vec_BBB_Intrinsic<"vsububs">;
 def int_ppc_altivec_vsubsbs : PowerPC_Vec_BBB_Intrinsic<"vsubsbs">;
 def int_ppc_altivec_vsubuhs : PowerPC_Vec_HHH_Intrinsic<"vsubuhs">;
 def int_ppc_altivec_vsubshs : PowerPC_Vec_HHH_Intrinsic<"vsubshs">;
 def int_ppc_altivec_vsubuws : PowerPC_Vec_WWW_Intrinsic<"vsubuws">;
 def int_ppc_altivec_vsubsws : PowerPC_Vec_WWW_Intrinsic<"vsubsws">;
 def int_ppc_altivec_vsubcuw : PowerPC_Vec_WWW_Intrinsic<"vsubcuw">;
+def int_ppc_altivec_vsubcuq : PowerPC_Vec_QQQ_Intrinsic<"vsubcuq">;
 
 let TargetPrefix = "ppc" in {  // All PPC intrinsics start with "llvm.ppc.".
   // Saturating multiply-adds.
   def int_ppc_altivec_vmhaddshs : GCCBuiltin<"__builtin_altivec_vmhaddshs">,
               Intrinsic<[llvm_v8i16_ty], [llvm_v8i16_ty,
                          llvm_v8i16_ty, llvm_v8i16_ty], [IntrNoMem]>;
   def int_ppc_altivec_vmhraddshs : GCCBuiltin<"__builtin_altivec_vmhraddshs">,
               Intrinsic<[llvm_v8i16_ty], [llvm_v8i16_ty,
let TargetPrefix = "ppc" in {  // All PPC intrinsics start with "llvm.ppc.".
   def int_ppc_altivec_vrfim : GCCBuiltin<"__builtin_altivec_vrfim">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty], [IntrNoMem]>;
   def int_ppc_altivec_vrfin : GCCBuiltin<"__builtin_altivec_vrfin">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty], [IntrNoMem]>;
   def int_ppc_altivec_vrfip : GCCBuiltin<"__builtin_altivec_vrfip">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty], [IntrNoMem]>;
   def int_ppc_altivec_vrfiz : GCCBuiltin<"__builtin_altivec_vrfiz">,
               Intrinsic<[llvm_v4f32_ty], [llvm_v4f32_ty], [IntrNoMem]>;
+
+  // Add Extended Quadword
+  def int_ppc_altivec_vaddeuqm : GCCBuiltin<"__builtin_altivec_vaddeuqm">,
+              Intrinsic<[llvm_v1i128_ty], 
+                        [llvm_v1i128_ty, llvm_v1i128_ty, llvm_v1i128_ty],
+                        [IntrNoMem]>;
+  def int_ppc_altivec_vaddecuq : GCCBuiltin<"__builtin_altivec_vaddecuq">,
+              Intrinsic<[llvm_v1i128_ty], 
+                        [llvm_v1i128_ty, llvm_v1i128_ty, llvm_v1i128_ty],
+                        [IntrNoMem]>;
+
+  // Sub Extended Quadword
+  def int_ppc_altivec_vsubeuqm : GCCBuiltin<"__builtin_altivec_vsubeuqm">,
+              Intrinsic<[llvm_v1i128_ty], 
+                        [llvm_v1i128_ty, llvm_v1i128_ty, llvm_v1i128_ty],
+                        [IntrNoMem]>;
+  def int_ppc_altivec_vsubecuq : GCCBuiltin<"__builtin_altivec_vsubecuq">,
+              Intrinsic<[llvm_v1i128_ty], 
+                        [llvm_v1i128_ty, llvm_v1i128_ty, llvm_v1i128_ty],
+                        [IntrNoMem]>;
 }
 
 def int_ppc_altivec_vsl   : PowerPC_Vec_WWW_Intrinsic<"vsl">;
 def int_ppc_altivec_vslo  : PowerPC_Vec_WWW_Intrinsic<"vslo">;
 
 def int_ppc_altivec_vslb  : PowerPC_Vec_BBB_Intrinsic<"vslb">;
 def int_ppc_altivec_vslh  : PowerPC_Vec_HHH_Intrinsic<"vslh">;
 def int_ppc_altivec_vslw  : PowerPC_Vec_WWW_Intrinsic<"vslw">;

lib/IR/Function.cpp

enum IIT_Info {
   IIT_EXTEND_ARG = 24,
   IIT_TRUNC_ARG = 25,
   IIT_ANYPTR = 26,
   IIT_V1   = 27,
   IIT_VARARG = 28,
   IIT_HALF_VEC_ARG = 29,
   IIT_SAME_VEC_WIDTH_ARG = 30,
   IIT_PTR_TO_ARG = 31,
-  IIT_VEC_OF_PTRS_TO_ELT = 32
+  IIT_VEC_OF_PTRS_TO_ELT = 32,
+  IIT_I128 = 33
 };
 
 
 static void DecodeIITType(unsigned &NextElt, ArrayRef<unsigned char> Infos,
                       SmallVectorImpl<Intrinsic::IITDescriptor> &OutputTable) {
   IIT_Info Info = IIT_Info(Infos[NextElt++]);
   unsigned StructElts = 2;
   using namespace Intrinsic;
case IIT_I16:
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Integer,16));
     return;
   case IIT_I32:
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Integer, 32));
     return;
   case IIT_I64:
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Integer, 64));
     return;
+  case IIT_I128:
+    OutputTable.push_back(IITDescriptor::get(IITDescriptor::Integer, 128));
+    return;
   case IIT_V1:
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Vector, 1));
     DecodeIITType(NextElt, Infos, OutputTable);
     return;
   case IIT_V2:
     OutputTable.push_back(IITDescriptor::get(IITDescriptor::Vector, 2));
     DecodeIITType(NextElt, Infos, OutputTable);
     return;

lib/Target/PowerPC/PPCISelLowering.cpp

if (Subtarget.use64BitRegs()) {
     setOperationAction(ISD::SRL_PARTS, MVT::i32, Custom);
   }
 
   if (Subtarget.hasAltivec()) {
     // First set operation action for all vector types to expand. Then we
     // will selectively turn on ones that can be effectively codegen'd.
     for (MVT VT : MVT::vector_valuetypes()) {
       // add/sub are legal for all supported vector VT's.
-      // This check is temporary until support for quadword add/sub is added
-      if (VT.SimpleTy != MVT::v1i128) {
-        setOperationAction(ISD::ADD , VT, Legal);
-        setOperationAction(ISD::SUB , VT, Legal);
-      }
-      else {
-        setOperationAction(ISD::ADD , VT, Expand);
-        setOperationAction(ISD::SUB , VT, Expand);
-      }
       
       // Vector instructions introduced in P8
       if (Subtarget.hasP8Altivec() && (VT.SimpleTy != MVT::v1i128)) {
         setOperationAction(ISD::CTPOP, VT, Legal);
         setOperationAction(ISD::CTLZ, VT, Legal);
       }
       else {
         setOperationAction(ISD::CTPOP, VT, Expand);
for (unsigned i = 0; i != NumOps; ++i) {
     default: llvm_unreachable("Unexpected ValueType for argument!");
     case MVT::i1:
     case MVT::i32:
     case MVT::i64:
       // These can be scalar arguments or elements of an integer array type
       // passed directly.  Clang may use those instead of "byval" aggregate
       // types to avoid forcing arguments to memory unnecessarily.
       if (GPR_idx != NumGPRs) {
         RegsToPass.push_back(std::make_pair(GPR[GPR_idx++], Arg));
       } else {
         if (CallConv == CallingConv::Fast)
           ComputePtrOff();
 
         LowerMemOpCallTo(DAG, MF, Chain, Arg, PtrOff, SPDiff, ArgOffset,
                          true, isTailCall, false, MemOpChains,
                          TailCallArguments, dl);
         if (CallConv == CallingConv::Fast)

lib/Target/PowerPC/PPCInstrAltivec.td

def VSRAD : VXForm_1<964, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                     "vsrad $vD, $vA, $vB", IIC_VecGeneral,
                     [(set v2i64:$vD, (sra v2i64:$vA, v2i64:$vB))]>;
 
 // Vector Integer Arithmetic Instructions
 let isCommutable = 1 in {
 def VADDUDM : VXForm_1<192, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                        "vaddudm $vD, $vA, $vB", IIC_VecGeneral,
                        [(set v2i64:$vD, (add v2i64:$vA, v2i64:$vB))]>;
+def VADDUQM : VXForm_1<256, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
+                       "vadduqm $vD, $vA, $vB", IIC_VecGeneral,
+                       [(set v1i128:$vD, (add v1i128:$vA, v1i128:$vB))]>;
 } // isCommutable
 
+// Vector Quadword Add
+def VADDEUQM : VA1a_Int_Ty<60, "vaddeuqm", int_ppc_altivec_vaddeuqm, v1i128>;
+def VADDCUQ  : VX1_Int_Ty<320, "vaddcuq", int_ppc_altivec_vaddcuq, v1i128>;
+def VADDECUQ : VA1a_Int_Ty<61, "vaddecuq", int_ppc_altivec_vaddecuq, v1i128>;
+
+// Vector Doubleword Subtract
 def VSUBUDM : VXForm_1<1216, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
                        "vsubudm $vD, $vA, $vB", IIC_VecGeneral,
                        [(set v2i64:$vD, (sub v2i64:$vA, v2i64:$vB))]>;
 
+// Vector Quadword Subtract
+def VSUBUQM : VXForm_1<1280, (outs vrrc:$vD), (ins vrrc:$vA, vrrc:$vB),
+                       "vsubuqm $vD, $vA, $vB", IIC_VecGeneral,
+                       [(set v1i128:$vD, (sub v1i128:$vA, v1i128:$vB))]>;
+def VSUBEUQM : VA1a_Int_Ty<62, "vsubeuqm", int_ppc_altivec_vsubeuqm, v1i128>;
+def VSUBCUQ  : VX1_Int_Ty<1344, "vsubcuq", int_ppc_altivec_vsubcuq, v1i128>;
+def VSUBECUQ : VA1a_Int_Ty<63, "vsubecuq", int_ppc_altivec_vsubecuq, v1i128>;
+
 // Count Leading Zeros
 def VCLZB : VXForm_2<1794, (outs vrrc:$vD), (ins vrrc:$vB),
                      "vclzb $vD, $vB", IIC_VecGeneral,
                      [(set v16i8:$vD, (ctlz v16i8:$vB))]>;
 def VCLZH : VXForm_2<1858, (outs vrrc:$vD), (ins vrrc:$vB),
                      "vclzh $vD, $vB", IIC_VecGeneral,
                      [(set v8i16:$vD, (ctlz v8i16:$vB))]>;
 def VCLZW : VXForm_2<1922, (outs vrrc:$vD), (ins vrrc:$vB),

test/CodeGen/PowerPC/ppc64-i128-abi.ll

 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=CHECK-LE
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s -check-prefix=CHECK-BE
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-NOVSX
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-NOVSX
 ; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-BE-NOVSX
 ; RUN: llc -mtriple=powerpc64le-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s -check-prefix=CHECK-LE-NOVSX
 
 @x = common global <1 x i128> zeroinitializer, align 16
 @y = common global <1 x i128> zeroinitializer, align 16
 @a = common global i128 zeroinitializer, align 16
 @b = common global i128 zeroinitializer, align 16
 
 ; VSX:
 ;   %a is passed in register 34
-;   On LE, ensure %a is swapped before being used (using xxswapd)
-;   Similarly, on LE ensure the results are swapped before being returned in 
-;   register 34
+;   The value of 1 is stored in the TOC.
+;   On LE, ensure the value of 1 is swapped before being used (using xxswapd).
 ; VMX (no VSX): 
 ;   %a is passed in register 2
-;   No swaps are necessary on LE
+;   The value of 1 is stored in the TOC.
+;   No swaps are necessary when using P8 Vector instructions on LE
 define <1 x i128> @v1i128_increment_by_one(<1 x i128> %a) nounwind {
        %tmp = add <1 x i128> %a, <i128 1>
        ret <1 x i128> %tmp  
 
+; FIXME: Seems a 128-bit literal is materialized by loading from the TOC. There
+;        should be a better way of doing this.
+
 ; CHECK-LE-LABEL: @v1i128_increment_by_one
-; CHECK-LE: xxswapd [[PARAM1:[0-9]+]], 34
-; CHECK-LE: stxvd2x [[PARAM1]], {{[0-9]+}}, {{[0-9]+}}
-; CHECK-LE: lxvd2x [[RESULT:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
-; CHECK-LE: xxswapd 34, [[RESULT]]
+; CHECK-LE: lxvd2x [[VAL:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
+; CHECK-LE: xxswapd 35, [[VAL]]
+; CHECK-LE: vadduqm 2, 2, 3
 ; CHECK-LE: blr
 
 ; CHECK-BE-LABEL: @v1i128_increment_by_one
-; CHECK-BE-NOT: xxswapd {{[0-9]+}}, 34
-; CHECK-BE: stxvd2x 34, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-BE: lxvd2x 34, {{[0-9]+}}, {{[0-9]+}}
+; CHECK-BE: lxvd2x 35, {{[0-9]+}}, {{[0-9]+}}
+; CHECK-BE-NOT: xxswapd 
+; CHECK-BE: vadduqm 2, 2, 3 
 ; CHECK-BE-NOT: xxswapd 34, {{[0-9]+}}
 ; CHECK-BE: blr
 
 ; CHECK-NOVSX-LABEL: @v1i128_increment_by_one
 ; CHECK-NOVSX-NOT: xxswapd {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-NOVSX-NOT: stxvd2x {{[0-9]+}}, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX: stvx 2, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX: lvx 2, {{[0-9]+}}, {{[0-9]+}}
+; CHECK-NOVSX: lvx [[VAL:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-NOVSX-NOT: lxvd2x {{[0-9]+}}, {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-NOVSX-NOT: xxswapd {{[0-9]+}}, {{[0-9]+}}
+; CHECK-NOVSX: vadduqm 2, 2, [[VAL]]
 ; CHECK-NOVSX: blr
 }
 
 ; VSX:
 ;   %a is passed in register 34
 ;   %b is passed in register 35
-;   On LE, ensure the contents of 34 and 35 are swapped before being used
-;   Similarly, on LE ensure the results are swapped before being returned in
-;   register 34
+;   No swaps are necessary when using P8 Vector instructions on LE
 ; VMX (no VSX):
 ;   %a is passewd in register 2
 ;   %b is passed in register 3
 ;   On LE, do not need to swap contents of 2 and 3 because the lvx/stvx 
 ;   instructions no not swap elements
 define <1 x i128> @v1i128_increment_by_val(<1 x i128> %a, <1 x i128> %b) nounwind {
        %tmp = add <1 x i128> %a, %b
        ret <1 x i128> %tmp
 
 ; CHECK-LE-LABEL: @v1i128_increment_by_val
-; CHECK-LE-DAG: xxswapd [[PARAM1:[0-9]+]], 34
-; CHECK-LE-DAG: xxswapd [[PARAM2:[0-9]+]], 35
-; CHECK-LE-DAG: stxvd2x [[PARAM1]], {{[0-9]+}}, {{[0-9]+}}
-; CHECK-LE-DAG: stxvd2x [[PARAM2]], {{[0-9]+}}, {{[0-9]+}}
-; CHECK-LE: lxvd2x [[RESULT:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
-; CHECK-LE: xxswapd 34, [[RESULT]]
+; CHECK-LE-NOT: xxswapd
+; CHECK-LE: adduqm 2, 2, 3
 ; CHECK-LE: blr
 
 ; CHECK-BE-LABEL: @v1i128_increment_by_val
 ; CHECK-BE-NOT: xxswapd {{[0-9]+}}, 34
 ; CHECK-BE-NOT: xxswapd {{[0-9]+}}, 35
-; CHECK-BE-DAG: stxvd2x 34, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-BE-DAG: stxvd2x 35, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-BE: lxvd2x [[RESULT:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-BE-NOT: xxswapd 34, [[RESULT]]
+; CHECK-BE: adduqm 2, 2, 3
 ; CHECK-BE: blr
 
 ; CHECK-NOVSX-LABEL: @v1i128_increment_by_val
-; CHECK-NOVSX-NOT: xxswapd {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX-NOT: xxswapd {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX-DAG: stvx 2, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX-DAG: stvx 3, {{[0-9]+}}, {{[0-9]+}}
-; CHECK-NOVSX: lvx [[RESULT:[0-9]+]], {{[0-9]+}}, {{[0-9]+}}
 ; CHECK-NOVSX-NOT: xxswapd 34, [[RESULT]]
+; CHECK-NOVSX: adduqm 2, 2, 3
 ; CHECK-NOVSX: blr
 }
 
 ; Little Endian (VSX and VMX):
 ;   Lower 64-bits of %a are passed in register 3
 ;   Upper 64-bits of %a are passed in register 4
 ;   Increment lower 64-bits using addic (immediate value of 1)
 ;   Increment upper 64-bits using add zero extended

test/CodeGen/PowerPC/vec_add_sub_quadword.ll

+; Check VMX 128-bit integer operations
+;
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 < %s | FileCheck %s
+; RUN: llc -mtriple=powerpc64-unknown-linux-gnu -mcpu=pwr8 -mattr=-vsx < %s | FileCheck %s
+
+define <1 x i128> @test_add(<1 x i128> %x, <1 x i128> %y) nounwind {
+       %result = add <1 x i128> %x, %y
+       ret <1 x i128> %result
+; CHECK-LABEL: @test_add
+; CHECK: vadduqm 2, 2, 3
+}
+
+define <1 x i128> @increment_by_one(<1 x i128> %x) nounwind {
+       %result = add <1 x i128> %x, <i128 1>
+       ret <1 x i128> %result
+; CHECK-LABEL: @increment_by_one
+; CHECK vadduqm 2, 2, 3
+}
+
+define <1 x i128> @increment_by_val(<1 x i128> %x, i128 %val) nounwind {
+       %tmpvec = insertelement <1 x i128> <i128 0>, i128 %val, i32 0
+       %tmpvec2 = insertelement <1 x i128> %tmpvec, i128 %val, i32 1
+       %result = add <1 x i128> %x, %tmpvec2
+       ret <1 x i128> %result
+; CHECK-LABEL: @increment_by_val
+; CHECK: vadduqm 2, 2, 3
+}
+
+define <1 x i128> @test_sub(<1 x i128> %x, <1 x i128> %y) nounwind {
+       %result = sub <1 x i128> %x, %y
+       ret <1 x i128> %result
+; CHECK-LABEL: @test_sub
+; CHECK: vsubuqm 2, 2, 3
+}
+
+define <1 x i128> @decrement_by_one(<1 x i128> %x) nounwind {
+       %result = sub <1 x i128> %x, <i128 1>
+       ret <1 x i128> %result
+; CHECK-LABEL: @decrement_by_one
+; CHECK vsubuqm 2, 2, 3
+}
+
+define <1 x i128> @decrement_by_val(<1 x i128> %x, i128 %val) nounwind {
+       %tmpvec = insertelement <1 x i128> <i128 0>, i128 %val, i32 0
+       %tmpvec2 = insertelement <1 x i128> %tmpvec, i128 %val, i32 1
+       %result = sub <1 x i128> %x, %tmpvec2
+       ret <1 x i128> %result
+; CHECK-LABEL: @decrement_by_val
+; CHECK vsubuqm   2, 2, 3
+}
+
+declare <1 x i128> @llvm.ppc.altivec.vaddeuqm(<1 x i128> %x, 
+                                              <1 x i128> %y, 
+                                              <1 x i128> %z) nounwind readnone
+declare <1 x i128> @llvm.ppc.altivec.vaddcuq(<1 x i128> %x, 
+                                             <1 x i128> %y) nounwind readnone
+declare <1 x i128> @llvm.ppc.altivec.vaddecuq(<1 x i128> %x, 
+                                              <1 x i128> %y, 
+                                              <1 x i128> %z) nounwind readnone
+declare <1 x i128> @llvm.ppc.altivec.vsubeuqm(<1 x i128> %x, 
+                                              <1 x i128> %y, 
+                                              <1 x i128> %z) nounwind readnone
+declare <1 x i128> @llvm.ppc.altivec.vsubcuq(<1 x i128> %x, 
+                                             <1 x i128> %y) nounwind readnone
+declare <1 x i128> @llvm.ppc.altivec.vsubecuq(<1 x i128> %x, 
+                                              <1 x i128> %y, 
+                                              <1 x i128> %z) nounwind readnone
+
+define <1 x i128> @test_vaddeuqm(<1 x i128> %x, 
+                               <1 x i128> %y, 
+                                 <1 x i128> %z) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vaddeuqm(<1 x i128> %x,
+                                                         <1 x i128> %y,
+                                                         <1 x i128> %z)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: @test_vaddeuqm
+; CHECK: vaddeuqm 2, 2, 3, 4
+}
+
+define <1 x i128> @test_vaddcuq(<1 x i128> %x, 
+                               <1 x i128> %y) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vaddcuq(<1 x i128> %x,
+                                                        <1 x i128> %y)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: @test_vaddcuq
+; CHECK: vaddcuq 2, 2, 3
+}
+
+define <1 x i128> @test_vaddecuq(<1 x i128> %x, 
+                               <1 x i128> %y, 
+                                 <1 x i128> %z) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vaddecuq(<1 x i128> %x,
+                                                         <1 x i128> %y,
+                                                         <1 x i128> %z)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: @test_vaddecuq
+; CHECK: vaddecuq 2, 2, 3, 4
+}
+
+define <1 x i128> @test_vsubeuqm(<1 x i128> %x, 
+                               <1 x i128> %y, 
+                                 <1 x i128> %z) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vsubeuqm(<1 x i128> %x,
+                                                         <1 x i128> %y,
+                                                         <1 x i128> %z)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: test_vsubeuqm
+; CHECK: vsubeuqm 2, 2, 3, 4
+}
+
+define <1 x i128> @test_vsubcuq(<1 x i128> %x, 
+                               <1 x i128> %y) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vsubcuq(<1 x i128> %x,
+                                                        <1 x i128> %y)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: test_vsubcuq
+; CHECK: vsubcuq 2, 2, 3
+}
+
+define <1 x i128> @test_vsubecuq(<1 x i128> %x, 
+                               <1 x i128> %y, 
+                                 <1 x i128> %z) nounwind {
+  %tmp = tail call <1 x i128> @llvm.ppc.altivec.vsubecuq(<1 x i128> %x,
+                                                         <1 x i128> %y,
+                                                         <1 x i128> %z)
+  ret <1 x i128> %tmp
+; CHECK-LABEL: test_vsubecuq
+; CHECK: vsubecuq 2, 2, 3, 4
+}
+

test/MC/Disassembler/PowerPC/ppc64-encoding-vmx.txt

 0x10 0x43 0x22 0x00
 
 # CHECK: vadduhs 2, 3, 4                 
 0x10 0x43 0x22 0x40
 
 # CHECK: vadduws 2, 3, 4                 
 0x10 0x43 0x22 0x80
 
+# CHECK: vadduqm 2, 3, 4
+0x10 0x43 0x21 0x00
+
+# CHECK: vaddeuqm 2, 3, 4, 5
+0x10 0x43 0x21 0x7c
+
+# CHECK: vaddcuq 2, 3, 4
+0x10 0x43 0x21 0x40
+
+# CHECK: vaddecuq 2, 3, 4, 5
+0x10 0x43 0x21 0x7d
+
 # CHECK: vsubcuw 2, 3, 4                 
 0x10 0x43 0x25 0x80
 
 # CHECK: vsubsbs 2, 3, 4                 
 0x10 0x43 0x27 0x00
 
 # CHECK: vsubshs 2, 3, 4                 
 0x10 0x43 0x27 0x40

 0x10 0x43 0x26 0x00
 
 # CHECK: vsubuhs 2, 3, 4                 
 0x10 0x43 0x26 0x40
 
 # CHECK: vsubuws 2, 3, 4                 
 0x10 0x43 0x26 0x80
 
+# CHECK: vsubuqm 2, 3, 4
+0x10 0x43 0x25 0x00
+
+# CHECK: vsubeuqm 2, 3, 4, 5
+0x10 0x43 0x21 0x7e
+
+# CHECK: vsubcuq 2, 3, 4
+0x10 0x43 0x25 0x40
+
+# CHECK: vsubecuq 2, 3, 4, 5
+0x10 0x43 0x21 0x7f
+
 # CHECK: vmulesb 2, 3, 4                 
 0x10 0x43 0x23 0x08
 
 # CHECK: vmulesh 2, 3, 4                 
 0x10 0x43 0x23 0x48
 
 # CHECK: vmulesw 2, 3, 4
 0x10 0x43 0x23 0x88

test/MC/PowerPC/ppc64-encoding-vmx.s

 # CHECK-LE: vaddubs 2, 3, 4                 # encoding: [0x00,0x22,0x43,0x10]
             vaddubs 2, 3, 4
 # CHECK-BE: vadduhs 2, 3, 4                 # encoding: [0x10,0x43,0x22,0x40]
 # CHECK-LE: vadduhs 2, 3, 4                 # encoding: [0x40,0x22,0x43,0x10]
             vadduhs 2, 3, 4
 # CHECK-BE: vadduws 2, 3, 4                 # encoding: [0x10,0x43,0x22,0x80]
 # CHECK-LE: vadduws 2, 3, 4                 # encoding: [0x80,0x22,0x43,0x10]
             vadduws 2, 3, 4
+# CHECK-BE: vadduqm 2, 3, 4                 # encoding: [0x10,0x43,0x21,0x00]
+# CHECK-LE: vadduqm 2, 3, 4                 # encoding: [0x00,0x21,0x43,0x10]
+            vadduqm 2, 3, 4
+# CHECK-BE: vaddeuqm 2, 3, 4, 5             # encoding: [0x10,0x43,0x21,0x7c]
+# CHECK-LE: vaddeuqm 2, 3, 4, 5             # encoding: [0x7c,0x21,0x43,0x10]
+            vaddeuqm 2, 3, 4, 5
+# CHECK-BE: vaddcuq 2, 3, 4                 # encoding: [0x10,0x43,0x21,0x40]
+# CHECK-LE: vaddcuq 2, 3, 4                 # encoding: [0x40,0x21,0x43,0x10]
+            vaddcuq 2, 3, 4
+# CHECK-BE: vaddecuq 2, 3, 4, 5             # encoding: [0x10,0x43,0x21,0x7d]
+# CHECK-LE: vaddecuq 2, 3, 4, 5             # encoding: [0x7d,0x21,0x43,0x10]
+            vaddecuq 2, 3, 4, 5   
   
 # CHECK-BE: vsubcuw 2, 3, 4                 # encoding: [0x10,0x43,0x25,0x80]
 # CHECK-LE: vsubcuw 2, 3, 4                 # encoding: [0x80,0x25,0x43,0x10]
             vsubcuw 2, 3, 4
 # CHECK-BE: vsubsbs 2, 3, 4                 # encoding: [0x10,0x43,0x27,0x00]
 # CHECK-LE: vsubsbs 2, 3, 4                 # encoding: [0x00,0x27,0x43,0x10]
             vsubsbs 2, 3, 4
 # CHECK-BE: vsubshs 2, 3, 4                 # encoding: [0x10,0x43,0x27,0x40]
 # CHECK-LE: vsubshs 2, 3, 4                 # encoding: [0x40,0x27,0x43,0x10]

 # CHECK-LE: vsububs 2, 3, 4                 # encoding: [0x00,0x26,0x43,0x10]
             vsububs 2, 3, 4
 # CHECK-BE: vsubuhs 2, 3, 4                 # encoding: [0x10,0x43,0x26,0x40]
 # CHECK-LE: vsubuhs 2, 3, 4                 # encoding: [0x40,0x26,0x43,0x10]
             vsubuhs 2, 3, 4
 # CHECK-BE: vsubuws 2, 3, 4                 # encoding: [0x10,0x43,0x26,0x80]
 # CHECK-LE: vsubuws 2, 3, 4                 # encoding: [0x80,0x26,0x43,0x10]
             vsubuws 2, 3, 4
+# CHECK-BE: vsubuqm 2, 3, 4                 # encoding: [0x10,0x43,0x25,0x00]
+# CHECK-LE: vsubuqm 2, 3, 4                 # encoding: [0x00,0x25,0x43,0x10]
+            vsubuqm 2, 3, 4
+# CHECK-BE: vsubeuqm 2, 3, 4, 5             # encoding: [0x10,0x43,0x21,0x7e]
+# CHECK-LE: vsubeuqm 2, 3, 4, 5             # encoding: [0x7e,0x21,0x43,0x10]
+            vsubeuqm 2, 3, 4, 5
+# CHECK-BE: vsubcuq 2, 3, 4                 # encoding: [0x10,0x43,0x25,0x40] 
+# CHECK-LE: vsubcuq 2, 3, 4                 # encoding: [0x40,0x25,0x43,0x10]
+            vsubcuq 2, 3, 4
+# CHECK-BE: vsubecuq 2, 3, 4, 5             # encoding: [0x10,0x43,0x21,0x7f]
+# CHECK-LE: vsubecuq 2, 3, 4, 5             # encoding: [0x7f,0x21,0x43,0x10]
+            vsubecuq 2, 3, 4, 5 
   
 # CHECK-BE: vmulesb 2, 3, 4                 # encoding: [0x10,0x43,0x23,0x08]
 # CHECK-LE: vmulesb 2, 3, 4                 # encoding: [0x08,0x23,0x43,0x10]
             vmulesb 2, 3, 4
 # CHECK-BE: vmulesh 2, 3, 4                 # encoding: [0x10,0x43,0x23,0x48]
 # CHECK-LE: vmulesh 2, 3, 4                 # encoding: [0x48,0x23,0x43,0x10]
             vmulesh 2, 3, 4
 # CHECK-BE: vmulesw 2, 3, 4                 # encoding: [0x10,0x43,0x23,0x88]
 # CHECK-LE: vmulesw 2, 3, 4                 # encoding: [0x88,0x23,0x43,0x10]

utils/TableGen/IntrinsicEmitter.cpp

enum IIT_Info {
   IIT_EXTEND_ARG = 24,
   IIT_TRUNC_ARG = 25,
   IIT_ANYPTR = 26,
   IIT_V1   = 27,
   IIT_VARARG = 28,
   IIT_HALF_VEC_ARG = 29,
   IIT_SAME_VEC_WIDTH_ARG = 30,
   IIT_PTR_TO_ARG = 31,
-  IIT_VEC_OF_PTRS_TO_ELT = 32
+  IIT_VEC_OF_PTRS_TO_ELT = 32,
+  IIT_I128 = 33
 };
 
 
 static void EncodeFixedValueType(MVT::SimpleValueType VT,
                                  std::vector<unsigned char> &Sig) {
   if (MVT(VT).isInteger()) {
     unsigned BitWidth = MVT(VT).getSizeInBits();
     switch (BitWidth) {
     default: PrintFatalError("unhandled integer type width in intrinsic!");
     case 1: return Sig.push_back(IIT_I1);
     case 8: return Sig.push_back(IIT_I8);
     case 16: return Sig.push_back(IIT_I16);
     case 32: return Sig.push_back(IIT_I32);
     case 64: return Sig.push_back(IIT_I64);
+    case 128: return Sig.push_back(IIT_I128);
     }
   }
 
   switch (VT) {
   default: PrintFatalError("unhandled MVT in intrinsic!");
   case MVT::f16: return Sig.push_back(IIT_F16);
   case MVT::f32: return Sig.push_back(IIT_F32);
   case MVT::f64: return Sig.push_back(IIT_F64);