[VE] VE Vector Predicated SDNode, vector add isel and tests

[simoll]

Summary

VE Vector Predicated (VVP) SDNodes form an intermediate layer between VE vector instructions and the initial SDNodes.

We introduce 'vvp_add' with isel and tests as the first of these VVP nodes. VVP nodes have a mask and explicit vector length operand, which we will make proper use of later.

[kaz7]

I add some comments inlined.

llvm/lib/Target/VE/VEISelLowering.cpp

1677

If you want to implement something generic method, you can check getBooleanContents(true, false) to check the immediate value of vector true. For the case of VE, it is 1 not -1.

llvm/lib/Target/VE/VEInstrInfo.td

248

Is there any definition about capitalization about PatLeaf? I used lozero, fplozero, and etc. Whole lower case. If capitalization is common, I'd like to modify exisiting PatLeaves.

llvm/lib/Target/VE/VVPInstrPatternsVec.td

36

I guess missing immediate variant is only "ivml". How about impelementing it?

On the other hand, we have another choice which is not to implement immediate variants anyway. Then, we can ask fold immediate to do so (Not implemented yet for vector instructions, though). We have too many immediate variants.

I think we can choose several ways here, 1) implement whole immediate variants, 2) not implement immediate variants anyway and use fold immediate, 3) implement some important immeidate variants only and use fold immeidate. Do you have any opinion here? I think 2 is fine if you would like to implement more VVP and don't have time to work in immediate variants.

In addition, fold immediate for vector instructions is not implemented yet, but soon. ;-)

llvm/lib/Target/VE/VVPNodes.inc

1 ↗

(On Diff #306438)

I have several suggestions.

1. How about using .def as a file extension, similar to include/llvm/BinaryFormat/ELFRelocs/VE.def and otehrs. I feel .inc is used for tablegen generated files.
2. How about using the method similar to others like '#define' and '#undef' at caller side.
3. You can write only VVP_MAP(VVP_ADD, ADD) in this file. Then, caller side can define VVP_MAP in several ways. e.g.,:

#define VVP_MAP(X, Y) TARGET_NODE_CASE(X)
#include ...
#undef VVP_MAP

or

#define VVP_MAP(X, Y) \
  case VEISD::X: \
  case ISD::Y: \
    return VEISD::X;
#include ...
#undef VVP_MAP

[simoll]

llvm/lib/Target/VE/VEISelLowering.cpp

1677

Using 1 for now.. i'll factor out stuff like this into a custom DAG class that wraps common dag.getNode/getConstant/, etc. cases later.

llvm/lib/Target/VE/VEInstrInfo.td

248

I was shying away from calling it true as it seemed too close to C++.. capitalizing it at least makes it stand out as something non-builtin.
I am not aware of capitalization rules for TableGen. Being consistent on our end is the least we can do however. Suggestions for a lower-case name?

llvm/lib/Target/VE/VVPInstrPatternsVec.td

36

Can we do 2), please? :) It makes sense on so many accounts

llvm/lib/Target/VE/VVPNodes.inc

1 ↗

(On Diff #306438)

I will rename this to VVPNodes.def and simplify it to use only a ADD_VVP_OP(VVP, ISD) macro.

Not sure about the macro-usage examples. I am already doing that, eg see the definition of the TARGET_NODE_NAME macro and include in VEISelLowering with this patch.

About caller/includer-side-only def/undef: the macros are undefined in the VVPNodes.inc/def file because when we include it, we only want to define a few of the macros that are actually used in the file to query specific properties (there will be more macros in the future). By undefining all macros in the def-file itself the includer is not poluted with macros they do not care about.

[kaz7]

llvm/lib/Target/VE/VEISelLowering.cpp

1677

Using 1 here is fine to me.

llvm/lib/Target/VE/VEInstrInfo.td

248

I have no real opinion, but suggest for a lower-case name for the consistency.

llvm/lib/Target/VE/VVPInstrPatternsVec.td

36

Thant's fine to me. It's better to remove existing pattern using "ivl" for the consistency. And it's much better to add some comments that we will use fold immediate in peephole optimizaiton for vector instructions having immediate operands.

llvm/lib/Target/VE/VVPNodes.inc

1 ↗

(On Diff #306438)

Wait. I was thinking examples containing only binary operators. So, I asked you to use single ADD_VVP_OP macro to simplify them.

However, I've understood what you want to write here is not only binary but also unary operators like below:

ADD_VVP_OP(VVP_ADD)       REGISTER_BINARY_VVP_OP(VVP_ADD, ADD)
ADD_VVP_OP(VVP_NOT)       REGISTER_UNARY_VVP_OP(VVP_NOT, NOT)

We cannot write ADD_VVP_OP(VVP_ADD, ADD) in this case since we want to separate binary and unary operators. So, my suggestion doesn't work in this case. Please leave macros as original your codes.

I still think we can simplify these macros even if we need to write binary, unary, and otehr operators, but I have no concrete examples at the moment.

[simoll]

• Simplified macros
• Keeping the vector inst+imm isel patterns and tests for now. When vector-immediate folding is upstream, we can remove that code

[simoll]

Removed immediate variants in isel and tests. Added comment for future immediate folding.

[kaz7]

Small checks like 80 columns and indentation. And one question about how you plan to implement test cases for fold immediates.

llvm/lib/Target/VE/VEISelLowering.cpp

259

Indentation.

llvm/lib/Target/VE/VEInstrInfo.td

248

Indentation before :

249

Additional empty line. It's OK if you intended it.

2229

Over 80 columns. Following several lines also has over 80 columns.

llvm/lib/Target/VE/VVPInstrInfo.td

2

80 columns.

11

Over 80 columns.

26

Over 80 columns.

llvm/lib/Target/VE/VVPInstrPatternsVec.td

21

Over 80 columns. Following lines also has over 80 columns.

llvm/test/CodeGen/VE/Vector/vec_add.ll

77

Removing regression test cases like add_iv_v256i64() means that you need to implement iv test cases for all vector instructions when you implement a FoldImmediate for all vector instructions. Just want to make sure that you intend it.

[kaz7]

llvm/test/CodeGen/VE/Vector/vec_add.ll

77

Just want to make sure that you intend it.

I meant that "Just want to make clear what you intend to do."

[simoll]

NFC, Adressed comments

llvm/test/CodeGen/VE/Vector/vec_add.ll

77

Not testing the immediate special case here is intentional since there neither is a code path to test - i think the test-cases-with-immediate should be added here with the patch for immediate folding pass.

[kaz7]

LGTM.

[simoll]

NFC. Attaching this here because the rebase required manual merging.

llvm/lib/Target/VE/VEISelLowering.h

enum NodeType : unsigned {
 
   MEMBARRIER, // Compiler barrier only; generate a no-op.
 
   VEC_BROADCAST,    // 0: scalar value, 1: VL
 
   CALL,            // A call instruction.
   RET_FLAG,        // Return with a flag operand.
   GLOBAL_BASE_REG, // Global base reg for PIC.
+
+  // VVP_* nodes.
+#define ADD_VVP_OP(VVP_NAME, ...) VVP_NAME,
+#include "VVPNodes.def"
 };
 }
 
 class VETargetLowering : public TargetLowering {
   const VESubtarget *Subtarget;
 
   void initRegisterClasses();
   void initSPUActions();
public:
   SDValue lowerSTORE(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerToTLSGeneralDynamicModel(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerVASTART(SDValue Op, SelectionDAG &DAG) const;
   SDValue lowerVAARG(SDValue Op, SelectionDAG &DAG) const;
 
   SDValue lowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   /// } Custom Lower
 
+  /// VVP Lowering {
+  SDValue lowerToVVP(SDValue Op, SelectionDAG &DAG) const;
+  /// } VVPLowering
+
   /// Custom DAGCombine {
   SDValue PerformDAGCombine(SDNode *N, DAGCombinerInfo &DCI) const override;
 
   SDValue combineTRUNCATE(SDNode *N, DAGCombinerInfo &DCI) const;
   /// } Custom DAGCombine
 
   SDValue withTargetFlags(SDValue Op, unsigned TF, SelectionDAG &DAG) const;
   SDValue makeHiLoPair(SDValue Op, unsigned HiTF, unsigned LoTF,

llvm/lib/Target/VE/VEISelLowering.cpp

void VETargetLowering::initSPUActions() {
 
   // Use custom inserter for ATOMIC_FENCE.
   setOperationAction(ISD::ATOMIC_FENCE, MVT::Other, Custom);
 
   /// } Atomic isntructions
 }
 
 void VETargetLowering::initVPUActions() {
-  for (MVT LegalVecVT : AllVectorVTs)
+  for (MVT LegalVecVT : AllVectorVTs) {
     setOperationAction(ISD::BUILD_VECTOR, LegalVecVT, Custom);
+    // Translate all vector instructions with legal element types to VVP_*
+    // nodes.
+    // TODO We will custom-widen into VVP_* nodes in the future. While we are
+    // buildling the infrastructure for this, we only do this for legal vector
+    // VTs.
+#define ADD_VVP_OP(VVP_NAME, ISD_NAME)                                         \
+  setOperationAction(ISD::ISD_NAME, LegalVecVT, Custom);
+#include "VVPNodes.def"
+  }
 }
 
 SDValue
 VETargetLowering::LowerReturn(SDValue Chain, CallingConv::ID CallConv,
                               bool IsVarArg,
                               const SmallVectorImpl<ISD::OutputArg> &Outs,
                               const SmallVectorImpl<SDValue> &OutVals,
                               const SDLoc &DL, SelectionDAG &DAG) const {
case VEISD::FIRST_NUMBER:
     TARGET_NODE_CASE(GETFUNPLT)
     TARGET_NODE_CASE(GETSTACKTOP)
     TARGET_NODE_CASE(GETTLSADDR)
     TARGET_NODE_CASE(MEMBARRIER)
     TARGET_NODE_CASE(CALL)
     TARGET_NODE_CASE(VEC_BROADCAST)
     TARGET_NODE_CASE(RET_FLAG)
     TARGET_NODE_CASE(GLOBAL_BASE_REG)
+
+    // Register the VVP_* SDNodes.
+#define ADD_VVP_OP(VVP_NAME, ...) TARGET_NODE_CASE(VVP_NAME)
+#include "VVPNodes.def"
   }
 #undef TARGET_NODE_CASE
   return nullptr;
 }
 
 EVT VETargetLowering::getSetCCResultType(const DataLayout &, LLVMContext &,
                                          EVT VT) const {
   return MVT::i32;
SDValue VETargetLowering::LowerOperation(SDValue Op, SelectionDAG &DAG) const {
   case ISD::BUILD_VECTOR:
     return lowerBUILD_VECTOR(Op, DAG);
   case ISD::STORE:
     return lowerSTORE(Op, DAG);
   case ISD::VASTART:
     return lowerVASTART(Op, DAG);
   case ISD::VAARG:
     return lowerVAARG(Op, DAG);
+
+#define ADD_BINARY_VVP_OP(VVP_NAME, ISD_NAME) case ISD::ISD_NAME:
+#include "VVPNodes.def"
+    return lowerToVVP(Op, DAG);
   }
 }
 /// } Custom Lower
 
 /// JumpTable for VE.
 ///
 ///   VE cannot generate relocatable symbol in jump table.  VE cannot
 ///   generate expressions using symbols in both text segment and data
bool VETargetLowering::hasAndNot(SDValue Y) const {
   // VE allows different immediate values for X and Y where ~X & Y.
   // Only simm7 works for X, and only mimm works for Y on VE.  However, this
   // function is used to check whether an immediate value is OK for and-not
   // instruction as both X and Y.  Generating additional instruction to
   // retrieve an immediate value is no good since the purpose of this
   // function is to convert a series of 3 instructions to another series of
   // 3 instructions with better parallelism.  Therefore, we return false
   // for all immediate values now.
   // FIXME: Change hasAndNot function to have two operands to make it work
   //        correctly with Aurora VE.
   if (isa<ConstantSDNode>(Y))
     return false;
 
   // It's ok for generic registers.
   return true;
 }
+
+/// \returns the VVP_* SDNode opcode corresponsing to \p OC.
+static Optional<unsigned> getVVPOpcode(unsigned OC) {
+  switch (OC) {
+#define ADD_VVP_OP(VVPNAME, SDNAME)                                            \
+  case VEISD::VVPNAME:                                                         \
+  case ISD::SDNAME:                                                            \
+    return VEISD::VVPNAME;
+#include "VVPNodes.def"
+  }
+  return None;
+}
+
+SDValue VETargetLowering::lowerToVVP(SDValue Op, SelectionDAG &DAG) const {
+  // Can we represent this as a VVP node.
+  auto OCOpt = getVVPOpcode(Op->getOpcode());
+  if (!OCOpt.hasValue())
+    return SDValue();
+  unsigned VVPOC = OCOpt.getValue();
+
+  // The representative and legalized vector type of this operation.
+  EVT OpVecVT = Op.getValueType();
+  EVT LegalVecVT = getTypeToTransformTo(*DAG.getContext(), OpVecVT);
+
+  // Materialize the VL parameter.
+  SDLoc DL(Op);
+  SDValue AVL = DAG.getConstant(OpVecVT.getVectorNumElements(), DL, MVT::i32);
+  MVT MaskVT = MVT::v256i1;
+  SDValue ConstTrue = DAG.getConstant(1, DL, MVT::i32);
+  SDValue Mask = DAG.getNode(VEISD::VEC_BROADCAST, DL, MaskVT,
+                             ConstTrue); // emit a VEISD::VEC_BROADCAST here.
+
+  // Categories we are interested in.
+  bool IsBinaryOp = false;
+
+  switch (VVPOC) {
+#define ADD_BINARY_VVP_OP(VVPNAME, ...)                                        \
+  case VEISD::VVPNAME:                                                         \
+    IsBinaryOp = true;                                                         \
+    break;
+#include "VVPNodes.def"
+  }
+
+  if (IsBinaryOp) {
+    assert(LegalVecVT.isSimple());
+    return DAG.getNode(VVPOC, DL, LegalVecVT, Op->getOperand(0),
+                       Op->getOperand(1), Mask, AVL);
+  }
+  llvm_unreachable("lowerToVVP called for unexpected SDNode.");
+}

llvm/lib/Target/VE/VEInstrInfo.td

 def lomsbzero   : PatLeaf<(imm), [{ return (N->getZExtValue() & 0x80000000)
                                       == 0; }]>;
 def lozero      : PatLeaf<(imm), [{ return (N->getZExtValue() & 0xffffffff)
                                       == 0; }]>;
 def fplomsbzero : PatLeaf<(fpimm), [{ return (getFpImmVal(N) & 0x80000000)
                                         == 0; }]>;
 def fplozero    : PatLeaf<(fpimm), [{ return (getFpImmVal(N) & 0xffffffff)
                                         == 0; }]>;
+def nonzero     : PatLeaf<(imm), [{ return N->getSExtValue() !=0 ; }]>;
 
 def CCSIOp : PatLeaf<(cond), [{
   switch (N->get()) {
   default:          return true;
   case ISD::SETULT:
   case ISD::SETULE:
   case ISD::SETUGT:
   case ISD::SETUGE: return false;
   }
def : Pat<(i32 (and i32:$val, 0xff)),
                      !add(56, 64)), sub_i32)>;
 def : Pat<(i32 (and i32:$val, 0xffff)),
           (EXTRACT_SUBREG
               (ANDrm (INSERT_SUBREG (i64 (IMPLICIT_DEF)), $val, sub_i32),
                      !add(48, 64)), sub_i32)>;
 def : Pat<(i64 (and i64:$val, 0xffffffff)),
           (ANDrm $val, !add(32, 64))>;
 
+//===----------------------------------------------------------------------===//
+// Vector Instruction Pattern Stuff
+//===----------------------------------------------------------------------===//
+
+// Custom intermediate ISDs.
+class IsVLVT<int OpIdx> : SDTCisVT<OpIdx,i32>;
+def vec_broadcast       : SDNode<"VEISD::VEC_BROADCAST", SDTypeProfile<1, 2,
+                                 [SDTCisVec<0>, IsVLVT<2>]>>;
+
+// Whether this is an all-true mask (assuming undef-bits above VL are all-true).
+def true_mask           : PatLeaf<
+                            (vec_broadcast (i32 nonzero), (i32 srcvalue))>;
+// Match any broadcast (ignoring VL).
+def any_broadcast       : PatFrag<(ops node:$sx),
+                                  (vec_broadcast node:$sx, (i32 srcvalue))>;
+
 // Vector instructions.
 include "VEInstrVec.td"
 
 // The vevlintrin
 include "VEInstrIntrinsicVL.td"
 
 // Patterns and intermediate SD nodes (VEC_*).
 include "VEInstrPatternsVec.td"
+
+// Patterns and intermediate SD nodes (VVP_*).
+include "VVPInstrPatternsVec.td"

llvm/lib/Target/VE/VEInstrPatternsVec.td

 // isel patterns.
 //
 //===----------------------------------------------------------------------===//
 
 //===----------------------------------------------------------------------===//
 // Instruction format superclass
 //===----------------------------------------------------------------------===//
 
-// Custom intermediate ISDs.
-class IsVLVT<int OpIdx> : SDTCisVT<OpIdx,i32>;
-def vec_broadcast   : SDNode<"VEISD::VEC_BROADCAST", SDTypeProfile<1, 2,  [SDTCisVec<0>, IsVLVT<2>]>>;
-
 multiclass vbrd_elem32<ValueType v32, ValueType s32, SDPatternOperator ImmOp, SDNodeXForm ImmCast, int SubRegIdx> {
   // VBRDil
   def : Pat<(v32 (vec_broadcast (s32 ImmOp:$sy), i32:$vl)),
             (VBRDil (ImmCast $sy), i32:$vl)>;
 
   // VBRDrl
   def : Pat<(v32 (vec_broadcast s32:$sy, i32:$vl)),
             (VBRDrl

llvm/lib/Target/VE/VVPInstrInfo.td

+//===-------------- VVPInstrInfo.td - VVP_* SDNode patterns ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the VE Vector Predicated SDNodes (VVP SDNodes).  VVP
+// SDNodes are an intermediate isel layer between the vector SDNodes emitted by
+// LLVM and the actual VE vector instructions. For example:
+//
+//  ADD(x,y)   -->   VVP_ADD(x,y,mask,evl)   -->   VADDSWSXrvml(x,y,mask,evl)
+//     ^                      ^                            ^
+//  The standard     The VVP layer SDNode.        The VE vector instruction.
+//  SDNode.
+//
+// TODO explain how VVP nodes relate to VP SDNodes once VP ISel is uptream.
+//===----------------------------------------------------------------------===//
+
+// Binary Operators {
+
+// BinaryOp(x,y,mask,vl)
+def SDTIntBinOpVVP : SDTypeProfile<1, 4, [     // vp_add, vp_and, etc.
+  SDTCisSameAs<0, 1>,
+  SDTCisSameAs<0, 2>,
+  SDTCisInt<0>,
+  SDTCisSameNumEltsAs<0, 3>,
+  IsVLVT<4>
+]>;
+
+// Binary operator commutative pattern.
+class vvp_commutative<SDNode RootOp> :
+  PatFrags<
+  (ops node:$lhs, node:$rhs, node:$mask, node:$vlen),
+  [(RootOp node:$lhs, node:$rhs, node:$mask, node:$vlen),
+   (RootOp node:$rhs, node:$lhs, node:$mask, node:$vlen)]>;
+
+// VVP node definitions.
+def vvp_add    : SDNode<"VEISD::VVP_ADD",  SDTIntBinOpVVP>;
+def c_vvp_add  : vvp_commutative<vvp_add>;
+
+// } Binary Operators

llvm/lib/Target/VE/VVPInstrPatternsVec.td

+//===----------- VVPInstrPatternsVec.td - VVP_* SDNode patterns -----------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file describes how VVP_* SDNodes are lowered to machine instructions.
+//
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+//
+// VVP SDNode definitions.
+//
+//===----------------------------------------------------------------------===//
+include "VVPInstrInfo.td"
+
+multiclass VectorBinaryArith<
+    SDPatternOperator OpNode,
+    ValueType ScalarVT, ValueType DataVT, ValueType MaskVT,
+    string OpBaseName,
+    SDPatternOperator ImmOp, SDNodeXForm ImmCast> {
+  // No mask.
+  def : Pat<(OpNode
+                (any_broadcast ScalarVT:$sx),
+                DataVT:$vy, (MaskVT true_mask), i32:$avl),
+            (!cast<Instruction>(OpBaseName#"rvl")
+                ScalarVT:$sx, $vy, $avl)>;
+  def : Pat<(OpNode DataVT:$vx, DataVT:$vy, (MaskVT true_mask), i32:$avl),
+            (!cast<Instruction>(OpBaseName#"vvl")
+                $vx, $vy, $avl)>;
+
+  // Mask.
+  def : Pat<(OpNode
+                (any_broadcast ScalarVT:$sx),
+                DataVT:$vy, MaskVT:$mask, i32:$avl),
+            (!cast<Instruction>(OpBaseName#"rvml")
+                ScalarVT:$sx, $vy, $mask, $avl)>;
+  def : Pat<(OpNode DataVT:$vx, DataVT:$vy, MaskVT:$mask, i32:$avl),
+            (!cast<Instruction>(OpBaseName#"vvml")
+                $vx, $vy, $mask, $avl)>;
+
+  // TODO We do not specify patterns for the immediate variants here. There
+  // will be an immediate folding pass that takes care of switching to the
+  // immediate variant where applicable.
+
+  // TODO Fold vvp_select into passthru.
+}
+
+// Expand both 64bit and 32 bit variant (256 elements)
+multiclass VectorBinaryArith_ShortLong<
+    SDPatternOperator OpNode,
+    ValueType LongScalarVT, ValueType LongDataVT, string LongOpBaseName,
+    ValueType ShortScalarVT, ValueType ShortDataVT, string ShortOpBaseName> {
+  defm : VectorBinaryArith<OpNode,
+                           LongScalarVT, LongDataVT, v256i1,
+                           LongOpBaseName, simm7, LO7>;
+  defm : VectorBinaryArith<OpNode,
+                           ShortScalarVT, ShortDataVT, v256i1,
+                           ShortOpBaseName, simm7, LO7>;
+}
+
+
+defm : VectorBinaryArith_ShortLong<c_vvp_add,
+                                   i64, v256i64, "VADDSL",
+                                   i32, v256i32, "VADDSWSX">;

llvm/lib/Target/VE/VVPNodes.def

+//===-- VVPNodes.def - Lists & properties of VE Vector Predication Nodes --===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines all VVP_* SDNodes and their properties
+//
+//===----------------------------------------------------------------------===//
+
+/// ADD_VVP_OP(VVPNAME,SDNAME)
+/// \p VVPName is a VVP SDNode operator.
+/// \p SDNAME is the generic SD opcode corresponding to \p VVPName.
+#ifndef ADD_VVP_OP
+#define ADD_VVP_OP(X, Y)
+#endif
+
+/// ADD_BINARY_VVP_OP(VVPNAME,SDNAME)
+/// \p VVPName is a VVP Binary operator.
+/// \p SDNAME is the generic SD opcode corresponding to \p VVPName.
+#ifndef ADD_BINARY_VVP_OP
+#define ADD_BINARY_VVP_OP(X,Y) ADD_VVP_OP(X,Y)
+#endif
+
+// Integer arithmetic.
+ADD_BINARY_VVP_OP(VVP_ADD,ADD)
+
+
+#undef ADD_BINARY_VVP_OP
+#undef ADD_VVP_OP

llvm/test/CodeGen/VE/Vector/vec_add.ll

+; RUN: llc < %s -mtriple=ve -mattr=+vpu | FileCheck %s
+
+; <256 x i32>
+
+; Function Attrs: nounwind
+define fastcc <256 x i32> @add_vv_v256i32(<256 x i32> %x, <256 x i32> %y) {
+; CHECK-LABEL: add_vv_v256i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s0, 256
+; CHECK-NEXT: lvl %s0
+; CHECK-NEXT: vadds.w.sx %v0, %v0, %v1
+; CHECK-NEXT: b.l.t (, %s10)
+  %z = add <256 x i32> %x, %y
+  ret <256 x i32> %z
+}
+
+; Function Attrs: nounwind
+define fastcc <256 x i32> @add_sv_v256i32(i32 %x, <256 x i32> %y) {
+; CHECK-LABEL: add_sv_v256i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: adds.w.sx %s0, %s0, (0)1
+; CHECK-NEXT: lea %s1, 256
+; CHECK-NEXT: lvl %s1
+; CHECK-NEXT: vadds.w.sx %v0, %s0, %v0
+; CHECK-NEXT: b.l.t (, %s10)
+  %xins = insertelement <256 x i32> undef, i32 %x, i32 0
+  %vx = shufflevector <256 x i32> %xins, <256 x i32> undef, <256 x i32> zeroinitializer
+  %z = add <256 x i32> %vx, %y
+  ret <256 x i32> %z
+}
+
+; Function Attrs: nounwind
+define fastcc <256 x i32> @add_vs_v256i32(<256 x i32> %x, i32 %y) {
+; CHECK-LABEL: add_vs_v256i32:
+; CHECK: # %bb.0:
+; CHECK-NEXT: adds.w.sx %s0, %s0, (0)1
+; CHECK-NEXT: lea %s1, 256
+; CHECK-NEXT: lvl %s1
+; CHECK-NEXT: vadds.w.sx %v0, %s0, %v0
+; CHECK-NEXT: b.l.t (, %s10)
+  %yins = insertelement <256 x i32> undef, i32 %y, i32 0
+  %vy = shufflevector <256 x i32> %yins, <256 x i32> undef, <256 x i32> zeroinitializer
+  %z = add <256 x i32> %x, %vy
+  ret <256 x i32> %z
+}
+
+
+
+; <256 x i64>
+
+; Function Attrs: nounwind
+define fastcc <256 x i64> @add_vv_v256i64(<256 x i64> %x, <256 x i64> %y) {
+; CHECK-LABEL: add_vv_v256i64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s0, 256
+; CHECK-NEXT: lvl %s0
+; CHECK-NEXT: vadds.l %v0, %v0, %v1
+; CHECK-NEXT: b.l.t (, %s10)
+  %z = add <256 x i64> %x, %y
+  ret <256 x i64> %z
+}
+
+; Function Attrs: nounwind
+define fastcc <256 x i64> @add_sv_v256i64(i64 %x, <256 x i64> %y) {
+; CHECK-LABEL: add_sv_v256i64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s1, 256
+; CHECK-NEXT: lvl %s1
+; CHECK-NEXT: vadds.l %v0, %s0, %v0
+; CHECK-NEXT: b.l.t (, %s10)
+  %xins = insertelement <256 x i64> undef, i64 %x, i32 0
+  %vx = shufflevector <256 x i64> %xins, <256 x i64> undef, <256 x i32> zeroinitializer
+  %z = add <256 x i64> %vx, %y
+  ret <256 x i64> %z
+}
+
+; Function Attrs: nounwind
+define fastcc <256 x i64> @add_vs_v256i64(<256 x i64> %x, i64 %y) {
+; CHECK-LABEL: add_vs_v256i64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s1, 256
+; CHECK-NEXT: lvl %s1
+; CHECK-NEXT: vadds.l %v0, %s0, %v0
+; CHECK-NEXT: b.l.t (, %s10)
+  %yins = insertelement <256 x i64> undef, i64 %y, i32 0
+  %vy = shufflevector <256 x i64> %yins, <256 x i64> undef, <256 x i32> zeroinitializer
+  %z = add <256 x i64> %x, %vy
+  ret <256 x i64> %z
+}
+
+; <128 x i64>
+; We expect this to be widened.
+
+; Function Attrs: nounwind
+define fastcc <128 x i64> @add_vv_v128i64(<128 x i64> %x, <128 x i64> %y) {
+; CHECK-LABEL: add_vv_v128i64:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s0, 256
+; CHECK-NEXT: lvl %s0
+; CHECK-NEXT: vadds.l %v0, %v0, %v1
+; CHECK-NEXT: b.l.t (, %s10)
+  %z = add <128 x i64> %x, %y
+  ret <128 x i64> %z
+}
+
+; <256 x i16>
+; We expect promotion.
+
+; Function Attrs: nounwind
+define fastcc <256 x i16> @add_vv_v256i16(<256 x i16> %x, <256 x i16> %y) {
+; CHECK-LABEL: add_vv_v256i16:
+; CHECK: # %bb.0:
+; CHECK-NEXT: lea %s0, 256
+; CHECK-NEXT: lvl %s0
+; CHECK-NEXT: vadds.w.sx %v0, %v0, %v1
+; CHECK-NEXT: b.l.t (, %s10)
+  %z = add <256 x i16> %x, %y
+  ret <256 x i16> %z
+}
+
+; <128 x i16>
+; We expect this to be scalarized (for now).
+
+; Function Attrs: nounwind
+define fastcc <128 x i16> @add_vv_v128i16(<128 x i16> %x, <128 x i16> %y) {
+; CHECK-LABEL: add_vv_v128i16:
+; CHECK: # %bb.0:
+; CHECK-NOT: vadd
+  %z = add <128 x i16> %x, %y
+  ret <128 x i16> %z
+}
+