[AArch64][SVE] Add SPLAT_VECTOR ISD Node

[huntergr]

Summary

Adds a new ISD node to replicate a scalar value across all elements of a vector. This is needed for scalable vectors, since BUILD_VECTOR cannot be used.

At present I only use this node for scalable vectors; although the default expand works and generates a build vector with the appropriate value, some backends expect a different canonical form in SDag for a splat. I suspect the code to generate those shuffles will need to be replicated in the expansion (or possibly moved to a shared function) before this can be used by default for all backends.

I've only added legalization for i8/i16/i32/i64 types in AArch64 for now.

[rkruppe]

This comment was removed by rkruppe.

[huntergr]

Needs to be updated with better isel patterns (and possibly a new ISD?), instead of just the bare minimum required for demonstration.

[huntergr]

• Removed intrinsic and supporting code
• Added new ISD as discussed on llvm-dev: http://lists.llvm.org/pipermail/llvm-dev/2019-August/134809.html

[jmolloy]

Hi Graham,

I really like the addition of a new ISD opcode for this. I do have one inline comment.

James

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3544 ↗

(On Diff #222386)

It feels to me like this is a canonicalization that should be a DAGCombine rather than in the builder. Is this here specifically because the ISD::vector_shuffle is not representable with scalable vectors?

If not, I'd prefer this be moved to a canonical dag combine. In general, I feel logic here should be at a minimum because it runs before targets can override or pattern match themselves.

[huntergr]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3544 ↗

(On Diff #222386)

It's because BUILD_VECTOR cannot be used for scalable vectors, and if we tried to use VECTOR_SHUFFLE, we'd still need to generate a mask with an unknown number of elements which we can't do at the moment.

I've limited it to scalable vectors only for now precisely because this does break some unit tests if I enable it for all targets -- they do get a splat, but not the expected splat. It may be better to move this code to SelectionDAG::getVectorShuffle near the code that checks TLI->hasVectorBlend(), which also seems to match and transform splats early on.

We do have a VECTOR_SHUFFLE_VAR node downstream which takes a third argument as a mask, and with nodes for vscale and stepvector we're able to generate the masks we need -- but that's for a separate discussion and patch.

[jmolloy]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3544 ↗

(On Diff #222386)

I suppose an extension of my comment is that this introduces a super-cool new node that a bunch of backends have wanted for a while, but doesn't plumb in any canonicalization to it for targets that aren't scalable vectors.

[huntergr]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3544 ↗

(On Diff #222386)

Is there a specific change you'd like to see? I'm not quite sure what's needed at this point.

[jmolloy]

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3544 ↗

(On Diff #222386)

The specific change I'd like to see is canonicalization of BUILD_VECTOR<x,x,x,x,x...> into SPLAT_VECTOR in DAGCombine. I'm opposed to taking a canonical pattern (BUILD_VECTOR<x,x,x,...>) and adding a new node to represent it (SPLAT_VECTOR) without canonicalizing the old pattern to the new node.

I think this will cause duplication of patterns in DAGCombine where BUILD_VECTOR<> has a set of optimizations and SPLAT_VECTOR has a different set, for no reason.

[huntergr]

Hi James,

So I implemented a combine to do this, but ran into a problem -- namely that the design suggested by @hfinkel in the llvm-dev discussion (to expand to build_vector when the backend hasn't marked the splat_vector operation as supported) conflicts with the combine and ends up in an infinite loop.

For now I've guarded the combine with a check on whether the operation action has been changed from Expand for that type. Would this be acceptable?

[huntergr]

• Added DAG combine to canonicalize a build_vector splat into a splat_vector if the target supports the splat_vector operation (by overriding the default Expand legalization action)

[huntergr]

Ping.

[lebedev.ri]

This needs more tests - other targets, some legalization tests

[RKSimon]

llvm/include/llvm/CodeGen/ISDOpcodes.h

409 ↗

(On Diff #223004)

Please can you add info on implicit truncation (same as we do for SCALAR_TO_VECTOR/BUILD_VECTOR/INSERT_VECTOR_ELT)

[jmolloy]

Thanks Graham for taking my feedback on board. I agree with Simon's request too.

I'm not convinced I agree with Roman's requests for tests on more targets; this feature is opt-in and the canonicalization is only enabled when a target marks it non-expand, which no target has apart from AArch64.

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17515 ↗

(On Diff #223004)

nit: Extraneous parens around the cast<>.

17516 ↗

(On Diff #223004)

This is probably worth a comment. I suppose we don't convert if the splatted value is undef because the entire build_vector is undef and you haven't written all the canonicalizations for that?

llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

1369 ↗

(On Diff #223004)

I know you've copied the formatting from above, but this formatting is weird and should be clang-formatted.

llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

3558 ↗

(On Diff #223004)

Can we have a comment that the DAGCombiner will perform build_vector->splat_vector conversion for non-scalable types?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

806 ↗

(On Diff #223004)

This if-nest is getting a bit deep, could these two ifs be merged?

[RKSimon]

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

17516 ↗

(On Diff #223004)

This if() might be better as an assert - we have the ISD::allOperandsUndef(N) test earlier in visitBUILD_VECTOR.

[huntergr]

• Fixed legalization action code, which was trying to use widening for all integer scalable vector types after the new MVT ranges were added
• Added unit tests for legalizing the result via promotion
• Minor cleanups, additional comments as requested

include/llvm/IR/Instructions.h

public:
   int getMaskValue(unsigned Elt) const {
     return getMaskValue(getMask(), Elt);
   }
 
   /// Convert the input shuffle mask operand to a vector of integers. Undefined
   /// elements of the mask are returned as -1.
   static void getShuffleMask(Constant *Mask, SmallVectorImpl<int> &Result);
 
+  static bool getShuffleMask(Value *Mask, SmallVectorImpl<int> &Result);
+
   /// Return the mask for this instruction as a vector of integers. Undefined
   /// elements of the mask are returned as -1.
   void getShuffleMask(SmallVectorImpl<int> &Result) const {
     return getShuffleMask(getMask(), Result);
   }
 
   SmallVector<int, 16> getShuffleMask() const {
     SmallVector<int, 16> Mask;

include/llvm/IR/Intrinsics.td

 def int_experimental_vector_stepvector : Intrinsic<[llvm_anyvector_ty],
                                                    [],
                                                    [IntrNoMem]>;
 
 def int_experimental_vector_vscale : Intrinsic<[llvm_anyint_ty],
                                                [],
                                                [IntrNoMem]>;
 
+def int_experimental_vector_splatvector : Intrinsic<[llvm_anyvector_ty],
+                                                    [LLVMVectorElementType<0>],
+                                                    [IntrNoMem]>;
+
 //===----------------------------------------------------------------------===//
 // Target-specific intrinsics
 //===----------------------------------------------------------------------===//
 
 include "llvm/IR/IntrinsicsPowerPC.td"
 include "llvm/IR/IntrinsicsX86.td"
 include "llvm/IR/IntrinsicsARM.td"
 include "llvm/IR/IntrinsicsAArch64.td"
 include "llvm/IR/IntrinsicsXCore.td"
 include "llvm/IR/IntrinsicsHexagon.td"
 include "llvm/IR/IntrinsicsNVVM.td"
 include "llvm/IR/IntrinsicsMips.td"
 include "llvm/IR/IntrinsicsAMDGPU.td"
 include "llvm/IR/IntrinsicsBPF.td"
 include "llvm/IR/IntrinsicsSystemZ.td"
 include "llvm/IR/IntrinsicsWebAssembly.td"

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

if (const ConstantVector *CV = dyn_cast<ConstantVector>(C)) {
       EVT EltVT =
           TLI.getValueType(DAG.getDataLayout(), VecTy->getElementType());
 
       SDValue Op;
       if (EltVT.isFloatingPoint())
         Op = DAG.getConstantFP(0, getCurSDLoc(), EltVT);
       else
         Op = DAG.getConstant(0, getCurSDLoc(), EltVT);
+
+      if (VT.isScalableVector()) {
+        auto INum = DAG.getConstant(Intrinsic::experimental_vector_splatvector,
+                                    getCurSDLoc(), MVT::i32);
+
+        auto Splat = DAG.getNode(ISD::INTRINSIC_WO_CHAIN, getCurSDLoc(), VT,
+                                 INum, Op);
+
+        return Splat;
+      }
+
       Ops.assign(NumElements, Op);
     }
 
     // Create a BUILD_VECTOR node.
     return NodeMap[V] = DAG.getBuildVector(VT, getCurSDLoc(), Ops);
   }
 
   // If this is a static alloca, generate it as the frameindex instead of
void SelectionDAGBuilder::visitExtractElement(const User &I) {
   setValue(&I, DAG.getNode(ISD::EXTRACT_VECTOR_ELT, getCurSDLoc(),
                            TLI.getValueType(DAG.getDataLayout(), I.getType()),
                            InVec, InIdx));
 }
 
 void SelectionDAGBuilder::visitShuffleVector(const User &I) {
   SDValue Src1 = getValue(I.getOperand(0));
   SDValue Src2 = getValue(I.getOperand(1));
+  Value *MaskV = I.getOperand(2);
   SDLoc DL = getCurSDLoc();
 
-  SmallVector<int, 8> Mask;
-  ShuffleVectorInst::getShuffleMask(cast<Constant>(I.getOperand(2)), Mask);
-  unsigned MaskNumElts = Mask.size();
-
   const TargetLowering &TLI = DAG.getTargetLoweringInfo();
   EVT VT = TLI.getValueType(DAG.getDataLayout(), I.getType());
+  bool IsScalable = VT.isScalableVector();
   EVT SrcVT = Src1.getValueType();
   unsigned SrcNumElts = SrcVT.getVectorNumElements();
 
+  SmallVector<int, 8> Mask;
+  if (!ShuffleVectorInst::getShuffleMask(MaskV, Mask)) {
+    SDValue Mask = getValue(I.getOperand(2));
+    unsigned NumElts = VT.getVectorNumElements();
+    // We don't currently support variable shuffles on fixed-length vectors
+    assert(IsScalable && "Non-constant shuffle mask on fixed-length vector");
+
+    // We haven't introduced a vector_shuffle_var intrinsic to support shuffles
+    // where we need to extract or merge vectors.
+    if (NumElts != SrcNumElts)
+      llvm_unreachable("Haven't implemented VECTOR_SHUFFLE_VAR intrinsic yet");
+
+    // Currently only handling splats of a single value for scalable vectors
+    if (auto *CMask = dyn_cast<Constant>(MaskV))
+      if (CMask->isNullValue()) {
+        // Splat of first element.
+        auto FirstElt = DAG.getNode(ISD::EXTRACT_VECTOR_ELT, DL,
+                                    SrcVT.getScalarType(), Src1,
+                                    DAG.getConstant(0, DL,
+                                    TLI.getVectorIdxTy(DAG.getDataLayout())));
+
+        auto INum = DAG.getConstant(Intrinsic::experimental_vector_splatvector,
+                                    DL, MVT::i32);
+
+        auto Splat = DAG.getNode(ISD::INTRINSIC_WO_CHAIN, DL, VT,
+                                 INum, FirstElt);
+
+        setValue(&I, Splat);
+        return;
+      }
+    llvm_unreachable("Haven't implemented VECTOR_SHUFFLE_VAR intrinsic yet");
+    return;
+  }
+
+  unsigned MaskNumElts = Mask.size();
   if (SrcNumElts == MaskNumElts) {
     setValue(&I, DAG.getVectorShuffle(VT, DL, Src1, Src2, Mask));
     return;
   }
 
   // Normalize the shuffle vector since mask and vector length don't match.
   if (SrcNumElts < MaskNumElts) {
     // Mask is longer than the source vectors. We can use concatenate vector to

lib/IR/ConstantFold.cpp

Constant *llvm::ConstantFoldExtractElementInstruction(Constant *Val,
   if (Val->isNullValue())  // ee(zero, x) -> zero
     return Constant::getNullValue(Val->getType()->getVectorElementType());
   // ee({w,x,y,z}, undef) -> undef
   if (isa<UndefValue>(Idx))
     return UndefValue::get(Val->getType()->getVectorElementType());
 
   if (ConstantInt *CIdx = dyn_cast<ConstantInt>(Idx)) {
     // ee({w,x,y,z}, wrong_value) -> undef
+    if (!Val->getType()->getVectorIsScalable())
       if (CIdx->uge(Val->getType()->getVectorNumElements()))
         return UndefValue::get(Val->getType()->getVectorElementType());
     return Val->getAggregateElement(CIdx->getZExtValue());
   }
   return nullptr;
 }
 
 Constant *llvm::ConstantFoldInsertElementInstruction(Constant *Val,
                                                      Constant *Elt,
                                                      Constant *Idx) {
   if (isa<UndefValue>(Idx))
     return UndefValue::get(Val->getType());
 
+  // Everything after this point assumes you can iterate across Val.
+  if (Val->getType()->getVectorIsScalable())
+    return nullptr;
+
   ConstantInt *CIdx = dyn_cast<ConstantInt>(Idx);
   if (!CIdx) return nullptr;
 
   unsigned NumElts = Val->getType()->getVectorNumElements();
   if (CIdx->uge(NumElts))
     return UndefValue::get(Val->getType());
 
   SmallVector<Constant*, 16> Result;
Constant *llvm::ConstantFoldInsertElementInstruction(Constant *Val,
   }
 
   return ConstantVector::get(Result);
 }
 
 Constant *llvm::ConstantFoldShuffleVectorInstruction(Constant *V1,
                                                      Constant *V2,
                                                      Constant *Mask) {
-  unsigned MaskNumElts = Mask->getType()->getVectorNumElements();
+  auto *MaskTy = cast<VectorType>(Mask->getType());
+  auto MaskNumElts = MaskTy->getElementCount();
   Type *EltTy = V1->getType()->getVectorElementType();
 
   // Undefined shuffle mask -> undefined value.
   if (isa<UndefValue>(Mask))
     return UndefValue::get(VectorType::get(EltTy, MaskNumElts));
 
   // Don't break the bitcode reader hack.
   if (isa<ConstantExpr>(Mask)) return nullptr;
 
+  if (MaskTy->isScalable()) {
+    // Is splat?
+    if (Mask->isNullValue()) {
+      Constant *Zero = Constant::getNullValue(MaskTy->getElementType());
+      Constant *SplatVal = ConstantFoldExtractElementInstruction(V1, Zero);
+      // Is splat of zero?
+      if (SplatVal && SplatVal->isNullValue())
+        return Constant::getNullValue(VectorType::get(EltTy, MaskNumElts));
+    }
+    return nullptr;
+  }
+
   unsigned SrcNumElts = V1->getType()->getVectorNumElements();
 
   // Loop over the shuffle mask, evaluating each element.
   SmallVector<Constant*, 32> Result;
-  for (unsigned i = 0; i != MaskNumElts; ++i) {
+  for (unsigned i = 0; i != MaskNumElts.Min; ++i) {
     int Elt = ShuffleVectorInst::getMaskValue(Mask, i);
     if (Elt == -1) {
       Result.push_back(UndefValue::get(EltTy));
       continue;
     }
     Constant *InElt;
     if (unsigned(Elt) >= SrcNumElts*2)
       InElt = UndefValue::get(EltTy);

lib/IR/Constants.cpp

Constant *ConstantExpr::getShuffleVector(Constant *V1, Constant *V2,
                                          Constant *Mask, Type *OnlyIfReducedTy) {
   assert(ShuffleVectorInst::isValidOperands(V1, V2, Mask) &&
          "Invalid shuffle vector constant expr operands!");
 
   if (Constant *FC = ConstantFoldShuffleVectorInstruction(V1, V2, Mask))
     return FC;          // Fold a few common cases.
 
   unsigned NElts = Mask->getType()->getVectorNumElements();
+  bool Scalable = Mask->getType()->getVectorIsScalable();
   Type *EltTy = V1->getType()->getVectorElementType();
-  Type *ShufTy = VectorType::get(EltTy, NElts);
+  Type *ShufTy = VectorType::get(EltTy, NElts, Scalable);
 
   if (OnlyIfReducedTy == ShufTy)
     return nullptr;
 
   // Look up the constant in the table first to ensure uniqueness
   Constant *ArgVec[] = { V1, V2, Mask };
   const ConstantExprKeyType Key(Instruction::ShuffleVector, ArgVec);
 

lib/IR/ConstantsContext.h

 /// ShuffleVectorConstantExpr - This class is private to
 /// Constants.cpp, and is used behind the scenes to implement
 /// shufflevector constant exprs.
 class ShuffleVectorConstantExpr : public ConstantExpr {
 public:
   ShuffleVectorConstantExpr(Constant *C1, Constant *C2, Constant *C3)
   : ConstantExpr(VectorType::get(
                    cast<VectorType>(C1->getType())->getElementType(),
-                   cast<VectorType>(C3->getType())->getNumElements()),
+                   cast<VectorType>(C3->getType())->getElementCount()),
                  Instruction::ShuffleVector,
                  &Op<0>(), 3) {
     Op<0>() = C1;
     Op<1>() = C2;
     Op<2>() = C3;
   }
 
   // allocate space for exactly three operands

lib/IR/Instructions.cpp

 //===----------------------------------------------------------------------===//
 //                      ShuffleVectorInst Implementation
 //===----------------------------------------------------------------------===//
 
 ShuffleVectorInst::ShuffleVectorInst(Value *V1, Value *V2, Value *Mask,
                                      const Twine &Name,
                                      Instruction *InsertBefore)
 : Instruction(VectorType::get(cast<VectorType>(V1->getType())->getElementType(),
-                cast<VectorType>(Mask->getType())->getNumElements()),
+                cast<VectorType>(Mask->getType())->getElementCount()),
               ShuffleVector,
               OperandTraits<ShuffleVectorInst>::op_begin(this),
               OperandTraits<ShuffleVectorInst>::operands(this),
               InsertBefore) {
   assert(isValidOperands(V1, V2, Mask) &&
          "Invalid shuffle vector instruction operands!");
   Op<0>() = V1;
   Op<1>() = V2;
   Op<2>() = Mask;
   setName(Name);
 }
 
 ShuffleVectorInst::ShuffleVectorInst(Value *V1, Value *V2, Value *Mask,
                                      const Twine &Name,
                                      BasicBlock *InsertAtEnd)
 : Instruction(VectorType::get(cast<VectorType>(V1->getType())->getElementType(),
-                cast<VectorType>(Mask->getType())->getNumElements()),
+                cast<VectorType>(Mask->getType())->getElementCount()),
               ShuffleVector,
               OperandTraits<ShuffleVectorInst>::op_begin(this),
               OperandTraits<ShuffleVectorInst>::operands(this),
               InsertAtEnd) {
   assert(isValidOperands(V1, V2, Mask) &&
          "Invalid shuffle vector instruction operands!");
 
   Op<0>() = V1;
void ShuffleVectorInst::getShuffleMask(Constant *Mask,
   }    
   for (unsigned i = 0; i != NumElts; ++i) {
     Constant *C = Mask->getAggregateElement(i);
     Result.push_back(isa<UndefValue>(C) ? -1 :
                      cast<ConstantInt>(C)->getZExtValue());
   }
 }
 
+bool ShuffleVectorInst::getShuffleMask(Value *Mask,
+                                       SmallVectorImpl<int> &Result) {
+  VectorType *VecTy = cast<VectorType>(Mask->getType());
+  if (VecTy->isScalable())
+    return false;
+
+  if (auto *CMask = dyn_cast<Constant>(Mask)) {
+    getShuffleMask(CMask, Result);
+    return true;
+  }
+
+  return false;
+}
+
 //===----------------------------------------------------------------------===//
 //                             InsertValueInst Class
 //===----------------------------------------------------------------------===//
 
 void InsertValueInst::init(Value *Agg, Value *Val, ArrayRef<unsigned> Idxs, 
                            const Twine &Name) {
   assert(getNumOperands() == 2 && "NumOperands not initialized?");
 

lib/Target/AArch64/AArch64ISelLowering.h

private:
   SDValue LowerVAARG(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerFRAMEADDR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerRETURNADDR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerINSERT_VECTOR_ELT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerEXTRACT_VECTOR_ELT(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerSCALAR_TO_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerBUILD_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVECTOR_SHUFFLE(SDValue Op, SelectionDAG &DAG) const;
+  SDValue LowerSPLAT_VECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerEXTRACT_SUBVECTOR(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVectorSRA_SRL_SHL(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShiftLeftParts(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerShiftRightParts(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerVSETCC(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerCTPOP(SDValue Op, SelectionDAG &DAG) const;
   SDValue LowerF128Call(SDValue Op, SelectionDAG &DAG,
                         RTLIB::Libcall Call) const;

lib/Target/AArch64/AArch64ISelLowering.cpp

AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
   // the subtarget, but not at the command line.
   unsigned MaxJT = STI.getMaximumJumpTableSize();
   if (MaxJT && getMaximumJumpTableSize() == 0)
     setMaximumJumpTableSize(MaxJT);
 
   setHasExtractBitsInsn(true);
 
   setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::Other, Custom);
+  setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i8, Custom);
+  setOperationAction(ISD::INTRINSIC_WO_CHAIN, MVT::i16, Custom);
 
   if (Subtarget->hasNEON()) {
     // FIXME: v1f64 shouldn't be legal if we can avoid it, because it leads to
     // silliness like this:
     setOperationAction(ISD::FABS, MVT::v1f64, Expand);
     setOperationAction(ISD::FADD, MVT::v1f64, Expand);
     setOperationAction(ISD::FCEIL, MVT::v1f64, Expand);
     setOperationAction(ISD::FCOPYSIGN, MVT::v1f64, Expand);
case Intrinsic::experimental_vector_stepvector: {
     // will use the 32b w regs instead.
     if (ScalarVT != MVT::i64)
       ScalarVT = MVT::i32;
     auto SeriesVec = DAG.getNode(AArch64ISD::SERIES_VECTOR, dl, VT,
                                  DAG.getConstant(0, dl, ScalarVT),
                                  DAG.getConstant(1, dl, ScalarVT));
     return SeriesVec;
   }
+  case Intrinsic::experimental_vector_splatvector:
+    return LowerSPLAT_VECTOR(Op, DAG);
   }
 }
 
 SDValue AArch64TargetLowering::LowerOperation(SDValue Op,
                                               SelectionDAG &DAG) const {
   LLVM_DEBUG(dbgs() << "Custom lowering: ");
   LLVM_DEBUG(Op.dump());
 
if (NumElts == 4) {
 
     if (Cost <= 4)
       return GeneratePerfectShuffle(PFEntry, V1, V2, DAG, dl);
   }
 
   return GenerateTBL(Op, ShuffleMask, DAG);
 }
 
+SDValue AArch64TargetLowering::LowerSPLAT_VECTOR(SDValue Op,
+                                                 SelectionDAG &DAG) const {
+  SDLoc dl(Op);
+  EVT VT = Op.getValueType();
+  EVT ElemVT = VT.getScalarType();
+
+  SDValue SplatVal = Op.getOperand(1);
+
+  switch (ElemVT.getSimpleVT().SimpleTy) {
+  case MVT::i8:
+  case MVT::i16:
+    SplatVal = DAG.getAnyExtOrTrunc(SplatVal, dl, MVT::i32);
+    break;
+  case MVT::i64:
+    SplatVal = DAG.getAnyExtOrTrunc(SplatVal, dl, MVT::i64);
+    break;
+  case MVT::i32:
+    // Fine as is
+    break;
+  default:
+    // Skip the other types for now; i1 will be lowered to
+    // ptrue/pfalse, and the fp types can be lowered to unpacked
+    // vectors so should just pass through.
+    llvm_unreachable("Unsupported SPLAT_VECTOR input operand type");
+    break;
+  }
+
+  return DAG.getNode(AArch64ISD::DUP, dl, VT, SplatVal);
+}
+
 static bool resolveBuildVector(BuildVectorSDNode *BVN, APInt &CnstBits,
                                APInt &UndefBits) {
   EVT VT = BVN->getValueType(0);
   APInt SplatBits, SplatUndef;
   unsigned SplatBitSize;
   bool HasAnyUndefs;
   if (BVN->isConstantSplat(SplatBits, SplatUndef, SplatBitSize, HasAnyUndefs)) {
     unsigned NumSplats = VT.getSizeInBits() / SplatBitSize;
void AArch64TargetLowering::ReplaceNodeResults(
   case ISD::FP_TO_UINT:
   case ISD::FP_TO_SINT:
     assert(N->getValueType(0) == MVT::i128 && "unexpected illegal conversion");
     // Let normal code take care of it by not adding anything to Results.
     return;
   case ISD::ATOMIC_CMP_SWAP:
     ReplaceCMP_SWAP_128Results(N, Results, DAG, Subtarget);
     return;
+  case ISD::INTRINSIC_WO_CHAIN:
+    // This is needed since the operation action for this type of intrinsic
+    // was set to 'custom' for i8 and i16, in order to legalize stepvector
+    // and splatvector intrinsics. We don't yet need to legalize the
+    // results, but this is where we can do it until we can promote to
+    // generic code.
+    return;
   }
 }
 
 bool AArch64TargetLowering::useLoadStackGuardNode() const {
   if (Subtarget->isTargetAndroid() || Subtarget->isTargetFuchsia())
     return TargetLowering::useLoadStackGuardNode();
   return true;
 }

lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVE] in {
   def : InstAlias<"mov $Zd, $Zn",
                   (ORR_ZZZ ZPR64:$Zd, ZPR64:$Zn, ZPR64:$Zn), 1>;
 
 
   // Simple codegen Patterns
   // Unoptimized vscale
   def : Pat<(AArch64vscale (simm6_32b:$imm)), (RDVLI_XI $imm)>;
 
+  // Duplicate GPR in all vector elements
+  def : Pat<(nxv16i8 (AArch64dup GPR32:$a)), (DUP_ZR_B $a)>;
+  def : Pat<(nxv8i16 (AArch64dup GPR32:$a)), (DUP_ZR_H $a)>;
+  def : Pat<(nxv4i32 (AArch64dup GPR32:$a)), (DUP_ZR_S $a)>;
+  def : Pat<(nxv2i64 (AArch64dup GPR64:$a)), (DUP_ZR_D $a)>;
 
   // Unoptimized seriesvector patterns
   def : Pat<(nxv16i8 (AArch64seriesvec GPR32:$start, GPR32:$step)),
             (INDEX_RR_B $start, $step)>;
   def : Pat<(nxv8i16 (AArch64seriesvec GPR32:$start, GPR32:$step)),
             (INDEX_RR_H $start, $step)>;
   def : Pat<(nxv4i32 (AArch64seriesvec GPR32:$start, GPR32:$step)),
             (INDEX_RR_S $start, $step)>;
   def : Pat<(nxv2i64 (AArch64seriesvec GPR64:$start, GPR64:$step)),
             (INDEX_RR_D $start, $step)>;
 
 }

test/CodeGen/AArch64/SVE/dup.ll

+; RUN: llc -verify-machineinstrs -mattr=+sve < %s | FileCheck %s
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64--linux-gnueabi"
+
+define <scalable 16 x i8> @dup_b(i8 %val) {
+; CHECK-LABEL: dup_b:
+; CHECK: mov z0.b, w0
+; CHECK-NEXT: ret
+  %insert = insertelement <scalable 16 x i8> undef, i8 %val, i32 0
+  %splat = shufflevector <scalable 16 x i8> %insert, <scalable 16 x i8> undef, <scalable 16 x i32> zeroinitializer
+  ret <scalable 16 x i8> %splat
+}
+
+define <scalable 8 x i16> @dup_h(i16 %val) {
+; CHECK-LABEL: dup_h:
+; CHECK: mov z0.h, w0
+; CHECK-NEXT: ret
+  %insert = insertelement <scalable 8 x i16> undef, i16 %val, i32 0
+  %splat = shufflevector <scalable 8 x i16> %insert, <scalable 8 x i16> undef, <scalable 8 x i32> zeroinitializer
+  ret <scalable 8 x i16> %splat
+}
+
+define <scalable 4 x i32> @dup_s(i32 %val) {
+; CHECK-LABEL: dup_s:
+; CHECK: mov z0.s, w0
+; CHECK-NEXT: ret
+  %insert = insertelement <scalable 4 x i32> undef, i32 %val, i32 0
+  %splat = shufflevector <scalable 4 x i32> %insert, <scalable 4 x i32> undef, <scalable 4 x i32> zeroinitializer
+  ret <scalable 4 x i32> %splat
+}
+
+define <scalable 2 x i64> @dup_d(i64 %val) {
+; CHECK-LABEL: dup_d:
+; CHECK: mov z0.d, x0
+; CHECK-NEXT: ret
+  %insert = insertelement <scalable 2 x i64> undef, i64 %val, i32 0
+  %splat = shufflevector <scalable 2 x i64> %insert, <scalable 2 x i64> undef, <scalable 2 x i32> zeroinitializer
+  ret <scalable 2 x i64> %splat
+}