[CostModel][X86] Add support for broadcast shuffle costs

[RKSimon]

Summary

Currently only for broadcasts with input and output of the same width.

[andreadb]

Hi Simon,

I noticed that getShuffleCost is becoming quite big.
Do you think it makes sense to split the logic in getShuffleCost in three parts (a function for each supported ShuffleKind)?. In case, getShuffleCost could be refactored in a separate commit.

lib/Target/X86/X86TargetTransformInfo.cpp

613

Shouldn't this be LT.first * Entry->Cost ?
I think that you are not accounting for the type legalization cost.

625

Same.

639

Same (also, see lines 654, 664, 677 and 686).

678–685

Can we not just simplify this code into something like this?

if (ST->hasSSE1() && LT.second == MVT::v4f32)
  return LT.first;

You are basically performing a lookup on a table with just a single entry.

[RKSimon]

In D27811#623731, @andreadb wrote:

I noticed that getShuffleCost is becoming quite big.

You're not kidding ;-)

Do you think it makes sense to split the logic in getShuffleCost in three parts (a function for each supported ShuffleKind)?. In case, getShuffleCost could be refactored in a separate commit.

I have considered hijacking the 'ISD::VECTOR_SHUFFLE' int entry in both CostTbleEmtry and CostTableLookup to do lookup based on TTI::ShuffleKind instead - what do you think?

lib/Target/X86/X86TargetTransformInfo.cpp

613

That's what I meant in the disclaimer at the top - as we're broadcasting we only reference the first input register and all the outputs are the same - so the costs aren't multiplied by the LT.first scale factor (num vectors). It doesn't account for any register moves that occur but that's true for most throughput costs.

[mkuper]

lib/Analysis/CostModel.cpp

93

We already have this helper in CGP (that version also checks if you're splatting *any* element, not just element 0.)
Maybe make it common? Not entirely sure what the appropriate place for it yes, though. Would it make sense for CGP to use CostModel?

[andreadb]

In D27811#623755, @RKSimon wrote:

In D27811#623731, @andreadb wrote:

I noticed that getShuffleCost is becoming quite big.

You're not kidding ;-)

Do you think it makes sense to split the logic in getShuffleCost in three parts (a function for each supported ShuffleKind)?. In case, getShuffleCost could be refactored in a separate commit.

I have considered hijacking the 'ISD::VECTOR_SHUFFLE' int entry in both CostTbleEmtry and CostTableLookup to do lookup based on TTI::ShuffleKind instead - what do you think?

I think it is a good idea :-). After all, the opcode can only be ISD::VECTOR_SHUFFLE in this method. So, that bit of information doesn't really need to be stored in any entries.

lib/Target/X86/X86TargetTransformInfo.cpp

613

Ah I see. That makes sense.

[RKSimon]

lib/Analysis/CostModel.cpp

93

Adding isSplatMask/isSplat/getSplatIndex support to ShuffleVectorInst (worth adding them to all to match ShuffleVectorSDNode ?) would be trivial, then both CodeGenPrepare and CostModel could use it easily.

It would probably be worth upgrading SK_Broadcast at the same time to support broadcasting any Index value (not just 0) - given that almost nothing uses it so far that shouldn't be a problem.

What other cost model cases did you have in mind? CodeGenPrepare::optimizeShuffleVectorInst seems quite limited.

[mkuper]

lib/Analysis/CostModel.cpp

93

It would probably be worth upgrading SK_Broadcast at the same time to support broadcasting any Index value (not just 0) - given that almost nothing uses it so far that shouldn't be a problem.

Thinking about it a bit more, I'm not entirely sure about this. Ignoring what we do in the DAG for a moment, in IR, I'd expect the canonical insert + splat pattern to use index 0.

What other cost model cases did you have in mind? CodeGenPrepare::optimizeShuffleVectorInst seems quite limited.

I didn't, really, just trying to avoid code duplication.

[RKSimon]

Added ShuffleVectorInst::isSplat as suggested.

Regarding the refactor to use TTI::ShuffleKind instead of ISD::SHUFFLE_VECTOR in the LUTs - is everyone happy with me to commit this to trunk and I'll then update this patch with the new scheme?

[delena]

In D27811#623924, @andreadb wrote:

In D27811#623755, @RKSimon wrote:

In D27811#623731, @andreadb wrote:

I noticed that getShuffleCost is becoming quite big.

You're not kidding ;-)

Do you think it makes sense to split the logic in getShuffleCost in three parts (a function for each supported ShuffleKind)?. In case, getShuffleCost could be refactored in a separate commit.

I have considered hijacking the 'ISD::VECTOR_SHUFFLE' int entry in both CostTbleEmtry and CostTableLookup to do lookup based on TTI::ShuffleKind instead - what do you think?

I think it is a good idea :-). After all, the opcode can only be ISD::VECTOR_SHUFFLE in this method. So, that bit of information doesn't really need to be stored in any entries.

May I ask you to postpone refactoring of getShuffleCost() to the next commit? Andrea, I have another patch https://reviews.llvm.org/D28118 that also changes ShuffleCost for X86 targets.

[RKSimon]

In D27811#632164, @delena wrote:

May I ask you to postpone refactoring of getShuffleCost() to the next commit? Andrea, I have another patch https://reviews.llvm.org/D28118 that also changes ShuffleCost for X86 targets.

No problem, I'll do the cleanup/refactor after your interleaving patch

[delena]

lib/Analysis/CostModel.cpp

519

I suggest to simplify the code - just one function:

static bool isZeroEltBroadcastVectorMask(SmallVectorImpl<int> &Mask) {
  for (unsigned i = 0; i < Mask.size(); ++i)
    if (Mask[i] > 0)
      return false;
  return true;
}

[RKSimon]

Updated having merged the Broadcast/Alternate/Reverse shuffle costs into a single set of LUTs.

[andreadb]

lib/Analysis/CostModel.cpp

520–522

Is this code still needed?
r290810 introduced a check for `isZeroEltBroadcastVectorMask' at line 530.

[RKSimon]

lib/Analysis/CostModel.cpp

520–522

Thanks - I missed that for some reason.

What should we do? Keep with the separate Broadcast matching helpers or merge and use ShuffleVectorInst::isSplat ?

[andreadb]

lib/Analysis/CostModel.cpp

520–522

Good question. I don't have a strong opinion since both approaches sound reasonable to me.

In general, I like the idea of having separate helpers to check shuffle masks, and I am not particularly worried about the small code duplication in isZeroEltBroadcastVectorMask (since that helper is very simple).

[delena]

lib/Target/X86/X86TargetTransformInfo.cpp

683

theoretically, v16i16 should not be more expensive than v32i8, vpshufb should work for v8i16 as it works for v16i8. (I don't know how it is implemented).

[RKSimon]

lib/Target/X86/X86TargetTransformInfo.cpp

683

It does currently have the vpshuflw + vpshufd + vinsertf128 pattern - I'll look improving this in shuffle combining.

[RKSimon]

Use existing broadcast shuffle matching

[andreadb]

LGTM. Thanks Simon!

include/llvm/IR/Instructions.h

public:
   }
 
   SmallVector<int, 16> getShuffleMask() const {
     SmallVector<int, 16> Mask;
     getShuffleMask(Mask);
     return Mask;
   }
 
+  /// Determine if the shuffle mask is a splat, possibly with undefined mask
+  /// indices as well. Returns true if the same shuffle index is found in all
+  /// defined elements and optionally returns the splat index. Returns false
+  /// if the mask is not a splat or all mask indices are undefined.
+  bool isSplat(int *SplatIndex = nullptr) const;
+
   // Methods for support type inquiry through isa, cast, and dyn_cast:
   static inline bool classof(const Instruction *I) {
     return I->getOpcode() == Instruction::ShuffleVector;
   }
   static inline bool classof(const Value *V) {
     return isa<Instruction>(V) && classof(cast<Instruction>(V));
   }
 };

lib/Analysis/CostModel.cpp

 CostModelAnalysis::runOnFunction(Function &F) {
  this->F = &F;
  auto *TTIWP = getAnalysisIfAvailable<TargetTransformInfoWrapperPass>();
  TTI = TTIWP ? &TTIWP->getTTI(F) : nullptr;
 
  return false;
 }
 
 static bool isReverseVectorMask(ArrayRef<int> Mask) {
   for (unsigned i = 0, MaskSize = Mask.size(); i < MaskSize; ++i)
     if (Mask[i] >= 0 && Mask[i] != (int)(MaskSize - 1 - i))
       return false;
   return true;
 }
 
 static bool isSingleSourceVectorMask(ArrayRef<int> Mask) {
   bool Vec0 = false;
unsigned CostModelAnalysis::getInstructionCost(const Instruction *I) const {
   }
   case Instruction::ShuffleVector: {
     const ShuffleVectorInst *Shuffle = cast<ShuffleVectorInst>(I);
     Type *VecTypOp0 = Shuffle->getOperand(0)->getType();
     unsigned NumVecElems = VecTypOp0->getVectorNumElements();
     SmallVector<int, 16> Mask = Shuffle->getShuffleMask();
 
     if (NumVecElems == Mask.size()) {
+      int BroadcastIndex = -1;
+      if (Shuffle->isSplat(&BroadcastIndex) && BroadcastIndex == 0)
+        return TTI->getShuffleCost(TargetTransformInfo::SK_Broadcast,
+                                   VecTypOp0, 0, nullptr);
       if (isReverseVectorMask(Mask))
         return TTI->getShuffleCost(TargetTransformInfo::SK_Reverse, VecTypOp0,
                                    0, nullptr);
       if (isAlternateVectorMask(Mask))
         return TTI->getShuffleCost(TargetTransformInfo::SK_Alternate,
                                    VecTypOp0, 0, nullptr);
 
       if (isZeroEltBroadcastVectorMask(Mask))

lib/CodeGen/CodeGenPrepare.cpp

for (auto It = ASI.rbegin(); It != ASI.rend(); ++It) {
     ++NumSelectsExpanded;
   }
 
   // Instruct OptimizeBlock to skip to the next block.
   CurInstIterator = StartBlock->end();
   return true;
 }
 
-static bool isBroadcastShuffle(ShuffleVectorInst *SVI) {
-  SmallVector<int, 16> Mask(SVI->getShuffleMask());
-  int SplatElem = -1;
-  for (unsigned i = 0; i < Mask.size(); ++i) {
-    if (SplatElem != -1 && Mask[i] != -1 && Mask[i] != SplatElem)
-      return false;
-    SplatElem = Mask[i];
-  }
-
-  return true;
-}
-
 /// Some targets have expensive vector shifts if the lanes aren't all the same
 /// (e.g. x86 only introduced "vpsllvd" and friends with AVX2). In these cases
 /// it's often worth sinking a shufflevector splat down to its use so that
 /// codegen can spot all lanes are identical.
 bool CodeGenPrepare::optimizeShuffleVectorInst(ShuffleVectorInst *SVI) {
   BasicBlock *DefBB = SVI->getParent();
 
   // Only do this xform if variable vector shifts are particularly expensive.
   if (!TLI || !TLI->isVectorShiftByScalarCheap(SVI->getType()))
     return false;
 
   // We only expect better codegen by sinking a shuffle if we can recognise a
   // constant splat.
-  if (!isBroadcastShuffle(SVI))
+  if (!SVI->isSplat())
     return false;
 
   // InsertedShuffles - Only insert a shuffle in each block once.
   DenseMap<BasicBlock*, Instruction*> InsertedShuffles;
 
   bool MadeChange = false;
   for (User *U : SVI->users()) {
     Instruction *UI = cast<Instruction>(U);

lib/IR/Instructions.cpp

void ShuffleVectorInst::getShuffleMask(Constant *Mask,
   }    
   for (unsigned i = 0; i != NumElts; ++i) {
     Constant *C = Mask->getAggregateElement(i);
     Result.push_back(isa<UndefValue>(C) ? -1 :
                      cast<ConstantInt>(C)->getZExtValue());
   }
 }
 
+bool ShuffleVectorInst::isSplat(int *SplatIndex /* = nullptr */) const {
+  SmallVector<int, 16> Mask;
+  getShuffleMask(Mask);
+
+  int SplatElem = -1;
+  for (int M : Mask) {
+    if (M < 0)
+      continue;
+    if (0 <= SplatElem && SplatElem != M)
+      return false;
+    SplatElem = M;
+  }
+
+  if (0 <= SplatElem && SplatIndex)
+    *SplatIndex = SplatElem;
+  return (0 <= SplatElem);
+}
 
 //===----------------------------------------------------------------------===//
 //                             InsertValueInst Class
 //===----------------------------------------------------------------------===//
 
 void InsertValueInst::init(Value *Agg, Value *Val, ArrayRef<unsigned> Idxs, 
                            const Twine &Name) {
   assert(getNumOperands() == 2 && "NumOperands not initialized?");

lib/Target/X86/X86TargetTransformInfo.cpp

if (const auto *Entry = CostTableLookup(SSE1FloatCostTable, ISD,
                                             LT.second))
       return LT.first * Entry->Cost;
   // Fallback to the default implementation.
   return BaseT::getArithmeticInstrCost(Opcode, Ty, Op1Info, Op2Info);
 }
 
 int X86TTIImpl::getShuffleCost(TTI::ShuffleKind Kind, Type *Tp, int Index,
                                Type *SubTp) {
-  if (Kind == TTI::SK_Reverse || Kind == TTI::SK_Alternate) {
+  if (Kind == TTI::SK_Reverse || Kind == TTI::SK_Alternate ||
+      Kind == TTI::SK_Broadcast) {
     // 64-bit packed float vectors (v2f32) are widened to type v4f32.
     // 64-bit packed integer vectors (v2i32) are promoted to type v2i64.
     std::pair<int, MVT> LT = TLI->getTypeLegalizationCost(DL, Tp);
 
+    // For Broadcasts we are splatting the first element from the first input
+    // register, so only need to reference that input and all the output
+    // registers are the same.
+    if (Kind == TTI::SK_Broadcast)
+      LT.first = 1;
+
     static const CostTblEntry AVX512VBMIShuffleTbl[] = {
       { TTI::SK_Reverse, MVT::v64i8,  1 }, // vpermb
       { TTI::SK_Reverse, MVT::v32i8,  1 }  // vpermb
     };
 
     if (ST->hasVBMI())
       if (const auto *Entry =
               CostTableLookup(AVX512VBMIShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry AVX512BWShuffleTbl[] = {
-      { TTI::SK_Reverse, MVT::v32i16, 1 }, // vpermw
-      { TTI::SK_Reverse, MVT::v16i16, 1 }, // vpermw
-      { TTI::SK_Reverse, MVT::v64i8,  6 }  // vextracti64x4 + 2*vperm2i128
+      { TTI::SK_Broadcast, MVT::v32i16, 1 }, // vpbroadcastw
+      { TTI::SK_Broadcast, MVT::v64i8,  1 }, // vpbroadcastb
+
+      { TTI::SK_Reverse,   MVT::v32i16, 1 }, // vpermw
+      { TTI::SK_Reverse,   MVT::v16i16, 1 }, // vpermw
+      { TTI::SK_Reverse,   MVT::v64i8,  6 }  // vextracti64x4 + 2*vperm2i128
                                              // + 2*pshufb + vinserti64x4
     };
 
     if (ST->hasBWI())
       if (const auto *Entry =
               CostTableLookup(AVX512BWShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry AVX512ShuffleTbl[] = {
-      { TTI::SK_Reverse, MVT::v8f64,  1 }, // vpermpd
-      { TTI::SK_Reverse, MVT::v16f32, 1 }, // vpermps
-      { TTI::SK_Reverse, MVT::v8i64,  1 }, // vpermq
-      { TTI::SK_Reverse, MVT::v16i32, 1 }, // vpermd
+      { TTI::SK_Broadcast, MVT::v8f64,  1 }, // vbroadcastpd
+      { TTI::SK_Broadcast, MVT::v16f32, 1 }, // vbroadcastps
+      { TTI::SK_Broadcast, MVT::v8i64,  1 }, // vpbroadcastq
+      { TTI::SK_Broadcast, MVT::v16i32, 1 }, // vpbroadcastd
+
+      { TTI::SK_Reverse,   MVT::v8f64,  1 }, // vpermpd
+      { TTI::SK_Reverse,   MVT::v16f32, 1 }, // vpermps
+      { TTI::SK_Reverse,   MVT::v8i64,  1 }, // vpermq
+      { TTI::SK_Reverse,   MVT::v16i32, 1 }  // vpermd
     };
 
     if (ST->hasAVX512())
       if (const auto *Entry =
               CostTableLookup(AVX512ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry AVX2ShuffleTbl[] = {
+      { TTI::SK_Broadcast, MVT::v4f64,  1 }, // vbroadcastpd
+      { TTI::SK_Broadcast, MVT::v8f32,  1 }, // vbroadcastps
+      { TTI::SK_Broadcast, MVT::v4i64,  1 }, // vpbroadcastq
+      { TTI::SK_Broadcast, MVT::v8i32,  1 }, // vpbroadcastd
+      { TTI::SK_Broadcast, MVT::v16i16, 1 }, // vpbroadcastw
+      { TTI::SK_Broadcast, MVT::v32i8,  1 }, // vpbroadcastb
+
       { TTI::SK_Reverse,   MVT::v4f64,  1 }, // vpermpd
       { TTI::SK_Reverse,   MVT::v8f32,  1 }, // vpermps
       { TTI::SK_Reverse,   MVT::v4i64,  1 }, // vpermq
       { TTI::SK_Reverse,   MVT::v8i32,  1 }, // vpermd
       { TTI::SK_Reverse,   MVT::v16i16, 2 }, // vperm2i128 + pshufb
       { TTI::SK_Reverse,   MVT::v32i8,  2 }, // vperm2i128 + pshufb
 
       { TTI::SK_Alternate, MVT::v16i16, 1 }, // vpblendw
       { TTI::SK_Alternate, MVT::v32i8,  1 }  // vpblendvb
     };
 
     if (ST->hasAVX2())
       if (const auto *Entry = CostTableLookup(AVX2ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry AVX1ShuffleTbl[] = {
+      { TTI::SK_Broadcast, MVT::v4f64,  2 }, // vperm2f128 + vpermilpd
+      { TTI::SK_Broadcast, MVT::v8f32,  2 }, // vperm2f128 + vpermilps
+      { TTI::SK_Broadcast, MVT::v4i64,  2 }, // vperm2f128 + vpermilpd
+      { TTI::SK_Broadcast, MVT::v8i32,  2 }, // vperm2f128 + vpermilps
+      { TTI::SK_Broadcast, MVT::v16i16, 3 }, // vpshuflw + vpshufd + vinsertf128
+      { TTI::SK_Broadcast, MVT::v32i8,  2 }, // vpshufb + vinsertf128
+
       { TTI::SK_Reverse,   MVT::v4f64,  2 }, // vperm2f128 + vpermilpd
       { TTI::SK_Reverse,   MVT::v8f32,  2 }, // vperm2f128 + vpermilps
       { TTI::SK_Reverse,   MVT::v4i64,  2 }, // vperm2f128 + vpermilpd
       { TTI::SK_Reverse,   MVT::v8i32,  2 }, // vperm2f128 + vpermilps
       { TTI::SK_Reverse,   MVT::v16i16, 4 }, // vextractf128 + 2*pshufb
                                              // + vinsertf128
       { TTI::SK_Reverse,   MVT::v32i8,  4 }, // vextractf128 + 2*pshufb
                                              // + vinsertf128
static const CostTblEntry SSE41ShuffleTbl[] = {
       { TTI::SK_Alternate, MVT::v16i8,  1 }  // pblendvb
     };
 
     if (ST->hasSSE41())
       if (const auto *Entry = CostTableLookup(SSE41ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry SSSE3ShuffleTbl[] = {
+      { TTI::SK_Broadcast, MVT::v8i16,  1 }, // pshufb
+      { TTI::SK_Broadcast, MVT::v16i8,  1 }, // pshufb
+
       { TTI::SK_Reverse,   MVT::v8i16,  1 }, // pshufb
       { TTI::SK_Reverse,   MVT::v16i8,  1 }, // pshufb
 
       { TTI::SK_Alternate, MVT::v8i16,  3 }, // pshufb + pshufb + por
       { TTI::SK_Alternate, MVT::v16i8,  3 }  // pshufb + pshufb + por
     };
 
     if (ST->hasSSSE3())
       if (const auto *Entry = CostTableLookup(SSSE3ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry SSE2ShuffleTbl[] = {
+      { TTI::SK_Broadcast, MVT::v2f64,  1 }, // shufpd
+      { TTI::SK_Broadcast, MVT::v2i64,  1 }, // pshufd
+      { TTI::SK_Broadcast, MVT::v4i32,  1 }, // pshufd
+      { TTI::SK_Broadcast, MVT::v8i16,  2 }, // pshuflw  + pshufd
+      { TTI::SK_Broadcast, MVT::v16i8,  3 }, // unpck + pshuflw + pshufd
+
       { TTI::SK_Reverse,   MVT::v2f64,  1 }, // shufpd
       { TTI::SK_Reverse,   MVT::v2i64,  1 }, // pshufd
       { TTI::SK_Reverse,   MVT::v4i32,  1 }, // pshufd
       { TTI::SK_Reverse,   MVT::v8i16,  3 }, // pshuflw + pshufhw  + pshufd
       { TTI::SK_Reverse,   MVT::v16i8,  9 }, // 2*pshuflw + 2*pshufhw
                                              // + 2*pshufd + 2*unpck + packus
 
       { TTI::SK_Alternate, MVT::v2i64,  1 }, // movsd
       { TTI::SK_Alternate, MVT::v2f64,  1 }, // movsd
       { TTI::SK_Alternate, MVT::v4i32,  2 }, // 2*shufps
       { TTI::SK_Alternate, MVT::v8i16,  3 }, // pand + pandn + por
       { TTI::SK_Alternate, MVT::v16i8,  3 }  // pand + pandn + por
     };
 
     if (ST->hasSSE2())
       if (const auto *Entry = CostTableLookup(SSE2ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 
     static const CostTblEntry SSE1ShuffleTbl[] = {
+      { TTI::SK_Broadcast, MVT::v4f32,  1 }, // shufps
       { TTI::SK_Reverse,   MVT::v4f32,  1 }, // shufps
       { TTI::SK_Alternate, MVT::v4f32,  2 }  // 2*shufps
     };
 
     if (ST->hasSSE1())
       if (const auto *Entry = CostTableLookup(SSE1ShuffleTbl, Kind, LT.second))
         return LT.first * Entry->Cost;
 

test/Analysis/CostModel/X86/shuffle-broadcast.ll

 ; CHECK-LABEL: 'test_vXf64'
 define void @test_vXf64(<2 x double> %src128, <4 x double> %src256, <8 x double> %src512) {
   ; SSE: cost of 1 {{.*}} %V128 = shufflevector
   ; AVX: cost of 1 {{.*}} %V128 = shufflevector
   ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
   %V128 = shufflevector <2 x double> %src128, <2 x double> undef, <2 x i32> zeroinitializer
 
   ; SSE: cost of 1 {{.*}} %V256 = shufflevector
-  ; AVX: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
   ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
   %V256 = shufflevector <4 x double> %src256, <4 x double> undef, <4 x i32> zeroinitializer
 
   ; SSE: cost of 1 {{.*}} %V512 = shufflevector
-  ; AVX: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
   ; AVX512: cost of 1 {{.*}} %V512 = shufflevector
   %V512 = shufflevector <8 x double> %src512, <8 x double> undef, <8 x i32> zeroinitializer
 
   ret void
 }
+
+; CHECK-LABEL: 'test_vXi64'
+define void @test_vXi64(<2 x i64> %src128, <4 x i64> %src256, <8 x i64> %src512) {
+  ; SSE: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
+  %V128 = shufflevector <2 x i64> %src128, <2 x i64> undef, <2 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
+  %V256 = shufflevector <4 x i64> %src256, <4 x i64> undef, <4 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V512 = shufflevector
+  %V512 = shufflevector <8 x i64> %src512, <8 x i64> undef, <8 x i32> zeroinitializer
+
+  ret void
+}
+
+; CHECK-LABEL: 'test_vXf32'
+define void @test_vXf32(<2 x float> %src64, <4 x float> %src128, <8 x float> %src256, <16 x float> %src512) {
+  ; SSE: cost of 1 {{.*}} %V64 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V64 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V64 = shufflevector
+  %V64 = shufflevector <2 x float> %src64, <2 x float> undef, <2 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
+  %V128 = shufflevector <4 x float> %src128, <4 x float> undef, <4 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
+  %V256 = shufflevector <8 x float> %src256, <8 x float> undef, <8 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V512 = shufflevector
+  %V512 = shufflevector <16 x float> %src512, <16 x float> undef, <16 x i32> zeroinitializer
+
+  ret void
+}
+
+; CHECK-LABEL: 'test_vXi32'
+define void @test_vXi32(<2 x i32> %src64, <4 x i32> %src128, <8 x i32> %src256, <16 x i32> %src512) {
+  ; SSE: cost of 1 {{.*}} %V64 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V64 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V64 = shufflevector
+  %V64 = shufflevector <2 x i32> %src64, <2 x i32> undef, <2 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
+  %V128 = shufflevector <4 x i32> %src128, <4 x i32> undef, <4 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
+  %V256 = shufflevector <8 x i32> %src256, <8 x i32> undef, <8 x i32> zeroinitializer
+
+  ; SSE: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V512 = shufflevector
+  %V512 = shufflevector <16 x i32> %src512, <16 x i32> undef, <16 x i32> zeroinitializer
+
+  ret void
+}
+
+; CHECK-LABEL: 'test_vXi16'
+define void @test_vXi16(<8 x i16> %src128, <16 x i16> %src256, <32 x i16> %src512) {
+  ; SSE2: cost of 2 {{.*}} %V128 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V128 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
+  %V128 = shufflevector <8 x i16> %src128, <8 x i16> undef, <8 x i32> zeroinitializer
+
+  ; SSE2: cost of 2 {{.*}} %V256 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V256 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 3 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
+  %V256 = shufflevector <16 x i16> %src256, <16 x i16> undef, <16 x i32> zeroinitializer
+
+  ; SSE2: cost of 2 {{.*}} %V512 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V512 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 3 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512F: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512BW: cost of 1 {{.*}} %V512 = shufflevector
+  %V512 = shufflevector <32 x i16> %src512, <32 x i16> undef, <32 x i32> zeroinitializer
+
+  ret void
+}
+
+; CHECK-LABEL: 'test_vXi8'
+define void @test_vXi8(<16 x i8> %src128, <32 x i8> %src256, <64 x i8> %src512) {
+  ; SSE2: cost of 3 {{.*}} %V128 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V128 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX: cost of 1 {{.*}} %V128 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V128 = shufflevector
+  %V128 = shufflevector <16 x i8> %src128, <16 x i8> undef, <16 x i32> zeroinitializer
+
+  ; SSE2: cost of 3 {{.*}} %V256 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V256 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V256 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V256 = shufflevector
+  ; AVX512: cost of 1 {{.*}} %V256 = shufflevector
+  %V256 = shufflevector <32 x i8> %src256, <32 x i8> undef, <32 x i32> zeroinitializer
+
+  ; SSE2: cost of 3 {{.*}} %V512 = shufflevector
+  ; SSSE3: cost of 1 {{.*}} %V512 = shufflevector
+  ; SSE42: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX1: cost of 2 {{.*}} %V512 = shufflevector
+  ; AVX2: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512F: cost of 1 {{.*}} %V512 = shufflevector
+  ; AVX512BW: cost of 1 {{.*}} %V512 = shufflevector
+  %V512 = shufflevector <64 x i8> %src512, <64 x i8> undef, <64 x i32> zeroinitializer
+
+  ret void
+}