[AArch64] Use NEON's tbl1 for 16xi8 and 8xi8 build vector with mask.

[lawben]

Summary

When using Clang's __builtin_shufflevector with a 16xi8 or 8xi8 source and runtime mask on an AArch64 target, LLVM currently generates 16 or 8 extract+and+insert operations. This patch replaces these inserts with (a vector AND +) NEON's tbl1 intruction.

Issue: https://github.com/llvm/llvm-project/issues/60515

[fhahn]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10214

Why limit this it v16i8?

10215

It looks like we are missing test coverage for most (all?) exits. It would be great if you could extend the test coverage.

[lawben]

Add tests for exit cases and extend to v8i8.

[lawben]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10214

Good point. I've added v8i8 too. I only had the v16i8 use case in mind when writing this. This also works for other vector sizes, but that requires some additional logic to multiply and and byte offsets, as TBL only supports v16i8 and v8i8 directly. I've seen some of that around the code and I think x86 instruction selection also does this. If you think this is valuable and the right place to do it, I'd be happy to implement that in a follow-up patch.

10215

I've added a few tests to cover the exits. Please let me know if you think something is missing. I also noticed that some of my assumptions were too pessimistic in the process, so I removed them.

[dmgreen]

Hi - I like the idea of generating tbl instructions from non-constant masks. Does the patch need an update to be based against main?

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10201–10294

Should something check that the inner extract element (MaskSource) has an index equal to i?

10201–10294

This doesn't need the cast any more.

10203

This can test against the EVT directly: VT != MVT::v16i8 && .. .
Other types could be handled too, by breaking the larger type up into i8 lanes. i.e a i32 would add [idx*4,idx*4+1,idx*4+2,idx*4+3] for example, if you get my meaning.
I'm not sure if that is worth adding to this patch, or handling in another but it would be good to keep in mind.

10209

You can concat with undef if all the needed elements are in the bottom half.

llvm/test/CodeGen/AArch64/neon-shuffle-vector-tbl.ll

12

It would be good to have a test case without this And mask, just using %mask directly.

[lawben]

Add a path without AND mask.

We now have a path that does not need an AND on the mask source. We now either expect all elements to have an AND or none.
Also addresses some review comments.

[lawben]

@dmgreen Thanks for the feedback. I've addressed the comments you made. I've renamed the method, as it does not require an AND anymore.

re: update to base against main. I tried to merge the current main branch into another patch (using git) and completely broke my setup, i.e., arc is doing really weird things, trying to lint hundreds of files, etc. So I'm not sure what the correct way is to do this. Do you maybe have a pointer for me? This is my first patch, so I'm new to this setup.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

10201–10294

Yes, good catch. I've added a check and a test.

10203

I also had this in mind but was thinking about putting it in another patch. v16i8 and v8i8 directly map, so they are easier to generate. Other sizes have some more overhead, so it makes sense to split them up. I've seen the logic for this in X86 instruction selection, so I guess that could just be reused. I'll do that in a follow-up, once this is merged.

llvm/test/CodeGen/AArch64/neon-shuffle-vector-tbl.ll

12

Makes sense. There is actually no reason to limit this to and-masked masks. So I added a path that either expects all elements to have an AND mask or none. If there are no ANDs, we can omit it and directly emit a tbl. I was looking at this from Clang's __builtin_shufflevector() (as this is my use) and that always adds an AND.

[dmgreen]

In D146212#4209438, @lawben wrote:

@dmgreen Thanks for the feedback. I've addressed the comments you made. I've renamed the method, as it does not require an AND anymore.

re: update to base against main. I tried to merge the current main branch into another patch (using git) and completely broke my setup, i.e., arc is doing really weird things, trying to lint hundreds of files, etc. So I'm not sure what the correct way is to do this. Do you maybe have a pointer for me? This is my first patch, so I'm new to this setup.

I'm afraid I'm a bit of a luddite who just uploads patches manually into phabricator and doesn't use arc. I usually use git rebase -i to squash patches into a single commit, and upload them from there. Maybe the same thing will help with arc?

[lawben]

Rebase main

[lawben]

@dmgreen Okay, I rebased onto main and submited via arc. I hope this did the right thing, but it doesn't look broken. I guess my mistake in the other patch was to create a merge commit instead of rebasing. For future reference, I'll just always squash into in commit.

[DavidSpickett]

@dmgreen Okay, I rebased onto main and submited via arc. I hope this did the right thing, but it doesn't look broken. I guess my mistake in the other patch was to create a merge commit instead of rebasing. For future reference, I'll just always squash into in commit.

I use arc a lot and rebase things frequently. What you did may not even be incorrect, but arc has a habit of doing crazy things for small mistakes. It's usually better to find a safe method rather than trying to understand why it did what it did.

I usually have the patches that are in review in a branch. So if I want to rebase them onto main, I update main, checkout the branch. Then git rebase main to update them all. Then to update each review I interactive rebase back to each patch and run arc diff from there.

You can apply the same thing if you have a series of say 5 patches, and someone asks you to update patch 1 which means you have to change all the rest. Rebase back to patch 1, marking all the commits to edit. Do your changes, arc diff, then continue the rebase, fixing up each commit as you go and updating the reviews.

[lawben]

An honest attempt to do a rebase with arc :)

[dmgreen]

Thanks - that does look better now. As far as I can see this LGTM. Thanks for the patch.

[lawben]

@fhahn is there anything missing from your side? If not, could one of you or @dmgreen merge this? As this is my first patch, I obviously do not have push access :)

[dmgreen]

Hello. I can merge, do you have a "name <[e-mail]>" I can use for git attribution?

[lawben]

Thanks. Could you please use "Lawrence Benson <[e-mail]>".

[fhahn]

Thanks, looks good from me side as well.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

 }
 
 /// getExtFactor - Determine the adjustment factor for the position when
 /// generating an "extract from vector registers" instruction.
 static unsigned getExtFactor(SDValue &V) {
   EVT EltType = V.getValueType().getVectorElementType();
   return EltType.getSizeInBits() / 8;
 }
 
+// Check if a vector is built from one vector via extracted elements of
+// another together with an AND mask, ensuring that all elements fit
+// within range. This can be reconstructed using AND and NEON's TBL1.
+SDValue ReconstructShuffleWithRuntimeMask(SDValue Op, SelectionDAG &DAG) {
+  assert(Op.getOpcode() == ISD::BUILD_VECTOR && "Unknown opcode!");
+  SDLoc dl(Op);
+  EVT VT = Op.getValueType();
+  assert(!VT.isScalableVector() &&
+         "Scalable vectors cannot be used with ISD::BUILD_VECTOR");
+
+  // Can only recreate a shuffle with 16xi8 or 8xi8 elements, as they map
+  // directly to TBL1.
+  if (VT != MVT::v16i8 && VT != MVT::v8i8)
+    return SDValue();
+
+  unsigned NumElts = VT.getVectorNumElements();
+  assert((NumElts == 8 || NumElts == 16) &&
+         "Need to have exactly 8 or 16 elements in vector.");
+
+  SDValue SourceVec;
+  SDValue MaskSourceVec;
+  SmallVector<SDValue, 16> AndMaskConstants;
+
+  for (unsigned i = 0; i < NumElts; ++i) {
+    SDValue V = Op.getOperand(i);
+    if (V.getOpcode() != ISD::EXTRACT_VECTOR_ELT)
+      return SDValue();
+
+    SDValue OperandSourceVec = V.getOperand(0);
+    if (!SourceVec)
+      SourceVec = OperandSourceVec;
+    else if (SourceVec != OperandSourceVec)
+      return SDValue();
+
+    // This only looks at shuffles with elements that are
+    // a) truncated by a constant AND mask extracted from a mask vector, or
+    // b) extracted directly from a mask vector.
+    SDValue MaskSource = V.getOperand(1);
+    if (MaskSource.getOpcode() == ISD::AND) {
+      if (!isa<ConstantSDNode>(MaskSource.getOperand(1)))
+        return SDValue();
+
+      AndMaskConstants.push_back(MaskSource.getOperand(1));
+      MaskSource = MaskSource->getOperand(0);
+    } else if (!AndMaskConstants.empty()) {
+      // Either all or no operands should have an AND mask.
+      return SDValue();
+    }
+
+    // An ANY_EXTEND may be inserted between the AND and the source vector
+    // extraction. We don't care about that, so we can just skip it.
+    if (MaskSource.getOpcode() == ISD::ANY_EXTEND)
+      MaskSource = MaskSource.getOperand(0);
+
+    if (MaskSource.getOpcode() != ISD::EXTRACT_VECTOR_ELT)
+      return SDValue();
+
+    SDValue MaskIdx = MaskSource.getOperand(1);
+    if (!isa<ConstantSDNode>(MaskIdx) ||
+        !cast<ConstantSDNode>(MaskIdx)->getConstantIntValue()->equalsInt(i))
+      return SDValue();
+
+    // We only apply this if all elements come from the same vector with the
+    // same vector type.
+    if (!MaskSourceVec) {
+      MaskSourceVec = MaskSource->getOperand(0);
+      if (MaskSourceVec.getValueType() != VT)
+        return SDValue();
+    } else if (MaskSourceVec != MaskSource->getOperand(0)) {
+      return SDValue();
+    }
+  }
+
+  // We need a v16i8 for TBL, so we extend the source with a placeholder vector
+  // for v8i8 to get a v16i8. As the pattern we are replacing is extract +
+  // insert, we know that the index in the mask must be smaller than the number
+  // of elements in the source, or we would have an out-of-bounds access.
+  if (NumElts == 8)
+    SourceVec = DAG.getNode(ISD::CONCAT_VECTORS, dl, MVT::v16i8, SourceVec,
+                            DAG.getUNDEF(VT));
+
+  // Preconditions met, so we can use a vector (AND +) TBL to build this vector.
+  if (!AndMaskConstants.empty())
+    MaskSourceVec = DAG.getNode(ISD::AND, dl, VT, MaskSourceVec,
+                                DAG.getBuildVector(VT, dl, AndMaskConstants));
+
+  return DAG.getNode(
+      ISD::INTRINSIC_WO_CHAIN, dl, VT,
+      DAG.getConstant(Intrinsic::aarch64_neon_tbl1, dl, MVT::i32), SourceVec,
+      MaskSourceVec);
+}
+
 // Gather data to see if the operation can be modelled as a
 // shuffle in combination with VEXTs.
 SDValue AArch64TargetLowering::ReconstructShuffle(SDValue Op,
                                                   SelectionDAG &DAG) const {
   assert(Op.getOpcode() == ISD::BUILD_VECTOR && "Unknown opcode!");
   LLVM_DEBUG(dbgs() << "AArch64TargetLowering::ReconstructShuffle\n");
   SDLoc dl(Op);
   EVT VT = Op.getValueType();
   assert(!VT.isScalableVector() &&
SDValue AArch64TargetLowering::LowerBUILD_VECTOR(SDValue Op,
   // Detect patterns of a0,a1,a2,a3,b0,b1,b2,b3,c0,c1,c2,c3,d0,d1,d2,d3 from
   // v4i32s. This is really a truncate, which we can construct out of (legal)
   // concats and truncate nodes.
   if (SDValue M = ReconstructTruncateFromBuildVector(Op, DAG))
     return M;
 
   // Empirical tests suggest this is rarely worth it for vectors of length <= 2.
   if (NumElts >= 4) {
-    if (SDValue shuffle = ReconstructShuffle(Op, DAG))
-      return shuffle;
+    if (SDValue Shuffle = ReconstructShuffle(Op, DAG))
+      return Shuffle;
+
+    if (SDValue Shuffle = ReconstructShuffleWithRuntimeMask(Op, DAG))
+      return Shuffle;
   }
 
   if (PreferDUPAndInsert) {
     // First, build a constant vector with the common element.
     SmallVector<SDValue, 8> Ops(NumElts, Value);
     SDValue NewVector = LowerBUILD_VECTOR(DAG.getBuildVector(VT, dl, Ops), DAG);
     // Next, insert the elements that do not match the common value.
     for (unsigned I = 0; I < NumElts; ++I)

llvm/test/CodeGen/AArch64/neon-shuffle-vector-tbl.ll

+; RUN: llc < %s -mtriple=arm64-eabi -aarch64-neon-syntax=apple | FileCheck %s
+
+; This is the IR generated by Clang's __builtin_shufflevector for two 16x uint8_t vectors.
+define <16 x i8> @shuffle16_with_and_mask(<16 x i8> %src, <16 x i8> %mask) {
+; CHECK-LABEL: shuffle16_with_and_mask:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movi.16b v2, #15
+; CHECK-NEXT:    and.16b v1, v1, v2
+; CHECK-NEXT:    tbl.16b v0, { v0 }, v1
+; CHECK-NEXT:    ret
+
+  %masked_mask = and <16 x i8> %mask, <i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15>
+  %1 = extractelement <16 x i8> %masked_mask, i64 0
+  %2 = extractelement <16 x i8> %src, i8 %1
+  %3 = insertelement <16 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <16 x i8> %masked_mask, i64 1
+  %5 = extractelement <16 x i8> %src, i8 %4
+  %6 = insertelement <16 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <16 x i8> %masked_mask, i64 2
+  %8 = extractelement <16 x i8> %src, i8 %7
+  %9 = insertelement <16 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <16 x i8> %masked_mask, i64 3
+  %11 = extractelement <16 x i8> %src, i8 %10
+  %12 = insertelement <16 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <16 x i8> %masked_mask, i64 4
+  %14 = extractelement <16 x i8> %src, i8 %13
+  %15 = insertelement <16 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <16 x i8> %masked_mask, i64 5
+  %17 = extractelement <16 x i8> %src, i8 %16
+  %18 = insertelement <16 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <16 x i8> %masked_mask, i64 6
+  %20 = extractelement <16 x i8> %src, i8 %19
+  %21 = insertelement <16 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <16 x i8> %masked_mask, i64 7
+  %23 = extractelement <16 x i8> %src, i8 %22
+  %24 = insertelement <16 x i8> %21, i8 %23, i64 7
+  %25 = extractelement <16 x i8> %masked_mask, i64 8
+  %26 = extractelement <16 x i8> %src, i8 %25
+  %27 = insertelement <16 x i8> %24, i8 %26, i64 8
+  %28 = extractelement <16 x i8> %masked_mask, i64 9
+  %29 = extractelement <16 x i8> %src, i8 %28
+  %30 = insertelement <16 x i8> %27, i8 %29, i64 9
+  %31 = extractelement <16 x i8> %masked_mask, i64 10
+  %32 = extractelement <16 x i8> %src, i8 %31
+  %33 = insertelement <16 x i8> %30, i8 %32, i64 10
+  %34 = extractelement <16 x i8> %masked_mask, i64 11
+  %35 = extractelement <16 x i8> %src, i8 %34
+  %36 = insertelement <16 x i8> %33, i8 %35, i64 11
+  %37 = extractelement <16 x i8> %masked_mask, i64 12
+  %38 = extractelement <16 x i8> %src, i8 %37
+  %39 = insertelement <16 x i8> %36, i8 %38, i64 12
+  %40 = extractelement <16 x i8> %masked_mask, i64 13
+  %41 = extractelement <16 x i8> %src, i8 %40
+  %42 = insertelement <16 x i8> %39, i8 %41, i64 13
+  %43 = extractelement <16 x i8> %masked_mask, i64 14
+  %44 = extractelement <16 x i8> %src, i8 %43
+  %45 = insertelement <16 x i8> %42, i8 %44, i64 14
+  %46 = extractelement <16 x i8> %masked_mask, i64 15
+  ; Make sure that ANY_EXTEND is ignored
+  %47 = zext i8 %46 to i32
+  %48 = extractelement <16 x i8> %src, i32 %47
+  %49 = insertelement <16 x i8> %45, i8 %48, i64 15
+  ret <16 x i8> %49
+}
+
+define <8 x i8> @shuffle8_with_and_mask(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: shuffle8_with_and_mask:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    movi.8b v2, #7
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-NEXT:    and.8b v1, v1, v2
+; CHECK-NEXT:    tbl.8b v0, { v0 }, v1
+; CHECK-NEXT:    ret
+
+  %masked_mask = and <8 x i8> %mask, <i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7>
+  %1 = extractelement <8 x i8> %masked_mask, i64 0
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %masked_mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <8 x i8> %masked_mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %masked_mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <8 x i8> %masked_mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <8 x i8> %masked_mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <8 x i8> %masked_mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <8 x i8> %masked_mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}
+
+define <8 x i8> @shuffle8_with_and_mask_different_constants(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: LCPI2_0:
+; CHECK-NEXT:  .byte  3
+; CHECK-NEXT:  .byte  1
+; CHECK-NEXT:  .byte  7
+; CHECK-NEXT:  .byte  1
+; CHECK-NEXT:  .byte  7
+; CHECK-NEXT:  .byte  3
+; CHECK-NEXT:  .byte  7
+; CHECK-NEXT:  .byte  7
+
+; CHECK-LABEL: shuffle8_with_and_mask_different_constants:
+; CHECK:        // %bb.0:
+; CHECK-NEXT:   adrp  x8, .LCPI2_0
+; CHECK-NEXT:   // kill: def $d0 killed $d0 def $q0
+; CHECK-NEXT:   ldr d2, [x8, :lo12:.LCPI2_0]
+; CHECK-NEXT:   and.8b  v1, v1, v2
+; CHECK-NEXT:   tbl.8b  v0, { v0 }, v1
+; CHECK-NEXT:   ret
+
+  %masked_mask = and <8 x i8> %mask, <i8 3, i8 1, i8 7, i8 1, i8 7, i8 3, i8 7, i8 7>
+  %1 = extractelement <8 x i8> %masked_mask, i64 0
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %masked_mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <8 x i8> %masked_mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %masked_mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <8 x i8> %masked_mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <8 x i8> %masked_mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <8 x i8> %masked_mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <8 x i8> %masked_mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}
+
+define <8 x i8> @shuffle8_with_mask(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: shuffle8_with_mask:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:  // kill: def $d0 killed $d0 def $q0
+; CHECK-NEXT:  tbl.8b v0, { v0 }, v1
+; CHECK-NEXT:  ret
+
+  %1 = extractelement <8 x i8> %mask, i64 0
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <8 x i8> %mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <8 x i8> %mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <8 x i8> %mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <8 x i8> %mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <8 x i8> %mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}
+
+define <8 x i8> @no_shuffle_only_some_and_constants(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: no_shuffle_only_some_and_constants:
+; CHECK:       // %bb.0:
+; CHECK-NOT:   tbl.16b
+
+  ; Element at 0 has a AND mask, element at 1 does not.
+  %1 = extractelement <8 x i8> %mask, i64 0
+  %masked_elt1 = and i8 %1, 7
+  %2 = extractelement <8 x i8> %src, i8 %masked_elt1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+
+  %7 = extractelement <8 x i8> %mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <8 x i8> %mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <8 x i8> %mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <8 x i8> %mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <8 x i8> %mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}
+
+; Takes alternating entries from two mask source vectors. Currently not supported.
+define <16 x i8> @no_shuffle_with_two_mask_sources(<16 x i8> %src, <16 x i8> %mask1, <16 x i8> %mask2) {
+; CHECK-LABEL: shuffle_with_two_mask_sources:
+; CHECK:       // %bb.0:
+; CHECK-NOT:    tbl.16b
+
+  %masked_mask1 = and <16 x i8> %mask1, <i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15>
+  %masked_mask2 = and <16 x i8> %mask2, <i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15, i8 15>
+  %1 = extractelement <16 x i8> %masked_mask1, i64 0
+  %2 = extractelement <16 x i8> %src, i8 %1
+  %3 = insertelement <16 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <16 x i8> %masked_mask2, i64 1
+  %5 = extractelement <16 x i8> %src, i8 %4
+  %6 = insertelement <16 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <16 x i8> %masked_mask1, i64 2
+  %8 = extractelement <16 x i8> %src, i8 %7
+  %9 = insertelement <16 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <16 x i8> %masked_mask2, i64 3
+  %11 = extractelement <16 x i8> %src, i8 %10
+  %12 = insertelement <16 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <16 x i8> %masked_mask1, i64 4
+  %14 = extractelement <16 x i8> %src, i8 %13
+  %15 = insertelement <16 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <16 x i8> %masked_mask2, i64 5
+  %17 = extractelement <16 x i8> %src, i8 %16
+  %18 = insertelement <16 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <16 x i8> %masked_mask1, i64 6
+  %20 = extractelement <16 x i8> %src, i8 %19
+  %21 = insertelement <16 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <16 x i8> %masked_mask2, i64 7
+  %23 = extractelement <16 x i8> %src, i8 %22
+  %24 = insertelement <16 x i8> %21, i8 %23, i64 7
+  %25 = extractelement <16 x i8> %masked_mask1, i64 8
+  %26 = extractelement <16 x i8> %src, i8 %25
+  %27 = insertelement <16 x i8> %24, i8 %26, i64 8
+  %28 = extractelement <16 x i8> %masked_mask2, i64 9
+  %29 = extractelement <16 x i8> %src, i8 %28
+  %30 = insertelement <16 x i8> %27, i8 %29, i64 9
+  %31 = extractelement <16 x i8> %masked_mask1, i64 10
+  %32 = extractelement <16 x i8> %src, i8 %31
+  %33 = insertelement <16 x i8> %30, i8 %32, i64 10
+  %34 = extractelement <16 x i8> %masked_mask2, i64 11
+  %35 = extractelement <16 x i8> %src, i8 %34
+  %36 = insertelement <16 x i8> %33, i8 %35, i64 11
+  %37 = extractelement <16 x i8> %masked_mask1, i64 12
+  %38 = extractelement <16 x i8> %src, i8 %37
+  %39 = insertelement <16 x i8> %36, i8 %38, i64 12
+  %40 = extractelement <16 x i8> %masked_mask2, i64 13
+  %41 = extractelement <16 x i8> %src, i8 %40
+  %42 = insertelement <16 x i8> %39, i8 %41, i64 13
+  %43 = extractelement <16 x i8> %masked_mask1, i64 14
+  %44 = extractelement <16 x i8> %src, i8 %43
+  %45 = insertelement <16 x i8> %42, i8 %44, i64 14
+  %46 = extractelement <16 x i8> %masked_mask2, i64 15
+  %47 = extractelement <16 x i8> %src, i8 %46
+  %48 = insertelement <16 x i8> %45, i8 %47, i64 15
+  ret <16 x i8> %48
+}
+
+; Non-supported vector type.
+define <4 x i32> @no_for_shuffle_int_vector(<4 x i32> %src, <4 x i32> %mask) {
+; CHECK-LABEL: no_for_shuffle_int_vector:
+; CHECK:       // %bb.0:
+; CHECK-NOT:    tbl.16b
+
+  %masked_mask = and <4 x i32> %mask, <i32 3, i32 3, i32 3, i32 3>
+  %1 = extractelement <4 x i32> %masked_mask, i64 0
+  %2 = extractelement <4 x i32> %src, i32 %1
+  %3 = insertelement <4 x i32> undef, i32 %2, i64 0
+  %4 = extractelement <4 x i32> %masked_mask, i64 1
+  %5 = extractelement <4 x i32> %src, i32 %4
+  %6 = insertelement <4 x i32> %3, i32 %5, i64 1
+  %7 = extractelement <4 x i32> %masked_mask, i64 2
+  %8 = extractelement <4 x i32> %src, i32 %7
+  %9 = insertelement <4 x i32> %6, i32 %8, i64 2
+  %10 = extractelement <4 x i32> %masked_mask, i64 3
+  %11 = extractelement <4 x i32> %src, i32 %10
+  %12 = insertelement <4 x i32> %9, i32 %11, i64 3
+  ret <4 x i32> %12
+}
+
+define <8 x i8> @no_shuffle_not_enough_elements(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: no_shuffle_not_enough_elements:
+; CHECK:       // %bb.0:
+; CHECK-NOT:    tbl.16b
+
+  %masked_mask = and <8 x i8> %mask, <i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7>
+  %1 = extractelement <8 x i8> %masked_mask, i64 0
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %masked_mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <8 x i8> %masked_mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %masked_mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  ret <8 x i8> %12
+}
+
+define <8 x i8> @no_shuffle_different_vector_types(<8 x i8> %src, <16 x i8> %mask) {
+; CHECK-LABEL: no_shuffle_different_vector_types:
+; CHECK:       // %bb.0:
+; CHECK-NOT:    tbl.16b
+
+  %masked_mask = and <16 x i8> %mask, <i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7>
+  %1 = extractelement <16 x i8> %masked_mask, i64 0
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <16 x i8> %masked_mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <16 x i8> %masked_mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <16 x i8> %masked_mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <16 x i8> %masked_mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <16 x i8> %masked_mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <16 x i8> %masked_mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <16 x i8> %masked_mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}
+
+define <8 x i8> @no_shuffle_bad_mask_index(<8 x i8> %src, <8 x i8> %mask) {
+; CHECK-LABEL: no_shuffle_bad_mask_index:
+; CHECK:       // %bb.0:
+; CHECK-NOT:    tbl.16b
+
+  %masked_mask = and <8 x i8> %mask, <i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7, i8 7>
+
+  ; This should extract at 0, but because it extracts at 1, the pattern does not match.
+  %1 = extractelement <8 x i8> %masked_mask, i64 1
+
+  %2 = extractelement <8 x i8> %src, i8 %1
+  %3 = insertelement <8 x i8> undef, i8 %2, i64 0
+  %4 = extractelement <8 x i8> %masked_mask, i64 1
+  %5 = extractelement <8 x i8> %src, i8 %4
+  %6 = insertelement <8 x i8> %3, i8 %5, i64 1
+  %7 = extractelement <8 x i8> %masked_mask, i64 2
+  %8 = extractelement <8 x i8> %src, i8 %7
+  %9 = insertelement <8 x i8> %6, i8 %8, i64 2
+  %10 = extractelement <8 x i8> %masked_mask, i64 3
+  %11 = extractelement <8 x i8> %src, i8 %10
+  %12 = insertelement <8 x i8> %9, i8 %11, i64 3
+  %13 = extractelement <8 x i8> %masked_mask, i64 4
+  %14 = extractelement <8 x i8> %src, i8 %13
+  %15 = insertelement <8 x i8> %12, i8 %14, i64 4
+  %16 = extractelement <8 x i8> %masked_mask, i64 5
+  %17 = extractelement <8 x i8> %src, i8 %16
+  %18 = insertelement <8 x i8> %15, i8 %17, i64 5
+  %19 = extractelement <8 x i8> %masked_mask, i64 6
+  %20 = extractelement <8 x i8> %src, i8 %19
+  %21 = insertelement <8 x i8> %18, i8 %20, i64 6
+  %22 = extractelement <8 x i8> %masked_mask, i64 7
+  %23 = extractelement <8 x i8> %src, i8 %22
+  %24 = insertelement <8 x i8> %21, i8 %23, i64 7
+  ret <8 x i8> %24
+}