[AArch64] Extending lowering of 'zext <Y x i8> %x to <Y x i8X>' to use tbl instructions

[nilanjana_basu]

Summary

Adding support for ZExt lowering for destination types beyond the existing support for (8|16) x i32

[AArch64] Patch for lowering zext instructions to 'tbl' for (8|16)xi8 -> (8|16)xi32 conversions in D120571 is extended to support zext to 'tbl' lowering for Y x i8 to Y x i8X. Any arbitrary number of vector elements & any destination element type whose size is a multiple of 8, greater than 16 and less than 64, is allowed for this transformation.

Related microbenchmarks are in D136274 & D138059

Depends on D120571

[nilanjana_basu]

Added more test cases for zext lowering of different vector types

[nilanjana_basu]

Allowed the zext to tbl lowering for all lengths of vectors

[nilanjana_basu]

Added Big-Endian checks for the test cases that I missed earlier

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

Allowed all element sizes in the destination element that is a multiple of 8

[dmgreen]

I know it's not your problem, but the code in optimizeExtendOrTruncateConversion doesn't feel like it is in the best place, to be honest. CGP has always described itself as a hack, but we shouldn't be hacking things that much. There will be some obvious cases where the extend/trunc can be optimized but the tbl blocks it.
As far as I understand, the code is only in CGP because it is trying limit the transforms to loops. I'm wondering if it would be better to add some sort of flag into ISel so that combines could tell that the current block is a loop, and behave differently because of it.

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444 ↗

(On Diff #473877)

I think this is worse, I'm afraid. We only want to use tbl if it would replace two instructions (it performs two truncate/zext steps). Otherwise we are just adding instructions to the loop header (and using more registers) for no gain.

[nilanjana_basu]

Minor update to comment

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13871

This conversion shows a regression in performance for some cases where there are multiple similar zext instructions present back to back. The generated code with the previous implementation could be folded into a more optimized set of instructions, which is not possible with 'tbl' instructions. One example is 16xi8->16xi64, where I find an increase in the number of instructions after being lowered to tbl on using a loop interleave count of 4, i.e. with 4 back to back zext instructions.

Is it better to rule out this case in this 'if' block or should we not allow tbl-lowering when there are multiple zext instructions of the same type present back to back?

[fhahn]

In D136722#3917024, @dmgreen wrote:

I know it's not your problem, but the code in optimizeExtendOrTruncateConversion doesn't feel like it is in the best place, to be honest. CGP has always described itself as a hack, but we shouldn't be hacking things that much. There will be some obvious cases where the extend/trunc can be optimized but the tbl blocks it.
As far as I understand, the code is only in CGP because it is trying limit the transforms to loops. I'm wondering if it would be better to add some sort of flag into ISel so that combines could tell that the current block is a loop, and behave differently because of it.

Yep the only reason for doing it in CGP is to work around SelDAG's limitation.I am not sure about extending SelDAG for this, as we are planning to transition to GIsel at least on Darwin platforms very soon, which won't require doing this in CGP. I think @nilanjana_basu will also look into implementing this in GIsel

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13871

This conversion shows a regression in performance for some cases where there are multiple similar zext instructions present back to back. The generated code with the previous implementation could be folded into a more optimized set of instructions, which is not possible with 'tbl' instructions. One example is 16xi8->16xi64, where I find an increase in the number of instructions after being lowered to tbl on using a loop interleave count of 4, i.e. with 4 back to back zext instructions.

is this covered by one of the unit tests? zext_v16i8_to_v16i64_in_loop looks fine to me, at least for little endian

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444 ↗

(On Diff #473877)

yep it looks like we should have a check for that. @nilanjana_basu could you update the patch and make sure this is also tested in zext-to-tbl.ll

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1399

could you ad those new tests separately?

[nilanjana_basu]

Removed cases where TBL lowering will not be beneficial

[nilanjana_basu]

Rebasing & merging on a recent commit

[nilanjana_basu]

Rebased on latest updated zext unit tests

[nilanjana_basu]

Trying to fix patching error because of rebasing

[nilanjana_basu]

Trying to fix patching error again

[nilanjana_basu]

Fixed rebasing error of duplicated tests

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13871

I was mistaken in checking the instruction count earlier, but I have still added a unit test in zext_v16i8_to_v16i64_in_sequence_in_loop, since I see a performance regression in my local setup.

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444 ↗

(On Diff #473877)

In the latest patch, I've blocked zext lowering to tbl for destination vectors with i16 element type, since those were the ones that don't benefit from it. For other vector types, a single zext/truncate too improves on the instruction count.

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1399

I added two new tests for 2 back-to-back zext instructions - zext_v8i8_to_v8i64_with_add_in_sequence_in_loop & zext_v16i8_to_v16i64_in_sequence_in_loop. The pre-patch codegen has been updated in the parent revision D137993.

There seems to be a slight increase in instruction count for zext_v8i8_to_v8i64_with_add_in_sequence_in_loop.

[nilanjana_basu]

Blocked tbl-conversion for destination element size above 64 since only 2 or less destination vector elements can be chosen with each tbl instruction in these cases, making it less beneficial

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

Removed tbl-conversion cases to destination vector element width above 64, due to observed performance regressions. Will move this to a later patch, once we find a fix.

[fhahn]

Thanks for the latest update! This looks good in general to me, with just one more inline comment about an edge case

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13774

I *think* we could have something like zext i8 to i33 and then the division will drop the remainder. Could you add a test to see if that's the case and make sure we don't perform an incorrect transformation? It should be fine to just ignore cases where there would be remainder.

[nilanjana_basu]

Re-based on newly added tests

[nilanjana_basu]

Trying to fix patching error

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13774

I have also added testing for the Global-ISel path.

[nilanjana_basu]

Added an assert for an extra check

[fhahn]

LGTM, thanks!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::shouldSinkOperands(
   default:
     return false;
   }
   return false;
 }
 
 static void createTblShuffleForZExt(ZExtInst *ZExt, bool IsLittleEndian) {
   Value *Op = ZExt->getOperand(0);
-  auto *SrcTy = dyn_cast<FixedVectorType>(Op->getType());
-  auto *DstTy = dyn_cast<FixedVectorType>(ZExt->getType());
+  auto *SrcTy = cast<FixedVectorType>(Op->getType());
+  auto *DstTy = cast<FixedVectorType>(ZExt->getType());
+  unsigned ZExtFactor =
+      (cast<IntegerType>(DstTy->getElementType())->getBitWidth()) /
+      (cast<IntegerType>(SrcTy->getElementType())->getBitWidth());
   unsigned NumElts = SrcTy->getNumElements();
   IRBuilder<> Builder(ZExt);
-  SmallVector<int> Mask(4 * NumElts, NumElts);
-  // Create a mask that selects <0,0,0,Op[i]> for each lane of vector of i32 to
-  // replace the original ZExt. This can later be lowered to a set of tbl
-  // instructions.
-  for (unsigned i = 0; i < NumElts; i++) {
-    if (IsLittleEndian)
-      Mask[i * 4] = i;
+  SmallVector<int> Mask;
+  // Create a mask that selects <0,...,Op[i]> for each lane of the destination
+  // vector to replace the original ZExt. This can later be lowered to a set of
+  // tbl instructions.
+  for (unsigned i = 0; i < NumElts * ZExtFactor; i++) {
+    if (IsLittleEndian) {
+      if (i % ZExtFactor == 0)
+        Mask.push_back(i / ZExtFactor);
+      else
+        Mask.push_back(NumElts);
+    } else {
+      if ((i + 1) % ZExtFactor == 0)
+        Mask.push_back((i - ZExtFactor + 1) / ZExtFactor);
       else
-      Mask[i * 4 + 3] = i;
+        Mask.push_back(NumElts);
+    }
   }
 
   auto *FirstEltZero = Builder.CreateInsertElement(
       PoisonValue::get(SrcTy), Builder.getInt8(0), uint64_t(0));
   Value *Result = Builder.CreateShuffleVector(Op, FirstEltZero, Mask);
   Result = Builder.CreateBitCast(Result, DstTy);
   ZExt->replaceAllUsesWith(Result);
   ZExt->eraseFromParent();
if (!L || L->getHeader() != I->getParent() || F->hasMinSize() ||
       F->hasOptSize())
     return false;
 
   auto *SrcTy = dyn_cast<FixedVectorType>(I->getOperand(0)->getType());
   auto *DstTy = dyn_cast<FixedVectorType>(I->getType());
   if (!SrcTy || !DstTy)
     return false;
 
-  // Convert 'zext <(8|16) x i8> %x to <(8|16) x i32>' to a shuffle that can be
-  // lowered to either 2 or 4 tbl instructions to insert the original i8
-  // elements into i32 lanes.
+  // Convert 'zext <Y x i8> %x to <Y x i8X>' to a shuffle that can be
+  // lowered to tbl instructions to insert the original i8 elements
+  // into i8x lanes. Conversion to <Y x i16> is ignored as using tbl is not
+  // optimal for this case
   auto *ZExt = dyn_cast<ZExtInst>(I);
-  if (ZExt && (SrcTy->getNumElements() == 8 || SrcTy->getNumElements() == 16) &&
-      SrcTy->getElementType()->isIntegerTy(8) &&
-      DstTy->getElementType()->isIntegerTy(32)) {
+  if (ZExt && SrcTy->getElementType()->isIntegerTy(8) &&
+      (cast<IntegerType>(DstTy->getElementType())->getBitWidth() % 8 == 0) &&
+      !DstTy->getElementType()->isIntegerTy(16)) {
     createTblShuffleForZExt(ZExt, Subtarget->isLittleEndian());
     return true;
   }
 
   auto *UIToFP = dyn_cast<UIToFPInst>(I);
-  if (UIToFP &&
-      (SrcTy->getNumElements() == 8 || SrcTy->getNumElements() == 16) &&
-      SrcTy->getElementType()->isIntegerTy(8) &&
+  if (UIToFP && SrcTy->getElementType()->isIntegerTy(8) &&
       DstTy->getElementType()->isFloatTy()) {
     IRBuilder<> Builder(I);
     auto *ZExt = cast<ZExtInst>(
         Builder.CreateZExt(I->getOperand(0), VectorType::getInteger(DstTy)));
     auto *UI = Builder.CreateUIToFP(ZExt, DstTy);
     I->replaceAllUsesWith(UI);
     I->eraseFromParent();
     createTblShuffleForZExt(ZExt, Subtarget->isLittleEndian());

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
-
-
 define void @zext_v16i8_to_v16i16_in_loop(i8* %src, i16* %dst) {
 ; CHECK-LABEL: zext_v16i8_to_v16i16_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  LBB5_1: ; %loop
-; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q0, [x0, x8]
-; CHECK-NEXT:    add x8, x8, #16
-; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    ushll2.8h v1, v0, #0
-; CHECK-NEXT:    ushll.8h v0, v0, #0
-; CHECK-NEXT:    stp q0, q1, [x1], #32
-; CHECK-NEXT:    b.ne LBB5_1
-; CHECK-NEXT:  ; %bb.2: ; %exit
-; CHECK-NEXT:    ret
+; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT:  LBB5_1:                                 ; %loop
+; CHECK-NEXT:                                          ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:   ldr q0, [x0, x8]
+; CHECK-NEXT:   add x8, x8, #16
+; CHECK-NEXT:   cmp x8, #128
+; CHECK-NEXT:   ushll2.8h v1, v0, #0
+; CHECK-NEXT:   ushll.8h v0, v0, #0
+; CHECK-NEXT:   stp q0, q1, [x1], #32
+; CHECK-NEXT:   b.ne  LBB5_1
+; CHECK-NEXT:  ; %bb.2:                                ; %exit
+; CHECK-NEXT:   ret
+
 ;
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i16_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
-; CHECK-BE-NEXT:    mov x8, xzr
-; CHECK-BE-NEXT:  .LBB5_1: // %loop
-; CHECK-BE-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #16
+; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:  .LBB5_1:                                // %loop
+; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
+; CHECK-BE-NEXT:    add x9, x0, x8
+; CHECK-BE-NEXT:    add x8, x8, #16
+; CHECK-BE-NEXT:    cmp x8, #128
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #16
 ; CHECK-BE-NEXT:    ushll v1.8h, v0.8b, #0
 ; CHECK-BE-NEXT:    ushll2 v0.8h, v0.16b, #0
-; CHECK-BE-NEXT:    st1 { v1.8h }, [x1]
-; CHECK-BE-NEXT:    add x1, x1, #32
-; CHECK-BE-NEXT:    st1 { v0.8h }, [x9]
-; CHECK-BE-NEXT:    b.ne .LBB5_1
-; CHECK-BE-NEXT:  // %bb.2: // %exit
+; CHECK-BE-NEXT:    st1 { v1.8h }, [x1]
+; CHECK-BE-NEXT:    add x1, x1, #32
+; CHECK-BE-NEXT:    st1 { v0.8h }, [x9]
+; CHECK-BE-NEXT:    b.ne  .LBB5_1
+; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
+
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <16 x i8>*
   %load = load <16 x i8>, <16 x i8>* %src.gep.cast
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI7_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_1:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_2:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_3:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_4:
+; CHECK-NEXT:   .byte 8                               ; 0x8
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 9                               ; 0x9
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_5:
+; CHECK-NEXT:   .byte 10                              ; 0xa
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 11                              ; 0xb
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_6:
+; CHECK-NEXT:   .byte 12                              ; 0xc
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 13                              ; 0xd
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_7:
+; CHECK-NEXT:   .byte 14                              ; 0xe
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 15                              ; 0xf
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI7_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI7_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI7_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .LCPI7_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+; CHECK-BE-NEXT:  .LCPI7_4:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 8                               // 0x8
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 9                               // 0x9
+; CHECK-BE-NEXT:  .LCPI7_5:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 10                              // 0xa
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 11                              // 0xb
+; CHECK-BE-NEXT:  .LCPI7_6:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 12                              // 0xc
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 13                              // 0xd
+; CHECK-BE-NEXT:  .LCPI7_7:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 14                              // 0xe
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 15                              // 0xf
+
 define void @zext_v16i8_to_v16i64_in_loop(i8* %src, i64* %dst) {
 ; CHECK-LABEL: zext_v16i8_to_v16i64_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  LBB7_1: ; %loop
-; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q0, [x0, x8]
-; CHECK-NEXT:    add x8, x8, #16
-; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    ushll.8h v1, v0, #0
-; CHECK-NEXT:    ushll2.8h v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v2, v1, #0
-; CHECK-NEXT:    ushll2.4s v3, v0, #0
-; CHECK-NEXT:    ushll.4s v0, v0, #0
-; CHECK-NEXT:    ushll2.2d v4, v3, #0
-; CHECK-NEXT:    ushll2.2d v5, v0, #0
-; CHECK-NEXT:    ushll.2d v0, v0, #0
-; CHECK-NEXT:    ushll.2d v3, v3, #0
-; CHECK-NEXT:    stp q0, q5, [x1, #64]
-; CHECK-NEXT:    ushll.4s v0, v1, #0
-; CHECK-NEXT:    stp q3, q4, [x1, #96]
-; CHECK-NEXT:    ushll2.2d v3, v2, #0
-; CHECK-NEXT:    ushll.2d v2, v2, #0
-; CHECK-NEXT:    ushll2.2d v1, v0, #0
-; CHECK-NEXT:    ushll.2d v0, v0, #0
-; CHECK-NEXT:    stp q2, q3, [x1, #32]
-; CHECK-NEXT:    stp q0, q1, [x1], #128
-; CHECK-NEXT:    b.ne LBB7_1
-; CHECK-NEXT:  ; %bb.2: ; %exit
-; CHECK-NEXT:    ret
-;
+; CHECK-NEXT:    Lloh12:
+; CHECK-NEXT:     adrp  x9, lCPI7_0@PAGE
+; CHECK-NEXT:    Lloh13:
+; CHECK-NEXT:     adrp  x10, lCPI7_1@PAGE
+; CHECK-NEXT:    Lloh14:
+; CHECK-NEXT:     adrp  x11, lCPI7_2@PAGE
+; CHECK-NEXT:     mov x8, xzr
+; CHECK-NEXT:    Lloh15:
+; CHECK-NEXT:     ldr q0, [x9, lCPI7_0@PAGEOFF]
+; CHECK-NEXT:    Lloh16:
+; CHECK-NEXT:     adrp  x9, lCPI7_3@PAGE
+; CHECK-NEXT:    Lloh17:
+; CHECK-NEXT:     ldr q1, [x10, lCPI7_1@PAGEOFF]
+; CHECK-NEXT:    Lloh18:
+; CHECK-NEXT:     adrp  x10, lCPI7_4@PAGE
+; CHECK-NEXT:    Lloh19:
+; CHECK-NEXT:     ldr q2, [x11, lCPI7_2@PAGEOFF]
+; CHECK-NEXT:    Lloh20:
+; CHECK-NEXT:     adrp  x11, lCPI7_5@PAGE
+; CHECK-NEXT:    Lloh21:
+; CHECK-NEXT:     ldr q3, [x9, lCPI7_3@PAGEOFF]
+; CHECK-NEXT:    Lloh22:
+; CHECK-NEXT:     adrp  x9, lCPI7_6@PAGE
+; CHECK-NEXT:    Lloh23:
+; CHECK-NEXT:     ldr q4, [x10, lCPI7_4@PAGEOFF]
+; CHECK-NEXT:    Lloh24:
+; CHECK-NEXT:     adrp  x10, lCPI7_7@PAGE
+; CHECK-NEXT:    Lloh25:
+; CHECK-NEXT:     ldr q5, [x11, lCPI7_5@PAGEOFF]
+; CHECK-NEXT:    Lloh26:
+; CHECK-NEXT:     ldr q6, [x9, lCPI7_6@PAGEOFF]
+; CHECK-NEXT:    Lloh27:
+; CHECK-NEXT:     ldr q7, [x10, lCPI7_7@PAGEOFF]
+; CHECK-NEXT:    LBB7_1:                                 ; %loop
+; CHECK-NEXT:                                            ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:     ldr q16, [x0, x8]
+; CHECK-NEXT:     add x8, x8, #16
+; CHECK-NEXT:     cmp x8, #128
+; CHECK-NEXT:     tbl.16b v17, { v16 }, v7
+; CHECK-NEXT:     tbl.16b v18, { v16 }, v6
+; CHECK-NEXT:     tbl.16b v19, { v16 }, v5
+; CHECK-NEXT:     tbl.16b v20, { v16 }, v4
+; CHECK-NEXT:     tbl.16b v21, { v16 }, v3
+; CHECK-NEXT:     stp q18, q17, [x1, #96]
+; CHECK-NEXT:     tbl.16b v17, { v16 }, v2
+; CHECK-NEXT:     tbl.16b v18, { v16 }, v1
+; CHECK-NEXT:     stp q20, q19, [x1, #64]
+; CHECK-NEXT:     tbl.16b v16, { v16 }, v0
+; CHECK-NEXT:     stp q17, q21, [x1, #32]
+; CHECK-NEXT:     stp q16, q18, [x1], #128
+; CHECK-NEXT:     b.ne  LBB7_1
+; CHECK-NEXT:    ; %bb.2:                                ; %exit
+; CHECK-NEXT:     ret
+; CHECK-NEXT:     .loh AdrpLdr  Lloh24, Lloh27
+; CHECK-NEXT:     .loh AdrpLdr  Lloh22, Lloh26
+; CHECK-NEXT:     .loh AdrpLdr  Lloh20, Lloh25
+; CHECK-NEXT:     .loh AdrpAdrp Lloh18, Lloh24
+; CHECK-NEXT:     .loh AdrpLdr  Lloh18, Lloh23
+; CHECK-NEXT:     .loh AdrpAdrp Lloh16, Lloh22
+; CHECK-NEXT:     .loh AdrpLdr  Lloh16, Lloh21
+; CHECK-NEXT:     .loh AdrpAdrp Lloh14, Lloh20
+; CHECK-NEXT:     .loh AdrpLdr  Lloh14, Lloh19
+; CHECK-NEXT:     .loh AdrpAdrp Lloh13, Lloh18
+; CHECK-NEXT:     .loh AdrpLdr  Lloh13, Lloh17
+; CHECK-NEXT:     .loh AdrpAdrp Lloh12, Lloh16
+; CHECK-NEXT:     .loh AdrpLdr  Lloh12, Lloh15
+
+
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i64_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
-; CHECK-BE-NEXT:    mov x8, xzr
-; CHECK-BE-NEXT:  .LBB7_1: // %loop
-; CHECK-BE-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x10, x1, #96
-; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #112
-; CHECK-BE-NEXT:    ushll2 v1.8h, v0.16b, #0
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2 v2.4s, v1.8h, #0
-; CHECK-BE-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-BE-NEXT:    ushll2 v3.2d, v2.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #80
-; CHECK-BE-NEXT:    ushll2 v3.2d, v1.4s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x10]
-; CHECK-BE-NEXT:    ushll2 v2.4s, v0.8h, #0
-; CHECK-BE-NEXT:    add x10, x1, #48
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #64
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    ushll2 v4.2d, v2.4s, #0
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x9]
-; CHECK-BE-NEXT:    ushll v1.2d, v0.2s, #0
-; CHECK-BE-NEXT:    add x9, x1, #16
-; CHECK-BE-NEXT:    st1 { v4.2d }, [x10]
-; CHECK-BE-NEXT:    add x10, x1, #32
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x1]
-; CHECK-BE-NEXT:    add x1, x1, #128
-; CHECK-BE-NEXT:    ushll2 v0.2d, v0.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x9]
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x10]
-; CHECK-BE-NEXT:    b.ne .LBB7_1
-; CHECK-BE-NEXT:  // %bb.2: // %exit
-; CHECK-BE-NEXT:    ret
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_2
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_3
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_4
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_4
+; CHECK-BE-NEXT:    ld1 { v4.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_5
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_5
+; CHECK-BE-NEXT:    ld1 { v5.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_6
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_6
+; CHECK-BE-NEXT:    ld1 { v6.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_7
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_7
+; CHECK-BE-NEXT:    ld1 { v7.16b }, [x8]
+; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:  .LBB7_1:                                // %loop
+; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
+; CHECK-BE-NEXT:    add x9, x0, x8
+; CHECK-BE-NEXT:    add x10, x1, #96
+; CHECK-BE-NEXT:    add x8, x8, #16
+; CHECK-BE-NEXT:    cmp x8, #128
+; CHECK-BE-NEXT:    ld1 { v16.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #112
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v7.16b
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v6.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v5.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #80
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v4.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x1, #64
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #48
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v3.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v2.16b
+; CHECK-BE-NEXT:    add x10, x1, #32
+; CHECK-BE-NEXT:    tbl v16.16b, { v16.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #16
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x1]
+; CHECK-BE-NEXT:    add x1, x1, #128
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x9]
+; CHECK-BE-NEXT:    b.ne  .LBB7_1
+; CHECK-BE-NEXT:  // %bb.2:                               // %exit
+; CHECK-BE-NEXT:    ret
+
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <16 x i8>*
   %load = load <16 x i8>, <16 x i8>* %src.gep.cast
   %ext = zext <16 x i8> %load to <16 x i64>
   %dst.gep = getelementptr i64, i64* %dst, i64 %iv
   %dst.gep.cast = bitcast i64* %dst.gep to <16 x i64>*
   store <16 x i64> %ext, <16 x i64>* %dst.gep.cast
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL:  lCPI8_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI8_1:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI8_2:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI8_3:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL: .LCPI8_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI8_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI8_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .LCPI8_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+
 define void @zext_v8i8_to_v8i64_in_loop(i8* %src, i64* %dst) {
-; CHECK-LABEL: _zext_v8i8_to_v8i64_in_loop:
-; CHECK: ; %bb.0:                                ; %entry
-; CHECK-NEXT:   mov x8, xzr
-; CHECK-NEXT:  LBB8_1:                                 ; %loop
-; CHECK-NEXT:                                          ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:   ldr d0, [x0, x8]
-; CHECK-NEXT:   add x8, x8, #16
-; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   ushll2.4s v1, v0, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   ushll2.2d v2, v1, #0
-; CHECK-NEXT:   ushll.2d  v1, v1, #0
-; CHECK-NEXT:   ushll2.2d v3, v0, #0
-; CHECK-NEXT:   ushll.2d  v0, v0, #0
-; CHECK-NEXT:   stp q1, q2, [x1, #32]
-; CHECK-NEXT:   stp q0, q3, [x1], #128
-; CHECK-NEXT:   b.ne  LBB8_1
-; CHECK-NEXT:  ; %bb.2:                                ; %exit
-; CHECK-NEXT:   ret
+; CHECK-LABEL:  _zext_v8i8_to_v8i64_in_loop:
+; CHECK:  ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh28:
+; CHECK-NEXT:   adrp  x9, lCPI8_0@PAGE
+; CHECK-NEXT: Lloh29:
+; CHECK-NEXT:   adrp  x10, lCPI8_1@PAGE
+; CHECK-NEXT: Lloh30:
+; CHECK-NEXT:   adrp  x11, lCPI8_2@PAGE
+; CHECK-NEXT: Lloh31:
+; CHECK-NEXT:   adrp  x12, lCPI8_3@PAGE
+; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh32:
+; CHECK-NEXT:   ldr q0, [x9, lCPI8_0@PAGEOFF]
+; CHECK-NEXT: Lloh33:
+; CHECK-NEXT:   ldr q1, [x10, lCPI8_1@PAGEOFF]
+; CHECK-NEXT: Lloh34:
+; CHECK-NEXT:   ldr q2, [x11, lCPI8_2@PAGEOFF]
+; CHECK-NEXT: Lloh35:
+; CHECK-NEXT:   ldr q3, [x12, lCPI8_3@PAGEOFF]
+; CHECK-NEXT: LBB8_1:                                 ; %loop
+; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:   ldr d4, [x0, x8]
+; CHECK-NEXT:   add x8, x8, #16
+; CHECK-NEXT:   cmp x8, #128
+; CHECK-NEXT:   tbl.16b v5, { v4 }, v3
+; CHECK-NEXT:   tbl.16b v6, { v4 }, v2
+; CHECK-NEXT:   tbl.16b v7, { v4 }, v1
+; CHECK-NEXT:   tbl.16b v4, { v4 }, v0
+; CHECK-NEXT:   stp q6, q5, [x1, #32]
+; CHECK-NEXT:   stp q4, q7, [x1], #128
+; CHECK-NEXT:   b.ne  LBB8_1
+; CHECK-NEXT: ; %bb.2:                                ; %exit
+; CHECK-NEXT:   ret
+; CHECK-NEXT:   .loh AdrpLdr  Lloh31, Lloh35
+; CHECK-NEXT:   .loh AdrpLdr  Lloh30, Lloh34
+; CHECK-NEXT:   .loh AdrpLdr  Lloh29, Lloh33
+; CHECK-NEXT:   .loh AdrpLdr  Lloh28, Lloh32
 
-; CHECK-BE-LABEL: zext_v8i8_to_v8i64_in_loop:
-; CHECK-BE: // %bb.0:                               // %entry
+; CHECK-BE-LABEL:  zext_v8i8_to_v8i64_in_loop:
+; CHECK-BE:  // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x8, .LCPI8_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI8_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI8_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI8_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI8_2
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI8_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI8_3
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI8_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x8]
 ; CHECK-BE-NEXT:    mov x8, xzr
 ; CHECK-BE-NEXT:  .LBB8_1:                                // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    add x9, x0, x8
 ; CHECK-BE-NEXT:    add x10, x1, #32
 ; CHECK-BE-NEXT:    add x8, x8, #16
 ; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.8b }, [x9]
+; CHECK-BE-NEXT:    ld1 { v4.8b }, [x9]
 ; CHECK-BE-NEXT:    add x9, x1, #48
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2  v1.4s, v0.8h, #0
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    ushll2  v2.2d, v1.4s, #0
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x9]
+; CHECK-BE-NEXT:    tbl v5.16b, { v4.16b }, v3.16b
+; CHECK-BE-NEXT:    tbl v6.16b, { v4.16b }, v0.16b
+; CHECK-BE-NEXT:    tbl v7.16b, { v4.16b }, v2.16b
+; CHECK-BE-NEXT:    tbl v4.16b, { v4.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v5.16b }, [x9]
 ; CHECK-BE-NEXT:    add x9, x1, #16
-; CHECK-BE-NEXT:    ushll v2.2d, v0.2s, #0
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x10]
-; CHECK-BE-NEXT:    ushll2  v0.2d, v0.4s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x1]
+; CHECK-BE-NEXT:    st1 { v6.16b }, [x1]
 ; CHECK-BE-NEXT:    add x1, x1, #128
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x9]
+; CHECK-BE-NEXT:    st1 { v7.16b }, [x10]
+; CHECK-BE-NEXT:    st1 { v4.16b }, [x9]
 ; CHECK-BE-NEXT:    b.ne  .LBB8_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
 entry:
   br label %loop
 
 loop:
loop:
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
 define void @zext_v8i8_to_v8i16_in_loop(i8* %src, i16* %dst) {
-; CHECK-LABEL:  _zext_v8i8_to_v8i16_in_loop:            ; @zext_v8i8_to_v8i16_in_loop
+; CHECK-LABEL:  _zext_v8i8_to_v8i16_in_loop:
 ; CHECK:  ; %bb.0:                                ; %entry
-; CHECK-NEXT:   mov x8, xzr
-; CHECK-NEXT: LBB9_1:                                 ; %loop
-; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:   ldr d0, [x0, x8]
-; CHECK-NEXT:   add x8, x8, #16
-; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   str q0, [x1], #32
-; CHECK-NEXT:   b.ne  LBB9_1
-; CHECK-NEXT: ; %bb.2:                                ; %exit
-; CHECK-NEXT:   ret
+; CHECK-NEXT: mov x8, xzr
+; CHECK-NEXT:LBB9_1:                                 ; %loop
+; CHECK-NEXT:                                        ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT: ldr d0, [x0, x8]
+; CHECK-NEXT: add x8, x8, #16
+; CHECK-NEXT: cmp x8, #128
+; CHECK-NEXT: ushll.8h  v0, v0, #0
+; CHECK-NEXT: str q0, [x1], #32
+; CHECK-NEXT: b.ne  LBB9_1
+; CHECK-NEXT:; %bb.2:                                ; %exit
+; CHECK-NEXT: ret
 
-; CHECK-BE-LABEL:  zext_v8i8_to_v8i16_in_loop:             // @zext_v8i8_to_v8i16_in_loop
-; CHECK-BE:   .cfi_startproc
-; CHECK-BE-NEXT:  // %bb.0:                               // %entry
-; CHECK-BE-NEXT:    mov x8, xzr
-; CHECK-BE-NEXT:  .LBB9_1:                                // %loop
-; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.8b }, [x9]
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    st1 { v0.8h }, [x1]
-; CHECK-BE-NEXT:    add x1, x1, #32
-; CHECK-BE-NEXT:    b.ne  .LBB9_1
-; CHECK-BE-NEXT:  // %bb.2:                               // %exit
-; CHECK-BE-NEXT:    ret
+
+; CHECK-BE-LABEL: zext_v8i8_to_v8i16_in_loop:
+; CHECK-BE:  // %bb.0:                               // %entry
+; CHECK-BE-NEXT:  mov x8, xzr
+; CHECK-BE-NEXT: .LBB9_1:                                // %loop
+; CHECK-BE-NEXT:                                         // =>This Inner Loop Header: Depth=1
+; CHECK-BE-NEXT:  add x9, x0, x8
+; CHECK-BE-NEXT:  add x8, x8, #16
+; CHECK-BE-NEXT:  cmp x8, #128
+; CHECK-BE-NEXT:  ld1 { v0.8b }, [x9]
+; CHECK-BE-NEXT:  ushll v0.8h, v0.8b, #0
+; CHECK-BE-NEXT:  st1 { v0.8h }, [x1]
+; CHECK-BE-NEXT:  add x1, x1, #32
+; CHECK-BE-NEXT:  b.ne  .LBB9_1
+; CHECK-BE-NEXT: // %bb.2:                               // %exit
+; CHECK-BE-NEXT:  ret
 
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <8 x i8>*
   %load = load <8 x i8>, <8 x i8>* %src.gep.cast
   %ext = zext <8 x i8> %load to <8 x i16>
   %dst.gep = getelementptr i16, i16* %dst, i64 %iv
   %dst.gep.cast = bitcast i16* %dst.gep to <8 x i16>*
   store <8 x i16> %ext, <8 x i16>* %dst.gep.cast
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128

 ; CHECK-BE-NEXT:    lsr w13, w14, #4
 ; CHECK-BE-NEXT:    lsr w14, w16, #4
 ; CHECK-BE-NEXT:    strh  w10, [x1, #18]
 ; CHECK-BE-NEXT:    extr  x9, x13, x9, #16
 ; CHECK-BE-NEXT:    strh  w12, [x1, #8]
 ; CHECK-BE-NEXT:    extr  x10, x14, x11, #16
 ; CHECK-BE-NEXT:    stur  x9, [x1, #10]
 ; CHECK-BE-NEXT:    str x10, [x1], #64
+; CHECK-BE-NEXT:    b.ne  .LBB10_1
+; CHECK-BE-NEXT:  // %bb.2:                               // %exit
+; CHECK-BE-NEXT:    ret
 
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <8 x i8>*
   %load = load <8 x i8>, <8 x i8>* %src.gep.cast
   %ext = zext <8 x i8> %load to <8 x i20>
   %dst.gep = getelementptr i20, i20* %dst, i64 %iv
   %dst.gep.cast = bitcast i20* %dst.gep to <8 x i20>*
   store <8 x i20> %ext, <8 x i20>* %dst.gep.cast
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI11_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI11_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+
 define void @zext_v4i8_to_v4i32_in_loop(i8* %src, i32* %dst) {
 ; CHECK-LABEL:  _zext_v4i8_to_v4i32_in_loop:
-; CHECK:  ; %bb.0:                                ; %entry
+; CHECK: ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh36:
+; CHECK-NEXT:   adrp  x9, lCPI11_0@PAGE
 ; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh37:
+; CHECK-NEXT:   ldr q0, [x9, lCPI11_0@PAGEOFF]
 ; CHECK-NEXT: LBB11_1:                                ; %loop
 ; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:   ldr s0, [x0, x8]
+; CHECK-NEXT:   ldr s1, [x0, x8]
 ; CHECK-NEXT:   add x8, x8, #16
 ; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   str q0, [x1], #64
+; CHECK-NEXT:   tbl.16b v1, { v1 }, v0
+; CHECK-NEXT:   str q1, [x1], #64
 ; CHECK-NEXT:   b.ne  LBB11_1
 ; CHECK-NEXT: ; %bb.2:                                ; %exit
 ; CHECK-NEXT:   ret
 
 ; CHECK-BE-LABEL: zext_v4i8_to_v4i32_in_loop:
 ; CHECK-BE: // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x8, .LCPI11_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI11_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    mov x8, xzr
 ; CHECK-BE-NEXT:  .LBB11_1:                               // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    ldr s0, [x0, x8]
+; CHECK-BE-NEXT:    ldr s1, [x0, x8]
 ; CHECK-BE-NEXT:    add x8, x8, #16
 ; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    rev32 v0.8b, v0.8b
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    st1 { v0.4s }, [x1]
+; CHECK-BE-NEXT:    rev32 v1.16b, v1.16b
+; CHECK-BE-NEXT:    tbl v1.16b, { v1.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v1.16b }, [x1]
 ; CHECK-BE-NEXT:    add x1, x1, #64
 ; CHECK-BE-NEXT:    b.ne  .LBB11_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
 entry:
   br label %loop
 
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI12_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI12_1:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI12_2:
+; CHECK-NEXT:   .byte 8                               ; 0x8
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 9                               ; 0x9
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 10                              ; 0xa
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 11                              ; 0xb
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL: .LCPI12_0:
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 0                               // 0x0
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 2                               // 0x2
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 3                               // 0x3
+; CHECK-BE-NEXT: .LCPI12_1:
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 4                               // 0x4
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 6                               // 0x6
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 7                               // 0x7
+; CHECK-BE-NEXT: .LCPI12_2:
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 8                               // 0x8
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 9                               // 0x9
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 10                              // 0xa
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 255                             // 0xff
+; CHECK-BE-NEXT:  .byte 11                              // 0xb
+
 define void @zext_v12i8_to_v12i32_in_loop(i8* %src, i32* %dst) {
 ; CHECK-LABEL:  _zext_v12i8_to_v12i32_in_loop:
 ; CHECK:  ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh38:
+; CHECK-NEXT:   adrp  x9, lCPI12_0@PAGE
+; CHECK-NEXT: Lloh39:
+; CHECK-NEXT:   adrp  x10, lCPI12_1@PAGE
+; CHECK-NEXT: Lloh40:
+; CHECK-NEXT:   adrp  x11, lCPI12_2@PAGE
 ; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh41:
+; CHECK-NEXT:   ldr q0, [x9, lCPI12_0@PAGEOFF]
+; CHECK-NEXT: Lloh42:
+; CHECK-NEXT:   ldr q1, [x10, lCPI12_1@PAGEOFF]
+; CHECK-NEXT: Lloh43:
+; CHECK-NEXT:   ldr q2, [x11, lCPI12_2@PAGEOFF]
 ; CHECK-NEXT: LBB12_1:                                ; %loop
 ; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:   ldr q0, [x0, x8]
+; CHECK-NEXT:   ldr q3, [x0, x8]
 ; CHECK-NEXT:   add x8, x8, #16
 ; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ushll2.8h v1, v0, #0
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   ushll.4s  v1, v1, #0
-; CHECK-NEXT:   ushll2.4s v2, v0, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   stp q2, q1, [x1, #16]
-; CHECK-NEXT:   str q0, [x1], #64
+; CHECK-NEXT:   tbl.16b v4, { v3 }, v2
+; CHECK-NEXT:   tbl.16b v5, { v3 }, v1
+; CHECK-NEXT:   tbl.16b v3, { v3 }, v0
+; CHECK-NEXT:   stp q5, q4, [x1, #16]
+; CHECK-NEXT:   str q3, [x1], #64
 ; CHECK-NEXT:   b.ne  LBB12_1
 ; CHECK-NEXT: ; %bb.2:                                ; %exit
 ; CHECK-NEXT:   ret
+; CHECK-NEXT:   .loh AdrpLdr  Lloh40, Lloh43
+; CHECK-NEXT:   .loh AdrpLdr  Lloh39, Lloh42
+; CHECK-NEXT:   .loh AdrpLdr  Lloh38, Lloh41
 
 ; CHECK-BE-LABEL: zext_v12i8_to_v12i32_in_loop:
 ; CHECK-BE: // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x8, .LCPI12_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI12_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI12_2
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x8]
 ; CHECK-BE-NEXT:    mov x8, xzr
 ; CHECK-BE-NEXT:  .LBB12_1:                               // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    add x9, x0, x8
 ; CHECK-BE-NEXT:    add x10, x1, #16
 ; CHECK-BE-NEXT:    add x8, x8, #16
 ; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x9]
 ; CHECK-BE-NEXT:    add x9, x1, #32
-; CHECK-BE-NEXT:    ushll v1.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2  v0.8h, v0.16b, #0
-; CHECK-BE-NEXT:    ushll v2.4s, v1.4h, #0
-; CHECK-BE-NEXT:    ushll2  v1.4s, v1.8h, #0
-; CHECK-BE-NEXT:    st1 { v2.4s }, [x1]
+; CHECK-BE-NEXT:    tbl v4.16b, { v3.16b }, v0.16b
+; CHECK-BE-NEXT:    tbl v5.16b, { v3.16b }, v2.16b
+; CHECK-BE-NEXT:    tbl v3.16b, { v3.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v4.16b }, [x1]
 ; CHECK-BE-NEXT:    add x1, x1, #64
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    st1 { v1.4s }, [x10]
-; CHECK-BE-NEXT:    st1 { v0.4s }, [x9]
+; CHECK-BE-NEXT:    st1 { v5.16b }, [x9]
+; CHECK-BE-NEXT:    st1 { v3.16b }, [x10]
 ; CHECK-BE-NEXT:    b.ne  .LBB12_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
 entry:
   br label %loop
 
 loop:
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI16_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_1:
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_2:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_3:
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_4:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_5:
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_6:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI16_7:
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI16_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:  .LCPI16_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI16_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:  .LCPI16_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI16_4:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:  .LCPI16_5:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .LCPI16_6:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:  .LCPI16_7:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+
 define void @zext_v8i8_to_v8i128_in_loop(i8* %src, i128* %dst) {
 ; CHECK-LABEL: _zext_v8i8_to_v8i128_in_loop:
 ; CHECK: ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh44:
+; CHECK-NEXT:   adrp  x9, lCPI16_0@PAGE
+; CHECK-NEXT: Lloh45:
+; CHECK-NEXT:   adrp  x10, lCPI16_1@PAGE
+; CHECK-NEXT: Lloh46:
+; CHECK-NEXT:   adrp  x11, lCPI16_2@PAGE
 ; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh47:
+; CHECK-NEXT:   ldr q0, [x9, lCPI16_0@PAGEOFF]
+; CHECK-NEXT: Lloh48:
+; CHECK-NEXT:   adrp  x9, lCPI16_3@PAGE
+; CHECK-NEXT: Lloh49:
+; CHECK-NEXT:   ldr q1, [x10, lCPI16_1@PAGEOFF]
+; CHECK-NEXT: Lloh50:
+; CHECK-NEXT:   adrp  x10, lCPI16_4@PAGE
+; CHECK-NEXT: Lloh51:
+; CHECK-NEXT:   ldr q2, [x11, lCPI16_2@PAGEOFF]
+; CHECK-NEXT: Lloh52:
+; CHECK-NEXT:   adrp  x11, lCPI16_5@PAGE
+; CHECK-NEXT: Lloh53:
+; CHECK-NEXT:   ldr q3, [x9, lCPI16_3@PAGEOFF]
+; CHECK-NEXT: Lloh54:
+; CHECK-NEXT:   adrp  x9, lCPI16_6@PAGE
+; CHECK-NEXT: Lloh55:
+; CHECK-NEXT:   ldr q4, [x10, lCPI16_4@PAGEOFF]
+; CHECK-NEXT: Lloh56:
+; CHECK-NEXT:   adrp  x10, lCPI16_7@PAGE
+; CHECK-NEXT: Lloh57:
+; CHECK-NEXT:   ldr q5, [x11, lCPI16_5@PAGEOFF]
+; CHECK-NEXT: Lloh58:
+; CHECK-NEXT:   ldr q6, [x9, lCPI16_6@PAGEOFF]
+; CHECK-NEXT: Lloh59:
+; CHECK-NEXT:   ldr q7, [x10, lCPI16_7@PAGEOFF]
 ; CHECK-NEXT: LBB16_1:                                ; %loop
 ; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:   ldr d0, [x0, x8]
-; CHECK-NEXT:   add x9, x1, #112
-; CHECK-NEXT:   add x10, x1, #80
-; CHECK-NEXT:   str xzr, [x1, #120]
-; CHECK-NEXT:   str xzr, [x1, #104]
+; CHECK-NEXT:   ldr d16, [x0, x8]
 ; CHECK-NEXT:   add x8, x8, #16
-; CHECK-NEXT:   str xzr, [x1, #88]
 ; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   str xzr, [x1, #72]
-; CHECK-NEXT:   str xzr, [x1, #56]
-; CHECK-NEXT:   ushll2.4s v1, v0, #0
-; CHECK-NEXT:   str xzr, [x1, #40]
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   str xzr, [x1, #24]
-; CHECK-NEXT:   ushll2.2d v2, v1, #0
-; CHECK-NEXT:   str xzr, [x1, #8]
-; CHECK-NEXT:   ushll.2d  v1, v1, #0
-; CHECK-NEXT:   st1.d { v2 }[1], [x9]
-; CHECK-NEXT:   add x9, x1, #48
-; CHECK-NEXT:   str d2, [x1, #96]
-; CHECK-NEXT:   ushll2.2d v2, v0, #0
-; CHECK-NEXT:   st1.d { v1 }[1], [x10]
-; CHECK-NEXT:   ushll.2d  v0, v0, #0
-; CHECK-NEXT:   str d1, [x1, #64]
-; CHECK-NEXT:   str d2, [x1, #32]
-; CHECK-NEXT:   add x10, x1, #16
-; CHECK-NEXT:   str d0, [x1]
-; CHECK-NEXT:   add x1, x1, #256
-; CHECK-NEXT:   st1.d { v2 }[1], [x9]
-; CHECK-NEXT:   st1.d { v0 }[1], [x10]
+; CHECK-NEXT:   tbl.16b v17, { v16 }, v7
+; CHECK-NEXT:   tbl.16b v18, { v16 }, v6
+; CHECK-NEXT:   tbl.16b v19, { v16 }, v5
+; CHECK-NEXT:   tbl.16b v20, { v16 }, v4
+; CHECK-NEXT:   tbl.16b v21, { v16 }, v3
+; CHECK-NEXT:   stp q18, q17, [x1, #96]
+; CHECK-NEXT:   tbl.16b v17, { v16 }, v2
+; CHECK-NEXT:   tbl.16b v18, { v16 }, v1
+; CHECK-NEXT:   stp q20, q19, [x1, #64]
+; CHECK-NEXT:   tbl.16b v16, { v16 }, v0
+; CHECK-NEXT:   stp q17, q21, [x1, #32]
+; CHECK-NEXT:   stp q16, q18, [x1], #256
 ; CHECK-NEXT:   b.ne  LBB16_1
 ; CHECK-NEXT: ; %bb.2:                                ; %exit
 ; CHECK-NEXT:   ret
+; CHECK-NEXT:   .loh AdrpLdr  Lloh56, Lloh59
+; CHECK-NEXT:   .loh AdrpLdr  Lloh54, Lloh58
+; CHECK-NEXT:   .loh AdrpLdr  Lloh52, Lloh57
+; CHECK-NEXT:   .loh AdrpAdrp Lloh50, Lloh56
+; CHECK-NEXT:   .loh AdrpLdr  Lloh50, Lloh55
+; CHECK-NEXT:   .loh AdrpAdrp Lloh48, Lloh54
+; CHECK-NEXT:   .loh AdrpLdr  Lloh48, Lloh53
+; CHECK-NEXT:   .loh AdrpAdrp Lloh46, Lloh52
+; CHECK-NEXT:   .loh AdrpLdr  Lloh46, Lloh51
+; CHECK-NEXT:   .loh AdrpAdrp Lloh45, Lloh50
+; CHECK-NEXT:   .loh AdrpLdr  Lloh45, Lloh49
+; CHECK-NEXT:   .loh AdrpAdrp Lloh44, Lloh48
+; CHECK-NEXT:   .loh AdrpLdr  Lloh44, Lloh47
 
 ; CHECK-BE-LABEL:  zext_v8i8_to_v8i128_in_loop:
 ; CHECK-BE:  // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_2
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_3
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_4
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_4
+; CHECK-BE-NEXT:    ld1 { v4.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_5
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_5
+; CHECK-BE-NEXT:    ld1 { v5.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_6
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_6
+; CHECK-BE-NEXT:    ld1 { v6.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI16_7
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI16_7
+; CHECK-BE-NEXT:    ld1 { v7.16b }, [x8]
 ; CHECK-BE-NEXT:    mov x8, xzr
 ; CHECK-BE-NEXT:  .LBB16_1:                               // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x10, x1, #88
+; CHECK-BE-NEXT:    add x10, x1, #96
 ; CHECK-BE-NEXT:    add x8, x8, #16
 ; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.8b }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #120
-; CHECK-BE-NEXT:    str xzr, [x1, #112]
-; CHECK-BE-NEXT:    str xzr, [x1, #96]
-; CHECK-BE-NEXT:    str xzr, [x1, #80]
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    str xzr, [x1, #64]
-; CHECK-BE-NEXT:    str xzr, [x1, #48]
-; CHECK-BE-NEXT:    ushll2  v1.4s, v0.8h, #0
-; CHECK-BE-NEXT:    str xzr, [x1, #32]
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    str xzr, [x1, #16]
-; CHECK-BE-NEXT:    ushll2  v2.2d, v1.4s, #0
-; CHECK-BE-NEXT:    str xzr, [x1]
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    st1 { v2.d }[1], [x9]
-; CHECK-BE-NEXT:    add x9, x1, #56
-; CHECK-BE-NEXT:    str d2, [x1, #104]
-; CHECK-BE-NEXT:    ushll2  v2.2d, v0.4s, #0
-; CHECK-BE-NEXT:    st1 { v1.d }[1], [x10]
-; CHECK-BE-NEXT:    ushll v0.2d, v0.2s, #0
-; CHECK-BE-NEXT:    str d1, [x1, #72]
-; CHECK-BE-NEXT:    str d2, [x1, #40]
-; CHECK-BE-NEXT:    add x10, x1, #24
-; CHECK-BE-NEXT:    str d0, [x1, #8]
+; CHECK-BE-NEXT:    ld1 { v16.8b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #112
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v7.16b
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v6.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v5.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #80
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v4.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x1, #64
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #48
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v3.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v2.16b
+; CHECK-BE-NEXT:    add x10, x1, #32
+; CHECK-BE-NEXT:    tbl v16.16b, { v16.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #16
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x1]
 ; CHECK-BE-NEXT:    add x1, x1, #256
-; CHECK-BE-NEXT:    st1 { v2.d }[1], [x9]
-; CHECK-BE-NEXT:    st1 { v0.d }[1], [x10]
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x9]
 ; CHECK-BE-NEXT:    b.ne  .LBB16_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <8 x i8>*
   %load = load <8 x i8>, <8 x i8>* %src.gep.cast
   %ext = zext <8 x i8> %load to <8 x i128>
   %dst.gep = getelementptr i128, i128* %dst, i64 %iv
   %dst.gep.cast = bitcast i128* %dst.gep to <8 x i128>*
   store <8 x i128> %ext, <8 x i128>* %dst.gep.cast
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI17_0:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI17_1:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI17_2:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI17_3:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI17_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI17_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI17_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+; CHECK-BE-NEXT:  .LCPI17_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5
+
+; multiple back-to-back 'zext' of similar type of vectors can lead to 
+; future optimizations that may not be possible with 'tbl' lowering
 define void @zext_v8i8_to_v8i64_with_add_in_sequence_in_loop(i8* %src, i64* %dst) {
 ; CHECK-LABEL:  _zext_v8i8_to_v8i64_with_add_in_sequence_in_loop:
 ; CHECK: ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh60:
+; CHECK-NEXT:   adrp  x9, lCPI17_0@PAGE
+; CHECK-NEXT: Lloh61:
+; CHECK-NEXT:   adrp  x10, lCPI17_1@PAGE
+; CHECK-NEXT: Lloh62:
+; CHECK-NEXT:   adrp  x11, lCPI17_2@PAGE
+; CHECK-NEXT: Lloh63:
+; CHECK-NEXT:   adrp  x12, lCPI17_3@PAGE
 ; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh64:
+; CHECK-NEXT:   ldr q0, [x9, lCPI17_0@PAGEOFF]
 ; CHECK-NEXT:   add x9, x0, #8
+; CHECK-NEXT: Lloh65:
+; CHECK-NEXT:   ldr q1, [x10, lCPI17_1@PAGEOFF]
+; CHECK-NEXT: Lloh66:
+; CHECK-NEXT:   ldr q2, [x11, lCPI17_2@PAGEOFF]
+; CHECK-NEXT: Lloh67:
+; CHECK-NEXT:   ldr q3, [x12, lCPI17_3@PAGEOFF]
 ; CHECK-NEXT: LBB17_1:                                ; %loop
 ; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:   add x10, x1, x8
 ; CHECK-NEXT:   add x8, x8, #128
-; CHECK-NEXT:   ldp d0, d1, [x9, #-8]
+; CHECK-NEXT:   ldp d4, d5, [x9, #-8]
 ; CHECK-NEXT:   add x9, x9, #16
 ; CHECK-NEXT:   cmp x8, #1024
-; CHECK-NEXT:   ldp q3, q2, [x10, #32]
-; CHECK-NEXT:   ushll.8h  v0, v0, #0
-; CHECK-NEXT:   ushll.8h  v1, v1, #0
-; CHECK-NEXT:   ushll2.4s v6, v0, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   ldp q5, q4, [x10]
-; CHECK-NEXT:   uaddw2.2d v2, v2, v6
-; CHECK-NEXT:   uaddw.2d  v3, v3, v6
-; CHECK-NEXT:   ushll2.4s v7, v1, #0
-; CHECK-NEXT:   ushll.4s  v1, v1, #0
-; CHECK-NEXT:   stp q3, q2, [x10, #32]
-; CHECK-NEXT:   ldp q17, q16, [x10, #96]
-; CHECK-NEXT:   uaddw2.2d v4, v4, v0
-; CHECK-NEXT:   uaddw.2d  v0, v5, v0
-; CHECK-NEXT:   uaddw.2d  v3, v17, v7
-; CHECK-NEXT:   stp q0, q4, [x10]
-; CHECK-NEXT:   ldp q6, q18, [x10, #64]
-; CHECK-NEXT:   uaddw2.2d v2, v16, v7
-; CHECK-NEXT:   stp q3, q2, [x10, #96]
-; CHECK-NEXT:   uaddw2.2d v0, v18, v1
-; CHECK-NEXT:   uaddw.2d  v1, v6, v1
-; CHECK-NEXT:   stp q1, q0, [x10, #64]
+; CHECK-NEXT:   ldp q7, q17, [x10, #32]
+; CHECK-NEXT:   tbl.16b v6, { v4 }, v0
+; CHECK-NEXT:   tbl.16b v16, { v4 }, v1
+; CHECK-NEXT:   tbl.16b v18, { v4 }, v2
+; CHECK-NEXT:   tbl.16b v4, { v4 }, v3
+; CHECK-NEXT:   ldp q19, q20, [x10]
+; CHECK-NEXT:   tbl.16b v21, { v5 }, v0
+; CHECK-NEXT:   tbl.16b v23, { v5 }, v1
+; CHECK-NEXT:   tbl.16b v25, { v5 }, v2
+; CHECK-NEXT:   tbl.16b v5, { v5 }, v3
+; CHECK-NEXT:   add.2d  v6, v7, v6
+; CHECK-NEXT:   ldp q22, q24, [x10, #96]
+; CHECK-NEXT:   add.2d  v7, v17, v16
+; CHECK-NEXT:   add.2d  v16, v19, v18
+; CHECK-NEXT:   add.2d  v4, v20, v4
+; CHECK-NEXT:   stp q6, q7, [x10, #32]
+; CHECK-NEXT:   add.2d  v6, v22, v21
+; CHECK-NEXT:   stp q16, q4, [x10]
+; CHECK-NEXT:   ldp q26, q27, [x10, #64]
+; CHECK-NEXT:   add.2d  v7, v24, v23
+; CHECK-NEXT:   stp q6, q7, [x10, #96]
+; CHECK-NEXT:   add.2d  v4, v27, v5
+; CHECK-NEXT:   add.2d  v5, v26, v25
+; CHECK-NEXT:   stp q5, q4, [x10, #64]
 ; CHECK-NEXT:   b.ne  LBB17_1
 ; CHECK-NEXT: ; %bb.2:                                ; %exit
 ; CHECK-NEXT:   ret
+; CHECK-NEXT:   .loh AdrpLdr  Lloh63, Lloh67
+; CHECK-NEXT:   .loh AdrpLdr  Lloh62, Lloh66
+; CHECK-NEXT:   .loh AdrpLdr  Lloh61, Lloh65
+; CHECK-NEXT:   .loh AdrpLdr  Lloh60, Lloh64
 
 ; CHECK-BE-LABEL: zext_v8i8_to_v8i64_with_add_in_sequence_in_loop:
 ; CHECK-BE:  // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x9, .LCPI17_0
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI17_0
 ; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI17_1
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI17_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI17_2
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI17_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI17_3
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI17_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x9]
 ; CHECK-BE-NEXT:    add x9, x0, #8
 ; CHECK-BE-NEXT:  .LBB17_1:                               // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    sub x12, x9, #8
 ; CHECK-BE-NEXT:    add x10, x1, x8
-; CHECK-BE-NEXT:    add x11, x10, #48
-; CHECK-BE-NEXT:    add x13, x10, #32
+; CHECK-BE-NEXT:    add x11, x10, #32
+; CHECK-BE-NEXT:    ld1 { v4.8b }, [x9]
+; CHECK-BE-NEXT:    add x13, x10, #48
 ; CHECK-BE-NEXT:    add x14, x10, #16
-; CHECK-BE-NEXT:    ld1 { v0.8b }, [x9]
-; CHECK-BE-NEXT:    ld1 { v2.8b }, [x12]
-; CHECK-BE-NEXT:    add x12, x10, #112
-; CHECK-BE-NEXT:    ld1 { v1.2d }, [x11]
+; CHECK-BE-NEXT:    ld1 { v6.8b }, [x12]
 ; CHECK-BE-NEXT:    add x15, x10, #96
+; CHECK-BE-NEXT:    ld1 { v5.2d }, [x11]
+; CHECK-BE-NEXT:    add x12, x10, #112
+; CHECK-BE-NEXT:    tbl v16.16b, { v4.16b }, v2.16b
 ; CHECK-BE-NEXT:    add x16, x10, #64
+; CHECK-BE-NEXT:    tbl v17.16b, { v4.16b }, v3.16b
 ; CHECK-BE-NEXT:    add x17, x10, #80
-; CHECK-BE-NEXT:    ld1 { v3.2d }, [x13]
+; CHECK-BE-NEXT:    tbl v19.16b, { v6.16b }, v3.16b
+; CHECK-BE-NEXT:    ld1 { v7.2d }, [x13]
+; CHECK-BE-NEXT:    tbl v20.16b, { v6.16b }, v2.16b
 ; CHECK-BE-NEXT:    add x8, x8, #128
-; CHECK-BE-NEXT:    ushll v2.8h, v2.8b, #0
+; CHECK-BE-NEXT:    tbl v22.16b, { v6.16b }, v1.16b
+; CHECK-BE-NEXT:    ld1 { v21.2d }, [x14]
+; CHECK-BE-NEXT:    tbl v6.16b, { v6.16b }, v0.16b
 ; CHECK-BE-NEXT:    add x9, x9, #16
-; CHECK-BE-NEXT:    ld1 { v5.2d }, [x10]
+; CHECK-BE-NEXT:    rev64 v19.16b, v19.16b
+; CHECK-BE-NEXT:    ld1 { v18.2d }, [x10]
+; CHECK-BE-NEXT:    tbl v24.16b, { v4.16b }, v1.16b
 ; CHECK-BE-NEXT:    cmp x8, #1024
-; CHECK-BE-NEXT:    ushll2  v7.4s, v2.8h, #0
-; CHECK-BE-NEXT:    ld1 { v6.2d }, [x14]
-; CHECK-BE-NEXT:    uaddw2  v1.2d, v1.2d, v7.4s
-; CHECK-BE-NEXT:    ushll v2.4s, v2.4h, #0
-; CHECK-BE-NEXT:    ld1 { v16.2d }, [x12]
-; CHECK-BE-NEXT:    uaddw v3.2d, v3.2d, v7.2s
-; CHECK-BE-NEXT:    ld1 { v17.2d }, [x15]
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x11]
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ld1 { v7.2d }, [x17]
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x13]
-; CHECK-BE-NEXT:    uaddw v5.2d, v5.2d, v2.2s
-; CHECK-BE-NEXT:    ld1 { v1.2d }, [x16]
-; CHECK-BE-NEXT:    uaddw2  v2.2d, v6.2d, v2.4s
-; CHECK-BE-NEXT:    ushll2  v4.4s, v0.8h, #0
-; CHECK-BE-NEXT:    st1 { v5.2d }, [x10]
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    uaddw2  v6.2d, v16.2d, v4.4s
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x14]
-; CHECK-BE-NEXT:    uaddw v3.2d, v17.2d, v4.2s
-; CHECK-BE-NEXT:    uaddw2  v2.2d, v7.2d, v0.4s
-; CHECK-BE-NEXT:    uaddw v0.2d, v1.2d, v0.2s
-; CHECK-BE-NEXT:    st1 { v6.2d }, [x12]
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x15]
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x17]
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x16]
+; CHECK-BE-NEXT:    tbl v4.16b, { v4.16b }, v0.16b
+; CHECK-BE-NEXT:    ld1 { v23.2d }, [x15]
+; CHECK-BE-NEXT:    rev64 v20.16b, v20.16b
+; CHECK-BE-NEXT:    rev64 v6.16b, v6.16b
+; CHECK-BE-NEXT:    ld1 { v25.2d }, [x12]
+; CHECK-BE-NEXT:    rev64 v22.16b, v22.16b
+; CHECK-BE-NEXT:    ld1 { v26.2d }, [x16]
+; CHECK-BE-NEXT:    rev64 v17.16b, v17.16b
+; CHECK-BE-NEXT:    ld1 { v27.2d }, [x17]
+; CHECK-BE-NEXT:    rev64 v16.16b, v16.16b
+; CHECK-BE-NEXT:    rev64 v24.16b, v24.16b
+; CHECK-BE-NEXT:    rev64 v4.16b, v4.16b
+; CHECK-BE-NEXT:    add v5.2d, v5.2d, v19.2d
+; CHECK-BE-NEXT:    add v7.2d, v7.2d, v20.2d
+; CHECK-BE-NEXT:    st1 { v5.2d }, [x11]
+; CHECK-BE-NEXT:    add v5.2d, v21.2d, v6.2d
+; CHECK-BE-NEXT:    st1 { v7.2d }, [x13]
+; CHECK-BE-NEXT:    add v18.2d, v18.2d, v22.2d
+; CHECK-BE-NEXT:    add v6.2d, v23.2d, v17.2d
+; CHECK-BE-NEXT:    st1 { v5.2d }, [x14]
+; CHECK-BE-NEXT:    add v7.2d, v25.2d, v16.2d
+; CHECK-BE-NEXT:    st1 { v18.2d }, [x10]
+; CHECK-BE-NEXT:    add v4.2d, v27.2d, v4.2d
+; CHECK-BE-NEXT:    st1 { v6.2d }, [x15]
+; CHECK-BE-NEXT:    add v5.2d, v26.2d, v24.2d
+; CHECK-BE-NEXT:    st1 { v7.2d }, [x12]
+; CHECK-BE-NEXT:    st1 { v4.2d }, [x17]
+; CHECK-BE-NEXT:    st1 { v5.2d }, [x16]
 ; CHECK-BE-NEXT:    b.ne  .LBB17_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
-
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <8 x i8>*
   %load = load <8 x i8>, <8 x i8>* %src.gep.cast
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI18_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_1:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_2:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_3:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_4:
+; CHECK-NEXT:   .byte 8                               ; 0x8
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 9                               ; 0x9
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_5:
+; CHECK-NEXT:   .byte 10                              ; 0xa
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 11                              ; 0xb
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_6:
+; CHECK-NEXT:   .byte 12                              ; 0xc
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 13                              ; 0xd
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI18_7:
+; CHECK-NEXT:   .byte 14                              ; 0xe
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 15                              ; 0xf
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI18_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI18_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI18_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .LCPI18_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+; CHECK-BE-NEXT:  .LCPI18_4:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 8                               // 0x8
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 9                               // 0x9
+; CHECK-BE-NEXT:  .LCPI18_5:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 10                              // 0xa
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 11                              // 0xb
+; CHECK-BE-NEXT:  .LCPI18_6:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 12                              // 0xc
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 13                              // 0xd
+; CHECK-BE-NEXT:  .LCPI18_7:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 14                              // 0xe
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 15                              // 0xf
+
+; multiple back-to-back 'zext' of similar type of vectors
 define void @zext_v16i8_to_v16i64_in_sequence_in_loop(i8* %src, i64* %dst) {
 ; CHECK-LABEL: _zext_v16i8_to_v16i64_in_sequence_in_loop:
 ; CHECK: ; %bb.0:                                ; %entry
+; CHECK-NEXT: Lloh68:
+; CHECK-NEXT:   adrp  x9, lCPI18_0@PAGE
+; CHECK-NEXT: Lloh69:
+; CHECK-NEXT:   adrp  x10, lCPI18_1@PAGE
+; CHECK-NEXT: Lloh70:
+; CHECK-NEXT:   adrp  x11, lCPI18_2@PAGE
 ; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT: Lloh71:
+; CHECK-NEXT:   ldr q0, [x9, lCPI18_0@PAGEOFF]
+; CHECK-NEXT: Lloh72:
+; CHECK-NEXT:   adrp  x9, lCPI18_3@PAGE
+; CHECK-NEXT: Lloh73:
+; CHECK-NEXT:   ldr q1, [x10, lCPI18_1@PAGEOFF]
+; CHECK-NEXT: Lloh74:
+; CHECK-NEXT:   adrp  x10, lCPI18_4@PAGE
+; CHECK-NEXT: Lloh75:
+; CHECK-NEXT:   ldr q2, [x11, lCPI18_2@PAGEOFF]
+; CHECK-NEXT: Lloh76:
+; CHECK-NEXT:   adrp  x11, lCPI18_5@PAGE
+; CHECK-NEXT: Lloh77:
+; CHECK-NEXT:   ldr q3, [x9, lCPI18_3@PAGEOFF]
+; CHECK-NEXT: Lloh78:
+; CHECK-NEXT:   adrp  x9, lCPI18_6@PAGE
+; CHECK-NEXT: Lloh79:
+; CHECK-NEXT:   ldr q4, [x10, lCPI18_4@PAGEOFF]
+; CHECK-NEXT: Lloh80:
+; CHECK-NEXT:   adrp  x10, lCPI18_7@PAGE
+; CHECK-NEXT: Lloh81:
+; CHECK-NEXT:   ldr q5, [x11, lCPI18_5@PAGEOFF]
+; CHECK-NEXT: Lloh82:
+; CHECK-NEXT:   ldr q6, [x9, lCPI18_6@PAGEOFF]
 ; CHECK-NEXT:   add x9, x1, #128
+; CHECK-NEXT: Lloh83:
+; CHECK-NEXT:   ldr q7, [x10, lCPI18_7@PAGEOFF]
 ; CHECK-NEXT: LBB18_1:                                ; %loop
 ; CHECK-NEXT:                                         ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:   add x10, x0, x8
 ; CHECK-NEXT:   add x8, x8, #16
 ; CHECK-NEXT:   cmp x8, #128
-; CHECK-NEXT:   ldp q0, q1, [x10]
-; CHECK-NEXT:   ushll.8h  v2, v0, #0
-; CHECK-NEXT:   ushll2.8h v0, v0, #0
-; CHECK-NEXT:   ushll.4s  v4, v2, #0
-; CHECK-NEXT:   ushll2.4s v5, v0, #0
-; CHECK-NEXT:   ushll2.4s v2, v2, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   ushll2.2d v6, v5, #0
-; CHECK-NEXT:   ushll.2d  v5, v5, #0
-; CHECK-NEXT:   ushll2.8h v3, v1, #0
-; CHECK-NEXT:   ushll2.2d v7, v0, #0
-; CHECK-NEXT:   stp q5, q6, [x9, #-32]
-; CHECK-NEXT:   ushll.2d  v0, v0, #0
-; CHECK-NEXT:   ushll2.2d v5, v2, #0
-; CHECK-NEXT:   ushll.2d  v2, v2, #0
-; CHECK-NEXT:   stp q0, q7, [x9, #-64]
-; CHECK-NEXT:   ushll2.2d v0, v4, #0
-; CHECK-NEXT:   stp q2, q5, [x9, #-96]
-; CHECK-NEXT:   ushll2.4s v5, v3, #0
-; CHECK-NEXT:   ushll.2d  v2, v4, #0
-; CHECK-NEXT:   ushll2.2d v4, v5, #0
-; CHECK-NEXT:   stp q2, q0, [x9, #-128]
-; CHECK-NEXT:   ushll.2d  v0, v5, #0
-; CHECK-NEXT:   ushll.4s  v2, v3, #0
-; CHECK-NEXT:   stp q0, q4, [x9, #96]
-; CHECK-NEXT:   ushll.8h  v0, v1, #0
-; CHECK-NEXT:   ushll2.2d v1, v2, #0
-; CHECK-NEXT:   ushll.2d  v2, v2, #0
-; CHECK-NEXT:   ushll2.4s v3, v0, #0
-; CHECK-NEXT:   stp q2, q1, [x9, #64]
-; CHECK-NEXT:   ushll2.2d v1, v3, #0
-; CHECK-NEXT:   ushll.2d  v2, v3, #0
-; CHECK-NEXT:   ushll.4s  v0, v0, #0
-; CHECK-NEXT:   stp q2, q1, [x9, #32]
-; CHECK-NEXT:   ushll2.2d v1, v0, #0
-; CHECK-NEXT:   ushll.2d  v0, v0, #0
-; CHECK-NEXT:   stp q0, q1, [x9], #128
+; CHECK-NEXT:   ldp q16, q17, [x10]
+; CHECK-NEXT:   tbl.16b v18, { v16 }, v7
+; CHECK-NEXT:   tbl.16b v19, { v16 }, v6
+; CHECK-NEXT:   tbl.16b v20, { v16 }, v5
+; CHECK-NEXT:   tbl.16b v21, { v16 }, v4
+; CHECK-NEXT:   tbl.16b v22, { v16 }, v3
+; CHECK-NEXT:   stp q19, q18, [x9, #-32]
+; CHECK-NEXT:   tbl.16b v18, { v16 }, v2
+; CHECK-NEXT:   tbl.16b v19, { v16 }, v1
+; CHECK-NEXT:   stp q21, q20, [x9, #-64]
+; CHECK-NEXT:   tbl.16b v16, { v16 }, v0
+; CHECK-NEXT:   tbl.16b v20, { v17 }, v7
+; CHECK-NEXT:   tbl.16b v21, { v17 }, v6
+; CHECK-NEXT:   stp q18, q22, [x9, #-96]
+; CHECK-NEXT:   tbl.16b v18, { v17 }, v5
+; CHECK-NEXT:   stp q16, q19, [x9, #-128]
+; CHECK-NEXT:   tbl.16b v19, { v17 }, v4
+; CHECK-NEXT:   tbl.16b v16, { v17 }, v3
+; CHECK-NEXT:   stp q21, q20, [x9, #96]
+; CHECK-NEXT:   tbl.16b v20, { v17 }, v2
+; CHECK-NEXT:   tbl.16b v21, { v17 }, v1
+; CHECK-NEXT:   tbl.16b v17, { v17 }, v0
+; CHECK-NEXT:   stp q19, q18, [x9, #64]
+; CHECK-NEXT:   stp q20, q16, [x9, #32]
+; CHECK-NEXT:   stp q17, q21, [x9], #128
 ; CHECK-NEXT:   b.ne  LBB18_1
 ; CHECK-NEXT: ; %bb.2:                                ; %exit
 ; CHECK-NEXT:   ret
+; CHECK-NEXT:   .loh AdrpLdr  Lloh80, Lloh83
+; CHECK-NEXT:   .loh AdrpLdr  Lloh78, Lloh82
+; CHECK-NEXT:   .loh AdrpLdr  Lloh76, Lloh81
+; CHECK-NEXT:   .loh AdrpAdrp Lloh74, Lloh80
+; CHECK-NEXT:   .loh AdrpLdr  Lloh74, Lloh79
+; CHECK-NEXT:   .loh AdrpAdrp Lloh72, Lloh78
+; CHECK-NEXT:   .loh AdrpLdr  Lloh72, Lloh77
+; CHECK-NEXT:   .loh AdrpAdrp Lloh70, Lloh76
+; CHECK-NEXT:   .loh AdrpLdr  Lloh70, Lloh75
+; CHECK-NEXT:   .loh AdrpAdrp Lloh69, Lloh74
+; CHECK-NEXT:   .loh AdrpLdr  Lloh69, Lloh73
+; CHECK-NEXT:   .loh AdrpAdrp Lloh68, Lloh72
+; CHECK-NEXT:   .loh AdrpLdr  Lloh68, Lloh71
 
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i64_in_sequence_in_loop:
 ; CHECK-BE:  // %bb.0:                               // %entry
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_0
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_0
 ; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_1
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_2
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_3
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_4
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_4
+; CHECK-BE-NEXT:    ld1 { v4.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_5
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_5
+; CHECK-BE-NEXT:    ld1 { v5.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_6
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_6
+; CHECK-BE-NEXT:    ld1 { v6.16b }, [x9]
+; CHECK-BE-NEXT:    adrp  x9, .LCPI18_7
+; CHECK-BE-NEXT:    add x9, x9, :lo12:.LCPI18_7
+; CHECK-BE-NEXT:    ld1 { v7.16b }, [x9]
 ; CHECK-BE-NEXT:    add x9, x1, #128
 ; CHECK-BE-NEXT:  .LBB18_1:                               // %loop
 ; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
 ; CHECK-BE-NEXT:    add x10, x0, x8
+; CHECK-BE-NEXT:    sub x11, x9, #16
+; CHECK-BE-NEXT:    sub x12, x9, #32
 ; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    add x11, x10, #16
 ; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x10]
-; CHECK-BE-NEXT:    sub x10, x9, #16
-; CHECK-BE-NEXT:    ld1 { v3.16b }, [x11]
-; CHECK-BE-NEXT:    sub x11, x9, #32
-; CHECK-BE-NEXT:    ushll2  v1.8h, v0.16b, #0
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2  v2.4s, v1.8h, #0
-; CHECK-BE-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-BE-NEXT:    ushll2  v4.2d, v2.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    st1 { v4.2d }, [x10]
+; CHECK-BE-NEXT:    ld1 { v16.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x10, #16
+; CHECK-BE-NEXT:    ld1 { v17.16b }, [x10]
 ; CHECK-BE-NEXT:    sub x10, x9, #48
-; CHECK-BE-NEXT:    ushll2  v4.2d, v1.4s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x11]
-; CHECK-BE-NEXT:    ushll2  v2.4s, v0.8h, #0
-; CHECK-BE-NEXT:    sub x11, x9, #80
-; CHECK-BE-NEXT:    st1 { v4.2d }, [x10]
-; CHECK-BE-NEXT:    sub x10, x9, #64
-; CHECK-BE-NEXT:    ushll2  v5.2d, v2.4s, #0
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    st1 { v5.2d }, [x11]
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v7.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v6.16b
+; CHECK-BE-NEXT:    tbl v20.16b, { v16.16b }, v5.16b
+; CHECK-BE-NEXT:    tbl v21.16b, { v16.16b }, v4.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x11]
+; CHECK-BE-NEXT:    sub x11, x9, #64
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x12]
+; CHECK-BE-NEXT:    st1 { v20.16b }, [x10]
+; CHECK-BE-NEXT:    sub x10, x9, #80
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v21.16b }, [x11]
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v1.16b
 ; CHECK-BE-NEXT:    sub x11, x9, #96
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x10]
-; CHECK-BE-NEXT:    ushll v4.8h, v3.8b, #0
-; CHECK-BE-NEXT:    sub x10, x9, #112
-; CHECK-BE-NEXT:    ushll2  v3.8h, v3.16b, #0
-; CHECK-BE-NEXT:    ushll v5.2d, v0.2s, #0
-; CHECK-BE-NEXT:    ushll2  v0.2d, v0.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    ushll v1.4s, v4.4h, #0
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x10]
-; CHECK-BE-NEXT:    ushll2  v6.4s, v3.8h, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x11]
+; CHECK-BE-NEXT:    tbl v20.16b, { v16.16b }, v2.16b
+; CHECK-BE-NEXT:    tbl v16.16b, { v16.16b }, v3.16b
+; CHECK-BE-NEXT:    tbl v23.16b, { v17.16b }, v5.16b
+; CHECK-BE-NEXT:    tbl v21.16b, { v17.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v20.16b }, [x11]
 ; CHECK-BE-NEXT:    sub x11, x9, #128
-; CHECK-BE-NEXT:    add x10, x9, #112
-; CHECK-BE-NEXT:    ushll v2.2d, v1.2s, #0
-; CHECK-BE-NEXT:    ushll2  v0.2d, v1.4s, #0
-; CHECK-BE-NEXT:    ushll2  v1.2d, v6.4s, #0
-; CHECK-BE-NEXT:    st1 { v5.2d }, [x11]
-; CHECK-BE-NEXT:    ushll v3.4s, v3.4h, #0
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x10]
+; CHECK-BE-NEXT:    sub x10, x9, #112
+; CHECK-BE-NEXT:    tbl v16.16b, { v17.16b }, v7.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x11]
 ; CHECK-BE-NEXT:    add x11, x9, #96
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x10]
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x9, #112
+; CHECK-BE-NEXT:    tbl v19.16b, { v17.16b }, v6.16b
+; CHECK-BE-NEXT:    st1 { v21.16b }, [x9]
+; CHECK-BE-NEXT:    tbl v22.16b, { v17.16b }, v3.16b
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x10]
 ; CHECK-BE-NEXT:    add x10, x9, #80
-; CHECK-BE-NEXT:    ushll v5.2d, v6.2s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x9]
-; CHECK-BE-NEXT:    ushll2  v1.4s, v4.8h, #0
-; CHECK-BE-NEXT:    ushll2  v4.2d, v3.4s, #0
-; CHECK-BE-NEXT:    st1 { v5.2d }, [x11]
-; CHECK-BE-NEXT:    add x11, x9, #48
-; CHECK-BE-NEXT:    st1 { v4.2d }, [x10]
-; CHECK-BE-NEXT:    add x10, x9, #64
-; CHECK-BE-NEXT:    ushll2  v5.2d, v1.4s, #0
-; CHECK-BE-NEXT:    ushll v3.2d, v3.2s, #0
-; CHECK-BE-NEXT:    st1 { v5.2d }, [x11]
-; CHECK-BE-NEXT:    add x11, x9, #16
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x10]
+; CHECK-BE-NEXT:    tbl v16.16b, { v17.16b }, v4.16b
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x11]
+; CHECK-BE-NEXT:    add x11, x9, #64
+; CHECK-BE-NEXT:    st1 { v23.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x9, #48
+; CHECK-BE-NEXT:    tbl v18.16b, { v17.16b }, v1.16b
+; CHECK-BE-NEXT:    tbl v17.16b, { v17.16b }, v2.16b
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x11]
+; CHECK-BE-NEXT:    st1 { v22.16b }, [x10]
 ; CHECK-BE-NEXT:    add x10, x9, #32
+; CHECK-BE-NEXT:    add x11, x9, #16
 ; CHECK-BE-NEXT:    add x9, x9, #128
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x11]
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x10]
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x11]
 ; CHECK-BE-NEXT:    b.ne  .LBB18_1
 ; CHECK-BE-NEXT:  // %bb.2:                               // %exit
 ; CHECK-BE-NEXT:    ret
 
-
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <16 x i8>*
   %load = load <16 x i8>, <16 x i8>* %src.gep.cast

llvm/test/Transforms/CodeGenPrepare/AArch64/zext-to-shuffle.ll

 define void @zext_v16i8_to_v16i64_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: @zext_v16i8_to_v16i64_in_loop(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[SRC_GEP:%.*]] = getelementptr i8, ptr [[SRC:%.*]], i64 [[IV]]
 ; CHECK-NEXT:    [[LOAD:%.*]] = load <16 x i8>, ptr [[SRC_GEP]], align 16
-; CHECK-NEXT:    [[EXT:%.*]] = zext <16 x i8> [[LOAD]] to <16 x i64>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <16 x i8> %load, <16 x i8> <i8 0, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison>, <128 x i32> <i32 0, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 1, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 2, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 3, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 4, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 5, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 6, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 7, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 8, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 9, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 10, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 11, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 12, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 13, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 14, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 15, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <128 x i8> [[TMP0]] to <16 x i64>
 ; CHECK-NEXT:    [[DST_GEP:%.*]] = getelementptr i64, ptr [[DST:%.*]], i64 [[IV]]
-; CHECK-NEXT:    store <16 x i64> [[EXT]], ptr [[DST_GEP]], align 128
+; CHECK-NEXT:    store <16 x i64> [[TMP1:%.*]], ptr [[DST_GEP]], align 128
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw i64 [[IV]], 16
 ; CHECK-NEXT:    [[EC:%.*]] = icmp eq i64 [[IV_NEXT]], 128
 ; CHECK-NEXT:    br i1 [[EC]], label [[EXIT:%.*]], label [[LOOP]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br label %loop