[AArch64] Extending lowering of 'zext <Y x i8> %x to <Y x i8X>' to use tbl instructions

[nilanjana_basu]

Summary

Adding support for ZExt lowering for destination types beyond the existing support for (8|16) x i32

[AArch64] Patch for lowering zext instructions to 'tbl' for (8|16)xi8 -> (8|16)xi32 conversions in D120571 is extended to support zext to 'tbl' lowering for Y x i8 to Y x i8X. Any arbitrary number of vector elements & any destination element type whose size is a multiple of 8, greater than 16 and less than 64, is allowed for this transformation.

Related microbenchmarks are in D136274 & D138059

Depends on D120571

[nilanjana_basu]

Added more test cases for zext lowering of different vector types

[nilanjana_basu]

Allowed the zext to tbl lowering for all lengths of vectors

[nilanjana_basu]

Added Big-Endian checks for the test cases that I missed earlier

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

Allowed all element sizes in the destination element that is a multiple of 8

[dmgreen]

I know it's not your problem, but the code in optimizeExtendOrTruncateConversion doesn't feel like it is in the best place, to be honest. CGP has always described itself as a hack, but we shouldn't be hacking things that much. There will be some obvious cases where the extend/trunc can be optimized but the tbl blocks it.
As far as I understand, the code is only in CGP because it is trying limit the transforms to loops. I'm wondering if it would be better to add some sort of flag into ISel so that combines could tell that the current block is a loop, and behave differently because of it.

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444

I think this is worse, I'm afraid. We only want to use tbl if it would replace two instructions (it performs two truncate/zext steps). Otherwise we are just adding instructions to the loop header (and using more registers) for no gain.

[nilanjana_basu]

Minor update to comment

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13708

This conversion shows a regression in performance for some cases where there are multiple similar zext instructions present back to back. The generated code with the previous implementation could be folded into a more optimized set of instructions, which is not possible with 'tbl' instructions. One example is 16xi8->16xi64, where I find an increase in the number of instructions after being lowered to tbl on using a loop interleave count of 4, i.e. with 4 back to back zext instructions.

Is it better to rule out this case in this 'if' block or should we not allow tbl-lowering when there are multiple zext instructions of the same type present back to back?

[fhahn]

In D136722#3917024, @dmgreen wrote:

I know it's not your problem, but the code in optimizeExtendOrTruncateConversion doesn't feel like it is in the best place, to be honest. CGP has always described itself as a hack, but we shouldn't be hacking things that much. There will be some obvious cases where the extend/trunc can be optimized but the tbl blocks it.
As far as I understand, the code is only in CGP because it is trying limit the transforms to loops. I'm wondering if it would be better to add some sort of flag into ISel so that combines could tell that the current block is a loop, and behave differently because of it.

Yep the only reason for doing it in CGP is to work around SelDAG's limitation.I am not sure about extending SelDAG for this, as we are planning to transition to GIsel at least on Darwin platforms very soon, which won't require doing this in CGP. I think @nilanjana_basu will also look into implementing this in GIsel

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13708

This conversion shows a regression in performance for some cases where there are multiple similar zext instructions present back to back. The generated code with the previous implementation could be folded into a more optimized set of instructions, which is not possible with 'tbl' instructions. One example is 16xi8->16xi64, where I find an increase in the number of instructions after being lowered to tbl on using a loop interleave count of 4, i.e. with 4 back to back zext instructions.

is this covered by one of the unit tests? zext_v16i8_to_v16i64_in_loop looks fine to me, at least for little endian

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444

yep it looks like we should have a check for that. @nilanjana_basu could you update the patch and make sure this is also tested in zext-to-tbl.ll

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1483

could you ad those new tests separately?

[nilanjana_basu]

Removed cases where TBL lowering will not be beneficial

[nilanjana_basu]

Rebasing & merging on a recent commit

[nilanjana_basu]

Rebased on latest updated zext unit tests

[nilanjana_basu]

Trying to fix patching error because of rebasing

[nilanjana_basu]

Trying to fix patching error again

[nilanjana_basu]

Fixed rebasing error of duplicated tests

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13708

I was mistaken in checking the instruction count earlier, but I have still added a unit test in zext_v16i8_to_v16i64_in_sequence_in_loop, since I see a performance regression in my local setup.

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

444

In the latest patch, I've blocked zext lowering to tbl for destination vectors with i16 element type, since those were the ones that don't benefit from it. For other vector types, a single zext/truncate too improves on the instruction count.

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1483

I added two new tests for 2 back-to-back zext instructions - zext_v8i8_to_v8i64_with_add_in_sequence_in_loop & zext_v16i8_to_v16i64_in_sequence_in_loop. The pre-patch codegen has been updated in the parent revision D137993.

There seems to be a slight increase in instruction count for zext_v8i8_to_v8i64_with_add_in_sequence_in_loop.

[nilanjana_basu]

Blocked tbl-conversion for destination element size above 64 since only 2 or less destination vector elements can be chosen with each tbl instruction in these cases, making it less beneficial

[nilanjana_basu]

Ran clang-format

[nilanjana_basu]

Removed tbl-conversion cases to destination vector element width above 64, due to observed performance regressions. Will move this to a later patch, once we find a fix.

[fhahn]

Thanks for the latest update! This looks good in general to me, with just one more inline comment about an edge case

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13609

I *think* we could have something like zext i8 to i33 and then the division will drop the remainder. Could you add a test to see if that's the case and make sure we don't perform an incorrect transformation? It should be fine to just ignore cases where there would be remainder.

[nilanjana_basu]

Re-based on newly added tests

[nilanjana_basu]

Trying to fix patching error

[nilanjana_basu]

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

13609

I have also added testing for the Global-ISel path.

[nilanjana_basu]

Added an assert for an extra check

[fhahn]

LGTM, thanks!

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

bool AArch64TargetLowering::shouldSinkOperands(
   }
   return false;
 }
 
 static void createTblShuffleForZExt(ZExtInst *ZExt, bool IsLittleEndian) {
   Value *Op = ZExt->getOperand(0);
   auto *SrcTy = dyn_cast<FixedVectorType>(Op->getType());
   auto *DstTy = dyn_cast<FixedVectorType>(ZExt->getType());
+  unsigned ZExtFactor =
+      (dyn_cast<IntegerType>(DstTy->getElementType())->getBitWidth()) /
+      (dyn_cast<IntegerType>(SrcTy->getElementType())->getBitWidth());
   unsigned NumElts = SrcTy->getNumElements();
   IRBuilder<> Builder(ZExt);
-  SmallVector<int> Mask(4 * NumElts, NumElts);
+  SmallVector<int> Mask;
   // Create a mask that selects <0,0,0,Op[i]> for each lane of vector of i32 to
   // replace the original ZExt. This can later be lowered to a set of tbl
   // instructions.
-  for (unsigned i = 0; i < NumElts; i++) {
-    if (IsLittleEndian)
-      Mask[i * 4] = i;
+  for (unsigned i = 0; i < NumElts * ZExtFactor; i++) {
+    if (IsLittleEndian) {
+      if (i % ZExtFactor == 0)
+        Mask.push_back(i / ZExtFactor);
       else
-      Mask[i * 4 + 3] = i;
+        Mask.push_back(NumElts);
+    } else {
+      if ((i + 1) % ZExtFactor == 0)
+        Mask.push_back((i - ZExtFactor + 1) / ZExtFactor);
+      else
+        Mask.push_back(NumElts);
+    }
   }
 
   auto *FirstEltZero = Builder.CreateInsertElement(
       PoisonValue::get(SrcTy), Builder.getInt8(0), uint64_t(0));
   Value *Result = Builder.CreateShuffleVector(Op, FirstEltZero, Mask);
   Result = Builder.CreateBitCast(Result, DstTy);
   ZExt->replaceAllUsesWith(Result);
   ZExt->eraseFromParent();
if (!SrcTy || !DstTy)
     return false;
 
   // Convert 'zext <(8|16) x i8> %x to <(8|16) x i32>' to a shuffle that can be
   // lowered to either 2 or 4 tbl instructions to insert the original i8
   // elements into i32 lanes.
   auto *ZExt = dyn_cast<ZExtInst>(I);
   if (ZExt && (SrcTy->getNumElements() == 8 || SrcTy->getNumElements() == 16) &&
       SrcTy->getElementType()->isIntegerTy(8) &&
-      DstTy->getElementType()->isIntegerTy(32)) {
+      (DstTy->getElementType()->isIntegerTy(16) ||
+       DstTy->getElementType()->isIntegerTy(32) ||
+       DstTy->getElementType()->isIntegerTy(64))) {
     createTblShuffleForZExt(ZExt, Subtarget->isLittleEndian());
     return true;
   }
 
   auto *UIToFP = dyn_cast<UIToFPInst>(I);
   if (UIToFP &&
       (SrcTy->getNumElements() == 8 || SrcTy->getNumElements() == 16) &&
       SrcTy->getElementType()->isIntegerTy(8) &&
       DstTy->getElementType()->isFloatTy()) {
     IRBuilder<> Builder(I);
     auto *ZExt = cast<ZExtInst>(
         Builder.CreateZExt(I->getOperand(0), VectorType::getInteger(DstTy)));
     auto *UI = Builder.CreateUIToFP(ZExt, DstTy);
     I->replaceAllUsesWith(UI);
     I->eraseFromParent();
     createTblShuffleForZExt(ZExt, Subtarget->isLittleEndian());

llvm/test/CodeGen/AArch64/aarch64-matrix-umull-smull.ll

 ; CHECK-NEXT:  // %bb.2:
 ; CHECK-NEXT:    mov x11, xzr
 ; CHECK-NEXT:    mov w8, wzr
 ; CHECK-NEXT:    b .LBB5_7
 ; CHECK-NEXT:  .LBB5_3:
 ; CHECK-NEXT:    mov w0, wzr
 ; CHECK-NEXT:    ret
 ; CHECK-NEXT:  .LBB5_4: // %vector.ph
-; CHECK-NEXT:    dup v2.8b, w9
-; CHECK-NEXT:    and x11, x10, #0xfffffff0
-; CHECK-NEXT:    movi v0.2d, #0000000000000000
-; CHECK-NEXT:    add x8, x0, #8
-; CHECK-NEXT:    movi v1.2d, #0000000000000000
-; CHECK-NEXT:    mov x12, x11
-; CHECK-NEXT:    sshll v2.8h, v2.8b, #0
-; CHECK-NEXT:  .LBB5_5: // %vector.body
-; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldp d3, d4, [x8, #-8]
-; CHECK-NEXT:    subs x12, x12, #16
-; CHECK-NEXT:    add x8, x8, #16
-; CHECK-NEXT:    ushll v3.8h, v3.8b, #0
-; CHECK-NEXT:    ushll v4.8h, v4.8b, #0
-; CHECK-NEXT:    mla v0.8h, v2.8h, v3.8h
-; CHECK-NEXT:    mla v1.8h, v2.8h, v4.8h
-; CHECK-NEXT:    b.ne .LBB5_5
+; CHECK-NEXT:   adrp  x12, .LCPI5_0
+; CHECK-NEXT:   dup v3.8b, w9
+; CHECK-NEXT:   and x11, x10, #0xfffffff0
+; CHECK-NEXT:   add x8, x0, #8
+; CHECK-NEXT:   movi  v0.2d, #0000000000000000
+; CHECK-NEXT:   movi  v1.2d, #0000000000000000
+; CHECK-NEXT:   ldr q2, [x12, :lo12:.LCPI5_0]
+; CHECK-NEXT:   mov x12, x11
+; CHECK-NEXT:   sshll v3.8h, v3.8b, #0
+; CHECK-NEXT: .LBB5_5:                                // %vector.body
+; CHECK-NEXT:                                         // =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:   ldp d4, d5, [x8, #-8]
+; CHECK-NEXT:   subs  x12, x12, #16
+; CHECK-NEXT:   add x8, x8, #16
+; CHECK-NEXT:   tbl v4.16b, { v4.16b }, v2.16b
+; CHECK-NEXT:   tbl v5.16b, { v5.16b }, v2.16b
+; CHECK-NEXT:   mla v0.8h, v3.8h, v4.8h
+; CHECK-NEXT:   mla v1.8h, v3.8h, v5.8h
+; CHECK-NEXT:   b.ne  .LBB5_5
 ; CHECK-NEXT:  // %bb.6: // %middle.block
 ; CHECK-NEXT:    add v0.8h, v1.8h, v0.8h
 ; CHECK-NEXT:    cmp x11, x10
 ; CHECK-NEXT:    addv h0, v0.8h
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    b.eq .LBB5_9
 ; CHECK-NEXT:  .LBB5_7: // %for.body.preheader1
 ; CHECK-NEXT:    sub x10, x10, x11

 
 exit:
   ret void
 }
 
 define void @sink_v8z16_0(i32 *%p, i32 *%d, i64 %n, <16 x i8> %a) {
 ; CHECK-LABEL: sink_v8z16_0:
 ; CHECK:       // %bb.0: // %entry
-; CHECK-NEXT:    dup v0.8b, v0.b[0]
-; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  .LBB8_1: // %loop
-; CHECK-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr d1, [x0]
-; CHECK-NEXT:    add x8, x8, #8
-; CHECK-NEXT:    subs x2, x2, #8
-; CHECK-NEXT:    umull v1.8h, v1.8b, v0.8b
-; CHECK-NEXT:    cmlt v1.8h, v1.8h, #0
-; CHECK-NEXT:    xtn v1.8b, v1.8h
-; CHECK-NEXT:    str d1, [x0], #32
-; CHECK-NEXT:    b.ne .LBB8_1
-; CHECK-NEXT:  // %bb.2: // %exit
-; CHECK-NEXT:    ret
+;CHECK-NEXT:    adrp  x9, .LCPI8_0
+;CHECK-NEXT:    dup v1.8b, v0.b[0]
+;CHECK-NEXT:    mov x8, xzr
+;CHECK-NEXT:    ldr q0, [x9, :lo12:.LCPI8_0]
+;CHECK-NEXT:    ushll v1.8h, v1.8b, #0
+;CHECK-NEXT:  .LBB8_1:                                // %loop
+;CHECK-NEXT:                                          // =>This Inner Loop Header: Depth=1
+;CHECK-NEXT:    ldr d2, [x0]
+;CHECK-NEXT:    add x8, x8, #8
+;CHECK-NEXT:    subs  x2, x2, #8
+;CHECK-NEXT:    tbl v2.16b, { v2.16b }, v0.16b
+;CHECK-NEXT:    mul v2.8h, v2.8h, v1.8h
+;CHECK-NEXT:    cmlt  v2.8h, v2.8h, #0
+;CHECK-NEXT:    xtn v2.8b, v2.8h
+;CHECK-NEXT:    str d2, [x0], #32
+;CHECK-NEXT:    b.ne  .LBB8_1
+;CHECK-NEXT:  // %bb.2:                               // %exit
+;CHECK-NEXT:    ret
 entry:
   %ext = zext <16 x i8> %a to <16 x i16>
   %broadcast.splat = shufflevector <16 x i16> %ext, <16 x i16> poison, <8 x i32> <i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0, i32 0>
   br label %loop
 
 loop:
   %index = phi i64 [ 0, %entry ], [ %index.next, %loop ]
   %g = getelementptr inbounds i32, i32 *%p, i64 %index

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI5_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI5_1:
+; CHECK-NEXT:   .byte 8                               ; 0x8
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 9                               ; 0x9
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 10                              ; 0xa
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 11                              ; 0xb
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 12                              ; 0xc
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 13                              ; 0xd
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 14                              ; 0xe
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 15                              ; 0xf
+; CHECK-NEXT:   .byte 255                             ; 0xff
 
+; CHECK-BE-LABEL:  .LCPI5_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+; CHECK-BE-NEXT:  .LCPI5_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 8                               // 0x8
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 9                               // 0x9
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 10                              // 0xa
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 11                              // 0xb
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 12                              // 0xc
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 13                              // 0xd
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 14                              // 0xe
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 15                              // 0xf
 
 define void @zext_v16i8_to_v16i16_in_loop(i8* %src, i16* %dst) {
 ; CHECK-LABEL: zext_v16i8_to_v16i16_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  LBB5_1: ; %loop
-; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q0, [x0, x8]
-; CHECK-NEXT:    add x8, x8, #16
-; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    ushll2.8h v1, v0, #0
-; CHECK-NEXT:    ushll.8h v0, v0, #0
-; CHECK-NEXT:    stp q0, q1, [x1], #32
-; CHECK-NEXT:    b.ne LBB5_1
-; CHECK-NEXT:  ; %bb.2: ; %exit
-; CHECK-NEXT:    ret
+; CHECK-NEXT:  Lloh8:
+; CHECK-NEXT:   adrp  x9, lCPI5_0@PAGE
+; CHECK-NEXT:  Lloh9:
+; CHECK-NEXT:   adrp  x10, lCPI5_1@PAGE
+; CHECK-NEXT:   mov x8, xzr
+; CHECK-NEXT:  Lloh10:
+; CHECK-NEXT:   ldr q0, [x9, lCPI5_0@PAGEOFF]
+; CHECK-NEXT:  Lloh11:
+; CHECK-NEXT:   ldr q1, [x10, lCPI5_1@PAGEOFF]
+; CHECK-NEXT:  LBB5_1:                                 ; %loop
+; CHECK-NEXT:                                          ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:   ldr q2, [x0, x8]
+; CHECK-NEXT:   add x8, x8, #16
+; CHECK-NEXT:   cmp x8, #128
+; CHECK-NEXT:   tbl.16b v3, { v2 }, v1
+; CHECK-NEXT:   tbl.16b v2, { v2 }, v0
+; CHECK-NEXT:   stp q2, q3, [x1], #32
+; CHECK-NEXT:   b.ne  LBB5_1
+; CHECK-NEXT:  ; %bb.2:                                ; %exit
+; CHECK-NEXT:   ret
+
 ;
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i16_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
-; CHECK-BE-NEXT:    mov x8, xzr
-; CHECK-BE-NEXT:  .LBB5_1: // %loop
-; CHECK-BE-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #16
-; CHECK-BE-NEXT:    ushll v1.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2 v0.8h, v0.16b, #0
-; CHECK-BE-NEXT:    st1 { v1.8h }, [x1]
-; CHECK-BE-NEXT:    add x1, x1, #32
-; CHECK-BE-NEXT:    st1 { v0.8h }, [x9]
-; CHECK-BE-NEXT:    b.ne .LBB5_1
-; CHECK-BE-NEXT:  // %bb.2: // %exit
-; CHECK-BE-NEXT:    ret
+; CHECK-BE-NEXT:    adrp  x8, .LCPI5_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI5_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI5_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI5_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:  .LBB5_1:                                // %loop
+; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
+; CHECK-BE-NEXT:    add x9, x0, x8
+; CHECK-BE-NEXT:    add x8, x8, #16
+; CHECK-BE-NEXT:    cmp x8, #128
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #16
+; CHECK-BE-NEXT:    tbl v3.16b, { v2.16b }, v0.16b
+; CHECK-BE-NEXT:    tbl v2.16b, { v2.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v3.16b }, [x1]
+; CHECK-BE-NEXT:    add x1, x1, #32
+; CHECK-BE-NEXT:    st1 { v2.16b }, [x9]
+; CHECK-BE-NEXT:    b.ne  .LBB5_1
+; CHECK-BE-NEXT:  // %bb.2:                               // %exit
+; CHECK-BE-NEXT:    ret
+
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <16 x i8>*
   %load = load <16 x i8>, <16 x i8>* %src.gep.cast

 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 7                               // 0x7
 
 define void @zext_v8i8_to_v8i32_in_loop(i8* %src, i32* %dst) {
 ; CHECK-LABEL: zext_v8i8_to_v8i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh8:
+; CHECK-NEXT:  Lloh12:
 ; CHECK-NEXT:    adrp x9, lCPI6_0@PAGE
-; CHECK-NEXT:  Lloh9:
+; CHECK-NEXT:  Lloh13:
 ; CHECK-NEXT:    adrp x10, lCPI6_1@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh10:
+; CHECK-NEXT:  Lloh14:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI6_0@PAGEOFF]
-; CHECK-NEXT:  Lloh11:
+; CHECK-NEXT:  Lloh15:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI6_1@PAGEOFF]
 ; CHECK-NEXT:  LBB6_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr d2, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    tbl.16b v3, { v2 }, v1
 ; CHECK-NEXT:    tbl.16b v2, { v2 }, v0
 ; CHECK-NEXT:    stp q2, q3, [x1], #64
 ; CHECK-NEXT:    b.ne LBB6_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh9, Lloh11
-; CHECK-NEXT:    .loh AdrpLdr Lloh8, Lloh10
+; CHECK-NEXT:    .loh AdrpLdr Lloh13, Lloh15
+; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh14
 ;
 ; CHECK-BE-LABEL: zext_v8i8_to_v8i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI6_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI6_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI6_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI6_1
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
+; CHECK-LABEL: lCPI7_0:
+; CHECK-NEXT:   .byte 0                               ; 0x0
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 1                               ; 0x1
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_1:
+; CHECK-NEXT:   .byte 2                               ; 0x2
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 3                               ; 0x3
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_2:
+; CHECK-NEXT:   .byte 4                               ; 0x4
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 5                               ; 0x5
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_3:
+; CHECK-NEXT:   .byte 6                               ; 0x6
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 7                               ; 0x7
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_4:
+; CHECK-NEXT:   .byte 8                               ; 0x8
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 9                               ; 0x9
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_5:
+; CHECK-NEXT:   .byte 10                              ; 0xa
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 11                              ; 0xb
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_6:
+; CHECK-NEXT:   .byte 12                              ; 0xc
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 13                              ; 0xd
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT: lCPI7_7:
+; CHECK-NEXT:   .byte 14                              ; 0xe
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 15                              ; 0xf
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+; CHECK-NEXT:   .byte 255                             ; 0xff
+
+; CHECK-BE-LABEL:  .LCPI7_0:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 0                               // 0x0
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 1                               // 0x1
+; CHECK-BE-NEXT:  .LCPI7_1:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 2                               // 0x2
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 3                               // 0x3
+; CHECK-BE-NEXT:  .LCPI7_2:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 4                               // 0x4
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 5                               // 0x5
+; CHECK-BE-NEXT:  .LCPI7_3:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 6                               // 0x6
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 7                               // 0x7
+; CHECK-BE-NEXT:  .LCPI7_4:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 8                               // 0x8
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 9                               // 0x9
+; CHECK-BE-NEXT:  .LCPI7_5:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 10                              // 0xa
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 11                              // 0xb
+; CHECK-BE-NEXT:  .LCPI7_6:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 12                              // 0xc
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 13                              // 0xd
+; CHECK-BE-NEXT:  .LCPI7_7:
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 14                              // 0xe
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 255                             // 0xff
+; CHECK-BE-NEXT:    .byte 15                              // 0xf
+
 define void @zext_v16i8_to_v16i64_in_loop(i8* %src, i64* %dst) {
 ; CHECK-LABEL: zext_v16i8_to_v16i64_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  LBB7_1: ; %loop
-; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q0, [x0, x8]
-; CHECK-NEXT:    add x8, x8, #16
-; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    ushll.8h v1, v0, #0
-; CHECK-NEXT:    ushll2.8h v0, v0, #0
-; CHECK-NEXT:    ushll2.4s v2, v1, #0
-; CHECK-NEXT:    ushll2.4s v3, v0, #0
-; CHECK-NEXT:    ushll.4s v0, v0, #0
-; CHECK-NEXT:    ushll2.2d v4, v3, #0
-; CHECK-NEXT:    ushll2.2d v5, v0, #0
-; CHECK-NEXT:    ushll.2d v0, v0, #0
-; CHECK-NEXT:    ushll.2d v3, v3, #0
-; CHECK-NEXT:    stp q0, q5, [x1, #64]
-; CHECK-NEXT:    ushll.4s v0, v1, #0
-; CHECK-NEXT:    stp q3, q4, [x1, #96]
-; CHECK-NEXT:    ushll2.2d v3, v2, #0
-; CHECK-NEXT:    ushll.2d v2, v2, #0
-; CHECK-NEXT:    ushll2.2d v1, v0, #0
-; CHECK-NEXT:    ushll.2d v0, v0, #0
-; CHECK-NEXT:    stp q2, q3, [x1, #32]
-; CHECK-NEXT:    stp q0, q1, [x1], #128
-; CHECK-NEXT:    b.ne LBB7_1
-; CHECK-NEXT:  ; %bb.2: ; %exit
-; CHECK-NEXT:    ret
-;
+; CHECK-NEXT:    Lloh16:
+; CHECK-NEXT:     adrp  x9, lCPI7_0@PAGE
+; CHECK-NEXT:    Lloh17:
+; CHECK-NEXT:     adrp  x10, lCPI7_1@PAGE
+; CHECK-NEXT:    Lloh18:
+; CHECK-NEXT:     adrp  x11, lCPI7_2@PAGE
+; CHECK-NEXT:     mov x8, xzr
+; CHECK-NEXT:    Lloh19:
+; CHECK-NEXT:     ldr q0, [x9, lCPI7_0@PAGEOFF]
+; CHECK-NEXT:    Lloh20:
+; CHECK-NEXT:     adrp  x9, lCPI7_3@PAGE
+; CHECK-NEXT:    Lloh21:
+; CHECK-NEXT:     ldr q1, [x10, lCPI7_1@PAGEOFF]
+; CHECK-NEXT:    Lloh22:
+; CHECK-NEXT:     adrp  x10, lCPI7_4@PAGE
+; CHECK-NEXT:    Lloh23:
+; CHECK-NEXT:     ldr q2, [x11, lCPI7_2@PAGEOFF]
+; CHECK-NEXT:    Lloh24:
+; CHECK-NEXT:     adrp  x11, lCPI7_5@PAGE
+; CHECK-NEXT:    Lloh25:
+; CHECK-NEXT:     ldr q3, [x9, lCPI7_3@PAGEOFF]
+; CHECK-NEXT:    Lloh26:
+; CHECK-NEXT:     adrp  x9, lCPI7_6@PAGE
+; CHECK-NEXT:    Lloh27:
+; CHECK-NEXT:     ldr q4, [x10, lCPI7_4@PAGEOFF]
+; CHECK-NEXT:    Lloh28:
+; CHECK-NEXT:     adrp  x10, lCPI7_7@PAGE
+; CHECK-NEXT:    Lloh29:
+; CHECK-NEXT:     ldr q5, [x11, lCPI7_5@PAGEOFF]
+; CHECK-NEXT:    Lloh30:
+; CHECK-NEXT:     ldr q6, [x9, lCPI7_6@PAGEOFF]
+; CHECK-NEXT:    Lloh31:
+; CHECK-NEXT:     ldr q7, [x10, lCPI7_7@PAGEOFF]
+; CHECK-NEXT:    LBB7_1:                                 ; %loop
+; CHECK-NEXT:                                            ; =>This Inner Loop Header: Depth=1
+; CHECK-NEXT:     ldr q16, [x0, x8]
+; CHECK-NEXT:     add x8, x8, #16
+; CHECK-NEXT:     cmp x8, #128
+; CHECK-NEXT:     tbl.16b v17, { v16 }, v7
+; CHECK-NEXT:     tbl.16b v18, { v16 }, v6
+; CHECK-NEXT:     tbl.16b v19, { v16 }, v5
+; CHECK-NEXT:     tbl.16b v20, { v16 }, v4
+; CHECK-NEXT:     tbl.16b v21, { v16 }, v3
+; CHECK-NEXT:     stp q18, q17, [x1, #96]
+; CHECK-NEXT:     tbl.16b v17, { v16 }, v2
+; CHECK-NEXT:     tbl.16b v18, { v16 }, v1
+; CHECK-NEXT:     stp q20, q19, [x1, #64]
+; CHECK-NEXT:     tbl.16b v16, { v16 }, v0
+; CHECK-NEXT:     stp q17, q21, [x1, #32]
+; CHECK-NEXT:     stp q16, q18, [x1], #128
+; CHECK-NEXT:     b.ne  LBB7_1
+; CHECK-NEXT:    ; %bb.2:                                ; %exit
+; CHECK-NEXT:     ret
+; CHECK-NEXT:     .loh AdrpLdr  Lloh28, Lloh31
+; CHECK-NEXT:     .loh AdrpLdr  Lloh26, Lloh30
+; CHECK-NEXT:     .loh AdrpLdr  Lloh24, Lloh29
+; CHECK-NEXT:     .loh AdrpAdrp Lloh22, Lloh28
+; CHECK-NEXT:     .loh AdrpLdr  Lloh22, Lloh27
+; CHECK-NEXT:     .loh AdrpAdrp Lloh20, Lloh26
+; CHECK-NEXT:     .loh AdrpLdr  Lloh20, Lloh25
+; CHECK-NEXT:     .loh AdrpAdrp Lloh18, Lloh24
+; CHECK-NEXT:     .loh AdrpLdr  Lloh18, Lloh23
+; CHECK-NEXT:     .loh AdrpAdrp Lloh17, Lloh22
+; CHECK-NEXT:     .loh AdrpLdr  Lloh17, Lloh21
+; CHECK-NEXT:     .loh AdrpAdrp Lloh16, Lloh20
+; CHECK-NEXT:     .loh AdrpLdr  Lloh16, Lloh19
+
+
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i64_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
-; CHECK-BE-NEXT:    mov x8, xzr
-; CHECK-BE-NEXT:  .LBB7_1: // %loop
-; CHECK-BE-NEXT:    // =>This Inner Loop Header: Depth=1
-; CHECK-BE-NEXT:    add x9, x0, x8
-; CHECK-BE-NEXT:    add x10, x1, #96
-; CHECK-BE-NEXT:    add x8, x8, #16
-; CHECK-BE-NEXT:    cmp x8, #128
-; CHECK-BE-NEXT:    ld1 { v0.16b }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #112
-; CHECK-BE-NEXT:    ushll2 v1.8h, v0.16b, #0
-; CHECK-BE-NEXT:    ushll v0.8h, v0.8b, #0
-; CHECK-BE-NEXT:    ushll2 v2.4s, v1.8h, #0
-; CHECK-BE-NEXT:    ushll v1.4s, v1.4h, #0
-; CHECK-BE-NEXT:    ushll2 v3.2d, v2.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #80
-; CHECK-BE-NEXT:    ushll2 v3.2d, v1.4s, #0
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x10]
-; CHECK-BE-NEXT:    ushll2 v2.4s, v0.8h, #0
-; CHECK-BE-NEXT:    add x10, x1, #48
-; CHECK-BE-NEXT:    st1 { v3.2d }, [x9]
-; CHECK-BE-NEXT:    add x9, x1, #64
-; CHECK-BE-NEXT:    ushll v1.2d, v1.2s, #0
-; CHECK-BE-NEXT:    ushll v0.4s, v0.4h, #0
-; CHECK-BE-NEXT:    ushll2 v4.2d, v2.4s, #0
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x9]
-; CHECK-BE-NEXT:    ushll v1.2d, v0.2s, #0
-; CHECK-BE-NEXT:    add x9, x1, #16
-; CHECK-BE-NEXT:    st1 { v4.2d }, [x10]
-; CHECK-BE-NEXT:    add x10, x1, #32
-; CHECK-BE-NEXT:    st1 { v1.2d }, [x1]
-; CHECK-BE-NEXT:    add x1, x1, #128
-; CHECK-BE-NEXT:    ushll2 v0.2d, v0.4s, #0
-; CHECK-BE-NEXT:    ushll v2.2d, v2.2s, #0
-; CHECK-BE-NEXT:    st1 { v0.2d }, [x9]
-; CHECK-BE-NEXT:    st1 { v2.2d }, [x10]
-; CHECK-BE-NEXT:    b.ne .LBB7_1
-; CHECK-BE-NEXT:  // %bb.2: // %exit
-; CHECK-BE-NEXT:    ret
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_0
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_0
+; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_1
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_1
+; CHECK-BE-NEXT:    ld1 { v1.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_2
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_2
+; CHECK-BE-NEXT:    ld1 { v2.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_3
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_3
+; CHECK-BE-NEXT:    ld1 { v3.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_4
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_4
+; CHECK-BE-NEXT:    ld1 { v4.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_5
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_5
+; CHECK-BE-NEXT:    ld1 { v5.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_6
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_6
+; CHECK-BE-NEXT:    ld1 { v6.16b }, [x8]
+; CHECK-BE-NEXT:    adrp  x8, .LCPI7_7
+; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI7_7
+; CHECK-BE-NEXT:    ld1 { v7.16b }, [x8]
+; CHECK-BE-NEXT:    mov x8, xzr
+; CHECK-BE-NEXT:  .LBB7_1:                                // %loop
+; CHECK-BE-NEXT:                                          // =>This Inner Loop Header: Depth=1
+; CHECK-BE-NEXT:    add x9, x0, x8
+; CHECK-BE-NEXT:    add x10, x1, #96
+; CHECK-BE-NEXT:    add x8, x8, #16
+; CHECK-BE-NEXT:    cmp x8, #128
+; CHECK-BE-NEXT:    ld1 { v16.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #112
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v7.16b
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v6.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v5.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #80
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v4.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x10]
+; CHECK-BE-NEXT:    add x10, x1, #64
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #48
+; CHECK-BE-NEXT:    tbl v18.16b, { v16.16b }, v3.16b
+; CHECK-BE-NEXT:    tbl v19.16b, { v16.16b }, v0.16b
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    tbl v17.16b, { v16.16b }, v2.16b
+; CHECK-BE-NEXT:    add x10, x1, #32
+; CHECK-BE-NEXT:    tbl v16.16b, { v16.16b }, v1.16b
+; CHECK-BE-NEXT:    st1 { v18.16b }, [x9]
+; CHECK-BE-NEXT:    add x9, x1, #16
+; CHECK-BE-NEXT:    st1 { v19.16b }, [x1]
+; CHECK-BE-NEXT:    add x1, x1, #128
+; CHECK-BE-NEXT:    st1 { v17.16b }, [x10]
+; CHECK-BE-NEXT:    st1 { v16.16b }, [x9]
+; CHECK-BE-NEXT:    b.ne  .LBB7_1
+; CHECK-BE-NEXT:  // %bb.2:                               // %exit
+; CHECK-BE-NEXT:    ret
+
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, i8* %src, i64 %iv
   %src.gep.cast = bitcast i8* %src.gep to <16 x i8>*
   %load = load <16 x i8>, <16 x i8>* %src.gep.cast
   %ext = zext <16 x i8> %load to <16 x i64>
   %dst.gep = getelementptr i64, i64* %dst, i64 %iv
   %dst.gep.cast = bitcast i64* %dst.gep to <16 x i64>*
   store <16 x i64> %ext, <16 x i64>* %dst.gep.cast
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }

llvm/test/Transforms/CodeGenPrepare/AArch64/zext-to-shuffle.ll

 ; CHECK-LABEL: @zext_v16i8_to_v16i16_in_loop(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
 ; CHECK:       loop:
 ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[SRC_GEP:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i64 [[IV]]
 ; CHECK-NEXT:    [[SRC_GEP_CAST:%.*]] = bitcast i8* [[SRC_GEP]] to <16 x i8>*
 ; CHECK-NEXT:    [[LOAD:%.*]] = load <16 x i8>, <16 x i8>* [[SRC_GEP_CAST]], align 16
-; CHECK-NEXT:    [[EXT:%.*]] = zext <16 x i8> [[LOAD]] to <16 x i16>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <16 x i8> %load, <16 x i8> <i8 0, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison>, <32 x i32> <i32 0, i32 16, i32 1, i32 16, i32 2, i32 16, i32 3, i32 16, i32 4, i32 16, i32 5, i32 16, i32 6, i32 16, i32 7, i32 16, i32 8, i32 16, i32 9, i32 16, i32 10, i32 16, i32 11, i32 16, i32 12, i32 16, i32 13, i32 16, i32 14, i32 16, i32 15, i32 16>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <32 x i8> [[TMP0:%.*]] to <16 x i16>
 ; CHECK-NEXT:    [[DST_GEP:%.*]] = getelementptr i16, i16* [[DST:%.*]], i64 [[IV]]
 ; CHECK-NEXT:    [[DST_GEP_CAST:%.*]] = bitcast i16* [[DST_GEP]] to <16 x i16>*
-; CHECK-NEXT:    store <16 x i16> [[EXT]], <16 x i16>* [[DST_GEP_CAST]], align 32
+; CHECK-NEXT:    store <16 x i16> [[TMP1:%.*]], <16 x i16>* [[DST_GEP_CAST]], align 32
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw i64 [[IV]], 16
 ; CHECK-NEXT:    [[EC:%.*]] = icmp eq i64 [[IV_NEXT]], 128
 ; CHECK-NEXT:    br i1 [[EC]], label [[EXIT:%.*]], label [[LOOP]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br label %loop

 exit:
   ret void
 }
 
 define void @zext_v16i8_to_v16i64_in_loop(i8* %src, i64* %dst) {
 ; CHECK-LABEL: @zext_v16i8_to_v16i64_in_loop(
 ; CHECK-NEXT:  entry:
 ; CHECK-NEXT:    br label [[LOOP:%.*]]
-; CHECK:       loop:
+
+;CHECK:        loop:                                             ; preds = %loop, %entry
 ; CHECK-NEXT:    [[IV:%.*]] = phi i64 [ 0, [[ENTRY:%.*]] ], [ [[IV_NEXT:%.*]], [[LOOP]] ]
 ; CHECK-NEXT:    [[SRC_GEP:%.*]] = getelementptr i8, i8* [[SRC:%.*]], i64 [[IV]]
 ; CHECK-NEXT:    [[SRC_GEP_CAST:%.*]] = bitcast i8* [[SRC_GEP]] to <16 x i8>*
 ; CHECK-NEXT:    [[LOAD:%.*]] = load <16 x i8>, <16 x i8>* [[SRC_GEP_CAST]], align 16
-; CHECK-NEXT:    [[EXT:%.*]] = zext <16 x i8> [[LOAD]] to <16 x i64>
+; CHECK-NEXT:    [[TMP0:%.*]] = shufflevector <16 x i8> %load, <16 x i8> <i8 0, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison, i8 poison>, <128 x i32> <i32 0, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 1, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 2, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 3, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 4, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 5, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 6, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 7, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 8, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 9, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 10, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 11, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 12, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 13, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 14, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 15, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16, i32 16>
+; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <128 x i8> [[TMP0]] to <16 x i64>
 ; CHECK-NEXT:    [[DST_GEP:%.*]] = getelementptr i64, i64* [[DST:%.*]], i64 [[IV]]
 ; CHECK-NEXT:    [[DST_GEP_CAST:%.*]] = bitcast i64* [[DST_GEP]] to <16 x i64>*
-; CHECK-NEXT:    store <16 x i64> [[EXT]], <16 x i64>* [[DST_GEP_CAST]], align 128
+; CHECK-NEXT:    store <16 x i64> [[TMP1]], <16 x i64>* [[DST_GEP_CAST]], align 128
 ; CHECK-NEXT:    [[IV_NEXT]] = add nuw i64 [[IV]], 16
 ; CHECK-NEXT:    [[EC:%.*]] = icmp eq i64 [[IV_NEXT]], 128
 ; CHECK-NEXT:    br i1 [[EC]], label [[EXIT:%.*]], label [[LOOP]]
 ; CHECK:       exit:
 ; CHECK-NEXT:    ret void
 ;
 entry:
   br label %loop