[AArch64][DAG] `canCombineShuffleToExtendVectorInreg()`: allow illegal types before legalization

[lebedev.ri]

Summary

[DAG] canCombineShuffleToExtendVectorInreg(): allow illegal types before legalization

This isn't all that interesting on itself, but this is a prerequisite
to get to the more interesting changes in a follow-up...

[lebedev.ri]

Rebased.

[lebedev.ri]

Another batch of regressions gone.

(@RKSimon: this is the next patch after D140811, but ignore this one for now.)

[fhahn]

llvm/test/CodeGen/AArch64/fp-conversion-to-tbl.ll

452–457

I think this is not quite as clear cut as D140676, but I think this is still a regression over the original code where the constant-pool loads are outside the loop.

[lebedev.ri]

llvm/test/CodeGen/AArch64/fp-conversion-to-tbl.ll

452–457

I see. I'm still stomping out many many X86 regressions here,
and didn't look into this AArch64-one.
If someone else wants to, please feel free to do so?

[lebedev.ri]

Rebased.

[lebedev.ri]

llvm/test/CodeGen/AArch64/fp-conversion-to-tbl.ll

452–457

Looks like that regression has been resolved, but other changes appeared.

[lebedev.ri]

llvm/test/CodeGen/X86/zero_extend_vector_inreg.ll

2713–2714 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

2931 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

3147 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

3347 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

3361–3366 ↗

(On Diff #486403)

Missing shuffle lowering strategy?

3703 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

4350–4353 ↗

(On Diff #486403)

???

4370–4376 ↗

(On Diff #486403)

???

4939 ↗

(On Diff #486403)

Looks like we lost UNDEF knowledge and can no longer eliminate these two redundant operations

llvm/lib/CodeGen/SelectionDAG/DAGCombiner.cpp

 }
 
 // Match shuffles that can be converted to zero_extend_vector_inreg.
 // This is often generated during legalization.
 // e.g. v4i32 <0,z,1,u> -> (v2i64 zero_extend_vector_inreg(v4i32 src))
 static SDValue combineShuffleToZeroExtendVectorInReg(ShuffleVectorSDNode *SVN,
                                                      SelectionDAG &DAG,
                                                      const TargetLowering &TLI,
+                                                     bool LegalTypes,
                                                      bool LegalOperations) {
-  bool LegalTypes = true;
   EVT VT = SVN->getValueType(0);
   assert(!VT.isScalableVector() && "Encountered scalable shuffle?");
   unsigned NumElts = VT.getVectorNumElements();
   unsigned EltSizeInBits = VT.getScalarSizeInBits();
 
   // TODO: add support for big-endian when we have a test case.
   bool IsBigEndian = DAG.getDataLayout().isBigEndian();
   if (!VT.isInteger() || IsBigEndian)
SDValue DAGCombiner::visitVECTOR_SHUFFLE(SDNode *N) {
 
   if (SDValue V = foldShuffleOfConcatUndefs(SVN, DAG))
     return V;
 
   // Match shuffles that can be converted to ISD::ZERO_EXTEND_VECTOR_INREG.
   // Perform this really late, because it could eliminate knowledge
   // of undef elements created by this shuffle.
   if (Level < AfterLegalizeTypes)
-    if (SDValue V = combineShuffleToZeroExtendVectorInReg(SVN, DAG, TLI,
-                                                          LegalOperations))
+    if (SDValue V = combineShuffleToZeroExtendVectorInReg(
+            SVN, DAG, TLI, LegalTypes, LegalOperations))
       return V;
 
   return SDValue();
 }
 
 SDValue DAGCombiner::visitSCALAR_TO_VECTOR(SDNode *N) {
   EVT VT = N->getValueType(0);
   if (!VT.isFixedLengthVector())

llvm/test/CodeGen/AArch64/fp-conversion-to-tbl.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -o - %s | FileCheck %s
+; RUN: llc -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI %s
 
 target datalayout = "e-m:o-i64:64-i128:128-n32:64-S128"
 target triple = "arm64-apple-ios"
 
 ; CHECK-LABEL: lCPI0_0:
 ; CHECK-NEXT:     .byte    0                               ; 0x0
 ; CHECK-NEXT:     .byte    4                               ; 0x4
 ; CHECK-NEXT:     .byte    8                               ; 0x8
loop:
   %iv.next = add i64 %iv, 1
   %ec = icmp eq i64 %iv.next, 1000
   br i1 %ec, label %loop, label %exit
 
 exit:
   ret void
 }
 
-; CHECK-LABEL: lCPI8_0:
-; CHECK-NEXT:   .byte   4                               ; 0x4
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   5                               ; 0x5
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   6                               ; 0x6
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   7                               ; 0x7
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT: lCPI8_1:
-; CHECK-NEXT:   .byte   0                               ; 0x0
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   1                               ; 0x1
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   2                               ; 0x2
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   3                               ; 0x3
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-; CHECK-NEXT:   .byte   255                             ; 0xff
-
 define void @uitofp_v8i8_to_v8f32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: uitofp_v8i8_to_v8f32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh10:
-; CHECK-NEXT:    adrp x9, lCPI8_0@PAGE
-; CHECK-NEXT:  Lloh11:
-; CHECK-NEXT:    adrp x10, lCPI8_1@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh12:
-; CHECK-NEXT:    ldr q0, [x9, lCPI8_0@PAGEOFF]
-; CHECK-NEXT:  Lloh13:
-; CHECK-NEXT:    ldr q1, [x10, lCPI8_1@PAGEOFF]
 ; CHECK-NEXT:  LBB8_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr d2, [x0, x8, lsl #3]
+; CHECK-NEXT:    ldr d1, [x0, x8, lsl #3]
 ; CHECK-NEXT:    add x9, x1, x8, lsl #5
 ; CHECK-NEXT:    add x8, x8, #1
 ; CHECK-NEXT:    cmp x8, #1000
-; CHECK-NEXT:    tbl.16b v3, { v2 }, v0
-; CHECK-NEXT:    tbl.16b v2, { v2 }, v1
-; CHECK-NEXT:    ucvtf.4s v3, v3
+; CHECK-NEXT:    dup.2s v2, v1[1]
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    zip1.16b v2, v2, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
+; CHECK-NEXT:    zip1.8h v2, v2, v0
+; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    ucvtf.4s v2, v2
-; CHECK-NEXT:    stp q2, q3, [x9]
+; CHECK-NEXT:    stp q1, q2, [x9]
 ; CHECK-NEXT:    b.eq LBB8_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh11, Lloh13
-; CHECK-NEXT:    .loh AdrpLdr Lloh10, Lloh12
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %gep.src = getelementptr inbounds <8 x i8>, ptr %src, i64 %iv
   %l = load <8 x i8>, ptr %gep.src
   %conv = uitofp <8 x i8> %l to <8 x float>
   %gep.dst = getelementptr inbounds <8 x float>, ptr %dst, i64 %iv
   store <8 x float> %conv, ptr %gep.dst
   %iv.next = add i64 %iv, 1
   %ec = icmp eq i64 %iv.next, 1000
   br i1 %ec, label %loop, label %exit
 
 exit:
   ret void
 }
 
-; CHECK-LABEL: lCPI9_0:
-; CHECK-NEXT:     .byte   12                              ; 0xc
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   13                              ; 0xd
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   14                              ; 0xe
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   15                              ; 0xf
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT: lCPI9_1:
-; CHECK-NEXT:     .byte   8                               ; 0x8
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   9                               ; 0x9
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   10                              ; 0xa
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   11                              ; 0xb
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT: lCPI9_2:
-; CHECK-NEXT:     .byte   4                               ; 0x4
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   5                               ; 0x5
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   6                               ; 0x6
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   7                               ; 0x7
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT: lCPI9_3:
-; CHECK-NEXT:     .byte   0                               ; 0x0
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   1                               ; 0x1
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   2                               ; 0x2
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   3                               ; 0x3
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-
 define void @uitofp_v16i8_to_v16f32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: uitofp_v16i8_to_v16f32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh14:
-; CHECK-NEXT:    adrp x9, lCPI9_0@PAGE
-; CHECK-NEXT:  Lloh15:
-; CHECK-NEXT:    adrp x10, lCPI9_1@PAGE
-; CHECK-NEXT:  Lloh16:
-; CHECK-NEXT:    adrp x11, lCPI9_2@PAGE
-; CHECK-NEXT:  Lloh17:
-; CHECK-NEXT:    adrp x12, lCPI9_3@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh18:
-; CHECK-NEXT:    ldr q0, [x9, lCPI9_0@PAGEOFF]
-; CHECK-NEXT:  Lloh19:
-; CHECK-NEXT:    ldr q1, [x10, lCPI9_1@PAGEOFF]
-; CHECK-NEXT:  Lloh20:
-; CHECK-NEXT:    ldr q2, [x11, lCPI9_2@PAGEOFF]
-; CHECK-NEXT:  Lloh21:
-; CHECK-NEXT:    ldr q3, [x12, lCPI9_3@PAGEOFF]
 ; CHECK-NEXT:  LBB9_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q4, [x0, x8, lsl #4]
+; CHECK-NEXT:    ldr q1, [x0, x8, lsl #4]
 ; CHECK-NEXT:    add x9, x1, x8, lsl #6
 ; CHECK-NEXT:    add x8, x8, #1
 ; CHECK-NEXT:    cmp x8, #1000
-; CHECK-NEXT:    tbl.16b v5, { v4 }, v0
-; CHECK-NEXT:    tbl.16b v6, { v4 }, v1
-; CHECK-NEXT:    tbl.16b v7, { v4 }, v2
-; CHECK-NEXT:    tbl.16b v4, { v4 }, v3
-; CHECK-NEXT:    ucvtf.4s v5, v5
-; CHECK-NEXT:    ucvtf.4s v6, v6
-; CHECK-NEXT:    ucvtf.4s v7, v7
+; CHECK-NEXT:    zip1.16b v2, v1, v0
+; CHECK-NEXT:    dup.4s v3, v1[1]
+; CHECK-NEXT:    dup.2d v1, v1[1]
+; CHECK-NEXT:    zip1.16b v3, v3, v0
+; CHECK-NEXT:    dup.4s v4, v1[1]
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    zip1.16b v4, v4, v0
+; CHECK-NEXT:    zip1.8h v2, v2, v0
+; CHECK-NEXT:    zip1.8h v3, v3, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
+; CHECK-NEXT:    zip1.8h v4, v4, v0
+; CHECK-NEXT:    ucvtf.4s v2, v2
+; CHECK-NEXT:    ucvtf.4s v3, v3
+; CHECK-NEXT:    ucvtf.4s v1, v1
 ; CHECK-NEXT:    ucvtf.4s v4, v4
-; CHECK-NEXT:    stp q6, q5, [x9, #32]
-; CHECK-NEXT:    stp q4, q7, [x9]
+; CHECK-NEXT:    stp q2, q3, [x9]
+; CHECK-NEXT:    stp q1, q4, [x9, #32]
 ; CHECK-NEXT:    b.eq LBB9_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh17, Lloh21
-; CHECK-NEXT:    .loh AdrpLdr Lloh16, Lloh20
-; CHECK-NEXT:    .loh AdrpLdr Lloh15, Lloh19
-; CHECK-NEXT:    .loh AdrpLdr Lloh14, Lloh18
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %gep.src = getelementptr inbounds <16 x i8>, ptr %src, i64 %iv
   %l = load <16 x i8>, ptr %gep.src
   %conv = uitofp <16 x i8> %l to <16 x float>

llvm/test/CodeGen/AArch64/vselect-ext.ll

entry:
   %cmp = icmp sgt <16 x i8> %a,  <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   ret <16 x i32> %sel
 }
 
 define void @extension_in_loop_v16i8_to_v16i32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: extension_in_loop_v16i8_to_v16i32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh2:
-; CHECK-NEXT:    adrp x9, lCPI24_0@PAGE
-; CHECK-NEXT:  Lloh3:
-; CHECK-NEXT:    adrp x10, lCPI24_1@PAGE
-; CHECK-NEXT:  Lloh4:
-; CHECK-NEXT:    adrp x11, lCPI24_2@PAGE
-; CHECK-NEXT:  Lloh5:
-; CHECK-NEXT:    adrp x12, lCPI24_3@PAGE
-; CHECK-NEXT:    movi.2d v2, #0xffffffffffffffff
+; CHECK-NEXT:    movi.2d v0, #0xffffffffffffffff
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh6:
-; CHECK-NEXT:    ldr q0, [x9, lCPI24_0@PAGEOFF]
-; CHECK-NEXT:  Lloh7:
-; CHECK-NEXT:    ldr q1, [x10, lCPI24_1@PAGEOFF]
-; CHECK-NEXT:  Lloh8:
-; CHECK-NEXT:    ldr q3, [x11, lCPI24_2@PAGEOFF]
-; CHECK-NEXT:  Lloh9:
-; CHECK-NEXT:    ldr q4, [x12, lCPI24_3@PAGEOFF]
+; CHECK-NEXT:    movi.2d v1, #0000000000000000
 ; CHECK-NEXT:  LBB24_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q5, [x0, x8]
+; CHECK-NEXT:    ldr q2, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    cmgt.16b v6, v5, v2
-; CHECK-NEXT:    tbl.16b v7, { v5 }, v0
-; CHECK-NEXT:    tbl.16b v16, { v5 }, v1
-; CHECK-NEXT:    sshll2.8h v18, v6, #0
-; CHECK-NEXT:    tbl.16b v17, { v5 }, v3
-; CHECK-NEXT:    sshll2.4s v19, v18, #0
-; CHECK-NEXT:    sshll.4s v18, v18, #0
-; CHECK-NEXT:    tbl.16b v5, { v5 }, v4
-; CHECK-NEXT:    sshll.8h v6, v6, #0
-; CHECK-NEXT:    and.16b v7, v7, v19
-; CHECK-NEXT:    and.16b v16, v16, v18
-; CHECK-NEXT:    stp q16, q7, [x1, #32]
-; CHECK-NEXT:    sshll2.4s v7, v6, #0
+; CHECK-NEXT:    dup.2d v4, v2[1]
+; CHECK-NEXT:    cmgt.16b v3, v2, v0
+; CHECK-NEXT:    zip1.16b v5, v2, v1
+; CHECK-NEXT:    dup.4s v2, v2[1]
+; CHECK-NEXT:    dup.4s v7, v4[1]
+; CHECK-NEXT:    zip1.16b v4, v4, v1
+; CHECK-NEXT:    zip1.16b v7, v7, v1
+; CHECK-NEXT:    zip1.16b v2, v2, v1
+; CHECK-NEXT:    sshll2.8h v6, v3, #0
+; CHECK-NEXT:    zip1.8h v4, v4, v1
+; CHECK-NEXT:    sshll.8h v3, v3, #0
+; CHECK-NEXT:    sshll2.4s v16, v6, #0
 ; CHECK-NEXT:    sshll.4s v6, v6, #0
-; CHECK-NEXT:    and.16b v7, v17, v7
-; CHECK-NEXT:    and.16b v5, v5, v6
-; CHECK-NEXT:    stp q5, q7, [x1], #64
+; CHECK-NEXT:    zip1.8h v7, v7, v1
+; CHECK-NEXT:    zip1.8h v5, v5, v1
+; CHECK-NEXT:    zip1.8h v2, v2, v1
+; CHECK-NEXT:    and.16b v4, v4, v6
+; CHECK-NEXT:    sshll2.4s v6, v3, #0
+; CHECK-NEXT:    sshll.4s v3, v3, #0
+; CHECK-NEXT:    and.16b v7, v7, v16
+; CHECK-NEXT:    and.16b v2, v2, v6
+; CHECK-NEXT:    stp q4, q7, [x1, #32]
+; CHECK-NEXT:    and.16b v3, v5, v3
+; CHECK-NEXT:    stp q3, q2, [x1], #64
 ; CHECK-NEXT:    b.ne LBB24_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh5, Lloh9
-; CHECK-NEXT:    .loh AdrpLdr Lloh4, Lloh8
-; CHECK-NEXT:    .loh AdrpLdr Lloh3, Lloh7
-; CHECK-NEXT:    .loh AdrpLdr Lloh2, Lloh6
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, ptr %src, i64 %iv
   %load = load <16 x i8>, ptr %src.gep
   %cmp = icmp sgt <16 x i8> %load,  <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
   %ext = zext <16 x i8> %load to <16 x i32>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   %dst.gep = getelementptr i32, ptr %dst, i64 %iv
   store <16 x i32> %sel, ptr %dst.gep
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
 define void @extension_in_loop_as_shuffle_v16i8_to_v16i32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: extension_in_loop_as_shuffle_v16i8_to_v16i32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh10:
+; CHECK-NEXT:  Lloh2:
 ; CHECK-NEXT:    adrp x9, lCPI25_0@PAGE
-; CHECK-NEXT:  Lloh11:
+; CHECK-NEXT:  Lloh3:
 ; CHECK-NEXT:    adrp x10, lCPI25_1@PAGE
-; CHECK-NEXT:  Lloh12:
+; CHECK-NEXT:  Lloh4:
 ; CHECK-NEXT:    adrp x11, lCPI25_2@PAGE
-; CHECK-NEXT:  Lloh13:
+; CHECK-NEXT:  Lloh5:
 ; CHECK-NEXT:    adrp x12, lCPI25_3@PAGE
 ; CHECK-NEXT:    movi.2d v2, #0xffffffffffffffff
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh14:
+; CHECK-NEXT:  Lloh6:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI25_0@PAGEOFF]
-; CHECK-NEXT:  Lloh15:
+; CHECK-NEXT:  Lloh7:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI25_1@PAGEOFF]
-; CHECK-NEXT:  Lloh16:
+; CHECK-NEXT:  Lloh8:
 ; CHECK-NEXT:    ldr q3, [x11, lCPI25_2@PAGEOFF]
-; CHECK-NEXT:  Lloh17:
+; CHECK-NEXT:  Lloh9:
 ; CHECK-NEXT:    ldr q4, [x12, lCPI25_3@PAGEOFF]
 ; CHECK-NEXT:  LBB25_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr q5, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    cmgt.16b v6, v5, v2
 ; CHECK-NEXT:    tbl.16b v7, { v5 }, v0

 ; CHECK-NEXT:    sshll2.4s v7, v6, #0
 ; CHECK-NEXT:    sshll.4s v6, v6, #0
 ; CHECK-NEXT:    and.16b v7, v17, v7
 ; CHECK-NEXT:    and.16b v5, v5, v6
 ; CHECK-NEXT:    stp q5, q7, [x1], #64
 ; CHECK-NEXT:    b.ne LBB25_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh13, Lloh17
-; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh16
-; CHECK-NEXT:    .loh AdrpLdr Lloh11, Lloh15
-; CHECK-NEXT:    .loh AdrpLdr Lloh10, Lloh14
+; CHECK-NEXT:    .loh AdrpLdr Lloh5, Lloh9
+; CHECK-NEXT:    .loh AdrpLdr Lloh4, Lloh8
+; CHECK-NEXT:    .loh AdrpLdr Lloh3, Lloh7
+; CHECK-NEXT:    .loh AdrpLdr Lloh2, Lloh6
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, ptr %src, i64 %iv
   %load = load <16 x i8>, ptr %src.gep
   %cmp = icmp sgt <16 x i8> %load,  <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>
   %ext.shuf = shufflevector <16 x i8> %load, <16 x i8> zeroinitializer, <64 x i32> <i32 16, i32 16, i32 16, i32 0, i32 16, i32 16, i32 16, i32 1, i32 16, i32 16, i32 16, i32 2, i32 16, i32 16, i32 16, i32 3, i32 16, i32 16, i32 16, i32 4, i32 16, i32 16, i32 16, i32 5, i32 16, i32 16, i32 16, i32 6, i32 16, i32 16, i32 16, i32 7, i32 16, i32 16, i32 16, i32 8, i32 16, i32 16, i32 16, i32 9, i32 16, i32 16, i32 16, i32 10, i32 16, i32 16, i32 16, i32 11, i32 16, i32 16, i32 16, i32 12, i32 16, i32 16, i32 16, i32 13, i32 16, i32 16, i32 16, i32 14, i32 16, i32 16, i32 16, i32 15>
   %ext = bitcast <64 x i8> %ext.shuf to <16 x i32>
   %sel = select <16 x i1> %cmp, <16 x i32> %ext, <16 x i32> zeroinitializer
   %dst.gep = getelementptr i32, ptr %dst, i64 %iv
   store <16 x i32> %sel, ptr %dst.gep
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
 define void @shuffle_in_loop_is_no_extend_v16i8_to_v16i32(ptr %src, ptr %dst) {
 ; CHECK-LABEL: shuffle_in_loop_is_no_extend_v16i8_to_v16i32:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh18:
+; CHECK-NEXT:  Lloh10:
 ; CHECK-NEXT:    adrp x9, lCPI26_0@PAGE
-; CHECK-NEXT:  Lloh19:
+; CHECK-NEXT:  Lloh11:
 ; CHECK-NEXT:    adrp x10, lCPI26_1@PAGE
-; CHECK-NEXT:  Lloh20:
+; CHECK-NEXT:  Lloh12:
 ; CHECK-NEXT:    adrp x11, lCPI26_2@PAGE
-; CHECK-NEXT:  Lloh21:
+; CHECK-NEXT:  Lloh13:
 ; CHECK-NEXT:    adrp x12, lCPI26_3@PAGE
 ; CHECK-NEXT:    movi.2d v2, #0xffffffffffffffff
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh22:
+; CHECK-NEXT:  Lloh14:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI26_0@PAGEOFF]
-; CHECK-NEXT:  Lloh23:
+; CHECK-NEXT:  Lloh15:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI26_1@PAGEOFF]
-; CHECK-NEXT:  Lloh24:
+; CHECK-NEXT:  Lloh16:
 ; CHECK-NEXT:    ldr q3, [x11, lCPI26_2@PAGEOFF]
-; CHECK-NEXT:  Lloh25:
+; CHECK-NEXT:  Lloh17:
 ; CHECK-NEXT:    ldr q4, [x12, lCPI26_3@PAGEOFF]
 ; CHECK-NEXT:  LBB26_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr q5, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    cmgt.16b v6, v5, v2
 ; CHECK-NEXT:    tbl.16b v7, { v5 }, v0

 ; CHECK-NEXT:    sshll2.4s v7, v6, #0
 ; CHECK-NEXT:    sshll.4s v6, v6, #0
 ; CHECK-NEXT:    and.16b v7, v17, v7
 ; CHECK-NEXT:    and.16b v5, v5, v6
 ; CHECK-NEXT:    stp q5, q7, [x1], #64
 ; CHECK-NEXT:    b.ne LBB26_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh21, Lloh25
-; CHECK-NEXT:    .loh AdrpLdr Lloh20, Lloh24
-; CHECK-NEXT:    .loh AdrpLdr Lloh19, Lloh23
-; CHECK-NEXT:    .loh AdrpLdr Lloh18, Lloh22
+; CHECK-NEXT:    .loh AdrpLdr Lloh13, Lloh17
+; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh16
+; CHECK-NEXT:    .loh AdrpLdr Lloh11, Lloh15
+; CHECK-NEXT:    .loh AdrpLdr Lloh10, Lloh14
 entry:
   br label %loop
 
 loop:
   %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
   %src.gep = getelementptr i8, ptr %src, i64 %iv
   %load = load <16 x i8>, ptr %src.gep
   %cmp = icmp sgt <16 x i8> %load,  <i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1, i8 -1>

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -mtriple=arm64-apple-ios -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI %s
 ; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI --check-prefix=CHECK-BE %s
 ; RUN: llc -mtriple=arm64-apple-ios -mattr=+global-isel -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI %s
 ; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+global-isel -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI --check-prefix=CHECK-BE %s
 
-; CHECK-LABEL: lCPI0_0:
-; CHECK-NEXT:    .byte   0                               ; 0x0
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   1                               ; 0x1
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   2                               ; 0x2
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   3                               ; 0x3
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:lCPI0_1:
-; CHECK-NEXT:    .byte   4                               ; 0x4
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   5                               ; 0x5
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   6                               ; 0x6
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   7                               ; 0x7
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:lCPI0_2:
-; CHECK-NEXT:    .byte   8                               ; 0x8
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   9                               ; 0x9
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   10                              ; 0xa
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   11                              ; 0xb
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:lCPI0_3:
-; CHECK-NEXT:    .byte   12                              ; 0xc
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   13                              ; 0xd
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   14                              ; 0xe
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   15                              ; 0xf
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-; CHECK-NEXT:    .byte   255                             ; 0xff
-
 ; CHECK-BE: .LCPI0_0:
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 0                               // 0x0
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff

 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 15                              // 0xf
 
 ; It's profitable to convert the zext to a shuffle, which in turn will be
 ; lowered to 4 tbl instructions. The masks are materialized outside the loop.
 define void @zext_v16i8_to_v16i32_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v16i8_to_v16i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh0:
-; CHECK-NEXT:    adrp x9, lCPI0_0@PAGE
-; CHECK-NEXT:  Lloh1:
-; CHECK-NEXT:    adrp x10, lCPI0_1@PAGE
-; CHECK-NEXT:  Lloh2:
-; CHECK-NEXT:    adrp x11, lCPI0_2@PAGE
-; CHECK-NEXT:  Lloh3:
-; CHECK-NEXT:    adrp x12, lCPI0_3@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh4:
-; CHECK-NEXT:    ldr q0, [x9, lCPI0_0@PAGEOFF]
-; CHECK-NEXT:  Lloh5:
-; CHECK-NEXT:    ldr q1, [x10, lCPI0_1@PAGEOFF]
-; CHECK-NEXT:  Lloh6:
-; CHECK-NEXT:    ldr q2, [x11, lCPI0_2@PAGEOFF]
-; CHECK-NEXT:  Lloh7:
-; CHECK-NEXT:    ldr q3, [x12, lCPI0_3@PAGEOFF]
 ; CHECK-NEXT:  LBB0_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q4, [x0, x8]
+; CHECK-NEXT:    ldr q1, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    tbl.16b v5, { v4 }, v3
-; CHECK-NEXT:    tbl.16b v6, { v4 }, v2
-; CHECK-NEXT:    tbl.16b v7, { v4 }, v1
-; CHECK-NEXT:    tbl.16b v4, { v4 }, v0
-; CHECK-NEXT:    stp q6, q5, [x1, #32]
-; CHECK-NEXT:    stp q4, q7, [x1], #64
+; CHECK-NEXT:    dup.2d v2, v1[1]
+; CHECK-NEXT:    dup.4s v3, v1[1]
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    dup.4s v4, v2[1]
+; CHECK-NEXT:    zip1.16b v3, v3, v0
+; CHECK-NEXT:    zip1.16b v2, v2, v0
+; CHECK-NEXT:    zip1.16b v4, v4, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
+; CHECK-NEXT:    zip1.8h v3, v3, v0
+; CHECK-NEXT:    zip1.8h v2, v2, v0
+; CHECK-NEXT:    zip1.8h v4, v4, v0
+; CHECK-NEXT:    stp q1, q3, [x1]
+; CHECK-NEXT:    stp q2, q4, [x1, #32]
+; CHECK-NEXT:    add x1, x1, #64
 ; CHECK-NEXT:    b.ne LBB0_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh3, Lloh7
-; CHECK-NEXT:    .loh AdrpLdr Lloh2, Lloh6
-; CHECK-NEXT:    .loh AdrpLdr Lloh1, Lloh5
-; CHECK-NEXT:    .loh AdrpLdr Lloh0, Lloh4
 ;
 ; CHECK-BE-LABEL: zext_v16i8_to_v16i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI0_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI0_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI0_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI0_1
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
-; CHECK-LABEL: lCPI6_0:
-; CHECK-NEXT:     .byte   0                               ; 0x0
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   1                               ; 0x1
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   2                               ; 0x2
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   3                               ; 0x3
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT: lCPI6_1:
-; CHECK-NEXT:     .byte   4                               ; 0x4
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   5                               ; 0x5
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   6                               ; 0x6
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   7                               ; 0x7
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-; CHECK-NEXT:     .byte   255                             ; 0xff
-
 ; CHECK-BE:       .LCPI6_0:
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 0                               // 0x0
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff

 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 7                               // 0x7
 
 define void @zext_v8i8_to_v8i32_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v8i8_to_v8i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh8:
-; CHECK-NEXT:    adrp x9, lCPI6_0@PAGE
-; CHECK-NEXT:  Lloh9:
-; CHECK-NEXT:    adrp x10, lCPI6_1@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh10:
-; CHECK-NEXT:    ldr q0, [x9, lCPI6_0@PAGEOFF]
-; CHECK-NEXT:  Lloh11:
-; CHECK-NEXT:    ldr q1, [x10, lCPI6_1@PAGEOFF]
 ; CHECK-NEXT:  LBB6_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr d2, [x0, x8]
+; CHECK-NEXT:    ldr d1, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    tbl.16b v3, { v2 }, v1
-; CHECK-NEXT:    tbl.16b v2, { v2 }, v0
-; CHECK-NEXT:    stp q2, q3, [x1], #64
+; CHECK-NEXT:    dup.2s v2, v1[1]
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    zip1.16b v2, v2, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
+; CHECK-NEXT:    zip1.8h v2, v2, v0
+; CHECK-NEXT:    stp q1, q2, [x1], #64
 ; CHECK-NEXT:    b.ne LBB6_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh9, Lloh11
-; CHECK-NEXT:    .loh AdrpLdr Lloh8, Lloh10
 ;
 ; CHECK-BE-LABEL: zext_v8i8_to_v8i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI6_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI6_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI6_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI6_1
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
-; CHECK-LABEL: lCPI12_0:
-; CHECK-NEXT:   .byte 0                               ; 0x0
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 1                               ; 0x1
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 2                               ; 0x2
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 3                               ; 0x3
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT: lCPI12_1:
-; CHECK-NEXT:   .byte 4                               ; 0x4
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 5                               ; 0x5
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 6                               ; 0x6
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 7                               ; 0x7
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT: lCPI12_2:
-; CHECK-NEXT:   .byte 8                               ; 0x8
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 9                               ; 0x9
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 10                              ; 0xa
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 11                              ; 0xb
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-
 ; CHECK-BE-LABEL: .LCPI12_0:
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 0                               // 0x0
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff

 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 11                              // 0xb
 
 define void @zext_v12i8_to_v12i32_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v12i8_to_v12i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh12:
-; CHECK-NEXT:    adrp x9, lCPI12_0@PAGE
-; CHECK-NEXT:  Lloh13:
-; CHECK-NEXT:    adrp x10, lCPI12_1@PAGE
-; CHECK-NEXT:  Lloh14:
-; CHECK-NEXT:    adrp x11, lCPI12_2@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh15:
-; CHECK-NEXT:    ldr q0, [x9, lCPI12_0@PAGEOFF]
-; CHECK-NEXT:  Lloh16:
-; CHECK-NEXT:    ldr q1, [x10, lCPI12_1@PAGEOFF]
-; CHECK-NEXT:  Lloh17:
-; CHECK-NEXT:    ldr q2, [x11, lCPI12_2@PAGEOFF]
 ; CHECK-NEXT:  LBB12_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
-; CHECK-NEXT:    ldr q3, [x0, x8]
+; CHECK-NEXT:    ldr q1, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    tbl.16b v4, { v3 }, v2
-; CHECK-NEXT:    tbl.16b v5, { v3 }, v1
-; CHECK-NEXT:    tbl.16b v3, { v3 }, v0
-; CHECK-NEXT:    stp q5, q4, [x1, #16]
-; CHECK-NEXT:    str q3, [x1], #64
+; CHECK-NEXT:    dup.2d v2, v1[1]
+; CHECK-NEXT:    dup.4s v3, v1[1]
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    zip1.16b v2, v2, v0
+; CHECK-NEXT:    zip1.16b v3, v3, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
+; CHECK-NEXT:    zip1.8h v2, v2, v0
+; CHECK-NEXT:    zip1.8h v3, v3, v0
+; CHECK-NEXT:    stp q3, q2, [x1, #16]
+; CHECK-NEXT:    str q1, [x1], #64
 ; CHECK-NEXT:    b.ne LBB12_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh14, Lloh17
-; CHECK-NEXT:    .loh AdrpLdr Lloh13, Lloh16
-; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh15
 ;
 ; CHECK-BE-LABEL: zext_v12i8_to_v12i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI12_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI12_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_1

 ; CHECK-BE-NEXT:    .byte 14                              // 0xe
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 15                              // 0xf
 
 define void @zext_v20i8_to_v20i24_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v20i8_to_v20i24_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh18:
+; CHECK-NEXT:  Lloh0:
 ; CHECK-NEXT:    adrp x9, lCPI20_0@PAGE
-; CHECK-NEXT:  Lloh19:
+; CHECK-NEXT:  Lloh1:
 ; CHECK-NEXT:    adrp x10, lCPI20_1@PAGE
-; CHECK-NEXT:  Lloh20:
+; CHECK-NEXT:  Lloh2:
 ; CHECK-NEXT:    adrp x11, lCPI20_2@PAGE
-; CHECK-NEXT:  Lloh21:
+; CHECK-NEXT:  Lloh3:
 ; CHECK-NEXT:    adrp x12, lCPI20_3@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh22:
+; CHECK-NEXT:  Lloh4:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI20_0@PAGEOFF]
-; CHECK-NEXT:  Lloh23:
+; CHECK-NEXT:  Lloh5:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI20_1@PAGEOFF]
-; CHECK-NEXT:  Lloh24:
+; CHECK-NEXT:  Lloh6:
 ; CHECK-NEXT:    ldr q2, [x11, lCPI20_2@PAGEOFF]
-; CHECK-NEXT:  Lloh25:
+; CHECK-NEXT:  Lloh7:
 ; CHECK-NEXT:    ldr q3, [x12, lCPI20_3@PAGEOFF]
 ; CHECK-NEXT:  LBB20_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    add x9, x0, x8
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    ldp q4, q5, [x9]
 ; CHECK-NEXT:    add x9, x1, #56
 ; CHECK-NEXT:    tbl.16b v6, { v4 }, v2
 ; CHECK-NEXT:    tbl.16b v7, { v4 }, v1
 ; CHECK-NEXT:    tbl.16b v4, { v4 }, v0
 ; CHECK-NEXT:    tbl.16b v5, { v5 }, v3
 ; CHECK-NEXT:    stp q7, q6, [x1, #16]
 ; CHECK-NEXT:    str q4, [x1]
 ; CHECK-NEXT:    str d5, [x1, #48]
 ; CHECK-NEXT:    add x1, x1, #64
 ; CHECK-NEXT:    st1.s { v5 }[2], [x9]
 ; CHECK-NEXT:    b.ne LBB20_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh21, Lloh25
-; CHECK-NEXT:    .loh AdrpLdr Lloh20, Lloh24
-; CHECK-NEXT:    .loh AdrpLdr Lloh19, Lloh23
-; CHECK-NEXT:    .loh AdrpLdr Lloh18, Lloh22
+; CHECK-NEXT:    .loh AdrpLdr Lloh3, Lloh7
+; CHECK-NEXT:    .loh AdrpLdr Lloh2, Lloh6
+; CHECK-NEXT:    .loh AdrpLdr Lloh1, Lloh5
+; CHECK-NEXT:    .loh AdrpLdr Lloh0, Lloh4
 ;
 ; CHECK-BE-LABEL: zext_v20i8_to_v20i24_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI20_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI20_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI20_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI20_1

 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 15                              // 0xf
 
 define void @zext_v23i8_to_v23i48_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v23i8_to_v23i48_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh26:
+; CHECK-NEXT:  Lloh8:
 ; CHECK-NEXT:    adrp x9, lCPI21_0@PAGE
-; CHECK-NEXT:  Lloh27:
+; CHECK-NEXT:  Lloh9:
 ; CHECK-NEXT:    adrp x10, lCPI21_1@PAGE
-; CHECK-NEXT:  Lloh28:
+; CHECK-NEXT:  Lloh10:
 ; CHECK-NEXT:    adrp x11, lCPI21_2@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh29:
+; CHECK-NEXT:  Lloh11:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI21_0@PAGEOFF]
-; CHECK-NEXT:  Lloh30:
+; CHECK-NEXT:  Lloh12:
 ; CHECK-NEXT:    adrp x9, lCPI21_3@PAGE
-; CHECK-NEXT:  Lloh31:
+; CHECK-NEXT:  Lloh13:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI21_1@PAGEOFF]
-; CHECK-NEXT:  Lloh32:
+; CHECK-NEXT:  Lloh14:
 ; CHECK-NEXT:    adrp x10, lCPI21_4@PAGE
-; CHECK-NEXT:  Lloh33:
+; CHECK-NEXT:  Lloh15:
 ; CHECK-NEXT:    ldr q2, [x11, lCPI21_2@PAGEOFF]
-; CHECK-NEXT:  Lloh34:
+; CHECK-NEXT:  Lloh16:
 ; CHECK-NEXT:    adrp x11, lCPI21_5@PAGE
-; CHECK-NEXT:  Lloh35:
+; CHECK-NEXT:  Lloh17:
 ; CHECK-NEXT:    ldr q3, [x9, lCPI21_3@PAGEOFF]
-; CHECK-NEXT:  Lloh36:
+; CHECK-NEXT:  Lloh18:
 ; CHECK-NEXT:    ldr q4, [x10, lCPI21_4@PAGEOFF]
-; CHECK-NEXT:  Lloh37:
+; CHECK-NEXT:  Lloh19:
 ; CHECK-NEXT:    ldr q5, [x11, lCPI21_5@PAGEOFF]
 ; CHECK-NEXT:  LBB21_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    add x9, x0, x8
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    movi.2d v6, #0000000000000000
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    ldp q16, q7, [x9]

 ; CHECK-NEXT:    tbl.16b v7, { v16 }, v1
 ; CHECK-NEXT:    tbl.16b v16, { v16 }, v0
 ; CHECK-NEXT:    stp q17, q20, [x1, #32]
 ; CHECK-NEXT:    stp q16, q7, [x1]
 ; CHECK-NEXT:    str x9, [x1, #128]!
 ; CHECK-NEXT:    b.ne LBB21_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh34, Lloh37
-; CHECK-NEXT:    .loh AdrpLdr Lloh32, Lloh36
-; CHECK-NEXT:    .loh AdrpLdr Lloh30, Lloh35
-; CHECK-NEXT:    .loh AdrpAdrp Lloh28, Lloh34
-; CHECK-NEXT:    .loh AdrpLdr Lloh28, Lloh33
-; CHECK-NEXT:    .loh AdrpAdrp Lloh27, Lloh32
-; CHECK-NEXT:    .loh AdrpLdr Lloh27, Lloh31
-; CHECK-NEXT:    .loh AdrpAdrp Lloh26, Lloh30
-; CHECK-NEXT:    .loh AdrpLdr Lloh26, Lloh29
+; CHECK-NEXT:    .loh AdrpLdr Lloh16, Lloh19
+; CHECK-NEXT:    .loh AdrpLdr Lloh14, Lloh18
+; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh17
+; CHECK-NEXT:    .loh AdrpAdrp Lloh10, Lloh16
+; CHECK-NEXT:    .loh AdrpLdr Lloh10, Lloh15
+; CHECK-NEXT:    .loh AdrpAdrp Lloh9, Lloh14
+; CHECK-NEXT:    .loh AdrpLdr Lloh9, Lloh13
+; CHECK-NEXT:    .loh AdrpAdrp Lloh8, Lloh12
+; CHECK-NEXT:    .loh AdrpLdr Lloh8, Lloh11
 ;
 ; CHECK-BE-LABEL: zext_v23i8_to_v23i48_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI21_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI21_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI21_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI21_1