[AArch64] `LowerZERO_EXTEND_VECTOR_INREG()`: recursively apply `zip1` until done

[lebedev.ri]

Summary

While the one-step zip1 lowering seemed obviously good,
here i feel like i should double-check: is 2 or more zip1
not worse than a tbl?
I'm guessing so, because it avoids constant pool load.

This comes up in a follow-up change to combineShuffleToZeroExtendVectorInReg().

[fhahn]

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

I think that for this lowering is slightly worse in general for CPUs that have efficient implementations of tbl, as tbl results in shorter dependency chains than having 2 zip1, with one depending on the other. The tbl lowering is only used in loops, when the load from the constant pool is hoisted outside the loop.

[lebedev.ri]

@fhahn IOW, this change is not an improvement,
and reverse changes are not regressions?

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

Note: i personally do not care what happens here, i'm only asking because this comes up
as a "regression" in further patches (D140677 e.g.), so i'm wondering if this needs to be dealt with.

[fhahn]

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

Yeah, IMO this is a regression that should be avoided (at least in loops)

[lebedev.ri]

@fhahn thank you for taking a look!

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

Err, i'm still confused. You are saying this patch is not needed, correct? :)

[fhahn]

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

Yep, it should not be needed.

[lebedev.ri]

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

1025

Cheers, thank you.

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

AArch64TargetLowering::LowerZERO_EXTEND_VECTOR_INREG(SDValue Op,
                                                      SelectionDAG &DAG) const {
   SDLoc dl(Op);
   EVT VT = Op.getValueType();
   SDValue SrcOp = Op.getOperand(0);
   EVT SrcVT = SrcOp.getValueType();
   assert(VT.getScalarSizeInBits() % SrcVT.getScalarSizeInBits() == 0 &&
          "Unexpected extension factor.");
   unsigned Scale = VT.getScalarSizeInBits() / SrcVT.getScalarSizeInBits();
-  // FIXME: support multi-step zipping?
-  if (Scale != 2)
+  // Only support simple power-of-2 extensions.
+  if (!isPowerOf2_32(Scale))
     return SDValue();
+  for (unsigned Step = 0, NumSteps = Log2_32(Scale); Step != NumSteps; ++Step) {
     SDValue Zeros = DAG.getConstant(0, dl, SrcVT);
-  return DAG.getBitcast(VT,
-                        DAG.getNode(AArch64ISD::ZIP1, dl, SrcVT, SrcOp, Zeros));
+    EVT NewVT = EVT::getVectorVT(
+        *DAG.getContext(),
+        EVT::getIntegerVT(*DAG.getContext(), 2 * SrcVT.getScalarSizeInBits()),
+        SrcVT.getVectorNumElements() / 2);
+    SrcOp = DAG.getNode(AArch64ISD::ZIP1, dl, SrcVT, SrcOp, Zeros);
+    SrcOp = DAG.getBitcast(NewVT, SrcOp);
+    SrcVT = NewVT;
+  }
+  return SrcOp;
 }
 
 SDValue AArch64TargetLowering::LowerVECTOR_SHUFFLE(SDValue Op,
                                                    SelectionDAG &DAG) const {
   SDLoc dl(Op);
   EVT VT = Op.getValueType();
 
   ShuffleVectorSDNode *SVN = cast<ShuffleVectorSDNode>(Op.getNode());

llvm/test/CodeGen/AArch64/zext-to-tbl.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=arm64-apple-ios -mattr=+sve -o - %s | FileCheck %s
-; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+sve -o - %s | FileCheck --check-prefix=CHECK-BE %s
-; RUN: llc -mtriple=arm64-apple-ios -mattr=+global-isel -mattr=+sve -o - %s | FileCheck %s
-; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+global-isel -mattr=+sve -o - %s | FileCheck --check-prefix=CHECK-BE %s
+; RUN: llc -mtriple=arm64-apple-ios -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI %s
+; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI --check-prefix=CHECK-BE %s
+; RUN: llc -mtriple=arm64-apple-ios -mattr=+global-isel -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI %s
+; RUN: llc -mtriple=aarch64_be-unknown-linux -mattr=+global-isel -mattr=+sve -o - %s | FileCheck --implicit-check-not=LCPI --implicit-check-not=lCPI --check-prefix=CHECK-BE %s
 
 ; CHECK-LABEL: lCPI0_0:
 ; CHECK-NEXT:    .byte   0                               ; 0x0
 ; CHECK-NEXT:    .byte   255                             ; 0xff
 ; CHECK-NEXT:    .byte   255                             ; 0xff
 ; CHECK-NEXT:    .byte   255                             ; 0xff
 ; CHECK-NEXT:    .byte   1                               ; 0x1
 ; CHECK-NEXT:    .byte   255                             ; 0xff
loop:
   %iv.next = add nuw i64 %iv, 16
   %ec = icmp eq i64 %iv.next, 128
   br i1 %ec, label %exit, label %loop
 
 exit:
   ret void
 }
 
-; CHECK-LABEL: lCPI11_0:
-; CHECK-NEXT:   .byte 0                               ; 0x0
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 1                               ; 0x1
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 2                               ; 0x2
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 3                               ; 0x3
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-; CHECK-NEXT:   .byte 255                             ; 0xff
-
 ; CHECK-BE-LABEL:  .LCPI11_0:
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 0                               // 0x0
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 1                               // 0x1
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 2                               // 0x2
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 3                               // 0x3
 
 define void @zext_v4i8_to_v4i32_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v4i8_to_v4i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh12:
-; CHECK-NEXT:    adrp x9, lCPI11_0@PAGE
+; CHECK-NEXT:    movi.2d v0, #0000000000000000
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh13:
-; CHECK-NEXT:    ldr q0, [x9, lCPI11_0@PAGEOFF]
 ; CHECK-NEXT:  LBB11_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr s1, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
-; CHECK-NEXT:    tbl.16b v1, { v1 }, v0
+; CHECK-NEXT:    zip1.16b v1, v1, v0
+; CHECK-NEXT:    zip1.8h v1, v1, v0
 ; CHECK-NEXT:    str q1, [x1], #64
 ; CHECK-NEXT:    b.ne LBB11_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh13
 ;
 ; CHECK-BE-LABEL: zext_v4i8_to_v4i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI11_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI11_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    mov x8, xzr
 ; CHECK-BE-NEXT:  .LBB11_1: // %loop

 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 255                             // 0xff
 ; CHECK-BE-NEXT:  .byte 11                              // 0xb
 
 define void @zext_v12i8_to_v12i32_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v12i8_to_v12i32_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh14:
+; CHECK-NEXT:  Lloh12:
 ; CHECK-NEXT:    adrp x9, lCPI12_0@PAGE
-; CHECK-NEXT:  Lloh15:
+; CHECK-NEXT:  Lloh13:
 ; CHECK-NEXT:    adrp x10, lCPI12_1@PAGE
-; CHECK-NEXT:  Lloh16:
+; CHECK-NEXT:  Lloh14:
 ; CHECK-NEXT:    adrp x11, lCPI12_2@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh17:
+; CHECK-NEXT:  Lloh15:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI12_0@PAGEOFF]
-; CHECK-NEXT:  Lloh18:
+; CHECK-NEXT:  Lloh16:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI12_1@PAGEOFF]
-; CHECK-NEXT:  Lloh19:
+; CHECK-NEXT:  Lloh17:
 ; CHECK-NEXT:    ldr q2, [x11, lCPI12_2@PAGEOFF]
 ; CHECK-NEXT:  LBB12_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    ldr q3, [x0, x8]
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    tbl.16b v4, { v3 }, v2
 ; CHECK-NEXT:    tbl.16b v5, { v3 }, v1
 ; CHECK-NEXT:    tbl.16b v3, { v3 }, v0
 ; CHECK-NEXT:    stp q5, q4, [x1, #16]
 ; CHECK-NEXT:    str q3, [x1], #64
 ; CHECK-NEXT:    b.ne LBB12_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh16, Lloh19
-; CHECK-NEXT:    .loh AdrpLdr Lloh15, Lloh18
 ; CHECK-NEXT:    .loh AdrpLdr Lloh14, Lloh17
+; CHECK-NEXT:    .loh AdrpLdr Lloh13, Lloh16
+; CHECK-NEXT:    .loh AdrpLdr Lloh12, Lloh15
 ;
 ; CHECK-BE-LABEL: zext_v12i8_to_v12i32_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI12_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI12_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI12_1

 ; CHECK-BE-NEXT:    .byte 14                              // 0xe
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 15                              // 0xf
 
 define void @zext_v20i8_to_v20i24_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v20i8_to_v20i24_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh20:
+; CHECK-NEXT:  Lloh18:
 ; CHECK-NEXT:    adrp x9, lCPI20_0@PAGE
-; CHECK-NEXT:  Lloh21:
+; CHECK-NEXT:  Lloh19:
 ; CHECK-NEXT:    adrp x10, lCPI20_1@PAGE
-; CHECK-NEXT:  Lloh22:
+; CHECK-NEXT:  Lloh20:
 ; CHECK-NEXT:    adrp x11, lCPI20_2@PAGE
-; CHECK-NEXT:  Lloh23:
+; CHECK-NEXT:  Lloh21:
 ; CHECK-NEXT:    adrp x12, lCPI20_3@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh24:
+; CHECK-NEXT:  Lloh22:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI20_0@PAGEOFF]
-; CHECK-NEXT:  Lloh25:
+; CHECK-NEXT:  Lloh23:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI20_1@PAGEOFF]
-; CHECK-NEXT:  Lloh26:
+; CHECK-NEXT:  Lloh24:
 ; CHECK-NEXT:    ldr q2, [x11, lCPI20_2@PAGEOFF]
-; CHECK-NEXT:  Lloh27:
+; CHECK-NEXT:  Lloh25:
 ; CHECK-NEXT:    ldr q3, [x12, lCPI20_3@PAGEOFF]
 ; CHECK-NEXT:  LBB20_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    add x9, x0, x8
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    ldp q4, q5, [x9]
 ; CHECK-NEXT:    add x9, x1, #56
 ; CHECK-NEXT:    tbl.16b v6, { v4 }, v2
 ; CHECK-NEXT:    tbl.16b v7, { v4 }, v1
 ; CHECK-NEXT:    tbl.16b v4, { v4 }, v0
 ; CHECK-NEXT:    tbl.16b v5, { v5 }, v3
 ; CHECK-NEXT:    stp q7, q6, [x1, #16]
 ; CHECK-NEXT:    str q4, [x1]
 ; CHECK-NEXT:    str d5, [x1, #48]
 ; CHECK-NEXT:    add x1, x1, #64
 ; CHECK-NEXT:    st1.s { v5 }[2], [x9]
 ; CHECK-NEXT:    b.ne LBB20_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh23, Lloh27
-; CHECK-NEXT:    .loh AdrpLdr Lloh22, Lloh26
 ; CHECK-NEXT:    .loh AdrpLdr Lloh21, Lloh25
 ; CHECK-NEXT:    .loh AdrpLdr Lloh20, Lloh24
+; CHECK-NEXT:    .loh AdrpLdr Lloh19, Lloh23
+; CHECK-NEXT:    .loh AdrpLdr Lloh18, Lloh22
 ;
 ; CHECK-BE-LABEL: zext_v20i8_to_v20i24_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI20_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI20_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI20_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI20_1

 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 255                             // 0xff
 ; CHECK-BE-NEXT:    .byte 15                              // 0xf
 
 define void @zext_v23i8_to_v23i48_in_loop(ptr %src, ptr %dst) {
 ; CHECK-LABEL: zext_v23i8_to_v23i48_in_loop:
 ; CHECK:       ; %bb.0: ; %entry
-; CHECK-NEXT:  Lloh28:
+; CHECK-NEXT:  Lloh26:
 ; CHECK-NEXT:    adrp x9, lCPI21_0@PAGE
-; CHECK-NEXT:  Lloh29:
+; CHECK-NEXT:  Lloh27:
 ; CHECK-NEXT:    adrp x10, lCPI21_1@PAGE
-; CHECK-NEXT:  Lloh30:
+; CHECK-NEXT:  Lloh28:
 ; CHECK-NEXT:    adrp x11, lCPI21_2@PAGE
 ; CHECK-NEXT:    mov x8, xzr
-; CHECK-NEXT:  Lloh31:
+; CHECK-NEXT:  Lloh29:
 ; CHECK-NEXT:    ldr q0, [x9, lCPI21_0@PAGEOFF]
-; CHECK-NEXT:  Lloh32:
+; CHECK-NEXT:  Lloh30:
 ; CHECK-NEXT:    adrp x9, lCPI21_3@PAGE
-; CHECK-NEXT:  Lloh33:
+; CHECK-NEXT:  Lloh31:
 ; CHECK-NEXT:    ldr q1, [x10, lCPI21_1@PAGEOFF]
-; CHECK-NEXT:  Lloh34:
+; CHECK-NEXT:  Lloh32:
 ; CHECK-NEXT:    adrp x10, lCPI21_4@PAGE
-; CHECK-NEXT:  Lloh35:
+; CHECK-NEXT:  Lloh33:
 ; CHECK-NEXT:    ldr q2, [x11, lCPI21_2@PAGEOFF]
-; CHECK-NEXT:  Lloh36:
+; CHECK-NEXT:  Lloh34:
 ; CHECK-NEXT:    adrp x11, lCPI21_5@PAGE
-; CHECK-NEXT:  Lloh37:
+; CHECK-NEXT:  Lloh35:
 ; CHECK-NEXT:    ldr q3, [x9, lCPI21_3@PAGEOFF]
-; CHECK-NEXT:  Lloh38:
+; CHECK-NEXT:  Lloh36:
 ; CHECK-NEXT:    ldr q4, [x10, lCPI21_4@PAGEOFF]
-; CHECK-NEXT:  Lloh39:
+; CHECK-NEXT:  Lloh37:
 ; CHECK-NEXT:    ldr q5, [x11, lCPI21_5@PAGEOFF]
 ; CHECK-NEXT:  LBB21_1: ; %loop
 ; CHECK-NEXT:    ; =>This Inner Loop Header: Depth=1
 ; CHECK-NEXT:    add x9, x0, x8
 ; CHECK-NEXT:    add x8, x8, #16
 ; CHECK-NEXT:    movi.2d v6, #0000000000000000
 ; CHECK-NEXT:    cmp x8, #128
 ; CHECK-NEXT:    ldp q16, q7, [x9]

 ; CHECK-NEXT:    tbl.16b v7, { v16 }, v1
 ; CHECK-NEXT:    tbl.16b v16, { v16 }, v0
 ; CHECK-NEXT:    stp q17, q20, [x1, #32]
 ; CHECK-NEXT:    stp q16, q7, [x1]
 ; CHECK-NEXT:    str x9, [x1, #128]!
 ; CHECK-NEXT:    b.ne LBB21_1
 ; CHECK-NEXT:  ; %bb.2: ; %exit
 ; CHECK-NEXT:    ret
-; CHECK-NEXT:    .loh AdrpLdr Lloh36, Lloh39
-; CHECK-NEXT:    .loh AdrpLdr Lloh34, Lloh38
-; CHECK-NEXT:    .loh AdrpLdr Lloh32, Lloh37
-; CHECK-NEXT:    .loh AdrpAdrp Lloh30, Lloh36
+; CHECK-NEXT:    .loh AdrpLdr Lloh34, Lloh37
+; CHECK-NEXT:    .loh AdrpLdr Lloh32, Lloh36
 ; CHECK-NEXT:    .loh AdrpLdr Lloh30, Lloh35
-; CHECK-NEXT:    .loh AdrpAdrp Lloh29, Lloh34
-; CHECK-NEXT:    .loh AdrpLdr Lloh29, Lloh33
-; CHECK-NEXT:    .loh AdrpAdrp Lloh28, Lloh32
-; CHECK-NEXT:    .loh AdrpLdr Lloh28, Lloh31
+; CHECK-NEXT:    .loh AdrpAdrp Lloh28, Lloh34
+; CHECK-NEXT:    .loh AdrpLdr Lloh28, Lloh33
+; CHECK-NEXT:    .loh AdrpAdrp Lloh27, Lloh32
+; CHECK-NEXT:    .loh AdrpLdr Lloh27, Lloh31
+; CHECK-NEXT:    .loh AdrpAdrp Lloh26, Lloh30
+; CHECK-NEXT:    .loh AdrpLdr Lloh26, Lloh29
 ;
 ; CHECK-BE-LABEL: zext_v23i8_to_v23i48_in_loop:
 ; CHECK-BE:       // %bb.0: // %entry
 ; CHECK-BE-NEXT:    adrp x8, .LCPI21_0
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI21_0
 ; CHECK-BE-NEXT:    ld1 { v0.16b }, [x8]
 ; CHECK-BE-NEXT:    adrp x8, .LCPI21_1
 ; CHECK-BE-NEXT:    add x8, x8, :lo12:.LCPI21_1