[AArch64] Add patterns for SVE predicated add/sub and mov combine

[NicolaLancellotti]

Summary

This patch creates predicated add and sub SVE instructions when the second operand of the add or sub LLVM instruction is a zext or a sext instruction.

I have already attempted to do that with two other patches by canonicalising zext and sext to a vselect instruction.

Patches:

• Canonicalize ZERO_EXTEND to VSELECT

This first patch is currently on the main branch but I noted that the tests for sub are wrong (the zext is the first operand instead of the second one, for add it doesn't matter because of the commutative property)

• Canonicalize SIGN_EXTEND to VSELECT

This patch is currently in review. Also, this patch needed more changes than the previous one, and it can be difficult in future to maintain it.

However, there is a simpler way to generate these predicated instructions by using the TableGen patterns proposed in this patch.
If you agree, I propose to revert the first patch, pre-commit the tests, and then commit these patterns and the updated tests.

[dmgreen]

I agree this sounds nicer than the sext patch - that one looked like there was much more involved than we were hoping for.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3160

The indenting can be a little lower here.

3175

I think these constants should be in the range 0-255. Only the bottom 8 bits are likely used, so -1 will mean the same thing, but it's best if we can generate instructions more precisely if we can.

Something like this link can be used to check the constants we need for different values: https://godbolt.org/z/9K7ozhYo7

[NicolaLancellotti]

Address comments

[NicolaLancellotti]

Thanks, @dmgreen for the review. I fixed the indent and replaced -1 with 255 and 65535 for i8 and i16.

[paulwalker-arm]

For SVE we've tried to minimise the number of patterns that emit multiple instructions so I'm wondering if instead this can be a post legalisation DAG combine that emits int_aarch64_sve_add/sub intrinsics?

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3159

This likely wants to be HasSVEorSME.

3160

Given the previous work that did land, do these patterns ever match? I mean we shouldn't have any zext of predicate vectors by this point should we? Or do you plan to revert that patch once this lands.

[NicolaLancellotti]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3160

Given the previous work that did land, do these patterns ever match? I mean we shouldn't have any zext of predicate vectors by this point should we? Or do you plan to revert that patch once this lands.

These zext patterns are added in case we want to revert the "Canonicalize ZERO_EXTEND to VSELECT" patch, otherwise, we don't need them.

[dmgreen]

I think the H/S/D should be 255 too, according to https://godbolt.org/z/zcYcdGjfh. It looks like it's a 8bit encoding, that gets treated as a signed value so 255 becomes -1.

For SVE we've tried to minimise the number of patterns that emit multiple instructions so I'm wondering if instead this can be a post legalisation DAG combine that emits int_aarch64_sve_add/sub intrinsics?

Do you think that is as important in this case? It is a good rule if the instructions could be combined into others (we try to use that rule in general for all of Arm/AArch64), but in this case the DUP_ZI_B are leaf nodes. They should all be CSE'd and remat trivially by the rest of the backend, and won't be combined into anything else.

[paulwalker-arm]

In D138570#3949132, @dmgreen wrote:

Do you think that is as important in this case?

It's not super important, it's just something that we made a conscious choice to avoid after hitting issues with our original downstream implementation. I'm not too bothered about the CSE side of things, it's more to keep the patterns maintainable. For example, if SVE gains a predicated add/sub imm instruction then you start needing to duplicate these patterns. Of course this can be refactored when such value can be gained so it's not a demand just a suggestion. I'd have a stronger view if AArch64add_m1 existed as an ISD node but it doesn't so I'm happy either way.

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3160

There's never a good reason to add unused patterns so you should pick one way or the other. For what it's worth I think consistency is best so if the isel patterns work best for sext and there's no downside to using them for zext then I'd prefer to revert the original work once this patch lands.

[NicolaLancellotti]

Address comments

[NicolaLancellotti]

In D138570#3949132, @dmgreen wrote:

I think the H/S/D should be 255 too, according to https://godbolt.org/z/zcYcdGjfh. It looks like it's a 8bit encoding, that gets treated as a signed value so 255 becomes -1.

You are right. Done.

[NicolaLancellotti]

I addressed the comments, if you agree, I'll proceed to revert the "Canonicalize ZERO_EXTEND to VSELECT" patch and land this patch.

[paulwalker-arm]

In D138570#3949484, @NicolaLancellotti wrote:

I addressed the comments, if you agree, I'll proceed to revert the "Canonicalize ZERO_EXTEND to VSELECT" patch and land this patch.

Yes, I'm happy with that.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3157–3159

It looks like you can merge these together.

3174–3175

Presumably you'll need zext variants of these patterns before reverting you're previous patch?

Perhaps it's worth canonicalising the constant just in case there are multiple instances of these operations. By which I mean:

add ZPR:$op, (sext ($pred)) -> SUB $op, $pred, DUP(1)
add ZPR:$op, (zext ($pred)) -> ADD $op, $pred, DUP(1)
sub ZPR:$op, (sext ($pred)) -> ADD $op, $pred, DUP(1)
sub ZPR:$op, (zext ($pred)) -> SUB $op, $pred, DUP(1)

llvm/test/CodeGen/AArch64/predicated-add-sub.ll

11–12

I know these are existing tests but is there a reason we don't just pass <vscale x 8 x i1> %v into the function?

[NicolaLancellotti]

Address comments

[NicolaLancellotti]

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3174–3175

As you can see from the tests, instead of subtracting 1, LLVM adds -1, which is the same.
So, we don't need them, do we?

llvm/test/CodeGen/AArch64/predicated-add-sub.ll

11–12

The tests were in a pre-commit test, I updated them.

[paulwalker-arm]

Thanks @NicolaLancellotti. I still think it would be better to canonicalise to two variants (i.e. add/sub op, #1) instead of the current three, but that can be changed later is it turns out be advantageous.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

3174–3175

I see what you mean.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

def : Pat<(vector_extract (nxv2f16 ZPR:$Zs), (i64 0)),
             (f16 (EXTRACT_SUBREG ZPR:$Zs, hsub))>;
   def : Pat<(vector_extract (nxv4f32 ZPR:$Zs), (i64 0)),
             (f32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv2f32 ZPR:$Zs), (i64 0)),
             (f32 (EXTRACT_SUBREG ZPR:$Zs, ssub))>;
   def : Pat<(vector_extract (nxv2f64 ZPR:$Zs), (i64 0)),
             (f64 (EXTRACT_SUBREG ZPR:$Zs, dsub))>;
   }
+
+  multiclass sve_predicated_add<SDNode extend, int value> {
+    def : Pat<(nxv16i8 (add ZPR:$op, (extend (nxv16i1 PPR:$pred)))),
+              (ADD_ZPmZ_B PPR:$pred, ZPR:$op, (DUP_ZI_B value, 0))>;
+    def : Pat<(nxv8i16 (add ZPR:$op, (extend (nxv8i1 PPR:$pred)))),
+              (ADD_ZPmZ_H PPR:$pred, ZPR:$op, (DUP_ZI_H value, 0))>;
+    def : Pat<(nxv4i32 (add ZPR:$op, (extend (nxv4i1 PPR:$pred)))),
+              (ADD_ZPmZ_S PPR:$pred, ZPR:$op, (DUP_ZI_S value, 0))>;
+    def : Pat<(nxv2i64 (add ZPR:$op, (extend (nxv2i1 PPR:$pred)))),
+              (ADD_ZPmZ_D PPR:$pred, ZPR:$op, (DUP_ZI_D value, 0))>;
+  }
+
+  defm : sve_predicated_add<zext, 1>;
+  defm : sve_predicated_add<sext, 255>;
+
+  def : Pat<(nxv16i8 (sub ZPR:$op, (sext (nxv16i1 PPR:$pred)))),
+            (SUB_ZPmZ_B PPR:$pred, ZPR:$op, (DUP_ZI_B 255, 0))>;
+  def : Pat<(nxv8i16 (sub ZPR:$op, (sext (nxv8i1 PPR:$pred)))),
+            (SUB_ZPmZ_H PPR:$pred, ZPR:$op, (DUP_ZI_H 255, 0))>;
+  def : Pat<(nxv4i32 (sub ZPR:$op, (sext (nxv4i1 PPR:$pred)))),
+            (SUB_ZPmZ_S PPR:$pred, ZPR:$op, (DUP_ZI_S 255, 0))>;
+  def : Pat<(nxv2i64 (sub ZPR:$op, (sext (nxv2i1 PPR:$pred)))),
+            (SUB_ZPmZ_D PPR:$pred, ZPR:$op, (DUP_ZI_D 255, 0))>;
 } // End HasSVEorSME
 
 let Predicates = [HasSVE, HasMatMulInt8] in {
   defm  SMMLA_ZZZ : sve_int_matmul<0b00, "smmla", int_aarch64_sve_smmla>;
   defm  UMMLA_ZZZ : sve_int_matmul<0b11, "ummla", int_aarch64_sve_ummla>;
   defm USMMLA_ZZZ : sve_int_matmul<0b10, "usmmla", int_aarch64_sve_usmmla>;
 } // End HasSVE, HasMatMulInt8
 
 let Predicates = [HasSVEorSME, HasMatMulInt8] in {
   defm USDOT_ZZZ  : sve_int_dot_mixed<"usdot", int_aarch64_sve_usdot>;
   defm USDOT_ZZZI : sve_int_dot_mixed_indexed<0, "usdot", int_aarch64_sve_usdot_lane>;
   defm SUDOT_ZZZI : sve_int_dot_mixed_indexed<1, "sudot", int_aarch64_sve_sudot_lane>;
 } // End HasSVEorSME, HasMatMulInt8
 
 let Predicates = [HasSVE, HasMatMulFP32] in {
   defm FMMLA_ZZZ_S : sve_fp_matrix_mla<0, "fmmla", ZPR32, int_aarch64_sve_fmmla, nxv4f32>;
 } // End HasSVE, HasMatMulFP32
 
 let Predicates = [HasSVE, HasMatMulFP64] in {
   defm FMMLA_ZZZ_D : sve_fp_matrix_mla<1, "fmmla", ZPR64, int_aarch64_sve_fmmla, nxv2f64>;
   defm LD1RO_B_IMM : sve_mem_ldor_si<0b00, "ld1rob", Z_b, ZPR8,  nxv16i8, nxv16i1, AArch64ld1ro_z>;
   defm LD1RO_H_IMM : sve_mem_ldor_si<0b01, "ld1roh", Z_h, ZPR16, nxv8i16, nxv8i1,  AArch64ld1ro_z>;
   defm LD1RO_W_IMM : sve_mem_ldor_si<0b10, "ld1row", Z_s, ZPR32, nxv4i32, nxv4i1,  AArch64ld1ro_z>;
   defm LD1RO_D_IMM : sve_mem_ldor_si<0b11, "ld1rod", Z_d, ZPR64, nxv2i64, nxv2i1,  AArch64ld1ro_z>;
   defm LD1RO_B     : sve_mem_ldor_ss<0b00, "ld1rob", Z_b, ZPR8,  GPR64NoXZRshifted8,  nxv16i8, nxv16i1, AArch64ld1ro_z, am_sve_regreg_lsl0>;
   defm LD1RO_H     : sve_mem_ldor_ss<0b01, "ld1roh", Z_h, ZPR16, GPR64NoXZRshifted16, nxv8i16, nxv8i1,  AArch64ld1ro_z, am_sve_regreg_lsl1>;
   defm LD1RO_W     : sve_mem_ldor_ss<0b10, "ld1row", Z_s, ZPR32, GPR64NoXZRshifted32, nxv4i32, nxv4i1,  AArch64ld1ro_z, am_sve_regreg_lsl2>;

llvm/test/CodeGen/AArch64/predicated-add-sub.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 
 target triple = "aarch64-unknown-linux"
 
 define <vscale x 8 x i8> @zext.add.8xi8(<vscale x 8 x i8> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.8xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #1 // =0x1
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i8>
   %result = add <vscale x 8 x i8> %a, %extend
   ret <vscale x 8 x i8> %result
 }
 
 define <vscale x 4 x i16> @zext.add.4xi16(<vscale x 4 x i16> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.4xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #1 // =0x1
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 4 x i1> %v to <vscale x 4 x i16>
   %result = add <vscale x 4 x i16> %a, %extend
   ret <vscale x 4 x i16> %result
 }
 
 define <vscale x 2 x i32> @zext.add.2xi32(<vscale x 2 x i32> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.2xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #1 // =0x1
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 2 x i1> %v to <vscale x 2 x i32>
   %result = add <vscale x 2 x i32> %a, %extend
   ret <vscale x 2 x i32> %result
 }
 
 define <vscale x 16 x i8> @zext.add.16xi8(<vscale x 16 x i8> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.16xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.b, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.b, z0.b, z1.b
+; CHECK-NEXT:    mov z1.b, #1 // =0x1
+; CHECK-NEXT:    add z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 16 x i1> %v to <vscale x 16 x i8>
   %result = add <vscale x 16 x i8> %a, %extend
   ret <vscale x 16 x i8> %result
 }
 
 define <vscale x 8 x i16> @zext.add.8xi16(<vscale x 8 x i16> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.8xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #1 // =0x1
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i16>
   %result = add <vscale x 8 x i16> %a, %extend
   ret <vscale x 8 x i16> %result
 }
 
 define <vscale x 4 x i32> @zext.add.4xi32(<vscale x 4 x i32> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.4xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #1 // =0x1
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 4 x i1> %v to <vscale x 4 x i32>
   %result = add <vscale x 4 x i32> %a, %extend
   ret <vscale x 4 x i32> %result
 }
 
 define <vscale x 2 x i64> @zext.add.2xi64(<vscale x 2 x i64> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.2xi64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #1 // =0x1
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 2 x i1> %v to <vscale x 2 x i64>
   %result = add <vscale x 2 x i64> %a, %extend
   ret <vscale x 2 x i64> %result
 }
 
 define <vscale x 8 x i32> @zext.add.8xi32(<vscale x 8 x i32> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.8xi32:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov z2.s, #1 // =0x1
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z2.s, p1/z, #1 // =0x1
-; CHECK-NEXT:    mov z3.s, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.s, z0.s, z3.s
-; CHECK-NEXT:    add z1.s, z1.s, z2.s
+; CHECK-NEXT:    add z1.s, p1/m, z1.s, z2.s
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i32>
   %result = add <vscale x 8 x i32> %a, %extend
   ret <vscale x 8 x i32> %result
 }
 
 define <vscale x 16 x i32> @zext.add.16xi32(<vscale x 16 x i32> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: zext.add.16xi32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
+; CHECK-NEXT:    mov z4.s, #1 // =0x1
 ; CHECK-NEXT:    punpkhi p2.h, p1.b
 ; CHECK-NEXT:    punpklo p1.h, p1.b
-; CHECK-NEXT:    mov z4.s, p2/z, #1 // =0x1
-; CHECK-NEXT:    punpkhi p2.h, p0.b
+; CHECK-NEXT:    punpkhi p3.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z5.s, p1/z, #1 // =0x1
-; CHECK-NEXT:    mov z6.s, p2/z, #1 // =0x1
-; CHECK-NEXT:    mov z7.s, p0/z, #1 // =0x1
-; CHECK-NEXT:    add z0.s, z0.s, z7.s
-; CHECK-NEXT:    add z1.s, z1.s, z6.s
-; CHECK-NEXT:    add z2.s, z2.s, z5.s
-; CHECK-NEXT:    add z3.s, z3.s, z4.s
+; CHECK-NEXT:    add z1.s, p3/m, z1.s, z4.s
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z4.s
+; CHECK-NEXT:    add z2.s, p1/m, z2.s, z4.s
+; CHECK-NEXT:    add z3.s, p2/m, z3.s, z4.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 16 x i1> %v to <vscale x 16 x i32>
   %result = add <vscale x 16 x i32> %a, %extend
   ret <vscale x 16 x i32> %result
 }
 
 define <vscale x 8 x i8> @zext.sub.8xi8(<vscale x 8 x i8> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.8xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i8>
   %result = sub <vscale x 8 x i8> %a, %extend
   ret <vscale x 8 x i8> %result
 }
 
 define <vscale x 4 x i16> @zext.sub.4xi16(<vscale x 4 x i16> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.4xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 4 x i1> %v to <vscale x 4 x i16>
   %result = sub <vscale x 4 x i16> %a, %extend
   ret <vscale x 4 x i16> %result
 }
 
 define <vscale x 2 x i32> @zext.sub.2xi32(<vscale x 2 x i32> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.2xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 2 x i1> %v to <vscale x 2 x i32>
   %result = sub <vscale x 2 x i32> %a, %extend
   ret <vscale x 2 x i32> %result
 }
 
 define <vscale x 16 x i8> @zext.sub.16xi8(<vscale x 16 x i8> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.16xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.b, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.b, z0.b, z1.b
+; CHECK-NEXT:    mov z1.b, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 16 x i1> %v to <vscale x 16 x i8>
   %result = sub <vscale x 16 x i8> %a, %extend
   ret <vscale x 16 x i8> %result
 }
 
 define <vscale x 8 x i16> @zext.sub.8xi16(<vscale x 8 x i16> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.8xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i16>
   %result = sub <vscale x 8 x i16> %a, %extend
   ret <vscale x 8 x i16> %result
 }
 
 define <vscale x 4 x i32> @zext.sub.4xi32(<vscale x 4 x i32> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.4xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 4 x i1> %v to <vscale x 4 x i32>
   %result = sub <vscale x 4 x i32> %a, %extend
   ret <vscale x 4 x i32> %result
 }
 
 define <vscale x 2 x i64> @zext.sub.2xi64(<vscale x 2 x i64> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.2xi64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 2 x i1> %v to <vscale x 2 x i64>
   %result = sub <vscale x 2 x i64> %a, %extend
   ret <vscale x 2 x i64> %result
 }
 
 define <vscale x 8 x i32> @zext.sub.8xi32(<vscale x 8 x i32> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.8xi32:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpklo p1.h, p0.b
 ; CHECK-NEXT:    punpkhi p0.h, p0.b
-; CHECK-NEXT:    mov z2.s, p1/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z3.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z2.s
-; CHECK-NEXT:    add z1.s, z1.s, z3.s
+; CHECK-NEXT:    add z0.s, p1/m, z0.s, z2.s
+; CHECK-NEXT:    add z1.s, p0/m, z1.s, z2.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 8 x i1> %v to <vscale x 8 x i32>
   %result = sub <vscale x 8 x i32> %a, %extend
   ret <vscale x 8 x i32> %result
 }
 
 define <vscale x 16 x i32> @zext.sub.16xi32(<vscale x 16 x i32> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: zext.sub.16xi32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    punpklo p1.h, p0.b
+; CHECK-NEXT:    mov z4.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpkhi p0.h, p0.b
 ; CHECK-NEXT:    punpklo p2.h, p1.b
 ; CHECK-NEXT:    punpkhi p1.h, p1.b
-; CHECK-NEXT:    mov z4.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    punpklo p2.h, p0.b
-; CHECK-NEXT:    mov z5.s, p1/z, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.s, p2/m, z0.s, z4.s
+; CHECK-NEXT:    add z1.s, p1/m, z1.s, z4.s
+; CHECK-NEXT:    punpklo p1.h, p0.b
 ; CHECK-NEXT:    punpkhi p0.h, p0.b
-; CHECK-NEXT:    add z0.s, z0.s, z4.s
-; CHECK-NEXT:    add z1.s, z1.s, z5.s
-; CHECK-NEXT:    mov z4.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z5.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z2.s, z2.s, z4.s
-; CHECK-NEXT:    add z3.s, z3.s, z5.s
+; CHECK-NEXT:    add z2.s, p1/m, z2.s, z4.s
+; CHECK-NEXT:    add z3.s, p0/m, z3.s, z4.s
 ; CHECK-NEXT:    ret
   %extend = zext <vscale x 16 x i1> %v to <vscale x 16 x i32>
   %result = sub <vscale x 16 x i32> %a, %extend
   ret <vscale x 16 x i32> %result
 }
 
 define <vscale x 8 x i8> @sext.add.8xi8(<vscale x 8 x i8> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.8xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i8>
   %result = add <vscale x 8 x i8> %a, %extend
   ret <vscale x 8 x i8> %result
 }
 
 define <vscale x 4 x i16> @sext.add.4xi16(<vscale x 4 x i16> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.4xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 4 x i1> %v to <vscale x 4 x i16>
   %result = add <vscale x 4 x i16> %a, %extend
   ret <vscale x 4 x i16> %result
 }
 
 define <vscale x 2 x i32> @sext.add.2xi32(<vscale x 2 x i32> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.2xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 2 x i1> %v to <vscale x 2 x i32>
   %result = add <vscale x 2 x i32> %a, %extend
   ret <vscale x 2 x i32> %result
 }
 
 define <vscale x 16 x i8> @sext.add.16xi8(<vscale x 16 x i8> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.16xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.b, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.b, z0.b, z1.b
+; CHECK-NEXT:    mov z1.b, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 16 x i1> %v to <vscale x 16 x i8>
   %result = add <vscale x 16 x i8> %a, %extend
   ret <vscale x 16 x i8> %result
 }
 
 define <vscale x 8 x i16> @sext.add.8xi16(<vscale x 8 x i16> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.8xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i16>
   %result = add <vscale x 8 x i16> %a, %extend
   ret <vscale x 8 x i16> %result
 }
 
 define <vscale x 4 x i32> @sext.add.4xi32(<vscale x 4 x i32> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.4xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 4 x i1> %v to <vscale x 4 x i32>
   %result = add <vscale x 4 x i32> %a, %extend
   ret <vscale x 4 x i32> %result
 }
 
 define <vscale x 2 x i64> @sext.add.2xi64(<vscale x 2 x i64> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.2xi64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    add z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 2 x i1> %v to <vscale x 2 x i64>
   %result = add <vscale x 2 x i64> %a, %extend
   ret <vscale x 2 x i64> %result
 }
 
 define <vscale x 8 x i32> @sext.add.8xi32(<vscale x 8 x i32> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.8xi32:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z2.s, p1/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z3.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z3.s
-; CHECK-NEXT:    add z1.s, z1.s, z2.s
+; CHECK-NEXT:    add z1.s, p1/m, z1.s, z2.s
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i32>
   %result = add <vscale x 8 x i32> %a, %extend
   ret <vscale x 8 x i32> %result
 }
 
 define <vscale x 16 x i32> @sext.add.16xi32(<vscale x 16 x i32> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: sext.add.16xi32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
+; CHECK-NEXT:    mov z4.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpkhi p2.h, p1.b
 ; CHECK-NEXT:    punpklo p1.h, p1.b
-; CHECK-NEXT:    mov z4.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    punpkhi p2.h, p0.b
+; CHECK-NEXT:    punpkhi p3.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z5.s, p1/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z6.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z7.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    add z0.s, z0.s, z7.s
-; CHECK-NEXT:    add z1.s, z1.s, z6.s
-; CHECK-NEXT:    add z2.s, z2.s, z5.s
-; CHECK-NEXT:    add z3.s, z3.s, z4.s
+; CHECK-NEXT:    add z1.s, p3/m, z1.s, z4.s
+; CHECK-NEXT:    add z0.s, p0/m, z0.s, z4.s
+; CHECK-NEXT:    add z2.s, p1/m, z2.s, z4.s
+; CHECK-NEXT:    add z3.s, p2/m, z3.s, z4.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 16 x i1> %v to <vscale x 16 x i32>
   %result = add <vscale x 16 x i32> %a, %extend
   ret <vscale x 16 x i32> %result
 }
 
 define <vscale x 8 x i8> @sext.sub.8xi8(<vscale x 8 x i8> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.8xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i8>
   %result = sub <vscale x 8 x i8> %a, %extend
   ret <vscale x 8 x i8> %result
 }
 
 define <vscale x 4 x i16> @sext.sub.4xi16(<vscale x 4 x i16> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.4xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 4 x i1> %v to <vscale x 4 x i16>
   %result = sub <vscale x 4 x i16> %a, %extend
   ret <vscale x 4 x i16> %result
 }
 
 define <vscale x 2 x i32> @sext.sub.2xi32(<vscale x 2 x i32> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.2xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 2 x i1> %v to <vscale x 2 x i32>
   %result = sub <vscale x 2 x i32> %a, %extend
   ret <vscale x 2 x i32> %result
 }
 
 define <vscale x 16 x i8> @sext.sub.16xi8(<vscale x 16 x i8> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.16xi8:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.b, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.b, z0.b, z1.b
+; CHECK-NEXT:    mov z1.b, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.b, p0/m, z0.b, z1.b
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 16 x i1> %v to <vscale x 16 x i8>
   %result = sub <vscale x 16 x i8> %a, %extend
   ret <vscale x 16 x i8> %result
 }
 
 define <vscale x 8 x i16> @sext.sub.8xi16(<vscale x 8 x i16> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.8xi16:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.h, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.h, z0.h, z1.h
+; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.h, p0/m, z0.h, z1.h
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i16>
   %result = sub <vscale x 8 x i16> %a, %extend
   ret <vscale x 8 x i16> %result
 }
 
 define <vscale x 4 x i32> @sext.sub.4xi32(<vscale x 4 x i32> %a, <vscale x 4 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.4xi32:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.s, z0.s, z1.s
+; CHECK-NEXT:    mov z1.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.s, p0/m, z0.s, z1.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 4 x i1> %v to <vscale x 4 x i32>
   %result = sub <vscale x 4 x i32> %a, %extend
   ret <vscale x 4 x i32> %result
 }
 
 define <vscale x 2 x i64> @sext.sub.2xi64(<vscale x 2 x i64> %a, <vscale x 2 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.2xi64:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    mov z1.d, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.d, z0.d, z1.d
+; CHECK-NEXT:    mov z1.d, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    sub z0.d, p0/m, z0.d, z1.d
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 2 x i1> %v to <vscale x 2 x i64>
   %result = sub <vscale x 2 x i64> %a, %extend
   ret <vscale x 2 x i64> %result
 }
 
 define <vscale x 8 x i32> @sext.sub.8xi32(<vscale x 8 x i32> %a, <vscale x 8 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.8xi32:
 ; CHECK:       // %bb.0:
+; CHECK-NEXT:    mov z2.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z2.s, p1/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z3.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.s, z0.s, z3.s
-; CHECK-NEXT:    sub z1.s, z1.s, z2.s
+; CHECK-NEXT:    sub z1.s, p1/m, z1.s, z2.s
+; CHECK-NEXT:    sub z0.s, p0/m, z0.s, z2.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 8 x i1> %v to <vscale x 8 x i32>
   %result = sub <vscale x 8 x i32> %a, %extend
   ret <vscale x 8 x i32> %result
 }
 
 define <vscale x 16 x i32> @sext.sub.16xi32(<vscale x 16 x i32> %a, <vscale x 16 x i1> %v) #0 {
 ; CHECK-LABEL: sext.sub.16xi32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    punpkhi p1.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
+; CHECK-NEXT:    mov z4.s, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    punpkhi p2.h, p1.b
 ; CHECK-NEXT:    punpklo p1.h, p1.b
-; CHECK-NEXT:    mov z4.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    punpkhi p2.h, p0.b
+; CHECK-NEXT:    punpkhi p3.h, p0.b
 ; CHECK-NEXT:    punpklo p0.h, p0.b
-; CHECK-NEXT:    mov z5.s, p1/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z6.s, p2/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    mov z7.s, p0/z, #-1 // =0xffffffffffffffff
-; CHECK-NEXT:    sub z0.s, z0.s, z7.s
-; CHECK-NEXT:    sub z1.s, z1.s, z6.s
-; CHECK-NEXT:    sub z2.s, z2.s, z5.s
-; CHECK-NEXT:    sub z3.s, z3.s, z4.s
+; CHECK-NEXT:    sub z1.s, p3/m, z1.s, z4.s
+; CHECK-NEXT:    sub z0.s, p0/m, z0.s, z4.s
+; CHECK-NEXT:    sub z2.s, p1/m, z2.s, z4.s
+; CHECK-NEXT:    sub z3.s, p2/m, z3.s, z4.s
 ; CHECK-NEXT:    ret
   %extend = sext <vscale x 16 x i1> %v to <vscale x 16 x i32>
   %result = sub <vscale x 16 x i32> %a, %extend
   ret <vscale x 16 x i32> %result
 }
 
 attributes #0 = { "target-features"="+sve" }