[AArch64] Autogenerate sve-fixed-length tests. NFC

[deadalnix]

Summary

As per title. This makes it easier to work onc hange that require "shotgun diffs" over the codebase.

[DavidSpickett]

Generating test output seems like a good change but I have no experience with it myself.

[paulwalker-arm]

For most of these tests we don't care about the exact "no sve" output and thus have a single CHECK for the whole file. Is there a parameter that can be passed to update_llc_test_checks.py so those CHECK lines are not generated?

[RKSimon]

llvm/test/CodeGen/AArch64/sve-fixed-length-extract-subvector.ll

3–4

Please can you -check-prefixes=CHECK,VBITS_GE_384 to the tests?

[deadalnix]

Use VBITS_GE_384

[paulwalker-arm]

Have you had chance to see if the extra NO_SVE lines can be removed. I really don't want these tests to require updates when SVE is not being used, whilst at the same time validating no SVE is used, which is what the current single NO_SVE-NOT lines verifies.

llvm/test/CodeGen/AArch64/sve-fixed-length-extract-subvector.ll

3–4

Sorry I didn't get to respond before you made the change but I don't think VBITS_GE_384 makes sense for some (if not all) of the tests you're changing. The fixed length support it limited to power-of-two values (regardless of the actual target vector length) and so the likely real fix is for the RUN lines where only aarch64-sve-vector-bits-min is set to use VBITS_GE_256. You can see this when looking at the output with VBITS_EQ_256 and VBITS_GE_384 being identical.

[RKSimon]

llvm/test/CodeGen/AArch64/sve-fixed-length-extract-subvector.ll

3–4

How about this?

; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefixes=CHECK,NO_SVE
; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_EQ_256
; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_EQ_256
; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512
; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512
; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512
; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512
; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024
; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -check-prefixes=CHECK,SVE,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048

[deadalnix]

Use VBITS_GE_256

[deadalnix]

I do not know if there is a way to simplify the NO_SVE case. Maybe an expert with utils/update_llc_test_checks.py could help?

[paulwalker-arm]

To be honest these tests are just not a good fit for auto-generation, which is why we didn't use update_llc_test_checks.py originally. The best I could come up with is to disable the NO_SVE run line when running the script and then enable it afterwards. I do have the workings of a patch to add a --ignored-check-prefixes option to update_llc_test_checks.py but even then the decision as to which CHECK lines are emitted doesn't really match what we wanted to achieve. If autogeneration is preferable and you're happy to wait a week or two I think it'll be better if I just restructure all the SVE fixed length tests (essentially split them based on register size) and enable auto-generation at the same time. I think this is worth doing anyway to increase our test coverage and test parallelism as the existing layout means these tests are the most time consuming ones within the AArch64 directory.

[RKSimon]

In D127118#3569796, @paulwalker-arm wrote:

To be honest these tests are just not a good fit for auto-generation, which is why we didn't use update_llc_test_checks.py originally. The best I could come up with is to disable the NO_SVE run line when running the script and then enable it afterwards.

That might be an acceptable solution to stop this blocking D127115 - by removing the '| FileCheck %s -check-prefix=NO_SVE' the update script will ignore those RUN lines. After that its just a question of manualy adding the FileCheck back and removing the 'NOTE:' top line - we have plenty of test files that are manually adjusted after the update script, removing the 'NOTE' just makes it clearer.

I'd possibly suggest replacing VBITS_EQ_256 with VBITS_GE_256 and adding it the 384 tests as well as the 256 tests (and get rid of the 384 specific checks - sorry my bad).

@paulwalker-arm Would that work or do you think the test refactor is imminent?

[deadalnix]

llvm/test/CodeGen/AArch64/sve-fixed-length-extract-subvector.ll

22

If what we want to check is that this is not present, maybe we could simply replace the filecheck for NO_SVE with a grep that ensure ptrue isn't present. this will allow the other cases to be autogenerated without polluting the tests too much.

Would that work?

[deadalnix]

Use not grep.

[deadalnix]

Rework the test a bit to improve the end result. Loop in a couple of extra test.

[paulwalker-arm]

@paulwalker-arm Would that work or do you think the test refactor is imminent?

I'm happy with this. I do plan to work on the refactoring this week but if landing these changes unblocks other work then let's do that.

llvm/test/CodeGen/AArch64/sve-fixed-length-float-compares.ll

360–362

Is this enforced by the update script? If not then please leave the comment style as was. There are a few other files with similar changes.

[deadalnix]

llvm/test/CodeGen/AArch64/sve-fixed-length-float-compares.ll

362

Has, this is leftovers from the cleanup I ran to get there. Let me restore these, and then I'll land this. Thanks.

[deadalnix]

Fixup erroneously remove ;

[deadalnix]

More ; fixups

[RKSimon]

@deadalnix @paulwalker-arm Should sve-fixed-length-fp-vselect.ll be updated to match as well?

[paulwalker-arm]

In D127118#3577687, @RKSimon wrote:

@deadalnix @paulwalker-arm Should sve-fixed-length-fp-vselect.ll be updated to match as well?

Sorry, I don't understand the question.

[RKSimon]

In D127118#3577754, @paulwalker-arm wrote:

In D127118#3577687, @RKSimon wrote:

@deadalnix @paulwalker-arm Should sve-fixed-length-fp-vselect.ll be updated to match as well?

Sorry, I don't understand the question.

Sorry copy-paste mistake - I meant sve-fixed-length-fp-select.ll which has NO_SVE checks for codegen as well as a test for ptrue.

[paulwalker-arm]

I see :)

There's many test files that this patch doesn't change. I just figured the patch only changed the currently problematic ones. In any case, for my refactoring I'll ensure all sve-fixed-length- files have auto-generated check lines.

llvm/test/CodeGen/AArch64/sve-fixed-length-bitselect.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=256 < %s | FileCheck %s
 
 target triple = "aarch64"
 
 ;
 ; NOTE: SVE lowering for the BSP pseudoinst is not currently implemented, so we
 ;       don't currently expect the code below to lower to BSL/BIT/BIF. Once
 ;       this is implemented, this test will be fleshed out.
 ;
 
 define <8 x i32> @fixed_bitselect_v8i32(<8 x i32>* %pre_cond_ptr, <8 x i32>* %left_ptr, <8 x i32>* %right_ptr) #0 {
 ; CHECK-LABEL: fixed_bitselect_v8i32:
-; CHECK-NOT:     bsl {{.*}}, {{.*}}, {{.*}}
-; CHECK-NOT:     bit {{.*}}, {{.*}}, {{.*}}
-; CHECK-NOT:     bif {{.*}}, {{.*}}, {{.*}}
-; CHECK:         ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    mov z3.s, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
+; CHECK-NEXT:    ld1w { z2.s }, p0/z, [x2]
+; CHECK-NEXT:    add z3.s, z0.s, z3.s
+; CHECK-NEXT:    subr z0.s, z0.s, #0 // =0x0
+; CHECK-NEXT:    and z0.d, z0.d, z1.d
+; CHECK-NEXT:    and z1.d, z3.d, z2.d
+; CHECK-NEXT:    orr z0.d, z1.d, z0.d
+; CHECK-NEXT:    st1w { z0.s }, p0, [x8]
+; CHECK-NEXT:    ret
   %pre_cond = load <8 x i32>, <8 x i32>* %pre_cond_ptr
   %left = load <8 x i32>, <8 x i32>* %left_ptr
   %right = load <8 x i32>, <8 x i32>* %right_ptr
 
   %neg_cond = sub <8 x i32> zeroinitializer, %pre_cond
   %min_cond = add <8 x i32> %pre_cond, <i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1, i32 -1>
   %left_bits_0 = and <8 x i32> %neg_cond, %left
   %right_bits_0 = and <8 x i32> %min_cond, %right
   %bsl0000 = or <8 x i32> %right_bits_0, %left_bits_0
   ret <8 x i32> %bsl0000
 }
 
 attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-extract-subvector.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Don't use SVE when its registers are no bigger than NEON.
 ; NO_SVE-NOT: ptrue
 
 ; i8
 
 ; Don't use SVE for 64-bit vectors.
 define <4 x i8> @extract_subvector_v8i8(<8 x i8> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v8i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    zip2 v0.8b, v0.8b, v0.8b
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v8i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    zip2 v0.8b, v0.8b, v0.8b
 ; CHECK-NEXT:    ret
   %ret = call <4 x i8> @llvm.experimental.vector.extract.v4i8.v8i8(<8 x i8> %op, i64 4)
   ret <4 x i8> %ret
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <8 x i8> @extract_subvector_v16i8(<16 x i8> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v16i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v16i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <8 x i8> @llvm.experimental.vector.extract.v8i8.v16i8(<16 x i8> %op, i64 8)
   ret <8 x i8> %ret
 }
 
 define void @extract_subvector_v32i8(<32 x i8>* %a, <16 x i8>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v32i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v32i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.b, vl32
 ; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <32 x i8>, <32 x i8>* %a
   %ret = call <16 x i8> @llvm.experimental.vector.extract.v16i8.v32i8(<32 x i8> %op, i64 16)
   store <16 x i8> %ret, <16 x i8>* %b
   ret void
 }
 
 define void @extract_subvector_v64i8(<64 x i8>* %a, <32 x i8>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v64i8:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov w8, #32
-; VBITS_EQ_256-NEXT:    ptrue p0.b, vl32
-; VBITS_EQ_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
-; VBITS_EQ_256-NEXT:    st1b { z0.b }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v64i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v64i8:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.b, vl64
 ; VBITS_GE_512-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.b, vl32
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1b { z0.b }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <64 x i8>, <64 x i8>* %a
   %ret = call <32 x i8> @llvm.experimental.vector.extract.v32i8.v64i8(<64 x i8> %op, i64 32)
   store <32 x i8> %ret, <32 x i8>* %b
   ret void
 }
 
 define void @extract_subvector_v128i8(<128 x i8>* %a, <64 x i8>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v128i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v128i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #64
+; VBITS_GE_256-NEXT:    mov w9, #96
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v128i8:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.b, vl128
 ; VBITS_GE_1024-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.b, vl64
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1b { z0.b }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <128 x i8>, <128 x i8>* %a
   %ret = call <64 x i8> @llvm.experimental.vector.extract.v64i8.v128i8(<128 x i8> %op, i64 64)
   store <64 x i8> %ret, <64 x i8>* %b
   ret void
 }
 
 define void @extract_subvector_v256i8(<256 x i8>* %a, <128 x i8>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v256i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v256i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #128
+; VBITS_GE_256-NEXT:    mov w9, #160
+; VBITS_GE_256-NEXT:    mov w10, #224
+; VBITS_GE_256-NEXT:    mov w11, #192
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x0, x10]
+; VBITS_GE_256-NEXT:    ld1b { z3.b }, p0/z, [x0, x11]
+; VBITS_GE_256-NEXT:    mov w8, #64
+; VBITS_GE_256-NEXT:    mov w9, #96
+; VBITS_GE_256-NEXT:    mov w10, #32
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1b { z3.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x1, x9]
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x1, x10]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v256i8:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.b, vl256
 ; VBITS_GE_2048-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.b, vl128
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1b { z0.b }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <256 x i8>, <256 x i8>* %a
   %ret = call <128 x i8> @llvm.experimental.vector.extract.v128i8.v256i8(<256 x i8> %op, i64 128)
   store <128 x i8> %ret, <128 x i8>* %b
   ret void
 }
 
 ; i16
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x i16> @extract_subvector_v4i16(<4 x i16> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v4i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    umov w8, v0.h[2]
+; NO_SVE-NEXT:    umov w9, v0.h[3]
+; NO_SVE-NEXT:    fmov s0, w8
+; NO_SVE-NEXT:    mov v0.s[1], w9
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    umov w8, v0.h[2]
 ; CHECK-NEXT:    umov w9, v0.h[3]
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    mov v0.s[1], w9
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <2 x i16> @llvm.experimental.vector.extract.v2i16.v4i16(<4 x i16> %op, i64 2)
   ret <2 x i16> %ret
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x i16> @extract_subvector_v8i16(<8 x i16> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v8i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v8i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <4 x i16> @llvm.experimental.vector.extract.v4i16.v8i16(<8 x i16> %op, i64 4)
   ret <4 x i16> %ret
 }
 
 define void @extract_subvector_v16i16(<16 x i16>* %a, <8 x i16>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v16i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v16i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <16 x i16>, <16 x i16>* %a
   %ret = call <8 x i16> @llvm.experimental.vector.extract.v8i16.v16i16(<16 x i16> %op, i64 8)
   store <8 x i16> %ret, <8 x i16>* %b
   ret void
 }
 
 define void @extract_subvector_v32i16(<32 x i16>* %a, <16 x i16>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v32i16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    st1h { z0.h }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v32i16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl16
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <32 x i16>, <32 x i16>* %a
   %ret = call <16 x i16> @llvm.experimental.vector.extract.v16i16.v32i16(<32 x i16> %op, i64 16)
   store <16 x i16> %ret, <16 x i16>* %b
   ret void
 }
 
 define void @extract_subvector_v64i16(<64 x i16>* %a, <32 x i16>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v64i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v64i16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <64 x i16>, <64 x i16>* %a
   %ret = call <32 x i16> @llvm.experimental.vector.extract.v32i16.v64i16(<64 x i16> %op, i64 32)
   store <32 x i16> %ret, <32 x i16>* %b
   ret void
 }
 
 define void @extract_subvector_v128i16(<128 x i16>* %a, <64 x i16>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v128i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v128i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #64
+; VBITS_GE_256-NEXT:    mov x9, #80
+; VBITS_GE_256-NEXT:    mov x10, #112
+; VBITS_GE_256-NEXT:    mov x11, #96
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v128i16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <128 x i16>, <128 x i16>* %a
   %ret = call <64 x i16> @llvm.experimental.vector.extract.v64i16.v128i16(<128 x i16> %op, i64 64)
   store <64 x i16> %ret, <64 x i16>* %b
   ret void
 }
 
 ; i32
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x i32> @extract_subvector_v2i32(<2 x i32> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v2i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2s, v0.s[1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    dup v0.2s, v0.s[1]
 ; CHECK-NEXT:    ret
   %ret = call <1 x i32> @llvm.experimental.vector.extract.v1i32.v2i32(<2 x i32> %op, i64 1)
   ret <1 x i32> %ret
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x i32> @extract_subvector_v4i32(<4 x i32> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v4i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <2 x i32> @llvm.experimental.vector.extract.v2i32.v4i32(<4 x i32> %op, i64 2)
   ret <2 x i32> %ret
 }
 
 define void @extract_subvector_v8i32(<8 x i32>* %a, <4 x i32>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v8i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v8i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <8 x i32>, <8 x i32>* %a
   %ret = call <4 x i32> @llvm.experimental.vector.extract.v4i32.v8i32(<8 x i32> %op, i64 4)
   store <4 x i32> %ret, <4 x i32>* %b
   ret void
 }
 
 define void @extract_subvector_v16i32(<16 x i32>* %a, <8 x i32>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v16i32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    st1w { z0.s }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v16i32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl8
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <16 x i32>, <16 x i32>* %a
   %ret = call <8 x i32> @llvm.experimental.vector.extract.v8i32.v16i32(<16 x i32> %op, i64 8)
   store <8 x i32> %ret, <8 x i32>* %b
   ret void
 }
 
 define void @extract_subvector_v32i32(<32 x i32>* %a, <16 x i32>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v32i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v32i32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <32 x i32>, <32 x i32>* %a
   %ret = call <16 x i32> @llvm.experimental.vector.extract.v16i32.v32i32(<32 x i32> %op, i64 16)
   store <16 x i32> %ret, <16 x i32>* %b
   ret void
 }
 
 define void @extract_subvector_v64i32(<64 x i32>* %a, <32 x i32>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v64i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #40
+; VBITS_GE_256-NEXT:    mov x10, #56
+; VBITS_GE_256-NEXT:    mov x11, #48
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v64i32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <64 x i32>, <64 x i32>* %a
   %ret = call <32 x i32> @llvm.experimental.vector.extract.v32i32.v64i32(<64 x i32> %op, i64 32)
   store <32 x i32> %ret, <32 x i32>* %b
   ret void
 }
 
 ; i64
 
 ; Don't use SVE for 128-bit vectors.
 define <1 x i64> @extract_subvector_v2i64(<2 x i64> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v2i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <1 x i64> @llvm.experimental.vector.extract.v1i64.v2i64(<2 x i64> %op, i64 1)
   ret <1 x i64> %ret
 }
 
 define void @extract_subvector_v4i64(<4 x i64>* %a, <2 x i64>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v4i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <4 x i64>, <4 x i64>* %a
   %ret = call <2 x i64> @llvm.experimental.vector.extract.v2i64.v4i64(<4 x i64> %op, i64 2)
   store <2 x i64> %ret, <2 x i64>* %b
   ret void
 }
 
 define void @extract_subvector_v8i64(<8 x i64>* %a, <4 x i64>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v8i64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    st1d { z0.d }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v8i64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl4
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <8 x i64>, <8 x i64>* %a
   %ret = call <4 x i64> @llvm.experimental.vector.extract.v4i64.v8i64(<8 x i64> %op, i64 4)
   store <4 x i64> %ret, <4 x i64>* %b
   ret void
 }
 
 define void @extract_subvector_v16i64(<16 x i64>* %a, <8 x i64>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v16i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v16i64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <16 x i64>, <16 x i64>* %a
   %ret = call <8 x i64> @llvm.experimental.vector.extract.v8i64.v16i64(<16 x i64> %op, i64 8)
   store <8 x i64> %ret, <8 x i64>* %b
   ret void
 }
 
 define void @extract_subvector_v32i64(<32 x i64>* %a, <16 x i64>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v32i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #20
+; VBITS_GE_256-NEXT:    mov x10, #28
+; VBITS_GE_256-NEXT:    mov x11, #24
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v32i64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <32 x i64>, <32 x i64>* %a
   %ret = call <16 x i64> @llvm.experimental.vector.extract.v16i64.v32i64(<32 x i64> %op, i64 16)
   store <16 x i64> %ret, <16 x i64>* %b
   ret void
 }
 
 ; f16
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x half> @extract_subvector_v4f16(<4 x half> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2s, v0.s[1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    dup v0.2s, v0.s[1]
 ; CHECK-NEXT:    ret
   %ret = call <2 x half> @llvm.experimental.vector.extract.v2f16.v4f16(<4 x half> %op, i64 2)
   ret <2 x half> %ret
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x half> @extract_subvector_v8f16(<8 x half> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v8f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <4 x half> @llvm.experimental.vector.extract.v4f16.v8f16(<8 x half> %op, i64 4)
   ret <4 x half> %ret
 }
 
 define void @extract_subvector_v16f16(<16 x half>* %a, <8 x half>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <16 x half>, <16 x half>* %a
   %ret = call <8 x half> @llvm.experimental.vector.extract.v8f16.v16f16(<16 x half> %op, i64 8)
   store <8 x half> %ret, <8 x half>* %b
   ret void
 }
 
 define void @extract_subvector_v32f16(<32 x half>* %a, <16 x half>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v32f16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    st1h { z0.h }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v32f16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl16
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <32 x half>, <32 x half>* %a
   %ret = call <16 x half> @llvm.experimental.vector.extract.v16f16.v32f16(<32 x half> %op, i64 16)
   store <16 x half> %ret, <16 x half>* %b
   ret void
 }
 
 define void @extract_subvector_v64f16(<64 x half>* %a, <32 x half>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v64f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v64f16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <64 x half>, <64 x half>* %a
   %ret = call <32 x half> @llvm.experimental.vector.extract.v32f16.v64f16(<64 x half> %op, i64 32)
   store <32 x half> %ret, <32 x half>* %b
   ret void
 }
 
 define void @extract_subvector_v128f16(<128 x half>* %a, <64 x half>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v128f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v128f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #64
+; VBITS_GE_256-NEXT:    mov x9, #80
+; VBITS_GE_256-NEXT:    mov x10, #112
+; VBITS_GE_256-NEXT:    mov x11, #96
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v128f16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <128 x half>, <128 x half>* %a
   %ret = call <64 x half> @llvm.experimental.vector.extract.v64f16.v128f16(<128 x half> %op, i64 64)
   store <64 x half> %ret, <64 x half>* %b
   ret void
 }
 
 ; f32
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x float> @extract_subvector_v2f32(<2 x float> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2s, v0.s[1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; CHECK-NEXT:    dup v0.2s, v0.s[1]
 ; CHECK-NEXT:    ret
   %ret = call <1 x float> @llvm.experimental.vector.extract.v1f32.v2f32(<2 x float> %op, i64 1)
   ret <1 x float> %ret
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x float> @extract_subvector_v4f32(<4 x float> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <2 x float> @llvm.experimental.vector.extract.v2f32.v4f32(<4 x float> %op, i64 2)
   ret <2 x float> %ret
 }
 
 define void @extract_subvector_v8f32(<8 x float>* %a, <4 x float>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <8 x float>, <8 x float>* %a
   %ret = call <4 x float> @llvm.experimental.vector.extract.v4f32.v8f32(<8 x float> %op, i64 4)
   store <4 x float> %ret, <4 x float>* %b
   ret void
 }
 
 define void @extract_subvector_v16f32(<16 x float>* %a, <8 x float>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v16f32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    st1w { z0.s }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v16f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl8
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <16 x float>, <16 x float>* %a
   %ret = call <8 x float> @llvm.experimental.vector.extract.v8f32.v16f32(<16 x float> %op, i64 8)
   store <8 x float> %ret, <8 x float>* %b
   ret void
 }
 
 define void @extract_subvector_v32f32(<32 x float>* %a, <16 x float>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v32f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <32 x float>, <32 x float>* %a
   %ret = call <16 x float> @llvm.experimental.vector.extract.v16f32.v32f32(<32 x float> %op, i64 16)
   store <16 x float> %ret, <16 x float>* %b
   ret void
 }
 
 define void @extract_subvector_v64f32(<64 x float>* %a, <32 x float>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #40
+; VBITS_GE_256-NEXT:    mov x10, #56
+; VBITS_GE_256-NEXT:    mov x11, #48
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v64f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret
   %op = load <64 x float>, <64 x float>* %a
   %ret = call <32 x float> @llvm.experimental.vector.extract.v32f32.v64f32(<64 x float> %op, i64 32)
   store <32 x float> %ret, <32 x float>* %b
   ret void
 }
 
 ; f64
 
 ; Don't use SVE for 128-bit vectors.
 define <1 x double> @extract_subvector_v2f64(<2 x double> %op) #0 {
+; NO_SVE-LABEL: extract_subvector_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v2f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
 ; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; CHECK-NEXT:    ret
   %ret = call <1 x double> @llvm.experimental.vector.extract.v1f64.v2f64(<2 x double> %op, i64 1)
   ret <1 x double> %ret
 }
 
 define void @extract_subvector_v4f64(<4 x double>* %a, <2 x double>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: extract_subvector_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
 ; CHECK-NEXT:    str q0, [x1]
 ; CHECK-NEXT:    ret
   %op = load <4 x double>, <4 x double>* %a
   %ret = call <2 x double> @llvm.experimental.vector.extract.v2f64.v4f64(<4 x double> %op, i64 2)
   store <2 x double> %ret, <2 x double>* %b
   ret void
 }
 
 define void @extract_subvector_v8f64(<8 x double>* %a, <4 x double>* %b) #0 {
-; VBITS_EQ_256-LABEL: extract_subvector_v8f64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    st1d { z0.d }, p0, [x1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: extract_subvector_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: extract_subvector_v8f64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl4
 ; VBITS_GE_512-NEXT:    ext z0.b, z0.b, z0.b, #32
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_512-NEXT:    ret
   %op = load <8 x double>, <8 x double>* %a
   %ret = call <4 x double> @llvm.experimental.vector.extract.v4f64.v8f64(<8 x double> %op, i64 4)
   store <4 x double> %ret, <4 x double>* %b
   ret void
 }
 
 define void @extract_subvector_v16f64(<16 x double>* %a, <8 x double>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q1, [x1, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: extract_subvector_v16f64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_1024-NEXT:    ext z0.b, z0.b, z0.b, #64
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_1024-NEXT:    ret
   %op = load <16 x double>, <16 x double>* %a
   %ret = call <8 x double> @llvm.experimental.vector.extract.v8f64.v16f64(<16 x double> %op, i64 8)
   store <8 x double> %ret, <8 x double>* %b
   ret void
 }
 
 define void @extract_subvector_v32f64(<32 x double>* %a, <16 x double>* %b) #0 {
+; NO_SVE-LABEL: extract_subvector_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #128]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #192]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: extract_subvector_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #20
+; VBITS_GE_256-NEXT:    mov x10, #28
+; VBITS_GE_256-NEXT:    mov x11, #24
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: extract_subvector_v32f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_2048-NEXT:    ext z0.b, z0.b, z0.b, #128
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x1]
 ; VBITS_GE_2048-NEXT:    ret

llvm/test/CodeGen/AArch64/sve-fixed-length-float-compares.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024

 ; NO_SVE-NOT: z{0-9}
 
 ;
 ; FCMP OEQ
 ;
 
 ; Don't use SVE for 64-bit vectors.
 define <4 x i16> @fcmp_oeq_v4f16(<4 x half> %op1, <4 x half> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq v0.4h, v0.4h, v1.4h
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v4f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq v0.4h, v0.4h, v1.4h
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <4 x half> %op1, %op2
   %sext = sext <4 x i1> %cmp to <4 x i16>
   ret <4 x i16> %sext
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <8 x i16> @fcmp_oeq_v8f16(<8 x half> %op1, <8 x half> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v8f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq v0.8h, v0.8h, v1.8h
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <8 x half> %op1, %op2
   %sext = sext <8 x i1> %cmp to <8 x i16>
   ret <8 x i16> %sext
 }
 
 define void @fcmp_oeq_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmeq v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp oeq <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 define void @fcmp_oeq_v32f16(<32 x half>* %a, <32 x half>* %b, <32 x i16>* %c) #0 {
 ; Ensure sensible type legalisation
-; VBITS_EQ_256-LABEL: fcmp_oeq_v32f16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z3.h }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    fcmeq p1.h, p0/z, z0.h, z2.h
-; VBITS_EQ_256-NEXT:    fcmeq p2.h, p0/z, z1.h, z3.h
-; VBITS_EQ_256-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    mov z1.h, p2/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    st1h { z0.h }, p0, [x2, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    st1h { z1.h }, p0, [x2]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: fcmp_oeq_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    fcmeq v2.8h, v2.8h, v5.8h
+; NO_SVE-NEXT:    ldp q6, q1, [x1]
+; NO_SVE-NEXT:    stp q0, q2, [x2, #32]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, v6.8h
+; NO_SVE-NEXT:    fcmeq v1.8h, v4.8h, v1.8h
+; NO_SVE-NEXT:    stp q3, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z0.h, z2.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z1.h, z3.h
+; VBITS_GE_256-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z1.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x2, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: fcmp_oeq_v32f16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p1.h, p0/z, z0.h, z1.h
 ; VBITS_GE_512-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x2]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <32 x half>, <32 x half>* %a
   %op2 = load <32 x half>, <32 x half>* %b
   %cmp = fcmp oeq <32 x half> %op1, %op2
   %sext = sext <32 x i1> %cmp to <32 x i16>
   store <32 x i16> %sext, <32 x i16>* %c
   ret void
 }
 
 define void @fcmp_oeq_v64f16(<64 x half>* %a, <64 x half>* %b, <64 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q17, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #96]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v5.8h, v0.8h
+; NO_SVE-NEXT:    ldp q7, q16, [x1, #96]
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, v7.8h
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v2.8h, v2.8h, v16.8h
+; NO_SVE-NEXT:    ldp q5, q7, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, v5.8h
+; NO_SVE-NEXT:    ldp q16, q5, [x0]
+; NO_SVE-NEXT:    fcmeq v4.8h, v4.8h, v7.8h
+; NO_SVE-NEXT:    ldp q7, q18, [x1]
+; NO_SVE-NEXT:    stp q3, q4, [x2, #64]
+; NO_SVE-NEXT:    stp q1, q2, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v1.8h, v6.8h, v17.8h
+; NO_SVE-NEXT:    fcmeq v2.8h, v16.8h, v7.8h
+; NO_SVE-NEXT:    fcmeq v3.8h, v5.8h, v18.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v64f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z5.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z6.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z7.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z2.h, z5.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z1.h, z4.h
+; VBITS_GE_256-NEXT:    mov z1.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z0.h, z6.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z3.h, z7.h
+; VBITS_GE_256-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x2, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x2, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x2, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: fcmp_oeq_v64f16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, z1.h
 ; VBITS_GE_1024-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x2]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <64 x half>, <64 x half>* %a
   %op2 = load <64 x half>, <64 x half>* %b
   %cmp = fcmp oeq <64 x half> %op1, %op2
   %sext = sext <64 x i1> %cmp to <64 x i16>
   store <64 x i16> %sext, <64 x i16>* %c
   ret void
 }
 
 define void @fcmp_oeq_v128f16(<128 x half>* %a, <128 x half>* %b, <128 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v128f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    str d10, [sp, #-32]! // 8-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 32
+; NO_SVE-NEXT:    stp d9, d8, [sp, #16] // 16-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_offset b8, -8
+; NO_SVE-NEXT:    .cfi_offset b9, -16
+; NO_SVE-NEXT:    .cfi_offset b10, -32
+; NO_SVE-NEXT:    ldp q25, q0, [x1, #224]
+; NO_SVE-NEXT:    ldp q2, q1, [x0, #224]
+; NO_SVE-NEXT:    fcmeq v2.8h, v2.8h, v25.8h
+; NO_SVE-NEXT:    ldp q6, q5, [x0, #160]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldp q29, q28, [x1, #160]
+; NO_SVE-NEXT:    ldp q4, q3, [x0, #192]
+; NO_SVE-NEXT:    fcmeq v5.8h, v5.8h, v28.8h
+; NO_SVE-NEXT:    ldp q27, q26, [x1, #192]
+; NO_SVE-NEXT:    fcmeq v4.8h, v4.8h, v27.8h
+; NO_SVE-NEXT:    ldp q16, q7, [x0, #128]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, v26.8h
+; NO_SVE-NEXT:    ldp q18, q17, [x0, #96]
+; NO_SVE-NEXT:    ldp q20, q19, [x0, #64]
+; NO_SVE-NEXT:    ldp q31, q30, [x1, #128]
+; NO_SVE-NEXT:    ldp q9, q8, [x1, #96]
+; NO_SVE-NEXT:    ldp q1, q28, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v1.8h, v20.8h, v1.8h
+; NO_SVE-NEXT:    ldp q22, q21, [x0, #32]
+; NO_SVE-NEXT:    ldp q24, q23, [x0]
+; NO_SVE-NEXT:    ldp q26, q25, [x1, #32]
+; NO_SVE-NEXT:    ldp q10, q27, [x1]
+; NO_SVE-NEXT:    stp q4, q3, [x2, #192]
+; NO_SVE-NEXT:    stp q2, q0, [x2, #224]
+; NO_SVE-NEXT:    fcmeq v0.8h, v6.8h, v29.8h
+; NO_SVE-NEXT:    fcmeq v2.8h, v7.8h, v30.8h
+; NO_SVE-NEXT:    fcmeq v3.8h, v16.8h, v31.8h
+; NO_SVE-NEXT:    fcmeq v4.8h, v17.8h, v8.8h
+; NO_SVE-NEXT:    stp q0, q5, [x2, #160]
+; NO_SVE-NEXT:    fcmeq v5.8h, v18.8h, v9.8h
+; NO_SVE-NEXT:    fcmeq v0.8h, v19.8h, v28.8h
+; NO_SVE-NEXT:    stp q3, q2, [x2, #128]
+; NO_SVE-NEXT:    fcmeq v2.8h, v21.8h, v25.8h
+; NO_SVE-NEXT:    fcmeq v3.8h, v22.8h, v26.8h
+; NO_SVE-NEXT:    stp q5, q4, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v4.8h, v23.8h, v27.8h
+; NO_SVE-NEXT:    stp q1, q0, [x2, #64]
+; NO_SVE-NEXT:    fcmeq v0.8h, v24.8h, v10.8h
+; NO_SVE-NEXT:    ldp d9, d8, [sp, #16] // 16-byte Folded Reload
+; NO_SVE-NEXT:    stp q3, q2, [x2, #32]
+; NO_SVE-NEXT:    stp q0, q4, [x2]
+; NO_SVE-NEXT:    ldr d10, [sp], #32 // 8-byte Folded Reload
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v128f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #96
+; VBITS_GE_256-NEXT:    mov x9, #112
+; VBITS_GE_256-NEXT:    mov x10, #64
+; VBITS_GE_256-NEXT:    mov x11, #80
+; VBITS_GE_256-NEXT:    mov x12, #32
+; VBITS_GE_256-NEXT:    mov x13, #48
+; VBITS_GE_256-NEXT:    mov x14, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z5.h }, p0/z, [x0, x13, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z6.h }, p0/z, [x0, x14, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z7.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1h { z16.h }, p0/z, [x1, x13, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z17.h }, p0/z, [x1, x14, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z20.h }, p0/z, [x1, x11, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z21.h }, p0/z, [x1, x12, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z19.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z22.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z18.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z23.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z6.h, z17.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z5.h, z16.h
+; VBITS_GE_256-NEXT:    mov z5.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z6.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z4.h, z21.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z3.h, z20.h
+; VBITS_GE_256-NEXT:    mov z3.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z4.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z2.h, z22.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z1.h, z19.h
+; VBITS_GE_256-NEXT:    mov z1.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.h, p0/z, z0.h, z18.h
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z7.h, z23.h
+; VBITS_GE_256-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z7.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x2, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x2, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x2, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x2, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x2, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z6.h }, p0, [x2, x13, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z5.h }, p0, [x2, x14, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: fcmp_oeq_v128f16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, z1.h
 ; VBITS_GE_2048-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x2]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <128 x half>, <128 x half>* %a
   %op2 = load <128 x half>, <128 x half>* %b
   %cmp = fcmp oeq <128 x half> %op1, %op2
   %sext = sext <128 x i1> %cmp to <128 x i16>
   store <128 x i16> %sext, <128 x i16>* %c
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x i32> @fcmp_oeq_v2f32(<2 x float> %op1, <2 x float> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq v0.2s, v0.2s, v1.2s
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq v0.2s, v0.2s, v1.2s
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <2 x float> %op1, %op2
   %sext = sext <2 x i1> %cmp to <2 x i32>
   ret <2 x i32> %sext
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x i32> @fcmp_oeq_v4f32(<4 x float> %op1, <4 x float> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq v0.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <4 x float> %op1, %op2
   %sext = sext <4 x i1> %cmp to <4 x i32>
   ret <4 x i32> %sext
 }
 
 define void @fcmp_oeq_v8f32(<8 x float>* %a, <8 x float>* %b, <8 x i32>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.4s, v1.4s, v0.4s
+; NO_SVE-NEXT:    fcmeq v1.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p1.s, p0/z, z0.s, z1.s
 ; CHECK-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <8 x float>, <8 x float>* %a
   %op2 = load <8 x float>, <8 x float>* %b
   %cmp = fcmp oeq <8 x float> %op1, %op2
   %sext = sext <8 x i1> %cmp to <8 x i32>
   store <8 x i32> %sext, <8 x i32>* %c
   ret void
 }
 
 define void @fcmp_oeq_v16f32(<16 x float>* %a, <16 x float>* %b, <16 x i32>* %c) #0 {
 ; Ensure sensible type legalisation
-; VBITS_EQ_256-LABEL: fcmp_oeq_v16f32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z3.s }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    fcmeq p1.s, p0/z, z0.s, z2.s
-; VBITS_EQ_256-NEXT:    fcmeq p2.s, p0/z, z1.s, z3.s
-; VBITS_EQ_256-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    mov z1.s, p2/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    st1w { z0.s }, p0, [x2, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    st1w { z1.s }, p0, [x2]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: fcmp_oeq_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.4s, v1.4s, v0.4s
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, v5.4s
+; NO_SVE-NEXT:    ldp q6, q1, [x1]
+; NO_SVE-NEXT:    stp q0, q2, [x2, #32]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, v6.4s
+; NO_SVE-NEXT:    fcmeq v1.4s, v4.4s, v1.4s
+; NO_SVE-NEXT:    stp q3, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z0.s, z2.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z1.s, z3.s
+; VBITS_GE_256-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z1.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x2, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: fcmp_oeq_v16f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p1.s, p0/z, z0.s, z1.s
 ; VBITS_GE_512-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x2]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <16 x float>, <16 x float>* %a
   %op2 = load <16 x float>, <16 x float>* %b
   %cmp = fcmp oeq <16 x float> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i32>
   store <16 x i32> %sext, <16 x i32>* %c
   ret void
 }
 
 define void @fcmp_oeq_v32f32(<32 x float>* %a, <32 x float>* %b, <32 x i32>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q17, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #96]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.4s, v5.4s, v0.4s
+; NO_SVE-NEXT:    ldp q7, q16, [x1, #96]
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, v7.4s
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, v16.4s
+; NO_SVE-NEXT:    ldp q5, q7, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, v5.4s
+; NO_SVE-NEXT:    ldp q16, q5, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, v7.4s
+; NO_SVE-NEXT:    ldp q7, q18, [x1]
+; NO_SVE-NEXT:    stp q3, q4, [x2, #64]
+; NO_SVE-NEXT:    stp q1, q2, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v1.4s, v6.4s, v17.4s
+; NO_SVE-NEXT:    fcmeq v2.4s, v16.4s, v7.4s
+; NO_SVE-NEXT:    fcmeq v3.4s, v5.4s, v18.4s
+; NO_SVE-NEXT:    stp q0, q1, [x2, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z2.s, z5.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z1.s, z4.s
+; VBITS_GE_256-NEXT:    mov z1.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z0.s, z6.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, z7.s
+; VBITS_GE_256-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x2, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x2, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x2, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: fcmp_oeq_v32f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.s, p0/z, z0.s, z1.s
 ; VBITS_GE_1024-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x2]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <32 x float>, <32 x float>* %a
   %op2 = load <32 x float>, <32 x float>* %b
   %cmp = fcmp oeq <32 x float> %op1, %op2
   %sext = sext <32 x i1> %cmp to <32 x i32>
   store <32 x i32> %sext, <32 x i32>* %c
   ret void
 }
 
 define void @fcmp_oeq_v64f32(<64 x float>* %a, <64 x float>* %b, <64 x i32>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    str d10, [sp, #-32]! // 8-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 32
+; NO_SVE-NEXT:    stp d9, d8, [sp, #16] // 16-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_offset b8, -8
+; NO_SVE-NEXT:    .cfi_offset b9, -16
+; NO_SVE-NEXT:    .cfi_offset b10, -32
+; NO_SVE-NEXT:    ldp q25, q0, [x1, #224]
+; NO_SVE-NEXT:    ldp q2, q1, [x0, #224]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, v25.4s
+; NO_SVE-NEXT:    ldp q6, q5, [x0, #160]
+; NO_SVE-NEXT:    fcmeq v0.4s, v1.4s, v0.4s
+; NO_SVE-NEXT:    ldp q29, q28, [x1, #160]
+; NO_SVE-NEXT:    ldp q4, q3, [x0, #192]
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, v28.4s
+; NO_SVE-NEXT:    ldp q27, q26, [x1, #192]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, v27.4s
+; NO_SVE-NEXT:    ldp q16, q7, [x0, #128]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, v26.4s
+; NO_SVE-NEXT:    ldp q18, q17, [x0, #96]
+; NO_SVE-NEXT:    ldp q20, q19, [x0, #64]
+; NO_SVE-NEXT:    ldp q31, q30, [x1, #128]
+; NO_SVE-NEXT:    ldp q9, q8, [x1, #96]
+; NO_SVE-NEXT:    ldp q1, q28, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v1.4s, v20.4s, v1.4s
+; NO_SVE-NEXT:    ldp q22, q21, [x0, #32]
+; NO_SVE-NEXT:    ldp q24, q23, [x0]
+; NO_SVE-NEXT:    ldp q26, q25, [x1, #32]
+; NO_SVE-NEXT:    ldp q10, q27, [x1]
+; NO_SVE-NEXT:    stp q4, q3, [x2, #192]
+; NO_SVE-NEXT:    stp q2, q0, [x2, #224]
+; NO_SVE-NEXT:    fcmeq v0.4s, v6.4s, v29.4s
+; NO_SVE-NEXT:    fcmeq v2.4s, v7.4s, v30.4s
+; NO_SVE-NEXT:    fcmeq v3.4s, v16.4s, v31.4s
+; NO_SVE-NEXT:    fcmeq v4.4s, v17.4s, v8.4s
+; NO_SVE-NEXT:    stp q0, q5, [x2, #160]
+; NO_SVE-NEXT:    fcmeq v5.4s, v18.4s, v9.4s
+; NO_SVE-NEXT:    fcmeq v0.4s, v19.4s, v28.4s
+; NO_SVE-NEXT:    stp q3, q2, [x2, #128]
+; NO_SVE-NEXT:    fcmeq v2.4s, v21.4s, v25.4s
+; NO_SVE-NEXT:    fcmeq v3.4s, v22.4s, v26.4s
+; NO_SVE-NEXT:    stp q5, q4, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v4.4s, v23.4s, v27.4s
+; NO_SVE-NEXT:    stp q1, q0, [x2, #64]
+; NO_SVE-NEXT:    fcmeq v0.4s, v24.4s, v10.4s
+; NO_SVE-NEXT:    ldp d9, d8, [sp, #16] // 16-byte Folded Reload
+; NO_SVE-NEXT:    stp q3, q2, [x2, #32]
+; NO_SVE-NEXT:    stp q0, q4, [x2]
+; NO_SVE-NEXT:    ldr d10, [sp], #32 // 8-byte Folded Reload
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #56
+; VBITS_GE_256-NEXT:    mov x10, #32
+; VBITS_GE_256-NEXT:    mov x11, #40
+; VBITS_GE_256-NEXT:    mov x12, #16
+; VBITS_GE_256-NEXT:    mov x13, #24
+; VBITS_GE_256-NEXT:    mov x14, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z16.s }, p0/z, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p0/z, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z20.s }, p0/z, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z21.s }, p0/z, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z19.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z22.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z18.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z23.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z6.s, z17.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z5.s, z16.s
+; VBITS_GE_256-NEXT:    mov z5.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z6.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z4.s, z21.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, z20.s
+; VBITS_GE_256-NEXT:    mov z3.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z4.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z2.s, z22.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z1.s, z19.s
+; VBITS_GE_256-NEXT:    mov z1.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z0.s, z18.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z7.s, z23.s
+; VBITS_GE_256-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z7.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x2, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x2, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x2, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x2, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x2, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p0, [x2, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x2, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: fcmp_oeq_v64f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, z1.s
 ; VBITS_GE_2048-NEXT:    mov z0.s, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x2]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <64 x float>, <64 x float>* %a
   %op2 = load <64 x float>, <64 x float>* %b
   %cmp = fcmp oeq <64 x float> %op1, %op2
   %sext = sext <64 x i1> %cmp to <64 x i32>
   store <64 x i32> %sext, <64 x i32>* %c
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x i64> @fcmp_oeq_v1f64(<1 x double> %op1, <1 x double> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v1f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq d0, d0, d1
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v1f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq d0, d0, d1
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <1 x double> %op1, %op2
   %sext = sext <1 x i1> %cmp to <1 x i64>
   ret <1 x i64> %sext
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x i64> @fcmp_oeq_v2f64(<2 x double> %op1, <2 x double> %op2) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, v1.2d
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v2f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    fcmeq v0.2d, v0.2d, v1.2d
 ; CHECK-NEXT:    ret
   %cmp = fcmp oeq <2 x double> %op1, %op2
   %sext = sext <2 x i1> %cmp to <2 x i64>
   ret <2 x i64> %sext
 }
 
 define void @fcmp_oeq_v4f64(<4 x double>* %a, <4 x double>* %b, <4 x i64>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.2d, v1.2d, v0.2d
+; NO_SVE-NEXT:    fcmeq v1.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oeq_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p1.d, p0/z, z0.d, z1.d
 ; CHECK-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <4 x double>, <4 x double>* %a
   %op2 = load <4 x double>, <4 x double>* %b
   %cmp = fcmp oeq <4 x double> %op1, %op2
   %sext = sext <4 x i1> %cmp to <4 x i64>
   store <4 x i64> %sext, <4 x i64>* %c
   ret void
 }
 
 define void @fcmp_oeq_v8f64(<8 x double>* %a, <8 x double>* %b, <8 x i64>* %c) #0 {
 ; Ensure sensible type legalisation
-; VBITS_EQ_256-LABEL: fcmp_oeq_v8f64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    fcmeq p1.d, p0/z, z0.d, z2.d
-; VBITS_EQ_256-NEXT:    fcmeq p2.d, p0/z, z1.d, z3.d
-; VBITS_EQ_256-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    mov z1.d, p2/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    st1d { z0.d }, p0, [x2, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    st1d { z1.d }, p0, [x2]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: fcmp_oeq_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.2d, v1.2d, v0.2d
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    ldp q6, q1, [x1]
+; NO_SVE-NEXT:    stp q0, q2, [x2, #32]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, v6.2d
+; NO_SVE-NEXT:    fcmeq v1.2d, v4.2d, v1.2d
+; NO_SVE-NEXT:    stp q3, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z0.d, z2.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, z3.d
+; VBITS_GE_256-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z1.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x2, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: fcmp_oeq_v8f64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p1.d, p0/z, z0.d, z1.d
 ; VBITS_GE_512-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x2]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <8 x double>, <8 x double>* %a
   %op2 = load <8 x double>, <8 x double>* %b
   %cmp = fcmp oeq <8 x double> %op1, %op2
   %sext = sext <8 x i1> %cmp to <8 x i64>
   store <8 x i64> %sext, <8 x i64>* %c
   ret void
 }
 
 define void @fcmp_oeq_v16f64(<16 x double>* %a, <16 x double>* %b, <16 x i64>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q17, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #96]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.2d, v5.2d, v0.2d
+; NO_SVE-NEXT:    ldp q7, q16, [x1, #96]
+; NO_SVE-NEXT:    fcmeq v1.2d, v1.2d, v7.2d
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    ldp q5, q7, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, v5.2d
+; NO_SVE-NEXT:    ldp q16, q5, [x0]
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, v7.2d
+; NO_SVE-NEXT:    ldp q7, q18, [x1]
+; NO_SVE-NEXT:    stp q3, q4, [x2, #64]
+; NO_SVE-NEXT:    stp q1, q2, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v1.2d, v6.2d, v17.2d
+; NO_SVE-NEXT:    fcmeq v2.2d, v16.2d, v7.2d
+; NO_SVE-NEXT:    fcmeq v3.2d, v5.2d, v18.2d
+; NO_SVE-NEXT:    stp q0, q1, [x2, #32]
+; NO_SVE-NEXT:    stp q2, q3, [x2]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z2.d, z5.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, z4.d
+; VBITS_GE_256-NEXT:    mov z1.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z0.d, z6.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z3.d, z7.d
+; VBITS_GE_256-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x2, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x2, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x2, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: fcmp_oeq_v16f64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.d, p0/z, z0.d, z1.d
 ; VBITS_GE_1024-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x2]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <16 x double>, <16 x double>* %a
   %op2 = load <16 x double>, <16 x double>* %b
   %cmp = fcmp oeq <16 x double> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i64>
   store <16 x i64> %sext, <16 x i64>* %c
   ret void
 }
 
 define void @fcmp_oeq_v32f64(<32 x double>* %a, <32 x double>* %b, <32 x i64>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oeq_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    str d10, [sp, #-32]! // 8-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 32
+; NO_SVE-NEXT:    stp d9, d8, [sp, #16] // 16-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_offset b8, -8
+; NO_SVE-NEXT:    .cfi_offset b9, -16
+; NO_SVE-NEXT:    .cfi_offset b10, -32
+; NO_SVE-NEXT:    ldp q25, q0, [x1, #224]
+; NO_SVE-NEXT:    ldp q2, q1, [x0, #224]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, v25.2d
+; NO_SVE-NEXT:    ldp q6, q5, [x0, #160]
+; NO_SVE-NEXT:    fcmeq v0.2d, v1.2d, v0.2d
+; NO_SVE-NEXT:    ldp q29, q28, [x1, #160]
+; NO_SVE-NEXT:    ldp q4, q3, [x0, #192]
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, v28.2d
+; NO_SVE-NEXT:    ldp q27, q26, [x1, #192]
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, v27.2d
+; NO_SVE-NEXT:    ldp q16, q7, [x0, #128]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, v26.2d
+; NO_SVE-NEXT:    ldp q18, q17, [x0, #96]
+; NO_SVE-NEXT:    ldp q20, q19, [x0, #64]
+; NO_SVE-NEXT:    ldp q31, q30, [x1, #128]
+; NO_SVE-NEXT:    ldp q9, q8, [x1, #96]
+; NO_SVE-NEXT:    ldp q1, q28, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v1.2d, v20.2d, v1.2d
+; NO_SVE-NEXT:    ldp q22, q21, [x0, #32]
+; NO_SVE-NEXT:    ldp q24, q23, [x0]
+; NO_SVE-NEXT:    ldp q26, q25, [x1, #32]
+; NO_SVE-NEXT:    ldp q10, q27, [x1]
+; NO_SVE-NEXT:    stp q4, q3, [x2, #192]
+; NO_SVE-NEXT:    stp q2, q0, [x2, #224]
+; NO_SVE-NEXT:    fcmeq v0.2d, v6.2d, v29.2d
+; NO_SVE-NEXT:    fcmeq v2.2d, v7.2d, v30.2d
+; NO_SVE-NEXT:    fcmeq v3.2d, v16.2d, v31.2d
+; NO_SVE-NEXT:    fcmeq v4.2d, v17.2d, v8.2d
+; NO_SVE-NEXT:    stp q0, q5, [x2, #160]
+; NO_SVE-NEXT:    fcmeq v5.2d, v18.2d, v9.2d
+; NO_SVE-NEXT:    fcmeq v0.2d, v19.2d, v28.2d
+; NO_SVE-NEXT:    stp q3, q2, [x2, #128]
+; NO_SVE-NEXT:    fcmeq v2.2d, v21.2d, v25.2d
+; NO_SVE-NEXT:    fcmeq v3.2d, v22.2d, v26.2d
+; NO_SVE-NEXT:    stp q5, q4, [x2, #96]
+; NO_SVE-NEXT:    fcmeq v4.2d, v23.2d, v27.2d
+; NO_SVE-NEXT:    stp q1, q0, [x2, #64]
+; NO_SVE-NEXT:    fcmeq v0.2d, v24.2d, v10.2d
+; NO_SVE-NEXT:    ldp d9, d8, [sp, #16] // 16-byte Folded Reload
+; NO_SVE-NEXT:    stp q3, q2, [x2, #32]
+; NO_SVE-NEXT:    stp q0, q4, [x2]
+; NO_SVE-NEXT:    ldr d10, [sp], #32 // 8-byte Folded Reload
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: fcmp_oeq_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #28
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    mov x12, #8
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov x14, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p0/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p0/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z20.d }, p0/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z21.d }, p0/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z19.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z22.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z23.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z6.d, z17.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z5.d, z16.d
+; VBITS_GE_256-NEXT:    mov z5.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z6.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z4.d, z21.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z3.d, z20.d
+; VBITS_GE_256-NEXT:    mov z3.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z4.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z2.d, z22.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, z19.d
+; VBITS_GE_256-NEXT:    mov z1.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z2.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z0.d, z18.d
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z7.d, z23.d
+; VBITS_GE_256-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z7.d, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x2, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x2, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x2, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x2, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x2, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x2, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x2, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: fcmp_oeq_v32f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.d, p0/z, z0.d, z1.d
 ; VBITS_GE_2048-NEXT:    mov z0.d, p1/z, #-1 // =0xffffffffffffffff
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x2]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <32 x double>, <32 x double>* %a
   %op2 = load <32 x double>, <32 x double>* %b
   %cmp = fcmp oeq <32 x double> %op1, %op2
   %sext = sext <32 x i1> %cmp to <32 x i64>
   store <32 x i64> %sext, <32 x i64>* %c
   ret void
 }
 
 ;
 ; FCMP UEQ
 ;
 
 define void @fcmp_ueq_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ueq_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q3, q0, [x1]
+; NO_SVE-NEXT:    ldp q2, q1, [x0]
+; NO_SVE-NEXT:    fcmgt v4.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    fcmgt v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    orr v0.16b, v0.16b, v4.16b
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    orr v1.16b, v2.16b, v1.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q1, q0, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ueq_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmuo p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    fcmeq p2.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov p1.b, p2/m, p2.b
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ueq <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP ONE
 ;
 
 define void @fcmp_one_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_one_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q3, q0, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v5.8h, v1.8h, v3.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v3.8h, v1.8h
+; NO_SVE-NEXT:    fcmgt v4.8h, v2.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v2.8h
+; NO_SVE-NEXT:    orr v1.16b, v1.16b, v5.16b
+; NO_SVE-NEXT:    orr v0.16b, v0.16b, v4.16b
+; NO_SVE-NEXT:    stp q1, q0, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_one_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    fcmgt p2.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov p1.b, p2/m, p2.b
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp one <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP UNE
 ;
 
 define void @fcmp_une_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_une_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmeq v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_une_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmne p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp une <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP OGT
 ;
 
 define void @fcmp_ogt_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ogt_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ogt_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ogt <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP UGT
 ;
 
 define void @fcmp_ugt_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ugt_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ugt_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    eor z0.d, z0.d, z1.d
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ugt <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP OLT
 ;
 
 define void @fcmp_olt_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_olt_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_olt_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp olt <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP ULT
 ;
 
 define void @fcmp_ult_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ult_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ult_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    eor z0.d, z0.d, z1.d
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ult <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP OGE
 ;
 
 define void @fcmp_oge_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_oge_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_oge_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp oge <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP UGE
 ;
 
 define void @fcmp_uge_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_uge_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_uge_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    eor z0.d, z0.d, z1.d
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp uge <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP OLE
 ;
 
 define void @fcmp_ole_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ole_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ole_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ole <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP ULE
 ;
 
 define void @fcmp_ule_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ule_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ule_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    eor z0.d, z0.d, z1.d
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ule <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP UNO
 ;
 
 define void @fcmp_uno_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_uno_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q3, q0, [x1]
+; NO_SVE-NEXT:    ldp q2, q1, [x0]
+; NO_SVE-NEXT:    fcmge v4.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    fcmgt v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    orr v0.16b, v0.16b, v4.16b
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    orr v1.16b, v2.16b, v1.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q1, q0, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_uno_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmuo p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp uno <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP ORD
 ;
 
 define void @fcmp_ord_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ord_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q3, q0, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v5.8h, v1.8h, v3.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v3.8h, v1.8h
+; NO_SVE-NEXT:    fcmge v4.8h, v2.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v0.8h, v0.8h, v2.8h
+; NO_SVE-NEXT:    orr v1.16b, v1.16b, v5.16b
+; NO_SVE-NEXT:    orr v0.16b, v0.16b, v4.16b
+; NO_SVE-NEXT:    stp q1, q0, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ord_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmuo p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z1.h, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    eor z0.d, z0.d, z1.d
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp ord <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP EQ
 ;
 
 define void @fcmp_eq_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_eq_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmeq v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_eq_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp fast oeq <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP NE
 ;
 
 define void @fcmp_ne_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ne_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmeq v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    mvn v0.16b, v0.16b
+; NO_SVE-NEXT:    mvn v1.16b, v1.16b
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ne_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmne p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp fast one <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP GT
 ;
 
 define void @fcmp_gt_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_gt_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmgt v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmgt v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_gt_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp fast ogt <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP LT
 ;
 
 define void @fcmp_lt_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_lt_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x1, #16]
+; NO_SVE-NEXT:    ldr q1, [x0, #16]
+; NO_SVE-NEXT:    mov h2, v0.h[1]
+; NO_SVE-NEXT:    mov h4, v0.h[3]
+; NO_SVE-NEXT:    mov h3, v1.h[1]
+; NO_SVE-NEXT:    mov h5, v1.h[3]
+; NO_SVE-NEXT:    mov h6, v1.h[6]
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h2, v0.h[2]
+; NO_SVE-NEXT:    mov h3, v1.h[2]
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    csetm w9, lt
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h2, v0.h[4]
+; NO_SVE-NEXT:    mov h3, v1.h[4]
+; NO_SVE-NEXT:    csetm w10, lt
+; NO_SVE-NEXT:    fcmp h5, h4
+; NO_SVE-NEXT:    mov h4, v0.h[5]
+; NO_SVE-NEXT:    mov h5, v1.h[5]
+; NO_SVE-NEXT:    mov h1, v1.h[7]
+; NO_SVE-NEXT:    csetm w11, lt
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h3, v0.h[6]
+; NO_SVE-NEXT:    ldr q2, [x0]
+; NO_SVE-NEXT:    mov h0, v0.h[7]
+; NO_SVE-NEXT:    csetm w12, lt
+; NO_SVE-NEXT:    fcmp h5, h4
+; NO_SVE-NEXT:    ldr q4, [x1]
+; NO_SVE-NEXT:    mov h5, v2.h[1]
+; NO_SVE-NEXT:    csetm w13, lt
+; NO_SVE-NEXT:    fcmp h6, h3
+; NO_SVE-NEXT:    mov h3, v4.h[1]
+; NO_SVE-NEXT:    mov h6, v4.h[3]
+; NO_SVE-NEXT:    csetm w14, lt
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    mov h0, v4.h[2]
+; NO_SVE-NEXT:    mov h1, v2.h[2]
+; NO_SVE-NEXT:    csetm w15, lt
+; NO_SVE-NEXT:    fcmp h5, h3
+; NO_SVE-NEXT:    fmov s3, w9
+; NO_SVE-NEXT:    csetm w16, lt
+; NO_SVE-NEXT:    fcmp h2, h4
+; NO_SVE-NEXT:    mov v3.h[1], w8
+; NO_SVE-NEXT:    csetm w17, lt
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    mov h0, v2.h[3]
+; NO_SVE-NEXT:    mov h1, v4.h[4]
+; NO_SVE-NEXT:    mov v3.h[2], w10
+; NO_SVE-NEXT:    fmov s5, w17
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h0, h6
+; NO_SVE-NEXT:    mov h0, v2.h[4]
+; NO_SVE-NEXT:    mov v5.h[1], w16
+; NO_SVE-NEXT:    mov h6, v4.h[5]
+; NO_SVE-NEXT:    mov v3.h[3], w11
+; NO_SVE-NEXT:    mov v5.h[2], w8
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h0, h1
+; NO_SVE-NEXT:    mov h0, v2.h[5]
+; NO_SVE-NEXT:    mov h1, v4.h[6]
+; NO_SVE-NEXT:    mov v3.h[4], w12
+; NO_SVE-NEXT:    mov h4, v4.h[7]
+; NO_SVE-NEXT:    mov v5.h[3], w8
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h0, h6
+; NO_SVE-NEXT:    mov h0, v2.h[6]
+; NO_SVE-NEXT:    mov h2, v2.h[7]
+; NO_SVE-NEXT:    mov v5.h[4], w8
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h0, h1
+; NO_SVE-NEXT:    mov v3.h[5], w13
+; NO_SVE-NEXT:    mov v5.h[5], w8
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    fcmp h2, h4
+; NO_SVE-NEXT:    mov v3.h[6], w14
+; NO_SVE-NEXT:    mov v5.h[6], w8
+; NO_SVE-NEXT:    csetm w8, lt
+; NO_SVE-NEXT:    mov v3.h[7], w15
+; NO_SVE-NEXT:    mov v5.h[7], w8
+; NO_SVE-NEXT:    stp q5, q3, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_lt_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmgt p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp fast olt <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP GE
 ;
 
 define void @fcmp_ge_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_ge_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q3, [x1]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmge v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    fcmge v1.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    stp q0, q1, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_ge_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z0.h, z1.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %cmp = fcmp fast oge <16 x half> %op1, %op2
   %sext = sext <16 x i1> %cmp to <16 x i16>
   store <16 x i16> %sext, <16 x i16>* %c
   ret void
 }
 
 ;
 ; FCMP LE
 ;
 
 define void @fcmp_le_v16f16(<16 x half>* %a, <16 x half>* %b, <16 x i16>* %c) #0 {
+; NO_SVE-LABEL: fcmp_le_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x1, #16]
+; NO_SVE-NEXT:    ldr q1, [x0, #16]
+; NO_SVE-NEXT:    mov h2, v0.h[1]
+; NO_SVE-NEXT:    mov h4, v0.h[3]
+; NO_SVE-NEXT:    mov h3, v1.h[1]
+; NO_SVE-NEXT:    mov h5, v1.h[3]
+; NO_SVE-NEXT:    mov h6, v1.h[6]
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h2, v0.h[2]
+; NO_SVE-NEXT:    mov h3, v1.h[2]
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    csetm w9, le
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h2, v0.h[4]
+; NO_SVE-NEXT:    mov h3, v1.h[4]
+; NO_SVE-NEXT:    csetm w10, le
+; NO_SVE-NEXT:    fcmp h5, h4
+; NO_SVE-NEXT:    mov h4, v0.h[5]
+; NO_SVE-NEXT:    mov h5, v1.h[5]
+; NO_SVE-NEXT:    mov h1, v1.h[7]
+; NO_SVE-NEXT:    csetm w11, le
+; NO_SVE-NEXT:    fcmp h3, h2
+; NO_SVE-NEXT:    mov h3, v0.h[6]
+; NO_SVE-NEXT:    ldr q2, [x0]
+; NO_SVE-NEXT:    mov h0, v0.h[7]
+; NO_SVE-NEXT:    csetm w12, le
+; NO_SVE-NEXT:    fcmp h5, h4
+; NO_SVE-NEXT:    ldr q4, [x1]
+; NO_SVE-NEXT:    mov h5, v2.h[1]
+; NO_SVE-NEXT:    csetm w13, le
+; NO_SVE-NEXT:    fcmp h6, h3
+; NO_SVE-NEXT:    mov h3, v4.h[1]
+; NO_SVE-NEXT:    mov h6, v4.h[3]
+; NO_SVE-NEXT:    csetm w14, le
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    mov h0, v4.h[2]
+; NO_SVE-NEXT:    mov h1, v2.h[2]
+; NO_SVE-NEXT:    csetm w15, le
+; NO_SVE-NEXT:    fcmp h5, h3
+; NO_SVE-NEXT:    fmov s3, w9
+; NO_SVE-NEXT:    csetm w16, le
+; NO_SVE-NEXT:    fcmp h2, h4
+; NO_SVE-NEXT:    mov v3.h[1], w8
+; NO_SVE-NEXT:    csetm w17, le
+; NO_SVE-NEXT:    fcmp h1, h0
+; NO_SVE-NEXT:    mov h0, v2.h[3]
+; NO_SVE-NEXT:    mov h1, v4.h[4]
+; NO_SVE-NEXT:    mov v3.h[2], w10
+; NO_SVE-NEXT:    fmov s5, w17
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h0, h6
+; NO_SVE-NEXT:    mov h0, v2.h[4]
+; NO_SVE-NEXT:    mov v5.h[1], w16
+; NO_SVE-NEXT:    mov h6, v4.h[5]
+; NO_SVE-NEXT:    mov v3.h[3], w11
+; NO_SVE-NEXT:    mov v5.h[2], w8
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h0, h1
+; NO_SVE-NEXT:    mov h0, v2.h[5]
+; NO_SVE-NEXT:    mov h1, v4.h[6]
+; NO_SVE-NEXT:    mov v3.h[4], w12
+; NO_SVE-NEXT:    mov h4, v4.h[7]
+; NO_SVE-NEXT:    mov v5.h[3], w8
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h0, h6
+; NO_SVE-NEXT:    mov h0, v2.h[6]
+; NO_SVE-NEXT:    mov h2, v2.h[7]
+; NO_SVE-NEXT:    mov v5.h[4], w8
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h0, h1
+; NO_SVE-NEXT:    mov v3.h[5], w13
+; NO_SVE-NEXT:    mov v5.h[5], w8
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    fcmp h2, h4
+; NO_SVE-NEXT:    mov v3.h[6], w14
+; NO_SVE-NEXT:    mov v5.h[6], w8
+; NO_SVE-NEXT:    csetm w8, le
+; NO_SVE-NEXT:    mov v3.h[7], w15
+; NO_SVE-NEXT:    mov v5.h[7], w8
+; NO_SVE-NEXT:    stp q5, q3, [x2]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: fcmp_le_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    fcmge p1.h, p0/z, z1.h, z0.h
 ; CHECK-NEXT:    mov z0.h, p1/z, #-1 // =0xffffffffffffffff
 ; CHECK-NEXT:    st1h { z0.h }, p0, [x2]

llvm/test/CodeGen/AArch64/sve-fixed-length-frame-offests-crash.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Ensure we don't crash by trying to fold fixed length frame indexes into
 ; loads/stores that don't support an appropriate addressing mode, hence creating
 ; too many extra vregs during frame lowering, when we don't have an emergency
 ; spill slot.
 
 define dso_local void @func1(i64* %v1, i64* %v2, i64* %v3, i64* %v4, i64* %v5, i64* %v6, i64* %v7, i64* %v8,
+; CHECK-LABEL: func1:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    str x25, [sp, #-64]! // 8-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 64
+; CHECK-NEXT:    stp x24, x23, [sp, #16] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x22, x21, [sp, #32] // 16-byte Folded Spill
+; CHECK-NEXT:    stp x20, x19, [sp, #48] // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_offset w19, -8
+; CHECK-NEXT:    .cfi_offset w20, -16
+; CHECK-NEXT:    .cfi_offset w21, -24
+; CHECK-NEXT:    .cfi_offset w22, -32
+; CHECK-NEXT:    .cfi_offset w23, -40
+; CHECK-NEXT:    .cfi_offset w24, -48
+; CHECK-NEXT:    .cfi_offset w25, -64
+; CHECK-NEXT:    add x8, sp, #64
+; CHECK-NEXT:    add x9, sp, #128
+; CHECK-NEXT:    add x10, sp, #160
+; CHECK-NEXT:    add x11, sp, #192
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    add x20, sp, #192
+; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x8]
+; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x9]
+; CHECK-NEXT:    ld1d { z2.d }, p0/z, [x10]
+; CHECK-NEXT:    ld1d { z3.d }, p0/z, [x11]
+; CHECK-NEXT:    ldp x18, x19, [sp, #368]
+; CHECK-NEXT:    add x21, sp, #160
+; CHECK-NEXT:    add x22, sp, #128
+; CHECK-NEXT:    ldp x24, x14, [sp, #296]
+; CHECK-NEXT:    add x23, sp, #64
+; CHECK-NEXT:    ldr x25, [sp, #288]
+; CHECK-NEXT:    ldp x9, x8, [sp, #344]
+; CHECK-NEXT:    ldp x11, x10, [sp, #328]
+; CHECK-NEXT:    ldp x13, x12, [sp, #312]
+; CHECK-NEXT:    ldr x15, [sp, #120]
+; CHECK-NEXT:    ldur q4, [sp, #104]
+; CHECK-NEXT:    ldp x16, x17, [sp, #224]
+; CHECK-NEXT:    st1d { z3.d }, p0, [x20]
+; CHECK-NEXT:    st1d { z2.d }, p0, [x21]
+; CHECK-NEXT:    st1d { z1.d }, p0, [x22]
+; CHECK-NEXT:    st1d { z0.d }, p0, [x23]
+; CHECK-NEXT:    stp x18, x19, [sp, #368]
+; CHECK-NEXT:    stp x25, x24, [sp, #288]
+; CHECK-NEXT:    ldp x20, x19, [sp, #48] // 16-byte Folded Reload
+; CHECK-NEXT:    stp x16, x17, [sp, #224]
+; CHECK-NEXT:    ldp x22, x21, [sp, #32] // 16-byte Folded Reload
+; CHECK-NEXT:    stur q4, [sp, #104]
+; CHECK-NEXT:    ldp x24, x23, [sp, #16] // 16-byte Folded Reload
+; CHECK-NEXT:    str x15, [sp, #120]
+; CHECK-NEXT:    stp x14, x13, [sp, #304]
+; CHECK-NEXT:    stp x12, x11, [sp, #320]
+; CHECK-NEXT:    stp x10, x9, [sp, #336]
+; CHECK-NEXT:    str x8, [sp, #352]
+; CHECK-NEXT:    ldr x25, [sp], #64 // 8-byte Folded Reload
+; CHECK-NEXT:    b func2
                              i64* %v9, i64* %v10, i64* %v11, i64* %v12, i64* %v13, i64* %v14,  i64* %v15, i64* %v16,
                              i64* %v17, i64* %v18, i64* %v19, i64* %v20, i64* %v21, i64* %v22, i64* %v23, i64* %v24,
                              i64* %v25, i64* %v26, i64* %v27, i64* %v28, i64* %v29, i64* %v30, i64* %v31, i64* %v32,
                              i64* %v33, i64* %v34, i64* %v35, i64* %v36, i64* %v37, i64* %v38, i64* %v39, i64* %v40,
                              i64* %v41, i64* %v42, i64* %v43, i64* %v44, i64* %v45, i64* %v46, i64* %v47, i64* %v48,
                              i64 %v49) #0 {
-; CHECK-LABEL: func1
   tail call void @func2(i64* %v1, i64* %v2, i64* %v3, i64* %v4, i64* %v5, i64* %v6, i64* %v7, i64* %v8,
                         i64* %v9, i64* %v10, i64* %v11, i64* %v12, i64* undef, i64* %v14, i64* %v15, i64* %v16,
                         i64* %v17, i64* %v18, i64* %v19, i64* %v20, i64* %v21, i64* %v22, i64* %v23, i64* %v24,
                         i64* %v25, i64* %v26, i64* %v27, i64* %v28, i64* %v29, i64* %v30, i64* undef, i64* undef,
                         i64* undef, i64* undef, i64* undef, i64* undef, i64* %v37, i64* %v38, i64* %v39, i64* %v40,
                         i64* %v41, i64* %v42, i64* %v43, i64* %v44, i64* %v45, i64* undef, i64* %v47, i64* %v48,
                         i64 undef)
   ret void

llvm/test/CodeGen/AArch64/sve-fixed-length-insert-vector-elt.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -D#VBYTES=16  -check-prefix=NO_SVE
 ; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=VBITS_GE_256,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=VBITS_GE_256,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=VBITS_GE_256,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=VBITS_GE_256,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=VBITS_GE_256,VBITS_GE_512,VBITS_GE_1024

 ; NO_SVE-NOT: ptrue
 
 ;
 ; insertelement
 ;
 
 ; Don't use SVE for 64-bit vectors.
 define <4 x half> @insertelement_v4f16(<4 x half> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov h1, #5.00000000
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    mov v0.h[3], v1.h[0]
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v4f16:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov h1, #5.00000000
 ; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; VBITS_GE_256-NEXT:    mov v0.h[3], v1.h[0]
 ; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <4 x half> %op1, half 5.0, i64 3
     ret <4 x half> %r
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <8 x half> @insertelement_v8f16(<8 x half> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov h1, #5.00000000
+; NO_SVE-NEXT:    mov v0.h[7], v1.h[0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v8f16:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov h1, #5.00000000
 ; VBITS_GE_256-NEXT:    mov v0.h[7], v1.h[0]
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <8 x half> %op1, half 5.0, i64 7
     ret <8 x half> %r
 }
 
 define <16 x half> @insertelement_v16f16(<16 x half>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov h0, #5.00000000
+; NO_SVE-NEXT:    ldr q1, [x0, #16]
+; NO_SVE-NEXT:    mov v1.h[7], v0.h[0]
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v16f16:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    mov w9, #15
 ; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
 ; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_256-NEXT:    fmov h2, #5.00000000
 ; VBITS_GE_256-NEXT:    index z3.h, #0, #1
 ; VBITS_GE_256-NEXT:    ptrue p1.h
 ; VBITS_GE_256-NEXT:    mov z1.h, w9
 ; VBITS_GE_256-NEXT:    cmpeq p1.h, p1/z, z3.h, z1.h
 ; VBITS_GE_256-NEXT:    mov z0.h, p1/m, h2
 ; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x8]
 ; VBITS_GE_256-NEXT:    ret
     %op1 = load <16 x half>, <16 x half>* %a
     %r = insertelement <16 x half> %op1, half 5.0, i64 15
     ret <16 x half> %r
 }
 
 define <32 x half> @insertelement_v32f16(<32 x half>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov h2, #5.00000000
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    mov v3.h[7], v2.h[0]
+; NO_SVE-NEXT:    ldr q2, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: insertelement_v32f16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    mov w9, #31
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    fmov h2, #5.00000000
 ; VBITS_GE_512-NEXT:    index z3.h, #0, #1
 ; VBITS_GE_512-NEXT:    ptrue p1.h
 ; VBITS_GE_512-NEXT:    mov z1.h, w9
 ; VBITS_GE_512-NEXT:    cmpeq p1.h, p1/z, z3.h, z1.h
 ; VBITS_GE_512-NEXT:    mov z0.h, p1/m, h2
 ; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x8]
 ; VBITS_GE_512-NEXT:    ret
     %op1 = load <32 x half>, <32 x half>* %a
     %r = insertelement <32 x half> %op1, half 5.0, i64 31
     ret <32 x half> %r
 }
 
 define <64 x half> @insertelement_v64f16(<64 x half>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    fmov h5, #5.00000000
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    mov v7.h[7], v5.h[0]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: insertelement_v64f16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    mov w9, #63
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    fmov h2, #5.00000000
 ; VBITS_GE_1024-NEXT:    index z3.h, #0, #1
 ; VBITS_GE_1024-NEXT:    ptrue p1.h
 ; VBITS_GE_1024-NEXT:    mov z1.h, w9
 ; VBITS_GE_1024-NEXT:    cmpeq p1.h, p1/z, z3.h, z1.h
 ; VBITS_GE_1024-NEXT:    mov z0.h, p1/m, h2
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x8]
 ; VBITS_GE_1024-NEXT:    ret
     %op1 = load <64 x half>, <64 x half>* %a
     %r = insertelement <64 x half> %op1, half 5.0, i64 63
     ret <64 x half> %r
 }
 
 define <128 x half> @insertelement_v128f16(<128 x half>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v128f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q23, q24, [x0, #224]
+; NO_SVE-NEXT:    fmov h0, #5.00000000
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    mov v24.h[7], v0.h[0]
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    ldp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q17, q18, [x0, #128]
+; NO_SVE-NEXT:    ldp q19, q20, [x0, #192]
+; NO_SVE-NEXT:    ldp q21, q22, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q2, [x8]
+; NO_SVE-NEXT:    stp q3, q4, [x8, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x8, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x8, #96]
+; NO_SVE-NEXT:    stp q17, q18, [x8, #128]
+; NO_SVE-NEXT:    stp q21, q22, [x8, #160]
+; NO_SVE-NEXT:    stp q19, q20, [x8, #192]
+; NO_SVE-NEXT:    stp q23, q24, [x8, #224]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: insertelement_v128f16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    mov w9, #127
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    fmov h2, #5.00000000
 ; VBITS_GE_2048-NEXT:    index z3.h, #0, #1
 ; VBITS_GE_2048-NEXT:    ptrue p1.h
 ; VBITS_GE_2048-NEXT:    mov z1.h, w9
 ; VBITS_GE_2048-NEXT:    cmpeq p1.h, p1/z, z3.h, z1.h
 ; VBITS_GE_2048-NEXT:    mov z0.h, p1/m, h2
 ; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x8]
 ; VBITS_GE_2048-NEXT:    ret
     %op1 = load <128 x half>, <128 x half>* %a
     %r = insertelement <128 x half> %op1, half 5.0, i64 127
     ret <128 x half> %r
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x float> @insertelement_v2f32(<2 x float> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov s1, #5.00000000
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    mov v0.s[1], v1.s[0]
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 killed $q0
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v2f32:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov s1, #5.00000000
 ; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 def $q0
 ; VBITS_GE_256-NEXT:    mov v0.s[1], v1.s[0]
 ; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 killed $q0
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <2 x float> %op1, float 5.0, i64 1
     ret <2 x float> %r
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x float> @insertelement_v4f32(<4 x float> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov s1, #5.00000000
+; NO_SVE-NEXT:    mov v0.s[3], v1.s[0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v4f32:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov s1, #5.00000000
 ; VBITS_GE_256-NEXT:    mov v0.s[3], v1.s[0]
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <4 x float> %op1, float 5.0, i64 3
     ret <4 x float> %r
 }
 
 define <8 x float> @insertelement_v8f32(<8 x float>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    fmov s2, #5.00000000
+; NO_SVE-NEXT:    mov v1.s[3], v2.s[0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v8f32:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    mov w9, #7
 ; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
 ; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_256-NEXT:    fmov s2, #5.00000000
 ; VBITS_GE_256-NEXT:    index z3.s, #0, #1
 ; VBITS_GE_256-NEXT:    ptrue p1.s
 ; VBITS_GE_256-NEXT:    mov z1.s, w9
 ; VBITS_GE_256-NEXT:    cmpeq p1.s, p1/z, z3.s, z1.s
 ; VBITS_GE_256-NEXT:    mov z0.s, p1/m, s2
 ; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x8]
 ; VBITS_GE_256-NEXT:    ret
     %op1 = load <8 x float>, <8 x float>* %a
     %r = insertelement <8 x float> %op1, float 5.0, i64 7
     ret <8 x float> %r
 }
 
 define <16 x float> @insertelement_v16f32(<16 x float>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    fmov s1, #5.00000000
+; NO_SVE-NEXT:    mov v3.s[3], v1.s[0]
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: insertelement_v16f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    mov w9, #15
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    fmov s2, #5.00000000
 ; VBITS_GE_512-NEXT:    index z3.s, #0, #1
 ; VBITS_GE_512-NEXT:    ptrue p1.s
 ; VBITS_GE_512-NEXT:    mov z1.s, w9
 ; VBITS_GE_512-NEXT:    cmpeq p1.s, p1/z, z3.s, z1.s
 ; VBITS_GE_512-NEXT:    mov z0.s, p1/m, s2
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x8]
 ; VBITS_GE_512-NEXT:    ret
     %op1 = load <16 x float>, <16 x float>* %a
     %r = insertelement <16 x float> %op1, float 5.0, i64 15
     ret <16 x float> %r
 }
 
 define <32 x float> @insertelement_v32f32(<32 x float>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    fmov s5, #5.00000000
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    mov v7.s[3], v5.s[0]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: insertelement_v32f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    mov w9, #31
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    fmov s2, #5.00000000
 ; VBITS_GE_1024-NEXT:    index z3.s, #0, #1
 ; VBITS_GE_1024-NEXT:    ptrue p1.s
 ; VBITS_GE_1024-NEXT:    mov z1.s, w9
 ; VBITS_GE_1024-NEXT:    cmpeq p1.s, p1/z, z3.s, z1.s
 ; VBITS_GE_1024-NEXT:    mov z0.s, p1/m, s2
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x8]
 ; VBITS_GE_1024-NEXT:    ret
     %op1 = load <32 x float>, <32 x float>* %a
     %r = insertelement <32 x float> %op1, float 5.0, i64 31
     ret <32 x float> %r
 }
 
 define <64 x float> @insertelement_v64f32(<64 x float>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q23, q24, [x0, #224]
+; NO_SVE-NEXT:    fmov s0, #5.00000000
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    mov v24.s[3], v0.s[0]
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    ldp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q17, q18, [x0, #128]
+; NO_SVE-NEXT:    ldp q19, q20, [x0, #192]
+; NO_SVE-NEXT:    ldp q21, q22, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q2, [x8]
+; NO_SVE-NEXT:    stp q3, q4, [x8, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x8, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x8, #96]
+; NO_SVE-NEXT:    stp q17, q18, [x8, #128]
+; NO_SVE-NEXT:    stp q21, q22, [x8, #160]
+; NO_SVE-NEXT:    stp q19, q20, [x8, #192]
+; NO_SVE-NEXT:    stp q23, q24, [x8, #224]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: insertelement_v64f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    mov w9, #63
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    fmov s2, #5.00000000
 ; VBITS_GE_2048-NEXT:    index z3.s, #0, #1
 ; VBITS_GE_2048-NEXT:    ptrue p1.s
 ; VBITS_GE_2048-NEXT:    mov z1.s, w9
 ; VBITS_GE_2048-NEXT:    cmpeq p1.s, p1/z, z3.s, z1.s
 ; VBITS_GE_2048-NEXT:    mov z0.s, p1/m, s2
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x8]
 ; VBITS_GE_2048-NEXT:    ret
     %op1 = load <64 x float>, <64 x float>* %a
     %r = insertelement <64 x float> %op1, float 5.0, i64 63
     ret <64 x float> %r
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x double> @insertelement_v1f64(<1 x double> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v1f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov d0, #5.00000000
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v1f64:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov d0, #5.00000000
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <1 x double> %op1, double 5.0, i64 0
     ret <1 x double> %r
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x double> @insertelement_v2f64(<2 x double> %op1) #0 {
+; NO_SVE-LABEL: insertelement_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov d1, #5.00000000
+; NO_SVE-NEXT:    mov v0.d[1], v1.d[0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v2f64:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    fmov d1, #5.00000000
 ; VBITS_GE_256-NEXT:    mov v0.d[1], v1.d[0]
 ; VBITS_GE_256-NEXT:    ret
     %r = insertelement <2 x double> %op1, double 5.0, i64 1
     ret <2 x double> %r
 }
 
 define <4 x double> @insertelement_v4f64(<4 x double>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov d0, #5.00000000
+; NO_SVE-NEXT:    ldr q1, [x0, #16]
+; NO_SVE-NEXT:    mov v1.d[1], v0.d[0]
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_256-LABEL: insertelement_v4f64:
 ; VBITS_GE_256:       // %bb.0:
 ; VBITS_GE_256-NEXT:    mov w9, #3
 ; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
 ; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_256-NEXT:    fmov d2, #5.00000000
 ; VBITS_GE_256-NEXT:    index z3.d, #0, #1
 ; VBITS_GE_256-NEXT:    ptrue p1.d
 ; VBITS_GE_256-NEXT:    mov z1.d, x9
 ; VBITS_GE_256-NEXT:    cmpeq p1.d, p1/z, z3.d, z1.d
 ; VBITS_GE_256-NEXT:    mov z0.d, p1/m, d2
 ; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x8]
 ; VBITS_GE_256-NEXT:    ret
     %op1 = load <4 x double>, <4 x double>* %a
     %r = insertelement <4 x double> %op1, double 5.0, i64 3
     ret <4 x double> %r
 }
 
 define <8 x double> @insertelement_v8f64(<8 x double>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov d2, #5.00000000
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    mov v3.d[1], v2.d[0]
+; NO_SVE-NEXT:    ldr q2, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: insertelement_v8f64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    mov w9, #7
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    fmov d2, #5.00000000
 ; VBITS_GE_512-NEXT:    index z3.d, #0, #1
 ; VBITS_GE_512-NEXT:    ptrue p1.d
 ; VBITS_GE_512-NEXT:    mov z1.d, x9
 ; VBITS_GE_512-NEXT:    cmpeq p1.d, p1/z, z3.d, z1.d
 ; VBITS_GE_512-NEXT:    mov z0.d, p1/m, d2
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x8]
 ; VBITS_GE_512-NEXT:    ret
     %op1 = load <8 x double>, <8 x double>* %a
     %r = insertelement <8 x double> %op1, double 5.0, i64 7
     ret <8 x double> %r
 }
 
 define <16 x double> @insertelement_v16f64(<16 x double>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    fmov d5, #5.00000000
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    mov v7.d[1], v5.d[0]
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: insertelement_v16f64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    mov w9, #15
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    fmov d2, #5.00000000
 ; VBITS_GE_1024-NEXT:    index z3.d, #0, #1
 ; VBITS_GE_1024-NEXT:    ptrue p1.d
 ; VBITS_GE_1024-NEXT:    mov z1.d, x9
 ; VBITS_GE_1024-NEXT:    cmpeq p1.d, p1/z, z3.d, z1.d
 ; VBITS_GE_1024-NEXT:    mov z0.d, p1/m, d2
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x8]
 ; VBITS_GE_1024-NEXT:    ret
     %op1 = load <16 x double>, <16 x double>* %a
     %r = insertelement <16 x double> %op1, double 5.0, i64 15
     ret <16 x double> %r
 }
 
 define <32 x double> @insertelement_v32f64(<32 x double>* %a) #0 {
+; NO_SVE-LABEL: insertelement_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q23, q24, [x0, #224]
+; NO_SVE-NEXT:    fmov d0, #5.00000000
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    mov v24.d[1], v0.d[0]
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    ldp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q17, q18, [x0, #128]
+; NO_SVE-NEXT:    ldp q19, q20, [x0, #192]
+; NO_SVE-NEXT:    ldp q21, q22, [x0, #160]
+; NO_SVE-NEXT:    stp q1, q2, [x8]
+; NO_SVE-NEXT:    stp q3, q4, [x8, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x8, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x8, #96]
+; NO_SVE-NEXT:    stp q17, q18, [x8, #128]
+; NO_SVE-NEXT:    stp q21, q22, [x8, #160]
+; NO_SVE-NEXT:    stp q19, q20, [x8, #192]
+; NO_SVE-NEXT:    stp q23, q24, [x8, #224]
+; NO_SVE-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: insertelement_v32f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    mov w9, #31
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    fmov d2, #5.00000000
 ; VBITS_GE_2048-NEXT:    index z3.d, #0, #1
 ; VBITS_GE_2048-NEXT:    ptrue p1.d

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-gather.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Don't use SVE when its registers are no bigger than NEON.
 ; NO_SVE-NOT: ptrue
 
 ;
 ; LD1B
 ;
 
 define void @masked_gather_v2i8(<2 x i8>* %a, <2 x i8*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldrb w8, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fmov s0, w8
+; NO_SVE-NEXT:    ldrb w8, [x0, #1]
+; NO_SVE-NEXT:    mov v0.s[1], w8
+; NO_SVE-NEXT:    cmeq v0.2s, v0.2s, #0
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB0_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[0], [x9]
+; NO_SVE-NEXT:  .LBB0_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB0_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[4], [x8]
+; NO_SVE-NEXT:  .LBB0_4: // %else2
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    strb w9, [x0]
+; NO_SVE-NEXT:    strb w8, [x0, #1]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldrb w8, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    ldrb w8, [x0, #1]
 ; CHECK-NEXT:    mov v0.s[1], w8
; CHECK-NEXT:    ret
   %ptrs = load <2 x i8*>, <2 x i8*>* %b
   %mask = icmp eq <2 x i8> %cval, zeroinitializer
   %vals = call <2 x i8> @llvm.masked.gather.v2i8(<2 x i8*> %ptrs, i32 8, <2 x i1> %mask, <2 x i8> undef)
   store <2 x i8> %vals, <2 x i8>* %a
   ret void
 }
 
 define void @masked_gather_v4i8(<4 x i8>* %a, <4 x i8*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr s0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    ushll v0.8h, v0.8b, #0
+; NO_SVE-NEXT:    cmeq v0.4h, v0.4h, #0
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB1_6
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB1_7
+; NO_SVE-NEXT:  .LBB1_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB1_8
+; NO_SVE-NEXT:  .LBB1_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB1_5
+; NO_SVE-NEXT:  .LBB1_4: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[6], [x8]
+; NO_SVE-NEXT:  .LBB1_5: // %else8
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    str s0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB1_6: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB1_2
+; NO_SVE-NEXT:  .LBB1_7: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB1_3
+; NO_SVE-NEXT:  .LBB1_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB1_4
+; NO_SVE-NEXT:    b .LBB1_5
+;
 ; CHECK-LABEL: masked_gather_v4i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    ushll v0.8h, v0.8b, #0
 ; CHECK-NEXT:    cmeq v0.4h, v0.4h, #0
 ; CHECK-NEXT:    sunpklo z0.s, z0.h
; CHECK-NEXT:    ret
   %mask = icmp eq <4 x i8> %cval, zeroinitializer
   %vals = call <4 x i8> @llvm.masked.gather.v4i8(<4 x i8*> %ptrs, i32 8, <4 x i1> %mask, <4 x i8> undef)
   store <4 x i8> %vals, <4 x i8>* %a
   ret void
 }
 
 define void @masked_gather_v8i8(<8 x i8>* %a, <8 x i8*>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: masked_gather_v8i8:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    ldr d0, [x0]
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    cmeq v0.8b, v0.8b, #0
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    zip2 v1.8b, v0.8b, v0.8b
-; VBITS_EQ_256-NEXT:    zip1 v0.8b, v0.8b, v0.8b
-; VBITS_EQ_256-NEXT:    shl v1.4h, v1.4h, #8
-; VBITS_EQ_256-NEXT:    shl v0.4h, v0.4h, #8
-; VBITS_EQ_256-NEXT:    sshr v1.4h, v1.4h, #8
-; VBITS_EQ_256-NEXT:    sshr v0.4h, v0.4h, #8
-; VBITS_EQ_256-NEXT:    sunpklo z1.s, z1.h
-; VBITS_EQ_256-NEXT:    sunpklo z0.s, z0.h
-; VBITS_EQ_256-NEXT:    sunpklo z1.d, z1.s
-; VBITS_EQ_256-NEXT:    sunpklo z0.d, z0.s
-; VBITS_EQ_256-NEXT:    cmpne p1.d, p0/z, z1.d, #0
-; VBITS_EQ_256-NEXT:    cmpne p0.d, p0/z, z0.d, #0
-; VBITS_EQ_256-NEXT:    ld1b { z0.d }, p1/z, [z2.d]
-; VBITS_EQ_256-NEXT:    ld1b { z1.d }, p0/z, [z3.d]
-; VBITS_EQ_256-NEXT:    uzp1 z0.s, z0.s, z0.s
-; VBITS_EQ_256-NEXT:    uzp1 z1.s, z1.s, z1.s
-; VBITS_EQ_256-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_EQ_256-NEXT:    uzp1 z1.h, z1.h, z1.h
-; VBITS_EQ_256-NEXT:    uzp1 v0.8b, v1.8b, v0.8b
-; VBITS_EQ_256-NEXT:    str d0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_v8i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.8b, v0.8b, #0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB2_10
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB2_11
+; NO_SVE-NEXT:  .LBB2_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB2_12
+; NO_SVE-NEXT:  .LBB2_3: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB2_13
+; NO_SVE-NEXT:  .LBB2_4: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB2_14
+; NO_SVE-NEXT:  .LBB2_5: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB2_15
+; NO_SVE-NEXT:  .LBB2_6: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB2_16
+; NO_SVE-NEXT:  .LBB2_7: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB2_9
+; NO_SVE-NEXT:  .LBB2_8: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[7], [x8]
+; NO_SVE-NEXT:  .LBB2_9: // %else20
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB2_10: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr b0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB2_2
+; NO_SVE-NEXT:  .LBB2_11: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB2_3
+; NO_SVE-NEXT:  .LBB2_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB2_4
+; NO_SVE-NEXT:  .LBB2_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB2_5
+; NO_SVE-NEXT:  .LBB2_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB2_6
+; NO_SVE-NEXT:  .LBB2_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB2_7
+; NO_SVE-NEXT:  .LBB2_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB2_8
+; NO_SVE-NEXT:    b .LBB2_9
+;
+; VBITS_GE_256-LABEL: masked_gather_v8i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ldr d0, [x0]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    cmeq v0.8b, v0.8b, #0
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    zip2 v1.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    zip1 v0.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    shl v1.4h, v1.4h, #8
+; VBITS_GE_256-NEXT:    shl v0.4h, v0.4h, #8
+; VBITS_GE_256-NEXT:    sshr v1.4h, v1.4h, #8
+; VBITS_GE_256-NEXT:    sshr v0.4h, v0.4h, #8
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    cmpne p1.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1b { z0.d }, p1/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1b { z1.d }, p0/z, [z3.d]
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 v0.8b, v1.8b, v0.8b
+; VBITS_GE_256-NEXT:    str d0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: masked_gather_v8i8:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ldr d0, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    cmeq v0.8b, v0.8b, #0
 ; VBITS_GE_512-NEXT:    sunpklo z0.h, z0.b
; VBITS_GE_512-NEXT:    ret
   %ptrs = load <8 x i8*>, <8 x i8*>* %b
   %mask = icmp eq <8 x i8> %cval, zeroinitializer
   %vals = call <8 x i8> @llvm.masked.gather.v8i8(<8 x i8*> %ptrs, i32 8, <8 x i1> %mask, <8 x i8> undef)
   store <8 x i8> %vals, <8 x i8>* %a
   ret void
 }
 
 define void @masked_gather_v16i8(<16 x i8>* %a, <16 x i8*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.16b, v0.16b, #0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[8]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v0.b[9]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v0.b[10]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v0.b[11]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v0.b[12]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v0.b[13]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v0.b[14]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[15]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB3_18
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB3_19
+; NO_SVE-NEXT:  .LBB3_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB3_20
+; NO_SVE-NEXT:  .LBB3_3: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB3_21
+; NO_SVE-NEXT:  .LBB3_4: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB3_22
+; NO_SVE-NEXT:  .LBB3_5: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB3_23
+; NO_SVE-NEXT:  .LBB3_6: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB3_24
+; NO_SVE-NEXT:  .LBB3_7: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB3_25
+; NO_SVE-NEXT:  .LBB3_8: // %else20
+; NO_SVE-NEXT:    ldr q1, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB3_26
+; NO_SVE-NEXT:  .LBB3_9: // %else23
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB3_27
+; NO_SVE-NEXT:  .LBB3_10: // %else26
+; NO_SVE-NEXT:    ldr q1, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB3_28
+; NO_SVE-NEXT:  .LBB3_11: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB3_29
+; NO_SVE-NEXT:  .LBB3_12: // %else32
+; NO_SVE-NEXT:    ldr q1, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB3_30
+; NO_SVE-NEXT:  .LBB3_13: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB3_31
+; NO_SVE-NEXT:  .LBB3_14: // %else38
+; NO_SVE-NEXT:    ldr q1, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB3_32
+; NO_SVE-NEXT:  .LBB3_15: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB3_17
+; NO_SVE-NEXT:  .LBB3_16: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[15], [x8]
+; NO_SVE-NEXT:  .LBB3_17: // %else44
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB3_18: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr b0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB3_2
+; NO_SVE-NEXT:  .LBB3_19: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB3_3
+; NO_SVE-NEXT:  .LBB3_20: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB3_4
+; NO_SVE-NEXT:  .LBB3_21: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB3_5
+; NO_SVE-NEXT:  .LBB3_22: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB3_6
+; NO_SVE-NEXT:  .LBB3_23: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB3_7
+; NO_SVE-NEXT:  .LBB3_24: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB3_8
+; NO_SVE-NEXT:  .LBB3_25: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[7], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB3_9
+; NO_SVE-NEXT:  .LBB3_26: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB3_10
+; NO_SVE-NEXT:  .LBB3_27: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[9], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB3_11
+; NO_SVE-NEXT:  .LBB3_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB3_12
+; NO_SVE-NEXT:  .LBB3_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[11], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB3_13
+; NO_SVE-NEXT:  .LBB3_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB3_14
+; NO_SVE-NEXT:  .LBB3_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[13], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB3_15
+; NO_SVE-NEXT:  .LBB3_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[14], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB3_16
+; NO_SVE-NEXT:    b .LBB3_17
+;
+; VBITS_GE_256-LABEL: masked_gather_v16i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ldr q0, [x0]
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    cmeq v0.16b, v0.16b, #0
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    zip2 v1.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    zip1 v2.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    ext v0.16b, v0.16b, v0.16b, #8
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    shl v1.4h, v1.4h, #8
+; VBITS_GE_256-NEXT:    shl v2.4h, v2.4h, #8
+; VBITS_GE_256-NEXT:    zip2 v3.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    zip1 v0.8b, v0.8b, v0.8b
+; VBITS_GE_256-NEXT:    sshr v1.4h, v1.4h, #8
+; VBITS_GE_256-NEXT:    sshr v2.4h, v2.4h, #8
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    sunpklo z2.s, z2.h
+; VBITS_GE_256-NEXT:    shl v3.4h, v3.4h, #8
+; VBITS_GE_256-NEXT:    shl v0.4h, v0.4h, #8
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z2.s
+; VBITS_GE_256-NEXT:    cmpne p1.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    sshr v1.4h, v3.4h, #8
+; VBITS_GE_256-NEXT:    sshr v0.4h, v0.4h, #8
+; VBITS_GE_256-NEXT:    cmpne p2.d, p0/z, z2.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    ld1b { z2.d }, p1/z, [z6.d]
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    cmpne p1.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    cmpne p0.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1b { z3.d }, p2/z, [z7.d]
+; VBITS_GE_256-NEXT:    ld1b { z0.d }, p1/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1b { z1.d }, p0/z, [z4.d]
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    mov v3.d[1], v2.d[0]
+; VBITS_GE_256-NEXT:    mov v1.d[1], v0.d[0]
+; VBITS_GE_256-NEXT:    uzp1 v0.16b, v3.16b, v1.16b
+; VBITS_GE_256-NEXT:    str q0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16i8:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ldr q0, [x0]
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    cmeq v0.16b, v0.16b, #0
 ; VBITS_GE_1024-NEXT:    sunpklo z0.h, z0.b
 ; VBITS_GE_1024-NEXT:    sunpklo z0.s, z0.h
; VBITS_GE_1024-NEXT:    ret
   %ptrs = load <16 x i8*>, <16 x i8*>* %b
   %mask = icmp eq <16 x i8> %cval, zeroinitializer
   %vals = call <16 x i8> @llvm.masked.gather.v16i8(<16 x i8*> %ptrs, i32 8, <16 x i1> %mask, <16 x i8> undef)
   store <16 x i8> %vals, <16 x i8>* %a
   ret void
 }
 
 define void @masked_gather_v32i8(<32 x i8>* %a, <32 x i8*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    cmeq v1.16b, v1.16b, #0
+; NO_SVE-NEXT:    cmeq v0.16b, v0.16b, #0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[8]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[9]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[10]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[11]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[1]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    umov w9, v1.b[2]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w10, v1.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[12]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    bfi w15, w14, #1, #1
+; NO_SVE-NEXT:    umov w14, v1.b[9]
+; NO_SVE-NEXT:    bfi w15, w9, #2, #1
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[6]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w15, w10, #3, #1
+; NO_SVE-NEXT:    umov w10, v1.b[7]
+; NO_SVE-NEXT:    bfi w15, w9, #4, #1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v0.b[13]
+; NO_SVE-NEXT:    bfi w15, w11, #5, #1
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[8]
+; NO_SVE-NEXT:    umov w9, v0.b[14]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w11, w15, w11, lsl #6
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w10, w11, w10, lsl #7
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[10]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #13
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[11]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #14
+; NO_SVE-NEXT:    orr w9, w10, w11, lsl #8
+; NO_SVE-NEXT:    umov w10, v1.b[12]
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[13]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[14]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v0.b[15]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[15]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB4_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr b0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB4_3
+; NO_SVE-NEXT:    b .LBB4_4
+; NO_SVE-NEXT:  .LBB4_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB4_4
+; NO_SVE-NEXT:  .LBB4_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:  .LBB4_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB4_20
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB4_21
+; NO_SVE-NEXT:  .LBB4_6: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB4_22
+; NO_SVE-NEXT:  .LBB4_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB4_23
+; NO_SVE-NEXT:  .LBB4_8: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB4_24
+; NO_SVE-NEXT:  .LBB4_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB4_25
+; NO_SVE-NEXT:  .LBB4_10: // %else20
+; NO_SVE-NEXT:    ldr q1, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB4_26
+; NO_SVE-NEXT:  .LBB4_11: // %else23
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB4_27
+; NO_SVE-NEXT:  .LBB4_12: // %else26
+; NO_SVE-NEXT:    ldr q1, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB4_28
+; NO_SVE-NEXT:  .LBB4_13: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB4_29
+; NO_SVE-NEXT:  .LBB4_14: // %else32
+; NO_SVE-NEXT:    ldr q1, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB4_30
+; NO_SVE-NEXT:  .LBB4_15: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB4_31
+; NO_SVE-NEXT:  .LBB4_16: // %else38
+; NO_SVE-NEXT:    ldr q1, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB4_32
+; NO_SVE-NEXT:  .LBB4_17: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB4_33
+; NO_SVE-NEXT:  .LBB4_18: // %else44
+; NO_SVE-NEXT:    ldr q2, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB4_34
+; NO_SVE-NEXT:  .LBB4_19: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB4_35
+; NO_SVE-NEXT:    b .LBB4_36
+; NO_SVE-NEXT:  .LBB4_20: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB4_6
+; NO_SVE-NEXT:  .LBB4_21: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB4_7
+; NO_SVE-NEXT:  .LBB4_22: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB4_8
+; NO_SVE-NEXT:  .LBB4_23: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB4_9
+; NO_SVE-NEXT:  .LBB4_24: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB4_10
+; NO_SVE-NEXT:  .LBB4_25: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[7], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB4_11
+; NO_SVE-NEXT:  .LBB4_26: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB4_12
+; NO_SVE-NEXT:  .LBB4_27: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[9], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB4_13
+; NO_SVE-NEXT:  .LBB4_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB4_14
+; NO_SVE-NEXT:  .LBB4_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[11], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB4_15
+; NO_SVE-NEXT:  .LBB4_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB4_16
+; NO_SVE-NEXT:  .LBB4_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[13], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB4_17
+; NO_SVE-NEXT:  .LBB4_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.b }[14], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB4_18
+; NO_SVE-NEXT:  .LBB4_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.b }[15], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB4_19
+; NO_SVE-NEXT:  .LBB4_34:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #17, .LBB4_36
+; NO_SVE-NEXT:  .LBB4_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[1], [x9]
+; NO_SVE-NEXT:  .LBB4_36: // %else50
+; NO_SVE-NEXT:    ldr q2, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB4_52
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB4_53
+; NO_SVE-NEXT:  .LBB4_38: // %else56
+; NO_SVE-NEXT:    ldr q2, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB4_54
+; NO_SVE-NEXT:  .LBB4_39: // %else59
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB4_55
+; NO_SVE-NEXT:  .LBB4_40: // %else62
+; NO_SVE-NEXT:    ldr q2, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB4_56
+; NO_SVE-NEXT:  .LBB4_41: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB4_57
+; NO_SVE-NEXT:  .LBB4_42: // %else68
+; NO_SVE-NEXT:    ldr q2, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB4_58
+; NO_SVE-NEXT:  .LBB4_43: // %else71
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB4_59
+; NO_SVE-NEXT:  .LBB4_44: // %else74
+; NO_SVE-NEXT:    ldr q2, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB4_60
+; NO_SVE-NEXT:  .LBB4_45: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB4_61
+; NO_SVE-NEXT:  .LBB4_46: // %else80
+; NO_SVE-NEXT:    ldr q2, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB4_62
+; NO_SVE-NEXT:  .LBB4_47: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB4_63
+; NO_SVE-NEXT:  .LBB4_48: // %else86
+; NO_SVE-NEXT:    ldr q2, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB4_64
+; NO_SVE-NEXT:  .LBB4_49: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB4_51
+; NO_SVE-NEXT:  .LBB4_50: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[15], [x8]
+; NO_SVE-NEXT:  .LBB4_51: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB4_52: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB4_38
+; NO_SVE-NEXT:  .LBB4_53: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB4_39
+; NO_SVE-NEXT:  .LBB4_54: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB4_40
+; NO_SVE-NEXT:  .LBB4_55: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[5], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB4_41
+; NO_SVE-NEXT:  .LBB4_56: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB4_42
+; NO_SVE-NEXT:  .LBB4_57: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[7], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB4_43
+; NO_SVE-NEXT:  .LBB4_58: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB4_44
+; NO_SVE-NEXT:  .LBB4_59: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[9], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB4_45
+; NO_SVE-NEXT:  .LBB4_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB4_46
+; NO_SVE-NEXT:  .LBB4_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[11], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB4_47
+; NO_SVE-NEXT:  .LBB4_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB4_48
+; NO_SVE-NEXT:  .LBB4_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.b }[13], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB4_49
+; NO_SVE-NEXT:  .LBB4_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.b }[14], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB4_50
+; NO_SVE-NEXT:    b .LBB4_51
+;
+; VBITS_GE_256-LABEL: masked_gather_v32i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; VBITS_GE_256-NEXT:    .cfi_def_cfa_offset 16
+; VBITS_GE_256-NEXT:    mov x29, sp
+; VBITS_GE_256-NEXT:    .cfi_def_cfa w29, 16
+; VBITS_GE_256-NEXT:    .cfi_offset w30, -8
+; VBITS_GE_256-NEXT:    .cfi_offset w29, -16
+; VBITS_GE_256-NEXT:    sub x9, sp, #48
+; VBITS_GE_256-NEXT:    and sp, x9, #0xffffffffffffffe0
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    mov x9, #28
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    cmpeq p1.b, p0/z, z0.b, #0
+; VBITS_GE_256-NEXT:    mov z4.b, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    zip2 v2.8b, v4.8b, v0.8b
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    shl v3.4h, v2.4h, #8
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    sshr v5.4h, v3.4h, #8
+; VBITS_GE_256-NEXT:    mov x8, #20
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z5.h
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1b { z5.d }, p2/z, [z7.d]
+; VBITS_GE_256-NEXT:    zip1 v7.8b, v4.8b, v0.8b
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    shl v7.4h, v7.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z5.h, z5.h, z5.h
+; VBITS_GE_256-NEXT:    umov w8, v5.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v5.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v5.h[1]
+; VBITS_GE_256-NEXT:    sshr v7.4h, v7.4h, #8
+; VBITS_GE_256-NEXT:    umov w11, v5.h[0]
+; VBITS_GE_256-NEXT:    mov z5.d, z4.d
+; VBITS_GE_256-NEXT:    sunpklo z7.s, z7.h
+; VBITS_GE_256-NEXT:    ext z5.b, z5.b, z4.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z7.d, z7.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #7]
+; VBITS_GE_256-NEXT:    strb w9, [sp, #6]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z7.d, #0
+; VBITS_GE_256-NEXT:    strb w10, [sp, #5]
+; VBITS_GE_256-NEXT:    strb w11, [sp, #4]
+; VBITS_GE_256-NEXT:    ld1b { z7.d }, p2/z, [z17.d]
+; VBITS_GE_256-NEXT:    zip2 v17.8b, v5.8b, v0.8b
+; VBITS_GE_256-NEXT:    ext v4.16b, v4.16b, v4.16b, #8
+; VBITS_GE_256-NEXT:    uzp1 z7.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    shl v17.4h, v17.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z7.h, z7.h, z7.h
+; VBITS_GE_256-NEXT:    umov w8, v7.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v7.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v7.h[1]
+; VBITS_GE_256-NEXT:    sshr v17.4h, v17.4h, #8
+; VBITS_GE_256-NEXT:    umov w11, v7.h[0]
+; VBITS_GE_256-NEXT:    sunpklo z7.s, z17.h
+; VBITS_GE_256-NEXT:    sunpklo z7.d, z7.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #3]
+; VBITS_GE_256-NEXT:    strb w9, [sp, #2]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z7.d, #0
+; VBITS_GE_256-NEXT:    strb w10, [sp, #1]
+; VBITS_GE_256-NEXT:    strb w11, [sp]
+; VBITS_GE_256-NEXT:    ld1b { z7.d }, p2/z, [z16.d]
+; VBITS_GE_256-NEXT:    zip1 v16.8b, v5.8b, v0.8b
+; VBITS_GE_256-NEXT:    uzp1 z7.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    shl v16.4h, v16.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z7.h, z7.h, z7.h
+; VBITS_GE_256-NEXT:    umov w8, v7.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v7.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v7.h[1]
+; VBITS_GE_256-NEXT:    sshr v16.4h, v16.4h, #8
+; VBITS_GE_256-NEXT:    umov w11, v7.h[0]
+; VBITS_GE_256-NEXT:    sunpklo z7.s, z16.h
+; VBITS_GE_256-NEXT:    sunpklo z7.d, z7.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #23]
+; VBITS_GE_256-NEXT:    strb w9, [sp, #22]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z7.d, #0
+; VBITS_GE_256-NEXT:    strb w10, [sp, #21]
+; VBITS_GE_256-NEXT:    zip2 v7.8b, v4.8b, v0.8b
+; VBITS_GE_256-NEXT:    strb w11, [sp, #20]
+; VBITS_GE_256-NEXT:    zip1 v4.8b, v4.8b, v0.8b
+; VBITS_GE_256-NEXT:    ld1b { z6.d }, p2/z, [z6.d]
+; VBITS_GE_256-NEXT:    shl v7.4h, v7.4h, #8
+; VBITS_GE_256-NEXT:    shl v4.4h, v4.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z6.s, z6.s, z6.s
+; VBITS_GE_256-NEXT:    sshr v7.4h, v7.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z6.h, z6.h, z6.h
+; VBITS_GE_256-NEXT:    sshr v4.4h, v4.4h, #8
+; VBITS_GE_256-NEXT:    umov w8, v6.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v6.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v6.h[1]
+; VBITS_GE_256-NEXT:    umov w11, v6.h[0]
+; VBITS_GE_256-NEXT:    sunpklo z6.s, z7.h
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z4.h
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z6.s
+; VBITS_GE_256-NEXT:    sunpklo z4.d, z4.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #19]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    strb w9, [sp, #18]
+; VBITS_GE_256-NEXT:    strb w10, [sp, #17]
+; VBITS_GE_256-NEXT:    strb w11, [sp, #16]
+; VBITS_GE_256-NEXT:    ld1b { z3.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z4.d, #0
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    umov w8, v3.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v3.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v3.h[1]
+; VBITS_GE_256-NEXT:    umov w11, v3.h[0]
+; VBITS_GE_256-NEXT:    ext v3.16b, v5.16b, v5.16b, #8
+; VBITS_GE_256-NEXT:    strb w8, [sp, #15]
+; VBITS_GE_256-NEXT:    strb w9, [sp, #14]
+; VBITS_GE_256-NEXT:    strb w10, [sp, #13]
+; VBITS_GE_256-NEXT:    zip2 v4.8b, v3.8b, v0.8b
+; VBITS_GE_256-NEXT:    strb w11, [sp, #12]
+; VBITS_GE_256-NEXT:    ld1b { z2.d }, p2/z, [z2.d]
+; VBITS_GE_256-NEXT:    shl v4.4h, v4.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    sshr v4.4h, v4.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    umov w8, v2.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v2.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v2.h[1]
+; VBITS_GE_256-NEXT:    umov w11, v2.h[0]
+; VBITS_GE_256-NEXT:    sunpklo z2.s, z4.h
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z2.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #11]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    strb w9, [sp, #10]
+; VBITS_GE_256-NEXT:    zip1 v2.8b, v3.8b, v0.8b
+; VBITS_GE_256-NEXT:    strb w10, [sp, #9]
+; VBITS_GE_256-NEXT:    strb w11, [sp, #8]
+; VBITS_GE_256-NEXT:    ld1b { z1.d }, p2/z, [z1.d]
+; VBITS_GE_256-NEXT:    shl v2.4h, v2.4h, #8
+; VBITS_GE_256-NEXT:    sshr v2.4h, v2.4h, #8
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    umov w8, v1.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v1.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v1.h[1]
+; VBITS_GE_256-NEXT:    umov w11, v1.h[0]
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z2.h
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    strb w8, [sp, #31]
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    strb w9, [sp, #30]
+; VBITS_GE_256-NEXT:    strb w10, [sp, #29]
+; VBITS_GE_256-NEXT:    strb w11, [sp, #28]
+; VBITS_GE_256-NEXT:    ld1b { z0.d }, p1/z, [z0.d]
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    umov w8, v0.h[3]
+; VBITS_GE_256-NEXT:    umov w9, v0.h[2]
+; VBITS_GE_256-NEXT:    umov w10, v0.h[1]
+; VBITS_GE_256-NEXT:    umov w11, v0.h[0]
+; VBITS_GE_256-NEXT:    strb w8, [sp, #27]
+; VBITS_GE_256-NEXT:    strb w9, [sp, #26]
+; VBITS_GE_256-NEXT:    strb w10, [sp, #25]
+; VBITS_GE_256-NEXT:    strb w11, [sp, #24]
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [sp]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    mov sp, x29
+; VBITS_GE_256-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32i8:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.b, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    cmpeq p1.b, p0/z, z0.b, #0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
; VBITS_GE_2048-NEXT:    ret
   ret void
 }
 
 ;
 ; LD1H
 ;
 
 define void @masked_gather_v2i16(<2 x i16>* %a, <2 x i16*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldrh w8, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fmov s0, w8
+; NO_SVE-NEXT:    ldrh w8, [x0, #2]
+; NO_SVE-NEXT:    mov v0.s[1], w8
+; NO_SVE-NEXT:    cmeq v0.2s, v0.2s, #0
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB5_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[0], [x9]
+; NO_SVE-NEXT:  .LBB5_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB5_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x8]
+; NO_SVE-NEXT:  .LBB5_4: // %else2
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    strh w9, [x0]
+; NO_SVE-NEXT:    strh w8, [x0, #2]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldrh w8, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    fmov s0, w8
 ; CHECK-NEXT:    ldrh w8, [x0, #2]
 ; CHECK-NEXT:    mov v0.s[1], w8
; CHECK-NEXT:    ret
   %ptrs = load <2 x i16*>, <2 x i16*>* %b
   %mask = icmp eq <2 x i16> %cval, zeroinitializer
   %vals = call <2 x i16> @llvm.masked.gather.v2i16(<2 x i16*> %ptrs, i32 8, <2 x i1> %mask, <2 x i16> undef)
   store <2 x i16> %vals, <2 x i16>* %a
   ret void
 }
 
 define void @masked_gather_v4i16(<4 x i16>* %a, <4 x i16*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.4h, v0.4h, #0
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB6_6
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB6_7
+; NO_SVE-NEXT:  .LBB6_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB6_8
+; NO_SVE-NEXT:  .LBB6_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB6_5
+; NO_SVE-NEXT:  .LBB6_4: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x8]
+; NO_SVE-NEXT:  .LBB6_5: // %else8
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB6_6: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB6_2
+; NO_SVE-NEXT:  .LBB6_7: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB6_3
+; NO_SVE-NEXT:  .LBB6_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB6_4
+; NO_SVE-NEXT:    b .LBB6_5
+;
 ; CHECK-LABEL: masked_gather_v4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    cmeq v0.4h, v0.4h, #0
 ; CHECK-NEXT:    sunpklo z0.s, z0.h
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1h { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; CHECK-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x i16>, <4 x i16>* %a
   %ptrs = load <4 x i16*>, <4 x i16*>* %b
   %mask = icmp eq <4 x i16> %cval, zeroinitializer
   %vals = call <4 x i16> @llvm.masked.gather.v4i16(<4 x i16*> %ptrs, i32 8, <4 x i1> %mask, <4 x i16> undef)
   store <4 x i16> %vals, <4 x i16>* %a
   ret void
 }
 
 define void @masked_gather_v8i16(<8 x i16>* %a, <8 x i16*>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: masked_gather_v8i16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    ldr q0, [x0]
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    cmeq v0.8h, v0.8h, #0
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
-; VBITS_EQ_256-NEXT:    sunpklo z0.s, z0.h
-; VBITS_EQ_256-NEXT:    sunpklo z0.d, z0.s
-; VBITS_EQ_256-NEXT:    cmpne p1.d, p0/z, z0.d, #0
-; VBITS_EQ_256-NEXT:    ld1h { z0.d }, p1/z, [z3.d]
-; VBITS_EQ_256-NEXT:    sunpklo z1.s, z1.h
-; VBITS_EQ_256-NEXT:    sunpklo z1.d, z1.s
-; VBITS_EQ_256-NEXT:    cmpne p0.d, p0/z, z1.d, #0
-; VBITS_EQ_256-NEXT:    ld1h { z1.d }, p0/z, [z2.d]
-; VBITS_EQ_256-NEXT:    uzp1 z0.s, z0.s, z0.s
-; VBITS_EQ_256-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_EQ_256-NEXT:    uzp1 z1.s, z1.s, z1.s
-; VBITS_EQ_256-NEXT:    uzp1 z1.h, z1.h, z1.h
-; VBITS_EQ_256-NEXT:    mov v0.d[1], v1.d[0]
-; VBITS_EQ_256-NEXT:    str q0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_v8i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.8h, v0.8h, #0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB7_10
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB7_11
+; NO_SVE-NEXT:  .LBB7_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB7_12
+; NO_SVE-NEXT:  .LBB7_3: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB7_13
+; NO_SVE-NEXT:  .LBB7_4: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB7_14
+; NO_SVE-NEXT:  .LBB7_5: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB7_15
+; NO_SVE-NEXT:  .LBB7_6: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB7_16
+; NO_SVE-NEXT:  .LBB7_7: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB7_9
+; NO_SVE-NEXT:  .LBB7_8: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB7_9: // %else20
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB7_10: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB7_2
+; NO_SVE-NEXT:  .LBB7_11: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB7_3
+; NO_SVE-NEXT:  .LBB7_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB7_4
+; NO_SVE-NEXT:  .LBB7_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB7_5
+; NO_SVE-NEXT:  .LBB7_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB7_6
+; NO_SVE-NEXT:  .LBB7_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB7_7
+; NO_SVE-NEXT:  .LBB7_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB7_8
+; NO_SVE-NEXT:    b .LBB7_9
+;
+; VBITS_GE_256-LABEL: masked_gather_v8i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ldr q0, [x0]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    cmeq v0.8h, v0.8h, #0
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p1/z, [z3.d]
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    cmpne p0.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p0/z, [z2.d]
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    mov v0.d[1], v1.d[0]
+; VBITS_GE_256-NEXT:    str q0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: masked_gather_v8i16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ldr q0, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    cmeq v0.8h, v0.8h, #0
 ; VBITS_GE_512-NEXT:    sunpklo z0.s, z0.h
 ; VBITS_GE_512-NEXT:    sunpklo z0.d, z0.s
 ; VBITS_GE_512-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; VBITS_GE_512-NEXT:    ld1h { z0.d }, p0/z, [z1.d]
 ; VBITS_GE_512-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_512-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; VBITS_GE_512-NEXT:    str q0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %cval = load <8 x i16>, <8 x i16>* %a
   %ptrs = load <8 x i16*>, <8 x i16*>* %b
   %mask = icmp eq <8 x i16> %cval, zeroinitializer
   %vals = call <8 x i16> @llvm.masked.gather.v8i16(<8 x i16*> %ptrs, i32 8, <8 x i1> %mask, <8 x i16> undef)
   store <8 x i16> %vals, <8 x i16>* %a
   ret void
 }
 
 define void @masked_gather_v16i16(<16 x i16>* %a, <16 x i16*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    cmeq v0.8h, v0.8h, #0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    cmeq v1.8h, v1.8h, #0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v1.b[1]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v1.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v1.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v1.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB8_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB8_3
+; NO_SVE-NEXT:    b .LBB8_4
+; NO_SVE-NEXT:  .LBB8_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB8_4
+; NO_SVE-NEXT:  .LBB8_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB8_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB8_12
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB8_13
+; NO_SVE-NEXT:  .LBB8_6: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB8_14
+; NO_SVE-NEXT:  .LBB8_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB8_15
+; NO_SVE-NEXT:  .LBB8_8: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB8_16
+; NO_SVE-NEXT:  .LBB8_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB8_17
+; NO_SVE-NEXT:  .LBB8_10: // %else20
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB8_18
+; NO_SVE-NEXT:  .LBB8_11: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB8_19
+; NO_SVE-NEXT:    b .LBB8_20
+; NO_SVE-NEXT:  .LBB8_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB8_6
+; NO_SVE-NEXT:  .LBB8_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB8_7
+; NO_SVE-NEXT:  .LBB8_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB8_8
+; NO_SVE-NEXT:  .LBB8_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB8_9
+; NO_SVE-NEXT:  .LBB8_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB8_10
+; NO_SVE-NEXT:  .LBB8_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB8_11
+; NO_SVE-NEXT:  .LBB8_18:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #9, .LBB8_20
+; NO_SVE-NEXT:  .LBB8_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB8_20: // %else26
+; NO_SVE-NEXT:    ldr q2, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB8_28
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB8_29
+; NO_SVE-NEXT:  .LBB8_22: // %else32
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB8_30
+; NO_SVE-NEXT:  .LBB8_23: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB8_31
+; NO_SVE-NEXT:  .LBB8_24: // %else38
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB8_32
+; NO_SVE-NEXT:  .LBB8_25: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB8_27
+; NO_SVE-NEXT:  .LBB8_26: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB8_27: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB8_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB8_22
+; NO_SVE-NEXT:  .LBB8_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB8_23
+; NO_SVE-NEXT:  .LBB8_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB8_24
+; NO_SVE-NEXT:  .LBB8_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB8_25
+; NO_SVE-NEXT:  .LBB8_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB8_26
+; NO_SVE-NEXT:    b .LBB8_27
+;
+; VBITS_GE_256-LABEL: masked_gather_v16i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    cmpeq p2.h, p0/z, z0.h, #0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov z2.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z2.h
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ext v5.16b, v2.16b, v2.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z4.d, z4.s
+; VBITS_GE_256-NEXT:    ext z2.b, z2.b, z2.b, #16
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z4.d, #0
+; VBITS_GE_256-NEXT:    ext v4.16b, v2.16b, v2.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z2.s, z2.h
+; VBITS_GE_256-NEXT:    ld1h { z6.d }, p2/z, [z6.d]
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z2.s
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z5.h
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z4.h
+; VBITS_GE_256-NEXT:    ld1h { z3.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z4.s
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p2/z, [z0.d]
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z6.s, z6.s
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p1/z, [z1.d]
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    mov v2.d[1], v3.d[0]
+; VBITS_GE_256-NEXT:    mov v0.d[1], v1.d[0]
+; VBITS_GE_256-NEXT:    splice z2.h, p1, z2.h, z0.h
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16i16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl16
 ; VBITS_GE_1024-NEXT:    ptrue p1.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_1024-NEXT:    cmpeq p1.h, p0/z, z0.h, #0
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    ld1h { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x i16>, <16 x i16>* %a
   %ptrs = load <16 x i16*>, <16 x i16*>* %b
   %mask = icmp eq <16 x i16> %cval, zeroinitializer
   %vals = call <16 x i16> @llvm.masked.gather.v16i16(<16 x i16*> %ptrs, i32 8, <16 x i1> %mask, <16 x i16> undef)
   store <16 x i16> %vals, <16 x i16>* %a
   ret void
 }
 
 define void @masked_gather_v32i16(<32 x i16>* %a, <32 x i16*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    cmeq v0.8h, v0.8h, #0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    ldp q2, q3, [x0]
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    cmeq v1.8h, v1.8h, #0
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    cmeq v2.8h, v2.8h, #0
+; NO_SVE-NEXT:    umov w16, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    umov w14, v2.b[1]
+; NO_SVE-NEXT:    and w16, w16, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    umov w15, v2.b[2]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w16, lsl #8
+; NO_SVE-NEXT:    umov w16, v2.b[0]
+; NO_SVE-NEXT:    umov w12, v1.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #9
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #10
+; NO_SVE-NEXT:    umov w10, v2.b[3]
+; NO_SVE-NEXT:    umov w13, v1.b[5]
+; NO_SVE-NEXT:    and w9, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #11
+; NO_SVE-NEXT:    umov w11, v2.b[4]
+; NO_SVE-NEXT:    and w15, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v2.b[5]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w15, w14, #1, #1
+; NO_SVE-NEXT:    bfi w15, w9, #2, #1
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #12
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    and w10, w11, #0x1
+; NO_SVE-NEXT:    umov w13, v2.b[6]
+; NO_SVE-NEXT:    cmeq v0.8h, v3.8h, #0
+; NO_SVE-NEXT:    and w11, w16, #0x1
+; NO_SVE-NEXT:    bfi w15, w9, #3, #1
+; NO_SVE-NEXT:    umov w9, v2.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w15, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w15, w11, #5, #1
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    umov w14, v0.b[1]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #13
+; NO_SVE-NEXT:    orr w11, w15, w11, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w11, w9, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB9_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB9_3
+; NO_SVE-NEXT:    b .LBB9_4
+; NO_SVE-NEXT:  .LBB9_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB9_4
+; NO_SVE-NEXT:  .LBB9_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB9_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB9_12
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB9_13
+; NO_SVE-NEXT:  .LBB9_6: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB9_14
+; NO_SVE-NEXT:  .LBB9_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB9_15
+; NO_SVE-NEXT:  .LBB9_8: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB9_16
+; NO_SVE-NEXT:  .LBB9_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB9_17
+; NO_SVE-NEXT:  .LBB9_10: // %else20
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB9_18
+; NO_SVE-NEXT:  .LBB9_11: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB9_19
+; NO_SVE-NEXT:    b .LBB9_20
+; NO_SVE-NEXT:  .LBB9_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB9_6
+; NO_SVE-NEXT:  .LBB9_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB9_7
+; NO_SVE-NEXT:  .LBB9_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB9_8
+; NO_SVE-NEXT:  .LBB9_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB9_9
+; NO_SVE-NEXT:  .LBB9_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB9_10
+; NO_SVE-NEXT:  .LBB9_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB9_11
+; NO_SVE-NEXT:  .LBB9_18:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #9, .LBB9_20
+; NO_SVE-NEXT:  .LBB9_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB9_20: // %else26
+; NO_SVE-NEXT:    ldr q2, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB9_28
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB9_29
+; NO_SVE-NEXT:  .LBB9_22: // %else32
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB9_30
+; NO_SVE-NEXT:  .LBB9_23: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB9_31
+; NO_SVE-NEXT:  .LBB9_24: // %else38
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB9_32
+; NO_SVE-NEXT:  .LBB9_25: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB9_33
+; NO_SVE-NEXT:  .LBB9_26: // %else44
+; NO_SVE-NEXT:    ldr q3, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB9_34
+; NO_SVE-NEXT:  .LBB9_27: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB9_35
+; NO_SVE-NEXT:    b .LBB9_36
+; NO_SVE-NEXT:  .LBB9_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB9_22
+; NO_SVE-NEXT:  .LBB9_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB9_23
+; NO_SVE-NEXT:  .LBB9_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB9_24
+; NO_SVE-NEXT:  .LBB9_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB9_25
+; NO_SVE-NEXT:  .LBB9_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB9_26
+; NO_SVE-NEXT:  .LBB9_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB9_27
+; NO_SVE-NEXT:  .LBB9_34:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #17, .LBB9_36
+; NO_SVE-NEXT:  .LBB9_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB9_36: // %else50
+; NO_SVE-NEXT:    ldr q3, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB9_44
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB9_45
+; NO_SVE-NEXT:  .LBB9_38: // %else56
+; NO_SVE-NEXT:    ldr q3, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB9_46
+; NO_SVE-NEXT:  .LBB9_39: // %else59
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB9_47
+; NO_SVE-NEXT:  .LBB9_40: // %else62
+; NO_SVE-NEXT:    ldr q3, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB9_48
+; NO_SVE-NEXT:  .LBB9_41: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB9_49
+; NO_SVE-NEXT:  .LBB9_42: // %else68
+; NO_SVE-NEXT:    ldr q4, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB9_50
+; NO_SVE-NEXT:  .LBB9_43: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB9_51
+; NO_SVE-NEXT:    b .LBB9_52
+; NO_SVE-NEXT:  .LBB9_44: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB9_38
+; NO_SVE-NEXT:  .LBB9_45: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB9_39
+; NO_SVE-NEXT:  .LBB9_46: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB9_40
+; NO_SVE-NEXT:  .LBB9_47: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB9_41
+; NO_SVE-NEXT:  .LBB9_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB9_42
+; NO_SVE-NEXT:  .LBB9_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB9_43
+; NO_SVE-NEXT:  .LBB9_50:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #25, .LBB9_52
+; NO_SVE-NEXT:  .LBB9_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB9_52: // %else74
+; NO_SVE-NEXT:    ldr q4, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB9_60
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB9_61
+; NO_SVE-NEXT:  .LBB9_54: // %else80
+; NO_SVE-NEXT:    ldr q4, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB9_62
+; NO_SVE-NEXT:  .LBB9_55: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB9_63
+; NO_SVE-NEXT:  .LBB9_56: // %else86
+; NO_SVE-NEXT:    ldr q4, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB9_64
+; NO_SVE-NEXT:  .LBB9_57: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB9_59
+; NO_SVE-NEXT:  .LBB9_58: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB9_59: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB9_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB9_54
+; NO_SVE-NEXT:  .LBB9_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB9_55
+; NO_SVE-NEXT:  .LBB9_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB9_56
+; NO_SVE-NEXT:  .LBB9_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB9_57
+; NO_SVE-NEXT:  .LBB9_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB9_58
+; NO_SVE-NEXT:    b .LBB9_59
+;
+; VBITS_GE_256-LABEL: masked_gather_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #28
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    cmpeq p2.h, p0/z, z3.h, #0
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov z3.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z18.s, z3.h
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ext v5.16b, v3.16b, v3.16b, #8
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z18.d, z18.s
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z18.d, #0
+; VBITS_GE_256-NEXT:    ext v18.16b, v3.16b, v3.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z3.s, z3.h
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z5.h
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    sunpklo z18.s, z18.h
+; VBITS_GE_256-NEXT:    ld1h { z17.d }, p2/z, [z17.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.h, p0/z, z4.h, #0
+; VBITS_GE_256-NEXT:    sunpklo z18.d, z18.s
+; VBITS_GE_256-NEXT:    ld1h { z4.d }, p3/z, [z16.d]
+; VBITS_GE_256-NEXT:    mov z16.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z18.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z3.d }, p2/z, [z7.d]
+; VBITS_GE_256-NEXT:    ld1h { z6.d }, p3/z, [z6.d]
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z7.h, z17.h, z17.h
+; VBITS_GE_256-NEXT:    uzp1 z4.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    uzp1 z6.s, z6.s, z6.s
+; VBITS_GE_256-NEXT:    mov v7.d[1], v4.d[0]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    ext v4.16b, v16.16b, v16.16b, #8
+; VBITS_GE_256-NEXT:    uzp1 z6.h, z6.h, z6.h
+; VBITS_GE_256-NEXT:    mov v3.d[1], v6.d[0]
+; VBITS_GE_256-NEXT:    sunpklo z6.s, z16.h
+; VBITS_GE_256-NEXT:    ext z16.b, z16.b, z16.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z6.s
+; VBITS_GE_256-NEXT:    ext v17.16b, v16.16b, v16.16b, #8
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z4.h
+; VBITS_GE_256-NEXT:    sunpklo z4.d, z4.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z4.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z4.d }, p2/z, [z5.d]
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z16.h
+; VBITS_GE_256-NEXT:    sunpklo z6.s, z17.h
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z6.s
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z2.d }, p3/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p2/z, [z1.d]
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p1/z, [z0.d]
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z4.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z7.h, p1, z7.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    mov v4.d[1], v2.d[0]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    mov v1.d[1], v0.d[0]
+; VBITS_GE_256-NEXT:    splice z4.h, p1, z4.h, z1.h
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32i16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    cmpeq p1.h, p0/z, z0.h, #0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
; VBITS_GE_2048-NEXT:    ret
   ret void
 }
 
 ;
 ; LD1W
 ;
 
 define void @masked_gather_v2i32(<2 x i32>* %a, <2 x i32*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.2s, v0.2s, #0
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB10_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:  .LBB10_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB10_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x8]
+; NO_SVE-NEXT:  .LBB10_4: // %else2
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    cmeq v0.2s, v0.2s, #0
 ; CHECK-NEXT:    sshll v0.2d, v0.2s, #0
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1w { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    xtn v0.2s, v0.2d
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <2 x i32>, <2 x i32>* %a
   %ptrs = load <2 x i32*>, <2 x i32*>* %b
   %mask = icmp eq <2 x i32> %cval, zeroinitializer
   %vals = call <2 x i32> @llvm.masked.gather.v2i32(<2 x i32*> %ptrs, i32 8, <2 x i1> %mask, <2 x i32> undef)
   store <2 x i32> %vals, <2 x i32>* %a
   ret void
 }
 
 define void @masked_gather_v4i32(<4 x i32>* %a, <4 x i32*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.4s, v0.4s, #0
+; NO_SVE-NEXT:    xtn v0.4h, v0.4s
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB11_6
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB11_7
+; NO_SVE-NEXT:  .LBB11_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB11_8
+; NO_SVE-NEXT:  .LBB11_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB11_5
+; NO_SVE-NEXT:  .LBB11_4: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB11_5: // %else8
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB11_6: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB11_2
+; NO_SVE-NEXT:  .LBB11_7: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB11_3
+; NO_SVE-NEXT:  .LBB11_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB11_4
+; NO_SVE-NEXT:    b .LBB11_5
+;
 ; CHECK-LABEL: masked_gather_v4i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr q0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    cmeq v0.4s, v0.4s, #0
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1w { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x i32>, <4 x i32>* %a
   %ptrs = load <4 x i32*>, <4 x i32*>* %b
   %mask = icmp eq <4 x i32> %cval, zeroinitializer
   %vals = call <4 x i32> @llvm.masked.gather.v4i32(<4 x i32*> %ptrs, i32 8, <4 x i1> %mask, <4 x i32> undef)
   store <4 x i32> %vals, <4 x i32>* %a
   ret void
 }
 
 define void @masked_gather_v8i32(<8 x i32>* %a, <8 x i32*>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: masked_gather_v8i32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    ptrue p1.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p1/z, [x1]
-; VBITS_EQ_256-NEXT:    cmpeq p2.s, p0/z, z0.s, #0
-; VBITS_EQ_256-NEXT:    mov z0.s, p2/z, #-1 // =0xffffffffffffffff
-; VBITS_EQ_256-NEXT:    punpklo p2.h, p2.b
-; VBITS_EQ_256-NEXT:    ext z0.b, z0.b, z0.b, #16
-; VBITS_EQ_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
-; VBITS_EQ_256-NEXT:    sunpklo z0.d, z0.s
-; VBITS_EQ_256-NEXT:    ld1w { z2.d }, p2/z, [z2.d]
-; VBITS_EQ_256-NEXT:    cmpne p1.d, p1/z, z0.d, #0
-; VBITS_EQ_256-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
-; VBITS_EQ_256-NEXT:    ptrue p1.s, vl4
-; VBITS_EQ_256-NEXT:    uzp1 z1.s, z2.s, z2.s
-; VBITS_EQ_256-NEXT:    uzp1 z0.s, z0.s, z0.s
-; VBITS_EQ_256-NEXT:    splice z1.s, p1, z1.s, z0.s
-; VBITS_EQ_256-NEXT:    st1w { z1.s }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_v8i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    cmeq v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    cmeq v0.4s, v0.4s, #0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB12_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB12_3
+; NO_SVE-NEXT:    b .LBB12_4
+; NO_SVE-NEXT:  .LBB12_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB12_4
+; NO_SVE-NEXT:  .LBB12_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB12_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB12_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB12_9
+; NO_SVE-NEXT:  .LBB12_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB12_10
+; NO_SVE-NEXT:  .LBB12_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB12_11
+; NO_SVE-NEXT:    b .LBB12_12
+; NO_SVE-NEXT:  .LBB12_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB12_6
+; NO_SVE-NEXT:  .LBB12_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB12_7
+; NO_SVE-NEXT:  .LBB12_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB12_12
+; NO_SVE-NEXT:  .LBB12_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB12_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB12_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB12_15
+; NO_SVE-NEXT:  .LBB12_14: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB12_15: // %else20
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB12_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB12_14
+; NO_SVE-NEXT:    b .LBB12_15
+;
+; VBITS_GE_256-LABEL: masked_gather_v8i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p2.s, p0/z, z0.s, #0
+; VBITS_GE_256-NEXT:    mov z0.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z2.d]
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z0.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: masked_gather_v8i32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl8
 ; VBITS_GE_512-NEXT:    ptrue p1.d, vl8
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_512-NEXT:    cmpeq p1.s, p0/z, z0.s, #0
 ; VBITS_GE_512-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_512-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_512-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %cval = load <8 x i32>, <8 x i32>* %a
   %ptrs = load <8 x i32*>, <8 x i32*>* %b
   %mask = icmp eq <8 x i32> %cval, zeroinitializer
   %vals = call <8 x i32> @llvm.masked.gather.v8i32(<8 x i32*> %ptrs, i32 8, <8 x i1> %mask, <8 x i32> undef)
   store <8 x i32> %vals, <8 x i32>* %a
   ret void
 }
 
 define void @masked_gather_v16i32(<16 x i32>* %a, <16 x i32*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    cmeq v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    cmeq v0.4s, v0.4s, #0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldp q2, q1, [x0, #32]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    cmeq v2.4s, v2.4s, #0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    cmeq v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    uzp1 v1.8h, v2.8h, v1.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w8, v0.b[6]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v1.b[1]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v1.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v1.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v1.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB13_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB13_3
+; NO_SVE-NEXT:    b .LBB13_4
+; NO_SVE-NEXT:  .LBB13_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB13_4
+; NO_SVE-NEXT:  .LBB13_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB13_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB13_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB13_9
+; NO_SVE-NEXT:  .LBB13_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB13_10
+; NO_SVE-NEXT:  .LBB13_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB13_11
+; NO_SVE-NEXT:    b .LBB13_12
+; NO_SVE-NEXT:  .LBB13_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB13_6
+; NO_SVE-NEXT:  .LBB13_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB13_7
+; NO_SVE-NEXT:  .LBB13_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB13_12
+; NO_SVE-NEXT:  .LBB13_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB13_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB13_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB13_17
+; NO_SVE-NEXT:  .LBB13_14: // %else20
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB13_18
+; NO_SVE-NEXT:  .LBB13_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB13_19
+; NO_SVE-NEXT:    b .LBB13_20
+; NO_SVE-NEXT:  .LBB13_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB13_14
+; NO_SVE-NEXT:  .LBB13_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB13_15
+; NO_SVE-NEXT:  .LBB13_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB13_20
+; NO_SVE-NEXT:  .LBB13_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB13_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB13_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB13_25
+; NO_SVE-NEXT:  .LBB13_22: // %else32
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB13_26
+; NO_SVE-NEXT:  .LBB13_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB13_27
+; NO_SVE-NEXT:    b .LBB13_28
+; NO_SVE-NEXT:  .LBB13_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB13_22
+; NO_SVE-NEXT:  .LBB13_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB13_23
+; NO_SVE-NEXT:  .LBB13_26:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #13, .LBB13_28
+; NO_SVE-NEXT:  .LBB13_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB13_28: // %else38
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB13_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB13_31
+; NO_SVE-NEXT:  .LBB13_30: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB13_31: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB13_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB13_30
+; NO_SVE-NEXT:    b .LBB13_31
+;
+; VBITS_GE_256-LABEL: masked_gather_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p2.s, p0/z, z0.s, #0
+; VBITS_GE_256-NEXT:    cmpeq p3.s, p0/z, z1.s, #0
+; VBITS_GE_256-NEXT:    mov z0.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z4.d }, p2/z, [z4.d]
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z0.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p3/z, [z3.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p2/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p1/z, [z2.d]
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z0.s
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z2.s
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16i32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_1024-NEXT:    ptrue p1.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_1024-NEXT:    cmpeq p1.s, p0/z, z0.s, #0
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x i32>, <16 x i32>* %a
   %ptrs = load <16 x i32*>, <16 x i32*>* %b
   %mask = icmp eq <16 x i32> %cval, zeroinitializer
   %vals = call <16 x i32> @llvm.masked.gather.v16i32(<16 x i32*> %ptrs, i32 8, <16 x i1> %mask, <16 x i32> undef)
   store <16 x i32> %vals, <16 x i32>* %a
   ret void
 }
 
 define void @masked_gather_v32i32(<32 x i32>* %a, <32 x i32*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    cmeq v2.4s, v2.4s, #0
+; NO_SVE-NEXT:    cmeq v3.4s, v3.4s, #0
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    cmeq v4.4s, v4.4s, #0
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    cmeq v5.4s, v5.4s, #0
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    uzp1 v3.8h, v4.8h, v5.8h
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    cmeq v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    cmeq v0.4s, v0.4s, #0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v3.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    umov w10, v3.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v3.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v3.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    ldp q1, q4, [x0, #32]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w14, v3.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v0.b[3]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[4]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    cmeq v2.4s, v4.4s, #0
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    cmeq v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    umov w15, v0.b[5]
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #3, #1
+; NO_SVE-NEXT:    umov w12, v0.b[7]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #13
+; NO_SVE-NEXT:    xtn v0.8b, v1.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v3.b[6]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[0]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #6
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #7
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v3.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB14_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB14_3
+; NO_SVE-NEXT:    b .LBB14_4
+; NO_SVE-NEXT:  .LBB14_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB14_4
+; NO_SVE-NEXT:  .LBB14_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB14_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB14_9
+; NO_SVE-NEXT:  .LBB14_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB14_10
+; NO_SVE-NEXT:  .LBB14_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB14_11
+; NO_SVE-NEXT:    b .LBB14_12
+; NO_SVE-NEXT:  .LBB14_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB14_6
+; NO_SVE-NEXT:  .LBB14_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB14_7
+; NO_SVE-NEXT:  .LBB14_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB14_12
+; NO_SVE-NEXT:  .LBB14_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB14_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB14_17
+; NO_SVE-NEXT:  .LBB14_14: // %else20
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB14_18
+; NO_SVE-NEXT:  .LBB14_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB14_19
+; NO_SVE-NEXT:    b .LBB14_20
+; NO_SVE-NEXT:  .LBB14_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB14_14
+; NO_SVE-NEXT:  .LBB14_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB14_15
+; NO_SVE-NEXT:  .LBB14_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB14_20
+; NO_SVE-NEXT:  .LBB14_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB14_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB14_25
+; NO_SVE-NEXT:  .LBB14_22: // %else32
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB14_26
+; NO_SVE-NEXT:  .LBB14_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB14_27
+; NO_SVE-NEXT:    b .LBB14_28
+; NO_SVE-NEXT:  .LBB14_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB14_22
+; NO_SVE-NEXT:  .LBB14_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB14_23
+; NO_SVE-NEXT:  .LBB14_26:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #13, .LBB14_28
+; NO_SVE-NEXT:  .LBB14_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_28: // %else38
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB14_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB14_33
+; NO_SVE-NEXT:  .LBB14_30: // %else44
+; NO_SVE-NEXT:    ldr q5, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB14_34
+; NO_SVE-NEXT:  .LBB14_31: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB14_35
+; NO_SVE-NEXT:    b .LBB14_36
+; NO_SVE-NEXT:  .LBB14_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB14_30
+; NO_SVE-NEXT:  .LBB14_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB14_31
+; NO_SVE-NEXT:  .LBB14_34:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #17, .LBB14_36
+; NO_SVE-NEXT:  .LBB14_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_36: // %else50
+; NO_SVE-NEXT:    ldr q5, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB14_40
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB14_41
+; NO_SVE-NEXT:  .LBB14_38: // %else56
+; NO_SVE-NEXT:    ldr q6, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB14_42
+; NO_SVE-NEXT:  .LBB14_39: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB14_43
+; NO_SVE-NEXT:    b .LBB14_44
+; NO_SVE-NEXT:  .LBB14_40: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB14_38
+; NO_SVE-NEXT:  .LBB14_41: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB14_39
+; NO_SVE-NEXT:  .LBB14_42:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #21, .LBB14_44
+; NO_SVE-NEXT:  .LBB14_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_44: // %else62
+; NO_SVE-NEXT:    ldr q6, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB14_48
+; NO_SVE-NEXT:  // %bb.45: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB14_49
+; NO_SVE-NEXT:  .LBB14_46: // %else68
+; NO_SVE-NEXT:    ldr q7, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB14_50
+; NO_SVE-NEXT:  .LBB14_47: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB14_51
+; NO_SVE-NEXT:    b .LBB14_52
+; NO_SVE-NEXT:  .LBB14_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB14_46
+; NO_SVE-NEXT:  .LBB14_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB14_47
+; NO_SVE-NEXT:  .LBB14_50:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #25, .LBB14_52
+; NO_SVE-NEXT:  .LBB14_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_52: // %else74
+; NO_SVE-NEXT:    ldr q7, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB14_56
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB14_57
+; NO_SVE-NEXT:  .LBB14_54: // %else80
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB14_58
+; NO_SVE-NEXT:  .LBB14_55: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB14_59
+; NO_SVE-NEXT:    b .LBB14_60
+; NO_SVE-NEXT:  .LBB14_56: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB14_54
+; NO_SVE-NEXT:  .LBB14_57: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB14_55
+; NO_SVE-NEXT:  .LBB14_58:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #29, .LBB14_60
+; NO_SVE-NEXT:  .LBB14_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB14_60: // %else86
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB14_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB14_63
+; NO_SVE-NEXT:  .LBB14_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB14_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB14_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB14_62
+; NO_SVE-NEXT:    b .LBB14_63
+;
+; VBITS_GE_256-LABEL: masked_gather_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    cmpeq p2.s, p0/z, z0.s, #0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    cmpeq p4.s, p0/z, z2.s, #0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z18.d]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [z17.d]
+; VBITS_GE_256-NEXT:    cmpeq p3.s, p0/z, z1.s, #0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    cmpeq p2.s, p0/z, z3.s, #0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [z16.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [z6.d]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [z7.d]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [z0.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [z4.d]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32i32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    cmpeq p1.s, p0/z, z0.s, #0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b

 }
 
 ;
 ; LD1D
 ;
 
 ; Scalarize 1 x i64 gathers
 define void @masked_gather_v1i64(<1 x i64>* %a, <1 x i64*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v1i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    fmov x8, d0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    cbnz x8, .LBB15_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    ldr d0, [x1]
+; NO_SVE-NEXT:    fmov x8, d0
+; NO_SVE-NEXT:    ldr d0, [x8]
+; NO_SVE-NEXT:  .LBB15_2: // %else
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v1i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    // implicit-def: $d0
 ; CHECK-NEXT:    cbnz x8, .LBB15_2
 ; CHECK-NEXT:  // %bb.1: // %cond.load
 ; CHECK-NEXT:    ldr d0, [x1]
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    ldr d0, [x8]
 ; CHECK-NEXT:  .LBB15_2: // %else
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <1 x i64>, <1 x i64>* %a
   %ptrs = load <1 x i64*>, <1 x i64*>* %b
   %mask = icmp eq <1 x i64> %cval, zeroinitializer
   %vals = call <1 x i64> @llvm.masked.gather.v1i64(<1 x i64*> %ptrs, i32 8, <1 x i1> %mask, <1 x i64> undef)
   store <1 x i64> %vals, <1 x i64>* %a
   ret void
 }
 
 define void @masked_gather_v2i64(<2 x i64>* %a, <2 x i64*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    cmeq v0.2d, v0.2d, #0
+; NO_SVE-NEXT:    xtn v0.2s, v0.2d
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB16_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:  .LBB16_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB16_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB16_4: // %else2
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr q0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    cmeq v0.2d, v0.2d, #0
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <2 x i64>, <2 x i64>* %a
   %ptrs = load <2 x i64*>, <2 x i64*>* %b
   %mask = icmp eq <2 x i64> %cval, zeroinitializer
   %vals = call <2 x i64> @llvm.masked.gather.v2i64(<2 x i64*> %ptrs, i32 8, <2 x i1> %mask, <2 x i64> undef)
   store <2 x i64> %vals, <2 x i64>* %a
   ret void
 }
 
 define void @masked_gather_v4i64(<4 x i64>* %a, <4 x i64*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    cmeq v1.2d, v1.2d, #0
+; NO_SVE-NEXT:    cmeq v0.2d, v0.2d, #0
+; NO_SVE-NEXT:    uzp1 v0.4s, v1.4s, v0.4s
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.4h, v0.4s
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbz w10, #0, .LBB17_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB17_3
+; NO_SVE-NEXT:    b .LBB17_4
+; NO_SVE-NEXT:  .LBB17_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB17_4
+; NO_SVE-NEXT:  .LBB17_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB17_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB17_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB17_7
+; NO_SVE-NEXT:    b .LBB17_8
+; NO_SVE-NEXT:  .LBB17_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB17_8
+; NO_SVE-NEXT:  .LBB17_7: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB17_8: // %else8
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v4i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x i64>, <4 x i64>* %a
   %ptrs = load <4 x i64*>, <4 x i64*>* %b
   %mask = icmp eq <4 x i64> %cval, zeroinitializer
   %vals = call <4 x i64> @llvm.masked.gather.v4i64(<4 x i64*> %ptrs, i32 8, <4 x i1> %mask, <4 x i64> undef)
   store <4 x i64> %vals, <4 x i64>* %a
   ret void
 }
 
 define void @masked_gather_v8i64(<8 x i64>* %a, <8 x i64*>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: masked_gather_v8i64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
-; VBITS_EQ_256-NEXT:    cmpeq p2.d, p0/z, z1.d, #0
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p1/z, [z2.d]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p2/z, [z3.d]
-; VBITS_EQ_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    st1d { z1.d }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    cmeq v0.2d, v0.2d, #0
+; NO_SVE-NEXT:    ldp q2, q3, [x0]
+; NO_SVE-NEXT:    cmeq v1.2d, v1.2d, #0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    cmeq v2.2d, v2.2d, #0
+; NO_SVE-NEXT:    cmeq v3.2d, v3.2d, #0
+; NO_SVE-NEXT:    uzp1 v1.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB18_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB18_3
+; NO_SVE-NEXT:    b .LBB18_4
+; NO_SVE-NEXT:  .LBB18_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB18_4
+; NO_SVE-NEXT:  .LBB18_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB18_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB18_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB18_7
+; NO_SVE-NEXT:    b .LBB18_8
+; NO_SVE-NEXT:  .LBB18_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB18_8
+; NO_SVE-NEXT:  .LBB18_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB18_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB18_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB18_11
+; NO_SVE-NEXT:    b .LBB18_12
+; NO_SVE-NEXT:  .LBB18_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB18_12
+; NO_SVE-NEXT:  .LBB18_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB18_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB18_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB18_15
+; NO_SVE-NEXT:    b .LBB18_16
+; NO_SVE-NEXT:  .LBB18_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB18_16
+; NO_SVE-NEXT:  .LBB18_15: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB18_16: // %else20
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: masked_gather_v8i64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
 
   %cval = load <8 x i64>, <8 x i64>* %a
   %ptrs = load <8 x i64*>, <8 x i64*>* %b
   %mask = icmp eq <8 x i64> %cval, zeroinitializer
   %vals = call <8 x i64> @llvm.masked.gather.v8i64(<8 x i64*> %ptrs, i32 8, <8 x i1> %mask, <8 x i64> undef)
   store <8 x i64> %vals, <8 x i64>* %a
   ret void
 }
 
 define void @masked_gather_v16i64(<16 x i64>* %a, <16 x i64*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    cmeq v2.2d, v2.2d, #0
+; NO_SVE-NEXT:    ldp q4, q5, [x0]
+; NO_SVE-NEXT:    cmeq v3.2d, v3.2d, #0
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    cmeq v4.2d, v4.2d, #0
+; NO_SVE-NEXT:    cmeq v5.2d, v5.2d, #0
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #96]
+; NO_SVE-NEXT:    uzp1 v3.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    cmeq v0.2d, v0.2d, #0
+; NO_SVE-NEXT:    uzp1 v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    cmeq v1.2d, v1.2d, #0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    cmeq v4.2d, v4.2d, #0
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    cmeq v3.2d, v5.2d, #0
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    uzp1 v2.4s, v4.4s, v3.4s
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    umov w12, v1.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v0.8h, v2.8h, v0.8h
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v1.b[6]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v1.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[0]
+; NO_SVE-NEXT:    umov w12, v0.b[1]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v0.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v0.b[3]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v0.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v0.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB19_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB19_3
+; NO_SVE-NEXT:    b .LBB19_4
+; NO_SVE-NEXT:  .LBB19_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB19_4
+; NO_SVE-NEXT:  .LBB19_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB19_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB19_7
+; NO_SVE-NEXT:    b .LBB19_8
+; NO_SVE-NEXT:  .LBB19_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB19_8
+; NO_SVE-NEXT:  .LBB19_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB19_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB19_11
+; NO_SVE-NEXT:    b .LBB19_12
+; NO_SVE-NEXT:  .LBB19_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB19_12
+; NO_SVE-NEXT:  .LBB19_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB19_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB19_15
+; NO_SVE-NEXT:    b .LBB19_16
+; NO_SVE-NEXT:  .LBB19_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB19_16
+; NO_SVE-NEXT:  .LBB19_15: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_16: // %else20
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB19_18
+; NO_SVE-NEXT:  // %bb.17: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB19_19
+; NO_SVE-NEXT:    b .LBB19_20
+; NO_SVE-NEXT:  .LBB19_18:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #9, .LBB19_20
+; NO_SVE-NEXT:  .LBB19_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_20: // %else26
+; NO_SVE-NEXT:    ldr q6, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB19_22
+; NO_SVE-NEXT:  // %bb.21: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB19_23
+; NO_SVE-NEXT:    b .LBB19_24
+; NO_SVE-NEXT:  .LBB19_22:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #11, .LBB19_24
+; NO_SVE-NEXT:  .LBB19_23: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_24: // %else32
+; NO_SVE-NEXT:    ldr q7, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB19_26
+; NO_SVE-NEXT:  // %bb.25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB19_27
+; NO_SVE-NEXT:    b .LBB19_28
+; NO_SVE-NEXT:  .LBB19_26:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #13, .LBB19_28
+; NO_SVE-NEXT:  .LBB19_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB19_28: // %else38
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB19_30
+; NO_SVE-NEXT:  // %bb.29: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB19_31
+; NO_SVE-NEXT:    b .LBB19_32
+; NO_SVE-NEXT:  .LBB19_30:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #15, .LBB19_32
+; NO_SVE-NEXT:  .LBB19_31: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB19_32: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    cmpeq p3.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z2.d, #0
+; VBITS_GE_256-NEXT:    cmpeq p4.d, p0/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p3/z, [z4.d]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z6.d]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p4/z, [z7.d]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16i64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x i64>, <16 x i64>* %a
   %ptrs = load <16 x i64*>, <16 x i64*>* %b
   %mask = icmp eq <16 x i64> %cval, zeroinitializer
   %vals = call <16 x i64> @llvm.masked.gather.v16i64(<16 x i64*> %ptrs, i32 8, <16 x i1> %mask, <16 x i64> undef)
   store <16 x i64> %vals, <16 x i64>* %a
   ret void
 }
 
 define void @masked_gather_v32i64(<32 x i64>* %a, <32 x i64*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #160]
+; NO_SVE-NEXT:    cmeq v2.2d, v2.2d, #0
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #128]
+; NO_SVE-NEXT:    cmeq v3.2d, v3.2d, #0
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    cmeq v4.2d, v4.2d, #0
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #224]
+; NO_SVE-NEXT:    cmeq v5.2d, v5.2d, #0
+; NO_SVE-NEXT:    uzp1 v4.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    cmeq v0.2d, v0.2d, #0
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #192]
+; NO_SVE-NEXT:    cmeq v1.2d, v1.2d, #0
+; NO_SVE-NEXT:    uzp1 v2.8h, v4.8h, v2.8h
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    cmeq v6.2d, v6.2d, #0
+; NO_SVE-NEXT:    cmeq v7.2d, v7.2d, #0
+; NO_SVE-NEXT:    ldp q3, q5, [x0]
+; NO_SVE-NEXT:    uzp1 v1.4s, v6.4s, v7.4s
+; NO_SVE-NEXT:    cmeq v3.2d, v3.2d, #0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    ldp q16, q17, [x0, #32]
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    umov w12, v1.b[4]
+; NO_SVE-NEXT:    umov w13, v1.b[5]
+; NO_SVE-NEXT:    cmeq v6.2d, v16.2d, #0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    cmeq v4.2d, v17.2d, #0
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    cmeq v2.2d, v5.2d, #0
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v4.4s, v6.4s, v4.4s
+; NO_SVE-NEXT:    umov w8, v0.b[0]
+; NO_SVE-NEXT:    uzp1 v2.4s, v3.4s, v2.4s
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w10, v0.b[1]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w11, v0.b[2]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    uzp1 v1.8h, v2.8h, v4.8h
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #7
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    umov w10, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #9
+; NO_SVE-NEXT:    umov w9, v0.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[1]
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    cmeq v3.2d, v3.2d, #0
+; NO_SVE-NEXT:    umov w13, v1.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #11
+; NO_SVE-NEXT:    and w10, w11, #0x1
+; NO_SVE-NEXT:    cmeq v4.2d, v4.2d, #0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #12
+; NO_SVE-NEXT:    cmeq v2.2d, v2.2d, #0
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    cmeq v5.2d, v5.2d, #0
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    umov w14, v1.b[4]
+; NO_SVE-NEXT:    uzp1 v4.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w11, w10, #1, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v4.8h
+; NO_SVE-NEXT:    bfi w11, w12, #2, #1
+; NO_SVE-NEXT:    and w12, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    bfi w11, w10, #3, #1
+; NO_SVE-NEXT:    umov w10, v1.b[7]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    bfi w11, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v0.b[6]
+; NO_SVE-NEXT:    bfi w11, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[0]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    orr w9, w11, w13, lsl #6
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #7
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #8
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB20_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB20_3
+; NO_SVE-NEXT:    b .LBB20_4
+; NO_SVE-NEXT:  .LBB20_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB20_4
+; NO_SVE-NEXT:  .LBB20_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB20_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB20_7
+; NO_SVE-NEXT:    b .LBB20_8
+; NO_SVE-NEXT:  .LBB20_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB20_8
+; NO_SVE-NEXT:  .LBB20_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB20_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB20_11
+; NO_SVE-NEXT:    b .LBB20_12
+; NO_SVE-NEXT:  .LBB20_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB20_12
+; NO_SVE-NEXT:  .LBB20_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB20_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB20_15
+; NO_SVE-NEXT:    b .LBB20_16
+; NO_SVE-NEXT:  .LBB20_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB20_16
+; NO_SVE-NEXT:  .LBB20_15: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_16: // %else20
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB20_18
+; NO_SVE-NEXT:  // %bb.17: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB20_19
+; NO_SVE-NEXT:    b .LBB20_20
+; NO_SVE-NEXT:  .LBB20_18:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #9, .LBB20_20
+; NO_SVE-NEXT:  .LBB20_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_20: // %else26
+; NO_SVE-NEXT:    ldr q6, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB20_22
+; NO_SVE-NEXT:  // %bb.21: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB20_23
+; NO_SVE-NEXT:    b .LBB20_24
+; NO_SVE-NEXT:  .LBB20_22:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #11, .LBB20_24
+; NO_SVE-NEXT:  .LBB20_23: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_24: // %else32
+; NO_SVE-NEXT:    ldr q7, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB20_26
+; NO_SVE-NEXT:  // %bb.25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB20_27
+; NO_SVE-NEXT:    b .LBB20_28
+; NO_SVE-NEXT:  .LBB20_26:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #13, .LBB20_28
+; NO_SVE-NEXT:  .LBB20_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_28: // %else38
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB20_30
+; NO_SVE-NEXT:  // %bb.29: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB20_31
+; NO_SVE-NEXT:    b .LBB20_32
+; NO_SVE-NEXT:  .LBB20_30:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #15, .LBB20_32
+; NO_SVE-NEXT:  .LBB20_31: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_32: // %else44
+; NO_SVE-NEXT:    ldr q17, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB20_34
+; NO_SVE-NEXT:  // %bb.33: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB20_35
+; NO_SVE-NEXT:    b .LBB20_36
+; NO_SVE-NEXT:  .LBB20_34:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    tbz w8, #17, .LBB20_36
+; NO_SVE-NEXT:  .LBB20_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_36: // %else50
+; NO_SVE-NEXT:    ldr q18, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB20_38
+; NO_SVE-NEXT:  // %bb.37: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d18
+; NO_SVE-NEXT:    ld1 { v17.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB20_39
+; NO_SVE-NEXT:    b .LBB20_40
+; NO_SVE-NEXT:  .LBB20_38:
+; NO_SVE-NEXT:    // implicit-def: $q17
+; NO_SVE-NEXT:    tbz w8, #19, .LBB20_40
+; NO_SVE-NEXT:  .LBB20_39: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v18.d[1]
+; NO_SVE-NEXT:    ld1 { v17.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_40: // %else56
+; NO_SVE-NEXT:    ldr q19, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB20_42
+; NO_SVE-NEXT:  // %bb.41: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d19
+; NO_SVE-NEXT:    ld1 { v18.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB20_43
+; NO_SVE-NEXT:    b .LBB20_44
+; NO_SVE-NEXT:  .LBB20_42:
+; NO_SVE-NEXT:    // implicit-def: $q18
+; NO_SVE-NEXT:    tbz w8, #21, .LBB20_44
+; NO_SVE-NEXT:  .LBB20_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v19.d[1]
+; NO_SVE-NEXT:    ld1 { v18.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_44: // %else62
+; NO_SVE-NEXT:    ldr q20, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB20_46
+; NO_SVE-NEXT:  // %bb.45: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d20
+; NO_SVE-NEXT:    ld1 { v19.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB20_47
+; NO_SVE-NEXT:    b .LBB20_48
+; NO_SVE-NEXT:  .LBB20_46:
+; NO_SVE-NEXT:    // implicit-def: $q19
+; NO_SVE-NEXT:    tbz w8, #23, .LBB20_48
+; NO_SVE-NEXT:  .LBB20_47: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v20.d[1]
+; NO_SVE-NEXT:    ld1 { v19.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_48: // %else68
+; NO_SVE-NEXT:    ldr q21, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB20_50
+; NO_SVE-NEXT:  // %bb.49: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d21
+; NO_SVE-NEXT:    ld1 { v20.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB20_51
+; NO_SVE-NEXT:    b .LBB20_52
+; NO_SVE-NEXT:  .LBB20_50:
+; NO_SVE-NEXT:    // implicit-def: $q20
+; NO_SVE-NEXT:    tbz w8, #25, .LBB20_52
+; NO_SVE-NEXT:  .LBB20_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v21.d[1]
+; NO_SVE-NEXT:    ld1 { v20.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_52: // %else74
+; NO_SVE-NEXT:    ldr q22, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB20_54
+; NO_SVE-NEXT:  // %bb.53: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d22
+; NO_SVE-NEXT:    ld1 { v21.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB20_55
+; NO_SVE-NEXT:    b .LBB20_56
+; NO_SVE-NEXT:  .LBB20_54:
+; NO_SVE-NEXT:    // implicit-def: $q21
+; NO_SVE-NEXT:    tbz w8, #27, .LBB20_56
+; NO_SVE-NEXT:  .LBB20_55: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v22.d[1]
+; NO_SVE-NEXT:    ld1 { v21.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_56: // %else80
+; NO_SVE-NEXT:    ldr q23, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB20_58
+; NO_SVE-NEXT:  // %bb.57: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d23
+; NO_SVE-NEXT:    ld1 { v22.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB20_59
+; NO_SVE-NEXT:    b .LBB20_60
+; NO_SVE-NEXT:  .LBB20_58:
+; NO_SVE-NEXT:    // implicit-def: $q22
+; NO_SVE-NEXT:    tbz w8, #29, .LBB20_60
+; NO_SVE-NEXT:  .LBB20_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v23.d[1]
+; NO_SVE-NEXT:    ld1 { v22.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB20_60: // %else86
+; NO_SVE-NEXT:    ldr q24, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB20_62
+; NO_SVE-NEXT:  // %bb.61: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d24
+; NO_SVE-NEXT:    ld1 { v23.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB20_63
+; NO_SVE-NEXT:    b .LBB20_64
+; NO_SVE-NEXT:  .LBB20_62:
+; NO_SVE-NEXT:    // implicit-def: $q23
+; NO_SVE-NEXT:    tbz w8, #31, .LBB20_64
+; NO_SVE-NEXT:  .LBB20_63: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v24.d[1]
+; NO_SVE-NEXT:    ld1 { v23.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB20_64: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    stp q16, q17, [x0, #128]
+; NO_SVE-NEXT:    stp q18, q19, [x0, #160]
+; NO_SVE-NEXT:    stp q20, q21, [x0, #192]
+; NO_SVE-NEXT:    stp q22, q23, [x0, #224]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    mov x13, #24
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z19.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z21.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z22.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z20.d }, p0/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p0/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p0/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p0/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z23.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p2/z, [z19.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z6.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z21.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p2/z, [z22.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p2/z, [z20.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z4.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p2/z, [z18.d]
+; VBITS_GE_256-NEXT:    cmpeq p2.d, p0/z, z5.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p2/z, [z17.d]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [z16.d]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z7.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [z23.d]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32i64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    cmpeq p1.d, p0/z, z0.d, #0
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cval = load <32 x i64>, <32 x i64>* %a
   %ptrs = load <32 x i64*>, <32 x i64*>* %b
   %mask = icmp eq <32 x i64> %cval, zeroinitializer
   %vals = call <32 x i64> @llvm.masked.gather.v32i64(<32 x i64*> %ptrs, i32 8, <32 x i1> %mask, <32 x i64> undef)
   store <32 x i64> %vals, <32 x i64>* %a
   ret void
 }
 
 ;
 ; LD1H (float)
 ;
 
 define void @masked_gather_v2f16(<2 x half>* %a, <2 x half*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr s0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.4h, v0.4h, #0.0
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[0]
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB21_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:  .LBB21_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB21_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x8]
+; NO_SVE-NEXT:  .LBB21_4: // %else2
+; NO_SVE-NEXT:    str s0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    movi d0, #0000000000000000
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    fcmeq v1.4h, v1.4h, #0.0
 ; CHECK-NEXT:    umov w8, v1.h[0]
 ; CHECK-NEXT:    umov w9, v1.h[1]
; CHECK-NEXT:    ret
   %ptrs = load <2 x half*>, <2 x half*>* %b
   %mask = fcmp oeq <2 x half> %cval, zeroinitializer
   %vals = call <2 x half> @llvm.masked.gather.v2f16(<2 x half*> %ptrs, i32 8, <2 x i1> %mask, <2 x half> undef)
   store <2 x half> %vals, <2 x half>* %a
   ret void
 }
 
 define void @masked_gather_v4f16(<4 x half>* %a, <4 x half*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.4h, v0.4h, #0.0
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB22_6
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB22_7
+; NO_SVE-NEXT:  .LBB22_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB22_8
+; NO_SVE-NEXT:  .LBB22_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB22_5
+; NO_SVE-NEXT:  .LBB22_4: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x8]
+; NO_SVE-NEXT:  .LBB22_5: // %else8
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB22_6: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB22_2
+; NO_SVE-NEXT:  .LBB22_7: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB22_3
+; NO_SVE-NEXT:  .LBB22_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB22_4
+; NO_SVE-NEXT:    b .LBB22_5
+;
 ; CHECK-LABEL: masked_gather_v4f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq v0.4h, v0.4h, #0.0
 ; CHECK-NEXT:    sunpklo z0.s, z0.h
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1h { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; CHECK-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x half>, <4 x half>* %a
   %ptrs = load <4 x half*>, <4 x half*>* %b
   %mask = fcmp oeq <4 x half> %cval, zeroinitializer
   %vals = call <4 x half> @llvm.masked.gather.v4f16(<4 x half*> %ptrs, i32 8, <4 x i1> %mask, <4 x half> undef)
   store <4 x half> %vals, <4 x half>* %a
   ret void
 }
 
 define void @masked_gather_v8f16(<8 x half>* %a, <8 x half*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB23_10
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB23_11
+; NO_SVE-NEXT:  .LBB23_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB23_12
+; NO_SVE-NEXT:  .LBB23_3: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB23_13
+; NO_SVE-NEXT:  .LBB23_4: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB23_14
+; NO_SVE-NEXT:  .LBB23_5: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB23_15
+; NO_SVE-NEXT:  .LBB23_6: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB23_16
+; NO_SVE-NEXT:  .LBB23_7: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB23_9
+; NO_SVE-NEXT:  .LBB23_8: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB23_9: // %else20
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB23_10: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB23_2
+; NO_SVE-NEXT:  .LBB23_11: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB23_3
+; NO_SVE-NEXT:  .LBB23_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB23_4
+; NO_SVE-NEXT:  .LBB23_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB23_5
+; NO_SVE-NEXT:  .LBB23_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB23_6
+; NO_SVE-NEXT:  .LBB23_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB23_7
+; NO_SVE-NEXT:  .LBB23_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB23_8
+; NO_SVE-NEXT:    b .LBB23_9
+;
+; VBITS_GE_256-LABEL: masked_gather_v8f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ldr q0, [x0]
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    ext v1.16b, v0.16b, v0.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    cmpne p1.d, p0/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p1/z, [z3.d]
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    cmpne p0.d, p0/z, z1.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p0/z, [z2.d]
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    mov v0.d[1], v1.d[0]
+; VBITS_GE_256-NEXT:    str q0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: masked_gather_v8f16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ldr q0, [x0]
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq v0.8h, v0.8h, #0.0
 ; VBITS_GE_512-NEXT:    sunpklo z0.s, z0.h
 ; VBITS_GE_512-NEXT:    sunpklo z0.d, z0.s
 ; VBITS_GE_512-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; VBITS_GE_512-NEXT:    ld1h { z0.d }, p0/z, [z1.d]
 ; VBITS_GE_512-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_512-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; VBITS_GE_512-NEXT:    str q0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %cval = load <8 x half>, <8 x half>* %a
   %ptrs = load <8 x half*>, <8 x half*>* %b
   %mask = fcmp oeq <8 x half> %cval, zeroinitializer
   %vals = call <8 x half> @llvm.masked.gather.v8f16(<8 x half*> %ptrs, i32 8, <8 x i1> %mask, <8 x half> undef)
   store <8 x half> %vals, <8 x half>* %a
   ret void
 }
 
 define void @masked_gather_v16f16(<16 x half>* %a, <16 x half*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v1.b[1]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v1.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v1.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v1.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB24_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB24_3
+; NO_SVE-NEXT:    b .LBB24_4
+; NO_SVE-NEXT:  .LBB24_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB24_4
+; NO_SVE-NEXT:  .LBB24_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB24_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB24_12
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB24_13
+; NO_SVE-NEXT:  .LBB24_6: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB24_14
+; NO_SVE-NEXT:  .LBB24_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB24_15
+; NO_SVE-NEXT:  .LBB24_8: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB24_16
+; NO_SVE-NEXT:  .LBB24_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB24_17
+; NO_SVE-NEXT:  .LBB24_10: // %else20
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB24_18
+; NO_SVE-NEXT:  .LBB24_11: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB24_19
+; NO_SVE-NEXT:    b .LBB24_20
+; NO_SVE-NEXT:  .LBB24_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB24_6
+; NO_SVE-NEXT:  .LBB24_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB24_7
+; NO_SVE-NEXT:  .LBB24_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB24_8
+; NO_SVE-NEXT:  .LBB24_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB24_9
+; NO_SVE-NEXT:  .LBB24_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB24_10
+; NO_SVE-NEXT:  .LBB24_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB24_11
+; NO_SVE-NEXT:  .LBB24_18:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #9, .LBB24_20
+; NO_SVE-NEXT:  .LBB24_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB24_20: // %else26
+; NO_SVE-NEXT:    ldr q2, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB24_28
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB24_29
+; NO_SVE-NEXT:  .LBB24_22: // %else32
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB24_30
+; NO_SVE-NEXT:  .LBB24_23: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB24_31
+; NO_SVE-NEXT:  .LBB24_24: // %else38
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB24_32
+; NO_SVE-NEXT:  .LBB24_25: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB24_27
+; NO_SVE-NEXT:  .LBB24_26: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB24_27: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB24_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB24_22
+; NO_SVE-NEXT:  .LBB24_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB24_23
+; NO_SVE-NEXT:  .LBB24_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB24_24
+; NO_SVE-NEXT:  .LBB24_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB24_25
+; NO_SVE-NEXT:  .LBB24_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB24_26
+; NO_SVE-NEXT:    b .LBB24_27
+;
+; VBITS_GE_256-LABEL: masked_gather_v16f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z0.h, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov z2.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z2.h
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ext v5.16b, v2.16b, v2.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z4.d, z4.s
+; VBITS_GE_256-NEXT:    ext z2.b, z2.b, z2.b, #16
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z4.d, #0
+; VBITS_GE_256-NEXT:    ext v4.16b, v2.16b, v2.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z2.s, z2.h
+; VBITS_GE_256-NEXT:    ld1h { z6.d }, p2/z, [z6.d]
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z2.s
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z5.h
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z4.h
+; VBITS_GE_256-NEXT:    ld1h { z3.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z4.s
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p2/z, [z0.d]
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z6.s, z6.s
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p1/z, [z1.d]
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    mov v2.d[1], v3.d[0]
+; VBITS_GE_256-NEXT:    mov v0.d[1], v1.d[0]
+; VBITS_GE_256-NEXT:    splice z2.h, p1, z2.h, z0.h
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16f16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl16
 ; VBITS_GE_1024-NEXT:    ptrue p1.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    ld1h { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x half>, <16 x half>* %a
   %ptrs = load <16 x half*>, <16 x half*>* %b
   %mask = fcmp oeq <16 x half> %cval, zeroinitializer
   %vals = call <16 x half> @llvm.masked.gather.v16f16(<16 x half*> %ptrs, i32 8, <16 x i1> %mask, <16 x half> undef)
   store <16 x half> %vals, <16 x half>* %a
   ret void
 }
 
 define void @masked_gather_v32f16(<32 x half>* %a, <32 x half*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    ldp q2, q3, [x0]
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    fcmeq v2.8h, v2.8h, #0.0
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w16, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[1]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w16, w16, #0x1
+; NO_SVE-NEXT:    umov w14, v2.b[0]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w16, lsl #8
+; NO_SVE-NEXT:    umov w16, v2.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #9
+; NO_SVE-NEXT:    umov w9, v2.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #10
+; NO_SVE-NEXT:    umov w10, v2.b[4]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #11
+; NO_SVE-NEXT:    umov w11, v2.b[5]
+; NO_SVE-NEXT:    bfi w14, w12, #1, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    and w15, w16, #0x1
+; NO_SVE-NEXT:    fcmeq v0.8h, v3.8h, #0.0
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #12
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w12, v2.b[6]
+; NO_SVE-NEXT:    bfi w14, w15, #2, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w14, w9, #3, #1
+; NO_SVE-NEXT:    umov w9, v2.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w14, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w14, w11, #5, #1
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[0]
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w11, w14, w11, lsl #6
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w11, w9, lsl #7
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #13
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB25_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB25_3
+; NO_SVE-NEXT:    b .LBB25_4
+; NO_SVE-NEXT:  .LBB25_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB25_4
+; NO_SVE-NEXT:  .LBB25_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB25_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB25_12
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB25_13
+; NO_SVE-NEXT:  .LBB25_6: // %else8
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB25_14
+; NO_SVE-NEXT:  .LBB25_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB25_15
+; NO_SVE-NEXT:  .LBB25_8: // %else14
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB25_16
+; NO_SVE-NEXT:  .LBB25_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB25_17
+; NO_SVE-NEXT:  .LBB25_10: // %else20
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB25_18
+; NO_SVE-NEXT:  .LBB25_11: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB25_19
+; NO_SVE-NEXT:    b .LBB25_20
+; NO_SVE-NEXT:  .LBB25_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB25_6
+; NO_SVE-NEXT:  .LBB25_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB25_7
+; NO_SVE-NEXT:  .LBB25_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB25_8
+; NO_SVE-NEXT:  .LBB25_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB25_9
+; NO_SVE-NEXT:  .LBB25_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB25_10
+; NO_SVE-NEXT:  .LBB25_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB25_11
+; NO_SVE-NEXT:  .LBB25_18:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #9, .LBB25_20
+; NO_SVE-NEXT:  .LBB25_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB25_20: // %else26
+; NO_SVE-NEXT:    ldr q2, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB25_28
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB25_29
+; NO_SVE-NEXT:  .LBB25_22: // %else32
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB25_30
+; NO_SVE-NEXT:  .LBB25_23: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB25_31
+; NO_SVE-NEXT:  .LBB25_24: // %else38
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB25_32
+; NO_SVE-NEXT:  .LBB25_25: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB25_33
+; NO_SVE-NEXT:  .LBB25_26: // %else44
+; NO_SVE-NEXT:    ldr q3, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB25_34
+; NO_SVE-NEXT:  .LBB25_27: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB25_35
+; NO_SVE-NEXT:    b .LBB25_36
+; NO_SVE-NEXT:  .LBB25_28: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB25_22
+; NO_SVE-NEXT:  .LBB25_29: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB25_23
+; NO_SVE-NEXT:  .LBB25_30: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB25_24
+; NO_SVE-NEXT:  .LBB25_31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB25_25
+; NO_SVE-NEXT:  .LBB25_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB25_26
+; NO_SVE-NEXT:  .LBB25_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB25_27
+; NO_SVE-NEXT:  .LBB25_34:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #17, .LBB25_36
+; NO_SVE-NEXT:  .LBB25_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB25_36: // %else50
+; NO_SVE-NEXT:    ldr q3, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB25_44
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB25_45
+; NO_SVE-NEXT:  .LBB25_38: // %else56
+; NO_SVE-NEXT:    ldr q3, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB25_46
+; NO_SVE-NEXT:  .LBB25_39: // %else59
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB25_47
+; NO_SVE-NEXT:  .LBB25_40: // %else62
+; NO_SVE-NEXT:    ldr q3, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB25_48
+; NO_SVE-NEXT:  .LBB25_41: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB25_49
+; NO_SVE-NEXT:  .LBB25_42: // %else68
+; NO_SVE-NEXT:    ldr q4, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB25_50
+; NO_SVE-NEXT:  .LBB25_43: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB25_51
+; NO_SVE-NEXT:    b .LBB25_52
+; NO_SVE-NEXT:  .LBB25_44: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB25_38
+; NO_SVE-NEXT:  .LBB25_45: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB25_39
+; NO_SVE-NEXT:  .LBB25_46: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB25_40
+; NO_SVE-NEXT:  .LBB25_47: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB25_41
+; NO_SVE-NEXT:  .LBB25_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB25_42
+; NO_SVE-NEXT:  .LBB25_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB25_43
+; NO_SVE-NEXT:  .LBB25_50:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #25, .LBB25_52
+; NO_SVE-NEXT:  .LBB25_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB25_52: // %else74
+; NO_SVE-NEXT:    ldr q4, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB25_60
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB25_61
+; NO_SVE-NEXT:  .LBB25_54: // %else80
+; NO_SVE-NEXT:    ldr q4, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB25_62
+; NO_SVE-NEXT:  .LBB25_55: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB25_63
+; NO_SVE-NEXT:  .LBB25_56: // %else86
+; NO_SVE-NEXT:    ldr q4, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB25_64
+; NO_SVE-NEXT:  .LBB25_57: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB25_59
+; NO_SVE-NEXT:  .LBB25_58: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB25_59: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB25_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB25_54
+; NO_SVE-NEXT:  .LBB25_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB25_55
+; NO_SVE-NEXT:  .LBB25_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB25_56
+; NO_SVE-NEXT:  .LBB25_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB25_57
+; NO_SVE-NEXT:  .LBB25_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB25_58
+; NO_SVE-NEXT:    b .LBB25_59
+;
+; VBITS_GE_256-LABEL: masked_gather_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #28
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z3.h, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov z3.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    sunpklo z18.s, z3.h
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ext v5.16b, v3.16b, v3.16b, #8
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z18.d, z18.s
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z18.d, #0
+; VBITS_GE_256-NEXT:    ext v18.16b, v3.16b, v3.16b, #8
+; VBITS_GE_256-NEXT:    sunpklo z3.s, z3.h
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z5.h
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    sunpklo z18.s, z18.h
+; VBITS_GE_256-NEXT:    ld1h { z17.d }, p2/z, [z17.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z4.h, #0.0
+; VBITS_GE_256-NEXT:    sunpklo z18.d, z18.s
+; VBITS_GE_256-NEXT:    ld1h { z4.d }, p3/z, [z16.d]
+; VBITS_GE_256-NEXT:    mov z16.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z18.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z3.d }, p2/z, [z7.d]
+; VBITS_GE_256-NEXT:    ld1h { z6.d }, p3/z, [z6.d]
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z7.h, z17.h, z17.h
+; VBITS_GE_256-NEXT:    uzp1 z4.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    uzp1 z6.s, z6.s, z6.s
+; VBITS_GE_256-NEXT:    mov v7.d[1], v4.d[0]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    ext v4.16b, v16.16b, v16.16b, #8
+; VBITS_GE_256-NEXT:    uzp1 z6.h, z6.h, z6.h
+; VBITS_GE_256-NEXT:    mov v3.d[1], v6.d[0]
+; VBITS_GE_256-NEXT:    sunpklo z6.s, z16.h
+; VBITS_GE_256-NEXT:    ext z16.b, z16.b, z16.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z6.s
+; VBITS_GE_256-NEXT:    ext v17.16b, v16.16b, v16.16b, #8
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    sunpklo z4.s, z4.h
+; VBITS_GE_256-NEXT:    sunpklo z4.d, z4.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z4.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z4.d }, p2/z, [z5.d]
+; VBITS_GE_256-NEXT:    sunpklo z5.s, z16.h
+; VBITS_GE_256-NEXT:    sunpklo z6.s, z17.h
+; VBITS_GE_256-NEXT:    sunpklo z5.d, z5.s
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z6.s
+; VBITS_GE_256-NEXT:    cmpne p2.d, p1/z, z5.d, #0
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    ld1h { z2.d }, p3/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1h { z1.d }, p2/z, [z1.d]
+; VBITS_GE_256-NEXT:    ld1h { z0.d }, p1/z, [z0.d]
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z4.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z7.h, p1, z7.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    mov v4.d[1], v2.d[0]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    mov v1.d[1], v0.d[0]
+; VBITS_GE_256-NEXT:    splice z4.h, p1, z4.h, z1.h
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32f16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
; VBITS_GE_2048-NEXT:    ret
   ret void
 }
 
 ;
 ; LD1W (float)
 ;
 
 define void @masked_gather_v2f32(<2 x float>* %a, <2 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.2s, v0.2s, #0.0
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB26_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:  .LBB26_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB26_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x8]
+; NO_SVE-NEXT:  .LBB26_4: // %else2
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    fcmeq v0.2s, v0.2s, #0.0
 ; CHECK-NEXT:    sshll v0.2d, v0.2s, #0
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1w { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    xtn v0.2s, v0.2d
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <2 x float>, <2 x float>* %a
   %ptrs = load <2 x float*>, <2 x float*>* %b
   %mask = fcmp oeq <2 x float> %cval, zeroinitializer
   %vals = call <2 x float> @llvm.masked.gather.v2f32(<2 x float*> %ptrs, i32 8, <2 x i1> %mask, <2 x float> undef)
   store <2 x float> %vals, <2 x float>* %a
   ret void
 }
 
 define void @masked_gather_v4f32(<4 x float>* %a, <4 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    xtn v0.4h, v0.4s
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB27_6
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB27_7
+; NO_SVE-NEXT:  .LBB27_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB27_8
+; NO_SVE-NEXT:  .LBB27_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB27_5
+; NO_SVE-NEXT:  .LBB27_4: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB27_5: // %else8
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB27_6: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB27_2
+; NO_SVE-NEXT:  .LBB27_7: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB27_3
+; NO_SVE-NEXT:  .LBB27_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB27_4
+; NO_SVE-NEXT:    b .LBB27_5
+;
 ; CHECK-LABEL: masked_gather_v4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr q0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq v0.4s, v0.4s, #0.0
 ; CHECK-NEXT:    sunpklo z0.d, z0.s
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1w { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x float>, <4 x float>* %a
   %ptrs = load <4 x float*>, <4 x float*>* %b
   %mask = fcmp oeq <4 x float> %cval, zeroinitializer
   %vals = call <4 x float> @llvm.masked.gather.v4f32(<4 x float*> %ptrs, i32 8, <4 x i1> %mask, <4 x float> undef)
   store <4 x float> %vals, <4 x float>* %a
   ret void
 }
 
 define void @masked_gather_v8f32(<8 x float>* %a, <8 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB28_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB28_3
+; NO_SVE-NEXT:    b .LBB28_4
+; NO_SVE-NEXT:  .LBB28_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB28_4
+; NO_SVE-NEXT:  .LBB28_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB28_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB28_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB28_9
+; NO_SVE-NEXT:  .LBB28_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB28_10
+; NO_SVE-NEXT:  .LBB28_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB28_11
+; NO_SVE-NEXT:    b .LBB28_12
+; NO_SVE-NEXT:  .LBB28_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB28_6
+; NO_SVE-NEXT:  .LBB28_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB28_7
+; NO_SVE-NEXT:  .LBB28_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB28_12
+; NO_SVE-NEXT:  .LBB28_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB28_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB28_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB28_15
+; NO_SVE-NEXT:  .LBB28_14: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB28_15: // %else20
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB28_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB28_14
+; NO_SVE-NEXT:    b .LBB28_15
+;
+; VBITS_GE_256-LABEL: masked_gather_v8f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov z0.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z2.d]
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z0.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z0.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: masked_gather_v8f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl8
 ; VBITS_GE_512-NEXT:    ptrue p1.d, vl8
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_512-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_512-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_512-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %cval = load <8 x float>, <8 x float>* %a
   %ptrs = load <8 x float*>, <8 x float*>* %b
   %mask = fcmp oeq <8 x float> %cval, zeroinitializer
   %vals = call <8 x float> @llvm.masked.gather.v8f32(<8 x float*> %ptrs, i32 8, <8 x i1> %mask, <8 x float> undef)
   store <8 x float> %vals, <8 x float>* %a
   ret void
 }
 
 define void @masked_gather_v16f32(<16 x float>* %a, <16 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldp q1, q2, [x0, #32]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[6]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v1.b[1]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v1.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v1.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v1.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB29_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB29_3
+; NO_SVE-NEXT:    b .LBB29_4
+; NO_SVE-NEXT:  .LBB29_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB29_4
+; NO_SVE-NEXT:  .LBB29_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB29_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB29_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB29_9
+; NO_SVE-NEXT:  .LBB29_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB29_10
+; NO_SVE-NEXT:  .LBB29_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB29_11
+; NO_SVE-NEXT:    b .LBB29_12
+; NO_SVE-NEXT:  .LBB29_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB29_6
+; NO_SVE-NEXT:  .LBB29_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB29_7
+; NO_SVE-NEXT:  .LBB29_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB29_12
+; NO_SVE-NEXT:  .LBB29_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB29_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB29_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB29_17
+; NO_SVE-NEXT:  .LBB29_14: // %else20
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB29_18
+; NO_SVE-NEXT:  .LBB29_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB29_19
+; NO_SVE-NEXT:    b .LBB29_20
+; NO_SVE-NEXT:  .LBB29_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB29_14
+; NO_SVE-NEXT:  .LBB29_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB29_15
+; NO_SVE-NEXT:  .LBB29_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB29_20
+; NO_SVE-NEXT:  .LBB29_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB29_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB29_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB29_25
+; NO_SVE-NEXT:  .LBB29_22: // %else32
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB29_26
+; NO_SVE-NEXT:  .LBB29_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB29_27
+; NO_SVE-NEXT:    b .LBB29_28
+; NO_SVE-NEXT:  .LBB29_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB29_22
+; NO_SVE-NEXT:  .LBB29_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB29_23
+; NO_SVE-NEXT:  .LBB29_26:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #13, .LBB29_28
+; NO_SVE-NEXT:  .LBB29_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB29_28: // %else38
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB29_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB29_31
+; NO_SVE-NEXT:  .LBB29_30: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB29_31: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB29_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB29_30
+; NO_SVE-NEXT:    b .LBB29_31
+;
+; VBITS_GE_256-LABEL: masked_gather_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z0.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z4.d }, p2/z, [z4.d]
+; VBITS_GE_256-NEXT:    sunpklo z0.d, z0.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z0.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p3/z, [z3.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p2/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p1/z, [z2.d]
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z0.s
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z2.s
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_1024-NEXT:    ptrue p1.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_1024-NEXT:    ld1w { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x float>, <16 x float>* %a
   %ptrs = load <16 x float*>, <16 x float*>* %b
   %mask = fcmp oeq <16 x float> %cval, zeroinitializer
   %vals = call <16 x float> @llvm.masked.gather.v16f32(<16 x float*> %ptrs, i32 8, <16 x i1> %mask, <16 x float> undef)
   store <16 x float> %vals, <16 x float>* %a
   ret void
 }
 
 define void @masked_gather_v32f32(<32 x float>* %a, <32 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    uzp1 v4.8h, v4.8h, v5.8h
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    umov w12, v1.b[4]
+; NO_SVE-NEXT:    umov w13, v1.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v4.8b, v4.8h
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v4.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v4.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v4.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v4.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w13, v4.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v2.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldr q3, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    fcmeq v1.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v4.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v1.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v4.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v4.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB30_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB30_3
+; NO_SVE-NEXT:    b .LBB30_4
+; NO_SVE-NEXT:  .LBB30_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB30_4
+; NO_SVE-NEXT:  .LBB30_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_4: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB30_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB30_9
+; NO_SVE-NEXT:  .LBB30_6: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB30_10
+; NO_SVE-NEXT:  .LBB30_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB30_11
+; NO_SVE-NEXT:    b .LBB30_12
+; NO_SVE-NEXT:  .LBB30_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB30_6
+; NO_SVE-NEXT:  .LBB30_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB30_7
+; NO_SVE-NEXT:  .LBB30_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #5, .LBB30_12
+; NO_SVE-NEXT:  .LBB30_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_12: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB30_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB30_17
+; NO_SVE-NEXT:  .LBB30_14: // %else20
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB30_18
+; NO_SVE-NEXT:  .LBB30_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB30_19
+; NO_SVE-NEXT:    b .LBB30_20
+; NO_SVE-NEXT:  .LBB30_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB30_14
+; NO_SVE-NEXT:  .LBB30_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB30_15
+; NO_SVE-NEXT:  .LBB30_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB30_20
+; NO_SVE-NEXT:  .LBB30_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB30_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB30_25
+; NO_SVE-NEXT:  .LBB30_22: // %else32
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB30_26
+; NO_SVE-NEXT:  .LBB30_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB30_27
+; NO_SVE-NEXT:    b .LBB30_28
+; NO_SVE-NEXT:  .LBB30_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB30_22
+; NO_SVE-NEXT:  .LBB30_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB30_23
+; NO_SVE-NEXT:  .LBB30_26:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #13, .LBB30_28
+; NO_SVE-NEXT:  .LBB30_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_28: // %else38
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB30_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB30_33
+; NO_SVE-NEXT:  .LBB30_30: // %else44
+; NO_SVE-NEXT:    ldr q5, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB30_34
+; NO_SVE-NEXT:  .LBB30_31: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB30_35
+; NO_SVE-NEXT:    b .LBB30_36
+; NO_SVE-NEXT:  .LBB30_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB30_30
+; NO_SVE-NEXT:  .LBB30_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB30_31
+; NO_SVE-NEXT:  .LBB30_34:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #17, .LBB30_36
+; NO_SVE-NEXT:  .LBB30_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_36: // %else50
+; NO_SVE-NEXT:    ldr q5, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB30_40
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB30_41
+; NO_SVE-NEXT:  .LBB30_38: // %else56
+; NO_SVE-NEXT:    ldr q6, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB30_42
+; NO_SVE-NEXT:  .LBB30_39: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB30_43
+; NO_SVE-NEXT:    b .LBB30_44
+; NO_SVE-NEXT:  .LBB30_40: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB30_38
+; NO_SVE-NEXT:  .LBB30_41: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB30_39
+; NO_SVE-NEXT:  .LBB30_42:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #21, .LBB30_44
+; NO_SVE-NEXT:  .LBB30_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_44: // %else62
+; NO_SVE-NEXT:    ldr q6, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB30_48
+; NO_SVE-NEXT:  // %bb.45: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB30_49
+; NO_SVE-NEXT:  .LBB30_46: // %else68
+; NO_SVE-NEXT:    ldr q7, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB30_50
+; NO_SVE-NEXT:  .LBB30_47: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB30_51
+; NO_SVE-NEXT:    b .LBB30_52
+; NO_SVE-NEXT:  .LBB30_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB30_46
+; NO_SVE-NEXT:  .LBB30_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB30_47
+; NO_SVE-NEXT:  .LBB30_50:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #25, .LBB30_52
+; NO_SVE-NEXT:  .LBB30_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_52: // %else74
+; NO_SVE-NEXT:    ldr q7, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB30_56
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB30_57
+; NO_SVE-NEXT:  .LBB30_54: // %else80
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB30_58
+; NO_SVE-NEXT:  .LBB30_55: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB30_59
+; NO_SVE-NEXT:    b .LBB30_60
+; NO_SVE-NEXT:  .LBB30_56: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB30_54
+; NO_SVE-NEXT:  .LBB30_57: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB30_55
+; NO_SVE-NEXT:  .LBB30_58:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #29, .LBB30_60
+; NO_SVE-NEXT:  .LBB30_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB30_60: // %else86
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB30_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB30_63
+; NO_SVE-NEXT:  .LBB30_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB30_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB30_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB30_62
+; NO_SVE-NEXT:    b .LBB30_63
+;
+; VBITS_GE_256-LABEL: masked_gather_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z18.d]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [z17.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [z16.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [z6.d]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [z7.d]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [z0.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [z4.d]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b

 }
 
 ;
 ; LD1D (float)
 ;
 
 ; Scalarize 1 x double gathers
 define void @masked_gather_v1f64(<1 x double>* %a, <1 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v1f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    fcmp d0, #0.0
+; NO_SVE-NEXT:    // implicit-def: $d0
+; NO_SVE-NEXT:    b.ne .LBB31_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    ldr d0, [x1]
+; NO_SVE-NEXT:    fmov x8, d0
+; NO_SVE-NEXT:    ldr d0, [x8]
+; NO_SVE-NEXT:  .LBB31_2: // %else
+; NO_SVE-NEXT:    str d0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v1f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    fcmp d0, #0.0
 ; CHECK-NEXT:    // implicit-def: $d0
 ; CHECK-NEXT:    b.ne .LBB31_2
 ; CHECK-NEXT:  // %bb.1: // %cond.load
 ; CHECK-NEXT:    ldr d0, [x1]
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    ldr d0, [x8]
 ; CHECK-NEXT:  .LBB31_2: // %else
 ; CHECK-NEXT:    str d0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <1 x double>, <1 x double>* %a
   %ptrs = load <1 x double*>, <1 x double*>* %b
   %mask = fcmp oeq <1 x double> %cval, zeroinitializer
   %vals = call <1 x double> @llvm.masked.gather.v1f64(<1 x double*> %ptrs, i32 8, <1 x i1> %mask, <1 x double> undef)
   store <1 x double> %vals, <1 x double>* %a
   ret void
 }
 
 define void @masked_gather_v2f64(<2 x double>* %a, <2 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    xtn v0.2s, v0.2d
+; NO_SVE-NEXT:    mov w8, v0.s[1]
+; NO_SVE-NEXT:    fmov w9, s0
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbz w9, #0, .LBB32_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:  .LBB32_2: // %else
+; NO_SVE-NEXT:    tbz w8, #1, .LBB32_4
+; NO_SVE-NEXT:  // %bb.3: // %cond.load1
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB32_4: // %else2
+; NO_SVE-NEXT:    str q0, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v2f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr q0, [x0]
 ; CHECK-NEXT:    ptrue p0.d, vl2
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    fcmeq v0.2d, v0.2d, #0.0
 ; CHECK-NEXT:    cmpne p0.d, p0/z, z0.d, #0
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [z1.d]
 ; CHECK-NEXT:    str q0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <2 x double>, <2 x double>* %a
   %ptrs = load <2 x double*>, <2 x double*>* %b
   %mask = fcmp oeq <2 x double> %cval, zeroinitializer
   %vals = call <2 x double> @llvm.masked.gather.v2f64(<2 x double*> %ptrs, i32 8, <2 x i1> %mask, <2 x double> undef)
   store <2 x double> %vals, <2 x double>* %a
   ret void
 }
 
 define void @masked_gather_v4f64(<4 x double>* %a, <4 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    fcmeq v1.2d, v1.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v0.4s, v1.4s, v0.4s
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.4h, v0.4s
+; NO_SVE-NEXT:    umov w8, v0.h[1]
+; NO_SVE-NEXT:    umov w9, v0.h[2]
+; NO_SVE-NEXT:    umov w10, v0.h[0]
+; NO_SVE-NEXT:    umov w11, v0.h[3]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbz w10, #0, .LBB33_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB33_3
+; NO_SVE-NEXT:    b .LBB33_4
+; NO_SVE-NEXT:  .LBB33_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB33_4
+; NO_SVE-NEXT:  .LBB33_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB33_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB33_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB33_7
+; NO_SVE-NEXT:    b .LBB33_8
+; NO_SVE-NEXT:  .LBB33_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB33_8
+; NO_SVE-NEXT:  .LBB33_7: // %cond.load7
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB33_8: // %else8
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_gather_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
 ; CHECK-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    ret
   %cval = load <4 x double>, <4 x double>* %a
   %ptrs = load <4 x double*>, <4 x double*>* %b
   %mask = fcmp oeq <4 x double> %cval, zeroinitializer
   %vals = call <4 x double> @llvm.masked.gather.v4f64(<4 x double*> %ptrs, i32 8, <4 x i1> %mask, <4 x double> undef)
   store <4 x double> %vals, <4 x double>* %a
   ret void
 }
 
 define void @masked_gather_v8f64(<8 x double>* %a, <8 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    ldp q2, q3, [x0]
+; NO_SVE-NEXT:    fcmeq v1.2d, v1.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v1.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB34_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB34_3
+; NO_SVE-NEXT:    b .LBB34_4
+; NO_SVE-NEXT:  .LBB34_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB34_4
+; NO_SVE-NEXT:  .LBB34_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB34_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB34_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB34_7
+; NO_SVE-NEXT:    b .LBB34_8
+; NO_SVE-NEXT:  .LBB34_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB34_8
+; NO_SVE-NEXT:  .LBB34_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB34_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB34_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB34_11
+; NO_SVE-NEXT:    b .LBB34_12
+; NO_SVE-NEXT:  .LBB34_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB34_12
+; NO_SVE-NEXT:  .LBB34_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB34_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB34_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB34_15
+; NO_SVE-NEXT:    b .LBB34_16
+; NO_SVE-NEXT:  .LBB34_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB34_16
+; NO_SVE-NEXT:  .LBB34_15: // %cond.load19
+; NO_SVE-NEXT:    mov x8, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB34_16: // %else20
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: masked_gather_v8f64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %cval = load <8 x double>, <8 x double>* %a
   %ptrs = load <8 x double*>, <8 x double*>* %b
   %mask = fcmp oeq <8 x double> %cval, zeroinitializer
   %vals = call <8 x double> @llvm.masked.gather.v8f64(<8 x double*> %ptrs, i32 8, <8 x i1> %mask, <8 x double> undef)
   store <8 x double> %vals, <8 x double>* %a
   ret void
 }
 
 define void @masked_gather_v16f64(<16 x double>* %a, <16 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, #0.0
+; NO_SVE-NEXT:    ldp q4, q5, [x0]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #64]
+; NO_SVE-NEXT:    uzp1 v3.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    uzp1 v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v1.2d, v1.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v5.2d, v6.2d, #0.0
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    fcmeq v4.2d, v7.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    uzp1 v3.4s, v5.4s, v4.4s
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v3.8h
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v2.b[6]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v2.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[0]
+; NO_SVE-NEXT:    umov w12, v0.b[1]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v0.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v0.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v0.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v0.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbz w9, #0, .LBB35_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB35_3
+; NO_SVE-NEXT:    b .LBB35_4
+; NO_SVE-NEXT:  .LBB35_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB35_4
+; NO_SVE-NEXT:  .LBB35_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB35_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB35_7
+; NO_SVE-NEXT:    b .LBB35_8
+; NO_SVE-NEXT:  .LBB35_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB35_8
+; NO_SVE-NEXT:  .LBB35_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB35_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB35_11
+; NO_SVE-NEXT:    b .LBB35_12
+; NO_SVE-NEXT:  .LBB35_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB35_12
+; NO_SVE-NEXT:  .LBB35_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB35_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB35_15
+; NO_SVE-NEXT:    b .LBB35_16
+; NO_SVE-NEXT:  .LBB35_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB35_16
+; NO_SVE-NEXT:  .LBB35_15: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_16: // %else20
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB35_18
+; NO_SVE-NEXT:  // %bb.17: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB35_19
+; NO_SVE-NEXT:    b .LBB35_20
+; NO_SVE-NEXT:  .LBB35_18:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #9, .LBB35_20
+; NO_SVE-NEXT:  .LBB35_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_20: // %else26
+; NO_SVE-NEXT:    ldr q6, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB35_22
+; NO_SVE-NEXT:  // %bb.21: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB35_23
+; NO_SVE-NEXT:    b .LBB35_24
+; NO_SVE-NEXT:  .LBB35_22:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #11, .LBB35_24
+; NO_SVE-NEXT:  .LBB35_23: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_24: // %else32
+; NO_SVE-NEXT:    ldr q7, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB35_26
+; NO_SVE-NEXT:  // %bb.25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB35_27
+; NO_SVE-NEXT:    b .LBB35_28
+; NO_SVE-NEXT:  .LBB35_26:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #13, .LBB35_28
+; NO_SVE-NEXT:  .LBB35_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB35_28: // %else38
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB35_30
+; NO_SVE-NEXT:  // %bb.29: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB35_31
+; NO_SVE-NEXT:    b .LBB35_32
+; NO_SVE-NEXT:  .LBB35_30:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #15, .LBB35_32
+; NO_SVE-NEXT:  .LBB35_31: // %cond.load43
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB35_32: // %else44
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.d, p0/z, z1.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z2.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p4.d, p0/z, z3.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p3/z, [z4.d]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z6.d]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p4/z, [z7.d]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_gather_v16f64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %cval = load <16 x double>, <16 x double>* %a
   %ptrs = load <16 x double*>, <16 x double*>* %b
   %mask = fcmp oeq <16 x double> %cval, zeroinitializer
   %vals = call <16 x double> @llvm.masked.gather.v16f64(<16 x double*> %ptrs, i32 8, <16 x i1> %mask, <16 x double> undef)
   store <16 x double> %vals, <16 x double>* %a
   ret void
 }
 
 define void @masked_gather_v32f64(<32 x double>* %a, <32 x double*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #160]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, #0.0
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #128]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #224]
+; NO_SVE-NEXT:    uzp1 v4.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    ldp q6, q7, [x0, #192]
+; NO_SVE-NEXT:    fcmeq v1.2d, v1.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v2.8h, v4.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.2d, v6.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    ldp q3, q5, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v7.2d, v7.2d, #0.0
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    ldp q16, q17, [x0]
+; NO_SVE-NEXT:    fcmeq v1.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    uzp1 v5.4s, v6.4s, v7.4s
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v6.2d, v16.2d, #0.0
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    uzp1 v0.8h, v5.8h, v0.8h
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    fcmeq v4.2d, v17.2d, #0.0
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v1.4s, v3.4s, v1.4s
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    uzp1 v3.4s, v6.4s, v4.4s
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w8, v0.b[0]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w10, v0.b[1]
+; NO_SVE-NEXT:    umov w11, v0.b[2]
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    umov w12, v0.b[3]
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    uzp1 v1.8h, v3.8h, v1.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    umov w13, v0.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    and w9, w12, #0x1
+; NO_SVE-NEXT:    umov w10, v0.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w13, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #12
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    umov w10, v1.b[1]
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    umov w11, v1.b[0]
+; NO_SVE-NEXT:    uzp1 v4.4s, v4.4s, v5.4s
+; NO_SVE-NEXT:    umov w13, v1.b[3]
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v3.4s
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v4.8h
+; NO_SVE-NEXT:    bfi w11, w10, #1, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w11, w12, #2, #1
+; NO_SVE-NEXT:    and w12, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    bfi w11, w10, #3, #1
+; NO_SVE-NEXT:    umov w10, v1.b[7]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    bfi w11, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v0.b[6]
+; NO_SVE-NEXT:    bfi w11, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[0]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    orr w9, w11, w13, lsl #6
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #7
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #8
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB36_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB36_3
+; NO_SVE-NEXT:    b .LBB36_4
+; NO_SVE-NEXT:  .LBB36_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB36_4
+; NO_SVE-NEXT:  .LBB36_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_4: // %else2
+; NO_SVE-NEXT:    ldr q2, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB36_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB36_7
+; NO_SVE-NEXT:    b .LBB36_8
+; NO_SVE-NEXT:  .LBB36_6:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB36_8
+; NO_SVE-NEXT:  .LBB36_7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB36_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB36_11
+; NO_SVE-NEXT:    b .LBB36_12
+; NO_SVE-NEXT:  .LBB36_10:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB36_12
+; NO_SVE-NEXT:  .LBB36_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB36_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB36_15
+; NO_SVE-NEXT:    b .LBB36_16
+; NO_SVE-NEXT:  .LBB36_14:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB36_16
+; NO_SVE-NEXT:  .LBB36_15: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_16: // %else20
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB36_18
+; NO_SVE-NEXT:  // %bb.17: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB36_19
+; NO_SVE-NEXT:    b .LBB36_20
+; NO_SVE-NEXT:  .LBB36_18:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #9, .LBB36_20
+; NO_SVE-NEXT:  .LBB36_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_20: // %else26
+; NO_SVE-NEXT:    ldr q6, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB36_22
+; NO_SVE-NEXT:  // %bb.21: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB36_23
+; NO_SVE-NEXT:    b .LBB36_24
+; NO_SVE-NEXT:  .LBB36_22:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #11, .LBB36_24
+; NO_SVE-NEXT:  .LBB36_23: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_24: // %else32
+; NO_SVE-NEXT:    ldr q7, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB36_26
+; NO_SVE-NEXT:  // %bb.25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB36_27
+; NO_SVE-NEXT:    b .LBB36_28
+; NO_SVE-NEXT:  .LBB36_26:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    tbz w8, #13, .LBB36_28
+; NO_SVE-NEXT:  .LBB36_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_28: // %else38
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB36_30
+; NO_SVE-NEXT:  // %bb.29: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB36_31
+; NO_SVE-NEXT:    b .LBB36_32
+; NO_SVE-NEXT:  .LBB36_30:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    tbz w8, #15, .LBB36_32
+; NO_SVE-NEXT:  .LBB36_31: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_32: // %else44
+; NO_SVE-NEXT:    ldr q17, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB36_34
+; NO_SVE-NEXT:  // %bb.33: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB36_35
+; NO_SVE-NEXT:    b .LBB36_36
+; NO_SVE-NEXT:  .LBB36_34:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    tbz w8, #17, .LBB36_36
+; NO_SVE-NEXT:  .LBB36_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_36: // %else50
+; NO_SVE-NEXT:    ldr q18, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB36_38
+; NO_SVE-NEXT:  // %bb.37: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d18
+; NO_SVE-NEXT:    ld1 { v17.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB36_39
+; NO_SVE-NEXT:    b .LBB36_40
+; NO_SVE-NEXT:  .LBB36_38:
+; NO_SVE-NEXT:    // implicit-def: $q17
+; NO_SVE-NEXT:    tbz w8, #19, .LBB36_40
+; NO_SVE-NEXT:  .LBB36_39: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v18.d[1]
+; NO_SVE-NEXT:    ld1 { v17.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_40: // %else56
+; NO_SVE-NEXT:    ldr q19, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB36_42
+; NO_SVE-NEXT:  // %bb.41: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d19
+; NO_SVE-NEXT:    ld1 { v18.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB36_43
+; NO_SVE-NEXT:    b .LBB36_44
+; NO_SVE-NEXT:  .LBB36_42:
+; NO_SVE-NEXT:    // implicit-def: $q18
+; NO_SVE-NEXT:    tbz w8, #21, .LBB36_44
+; NO_SVE-NEXT:  .LBB36_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v19.d[1]
+; NO_SVE-NEXT:    ld1 { v18.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_44: // %else62
+; NO_SVE-NEXT:    ldr q20, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB36_46
+; NO_SVE-NEXT:  // %bb.45: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d20
+; NO_SVE-NEXT:    ld1 { v19.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB36_47
+; NO_SVE-NEXT:    b .LBB36_48
+; NO_SVE-NEXT:  .LBB36_46:
+; NO_SVE-NEXT:    // implicit-def: $q19
+; NO_SVE-NEXT:    tbz w8, #23, .LBB36_48
+; NO_SVE-NEXT:  .LBB36_47: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v20.d[1]
+; NO_SVE-NEXT:    ld1 { v19.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_48: // %else68
+; NO_SVE-NEXT:    ldr q21, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB36_50
+; NO_SVE-NEXT:  // %bb.49: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d21
+; NO_SVE-NEXT:    ld1 { v20.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB36_51
+; NO_SVE-NEXT:    b .LBB36_52
+; NO_SVE-NEXT:  .LBB36_50:
+; NO_SVE-NEXT:    // implicit-def: $q20
+; NO_SVE-NEXT:    tbz w8, #25, .LBB36_52
+; NO_SVE-NEXT:  .LBB36_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v21.d[1]
+; NO_SVE-NEXT:    ld1 { v20.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_52: // %else74
+; NO_SVE-NEXT:    ldr q22, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB36_54
+; NO_SVE-NEXT:  // %bb.53: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d22
+; NO_SVE-NEXT:    ld1 { v21.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB36_55
+; NO_SVE-NEXT:    b .LBB36_56
+; NO_SVE-NEXT:  .LBB36_54:
+; NO_SVE-NEXT:    // implicit-def: $q21
+; NO_SVE-NEXT:    tbz w8, #27, .LBB36_56
+; NO_SVE-NEXT:  .LBB36_55: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v22.d[1]
+; NO_SVE-NEXT:    ld1 { v21.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_56: // %else80
+; NO_SVE-NEXT:    ldr q23, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB36_58
+; NO_SVE-NEXT:  // %bb.57: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d23
+; NO_SVE-NEXT:    ld1 { v22.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB36_59
+; NO_SVE-NEXT:    b .LBB36_60
+; NO_SVE-NEXT:  .LBB36_58:
+; NO_SVE-NEXT:    // implicit-def: $q22
+; NO_SVE-NEXT:    tbz w8, #29, .LBB36_60
+; NO_SVE-NEXT:  .LBB36_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v23.d[1]
+; NO_SVE-NEXT:    ld1 { v22.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB36_60: // %else86
+; NO_SVE-NEXT:    ldr q24, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB36_62
+; NO_SVE-NEXT:  // %bb.61: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d24
+; NO_SVE-NEXT:    ld1 { v23.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB36_63
+; NO_SVE-NEXT:    b .LBB36_64
+; NO_SVE-NEXT:  .LBB36_62:
+; NO_SVE-NEXT:    // implicit-def: $q23
+; NO_SVE-NEXT:    tbz w8, #31, .LBB36_64
+; NO_SVE-NEXT:  .LBB36_63: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v24.d[1]
+; NO_SVE-NEXT:    ld1 { v23.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB36_64: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    stp q16, q17, [x0, #128]
+; NO_SVE-NEXT:    stp q18, q19, [x0, #160]
+; NO_SVE-NEXT:    stp q20, q21, [x0, #192]
+; NO_SVE-NEXT:    stp q22, q23, [x0, #224]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #12
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    mov x13, #24
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z19.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z21.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z22.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z20.d }, p0/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z0.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p0/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p0/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p0/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z23.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p2/z, [z19.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z6.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [z21.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z2.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p2/z, [z22.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z3.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p2/z, [z20.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z4.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p2/z, [z18.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z5.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p2/z, [z17.d]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [z16.d]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z7.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [z23.d]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_v32f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p1/z, [z1.d]
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cval = load <32 x double>, <32 x double>* %a
   %ptrs = load <32 x double*>, <32 x double*>* %b
   %mask = fcmp oeq <32 x double> %cval, zeroinitializer
   %vals = call <32 x double> @llvm.masked.gather.v32f64(<32 x double*> %ptrs, i32 8, <32 x i1> %mask, <32 x double> undef)
   store <32 x double> %vals, <32 x double>* %a
   ret void
 }
 
 ; The above tests test the types, the below tests check that the addressing
 ; modes still function
 
 define void @masked_gather_32b_scaled_sext_f16(<32 x half>* %a, <32 x i32>* %b, half* %base) #0 {
-; VBITS_GE_2048-LABEL: masked_gather_32b_scaled_sext_f16:
-; VBITS_GE_2048:       // %bb.0:
-; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
-; VBITS_GE_2048-NEXT:    ptrue p1.s, vl32
-; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
-; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p1/z, [x1]
-; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
-; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
-; VBITS_GE_2048-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, sxtw #1]
-; VBITS_GE_2048-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
-; VBITS_GE_2048-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_32b_scaled_sext_f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    ldr q2, [x1]
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v1.b[0]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v3.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v3.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v3.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v3.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w12, v3.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v3.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v0.8h, v4.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v3.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v1.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    sshll v0.2d, v2.2s, #1
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB37_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    b .LBB37_3
+; NO_SVE-NEXT:  .LBB37_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:  .LBB37_3: // %else
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    sshll2 v2.2d, v2.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB37_13
+; NO_SVE-NEXT:  // %bb.4: // %else2
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB37_14
+; NO_SVE-NEXT:  .LBB37_5: // %else5
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB37_15
+; NO_SVE-NEXT:  .LBB37_6: // %else8
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB37_8
+; NO_SVE-NEXT:  .LBB37_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB37_8: // %else11
+; NO_SVE-NEXT:    ldr q4, [x1, #32]
+; NO_SVE-NEXT:    sshll2 v3.2d, v3.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB37_16
+; NO_SVE-NEXT:  // %bb.9: // %else14
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB37_17
+; NO_SVE-NEXT:  .LBB37_10: // %else17
+; NO_SVE-NEXT:    sshll v3.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB37_18
+; NO_SVE-NEXT:  .LBB37_11: // %else20
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB37_19
+; NO_SVE-NEXT:  .LBB37_12: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB37_20
+; NO_SVE-NEXT:  .LBB37_13: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #2, .LBB37_5
+; NO_SVE-NEXT:  .LBB37_14: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB37_6
+; NO_SVE-NEXT:  .LBB37_15: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB37_7
+; NO_SVE-NEXT:    b .LBB37_8
+; NO_SVE-NEXT:  .LBB37_16: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #6, .LBB37_10
+; NO_SVE-NEXT:  .LBB37_17: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    sshll v3.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbz w8, #7, .LBB37_11
+; NO_SVE-NEXT:  .LBB37_18: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB37_12
+; NO_SVE-NEXT:  .LBB37_19:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:  .LBB37_20: // %else23
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    sshll2 v4.2d, v4.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB37_30
+; NO_SVE-NEXT:  // %bb.21: // %else26
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB37_31
+; NO_SVE-NEXT:  .LBB37_22: // %else29
+; NO_SVE-NEXT:    sshll v5.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB37_32
+; NO_SVE-NEXT:  .LBB37_23: // %else32
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB37_25
+; NO_SVE-NEXT:  .LBB37_24: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB37_25: // %else35
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    sshll2 v3.2d, v3.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB37_33
+; NO_SVE-NEXT:  // %bb.26: // %else38
+; NO_SVE-NEXT:    add v3.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB37_34
+; NO_SVE-NEXT:  .LBB37_27: // %else41
+; NO_SVE-NEXT:    sshll v4.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB37_35
+; NO_SVE-NEXT:  .LBB37_28: // %else44
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB37_36
+; NO_SVE-NEXT:  .LBB37_29: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB37_37
+; NO_SVE-NEXT:  .LBB37_30: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #10, .LBB37_22
+; NO_SVE-NEXT:  .LBB37_31: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    sshll v5.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbz w8, #11, .LBB37_23
+; NO_SVE-NEXT:  .LBB37_32: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB37_24
+; NO_SVE-NEXT:    b .LBB37_25
+; NO_SVE-NEXT:  .LBB37_33: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:    add v3.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #14, .LBB37_27
+; NO_SVE-NEXT:  .LBB37_34: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    sshll v4.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbz w8, #15, .LBB37_28
+; NO_SVE-NEXT:  .LBB37_35: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB37_29
+; NO_SVE-NEXT:  .LBB37_36:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:  .LBB37_37: // %else47
+; NO_SVE-NEXT:    ldr q4, [x1, #80]
+; NO_SVE-NEXT:    sshll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB37_47
+; NO_SVE-NEXT:  // %bb.38: // %else50
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB37_48
+; NO_SVE-NEXT:  .LBB37_39: // %else53
+; NO_SVE-NEXT:    sshll v6.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB37_49
+; NO_SVE-NEXT:  .LBB37_40: // %else56
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB37_42
+; NO_SVE-NEXT:  .LBB37_41: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB37_42: // %else59
+; NO_SVE-NEXT:    ldr q6, [x1, #96]
+; NO_SVE-NEXT:    sshll2 v4.2d, v4.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB37_50
+; NO_SVE-NEXT:  // %bb.43: // %else62
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB37_51
+; NO_SVE-NEXT:  .LBB37_44: // %else65
+; NO_SVE-NEXT:    sshll v5.2d, v6.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB37_52
+; NO_SVE-NEXT:  .LBB37_45: // %else68
+; NO_SVE-NEXT:    add v7.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB37_53
+; NO_SVE-NEXT:  .LBB37_46: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v4.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB37_54
+; NO_SVE-NEXT:  .LBB37_47: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #18, .LBB37_39
+; NO_SVE-NEXT:  .LBB37_48: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    sshll v6.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbz w8, #19, .LBB37_40
+; NO_SVE-NEXT:  .LBB37_49: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB37_41
+; NO_SVE-NEXT:    b .LBB37_42
+; NO_SVE-NEXT:  .LBB37_50: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #22, .LBB37_44
+; NO_SVE-NEXT:  .LBB37_51: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    sshll v5.2d, v6.2s, #1
+; NO_SVE-NEXT:    tbz w8, #23, .LBB37_45
+; NO_SVE-NEXT:  .LBB37_52: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x9]
+; NO_SVE-NEXT:    add v7.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB37_46
+; NO_SVE-NEXT:  .LBB37_53:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:  .LBB37_54: // %else71
+; NO_SVE-NEXT:    ldr q5, [x1, #112]
+; NO_SVE-NEXT:    sshll2 v6.2d, v6.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB37_63
+; NO_SVE-NEXT:  // %bb.55: // %else74
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB37_64
+; NO_SVE-NEXT:  .LBB37_56: // %else77
+; NO_SVE-NEXT:    sshll v7.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB37_65
+; NO_SVE-NEXT:  .LBB37_57: // %else80
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB37_66
+; NO_SVE-NEXT:  .LBB37_58: // %else83
+; NO_SVE-NEXT:    sshll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB37_67
+; NO_SVE-NEXT:  .LBB37_59: // %else86
+; NO_SVE-NEXT:    add v1.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB37_68
+; NO_SVE-NEXT:  .LBB37_60: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB37_62
+; NO_SVE-NEXT:  .LBB37_61: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB37_62: // %else92
+; NO_SVE-NEXT:    stp q0, q2, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB37_63: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[1], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #26, .LBB37_56
+; NO_SVE-NEXT:  .LBB37_64: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[2], [x9]
+; NO_SVE-NEXT:    sshll v7.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbz w8, #27, .LBB37_57
+; NO_SVE-NEXT:  .LBB37_65: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB37_58
+; NO_SVE-NEXT:  .LBB37_66: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[4], [x9]
+; NO_SVE-NEXT:    sshll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbz w8, #29, .LBB37_59
+; NO_SVE-NEXT:  .LBB37_67: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[5], [x9]
+; NO_SVE-NEXT:    add v1.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #30, .LBB37_60
+; NO_SVE-NEXT:  .LBB37_68: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v4.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB37_61
+; NO_SVE-NEXT:    b .LBB37_62
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_scaled_sext_f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p1/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p1/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p1/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z0.h, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.h, p0/z, z1.h, #0.0
+; VBITS_GE_256-NEXT:    mov z0.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    mov z1.h, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    ld1h { z4.s }, p2/z, [x2, z4.s, sxtw #1]
+; VBITS_GE_256-NEXT:    cmpne p2.s, p1/z, z0.s, #0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1h { z0.s }, p2/z, [x2, z3.s, sxtw #1]
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.s, p1/z, z1.s, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.s }, p2/z, [x2, z5.s, sxtw #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.s }, p1/z, [x2, z2.s, sxtw #1]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z3.h, p1, z3.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    splice z1.h, p1, z1.h, z2.h
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: masked_gather_32b_scaled_sext_f16:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_1024-NEXT:    ptrue p1.s, vl32
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p1/z, [x1]
+; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
+; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
+; VBITS_GE_1024-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, sxtw #1]
+; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %cvals = load <32 x half>, <32 x half>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = sext <32 x i32> %idxs to <32 x i64>
   %ptrs = getelementptr half, half* %base, <32 x i64> %ext
   %mask = fcmp oeq <32 x half> %cvals, zeroinitializer
   %vals = call <32 x half> @llvm.masked.gather.v32f16(<32 x half*> %ptrs, i32 8, <32 x i1> %mask, <32 x half> undef)
   store <32 x half> %vals, <32 x half>* %a
   ret void
 }
 
 define void @masked_gather_32b_scaled_sext_f32(<32 x float>* %a, <32 x i32>* %b, float* %base) #0 {
-; VBITS_GE_2048-LABEL: masked_gather_32b_scaled_sext_f32:
-; VBITS_GE_2048:       // %bb.0:
-; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
-; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
-; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p0/z, [x1]
-; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
-; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p1/z, [x2, z1.s, sxtw #2]
-; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
-; VBITS_GE_2048-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_32b_scaled_sext_f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldp q0, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v4.8h
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.4s, v6.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #48]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v5.8h, v5.8h, v6.8h
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v5.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w13, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v3.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    fcmeq v2.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v0.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[6]
+; NO_SVE-NEXT:    umov w14, v5.b[5]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v5.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v2.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    sshll v0.2d, v1.2s, #2
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB38_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    b .LBB38_3
+; NO_SVE-NEXT:  .LBB38_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:  .LBB38_3: // %else
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    sshll2 v1.2d, v1.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB38_8
+; NO_SVE-NEXT:  // %bb.4: // %else2
+; NO_SVE-NEXT:    add v1.2d, v2.2d, v1.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB38_9
+; NO_SVE-NEXT:  .LBB38_5: // %else5
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB38_10
+; NO_SVE-NEXT:  .LBB38_6: // %else8
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB38_11
+; NO_SVE-NEXT:  .LBB38_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_12
+; NO_SVE-NEXT:  .LBB38_8: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    add v1.2d, v2.2d, v1.2d
+; NO_SVE-NEXT:    tbz w8, #2, .LBB38_5
+; NO_SVE-NEXT:  .LBB38_9: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #2
+; NO_SVE-NEXT:    tbz w8, #3, .LBB38_6
+; NO_SVE-NEXT:  .LBB38_10: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB38_7
+; NO_SVE-NEXT:  .LBB38_11:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:  .LBB38_12: // %else11
+; NO_SVE-NEXT:    ldr q4, [x1, #32]
+; NO_SVE-NEXT:    sshll2 v3.2d, v3.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB38_17
+; NO_SVE-NEXT:  // %bb.13: // %else14
+; NO_SVE-NEXT:    add v3.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB38_18
+; NO_SVE-NEXT:  .LBB38_14: // %else17
+; NO_SVE-NEXT:    sshll v5.2d, v4.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB38_19
+; NO_SVE-NEXT:  .LBB38_15: // %else20
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB38_20
+; NO_SVE-NEXT:  .LBB38_16: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_21
+; NO_SVE-NEXT:  .LBB38_17: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    add v3.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #6, .LBB38_14
+; NO_SVE-NEXT:  .LBB38_18: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v5.2d, v4.2s, #2
+; NO_SVE-NEXT:    tbz w8, #7, .LBB38_15
+; NO_SVE-NEXT:  .LBB38_19: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB38_16
+; NO_SVE-NEXT:  .LBB38_20:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:  .LBB38_21: // %else23
+; NO_SVE-NEXT:    ldr q5, [x1, #48]
+; NO_SVE-NEXT:    sshll2 v4.2d, v4.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB38_26
+; NO_SVE-NEXT:  // %bb.22: // %else26
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB38_27
+; NO_SVE-NEXT:  .LBB38_23: // %else29
+; NO_SVE-NEXT:    sshll v6.2d, v5.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB38_28
+; NO_SVE-NEXT:  .LBB38_24: // %else32
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB38_29
+; NO_SVE-NEXT:  .LBB38_25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_30
+; NO_SVE-NEXT:  .LBB38_26: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #10, .LBB38_23
+; NO_SVE-NEXT:  .LBB38_27: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v6.2d, v5.2s, #2
+; NO_SVE-NEXT:    tbz w8, #11, .LBB38_24
+; NO_SVE-NEXT:  .LBB38_28: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB38_25
+; NO_SVE-NEXT:  .LBB38_29:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:  .LBB38_30: // %else35
+; NO_SVE-NEXT:    ldr q6, [x1, #64]
+; NO_SVE-NEXT:    sshll2 v5.2d, v5.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB38_35
+; NO_SVE-NEXT:  // %bb.31: // %else38
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB38_36
+; NO_SVE-NEXT:  .LBB38_32: // %else41
+; NO_SVE-NEXT:    sshll v7.2d, v6.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB38_37
+; NO_SVE-NEXT:  .LBB38_33: // %else44
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB38_38
+; NO_SVE-NEXT:  .LBB38_34: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_39
+; NO_SVE-NEXT:  .LBB38_35: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #14, .LBB38_32
+; NO_SVE-NEXT:  .LBB38_36: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v7.2d, v6.2s, #2
+; NO_SVE-NEXT:    tbz w8, #15, .LBB38_33
+; NO_SVE-NEXT:  .LBB38_37: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB38_34
+; NO_SVE-NEXT:  .LBB38_38:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:  .LBB38_39: // %else47
+; NO_SVE-NEXT:    ldr q7, [x1, #80]
+; NO_SVE-NEXT:    sshll2 v6.2d, v6.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB38_44
+; NO_SVE-NEXT:  // %bb.40: // %else50
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB38_45
+; NO_SVE-NEXT:  .LBB38_41: // %else53
+; NO_SVE-NEXT:    sshll v16.2d, v7.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB38_46
+; NO_SVE-NEXT:  .LBB38_42: // %else56
+; NO_SVE-NEXT:    add v17.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB38_47
+; NO_SVE-NEXT:  .LBB38_43: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_48
+; NO_SVE-NEXT:  .LBB38_44: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #18, .LBB38_41
+; NO_SVE-NEXT:  .LBB38_45: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v16.2d, v7.2s, #2
+; NO_SVE-NEXT:    tbz w8, #19, .LBB38_42
+; NO_SVE-NEXT:  .LBB38_46: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    add v17.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB38_43
+; NO_SVE-NEXT:  .LBB38_47:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:  .LBB38_48: // %else59
+; NO_SVE-NEXT:    ldr q16, [x1, #96]
+; NO_SVE-NEXT:    sshll2 v7.2d, v7.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB38_53
+; NO_SVE-NEXT:  // %bb.49: // %else62
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB38_54
+; NO_SVE-NEXT:  .LBB38_50: // %else65
+; NO_SVE-NEXT:    sshll v17.2d, v16.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB38_55
+; NO_SVE-NEXT:  .LBB38_51: // %else68
+; NO_SVE-NEXT:    add v18.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB38_56
+; NO_SVE-NEXT:  .LBB38_52: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d18
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    b .LBB38_57
+; NO_SVE-NEXT:  .LBB38_53: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #22, .LBB38_50
+; NO_SVE-NEXT:  .LBB38_54: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v17.2d, v16.2s, #2
+; NO_SVE-NEXT:    tbz w8, #23, .LBB38_51
+; NO_SVE-NEXT:  .LBB38_55: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    add v18.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB38_52
+; NO_SVE-NEXT:  .LBB38_56:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:  .LBB38_57: // %else71
+; NO_SVE-NEXT:    ldr q17, [x1, #112]
+; NO_SVE-NEXT:    sshll2 v16.2d, v16.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB38_62
+; NO_SVE-NEXT:  // %bb.58: // %else74
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB38_63
+; NO_SVE-NEXT:  .LBB38_59: // %else77
+; NO_SVE-NEXT:    sshll v18.2d, v17.2s, #2
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB38_64
+; NO_SVE-NEXT:  .LBB38_60: // %else80
+; NO_SVE-NEXT:    add v18.2d, v2.2d, v18.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB38_65
+; NO_SVE-NEXT:  .LBB38_61: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d18
+; NO_SVE-NEXT:    ld1 { v16.s }[0], [x9]
+; NO_SVE-NEXT:    sshll2 v17.2d, v17.4s, #2
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB38_66
+; NO_SVE-NEXT:    b .LBB38_67
+; NO_SVE-NEXT:  .LBB38_62: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v18.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #26, .LBB38_59
+; NO_SVE-NEXT:  .LBB38_63: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    sshll v18.2d, v17.2s, #2
+; NO_SVE-NEXT:    tbz w8, #27, .LBB38_60
+; NO_SVE-NEXT:  .LBB38_64: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    add v18.2d, v2.2d, v18.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB38_61
+; NO_SVE-NEXT:  .LBB38_65:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    sshll2 v17.2d, v17.4s, #2
+; NO_SVE-NEXT:    tbz w8, #29, .LBB38_67
+; NO_SVE-NEXT:  .LBB38_66: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v18.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB38_67: // %else86
+; NO_SVE-NEXT:    add v2.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB38_71
+; NO_SVE-NEXT:  // %bb.68: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB38_70
+; NO_SVE-NEXT:  .LBB38_69: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB38_70: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB38_71: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB38_69
+; NO_SVE-NEXT:    b .LBB38_70
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_scaled_sext_f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p3/z, [x2, z4.s, sxtw #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p2/z, [x2, z6.s, sxtw #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p1/z, [x2, z5.s, sxtw #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p4/z, [x2, z7.s, sxtw #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: masked_gather_32b_scaled_sext_f32:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
+; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p0/z, [x1]
+; VBITS_GE_1024-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
+; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p1/z, [x2, z1.s, sxtw #2]
+; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = sext <32 x i32> %idxs to <32 x i64>
   %ptrs = getelementptr float, float* %base, <32 x i64> %ext
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> undef)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_32b_scaled_sext_f64(<32 x double>* %a, <32 x i32>* %b, double* %base) #0 {
+; NO_SVE-LABEL: masked_gather_32b_scaled_sext_f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #160]
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #128]
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v3.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    ldp q0, q2, [x0, #224]
+; NO_SVE-NEXT:    fcmeq v6.2d, v6.2d, #0.0
+; NO_SVE-NEXT:    fcmeq v0.2d, v0.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v5.4s, v5.4s, v6.4s
+; NO_SVE-NEXT:    ldp q7, q16, [x0, #192]
+; NO_SVE-NEXT:    fcmeq v2.2d, v2.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v5.8h, v3.8h
+; NO_SVE-NEXT:    fcmeq v7.2d, v7.2d, #0.0
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v2.4s
+; NO_SVE-NEXT:    fcmeq v16.2d, v16.2d, #0.0
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    ldp q17, q18, [x0]
+; NO_SVE-NEXT:    uzp1 v2.4s, v7.4s, v16.4s
+; NO_SVE-NEXT:    fcmeq v7.2d, v17.2d, #0.0
+; NO_SVE-NEXT:    ldp q19, q4, [x0, #32]
+; NO_SVE-NEXT:    uzp1 v0.8h, v2.8h, v0.8h
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    fcmeq v6.2d, v18.2d, #0.0
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    fcmeq v5.2d, v19.2d, #0.0
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v3.4s, v5.4s, v4.4s
+; NO_SVE-NEXT:    umov w8, v0.b[0]
+; NO_SVE-NEXT:    uzp1 v4.4s, v7.4s, v6.4s
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w10, v0.b[1]
+; NO_SVE-NEXT:    umov w11, v0.b[2]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    umov w13, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #7
+; NO_SVE-NEXT:    uzp1 v2.8h, v4.8h, v3.8h
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #10
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w9, v0.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #11
+; NO_SVE-NEXT:    and w10, w11, #0x1
+; NO_SVE-NEXT:    fcmeq v6.2d, v6.2d, #0.0
+; NO_SVE-NEXT:    umov w12, v2.b[2]
+; NO_SVE-NEXT:    fcmeq v5.2d, v5.2d, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    fcmeq v4.2d, v4.2d, #0.0
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v3.2d, v3.2d, #0.0
+; NO_SVE-NEXT:    umov w10, v2.b[1]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w11, v2.b[0]
+; NO_SVE-NEXT:    umov w13, v2.b[3]
+; NO_SVE-NEXT:    uzp1 v5.4s, v5.4s, v6.4s
+; NO_SVE-NEXT:    umov w14, v2.b[4]
+; NO_SVE-NEXT:    uzp1 v3.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[5]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v5.8h
+; NO_SVE-NEXT:    bfi w11, w10, #1, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w11, w12, #2, #1
+; NO_SVE-NEXT:    and w12, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    dup v5.2d, x2
+; NO_SVE-NEXT:    bfi w11, w10, #3, #1
+; NO_SVE-NEXT:    umov w10, v2.b[7]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    bfi w11, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v0.b[6]
+; NO_SVE-NEXT:    bfi w11, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v2.b[0]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[1]
+; NO_SVE-NEXT:    orr w9, w11, w13, lsl #6
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v2.b[2]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #7
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v2.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #8
+; NO_SVE-NEXT:    umov w10, v2.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v2.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v2.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    sshll v0.2d, v1.2s, #3
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v2.2d, v5.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB39_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ldr d0, [x9]
+; NO_SVE-NEXT:    b .LBB39_3
+; NO_SVE-NEXT:  .LBB39_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:  .LBB39_3: // %else
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    sshll2 v1.2d, v1.4s, #3
+; NO_SVE-NEXT:    tbz w8, #1, .LBB39_5
+; NO_SVE-NEXT:  // %bb.4: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_5: // %else2
+; NO_SVE-NEXT:    add v2.2d, v5.2d, v1.2d
+; NO_SVE-NEXT:    tbz w8, #2, .LBB39_7
+; NO_SVE-NEXT:  // %bb.6: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB39_8
+; NO_SVE-NEXT:    b .LBB39_9
+; NO_SVE-NEXT:  .LBB39_7:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    sshll v4.2d, v3.2s, #3
+; NO_SVE-NEXT:    tbz w8, #3, .LBB39_9
+; NO_SVE-NEXT:  .LBB39_8: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_9: // %else8
+; NO_SVE-NEXT:    add v4.2d, v5.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB39_11
+; NO_SVE-NEXT:  // %bb.10: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_12
+; NO_SVE-NEXT:  .LBB39_11:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:  .LBB39_12: // %else11
+; NO_SVE-NEXT:    ldr q6, [x1, #32]
+; NO_SVE-NEXT:    sshll2 v3.2d, v3.4s, #3
+; NO_SVE-NEXT:    tbz w8, #5, .LBB39_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_14: // %else14
+; NO_SVE-NEXT:    add v4.2d, v5.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #6, .LBB39_16
+; NO_SVE-NEXT:  // %bb.15: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v7.2d, v6.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB39_17
+; NO_SVE-NEXT:    b .LBB39_18
+; NO_SVE-NEXT:  .LBB39_16:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    sshll v7.2d, v6.2s, #3
+; NO_SVE-NEXT:    tbz w8, #7, .LBB39_18
+; NO_SVE-NEXT:  .LBB39_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_18: // %else20
+; NO_SVE-NEXT:    add v7.2d, v5.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB39_20
+; NO_SVE-NEXT:  // %bb.19: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v4.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_21
+; NO_SVE-NEXT:  .LBB39_20:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:  .LBB39_21: // %else23
+; NO_SVE-NEXT:    ldr q16, [x1, #48]
+; NO_SVE-NEXT:    sshll2 v6.2d, v6.4s, #3
+; NO_SVE-NEXT:    tbz w8, #9, .LBB39_23
+; NO_SVE-NEXT:  // %bb.22: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v4.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_23: // %else26
+; NO_SVE-NEXT:    add v7.2d, v5.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #10, .LBB39_25
+; NO_SVE-NEXT:  // %bb.24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v17.2d, v16.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB39_26
+; NO_SVE-NEXT:    b .LBB39_27
+; NO_SVE-NEXT:  .LBB39_25:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    sshll v17.2d, v16.2s, #3
+; NO_SVE-NEXT:    tbz w8, #11, .LBB39_27
+; NO_SVE-NEXT:  .LBB39_26: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_27: // %else32
+; NO_SVE-NEXT:    add v17.2d, v5.2d, v17.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB39_29
+; NO_SVE-NEXT:  // %bb.28: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v7.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_30
+; NO_SVE-NEXT:  .LBB39_29:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:  .LBB39_30: // %else35
+; NO_SVE-NEXT:    ldr q18, [x1, #64]
+; NO_SVE-NEXT:    sshll2 v16.2d, v16.4s, #3
+; NO_SVE-NEXT:    tbz w8, #13, .LBB39_32
+; NO_SVE-NEXT:  // %bb.31: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v7.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_32: // %else38
+; NO_SVE-NEXT:    add v17.2d, v5.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #14, .LBB39_34
+; NO_SVE-NEXT:  // %bb.33: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v19.2d, v18.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB39_35
+; NO_SVE-NEXT:    b .LBB39_36
+; NO_SVE-NEXT:  .LBB39_34:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    sshll v19.2d, v18.2s, #3
+; NO_SVE-NEXT:    tbz w8, #15, .LBB39_36
+; NO_SVE-NEXT:  .LBB39_35: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_36: // %else44
+; NO_SVE-NEXT:    add v19.2d, v5.2d, v19.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB39_38
+; NO_SVE-NEXT:  // %bb.37: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d19
+; NO_SVE-NEXT:    ld1 { v17.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_39
+; NO_SVE-NEXT:  .LBB39_38:
+; NO_SVE-NEXT:    // implicit-def: $q17
+; NO_SVE-NEXT:  .LBB39_39: // %else47
+; NO_SVE-NEXT:    ldr q20, [x1, #80]
+; NO_SVE-NEXT:    sshll2 v18.2d, v18.4s, #3
+; NO_SVE-NEXT:    tbz w8, #17, .LBB39_41
+; NO_SVE-NEXT:  // %bb.40: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v19.d[1]
+; NO_SVE-NEXT:    ld1 { v17.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_41: // %else50
+; NO_SVE-NEXT:    add v19.2d, v5.2d, v18.2d
+; NO_SVE-NEXT:    tbz w8, #18, .LBB39_43
+; NO_SVE-NEXT:  // %bb.42: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d19
+; NO_SVE-NEXT:    ld1 { v18.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v21.2d, v20.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB39_44
+; NO_SVE-NEXT:    b .LBB39_45
+; NO_SVE-NEXT:  .LBB39_43:
+; NO_SVE-NEXT:    // implicit-def: $q18
+; NO_SVE-NEXT:    sshll v21.2d, v20.2s, #3
+; NO_SVE-NEXT:    tbz w8, #19, .LBB39_45
+; NO_SVE-NEXT:  .LBB39_44: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v19.d[1]
+; NO_SVE-NEXT:    ld1 { v18.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_45: // %else56
+; NO_SVE-NEXT:    add v21.2d, v5.2d, v21.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB39_47
+; NO_SVE-NEXT:  // %bb.46: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d21
+; NO_SVE-NEXT:    ld1 { v19.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_48
+; NO_SVE-NEXT:  .LBB39_47:
+; NO_SVE-NEXT:    // implicit-def: $q19
+; NO_SVE-NEXT:  .LBB39_48: // %else59
+; NO_SVE-NEXT:    ldr q22, [x1, #96]
+; NO_SVE-NEXT:    sshll2 v20.2d, v20.4s, #3
+; NO_SVE-NEXT:    tbz w8, #21, .LBB39_50
+; NO_SVE-NEXT:  // %bb.49: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v21.d[1]
+; NO_SVE-NEXT:    ld1 { v19.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_50: // %else62
+; NO_SVE-NEXT:    add v21.2d, v5.2d, v20.2d
+; NO_SVE-NEXT:    tbz w8, #22, .LBB39_52
+; NO_SVE-NEXT:  // %bb.51: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d21
+; NO_SVE-NEXT:    ld1 { v20.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v23.2d, v22.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB39_53
+; NO_SVE-NEXT:    b .LBB39_54
+; NO_SVE-NEXT:  .LBB39_52:
+; NO_SVE-NEXT:    // implicit-def: $q20
+; NO_SVE-NEXT:    sshll v23.2d, v22.2s, #3
+; NO_SVE-NEXT:    tbz w8, #23, .LBB39_54
+; NO_SVE-NEXT:  .LBB39_53: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v21.d[1]
+; NO_SVE-NEXT:    ld1 { v20.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_54: // %else68
+; NO_SVE-NEXT:    add v23.2d, v5.2d, v23.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB39_56
+; NO_SVE-NEXT:  // %bb.55: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d23
+; NO_SVE-NEXT:    ld1 { v21.d }[0], [x9]
+; NO_SVE-NEXT:    b .LBB39_57
+; NO_SVE-NEXT:  .LBB39_56:
+; NO_SVE-NEXT:    // implicit-def: $q21
+; NO_SVE-NEXT:  .LBB39_57: // %else71
+; NO_SVE-NEXT:    ldr q24, [x1, #112]
+; NO_SVE-NEXT:    sshll2 v22.2d, v22.4s, #3
+; NO_SVE-NEXT:    tbz w8, #25, .LBB39_59
+; NO_SVE-NEXT:  // %bb.58: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v23.d[1]
+; NO_SVE-NEXT:    ld1 { v21.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_59: // %else74
+; NO_SVE-NEXT:    add v23.2d, v5.2d, v22.2d
+; NO_SVE-NEXT:    tbz w8, #26, .LBB39_61
+; NO_SVE-NEXT:  // %bb.60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d23
+; NO_SVE-NEXT:    ld1 { v22.d }[0], [x9]
+; NO_SVE-NEXT:    sshll v25.2d, v24.2s, #3
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB39_62
+; NO_SVE-NEXT:    b .LBB39_63
+; NO_SVE-NEXT:  .LBB39_61:
+; NO_SVE-NEXT:    // implicit-def: $q22
+; NO_SVE-NEXT:    sshll v25.2d, v24.2s, #3
+; NO_SVE-NEXT:    tbz w8, #27, .LBB39_63
+; NO_SVE-NEXT:  .LBB39_62: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v23.d[1]
+; NO_SVE-NEXT:    ld1 { v22.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_63: // %else80
+; NO_SVE-NEXT:    add v25.2d, v5.2d, v25.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB39_65
+; NO_SVE-NEXT:  // %bb.64: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d25
+; NO_SVE-NEXT:    ld1 { v23.d }[0], [x9]
+; NO_SVE-NEXT:    sshll2 v24.2d, v24.4s, #3
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB39_66
+; NO_SVE-NEXT:    b .LBB39_67
+; NO_SVE-NEXT:  .LBB39_65:
+; NO_SVE-NEXT:    // implicit-def: $q23
+; NO_SVE-NEXT:    sshll2 v24.2d, v24.4s, #3
+; NO_SVE-NEXT:    tbz w8, #29, .LBB39_67
+; NO_SVE-NEXT:  .LBB39_66: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v25.d[1]
+; NO_SVE-NEXT:    ld1 { v23.d }[1], [x9]
+; NO_SVE-NEXT:  .LBB39_67: // %else86
+; NO_SVE-NEXT:    add v24.2d, v5.2d, v24.2d
+; NO_SVE-NEXT:    tbz w8, #30, .LBB39_69
+; NO_SVE-NEXT:  // %bb.68: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d24
+; NO_SVE-NEXT:    ld1 { v5.d }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB39_70
+; NO_SVE-NEXT:    b .LBB39_71
+; NO_SVE-NEXT:  .LBB39_69:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    tbz w8, #31, .LBB39_71
+; NO_SVE-NEXT:  .LBB39_70: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v24.d[1]
+; NO_SVE-NEXT:    ld1 { v5.d }[1], [x8]
+; NO_SVE-NEXT:  .LBB39_71: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    stp q17, q18, [x0, #128]
+; NO_SVE-NEXT:    stp q19, q20, [x0, #160]
+; NO_SVE-NEXT:    stp q21, q22, [x0, #192]
+; NO_SVE-NEXT:    stp q23, q5, [x0, #224]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_scaled_sext_f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    mov x13, #28
+; VBITS_GE_256-NEXT:    mov x14, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl8
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z18.s }, p1/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p1/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z16.s }, p1/z, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z19.s }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z0.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z5.d, #0.0
+; VBITS_GE_256-NEXT:    sunpklo z22.d, z18.s
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z18.d, z18.s
+; VBITS_GE_256-NEXT:    sunpklo z21.d, z17.s
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p2/z, [x2, z18.d, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z1.d, #0.0
+; VBITS_GE_256-NEXT:    ext z17.b, z17.b, z17.b, #16
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p2/z, [x2, z22.d, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z2.d, #0.0
+; VBITS_GE_256-NEXT:    sunpklo z20.d, z16.s
+; VBITS_GE_256-NEXT:    ext z16.b, z16.b, z16.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z17.d, z17.s
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p2/z, [x2, z21.d, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z3.d, #0.0
+; VBITS_GE_256-NEXT:    sunpklo z16.d, z16.s
+; VBITS_GE_256-NEXT:    sunpklo z23.d, z19.s
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p2/z, [x2, z17.d, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z4.d, #0.0
+; VBITS_GE_256-NEXT:    sunpklo z19.d, z19.s
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p2/z, [x2, z20.d, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x2, z16.d, lsl #3]
+; VBITS_GE_256-NEXT:    fcmeq p1.d, p0/z, z6.d, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p2.d, p0/z, z7.d, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p2/z, [x2, z23.d, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x2, z19.d, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_32b_scaled_sext_f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1sw { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.d, p0/z, z0.d, #0.0
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p1/z, [x2, z1.d, lsl #3]
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x double>, <32 x double>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = sext <32 x i32> %idxs to <32 x i64>
   %ptrs = getelementptr double, double* %base, <32 x i64> %ext
   %mask = fcmp oeq <32 x double> %cvals, zeroinitializer
   %vals = call <32 x double> @llvm.masked.gather.v32f64(<32 x double*> %ptrs, i32 8, <32 x i1> %mask, <32 x double> undef)
   store <32 x double> %vals, <32 x double>* %a
   ret void
 }
 
 define void @masked_gather_32b_scaled_zext(<32 x half>* %a, <32 x i32>* %b, half* %base) #0 {
-; VBITS_GE_2048-LABEL: masked_gather_32b_scaled_zext:
-; VBITS_GE_2048:       // %bb.0:
-; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
-; VBITS_GE_2048-NEXT:    ptrue p1.s, vl32
-; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
-; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p1/z, [x1]
-; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
-; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
-; VBITS_GE_2048-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, uxtw #1]
-; VBITS_GE_2048-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
-; VBITS_GE_2048-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_32b_scaled_zext:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    ldr q2, [x1]
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v1.b[0]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v3.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v3.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v3.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v3.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w12, v3.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v3.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v0.8h, v4.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v3.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v1.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    ushll v0.2d, v2.2s, #1
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB40_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    b .LBB40_3
+; NO_SVE-NEXT:  .LBB40_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:  .LBB40_3: // %else
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    ushll2 v2.2d, v2.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB40_13
+; NO_SVE-NEXT:  // %bb.4: // %else2
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB40_14
+; NO_SVE-NEXT:  .LBB40_5: // %else5
+; NO_SVE-NEXT:    ushll v4.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB40_15
+; NO_SVE-NEXT:  .LBB40_6: // %else8
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB40_8
+; NO_SVE-NEXT:  .LBB40_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB40_8: // %else11
+; NO_SVE-NEXT:    ldr q4, [x1, #32]
+; NO_SVE-NEXT:    ushll2 v3.2d, v3.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB40_16
+; NO_SVE-NEXT:  // %bb.9: // %else14
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB40_17
+; NO_SVE-NEXT:  .LBB40_10: // %else17
+; NO_SVE-NEXT:    ushll v3.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB40_18
+; NO_SVE-NEXT:  .LBB40_11: // %else20
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB40_19
+; NO_SVE-NEXT:  .LBB40_12: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB40_20
+; NO_SVE-NEXT:  .LBB40_13: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #2, .LBB40_5
+; NO_SVE-NEXT:  .LBB40_14: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    ushll v4.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbz w8, #3, .LBB40_6
+; NO_SVE-NEXT:  .LBB40_15: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB40_7
+; NO_SVE-NEXT:    b .LBB40_8
+; NO_SVE-NEXT:  .LBB40_16: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    add v2.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #6, .LBB40_10
+; NO_SVE-NEXT:  .LBB40_17: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    ushll v3.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbz w8, #7, .LBB40_11
+; NO_SVE-NEXT:  .LBB40_18: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB40_12
+; NO_SVE-NEXT:  .LBB40_19:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:  .LBB40_20: // %else23
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    ushll2 v4.2d, v4.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB40_30
+; NO_SVE-NEXT:  // %bb.21: // %else26
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB40_31
+; NO_SVE-NEXT:  .LBB40_22: // %else29
+; NO_SVE-NEXT:    ushll v5.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB40_32
+; NO_SVE-NEXT:  .LBB40_23: // %else32
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB40_25
+; NO_SVE-NEXT:  .LBB40_24: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB40_25: // %else35
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    ushll2 v3.2d, v3.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB40_33
+; NO_SVE-NEXT:  // %bb.26: // %else38
+; NO_SVE-NEXT:    add v3.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB40_34
+; NO_SVE-NEXT:  .LBB40_27: // %else41
+; NO_SVE-NEXT:    ushll v4.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB40_35
+; NO_SVE-NEXT:  .LBB40_28: // %else44
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB40_36
+; NO_SVE-NEXT:  .LBB40_29: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB40_37
+; NO_SVE-NEXT:  .LBB40_30: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #10, .LBB40_22
+; NO_SVE-NEXT:  .LBB40_31: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    ushll v5.2d, v3.2s, #1
+; NO_SVE-NEXT:    tbz w8, #11, .LBB40_23
+; NO_SVE-NEXT:  .LBB40_32: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB40_24
+; NO_SVE-NEXT:    b .LBB40_25
+; NO_SVE-NEXT:  .LBB40_33: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:    add v3.2d, v1.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #14, .LBB40_27
+; NO_SVE-NEXT:  .LBB40_34: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    ushll v4.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbz w8, #15, .LBB40_28
+; NO_SVE-NEXT:  .LBB40_35: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB40_29
+; NO_SVE-NEXT:  .LBB40_36:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:  .LBB40_37: // %else47
+; NO_SVE-NEXT:    ldr q4, [x1, #80]
+; NO_SVE-NEXT:    ushll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB40_47
+; NO_SVE-NEXT:  // %bb.38: // %else50
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB40_48
+; NO_SVE-NEXT:  .LBB40_39: // %else53
+; NO_SVE-NEXT:    ushll v6.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB40_49
+; NO_SVE-NEXT:  .LBB40_40: // %else56
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB40_42
+; NO_SVE-NEXT:  .LBB40_41: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:  .LBB40_42: // %else59
+; NO_SVE-NEXT:    ldr q6, [x1, #96]
+; NO_SVE-NEXT:    ushll2 v4.2d, v4.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB40_50
+; NO_SVE-NEXT:  // %bb.43: // %else62
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB40_51
+; NO_SVE-NEXT:  .LBB40_44: // %else65
+; NO_SVE-NEXT:    ushll v5.2d, v6.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB40_52
+; NO_SVE-NEXT:  .LBB40_45: // %else68
+; NO_SVE-NEXT:    add v7.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB40_53
+; NO_SVE-NEXT:  .LBB40_46: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v4.h }[0], [x9]
+; NO_SVE-NEXT:    b .LBB40_54
+; NO_SVE-NEXT:  .LBB40_47: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #18, .LBB40_39
+; NO_SVE-NEXT:  .LBB40_48: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    ushll v6.2d, v4.2s, #1
+; NO_SVE-NEXT:    tbz w8, #19, .LBB40_40
+; NO_SVE-NEXT:  .LBB40_49: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB40_41
+; NO_SVE-NEXT:    b .LBB40_42
+; NO_SVE-NEXT:  .LBB40_50: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:    add v4.2d, v1.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #22, .LBB40_44
+; NO_SVE-NEXT:  .LBB40_51: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    ushll v5.2d, v6.2s, #1
+; NO_SVE-NEXT:    tbz w8, #23, .LBB40_45
+; NO_SVE-NEXT:  .LBB40_52: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x9]
+; NO_SVE-NEXT:    add v7.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB40_46
+; NO_SVE-NEXT:  .LBB40_53:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:  .LBB40_54: // %else71
+; NO_SVE-NEXT:    ldr q5, [x1, #112]
+; NO_SVE-NEXT:    ushll2 v6.2d, v6.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB40_63
+; NO_SVE-NEXT:  // %bb.55: // %else74
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB40_64
+; NO_SVE-NEXT:  .LBB40_56: // %else77
+; NO_SVE-NEXT:    ushll v7.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB40_65
+; NO_SVE-NEXT:  .LBB40_57: // %else80
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB40_66
+; NO_SVE-NEXT:  .LBB40_58: // %else83
+; NO_SVE-NEXT:    ushll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB40_67
+; NO_SVE-NEXT:  .LBB40_59: // %else86
+; NO_SVE-NEXT:    add v1.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB40_68
+; NO_SVE-NEXT:  .LBB40_60: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB40_62
+; NO_SVE-NEXT:  .LBB40_61: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB40_62: // %else92
+; NO_SVE-NEXT:    stp q0, q2, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB40_63: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[1], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #26, .LBB40_56
+; NO_SVE-NEXT:  .LBB40_64: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[2], [x9]
+; NO_SVE-NEXT:    ushll v7.2d, v5.2s, #1
+; NO_SVE-NEXT:    tbz w8, #27, .LBB40_57
+; NO_SVE-NEXT:  .LBB40_65: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v1.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB40_58
+; NO_SVE-NEXT:  .LBB40_66: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[4], [x9]
+; NO_SVE-NEXT:    ushll2 v5.2d, v5.4s, #1
+; NO_SVE-NEXT:    tbz w8, #29, .LBB40_59
+; NO_SVE-NEXT:  .LBB40_67: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[5], [x9]
+; NO_SVE-NEXT:    add v1.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #30, .LBB40_60
+; NO_SVE-NEXT:  .LBB40_68: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v4.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB40_61
+; NO_SVE-NEXT:    b .LBB40_62
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_scaled_zext:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p1/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p1/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p1/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z0.h, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.h, p0/z, z1.h, #0.0
+; VBITS_GE_256-NEXT:    mov z0.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    mov z1.h, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    ld1h { z4.s }, p2/z, [x2, z4.s, uxtw #1]
+; VBITS_GE_256-NEXT:    cmpne p2.s, p1/z, z0.s, #0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1h { z0.s }, p2/z, [x2, z3.s, uxtw #1]
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.s, p1/z, z1.s, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.s }, p2/z, [x2, z5.s, uxtw #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.s }, p1/z, [x2, z2.s, uxtw #1]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z3.h, p1, z3.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    splice z1.h, p1, z1.h, z2.h
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: masked_gather_32b_scaled_zext:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_1024-NEXT:    ptrue p1.s, vl32
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p1/z, [x1]
+; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
+; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
+; VBITS_GE_1024-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, uxtw #1]
+; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %cvals = load <32 x half>, <32 x half>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = zext <32 x i32> %idxs to <32 x i64>
   %ptrs = getelementptr half, half* %base, <32 x i64> %ext
   %mask = fcmp oeq <32 x half> %cvals, zeroinitializer
   %vals = call <32 x half> @llvm.masked.gather.v32f16(<32 x half*> %ptrs, i32 8, <32 x i1> %mask, <32 x half> undef)
   store <32 x half> %vals, <32 x half>* %a
   ret void
 }
 
 define void @masked_gather_32b_unscaled_sext(<32 x half>* %a, <32 x i32>* %b, i8* %base) #0 {
-; VBITS_GE_2048-LABEL: masked_gather_32b_unscaled_sext:
-; VBITS_GE_2048:       // %bb.0:
-; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
-; VBITS_GE_2048-NEXT:    ptrue p1.s, vl32
-; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
-; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p1/z, [x1]
-; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
-; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
-; VBITS_GE_2048-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, sxtw]
-; VBITS_GE_2048-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
-; VBITS_GE_2048-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_32b_unscaled_sext:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    ldr q2, [x1]
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v1.b[0]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v3.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v3.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v3.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v3.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w12, v3.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v3.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v0.8h, v4.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v3.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v1.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    saddw v3.2d, v1.2d, v2.2s
+; NO_SVE-NEXT:    tbz w8, #0, .LBB41_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB41_3
+; NO_SVE-NEXT:    b .LBB41_4
+; NO_SVE-NEXT:  .LBB41_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB41_4
+; NO_SVE-NEXT:  .LBB41_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB41_4: // %else2
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    saddw2 v2.2d, v1.2d, v2.4s
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB41_13
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB41_14
+; NO_SVE-NEXT:  .LBB41_6: // %else8
+; NO_SVE-NEXT:    saddw v2.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB41_15
+; NO_SVE-NEXT:  .LBB41_7: // %else11
+; NO_SVE-NEXT:    tbz w8, #5, .LBB41_9
+; NO_SVE-NEXT:  .LBB41_8: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB41_9: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #32]
+; NO_SVE-NEXT:    saddw2 v2.2d, v1.2d, v3.4s
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB41_16
+; NO_SVE-NEXT:  // %bb.10: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB41_17
+; NO_SVE-NEXT:  .LBB41_11: // %else20
+; NO_SVE-NEXT:    saddw v3.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbz w8, #8, .LBB41_18
+; NO_SVE-NEXT:  .LBB41_12: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB41_19
+; NO_SVE-NEXT:    b .LBB41_20
+; NO_SVE-NEXT:  .LBB41_13: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB41_6
+; NO_SVE-NEXT:  .LBB41_14: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    saddw v2.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbz w8, #4, .LBB41_7
+; NO_SVE-NEXT:  .LBB41_15: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB41_8
+; NO_SVE-NEXT:    b .LBB41_9
+; NO_SVE-NEXT:  .LBB41_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB41_11
+; NO_SVE-NEXT:  .LBB41_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    saddw v3.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB41_12
+; NO_SVE-NEXT:  .LBB41_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB41_20
+; NO_SVE-NEXT:  .LBB41_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB41_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    saddw2 v4.2d, v1.2d, v4.4s
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB41_29
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB41_30
+; NO_SVE-NEXT:  .LBB41_22: // %else32
+; NO_SVE-NEXT:    saddw v4.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB41_31
+; NO_SVE-NEXT:  .LBB41_23: // %else35
+; NO_SVE-NEXT:    tbz w8, #13, .LBB41_25
+; NO_SVE-NEXT:  .LBB41_24: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB41_25: // %else38
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    saddw2 v3.2d, v1.2d, v3.4s
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB41_32
+; NO_SVE-NEXT:  // %bb.26: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB41_33
+; NO_SVE-NEXT:  .LBB41_27: // %else44
+; NO_SVE-NEXT:    saddw v4.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbz w8, #16, .LBB41_34
+; NO_SVE-NEXT:  .LBB41_28: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB41_35
+; NO_SVE-NEXT:    b .LBB41_36
+; NO_SVE-NEXT:  .LBB41_29: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB41_22
+; NO_SVE-NEXT:  .LBB41_30: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    saddw v4.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbz w8, #12, .LBB41_23
+; NO_SVE-NEXT:  .LBB41_31: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB41_24
+; NO_SVE-NEXT:    b .LBB41_25
+; NO_SVE-NEXT:  .LBB41_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB41_27
+; NO_SVE-NEXT:  .LBB41_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    saddw v4.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB41_28
+; NO_SVE-NEXT:  .LBB41_34:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #17, .LBB41_36
+; NO_SVE-NEXT:  .LBB41_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB41_36: // %else50
+; NO_SVE-NEXT:    ldr q4, [x1, #80]
+; NO_SVE-NEXT:    saddw2 v5.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB41_45
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB41_46
+; NO_SVE-NEXT:  .LBB41_38: // %else56
+; NO_SVE-NEXT:    saddw v5.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB41_47
+; NO_SVE-NEXT:  .LBB41_39: // %else59
+; NO_SVE-NEXT:    tbz w8, #21, .LBB41_41
+; NO_SVE-NEXT:  .LBB41_40: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB41_41: // %else62
+; NO_SVE-NEXT:    ldr q6, [x1, #96]
+; NO_SVE-NEXT:    saddw2 v4.2d, v1.2d, v4.4s
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB41_48
+; NO_SVE-NEXT:  // %bb.42: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB41_49
+; NO_SVE-NEXT:  .LBB41_43: // %else68
+; NO_SVE-NEXT:    saddw v5.2d, v1.2d, v6.2s
+; NO_SVE-NEXT:    tbz w8, #24, .LBB41_50
+; NO_SVE-NEXT:  .LBB41_44: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB41_51
+; NO_SVE-NEXT:    b .LBB41_52
+; NO_SVE-NEXT:  .LBB41_45: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB41_38
+; NO_SVE-NEXT:  .LBB41_46: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    saddw v5.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbz w8, #20, .LBB41_39
+; NO_SVE-NEXT:  .LBB41_47: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB41_40
+; NO_SVE-NEXT:    b .LBB41_41
+; NO_SVE-NEXT:  .LBB41_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB41_43
+; NO_SVE-NEXT:  .LBB41_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x9]
+; NO_SVE-NEXT:    saddw v5.2d, v1.2d, v6.2s
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB41_44
+; NO_SVE-NEXT:  .LBB41_50:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #25, .LBB41_52
+; NO_SVE-NEXT:  .LBB41_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB41_52: // %else74
+; NO_SVE-NEXT:    ldr q5, [x1, #112]
+; NO_SVE-NEXT:    saddw2 v6.2d, v1.2d, v6.4s
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB41_60
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB41_61
+; NO_SVE-NEXT:  .LBB41_54: // %else80
+; NO_SVE-NEXT:    saddw v6.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB41_62
+; NO_SVE-NEXT:  .LBB41_55: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB41_63
+; NO_SVE-NEXT:  .LBB41_56: // %else86
+; NO_SVE-NEXT:    saddw2 v1.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB41_64
+; NO_SVE-NEXT:  .LBB41_57: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB41_59
+; NO_SVE-NEXT:  .LBB41_58: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB41_59: // %else92
+; NO_SVE-NEXT:    stp q0, q2, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB41_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB41_54
+; NO_SVE-NEXT:  .LBB41_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[3], [x9]
+; NO_SVE-NEXT:    saddw v6.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbz w8, #28, .LBB41_55
+; NO_SVE-NEXT:  .LBB41_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB41_56
+; NO_SVE-NEXT:  .LBB41_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[5], [x9]
+; NO_SVE-NEXT:    saddw2 v1.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbz w8, #30, .LBB41_57
+; NO_SVE-NEXT:  .LBB41_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v4.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB41_58
+; NO_SVE-NEXT:    b .LBB41_59
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_unscaled_sext:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p1/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p1/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p1/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z0.h, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.h, p0/z, z1.h, #0.0
+; VBITS_GE_256-NEXT:    mov z0.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    mov z1.h, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    ld1h { z4.s }, p2/z, [x2, z4.s, sxtw]
+; VBITS_GE_256-NEXT:    cmpne p2.s, p1/z, z0.s, #0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1h { z0.s }, p2/z, [x2, z3.s, sxtw]
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.s, p1/z, z1.s, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.s }, p2/z, [x2, z5.s, sxtw]
+; VBITS_GE_256-NEXT:    ld1h { z2.s }, p1/z, [x2, z2.s, sxtw]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z3.h, p1, z3.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    splice z1.h, p1, z1.h, z2.h
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: masked_gather_32b_unscaled_sext:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_1024-NEXT:    ptrue p1.s, vl32
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p1/z, [x1]
+; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
+; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
+; VBITS_GE_1024-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, sxtw]
+; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %cvals = load <32 x half>, <32 x half>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = sext <32 x i32> %idxs to <32 x i64>
   %byte_ptrs = getelementptr i8, i8* %base, <32 x i64> %ext
   %ptrs = bitcast <32 x i8*> %byte_ptrs to <32 x half*>
   %mask = fcmp oeq <32 x half> %cvals, zeroinitializer
   %vals = call <32 x half> @llvm.masked.gather.v32f16(<32 x half*> %ptrs, i32 8, <32 x i1> %mask, <32 x half> undef)
   store <32 x half> %vals, <32 x half>* %a
   ret void
 }
 
 define void @masked_gather_32b_unscaled_zext(<32 x half>* %a, <32 x i32>* %b, i8* %base) #0 {
-; VBITS_GE_2048-LABEL: masked_gather_32b_unscaled_zext:
-; VBITS_GE_2048:       // %bb.0:
-; VBITS_GE_2048-NEXT:    ptrue p0.h, vl32
-; VBITS_GE_2048-NEXT:    ptrue p1.s, vl32
-; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
-; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p1/z, [x1]
-; VBITS_GE_2048-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
-; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
-; VBITS_GE_2048-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, uxtw]
-; VBITS_GE_2048-NEXT:    uzp1 z0.h, z0.h, z0.h
-; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
-; VBITS_GE_2048-NEXT:    ret
+; NO_SVE-LABEL: masked_gather_32b_unscaled_zext:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    fcmeq v0.8h, v0.8h, #0.0
+; NO_SVE-NEXT:    fcmeq v1.8h, v1.8h, #0.0
+; NO_SVE-NEXT:    ldr q2, [x1]
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v1.b[0]
+; NO_SVE-NEXT:    fcmeq v3.8h, v3.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v1.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v3.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v3.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v3.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v3.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w12, v3.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v3.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v0.8h, v4.8h, #0.0
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v3.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v1.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v1.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    uaddw v3.2d, v1.2d, v2.2s
+; NO_SVE-NEXT:    tbz w8, #0, .LBB42_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ldr h0, [x9]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB42_3
+; NO_SVE-NEXT:    b .LBB42_4
+; NO_SVE-NEXT:  .LBB42_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    tbz w8, #1, .LBB42_4
+; NO_SVE-NEXT:  .LBB42_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB42_4: // %else2
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    uaddw2 v2.2d, v1.2d, v2.4s
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB42_13
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB42_14
+; NO_SVE-NEXT:  .LBB42_6: // %else8
+; NO_SVE-NEXT:    uaddw v2.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB42_15
+; NO_SVE-NEXT:  .LBB42_7: // %else11
+; NO_SVE-NEXT:    tbz w8, #5, .LBB42_9
+; NO_SVE-NEXT:  .LBB42_8: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB42_9: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #32]
+; NO_SVE-NEXT:    uaddw2 v2.2d, v1.2d, v3.4s
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB42_16
+; NO_SVE-NEXT:  // %bb.10: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB42_17
+; NO_SVE-NEXT:  .LBB42_11: // %else20
+; NO_SVE-NEXT:    uaddw v3.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbz w8, #8, .LBB42_18
+; NO_SVE-NEXT:  .LBB42_12: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB42_19
+; NO_SVE-NEXT:    b .LBB42_20
+; NO_SVE-NEXT:  .LBB42_13: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB42_6
+; NO_SVE-NEXT:  .LBB42_14: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    uaddw v2.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbz w8, #4, .LBB42_7
+; NO_SVE-NEXT:  .LBB42_15: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB42_8
+; NO_SVE-NEXT:    b .LBB42_9
+; NO_SVE-NEXT:  .LBB42_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB42_11
+; NO_SVE-NEXT:  .LBB42_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v0.h }[7], [x9]
+; NO_SVE-NEXT:    uaddw v3.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB42_12
+; NO_SVE-NEXT:  .LBB42_18:
+; NO_SVE-NEXT:    // implicit-def: $q2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB42_20
+; NO_SVE-NEXT:  .LBB42_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB42_20: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    uaddw2 v4.2d, v1.2d, v4.4s
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB42_29
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB42_30
+; NO_SVE-NEXT:  .LBB42_22: // %else32
+; NO_SVE-NEXT:    uaddw v4.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB42_31
+; NO_SVE-NEXT:  .LBB42_23: // %else35
+; NO_SVE-NEXT:    tbz w8, #13, .LBB42_25
+; NO_SVE-NEXT:  .LBB42_24: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB42_25: // %else38
+; NO_SVE-NEXT:    ldr q5, [x1, #64]
+; NO_SVE-NEXT:    uaddw2 v3.2d, v1.2d, v3.4s
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB42_32
+; NO_SVE-NEXT:  // %bb.26: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB42_33
+; NO_SVE-NEXT:  .LBB42_27: // %else44
+; NO_SVE-NEXT:    uaddw v4.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbz w8, #16, .LBB42_34
+; NO_SVE-NEXT:  .LBB42_28: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB42_35
+; NO_SVE-NEXT:    b .LBB42_36
+; NO_SVE-NEXT:  .LBB42_29: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB42_22
+; NO_SVE-NEXT:  .LBB42_30: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    uaddw v4.2d, v1.2d, v3.2s
+; NO_SVE-NEXT:    tbz w8, #12, .LBB42_23
+; NO_SVE-NEXT:  .LBB42_31: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB42_24
+; NO_SVE-NEXT:    b .LBB42_25
+; NO_SVE-NEXT:  .LBB42_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB42_27
+; NO_SVE-NEXT:  .LBB42_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    uaddw v4.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB42_28
+; NO_SVE-NEXT:  .LBB42_34:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    tbz w8, #17, .LBB42_36
+; NO_SVE-NEXT:  .LBB42_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB42_36: // %else50
+; NO_SVE-NEXT:    ldr q4, [x1, #80]
+; NO_SVE-NEXT:    uaddw2 v5.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB42_45
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB42_46
+; NO_SVE-NEXT:  .LBB42_38: // %else56
+; NO_SVE-NEXT:    uaddw v5.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB42_47
+; NO_SVE-NEXT:  .LBB42_39: // %else59
+; NO_SVE-NEXT:    tbz w8, #21, .LBB42_41
+; NO_SVE-NEXT:  .LBB42_40: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[5], [x9]
+; NO_SVE-NEXT:  .LBB42_41: // %else62
+; NO_SVE-NEXT:    ldr q6, [x1, #96]
+; NO_SVE-NEXT:    uaddw2 v4.2d, v1.2d, v4.4s
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB42_48
+; NO_SVE-NEXT:  // %bb.42: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB42_49
+; NO_SVE-NEXT:  .LBB42_43: // %else68
+; NO_SVE-NEXT:    uaddw v5.2d, v1.2d, v6.2s
+; NO_SVE-NEXT:    tbz w8, #24, .LBB42_50
+; NO_SVE-NEXT:  .LBB42_44: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.h }[0], [x9]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB42_51
+; NO_SVE-NEXT:    b .LBB42_52
+; NO_SVE-NEXT:  .LBB42_45: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB42_38
+; NO_SVE-NEXT:  .LBB42_46: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[3], [x9]
+; NO_SVE-NEXT:    uaddw v5.2d, v1.2d, v4.2s
+; NO_SVE-NEXT:    tbz w8, #20, .LBB42_39
+; NO_SVE-NEXT:  .LBB42_47: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v3.h }[4], [x9]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB42_40
+; NO_SVE-NEXT:    b .LBB42_41
+; NO_SVE-NEXT:  .LBB42_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB42_43
+; NO_SVE-NEXT:  .LBB42_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.h }[7], [x9]
+; NO_SVE-NEXT:    uaddw v5.2d, v1.2d, v6.2s
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB42_44
+; NO_SVE-NEXT:  .LBB42_50:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    tbz w8, #25, .LBB42_52
+; NO_SVE-NEXT:  .LBB42_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[1], [x9]
+; NO_SVE-NEXT:  .LBB42_52: // %else74
+; NO_SVE-NEXT:    ldr q5, [x1, #112]
+; NO_SVE-NEXT:    uaddw2 v6.2d, v1.2d, v6.4s
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB42_60
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB42_61
+; NO_SVE-NEXT:  .LBB42_54: // %else80
+; NO_SVE-NEXT:    uaddw v6.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB42_62
+; NO_SVE-NEXT:  .LBB42_55: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB42_63
+; NO_SVE-NEXT:  .LBB42_56: // %else86
+; NO_SVE-NEXT:    uaddw2 v1.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB42_64
+; NO_SVE-NEXT:  .LBB42_57: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB42_59
+; NO_SVE-NEXT:  .LBB42_58: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[7], [x8]
+; NO_SVE-NEXT:  .LBB42_59: // %else92
+; NO_SVE-NEXT:    stp q0, q2, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB42_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB42_54
+; NO_SVE-NEXT:  .LBB42_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[3], [x9]
+; NO_SVE-NEXT:    uaddw v6.2d, v1.2d, v5.2s
+; NO_SVE-NEXT:    tbz w8, #28, .LBB42_55
+; NO_SVE-NEXT:  .LBB42_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v4.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB42_56
+; NO_SVE-NEXT:  .LBB42_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v4.h }[5], [x9]
+; NO_SVE-NEXT:    uaddw2 v1.2d, v1.2d, v5.4s
+; NO_SVE-NEXT:    tbz w8, #30, .LBB42_57
+; NO_SVE-NEXT:  .LBB42_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v4.h }[6], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB42_58
+; NO_SVE-NEXT:    b .LBB42_59
+;
+; VBITS_GE_256-LABEL: masked_gather_32b_unscaled_zext:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #24
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p1/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p1/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p1/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p2.h, p0/z, z0.h, #0.0
+; VBITS_GE_256-NEXT:    fcmeq p3.h, p0/z, z1.h, #0.0
+; VBITS_GE_256-NEXT:    mov z0.h, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    ext z0.b, z0.b, z0.b, #16
+; VBITS_GE_256-NEXT:    mov z1.h, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z0.s, z0.h
+; VBITS_GE_256-NEXT:    ld1h { z4.s }, p2/z, [x2, z4.s, uxtw]
+; VBITS_GE_256-NEXT:    cmpne p2.s, p1/z, z0.s, #0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    ld1h { z0.s }, p2/z, [x2, z3.s, uxtw]
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.s, z1.h
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.s, p1/z, z1.s, #0
+; VBITS_GE_256-NEXT:    ld1h { z1.s }, p2/z, [x2, z5.s, uxtw]
+; VBITS_GE_256-NEXT:    ld1h { z2.s }, p1/z, [x2, z2.s, uxtw]
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z4.h, z4.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z3.h, p1, z3.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    splice z1.h, p1, z1.h, z2.h
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: masked_gather_32b_unscaled_zext:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_1024-NEXT:    ptrue p1.s, vl32
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p1/z, [x1]
+; VBITS_GE_1024-NEXT:    fcmeq p1.h, p0/z, z0.h, #0.0
+; VBITS_GE_1024-NEXT:    punpklo p1.h, p1.b
+; VBITS_GE_1024-NEXT:    ld1h { z0.s }, p1/z, [x2, z1.s, uxtw]
+; VBITS_GE_1024-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %cvals = load <32 x half>, <32 x half>* %a
   %idxs = load <32 x i32>, <32 x i32>* %b
   %ext = zext <32 x i32> %idxs to <32 x i64>
   %byte_ptrs = getelementptr i8, i8* %base, <32 x i64> %ext
   %ptrs = bitcast <32 x i8*> %byte_ptrs to <32 x half*>
   %mask = fcmp oeq <32 x half> %cvals, zeroinitializer
   %vals = call <32 x half> @llvm.masked.gather.v32f16(<32 x half*> %ptrs, i32 8, <32 x i1> %mask, <32 x half> undef)
   store <32 x half> %vals, <32 x half>* %a
   ret void
 }
 
 define void @masked_gather_64b_scaled(<32 x float>* %a, <32 x i64>* %b, float* %base) #0 {
+; NO_SVE-LABEL: masked_gather_64b_scaled:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v4.8h
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.4s, v6.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #48]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldr q0, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v5.8h, v5.8h, v6.8h
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v5.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w13, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v3.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    fcmeq v2.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[6]
+; NO_SVE-NEXT:    umov w14, v5.b[5]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v5.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v2.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    shl v0.2d, v0.2d, #2
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v3.2d, v2.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB43_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    shl v1.2d, v1.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB43_3
+; NO_SVE-NEXT:    b .LBB43_4
+; NO_SVE-NEXT:  .LBB43_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    shl v1.2d, v1.2d, #2
+; NO_SVE-NEXT:    tbz w8, #1, .LBB43_4
+; NO_SVE-NEXT:  .LBB43_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_4: // %else2
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    add v1.2d, v2.2d, v1.2d
+; NO_SVE-NEXT:    tbz w8, #2, .LBB43_6
+; NO_SVE-NEXT:  // %bb.5: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_6: // %else5
+; NO_SVE-NEXT:    shl v4.2d, v3.2d, #2
+; NO_SVE-NEXT:    tbz w8, #3, .LBB43_8
+; NO_SVE-NEXT:  // %bb.7: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_8: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB43_10
+; NO_SVE-NEXT:  // %bb.9: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    shl v3.2d, v3.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB43_11
+; NO_SVE-NEXT:    b .LBB43_12
+; NO_SVE-NEXT:  .LBB43_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    shl v3.2d, v3.2d, #2
+; NO_SVE-NEXT:    tbz w8, #5, .LBB43_12
+; NO_SVE-NEXT:  .LBB43_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_12: // %else14
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    add v3.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #6, .LBB43_14
+; NO_SVE-NEXT:  // %bb.13: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_14: // %else17
+; NO_SVE-NEXT:    shl v4.2d, v4.2d, #2
+; NO_SVE-NEXT:    tbz w8, #7, .LBB43_16
+; NO_SVE-NEXT:  // %bb.15: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_16: // %else20
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB43_18
+; NO_SVE-NEXT:  // %bb.17: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    shl v6.2d, v5.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB43_19
+; NO_SVE-NEXT:    b .LBB43_20
+; NO_SVE-NEXT:  .LBB43_18:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    shl v6.2d, v5.2d, #2
+; NO_SVE-NEXT:    tbz w8, #9, .LBB43_20
+; NO_SVE-NEXT:  .LBB43_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_20: // %else26
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #10, .LBB43_22
+; NO_SVE-NEXT:  // %bb.21: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_22: // %else29
+; NO_SVE-NEXT:    shl v5.2d, v5.2d, #2
+; NO_SVE-NEXT:    tbz w8, #11, .LBB43_24
+; NO_SVE-NEXT:  // %bb.23: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_24: // %else32
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB43_26
+; NO_SVE-NEXT:  // %bb.25: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    shl v7.2d, v6.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB43_27
+; NO_SVE-NEXT:    b .LBB43_28
+; NO_SVE-NEXT:  .LBB43_26:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    shl v7.2d, v6.2d, #2
+; NO_SVE-NEXT:    tbz w8, #13, .LBB43_28
+; NO_SVE-NEXT:  .LBB43_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_28: // %else38
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #14, .LBB43_30
+; NO_SVE-NEXT:  // %bb.29: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_30: // %else41
+; NO_SVE-NEXT:    shl v6.2d, v6.2d, #2
+; NO_SVE-NEXT:    tbz w8, #15, .LBB43_32
+; NO_SVE-NEXT:  // %bb.31: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_32: // %else44
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB43_34
+; NO_SVE-NEXT:  // %bb.33: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    shl v16.2d, v7.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB43_35
+; NO_SVE-NEXT:    b .LBB43_36
+; NO_SVE-NEXT:  .LBB43_34:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    shl v16.2d, v7.2d, #2
+; NO_SVE-NEXT:    tbz w8, #17, .LBB43_36
+; NO_SVE-NEXT:  .LBB43_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_36: // %else50
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #18, .LBB43_38
+; NO_SVE-NEXT:  // %bb.37: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_38: // %else53
+; NO_SVE-NEXT:    shl v7.2d, v7.2d, #2
+; NO_SVE-NEXT:    tbz w8, #19, .LBB43_40
+; NO_SVE-NEXT:  // %bb.39: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_40: // %else56
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB43_42
+; NO_SVE-NEXT:  // %bb.41: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    shl v17.2d, v16.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB43_43
+; NO_SVE-NEXT:    b .LBB43_44
+; NO_SVE-NEXT:  .LBB43_42:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    shl v17.2d, v16.2d, #2
+; NO_SVE-NEXT:    tbz w8, #21, .LBB43_44
+; NO_SVE-NEXT:  .LBB43_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_44: // %else62
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbz w8, #22, .LBB43_46
+; NO_SVE-NEXT:  // %bb.45: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_46: // %else65
+; NO_SVE-NEXT:    shl v16.2d, v16.2d, #2
+; NO_SVE-NEXT:    tbz w8, #23, .LBB43_48
+; NO_SVE-NEXT:  // %bb.47: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_48: // %else68
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB43_50
+; NO_SVE-NEXT:  // %bb.49: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    shl v18.2d, v17.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB43_51
+; NO_SVE-NEXT:    b .LBB43_52
+; NO_SVE-NEXT:  .LBB43_50:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    shl v18.2d, v17.2d, #2
+; NO_SVE-NEXT:    tbz w8, #25, .LBB43_52
+; NO_SVE-NEXT:  .LBB43_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_52: // %else74
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v18.2d
+; NO_SVE-NEXT:    tbz w8, #26, .LBB43_54
+; NO_SVE-NEXT:  // %bb.53: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:  .LBB43_54: // %else77
+; NO_SVE-NEXT:    shl v17.2d, v17.2d, #2
+; NO_SVE-NEXT:    tbz w8, #27, .LBB43_56
+; NO_SVE-NEXT:  // %bb.55: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB43_56: // %else80
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    add v17.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB43_58
+; NO_SVE-NEXT:  // %bb.57: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.s }[0], [x9]
+; NO_SVE-NEXT:    shl v18.2d, v18.2d, #2
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB43_59
+; NO_SVE-NEXT:    b .LBB43_60
+; NO_SVE-NEXT:  .LBB43_58:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    shl v18.2d, v18.2d, #2
+; NO_SVE-NEXT:    tbz w8, #29, .LBB43_60
+; NO_SVE-NEXT:  .LBB43_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB43_60: // %else86
+; NO_SVE-NEXT:    add v2.2d, v2.2d, v18.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB43_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB43_63
+; NO_SVE-NEXT:  .LBB43_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB43_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB43_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB43_62
+; NO_SVE-NEXT:    b .LBB43_63
+;
+; VBITS_GE_256-LABEL: masked_gather_64b_scaled:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [x2, z18.d, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [x2, z17.d, lsl #2]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [x2, z16.d, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [x2, z6.d, lsl #2]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [x2, z7.d, lsl #2]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [x2, z5.d, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [x2, z0.d, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [x2, z4.d, lsl #2]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_64b_scaled:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_2048-NEXT:    ld1w { z0.d }, p1/z, [x2, z1.d, lsl #2]
 ; VBITS_GE_2048-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %idxs = load <32 x i64>, <32 x i64>* %b
   %ptrs = getelementptr float, float* %base, <32 x i64> %idxs
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> undef)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_64b_unscaled(<32 x float>* %a, <32 x i64>* %b, i8* %base) #0 {
+; NO_SVE-LABEL: masked_gather_64b_unscaled:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v4.8h
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.4s, v6.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #48]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldr q0, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v5.8h, v5.8h, v6.8h
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v5.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w13, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v3.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    fcmeq v2.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[6]
+; NO_SVE-NEXT:    umov w14, v5.b[5]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v5.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v2.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v1.2d, v2.2d, v0.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB44_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB44_3
+; NO_SVE-NEXT:    b .LBB44_4
+; NO_SVE-NEXT:  .LBB44_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB44_4
+; NO_SVE-NEXT:  .LBB44_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_4: // %else2
+; NO_SVE-NEXT:    add v1.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB44_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB44_9
+; NO_SVE-NEXT:  .LBB44_6: // %else8
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB44_10
+; NO_SVE-NEXT:  .LBB44_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB44_11
+; NO_SVE-NEXT:    b .LBB44_12
+; NO_SVE-NEXT:  .LBB44_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB44_6
+; NO_SVE-NEXT:  .LBB44_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB44_7
+; NO_SVE-NEXT:  .LBB44_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB44_12
+; NO_SVE-NEXT:  .LBB44_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_12: // %else14
+; NO_SVE-NEXT:    add v3.2d, v2.2d, v3.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB44_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB44_17
+; NO_SVE-NEXT:  .LBB44_14: // %else20
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB44_18
+; NO_SVE-NEXT:  .LBB44_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB44_19
+; NO_SVE-NEXT:    b .LBB44_20
+; NO_SVE-NEXT:  .LBB44_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB44_14
+; NO_SVE-NEXT:  .LBB44_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v4.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB44_15
+; NO_SVE-NEXT:  .LBB44_18:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB44_20
+; NO_SVE-NEXT:  .LBB44_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_20: // %else26
+; NO_SVE-NEXT:    add v4.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB44_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB44_25
+; NO_SVE-NEXT:  .LBB44_22: // %else32
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB44_26
+; NO_SVE-NEXT:  .LBB44_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB44_27
+; NO_SVE-NEXT:    b .LBB44_28
+; NO_SVE-NEXT:  .LBB44_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB44_22
+; NO_SVE-NEXT:  .LBB44_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v5.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB44_23
+; NO_SVE-NEXT:  .LBB44_26:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB44_28
+; NO_SVE-NEXT:  .LBB44_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_28: // %else38
+; NO_SVE-NEXT:    add v5.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB44_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB44_33
+; NO_SVE-NEXT:  .LBB44_30: // %else44
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB44_34
+; NO_SVE-NEXT:  .LBB44_31: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB44_35
+; NO_SVE-NEXT:    b .LBB44_36
+; NO_SVE-NEXT:  .LBB44_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB44_30
+; NO_SVE-NEXT:  .LBB44_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v6.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB44_31
+; NO_SVE-NEXT:  .LBB44_34:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB44_36
+; NO_SVE-NEXT:  .LBB44_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_36: // %else50
+; NO_SVE-NEXT:    add v6.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB44_40
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB44_41
+; NO_SVE-NEXT:  .LBB44_38: // %else56
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB44_42
+; NO_SVE-NEXT:  .LBB44_39: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB44_43
+; NO_SVE-NEXT:    b .LBB44_44
+; NO_SVE-NEXT:  .LBB44_40: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB44_38
+; NO_SVE-NEXT:  .LBB44_41: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB44_39
+; NO_SVE-NEXT:  .LBB44_42:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB44_44
+; NO_SVE-NEXT:  .LBB44_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_44: // %else62
+; NO_SVE-NEXT:    add v7.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB44_48
+; NO_SVE-NEXT:  // %bb.45: // %else65
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB44_49
+; NO_SVE-NEXT:  .LBB44_46: // %else68
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB44_50
+; NO_SVE-NEXT:  .LBB44_47: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB44_51
+; NO_SVE-NEXT:    b .LBB44_52
+; NO_SVE-NEXT:  .LBB44_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB44_46
+; NO_SVE-NEXT:  .LBB44_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v16.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB44_47
+; NO_SVE-NEXT:  .LBB44_50:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB44_52
+; NO_SVE-NEXT:  .LBB44_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_52: // %else74
+; NO_SVE-NEXT:    add v16.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB44_56
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB44_57
+; NO_SVE-NEXT:  .LBB44_54: // %else80
+; NO_SVE-NEXT:    add v17.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB44_58
+; NO_SVE-NEXT:  .LBB44_55: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB44_59
+; NO_SVE-NEXT:    b .LBB44_60
+; NO_SVE-NEXT:  .LBB44_56: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB44_54
+; NO_SVE-NEXT:  .LBB44_57: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    add v17.2d, v2.2d, v17.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB44_55
+; NO_SVE-NEXT:  .LBB44_58:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB44_60
+; NO_SVE-NEXT:  .LBB44_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB44_60: // %else86
+; NO_SVE-NEXT:    add v2.2d, v2.2d, v18.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB44_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB44_63
+; NO_SVE-NEXT:  .LBB44_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB44_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB44_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB44_62
+; NO_SVE-NEXT:    b .LBB44_63
+;
+; VBITS_GE_256-LABEL: masked_gather_64b_unscaled:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [x2, z18.d]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [x2, z17.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [x2, z16.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [x2, z6.d]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [x2, z7.d]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [x2, z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [x2, z0.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [x2, z4.d]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_64b_unscaled:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_2048-NEXT:    ld1w { z0.d }, p1/z, [x2, z1.d]
 ; VBITS_GE_2048-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %idxs = load <32 x i64>, <32 x i64>* %b
   %byte_ptrs = getelementptr i8, i8* %base, <32 x i64> %idxs
   %ptrs = bitcast <32 x i8*> %byte_ptrs to <32 x float*>
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> undef)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_vec_plus_reg(<32 x float>* %a, <32 x i8*>* %b, i64 %off) #0 {
+; NO_SVE-LABEL: masked_gather_vec_plus_reg:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v4.8h
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.4s, v6.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #48]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldr q0, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v5.8h, v5.8h, v6.8h
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v5.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w13, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v3.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    fcmeq v2.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v1.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[6]
+; NO_SVE-NEXT:    umov w14, v5.b[5]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v1.b[7]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v5.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v1.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v1.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v1.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    dup v2.2d, x2
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    add v1.2d, v0.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB45_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB45_3
+; NO_SVE-NEXT:    b .LBB45_4
+; NO_SVE-NEXT:  .LBB45_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB45_4
+; NO_SVE-NEXT:  .LBB45_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_4: // %else2
+; NO_SVE-NEXT:    add v1.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB45_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB45_9
+; NO_SVE-NEXT:  .LBB45_6: // %else8
+; NO_SVE-NEXT:    add v4.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB45_10
+; NO_SVE-NEXT:  .LBB45_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB45_11
+; NO_SVE-NEXT:    b .LBB45_12
+; NO_SVE-NEXT:  .LBB45_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB45_6
+; NO_SVE-NEXT:  .LBB45_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB45_7
+; NO_SVE-NEXT:  .LBB45_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB45_12
+; NO_SVE-NEXT:  .LBB45_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_12: // %else14
+; NO_SVE-NEXT:    add v3.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB45_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB45_17
+; NO_SVE-NEXT:  .LBB45_14: // %else20
+; NO_SVE-NEXT:    add v4.2d, v4.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB45_18
+; NO_SVE-NEXT:  .LBB45_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB45_19
+; NO_SVE-NEXT:    b .LBB45_20
+; NO_SVE-NEXT:  .LBB45_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB45_14
+; NO_SVE-NEXT:  .LBB45_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v4.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB45_15
+; NO_SVE-NEXT:  .LBB45_18:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB45_20
+; NO_SVE-NEXT:  .LBB45_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_20: // %else26
+; NO_SVE-NEXT:    add v4.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB45_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB45_25
+; NO_SVE-NEXT:  .LBB45_22: // %else32
+; NO_SVE-NEXT:    add v5.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB45_26
+; NO_SVE-NEXT:  .LBB45_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB45_27
+; NO_SVE-NEXT:    b .LBB45_28
+; NO_SVE-NEXT:  .LBB45_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB45_22
+; NO_SVE-NEXT:  .LBB45_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB45_23
+; NO_SVE-NEXT:  .LBB45_26:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB45_28
+; NO_SVE-NEXT:  .LBB45_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_28: // %else38
+; NO_SVE-NEXT:    add v5.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB45_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB45_33
+; NO_SVE-NEXT:  .LBB45_30: // %else44
+; NO_SVE-NEXT:    add v6.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB45_34
+; NO_SVE-NEXT:  .LBB45_31: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB45_35
+; NO_SVE-NEXT:    b .LBB45_36
+; NO_SVE-NEXT:  .LBB45_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB45_30
+; NO_SVE-NEXT:  .LBB45_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB45_31
+; NO_SVE-NEXT:  .LBB45_34:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB45_36
+; NO_SVE-NEXT:  .LBB45_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_36: // %else50
+; NO_SVE-NEXT:    add v6.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB45_40
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB45_41
+; NO_SVE-NEXT:  .LBB45_38: // %else56
+; NO_SVE-NEXT:    add v7.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB45_42
+; NO_SVE-NEXT:  .LBB45_39: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB45_43
+; NO_SVE-NEXT:    b .LBB45_44
+; NO_SVE-NEXT:  .LBB45_40: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB45_38
+; NO_SVE-NEXT:  .LBB45_41: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    add v7.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB45_39
+; NO_SVE-NEXT:  .LBB45_42:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB45_44
+; NO_SVE-NEXT:  .LBB45_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_44: // %else62
+; NO_SVE-NEXT:    add v7.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB45_48
+; NO_SVE-NEXT:  // %bb.45: // %else65
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB45_49
+; NO_SVE-NEXT:  .LBB45_46: // %else68
+; NO_SVE-NEXT:    add v16.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB45_50
+; NO_SVE-NEXT:  .LBB45_47: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB45_51
+; NO_SVE-NEXT:    b .LBB45_52
+; NO_SVE-NEXT:  .LBB45_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB45_46
+; NO_SVE-NEXT:  .LBB45_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    add v16.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB45_47
+; NO_SVE-NEXT:  .LBB45_50:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB45_52
+; NO_SVE-NEXT:  .LBB45_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_52: // %else74
+; NO_SVE-NEXT:    add v16.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB45_56
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB45_57
+; NO_SVE-NEXT:  .LBB45_54: // %else80
+; NO_SVE-NEXT:    add v17.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB45_58
+; NO_SVE-NEXT:  .LBB45_55: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB45_59
+; NO_SVE-NEXT:    b .LBB45_60
+; NO_SVE-NEXT:  .LBB45_56: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB45_54
+; NO_SVE-NEXT:  .LBB45_57: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    add v17.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB45_55
+; NO_SVE-NEXT:  .LBB45_58:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB45_60
+; NO_SVE-NEXT:  .LBB45_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB45_60: // %else86
+; NO_SVE-NEXT:    add v2.2d, v18.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB45_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB45_63
+; NO_SVE-NEXT:  .LBB45_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB45_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB45_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB45_62
+; NO_SVE-NEXT:    b .LBB45_63
+;
+; VBITS_GE_256-LABEL: masked_gather_vec_plus_reg:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [x2, z18.d]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [x2, z17.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [x2, z16.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [x2, z6.d]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [x2, z7.d]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [x2, z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [x2, z0.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [x2, z4.d]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_vec_plus_reg:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_2048-NEXT:    ld1w { z0.d }, p1/z, [x2, z1.d]
 ; VBITS_GE_2048-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %bases = load <32 x i8*>, <32 x i8*>* %b
   %byte_ptrs = getelementptr i8, <32 x i8*> %bases, i64 %off
   %ptrs = bitcast <32 x i8*> %byte_ptrs to <32 x float*>
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> undef)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_vec_plus_imm(<32 x float>* %a, <32 x i8*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_vec_plus_imm:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q3, [x0, #64]
+; NO_SVE-NEXT:    mov w9, #4
+; NO_SVE-NEXT:    fcmeq v6.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v6.8h, v3.8h
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldp q0, q7, [x0, #32]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w11, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v4.8h, v4.8h, v5.8h
+; NO_SVE-NEXT:    umov w10, v2.b[0]
+; NO_SVE-NEXT:    umov w12, v2.b[3]
+; NO_SVE-NEXT:    umov w13, v2.b[4]
+; NO_SVE-NEXT:    umov w14, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    xtn v3.8b, v4.8h
+; NO_SVE-NEXT:    umov w15, v2.b[6]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w16, v2.b[7]
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w8, v3.b[0]
+; NO_SVE-NEXT:    bfi w10, w11, #2, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w11, v3.b[1]
+; NO_SVE-NEXT:    bfi w10, w12, #3, #1
+; NO_SVE-NEXT:    umov w12, v3.b[2]
+; NO_SVE-NEXT:    bfi w10, w13, #4, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w13, v3.b[3]
+; NO_SVE-NEXT:    bfi w10, w14, #5, #1
+; NO_SVE-NEXT:    and w16, w16, #0x1
+; NO_SVE-NEXT:    xtn v1.8b, v1.8h
+; NO_SVE-NEXT:    umov w14, v3.b[4]
+; NO_SVE-NEXT:    orr w10, w10, w15, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w16, lsl #7
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w16, v1.b[1]
+; NO_SVE-NEXT:    orr w8, w10, w8, lsl #8
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #9
+; NO_SVE-NEXT:    umov w11, v1.b[2]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    fcmeq v6.4s, v7.4s, #0.0
+; NO_SVE-NEXT:    umov w10, v1.b[0]
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #10
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v1.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #11
+; NO_SVE-NEXT:    and w13, w16, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #12
+; NO_SVE-NEXT:    umov w14, v1.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w16, v1.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v6.8h
+; NO_SVE-NEXT:    bfi w10, w13, #1, #1
+; NO_SVE-NEXT:    bfi w10, w11, #2, #1
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    umov w15, v3.b[5]
+; NO_SVE-NEXT:    and w13, w16, #0x1
+; NO_SVE-NEXT:    bfi w10, w12, #3, #1
+; NO_SVE-NEXT:    umov w12, v1.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    bfi w10, w11, #4, #1
+; NO_SVE-NEXT:    umov w11, v3.b[6]
+; NO_SVE-NEXT:    bfi w10, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[0]
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    umov w16, v0.b[1]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w13, lsl #6
+; NO_SVE-NEXT:    umov w13, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w15, lsl #13
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[3]
+; NO_SVE-NEXT:    orr w10, w10, w12, lsl #7
+; NO_SVE-NEXT:    and w12, w14, #0x1
+; NO_SVE-NEXT:    and w14, w16, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    umov w11, v0.b[4]
+; NO_SVE-NEXT:    orr w10, w10, w12, lsl #8
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    orr w10, w10, w14, lsl #9
+; NO_SVE-NEXT:    and w14, w15, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[6]
+; NO_SVE-NEXT:    orr w10, w10, w12, lsl #10
+; NO_SVE-NEXT:    umov w12, v3.b[7]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w14, lsl #11
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[7]
+; NO_SVE-NEXT:    orr w10, w10, w11, lsl #12
+; NO_SVE-NEXT:    and w11, w15, #0x1
+; NO_SVE-NEXT:    orr w12, w8, w12, lsl #15
+; NO_SVE-NEXT:    orr w8, w10, w13, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    ldr q0, [x1]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #15
+; NO_SVE-NEXT:    dup v2.2d, x9
+; NO_SVE-NEXT:    bfi w8, w12, #16, #16
+; NO_SVE-NEXT:    add v1.2d, v0.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #0, .LBB46_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ldr s0, [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB46_3
+; NO_SVE-NEXT:    b .LBB46_4
+; NO_SVE-NEXT:  .LBB46_2:
+; NO_SVE-NEXT:    // implicit-def: $q0
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB46_4
+; NO_SVE-NEXT:  .LBB46_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_4: // %else2
+; NO_SVE-NEXT:    add v1.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB46_8
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB46_9
+; NO_SVE-NEXT:  .LBB46_6: // %else8
+; NO_SVE-NEXT:    add v4.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #4, .LBB46_10
+; NO_SVE-NEXT:  .LBB46_7: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB46_11
+; NO_SVE-NEXT:    b .LBB46_12
+; NO_SVE-NEXT:  .LBB46_8: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB46_6
+; NO_SVE-NEXT:  .LBB46_9: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB46_7
+; NO_SVE-NEXT:  .LBB46_10:
+; NO_SVE-NEXT:    // implicit-def: $q1
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB46_12
+; NO_SVE-NEXT:  .LBB46_11: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_12: // %else14
+; NO_SVE-NEXT:    add v3.2d, v3.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB46_16
+; NO_SVE-NEXT:  // %bb.13: // %else17
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB46_17
+; NO_SVE-NEXT:  .LBB46_14: // %else20
+; NO_SVE-NEXT:    add v4.2d, v4.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #8, .LBB46_18
+; NO_SVE-NEXT:  .LBB46_15: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB46_19
+; NO_SVE-NEXT:    b .LBB46_20
+; NO_SVE-NEXT:  .LBB46_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB46_14
+; NO_SVE-NEXT:  .LBB46_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    add v4.2d, v4.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB46_15
+; NO_SVE-NEXT:  .LBB46_18:
+; NO_SVE-NEXT:    // implicit-def: $q3
+; NO_SVE-NEXT:    ldr q5, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB46_20
+; NO_SVE-NEXT:  .LBB46_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_20: // %else26
+; NO_SVE-NEXT:    add v4.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB46_24
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB46_25
+; NO_SVE-NEXT:  .LBB46_22: // %else32
+; NO_SVE-NEXT:    add v5.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #12, .LBB46_26
+; NO_SVE-NEXT:  .LBB46_23: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB46_27
+; NO_SVE-NEXT:    b .LBB46_28
+; NO_SVE-NEXT:  .LBB46_24: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB46_22
+; NO_SVE-NEXT:  .LBB46_25: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    add v5.2d, v5.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB46_23
+; NO_SVE-NEXT:  .LBB46_26:
+; NO_SVE-NEXT:    // implicit-def: $q4
+; NO_SVE-NEXT:    ldr q6, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB46_28
+; NO_SVE-NEXT:  .LBB46_27: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_28: // %else38
+; NO_SVE-NEXT:    add v5.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB46_32
+; NO_SVE-NEXT:  // %bb.29: // %else41
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB46_33
+; NO_SVE-NEXT:  .LBB46_30: // %else44
+; NO_SVE-NEXT:    add v6.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #16, .LBB46_34
+; NO_SVE-NEXT:  .LBB46_31: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB46_35
+; NO_SVE-NEXT:    b .LBB46_36
+; NO_SVE-NEXT:  .LBB46_32: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB46_30
+; NO_SVE-NEXT:  .LBB46_33: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    add v6.2d, v6.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB46_31
+; NO_SVE-NEXT:  .LBB46_34:
+; NO_SVE-NEXT:    // implicit-def: $q5
+; NO_SVE-NEXT:    ldr q7, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB46_36
+; NO_SVE-NEXT:  .LBB46_35: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_36: // %else50
+; NO_SVE-NEXT:    add v6.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB46_40
+; NO_SVE-NEXT:  // %bb.37: // %else53
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB46_41
+; NO_SVE-NEXT:  .LBB46_38: // %else56
+; NO_SVE-NEXT:    add v7.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #20, .LBB46_42
+; NO_SVE-NEXT:  .LBB46_39: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB46_43
+; NO_SVE-NEXT:    b .LBB46_44
+; NO_SVE-NEXT:  .LBB46_40: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB46_38
+; NO_SVE-NEXT:  .LBB46_41: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    add v7.2d, v7.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB46_39
+; NO_SVE-NEXT:  .LBB46_42:
+; NO_SVE-NEXT:    // implicit-def: $q6
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB46_44
+; NO_SVE-NEXT:  .LBB46_43: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_44: // %else62
+; NO_SVE-NEXT:    add v7.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB46_48
+; NO_SVE-NEXT:  // %bb.45: // %else65
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB46_49
+; NO_SVE-NEXT:  .LBB46_46: // %else68
+; NO_SVE-NEXT:    add v16.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #24, .LBB46_50
+; NO_SVE-NEXT:  .LBB46_47: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB46_51
+; NO_SVE-NEXT:    b .LBB46_52
+; NO_SVE-NEXT:  .LBB46_48: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB46_46
+; NO_SVE-NEXT:  .LBB46_49: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    add v16.2d, v16.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB46_47
+; NO_SVE-NEXT:  .LBB46_50:
+; NO_SVE-NEXT:    // implicit-def: $q7
+; NO_SVE-NEXT:    ldr q17, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB46_52
+; NO_SVE-NEXT:  .LBB46_51: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_52: // %else74
+; NO_SVE-NEXT:    add v16.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB46_56
+; NO_SVE-NEXT:  // %bb.53: // %else77
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB46_57
+; NO_SVE-NEXT:  .LBB46_54: // %else80
+; NO_SVE-NEXT:    add v17.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbz w8, #28, .LBB46_58
+; NO_SVE-NEXT:  .LBB46_55: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d17
+; NO_SVE-NEXT:    ld1 { v16.s }[0], [x9]
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB46_59
+; NO_SVE-NEXT:    b .LBB46_60
+; NO_SVE-NEXT:  .LBB46_56: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    ldr q17, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB46_54
+; NO_SVE-NEXT:  .LBB46_57: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    add v17.2d, v17.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB46_55
+; NO_SVE-NEXT:  .LBB46_58:
+; NO_SVE-NEXT:    // implicit-def: $q16
+; NO_SVE-NEXT:    ldr q18, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB46_60
+; NO_SVE-NEXT:  .LBB46_59: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v17.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB46_60: // %else86
+; NO_SVE-NEXT:    add v2.2d, v18.2d, v2.2d
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB46_64
+; NO_SVE-NEXT:  // %bb.61: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB46_63
+; NO_SVE-NEXT:  .LBB46_62: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v16.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB46_63: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    stp q5, q6, [x0, #64]
+; NO_SVE-NEXT:    stp q7, q16, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB46_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB46_62
+; NO_SVE-NEXT:    b .LBB46_63
+;
+; VBITS_GE_256-LABEL: masked_gather_vec_plus_imm:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z18.d, #4]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [z17.d, #4]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [z16.d, #4]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [z6.d, #4]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [z7.d, #4]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [z5.d, #4]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [z0.d, #4]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [z4.d, #4]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_vec_plus_imm:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b
 ; VBITS_GE_2048-NEXT:    ld1w { z0.d }, p1/z, [z1.d, #4]
 ; VBITS_GE_2048-NEXT:    uzp1 z0.s, z0.s, z0.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %bases = load <32 x i8*>, <32 x i8*>* %b
   %byte_ptrs = getelementptr i8, <32 x i8*> %bases, i64 4
   %ptrs = bitcast <32 x i8*> %byte_ptrs to <32 x float*>
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> undef)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_passthru(<32 x float>* %a, <32 x float*>* %b, <32 x float>* %c) #0 {
+; NO_SVE-LABEL: masked_gather_passthru:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldp q0, q2, [x0]
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v3.8h, v3.8h, v4.8h
+; NO_SVE-NEXT:    ldp q5, q6, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v2.8h
+; NO_SVE-NEXT:    fcmeq v6.4s, v6.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #48]
+; NO_SVE-NEXT:    xtn v2.8b, v3.8h
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    uzp1 v5.8h, v5.8h, v6.8h
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v5.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w13, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v3.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    ldr q4, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    fcmeq v2.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w13, v0.b[4]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[6]
+; NO_SVE-NEXT:    umov w14, v5.b[5]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v2.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v5.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    ldr q0, [x2]
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbnz w8, #0, .LBB47_41
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB47_42
+; NO_SVE-NEXT:  .LBB47_2: // %else2
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB47_43
+; NO_SVE-NEXT:  .LBB47_3: // %else5
+; NO_SVE-NEXT:    tbz w8, #3, .LBB47_5
+; NO_SVE-NEXT:  .LBB47_4: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_5: // %else8
+; NO_SVE-NEXT:    ldr q2, [x1, #32]
+; NO_SVE-NEXT:    ldr q1, [x2, #16]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB47_44
+; NO_SVE-NEXT:  // %bb.6: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB47_45
+; NO_SVE-NEXT:  .LBB47_7: // %else14
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB47_46
+; NO_SVE-NEXT:  .LBB47_8: // %else17
+; NO_SVE-NEXT:    tbz w8, #7, .LBB47_10
+; NO_SVE-NEXT:  .LBB47_9: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_10: // %else20
+; NO_SVE-NEXT:    ldr q3, [x1, #64]
+; NO_SVE-NEXT:    ldr q2, [x2, #32]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB47_47
+; NO_SVE-NEXT:  // %bb.11: // %else23
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB47_48
+; NO_SVE-NEXT:  .LBB47_12: // %else26
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB47_49
+; NO_SVE-NEXT:  .LBB47_13: // %else29
+; NO_SVE-NEXT:    tbz w8, #11, .LBB47_15
+; NO_SVE-NEXT:  .LBB47_14: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_15: // %else32
+; NO_SVE-NEXT:    ldr q4, [x1, #96]
+; NO_SVE-NEXT:    ldr q3, [x2, #48]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB47_50
+; NO_SVE-NEXT:  // %bb.16: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB47_51
+; NO_SVE-NEXT:  .LBB47_17: // %else38
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB47_52
+; NO_SVE-NEXT:  .LBB47_18: // %else41
+; NO_SVE-NEXT:    tbz w8, #15, .LBB47_20
+; NO_SVE-NEXT:  .LBB47_19: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_20: // %else44
+; NO_SVE-NEXT:    ldr q5, [x1, #128]
+; NO_SVE-NEXT:    ldr q4, [x2, #64]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB47_53
+; NO_SVE-NEXT:  // %bb.21: // %else47
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB47_54
+; NO_SVE-NEXT:  .LBB47_22: // %else50
+; NO_SVE-NEXT:    ldr q5, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB47_55
+; NO_SVE-NEXT:  .LBB47_23: // %else53
+; NO_SVE-NEXT:    tbz w8, #19, .LBB47_25
+; NO_SVE-NEXT:  .LBB47_24: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_25: // %else56
+; NO_SVE-NEXT:    ldr q6, [x1, #160]
+; NO_SVE-NEXT:    ldr q5, [x2, #80]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB47_56
+; NO_SVE-NEXT:  // %bb.26: // %else59
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB47_57
+; NO_SVE-NEXT:  .LBB47_27: // %else62
+; NO_SVE-NEXT:    ldr q6, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB47_58
+; NO_SVE-NEXT:  .LBB47_28: // %else65
+; NO_SVE-NEXT:    tbz w8, #23, .LBB47_30
+; NO_SVE-NEXT:  .LBB47_29: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_30: // %else68
+; NO_SVE-NEXT:    ldr q7, [x1, #192]
+; NO_SVE-NEXT:    ldr q6, [x2, #96]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB47_59
+; NO_SVE-NEXT:  // %bb.31: // %else71
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB47_60
+; NO_SVE-NEXT:  .LBB47_32: // %else74
+; NO_SVE-NEXT:    ldr q7, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB47_61
+; NO_SVE-NEXT:  .LBB47_33: // %else77
+; NO_SVE-NEXT:    tbz w8, #27, .LBB47_35
+; NO_SVE-NEXT:  .LBB47_34: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:  .LBB47_35: // %else80
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    ldr q7, [x2, #112]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB47_62
+; NO_SVE-NEXT:  // %bb.36: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB47_63
+; NO_SVE-NEXT:  .LBB47_37: // %else86
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB47_64
+; NO_SVE-NEXT:  .LBB47_38: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB47_40
+; NO_SVE-NEXT:  .LBB47_39: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB47_40: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB47_41: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB47_2
+; NO_SVE-NEXT:  .LBB47_42: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v1.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB47_3
+; NO_SVE-NEXT:  .LBB47_43: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d1
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB47_4
+; NO_SVE-NEXT:    b .LBB47_5
+; NO_SVE-NEXT:  .LBB47_44: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB47_7
+; NO_SVE-NEXT:  .LBB47_45: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v2.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q2, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB47_8
+; NO_SVE-NEXT:  .LBB47_46: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d2
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB47_9
+; NO_SVE-NEXT:    b .LBB47_10
+; NO_SVE-NEXT:  .LBB47_47: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB47_12
+; NO_SVE-NEXT:  .LBB47_48: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #80]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB47_13
+; NO_SVE-NEXT:  .LBB47_49: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB47_14
+; NO_SVE-NEXT:    b .LBB47_15
+; NO_SVE-NEXT:  .LBB47_50: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB47_17
+; NO_SVE-NEXT:  .LBB47_51: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v4.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q4, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB47_18
+; NO_SVE-NEXT:  .LBB47_52: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d4
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB47_19
+; NO_SVE-NEXT:    b .LBB47_20
+; NO_SVE-NEXT:  .LBB47_53: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB47_22
+; NO_SVE-NEXT:  .LBB47_54: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v5.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q5, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB47_23
+; NO_SVE-NEXT:  .LBB47_55: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d5
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB47_24
+; NO_SVE-NEXT:    b .LBB47_25
+; NO_SVE-NEXT:  .LBB47_56: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB47_27
+; NO_SVE-NEXT:  .LBB47_57: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v6.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q6, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB47_28
+; NO_SVE-NEXT:  .LBB47_58: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d6
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB47_29
+; NO_SVE-NEXT:    b .LBB47_30
+; NO_SVE-NEXT:  .LBB47_59: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB47_32
+; NO_SVE-NEXT:  .LBB47_60: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v7.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q7, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB47_33
+; NO_SVE-NEXT:  .LBB47_61: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d7
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB47_34
+; NO_SVE-NEXT:    b .LBB47_35
+; NO_SVE-NEXT:  .LBB47_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB47_37
+; NO_SVE-NEXT:  .LBB47_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB47_38
+; NO_SVE-NEXT:  .LBB47_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB47_39
+; NO_SVE-NEXT:    b .LBB47_40
+;
+; VBITS_GE_256-LABEL: masked_gather_passthru:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z20.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z23.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z4.s, #0.0
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    punpklo p3.h, p2.b
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    ld1d { z19.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z21.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z22.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x2, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x2, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z16.s }, p0/z, [x2, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x2]
+; VBITS_GE_256-NEXT:    ld1w { z4.d }, p3/z, [z23.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z17.s, #0.0
+; VBITS_GE_256-NEXT:    mov z17.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z4.s, z4.s
+; VBITS_GE_256-NEXT:    bif v4.16b, v16.16b, v17.16b
+; VBITS_GE_256-NEXT:    ext z17.b, z17.b, z17.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z23.d, z17.s
+; VBITS_GE_256-NEXT:    ext z16.b, z16.b, z16.b, #16
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z23.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z22.d }, p4/z, [z22.d]
+; VBITS_GE_256-NEXT:    ld1w { z21.d }, p2/z, [z21.d]
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z20.s, #0.0
+; VBITS_GE_256-NEXT:    mov z20.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p2.b
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    uzp1 z21.s, z21.s, z21.s
+; VBITS_GE_256-NEXT:    uzp1 z22.s, z22.s, z22.s
+; VBITS_GE_256-NEXT:    bif v21.16b, v5.16b, v20.16b
+; VBITS_GE_256-NEXT:    ext z20.b, z20.b, z20.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z23.d, z20.s
+; VBITS_GE_256-NEXT:    ext z5.b, z5.b, z5.b, #16
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z23.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z19.d }, p4/z, [z19.d]
+; VBITS_GE_256-NEXT:    ld1w { z18.d }, p3/z, [z18.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z7.s, #0.0
+; VBITS_GE_256-NEXT:    mov z7.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p2.h, p3.b
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    uzp1 z18.s, z18.s, z18.s
+; VBITS_GE_256-NEXT:    bif v18.16b, v1.16b, v7.16b
+; VBITS_GE_256-NEXT:    ext z7.b, z7.b, z7.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z23.d, z7.s
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z23.d, #0
+; VBITS_GE_256-NEXT:    mov z23.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p4/z, [z2.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p2/z, [z3.d]
+; VBITS_GE_256-NEXT:    bit v16.16b, v22.16b, v17.16b
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    bif v3.16b, v6.16b, v23.16b
+; VBITS_GE_256-NEXT:    ext z23.b, z23.b, z23.b, #16
+; VBITS_GE_256-NEXT:    sunpklo z17.d, z23.s
+; VBITS_GE_256-NEXT:    ext z6.b, z6.b, z6.b, #16
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z17.d, #0
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z19.s, z19.s
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p1/z, [z0.d]
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    bit v5.16b, v17.16b, v20.16b
+; VBITS_GE_256-NEXT:    splice z4.s, p1, z4.s, z16.s
+; VBITS_GE_256-NEXT:    bit v1.16b, v2.16b, v7.16b
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    bif v0.16b, v6.16b, v23.16b
+; VBITS_GE_256-NEXT:    splice z21.s, p1, z21.s, z5.s
+; VBITS_GE_256-NEXT:    splice z18.s, p1, z18.s, z1.s
+; VBITS_GE_256-NEXT:    st1w { z21.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z18.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z0.s
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_passthru:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x2]
 ; VBITS_GE_2048-NEXT:    punpklo p2.h, p1.b
 ; VBITS_GE_2048-NEXT:    ld1w { z1.d }, p2/z, [z1.d]
 ; VBITS_GE_2048-NEXT:    uzp1 z1.s, z1.s, z1.s
 ; VBITS_GE_2048-NEXT:    mov z0.s, p1/m, z1.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %cvals = load <32 x float>, <32 x float>* %a
   %ptrs = load <32 x float*>, <32 x float*>* %b
   %passthru = load <32 x float>, <32 x float>* %c
   %mask = fcmp oeq <32 x float> %cvals, zeroinitializer
   %vals = call <32 x float> @llvm.masked.gather.v32f32(<32 x float*> %ptrs, i32 8, <32 x i1> %mask, <32 x float> %passthru)
   store <32 x float> %vals, <32 x float>* %a
   ret void
 }
 
 define void @masked_gather_passthru_0(<32 x float>* %a, <32 x float*>* %b) #0 {
+; NO_SVE-LABEL: masked_gather_passthru_0:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q3, [x0, #64]
+; NO_SVE-NEXT:    fcmeq v2.4s, v2.4s, #0.0
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    fcmeq v3.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v0.4s, v0.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    ldp q4, q5, [x0, #96]
+; NO_SVE-NEXT:    fcmeq v1.4s, v1.4s, #0.0
+; NO_SVE-NEXT:    fcmeq v4.4s, v4.4s, #0.0
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    fcmeq v5.4s, v5.4s, #0.0
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    xtn v1.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v1.b[1]
+; NO_SVE-NEXT:    umov w10, v1.b[2]
+; NO_SVE-NEXT:    uzp1 v4.8h, v4.8h, v5.8h
+; NO_SVE-NEXT:    umov w9, v1.b[0]
+; NO_SVE-NEXT:    umov w11, v1.b[3]
+; NO_SVE-NEXT:    umov w12, v1.b[4]
+; NO_SVE-NEXT:    umov w13, v1.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    xtn v4.8b, v4.8h
+; NO_SVE-NEXT:    umov w14, v1.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v1.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w8, v4.b[0]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w10, v4.b[1]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v4.b[2]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w12, v4.b[3]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w13, v4.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    fcmeq v2.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    ldr q3, [x0, #32]
+; NO_SVE-NEXT:    orr w9, w9, w15, lsl #7
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #9
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    fcmeq v1.4s, v3.4s, #0.0
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #11
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #12
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    umov w13, v0.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    uzp1 v1.8h, v1.8h, v2.8h
+; NO_SVE-NEXT:    bfi w9, w15, #1, #1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w14, v4.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    xtn v0.8b, v1.8h
+; NO_SVE-NEXT:    bfi w9, w10, #4, #1
+; NO_SVE-NEXT:    umov w10, v4.b[6]
+; NO_SVE-NEXT:    bfi w9, w12, #5, #1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[0]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v0.b[1]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #6
+; NO_SVE-NEXT:    umov w12, v0.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #13
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v0.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v0.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v0.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v4.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v0.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    ldr q3, [x1]
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    movi v0.2d, #0000000000000000
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbz w8, #0, .LBB48_2
+; NO_SVE-NEXT:  // %bb.1: // %cond.load
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    movi v1.2d, #0000000000000000
+; NO_SVE-NEXT:    ld1 { v0.s }[0], [x9]
+; NO_SVE-NEXT:    movi v2.2d, #0000000000000000
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB48_3
+; NO_SVE-NEXT:    b .LBB48_4
+; NO_SVE-NEXT:  .LBB48_2:
+; NO_SVE-NEXT:    movi v1.2d, #0000000000000000
+; NO_SVE-NEXT:    movi v2.2d, #0000000000000000
+; NO_SVE-NEXT:    tbz w8, #1, .LBB48_4
+; NO_SVE-NEXT:  .LBB48_3: // %cond.load1
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB48_4: // %else2
+; NO_SVE-NEXT:    ldr q3, [x1, #16]
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB48_12
+; NO_SVE-NEXT:  // %bb.5: // %else5
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB48_13
+; NO_SVE-NEXT:  .LBB48_6: // %else8
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB48_14
+; NO_SVE-NEXT:  .LBB48_7: // %else11
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB48_15
+; NO_SVE-NEXT:  .LBB48_8: // %else14
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB48_16
+; NO_SVE-NEXT:  .LBB48_9: // %else17
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB48_17
+; NO_SVE-NEXT:  .LBB48_10: // %else20
+; NO_SVE-NEXT:    ldr q16, [x1, #64]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB48_18
+; NO_SVE-NEXT:  .LBB48_11: // %cond.load22
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    mov v17.16b, v2.16b
+; NO_SVE-NEXT:    mov v3.16b, v2.16b
+; NO_SVE-NEXT:    ld1 { v17.s }[0], [x9]
+; NO_SVE-NEXT:    mov v4.16b, v2.16b
+; NO_SVE-NEXT:    mov v5.16b, v2.16b
+; NO_SVE-NEXT:    mov v6.16b, v2.16b
+; NO_SVE-NEXT:    mov v7.16b, v2.16b
+; NO_SVE-NEXT:    mov v2.16b, v17.16b
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB48_19
+; NO_SVE-NEXT:    b .LBB48_20
+; NO_SVE-NEXT:  .LBB48_12: // %cond.load4
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB48_6
+; NO_SVE-NEXT:  .LBB48_13: // %cond.load7
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v0.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #32]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB48_7
+; NO_SVE-NEXT:  .LBB48_14: // %cond.load10
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB48_8
+; NO_SVE-NEXT:  .LBB48_15: // %cond.load13
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q3, [x1, #48]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB48_9
+; NO_SVE-NEXT:  .LBB48_16: // %cond.load16
+; NO_SVE-NEXT:    fmov x9, d3
+; NO_SVE-NEXT:    ld1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB48_10
+; NO_SVE-NEXT:  .LBB48_17: // %cond.load19
+; NO_SVE-NEXT:    mov x9, v3.d[1]
+; NO_SVE-NEXT:    ld1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #64]
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB48_11
+; NO_SVE-NEXT:  .LBB48_18:
+; NO_SVE-NEXT:    mov v3.16b, v2.16b
+; NO_SVE-NEXT:    mov v4.16b, v2.16b
+; NO_SVE-NEXT:    mov v5.16b, v2.16b
+; NO_SVE-NEXT:    mov v6.16b, v2.16b
+; NO_SVE-NEXT:    mov v7.16b, v2.16b
+; NO_SVE-NEXT:    tbz w8, #9, .LBB48_20
+; NO_SVE-NEXT:  .LBB48_19: // %cond.load25
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:  .LBB48_20: // %else26
+; NO_SVE-NEXT:    ldr q16, [x1, #80]
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB48_44
+; NO_SVE-NEXT:  // %bb.21: // %else29
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB48_45
+; NO_SVE-NEXT:  .LBB48_22: // %else32
+; NO_SVE-NEXT:    ldr q16, [x1, #96]
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB48_46
+; NO_SVE-NEXT:  .LBB48_23: // %else35
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB48_47
+; NO_SVE-NEXT:  .LBB48_24: // %else38
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB48_48
+; NO_SVE-NEXT:  .LBB48_25: // %else41
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB48_49
+; NO_SVE-NEXT:  .LBB48_26: // %else44
+; NO_SVE-NEXT:    ldr q16, [x1, #128]
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB48_50
+; NO_SVE-NEXT:  .LBB48_27: // %else47
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB48_51
+; NO_SVE-NEXT:  .LBB48_28: // %else50
+; NO_SVE-NEXT:    ldr q16, [x1, #144]
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB48_52
+; NO_SVE-NEXT:  .LBB48_29: // %else53
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB48_53
+; NO_SVE-NEXT:  .LBB48_30: // %else56
+; NO_SVE-NEXT:    ldr q16, [x1, #160]
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB48_54
+; NO_SVE-NEXT:  .LBB48_31: // %else59
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB48_55
+; NO_SVE-NEXT:  .LBB48_32: // %else62
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB48_56
+; NO_SVE-NEXT:  .LBB48_33: // %else65
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB48_57
+; NO_SVE-NEXT:  .LBB48_34: // %else68
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB48_58
+; NO_SVE-NEXT:  .LBB48_35: // %else71
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB48_59
+; NO_SVE-NEXT:  .LBB48_36: // %else74
+; NO_SVE-NEXT:    ldr q16, [x1, #208]
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB48_60
+; NO_SVE-NEXT:  .LBB48_37: // %else77
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB48_61
+; NO_SVE-NEXT:  .LBB48_38: // %else80
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB48_62
+; NO_SVE-NEXT:  .LBB48_39: // %else83
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB48_63
+; NO_SVE-NEXT:  .LBB48_40: // %else86
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB48_64
+; NO_SVE-NEXT:  .LBB48_41: // %else89
+; NO_SVE-NEXT:    tbz w8, #31, .LBB48_43
+; NO_SVE-NEXT:  .LBB48_42: // %cond.load91
+; NO_SVE-NEXT:    mov x8, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB48_43: // %else92
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    stp q2, q3, [x0, #32]
+; NO_SVE-NEXT:    stp q4, q5, [x0, #64]
+; NO_SVE-NEXT:    stp q6, q7, [x0, #96]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB48_44: // %cond.load28
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB48_22
+; NO_SVE-NEXT:  .LBB48_45: // %cond.load31
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #96]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB48_23
+; NO_SVE-NEXT:  .LBB48_46: // %cond.load34
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v3.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB48_24
+; NO_SVE-NEXT:  .LBB48_47: // %cond.load37
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #112]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB48_25
+; NO_SVE-NEXT:  .LBB48_48: // %cond.load40
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB48_26
+; NO_SVE-NEXT:  .LBB48_49: // %cond.load43
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #128]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB48_27
+; NO_SVE-NEXT:  .LBB48_50: // %cond.load46
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v4.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB48_28
+; NO_SVE-NEXT:  .LBB48_51: // %cond.load49
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #144]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB48_29
+; NO_SVE-NEXT:  .LBB48_52: // %cond.load52
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB48_30
+; NO_SVE-NEXT:  .LBB48_53: // %cond.load55
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #160]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB48_31
+; NO_SVE-NEXT:  .LBB48_54: // %cond.load58
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v5.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB48_32
+; NO_SVE-NEXT:  .LBB48_55: // %cond.load61
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #176]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB48_33
+; NO_SVE-NEXT:  .LBB48_56: // %cond.load64
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB48_34
+; NO_SVE-NEXT:  .LBB48_57: // %cond.load67
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #192]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB48_35
+; NO_SVE-NEXT:  .LBB48_58: // %cond.load70
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v6.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB48_36
+; NO_SVE-NEXT:  .LBB48_59: // %cond.load73
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #208]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB48_37
+; NO_SVE-NEXT:  .LBB48_60: // %cond.load76
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB48_38
+; NO_SVE-NEXT:  .LBB48_61: // %cond.load79
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #224]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB48_39
+; NO_SVE-NEXT:  .LBB48_62: // %cond.load82
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB48_40
+; NO_SVE-NEXT:  .LBB48_63: // %cond.load85
+; NO_SVE-NEXT:    mov x9, v16.d[1]
+; NO_SVE-NEXT:    ld1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:    ldr q16, [x1, #240]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB48_41
+; NO_SVE-NEXT:  .LBB48_64: // %cond.load88
+; NO_SVE-NEXT:    fmov x9, d16
+; NO_SVE-NEXT:    ld1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB48_42
+; NO_SVE-NEXT:    b .LBB48_43
+;
+; VBITS_GE_256-LABEL: masked_gather_passthru_0:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x14, #28
+; VBITS_GE_256-NEXT:    ptrue p1.d, vl4
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p1/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z18.d }, p1/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #20
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z0.s, #0.0
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov z19.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z2.s, #0.0
+; VBITS_GE_256-NEXT:    ext z19.b, z19.b, z19.b, #16
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z2.d, z19.s
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p3.d, p1/z, z2.d, #0
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p1/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p1/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p1/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p1/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p1/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p1/z, [x1]
+; VBITS_GE_256-NEXT:    ld1w { z2.d }, p2/z, [z18.d]
+; VBITS_GE_256-NEXT:    ld1w { z17.d }, p3/z, [z17.d]
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z1.s, #0.0
+; VBITS_GE_256-NEXT:    mov z1.s, p3/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z3.s, #0.0
+; VBITS_GE_256-NEXT:    ext z1.b, z1.b, z1.b, #16
+; VBITS_GE_256-NEXT:    mov z18.s, p4/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p2/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    punpklo p3.h, p3.b
+; VBITS_GE_256-NEXT:    sunpklo z1.d, z1.s
+; VBITS_GE_256-NEXT:    and p3.b, p3/z, p3.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p5.d, p1/z, z1.d, #0
+; VBITS_GE_256-NEXT:    punpklo p4.h, p4.b
+; VBITS_GE_256-NEXT:    ext z18.b, z18.b, z18.b, #16
+; VBITS_GE_256-NEXT:    ext z3.b, z3.b, z3.b, #16
+; VBITS_GE_256-NEXT:    ld1w { z16.d }, p3/z, [z16.d]
+; VBITS_GE_256-NEXT:    ld1w { z1.d }, p5/z, [z6.d]
+; VBITS_GE_256-NEXT:    and p4.b, p4/z, p4.b, p1.b
+; VBITS_GE_256-NEXT:    sunpklo z6.d, z18.s
+; VBITS_GE_256-NEXT:    punpklo p2.h, p2.b
+; VBITS_GE_256-NEXT:    sunpklo z3.d, z3.s
+; VBITS_GE_256-NEXT:    ld1w { z7.d }, p4/z, [z7.d]
+; VBITS_GE_256-NEXT:    cmpne p4.d, p1/z, z6.d, #0
+; VBITS_GE_256-NEXT:    and p2.b, p2/z, p2.b, p1.b
+; VBITS_GE_256-NEXT:    cmpne p1.d, p1/z, z3.d, #0
+; VBITS_GE_256-NEXT:    ld1w { z5.d }, p4/z, [z5.d]
+; VBITS_GE_256-NEXT:    ld1w { z0.d }, p2/z, [z0.d]
+; VBITS_GE_256-NEXT:    ld1w { z3.d }, p1/z, [z4.d]
+; VBITS_GE_256-NEXT:    ptrue p3.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z4.s, z16.s, z16.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z17.s, z17.s, z17.s
+; VBITS_GE_256-NEXT:    splice z4.s, p3, z4.s, z1.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z7.s, z7.s
+; VBITS_GE_256-NEXT:    uzp1 z5.s, z5.s, z5.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    splice z2.s, p3, z2.s, z17.s
+; VBITS_GE_256-NEXT:    splice z1.s, p3, z1.s, z5.s
+; VBITS_GE_256-NEXT:    splice z0.s, p3, z0.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_gather_passthru_0:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_2048-NEXT:    ptrue p1.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p1/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p1.s, p0/z, z0.s, #0.0
 ; VBITS_GE_2048-NEXT:    punpklo p1.h, p1.b

llvm/test/CodeGen/AArch64/sve-fixed-length-masked-stores.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -D#VBYTES=16  -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -D#VBYTES=32  -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -D#VBYTES=64  -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -D#VBYTES=128 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -D#VBYTES=256 -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Don't use SVE when its registers are no bigger than NEON.
 ; NO_SVE-NOT: ptrue
 
 ;;
 ;; Masked Stores
 ;;
 define void @masked_store_v2f16(<2 x half>* %ap, <2 x half>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v2f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr s0, [x0]
+; NO_SVE-NEXT:    ldr s1, [x1]
+; NO_SVE-NEXT:    fcmeq v1.4h, v0.4h, v1.4h
+; NO_SVE-NEXT:    umov w8, v1.h[1]
+; NO_SVE-NEXT:    umov w9, v1.h[0]
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB0_3
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB0_4
+; NO_SVE-NEXT:  .LBB0_2: // %else2
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB0_3: // %cond.store
+; NO_SVE-NEXT:    str h0, [x1]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB0_2
+; NO_SVE-NEXT:  .LBB0_4: // %cond.store1
+; NO_SVE-NEXT:    add x8, x1, #2
+; NO_SVE-NEXT:    st1 { v0.h }[1], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_store_v2f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr s1, [x0]
 ; CHECK-NEXT:    movi d0, #0000000000000000
 ; CHECK-NEXT:    ldr s2, [x1]
 ; CHECK-NEXT:    ptrue p0.h, vl4
 ; CHECK-NEXT:    fcmeq v2.4h, v1.4h, v2.4h
 ; CHECK-NEXT:    umov w8, v2.h[0]
; CHECK-NEXT:    ret
   %b = load <2 x half>, <2 x half>* %bp
   %mask = fcmp oeq <2 x half> %a, %b
   call void @llvm.masked.store.v2f16(<2 x half> %a, <2 x half>* %bp, i32 8, <2 x i1> %mask)
   ret void
 }
 
 
 define void @masked_store_v2f32(<2 x float>* %ap, <2 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr d0, [x0]
+; NO_SVE-NEXT:    ldr d1, [x1]
+; NO_SVE-NEXT:    fcmeq v1.2s, v0.2s, v1.2s
+; NO_SVE-NEXT:    mov w8, v1.s[1]
+; NO_SVE-NEXT:    fmov w9, s1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #31
+; NO_SVE-NEXT:    and w8, w9, #0x3
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB1_3
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB1_4
+; NO_SVE-NEXT:  .LBB1_2: // %else2
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB1_3: // %cond.store
+; NO_SVE-NEXT:    str s0, [x1]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB1_2
+; NO_SVE-NEXT:  .LBB1_4: // %cond.store1
+; NO_SVE-NEXT:    add x8, x1, #4
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_store_v2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr d0, [x0]
 ; CHECK-NEXT:    ptrue p0.s, vl2
 ; CHECK-NEXT:    ldr d1, [x1]
 ; CHECK-NEXT:    fcmeq v1.2s, v0.2s, v1.2s
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
 ; CHECK-NEXT:    ret
   %a = load <2 x float>, <2 x float>* %ap
   %b = load <2 x float>, <2 x float>* %bp
   %mask = fcmp oeq <2 x float> %a, %b
   call void @llvm.masked.store.v2f32(<2 x float> %a, <2 x float>* %bp, i32 8, <2 x i1> %mask)
   ret void
 }
 
 define void @masked_store_v4f32(<4 x float>* %ap, <4 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    fcmeq v1.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    xtn v1.4h, v1.4s
+; NO_SVE-NEXT:    umov w8, v1.h[1]
+; NO_SVE-NEXT:    umov w9, v1.h[2]
+; NO_SVE-NEXT:    umov w10, v1.h[0]
+; NO_SVE-NEXT:    umov w11, v1.h[3]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #29
+; NO_SVE-NEXT:    and w8, w10, #0xf
+; NO_SVE-NEXT:    tbnz w10, #0, .LBB2_5
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB2_6
+; NO_SVE-NEXT:  .LBB2_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB2_7
+; NO_SVE-NEXT:  .LBB2_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB2_8
+; NO_SVE-NEXT:  .LBB2_4: // %else6
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB2_5: // %cond.store
+; NO_SVE-NEXT:    str s0, [x1]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB2_2
+; NO_SVE-NEXT:  .LBB2_6: // %cond.store1
+; NO_SVE-NEXT:    add x9, x1, #4
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB2_3
+; NO_SVE-NEXT:  .LBB2_7: // %cond.store3
+; NO_SVE-NEXT:    add x9, x1, #8
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB2_4
+; NO_SVE-NEXT:  .LBB2_8: // %cond.store5
+; NO_SVE-NEXT:    add x8, x1, #12
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_store_v4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldr q0, [x0]
 ; CHECK-NEXT:    ptrue p0.s, vl4
 ; CHECK-NEXT:    ldr q1, [x1]
 ; CHECK-NEXT:    fcmeq v1.4s, v0.4s, v1.4s
 ; CHECK-NEXT:    cmpne p0.s, p0/z, z1.s, #0
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
 ; CHECK-NEXT:    ret
   %a = load <4 x float>, <4 x float>* %ap
   %b = load <4 x float>, <4 x float>* %bp
   %mask = fcmp oeq <4 x float> %a, %b
   call void @llvm.masked.store.v4f32(<4 x float> %a, <4 x float>* %bp, i32 8, <4 x i1> %mask)
   ret void
 }
 
 define void @masked_store_v8f32(<8 x float>* %ap, <8 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    ldp q3, q2, [x1]
+; NO_SVE-NEXT:    fcmeq v3.4s, v1.4s, v3.4s
+; NO_SVE-NEXT:    fcmeq v2.4s, v0.4s, v2.4s
+; NO_SVE-NEXT:    uzp1 v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v2.b[7]
+; NO_SVE-NEXT:    bfi w9, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB3_9
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB3_10
+; NO_SVE-NEXT:  .LBB3_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB3_11
+; NO_SVE-NEXT:  .LBB3_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB3_12
+; NO_SVE-NEXT:  .LBB3_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB3_13
+; NO_SVE-NEXT:  .LBB3_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB3_14
+; NO_SVE-NEXT:  .LBB3_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB3_15
+; NO_SVE-NEXT:  .LBB3_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB3_16
+; NO_SVE-NEXT:  .LBB3_8: // %else14
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB3_9: // %cond.store
+; NO_SVE-NEXT:    str s1, [x1]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB3_2
+; NO_SVE-NEXT:  .LBB3_10: // %cond.store1
+; NO_SVE-NEXT:    add x9, x1, #4
+; NO_SVE-NEXT:    st1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB3_3
+; NO_SVE-NEXT:  .LBB3_11: // %cond.store3
+; NO_SVE-NEXT:    add x9, x1, #8
+; NO_SVE-NEXT:    st1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB3_4
+; NO_SVE-NEXT:  .LBB3_12: // %cond.store5
+; NO_SVE-NEXT:    add x9, x1, #12
+; NO_SVE-NEXT:    st1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB3_5
+; NO_SVE-NEXT:  .LBB3_13: // %cond.store7
+; NO_SVE-NEXT:    str s0, [x1, #16]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB3_6
+; NO_SVE-NEXT:  .LBB3_14: // %cond.store9
+; NO_SVE-NEXT:    add x9, x1, #20
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB3_7
+; NO_SVE-NEXT:  .LBB3_15: // %cond.store11
+; NO_SVE-NEXT:    add x9, x1, #24
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB3_8
+; NO_SVE-NEXT:  .LBB3_16: // %cond.store13
+; NO_SVE-NEXT:    add x8, x1, #28
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: masked_store_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; CHECK-NEXT:    fcmeq p0.s, p0/z, z0.s, z1.s
 ; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
 ; CHECK-NEXT:    ret
   %a = load <8 x float>, <8 x float>* %ap
   %b = load <8 x float>, <8 x float>* %bp
   %mask = fcmp oeq <8 x float> %a, %b
   call void @llvm.masked.store.v8f32(<8 x float> %a, <8 x float>* %bp, i32 8, <8 x i1> %mask)
   ret void
 }
 
 define void @masked_store_v16f32(<16 x float>* %ap, <16 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    ldp q1, q0, [x1]
+; NO_SVE-NEXT:    fcmeq v1.4s, v3.4s, v1.4s
+; NO_SVE-NEXT:    fcmeq v4.4s, v2.4s, v0.4s
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #32]
+; NO_SVE-NEXT:    uzp1 v4.8h, v1.8h, v4.8h
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    xtn v4.8b, v4.8h
+; NO_SVE-NEXT:    umov w8, v4.b[1]
+; NO_SVE-NEXT:    umov w10, v4.b[2]
+; NO_SVE-NEXT:    fcmeq v6.4s, v1.4s, v6.4s
+; NO_SVE-NEXT:    umov w9, v4.b[0]
+; NO_SVE-NEXT:    umov w11, v4.b[3]
+; NO_SVE-NEXT:    fcmeq v5.4s, v0.4s, v5.4s
+; NO_SVE-NEXT:    umov w12, v4.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v4.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    uzp1 v5.8h, v6.8h, v5.8h
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v4.b[6]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v4.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w11, v5.b[0]
+; NO_SVE-NEXT:    umov w12, v5.b[1]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w13, v5.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v5.b[3]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v5.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v5.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v5.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    and w9, w11, #0x1
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB4_17
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB4_18
+; NO_SVE-NEXT:  .LBB4_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB4_19
+; NO_SVE-NEXT:  .LBB4_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB4_20
+; NO_SVE-NEXT:  .LBB4_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB4_21
+; NO_SVE-NEXT:  .LBB4_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB4_22
+; NO_SVE-NEXT:  .LBB4_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB4_23
+; NO_SVE-NEXT:  .LBB4_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB4_24
+; NO_SVE-NEXT:  .LBB4_8: // %else14
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB4_25
+; NO_SVE-NEXT:  .LBB4_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB4_26
+; NO_SVE-NEXT:  .LBB4_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB4_27
+; NO_SVE-NEXT:  .LBB4_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB4_28
+; NO_SVE-NEXT:  .LBB4_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB4_29
+; NO_SVE-NEXT:  .LBB4_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB4_30
+; NO_SVE-NEXT:  .LBB4_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB4_31
+; NO_SVE-NEXT:  .LBB4_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB4_32
+; NO_SVE-NEXT:  .LBB4_16: // %else30
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB4_17: // %cond.store
+; NO_SVE-NEXT:    str s3, [x0]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB4_2
+; NO_SVE-NEXT:  .LBB4_18: // %cond.store1
+; NO_SVE-NEXT:    add x9, x0, #4
+; NO_SVE-NEXT:    st1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB4_3
+; NO_SVE-NEXT:  .LBB4_19: // %cond.store3
+; NO_SVE-NEXT:    add x9, x0, #8
+; NO_SVE-NEXT:    st1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB4_4
+; NO_SVE-NEXT:  .LBB4_20: // %cond.store5
+; NO_SVE-NEXT:    add x9, x0, #12
+; NO_SVE-NEXT:    st1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB4_5
+; NO_SVE-NEXT:  .LBB4_21: // %cond.store7
+; NO_SVE-NEXT:    str s2, [x0, #16]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB4_6
+; NO_SVE-NEXT:  .LBB4_22: // %cond.store9
+; NO_SVE-NEXT:    add x9, x0, #20
+; NO_SVE-NEXT:    st1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB4_7
+; NO_SVE-NEXT:  .LBB4_23: // %cond.store11
+; NO_SVE-NEXT:    add x9, x0, #24
+; NO_SVE-NEXT:    st1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB4_8
+; NO_SVE-NEXT:  .LBB4_24: // %cond.store13
+; NO_SVE-NEXT:    add x9, x0, #28
+; NO_SVE-NEXT:    st1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB4_9
+; NO_SVE-NEXT:  .LBB4_25: // %cond.store15
+; NO_SVE-NEXT:    str s1, [x0, #32]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB4_10
+; NO_SVE-NEXT:  .LBB4_26: // %cond.store17
+; NO_SVE-NEXT:    add x9, x0, #36
+; NO_SVE-NEXT:    st1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB4_11
+; NO_SVE-NEXT:  .LBB4_27: // %cond.store19
+; NO_SVE-NEXT:    add x9, x0, #40
+; NO_SVE-NEXT:    st1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB4_12
+; NO_SVE-NEXT:  .LBB4_28: // %cond.store21
+; NO_SVE-NEXT:    add x9, x0, #44
+; NO_SVE-NEXT:    st1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB4_13
+; NO_SVE-NEXT:  .LBB4_29: // %cond.store23
+; NO_SVE-NEXT:    str s0, [x0, #48]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB4_14
+; NO_SVE-NEXT:  .LBB4_30: // %cond.store25
+; NO_SVE-NEXT:    add x9, x0, #52
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB4_15
+; NO_SVE-NEXT:  .LBB4_31: // %cond.store27
+; NO_SVE-NEXT:    add x9, x0, #56
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB4_16
+; NO_SVE-NEXT:  .LBB4_32: // %cond.store29
+; NO_SVE-NEXT:    add x8, x0, #60
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z0.s, z2.s
+; VBITS_GE_256-NEXT:    fcmeq p0.s, p0/z, z1.s, z3.s
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p1, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: masked_store_v16f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    fcmeq p0.s, p0/z, z0.s, z1.s
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %a = load <16 x float>, <16 x float>* %ap
   %b = load <16 x float>, <16 x float>* %bp
   %mask = fcmp oeq <16 x float> %a, %b
   call void @llvm.masked.store.v16f32(<16 x float> %a, <16 x float>* %ap, i32 8, <16 x i1> %mask)
   ret void
 }
 
 define void @masked_store_v32f32(<32 x float>* %ap, <32 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q0, q1, [x1, #64]
+; NO_SVE-NEXT:    fcmeq v7.4s, v3.4s, v0.4s
+; NO_SVE-NEXT:    fcmeq v5.4s, v2.4s, v1.4s
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    uzp1 v5.8h, v7.8h, v5.8h
+; NO_SVE-NEXT:    ldp q16, q17, [x1, #96]
+; NO_SVE-NEXT:    xtn v20.8b, v5.8h
+; NO_SVE-NEXT:    umov w8, v20.b[1]
+; NO_SVE-NEXT:    umov w10, v20.b[2]
+; NO_SVE-NEXT:    fcmeq v16.4s, v1.4s, v16.4s
+; NO_SVE-NEXT:    umov w9, v20.b[0]
+; NO_SVE-NEXT:    umov w11, v20.b[3]
+; NO_SVE-NEXT:    fcmeq v17.4s, v0.4s, v17.4s
+; NO_SVE-NEXT:    umov w12, v20.b[4]
+; NO_SVE-NEXT:    ldp q6, q4, [x0]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v20.b[5]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v20.b[6]
+; NO_SVE-NEXT:    uzp1 v16.8h, v16.8h, v17.8h
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v20.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    ldp q18, q7, [x1]
+; NO_SVE-NEXT:    xtn v16.8b, v16.8h
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w10, v16.b[0]
+; NO_SVE-NEXT:    umov w11, v16.b[1]
+; NO_SVE-NEXT:    fcmeq v18.4s, v6.4s, v18.4s
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v16.b[2]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    fcmeq v19.4s, v4.4s, v7.4s
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    ldp q7, q5, [x0, #32]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w13, v16.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #6
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    uzp1 v17.8h, v18.8h, v19.8h
+; NO_SVE-NEXT:    umov w14, v16.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #7
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #8
+; NO_SVE-NEXT:    ldp q18, q19, [x1, #32]
+; NO_SVE-NEXT:    xtn v17.8b, v17.8h
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #9
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #10
+; NO_SVE-NEXT:    umov w10, v17.b[1]
+; NO_SVE-NEXT:    umov w12, v17.b[2]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w11, v17.b[0]
+; NO_SVE-NEXT:    fcmeq v18.4s, v7.4s, v18.4s
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #11
+; NO_SVE-NEXT:    fcmeq v19.4s, v5.4s, v19.4s
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #12
+; NO_SVE-NEXT:    umov w13, v17.b[3]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v17.b[4]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w15, v17.b[5]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    uzp1 v18.8h, v18.8h, v19.8h
+; NO_SVE-NEXT:    bfi w11, w10, #1, #1
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    bfi w11, w12, #2, #1
+; NO_SVE-NEXT:    and w12, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v17.b[6]
+; NO_SVE-NEXT:    umov w9, v16.b[5]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    bfi w11, w10, #3, #1
+; NO_SVE-NEXT:    umov w10, v17.b[7]
+; NO_SVE-NEXT:    xtn v17.8b, v18.8h
+; NO_SVE-NEXT:    bfi w11, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v16.b[6]
+; NO_SVE-NEXT:    bfi w11, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v17.b[0]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w15, v17.b[1]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    orr w9, w11, w13, lsl #6
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v17.b[2]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #7
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v17.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #8
+; NO_SVE-NEXT:    umov w10, v17.b[4]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v17.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v17.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v16.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v17.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbnz w8, #0, .LBB5_33
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB5_34
+; NO_SVE-NEXT:  .LBB5_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB5_35
+; NO_SVE-NEXT:  .LBB5_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB5_36
+; NO_SVE-NEXT:  .LBB5_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB5_37
+; NO_SVE-NEXT:  .LBB5_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB5_38
+; NO_SVE-NEXT:  .LBB5_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB5_39
+; NO_SVE-NEXT:  .LBB5_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB5_40
+; NO_SVE-NEXT:  .LBB5_8: // %else14
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB5_41
+; NO_SVE-NEXT:  .LBB5_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB5_42
+; NO_SVE-NEXT:  .LBB5_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB5_43
+; NO_SVE-NEXT:  .LBB5_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB5_44
+; NO_SVE-NEXT:  .LBB5_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB5_45
+; NO_SVE-NEXT:  .LBB5_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB5_46
+; NO_SVE-NEXT:  .LBB5_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB5_47
+; NO_SVE-NEXT:  .LBB5_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB5_48
+; NO_SVE-NEXT:  .LBB5_16: // %else30
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB5_49
+; NO_SVE-NEXT:  .LBB5_17: // %else32
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB5_50
+; NO_SVE-NEXT:  .LBB5_18: // %else34
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB5_51
+; NO_SVE-NEXT:  .LBB5_19: // %else36
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB5_52
+; NO_SVE-NEXT:  .LBB5_20: // %else38
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB5_53
+; NO_SVE-NEXT:  .LBB5_21: // %else40
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB5_54
+; NO_SVE-NEXT:  .LBB5_22: // %else42
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB5_55
+; NO_SVE-NEXT:  .LBB5_23: // %else44
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB5_56
+; NO_SVE-NEXT:  .LBB5_24: // %else46
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB5_57
+; NO_SVE-NEXT:  .LBB5_25: // %else48
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB5_58
+; NO_SVE-NEXT:  .LBB5_26: // %else50
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB5_59
+; NO_SVE-NEXT:  .LBB5_27: // %else52
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB5_60
+; NO_SVE-NEXT:  .LBB5_28: // %else54
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB5_61
+; NO_SVE-NEXT:  .LBB5_29: // %else56
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB5_62
+; NO_SVE-NEXT:  .LBB5_30: // %else58
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB5_63
+; NO_SVE-NEXT:  .LBB5_31: // %else60
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB5_64
+; NO_SVE-NEXT:  .LBB5_32: // %else62
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB5_33: // %cond.store
+; NO_SVE-NEXT:    str s6, [x0]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB5_2
+; NO_SVE-NEXT:  .LBB5_34: // %cond.store1
+; NO_SVE-NEXT:    add x9, x0, #4
+; NO_SVE-NEXT:    st1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB5_3
+; NO_SVE-NEXT:  .LBB5_35: // %cond.store3
+; NO_SVE-NEXT:    add x9, x0, #8
+; NO_SVE-NEXT:    st1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB5_4
+; NO_SVE-NEXT:  .LBB5_36: // %cond.store5
+; NO_SVE-NEXT:    add x9, x0, #12
+; NO_SVE-NEXT:    st1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB5_5
+; NO_SVE-NEXT:  .LBB5_37: // %cond.store7
+; NO_SVE-NEXT:    str s4, [x0, #16]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB5_6
+; NO_SVE-NEXT:  .LBB5_38: // %cond.store9
+; NO_SVE-NEXT:    add x9, x0, #20
+; NO_SVE-NEXT:    st1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB5_7
+; NO_SVE-NEXT:  .LBB5_39: // %cond.store11
+; NO_SVE-NEXT:    add x9, x0, #24
+; NO_SVE-NEXT:    st1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB5_8
+; NO_SVE-NEXT:  .LBB5_40: // %cond.store13
+; NO_SVE-NEXT:    add x9, x0, #28
+; NO_SVE-NEXT:    st1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB5_9
+; NO_SVE-NEXT:  .LBB5_41: // %cond.store15
+; NO_SVE-NEXT:    str s7, [x0, #32]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB5_10
+; NO_SVE-NEXT:  .LBB5_42: // %cond.store17
+; NO_SVE-NEXT:    add x9, x0, #36
+; NO_SVE-NEXT:    st1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB5_11
+; NO_SVE-NEXT:  .LBB5_43: // %cond.store19
+; NO_SVE-NEXT:    add x9, x0, #40
+; NO_SVE-NEXT:    st1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB5_12
+; NO_SVE-NEXT:  .LBB5_44: // %cond.store21
+; NO_SVE-NEXT:    add x9, x0, #44
+; NO_SVE-NEXT:    st1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB5_13
+; NO_SVE-NEXT:  .LBB5_45: // %cond.store23
+; NO_SVE-NEXT:    str s5, [x0, #48]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB5_14
+; NO_SVE-NEXT:  .LBB5_46: // %cond.store25
+; NO_SVE-NEXT:    add x9, x0, #52
+; NO_SVE-NEXT:    st1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB5_15
+; NO_SVE-NEXT:  .LBB5_47: // %cond.store27
+; NO_SVE-NEXT:    add x9, x0, #56
+; NO_SVE-NEXT:    st1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB5_16
+; NO_SVE-NEXT:  .LBB5_48: // %cond.store29
+; NO_SVE-NEXT:    add x9, x0, #60
+; NO_SVE-NEXT:    st1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB5_17
+; NO_SVE-NEXT:  .LBB5_49: // %cond.store31
+; NO_SVE-NEXT:    str s3, [x0, #64]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB5_18
+; NO_SVE-NEXT:  .LBB5_50: // %cond.store33
+; NO_SVE-NEXT:    add x9, x0, #68
+; NO_SVE-NEXT:    st1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB5_19
+; NO_SVE-NEXT:  .LBB5_51: // %cond.store35
+; NO_SVE-NEXT:    add x9, x0, #72
+; NO_SVE-NEXT:    st1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB5_20
+; NO_SVE-NEXT:  .LBB5_52: // %cond.store37
+; NO_SVE-NEXT:    add x9, x0, #76
+; NO_SVE-NEXT:    st1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB5_21
+; NO_SVE-NEXT:  .LBB5_53: // %cond.store39
+; NO_SVE-NEXT:    str s2, [x0, #80]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB5_22
+; NO_SVE-NEXT:  .LBB5_54: // %cond.store41
+; NO_SVE-NEXT:    add x9, x0, #84
+; NO_SVE-NEXT:    st1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB5_23
+; NO_SVE-NEXT:  .LBB5_55: // %cond.store43
+; NO_SVE-NEXT:    add x9, x0, #88
+; NO_SVE-NEXT:    st1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB5_24
+; NO_SVE-NEXT:  .LBB5_56: // %cond.store45
+; NO_SVE-NEXT:    add x9, x0, #92
+; NO_SVE-NEXT:    st1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB5_25
+; NO_SVE-NEXT:  .LBB5_57: // %cond.store47
+; NO_SVE-NEXT:    str s1, [x0, #96]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB5_26
+; NO_SVE-NEXT:  .LBB5_58: // %cond.store49
+; NO_SVE-NEXT:    add x9, x0, #100
+; NO_SVE-NEXT:    st1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB5_27
+; NO_SVE-NEXT:  .LBB5_59: // %cond.store51
+; NO_SVE-NEXT:    add x9, x0, #104
+; NO_SVE-NEXT:    st1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB5_28
+; NO_SVE-NEXT:  .LBB5_60: // %cond.store53
+; NO_SVE-NEXT:    add x9, x0, #108
+; NO_SVE-NEXT:    st1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB5_29
+; NO_SVE-NEXT:  .LBB5_61: // %cond.store55
+; NO_SVE-NEXT:    str s0, [x0, #112]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB5_30
+; NO_SVE-NEXT:  .LBB5_62: // %cond.store57
+; NO_SVE-NEXT:    add x9, x0, #116
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB5_31
+; NO_SVE-NEXT:  .LBB5_63: // %cond.store59
+; NO_SVE-NEXT:    add x9, x0, #120
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #31, .LBB5_32
+; NO_SVE-NEXT:  .LBB5_64: // %cond.store61
+; NO_SVE-NEXT:    add x8, x0, #124
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z2.s, z5.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z1.s, z4.s
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z0.s, z6.s
+; VBITS_GE_256-NEXT:    fcmeq p0.s, p0/z, z3.s, z7.s
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p3, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p2, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p1, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: masked_store_v32f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    fcmeq p0.s, p0/z, z0.s, z1.s
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %a = load <32 x float>, <32 x float>* %ap
   %b = load <32 x float>, <32 x float>* %bp
   %mask = fcmp oeq <32 x float> %a, %b
   call void @llvm.masked.store.v32f32(<32 x float> %a, <32 x float>* %ap, i32 8, <32 x i1> %mask)
   ret void
 }
 
 define void @masked_store_v64f32(<64 x float>* %ap, <64 x float>* %bp) #0 {
+; NO_SVE-LABEL: masked_store_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #48
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 48
+; NO_SVE-NEXT:    stp d11, d10, [sp, #16] // 16-byte Folded Spill
+; NO_SVE-NEXT:    stp d9, d8, [sp, #32] // 16-byte Folded Spill
+; NO_SVE-NEXT:    .cfi_offset b8, -8
+; NO_SVE-NEXT:    .cfi_offset b9, -16
+; NO_SVE-NEXT:    .cfi_offset b10, -24
+; NO_SVE-NEXT:    .cfi_offset b11, -32
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #192]
+; NO_SVE-NEXT:    ldp q4, q5, [x1, #192]
+; NO_SVE-NEXT:    fcmeq v4.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    fcmeq v5.4s, v2.4s, v5.4s
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #224]
+; NO_SVE-NEXT:    uzp1 v4.8h, v4.8h, v5.8h
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #224]
+; NO_SVE-NEXT:    xtn v24.8b, v4.8h
+; NO_SVE-NEXT:    umov w8, v24.b[1]
+; NO_SVE-NEXT:    umov w10, v24.b[2]
+; NO_SVE-NEXT:    fcmeq v6.4s, v1.4s, v6.4s
+; NO_SVE-NEXT:    umov w9, v24.b[0]
+; NO_SVE-NEXT:    umov w11, v24.b[3]
+; NO_SVE-NEXT:    fcmeq v7.4s, v0.4s, v7.4s
+; NO_SVE-NEXT:    umov w12, v24.b[4]
+; NO_SVE-NEXT:    umov w13, v24.b[5]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    ldp q26, q27, [x1, #128]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w14, v24.b[6]
+; NO_SVE-NEXT:    uzp1 v5.8h, v6.8h, v7.8h
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w15, v24.b[7]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #128]
+; NO_SVE-NEXT:    xtn v28.8b, v5.8h
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w10, v28.b[0]
+; NO_SVE-NEXT:    bfi w9, w8, #5, #1
+; NO_SVE-NEXT:    fcmeq v30.4s, v7.4s, v26.4s
+; NO_SVE-NEXT:    orr w8, w9, w11, lsl #6
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    umov w11, v28.b[2]
+; NO_SVE-NEXT:    fcmeq v29.4s, v6.4s, v27.4s
+; NO_SVE-NEXT:    umov w14, v28.b[6]
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    umov w10, v28.b[1]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #7
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    uzp1 v31.8h, v30.8h, v29.8h
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    umov w10, v28.b[3]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #160]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #9
+; NO_SVE-NEXT:    xtn v31.8b, v31.8h
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    and w9, w10, #0x1
+; NO_SVE-NEXT:    umov w10, v28.b[4]
+; NO_SVE-NEXT:    umov w11, v31.b[1]
+; NO_SVE-NEXT:    umov w12, v31.b[0]
+; NO_SVE-NEXT:    umov w13, v31.b[2]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    ldp q10, q11, [x1, #160]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    umov w9, v28.b[5]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v31.b[6]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v31.b[3]
+; NO_SVE-NEXT:    bfi w10, w11, #1, #1
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v31.b[4]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    fcmeq v11.4s, v4.4s, v11.4s
+; NO_SVE-NEXT:    fcmeq v10.4s, v5.4s, v10.4s
+; NO_SVE-NEXT:    bfi w10, w11, #2, #1
+; NO_SVE-NEXT:    ldp q19, q18, [x0, #64]
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v31.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #13
+; NO_SVE-NEXT:    uzp1 v10.8h, v10.8h, v11.8h
+; NO_SVE-NEXT:    and w9, w14, #0x1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #1
+; NO_SVE-NEXT:    umov w14, v28.b[7]
+; NO_SVE-NEXT:    bfi w10, w12, #4, #1
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    ldp q8, q9, [x1, #64]
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v31.b[7]
+; NO_SVE-NEXT:    xtn v31.8b, v10.8h
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w14, lsl #15
+; NO_SVE-NEXT:    fcmeq v8.4s, v19.4s, v8.4s
+; NO_SVE-NEXT:    bfi w10, w11, #5, #1
+; NO_SVE-NEXT:    umov w11, v31.b[0]
+; NO_SVE-NEXT:    orr w9, w10, w12, lsl #6
+; NO_SVE-NEXT:    fcmeq v9.4s, v18.4s, v9.4s
+; NO_SVE-NEXT:    umov w12, v31.b[1]
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v31.b[2]
+; NO_SVE-NEXT:    ldp q17, q16, [x0, #96]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #7
+; NO_SVE-NEXT:    uzp1 v8.8h, v8.8h, v9.8h
+; NO_SVE-NEXT:    umov w10, v31.b[3]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v31.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #9
+; NO_SVE-NEXT:    umov w14, v31.b[6]
+; NO_SVE-NEXT:    xtn v28.8b, v8.8h
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    ldp q29, q30, [x1, #96]
+; NO_SVE-NEXT:    umov w11, v28.b[1]
+; NO_SVE-NEXT:    orr w9, w9, w12, lsl #10
+; NO_SVE-NEXT:    umov w12, v31.b[5]
+; NO_SVE-NEXT:    umov w15, v28.b[0]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #11
+; NO_SVE-NEXT:    and w10, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v28.b[2]
+; NO_SVE-NEXT:    umov w16, v28.b[5]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    fcmeq v30.4s, v16.4s, v30.4s
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    fcmeq v29.4s, v17.4s, v29.4s
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w12, w11, #1, #1
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v28.b[3]
+; NO_SVE-NEXT:    umov w15, v28.b[4]
+; NO_SVE-NEXT:    ldp q23, q22, [x0]
+; NO_SVE-NEXT:    uzp1 v29.8h, v29.8h, v30.8h
+; NO_SVE-NEXT:    bfi w12, w11, #2, #1
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #13
+; NO_SVE-NEXT:    umov w10, v28.b[7]
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    and w15, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v28.b[6]
+; NO_SVE-NEXT:    xtn v28.8b, v29.8h
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    ldp q26, q27, [x1]
+; NO_SVE-NEXT:    bfi w12, w11, #3, #1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    bfi w12, w13, #4, #1
+; NO_SVE-NEXT:    and w11, w16, #0x1
+; NO_SVE-NEXT:    umov w13, v28.b[0]
+; NO_SVE-NEXT:    bfi w12, w15, #5, #1
+; NO_SVE-NEXT:    fcmeq v26.4s, v23.4s, v26.4s
+; NO_SVE-NEXT:    orr w11, w12, w11, lsl #6
+; NO_SVE-NEXT:    umov w12, v28.b[1]
+; NO_SVE-NEXT:    orr w10, w11, w10, lsl #7
+; NO_SVE-NEXT:    fcmeq v27.4s, v22.4s, v27.4s
+; NO_SVE-NEXT:    umov w15, v28.b[3]
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v28.b[2]
+; NO_SVE-NEXT:    ldp q21, q20, [x0, #32]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w11, lsl #8
+; NO_SVE-NEXT:    uzp1 v26.8h, v26.8h, v27.8h
+; NO_SVE-NEXT:    umov w11, v28.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #14
+; NO_SVE-NEXT:    umov w14, v31.b[7]
+; NO_SVE-NEXT:    orr w10, w10, w12, lsl #9
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v28.b[5]
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    ldp q24, q25, [x1, #32]
+; NO_SVE-NEXT:    xtn v26.8b, v26.8h
+; NO_SVE-NEXT:    orr w10, w10, w12, lsl #10
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w14, lsl #15
+; NO_SVE-NEXT:    umov w16, v26.b[1]
+; NO_SVE-NEXT:    orr w10, w10, w15, lsl #11
+; NO_SVE-NEXT:    umov w14, v26.b[2]
+; NO_SVE-NEXT:    orr w10, w10, w11, lsl #12
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    umov w13, v26.b[0]
+; NO_SVE-NEXT:    fcmeq v25.4s, v20.4s, v25.4s
+; NO_SVE-NEXT:    umov w15, v26.b[3]
+; NO_SVE-NEXT:    fcmeq v24.4s, v21.4s, v24.4s
+; NO_SVE-NEXT:    and w12, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v26.b[4]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w17, v26.b[5]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w13, w12, #1, #1
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    uzp1 v24.8h, v24.8h, v25.8h
+; NO_SVE-NEXT:    bfi w13, w14, #2, #1
+; NO_SVE-NEXT:    and w14, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v26.b[6]
+; NO_SVE-NEXT:    and w15, w17, #0x1
+; NO_SVE-NEXT:    bfi w13, w12, #3, #1
+; NO_SVE-NEXT:    umov w12, v26.b[7]
+; NO_SVE-NEXT:    orr w10, w10, w11, lsl #13
+; NO_SVE-NEXT:    xtn v24.8b, v24.8h
+; NO_SVE-NEXT:    bfi w13, w14, #4, #1
+; NO_SVE-NEXT:    umov w14, v28.b[6]
+; NO_SVE-NEXT:    bfi w13, w15, #5, #1
+; NO_SVE-NEXT:    and w15, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v24.b[0]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w17, v24.b[1]
+; NO_SVE-NEXT:    orr w11, w13, w15, lsl #6
+; NO_SVE-NEXT:    bfi w9, w8, #16, #16
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v24.b[2]
+; NO_SVE-NEXT:    orr w11, w11, w12, lsl #7
+; NO_SVE-NEXT:    and w12, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v24.b[3]
+; NO_SVE-NEXT:    and w15, w17, #0x1
+; NO_SVE-NEXT:    orr w11, w11, w12, lsl #8
+; NO_SVE-NEXT:    umov w12, v24.b[4]
+; NO_SVE-NEXT:    orr w10, w10, w13, lsl #14
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v24.b[5]
+; NO_SVE-NEXT:    orr w11, w11, w15, lsl #9
+; NO_SVE-NEXT:    and w15, w16, #0x1
+; NO_SVE-NEXT:    umov w16, v24.b[6]
+; NO_SVE-NEXT:    orr w11, w11, w13, lsl #10
+; NO_SVE-NEXT:    umov w13, v28.b[7]
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    orr w11, w11, w15, lsl #11
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    umov w15, v24.b[7]
+; NO_SVE-NEXT:    orr w11, w11, w12, lsl #12
+; NO_SVE-NEXT:    and w12, w16, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w13, lsl #15
+; NO_SVE-NEXT:    orr w11, w11, w14, lsl #13
+; NO_SVE-NEXT:    orr w8, w11, w12, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w15, lsl #15
+; NO_SVE-NEXT:    bfi w8, w10, #16, #16
+; NO_SVE-NEXT:    bfi x8, x9, #32, #32
+; NO_SVE-NEXT:    tbnz w8, #0, .LBB6_66
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB6_67
+; NO_SVE-NEXT:  .LBB6_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB6_68
+; NO_SVE-NEXT:  .LBB6_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB6_69
+; NO_SVE-NEXT:  .LBB6_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB6_70
+; NO_SVE-NEXT:  .LBB6_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB6_71
+; NO_SVE-NEXT:  .LBB6_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB6_72
+; NO_SVE-NEXT:  .LBB6_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB6_73
+; NO_SVE-NEXT:  .LBB6_8: // %else14
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB6_74
+; NO_SVE-NEXT:  .LBB6_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB6_75
+; NO_SVE-NEXT:  .LBB6_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB6_76
+; NO_SVE-NEXT:  .LBB6_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB6_77
+; NO_SVE-NEXT:  .LBB6_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB6_78
+; NO_SVE-NEXT:  .LBB6_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB6_79
+; NO_SVE-NEXT:  .LBB6_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB6_80
+; NO_SVE-NEXT:  .LBB6_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB6_81
+; NO_SVE-NEXT:  .LBB6_16: // %else30
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB6_82
+; NO_SVE-NEXT:  .LBB6_17: // %else32
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB6_83
+; NO_SVE-NEXT:  .LBB6_18: // %else34
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB6_84
+; NO_SVE-NEXT:  .LBB6_19: // %else36
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB6_85
+; NO_SVE-NEXT:  .LBB6_20: // %else38
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB6_86
+; NO_SVE-NEXT:  .LBB6_21: // %else40
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB6_87
+; NO_SVE-NEXT:  .LBB6_22: // %else42
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB6_88
+; NO_SVE-NEXT:  .LBB6_23: // %else44
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB6_89
+; NO_SVE-NEXT:  .LBB6_24: // %else46
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB6_90
+; NO_SVE-NEXT:  .LBB6_25: // %else48
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB6_91
+; NO_SVE-NEXT:  .LBB6_26: // %else50
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB6_92
+; NO_SVE-NEXT:  .LBB6_27: // %else52
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB6_93
+; NO_SVE-NEXT:  .LBB6_28: // %else54
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB6_94
+; NO_SVE-NEXT:  .LBB6_29: // %else56
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB6_95
+; NO_SVE-NEXT:  .LBB6_30: // %else58
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB6_96
+; NO_SVE-NEXT:  .LBB6_31: // %else60
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB6_97
+; NO_SVE-NEXT:  .LBB6_32: // %else62
+; NO_SVE-NEXT:    tbnz x8, #32, .LBB6_98
+; NO_SVE-NEXT:  .LBB6_33: // %else64
+; NO_SVE-NEXT:    tbnz x8, #33, .LBB6_99
+; NO_SVE-NEXT:  .LBB6_34: // %else66
+; NO_SVE-NEXT:    tbnz x8, #34, .LBB6_100
+; NO_SVE-NEXT:  .LBB6_35: // %else68
+; NO_SVE-NEXT:    tbnz x8, #35, .LBB6_101
+; NO_SVE-NEXT:  .LBB6_36: // %else70
+; NO_SVE-NEXT:    tbnz x8, #36, .LBB6_102
+; NO_SVE-NEXT:  .LBB6_37: // %else72
+; NO_SVE-NEXT:    tbnz x8, #37, .LBB6_103
+; NO_SVE-NEXT:  .LBB6_38: // %else74
+; NO_SVE-NEXT:    tbnz x8, #38, .LBB6_104
+; NO_SVE-NEXT:  .LBB6_39: // %else76
+; NO_SVE-NEXT:    tbnz x8, #39, .LBB6_105
+; NO_SVE-NEXT:  .LBB6_40: // %else78
+; NO_SVE-NEXT:    tbnz x8, #40, .LBB6_106
+; NO_SVE-NEXT:  .LBB6_41: // %else80
+; NO_SVE-NEXT:    tbnz x8, #41, .LBB6_107
+; NO_SVE-NEXT:  .LBB6_42: // %else82
+; NO_SVE-NEXT:    tbnz x8, #42, .LBB6_108
+; NO_SVE-NEXT:  .LBB6_43: // %else84
+; NO_SVE-NEXT:    tbnz x8, #43, .LBB6_109
+; NO_SVE-NEXT:  .LBB6_44: // %else86
+; NO_SVE-NEXT:    tbnz x8, #44, .LBB6_110
+; NO_SVE-NEXT:  .LBB6_45: // %else88
+; NO_SVE-NEXT:    tbnz x8, #45, .LBB6_111
+; NO_SVE-NEXT:  .LBB6_46: // %else90
+; NO_SVE-NEXT:    tbnz x8, #46, .LBB6_112
+; NO_SVE-NEXT:  .LBB6_47: // %else92
+; NO_SVE-NEXT:    tbnz x8, #47, .LBB6_113
+; NO_SVE-NEXT:  .LBB6_48: // %else94
+; NO_SVE-NEXT:    tbnz x8, #48, .LBB6_114
+; NO_SVE-NEXT:  .LBB6_49: // %else96
+; NO_SVE-NEXT:    tbnz x8, #49, .LBB6_115
+; NO_SVE-NEXT:  .LBB6_50: // %else98
+; NO_SVE-NEXT:    tbnz x8, #50, .LBB6_116
+; NO_SVE-NEXT:  .LBB6_51: // %else100
+; NO_SVE-NEXT:    tbnz x8, #51, .LBB6_117
+; NO_SVE-NEXT:  .LBB6_52: // %else102
+; NO_SVE-NEXT:    tbnz x8, #52, .LBB6_118
+; NO_SVE-NEXT:  .LBB6_53: // %else104
+; NO_SVE-NEXT:    tbnz x8, #53, .LBB6_119
+; NO_SVE-NEXT:  .LBB6_54: // %else106
+; NO_SVE-NEXT:    tbnz x8, #54, .LBB6_120
+; NO_SVE-NEXT:  .LBB6_55: // %else108
+; NO_SVE-NEXT:    tbnz x8, #55, .LBB6_121
+; NO_SVE-NEXT:  .LBB6_56: // %else110
+; NO_SVE-NEXT:    tbnz x8, #56, .LBB6_122
+; NO_SVE-NEXT:  .LBB6_57: // %else112
+; NO_SVE-NEXT:    tbnz x8, #57, .LBB6_123
+; NO_SVE-NEXT:  .LBB6_58: // %else114
+; NO_SVE-NEXT:    tbnz x8, #58, .LBB6_124
+; NO_SVE-NEXT:  .LBB6_59: // %else116
+; NO_SVE-NEXT:    tbnz x8, #59, .LBB6_125
+; NO_SVE-NEXT:  .LBB6_60: // %else118
+; NO_SVE-NEXT:    tbnz x8, #60, .LBB6_126
+; NO_SVE-NEXT:  .LBB6_61: // %else120
+; NO_SVE-NEXT:    tbnz x8, #61, .LBB6_127
+; NO_SVE-NEXT:  .LBB6_62: // %else122
+; NO_SVE-NEXT:    tbnz x8, #62, .LBB6_128
+; NO_SVE-NEXT:  .LBB6_63: // %else124
+; NO_SVE-NEXT:    tbz x8, #63, .LBB6_65
+; NO_SVE-NEXT:  .LBB6_64: // %cond.store125
+; NO_SVE-NEXT:    add x8, x0, #252
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:  .LBB6_65: // %else126
+; NO_SVE-NEXT:    ldp d9, d8, [sp, #32] // 16-byte Folded Reload
+; NO_SVE-NEXT:    ldp d11, d10, [sp, #16] // 16-byte Folded Reload
+; NO_SVE-NEXT:    add sp, sp, #48
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB6_66: // %cond.store
+; NO_SVE-NEXT:    str s23, [x0]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB6_2
+; NO_SVE-NEXT:  .LBB6_67: // %cond.store1
+; NO_SVE-NEXT:    add x9, x0, #4
+; NO_SVE-NEXT:    st1 { v23.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB6_3
+; NO_SVE-NEXT:  .LBB6_68: // %cond.store3
+; NO_SVE-NEXT:    add x9, x0, #8
+; NO_SVE-NEXT:    st1 { v23.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB6_4
+; NO_SVE-NEXT:  .LBB6_69: // %cond.store5
+; NO_SVE-NEXT:    add x9, x0, #12
+; NO_SVE-NEXT:    st1 { v23.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB6_5
+; NO_SVE-NEXT:  .LBB6_70: // %cond.store7
+; NO_SVE-NEXT:    str s22, [x0, #16]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB6_6
+; NO_SVE-NEXT:  .LBB6_71: // %cond.store9
+; NO_SVE-NEXT:    add x9, x0, #20
+; NO_SVE-NEXT:    st1 { v22.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB6_7
+; NO_SVE-NEXT:  .LBB6_72: // %cond.store11
+; NO_SVE-NEXT:    add x9, x0, #24
+; NO_SVE-NEXT:    st1 { v22.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB6_8
+; NO_SVE-NEXT:  .LBB6_73: // %cond.store13
+; NO_SVE-NEXT:    add x9, x0, #28
+; NO_SVE-NEXT:    st1 { v22.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB6_9
+; NO_SVE-NEXT:  .LBB6_74: // %cond.store15
+; NO_SVE-NEXT:    str s21, [x0, #32]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB6_10
+; NO_SVE-NEXT:  .LBB6_75: // %cond.store17
+; NO_SVE-NEXT:    add x9, x0, #36
+; NO_SVE-NEXT:    st1 { v21.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB6_11
+; NO_SVE-NEXT:  .LBB6_76: // %cond.store19
+; NO_SVE-NEXT:    add x9, x0, #40
+; NO_SVE-NEXT:    st1 { v21.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB6_12
+; NO_SVE-NEXT:  .LBB6_77: // %cond.store21
+; NO_SVE-NEXT:    add x9, x0, #44
+; NO_SVE-NEXT:    st1 { v21.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB6_13
+; NO_SVE-NEXT:  .LBB6_78: // %cond.store23
+; NO_SVE-NEXT:    str s20, [x0, #48]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB6_14
+; NO_SVE-NEXT:  .LBB6_79: // %cond.store25
+; NO_SVE-NEXT:    add x9, x0, #52
+; NO_SVE-NEXT:    st1 { v20.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB6_15
+; NO_SVE-NEXT:  .LBB6_80: // %cond.store27
+; NO_SVE-NEXT:    add x9, x0, #56
+; NO_SVE-NEXT:    st1 { v20.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB6_16
+; NO_SVE-NEXT:  .LBB6_81: // %cond.store29
+; NO_SVE-NEXT:    add x9, x0, #60
+; NO_SVE-NEXT:    st1 { v20.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #16, .LBB6_17
+; NO_SVE-NEXT:  .LBB6_82: // %cond.store31
+; NO_SVE-NEXT:    str s19, [x0, #64]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB6_18
+; NO_SVE-NEXT:  .LBB6_83: // %cond.store33
+; NO_SVE-NEXT:    add x9, x0, #68
+; NO_SVE-NEXT:    st1 { v19.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB6_19
+; NO_SVE-NEXT:  .LBB6_84: // %cond.store35
+; NO_SVE-NEXT:    add x9, x0, #72
+; NO_SVE-NEXT:    st1 { v19.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB6_20
+; NO_SVE-NEXT:  .LBB6_85: // %cond.store37
+; NO_SVE-NEXT:    add x9, x0, #76
+; NO_SVE-NEXT:    st1 { v19.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB6_21
+; NO_SVE-NEXT:  .LBB6_86: // %cond.store39
+; NO_SVE-NEXT:    str s18, [x0, #80]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB6_22
+; NO_SVE-NEXT:  .LBB6_87: // %cond.store41
+; NO_SVE-NEXT:    add x9, x0, #84
+; NO_SVE-NEXT:    st1 { v18.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB6_23
+; NO_SVE-NEXT:  .LBB6_88: // %cond.store43
+; NO_SVE-NEXT:    add x9, x0, #88
+; NO_SVE-NEXT:    st1 { v18.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB6_24
+; NO_SVE-NEXT:  .LBB6_89: // %cond.store45
+; NO_SVE-NEXT:    add x9, x0, #92
+; NO_SVE-NEXT:    st1 { v18.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB6_25
+; NO_SVE-NEXT:  .LBB6_90: // %cond.store47
+; NO_SVE-NEXT:    str s17, [x0, #96]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB6_26
+; NO_SVE-NEXT:  .LBB6_91: // %cond.store49
+; NO_SVE-NEXT:    add x9, x0, #100
+; NO_SVE-NEXT:    st1 { v17.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB6_27
+; NO_SVE-NEXT:  .LBB6_92: // %cond.store51
+; NO_SVE-NEXT:    add x9, x0, #104
+; NO_SVE-NEXT:    st1 { v17.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB6_28
+; NO_SVE-NEXT:  .LBB6_93: // %cond.store53
+; NO_SVE-NEXT:    add x9, x0, #108
+; NO_SVE-NEXT:    st1 { v17.s }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB6_29
+; NO_SVE-NEXT:  .LBB6_94: // %cond.store55
+; NO_SVE-NEXT:    str s16, [x0, #112]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB6_30
+; NO_SVE-NEXT:  .LBB6_95: // %cond.store57
+; NO_SVE-NEXT:    add x9, x0, #116
+; NO_SVE-NEXT:    st1 { v16.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB6_31
+; NO_SVE-NEXT:  .LBB6_96: // %cond.store59
+; NO_SVE-NEXT:    add x9, x0, #120
+; NO_SVE-NEXT:    st1 { v16.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #31, .LBB6_32
+; NO_SVE-NEXT:  .LBB6_97: // %cond.store61
+; NO_SVE-NEXT:    add x9, x0, #124
+; NO_SVE-NEXT:    st1 { v16.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #32, .LBB6_33
+; NO_SVE-NEXT:  .LBB6_98: // %cond.store63
+; NO_SVE-NEXT:    str s7, [x0, #128]
+; NO_SVE-NEXT:    tbz x8, #33, .LBB6_34
+; NO_SVE-NEXT:  .LBB6_99: // %cond.store65
+; NO_SVE-NEXT:    add x9, x0, #132
+; NO_SVE-NEXT:    st1 { v7.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #34, .LBB6_35
+; NO_SVE-NEXT:  .LBB6_100: // %cond.store67
+; NO_SVE-NEXT:    add x9, x0, #136
+; NO_SVE-NEXT:    st1 { v7.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #35, .LBB6_36
+; NO_SVE-NEXT:  .LBB6_101: // %cond.store69
+; NO_SVE-NEXT:    add x9, x0, #140
+; NO_SVE-NEXT:    st1 { v7.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #36, .LBB6_37
+; NO_SVE-NEXT:  .LBB6_102: // %cond.store71
+; NO_SVE-NEXT:    str s6, [x0, #144]
+; NO_SVE-NEXT:    tbz x8, #37, .LBB6_38
+; NO_SVE-NEXT:  .LBB6_103: // %cond.store73
+; NO_SVE-NEXT:    add x9, x0, #148
+; NO_SVE-NEXT:    st1 { v6.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #38, .LBB6_39
+; NO_SVE-NEXT:  .LBB6_104: // %cond.store75
+; NO_SVE-NEXT:    add x9, x0, #152
+; NO_SVE-NEXT:    st1 { v6.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #39, .LBB6_40
+; NO_SVE-NEXT:  .LBB6_105: // %cond.store77
+; NO_SVE-NEXT:    add x9, x0, #156
+; NO_SVE-NEXT:    st1 { v6.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #40, .LBB6_41
+; NO_SVE-NEXT:  .LBB6_106: // %cond.store79
+; NO_SVE-NEXT:    str s5, [x0, #160]
+; NO_SVE-NEXT:    tbz x8, #41, .LBB6_42
+; NO_SVE-NEXT:  .LBB6_107: // %cond.store81
+; NO_SVE-NEXT:    add x9, x0, #164
+; NO_SVE-NEXT:    st1 { v5.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #42, .LBB6_43
+; NO_SVE-NEXT:  .LBB6_108: // %cond.store83
+; NO_SVE-NEXT:    add x9, x0, #168
+; NO_SVE-NEXT:    st1 { v5.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #43, .LBB6_44
+; NO_SVE-NEXT:  .LBB6_109: // %cond.store85
+; NO_SVE-NEXT:    add x9, x0, #172
+; NO_SVE-NEXT:    st1 { v5.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #44, .LBB6_45
+; NO_SVE-NEXT:  .LBB6_110: // %cond.store87
+; NO_SVE-NEXT:    str s4, [x0, #176]
+; NO_SVE-NEXT:    tbz x8, #45, .LBB6_46
+; NO_SVE-NEXT:  .LBB6_111: // %cond.store89
+; NO_SVE-NEXT:    add x9, x0, #180
+; NO_SVE-NEXT:    st1 { v4.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #46, .LBB6_47
+; NO_SVE-NEXT:  .LBB6_112: // %cond.store91
+; NO_SVE-NEXT:    add x9, x0, #184
+; NO_SVE-NEXT:    st1 { v4.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #47, .LBB6_48
+; NO_SVE-NEXT:  .LBB6_113: // %cond.store93
+; NO_SVE-NEXT:    add x9, x0, #188
+; NO_SVE-NEXT:    st1 { v4.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #48, .LBB6_49
+; NO_SVE-NEXT:  .LBB6_114: // %cond.store95
+; NO_SVE-NEXT:    str s3, [x0, #192]
+; NO_SVE-NEXT:    tbz x8, #49, .LBB6_50
+; NO_SVE-NEXT:  .LBB6_115: // %cond.store97
+; NO_SVE-NEXT:    add x9, x0, #196
+; NO_SVE-NEXT:    st1 { v3.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #50, .LBB6_51
+; NO_SVE-NEXT:  .LBB6_116: // %cond.store99
+; NO_SVE-NEXT:    add x9, x0, #200
+; NO_SVE-NEXT:    st1 { v3.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #51, .LBB6_52
+; NO_SVE-NEXT:  .LBB6_117: // %cond.store101
+; NO_SVE-NEXT:    add x9, x0, #204
+; NO_SVE-NEXT:    st1 { v3.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #52, .LBB6_53
+; NO_SVE-NEXT:  .LBB6_118: // %cond.store103
+; NO_SVE-NEXT:    str s2, [x0, #208]
+; NO_SVE-NEXT:    tbz x8, #53, .LBB6_54
+; NO_SVE-NEXT:  .LBB6_119: // %cond.store105
+; NO_SVE-NEXT:    add x9, x0, #212
+; NO_SVE-NEXT:    st1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #54, .LBB6_55
+; NO_SVE-NEXT:  .LBB6_120: // %cond.store107
+; NO_SVE-NEXT:    add x9, x0, #216
+; NO_SVE-NEXT:    st1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #55, .LBB6_56
+; NO_SVE-NEXT:  .LBB6_121: // %cond.store109
+; NO_SVE-NEXT:    add x9, x0, #220
+; NO_SVE-NEXT:    st1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #56, .LBB6_57
+; NO_SVE-NEXT:  .LBB6_122: // %cond.store111
+; NO_SVE-NEXT:    str s1, [x0, #224]
+; NO_SVE-NEXT:    tbz x8, #57, .LBB6_58
+; NO_SVE-NEXT:  .LBB6_123: // %cond.store113
+; NO_SVE-NEXT:    add x9, x0, #228
+; NO_SVE-NEXT:    st1 { v1.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #58, .LBB6_59
+; NO_SVE-NEXT:  .LBB6_124: // %cond.store115
+; NO_SVE-NEXT:    add x9, x0, #232
+; NO_SVE-NEXT:    st1 { v1.s }[2], [x9]
+; NO_SVE-NEXT:    tbz x8, #59, .LBB6_60
+; NO_SVE-NEXT:  .LBB6_125: // %cond.store117
+; NO_SVE-NEXT:    add x9, x0, #236
+; NO_SVE-NEXT:    st1 { v1.s }[3], [x9]
+; NO_SVE-NEXT:    tbz x8, #60, .LBB6_61
+; NO_SVE-NEXT:  .LBB6_126: // %cond.store119
+; NO_SVE-NEXT:    str s0, [x0, #240]
+; NO_SVE-NEXT:    tbz x8, #61, .LBB6_62
+; NO_SVE-NEXT:  .LBB6_127: // %cond.store121
+; NO_SVE-NEXT:    add x9, x0, #244
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz x8, #62, .LBB6_63
+; NO_SVE-NEXT:  .LBB6_128: // %cond.store123
+; NO_SVE-NEXT:    add x9, x0, #248
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbnz x8, #63, .LBB6_64
+; NO_SVE-NEXT:    b .LBB6_65
+;
+; VBITS_GE_256-LABEL: masked_store_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #40
+; VBITS_GE_256-NEXT:    mov x11, #32
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    mov x13, #16
+; VBITS_GE_256-NEXT:    mov x14, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z16.s }, p0/z, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p0/z, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z18.s }, p0/z, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z19.s }, p0/z, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z20.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z21.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z22.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z23.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fcmeq p1.s, p0/z, z6.s, z17.s
+; VBITS_GE_256-NEXT:    fcmeq p2.s, p0/z, z5.s, z16.s
+; VBITS_GE_256-NEXT:    fcmeq p3.s, p0/z, z4.s, z19.s
+; VBITS_GE_256-NEXT:    fcmeq p4.s, p0/z, z3.s, z18.s
+; VBITS_GE_256-NEXT:    fcmeq p5.s, p0/z, z2.s, z21.s
+; VBITS_GE_256-NEXT:    fcmeq p6.s, p0/z, z1.s, z20.s
+; VBITS_GE_256-NEXT:    fcmeq p7.s, p0/z, z0.s, z22.s
+; VBITS_GE_256-NEXT:    fcmeq p0.s, p0/z, z7.s, z23.s
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p7, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p6, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p5, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p4, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p3, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p2, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p1, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: masked_store_v64f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    fcmeq p0.s, p0/z, z0.s, z1.s
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %a = load <64 x float>, <64 x float>* %ap
   %b = load <64 x float>, <64 x float>* %bp
   %mask = fcmp oeq <64 x float> %a, %b
   call void @llvm.masked.store.v64f32(<64 x float> %a, <64 x float>* %ap, i32 8, <64 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v8i64i8(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i8>* %dest) #0 {
+; NO_SVE-LABEL: masked_store_trunc_v8i64i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    cmeq v0.2d, v3.2d, v0.2d
+; NO_SVE-NEXT:    ldp q6, q7, [x1]
+; NO_SVE-NEXT:    cmeq v5.2d, v4.2d, v5.2d
+; NO_SVE-NEXT:    uzp1 v3.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v5.4s
+; NO_SVE-NEXT:    cmeq v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    uzp1 v1.4s, v1.4s, v2.4s
+; NO_SVE-NEXT:    cmeq v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    uzp1 v5.4s, v6.4s, v7.4s
+; NO_SVE-NEXT:    uzp1 v0.8h, v5.8h, v0.8h
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w8, #5, #1
+; NO_SVE-NEXT:    and w8, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB7_9
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB7_10
+; NO_SVE-NEXT:  .LBB7_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB7_11
+; NO_SVE-NEXT:  .LBB7_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB7_12
+; NO_SVE-NEXT:  .LBB7_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB7_13
+; NO_SVE-NEXT:  .LBB7_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB7_14
+; NO_SVE-NEXT:  .LBB7_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB7_15
+; NO_SVE-NEXT:  .LBB7_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB7_16
+; NO_SVE-NEXT:  .LBB7_8: // %else14
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB7_9: // %cond.store
+; NO_SVE-NEXT:    st1 { v0.b }[0], [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB7_2
+; NO_SVE-NEXT:  .LBB7_10: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #1
+; NO_SVE-NEXT:    st1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB7_3
+; NO_SVE-NEXT:  .LBB7_11: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #2
+; NO_SVE-NEXT:    st1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB7_4
+; NO_SVE-NEXT:  .LBB7_12: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #3
+; NO_SVE-NEXT:    st1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB7_5
+; NO_SVE-NEXT:  .LBB7_13: // %cond.store7
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB7_6
+; NO_SVE-NEXT:  .LBB7_14: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #5
+; NO_SVE-NEXT:    st1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB7_7
+; NO_SVE-NEXT:  .LBB7_15: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #6
+; NO_SVE-NEXT:    st1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB7_8
+; NO_SVE-NEXT:  .LBB7_16: // %cond.store13
+; NO_SVE-NEXT:    add x8, x2, #7
+; NO_SVE-NEXT:    st1 { v0.b }[7], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v8i64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z0.d, z2.d
+; VBITS_GE_256-NEXT:    cmpeq p0.d, p0/z, z1.d, z3.d
+; VBITS_GE_256-NEXT:    mov z2.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.d, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    cmpne p0.s, p0/z, z3.s, #0
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z0.s
+; VBITS_GE_256-NEXT:    st1b { z1.s }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_512-LABEL: masked_store_trunc_v8i64i8:
 ; VBITS_GE_512:       // %bb.0:
-; VBITS_GE_512-NEXT: ptrue p[[P0:[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: ld1d { [[Z0:z[0-9]+]].d }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1d { [[Z1:z[0-9]+]].d }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0]].d, [[Z1]].d
-; VBITS_GE_512-NEXT: st1b { [[Z0]].d }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.d, p0/z, z0.d, z1.d
+; VBITS_GE_512-NEXT:    st1b { z0.d }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
 
   %a = load <8 x i64>, <8 x i64>* %ap
   %b = load <8 x i64>, <8 x i64>* %bp
   %mask = icmp eq <8 x i64> %a, %b
   %val = trunc <8 x i64> %a to <8 x i8>
   call void @llvm.masked.store.v8i8(<8 x i8> %val, <8 x i8>* %dest, i32 8, <8 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v8i64i16(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i16>* %dest) #0 {
-; CHECK-LABEL: masked_store_trunc_v8i64i16:
-; VBITS_GE_512: ptrue p[[P0:[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: ld1d { [[Z0:z[0-9]+]].d }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1d { [[Z1:z[0-9]+]].d }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0]].d, [[Z1]].d
-; VBITS_GE_512-NEXT: st1h { [[Z0]].d }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: masked_store_trunc_v8i64i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q1, q2, [x0]
+; NO_SVE-NEXT:    ldp q3, q4, [x0, #32]
+; NO_SVE-NEXT:    cmeq v0.2d, v3.2d, v0.2d
+; NO_SVE-NEXT:    ldp q6, q7, [x1]
+; NO_SVE-NEXT:    cmeq v5.2d, v4.2d, v5.2d
+; NO_SVE-NEXT:    uzp1 v3.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v5.4s
+; NO_SVE-NEXT:    cmeq v6.2d, v1.2d, v6.2d
+; NO_SVE-NEXT:    cmeq v7.2d, v2.2d, v7.2d
+; NO_SVE-NEXT:    uzp1 v5.4s, v6.4s, v7.4s
+; NO_SVE-NEXT:    uzp1 v0.8h, v5.8h, v0.8h
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    umov w8, v0.b[1]
+; NO_SVE-NEXT:    umov w9, v0.b[0]
+; NO_SVE-NEXT:    umov w10, v0.b[2]
+; NO_SVE-NEXT:    umov w11, v0.b[3]
+; NO_SVE-NEXT:    umov w12, v0.b[4]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v0.b[5]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v0.b[6]
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v0.b[7]
+; NO_SVE-NEXT:    uzp1 v0.4s, v1.4s, v2.4s
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    bfi w9, w8, #5, #1
+; NO_SVE-NEXT:    orr w8, w9, w10, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v3.8h
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB8_9
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB8_10
+; NO_SVE-NEXT:  .LBB8_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB8_11
+; NO_SVE-NEXT:  .LBB8_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB8_12
+; NO_SVE-NEXT:  .LBB8_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB8_13
+; NO_SVE-NEXT:  .LBB8_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB8_14
+; NO_SVE-NEXT:  .LBB8_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB8_15
+; NO_SVE-NEXT:  .LBB8_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB8_16
+; NO_SVE-NEXT:  .LBB8_8: // %else14
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB8_9: // %cond.store
+; NO_SVE-NEXT:    str h0, [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB8_2
+; NO_SVE-NEXT:  .LBB8_10: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #2
+; NO_SVE-NEXT:    st1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB8_3
+; NO_SVE-NEXT:  .LBB8_11: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB8_4
+; NO_SVE-NEXT:  .LBB8_12: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #6
+; NO_SVE-NEXT:    st1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB8_5
+; NO_SVE-NEXT:  .LBB8_13: // %cond.store7
+; NO_SVE-NEXT:    add x9, x2, #8
+; NO_SVE-NEXT:    st1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB8_6
+; NO_SVE-NEXT:  .LBB8_14: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #10
+; NO_SVE-NEXT:    st1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB8_7
+; NO_SVE-NEXT:  .LBB8_15: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #12
+; NO_SVE-NEXT:    st1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB8_8
+; NO_SVE-NEXT:  .LBB8_16: // %cond.store13
+; NO_SVE-NEXT:    add x8, x2, #14
+; NO_SVE-NEXT:    st1 { v0.h }[7], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v8i64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z0.d, z2.d
+; VBITS_GE_256-NEXT:    cmpeq p0.d, p0/z, z1.d, z3.d
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    mov z2.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.d, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl8
+; VBITS_GE_256-NEXT:    mov v1.d[1], v0.d[0]
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    cmpne p0.h, p0/z, z0.h, #0
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: masked_store_trunc_v8i64i16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.d, p0/z, z0.d, z1.d
+; VBITS_GE_512-NEXT:    st1h { z0.d }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
   %a = load <8 x i64>, <8 x i64>* %ap
   %b = load <8 x i64>, <8 x i64>* %bp
   %mask = icmp eq <8 x i64> %a, %b
   %val = trunc <8 x i64> %a to <8 x i16>
   call void @llvm.masked.store.v8i16(<8 x i16> %val, <8 x i16>* %dest, i32 8, <8 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v8i64i32(<8 x i64>* %ap, <8 x i64>* %bp, <8 x i32>* %dest) #0 {
-; CHECK-LABEL: masked_store_trunc_v8i64i32:
-; VBITS_GE_512: ptrue p[[P0:[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: ld1d { [[Z0:z[0-9]+]].d }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1d { [[Z1:z[0-9]+]].d }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].d, p[[P0]]/z, [[Z0]].d, [[Z1]].d
-; VBITS_GE_512-NEXT: st1w { [[Z0]].d }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: masked_store_trunc_v8i64i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q2, q5, [x1, #32]
+; NO_SVE-NEXT:    ldp q3, q4, [x0]
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    cmeq v2.2d, v0.2d, v2.2d
+; NO_SVE-NEXT:    ldp q6, q7, [x1]
+; NO_SVE-NEXT:    cmeq v5.2d, v1.2d, v5.2d
+; NO_SVE-NEXT:    uzp1 v2.4s, v2.4s, v5.4s
+; NO_SVE-NEXT:    cmeq v6.2d, v3.2d, v6.2d
+; NO_SVE-NEXT:    cmeq v7.2d, v4.2d, v7.2d
+; NO_SVE-NEXT:    uzp1 v5.4s, v6.4s, v7.4s
+; NO_SVE-NEXT:    uzp1 v2.8h, v5.8h, v2.8h
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w9, v2.b[2]
+; NO_SVE-NEXT:    umov w10, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    umov w14, v2.b[6]
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    and w8, w12, #0x1
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    and w9, w13, #0x1
+; NO_SVE-NEXT:    bfi w10, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v2.b[7]
+; NO_SVE-NEXT:    bfi w10, w8, #4, #1
+; NO_SVE-NEXT:    and w8, w14, #0x1
+; NO_SVE-NEXT:    bfi w10, w9, #5, #1
+; NO_SVE-NEXT:    orr w8, w10, w8, lsl #6
+; NO_SVE-NEXT:    orr w9, w8, w11, lsl #7
+; NO_SVE-NEXT:    uzp1 v2.4s, v3.4s, v4.4s
+; NO_SVE-NEXT:    and w8, w9, #0xff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB9_9
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB9_10
+; NO_SVE-NEXT:  .LBB9_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB9_11
+; NO_SVE-NEXT:  .LBB9_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB9_12
+; NO_SVE-NEXT:  .LBB9_4: // %else6
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB9_13
+; NO_SVE-NEXT:  .LBB9_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB9_14
+; NO_SVE-NEXT:  .LBB9_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB9_15
+; NO_SVE-NEXT:  .LBB9_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB9_16
+; NO_SVE-NEXT:  .LBB9_8: // %else14
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB9_9: // %cond.store
+; NO_SVE-NEXT:    str s2, [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB9_2
+; NO_SVE-NEXT:  .LBB9_10: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v2.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB9_3
+; NO_SVE-NEXT:  .LBB9_11: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #8
+; NO_SVE-NEXT:    st1 { v2.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB9_4
+; NO_SVE-NEXT:  .LBB9_12: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #12
+; NO_SVE-NEXT:    st1 { v2.s }[3], [x9]
+; NO_SVE-NEXT:    uzp1 v0.4s, v0.4s, v1.4s
+; NO_SVE-NEXT:    tbz w8, #4, .LBB9_5
+; NO_SVE-NEXT:  .LBB9_13: // %cond.store7
+; NO_SVE-NEXT:    str s0, [x2, #16]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB9_6
+; NO_SVE-NEXT:  .LBB9_14: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #20
+; NO_SVE-NEXT:    st1 { v0.s }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB9_7
+; NO_SVE-NEXT:  .LBB9_15: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #24
+; NO_SVE-NEXT:    st1 { v0.s }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB9_8
+; NO_SVE-NEXT:  .LBB9_16: // %cond.store13
+; NO_SVE-NEXT:    add x8, x2, #28
+; NO_SVE-NEXT:    st1 { v0.s }[3], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v8i64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.d, p0/z, z0.d, z2.d
+; VBITS_GE_256-NEXT:    cmpeq p0.d, p0/z, z1.d, z3.d
+; VBITS_GE_256-NEXT:    mov z2.d, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.d, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ptrue p1.s, vl4
+; VBITS_GE_256-NEXT:    uzp1 z2.s, z2.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z3.s, z3.s, z3.s
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    splice z3.s, p1, z3.s, z2.s
+; VBITS_GE_256-NEXT:    uzp1 z0.s, z0.s, z0.s
+; VBITS_GE_256-NEXT:    uzp1 z1.s, z1.s, z1.s
+; VBITS_GE_256-NEXT:    cmpne p0.s, p0/z, z3.s, #0
+; VBITS_GE_256-NEXT:    splice z1.s, p1, z1.s, z0.s
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: masked_store_trunc_v8i64i32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.d, p0/z, z0.d, z1.d
+; VBITS_GE_512-NEXT:    st1w { z0.d }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
   %a = load <8 x i64>, <8 x i64>* %ap
   %b = load <8 x i64>, <8 x i64>* %bp
   %mask = icmp eq <8 x i64> %a, %b
   %val = trunc <8 x i64> %a to <8 x i32>
   call void @llvm.masked.store.v8i32(<8 x i32> %val, <8 x i32>* %dest, i32 8, <8 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v16i32i8(<16 x i32>* %ap, <16 x i32>* %bp, <16 x i8>* %dest) #0 {
-; CHECK-LABEL: masked_store_trunc_v16i32i8:
-; VBITS_GE_512: ptrue p[[P0:[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: ld1w { [[Z0:z[0-9]+]].s }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1w { [[Z1:z[0-9]+]].s }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].s, p[[P0]]/z, [[Z0]].s, [[Z1]].s
-; VBITS_GE_512-NEXT: st1b { [[Z0]].s }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: masked_store_trunc_v16i32i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q3, q2, [x1]
+; NO_SVE-NEXT:    ldp q0, q1, [x0]
+; NO_SVE-NEXT:    cmeq v3.4s, v0.4s, v3.4s
+; NO_SVE-NEXT:    cmeq v2.4s, v1.4s, v2.4s
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    uzp1 v2.8h, v3.8h, v2.8h
+; NO_SVE-NEXT:    uzp1 v0.8h, v0.8h, v1.8h
+; NO_SVE-NEXT:    xtn v2.8b, v2.8h
+; NO_SVE-NEXT:    ldp q6, q3, [x1, #32]
+; NO_SVE-NEXT:    umov w8, v2.b[1]
+; NO_SVE-NEXT:    umov w10, v2.b[2]
+; NO_SVE-NEXT:    umov w9, v2.b[0]
+; NO_SVE-NEXT:    umov w11, v2.b[3]
+; NO_SVE-NEXT:    umov w12, v2.b[4]
+; NO_SVE-NEXT:    umov w13, v2.b[5]
+; NO_SVE-NEXT:    cmeq v6.4s, v5.4s, v6.4s
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    cmeq v3.4s, v4.4s, v3.4s
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v3.8h, v6.8h, v3.8h
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v2.b[6]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v2.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    xtn v3.8b, v3.8h
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w11, v3.b[0]
+; NO_SVE-NEXT:    umov w12, v3.b[1]
+; NO_SVE-NEXT:    umov w13, v3.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v3.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v3.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v2.8h, v5.8h, v4.8h
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v3.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v3.b[6]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    umov w9, v3.b[7]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v0.16b, v0.16b, v2.16b
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w9, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB10_17
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB10_18
+; NO_SVE-NEXT:  .LBB10_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB10_19
+; NO_SVE-NEXT:  .LBB10_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB10_20
+; NO_SVE-NEXT:  .LBB10_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB10_21
+; NO_SVE-NEXT:  .LBB10_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB10_22
+; NO_SVE-NEXT:  .LBB10_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB10_23
+; NO_SVE-NEXT:  .LBB10_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB10_24
+; NO_SVE-NEXT:  .LBB10_8: // %else14
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB10_25
+; NO_SVE-NEXT:  .LBB10_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB10_26
+; NO_SVE-NEXT:  .LBB10_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB10_27
+; NO_SVE-NEXT:  .LBB10_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB10_28
+; NO_SVE-NEXT:  .LBB10_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB10_29
+; NO_SVE-NEXT:  .LBB10_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB10_30
+; NO_SVE-NEXT:  .LBB10_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB10_31
+; NO_SVE-NEXT:  .LBB10_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB10_32
+; NO_SVE-NEXT:  .LBB10_16: // %else30
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB10_17: // %cond.store
+; NO_SVE-NEXT:    st1 { v0.b }[0], [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB10_2
+; NO_SVE-NEXT:  .LBB10_18: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #1
+; NO_SVE-NEXT:    st1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB10_3
+; NO_SVE-NEXT:  .LBB10_19: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #2
+; NO_SVE-NEXT:    st1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB10_4
+; NO_SVE-NEXT:  .LBB10_20: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #3
+; NO_SVE-NEXT:    st1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB10_5
+; NO_SVE-NEXT:  .LBB10_21: // %cond.store7
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB10_6
+; NO_SVE-NEXT:  .LBB10_22: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #5
+; NO_SVE-NEXT:    st1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB10_7
+; NO_SVE-NEXT:  .LBB10_23: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #6
+; NO_SVE-NEXT:    st1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB10_8
+; NO_SVE-NEXT:  .LBB10_24: // %cond.store13
+; NO_SVE-NEXT:    add x9, x2, #7
+; NO_SVE-NEXT:    st1 { v0.b }[7], [x9]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB10_9
+; NO_SVE-NEXT:  .LBB10_25: // %cond.store15
+; NO_SVE-NEXT:    add x9, x2, #8
+; NO_SVE-NEXT:    st1 { v0.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB10_10
+; NO_SVE-NEXT:  .LBB10_26: // %cond.store17
+; NO_SVE-NEXT:    add x9, x2, #9
+; NO_SVE-NEXT:    st1 { v0.b }[9], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB10_11
+; NO_SVE-NEXT:  .LBB10_27: // %cond.store19
+; NO_SVE-NEXT:    add x9, x2, #10
+; NO_SVE-NEXT:    st1 { v0.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB10_12
+; NO_SVE-NEXT:  .LBB10_28: // %cond.store21
+; NO_SVE-NEXT:    add x9, x2, #11
+; NO_SVE-NEXT:    st1 { v0.b }[11], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB10_13
+; NO_SVE-NEXT:  .LBB10_29: // %cond.store23
+; NO_SVE-NEXT:    add x9, x2, #12
+; NO_SVE-NEXT:    st1 { v0.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB10_14
+; NO_SVE-NEXT:  .LBB10_30: // %cond.store25
+; NO_SVE-NEXT:    add x9, x2, #13
+; NO_SVE-NEXT:    st1 { v0.b }[13], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB10_15
+; NO_SVE-NEXT:  .LBB10_31: // %cond.store27
+; NO_SVE-NEXT:    add x9, x2, #14
+; NO_SVE-NEXT:    st1 { v0.b }[14], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB10_16
+; NO_SVE-NEXT:  .LBB10_32: // %cond.store29
+; NO_SVE-NEXT:    add x8, x2, #15
+; NO_SVE-NEXT:    st1 { v0.b }[15], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v16i32i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.s, p0/z, z0.s, z2.s
+; VBITS_GE_256-NEXT:    cmpeq p0.s, p0/z, z1.s, z3.s
+; VBITS_GE_256-NEXT:    mov z2.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    uzp1 z2.b, z2.b, z2.b
+; VBITS_GE_256-NEXT:    uzp1 z3.b, z3.b, z3.b
+; VBITS_GE_256-NEXT:    mov v3.d[1], v2.d[0]
+; VBITS_GE_256-NEXT:    uzp1 z0.b, z0.b, z0.b
+; VBITS_GE_256-NEXT:    uzp1 z1.b, z1.b, z1.b
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl16
+; VBITS_GE_256-NEXT:    mov v1.d[1], v0.d[0]
+; VBITS_GE_256-NEXT:    cmpne p0.b, p0/z, z3.b, #0
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: masked_store_trunc_v16i32i8:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.s, p0/z, z0.s, z1.s
+; VBITS_GE_512-NEXT:    st1b { z0.s }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
   %a = load <16 x i32>, <16 x i32>* %ap
   %b = load <16 x i32>, <16 x i32>* %bp
   %mask = icmp eq <16 x i32> %a, %b
   %val = trunc <16 x i32> %a to <16 x i8>
   call void @llvm.masked.store.v16i8(<16 x i8> %val, <16 x i8>* %dest, i32 8, <16 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v16i32i16(<16 x i32>* %ap, <16 x i32>* %bp, <16 x i16>* %dest) #0 {
-; CHECK-LABEL: masked_store_trunc_v16i32i16:
-; VBITS_GE_512: ptrue p[[P0:[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: ld1w { [[Z0:z[0-9]+]].s }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1w { [[Z1:z[0-9]+]].s }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].s, p[[P0]]/z, [[Z0]].s, [[Z1]].s
-; VBITS_GE_512-NEXT: st1h { [[Z0]].s }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: masked_store_trunc_v16i32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q1, q0, [x1]
+; NO_SVE-NEXT:    ldp q2, q3, [x0]
+; NO_SVE-NEXT:    cmeq v1.4s, v2.4s, v1.4s
+; NO_SVE-NEXT:    cmeq v4.4s, v3.4s, v0.4s
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #32]
+; NO_SVE-NEXT:    uzp1 v4.8h, v1.8h, v4.8h
+; NO_SVE-NEXT:    uzp1 v2.8h, v2.8h, v3.8h
+; NO_SVE-NEXT:    xtn v4.8b, v4.8h
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    umov w8, v4.b[1]
+; NO_SVE-NEXT:    umov w10, v4.b[2]
+; NO_SVE-NEXT:    umov w9, v4.b[0]
+; NO_SVE-NEXT:    umov w11, v4.b[3]
+; NO_SVE-NEXT:    umov w12, v4.b[4]
+; NO_SVE-NEXT:    umov w13, v4.b[5]
+; NO_SVE-NEXT:    cmeq v6.4s, v1.4s, v6.4s
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    cmeq v5.4s, v0.4s, v5.4s
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    uzp1 v5.8h, v6.8h, v5.8h
+; NO_SVE-NEXT:    bfi w9, w8, #1, #1
+; NO_SVE-NEXT:    umov w8, v4.b[6]
+; NO_SVE-NEXT:    bfi w9, w10, #2, #1
+; NO_SVE-NEXT:    umov w10, v4.b[7]
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    umov w11, v5.b[0]
+; NO_SVE-NEXT:    umov w12, v5.b[1]
+; NO_SVE-NEXT:    umov w13, v5.b[2]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w9, w8, lsl #6
+; NO_SVE-NEXT:    umov w9, v5.b[3]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #7
+; NO_SVE-NEXT:    umov w10, v5.b[4]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #8
+; NO_SVE-NEXT:    umov w11, v5.b[5]
+; NO_SVE-NEXT:    orr w8, w8, w12, lsl #9
+; NO_SVE-NEXT:    umov w12, v5.b[6]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #10
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #11
+; NO_SVE-NEXT:    umov w9, v5.b[7]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w9, w8, w9, lsl #15
+; NO_SVE-NEXT:    and w8, w9, #0xffff
+; NO_SVE-NEXT:    tbnz w9, #0, .LBB11_17
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB11_18
+; NO_SVE-NEXT:  .LBB11_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB11_19
+; NO_SVE-NEXT:  .LBB11_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB11_20
+; NO_SVE-NEXT:  .LBB11_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB11_21
+; NO_SVE-NEXT:  .LBB11_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB11_22
+; NO_SVE-NEXT:  .LBB11_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB11_23
+; NO_SVE-NEXT:  .LBB11_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB11_24
+; NO_SVE-NEXT:  .LBB11_8: // %else14
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB11_25
+; NO_SVE-NEXT:  .LBB11_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB11_26
+; NO_SVE-NEXT:  .LBB11_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB11_27
+; NO_SVE-NEXT:  .LBB11_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB11_28
+; NO_SVE-NEXT:  .LBB11_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB11_29
+; NO_SVE-NEXT:  .LBB11_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB11_30
+; NO_SVE-NEXT:  .LBB11_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB11_31
+; NO_SVE-NEXT:  .LBB11_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB11_32
+; NO_SVE-NEXT:  .LBB11_16: // %else30
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB11_17: // %cond.store
+; NO_SVE-NEXT:    str h2, [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB11_2
+; NO_SVE-NEXT:  .LBB11_18: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #2
+; NO_SVE-NEXT:    st1 { v2.h }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB11_3
+; NO_SVE-NEXT:  .LBB11_19: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v2.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB11_4
+; NO_SVE-NEXT:  .LBB11_20: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #6
+; NO_SVE-NEXT:    st1 { v2.h }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB11_5
+; NO_SVE-NEXT:  .LBB11_21: // %cond.store7
+; NO_SVE-NEXT:    add x9, x2, #8
+; NO_SVE-NEXT:    st1 { v2.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB11_6
+; NO_SVE-NEXT:  .LBB11_22: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #10
+; NO_SVE-NEXT:    st1 { v2.h }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB11_7
+; NO_SVE-NEXT:  .LBB11_23: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #12
+; NO_SVE-NEXT:    st1 { v2.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB11_8
+; NO_SVE-NEXT:  .LBB11_24: // %cond.store13
+; NO_SVE-NEXT:    add x9, x2, #14
+; NO_SVE-NEXT:    st1 { v2.h }[7], [x9]
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    tbz w8, #8, .LBB11_9
+; NO_SVE-NEXT:  .LBB11_25: // %cond.store15
+; NO_SVE-NEXT:    str h0, [x2, #16]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB11_10
+; NO_SVE-NEXT:  .LBB11_26: // %cond.store17
+; NO_SVE-NEXT:    add x9, x2, #18
+; NO_SVE-NEXT:    st1 { v0.h }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB11_11
+; NO_SVE-NEXT:  .LBB11_27: // %cond.store19
+; NO_SVE-NEXT:    add x9, x2, #20
+; NO_SVE-NEXT:    st1 { v0.h }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB11_12
+; NO_SVE-NEXT:  .LBB11_28: // %cond.store21
+; NO_SVE-NEXT:    add x9, x2, #22
+; NO_SVE-NEXT:    st1 { v0.h }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB11_13
+; NO_SVE-NEXT:  .LBB11_29: // %cond.store23
+; NO_SVE-NEXT:    add x9, x2, #24
+; NO_SVE-NEXT:    st1 { v0.h }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB11_14
+; NO_SVE-NEXT:  .LBB11_30: // %cond.store25
+; NO_SVE-NEXT:    add x9, x2, #26
+; NO_SVE-NEXT:    st1 { v0.h }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB11_15
+; NO_SVE-NEXT:  .LBB11_31: // %cond.store27
+; NO_SVE-NEXT:    add x9, x2, #28
+; NO_SVE-NEXT:    st1 { v0.h }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB11_16
+; NO_SVE-NEXT:  .LBB11_32: // %cond.store29
+; NO_SVE-NEXT:    add x8, x2, #30
+; NO_SVE-NEXT:    st1 { v0.h }[7], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v16i32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.s, p0/z, z0.s, z2.s
+; VBITS_GE_256-NEXT:    cmpeq p0.s, p0/z, z1.s, z3.s
+; VBITS_GE_256-NEXT:    mov z2.s, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.s, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    uzp1 z2.h, z2.h, z2.h
+; VBITS_GE_256-NEXT:    uzp1 z3.h, z3.h, z3.h
+; VBITS_GE_256-NEXT:    uzp1 z2.b, z2.b, z2.b
+; VBITS_GE_256-NEXT:    uzp1 z3.b, z3.b, z3.b
+; VBITS_GE_256-NEXT:    mov v3.d[1], v2.d[0]
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    uzp1 z0.h, z0.h, z0.h
+; VBITS_GE_256-NEXT:    uzp1 z1.h, z1.h, z1.h
+; VBITS_GE_256-NEXT:    ptrue p1.h, vl8
+; VBITS_GE_256-NEXT:    splice z1.h, p1, z1.h, z0.h
+; VBITS_GE_256-NEXT:    sunpklo z2.h, z3.b
+; VBITS_GE_256-NEXT:    cmpne p0.h, p0/z, z2.h, #0
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: masked_store_trunc_v16i32i16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.s, p0/z, z0.s, z1.s
+; VBITS_GE_512-NEXT:    st1h { z0.s }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
   %a = load <16 x i32>, <16 x i32>* %ap
   %b = load <16 x i32>, <16 x i32>* %bp
   %mask = icmp eq <16 x i32> %a, %b
   %val = trunc <16 x i32> %a to <16 x i16>
   call void @llvm.masked.store.v16i16(<16 x i16> %val, <16 x i16>* %dest, i32 8, <16 x i1> %mask)
   ret void
 }
 
 define void @masked_store_trunc_v32i16i8(<32 x i16>* %ap, <32 x i16>* %bp, <32 x i8>* %dest) #0 {
-; CHECK-LABEL: masked_store_trunc_v32i16i8:
-; VBITS_GE_512: ptrue p[[P0:[0-9]+]].h, vl32
-; VBITS_GE_512-NEXT: ld1h { [[Z0:z[0-9]+]].h }, p0/z, [x0]
-; VBITS_GE_512-NEXT: ld1h { [[Z1:z[0-9]+]].h }, p0/z, [x1]
-; VBITS_GE_512-NEXT: cmpeq p[[P1:[0-9]+]].h, p[[P0]]/z, [[Z0]].h, [[Z1]].h
-; VBITS_GE_512-NEXT: st1b { [[Z0]].h }, p[[P1]], [x{{[0-9]+}}]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: masked_store_trunc_v32i16i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    sub sp, sp, #16
+; NO_SVE-NEXT:    .cfi_def_cfa_offset 16
+; NO_SVE-NEXT:    ldp q0, q1, [x0, #32]
+; NO_SVE-NEXT:    ldp q2, q4, [x1, #32]
+; NO_SVE-NEXT:    cmeq v5.8h, v0.8h, v2.8h
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    cmeq v4.8h, v1.8h, v4.8h
+; NO_SVE-NEXT:    umov w8, v5.b[1]
+; NO_SVE-NEXT:    umov w9, v5.b[2]
+; NO_SVE-NEXT:    umov w10, v5.b[0]
+; NO_SVE-NEXT:    umov w11, v5.b[3]
+; NO_SVE-NEXT:    umov w12, v5.b[4]
+; NO_SVE-NEXT:    umov w13, v5.b[5]
+; NO_SVE-NEXT:    xtn v4.8b, v4.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[6]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v5.b[7]
+; NO_SVE-NEXT:    bfi w10, w8, #1, #1
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w16, v4.b[0]
+; NO_SVE-NEXT:    bfi w10, w9, #2, #1
+; NO_SVE-NEXT:    and w13, w13, #0x1
+; NO_SVE-NEXT:    umov w8, v4.b[1]
+; NO_SVE-NEXT:    ldp q7, q6, [x1]
+; NO_SVE-NEXT:    bfi w10, w11, #3, #1
+; NO_SVE-NEXT:    umov w9, v4.b[2]
+; NO_SVE-NEXT:    and w14, w14, #0x1
+; NO_SVE-NEXT:    bfi w10, w12, #4, #1
+; NO_SVE-NEXT:    umov w11, v4.b[3]
+; NO_SVE-NEXT:    and w15, w15, #0x1
+; NO_SVE-NEXT:    cmeq v5.8h, v3.8h, v7.8h
+; NO_SVE-NEXT:    bfi w10, w13, #5, #1
+; NO_SVE-NEXT:    and w16, w16, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w14, lsl #6
+; NO_SVE-NEXT:    xtn v5.8b, v5.8h
+; NO_SVE-NEXT:    and w8, w8, #0x1
+; NO_SVE-NEXT:    umov w12, v4.b[4]
+; NO_SVE-NEXT:    orr w10, w10, w15, lsl #7
+; NO_SVE-NEXT:    umov w13, v5.b[1]
+; NO_SVE-NEXT:    umov w14, v5.b[2]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w10, w10, w16, lsl #8
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    orr w8, w10, w8, lsl #9
+; NO_SVE-NEXT:    orr w8, w8, w9, lsl #10
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    umov w9, v5.b[0]
+; NO_SVE-NEXT:    orr w8, w8, w11, lsl #11
+; NO_SVE-NEXT:    umov w11, v4.b[5]
+; NO_SVE-NEXT:    and w12, w13, #0x1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[3]
+; NO_SVE-NEXT:    umov w15, v5.b[4]
+; NO_SVE-NEXT:    umov w16, v5.b[5]
+; NO_SVE-NEXT:    and w9, w9, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w11, #0x1
+; NO_SVE-NEXT:    bfi w9, w12, #1, #1
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[6]
+; NO_SVE-NEXT:    and w12, w15, #0x1
+; NO_SVE-NEXT:    bfi w9, w13, #2, #1
+; NO_SVE-NEXT:    cmeq v6.8h, v2.8h, v6.8h
+; NO_SVE-NEXT:    and w13, w16, #0x1
+; NO_SVE-NEXT:    bfi w9, w11, #3, #1
+; NO_SVE-NEXT:    umov w11, v5.b[7]
+; NO_SVE-NEXT:    xtn v5.8b, v6.8h
+; NO_SVE-NEXT:    bfi w9, w12, #4, #1
+; NO_SVE-NEXT:    umov w12, v4.b[6]
+; NO_SVE-NEXT:    bfi w9, w13, #5, #1
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[0]
+; NO_SVE-NEXT:    and w11, w11, #0x1
+; NO_SVE-NEXT:    umov w15, v5.b[1]
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #13
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #6
+; NO_SVE-NEXT:    and w10, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[2]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #7
+; NO_SVE-NEXT:    and w11, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[3]
+; NO_SVE-NEXT:    and w13, w15, #0x1
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    umov w10, v5.b[4]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #8
+; NO_SVE-NEXT:    and w11, w12, #0x1
+; NO_SVE-NEXT:    umov w12, v5.b[5]
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #9
+; NO_SVE-NEXT:    and w13, w14, #0x1
+; NO_SVE-NEXT:    umov w14, v5.b[6]
+; NO_SVE-NEXT:    orr w9, w9, w11, lsl #10
+; NO_SVE-NEXT:    umov w11, v4.b[7]
+; NO_SVE-NEXT:    and w10, w10, #0x1
+; NO_SVE-NEXT:    orr w9, w9, w13, lsl #11
+; NO_SVE-NEXT:    and w12, w12, #0x1
+; NO_SVE-NEXT:    umov w13, v5.b[7]
+; NO_SVE-NEXT:    orr w9, w9, w10, lsl #12
+; NO_SVE-NEXT:    and w10, w14, #0x1
+; NO_SVE-NEXT:    orr w11, w8, w11, lsl #15
+; NO_SVE-NEXT:    orr w8, w9, w12, lsl #13
+; NO_SVE-NEXT:    orr w8, w8, w10, lsl #14
+; NO_SVE-NEXT:    orr w8, w8, w13, lsl #15
+; NO_SVE-NEXT:    uzp1 v2.16b, v3.16b, v2.16b
+; NO_SVE-NEXT:    bfi w8, w11, #16, #16
+; NO_SVE-NEXT:    tbnz w8, #0, .LBB12_33
+; NO_SVE-NEXT:  // %bb.1: // %else
+; NO_SVE-NEXT:    tbnz w8, #1, .LBB12_34
+; NO_SVE-NEXT:  .LBB12_2: // %else2
+; NO_SVE-NEXT:    tbnz w8, #2, .LBB12_35
+; NO_SVE-NEXT:  .LBB12_3: // %else4
+; NO_SVE-NEXT:    tbnz w8, #3, .LBB12_36
+; NO_SVE-NEXT:  .LBB12_4: // %else6
+; NO_SVE-NEXT:    tbnz w8, #4, .LBB12_37
+; NO_SVE-NEXT:  .LBB12_5: // %else8
+; NO_SVE-NEXT:    tbnz w8, #5, .LBB12_38
+; NO_SVE-NEXT:  .LBB12_6: // %else10
+; NO_SVE-NEXT:    tbnz w8, #6, .LBB12_39
+; NO_SVE-NEXT:  .LBB12_7: // %else12
+; NO_SVE-NEXT:    tbnz w8, #7, .LBB12_40
+; NO_SVE-NEXT:  .LBB12_8: // %else14
+; NO_SVE-NEXT:    tbnz w8, #8, .LBB12_41
+; NO_SVE-NEXT:  .LBB12_9: // %else16
+; NO_SVE-NEXT:    tbnz w8, #9, .LBB12_42
+; NO_SVE-NEXT:  .LBB12_10: // %else18
+; NO_SVE-NEXT:    tbnz w8, #10, .LBB12_43
+; NO_SVE-NEXT:  .LBB12_11: // %else20
+; NO_SVE-NEXT:    tbnz w8, #11, .LBB12_44
+; NO_SVE-NEXT:  .LBB12_12: // %else22
+; NO_SVE-NEXT:    tbnz w8, #12, .LBB12_45
+; NO_SVE-NEXT:  .LBB12_13: // %else24
+; NO_SVE-NEXT:    tbnz w8, #13, .LBB12_46
+; NO_SVE-NEXT:  .LBB12_14: // %else26
+; NO_SVE-NEXT:    tbnz w8, #14, .LBB12_47
+; NO_SVE-NEXT:  .LBB12_15: // %else28
+; NO_SVE-NEXT:    tbnz w8, #15, .LBB12_48
+; NO_SVE-NEXT:  .LBB12_16: // %else30
+; NO_SVE-NEXT:    uzp1 v0.16b, v0.16b, v1.16b
+; NO_SVE-NEXT:    tbnz w8, #16, .LBB12_49
+; NO_SVE-NEXT:  .LBB12_17: // %else32
+; NO_SVE-NEXT:    tbnz w8, #17, .LBB12_50
+; NO_SVE-NEXT:  .LBB12_18: // %else34
+; NO_SVE-NEXT:    tbnz w8, #18, .LBB12_51
+; NO_SVE-NEXT:  .LBB12_19: // %else36
+; NO_SVE-NEXT:    tbnz w8, #19, .LBB12_52
+; NO_SVE-NEXT:  .LBB12_20: // %else38
+; NO_SVE-NEXT:    tbnz w8, #20, .LBB12_53
+; NO_SVE-NEXT:  .LBB12_21: // %else40
+; NO_SVE-NEXT:    tbnz w8, #21, .LBB12_54
+; NO_SVE-NEXT:  .LBB12_22: // %else42
+; NO_SVE-NEXT:    tbnz w8, #22, .LBB12_55
+; NO_SVE-NEXT:  .LBB12_23: // %else44
+; NO_SVE-NEXT:    tbnz w8, #23, .LBB12_56
+; NO_SVE-NEXT:  .LBB12_24: // %else46
+; NO_SVE-NEXT:    tbnz w8, #24, .LBB12_57
+; NO_SVE-NEXT:  .LBB12_25: // %else48
+; NO_SVE-NEXT:    tbnz w8, #25, .LBB12_58
+; NO_SVE-NEXT:  .LBB12_26: // %else50
+; NO_SVE-NEXT:    tbnz w8, #26, .LBB12_59
+; NO_SVE-NEXT:  .LBB12_27: // %else52
+; NO_SVE-NEXT:    tbnz w8, #27, .LBB12_60
+; NO_SVE-NEXT:  .LBB12_28: // %else54
+; NO_SVE-NEXT:    tbnz w8, #28, .LBB12_61
+; NO_SVE-NEXT:  .LBB12_29: // %else56
+; NO_SVE-NEXT:    tbnz w8, #29, .LBB12_62
+; NO_SVE-NEXT:  .LBB12_30: // %else58
+; NO_SVE-NEXT:    tbnz w8, #30, .LBB12_63
+; NO_SVE-NEXT:  .LBB12_31: // %else60
+; NO_SVE-NEXT:    tbnz w8, #31, .LBB12_64
+; NO_SVE-NEXT:  .LBB12_32: // %else62
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+; NO_SVE-NEXT:  .LBB12_33: // %cond.store
+; NO_SVE-NEXT:    st1 { v2.b }[0], [x2]
+; NO_SVE-NEXT:    tbz w8, #1, .LBB12_2
+; NO_SVE-NEXT:  .LBB12_34: // %cond.store1
+; NO_SVE-NEXT:    add x9, x2, #1
+; NO_SVE-NEXT:    st1 { v2.b }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #2, .LBB12_3
+; NO_SVE-NEXT:  .LBB12_35: // %cond.store3
+; NO_SVE-NEXT:    add x9, x2, #2
+; NO_SVE-NEXT:    st1 { v2.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #3, .LBB12_4
+; NO_SVE-NEXT:  .LBB12_36: // %cond.store5
+; NO_SVE-NEXT:    add x9, x2, #3
+; NO_SVE-NEXT:    st1 { v2.b }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #4, .LBB12_5
+; NO_SVE-NEXT:  .LBB12_37: // %cond.store7
+; NO_SVE-NEXT:    add x9, x2, #4
+; NO_SVE-NEXT:    st1 { v2.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #5, .LBB12_6
+; NO_SVE-NEXT:  .LBB12_38: // %cond.store9
+; NO_SVE-NEXT:    add x9, x2, #5
+; NO_SVE-NEXT:    st1 { v2.b }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #6, .LBB12_7
+; NO_SVE-NEXT:  .LBB12_39: // %cond.store11
+; NO_SVE-NEXT:    add x9, x2, #6
+; NO_SVE-NEXT:    st1 { v2.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #7, .LBB12_8
+; NO_SVE-NEXT:  .LBB12_40: // %cond.store13
+; NO_SVE-NEXT:    add x9, x2, #7
+; NO_SVE-NEXT:    st1 { v2.b }[7], [x9]
+; NO_SVE-NEXT:    tbz w8, #8, .LBB12_9
+; NO_SVE-NEXT:  .LBB12_41: // %cond.store15
+; NO_SVE-NEXT:    add x9, x2, #8
+; NO_SVE-NEXT:    st1 { v2.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #9, .LBB12_10
+; NO_SVE-NEXT:  .LBB12_42: // %cond.store17
+; NO_SVE-NEXT:    add x9, x2, #9
+; NO_SVE-NEXT:    st1 { v2.b }[9], [x9]
+; NO_SVE-NEXT:    tbz w8, #10, .LBB12_11
+; NO_SVE-NEXT:  .LBB12_43: // %cond.store19
+; NO_SVE-NEXT:    add x9, x2, #10
+; NO_SVE-NEXT:    st1 { v2.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #11, .LBB12_12
+; NO_SVE-NEXT:  .LBB12_44: // %cond.store21
+; NO_SVE-NEXT:    add x9, x2, #11
+; NO_SVE-NEXT:    st1 { v2.b }[11], [x9]
+; NO_SVE-NEXT:    tbz w8, #12, .LBB12_13
+; NO_SVE-NEXT:  .LBB12_45: // %cond.store23
+; NO_SVE-NEXT:    add x9, x2, #12
+; NO_SVE-NEXT:    st1 { v2.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #13, .LBB12_14
+; NO_SVE-NEXT:  .LBB12_46: // %cond.store25
+; NO_SVE-NEXT:    add x9, x2, #13
+; NO_SVE-NEXT:    st1 { v2.b }[13], [x9]
+; NO_SVE-NEXT:    tbz w8, #14, .LBB12_15
+; NO_SVE-NEXT:  .LBB12_47: // %cond.store27
+; NO_SVE-NEXT:    add x9, x2, #14
+; NO_SVE-NEXT:    st1 { v2.b }[14], [x9]
+; NO_SVE-NEXT:    tbz w8, #15, .LBB12_16
+; NO_SVE-NEXT:  .LBB12_48: // %cond.store29
+; NO_SVE-NEXT:    add x9, x2, #15
+; NO_SVE-NEXT:    st1 { v2.b }[15], [x9]
+; NO_SVE-NEXT:    uzp1 v0.16b, v0.16b, v1.16b
+; NO_SVE-NEXT:    tbz w8, #16, .LBB12_17
+; NO_SVE-NEXT:  .LBB12_49: // %cond.store31
+; NO_SVE-NEXT:    add x9, x2, #16
+; NO_SVE-NEXT:    st1 { v0.b }[0], [x9]
+; NO_SVE-NEXT:    tbz w8, #17, .LBB12_18
+; NO_SVE-NEXT:  .LBB12_50: // %cond.store33
+; NO_SVE-NEXT:    add x9, x2, #17
+; NO_SVE-NEXT:    st1 { v0.b }[1], [x9]
+; NO_SVE-NEXT:    tbz w8, #18, .LBB12_19
+; NO_SVE-NEXT:  .LBB12_51: // %cond.store35
+; NO_SVE-NEXT:    add x9, x2, #18
+; NO_SVE-NEXT:    st1 { v0.b }[2], [x9]
+; NO_SVE-NEXT:    tbz w8, #19, .LBB12_20
+; NO_SVE-NEXT:  .LBB12_52: // %cond.store37
+; NO_SVE-NEXT:    add x9, x2, #19
+; NO_SVE-NEXT:    st1 { v0.b }[3], [x9]
+; NO_SVE-NEXT:    tbz w8, #20, .LBB12_21
+; NO_SVE-NEXT:  .LBB12_53: // %cond.store39
+; NO_SVE-NEXT:    add x9, x2, #20
+; NO_SVE-NEXT:    st1 { v0.b }[4], [x9]
+; NO_SVE-NEXT:    tbz w8, #21, .LBB12_22
+; NO_SVE-NEXT:  .LBB12_54: // %cond.store41
+; NO_SVE-NEXT:    add x9, x2, #21
+; NO_SVE-NEXT:    st1 { v0.b }[5], [x9]
+; NO_SVE-NEXT:    tbz w8, #22, .LBB12_23
+; NO_SVE-NEXT:  .LBB12_55: // %cond.store43
+; NO_SVE-NEXT:    add x9, x2, #22
+; NO_SVE-NEXT:    st1 { v0.b }[6], [x9]
+; NO_SVE-NEXT:    tbz w8, #23, .LBB12_24
+; NO_SVE-NEXT:  .LBB12_56: // %cond.store45
+; NO_SVE-NEXT:    add x9, x2, #23
+; NO_SVE-NEXT:    st1 { v0.b }[7], [x9]
+; NO_SVE-NEXT:    tbz w8, #24, .LBB12_25
+; NO_SVE-NEXT:  .LBB12_57: // %cond.store47
+; NO_SVE-NEXT:    add x9, x2, #24
+; NO_SVE-NEXT:    st1 { v0.b }[8], [x9]
+; NO_SVE-NEXT:    tbz w8, #25, .LBB12_26
+; NO_SVE-NEXT:  .LBB12_58: // %cond.store49
+; NO_SVE-NEXT:    add x9, x2, #25
+; NO_SVE-NEXT:    st1 { v0.b }[9], [x9]
+; NO_SVE-NEXT:    tbz w8, #26, .LBB12_27
+; NO_SVE-NEXT:  .LBB12_59: // %cond.store51
+; NO_SVE-NEXT:    add x9, x2, #26
+; NO_SVE-NEXT:    st1 { v0.b }[10], [x9]
+; NO_SVE-NEXT:    tbz w8, #27, .LBB12_28
+; NO_SVE-NEXT:  .LBB12_60: // %cond.store53
+; NO_SVE-NEXT:    add x9, x2, #27
+; NO_SVE-NEXT:    st1 { v0.b }[11], [x9]
+; NO_SVE-NEXT:    tbz w8, #28, .LBB12_29
+; NO_SVE-NEXT:  .LBB12_61: // %cond.store55
+; NO_SVE-NEXT:    add x9, x2, #28
+; NO_SVE-NEXT:    st1 { v0.b }[12], [x9]
+; NO_SVE-NEXT:    tbz w8, #29, .LBB12_30
+; NO_SVE-NEXT:  .LBB12_62: // %cond.store57
+; NO_SVE-NEXT:    add x9, x2, #29
+; NO_SVE-NEXT:    st1 { v0.b }[13], [x9]
+; NO_SVE-NEXT:    tbz w8, #30, .LBB12_31
+; NO_SVE-NEXT:  .LBB12_63: // %cond.store59
+; NO_SVE-NEXT:    add x9, x2, #30
+; NO_SVE-NEXT:    st1 { v0.b }[14], [x9]
+; NO_SVE-NEXT:    tbz w8, #31, .LBB12_32
+; NO_SVE-NEXT:  .LBB12_64: // %cond.store61
+; NO_SVE-NEXT:    add x8, x2, #31
+; NO_SVE-NEXT:    st1 { v0.b }[15], [x8]
+; NO_SVE-NEXT:    add sp, sp, #16
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: masked_store_trunc_v32i16i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    cmpeq p1.h, p0/z, z0.h, z2.h
+; VBITS_GE_256-NEXT:    cmpeq p0.h, p0/z, z1.h, z3.h
+; VBITS_GE_256-NEXT:    mov z2.h, p1/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    mov z3.h, p0/z, #-1 // =0xffffffffffffffff
+; VBITS_GE_256-NEXT:    ptrue p1.b, vl16
+; VBITS_GE_256-NEXT:    uzp1 z2.b, z2.b, z2.b
+; VBITS_GE_256-NEXT:    uzp1 z3.b, z3.b, z3.b
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    splice z3.b, p1, z3.b, z2.b
+; VBITS_GE_256-NEXT:    uzp1 z0.b, z0.b, z0.b
+; VBITS_GE_256-NEXT:    uzp1 z1.b, z1.b, z1.b
+; VBITS_GE_256-NEXT:    cmpne p0.b, p0/z, z3.b, #0
+; VBITS_GE_256-NEXT:    splice z1.b, p1, z1.b, z0.b
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: masked_store_trunc_v32i16i8:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    ld1h { z1.h }, p0/z, [x1]
+; VBITS_GE_512-NEXT:    cmpeq p0.h, p0/z, z0.h, z1.h
+; VBITS_GE_512-NEXT:    st1b { z0.h }, p0, [x2]
+; VBITS_GE_512-NEXT:    ret
   %a = load <32 x i16>, <32 x i16>* %ap
   %b = load <32 x i16>, <32 x i16>* %bp
   %mask = icmp eq <32 x i16> %a, %b
   %val = trunc <32 x i16> %a to <32 x i8>
   call void @llvm.masked.store.v32i8(<32 x i8> %val, <32 x i8>* %dest, i32 8, <32 x i1> %mask)
   ret void
 }
 

llvm/test/CodeGen/AArch64/sve-fixed-length-sdiv-pow2.ll

-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024

 ; CHECK-NEXT:    ret
   %op1 = load <32 x i8>, <32 x i8>* %a
   %res = sdiv <32 x i8> %op1, shufflevector (<32 x i8> insertelement (<32 x i8> poison, i8 32, i32 0), <32 x i8> poison, <32 x i32> zeroinitializer)
   store <32 x i8> %res, <32 x i8>* %a
   ret void
 }
 
 define void @sdiv_v64i8(<64 x i8>* %a) #0 {
-; VBITS_EQ_256-LABEL: sdiv_v64i8:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov w8, #32
-; VBITS_EQ_256-NEXT:    ptrue p0.b, vl32
-; VBITS_EQ_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
-; VBITS_EQ_256-NEXT:    ld1b { z1.b }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    asrd z0.b, p0/m, z0.b, #5
-; VBITS_EQ_256-NEXT:    asrd z1.b, p0/m, z1.b, #5
-; VBITS_EQ_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
-; VBITS_EQ_256-NEXT:    st1b { z1.b }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; VBITS_GE_256-LABEL: sdiv_v64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z0.b, p0/m, z0.b, #5
+; VBITS_GE_256-NEXT:    asrd z1.b, p0/m, z1.b, #5
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: sdiv_v64i8:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.b, vl64
 ; VBITS_GE_512-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    asrd z0.b, p0/m, z0.b, #5
 ; VBITS_GE_512-NEXT:    st1b { z0.b }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <64 x i8>, <64 x i8>* %a
   %res = sdiv <64 x i8> %op1, shufflevector (<64 x i8> insertelement (<64 x i8> poison, i8 32, i32 0), <64 x i8> poison, <64 x i32> zeroinitializer)
   store <64 x i8> %res, <64 x i8>* %a
   ret void
 }
 
 define void @sdiv_v128i8(<128 x i8>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v128i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #96
+; VBITS_GE_256-NEXT:    mov w9, #32
+; VBITS_GE_256-NEXT:    mov w10, #64
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x0, x10]
+; VBITS_GE_256-NEXT:    ld1b { z3.b }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.b, p0/m, z1.b, #5
+; VBITS_GE_256-NEXT:    asrd z0.b, p0/m, z0.b, #5
+; VBITS_GE_256-NEXT:    asrd z2.b, p0/m, z2.b, #5
+; VBITS_GE_256-NEXT:    asrd z3.b, p0/m, z3.b, #5
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x0, x10]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x0, x9]
+; VBITS_GE_256-NEXT:    st1b { z3.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: sdiv_v128i8:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.b, vl128
 ; VBITS_GE_1024-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    asrd z0.b, p0/m, z0.b, #5
 ; VBITS_GE_1024-NEXT:    st1b { z0.b }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <128 x i8>, <128 x i8>* %a
   %res = sdiv <128 x i8> %op1, shufflevector (<128 x i8> insertelement (<128 x i8> poison, i8 32, i32 0), <128 x i8> poison, <128 x i32> zeroinitializer)
   store <128 x i8> %res, <128 x i8>* %a
   ret void
 }
 
 define void @sdiv_v256i8(<256 x i8>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v256i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #192
+; VBITS_GE_256-NEXT:    mov w9, #96
+; VBITS_GE_256-NEXT:    mov w10, #32
+; VBITS_GE_256-NEXT:    mov w11, #160
+; VBITS_GE_256-NEXT:    mov w12, #64
+; VBITS_GE_256-NEXT:    mov w13, #224
+; VBITS_GE_256-NEXT:    mov w14, #128
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x10]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x0, x11]
+; VBITS_GE_256-NEXT:    ld1b { z3.b }, p0/z, [x0, x12]
+; VBITS_GE_256-NEXT:    ld1b { z4.b }, p0/z, [x0, x13]
+; VBITS_GE_256-NEXT:    ld1b { z5.b }, p0/z, [x0, x14]
+; VBITS_GE_256-NEXT:    ld1b { z6.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z7.b }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.b, p0/m, z1.b, #5
+; VBITS_GE_256-NEXT:    asrd z0.b, p0/m, z0.b, #5
+; VBITS_GE_256-NEXT:    asrd z3.b, p0/m, z3.b, #5
+; VBITS_GE_256-NEXT:    asrd z2.b, p0/m, z2.b, #5
+; VBITS_GE_256-NEXT:    asrd z5.b, p0/m, z5.b, #5
+; VBITS_GE_256-NEXT:    asrd z4.b, p0/m, z4.b, #5
+; VBITS_GE_256-NEXT:    asrd z6.b, p0/m, z6.b, #5
+; VBITS_GE_256-NEXT:    asrd z7.b, p0/m, z7.b, #5
+; VBITS_GE_256-NEXT:    st1b { z6.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    st1b { z4.b }, p0, [x0, x13]
+; VBITS_GE_256-NEXT:    st1b { z5.b }, p0, [x0, x14]
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x0, x11]
+; VBITS_GE_256-NEXT:    st1b { z3.b }, p0, [x0, x12]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x9]
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x0, x10]
+; VBITS_GE_256-NEXT:    st1b { z7.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: sdiv_v256i8:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.b, vl256
 ; VBITS_GE_2048-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    asrd z0.b, p0/m, z0.b, #5
 ; VBITS_GE_2048-NEXT:    st1b { z0.b }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <256 x i8>, <256 x i8>* %a

 ; CHECK-NEXT:    ret
   %op1 = load <16 x i16>, <16 x i16>* %a
   %res = sdiv <16 x i16> %op1, shufflevector (<16 x i16> insertelement (<16 x i16> poison, i16 32, i32 0), <16 x i16> poison, <16 x i32> zeroinitializer)
   store <16 x i16> %res, <16 x i16>* %a
   ret void
 }
 
 define void @sdiv_v32i16(<32 x i16>* %a) #0 {
-; VBITS_EQ_256-LABEL: sdiv_v32i16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    asrd z0.h, p0/m, z0.h, #5
-; VBITS_EQ_256-NEXT:    asrd z1.h, p0/m, z1.h, #5
-; VBITS_EQ_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    st1h { z1.h }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; VBITS_GE_256-LABEL: sdiv_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z0.h, p0/m, z0.h, #5
+; VBITS_GE_256-NEXT:    asrd z1.h, p0/m, z1.h, #5
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: sdiv_v32i16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    asrd z0.h, p0/m, z0.h, #5
 ; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <32 x i16>, <32 x i16>* %a
   %res = sdiv <32 x i16> %op1, shufflevector (<32 x i16> insertelement (<32 x i16> poison, i16 32, i32 0), <32 x i16> poison, <32 x i32> zeroinitializer)
   store <32 x i16> %res, <32 x i16>* %a
   ret void
 }
 
 define void @sdiv_v64i16(<64 x i16>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #32
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.h, p0/m, z1.h, #5
+; VBITS_GE_256-NEXT:    asrd z0.h, p0/m, z0.h, #5
+; VBITS_GE_256-NEXT:    asrd z2.h, p0/m, z2.h, #5
+; VBITS_GE_256-NEXT:    asrd z3.h, p0/m, z3.h, #5
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: sdiv_v64i16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    asrd z0.h, p0/m, z0.h, #5
 ; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <64 x i16>, <64 x i16>* %a
   %res = sdiv <64 x i16> %op1, shufflevector (<64 x i16> insertelement (<64 x i16> poison, i16 32, i32 0), <64 x i16> poison, <64 x i32> zeroinitializer)
   store <64 x i16> %res, <64 x i16>* %a
   ret void
 }
 
 define void @sdiv_v128i16(<128 x i16>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v128i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #96
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x11, #80
+; VBITS_GE_256-NEXT:    mov x12, #32
+; VBITS_GE_256-NEXT:    mov x13, #112
+; VBITS_GE_256-NEXT:    mov x14, #64
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x0, x13, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z5.h }, p0/z, [x0, x14, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z6.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z7.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.h, p0/m, z1.h, #5
+; VBITS_GE_256-NEXT:    asrd z0.h, p0/m, z0.h, #5
+; VBITS_GE_256-NEXT:    asrd z3.h, p0/m, z3.h, #5
+; VBITS_GE_256-NEXT:    asrd z2.h, p0/m, z2.h, #5
+; VBITS_GE_256-NEXT:    asrd z5.h, p0/m, z5.h, #5
+; VBITS_GE_256-NEXT:    asrd z4.h, p0/m, z4.h, #5
+; VBITS_GE_256-NEXT:    asrd z6.h, p0/m, z6.h, #5
+; VBITS_GE_256-NEXT:    asrd z7.h, p0/m, z7.h, #5
+; VBITS_GE_256-NEXT:    st1h { z6.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0, x13, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z5.h }, p0, [x0, x14, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: sdiv_v128i16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    asrd z0.h, p0/m, z0.h, #5
 ; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <128 x i16>, <128 x i16>* %a

 ; CHECK-NEXT:    ret
   %op1 = load <8 x i32>, <8 x i32>* %a
   %res = sdiv <8 x i32> %op1, shufflevector (<8 x i32> insertelement (<8 x i32> poison, i32 32, i32 0), <8 x i32> poison, <8 x i32> zeroinitializer)
   store <8 x i32> %res, <8 x i32>* %a
   ret void
 }
 
 define void @sdiv_v16i32(<16 x i32>* %a) #0 {
-; VBITS_EQ_256-LABEL: sdiv_v16i32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    asrd z0.s, p0/m, z0.s, #5
-; VBITS_EQ_256-NEXT:    asrd z1.s, p0/m, z1.s, #5
-; VBITS_EQ_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    st1w { z1.s }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; VBITS_GE_256-LABEL: sdiv_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z0.s, p0/m, z0.s, #5
+; VBITS_GE_256-NEXT:    asrd z1.s, p0/m, z1.s, #5
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: sdiv_v16i32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    asrd z0.s, p0/m, z0.s, #5
 ; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <16 x i32>, <16 x i32>* %a
   %res = sdiv <16 x i32> %op1, shufflevector (<16 x i32> insertelement (<16 x i32> poison, i32 32, i32 0), <16 x i32> poison, <16 x i32> zeroinitializer)
   store <16 x i32> %res, <16 x i32>* %a
   ret void
 }
 
 define void @sdiv_v32i32(<32 x i32>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.s, p0/m, z1.s, #5
+; VBITS_GE_256-NEXT:    asrd z0.s, p0/m, z0.s, #5
+; VBITS_GE_256-NEXT:    asrd z2.s, p0/m, z2.s, #5
+; VBITS_GE_256-NEXT:    asrd z3.s, p0/m, z3.s, #5
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: sdiv_v32i32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    asrd z0.s, p0/m, z0.s, #5
 ; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <32 x i32>, <32 x i32>* %a
   %res = sdiv <32 x i32> %op1, shufflevector (<32 x i32> insertelement (<32 x i32> poison, i32 32, i32 0), <32 x i32> poison, <32 x i32> zeroinitializer)
   store <32 x i32> %res, <32 x i32>* %a
   ret void
 }
 
 define void @sdiv_v64i32(<64 x i32>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    mov x11, #40
+; VBITS_GE_256-NEXT:    mov x12, #16
+; VBITS_GE_256-NEXT:    mov x13, #56
+; VBITS_GE_256-NEXT:    mov x14, #32
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.s, p0/m, z1.s, #5
+; VBITS_GE_256-NEXT:    asrd z0.s, p0/m, z0.s, #5
+; VBITS_GE_256-NEXT:    asrd z3.s, p0/m, z3.s, #5
+; VBITS_GE_256-NEXT:    asrd z2.s, p0/m, z2.s, #5
+; VBITS_GE_256-NEXT:    asrd z5.s, p0/m, z5.s, #5
+; VBITS_GE_256-NEXT:    asrd z4.s, p0/m, z4.s, #5
+; VBITS_GE_256-NEXT:    asrd z6.s, p0/m, z6.s, #5
+; VBITS_GE_256-NEXT:    asrd z7.s, p0/m, z7.s, #5
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: sdiv_v64i32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    asrd z0.s, p0/m, z0.s, #5
 ; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <64 x i32>, <64 x i32>* %a

 ; CHECK-NEXT:    ret
   %op1 = load <4 x i64>, <4 x i64>* %a
   %res = sdiv <4 x i64> %op1, shufflevector (<4 x i64> insertelement (<4 x i64> poison, i64 32, i32 0), <4 x i64> poison, <4 x i32> zeroinitializer)
   store <4 x i64> %res, <4 x i64>* %a
   ret void
 }
 
 define void @sdiv_v8i64(<8 x i64>* %a) #0 {
-; VBITS_EQ_256-LABEL: sdiv_v8i64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
-; VBITS_EQ_256-NEXT:    asrd z0.d, p0/m, z0.d, #5
-; VBITS_EQ_256-NEXT:    asrd z1.d, p0/m, z1.d, #5
-; VBITS_EQ_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    st1d { z1.d }, p0, [x0]
-; VBITS_EQ_256-NEXT:    ret
+; VBITS_GE_256-LABEL: sdiv_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z0.d, p0/m, z0.d, #5
+; VBITS_GE_256-NEXT:    asrd z1.d, p0/m, z1.d, #5
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: sdiv_v8i64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    asrd z0.d, p0/m, z0.d, #5
 ; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <8 x i64>, <8 x i64>* %a
   %res = sdiv <8 x i64> %op1, shufflevector (<8 x i64> insertelement (<8 x i64> poison, i64 32, i32 0), <8 x i64> poison, <8 x i32> zeroinitializer)
   store <8 x i64> %res, <8 x i64>* %a
   ret void
 }
 
 define void @sdiv_v16i64(<16 x i64>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.d, p0/m, z1.d, #5
+; VBITS_GE_256-NEXT:    asrd z0.d, p0/m, z0.d, #5
+; VBITS_GE_256-NEXT:    asrd z2.d, p0/m, z2.d, #5
+; VBITS_GE_256-NEXT:    asrd z3.d, p0/m, z3.d, #5
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: sdiv_v16i64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    asrd z0.d, p0/m, z0.d, #5
 ; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <16 x i64>, <16 x i64>* %a
   %res = sdiv <16 x i64> %op1, shufflevector (<16 x i64> insertelement (<16 x i64> poison, i64 32, i32 0), <16 x i64> poison, <16 x i32> zeroinitializer)
   store <16 x i64> %res, <16 x i64>* %a
   ret void
 }
 
 define void @sdiv_v32i64(<32 x i64>* %a) #0 {
+; VBITS_GE_256-LABEL: sdiv_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    mov x12, #8
+; VBITS_GE_256-NEXT:    mov x13, #28
+; VBITS_GE_256-NEXT:    mov x14, #16
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    asrd z1.d, p0/m, z1.d, #5
+; VBITS_GE_256-NEXT:    asrd z0.d, p0/m, z0.d, #5
+; VBITS_GE_256-NEXT:    asrd z3.d, p0/m, z3.d, #5
+; VBITS_GE_256-NEXT:    asrd z2.d, p0/m, z2.d, #5
+; VBITS_GE_256-NEXT:    asrd z5.d, p0/m, z5.d, #5
+; VBITS_GE_256-NEXT:    asrd z4.d, p0/m, z4.d, #5
+; VBITS_GE_256-NEXT:    asrd z6.d, p0/m, z6.d, #5
+; VBITS_GE_256-NEXT:    asrd z7.d, p0/m, z7.d, #5
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: sdiv_v32i64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    asrd z0.d, p0/m, z0.d, #5
 ; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <32 x i64>, <32 x i64>* %a
   %res = sdiv <32 x i64> %op1, shufflevector (<32 x i64> insertelement (<32 x i64> poison, i64 32, i32 0), <32 x i64> poison, <32 x i32> zeroinitializer)
   store <32 x i64> %res, <32 x i64>* %a
   ret void
 }
 
 attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-shuffles.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc < %s | FileCheck %s
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Currently there is no custom lowering for vector shuffles operating on types
 ; bigger than NEON. However, having no support opens us up to a code generator
 ; hang when expanding BUILD_VECTOR. Here we just validate the promblematic case
 ; successfully exits code generation.
 define void @hang_when_merging_stores_after_legalisation(<8 x i32>* %a, <2 x i32> %b) #0 {
 ; CHECK-LABEL: hang_when_merging_stores_after_legalisation:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    mov x29, sp
+; CHECK-NEXT:    .cfi_def_cfa w29, 16
+; CHECK-NEXT:    .cfi_offset w30, -8
+; CHECK-NEXT:    .cfi_offset w29, -16
+; CHECK-NEXT:    sub x9, sp, #48
+; CHECK-NEXT:    and sp, x9, #0xffffffffffffffe0
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    stp s0, s0, [sp, #24]
+; CHECK-NEXT:    stp s0, s0, [sp, #16]
+; CHECK-NEXT:    stp s0, s0, [sp, #8]
+; CHECK-NEXT:    stp s0, s0, [sp]
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [sp]
+; CHECK-NEXT:    mov z1.d, z0.d
+; CHECK-NEXT:    ext z1.b, z1.b, z1.b, #16
+; CHECK-NEXT:    st2 { v0.4s, v1.4s }, [x0]
+; CHECK-NEXT:    mov sp, x29
+; CHECK-NEXT:    ldp x29, x30, [sp], #16 // 16-byte Folded Reload
+; CHECK-NEXT:    ret
   %splat = shufflevector <2 x i32> %b, <2 x i32> undef, <8 x i32> zeroinitializer
   %interleaved.vec = shufflevector <8 x i32> %splat, <8 x i32> undef, <8 x i32> <i32 0, i32 4, i32 1, i32 5, i32 2, i32 6, i32 3, i32 7>
   store <8 x i32> %interleaved.vec, <8 x i32>* %a, align 4
   ret void
 }
 
 ; Ensure we don't crash when trying to lower a shuffle via and extract
 define void @crash_when_lowering_extract_shuffle(<32 x i32>* %dst, i1 %cond) #0 {
 ; CHECK-LABEL: crash_when_lowering_extract_shuffle:
-; CHECK:       ld1w { z3.s }, p0/z, [x0]
-; CHECK:       st1w { z3.s }, p0, [x0]
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    tbnz w1, #0, .LBB1_2
+; CHECK-NEXT:  // %bb.1: // %vector.body
+; CHECK-NEXT:    mov z0.b, #0 // =0x0
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    umov w8, v0.b[8]
+; CHECK-NEXT:    umov w9, v0.b[1]
+; CHECK-NEXT:    umov w10, v0.b[9]
+; CHECK-NEXT:    umov w11, v0.b[2]
+; CHECK-NEXT:    mov v1.16b, v0.16b
+; CHECK-NEXT:    fmov s2, w8
+; CHECK-NEXT:    umov w8, v0.b[10]
+; CHECK-NEXT:    mov v1.b[1], w9
+; CHECK-NEXT:    umov w9, v0.b[3]
+; CHECK-NEXT:    mov v2.b[1], w10
+; CHECK-NEXT:    umov w10, v0.b[11]
+; CHECK-NEXT:    mov v1.b[2], w11
+; CHECK-NEXT:    umov w11, v0.b[7]
+; CHECK-NEXT:    mov v2.b[2], w8
+; CHECK-NEXT:    umov w8, v0.b[4]
+; CHECK-NEXT:    mov v1.b[3], w9
+; CHECK-NEXT:    umov w9, v0.b[12]
+; CHECK-NEXT:    mov v2.b[3], w10
+; CHECK-NEXT:    umov w10, v0.b[5]
+; CHECK-NEXT:    mov v1.b[4], w8
+; CHECK-NEXT:    umov w8, v0.b[13]
+; CHECK-NEXT:    mov v2.b[4], w9
+; CHECK-NEXT:    umov w9, v0.b[6]
+; CHECK-NEXT:    mov v1.b[5], w10
+; CHECK-NEXT:    umov w10, v0.b[14]
+; CHECK-NEXT:    mov v2.b[5], w8
+; CHECK-NEXT:    mov x8, #16
+; CHECK-NEXT:    mov v1.b[6], w9
+; CHECK-NEXT:    mov x9, #24
+; CHECK-NEXT:    ld1w { z4.s }, p0/z, [x0, x8, lsl #2]
+; CHECK-NEXT:    mov v2.b[6], w10
+; CHECK-NEXT:    umov w10, v0.b[15]
+; CHECK-NEXT:    ext z0.b, z0.b, z0.b, #16
+; CHECK-NEXT:    ld1w { z5.s }, p0/z, [x0, x9, lsl #2]
+; CHECK-NEXT:    dup v3.2d, v0.d[1]
+; CHECK-NEXT:    uunpklo z0.h, z0.b
+; CHECK-NEXT:    mov v1.b[7], w11
+; CHECK-NEXT:    uunpklo z0.s, z0.h
+; CHECK-NEXT:    mov v2.b[7], w10
+; CHECK-NEXT:    uunpklo z3.h, z3.b
+; CHECK-NEXT:    uunpklo z3.s, z3.h
+; CHECK-NEXT:    mov x11, #8
+; CHECK-NEXT:    lsl z0.s, z0.s, #31
+; CHECK-NEXT:    lsl z3.s, z3.s, #31
+; CHECK-NEXT:    asr z0.s, z0.s, #31
+; CHECK-NEXT:    asr z3.s, z3.s, #31
+; CHECK-NEXT:    uunpklo z1.h, z1.b
+; CHECK-NEXT:    uunpklo z2.h, z2.b
+; CHECK-NEXT:    and z0.s, z0.s, #0x1
+; CHECK-NEXT:    and z3.s, z3.s, #0x1
+; CHECK-NEXT:    uunpklo z1.s, z1.h
+; CHECK-NEXT:    uunpklo z2.s, z2.h
+; CHECK-NEXT:    cmpne p1.s, p0/z, z0.s, #0
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0, x11, lsl #2]
+; CHECK-NEXT:    cmpne p2.s, p0/z, z3.s, #0
+; CHECK-NEXT:    ld1w { z3.s }, p0/z, [x0]
+; CHECK-NEXT:    lsl z1.s, z1.s, #31
+; CHECK-NEXT:    lsl z2.s, z2.s, #31
+; CHECK-NEXT:    asr z1.s, z1.s, #31
+; CHECK-NEXT:    asr z2.s, z2.s, #31
+; CHECK-NEXT:    and z1.s, z1.s, #0x1
+; CHECK-NEXT:    and z2.s, z2.s, #0x1
+; CHECK-NEXT:    mov z4.s, p1/m, #0 // =0x0
+; CHECK-NEXT:    mov z5.s, p2/m, #0 // =0x0
+; CHECK-NEXT:    cmpne p1.s, p0/z, z1.s, #0
+; CHECK-NEXT:    cmpne p2.s, p0/z, z2.s, #0
+; CHECK-NEXT:    mov z3.s, p1/m, #0 // =0x0
+; CHECK-NEXT:    mov z0.s, p2/m, #0 // =0x0
+; CHECK-NEXT:    st1w { z4.s }, p0, [x0, x8, lsl #2]
+; CHECK-NEXT:    st1w { z5.s }, p0, [x0, x9, lsl #2]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x0, x11, lsl #2]
+; CHECK-NEXT:    st1w { z3.s }, p0, [x0]
+; CHECK-NEXT:  .LBB1_2: // %exit
+; CHECK-NEXT:    ret
   %broadcast.splat = shufflevector <32 x i1> zeroinitializer, <32 x i1> zeroinitializer, <32 x i32> zeroinitializer
   br i1 %cond, label %exit, label %vector.body
 
 vector.body:
   %1 = load <32 x i32>, <32 x i32>* %dst, align 16
   %predphi = select <32 x i1> %broadcast.splat, <32 x i32> zeroinitializer, <32 x i32> %1
   store <32 x i32> %predphi, <32 x i32>* %dst, align 16
   br label %exit
 
 exit:
   ret void
 }
 
 attributes #0 = { vscale_range(2,2) "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-splat-vector.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024

 ; NO_SVE-NOT: ptrue
 
 ;
 ; DUP (integer)
 ;
 
 ; Don't use SVE for 64-bit vectors.
 define <8 x i8> @splat_v8i8(i8 %a) #0 {
+; NO_SVE-LABEL: splat_v8i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8b, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v8i8:
-; CHECK: dup v0.8b, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.8b, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <8 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <8 x i8> %insert, <8 x i8> undef, <8 x i32> zeroinitializer
   ret <8 x i8> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <16 x i8> @splat_v16i8(i8 %a) #0 {
+; NO_SVE-LABEL: splat_v16i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.16b, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v16i8:
-; CHECK: dup v0.16b, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.16b, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <16 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <16 x i8> %insert, <16 x i8> undef, <16 x i32> zeroinitializer
   ret <16 x i8> %splat
 }
 
 define void @splat_v32i8(i8 %a, <32 x i8>* %b) #0 {
+; NO_SVE-LABEL: splat_v32i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.16b, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v32i8:
-; CHECK-DAG: mov [[RES:z[0-9]+]].b, w0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].b, vl32
-; CHECK-NEXT: st1b { [[RES]].b }, [[PG]], [x1]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b, vl32
+; CHECK-NEXT:    mov z0.b, w0
+; CHECK-NEXT:    st1b { z0.b }, p0, [x1]
+; CHECK-NEXT:    ret
   %insert = insertelement <32 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <32 x i8> %insert, <32 x i8> undef, <32 x i32> zeroinitializer
   store <32 x i8> %splat, <32 x i8>* %b
   ret void
 }
 
 define void @splat_v64i8(i8 %a, <64 x i8>* %b) #0 {
-; CHECK-LABEL: splat_v64i8:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].b, w0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].b, vl64
-; VBITS_GE_512-NEXT: st1b { [[RES]].b }, [[PG]], [x1]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v64i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.16b, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    mov z0.b, w0
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v64i8:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.b, vl64
+; VBITS_GE_512-NEXT:    mov z0.b, w0
+; VBITS_GE_512-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].b, w0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].b, vl32
-; VBITS_EQ_256-DAG: mov w[[NUMELTS:[0-9]+]], #32
-; VBITS_EQ_256-DAG: st1b { [[RES]].b }, [[PG]], [x1]
-; VBITS_EQ_256-DAG: st1b { [[RES]].b }, [[PG]], [x1, x[[NUMELTS]]]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <64 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <64 x i8> %insert, <64 x i8> undef, <64 x i32> zeroinitializer
   store <64 x i8> %splat, <64 x i8>* %b
   ret void
 }
 
 define void @splat_v128i8(i8 %a, <128 x i8>* %b) #0 {
-; CHECK-LABEL: splat_v128i8:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].b, w0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].b, vl128
-; VBITS_GE_1024-NEXT: st1b { [[RES]].b }, [[PG]], [x1]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v128i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.16b, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v128i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #96
+; VBITS_GE_256-NEXT:    mov w9, #64
+; VBITS_GE_256-NEXT:    mov w10, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    mov z0.b, w0
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x9]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x10]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v128i8:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.b, vl128
+; VBITS_GE_1024-NEXT:    mov z0.b, w0
+; VBITS_GE_1024-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <128 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <128 x i8> %insert, <128 x i8> undef, <128 x i32> zeroinitializer
   store <128 x i8> %splat, <128 x i8>* %b
   ret void
 }
 
 define void @splat_v256i8(i8 %a, <256 x i8>* %b) #0 {
-; CHECK-LABEL: splat_v256i8:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].b, w0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].b, vl256
-; VBITS_GE_2048-NEXT: st1b { [[RES]].b }, [[PG]], [x1]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v256i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.16b, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v256i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #224
+; VBITS_GE_256-NEXT:    mov w9, #192
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    mov z0.b, w0
+; VBITS_GE_256-NEXT:    mov w10, #160
+; VBITS_GE_256-NEXT:    mov w11, #128
+; VBITS_GE_256-NEXT:    mov w12, #96
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    mov w8, #64
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x9]
+; VBITS_GE_256-NEXT:    mov w9, #32
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x10]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x11]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x12]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x8]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x1, x9]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v256i8:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.b, vl256
+; VBITS_GE_2048-NEXT:    mov z0.b, w0
+; VBITS_GE_2048-NEXT:    st1b { z0.b }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <256 x i8> undef, i8 %a, i64 0
   %splat = shufflevector <256 x i8> %insert, <256 x i8> undef, <256 x i32> zeroinitializer
   store <256 x i8> %splat, <256 x i8>* %b
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <4 x i16> @splat_v4i16(i16 %a) #0 {
+; NO_SVE-LABEL: splat_v4i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4h, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4i16:
-; CHECK: dup v0.4h, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.4h, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <4 x i16> %insert, <4 x i16> undef, <4 x i32> zeroinitializer
   ret <4 x i16> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <8 x i16> @splat_v8i16(i16 %a) #0 {
+; NO_SVE-LABEL: splat_v8i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8h, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v8i16:
-; CHECK: dup v0.8h, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.8h, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <8 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <8 x i16> %insert, <8 x i16> undef, <8 x i32> zeroinitializer
   ret <8 x i16> %splat
 }
 
 define void @splat_v16i16(i16 %a, <16 x i16>* %b) #0 {
+; NO_SVE-LABEL: splat_v16i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8h, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v16i16:
-; CHECK-DAG: mov [[RES:z[0-9]+]].h, w0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].h, vl16
-; CHECK-NEXT: st1h { [[RES]].h }, [[PG]], [x1]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h, vl16
+; CHECK-NEXT:    mov z0.h, w0
+; CHECK-NEXT:    st1h { z0.h }, p0, [x1]
+; CHECK-NEXT:    ret
   %insert = insertelement <16 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <16 x i16> %insert, <16 x i16> undef, <16 x i32> zeroinitializer
   store <16 x i16> %splat, <16 x i16>* %b
   ret void
 }
 
 define void @splat_v32i16(i16 %a, <32 x i16>* %b) #0 {
-; CHECK-LABEL: splat_v32i16:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].h, w0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512-NEXT: st1h { [[RES]].h }, [[PG]], [x1]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8h, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov z0.h, w0
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v32i16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    mov z0.h, w0
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].h, w0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].h, vl16
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #16
-; VBITS_EQ_256-DAG: st1h { [[RES]].h }, [[PG]], [x1]
-; VBITS_EQ_256-DAG: st1h { [[RES]].h }, [[PG]], [x1, x[[NUMELTS]], lsl #1]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <32 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <32 x i16> %insert, <32 x i16> undef, <32 x i32> zeroinitializer
   store <32 x i16> %splat, <32 x i16>* %b
   ret void
 }
 
 define void @splat_v64i16(i16 %a, <64 x i16>* %b) #0 {
-; CHECK-LABEL: splat_v64i16:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].h, w0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].h, vl64
-; VBITS_GE_1024-NEXT: st1h { [[RES]].h }, [[PG]], [x1]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v64i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8h, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #32
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov z0.h, w0
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v64i16:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
+; VBITS_GE_1024-NEXT:    mov z0.h, w0
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <64 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <64 x i16> %insert, <64 x i16> undef, <64 x i32> zeroinitializer
   store <64 x i16> %splat, <64 x i16>* %b
   ret void
 }
 
 define void @splat_v128i16(i16 %a, <128 x i16>* %b) #0 {
-; CHECK-LABEL: splat_v128i16:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].h, w0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].h, vl128
-; VBITS_GE_2048-NEXT: st1h { [[RES]].h }, [[PG]], [x1]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v128i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.8h, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v128i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #112
+; VBITS_GE_256-NEXT:    mov x9, #96
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov z0.h, w0
+; VBITS_GE_256-NEXT:    mov x10, #80
+; VBITS_GE_256-NEXT:    mov x11, #64
+; VBITS_GE_256-NEXT:    mov x12, #48
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v128i16:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
+; VBITS_GE_2048-NEXT:    mov z0.h, w0
+; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <128 x i16> undef, i16 %a, i64 0
   %splat = shufflevector <128 x i16> %insert, <128 x i16> undef, <128 x i32> zeroinitializer
   store <128 x i16> %splat, <128 x i16>* %b
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x i32> @splat_v2i32(i32 %a) #0 {
+; NO_SVE-LABEL: splat_v2i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2s, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v2i32:
-; CHECK: dup v0.2s, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.2s, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <2 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <2 x i32> %insert, <2 x i32> undef, <2 x i32> zeroinitializer
   ret <2 x i32> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x i32> @splat_v4i32(i32 %a) #0 {
+; NO_SVE-LABEL: splat_v4i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4s, w0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4i32:
-; CHECK: dup v0.4s, w0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.4s, w0
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <4 x i32> %insert, <4 x i32> undef, <4 x i32> zeroinitializer
   ret <4 x i32> %splat
 }
 
 define void @splat_v8i32(i32 %a, <8 x i32>* %b) #0 {
+; NO_SVE-LABEL: splat_v8i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4s, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v8i32:
-; CHECK-DAG: mov [[RES:z[0-9]+]].s, w0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x1]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    mov z0.s, w0
+; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
+; CHECK-NEXT:    ret
   %insert = insertelement <8 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <8 x i32> %insert, <8 x i32> undef, <8 x i32> zeroinitializer
   store <8 x i32> %splat, <8 x i32>* %b
   ret void
 }
 
 define void @splat_v16i32(i32 %a, <16 x i32>* %b) #0 {
-; CHECK-LABEL: splat_v16i32:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].s, w0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: st1w { [[RES]].s }, [[PG]], [x1]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4s, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov z0.s, w0
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v16i32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    mov z0.s, w0
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].s, w0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].s, vl8
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #8
-; VBITS_EQ_256-DAG: st1w { [[RES]].s }, [[PG]], [x1]
-; VBITS_EQ_256-DAG: st1w { [[RES]].s }, [[PG]], [x1, x[[NUMELTS]], lsl #2]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <16 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <16 x i32> %insert, <16 x i32> undef, <16 x i32> zeroinitializer
   store <16 x i32> %splat, <16 x i32>* %b
   ret void
 }
 
 define void @splat_v32i32(i32 %a, <32 x i32>* %b) #0 {
-; CHECK-LABEL: splat_v32i32:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].s, w0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].s, vl32
-; VBITS_GE_1024-NEXT: st1w { [[RES]].s }, [[PG]], [x1]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v32i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4s, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov z0.s, w0
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v32i32:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
+; VBITS_GE_1024-NEXT:    mov z0.s, w0
+; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <32 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <32 x i32> %insert, <32 x i32> undef, <32 x i32> zeroinitializer
   store <32 x i32> %splat, <32 x i32>* %b
   ret void
 }
 
 define void @splat_v64i32(i32 %a, <64 x i32>* %b) #0 {
-; CHECK-LABEL: splat_v64i32:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].s, w0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].s, vl64
-; VBITS_GE_2048-NEXT: st1w { [[RES]].s }, [[PG]], [x1]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v64i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.4s, w0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov z0.s, w0
+; VBITS_GE_256-NEXT:    mov x10, #40
+; VBITS_GE_256-NEXT:    mov x11, #32
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v64i32:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
+; VBITS_GE_2048-NEXT:    mov z0.s, w0
+; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <64 x i32> undef, i32 %a, i64 0
   %splat = shufflevector <64 x i32> %insert, <64 x i32> undef, <64 x i32> zeroinitializer
   store <64 x i32> %splat, <64 x i32>* %b
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x i64> @splat_v1i64(i64 %a) #0 {
+; NO_SVE-LABEL: splat_v1i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov d0, x0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v1i64:
-; CHECK: fmov d0, x0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    fmov d0, x0
+; CHECK-NEXT:    ret
   %insert = insertelement <1 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <1 x i64> %insert, <1 x i64> undef, <1 x i32> zeroinitializer
   ret <1 x i64> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x i64> @splat_v2i64(i64 %a) #0 {
+; NO_SVE-LABEL: splat_v2i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2d, x0
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v2i64:
-; CHECK: dup v0.2d, x0
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    dup v0.2d, x0
+; CHECK-NEXT:    ret
   %insert = insertelement <2 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <2 x i64> %insert, <2 x i64> undef, <2 x i32> zeroinitializer
   ret <2 x i64> %splat
 }
 
 define void @splat_v4i64(i64 %a, <4 x i64>* %b) #0 {
+; NO_SVE-LABEL: splat_v4i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2d, x0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4i64:
-; CHECK-DAG: mov [[RES:z[0-9]+]].d, x0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].d, vl4
-; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x1]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d, vl4
+; CHECK-NEXT:    mov z0.d, x0
+; CHECK-NEXT:    st1d { z0.d }, p0, [x1]
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <4 x i64> %insert, <4 x i64> undef, <4 x i32> zeroinitializer
   store <4 x i64> %splat, <4 x i64>* %b
   ret void
 }
 
 define void @splat_v8i64(i64 %a, <8 x i64>* %b) #0 {
-; CHECK-LABEL: splat_v8i64:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].d, x0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: st1d { [[RES]].d }, [[PG]], [x1]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2d, x0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    mov z0.d, x0
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v8i64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    mov z0.d, x0
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].d, x0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].d, vl4
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #4
-; VBITS_EQ_256-DAG: st1d { [[RES]].d }, [[PG]], [x1]
-; VBITS_EQ_256-DAG: st1d { [[RES]].d }, [[PG]], [x1, x[[NUMELTS]], lsl #3]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <8 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <8 x i64> %insert, <8 x i64> undef, <8 x i32> zeroinitializer
   store <8 x i64> %splat, <8 x i64>* %b
   ret void
 }
 
 define void @splat_v16i64(i64 %a, <16 x i64>* %b) #0 {
-; CHECK-LABEL: splat_v16i64:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].d, x0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].d, vl16
-; VBITS_GE_1024-NEXT: st1d { [[RES]].d }, [[PG]], [x1]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v16i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2d, x0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov z0.d, x0
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v16i64:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
+; VBITS_GE_1024-NEXT:    mov z0.d, x0
+; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <16 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <16 x i64> %insert, <16 x i64> undef, <16 x i32> zeroinitializer
   store <16 x i64> %splat, <16 x i64>* %b
   ret void
 }
 
 define void @splat_v32i64(i64 %a, <32 x i64>* %b) #0 {
-; CHECK-LABEL: splat_v32i64:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].d, x0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].d, vl32
-; VBITS_GE_2048-NEXT: st1d { [[RES]].d }, [[PG]], [x1]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v32i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    dup v0.2d, x0
+; NO_SVE-NEXT:    stp q0, q0, [x1]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #28
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov z0.d, x0
+; VBITS_GE_256-NEXT:    mov x10, #20
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #12
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v32i64:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
+; VBITS_GE_2048-NEXT:    mov z0.d, x0
+; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <32 x i64> undef, i64 %a, i64 0
   %splat = shufflevector <32 x i64> %insert, <32 x i64> undef, <32 x i32> zeroinitializer
   store <32 x i64> %splat, <32 x i64>* %b
   ret void
 }
 
 ;
 ; DUP (floating-point)
 ;
 
 ; Don't use SVE for 64-bit vectors.
 define <4 x half> @splat_v4f16(half %a) #0 {
+; NO_SVE-LABEL: splat_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.4h, v0.h[0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4f16:
-; CHECK: dup v0.4h, v0.h[0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $h0 killed $h0 def $q0
+; CHECK-NEXT:    dup v0.4h, v0.h[0]
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x half> undef, half %a, i64 0
   %splat = shufflevector <4 x half> %insert, <4 x half> undef, <4 x i32> zeroinitializer
   ret <4 x half> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <8 x half> @splat_v8f16(half %a) #0 {
+; NO_SVE-LABEL: splat_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.8h, v0.h[0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v8f16:
-; CHECK: dup v0.8h, v0.h[0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $h0 killed $h0 def $q0
+; CHECK-NEXT:    dup v0.8h, v0.h[0]
+; CHECK-NEXT:    ret
   %insert = insertelement <8 x half> undef, half %a, i64 0
   %splat = shufflevector <8 x half> %insert, <8 x half> undef, <8 x i32> zeroinitializer
   ret <8 x half> %splat
 }
 
 define void @splat_v16f16(half %a, <16 x half>* %b) #0 {
+; NO_SVE-LABEL: splat_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.8h, v0.h[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v16f16:
-; CHECK-DAG: mov [[RES:z[0-9]+]].h, h0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].h, vl16
-; CHECK-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $h0 killed $h0 def $z0
+; CHECK-NEXT:    ptrue p0.h, vl16
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    st1h { z0.h }, p0, [x0]
+; CHECK-NEXT:    ret
   %insert = insertelement <16 x half> undef, half %a, i64 0
   %splat = shufflevector <16 x half> %insert, <16 x half> undef, <16 x i32> zeroinitializer
   store <16 x half> %splat, <16 x half>* %b
   ret void
 }
 
 define void @splat_v32f16(half %a, <32 x half>* %b) #0 {
-; CHECK-LABEL: splat_v32f16:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].h, h0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.8h, v0.h[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov z0.h, h0
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v32f16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    mov z0.h, h0
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].h, h0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].h, vl16
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #16
-; VBITS_EQ_256-DAG: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_EQ_256-DAG: st1h { [[RES]].h }, [[PG]], [x0, x[[NUMELTS]], lsl #1]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <32 x half> undef, half %a, i64 0
   %splat = shufflevector <32 x half> %insert, <32 x half> undef, <32 x i32> zeroinitializer
   store <32 x half> %splat, <32 x half>* %b
   ret void
 }
 
 define void @splat_v64f16(half %a, <64 x half>* %b) #0 {
-; CHECK-LABEL: splat_v64f16:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].h, h0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].h, vl64
-; VBITS_GE_1024-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.8h, v0.h[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v64f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #32
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov z0.h, h0
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v64f16:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
+; VBITS_GE_1024-NEXT:    mov z0.h, h0
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <64 x half> undef, half %a, i64 0
   %splat = shufflevector <64 x half> %insert, <64 x half> undef, <64 x i32> zeroinitializer
   store <64 x half> %splat, <64 x half>* %b
   ret void
 }
 
 define void @splat_v128f16(half %a, <128 x half>* %b) #0 {
-; CHECK-LABEL: splat_v128f16:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].h, h0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].h, vl128
-; VBITS_GE_2048-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v128f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $h0 killed $h0 def $q0
+; NO_SVE-NEXT:    dup v0.8h, v0.h[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v128f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #112
+; VBITS_GE_256-NEXT:    mov x9, #96
+; VBITS_GE_256-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #80
+; VBITS_GE_256-NEXT:    mov z0.h, h0
+; VBITS_GE_256-NEXT:    mov x11, #64
+; VBITS_GE_256-NEXT:    mov x12, #48
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v128f16:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    // kill: def $h0 killed $h0 def $z0
+; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
+; VBITS_GE_2048-NEXT:    mov z0.h, h0
+; VBITS_GE_2048-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <128 x half> undef, half %a, i64 0
   %splat = shufflevector <128 x half> %insert, <128 x half> undef, <128 x i32> zeroinitializer
   store <128 x half> %splat, <128 x half>* %b
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <2 x float> @splat_v2f32(float %a, <2 x float> %op2) #0 {
+; NO_SVE-LABEL: splat_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.2s, v0.s[0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v2f32:
-; CHECK: dup v0.2s, v0.s[0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $s0 killed $s0 def $q0
+; CHECK-NEXT:    dup v0.2s, v0.s[0]
+; CHECK-NEXT:    ret
   %insert = insertelement <2 x float> undef, float %a, i64 0
   %splat = shufflevector <2 x float> %insert, <2 x float> undef, <2 x i32> zeroinitializer
   ret <2 x float> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <4 x float> @splat_v4f32(float %a, <4 x float> %op2) #0 {
+; NO_SVE-LABEL: splat_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.4s, v0.s[0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4f32:
-; CHECK: dup v0.4s, v0.s[0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $s0 killed $s0 def $q0
+; CHECK-NEXT:    dup v0.4s, v0.s[0]
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x float> undef, float %a, i64 0
   %splat = shufflevector <4 x float> %insert, <4 x float> undef, <4 x i32> zeroinitializer
   ret <4 x float> %splat
 }
 
 define void @splat_v8f32(float %a, <8 x float>* %b) #0 {
+; NO_SVE-LABEL: splat_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.4s, v0.s[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v8f32:
-; CHECK-DAG: mov [[RES:z[0-9]+]].s, s0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $s0 killed $s0 def $z0
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    st1w { z0.s }, p0, [x0]
+; CHECK-NEXT:    ret
   %insert = insertelement <8 x float> undef, float %a, i64 0
   %splat = shufflevector <8 x float> %insert, <8 x float> undef, <8 x i32> zeroinitializer
   store <8 x float> %splat, <8 x float>* %b
   ret void
 }
 
 define void @splat_v16f32(float %a, <16 x float>* %b) #0 {
-; CHECK-LABEL: splat_v16f32:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].s, s0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.4s, v0.s[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov z0.s, s0
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v16f32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    mov z0.s, s0
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].s, s0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].s, vl8
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #8
-; VBITS_EQ_256-DAG: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_EQ_256-DAG: st1w { [[RES]].s }, [[PG]], [x0, x[[NUMELTS]], lsl #2]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <16 x float> undef, float %a, i64 0
   %splat = shufflevector <16 x float> %insert, <16 x float> undef, <16 x i32> zeroinitializer
   store <16 x float> %splat, <16 x float>* %b
   ret void
 }
 
 define void @splat_v32f32(float %a, <32 x float>* %b) #0 {
-; CHECK-LABEL: splat_v32f32:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].s, s0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].s, vl32
-; VBITS_GE_1024-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.4s, v0.s[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov z0.s, s0
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v32f32:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
+; VBITS_GE_1024-NEXT:    mov z0.s, s0
+; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <32 x float> undef, float %a, i64 0
   %splat = shufflevector <32 x float> %insert, <32 x float> undef, <32 x i32> zeroinitializer
   store <32 x float> %splat, <32 x float>* %b
   ret void
 }
 
 define void @splat_v64f32(float %a, <64 x float>* %b) #0 {
-; CHECK-LABEL: splat_v64f32:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].s, s0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].s, vl64
-; VBITS_GE_2048-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $s0 killed $s0 def $q0
+; NO_SVE-NEXT:    dup v0.4s, v0.s[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #40
+; VBITS_GE_256-NEXT:    mov z0.s, s0
+; VBITS_GE_256-NEXT:    mov x11, #32
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v64f32:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    // kill: def $s0 killed $s0 def $z0
+; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
+; VBITS_GE_2048-NEXT:    mov z0.s, s0
+; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <64 x float> undef, float %a, i64 0
   %splat = shufflevector <64 x float> %insert, <64 x float> undef, <64 x i32> zeroinitializer
   store <64 x float> %splat, <64 x float>* %b
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors.
 define <1 x double> @splat_v1f64(double %a, <1 x double> %op2) #0 {
+; NO_SVE-LABEL: splat_v1f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v1f64:
-; CHECK: // %bb.0:
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ret
   %insert = insertelement <1 x double> undef, double %a, i64 0
   %splat = shufflevector <1 x double> %insert, <1 x double> undef, <1 x i32> zeroinitializer
   ret <1 x double> %splat
 }
 
 ; Don't use SVE for 128-bit vectors.
 define <2 x double> @splat_v2f64(double %a, <2 x double> %op2) #0 {
+; NO_SVE-LABEL: splat_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2d, v0.d[0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v2f64:
-; CHECK: dup v0.2d, v0.d[0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $q0
+; CHECK-NEXT:    dup v0.2d, v0.d[0]
+; CHECK-NEXT:    ret
   %insert = insertelement <2 x double> undef, double %a, i64 0
   %splat = shufflevector <2 x double> %insert, <2 x double> undef, <2 x i32> zeroinitializer
   ret <2 x double> %splat
 }
 
 define void @splat_v4f64(double %a, <4 x double>* %b) #0 {
+; NO_SVE-LABEL: splat_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2d, v0.d[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: splat_v4f64:
-; CHECK-DAG: mov [[RES:z[0-9]+]].d, d0
-; CHECK-DAG: ptrue [[PG:p[0-9]+]].d, vl4
-; CHECK-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; CHECK-NEXT: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    // kill: def $d0 killed $d0 def $z0
+; CHECK-NEXT:    ptrue p0.d, vl4
+; CHECK-NEXT:    mov z0.d, d0
+; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
+; CHECK-NEXT:    ret
   %insert = insertelement <4 x double> undef, double %a, i64 0
   %splat = shufflevector <4 x double> %insert, <4 x double> undef, <4 x i32> zeroinitializer
   store <4 x double> %splat, <4 x double>* %b
   ret void
 }
 
 define void @splat_v8f64(double %a, <8 x double>* %b) #0 {
-; CHECK-LABEL: splat_v8f64:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].d, d0
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2d, v0.d[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov z0.d, d0
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_v8f64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    mov z0.d, d0
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
 
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-DAG: mov [[RES:z[0-9]+]].d, d0
-; VBITS_EQ_256-DAG: ptrue [[PG:p[0-9]+]].d, vl4
-; VBITS_EQ_256-DAG: mov x[[NUMELTS:[0-9]+]], #4
-; VBITS_EQ_256-DAG: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_EQ_256-DAG: st1d { [[RES]].d }, [[PG]], [x0, x[[NUMELTS]], lsl #3]
-; VBITS_EQ_256-NEXT: ret
   %insert = insertelement <8 x double> undef, double %a, i64 0
   %splat = shufflevector <8 x double> %insert, <8 x double> undef, <8 x i32> zeroinitializer
   store <8 x double> %splat, <8 x double>* %b
   ret void
 }
 
 define void @splat_v16f64(double %a, <16 x double>* %b) #0 {
-; CHECK-LABEL: splat_v16f64:
-; VBITS_GE_1024-DAG: mov [[RES:z[0-9]+]].d, d0
-; VBITS_GE_1024-DAG: ptrue [[PG:p[0-9]+]].d, vl16
-; VBITS_GE_1024-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_GE_1024-NEXT: ret
+; NO_SVE-LABEL: splat_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2d, v0.d[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov z0.d, d0
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: splat_v16f64:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
+; VBITS_GE_1024-NEXT:    mov z0.d, d0
+; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_1024-NEXT:    ret
   %insert = insertelement <16 x double> undef, double %a, i64 0
   %splat = shufflevector <16 x double> %insert, <16 x double> undef, <16 x i32> zeroinitializer
   store <16 x double> %splat, <16 x double>* %b
   ret void
 }
 
 define void @splat_v32f64(double %a, <32 x double>* %b) #0 {
-; CHECK-LABEL: splat_v32f64:
-; VBITS_GE_2048-DAG: mov [[RES:z[0-9]+]].d, d0
-; VBITS_GE_2048-DAG: ptrue [[PG:p[0-9]+]].d, vl32
-; VBITS_GE_2048-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_GE_2048-NEXT: ret
+; NO_SVE-LABEL: splat_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    // kill: def $d0 killed $d0 def $q0
+; NO_SVE-NEXT:    dup v0.2d, v0.d[0]
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #64]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #96]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #128]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #192]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #28
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov x10, #20
+; VBITS_GE_256-NEXT:    mov z0.d, d0
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #12
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: splat_v32f64:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    // kill: def $d0 killed $d0 def $z0
+; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
+; VBITS_GE_2048-NEXT:    mov z0.d, d0
+; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_2048-NEXT:    ret
   %insert = insertelement <32 x double> undef, double %a, i64 0
   %splat = shufflevector <32 x double> %insert, <32 x double> undef, <32 x i32> zeroinitializer
   store <32 x double> %splat, <32 x double>* %b
   ret void
 }
 
 ;
 ; DUP (integer immediate)
 ;
 
 define void @splat_imm_v64i8(<64 x i8>* %a) #0 {
-; CHECK-LABEL: splat_imm_v64i8:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].b, #1
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].b, vl64
-; VBITS_GE_512-NEXT: st1b { [[RES]].b }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v64i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    movi v0.16b, #1
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    mov z0.b, #1 // =0x1
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v64i8:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    mov z0.b, #1 // =0x1
+; VBITS_GE_512-NEXT:    ptrue p0.b, vl64
+; VBITS_GE_512-NEXT:    st1b { z0.b }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <64 x i8> undef, i8 1, i64 0
   %splat = shufflevector <64 x i8> %insert, <64 x i8> undef, <64 x i32> zeroinitializer
   store <64 x i8> %splat, <64 x i8>* %a
   ret void
 }
 
 define void @splat_imm_v32i16(<32 x i16>* %a) #0 {
-; CHECK-LABEL: splat_imm_v32i16:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].h, #2
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    movi v0.8h, #2
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov z0.h, #2 // =0x2
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v32i16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    mov z0.h, #2 // =0x2
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <32 x i16> undef, i16 2, i64 0
   %splat = shufflevector <32 x i16> %insert, <32 x i16> undef, <32 x i32> zeroinitializer
   store <32 x i16> %splat, <32 x i16>* %a
   ret void
 }
 
 define void @splat_imm_v16i32(<16 x i32>* %a) #0 {
-; CHECK-LABEL: splat_imm_v16i32:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].s, #3
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    movi v0.4s, #3
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov z0.s, #3 // =0x3
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v16i32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    mov z0.s, #3 // =0x3
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <16 x i32> undef, i32 3, i64 0
   %splat = shufflevector <16 x i32> %insert, <16 x i32> undef, <16 x i32> zeroinitializer
   store <16 x i32> %splat, <16 x i32>* %a
   ret void
 }
 
 define void @splat_imm_v8i64(<8 x i64>* %a) #0 {
-; CHECK-LABEL: splat_imm_v8i64:
-; VBITS_GE_512-DAG: mov [[RES:z[0-9]+]].d, #4
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    mov w8, #4
+; NO_SVE-NEXT:    dup v0.2d, x8
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    mov z0.d, #4 // =0x4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v8i64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    mov z0.d, #4 // =0x4
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <8 x i64> undef, i64 4, i64 0
   %splat = shufflevector <8 x i64> %insert, <8 x i64> undef, <8 x i32> zeroinitializer
   store <8 x i64> %splat, <8 x i64>* %a
   ret void
 }
 
 ;
 ; DUP (floating-point immediate)
 ;
 
 define void @splat_imm_v32f16(<32 x half>* %a) #0 {
-; CHECK-LABEL: splat_imm_v32f16:
-; VBITS_GE_512-DAG: fmov [[RES:z[0-9]+]].h, #5.00000000
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512-NEXT: st1h { [[RES]].h }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    movi v0.8h, #69, lsl #8
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    fmov z0.h, #5.00000000
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v32f16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    fmov z0.h, #5.00000000
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <32 x half> undef, half 5.0, i64 0
   %splat = shufflevector <32 x half> %insert, <32 x half> undef, <32 x i32> zeroinitializer
   store <32 x half> %splat, <32 x half>* %a
   ret void
 }
 
 define void @splat_imm_v16f32(<16 x float>* %a) #0 {
-; CHECK-LABEL: splat_imm_v16f32:
-; VBITS_GE_512-DAG: fmov [[RES:z[0-9]+]].s, #6.00000000
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512-NEXT: st1w { [[RES]].s }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov v0.4s, #6.00000000
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    fmov z0.s, #6.00000000
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v16f32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    fmov z0.s, #6.00000000
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <16 x float> undef, float 6.0, i64 0
   %splat = shufflevector <16 x float> %insert, <16 x float> undef, <16 x i32> zeroinitializer
   store <16 x float> %splat, <16 x float>* %a
   ret void
 }
 
 define void @splat_imm_v8f64(<8 x double>* %a) #0 {
-; CHECK-LABEL: splat_imm_v8f64:
-; VBITS_GE_512-DAG: fmov [[RES:z[0-9]+]].d, #7.00000000
-; VBITS_GE_512-DAG: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512-NEXT: st1d { [[RES]].d }, [[PG]], [x0]
-; VBITS_GE_512-NEXT: ret
+; NO_SVE-LABEL: splat_imm_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    fmov v0.2d, #7.00000000
+; NO_SVE-NEXT:    stp q0, q0, [x0]
+; NO_SVE-NEXT:    stp q0, q0, [x0, #32]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: splat_imm_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    fmov z0.d, #7.00000000
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: splat_imm_v8f64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    fmov z0.d, #7.00000000
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x0]
+; VBITS_GE_512-NEXT:    ret
   %insert = insertelement <8 x double> undef, double 7.0, i64 0
   %splat = shufflevector <8 x double> %insert, <8 x double> undef, <8 x i32> zeroinitializer
   store <8 x double> %splat, <8 x double>* %a
   ret void
 }
 attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-subvector.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK
-; RUN: llc -aarch64-sve-vector-bits-min=384  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 -aarch64-enable-atomic-cfg-tidy=false < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024

 ; ensuring insert/extract sequences are not folded away.
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Don't use SVE when its registers are no bigger than NEON.
 ; NO_SVE-NOT: ptrue
 
 define void @subvector_v8i16(<8 x i16> *%in, <8 x i16>* %out) #0 {
+; NO_SVE-LABEL: subvector_v8i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v8i16:
-; CHECK: ldr [[DATA:q[0-9]+]], [x0]
-; CHECK: str [[DATA]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
   %a = load <8 x i16>, <8 x i16>* %in
   br label %bb1
 
 bb1:
   store <8 x i16> %a, <8 x i16>* %out
   ret void
 }
 
 define void @subvector_v16i16(<16 x i16> *%in, <16 x i16>* %out) #0 {
+; NO_SVE-LABEL: subvector_v16i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v16i16:
-; CHECK: ptrue [[PG:p[0-9]+]].h, vl16
-; CHECK: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; CHECK: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h, vl16
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    st1h { z0.h }, p0, [x1]
+; CHECK-NEXT:    ret
   %a = load <16 x i16>, <16 x i16>* %in
   br label %bb1
 
 bb1:
   store <16 x i16> %a, <16 x i16>* %out
   ret void
 }
 
 define void @subvector_v32i16(<32 x i16> *%in, <32 x i16>* %out) #0 {
-; CHECK-LABEL: subvector_v32i16:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v32i16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <32 x i16>, <32 x i16>* %in
   br label %bb1
 
 bb1:
   store <32 x i16> %a, <32 x i16>* %out
   ret void
 }
 
 define void @subvector_v64i16(<64 x i16> *%in, <64 x i16>* %out) #0 {
-; CHECK-LABEL: subvector_v64i16:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].h, vl64
-; VBITS_GE_1024: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v64i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #32
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v64i16:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <64 x i16>, <64 x i16>* %in
   br label %bb1
 
 bb1:
   store <64 x i16> %a, <64 x i16>* %out
   ret void
 }
 
 define void @subvector_v8i32(<8 x i32> *%in, <8 x i32>* %out) #0 {
+; NO_SVE-LABEL: subvector_v8i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v8i32:
-; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; CHECK: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
+; CHECK-NEXT:    ret
   %a = load <8 x i32>, <8 x i32>* %in
   br label %bb1
 
 bb1:
   store <8 x i32> %a, <8 x i32>* %out
   ret void
 }
 
 define void @subvector_v16i32(<16 x i32> *%in, <16 x i32>* %out) #0 {
-; CHECK-LABEL: subvector_v16i32:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v16i32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <16 x i32>, <16 x i32>* %in
   br label %bb1
 
 bb1:
   store <16 x i32> %a, <16 x i32>* %out
   ret void
 }
 
 define void @subvector_v32i32(<32 x i32> *%in, <32 x i32>* %out) #0 {
-; CHECK-LABEL: subvector_v32i32:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].s, vl32
-; VBITS_GE_1024: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v32i32:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
+; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <32 x i32>, <32 x i32>* %in
   br label %bb1
 
 bb1:
   store <32 x i32> %a, <32 x i32>* %out
   ret void
 }
 
 define void @subvector_v64i32(<64 x i32> *%in, <64 x i32>* %out) #0 {
-; CHECK-LABEL: subvector_v64i32:
-; VBITS_GE_2048: ptrue [[PG:p[0-9]+]].s, vl64
-; VBITS_GE_2048: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_2048: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v64i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #224]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #192]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #160]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #128]
+; NO_SVE-NEXT:    ldp q17, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q19, q18, [x0, #64]
+; NO_SVE-NEXT:    ldp q21, q20, [x0, #32]
+; NO_SVE-NEXT:    ldp q23, q22, [x0]
+; NO_SVE-NEXT:    stp q21, q20, [x1, #32]
+; NO_SVE-NEXT:    stp q19, q18, [x1, #64]
+; NO_SVE-NEXT:    stp q17, q16, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #128]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #160]
+; NO_SVE-NEXT:    stp q23, q22, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #192]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #40
+; VBITS_GE_256-NEXT:    mov x11, #32
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    mov x13, #16
+; VBITS_GE_256-NEXT:    mov x14, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p0, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: subvector_v64i32:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
+; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %a = load <64 x i32>, <64 x i32>* %in
   br label %bb1
 
 bb1:
   store <64 x i32> %a, <64 x i32>* %out
   ret void
 }
 
 
 define void @subvector_v8i64(<8 x i64> *%in, <8 x i64>* %out) #0 {
-; CHECK-LABEL: subvector_v8i64:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v8i64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <8 x i64>, <8 x i64>* %in
   br label %bb1
 
 bb1:
   store <8 x i64> %a, <8 x i64>* %out
   ret void
 }
 
 define void @subvector_v16i64(<16 x i64> *%in, <16 x i64>* %out) #0 {
-; CHECK-LABEL: subvector_v16i64:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].d, vl16
-; VBITS_GE_1024: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v16i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v16i64:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
+; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <16 x i64>, <16 x i64>* %in
   br label %bb1
 
 bb1:
   store <16 x i64> %a, <16 x i64>* %out
   ret void
 }
 
 define void @subvector_v32i64(<32 x i64> *%in, <32 x i64>* %out) #0 {
-; CHECK-LABEL: subvector_v32i64:
-; VBITS_GE_2048: ptrue [[PG:p[0-9]+]].d, vl32
-; VBITS_GE_2048: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_2048: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #224]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #192]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #160]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #128]
+; NO_SVE-NEXT:    ldp q17, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q19, q18, [x0, #64]
+; NO_SVE-NEXT:    ldp q21, q20, [x0, #32]
+; NO_SVE-NEXT:    ldp q23, q22, [x0]
+; NO_SVE-NEXT:    stp q21, q20, [x1, #32]
+; NO_SVE-NEXT:    stp q19, q18, [x1, #64]
+; NO_SVE-NEXT:    stp q17, q16, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #128]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #160]
+; NO_SVE-NEXT:    stp q23, q22, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #192]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #28
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #20
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #12
+; VBITS_GE_256-NEXT:    mov x13, #8
+; VBITS_GE_256-NEXT:    mov x14, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: subvector_v32i64:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
+; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %a = load <32 x i64>, <32 x i64>* %in
   br label %bb1
 
 bb1:
   store <32 x i64> %a, <32 x i64>* %out
   ret void
 }
 
 define void @subvector_v8f16(<8 x half> *%in, <8 x half>* %out) #0 {
+; NO_SVE-LABEL: subvector_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0]
+; NO_SVE-NEXT:    str q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v8f16:
-; CHECK: ldr [[DATA:q[0-9]+]], [x0]
-; CHECK: str [[DATA]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr q0, [x0]
+; CHECK-NEXT:    str q0, [x1]
+; CHECK-NEXT:    ret
   %a = load <8 x half>, <8 x half>* %in
   br label %bb1
 
 bb1:
   store <8 x half> %a, <8 x half>* %out
   ret void
 }
 
 define void @subvector_v16f16(<16 x half> *%in, <16 x half>* %out) #0 {
+; NO_SVE-LABEL: subvector_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v16f16:
-; CHECK: ptrue [[PG:p[0-9]+]].h, vl16
-; CHECK: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; CHECK: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h, vl16
+; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    st1h { z0.h }, p0, [x1]
+; CHECK-NEXT:    ret
   %a = load <16 x half>, <16 x half>* %in
   br label %bb1
 
 bb1:
   store <16 x half> %a, <16 x half>* %out
   ret void
 }
 
 define void @subvector_v32f16(<32 x half> *%in, <32 x half>* %out) #0 {
-; CHECK-LABEL: subvector_v32f16:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].h, vl32
-; VBITS_GE_512: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v32f16:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
+; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <32 x half>, <32 x half>* %in
   br label %bb1
 
 bb1:
   store <32 x half> %a, <32 x half>* %out
   ret void
 }
 
 define void @subvector_v64f16(<64 x half> *%in, <64 x half>* %out) #0 {
-; CHECK-LABEL: subvector_v64f16:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].h, vl64
-; VBITS_GE_1024: ld1h { [[DATA:z[0-9]+.h]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1h { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v64f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #48
+; VBITS_GE_256-NEXT:    mov x9, #32
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v64f16:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
+; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1h { z0.h }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <64 x half>, <64 x half>* %in
   br label %bb1
 
 bb1:
   store <64 x half> %a, <64 x half>* %out
   ret void
 }
 
 define void @subvector_v8f32(<8 x float> *%in, <8 x float>* %out) #0 {
+; NO_SVE-LABEL: subvector_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: subvector_v8f32:
-; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; CHECK: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
+; CHECK-NEXT:    ret
   %a = load <8 x float>, <8 x float>* %in
   br label %bb1
 
 bb1:
   store <8 x float> %a, <8 x float>* %out
   ret void
 }
 
 define void @subvector_v16f32(<16 x float> *%in, <16 x float>* %out) #0 {
-; CHECK-LABEL: subvector_v16f32:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].s, vl16
-; VBITS_GE_512: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v16f32:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
+; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <16 x float>, <16 x float>* %in
   br label %bb1
 
 bb1:
   store <16 x float> %a, <16 x float>* %out
   ret void
 }
 
 define void @subvector_v32f32(<32 x float> *%in, <32 x float>* %out) #0 {
-; CHECK-LABEL: subvector_v32f32:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].s, vl32
-; VBITS_GE_1024: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #24
+; VBITS_GE_256-NEXT:    mov x9, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v32f32:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
+; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <32 x float>, <32 x float>* %in
   br label %bb1
 
 bb1:
   store <32 x float> %a, <32 x float>* %out
   ret void
 }
 
 define void @subvector_v64f32(<64 x float> *%in, <64 x float>* %out) #0 {
-; CHECK-LABEL: subvector_v64f32:
-; VBITS_GE_2048: ptrue [[PG:p[0-9]+]].s, vl64
-; VBITS_GE_2048: ld1w { [[DATA:z[0-9]+.s]] }, [[PG]]/z, [x0]
-; VBITS_GE_2048: st1w { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #224]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #192]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #160]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #128]
+; NO_SVE-NEXT:    ldp q17, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q19, q18, [x0, #64]
+; NO_SVE-NEXT:    ldp q21, q20, [x0, #32]
+; NO_SVE-NEXT:    ldp q23, q22, [x0]
+; NO_SVE-NEXT:    stp q21, q20, [x1, #32]
+; NO_SVE-NEXT:    stp q19, q18, [x1, #64]
+; NO_SVE-NEXT:    stp q17, q16, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #128]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #160]
+; NO_SVE-NEXT:    stp q23, q22, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #192]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #40
+; VBITS_GE_256-NEXT:    mov x11, #32
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    mov x13, #16
+; VBITS_GE_256-NEXT:    mov x14, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p0, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: subvector_v64f32:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
+; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; VBITS_GE_2048-NEXT:    st1w { z0.s }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %a = load <64 x float>, <64 x float>* %in
   br label %bb1
 
 bb1:
   store <64 x float> %a, <64 x float>* %out
   ret void
 }
 define void @subvector_v8f64(<8 x double> *%in, <8 x double>* %out) #0 {
-; CHECK-LABEL: subvector_v8f64:
-; VBITS_GE_512: ptrue [[PG:p[0-9]+]].d, vl8
-; VBITS_GE_512: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_512: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #32]
+; NO_SVE-NEXT:    ldp q3, q2, [x0]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_512-LABEL: subvector_v8f64:
+; VBITS_GE_512:       // %bb.0:
+; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
+; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_512-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_512-NEXT:    ret
   %a = load <8 x double>, <8 x double>* %in
   br label %bb1
 
 bb1:
   store <8 x double> %a, <8 x double>* %out
   ret void
 }
 
 define void @subvector_v16f64(<16 x double> *%in, <16 x double>* %out) #0 {
-; CHECK-LABEL: subvector_v16f64:
-; VBITS_GE_1024: ptrue [[PG:p[0-9]+]].d, vl16
-; VBITS_GE_1024: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_1024: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #96]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #64]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #32]
+; NO_SVE-NEXT:    ldp q7, q6, [x0]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #32]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #64]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #12
+; VBITS_GE_256-NEXT:    mov x9, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_1024-LABEL: subvector_v16f64:
+; VBITS_GE_1024:       // %bb.0:
+; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
+; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_1024-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_1024-NEXT:    ret
   %a = load <16 x double>, <16 x double>* %in
   br label %bb1
 
 bb1:
   store <16 x double> %a, <16 x double>* %out
   ret void
 }
 
 define void @subvector_v32f64(<32 x double> *%in, <32 x double>* %out) #0 {
-; CHECK-LABEL: subvector_v32f64:
-; VBITS_GE_2048: ptrue [[PG:p[0-9]+]].d, vl32
-; VBITS_GE_2048: ld1d { [[DATA:z[0-9]+.d]] }, [[PG]]/z, [x0]
-; VBITS_GE_2048: st1d { [[DATA]] }, [[PG]], [x1]
-; CHECK: ret
+; NO_SVE-LABEL: subvector_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0, #224]
+; NO_SVE-NEXT:    ldp q3, q2, [x0, #192]
+; NO_SVE-NEXT:    ldp q5, q4, [x0, #160]
+; NO_SVE-NEXT:    ldp q7, q6, [x0, #128]
+; NO_SVE-NEXT:    ldp q17, q16, [x0, #96]
+; NO_SVE-NEXT:    ldp q19, q18, [x0, #64]
+; NO_SVE-NEXT:    ldp q21, q20, [x0, #32]
+; NO_SVE-NEXT:    ldp q23, q22, [x0]
+; NO_SVE-NEXT:    stp q21, q20, [x1, #32]
+; NO_SVE-NEXT:    stp q19, q18, [x1, #64]
+; NO_SVE-NEXT:    stp q17, q16, [x1, #96]
+; NO_SVE-NEXT:    stp q7, q6, [x1, #128]
+; NO_SVE-NEXT:    stp q5, q4, [x1, #160]
+; NO_SVE-NEXT:    stp q23, q22, [x1]
+; NO_SVE-NEXT:    stp q3, q2, [x1, #192]
+; NO_SVE-NEXT:    stp q1, q0, [x1, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: subvector_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #28
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #20
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    mov x12, #12
+; VBITS_GE_256-NEXT:    mov x13, #8
+; VBITS_GE_256-NEXT:    mov x14, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x1]
+; VBITS_GE_256-NEXT:    ret
+;
+; VBITS_GE_2048-LABEL: subvector_v32f64:
+; VBITS_GE_2048:       // %bb.0:
+; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
+; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
+; VBITS_GE_2048-NEXT:    st1d { z0.d }, p0, [x1]
+; VBITS_GE_2048-NEXT:    ret
   %a = load <32 x double>, <32 x double>* %in
   br label %bb1
 
 bb1:
   store <32 x double> %a, <32 x double>* %out
   ret void
 }
 
 define <8 x i1> @no_warn_dropped_scalable(<8 x i32>* %in) #0 {
+; NO_SVE-LABEL: no_warn_dropped_scalable:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    cmgt v1.4s, v1.4s, #0
+; NO_SVE-NEXT:    cmgt v0.4s, v0.4s, #0
+; NO_SVE-NEXT:    uzp1 v0.8h, v1.8h, v0.8h
+; NO_SVE-NEXT:    xtn v0.8b, v0.8h
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: no_warn_dropped_scalable:
-; CHECK: ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK: ld1w { [[A:z[0-9]+]].s }, [[PG]]/z, [x0]
-; CHECK: cmpgt p{{[0-9]}}.s, [[PG]]/z, [[A]].s, #0
-; CHECK: ret
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    cmpgt p0.s, p0/z, z0.s, #0
+; CHECK-NEXT:    mov z0.s, p0/z, #-1 // =0xffffffffffffffff
+; CHECK-NEXT:    uzp1 z0.h, z0.h, z0.h
+; CHECK-NEXT:    uzp1 z0.b, z0.b, z0.b
+; CHECK-NEXT:    // kill: def $d0 killed $d0 killed $z0
+; CHECK-NEXT:    ret
   %a = load <8 x i32>, <8 x i32>* %in
   br label %bb1
 
 bb1:
   %cond = icmp sgt <8 x i32> %a, zeroinitializer
   ret <8 x i1> %cond
 }
 
 ; binop(insert_subvec(a), insert_subvec(b)) -> insert_subvec(binop(a,b)) like
 ; combines remove redundant subvector operations. This test ensures it's not
 ; performed when the input idiom is the result of operation legalisation. When
 ; not prevented the test triggers infinite combine->legalise->combine->...
 define void @no_subvector_binop_hang(<8 x i32>* %in, <8 x i32>* %out, i1 %cond) #0 {
+; NO_SVE-LABEL: no_subvector_binop_hang:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    tbz w2, #0, .LBB23_2
+; NO_SVE-NEXT:  // %bb.1: // %bb.1
+; NO_SVE-NEXT:    ldp q1, q0, [x0]
+; NO_SVE-NEXT:    ldp q2, q3, [x1]
+; NO_SVE-NEXT:    orr v1.16b, v1.16b, v2.16b
+; NO_SVE-NEXT:    orr v0.16b, v0.16b, v3.16b
+; NO_SVE-NEXT:    stp q1, q0, [x1]
+; NO_SVE-NEXT:  .LBB23_2: // %bb.2
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: no_subvector_binop_hang:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue [[PG:p[0-9]+]].s, vl8
-; CHECK-NEXT:    ld1w { [[A:z[0-9]+]].s }, [[PG]]/z, [x0]
-; CHECK-NEXT:    ld1w { [[B:z[0-9]+]].s }, [[PG]]/z, [x1]
-; CHECK-NEXT:    tbz w2, #0, [[LABEL:\.[A-z0-9_]+]]
+; CHECK-NEXT:    ptrue p0.s, vl8
+; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
+; CHECK-NEXT:    tbz w2, #0, .LBB23_2
 ; CHECK-NEXT:  // %bb.1: // %bb.1
-; CHECK-NEXT:    orr [[OR:z[0-9]+]].d, [[A]].d, [[B]].d
-; CHECK-NEXT:    st1w { [[OR]].s }, [[PG]], [x1]
-; CHECK-NEXT:  [[LABEL]]: // %bb.2
+; CHECK-NEXT:    orr z0.d, z0.d, z1.d
+; CHECK-NEXT:    st1w { z0.s }, p0, [x1]
+; CHECK-NEXT:  .LBB23_2: // %bb.2
 ; CHECK-NEXT:    ret
   %a = load <8 x i32>, <8 x i32>* %in
   %b = load <8 x i32>, <8 x i32>* %out
   br i1 %cond, label %bb.1, label %bb.2
 
 bb.1:
   %or = or <8 x i32> %a, %b
   store <8 x i32> %or, <8 x i32>* %out
   br label %bb.2
 
 bb.2:
   ret void
 }
 
 attributes #0 = { "target-features"="+sve" }

llvm/test/CodeGen/AArch64/sve-fixed-length-vector-shuffle.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
 ; RUN: llc -aarch64-sve-vector-bits-min=128  < %s | FileCheck %s -check-prefix=NO_SVE
-; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_EQ_256
-; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK
+; RUN: llc -aarch64-sve-vector-bits-min=256  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
+; RUN: llc -aarch64-sve-vector-bits-min=384  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_256
 ; RUN: llc -aarch64-sve-vector-bits-min=512  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=640  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=768  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=896  < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512
 ; RUN: llc -aarch64-sve-vector-bits-min=1024 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1152 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1280 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1408 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1536 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1664 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1792 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=1920 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024
 ; RUN: llc -aarch64-sve-vector-bits-min=2048 < %s | FileCheck %s -check-prefixes=CHECK,VBITS_GE_512,VBITS_GE_1024,VBITS_GE_2048
 
 target triple = "aarch64-unknown-linux-gnu"
 
 ; Don't use SVE when its registers are no bigger than NEON.
 ; NO_SVE-NOT: ptrue
 
 ; Don't use SVE for 64-bit vectors
 define <8 x i8> @shuffle_ext_byone_v8i8(<8 x i8> %op1, <8 x i8> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v8i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.8b, v0.8b, v1.8b, #7
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v8i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.8b, v0.8b, v1.8b, #7
 ; CHECK-NEXT:    ret
   %ret = shufflevector <8 x i8> %op1, <8 x i8> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   ret <8 x i8> %ret
 }
 
 ; Don't use SVE for 128-bit vectors
 define <16 x i8> @shuffle_ext_byone_v16i8(<16 x i8> %op1, <16 x i8> %op2) {
+; NO_SVE-LABEL: shuffle_ext_byone_v16i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #15
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v16i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #15
 ; CHECK-NEXT:    ret
   %ret = shufflevector <16 x i8> %op1, <16 x i8> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                    i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   ret <16 x i8> %ret
 }
 
 define void @shuffle_ext_byone_v32i8(<32 x i8>* %a, <32 x i8>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v32i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #15
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #15
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v32i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.b, vl32
 ; CHECK-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; CHECK-NEXT:    ld1b { z1.b }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.b, z0.b[31]
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    insr z1.b, w8
 ; CHECK-NEXT:    st1b { z1.b }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <32 x i8>, <32 x i8>* %a
   %op2 = load <32 x i8>, <32 x i8>* %b
   %ret = shufflevector <32 x i8> %op1, <32 x i8> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                    i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                    i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                    i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x i8> %ret, <32 x i8>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v64i8(<64 x i8>* %a, <64 x i8>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v64i8:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov w8, #32
-; VBITS_EQ_256-NEXT:    ptrue p0.b, vl32
-; VBITS_EQ_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
-; VBITS_EQ_256-NEXT:    ld1b { z1.b }, p0/z, [x1, x8]
-; VBITS_EQ_256-NEXT:    ld1b { z2.b }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.b, z0.b[31]
-; VBITS_EQ_256-NEXT:    mov z3.b, z2.b[31]
-; VBITS_EQ_256-NEXT:    fmov w9, s0
-; VBITS_EQ_256-NEXT:    fmov w10, s3
-; VBITS_EQ_256-NEXT:    insr z2.b, w9
-; VBITS_EQ_256-NEXT:    insr z1.b, w10
-; VBITS_EQ_256-NEXT:    st1b { z2.b }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1b { z1.b }, p0, [x0, x8]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v64i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #15
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #15
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #15
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #15
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v64i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x0, x8]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x1, x8]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.b, z0.b[31]
+; VBITS_GE_256-NEXT:    mov z3.b, z2.b[31]
+; VBITS_GE_256-NEXT:    fmov w9, s0
+; VBITS_GE_256-NEXT:    fmov w10, s3
+; VBITS_GE_256-NEXT:    insr z2.b, w9
+; VBITS_GE_256-NEXT:    insr z1.b, w10
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1b { z1.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v64i8:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.b, vl64
 ; VBITS_GE_512-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1b { z1.b }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.b, z0.b[63]
 ; VBITS_GE_512-NEXT:    fmov w8, s0
%ret = shufflevector <64 x i8> %op1, <64 x i8> %op2, <64 x i32> <i32 63, i32 64, i32 65, i32 66, i32 67, i32 68, i32 69, i32 70,
                                                                    i32 103, i32 104, i32 105, i32 106, i32 107, i32 108, i32 109, i32 110,
                                                                    i32 111, i32 112, i32 113, i32 114, i32 115, i32 116, i32 117, i32 118,
                                                                    i32 119, i32 120, i32 121, i32 122, i32 123, i32 124, i32 125, i32 126>
   store <64 x i8> %ret, <64 x i8>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v128i8(<128 x i8>* %a, <128 x i8>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v128i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #15
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #15
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #15
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #15
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #15
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #15
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #15
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #15
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v128i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #64
+; VBITS_GE_256-NEXT:    mov w10, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    mov w9, #96
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x1, x8]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x1, x10]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    ld1b { z3.b }, p0/z, [x1, x9]
+; VBITS_GE_256-NEXT:    ld1b { z4.b }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.b, z0.b[31]
+; VBITS_GE_256-NEXT:    fmov w11, s5
+; VBITS_GE_256-NEXT:    mov z5.b, z2.b[31]
+; VBITS_GE_256-NEXT:    mov z1.b, z1.b[31]
+; VBITS_GE_256-NEXT:    fmov w12, s5
+; VBITS_GE_256-NEXT:    mov z5.b, z4.b[31]
+; VBITS_GE_256-NEXT:    fmov w13, s1
+; VBITS_GE_256-NEXT:    fmov w14, s5
+; VBITS_GE_256-NEXT:    insr z3.b, w11
+; VBITS_GE_256-NEXT:    insr z0.b, w12
+; VBITS_GE_256-NEXT:    st1b { z3.b }, p0, [x0, x9]
+; VBITS_GE_256-NEXT:    insr z4.b, w13
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    insr z2.b, w14
+; VBITS_GE_256-NEXT:    st1b { z4.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x0, x10]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v128i8:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.b, vl128
 ; VBITS_GE_1024-NEXT:    mov w8, #127
 ; VBITS_GE_1024-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1b { z1.b }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.b, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb w8, p1, z0.b
%ret = shufflevector <128 x i8> %op1, <128 x i8> %op2, <128 x i32> <i32 127,  i32 128,  i32 129,  i32 130,  i32 131,  i32 132,  i32 133,  i32 134,
                                                                       i32 231,  i32 232,  i32 233,  i32 234,  i32 235,  i32 236,  i32 237,  i32 238,
                                                                       i32 239,  i32 240,  i32 241,  i32 242,  i32 243,  i32 244,  i32 245,  i32 246,
                                                                       i32 247,  i32 248,  i32 249,  i32 250,  i32 251,  i32 252,  i32 253,  i32 254>
   store <128 x i8> %ret, <128 x i8>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v256i8(<256 x i8>* %a, <256 x i8>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v256i8:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #15
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #15
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #15
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #15
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #15
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #15
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #15
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #15
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #15
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #15
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #15
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #15
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #15
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #15
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #15
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #15
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v256i8:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov w8, #32
+; VBITS_GE_256-NEXT:    ptrue p0.b, vl32
+; VBITS_GE_256-NEXT:    mov w11, #128
+; VBITS_GE_256-NEXT:    mov w13, #64
+; VBITS_GE_256-NEXT:    mov w12, #96
+; VBITS_GE_256-NEXT:    mov w14, #160
+; VBITS_GE_256-NEXT:    ld1b { z0.b }, p0/z, [x1, x8]
+; VBITS_GE_256-NEXT:    mov w10, #192
+; VBITS_GE_256-NEXT:    ld1b { z3.b }, p0/z, [x1, x11]
+; VBITS_GE_256-NEXT:    ld1b { z5.b }, p0/z, [x1, x13]
+; VBITS_GE_256-NEXT:    mov w9, #224
+; VBITS_GE_256-NEXT:    ld1b { z7.b }, p0/z, [x1, x12]
+; VBITS_GE_256-NEXT:    ld1b { z4.b }, p0/z, [x1, x10]
+; VBITS_GE_256-NEXT:    mov z6.b, z0.b[31]
+; VBITS_GE_256-NEXT:    fmov w15, s6
+; VBITS_GE_256-NEXT:    ld1b { z6.b }, p0/z, [x1, x14]
+; VBITS_GE_256-NEXT:    mov z16.b, z3.b[31]
+; VBITS_GE_256-NEXT:    ld1b { z1.b }, p0/z, [x0, x9]
+; VBITS_GE_256-NEXT:    ld1b { z2.b }, p0/z, [x1, x9]
+; VBITS_GE_256-NEXT:    ld1b { z17.b }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fmov w16, s16
+; VBITS_GE_256-NEXT:    mov z16.b, z5.b[31]
+; VBITS_GE_256-NEXT:    insr z5.b, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.b, z7.b[31]
+; VBITS_GE_256-NEXT:    mov z1.b, z1.b[31]
+; VBITS_GE_256-NEXT:    fmov w17, s16
+; VBITS_GE_256-NEXT:    mov z16.b, z6.b[31]
+; VBITS_GE_256-NEXT:    fmov w18, s16
+; VBITS_GE_256-NEXT:    mov z16.b, z4.b[31]
+; VBITS_GE_256-NEXT:    insr z7.b, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.b, z17.b[31]
+; VBITS_GE_256-NEXT:    fmov w1, s1
+; VBITS_GE_256-NEXT:    fmov w2, s16
+; VBITS_GE_256-NEXT:    insr z3.b, w17
+; VBITS_GE_256-NEXT:    insr z6.b, w16
+; VBITS_GE_256-NEXT:    insr z4.b, w18
+; VBITS_GE_256-NEXT:    insr z2.b, w15
+; VBITS_GE_256-NEXT:    insr z17.b, w1
+; VBITS_GE_256-NEXT:    insr z0.b, w2
+; VBITS_GE_256-NEXT:    st1b { z2.b }, p0, [x0, x9]
+; VBITS_GE_256-NEXT:    st1b { z0.b }, p0, [x0, x8]
+; VBITS_GE_256-NEXT:    st1b { z4.b }, p0, [x0, x10]
+; VBITS_GE_256-NEXT:    st1b { z3.b }, p0, [x0, x11]
+; VBITS_GE_256-NEXT:    st1b { z7.b }, p0, [x0, x12]
+; VBITS_GE_256-NEXT:    st1b { z5.b }, p0, [x0, x13]
+; VBITS_GE_256-NEXT:    st1b { z6.b }, p0, [x0, x14]
+; VBITS_GE_256-NEXT:    st1b { z17.b }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v256i8:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.b, vl256
 ; VBITS_GE_2048-NEXT:    mov w8, #255
 ; VBITS_GE_2048-NEXT:    ld1b { z0.b }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1b { z1.b }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.b, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb w8, p1, z0.b
%ret = shufflevector <256 x i8> %op1, <256 x i8> %op2, <256 x i32> <i32 255,  i32 256,  i32 257,  i32 258,  i32 259,  i32 260,  i32 261,  i32 262,
                                                                       i32 495,  i32 496,  i32 497,  i32 498,  i32 499,  i32 500,  i32 501,  i32 502,
                                                                       i32 503,  i32 504,  i32 505,  i32 506,  i32 507,  i32 508,  i32 509,  i32 510>
   store <256 x i8> %ret, <256 x i8>* %a
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors
 define <4 x i16> @shuffle_ext_byone_v4i16(<4 x i16> %op1, <4 x i16> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.8b, v0.8b, v1.8b, #6
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.8b, v0.8b, v1.8b, #6
 ; CHECK-NEXT:    ret
   %ret = shufflevector <4 x i16> %op1, <4 x i16> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   ret <4 x i16> %ret
 }
 
 ; Don't use SVE for 128-bit vectors
 define <8 x i16> @shuffle_ext_byone_v8i16(<8 x i16> %op1, <8 x i16> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v8i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v8i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
 ; CHECK-NEXT:    ret
   %ret = shufflevector <8 x i16> %op1, <8 x i16> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   ret <8 x i16> %ret
 }
 
 define void @shuffle_ext_byone_v16i16(<16 x i16>* %a, <16 x i16>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v16i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #14
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v16i16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.h, z0.h[15]
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    insr z1.h, w8
 ; CHECK-NEXT:    st1h { z1.h }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x i16>, <16 x i16>* %a
   %op2 = load <16 x i16>, <16 x i16>* %b
   %ret = shufflevector <16 x i16> %op1, <16 x i16> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                      i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x i16> %ret, <16 x i16>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32i16(<32 x i16>* %a, <32 x i16>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v32i16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z2.h }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.h, z0.h[15]
-; VBITS_EQ_256-NEXT:    mov z3.h, z2.h[15]
-; VBITS_EQ_256-NEXT:    fmov w9, s0
-; VBITS_EQ_256-NEXT:    fmov w10, s3
-; VBITS_EQ_256-NEXT:    insr z2.h, w9
-; VBITS_EQ_256-NEXT:    insr z1.h, w10
-; VBITS_EQ_256-NEXT:    st1h { z2.h }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1h { z1.h }, p0, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v32i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #14
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #14
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.h, z0.h[15]
+; VBITS_GE_256-NEXT:    mov z3.h, z2.h[15]
+; VBITS_GE_256-NEXT:    fmov w9, s0
+; VBITS_GE_256-NEXT:    fmov w10, s3
+; VBITS_GE_256-NEXT:    insr z2.h, w9
+; VBITS_GE_256-NEXT:    insr z1.h, w10
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v32i16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.h, z0.h[31]
 ; VBITS_GE_512-NEXT:    fmov w8, s0
 ; VBITS_GE_512-NEXT:    insr z1.h, w8
 ; VBITS_GE_512-NEXT:    st1h { z1.h }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <32 x i16>, <32 x i16>* %a
   %op2 = load <32 x i16>, <32 x i16>* %b
   %ret = shufflevector <32 x i16> %op1, <32 x i16> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                      i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                      i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                      i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x i16> %ret, <32 x i16>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v64i16(<64 x i16>* %a, <64 x i16>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v64i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #14
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #14
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #14
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #14
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v64i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.h, z0.h[15]
+; VBITS_GE_256-NEXT:    fmov w11, s5
+; VBITS_GE_256-NEXT:    mov z5.h, z2.h[15]
+; VBITS_GE_256-NEXT:    mov z1.h, z1.h[15]
+; VBITS_GE_256-NEXT:    fmov w12, s5
+; VBITS_GE_256-NEXT:    mov z5.h, z4.h[15]
+; VBITS_GE_256-NEXT:    fmov w13, s1
+; VBITS_GE_256-NEXT:    fmov w14, s5
+; VBITS_GE_256-NEXT:    insr z3.h, w11
+; VBITS_GE_256-NEXT:    insr z0.h, w12
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    insr z4.h, w13
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    insr z2.h, w14
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v64i16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    mov w8, #63
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.h, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb w8, p1, z0.h
%ret = shufflevector <64 x i16> %op1, <64 x i16> %op2, <64 x i32> <i32 63, i32 64, i32 65, i32 66, i32 67, i32 68, i32 69, i32 70,
                                                                      i32 103, i32 104, i32 105, i32 106, i32 107, i32 108, i32 109, i32 110,
                                                                      i32 111, i32 112, i32 113, i32 114, i32 115, i32 116, i32 117, i32 118,
                                                                      i32 119, i32 120, i32 121, i32 122, i32 123, i32 124, i32 125, i32 126>
   store <64 x i16> %ret, <64 x i16>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v128i16(<128 x i16>* %a, <128 x i16>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v128i16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #14
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #14
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #14
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #14
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #14
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #14
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #14
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #14
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #14
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #14
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #14
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v128i16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x10, #64
+; VBITS_GE_256-NEXT:    mov x13, #32
+; VBITS_GE_256-NEXT:    mov x14, #48
+; VBITS_GE_256-NEXT:    mov x11, #80
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    mov x12, #96
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z5.h }, p0/z, [x1, x13, lsl #1]
+; VBITS_GE_256-NEXT:    mov x9, #112
+; VBITS_GE_256-NEXT:    ld1h { z7.h }, p0/z, [x1, x14, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x1, x12, lsl #1]
+; VBITS_GE_256-NEXT:    mov z6.h, z0.h[15]
+; VBITS_GE_256-NEXT:    fmov w15, s6
+; VBITS_GE_256-NEXT:    ld1h { z6.h }, p0/z, [x1, x11, lsl #1]
+; VBITS_GE_256-NEXT:    mov z16.h, z2.h[15]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z17.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fmov w16, s16
+; VBITS_GE_256-NEXT:    mov z16.h, z5.h[15]
+; VBITS_GE_256-NEXT:    insr z5.h, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.h, z7.h[15]
+; VBITS_GE_256-NEXT:    mov z1.h, z1.h[15]
+; VBITS_GE_256-NEXT:    fmov w17, s16
+; VBITS_GE_256-NEXT:    mov z16.h, z6.h[15]
+; VBITS_GE_256-NEXT:    fmov w18, s16
+; VBITS_GE_256-NEXT:    mov z16.h, z4.h[15]
+; VBITS_GE_256-NEXT:    insr z7.h, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.h, z17.h[15]
+; VBITS_GE_256-NEXT:    fmov w1, s1
+; VBITS_GE_256-NEXT:    fmov w2, s16
+; VBITS_GE_256-NEXT:    insr z2.h, w17
+; VBITS_GE_256-NEXT:    insr z6.h, w16
+; VBITS_GE_256-NEXT:    insr z4.h, w18
+; VBITS_GE_256-NEXT:    insr z3.h, w15
+; VBITS_GE_256-NEXT:    insr z17.h, w1
+; VBITS_GE_256-NEXT:    insr z0.h, w2
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x0, x14, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z5.h }, p0, [x0, x13, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z6.h }, p0, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z17.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v128i16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    mov w8, #127
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.h, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb w8, p1, z0.h
%ret = shufflevector <128 x i16> %op1, <128 x i16> %op2, <128 x i32> <i32 127,  i32 128,  i32 129,  i32 130,  i32 131,  i32 132,  i32 133,  i32 134,
                                                                         i32 239,  i32 240,  i32 241,  i32 242,  i32 243,  i32 244,  i32 245,  i32 246,
                                                                         i32 247,  i32 248,  i32 249,  i32 250,  i32 251,  i32 252,  i32 253,  i32 254>
   store <128 x i16> %ret, <128 x i16>* %a
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors
 define <2 x i32> @shuffle_ext_byone_v2i32(<2 x i32> %op1, <2 x i32> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v2i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.8b, v0.8b, v1.8b, #4
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v2i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.8b, v0.8b, v1.8b, #4
 ; CHECK-NEXT:    ret
   %ret = shufflevector <2 x i32> %op1, <2 x i32> %op2, <2 x i32> <i32 1, i32 2>
   ret <2 x i32> %ret
 }
 
 ; Don't use SVE for 128-bit vectors
 define <4 x i32> @shuffle_ext_byone_v4i32(<4 x i32> %op1, <4 x i32> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
 ; CHECK-NEXT:    ret
   %ret = shufflevector <4 x i32> %op1, <4 x i32> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   ret <4 x i32> %ret
 }
 
 define void @shuffle_ext_byone_v8i32(<8 x i32>* %a, <8 x i32>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v8i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #12
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v8i32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.s, z0.s[7]
 ; CHECK-NEXT:    fmov w8, s0
 ; CHECK-NEXT:    insr z1.s, w8
 ; CHECK-NEXT:    st1w { z1.s }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <8 x i32>, <8 x i32>* %a
   %op2 = load <8 x i32>, <8 x i32>* %b
   %ret = shufflevector <8 x i32> %op1, <8 x i32> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   store <8 x i32> %ret, <8 x i32>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v16i32(<16 x i32>* %a, <16 x i32>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v16i32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z2.s }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.s, z0.s[7]
-; VBITS_EQ_256-NEXT:    mov z3.s, z2.s[7]
-; VBITS_EQ_256-NEXT:    fmov w9, s0
-; VBITS_EQ_256-NEXT:    fmov w10, s3
-; VBITS_EQ_256-NEXT:    insr z2.s, w9
-; VBITS_EQ_256-NEXT:    insr z1.s, w10
-; VBITS_EQ_256-NEXT:    st1w { z2.s }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1w { z1.s }, p0, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v16i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #12
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #12
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v16i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.s, z0.s[7]
+; VBITS_GE_256-NEXT:    mov z3.s, z2.s[7]
+; VBITS_GE_256-NEXT:    fmov w9, s0
+; VBITS_GE_256-NEXT:    fmov w10, s3
+; VBITS_GE_256-NEXT:    insr z2.s, w9
+; VBITS_GE_256-NEXT:    insr z1.s, w10
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v16i32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.s, z0.s[15]
 ; VBITS_GE_512-NEXT:    fmov w8, s0
 ; VBITS_GE_512-NEXT:    insr z1.s, w8
 ; VBITS_GE_512-NEXT:    st1w { z1.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <16 x i32>, <16 x i32>* %a
   %op2 = load <16 x i32>, <16 x i32>* %b
   %ret = shufflevector <16 x i32> %op1, <16 x i32> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                      i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x i32> %ret, <16 x i32>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32i32(<32 x i32>* %a, <32 x i32>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v32i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #12
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #12
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #12
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #12
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.s, z0.s[7]
+; VBITS_GE_256-NEXT:    fmov w11, s5
+; VBITS_GE_256-NEXT:    mov z5.s, z2.s[7]
+; VBITS_GE_256-NEXT:    mov z1.s, z1.s[7]
+; VBITS_GE_256-NEXT:    fmov w12, s5
+; VBITS_GE_256-NEXT:    mov z5.s, z4.s[7]
+; VBITS_GE_256-NEXT:    fmov w13, s1
+; VBITS_GE_256-NEXT:    fmov w14, s5
+; VBITS_GE_256-NEXT:    insr z3.s, w11
+; VBITS_GE_256-NEXT:    insr z0.s, w12
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    insr z4.s, w13
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    insr z2.s, w14
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v32i32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    mov w8, #31
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.s, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb w8, p1, z0.s
 ; VBITS_GE_1024-NEXT:    insr z1.s, w8
 ; VBITS_GE_1024-NEXT:    st1w { z1.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <32 x i32>, <32 x i32>* %a
   %op2 = load <32 x i32>, <32 x i32>* %b
   %ret = shufflevector <32 x i32> %op1, <32 x i32> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                      i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                      i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                      i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x i32> %ret, <32 x i32>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v64i32(<64 x i32>* %a, <64 x i32>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v64i32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #12
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #12
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #12
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #12
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #12
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #12
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #12
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #12
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #12
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #12
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #12
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v64i32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x10, #32
+; VBITS_GE_256-NEXT:    mov x13, #16
+; VBITS_GE_256-NEXT:    mov x14, #24
+; VBITS_GE_256-NEXT:    mov x11, #40
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    mov x12, #48
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    mov x9, #56
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    mov z6.s, z0.s[7]
+; VBITS_GE_256-NEXT:    fmov w15, s6
+; VBITS_GE_256-NEXT:    ld1w { z6.s }, p0/z, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    mov z16.s, z2.s[7]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fmov w16, s16
+; VBITS_GE_256-NEXT:    mov z16.s, z5.s[7]
+; VBITS_GE_256-NEXT:    insr z5.s, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.s, z7.s[7]
+; VBITS_GE_256-NEXT:    mov z1.s, z1.s[7]
+; VBITS_GE_256-NEXT:    fmov w17, s16
+; VBITS_GE_256-NEXT:    mov z16.s, z6.s[7]
+; VBITS_GE_256-NEXT:    fmov w18, s16
+; VBITS_GE_256-NEXT:    mov z16.s, z4.s[7]
+; VBITS_GE_256-NEXT:    insr z7.s, w15
+; VBITS_GE_256-NEXT:    fmov w15, s16
+; VBITS_GE_256-NEXT:    mov z16.s, z17.s[7]
+; VBITS_GE_256-NEXT:    fmov w1, s1
+; VBITS_GE_256-NEXT:    fmov w2, s16
+; VBITS_GE_256-NEXT:    insr z2.s, w17
+; VBITS_GE_256-NEXT:    insr z6.s, w16
+; VBITS_GE_256-NEXT:    insr z4.s, w18
+; VBITS_GE_256-NEXT:    insr z3.s, w15
+; VBITS_GE_256-NEXT:    insr z17.s, w1
+; VBITS_GE_256-NEXT:    insr z0.s, w2
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z6.s }, p0, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z17.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v64i32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    mov w8, #63
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.s, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb w8, p1, z0.s
%ret = shufflevector <64 x i32> %op1, <64 x i32> %op2, <64 x i32> <i32 63, i32 64, i32 65, i32 66, i32 67, i32 68, i32 69, i32 70,
                                                                      i32 111, i32 112, i32 113, i32 114, i32 115, i32 116, i32 117, i32 118,
                                                                      i32 119, i32 120, i32 121, i32 122, i32 123, i32 124, i32 125, i32 126>
   store <64 x i32> %ret, <64 x i32>* %a
   ret void
 }
 
 ; Don't use SVE for 128-bit vectors
 define <2 x i64> @shuffle_ext_byone_v2i64(<2 x i64> %op1, <2 x i64> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v2i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v2i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
 ; CHECK-NEXT:    ret
   %ret = shufflevector <2 x i64> %op1, <2 x i64> %op2, <2 x i32> <i32 1, i32 2>
   ret <2 x i64> %ret
 }
 
 define void @shuffle_ext_byone_v4i64(<4 x i64>* %a, <4 x i64>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #8
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4i64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.d, z0.d[3]
 ; CHECK-NEXT:    fmov x8, d0
 ; CHECK-NEXT:    insr z1.d, x8
 ; CHECK-NEXT:    st1d { z1.d }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <4 x i64>, <4 x i64>* %a
   %op2 = load <4 x i64>, <4 x i64>* %b
   %ret = shufflevector <4 x i64> %op1, <4 x i64> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   store <4 x i64> %ret, <4 x i64>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v8i64(<8 x i64>* %a, <8 x i64>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v8i64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.d, z0.d[3]
-; VBITS_EQ_256-NEXT:    mov z3.d, z2.d[3]
-; VBITS_EQ_256-NEXT:    fmov x9, d0
-; VBITS_EQ_256-NEXT:    fmov x10, d3
-; VBITS_EQ_256-NEXT:    insr z2.d, x9
-; VBITS_EQ_256-NEXT:    insr z1.d, x10
-; VBITS_EQ_256-NEXT:    st1d { z2.d }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1d { z1.d }, p0, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v8i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #8
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #8
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v8i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.d, z0.d[3]
+; VBITS_GE_256-NEXT:    mov z3.d, z2.d[3]
+; VBITS_GE_256-NEXT:    fmov x9, d0
+; VBITS_GE_256-NEXT:    fmov x10, d3
+; VBITS_GE_256-NEXT:    insr z2.d, x9
+; VBITS_GE_256-NEXT:    insr z1.d, x10
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v8i64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.d, z0.d[7]
 ; VBITS_GE_512-NEXT:    fmov x8, d0
 ; VBITS_GE_512-NEXT:    insr z1.d, x8
 ; VBITS_GE_512-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <8 x i64>, <8 x i64>* %a
   %op2 = load <8 x i64>, <8 x i64>* %b
   %ret = shufflevector <8 x i64> %op1, <8 x i64> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   store <8 x i64> %ret, <8 x i64>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v16i64(<16 x i64>* %a, <16 x i64>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v16i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #8
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #8
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #8
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #8
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v16i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.d, z0.d[3]
+; VBITS_GE_256-NEXT:    fmov x11, d5
+; VBITS_GE_256-NEXT:    mov z5.d, z2.d[3]
+; VBITS_GE_256-NEXT:    mov z1.d, z1.d[3]
+; VBITS_GE_256-NEXT:    fmov x12, d5
+; VBITS_GE_256-NEXT:    mov z5.d, z4.d[3]
+; VBITS_GE_256-NEXT:    fmov x13, d1
+; VBITS_GE_256-NEXT:    fmov x14, d5
+; VBITS_GE_256-NEXT:    insr z3.d, x11
+; VBITS_GE_256-NEXT:    insr z0.d, x12
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    insr z4.d, x13
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    insr z2.d, x14
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v16i64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    mov w8, #15
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.d, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb x8, p1, z0.d
 ; VBITS_GE_1024-NEXT:    insr z1.d, x8
 ; VBITS_GE_1024-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <16 x i64>, <16 x i64>* %a
   %op2 = load <16 x i64>, <16 x i64>* %b
   %ret = shufflevector <16 x i64> %op1, <16 x i64> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                      i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x i64> %ret, <16 x i64>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32i64(<32 x i64>* %a, <32 x i64>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v32i64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #8
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #8
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #8
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #8
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #8
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #8
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #8
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #8
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #8
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #8
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #8
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32i64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x13, #8
+; VBITS_GE_256-NEXT:    mov x14, #12
+; VBITS_GE_256-NEXT:    mov x11, #20
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    mov x12, #24
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    mov x9, #28
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    mov z6.d, z0.d[3]
+; VBITS_GE_256-NEXT:    fmov x15, d6
+; VBITS_GE_256-NEXT:    ld1d { z6.d }, p0/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    mov z16.d, z2.d[3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    fmov x16, d16
+; VBITS_GE_256-NEXT:    mov z16.d, z5.d[3]
+; VBITS_GE_256-NEXT:    insr z5.d, x15
+; VBITS_GE_256-NEXT:    fmov x15, d16
+; VBITS_GE_256-NEXT:    mov z16.d, z7.d[3]
+; VBITS_GE_256-NEXT:    mov z1.d, z1.d[3]
+; VBITS_GE_256-NEXT:    fmov x17, d16
+; VBITS_GE_256-NEXT:    mov z16.d, z6.d[3]
+; VBITS_GE_256-NEXT:    fmov x18, d16
+; VBITS_GE_256-NEXT:    mov z16.d, z4.d[3]
+; VBITS_GE_256-NEXT:    insr z7.d, x15
+; VBITS_GE_256-NEXT:    fmov x15, d16
+; VBITS_GE_256-NEXT:    mov z16.d, z17.d[3]
+; VBITS_GE_256-NEXT:    fmov x1, d1
+; VBITS_GE_256-NEXT:    fmov x2, d16
+; VBITS_GE_256-NEXT:    insr z2.d, x17
+; VBITS_GE_256-NEXT:    insr z6.d, x16
+; VBITS_GE_256-NEXT:    insr z4.d, x18
+; VBITS_GE_256-NEXT:    insr z3.d, x15
+; VBITS_GE_256-NEXT:    insr z17.d, x1
+; VBITS_GE_256-NEXT:    insr z0.d, x2
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z6.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z17.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v32i64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    mov w8, #31
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.d, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb x8, p1, z0.d
 ; VBITS_GE_2048-NEXT:    insr z1.d, x8
 ; VBITS_GE_2048-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <32 x i64>, <32 x i64>* %a
   %op2 = load <32 x i64>, <32 x i64>* %b
   %ret = shufflevector <32 x i64> %op1, <32 x i64> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                      i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                      i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                      i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x i64> %ret, <32 x i64>* %a
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors
 define <4 x half> @shuffle_ext_byone_v4f16(<4 x half> %op1, <4 x half> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.8b, v0.8b, v1.8b, #6
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.8b, v0.8b, v1.8b, #6
 ; CHECK-NEXT:    ret
   %ret = shufflevector <4 x half> %op1, <4 x half> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   ret <4 x half> %ret
 }
 
 ; Don't use SVE for 128-bit vectors
 define <8 x half> @shuffle_ext_byone_v8f16(<8 x half> %op1, <8 x half> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v8f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v8f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
 ; CHECK-NEXT:    ret
   %ret = shufflevector <8 x half> %op1, <8 x half> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   ret <8 x half> %ret
 }
 
 define void @shuffle_ext_byone_v16f16(<16 x half>* %a, <16 x half>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v16f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #14
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v16f16:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.h, vl16
 ; CHECK-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.h, z0.h[15]
 ; CHECK-NEXT:    insr z1.h, h0
 ; CHECK-NEXT:    st1h { z1.h }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <16 x half>, <16 x half>* %a
   %op2 = load <16 x half>, <16 x half>* %b
   %ret = shufflevector <16 x half> %op1, <16 x half> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                        i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x half> %ret, <16 x half>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32f16(<32 x half>* %a, <32 x half>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v32f16:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #16
-; VBITS_EQ_256-NEXT:    ptrue p0.h, vl16
-; VBITS_EQ_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ld1h { z2.h }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.h, z0.h[15]
-; VBITS_EQ_256-NEXT:    mov z3.h, z2.h[15]
-; VBITS_EQ_256-NEXT:    insr z2.h, h0
-; VBITS_EQ_256-NEXT:    insr z1.h, h3
-; VBITS_EQ_256-NEXT:    st1h { z2.h }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1h { z1.h }, p0, [x0, x8, lsl #1]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v32f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #14
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #14
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.h, z0.h[15]
+; VBITS_GE_256-NEXT:    mov z3.h, z2.h[15]
+; VBITS_GE_256-NEXT:    insr z2.h, h0
+; VBITS_GE_256-NEXT:    insr z1.h, h3
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v32f16:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.h, vl32
 ; VBITS_GE_512-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.h, z0.h[31]
 ; VBITS_GE_512-NEXT:    insr z1.h, h0
 ; VBITS_GE_512-NEXT:    st1h { z1.h }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <32 x half>, <32 x half>* %a
   %op2 = load <32 x half>, <32 x half>* %b
   %ret = shufflevector <32 x half> %op1, <32 x half> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                        i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                        i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                        i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x half> %ret, <32 x half>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v64f16(<64 x half>* %a, <64 x half>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v64f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #14
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #14
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #14
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #14
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v64f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #32
+; VBITS_GE_256-NEXT:    mov x9, #48
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.h, z0.h[15]
+; VBITS_GE_256-NEXT:    insr z1.h, h5
+; VBITS_GE_256-NEXT:    mov z5.h, z3.h[15]
+; VBITS_GE_256-NEXT:    mov z2.h, z2.h[15]
+; VBITS_GE_256-NEXT:    insr z0.h, h5
+; VBITS_GE_256-NEXT:    mov z5.h, z4.h[15]
+; VBITS_GE_256-NEXT:    insr z4.h, h2
+; VBITS_GE_256-NEXT:    insr z3.h, h5
+; VBITS_GE_256-NEXT:    st1h { z0.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v64f16:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.h, vl64
 ; VBITS_GE_1024-NEXT:    mov w8, #63
 ; VBITS_GE_1024-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.h, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb h0, p1, z0.h
%ret = shufflevector <64 x half> %op1, <64 x half> %op2, <64 x i32> <i32 63, i32 64, i32 65, i32 66, i32 67, i32 68, i32 69, i32 70,
                                                                        i32 103, i32 104, i32 105, i32 106, i32 107, i32 108, i32 109, i32 110,
                                                                        i32 111, i32 112, i32 113, i32 114, i32 115, i32 116, i32 117, i32 118,
                                                                        i32 119, i32 120, i32 121, i32 122, i32 123, i32 124, i32 125, i32 126>
   store <64 x half> %ret, <64 x half>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v128f16(<128 x half>* %a, <128 x half>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v128f16:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #14
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #14
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #14
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #14
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #14
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #14
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #14
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #14
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #14
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #14
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #14
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #14
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #14
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #14
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #14
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v128f16:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x10, #64
+; VBITS_GE_256-NEXT:    mov x9, #80
+; VBITS_GE_256-NEXT:    mov x11, #16
+; VBITS_GE_256-NEXT:    ptrue p0.h, vl16
+; VBITS_GE_256-NEXT:    mov x12, #32
+; VBITS_GE_256-NEXT:    mov x13, #48
+; VBITS_GE_256-NEXT:    mov x8, #112
+; VBITS_GE_256-NEXT:    mov x14, #96
+; VBITS_GE_256-NEXT:    ld1h { z3.h }, p0/z, [x1, x10, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z1.h }, p0/z, [x1, x9, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z5.h }, p0/z, [x1, x11, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z4.h }, p0/z, [x1, x12, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z16.h }, p0/z, [x1, x13, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z0.h }, p0/z, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z2.h }, p0/z, [x1, x8, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z7.h }, p0/z, [x1, x14, lsl #1]
+; VBITS_GE_256-NEXT:    ld1h { z17.h }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z18.h, z3.h[15]
+; VBITS_GE_256-NEXT:    mov z6.h, z1.h[15]
+; VBITS_GE_256-NEXT:    insr z1.h, h18
+; VBITS_GE_256-NEXT:    mov z18.h, z5.h[15]
+; VBITS_GE_256-NEXT:    mov z19.h, z4.h[15]
+; VBITS_GE_256-NEXT:    insr z4.h, h18
+; VBITS_GE_256-NEXT:    mov z18.h, z16.h[15]
+; VBITS_GE_256-NEXT:    insr z3.h, h18
+; VBITS_GE_256-NEXT:    mov z18.h, z7.h[15]
+; VBITS_GE_256-NEXT:    insr z7.h, h6
+; VBITS_GE_256-NEXT:    mov z0.h, z0.h[15]
+; VBITS_GE_256-NEXT:    mov z6.h, z17.h[15]
+; VBITS_GE_256-NEXT:    insr z16.h, h19
+; VBITS_GE_256-NEXT:    insr z2.h, h18
+; VBITS_GE_256-NEXT:    insr z17.h, h0
+; VBITS_GE_256-NEXT:    insr z5.h, h6
+; VBITS_GE_256-NEXT:    st1h { z2.h }, p0, [x0, x8, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z7.h }, p0, [x0, x14, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z3.h }, p0, [x0, x10, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z16.h }, p0, [x0, x13, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z4.h }, p0, [x0, x12, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z5.h }, p0, [x0, x11, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z1.h }, p0, [x0, x9, lsl #1]
+; VBITS_GE_256-NEXT:    st1h { z17.h }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v128f16:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.h, vl128
 ; VBITS_GE_2048-NEXT:    mov w8, #127
 ; VBITS_GE_2048-NEXT:    ld1h { z0.h }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1h { z1.h }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.h, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb h0, p1, z0.h
%ret = shufflevector <128 x half> %op1, <128 x half> %op2, <128 x i32> <i32 127,  i32 128,  i32 129,  i32 130,  i32 131,  i32 132,  i32 133,  i32 134,
                                                                           i32 239,  i32 240,  i32 241,  i32 242,  i32 243,  i32 244,  i32 245,  i32 246,
                                                                           i32 247,  i32 248,  i32 249,  i32 250,  i32 251,  i32 252,  i32 253,  i32 254>
   store <128 x half> %ret, <128 x half>* %a
   ret void
 }
 
 ; Don't use SVE for 64-bit vectors
 define <2 x float> @shuffle_ext_byone_v2f32(<2 x float> %op1, <2 x float> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v2f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.8b, v0.8b, v1.8b, #4
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v2f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.8b, v0.8b, v1.8b, #4
 ; CHECK-NEXT:    ret
   %ret = shufflevector <2 x float> %op1, <2 x float> %op2, <2 x i32> <i32 1, i32 2>
   ret <2 x float> %ret
 }
 
 ; Don't use SVE for 128-bit vectors
 define <4 x float> @shuffle_ext_byone_v4f32(<4 x float> %op1, <4 x float> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
 ; CHECK-NEXT:    ret
   %ret = shufflevector <4 x float> %op1, <4 x float> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   ret <4 x float> %ret
 }
 
 define void @shuffle_ext_byone_v8f32(<8 x float>* %a, <8 x float>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v8f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #12
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v8f32:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.s, vl8
 ; CHECK-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.s, z0.s[7]
 ; CHECK-NEXT:    insr z1.s, s0
 ; CHECK-NEXT:    st1w { z1.s }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <8 x float>, <8 x float>* %a
   %op2 = load <8 x float>, <8 x float>* %b
   %ret = shufflevector <8 x float> %op1, <8 x float> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   store <8 x float> %ret, <8 x float>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v16f32(<16 x float>* %a, <16 x float>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v16f32:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #8
-; VBITS_EQ_256-NEXT:    ptrue p0.s, vl8
-; VBITS_EQ_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ld1w { z2.s }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.s, z0.s[7]
-; VBITS_EQ_256-NEXT:    mov z3.s, z2.s[7]
-; VBITS_EQ_256-NEXT:    insr z2.s, s0
-; VBITS_EQ_256-NEXT:    insr z1.s, s3
-; VBITS_EQ_256-NEXT:    st1w { z2.s }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1w { z1.s }, p0, [x0, x8, lsl #2]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v16f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #12
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #12
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v16f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.s, z0.s[7]
+; VBITS_GE_256-NEXT:    mov z3.s, z2.s[7]
+; VBITS_GE_256-NEXT:    insr z2.s, s0
+; VBITS_GE_256-NEXT:    insr z1.s, s3
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v16f32:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.s, vl16
 ; VBITS_GE_512-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.s, z0.s[15]
 ; VBITS_GE_512-NEXT:    insr z1.s, s0
 ; VBITS_GE_512-NEXT:    st1w { z1.s }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <16 x float>, <16 x float>* %a
   %op2 = load <16 x float>, <16 x float>* %b
   %ret = shufflevector <16 x float> %op1, <16 x float> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                          i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x float> %ret, <16 x float>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32f32(<32 x float>* %a, <32 x float>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v32f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #12
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #12
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #12
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #12
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #16
+; VBITS_GE_256-NEXT:    mov x9, #24
+; VBITS_GE_256-NEXT:    mov x10, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.s, z0.s[7]
+; VBITS_GE_256-NEXT:    insr z1.s, s5
+; VBITS_GE_256-NEXT:    mov z5.s, z3.s[7]
+; VBITS_GE_256-NEXT:    mov z2.s, z2.s[7]
+; VBITS_GE_256-NEXT:    insr z0.s, s5
+; VBITS_GE_256-NEXT:    mov z5.s, z4.s[7]
+; VBITS_GE_256-NEXT:    insr z4.s, s2
+; VBITS_GE_256-NEXT:    insr z3.s, s5
+; VBITS_GE_256-NEXT:    st1w { z0.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v32f32:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.s, vl32
 ; VBITS_GE_1024-NEXT:    mov w8, #31
 ; VBITS_GE_1024-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.s, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb s0, p1, z0.s
 ; VBITS_GE_1024-NEXT:    insr z1.s, s0
 ; VBITS_GE_1024-NEXT:    st1w { z1.s }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <32 x float>, <32 x float>* %a
   %op2 = load <32 x float>, <32 x float>* %b
   %ret = shufflevector <32 x float> %op1, <32 x float> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                          i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                          i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                          i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x float> %ret, <32 x float>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v64f32(<64 x float>* %a, <64 x float>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v64f32:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #12
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #12
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #12
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #12
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #12
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #12
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #12
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #12
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #12
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #12
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #12
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #12
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #12
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #12
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #12
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v64f32:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x10, #32
+; VBITS_GE_256-NEXT:    mov x9, #40
+; VBITS_GE_256-NEXT:    mov x11, #8
+; VBITS_GE_256-NEXT:    ptrue p0.s, vl8
+; VBITS_GE_256-NEXT:    mov x12, #16
+; VBITS_GE_256-NEXT:    mov x13, #24
+; VBITS_GE_256-NEXT:    mov x8, #56
+; VBITS_GE_256-NEXT:    mov x14, #48
+; VBITS_GE_256-NEXT:    ld1w { z3.s }, p0/z, [x1, x10, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z1.s }, p0/z, [x1, x9, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z5.s }, p0/z, [x1, x11, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z4.s }, p0/z, [x1, x12, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z16.s }, p0/z, [x1, x13, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z0.s }, p0/z, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z2.s }, p0/z, [x1, x8, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z7.s }, p0/z, [x1, x14, lsl #2]
+; VBITS_GE_256-NEXT:    ld1w { z17.s }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z18.s, z3.s[7]
+; VBITS_GE_256-NEXT:    mov z6.s, z1.s[7]
+; VBITS_GE_256-NEXT:    insr z1.s, s18
+; VBITS_GE_256-NEXT:    mov z18.s, z5.s[7]
+; VBITS_GE_256-NEXT:    mov z19.s, z4.s[7]
+; VBITS_GE_256-NEXT:    insr z4.s, s18
+; VBITS_GE_256-NEXT:    mov z18.s, z16.s[7]
+; VBITS_GE_256-NEXT:    insr z3.s, s18
+; VBITS_GE_256-NEXT:    mov z18.s, z7.s[7]
+; VBITS_GE_256-NEXT:    insr z7.s, s6
+; VBITS_GE_256-NEXT:    mov z0.s, z0.s[7]
+; VBITS_GE_256-NEXT:    mov z6.s, z17.s[7]
+; VBITS_GE_256-NEXT:    insr z16.s, s19
+; VBITS_GE_256-NEXT:    insr z2.s, s18
+; VBITS_GE_256-NEXT:    insr z17.s, s0
+; VBITS_GE_256-NEXT:    insr z5.s, s6
+; VBITS_GE_256-NEXT:    st1w { z2.s }, p0, [x0, x8, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z7.s }, p0, [x0, x14, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z3.s }, p0, [x0, x10, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z16.s }, p0, [x0, x13, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z4.s }, p0, [x0, x12, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z5.s }, p0, [x0, x11, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z1.s }, p0, [x0, x9, lsl #2]
+; VBITS_GE_256-NEXT:    st1w { z17.s }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v64f32:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.s, vl64
 ; VBITS_GE_2048-NEXT:    mov w8, #63
 ; VBITS_GE_2048-NEXT:    ld1w { z0.s }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1w { z1.s }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.s, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb s0, p1, z0.s
%ret = shufflevector <64 x float> %op1, <64 x float> %op2, <64 x i32> <i32 63, i32 64, i32 65, i32 66, i32 67, i32 68, i32 69, i32 70,
                                                                          i32 111, i32 112, i32 113, i32 114, i32 115, i32 116, i32 117, i32 118,
                                                                          i32 119, i32 120, i32 121, i32 122, i32 123, i32 124, i32 125, i32 126>
   store <64 x float> %ret, <64 x float>* %a
   ret void
 }
 
 ; Don't use SVE for 128-bit vectors
 define <2 x double> @shuffle_ext_byone_v2f64(<2 x double> %op1, <2 x double> %op2) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v2f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v2f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
 ; CHECK-NEXT:    ret
   %ret = shufflevector <2 x double> %op1, <2 x double> %op2, <2 x i32> <i32 1, i32 2>
   ret <2 x double> %ret
 }
 
 define void @shuffle_ext_byone_v4f64(<4 x double>* %a, <4 x double>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v4f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q2, [x1]
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v1.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v2.16b, #8
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_v4f64:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    mov z0.d, z0.d[3]
 ; CHECK-NEXT:    insr z1.d, d0
 ; CHECK-NEXT:    st1d { z1.d }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <4 x double>, <4 x double>* %a
   %op2 = load <4 x double>, <4 x double>* %b
   %ret = shufflevector <4 x double> %op1, <4 x double> %op2, <4 x i32> <i32 3, i32 4, i32 5, i32 6>
   store <4 x double> %ret, <4 x double>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v8f64(<8 x double>* %a, <8 x double>* %b) #0 {
 ; Ensure sensible type legalisation.
-; VBITS_EQ_256-LABEL: shuffle_ext_byone_v8f64:
-; VBITS_EQ_256:       // %bb.0:
-; VBITS_EQ_256-NEXT:    mov x8, #4
-; VBITS_EQ_256-NEXT:    ptrue p0.d, vl4
-; VBITS_EQ_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ld1d { z2.d }, p0/z, [x1]
-; VBITS_EQ_256-NEXT:    mov z0.d, z0.d[3]
-; VBITS_EQ_256-NEXT:    mov z3.d, z2.d[3]
-; VBITS_EQ_256-NEXT:    insr z2.d, d0
-; VBITS_EQ_256-NEXT:    insr z1.d, d3
-; VBITS_EQ_256-NEXT:    st1d { z2.d }, p0, [x0]
-; VBITS_EQ_256-NEXT:    st1d { z1.d }, p0, [x0, x8, lsl #3]
-; VBITS_EQ_256-NEXT:    ret
+; NO_SVE-LABEL: shuffle_ext_byone_v8f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #32]
+; NO_SVE-NEXT:    ldp q4, q2, [x1]
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #8
+; NO_SVE-NEXT:    ldr q3, [x0, #48]
+; NO_SVE-NEXT:    ext v5.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v4.16b, v2.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v4.16b, #8
+; NO_SVE-NEXT:    stp q5, q0, [x0, #32]
+; NO_SVE-NEXT:    stp q1, q2, [x0]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v8f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z0.d, z0.d[3]
+; VBITS_GE_256-NEXT:    mov z3.d, z2.d[3]
+; VBITS_GE_256-NEXT:    insr z2.d, d0
+; VBITS_GE_256-NEXT:    insr z1.d, d3
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ret
 ;
 ; VBITS_GE_512-LABEL: shuffle_ext_byone_v8f64:
 ; VBITS_GE_512:       // %bb.0:
 ; VBITS_GE_512-NEXT:    ptrue p0.d, vl8
 ; VBITS_GE_512-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_512-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_512-NEXT:    mov z0.d, z0.d[7]
 ; VBITS_GE_512-NEXT:    insr z1.d, d0
 ; VBITS_GE_512-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_512-NEXT:    ret
   %op1 = load <8 x double>, <8 x double>* %a
   %op2 = load <8 x double>, <8 x double>* %b
   %ret = shufflevector <8 x double> %op1, <8 x double> %op2, <8 x i32> <i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14>
   store <8 x double> %ret, <8 x double>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v16f64(<16 x double>* %a, <16 x double>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v16f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q2, q1, [x1, #96]
+; NO_SVE-NEXT:    ldp q6, q5, [x1, #64]
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ldp q4, q3, [x1, #32]
+; NO_SVE-NEXT:    ext v2.16b, v5.16b, v2.16b, #8
+; NO_SVE-NEXT:    ldp q16, q7, [x1]
+; NO_SVE-NEXT:    ext v17.16b, v4.16b, v3.16b, #8
+; NO_SVE-NEXT:    ldr q0, [x0, #112]
+; NO_SVE-NEXT:    stp q2, q1, [x0, #96]
+; NO_SVE-NEXT:    ext v4.16b, v7.16b, v4.16b, #8
+; NO_SVE-NEXT:    ext v7.16b, v16.16b, v7.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v16.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v3.16b, v6.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v6.16b, v5.16b, #8
+; NO_SVE-NEXT:    stp q4, q17, [x0, #32]
+; NO_SVE-NEXT:    stp q0, q7, [x0]
+; NO_SVE-NEXT:    stp q1, q2, [x0, #64]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v16f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x8, #8
+; VBITS_GE_256-NEXT:    mov x9, #12
+; VBITS_GE_256-NEXT:    mov x10, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z5.d, z0.d[3]
+; VBITS_GE_256-NEXT:    insr z1.d, d5
+; VBITS_GE_256-NEXT:    mov z5.d, z3.d[3]
+; VBITS_GE_256-NEXT:    mov z2.d, z2.d[3]
+; VBITS_GE_256-NEXT:    insr z0.d, d5
+; VBITS_GE_256-NEXT:    mov z5.d, z4.d[3]
+; VBITS_GE_256-NEXT:    insr z4.d, d2
+; VBITS_GE_256-NEXT:    insr z3.d, d5
+; VBITS_GE_256-NEXT:    st1d { z0.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_1024-LABEL: shuffle_ext_byone_v16f64:
 ; VBITS_GE_1024:       // %bb.0:
 ; VBITS_GE_1024-NEXT:    ptrue p0.d, vl16
 ; VBITS_GE_1024-NEXT:    mov w8, #15
 ; VBITS_GE_1024-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_1024-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_1024-NEXT:    whilels p1.d, xzr, x8
 ; VBITS_GE_1024-NEXT:    lastb d0, p1, z0.d
 ; VBITS_GE_1024-NEXT:    insr z1.d, d0
 ; VBITS_GE_1024-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_1024-NEXT:    ret
   %op1 = load <16 x double>, <16 x double>* %a
   %op2 = load <16 x double>, <16 x double>* %b
   %ret = shufflevector <16 x double> %op1, <16 x double> %op2, <16 x i32> <i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22,
                                                                            i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30>
   store <16 x double> %ret, <16 x double>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_v32f64(<32 x double>* %a, <32 x double>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_v32f64:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q1, q0, [x1, #160]
+; NO_SVE-NEXT:    ldp q3, q2, [x1, #128]
+; NO_SVE-NEXT:    ldp q6, q7, [x1, #192]
+; NO_SVE-NEXT:    ext v19.16b, v3.16b, v2.16b, #8
+; NO_SVE-NEXT:    ldp q18, q16, [x1, #96]
+; NO_SVE-NEXT:    ext v21.16b, v6.16b, v7.16b, #8
+; NO_SVE-NEXT:    ext v6.16b, v0.16b, v6.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v1.16b, v0.16b, #8
+; NO_SVE-NEXT:    ext v1.16b, v2.16b, v1.16b, #8
+; NO_SVE-NEXT:    ldp q5, q4, [x1, #64]
+; NO_SVE-NEXT:    ext v3.16b, v16.16b, v3.16b, #8
+; NO_SVE-NEXT:    ldp q22, q20, [x1, #32]
+; NO_SVE-NEXT:    ext v17.16b, v5.16b, v4.16b, #8
+; NO_SVE-NEXT:    ldp q24, q23, [x1]
+; NO_SVE-NEXT:    ldp q25, q2, [x1, #224]
+; NO_SVE-NEXT:    stp q3, q19, [x0, #128]
+; NO_SVE-NEXT:    stp q1, q0, [x0, #160]
+; NO_SVE-NEXT:    stp q6, q21, [x0, #192]
+; NO_SVE-NEXT:    ext v6.16b, v18.16b, v16.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v4.16b, v18.16b, #8
+; NO_SVE-NEXT:    ext v4.16b, v20.16b, v5.16b, #8
+; NO_SVE-NEXT:    ldr q1, [x0, #240]
+; NO_SVE-NEXT:    ext v3.16b, v22.16b, v20.16b, #8
+; NO_SVE-NEXT:    ext v5.16b, v23.16b, v22.16b, #8
+; NO_SVE-NEXT:    stp q0, q6, [x0, #96]
+; NO_SVE-NEXT:    ext v0.16b, v24.16b, v23.16b, #8
+; NO_SVE-NEXT:    stp q4, q17, [x0, #64]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v24.16b, #8
+; NO_SVE-NEXT:    ext v2.16b, v25.16b, v2.16b, #8
+; NO_SVE-NEXT:    stp q5, q3, [x0, #32]
+; NO_SVE-NEXT:    ext v3.16b, v7.16b, v25.16b, #8
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    stp q3, q2, [x0, #224]
+; NO_SVE-NEXT:    ret
+;
+; VBITS_GE_256-LABEL: shuffle_ext_byone_v32f64:
+; VBITS_GE_256:       // %bb.0:
+; VBITS_GE_256-NEXT:    mov x10, #16
+; VBITS_GE_256-NEXT:    mov x9, #20
+; VBITS_GE_256-NEXT:    mov x11, #4
+; VBITS_GE_256-NEXT:    ptrue p0.d, vl4
+; VBITS_GE_256-NEXT:    mov x12, #8
+; VBITS_GE_256-NEXT:    mov x13, #12
+; VBITS_GE_256-NEXT:    mov x8, #28
+; VBITS_GE_256-NEXT:    mov x14, #24
+; VBITS_GE_256-NEXT:    ld1d { z3.d }, p0/z, [x1, x10, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z1.d }, p0/z, [x1, x9, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z5.d }, p0/z, [x1, x11, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z4.d }, p0/z, [x1, x12, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z16.d }, p0/z, [x1, x13, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z0.d }, p0/z, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z2.d }, p0/z, [x1, x8, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z7.d }, p0/z, [x1, x14, lsl #3]
+; VBITS_GE_256-NEXT:    ld1d { z17.d }, p0/z, [x1]
+; VBITS_GE_256-NEXT:    mov z18.d, z3.d[3]
+; VBITS_GE_256-NEXT:    mov z6.d, z1.d[3]
+; VBITS_GE_256-NEXT:    insr z1.d, d18
+; VBITS_GE_256-NEXT:    mov z18.d, z5.d[3]
+; VBITS_GE_256-NEXT:    mov z19.d, z4.d[3]
+; VBITS_GE_256-NEXT:    insr z4.d, d18
+; VBITS_GE_256-NEXT:    mov z18.d, z16.d[3]
+; VBITS_GE_256-NEXT:    insr z3.d, d18
+; VBITS_GE_256-NEXT:    mov z18.d, z7.d[3]
+; VBITS_GE_256-NEXT:    insr z7.d, d6
+; VBITS_GE_256-NEXT:    mov z0.d, z0.d[3]
+; VBITS_GE_256-NEXT:    mov z6.d, z17.d[3]
+; VBITS_GE_256-NEXT:    insr z16.d, d19
+; VBITS_GE_256-NEXT:    insr z2.d, d18
+; VBITS_GE_256-NEXT:    insr z17.d, d0
+; VBITS_GE_256-NEXT:    insr z5.d, d6
+; VBITS_GE_256-NEXT:    st1d { z2.d }, p0, [x0, x8, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z7.d }, p0, [x0, x14, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z3.d }, p0, [x0, x10, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z16.d }, p0, [x0, x13, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z4.d }, p0, [x0, x12, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z5.d }, p0, [x0, x11, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z1.d }, p0, [x0, x9, lsl #3]
+; VBITS_GE_256-NEXT:    st1d { z17.d }, p0, [x0]
+; VBITS_GE_256-NEXT:    ret
+;
 ; VBITS_GE_2048-LABEL: shuffle_ext_byone_v32f64:
 ; VBITS_GE_2048:       // %bb.0:
 ; VBITS_GE_2048-NEXT:    ptrue p0.d, vl32
 ; VBITS_GE_2048-NEXT:    mov w8, #31
 ; VBITS_GE_2048-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; VBITS_GE_2048-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; VBITS_GE_2048-NEXT:    whilels p1.d, xzr, x8
 ; VBITS_GE_2048-NEXT:    lastb d0, p1, z0.d
 ; VBITS_GE_2048-NEXT:    insr z1.d, d0
 ; VBITS_GE_2048-NEXT:    st1d { z1.d }, p0, [x0]
 ; VBITS_GE_2048-NEXT:    ret
   %op1 = load <32 x double>, <32 x double>* %a
   %op2 = load <32 x double>, <32 x double>* %b
   %ret = shufflevector <32 x double> %op1, <32 x double> %op2, <32 x i32> <i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38,
                                                                            i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46,
                                                                            i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54,
                                                                            i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62>
   store <32 x double> %ret, <32 x double>* %a
   ret void
 }
 
 define void @shuffle_ext_byone_reverse(<4 x double>* %a, <4 x double>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_byone_reverse:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldp q0, q2, [x0]
+; NO_SVE-NEXT:    ldr q1, [x1, #16]
+; NO_SVE-NEXT:    ext v1.16b, v1.16b, v0.16b, #8
+; NO_SVE-NEXT:    ext v0.16b, v0.16b, v2.16b, #8
+; NO_SVE-NEXT:    stp q1, q0, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_byone_reverse:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ptrue p0.d, vl4
 ; CHECK-NEXT:    ld1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ld1d { z1.d }, p0/z, [x1]
 ; CHECK-NEXT:    mov z1.d, z1.d[3]
 ; CHECK-NEXT:    insr z0.d, d1
 ; CHECK-NEXT:    st1d { z0.d }, p0, [x0]
 ; CHECK-NEXT:    ret
   %op1 = load <4 x double>, <4 x double>* %a
   %op2 = load <4 x double>, <4 x double>* %b
   %ret = shufflevector <4 x double> %op1, <4 x double> %op2, <4 x i32> <i32 7, i32 0, i32 1, i32 2>
   store <4 x double> %ret, <4 x double>* %a
   ret void
 }
 
 define void @shuffle_ext_invalid(<4 x double>* %a, <4 x double>* %b) #0 {
+; NO_SVE-LABEL: shuffle_ext_invalid:
+; NO_SVE:       // %bb.0:
+; NO_SVE-NEXT:    ldr q0, [x0, #16]
+; NO_SVE-NEXT:    ldr q1, [x1]
+; NO_SVE-NEXT:    stp q0, q1, [x0]
+; NO_SVE-NEXT:    ret
+;
 ; CHECK-LABEL: shuffle_ext_invalid:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    stp x29, x30, [sp, #-16]! // 16-byte Folded Spill
 ; CHECK-NEXT:    .cfi_def_cfa_offset 16
 ; CHECK-NEXT:    mov x29, sp
 ; CHECK-NEXT:    .cfi_def_cfa w29, 16
 ; CHECK-NEXT:    .cfi_offset w30, -8
 ; CHECK-NEXT:    .cfi_offset w29, -16