[CodeGen] Disable FP LD1RX instructions generation for Neoverse-V1

[igor.kirillov]

Summary

These instructions show worse performance on Neoverse-V1 compared
to pair of LDR(LDP)/MOV instructions.
This patch adds no-sve-fp-ld1r sub-target feature, which is enabled
only on Neoverse-V1.

Fixes https://github.com/llvm/llvm-project/issues/64498

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64.td

151

This doesn't accurately represent the "feature" you want to model.

Firstly the effect is specific to SVE rather than wanting to avoid all uses of ld1r (plus below I reason that we also might want to keep the integer variants as is).

Secondly the issue is that on Neoverse V1 there are fewer LS pipelines for SVE than NEON. This is not normally a problem because the SVE registers are twice the size of NEON and so the overall bandwidth is greater. However, when loading 128-bit or smaller datatypes the bandwidth switches in favour of NEON with its extra LS pipe. (Noting that on V1 the latency of LD1R is the same are LDR+DUP)

The choice is yours but as a minimum, and based on agreement regarding the integer variants, I'd be happy with "sve-avoid-fp-ld1r" but if there's a nice way to sum up the second point above then that'll be perfect.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2358–2366

I believe this should be restricted to only the floating point patterns. The issue relates to instruction bandwidth and the GPR variants of DUP on V1 go down a single pipe and thus are likely worse than the LS bandwidth issue you're trying to prevent.

[igor.kirillov]

• Rename new feature avoid-ldr1 to no-sve-fp-ld1r
• Now change is applied only to FP LD1R instructions
• Some other renaming inside code

[igor.kirillov]

llvm/lib/Target/AArch64/AArch64.td

151

I've decided to use no- prefix rather than avoid as it seems to be the way it is done.

[igor.kirillov]

Update CHECK lines

[paulwalker-arm]

llvm/lib/Target/AArch64/AArch64.td

152

Should this be HasNoSVEFPLD1R? Otherwise UseSVEFPLD1R = !Subtarget->hasSVEFPLD1R code looks weird.

[igor.kirillov]

Renaming

[paulwalker-arm]

A couple of late requests but otherwise looks good.

llvm/lib/Target/AArch64/AArch64.td

152

The LDP is not relevant here. It was just a quirk of the code where you observed the issue.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

2358–2366

Sorry I missed this before but this'll override the existing value for Predicates and so you'll need to add HasSVEorSME to be correct.

[igor.kirillov]

• Add RUN line to sve-ld1r.ll test with -mcpu=neoverse-v1
• Update feature description
• Add missing Predicate

[igor.kirillov]

Remove RUN with -mcpu=neoverse-v1 due to too many changes

[igor.kirillov]

Added a test for neoverse-v1

[dmgreen]

This sounds like an interesting one.. We have certainly seen cases before where instructions are worth splitting out into multiple parts, but it often helps in one case and hurts in others. It seems like the idea here is that the throughput of sve loads is limited to 2, but with scalar fp loads that can go up to 3? So in load-throughput limited situations the expanded nodes win out (especially if they can use ldp). Wouldn't the opposite be true too though? If it was vector-instruction limited or frontend limited then multiple instructions will be worse? You could imagine it being done in the load/store optimizer if it could detect cases where it could use ldp.

If this is better (and I imagine it might be in many situations), then it can equally apply to integer too. It would just need to be changed to an fp load, to make sure it didn't pay the cost of crossing between register banks.

llvm/test/CodeGen/AArch64/sve-ld1r.ll

1254–1255

These with multiple extra instructions look quite a bit worse. It might not apply for predicated instructions with zeros.

[paulwalker-arm]

I agree and this is very much a last minute fix for a new regression observed within LLVM 17. My feeling is that for these sorts of issues we should be using the scheduling models to tell us about bandwidth limitations.

llvm/lib/Target/AArch64/AArch64.td

: SubtargetFeature<"use-experimental-zeroing-pseudos",
                        "UseExperimentalZeroingPseudos", "true",
                        "Hint to the compiler that the MOVPRFX instruction is "
                        "merged with destructive operations",
                        []>;
 
 def FeatureUseScalarIncVL : SubtargetFeature<"use-scalar-inc-vl",
   "UseScalarIncVL", "true", "Prefer inc/dec over add+cnt">;
 
+def FeatureNoSVEFPLD1R : SubtargetFeature<"no-sve-fp-ld1r",
+  "NoSVEFPLD1R", "true", "Avoid using LD1RX instructions for FP">;
+
 def FeatureSVE2 : SubtargetFeature<"sve2", "HasSVE2", "true",
   "Enable Scalable Vector Extension 2 (SVE2) instructions (FEAT_SVE2)",
   [FeatureSVE, FeatureUseScalarIncVL]>;
 
 def FeatureSVE2AES : SubtargetFeature<"sve2-aes", "HasSVE2AES", "true",
   "Enable AES SVE2 instructions (FEAT_SVE_AES, FEAT_SVE_PMULL128)",
   [FeatureSVE2, FeatureAES]>;
 

 
 def TuneNeoverseV1 : SubtargetFeature<"neoversev1", "ARMProcFamily", "NeoverseV1",
                                       "Neoverse V1 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureFuseAdrpAdd,
                                       FeatureLSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
-                                      FeaturePredictableSelectIsExpensive]>;
+                                      FeaturePredictableSelectIsExpensive,
+                                      FeatureNoSVEFPLD1R]>;
 
 def TuneNeoverseV2 : SubtargetFeature<"neoversev2", "ARMProcFamily", "NeoverseV2",
                                       "Neoverse V2 ARM processors", [
                                       FeatureFuseAES,
                                       FeatureLSLFast,
                                       FeaturePostRAScheduler,
                                       FeatureEnableSelectOptimize,
                                       FeaturePredictableSelectIsExpensive]>;

llvm/lib/Target/AArch64/AArch64InstrInfo.td

def UseAlternateSExtLoadCVTF32
     : Predicate<"Subtarget->useAlternateSExtLoadCVTF32Pattern()">;
 
 def UseNegativeImmediates
     : Predicate<"false">, AssemblerPredicate<(all_of (not FeatureNoNegativeImmediates)),
                                              "NegativeImmediates">;
 
 def UseScalarIncVL : Predicate<"Subtarget->useScalarIncVL()">;
 
+def UseSVEFPLD1R : Predicate<"!Subtarget->noSVEFPLD1R()">;
+
 def IsNeonAvailable : Predicate<"Subtarget->isNeonAvailable()">;
 
 def AArch64LocalRecover : SDNode<"ISD::LOCAL_RECOVER",
                                   SDTypeProfile<1, 1, [SDTCisSameAs<0, 1>,
                                                        SDTCisInt<1>]>>;
 
 
 //===----------------------------------------------------------------------===//

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

let Predicates = [HasSVEorSME] in {
   // LDR1 of 32-bit data
   defm : LD1RPat<nxv4i32, load,        LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
   defm : LD1RPat<nxv2i64, zextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
   defm : LD1RPat<nxv2i64, sextloadi32, LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
 
   // LDR1 of 64-bit data
   defm : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
 
+  let Predicates = [HasSVEorSME, UseSVEFPLD1R] in {
     // LD1R of FP data
     defm : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
     defm : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
     defm : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
     defm : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
     defm : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
     defm : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
+  }
 
 // LD1R of 128-bit masked data
   multiclass ld1rq_pat<ValueType vt1, SDPatternOperator op, Instruction load_instr, ComplexPattern AddrCP>{
     def : Pat<(vt1 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
           (!cast<Instruction>(load_instr # _IMM) $gp, $base, (i64 0))>;
     let AddedComplexity = 2 in {
       def : Pat<(vt1 (op PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
             (!cast<Instruction>(load_instr # _IMM) $gp, $base, simm4s16:$imm)>;

llvm/lib/Target/AArch64/SVEInstrFormats.td

def : InstAlias<"mov $Zd, $Pg/m, $Vn",
                   (!cast<Instruction>(NAME # _H) ZPR16:$Zd, PPR3bAny:$Pg, FPR16:$Vn), 1>;
   def : InstAlias<"mov $Zd, $Pg/m, $Vn",
                   (!cast<Instruction>(NAME # _S) ZPR32:$Zd, PPR3bAny:$Pg, FPR32:$Vn), 1>;
   def : InstAlias<"mov $Zd, $Pg/m, $Vn",
                   (!cast<Instruction>(NAME # _D) ZPR64:$Zd, PPR3bAny:$Pg, FPR64:$Vn), 1>;
 
   def : Pat<(nxv8f16 (op nxv8i1:$pg, f16:$splat, nxv8f16:$passthru)),
             (!cast<Instruction>(NAME # _H) $passthru, $pg, $splat)>;
+  def : Pat<(nxv4f16 (op nxv4i1:$pg, f16:$splat, nxv4f16:$passthru)),
+            (!cast<Instruction>(NAME # _H) $passthru, $pg, $splat)>;
+  def : Pat<(nxv2f16 (op nxv2i1:$pg, f16:$splat, nxv2f16:$passthru)),
+            (!cast<Instruction>(NAME # _H) $passthru, $pg, $splat)>;
   def : Pat<(nxv2f32 (op nxv2i1:$pg, f32:$splat, nxv2f32:$passthru)),
             (!cast<Instruction>(NAME # _S) $passthru, $pg, $splat)>;
   def : Pat<(nxv4f32 (op nxv4i1:$pg, f32:$splat, nxv4f32:$passthru)),
             (!cast<Instruction>(NAME # _S) $passthru, $pg, $splat)>;
   def : Pat<(nxv2f64 (op nxv2i1:$pg, f64:$splat, nxv2f64:$passthru)),
             (!cast<Instruction>(NAME # _D) $passthru, $pg, $splat)>;
 
   def : Pat<(nxv8bf16 (op nxv8i1:$pg, bf16:$splat, nxv8bf16:$passthru)),

llvm/test/CodeGen/AArch64/sve-ld1r.ll

 ; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
-; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s --check-prefixes=CHECK,CHECK-LD1R
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve,+no-sve-fp-ld1r < %s | FileCheck %s --check-prefixes=CHECK,CHECK-NO-LD1R
 ;
 ; Check that ldr1* instruction is generated to splat scalar during load,
 ; rather than mov from scalar to vector register (which would require the vector unit).
 ;
 ; one-off: ld1r_stack checks that ldr1b works with stack objects.
 ;
 ; Test axes:
 ;   types = [i8, i16, i32, i64, half, float, double]
; CHECK-NEXT:    ret
   %valp2 = getelementptr i64, ptr %valp, i32 -1
   %val = load i64, ptr %valp2
   %ins = insertelement <vscale x 2 x i64> undef, i64 %val, i32 0
   %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x i64> %shf
 }
 
 define <vscale x 8 x half> @ld1rh_half(ptr %valp) {
-; CHECK-LABEL: ld1rh_half:
+; CHECK-LD1R-LABEL: ld1rh_half:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.h
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
+  %val = load half, ptr %valp
+  %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %shf
+}
+
+define <vscale x 8 x half> @ld1rh_half_neoverse(ptr %valp) #1 {
+; CHECK-LABEL: ld1rh_half_neoverse:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ldr h0, [x0]
+; CHECK-NEXT:    mov z0.h, h0
 ; CHECK-NEXT:    ret
   %val = load half, ptr %valp
   %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x half> %shf
 }
 
 define <vscale x 8 x half> @ld1rh_half_gep(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.h
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #126]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 63
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x half> %shf
 }
 
 define <vscale x 8 x half> @ld1rh_half_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #128
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #128
+; CHECK-LD1R-NEXT:    ptrue p0.h
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #128]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 64
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x half> %shf
 }
 
 define <vscale x 8 x half> @ld1rh_half_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #2
-; CHECK-NEXT:    ptrue p0.h
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #2
+; CHECK-LD1R-NEXT:    ptrue p0.h
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 -1
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
   ret <vscale x 8 x half> %shf
 }
 
 define <vscale x 4 x half> @ld1rh_half_unpacked4(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked4:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked4:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked4:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %val = load half, ptr %valp
   %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x half> %shf
 }
 
 define <vscale x 4 x half> @ld1rh_half_unpacked4_gep(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked4_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0, #126]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked4_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x0, #126]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked4_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #126]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 63
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x half> %shf
 }
 
 define <vscale x 4 x half> @ld1rh_half_unpacked4_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked4_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #128
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked4_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #128
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked4_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #128]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 64
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x half> %shf
 }
 
 define <vscale x 4 x half> @ld1rh_half_unpacked4_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked4_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #2
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked4_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #2
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked4_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 -1
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x half> %shf
 }
 
 define <vscale x 2 x half> @ld1rh_half_unpacked2(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked2:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked2:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked2:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %val = load half, ptr %valp
   %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x half> %shf
 }
 
 define <vscale x 2 x half> @ld1rh_half_unpacked2_gep(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked2_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0, #126]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked2_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rh { z0.d }, p0/z, [x0, #126]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked2_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #126]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 63
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x half> %shf
 }
 
 define <vscale x 2 x half> @ld1rh_half_unpacked2_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked2_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #128
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked2_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #128
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rh { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked2_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0, #128]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 64
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x half> %shf
 }
 
 define <vscale x 2 x half> @ld1rh_half_unpacked2_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rh_half_unpacked2_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #2
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rh_half_unpacked2_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #2
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rh { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rh_half_unpacked2_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, h0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr half, ptr %valp, i32 -1
   %val = load half, ptr %valp2
   %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
   %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x half> %shf
 }
 
 define <vscale x 4 x float> @ld1rw_float(ptr %valp) {
-; CHECK-LABEL: ld1rw_float:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %val = load float, ptr %valp
   %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x float> %shf
 }
 
 define <vscale x 4 x float> @ld1rw_float_gep(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0, #252]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 63
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x float> %shf
 }
 
 define <vscale x 4 x float> @ld1rw_float_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #256
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #256
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0, #256]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 64
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x float> %shf
 }
 
 define <vscale x 4 x float> @ld1rw_float_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #4
-; CHECK-NEXT:    ptrue p0.s
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #4
+; CHECK-LD1R-NEXT:    ptrue p0.s
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur s0, [x0, #-4]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 -1
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
   ret <vscale x 4 x float> %shf
 }
 
 define <vscale x 2 x float> @ld1rw_float_unpacked2(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_unpacked2:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_unpacked2:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_unpacked2:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %val = load float, ptr %valp
   %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x float> %shf
 }
 
 define <vscale x 2 x float> @ld1rw_float_unpacked2_gep(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_unpacked2_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0, #252]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_unpacked2_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rw { z0.d }, p0/z, [x0, #252]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_unpacked2_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0, #252]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 63
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x float> %shf
 }
 
 define <vscale x 2 x float> @ld1rw_float_unpacked2_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_unpacked2_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #256
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_unpacked2_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #256
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rw { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_unpacked2_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0, #256]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 64
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x float> %shf
 }
 
 define <vscale x 2 x float> @ld1rw_float_unpacked2_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rw_float_unpacked2_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #4
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rw_float_unpacked2_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #4
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rw { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rw_float_unpacked2_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur s0, [x0, #-4]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, s0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr float, ptr %valp, i32 -1
   %val = load float, ptr %valp2
   %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
   %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x float> %shf
 }
 
 define <vscale x 2 x double> @ld1rd_double(ptr %valp) {
-; CHECK-LABEL: ld1rd_double:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rd_double:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rd_double:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr d0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, d0
+; CHECK-NO-LD1R-NEXT:    ret
   %val = load double, ptr %valp
   %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
   %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x double> %shf
 }
 
 define <vscale x 2 x double> @ld1rd_double_gep(ptr %valp) {
-; CHECK-LABEL: ld1rd_double_gep:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rd_double_gep:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rd_double_gep:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr d0, [x0, #504]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, d0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr double, ptr %valp, i32 63
   %val = load double, ptr %valp2
   %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
   %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x double> %shf
 }
 
 define <vscale x 2 x double> @ld1rd_double_gep_out_of_range_up(ptr %valp) {
-; CHECK-LABEL: ld1rd_double_gep_out_of_range_up:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    add x8, x0, #512
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rd_double_gep_out_of_range_up:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    add x8, x0, #512
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rd_double_gep_out_of_range_up:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr d0, [x0, #512]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, d0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr double, ptr %valp, i32 64
   %val = load double, ptr %valp2
   %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
   %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x double> %shf
 }
 
 define <vscale x 2 x double> @ld1rd_double_gep_out_of_range_down(ptr %valp) {
-; CHECK-LABEL: ld1rd_double_gep_out_of_range_down:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    sub x8, x0, #8
-; CHECK-NEXT:    ptrue p0.d
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: ld1rd_double_gep_out_of_range_down:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    sub x8, x0, #8
+; CHECK-LD1R-NEXT:    ptrue p0.d
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: ld1rd_double_gep_out_of_range_down:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldur d0, [x0, #-8]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, d0
+; CHECK-NO-LD1R-NEXT:    ret
   %valp2 = getelementptr double, ptr %valp, i32 -1
   %val = load double, ptr %valp2
   %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
   %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
   ret <vscale x 2 x double> %shf
 }
 
 define <vscale x 2 x double> @dupq_ld1rqd_f64(ptr %a) {

 ; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ret
     %ld = load i32, ptr %addr
     %ext = zext i32 %ld to i64
     %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
     ret <vscale x 2 x i64> %res
 }
 define <vscale x 8 x half> @dup_ld1rh_half_passthruundef_nxv8f16(<vscale x 8 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rh_half_passthruundef_nxv8f16:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rh_half_passthruundef_nxv8f16:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rh_half_passthruundef_nxv8f16:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, p0/m, h0
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load half, ptr %addr
     %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x i1> %pg, half %ld)
     ret <vscale x 8 x half> %res
 }
 define <vscale x 4 x float> @dup_ld1rs_float_passthruundef_nxv4f32(<vscale x 4 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rs_float_passthruundef_nxv4f32:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rs_float_passthruundef_nxv4f32:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rs_float_passthruundef_nxv4f32:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, p0/m, s0
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load float, ptr %addr
     %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x i1> %pg, float %ld)
     ret <vscale x 4 x float> %res
 }
 define <vscale x 2 x double> @dup_ld1rd_double_passthruundef_nxv2f64(<vscale x 2 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rd_double_passthruundef_nxv2f64:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rd_double_passthruundef_nxv2f64:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rd_double_passthruundef_nxv2f64:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr d0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, p0/m, d0
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load double, ptr %addr
     %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x i1> %pg, double %ld)
     ret <vscale x 2 x double> %res
 }
 define <vscale x 4 x half> @dup_ld1rh_half_passthruundef_nxv4f16(<vscale x 4 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rh_half_passthruundef_nxv4f16:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rh_half_passthruundef_nxv4f16:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rh_half_passthruundef_nxv4f16:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h0, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, p0/m, h0
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load half, ptr %addr
     %res = call <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x i1> %pg, half %ld)
     ret <vscale x 4 x half> %res
 }
 define <vscale x 16 x i8> @dup_ld1rb_i8_passthruzero_nxv16i8(<vscale x 16 x i1> %pg, ptr %addr) {
 ; CHECK-LABEL: dup_ld1rb_i8_passthruzero_nxv16i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]

 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    ret
     %ld = load i64, ptr %addr
     %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> zeroinitializer, <vscale x 2 x i1> %pg, i64 %ld)
     ret <vscale x 2 x i64> %res
 }
 define <vscale x 8 x half> @dup_ld1rh_half_passthruzero_nxv8f16(<vscale x 8 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv8f16:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv8f16:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv8f16:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load half, ptr %addr
     %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> zeroinitializer, <vscale x 8 x i1> %pg, half %ld)
     ret <vscale x 8 x half> %res
 }
 define <vscale x 4 x float> @dup_ld1rs_float_passthruzero_nxv4f32(<vscale x 4 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rs_float_passthruzero_nxv4f32:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rs_float_passthruzero_nxv4f32:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rs_float_passthruzero_nxv4f32:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load float, ptr %addr
     %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> zeroinitializer, <vscale x 4 x i1> %pg, float %ld)
     ret <vscale x 4 x float> %res
 }
 define <vscale x 2 x double> @dup_ld1rd_double_passthruzero_nxv2f64(<vscale x 2 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rd_double_passthruzero_nxv2f64:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rd_double_passthruzero_nxv2f64:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rd_double_passthruzero_nxv2f64:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr d1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.d, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.d, p0/m, d1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load double, ptr %addr
     %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> zeroinitializer, <vscale x 2 x i1> %pg, double %ld)
     ret <vscale x 2 x double> %res
 }
 define <vscale x 4 x half> @dup_ld1rh_half_passthruzero_nxv4f16(<vscale x 4 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv4f16:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv4f16:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv4f16:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load half, ptr %addr
     %res = call <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half> zeroinitializer, <vscale x 4 x i1> %pg, half %ld)
     ret <vscale x 4 x half> %res
 }
 define <vscale x 2 x half> @dup_ld1rh_half_passthruzero_nxv2f16(<vscale x 2 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv2f16:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv2f16:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rh_half_passthruzero_nxv2f16:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr h1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.h, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load half, ptr %addr
     %res = call <vscale x 2 x half> @llvm.aarch64.sve.dup.nxv2f16(<vscale x 2 x half> zeroinitializer, <vscale x 2 x i1> %pg, half %ld)
     ret <vscale x 2 x half> %res
 }
 define <vscale x 2 x float> @dup_ld1rs_float_passthruzero_nxv2f32(<vscale x 2 x i1> %pg, ptr %addr) {
-; CHECK-LABEL: dup_ld1rs_float_passthruzero_nxv2f32:
-; CHECK:       // %bb.0:
-; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
-; CHECK-NEXT:    ret
+; CHECK-LD1R-LABEL: dup_ld1rs_float_passthruzero_nxv2f32:
+; CHECK-LD1R:       // %bb.0:
+; CHECK-LD1R-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-LD1R-NEXT:    ret
+;
+; CHECK-NO-LD1R-LABEL: dup_ld1rs_float_passthruzero_nxv2f32:
+; CHECK-NO-LD1R:       // %bb.0:
+; CHECK-NO-LD1R-NEXT:    ldr s1, [x0]
+; CHECK-NO-LD1R-NEXT:    mov z0.s, #0 // =0x0
+; CHECK-NO-LD1R-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NO-LD1R-NEXT:    ret
     %ld = load float, ptr %addr
     %res = call <vscale x 2 x float> @llvm.aarch64.sve.dup.nxv2f32(<vscale x 2 x float> zeroinitializer, <vscale x 2 x i1> %pg, float %ld)
     ret <vscale x 2 x float> %res
 }
 define <vscale x 16 x i8> @negtest_dup_ld1rb_i8_passthru_nxv16i8(<vscale x 16 x i8> %pt, <vscale x 16 x i1> %pg, ptr %addr) {
 ; CHECK-LABEL: negtest_dup_ld1rb_i8_passthru_nxv16i8:
 ; CHECK:       // %bb.0:
 ; CHECK-NEXT:    ldrb w8, [x0]

 declare <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float>, <vscale x 4 x i1>, float)
 declare <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double>, <vscale x 2 x i1>, double)
 declare <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half>, <vscale x 4 x i1>, half)
 declare <vscale x 2 x half> @llvm.aarch64.sve.dup.nxv2f16(<vscale x 2 x half>, <vscale x 2 x i1>, half)
 declare <vscale x 2 x float> @llvm.aarch64.sve.dup.nxv2f32(<vscale x 2 x float>, <vscale x 2 x i1>, float)
 
 
 attributes #0 = { "target-features"="+sve,+bf16" }
+attributes #1 = { "target-cpu"="neoverse-v1" }