[CodeGen][AArch64][SVE] Use ld1r[bhsd] for vector splat from memory

[peterwaller-arm]

Summary

This avoids the use of the vector unit for copying from scalar to
vector. There is an extra ptrue instruction, but a predicate register
with the ptrue pattern populated is likely to be free in the context of
real code.

Tests were generated from a template to cover the axes mentioned at the
top of the test file.

Co-authored-by: Francesco Petrogalli <[e-mail]>

[peterwaller-arm]

• Fix nxv2f64: uimm6s{4 => 8}

[peterwaller-arm]

• Protect against values living on the stack. Intend to introduce this later.

Please note I'll be on leave for the next week, returning June 7th.

[paulwalker-arm]

I'll take a more detailed look later but structurally it looks fine to me. I've added a comment that hopefully means we don't need the frame index limitation.

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

135

Rather than this I think you just need to add entries for the LD1R instructions to functions like getLoadStoreImmIdx and getMemOpInfo.

[efriedma]

Maybe add a testcase for splat of a constant? Not sure what the code generation should look like, but it would be good to have coverage.

llvm/test/CodeGen/AArch64/sve-ld1r.ll

301

Is ldr+mov really better than add+ld1rw? (I think it's coming out this way because SelectAddrModeIndexedUImm bails instead of forcing the offset to zero.)

[peterwaller-arm]

Apologies for delayed response. Should be much faster on further updates.

• Address most review comments.
  • Fix the issue of stack objects.
  • Force offset to zero in SelectAddrModeIndexedUImm.
• Add a mir test for frame lowering, which crashes on main and passes here.
• Drop addition of azextload and use more specific patterns instead. I couldn't find a sense in which they were necessary.

[peterwaller-arm]

In D103170#2808815, @efriedma wrote:

Maybe add a testcase for splat of a constant? Not sure what the code generation should look like, but it would be good to have coverage.

@efriedma I wasn't sure why you might be after for splat-of-constant, since this is about reading from memory. Are you thinking of splat-of-constant where the constant is out of range for an immediate? Something like https://github.com/llvm/llvm-project/blob/0596f7d828436e7db85154f2815eb3ff32d505af/llvm/test/CodeGen/AArch64/sve-intrinsics-dup-x.ll ? Or something else? Thanks for your review!

[efriedma]

In D103170#2852580, @peterwaller-arm wrote:

In D103170#2808815, @efriedma wrote:

Maybe add a testcase for splat of a constant? Not sure what the code generation should look like, but it would be good to have coverage.

@efriedma I wasn't sure why you might be after for splat-of-constant, since this is about reading from memory. Are you thinking of splat-of-constant where the constant is out of range for an immediate? Something like https://github.com/llvm/llvm-project/blob/0596f7d828436e7db85154f2815eb3ff32d505af/llvm/test/CodeGen/AArch64/sve-intrinsics-dup-x.ll ? Or something else? Thanks for your review!

Yes, an splat with an out-of-range immediate. We generate a load for some floating-point constants. For example:

define <vscale x 2 x double> @z() {
  %1 = insertelement <vscale x 2 x double> undef, double 3.33, i32 0
  %2 = shufflevector <vscale x 2 x double> %1, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
  ret <vscale x 2 x double> %2
}

Currently lowers to:

adrp    x8, .LCPI0_0
ldr     d0, [x8, :lo12:.LCPI0_0]
mov     z0.d, d0
ret

[peterwaller-arm]

• Add test for splat-of-float-constant which is not representable as an immediate.
• Run update_llc_test_checks on sve-vector-splat.ll
• Address code formater complaints.

[peterwaller-arm]

Thanks again Eli. The code change is apparent in splat_nxv2f64_imm_out_of_range. I added float tests for good measure.

[efriedma]

LGTM

[peterwaller-arm]

• Rebase, update tests.

llvm/include/llvm/Target/TargetSelectionDAG.td

 def sextload  : PatFrag<(ops node:$ptr), (unindexedload node:$ptr)> {
   let IsLoad = true;
   let IsSignExtLoad = true;
 }
 def zextload  : PatFrag<(ops node:$ptr), (unindexedload node:$ptr)> {
   let IsLoad = true;
   let IsZeroExtLoad = true;
 }
+def azextload : PatFrag<(ops node:$ptr), (unindexedload node:$ptr), [{
+  auto Type = cast<LoadSDNode>(N)->getExtensionType();
+  return Type == ISD::ZEXTLOAD || Type == ISD::EXTLOAD;
+}]>;
 
 def extloadi1  : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let MemoryVT = i1;
 }
 def extloadi8  : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let MemoryVT = i8;
def zextloadi16 : PatFrag<(ops node:$ptr), (zextload node:$ptr)> {
   let IsLoad = true;
   let MemoryVT = i16;
 }
 def zextloadi32 : PatFrag<(ops node:$ptr), (zextload node:$ptr)> {
   let IsLoad = true;
   let MemoryVT = i32;
 }
 
+def azextloadi1 : PatFrag<(ops node:$ptr), (azextload node:$ptr), [{
+  return cast<LoadSDNode>(N)->getMemoryVT() == MVT::i1;
+}]>;
+def azextloadi8 : PatFrag<(ops node:$ptr), (azextload node:$ptr), [{
+  return cast<LoadSDNode>(N)->getMemoryVT() == MVT::i8;
+}]>;
+def azextloadi16 : PatFrag<(ops node:$ptr), (azextload node:$ptr), [{
+  return cast<LoadSDNode>(N)->getMemoryVT() == MVT::i16;
+}]>;
+def azextloadi32 : PatFrag<(ops node:$ptr), (azextload node:$ptr), [{
+  return cast<LoadSDNode>(N)->getMemoryVT() == MVT::i32;
+}]>;
+
 def extloadvi1  : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = i1;
 }
 def extloadvi8  : PatFrag<(ops node:$ptr), (extload node:$ptr)> {
   let IsLoad = true;
   let ScalarMemoryVT = i8;
 }

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

bool SelectAddrModeUnscaled32(SDValue N, SDValue &Base, SDValue &OffImm) {
     return SelectAddrModeUnscaled(N, 4, Base, OffImm);
   }
   bool SelectAddrModeUnscaled64(SDValue N, SDValue &Base, SDValue &OffImm) {
     return SelectAddrModeUnscaled(N, 8, Base, OffImm);
   }
   bool SelectAddrModeUnscaled128(SDValue N, SDValue &Base, SDValue &OffImm) {
     return SelectAddrModeUnscaled(N, 16, Base, OffImm);
   }
+  template <unsigned Size, unsigned Max>
+  bool SelectAddrModeIndexedUImm(SDValue N, SDValue &Base, SDValue &OffImm) {
+    bool Found = SelectAddrModeIndexed(N, Size, Base, OffImm);
+    if (Found) {
+      if (dyn_cast<FrameIndexSDNode>(Base))
+        return false;
+      if (auto CI = dyn_cast<ConstantSDNode>(OffImm)) {
+        int64_t C = CI->getSExtValue();
+        if (C <= Max)
+          return true;
+      }
+    }
+
+    return false;
+  }
 
   template<int Width>
   bool SelectAddrModeWRO(SDValue N, SDValue &Base, SDValue &Offset,
                          SDValue &SignExtend, SDValue &DoShift) {
     return SelectAddrModeWRO(N, Width / 8, Base, Offset, SignExtend, DoShift);
   }
 
   template<int Width>

llvm/lib/Target/AArch64/AArch64InstrFormats.td

 // (unsigned immediate)
 // Indexed for 8-bit registers. offset is in range [0,4095].
 def am_indexed8 : ComplexPattern<i64, 2, "SelectAddrModeIndexed8", []>;
 def am_indexed16 : ComplexPattern<i64, 2, "SelectAddrModeIndexed16", []>;
 def am_indexed32 : ComplexPattern<i64, 2, "SelectAddrModeIndexed32", []>;
 def am_indexed64 : ComplexPattern<i64, 2, "SelectAddrModeIndexed64", []>;
 def am_indexed128 : ComplexPattern<i64, 2, "SelectAddrModeIndexed128", []>;
 
+// (unsigned immediate)
+// Indexed for 8-bit registers. offset is in range [0,63].
+def am_indexed8_6b : ComplexPattern<i64, 2, "SelectAddrModeIndexedUImm<1,63>", []>;
+def am_indexed16_6b : ComplexPattern<i64, 2, "SelectAddrModeIndexedUImm<2,63>", []>;
+def am_indexed32_6b : ComplexPattern<i64, 2, "SelectAddrModeIndexedUImm<4,63>", []>;
+def am_indexed64_6b : ComplexPattern<i64, 2, "SelectAddrModeIndexedUImm<8,63>", []>;
+
 def gi_am_indexed8 :
     GIComplexOperandMatcher<s64, "selectAddrModeIndexed<8>">,
     GIComplexPatternEquiv<am_indexed8>;
 def gi_am_indexed16 :
     GIComplexOperandMatcher<s64, "selectAddrModeIndexed<16>">,
     GIComplexPatternEquiv<am_indexed16>;
 def gi_am_indexed32 :
     GIComplexOperandMatcher<s64, "selectAddrModeIndexed<32>">,

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

def : Pat<(AArch64ptest (nxv16i1 PPR:$pg), (nxv16i1 PPR:$src)),
             (PTEST_PP PPR:$pg, PPR:$src)>;
   def : Pat<(AArch64ptest (nxv8i1 PPR:$pg), (nxv8i1 PPR:$src)),
             (PTEST_PP PPR:$pg, PPR:$src)>;
   def : Pat<(AArch64ptest (nxv4i1 PPR:$pg), (nxv4i1 PPR:$src)),
             (PTEST_PP PPR:$pg, PPR:$src)>;
   def : Pat<(AArch64ptest (nxv2i1 PPR:$pg), (nxv2i1 PPR:$src)),
             (PTEST_PP PPR:$pg, PPR:$src)>;
 
+  let AddedComplexity = 1 in {
+  class LD1RPat<ValueType vt, SDPatternOperator operator,
+                Instruction load, Instruction ptrue, ValueType index_vt, ComplexPattern CP, Operand immtype> :
+        Pat<(vt (AArch64dup (index_vt (operator (CP GPR64:$base, immtype:$offset))))),
+            (load (ptrue 31), GPR64:$base, $offset)>;
+  }
+
+  // LDR1 of 8-bit data
+  def : LD1RPat<nxv16i8, azextloadi8, LD1RB_IMM,    PTRUE_B, i32, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv8i16, azextloadi8, LD1RB_H_IMM,  PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv4i32, azextloadi8, LD1RB_S_IMM,  PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv2i64, azextloadi8, LD1RB_D_IMM,  PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv8i16, sextloadi8,  LD1RSB_H_IMM, PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv4i32, sextloadi8,  LD1RSB_S_IMM, PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  def : LD1RPat<nxv2i64, sextloadi8,  LD1RSB_D_IMM, PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+
+  // LDR1 of 16-bit data
+  def : LD1RPat<nxv8i16, azextloadi16, LD1RH_IMM,    PTRUE_H, i32, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv4i32, azextloadi16, LD1RH_S_IMM,  PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv2i64, azextloadi16, LD1RH_D_IMM,  PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv4i32, sextloadi16,  LD1RSH_S_IMM, PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv2i64, sextloadi16,  LD1RSH_D_IMM, PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+
+  // LDR1 of 32-bit data
+  def : LD1RPat<nxv4i32, load,         LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
+  def : LD1RPat<nxv2i64, azextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+  def : LD1RPat<nxv2i64, sextloadi32,  LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+
+  // LDR1 of 64-bit data
+  def : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
+
+  // LD1R of FP data
+  def : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
+  def : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
+  def : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
+  def : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
+
   // LD1R of 128-bit masked data
   def : Pat<(nxv16i8 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
             (LD1RQ_B_IMM $gp, $base, (i64 0))>;
   def : Pat<(nxv8i16 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
             (LD1RQ_H_IMM $gp, $base, (i64 0))>;
   def : Pat<(nxv4i32 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
             (LD1RQ_W_IMM $gp, $base, (i64 0))>;
   def : Pat<(nxv2i64 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),

llvm/test/CodeGen/AArch64/sve-ld-post-inc.ll

; CHECK-NEXT:    ret
   %gep = getelementptr i32, i32* %a, i64 %inc
   store i32* %gep, i32** %ptr
   ret <vscale x 4 x i32> %ins
 }
 
 define <vscale x 2 x double> @test_post_ld1_dup(double* %a, double** %ptr, i64 %inc) {
 ; CHECK-LABEL: test_post_ld1_dup:
 ; CHECK:       // %bb.0:
-; CHECK-NEXT:    ldr d0, [x0]
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
 ; CHECK-NEXT:    add x8, x0, x2, lsl #3
 ; CHECK-NEXT:    str x8, [x1]
-; CHECK-NEXT:    mov z0.d, d0
 ; CHECK-NEXT:    ret
   %load = load double, double* %a
   %dup = call <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double %load)
   %gep = getelementptr double, double* %a, i64 %inc
   store double* %gep, double** %ptr
   ret <vscale x 2 x double> %dup
 }
 
 declare <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double)

llvm/test/CodeGen/AArch64/sve-ld1r.ll

+; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+;
+; Check that ldr1* instruction is generated to splat scalar during load,
+; rather than mov from scalar to vector register (which would require the vector unit).
+;
+; one-off: ld1r_stack checks that the compiler doesn't crash on a stack pointer.
+;
+; Test axes:
+;   types = [i8, i16, i32, i64, half, float, double]
+;   methods = [direct load, gep upper bound - 1, gep out of range x {neg,pos}, sext..., zext..., unpacked_floats...]
+;
+
+@g8 = external global i8
+
+; Not yet implemented: value being splatted to vector is loaded from the stack.
+define <vscale x 16 x i8> @ld1r_stack() {
+; CHECK-LABEL: ld1r_stack:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub sp, sp, #16 // =16
+; CHECK-NEXT:    .cfi_def_cfa_offset 16
+; CHECK-NEXT:    adrp x8, :got:g8
+; CHECK-NEXT:    ldr x8, [x8, :got_lo12:g8]
+; CHECK-NEXT:    ldrb w8, [x8]
+; CHECK-NEXT:    strb w8, [sp, #12]
+; CHECK-NEXT:    ldrb w8, [sp, #14]
+; CHECK-NEXT:    mov z0.b, w8
+; CHECK-NEXT:    add sp, sp, #16 // =16
+; CHECK-NEXT:    ret
+  %valp = alloca i8
+  %valp2  = load volatile i8, i8* @g8
+  store volatile i8 %valp2, i8* %valp
+  %valp3 = getelementptr i8, i8* %valp, i32 2
+  %val = load i8, i8* %valp3
+  %1 = insertelement <vscale x 16 x i8> undef, i8 %val, i32 0
+  %2 = shufflevector <vscale x 16 x i8> %1, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+  ret <vscale x 16 x i8> %2
+}
+
+define <vscale x 16 x i8> @ld1rb(i8* %valp) {
+; CHECK-LABEL: ld1rb:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ins = insertelement <vscale x 16 x i8> undef, i8 %val, i32 0
+  %shf = shufflevector <vscale x 16 x i8> %ins, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+  ret <vscale x 16 x i8> %shf
+}
+
+define <vscale x 16 x i8> @ld1rb_gep(i8* %valp) {
+; CHECK-LABEL: ld1rb_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0, #63]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i8, i8* %valp, i32 63
+  %val = load i8, i8* %valp2
+  %ins = insertelement <vscale x 16 x i8> undef, i8 %val, i32 0
+  %shf = shufflevector <vscale x 16 x i8> %ins, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+  ret <vscale x 16 x i8> %shf
+}
+
+define <vscale x 16 x i8> @ld1rb_gep_out_of_range_up(i8* %valp) {
+; CHECK-LABEL: ld1rb_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrb w8, [x0, #64]
+; CHECK-NEXT:    mov z0.b, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i8, i8* %valp, i32 64
+  %val = load i8, i8* %valp2
+  %ins = insertelement <vscale x 16 x i8> undef, i8 %val, i32 0
+  %shf = shufflevector <vscale x 16 x i8> %ins, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+  ret <vscale x 16 x i8> %shf
+}
+
+define <vscale x 16 x i8> @ld1rb_gep_out_of_range_down(i8* %valp) {
+; CHECK-LABEL: ld1rb_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldurb w8, [x0, #-1]
+; CHECK-NEXT:    mov z0.b, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i8, i8* %valp, i32 -1
+  %val = load i8, i8* %valp2
+  %ins = insertelement <vscale x 16 x i8> undef, i8 %val, i32 0
+  %shf = shufflevector <vscale x 16 x i8> %ins, <vscale x 16 x i8> undef, <vscale x 16 x i32> zeroinitializer
+  ret <vscale x 16 x i8> %shf
+}
+
+define <vscale x 8 x i16> @ld1rb_i8_i16_zext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i16_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rb { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = zext i8 %val to i16
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %ext, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 8 x i16> @ld1rb_i8_i16_sext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i16_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rsb { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = sext i8 %val to i16
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %ext, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 4 x i32> @ld1rb_i8_i32_zext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i32_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rb { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = zext i8 %val to i32
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %ext, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 4 x i32> @ld1rb_i8_i32_sext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i32_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rsb { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = sext i8 %val to i32
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %ext, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 2 x i64> @ld1rb_i8_i64_zext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rb { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = zext i8 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rb_i8_i64_sext(i8* %valp) {
+; CHECK-LABEL: ld1rb_i8_i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rsb { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i8, i8* %valp
+  %ext = sext i8 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 8 x i16> @ld1rh(i16* %valp) {
+; CHECK-LABEL: ld1rh:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i16, i16* %valp
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %val, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 8 x i16> @ld1rh_gep(i16* %valp) {
+; CHECK-LABEL: ld1rh_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i16, i16* %valp, i32 63
+  %val = load i16, i16* %valp2
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %val, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 8 x i16> @ld1rh_gep_out_of_range_up(i16* %valp) {
+; CHECK-LABEL: ld1rh_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrh w8, [x0, #128]
+; CHECK-NEXT:    mov z0.h, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i16, i16* %valp, i32 64
+  %val = load i16, i16* %valp2
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %val, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 8 x i16> @ld1rh_gep_out_of_range_down(i16* %valp) {
+; CHECK-LABEL: ld1rh_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldurh w8, [x0, #-2]
+; CHECK-NEXT:    mov z0.h, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i16, i16* %valp, i32 -1
+  %val = load i16, i16* %valp2
+  %ins = insertelement <vscale x 8 x i16> undef, i16 %val, i32 0
+  %shf = shufflevector <vscale x 8 x i16> %ins, <vscale x 8 x i16> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x i16> %shf
+}
+
+define <vscale x 4 x i32> @ld1rh_i16_i32_zext(i16* %valp) {
+; CHECK-LABEL: ld1rh_i16_i32_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i16, i16* %valp
+  %ext = zext i16 %val to i32
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %ext, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 4 x i32> @ld1rh_i16_i32_sext(i16* %valp) {
+; CHECK-LABEL: ld1rh_i16_i32_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rsh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i16, i16* %valp
+  %ext = sext i16 %val to i32
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %ext, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 2 x i64> @ld1rh_i16_i64_zext(i16* %valp) {
+; CHECK-LABEL: ld1rh_i16_i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i16, i16* %valp
+  %ext = zext i16 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rh_i16_i64_sext(i16* %valp) {
+; CHECK-LABEL: ld1rh_i16_i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rsh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i16, i16* %valp
+  %ext = sext i16 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 4 x i32> @ld1rw(i32* %valp) {
+; CHECK-LABEL: ld1rw:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i32, i32* %valp
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %val, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 4 x i32> @ld1rw_gep(i32* %valp) {
+; CHECK-LABEL: ld1rw_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i32, i32* %valp, i32 63
+  %val = load i32, i32* %valp2
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %val, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 4 x i32> @ld1rw_gep_out_of_range_up(i32* %valp) {
+; CHECK-LABEL: ld1rw_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr w8, [x0, #256]
+; CHECK-NEXT:    mov z0.s, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i32, i32* %valp, i32 64
+  %val = load i32, i32* %valp2
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %val, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 4 x i32> @ld1rw_gep_out_of_range_down(i32* %valp) {
+; CHECK-LABEL: ld1rw_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur w8, [x0, #-4]
+; CHECK-NEXT:    mov z0.s, w8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i32, i32* %valp, i32 -1
+  %val = load i32, i32* %valp2
+  %ins = insertelement <vscale x 4 x i32> undef, i32 %val, i32 0
+  %shf = shufflevector <vscale x 4 x i32> %ins, <vscale x 4 x i32> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x i32> %shf
+}
+
+define <vscale x 2 x i64> @ld1rw_i32_i64_zext(i32* %valp) {
+; CHECK-LABEL: ld1rw_i32_i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i32, i32* %valp
+  %ext = zext i32 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rw_i32_i64_sext(i32* %valp) {
+; CHECK-LABEL: ld1rw_i32_i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rsw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i32, i32* %valp
+  %ext = sext i32 %val to i64
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %ext, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rd(i64* %valp) {
+; CHECK-LABEL: ld1rd:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load i64, i64* %valp
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %val, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rd_gep(i64* %valp) {
+; CHECK-LABEL: ld1rd_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i64, i64* %valp, i32 63
+  %val = load i64, i64* %valp2
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %val, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rd_gep_out_of_range_up(i64* %valp) {
+; CHECK-LABEL: ld1rd_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr x8, [x0, #512]
+; CHECK-NEXT:    mov z0.d, x8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i64, i64* %valp, i32 64
+  %val = load i64, i64* %valp2
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %val, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 2 x i64> @ld1rd_gep_out_of_range_down(i64* %valp) {
+; CHECK-LABEL: ld1rd_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur x8, [x0, #-8]
+; CHECK-NEXT:    mov z0.d, x8
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr i64, i64* %valp, i32 -1
+  %val = load i64, i64* %valp2
+  %ins = insertelement <vscale x 2 x i64> undef, i64 %val, i32 0
+  %shf = shufflevector <vscale x 2 x i64> %ins, <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x i64> %shf
+}
+
+define <vscale x 8 x half> @ld1rh_half(half* %valp) {
+; CHECK-LABEL: ld1rh_half:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load half, half* %valp
+  %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %shf
+}
+
+define <vscale x 8 x half> @ld1rh_half_gep(half* %valp) {
+; CHECK-LABEL: ld1rh_half_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 63
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %shf
+}
+
+define <vscale x 8 x half> @ld1rh_half_gep_out_of_range_up(half* %valp) {
+; CHECK-LABEL: ld1rh_half_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0, #128]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 64
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %shf
+}
+
+define <vscale x 8 x half> @ld1rh_half_gep_out_of_range_down(half* %valp) {
+; CHECK-LABEL: ld1rh_half_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 -1
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 8 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 8 x half> %ins, <vscale x 8 x half> undef, <vscale x 8 x i32> zeroinitializer
+  ret <vscale x 8 x half> %shf
+}
+
+define <vscale x 4 x half> @ld1rh_half_unpacked4(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked4:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load half, half* %valp
+  %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x half> %shf
+}
+
+define <vscale x 4 x half> @ld1rh_half_unpacked4_gep(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked4_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 63
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x half> %shf
+}
+
+define <vscale x 4 x half> @ld1rh_half_unpacked4_gep_out_of_range_up(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked4_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0, #128]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 64
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x half> %shf
+}
+
+define <vscale x 4 x half> @ld1rh_half_unpacked4_gep_out_of_range_down(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked4_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 -1
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 4 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 4 x half> %ins, <vscale x 4 x half> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x half> %shf
+}
+
+define <vscale x 2 x half> @ld1rh_half_unpacked2(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked2:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load half, half* %valp
+  %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x half> %shf
+}
+
+define <vscale x 2 x half> @ld1rh_half_unpacked2_gep(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked2_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 63
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x half> %shf
+}
+
+define <vscale x 2 x half> @ld1rh_half_unpacked2_gep_out_of_range_up(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked2_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h0, [x0, #128]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 64
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x half> %shf
+}
+
+define <vscale x 2 x half> @ld1rh_half_unpacked2_gep_out_of_range_down(half* %valp) {
+; CHECK-LABEL: ld1rh_half_unpacked2_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur h0, [x0, #-2]
+; CHECK-NEXT:    mov z0.h, h0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr half, half* %valp, i32 -1
+  %val = load half, half* %valp2
+  %ins = insertelement <vscale x 2 x half> undef, half %val, i32 0
+  %shf = shufflevector <vscale x 2 x half> %ins, <vscale x 2 x half> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x half> %shf
+}
+
+define <vscale x 4 x float> @ld1rw_float(float* %valp) {
+; CHECK-LABEL: ld1rw_float:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load float, float* %valp
+  %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x float> %shf
+}
+
+define <vscale x 4 x float> @ld1rw_float_gep(float* %valp) {
+; CHECK-LABEL: ld1rw_float_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 63
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x float> %shf
+}
+
+define <vscale x 4 x float> @ld1rw_float_gep_out_of_range_up(float* %valp) {
+; CHECK-LABEL: ld1rw_float_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s0, [x0, #256]
+; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 64
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x float> %shf
+}
+
+define <vscale x 4 x float> @ld1rw_float_gep_out_of_range_down(float* %valp) {
+; CHECK-LABEL: ld1rw_float_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-4]
+; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 -1
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 4 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 4 x float> %ins, <vscale x 4 x float> undef, <vscale x 4 x i32> zeroinitializer
+  ret <vscale x 4 x float> %shf
+}
+
+define <vscale x 2 x float> @ld1rw_float_unpacked2(float* %valp) {
+; CHECK-LABEL: ld1rw_float_unpacked2:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load float, float* %valp
+  %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x float> %shf
+}
+
+define <vscale x 2 x float> @ld1rw_float_unpacked2_gep(float* %valp) {
+; CHECK-LABEL: ld1rw_float_unpacked2_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0, #252]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 63
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x float> %shf
+}
+
+define <vscale x 2 x float> @ld1rw_float_unpacked2_gep_out_of_range_up(float* %valp) {
+; CHECK-LABEL: ld1rw_float_unpacked2_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s0, [x0, #256]
+; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 64
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x float> %shf
+}
+
+define <vscale x 2 x float> @ld1rw_float_unpacked2_gep_out_of_range_down(float* %valp) {
+; CHECK-LABEL: ld1rw_float_unpacked2_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur s0, [x0, #-4]
+; CHECK-NEXT:    mov z0.s, s0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr float, float* %valp, i32 -1
+  %val = load float, float* %valp2
+  %ins = insertelement <vscale x 2 x float> undef, float %val, i32 0
+  %shf = shufflevector <vscale x 2 x float> %ins, <vscale x 2 x float> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x float> %shf
+}
+
+define <vscale x 2 x double> @ld1rd_double(double* %valp) {
+; CHECK-LABEL: ld1rd_double:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+  %val = load double, double* %valp
+  %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
+  %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x double> %shf
+}
+
+define <vscale x 2 x double> @ld1rd_double_gep(double* %valp) {
+; CHECK-LABEL: ld1rd_double_gep:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr double, double* %valp, i32 63
+  %val = load double, double* %valp2
+  %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
+  %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x double> %shf
+}
+
+define <vscale x 2 x double> @ld1rd_double_gep_out_of_range_up(double* %valp) {
+; CHECK-LABEL: ld1rd_double_gep_out_of_range_up:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d0, [x0, #512]
+; CHECK-NEXT:    mov z0.d, d0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr double, double* %valp, i32 64
+  %val = load double, double* %valp2
+  %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
+  %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x double> %shf
+}
+
+define <vscale x 2 x double> @ld1rd_double_gep_out_of_range_down(double* %valp) {
+; CHECK-LABEL: ld1rd_double_gep_out_of_range_down:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldur d0, [x0, #-8]
+; CHECK-NEXT:    mov z0.d, d0
+; CHECK-NEXT:    ret
+  %valp2 = getelementptr double, double* %valp, i32 -1
+  %val = load double, double* %valp2
+  %ins = insertelement <vscale x 2 x double> undef, double %val, i32 0
+  %shf = shufflevector <vscale x 2 x double> %ins, <vscale x 2 x double> undef, <vscale x 2 x i32> zeroinitializer
+  ret <vscale x 2 x double> %shf
+}