[llvm][SVE] Addressing mode for FF/NF loads.

[fpetrogalli]

Summary

This patch adds addressing mode computation for the following SVE
instructions:

• ldff1{s}<T1> { <Zt>.<T2> }, <Pg>/Z, [<Xn|SP>{, <Xm>{, lsl #imm}}]
• ldnf1{s}<T1> { <Zt>.<T2> }, <Pg>/Z, [<Xn|SP>{, #<imm>, mul vl}]

[andwar]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4504

Is this change needed here?

4507

Why do you have to update this method? I've tried to find the relation with other changes in this file and I failed :)

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

1550

[Nit] - unrelated

llvm/test/CodeGen/AArch64/sve-intrinsics-loads-ff.ll

380

FIXME

[fpetrogalli]

Remove unused parameter from test case, and remove empty line.

[fpetrogalli]

Thank you for the review @andwar !

Francesco

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4504

I wouldn't say needed, but:

1. as far as I know, all types handled by this component in LLVM are legal: which meas that it is very unlikely that we will ever enter the brnach code following. If we ever do enter this branch, it means that something elase is broken before in the pipeline. Addressing mode decomposition doesn't seem to me the right place to catch it.

if (NumElts != 2 && NumElts != 4 && NumElts != 8 && NumElts != 16)
  return EVT();

2. What this method does is "give me a scalable vector of N packed lanes of integers". Whether the "N" comes of a vector of i1 lanes, it doesn't matter. Hence, I think it is better to use ElementCount in input, not a vector EVT of i1lanes. So, I think we can also remove the first if condition.

All in all, I think it makes sense to make the changes you see in this method.

4507

The method SelectAddrModeIndexedSVE operates on the "memory VT" of the underlying SDNode. One it determines it, it uses it to computed the decomposition of the ADD into a Base and an integer MUL VL Offset.

I had to modify this method because it needs to learn how to extract the MemVT from the SDNodes that represent the LDNF/LDFF. I then rearranged the code for the PRF intrinsic because (to me) it made mode sense to handle all cases under the same switch.

It would be nice if we could use the same method to retrieve the memory VT for all the SVE LD intrinsic, but for the custom ISD nodes there is not such thing as getMemoryVT(), like there is for the MemSDNode class.

We could argue to use MemIntrinsicSDNode (which have the getMemoryVT method), but at this point in the lowering sequence the AArch64ISD nodes have replaced the input Intrinsic::aarch64_sve_ld... nodes, so I cannot cast the nodes into instances of MemIntrinsicSDNode.

[andwar]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4504

Thank you for the explanation, it makes sense!

• Still, these changes are unrelated to what this patch is for. I think that this sort of refactoring should happen in a separate patch. If you want to keep it here, could you please list the NFC changes in the commit message?
• I think that with these changes this method can be replaced with getSVEContainerType.

4507

OK, now I understand, thanks! I think that it would be very helpful to explain this in a comment somewhere within the method. In particular, why does this method differentiate between custom ISD nodes and intrinsics? Otherwise the code is a bit counterintuitive.

It's unfortunate that such a small method requires nested switch statements.

4533

This llvm_unreachable is a bit confusing here (relates to the nested switch). Wouldn't it make more sense to have a default case within the nested switch block?

llvm/test/CodeGen/AArch64/sve-intrinsics-loads-nf.ll

18

CHECK-NEXT?

[fpetrogalli]

Thank you for your review @andwar.

I have restored the original code. I will address merging getPackedVectorTypeFromPredicateType into getSVEContainerType in a separate patch, marked as NFC.

Francesco

[fpetrogalli]

I have added CHECK-NEXT where needed.

Francesco

[fpetrogalli]

Cosmetic changes to the formatting of the codegen pattern defs. NFC

[andwar]

Cheers for working on this, LGTM!

[andwar]

LGTM

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

static EVT getPackedVectorTypeFromPredicateType(LLVMContext &Ctx, EVT PredVT) {
 
   if (NumElts != 2 && NumElts != 4 && NumElts != 8 && NumElts != 16)
     return EVT();
 
   EVT ScalarVT = EVT::getIntegerVT(Ctx, AArch64::SVEBitsPerBlock / NumElts);
   EVT MemVT = EVT::getVectorVT(Ctx, ScalarVT, NumElts, /*IsScalable=*/true);
   return MemVT;
 }
 
 /// Return the EVT of the data associated to a memory operation in \p
 /// Root. If such EVT cannot be retrived, it returns an invalid EVT.
 static EVT getMemVTFromNode(LLVMContext &Ctx, SDNode *Root) {
   if (isa<MemSDNode>(Root))
     return cast<MemSDNode>(Root)->getMemoryVT();
 
   const unsigned Opcode = Root->getOpcode();
+  // For custom ISD nodes, we have to look at them individually to extract the
+  // type of the data moved to/from memory.
+  switch (Opcode) {
+  case AArch64ISD::LDNF1:
+  case AArch64ISD::LDNF1S:
+    return cast<VTSDNode>(Root->getOperand(3))->getVT();
+  default:
+    break;
+  }
+
   if (Opcode != ISD::INTRINSIC_VOID)
     return EVT();
 
   const unsigned IntNo =
       cast<ConstantSDNode>(Root->getOperand(1))->getZExtValue();
   if (IntNo != Intrinsic::aarch64_sve_prf)
     return EVT();
 
   // We are using an SVE prefetch intrinsic. Type must be inferred
   // from the width of the predicate.
   return getPackedVectorTypeFromPredicateType(
       Ctx, Root->getOperand(2)->getValueType(0));
 }
 
 /// SelectAddrModeIndexedSVE - Attempt selection of the addressing mode:
 /// Base + OffImm * sizeof(MemVT) for Min >= OffImm <= Max
 /// where Root is the memory access using N for its address.
 template <int64_t Min, int64_t Max>
 bool AArch64DAGToDAGISel::SelectAddrModeIndexedSVE(SDNode *Root, SDValue N,
                                                    SDValue &Base,

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

multiclass sve_prefetch<SDPatternOperator prefetch, ValueType PredTy, Instruction RegImmInst, Instruction RegRegInst, int scale, ComplexPattern AddrCP> {
   }
 
   defm Pat_Load_P16 : unpred_load_predicate<nxv16i1, LDR_PXI>;
   defm Pat_Load_P8  : unpred_load_predicate<nxv8i1, LDR_PXI>;
   defm Pat_Load_P4  : unpred_load_predicate<nxv4i1, LDR_PXI>;
   defm Pat_Load_P2  : unpred_load_predicate<nxv2i1, LDR_PXI>;
 
   multiclass ldnf1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT> {
+    // scalar + immediate (mul vl)
+    let AddedComplexity = 1 in {
+      def : Pat<(Ty (Load  (PredTy PPR:$gp), (am_sve_indexed_s4 GPR64sp:$base, simm4s1:$offset), MemVT)),
+                (I PPR:$gp, GPR64sp:$base, simm4s1:$offset)>;
+    }
+
     // base
     def : Pat<(Ty (Load  (PredTy PPR:$gp), GPR64:$base, MemVT)),
               (I PPR:$gp, GPR64sp:$base, (i64 0))>;
   }
 
   // 2-element contiguous non-faulting loads
   defm : ldnf1<LDNF1B_D_IMM,  nxv2i64, AArch64ldnf1,  nxv2i1, nxv2i8>;
   defm : ldnf1<LDNF1SB_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i8>;
   defm : ldnf1<LDNF1H_D_IMM,  nxv2i64, AArch64ldnf1,  nxv2i1, nxv2i16>;
   defm : ldnf1<LDNF1SH_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i16>;
   defm : ldnf1<LDNF1W_D_IMM,  nxv2i64, AArch64ldnf1,  nxv2i1, nxv2i32>;
   defm : ldnf1<LDNF1SW_D_IMM, nxv2i64, AArch64ldnf1s, nxv2i1, nxv2i32>;
multiclass sve_prefetch<SDPatternOperator prefetch, ValueType PredTy, Instruction RegImmInst, Instruction RegRegInst, int scale, ComplexPattern AddrCP> {
   defm : ldnf1<LDNF1B_H_IMM,  nxv8i16, AArch64ldnf1,  nxv8i1, nxv8i8>;
   defm : ldnf1<LDNF1SB_H_IMM, nxv8i16, AArch64ldnf1s, nxv8i1, nxv8i8>;
   defm : ldnf1<LDNF1H_IMM,    nxv8i16, AArch64ldnf1,  nxv8i1, nxv8i16>;
   defm : ldnf1<LDNF1H_IMM,    nxv8f16, AArch64ldnf1,  nxv8i1, nxv8f16>;
 
   // 16-element contiguous non-faulting loads
   defm : ldnf1<LDNF1B_IMM,    nxv16i8, AArch64ldnf1, nxv16i1, nxv16i8>;
 
-  multiclass ldff1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT> {
-    // Add more complex addressing modes here as required.
+  multiclass ldff1<Instruction I, ValueType Ty, SDPatternOperator Load, ValueType PredTy, ValueType MemVT, ComplexPattern AddrCP> {
+    // reg + reg
+    let AddedComplexity = 1 in {
+      def : Pat<(Ty (Load  (PredTy PPR:$gp), (AddrCP GPR64:$base, GPR64:$offset), MemVT)),
+                (I PPR:$gp, GPR64sp:$base, GPR64:$offset)>;
+    }
+
     // Base
     def : Pat<(Ty (Load  (PredTy PPR:$gp), GPR64:$base, MemVT)),
               (I PPR:$gp, GPR64sp:$base, XZR)>;
   }
 
   // 2-element contiguous first faulting loads
-  defm : ldff1<LDFF1B_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i8>;
-  defm : ldff1<LDFF1SB_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i8>;
-  defm : ldff1<LDFF1H_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i16>;
-  defm : ldff1<LDFF1SH_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i16>;
-  defm : ldff1<LDFF1W_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i32>;
-  defm : ldff1<LDFF1SW_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i32>;
-  defm : ldff1<LDFF1D,    nxv2i64, AArch64ldff1,  nxv2i1, nxv2i64>;
-  defm : ldff1<LDFF1W_D,  nxv2f32, AArch64ldff1,  nxv2i1, nxv2f32>;
-  defm : ldff1<LDFF1D,    nxv2f64, AArch64ldff1,  nxv2i1, nxv2f64>;
+  defm : ldff1<LDFF1B_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1SB_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1H_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
+  defm : ldff1<LDFF1SH_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i16, am_sve_regreg_lsl1>;
+  defm : ldff1<LDFF1W_D,  nxv2i64, AArch64ldff1,  nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
+  defm : ldff1<LDFF1SW_D, nxv2i64, AArch64ldff1s, nxv2i1, nxv2i32, am_sve_regreg_lsl2>;
+  defm : ldff1<LDFF1D,    nxv2i64, AArch64ldff1,  nxv2i1, nxv2i64, am_sve_regreg_lsl3>;
+  defm : ldff1<LDFF1W_D,  nxv2f32, AArch64ldff1,  nxv2i1, nxv2f32, am_sve_regreg_lsl2>;
+  defm : ldff1<LDFF1D,    nxv2f64, AArch64ldff1,  nxv2i1, nxv2f64, am_sve_regreg_lsl3>;
 
   // 4-element contiguous first faulting loads
-  defm : ldff1<LDFF1B_S,  nxv4i32, AArch64ldff1,  nxv4i1, nxv4i8>;
-  defm : ldff1<LDFF1SB_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i8>;
-  defm : ldff1<LDFF1H_S,  nxv4i32, AArch64ldff1,  nxv4i1, nxv4i16>;
-  defm : ldff1<LDFF1SH_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i16>;
-  defm : ldff1<LDFF1W,    nxv4i32, AArch64ldff1,  nxv4i1, nxv4i32>;
-  defm : ldff1<LDFF1W,    nxv4f32, AArch64ldff1,  nxv4i1, nxv4f32>;
+  defm : ldff1<LDFF1B_S,  nxv4i32, AArch64ldff1,  nxv4i1, nxv4i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1SB_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1H_S,  nxv4i32, AArch64ldff1,  nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
+  defm : ldff1<LDFF1SH_S, nxv4i32, AArch64ldff1s, nxv4i1, nxv4i16, am_sve_regreg_lsl1>;
+  defm : ldff1<LDFF1W,    nxv4i32, AArch64ldff1,  nxv4i1, nxv4i32, am_sve_regreg_lsl2>;
+  defm : ldff1<LDFF1W,    nxv4f32, AArch64ldff1,  nxv4i1, nxv4f32, am_sve_regreg_lsl2>;
 
   // 8-element contiguous first faulting loads
-  defm : ldff1<LDFF1B_H,  nxv8i16, AArch64ldff1,  nxv8i1, nxv8i8>;
-  defm : ldff1<LDFF1SB_H, nxv8i16, AArch64ldff1s, nxv8i1, nxv8i8>;
-  defm : ldff1<LDFF1H,    nxv8i16, AArch64ldff1,  nxv8i1, nxv8i16>;
-  defm : ldff1<LDFF1H,    nxv8f16, AArch64ldff1,  nxv8i1, nxv8f16>;
+  defm : ldff1<LDFF1B_H,  nxv8i16, AArch64ldff1,  nxv8i1, nxv8i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1SB_H, nxv8i16, AArch64ldff1s, nxv8i1, nxv8i8,  am_sve_regreg_lsl0>;
+  defm : ldff1<LDFF1H,    nxv8i16, AArch64ldff1,  nxv8i1, nxv8i16, am_sve_regreg_lsl1>;
+  defm : ldff1<LDFF1H,    nxv8f16, AArch64ldff1,  nxv8i1, nxv8f16, am_sve_regreg_lsl1>;
 
   // 16-element contiguous first faulting loads
-  defm : ldff1<LDFF1B, nxv16i8, AArch64ldff1, nxv16i1, nxv16i8>;
+  defm : ldff1<LDFF1B, nxv16i8, AArch64ldff1, nxv16i1, nxv16i8, am_sve_regreg_lsl0>;
 }
 
 let Predicates = [HasSVE2] in {
   // SVE2 integer multiply-add (indexed)
   defm MLA_ZZZI : sve2_int_mla_by_indexed_elem<0b01, 0b0, "mla", int_aarch64_sve_mla_lane>;
   defm MLS_ZZZI : sve2_int_mla_by_indexed_elem<0b01, 0b1, "mls", int_aarch64_sve_mls_lane>;
 
   // SVE2 saturating multiply-add high (indexed)

llvm/test/CodeGen/AArch64/sve-intrinsics-loads-ff.ll

 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
 
 ;
 ; LDFF1B
 ;
 
 define <vscale x 16 x i8> @ldff1b(<vscale x 16 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1b:
 ; CHECK: ldff1b { z0.b }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldff1.nxv16i8(<vscale x 16 x i1> %pg, i8* %a)
   ret <vscale x 16 x i8> %load
 }
 
+define <vscale x 16 x i8> @ldff1b_reg(<vscale x 16 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1b_reg:
+; CHECK: ldff1b { z0.b }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldff1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base)
+  ret <vscale x 16 x i8> %load
+}
+
 define <vscale x 8 x i16> @ldff1b_h(<vscale x 8 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1b_h:
 ; CHECK: ldff1b { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldff1.nxv8i8(<vscale x 8 x i1> %pg, i8* %a)
   %res = zext <vscale x 8 x i8> %load to <vscale x 8 x i16>
   ret <vscale x 8 x i16> %res
 }
 
+define <vscale x 8 x i16> @ldff1b_h_reg(<vscale x 8 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1b_h_reg:
+; CHECK: ldff1b { z0.h }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldff1.nxv8i8(<vscale x 8 x i1> %pg, i8* %base)
+  %res = zext <vscale x 8 x i8> %load to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %res
+}
+
 define <vscale x 4 x i32> @ldff1b_s(<vscale x 4 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1b_s:
 ; CHECK: ldff1b { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldff1.nxv4i8(<vscale x 4 x i1> %pg, i8* %a)
   %res = zext <vscale x 4 x i8> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldff1b_s_reg(<vscale x 4 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1b_s_reg:
+; CHECK: ldff1b { z0.s }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldff1.nxv4i8(<vscale x 4 x i1> %pg, i8* %base)
+  %res = zext <vscale x 4 x i8> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 2 x i64> @ldff1b_d(<vscale x 2 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1b_d:
 ; CHECK: ldff1b { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldff1.nxv2i8(<vscale x 2 x i1> %pg, i8* %a)
   %res = zext <vscale x 2 x i8> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1b_d_reg(<vscale x 2 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1b_d_reg:
+; CHECK: ldff1b { z0.d }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldff1.nxv2i8(<vscale x 2 x i1> %pg, i8* %base)
+  %res = zext <vscale x 2 x i8> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 ;
 ; LDFF1SB
 ;
 
 define <vscale x 8 x i16> @ldff1sb_h(<vscale x 8 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1sb_h:
 ; CHECK: ldff1sb { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldff1.nxv8i8(<vscale x 8 x i1> %pg, i8* %a)
   %res = sext <vscale x 8 x i8> %load to <vscale x 8 x i16>
   ret <vscale x 8 x i16> %res
 }
 
+define <vscale x 8 x i16> @ldff1sb_h_reg(<vscale x 8 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sb_h_reg:
+; CHECK: ldff1sb { z0.h }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldff1.nxv8i8(<vscale x 8 x i1> %pg, i8* %base)
+  %res = sext <vscale x 8 x i8> %load to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %res
+}
+
 define <vscale x 4 x i32> @ldff1sb_s(<vscale x 4 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1sb_s:
 ; CHECK: ldff1sb { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldff1.nxv4i8(<vscale x 4 x i1> %pg, i8* %a)
   %res = sext <vscale x 4 x i8> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldff1sb_s_reg(<vscale x 4 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sb_s_reg:
+; CHECK: ldff1sb { z0.s }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldff1.nxv4i8(<vscale x 4 x i1> %pg, i8* %base)
+  %res = sext <vscale x 4 x i8> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 2 x i64> @ldff1sb_d(<vscale x 2 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldff1sb_d:
 ; CHECK: ldff1sb { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldff1.nxv2i8(<vscale x 2 x i1> %pg, i8* %a)
   %res = sext <vscale x 2 x i8> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1sb_d_reg(<vscale x 2 x i1> %pg, i8* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sb_d_reg:
+; CHECK: ldff1sb { z0.d }, p0/z, [x0, x1]
+; CHECK-NEXT: ret
+  %base = getelementptr i8, i8* %a, i64 %offset
+  %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldff1.nxv2i8(<vscale x 2 x i1> %pg, i8* %base)
+  %res = sext <vscale x 2 x i8> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 ;
 ; LDFF1H
 ;
 
 define <vscale x 8 x i16> @ldff1h(<vscale x 8 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldff1h:
 ; CHECK: ldff1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i16> @llvm.aarch64.sve.ldff1.nxv8i16(<vscale x 8 x i1> %pg, i16* %a)
   ret <vscale x 8 x i16> %load
 }
 
+define <vscale x 8 x i16> @ldff1h_reg(<vscale x 8 x i1> %pg, i16* %a, i64 %offset) {
+; CHECK-LABEL: ldff1h_reg:
+; CHECK: ldff1h { z0.h }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr i16, i16* %a, i64 %offset
+  %load = call <vscale x 8 x i16> @llvm.aarch64.sve.ldff1.nxv8i16(<vscale x 8 x i1> %pg, i16* %base)
+  ret <vscale x 8 x i16> %load
+}
+
 define <vscale x 4 x i32> @ldff1h_s(<vscale x 4 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldff1h_s:
 ; CHECK: ldff1h { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldff1.nxv4i16(<vscale x 4 x i1> %pg, i16* %a)
   %res = zext <vscale x 4 x i16> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldff1h_s_reg(<vscale x 4 x i1> %pg, i16* %a, i64 %offset) {
+; CHECK-LABEL: ldff1h_s_reg:
+; CHECK: ldff1h { z0.s }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr i16, i16* %a, i64 %offset
+  %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldff1.nxv4i16(<vscale x 4 x i1> %pg, i16* %base)
+  %res = zext <vscale x 4 x i16> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 2 x i64> @ldff1h_d(<vscale x 2 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldff1h_d:
 ; CHECK: ldff1h { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldff1.nxv2i16(<vscale x 2 x i1> %pg, i16* %a)
   %res = zext <vscale x 2 x i16> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1h_d_reg(<vscale x 2 x i1> %pg, i16* %a, i64 %offset) {
+; CHECK-LABEL: ldff1h_d_reg:
+; CHECK: ldff1h { z0.d }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr i16, i16* %a, i64 %offset
+  %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldff1.nxv2i16(<vscale x 2 x i1> %pg, i16* %base)
+  %res = zext <vscale x 2 x i16> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 8 x half> @ldff1h_f16(<vscale x 8 x i1> %pg, half* %a) {
 ; CHECK-LABEL: ldff1h_f16:
 ; CHECK: ldff1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x half> @llvm.aarch64.sve.ldff1.nxv8f16(<vscale x 8 x i1> %pg, half* %a)
   ret <vscale x 8 x half> %load
 }
 
+define <vscale x 8 x half> @ldff1h_f16_reg(<vscale x 8 x i1> %pg, half* %a, i64 %offset) {
+; CHECK-LABEL: ldff1h_f16_reg:
+; CHECK: ldff1h { z0.h }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr half, half* %a, i64 %offset
+  %load = call <vscale x 8 x half> @llvm.aarch64.sve.ldff1.nxv8f16(<vscale x 8 x i1> %pg, half* %base)
+  ret <vscale x 8 x half> %load
+}
+
 ;
 ; LDFF1SH
 ;
 
 define <vscale x 4 x i32> @ldff1sh_s(<vscale x 4 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldff1sh_s:
 ; CHECK: ldff1sh { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldff1.nxv4i16(<vscale x 4 x i1> %pg, i16* %a)
   %res = sext <vscale x 4 x i16> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldff1sh_s_reg(<vscale x 4 x i1> %pg, i16* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sh_s_reg:
+; CHECK: ldff1sh { z0.s }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr i16, i16* %a, i64 %offset
+  %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldff1.nxv4i16(<vscale x 4 x i1> %pg, i16* %base)
+  %res = sext <vscale x 4 x i16> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 2 x i64> @ldff1sh_d(<vscale x 2 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldff1sh_d:
 ; CHECK: ldff1sh { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldff1.nxv2i16(<vscale x 2 x i1> %pg, i16* %a)
   %res = sext <vscale x 2 x i16> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1sh_d_reg(<vscale x 2 x i1> %pg, i16* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sh_d_reg:
+; CHECK: ldff1sh { z0.d }, p0/z, [x0, x1, lsl #1]
+; CHECK-NEXT: ret
+  %base = getelementptr i16, i16* %a, i64 %offset
+  %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldff1.nxv2i16(<vscale x 2 x i1> %pg, i16* %base)
+  %res = sext <vscale x 2 x i16> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 ;
 ; LDFF1W
 ;
 
 define <vscale x 4 x i32> @ldff1w(<vscale x 4 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldff1w:
 ; CHECK: ldff1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i32> @llvm.aarch64.sve.ldff1.nxv4i32(<vscale x 4 x i1> %pg, i32* %a)
   ret <vscale x 4 x i32> %load
 }
 
+define <vscale x 4 x i32> @ldff1w_reg(<vscale x 4 x i1> %pg, i32* %a, i64 %offset) {
+; CHECK-LABEL: ldff1w_reg:
+; CHECK: ldff1w { z0.s }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base = getelementptr i32, i32* %a, i64 %offset
+  %load = call <vscale x 4 x i32> @llvm.aarch64.sve.ldff1.nxv4i32(<vscale x 4 x i1> %pg, i32* %base)
+  ret <vscale x 4 x i32> %load
+}
+
 define <vscale x 2 x i64> @ldff1w_d(<vscale x 2 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldff1w_d:
 ; CHECK: ldff1w { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldff1.nxv2i32(<vscale x 2 x i1> %pg, i32* %a)
   %res = zext <vscale x 2 x i32> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1w_d_reg(<vscale x 2 x i1> %pg, i32* %a, i64 %offset) {
+; CHECK-LABEL: ldff1w_d_reg:
+; CHECK: ldff1w { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base = getelementptr i32, i32* %a, i64 %offset
+  %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldff1.nxv2i32(<vscale x 2 x i1> %pg, i32* %base)
+  %res = zext <vscale x 2 x i32> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 4 x float> @ldff1w_f32(<vscale x 4 x i1> %pg, float* %a) {
 ; CHECK-LABEL: ldff1w_f32:
 ; CHECK: ldff1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x float> @llvm.aarch64.sve.ldff1.nxv4f32(<vscale x 4 x i1> %pg, float* %a)
   ret <vscale x 4 x float> %load
 }
 
+define <vscale x 4 x float> @ldff1w_f32_reg(<vscale x 4 x i1> %pg, float* %a, i64 %offset) {
+; CHECK-LABEL: ldff1w_f32_reg:
+; CHECK: ldff1w { z0.s }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base = getelementptr float, float* %a, i64 %offset
+  %load = call <vscale x 4 x float> @llvm.aarch64.sve.ldff1.nxv4f32(<vscale x 4 x i1> %pg, float* %base)
+  ret <vscale x 4 x float> %load
+}
+
 define <vscale x 2 x float> @ldff1w_2f32(<vscale x 2 x i1> %pg, float* %a) {
 ; CHECK-LABEL: ldff1w_2f32:
 ; CHECK: ldff1w { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x float> @llvm.aarch64.sve.ldff1.nxv2f32(<vscale x 2 x i1> %pg, float* %a)
   ret <vscale x 2 x float> %load
 }
 
+define <vscale x 2 x float> @ldff1w_2f32_reg(<vscale x 2 x i1> %pg, float* %a, i64 %offset) {
+; CHECK-LABEL: ldff1w_2f32_reg:
+; CHECK: ldff1w { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base = getelementptr float, float* %a, i64 %offset
+  %load = call <vscale x 2 x float> @llvm.aarch64.sve.ldff1.nxv2f32(<vscale x 2 x i1> %pg, float* %base)
+  ret <vscale x 2 x float> %load
+}
+
 ;
 ; LDFF1SW
 ;
 
 define <vscale x 2 x i64> @ldff1sw_d(<vscale x 2 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldff1sw_d:
 ; CHECK: ldff1sw { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldff1.nxv2i32(<vscale x 2 x i1> %pg, i32* %a)
   %res = sext <vscale x 2 x i32> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldff1sw_d_reg(<vscale x 2 x i1> %pg, i32* %a, i64 %offset) {
+; CHECK-LABEL: ldff1sw_d_reg:
+; CHECK: ldff1sw { z0.d }, p0/z, [x0, x1, lsl #2]
+; CHECK-NEXT: ret
+  %base = getelementptr i32, i32* %a, i64 %offset
+  %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldff1.nxv2i32(<vscale x 2 x i1> %pg, i32* %base)
+  %res = sext <vscale x 2 x i32> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 ;
 ; LDFF1D
 ;
 
 define <vscale x 2 x i64> @ldff1d(<vscale x 2 x i1> %pg, i64* %a) {
 ; CHECK-LABEL: ldff1d:
 ; CHECK: ldff1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i64> @llvm.aarch64.sve.ldff1.nxv2i64(<vscale x 2 x i1> %pg, i64* %a)
   ret <vscale x 2 x i64> %load
 }
 
+define <vscale x 2 x i64> @ldff1d_reg(<vscale x 2 x i1> %pg, i64* %a, i64 %offset) {
+; CHECK-LABEL: ldff1d_reg:
+; CHECK: ldff1d { z0.d }, p0/z, [x0, x1, lsl #3]
+; CHECK-NEXT: ret
+  %base = getelementptr i64, i64* %a, i64 %offset
+  %load = call <vscale x 2 x i64> @llvm.aarch64.sve.ldff1.nxv2i64(<vscale x 2 x i1> %pg, i64* %base)
+  ret <vscale x 2 x i64> %load
+}
+
 
 define <vscale x 2 x double> @ldff1d_f64(<vscale x 2 x i1> %pg, double* %a) {
 ; CHECK-LABEL: ldff1d_f64:
 ; CHECK: ldff1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x double> @llvm.aarch64.sve.ldff1.nxv2f64(<vscale x 2 x i1> %pg, double* %a)
   ret <vscale x 2 x double> %load
 }
 
+define <vscale x 2 x double> @ldff1d_f64_reg(<vscale x 2 x i1> %pg, double* %a, i64 %offset) {
+; CHECK-LABEL: ldff1d_f64_reg:
+; CHECK: ldff1d { z0.d }, p0/z, [x0, x1, lsl #3]
+; CHECK-NEXT: ret
+  %base = getelementptr double, double* %a, i64 %offset
+  %load = call <vscale x 2 x double> @llvm.aarch64.sve.ldff1.nxv2f64(<vscale x 2 x i1> %pg, double* %base)
+  ret <vscale x 2 x double> %load
+}
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.ldff1.nxv16i8(<vscale x 16 x i1>, i8*)
 
 declare <vscale x 8 x i8> @llvm.aarch64.sve.ldff1.nxv8i8(<vscale x 8 x i1>, i8*)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.ldff1.nxv8i16(<vscale x 8 x i1>, i16*)
 declare <vscale x 8 x half> @llvm.aarch64.sve.ldff1.nxv8f16(<vscale x 8 x i1>, half*)
 
 declare <vscale x 4 x i8> @llvm.aarch64.sve.ldff1.nxv4i8(<vscale x 4 x i1>, i8*)
 declare <vscale x 4 x i16> @llvm.aarch64.sve.ldff1.nxv4i16(<vscale x 4 x i1>, i16*)

llvm/test/CodeGen/AArch64/sve-intrinsics-loads-nf.ll

 ; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
 
+; Range testing for the immediate in the reg+imm(mulvl) addressing
+; mode is done only for one instruction. The rest of the instrucions
+; test only one immediate value in bound.
+
 define <vscale x 16 x i8> @ldnf1b(<vscale x 16 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1b:
 ; CHECK: ldnf1b { z0.b }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %a)
   ret <vscale x 16 x i8> %load
 }
 
+define <vscale x 16 x i8> @ldnf1b_out_of_lower_bound(<vscale x 16 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_out_of_lower_bound:
+; CHECK:       rdvl    x[[OFFSET:[0-9]+]], #-9
+; CHECK-NEXT:  add     x[[BASE:[0-9]+]], x0, x[[OFFSET]]
+; CHECK-NEXT:  ldnf1b { z0.b }, p0/z, [x[[BASE]]]
+; CHECK-NEXT:  ret
+  %base_scalable = bitcast i8* %a to <vscale x 16 x i8>*
+  %base = getelementptr <vscale x 16 x i8>, <vscale x 16 x i8>* %base_scalable, i64 -9
+  %base_scalar = bitcast <vscale x 16 x i8>* %base to i8*
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base_scalar)
+  ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 16 x i8> @ldnf1b_lower_bound(<vscale x 16 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_lower_bound:
+; CHECK: ldnf1b { z0.b }, p0/z, [x0, #-8, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 16 x i8>*
+  %base = getelementptr <vscale x 16 x i8>, <vscale x 16 x i8>* %base_scalable, i64 -8
+  %base_scalar = bitcast <vscale x 16 x i8>* %base to i8*
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base_scalar)
+  ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 16 x i8> @ldnf1b_inbound(<vscale x 16 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_inbound:
+; CHECK: ldnf1b { z0.b }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 16 x i8>*
+  %base = getelementptr <vscale x 16 x i8>, <vscale x 16 x i8>* %base_scalable, i64 1
+  %base_scalar = bitcast <vscale x 16 x i8>* %base to i8*
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base_scalar)
+  ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 16 x i8> @ldnf1b_upper_bound(<vscale x 16 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_upper_bound:
+; CHECK: ldnf1b { z0.b }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 16 x i8>*
+  %base = getelementptr <vscale x 16 x i8>, <vscale x 16 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 16 x i8>* %base to i8*
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base_scalar)
+  ret <vscale x 16 x i8> %load
+}
+
+define <vscale x 16 x i8> @ldnf1b_out_of_upper_bound(<vscale x 16 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_out_of_upper_bound:
+; CHECK:       rdvl    x[[OFFSET:[0-9]+]], #8
+; CHECK-NEXT:  add     x[[BASE:[0-9]+]], x0, x[[OFFSET]]
+; CHECK-NEXT:  ldnf1b { z0.b }, p0/z, [x[[BASE]]]
+; CHECK-NEXT:  ret
+  %base_scalable = bitcast i8* %a to <vscale x 16 x i8>*
+  %base = getelementptr <vscale x 16 x i8>, <vscale x 16 x i8>* %base_scalable, i64 8
+  %base_scalar = bitcast <vscale x 16 x i8>* %base to i8*
+  %load = call <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1> %pg, i8* %base_scalar)
+  ret <vscale x 16 x i8> %load
+}
+
 define <vscale x 8 x i16> @ldnf1b_h(<vscale x 8 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1b_h:
 ; CHECK: ldnf1b { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldnf1.nxv8i8(<vscale x 8 x i1> %pg, i8* %a)
   %res = zext <vscale x 8 x i8> %load to <vscale x 8 x i16>
   ret <vscale x 8 x i16> %res
 }
 
+define <vscale x 8 x i16> @ldnf1b_h_inbound(<vscale x 8 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_h_inbound:
+; CHECK: ldnf1b { z0.h }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 8 x i8>*
+  %base = getelementptr <vscale x 8 x i8>, <vscale x 8 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 8 x i8>* %base to i8*
+  %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldnf1.nxv8i8(<vscale x 8 x i1> %pg, i8* %base_scalar)
+  %res = zext <vscale x 8 x i8> %load to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %res
+}
+
 define <vscale x 8 x i16> @ldnf1sb_h(<vscale x 8 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1sb_h:
 ; CHECK: ldnf1sb { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldnf1.nxv8i8(<vscale x 8 x i1> %pg, i8* %a)
   %res = sext <vscale x 8 x i8> %load to <vscale x 8 x i16>
   ret <vscale x 8 x i16> %res
 }
 
+define <vscale x 8 x i16> @ldnf1sb_h_inbound(<vscale x 8 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1sb_h_inbound:
+; CHECK: ldnf1sb { z0.h }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 8 x i8>*
+  %base = getelementptr <vscale x 8 x i8>, <vscale x 8 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 8 x i8>* %base to i8*
+  %load = call <vscale x 8 x i8> @llvm.aarch64.sve.ldnf1.nxv8i8(<vscale x 8 x i1> %pg, i8* %base_scalar)
+  %res = sext <vscale x 8 x i8> %load to <vscale x 8 x i16>
+  ret <vscale x 8 x i16> %res
+}
+
 define <vscale x 8 x i16> @ldnf1h(<vscale x 8 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldnf1h:
 ; CHECK: ldnf1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x i16> @llvm.aarch64.sve.ldnf1.nxv8i16(<vscale x 8 x i1> %pg, i16* %a)
   ret <vscale x 8 x i16> %load
 }
 
+define <vscale x 8 x i16> @ldnf1h_inbound(<vscale x 8 x i1> %pg, i16* %a) {
+; CHECK-LABEL: ldnf1h_inbound:
+; CHECK: ldnf1h { z0.h }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i16* %a to <vscale x 8 x i16>*
+  %base = getelementptr <vscale x 8 x i16>, <vscale x 8 x i16>* %base_scalable, i64 1
+  %base_scalar = bitcast <vscale x 8 x i16>* %base to i16*
+  %load = call <vscale x 8 x i16> @llvm.aarch64.sve.ldnf1.nxv8i16(<vscale x 8 x i1> %pg, i16* %base_scalar)
+  ret <vscale x 8 x i16> %load
+}
+
 define <vscale x 8 x half> @ldnf1h_f16(<vscale x 8 x i1> %pg, half* %a) {
 ; CHECK-LABEL: ldnf1h_f16:
 ; CHECK: ldnf1h { z0.h }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 8 x half> @llvm.aarch64.sve.ldnf1.nxv8f16(<vscale x 8 x i1> %pg, half* %a)
   ret <vscale x 8 x half> %load
 }
 
+define <vscale x 8 x half> @ldnf1h_f16_inbound(<vscale x 8 x i1> %pg, half* %a) {
+; CHECK-LABEL: ldnf1h_f16_inbound:
+; CHECK: ldnf1h { z0.h }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast half* %a to <vscale x 8 x half>*
+  %base = getelementptr <vscale x 8 x half>, <vscale x 8 x half>* %base_scalable, i64 1
+  %base_scalar = bitcast <vscale x 8 x half>* %base to half*
+  %load = call <vscale x 8 x half> @llvm.aarch64.sve.ldnf1.nxv8f16(<vscale x 8 x i1> %pg, half* %base_scalar)
+  ret <vscale x 8 x half> %load
+}
+
 define <vscale x 4 x i32> @ldnf1b_s(<vscale x 4 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1b_s:
 ; CHECK: ldnf1b { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldnf1.nxv4i8(<vscale x 4 x i1> %pg, i8* %a)
   %res = zext <vscale x 4 x i8> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldnf1b_s_inbound(<vscale x 4 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_s_inbound:
+; CHECK: ldnf1b { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 4 x i8>*
+  %base = getelementptr <vscale x 4 x i8>, <vscale x 4 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x i8>* %base to i8*
+  %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldnf1.nxv4i8(<vscale x 4 x i1> %pg, i8* %base_scalar)
+  %res = zext <vscale x 4 x i8> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 4 x i32> @ldnf1sb_s(<vscale x 4 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1sb_s:
 ; CHECK: ldnf1sb { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldnf1.nxv4i8(<vscale x 4 x i1> %pg, i8* %a)
   %res = sext <vscale x 4 x i8> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldnf1sb_s_inbound(<vscale x 4 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1sb_s_inbound:
+; CHECK: ldnf1sb { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 4 x i8>*
+  %base = getelementptr <vscale x 4 x i8>, <vscale x 4 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x i8>* %base to i8*
+  %load = call <vscale x 4 x i8> @llvm.aarch64.sve.ldnf1.nxv4i8(<vscale x 4 x i1> %pg, i8* %base_scalar)
+  %res = sext <vscale x 4 x i8> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 4 x i32> @ldnf1h_s(<vscale x 4 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldnf1h_s:
 ; CHECK: ldnf1h { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldnf1.nxv4i16(<vscale x 4 x i1> %pg, i16* %a)
   %res = zext <vscale x 4 x i16> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldnf1h_s_inbound(<vscale x 4 x i1> %pg, i16* %a) {
+; CHECK-LABEL: ldnf1h_s_inbound:
+; CHECK: ldnf1h { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i16* %a to <vscale x 4 x i16>*
+  %base = getelementptr <vscale x 4 x i16>, <vscale x 4 x i16>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x i16>* %base to i16*
+  %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldnf1.nxv4i16(<vscale x 4 x i1> %pg, i16* %base_scalar)
+  %res = zext <vscale x 4 x i16> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 4 x i32> @ldnf1sh_s(<vscale x 4 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldnf1sh_s:
 ; CHECK: ldnf1sh { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldnf1.nxv4i16(<vscale x 4 x i1> %pg, i16* %a)
   %res = sext <vscale x 4 x i16> %load to <vscale x 4 x i32>
   ret <vscale x 4 x i32> %res
 }
 
+define <vscale x 4 x i32> @ldnf1sh_s_inbound(<vscale x 4 x i1> %pg, i16* %a) {
+; CHECK-LABEL: ldnf1sh_s_inbound:
+; CHECK: ldnf1sh { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i16* %a to <vscale x 4 x i16>*
+  %base = getelementptr <vscale x 4 x i16>, <vscale x 4 x i16>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x i16>* %base to i16*
+  %load = call <vscale x 4 x i16> @llvm.aarch64.sve.ldnf1.nxv4i16(<vscale x 4 x i1> %pg, i16* %base_scalar)
+  %res = sext <vscale x 4 x i16> %load to <vscale x 4 x i32>
+  ret <vscale x 4 x i32> %res
+}
+
 define <vscale x 4 x i32> @ldnf1w(<vscale x 4 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldnf1w:
 ; CHECK: ldnf1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x i32> @llvm.aarch64.sve.ldnf1.nxv4i32(<vscale x 4 x i1> %pg, i32* %a)
   ret <vscale x 4 x i32> %load
 }
 
+define <vscale x 4 x i32> @ldnf1w_inbound(<vscale x 4 x i1> %pg, i32* %a) {
+; CHECK-LABEL: ldnf1w_inbound:
+; CHECK: ldnf1w { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i32* %a to <vscale x 4 x i32>*
+  %base = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x i32>* %base to i32*
+  %load = call <vscale x 4 x i32> @llvm.aarch64.sve.ldnf1.nxv4i32(<vscale x 4 x i1> %pg, i32* %base_scalar)
+  ret <vscale x 4 x i32> %load
+}
+
 define <vscale x 4 x float> @ldnf1w_f32(<vscale x 4 x i1> %pg, float* %a) {
 ; CHECK-LABEL: ldnf1w_f32:
 ; CHECK: ldnf1w { z0.s }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 4 x float> @llvm.aarch64.sve.ldnf1.nxv4f32(<vscale x 4 x i1> %pg, float* %a)
   ret <vscale x 4 x float> %load
 }
 
+define <vscale x 4 x float> @ldnf1w_f32_inbound(<vscale x 4 x i1> %pg, float* %a) {
+; CHECK-LABEL: ldnf1w_f32_inbound:
+; CHECK: ldnf1w { z0.s }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast float* %a to <vscale x 4 x float>*
+  %base = getelementptr <vscale x 4 x float>, <vscale x 4 x float>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 4 x float>* %base to float*
+  %load = call <vscale x 4 x float> @llvm.aarch64.sve.ldnf1.nxv4f32(<vscale x 4 x i1> %pg, float* %base_scalar)
+  ret <vscale x 4 x float> %load
+}
+
 define <vscale x 2 x i64> @ldnf1b_d(<vscale x 2 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1b_d:
 ; CHECK: ldnf1b { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldnf1.nxv2i8(<vscale x 2 x i1> %pg, i8* %a)
   %res = zext <vscale x 2 x i8> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1b_d_inbound(<vscale x 2 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1b_d_inbound:
+; CHECK: ldnf1b { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 2 x i8>*
+  %base = getelementptr <vscale x 2 x i8>, <vscale x 2 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i8>* %base to i8*
+  %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldnf1.nxv2i8(<vscale x 2 x i1> %pg, i8* %base_scalar)
+  %res = zext <vscale x 2 x i8> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1sb_d(<vscale x 2 x i1> %pg, i8* %a) {
 ; CHECK-LABEL: ldnf1sb_d:
 ; CHECK: ldnf1sb { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldnf1.nxv2i8(<vscale x 2 x i1> %pg, i8* %a)
   %res = sext <vscale x 2 x i8> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1sb_d_inbound(<vscale x 2 x i1> %pg, i8* %a) {
+; CHECK-LABEL: ldnf1sb_d_inbound:
+; CHECK: ldnf1sb { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i8* %a to <vscale x 2 x i8>*
+  %base = getelementptr <vscale x 2 x i8>, <vscale x 2 x i8>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i8>* %base to i8*
+  %load = call <vscale x 2 x i8> @llvm.aarch64.sve.ldnf1.nxv2i8(<vscale x 2 x i1> %pg, i8* %base_scalar)
+  %res = sext <vscale x 2 x i8> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1h_d(<vscale x 2 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldnf1h_d:
 ; CHECK: ldnf1h { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldnf1.nxv2i16(<vscale x 2 x i1> %pg, i16* %a)
   %res = zext <vscale x 2 x i16> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1h_d_inbound(<vscale x 2 x i1> %pg, i16* %a) {
+; CHECK-LABEL: ldnf1h_d_inbound:
+; CHECK: ldnf1h { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i16* %a to <vscale x 2 x i16>*
+  %base = getelementptr <vscale x 2 x i16>, <vscale x 2 x i16>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i16>* %base to i16*
+  %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldnf1.nxv2i16(<vscale x 2 x i1> %pg, i16* %base_scalar)
+  %res = zext <vscale x 2 x i16> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1sh_d(<vscale x 2 x i1> %pg, i16* %a) {
 ; CHECK-LABEL: ldnf1sh_d:
 ; CHECK: ldnf1sh { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldnf1.nxv2i16(<vscale x 2 x i1> %pg, i16* %a)
   %res = sext <vscale x 2 x i16> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1sh_d_inbound(<vscale x 2 x i1> %pg, i16* %a) {
+; CHECK-LABEL: ldnf1sh_d_inbound:
+; CHECK: ldnf1sh { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i16* %a to <vscale x 2 x i16>*
+  %base = getelementptr <vscale x 2 x i16>, <vscale x 2 x i16>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i16>* %base to i16*
+  %load = call <vscale x 2 x i16> @llvm.aarch64.sve.ldnf1.nxv2i16(<vscale x 2 x i1> %pg, i16* %base_scalar)
+  %res = sext <vscale x 2 x i16> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1w_d(<vscale x 2 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldnf1w_d:
 ; CHECK: ldnf1w { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldnf1.nxv2i32(<vscale x 2 x i1> %pg, i32* %a)
   %res = zext <vscale x 2 x i32> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1w_d_inbound(<vscale x 2 x i1> %pg, i32* %a) {
+; CHECK-LABEL: ldnf1w_d_inbound:
+; CHECK: ldnf1w { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i32* %a to <vscale x 2 x i32>*
+  %base = getelementptr <vscale x 2 x i32>, <vscale x 2 x i32>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i32>* %base to i32*
+  %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldnf1.nxv2i32(<vscale x 2 x i1> %pg, i32* %base_scalar)
+  %res = zext <vscale x 2 x i32> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1sw_d(<vscale x 2 x i1> %pg, i32* %a) {
 ; CHECK-LABEL: ldnf1sw_d:
 ; CHECK: ldnf1sw { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldnf1.nxv2i32(<vscale x 2 x i1> %pg, i32* %a)
   %res = sext <vscale x 2 x i32> %load to <vscale x 2 x i64>
   ret <vscale x 2 x i64> %res
 }
 
+define <vscale x 2 x i64> @ldnf1sw_d_inbound(<vscale x 2 x i1> %pg, i32* %a) {
+; CHECK-LABEL: ldnf1sw_d_inbound:
+; CHECK: ldnf1sw { z0.d }, p0/z, [x0, #7, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i32* %a to <vscale x 2 x i32>*
+  %base = getelementptr <vscale x 2 x i32>, <vscale x 2 x i32>* %base_scalable, i64 7
+  %base_scalar = bitcast <vscale x 2 x i32>* %base to i32*
+  %load = call <vscale x 2 x i32> @llvm.aarch64.sve.ldnf1.nxv2i32(<vscale x 2 x i1> %pg, i32* %base_scalar)
+  %res = sext <vscale x 2 x i32> %load to <vscale x 2 x i64>
+  ret <vscale x 2 x i64> %res
+}
+
 define <vscale x 2 x i64> @ldnf1d(<vscale x 2 x i1> %pg, i64* %a) {
 ; CHECK-LABEL: ldnf1d:
 ; CHECK: ldnf1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x i64> @llvm.aarch64.sve.ldnf1.nxv2i64(<vscale x 2 x i1> %pg, i64* %a)
   ret <vscale x 2 x i64> %load
 }
 
+define <vscale x 2 x i64> @ldnf1d_inbound(<vscale x 2 x i1> %pg, i64* %a) {
+; CHECK-LABEL: ldnf1d_inbound:
+; CHECK: ldnf1d { z0.d }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast i64* %a to <vscale x 2 x i64>*
+  %base = getelementptr <vscale x 2 x i64>, <vscale x 2 x i64>* %base_scalable, i64 1
+  %base_scalar = bitcast <vscale x 2 x i64>* %base to i64*
+  %load = call <vscale x 2 x i64> @llvm.aarch64.sve.ldnf1.nxv2i64(<vscale x 2 x i1> %pg, i64* %base_scalar)
+  ret <vscale x 2 x i64> %load
+}
+
 define <vscale x 2 x double> @ldnf1d_f64(<vscale x 2 x i1> %pg, double* %a) {
 ; CHECK-LABEL: ldnf1d_f64:
 ; CHECK: ldnf1d { z0.d }, p0/z, [x0]
 ; CHECK-NEXT: ret
   %load = call <vscale x 2 x double> @llvm.aarch64.sve.ldnf1.nxv2f64(<vscale x 2 x i1> %pg, double* %a)
   ret <vscale x 2 x double> %load
 }
 
+define <vscale x 2 x double> @ldnf1d_f64_inbound(<vscale x 2 x i1> %pg, double* %a) {
+; CHECK-LABEL: ldnf1d_f64_inbound:
+; CHECK: ldnf1d { z0.d }, p0/z, [x0, #1, mul vl]
+; CHECK-NEXT: ret
+  %base_scalable = bitcast double* %a to <vscale x 2 x double>*
+  %base = getelementptr <vscale x 2 x double>, <vscale x 2 x double>* %base_scalable, i64 1
+  %base_scalar = bitcast <vscale x 2 x double>* %base to double*
+  %load = call <vscale x 2 x double> @llvm.aarch64.sve.ldnf1.nxv2f64(<vscale x 2 x i1> %pg, double* %base_scalar)
+  ret <vscale x 2 x double> %load
+}
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.ldnf1.nxv16i8(<vscale x 16 x i1>, i8*)
 
 declare <vscale x 8 x i8> @llvm.aarch64.sve.ldnf1.nxv8i8(<vscale x 8 x i1>, i8*)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.ldnf1.nxv8i16(<vscale x 8 x i1>, i16*)
 declare <vscale x 8 x half> @llvm.aarch64.sve.ldnf1.nxv8f16(<vscale x 8 x i1>, half*)
 
 declare <vscale x 4 x i8> @llvm.aarch64.sve.ldnf1.nxv4i8(<vscale x 4 x i1>, i8*)
 declare <vscale x 4 x i16> @llvm.aarch64.sve.ldnf1.nxv4i16(<vscale x 4 x i1>, i16*)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.ldnf1.nxv4i32(<vscale x 4 x i1>, i32*)
 declare <vscale x 4 x float> @llvm.aarch64.sve.ldnf1.nxv4f32(<vscale x 4 x i1>, float*)
 
 declare <vscale x 2 x i8> @llvm.aarch64.sve.ldnf1.nxv2i8(<vscale x 2 x i1>, i8*)
 declare <vscale x 2 x i16> @llvm.aarch64.sve.ldnf1.nxv2i16(<vscale x 2 x i1>, i16*)
 declare <vscale x 2 x i32> @llvm.aarch64.sve.ldnf1.nxv2i32(<vscale x 2 x i1>, i32*)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.ldnf1.nxv2i64(<vscale x 2 x i1>, i64*)
 declare <vscale x 2 x double> @llvm.aarch64.sve.ldnf1.nxv2f64(<vscale x 2 x i1>, double*)