[llvm][CodeGen][aarch64] Add contiguous prefetch intrinsics for SVE.

[fpetrogalli]

Summary

The patch covers both register/register and register/immediate addressing modes.

[fpetrogalli]

This is not yet ready to review.

[fpetrogalli]

Replace <n x ... with <vscale x ... in the test.

[fpetrogalli]

The patch now is ready to review. It covers only the contiguous prefetches. I will create a separate revision for the gather prefetches.

[fpetrogalli]

llvm/include/llvm/IR/IntrinsicsAArch64.td

1238–1240

Ideally, I would have liked to define this as something like:

def int_aarch64_sve_prf : Intrinsic<[], [PointerTo<llvm_anyvector_ty>,
                                         LLVMScalarOrSameVectorWidth<0, llvm_i1_ty>,
                                         llvm_i32_ty], [IntrArgMemOnly]>;

For some definition of PointerTo, but I am not aware of anything in the current intrinsic generator that can be used for that.

[sdesmalen]

nit: in the title:
s/continuous/contiguous/

LGTM otherwise

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4414 ↗

(On Diff #245526)

nit: Any reason you're using an anonymous namespace over static for these functions?

4424 ↗

(On Diff #245526)

nit: you can fold this into the condition above.

4432 ↗

(On Diff #245526)

instead of 128, can you use AArch64::SVEBitsPerBlock?

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4414 ↗

(On Diff #245526)

I should have really used static to adhere to the style of the rest of the file... Done.

[fpetrogalli]

Address code review from @sdesmalen.

[andwar]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4443 ↗

(On Diff #245961)

[nit] corresponding?

4491 ↗

(On Diff #245961)

What's this check for? Wouldn't some hard-coded null-value be more idiomatic?

llvm/test/CodeGen/AArch64/sve-intrinsics-contiguous-prefetches.ll

100 ↗

(On Diff #245961)

Would it make sense to add checks for what's expected instead?

[fpetrogalli]

llvm/lib/Target/AArch64/AArch64ISelDAGToDAG.cpp

4491 ↗

(On Diff #245961)

I think EVT() is there to simulate a nullptr, see http://llvm.org/doxygen/ValueTypes_8h_source.html:

struct EVT {
private:
  MVT V = MVT::INVALID_SIMPLE_VALUE_TYPE;
  Type *LLVMTy = nullptr;
 
public:
  constexpr EVT() = default;

Essentially, if statement checks that the MemVT is not { MVT::INVALID_SIMPLE_VALUE_TYPE, nullptr}.

llvm/test/CodeGen/AArch64/sve-intrinsics-contiguous-prefetches.ll

100 ↗

(On Diff #245961)

I guess you are right. We could miss a failure in the addressing mode generation if we have (at the same time) a failure in the rendering of the immediate constant... quite a remote possibility, which I am sure it will be caught in time in some other tests.

So I think we can leave it as it is.

llvm/include/llvm/IR/IntrinsicsAArch64.td

 
 //
 // Stores
 //
 
 def int_aarch64_sve_stnt1 : AdvSIMD_1Vec_PredStore_Intrinsic;
 
 //
+// Prefetches
+//
+
+def int_aarch64_sve_prf : Intrinsic<[], [llvm_anyvector_ty,
+                                         llvm_ptr_ty,
+                                         llvm_i32_ty], [IntrArgMemOnly]>;
+def int_aarch64_sve_prfb_gather : Intrinsic<[],
+                                            [llvm_anyvector_ty,
+                                             LLVMPointerType<llvm_i8_ty>,
+                                             LLVMScalarOrSameVectorWidth<0,llvm_i64_ty>,
+                                             llvm_i32_ty],
+                                            [IntrArgMemOnly]>;
+def int_aarch64_sve_prfh_gather : Intrinsic<[],
+                                            [llvm_anyvector_ty,
+                                             LLVMPointerType<llvm_i8_ty>,
+                                             LLVMScalarOrSameVectorWidth<0,llvm_i64_ty>,
+                                             llvm_i32_ty],
+                                            [IntrArgMemOnly]>;
+def int_aarch64_sve_prfw_gather : Intrinsic<[],
+                                            [llvm_anyvector_ty,
+                                             LLVMPointerType<llvm_i8_ty>,
+                                             LLVMScalarOrSameVectorWidth<0,llvm_i64_ty>,
+                                             llvm_i32_ty],
+                                            [IntrArgMemOnly]>;
+def int_aarch64_sve_prfd_gather : Intrinsic<[],
+                                            [llvm_anyvector_ty,
+                                             LLVMPointerType<llvm_i8_ty>,
+                                             LLVMScalarOrSameVectorWidth<0,llvm_i64_ty>,
+                                             llvm_i32_ty],
+                                            [IntrArgMemOnly]>;
+
+//
 // Scalar to vector operations
 //
 
 def int_aarch64_sve_dup : AdvSIMD_SVE_DUP_Intrinsic;
 
 //
 // Integer arithmetic
 //

llvm/lib/Target/AArch64/AArch64ISelLowering.h

enum NodeType : unsigned {
 
   STG,
   STZG,
   ST2G,
   STZ2G,
 
   LDP,
   STP,
-  STNP
+  STNP,
+
+  // SVE prefetch
+  GPRF_S_IMM,        
+  GPRF_D_IMM,        
+  GPRF_D_SCALED,     
+  GPRF_S_SXTW_SCALED,
+  GPRF_S_UXTW_SCALED,
+  GPRF_D_SXTW_SCALED,
+  GPRF_D_UXTW_SCALED,    
+    
 };
 
 } // end namespace AArch64ISD
 
 namespace {
 
 // Any instruction that defines a 32-bit result zeros out the high half of the
 // register. Truncate can be lowered to EXTRACT_SUBREG. CopyFromReg may

llvm/lib/Target/AArch64/AArch64ISelLowering.cpp

const char *AArch64TargetLowering::getTargetNodeName(unsigned Opcode) const {
   case AArch64ISD::SST1_UXTW:         return "AArch64ISD::SST1_UXTW";
   case AArch64ISD::SST1_SXTW_SCALED:  return "AArch64ISD::SST1_SXTW_SCALED";
   case AArch64ISD::SST1_UXTW_SCALED:  return "AArch64ISD::SST1_UXTW_SCALED";
   case AArch64ISD::SST1_IMM:          return "AArch64ISD::SST1_IMM";
   case AArch64ISD::LDP:               return "AArch64ISD::LDP";
   case AArch64ISD::STP:               return "AArch64ISD::STP";
   case AArch64ISD::STNP:              return "AArch64ISD::STNP";
   case AArch64ISD::DUP_PRED:          return "AArch64ISD::DUP_PRED";
+  case AArch64ISD::GPRF_S_IMM:         return "AArch64ISD::GPRF_S_IMM";
+  case AArch64ISD::GPRF_D_IMM:         return "AArch64ISD::GPRF_D_IMM";
+  case AArch64ISD::GPRF_D_SCALED:      return "AArch64ISD::GPRF_D_SCALED";
+  case AArch64ISD::GPRF_S_SXTW_SCALED: return "AArch64ISD::GPRF_S_SXTW_SCALED";
+  case AArch64ISD::GPRF_S_UXTW_SCALED: return "AArch64ISD::GPRF_S_UXTW_SCALED";
+  case AArch64ISD::GPRF_D_SXTW_SCALED: return "AArch64ISD::GPRF_D_SXTW_SCALED";
+  case AArch64ISD::GPRF_D_UXTW_SCALED: return "AArch64ISD::GPRF_D_UXTW_SCALED";
   }
   return nullptr;
 }
 
 MachineBasicBlock *
 AArch64TargetLowering::EmitF128CSEL(MachineInstr &MI,
                                     MachineBasicBlock *MBB) const {
   // We materialise the F128CSEL pseudo-instruction as some control flow and a

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

 //=- AArch64SVEInstrInfo.td -  AArch64 SVE Instructions -*- tablegen -*-----=//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // AArch64 Scalable Vector Extension (SVE) Instruction definitions.
 //
 //===----------------------------------------------------------------------===//
 
+def SDT_AArch64_GPRF : SDTypeProfile< 0, 5, [
+  SDTCisVec<0>, SDTCisPtrTy<1>, SDTCisVec<2>, SDTCisInt<3>,
+  SDTCVecEltisVT<0,i1>, SDTCisSameNumEltsAs<0, 2>
+]>;
+
+def AArch64prf_gather_s_imm         : SDNode<"AArch64ISD::GPRF_S_IMM",         SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_d_imm         : SDNode<"AArch64ISD::GPRF_D_IMM",         SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_d_scaled      : SDNode<"AArch64ISD::GPRF_D_SCALED",      SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_s_sxtw_scaled : SDNode<"AArch64ISD::GPRF_S_SXTW_SCALED", SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_s_uxtw_scaled : SDNode<"AArch64ISD::GPRF_S_UXTW_SCALED", SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_d_sxtw_scaled : SDNode<"AArch64ISD::GPRF_D_SXTW_SCALED", SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+def AArch64prf_gather_d_uxtw_scaled : SDNode<"AArch64ISD::GPRF_D_UXTW_SCALED", SDT_AArch64_GPRF, [SDNPHasChain, SDNPMayLoad]>;
+
 def SDT_AArch64_LDNF1 : SDTypeProfile<1, 3, [
   SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>,
   SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
 ]>;
 
 def SDT_AArch64_GLD1 : SDTypeProfile<1, 4, [
   SDTCisVec<0>, SDTCisVec<1>, SDTCisPtrTy<2>, SDTCisVec<3>, SDTCisVT<4, OtherVT>,
   SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0,1>
let Predicates = [HasSVE] in {
   // Contiguous prefetch (register + register)
   def PRFB_PRR : sve_mem_prfm_ss<0b001, "prfb", GPR64NoXZRshifted8>;
   def PRFH_PRR : sve_mem_prfm_ss<0b011, "prfh", GPR64NoXZRshifted16>;
   def PRFS_PRR : sve_mem_prfm_ss<0b101, "prfw", GPR64NoXZRshifted32>;
   def PRFD_PRR : sve_mem_prfm_ss<0b111, "prfd", GPR64NoXZRshifted64>;
 
   // Gather prefetch using scaled 32-bit offsets, e.g.
   //    prfh pldl1keep, p0, [x0, z0.s, uxtw #1]
-  defm PRFB_S : sve_mem_32b_prfm_sv_scaled<0b00, "prfb", ZPR32ExtSXTW8Only,  ZPR32ExtUXTW8Only>;
-  defm PRFH_S : sve_mem_32b_prfm_sv_scaled<0b01, "prfh", ZPR32ExtSXTW16, ZPR32ExtUXTW16>;
-  defm PRFW_S : sve_mem_32b_prfm_sv_scaled<0b10, "prfw", ZPR32ExtSXTW32, ZPR32ExtUXTW32>;
-  defm PRFD_S : sve_mem_32b_prfm_sv_scaled<0b11, "prfd", ZPR32ExtSXTW64, ZPR32ExtUXTW64>;
+  defm PRFB_S : sve_mem_32b_prfm_sv_scaled<0b00, "prfb", AArch64prf_gather_s_sxtw_scaled, AArch64prf_gather_s_uxtw_scaled, ZPR32ExtSXTW8Only, ZPR32ExtUXTW8Only, i8>;
+  defm PRFH_S : sve_mem_32b_prfm_sv_scaled<0b01, "prfh", AArch64prf_gather_s_sxtw_scaled, AArch64prf_gather_s_uxtw_scaled, ZPR32ExtSXTW16, ZPR32ExtUXTW16, i16>;
+  defm PRFW_S : sve_mem_32b_prfm_sv_scaled<0b10, "prfw", AArch64prf_gather_s_sxtw_scaled, AArch64prf_gather_s_uxtw_scaled, ZPR32ExtSXTW32, ZPR32ExtUXTW32, i32>;
+  defm PRFD_S : sve_mem_32b_prfm_sv_scaled<0b11, "prfd", AArch64prf_gather_s_sxtw_scaled, AArch64prf_gather_s_uxtw_scaled, ZPR32ExtSXTW64, ZPR32ExtUXTW64, i64>;
 
   // Gather prefetch using unpacked, scaled 32-bit offsets, e.g.
   //    prfh pldl1keep, p0, [x0, z0.d, uxtw #1]
-  defm PRFB_D : sve_mem_64b_prfm_sv_ext_scaled<0b00, "prfb", ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only>;
-  defm PRFH_D : sve_mem_64b_prfm_sv_ext_scaled<0b01, "prfh", ZPR64ExtSXTW16, ZPR64ExtUXTW16>;
-  defm PRFW_D : sve_mem_64b_prfm_sv_ext_scaled<0b10, "prfw", ZPR64ExtSXTW32, ZPR64ExtUXTW32>;
-  defm PRFD_D : sve_mem_64b_prfm_sv_ext_scaled<0b11, "prfd", ZPR64ExtSXTW64, ZPR64ExtUXTW64>;
+  defm PRFB_D : sve_mem_64b_prfm_sv_ext_scaled<0b00, "prfb", AArch64prf_gather_d_sxtw_scaled, AArch64prf_gather_d_uxtw_scaled, ZPR64ExtSXTW8Only, ZPR64ExtUXTW8Only, i8>;
+  defm PRFH_D : sve_mem_64b_prfm_sv_ext_scaled<0b01, "prfh", AArch64prf_gather_d_sxtw_scaled, AArch64prf_gather_d_uxtw_scaled, ZPR64ExtSXTW16, ZPR64ExtUXTW16, i16>;
+  defm PRFW_D : sve_mem_64b_prfm_sv_ext_scaled<0b10, "prfw", AArch64prf_gather_d_sxtw_scaled, AArch64prf_gather_d_uxtw_scaled, ZPR64ExtSXTW32, ZPR64ExtUXTW32, i32>;
+  defm PRFD_D : sve_mem_64b_prfm_sv_ext_scaled<0b11, "prfd", AArch64prf_gather_d_sxtw_scaled, AArch64prf_gather_d_uxtw_scaled, ZPR64ExtSXTW64, ZPR64ExtUXTW64, i64>;
 
   // Gather prefetch using scaled 64-bit offsets, e.g.
   //    prfh pldl1keep, p0, [x0, z0.d, lsl #1]
-  defm PRFB_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b00, "prfb", ZPR64ExtLSL8>;
-  defm PRFH_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b01, "prfh", ZPR64ExtLSL16>;
-  defm PRFW_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b10, "prfw", ZPR64ExtLSL32>;
-  defm PRFD_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b11, "prfd", ZPR64ExtLSL64>;
+  defm PRFB_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b00, "prfb", AArch64prf_gather_d_scaled, ZPR64ExtLSL8,  i8>;
+  defm PRFH_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b01, "prfh", AArch64prf_gather_d_scaled, ZPR64ExtLSL16, i16>;
+  defm PRFW_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b10, "prfw", AArch64prf_gather_d_scaled, ZPR64ExtLSL32, i32>;
+  defm PRFD_D_SCALED : sve_mem_64b_prfm_sv_lsl_scaled<0b11, "prfd", AArch64prf_gather_d_scaled, ZPR64ExtLSL64, i64>;
 
   // Gather prefetch using 32/64-bit pointers with offset, e.g.
   //    prfh pldl1keep, p0, [z0.s, #16]
   //    prfh pldl1keep, p0, [z0.d, #16]
-  defm PRFB_S_PZI : sve_mem_32b_prfm_vi<0b00, "prfb", imm0_31>;
-  defm PRFH_S_PZI : sve_mem_32b_prfm_vi<0b01, "prfh", uimm5s2>;
-  defm PRFW_S_PZI : sve_mem_32b_prfm_vi<0b10, "prfw", uimm5s4>;
-  defm PRFD_S_PZI : sve_mem_32b_prfm_vi<0b11, "prfd", uimm5s8>;
-
-  defm PRFB_D_PZI : sve_mem_64b_prfm_vi<0b00, "prfb", imm0_31>;
-  defm PRFH_D_PZI : sve_mem_64b_prfm_vi<0b01, "prfh", uimm5s2>;
-  defm PRFW_D_PZI : sve_mem_64b_prfm_vi<0b10, "prfw", uimm5s4>;
-  defm PRFD_D_PZI : sve_mem_64b_prfm_vi<0b11, "prfd", uimm5s8>;
+  defm PRFB_S_PZI : sve_mem_32b_prfm_vi<0b00, "prfb", imm0_31, AArch64prf_gather_s_imm, i8 >;
+  defm PRFH_S_PZI : sve_mem_32b_prfm_vi<0b01, "prfh", uimm5s2, AArch64prf_gather_s_imm, i16>;
+  defm PRFW_S_PZI : sve_mem_32b_prfm_vi<0b10, "prfw", uimm5s4, AArch64prf_gather_s_imm, i32>;
+  defm PRFD_S_PZI : sve_mem_32b_prfm_vi<0b11, "prfd", uimm5s8, AArch64prf_gather_s_imm, i64>;
+
+  defm PRFB_D_PZI : sve_mem_64b_prfm_vi<0b00, "prfb", imm0_31, AArch64prf_gather_d_imm, i8 >;
+  defm PRFH_D_PZI : sve_mem_64b_prfm_vi<0b01, "prfh", uimm5s2, AArch64prf_gather_d_imm, i16>;
+  defm PRFW_D_PZI : sve_mem_64b_prfm_vi<0b10, "prfw", uimm5s4, AArch64prf_gather_d_imm, i32>;
+  defm PRFD_D_PZI : sve_mem_64b_prfm_vi<0b11, "prfd", uimm5s8, AArch64prf_gather_d_imm, i64>;
 
   defm ADR_SXTW_ZZZ_D : sve_int_bin_cons_misc_0_a_sxtw<0b00, "adr">;
   defm ADR_UXTW_ZZZ_D : sve_int_bin_cons_misc_0_a_uxtw<0b01, "adr">;
   defm ADR_LSL_ZZZ_S  : sve_int_bin_cons_misc_0_a_32_lsl<0b10, "adr">;
   defm ADR_LSL_ZZZ_D  : sve_int_bin_cons_misc_0_a_64_lsl<0b11, "adr">;
 
   defm TBL_ZZZ  : sve_int_perm_tbl<"tbl", AArch64tbl>;
 

llvm/lib/Target/AArch64/SVEInstrFormats.td

: I<(outs), (ins sve_prfop:$prfop, PPR3bAny:$Pg, GPR64sp:$Rn, zprext:$Zm),
   let Inst{9-5}   = Rn;
   let Inst{4}     = 0b0;
   let Inst{3-0}   = prfop;
 
   let hasSideEffects = 1;
 }
 
 multiclass sve_mem_32b_prfm_sv_scaled<bits<2> msz, string asm,
+                                      SDPatternOperator sxtw_op,
+                                      SDPatternOperator uxtw_op,
                                       RegisterOperand sxtw_opnd,
-                                      RegisterOperand uxtw_opnd> {
+                                      RegisterOperand uxtw_opnd,
+                                      ValueType vt> {
   def _UXTW_SCALED : sve_mem_32b_prfm_sv<msz, 0, asm, uxtw_opnd>;
   def _SXTW_SCALED : sve_mem_32b_prfm_sv<msz, 1, asm, sxtw_opnd>;
+
+  def : Pat<(uxtw_op (nxv4i1 PPR:$gp), GPR64sp:$base, (nxv4i32 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME # _UXTW_SCALED) sve_prfop:$prfop, PPR:$gp, GPR64sp:$base, ZPR:$indices)>;
+  def : Pat<(sxtw_op (nxv4i1 PPR:$gp), GPR64sp:$base, (nxv4i32 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME # _SXTW_SCALED) sve_prfop:$prfop, PPR:$gp, GPR64sp:$base, ZPR:$indices)>;
 }
 
 class sve_mem_32b_prfm_vi<bits<2> msz, string asm, Operand imm_ty>
 : I<(outs), (ins sve_prfop:$prfop, PPR3bAny:$Pg, ZPR32:$Zn, imm_ty:$imm5),
   asm, "\t$prfop, $Pg, [$Zn, $imm5]",
   "",
   []>, Sched<[]> {
   bits<3> Pg;
   bits<5> Zn;
   bits<5> imm5;
   bits<4> prfop;
   let Inst{31-25} = 0b1000010;
   let Inst{24-23} = msz;
   let Inst{22-21} = 0b00;
   let Inst{20-16} = imm5;
   let Inst{15-13} = 0b111;
   let Inst{12-10} = Pg;
   let Inst{9-5}   = Zn;
   let Inst{4}     = 0b0;
   let Inst{3-0}   = prfop;
 }
 
-multiclass sve_mem_32b_prfm_vi<bits<2> msz, string asm, Operand imm_ty> {
+multiclass sve_mem_32b_prfm_vi<bits<2> msz, string asm, Operand imm_ty,
+                               SDPatternOperator prefetch, ValueType vt> {
   def NAME : sve_mem_32b_prfm_vi<msz, asm, imm_ty>;
 
+  def : Pat<(prefetch (nxv4i1 PPR:$gp), (i64 imm_ty:$imm5), (nxv4i32 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME) sve_prfop:$prfop, PPR:$gp, ZPR:$indices, imm_ty:$imm5)>;
+
   def : InstAlias<asm # "\t$prfop, $Pg, [$Zn]",
                   (!cast<Instruction>(NAME) sve_prfop:$prfop, PPR3bAny:$Pg, ZPR32:$Zn, 0), 1>;
 }
 
 class sve_mem_z_fill<string asm>
 : I<(outs ZPRAny:$Zt), (ins GPR64sp:$Rn, simm9:$imm9),
   asm, "\t$Zt, [$Rn, $imm9, mul vl]",
   "",
: I<(outs), (ins sve_prfop:$prfop, PPR3bAny:$Pg, GPR64sp:$Rn, zprext:$Zm),
   let Inst{9-5}   = Rn;
   let Inst{4}     = 0b0;
   let Inst{3-0}   = prfop;
 
   let hasSideEffects = 1;
 }
 
 multiclass sve_mem_64b_prfm_sv_ext_scaled<bits<2> msz, string asm,
+                                          SDPatternOperator sxtw_op,
+                                          SDPatternOperator uxtw_op,
                                           RegisterOperand sxtw_opnd,
-                                          RegisterOperand uxtw_opnd> {
+                                          RegisterOperand uxtw_opnd,
+                                          ValueType vt> {
   def _UXTW_SCALED : sve_mem_64b_prfm_sv<msz, 0, 0, asm, uxtw_opnd>;
   def _SXTW_SCALED : sve_mem_64b_prfm_sv<msz, 1, 0, asm, sxtw_opnd>;
+
+  def : Pat<(uxtw_op (nxv2i1 PPR:$gp), GPR64sp:$base, (nxv2i64 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME # _UXTW_SCALED) sve_prfop:$prfop, PPR:$gp, GPR64sp:$base, ZPR:$indices)>;
+  def : Pat<(sxtw_op (nxv2i1 PPR:$gp), GPR64sp:$base, (nxv2i64 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME # _SXTW_SCALED) sve_prfop:$prfop, PPR:$gp, GPR64sp:$base, ZPR:$indices)>;
 }
 
 multiclass sve_mem_64b_prfm_sv_lsl_scaled<bits<2> msz, string asm,
-                                          RegisterOperand zprext> {
+                                         SDPatternOperator op,
+                                         RegisterOperand zprext, ValueType vt> {
   def NAME : sve_mem_64b_prfm_sv<msz, 1, 1, asm, zprext>;
+
+  def : Pat<(op (nxv2i1 PPR:$gp), GPR64sp:$base, (nxv2i64 ZPR:$indices),
+                (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME) sve_prfop:$prfop, PPR:$gp, GPR64sp:$base,
+                                      ZPR:$indices)>;
 }
 
 
 class sve_mem_64b_prfm_vi<bits<2> msz, string asm, Operand imm_ty>
 : I<(outs), (ins sve_prfop:$prfop, PPR3bAny:$Pg, ZPR64:$Zn, imm_ty:$imm5),
   asm, "\t$prfop, $Pg, [$Zn, $imm5]",
   "",
   []>, Sched<[]> {
: I<(outs), (ins sve_prfop:$prfop, PPR3bAny:$Pg, ZPR64:$Zn, imm_ty:$imm5),
   let Inst{12-10} = Pg;
   let Inst{9-5}   = Zn;
   let Inst{4}     = 0b0;
   let Inst{3-0}   = prfop;
 
   let hasSideEffects = 1;
 }
 
-multiclass sve_mem_64b_prfm_vi<bits<2> msz, string asm, Operand imm_ty> {
+multiclass sve_mem_64b_prfm_vi<bits<2> msz, string asm, Operand imm_ty,
+                               SDPatternOperator prefetch, ValueType vt> {
   def NAME : sve_mem_64b_prfm_vi<msz, asm, imm_ty>;
 
+  def : Pat<(prefetch (nxv2i1 PPR:$gp), (i64 imm_ty:$imm5), (nxv2i64 ZPR:$indices), (i32 sve_prfop:$prfop), vt),
+            (!cast<Instruction>(NAME) sve_prfop:$prfop, PPR:$gp, ZPR:$indices, imm_ty:$imm5)>;
+
   def : InstAlias<asm # "\t$prfop, $Pg, [$Zn]",
                   (!cast<Instruction>(NAME) sve_prfop:$prfop, PPR3bAny:$Pg, ZPR64:$Zn, 0), 1>;
 }
 
-
 //===----------------------------------------------------------------------===//
 // SVE Compute Vector Address Group
 //===----------------------------------------------------------------------===//
 
 class sve_int_bin_cons_misc_0_a<bits<2> opc, bits<2> msz, string asm,
                                 ZPRRegOp zprty, RegisterOperand zprext>
 : I<(outs zprty:$Zd), (ins zprty:$Zn, zprext:$Zm),
   asm, "\t$Zd, [$Zn, $Zm]",

llvm/test/CodeGen/AArch64/sve-intrinsics-prefetches.ll

+; RUN: llc -mtriple=aarch64-linux-gnu -mattr=+sve < %s | FileCheck %s
+
+;
+; Testing prfop encodings
+;
+define void @test_svprf_pldl1strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pldl1strm
+; CHECK: prfb pldl1strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 1)
+  ret void
+}
+
+define void @test_svprf_pldl2keep(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pldl2keep
+; CHECK: prfb pldl2keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 2)
+  ret void
+}
+
+define void @test_svprf_pldl2strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pldl2strm
+; CHECK: prfb pldl2strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 3)
+  ret void
+}
+
+define void @test_svprf_pldl3keep(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pldl3keep
+; CHECK: prfb pldl3keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 4)
+  ret void
+}
+
+define void @test_svprf_pldl3strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pldl3strm
+; CHECK: prfb pldl3strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 5)
+  ret void
+}
+
+define void @test_svprf_pstl1keep(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl1keep
+; CHECK: prfb pstl1keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 8)
+  ret void
+}
+
+define void @test_svprf_pstl1strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl1strm
+; CHECK: prfb pstl1strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 9)
+  ret void
+}
+
+define void @test_svprf_pstl2keep(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl2keep
+; CHECK: prfb pstl2keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 10)
+  ret void
+}
+
+define void @test_svprf_pstl2strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl2strm
+; CHECK: prfb pstl2strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 11)
+  ret void
+}
+
+define void @test_svprf_pstl3keep(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl3keep
+; CHECK: prfb pstl3keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 12)
+  ret void
+}
+
+define void @test_svprf_pstl3strm(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprf_pstl3strm
+; CHECK: prfb pstl3strm, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 13)
+  ret void
+}
+
+;
+; Testing imm limits of SI form
+;
+
+define void @test_svprf_vnum_under(<vscale x 16 x i1> %pg, <vscale x 16 x i8>* %base) {
+; CHECK-LABEL: test_svprf_vnum_under
+; CHECK-NOT: prfb pstl3strm, p0, [x0, #-33, mul vl]
+entry:
+  %gep = getelementptr inbounds <vscale x 16 x i8>, <vscale x 16 x i8>* %base, i64 -33, i64 0
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %gep, i32 13)
+  ret void
+}
+
+define void @test_svprf_vnum_min(<vscale x 16 x i1> %pg, <vscale x 16 x i8>* %base) {
+; CHECK-LABEL: test_svprf_vnum_min
+; CHECK: prfb pstl3strm, p0, [x0, #-32, mul vl]
+entry:
+  %gep = getelementptr inbounds <vscale x 16 x i8>, <vscale x 16 x i8>* %base, i64 -32, i64 0
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %gep, i32 13)
+  ret void
+}
+
+define void @test_svprf_vnum_over(<vscale x 16 x i1> %pg, <vscale x 16 x i8>* %base) {
+; CHECK-LABEL: test_svprf_vnum_over
+; CHECK-NOT: prfb pstl3strm, p0, [x0, #32, mul vl]
+entry:
+  %gep = getelementptr inbounds <vscale x 16 x i8>, <vscale x 16 x i8>* %base, i64 32, i64 0
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %gep, i32 13)
+  ret void
+}
+
+define void @test_svprf_vnum_max(<vscale x 16 x i1> %pg, <vscale x 16 x i8>* %base) {
+; CHECK-LABEL: test_svprf_vnum_max
+; CHECK: prfb pstl3strm, p0, [x0, #31, mul vl]
+entry:
+  %gep = getelementptr inbounds <vscale x 16 x i8>, <vscale x 16 x i8>* %base, i64 31, i64 0
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %gep, i32 13)
+  ret void
+}
+
+;
+; scalar contiguous
+;
+
+define void @test_svprfb(<vscale x 16 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprfb
+; CHECK: prfb pldl1keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %base, i32 0)
+  ret void
+}
+
+define void @test_svprfh(<vscale x 8 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprfh
+; CHECK: prfh pldl1keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv8i1(<vscale x 8 x i1> %pg, i8* %base, i32 0)
+  ret void
+}
+
+define void @test_svprfw(<vscale x 4 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprfw
+; CHECK: prfw pldl1keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, i32 0)
+  ret void
+}
+
+define void @test_svprfd(<vscale x 2 x i1> %pg, i8* %base) {
+; CHECK-LABEL: test_svprfd
+; CHECK: prfd pldl1keep, p0, [x0]
+entry:
+  tail call void @llvm.aarch64.sve.prf.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, i32 0)
+  ret void
+}
+
+;
+; scalar + imm contiguous
+;
+; imm form of prfb is tested above
+
+define void @test_svprfh_vnum(<vscale x 8 x i1> %pg, <vscale x 8 x i16>* %base) {
+; CHECK-LABEL: test_svprfh_vnum
+; CHECK: prfh pstl3strm, p0, [x0, #31, mul vl]
+entry:
+  %gep = getelementptr <vscale x 8 x i16>, <vscale x 8 x i16>* %base, i64 31
+  %addr = bitcast <vscale x 8 x i16>* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv8i1(<vscale x 8 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+define void @test_svprfw_vnum(<vscale x 4 x i1> %pg, <vscale x 4 x i32>* %base) {
+; CHECK-LABEL: test_svprfw_vnum
+; CHECK: prfw pstl3strm, p0, [x0, #31, mul vl]
+entry:
+  %gep = getelementptr <vscale x 4 x i32>, <vscale x 4 x i32>* %base, i64 31
+  %addr = bitcast <vscale x 4 x i32>* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv4i1(<vscale x 4 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+define void @test_svprfd_vnum(<vscale x 2 x i1> %pg, <vscale x 2 x i64>* %base) {
+; CHECK-LABEL: test_svprfd_vnum
+; CHECK: prfd pstl3strm, p0, [x0, #31, mul vl]
+entry:
+  %gep = getelementptr <vscale x 2 x i64>, <vscale x 2 x i64>* %base, i64 31
+  %addr = bitcast <vscale x 2 x i64>* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv2i1(<vscale x 2 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+;
+; scalar + scaled scalar contiguous
+;
+
+define void @test_svprfb_ss(<vscale x 16 x i1> %pg, i8* %base, i64 %offset) {
+; CHECK-LABEL: test_svprfb_ss
+; CHECK: prfb pstl3strm, p0, [x0, x1]
+entry:
+  %addr = getelementptr i8, i8* %base, i64 %offset
+  tail call void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+define void @test_svprfh_ss(<vscale x 8 x i1> %pg, i16* %base, i64 %offset) {
+; CHECK-LABEL: test_svprfh_ss
+; CHECK: prfh pstl3strm, p0, [x0, x1, lsl #1]
+entry:
+  %gep = getelementptr i16, i16* %base, i64 %offset
+  %addr = bitcast i16* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv8i1(<vscale x 8 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+define void @test_svprfw_ss(<vscale x 4 x i1> %pg, i32* %base, i64 %offset) {
+; CHECK-LABEL: test_svprfw_ss
+; CHECK: prfw pstl3strm, p0, [x0, x1, lsl #2]
+entry:
+  %gep = getelementptr i32, i32* %base, i64 %offset
+  %addr = bitcast i32* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv4i1(<vscale x 4 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+define void @test_svprfd_ss(<vscale x 2 x i1> %pg, i64* %base, i64 %offset) {
+; CHECK-LABEL: test_svprfd_ss
+; CHECK: prfd pstl3strm, p0, [x0, x1, lsl #3]
+entry:
+  %gep = getelementptr i64, i64* %base, i64 %offset
+  %addr = bitcast i64* %gep to i8*
+  tail call void @llvm.aarch64.sve.prf.nxv2i1(<vscale x 2 x i1> %pg, i8* %addr, i32 13)
+  ret void
+}
+
+
+;
+; scalar + vector gather - 32-bit scaled offset
+;
+define void @test_svprfb_u32offset(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %offsets) {
+; CHECK-LABEL: test_svprfb_u32offset
+; CHECK: prfb pstl3strm, p0, [x0, z0.s, uxtw]
+entry:
+  %i64_offsets = zext <vscale x 4 x i32> %offsets to <vscale x 4 x i64>
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %i64_offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32index(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfh_u32index
+; CHECK: prfh pstl3strm, p0, [x0, z0.s, uxtw #1]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = shl <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 1, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32index_mul(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfh_u32index_mul
+; CHECK: prfh pstl3strm, p0, [x0, z0.s, uxtw #1]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = mul <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 2, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32index(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfw_u32index
+; CHECK: prfw pstl3strm, p0, [x0, z0.s, uxtw #2]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = shl <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 2, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32index_mul(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfw_u32index_mul
+; CHECK: prfw pstl3strm, p0, [x0, z0.s, uxtw #2]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = mul <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 4, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32index(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfd_u32index
+; CHECK: prfd pstl3strm, p0, [x0, z0.s, uxtw #3]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = shl <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 3, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32index_mul(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i32> %indices) {
+; CHECK-LABEL: test_svprfd_u32index_mul
+; CHECK: prfd pstl3strm, p0, [x0, z0.s, uxtw #3]
+entry:
+  %i64_indices = zext <vscale x 4 x i32> %indices to <vscale x 4 x i64>
+  %offsets = mul <vscale x 4 x i64> %i64_indices, shufflevector (<vscale x 4 x i64> insertelement (<vscale x 4 x i64> undef, i64 8, i32 0), <vscale x 4 x i64> undef, <vscale x 4 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %base, <vscale x 4 x i64> %offsets, i32 13)
+  ret void
+}
+
+;
+; scalar + vector gather - 32-bit unpacked scaled offset
+;
+define void @test_svprfb_u32offset_unpacked(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %offsets) {
+; CHECK-LABEL: test_svprfb_u32offset_unpacked
+; CHECK: prfb pstl3strm, p0, [x0, z0.d, uxtw]
+entry:
+  %i64_offsets = zext <vscale x 2 x i32> %offsets to <vscale x 2 x i64>
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %i64_offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32index_unpacked(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfh_u32index_unpacked
+; CHECK: prfh pstl3strm, p0, [x0, z0.d, uxtw #1]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = shl <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 1, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32index_unpacked_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfh_u32index_unpacked_mul
+; CHECK: prfh pstl3strm, p0, [x0, z0.d, uxtw #1]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = mul <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 2, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32index_unpacked(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfw_u32index_unpacked
+; CHECK: prfw pstl3strm, p0, [x0, z0.d, uxtw #2]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = shl <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 2, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32index_unpacked_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfw_u32index_unpacked_mul
+; CHECK: prfw pstl3strm, p0, [x0, z0.d, uxtw #2]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = mul <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 4, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32index_unpacked(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfd_u32index_unpacked
+; CHECK: prfd pstl3strm, p0, [x0, z0.d, uxtw #3]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = shl <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 3, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32index_unpacked_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i32> %indices) {
+; CHECK-LABEL: test_svprfd_u32index_unpacked_mul
+; CHECK: prfd pstl3strm, p0, [x0, z0.d, uxtw #3]
+entry:
+  %i64_indices = zext <vscale x 2 x i32> %indices to <vscale x 2 x i64>
+  %offsets = mul <vscale x 2 x i64> %i64_indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 8, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+;
+; scalar + vector gather - 64-bit scaled offset
+;
+define void @test_svprfb_u64offset(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets) {
+; CHECK-LABEL: test_svprfb_u64offset
+; CHECK: prfb pstl3strm, p0, [x0, z0.d]
+entry:
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64offset(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfh_u64offset
+; CHECK: prfh pstl3strm, p0, [x0, z0.d, lsl #1]
+entry:
+  %offsets = shl <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 1, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64offset_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfh_u64offset_mul
+; CHECK: prfh pstl3strm, p0, [x0, z0.d, lsl #1]
+entry:
+  %offsets = mul <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 2, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64offset(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfw_u64offset
+; CHECK: prfw pstl3strm, p0, [x0, z0.d, lsl #2]
+entry:
+  %offsets = shl <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 2, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64offset_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfw_u64offset_mul
+; CHECK: prfw pstl3strm, p0, [x0, z0.d, lsl #2]
+entry:
+  %offsets = mul <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 4, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64offset(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfd_u64offset
+; CHECK: prfd pstl3strm, p0, [x0, z0.d, lsl #3]
+entry:
+  %offsets = shl <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 3, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64offset_mul(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %indices) {
+; CHECK-LABEL: test_svprfd_u64offset_mul
+; CHECK: prfd pstl3strm, p0, [x0, z0.d, lsl #3]
+entry:
+  %offsets = mul <vscale x 2 x i64> %indices, shufflevector (<vscale x 2 x i64> insertelement (<vscale x 2 x i64> undef, i64 8, i32 0), <vscale x 2 x i64> undef, <vscale x 2 x i32> zeroinitializer)
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %base, <vscale x 2 x i64> %offsets, i32 13)
+  ret void
+}
+
+;
+; vector plus imm gather - 32-bit element
+;
+define void @test_svprfb_u32base_vnum_under(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfb_u32base_vnum_under
+; CHECK: mov x[[BASE:[0-9]+]], #-1
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u32base_vnum_min(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfb_u32base_vnum_min
+; CHECK: prfb pstl3strm, p0, [z0.s]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* null, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u32base_vnum_max(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfb_u32base_vnum_max
+; CHECK: prfb pstl3strm, p0, [z0.s, #31]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 31 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u32base_vnum_over(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfb_u32base_vnum_over
+; CHECK: mov w[[BASE:[0-9]+]], #32
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 32 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32base_vnum_under(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfh_u32base_vnum_under
+; CHECK: mov x[[BASE:[0-9]+]], #-1
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32base_vnum_min(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfh_u32base_vnum_min
+; CHECK: prfh pstl3strm, p0, [z0.s]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* null, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32base_vnum_max(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfh_u32base_vnum_max
+; CHECK: prfh pstl3strm, p0, [z0.s, #62]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 62 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u32base_vnum_over(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfh_u32base_vnum_over
+; CHECK: mov w[[BASE:[0-9]+]], #64
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 64 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32base_vnum_under(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfw_u32base_vnum_under
+; CHECK: mov x[[BASE:[0-9]+]], #-1
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32base_vnum_min(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfw_u32base_vnum_min
+; CHECK: prfw pstl3strm, p0, [z0.s]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* null, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u32base_vnum_max(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfw_u32base_vnum_max
+; CHECK: prfw pstl3strm, p0, [z0.s, #124]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 124 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+define void @test_svprfw_u32base_vnum_over(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfw_u32base_vnum_over
+; CHECK: mov w[[BASE:[0-9]+]], #128
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 128 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32base_vnum_under(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfd_u32base_vnum_under
+; CHECK: mov x[[BASE:[0-9]+]], #-1
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32base_vnum_min(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfd_u32base_vnum_min
+; CHECK: prfd pstl3strm, p0, [z0.s]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* null, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32base_vnum_max(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfd_u32base_vnum_max
+; CHECK: prfd pstl3strm, p0, [z0.s, #248]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 248 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u32base_vnum_over(<vscale x 4 x i1> %pg, <vscale x 4 x i32> %bases)
+{
+; CHECK-LABEL: test_svprfd_u32base_vnum_over
+; CHECK: mov w[[BASE:[0-9]+]], #256
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.s, uxtw]
+entry:
+  %i64_bases = zext <vscale x 4 x i32> %bases to <vscale x 4 x i64>
+  %offset = inttoptr i64 256 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1> %pg, i8* %offset, <vscale x 4 x i64> %i64_bases, i32 13)
+  ret void
+}
+
+;
+; vector plus imm gather - 64-bit element
+;
+define void @test_svprfb_u64base_vnum_under(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfb_u64base_vnum_under
+; CHECK: mov x[[BASE:[0-9]+]], #-1
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.d]
+entry:
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u64base_vnum_min(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfb_u64base_vnum_min
+; CHECK: prfb pstl3strm, p0, [z0.d]
+entry:
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* null, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u64base_vnum_max(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfb_u64base_vnum_max
+; CHECK: prfb pstl3strm, p0, [z0.d, #31]
+entry:
+  %offset = inttoptr i64 31 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfb_u64base_vnum_over(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfb_u64base_vnum_over
+; CHECK: mov w[[BASE:[0-9]+]], #32
+; CHECK: prfb pstl3strm, p0, [x[[BASE]], z0.d]
+entry:
+  %offset = inttoptr i64 32 to i8*
+  tail call void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64base_vnum_under(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfh_u64base_vnum_under
+; CHECK: mov z[[SPLAT:[0-9]+]].d, #-1
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, z[[SPLAT]].d
+; CHECK: prfh pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64base_vnum_min(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfh_u64base_vnum_min
+; CHECK: prfh pstl3strm, p0, [z0.d]
+entry:
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* null, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64base_vnum_max(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfh_u64base_vnum_max
+; CHECK: prfh pstl3strm, p0, [z0.d, #62]
+entry:
+  %offset = inttoptr i64 62 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfh_u64base_vnum_over(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfh_u64base_vnum_over
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, #64
+; CHECK: prfh pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 64 to i8*
+  tail call void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64base_vnum_under(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfw_u64base_vnum_under
+; CHECK: mov z[[SPLAT:[0-9]+]].d, #-1
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, z[[SPLAT]].d
+; CHECK: prfw pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64base_vnum_min(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfw_u64base_vnum_min
+; CHECK: prfw pstl3strm, p0, [z0.d]
+entry:
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* null, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64base_vnum_max(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfw_u64base_vnum_max
+; CHECK: prfw pstl3strm, p0, [z0.d, #124]
+entry:
+  %offset = inttoptr i64 124 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfw_u64base_vnum_over(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfw_u64base_vnum_over
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, #128
+; CHECK: prfw pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 128 to i8*
+  tail call void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64base_vnum_under(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfd_u64base_vnum_under
+; CHECK: mov z[[SPLAT:[0-9]+]].d, #-1
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, z[[SPLAT]].d
+; CHECK: prfd pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 -1 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64base_vnum_min(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfd_u64base_vnum_min
+; CHECK: prfd pstl3strm, p0, [z0.d]
+entry:
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* null, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64base_vnum_max(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfd_u64base_vnum_max
+; CHECK: prfd pstl3strm, p0, [z0.d, #248]
+entry:
+  %offset = inttoptr i64 248 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+define void @test_svprfd_u64base_vnum_over(<vscale x 2 x i1> %pg, <vscale x 2 x i64> %bases)
+{
+; CHECK-LABEL: test_svprfd_u64base_vnum_over
+; CHECK: add z[[ADDR:[0-9]+]].d, z0.d, #256
+; CHECK: prfd pstl3strm, p0, [z[[ADDR]].d]
+entry:
+  %offset = inttoptr i64 256 to i8*
+  tail call void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1> %pg, i8* %offset, <vscale x 2 x i64> %bases, i32 13)
+  ret void
+}
+
+
+
+declare void @llvm.aarch64.sve.prf.nxv16i1(<vscale x 16 x i1>, i8*, i32)
+declare void @llvm.aarch64.sve.prf.nxv8i1(<vscale x 8 x i1>,  i8*, i32)
+declare void @llvm.aarch64.sve.prf.nxv4i1(<vscale x 4 x i1>,  i8*, i32)
+declare void @llvm.aarch64.sve.prf.nxv2i1(<vscale x 2 x i1>,  i8*, i32)
+declare void @llvm.aarch64.sve.prfb.gather.nxv4i1(<vscale x 4 x i1>, i8*, <vscale x 4 x i64>, i32)
+declare void @llvm.aarch64.sve.prfh.gather.nxv4i1(<vscale x 4 x i1>, i8*, <vscale x 4 x i64>, i32)
+declare void @llvm.aarch64.sve.prfw.gather.nxv4i1(<vscale x 4 x i1>, i8*, <vscale x 4 x i64>, i32)
+declare void @llvm.aarch64.sve.prfd.gather.nxv4i1(<vscale x 4 x i1>, i8*, <vscale x 4 x i64>, i32)
+declare void @llvm.aarch64.sve.prfb.gather.nxv2i1(<vscale x 2 x i1>, i8*, <vscale x 2 x i64>, i32)
+declare void @llvm.aarch64.sve.prfh.gather.nxv2i1(<vscale x 2 x i1>, i8*, <vscale x 2 x i64>, i32)
+declare void @llvm.aarch64.sve.prfw.gather.nxv2i1(<vscale x 2 x i1>, i8*, <vscale x 2 x i64>, i32)
+declare void @llvm.aarch64.sve.prfd.gather.nxv2i1(<vscale x 2 x i1>, i8*, <vscale x 2 x i64>, i32)