[AArch64][SVE][ISel] Combine dup of load to replicating load

[peterwaller-arm]

Summary

(dup (load) z_or_x_passthrough) => (replicating load)

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

2574–2575

I've pushed https://reviews.llvm.org/rGb4028fbc1a88 which fixes this issue.

[peterwaller-arm]

• Rebase after b4028fbc1a88

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

2574–2575

Thanks, much appreciated. Rebased.

[paulwalker-arm]

Some test recommendations but otherwise looks good.

llvm/test/CodeGen/AArch64/sve-ld1r.ll

843

To cover all the new variants you also want sign/zext forms for the integer types.

873

To cover all the new variants you also want unpacked forms for the floating-point types.

963–973

I don't think dup.x tests are necessary because we don't have any isel patterns for them. Instead we always lower such intrinsics to splat_vector, which is already tested?

That said, the dup.x tests are validating the immediate range so perhaps they can be converted to use dup into undef. That that said, I suppose there's an argument that given the immediate operand type is shared by the other pattern within the multiclass it's already being sufficiently tested. It's up to you depending on how paranoid you want to be.

[peterwaller-arm]

• Test zext/sext.
• Test unpacked forms (no negative tests, since dup with unpacked types is not specifically supported currently as there is currently no known way to get a DUP_MERGE_PASSTHRU from C/C++ code).
• Drop immediate tests since those are covered.

[peterwaller-arm]

• Remove 'q' accidentally copied into the test names.

[Allen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

Excuse me, I'm curious why the above changes affect the SideEffects attribute here. Would you give me some guidance? Thank you.

[dmgreen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

The way that hasSideEffects for an instruction works is that:

• It defaults to true.
• It gets set to false if there is a tablegen pattern that generated that instruction (I believe it might needs to be a single instruction generated by the pattern).
• It can be overridden with let hasSideEffects=1/0 on the instruction.

If you are looking to remove the side effects flags, and the instruction doesn't have a pattern, adding hasSideEffects=0 is usually a good way to go. Removing the side effects can help improve scheduling freedom and other code motion in the backend, so is usually good to do so long as the instruction doesn't do anything odd.

[Allen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

Thansk @dmgreen very much.

• As the above changes don't have some overridden with let hasSideEffects=1/0 , so I think the 2nd guidelines works in this case.
• Is the def : LD1RPat<nxv4i32, load, LD1RW_IMM, PTRUE_S, i32, am_indexed32_6b, uimm6s4>; before the change a tablegen pattern, and it also should set the instruction to false?

[dmgreen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

Yeah - IIRC, I think the logic in Tablegen might only apply when the pattern produces a single instruction. So the pattern with (load (ptrue 31), .. would not set hasSideEffects=0 in the same way as the new (load $pg, .. pattern.

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

Does hasSideEffects default to false? I investigated this after D140680 and spotted bit hasSideEffects = ?; with none of the base SVE instructions classes that sit above it setting it. I suspect there was some AArch64 upstream refactoring between our original downstream implementation and when it was upstreamed that we missed and so this property has been dandling ever since (either that or it's just always been wrong).

Eitherway, as agreed on D140680 I'll have a patch in the next few days that should resolve it. For this patch you can just accept the new value as has been the case of other patches where this flip has occurred.

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475–4490

Oops. Please ignore the final comment, I hadn't spotted this was not a new patch.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

(AArch64fadda_p_node (SVEAllActive), node:$op2,
              (vselect node:$op1, node:$op3, (splat_vector (f32 fpimm_minus0)))),
      (AArch64fadda_p_node (SVEAllActive), node:$op2,
              (vselect node:$op1, node:$op3, (splat_vector (f64 fpimm_minus0))))]>;
 
 def SDT_AArch64PTest : SDTypeProfile<0, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
 def AArch64ptest     : SDNode<"AArch64ISD::PTEST", SDT_AArch64PTest>;
 def AArch64ptest_any : SDNode<"AArch64ISD::PTEST_ANY", SDT_AArch64PTest>;
 
-def SDT_AArch64DUP_PRED  : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0, 3>, SDTCisVec<1>, SDTCVecEltisVT<1,i1>]>;
+def SDT_AArch64DUP_PRED  : SDTypeProfile<1, 3,
+  [SDTCisVec<0>, SDTCisSameAs<0, 3>, SDTCisVec<1>, SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0, 1>]>;
 def AArch64dup_mt : SDNode<"AArch64ISD::DUP_MERGE_PASSTHRU", SDT_AArch64DUP_PRED>;
 
 def AArch64splice : SDNode<"AArch64ISD::SPLICE", SDT_AArch64Arith>;
 
 def reinterpret_cast : SDNode<"AArch64ISD::REINTERPRET_CAST", SDTUnaryOp>;
 
 def AArch64mul_p_oneuse : PatFrag<(ops node:$pred, node:$src1, node:$src2),
                                   (AArch64mul_p node:$pred, node:$src1, node:$src2), [{
let Predicates = [HasSVEorSME] in {
   }
   let mayStore = 1, hasSideEffects = 0 in {
     def STR_ZZXI   : Pseudo<(outs), (ins   ZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
     def STR_ZZZXI  : Pseudo<(outs), (ins  ZZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
     def STR_ZZZZXI : Pseudo<(outs), (ins ZZZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
   }
 
   let AddedComplexity = 1 in {
-  class LD1RPat<ValueType vt, SDPatternOperator operator,
-                Instruction load, Instruction ptrue, ValueType index_vt, ComplexPattern CP, Operand immtype> :
-        Pat<(vt (splat_vector (index_vt (operator (CP GPR64:$base, immtype:$offset))))),
+    multiclass LD1RPat<ValueType vt, SDPatternOperator operator,
+                  Instruction load, Instruction ptrue, ValueType index_vt, ComplexPattern CP, Operand immtype> {
+      def : Pat<(vt (splat_vector (index_vt (operator (CP GPR64:$base, immtype:$offset))))),
                 (load (ptrue 31), GPR64:$base, $offset)>;
+      def : Pat<(vt (AArch64dup_mt PPR:$pg, (index_vt (operator (CP GPR64:$base, immtype:$offset))), (SVEDup0Undef))),
+                (load $pg, GPR64:$base, $offset)>;
+    }
   }
 
   // LDR1 of 8-bit data
-  def : LD1RPat<nxv16i8, extloadi8,  LD1RB_IMM,    PTRUE_B, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv8i16, zextloadi8, LD1RB_H_IMM,  PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv4i32, zextloadi8, LD1RB_S_IMM,  PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv2i64, zextloadi8, LD1RB_D_IMM,  PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv8i16, sextloadi8, LD1RSB_H_IMM, PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv4i32, sextloadi8, LD1RSB_S_IMM, PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv2i64, sextloadi8, LD1RSB_D_IMM, PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv16i8, extloadi8,  LD1RB_IMM,    PTRUE_B, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv8i16, zextloadi8, LD1RB_H_IMM,  PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv4i32, zextloadi8, LD1RB_S_IMM,  PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv2i64, zextloadi8, LD1RB_D_IMM,  PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv8i16, sextloadi8, LD1RSB_H_IMM, PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv4i32, sextloadi8, LD1RSB_S_IMM, PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv2i64, sextloadi8, LD1RSB_D_IMM, PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
 
   // LDR1 of 16-bit data
-  def : LD1RPat<nxv8i16, extloadi16,  LD1RH_IMM,    PTRUE_H, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4i32, zextloadi16, LD1RH_S_IMM,  PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2i64, zextloadi16, LD1RH_D_IMM,  PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4i32, sextloadi16, LD1RSH_S_IMM, PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2i64, sextloadi16, LD1RSH_D_IMM, PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv8i16, extloadi16,  LD1RH_IMM,    PTRUE_H, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4i32, zextloadi16, LD1RH_S_IMM,  PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2i64, zextloadi16, LD1RH_D_IMM,  PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4i32, sextloadi16, LD1RSH_S_IMM, PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2i64, sextloadi16, LD1RSH_D_IMM, PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
 
   // LDR1 of 32-bit data
-  def : LD1RPat<nxv4i32, load,        LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2i64, zextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2i64, sextloadi32, LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv4i32, load,        LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2i64, zextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2i64, sextloadi32, LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
 
   // LDR1 of 64-bit data
-  def : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
+  defm : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
 
   // LD1R of FP data
-  def : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
+  defm : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
 
 // LD1R of 128-bit masked data
   multiclass ld1rq_pat<ValueType vt1, SDPatternOperator op, Instruction load_instr, ComplexPattern AddrCP>{
     def : Pat<(vt1 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
           (!cast<Instruction>(load_instr # _IMM) $gp, $base, (i64 0))>;
     let AddedComplexity = 2 in {
       def : Pat<(vt1 (op PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
             (!cast<Instruction>(load_instr # _IMM) $gp, $base, simm4s16:$imm)>;

llvm/test/CodeGen/AArch64/sve-ld1r.ll

 ; CHECK-NEXT:    ld1rqb { z0.b }, p0/z, [x0]
 ; CHECK-NEXT:    ret
   %1 = load <16 x i8>, <16 x i8>* %a
   %2 = tail call <vscale x 16 x i8> @llvm.vector.insert.nxv16i8.v16i8(<vscale x 16 x i8> undef, <16 x i8> %1, i64 0)
   %3 = tail call <vscale x 16 x i8> @llvm.aarch64.sve.dupq.lane.nxv16i8(<vscale x 16 x i8> %2, i64 0)
   ret <vscale x 16 x i8> %3
 }
 
+;
+;
+; Tests for dup and dupx:
+;
+; Positive tests:
+; * dup with passthru=undef or passthrue=zero.
+; * dupx with an in-range immediate (hi, lo) for address offset.
+;
+; Negative tests:
+; * dup with passthru as a parameter.
+; * dupx with an out of range immediate for address offset
+;
+;
+
+
+define <vscale x 16 x i8> @dup_ld1rqb_i8_passthruundef(<vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rqb_i8_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dup_ld1rqh_i16_passthruundef(<vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rqh_i16_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dup_ld1rqs_i32_passthruundef(<vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rqs_i32_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dup_ld1rqd_i64_passthruundef(<vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: dup_ld1rqd_i64_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dup_ld1rqh_half_passthruundef(<vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rqh_half_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dup_ld1rqs_float_passthruundef(<vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: dup_ld1rqs_float_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dup_ld1rqd_double_passthruundef(<vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: dup_ld1rqd_double_passthruundef:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @dup_ld1rqb_i8_passthruzero(<vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rqb_i8_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> zeroinitializer, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dup_ld1rqh_i16_passthruzero(<vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rqh_i16_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> zeroinitializer, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dup_ld1rqs_i32_passthruzero(<vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rqs_i32_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> zeroinitializer, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dup_ld1rqd_i64_passthruzero(<vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: dup_ld1rqd_i64_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> zeroinitializer, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dup_ld1rqh_half_passthruzero(<vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rqh_half_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> zeroinitializer, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dup_ld1rqs_float_passthruzero(<vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: dup_ld1rqs_float_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> zeroinitializer, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dup_ld1rqd_double_passthruzero(<vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: dup_ld1rqd_double_passthruzero:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> zeroinitializer, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @dupx_ld1rqb_i8_immlo(i8* %addr) {
+; CHECK-LABEL: dupx_ld1rqb_i8_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0, #1]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i8, i8* %addr, i32 1
+    %ld = load i8, i8* %gep
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.x.nxv16i8(i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dupx_ld1rqh_i16_immlo(i16* %addr) {
+; CHECK-LABEL: dupx_ld1rqh_i16_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #2]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i16, i16* %addr, i32 1
+    %ld = load i16, i16* %gep
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.x.nxv8i16(i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dupx_ld1rqs_i32_immlo(i32* %addr) {
+; CHECK-LABEL: dupx_ld1rqs_i32_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #4]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i32, i32* %addr, i32 1
+    %ld = load i32, i32* %gep
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.x.nxv4i32(i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dupx_ld1rqd_i64_immlo(i64* %addr) {
+; CHECK-LABEL: dupx_ld1rqd_i64_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i64, i64* %addr, i32 1
+    %ld = load i64, i64* %gep
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.x.nxv2i64(i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dupx_ld1rqh_half_immlo(half* %addr) {
+; CHECK-LABEL: dupx_ld1rqh_half_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #2]
+; CHECK-NEXT:    ret
+    %gep = getelementptr half, half* %addr, i32 1
+    %ld = load half, half* %gep
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.x.nxv8f16(half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dupx_ld1rqs_float_immlo(float* %addr) {
+; CHECK-LABEL: dupx_ld1rqs_float_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #4]
+; CHECK-NEXT:    ret
+    %gep = getelementptr float, float* %addr, i32 1
+    %ld = load float, float* %gep
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.x.nxv4f32(float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dupx_ld1rqd_double_immlo(double* %addr) {
+; CHECK-LABEL: dupx_ld1rqd_double_immlo:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr double, double* %addr, i32 1
+    %ld = load double, double* %gep
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @dupx_ld1rqb_i8_immhi(i8* %addr) {
+; CHECK-LABEL: dupx_ld1rqb_i8_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0, #63]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i8, i8* %addr, i32 63
+    %ld = load i8, i8* %gep
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.x.nxv16i8(i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dupx_ld1rqh_i16_immhi(i16* %addr) {
+; CHECK-LABEL: dupx_ld1rqh_i16_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i16, i16* %addr, i32 63
+    %ld = load i16, i16* %gep
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.x.nxv8i16(i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dupx_ld1rqs_i32_immhi(i32* %addr) {
+; CHECK-LABEL: dupx_ld1rqs_i32_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i32, i32* %addr, i32 63
+    %ld = load i32, i32* %gep
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.x.nxv4i32(i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dupx_ld1rqd_i64_immhi(i64* %addr) {
+; CHECK-LABEL: dupx_ld1rqd_i64_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i64, i64* %addr, i32 63
+    %ld = load i64, i64* %gep
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.x.nxv2i64(i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dupx_ld1rqh_half_immhi(half* %addr) {
+; CHECK-LABEL: dupx_ld1rqh_half_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0, #126]
+; CHECK-NEXT:    ret
+    %gep = getelementptr half, half* %addr, i32 63
+    %ld = load half, half* %gep
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.x.nxv8f16(half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dupx_ld1rqs_float_immhi(float* %addr) {
+; CHECK-LABEL: dupx_ld1rqs_float_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0, #252]
+; CHECK-NEXT:    ret
+    %gep = getelementptr float, float* %addr, i32 63
+    %ld = load float, float* %gep
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.x.nxv4f32(float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dupx_ld1rqd_double_immhi(double* %addr) {
+; CHECK-LABEL: dupx_ld1rqd_double_immhi:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0, #504]
+; CHECK-NEXT:    ret
+    %gep = getelementptr double, double* %addr, i32 63
+    %ld = load double, double* %gep
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @negtest_dup_ld1rqb_i8_passthru(<vscale x 16 x i8> %pt, <vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqb_i8_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrb w8, [x0]
+; CHECK-NEXT:    mov z0.b, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> %pt, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @negtest_dup_ld1rqh_i16_passthru(<vscale x 8 x i16> %pt, <vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqh_i16_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrh w8, [x0]
+; CHECK-NEXT:    mov z0.h, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> %pt, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @negtest_dup_ld1rqs_i32_passthru(<vscale x 4 x i32> %pt, <vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqs_i32_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr w8, [x0]
+; CHECK-NEXT:    mov z0.s, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> %pt, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @negtest_dup_ld1rqd_i64_passthru(<vscale x 2 x i64> %pt, <vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqd_i64_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr x8, [x0]
+; CHECK-NEXT:    mov z0.d, p0/m, x8
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> %pt, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @negtest_dup_ld1rqh_half_passthru(<vscale x 8 x half> %pt, <vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqh_half_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h1, [x0]
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> %pt, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @negtest_dup_ld1rqs_float_passthru(<vscale x 4 x float> %pt, <vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqs_float_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s1, [x0]
+; CHECK-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> %pt, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @negtest_dup_ld1rqd_double_passthru(<vscale x 2 x double> %pt, <vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rqd_double_passthru:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d1, [x0]
+; CHECK-NEXT:    mov z0.d, p0/m, d1
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> %pt, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @negtest_dupx_ld1rqb_i8_immlo_outrange(i8* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqb_i8_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #1
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i8, i8* %addr, i32 -1
+    %ld = load i8, i8* %gep
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.x.nxv16i8(i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @negtest_dupx_ld1rqh_i16_immlo_outrange(i16* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqh_i16_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #2
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i16, i16* %addr, i32 -1
+    %ld = load i16, i16* %gep
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.x.nxv8i16(i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @negtest_dupx_ld1rqs_i32_immlo_outrange(i32* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqs_i32_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #4
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i32, i32* %addr, i32 -1
+    %ld = load i32, i32* %gep
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.x.nxv4i32(i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @negtest_dupx_ld1rqd_i64_immlo_outrange(i64* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqd_i64_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #8
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i64, i64* %addr, i32 -1
+    %ld = load i64, i64* %gep
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.x.nxv2i64(i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @negtest_dupx_ld1rqh_half_immlo_outrange(half* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqh_half_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #2
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr half, half* %addr, i32 -1
+    %ld = load half, half* %gep
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.x.nxv8f16(half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @negtest_dupx_ld1rqs_float_immlo_outrange(float* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqs_float_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #4
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr float, float* %addr, i32 -1
+    %ld = load float, float* %gep
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.x.nxv4f32(float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @negtest_dupx_ld1rqd_double_immlo_outrange(double* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqd_double_immlo_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    sub x8, x0, #8
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr double, double* %addr, i32 -1
+    %ld = load double, double* %gep
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 16 x i8> @negtest_dupx_ld1rqb_i8_immhi_outrange(i8* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqb_i8_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #64
+; CHECK-NEXT:    ptrue p0.b
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i8, i8* %addr, i32 64
+    %ld = load i8, i8* %gep
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.x.nxv16i8(i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+
+define <vscale x 8 x i16> @negtest_dupx_ld1rqh_i16_immhi_outrange(i16* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqh_i16_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #128
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i16, i16* %addr, i32 64
+    %ld = load i16, i16* %gep
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.x.nxv8i16(i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+
+define <vscale x 4 x i32> @negtest_dupx_ld1rqs_i32_immhi_outrange(i32* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqs_i32_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #256
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i32, i32* %addr, i32 64
+    %ld = load i32, i32* %gep
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.x.nxv4i32(i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+
+define <vscale x 2 x i64> @negtest_dupx_ld1rqd_i64_immhi_outrange(i64* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqd_i64_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #512
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr i64, i64* %addr, i32 64
+    %ld = load i64, i64* %gep
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.x.nxv2i64(i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+
+define <vscale x 8 x half> @negtest_dupx_ld1rqh_half_immhi_outrange(half* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqh_half_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #128
+; CHECK-NEXT:    ptrue p0.h
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr half, half* %addr, i32 64
+    %ld = load half, half* %gep
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.x.nxv8f16(half %ld)
+    ret <vscale x 8 x half> %res
+}
+
+define <vscale x 4 x float> @negtest_dupx_ld1rqs_float_immhi_outrange(float* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqs_float_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #256
+; CHECK-NEXT:    ptrue p0.s
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr float, float* %addr, i32 64
+    %ld = load float, float* %gep
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.x.nxv4f32(float %ld)
+    ret <vscale x 4 x float> %res
+}
+
+define <vscale x 2 x double> @negtest_dupx_ld1rqd_double_immhi_outrange(double* %addr) {
+; CHECK-LABEL: negtest_dupx_ld1rqd_double_immhi_outrange:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    add x8, x0, #512
+; CHECK-NEXT:    ptrue p0.d
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x8]
+; CHECK-NEXT:    ret
+    %gep = getelementptr double, double* %addr, i32 64
+    %ld = load double, double* %gep
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double %ld)
+    ret <vscale x 2 x double> %res
+}
+
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.dupq.lane.nxv16i8(<vscale x 16 x i8>, i64)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.dupq.lane.nxv8i16(<vscale x 8 x i16>, i64)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.dupq.lane.nxv4i32(<vscale x 4 x i32>, i64)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.dupq.lane.nxv2i64(<vscale x 2 x i64>, i64)
 declare <vscale x 8 x half> @llvm.aarch64.sve.dupq.lane.nxv8f16(<vscale x 8 x half>, i64)
 declare <vscale x 8 x bfloat> @llvm.aarch64.sve.dupq.lane.nxv8bf16(<vscale x 8 x bfloat>, i64)
 declare <vscale x 4 x float> @llvm.aarch64.sve.dupq.lane.nxv4f32(<vscale x 4 x float>, i64)
 declare <vscale x 2 x double> @llvm.aarch64.sve.dupq.lane.nxv2f64(<vscale x 2 x double>, i64)
 
 declare <vscale x 2 x double> @llvm.vector.insert.nxv2f64.v2f64(<vscale x 2 x double>, <2 x double>, i64)
 declare <vscale x 4 x float> @llvm.vector.insert.nxv4f32.v4f32(<vscale x 4 x float>, <4 x float>, i64)
 declare <vscale x 8 x half> @llvm.vector.insert.nxv8f16.v8f16(<vscale x 8 x half>, <8 x half>, i64)
 declare <vscale x 2 x i64> @llvm.vector.insert.nxv2i64.v2i64(<vscale x 2 x i64>, <2 x i64>, i64)
 declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v4i32(<vscale x 4 x i32>, <4 x i32>, i64)
 declare <vscale x 8 x i16> @llvm.vector.insert.nxv8i16.v8i16(<vscale x 8 x i16>, <8 x i16>, i64)
 declare <vscale x 16 x i8> @llvm.vector.insert.nxv16i8.v16i8(<vscale x 16 x i8>, <16 x i8>, i64)
 declare <vscale x 8 x bfloat> @llvm.vector.insert.nxv8bf16.v8bf16(<vscale x 8 x bfloat>, <8 x bfloat>, i64)
 
+declare <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i1>, i8)
+declare <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i1>, i16)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i1>, i64)
+declare <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half>, <vscale x 8 x i1>, half)
+declare <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float>, <vscale x 4 x i1>, float)
+declare <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double>, <vscale x 2 x i1>, double)
+
+declare <vscale x 16 x i8> @llvm.aarch64.sve.dup.x.nxv16i8(i8)
+declare <vscale x 8 x i16> @llvm.aarch64.sve.dup.x.nxv8i16(i16)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.dup.x.nxv4i32(i32)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.dup.x.nxv2i64(i64)
+declare <vscale x 8 x half> @llvm.aarch64.sve.dup.x.nxv8f16(half)
+declare <vscale x 4 x float> @llvm.aarch64.sve.dup.x.nxv4f32(float)
+declare <vscale x 2 x double> @llvm.aarch64.sve.dup.x.nxv2f64(double)
+
+
 attributes #0 = { "target-features"="+sve,+bf16" }

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

 # CHECK-NEXT:  1      1     0.50                        addvl x0, x0, #-32
 # CHECK-NEXT:  1      1     0.50                        addvl x21, x21, #0
 # CHECK-NEXT:  1      1     0.50                        addvl x23, x8, #-1
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #1]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #2]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #1]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #2]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #1]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #2]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #1]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #2]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #3]

 # CHECK-NEXT:  1      16    2.00    *             U     ld1h  { z31.d }, p7/z, [z31.d, #62]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1h  { z31.h }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1h  { z31.s }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, sxtw #1]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, uxtw #1]
 # CHECK-NEXT:  1      19    4.00    *             U     ld1h  { z31.s }, p7/z, [z31.s, #62]
 # CHECK-NEXT:  1      11    0.50    *                   ld1h  { z5.h }, p3/z, [sp, x16, lsl #1]
 # CHECK-NEXT:  1      11    0.50    *                   ld1h  { z5.h }, p3/z, [x17, x16, lsl #1]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.b }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.b }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rd { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rd { z31.d }, p7/z, [sp, #504]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.h }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.b }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.b }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rd { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rd { z31.d }, p7/z, [sp, #504]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.h }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.s }, p7/z, [sp, #126]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z0.b }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z0.b }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z21.b }, p5/z, [x10, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z23.b }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z31.b }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z0.d }, p0/z, [x0, x0, lsl #3]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z23.d }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z23.d }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z31.d }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z0.h }, p0/z, [x0, x0, lsl #1]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z23.h }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z23.h }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z31.h }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z0.s }, p0/z, [x0, x0, lsl #2]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z23.s }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z23.s }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z31.s }, p7/z, [sp, #-16]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z31.s }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsw  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsw  { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z31.s }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsw  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsw  { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z31.s }, p7/z, [sp, #252]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      16    2.00    *             U     ld1sb { z0.d }, p0/z, [z0.d]
 # CHECK-NEXT:  1      11    0.50    *                   ld1sb { z0.h }, p0/z, [sp, x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1sb { z0.h }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1sb { z0.s }, p0/z, [x0, z0.s, sxtw]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      19    4.00    *             U     ld1sb { z0.s }, p0/z, [z0.s]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

 # CHECK-NEXT:  4      9     1.00    *             U     ld1h  { z31.d }, p7/z, [z31.d, #62]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1h  { z31.h }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1h  { z31.s }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  4      10    1.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, sxtw #1]
 # CHECK-NEXT:  4      10    1.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, uxtw #1]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1h  { z31.s }, p7/z, [z31.s, #62]
 # CHECK-NEXT:  1      6     0.50    *                   ld1h  { z5.h }, p3/z, [sp, x16, lsl #1]
 # CHECK-NEXT:  1      6     0.50    *                   ld1h  { z5.h }, p3/z, [x17, x16, lsl #1]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.b }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.b }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rd { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rd { z31.d }, p7/z, [sp, #504]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.h }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.b }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.b }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rd { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rd { z31.d }, p7/z, [sp, #504]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.h }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.s }, p7/z, [sp, #126]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z0.b }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z0.b }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z21.b }, p5/z, [x10, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z23.b }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z31.b }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z0.d }, p0/z, [x0, x0, lsl #3]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z23.d }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z23.d }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z31.d }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z0.h }, p0/z, [x0, x0, lsl #1]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z23.h }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z23.h }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z31.h }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z0.s }, p0/z, [x0, x0, lsl #2]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z23.s }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z23.s }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z31.s }, p7/z, [sp, #-16]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z31.s }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsw  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsw  { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z31.s }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsw  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsw  { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z31.s }, p7/z, [sp, #252]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  4      9     1.00    *             U     ld1sb { z0.d }, p0/z, [z0.d]
 # CHECK-NEXT:  1      6     0.50    *                   ld1sb { z0.h }, p0/z, [sp, x0]
 # CHECK-NEXT:  1      6     0.50    *                   ld1sb { z0.h }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1sb { z0.s }, p0/z, [x0, z0.s, sxtw]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1sb { z0.s }, p0/z, [z0.s]