[AArch64][SVE][ISel] Combine dup of load to replicating load

[peterwaller-arm]

Summary

(dup (load) z_or_x_passthrough) => (replicating load)

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

2574–2575

I've pushed https://reviews.llvm.org/rGb4028fbc1a88 which fixes this issue.

[peterwaller-arm]

• Rebase after b4028fbc1a88

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

2574–2575

Thanks, much appreciated. Rebased.

[paulwalker-arm]

Some test recommendations but otherwise looks good.

llvm/test/CodeGen/AArch64/sve-ld1r.ll

843

To cover all the new variants you also want sign/zext forms for the integer types.

873

To cover all the new variants you also want unpacked forms for the floating-point types.

963–973

I don't think dup.x tests are necessary because we don't have any isel patterns for them. Instead we always lower such intrinsics to splat_vector, which is already tested?

That said, the dup.x tests are validating the immediate range so perhaps they can be converted to use dup into undef. That that said, I suppose there's an argument that given the immediate operand type is shared by the other pattern within the multiclass it's already being sufficiently tested. It's up to you depending on how paranoid you want to be.

[peterwaller-arm]

• Test zext/sext.
• Test unpacked forms (no negative tests, since dup with unpacked types is not specifically supported currently as there is currently no known way to get a DUP_MERGE_PASSTHRU from C/C++ code).
• Drop immediate tests since those are covered.

[peterwaller-arm]

• Remove 'q' accidentally copied into the test names.

[Allen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

Excuse me, I'm curious why the above changes affect the SideEffects attribute here. Would you give me some guidance? Thank you.

[dmgreen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

The way that hasSideEffects for an instruction works is that:

• It defaults to true.
• It gets set to false if there is a tablegen pattern that generated that instruction (I believe it might needs to be a single instruction generated by the pattern).
• It can be overridden with let hasSideEffects=1/0 on the instruction.

If you are looking to remove the side effects flags, and the instruction doesn't have a pattern, adding hasSideEffects=0 is usually a good way to go. Removing the side effects can help improve scheduling freedom and other code motion in the backend, so is usually good to do so long as the instruction doesn't do anything odd.

[Allen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

Thansk @dmgreen very much.

• As the above changes don't have some overridden with let hasSideEffects=1/0 , so I think the 2nd guidelines works in this case.
• Is the def : LD1RPat<nxv4i32, load, LD1RW_IMM, PTRUE_S, i32, am_indexed32_6b, uimm6s4>; before the change a tablegen pattern, and it also should set the instruction to false?

[dmgreen]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

Yeah - IIRC, I think the logic in Tablegen might only apply when the pattern produces a single instruction. So the pattern with (load (ptrue 31), .. would not set hasSideEffects=0 in the same way as the new (load $pg, .. pattern.

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

Does hasSideEffects default to false? I investigated this after D140680 and spotted bit hasSideEffects = ?; with none of the base SVE instructions classes that sit above it setting it. I suspect there was some AArch64 upstream refactoring between our original downstream implementation and when it was upstreamed that we missed and so this property has been dandling ever since (either that or it's just always been wrong).

Eitherway, as agreed on D140680 I'll have a patch in the next few days that should resolve it. For this patch you can just accept the new value as has been the case of other patches where this flip has occurred.

[paulwalker-arm]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

4475

Oops. Please ignore the final comment, I hadn't spotted this was not a new patch.

llvm/lib/Target/AArch64/AArch64SVEInstrInfo.td

(AArch64fadda_p_node (SVEAllActive), node:$op2,
              (vselect node:$op1, node:$op3, (splat_vector (f32 fpimm_minus0)))),
      (AArch64fadda_p_node (SVEAllActive), node:$op2,
              (vselect node:$op1, node:$op3, (splat_vector (f64 fpimm_minus0))))]>;
 
 def SDT_AArch64PTest : SDTypeProfile<0, 2, [SDTCisVec<0>, SDTCisSameAs<0,1>]>;
 def AArch64ptest     : SDNode<"AArch64ISD::PTEST", SDT_AArch64PTest>;
 def AArch64ptest_any : SDNode<"AArch64ISD::PTEST_ANY", SDT_AArch64PTest>;
 
-def SDT_AArch64DUP_PRED  : SDTypeProfile<1, 3, [SDTCisVec<0>, SDTCisSameAs<0, 3>, SDTCisVec<1>, SDTCVecEltisVT<1,i1>]>;
+def SDT_AArch64DUP_PRED  : SDTypeProfile<1, 3,
+  [SDTCisVec<0>, SDTCisSameAs<0, 3>, SDTCisVec<1>, SDTCVecEltisVT<1,i1>, SDTCisSameNumEltsAs<0, 1>]>;
 def AArch64dup_mt : SDNode<"AArch64ISD::DUP_MERGE_PASSTHRU", SDT_AArch64DUP_PRED>;
 
 def AArch64splice : SDNode<"AArch64ISD::SPLICE", SDT_AArch64Arith>;
 
 def reinterpret_cast : SDNode<"AArch64ISD::REINTERPRET_CAST", SDTUnaryOp>;
 
 def AArch64mul_p_oneuse : PatFrag<(ops node:$pred, node:$src1, node:$src2),
                                   (AArch64mul_p node:$pred, node:$src1, node:$src2), [{
let Predicates = [HasSVEorSME] in {
   }
   let mayStore = 1, hasSideEffects = 0 in {
     def STR_ZZXI   : Pseudo<(outs), (ins   ZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
     def STR_ZZZXI  : Pseudo<(outs), (ins  ZZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
     def STR_ZZZZXI : Pseudo<(outs), (ins ZZZZ_b:$Zs, GPR64sp:$sp, simm4s1:$offset),[]>, Sched<[]>;
   }
 
   let AddedComplexity = 1 in {
-  class LD1RPat<ValueType vt, SDPatternOperator operator,
-                Instruction load, Instruction ptrue, ValueType index_vt, ComplexPattern CP, Operand immtype> :
-        Pat<(vt (splat_vector (index_vt (operator (CP GPR64:$base, immtype:$offset))))),
+    multiclass LD1RPat<ValueType vt, SDPatternOperator operator,
+                  Instruction load, Instruction ptrue, ValueType index_vt, ComplexPattern CP, Operand immtype> {
+      def : Pat<(vt (splat_vector (index_vt (operator (CP GPR64:$base, immtype:$offset))))),
                 (load (ptrue 31), GPR64:$base, $offset)>;
+      def : Pat<(vt (AArch64dup_mt PPR:$pg, (index_vt (operator (CP GPR64:$base, immtype:$offset))), (SVEDup0Undef))),
+                (load $pg, GPR64:$base, $offset)>;
+    }
   }
 
   // LDR1 of 8-bit data
-  def : LD1RPat<nxv16i8, extloadi8,  LD1RB_IMM,    PTRUE_B, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv8i16, zextloadi8, LD1RB_H_IMM,  PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv4i32, zextloadi8, LD1RB_S_IMM,  PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv2i64, zextloadi8, LD1RB_D_IMM,  PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv8i16, sextloadi8, LD1RSB_H_IMM, PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv4i32, sextloadi8, LD1RSB_S_IMM, PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
-  def : LD1RPat<nxv2i64, sextloadi8, LD1RSB_D_IMM, PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv16i8, extloadi8,  LD1RB_IMM,    PTRUE_B, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv8i16, zextloadi8, LD1RB_H_IMM,  PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv4i32, zextloadi8, LD1RB_S_IMM,  PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv2i64, zextloadi8, LD1RB_D_IMM,  PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv8i16, sextloadi8, LD1RSB_H_IMM, PTRUE_H, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv4i32, sextloadi8, LD1RSB_S_IMM, PTRUE_S, i32, am_indexed8_6b, uimm6s1>;
+  defm : LD1RPat<nxv2i64, sextloadi8, LD1RSB_D_IMM, PTRUE_D, i64, am_indexed8_6b, uimm6s1>;
 
   // LDR1 of 16-bit data
-  def : LD1RPat<nxv8i16, extloadi16,  LD1RH_IMM,    PTRUE_H, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4i32, zextloadi16, LD1RH_S_IMM,  PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2i64, zextloadi16, LD1RH_D_IMM,  PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4i32, sextloadi16, LD1RSH_S_IMM, PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2i64, sextloadi16, LD1RSH_D_IMM, PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv8i16, extloadi16,  LD1RH_IMM,    PTRUE_H, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4i32, zextloadi16, LD1RH_S_IMM,  PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2i64, zextloadi16, LD1RH_D_IMM,  PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4i32, sextloadi16, LD1RSH_S_IMM, PTRUE_S, i32, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2i64, sextloadi16, LD1RSH_D_IMM, PTRUE_D, i64, am_indexed16_6b, uimm6s2>;
 
   // LDR1 of 32-bit data
-  def : LD1RPat<nxv4i32, load,        LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2i64, zextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2i64, sextloadi32, LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv4i32, load,        LD1RW_IMM,   PTRUE_S, i32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2i64, zextloadi32, LD1RW_D_IMM, PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2i64, sextloadi32, LD1RSW_IMM,  PTRUE_D, i64, am_indexed32_6b, uimm6s4>;
 
   // LDR1 of 64-bit data
-  def : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
+  defm : LD1RPat<nxv2i64, load, LD1RD_IMM, PTRUE_D, i64, am_indexed64_6b, uimm6s8>;
 
   // LD1R of FP data
-  def : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
-  def : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
-  def : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
+  defm : LD1RPat<nxv8f16, load, LD1RH_IMM,   PTRUE_H, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4f16, load, LD1RH_S_IMM, PTRUE_S, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv2f16, load, LD1RH_D_IMM, PTRUE_D, f16, am_indexed16_6b, uimm6s2>;
+  defm : LD1RPat<nxv4f32, load, LD1RW_IMM,   PTRUE_S, f32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2f32, load, LD1RW_D_IMM, PTRUE_D, f32, am_indexed32_6b, uimm6s4>;
+  defm : LD1RPat<nxv2f64, load, LD1RD_IMM,   PTRUE_D, f64, am_indexed64_6b, uimm6s8>;
 
 // LD1R of 128-bit masked data
   multiclass ld1rq_pat<ValueType vt1, SDPatternOperator op, Instruction load_instr, ComplexPattern AddrCP>{
     def : Pat<(vt1 (AArch64ld1rq_z PPR:$gp, GPR64:$base)),
           (!cast<Instruction>(load_instr # _IMM) $gp, $base, (i64 0))>;
     let AddedComplexity = 2 in {
       def : Pat<(vt1 (op PPR:$gp, (add GPR64:$base, (i64 simm4s16:$imm)))),
             (!cast<Instruction>(load_instr # _IMM) $gp, $base, simm4s16:$imm)>;

llvm/test/CodeGen/AArch64/sve-ld1r.ll

 ; CHECK-NEXT:    ld1rqb { z0.b }, p0/z, [x0]
 ; CHECK-NEXT:    ret
   %1 = load <16 x i8>, <16 x i8>* %a
   %2 = tail call <vscale x 16 x i8> @llvm.vector.insert.nxv16i8.v16i8(<vscale x 16 x i8> undef, <16 x i8> %1, i64 0)
   %3 = tail call <vscale x 16 x i8> @llvm.aarch64.sve.dupq.lane.nxv16i8(<vscale x 16 x i8> %2, i64 0)
   ret <vscale x 16 x i8> %3
 }
 
+;
+;
+; Tests for dup:
+;
+; Positive tests:
+; * dup with passthru=undef or passthrue=zero.
+; * sign/zero extending.
+; * unpacked types.
+;
+; Negative tests:
+; * dup with passthru as a parameter.
+;
+;
+
+define <vscale x 16 x i8> @dup_ld1rb_i8_passthruundef_nxv16i8(<vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rb_i8_passthruundef_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> undef, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dup_ld1rh_i16_passthruundef_nxv8i16(<vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rh_i16_passthruundef_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 8 x i16> @dup_ld1rh_i8_passthruundef_nxv8i16_sext(<vscale x 8 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rh_i8_passthruundef_nxv8i16_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsb { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = sext i8 %ld to i16
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i1> %pg, i16 %ext)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 8 x i16> @dup_ld1rh_i8_passthruundef_nxv8i16_zext(<vscale x 8 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rh_i8_passthruundef_nxv8i16_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = zext i8 %ld to i16
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> undef, <vscale x 8 x i1> %pg, i16 %ext)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i32_passthruundef_nxv4i32(<vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rs_i32_passthruundef_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i8_passthruundef_nxv4i32_sext(<vscale x 4 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rs_i8_passthruundef_nxv4i32_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsb { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = sext i8 %ld to i32
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ext)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i8_passthruundef_nxv4i32_zext(<vscale x 4 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rs_i8_passthruundef_nxv4i32_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = zext i8 %ld to i32
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ext)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i16_passthruundef_nxv4i32_sext(<vscale x 4 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rs_i16_passthruundef_nxv4i32_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %ext = sext i16 %ld to i32
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ext)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i16_passthruundef_nxv4i32_zext(<vscale x 4 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rs_i16_passthruundef_nxv4i32_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %ext = zext i16 %ld to i32
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> undef, <vscale x 4 x i1> %pg, i32 %ext)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dup_ld1rd_i64_passthruundef_nxv2i64(<vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: dup_ld1rd_i64_passthruundef_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i8_passthruundef_nxv2i64_sext(<vscale x 2 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rs_i8_passthruundef_nxv2i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsb { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = sext i8 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i8_passthruundef_nxv2i64_zext(<vscale x 2 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rs_i8_passthruundef_nxv2i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %ext = zext i8 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i16_passthruundef_nxv2i64_sext(<vscale x 2 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rs_i16_passthruundef_nxv2i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %ext = sext i16 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i16_passthruundef_nxv2i64_zext(<vscale x 2 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rs_i16_passthruundef_nxv2i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %ext = zext i16 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i32_passthruundef_nxv2i64_sext(<vscale x 2 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rs_i32_passthruundef_nxv2i64_sext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rsw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %ext = sext i32 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 2 x i64> @dup_ld1rs_i32_passthruundef_nxv2i64_zext(<vscale x 2 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rs_i32_passthruundef_nxv2i64_zext:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %ext = zext i32 %ld to i64
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> undef, <vscale x 2 x i1> %pg, i64 %ext)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dup_ld1rh_half_passthruundef_nxv8f16(<vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rh_half_passthruundef_nxv8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> undef, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dup_ld1rs_float_passthruundef_nxv4f32(<vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: dup_ld1rs_float_passthruundef_nxv4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> undef, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dup_ld1rd_double_passthruundef_nxv2f64(<vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: dup_ld1rd_double_passthruundef_nxv2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> undef, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 4 x half> @dup_ld1rh_half_passthruundef_nxv4f16(<vscale x 4 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rh_half_passthruundef_nxv4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half> undef, <vscale x 4 x i1> %pg, half %ld)
+    ret <vscale x 4 x half> %res
+}
+define <vscale x 16 x i8> @dup_ld1rb_i8_passthruzero_nxv16i8(<vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: dup_ld1rb_i8_passthruzero_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rb { z0.b }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> zeroinitializer, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @dup_ld1rh_i16_passthruzero_nxv8i16(<vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: dup_ld1rh_i16_passthruzero_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> zeroinitializer, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @dup_ld1rs_i32_passthruzero_nxv4i32(<vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: dup_ld1rs_i32_passthruzero_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> zeroinitializer, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @dup_ld1rd_i64_passthruzero_nxv2i64(<vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: dup_ld1rd_i64_passthruzero_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> zeroinitializer, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @dup_ld1rh_half_passthruzero_nxv8f16(<vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.h }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> zeroinitializer, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @dup_ld1rs_float_passthruzero_nxv4f32(<vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: dup_ld1rs_float_passthruzero_nxv4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> zeroinitializer, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @dup_ld1rd_double_passthruzero_nxv2f64(<vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: dup_ld1rd_double_passthruzero_nxv2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rd { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> zeroinitializer, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+define <vscale x 4 x half> @dup_ld1rh_half_passthruzero_nxv4f16(<vscale x 4 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv4f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.s }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half> zeroinitializer, <vscale x 4 x i1> %pg, half %ld)
+    ret <vscale x 4 x half> %res
+}
+define <vscale x 2 x half> @dup_ld1rh_half_passthruzero_nxv2f16(<vscale x 2 x i1> %pg, half* %addr) {
+; CHECK-LABEL: dup_ld1rh_half_passthruzero_nxv2f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rh { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 2 x half> @llvm.aarch64.sve.dup.nxv2f16(<vscale x 2 x half> zeroinitializer, <vscale x 2 x i1> %pg, half %ld)
+    ret <vscale x 2 x half> %res
+}
+define <vscale x 2 x float> @dup_ld1rs_float_passthruzero_nxv2f32(<vscale x 2 x i1> %pg, float* %addr) {
+; CHECK-LABEL: dup_ld1rs_float_passthruzero_nxv2f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ld1rw { z0.d }, p0/z, [x0]
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 2 x float> @llvm.aarch64.sve.dup.nxv2f32(<vscale x 2 x float> zeroinitializer, <vscale x 2 x i1> %pg, float %ld)
+    ret <vscale x 2 x float> %res
+}
+define <vscale x 16 x i8> @negtest_dup_ld1rb_i8_passthru_nxv16i8(<vscale x 16 x i8> %pt, <vscale x 16 x i1> %pg, i8* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rb_i8_passthru_nxv16i8:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrb w8, [x0]
+; CHECK-NEXT:    mov z0.b, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i8, i8* %addr
+    %res = call <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8> %pt, <vscale x 16 x i1> %pg, i8 %ld)
+    ret <vscale x 16 x i8> %res
+}
+define <vscale x 8 x i16> @negtest_dup_ld1rh_i16_passthru_nxv8i16(<vscale x 8 x i16> %pt, <vscale x 8 x i1> %pg, i16* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rh_i16_passthru_nxv8i16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldrh w8, [x0]
+; CHECK-NEXT:    mov z0.h, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i16, i16* %addr
+    %res = call <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16> %pt, <vscale x 8 x i1> %pg, i16 %ld)
+    ret <vscale x 8 x i16> %res
+}
+define <vscale x 4 x i32> @negtest_dup_ld1rs_i32_passthru_nxv4i32(<vscale x 4 x i32> %pt, <vscale x 4 x i1> %pg, i32* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rs_i32_passthru_nxv4i32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr w8, [x0]
+; CHECK-NEXT:    mov z0.s, p0/m, w8
+; CHECK-NEXT:    ret
+    %ld = load i32, i32* %addr
+    %res = call <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32> %pt, <vscale x 4 x i1> %pg, i32 %ld)
+    ret <vscale x 4 x i32> %res
+}
+define <vscale x 2 x i64> @negtest_dup_ld1rd_i64_passthru_nxv2i64(<vscale x 2 x i64> %pt, <vscale x 2 x i1> %pg, i64* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rd_i64_passthru_nxv2i64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr x8, [x0]
+; CHECK-NEXT:    mov z0.d, p0/m, x8
+; CHECK-NEXT:    ret
+    %ld = load i64, i64* %addr
+    %res = call <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64> %pt, <vscale x 2 x i1> %pg, i64 %ld)
+    ret <vscale x 2 x i64> %res
+}
+define <vscale x 8 x half> @negtest_dup_ld1rh_half_passthru_nxv8f16(<vscale x 8 x half> %pt, <vscale x 8 x i1> %pg, half* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rh_half_passthru_nxv8f16:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr h1, [x0]
+; CHECK-NEXT:    mov z0.h, p0/m, h1
+; CHECK-NEXT:    ret
+    %ld = load half, half* %addr
+    %res = call <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half> %pt, <vscale x 8 x i1> %pg, half %ld)
+    ret <vscale x 8 x half> %res
+}
+define <vscale x 4 x float> @negtest_dup_ld1rs_float_passthru_nxv4f32(<vscale x 4 x float> %pt, <vscale x 4 x i1> %pg, float* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rs_float_passthru_nxv4f32:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr s1, [x0]
+; CHECK-NEXT:    mov z0.s, p0/m, s1
+; CHECK-NEXT:    ret
+    %ld = load float, float* %addr
+    %res = call <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float> %pt, <vscale x 4 x i1> %pg, float %ld)
+    ret <vscale x 4 x float> %res
+}
+define <vscale x 2 x double> @negtest_dup_ld1rd_double_passthru_nxv2f64(<vscale x 2 x double> %pt, <vscale x 2 x i1> %pg, double* %addr) {
+; CHECK-LABEL: negtest_dup_ld1rd_double_passthru_nxv2f64:
+; CHECK:       // %bb.0:
+; CHECK-NEXT:    ldr d1, [x0]
+; CHECK-NEXT:    mov z0.d, p0/m, d1
+; CHECK-NEXT:    ret
+    %ld = load double, double* %addr
+    %res = call <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double> %pt, <vscale x 2 x i1> %pg, double %ld)
+    ret <vscale x 2 x double> %res
+}
+
 declare <vscale x 16 x i8> @llvm.aarch64.sve.dupq.lane.nxv16i8(<vscale x 16 x i8>, i64)
 declare <vscale x 8 x i16> @llvm.aarch64.sve.dupq.lane.nxv8i16(<vscale x 8 x i16>, i64)
 declare <vscale x 4 x i32> @llvm.aarch64.sve.dupq.lane.nxv4i32(<vscale x 4 x i32>, i64)
 declare <vscale x 2 x i64> @llvm.aarch64.sve.dupq.lane.nxv2i64(<vscale x 2 x i64>, i64)
 declare <vscale x 8 x half> @llvm.aarch64.sve.dupq.lane.nxv8f16(<vscale x 8 x half>, i64)
 declare <vscale x 8 x bfloat> @llvm.aarch64.sve.dupq.lane.nxv8bf16(<vscale x 8 x bfloat>, i64)
 declare <vscale x 4 x float> @llvm.aarch64.sve.dupq.lane.nxv4f32(<vscale x 4 x float>, i64)
 declare <vscale x 2 x double> @llvm.aarch64.sve.dupq.lane.nxv2f64(<vscale x 2 x double>, i64)
 
 declare <vscale x 2 x double> @llvm.vector.insert.nxv2f64.v2f64(<vscale x 2 x double>, <2 x double>, i64)
 declare <vscale x 4 x float> @llvm.vector.insert.nxv4f32.v4f32(<vscale x 4 x float>, <4 x float>, i64)
 declare <vscale x 8 x half> @llvm.vector.insert.nxv8f16.v8f16(<vscale x 8 x half>, <8 x half>, i64)
 declare <vscale x 2 x i64> @llvm.vector.insert.nxv2i64.v2i64(<vscale x 2 x i64>, <2 x i64>, i64)
 declare <vscale x 4 x i32> @llvm.vector.insert.nxv4i32.v4i32(<vscale x 4 x i32>, <4 x i32>, i64)
 declare <vscale x 8 x i16> @llvm.vector.insert.nxv8i16.v8i16(<vscale x 8 x i16>, <8 x i16>, i64)
 declare <vscale x 16 x i8> @llvm.vector.insert.nxv16i8.v16i8(<vscale x 16 x i8>, <16 x i8>, i64)
 declare <vscale x 8 x bfloat> @llvm.vector.insert.nxv8bf16.v8bf16(<vscale x 8 x bfloat>, <8 x bfloat>, i64)
 
+declare <vscale x 16 x i8> @llvm.aarch64.sve.dup.nxv16i8(<vscale x 16 x i8>, <vscale x 16 x i1>, i8)
+declare <vscale x 8 x i16> @llvm.aarch64.sve.dup.nxv8i16(<vscale x 8 x i16>, <vscale x 8 x i1>, i16)
+declare <vscale x 4 x i32> @llvm.aarch64.sve.dup.nxv4i32(<vscale x 4 x i32>, <vscale x 4 x i1>, i32)
+declare <vscale x 2 x i64> @llvm.aarch64.sve.dup.nxv2i64(<vscale x 2 x i64>, <vscale x 2 x i1>, i64)
+declare <vscale x 8 x half> @llvm.aarch64.sve.dup.nxv8f16(<vscale x 8 x half>, <vscale x 8 x i1>, half)
+declare <vscale x 4 x float> @llvm.aarch64.sve.dup.nxv4f32(<vscale x 4 x float>, <vscale x 4 x i1>, float)
+declare <vscale x 2 x double> @llvm.aarch64.sve.dup.nxv2f64(<vscale x 2 x double>, <vscale x 2 x i1>, double)
+declare <vscale x 4 x half> @llvm.aarch64.sve.dup.nxv4f16(<vscale x 4 x half>, <vscale x 4 x i1>, half)
+declare <vscale x 2 x half> @llvm.aarch64.sve.dup.nxv2f16(<vscale x 2 x half>, <vscale x 2 x i1>, half)
+declare <vscale x 2 x float> @llvm.aarch64.sve.dup.nxv2f32(<vscale x 2 x float>, <vscale x 2 x i1>, float)
+
+
 attributes #0 = { "target-features"="+sve,+bf16" }

llvm/test/tools/llvm-mca/AArch64/A64FX/A64FX-sve-instructions.s

 # CHECK-NEXT:  1      1     0.50                        addvl x0, x0, #-32
 # CHECK-NEXT:  1      1     0.50                        addvl x21, x21, #0
 # CHECK-NEXT:  1      1     0.50                        addvl x23, x8, #-1
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #1]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #2]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d, lsl #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #1]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #2]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, sxtw]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #1]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #2]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw #3]
 # CHECK-NEXT:  1      4     0.50                        adr z0.d, [z0.d, z0.d, uxtw]
 # CHECK-NEXT:  2      5     2.00                        adr z0.d, [z0.d, z0.d]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #1]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #2]
 # CHECK-NEXT:  2      5     2.00                        adr z0.s, [z0.s, z0.s, lsl #3]

 # CHECK-NEXT:  1      16    2.00    *             U     ld1h  { z31.d }, p7/z, [z31.d, #62]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1h  { z31.h }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1h  { z31.s }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, sxtw #1]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, uxtw #1]
 # CHECK-NEXT:  1      19    4.00    *             U     ld1h  { z31.s }, p7/z, [z31.s, #62]
 # CHECK-NEXT:  1      11    0.50    *                   ld1h  { z5.h }, p3/z, [sp, x16, lsl #1]
 # CHECK-NEXT:  1      11    0.50    *                   ld1h  { z5.h }, p3/z, [x17, x16, lsl #1]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.b }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.b }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rb { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rd { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rd { z31.d }, p7/z, [sp, #504]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.h }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rh { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.b }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.b }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rb { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rd { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rd { z31.d }, p7/z, [sp, #504]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.h }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rh { z31.s }, p7/z, [sp, #126]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z0.b }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z0.b }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z21.b }, p5/z, [x10, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z23.b }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqb  { z31.b }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z0.d }, p0/z, [x0, x0, lsl #3]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z23.d }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z23.d }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqd  { z31.d }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z0.h }, p0/z, [x0, x0, lsl #1]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z23.h }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z23.h }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqh  { z31.h }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z0.s }, p0/z, [x0, x0, lsl #2]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z23.s }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z23.s }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      11    0.50    *                   ld1rqw  { z31.s }, p7/z, [sp, #-16]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsb  { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsh  { z31.s }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsw  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rsw  { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      11    0.50    *             U     ld1rw { z31.s }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsb  { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsh  { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsw  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rsw  { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      11    0.50    *                   ld1rw { z31.s }, p7/z, [sp, #252]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      16    2.00    *             U     ld1sb { z0.d }, p0/z, [z0.d]
 # CHECK-NEXT:  1      11    0.50    *                   ld1sb { z0.h }, p0/z, [sp, x0]
 # CHECK-NEXT:  1      11    0.50    *                   ld1sb { z0.h }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      23    4.00    *             U     ld1sb { z0.s }, p0/z, [x0, z0.s, sxtw]
 # CHECK-NEXT:  1      11    0.50    *             U     ld1sb { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      19    4.00    *             U     ld1sb { z0.s }, p0/z, [z0.s]

llvm/test/tools/llvm-mca/AArch64/Neoverse/N2-sve-instructions.s

 # CHECK-NEXT:  4      9     1.00    *             U     ld1h  { z31.d }, p7/z, [z31.d, #62]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1h  { z31.h }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1h  { z31.s }, p7/z, [sp, #-1, mul vl]
 # CHECK-NEXT:  4      10    1.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, sxtw #1]
 # CHECK-NEXT:  4      10    1.00    *             U     ld1h  { z31.s }, p7/z, [sp, z31.s, uxtw #1]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1h  { z31.s }, p7/z, [z31.s, #62]
 # CHECK-NEXT:  1      6     0.50    *                   ld1h  { z5.h }, p3/z, [sp, x16, lsl #1]
 # CHECK-NEXT:  1      6     0.50    *                   ld1h  { z5.h }, p3/z, [x17, x16, lsl #1]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.b }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.b }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rb { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rd { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rd { z31.d }, p7/z, [sp, #504]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.h }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rh { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.b }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.b }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rb { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rd { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rd { z31.d }, p7/z, [sp, #504]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.h }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rh { z31.s }, p7/z, [sp, #126]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z0.b }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z0.b }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z21.b }, p5/z, [x10, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z23.b }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqb  { z31.b }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z0.d }, p0/z, [x0, x0, lsl #3]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z23.d }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z23.d }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqd  { z31.d }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z0.h }, p0/z, [x0, x0, lsl #1]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z23.h }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z23.h }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqh  { z31.h }, p7/z, [sp, #-16]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z0.s }, p0/z, [x0, x0, lsl #2]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z23.s }, p3/z, [x13, #-128]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z23.s }, p3/z, [x13, #112]
 # CHECK-NEXT:  1      6     0.33    *                   ld1rqw  { z31.s }, p7/z, [sp, #-16]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.h }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.d }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.h }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsb  { z31.s }, p7/z, [sp, #63]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z31.d }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsh  { z31.s }, p7/z, [sp, #126]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsw  { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rsw  { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z0.d }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z0.s }, p0/z, [x0]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z31.d }, p7/z, [sp, #252]
-# CHECK-NEXT:  1      6     0.33    *             U     ld1rw { z31.s }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.h }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.d }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.h }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsb  { z31.s }, p7/z, [sp, #63]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z31.d }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsh  { z31.s }, p7/z, [sp, #126]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsw  { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rsw  { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z0.d }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z0.s }, p0/z, [x0]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z31.d }, p7/z, [sp, #252]
+# CHECK-NEXT:  1      6     0.33    *                   ld1rw { z31.s }, p7/z, [sp, #252]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.d }, p0/z, [x0]
 # CHECK-NEXT:  4      9     1.00    *             U     ld1sb { z0.d }, p0/z, [z0.d]
 # CHECK-NEXT:  1      6     0.50    *                   ld1sb { z0.h }, p0/z, [sp, x0]
 # CHECK-NEXT:  1      6     0.50    *                   ld1sb { z0.h }, p0/z, [x0, x0]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.h }, p0/z, [x0]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1sb { z0.s }, p0/z, [x0, z0.s, sxtw]
 # CHECK-NEXT:  1      6     0.33    *             U     ld1sb { z0.s }, p0/z, [x0]
 # CHECK-NEXT:  2      9     0.50    *             U     ld1sb { z0.s }, p0/z, [z0.s]