[Clang][AArch64][SME] Add vector load/store (ld1/st1) intrinsics

[bryanpkc]

Summary

[Clang][AArch64][SME] Add vector load/store (ld1/st1) intrinsics

This patch adds support for the following SME ACLE intrinsics (as defined
in https://arm-software.github.io/acle/main/acle.html):

• svld1_hor_za8 // also for _za16, _za32, _za64 and _za128
• svld1_hor_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svld1_ver_za8 // also for _za16, _za32, _za64 and _za128
• svld1_ver_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svst1_hor_za8 // also for _za16, _za32, _za64 and _za128
• svst1_hor_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svst1_ver_za8 // also for _za16, _za32, _za64 and _za128
• svst1_ver_vnum_za8 // also for _za16, _za32, _za64 and _za128

SveEmitter.cpp is extended to generate arm_sme.h (currently named
arm_sme_draft_spec_subject_to_change.h) and other SME definitions from
arm_sme.td, which is modeled after arm_sve.td. Common TableGen definitions
are moved into arm_sve_sme_incl.td.

Co-authored-by: Sagar Kulkarni <[e-mail]>

[sdesmalen]

Hi @sagarkulkarni19, this patch will have to wait until the ABI is implemented so that the builtins can be decorated with the attributes. A first patch proposing these attributes is in D127762. LLVM patches that use these attributes to implement the ABI will follow soon. We could add you as reviewer to these patches if you'd like to help us with that?

clang/include/clang/Basic/arm_sve.td

209

Is there value in having both IsSME and IsSMELoadStore?

clang/lib/Basic/Targets/AArch64.cpp

342

When this macro is non-zero, it suggests that the compiler implements the full SME ACLE. That is currently not yet the case, so until then we should leave this macro undefined.

[sagarkulkarni19]

In D127910#3589068, @sdesmalen wrote:

Hi @sagarkulkarni19, this patch will have to wait until the ABI is implemented so that the builtins can be decorated with the attributes. A first patch proposing these attributes is in D127762. LLVM patches that use these attributes to implement the ABI will follow soon. We could add you as reviewer to these patches if you'd like to help us with that?

Hi @sdesmalen, sounds good. I can update this patch accordingly when the attributes patches are in. Sure, please add me as a reviewer to those patches. We are porting much of our downstream C intrinsics implementation to support the ACLE instead and can contribute this effort upstream. I have also created an article on discourse (https://discourse.llvm.org/t/clang-sme-acle-intrinsics/63223) for further discussion on this.

[sagarkulkarni19]

clang/include/clang/Basic/arm_sve.td

209

IsSME flag is checked in the SveEmitter.cpp : createSMEHeader(...) to generate arm_sme.h with only the SME intrinsics, whereas IsSMELoadStore is used to correctly CodeGen (CGBuiltins.cpp) load and store intrinsics.

clang/lib/Basic/Targets/AArch64.cpp

342

Okay makes sense. But until all SME ACLE is implemented, do we need some sort of guard in the meantime while the SME intrinsics that are implemented incrementally?

[sdesmalen]

clang/lib/Basic/Targets/AArch64.cpp

342

The tests need to have manual -D__ARM_FEATURE_SME in the RUN lines. Once the feature is implemented and we add the definition automatically when +sme is specified, we update the tests. See for example D81725 where we did this for SVE.

[sagarkulkarni19]

Updated testcases and also added the vnum variant of the ld1 and st1 intrinsics.

[sagarkulkarni19]

clang/lib/Basic/Targets/AArch64.cpp

342

Thanks. That makes sense, now the macro is undefined and I have updated the testcases accordingly.

[sagarkulkarni19]

• Support Opaque pointers
• Correct predicate types for the intrinsics.
• Decorate intrinsics with SME attributes.

[sagarkulkarni19]

Update testcases by adding "arm_streaming" attribute to the callee.

[david-arm]

Hi @sagarkulkarni19, thank you for working on the ACLE builtins for SME! I've had a look through and I have a few comments, mostly around how the code is structured. It would be good if you could try to separate SVE from SME in this implementation, in the same way that NEON and SVE are distinct. It's possible to do this whilst reusing much of the code in SveEmitter.cpp.

clang/include/clang/Basic/arm_sve.td

209

You don't need the IsSME flag either because in CodeGenFunction::EmitAArch64BuiltinExpr you can do exactly the same thing as SVE, i.e. something like

if (BuiltinID >= AArch64::FirstSMEBuiltin &&
    BuiltinID <= AArch64::LastSMEBuiltin)
  return EmitAArch64SMEBuiltinExpr(BuiltinID, E);

210

We don't need these new flags 'IsSMELd1' and 'IsSMESt1'. Please can you reuse the existing IsLoad and IsStore flags?

548

SME is really a distinct architecture and so I think it should live in it's own arm_sme.td file in the same way that we have arm_neon.td and arm_sve.td. It's possible to do this and still reuse the SveEmitter.cpp code. If you look at SveEmitter.cpp you'll see these functions:

void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createHeader(OS);
}

void EmitSveBuiltins(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createBuiltins(OS);
}

void EmitSveBuiltinCG(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createCodeGenMap(OS);
}

void EmitSveRangeChecks(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createRangeChecks(OS);
}

It would be good to add similar ones for SME, i.e. EmitSmeRangeChecks, etc.

clang/lib/CodeGen/CGBuiltin.cpp

9170

I would prefer this to be handled inside it's own EmitAArch64SMEBuiltinExpr, since it's a separate architecture.

clang/utils/TableGen/SveEmitter.cpp

858

I would prefer this to be done more precisely via separate attribute flags, i.e. in the .td file decorate each ACLE builtin with something like IsShared, IsStreaming, IsStreamingCompatible, etc. otherwise you'll end up needing loads of flags for all different possible combinations. That way you can do:

if (this->Flags & Emitter.getEnumValueForFlag("IsStreaming"))
  this->SMEAttributes += "arm_streaming";

etc.

1018

If you move the builtins to their own file in arm_sme.td and emit the records using interfaces like EmitSmeBuiltins, etc. then you already know they are SME builtins and so don't need the flag.

[david-arm]

Hi @sagarkulkarni19, just a gentle ping to see if you are still planning to do more work on this patch?

[bryanpkc]

In D127910#3885699, @david-arm wrote:

Hi @sagarkulkarni19, just a gentle ping to see if you are still planning to do more work on this patch?

@david-arm we are going to update this patch very soon. Sorry for the delays.

[bryanpkc]

Updated the patch according to review comments which suggest to treat SME as a different architecture.

[bryanpkc]

@david-arm, I have moved all the SME definitions into a new file, arm_sme.td. I have moved the common definitions into another file, arm_sve_sme_incl.td, which will be included by both arm_sve.td and arm_sme.td. SveEmitter has been updated to incorporate your suggestions.

[bryanpkc]

Removed some diffs that weren't necessary.

[bryanpkc]

Removed some more unnecessary lines.

[bryanpkc]

Updated the patch to define the __ARM_FEATURE_SME macro when the feature is enabled.

[bryanpkc]

Minor clean-up. Sorry for the noise.

[david-arm]

Hi @bryanpkc, this is looking a lot better now and thanks for addressing the comments! I've not reviewed all of the patch yet, but I do have a few more comments. The most important ones are about performing immediate range checks for the builtins and not declaring the __ARM_FEATURE_SME yet.

clang/include/clang/Basic/TargetBuiltins.h

292

I actually don't think you need to add this class - we should be able to just reuse the existing SVETypeFlags structure. I think this is fine because you've commonised the flags between SME and SVE.

clang/include/clang/Basic/arm_sme.td

21 ↗

(On Diff #492586)

I think all the load and store instructions need immediate checks for the tile and slice_offset here such as:

[ImmCheck<0, ImmCheck0>, ImmCheck<2, ImmCheck0_15>]

for SVLD1_HOR_ZA8 and the others. It's mentioned in the ACLE - https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics:

15.4.3.1 Common Rules

...
Every argument named tile, slice_offset or tile_mask must be an integer constant expression in the range of the underlying instruction.

clang/include/clang/Basic/arm_sve_sme_incl.td

126 ↗

(On Diff #492586)

Please can you add a comment here for the new Prototype modifier you added - '%'?

clang/lib/Basic/Targets/AArch64.cpp

414

Can you remove this please? We can't really set this macro until the SME ABI and ACLE is feature complete.

clang/lib/CodeGen/CGBuiltin.cpp

9030

I think that instead of this switch statement you should just be able to write something like:

Ops[3] = EmitSVEPredicateCast(Ops[3],  getSVEVectorForElementType(SVEBuiltinMemEltTy(TypeFlags)))

9072

I think you can just call

CGM.getIntrinsic(Intrinsic::aarch64_sme_cntsb)

9074

Again, I think you can just do

Builder.CreateCall(StreamingVectorLength)

9083

nit: Function *F = CGM.getIntrinsic(IntID);

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_int_const_expr_error.c

5 ↗

(On Diff #492586)

Once you've added the immediate range checks for the loads and stores it would be good add checks here for immediates outside the range for each instruction.

clang/utils/TableGen/SveEmitter.cpp

1681

If you reuse the existing SVETypeFlags rather than create a new SMETypeFlags then you only need the LLVM_GET_SME_IMMCHECKTYPES bit.

[dmgreen]

clang/utils/TableGen/SveEmitter.cpp

1524

We have been changing how the existing SVE and NEON instrinsics work a little. There are some details in https://reviews.llvm.org/D131064. The short version is it is best to not rely upon preprocessor macros, and instead define the intrinsics so that they can be used if the right target features are available. This allows us to do things like this below, even without a -march that supports sme, and have them callable at runtime under the right situations. We should be doing the same for SME.

__attribute__((target("+sme")))
void sme_func() {
  somesmeintrinsics();
}

[bryanpkc]

clang/include/clang/Basic/TargetBuiltins.h

292

Thanks for the suggestion. I have removed this class.

clang/include/clang/Basic/arm_sme.td

21 ↗

(On Diff #492586)

Done.

clang/include/clang/Basic/arm_sve_sme_incl.td

126 ↗

(On Diff #492586)

A comment for '%' already exists on line 103.

clang/lib/Basic/Targets/AArch64.cpp

414

OK. Could you educate me what else is needed for SME ABI and ACLE to be feature-complete? How can I help?

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_int_const_expr_error.c

5 ↗

(On Diff #492586)

Done, thanks for the suggestion. I have also moved these tests into clang/test/Sema/arm-sme-intrinsics/.

clang/utils/TableGen/SveEmitter.cpp

1524

I have refactored the SME intrinsics in a similar fashion. Could you confirm if I did it correctly?

[bryanpkc]

Rebased on trunk and addressed review comments.

[david-arm]

Thanks a lot for making all the changes @bryanpkc - it's looking really good now! I just have a few minor comments/suggestions and then I think it looks good to go.

clang/include/clang/Basic/arm_sme.td

22 ↗

(On Diff #495592)

This is just a suggestion, but you could reduce the lines of code here if you want by creating a multiclass that creates both the horizontal and vertical variants for each size, i.e. something like

multiclass MyMultiClass<..> {
  def NAME # _H : MInst<...>
  def NAME # _V : MInst<...>
}

defm SVLD1_ZA8 : MyMultiClass<...>;

or whatever naming scheme you prefer, and same for the stores. Feel free to ignore this suggestion though if it doesn't help you!

clang/lib/Basic/Targets/AArch64.cpp

414

It should have complete support for the SME ABI and ACLE in terms of supporting the C/C++ level attributes as described here - https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics. For example, the compiler should support cases where a normal function calls a arm_streaming function and sets up the state correctly, etc. You can see @sdesmalen's patch to add the clang-side attributes here D127762. There should also be full support for all of the SME ACLE builtins.

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_imm.cpp

3 ↗

(On Diff #495592)

I think you can remove the -target-feature +sve flags from the RUN lines because +sme should imply that.

16 ↗

(On Diff #495592)

These tests are great! Thanks for doing this. Could you also add the _vnum variants too?

[sdesmalen]

clang/lib/Headers/CMakeLists.txt

310

The ACLE specification is still in a draft (ALP) state, which means there may still be subject to significant changes. To avoid users from using this implementation with the expectation that their code is compliant going forward, it would be good to rename the header file to something that makes it very clear this feature is not yet ready to use. I'm thinking of something like arm_sme_draft_spec_subject_to_change.h. When the specification goes out of draft, we can rename it to arm_sme.h. Could you rename the file for now?

[bryanpkc]

clang/lib/Headers/CMakeLists.txt

310

Would something shorter like arm_sme_draft.h or arm_sme_experimental.h suffice?

[bryanpkc]

Addressed review comments.

[bryanpkc]

clang/include/clang/Basic/arm_sme.td

22 ↗

(On Diff #495592)

Thanks for the suggestion. Refactored.

clang/lib/Headers/CMakeLists.txt

310

Renamed to arm_sme_experimental.h.

[bryanpkc]

Fixed minor bugs in the previous upload.

[dmgreen]

clang/utils/TableGen/SveEmitter.cpp

1524

It looks OK - as far as I can see. It might be worth adding a test for it? But otherwise it looks good. Thanks.

[sdesmalen]

clang/lib/Headers/CMakeLists.txt

310

While arm_sme_experimental.h is indeed shorter, it should be unequivocally clear to the user that they shouldn't rely on the function prototypes defined in this header file yet because the specification itself is not finalised. I think that adding the draft_spec_subject_to_change to the name makes that more clear than experimental, as the latter might suggest that the support is not yet entirely stable or complete. There probably isn't that much to gain from making the user experience better by using a shorter name, if the whole point is to prevent people from using it for any production code. So from that perspective, I still have a slight preference for arm_sme_draft_spec_subject_to_change.h.

[bryanpkc]

Addressed review comments.

[bryanpkc]

clang/lib/Headers/CMakeLists.txt

310

Renamed as suggested.

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_imm.cpp

3 ↗

(On Diff #495592)

If I understand llvm/lib/Target/AArch64/AArch64.td and clang/lib/Basic/Targets/AArch64.cpp correctly, -target-feature +sme does not imply -target-feature +sve. I can remove the -D__ARM_FEATURE_SME though.

On the other hand, -march=armv8+sme will imply -target-feature +sve, which I have implemented in D142702.

clang/utils/TableGen/SveEmitter.cpp

1524

Added a test case.

[kmclaughlin]

clang/lib/CodeGen/CGBuiltin.cpp

8572

Is it necessary to add an EltTypeBool128? I think the EmitSVEPredicateCast call in EmitSMELd1St1 is creating a vector based on the memory element type and not the predicate type?

[bryanpkc]

clang/lib/CodeGen/CGBuiltin.cpp

8572

You are right, EltTypeBool128 is not used right now; I can remove it. When we started working on the pre-ACLE intrinsics in our downstream compiler, we were planning to add something like svptrue_b128, even though the hardware PTRUE instruction doesn't support the Q size specifier. It is still not clear to me how the ACLE wants users to create a predicate vector for a call to a _za128 intrinsic.

[kmclaughlin]

Thank you for checking and removing EltTypeBool128. I think you have addressed all of the other comments on this patch too, so it looks good to me!

Please can you update the commit message before landing, to change the reference to arm_sme_experimental.h with arm_sme_draft_spec_subject_to_change.h?

clang/lib/CodeGen/CGBuiltin.cpp

8572

I think the reason why svptrue_b128 wasn't added is because there is no ptrue.q instruction as you say, and it is possible to use any of the other forms (.b, .h, etc) with the 128 bit instructions since they only require every 16th lane.
It would be possible for the user to create a 128 bit predicate with either svunpk or svzip of svbool_t with pfalse where necessary. Otherwise using the other forms, e.g svptrue_b64(), will achieve the same result and require fewer instructions.

[bryanpkc]

Removed EltTypeBool128 as suggested.

[bryanpkc]

Moved test case acle_target_sme.c into clang/test/Sema/aarch64-sme-intrinsics/.

[bryanpkc]

Fix some bugs in the range checks of immediate operands.

[sdesmalen]

clang/lib/Basic/Targets/AArch64.cpp

549–550

Why did you remove this?

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

17

Hi @bryanpkc, would you be happy to remove the dependence on the attributes patch for now, so that we can move forward to review/land this patch series?

I'm thinking of doing something like:

// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES ....
...

#ifdef DISABLE_SME_ATTRIBUTES
#define ARM_STREAMING_ATTR
#else
#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
#endif

...

ARM_STREAMING_ATTR void test_svld1_hor_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
  svld1_hor_za16(0, slice_base, 0, pg, ptr);
  svld1_hor_za16(1, slice_base, 7, pg, ptr);
}

With that the tests all pass, and when the attribute patches have landed we can just remove the -DDISABLE_SME_ATTRIBUTES from the RUN lines.

[bryanpkc]

clang/lib/Basic/Targets/AArch64.cpp

549–550

The Feature == "+sme" case is also handled below on line 782. Perhaps we should delete this block and add HasFullFP16 below.

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

17

Sounds good, I will give that a shot. If we can unblock this patch, I will also rebase and update the rest of the patch series. Thanks!

[bryanpkc]

@sdesmalen How does the latest version look to you? If you approve, I will land this and finish updating the rest of the patch series. One of the pre-merge builds failed for an unrelated reason, but other jobs were successful.

[sdesmalen]

Thanks @bryanpkc this patch looks good to me now. I'll make some time to review the other patches in the series as well after you update them.

clang/include/clang/Basic/TargetBuiltins.h

#undef LLVM_GET_SVE_IMMCHECKTYPES
     bool isAppendSVALL() const { return Flags & IsAppendSVALL; }
     bool isInsertOp1SVALL() const { return Flags & IsInsertOp1SVALL; }
     bool isGatherPrefetch() const { return Flags & IsGatherPrefetch; }
     bool isReverseUSDOT() const { return Flags & ReverseUSDOT; }
     bool isUndef() const { return Flags & IsUndef; }
     bool isTupleCreate() const { return Flags & IsTupleCreate; }
     bool isTupleGet() const { return Flags & IsTupleGet; }
     bool isTupleSet() const { return Flags & IsTupleSet; }
+    bool isSME() const { return Flags & IsSME; }
+    bool isSMELoadStore() const { return Flags & IsSMELoadStore; }
 
     uint64_t getBits() const { return Flags; }
     bool isFlagSet(uint64_t Flag) const { return Flags & Flag; }
   };
 
   /// Hexagon builtins
   namespace Hexagon {
     enum {
         LastTIBuiltin = clang::Builtin::FirstTSBuiltin-1,
 #define BUILTIN(ID, TYPE, ATTRS) BI##ID,
 #include "clang/Basic/BuiltinsHexagon.def"
         LastTSBuiltin
     };
   }
 

clang/include/clang/Basic/arm_sve.td

 // t: svint32_t
 // z: svuint32_t
 // g: svuint64_t
 // O: svfloat16_t
 // M: svfloat32_t
 // N: svfloat64_t
 
 // J: Prefetch type (sv_prfop)
+// %: pointer to void
 // A: pointer to int8_t
 // B: pointer to int16_t
 // C: pointer to int32_t
 // D: pointer to int64_t
 
 // E: pointer to uint8_t
 // F: pointer to uint16_t
 // G: pointer to uint32_t

 def IsPrefetch                : FlagType<0x08000000>; // Contiguous prefetches.
 def IsGatherPrefetch          : FlagType<0x10000000>;
 def ReverseCompare            : FlagType<0x20000000>; // Compare operands must be swapped.
 def ReverseUSDOT              : FlagType<0x40000000>; // Unsigned/signed operands must be swapped.
 def IsUndef                   : FlagType<0x80000000>; // Codegen `undef` of given type.
 def IsTupleCreate             : FlagType<0x100000000>;
 def IsTupleGet                : FlagType<0x200000000>;
 def IsTupleSet                : FlagType<0x400000000>;
+def IsSME                     : FlagType<0x800000000>;
+def IsSMELoadStore            : FlagType<0x1000000000>;
 
 // These must be kept in sync with the flags in include/clang/Basic/TargetBuiltins.h
 class ImmCheckType<int val> {
   int Value = val;
 }
 def ImmCheck0_31                : ImmCheckType<0>;  // 0..31 (used for e.g. predicate patterns)
 def ImmCheck1_16                : ImmCheckType<1>;  // 1..16
 def ImmCheckExtract             : ImmCheckType<2>;  // 0..(2048/sizeinbits(elt) - 1)
let ArchGuard = "defined(__ARM_FEATURE_SVE_BF16)" in {
   def SVBFDOT_N      : SInst<"svbfdot[_n_{0}]",      "MMda",  "b", MergeNone, "aarch64_sve_bfdot",        [IsOverloadNone]>;
   def SVBFMLAL_N     : SInst<"svbfmlalb[_n_{0}]",    "MMda",  "b", MergeNone, "aarch64_sve_bfmlalb",      [IsOverloadNone]>;
   def SVBFMLALT_N    : SInst<"svbfmlalt[_n_{0}]",    "MMda",  "b", MergeNone, "aarch64_sve_bfmlalt",      [IsOverloadNone]>;
   def SVBFDOT_LANE   : SInst<"svbfdot_lane[_{0}]",   "MMddn", "b", MergeNone, "aarch64_sve_bfdot_lane",   [IsOverloadNone], [ImmCheck<3, ImmCheck0_3>]>;
   def SVBFMLALB_LANE : SInst<"svbfmlalb_lane[_{0}]", "MMddn", "b", MergeNone, "aarch64_sve_bfmlalb_lane", [IsOverloadNone], [ImmCheck<3, ImmCheck0_7>]>;
   def SVBFMLALT_LANE : SInst<"svbfmlalt_lane[_{0}]", "MMddn", "b", MergeNone, "aarch64_sve_bfmlalt_lane", [IsOverloadNone], [ImmCheck<3, ImmCheck0_7>]>;
 }
 
+def SVLD1_HOR_ZA8 : MInst<"svld1_hor_za8", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1b_horiz">;
+def SVLD1_HOR_ZA16 : MInst<"svld1_hor_za16", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1h_horiz">;
+def SVLD1_HOR_ZA32 : MInst<"svld1_hor_za32", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1w_horiz">;
+def SVLD1_HOR_ZA64 : MInst<"svld1_hor_za64", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1d_horiz">;
+def SVLD1_HOR_ZA128 : MInst<"svld1_hor_za128", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1q_horiz">;
+def SVLD1_VER_ZA8 : MInst<"svld1_ver_za8", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1b_vert">;
+def SVLD1_VER_ZA16 : MInst<"svld1_ver_za16", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1h_vert">;
+def SVLD1_VER_ZA32 : MInst<"svld1_ver_za32", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1w_vert">;
+def SVLD1_VER_ZA64 : MInst<"svld1_ver_za64", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1d_vert">;
+def SVLD1_VER_ZA128 : MInst<"svld1_ver_za128", "vnmnPQ", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_ld1q_vert">;
+
 ////////////////////////////////////////////////////////////////////////////////
 // Stores
 
 // Store one vector (scalar base)
 def SVST1    : MInst<"svst1[_{d}]",  "vPpd", "csilUcUsUiUlhfd", [IsStore], MemEltTyDefault, "aarch64_sve_st1">;
 def SVST1B_S : MInst<"svst1b[_{d}]", "vPAd", "sil",             [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1B_U : MInst<"svst1b[_{d}]", "vPEd", "UsUiUl",          [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1H_S : MInst<"svst1h[_{d}]", "vPBd", "il",              [IsStore], MemEltTyInt16,   "aarch64_sve_st1">;

 // Store one vector, with no truncation, non-temporal (scalar base, VL displacement)
 def SVSTNT1_VNUM : MInst<"svstnt1_vnum[_{d}]", "vPpld", "csilUcUsUiUlhfd", [IsStore], MemEltTyDefault, "aarch64_sve_stnt1">;
 
 let ArchGuard = "defined(__ARM_FEATURE_SVE_BF16)" in {
   def SVSTNT1_BF      : MInst<"svstnt1[_{d}]",      "vPpd",  "b", [IsStore], MemEltTyDefault, "aarch64_sve_stnt1">;
   def SVSTNT1_VNUM_BF : MInst<"svstnt1_vnum[_{d}]", "vPpld", "b", [IsStore], MemEltTyDefault, "aarch64_sve_stnt1">;
 }
 
+def SVST1_HOR_ZA8 : MInst<"svst1_hor_za8", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1b_horiz">;
+def SVST1_HOR_ZA16 : MInst<"svst1_hor_za16", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1h_horiz">;
+def SVST1_HOR_ZA32 : MInst<"svst1_hor_za32", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1w_horiz">;
+def SVST1_HOR_ZA64 : MInst<"svst1_hor_za64", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1d_horiz">;
+def SVST1_HOR_ZA128 : MInst<"svst1_hor_za128", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1q_horiz">;
+def SVST1_VER_ZA8 : MInst<"svst1_ver_za8", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1b_vert">;
+def SVST1_VER_ZA16 : MInst<"svst1_ver_za16", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1h_vert">;
+def SVST1_VER_ZA32 : MInst<"svst1_ver_za32", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1w_vert">;
+def SVST1_VER_ZA64 : MInst<"svst1_ver_za64", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1d_vert">;
+def SVST1_VER_ZA128 : MInst<"svst1_ver_za128", "vnmnP%", "", [IsOverloadNone, IsSME, IsSMELoadStore], MemEltTyDefault, "aarch64_sme_st1q_vert">;
+
 ////////////////////////////////////////////////////////////////////////////////
 // Prefetches
 
 // Prefetch (Scalar base)
 def SVPRFB : MInst<"svprfb", "vPQJ", "c", [IsPrefetch], MemEltTyInt8,  "aarch64_sve_prf">;
 def SVPRFH : MInst<"svprfh", "vPQJ", "s", [IsPrefetch], MemEltTyInt16, "aarch64_sve_prf">;
 def SVPRFW : MInst<"svprfw", "vPQJ", "i", [IsPrefetch], MemEltTyInt32, "aarch64_sve_prf">;
 def SVPRFD : MInst<"svprfd", "vPQJ", "l", [IsPrefetch], MemEltTyInt64, "aarch64_sve_prf">;

clang/lib/Basic/Targets/AArch64.h

class LLVM_LIBRARY_VISIBILITY AArch64TargetInfo : public TargetInfo {
   bool HasLS64;
   bool HasRandGen;
   bool HasMatMul;
   bool HasSVE2;
   bool HasSVE2AES;
   bool HasSVE2SHA3;
   bool HasSVE2SM4;
   bool HasSVE2BitPerm;
+  bool HasSME;
   bool HasMatmulFP64;
   bool HasMatmulFP32;
   bool HasLSE;
   bool HasFlagM;
   bool HasMOPS;
 
   llvm::AArch64::ArchKind ArchKind;
 

clang/lib/Basic/Targets/AArch64.cpp

if (HasSVE2 && HasSVE2BitPerm)
     Builder.defineMacro("__ARM_FEATURE_SVE2_BITPERM", "1");
 
   if (HasSVE2 && HasSVE2SHA3)
     Builder.defineMacro("__ARM_FEATURE_SVE2_SHA3", "1");
 
   if (HasSVE2 && HasSVE2SM4)
     Builder.defineMacro("__ARM_FEATURE_SVE2_SM4", "1");
 
+  if (HasSME)
+    Builder.defineMacro("__ARM_FEATURE_SME", "1");
+
   if (HasCRC)
     Builder.defineMacro("__ARM_FEATURE_CRC32", "1");
 
   // The __ARM_FEATURE_CRYPTO is deprecated in favor of finer grained feature
   // macros for AES, SHA2, SHA3 and SM4
   if (HasAES && HasSHA2)
     Builder.defineMacro("__ARM_FEATURE_CRYPTO", "1");
 
if ((FPU & SveMode) && HasMatmulFP32)
     Builder.defineMacro("__ARM_FEATURE_SVE_MATMUL_FP32", "1");
 
   if ((FPU & SveMode) && HasMatMul)
     Builder.defineMacro("__ARM_FEATURE_SVE_MATMUL_INT8", "1");
 
   if ((FPU & NeonMode) && HasFP16FML)
     Builder.defineMacro("__ARM_FEATURE_FP16_FML", "1");
 
   if (Opts.hasSignReturnAddress()) {
     // Bitmask:
     // 0: Protection using the A key
     // 1: Protection using the B key
     // 2: Protection including leaf functions
     unsigned Value = 0;
 
     if (Opts.isSignReturnAddressWithAKey())
       Value |= (1 << 0);
bool AArch64TargetInfo::handleTargetFeatures(std::vector<std::string> &Features,
   HasRandGen = false;
   HasMatMul = false;
   HasBFloat16 = false;
   HasSVE2 = false;
   HasSVE2AES = false;
   HasSVE2SHA3 = false;
   HasSVE2SM4 = false;
   HasSVE2BitPerm = false;
+  HasSME = false;
   HasMatmulFP64 = false;
   HasMatmulFP32 = false;
   HasLSE = false;
   HasMOPS = false;
 
   ArchKind = llvm::AArch64::ArchKind::INVALID;
 
   for (const auto &Feature : Features) {
     if (Feature == "+neon")
       FPU |= NeonMode;
     if (Feature == "+sve") {
       FPU |= SveMode;
       HasFullFP16 = true;
     }
     if (Feature == "+sve2") {
       FPU |= SveMode;
if (Feature == "+sve2-sm4") {
       HasSVE2SM4 = true;
     }
     if (Feature == "+sve2-bitperm") {
       FPU |= SveMode;
       HasFullFP16 = true;
       HasSVE2 = true;
       HasSVE2BitPerm = true;
     }
+    if (Feature == "+sme") {
+      HasSME = true;
+    }
     if (Feature == "+f32mm") {
       FPU |= SveMode;
       HasMatmulFP32 = true;
     }
     if (Feature == "+f64mm") {
       FPU |= SveMode;
       HasMatmulFP64 = true;
     }

clang/lib/CodeGen/CGBuiltin.cpp

llvm::Type *CodeGenFunction::getEltType(const SVETypeFlags &TypeFlags) {
 
   case SVETypeFlags::EltTyBFloat16:
     return Builder.getBFloatTy();
 
   case SVETypeFlags::EltTyBool8:
   case SVETypeFlags::EltTyBool16:
   case SVETypeFlags::EltTyBool32:
   case SVETypeFlags::EltTyBool64:
     return Builder.getInt1Ty();
   }
 }
 
 // Return the llvm predicate vector type corresponding to the specified element
 // TypeFlags.
 llvm::ScalableVectorType *
 CodeGenFunction::getSVEPredType(const SVETypeFlags &TypeFlags) {
   switch (TypeFlags.getEltType()) {
Value *CodeGenFunction::EmitSVEMaskedStore(const CallExpr *E,
   Function *F = CGM.getIntrinsic(BuiltinID, MemoryTy);
   auto *Store =
       cast<llvm::Instruction>(Builder.CreateCall(F, {Val, Predicate, BasePtr}));
   auto TBAAInfo = CGM.getTBAAAccessInfo(LangPTy->getPointeeType());
   CGM.DecorateInstructionWithTBAA(Store, TBAAInfo);
   return Store;
 }
 
+Value *CodeGenFunction::EmitSMELoadStore(SVETypeFlags TypeFlags,
+                                         SmallVectorImpl<Value *> &Ops,
+                                         unsigned IntID) {
+  SmallVector<Value *> NewOps;
+  NewOps.push_back(Ops[3]);
+  llvm::Value *BasePtr;
+  switch (IntID) {
+  case Intrinsic::aarch64_sme_ld1h_horiz:
+  case Intrinsic::aarch64_sme_ld1h_vert:
+  case Intrinsic::aarch64_sme_st1h_horiz:
+  case Intrinsic::aarch64_sme_st1h_vert:
+    BasePtr =
+        Builder.CreatePointerCast(Ops[4], llvm::PointerType::get(Int16Ty, 0));
+    break;
+  case Intrinsic::aarch64_sme_ld1w_horiz:
+  case Intrinsic::aarch64_sme_ld1w_vert:
+  case Intrinsic::aarch64_sme_st1w_horiz:
+  case Intrinsic::aarch64_sme_st1w_vert:
+    BasePtr =
+        Builder.CreatePointerCast(Ops[4], llvm::PointerType::get(Int32Ty, 0));
+    break;
+  case Intrinsic::aarch64_sme_ld1d_horiz:
+  case Intrinsic::aarch64_sme_ld1d_vert:
+  case Intrinsic::aarch64_sme_st1d_horiz:
+  case Intrinsic::aarch64_sme_st1d_vert:
+    BasePtr =
+        Builder.CreatePointerCast(Ops[4], llvm::PointerType::get(Int64Ty, 0));
+    break;
+  case Intrinsic::aarch64_sme_ld1q_horiz:
+  case Intrinsic::aarch64_sme_ld1q_vert:
+  case Intrinsic::aarch64_sme_st1q_horiz:
+  case Intrinsic::aarch64_sme_st1q_vert:
+    BasePtr = Builder.CreatePointerCast(
+        Ops[4], llvm::PointerType::get(
+                    llvm::IntegerType::get(getLLVMContext(), 128), 0));
+    break;
+  default:
+    BasePtr = Ops[4];
+    break;
+  }
+  NewOps.push_back(BasePtr);
+  NewOps.push_back(Ops[0]);
+  llvm::Value *CastOffset = Builder.CreateIntCast(Ops[2], Int32Ty, false);
+  NewOps.push_back(Builder.CreateAdd(Ops[1], CastOffset, "tileslice"));
+  Function *F = CGM.getIntrinsic(IntID, {});
+  return Builder.CreateCall(F, NewOps);
+}
+
 // Limit the usage of scalable llvm IR generated by the ACLE by using the
 // sve dup.x intrinsic instead of IRBuilder::CreateVectorSplat.
 Value *CodeGenFunction::EmitSVEDupX(Value *Scalar, llvm::Type *Ty) {
   auto F = CGM.getIntrinsic(Intrinsic::aarch64_sve_dup_x, Ty);
   return Builder.CreateCall(F, Scalar);
 }
 
 Value *CodeGenFunction::EmitSVEDupX(Value* Scalar) {
   return EmitSVEDupX(Scalar, getSVEVectorForElementType(Scalar->getType()));
 }
 
 Value *CodeGenFunction::EmitSVEReinterpret(Value *Val, llvm::Type *Ty) {
   // FIXME: For big endian this needs an additional REV, or needs a separate
   // intrinsic that is code-generated as a no-op, because the LLVM bitcast
   // instruction is defined as 'bitwise' equivalent from memory point of
   // view (when storing/reloading), whereas the svreinterpret builtin
   // implements bitwise equivalent cast from register point of view.
   // LLVM CodeGen for a bitcast must add an explicit REV for big-endian.
   return Builder.CreateBitCast(Val, Ty);
 }
 
 static void InsertExplicitZeroOperand(CGBuilderTy &Builder, llvm::Type *Ty,
Value *CodeGenFunction::EmitAArch64SVEBuiltinExpr(unsigned BuiltinID,
   auto *Builtin = findARMVectorIntrinsicInMap(AArch64SVEIntrinsicMap, BuiltinID,
                                               AArch64SVEIntrinsicsProvenSorted);
   SVETypeFlags TypeFlags(Builtin->TypeModifier);
   if (TypeFlags.isLoad())
     return EmitSVEMaskedLoad(E, Ty, Ops, Builtin->LLVMIntrinsic,
                              TypeFlags.isZExtReturn());
   else if (TypeFlags.isStore())
     return EmitSVEMaskedStore(E, Ops, Builtin->LLVMIntrinsic);
+  else if (TypeFlags.isSMELoadStore())
+    return EmitSMELoadStore(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isGatherLoad())
     return EmitSVEGatherLoad(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isScatterStore())
     return EmitSVEScatterStore(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isPrefetch())
     return EmitSVEPrefetchLoad(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isGatherPrefetch())
     return EmitSVEGatherPrefetch(TypeFlags, Ops, Builtin->LLVMIntrinsic);

clang/lib/CodeGen/CodeGenFunction.h

llvm::Value *EmitSVEScatterStore(const SVETypeFlags &TypeFlags,
                                    llvm::SmallVectorImpl<llvm::Value *> &Ops,
                                    unsigned IntID);
   llvm::Value *EmitSVEMaskedLoad(const CallExpr *, llvm::Type *ReturnTy,
                                  SmallVectorImpl<llvm::Value *> &Ops,
                                  unsigned BuiltinID, bool IsZExtReturn);
   llvm::Value *EmitSVEMaskedStore(const CallExpr *,
                                   SmallVectorImpl<llvm::Value *> &Ops,
                                   unsigned BuiltinID);
+  llvm::Value *EmitSMELoadStore(SVETypeFlags TypeFlags,
+                                llvm::SmallVectorImpl<llvm::Value *> &Ops,
+                                unsigned IntID);
   llvm::Value *EmitSVEPrefetchLoad(const SVETypeFlags &TypeFlags,
                                    SmallVectorImpl<llvm::Value *> &Ops,
                                    unsigned BuiltinID);
   llvm::Value *EmitSVEGatherPrefetch(const SVETypeFlags &TypeFlags,
                                      SmallVectorImpl<llvm::Value *> &Ops,
                                      unsigned IntID);
   llvm::Value *EmitSVEStructLoad(const SVETypeFlags &TypeFlags,
                                  SmallVectorImpl<llvm::Value *> &Ops,

clang/lib/Headers/CMakeLists.txt

 # Generate header files and copy them to the build directory
 if(ARM IN_LIST LLVM_TARGETS_TO_BUILD OR AArch64 IN_LIST LLVM_TARGETS_TO_BUILD)
   # Generate arm_neon.h
   clang_generate_header(-gen-arm-neon arm_neon.td arm_neon.h)
   # Generate arm_fp16.h
   clang_generate_header(-gen-arm-fp16 arm_fp16.td arm_fp16.h)
   # Generate arm_sve.h
   clang_generate_header(-gen-arm-sve-header arm_sve.td arm_sve.h)
+  # Generate arm_sme.h
+  clang_generate_header(-gen-arm-sme-header arm_sve.td arm_sme.h)
   # Generate arm_bf16.h
   clang_generate_header(-gen-arm-bf16 arm_bf16.td arm_bf16.h)
   # Generate arm_mve.h
   clang_generate_header(-gen-arm-mve-header arm_mve.td arm_mve.h)
   # Generate arm_cde.h
   clang_generate_header(-gen-arm-cde-header arm_cde.td arm_cde.h)
   # Copy arm_neon_sve_bridge.h
   copy_header_to_output_dir(${CMAKE_CURRENT_SOURCE_DIR}
     arm_neon_sve_bridge.h
   )
 
   # Add headers to target specific lists
   list(APPEND arm_common_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_neon.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_fp16.h"
     )
 
   list(APPEND arm_only_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_mve.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_cde.h"
     )
 
   list(APPEND aarch64_only_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_sve.h"
+    "${CMAKE_CURRENT_BINARY_DIR}/arm_sme.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_bf16.h"
     "${output_dir}/arm_neon_sve_bridge.h"
     )
 endif()
 if(RISCV IN_LIST LLVM_TARGETS_TO_BUILD)
   # Generate riscv_vector.h
   clang_generate_header(-gen-riscv-vector-header riscv_vector.td riscv_vector.h)
   list(APPEND riscv_generated_files

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefix=CPP-CHECK
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme.h>
+
+// CHECK-LABEL: @test_svld1_hor_za8(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z18test_svld1_hor_za8ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CPP-CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CPP-CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CPP-CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CPP-CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CPP-CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CPP-CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CPP-CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_hor_za8(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za8(0, slice_base, 0, pg, ptr);
+  svld1_hor_za8(0, slice_base, 1, pg, ptr);
+  svld1_hor_za8(0, slice_base, 2, pg, ptr);
+  svld1_hor_za8(0, slice_base, 3, pg, ptr);
+  svld1_hor_za8(0, slice_base, 4, pg, ptr);
+  svld1_hor_za8(0, slice_base, 5, pg, ptr);
+  svld1_hor_za8(0, slice_base, 6, pg, ptr);
+  svld1_hor_za8(0, slice_base, 7, pg, ptr);
+  svld1_hor_za8(0, slice_base, 8, pg, ptr);
+  svld1_hor_za8(0, slice_base, 9, pg, ptr);
+  svld1_hor_za8(0, slice_base, 10, pg, ptr);
+  svld1_hor_za8(0, slice_base, 11, pg, ptr);
+  svld1_hor_za8(0, slice_base, 12, pg, ptr);
+  svld1_hor_za8(0, slice_base, 13, pg, ptr);
+  svld1_hor_za8(0, slice_base, 14, pg, ptr);
+  svld1_hor_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_hor_za16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_hor_za16ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_hor_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za16(0, slice_base, 0, pg, ptr);
+  svld1_hor_za16(0, slice_base, 1, pg, ptr);
+  svld1_hor_za16(0, slice_base, 2, pg, ptr);
+  svld1_hor_za16(0, slice_base, 3, pg, ptr);
+  svld1_hor_za16(0, slice_base, 4, pg, ptr);
+  svld1_hor_za16(0, slice_base, 5, pg, ptr);
+  svld1_hor_za16(0, slice_base, 6, pg, ptr);
+  svld1_hor_za16(0, slice_base, 7, pg, ptr);
+  svld1_hor_za16(1, slice_base, 1, pg, ptr);
+  svld1_hor_za16(1, slice_base, 2, pg, ptr);
+  svld1_hor_za16(1, slice_base, 3, pg, ptr);
+  svld1_hor_za16(1, slice_base, 4, pg, ptr);
+  svld1_hor_za16(1, slice_base, 5, pg, ptr);
+  svld1_hor_za16(1, slice_base, 6, pg, ptr);
+  svld1_hor_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_hor_za32(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_hor_za32ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_hor_za32(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za32(0, slice_base, 0, pg, ptr);
+  svld1_hor_za32(0, slice_base, 1, pg, ptr);
+  svld1_hor_za32(0, slice_base, 2, pg, ptr);
+  svld1_hor_za32(0, slice_base, 3, pg, ptr);
+  svld1_hor_za32(1, slice_base, 0, pg, ptr);
+  svld1_hor_za32(1, slice_base, 1, pg, ptr);
+  svld1_hor_za32(1, slice_base, 2, pg, ptr);
+  svld1_hor_za32(1, slice_base, 3, pg, ptr);
+  svld1_hor_za32(2, slice_base, 0, pg, ptr);
+  svld1_hor_za32(2, slice_base, 1, pg, ptr);
+  svld1_hor_za32(2, slice_base, 2, pg, ptr);
+  svld1_hor_za32(2, slice_base, 3, pg, ptr);
+  svld1_hor_za32(3, slice_base, 0, pg, ptr);
+  svld1_hor_za32(3, slice_base, 1, pg, ptr);
+  svld1_hor_za32(3, slice_base, 2, pg, ptr);
+  svld1_hor_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_hor_za64(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_hor_za64ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_hor_za64(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za64(0, slice_base, 0, pg, ptr);
+  svld1_hor_za64(0, slice_base, 1, pg, ptr);
+  svld1_hor_za64(1, slice_base, 0, pg, ptr);
+  svld1_hor_za64(1, slice_base, 1, pg, ptr);
+  svld1_hor_za64(2, slice_base, 0, pg, ptr);
+  svld1_hor_za64(2, slice_base, 1, pg, ptr);
+  svld1_hor_za64(3, slice_base, 0, pg, ptr);
+  svld1_hor_za64(3, slice_base, 1, pg, ptr);
+  svld1_hor_za64(4, slice_base, 0, pg, ptr);
+  svld1_hor_za64(4, slice_base, 1, pg, ptr);
+  svld1_hor_za64(5, slice_base, 0, pg, ptr);
+  svld1_hor_za64(5, slice_base, 1, pg, ptr);
+  svld1_hor_za64(6, slice_base, 0, pg, ptr);
+  svld1_hor_za64(6, slice_base, 1, pg, ptr);
+  svld1_hor_za64(7, slice_base, 0, pg, ptr);
+  svld1_hor_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_hor_za128(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z20test_svld1_hor_za128ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_hor_za128(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za128(0, slice_base, 0, pg, ptr);
+  svld1_hor_za128(1, slice_base, 0, pg, ptr);
+  svld1_hor_za128(2, slice_base, 0, pg, ptr);
+  svld1_hor_za128(3, slice_base, 0, pg, ptr);
+  svld1_hor_za128(4, slice_base, 0, pg, ptr);
+  svld1_hor_za128(5, slice_base, 0, pg, ptr);
+  svld1_hor_za128(6, slice_base, 0, pg, ptr);
+  svld1_hor_za128(7, slice_base, 0, pg, ptr);
+  svld1_hor_za128(8, slice_base, 0, pg, ptr);
+  svld1_hor_za128(9, slice_base, 0, pg, ptr);
+  svld1_hor_za128(10, slice_base, 0, pg, ptr);
+  svld1_hor_za128(11, slice_base, 0, pg, ptr);
+  svld1_hor_za128(12, slice_base, 0, pg, ptr);
+  svld1_hor_za128(13, slice_base, 0, pg, ptr);
+  svld1_hor_za128(14, slice_base, 0, pg, ptr);
+  svld1_hor_za128(15, slice_base, 0, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_ver_za8(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z18test_svld1_ver_za8ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CPP-CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CPP-CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CPP-CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CPP-CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CPP-CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CPP-CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CPP-CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_ver_za8(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za8(0, slice_base, 0, pg, ptr);
+  svld1_ver_za8(0, slice_base, 1, pg, ptr);
+  svld1_ver_za8(0, slice_base, 2, pg, ptr);
+  svld1_ver_za8(0, slice_base, 3, pg, ptr);
+  svld1_ver_za8(0, slice_base, 4, pg, ptr);
+  svld1_ver_za8(0, slice_base, 5, pg, ptr);
+  svld1_ver_za8(0, slice_base, 6, pg, ptr);
+  svld1_ver_za8(0, slice_base, 7, pg, ptr);
+  svld1_ver_za8(0, slice_base, 8, pg, ptr);
+  svld1_ver_za8(0, slice_base, 9, pg, ptr);
+  svld1_ver_za8(0, slice_base, 10, pg, ptr);
+  svld1_ver_za8(0, slice_base, 11, pg, ptr);
+  svld1_ver_za8(0, slice_base, 12, pg, ptr);
+  svld1_ver_za8(0, slice_base, 13, pg, ptr);
+  svld1_ver_za8(0, slice_base, 14, pg, ptr);
+  svld1_ver_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_ver_za16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_ver_za16ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_ver_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za16(0, slice_base, 0, pg, ptr);
+  svld1_ver_za16(0, slice_base, 1, pg, ptr);
+  svld1_ver_za16(0, slice_base, 2, pg, ptr);
+  svld1_ver_za16(0, slice_base, 3, pg, ptr);
+  svld1_ver_za16(0, slice_base, 4, pg, ptr);
+  svld1_ver_za16(0, slice_base, 5, pg, ptr);
+  svld1_ver_za16(0, slice_base, 6, pg, ptr);
+  svld1_ver_za16(0, slice_base, 7, pg, ptr);
+  svld1_ver_za16(1, slice_base, 1, pg, ptr);
+  svld1_ver_za16(1, slice_base, 2, pg, ptr);
+  svld1_ver_za16(1, slice_base, 3, pg, ptr);
+  svld1_ver_za16(1, slice_base, 4, pg, ptr);
+  svld1_ver_za16(1, slice_base, 5, pg, ptr);
+  svld1_ver_za16(1, slice_base, 6, pg, ptr);
+  svld1_ver_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_ver_za32(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_ver_za32ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_ver_za32(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za32(0, slice_base, 0, pg, ptr);
+  svld1_ver_za32(0, slice_base, 1, pg, ptr);
+  svld1_ver_za32(0, slice_base, 2, pg, ptr);
+  svld1_ver_za32(0, slice_base, 3, pg, ptr);
+  svld1_ver_za32(1, slice_base, 0, pg, ptr);
+  svld1_ver_za32(1, slice_base, 1, pg, ptr);
+  svld1_ver_za32(1, slice_base, 2, pg, ptr);
+  svld1_ver_za32(1, slice_base, 3, pg, ptr);
+  svld1_ver_za32(2, slice_base, 0, pg, ptr);
+  svld1_ver_za32(2, slice_base, 1, pg, ptr);
+  svld1_ver_za32(2, slice_base, 2, pg, ptr);
+  svld1_ver_za32(2, slice_base, 3, pg, ptr);
+  svld1_ver_za32(3, slice_base, 0, pg, ptr);
+  svld1_ver_za32(3, slice_base, 1, pg, ptr);
+  svld1_ver_za32(3, slice_base, 2, pg, ptr);
+  svld1_ver_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_ver_za64(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svld1_ver_za64ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_ver_za64(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za64(0, slice_base, 0, pg, ptr);
+  svld1_ver_za64(0, slice_base, 1, pg, ptr);
+  svld1_ver_za64(1, slice_base, 0, pg, ptr);
+  svld1_ver_za64(1, slice_base, 1, pg, ptr);
+  svld1_ver_za64(2, slice_base, 0, pg, ptr);
+  svld1_ver_za64(2, slice_base, 1, pg, ptr);
+  svld1_ver_za64(3, slice_base, 0, pg, ptr);
+  svld1_ver_za64(3, slice_base, 1, pg, ptr);
+  svld1_ver_za64(4, slice_base, 0, pg, ptr);
+  svld1_ver_za64(4, slice_base, 1, pg, ptr);
+  svld1_ver_za64(5, slice_base, 0, pg, ptr);
+  svld1_ver_za64(5, slice_base, 1, pg, ptr);
+  svld1_ver_za64(6, slice_base, 0, pg, ptr);
+  svld1_ver_za64(6, slice_base, 1, pg, ptr);
+  svld1_ver_za64(7, slice_base, 0, pg, ptr);
+  svld1_ver_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svld1_ver_za128(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z20test_svld1_ver_za128ju10__SVBool_tPKv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.ld1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svld1_ver_za128(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za128(0, slice_base, 0, pg, ptr);
+  svld1_ver_za128(1, slice_base, 0, pg, ptr);
+  svld1_ver_za128(2, slice_base, 0, pg, ptr);
+  svld1_ver_za128(3, slice_base, 0, pg, ptr);
+  svld1_ver_za128(4, slice_base, 0, pg, ptr);
+  svld1_ver_za128(5, slice_base, 0, pg, ptr);
+  svld1_ver_za128(6, slice_base, 0, pg, ptr);
+  svld1_ver_za128(7, slice_base, 0, pg, ptr);
+  svld1_ver_za128(8, slice_base, 0, pg, ptr);
+  svld1_ver_za128(9, slice_base, 0, pg, ptr);
+  svld1_ver_za128(10, slice_base, 0, pg, ptr);
+  svld1_ver_za128(11, slice_base, 0, pg, ptr);
+  svld1_ver_za128(12, slice_base, 0, pg, ptr);
+  svld1_ver_za128(13, slice_base, 0, pg, ptr);
+  svld1_ver_za128(14, slice_base, 0, pg, ptr);
+  svld1_ver_za128(15, slice_base, 0, pg, ptr);
+}

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_st1.c

+// NOTE: Assertions have been autogenerated by utils/update_cc_test_checks.py
+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefix=CPP-CHECK
+// RUN: %clang_cc1 -no-opaque-pointers -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fallow-half-arguments-and-returns -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme.h>
+
+// CHECK-LABEL: @test_svst1_hor_za8(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z18test_svst1_hor_za8ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CPP-CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CPP-CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CPP-CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CPP-CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CPP-CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CPP-CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CPP-CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_hor_za8(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za8(0, slice_base, 0, pg, ptr);
+  svst1_hor_za8(0, slice_base, 1, pg, ptr);
+  svst1_hor_za8(0, slice_base, 2, pg, ptr);
+  svst1_hor_za8(0, slice_base, 3, pg, ptr);
+  svst1_hor_za8(0, slice_base, 4, pg, ptr);
+  svst1_hor_za8(0, slice_base, 5, pg, ptr);
+  svst1_hor_za8(0, slice_base, 6, pg, ptr);
+  svst1_hor_za8(0, slice_base, 7, pg, ptr);
+  svst1_hor_za8(0, slice_base, 8, pg, ptr);
+  svst1_hor_za8(0, slice_base, 9, pg, ptr);
+  svst1_hor_za8(0, slice_base, 10, pg, ptr);
+  svst1_hor_za8(0, slice_base, 11, pg, ptr);
+  svst1_hor_za8(0, slice_base, 12, pg, ptr);
+  svst1_hor_za8(0, slice_base, 13, pg, ptr);
+  svst1_hor_za8(0, slice_base, 14, pg, ptr);
+  svst1_hor_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_hor_za16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_hor_za16ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.horiz(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_hor_za16(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za16(0, slice_base, 0, pg, ptr);
+  svst1_hor_za16(0, slice_base, 1, pg, ptr);
+  svst1_hor_za16(0, slice_base, 2, pg, ptr);
+  svst1_hor_za16(0, slice_base, 3, pg, ptr);
+  svst1_hor_za16(0, slice_base, 4, pg, ptr);
+  svst1_hor_za16(0, slice_base, 5, pg, ptr);
+  svst1_hor_za16(0, slice_base, 6, pg, ptr);
+  svst1_hor_za16(0, slice_base, 7, pg, ptr);
+  svst1_hor_za16(1, slice_base, 1, pg, ptr);
+  svst1_hor_za16(1, slice_base, 2, pg, ptr);
+  svst1_hor_za16(1, slice_base, 3, pg, ptr);
+  svst1_hor_za16(1, slice_base, 4, pg, ptr);
+  svst1_hor_za16(1, slice_base, 5, pg, ptr);
+  svst1_hor_za16(1, slice_base, 6, pg, ptr);
+  svst1_hor_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_hor_za32(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_hor_za32ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.horiz(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_hor_za32(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za32(0, slice_base, 0, pg, ptr);
+  svst1_hor_za32(0, slice_base, 1, pg, ptr);
+  svst1_hor_za32(0, slice_base, 2, pg, ptr);
+  svst1_hor_za32(0, slice_base, 3, pg, ptr);
+  svst1_hor_za32(1, slice_base, 0, pg, ptr);
+  svst1_hor_za32(1, slice_base, 1, pg, ptr);
+  svst1_hor_za32(1, slice_base, 2, pg, ptr);
+  svst1_hor_za32(1, slice_base, 3, pg, ptr);
+  svst1_hor_za32(2, slice_base, 0, pg, ptr);
+  svst1_hor_za32(2, slice_base, 1, pg, ptr);
+  svst1_hor_za32(2, slice_base, 2, pg, ptr);
+  svst1_hor_za32(2, slice_base, 3, pg, ptr);
+  svst1_hor_za32(3, slice_base, 0, pg, ptr);
+  svst1_hor_za32(3, slice_base, 1, pg, ptr);
+  svst1_hor_za32(3, slice_base, 2, pg, ptr);
+  svst1_hor_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_hor_za64(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_hor_za64ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.horiz(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_hor_za64(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za64(0, slice_base, 0, pg, ptr);
+  svst1_hor_za64(0, slice_base, 1, pg, ptr);
+  svst1_hor_za64(1, slice_base, 0, pg, ptr);
+  svst1_hor_za64(1, slice_base, 1, pg, ptr);
+  svst1_hor_za64(2, slice_base, 0, pg, ptr);
+  svst1_hor_za64(2, slice_base, 1, pg, ptr);
+  svst1_hor_za64(3, slice_base, 0, pg, ptr);
+  svst1_hor_za64(3, slice_base, 1, pg, ptr);
+  svst1_hor_za64(4, slice_base, 0, pg, ptr);
+  svst1_hor_za64(4, slice_base, 1, pg, ptr);
+  svst1_hor_za64(5, slice_base, 0, pg, ptr);
+  svst1_hor_za64(5, slice_base, 1, pg, ptr);
+  svst1_hor_za64(6, slice_base, 0, pg, ptr);
+  svst1_hor_za64(6, slice_base, 1, pg, ptr);
+  svst1_hor_za64(7, slice_base, 0, pg, ptr);
+  svst1_hor_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_hor_za128(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z20test_svst1_hor_za128ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.horiz(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_hor_za128(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za128(0, slice_base, 0, pg, ptr);
+  svst1_hor_za128(1, slice_base, 0, pg, ptr);
+  svst1_hor_za128(2, slice_base, 0, pg, ptr);
+  svst1_hor_za128(3, slice_base, 0, pg, ptr);
+  svst1_hor_za128(4, slice_base, 0, pg, ptr);
+  svst1_hor_za128(5, slice_base, 0, pg, ptr);
+  svst1_hor_za128(6, slice_base, 0, pg, ptr);
+  svst1_hor_za128(7, slice_base, 0, pg, ptr);
+  svst1_hor_za128(8, slice_base, 0, pg, ptr);
+  svst1_hor_za128(9, slice_base, 0, pg, ptr);
+  svst1_hor_za128(10, slice_base, 0, pg, ptr);
+  svst1_hor_za128(11, slice_base, 0, pg, ptr);
+  svst1_hor_za128(12, slice_base, 0, pg, ptr);
+  svst1_hor_za128(13, slice_base, 0, pg, ptr);
+  svst1_hor_za128(14, slice_base, 0, pg, ptr);
+  svst1_hor_za128(15, slice_base, 0, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_ver_za8(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z18test_svst1_ver_za8ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG:%.*]], i8* [[PTR:%.*]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    [[TILESLICE8:%.*]] = add i32 [[SLICE_BASE]], 8
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE8]])
+// CPP-CHECK-NEXT:    [[TILESLICE9:%.*]] = add i32 [[SLICE_BASE]], 9
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE9]])
+// CPP-CHECK-NEXT:    [[TILESLICE10:%.*]] = add i32 [[SLICE_BASE]], 10
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE10]])
+// CPP-CHECK-NEXT:    [[TILESLICE11:%.*]] = add i32 [[SLICE_BASE]], 11
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE11]])
+// CPP-CHECK-NEXT:    [[TILESLICE12:%.*]] = add i32 [[SLICE_BASE]], 12
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE12]])
+// CPP-CHECK-NEXT:    [[TILESLICE13:%.*]] = add i32 [[SLICE_BASE]], 13
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE13]])
+// CPP-CHECK-NEXT:    [[TILESLICE14:%.*]] = add i32 [[SLICE_BASE]], 14
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE14]])
+// CPP-CHECK-NEXT:    [[TILESLICE15:%.*]] = add i32 [[SLICE_BASE]], 15
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], i8* [[PTR]], i64 0, i32 [[TILESLICE15]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_ver_za8(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za8(0, slice_base, 0, pg, ptr);
+  svst1_ver_za8(0, slice_base, 1, pg, ptr);
+  svst1_ver_za8(0, slice_base, 2, pg, ptr);
+  svst1_ver_za8(0, slice_base, 3, pg, ptr);
+  svst1_ver_za8(0, slice_base, 4, pg, ptr);
+  svst1_ver_za8(0, slice_base, 5, pg, ptr);
+  svst1_ver_za8(0, slice_base, 6, pg, ptr);
+  svst1_ver_za8(0, slice_base, 7, pg, ptr);
+  svst1_ver_za8(0, slice_base, 8, pg, ptr);
+  svst1_ver_za8(0, slice_base, 9, pg, ptr);
+  svst1_ver_za8(0, slice_base, 10, pg, ptr);
+  svst1_ver_za8(0, slice_base, 11, pg, ptr);
+  svst1_ver_za8(0, slice_base, 12, pg, ptr);
+  svst1_ver_za8(0, slice_base, 13, pg, ptr);
+  svst1_ver_za8(0, slice_base, 14, pg, ptr);
+  svst1_ver_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_ver_za16(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_ver_za16ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i16*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG:%.*]], i16* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    [[TILESLICE4:%.*]] = add i32 [[SLICE_BASE]], 4
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    [[TILESLICE5:%.*]] = add i32 [[SLICE_BASE]], 5
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    [[TILESLICE6:%.*]] = add i32 [[SLICE_BASE]], 6
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    [[TILESLICE7:%.*]] = add i32 [[SLICE_BASE]], 7
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 0, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE4]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE5]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE6]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1h.vert(<vscale x 16 x i1> [[PG]], i16* [[TMP0]], i64 1, i32 [[TILESLICE7]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_ver_za16(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za16(0, slice_base, 0, pg, ptr);
+  svst1_ver_za16(0, slice_base, 1, pg, ptr);
+  svst1_ver_za16(0, slice_base, 2, pg, ptr);
+  svst1_ver_za16(0, slice_base, 3, pg, ptr);
+  svst1_ver_za16(0, slice_base, 4, pg, ptr);
+  svst1_ver_za16(0, slice_base, 5, pg, ptr);
+  svst1_ver_za16(0, slice_base, 6, pg, ptr);
+  svst1_ver_za16(0, slice_base, 7, pg, ptr);
+  svst1_ver_za16(1, slice_base, 1, pg, ptr);
+  svst1_ver_za16(1, slice_base, 2, pg, ptr);
+  svst1_ver_za16(1, slice_base, 3, pg, ptr);
+  svst1_ver_za16(1, slice_base, 4, pg, ptr);
+  svst1_ver_za16(1, slice_base, 5, pg, ptr);
+  svst1_ver_za16(1, slice_base, 6, pg, ptr);
+  svst1_ver_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_ver_za32(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_ver_za32ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i32*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG:%.*]], i32* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 2
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    [[TILESLICE3:%.*]] = add i32 [[SLICE_BASE]], 3
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 0, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 1, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 2, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE2]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1w.vert(<vscale x 16 x i1> [[PG]], i32* [[TMP0]], i64 3, i32 [[TILESLICE3]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_ver_za32(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za32(0, slice_base, 0, pg, ptr);
+  svst1_ver_za32(0, slice_base, 1, pg, ptr);
+  svst1_ver_za32(0, slice_base, 2, pg, ptr);
+  svst1_ver_za32(0, slice_base, 3, pg, ptr);
+  svst1_ver_za32(1, slice_base, 0, pg, ptr);
+  svst1_ver_za32(1, slice_base, 1, pg, ptr);
+  svst1_ver_za32(1, slice_base, 2, pg, ptr);
+  svst1_ver_za32(1, slice_base, 3, pg, ptr);
+  svst1_ver_za32(2, slice_base, 0, pg, ptr);
+  svst1_ver_za32(2, slice_base, 1, pg, ptr);
+  svst1_ver_za32(2, slice_base, 2, pg, ptr);
+  svst1_ver_za32(2, slice_base, 3, pg, ptr);
+  svst1_ver_za32(3, slice_base, 0, pg, ptr);
+  svst1_ver_za32(3, slice_base, 1, pg, ptr);
+  svst1_ver_za32(3, slice_base, 2, pg, ptr);
+  svst1_ver_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_ver_za64(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z19test_svst1_ver_za64ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i64*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG:%.*]], i64* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 0, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 1, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 2, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 3, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 4, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 5, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 6, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1d.vert(<vscale x 16 x i1> [[PG]], i64* [[TMP0]], i64 7, i32 [[TILESLICE1]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_ver_za64(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za64(0, slice_base, 0, pg, ptr);
+  svst1_ver_za64(0, slice_base, 1, pg, ptr);
+  svst1_ver_za64(1, slice_base, 0, pg, ptr);
+  svst1_ver_za64(1, slice_base, 1, pg, ptr);
+  svst1_ver_za64(2, slice_base, 0, pg, ptr);
+  svst1_ver_za64(2, slice_base, 1, pg, ptr);
+  svst1_ver_za64(3, slice_base, 0, pg, ptr);
+  svst1_ver_za64(3, slice_base, 1, pg, ptr);
+  svst1_ver_za64(4, slice_base, 0, pg, ptr);
+  svst1_ver_za64(4, slice_base, 1, pg, ptr);
+  svst1_ver_za64(5, slice_base, 0, pg, ptr);
+  svst1_ver_za64(5, slice_base, 1, pg, ptr);
+  svst1_ver_za64(6, slice_base, 0, pg, ptr);
+  svst1_ver_za64(6, slice_base, 1, pg, ptr);
+  svst1_ver_za64(7, slice_base, 0, pg, ptr);
+  svst1_ver_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-LABEL: @test_svst1_ver_za128(
+// CHECK-NEXT:  entry:
+// CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:    ret void
+//
+// CPP-CHECK-LABEL: @_Z20test_svst1_ver_za128ju10__SVBool_tPv(
+// CPP-CHECK-NEXT:  entry:
+// CPP-CHECK-NEXT:    [[TMP0:%.*]] = bitcast i8* [[PTR:%.*]] to i128*
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG:%.*]], i128* [[TMP0]], i64 0, i32 [[SLICE_BASE:%.*]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 1, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 2, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 3, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 4, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 5, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 6, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 7, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 8, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 9, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 10, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 11, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 12, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 13, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 14, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    call void @llvm.aarch64.sme.st1q.vert(<vscale x 16 x i1> [[PG]], i128* [[TMP0]], i64 15, i32 [[SLICE_BASE]])
+// CPP-CHECK-NEXT:    ret void
+//
+void test_svst1_ver_za128(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za128(0, slice_base, 0, pg, ptr);
+  svst1_ver_za128(1, slice_base, 0, pg, ptr);
+  svst1_ver_za128(2, slice_base, 0, pg, ptr);
+  svst1_ver_za128(3, slice_base, 0, pg, ptr);
+  svst1_ver_za128(4, slice_base, 0, pg, ptr);
+  svst1_ver_za128(5, slice_base, 0, pg, ptr);
+  svst1_ver_za128(6, slice_base, 0, pg, ptr);
+  svst1_ver_za128(7, slice_base, 0, pg, ptr);
+  svst1_ver_za128(8, slice_base, 0, pg, ptr);
+  svst1_ver_za128(9, slice_base, 0, pg, ptr);
+  svst1_ver_za128(10, slice_base, 0, pg, ptr);
+  svst1_ver_za128(11, slice_base, 0, pg, ptr);
+  svst1_ver_za128(12, slice_base, 0, pg, ptr);
+  svst1_ver_za128(13, slice_base, 0, pg, ptr);
+  svst1_ver_za128(14, slice_base, 0, pg, ptr);
+  svst1_ver_za128(15, slice_base, 0, pg, ptr);
+}

clang/utils/TableGen/SveEmitter.cpp

public:
   }
 
   // Returns the SVETypeFlags value for the given SVEType.
   uint64_t encodeTypeFlags(const SVEType &T);
 
   /// Emit arm_sve.h.
   void createHeader(raw_ostream &o);
 
+  /// Emit arm_sme.h.
+  void createSMEHeader(raw_ostream &o);
+
   /// Emit all the __builtin prototypes and code needed by Sema.
   void createBuiltins(raw_ostream &o);
 
   /// Emit all the information needed to map builtin -> LLVM IR intrinsic.
   void createCodeGenMap(raw_ostream &o);
 
   /// Emit all the range checks for the immediates.
   void createRangeChecks(raw_ostream &o);
 
   /// Create the SVETypeFlags used in CGBuiltins
   void createTypeFlags(raw_ostream &o);
 
   /// Create intrinsic and add it to \p Out
-  void createIntrinsic(Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out);
+  void createIntrinsic(Record *R,
+                       SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out,
+                       bool IsSME = false);
 };
 
 } // end anonymous namespace
 
 
 //===----------------------------------------------------------------------===//
 // Type implementation
 //===----------------------------------------------------------------------===//
case 'Y':
     break;
   case 'U':
     Constant = true;
     Pointer = true;
     ElementBitwidth = Bitwidth = 32;
     NumVectors = 0;
     Signed = true;
     break;
+  case '%':
+    Pointer = true;
+    Void = true;
+    NumVectors = 0;
+    break;
   case 'A':
     Pointer = true;
     ElementBitwidth = Bitwidth = 8;
     NumVectors = 0;
     Signed = true;
     break;
   case 'B':
     Pointer = true;
Intrinsic::Intrinsic(StringRef Name, StringRef Proto, uint64_t MergeTy,
   this->Flags |= Emitter.encodeMergeType(MergeTy);
   if (hasSplat())
     this->Flags |= Emitter.encodeSplatOperand(getSplatIdx());
 }
 
 std::string Intrinsic::getBuiltinTypeStr() {
   std::string S = getReturnType().builtin_str();
   for (unsigned I = 0; I < getNumParams(); ++I)
     S += getParamType(I).builtin_str();
 
   return S;
 }
 
 std::string Intrinsic::replaceTemplatedArgs(std::string Name, TypeSpec TS,
                                             std::string Proto) const {
   std::string Ret = Name;
   while (Ret.find('{') != std::string::npos) {
uint64_t SVEEmitter::encodeTypeFlags(const SVEType &T) {
   case 64:
     return encodeEltType("EltTyInt64");
   default:
     llvm_unreachable("Unhandled integer element bitwidth!");
   }
 }
 
 void SVEEmitter::createIntrinsic(
-    Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out) {
+    Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out, bool IsSME) {
   StringRef Name = R->getValueAsString("Name");
   StringRef Proto = R->getValueAsString("Prototype");
   StringRef Types = R->getValueAsString("Types");
   StringRef Guard = R->getValueAsString("ArchGuard");
   StringRef LLVMName = R->getValueAsString("LLVMIntrinsic");
   uint64_t Merge = R->getValueAsInt("Merge");
   StringRef MergeSuffix = R->getValueAsString("MergeSuffix");
   uint64_t MemEltType = R->getValueAsInt("MemEltType");
   std::vector<Record*> FlagsList = R->getValueAsListOfDefs("Flags");
   std::vector<Record*> ImmCheckList = R->getValueAsListOfDefs("ImmChecks");
 
   int64_t Flags = 0;
   for (auto FlagRec : FlagsList)
     Flags |= FlagRec->getValueAsInt("Value");
 
+  bool SMEFlag = Flags & getEnumValueForFlag("IsSME");
+  if (SMEFlag != IsSME)
+    return;
   // Create a dummy TypeSpec for non-overloaded builtins.
   if (Types.empty()) {
     assert((Flags & getEnumValueForFlag("IsOverloadNone")) &&
            "Expect TypeSpec for overloaded builtin!");
     Types = "i";
   }
 
   // Extract type specs from string
void SVEEmitter::createHeader(raw_ostream &OS) {
 
   OS << "#ifdef __cplusplus\n";
   OS << "} // extern \"C\"\n";
   OS << "#endif\n\n";
   OS << "#endif /*__ARM_FEATURE_SVE */\n\n";
   OS << "#endif /* __ARM_SVE_H */\n";
 }
 
+void SVEEmitter::createSMEHeader(raw_ostream &OS) {
+  OS << "/*===---- arm_sme.h - ARM SME intrinsics "
+        "-----------------------------------===\n"
+        " *\n"
+        " *\n"
+        " * Part of the LLVM Project, under the Apache License v2.0 with LLVM "
+        "Exceptions.\n"
+        " * See https://llvm.org/LICENSE.txt for license information.\n"
+        " * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception\n"
+        " *\n"
+        " *===-----------------------------------------------------------------"
+        "------===\n"
+        " */\n\n";
+
+  OS << "#ifndef __ARM_SME_H\n";
+  OS << "#define __ARM_SME_H\n\n";
+
+  OS << "#if !defined(__ARM_FEATURE_SME)\n";
+  OS << "#error \"SME support not enabled\"\n";
+  OS << "#else\n\n";
+
+  OS << "#include <arm_sve.h> \n\n";
+
+  OS << "#ifdef  __cplusplus\n";
+  OS << "extern \"C\" {\n";
+  OS << "#endif\n\n";
+
+  SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  for (auto *R : RV)
+    createIntrinsic(R, Defs, true);
+
+  // Sort intrinsics in header file by following order/priority similar to SVE:
+  // - Architectural guard
+  // - Class (is intrinsic overloaded or not)
+  // - Intrinsic name
+  std::stable_sort(Defs.begin(), Defs.end(),
+                   [](const std::unique_ptr<Intrinsic> &A,
+                      const std::unique_ptr<Intrinsic> &B) {
+                     auto ToTuple = [](const std::unique_ptr<Intrinsic> &I) {
+                       return std::make_tuple(I->getGuard(),
+                                              (unsigned)I->getClassKind(),
+                                              I->getName());
+                     };
+                     return ToTuple(A) < ToTuple(B);
+                   });
+
+  StringRef InGuard = "";
+  for (auto &I : Defs) {
+    // Emit #endif/#if pair if needed.
+    if (I->getGuard() != InGuard) {
+      if (!InGuard.empty())
+        OS << "#endif  //" << InGuard << "\n";
+      InGuard = I->getGuard();
+      if (!InGuard.empty())
+        OS << "\n#if " << InGuard << "\n";
+    }
+
+    // Actually emit the intrinsic declaration.
+    I->emitIntrinsic(OS);
+  }
+
+  if (!InGuard.empty())
+    OS << "#endif  //" << InGuard << "\n";
+
+  OS << "#ifdef __cplusplus\n";
+  OS << "} // extern \"C\"\n";
+  OS << "#endif\n\n";
+  OS << "#endif /*__ARM_FEATURE_SME */\n\n";
+  OS << "#endif /* __ARM_SME_H */\n";
+}
+
 void SVEEmitter::createBuiltins(raw_ostream &OS) {
   std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
   SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
-  for (auto *R : RV)
+  for (auto *R : RV) {
     createIntrinsic(R, Defs);
+    createIntrinsic(R, Defs, true);
+  }
 
   // The mappings must be sorted based on BuiltinID.
   llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
                       const std::unique_ptr<Intrinsic> &B) {
     return A->getMangledName() < B->getMangledName();
   });
 
   OS << "#ifdef GET_SVE_BUILTINS\n";
for (const ReinterpretTypeInfo &To : Reinterprets)
          << "\", \"n\")\n";
 
   OS << "#endif\n\n";
   }
 
 void SVEEmitter::createCodeGenMap(raw_ostream &OS) {
   std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
   SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
-  for (auto *R : RV)
+  for (auto *R : RV) {
     createIntrinsic(R, Defs);
+    createIntrinsic(R, Defs, true);
+  }
 
   // The mappings must be sorted based on BuiltinID.
   llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
                       const std::unique_ptr<Intrinsic> &B) {
     return A->getMangledName() < B->getMangledName();
   });
 
   OS << "#ifdef GET_SVE_LLVM_INTRINSIC_MAP\n";
else
       OS << "SVEMAP2(" << Builtin << ", " << FlagString << "),\n";
   }
   OS << "#endif\n\n";
 }
 
 void SVEEmitter::createRangeChecks(raw_ostream &OS) {
   std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
   SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
-  for (auto *R : RV)
+  for (auto *R : RV) {
     createIntrinsic(R, Defs);
+    createIntrinsic(R, Defs, true);
+  }
 
   // The mappings must be sorted based on BuiltinID.
   llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
                       const std::unique_ptr<Intrinsic> &B) {
     return A->getMangledName() < B->getMangledName();
   });
 
 
void SVEEmitter::createTypeFlags(raw_ostream &OS) {
   OS << "#endif\n\n";
 }
 
 namespace clang {
 void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createHeader(OS);
 }
 
+void EmitSmeHeader(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).createSMEHeader(OS);
+}
+
 void EmitSveBuiltins(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createBuiltins(OS);
 }
 
 void EmitSveBuiltinCG(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createCodeGenMap(OS);
 }
 
 void EmitSveRangeChecks(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createRangeChecks(OS);
 }
 
 void EmitSveTypeFlags(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createTypeFlags(OS);
 }
 
 } // End namespace clang

clang/utils/TableGen/TableGen.cpp

enum ActionType {
   GenArmMveBuiltinSema,
   GenArmMveBuiltinCG,
   GenArmMveBuiltinAliases,
   GenArmSveHeader,
   GenArmSveBuiltins,
   GenArmSveBuiltinCG,
   GenArmSveTypeFlags,
   GenArmSveRangeChecks,
+  GenArmSmeHeader,
   GenArmCdeHeader,
   GenArmCdeBuiltinDef,
   GenArmCdeBuiltinSema,
   GenArmCdeBuiltinCG,
   GenArmCdeBuiltinAliases,
   GenRISCVVectorHeader,
   GenRISCVVectorBuiltins,
   GenRISCVVectorBuiltinCG,
cl::values(
         clEnumValN(GenArmSveBuiltins, "gen-arm-sve-builtins",
                    "Generate arm_sve_builtins.inc for clang"),
         clEnumValN(GenArmSveBuiltinCG, "gen-arm-sve-builtin-codegen",
                    "Generate arm_sve_builtin_cg_map.inc for clang"),
         clEnumValN(GenArmSveTypeFlags, "gen-arm-sve-typeflags",
                    "Generate arm_sve_typeflags.inc for clang"),
         clEnumValN(GenArmSveRangeChecks, "gen-arm-sve-sema-rangechecks",
                    "Generate arm_sve_sema_rangechecks.inc for clang"),
+        clEnumValN(GenArmSmeHeader, "gen-arm-sme-header",
+                   "Generate arm_sme.h for clang"),
         clEnumValN(GenArmMveHeader, "gen-arm-mve-header",
                    "Generate arm_mve.h for clang"),
         clEnumValN(GenArmMveBuiltinDef, "gen-arm-mve-builtin-def",
                    "Generate ARM MVE builtin definitions for clang"),
         clEnumValN(GenArmMveBuiltinSema, "gen-arm-mve-builtin-sema",
                    "Generate ARM MVE builtin sema checks for clang"),
         clEnumValN(GenArmMveBuiltinCG, "gen-arm-mve-builtin-codegen",
                    "Generate ARM MVE builtin code-generator for clang"),
case GenArmSveBuiltinCG:
     EmitSveBuiltinCG(Records, OS);
     break;
   case GenArmSveTypeFlags:
     EmitSveTypeFlags(Records, OS);
     break;
   case GenArmSveRangeChecks:
     EmitSveRangeChecks(Records, OS);
     break;
+  case GenArmSmeHeader:
+    EmitSmeHeader(Records, OS);
+    break;
   case GenArmCdeHeader:
     EmitCdeHeader(Records, OS);
     break;
   case GenArmCdeBuiltinDef:
     EmitCdeBuiltinDef(Records, OS);
     break;
   case GenArmCdeBuiltinSema:
     EmitCdeBuiltinSema(Records, OS);

clang/utils/TableGen/TableGenBackends.h

 void EmitNeonTest(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitSveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveTypeFlags(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveRangeChecks(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
+void EmitSmeHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+
 void EmitMveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinDef(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinSema(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinAliases(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitRVVHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitRVVBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);