[Clang][AArch64][SME] Add vector load/store (ld1/st1) intrinsics

[bryanpkc]

Summary

[Clang][AArch64][SME] Add vector load/store (ld1/st1) intrinsics

This patch adds support for the following SME ACLE intrinsics (as defined
in https://arm-software.github.io/acle/main/acle.html):

• svld1_hor_za8 // also for _za16, _za32, _za64 and _za128
• svld1_hor_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svld1_ver_za8 // also for _za16, _za32, _za64 and _za128
• svld1_ver_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svst1_hor_za8 // also for _za16, _za32, _za64 and _za128
• svst1_hor_vnum_za8 // also for _za16, _za32, _za64 and _za128
• svst1_ver_za8 // also for _za16, _za32, _za64 and _za128
• svst1_ver_vnum_za8 // also for _za16, _za32, _za64 and _za128

SveEmitter.cpp is extended to generate arm_sme.h (currently named
arm_sme_draft_spec_subject_to_change.h) and other SME definitions from
arm_sme.td, which is modeled after arm_sve.td. Common TableGen definitions
are moved into arm_sve_sme_incl.td.

Co-authored-by: Sagar Kulkarni <[e-mail]>

[sdesmalen]

Hi @sagarkulkarni19, this patch will have to wait until the ABI is implemented so that the builtins can be decorated with the attributes. A first patch proposing these attributes is in D127762. LLVM patches that use these attributes to implement the ABI will follow soon. We could add you as reviewer to these patches if you'd like to help us with that?

clang/include/clang/Basic/arm_sve.td

16

Is there value in having both IsSME and IsSMELoadStore?

clang/lib/Basic/Targets/AArch64.cpp

416

When this macro is non-zero, it suggests that the compiler implements the full SME ACLE. That is currently not yet the case, so until then we should leave this macro undefined.

[sagarkulkarni19]

In D127910#3589068, @sdesmalen wrote:

Hi @sagarkulkarni19, this patch will have to wait until the ABI is implemented so that the builtins can be decorated with the attributes. A first patch proposing these attributes is in D127762. LLVM patches that use these attributes to implement the ABI will follow soon. We could add you as reviewer to these patches if you'd like to help us with that?

Hi @sdesmalen, sounds good. I can update this patch accordingly when the attributes patches are in. Sure, please add me as a reviewer to those patches. We are porting much of our downstream C intrinsics implementation to support the ACLE instead and can contribute this effort upstream. I have also created an article on discourse (https://discourse.llvm.org/t/clang-sme-acle-intrinsics/63223) for further discussion on this.

[sagarkulkarni19]

clang/include/clang/Basic/arm_sve.td

16

IsSME flag is checked in the SveEmitter.cpp : createSMEHeader(...) to generate arm_sme.h with only the SME intrinsics, whereas IsSMELoadStore is used to correctly CodeGen (CGBuiltins.cpp) load and store intrinsics.

clang/lib/Basic/Targets/AArch64.cpp

416

Okay makes sense. But until all SME ACLE is implemented, do we need some sort of guard in the meantime while the SME intrinsics that are implemented incrementally?

[sdesmalen]

clang/lib/Basic/Targets/AArch64.cpp

416

The tests need to have manual -D__ARM_FEATURE_SME in the RUN lines. Once the feature is implemented and we add the definition automatically when +sme is specified, we update the tests. See for example D81725 where we did this for SVE.

[sagarkulkarni19]

Updated testcases and also added the vnum variant of the ld1 and st1 intrinsics.

[sagarkulkarni19]

clang/lib/Basic/Targets/AArch64.cpp

416

Thanks. That makes sense, now the macro is undefined and I have updated the testcases accordingly.

[sagarkulkarni19]

• Support Opaque pointers
• Correct predicate types for the intrinsics.
• Decorate intrinsics with SME attributes.

[sagarkulkarni19]

Update testcases by adding "arm_streaming" attribute to the callee.

[david-arm]

Hi @sagarkulkarni19, thank you for working on the ACLE builtins for SME! I've had a look through and I have a few comments, mostly around how the code is structured. It would be good if you could try to separate SVE from SME in this implementation, in the same way that NEON and SVE are distinct. It's possible to do this whilst reusing much of the code in SveEmitter.cpp.

clang/include/clang/Basic/arm_sve.td

16

You don't need the IsSME flag either because in CodeGenFunction::EmitAArch64BuiltinExpr you can do exactly the same thing as SVE, i.e. something like

if (BuiltinID >= AArch64::FirstSMEBuiltin &&
    BuiltinID <= AArch64::LastSMEBuiltin)
  return EmitAArch64SMEBuiltinExpr(BuiltinID, E);

17

We don't need these new flags 'IsSMELd1' and 'IsSMESt1'. Please can you reuse the existing IsLoad and IsStore flags?

301

SME is really a distinct architecture and so I think it should live in it's own arm_sme.td file in the same way that we have arm_neon.td and arm_sve.td. It's possible to do this and still reuse the SveEmitter.cpp code. If you look at SveEmitter.cpp you'll see these functions:

void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createHeader(OS);
}

void EmitSveBuiltins(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createBuiltins(OS);
}

void EmitSveBuiltinCG(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createCodeGenMap(OS);
}

void EmitSveRangeChecks(RecordKeeper &Records, raw_ostream &OS) {
  SVEEmitter(Records).createRangeChecks(OS);
}

It would be good to add similar ones for SME, i.e. EmitSmeRangeChecks, etc.

clang/lib/CodeGen/CGBuiltin.cpp

9580

I would prefer this to be handled inside it's own EmitAArch64SMEBuiltinExpr, since it's a separate architecture.

clang/utils/TableGen/SveEmitter.cpp

870

I would prefer this to be done more precisely via separate attribute flags, i.e. in the .td file decorate each ACLE builtin with something like IsShared, IsStreaming, IsStreamingCompatible, etc. otherwise you'll end up needing loads of flags for all different possible combinations. That way you can do:

if (this->Flags & Emitter.getEnumValueForFlag("IsStreaming"))
  this->SMEAttributes += "arm_streaming";

etc.

1046

If you move the builtins to their own file in arm_sme.td and emit the records using interfaces like EmitSmeBuiltins, etc. then you already know they are SME builtins and so don't need the flag.

[david-arm]

Hi @sagarkulkarni19, just a gentle ping to see if you are still planning to do more work on this patch?

[bryanpkc]

In D127910#3885699, @david-arm wrote:

Hi @sagarkulkarni19, just a gentle ping to see if you are still planning to do more work on this patch?

@david-arm we are going to update this patch very soon. Sorry for the delays.

[bryanpkc]

Updated the patch according to review comments which suggest to treat SME as a different architecture.

[bryanpkc]

@david-arm, I have moved all the SME definitions into a new file, arm_sme.td. I have moved the common definitions into another file, arm_sve_sme_incl.td, which will be included by both arm_sve.td and arm_sme.td. SveEmitter has been updated to incorporate your suggestions.

[bryanpkc]

Removed some diffs that weren't necessary.

[bryanpkc]

Removed some more unnecessary lines.

[bryanpkc]

Updated the patch to define the __ARM_FEATURE_SME macro when the feature is enabled.

[bryanpkc]

Minor clean-up. Sorry for the noise.

[david-arm]

Hi @bryanpkc, this is looking a lot better now and thanks for addressing the comments! I've not reviewed all of the patch yet, but I do have a few more comments. The most important ones are about performing immediate range checks for the builtins and not declaring the __ARM_FEATURE_SME yet.

clang/include/clang/Basic/TargetBuiltins.h

316

I actually don't think you need to add this class - we should be able to just reuse the existing SVETypeFlags structure. I think this is fine because you've commonised the flags between SME and SVE.

clang/include/clang/Basic/arm_sme.td

22

I think all the load and store instructions need immediate checks for the tile and slice_offset here such as:

[ImmCheck<0, ImmCheck0>, ImmCheck<2, ImmCheck0_15>]

for SVLD1_HOR_ZA8 and the others. It's mentioned in the ACLE - https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics:

15.4.3.1 Common Rules

...
Every argument named tile, slice_offset or tile_mask must be an integer constant expression in the range of the underlying instruction.

clang/include/clang/Basic/arm_sve_sme_incl.td

127

Please can you add a comment here for the new Prototype modifier you added - '%'?

clang/lib/Basic/Targets/AArch64.cpp

494

Can you remove this please? We can't really set this macro until the SME ABI and ACLE is feature complete.

clang/lib/CodeGen/CGBuiltin.cpp

9423

I think that instead of this switch statement you should just be able to write something like:

Ops[3] = EmitSVEPredicateCast(Ops[3],  getSVEVectorForElementType(SVEBuiltinMemEltTy(TypeFlags)))

9465

I think you can just call

CGM.getIntrinsic(Intrinsic::aarch64_sme_cntsb)

9467

Again, I think you can just do

Builder.CreateCall(StreamingVectorLength)

9476

nit: Function *F = CGM.getIntrinsic(IntID);

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_int_const_expr_error.c

5 ↗

(On Diff #492586)

Once you've added the immediate range checks for the loads and stores it would be good add checks here for immediates outside the range for each instruction.

clang/utils/TableGen/SveEmitter.cpp

1591

If you reuse the existing SVETypeFlags rather than create a new SMETypeFlags then you only need the LLVM_GET_SME_IMMCHECKTYPES bit.

[dmgreen]

clang/utils/TableGen/SveEmitter.cpp

1434

We have been changing how the existing SVE and NEON instrinsics work a little. There are some details in https://reviews.llvm.org/D131064. The short version is it is best to not rely upon preprocessor macros, and instead define the intrinsics so that they can be used if the right target features are available. This allows us to do things like this below, even without a -march that supports sme, and have them callable at runtime under the right situations. We should be doing the same for SME.

__attribute__((target("+sme")))
void sme_func() {
  somesmeintrinsics();
}

[bryanpkc]

clang/include/clang/Basic/TargetBuiltins.h

316

Thanks for the suggestion. I have removed this class.

clang/include/clang/Basic/arm_sme.td

22

Done.

clang/include/clang/Basic/arm_sve_sme_incl.td

127

A comment for '%' already exists on line 103.

clang/lib/Basic/Targets/AArch64.cpp

494

OK. Could you educate me what else is needed for SME ABI and ACLE to be feature-complete? How can I help?

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_int_const_expr_error.c

5 ↗

(On Diff #492586)

Done, thanks for the suggestion. I have also moved these tests into clang/test/Sema/arm-sme-intrinsics/.

clang/utils/TableGen/SveEmitter.cpp

1434

I have refactored the SME intrinsics in a similar fashion. Could you confirm if I did it correctly?

[bryanpkc]

Rebased on trunk and addressed review comments.

[david-arm]

Thanks a lot for making all the changes @bryanpkc - it's looking really good now! I just have a few minor comments/suggestions and then I think it looks good to go.

clang/include/clang/Basic/arm_sme.td

23

This is just a suggestion, but you could reduce the lines of code here if you want by creating a multiclass that creates both the horizontal and vertical variants for each size, i.e. something like

multiclass MyMultiClass<..> {
  def NAME # _H : MInst<...>
  def NAME # _V : MInst<...>
}

defm SVLD1_ZA8 : MyMultiClass<...>;

or whatever naming scheme you prefer, and same for the stores. Feel free to ignore this suggestion though if it doesn't help you!

clang/lib/Basic/Targets/AArch64.cpp

494

It should have complete support for the SME ABI and ACLE in terms of supporting the C/C++ level attributes as described here - https://arm-software.github.io/acle/main/acle.html#sme-language-extensions-and-intrinsics. For example, the compiler should support cases where a normal function calls a arm_streaming function and sets up the state correctly, etc. You can see @sdesmalen's patch to add the clang-side attributes here D127762. There should also be full support for all of the SME ACLE builtins.

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_imm.cpp

4

I think you can remove the -target-feature +sve flags from the RUN lines because +sme should imply that.

17

These tests are great! Thanks for doing this. Could you also add the _vnum variants too?

[sdesmalen]

clang/lib/Headers/CMakeLists.txt

344

The ACLE specification is still in a draft (ALP) state, which means there may still be subject to significant changes. To avoid users from using this implementation with the expectation that their code is compliant going forward, it would be good to rename the header file to something that makes it very clear this feature is not yet ready to use. I'm thinking of something like arm_sme_draft_spec_subject_to_change.h. When the specification goes out of draft, we can rename it to arm_sme.h. Could you rename the file for now?

[bryanpkc]

clang/lib/Headers/CMakeLists.txt

344

Would something shorter like arm_sme_draft.h or arm_sme_experimental.h suffice?

[bryanpkc]

Addressed review comments.

[bryanpkc]

clang/include/clang/Basic/arm_sme.td

23

Thanks for the suggestion. Refactored.

clang/lib/Headers/CMakeLists.txt

344

Renamed to arm_sme_experimental.h.

[bryanpkc]

Fixed minor bugs in the previous upload.

[dmgreen]

clang/utils/TableGen/SveEmitter.cpp

1434

It looks OK - as far as I can see. It might be worth adding a test for it? But otherwise it looks good. Thanks.

[sdesmalen]

clang/lib/Headers/CMakeLists.txt

344

While arm_sme_experimental.h is indeed shorter, it should be unequivocally clear to the user that they shouldn't rely on the function prototypes defined in this header file yet because the specification itself is not finalised. I think that adding the draft_spec_subject_to_change to the name makes that more clear than experimental, as the latter might suggest that the support is not yet entirely stable or complete. There probably isn't that much to gain from making the user experience better by using a shorter name, if the whole point is to prevent people from using it for any production code. So from that perspective, I still have a slight preference for arm_sme_draft_spec_subject_to_change.h.

[bryanpkc]

Addressed review comments.

[bryanpkc]

clang/lib/Headers/CMakeLists.txt

344

Renamed as suggested.

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_imm.cpp

4

If I understand llvm/lib/Target/AArch64/AArch64.td and clang/lib/Basic/Targets/AArch64.cpp correctly, -target-feature +sme does not imply -target-feature +sve. I can remove the -D__ARM_FEATURE_SME though.

On the other hand, -march=armv8+sme will imply -target-feature +sve, which I have implemented in D142702.

clang/utils/TableGen/SveEmitter.cpp

1434

Added a test case.

[kmclaughlin]

clang/lib/CodeGen/CGBuiltin.cpp

8950

Is it necessary to add an EltTypeBool128? I think the EmitSVEPredicateCast call in EmitSMELd1St1 is creating a vector based on the memory element type and not the predicate type?

[bryanpkc]

clang/lib/CodeGen/CGBuiltin.cpp

8950

You are right, EltTypeBool128 is not used right now; I can remove it. When we started working on the pre-ACLE intrinsics in our downstream compiler, we were planning to add something like svptrue_b128, even though the hardware PTRUE instruction doesn't support the Q size specifier. It is still not clear to me how the ACLE wants users to create a predicate vector for a call to a _za128 intrinsic.

[kmclaughlin]

Thank you for checking and removing EltTypeBool128. I think you have addressed all of the other comments on this patch too, so it looks good to me!

Please can you update the commit message before landing, to change the reference to arm_sme_experimental.h with arm_sme_draft_spec_subject_to_change.h?

clang/lib/CodeGen/CGBuiltin.cpp

8950

I think the reason why svptrue_b128 wasn't added is because there is no ptrue.q instruction as you say, and it is possible to use any of the other forms (.b, .h, etc) with the 128 bit instructions since they only require every 16th lane.
It would be possible for the user to create a 128 bit predicate with either svunpk or svzip of svbool_t with pfalse where necessary. Otherwise using the other forms, e.g svptrue_b64(), will achieve the same result and require fewer instructions.

[bryanpkc]

Removed EltTypeBool128 as suggested.

[bryanpkc]

Moved test case acle_target_sme.c into clang/test/Sema/aarch64-sme-intrinsics/.

[bryanpkc]

Fix some bugs in the range checks of immediate operands.

[sdesmalen]

clang/lib/Basic/Targets/AArch64.cpp

720–721

Why did you remove this?

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

17

Hi @bryanpkc, would you be happy to remove the dependence on the attributes patch for now, so that we can move forward to review/land this patch series?

I'm thinking of doing something like:

// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES ....
...

#ifdef DISABLE_SME_ATTRIBUTES
#define ARM_STREAMING_ATTR
#else
#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
#endif

...

ARM_STREAMING_ATTR void test_svld1_hor_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
  svld1_hor_za16(0, slice_base, 0, pg, ptr);
  svld1_hor_za16(1, slice_base, 7, pg, ptr);
}

With that the tests all pass, and when the attribute patches have landed we can just remove the -DDISABLE_SME_ATTRIBUTES from the RUN lines.

[bryanpkc]

clang/lib/Basic/Targets/AArch64.cpp

720–721

The Feature == "+sme" case is also handled below on line 782. Perhaps we should delete this block and add HasFullFP16 below.

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

17

Sounds good, I will give that a shot. If we can unblock this patch, I will also rebase and update the rest of the patch series. Thanks!

[bryanpkc]

@sdesmalen How does the latest version look to you? If you approve, I will land this and finish updating the rest of the patch series. One of the pre-merge builds failed for an unrelated reason, but other jobs were successful.

[sdesmalen]

Thanks @bryanpkc this patch looks good to me now. I'll make some time to review the other patches in the series as well after you update them.

clang/include/clang/Basic/BuiltinsAArch64.def

 
 TARGET_HEADER_BUILTIN(__readx18byte,  "UcULi", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(__readx18word,  "UsULi", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(__readx18dword, "ULiULi", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(__readx18qword, "ULLiULi", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 
 #undef BUILTIN
 #undef LANGBUILTIN
+#undef TARGET_BUILTIN
 #undef TARGET_HEADER_BUILTIN

clang/include/clang/Basic/BuiltinsARM.def

 TARGET_HEADER_BUILTIN(_InterlockedDecrement_nf,    "NiNiD*",   "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement_rel,   "NiNiD*",   "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement64_acq, "LLiLLiD*", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement64_nf,  "LLiLLiD*", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 TARGET_HEADER_BUILTIN(_InterlockedDecrement64_rel, "LLiLLiD*", "nh", INTRIN_H, ALL_MS_LANGUAGES, "")
 
 #undef BUILTIN
 #undef LANGBUILTIN
+#undef TARGET_BUILTIN
 #undef TARGET_HEADER_BUILTIN

clang/include/clang/Basic/BuiltinsNEON.def

 // The format of this database matches clang/Basic/Builtins.def.
 
 #define GET_NEON_BUILTINS
 #include "clang/Basic/arm_neon.inc"
 #include "clang/Basic/arm_fp16.inc"
 #undef GET_NEON_BUILTINS
 
 #undef BUILTIN
+#undef TARGET_BUILTIN

clang/include/clang/Basic/BuiltinsSME.def

+//===--- BuiltinsSME.def - SME Builtin function database --------*- C++ -*-===//
+//
+//  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//  See https://llvm.org/LICENSE.txt for license information.
+//  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the SME-specific builtin function database.  Users of
+// this file must define the BUILTIN macro to make use of this information.
+//
+//===----------------------------------------------------------------------===//
+
+// The format of this database matches clang/Basic/Builtins.def.
+
+#define GET_SME_BUILTINS
+#include "clang/Basic/arm_sme_builtins.inc"
+#undef GET_SME_BUILTINS
+
+#undef BUILTIN
+#undef TARGET_BUILTIN

clang/include/clang/Basic/CMakeLists.txt

clang_tablegen(arm_sve_builtin_cg.inc -gen-arm-sve-builtin-codegen
   SOURCE arm_sve.td
   TARGET ClangARMSveBuiltinCG)
 clang_tablegen(arm_sve_typeflags.inc -gen-arm-sve-typeflags
   SOURCE arm_sve.td
   TARGET ClangARMSveTypeFlags)
 clang_tablegen(arm_sve_sema_rangechecks.inc -gen-arm-sve-sema-rangechecks
   SOURCE arm_sve.td
   TARGET ClangARMSveSemaRangeChecks)
+clang_tablegen(arm_sme_builtins.inc -gen-arm-sme-builtins
+  SOURCE arm_sme.td
+  TARGET ClangARMSmeBuiltins)
+clang_tablegen(arm_sme_builtin_cg.inc -gen-arm-sme-builtin-codegen
+  SOURCE arm_sme.td
+  TARGET ClangARMSmeBuiltinCG)
+clang_tablegen(arm_sme_sema_rangechecks.inc -gen-arm-sme-sema-rangechecks
+  SOURCE arm_sme.td
+  TARGET ClangARMSmeSemaRangeChecks)
 clang_tablegen(arm_cde_builtins.inc -gen-arm-cde-builtin-def
   SOURCE arm_cde.td
   TARGET ClangARMCdeBuiltinsDef)
 clang_tablegen(arm_cde_builtin_cg.inc -gen-arm-cde-builtin-codegen
   SOURCE arm_cde.td
   TARGET ClangARMCdeBuiltinCG)
 clang_tablegen(arm_cde_builtin_sema.inc -gen-arm-cde-builtin-sema
   SOURCE arm_cde.td

clang/include/clang/Basic/TargetBuiltins.h

#include "clang/Basic/BuiltinsARM.def"
       LastTSBuiltin
     };
   }
 
   namespace SVE {
   enum {
     LastNEONBuiltin = NEON::FirstTSBuiltin - 1,
 #define BUILTIN(ID, TYPE, ATTRS) BI##ID,
+#define TARGET_BUILTIN(ID, TYPE, ATTRS, FEATURE) BI##ID,
 #include "clang/Basic/BuiltinsSVE.def"
     FirstTSBuiltin,
   };
   }
 
+  namespace SME {
+  enum {
+    LastSVEBuiltin = SVE::FirstTSBuiltin - 1,
+#define BUILTIN(ID, TYPE, ATTRS) BI##ID,
+#define TARGET_BUILTIN(ID, TYPE, ATTRS, FEATURE) BI##ID,
+#include "clang/Basic/BuiltinsSME.def"
+    FirstTSBuiltin,
+  };
+  }
+
   /// AArch64 builtins
   namespace AArch64 {
   enum {
     LastTIBuiltin = clang::Builtin::FirstTSBuiltin - 1,
     LastNEONBuiltin = NEON::FirstTSBuiltin - 1,
     FirstSVEBuiltin = NEON::FirstTSBuiltin,
     LastSVEBuiltin = SVE::FirstTSBuiltin - 1,
+    FirstSMEBuiltin = SVE::FirstTSBuiltin,
+    LastSMEBuiltin = SME::FirstTSBuiltin - 1,
   #define BUILTIN(ID, TYPE, ATTRS) BI##ID,
   #include "clang/Basic/BuiltinsAArch64.def"
     LastTSBuiltin
   };
   }
 
   /// BPF builtins
   namespace BPF {
#undef LLVM_GET_SVE_IMMCHECKTYPES
     bool isTupleGet() const { return Flags & IsTupleGet; }
     bool isTupleSet() const { return Flags & IsTupleSet; }
 
     uint64_t getBits() const { return Flags; }
     bool isFlagSet(uint64_t Flag) const { return Flags & Flag; }
   };
 
   /// Hexagon builtins
   namespace Hexagon {
     enum {
         LastTIBuiltin = clang::Builtin::FirstTSBuiltin-1,
 #define BUILTIN(ID, TYPE, ATTRS) BI##ID,
 #include "clang/Basic/BuiltinsHexagon.def"
         LastTSBuiltin
     };
   }
 

clang/include/clang/Basic/arm_sme.td

+//===--- arm_sme.td - ARM SME compiler interface ------------------------===//
+//
+//  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//  See https://llvm.org/LICENSE.txt for license information.
+//  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file defines the TableGen definitions from which the ARM SME header
+//  file will be generated.  See:
+//
+//      https://developer.arm.com/architectures/system-architectures/software-standards/acle
+//
+//===----------------------------------------------------------------------===//
+
+include "arm_sve_sme_incl.td"
+
+////////////////////////////////////////////////////////////////////////////////
+// Loads
+
+multiclass ZALoad<string n_suffix, string t, string i_prefix, list<ImmCheck> ch> {
+  let TargetGuard = "sme" in {
+    def NAME # _H : MInst<"svld1_hor_" # n_suffix, "vimiPQ", t,
+                          [IsLoad, IsOverloadNone, IsStreaming, IsSharedZA],
+                          MemEltTyDefault, i_prefix # "_horiz", ch>;
+
+    def NAME # _H_VNUM : MInst<"svld1_hor_vnum_" # n_suffix, "vimiPQl", t,
+                               [IsLoad, IsOverloadNone, IsStreaming, IsSharedZA],
+                               MemEltTyDefault, i_prefix # "_horiz", ch>;
+
+    def NAME # _V : MInst<"svld1_ver_" # n_suffix, "vimiPQ", t,
+                          [IsLoad, IsOverloadNone, IsStreaming, IsSharedZA],
+                          MemEltTyDefault, i_prefix # "_vert", ch>;
+
+    def NAME # _V_VNUM : MInst<"svld1_ver_vnum_" # n_suffix, "vimiPQl", t,
+                               [IsLoad, IsOverloadNone, IsStreaming, IsSharedZA],
+                               MemEltTyDefault, i_prefix # "_vert", ch>;
+  }
+}
+
+defm SVLD1_ZA8 : ZALoad<"za8", "c", "aarch64_sme_ld1b", [ImmCheck<0, ImmCheck0_0>, ImmCheck<2, ImmCheck0_15>]>;
+defm SVLD1_ZA16 : ZALoad<"za16", "s", "aarch64_sme_ld1h", [ImmCheck<0, ImmCheck0_1>, ImmCheck<2, ImmCheck0_7>]>;
+defm SVLD1_ZA32 : ZALoad<"za32", "i", "aarch64_sme_ld1w", [ImmCheck<0, ImmCheck0_3>, ImmCheck<2, ImmCheck0_3>]>;
+defm SVLD1_ZA64 : ZALoad<"za64", "l", "aarch64_sme_ld1d", [ImmCheck<0, ImmCheck0_7>, ImmCheck<2, ImmCheck0_1>]>;
+defm SVLD1_ZA128 : ZALoad<"za128", "q", "aarch64_sme_ld1q", [ImmCheck<0, ImmCheck0_15>, ImmCheck<2, ImmCheck0_0>]>;
+
+////////////////////////////////////////////////////////////////////////////////
+// Stores
+
+multiclass ZAStore<string n_suffix, string t, string i_prefix, list<ImmCheck> ch> {
+  let TargetGuard = "sme" in {
+    def NAME # _H : MInst<"svst1_hor_" # n_suffix, "vimiP%", t,
+                          [IsStore, IsOverloadNone, IsStreaming, IsSharedZA, IsPreservesZA],
+                          MemEltTyDefault, i_prefix # "_horiz", ch>;
+
+    def NAME # _H_VNUM : MInst<"svst1_hor_vnum_" # n_suffix, "vimiP%l", t,
+                               [IsStore, IsOverloadNone, IsStreaming, IsSharedZA, IsPreservesZA],
+                               MemEltTyDefault, i_prefix # "_horiz", ch>;
+
+    def NAME # _V : MInst<"svst1_ver_" # n_suffix, "vimiP%", t,
+                          [IsStore, IsOverloadNone, IsStreaming, IsSharedZA, IsPreservesZA],
+                          MemEltTyDefault, i_prefix # "_vert", ch>;
+
+    def NAME # _V_VNUM : MInst<"svst1_ver_vnum_" # n_suffix, "vimiP%l", t,
+                               [IsStore, IsOverloadNone, IsStreaming, IsSharedZA, IsPreservesZA],
+                               MemEltTyDefault, i_prefix # "_vert", ch>;
+  }
+}
+
+defm SVST1_ZA8 : ZAStore<"za8", "c", "aarch64_sme_st1b", [ImmCheck<0, ImmCheck0_0>, ImmCheck<2, ImmCheck0_15>]>;
+defm SVST1_ZA16 : ZAStore<"za16", "s", "aarch64_sme_st1h", [ImmCheck<0, ImmCheck0_1>, ImmCheck<2, ImmCheck0_7>]>;
+defm SVST1_ZA32 : ZAStore<"za32", "i", "aarch64_sme_st1w", [ImmCheck<0, ImmCheck0_3>, ImmCheck<2, ImmCheck0_3>]>;
+defm SVST1_ZA64 : ZAStore<"za64", "l", "aarch64_sme_st1d", [ImmCheck<0, ImmCheck0_7>, ImmCheck<2, ImmCheck0_1>]>;
+defm SVST1_ZA128 : ZAStore<"za128", "q", "aarch64_sme_st1q", [ImmCheck<0, ImmCheck0_15>, ImmCheck<2, ImmCheck0_0>]>;

clang/include/clang/Basic/arm_sve.td

 //===--- arm_sve.td - ARM SVE compiler interface ------------------------===//
 //
 //  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 //  See https://llvm.org/LICENSE.txt for license information.
 //  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 //  This file defines the TableGen definitions from which the ARM SVE header
 //  file will be generated.  See:
 //
 //      https://developer.arm.com/architectures/system-architectures/software-standards/acle
 //
 //===----------------------------------------------------------------------===//
 
-//===----------------------------------------------------------------------===//
-// Instruction definitions
-//===----------------------------------------------------------------------===//
-// Every intrinsic subclasses "Inst". An intrinsic has a name, a prototype and
-// a sequence of typespecs.
-//
-// The name is the base name of the intrinsic, for example "svld1". This is
-// then mangled by the tblgen backend to add type information ("svld1_s16").
-//
-// A typespec is a sequence of uppercase characters (modifiers) followed by one
-// lowercase character. A typespec encodes a particular "base type" of the
-// intrinsic.
-//
-// An example typespec is "Us" - unsigned short - svuint16_t. The available
-// typespec codes are given below.
-//
-// The string given to an Inst class is a sequence of typespecs. The intrinsic
-// is instantiated for every typespec in the sequence. For example "sdUsUd".
-//
-// The prototype is a string that defines the return type of the intrinsic
-// and the type of each argument. The return type and every argument gets a
-// "modifier" that can change in some way the "base type" of the intrinsic.
-//
-// The modifier 'd' means "default" and does not modify the base type in any
-// way. The available modifiers are given below.
-//
-// Typespecs
-// ---------
-// c: char
-// s: short
-// i: int
-// l: long
-// f: float
-// h: half-float
-// d: double
-// b: bfloat
-
-// Typespec modifiers
-// ------------------
-// P: boolean
-// U: unsigned
-
-// Prototype modifiers
-// -------------------
-// prototype: return (arg, arg, ...)
-//
-// 2,3,4: array of default vectors
-// v: void
-// x: vector of signed integers
-// u: vector of unsigned integers
-// d: default
-// c: const pointer type
-// P: predicate type
-// s: scalar of element type
-// a: scalar of element type (splat to vector type)
-// R: scalar of 1/2 width element type (splat to vector type)
-// r: scalar of 1/4 width element type (splat to vector type)
-// @: unsigned scalar of 1/4 width element type (splat to vector type)
-// e: 1/2 width unsigned elements, 2x element count
-// b: 1/4 width unsigned elements, 4x element count
-// h: 1/2 width elements, 2x element count
-// q: 1/4 width elements, 4x element count
-// o: 4x width elements, 1/4 element count
-//
-// w: vector of element type promoted to 64bits, vector maintains
-//    signedness of its element type.
-// f: element type promoted to uint64_t (splat to vector type)
-// j: element type promoted to 64bits (splat to vector type)
-// K: element type bitcast to a signed integer (splat to vector type)
-// L: element type bitcast to an unsigned integer (splat to vector type)
-//
-// i: constant uint64_t
-// k: int32_t
-// l: int64_t
-// m: uint32_t
-// n: uint64_t
-
-// t: svint32_t
-// z: svuint32_t
-// g: svuint64_t
-// O: svfloat16_t
-// M: svfloat32_t
-// N: svfloat64_t
-
-// J: Prefetch type (sv_prfop)
-// A: pointer to int8_t
-// B: pointer to int16_t
-// C: pointer to int32_t
-// D: pointer to int64_t
-
-// E: pointer to uint8_t
-// F: pointer to uint16_t
-// G: pointer to uint32_t
-// H: pointer to uint64_t
-
-// Q: const pointer to void
-
-// S: const pointer to int8_t
-// T: const pointer to int16_t
-// U: const pointer to int32_t
-// V: const pointer to int64_t
-//
-// W: const pointer to uint8_t
-// X: const pointer to uint16_t
-// Y: const pointer to uint32_t
-// Z: const pointer to uint64_t
-
-class MergeType<int val, string suffix=""> {
-  int Value = val;
-  string Suffix = suffix;
-}
-def MergeNone    : MergeType<0>;
-def MergeAny     : MergeType<1, "_x">;
-def MergeOp1     : MergeType<2, "_m">;
-def MergeZero    : MergeType<3, "_z">;
-def MergeAnyExp  : MergeType<4, "_x">; // Use merged builtin with explicit
-def MergeZeroExp : MergeType<5, "_z">; // generation of its inactive argument.
-
-class EltType<int val> {
-  int Value = val;
-}
-def EltTyInvalid : EltType<0>;
-def EltTyInt8    : EltType<1>;
-def EltTyInt16   : EltType<2>;
-def EltTyInt32   : EltType<3>;
-def EltTyInt64   : EltType<4>;
-def EltTyFloat16 : EltType<5>;
-def EltTyFloat32 : EltType<6>;
-def EltTyFloat64 : EltType<7>;
-def EltTyBool8   : EltType<8>;
-def EltTyBool16  : EltType<9>;
-def EltTyBool32  : EltType<10>;
-def EltTyBool64  : EltType<11>;
-def EltTyBFloat16 : EltType<12>;
-
-class MemEltType<int val> {
-  int Value = val;
-}
-def MemEltTyDefault   : MemEltType<0>;
-def MemEltTyInt8      : MemEltType<1>;
-def MemEltTyInt16     : MemEltType<2>;
-def MemEltTyInt32     : MemEltType<3>;
-def MemEltTyInt64     : MemEltType<4>;
-
-class FlagType<int val> {
-  int Value = val;
-}
-
-// These must be kept in sync with the flags in utils/TableGen/SveEmitter.h
-// and include/clang/Basic/TargetBuiltins.h
-def NoFlags                   : FlagType<0x00000000>;
-def FirstEltType              : FlagType<0x00000001>;
-//      :                                     :
-//      :                                     :
-def EltTypeMask               : FlagType<0x0000000f>;
-def FirstMemEltType           : FlagType<0x00000010>;
-//      :                                     :
-//      :                                     :
-def MemEltTypeMask            : FlagType<0x00000070>;
-def FirstMergeTypeMask        : FlagType<0x00000080>;
-//      :                                     :
-//      :                                     :
-def MergeTypeMask             : FlagType<0x00000380>;
-def FirstSplatOperand         : FlagType<0x00000400>;
-//      :                                     :
-// These flags are used to specify which scalar operand
-// needs to be duplicated/splatted into a vector.
-//      :                                     :
-def SplatOperandMask          : FlagType<0x00001C00>;
-def IsLoad                    : FlagType<0x00002000>;
-def IsStore                   : FlagType<0x00004000>;
-def IsGatherLoad              : FlagType<0x00008000>;
-def IsScatterStore            : FlagType<0x00010000>;
-def IsStructLoad              : FlagType<0x00020000>;
-def IsStructStore             : FlagType<0x00040000>;
-def IsZExtReturn              : FlagType<0x00080000>; // Return value is sign-extend by default
-def IsOverloadNone            : FlagType<0x00100000>; // Intrinsic does not take any overloaded types.
-def IsOverloadWhile           : FlagType<0x00200000>; // Use {default type, typeof(operand1)} as overloaded types.
-def IsOverloadWhileRW         : FlagType<0x00400000>; // Use {pred(default type), typeof(operand0)} as overloaded types.
-def IsOverloadCvt             : FlagType<0x00800000>; // Use {typeof(operand0), typeof(last operand)} as overloaded types.
-def OverloadKindMask          : FlagType<0x00E00000>; // When the masked values are all '0', the default type is used as overload type.
-def IsByteIndexed             : FlagType<0x01000000>;
-def IsAppendSVALL             : FlagType<0x02000000>; // Appends SV_ALL as the last operand.
-def IsInsertOp1SVALL          : FlagType<0x04000000>; // Inserts SV_ALL as the second operand.
-def IsPrefetch                : FlagType<0x08000000>; // Contiguous prefetches.
-def IsGatherPrefetch          : FlagType<0x10000000>;
-def ReverseCompare            : FlagType<0x20000000>; // Compare operands must be swapped.
-def ReverseUSDOT              : FlagType<0x40000000>; // Unsigned/signed operands must be swapped.
-def IsUndef                   : FlagType<0x80000000>; // Codegen `undef` of given type.
-def IsTupleCreate             : FlagType<0x100000000>;
-def IsTupleGet                : FlagType<0x200000000>;
-def IsTupleSet                : FlagType<0x400000000>;
-def ReverseMergeAnyBinOp      : FlagType<0x800000000>; // e.g. Implement SUBR_X using SUB_X.
-def ReverseMergeAnyAccOp      : FlagType<0x1000000000>; // e.g. Implement MSB_X using MLS_X.
-
-// These must be kept in sync with the flags in include/clang/Basic/TargetBuiltins.h
-class ImmCheckType<int val> {
-  int Value = val;
-}
-def ImmCheck0_31                : ImmCheckType<0>;  // 0..31 (used for e.g. predicate patterns)
-def ImmCheck1_16                : ImmCheckType<1>;  // 1..16
-def ImmCheckExtract             : ImmCheckType<2>;  // 0..(2048/sizeinbits(elt) - 1)
-def ImmCheckShiftRight          : ImmCheckType<3>;  // 1..sizeinbits(elt)
-def ImmCheckShiftRightNarrow    : ImmCheckType<4>;  // 1..sizeinbits(elt)/2
-def ImmCheckShiftLeft           : ImmCheckType<5>;  // 0..(sizeinbits(elt) - 1)
-def ImmCheck0_7                 : ImmCheckType<6>;  // 0..7
-def ImmCheckLaneIndex           : ImmCheckType<7>;  // 0..(128/(1*sizeinbits(elt)) - 1)
-def ImmCheckLaneIndexCompRotate : ImmCheckType<8>;  // 0..(128/(2*sizeinbits(elt)) - 1)
-def ImmCheckLaneIndexDot        : ImmCheckType<9>;  // 0..(128/(4*sizeinbits(elt)) - 1)
-def ImmCheckComplexRot90_270    : ImmCheckType<10>; // [90,270]
-def ImmCheckComplexRotAll90     : ImmCheckType<11>; // [0, 90, 180,270]
-def ImmCheck0_13                : ImmCheckType<12>; // 0..13
-def ImmCheck0_1                 : ImmCheckType<13>; // 0..1
-def ImmCheck0_2                 : ImmCheckType<14>; // 0..2
-def ImmCheck0_3                 : ImmCheckType<15>; // 0..3
-
-class ImmCheck<int arg, ImmCheckType kind, int eltSizeArg = -1> {
-  int Arg = arg;
-  int EltSizeArg = eltSizeArg;
-  ImmCheckType Kind = kind;
-}
-
-class Inst<string n, string p, string t, MergeType mt, string i,
-           list<FlagType> ft, list<ImmCheck> ch, MemEltType met> {
-  string Name = n;
-  string Prototype = p;
-  string Types = t;
-  string TargetGuard = "sve";
-  int Merge = mt.Value;
-  string MergeSuffix = mt.Suffix;
-  string LLVMIntrinsic = i;
-  list<FlagType> Flags = ft;
-  list<ImmCheck> ImmChecks = ch;
-  int MemEltType = met.Value;
-}
-
-// SInst: Instruction with signed/unsigned suffix (e.g., "s8", "u8")
-class SInst<string n, string p, string t, MergeType mt, string i = "",
-            list<FlagType> ft = [], list<ImmCheck> ch = []>
-    : Inst<n, p, t, mt, i, ft, ch, MemEltTyDefault> {
-}
-
-// MInst: Instructions which access memory
-class MInst<string n, string p, string t, list<FlagType> f,
-            MemEltType met = MemEltTyDefault, string i = "">
-    : Inst<n, p, t, MergeNone, i, f, [], met> {
-}
+include "arm_sve_sme_incl.td"
 
 ////////////////////////////////////////////////////////////////////////////////
 // Loads
 
 // Load one vector (scalar base)
 def SVLD1   : MInst<"svld1[_{2}]", "dPc", "csilUcUsUiUlhfd", [IsLoad],               MemEltTyDefault, "aarch64_sve_ld1">;
 def SVLD1SB : MInst<"svld1sb_{d}", "dPS", "silUsUiUl",       [IsLoad],               MemEltTyInt8,    "aarch64_sve_ld1">;
 def SVLD1UB : MInst<"svld1ub_{d}", "dPW", "silUsUiUl",       [IsLoad, IsZExtReturn], MemEltTyInt8,    "aarch64_sve_ld1">;
 def SVLD1SH : MInst<"svld1sh_{d}", "dPT", "ilUiUl",          [IsLoad],               MemEltTyInt16,   "aarch64_sve_ld1">;
let TargetGuard = "sve,bf16" in {
   def SVBFDOT_N      : SInst<"svbfdot[_n_{0}]",      "MMda",  "b", MergeNone, "aarch64_sve_bfdot",        [IsOverloadNone]>;
   def SVBFMLAL_N     : SInst<"svbfmlalb[_n_{0}]",    "MMda",  "b", MergeNone, "aarch64_sve_bfmlalb",      [IsOverloadNone]>;
   def SVBFMLALT_N    : SInst<"svbfmlalt[_n_{0}]",    "MMda",  "b", MergeNone, "aarch64_sve_bfmlalt",      [IsOverloadNone]>;
   def SVBFDOT_LANE   : SInst<"svbfdot_lane[_{0}]",   "MMddi", "b", MergeNone, "aarch64_sve_bfdot_lane_v2",   [IsOverloadNone], [ImmCheck<3, ImmCheck0_3>]>;
   def SVBFMLALB_LANE : SInst<"svbfmlalb_lane[_{0}]", "MMddi", "b", MergeNone, "aarch64_sve_bfmlalb_lane_v2", [IsOverloadNone], [ImmCheck<3, ImmCheck0_7>]>;
   def SVBFMLALT_LANE : SInst<"svbfmlalt_lane[_{0}]", "MMddi", "b", MergeNone, "aarch64_sve_bfmlalt_lane_v2", [IsOverloadNone], [ImmCheck<3, ImmCheck0_7>]>;
 }
 
 ////////////////////////////////////////////////////////////////////////////////
 // Stores
 
 // Store one vector (scalar base)
 def SVST1    : MInst<"svst1[_{d}]",  "vPpd", "csilUcUsUiUlhfd", [IsStore], MemEltTyDefault, "aarch64_sve_st1">;
 def SVST1B_S : MInst<"svst1b[_{d}]", "vPAd", "sil",             [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1B_U : MInst<"svst1b[_{d}]", "vPEd", "UsUiUl",          [IsStore], MemEltTyInt8,    "aarch64_sve_st1">;
 def SVST1H_S : MInst<"svst1h[_{d}]", "vPBd", "il",              [IsStore], MemEltTyInt16,   "aarch64_sve_st1">;
 def SVST1H_U : MInst<"svst1h[_{d}]", "vPFd", "UiUl",            [IsStore], MemEltTyInt16,   "aarch64_sve_st1">;

clang/include/clang/Basic/arm_sve_sme_incl.td

+//===--- arm_sve_sme_incl.td - ARM SVE/SME compiler interface -------------===//
+//
+//  Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+//  See https://llvm.org/LICENSE.txt for license information.
+//  SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+//  This file defines common properites of TableGen definitions use for both
+//  SVE and SME intrinsics.
+//
+//      https://developer.arm.com/architectures/system-architectures/software-standards/acle
+//
+//===----------------------------------------------------------------------===//
+
+//===----------------------------------------------------------------------===//
+// Instruction definitions
+//===----------------------------------------------------------------------===//
+// Every intrinsic subclasses "Inst". An intrinsic has a name, a prototype and
+// a sequence of typespecs.
+//
+// The name is the base name of the intrinsic, for example "svld1". This is
+// then mangled by the tblgen backend to add type information ("svld1_s16").
+//
+// A typespec is a sequence of uppercase characters (modifiers) followed by one
+// lowercase character. A typespec encodes a particular "base type" of the
+// intrinsic.
+//
+// An example typespec is "Us" - unsigned short - svuint16_t. The available
+// typespec codes are given below.
+//
+// The string given to an Inst class is a sequence of typespecs. The intrinsic
+// is instantiated for every typespec in the sequence. For example "sdUsUd".
+//
+// The prototype is a string that defines the return type of the intrinsic
+// and the type of each argument. The return type and every argument gets a
+// "modifier" that can change in some way the "base type" of the intrinsic.
+//
+// The modifier 'd' means "default" and does not modify the base type in any
+// way. The available modifiers are given below.
+//
+// Typespecs
+// ---------
+// c: char
+// s: short
+// i: int
+// l: long
+// q: int128_t
+// f: float
+// h: half-float
+// d: double
+// b: bfloat
+
+// Typespec modifiers
+// ------------------
+// P: boolean
+// U: unsigned
+
+// Prototype modifiers
+// -------------------
+// prototype: return (arg, arg, ...)
+//
+// 2,3,4: array of default vectors
+// v: void
+// x: vector of signed integers
+// u: vector of unsigned integers
+// d: default
+// c: const pointer type
+// P: predicate type
+// s: scalar of element type
+// a: scalar of element type (splat to vector type)
+// R: scalar of 1/2 width element type (splat to vector type)
+// r: scalar of 1/4 width element type (splat to vector type)
+// @: unsigned scalar of 1/4 width element type (splat to vector type)
+// e: 1/2 width unsigned elements, 2x element count
+// b: 1/4 width unsigned elements, 4x element count
+// h: 1/2 width elements, 2x element count
+// q: 1/4 width elements, 4x element count
+// o: 4x width elements, 1/4 element count
+//
+// w: vector of element type promoted to 64bits, vector maintains
+//    signedness of its element type.
+// f: element type promoted to uint64_t (splat to vector type)
+// j: element type promoted to 64bits (splat to vector type)
+// K: element type bitcast to a signed integer (splat to vector type)
+// L: element type bitcast to an unsigned integer (splat to vector type)
+//
+// i: constant uint64_t
+// k: int32_t
+// l: int64_t
+// m: uint32_t
+// n: uint64_t
+
+// t: svint32_t
+// z: svuint32_t
+// g: svuint64_t
+// O: svfloat16_t
+// M: svfloat32_t
+// N: svfloat64_t
+
+// J: Prefetch type (sv_prfop)
+
+// %: pointer to void
+
+// A: pointer to int8_t
+// B: pointer to int16_t
+// C: pointer to int32_t
+// D: pointer to int64_t
+
+// E: pointer to uint8_t
+// F: pointer to uint16_t
+// G: pointer to uint32_t
+// H: pointer to uint64_t
+
+// Q: const pointer to void
+
+// S: const pointer to int8_t
+// T: const pointer to int16_t
+// U: const pointer to int32_t
+// V: const pointer to int64_t
+//
+// W: const pointer to uint8_t
+// X: const pointer to uint16_t
+// Y: const pointer to uint32_t
+// Z: const pointer to uint64_t
+
+class MergeType<int val, string suffix=""> {
+  int Value = val;
+  string Suffix = suffix;
+}
+def MergeNone    : MergeType<0>;
+def MergeAny     : MergeType<1, "_x">;
+def MergeOp1     : MergeType<2, "_m">;
+def MergeZero    : MergeType<3, "_z">;
+def MergeAnyExp  : MergeType<4, "_x">; // Use merged builtin with explicit
+def MergeZeroExp : MergeType<5, "_z">; // generation of its inactive argument.
+
+class EltType<int val> {
+  int Value = val;
+}
+def EltTyInvalid : EltType<0>;
+def EltTyInt8    : EltType<1>;
+def EltTyInt16   : EltType<2>;
+def EltTyInt32   : EltType<3>;
+def EltTyInt64   : EltType<4>;
+def EltTyInt128  : EltType<5>;
+def EltTyFloat16 : EltType<6>;
+def EltTyFloat32 : EltType<7>;
+def EltTyFloat64 : EltType<8>;
+def EltTyBool8   : EltType<9>;
+def EltTyBool16  : EltType<10>;
+def EltTyBool32  : EltType<11>;
+def EltTyBool64  : EltType<12>;
+def EltTyBFloat16 : EltType<13>;
+
+class MemEltType<int val> {
+  int Value = val;
+}
+def MemEltTyDefault   : MemEltType<0>;
+def MemEltTyInt8      : MemEltType<1>;
+def MemEltTyInt16     : MemEltType<2>;
+def MemEltTyInt32     : MemEltType<3>;
+def MemEltTyInt64     : MemEltType<4>;
+
+class FlagType<int val> {
+  int Value = val;
+}
+
+// These must be kept in sync with the flags in utils/TableGen/SveEmitter.h
+// and include/clang/Basic/TargetBuiltins.h
+def NoFlags                   : FlagType<0x00000000>;
+def FirstEltType              : FlagType<0x00000001>;
+//      :                                     :
+//      :                                     :
+def EltTypeMask               : FlagType<0x0000000f>;
+def FirstMemEltType           : FlagType<0x00000010>;
+//      :                                     :
+//      :                                     :
+def MemEltTypeMask            : FlagType<0x00000070>;
+def FirstMergeTypeMask        : FlagType<0x00000080>;
+//      :                                     :
+//      :                                     :
+def MergeTypeMask             : FlagType<0x00000380>;
+def FirstSplatOperand         : FlagType<0x00000400>;
+//      :                                     :
+// These flags are used to specify which scalar operand
+// needs to be duplicated/splatted into a vector.
+//      :                                     :
+def SplatOperandMask          : FlagType<0x00001C00>;
+def IsLoad                    : FlagType<0x00002000>;
+def IsStore                   : FlagType<0x00004000>;
+def IsGatherLoad              : FlagType<0x00008000>;
+def IsScatterStore            : FlagType<0x00010000>;
+def IsStructLoad              : FlagType<0x00020000>;
+def IsStructStore             : FlagType<0x00040000>;
+def IsZExtReturn              : FlagType<0x00080000>; // Return value is sign-extend by default
+def IsOverloadNone            : FlagType<0x00100000>; // Intrinsic does not take any overloaded types.
+def IsOverloadWhile           : FlagType<0x00200000>; // Use {default type, typeof(operand1)} as overloaded types.
+def IsOverloadWhileRW         : FlagType<0x00400000>; // Use {pred(default type), typeof(operand0)} as overloaded types.
+def IsOverloadCvt             : FlagType<0x00800000>; // Use {typeof(operand0), typeof(last operand)} as overloaded types.
+def OverloadKindMask          : FlagType<0x00E00000>; // When the masked values are all '0', the default type is used as overload type.
+def IsByteIndexed             : FlagType<0x01000000>;
+def IsAppendSVALL             : FlagType<0x02000000>; // Appends SV_ALL as the last operand.
+def IsInsertOp1SVALL          : FlagType<0x04000000>; // Inserts SV_ALL as the second operand.
+def IsPrefetch                : FlagType<0x08000000>; // Contiguous prefetches.
+def IsGatherPrefetch          : FlagType<0x10000000>;
+def ReverseCompare            : FlagType<0x20000000>; // Compare operands must be swapped.
+def ReverseUSDOT              : FlagType<0x40000000>; // Unsigned/signed operands must be swapped.
+def IsUndef                   : FlagType<0x80000000>; // Codegen `undef` of given type.
+def IsTupleCreate             : FlagType<0x100000000>;
+def IsTupleGet                : FlagType<0x200000000>;
+def IsTupleSet                : FlagType<0x400000000>;
+def ReverseMergeAnyBinOp      : FlagType<0x800000000>; // e.g. Implement SUBR_X using SUB_X.
+def ReverseMergeAnyAccOp      : FlagType<0x1000000000>; // e.g. Implement MSB_X using MLS_X.
+def IsStreaming               : FlagType<0x2000000000>;
+def IsStreamingCompatible     : FlagType<0x4000000000>;
+def IsSharedZA                : FlagType<0x8000000000>;
+def IsPreservesZA             : FlagType<0x10000000000>;
+
+// These must be kept in sync with the flags in include/clang/Basic/TargetBuiltins.h
+class ImmCheckType<int val> {
+  int Value = val;
+}
+def ImmCheck0_31                : ImmCheckType<0>;  // 0..31 (used for e.g. predicate patterns)
+def ImmCheck1_16                : ImmCheckType<1>;  // 1..16
+def ImmCheckExtract             : ImmCheckType<2>;  // 0..(2048/sizeinbits(elt) - 1)
+def ImmCheckShiftRight          : ImmCheckType<3>;  // 1..sizeinbits(elt)
+def ImmCheckShiftRightNarrow    : ImmCheckType<4>;  // 1..sizeinbits(elt)/2
+def ImmCheckShiftLeft           : ImmCheckType<5>;  // 0..(sizeinbits(elt) - 1)
+def ImmCheck0_7                 : ImmCheckType<6>;  // 0..7
+def ImmCheckLaneIndex           : ImmCheckType<7>;  // 0..(128/(1*sizeinbits(elt)) - 1)
+def ImmCheckLaneIndexCompRotate : ImmCheckType<8>;  // 0..(128/(2*sizeinbits(elt)) - 1)
+def ImmCheckLaneIndexDot        : ImmCheckType<9>;  // 0..(128/(4*sizeinbits(elt)) - 1)
+def ImmCheckComplexRot90_270    : ImmCheckType<10>; // [90,270]
+def ImmCheckComplexRotAll90     : ImmCheckType<11>; // [0, 90, 180,270]
+def ImmCheck0_13                : ImmCheckType<12>; // 0..13
+def ImmCheck0_1                 : ImmCheckType<13>; // 0..1
+def ImmCheck0_2                 : ImmCheckType<14>; // 0..2
+def ImmCheck0_3                 : ImmCheckType<15>; // 0..3
+def ImmCheck0_0                 : ImmCheckType<16>; // 0..0
+def ImmCheck0_15                : ImmCheckType<17>; // 0..15
+
+class ImmCheck<int arg, ImmCheckType kind, int eltSizeArg = -1> {
+  int Arg = arg;
+  int EltSizeArg = eltSizeArg;
+  ImmCheckType Kind = kind;
+}
+
+class Inst<string n, string p, string t, MergeType mt, string i,
+           list<FlagType> ft, list<ImmCheck> ch, MemEltType met> {
+  string Name = n;
+  string Prototype = p;
+  string Types = t;
+  string TargetGuard = "sve";
+  int Merge = mt.Value;
+  string MergeSuffix = mt.Suffix;
+  string LLVMIntrinsic = i;
+  list<FlagType> Flags = ft;
+  list<ImmCheck> ImmChecks = ch;
+  int MemEltType = met.Value;
+}
+
+// SInst: Instruction with signed/unsigned suffix (e.g., "s8", "u8")
+class SInst<string n, string p, string t, MergeType mt, string i = "",
+            list<FlagType> ft = [], list<ImmCheck> ch = []>
+    : Inst<n, p, t, mt, i, ft, ch, MemEltTyDefault> {
+}
+
+// MInst: Instructions which access memory
+class MInst<string n, string p, string t, list<FlagType> f,
+            MemEltType met = MemEltTyDefault, string i = "",
+            list<ImmCheck> ch = []>
+    : Inst<n, p, t, MergeNone, i, f, ch, met> {
+}

clang/lib/Basic/Targets/AArch64.cpp

 #define BUILTIN(ID, TYPE, ATTRS)                                               \
   {#ID, TYPE, ATTRS, nullptr, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
 #define TARGET_BUILTIN(ID, TYPE, ATTRS, FEATURE)                               \
   {#ID, TYPE, ATTRS, FEATURE, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
 #include "clang/Basic/BuiltinsSVE.def"
 
 #define BUILTIN(ID, TYPE, ATTRS)                                               \
   {#ID, TYPE, ATTRS, nullptr, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
+#define TARGET_BUILTIN(ID, TYPE, ATTRS, FEATURE)                               \
+  {#ID, TYPE, ATTRS, FEATURE, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
+#include "clang/Basic/BuiltinsSME.def"
+
+#define BUILTIN(ID, TYPE, ATTRS)                                               \
+  {#ID, TYPE, ATTRS, nullptr, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
 #define LANGBUILTIN(ID, TYPE, ATTRS, LANG)                                     \
   {#ID, TYPE, ATTRS, nullptr, HeaderDesc::NO_HEADER, LANG},
 #define TARGET_BUILTIN(ID, TYPE, ATTRS, FEATURE)                               \
   {#ID, TYPE, ATTRS, FEATURE, HeaderDesc::NO_HEADER, ALL_LANGUAGES},
 #define TARGET_HEADER_BUILTIN(ID, TYPE, ATTRS, HEADER, LANGS, FEATURE)         \
   {#ID, TYPE, ATTRS, FEATURE, HeaderDesc::HEADER, LANGS},
 #include "clang/Basic/BuiltinsAArch64.def"
 };
void AArch64TargetInfo::getTargetDefines(const LangOptions &Opts,
 
   if (HasSVE2 && HasSVE2SHA3)
     Builder.defineMacro("__ARM_FEATURE_SVE2_SHA3", "1");
 
   if (HasSVE2 && HasSVE2SM4)
     Builder.defineMacro("__ARM_FEATURE_SVE2_SM4", "1");
 
   if (HasCRC)
     Builder.defineMacro("__ARM_FEATURE_CRC32", "1");
 
   if (HasRCPC)
     Builder.defineMacro("__ARM_FEATURE_RCPC", "1");
 
   if (HasFMV)
     Builder.defineMacro("__HAVE_FUNCTION_MULTI_VERSIONING", "1");
 
   // The __ARM_FEATURE_CRYPTO is deprecated in favor of finer grained feature
if ((FPU & SveMode) && HasMatmulFP32)
     Builder.defineMacro("__ARM_FEATURE_SVE_MATMUL_FP32", "1");
 
   if ((FPU & SveMode) && HasMatMul)
     Builder.defineMacro("__ARM_FEATURE_SVE_MATMUL_INT8", "1");
 
   if ((FPU & NeonMode) && HasFP16FML)
     Builder.defineMacro("__ARM_FEATURE_FP16_FML", "1");
 
   if (Opts.hasSignReturnAddress()) {
     // Bitmask:
     // 0: Protection using the A key
     // 1: Protection using the B key
     // 2: Protection including leaf functions
     unsigned Value = 0;
 
     if (Opts.isSignReturnAddressWithAKey())
       Value |= (1 << 0);
for (const auto &Feature : Features) {
     if (Feature == "+jscvt") {
       HasJSCVT = true;
       FPU |= NeonMode;
     }
     if (Feature == "+fcma") {
       HasFCMA = true;
       FPU |= NeonMode;
     }
 
     if (Feature == "+sve") {
       FPU |= NeonMode;
       FPU |= SveMode;
       HasFullFP16 = true;
     }
     if (Feature == "+sve2") {
       FPU |= NeonMode;
       FPU |= SveMode;
       HasFullFP16 = true;
if (Feature == "+f64mm") {
       FPU |= NeonMode;
       FPU |= SveMode;
       HasFullFP16 = true;
       HasMatmulFP64 = true;
     }
     if (Feature == "+sme") {
       HasSME = true;
       HasBFloat16 = true;
+      HasFullFP16 = true;
     }
     if (Feature == "+sme-f64f64") {
       HasSME = true;
       HasSMEF64F64 = true;
       HasBFloat16 = true;
+      HasFullFP16 = true;
     }
     if (Feature == "+sme-i16i64") {
       HasSME = true;
       HasSMEI16I64 = true;
       HasBFloat16 = true;
+      HasFullFP16 = true;
     }
     if (Feature == "+sb")
       HasSB = true;
     if (Feature == "+predres")
       HasPredRes = true;
     if (Feature == "+ssbs")
       HasSSBS = true;
     if (Feature == "+bti")

clang/lib/CodeGen/CGBuiltin.cpp

 #include "clang/Basic/arm_sve_builtin_cg.inc"
 #include "clang/Basic/BuiltinsAArch64NeonSVEBridge_cg.def"
 #undef GET_SVE_LLVM_INTRINSIC_MAP
 };
 
 #undef SVEMAP1
 #undef SVEMAP2
 
+#define SMEMAP1(NameBase, LLVMIntrinsic, TypeModifier)                         \
+  {                                                                            \
+    #NameBase, SME::BI__builtin_sme_##NameBase, Intrinsic::LLVMIntrinsic, 0,   \
+        TypeModifier                                                           \
+  }
+
+#define SMEMAP2(NameBase, TypeModifier)                                        \
+  { #NameBase, SME::BI__builtin_sme_##NameBase, 0, 0, TypeModifier }
+static const ARMVectorIntrinsicInfo AArch64SMEIntrinsicMap[] = {
+#define GET_SME_LLVM_INTRINSIC_MAP
+#include "clang/Basic/arm_sme_builtin_cg.inc"
+#undef GET_SME_LLVM_INTRINSIC_MAP
+};
+
+#undef SMEMAP1
+#undef SMEMAP2
+
 static bool NEONSIMDIntrinsicsProvenSorted = false;
 
 static bool AArch64SIMDIntrinsicsProvenSorted = false;
 static bool AArch64SISDIntrinsicsProvenSorted = false;
 static bool AArch64SVEIntrinsicsProvenSorted = false;
+static bool AArch64SMEIntrinsicsProvenSorted = false;
 
 static const ARMVectorIntrinsicInfo *
 findARMVectorIntrinsicInMap(ArrayRef<ARMVectorIntrinsicInfo> IntrinsicMap,
                             unsigned BuiltinID, bool &MapProvenSorted) {
 
 #ifndef NDEBUG
   if (!MapProvenSorted) {
     assert(llvm::is_sorted(IntrinsicMap));
llvm::Type *CodeGenFunction::getEltType(const SVETypeFlags &TypeFlags) {
   case SVETypeFlags::EltTyInt8:
     return Builder.getInt8Ty();
   case SVETypeFlags::EltTyInt16:
     return Builder.getInt16Ty();
   case SVETypeFlags::EltTyInt32:
     return Builder.getInt32Ty();
   case SVETypeFlags::EltTyInt64:
     return Builder.getInt64Ty();
+  case SVETypeFlags::EltTyInt128:
+    return Builder.getInt128Ty();
 
   case SVETypeFlags::EltTyFloat16:
     return Builder.getHalfTy();
   case SVETypeFlags::EltTyFloat32:
     return Builder.getFloatTy();
   case SVETypeFlags::EltTyFloat64:
     return Builder.getDoubleTy();
 
   case SVETypeFlags::EltTyBFloat16:
     return Builder.getBFloatTy();
 
   case SVETypeFlags::EltTyBool8:
   case SVETypeFlags::EltTyBool16:
   case SVETypeFlags::EltTyBool32:
   case SVETypeFlags::EltTyBool64:
     return Builder.getInt1Ty();
   }
 }
 
 // Return the llvm predicate vector type corresponding to the specified element
 // TypeFlags.
 llvm::ScalableVectorType *
 CodeGenFunction::getSVEPredType(const SVETypeFlags &TypeFlags) {
   switch (TypeFlags.getEltType()) {
Value *CodeGenFunction::EmitSVEPredicateCast(Value *Pred,
   if (Pred->getType() == RTy)
     return Pred;
 
   unsigned IntID;
   llvm::Type *IntrinsicTy;
   switch (VTy->getMinNumElements()) {
   default:
     llvm_unreachable("unsupported element count!");
+  case 1:
   case 2:
   case 4:
   case 8:
     IntID = Intrinsic::aarch64_sve_convert_from_svbool;
     IntrinsicTy = RTy;
     break;
   case 16:
     IntID = Intrinsic::aarch64_sve_convert_to_svbool;
Value *CodeGenFunction::EmitSVEMaskedStore(const CallExpr *E,
   Function *F = CGM.getIntrinsic(BuiltinID, MemoryTy);
   auto *Store =
       cast<llvm::Instruction>(Builder.CreateCall(F, {Val, Predicate, BasePtr}));
   auto TBAAInfo = CGM.getTBAAAccessInfo(LangPTy->getPointeeType());
   CGM.DecorateInstructionWithTBAA(Store, TBAAInfo);
   return Store;
 }
 
+Value *CodeGenFunction::EmitTileslice(Value *Offset, Value *Base) {
+  llvm::Value *CastOffset = Builder.CreateIntCast(Offset, Int32Ty, false);
+  return Builder.CreateAdd(Base, CastOffset, "tileslice");
+}
+
+Value *CodeGenFunction::EmitSMELd1St1(SVETypeFlags TypeFlags,
+                                      SmallVectorImpl<Value *> &Ops,
+                                      unsigned IntID) {
+  Ops[3] = EmitSVEPredicateCast(
+      Ops[3], getSVEVectorForElementType(SVEBuiltinMemEltTy(TypeFlags)));
+
+  SmallVector<Value *> NewOps;
+  NewOps.push_back(Ops[3]);
+
+  llvm::Value *BasePtr = Ops[4];
+
+  // If the intrinsic contains the vnum parameter, multiply it with the vector
+  // size in bytes.
+  if (Ops.size() == 6) {
+    Function *StreamingVectorLength =
+        CGM.getIntrinsic(Intrinsic::aarch64_sme_cntsb);
+    llvm::Value *StreamingVectorLengthCall =
+        Builder.CreateCall(StreamingVectorLength);
+    llvm::Value *Mulvl =
+        Builder.CreateMul(StreamingVectorLengthCall, Ops[5], "mulvl");
+    // The type of the ptr parameter is void *, so use Int8Ty here.
+    BasePtr = Builder.CreateGEP(Int8Ty, Ops[4], Mulvl);
+  }
+  NewOps.push_back(BasePtr);
+  NewOps.push_back(Ops[0]);
+  NewOps.push_back(EmitTileslice(Ops[2], Ops[1]));
+  Function *F = CGM.getIntrinsic(IntID);
+  return Builder.CreateCall(F, NewOps);
+}
+
 // Limit the usage of scalable llvm IR generated by the ACLE by using the
 // sve dup.x intrinsic instead of IRBuilder::CreateVectorSplat.
 Value *CodeGenFunction::EmitSVEDupX(Value *Scalar, llvm::Type *Ty) {
   return Builder.CreateVectorSplat(
       cast<llvm::VectorType>(Ty)->getElementCount(), Scalar);
 }
 
 Value *CodeGenFunction::EmitSVEDupX(Value* Scalar) {
   return EmitSVEDupX(Scalar, getSVEVectorForElementType(Scalar->getType()));
 }
 
 Value *CodeGenFunction::EmitSVEReinterpret(Value *Val, llvm::Type *Ty) {
   // FIXME: For big endian this needs an additional REV, or needs a separate
   // intrinsic that is code-generated as a no-op, because the LLVM bitcast
   // instruction is defined as 'bitwise' equivalent from memory point of
   // view (when storing/reloading), whereas the svreinterpret builtin
   // implements bitwise equivalent cast from register point of view.
   // LLVM CodeGen for a bitcast must add an explicit REV for big-endian.
   return Builder.CreateBitCast(Val, Ty);
 }
 
 static void InsertExplicitZeroOperand(CGBuilderTy &Builder, llvm::Type *Ty,
                                       SmallVectorImpl<Value *> &Ops) {
   auto *SplatZero = Constant::getNullValue(Ty);
   Ops.insert(Ops.begin(), SplatZero);
 }
 
 static void InsertExplicitUndefOperand(CGBuilderTy &Builder, llvm::Type *Ty,
                                        SmallVectorImpl<Value *> &Ops) {
   auto *SplatUndef = UndefValue::get(Ty);
   Ops.insert(Ops.begin(), SplatUndef);
 }
 
 SmallVector<llvm::Type *, 2>
 CodeGenFunction::getSVEOverloadTypes(const SVETypeFlags &TypeFlags,
Value *CodeGenFunction::EmitAArch64SVEBuiltinExpr(unsigned BuiltinID,
   auto *Builtin = findARMVectorIntrinsicInMap(AArch64SVEIntrinsicMap, BuiltinID,
                                               AArch64SVEIntrinsicsProvenSorted);
   SVETypeFlags TypeFlags(Builtin->TypeModifier);
   if (TypeFlags.isLoad())
     return EmitSVEMaskedLoad(E, Ty, Ops, Builtin->LLVMIntrinsic,
                              TypeFlags.isZExtReturn());
   else if (TypeFlags.isStore())
     return EmitSVEMaskedStore(E, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isGatherLoad())
     return EmitSVEGatherLoad(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isScatterStore())
     return EmitSVEScatterStore(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isPrefetch())
     return EmitSVEPrefetchLoad(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isGatherPrefetch())
     return EmitSVEGatherPrefetch(TypeFlags, Ops, Builtin->LLVMIntrinsic);
   else if (TypeFlags.isStructLoad())
return Builder.CreateIntrinsic(Intrinsic::aarch64_sve_dupq_lane, {Ty},
                                    {Insert, Builder.getInt64(0)});
   }
   }
 
   /// Should not happen
   return nullptr;
 }
 
+Value *CodeGenFunction::EmitAArch64SMEBuiltinExpr(unsigned BuiltinID,
+                                                  const CallExpr *E) {
+  // Find out if any arguments are required to be integer constant expressions.
+  unsigned ICEArguments = 0;
+  ASTContext::GetBuiltinTypeError Error;
+  getContext().GetBuiltinType(BuiltinID, Error, &ICEArguments);
+  assert(Error == ASTContext::GE_None && "Should not codegen an error");
+
+  llvm::SmallVector<Value *, 4> Ops;
+  for (unsigned i = 0, e = E->getNumArgs(); i != e; i++) {
+    if ((ICEArguments & (1 << i)) == 0)
+      Ops.push_back(EmitScalarExpr(E->getArg(i)));
+    else {
+      // If this is required to be a constant, constant fold it so that we know
+      // that the generated intrinsic gets a ConstantInt.
+      std::optional<llvm::APSInt> Result =
+          E->getArg(i)->getIntegerConstantExpr(getContext());
+      assert(Result && "Expected argument to be a constant");
+
+      // Immediates for SVE llvm intrinsics are always 32bit.  We can safely
+      // truncate because the immediate has been range checked and no valid
+      // immediate requires more than a handful of bits.
+      *Result = Result->extOrTrunc(32);
+      Ops.push_back(llvm::ConstantInt::get(getLLVMContext(), *Result));
+    }
+  }
+
+  auto *Builtin = findARMVectorIntrinsicInMap(AArch64SMEIntrinsicMap, BuiltinID,
+                                              AArch64SMEIntrinsicsProvenSorted);
+  SVETypeFlags TypeFlags(Builtin->TypeModifier);
+  if (TypeFlags.isLoad() || TypeFlags.isStore())
+    return EmitSMELd1St1(TypeFlags, Ops, Builtin->LLVMIntrinsic);
+
+  /// Should not happen
+  return nullptr;
+}
+
 Value *CodeGenFunction::EmitAArch64BuiltinExpr(unsigned BuiltinID,
                                                const CallExpr *E,
                                                llvm::Triple::ArchType Arch) {
   if (BuiltinID >= clang::AArch64::FirstSVEBuiltin &&
       BuiltinID <= clang::AArch64::LastSVEBuiltin)
     return EmitAArch64SVEBuiltinExpr(BuiltinID, E);
 
+  if (BuiltinID >= clang::AArch64::FirstSMEBuiltin &&
+      BuiltinID <= clang::AArch64::LastSMEBuiltin)
+    return EmitAArch64SMEBuiltinExpr(BuiltinID, E);
+
   unsigned HintID = static_cast<unsigned>(-1);
   switch (BuiltinID) {
   default: break;
   case clang::AArch64::BI__builtin_arm_nop:
     HintID = 0;
     break;
   case clang::AArch64::BI__builtin_arm_yield:
   case clang::AArch64::BI__yield:

clang/lib/CodeGen/CodeGenFunction.h

llvm::Value *EmitSVEScatterStore(const SVETypeFlags &TypeFlags,
                                    llvm::SmallVectorImpl<llvm::Value *> &Ops,
                                    unsigned IntID);
   llvm::Value *EmitSVEMaskedLoad(const CallExpr *, llvm::Type *ReturnTy,
                                  SmallVectorImpl<llvm::Value *> &Ops,
                                  unsigned BuiltinID, bool IsZExtReturn);
   llvm::Value *EmitSVEMaskedStore(const CallExpr *,
                                   SmallVectorImpl<llvm::Value *> &Ops,
                                   unsigned BuiltinID);
+  llvm::Value *EmitTileslice(llvm::Value *Offset, llvm::Value *Base);
   llvm::Value *EmitSVEPrefetchLoad(const SVETypeFlags &TypeFlags,
                                    SmallVectorImpl<llvm::Value *> &Ops,
                                    unsigned BuiltinID);
   llvm::Value *EmitSVEGatherPrefetch(const SVETypeFlags &TypeFlags,
                                      SmallVectorImpl<llvm::Value *> &Ops,
                                      unsigned IntID);
   llvm::Value *EmitSVEStructLoad(const SVETypeFlags &TypeFlags,
                                  SmallVectorImpl<llvm::Value *> &Ops,
                                  unsigned IntID);
   llvm::Value *EmitSVEStructStore(const SVETypeFlags &TypeFlags,
                                   SmallVectorImpl<llvm::Value *> &Ops,
                                   unsigned IntID);
   llvm::Value *EmitAArch64SVEBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
 
+  llvm::Value *EmitSMELd1St1(SVETypeFlags TypeFlags,
+                             llvm::SmallVectorImpl<llvm::Value *> &Ops,
+                             unsigned IntID);
+  llvm::Value *EmitAArch64SMEBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
+
   llvm::Value *EmitAArch64BuiltinExpr(unsigned BuiltinID, const CallExpr *E,
                                       llvm::Triple::ArchType Arch);
   llvm::Value *EmitBPFBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
 
   llvm::Value *BuildVector(ArrayRef<llvm::Value*> Ops);
   llvm::Value *EmitX86BuiltinExpr(unsigned BuiltinID, const CallExpr *E);
   llvm::Value *EmitPPCBuiltinExpr(unsigned BuiltinID, const CallExpr *E);
   llvm::Value *EmitAMDGPUBuiltinExpr(unsigned BuiltinID, const CallExpr *E);

clang/lib/Headers/CMakeLists.txt

 # Generate header files and copy them to the build directory
 if(ARM IN_LIST LLVM_TARGETS_TO_BUILD OR AArch64 IN_LIST LLVM_TARGETS_TO_BUILD)
   # Generate arm_neon.h
   clang_generate_header(-gen-arm-neon arm_neon.td arm_neon.h)
   # Generate arm_fp16.h
   clang_generate_header(-gen-arm-fp16 arm_fp16.td arm_fp16.h)
   # Generate arm_sve.h
   clang_generate_header(-gen-arm-sve-header arm_sve.td arm_sve.h)
+  # Generate arm_sme_draft_spec_subject_to_change.h
+  clang_generate_header(-gen-arm-sme-header arm_sme.td arm_sme_draft_spec_subject_to_change.h)
   # Generate arm_bf16.h
   clang_generate_header(-gen-arm-bf16 arm_bf16.td arm_bf16.h)
   # Generate arm_mve.h
   clang_generate_header(-gen-arm-mve-header arm_mve.td arm_mve.h)
   # Generate arm_cde.h
   clang_generate_header(-gen-arm-cde-header arm_cde.td arm_cde.h)
 
   # Add headers to target specific lists
   list(APPEND arm_common_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_neon.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_fp16.h"
     )
 
   list(APPEND arm_only_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_mve.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_cde.h"
     )
 
   list(APPEND aarch64_only_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/arm_sve.h"
+    "${CMAKE_CURRENT_BINARY_DIR}/arm_sme_draft_spec_subject_to_change.h"
     "${CMAKE_CURRENT_BINARY_DIR}/arm_bf16.h"
     )
 endif()
 if(RISCV IN_LIST LLVM_TARGETS_TO_BUILD)
   # Generate riscv_vector.h
   clang_generate_header(-gen-riscv-vector-header riscv_vector.td riscv_vector.h)
   list(APPEND riscv_generated_files
     "${CMAKE_CURRENT_BINARY_DIR}/riscv_vector.h"

clang/lib/Sema/SemaChecking.cpp

bool Sema::CheckSVEBuiltinFunctionCall(unsigned BuiltinID, CallExpr *TheCall) {
   SmallVector<std::tuple<int,int,int>, 3> ImmChecks;
 
   switch (BuiltinID) {
   default:
     return false;
 #define GET_SVE_IMMEDIATE_CHECK
 #include "clang/Basic/arm_sve_sema_rangechecks.inc"
 #undef GET_SVE_IMMEDIATE_CHECK
+#define GET_SME_IMMEDIATE_CHECK
+#include "clang/Basic/arm_sme_sema_rangechecks.inc"
+#undef GET_SME_IMMEDIATE_CHECK
   }
 
   // Perform all the immediate checks for this builtin call.
   bool HasError = false;
   for (auto &I : ImmChecks) {
     int ArgNum, CheckTy, ElementSizeInBits;
     std::tie(ArgNum, CheckTy, ElementSizeInBits) = I;
 
for (auto &I : ImmChecks) {
     case SVETypeFlags::ImmCheck0_2:
       if (SemaBuiltinConstantArgRange(TheCall, ArgNum, 0, 2))
         HasError = true;
       break;
     case SVETypeFlags::ImmCheck0_3:
       if (SemaBuiltinConstantArgRange(TheCall, ArgNum, 0, 3))
         HasError = true;
       break;
+    case SVETypeFlags::ImmCheck0_0:
+      if (SemaBuiltinConstantArgRange(TheCall, ArgNum, 0, 0))
+        HasError = true;
+      break;
+    case SVETypeFlags::ImmCheck0_15:
+      if (SemaBuiltinConstantArgRange(TheCall, ArgNum, 0, 15))
+        HasError = true;
+      break;
     }
   }
 
   return HasError;
 }
 
 bool Sema::CheckNeonBuiltinFunctionCall(const TargetInfo &TI,
                                         unsigned BuiltinID, CallExpr *TheCall) {

clang/lib/Sema/SemaDeclAttr.cpp

 static bool ArmSveAliasValid(ASTContext &Context, unsigned BuiltinID,
                              StringRef AliasName) {
   if (Context.BuiltinInfo.isAuxBuiltinID(BuiltinID))
     BuiltinID = Context.BuiltinInfo.getAuxBuiltinID(BuiltinID);
   return BuiltinID >= AArch64::FirstSVEBuiltin &&
          BuiltinID <= AArch64::LastSVEBuiltin;
 }
 
+static bool ArmSmeAliasValid(ASTContext &Context, unsigned BuiltinID,
+                             StringRef AliasName) {
+  if (Context.BuiltinInfo.isAuxBuiltinID(BuiltinID))
+    BuiltinID = Context.BuiltinInfo.getAuxBuiltinID(BuiltinID);
+  return BuiltinID >= AArch64::FirstSMEBuiltin &&
+         BuiltinID <= AArch64::LastSMEBuiltin;
+}
+
 static void handleArmBuiltinAliasAttr(Sema &S, Decl *D, const ParsedAttr &AL) {
   if (!AL.isArgIdent(0)) {
     S.Diag(AL.getLoc(), diag::err_attribute_argument_n_type)
         << AL << 1 << AANT_ArgumentIdentifier;
     return;
   }
 
   IdentifierInfo *Ident = AL.getArgAsIdent(0)->Ident;
   unsigned BuiltinID = Ident->getBuiltinID();
   StringRef AliasName = cast<FunctionDecl>(D)->getIdentifier()->getName();
 
   bool IsAArch64 = S.Context.getTargetInfo().getTriple().isAArch64();
-  if ((IsAArch64 && !ArmSveAliasValid(S.Context, BuiltinID, AliasName)) ||
+  if ((IsAArch64 && !ArmSveAliasValid(S.Context, BuiltinID, AliasName) &&
+       !ArmSmeAliasValid(S.Context, BuiltinID, AliasName)) ||
       (!IsAArch64 && !ArmMveAliasValid(BuiltinID, AliasName) &&
        !ArmCdeAliasValid(BuiltinID, AliasName))) {
     S.Diag(AL.getLoc(), diag::err_attribute_arm_builtin_alias);
     return;
   }
 
   D->addAttr(::new (S.Context) ArmBuiltinAliasAttr(S.Context, AL, Ident));
 }

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s -check-prefixes=CHECK,CHECK-C
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefixes=CHECK,CHECK-CXX
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+#ifdef DISABLE_SME_ATTRIBUTES
+#define ARM_STREAMING_ATTR
+#else
+#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
+#endif
+
+// CHECK-C-LABEL:   @test_svld1_hor_za8(
+// CHECK-CXX-LABEL: @_Z18test_svld1_hor_za8ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY:ptr|i8\*]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[PTR]], i32 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_za8(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za8(0, slice_base, 0, pg, ptr);
+  svld1_hor_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_za16(
+// CHECK-CXX-LABEL: @_Z19test_svld1_hor_za16ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za16(0, slice_base, 0, pg, ptr);
+  svld1_hor_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_za32(
+// CHECK-CXX-LABEL: @_Z19test_svld1_hor_za32ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_za32(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za32(0, slice_base, 0, pg, ptr);
+  svld1_hor_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_za64(
+// CHECK-CXX-LABEL: @_Z19test_svld1_hor_za64ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_za64(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za64(0, slice_base, 0, pg, ptr);
+  svld1_hor_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_za128(
+// CHECK-CXX-LABEL: @_Z20test_svld1_hor_za128ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_za128(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_hor_za128(0, slice_base, 0, pg, ptr);
+  svld1_hor_za128(15, slice_base, 0, pg, ptr);
+}
+
+// CHECK-C-LABEL: @test_svld1_ver_za8(
+// CHECK-CXX-LABEL: @_Z18test_svld1_ver_za8ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[PTR]], i32 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_za8(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za8(0, slice_base, 0, pg, ptr);
+  svld1_ver_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_za16(
+// CHECK-CXX-LABEL: @_Z19test_svld1_ver_za16ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_za16(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za16(0, slice_base, 0, pg, ptr);
+  svld1_ver_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_za32(
+// CHECK-CXX-LABEL: @_Z19test_svld1_ver_za32ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_za32(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za32(0, slice_base, 0, pg, ptr);
+  svld1_ver_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_za64(
+// CHECK-CXX-LABEL: @_Z19test_svld1_ver_za64ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_za64(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za64(0, slice_base, 0, pg, ptr);
+  svld1_ver_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_za128(
+// CHECK-CXX-LABEL: @_Z20test_svld1_ver_za128ju10__SVBool_tPKv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_za128(uint32_t slice_base, svbool_t pg, const void *ptr) {
+  svld1_ver_za128(0, slice_base, 0, pg, ptr);
+  svld1_ver_za128(15, slice_base, 0, pg, ptr);
+}

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_ld1_vnum.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s -check-prefixes=CHECK,CHECK-C
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefixes=CHECK,CHECK-CXX
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+#ifdef DISABLE_SME_ATTRIBUTES
+#define ARM_STREAMING_ATTR
+#else
+#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
+#endif
+
+// CHECK-C-LABEL:   @test_svld1_hor_vnum_za8(
+// CHECK-CXX-LABEL: @_Z23test_svld1_hor_vnum_za8ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP0]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP1:%.*]] = getelementptr i8, [[PTRTY:ptr|i8\*]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[TMP1]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.horiz(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[TMP1]], i32 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_vnum_za8(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_hor_vnum_za8(0, slice_base, 0, pg, ptr, vnum);
+  svld1_hor_vnum_za8(0, slice_base, 15, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_vnum_za16(
+// CHECK-CXX-LABEL: @_Z24test_svld1_hor_vnum_za16ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_vnum_za16(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_hor_vnum_za16(0, slice_base, 0, pg, ptr, vnum);
+  svld1_hor_vnum_za16(1, slice_base, 7, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_vnum_za32(
+// CHECK-CXX-LABEL: @_Z24test_svld1_hor_vnum_za32ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_vnum_za32(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_hor_vnum_za32(0, slice_base, 0, pg, ptr, vnum);
+  svld1_hor_vnum_za32(3, slice_base, 3, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_vnum_za64(
+// CHECK-CXX-LABEL: @_Z24test_svld1_hor_vnum_za64ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 7, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_vnum_za64(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_hor_vnum_za64(0, slice_base, 0, pg, ptr, vnum);
+  svld1_hor_vnum_za64(7, slice_base, 1, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_hor_vnum_za128(
+// CHECK-CXX-LABEL: @_Z25test_svld1_hor_vnum_za128ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_hor_vnum_za128(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_hor_vnum_za128(0, slice_base, 0, pg, ptr, vnum);
+  svld1_hor_vnum_za128(15, slice_base, 0, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_hor_za8(
+// CHECK-CXX-LABEL: @_Z22test_svld1_ver_hor_za8ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP0]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP1:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[TMP1]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1b.vert(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[TMP1]], i32 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_hor_za8(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_ver_vnum_za8(0, slice_base, 0, pg, ptr, vnum);
+  svld1_ver_vnum_za8(0, slice_base, 15, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_vnum_za16(
+// CHECK-CXX-LABEL: @_Z24test_svld1_ver_vnum_za16ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_vnum_za16(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_ver_vnum_za16(0, slice_base, 0, pg, ptr, vnum);
+  svld1_ver_vnum_za16(1, slice_base, 7, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_vnum_za32(
+// CHECK-CXX-LABEL: @_Z24test_svld1_ver_vnum_za32ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_vnum_za32(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_ver_vnum_za32(0, slice_base, 0, pg, ptr, vnum);
+  svld1_ver_vnum_za32(3, slice_base, 3, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_vnum_za64(
+// CHECK-CXX-LABEL: @_Z24test_svld1_ver_vnum_za64ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 7, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_vnum_za64(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_ver_vnum_za64(0, slice_base, 0, pg, ptr, vnum);
+  svld1_ver_vnum_za64(7, slice_base, 1, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svld1_ver_vnum_za128(
+// CHECK-CXX-LABEL: @_Z25test_svld1_ver_vnum_za128ju10__SVBool_tPKvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.ld1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svld1_ver_vnum_za128(uint32_t slice_base, svbool_t pg, const void *ptr, int64_t vnum) {
+  svld1_ver_vnum_za128(0, slice_base, 0, pg, ptr, vnum);
+  svld1_ver_vnum_za128(15, slice_base, 0, pg, ptr, vnum);
+}

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_st1.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s -check-prefixes=CHECK,CHECK-C
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefixes=CHECK,CHECK-CXX
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+#ifdef DISABLE_SME_ATTRIBUTES
+#define ARM_STREAMING_ATTR
+#else
+#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
+#endif
+
+// CHECK-C-LABEL:   @test_svst1_hor_za8(
+// CHECK-CXX-LABEL: @_Z18test_svst1_hor_za8ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY:ptr|i8\*]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[PTR]], i32 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_za8(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za8(0, slice_base, 0, pg, ptr);
+  svst1_hor_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_za16(
+// CHECK-CXX-LABEL: @_Z19test_svst1_hor_za16ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_za16(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za16(0, slice_base, 0, pg, ptr);
+  svst1_hor_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_za32(
+// CHECK-CXX-LABEL: @_Z19test_svst1_hor_za32ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_za32(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za32(0, slice_base, 0, pg, ptr);
+  svst1_hor_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_za64(
+// CHECK-CXX-LABEL: @_Z19test_svst1_hor_za64ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_za64(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za64(0, slice_base, 0, pg, ptr);
+  svst1_hor_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_za128(
+// CHECK-CXX-LABEL: @_Z20test_svst1_hor_za128ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_za128(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_hor_za128(0, slice_base, 0, pg, ptr);
+  svst1_hor_za128(15, slice_base, 0, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_za8(
+// CHECK-CXX-LABEL: @_Z18test_svst1_ver_za8ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[PTR]], i32 0, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_za8(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za8(0, slice_base, 0, pg, ptr);
+  svst1_ver_za8(0, slice_base, 15, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_za16(
+// CHECK-CXX-LABEL: @_Z19test_svst1_ver_za16ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 1, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_za16(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za16(0, slice_base, 0, pg, ptr);
+  svst1_ver_za16(1, slice_base, 7, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_za32(
+// CHECK-CXX-LABEL: @_Z19test_svst1_ver_za32ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 3, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_za32(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za32(0, slice_base, 0, pg, ptr);
+  svst1_ver_za32(3, slice_base, 3, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_za64(
+// CHECK-CXX-LABEL: @_Z19test_svst1_ver_za64ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE1:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 7, i32 [[TILESLICE1]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_za64(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za64(0, slice_base, 0, pg, ptr);
+  svst1_ver_za64(7, slice_base, 1, pg, ptr);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_za128(
+// CHECK-CXX-LABEL: @_Z20test_svst1_ver_za128ju10__SVBool_tPv(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR:%.*]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[PTR]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_za128(uint32_t slice_base, svbool_t pg, void *ptr) {
+  svst1_ver_za128(0, slice_base, 0, pg, ptr);
+  svst1_ver_za128(15, slice_base, 0, pg, ptr);
+}

clang/test/CodeGen/aarch64-sme-intrinsics/acle_sme_st1_vnum.c

+// REQUIRES: aarch64-registered-target
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - %s | FileCheck %s -check-prefixes=CHECK,CHECK-C
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -emit-llvm -o - -x c++ %s | FileCheck %s -check-prefixes=CHECK,CHECK-CXX
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -S -O1 -Werror -o /dev/null %s
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+#ifdef DISABLE_SME_ATTRIBUTES
+#define ARM_STREAMING_ATTR
+#else
+#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
+#endif
+
+// CHECK-C-LABEL:   @test_svst1_hor_vnum_za8(
+// CHECK-CXX-LABEL: @_Z23test_svst1_hor_vnum_za8ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP0]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP1:%.*]] = getelementptr i8, [[PTRTY:ptr|i8\*]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[TMP1]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.horiz(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[TMP1]], i32 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_vnum_za8(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_hor_vnum_za8(0, slice_base, 0, pg, ptr, vnum);
+  svst1_hor_vnum_za8(0, slice_base, 15, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_vnum_za16(
+// CHECK-CXX-LABEL: @_Z24test_svst1_hor_vnum_za16ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.horiz(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_vnum_za16(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_hor_vnum_za16(0, slice_base, 0, pg, ptr, vnum);
+  svst1_hor_vnum_za16(1, slice_base, 7, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_vnum_za32(
+// CHECK-CXX-LABEL: @_Z24test_svst1_hor_vnum_za32ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.horiz(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_vnum_za32(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_hor_vnum_za32(0, slice_base, 0, pg, ptr, vnum);
+  svst1_hor_vnum_za32(3, slice_base, 3, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_vnum_za64(
+// CHECK-CXX-LABEL: @_Z24test_svst1_hor_vnum_za64ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.horiz(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 7, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_vnum_za64(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_hor_vnum_za64(0, slice_base, 0, pg, ptr, vnum);
+  svst1_hor_vnum_za64(7, slice_base, 1, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_hor_vnum_za128(
+// CHECK-CXX-LABEL: @_Z25test_svst1_hor_vnum_za128ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.horiz(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_hor_vnum_za128(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_hor_vnum_za128(0, slice_base, 0, pg, ptr, vnum);
+  svst1_hor_vnum_za128(15, slice_base, 0, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_vnum_za8(
+// CHECK-CXX-LABEL: @_Z23test_svst1_ver_vnum_za8ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP0]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP1:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG:%.*]], [[PTRTY]] [[TMP1]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 15
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1b.vert(<vscale x 16 x i1> [[PG]], [[PTRTY]] [[TMP1]], i32 0, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_vnum_za8(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_ver_vnum_za8(0, slice_base, 0, pg, ptr, vnum);
+  svst1_ver_vnum_za8(0, slice_base, 15, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_vnum_za16(
+// CHECK-CXX-LABEL: @_Z24test_svst1_ver_vnum_za16ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 8 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv8i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 7
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1h.vert(<vscale x 8 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 1, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_vnum_za16(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_ver_vnum_za16(0, slice_base, 0, pg, ptr, vnum);
+  svst1_ver_vnum_za16(1, slice_base, 7, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_vnum_za32(
+// CHECK-CXX-LABEL: @_Z24test_svst1_ver_vnum_za32ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 4 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv4i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 3
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1w.vert(<vscale x 4 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 3, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_vnum_za32(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_ver_vnum_za32(0, slice_base, 0, pg, ptr, vnum);
+  svst1_ver_vnum_za32(3, slice_base, 3, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_vnum_za64(
+// CHECK-CXX-LABEL: @_Z24test_svst1_ver_vnum_za64ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 2 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv2i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        [[TILESLICE2:%.*]] = add i32 [[SLICE_BASE]], 1
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1d.vert(<vscale x 2 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 7, i32 [[TILESLICE2]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_vnum_za64(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_ver_vnum_za64(0, slice_base, 0, pg, ptr, vnum);
+  svst1_ver_vnum_za64(7, slice_base, 1, pg, ptr, vnum);
+}
+
+// CHECK-C-LABEL:   @test_svst1_ver_vnum_za128(
+// CHECK-CXX-LABEL: @_Z25test_svst1_ver_vnum_za128ju10__SVBool_tPvl(
+// CHECK-NEXT:      entry:
+// CHECK-NEXT:        [[TMP0:%.*]] = tail call <vscale x 1 x i1> @llvm.aarch64.sve.convert.from.svbool.nxv1i1(<vscale x 16 x i1> [[PG:%.*]])
+// CHECK-NEXT:        [[TMP1:%.*]] = tail call i64 @llvm.aarch64.sme.cntsb()
+// CHECK-NEXT:        [[MULVL:%.*]] = mul i64 [[TMP1]], [[VNUM:%.*]]
+// CHECK-NEXT:        [[TMP2:%.*]] = getelementptr i8, [[PTRTY]] [[PTR:%.*]], i64 [[MULVL]]
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 0, i32 [[SLICE_BASE:%.*]])
+// CHECK-NEXT:        tail call void @llvm.aarch64.sme.st1q.vert(<vscale x 1 x i1> [[TMP0]], [[PTRTY]] [[TMP2]], i32 15, i32 [[SLICE_BASE]])
+// CHECK-NEXT:        ret void
+//
+ARM_STREAMING_ATTR void test_svst1_ver_vnum_za128(uint32_t slice_base, svbool_t pg, void *ptr, int64_t vnum) {
+  svst1_ver_vnum_za128(0, slice_base, 0, pg, ptr, vnum);
+  svst1_ver_vnum_za128(15, slice_base, 0, pg, ptr, vnum);
+}

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_imm.cpp

+// REQUIRES: aarch64-registered-target
+
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fsyntax-only -verify -verify-ignore-unexpected=error %s
+// RUN: %clang_cc1 -DDISABLE_SME_ATTRIBUTES -DSVE_OVERLOADED_FORMS -triple aarch64-none-linux-gnu -target-feature +sme -target-feature +sve -fsyntax-only -verify -verify-ignore-unexpected=error %s
+
+#ifdef SVE_OVERLOADED_FORMS
+// A simple used,unused... macro, long enough to represent any SVE builtin.
+#define SVE_ACLE_FUNC(A1,A2_UNUSED,A3,A4_UNUSED) A1##A3
+#else
+#define SVE_ACLE_FUNC(A1,A2,A3,A4) A1##A2##A3##A4
+#endif
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+#ifdef DISABLE_SME_ATTRIBUTES
+#define ARM_STREAMING_ATTR
+#else
+#define ARM_STREAMING_ATTR __attribute__((arm_streaming))
+#endif
+
+ARM_STREAMING_ATTR
+void test_range_0_0(svbool_t pg, void *ptr) {
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svld1_hor_za8,,,)(-1, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 1 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svst1_ver_za8,,,)(1, -1, 15, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svld1_hor_za128,,,)(0, -1, -1, pg, ptr);
+  // expected-error@+1 {{argument value 1 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svst1_ver_za128,,,)(15, -1, 1, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za8,,,)(-1, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 1 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za8,,,)(1, -1, 15, pg, ptr, 1);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za128,,,)(0, -1, -1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 1 is outside the valid range [0, 0]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za128,,,)(15, -1, 1, pg, ptr, 1);
+}
+
+ARM_STREAMING_ATTR
+void test_range_0_1(svbool_t pg, void *ptr) {
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svld1_hor_za16,,,)(-1, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 2 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svst1_ver_za16,,,)(2, -1, 7, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svld1_hor_za64,,,)(0, -1, -1, pg, ptr);
+  // expected-error@+1 {{argument value 2 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svst1_ver_za64,,,)(7, -1, 2, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za16,,,)(-1, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 2 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za16,,,)(2, -1, 7, pg, ptr, 1);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za64,,,)(0, -1, -1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 2 is outside the valid range [0, 1]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za64,,,)(7, -1, 2, pg, ptr, 1);
+}
+
+ARM_STREAMING_ATTR
+void test_range_0_3(svbool_t pg, void *ptr) {
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svld1_hor_za32,,,)(-1, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 4 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svst1_ver_za32,,,)(4, -1, 3, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svld1_hor_za32,,,)(0, -1, -1, pg, ptr);
+  // expected-error@+1 {{argument value 4 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svst1_ver_za32,,,)(3, -1, 4, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za32,,,)(-1, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 4 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za32,,,)(4, -1, 3, pg, ptr, 1);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za32,,,)(0, -1, -1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 4 is outside the valid range [0, 3]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za32,,,)(3, -1, 4, pg, ptr, 1);
+}
+
+ARM_STREAMING_ATTR
+void test_range_0_7(svbool_t pg, void *ptr) {
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svld1_hor_za64,,,)(-1, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 8 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svst1_ver_za64,,,)(8, -1, 1, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svld1_hor_za16,,,)(0, -1, -1, pg, ptr);
+  // expected-error@+1 {{argument value 8 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svst1_ver_za16,,,)(1, -1, 8, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za64,,,)(-1, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 8 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za64,,,)(8, -1, 1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za16,,,)(0, -1, -1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 8 is outside the valid range [0, 7]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za16,,,)(1, -1, 8, pg, ptr, 1);
+}
+
+ARM_STREAMING_ATTR
+void test_range_0_15(svbool_t pg, void *ptr) {
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svld1_hor_za128,,,)(-1, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 16 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svst1_ver_za128,,,)(16, -1, 0, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svld1_hor_za8,,,)(0, -1, -1, pg, ptr);
+  // expected-error@+1 {{argument value 16 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svst1_ver_za8,,,)(0, -1, 16, pg, ptr);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za128,,,)(-1, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 16 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za128,,,)(16, -1, 0, pg, ptr, 1);
+  // expected-error@+1 {{argument value 18446744073709551615 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za8,,,)(0, -1, -1, pg, ptr, 1);
+  // expected-error@+1 {{argument value 16 is outside the valid range [0, 15]}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za8,,,)(0, -1, 16, pg, ptr, 1);
+}
+
+ARM_STREAMING_ATTR
+void test_constant(uint64_t u64, svbool_t pg, void *ptr) {
+  SVE_ACLE_FUNC(svld1_hor_za8,,,)(u64, u64, 0, pg, ptr);  // expected-error {{argument to 'svld1_hor_za8' must be a constant integer}}
+  SVE_ACLE_FUNC(svld1_ver_za16,,,)(0, u64, u64, pg, ptr); // expected-error {{argument to 'svld1_ver_za16' must be a constant integer}}
+  SVE_ACLE_FUNC(svst1_hor_za32,,,)(u64, u64, 0, pg, ptr); // expected-error {{argument to 'svst1_hor_za32' must be a constant integer}}
+  SVE_ACLE_FUNC(svst1_ver_za64,,,)(0, u64, u64, pg, ptr); // expected-error {{argument to 'svst1_ver_za64' must be a constant integer}}
+  SVE_ACLE_FUNC(svld1_hor_vnum_za8,,,)(u64, u64, 0, pg, ptr, u64);  // expected-error {{argument to 'svld1_hor_vnum_za8' must be a constant integer}}
+  SVE_ACLE_FUNC(svld1_ver_vnum_za16,,,)(0, u64, u64, pg, ptr, u64); // expected-error {{argument to 'svld1_ver_vnum_za16' must be a constant integer}}
+  SVE_ACLE_FUNC(svst1_hor_vnum_za32,,,)(u64, u64, 0, pg, ptr, u64); // expected-error {{argument to 'svst1_hor_vnum_za32' must be a constant integer}}
+  SVE_ACLE_FUNC(svst1_ver_vnum_za64,,,)(0, u64, u64, pg, ptr, u64); // expected-error {{argument to 'svst1_ver_vnum_za64' must be a constant integer}}
+}

clang/test/Sema/aarch64-sme-intrinsics/acle_sme_target.c

+// RUN: %clang_cc1 -triple aarch64-none-linux-gnu -target-feature +sve -fsyntax-only -verify -emit-llvm -o - %s
+// REQUIRES: aarch64-registered-target
+
+// Test that functions with the correct target attributes can use the correct SME intrinsics.
+
+#include <arm_sme_draft_spec_subject_to_change.h>
+
+__attribute__((target("sme")))
+void test_sme(svbool_t pg, void *ptr) {
+  svld1_hor_za8(0, 0, 0, pg, ptr);
+}
+
+__attribute__((target("arch=armv8-a+sme")))
+void test_arch_sme(svbool_t pg, void *ptr) {
+  svld1_hor_vnum_za32(0, 0, 0, pg, ptr, 0);
+}
+
+__attribute__((target("+sme")))
+void test_plus_sme(svbool_t pg, void *ptr) {
+  svst1_ver_za16(0, 0, 0, pg, ptr);
+}
+
+void undefined(svbool_t pg, void *ptr) {
+  svst1_ver_vnum_za64(0, 0, 0, pg, ptr, 0); // expected-error {{'svst1_ver_vnum_za64' needs target feature sme}}
+}

clang/utils/TableGen/SveEmitter.cpp

unsigned getSplatIdx() const {
     // These prototype modifiers are described in arm_sve.td.
     auto Idx = Proto.find_first_of("ajfrKLR@");
     assert(Idx != std::string::npos && Idx > 0 &&
            "Prototype has no splat operand");
     return Idx - 1;
   }
 
   /// Emits the intrinsic declaration to the ostream.
-  void emitIntrinsic(raw_ostream &OS) const;
+  void emitIntrinsic(raw_ostream &OS, SVEEmitter &Emitter) const;
 
 private:
   std::string getMergeSuffix() const { return MergeSuffix; }
   std::string mangleName(ClassKind LocalCK) const;
   std::string replaceTemplatedArgs(std::string Name, TypeSpec TS,
                                    std::string Proto) const;
 };
 
public:
   void createCodeGenMap(raw_ostream &o);
 
   /// Emit all the range checks for the immediates.
   void createRangeChecks(raw_ostream &o);
 
   /// Create the SVETypeFlags used in CGBuiltins
   void createTypeFlags(raw_ostream &o);
 
+  /// Emit arm_sme.h.
+  void createSMEHeader(raw_ostream &o);
+
+  /// Emit all the SME __builtin prototypes and code needed by Sema.
+  void createSMEBuiltins(raw_ostream &o);
+
+  /// Emit all the information needed to map builtin -> LLVM IR intrinsic.
+  void createSMECodeGenMap(raw_ostream &o);
+
+  /// Emit all the range checks for the immediates.
+  void createSMERangeChecks(raw_ostream &o);
+
   /// Create intrinsic and add it to \p Out
-  void createIntrinsic(Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out);
+  void createIntrinsic(Record *R,
+                       SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out);
 };
 
 } // end anonymous namespace
 
 
 //===----------------------------------------------------------------------===//
 // Type implementation
 //===----------------------------------------------------------------------===//
case 's':
       ElementBitwidth = 16;
       break;
     case 'i':
       ElementBitwidth = 32;
       break;
     case 'l':
       ElementBitwidth = 64;
       break;
+    case 'q':
+      ElementBitwidth = 128;
+      break;
     case 'h':
       Float = true;
       ElementBitwidth = 16;
       break;
     case 'f':
       Float = true;
       ElementBitwidth = 32;
       break;
case 'Y':
     break;
   case 'U':
     Constant = true;
     Pointer = true;
     ElementBitwidth = Bitwidth = 32;
     NumVectors = 0;
     Signed = true;
     break;
+  case '%':
+    Pointer = true;
+    Void = true;
+    NumVectors = 0;
+    break;
   case 'A':
     Pointer = true;
     ElementBitwidth = Bitwidth = 8;
     NumVectors = 0;
     Signed = true;
     break;
   case 'B':
     Pointer = true;
Intrinsic::Intrinsic(StringRef Name, StringRef Proto, uint64_t MergeTy,
   this->Flags |= Emitter.encodeMergeType(MergeTy);
   if (hasSplat())
     this->Flags |= Emitter.encodeSplatOperand(getSplatIdx());
 }
 
 std::string Intrinsic::getBuiltinTypeStr() {
   std::string S = getReturnType().builtin_str();
   for (unsigned I = 0; I < getNumParams(); ++I)
     S += getParamType(I).builtin_str();
 
   return S;
 }
 
 std::string Intrinsic::replaceTemplatedArgs(std::string Name, TypeSpec TS,
                                             std::string Proto) const {
   std::string Ret = Name;
   while (Ret.find('{') != std::string::npos) {
if (LocalCK == ClassG) {
     }
   }
 
   // Replace all {d} like expressions with e.g. 'u32'
   return replaceTemplatedArgs(S, getBaseTypeSpec(), getProto()) +
          getMergeSuffix();
 }
 
-void Intrinsic::emitIntrinsic(raw_ostream &OS) const {
+void Intrinsic::emitIntrinsic(raw_ostream &OS, SVEEmitter &Emitter) const {
   bool IsOverloaded = getClassKind() == ClassG && getProto().size() > 1;
 
   std::string FullName = mangleName(ClassS);
   std::string ProtoName = mangleName(getClassKind());
+  std::string SMEAttrs = "";
+
+  if (Flags & Emitter.getEnumValueForFlag("IsStreaming"))
+    SMEAttrs += ", arm_streaming";
+  if (Flags & Emitter.getEnumValueForFlag("IsStreamingCompatible"))
+    SMEAttrs += ", arm_streaming_compatible";
+  if (Flags & Emitter.getEnumValueForFlag("IsSharedZA"))
+    SMEAttrs += ", arm_shared_za";
+  if (Flags & Emitter.getEnumValueForFlag("IsPreservesZA"))
+    SMEAttrs += ", arm_preserves_za";
 
   OS << (IsOverloaded ? "__aio " : "__ai ")
      << "__attribute__((__clang_arm_builtin_alias("
-     << "__builtin_sve_" << FullName << ")))\n";
+     << (SMEAttrs.empty() ? "__builtin_sve_" : "__builtin_sme_")
+     << FullName << ")";
+  if (!SMEAttrs.empty())
+    OS << SMEAttrs;
+  OS << "))\n";
 
   OS << getTypes()[0].str() << " " << ProtoName << "(";
   for (unsigned I = 0; I < getTypes().size() - 1; ++I) {
     if (I != 0)
       OS << ", ";
     OS << getTypes()[I + 1].str();
   }
   OS << ");\n";
uint64_t SVEEmitter::encodeTypeFlags(const SVEType &T) {
   case 8:
     return encodeEltType("EltTyInt8");
   case 16:
     return encodeEltType("EltTyInt16");
   case 32:
     return encodeEltType("EltTyInt32");
   case 64:
     return encodeEltType("EltTyInt64");
+  case 128:
+    return encodeEltType("EltTyInt128");
   default:
     llvm_unreachable("Unhandled integer element bitwidth!");
   }
 }
 
 void SVEEmitter::createIntrinsic(
     Record *R, SmallVectorImpl<std::unique_ptr<Intrinsic>> &Out) {
   StringRef Name = R->getValueAsString("Name");
   StringRef Proto = R->getValueAsString("Prototype");
   StringRef Types = R->getValueAsString("Types");
   StringRef Guard = R->getValueAsString("TargetGuard");
   StringRef LLVMName = R->getValueAsString("LLVMIntrinsic");
   uint64_t Merge = R->getValueAsInt("Merge");
   StringRef MergeSuffix = R->getValueAsString("MergeSuffix");
   uint64_t MemEltType = R->getValueAsInt("MemEltType");
   std::vector<Record*> FlagsList = R->getValueAsListOfDefs("Flags");
   std::vector<Record*> ImmCheckList = R->getValueAsListOfDefs("ImmChecks");
 
   int64_t Flags = 0;
   for (auto FlagRec : FlagsList)
     Flags |= FlagRec->getValueAsInt("Value");
 
   // Create a dummy TypeSpec for non-overloaded builtins.
   if (Types.empty()) {
     assert((Flags & getEnumValueForFlag("IsOverloadNone")) &&
            "Expect TypeSpec for overloaded builtin!");
     Types = "i";
   }
 
   // Extract type specs from string
   SmallVector<TypeSpec, 8> TypeSpecs;
std::stable_sort(
         auto ToTuple = [](const std::unique_ptr<Intrinsic> &I) {
           return std::make_tuple(I->getGuard(), (unsigned)I->getClassKind(), I->getName());
         };
         return ToTuple(A) < ToTuple(B);
       });
 
   // Actually emit the intrinsic declarations.
   for (auto &I : Defs)
-    I->emitIntrinsic(OS);
+    I->emitIntrinsic(OS, *this);
 
   OS << "#define svcvtnt_bf16_x      svcvtnt_bf16_m\n";
   OS << "#define svcvtnt_bf16_f32_x  svcvtnt_bf16_f32_m\n";
 
   OS << "#define svcvtnt_f16_x      svcvtnt_f16_m\n";
   OS << "#define svcvtnt_f16_f32_x  svcvtnt_f16_f32_m\n";
   OS << "#define svcvtnt_f32_x      svcvtnt_f32_m\n";
   OS << "#define svcvtnt_f32_f64_x  svcvtnt_f32_f64_m\n\n";
void SVEEmitter::createTypeFlags(raw_ostream &OS) {
   OS << "#endif\n\n";
 
   OS << "#ifdef LLVM_GET_SVE_IMMCHECKTYPES\n";
   for (auto &KV : ImmCheckTypes)
     OS << "  " << KV.getKey() << " = " << KV.getValue() << ",\n";
   OS << "#endif\n\n";
 }
 
+void SVEEmitter::createSMEHeader(raw_ostream &OS) {
+  OS << "/*===---- arm_sme_draft_spec_subject_to_change.h - ARM SME intrinsics "
+        "------===\n"
+        " *\n"
+        " *\n"
+        " * Part of the LLVM Project, under the Apache License v2.0 with LLVM "
+        "Exceptions.\n"
+        " * See https://llvm.org/LICENSE.txt for license information.\n"
+        " * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception\n"
+        " *\n"
+        " *===-----------------------------------------------------------------"
+        "------===\n"
+        " */\n\n";
+
+  OS << "#ifndef __ARM_SME_H\n";
+  OS << "#define __ARM_SME_H\n\n";
+
+  OS << "#if !defined(__LITTLE_ENDIAN__)\n";
+  OS << "#error \"Big endian is currently not supported for arm_sme_draft_spec_subject_to_change.h\"\n";
+  OS << "#endif\n";
+
+  OS << "#include <arm_sve.h> \n\n";
+
+  OS << "/* Function attributes */\n";
+  OS << "#define __ai static __inline__ __attribute__((__always_inline__, "
+        "__nodebug__))\n\n";
+
+  OS << "#ifdef  __cplusplus\n";
+  OS << "extern \"C\" {\n";
+  OS << "#endif\n\n";
+
+  SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  for (auto *R : RV)
+    createIntrinsic(R, Defs);
+
+  // Sort intrinsics in header file by following order/priority similar to SVE:
+  // - Architectural guard
+  // - Class (is intrinsic overloaded or not)
+  // - Intrinsic name
+  std::stable_sort(Defs.begin(), Defs.end(),
+                   [](const std::unique_ptr<Intrinsic> &A,
+                      const std::unique_ptr<Intrinsic> &B) {
+                     auto ToTuple = [](const std::unique_ptr<Intrinsic> &I) {
+                       return std::make_tuple(I->getGuard(),
+                                              (unsigned)I->getClassKind(),
+                                              I->getName());
+                     };
+                     return ToTuple(A) < ToTuple(B);
+                   });
+
+  // Actually emit the intrinsic declaration.
+  for (auto &I : Defs) {
+    I->emitIntrinsic(OS, *this);
+  }
+
+  OS << "#ifdef __cplusplus\n";
+  OS << "} // extern \"C\"\n";
+  OS << "#endif\n\n";
+  OS << "#undef __ai\n\n";
+  OS << "#endif /* __ARM_SME_H */\n";
+}
+
+void SVEEmitter::createSMEBuiltins(raw_ostream &OS) {
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
+  for (auto *R : RV) {
+    createIntrinsic(R, Defs);
+  }
+
+  // The mappings must be sorted based on BuiltinID.
+  llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
+                      const std::unique_ptr<Intrinsic> &B) {
+    return A->getMangledName() < B->getMangledName();
+  });
+
+  OS << "#ifdef GET_SME_BUILTINS\n";
+  for (auto &Def : Defs) {
+    // Only create BUILTINs for non-overloaded intrinsics, as overloaded
+    // declarations only live in the header file.
+    if (Def->getClassKind() != ClassG)
+      OS << "TARGET_BUILTIN(__builtin_sme_" << Def->getMangledName() << ", \""
+         << Def->getBuiltinTypeStr() << "\", \"n\", \"" << Def->getGuard()
+         << "\")\n";
+  }
+
+  OS << "#endif\n\n";
+}
+
+void SVEEmitter::createSMECodeGenMap(raw_ostream &OS) {
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
+  for (auto *R : RV) {
+    createIntrinsic(R, Defs);
+  }
+
+  // The mappings must be sorted based on BuiltinID.
+  llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
+                      const std::unique_ptr<Intrinsic> &B) {
+    return A->getMangledName() < B->getMangledName();
+  });
+
+  OS << "#ifdef GET_SME_LLVM_INTRINSIC_MAP\n";
+  for (auto &Def : Defs) {
+    // Builtins only exist for non-overloaded intrinsics, overloaded
+    // declarations only live in the header file.
+    if (Def->getClassKind() == ClassG)
+      continue;
+
+    uint64_t Flags = Def->getFlags();
+    auto FlagString = std::to_string(Flags);
+
+    std::string LLVMName = Def->getLLVMName();
+    std::string Builtin = Def->getMangledName();
+    if (!LLVMName.empty())
+      OS << "SMEMAP1(" << Builtin << ", " << LLVMName << ", " << FlagString
+         << "),\n";
+    else
+      OS << "SMEMAP2(" << Builtin << ", " << FlagString << "),\n";
+  }
+  OS << "#endif\n\n";
+}
+
+void SVEEmitter::createSMERangeChecks(raw_ostream &OS) {
+  std::vector<Record *> RV = Records.getAllDerivedDefinitions("Inst");
+  SmallVector<std::unique_ptr<Intrinsic>, 128> Defs;
+  for (auto *R : RV) {
+    createIntrinsic(R, Defs);
+  }
+
+  // The mappings must be sorted based on BuiltinID.
+  llvm::sort(Defs, [](const std::unique_ptr<Intrinsic> &A,
+                      const std::unique_ptr<Intrinsic> &B) {
+    return A->getMangledName() < B->getMangledName();
+  });
+
+
+  OS << "#ifdef GET_SME_IMMEDIATE_CHECK\n";
+
+  // Ensure these are only emitted once.
+  std::set<std::string> Emitted;
+
+  for (auto &Def : Defs) {
+    if (Emitted.find(Def->getMangledName()) != Emitted.end() ||
+        Def->getImmChecks().empty())
+      continue;
+
+    OS << "case SME::BI__builtin_sme_" << Def->getMangledName() << ":\n";
+    for (auto &Check : Def->getImmChecks())
+      OS << "ImmChecks.push_back(std::make_tuple(" << Check.getArg() << ", "
+         << Check.getKind() << ", " << Check.getElementSizeInBits() << "));\n";
+    OS << "  break;\n";
+
+    Emitted.insert(Def->getMangledName());
+  }
+
+  OS << "#endif\n\n";
+}
+
 namespace clang {
 void EmitSveHeader(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createHeader(OS);
 }
 
 void EmitSveBuiltins(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createBuiltins(OS);
 }
 
 void EmitSveBuiltinCG(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createCodeGenMap(OS);
 }
 
 void EmitSveRangeChecks(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createRangeChecks(OS);
 }
 
 void EmitSveTypeFlags(RecordKeeper &Records, raw_ostream &OS) {
   SVEEmitter(Records).createTypeFlags(OS);
 }
 
+void EmitSmeHeader(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).createSMEHeader(OS);
+}
+
+void EmitSmeBuiltins(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).createSMEBuiltins(OS);
+}
+
+void EmitSmeBuiltinCG(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).createSMECodeGenMap(OS);
+}
+
+void EmitSmeRangeChecks(RecordKeeper &Records, raw_ostream &OS) {
+  SVEEmitter(Records).createSMERangeChecks(OS);
+}
 } // End namespace clang

clang/utils/TableGen/TableGen.cpp

enum ActionType {
   GenArmMveBuiltinSema,
   GenArmMveBuiltinCG,
   GenArmMveBuiltinAliases,
   GenArmSveHeader,
   GenArmSveBuiltins,
   GenArmSveBuiltinCG,
   GenArmSveTypeFlags,
   GenArmSveRangeChecks,
+  GenArmSmeHeader,
+  GenArmSmeBuiltins,
+  GenArmSmeBuiltinCG,
+  GenArmSmeRangeChecks,
   GenArmCdeHeader,
   GenArmCdeBuiltinDef,
   GenArmCdeBuiltinSema,
   GenArmCdeBuiltinCG,
   GenArmCdeBuiltinAliases,
   GenRISCVVectorHeader,
   GenRISCVVectorBuiltins,
   GenRISCVVectorBuiltinCG,
cl::values(
         clEnumValN(GenArmSveBuiltins, "gen-arm-sve-builtins",
                    "Generate arm_sve_builtins.inc for clang"),
         clEnumValN(GenArmSveBuiltinCG, "gen-arm-sve-builtin-codegen",
                    "Generate arm_sve_builtin_cg_map.inc for clang"),
         clEnumValN(GenArmSveTypeFlags, "gen-arm-sve-typeflags",
                    "Generate arm_sve_typeflags.inc for clang"),
         clEnumValN(GenArmSveRangeChecks, "gen-arm-sve-sema-rangechecks",
                    "Generate arm_sve_sema_rangechecks.inc for clang"),
+        clEnumValN(GenArmSmeHeader, "gen-arm-sme-header",
+                   "Generate arm_sme.h for clang"),
+        clEnumValN(GenArmSmeBuiltins, "gen-arm-sme-builtins",
+                   "Generate arm_sme_builtins.inc for clang"),
+        clEnumValN(GenArmSmeBuiltinCG, "gen-arm-sme-builtin-codegen",
+                   "Generate arm_sme_builtin_cg_map.inc for clang"),
+        clEnumValN(GenArmSmeRangeChecks, "gen-arm-sme-sema-rangechecks",
+                   "Generate arm_sme_sema_rangechecks.inc for clang"),
         clEnumValN(GenArmMveHeader, "gen-arm-mve-header",
                    "Generate arm_mve.h for clang"),
         clEnumValN(GenArmMveBuiltinDef, "gen-arm-mve-builtin-def",
                    "Generate ARM MVE builtin definitions for clang"),
         clEnumValN(GenArmMveBuiltinSema, "gen-arm-mve-builtin-sema",
                    "Generate ARM MVE builtin sema checks for clang"),
         clEnumValN(GenArmMveBuiltinCG, "gen-arm-mve-builtin-codegen",
                    "Generate ARM MVE builtin code-generator for clang"),
case GenArmSveBuiltinCG:
     EmitSveBuiltinCG(Records, OS);
     break;
   case GenArmSveTypeFlags:
     EmitSveTypeFlags(Records, OS);
     break;
   case GenArmSveRangeChecks:
     EmitSveRangeChecks(Records, OS);
     break;
+  case GenArmSmeHeader:
+    EmitSmeHeader(Records, OS);
+    break;
+  case GenArmSmeBuiltins:
+    EmitSmeBuiltins(Records, OS);
+    break;
+  case GenArmSmeBuiltinCG:
+    EmitSmeBuiltinCG(Records, OS);
+    break;
+  case GenArmSmeRangeChecks:
+    EmitSmeRangeChecks(Records, OS);
+    break;
   case GenArmCdeHeader:
     EmitCdeHeader(Records, OS);
     break;
   case GenArmCdeBuiltinDef:
     EmitCdeBuiltinDef(Records, OS);
     break;
   case GenArmCdeBuiltinSema:
     EmitCdeBuiltinSema(Records, OS);

clang/utils/TableGen/TableGenBackends.h

 void EmitNeonTest(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitSveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveTypeFlags(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitSveRangeChecks(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
+void EmitSmeHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+void EmitSmeBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+void EmitSmeBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+void EmitSmeRangeChecks(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
+
 void EmitMveHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinDef(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinSema(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinCG(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitMveBuiltinAliases(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 
 void EmitRVVHeader(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);
 void EmitRVVBuiltins(llvm::RecordKeeper &Records, llvm::raw_ostream &OS);