[compiler-rt][aarch64] Add SME ABI support routines.

[sdesmalen]

Summary

When compiling for SME and using the attributes to use PSTATE.ZA,
Clang will emit calls to SME ABI support routines to save and
restore ZA state.

[paulwalker-arm]

compiler-rt/lib/builtins/aarch64/sme-abi.S

16

I cannot see any uses of RDSVL_X15_1.

41

Is calling __arm_sme_state strictly necessary? Not sure if I've missed something but it looks like we only abort based on the value of TPIDR2_EL0 or the block to which it points. Which means we only abort when SME is known available and so can read SVCR directly?

42

Is masking necessary here? given you're only testing a single bit where you could use x0 directly.

48

Not sure if this merit in this idea but given you don't need to preserve state I wondered if a normal branch would be better here. This way any backtrace will show the parent SME ABI function, which might be more meaningful? This kind of going in hand with my following comments about not needing to persevere x29 and x30 within those ABI functions.

56–57

This looks weird. Shouldn't x16 already contain the contents of CPTR_EL3 plus you already know SME is not available by this point of execution.

63

Can this be hex so it's easier to understand what's happening?

67

You already know SME is available by this point. Looking at the code I think you it can be simplified to:

if SME not available goto A:
do SME stuff
ret;

A:
x0 = x1= 0;
reet

75

Is this necessary?

76

The calling convention says x14-x18 are free and looking at the code you're not using x17 or x18 so perhaps don't need to split x12?

90

Using addsvl here will remove the need for rdsvl and free up another register.

104

Is this necessary? I cannot see any legitimate changes of these registers.

105

As above there shouldn't be a need to use x12.

124

As above, can you use addsvl here?

143–144

By this point you already know SME is available and this the branch will never be taken.

166

Can you use xzr directly?

[MaskRay]

Add code owner @compnerd

compiler-rt/lib/builtins/aarch64/sme-abi.S

8

I feel that this guard is not necessary. we should just not compile these files for non-aarch64.

[rsandifo-arm]

compiler-rt/lib/builtins/aarch64/sme-abi.S

41

True, but it might be more future-proof to assume that “SME is available” is a sufficient but not necessary condition for “TPIDR2_EL0 is available”. So SME ⇒ TPIDR2_EL0 is safe but TPIDR2_EL0 ⇒ SME seems less safe.

41

Since this function can change VL, it should save the old VG value on the stack: https://github.com/ARM-software/abi-aa/blob/main/aadwarf64/aadwarf64.rst#changes-in-vector-length-alpha

[compnerd]

compiler-rt/lib/builtins/aarch64/sme-abi.S

8

I believe that CMake is already setup that way. __arch64__ may also be spelt _M_ARM64 or __arm64__. I think that doing what @MaskRay is suggesting here and dropping the guard is the better thing.

48

IIRC we did have some macros for function names. Please do not call abort here but rather with macro wrapping it to account for __USER_LABEL_PREFIX__.

[aemerson]

Can't we compile these instructions using the just-built clang and therefore directly use the real assembly mnemonics?

[sdesmalen]

Thanks everyone for your comments. I've tried to address all of them. It did mean rewriting large parts of the routines and adding some new functionality.

Changes:

• Avoid spilling/filling of x29 and x30, mostly by making the calls to do_abort a tail call.
• Added a new file sme-abi-init.c which adds code to test at compiler-rt startup whether SME is available on the platform.

[sdesmalen]

In D154045#4626984, @aemerson wrote:

Can't we compile these instructions using the just-built clang and therefore directly use the real assembly mnemonics?

We could, but there is no requirement that compiler-rt must be built with a compiler that supports these mnemonics, it's perfectly fine to build compiler-rt with an older compiler.

compiler-rt/lib/builtins/aarch64/sme-abi.S

8

Removing it makes sense. I mostly just copied what was done in other files.

41

Thanks, I didn't realise that was required!

48

Not sure if this merit in this idea but given you don't need to preserve state I wondered if a normal branch would be better here. This way any backtrace will show the parent SME ABI function, which might be more meaningful? This kind of going in hand with my following comments about not needing to persevere x29 and x30 within those ABI functions.

I think that you're right that we can use b instead of bl for the other ABI functions, but for do_abort specifically we'll need to use bl, as the unwinder may need to come back to this point to restore VG, as @rsandifo-arm pointed out.

75

I've tried to remove unnecessary spills/fills of x29/x30 (which mostly comes from tail call optimising do_abort)

76

I think it did this because only x12-x15 can be used for the ldr za addressing mode. If we rename some of the registers and use x17/x18, then spilling x12 is indeed unnecessary. Anyway, I've tried to address this in the latest revision.

[nsz]

compiler-rt/cmake/builtin-config-ix.cmake

47

AT_HWCAP2 and HWCAP2_SME are linux ABI (cannot change and no need to check if they are defined in the libc headers on a linux platform), only the getauxval symbol needs a check.

on non-linux platforms with SME support the flag may not be in AT_HWCAP2 nor at 1<<23 bit.

actually getauxval is the wrong symbol: calling that from compiler-rt violates iso c and posix link namespace so it can break conforming code.

glibc has __getauxval for this purpose (i guess bionic, musl etc needs to add this symbol too to allow checking auxval bits from a compiler runtime)

compiler-rt/lib/builtins/aarch64/sme-abi-init.c

7

why unsigned long and not say bool?

10

why a separate value when it is always the same as __aarch64_has_sme and always used together?

i think it is enough to add a separate symbol when there is logic that sets them differently (or used in different places but then the definitions should be in different TU for static linking).

25

i'm not sure if weakref is a good idea.

in case of static linking weakref does not cause link of the definition from a library archive so libc can only define it in a crt.o, which then cannot be optimized away when SME is not in use.

in case of shared compiler-rt lib this is ok i guess (with the caveat that -Wl,-as-needed would not link the shared lib with the definition, but since we expect the definition in libc.so that can be assumed to be linked for other reasons)

[sdesmalen]

Addressed @nsz's review comments, specifically:

• Merged __aarch64_has_sme and __aarch64_has_tpidr2_el0 into a single __aarch64_has_sme_and_tpidr2_el0
• Also changed the type of this variable from unsigned long -> _Bool
• Test for AT_HWCAP2 instead of getauxval(AT_HWCAP2) in CMake config
• Use __getauxval() instead of getauxval()
• No longer use weak ref for __aarch64_sme_accessible

[aemerson]

We should move this onto GitHub PRs.

Anyway, having a static initializer seems like a bad idea. It seems wasteful to force static initializers to be generated & run just so we can check if we have SME, which in reality is never going to be false if we're in these routines. @rsandifo-arm can we relax this check requirement?

[rsandifo-arm]

In D154045#4652092, @aemerson wrote:

We should move this onto GitHub PRs.

Anyway, having a static initializer seems like a bad idea. It seems wasteful to force static initializers to be generated & run just so we can check if we have SME, which in reality is never going to be false if we're in these routines. @rsandifo-arm can we relax this check requirement?

The requirement exists for streaming-compatible calls to normal functions.

A streaming-compatible function that calls printf (say) must force PSTATE.SM to 0 before the call and then restore it to its original value after the call. To do that, it needs to read the current PSTATE.SM. But in general, streaming-compatible functions are agnostic not just about whether the processor is in streaming mode, but also about whether SME is present at all. E.g. it's possible to compile a streaming-compatible function with base -march=armv8-a (without even SVE).

So a general streaming-compatible function can't read SVCR unconditionally. There needs to be a “safe” way of reading PSTATE.SM without knowing whether SME is present.

[nsz]

In D154045#4652199, @rsandifo-arm wrote:

A streaming-compatible function that calls printf (say) must force PSTATE.SM to 0 before the call and then restore it to its original value after the call. To do that, it needs to read the current PSTATE.SM. But in general, streaming-compatible functions are agnostic not just about whether the processor is in streaming mode, but also about whether SME is present at all. E.g. it's possible to compile a streaming-compatible function with base -march=armv8-a (without even SVE).

yeah and the unwinder has to deal with SM and ZA states too and must work without SME (and often not part of the libc so does not have direct information about the presence of SME).

so the libc needs an ABI to detect if SME is accessible, if this ABI is fast then no global initializer is needed for caching the result in principle:
e.g. the sme runtime functions can be ifunc dispatched (works on glibc) or the libc can set a flag at a fixed offset from TP.

i think it makes sense to have a generic solution that a platform may override with optimizations if necessary.

[sdesmalen]

• Use ldrb instead of ldr for loading _Bool (this solves relocation errors)
• Use /*AT_HWCAP2*/26 and /*HWCAP2_SME*/(1 << 23) directly without checking for support at compiler-rt build time. We might otherwise incorrectly assume SME is not available based on the target we compile for, even when the eventual runtime supports it.

[paulwalker-arm]

compiler-rt/cmake/Modules/AddCompilerRT.cmake

315

The next block suggests this can be just type?

compiler-rt/lib/builtins/aarch64/sme-abi-init.c

19

If missing defines is a concern the following idiom is better because it means we'll use the correct value if for some reason it is changed.

#ifndef AT_HWCAP2                                                                   
#define AT_HWCAP2 26                                                                
#endif

I can see this is the style used by lib/builtins/cpu_model.c also.

39

It is defined somewhere what 90 means? and thus why that's the chosen priority.

compiler-rt/lib/builtins/aarch64/sme-abi.S

7

Is it worth pointing to the ABI specification as documentation for these routines?

14

RDSVL_X14_1 looks unused?

30

I ask this below but can we not assume SVE is available?

44

My dwarf knowledge is not great but is this safe? I mean you're essentially marking uninitialised data until you actually store the value of VG. What happens if we start unwinding from inside the call to __arm_sme_state? Is it possible to store the result of cntd as part of the prologue or can we not assume SVE is available?

70–71

Are the uses of SYMBOL_NAME required here? You're defining the name above so I'm assuming it cannot be changed by the time you get here?

120

As above, is SYMBOL_NAME required here?

[aemerson]

compiler-rt/lib/builtins/aarch64/sme-abi.S

70–71

For cross-platform code, this is needed so that the right prefix gets added to the symbol name. For example on Darwin it prepends "_".

[sdesmalen]

Addressed review comments:

• Use #ifndef <macro>, #define <macro> for HWCAP macros.
• Removed SYMBOL_NAME(..) for cases where mangling is not allowed (the public ABI routines)
• Call cntd earlier to populate VG value on the stack.
• Use manual CFI expression for VG if the compiler does not support SME assembler syntax.

compiler-rt/lib/builtins/aarch64/sme-abi-init.c

39

https://clang.llvm.org/docs/AttributeReference.html#constructor

Specifically:

The priority is given as an integer constant expression between 101 and 65535 (inclusive). Priorities outside of that range are reserved for use by the implementation. A lower value indicates a higher priority of initialization.

I interpreted this as priorities between the range 101 and 65535 being user-space initialisation, meaning that a value of 90 will mean this constructor is called as part of the compiler's (implementation, i.e.) internal runtime and executed before any user-space initialisation.

compiler-rt/lib/builtins/aarch64/sme-abi.S

30

At runtime, yes, but the compiler may not be able to support the asm syntax.

44

Good point, I think this is indeed unsafe. We can assume that SVE is available at runtime, so we can execute cntd earlier.

[paulwalker-arm]

A couple of extra optimisations but otherwise this looks good to me.

compiler-rt/lib/builtins/aarch64/sme-abi.S

54–55

Can you just tbz to 2 and remove the need for the extra branch?

147

Looking at the docs both x17 and x18 are available? Which means if you use x17 to hold REG_TPIDR2_EL0 then you can remove this extra load.

[sdesmalen]

Removed unnecessary branch and load (addressing @paulwalker-arm's comments)

[hnrklssn]

This commit seems be causing build failures on green dragon:
https://green.lab.llvm.org/green/job/clang-san-iossim/19806/consoleFull#25062429549ba4694-19c4-4d7e-bec5-911270d8a58c
https://green.lab.llvm.org/green/job/clang-stage1-RA/lastFailedBuild/consoleFull#20418663849ba4694-19c4-4d7e-bec5-911270d8a58c

/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:55:3: error: ADR/ADRP relocations must be GOT relative
  adrp x16, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:55:3: error: unknown AArch64 fixup kind!
  adrp x16, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:56:3: error: unknown AArch64 fixup kind!
  ldrb w16, [x16, :lo12:___aarch64_has_sme_and_tpidr2_el0]
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:105:3: error: ADR/ADRP relocations must be GOT relative
  adrp x14, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:105:3: error: unknown AArch64 fixup kind!
  adrp x14, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:106:3: error: unknown AArch64 fixup kind!
  ldrb w14, [x14, :lo12:___aarch64_has_sme_and_tpidr2_el0]
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:144:3: error: ADR/ADRP relocations must be GOT relative
  adrp x14, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:144:3: error: unknown AArch64 fixup kind!
  adrp x14, ___aarch64_has_sme_and_tpidr2_el0
  ^
/var/folders/09/r4vw4v8n5kb67jl66zvlbljw0000gn/T/sme-abi-e48b7e.s:145:3: error: unknown AArch64 fixup kind!
  ldrb w14, [x14, :lo12:___aarch64_has_sme_and_tpidr2_el0]
  ^

[SjoerdMeijer]

We are seeing build problems because of this change on some of our systems. I am suspecting there might be multiple problems, but let's start at the very beginning with:

When compiling for SME ...

New files sme-abi.S and aarch64/sme-abi-init.c get built depending on successful compilation of:

builtin_check_c_compiler_source(COMPILER_RT_HAS_ASM_SME
"
asm(\".arch armv9-a+sme\");
asm(\"smstart\");
")

This check depends on the host compiler and assembler du jour and whether that can assemble SME or not.

That doesn't seem like the way we want to do architecture extensions checks. But perhaps I am missing something @sdesmalen / @paulwalker-arm ?

[sdesmalen]

In D154045#4655360, @SjoerdMeijer wrote:

We are seeing build problems because of this change on some of our systems. I am suspecting there might be multiple problems, but let's start at the very beginning with:

When compiling for SME ...

New files sme-abi.S and aarch64/sme-abi-init.c get built depending on successful compilation of:

builtin_check_c_compiler_source(COMPILER_RT_HAS_ASM_SME
"
asm(\".arch armv9-a+sme\");
asm(\"smstart\");
")

This check depends on the host compiler and assembler du jour and whether that can assemble SME or not.

That doesn't seem like the way we want to do architecture extensions checks. But perhaps I am missing something @sdesmalen / @paulwalker-arm ?

Are you using the latest top-of-tree? We should only build these files when targeting a platform with sys/auxv.h or baremetal, see https://github.com/llvm/llvm-project/pull/69423.

[SjoerdMeijer]

Are you using the latest top-of-tree? We should only build these files when targeting a platform with sys/auxv.h or baremetal, see https://github.com/llvm/llvm-project/pull/69423.

Yes, I am building ToT OSS Clang/LLVM. I see that check:

if(COMPILER_RT_HAS_ASM_SME AND (COMPILER_RT_HAS_AUXV OR COMPILER_RT_BAREMETAL_BUILD))

COMPILER_RT_HAS_ASM_SME is true and since I see auxv.h on our systems, COMPILER_RT_HAS_AUXV is true.
And it must be true, because I see compilation of sme-abi.S which is where we have problems (that's probably another problem).
The first problem is that I don't expect COMPILER_RT_HAS_ASM_SME to be true on our system. It happens to be true, because a GNU as is picked up that is happy to assemble SME, but I don't think that is the check that we should be doing.

Can this be reverted? OSS Clang/LLVM builds are failing on our AArch64 platforms.

[sdesmalen]

The first problem is that I don't expect COMPILER_RT_HAS_ASM_SME to be true on our system. It happens to be true, because a GNU as is picked up that is happy to assemble SME, but I don't think that is the check that we should be doing.

If the toolchain you're using to build compiler-rt for the selected target supports assembling SME instructions, then there shouldn't be a problem assembling sme-abi.S. A similar approach is taken for the LSE builtins.

What exactly is the problem that you're encountering with building sme-abi.S?

[SjoerdMeijer]

In D154045#4655394, @sdesmalen wrote:

The first problem is that I don't expect COMPILER_RT_HAS_ASM_SME to be true on our system. It happens to be true, because a GNU as is picked up that is happy to assemble SME, but I don't think that is the check that we should be doing.

If the toolchain you're using to build compiler-rt for the selected target supports assembling SME instructions, then there shouldn't be a problem assembling sme-abi.S. A similar approach is taken for the LSE builtins.

This results in completely arbitrary results. On one ubuntu system binutils 2.34 is picked up that doesn't support SME, on another system binutils 2.38 is picked up which apparantly does support it.
My question is if the binutils "du jour" should be determining this, or something else?

I have indeed seen the precedent for LSE, but I am not sure I agree with that one either (but I don't mind at the moment).

What exactly is the problem that you're encountering with building sme-abi.S?

We are getting a build failure:

compiler-rt/lib/builtins/aarch64/sme-abi.S: Assembler messages:
compiler-rt/lib/builtins/aarch64/sme-abi.S:38: Error: bad register expression

Apparently the GNU assembler doesn't like this:

.cfi_offset vg, -16

I have not yet looked what this is about, if this GNU assembler is not supporting this or if something else is going on...

My question is why we are compiling this in the first place.

[SjoerdMeijer]

Just wanted to add that I am running vanilla Ubuntu 20.04.5 LTS and Ubuntu 22.04.1 LTS.

I really want to get my builds going again. I will revert this on Monday.

[sdesmalen]

In D154045#4655395, @SjoerdMeijer wrote:

My question is why we are compiling this in the first place.

As far as I understand it compiler-rt should provide any runtime support that Clang supports. In this case, when building a toolchain for AArch64, Clang supports the SME ACLE intrinsics and so compiler-rt should also provide the corresponding SME ABI routines. If for some reason the toolchain used to build compiler-rt is not 'new' enough to support the SME assembly, CMake emits a warning that the SME ABI routines are not built. In this case it seems that the issue here is that assembler we use to build compiler-rt supports SME but apparently not the SVE vg in the .cfi_offset directive.

Can you try replacing .cfi_offset vg, -16 by .cfi_offset 46, -16 and let me know if that solves the build issues you encountered?

[SjoerdMeijer]

Thanks for e79f0506cf6fd32ed466fdd2ccbb68dcacbb4bf, that solved it.

[efriedma]

Looking at the discussion of initialization, was there any consideration of lazily initializing the "__aarch64_has_sme_and_tpidr2_el0" bit? I mean, I see an explanation for why the bit is cached, but I don't see any constraint on when that computation happens.

[nsz]

In D154045#4656377, @efriedma wrote:

Looking at the discussion of initialization, was there any consideration of lazily initializing the "__aarch64_has_sme_and_tpidr2_el0" bit? I mean, I see an explanation for why the bit is cached, but I don't see any constraint on when that computation happens.

the first use can be in an async signal handler and getauxval is not documented to be async signal safe (i'd expect it to work in practice, but there is no guarantee).

and i think a lot of registers have to be preserved (including sve regs which can cause stack overflow on a large sve implementation) as well as errno (getauxval clobbers it in case the kernel does not set AT_HWCAP2).

(you would also need an extra conditional branch to check for init and keep the is-initialized and has-sme bits in the same word to avoid atomics in the hot path)

compiler-rt/cmake/Modules/AddCompilerRT.cmake

foreach(libname ${libnames})
     # If you are using a multi-configuration generator we don't generate
     # per-library install rules, so we fall back to the parent target COMPONENT
     if(CMAKE_CONFIGURATION_TYPES AND LIB_PARENT_TARGET)
       set(COMPONENT_OPTION COMPONENT ${LIB_PARENT_TARGET})
     else()
       set(COMPONENT_OPTION COMPONENT ${libname})
     endif()
 
+    if(type STREQUAL "SHARED")
+      list(APPEND LIB_DEFS COMPILER_RT_SHARED_LIB)
+    endif()
+
     if(type STREQUAL "OBJECT")
       if(CMAKE_C_COMPILER_ID MATCHES Clang AND CMAKE_C_COMPILER_TARGET)
         list(APPEND extra_cflags_${libname} "--target=${CMAKE_C_COMPILER_TARGET}")
       endif()
       if(CMAKE_SYSROOT)
         list(APPEND extra_cflags_${libname} "--sysroot=${CMAKE_SYSROOT}")
       endif()
       string(REPLACE ";" " " extra_cflags_${libname} "${extra_cflags_${libname}}")

compiler-rt/cmake/builtin-config-ix.cmake

 ")
 
 builtin_check_c_compiler_source(COMPILER_RT_HAS_ASM_LSE
 "
 asm(\".arch armv8-a+lse\");
 asm(\"cas w0, w1, [x2]\");
 ")
 
+builtin_check_c_compiler_source(COMPILER_RT_HAS_ASM_SME
+"
+asm(\".arch armv9-a+sme\");
+asm(\"smstart\");
+")
+
 if(ANDROID)
   set(OS_NAME "Android")
 else()
   set(OS_NAME "${CMAKE_SYSTEM_NAME}")
 endif()
 
 set(ARM64 aarch64)
 set(ARM32 arm armhf armv4t armv5te armv6 armv6m armv7m armv7em armv7 armv7s armv7k armv8m.base armv8m.main armv8.1m.main)
 set(AVR avr)
 set(HEXAGON hexagon)
 set(X86 i386)
 set(X86_64 x86_64)
 set(LOONGARCH64 loongarch64)
 set(MIPS32 mips mipsel)

compiler-rt/lib/builtins/CMakeLists.txt

elseif(NOT WIN32)
   )
 endif()
 
 set(aarch64_SOURCES
   ${GENERIC_TF_SOURCES}
   ${GENERIC_SOURCES}
   cpu_model.c
   aarch64/fp_mode.c
+  aarch64/sme-abi.S
+  aarch64/sme-abi-init.c
 )
 
 # Generate outline atomics helpers from lse.S base
 set(OA_HELPERS_DIR "${CMAKE_CURRENT_BINARY_DIR}/outline_atomic_helpers.dir")
 file(MAKE_DIRECTORY "${OA_HELPERS_DIR}")
 
 if(CMAKE_HOST_UNIX)
   set(COMPILER_RT_LINK_OR_COPY create_symlink)
if(COMPILER_RT_BUILTINS_HIDE_SYMBOLS)
     append_list_if(COMPILER_RT_HAS_VISIBILITY_HIDDEN_FLAG VISIBILITY_HIDDEN BUILTIN_DEFS)
   endif()
 
   if(COMPILER_RT_DISABLE_AARCH64_FMV)
     list(APPEND BUILTIN_DEFS DISABLE_AARCH64_FMV)
   endif()
 
   append_list_if(COMPILER_RT_HAS_ASM_LSE HAS_ASM_LSE BUILTIN_DEFS)
+  append_list_if(COMPILER_RT_HAS_ASM_SME HAS_ASM_SME BUILTIN_DEFS)
 
   foreach (arch ${BUILTIN_SUPPORTED_ARCH})
     if (CAN_TARGET_${arch})
       cmake_push_check_state()
       # TODO: we should probably make most of the checks in builtin-config depend on the target flags.
       message(STATUS "Performing additional configure checks with target flags: ${TARGET_${arch}_CFLAGS}")
       set(BUILTIN_CFLAGS_${arch} ${BUILTIN_CFLAGS})
       list(APPEND CMAKE_REQUIRED_FLAGS ${TARGET_${arch}_CFLAGS} ${BUILTIN_CFLAGS_${arch}})

compiler-rt/lib/builtins/aarch64/sme-abi-init.c

+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+__attribute__((visibility("hidden"), nocommon))
+_Bool __aarch64_has_sme_and_tpidr2_el0;
+
+// We have multiple ways to check that the function has SME, depending on our
+// target.
+// * For Linux we can use __getauxval().
+// * For newlib we can use __aarch64_sme_accessible().
+
+#if defined(__linux__)
+
+#ifndef AT_HWCAP2
+#define AT_HWCAP2 26
+#endif
+
+#ifndef HWCAP2_SME
+#define HWCAP2_SME (1 << 23)
+#endif
+
+extern unsigned long int __getauxval (unsigned long int);
+
+static _Bool has_sme(void) {
+  return __getauxval(AT_HWCAP2) & HWCAP2_SME;
+}
+
+#else  // defined(__linux__)
+
+#if defined(COMPILER_RT_SHARED_LIB)
+__attribute__((weak))
+#endif
+extern _Bool __aarch64_sme_accessible(void);
+
+static _Bool has_sme(void)  {
+#if defined(COMPILER_RT_SHARED_LIB)
+  if (!__aarch64_sme_accessible)
+    return 0;
+#endif
+  return __aarch64_sme_accessible();
+}
+
+#endif // defined(__linux__)
+
+__attribute__((constructor(90)))
+static void init_aarch64_has_sme(void) {
+  __aarch64_has_sme_and_tpidr2_el0 = has_sme();
+}

compiler-rt/lib/builtins/aarch64/sme-abi.S

+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+// This patch implements the support routines for the SME ABI,
+// described here:
+//  https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst#sme-support-routines
+
+#include "../assembly.h"
+
+#ifdef HAS_ASM_SME
+#define ARCH armv9-a+sme
+#define SMSTOP_SM smstop sm
+#define SMSTOP_ZA smstop za
+#define REG_TPIDR2_EL0 TPIDR2_EL0
+#define REG_SVCR SVCR
+#define ADDSVL_X16_X16_1 addsvl x16, x16, #1
+#define LDR_ZA_W15_0_X16 ldr za[w15,0], [x16]
+#define STR_ZA_W15_0_X16 str za[w15,0], [x16]
+#define CNTD_X0 cntd x0
+#define CFI_OFFSET_VG_MINUS_16 .cfi_offset vg, -16
+#else
+#define ARCH armv8-a
+#define SMSTOP_SM .inst 0xd503427f
+#define SMSTOP_ZA .inst 0xd503447f
+#define REG_TPIDR2_EL0 S3_3_C13_C0_5
+#define REG_SVCR S3_3_C4_C2_2
+#define ADDSVL_X16_X16_1 .inst 0x04305830
+#define LDR_ZA_W15_0_X16 .inst 0xe1006200
+#define STR_ZA_W15_0_X16 .inst 0xe1206200
+#define CNTD_X0 .inst 0x04e0e3e0
+#define CFI_OFFSET_VG_MINUS_16 .cfi_escape 0x10, 0x2e, 0x03, 0x11, 0x70, 0x22 // $vg  @ cfa - 16
+#endif
+
+.arch ARCH
+
+// Utility function which calls a system's abort() routine. Because the function
+// is streaming-compatible it should disable streaming-SVE mode before calling
+// abort(). Note that there is no need to preserve any state before the call,
+// because the function does not return.
+DEFINE_COMPILERRT_PRIVATE_FUNCTION(do_abort)
+.cfi_startproc
+  .variant_pcs  SYMBOL_NAME(do_abort)
+  stp x29, x30, [sp, #-32]!
+  CNTD_X0
+  // Store VG to a stack location that we describe with .cfi_offset
+  str x0, [sp, #16]
+  .cfi_def_cfa_offset 32
+  .cfi_offset w30, -24
+  .cfi_offset w29, -32
+  CFI_OFFSET_VG_MINUS_16
+  bl  __arm_sme_state
+  tbz x0, #0, 2f
+1:
+  SMSTOP_SM
+2:
+  // We can't make this into a tail-call because the unwinder would
+  // need to restore the value of VG.
+  bl  SYMBOL_NAME(abort)
+.cfi_endproc
+END_COMPILERRT_FUNCTION(do_abort)
+
+// __arm_sme_state fills the result registers based on a local
+// that is set as part of the compiler-rt startup code.
+//   __aarch64_has_sme_and_tpidr2_el0
+DEFINE_COMPILERRT_OUTLINE_FUNCTION_UNMANGLED(__arm_sme_state)
+  .variant_pcs  __arm_sme_state
+  mov x0, xzr
+  mov x1, xzr
+
+  adrp  x16, SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)
+  ldrb w16, [x16, :lo12:SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)]
+  cbz w16, 1f
+0:
+  orr x0, x0, #0xC000000000000000
+  mrs x16, REG_SVCR
+  bfxil x0, x16, #0, #2
+  mrs x1, REG_TPIDR2_EL0
+1:
+  ret
+END_COMPILERRT_OUTLINE_FUNCTION(__arm_sme_state)
+
+DEFINE_COMPILERRT_OUTLINE_FUNCTION_UNMANGLED(__arm_tpidr2_restore)
+  .variant_pcs  __arm_tpidr2_restore
+  // If TPIDR2_EL0 is nonnull, the subroutine aborts in some platform-specific
+  // manner.
+  mrs x14, REG_TPIDR2_EL0
+  cbnz  x14, 2f
+
+  // If any of the reserved bytes in the first 16 bytes of BLK are nonzero,
+  // the subroutine [..] aborts in some platform-defined manner.
+  ldrh  w14, [x0, #10]
+  cbnz  w14, 2f
+  ldr w14, [x0, #12]
+  cbnz  w14, 2f
+
+  // If BLK.za_save_buffer is NULL, the subroutine does nothing.
+  ldr x16, [x0]
+  cbz x16, 1f
+
+  // If BLK.num_za_save_slices is zero, the subroutine does nothing.
+  ldrh  w14, [x0, #8]
+  cbz x14, 1f
+
+  mov x15, xzr
+0:
+  LDR_ZA_W15_0_X16
+  ADDSVL_X16_X16_1
+  add x15, x15, #1
+  cmp x14, x15
+  b.ne  0b
+1:
+  ret
+2:
+  b  SYMBOL_NAME(do_abort)
+END_COMPILERRT_OUTLINE_FUNCTION(__arm_tpidr2_restore)
+
+DEFINE_COMPILERRT_OUTLINE_FUNCTION_UNMANGLED(__arm_tpidr2_save)
+  // If the current thread does not have access to TPIDR2_EL0, the subroutine
+  // does nothing.
+  adrp  x14, SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)
+  ldrb w14, [x14, :lo12:SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)]
+  cbz w14, 1f
+
+  // If TPIDR2_EL0 is null, the subroutine does nothing.
+  mrs x16, REG_TPIDR2_EL0
+  cbz x16, 1f
+
+  // If any of the reserved bytes in the first 16 bytes of the TPIDR2 block are
+  // nonzero, the subroutine [..] aborts in some platform-defined manner.
+  ldrh  w14, [x16, #10]
+  cbnz  w14, 2f
+  ldr w14, [x16, #12]
+  cbnz  w14, 2f
+
+  // If num_za_save_slices is zero, the subroutine does nothing.
+  ldrh  w14, [x16, #8]
+  cbz x14, 1f
+
+  // If za_save_buffer is NULL, the subroutine does nothing.
+  ldr x16, [x16]
+  cbz x16, 1f
+
+  mov x15, xzr
+0:
+  STR_ZA_W15_0_X16
+  ADDSVL_X16_X16_1
+  add x15, x15, #1
+  cmp x14, x15
+  b.ne  0b
+1:
+  ret
+2:
+  b  SYMBOL_NAME(do_abort)
+END_COMPILERRT_OUTLINE_FUNCTION(__arm_tpidr2_save)
+
+DEFINE_COMPILERRT_OUTLINE_FUNCTION_UNMANGLED(__arm_za_disable)
+  // If the current thread does not have access to SME, the subroutine does
+  // nothing.
+  adrp  x14, SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)
+  ldrb w14, [x14, :lo12:SYMBOL_NAME(__aarch64_has_sme_and_tpidr2_el0)]
+  cbz w14, 0f
+
+  // Otherwise, the subroutine behaves as if it did the following:
+  // * Call __arm_tpidr2_save.
+  stp x29, x30, [sp, #-16]!
+  .cfi_def_cfa_offset 16
+  mov x29, sp
+  .cfi_def_cfa w29, 16
+  .cfi_offset w30, -8
+  .cfi_offset w29, -16
+  bl  __arm_tpidr2_save
+
+  // * Set TPIDR2_EL0 to null.
+  msr REG_TPIDR2_EL0, xzr
+
+  // * Set PSTATE.ZA to 0.
+  SMSTOP_ZA
+
+  .cfi_def_cfa wsp, 16
+  ldp x29, x30, [sp], #16
+  .cfi_def_cfa_offset 0
+  .cfi_restore w30
+  .cfi_restore w29
+0:
+  ret
+END_COMPILERRT_OUTLINE_FUNCTION(__arm_za_disable)