[sanitizer][LoongArch] Port sanitizer_common to LoongArch

[xry111]

Summary

Initial libsanitizer support for LoongArch. It survived all GCC UBSan tests.

Major changes:

1. LoongArch port of Linux kernel only supports statx for stat and its families. So we need to add statx_to_stat and use it for stat-like libcalls. The logic is "borrowed" from Glibc.
2. sanitizer_syscall_linux_loongarch64.inc is mostly duplicated from RISC-V port, as the syscall interface is almost same.

[xen0n]

compiler-rt/lib/sanitizer_common/sanitizer_platform.h

280

Why not defined(__loongarch64)? I think the future ILP32 ABIs of LA64 will also have GRLEN=64, but the changes above seem to imply the LP64* ABIs.

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp

273

nit: minimum

[xry111]

compiler-rt/lib/sanitizer_common/sanitizer_platform.h

280

I'll change it to use __loongarch_lp64. The LoongArch Toolchain Conventions doc does not mention __loongarch64.

[xry111]

Use __loongarch_lp64 instead of __loongarch_grlen == 64 (as an ILP32 environment on LA64 CPU can also report __loongarch_grlen == 64).

__loongarch64 is not documented, and it may have same issue with __loongarch_grlen. On x86_64-linux-gnux32 and mips64-linux-gnuabin32 we also have __x86_64__ or __mips64 (the latter has already caused some problems in sanitizer). But __loongarch_lp64 is explicit enough, I believe.

Nit: minimal -> minimum.

[SixWeining]

Add @XiaodongLoong who ever ported compiler-rt to LoongArch on LLVM8. Thanks.

[XiaodongLoong]

It is better to upload diffs with full context (-U99999) to make it possible to browse the whole file.

// llvm/docs/Phabricator.rst
To get a full diff, use one of the following commands (or just use Arcanist
to upload your patch):

* ``git show HEAD -U999999 > mypatch.patch``
* ``git diff -U999999 @{u} > mypatch.patch``
* ``git diff HEAD~1 -U999999 > mypatch.patch``

[xry111]

Use full diff

[vitalybuka]

LGTM, If you want feedback from particular people please set them as "blocking reviewers"

[MaskRay]

compiler-rt/lib/sanitizer_common/sanitizer_platform.h

280

Removing defined(__loongarch__) && looks good.

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_linux.cpp

67

Do newer kernel ports use another struct stat?

[xry111]

compiler-rt/lib/sanitizer_common/sanitizer_platform.h

280

Will do.

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_linux.cpp

67

Initially I used "0" for struct_kernel_stat_sz, because the kernel does not provide struct stat anymore. At last it turn out we should use the size of struct stat in libc, not kernel.

I'll remove this #if pair.

But I'd say _kernel in the name struct_kernel_stat_sz is completely misleading, and it's [already causing misunderstandings][1]. Maybe we should rename struct_kernel_*_sz to struct_os_stat_sz or something later.

[1]: https://gcc.gnu.org/pipermail/gcc-patches/2022-July/598096.html

[xry111]

Restore compiler check for sizeof(struct stat).

Simplify the definition of SANITIZER_LOONGARCH64.

[XiaodongLoong]

Thanks! LGTM.

[SixWeining]

LGTM. Thanks.

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

382

Nit: tab -> spaces

414

Ditto.

[vitalybuka]

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

414

for spaces, just keep whatever clang-format does:
git clang-format -f --extensions=inc,cc,h,c,cpp,rst HEAD^

[xry111]

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

382

They are spaces, but with a different indent than our standard I guess.

414

When I run this command, it changes the indent of some #if lines but the corresponding #else/#endif lines are not changed. Should I keep it as-is or manually fix those #else/#endif lines?

[xry111]

run git-clang-format

[MaskRay]

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp

273

llvm-project comments prefer one space after .

[xry111]

nit: use one space instead of two after ..

[SixWeining]

LGTM.

LoongArch is a new port. Although clang hasn't support LoongArch, but I think this revision can land becuase it doesn't break other targets' build and @xry111 has checked GCC's UBSan tests. (We also double checked by same way as @xry111).

If there is objection, please let me know. Thanks.

[MaskRay]

remove unneeded indentation

[MaskRay]

reduce more indentation

compiler-rt/cmake/Modules/AllSupportedArchDefs.cmake

 set(ARM64 aarch64)
 set(ARM32 arm armhf)
 set(HEXAGON hexagon)
 set(X86 i386)
 set(X86_64 x86_64)
+set(LOONGARCH64 loongarch64)
 set(MIPS32 mips mipsel)
 set(MIPS64 mips64 mips64el)
 set(PPC32 powerpc powerpcspe)
 set(PPC64 powerpc64 powerpc64le)
 set(RISCV32 riscv32)
 set(RISCV64 riscv64)
 set(S390X s390x)
 set(SPARC sparc)
 set(SPARCV9 sparcv9)
 set(WASM32 wasm32)
 set(WASM64 wasm64)
 set(VE ve)
 
 if(APPLE)
   set(ARM64 arm64)
   set(ARM32 armv7 armv7s armv7k)
   set(X86_64 x86_64 x86_64h)
 endif()
 
 set(ALL_SANITIZER_COMMON_SUPPORTED_ARCH ${X86} ${X86_64} ${PPC64} ${RISCV64}
     ${ARM32} ${ARM64} ${MIPS32} ${MIPS64} ${S390X} ${SPARC} ${SPARCV9}
-    ${HEXAGON})
+    ${HEXAGON} ${LOONGARCH64})
 set(ALL_ASAN_SUPPORTED_ARCH ${X86} ${X86_64} ${ARM32} ${ARM64} ${RISCV64}
     ${MIPS32} ${MIPS64} ${PPC64} ${S390X} ${SPARC} ${SPARCV9} ${HEXAGON})
 set(ALL_DFSAN_SUPPORTED_ARCH ${X86_64} ${MIPS64} ${ARM64})
 
 if(ANDROID)
   set(OS_NAME "Android")
 else()
   set(OS_NAME "${CMAKE_SYSTEM_NAME}")

compiler-rt/lib/sanitizer_common/CMakeLists.txt

set(SANITIZER_IMPL_HEADERS
   sanitizer_symbolizer_internal.h
   sanitizer_symbolizer_libbacktrace.h
   sanitizer_symbolizer_mac.h
   sanitizer_syscall_generic.inc
   sanitizer_syscall_linux_aarch64.inc
   sanitizer_syscall_linux_arm.inc
   sanitizer_syscall_linux_x86_64.inc
   sanitizer_syscall_linux_riscv64.inc
+  sanitizer_syscall_linux_loongarch64.inc
   sanitizer_syscalls_netbsd.inc
   sanitizer_thread_registry.h
   sanitizer_thread_safety.h
   sanitizer_tls_get_addr.h
   sanitizer_vector.h
   sanitizer_win.h
   sanitizer_win_defs.h
   sanitizer_win_dll_thunk.h

compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp

 #if SANITIZER_LINUX
 #include <sys/utsname.h>
 #endif
 
 #if SANITIZER_LINUX && !SANITIZER_ANDROID
 #include <sys/personality.h>
 #endif
 
+#if SANITIZER_LINUX && defined(__loongarch__)
+#include <sys/sysmacros.h>
+#endif
+
 #if SANITIZER_FREEBSD
 #include <sys/exec.h>
 #include <sys/procctl.h>
 #include <sys/sysctl.h>
 #include <machine/atomic.h>
 extern "C" {
 // <sys/umtx.h> must be included after <errno.h> and <sys/types.h> on
 // FreeBSD 9.2 and 10.0.

 #  elif SANITIZER_LINUX && SANITIZER_RISCV64
 #    include "sanitizer_syscall_linux_riscv64.inc"
 #  elif SANITIZER_LINUX && defined(__aarch64__)
 #    include "sanitizer_syscall_linux_aarch64.inc"
 #  elif SANITIZER_LINUX && defined(__arm__)
 #    include "sanitizer_syscall_linux_arm.inc"
 #  elif SANITIZER_LINUX && defined(__hexagon__)
 #    include "sanitizer_syscall_linux_hexagon.inc"
+#  elif SANITIZER_LINUX && SANITIZER_LOONGARCH64
+#    include "sanitizer_syscall_linux_loongarch64.inc"
 #  else
 #    include "sanitizer_syscall_generic.inc"
 #  endif
 
 // --------------- sanitizer_libc.h
 #if !SANITIZER_SOLARIS && !SANITIZER_NETBSD
 #if !SANITIZER_S390
 uptr internal_mmap(void *addr, uptr length, int prot, int flags, int fd,
static void stat64_to_stat(struct stat64 *in, struct stat *out) {
   out->st_blksize = in->st_blksize;
   out->st_blocks = in->st_blocks;
   out->st_atime = in->st_atime;
   out->st_mtime = in->st_mtime;
   out->st_ctime = in->st_ctime;
 }
 #endif
 
+#if SANITIZER_LINUX && defined(__loongarch__)
+static void statx_to_stat(struct statx *in, struct stat *out) {
+  internal_memset(out, 0, sizeof(*out));
+  out->st_dev = makedev(in->stx_dev_major, in->stx_dev_minor);
+  out->st_ino = in->stx_ino;
+  out->st_mode = in->stx_mode;
+  out->st_nlink = in->stx_nlink;
+  out->st_uid = in->stx_uid;
+  out->st_gid = in->stx_gid;
+  out->st_rdev = makedev(in->stx_rdev_major, in->stx_rdev_minor);
+  out->st_size = in->stx_size;
+  out->st_blksize = in->stx_blksize;
+  out->st_blocks = in->stx_blocks;
+  out->st_atime = in->stx_atime.tv_sec;
+  out->st_atim.tv_nsec = in->stx_atime.tv_nsec;
+  out->st_mtime = in->stx_mtime.tv_sec;
+  out->st_mtim.tv_nsec = in->stx_mtime.tv_nsec;
+  out->st_ctime = in->stx_ctime.tv_sec;
+  out->st_ctim.tv_nsec = in->stx_ctime.tv_nsec;
+}
+#endif
+
 #if SANITIZER_MIPS64
 // Undefine compatibility macros from <sys/stat.h>
 // so that they would not clash with the kernel_stat
 // st_[a|m|c]time fields
 #if !SANITIZER_GO
 #undef st_atime
 #undef st_mtime
 #undef st_ctime

 #endif
 }
 #endif
 
 uptr internal_stat(const char *path, void *buf) {
 #if SANITIZER_FREEBSD
   return internal_syscall(SYSCALL(fstatat), AT_FDCWD, (uptr)path, (uptr)buf, 0);
 #    elif SANITIZER_LINUX
-#      if (SANITIZER_WORDSIZE == 64 || SANITIZER_X32 ||    \
+#      if defined(__loongarch__)
+  struct statx bufx;
+  int res = internal_syscall(SYSCALL(statx), AT_FDCWD, (uptr)path,
+                             AT_NO_AUTOMOUNT, STATX_BASIC_STATS, (uptr)&bufx);
+  statx_to_stat(&bufx, (struct stat *)buf);
+  return res;
+#      elif (SANITIZER_WORDSIZE == 64 || SANITIZER_X32 ||    \
              (defined(__mips__) && _MIPS_SIM == _ABIN32)) && \
             !SANITIZER_SPARC
   return internal_syscall(SYSCALL(newfstatat), AT_FDCWD, (uptr)path, (uptr)buf,
                           0);
 #      else
   struct stat64 buf64;
   int res = internal_syscall(SYSCALL(fstatat64), AT_FDCWD, (uptr)path,
                              (uptr)&buf64, 0);
   stat64_to_stat(&buf64, (struct stat *)buf);
   return res;
 #      endif
 #    else
   struct stat64 buf64;
   int res = internal_syscall(SYSCALL(stat64), path, &buf64);
   stat64_to_stat(&buf64, (struct stat *)buf);
   return res;
 #    endif
 }
 
 uptr internal_lstat(const char *path, void *buf) {
 #if SANITIZER_FREEBSD
   return internal_syscall(SYSCALL(fstatat), AT_FDCWD, (uptr)path, (uptr)buf,
                           AT_SYMLINK_NOFOLLOW);
 #    elif SANITIZER_LINUX
-#      if (defined(_LP64) || SANITIZER_X32 ||              \
+#      if defined(__loongarch__)
+  struct statx bufx;
+  int res = internal_syscall(SYSCALL(statx), AT_FDCWD, (uptr)path,
+                             AT_SYMLINK_NOFOLLOW | AT_NO_AUTOMOUNT,
+                             STATX_BASIC_STATS, (uptr)&bufx);
+  statx_to_stat(&bufx, (struct stat *)buf);
+  return res;
+#      elif (defined(_LP64) || SANITIZER_X32 ||              \
              (defined(__mips__) && _MIPS_SIM == _ABIN32)) && \
             !SANITIZER_SPARC
   return internal_syscall(SYSCALL(newfstatat), AT_FDCWD, (uptr)path, (uptr)buf,
                           AT_SYMLINK_NOFOLLOW);
 #      else
   struct stat64 buf64;
   int res = internal_syscall(SYSCALL(fstatat64), AT_FDCWD, (uptr)path,
                              (uptr)&buf64, AT_SYMLINK_NOFOLLOW);
   stat64_to_stat(&buf64, (struct stat *)buf);
   return res;

 uptr internal_fstat(fd_t fd, void *buf) {
 #if SANITIZER_FREEBSD || SANITIZER_LINUX_USES_64BIT_SYSCALLS
 #if SANITIZER_MIPS64
   // For mips64, fstat syscall fills buffer in the format of kernel_stat
   struct kernel_stat kbuf;
   int res = internal_syscall(SYSCALL(fstat), fd, &kbuf);
   kernel_stat_to_stat(&kbuf, (struct stat *)buf);
   return res;
+# elif SANITIZER_LINUX && defined(__loongarch__)
+  struct statx bufx;
+  int res = internal_syscall(SYSCALL(statx), fd, 0, AT_EMPTY_PATH,
+                             STATX_BASIC_STATS, (uptr)&bufx);
+  statx_to_stat(&bufx, (struct stat *)buf);
+  return res;
 # else
   return internal_syscall(SYSCALL(fstat), fd, (uptr)buf);
 # endif
 #else
   struct stat64 buf64;
   int res = internal_syscall(SYSCALL(fstat64), fd, &buf64);
   stat64_to_stat(&buf64, (struct stat *)buf);
   return res;

 #    if SANITIZER_LINUX
   return internal_syscall(SYSCALL(unlinkat), AT_FDCWD, (uptr)path, 0);
 #else
   return internal_syscall(SYSCALL(unlink), (uptr)path);
 #endif
 }
 
 uptr internal_rename(const char *oldpath, const char *newpath) {
-#if defined(__riscv) && defined(__linux__)
+#if (defined(__riscv) || defined(__loongarch__)) && defined(__linux__)
   return internal_syscall(SYSCALL(renameat2), AT_FDCWD, (uptr)oldpath, AT_FDCWD,
                           (uptr)newpath, 0);
 #    elif SANITIZER_LINUX
   return internal_syscall(SYSCALL(renameat), AT_FDCWD, (uptr)oldpath, AT_FDCWD,
                           (uptr)newpath);
 #    else
   return internal_syscall(SYSCALL(rename), (uptr)oldpath, (uptr)newpath);
 #    endif
#    else
   *bp = ucontext->uc_mcontext.__gregs[REG_S0];
   *sp = ucontext->uc_mcontext.__gregs[REG_SP];
 #    endif
 #  elif defined(__hexagon__)
   ucontext_t *ucontext = (ucontext_t *)context;
   *pc = ucontext->uc_mcontext.pc;
   *bp = ucontext->uc_mcontext.r30;
   *sp = ucontext->uc_mcontext.r29;
+#  elif defined(__loongarch__)
+  ucontext_t *ucontext = (ucontext_t*)context;
+  *pc = ucontext->uc_mcontext.__pc;
+  *bp = ucontext->uc_mcontext.__gregs[22];
+  *sp = ucontext->uc_mcontext.__gregs[3];
 #  else
 #    error "Unsupported arch"
 #  endif
 }
 
 void SignalContext::InitPcSpBp() { GetPcSpBp(context, &pc, &sp, &bp); }
 
 void InitializePlatformEarly() {

compiler-rt/lib/sanitizer_common/sanitizer_platform.h

 #endif
 
 #if defined(__riscv) && (__riscv_xlen == 64)
 #  define SANITIZER_RISCV64 1
 #else
 #  define SANITIZER_RISCV64 0
 #endif
 
+#if defined(__loongarch__) && defined(__loongarch_lp64)
+#  define SANITIZER_LOONGARCH64 1
+#else
+#  define SANITIZER_LOONGARCH64 0
+#endif
+
 // By default we allow to use SizeClassAllocator64 on 64-bit platform.
 // But in some cases (e.g. AArch64's 39-bit address space) SizeClassAllocator64
 // does not work well and we need to fallback to SizeClassAllocator32.
 // For such platforms build this code with -DSANITIZER_CAN_USE_ALLOCATOR64=0 or
 // change the definition of SANITIZER_CAN_USE_ALLOCATOR64 here.
 #ifndef SANITIZER_CAN_USE_ALLOCATOR64
 #  if (SANITIZER_ANDROID && defined(__aarch64__)) || SANITIZER_FUCHSIA
 #    define SANITIZER_CAN_USE_ALLOCATOR64 1

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_linux.cpp

 #    include <sys/statfs.h>
 #    include <linux/perf_event.h>
 #  endif
 
 using namespace __sanitizer;
 
 #  if !defined(__powerpc64__) && !defined(__x86_64__) &&                   \
       !defined(__aarch64__) && !defined(__mips__) && !defined(__s390__) && \
-      !defined(__sparc__) && !defined(__riscv) && !defined(__hexagon__)
+      !defined(__sparc__) && !defined(__riscv) && !defined(__hexagon__) && \
+      !defined(__loongarch__)
 COMPILER_CHECK(struct___old_kernel_stat_sz == sizeof(struct __old_kernel_stat));
 #endif
 
+#if !defined(__loongarch__)
 COMPILER_CHECK(struct_kernel_stat_sz == sizeof(struct stat));
+#endif
 
 #if defined(__i386__)
 COMPILER_CHECK(struct_kernel_stat64_sz == sizeof(struct stat64));
 #endif
 
 CHECK_TYPE_SIZE(io_event);
 CHECK_SIZE_AND_OFFSET(io_event, data);
 CHECK_SIZE_AND_OFFSET(io_event, obj);

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h

 const unsigned struct_kernel_stat_sz = 64;
 const unsigned struct_kernel_stat64_sz = 104;
 #elif SANITIZER_RISCV64
 const unsigned struct_kernel_stat_sz = 128;
 const unsigned struct_kernel_stat64_sz = 0;  // RISCV64 does not use stat64
 #    elif defined(__hexagon__)
 const unsigned struct_kernel_stat_sz = 128;
 const unsigned struct_kernel_stat64_sz = 0;
+#    elif defined(__loongarch__)
+const unsigned struct_kernel_stat_sz = 128;
+const unsigned struct_kernel_stat64_sz = 0;
 #    endif
 struct __sanitizer_perf_event_attr {
   unsigned type;
   unsigned size;
   // More fields that vary with the kernel version.
 };
 
 extern unsigned struct_epoll_event_sz;
 extern unsigned struct_sysinfo_sz;
 extern unsigned __user_cap_header_struct_sz;
 extern unsigned __user_cap_data_struct_sz;
 extern unsigned struct_new_utsname_sz;
 extern unsigned struct_old_utsname_sz;
 extern unsigned struct_oldold_utsname_sz;
 
 const unsigned struct_kexec_segment_sz = 4 * sizeof(unsigned long);
 #endif  // SANITIZER_LINUX
 
 #if SANITIZER_LINUX
 
-#if defined(__powerpc64__) || defined(__s390__)
+#if defined(__powerpc64__) || defined(__s390__) || defined(__loongarch__)
 const unsigned struct___old_kernel_stat_sz = 0;
 #elif !defined(__sparc__)
 const unsigned struct___old_kernel_stat_sz = 32;
 #endif
 
 extern unsigned struct_rlimit_sz;
 extern unsigned struct_utimbuf_sz;
 extern unsigned struct_timespec_sz;

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp

 #if defined(__aarch64__) || defined(__s390x__) || defined(__mips64) ||     \
     defined(__powerpc64__) || defined(__arch64__) || defined(__sparcv9) || \
     defined(__x86_64__) || SANITIZER_RISCV64
 #define SIZEOF_STRUCT_USTAT 32
 #    elif defined(__arm__) || defined(__i386__) || defined(__mips__) ||    \
         defined(__powerpc__) || defined(__s390__) || defined(__sparc__) || \
         defined(__hexagon__)
 #      define SIZEOF_STRUCT_USTAT 20
+#    elif defined(__loongarch__)
+       // Not used.  The minimum Glibc version available for LoongArch is 2.36
+       // so ustat() wrapper is already gone.
+#      define SIZEOF_STRUCT_USTAT 0
 #    else
 #      error Unknown size of struct ustat
 #    endif
   unsigned struct_ustat_sz = SIZEOF_STRUCT_USTAT;
   unsigned struct_rlimit64_sz = sizeof(struct rlimit64);
   unsigned struct_statvfs64_sz = sizeof(struct statvfs64);
 #endif // SANITIZER_LINUX && !SANITIZER_ANDROID
 

compiler-rt/lib/sanitizer_common/sanitizer_syscall_linux_loongarch64.inc

+//===-- sanitizer_syscall_linux_loongarch64.inc -----------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Implementations of internal_syscall and internal_iserror for Linux/loongarch64.
+//
+//===----------------------------------------------------------------------===//
+
+// About local register variables:
+// https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
+//
+// Kernel ABI...
+//  syscall number is passed in a7
+//  (http://man7.org/linux/man-pages/man2/syscall.2.html) results are return in
+//  a0 and a1 (http://man7.org/linux/man-pages/man2/syscall.2.html) arguments
+//  are passed in: a0-a7 (confirmed by inspecting glibc sources).
+#define SYSCALL(name) __NR_##name
+
+#define INTERNAL_SYSCALL_CLOBBERS "memory"
+
+static uptr __internal_syscall(u64 nr) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0");
+  __asm__ volatile("syscall 0\n\t"
+                   : "=r"(a0)
+                   : "r"(a7)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall0(n) (__internal_syscall)(n)
+
+static uptr __internal_syscall(u64 nr, u64 arg1) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall1(n, a1) (__internal_syscall)(n, (u64)(a1))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall2(n, a1, a2) \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2, long arg3) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  register u64 a2 asm("a2") = arg3;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1), "r"(a2)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall3(n, a1, a2, a3) \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2), (long)(a3))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2, long arg3,
+                               u64 arg4) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  register u64 a2 asm("a2") = arg3;
+  register u64 a3 asm("a3") = arg4;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1), "r"(a2), "r"(a3)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall4(n, a1, a2, a3, a4) \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2), (long)(a3), (long)(a4))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2, long arg3, u64 arg4,
+                               long arg5) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  register u64 a2 asm("a2") = arg3;
+  register u64 a3 asm("a3") = arg4;
+  register u64 a4 asm("a4") = arg5;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1), "r"(a2), "r"(a3), "r"(a4)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall5(n, a1, a2, a3, a4, a5)                       \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2), (long)(a3), (long)(a4), \
+                       (u64)(a5))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2, long arg3, u64 arg4,
+                               long arg5, long arg6) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  register u64 a2 asm("a2") = arg3;
+  register u64 a3 asm("a3") = arg4;
+  register u64 a4 asm("a4") = arg5;
+  register u64 a5 asm("a5") = arg6;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1), "r"(a2), "r"(a3), "r"(a4), "r"(a5)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall6(n, a1, a2, a3, a4, a5, a6)                   \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2), (long)(a3), (long)(a4), \
+                       (u64)(a5), (long)(a6))
+
+static uptr __internal_syscall(u64 nr, u64 arg1, long arg2, long arg3, u64 arg4,
+                               long arg5, long arg6, long arg7) {
+  register u64 a7 asm("a7") = nr;
+  register u64 a0 asm("a0") = arg1;
+  register u64 a1 asm("a1") = arg2;
+  register u64 a2 asm("a2") = arg3;
+  register u64 a3 asm("a3") = arg4;
+  register u64 a4 asm("a4") = arg5;
+  register u64 a5 asm("a5") = arg6;
+  register u64 a6 asm("a6") = arg7;
+  __asm__ volatile("syscall 0\n\t"
+                   : "+r"(a0)
+                   : "r"(a7), "r"(a1), "r"(a2), "r"(a3), "r"(a4), "r"(a5),
+                     "r"(a6)
+                   : INTERNAL_SYSCALL_CLOBBERS);
+  return a0;
+}
+#define __internal_syscall7(n, a1, a2, a3, a4, a5, a6, a7)               \
+  (__internal_syscall)(n, (u64)(a1), (long)(a2), (long)(a3), (long)(a4), \
+                       (u64)(a5), (long)(a6), (long)(a7))
+
+#define __SYSCALL_NARGS_X(a1, a2, a3, a4, a5, a6, a7, a8, n, ...) n
+#define __SYSCALL_NARGS(...) \
+  __SYSCALL_NARGS_X(__VA_ARGS__, 7, 6, 5, 4, 3, 2, 1, 0, )
+#define __SYSCALL_CONCAT_X(a, b) a##b
+#define __SYSCALL_CONCAT(a, b) __SYSCALL_CONCAT_X(a, b)
+#define __SYSCALL_DISP(b, ...) \
+  __SYSCALL_CONCAT(b, __SYSCALL_NARGS(__VA_ARGS__))(__VA_ARGS__)
+
+#define internal_syscall(...) __SYSCALL_DISP(__internal_syscall, __VA_ARGS__)
+
+// Helper function used to avoid clobbering of errno.
+bool internal_iserror(uptr retval, int *internal_errno) {
+  if (retval >= (uptr)-4095) {
+    if (internal_errno)
+      *internal_errno = -retval;
+    return true;
+  }
+  return false;
+}