[scudo] Implement Fuchsia backend for the new MemMap API

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

23

nit: Suggest moving static functions (here and below) outside scudo namespace.

68–70

nit:single statement

93–95

nit:single statement

117–119

nit:single statement

168–170

nit:single statement

179

Do you want to handle MAP_ALLOWNOMEM here?

188–190

nit: omit curly brackets if it's single statement

194–196

nit: single statement

217–219

nit: single statement

[fabio-d]

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

23

What goal do you have in mind? I can do that, but I'm not sure it's more readable as I'll have to add the scudo:: prefix to all the references to other Scudo functions and types, e.g.

static void NORETURN dieOnError(zx_status_t Status, const char *FnName,
                                scudo::uptr Size) {
  char Error[128];
  scudo::formatString(Error, sizeof(Error),
                      "SCUDO ERROR: %s failed with size %zuKB (%s)", FnName,
                      Size >> 10, _zx_status_get_string(Status));
  scudo::outputRaw(Error);
  scudo::die();
}

179

Nice catch! The zx_vmar_map documentation says that userspace has no way to handle ZX_ERR_NO_MEMORY in a good way, and that it will become impossible to get this error in the future. Anyway, I guess that an extra check within an already-UNLIKELY block doesn't hurt, so I've added it.

For completeness, dispatch could also result in the same error (because of the zx_vmo_create call). In theory, I think we should make dispatch similarly fallible and with an option to pass MAP_ALLOWNOMEM. In practice, I think that it's OK to just die if dispatch fails, given who the current users are.

[Caslyn]

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

162

I recall the Size passed to decommit must be page aligned for the op to succeed - want to make sure, is that always the case here?

[fabio-d]

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

162

Good point! Actually, all the addresses, sizes and capacities must be page aligned.

@Chia-hungDuan I'd say that this and similar checks should be DCHECKs in mem_map_base.h, so that they are checked on Linux too. I think think we even discussed it, but there was some caveat that I don't remember about Trusty. I saw that Trusty's impl has changed since then. Would you see any issue in enforcing page alignment there too?

[Chia-hungDuan]

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

23

This is just a coding convention that I usually have in some llvm projects. Using anonymous namespace and static to manage the code that has the scope only in a compilation unit.

In general, we put local class declaration in anonymous namespace only. Other stuff will be done with static in global namespace.

In the case here, it makes the scudo namespace clear. Not a strong opinion. feel free to stick on your preference.

162

On Linux, it only requires the Size to be greater than 0. It does the round up stuff in the kernel. Luke said he would like to do the same thing in Fuchsia.

I think it's fine to add this constraint at one level above platform specific layer. I need to review the rationale of enforcing that (include Trusty's stuff)

compiler-rt/lib/scudo/standalone/CMakeLists.txt

set(SCUDO_HEADERS
   fuchsia.h
   internal_defs.h
   linux.h
   list.h
   local_cache.h
   memtag.h
   mem_map.h
   mem_map_base.h
+  mem_map_fuchsia.h
   mutex.h
   options.h
   platform.h
   primary32.h
   primary64.h
   quarantine.h
   release.h
   report.h
set(SCUDO_SOURCES
   checksum.cpp
   common.cpp
   crc32_hw.cpp
   flags_parser.cpp
   flags.cpp
   fuchsia.cpp
   linux.cpp
   mem_map.cpp
+  mem_map_fuchsia.cpp
   release.cpp
   report.cpp
   rss_limit_checker.cpp
   string_utils.cpp
   timing.cpp
   )
 
 # Temporary hack until LLVM libc supports inttypes.h print format macros

compiler-rt/lib/scudo/standalone/mem_map.h

 
 // TODO: This is only used for `MapPlatformData`. Remove these includes when we
 // have all three platform specific `MemMap` and `ReservedMemory`
 // implementations.
 #include "fuchsia.h"
 #include "linux.h"
 #include "trusty.h"
 
+#include "mem_map_fuchsia.h"
+
 namespace scudo {
 
 // This will be deprecated when every allocator has been supported by each
 // platform's `MemMap` implementation.
 class MemMapDefault final : public MemMapBase<MemMapDefault> {
 public:
   constexpr MemMapDefault() = default;
   MemMapDefault(uptr Base, uptr Capacity) : Base(Base), Capacity(Capacity) {}
private:
   uptr Capacity = 0;
   MapPlatformData Data = {};
 };
 
 #if SCUDO_LINUX
 using ReservedMemoryT = ReservedMemoryDefault;
 using MemMapT = ReservedMemoryT::MemMapT;
 #elif SCUDO_FUCHSIA
-using ReservedMemoryT = ReservedMemoryDefault;
+using ReservedMemoryT = ReservedMemoryFuchsia;
 using MemMapT = ReservedMemoryT::MemMapT;
 #elif SCUDO_TRUSTY
 using ReservedMemoryT = ReservedMemoryDefault;
 using MemMapT = ReservedMemoryT::MemMapT;
 #else
 #error                                                                         \
     "Unsupported platform, please implement the ReservedMemory for your platform!"
 #endif
 
 } // namespace scudo
 
 #endif // SCUDO_MEM_MAP_H_

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.h

+//===-- mem_map_fuchsia.h ---------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef SCUDO_MEM_MAP_FUCHSIA_H_
+#define SCUDO_MEM_MAP_FUCHSIA_H_
+
+#include "mem_map_base.h"
+
+#if SCUDO_FUCHSIA
+
+#include <stdint.h>
+#include <zircon/types.h>
+
+namespace scudo {
+
+class MemMapFuchsia final : public MemMapBase<MemMapFuchsia> {
+public:
+  constexpr MemMapFuchsia() = default;
+
+  // Impls for base functions.
+  bool mapImpl(uptr Addr, uptr Size, const char *Name, uptr Flags);
+  void unmapImpl(uptr Addr, uptr Size);
+  bool remapImpl(uptr Addr, uptr Size, const char *Name, uptr Flags);
+  void setMemoryPermissionImpl(uptr Addr, uptr Size, uptr Flags);
+  void releasePagesToOSImpl(uptr From, uptr Size) {
+    return releaseAndZeroPagesToOSImpl(From, Size);
+  }
+  void releaseAndZeroPagesToOSImpl(uptr From, uptr Size);
+  uptr getBaseImpl() { return WindowBase; }
+  uptr getCapacityImpl() { return WindowSize; }
+
+private:
+  friend class ReservedMemoryFuchsia;
+
+  // Used by ReservedMemoryFuchsia::dispatch.
+  MemMapFuchsia(uptr Base, uptr Capacity);
+
+  // Virtual memory address corresponding to VMO offset 0.
+  uptr MapAddr = 0;
+
+  // Virtual memory base address and size of the VMO subrange that is still in
+  // use. unmapImpl() can shrink this range, either at the beginning or at the
+  // end.
+  uptr WindowBase = 0;
+  uptr WindowSize = 0;
+
+  zx_handle_t Vmo = ZX_HANDLE_INVALID;
+};
+
+class ReservedMemoryFuchsia final
+    : public ReservedMemory<ReservedMemoryFuchsia, MemMapFuchsia> {
+public:
+  constexpr ReservedMemoryFuchsia() = default;
+
+  bool createImpl(uptr Addr, uptr Size, const char *Name, uptr Flags);
+  void releaseImpl();
+  MemMapT dispatchImpl(uptr Addr, uptr Size);
+  uptr getBaseImpl() { return Base; }
+  uptr getCapacityImpl() { return Capacity; }
+
+private:
+  uptr Base = 0;
+  uptr Capacity = 0;
+};
+
+} // namespace scudo
+
+#endif // SCUDO_FUCHSIA
+
+#endif // SCUDO_MEM_MAP_FUCHSIA_H_

compiler-rt/lib/scudo/standalone/mem_map_fuchsia.cpp

+//===-- mem_map_fuchsia.cpp -------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "mem_map_fuchsia.h"
+
+#include "atomic_helpers.h"
+#include "common.h"
+#include "string_utils.h"
+
+#if SCUDO_FUCHSIA
+
+#include <zircon/process.h>
+#include <zircon/status.h>
+#include <zircon/syscalls.h>
+
+namespace scudo {
+
+static void NORETURN dieOnError(zx_status_t Status, const char *FnName,
+                                uptr Size) {
+  char Error[128];
+  formatString(Error, sizeof(Error),
+               "SCUDO ERROR: %s failed with size %zuKB (%s)", FnName,
+               Size >> 10, _zx_status_get_string(Status));
+  outputRaw(Error);
+  die();
+}
+
+static void setVmoName(zx_handle_t Vmo, const char *Name) {
+  size_t Len = strlen(Name);
+  DCHECK_LT(Len, ZX_MAX_NAME_LEN);
+  zx_status_t Status = _zx_object_set_property(Vmo, ZX_PROP_NAME, Name, Len);
+  CHECK_EQ(Status, ZX_OK);
+}
+
+// Returns the (cached) base address of the root VMAR.
+static uptr getRootVmarBase() {
+  static atomic_uptr CachedResult = {0};
+
+  uptr Result = atomic_load_relaxed(&CachedResult);
+  if (UNLIKELY(!Result)) {
+    zx_info_vmar_t VmarInfo;
+    zx_status_t Status =
+        _zx_object_get_info(_zx_vmar_root_self(), ZX_INFO_VMAR, &VmarInfo,
+                            sizeof(VmarInfo), nullptr, nullptr);
+    CHECK_EQ(Status, ZX_OK);
+    CHECK_NE(VmarInfo.base, 0);
+
+    atomic_store_relaxed(&CachedResult, VmarInfo.base);
+    Result = VmarInfo.base;
+  }
+
+  return Result;
+}
+
+// Lazily creates and then always returns the same zero-sized VMO.
+static zx_handle_t getPlaceholderVmo() {
+  static atomic_u32 StoredVmo = {ZX_HANDLE_INVALID};
+
+  zx_handle_t Vmo = atomic_load_relaxed(&StoredVmo);
+  if (UNLIKELY(Vmo == ZX_HANDLE_INVALID)) {
+    // Create a zero-sized placeholder VMO.
+    zx_status_t Status = _zx_vmo_create(0, 0, &Vmo);
+    if (UNLIKELY(Status != ZX_OK)) {
+      dieOnError(Status, "zx_vmo_create", 0);
+    }
+
+    setVmoName(Vmo, "scudo:reserved");
+
+    // Atomically store its handle. If some other thread wins the race, use its
+    // handle and discard ours.
+    zx_handle_t OldValue =
+        atomic_compare_exchange(&StoredVmo, ZX_HANDLE_INVALID, Vmo);
+    if (OldValue != ZX_HANDLE_INVALID) {
+      Status = _zx_handle_close(Vmo);
+      CHECK_EQ(Status, ZX_OK);
+
+      Vmo = OldValue;
+    }
+  }
+
+  return Vmo;
+}
+
+MemMapFuchsia::MemMapFuchsia(uptr Base, uptr Capacity)
+    : MapAddr(Base), WindowBase(Base), WindowSize(Capacity) {
+  // Create the VMO.
+  zx_status_t Status = _zx_vmo_create(Capacity, 0, &Vmo);
+  if (UNLIKELY(Status != ZX_OK)) {
+    dieOnError(Status, "zx_vmo_create", Capacity);
+  }
+}
+
+bool MemMapFuchsia::mapImpl(UNUSED uptr Addr, uptr Size, const char *Name,
+                            uptr Flags) {
+  const bool AllowNoMem = !!(Flags & MAP_ALLOWNOMEM);
+  const bool PreCommit = !!(Flags & MAP_PRECOMMIT);
+  const bool NoAccess = !!(Flags & MAP_NOACCESS);
+
+  // Create the VMO.
+  zx_status_t Status = _zx_vmo_create(Size, 0, &Vmo);
+  if (UNLIKELY(Status != ZX_OK)) {
+    if (Status != ZX_ERR_NO_MEMORY || !AllowNoMem)
+      dieOnError(Status, "zx_vmo_create", Size);
+    return false;
+  }
+
+  if (Name != nullptr)
+    setVmoName(Vmo, Name);
+
+  // Map it.
+  zx_vm_option_t MapFlags = ZX_VM_ALLOW_FAULTS;
+  if (!NoAccess) {
+    MapFlags |= ZX_VM_PERM_READ | ZX_VM_PERM_WRITE;
+  }
+  Status =
+      _zx_vmar_map(_zx_vmar_root_self(), MapFlags, 0, Vmo, 0, Size, &MapAddr);
+  if (UNLIKELY(Status != ZX_OK)) {
+    if (Status != ZX_ERR_NO_MEMORY || !AllowNoMem)
+      dieOnError(Status, "zx_vmar_map", Size);
+
+    Status = _zx_handle_close(Vmo);
+    CHECK_EQ(Status, ZX_OK);
+
+    MapAddr = 0;
+    Vmo = ZX_HANDLE_INVALID;
+    return false;
+  }
+
+  if (PreCommit) {
+    Status = _zx_vmar_op_range(_zx_vmar_root_self(), ZX_VMAR_OP_COMMIT, MapAddr,
+                               Size, nullptr, 0);
+    CHECK_EQ(Status, ZX_OK);
+  }
+
+  WindowBase = MapAddr;
+  WindowSize = Size;
+  return true;
+}
+
+void MemMapFuchsia::unmapImpl(uptr Addr, uptr Size) {
+  zx_status_t Status;
+
+  if (Size == WindowSize) {
+    // NOTE: Closing first and then unmapping seems slightly faster than doing
+    // the same operations in the opposite order.
+    Status = _zx_handle_close(Vmo);
+    CHECK_EQ(Status, ZX_OK);
+    Status = _zx_vmar_unmap(_zx_vmar_root_self(), Addr, Size);
+    CHECK_EQ(Status, ZX_OK);
+
+    MapAddr = WindowBase = WindowSize = 0;
+    Vmo = ZX_HANDLE_INVALID;
+  } else {
+    // Unmap the subrange.
+    Status = _zx_vmar_unmap(_zx_vmar_root_self(), Addr, Size);
+    CHECK_EQ(Status, ZX_OK);
+
+    // Decommit the pages that we just unmapped.
+    Status = _zx_vmo_op_range(Vmo, ZX_VMO_OP_DECOMMIT, Addr - MapAddr, Size,
+                              nullptr, 0);
+    CHECK_EQ(Status, ZX_OK);
+
+    if (Addr == WindowBase) {
+      WindowBase += Size;
+    }
+    WindowSize -= Size;
+  }
+}
+
+bool MemMapFuchsia::remapImpl(uptr Addr, uptr Size, const char *Name,
+                              uptr Flags) {
+  const bool PreCommit = !!(Flags & MAP_PRECOMMIT);
+  const bool NoAccess = !!(Flags & MAP_NOACCESS);
+
+  // NOTE: This will rename the *whole* VMO, not only the requested portion of
+  // it. But we cannot do better than this given the MemMap API. In practice,
+  // the upper layers of Scudo always pass the same Name for a given MemMap.
+  if (Name != nullptr)
+    setVmoName(Vmo, Name);
+
+  uptr MappedAddr;
+  zx_vm_option_t MapFlags = ZX_VM_ALLOW_FAULTS | ZX_VM_SPECIFIC_OVERWRITE;
+  if (!NoAccess) {
+    MapFlags |= ZX_VM_PERM_READ | ZX_VM_PERM_WRITE;
+  }
+  zx_status_t Status =
+      _zx_vmar_map(_zx_vmar_root_self(), MapFlags, Addr - getRootVmarBase(),
+                   Vmo, Addr - MapAddr, Size, &MappedAddr);
+  if (UNLIKELY(Status != ZX_OK)) {
+    dieOnError(Status, "zx_vmar_map", Size);
+  }
+  DCHECK_EQ(Addr, MappedAddr);
+
+  if (PreCommit) {
+    Status = _zx_vmar_op_range(_zx_vmar_root_self(), ZX_VMAR_OP_COMMIT, MapAddr,
+                               Size, nullptr, 0);
+    CHECK_EQ(Status, ZX_OK);
+  }
+
+  return true;
+}
+
+void MemMapFuchsia::releaseAndZeroPagesToOSImpl(uptr From, uptr Size) {
+  zx_status_t Status = _zx_vmo_op_range(Vmo, ZX_VMO_OP_DECOMMIT, From - MapAddr,
+                                        Size, nullptr, 0);
+  CHECK_EQ(Status, ZX_OK);
+}
+
+void MemMapFuchsia::setMemoryPermissionImpl(uptr Addr, uptr Size, uptr Flags) {
+  const bool NoAccess = !!(Flags & MAP_NOACCESS);
+  zx_vm_option_t MapFlags = 0;
+  if (!NoAccess) {
+    MapFlags |= ZX_VM_PERM_READ | ZX_VM_PERM_WRITE;
+  }
+
+  zx_status_t Status =
+      _zx_vmar_protect(_zx_vmar_root_self(), MapFlags, Addr, Size);
+  CHECK_EQ(Status, ZX_OK);
+}
+
+bool ReservedMemoryFuchsia::createImpl(UNUSED uptr Addr, uptr Size,
+                                       UNUSED const char *Name, uptr Flags) {
+  const bool AllowNoMem = !!(Flags & MAP_ALLOWNOMEM);
+
+  // Reserve memory by mapping the placeholder VMO without any permission.
+  zx_status_t Status = _zx_vmar_map(_zx_vmar_root_self(), ZX_VM_ALLOW_FAULTS, 0,
+                                    getPlaceholderVmo(), 0, Size, &Base);
+  if (UNLIKELY(Status != ZX_OK)) {
+    if (Status != ZX_ERR_NO_MEMORY || !AllowNoMem)
+      dieOnError(Status, "zx_vmar_map", Size);
+    return false;
+  }
+
+  Capacity = Size;
+  return true;
+}
+
+void ReservedMemoryFuchsia::releaseImpl() {
+  zx_status_t Status = _zx_vmar_unmap(_zx_vmar_root_self(), Base, Capacity);
+  CHECK_EQ(Status, ZX_OK);
+}
+
+ReservedMemoryFuchsia::MemMapT ReservedMemoryFuchsia::dispatchImpl(uptr Addr,
+                                                                   uptr Size) {
+  return ReservedMemoryFuchsia::MemMapT(Addr, Size);
+}
+
+} // namespace scudo
+
+#endif // SCUDO_FUCHSIA