[sanitizer] Use "fast mmap" kernel flag for shadow memory on macOS 10.13.4+

[kubamracek]

Summary

This speeds up process startup and teardown and also reduces lock contention when running multiple ASanified/TSanified processes simultaneously. Should greatly improve lit testing time.

[vitalybuka]

lib/sanitizer_common/sanitizer_mac.cc

758

Would you like to resolve this by moving into your new function?
Probably in the next patch.

[kubamracek]

lib/sanitizer_common/sanitizer_mac.cc

758

Yes, nice catch! Will send another patch after this one.

[delcypher]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

333

Is there a reason the use_xnu_fast_mmap isn't pushed into the Darwin implementation of internal_mmap()? Wouldn't it make more sense to be there given that xnu_fast_mmap is Darwin specific?

400

Again can't we move this into the Darwin implementation of internal_mmap(). This will also mean we avoid duplicating code.

lib/tsan/rtl/tsan_rtl.cc

365–367

__sanitizer::InitializePlatformEarly() is different from InitializePlatformEarly() that's called immediately afterwards?

[kubamracek]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

333

We don't want to use this flag in internal_mmap, that would apply to all mmap calls. We really only want this flag for the shadow memory. I can add another argument to internal_mmap, but that looks to me like it would complicate things more than clarify them.

lib/tsan/rtl/tsan_rtl.cc

365–367

Yes. Tsan has __tsan::InitializePlatformEarly.

[delcypher]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

333

Why wouldn't you want to take the fast path for all mmap calls? I suppose it's the fast path for large allocations but not necessarily smaller ones?

If we're going with this approach there should probably be a comment or note in the commit message explaining why the decision to use fast_mmap has to belong in these locations and not in internal_mmap.

I'd kind of feel that the fast mmap option really should be a flag because that would allow us to pass that flag and then handle it in internal_mmap() (i.e. on non Darwin platforms, strip out the flag and use it appropriately on Darwin). That would avoid having to change all call sites to internal_mmap().

The problem is mmap flag values are likely reserved and adding our own special value makes me feel uneasy because if another platform (or even Darwin) ever uses that flag value to mean something else, then semantics of internal_mmap will be broken. So I guess we have to do something else.

lib/tsan/rtl/tsan_rtl.cc

365–367

Ah. Personally I'd rather it read as

__sanitizer::InitializePlatformEarly();
__tsan::InitializePlatformEarly();

for clarity but I won't insist on this if you really want to write it how it is now.

[kubamracek]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

333

I'll add the comment with a bit of explanation, but I think you won't be fully satisfied because I don't understand what exactly the behavior of this flag is either. It was added into xnu specifically for the sanitizers and it's supposed to only be used for the shadow memory (giant region sizes), and not for other memory allocations. I don't know why.

The interface of internal_mmap is exactly the same as mmap, so I don't think we want to change it, or pass "fd" into "flags".

lib/tsan/rtl/tsan_rtl.cc

365–367

Sure, I'll change it.

[delcypher]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

336

Side note. This implementation will prevent the VM tag being set (see https://reviews.llvm.org/D49603 ) with fast mmap is being used. This needs to be fixed.

400

Side note. This implementation will prevent the VM tag being set (see https://reviews.llvm.org/D49603 ) with fast mmap is being used. This needs to be fixed.

[kubamracek]

Updating patch.

[delcypher]

lib/sanitizer_common/sanitizer_mac.cc

128

Thanks for fixing the VM tag issue!

lib/sanitizer_common/sanitizer_posix_libcdep.cc

338–340

The name internal_mmap_shadow(...) implies that this function is only for use in allocating shadow regions. However the name MmapFixedNoReserve() doesn't imply that. However, looking at the uses
of MmapFixedNoReserve() it looks like they are only used for allocating "shadow" memory currently, so this is probably okay but maybe it should be named internal_mmap_large() instead because semantically that's what differentiates it from internal_mmap()?

[kubamracek]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

338–340

“large” to me doesn’t imply what I mean here. A 10 MB mmap is “large”. But “shadow” regions imply they are proportional in size to the entire VM address space.

[delcypher]

lib/sanitizer_common/sanitizer_posix_libcdep.cc

338–340

Ok. By "large" I mean extremely "large" but I guess that could also be confusing. I guess the remaining thing to address is MmapFixedNoReserve() calling internal_mmap_shadow(). I don't like MmapFixedNoReserve() (which has a name that does not imply it is for shadow memory) calling a function whose name implies it is for use for shadow memory.

A comment on MmapFixedNoReserve() saying it is meant for allocating shadow memory might be sufficient.

[kubamracek]

Ok. By "large" I mean extremely "large" but I guess that could also be confusing. I guess the remaining thing to address is MmapFixedNoReserve() calling internal_mmap_shadow(). I don't like MmapFixedNoReserve() (which has a name that does not imply it is for shadow memory) calling a function whose name implies it is for use for shadow memory.

I've updated the patch to now split out MmapFixedNoReserve into MmapFixedNoReserveShadow. The latter is used when allocating shadow regions.

[delcypher]

Other than the kXnuFastMmapFd declaration. LGTM.

lib/sanitizer_common/sanitizer_mac.cc

121

Any particular reason to use a macro rather than static const int kXnuFastMmapFd = 0x4;? The later style seems used more in the sanitizer code base (at least the bits I've seen). This is a very minor issue though so I'll leave changing this to your discretion.

[kubamracek]

Changing the #define to a constant. Rebasing patch.

[kubamracek]

Adding MmapFixedNoAccessShadow.

[kubamracek]

This revision is marked as "Accepted", but the last round of changes are touching other files than before. @kcc, @dvyukov, @eugenis, are you okay with these changes?

[eugenis]

I don't like the new Mmap*Shadow functions. The number of different Mmap* functions is growing exponentially with any new feature, let's use flags and/or arguments instead.

It is also not clear from the user side what this Shadow suffix means exactly. If that's an optimization for huge mappings, can't we define a size threshold when it is more efficient than the alternative, and use it under the hood in common Mmap* calls?

lib/sanitizer_common/sanitizer_mac.cc

113

Leftover debug? More of the same below.

[kubamracek]

Okay, simplifying this patch back, and only applying the "fast mmap" flag based on a size threshold.

[eugenis]

LGTM

[kubamracek]

@earthdok

r346264 | d0k | 2018-11-06 12:42:19 -0800 (Tue, 06 Nov 2018) | 11 lines

[dfsan] Fix build after r346262

compiler-rt/lib/dfsan/dfsan.cc:426:3: error: call to 'InitializePlatformEarly' is ambiguous
  InitializePlatformEarly();
  ^~~~~~~~~~~~~~~~~~~~~~~
compiler-rt/lib/dfsan/../sanitizer_common/sanitizer_common.h:901:6: note: candidate function
void InitializePlatformEarly();
     ^
compiler-rt/lib/dfsan/dfsan.cc:391:13: note: candidate function
static void InitializePlatformEarly() {
            ^

Thanks and sorry about this! Please let me know if the failure is still there!

lib/asan/asan_rtl.cc

static void AsanInitInternal() {
   SetCheckFailedCallback(AsanCheckFailed);
   SetPrintfAndReportCallback(AppendToErrorMessageBuffer);
 
   __sanitizer_set_report_path(common_flags()->log_path);
 
   __asan_option_detect_stack_use_after_return =
       flags()->detect_stack_use_after_return;
 
+  __sanitizer::InitializePlatformEarly();
+
   // Re-exec ourselves if we need to set additional env or command line args.
   MaybeReexec();
 
   // Setup internal allocator callback.
   SetLowLevelAllocateMinAlignment(SHADOW_GRANULARITY);
   SetLowLevelAllocateCallback(OnLowLevelAllocate);
 
   InitializeAsanInterceptors();

lib/asan/asan_shadow_setup.cc

 // ---------------------- mmap -------------------- {{{1
 // Reserve memory range [beg, end].
 // We need to use inclusive range because end+1 may not be representable.
 void ReserveShadowMemoryRange(uptr beg, uptr end, const char *name) {
   CHECK_EQ((beg % GetMmapGranularity()), 0);
   CHECK_EQ(((end + 1) % GetMmapGranularity()), 0);
   uptr size = end - beg + 1;
   DecreaseTotalMmap(size);  // Don't count the shadow against mmap_limit_mb.
-  if (!MmapFixedNoReserve(beg, size, name)) {
+  if (!MmapFixedNoReserveShadow(beg, size, name)) {
     Report(
         "ReserveShadowMemoryRange failed while trying to map 0x%zx bytes. "
         "Perhaps you're using ulimit -v\n",
         size);
     Abort();
   }
   if (common_flags()->no_huge_pages_for_shadow) NoHugePagesInRegion(beg, size);
   if (common_flags()->use_madv_dontdump) DontDumpShadowMemory(beg, size);
if (Verbosity())
           "protect_shadow_gap=0:"
           " not protecting shadow gap, allocating gap's shadow\n"
           "|| `[%p, %p]` || ShadowGap's shadow ||\n",
           GapShadowBeg, GapShadowEnd);
     ReserveShadowMemoryRange(GapShadowBeg, GapShadowEnd,
                              "unprotected gap shadow");
     return;
   }
-  void *res = MmapFixedNoAccess(addr, size, "shadow gap");
+  void *res = MmapFixedNoAccessShadow(addr, size, "shadow gap");
   if (addr == (uptr)res) return;
   // A few pages at the start of the address space can not be protected.
   // But we really want to protect as much as possible, to prevent this memory
   // being returned as a result of a non-FIXED mmap().
   if (addr == kZeroBaseShadowStart) {
     uptr step = GetMmapGranularity();
     while (size > step && addr < kZeroBaseMaxShadowStart) {
       addr += step;
       size -= step;
-      void *res = MmapFixedNoAccess(addr, size, "shadow gap");
+      void *res = MmapFixedNoAccessShadow(addr, size, "shadow gap");
       if (addr == (uptr)res) return;
     }
   }
 
   Report(
       "ERROR: Failed to protect the shadow gap. "
       "ASan cannot proceed correctly. ABORTING.\n");
   DumpProcessMap();

lib/dfsan/dfsan.cc

static void dfsan_fini() {
   }
 }
 
 static void dfsan_init(int argc, char **argv, char **envp) {
   InitializeFlags();
 
   InitializePlatformEarly();
 
-  if (!MmapFixedNoReserve(ShadowAddr(), UnusedAddr() - ShadowAddr()))
+  if (!MmapFixedNoReserveShadow(ShadowAddr(), UnusedAddr() - ShadowAddr()))
     Die();
 
   // Protect the region of memory we don't use, to preserve the one-to-one
   // mapping from application to shadow memory. But if ASLR is disabled, Linux
   // will load our executable in the middle of our unused region. This mostly
   // works so long as the program doesn't use too much memory. We support this
   // case by disabling memory protection when ASLR is disabled.
   uptr init_addr = (uptr)&dfsan_init;
   if (!(init_addr >= UnusedAddr() && init_addr < AppAddr()))
-    MmapFixedNoAccess(UnusedAddr(), AppAddr() - UnusedAddr());
+    MmapFixedNoAccessShadow(UnusedAddr(), AppAddr() - UnusedAddr());
 
   InitializeInterceptors();
 
   // Register the fini callback to run when the program terminates successfully
   // or it is killed by the runtime.
   Atexit(dfsan_fini);
   AddDieCallback(dfsan_fini);
 
   __dfsan_label_info[kInitializingLabel].desc = "<init label>";
 }
 
 #if SANITIZER_CAN_USE_PREINIT_ARRAY
 __attribute__((section(".preinit_array"), used))
 static void (*dfsan_init_ptr)(int, char **, char **) = dfsan_init;
 #endif

lib/esan/esan.cpp

static void initializeShadow() {
   for (int i = 0; getShadowRegion(i, &ShadowStart, &ShadowEnd); ++i) {
     VPrintf(1, "Shadow #%d: [%zx-%zx) (%zuGB)\n", i, ShadowStart, ShadowEnd,
             (ShadowEnd - ShadowStart) >> 30);
 
     uptr Map = 0;
     if (__esan_which_tool == ESAN_WorkingSet) {
       // We want to identify all shadow pages that are touched so we start
       // out inaccessible.
-      Map = (uptr)MmapFixedNoAccess(ShadowStart, ShadowEnd- ShadowStart,
+      Map = (uptr)MmapFixedNoAccessShadow(ShadowStart, ShadowEnd- ShadowStart,
                                           "shadow");
     } else {
-      if (MmapFixedNoReserve(ShadowStart, ShadowEnd - ShadowStart, "shadow"))
+      if (MmapFixedNoReserveShadow(ShadowStart, ShadowEnd - ShadowStart,
+                                   "shadow"))
         Map = ShadowStart;
     }
     if (Map != ShadowStart) {
       Printf("FATAL: EfficiencySanitizer failed to map its shadow memory.\n");
       Die();
     }
 
     if (common_flags()->no_huge_pages_for_shadow)

lib/hwasan/hwasan_linux.cc

 
 namespace __hwasan {
 
 static void ReserveShadowMemoryRange(uptr beg, uptr end, const char *name) {
   CHECK_EQ((beg % GetMmapGranularity()), 0);
   CHECK_EQ(((end + 1) % GetMmapGranularity()), 0);
   uptr size = end - beg + 1;
   DecreaseTotalMmap(size);  // Don't count the shadow against mmap_limit_mb.
-  if (!MmapFixedNoReserve(beg, size, name)) {
+  if (!MmapFixedNoReserveShadow(beg, size, name)) {
     Report(
         "ReserveShadowMemoryRange failed while trying to map 0x%zx bytes. "
         "Perhaps you're using ulimit -v\n",
         size);
     Abort();
   }
 }
 
 static void ProtectGap(uptr addr, uptr size) {
   if (!size)
     return;
-  void *res = MmapFixedNoAccess(addr, size, "shadow gap");
+  void *res = MmapFixedNoAccessShadow(addr, size, "shadow gap");
   if (addr == (uptr)res)
     return;
   // A few pages at the start of the address space can not be protected.
   // But we really want to protect as much as possible, to prevent this memory
   // being returned as a result of a non-FIXED mmap().
   if (addr == 0) {
     uptr step = GetMmapGranularity();
     while (size > step) {
       addr += step;
       size -= step;
-      void *res = MmapFixedNoAccess(addr, size, "shadow gap");
+      void *res = MmapFixedNoAccessShadow(addr, size, "shadow gap");
       if (addr == (uptr)res)
         return;
     }
   }
 
   Report(
       "ERROR: Failed to protect shadow gap [%p, %p]. "
       "HWASan cannot proceed correctly. ABORTING.\n", (void *)addr,

lib/msan/msan_linux.cc

if (!MemoryRangeIsAvailable(beg, end)) {
       return false;
     }
   }
   return true;
 }
 
 static bool ProtectMemoryRange(uptr beg, uptr size, const char *name) {
   if (size > 0) {
-    void *addr = MmapFixedNoAccess(beg, size, name);
+    void *addr = MmapFixedNoAccessShadow(beg, size, name);
     if (beg == 0 && addr) {
       // Depending on the kernel configuration, we may not be able to protect
       // the page at address zero.
       uptr gap = 16 * GetPageSizeCached();
       beg += gap;
       size -= gap;
-      addr = MmapFixedNoAccess(beg, size, name);
+      addr = MmapFixedNoAccessShadow(beg, size, name);
     }
     if ((uptr)addr != beg) {
       uptr end = beg + size - 1;
       Printf("FATAL: Cannot protect memory range %p - %p (%s).\n", beg, end,
              name);
       return false;
     }
   }
for (unsigned i = 0; i < kMemoryLayoutSize; ++i) {
     bool protect = type == MappingDesc::INVALID ||
                    (!init_origins && type == MappingDesc::ORIGIN);
     CHECK(!(map && protect));
     if (!map && !protect)
       CHECK(type == MappingDesc::APP);
     if (map) {
       if (!CheckMemoryRangeAvailability(start, size))
         return false;
-      if (!MmapFixedNoReserve(start, size, kMemoryLayout[i].name))
+      if (!MmapFixedNoReserveShadow(start, size, kMemoryLayout[i].name))
         return false;
       if (common_flags()->use_madv_dontdump)
         DontDumpShadowMemory(start, size);
     }
     if (protect) {
       if (!CheckMemoryRangeAvailability(start, size))
         return false;
       if (!ProtectMemoryRange(start, size, kMemoryLayout[i].name))

lib/msan/msan_poisoning.cc

if (page_beg >= page_end) {
       REAL(memset)((void *)shadow_beg, 0, shadow_end - shadow_beg);
     } else {
       if (page_beg != shadow_beg) {
         REAL(memset)((void *)shadow_beg, 0, page_beg - shadow_beg);
       }
       if (page_end != shadow_end) {
         REAL(memset)((void *)page_end, 0, shadow_end - page_end);
       }
-      if (!MmapFixedNoReserve(page_beg, page_end - page_beg))
+      if (!MmapFixedNoReserveShadow(page_beg, page_end - page_beg))
         Die();
     }
   }
 }
 
 void SetOrigin(const void *dst, uptr size, u32 origin) {
   // Origin mapping is 4 bytes per 4 bytes of application memory.
   // Here we extend the range such that its left and right bounds are both

lib/sanitizer_common/sanitizer_common.h

INLINE void *MmapOrDieQuietly(uptr size, const char *mem_type) {
   return MmapOrDie(size, mem_type, /*raw_report*/ true);
 }
 void UnmapOrDie(void *addr, uptr size);
 // Behaves just like MmapOrDie, but tolerates out of memory condition, in that
 // case returns nullptr.
 void *MmapOrDieOnFatalError(uptr size, const char *mem_type);
 bool MmapFixedNoReserve(uptr fixed_addr, uptr size, const char *name = nullptr)
      WARN_UNUSED_RESULT;
+bool MmapFixedNoReserveShadow(uptr fixed_addr, uptr size,
+                              const char *name = nullptr) WARN_UNUSED_RESULT;
 void *MmapNoReserveOrDie(uptr size, const char *mem_type);
 void *MmapFixedOrDie(uptr fixed_addr, uptr size);
 // Behaves just like MmapFixedOrDie, but tolerates out of memory condition, in
 // that case returns nullptr.
 void *MmapFixedOrDieOnFatalError(uptr fixed_addr, uptr size);
 void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name = nullptr);
+void *MmapFixedNoAccessShadow(uptr fixed_addr, uptr size,
+                              const char *name = nullptr);
 void *MmapNoAccess(uptr size);
 // Map aligned chunk of address space; size and alignment are powers of two.
 // Dies on all but out of memory errors, in the latter case returns nullptr.
 void *MmapAlignedOrDieOnFatalError(uptr size, uptr alignment,
                                    const char *mem_type);
 // Disallow access to a memory range.  Use MmapFixedNoAccess to allocate an
 // unaccessible memory.
 bool MprotectNoAccess(uptr addr, uptr size);

  private:
   // Platform specific initialization.
   void InitPcSpBp();
   uptr GetAddress() const;
   WriteFlag GetWriteFlag() const;
   bool IsMemoryAccess() const;
 };
 
+void InitializePlatformEarly();
 void MaybeReexec();
 
 template <typename Fn>
 class RunOnDestruction {
  public:
   explicit RunOnDestruction(Fn fn) : fn_(fn) {}
   ~RunOnDestruction() { fn_(); }
 

lib/sanitizer_common/sanitizer_fuchsia.cc

void GetThreadStackTopAndBottom(bool, uptr *stack_top, uptr *stack_bottom) {
   size_t size;
   CHECK_EQ(pthread_attr_getstack(&attr, &base, &size), 0);
   CHECK_EQ(pthread_attr_destroy(&attr), 0);
 
   *stack_bottom = reinterpret_cast<uptr>(base);
   *stack_top = *stack_bottom + size;
 }
 
+void InitializePlatformEarly() {}
 void MaybeReexec() {}
 void CheckASLR() {}
 void PlatformPrepareForSandboxing(__sanitizer_sandbox_arguments *args) {}
 void DisableCoreDumperIfNecessary() {}
 void InstallDeadlySignalHandlers(SignalHandlerType handler) {}
 void SetAlternateSignalStack() {}
 void UnsetAlternateSignalStack() {}
 void InitTlsSize() {}
UnmapOrDieVmar(reinterpret_cast<void *>(addr), size,
                  static_cast<zx_handle_t>(os_handle_));
 }
 
 // This should never be called.
 void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name) {
   UNIMPLEMENTED();
 }
 
+void *MmapFixedNoAccessShadow(uptr fixed_addr, uptr size, const char *name) {
+  UNIMPLEMENTED();
+}
+
 void *MmapAlignedOrDieOnFatalError(uptr size, uptr alignment,
                                    const char *mem_type) {
   CHECK_GE(size, PAGE_SIZE);
   CHECK(IsPowerOfTwo(size));
   CHECK(IsPowerOfTwo(alignment));
 
   zx_handle_t vmo;
   zx_status_t status = _zx_vmo_create(size, 0, &vmo);

lib/sanitizer_common/sanitizer_linux.cc

return internal_syscall(SYSCALL(mmap), (uptr)addr, length, prot, flags, fd,
                           offset);
 #else
   // mmap2 specifies file offset in 4096-byte units.
   CHECK(IsAligned(offset, 4096));
   return internal_syscall(SYSCALL(mmap2), addr, length, prot, flags, fd,
                           offset / 4096);
 #endif
 }
-#endif // !SANITIZER_S390 && !SANITIZER_OPENBSD
+uptr internal_mmap_shadow(void *addr, uptr length, int prot, int flags, int fd,
+                          OFF_T offset) {
+  return internal_mmap(addr, length, prot, flags, fd, offset);
+}
+#endif  // !SANITIZER_S390 && !SANITIZER_OPENBSD
 
 #if !SANITIZER_OPENBSD
 uptr internal_munmap(void *addr, uptr length) {
   return internal_syscall_ptr(SYSCALL(munmap), (uptr)addr, length);
 }
 
 int internal_mprotect(void *addr, uptr length, int prot) {
   return internal_syscall_ptr(SYSCALL(mprotect), (uptr)addr, length, prot);
# endif
   *sp = ucontext->uc_mcontext.gregs[15];
 #else
 # error "Unsupported arch"
 #endif
 }
 
 void SignalContext::InitPcSpBp() { GetPcSpBp(context, &pc, &sp, &bp); }
 
+void InitializePlatformEarly() {
+  // Do nothing.
+}
+
 void MaybeReexec() {
   // No need to re-exec on Linux.
 }
 
 void CheckASLR() {
 #if SANITIZER_NETBSD
   int mib[3];
   int paxflags;

lib/sanitizer_common/sanitizer_linux_s390.cc

# endif
   };
 # ifdef __s390x__
   return syscall(__NR_mmap, &params);
 # else
   return syscall(__NR_mmap2, &params);
 # endif
 }
 
+uptr internal_mmap_shadow(void *addr, uptr length, int prot, int flags, int fd,
+                          OFF_T offset) {
+  return internal_mmap(addr, length, prot, flags, fd, offset);
+}
+
 uptr internal_clone(int (*fn)(void *), void *child_stack, int flags, void *arg,
                     int *parent_tidptr, void *newtls, int *child_tidptr) {
   if (!fn || !child_stack)
     return -EINVAL;
   CHECK_EQ(0, (uptr)child_stack % 16);
   // Minimum frame size.
 #ifdef __s390x__
   child_stack = (char *)child_stack - 160;

lib/sanitizer_common/sanitizer_mac.h

enum MacosVersion {
   MACOS_VERSION_SNOW_LEOPARD,
   MACOS_VERSION_LION,
   MACOS_VERSION_MOUNTAIN_LION,
   MACOS_VERSION_MAVERICKS,
   MACOS_VERSION_YOSEMITE,
   MACOS_VERSION_EL_CAPITAN,
   MACOS_VERSION_SIERRA,
   MACOS_VERSION_HIGH_SIERRA,
+  MACOS_VERSION_HIGH_SIERRA_DOT_RELEASE_4,
   MACOS_VERSION_MOJAVE,
   MACOS_VERSION_UNKNOWN_NEWER
 };
 
 MacosVersion GetMacosVersion();
 
 char **GetEnviron();
 

lib/sanitizer_common/sanitizer_mac.cc

 
 // From <mach/vm_statistics.h>, but not on older OSs.
 #ifndef VM_MEMORY_SANITIZER
 #define VM_MEMORY_SANITIZER 99
 #endif
 
 uptr internal_mmap(void *addr, size_t length, int prot, int flags,
                    int fd, u64 offset) {
+  Report("internal_mmap(%p,%p)\n", addr, length);
   if (fd == -1) fd = VM_MAKE_TAG(VM_MEMORY_SANITIZER);
   if (&__mmap) return (uptr)__mmap(addr, length, prot, flags, fd, offset);
   return (uptr)mmap(addr, length, prot, flags, fd, offset);
 }
 
+// XNU on Darwin provides a mmap flag that optimizes allocation/deallocation of
+// giant memory regions (i.e. shadow memory regions).
+static const long kXnuFastMmapFd = 0x4;
+static bool use_xnu_fast_mmap = false;
+
+uptr internal_mmap_shadow(void *addr, size_t length, int prot, int flags,
+                          int fd, u64 offset) {
+  Report("internal_mmap_shadow(%p,%p)\n", addr, length);
+  if (fd == -1) {
+    int fd = VM_MAKE_TAG(VM_MEMORY_SANITIZER);
+    if (use_xnu_fast_mmap) fd |= kXnuFastMmapFd;
+  }
+  if (&__mmap) return (uptr)__mmap(addr, length, prot, flags, fd, offset);
+  return (uptr)mmap(addr, length, prot, flags, fd, offset);
+}
+
 uptr internal_munmap(void *addr, uptr length) {
   if (&__munmap) return __munmap(addr, length);
   return munmap(addr, length);
 }
 
 int internal_mprotect(void *addr, uptr length, int prot) {
   return mprotect(addr, length, prot);
 }
MacosVersion GetMacosVersionInternal() {
   int mib[2] = { CTL_KERN, KERN_OSRELEASE };
   char version[100];
   uptr len = 0, maxlen = sizeof(version) / sizeof(version[0]);
   for (uptr i = 0; i < maxlen; i++) version[i] = '\0';
   // Get the version length.
   CHECK_NE(sysctl(mib, 2, 0, &len, 0, 0), -1);
   CHECK_LT(len, maxlen);
   CHECK_NE(sysctl(mib, 2, version, &len, 0, 0), -1);
-  switch (version[0]) {
-    case '9': return MACOS_VERSION_LEOPARD;
-    case '1': {
-      switch (version[1]) {
-        case '0': return MACOS_VERSION_SNOW_LEOPARD;
-        case '1': return MACOS_VERSION_LION;
-        case '2': return MACOS_VERSION_MOUNTAIN_LION;
-        case '3': return MACOS_VERSION_MAVERICKS;
-        case '4': return MACOS_VERSION_YOSEMITE;
-        case '5': return MACOS_VERSION_EL_CAPITAN;
-        case '6': return MACOS_VERSION_SIERRA;
-        case '7': return MACOS_VERSION_HIGH_SIERRA;
-        case '8': return MACOS_VERSION_MOJAVE;
-        default:
-          if (IsDigit(version[1]))
-            return MACOS_VERSION_UNKNOWN_NEWER;
-          else
-            return MACOS_VERSION_UNKNOWN;
-      }
-    }
-    default: return MACOS_VERSION_UNKNOWN;
+
+  // Expect <major>.<minor>(.<patch>)
+  CHECK_GE(len, 3);
+  const char *p = version;
+  int major = internal_simple_strtoll(p, &p, /*base=*/10);
+  if (*p != '.') return MACOS_VERSION_UNKNOWN;
+  p += 1;
+  int minor = internal_simple_strtoll(p, &p, /*base=*/10);
+  if (*p != '.') return MACOS_VERSION_UNKNOWN;
+
+  switch (major) {
+    case 9: return MACOS_VERSION_LEOPARD;
+    case 10: return MACOS_VERSION_SNOW_LEOPARD;
+    case 11: return MACOS_VERSION_LION;
+    case 12: return MACOS_VERSION_MOUNTAIN_LION;
+    case 13: return MACOS_VERSION_MAVERICKS;
+    case 14: return MACOS_VERSION_YOSEMITE;
+    case 15: return MACOS_VERSION_EL_CAPITAN;
+    case 16: return MACOS_VERSION_SIERRA;
+    case 17:
+      // Not a typo, 17.5 Darwin Kernel Version maps to High Sierra 10.13.4.
+      if (minor >= 5)
+        return MACOS_VERSION_HIGH_SIERRA_DOT_RELEASE_4;
+      return MACOS_VERSION_HIGH_SIERRA;
+    case 18:
+      return MACOS_VERSION_MOJAVE;
+      if (major < 9) return MACOS_VERSION_UNKNOWN;
   }
 }
 
 MacosVersion GetMacosVersion() {
   atomic_uint32_t *cache =
       reinterpret_cast<atomic_uint32_t*>(&cached_macos_version);
   MacosVersion result =
       static_cast<MacosVersion>(atomic_load(cache, memory_order_acquire));
# elif defined(__i386__)
   *sp = ucontext->uc_mcontext->__ss.__esp;
 # else
 # error "Unknown architecture"
 # endif
 }
 
 void SignalContext::InitPcSpBp() { GetPcSpBp(context, &pc, &sp, &bp); }
 
+void InitializePlatformEarly() {
+  // Only use xnu_fast_mmap when on x86_64 and the OS supports it.
+  use_xnu_fast_mmap =
+#if defined(__x86_64__)
+      GetMacosVersion() >= MACOS_VERSION_HIGH_SIERRA_DOT_RELEASE_4;
+#else
+      false;
+#endif
+}
+
 #if !SANITIZER_GO
 static const char kDyldInsertLibraries[] = "DYLD_INSERT_LIBRARIES";
 LowLevelAllocator allocator_for_env;
 
 // Change the value of the env var |name|, leaking the original value.
 // If |name_value| is NULL, the variable is deleted from the environment,
 // otherwise the corresponding "NAME=value" string is replaced with
 // |name_value|.
bool DyldNeedsEnvVariable() {
   // DYLD_INSERT_LIBRARIES is not set. However, checking OS version via
   // GetMacosVersion() doesn't work for the simulator. Let's instead check
   // `dyldVersionNumber`, which is exported by dyld, against a known version
   // number from the first OS release where this appeared.
   return dyldVersionNumber < kMinDyldVersionWithAutoInterposition;
 }
 
 void MaybeReexec() {
   // FIXME: This should really live in some "InitializePlatform" method.
   MonotonicNanoTime();
 
   if (ReexecDisabled()) return;
 
   // Make sure the dynamic runtime library is preloaded so that the
   // wrappers work. If it is not, set DYLD_INSERT_LIBRARIES and re-exec
   // ourselves.
   Dl_info info;

lib/sanitizer_common/sanitizer_mac_libcdep.cc

 #include "sanitizer_platform.h"
 #if SANITIZER_MAC
 #include "sanitizer_mac.h"
 
 namespace __sanitizer {
 
 void RestrictMemoryToMaxAddress(uptr max_address) {
   uptr size_to_mmap = GetMaxUserVirtualAddress() + 1 - max_address;
-  void *res = MmapFixedNoAccess(max_address, size_to_mmap, "high gap");
+  void *res = MmapFixedNoAccessShadow(max_address, size_to_mmap, "high gap");
   CHECK(res);
 }
 
 }  // namespace __sanitizer
 
 #endif  // SANITIZER_MAC

lib/sanitizer_common/sanitizer_openbsd.cc

 
 namespace __sanitizer {
 
 uptr internal_mmap(void *addr, size_t length, int prot, int flags, int fd,
                    u64 offset) {
   return (uptr)mmap(addr, length, prot, flags, fd, offset);
 }
 
+uptr internal_mmap_shadow(void *addr, uptr length, int prot, int flags, int fd,
+                          OFF_T offset) {
+  return internal_mmap(addr, length, prot, flags, fd, offset);
+}
+
 uptr internal_munmap(void *addr, uptr length) { return munmap(addr, length); }
 
 int internal_mprotect(void *addr, uptr length, int prot) {
   return mprotect(addr, length, prot);
 }
 
 uptr ReadBinaryName(/*out*/char *buf, uptr buf_len) {
   // On OpenBSD we cannot get the full path

lib/sanitizer_common/sanitizer_posix.h

 uptr internal_open(const char *filename, int flags);
 uptr internal_open(const char *filename, int flags, u32 mode);
 uptr internal_close(fd_t fd);
 
 uptr internal_read(fd_t fd, void *buf, uptr count);
 uptr internal_write(fd_t fd, const void *buf, uptr count);
 
 // Memory
-uptr internal_mmap(void *addr, uptr length, int prot, int flags,
-                   int fd, OFF_T offset);
+uptr internal_mmap(void *addr, uptr length, int prot, int flags, int fd,
+                   OFF_T offset);
+uptr internal_mmap_shadow(void *addr, uptr length, int prot, int flags, int fd,
+                          OFF_T offset);
 uptr internal_munmap(void *addr, uptr length);
 int internal_mprotect(void *addr, uptr length, int prot);
 
 // OS
 uptr internal_filesize(fd_t fd);  // -1 on error.
 uptr internal_stat(const char *path, void *buf);
 uptr internal_lstat(const char *path, void *buf);
 uptr internal_fstat(fd_t fd, void *buf);

lib/sanitizer_common/sanitizer_posix_libcdep.cc

int GetNamedMappingFd(const char *name, uptr size) {
   int res = internal_ftruncate(fd, size);
   CHECK_EQ(0, res);
   res = shm_unlink(shmname);
   CHECK_EQ(0, res);
   return fd;
 }
 #endif
 
-bool MmapFixedNoReserve(uptr fixed_addr, uptr size, const char *name) {
+static bool MmapFixedNoReserveCommon(uptr fixed_addr, uptr size,
+                                     const char *name, bool shadow) {
   int fd = name ? GetNamedMappingFd(name, size) : -1;
   unsigned flags = MAP_PRIVATE | MAP_FIXED | MAP_NORESERVE;
   if (fd == -1) flags |= MAP_ANON;
 
   uptr PageSize = GetPageSizeCached();
-  uptr p = internal_mmap((void *)(fixed_addr & ~(PageSize - 1)),
-                         RoundUpTo(size, PageSize), PROT_READ | PROT_WRITE,
-                         flags, fd, 0);
+  uptr p = (shadow ? internal_mmap_shadow : internal_mmap)(
+      (void *)(fixed_addr & ~(PageSize - 1)), RoundUpTo(size, PageSize),
+      PROT_READ | PROT_WRITE, flags, fd, 0);
   int reserrno;
   if (internal_iserror(p, &reserrno)) {
     Report("ERROR: %s failed to "
            "allocate 0x%zx (%zd) bytes at address %zx (errno: %d)\n",
            SanitizerToolName, size, size, fixed_addr, reserrno);
     return false;
   }
   IncreaseTotalMmap(size);
   return true;
 }
 
+bool MmapFixedNoReserve(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoReserveCommon(fixed_addr, size, name, /*shadow=*/ false);
+}
+
+bool MmapFixedNoReserveShadow(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoReserveCommon(fixed_addr, size, name, /*shadow=*/ true);
+}
+
 uptr ReservedAddressRange::Init(uptr size, const char *name, uptr fixed_addr) {
   // We don't pass `name` along because, when you enable `decorate_proc_maps`
   // AND actually use a named mapping AND are using a sanitizer intercepting
   // `open` (e.g. TSAN, ESAN), then you'll get a failure during initialization.
   // TODO(flowerhack): Fix the implementation of GetNamedMappingFd to solve
   // this problem.
-  base_ = fixed_addr ? MmapFixedNoAccess(fixed_addr, size) : MmapNoAccess(size);
+  base_ = fixed_addr ? MmapFixedNoAccessShadow(fixed_addr, size)
+                     : MmapNoAccess(size);
   size_ = size;
   name_ = name;
   (void)os_handle_;  // unsupported
   return reinterpret_cast<uptr>(base_);
 }
 
 // Uses fixed_addr for now.
 // Will use offset instead once we've implemented this function for real.
if (addr == reinterpret_cast<uptr>(base_))
     // If we unmap the whole range, just null out the base.
     base_ = (size == size_) ? nullptr : reinterpret_cast<void*>(addr + size);
   else
     CHECK_EQ(addr + size, reinterpret_cast<uptr>(base_) + size_);
   size_ -= size;
   UnmapOrDie(reinterpret_cast<void*>(addr), size);
 }
 
-void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name) {
+void *MmapFixedNoAccessCommon(uptr fixed_addr, uptr size, const char *name,
+                              bool shadow) {
   int fd = name ? GetNamedMappingFd(name, size) : -1;
   unsigned flags = MAP_PRIVATE | MAP_FIXED | MAP_NORESERVE;
   if (fd == -1) flags |= MAP_ANON;
 
-  uptr p = internal_mmap((void *)fixed_addr, size, PROT_NONE, flags, fd, 0);
+  uptr p = (shadow ? internal_mmap_shadow : internal_mmap)(
+      (void *)fixed_addr, size, PROT_NONE, flags, fd, 0);
   if (internal_iserror(p))
     return nullptr;
   return (void *)p;
 }
 
+void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoAccessCommon(fixed_addr, size, name, /*shadow=*/ false);
+}
+
+void *MmapFixedNoAccessShadow(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoAccessCommon(fixed_addr, size, name, /*shadow=*/ true);
+}
+
 void *MmapNoAccess(uptr size) {
   unsigned flags = MAP_PRIVATE | MAP_ANON | MAP_NORESERVE;
   uptr p = internal_mmap(nullptr, size, PROT_NONE, flags, -1, 0);
   if (internal_iserror(p))
     return nullptr;
   return (void *)p;
 }
 

lib/sanitizer_common/sanitizer_rtems.cc

void GetThreadStackAndTls(bool main, uptr *stk_addr, uptr *stk_size,
                           uptr *tls_addr, uptr *tls_size) {
   uptr stack_top, stack_bottom;
   GetThreadStackTopAndBottom(main, &stack_top, &stack_bottom);
   *stk_addr = stack_bottom;
   *stk_size = stack_top - stack_bottom;
   *tls_addr = *tls_size = 0;
 }
 
+void InitializePlatformEarly() {}
 void MaybeReexec() {}
 void CheckASLR() {}
 void DisableCoreDumperIfNecessary() {}
 void InstallDeadlySignalHandlers(SignalHandlerType handler) {}
 void SetAlternateSignalStack() {}
 void UnsetAlternateSignalStack() {}
 void InitTlsSize() {}
 

lib/sanitizer_common/sanitizer_win.cc

if (p == 0) {
     Report("ERROR: %s failed to "
            "allocate %p (%zd) bytes at %p (error code: %d)\n",
            SanitizerToolName, size, size, fixed_addr, GetLastError());
     return false;
   }
   return true;
 }
 
+bool MmapFixedNoReserveShadow(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoReserve(fixed_addr, size, name);
+}
+
 // Memory space mapped by 'MmapFixedOrDie' must have been reserved by
 // 'MmapFixedNoAccess'.
 void *MmapFixedOrDie(uptr fixed_addr, uptr size) {
   void *p = VirtualAlloc((LPVOID)fixed_addr, size,
       MEM_COMMIT, PAGE_READWRITE);
   if (p == 0) {
     char mem_type[30];
     internal_snprintf(mem_type, sizeof(mem_type), "memory at address 0x%zx",

 }
 
 void *MmapNoReserveOrDie(uptr size, const char *mem_type) {
   // FIXME: make this really NoReserve?
   return MmapOrDie(size, mem_type);
 }
 
 uptr ReservedAddressRange::Init(uptr size, const char *name, uptr fixed_addr) {
-  base_ = fixed_addr ? MmapFixedNoAccess(fixed_addr, size) : MmapNoAccess(size);
+  base_ = fixed_addr ? MmapFixedNoAccessShadow(fixed_addr, size)
+                     : MmapNoAccess(size);
   size_ = size;
   name_ = name;
   (void)os_handle_;  // unsupported
   return reinterpret_cast<uptr>(base_);
 }
 
 
 void *MmapFixedNoAccess(uptr fixed_addr, uptr size, const char *name) {
   (void)name; // unsupported
   void *res = VirtualAlloc((LPVOID)fixed_addr, size,
                            MEM_RESERVE, PAGE_NOACCESS);
   if (res == 0)
     Report("WARNING: %s failed to "
            "mprotect %p (%zd) bytes at %p (error code: %d)\n",
            SanitizerToolName, size, size, fixed_addr, GetLastError());
   return res;
 }
 
+void *MmapFixedNoAccessShadow(uptr fixed_addr, uptr size, const char *name) {
+  return MmapFixedNoAccess(fixed_addr, size, name);
+}
+
 void *MmapNoAccess(uptr size) {
   void *res = VirtualAlloc(nullptr, size, MEM_RESERVE, PAGE_NOACCESS);
   if (res == 0)
     Report("WARNING: %s failed to "
            "mprotect %p (%zd) bytes (error code: %d)\n",
            SanitizerToolName, size, size, GetLastError());
   return res;
 }

 uptr ReadLongProcessName(/*out*/char *buf, uptr buf_len) {
   return ReadBinaryName(buf, buf_len);
 }
 
 void CheckVMASize() {
   // Do nothing.
 }
 
+void InitializePlatformEarly() {
+  // Do nothing.
+}
+
 void MaybeReexec() {
   // No need to re-exec on Windows.
 }
 
 void CheckASLR() {
   // Do nothing
 }
 

lib/tsan/rtl/tsan_platform_posix.cc

if (!DontDumpShadowMemory(addr, size)) {
       Printf(kShadowMemoryMappingHint, "MADV_DONTDUMP", "use_madv_dontdump");
       Die();
     }
 }
 
 #if !SANITIZER_GO
 void InitializeShadowMemory() {
   // Map memory shadow.
-  if (!MmapFixedNoReserve(ShadowBeg(), ShadowEnd() - ShadowBeg(), "shadow")) {
+  if (!MmapFixedNoReserveShadow(ShadowBeg(), ShadowEnd() - ShadowBeg(),
+                                "shadow")) {
     Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
     Printf("FATAL: Make sure to compile with -fPIE and to link with -pie.\n");
     Die();
   }
   // This memory range is used for thread stacks and large user mmaps.
   // Frequently a thread uses only a small part of stack and similarly
   // a program uses a small part of large mmap. On some programs
   // we see 20% memory usage reduction without huge pages for this range.
#endif
   DontDumpShadow(ShadowBeg(), ShadowEnd() - ShadowBeg());
   DPrintf("memory shadow: %zx-%zx (%zuGB)\n",
       ShadowBeg(), ShadowEnd(),
       (ShadowEnd() - ShadowBeg()) >> 30);
 
   // Map meta shadow.
   const uptr meta = MetaShadowBeg();
   const uptr meta_size = MetaShadowEnd() - meta;
-  if (!MmapFixedNoReserve(meta, meta_size, "meta shadow")) {
+  if (!MmapFixedNoReserveShadow(meta, meta_size, "meta shadow")) {
     Printf("FATAL: ThreadSanitizer can not mmap the shadow memory\n");
     Printf("FATAL: Make sure to compile with -fPIE and to link with -pie.\n");
     Die();
   }
   NoHugePagesInShadow(meta, meta_size);
   DontDumpShadow(meta, meta_size);
   DPrintf("meta shadow: %zx-%zx (%zuGB)\n",
       meta, meta + meta_size, meta_size >> 30);
 
   InitializeShadowMemoryPlatform();
 }
 
 static void ProtectRange(uptr beg, uptr end) {
   CHECK_LE(beg, end);
   if (beg == end)
     return;
-  if (beg != (uptr)MmapFixedNoAccess(beg, end - beg)) {
+  if (beg != (uptr)MmapFixedNoAccessShadow(beg, end - beg)) {
     Printf("FATAL: ThreadSanitizer can not protect [%zx,%zx]\n", beg, end);
     Printf("FATAL: Make sure you are not using unlimited stack\n");
     Die();
   }
 }
 
 void CheckAndProtect() {
   // Ensure that the binary is indeed compiled with -pie.

lib/tsan/rtl/tsan_rtl.cc

 
 void MapShadow(uptr addr, uptr size) {
   // Global data is not 64K aligned, but there are no adjacent mappings,
   // so we can get away with unaligned mapping.
   // CHECK_EQ(addr, addr & ~((64 << 10) - 1));  // windows wants 64K alignment
   const uptr kPageSize = GetPageSizeCached();
   uptr shadow_begin = RoundDownTo((uptr)MemToShadow(addr), kPageSize);
   uptr shadow_end = RoundUpTo((uptr)MemToShadow(addr + size), kPageSize);
-  if (!MmapFixedNoReserve(shadow_begin, shadow_end - shadow_begin, "shadow"))
+  if (!MmapFixedNoReserveShadow(shadow_begin, shadow_end - shadow_begin,
+                                "shadow"))
     Die();
 
   // Meta shadow is 2:1, so tread carefully.
   static bool data_mapped = false;
   static uptr mapped_meta_end = 0;
   uptr meta_begin = (uptr)MemToMeta(addr);
   uptr meta_end = (uptr)MemToMeta(addr + size);
   meta_begin = RoundDownTo(meta_begin, 64 << 10);
   meta_end = RoundUpTo(meta_end, 64 << 10);
   if (!data_mapped) {
     // First call maps data+bss.
     data_mapped = true;
-    if (!MmapFixedNoReserve(meta_begin, meta_end - meta_begin, "meta shadow"))
+    if (!MmapFixedNoReserveShadow(meta_begin, meta_end - meta_begin,
+                                  "meta shadow"))
       Die();
   } else {
     // Mapping continous heap.
     // Windows wants 64K alignment.
     meta_begin = RoundDownTo(meta_begin, 64 << 10);
     meta_end = RoundUpTo(meta_end, 64 << 10);
     if (meta_end <= mapped_meta_end)
       return;
     if (meta_begin < mapped_meta_end)
       meta_begin = mapped_meta_end;
-    if (!MmapFixedNoReserve(meta_begin, meta_end - meta_begin, "meta shadow"))
+    if (!MmapFixedNoReserveShadow(meta_begin, meta_end - meta_begin,
+                                  "meta shadow"))
       Die();
     mapped_meta_end = meta_end;
   }
   VPrintf(2, "mapped meta shadow for (%p-%p) at (%p-%p)\n",
       addr, addr+size, meta_begin, meta_end);
 }
 
 void MapThreadTrace(uptr addr, uptr size, const char *name) {
void Initialize(ThreadState *thr) {
   SetCheckFailedCallback(TsanCheckFailed);
 
   ctx = new(ctx_placeholder) Context;
   const char *options = GetEnv(SANITIZER_GO ? "GORACE" : "TSAN_OPTIONS");
   CacheBinaryName();
   CheckASLR();
   InitializeFlags(&ctx->flags, options);
   AvoidCVE_2016_2143();
-  InitializePlatformEarly();
+  __sanitizer::InitializePlatformEarly();
+  __tsan::InitializePlatformEarly();
+
 #if !SANITIZER_GO
   // Re-exec ourselves if we need to set additional env or command line args.
   MaybeReexec();
 
   InitializeAllocator();
   ReplaceSystemMalloc();
 #endif
   if (common_flags()->detect_deadlocks)