[sanitizer] Introduce a vDSO aware time function, and use it in the allocator [redo]

[cryptoad]

Summary

Redo of D40657, which had the initial discussion. The initial code had to move
into a libcdep file, and things had to be shuffled accordingly.

NanoTime is a time sink when checking whether or not to release memory to
the OS. While reducing the amount of calls to said function is in the works,
another solution that was found to be beneficial was to use a timing function
that can leverage the vDSO.

We hit a couple of snags along the way, like the fact that the glibc crashes
when clock_gettime is called from a preinit_array, or the fact that
__vdso_clock_gettime is mangled (for security purposes) and can't be used
directly, and also that clock_gettime can be intercepted.

The proposed solution takes care of all this as far as I can tell, and
significantly improve performances and some Scudo load tests with memory
reclaiming enabled.

@mcgrathr: please feel free to follow up on
https://reviews.llvm.org/D40657#940857 here. I posted a reply at
https://reviews.llvm.org/D40657#940974.

[krytarowski]

lib/sanitizer_common/sanitizer_common_interceptors.inc

2051

This symbol name needs to be mangled for NetBSD.

lib/sanitizer_common/sanitizer_linux.cc

464

Perhaps we could define clock_gettime_symname similar to sigaction_symname.

[cryptoad]

lib/sanitizer_common/sanitizer_common_interceptors.inc

2051

So I don't think I understand what you mean here.
I have never touched a BSD, I am sorry if I am bit dense on this.
Which one has to be mangled and how?
It looks like the interceptor above also calls REAL(clock_gettime), so it would have triggered some NetBSD issue as well?

lib/sanitizer_common/sanitizer_linux.cc

464

Will do.

[alekseyshl]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

716

This name is a bit too generic, can we change it to something like CanUseLibcClockGetTime()?

734

Does it mean that our allocator is going to be slow on BSD? Should we turn release to OS off by default there then?

[cryptoad]

s/CanUseVDSO/CanUseLibcClockGetTime

[cryptoad]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

734

Currently BSD doesn't have Scudo, I am not sure it's a big deal for the *San allocators.

[krytarowski]

lib/sanitizer_common/sanitizer_common_interceptors.inc

2051

Ah right, good catch. We need to fix more calls in this file: __clock_gettime50, __clock_getres50, __clock_settime50, __setitimer50, __getitimer50 and few more: __opendir30, __readdir30, __time50, __localtime_r50, __gmtime_r50, __gmtime50, __ctime50, __ctime_r50, __mktime50, __getpwnam50, __getpwuid50, __getpwnam_r50, __getpwuid_r50, __getpwent50, __glob30, __wait350, __wait450, __readdir_r30, __setlocale50, __scandir30, __sigtimedwait50, __sigemptyset14, __sigfillset14, __sigpending14, __sigprocmask14, __shmctl50, __times13, __stat50, __getutent50, __getutxent50, __getutxid50, __getutxline50.

Please ignore symbol demangling yourself, I will send a patch myself.

[alekseyshl]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

734

Ok then, what's a proper BSD way to get time fast? Is there an alternative to syscall? It's a question to BSD folks, indeed.

[krytarowski]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

734

I have got a simple local Scudo/NetBSD port, however I need to port tests to !GLIBC in order to make sure that it works reliably.

There is no vDSO as of now on NetBSD and no quick alternative.

[krytarowski]

lib/sanitizer_common/sanitizer_common_interceptors.inc

2051

D40765

[cryptoad]

Rebasing.
Use MonotonicNanoTime for the Scudo RSS checks.

[cryptoad]

Moving the NetBSD clock_gettime syscall mangling to
sanitizer_syscall_generic.inc as is the case for others.

[cryptoad]

@krytarowski: I'd be grateful if you could confirm that this would work on NetBSD. Thanks!

[krytarowski]

Looks fine.

[cryptoad]

Adding a MonotonicNanoTime stub for mac.

[cryptoad]

Rebasing one last time.

lib/sanitizer_common/sanitizer_allocator_primary64.h

if (LIKELY(total_user_bytes > region->mapped_user)) {
           region->rand_state = static_cast<u32>(region_beg >> 12);
         // Postpone the first release to OS attempt for ReleaseToOSIntervalMs,
         // preventing just allocated memory from being released sooner than
         // necessary and also preventing extraneous ReleaseMemoryPagesToOS calls
         // for short lived processes.
         // Do it only when the feature is turned on, to avoid a potentially
         // extraneous syscall.
         if (ReleaseToOSIntervalMs() >= 0)
-          region->rtoi.last_release_at_ns = NanoTime();
+          region->rtoi.last_release_at_ns = MonotonicNanoTime();
       }
       // Do the mmap for the user memory.
       const uptr user_map_size =
           RoundUpTo(total_user_bytes - region->mapped_user, kUserMapSize);
       if (UNLIKELY(IsRegionExhausted(region, class_id, user_map_size)))
         return false;
       if (UNLIKELY(!MapWithCallback(region_beg + region->mapped_user,
                                     user_map_size)))
void MaybeReleaseToOS(uptr class_id, bool force) {
     }
 
     if (!force) {
       s32 interval_ms = ReleaseToOSIntervalMs();
       if (interval_ms < 0)
         return;
 
       if (region->rtoi.last_release_at_ns + interval_ms * 1000000ULL >
-          NanoTime()) {
+          MonotonicNanoTime()) {
         return;  // Memory was returned recently.
       }
     }
 
     MemoryMapper memory_mapper(*this, class_id);
 
     ReleaseFreeMemoryToOS<MemoryMapper>(
         GetFreeArray(GetRegionBeginBySizeClass(class_id)), n, chunk_size,
         RoundUpTo(region->allocated_user, page_size) / page_size,
         &memory_mapper);
 
     if (memory_mapper.GetReleasedRangesCount() > 0) {
       region->rtoi.n_freed_at_last_release = region->stats.n_freed;
       region->rtoi.num_releases += memory_mapper.GetReleasedRangesCount();
       region->rtoi.last_released_bytes = memory_mapper.GetReleasedBytes();
     }
-    region->rtoi.last_release_at_ns = NanoTime();
+    region->rtoi.last_release_at_ns = MonotonicNanoTime();
   }
 };

lib/sanitizer_common/sanitizer_common.h

 
 void InitTlsSize();
 uptr GetTlsSize();
 
 // Other
 void SleepForSeconds(int seconds);
 void SleepForMillis(int millis);
 u64 NanoTime();
+u64 MonotonicNanoTime();
 int Atexit(void (*function)(void));
 void SortArray(uptr *array, uptr size);
 void SortArray(u32 *array, uptr size);
 bool TemplateMatch(const char *templ, const char *str);
 
 // Exit
 void NORETURN Abort();
 void NORETURN Die();

lib/sanitizer_common/sanitizer_common_interceptors.inc

INTERCEPTOR(int, clock_gettime, u32 clk_id, void *tp) {
   // its metadata. See
   // https://github.com/google/sanitizers/issues/321.
   int res = REAL(clock_gettime)(clk_id, tp);
   if (!res) {
     COMMON_INTERCEPTOR_WRITE_RANGE(ctx, tp, struct_timespec_sz);
   }
   return res;
 }
+namespace __sanitizer {
+extern "C" {
+int real_clock_gettime(u32 clk_id, void *tp) {
+  return REAL(clock_gettime)(clk_id, tp);
+}
+}  // extern "C"
+}  // namespace __sanitizer
 INTERCEPTOR(int, clock_settime, u32 clk_id, const void *tp) {
   void *ctx;
   COMMON_INTERCEPTOR_ENTER(ctx, clock_settime, clk_id, tp);
   COMMON_INTERCEPTOR_READ_RANGE(ctx, tp, struct_timespec_sz);
   return REAL(clock_settime)(clk_id, tp);
 }
 #define INIT_CLOCK_GETTIME                  \
   COMMON_INTERCEPT_FUNCTION(clock_getres);  \

lib/sanitizer_common/sanitizer_fuchsia.cc

 
 unsigned int internal_sleep(unsigned int seconds) {
   internal_nanosleep(ZX_SEC(seconds));
   return 0;
 }
 
 u64 NanoTime() { return _zx_time_get(ZX_CLOCK_UTC); }
 
+u64 MonotonicNanoTime() { return _zx_time_get(ZX_CLOCK_MONOTONIC); }
+
 uptr internal_getpid() {
   zx_info_handle_basic_t info;
   zx_status_t status =
       _zx_object_get_info(_zx_process_self(), ZX_INFO_HANDLE_BASIC, &info,
                           sizeof(info), NULL, NULL);
   CHECK_EQ(status, ZX_OK);
   uptr pid = static_cast<uptr>(info.koid);
   CHECK_EQ(pid, info.koid);

lib/sanitizer_common/sanitizer_linux.h

 
 void ReadProcMaps(ProcSelfMapsBuff *proc_maps);
 
 // Syscall wrappers.
 uptr internal_getdents(fd_t fd, struct linux_dirent *dirp, unsigned int count);
 uptr internal_sigaltstack(const void* ss, void* oss);
 uptr internal_sigprocmask(int how, __sanitizer_sigset_t *set,
     __sanitizer_sigset_t *oldset);
+uptr internal_clock_gettime(__sanitizer_clockid_t clk_id, void *tp);
 
 // Linux-only syscalls.
 #if SANITIZER_LINUX
 uptr internal_prctl(int option, uptr arg2, uptr arg3, uptr arg4, uptr arg5);
 // Used only by sanitizer_stoptheworld. Signal handlers that are actually used
 // (like the process-wide error reporting SEGV handler) must use
 // internal_sigaction instead.
 int internal_sigaction_norestorer(int signum, const void *act, void *oldact);

lib/sanitizer_common/sanitizer_linux.cc

 #else
   kernel_timeval tv;
 #endif
   internal_memset(&tv, 0, sizeof(tv));
   internal_syscall_ptr(SYSCALL(gettimeofday), &tv, 0);
   return (u64)tv.tv_sec * 1000*1000*1000 + tv.tv_usec * 1000;
 }
 
+uptr internal_clock_gettime(__sanitizer_clockid_t clk_id, void *tp) {
+  return internal_syscall_ptr(SYSCALL(clock_gettime), clk_id, tp);
+}
+
 // Like getenv, but reads env directly from /proc (on Linux) or parses the
 // 'environ' array (on FreeBSD) and does not use libc. This function should be
 // called first inside __asan_init.
 const char *GetEnv(const char *name) {
 #if SANITIZER_FREEBSD || SANITIZER_NETBSD
   if (::environ != 0) {
     uptr NameLen = internal_strlen(name);
     for (char **Env = ::environ; *Env != 0; Env++) {

lib/sanitizer_common/sanitizer_linux_libcdep.cc

 
 void LogMessageOnPrintf(const char *str) {
   if (common_flags()->log_to_syslog && ShouldLogAfterPrintf())
     WriteToSyslog(str);
 }
 
 #endif  // SANITIZER_LINUX
 
+// glibc cannot use clock_gettime from a preinit_array function as the vDSO
+// function pointers haven't been initialized yet. To prevent a crash, we check
+// for the presence of the glibc symbol __vdso_clock_gettime, and verify that it
+// is not null (it can be mangled so we can't use it directly). Bionic's
+// clock_gettime actually falls back to the syscall in the same situation.
+extern "C" SANITIZER_WEAK_ATTRIBUTE void *__vdso_clock_gettime;
+bool CanUseLibcClockGetTime() {
+  return !SANITIZER_FREEBSD && !SANITIZER_NETBSD &&
+      (SANITIZER_ANDROID || (&__vdso_clock_gettime && __vdso_clock_gettime));
+}
+
+// MonotonicNanoTime is a timing function that can leverage the vDSO by calling
+// clock_gettime. real_clock_gettime only exists if clock_gettime is
+// intercepted, so define it weakly and use it if available.
+extern "C" SANITIZER_WEAK_ATTRIBUTE
+int real_clock_gettime(u32 clk_id, void *tp);
+u64 MonotonicNanoTime() {
+  timespec ts;
+  if (CanUseLibcClockGetTime()) {
+    if (&real_clock_gettime)
+      real_clock_gettime(CLOCK_MONOTONIC, &ts);
+    else
+      clock_gettime(CLOCK_MONOTONIC, &ts);
+  } else {
+    internal_clock_gettime(CLOCK_MONOTONIC, &ts);
+  }
+  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
+}
+
 } // namespace __sanitizer
 
 #endif // SANITIZER_FREEBSD || SANITIZER_LINUX

lib/sanitizer_common/sanitizer_mac.cc

 void BlockingMutex::CheckLocked() {
   CHECK_NE(*(OSSpinLock*)&opaque_storage_, 0);
 }
 
 u64 NanoTime() {
   return 0;
 }
 
+u64 MonotonicNanoTime() {
+  return 0;
+}
+
 uptr GetTlsSize() {
   return 0;
 }
 
 void InitTlsSize() {
 }
 
 uptr TlsBaseAddr() {

lib/sanitizer_common/sanitizer_syscall_generic.inc

 # define SYS_lstat SYS___lstat50
 # define SYS_fstat SYS___fstat50
 # define SYS_gettimeofday SYS___gettimeofday50
 # define SYS_wait4 SYS___wait450
 # define SYS_getdents SYS___getdents30
 # define SYS_sigaltstack SYS___sigaltstack14
 # define SYS_sigprocmask SYS___sigprocmask14
 # define SYS_nanosleep SYS___nanosleep50
+# define SYS_clock_gettime SYS___clock_gettime50
 # if SANITIZER_WORDSIZE == 64
 #  define internal_syscall_ptr  __syscall
 # else
 #  define internal_syscall_ptr  syscall
 # endif
 #elif defined(__x86_64__) && (SANITIZER_FREEBSD || SANITIZER_MAC)
 # define internal_syscall __syscall
 # define internal_syscall64 __syscall

lib/sanitizer_common/sanitizer_win.cc

 void SleepForMillis(int millis) {
   Sleep(millis);
 }
 
 u64 NanoTime() {
   return 0;
 }
 
+u64 MonotonicNanoTime() {
+  return 0;
+}
+
 void Abort() {
   internal__exit(3);
 }
 
 #if !SANITIZER_GO
 // Read the file to extract the ImageBase field from the PE header. If ASLR is
 // disabled and this virtual address is available, the loader will typically
 // load the image at this address. Therefore, we call it the preferred base. Any

lib/scudo/scudo_allocator.cpp

void init() {
     if (UNLIKELY(!GetRandom(reinterpret_cast<void *>(&Cookie), sizeof(Cookie),
                             /*blocking=*/false))) {
       Cookie = static_cast<u32>((NanoTime() >> 12) ^
                                 (reinterpret_cast<uptr>(this) >> 4));
     }
 
     CheckRssLimit = HardRssLimitMb || SoftRssLimitMb;
     if (CheckRssLimit)
-      atomic_store_relaxed(&RssLastCheckedAtNS, NanoTime());
+      atomic_store_relaxed(&RssLastCheckedAtNS, MonotonicNanoTime());
   }
 
   // Helper function that checks for a valid Scudo chunk. nullptr isn't.
   bool isValidPointer(const void *UserPtr) {
     initThreadMaybe();
     if (UNLIKELY(!UserPtr))
       return false;
     uptr UserBeg = reinterpret_cast<uptr>(UserPtr);
     if (!IsAligned(UserBeg, MinAlignment))
       return false;
     return getScudoChunk(UserBeg)->isValid();
   }
 
   // Opportunistic RSS limit check. This will update the RSS limit status, if
   // it can, every 100ms, otherwise it will just return the current one.
   bool isRssLimitExceeded() {
     u64 LastCheck = atomic_load_relaxed(&RssLastCheckedAtNS);
-    const u64 CurrentCheck = NanoTime();
+    const u64 CurrentCheck = MonotonicNanoTime();
     if (LIKELY(CurrentCheck < LastCheck + (100ULL * 1000000ULL)))
       return atomic_load_relaxed(&RssLimitExceeded);
     if (!atomic_compare_exchange_weak(&RssLastCheckedAtNS, &LastCheck,
                                       CurrentCheck, memory_order_relaxed))
       return atomic_load_relaxed(&RssLimitExceeded);
     // TODO(kostyak): We currently use sanitizer_common's GetRSS which reads the
     //                RSS from /proc/self/statm by default. We might want to
     //                call getrusage directly, even if it's less accurate.

lib/scudo/scudo_tsd.h

struct ALIGNED(64) ScudoTSD {
   void commitBack();
 
   INLINE bool tryLock() {
     if (Mutex.TryLock()) {
       atomic_store_relaxed(&Precedence, 0);
       return true;
     }
     if (atomic_load_relaxed(&Precedence) == 0)
-      atomic_store_relaxed(&Precedence, NanoTime());
+      atomic_store_relaxed(&Precedence, MonotonicNanoTime());
     return false;
   }
 
   INLINE void lock() {
     Mutex.Lock();
     atomic_store_relaxed(&Precedence, 0);
   }