[sanitizer] Introduce a vDSO aware time function, and use it in the allocator

[cryptoad]

Summary

NanoTime is a time sink when checking whether or not to release memory to
the OS. While reducing the amount of calls to said function is in the works,
another solution that was found to be beneficial was to use a timing function
that can leverage the vDSO.

We hit a couple of snags along the way, like the fact that the glibc crashes
when clock_gettime is called from a preinit_array, or the fact that
__vdso_clock_gettime is mangled (for security purposes) and can't be used
directly, and also that clock_gettime can be intercepted.

The proposed solution takes care of all this as far as I can tell, and
significantly improve performances and some Scudo load tests with memory
reclaiming enabled.

@krytarowski: could you please chime in on the NetBSD aspect of it?

[alekseyshl]

lib/sanitizer_common/sanitizer_linux.cc

466

It feels fragile to depend on the implementation, but I don't really see what else can we do. Have a test to check that it works from preinit_array on Android?

470

What about __kernel_clock_gettime?

[krytarowski]

NetBSD does not ship with vDSO and the code should/could be disabled for it.

[mcgrathr]

Fuchsia bit LGTM. Linux bit highly suspect.

lib/sanitizer_common/sanitizer_linux.cc

468

Don't rely on these libc internals. Just look up the real __vdso_clock_gettime symbol in the vDSO itself. On some machines glibc defines a __vdso_clock_gettime symbol also, which is a mangled pointer, not a function as declared here. But that's purely a libc implementation detail (it's a GLIBC_PRIVATE symbol) and you shouldn't pay attention to that at all. On x86, there is no such symbol in libc and so __vdso_clock_gettime will reach the actual vDSO function. But since the name is sometimes overloaded, what you probably want to do is find the vDSO with dlopen or suchlike and then use dlsym to look up __vdso_clock_gettime directly in the vDSO.

[krytarowski]

Additionally NetBSD requires symbol mangling: clock_gettime -> __clock_gettime50.

[cryptoad]

lib/sanitizer_common/sanitizer_linux.cc

466

I have test/scudo/preinit.c that I could enable again on Android (it crashes on N for another reason due to a Bionic bug).
Otherwise ASan has a few tests that leverage the preinit_array, I am going to check if they actually end up calling the function.

470

So it turns out that __kernel_clock_gettime@version is the vDSO symbol name.
The glibc looks it up, and stores it in __vdso_clock_gettime (without appendix, glibc symbol, not vDSO): https://github.com/lattera/glibc/blob/a2f34833b1042d5d8eeb263b4cf4caaea138c4ad/sysdeps/unix/sysv/linux/s390/init-first.c#L42 (example for s390).
What we really want is to make sure that the glibc symbol isn't null.

[cryptoad]

@krytarowski: so directly call the syscall on NetBSD or is there a possible performance gain calling the libc function as opposed to the syscall?

lib/sanitizer_common/sanitizer_linux.cc

468

So __vdso_clock_gettime here is not used in any capacity but very that if it exists, it's not null.
The reason being that calling clock_gettime from a preinit_array crashes on a NULL deref (or bad demangled pointer deref).
In an ideal world I don't think that should be the case, so I am trying to work around this.

While I totally agree that it's hackish, it felt more portable than going through our own vDSO symbol resolution (with multiple architectures, multiple symbold names, etc), which it's already done at some point by the libc.
Regarding alternatives:

• dlsym uses calloc and I am not really keen on adding a special calloc case from dlsym as ASan does;
• doing our own ELF parsing could be the alternative and getting the base of the vDSO via getauxval, but the ratio of additional code & complexity versus value seems low.

[krytarowski]

In D40657#940974, @cryptoad wrote:

@krytarowski: so directly call the syscall on NetBSD or is there a possible performance gain calling the libc function as opposed to the syscall?

Just call the syscall for NetBSD. "clock_gettime" under mangled name "__clock_gettime50".

[cryptoad]

Updating BSD support.

[cryptoad]

I am going to redo this because it's actually libc specific, so for Linux it should be in linuc_libcdep, but then a flurry of other issues arise.

lib/sanitizer_common/sanitizer_allocator_primary64.h

if (new_space_end > region->mapped_user) {
           region->rand_state = static_cast<u32>(region_beg >> 12);
         // Postpone the first release to OS attempt for ReleaseToOSIntervalMs,
         // preventing just allocated memory from being released sooner than
         // necessary and also preventing extraneous ReleaseMemoryPagesToOS calls
         // for short lived processes.
         // Do it only when the feature is turned on, to avoid a potentially
         // extraneous syscall.
         if (ReleaseToOSIntervalMs() >= 0)
-          region->rtoi.last_release_at_ns = NanoTime();
+          region->rtoi.last_release_at_ns = MonotonicNanoTime();
       }
       // Do the mmap for the user memory.
       uptr map_size = kUserMapSize;
       while (new_space_end > region->mapped_user + map_size)
         map_size += kUserMapSize;
       CHECK_GE(region->mapped_user + map_size, new_space_end);
       if (UNLIKELY(!MapWithCallback(region_beg + region->mapped_user,
                                     map_size)))
void MaybeReleaseToOS(uptr class_id, bool force) {
     }
 
     if (!force) {
       s32 interval_ms = ReleaseToOSIntervalMs();
       if (interval_ms < 0)
         return;
 
       if (region->rtoi.last_release_at_ns + interval_ms * 1000000ULL >
-          NanoTime()) {
+          MonotonicNanoTime()) {
         return;  // Memory was returned recently.
       }
     }
 
     MemoryMapper memory_mapper(*this, class_id);
 
     ReleaseFreeMemoryToOS<MemoryMapper>(
         GetFreeArray(GetRegionBeginBySizeClass(class_id)), n, chunk_size,
         RoundUpTo(region->allocated_user, page_size) / page_size,
         &memory_mapper);
 
     if (memory_mapper.GetReleasedRangesCount() > 0) {
       region->rtoi.n_freed_at_last_release = region->stats.n_freed;
       region->rtoi.num_releases += memory_mapper.GetReleasedRangesCount();
       region->rtoi.last_released_bytes = memory_mapper.GetReleasedBytes();
     }
-    region->rtoi.last_release_at_ns = NanoTime();
+    region->rtoi.last_release_at_ns = MonotonicNanoTime();
   }
 };

lib/sanitizer_common/sanitizer_common.h

 
 void InitTlsSize();
 uptr GetTlsSize();
 
 // Other
 void SleepForSeconds(int seconds);
 void SleepForMillis(int millis);
 u64 NanoTime();
+u64 MonotonicNanoTime();
 int Atexit(void (*function)(void));
 void SortArray(uptr *array, uptr size);
 void SortArray(u32 *array, uptr size);
 bool TemplateMatch(const char *templ, const char *str);
 
 // Exit
 void NORETURN Abort();
 void NORETURN Die();

lib/sanitizer_common/sanitizer_common_interceptors.inc

INTERCEPTOR(int, clock_gettime, u32 clk_id, void *tp) {
   // its metadata. See
   // https://github.com/google/sanitizers/issues/321.
   int res = REAL(clock_gettime)(clk_id, tp);
   if (!res) {
     COMMON_INTERCEPTOR_WRITE_RANGE(ctx, tp, struct_timespec_sz);
   }
   return res;
 }
+namespace __sanitizer {
+extern "C" {
+int real_clock_gettime(u32 clk_id, void *tp) {
+  return REAL(clock_gettime)(clk_id, tp);
+}
+}  // extern "C"
+}  // namespace __sanitizer
 INTERCEPTOR(int, clock_settime, u32 clk_id, const void *tp) {
   void *ctx;
   COMMON_INTERCEPTOR_ENTER(ctx, clock_settime, clk_id, tp);
   COMMON_INTERCEPTOR_READ_RANGE(ctx, tp, struct_timespec_sz);
   return REAL(clock_settime)(clk_id, tp);
 }
 #define INIT_CLOCK_GETTIME                  \
   COMMON_INTERCEPT_FUNCTION(clock_getres);  \

lib/sanitizer_common/sanitizer_fuchsia.cc

 
 unsigned int internal_sleep(unsigned int seconds) {
   internal_nanosleep(ZX_SEC(seconds));
   return 0;
 }
 
 u64 NanoTime() { return _zx_time_get(ZX_CLOCK_UTC); }
 
+u64 MonotonicNanoTime() { return _zx_time_get(ZX_CLOCK_MONOTONIC); }
+
 uptr internal_getpid() {
   zx_info_handle_basic_t info;
   zx_status_t status =
       _zx_object_get_info(_zx_process_self(), ZX_INFO_HANDLE_BASIC, &info,
                           sizeof(info), NULL, NULL);
   CHECK_EQ(status, ZX_OK);
   uptr pid = static_cast<uptr>(info.koid);
   CHECK_EQ(pid, info.koid);

lib/sanitizer_common/sanitizer_linux.cc

 #else
   kernel_timeval tv;
 #endif
   internal_memset(&tv, 0, sizeof(tv));
   internal_syscall_ptr(SYSCALL(gettimeofday), &tv, 0);
   return (u64)tv.tv_sec * 1000*1000*1000 + tv.tv_usec * 1000;
 }
 
+// glibc cannot use clock_gettime from a preinit_array function as the vDSO
+// function pointers haven't been initialized yet. To prevent a crash, we check
+// for the presence of the glibc symbol __vdso_clock_gettime, and verify that it
+// is not null (it can be mangled so we can't use it directly). Bionic's
+// clock_gettime actually falls back to the syscall in the same situation.
+extern "C" SANITIZER_WEAK_ATTRIBUTE
+int __vdso_clock_gettime(clockid_t clock, struct timespec* ts);
+bool CanUseVDSO() {
+  return !SANITIZER_FREEBSD && !SANITIZER_NETBSD &&
+      (SANITIZER_ANDROID || (&__vdso_clock_gettime && __vdso_clock_gettime));
+}
+
+// MonotonicNanoTime is a timing function that can leverage the vDSO by calling
+// clock_gettime. real_clock_gettime only exists if clock_gettime is
+// intercepted, so define it weakly and use it if available.
+extern "C" SANITIZER_WEAK_ATTRIBUTE
+int real_clock_gettime(u32 clk_id, void *tp);
+u64 MonotonicNanoTime() {
+  timespec ts;
+  if (CanUseVDSO()) {
+    if (&real_clock_gettime)
+      real_clock_gettime(CLOCK_MONOTONIC, &ts);
+    else
+      clock_gettime(CLOCK_MONOTONIC, &ts);
+  } else {
+#if SANITIZER_NETBSD
+    internal_syscall_ptr(SYSCALL(__clock_gettime50), CLOCK_MONOTONIC, &ts);
+#else
+    internal_syscall_ptr(SYSCALL(clock_gettime), CLOCK_MONOTONIC, &ts);
+#endif
+  }
+  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
+}
+
 // Like getenv, but reads env directly from /proc (on Linux) or parses the
 // 'environ' array (on FreeBSD) and does not use libc. This function should be
 // called first inside __asan_init.
 const char *GetEnv(const char *name) {
 #if SANITIZER_FREEBSD || SANITIZER_NETBSD
   if (::environ != 0) {
     uptr NameLen = internal_strlen(name);
     for (char **Env = ::environ; *Env != 0; Env++) {

lib/sanitizer_common/sanitizer_win.cc

 void SleepForMillis(int millis) {
   Sleep(millis);
 }
 
 u64 NanoTime() {
   return 0;
 }
 
+u64 MonotonicNanoTime() {
+  return 0;
+}
+
 void Abort() {
   internal__exit(3);
 }
 
 #if !SANITIZER_GO
 // Read the file to extract the ImageBase field from the PE header. If ASLR is
 // disabled and this virtual address is available, the loader will typically
 // load the image at this address. Therefore, we call it the preferred base. Any

lib/scudo/scudo_tsd.h

struct ALIGNED(64) ScudoTSD {
   void commitBack();
 
   INLINE bool tryLock() {
     if (Mutex.TryLock()) {
       atomic_store_relaxed(&Precedence, 0);
       return true;
     }
     if (atomic_load_relaxed(&Precedence) == 0)
-      atomic_store_relaxed(&Precedence, NanoTime());
+      atomic_store_relaxed(&Precedence, MonotonicNanoTime());
     return false;
   }
 
   INLINE void lock() {
     Mutex.Lock();
     atomic_store_relaxed(&Precedence, 0);
   }