[sanitizer] Introduce a vDSO aware timing function

[cryptoad]

Summary

See D40657 & D40679 for previous versions of this patch & description.

A couple of things were fixed here to have it not break some bots.
Weak symbols can't be used with SANITIZER_GO so the previous version was
breakin TsanGo. I set up some additional local tests and those pass now.

I changed the workaround for the glibc vDSO issue: __progname is initialized
after the vDSO and is actually public and of known type, unlike
__vdso_clock_gettime. This works better, and with all compilers.

The rest is the same.

[krytarowski]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

723

Can we restrict this symbol to Linux only?

[cryptoad]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

723

I can do that, just to make sure: since it's defined as weak, and the the BSD part will return before that check, shouldn't that work as in on NetBSD?

[krytarowski]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

723

I don't like pollution of symbols.. but I will try to test build it tonight.

[cryptoad]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

723

No worries, I'll hide it in the same define as !SANITIZER_GO.

[cryptoad]

Moving the vDSO check under a SANITIZER_LINUX define.

[alekseyshl]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

747

Now it seems that it would be easier to parse if we separate implementations completely:

#if SANITIZER_LINUX && !SANITIZER_GO

...all the defs...

u64 MonotonicNanoTime() {
  timespec ts;
  if (CanUseVDSO()) {
    if (&real_clock_gettime)
      real_clock_gettime(CLOCK_MONOTONIC, &ts);
    else
      clock_gettime(CLOCK_MONOTONIC, &ts);
  } else {
    internal_clock_gettime(CLOCK_MONOTONIC, &ts);
  }
  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
}

#else // !(SANITIZER_LINUX && !SANITIZER_GO)

u64 MonotonicNanoTime() {
  timespec ts;
  internal_clock_gettime(CLOCK_MONOTONIC, &ts);
  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
}

#endif // SANITIZER_LINUX && !SANITIZER_GO

Also can do

u64 MonotonicNanoTime() {
  timespec ts;
  MonotonicNanoTimeImpl(&ts)
  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
}

and define inlined MonotonicNanoTimeImpls for each case, but that is up to you.

[cryptoad]

As suggested, split the code cleanly between Linux-non-Go and the rest.

[cryptoad]

Correct the last #endif comment of the file while I am at it, since it
didn't include SANITIZER_NETBSD.

[krytarowski]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

747

This split looks good!

[alekseyshl]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

743

No action required at the moment, but this comment sounds a bit silly in a file named sanitizer_LINUX_libcdep.cc.

[cryptoad]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

743

// BSDs & go maybe?

[krytarowski]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

743

I agree that this naming and OS-specific vs generic parts deserves refactoring. We are now adding here Solaris support in sanitizer_linux_libcdep.cc. D40898

[cryptoad]

lib/sanitizer_common/sanitizer_linux_libcdep.cc

743

Well given this new information I guess Non-Linux makes sense until some sort of refactor happens.

lib/sanitizer_common/sanitizer_allocator_primary64.h

if (LIKELY(total_user_bytes > region->mapped_user)) {
           region->rand_state = static_cast<u32>(region_beg >> 12);
         // Postpone the first release to OS attempt for ReleaseToOSIntervalMs,
         // preventing just allocated memory from being released sooner than
         // necessary and also preventing extraneous ReleaseMemoryPagesToOS calls
         // for short lived processes.
         // Do it only when the feature is turned on, to avoid a potentially
         // extraneous syscall.
         if (ReleaseToOSIntervalMs() >= 0)
-          region->rtoi.last_release_at_ns = NanoTime();
+          region->rtoi.last_release_at_ns = MonotonicNanoTime();
       }
       // Do the mmap for the user memory.
       const uptr user_map_size =
           RoundUpTo(total_user_bytes - region->mapped_user, kUserMapSize);
       if (UNLIKELY(IsRegionExhausted(region, class_id, user_map_size)))
         return false;
       if (UNLIKELY(!MapWithCallback(region_beg + region->mapped_user,
                                     user_map_size)))
void MaybeReleaseToOS(uptr class_id, bool force) {
     }
 
     if (!force) {
       s32 interval_ms = ReleaseToOSIntervalMs();
       if (interval_ms < 0)
         return;
 
       if (region->rtoi.last_release_at_ns + interval_ms * 1000000ULL >
-          NanoTime()) {
+          MonotonicNanoTime()) {
         return;  // Memory was returned recently.
       }
     }
 
     MemoryMapper memory_mapper(*this, class_id);
 
     ReleaseFreeMemoryToOS<MemoryMapper>(
         GetFreeArray(GetRegionBeginBySizeClass(class_id)), n, chunk_size,
         RoundUpTo(region->allocated_user, page_size) / page_size,
         &memory_mapper);
 
     if (memory_mapper.GetReleasedRangesCount() > 0) {
       region->rtoi.n_freed_at_last_release = region->stats.n_freed;
       region->rtoi.num_releases += memory_mapper.GetReleasedRangesCount();
       region->rtoi.last_released_bytes = memory_mapper.GetReleasedBytes();
     }
-    region->rtoi.last_release_at_ns = NanoTime();
+    region->rtoi.last_release_at_ns = MonotonicNanoTime();
   }
 };

lib/sanitizer_common/sanitizer_common.h

 
 void InitTlsSize();
 uptr GetTlsSize();
 
 // Other
 void SleepForSeconds(int seconds);
 void SleepForMillis(int millis);
 u64 NanoTime();
+u64 MonotonicNanoTime();
 int Atexit(void (*function)(void));
 void SortArray(uptr *array, uptr size);
 void SortArray(u32 *array, uptr size);
 bool TemplateMatch(const char *templ, const char *str);
 
 // Exit
 void NORETURN Abort();
 void NORETURN Die();

lib/sanitizer_common/sanitizer_common_interceptors.inc

INTERCEPTOR(int, clock_gettime, u32 clk_id, void *tp) {
   // its metadata. See
   // https://github.com/google/sanitizers/issues/321.
   int res = REAL(clock_gettime)(clk_id, tp);
   if (!res) {
     COMMON_INTERCEPTOR_WRITE_RANGE(ctx, tp, struct_timespec_sz);
   }
   return res;
 }
+namespace __sanitizer {
+extern "C" {
+int real_clock_gettime(u32 clk_id, void *tp) {
+  return REAL(clock_gettime)(clk_id, tp);
+}
+}  // extern "C"
+}  // namespace __sanitizer
 INTERCEPTOR(int, clock_settime, u32 clk_id, const void *tp) {
   void *ctx;
   COMMON_INTERCEPTOR_ENTER(ctx, clock_settime, clk_id, tp);
   COMMON_INTERCEPTOR_READ_RANGE(ctx, tp, struct_timespec_sz);
   return REAL(clock_settime)(clk_id, tp);
 }
 #define INIT_CLOCK_GETTIME                  \
   COMMON_INTERCEPT_FUNCTION(clock_getres);  \

lib/sanitizer_common/sanitizer_fuchsia.cc

 
 unsigned int internal_sleep(unsigned int seconds) {
   internal_nanosleep(ZX_SEC(seconds));
   return 0;
 }
 
 u64 NanoTime() { return _zx_time_get(ZX_CLOCK_UTC); }
 
+u64 MonotonicNanoTime() { return _zx_time_get(ZX_CLOCK_MONOTONIC); }
+
 uptr internal_getpid() {
   zx_info_handle_basic_t info;
   zx_status_t status =
       _zx_object_get_info(_zx_process_self(), ZX_INFO_HANDLE_BASIC, &info,
                           sizeof(info), NULL, NULL);
   CHECK_EQ(status, ZX_OK);
   uptr pid = static_cast<uptr>(info.koid);
   CHECK_EQ(pid, info.koid);

lib/sanitizer_common/sanitizer_linux.h

 
 void ReadProcMaps(ProcSelfMapsBuff *proc_maps);
 
 // Syscall wrappers.
 uptr internal_getdents(fd_t fd, struct linux_dirent *dirp, unsigned int count);
 uptr internal_sigaltstack(const void* ss, void* oss);
 uptr internal_sigprocmask(int how, __sanitizer_sigset_t *set,
     __sanitizer_sigset_t *oldset);
+uptr internal_clock_gettime(__sanitizer_clockid_t clk_id, void *tp);
 
 // Linux-only syscalls.
 #if SANITIZER_LINUX
 uptr internal_prctl(int option, uptr arg2, uptr arg3, uptr arg4, uptr arg5);
 // Used only by sanitizer_stoptheworld. Signal handlers that are actually used
 // (like the process-wide error reporting SEGV handler) must use
 // internal_sigaction instead.
 int internal_sigaction_norestorer(int signum, const void *act, void *oldact);

lib/sanitizer_common/sanitizer_linux.cc

 #else
   kernel_timeval tv;
 #endif
   internal_memset(&tv, 0, sizeof(tv));
   internal_syscall_ptr(SYSCALL(gettimeofday), &tv, 0);
   return (u64)tv.tv_sec * 1000*1000*1000 + tv.tv_usec * 1000;
 }
 
+uptr internal_clock_gettime(__sanitizer_clockid_t clk_id, void *tp) {
+  return internal_syscall_ptr(SYSCALL(clock_gettime), clk_id, tp);
+}
+
 // Like getenv, but reads env directly from /proc (on Linux) or parses the
 // 'environ' array (on FreeBSD) and does not use libc. This function should be
 // called first inside __asan_init.
 const char *GetEnv(const char *name) {
 #if SANITIZER_FREEBSD || SANITIZER_NETBSD
   if (::environ != 0) {
     uptr NameLen = internal_strlen(name);
     for (char **Env = ::environ; *Env != 0; Env++) {

lib/sanitizer_common/sanitizer_linux_libcdep.cc

 
 void LogMessageOnPrintf(const char *str) {
   if (common_flags()->log_to_syslog && ShouldLogAfterPrintf())
     WriteToSyslog(str);
 }
 
 #endif  // SANITIZER_LINUX
 
+#if SANITIZER_LINUX && !SANITIZER_GO
+// glibc crashes when using clock_gettime from a preinit_array function as the
+// vDSO function pointers haven't been initialized yet. __progname is
+// initialized after the vDSO function pointers, so if it exists, is not null
+// and is not empty, we can use clock_gettime.
+extern "C" SANITIZER_WEAK_ATTRIBUTE char *__progname;
+INLINE bool CanUseVDSO() {
+  // Bionic is safe, it checks for the vDSO function pointers to be initialized.
+  if (SANITIZER_ANDROID)
+    return true;
+  if (&__progname && __progname && *__progname)
+    return true;
+  return false;
+}
+
+// MonotonicNanoTime is a timing function that can leverage the vDSO by calling
+// clock_gettime. real_clock_gettime only exists if clock_gettime is
+// intercepted, so define it weakly and use it if available.
+extern "C" SANITIZER_WEAK_ATTRIBUTE
+int real_clock_gettime(u32 clk_id, void *tp);
+u64 MonotonicNanoTime() {
+  timespec ts;
+  if (CanUseVDSO()) {
+    if (&real_clock_gettime)
+      real_clock_gettime(CLOCK_MONOTONIC, &ts);
+    else
+      clock_gettime(CLOCK_MONOTONIC, &ts);
+  } else {
+    internal_clock_gettime(CLOCK_MONOTONIC, &ts);
+  }
+  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
+}
+#else
+// Non-Linux & Go always use the syscall.
+u64 MonotonicNanoTime() {
+  timespec ts;
+  internal_clock_gettime(CLOCK_MONOTONIC, &ts);
+  return (u64)ts.tv_sec * (1000ULL * 1000 * 1000) + ts.tv_nsec;
+}
+#endif  // SANITIZER_LINUX && !SANITIZER_GO
+
 } // namespace __sanitizer
 
-#endif // SANITIZER_FREEBSD || SANITIZER_LINUX
+#endif  // SANITIZER_FREEBSD || SANITIZER_LINUX || SANITIZER_NETBSD

lib/sanitizer_common/sanitizer_mac.cc

 void BlockingMutex::CheckLocked() {
   CHECK_NE(*(OSSpinLock*)&opaque_storage_, 0);
 }
 
 u64 NanoTime() {
   return 0;
 }
 
+u64 MonotonicNanoTime() {
+  return 0;
+}
+
 uptr GetTlsSize() {
   return 0;
 }
 
 void InitTlsSize() {
 }
 
 uptr TlsBaseAddr() {

lib/sanitizer_common/sanitizer_syscall_generic.inc

 # define SYS_lstat SYS___lstat50
 # define SYS_fstat SYS___fstat50
 # define SYS_gettimeofday SYS___gettimeofday50
 # define SYS_wait4 SYS___wait450
 # define SYS_getdents SYS___getdents30
 # define SYS_sigaltstack SYS___sigaltstack14
 # define SYS_sigprocmask SYS___sigprocmask14
 # define SYS_nanosleep SYS___nanosleep50
+# define SYS_clock_gettime SYS___clock_gettime50
 # if SANITIZER_WORDSIZE == 64
 #  define internal_syscall_ptr  __syscall
 # else
 #  define internal_syscall_ptr  syscall
 # endif
 #elif defined(__x86_64__) && (SANITIZER_FREEBSD || SANITIZER_MAC)
 # define internal_syscall __syscall
 # define internal_syscall64 __syscall

lib/sanitizer_common/sanitizer_win.cc

 void SleepForMillis(int millis) {
   Sleep(millis);
 }
 
 u64 NanoTime() {
   return 0;
 }
 
+u64 MonotonicNanoTime() {
+  return 0;
+}
+
 void Abort() {
   internal__exit(3);
 }
 
 #if !SANITIZER_GO
 // Read the file to extract the ImageBase field from the PE header. If ASLR is
 // disabled and this virtual address is available, the loader will typically
 // load the image at this address. Therefore, we call it the preferred base. Any

lib/scudo/scudo_allocator.cpp

void init() {
     if (UNLIKELY(!GetRandom(reinterpret_cast<void *>(&Cookie), sizeof(Cookie),
                             /*blocking=*/false))) {
       Cookie = static_cast<u32>((NanoTime() >> 12) ^
                                 (reinterpret_cast<uptr>(this) >> 4));
     }
 
     CheckRssLimit = HardRssLimitMb || SoftRssLimitMb;
     if (CheckRssLimit)
-      atomic_store_relaxed(&RssLastCheckedAtNS, NanoTime());
+      atomic_store_relaxed(&RssLastCheckedAtNS, MonotonicNanoTime());
   }
 
   // Helper function that checks for a valid Scudo chunk. nullptr isn't.
   bool isValidPointer(const void *UserPtr) {
     initThreadMaybe();
     if (UNLIKELY(!UserPtr))
       return false;
     uptr UserBeg = reinterpret_cast<uptr>(UserPtr);
     if (!IsAligned(UserBeg, MinAlignment))
       return false;
     return getScudoChunk(UserBeg)->isValid();
   }
 
   // Opportunistic RSS limit check. This will update the RSS limit status, if
   // it can, every 100ms, otherwise it will just return the current one.
   bool isRssLimitExceeded() {
     u64 LastCheck = atomic_load_relaxed(&RssLastCheckedAtNS);
-    const u64 CurrentCheck = NanoTime();
+    const u64 CurrentCheck = MonotonicNanoTime();
     if (LIKELY(CurrentCheck < LastCheck + (100ULL * 1000000ULL)))
       return atomic_load_relaxed(&RssLimitExceeded);
     if (!atomic_compare_exchange_weak(&RssLastCheckedAtNS, &LastCheck,
                                       CurrentCheck, memory_order_relaxed))
       return atomic_load_relaxed(&RssLimitExceeded);
     // TODO(kostyak): We currently use sanitizer_common's GetRSS which reads the
     //                RSS from /proc/self/statm by default. We might want to
     //                call getrusage directly, even if it's less accurate.

lib/scudo/scudo_tsd.h

struct ALIGNED(64) ScudoTSD {
   void commitBack();
 
   INLINE bool tryLock() {
     if (Mutex.TryLock()) {
       atomic_store_relaxed(&Precedence, 0);
       return true;
     }
     if (atomic_load_relaxed(&Precedence) == 0)
-      atomic_store_relaxed(&Precedence, NanoTime());
+      atomic_store_relaxed(&Precedence, MonotonicNanoTime());
     return false;
   }
 
   INLINE void lock() {
     Mutex.Lock();
     atomic_store_relaxed(&Precedence, 0);
   }