[sanitizer] Use private futex operations for BlockingMutex

[cryptoad]

Summary

Use FUTEX_PRIVATE_FLAG in conjunction with the wait & wake operations
employed by BlockingMutex. As far as I can tell, the mutexes are
process-private, and there is an actual performance benefit at employing the
private operations. There should be no downside to switching to it.

[alekseyshl]

Have you noticed any difference in performance? I tried the same back in March I have not noticed any perf gain on the couple of benchmarks. I am merely curious, I think this is a right change to make anyways.

[cryptoad]

In D50910#1205003, @alekseyshl wrote:

Have you noticed any difference in performance? I tried the same back in March I have not noticed any perf gain on the couple of benchmarks. I am merely curious, I think this is a right change to make anyways.

The performance gain is noticeable with the shared TSD model for Scudo, in some of the most contentious scenario.
With rpc2-benchmark (async mode, no client/server thread limitation, 72-core machines, TSD pool size of 32) the average QPS over 10 runs jumps from 347K to 375K.
In other cases, the gains are more marginal.

lib/sanitizer_common/sanitizer_linux.cc

 struct kernel_timeval {
   long tv_sec;
   long tv_usec;
 };
 
 // <linux/futex.h> is broken on some linux distributions.
 const int FUTEX_WAIT = 0;
 const int FUTEX_WAKE = 1;
+const int FUTEX_PRIVATE_FLAG = 128;
+const int FUTEX_WAIT_PRIVATE = FUTEX_WAIT | FUTEX_PRIVATE_FLAG;
+const int FUTEX_WAKE_PRIVATE = FUTEX_WAKE | FUTEX_PRIVATE_FLAG;
 #endif  // SANITIZER_LINUX
 
 // Are we using 32-bit or 64-bit Linux syscalls?
 // x32 (which defines __x86_64__) has SANITIZER_WORDSIZE == 32
 // but it still needs to use 64-bit syscalls.
 #if SANITIZER_LINUX && (defined(__x86_64__) || defined(__powerpc64__) ||       \
                         SANITIZER_WORDSIZE == 64)
 # define SANITIZER_LINUX_USES_64BIT_SYSCALLS 1
void BlockingMutex::Lock() {
   if (atomic_exchange(m, MtxLocked, memory_order_acquire) == MtxUnlocked)
     return;
   while (atomic_exchange(m, MtxSleeping, memory_order_acquire) != MtxUnlocked) {
 #if SANITIZER_FREEBSD
     _umtx_op(m, UMTX_OP_WAIT_UINT, MtxSleeping, 0, 0);
 #elif SANITIZER_NETBSD
     sched_yield(); /* No userspace futex-like synchronization */
 #else
-    internal_syscall(SYSCALL(futex), (uptr)m, FUTEX_WAIT, MtxSleeping, 0, 0, 0);
+    internal_syscall(SYSCALL(futex), (uptr)m, FUTEX_WAIT_PRIVATE, MtxSleeping,
+                     0, 0, 0);
 #endif
   }
 }
 
 void BlockingMutex::Unlock() {
   atomic_uint32_t *m = reinterpret_cast<atomic_uint32_t *>(&opaque_storage_);
   u32 v = atomic_exchange(m, MtxUnlocked, memory_order_release);
   CHECK_NE(v, MtxUnlocked);
   if (v == MtxSleeping) {
 #if SANITIZER_FREEBSD
     _umtx_op(m, UMTX_OP_WAKE, 1, 0, 0);
 #elif SANITIZER_NETBSD
                    /* No userspace futex-like synchronization */
 #else
-    internal_syscall(SYSCALL(futex), (uptr)m, FUTEX_WAKE, 1, 0, 0, 0);
+    internal_syscall(SYSCALL(futex), (uptr)m, FUTEX_WAKE_PRIVATE, 1, 0, 0, 0);
 #endif
   }
 }
 
 void BlockingMutex::CheckLocked() {
   atomic_uint32_t *m = reinterpret_cast<atomic_uint32_t *>(&opaque_storage_);
   CHECK_NE(MtxUnlocked, atomic_load(m, memory_order_relaxed));
 }