[Sanitizers] Cleanup handling of stat64/statfs64

[wrotki]

Summary

There are differences in handling of stat64/statfs64 calls by sanitizers between Linux and macOS. Versions of macOS starting with 10.6 drop the stat64/statfs64 APIs completely, relying on the linker to redirect stat/statfs to the appropriate 64 bit versions. Emitting variables needed by sanitizers is thus controlled by convoluted sets of conditions, involving Linux, IOS, macOS and Android, sprinkled around files.

This change adresses it, allowing to specify presence/absence of stat64/statfs64 for each platform, in a single location. Also, it adresses the Android case which handles stat64, but not statfs64.

Adding Vitaly as a reviewer since he seems to be actively working on sanitizers, perhaps can comment on the Android bit

[yln]

LGTM, thanks!

@vitalybuka @eugenis: any concerns?

[dyung]

This seems to be causing 7 test failures on our internal linux buildbot:

• MemorySanitizer-X86_64 :: Linux/glob_altdirfunc.cpp
• MemorySanitizer-X86_64 :: stat64.cpp
• MemorySanitizer-lld-X86_64 :: Linux/glob_altdirfunc.cpp
• MemorySanitizer-lld-X86_64 :: stat64.cpp
• SanitizerCommon-asan-x86_64-Linux :: Posix/lstat64.cpp
• SanitizerCommon-msan-x86_64-Linux :: Posix/lstat64.cpp
• SanitizerCommon-tsan-x86_64-Linux :: Posix/lstat64.cpp

The failures are all similar and look like this:

FAIL: MemorySanitizer-X86_64 :: Linux/glob_altdirfunc.cpp (2151 of 68143)
******************** TEST 'MemorySanitizer-X86_64 :: Linux/glob_altdirfunc.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';      /home/siadmin/jenkins/w/opensource/opensource_build/build/./bin/clang  --driver-mode=g++ -fsanitize=memory -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls  -m64  -gline-tables-only -O0 /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp -o /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp &&  /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux 2>&1 | FileCheck /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp
: 'RUN: at line 2';      /home/siadmin/jenkins/w/opensource/opensource_build/build/./bin/clang  --driver-mode=g++ -fsanitize=memory -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls  -m64  -gline-tables-only -O0 -D_FILE_OFFSET_BITS=64 /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp -o /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp &&  /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux 2>&1 | FileCheck /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp
: 'RUN: at line 3';      /home/siadmin/jenkins/w/opensource/opensource_build/build/./bin/clang  --driver-mode=g++ -fsanitize=memory -mno-omit-leaf-frame-pointer -fno-omit-frame-pointer -fno-optimize-sibling-calls  -m64  -gline-tables-only -O3 /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp -o /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp &&  /home/siadmin/jenkins/w/opensource/opensource_build/build/projects/compiler-rt/test/msan/X86_64/Linux/Output/glob_altdirfunc.cpp.tmp /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux 2>&1 | FileCheck /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp
--
Exit Code: 1

Command Output (stderr):
--
/home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp:76:12: error: CHECK: expected string not found in input
 // CHECK: PASS
           ^
<stdin>:1:1: note: scanning from here
MemorySanitizer:DEADLYSIGNAL
^
<stdin>:1:4: note: possible intended match here
MemorySanitizer:DEADLYSIGNAL
   ^

Input file: <stdin>
Check file: /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/msan/Linux/glob_altdirfunc.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: MemorySanitizer:DEADLYSIGNAL 
check:76'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:76'1        ?                          possible intended match
            2: ==24041==ERROR: MemorySanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 bp 0x7ffc037a9bc0 sp 0x7ffc037a9b98 T24041) 
check:76'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            3: ==24041==Hint: pc points to the zero page. 
check:76'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            4: ==24041==The signal is caused by a READ memory access. 
check:76'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            5: ==24041==Hint: address points to the zero page. 
check:76'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6:  #0 0x0 (<unknown module>) 
check:76'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~
            .
            .
            .
>>>>>>

--

********************

Can you take a look and revert if you need time to investigate?

[abrachet]

Likewise this is causing failures for us see https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/clang-linux-x64/b8811805236073129009/overview

[wrotki]

Okay, will revert and investigate - thanks for notifying.

[hctim]

(jumping on the bandwagon, also failing on the sanitizer-powerpc bot: https://lab.llvm.org/buildbot/#/builders/105/builds/26531. Maybe this is easier to diagnose than internal bots.)

compiler-rt/lib/sanitizer_common/sanitizer_platform_interceptors.h

 #define SANITIZER_INTERCEPT_SIGPROCMASK SI_POSIX
 #define SANITIZER_INTERCEPT_PTHREAD_SIGMASK SI_POSIX
 #define SANITIZER_INTERCEPT_BACKTRACE \
   (SI_FREEBSD || SI_NETBSD || SI_GLIBC || SI_SOLARIS)
 #define SANITIZER_INTERCEPT_GETMNTENT SI_LINUX
 #define SANITIZER_INTERCEPT_GETMNTENT_R SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_STATFS \
   (SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID || SI_SOLARIS)
-#define SANITIZER_INTERCEPT_STATFS64 \
-  (((SI_MAC && !TARGET_CPU_ARM64) && !SI_IOS) || SI_LINUX_NOT_ANDROID)
+#define SANITIZER_INTERCEPT_STATFS64 SANITIZER_HAS_STATFS64
 #define SANITIZER_INTERCEPT_STATVFS \
   (SI_FREEBSD || SI_NETBSD || SI_LINUX_NOT_ANDROID)
 #define SANITIZER_INTERCEPT_STATVFS64 SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_INITGROUPS SI_POSIX
 #define SANITIZER_INTERCEPT_ETHER_NTOA_ATON SI_POSIX
 #define SANITIZER_INTERCEPT_ETHER_HOST \
   (SI_FREEBSD || SI_MAC || SI_LINUX_NOT_ANDROID)
 #define SANITIZER_INTERCEPT_ETHER_R (SI_FREEBSD || SI_LINUX_NOT_ANDROID)

 #define SANITIZER_INTERCEPT_RECV_RECVFROM SI_POSIX
 #define SANITIZER_INTERCEPT_SEND_SENDTO SI_POSIX
 #define SANITIZER_INTERCEPT_EVENTFD_READ_WRITE SI_LINUX
 
 #define SI_STAT_LINUX (SI_LINUX && __GLIBC_PREREQ(2, 33))
 #define SANITIZER_INTERCEPT_STAT                                        \
   (SI_FREEBSD || SI_MAC || SI_ANDROID || SI_NETBSD || SI_SOLARIS ||     \
    SI_STAT_LINUX)
-#define SANITIZER_INTERCEPT_STAT64 SI_STAT_LINUX
+#define SANITIZER_INTERCEPT_STAT64 SI_STAT_LINUX || SANITIZER_HAS_STAT64
 #define SANITIZER_INTERCEPT_LSTAT (SI_NETBSD || SI_FREEBSD || SI_STAT_LINUX)
 #define SANITIZER_INTERCEPT___XSTAT \
   ((!SANITIZER_INTERCEPT_STAT && SI_POSIX) || SI_STAT_LINUX)
 #define SANITIZER_INTERCEPT___XSTAT64 SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT___LXSTAT SANITIZER_INTERCEPT___XSTAT
 #define SANITIZER_INTERCEPT___LXSTAT64 SI_LINUX_NOT_ANDROID
 
 #define SANITIZER_INTERCEPT_UTMP \

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_linux.cpp

 
 #  if !SANITIZER_ANDROID
 #    include <sys/statfs.h>
 #    include <linux/perf_event.h>
 #  endif
 
 using namespace __sanitizer;
 
-namespace __sanitizer {
-#if !SANITIZER_ANDROID
-  unsigned struct_statfs64_sz = sizeof(struct statfs64);
-#endif
-}  // namespace __sanitizer
-
 #  if !defined(__powerpc64__) && !defined(__x86_64__) &&                   \
       !defined(__aarch64__) && !defined(__mips__) && !defined(__s390__) && \
       !defined(__sparc__) && !defined(__riscv) && !defined(__hexagon__)
 COMPILER_CHECK(struct___old_kernel_stat_sz == sizeof(struct __old_kernel_stat));
 #endif
 
 COMPILER_CHECK(struct_kernel_stat_sz == sizeof(struct stat));
 

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.h

 #ifndef SANITIZER_PLATFORM_LIMITS_POSIX_H
 #define SANITIZER_PLATFORM_LIMITS_POSIX_H
 
 #if SANITIZER_LINUX || SANITIZER_APPLE
 
 #include "sanitizer_internal_defs.h"
 #include "sanitizer_platform.h"
 
+#if SANITIZER_APPLE
+#include <sys/cdefs.h>
+#if !__DARWIN_ONLY_64_BIT_INO_T
+#define SANITIZER_HAS_STAT64 1
+#define SANITIZER_HAS_STATFS64 1
+#else
+#define SANITIZER_HAS_STAT64 0
+#define SANITIZER_HAS_STATFS64 0
+#endif
+#else
+// Must be SANITIZER_LINUX then
+#define SANITIZER_HAS_STAT64 1
+#if SANITIZER_ANDROID
+#define SANITIZER_HAS_STATFS64 0 // Intercepting statfs64 seems to be broken on Android
+#else
+#define SANITIZER_HAS_STATFS64 1
+#endif
+#endif
+
 #if defined(__sparc__)
 // FIXME: This can't be included from tsan which does not support sparc yet.
 #include "sanitizer_glibc_version.h"
 #endif
 
 # define GET_LINK_MAP_BY_DLOPEN_HANDLE(handle) ((link_map*)(handle))
 
 namespace __sanitizer {
 extern unsigned struct_utsname_sz;
 extern unsigned struct_stat_sz;
-#if !SANITIZER_IOS
+#if SANITIZER_HAS_STAT64
 extern unsigned struct_stat64_sz;
 #endif
 extern unsigned struct_rusage_sz;
 extern unsigned siginfo_t_sz;
 extern unsigned struct_itimerval_sz;
 extern unsigned pthread_t_sz;
 extern unsigned pthread_mutex_t_sz;
 extern unsigned pthread_cond_t_sz;
 extern unsigned pid_t_sz;
 extern unsigned timeval_sz;
 extern unsigned uid_t_sz;
 extern unsigned gid_t_sz;
 extern unsigned mbstate_t_sz;
 extern unsigned struct_timezone_sz;
 extern unsigned struct_tms_sz;
 extern unsigned struct_itimerspec_sz;
 extern unsigned struct_sigevent_sz;
 extern unsigned struct_stack_t_sz;
 extern unsigned struct_sched_param_sz;
+#if SANITIZER_HAS_STATFS64
 extern unsigned struct_statfs64_sz;
+#endif
 extern unsigned struct_regex_sz;
 extern unsigned struct_regmatch_sz;
 
 #if !SANITIZER_ANDROID
 extern unsigned struct_fstab_sz;
 extern unsigned struct_statfs_sz;
 extern unsigned struct_sockaddr_sz;
 unsigned ucontext_t_sz(void *uctx);

compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp

 
 #if SANITIZER_INTERCEPT_CRYPT_R
 #include <crypt.h>
 #endif
 
 namespace __sanitizer {
   unsigned struct_utsname_sz = sizeof(struct utsname);
   unsigned struct_stat_sz = sizeof(struct stat);
-#if !SANITIZER_IOS && !(SANITIZER_APPLE && TARGET_CPU_ARM64)
+#if SANITIZER_HAS_STAT64
   unsigned struct_stat64_sz = sizeof(struct stat64);
-#endif // !SANITIZER_IOS && !(SANITIZER_APPLE && TARGET_CPU_ARM64)
+#endif // SANITIZER_HAS_STAT64
   unsigned struct_rusage_sz = sizeof(struct rusage);
   unsigned struct_tm_sz = sizeof(struct tm);
   unsigned struct_passwd_sz = sizeof(struct passwd);
   unsigned struct_group_sz = sizeof(struct group);
   unsigned siginfo_t_sz = sizeof(siginfo_t);
   unsigned struct_sigaction_sz = sizeof(struct sigaction);
   unsigned struct_stack_t_sz = sizeof(stack_t);
   unsigned struct_itimerval_sz = sizeof(struct itimerval);
   unsigned pthread_t_sz = sizeof(pthread_t);
   unsigned pthread_mutex_t_sz = sizeof(pthread_mutex_t);
   unsigned pthread_cond_t_sz = sizeof(pthread_cond_t);
   unsigned pid_t_sz = sizeof(pid_t);
   unsigned timeval_sz = sizeof(timeval);
   unsigned uid_t_sz = sizeof(uid_t);
   unsigned gid_t_sz = sizeof(gid_t);
   unsigned mbstate_t_sz = sizeof(mbstate_t);
   unsigned sigset_t_sz = sizeof(sigset_t);
   unsigned struct_timezone_sz = sizeof(struct timezone);
   unsigned struct_tms_sz = sizeof(struct tms);
   unsigned struct_sigevent_sz = sizeof(struct sigevent);
   unsigned struct_sched_param_sz = sizeof(struct sched_param);
   unsigned struct_regex_sz = sizeof(regex_t);
   unsigned struct_regmatch_sz = sizeof(regmatch_t);
 
-#if (SANITIZER_APPLE && !TARGET_CPU_ARM64) && !SANITIZER_IOS
+#if SANITIZER_HAS_STATFS64
   unsigned struct_statfs64_sz = sizeof(struct statfs64);
-#endif // (SANITIZER_APPLE && !TARGET_CPU_ARM64) && !SANITIZER_IOS
+#endif // SANITIZER_HAS_STATFS64
 
 #if SANITIZER_GLIBC || SANITIZER_FREEBSD || SANITIZER_NETBSD || SANITIZER_APPLE
   unsigned struct_fstab_sz = sizeof(struct fstab);
 #endif  // SANITIZER_GLIBC || SANITIZER_FREEBSD || SANITIZER_NETBSD ||
         // SANITIZER_APPLE
 #if !SANITIZER_ANDROID
   unsigned struct_statfs_sz = sizeof(struct statfs);
   unsigned struct_sockaddr_sz = sizeof(struct sockaddr);