[asancov] Write coverage directly to a memory-mapped file

[eugenis]

Summary

This way does not require a __sanitizer_cov_dump() call. That's important on Android, where apps can be killed at arbitrary time.

We write raw PCs to disk instead of module offsets; we also write memory layout to a separate file. This increases dump size by the factor of 2 on 64-bit systems.

[kcc]

lib/sanitizer_common/sanitizer_coverage_direct.cc

1 ↗

(On Diff #9693)

This file deserves an extended comment.

[eugenis]

lib/sanitizer_common/sanitizer_coverage_direct.cc

1 ↗

(On Diff #9693)

done

[kcc]

lib/sanitizer_common/sanitizer_coverage_direct.cc

17 ↗

(On Diff #9695)

You mean, 32 or 64?

73 ↗

(On Diff #9695)

Just in case, I would replace this with

for (int i = 0; i < (1<<30); i++) {
      ....
}
CHECK(0 && "CoverageMap::TryAdd wanted to loop forever")

87 ↗

(On Diff #9695)

I wonder if this thing is AS-safe.
This at least deserves a comment.

test/asan/TestCases/Linux/coverage-direct-large.cc

20

don't you need to cleanup after the test, as other coverage tests do?

[eugenis]

lib/sanitizer_common/sanitizer_coverage_direct.cc

17 ↗

(On Diff #9695)

sure, I expanded the comment
we need this to parse the .sancov.raw file from a 32-bit process on a 64-bit machine.

73 ↗

(On Diff #9695)

This is gone in the new patch.

87 ↗

(On Diff #9695)

It was not. Good point. The new implementation is.

test/asan/TestCases/Linux/coverage-direct-large.cc

20

I think it's better to cleanup before the test, and leave test artifacts (under the Output dir) for inspection.

This way an interrupted test won't mess up the next run.

[eugenis]

See D3898 for instrumentation-side changes.
PTAL.

[kcc]

lib/sanitizer_common/sanitizer_coverage.cc

120

this function deserves a comment.

[eugenis]

lib/sanitizer_common/sanitizer_coverage.cc

120

done

[kcc]

LGTM

[eugenis]

I've had to move a bunch of functions to *_libcdep.cc because now coverage depends on GetListOfModules, which is implemented via dl_iterate_get_phdr.
Also, dlopen/dlclose interceptors are special in that they are not affected by and ignores and blacklists, so I added another macro in sanitizer_common_interceptors to handle them.

[dvyukov]

lib/sanitizer_common/sanitizer_coverage_libcdep.cc

94 ↗

(On Diff #9826)

same comment as for sancov.map

129 ↗

(On Diff #9826)

You need to increment it after MapWritableFileToMemory. Otherwise the CHECK in Add does not work.

130 ↗

(On Diff #9826)

Why do you need this to be atomic? It's accessed only in Init or under the mutex.

138 ↗

(On Diff #9826)

please print a readable error message with errno and Die
this is not a programming error, it can happen due to out of disk space of networking error

153 ↗

(On Diff #9826)

memory_order_acquire in CHECK looks suspicious
I think it needs to be just memory_order_relaxed

lib/sanitizer_common/sanitizer_coverage_mapping_libcdep.cc

43 ↗

(On Diff #9826)

It's just a "zd.sancov.map" string, right?
64 would be enough. 1024 makes me think that it's a full path, while it is not.

46 ↗

(On Diff #9826)

If the process is killed in the middle of this function, then we will get broken cov profile. And working in the presence of sudden kills was the main motivation for this changes.
I would write the data to ancov.map.tmp and then rename it to ancov.map.

lib/sanitizer_common/sanitizer_flags.cc

133

please explain when it can be useful (if the process is suddenly killed, it will leave some valid coverage file)

lib/tsan/rtl/tsan_interceptors.cc

211

debug leftover?

[eugenis]

lib/sanitizer_common/sanitizer_coverage_libcdep.cc

94 ↗

(On Diff #9826)

done

129 ↗

(On Diff #9826)

done

130 ↗

(On Diff #9826)

done

138 ↗

(On Diff #9826)

done

153 ↗

(On Diff #9826)

But then release-store to pc_array_size has no corresponding acquire-load.
Looks fine to me: this ordering is only needed for the CHECK to work, fetch_add one line above can be relaxed.

lib/sanitizer_common/sanitizer_coverage_mapping_libcdep.cc

43 ↗

(On Diff #9826)

done. but I'll increase it back to 1024 with the next commit anyway.

46 ↗

(On Diff #9826)

done

lib/sanitizer_common/sanitizer_flags.cc

133

done

[dvyukov]

LGTM

[Eugene.Zelenko]

Committed in rL209653.

lib/asan/asan_interceptors.cc

 // But asan does not remember UserId's for threads (pthread_t);
 // and remembers all ever existed threads, so the linear search by UserId
 // can be slow.
 #define COMMON_INTERCEPTOR_SET_PTHREAD_NAME(ctx, thread, name) \
   do {                                                         \
   } while (false)
 #define COMMON_INTERCEPTOR_BLOCK_REAL(name) REAL(name)
 #define COMMON_INTERCEPTOR_ON_EXIT(ctx) OnExit()
+#define COMMON_INTERCEPTOR_LIBRARY_LOADED(filename) CovUpdateMapping()
+#define COMMON_INTERCEPTOR_LIBRARY_UNLOADED() CovUpdateMapping()
 #include "sanitizer_common/sanitizer_common_interceptors.inc"
 
 #define COMMON_SYSCALL_PRE_READ_RANGE(p, s) ASAN_READ_RANGE(p, s)
 #define COMMON_SYSCALL_PRE_WRITE_RANGE(p, s) ASAN_WRITE_RANGE(p, s)
 #define COMMON_SYSCALL_POST_READ_RANGE(p, s) \
   do {                                       \
     (void)(p);                               \
     (void)(s);                               \

lib/sanitizer_common/CMakeLists.txt

 # Build system for the common Sanitizer runtime support library components.
 # These components are shared between AddressSanitizer and ThreadSanitizer.
 
 set(SANITIZER_SOURCES
   sanitizer_allocator.cc
   sanitizer_common.cc
   sanitizer_coverage.cc
+  sanitizer_coverage_mapping.cc
   sanitizer_deadlock_detector1.cc
   sanitizer_deadlock_detector2.cc
   sanitizer_flags.cc
   sanitizer_libc.cc
   sanitizer_libignore.cc
   sanitizer_linux.cc
   sanitizer_mac.cc
   sanitizer_persistent_allocator.cc
set(SANITIZER_HEADERS
   sanitizer_atomic_msvc.h
   sanitizer_bitvector.h
   sanitizer_bvgraph.h
   sanitizer_common.h
   sanitizer_common_interceptors.inc
   sanitizer_common_interceptors_ioctl.inc
   sanitizer_common_interceptors_format.inc
   sanitizer_common_syscalls.inc
+  sanitizer_coverage_mapping.h
   sanitizer_deadlock_detector.h
   sanitizer_deadlock_detector_interface.h
   sanitizer_flags.h
   sanitizer_internal_defs.h
   sanitizer_lfstack.h
   sanitizer_libc.h
   sanitizer_libignore.h
   sanitizer_linux.h

lib/sanitizer_common/sanitizer_common.h

 // The size of the mmaped region is stored in '*buff_size',
 // Returns the number of read bytes or 0 if file can not be opened.
 uptr ReadFileToBuffer(const char *file_name, char **buff,
                       uptr *buff_size, uptr max_len);
 // Maps given file to virtual memory, and returns pointer to it
 // (or NULL if the mapping failes). Stores the size of mmaped region
 // in '*buff_size'.
 void *MapFileToMemory(const char *file_name, uptr *buff_size);
+void *MapWritableFileToMemory(void *addr, uptr size, uptr fd, uptr offset);
 
 // Error report formatting.
 const char *StripPathPrefix(const char *filepath,
                             const char *strip_file_prefix);
 void PrintSourceLocation(InternalScopedString *buffer, const char *file,
                          int line, int column);
 void PrintModuleAndOffset(InternalScopedString *buffer,
                           const char *module, uptr offset);

 void ReExec();
 bool StackSizeIsUnlimited();
 void SetStackSizeLimitInBytes(uptr limit);
 void AdjustStackSize(void *attr);
 void PrepareForSandboxing(__sanitizer_sandbox_arguments *args);
 void CovPrepareForSandboxing(__sanitizer_sandbox_arguments *args);
 void SetSandboxingCallback(void (*f)());
 
+void CovUpdateMapping();
+
 void InitTlsSize();
 uptr GetTlsSize();
 
 // Other
 void SleepForSeconds(int seconds);
 void SleepForMillis(int millis);
 u64 NanoTime();
 int Atexit(void (*function)(void));

  public:
   LoadedModule(const char *module_name, uptr base_address);
   void addAddressRange(uptr beg, uptr end);
   bool containsAddress(uptr address) const;
 
   const char *full_name() const { return full_name_; }
   uptr base_address() const { return base_address_; }
 
+  uptr n_ranges() const { return n_ranges_; }
+  uptr address_range_start(int i) const { return ranges_[i].beg; }
+  uptr address_range_end(int i) const { return ranges_[i].end; }
+
  private:
   struct AddressRange {
     uptr beg;
     uptr end;
   };
   char *full_name_;
   uptr base_address_;
   static const uptr kMaxNumberOfAddressRanges = 6;

lib/sanitizer_common/sanitizer_common_interceptors.inc

 #ifndef COMMON_INTERCEPTOR_FILE_OPEN
 #define COMMON_INTERCEPTOR_FILE_OPEN(ctx, file, path) {}
 #endif
 
 #ifndef COMMON_INTERCEPTOR_FILE_CLOSE
 #define COMMON_INTERCEPTOR_FILE_CLOSE(ctx, file) {}
 #endif
 
+#ifndef COMMON_INTERCEPTOR_LIBRARY_LOADED
+#define COMMON_INTERCEPTOR_LIBRARY_LOADED(filename) {}
+#endif
+
+#ifndef COMMON_INTERCEPTOR_LIBRARY_UNLOADED
+#define COMMON_INTERCEPTOR_LIBRARY_UNLOADED() {}
+#endif
+
 struct FileMetadata {
   // For open_memstream().
   char **addr;
   SIZE_T *size;
 };
 
 struct CommonInterceptorMetadata {
   enum {
INTERCEPTOR(int, fclose, __sanitizer_FILE *fp) {
   }
   return REAL(fclose)(fp);
 }
 #define INIT_FCLOSE COMMON_INTERCEPT_FUNCTION(fclose);
 #else
 #define INIT_FCLOSE
 #endif
 
+#if SANITIZER_INTERCEPT_DLOPEN_DLCLOSE
+INTERCEPTOR(void*, dlopen, const char *filename, int flag) {
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, dlopen, filename, flag);
+  void *res = REAL(dlopen)(filename, flag);
+  COMMON_INTERCEPTOR_LIBRARY_LOADED(filename);
+  return res;
+}
+
+INTERCEPTOR(int, dlclose, void *handle) {
+  void *ctx;
+  COMMON_INTERCEPTOR_ENTER(ctx, dlclose, handle);
+  int res = REAL(dlclose)(handle);
+  COMMON_INTERCEPTOR_LIBRARY_UNLOADED();
+  return res;
+}
+#define INIT_DLOPEN_DLCLOSE          \
+  COMMON_INTERCEPT_FUNCTION(dlopen); \
+  COMMON_INTERCEPT_FUNCTION(dlclose);
+#else
+#define INIT_DLOPEN_DLCLOSE
+#endif
+
 static void InitializeCommonInterceptors() {
   static u64 metadata_mem[sizeof(MetadataHashMap) / sizeof(u64) + 1];
   interceptor_metadata_map = new((void *)&metadata_mem) MetadataHashMap();
 
   INIT_TEXTDOMAIN;
   INIT_STRCMP;
   INIT_STRNCMP;
   INIT_STRCASECMP;
static void InitializeCommonInterceptors() {
   INIT_TSEARCH;
   INIT_LIBIO_INTERNALS;
   INIT_FOPEN;
   INIT_FOPEN64;
   INIT_OPEN_MEMSTREAM;
   INIT_OBSTACK;
   INIT_FFLUSH;
   INIT_FCLOSE;
+  INIT_DLOPEN_DLCLOSE;
 }

lib/sanitizer_common/sanitizer_coverage.cc

 #include "sanitizer_stacktrace.h"
 #include "sanitizer_flags.h"
 
 atomic_uint32_t dump_once_guard;  // Ensure that CovDump runs only once.
 
 // pc_array is the array containing the covered PCs.
 // To make the pc_array thread- and async-signal-safe it has to be large enough.
 // 128M counters "ought to be enough for anybody" (4M on 32-bit).
-// pc_array is allocated with MmapNoReserveOrDie and so it uses only as
-// much RAM as it really needs.
-static const uptr kPcArraySize = FIRST_32_SECOND_64(1 << 22, 1 << 27);
-static uptr *pc_array;
-static atomic_uintptr_t pc_array_index;
+
+// With coverage_direct=1 in ASAN_OPTIONS, pc_array memory is mapped to a file.
+// In this mode, __sanitizer_cov_dump does nothing, and CovUpdateMapping()
+// dump current memory layout to another file.
 
 static bool cov_sandboxed = false;
 static int cov_fd = kInvalidFd;
 static unsigned int cov_max_block_size = 0;
 
 namespace __sanitizer {
 
+class CoverageData {
+ public:
+  void Init();
+  void Extend(uptr npcs);
+  void Add(uptr pc);
+
+  uptr *data();
+  uptr size();
+
+ private:
+  // Maximal size pc array may ever grow.
+  // We MmapNoReserve this space to ensure that the array is contiguous.
+  static const uptr kPcArrayMaxSize = FIRST_32_SECOND_64(1 << 22, 1 << 27);
+  // The amount file mapping for the pc array is grown by.
+  static const uptr kPcArrayMmapSize = 64 * 1024;
+
+  // pc_array is allocated with MmapNoReserveOrDie and so it uses only as
+  // much RAM as it really needs.
+  uptr *pc_array;
+  // Index of the first available pc_array slot.
+  atomic_uintptr_t pc_array_index;
+  // Array size.
+  atomic_uintptr_t pc_array_size;
+  // Current file mapped size of the pc array.
+  atomic_uintptr_t pc_array_mapped_size;
+  // Descriptor of the file mapped pc array.
+  int pc_fd;
+  StaticSpinMutex mu;
+
+  void DirectInit();
+};
+
+static CoverageData coverage_data;
+
+void CoverageData::DirectInit() {
+  InternalScopedString path(1024);
+  internal_snprintf((char *)path.data(), path.size(), "%zd.sancov.raw",
+                    internal_getpid());
+  pc_fd = OpenFile(path.data(), true);
+  if (internal_iserror(pc_fd)) {
+    Report(" Coverage: failed to open %s for writing\n", path.data());
+    Die();
+  }
+
+  atomic_store(&pc_array_size, 0, memory_order_relaxed);
+  atomic_store(&pc_array_mapped_size, 0, memory_order_relaxed);
+
+  CovUpdateMapping();
+}
+
+void CoverageData::Init() {
+  pc_array = reinterpret_cast<uptr *>(
+      MmapNoReserveOrDie(sizeof(uptr) * kPcArrayMaxSize, "CovInit"));
+  if (common_flags()->coverage_direct) {
+    DirectInit();
+  } else {
+    pc_fd = 0;
+    atomic_store(&pc_array_size, kPcArrayMaxSize, memory_order_relaxed);
+  }
+}
+
+void CoverageData::Extend(uptr npcs) {
+  if (!pc_fd) return;
+  SpinMutexLock l(&mu);
+
+  uptr delta = npcs * sizeof(uptr);
+  uptr size =
+      atomic_fetch_add(&pc_array_size, delta, memory_order_relaxed) + delta;
+  uptr mapped_size = atomic_load(&pc_array_mapped_size, memory_order_relaxed);
+  if (size <= mapped_size) return;
+
+  uptr new_mapped_size = mapped_size;
+  while (size > new_mapped_size) new_mapped_size += kPcArrayMmapSize;
+
+  uptr res = internal_ftruncate(pc_fd, new_mapped_size);
+  CHECK_EQ(res, 0);
+  void *p = MapWritableFileToMemory(pc_array + mapped_size,
+                                    new_mapped_size - mapped_size, pc_fd,
+                                    mapped_size);
+  CHECK_EQ(p, pc_array + mapped_size);
+
+  atomic_store(&pc_array_mapped_size, new_mapped_size, memory_order_release);
+}
+
 // Simply add the pc into the vector under lock. If the function is called more
 // than once for a given PC it will be inserted multiple times, which is fine.
-static void CovAdd(uptr pc) {
+void CoverageData::Add(uptr pc) {
   if (!pc_array) return;
   uptr idx = atomic_fetch_add(&pc_array_index, 1, memory_order_relaxed);
-  CHECK_LT(idx, kPcArraySize);
+  CHECK_LT(idx * sizeof(uptr),
+           atomic_load(&pc_array_size, memory_order_acquire));
   pc_array[idx] = pc;
 }
 
-void CovInit() {
-  pc_array = reinterpret_cast<uptr *>(
-      MmapNoReserveOrDie(sizeof(uptr) * kPcArraySize, "CovInit"));
+uptr *CoverageData::data() {
+  return pc_array;
 }
 
-static inline bool CompareLess(const uptr &a, const uptr &b) {
-  return a < b;
+uptr CoverageData::size() {
+  return atomic_load(&pc_array_index, memory_order_relaxed);
 }
 
 // Block layout for packed file format: header, followed by module name (no
 // trailing zero), followed by data blob.
 struct CovHeader {
   int pid;
   unsigned int module_name_length;
   unsigned int data_length;
while (blob_size > 0) {
       internal_write(cov_fd, block.data(),
                      header_size_with_module + payload_size);
     }
   }
 }
 
 // Dump the coverage on disk.
 static void CovDump() {
-  if (!common_flags()->coverage) return;
+  if (!common_flags()->coverage || common_flags()->coverage_direct) return;
 #if !SANITIZER_WINDOWS
   if (atomic_fetch_add(&dump_once_guard, 1, memory_order_relaxed))
     return;
-  uptr size = atomic_load(&pc_array_index, memory_order_relaxed);
-  InternalSort(&pc_array, size, CompareLess);
+  uptr size = coverage_data.size();
   InternalMmapVector<u32> offsets(size);
-  const uptr *vb = pc_array;
-  const uptr *ve = vb + size;
+  uptr *vb = coverage_data.data();
+  uptr *ve = vb + size;
+  SortArray(vb, size);
   MemoryMappingLayout proc_maps(/*cache_enabled*/true);
   uptr mb, me, off, prot;
   InternalScopedBuffer<char> module(4096);
   InternalScopedBuffer<char> path(4096 * 2);
   for (int i = 0;
        proc_maps.Next(&mb, &me, &off, module.data(), module.size(), &prot);
        i++) {
     if ((prot & MemoryMappingLayout::kProtectionExecute) == 0)
if (cov_fd < 0)
     // Pre-open the file now. The sandbox won't allow us to do it later.
     OpenPackedFileForWriting();
 }
 
 }  // namespace __sanitizer
 
 extern "C" {
 SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov() {
-  CovAdd(StackTrace::GetPreviousInstructionPc(GET_CALLER_PC()));
+  coverage_data.Add(StackTrace::GetPreviousInstructionPc(GET_CALLER_PC()));
 }
 SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_dump() { CovDump(); }
-SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_init() { CovInit(); }
+SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_init() {
+  coverage_data.Init();
+}
+SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_module_init(uptr npcs) {
+  coverage_data.Extend(npcs);
+}
 }  // extern "C"

lib/sanitizer_common/sanitizer_coverage_mapping.cc

+//===-- sanitizer_coverage_mapping.cc -------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// Mmap-based implementation of sanitizer coverage.
+//
+// This is part of the implementation of code coverage that does not require
+// __sanitizer_cov_dump() call. Data is stored in 2 files per process.
+//
+// $pid.sancov.map describes process memory layout in the following text-based
+// format:
+// <pointer size in bits>  // 1 line, 32 or 64
+// <mapping start> <mapping end> <base address> <dso name> // repeated
+// ...
+// Mapping lines are NOT sorted. This file is updated every time memory layout
+// is changed (i.e. in dlopen() and dlclose() interceptors).
+//
+// $pid.sancov.raw is a binary dump of PC values, sizeof(uptr) each. Again, not
+// sorted. This file is extended by 64Kb at a time and mapped into memory. It
+// contains one or more 0 words at the end, up to the next 64Kb aligned offset.
+//
+// To convert these 2 files to the usual .sancov format, run sancov.py rawunpack
+// $pid.sancov.raw.
+//
+//===----------------------------------------------------------------------===//
+
+#include "sanitizer_allocator_internal.h"
+#include "sanitizer_libc.h"
+#include "sanitizer_procmaps.h"
+
+namespace __sanitizer {
+
+static const uptr kMaxNumberOfModules = 1 << 14;
+
+void CovUpdateMapping() {
+  if (!common_flags()->coverage || !common_flags()->coverage_direct) return;
+
+  InternalScopedString path(1024);
+  internal_snprintf((char *)path.data(), path.size(), "%zd.sancov.map",
+                    internal_getpid());
+  uptr map_fd = OpenFile(path.data(), true);
+  if (internal_iserror(map_fd)) {
+    Report(" Coverage: failed to open %s for writing\n", path.data());
+    Die();
+  }
+
+  InternalScopedBuffer<char> modules_data(kMaxNumberOfModules *
+                                          sizeof(LoadedModule));
+  LoadedModule *modules = (LoadedModule *)modules_data.data();
+  CHECK(modules);
+  int n_modules = GetListOfModules(modules, kMaxNumberOfModules,
+                                   /* filter */ 0);
+
+  InternalScopedString line(4096);
+  line.append("%d\n", sizeof(uptr) * 8);
+  uptr res = internal_write(map_fd, line.data(), line.length());
+  CHECK_EQ(res, line.length());
+  line.clear();
+
+  for (int i = 0; i < n_modules; ++i) {
+    char *module_name = StripModuleName(modules[i].full_name());
+    for (unsigned j = 0; j < modules[i].n_ranges(); ++j) {
+      line.append("%zx %zx %zx %s\n", modules[i].address_range_start(j),
+                  modules[i].address_range_end(j), modules[i].base_address(),
+                  module_name);
+      uptr res = internal_write(map_fd, line.data(), line.length());
+      CHECK_EQ(res, line.length());
+      line.clear();
+    }
+    InternalFree(module_name);
+  }
+
+  internal_close(map_fd);
+}
+
+}  // namespace __sanitizer

lib/sanitizer_common/sanitizer_flags.h

struct CommonFlags {
   bool detect_deadlocks;
   uptr clear_shadow_mmap_threshold;
   const char *color;
   bool legacy_pthread_cond;
   bool intercept_tls_get_addr;
   bool help;
   uptr mmap_limit_mb;
   bool coverage;
+  bool coverage_direct;
   bool full_address_space;
 };
 
 inline CommonFlags *common_flags() {
   extern CommonFlags common_flags_dont_use;
   return &common_flags_dont_use;
 }
 
 void SetCommonFlagsDefaults(CommonFlags *f);
 void ParseCommonFlagsFromString(CommonFlags *f, const char *str);
 void PrintFlagDescriptions();
 
 }  // namespace __sanitizer
 
 #endif  // SANITIZER_FLAGS_H

lib/sanitizer_common/sanitizer_flags.cc

void SetCommonFlagsDefaults(CommonFlags *f) {
   f->allow_user_segv_handler = false;
   f->use_sigaltstack = true;
   f->detect_deadlocks = false;
   f->clear_shadow_mmap_threshold = 64 * 1024;
   f->color = "auto";
   f->legacy_pthread_cond = false;
   f->intercept_tls_get_addr = false;
   f->coverage = false;
+  f->coverage_direct = false;
   f->full_address_space = false;
 }
 
 void ParseCommonFlagsFromString(CommonFlags *f, const char *str) {
   ParseFlag(str, &f->symbolize, "symbolize",
       "If set, use the online symbolizer from common sanitizer runtime to turn "
       "virtual addresses to file/line locations.");
   ParseFlag(str, &f->external_symbolizer_path, "external_symbolizer_path",
ParseFlag(str, &f->intercept_tls_get_addr, "intercept_tls_get_addr",
             "Intercept __tls_get_addr.");
   ParseFlag(str, &f->help, "help", "Print the flag descriptions.");
   ParseFlag(str, &f->mmap_limit_mb, "mmap_limit_mb",
             "Limit the amount of mmap-ed memory (excluding shadow) in Mb; "
             "not a user-facing flag, used mosly for testing the tools");
   ParseFlag(str, &f->coverage, "coverage",
       "If set, coverage information will be dumped at program shutdown (if the "
       "coverage instrumentation was enabled at compile time).");
+  ParseFlag(str, &f->coverage_direct, "coverage_direct",
+            "If set, coverage information will be dumped directly to a memory "
+            "mapped file.");
   ParseFlag(str, &f->full_address_space, "full_address_space",
             "Sanitize complete address space; "
             "by default kernel area on 32-bit platforms will not be sanitized");
 
   // Do a sanity check for certain flags.
   if (f->malloc_context_size < 1)
     f->malloc_context_size = 1;
 }

lib/sanitizer_common/sanitizer_libc.h

 int internal_isatty(fd_t fd);
 
 // Use __sanitizer::OpenFile() instead.
 uptr internal_open(const char *filename, int flags);
 uptr internal_open(const char *filename, int flags, u32 mode);
 
 uptr internal_read(fd_t fd, void *buf, uptr count);
 uptr internal_write(fd_t fd, const void *buf, uptr count);
+uptr internal_ftruncate(fd_t fd, uptr size);
 
 // OS
 uptr internal_filesize(fd_t fd);  // -1 on error.
 uptr internal_stat(const char *path, void *buf);
 uptr internal_lstat(const char *path, void *buf);
 uptr internal_fstat(fd_t fd, void *buf);
 uptr internal_dup2(int oldfd, int newfd);
 uptr internal_readlink(const char *path, char *buf, uptr bufsize);

lib/sanitizer_common/sanitizer_linux.cc

#if SANITIZER_USES_CANONICAL_LINUX_SYSCALLS
   return internal_syscall(SYSCALL(openat), AT_FDCWD, (uptr)filename, flags,
                           mode);
 #else
   return internal_syscall(SYSCALL(open), (uptr)filename, flags, mode);
 #endif
 }
 
 uptr OpenFile(const char *filename, bool write) {
-  return internal_open(filename,
-      write ? O_WRONLY | O_CREAT /*| O_CLOEXEC*/ : O_RDONLY, 0660);
+  return internal_open(
+      filename, write ? O_RDWR | O_CREAT /*| O_CLOEXEC*/ : O_RDONLY, 0660);
 }
 
 uptr internal_read(fd_t fd, void *buf, uptr count) {
   sptr res;
   HANDLE_EINTR(res, (sptr)internal_syscall(SYSCALL(read), fd, (uptr)buf,
                count));
   return res;
 }
 
 uptr internal_write(fd_t fd, const void *buf, uptr count) {
   sptr res;
   HANDLE_EINTR(res, (sptr)internal_syscall(SYSCALL(write), fd, (uptr)buf,
                count));
   return res;
 }
 
+uptr internal_ftruncate(fd_t fd, uptr size) {
+  sptr res;
+  HANDLE_EINTR(res, (sptr)internal_syscall(SYSCALL(ftruncate), fd, size));
+  return res;
+}
+
 #if !SANITIZER_LINUX_USES_64BIT_SYSCALLS && !SANITIZER_FREEBSD
 static void stat64_to_stat(struct stat64 *in, struct stat *out) {
   internal_memset(out, 0, sizeof(*out));
   out->st_dev = in->st_dev;
   out->st_ino = in->st_ino;
   out->st_mode = in->st_mode;
   out->st_nlink = in->st_nlink;
   out->st_uid = in->st_uid;

lib/sanitizer_common/sanitizer_platform_interceptors.h

 #define SANITIZER_INTERCEPT_TSEARCH SI_LINUX_NOT_ANDROID || SI_MAC
 #define SANITIZER_INTERCEPT_LIBIO_INTERNALS SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_FOPEN SI_NOT_WINDOWS
 #define SANITIZER_INTERCEPT_FOPEN64 SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_OPEN_MEMSTREAM SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_OBSTACK SI_LINUX_NOT_ANDROID
 #define SANITIZER_INTERCEPT_FFLUSH SI_NOT_WINDOWS
 #define SANITIZER_INTERCEPT_FCLOSE SI_NOT_WINDOWS
+#define SANITIZER_INTERCEPT_DLOPEN_DLCLOSE SI_NOT_WINDOWS
 
 #endif  // #ifndef SANITIZER_PLATFORM_INTERCEPTORS_H

lib/sanitizer_common/sanitizer_posix.cc

void *MapFileToMemory(const char *file_name, uptr *buff_size) {
   uptr fsize = internal_filesize(fd);
   CHECK_NE(fsize, (uptr)-1);
   CHECK_GT(fsize, 0);
   *buff_size = RoundUpTo(fsize, GetPageSizeCached());
   uptr map = internal_mmap(0, *buff_size, PROT_READ, MAP_PRIVATE, fd, 0);
   return internal_iserror(map) ? 0 : (void *)map;
 }
 
+void *MapWritableFileToMemory(void *addr, uptr size, uptr fd, uptr offset) {
+  uptr flags = MAP_SHARED;
+  if (addr) flags |= MAP_FIXED;
+  uptr p = internal_mmap(addr, size, PROT_READ | PROT_WRITE, flags, fd, offset);
+  if (internal_iserror(p)) {
+    Printf("could not map writable file (%zd, %zu, %zu): %zd\n", fd, offset,
+           size, p);
+    return 0;
+  }
+  return (void *)p;
+}
 
 static inline bool IntervalsAreSeparate(uptr start1, uptr end1,
                                         uptr start2, uptr end2) {
   CHECK(start1 <= end1);
   CHECK(start2 <= end2);
   return (end1 < start2) || (end2 < start1);
 }
 

lib/sanitizer_common/scripts/sancov.py

 #!/usr/bin/env python
 # Merge or print the coverage data collected by asan's coverage.
 # Input files are sequences of 4-byte integers.
 # We need to merge these integers into a set and then
 # either print them (as hex) or dump them into another file.
 import array
 import struct
 import sys
+import bisect
+import os.path
 
 prog_name = "";
 
 def Usage():
   print >> sys.stderr, "Usage: \n" + \
       " " + prog_name + " merge file1 [file2 ...]  > output\n" \
       " " + prog_name + " print file1 [file2 ...]\n" \
       " " + prog_name + " unpack file1 [file2 ...]\n"
with open(path, mode="rb") as f:
     # fail
     raise Exception('Error reading file %s' % path)
 
 
 def Unpack(files):
   for f in files:
     UnpackOneFile(f)
 
+def UnpackOneRawFile(path, map_path):
+  mem_map = []
+  with open(map_path, mode="rt") as f_map:
+    print >> sys.stderr, "%s: reading map %s" % (prog_name, map_path)
+    bits = int(f_map.readline())
+    for line in f_map:
+      parts = line.rstrip().split()
+      assert len(parts) == 4
+      mem_map.append((int(parts[0], 16),
+                  int(parts[1], 16),
+                  int(parts[2], 16),
+                  parts[3]))
+  mem_map.sort(key=lambda m : m[0])
+  mem_map_keys = [m[0] for m in mem_map]
+
+  print mem_map
+  with open(path, mode="rb") as f:
+    print >> sys.stderr, "%s: unpacking %s" % (prog_name, path)
+
+    f.seek(0, 2)
+    size = f.tell()
+    f.seek(0, 0)
+    if bits == 64:
+      typecode = 'L'
+    else:
+      typecode = 'I'
+    pcs = array.array(typecode, f.read(size))
+    mem_map_pcs = [[] for i in range(0, len(mem_map))]
+
+    for pc in pcs:
+      if pc == 0: continue
+      map_idx = bisect.bisect(mem_map_keys, pc) - 1
+      (start, end, base, module_path) = mem_map[map_idx]
+      print pc
+      print start, end, base, module_path
+      assert pc >= start
+      if pc >= end:
+        print >> sys.stderr, "warning: %s: pc %x outside of any known mapping" % (prog_name, pc)
+        continue
+      mem_map_pcs[map_idx].append(pc - base)
+
+    for ((start, end, base, module_path), pc_list) in zip(mem_map, mem_map_pcs):
+      if len(pc_list) == 0: continue
+      assert path.endswith('.sancov.raw')
+      dst_path = module_path + '.' + os.path.basename(path)[:-4]
+      print "writing %d PCs to %s" % (len(pc_list), dst_path)
+      arr = array.array('I')
+      arr.fromlist(sorted(pc_list))
+      with open(dst_path, 'ab') as f2:
+        arr.tofile(f2)
+
+def RawUnpack(files):
+  for f in files:
+    if not f.endswith('.sancov.raw'):
+      raise Exception('Unexpected raw file name %s' % f)
+    f_map = f[:-3] + 'map'
+    UnpackOneRawFile(f, f_map)
 
 if __name__ == '__main__':
   prog_name = sys.argv[0]
   if len(sys.argv) <= 2:
     Usage();
   if sys.argv[1] == "print":
     PrintFiles(sys.argv[2:])
   elif sys.argv[1] == "merge":
     MergeAndPrint(sys.argv[2:])
   elif sys.argv[1] == "unpack":
     Unpack(sys.argv[2:])
+  elif sys.argv[1] == "rawunpack":
+    RawUnpack(sys.argv[2:])
   else:
     Usage()

lib/tsan/rtl/tsan_interceptors.cc

 /**/
 
 #define SCOPED_TSAN_INTERCEPTOR(func, ...) \
     SCOPED_INTERCEPTOR_RAW(func, __VA_ARGS__); \
     if (REAL(func) == 0) { \
       Report("FATAL: ThreadSanitizer: failed to intercept %s\n", #func); \
       Die(); \
     } \
     if (thr->ignore_interceptors || thr->in_ignored_lib) \
       return REAL(func)(__VA_ARGS__); \
 /**/
 
 #define TSAN_INTERCEPTOR(ret, func, ...) INTERCEPTOR(ret, func, __VA_ARGS__)
 #define TSAN_INTERCEPT(func) INTERCEPT_FUNCTION(func)
 #define TSAN_INTERCEPT_VER(func, ver) INTERCEPT_FUNCTION_VER(func, ver)
 
 #define BLOCK_REAL(name) (BlockingCall(thr), REAL(name))

 
 TSAN_INTERCEPTOR(int, nanosleep, void *req, void *rem) {
   SCOPED_TSAN_INTERCEPTOR(nanosleep, req, rem);
   int res = BLOCK_REAL(nanosleep)(req, rem);
   AfterSleep(thr, pc);
   return res;
 }
 
-TSAN_INTERCEPTOR(void*, dlopen, const char *filename, int flag) {
-  SCOPED_INTERCEPTOR_RAW(dlopen, filename, flag);
-  void *res = REAL(dlopen)(filename, flag);
-  libignore()->OnLibraryLoaded(filename);
-  return res;
-}
-
-TSAN_INTERCEPTOR(int, dlclose, void *handle) {
-  SCOPED_INTERCEPTOR_RAW(dlclose, handle);
-  int res = REAL(dlclose)(handle);
-  libignore()->OnLibraryUnloaded();
-  return res;
-}
-
 class AtExitContext {
  public:
   AtExitContext()
     : mtx_(MutexTypeAtExit, StatMtxAtExit)
     , pos_() {
   }
 
   typedef void(*atexit_t)();
#define COMMON_INTERCEPTOR_FILE_OPEN(ctx, file, path) \
   }
 
 #define COMMON_INTERCEPTOR_FILE_CLOSE(ctx, file) \
   if (file) {                                    \
     int fd = fileno_unlocked(file);              \
     if (fd >= 0) FdClose(thr, pc, fd);           \
   }
 
+#define COMMON_INTERCEPTOR_LIBRARY_LOADED(filename) \
+  libignore()->OnLibraryLoaded(filename)
+
+#define COMMON_INTERCEPTOR_LIBRARY_UNLOADED() \
+  libignore()->OnLibraryUnloaded()
+
 #define COMMON_INTERCEPTOR_FD_ACQUIRE(ctx, fd) \
   FdAcquire(((TsanInterceptorContext *) ctx)->thr, pc, fd)
 
 #define COMMON_INTERCEPTOR_FD_RELEASE(ctx, fd) \
   FdRelease(((TsanInterceptorContext *) ctx)->thr, pc, fd)
 
 #define COMMON_INTERCEPTOR_FD_ACCESS(ctx, fd) \
   FdAccess(((TsanInterceptorContext *) ctx)->thr, pc, fd)
void InitializeInterceptors() {
 
   TSAN_INTERCEPT(mlock);
   TSAN_INTERCEPT(munlock);
   TSAN_INTERCEPT(mlockall);
   TSAN_INTERCEPT(munlockall);
 
   TSAN_INTERCEPT(fork);
   TSAN_INTERCEPT(vfork);
-  TSAN_INTERCEPT(dlopen);
-  TSAN_INTERCEPT(dlclose);
   TSAN_INTERCEPT(on_exit);
   TSAN_INTERCEPT(__cxa_atexit);
   TSAN_INTERCEPT(_exit);
 
   // Need to setup it, because interceptors check that the function is resolved.
   // But atexit is emitted directly into the module, so can't be resolved.
   REAL(atexit) = (int(*)(void(*)()))unreachable;
   atexit_ctx = new(internal_alloc(MBlockAtExit, sizeof(AtExitContext)))

test/asan/TestCases/Linux/coverage-direct-large.cc

+// Test for direct coverage writing with lots of data.
+// Current implementation maps output file in chunks of 64K. This test overflows
+// 1 chunk.
+// RUN: %clangxx_asan -mllvm -asan-coverage=1 -O0 %s -o %t
+
+// RUN: rm -rf %T/coverage-direct-large
+
+// RUN: mkdir -p %T/coverage-direct-large/normal && cd %T/coverage-direct-large/normal
+// RUN: ASAN_OPTIONS=coverage=1:coverage_direct=0:verbosity=1 %run %t
+// RUN: %sancov print *.sancov >out.txt
+// RUN: cd ../..
+
+// RUN: mkdir -p %T/coverage-direct-large/direct && cd %T/coverage-direct-large/direct
+// RUN: ASAN_OPTIONS=coverage=1:coverage_direct=1:verbosity=1 %run %t
+// RUN: %sancov rawunpack *.sancov.raw
+// RUN: %sancov print *.sancov >out.txt
+// RUN: cd ../..
+
+// RUN: diff -u coverage-direct-large/normal/out.txt coverage-direct-large/direct/out.txt
+//
+// XFAIL: android
+
+#define F0(Q, x) Q(x)
+#define F1(Q, x)                                                          \
+  F0(Q, x##0) F0(Q, x##1) F0(Q, x##2) F0(Q, x##3) F0(Q, x##4) F0(Q, x##5) \
+      F0(Q, x##6) F0(Q, x##7) F0(Q, x##8) F0(Q, x##9)
+#define F2(Q, x)                                                          \
+  F1(Q, x##0) F1(Q, x##1) F1(Q, x##2) F1(Q, x##3) F1(Q, x##4) F1(Q, x##5) \
+      F1(Q, x##6) F1(Q, x##7) F1(Q, x##8) F1(Q, x##9)
+#define F3(Q, x)                                                          \
+  F2(Q, x##0) F2(Q, x##1) F2(Q, x##2) F2(Q, x##3) F2(Q, x##4) F2(Q, x##5) \
+      F2(Q, x##6) F2(Q, x##7) F2(Q, x##8) F2(Q, x##9)
+#define F4(Q, x)                                                          \
+  F3(Q, x##0) F3(Q, x##1) F3(Q, x##2) F3(Q, x##3) F3(Q, x##4) F3(Q, x##5) \
+      F3(Q, x##6) F3(Q, x##7) F3(Q, x##8) F3(Q, x##9)
+
+#define DECL(x) __attribute__((noinline)) void x() {}
+#define CALL(x) x();
+
+F4(DECL, f)
+
+int main(void) {
+  F4(CALL, f)
+  return 0;
+}

test/asan/TestCases/Linux/coverage-direct.cc

+// Test for direct coverage writing with dlopen.
+// RUN: %clangxx_asan -mllvm -asan-coverage=1 -DSHARED %s -shared -o %T/libcoverage_direct_test_1.so -fPIC
+// RUN: %clangxx_asan -mllvm -asan-coverage=1 -DSO_DIR=\"%T\" %s -o %t
+
+// RUN: rm -rf %T/coverage-direct
+
+// RUN: mkdir -p %T/coverage-direct/normal && cd %T/coverage-direct/normal
+// RUN: ASAN_OPTIONS=coverage=1:coverage_direct=0:verbosity=1 %run %t
+// RUN: %sancov print *.sancov >out.txt
+// RUN: cd ../..
+
+// RUN: mkdir -p %T/coverage-direct/direct && cd %T/coverage-direct/direct
+// RUN: ASAN_OPTIONS=coverage=1:coverage_direct=1:verbosity=1 %run %t
+// RUN: %sancov rawunpack *.sancov.raw
+// RUN: %sancov print *.sancov >out.txt
+// RUN: cd ../..
+
+// RUN: diff -u coverage-direct/normal/out.txt coverage-direct/direct/out.txt
+//
+// XFAIL: android
+
+#include <assert.h>
+#include <dlfcn.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#ifdef SHARED
+extern "C" {
+void bar() { printf("bar\n"); }
+}
+#else
+
+int main(int argc, char **argv) {
+  fprintf(stderr, "PID: %d\n", getpid());
+  void *handle1 =
+      dlopen(SO_DIR "/libcoverage_direct_test_1.so", RTLD_LAZY);
+  assert(handle1);
+  void (*bar1)() = (void (*)())dlsym(handle1, "bar");
+  assert(bar1);
+  bar1();
+
+  return 0;
+}
+#endif