[sanitizer] Support sandboxing in sanitizer coverage.

[earthdok]

Summary

Sandboxed code may now pass additional arguments to
__sanitizer_sandbox_on_notify() to force all coverage data to be dumped to a
single file (the default is one file per module). The user may supply a file or
socket to write to. The latter option can be used to broker out the file writing
functionality. If -1 is passed, we pre-open a file.

[earthdok]

• whitespace

[kcc]

+glider

[kcc]

include/sanitizer/common_interface_defs.h

27

use typedef so that you don't need to repeat struct every time

29

I wonder how this is going to work on Windows

30

comments?

lib/sanitizer_common/sanitizer_coverage.cc

78

comments?

[glider]

include/sanitizer/common_interface_defs.h

27

I think we need some "version" parameter to distinguish between different versions of this structure.

30

Should this really be passed through the sandbox arguments? I don't think the clients need to care much.

lib/sanitizer_common/sanitizer_coverage.cc

46

While at it, can you please replace "AS- safe" with "async-signal-safe"?

55

Please use kInvalidFd.

80

This should be u32, like the offsets.
I'd even suggest to define something like module_off_t in this file.

96

I think it's more common to keep the trailing zeroes, YMMV.

105

s/block_bytes/block_pos ?

116

Please define "proper"

162

Are you adding the leading whitespaces on purpose (here and below)? Why?

186

CHECK(cov_fd != kInvalidFd), remove NOLINT.

lib/sanitizer_common/sanitizer_internal_defs.h

98

You shouldn't need to redefine this struct.

lib/sanitizer_common/scripts/sancov.py

48

Two lines between the function declarations, please.

[earthdok]

include/sanitizer/common_interface_defs.h

29

It's hard to speculate in the absense of any users of __sanitizer_sandbox_on_notify() on Windows.

30

We must not send a longer message than the other side is prepared to receive, otherwise the message gets truncated.

lib/sanitizer_common/sanitizer_coverage.cc

80

This is string length, not module data length. I agree the name is confusing, but it follows pre-existing naming in this file. I should probably add "_name" everywhere.

96

I agree. However, dropping the zero makes the code here and in the parser script a bit more straightforward, and I see no other major reasons to pick one over the other.

162

Following the style set by the two pre-existing Report() calls in this file.

lib/sanitizer_common/sanitizer_internal_defs.h

98

Suggestions? We don't include the public header into private headers, or vice versa.

[earthdok]

include/sanitizer/common_interface_defs.h

27

done

30

done

lib/sanitizer_common/sanitizer_coverage.cc

46

done

55

done

78

done

80

done

105

changed naming

116

replaced with an explicit alignment check

186

done

[earthdok]

• address comments

[earthdok]

I think we need some "version" parameter to distinguish between different versions of this structure.

This basically only matters when the runtime is out of sync with the header for some reason. Kostya, do you think safeguarding against this case is worth the trouble?

[kcc]

Looks reasonable, but I'd like to see a lit test :)

lib/sanitizer_common/sanitizer_coverage.cc

90

use C++ initializer:

[kcc]

This basically only matters when the runtime is out of sync with the header for some reason. Kostya, do you think safeguarding against this case is worth the trouble?

I wouldn't worry at this point.

[earthdok]

• whitespace
• address comments
• use c++ initializer
• add test

[earthdok]

Test is up. I also switched to 'unsigned int' for the blob size, to keep the file format consistent between 32/64 bit. No one is going to send 4 GB messages over the socket anyway.

[rnk]

include/sanitizer/common_interface_defs.h

29

I think we are going to need it, and it will probably look similar, except this will be a HANDLE (aka void*) instead of an int.

When you add Windows support, you could do #ifdef WIN32 to avoid breaking ABI on Linux, but it might be better to use a uptr instead of an int to make it the same.

[kcc]

LGTM
Great test.

[earthdok]

• whitespace
• address comments
• use c++ initializer
• add test
• do not leave files after test
• use intptr_t for fd

[earthdok]

r209121

include/sanitizer/common_interface_defs.h

 // GCC does not understand __has_feature.
 #if !defined(__has_feature)
 # define __has_feature(x) 0
 #endif
 
 #ifdef __cplusplus
 extern "C" {
 #endif
+  // Arguments for __sanitizer_sandbox_on_notify() below.
+  typedef struct {
+    // Enable sandbox support in sanitizer coverage.
+    int coverage_sandboxed;
+    // File descriptor to write coverage data to. If -1 is passed, a file will
+    // be pre-opened by __sanitizer_sandobx_on_notify(). This field has no
+    // effect if coverage_sandboxed == 0.
+    int coverage_fd;
+    // If non-zero, split the coverage data into well-formed blocks. This is
+    // useful when coverage_fd is a socket descriptor. Each block will contain
+    // a header, allowing data from multiple processes to be sent over the same
+    // socket.
+    size_t coverage_max_block_size;
+  } __sanitizer_sandbox_arguments;
+
   // Tell the tools to write their reports to "path.<pid>" instead of stderr.
   void __sanitizer_set_report_path(const char *path);
 
   // Notify the tools that the sandbox is going to be turned on. The reserved
   // parameter will be used in the future to hold a structure with functions
   // that the tools may call to bypass the sandbox.
-  void __sanitizer_sandbox_on_notify(void *reserved);
+  void __sanitizer_sandbox_on_notify(__sanitizer_sandbox_arguments *args);
 
   // This function is called by the tool when it has just finished reporting
   // an error. 'error_summary' is a one-line string that summarizes
   // the error message. This function can be overridden by the client.
   void __sanitizer_report_error_summary(const char *error_summary);
 
   // Some of the sanitizers (e.g. asan/tsan) may miss bugs that happen
   // in unaligned loads/stores. In order to find such bugs reliably one needs

lib/sanitizer_common/sanitizer_common.h

 bool SetEnv(const char *name, const char *value);
 const char *GetPwd();
 char *FindPathToBinary(const char *name);
 u32 GetUid();
 void ReExec();
 bool StackSizeIsUnlimited();
 void SetStackSizeLimitInBytes(uptr limit);
 void AdjustStackSize(void *attr);
-void PrepareForSandboxing();
+void PrepareForSandboxing(__sanitizer_sandbox_arguments *args);
+void CovPrepareForSandboxing(__sanitizer_sandbox_arguments *args);
 void SetSandboxingCallback(void (*f)());
 
 void InitTlsSize();
 uptr GetTlsSize();
 
 // Other
 void SleepForSeconds(int seconds);
 void SleepForMillis(int millis);

lib/sanitizer_common/sanitizer_common.cc

if (internal_strcmp(path, "stdout") == 0) {
     report_fd = kStderrFd;
   } else {
     internal_strncpy(report_path_prefix, path, sizeof(report_path_prefix));
     report_path_prefix[len] = '\0';
     log_to_file = true;
   }
 }
 
-void NOINLINE __sanitizer_sandbox_on_notify(void *reserved) {
-  (void)reserved;
-  PrepareForSandboxing();
+void NOINLINE
+__sanitizer_sandbox_on_notify(__sanitizer_sandbox_arguments *args) {
+  PrepareForSandboxing(args);
   if (sandboxing_callback)
     sandboxing_callback();
 }
 
 void __sanitizer_report_error_summary(const char *error_summary) {
   Printf("%s\n", error_summary);
 }
 }  // extern "C"

lib/sanitizer_common/sanitizer_coverage.cc

 #include "sanitizer_mutex.h"
 #include "sanitizer_procmaps.h"
 #include "sanitizer_stacktrace.h"
 #include "sanitizer_flags.h"
 
 atomic_uint32_t dump_once_guard;  // Ensure that CovDump runs only once.
 
 // pc_array is the array containing the covered PCs.
-// To make the pc_array thread- and AS- safe it has to be large enough.
+// To make the pc_array thread- and async-signal-safe it has to be large enough.
 // 128M counters "ought to be enough for anybody" (4M on 32-bit).
 // pc_array is allocated with MmapNoReserveOrDie and so it uses only as
 // much RAM as it really needs.
 static const uptr kPcArraySize = FIRST_32_SECOND_64(1 << 22, 1 << 27);
 static uptr *pc_array;
 static atomic_uintptr_t pc_array_index;
 
+static bool cov_sandboxed = false;
+static int cov_fd = kInvalidFd;
+static uptr cov_max_block_size = 0;
+
 namespace __sanitizer {
 
 // Simply add the pc into the vector under lock. If the function is called more
 // than once for a given PC it will be inserted multiple times, which is fine.
 static void CovAdd(uptr pc) {
   if (!pc_array) return;
   uptr idx = atomic_fetch_add(&pc_array_index, 1, memory_order_relaxed);
   CHECK_LT(idx, kPcArraySize);
   pc_array[idx] = pc;
 }
 
 void CovInit() {
   pc_array = reinterpret_cast<uptr *>(
       MmapNoReserveOrDie(sizeof(uptr) * kPcArraySize, "CovInit"));
 }
 
 static inline bool CompareLess(const uptr &a, const uptr &b) {
   return a < b;
 }
 
+// Block layout for packed file format: header, followed by module name (no
+// trailing zero), followed by data blob.
+struct CovHeader {
+  int pid;
+  unsigned int module_name_length;
+  uptr data_length;
+};
+
+static void CovWritePacked(int pid, const char *module, const void *blob,
+                           uptr blob_size) {
+  CHECK_GE(cov_fd, 0);
+  unsigned module_name_length = internal_strlen(module);
+  CovHeader header;
+  header.pid = pid;
+  header.module_name_length = module_name_length;
+  header.data_length = blob_size;
+
+  if (cov_max_block_size == 0) {
+    // Writing to a file. Just go ahead.
+    internal_write(cov_fd, &header, sizeof(header));
+    internal_write(cov_fd, module, module_name_length);
+    internal_write(cov_fd, blob, blob_size);
+  } else {
+    // Writing to a socket. We want to split the data into appropriately sized
+    // blocks.
+    InternalScopedBuffer<char> block(cov_max_block_size);
+    CHECK_EQ((uptr)block.data(), (uptr)(CovHeader *)block.data());
+    uptr header_size_with_module = sizeof(header) + module_name_length;
+    CHECK_LT(header_size_with_module, cov_max_block_size);
+    uptr max_payload_size = cov_max_block_size - header_size_with_module;
+    char *block_pos = block.data();
+    internal_memcpy(block_pos, &header, sizeof(header));
+    block_pos += sizeof(header);
+    internal_memcpy(block_pos, module, module_name_length);
+    block_pos += module_name_length;
+    char *block_data_begin = block_pos;
+    char *blob_pos = (char *)blob;
+    while (blob_size > 0) {
+      uptr payload_size = Min(blob_size, max_payload_size);
+      blob_size -= payload_size;
+      internal_memcpy(block_data_begin, blob_pos, payload_size);
+      blob_pos += payload_size;
+      ((CovHeader *)block.data())->data_length = payload_size;
+      internal_write(cov_fd, block.data(),
+                     header_size_with_module + payload_size);
+    }
+  }
+}
+
 // Dump the coverage on disk.
-void CovDump() {
+static void CovDump() {
+  if (!common_flags()->coverage) return;
 #if !SANITIZER_WINDOWS
   if (atomic_fetch_add(&dump_once_guard, 1, memory_order_relaxed))
     return;
   uptr size = atomic_load(&pc_array_index, memory_order_relaxed);
   InternalSort(&pc_array, size, CompareLess);
   InternalMmapVector<u32> offsets(size);
   const uptr *vb = pc_array;
   const uptr *ve = vb + size;
-  MemoryMappingLayout proc_maps(/*cache_enabled*/false);
+  MemoryMappingLayout proc_maps(/*cache_enabled*/true);
   uptr mb, me, off, prot;
   InternalScopedBuffer<char> module(4096);
   InternalScopedBuffer<char> path(4096 * 2);
   for (int i = 0;
        proc_maps.Next(&mb, &me, &off, module.data(), module.size(), &prot);
        i++) {
     if ((prot & MemoryMappingLayout::kProtectionExecute) == 0)
       continue;
     while (vb < ve && *vb < mb) vb++;
     if (vb >= ve) break;
     if (*vb < me) {
       offsets.clear();
       const uptr *old_vb = vb;
       CHECK_LE(off, *vb);
       for (; vb < ve && *vb < me; vb++) {
         uptr diff = *vb - (i ? mb : 0) + off;
         CHECK_LE(diff, 0xffffffffU);
         offsets.push_back(static_cast<u32>(diff));
       }
       char *module_name = StripModuleName(module.data());
+      if (cov_sandboxed) {
+        CovWritePacked(internal_getpid(), module_name, offsets.data(),
+                       offsets.size() * sizeof(u32));
+        VReport(1, " CovDump: %zd PCs written to packed file\n", vb - old_vb);
+      } else {
+        // One file per module per process.
         internal_snprintf((char *)path.data(), path.size(), "%s.%zd.sancov",
                           module_name, internal_getpid());
-      InternalFree(module_name);
         uptr fd = OpenFile(path.data(), true);
         if (internal_iserror(fd)) {
           Report(" CovDump: failed to open %s for writing\n", path.data());
         } else {
           internal_write(fd, offsets.data(), offsets.size() * sizeof(u32));
           internal_close(fd);
-        VReport(1, " CovDump: %s: %zd PCs written\n", path.data(), vb - old_vb);
+          VReport(1, " CovDump: %s: %zd PCs written\n", path.data(),
+                  vb - old_vb);
         }
       }
+      InternalFree(module_name);
     }
+  }
+  if (cov_fd >= 0)
+    internal_close(cov_fd);
 #endif  // !SANITIZER_WINDOWS
 }
 
+static void OpenPackedFileForWriting() {
+  CHECK(cov_fd == kInvalidFd);
+  InternalScopedBuffer<char> path(1024);
+  internal_snprintf((char *)path.data(), path.size(), "%zd.sancov.packed",
+                    internal_getpid());
+  uptr fd = OpenFile(path.data(), true);
+  if (internal_iserror(fd)) {
+    Report(" Coverage: failed to open %s for writing\n", path.data());
+    Die();
+  }
+  cov_fd = fd;
+}
+
+void CovPrepareForSandboxing(__sanitizer_sandbox_arguments *args) {
+  if (!args) return;
+  if (!common_flags()->coverage) return;
+  cov_sandboxed = args->coverage_sandboxed;
+  if (!cov_sandboxed) return;
+  cov_fd = args->coverage_fd;
+  cov_max_block_size = args->coverage_max_block_size;
+  if (cov_fd < 0)
+    // Pre-open the file now. The sandbox won't allow us to do it later.
+    OpenPackedFileForWriting();
+}
+
 }  // namespace __sanitizer
 
 extern "C" {
 SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov() {
   CovAdd(StackTrace::GetPreviousInstructionPc(GET_CALLER_PC()));
 }
 SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_dump() { CovDump(); }
 SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_init() { CovInit(); }
 }  // extern "C"

lib/sanitizer_common/sanitizer_internal_defs.h

 }  // namespace __sanitizer
 
 extern "C" {
   // Tell the tools to write their reports to "path.<pid>" instead of stderr.
   // The special values are "stdout" and "stderr".
   SANITIZER_INTERFACE_ATTRIBUTE
   void __sanitizer_set_report_path(const char *path);
 
-  // Notify the tools that the sandbox is going to be turned on. The reserved
-  // parameter will be used in the future to hold a structure with functions
-  // that the tools may call to bypass the sandbox.
-  SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
-  void __sanitizer_sandbox_on_notify(void *reserved);
+  typedef struct {
+      int coverage_sandboxed;
+      int coverage_fd;
+      __sanitizer::uptr coverage_max_block_size;
+  } __sanitizer_sandbox_arguments;
+
+  // Notify the tools that the sandbox is going to be turned on.
+  SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE void
+      __sanitizer_sandbox_on_notify(__sanitizer_sandbox_arguments *args);
 
   // This function is called by the tool when it has just finished reporting
   // an error. 'error_summary' is a one-line string that summarizes
   // the error message. This function can be overridden by the client.
   SANITIZER_INTERFACE_ATTRIBUTE SANITIZER_WEAK_ATTRIBUTE
   void __sanitizer_report_error_summary(const char *error_summary);
 
   SANITIZER_INTERFACE_ATTRIBUTE void __sanitizer_cov_dump();

lib/sanitizer_common/sanitizer_linux.cc

void GetThreadStackAndTls(bool main, uptr *stk_addr, uptr *stk_size,
                           uptr *tls_addr, uptr *tls_size) {
   *stk_addr = 0;
   *stk_size = 0;
   *tls_addr = 0;
   *tls_size = 0;
 }
 #endif  // SANITIZER_GO
 
-void PrepareForSandboxing() {
+void PrepareForSandboxing(__sanitizer_sandbox_arguments *args) {
   // Some kinds of sandboxes may forbid filesystem access, so we won't be able
   // to read the file mappings from /proc/self/maps. Luckily, neither the
   // process will be able to load additional libraries, so it's fine to use the
   // cached mappings.
   MemoryMappingLayout::CacheMemoryMappings();
   // Same for /proc/self/exe in the symbolizer.
 #if !SANITIZER_GO
   if (Symbolizer *sym = Symbolizer::GetOrNull())
     sym->PrepareForSandboxing();
 #endif
+  CovPrepareForSandboxing(args);
 }
 
 enum MutexState {
   MtxUnlocked = 0,
   MtxLocked = 1,
   MtxSleeping = 2
 };
 

lib/sanitizer_common/sanitizer_mac.cc

const char *GetEnv(const char *name) {
   }
   return 0;
 }
 
 void ReExec() {
   UNIMPLEMENTED();
 }
 
-void PrepareForSandboxing() {
+void PrepareForSandboxing(__sanitizer_sandbox_arguments *args) {
+  (void)args;
   // Nothing here for now.
 }
 
 uptr GetPageSize() {
   return sysconf(_SC_PAGESIZE);
 }
 
 BlockingMutex::BlockingMutex(LinkerInitialized) {

lib/sanitizer_common/sanitizer_win.cc

 void DisableCoreDumper() {
   // Do nothing.
 }
 
 void ReExec() {
   UNIMPLEMENTED();
 }
 
-void PrepareForSandboxing() {
+void PrepareForSandboxing(__sanitizer_sandbox_arguments *args) {
+  (void)args;
   // Nothing here for now.
 }
 
 bool StackSizeIsUnlimited() {
   UNIMPLEMENTED();
 }
 
 void SetStackSizeLimitInBytes(uptr limit) {

lib/sanitizer_common/scripts/sancov.py

 #!/usr/bin/env python
 # Merge or print the coverage data collected by asan's coverage.
 # Input files are sequences of 4-byte integers.
 # We need to merge these integers into a set and then
 # either print them (as hex) or dump them into another file.
 import array
+import struct
 import sys
 
 prog_name = "";
 
 def Usage():
   print >> sys.stderr, "Usage: \n" + \
       " " + prog_name + " merge file1 [file2 ...]  > output\n" \
-      " " + prog_name + " print file1 [file2 ...]\n"
+      " " + prog_name + " print file1 [file2 ...]\n" \
+      " " + prog_name + " unpack file1 [file2 ...]\n"
   exit(1)
 
 def ReadOneFile(path):
-  f = open(path, mode="rb")
+  with open(path, mode="rb") as f:
     f.seek(0, 2)
     size = f.tell()
     f.seek(0, 0)
     s = set(array.array('I', f.read(size)))
-  f.close()
   print >>sys.stderr, "%s: read %d PCs from %s" % (prog_name, size / 4, path)
   return s
 
 def Merge(files):
   s = set()
   for f in files:
     s = s.union(ReadOneFile(f))
   print >> sys.stderr, "%s: %d files merged; %d PCs total" % \
     (prog_name, len(files), len(s))
   return sorted(s)
 
 def PrintFiles(files):
   s = Merge(files)
   for i in s:
     print "0x%x" % i
 
 def MergeAndPrint(files):
   if sys.stdout.isatty():
     Usage()
   s = Merge(files)
   a = array.array('I', s)
   a.tofile(sys.stdout)
 
+
+def UnpackOneFile(path):
+  with open(path, mode="rb") as f:
+    print >> sys.stderr, "%s: unpacking %s" % (prog_name, path)
+    while True:
+      header = f.read(16)
+      if not header: return
+      if len(header) < 16:
+        break
+      pid, module_length, blob_size = struct.unpack('iIL', header)
+      module = f.read(module_length)
+      blob = f.read(blob_size)
+      assert(len(module) == module_length)
+      assert(len(blob) == blob_size)
+      extracted_file = "%s.%d.sancov" % (module, pid)
+      print >> sys.stderr, "%s: extracting %s" % \
+        (prog_name, extracted_file)
+      with open(extracted_file, 'wb') as f2:
+        # The packed file may contain multiple blobs for the same pid/module
+        # pair. Append to the end of the file instead of overwriting.
+        f2.seek(0, 2)
+        f2.write(blob)
+    # fail
+    raise Exception('Error reading file %s' % path)
+
+
+def Unpack(files):
+  for f in files:
+    UnpackOneFile(f)
+
+
 if __name__ == '__main__':
   prog_name = sys.argv[0]
   if len(sys.argv) <= 2:
     Usage();
   if sys.argv[1] == "print":
     PrintFiles(sys.argv[2:])
   elif sys.argv[1] == "merge":
     MergeAndPrint(sys.argv[2:])
+  elif sys.argv[1] == "unpack":
+    Unpack(sys.argv[2:])
   else:
     Usage()