[Darwin] Fix symbolization for recent simulator runtimes.

[delcypher]

Summary

Due to sandbox restrictions in the recent versions of the simulator runtime the
atos program is no longer able to access the task port of a parent process
without additional help.

This patch fixes this by registering a task port for the parent process
before spawning atos and also tells atos to look for this by setting
a special environment variable.

This patch is based on an Apple internal fix (rdar://problem/43693565) that
unfortunately contained a bug (rdar://problem/58789439) because it used
setenv() to set the special environment variable. This is not safe because in
certain circumstances this can trigger a call to realloc() which can fail
during symbolization leading to deadlock. A test case is included that captures
this problem.

This patch takes a different approach by modifying a copy of the environment
that is passed to atos and so fixes both rdar://problem/43693565 and
rdar://problem/58789439.

In more detail calling setenv() during symbolization sometimes caused
a deadlock in the following situation:

1. Before TSan detects an issue Something calls setenv() that sets a

new environment variable that wasn't previously set. This triggers a
call to malloc() to allocate a new environment array. This uses TSan's
normal user-facing allocator. LibC stores this pointer for future use
later.

2. TSan detects an issue and tries to launch the symbolizer. When we are in the

symbolizer we switch to a different (internal allocator) and then we call
setenv() to set a new environment variable. When this happen setenv() sees
that it needs to make the environment array larger and calls realloc() on the
existing enviroment array because it remembers that it previously allocated
memory for it. Calling realloc() fails here because it is being called on a
pointer its never seen before.

The included test case closely reproduces the originally reported
problem but it doesn't replicate the `((kBlockMagic)) ==
((((u64*)addr)[0])` assertion failure exactly. This is due to the way
TSan's normal allocator allocates the environment array the first time
it is allocated. In the test program addr[0] accesses an inaccessible
page and raises SIGBUS. If TSan's SIGBUS signal handler is active, the
signal is caught and symbolication is attempted again which results in
deadlock.

In the originally reported problem the pointer is successfully derefenced but
then the assert fails due to the provided pointer not coming from the active
allocator. When the assert fails TSan tries to symbolicate the stacktrace while
already being in the middle of symbolication which results in deadlock.

rdar://problem/58789439

[delcypher]

Refactor

[delcypher]

typo fix.

[delcypher]

• Bug fix. If we are going to overwrite and existing entry but need to truncate use the existing entry slot.

[kubamracek]

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp

79

I would like this to be done with a sequence of call for helper functions: 1) Copy, 2) AddOrReplace. Please separate the symbolizer-specific logic (printing the warning, using custom_env_ as the destination) from the env manipulation logic.

[delcypher]

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp

79

To make that work AddOrReplace would also need to be given the storage for the entry being added (as well as the storage for the copy of the environment array) because AtosSymbolizerProcess owns the storage (atos_mach_port_env_entry_).

These helper functions you are proposing will be extremely cumbersome to use (due to needing to pass the storage) which makes we wonder if they will ever be re-used elsewhere in the code. Do you foresee places in the code where we would want to use them? If the answer is no then I think refactoring now would be premature.

[delcypher]

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp

79

@kubamracek I've come up with an abstraction for buffers that actually makes what you're asking for a lot more elegant and feasible. I'll rework my patches based on this.

template <class T>
struct BufferRef {
  T *data;
  const uptr size; // number of elements of type T in buffer.
  BufferRef() : data(nullptr), size(0) {}
  BufferRef(T *data, uptr size) : data(data), size(size) {}
  T &operator[](uptr idx) {
    CHECK_LT(idx, size);
    return data[idx];
  }
  void copyFrom(BufferRef<T> &src) {
    CHECK_GE(size, src.size);
    internal_memcpy(data, src.data, sizeof(T) * src.size);
  }
};

[delcypher]

Use new EnvArray type.

[delcypher]

Abandoning this change in favour of:

• https://reviews.llvm.org/D78178
• https://reviews.llvm.org/D78179

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp

 //===-- sanitizer_symbolizer_mac.cpp --------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file is shared between various sanitizers' runtime libraries.
 //
 // Implementation of Mac-specific "atos" symbolizer.
 //===----------------------------------------------------------------------===//
 
 #include "sanitizer_platform.h"
 #if SANITIZER_MAC
 
 #include <dlfcn.h>
 #include <errno.h>
+#include <mach/mach.h>
 #include <stdlib.h>
 #include <sys/wait.h>
 #include <unistd.h>
 #include <util.h>
 
 #include "sanitizer_allocator_internal.h"
 #include "sanitizer_env_array.h"
 #include "sanitizer_mac.h"
bool DlAddrSymbolizer::SymbolizeData(uptr addr, DataInfo *datainfo) {
   int result = dladdr((const void *)addr, &info);
   if (!result) return false;
   const char *demangled = DemangleSwiftAndCXX(info.dli_sname);
   datainfo->name = internal_strdup(demangled);
   datainfo->start = (uptr)info.dli_saddr;
   return true;
 }
 
+static const char kAtosMachPortEnvVar[] = "__check_mach_ports_lookup";
+
 class AtosSymbolizerProcess : public SymbolizerProcess {
  public:
   explicit AtosSymbolizerProcess(const char *path)
       : SymbolizerProcess(path, /*use_posix_spawn*/ true) {
     pid_str_[0] = '\0';
   }
 
  private:
   bool StartSymbolizerSubprocess() override {
     // Configure sandbox before starting atos process.
 
     // Put the string command line argument in the object so that it outlives
     // the call to GetArgV.
     internal_snprintf(pid_str_, sizeof(pid_str_), "%d", internal_getpid());
 
     // Prepare the environment.
     custom_env_.reset();
 
     // Copy the existing environment.
     // FIXME(dliew): This isn't quite right. There's a race here because we
     // shallow copy the strings which we don't own which could be free'd between
     // now and the time we launch the process. Deep copying wouldn't help much
     // because the exact same race exists. Really we need to intercept `setenv`
     // and friends and take a lock until we finish launching the process.
     uptr entries_copied = custom_env_.copyFromRawEnv(
         GetEnviron(), /*shallow_copy=*/true, /*allow_heap_allocations=*/false,
         /*overwrite=*/true);
     uptr num_entries = GetNumEnvironEntries(GetEnviron());
     if (entries_copied < num_entries) {
       Report(
           "WARNING: Truncating symbolizer environment. Needed %llu entries but "
           "only %llu are available.\n",
           num_entries, custom_env_.getNumEnvEntries());
     }
-    return SymbolizerProcess::StartSymbolizerSubprocess();
+
+    if (SANITIZER_IOSSIM) {
+      // `atos` in the simulator is restricted in its ability to retrieve the
+      // task port for the target process (us) so we need to do extra work
+      // to pass our task port to it.
+      mach_port_t ports[]{mach_task_self()};
+      kern_return_t ret =
+          mach_ports_register(mach_task_self(), ports, /*count=*/1);
+      CHECK_EQ(ret, KERN_SUCCESS);
+
+      // Add environment variable that signals to `atos` that it should look
+      // for our task port. We try to do this without heap allocations because
+      // the symbolizer might be spawned in a low memory situation.
+      bool success =
+          custom_env_.set(kAtosMachPortEnvVar, pid_str_, /*overwrite=*/true,
+                          /*allow_heap_allocations=*/false);
+      if (!success) {
+        CHECK_EQ(custom_env_.getNumAvailableStaticEnvEntries(), 0);
+        // Insertion failed. This should only happen if we ran out of static
+        // entry slots. Try removing the most recently added entry and add
+        // again.
+        CHECK_EQ(custom_env_.getNumAvailableStaticEnvEntries(), 0);
+        custom_env_.removeEntryAt(custom_env_.getNumEnvEntries() - 1);
+        // Try again.
+        success =
+            custom_env_.set(kAtosMachPortEnvVar, pid_str_, /*overwrite=*/true,
+                            /*allow_heap_allocations=*/false);
+        CHECK(success);
+      }
+      CHECK(!custom_env_.usingHeap());
     }
 
+    return SymbolizerProcess::StartSymbolizerSubprocess();
+  }
   bool ReachedEndOfOutput(const char *buffer, uptr length) const override {
     return (length >= 1 && buffer[length - 1] == '\n');
   }
 
   char **GetEnvP() override { return custom_env_.getStorage().data(); }
 
   void GetArgV(const char *path_to_binary,
                const char *(&argv)[kArgVMax]) const override {
     int i = 0;
     argv[i++] = path_to_binary;
     argv[i++] = "-p";
     argv[i++] = &pid_str_[0];
     if (GetMacosVersion() == MACOS_VERSION_MAVERICKS) {
       // On Mavericks atos prints a deprecation warning which we suppress by
       // passing -d. The warning isn't present on other OSX versions, even the
       // newer ones.
       argv[i++] = "-d";
     }
     argv[i++] = nullptr;
   }
 
   char pid_str_[16];
-  EnvArray<512, 1> custom_env_;
+  // Enough static storage for 511 environment variables and
+  // the addition of the kAtosMachPortEnvVar entry we might add.
+  EnvArray<512, sizeof(kAtosMachPortEnvVar) + sizeof(pid_str_) - 1> custom_env_;
 };
 
 static bool ParseCommandOutput(const char *str, uptr addr, char **out_name,
                                char **out_module, char **out_file, uptr *line,
                                uptr *start_address) {
   // Trim ending newlines.
   char *trim;
   ExtractTokenUpToDelimiter(str, "\n", &trim);

compiler-rt/test/tsan/Darwin/no_call_setenv_in_symbolize.cpp

+// RUN: %clangxx_tsan -O1 %s -o %t
+// `handle_sigbus=0` is required because when the rdar://problem/58789439 bug was
+// present TSan's runtime could derefence bad memory leading to SIGBUS being raised.
+// If the signal was caught TSan would deadlock because it would try to run the
+// symbolizer again.
+// RUN: %env_tsan_opts=handle_sigbus=0,symbolize=1 %run %t 2>&1 | FileCheck %s
+// RUN: %env_tsan_opts=handle_sigbus=0,symbolize=1 _check_mach_ports_lookup=some_value %run %t 2>&1 | FileCheck %s
+#include <sanitizer/common_interface_defs.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+const char *kEnvName = "__UNLIKELY_ENV_VAR_NAME__";
+
+int main() {
+  if (getenv(kEnvName)) {
+    fprintf(stderr, "Env var %s should not be set\n", kEnvName);
+    abort();
+  }
+
+  // This will set an environment variable that isn't already in
+  // the environment array. This will cause Darwin's Libc to
+  // malloc() a new array.
+  if (setenv(kEnvName, "some_value", /*overwrite=*/1)) {
+    fprintf(stderr, "Failed to set %s \n", kEnvName);
+    abort();
+  }
+
+  // rdar://problem/58789439
+  // Now trigger symbolization. If symbolization tries to call
+  // to `setenv` that adds a new environment variable, then Darwin
+  // Libc will call `realloc()` and TSan's runtime will hit
+  // an assertion failure because TSan's runtime uses a different
+  // allocator during symbolization which leads to `realloc()` being
+  // called on a pointer that the allocator didn't allocate.
+  //
+  // CHECK: #{{[0-9]}} main {{.*}}no_call_setenv_in_symbolize.cpp:[[@LINE+1]]
+  __sanitizer_print_stack_trace();
+
+  // CHECK: DONE
+  fprintf(stderr, "DONE\n");
+
+  return 0;
+}

compiler-rt/test/tsan/Darwin/symbolizer-large-env.cpp

+// FIXME(dliew): This duplicates a sanitizer_common test case because it doesn't run for iOSSim (rdar://problem/47333049).
+// RUN: %clangxx_tsan %s -g -O0 -o %t
+// RUN: %run %t 2>&1 | FileCheck %s
+#include <assert.h>
+#include <sanitizer/common_interface_defs.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+
+extern "C" char ***_NSGetEnviron(void);
+
+size_t get_num_env_entries() {
+  char **envp = *_NSGetEnviron();
+  size_t count = 0;
+  for (; envp[count]; ++count) {
+  }
+  return count;
+}
+
+const size_t kMaxNumEnvPEntries = 511;
+
+int main() {
+  size_t current_entries = get_num_env_entries();
+  fprintf(stderr, "current_entries: %lu\n", current_entries);
+  size_t entries_to_add = (kMaxNumEnvPEntries - current_entries) + 1;
+  fprintf(stderr, "Adding %lu entries\n", entries_to_add);
+
+  // Add a large number of entries. In particular more than the sanitizer
+  // run time allows for.
+  char buffer[256];
+  for (int count = 0; count < entries_to_add; ++count) {
+    int chars_written = snprintf(buffer, sizeof(buffer), "__UNLIKELY_ENV_VAR__%d", count);
+    assert(chars_written);
+    if (setenv(buffer, "some_value", /*overwrite*/ 1)) {
+      fprintf(stderr, "Failed to write \"%s\" to env\n", buffer);
+      return 1;
+    }
+  }
+  current_entries = get_num_env_entries();
+  assert(current_entries == kMaxNumEnvPEntries + 1);
+
+  // Trigger symbolization.
+  // CHECK: WARNING: Truncating symbolizer environment. Needed 512 entries but only 511 are available.
+  // CHECK: #0{{( *0x.* *in *)?}}  __sanitizer_print_stack_trace
+  // CHECK: #1{{( *0x.* *in *)?}} main {{.*}}symbolizer-large-env.cpp:[[@LINE+1]]
+  __sanitizer_print_stack_trace();
+
+  return 0;
+}