This is an archive of the discontinued LLVM Phabricator instance.

[Darwin] Fix symbolization for recent simulator runtimes.
ClosedPublic

Authored by delcypher on Apr 14 2020, 10:46 PM.

Details

Summary

Due to sandbox restrictions in the recent versions of the simulator runtime the
atos program is no longer able to access the task port of a parent process
without additional help.

This patch fixes this by registering a task port for the parent process
before spawning atos and also tells atos to look for this by setting
a special environment variable.

This patch is based on an Apple internal fix (rdar://problem/43693565) that
unfortunately contained a bug (rdar://problem/58789439) because it used
setenv() to set the special environment variable. This is not safe because in
certain circumstances this can trigger a call to realloc() which can fail
during symbolization leading to deadlock. A test case is included that captures
this problem.

This patch takes a different approach by

  1. Calling putenv() early during process init (but late enough that

malloc/realloc works) to set a dummy value for the environment variable.

  1. Just before atos is spawned the storage for the environment

variable is modified to contain the correct PID.

The issue reported in rdar://problem/58789439 manifested as a deadlock
during symbolization in the following situation:

  1. Before TSan detects an issue Something calls setenv() that sets a

new environment variable that wasn't previously set. This triggers a
call to malloc() to allocate a new environment array. This uses TSan's
normal user-facing allocator. LibC stores this pointer for future use
later.

  1. TSan detects an issue and tries to launch the symbolizer. When we are in the

symbolizer we switch to a different (internal allocator) and then we call
setenv() to set a new environment variable. When this happen setenv() sees
that it needs to make the environment array larger and calls realloc() on the
existing enviroment array because it remembers that it previously allocated
memory for it. Calling realloc() fails here because it is being called on a
pointer its never seen before.

The included test case closely reproduces the originally reported
problem but it doesn't replicate the `((kBlockMagic)) ==
((((u64*)addr)[0])` assertion failure exactly. This is due to the way
TSan's normal allocator allocates the environment array the first time
it is allocated. In the test program addr[0] accesses an inaccessible
page and raises SIGBUS. If TSan's SIGBUS signal handler is active, the
signal is caught and symbolication is attempted again which results in
deadlock.

In the originally reported problem the pointer is successfully derefenced but
then the assert fails due to the provided pointer not coming from the active
allocator. When the assert fails TSan tries to symbolicate the stacktrace while
already being in the middle of symbolication which results in deadlock.

rdar://problem/58789439

Diff Detail

Event Timeline

delcypher created this revision.Apr 14 2020, 10:46 PM
Herald added a project: Restricted Project. · View Herald TranscriptApr 14 2020, 10:46 PM
Herald added subscribers: Restricted Project, jfb. · View Herald Transcript
delcypher updated this revision to Diff 257749.Apr 15 2020, 9:21 AM
  • Detect env var being messed with gracefully.
  • Small clean up.
kubamracek added inline comments.Apr 15 2020, 10:19 AM
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
32

Can you drop these non-essential changes, please?

53

Can you sink this into the class please?

74

Just for readability, please move the putenv call outside of the CHECK.

102

I'd rather not call getenv here too. We can just access atos_mach_port_env_var_entry_ directly, no? Also, why not just unconditionally call updateAtosEnvVar?

108

All methods here in this class start with uppercase letter. Let's keep it that way, please.

247

Who's calling this?

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.h
38

I don't see a virtual LateInitialize in SymbolizerTool... is this supposed to not be an "override"?

delcypher marked 5 inline comments as done.Apr 15 2020, 11:37 AM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
32

Sorry clang-format changed this and I didn't notice.

74

Sure.

102

The reason is that I wanted to check our assumption is that the __check_mach_ports_lookup environment variable is still in the environment and that we still control it.

This assumption could be broken by the user's code changing the environment.

If you don't care about detecting this case so that we can fail more gracefully then I can rip this out which also means I can rip out getValueFromEnvEntry() as well.

108

Oops I missed this. I'll fix that.

247

This will be called via Symbolizer::LateInitialize()

Please see the patch (https://reviews.llvm.org/D78178) that this patch depends on.

delcypher marked an inline comment as done.Apr 15 2020, 11:43 AM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
53

If I try to sink this into the class then this doesn't compile.

/Volumes/data/dev/llvm/monorepo_upstream/master/llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp:152:21: error: in-class initializer for static data member of type 'const char [26]' requires 'constexpr' specifier
  static const char kAtosEnvVar[] = "__check_mach_ports_lookup";
                    ^               ~~~~~~~~~~~~~~~~~~~~~~~~~~~
  constexpr

If I make it constexpr then we'll get linking error because we need the memory to exist at runtime.

Undefined symbols for architecture i386:
  "__sanitizer::AtosSymbolizerProcess::kAtosEnvVar", referenced from:
      __sanitizer::AtosSymbolizer::AtosSymbolizer(char const*, __sanitizer::LowLevelAllocator*) in sanitizer_symbolizer_mac.cpp.o
      __sanitizer::AtosSymbolizer::AtosSymbolizer(char const*, __sanitizer::LowLevelAllocator*) in sanitizer_symbolizer_mac.cpp.o
      __sanitizer::AtosSymbolizerProcess::StartSymbolizerSubprocess() in sanitizer_symbolizer_mac.cpp.o

To fix this I'd have initialize the static data member outside the class which means I'd have to do...

// In class
static const char kAtosEnvVar[26];

// Outside of class
const char AtosSymbolizerProcess::kAtosEnvVar[] = "__check_mach_ports_lookup";

This seems clumsy to me because I have to hardcode the 26 size.

delcypher marked an inline comment as done.Apr 15 2020, 11:44 AM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.h
38

Please see the patch that this patch depends on: https://reviews.llvm.org/D78178

kubamracek added inline comments.Apr 15 2020, 1:09 PM
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
102

Yes, let's not worry about the user overwriting that env var.

delcypher marked 2 inline comments as done.Apr 15 2020, 7:28 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
53

Do you have a preference between the global var and the clumsy static data member?

102

Are you have for me to drop the check of the assumption?

delcypher updated this revision to Diff 257942.Apr 15 2020, 7:36 PM
  • Address first round of feedback.

@kubamracek Any additional comments?

delcypher updated this revision to Diff 257943.Apr 15 2020, 7:40 PM

Correct typo in comment.

delcypher updated this revision to Diff 257945.Apr 15 2020, 7:43 PM
  • Update radar number.
delcypher edited the summary of this revision. (Show Details)Apr 15 2020, 7:45 PM
Harbormaster failed remote builds in B53494: Diff 257943!
Harbormaster failed remote builds in B53496: Diff 257945!
yln added inline comments.Apr 16 2020, 9:52 AM
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
68

What's our stance on if(<defined-value>) vs #ifdef? In compiler-rt we seem to usually use #ifdef. Should we aim for consistency here?

69

Thanks for the explanation of why this is needed! :)

102

If you remove the unnecessary call in the constructor, then this is the only usage of the function and we can inline it.

138

I think you can get away without the defining kAtosEnvVar_ at all:
Here:
char atos_mach_port_env_var_entry_[sizeof(K_ATOS_ENV_VAR) + sizeof(pid_str_)];
Above: make it part of the format string:
internal_snprintf(.., .., K_ATOS_ENV_VAR "=%s", pid_str);

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.h
38

Just a quick question: will the principled resolution (where we modify the env only for posix_spawn, but not in our own process) require this as well. If not, is the plan to remove it again?

delcypher marked 2 inline comments as done.Apr 16 2020, 12:46 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
68

SANITIZER_IOSSIM is always defined to be 0 or 1. I thought it was better than this code be compiled on all paths because not everyone builds for iOS simulators in open source.

102

@yln

If you remove the unnecessary call in the constructor, then this is the only usage of the function and we can inline it.

The call in the constructor is essential. When you call putenv() Darwin's LibC checks the data is well formed so we have to initialize atos_mach_port_env_var_entry_ before giving it to putenv().

Later on we need to modify atos_mach_port_env_var_entry_ again to have it contain the right PID so it made sense to refactor this repeated action into UpdateAtosEnvVar().

delcypher marked an inline comment as done.Apr 16 2020, 12:53 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
138

@yln
Originally this was a lot cleaner. It was just this at global scope.

static const char kAtosEnvVar[] = "__check_mach_ports_lookup";

However @kubamracek wanted this sunk into the class itself. This resulted in the mess that's in this current patch. I much preferred how it was before because I didn't need to use macros at all.

The approach you've laid out is a definite improvement over what's currently in the patch but it still involves macros.

@kubamracek @yln Can we pick something? Either what @yln is suggesting or my original approach (global variable). I prefer the original approach because it doesn't involve any macros.

delcypher marked an inline comment as done.Apr 16 2020, 1:27 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
102

@yln Sorry a little bit more on this. The call to UpdateAtosEnvVar() is essential but it doesn't actually need to be in the constructor. We could move it into LateInitialize() to set atos_mach_port_env_var_entry_ just before we give it to putenv(). Is that what you'd prefer?

yln added inline comments.Apr 16 2020, 1:54 PM
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
102

How about the following?

Initialize to default that is good enough to use for putenv():

// Comment about '\0' and '='
char AtosSymbolizerProcess::atos_mach_port_env_var_entry_[sizeof(K_ATOS_ENV_VAR) + sizeof(pid_str_)] = K_ATOS_ENV_VAR "=0";

Do nothing in the ctor (UpdateAtosEnvVar can be inlined).

UpdateAtosEnvVar() becomes:

dst = &atos_mach_port_env_var_entry_[sizeof(K_ATOS_ENV_VAR)];
strlcpy(dst, pid_str, ...);

Nit: use less verbose name for atos_mach_port_env_var_entry_

Double check the buffer sizes/string offsets/space for =...

  • Adopt Julian's macro suggestions.
  • Inline UpdateAtosEnvVar().
  • atos_mach_port_env_var_entry_ -> mach_port_env_var_entry_
delcypher marked 3 inline comments as done.Apr 17 2020, 11:55 AM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
102

I tried doing this with constexpr but I gave up as I would need to be able to concatenate strings and I'd have to implement a bunch of infrastructure to do that. So I went with your macro approach.

delcypher marked 4 inline comments as done.Apr 17 2020, 12:03 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.h
38

The principled resolution does not depend on https://reviews.llvm.org/D78178.
However, I'm abandoning the principled resolution due to complexity involved in copying the environment and modifying the copy. I had it all implemented but it needs a lot of clean up for adoption in open source which I don't have time to do.

yln accepted this revision.Apr 17 2020, 1:28 PM

LGTM, with two more tiny nits.

compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
105

We could assert "equal" on the exact count sizeof(K_ATOS_ENV_VAR) + 1 + internal_strlen(pid_str_)

143

Is this considered good hygiene? Personally, I would drop it, but I am fine either way.

This revision is now accepted and ready to land.Apr 17 2020, 1:28 PM
delcypher updated this revision to Diff 258435.Apr 17 2020, 3:01 PM
  • Make check on return of internal_snprintf() tighter.
delcypher marked 3 inline comments as done.Apr 17 2020, 3:02 PM
delcypher added inline comments.
compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_mac.cpp
105

Done. You don't need the +1 though .

143

I consider it good hygiene for the macro to be defined for only as long as its needed.

delcypher marked an inline comment as done.Apr 17 2020, 3:03 PM
kubamracek accepted this revision.Apr 17 2020, 3:07 PM
This revision was automatically updated to reflect the committed changes.