[sanitizer] Don't reuse the main thread in ThreadRegistry

[kubamracek]

Summary

There is a hard-to-reproduce crash happening on OS X that involves terminating the main thread (dispatch_main does that, see discussion at http://reviews.llvm.org/D18496) and later reusing the main thread's ThreadContext. This patch disables reuse of the main thread. I believe this problem exists only on OS X, because on other systems the main thread cannot be terminated without exiting the process.

[dvyukov]

Why is it difficult to reproduce? The quarantine if FIFO queue of size 16 IIRC. So if you create/destroy 16 threads, next thread creation will reuse the oldest thread context. Or the crash does not always happen after reuse?
I am hinting on a test.

[kubamracek]

The crash happens with GCD worker threads. The only way to wait for a worker thread to be destroyed that I know of, is using long sleep()s and even then it's non-deterministic and the actual delays differ in different OS versions.

[dvyukov]

If you can make the test crash once in 100 runs that's still better than no test. If we have a regression, the failure will be detected eventually (on bots or in manual test runs).

[kubamracek]

The test would need to sleep for a long time (~30 seconds total). Is that okay?

[dvyukov]

No, it is not OK. How does the look like?
Note to get thread id reuse, you don't need to use GCD worker threads, you can create 16 normal threads, ensure that they started and join them.

[kubamracek]

In D19722#418402, @dvyukov wrote:

No, it is not OK. How does the look like?
Note to get thread id reuse, you don't need to use GCD worker threads, you can create 16 normal threads, ensure that they started and join them.

The crash only happens when the main thread is reused as a worker thread. It's hard to trigger that, since you can't control the creation and termination of worker threads. I don't know how would throwing in some regular pthreads help.

[dvyukov]

OK, let's leave it as is.

compiler-rt/trunk/lib/sanitizer_common/sanitizer_thread_registry.cc

void ThreadRegistry::StartThread(u32 tid, uptr os_id, void *arg) {
   CHECK_LT(tid, n_contexts_);
   ThreadContextBase *tctx = threads_[tid];
   CHECK_NE(tctx, 0);
   CHECK_EQ(ThreadStatusCreated, tctx->status);
   tctx->SetStarted(os_id, arg);
 }
 
 void ThreadRegistry::QuarantinePush(ThreadContextBase *tctx) {
+  if (tctx->tid == 0)
+    return;  // Don't reuse the main thread.  It's a special snowflake.
   dead_threads_.push_back(tctx);
   if (dead_threads_.size() <= thread_quarantine_size_)
     return;
   tctx = dead_threads_.front();
   dead_threads_.pop_front();
   CHECK_EQ(tctx->status, ThreadStatusDead);
   tctx->Reset();
   tctx->reuse_count++;