[libcxxabi] Refactor pthread usage into a separate API

[rmaprath]

Summary

This is simply a cleanup of D18482 (patch taken with permission) while adapting it to match what we have already implemented for libcxx.

Note that I haven't included the external-threading stuff yet, this is only refactoring pthread usage of libcxxabi. A follow-up patch will add the external-threading support.

[rmaprath]

Use __internal_linkage__ attribute to control visibility following D24642.

[compnerd]

src/config.h

41

Can you change this to defined(__sun__) && defined(_POSIX_THREADS) at the very least? There is no reason to assume pthreads on Solaris. Solaris Threads are a valid threading model.

42

Can we use _LIBCXXABI_USE_THREAD_API_PTHREAD instead? You can have more than one threading API on a platform, and wish to use a specific one.

46

I really think that we should use _POSIX_THREADS macro rather than this enumeration approach.

test/test_fallback_malloc.pass.cpp

13

Can we add the include directory as a header search path for the tests? Seems it may be nicer, but, obviously, we can do that later.

[EricWF]

LGTM other than the inline comments. I'll give it a final once over tonight or tomorrow.

include/__cxxabi_config.h

36 ↗

(On Diff #72478)

Does this configuration need to be in a public header if it's only used internally? I think config.h or threading_support.h would be a better place for this to live.

src/config.h

22

What's the point of defining _LIBCXXABI_WARNING? It's unused and seems unneeded.

src/fallback_malloc.ipp

30 ↗

(On Diff #72478)

s/PTHREAD_MUTEX_INITIALIZER/_LIBCXXABI_MUTEX_INITIALIZER

[rmaprath]

Address review comments from @compnerd and @EricWF.

[rmaprath]

src/config.h

22

Good catch. I have copy-pasted this lot without checking.

41

Not sure if I understand completely, libcxxabi currently either supports pthread or no-threads, either of which can be explicitly configured through a cmake option. So I don't see the point in checking for _POSIX_THREADS here as the only current threading option is pthread. When we add another threading system into the mix (like external threading), that too will be configurable with a cmake option, the defaulting to pthread is only when no particular threading system is selected.

Or is it that pthread.h et. al can be provided by Solaris Threads as well? So, the pthread API has two implementations on Solaris?

42

Makes sense. Need to change the convention used in libcxx too, but that can be done later.

46

_LIBCXXABI_USE_THREAD_API_XXXX allows us to be consistent with naming of the different threading variants (e.g. I'm going to add _LIBCXXABI_USE_THREAD_API_EXTENRAL soon). Is there some functional benefit to using _POSIX_THREADS for the pthread variant?

src/fallback_malloc.ipp

30 ↗

(On Diff #72653)

Thanks for the catch.

[rmaprath]

@EricWF: Gentle ping.

[EricWF]

LGTM once the _POSIX_THREADS comment is addressed.

src/config.h

46

I think what @compnerd means is that _POSIX_THREADS can be used to detect if the current platform supports/provides pthreads.h. So instead of enumerating the OS's that support them manually we can just replace that check with defined(_POSIX_THREADS) after including unistd.h.

[rmaprath]

Updated with the following changes:

• Address review comments from @EricWF and @compnerd regarding using _POSIX_THREADS for detecting pthread availability. Now the patch is checking for defined(_POSIX_THREADS) && _POSIX_THREADS >= 0 instead of enumerating the platforms.
• Fix a missed pthread dependency in cxa_guard.cpp. The call to pthread_mach_thread_np is now replaced with __libcxxabi_thread_get_current_id().
• Fix a couple of missing pthread dependencies in test_exception_storage.cpp (see patch below).

@EricWF: Hope the new changes look OK? I'm going to test this on a Mac before committing.

I will also ping D21803 and D17815, as they'll be affected by this patch (those patches add more pthread calls).

[rmaprath]

In D24864#556845, @rmaprath wrote:

I'm going to test this on a Mac before committing.

Good call that was, wouldn't have compiled there. Attaching updated patch soon.

[rmaprath]

pthread_mach_thread_np is Mac specific. I've introduced a Mac-only __libcxxabi_thread_get_port() API call to sort this out (didn't think it would be nice to leave a pthread call even if it is Mac specific).

I can revisit this if there is a better approach. Will commit now.

[rmaprath]

Sigh. I've bumped into some downstream problems. Will commit as soon as I've verified these problems are not related to the patch.

[rmaprath]

Patch re-based on the latest trunk.

I've resolved my downstream issues, will be committing soon.

/ Asiri

[rmaprath]

Committed as r284128.

[vitalybuka]

This change breaks http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux/builds/25625/steps/build%2032-bit%20symbolizer%20for%20compiler_rt_build/logs/stdio

[vitalybuka]

So there is:

• Looking for __cxa_thread_atexit_impl in c
• Looking for __cxa_thread_atexit_impl in c - not found

and libcxx is configured with -DLIBCXX_ENABLE_THREADS=OFF

[rmaprath]

In D24864#570924, @vitalybuka wrote:

So there is:

• Looking for __cxa_thread_atexit_impl in c
• Looking for __cxa_thread_atexit_impl in c - not found

and libcxx is configured with -DLIBCXX_ENABLE_THREADS=OFF

I think, the problem here is that cxa_thread_atexit.cpp is not properly guarded against the non-threaded use-case. If you look at the source prior to this patch, it refers to pthread.h and pthread_key_t unconditionally, it may have worked before because pthread got somehow dragged in.

The fix should be fairly straightforward. I will do a patch tomorrow (bit late in the day here), hope that's OK?

Cheers,

/ Asiri

[vitalybuka]

Maybe?

• if (UNIX AND NOT (APPLE OR CYGWIN))

+ if (LIBCXXABI_ENABLE_THREADS AND UNIX AND NOT (APPLE OR CYGWIN))

list(APPEND LIBCXXABI_SOURCES cxa_thread_atexit.cpp)

endif()

[rmaprath]

In D24864#570935, @vitalybuka wrote:

Maybe?

- if (UNIX AND NOT (APPLE OR CYGWIN))
+ if (LIBCXXABI_ENABLE_THREADS AND UNIX AND NOT (APPLE OR CYGWIN))
list(APPEND LIBCXXABI_SOURCES cxa_thread_atexit.cpp)
endif()

Yes!

I was just about to say I don't know why it's even trying to build this, given that the library is configured with -DLIBCXXABI_ENABLE_THREADS=OFF.

Feel free to commit a fix :)

Cheers,

/ Asiri

[vitalybuka]

Thanks, done D25636

[rmaprath]

In D24864#570954, @vitalybuka wrote:

Thanks, done D25636

Thanks for the fix!

/ Asiri

CMakeLists.txt

                         " be set to ON when LIBCXXABI_ENABLE_THREADS"
                         " is also set to ON.")
   endif()
   add_definitions(-D_LIBCXXABI_HAS_NO_THREADS)
 endif()
 
 if (LIBCXXABI_HAS_PTHREAD_API)
   add_definitions(-D_LIBCPP_HAS_THREAD_API_PTHREAD)
+  add_definitions(-D_LIBCXXABI_USE_THREAD_API_PTHREAD)
 endif()
 
 if (MSVC)
   add_definitions(-D_CRT_SECURE_NO_WARNINGS)
 endif()
 
 # Define LIBCXXABI_USE_LLVM_UNWINDER for conditional compilation.
 if (LIBCXXABI_USE_LLVM_UNWINDER OR LLVM_NATIVE_ARCH MATCHES ARM)

src/config.h

 //===----------------------------------------------------------------------===//
 
 
 #ifndef LIBCXXABI_CONFIG_H
 #define LIBCXXABI_CONFIG_H
 
 #include <unistd.h>
 
+// Configure inline visibility attributes
+#if defined(_WIN32)
+ #if defined(_MSC_VER) && !defined(__clang__)
+  // Using Microsoft Visual C++ compiler
+  #define _LIBCXXABI_INLINE_VISIBILITY __forceinline
+ #else
+  #if __has_attribute(__internal_linkage__)
+   #define _LIBCXXABI_INLINE_VISIBILITY __attribute__ ((__internal_linkage__, __always_inline__))
+  #else
+   #define _LIBCXXABI_INLINE_VISIBILITY __attribute__ ((__always_inline__))
+  #endif
+ #endif
+#else
+ #if __has_attribute(__internal_linkage__)
+  #define _LIBCXXABI_INLINE_VISIBILITY __attribute__ ((__internal_linkage__, __always_inline__))
+ #else
+  #define _LIBCXXABI_INLINE_VISIBILITY __attribute__ ((__visibility__("hidden"), __always_inline__))
+ #endif
+#endif
+
+// Try and deduce a threading api if one has not been explicitly set.
+#if !defined(_LIBCXXABI_HAS_NO_THREADS) && \
+    !defined(_LIBCXXABI_USE_THREAD_API_PTHREAD)
+  #if defined(_POSIX_THREADS) && _POSIX_THREADS >= 0
+    #define _LIBCXXABI_USE_THREAD_API_PTHREAD
+  #else
+    #error "No thread API"
+  #endif
+#endif
+
 // Set this in the CXXFLAGS when you need it, because otherwise we'd have to
 // #if !defined(__linux__) && !defined(__APPLE__) && ...
 // and so-on for *every* platform.
 #ifndef LIBCXXABI_BAREMETAL
 #  define LIBCXXABI_BAREMETAL 0
 #endif
 
 #endif // LIBCXXABI_CONFIG_H

src/cxa_exception.cpp

 //===------------------------- cxa_exception.cpp --------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //  
 //  This file implements the "Exception Handling APIs"
 //  http://mentorembedded.github.io/cxx-abi/abi-eh.html
 //  
 //===----------------------------------------------------------------------===//
 
 #include "config.h"
+#include "threading_support.h"
 #include "cxxabi.h"
 
 #include <exception>        // for std::terminate
 #include <cstring>          // for memset
 #include "cxa_exception.hpp"
 #include "cxa_handlers.hpp"
 #include "fallback_malloc.h"
 

src/cxa_exception_storage.cpp

 //===--------------------- cxa_exception_storage.cpp ----------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //  
 //  This file implements the storage for the "Caught Exception Stack"
 //  http://mentorembedded.github.io/cxx-abi/abi-eh.html (section 2.2.2)
 //  
 //===----------------------------------------------------------------------===//
 
 #include "cxa_exception.hpp"
 
 #include "config.h"
+#include "threading_support.h"
 
 #if defined(_LIBCXXABI_HAS_NO_THREADS)
 
 namespace __cxxabiv1 {
 extern "C" {
     static __cxa_eh_globals eh_globals;
     __cxa_eh_globals *__cxa_get_globals() { return &eh_globals; }
     __cxa_eh_globals *__cxa_get_globals_fast() { return &eh_globals; }

 extern "C" {
     __cxa_eh_globals * __cxa_get_globals      () { return __globals (); }
     __cxa_eh_globals * __cxa_get_globals_fast () { return __globals (); }
     }
 }
 
 #else
 
-#include <pthread.h>
 #include "abort_message.h"
 #include "fallback_malloc.h"
 
-//  In general, we treat all pthread errors as fatal.
+//  In general, we treat all threading errors as fatal.
 //  We cannot call std::terminate() because that will in turn
 //  call __cxa_get_globals() and cause infinite recursion.
 
 namespace __cxxabiv1 {
 namespace {
-    pthread_key_t  key_;
-    pthread_once_t flag_ = PTHREAD_ONCE_INIT;
+    __libcxxabi_tls_key key_;
+    __libcxxabi_exec_once_flag flag_ = _LIBCXXABI_EXEC_ONCE_INITIALIZER;
 
     void destruct_ (void *p) {
         __free_with_fallback ( p );
-        if ( 0 != ::pthread_setspecific ( key_, NULL ) ) 
+        if ( 0 != __libcxxabi_tls_set ( key_, NULL ) )  
             abort_message("cannot zero out thread value for __cxa_get_globals()");
         }
 
     void construct_ () {
-        if ( 0 != pthread_key_create ( &key_, destruct_ ) )
-            abort_message("cannot create pthread key for __cxa_get_globals()");
+        if ( 0 != __libcxxabi_tls_create ( &key_, destruct_ ) )
+            abort_message("cannot create thread specific key for __cxa_get_globals()");
         }
 }   
 
 extern "C" {
     __cxa_eh_globals * __cxa_get_globals () {
     //  Try to get the globals for this thread
         __cxa_eh_globals* retVal = __cxa_get_globals_fast ();
     
     //  If this is the first time we've been asked for these globals, create them
         if ( NULL == retVal ) {
             retVal = static_cast<__cxa_eh_globals*>
                         (__calloc_with_fallback (1, sizeof (__cxa_eh_globals)));
             if ( NULL == retVal )
                 abort_message("cannot allocate __cxa_eh_globals");
-            if ( 0 != pthread_setspecific ( key_, retVal ) )
-               abort_message("pthread_setspecific failure in __cxa_get_globals()");
+            if ( 0 != __libcxxabi_tls_set ( key_, retVal ) )
+               abort_message("__libcxxabi_tls_set failure in __cxa_get_globals()");
            }
         return retVal;
         }
 
     // Note that this implementation will reliably return NULL if not
     // preceded by a call to __cxa_get_globals().  This is an extension
     // to the Itanium ABI and is taken advantage of in several places in
     // libc++abi.
     __cxa_eh_globals * __cxa_get_globals_fast () {
     //  First time through, create the key.
-        if (0 != pthread_once(&flag_, construct_))
-            abort_message("pthread_once failure in __cxa_get_globals_fast()");
+        if (0 != __libcxxabi_execute_once(&flag_, construct_))
+            abort_message("execute once failure in __cxa_get_globals_fast()");
 //        static int init = construct_();
-        return static_cast<__cxa_eh_globals*>(::pthread_getspecific(key_));
+        return static_cast<__cxa_eh_globals*>(__libcxxabi_tls_get(key_));
         }
     
 }
 }
 #endif

src/cxa_guard.cpp

 //===---------------------------- cxa_guard.cpp ---------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "__cxxabi_config.h"
 
 #include "abort_message.h"
 #include "config.h"
+#include "threading_support.h"
 
-#ifndef _LIBCXXABI_HAS_NO_THREADS
-#  include <pthread.h>
-#endif
 #include <stdint.h>
 
 /*
     This implementation must be careful to not call code external to this file
     which will turn around and try to call __cxa_guard_acquire reentrantly.
     For this reason, the headers of this file are as restricted as possible.
     Previous implementations of this code for __APPLE__ have used
-    pthread_mutex_lock and the abort_message utility without problem.  This
-    implementation also uses pthread_cond_wait which has tested to not be a
-    problem.
+    __libcxxabi_mutex_lock and the abort_message utility without problem.  This
+    implementation also uses __libcxxabi_condvar_wait which has tested
+    to not be a problem.
 */
 
 namespace __cxxabiv1
 {
 
 namespace
 {
 
bool is_initialized(guard_type* guard_object) {
     char* initialized = (char*)guard_object;
     return *initialized;
 }
 
 #endif
 #endif
 
 #ifndef _LIBCXXABI_HAS_NO_THREADS
-pthread_mutex_t guard_mut = PTHREAD_MUTEX_INITIALIZER;
-pthread_cond_t  guard_cv  = PTHREAD_COND_INITIALIZER;
+__libcxxabi_mutex_t guard_mut = _LIBCXXABI_MUTEX_INITIALIZER;
+__libcxxabi_condvar_t guard_cv = _LIBCXXABI_CONDVAR_INITIALIZER;
 #endif
 
 #if defined(__APPLE__) && !defined(__arm__)
 
 typedef uint32_t lock_type;
 
 #if __LITTLE_ENDIAN__
 

 }  // unnamed namespace
 
 extern "C"
 {
 
 #ifndef _LIBCXXABI_HAS_NO_THREADS
 _LIBCXXABI_FUNC_VIS int __cxa_guard_acquire(guard_type *guard_object) {
     char* initialized = (char*)guard_object;
-    if (pthread_mutex_lock(&guard_mut))
+    if (__libcxxabi_mutex_lock(&guard_mut))
         abort_message("__cxa_guard_acquire failed to acquire mutex");
     int result = *initialized == 0;
     if (result)
     {
 #if defined(__APPLE__) && !defined(__arm__)
-        const lock_type id = pthread_mach_thread_np(pthread_self());
+        const lock_type id = __libcxxabi_thread_get_port();
         lock_type lock = get_lock(*guard_object);
         if (lock)
         {
             // if this thread set lock for this same guard_object, abort
             if (lock == id)
                 abort_message("__cxa_guard_acquire detected deadlock");
             do
             {
-                if (pthread_cond_wait(&guard_cv, &guard_mut))
+                if (__libcxxabi_condvar_wait(&guard_cv, &guard_mut))
                     abort_message("__cxa_guard_acquire condition variable wait failed");
                 lock = get_lock(*guard_object);
             } while (lock);
             result = !is_initialized(guard_object);
             if (result)
                 set_lock(*guard_object, id);
         }
         else
             set_lock(*guard_object, id);
 #else  // !__APPLE__ || __arm__
         while (get_lock(*guard_object))
-            if (pthread_cond_wait(&guard_cv, &guard_mut))
+            if (__libcxxabi_condvar_wait(&guard_cv, &guard_mut))
                 abort_message("__cxa_guard_acquire condition variable wait failed");
         result = *initialized == 0;
         if (result)
             set_lock(*guard_object, true);
 #endif  // !__APPLE__ || __arm__
     }
-    if (pthread_mutex_unlock(&guard_mut))
+    if (__libcxxabi_mutex_unlock(&guard_mut))
         abort_message("__cxa_guard_acquire failed to release mutex");
     return result;
 }
 
 _LIBCXXABI_FUNC_VIS void __cxa_guard_release(guard_type *guard_object) {
-    if (pthread_mutex_lock(&guard_mut))
+    if (__libcxxabi_mutex_lock(&guard_mut))
         abort_message("__cxa_guard_release failed to acquire mutex");
     *guard_object = 0;
     set_initialized(guard_object);
-    if (pthread_mutex_unlock(&guard_mut))
+    if (__libcxxabi_mutex_unlock(&guard_mut))
         abort_message("__cxa_guard_release failed to release mutex");
-    if (pthread_cond_broadcast(&guard_cv))
+    if (__libcxxabi_condvar_broadcast(&guard_cv))
         abort_message("__cxa_guard_release failed to broadcast condition variable");
 }
 
 _LIBCXXABI_FUNC_VIS void __cxa_guard_abort(guard_type *guard_object) {
-    if (pthread_mutex_lock(&guard_mut))
+    if (__libcxxabi_mutex_lock(&guard_mut))
         abort_message("__cxa_guard_abort failed to acquire mutex");
     *guard_object = 0;
-    if (pthread_mutex_unlock(&guard_mut))
+    if (__libcxxabi_mutex_unlock(&guard_mut))
         abort_message("__cxa_guard_abort failed to release mutex");
-    if (pthread_cond_broadcast(&guard_cv))
+    if (__libcxxabi_condvar_broadcast(&guard_cv))
         abort_message("__cxa_guard_abort failed to broadcast condition variable");
 }
 
 #else // _LIBCXXABI_HAS_NO_THREADS
 
 _LIBCXXABI_FUNC_VIS int __cxa_guard_acquire(guard_type *guard_object) {
     return !is_initialized(guard_object);
 }

src/cxa_thread_atexit.cpp

 //===----------------------- cxa_thread_atexit.cpp ------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "abort_message.h"
 #include "cxxabi.h"
+#include "threading_support.h"
 #include <cstdlib>
-#include <pthread.h>
 
 namespace __cxxabiv1 {
 
   using Dtor = void(*)(void*);
 
   extern "C"
 #ifndef HAVE___CXA_THREAD_ATEXIT_IMPL
   // A weak symbol is used to detect this function's presence in the C library
namespace {
   // - dso_symbol is ignored.  This means that a shared library may be unloaded
   //   (via dlclose()) before its thread_local destructors have run.
   //
   // - thread_local destructors for the main thread are run by the destructor of
   //   a static object.  This is later than expected; they should run before the
   //   destructors of any objects with static storage duration.
   //
   // - thread_local destructors on non-main threads run on the first iteration
-  //   through the pthread_key destructors.  std::notify_all_at_thread_exit()
-  //   and similar functions must be careful to wait until the second iteration
-  //   to provide their intended ordering guarantees.
+  //   through the __libcxxabi_tls_key destructors.
+  //   std::notify_all_at_thread_exit() and similar functions must be careful to
+  //   wait until the second iteration to provide their intended ordering
+  //   guarantees.
   //
   // Another limitation, though one shared with ..._impl(), is that any
   // thread_locals that are first initialized after non-thread_local global
   // destructors begin to run will not be destroyed.  [basic.start.term] states
   // that all thread_local destructors are sequenced before the destruction of
   // objects with static storage duration, resulting in a contradiction if a
   // thread_local is constructed after that point.  Thus we consider such
   // programs ill-formed, and don't bother to run those destructors.  (If the
   // program terminates abnormally after such a thread_local is constructed,
   // the destructor is not expected to run and thus there is no contradiction.
   // So construction still has to work.)
 
   struct DtorList {
     Dtor dtor;
     void* obj;
     DtorList* next;
   };
 
   // The linked list of thread-local destructors to run
   __thread DtorList* dtors = nullptr;
   // True if the destructors are currently scheduled to run on this thread
   __thread bool dtors_alive = false;
   // Used to trigger destructors on thread exit; value is ignored
-  pthread_key_t dtors_key;
+  __libcxxabi_tls_key dtors_key;
 
   void run_dtors(void*) {
     while (auto head = dtors) {
       dtors = head->next;
       head->dtor(head->obj);
       std::free(head);
     }
 
     dtors_alive = false;
   }
 
   struct DtorsManager {
     DtorsManager() {
-      // There is intentionally no matching pthread_key_delete call, as
+      // There is intentionally no matching __libcxxabi_tls_delete call, as
       // __cxa_thread_atexit() may be called arbitrarily late (for example, from
       // global destructors or atexit() handlers).
-      if (pthread_key_create(&dtors_key, run_dtors) != 0) {
-        abort_message("pthread_key_create() failed in __cxa_thread_atexit()");
+      if (__libcxxabi_tls_create(&dtors_key, run_dtors) != 0) {
+        abort_message("__libcxxabi_tls_create() failed in __cxa_thread_atexit()");
       }
     }
 
     ~DtorsManager() {
-      // pthread_key destructors do not run on threads that call exit()
+      // __libcxxabi_tls_key destructors do not run on threads that call exit()
       // (including when the main thread returns from main()), so we explicitly
       // call the destructor here.  This runs at exit time (potentially earlier
       // if libc++abi is dlclose()'d).  Any thread_locals initialized after this
       // point will not be destroyed.
       run_dtors(nullptr);
     }
   };
 } // namespace
 
 #endif // HAVE__CXA_THREAD_ATEXIT_IMPL
 
 extern "C" {
 
   _LIBCXXABI_FUNC_VIS int __cxa_thread_atexit(Dtor dtor, void* obj, void* dso_symbol) throw() {
 #ifdef HAVE___CXA_THREAD_ATEXIT_IMPL
     return __cxa_thread_atexit_impl(dtor, obj, dso_symbol);
 #else
     if (__cxa_thread_atexit_impl) {
       return __cxa_thread_atexit_impl(dtor, obj, dso_symbol);
     } else {
-      // Initialize the dtors pthread_key (uses __cxa_guard_*() for one-time
-      // initialization and __cxa_atexit() for destruction)
+      // Initialize the dtors __libcxxabi_tls_key (uses __cxa_guard_*() for
+      // one-time initialization and __cxa_atexit() for destruction)
       static DtorsManager manager;
 
       if (!dtors_alive) {
-        if (pthread_setspecific(dtors_key, &dtors_key) != 0) {
+        if (__libcxxabi_tls_set(dtors_key, &dtors_key) != 0) {
           return -1;
         }
         dtors_alive = true;
       }
 
       auto head = static_cast<DtorList*>(std::malloc(sizeof(DtorList)));
       if (!head) {
         return -1;

src/fallback_malloc.cpp

 //===------------------------ fallback_malloc.cpp -------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "fallback_malloc.h"
 
 #include "config.h"
+#include "threading_support.h"
 
 #include <cstdlib> // for malloc, calloc, free
 #include <cstring> // for memset
 
-#ifndef _LIBCXXABI_HAS_NO_THREADS
-#include <pthread.h> // for mutexes
-#endif
-
 //  A small, simple heap manager based (loosely) on
 //  the startup heap manager from FreeBSD, optimized for space.
 //
 //  Manages a fixed-size memory pool, supports malloc and free only.
 //  No support for realloc.
 //
 //  Allocates chunks in multiples of four bytes, with a four byte header
 //  for each chunk. The overhead of each chunk is kept low by keeping pointers
 //  as two byte offsets within the heap, rather than (4 or 8 byte) pointers.
 
 namespace {
 
 // When POSIX threads are not available, make the mutex operations a nop
 #ifndef _LIBCXXABI_HAS_NO_THREADS
-static pthread_mutex_t heap_mutex = PTHREAD_MUTEX_INITIALIZER;
+static __libcxxabi_mutex_t heap_mutex = _LIBCXXABI_MUTEX_INITIALIZER;
 #else
 static void * heap_mutex = 0;
 #endif
 
 class mutexor {
 public:
 #ifndef _LIBCXXABI_HAS_NO_THREADS
-    mutexor ( pthread_mutex_t *m ) : mtx_(m) { pthread_mutex_lock ( mtx_ ); }
-    ~mutexor () { pthread_mutex_unlock ( mtx_ ); }
+    mutexor ( __libcxxabi_mutex_t *m ) : mtx_(m) { __libcxxabi_mutex_lock ( mtx_ ); }
+    ~mutexor () { __libcxxabi_mutex_unlock ( mtx_ ); }
 #else
     mutexor ( void * ) {}
     ~mutexor () {}
 #endif
 private:
     mutexor ( const mutexor &rhs );
     mutexor & operator = ( const mutexor &rhs );
 #ifndef _LIBCXXABI_HAS_NO_THREADS
-    pthread_mutex_t *mtx_;
+    __libcxxabi_mutex_t *mtx_;
 #endif
     };
 
 
 static const size_t HEAP_SIZE = 512;
 char heap [ HEAP_SIZE ] __attribute__((aligned));
 
 typedef unsigned short heap_offset;

src/threading_support.h

+//===------------------------ threading_support.h -------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCXXABI_THREADING_SUPPORT_H
+#define _LIBCXXABI_THREADING_SUPPORT_H
+
+#include "__cxxabi_config.h"
+#include "config.h"
+
+#ifndef _LIBCXXABI_HAS_NO_THREADS
+
+#if defined(_LIBCXXABI_USE_THREAD_API_PTHREAD)
+#include <pthread.h>
+
+#define _LIBCXXABI_THREAD_ABI_VISIBILITY inline _LIBCXXABI_INLINE_VISIBILITY
+
+// Mutex
+typedef pthread_mutex_t __libcxxabi_mutex_t;
+#define _LIBCXXABI_MUTEX_INITIALIZER PTHREAD_MUTEX_INITIALIZER
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_mutex_lock(__libcxxabi_mutex_t *mutex) {
+  return pthread_mutex_lock(mutex);
+}
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_mutex_unlock(__libcxxabi_mutex_t *mutex) {
+  return pthread_mutex_unlock(mutex);
+}
+
+// Condition variable
+typedef pthread_cond_t __libcxxabi_condvar_t;
+#define _LIBCXXABI_CONDVAR_INITIALIZER PTHREAD_COND_INITIALIZER
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_condvar_wait(__libcxxabi_condvar_t *cv,
+                             __libcxxabi_mutex_t *mutex) {
+  return pthread_cond_wait(cv, mutex);
+}
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_condvar_broadcast(__libcxxabi_condvar_t *cv) {
+  return pthread_cond_broadcast(cv);
+}
+
+// Execute once
+typedef pthread_once_t __libcxxabi_exec_once_flag;
+#define _LIBCXXABI_EXEC_ONCE_INITIALIZER PTHREAD_ONCE_INIT
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_execute_once(__libcxxabi_exec_once_flag *flag,
+                             void (*init_routine)(void)) {
+  return pthread_once(flag, init_routine);
+}
+
+// Thread id
+#if defined(__APPLE__) && !defined(__arm__)
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+mach_port_t __libcxxabi_thread_get_port()
+{
+    return pthread_mach_thread_np(pthread_self());
+}
+#endif
+
+// Thread
+typedef pthread_t __libcxxabi_thread_t;
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_thread_create(__libcxxabi_thread_t* __t,
+                           void* (*__func)(void*), void* __arg)
+{
+    return pthread_create(__t, 0, __func, __arg);
+}
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_thread_join(__libcxxabi_thread_t* __t)
+{
+    return pthread_join(*__t, 0);
+}
+
+// TLS
+typedef pthread_key_t __libcxxabi_tls_key;
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_tls_create(__libcxxabi_tls_key *key,
+                           void (*destructor)(void *)) {
+  return pthread_key_create(key, destructor);
+}
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+void *__libcxxabi_tls_get(__libcxxabi_tls_key key) {
+  return pthread_getspecific(key);
+}
+
+_LIBCXXABI_THREAD_ABI_VISIBILITY
+int __libcxxabi_tls_set(__libcxxabi_tls_key key, void *value) {
+  return pthread_setspecific(key, value);
+}
+#endif // _LIBCXXABI_USE_THREAD_API_PTHREAD
+#endif // !_LIBCXXABI_HAS_NO_THREADS
+#endif // _LIBCXXABI_THREADING_SUPPORT_H

test/test_exception_storage.pass.cpp

 //===-------------------- test_exception_storage.cpp ----------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "../src/config.h"
 
 #include <cstdlib>
 #include <algorithm>
 #include <iostream>
-#ifndef _LIBCXXABI_HAS_NO_THREADS
-#  include <pthread.h>
-#endif
+#include "../src/threading_support.h"
 #include <unistd.h>
 
 #include "../src/cxa_exception.hpp"
 
 typedef __cxxabiv1::__cxa_eh_globals globals_t ;
 
 void *thread_code (void *parm) {
     size_t *result = (size_t *) parm;
void *thread_code (void *parm) {
     
     *result = (size_t) glob1;
     sleep ( 1 );
     return parm;
     }
 
 #ifndef _LIBCXXABI_HAS_NO_THREADS
 #define NUMTHREADS  10
-size_t      thread_globals [ NUMTHREADS ] = { 0 };
-pthread_t   threads        [ NUMTHREADS ];
+size_t                 thread_globals [ NUMTHREADS ] = { 0 };
+__libcxxabi_thread_t   threads        [ NUMTHREADS ];
 #endif
 
 int main ( int argc, char *argv [] ) {
     int retVal = 0;
 
 #ifndef _LIBCXXABI_HAS_NO_THREADS
 //  Make the threads, let them run, and wait for them to finish
     for ( int i = 0; i < NUMTHREADS; ++i )
-        pthread_create( threads + i, NULL, thread_code, (void *) (thread_globals + i));
+        __libcxxabi_thread_create ( threads + i, thread_code, (void *) (thread_globals + i));
     for ( int i = 0; i < NUMTHREADS; ++i )
-        pthread_join ( threads [ i ], NULL );
+        __libcxxabi_thread_join ( &threads [ i ] );
 
     for ( int i = 0; i < NUMTHREADS; ++i )
         if ( 0 == thread_globals [ i ] ) {
             std::cerr << "Thread #" << i << " had a zero global" << std::endl;
             retVal = 1;
             }
         
     std::sort ( thread_globals, thread_globals + NUMTHREADS );

test/test_fallback_malloc.pass.cpp

 //===--------------------- test_fallback_malloc.cpp -----------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include <iostream>
 #include <deque>
 
-#include <pthread.h>
+#include "../src/threading_support.h"
 
 typedef std::deque<void *> container;
 
 // #define  DEBUG_FALLBACK_MALLOC
 #define INSTRUMENT_FALLBACK_MALLOC
 #include "../src/fallback_malloc.cpp"
 
 container alloc_series ( size_t sz ) {