[libcxx] Externally threaded libc++ variant

[rmaprath]

Summary

This patch builds on the work done under D19412 where all pthread dependencies were refactored under a separate thread-support API.

The current patch enables building a new variant of libc++ where all the threading dependencies are exported into a runtime API. This allows platform vendors to re-target these threading calls in a platform-defined manner (at runtime), paving the way for libc++ on non-pthread platforms.

For running the libc++ test suite, I've introduced a sample implementation of this external-thread-API using pthreads. This implementation should give an idea of what the end goal is.

This patch supersedes the sketch presented in D19415.

[rmaprath]

@mclow.lists, @EricWF: Any comments on this? If it helps, I could split off the library source changes and test setup into two separate patches. I thought to keep the two together as the test setup gives an idea of the overall objective.

Thanks.

/ Asiri

[bcraig]

Note: You'll want to look at http://reviews.llvm.org/D20573, as there will be confilicts for whoever submits second.

include/__threading_support

201

I'm not sure I like taking the freedom to define _LIBCPP_MUTEX_INITIALIZER away from implementers.

Would it be too terrible to replace this entire #elif block with something like the following?

#if !defined(__has_include) || __has_include(<os_provided_thread.h>)
#include <os_provided_thread.h>
#else
#error "_LIBCPP_THREAD_API_EXTERNAL requires the implementer to provide <os_provided_thread.h> in the include path"
#endif

[rmaprath]

include/__threading_support

201

The problem is that, std::mutex constructor needs to be constexpr (as you pointed out earlier). And since __libcpp_mutex_t is a pointer type (for this externally threaded variant), sensible definitions for _LIBCPP_MUTEX_INITIALIZER are limited.

Other than nullptr, one may be able to define _LIBCPP_MUTEX_INITIALIZER to be a pointer to some constant mutex (presuming that make it constexpr OK?) but I'm not convinced if such a setup would be very useful.

Hope that sounds sensible?

[bcraig]

include/__threading_support

201

If the implementer gets to provide an entire header, then they also get to choose what libcpp_mutex_t will be. They could make it a struct.

[rmaprath]

include/__threading_support

201

This externally-threaded library variant needs to be compiled against a set API, so we have this header with declarations like __libcpp_mutex_lock() which are referred from within the library sources (same function signatures as those used in LIBCPP_THREAD_API_PTHREAD - this allows us to keep the library source changes to a minimum).

Now, in this particular library variant, we want to differ these calls to runtime rather than libcxx compile-time!

On the other hand, I think you are proposing a compile-time (static) thread porting setup where the platform vendors need to supply a header file with appropriate definitions for these functions / types, and they would compile libcxx themselves? That sounds like a good idea, but the current patch is aiming for a different goal: we provide a pre-compiled libcxx library which calls out to those __libcpp_xxx functions at runtime, and platform vendors need to provide those functions, they don't have to compile libcxx themselves. This is what forces us to use opaque pointer types to capture platform-defined threading primitives.

Perhaps I could improve the naming here, may be LIBCPP_HAS_RUNTIME_THREAD_API? Or LIBCPP_HAS_DYNAMIC_THREAD_API?

[bcraig]

include/__threading_support

201

That is an excellent summary, and it does a good job of explaining the disconnect we were having. I'm going to push forward with the disconnect though :)

I think you can implement the dynamic case in terms of the static case without loss of generality or performance. You ("Vendor X") could pre-compile your library with a pointer-sized libcpp_mutex_t defined in <os_provided_thread.h>, and at runtime call out to a different library that has an implementation of libcpp_mutex_lock. Someone else ("Vendor Y") could have a larger libcpp_mutex_t defined, and provide a static inline definition of libcpp_mutex_lock.

[rmaprath]

include/__threading_support

201

OK, I think I understand your point :)

Will meditate on it a bit and spin a new patch tomorrow. Thanks!

[rmaprath]

include/__threading_support

201

There is a slight complication here. If we fully offload the external thread-API to platform vendors, nobody will be able to build this externally-threaded library variant using the vanilla libcxx sources, because they would have to first come up with a platform_threads.h header.

We could provide a dynamic_threads.h header with the method signatures only (which is selected if the platform_threads.h header is absent - and you get a "dynamically-threaded" library build). Now the library can be built + tested with the vanilla upstream sources. But note that we are introducing yet another header. I remember @mclow.lists mentioned that each additional header adds to the overhead of header lookup and we should try to keep that to a minimum.

We can workaround this additional header by not collecting it when building the normal library variant (so it doesn't add to the header lookup overhead).

Does that sound like an OK approach?

[bcraig]

include/__threading_support

201

Sounds reasonable.

[rmaprath]

Addressed review comments from @bcraig.

@EricWF: Gentle ping.

[bcraig]

include/__threading_support

201

This #include deserves a comment. Some perplexed developer is going to come along later and wonder where __static_threading is supposed to come from.

[rmaprath]

Added a comment about __static_threading and __dynamic_threading header includes.

@EricWF: Ping (sorry).

[rmaprath]

@mclow.lists, @EricWF: Ping.

@bcraig: I guess you wouldn't be able to let this through on your own? (in case if there isn't much interest in this functionality from others). Eric might be away though (didn't see many emails from him), so I'll hold off for now.

/ Asiri

[mclow.lists]

Some nits while reading this. More to come.

Also, I don't see how this can be retargeted "at runtime"; are you implying that someone can choose at program launch time what threading system to use?

How could that work given (say) a constexpr constructor for a mutex type?
[ I read the previous discussion, but still don't see how to make it work]

src/algorithm.cpp

99

Why the undef here?

src/mutex.cpp

277

Again; why undef at the end of a source file (not an include)?

For that matter, there's no need to use reserved identifiers here - __LOCKXXX

[rmaprath]

In D20328#451178, @mclow.lists wrote:

Also, I don't see how this can be retargeted "at runtime"; are you implying that someone can choose at program launch time what threading system to use?

Yup, users can link their objects with an implementation of the dynamic threading API and it should all be fine.

How could that work given (say) a constexpr constructor for a mutex type?
[ I read the previous discussion, but still don't see how to make it work]

Yeah, that is a real pain point. I have provided a sample implementation of the API in test/support/external_threads.cpp (last file in this patch) where I've adopted a initialize-on-first-use policy to workaround this particular problem. I've put a bunch of comments there to explain the details.

The constexpr mutex constructor is apparently also a pain point on windows [1], I'm sure @STL_MSFT knows more about it.

Thanks!

/ Asiri

[1] https://blogs.msdn.microsoft.com/vcblog/2015/06/02/constexpr-complete-for-vs-2015-rtm-c11-compiler-c17-stl/

[rmaprath]

In D20328#451197, @rmaprath wrote:

In D20328#451178, @mclow.lists wrote:

Also, I don't see how this can be retargeted "at runtime"; are you implying that someone can choose at program launch time what threading system to use?

Yup, users can link their objects with an implementation of the dynamic threading API and it should all be fine.

Apologies, not at program launch time but link time.

[STL_MSFT]

[Asiri Rathnayake]

The constexpr mutex constructor is apparently also a
pain point on windows [1], I'm sure @STL_MSFT knows more about it.

That's because of MSVC-specific issues, where we need to dynamically switch between WinAPI and ConcRT implementations depending on whether the end user's machine is Win7+/Vista/XP. If we could assume Win7+, we wouldn't need this squirrelly machinery.

STL

[rmaprath]

In D20328#451210, @STL_MSFT wrote:

[Asiri Rathnayake]

The constexpr mutex constructor is apparently also a
pain point on windows [1], I'm sure @STL_MSFT knows more about it.

That's because of MSVC-specific issues, where we need to dynamically switch between WinAPI and ConcRT implementations depending on whether the end user's machine is Win7+/Vista/XP. If we could assume Win7+, we wouldn't need this squirrelly machinery.

STL

Thanks. I was thinking may be it's because you also need to call some dynamic initialization routine from within the std::mutex constructor (like, to initialize a kernel provided mutex). Good to know that this isn't the case.

Cheers,

/ Asiri

[rmaprath]

Addressed comments from @mclow.lists.

[rmaprath]

@mclow.lists, @EricWF: Gentle (and shameless) ping!

[rmaprath]

@mclow.lists, @EricWF: Ping.

[mclow.lists]

Apologies, not at program launch time but link time.

I'm OK with that; I think that's unnecessary complication, but not a deal-breaker.
The choosing at program launch seems unworkable to me.

[ I think that the threading implementation should be chosen by the provider of libc++, frankly. ]

[mclow.lists]

I've adopted a initialize-on-first-use policy to workaround this particular problem.

That's not constexpr.

You say it in your comment "This prohibits any prospects of calling a runtime initialization routine", but then you have a runtime initialization routine.

[rmaprath]

In D20328#456271, @mclow.lists wrote:

I've adopted a initialize-on-first-use policy to workaround this particular problem.

That's not constexpr.

You say it in your comment "This prohibits any prospects of calling a runtime initialization routine", but then you have a runtime initialization routine.

std::mutex() constructor is still constexpr, what I've done is to defer the initialization to lock() and unlock() methods. This is pretty much the only way I can keep the constructor constexpr and allow an underlying platform implementation to provide the guts of std::mutex.

Hope that makes sense?

[rmaprath]

In D20328#456241, @mclow.lists wrote:

Apologies, not at program launch time but link time.

I'm OK with that; I think that's unnecessary complication, but not a deal-breaker.
The choosing at program launch seems unworkable to me.

[ I think that the threading implementation should be chosen by the provider of libc++, frankly. ]

The problem with the latter is the number of different platforms we have to support, each with different threading primitives. For most RTOS platforms, pthreads is a big ask and even if they can provide a subset of <pthread.h>, different platforms will have different definitions of PTHREAD_MUTEX_INITIALIZER for example, making it very difficult for us to build libc++ in a platform agnostic way.

[rmaprath]

Reason for abandoning: This patch uses opaque pointers to delegate std::mutex and std::condition_variable types to the underlying platform implementation.

This creates problems given that these two types have constexpr constructors. We must make std::mutex::lock() (and the like) initialize the internal opaque pointer type before it is used (cannot initialize from the constructor), which in turn implies that the std::mutex::lock() method could fail if the implementation cannot allocate memory to hold the underlying platform mutex type - this is not a valid condition for the std::mutex::lock() method to fail. Besides, with this workaround, we are essentially trying to beat the purpose of a constexpr mutex constructor (be able to statically declare a mutex and use it without any explicit initialization).

After discussion with @mclow.lists, we think it's best to have libc++ vendors know the underlying mutex / condition variable types in advance (libc++ build time). I will create a new patch to implement this approach soon.

CMakeLists.txt

   "Build libc++ with support for a monotonic clock.
    This option may only be set to OFF when LIBCXX_ENABLE_THREADS=OFF." ON)
 option(LIBCXX_HAS_MUSL_LIBC "Build libc++ with support for the Musl C library" OFF)
-option(LIBCXX_HAS_PTHREAD_API "Ignore auto-detection and force use of pthread API" OFF)
+option(LIBCXX_HAS_PTHREAD_THREAD_API "Ignore auto-detection and force use of pthread API" OFF)
+option(LIBCXX_HAS_EXTERNAL_THREAD_API
+  "Build libc++ with an externalized threading API.
+   This option may only be set to ON when LIBCXX_ENABLE_THREADS=ON." OFF)
 
 # Misc options ----------------------------------------------------------------
 # FIXME: Turn -pedantic back ON. It is currently off because it warns

 # Check option configurations
 #===============================================================================
 
-# Ensure LIBCXX_ENABLE_MONOTONIC_CLOCK is set to ON only when
+# Ensure LIBCXX_ENABLE_MONOTONIC_CLOCK is always set to ON when
 # LIBCXX_ENABLE_THREADS is on.
 if(LIBCXX_ENABLE_THREADS AND NOT LIBCXX_ENABLE_MONOTONIC_CLOCK)
   message(FATAL_ERROR "LIBCXX_ENABLE_MONOTONIC_CLOCK can only be set to OFF"
                       " when LIBCXX_ENABLE_THREADS is also set to OFF.")
 endif()
 
-if(LIBCXX_HAS_PTHREAD_API AND NOT LIBCXX_ENABLE_THREADS)
-  message(FATAL_ERROR "LIBCXX_HAS_PTHREAD_API can only be set to ON"
+if(LIBCXX_HAS_PTHREAD_THREAD_API AND NOT LIBCXX_ENABLE_THREADS)
+  message(FATAL_ERROR "LIBCXX_HAS_PTHREAD_THREAD_API can only be set to ON"
+                      " when LIBCXX_ENABLE_THREADS is also set to ON.")
+endif()
+
+if(LIBCXX_HAS_EXTERNAL_THREAD_API AND NOT LIBCXX_ENABLE_THREADS)
+  message(FATAL_ERROR "LIBCXX_HAS_EXTERNAL_THREAD_API can only be set to ON"
                       " when LIBCXX_ENABLE_THREADS is also set to ON.")
 endif()
 

 config_define_if_not(LIBCXX_ENABLE_MONOTONIC_CLOCK _LIBCPP_HAS_NO_MONOTONIC_CLOCK)
 config_define_if_not(LIBCXX_ENABLE_THREAD_UNSAFE_C_FUNCTIONS _LIBCPP_HAS_NO_THREAD_UNSAFE_C_FUNCTIONS)
 
-config_define_if(LIBCXX_HAS_PTHREAD_API _LIBCPP_HAS_THREAD_API_PTHREAD)
+config_define_if(LIBCXX_HAS_PTHREAD_THREAD_API _LIBCPP_HAS_THREAD_API_PTHREAD)
+config_define_if(LIBCXX_HAS_EXTERNAL_THREAD_API _LIBCPP_HAS_THREAD_API_EXTERNAL)
 config_define_if(LIBCXX_HAS_MUSL_LIBC _LIBCPP_HAS_MUSL_LIBC)
 
 if (LIBCXX_NEEDS_SITE_CONFIG)

include/__config

 #endif
 
 // Thread API
-#if !defined(_LIBCPP_HAS_NO_THREADS) && !defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
+#if !defined(_LIBCPP_HAS_NO_THREADS) && \
+    !defined(_LIBCPP_HAS_THREAD_API_PTHREAD) && \
+    !defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
 # if defined(__FreeBSD__) || \
     defined(__NetBSD__) || \
     defined(__linux__) || \

          _LIBCPP_HAS_NO_THREADS is not defined.
 #endif
 
+#if defined(_LIBCPP_HAS_NO_THREADS) && defined(_LIBCPP_HAS_THREAD_API_EXTERNAL) 
+#  error _LIBCPP_HAS_EXTERNAL_THREAD_API may not be defined when \
+         _LIBCPP_HAS_NO_THREADS is defined.
+#endif
+
 #if defined(_LIBCPP_HAS_NO_MONOTONIC_CLOCK) && !defined(_LIBCPP_HAS_NO_THREADS)
 #  error _LIBCPP_HAS_NO_MONOTONIC_CLOCK may only be defined when \
          _LIBCPP_HAS_NO_THREADS is defined.

include/__config_site.in

 #cmakedefine _LIBCPP_HAS_NO_THREAD_UNSAFE_C_FUNCTIONS
 #cmakedefine _LIBCPP_HAS_MUSL_LIBC
 #cmakedefine _LIBCPP_HAS_THREAD_API_PTHREAD
+#cmakedefine _LIBCPP_HAS_THREAD_API_EXTERNAL
 
 #endif // _LIBCPP_CONFIG_SITE

include/__dynamic_threading

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_DYNAMIC_THREADING
+#define _LIBCPP_DYNAMIC_THREADING
+
+#include <__config>
+
+#ifndef _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
+#pragma GCC system_header
+#endif
+
+#if !defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+#  error "Only valid when _LIBCPP_HAS_THREAD_API_EXTERNAL is defined"
+#endif
+
+#include <time.h>
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+// Mutex
+#define _LIBCPP_MUTEX_INITIALIZER nullptr
+struct __libcpp_platform_mutex_t;
+typedef __libcpp_platform_mutex_t *__libcpp_mutex_t;
+
+int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m);
+int __libcpp_mutex_lock(__libcpp_mutex_t* __m);
+int __libcpp_mutex_trylock(__libcpp_mutex_t* __m);
+int __libcpp_mutex_unlock(__libcpp_mutex_t* __m);
+int __libcpp_mutex_destroy(__libcpp_mutex_t* __m);
+
+// Condition variable
+#define _LIBCPP_CONDVAR_INITIALIZER nullptr
+struct __libcpp_platform_condvar_t;
+typedef __libcpp_platform_condvar_t *__libcpp_condvar_t;
+
+int __libcpp_condvar_signal(__libcpp_condvar_t* __cv);
+int __libcpp_condvar_broadcast(__libcpp_condvar_t* __cv);
+int __libcpp_condvar_wait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m);
+int __libcpp_condvar_timedwait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m, timespec* __ts);
+int __libcpp_condvar_destroy(__libcpp_condvar_t* __cv);
+
+// Thread id
+typedef unsigned long __libcpp_thread_id;
+bool __libcpp_thread_id_equal(__libcpp_thread_id t1, __libcpp_thread_id t2);
+bool __libcpp_thread_id_less(__libcpp_thread_id t1, __libcpp_thread_id t2);
+
+// Thread
+struct __libcpp_platform_thread_t;
+typedef __libcpp_platform_thread_t *__libcpp_thread_t;
+int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg);
+__libcpp_thread_id __libcpp_thread_get_current_id();
+__libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t* __t);
+int __libcpp_thread_join(__libcpp_thread_t* __t);
+int __libcpp_thread_detach(__libcpp_thread_t* __t);
+void __libcpp_thread_yield();
+void __libcpp_thread_finalize(__libcpp_thread_t* __t);
+
+// Thread local storage
+struct __libcpp_platform_tlskey_t;
+typedef __libcpp_platform_tlskey_t *__libcpp_tl_key;
+int __libcpp_tl_create(__libcpp_tl_key* __key, void (*__at_exit)(void*));
+void* __libcpp_tl_get(__libcpp_tl_key __key);
+void __libcpp_tl_set(__libcpp_tl_key __key, void* __p);
+void __libcpp_tl_finalize(__libcpp_tl_key __key);
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP_DYNAMIC_THREADING

include/__threading_support

 #include <sched.h>
 #endif
 
-_LIBCPP_BEGIN_NAMESPACE_STD
-
 #if defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
 
+_LIBCPP_BEGIN_NAMESPACE_STD
+
 // Mutex
 #define _LIBCPP_MUTEX_INITIALIZER PTHREAD_MUTEX_INITIALIZER
 typedef pthread_mutex_t __libcpp_mutex_t;

     pthread_setspecific(__key, __p);
 }
 
-#else // !_LIBCPP_HAS_THREAD_API_PTHREAD
+_LIBCPP_END_NAMESPACE_STD
+
+#elif defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+/* !!!FOR LIBRARY IMPLEMENTORS ONLY!!! */
+#  if !defined(__has_include) || __has_include(<__static_threading>)
+     /* For compiling libc++ against an external thread implementation.
+        This header should provide the definitions of various __libcpp_*
+        functions declared above. These functions should be defined as
+        _LIBCPP_ALWAYS_INLINE so that they are inlined into the library
+        code. */
+#    include<__static_threading>
+#  else
+     /* For compiling libc++ with an externalized (runtime) thread API. This
+        is the default when no __static_threading header is provided. */
+#    include<__dynamic_threading>
+#  endif
+#else
   #error "No thread API selected."
 #endif
 
-_LIBCPP_END_NAMESPACE_STD
-
 #endif // _LIBCPP_HAS_NO_THREADS
 
 #endif // _LIBCPP_THREADING_SUPPORT

include/thread

     // so this destructor is only invoked during program termination. Invoking
     // pthread_key_delete(__key_) may prevent other threads from deleting their
     // thread local data. For this reason we leak the key.
+
+#if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+    __libcpp_tl_finalize(__key_);
+#endif
 }
 
 template <class _Tp>

lib/CMakeLists.txt

   )
 endif()
 
+if (LIBCXX_HAS_EXTERNAL_THREAD_API)
+  file(GLOB LIBCXX_EXTERNAL_THREADING_SUPPORT_SOURCES ../test/support/external_threads.cpp)
+  add_library(cxx_external_threading_support STATIC ${LIBCXX_EXTERNAL_THREADING_SUPPORT_SOURCES})
+  target_link_libraries(cxx_external_threading_support cxx)
+
+  set_target_properties(cxx_external_threading_support
+    PROPERTIES
+      COMPILE_FLAGS "${LIBCXX_COMPILE_FLAGS}"
+      OUTPUT_NAME   "c++external_threading_support"
+  )
+endif()
+
 # Generate a linker script inplace of a libc++.so symlink. Rerun this command
 # after cxx builds.
 if (LIBCXX_ENABLE_ABI_LINKER_SCRIPT)

src/algorithm.cpp

 template unsigned __sort5<__less<long double>&, long double*>(long double*, long double*, long double*, long double*, long double*, __less<long double>&);
 
 #ifndef _LIBCPP_HAS_NO_THREADS
+# if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+static mutex __rs_mut;
+#  define LOCK_LOCK() __rs_mut.lock();
+#  define LOCK_UNLOCK() __rs_mut.unlock();
+# else
 static __libcpp_mutex_t __rs_mut = _LIBCPP_MUTEX_INITIALIZER;
+#  define LOCK_LOCK() __libcpp_mutex_lock(&__rs_mut);
+#  define LOCK_UNLOCK() __libcpp_mutex_unlock(&__rs_mut);
+# endif
 #endif
 unsigned __rs_default::__c_ = 0;
 
 __rs_default::__rs_default()
 {
 #ifndef _LIBCPP_HAS_NO_THREADS
-    __libcpp_mutex_lock(&__rs_mut);
+    LOCK_LOCK()
 #endif
     __c_ = 1;
 }

 {
 #ifndef _LIBCPP_HAS_NO_THREADS
     if (--__c_ == 0)
-       __libcpp_mutex_unlock(&__rs_mut);
+       LOCK_UNLOCK()
 #else
     --__c_;
 #endif

src/memory.cpp

 #if defined(_LIBCPP_HAS_C_ATOMIC_IMP) && !defined(_LIBCPP_HAS_NO_THREADS)
 
 static const std::size_t __sp_mut_count = 16;
+#if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+static mutex mut_back_imp[__sp_mut_count];
+#else
 static __libcpp_mutex_t mut_back_imp[__sp_mut_count] =
 {
     _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER,

     _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER,
     _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER, _LIBCPP_MUTEX_INITIALIZER
 };
+#endif
 
 static mutex* mut_back = reinterpret_cast<std::mutex*>(mut_back_imp);
 

src/mutex.cpp

 // keep in sync with:  7741191.
 
 #ifndef _LIBCPP_HAS_NO_THREADS
+# if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+static mutex mut;
+static condition_variable cv;
+#  define LOCK_INIT() unique_lock<mutex> lk(mut);
+#  define LOCK_LOCK() lk.lock();
+#  define LOCK_UNLOCK() lk.unlock();
+#  define CV_WAIT_LOCK() cv.wait(lk);
+#  define CV_NOTIFY_ALL() cv.notify_all();
+# else
 static __libcpp_mutex_t mut = _LIBCPP_MUTEX_INITIALIZER;
 static __libcpp_condvar_t cv = _LIBCPP_CONDVAR_INITIALIZER;
+#  define LOCK_INIT() __libcpp_mutex_lock(&mut);
+#  define LOCK_LOCK() __libcpp_mutex_lock(&mut);
+#  define LOCK_UNLOCK() __libcpp_mutex_unlock(&mut);
+#  define CV_WAIT_LOCK() __libcpp_condvar_wait(&cv, &mut);
+#  define CV_NOTIFY_ALL() __libcpp_condvar_broadcast(&cv);
+# endif
 #endif
 
 /// NOTE: Changes to flag are done via relaxed atomic stores

 #endif  // _LIBCPP_NO_EXCEPTIONS
     }
 #else // !_LIBCPP_HAS_NO_THREADS
-    __libcpp_mutex_lock(&mut);
+    LOCK_INIT()
     while (flag == 1)
-        __libcpp_condvar_wait(&cv, &mut);
+        CV_WAIT_LOCK()
     if (flag == 0)
     {
 #ifndef _LIBCPP_NO_EXCEPTIONS

         {
 #endif  // _LIBCPP_NO_EXCEPTIONS
             __libcpp_relaxed_store(&flag, 1ul);
-            __libcpp_mutex_unlock(&mut);
+            LOCK_UNLOCK()
             func(arg);
-            __libcpp_mutex_lock(&mut);
+            LOCK_LOCK()
             __libcpp_relaxed_store(&flag, ~0ul);
-            __libcpp_mutex_unlock(&mut);
-            __libcpp_condvar_broadcast(&cv);
+            LOCK_UNLOCK()
+            CV_NOTIFY_ALL()
 #ifndef _LIBCPP_NO_EXCEPTIONS
         }
         catch (...)
         {
-            __libcpp_mutex_lock(&mut);
+            LOCK_LOCK()
             __libcpp_relaxed_store(&flag, 0ul);
-            __libcpp_mutex_unlock(&mut);
-            __libcpp_condvar_broadcast(&cv);
+            LOCK_UNLOCK()
+            CV_NOTIFY_ALL()
             throw;
         }
 #endif  // _LIBCPP_NO_EXCEPTIONS
     }
     else
-        __libcpp_mutex_unlock(&mut);
+        LOCK_UNLOCK()
 #endif // !_LIBCPP_HAS_NO_THREADS
 
 }

src/thread.cpp

 
 thread::~thread()
 {
+#if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+    __libcpp_thread_finalize(&__t_);
+#endif
     if (__t_ != 0)
         terminate();
 }

test/CMakeLists.txt

 pythonize_bool(LIBCXXABI_ENABLE_SHARED)
 pythonize_bool(LIBCXXABI_USE_LLVM_UNWINDER)
 pythonize_bool(LIBCXX_HAS_ATOMIC_LIB)
+pythonize_bool(LIBCXX_HAS_EXTERNAL_THREAD_API)
 
 # The tests shouldn't link to any ABI library when it has been linked into
 # libc++ statically or via a linker script.

   set(experimental_dep cxx_experimental)
 endif()
 
+if (LIBCXX_HAS_EXTERNAL_THREAD_API)
+  set(external_threading_support_dep cxx_external_threading_support)
+endif()
+
 add_lit_testsuite(check-libcxx
   "Running libcxx tests"
   ${CMAKE_CURRENT_BINARY_DIR}
-  DEPENDS cxx ${experimental_dep})
+  DEPENDS cxx ${experimental_dep} ${external_threading_support_dep})
 
 if (LIBCXX_GENERATE_COVERAGE)
   include(CodeCoverage)

test/libcxx/test/config.py

                 self.cxx.link_flags += [abs_path]
             else:
                 self.cxx.link_flags += ['-lc++']
+        # This needs to come after -lc++ as we want its unresolved thread-api symbols
+        # to be picked up from this one.
+        if self.get_lit_bool('libcxx_external_thread_api', default=False):
+            self.cxx.link_flags += ['-lc++external_threading_support']
 
     def configure_link_flags_abi_library(self):
         cxx_abi = self.get_lit_conf('cxx_abi', 'libcxxabi')

test/lit.site.cfg.in

 config.llvm_unwinder            = "@LIBCXXABI_USE_LLVM_UNWINDER@"
 config.use_libatomic            = "@LIBCXX_HAS_ATOMIC_LIB@"
 config.libcxxabi_shared         = "@LIBCXXABI_ENABLE_SHARED@"
+config.libcxx_external_thread_api  = "@LIBCXX_HAS_EXTERNAL_THREAD_API@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@LIBCXX_SOURCE_DIR@/test/lit.cfg")

test/std/thread/thread.condition/thread.condition.condvar/native_handle.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <condition_variable>
 

test/std/thread/thread.mutex/thread.mutex.requirements/thread.mutex.requirements.mutex/thread.mutex.class/native_handle.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <mutex>
 

test/std/thread/thread.mutex/thread.mutex.requirements/thread.mutex.requirements.mutex/thread.mutex.recursive/native_handle.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <mutex>
 

test/std/thread/thread.threads/thread.thread.class/thread.thread.member/native_handle.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <thread>
 

test/std/thread/thread.threads/thread.thread.class/types.pass.cpp

 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <thread>
 

test/support/external_threads.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#ifndef SUPPORT_EXTERNAL_THREADS
+#define SUPPORT_EXTERNAL_THREADS
+
+#include <__threading_support>
+
+// Only define these symbols if using the __dynamic_threading header
+#if defined(_LIBCPP_DYNAMIC_THREADING)
+
+#include <pthread.h>
+#include <stdlib.h>
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+/* In the pthread-based (or statically-threaded) libc++ variant, pthread
+   primitives (like pthread_mutex_t) become part of the internal states of the
+   corresponding C++ constructs (e.g. C++ mutex class holds an internal
+   pthread_mutex_t object). In contrast, in the externally-threaded libc++
+   variant, C++ constructs hold (opaque) pointers to the underlying platform-
+   defined threading primitives. The following macros are used to convert the
+   opaque pointer types held within the C++ constructs and cast them as pointers
+   to the appropriate platform-defined thread primitive type (in this case,
+   pthread types again).
+*/
+#define AS_PTHREAD_MUTPTR(x) reinterpret_cast<pthread_mutex_t*>(x)
+#define AS_PTHREAD_CONDPTR(x) reinterpret_cast<pthread_cond_t*>(x)
+#define AS_PTHREAD_TPTR(x) reinterpret_cast<pthread_t*>(x)
+#define AS_PTHREAD_KPTR(x) reinterpret_cast<pthread_key_t*>(x)
+
+//-- Mutex --//
+
+/* This method is invoked from within the std::recursive_mutex constructor, as
+   such, it does not need to be thread-safe when initializing the *__m pointer.
+*/
+int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m)
+{
+    // Populate the internal opaque pointer *__m
+    pthread_mutex_t *mut = (pthread_mutex_t*) malloc(sizeof(pthread_mutex_t));
+    if (mut == nullptr)
+        return -1;
+    *__m = reinterpret_cast<__libcpp_mutex_t> (mut);
+
+    // Initialize the allocated pthread_mutex_t object as a recursive mutex
+    pthread_mutexattr_t attr;
+    int __ec = pthread_mutexattr_init(&attr);
+    if (__ec)
+      goto fail;
+
+    __ec = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
+    if (__ec)
+    {
+        pthread_mutexattr_destroy(&attr);
+        goto fail;
+    }
+
+    __ec = pthread_mutex_init(AS_PTHREAD_MUTPTR(*__m), &attr);
+    if (__ec)
+    {
+        pthread_mutexattr_destroy(&attr);
+        goto fail;
+    }
+
+    __ec = pthread_mutexattr_destroy(&attr);
+    if (__ec)
+    {
+        pthread_mutex_destroy(AS_PTHREAD_MUTPTR(*__m));
+        goto fail;
+    }
+
+    return 0;
+
+fail:
+    free(*__m);
+    return __ec;
+}
+
+/* C++11 standard requires mutex and condition_variable constructors to be
+   constexpr qualifying. This prohibits any prospects of calling a runtime
+   initialization routine from within mutex / condition_varialbe constructors.
+   For this reason, the external thread API must adopt an initialize-on-first-
+   use policy for mutexes and condition variables. With this strategy, we need
+   to be especially careful when initializing the internal opaque pointer *__m,
+   as multiple threads could attempt to lock the same mutex at once. In the
+   following routine, I've used a plain pthread mutex to make this initialization
+   thread-safe. It's also possible to use atomic compare-exchange builtins (where
+   available) to achieve the same goal.
+*/
+pthread_mutex_t mutex_init_access = PTHREAD_MUTEX_INITIALIZER;
+
+int mutex_safe_init(volatile __libcpp_mutex_t* __m) {
+    pthread_mutex_lock(&mutex_init_access);
+    if (*__m == nullptr) {
+        pthread_mutex_t *mut = (pthread_mutex_t*) malloc(sizeof(pthread_mutex_t));
+        if (mut == nullptr)
+            return -1;
+        if (pthread_mutex_init(mut, nullptr)) {
+            free(mut);
+            return -1;
+  }
+
+        *__m = reinterpret_cast<__libcpp_mutex_t> (mut);
+    }
+    pthread_mutex_unlock(&mutex_init_access);
+
+    return 0;
+}
+
+int __libcpp_mutex_lock(__libcpp_mutex_t* __m)
+{
+    // initialize-on-first-use
+    if (*__m == nullptr && mutex_safe_init(__m))
+        return -1;
+
+    return pthread_mutex_lock(AS_PTHREAD_MUTPTR(*__m));
+}
+
+int __libcpp_mutex_trylock(__libcpp_mutex_t* __m)
+{
+    // initialize-on-first-use
+    if (*__m == nullptr && mutex_safe_init(__m))
+        return -1;
+
+    return pthread_mutex_trylock(AS_PTHREAD_MUTPTR(*__m));
+}
+
+int __libcpp_mutex_unlock(__libcpp_mutex_t* __m)
+{
+    if (*__m == nullptr)
+        return 0;
+
+    return pthread_mutex_unlock(AS_PTHREAD_MUTPTR(*__m));
+}
+
+int __libcpp_mutex_destroy(__libcpp_mutex_t* __m) {
+    if (*__m == nullptr)
+        return 0;
+
+    int ec = pthread_mutex_destroy(AS_PTHREAD_MUTPTR(*__m));
+    free(*__m);
+    return ec;
+}
+
+//-- Condition variable --//
+
+int __libcpp_condvar_signal(__libcpp_condvar_t* __cv)
+{
+    if (*__cv == nullptr)
+      return 0;
+
+    return pthread_cond_signal(AS_PTHREAD_CONDPTR(*__cv));
+}
+
+int __libcpp_condvar_broadcast(__libcpp_condvar_t* __cv)
+{
+    if (*__cv == nullptr)
+      return 0;
+
+    return pthread_cond_broadcast(AS_PTHREAD_CONDPTR(*__cv));
+}
+
+// See comment for mutex_safe_init()
+pthread_mutex_t condvar_init_access = PTHREAD_MUTEX_INITIALIZER;
+
+int condvar_safe_init(__libcpp_condvar_t* __cv) {
+    pthread_mutex_lock(&condvar_init_access);
+    if (*__cv == nullptr) {
+        pthread_cond_t *cv = (pthread_cond_t*) malloc(sizeof(pthread_cond_t));
+        if (cv == nullptr)
+            return -1;
+        if (pthread_cond_init(cv, nullptr)) {
+      free(cv);
+            return -1;
+        }
+
+        *__cv = reinterpret_cast<__libcpp_condvar_t> (cv);
+    }
+    pthread_mutex_unlock(&condvar_init_access);
+
+    return 0;
+}
+
+int __libcpp_condvar_wait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m)
+{
+    // initialize-on-first-use
+    if (*__cv == nullptr && condvar_safe_init(__cv))
+        return -1;
+
+    return pthread_cond_wait(AS_PTHREAD_CONDPTR(*__cv), AS_PTHREAD_MUTPTR(*__m));
+}
+
+int __libcpp_condvar_timedwait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m, timespec* __ts)
+{
+    // initialize-on-first-use
+    if (*__cv == nullptr && condvar_safe_init(__cv))
+        return -1;
+
+    return pthread_cond_timedwait(AS_PTHREAD_CONDPTR(*__cv), AS_PTHREAD_MUTPTR(*__m), __ts);
+}
+
+int __libcpp_condvar_destroy(__libcpp_condvar_t* __cv)
+{
+    if (*__cv == nullptr)
+        return 0;
+
+    int ec = pthread_cond_destroy(AS_PTHREAD_CONDPTR(*__cv));
+    free(*__cv);
+    return ec;
+}
+
+//-- Thread --//
+
+bool __libcpp_thread_id_equal(__libcpp_thread_id t1, __libcpp_thread_id t2) {
+    return pthread_equal(t1, t2) != 0;
+}
+
+bool __libcpp_thread_id_less(__libcpp_thread_id t1, __libcpp_thread_id t2) {
+    return t1 < t2;
+}
+
+/* This method is invoked from within the std::thread constructor, as such, it
+   does not need to be thread-safe when initializing the *__t pointer.
+*/
+int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg)
+{
+    pthread_t *td = (pthread_t*) malloc(sizeof(pthread_t));
+    if (td == nullptr)
+      return -1;
+
+    *__t = reinterpret_cast<__libcpp_thread_t>(td);
+
+    return pthread_create(AS_PTHREAD_TPTR(*__t), 0, __func, __arg);
+}
+
+__libcpp_thread_id __libcpp_thread_get_current_id()
+{
+    return static_cast<__libcpp_thread_id>(pthread_self());
+}
+
+__libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t* __t)
+{
+    if (*__t == 0)
+        return 0;
+
+    return static_cast<__libcpp_thread_id>(*AS_PTHREAD_TPTR(*__t));
+}
+
+int __libcpp_thread_join(__libcpp_thread_t* __t)
+{
+    // Must return non-zero if the internal state has already been invalidated
+    // This can happen for example, if std::thread::join() has been called
+    // before.
+    if (*__t == 0)
+        return -1;
+
+    // Must cleanup memory allocated in __libcpp_thread_create() here, as the
+    // library will set *__t to zero after this call.
+    int ec = pthread_join(*AS_PTHREAD_TPTR(*__t), 0);
+    free(*__t);
+    return ec;
+}
+
+int __libcpp_thread_detach(__libcpp_thread_t* __t)
+{
+    // Must return non-zero if the internal state has already been invalidated
+    // This can happen for example, if std::thread::detach() has been called
+    // before.
+    if (*__t == 0)
+        return -1;
+
+    // Must cleanup memory allocated in __libcpp_thread_create() here, as the
+    // library will set *__t to zero after this call.
+    int ec = pthread_detach(*AS_PTHREAD_TPTR(*__t));
+    free(*__t);
+    return ec;
+}
+
+void __libcpp_thread_yield()
+{
+    sched_yield();
+}
+
+void __libcpp_thread_finalize(__libcpp_thread_t* __t) {
+    // Cleanup memory allocated in __libcpp_thread_create()
+    if (*__t != 0)
+        free(*__t);
+}
+
+//-- TLS --//
+
+/* This method is invoked from within the std::__thread_specific_ptr
+   constructor, as such, it does not need to be thread-safe when initializing
+   the *__t pointer. 
+*/
+int __libcpp_tl_create(__libcpp_tl_key* __key, void (*__at_exit)(void*))
+{
+    pthread_key_t *key = (pthread_key_t*) malloc(sizeof(pthread_key_t));
+    if (key == nullptr)
+      return -1;
+
+    *__key = reinterpret_cast<__libcpp_tl_key>(key);
+
+    return pthread_key_create(AS_PTHREAD_KPTR(*__key), __at_exit);
+}
+
+void* __libcpp_tl_get(__libcpp_tl_key __key)
+{
+    return pthread_getspecific(*AS_PTHREAD_KPTR(__key));
+}
+
+void __libcpp_tl_set(__libcpp_tl_key __key, void* __p)
+{
+    pthread_setspecific(*AS_PTHREAD_KPTR(__key), __p);
+}
+
+void __libcpp_tl_finalize(__libcpp_tl_key __key)
+{
+    // Only deallocate the memory allocated in __libcpp_tl_create. Do not
+    // call pthread_key_delete(__key). See comment for __thread_specific_ptr
+    // destructor.
+    if (__key != nullptr)
+      free(__key);
+}
+
+#undef AS_PTHREAD_MUTPTR
+#undef AS_PTHREAD_CONDPTR
+#undef AS_PTHREAD_TPTR
+#undef AS_PTHREAD_KPTR
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP_DYNAMIC_THREADING
+
+#endif // SUPPORT_EXTERNAL_THREADS