[libcxx] Externally threaded libc++ variant - Take 2

[rmaprath]

Summary

This is the replacement for D20328.

Rather than using opaque pointer types to delegate the internal representations of mutex / condvar / thread types, in this patch we expect the libc++ vendor to explicitly define the implementation types upfront (at library compile time). The API is still left open for platform vendors to implement, it's only the types that need to be agreed upon.

[bcraig]

CMakeLists.txt

131

I would prefer that you not change this option name, as that will break my builds. It's easy enough to fix on my side, but unless there's a really compelling reason, I'd rather not deal with the breakage.

include/__external_threading

26 ↗

(On Diff #62670)

So users of external pthreading (or their compiler driver) would need to provide a pile of -D options on the command line? Or is it expected that users will have their own __external_threading header that comes earlier in the include path? Please document your expectation.

include/__threading_support

33

More context would be useful here. "-U99999" perhaps (if using git).

[rmaprath]

Addressed review comments from @bcraig:

• Got rid of some of the option name cleanups (done to make them more consistent, but not relevant to the patch)
• Arranged it so that libc++ vendors can drop in __external_threading header to override the default setup
  • By default, __threading_support declares the function prototypes and relies on pthread types - this makes it possible to build+test an externally-threaded libc++ variant with vanilla sources. The implementation of the thread api (in test/support/external_threads.cpp) is separate from the library, the two are glued together by __threading_support.

[rmaprath]

Improve comment. NFC.

[bcraig]

LGTM. As usual, you'll want to get one of the code owner's sign off first though.

[rmaprath]

@mclow.lists, @EricWF: Gentle ping.

[rmaprath]

Minor cleanup + Ping.

[rmaprath]

Rebased.

[compnerd]

CMakeLists.txt

139

Can you use cmake_dependent_option here instead please?

include/__config

830

clang-format?

[rmaprath]

CMakeLists.txt

139

I have two problems with this:

• Currently we don't use the CMakeDepedentOption module anywhere else in the llvm project
• This option silently forces the cmake switch in question to the FORCE (last) argument when the dependency in question is not satisfied. Our current approach on the other hand is to issue an error to the user indicating the incompatible options (see hunk below).

I'm slightly leaning towards keeping the current approach. @EricWF: WDYT?

include/__config

830

Running clang-format -style=llvm on this file shows that it's going to do a lot of changes to it. Do we want that? Perhaps it should be a separate patch?

Not sure if this is what you meant.

[rmaprath]

@compnerd: I plan to look at D18482 soon (going to need it when this lands). Got moved to a new machine, still setting up the environment :)

@mclow.lists: Ping?

[mclow.lists]

include/__external_threading

26 ↗

(On Diff #62670)

My expectation is that system vendors who have a different threading implementation will ship a customized version of the libc++ headers and dylib.

The headers will include a replacement include/__external_threading, and a modified include/__config that contains additional #defines

What this patch does is to provide a customization point to produce such libraries.

[rmaprath]

include/__external_threading

26 ↗

(On Diff #62670)

This is correct.

With the latest revision, I (following @bcraig's comments) have made this a bit more simpler; system vendors can simply drop in the header include/__external_threading to override the default threading implementation. When this header is provided (and _LIBCPP_HAS_THREAD_API_EXTERNAL is defined - which is automatically wired into __config header through __config_site.in at build time), the built library will end up using the new threading implementation.

[rmaprath]

Comments from @mclow.lists on the latest revision (received offline):

• Need instructions on how to build and test the patch...
• Need to be able to build+test on Mac (looks like there is a small problem in the patch w.r.t pthreads that makes the build fail on Mac)
• Need to work on shared library builds as well

I will address these comments soon (as soon as I get familiar with how Macs work...)

[rmaprath]

Addressed above comments from @mclow.lists.

Getting this to work on shared library builds was a bit tricky, had to use -undefined dynamic_lookup linker option on OSX and strip off the default -Wl,-z,defs linker option on linux.

Now this should build+test on OSX with the usual cmake options.

[compnerd]

CMakeLists.txt

372

There is a similar change in swift to deal with this pollution :-(.

include/__config

830

I meant on the lines that you added :-).

include/__threading_support

294

I think we should explicitly spell out tls. The extra character won't hurt.

lib/CMakeLists.txt

216

Have you considered a slightly alternative approach of basically implementing pthreads as an "external" threading model as well? You could override it, or take the default implementation that is provided. It avoids having a second supporting DSO for threading as we could always statically link against it.

test/support/external_threads.cpp

27

I think this would be nicer as:

if (int error = pthread_mutexattr_init(&attr))
  return error;

53

I don't think that the label for failure is adding anything since you really are handling the destruction of the mutex at all failure sites.

71

Please choose a single style.

104

Please choose a single style (and below).

132

Can you compare to nullptr instead?

143

Same.

[rmaprath]

include/__config

830

Will update to reflect rest of the source file.

include/__threading_support

294

Will fix.

lib/CMakeLists.txt

216

So, the particulars of this design was discussed and (generally) agreed upon sometime back.

Currently pthreads is hidden behind the same threading API as the externally threaded variant (on this patch). To override the default (pthread) implementation, you just have to drop in an __external_threading header and specify the -DLIBCXX_HAS_EXTERNAL_THREAD_API cmake option.

There's no need to provide a second DSO if your __external_threading header (and your thread-system sources) are available at libc++ build time.

But we have a requirement that the library needs to be build-able in such a way you can provide a second DSO holding the implementation of the API. That is, we need to be able to build and ship a library to which customers can drop-in their own thread implementation (i.e. the thread-support DSO).

Note that in one of my earlier patches, I proposed an even more de-coupled external threading system where even the types like __libcpp_mutex_t were opaque pointers. @mclow.lists opposed that (perhaps rightfully so, now I think about it) and currently these types need to be known at libc++ compile time. The implementation of the API functions like __libcpp_mutex_lock() can be deferred to link time (this is what is provided in the second DSO).

Hope that clarifies everything here?

test/support/external_threads.cpp

27

I think I'm following the conventions used in the libc++ source files here. Will double check tomorrow.

53

Not sure what I did here. Will double check and fix.

71

Will fix.

104

Will fix.

132

I can't remember exactly why, but I think there was a problem with using nullptr here. I will check and get back.

143

Need to check.

[EricWF]

This looks great. Two comments:

1. The declarations should be used in both the inline and external pthread implementation. They also need visibility declarations.
2. Why can't we use the inline implementation to implement external_threads.cpp?

I took a stab at it here.

After that this LGTM.

lib/CMakeLists.txt

211

When this is building as a shared library it needs to do the -nodefaultlibs dance to prevent linking to libstdc++.so or other unwanted system libraries.

[mclow.lists]

The patch (again) doesn't apply cleanly (test/CMakeLists.txt), so I applied it manually.
After that, it builds w/o errors, and passes all the tests.

After addressing @EricWF's concerns, I think this is ready to land.

[rmaprath]

In D21968#534462, @EricWF wrote:

This looks great. Two comments:

1. The declarations should be used in both the inline and external pthread implementation. They also need visibility declarations.
2. Why can't we use the inline implementation to implement external_threads.cpp?

I took a stab at it here.

Yup, that looks better.

I will incorporate this change and also address the remaining comments from @compnerd before I commit.

Thanks all!

/ Asiri

[rmaprath]

Final patch incorporating all the changes from @EricWF and @compnerd.

Will commit tomorrow (@mclow.lists gave approval earlier)

/ Asiri

CMakeLists.txt

 option(LIBCXXABI_USE_LLVM_UNWINDER "Build and use the LLVM unwinder." OFF)
 
 # Target options --------------------------------------------------------------
 option(LIBCXX_BUILD_32_BITS "Build 32 bit libc++." ${LLVM_BUILD_32_BITS})
 set(LIBCXX_SYSROOT "" CACHE STRING "Use alternate sysroot.")
 set(LIBCXX_GCC_TOOLCHAIN "" CACHE STRING "Use alternate GCC toolchain.")
 
 # Feature options -------------------------------------------------------------
 option(LIBCXX_ENABLE_EXCEPTIONS "Use exceptions." ON)
 option(LIBCXX_ENABLE_RTTI "Use run time type information." ON)
 option(LIBCXX_ENABLE_GLOBAL_FILESYSTEM_NAMESPACE "Build libc++ with support for the global filesystem namespace." ON)
 option(LIBCXX_ENABLE_STDIN "Build libc++ with support for stdin/std::cin." ON)
 option(LIBCXX_ENABLE_STDOUT "Build libc++ with support for stdout/std::cout." ON)
 option(LIBCXX_ENABLE_THREADS "Build libc++ with support for threads." ON)
 option(LIBCXX_ENABLE_THREAD_UNSAFE_C_FUNCTIONS "Build libc++ with support for thread-unsafe C functions" ON)
 option(LIBCXX_ENABLE_MONOTONIC_CLOCK
   "Build libc++ with support for a monotonic clock.
    This option may only be set to OFF when LIBCXX_ENABLE_THREADS=OFF." ON)
 option(LIBCXX_HAS_MUSL_LIBC "Build libc++ with support for the Musl C library" OFF)
 option(LIBCXX_HAS_PTHREAD_API "Ignore auto-detection and force use of pthread API" OFF)
+option(LIBCXX_HAS_EXTERNAL_THREAD_API
+  "Build libc++ with an externalized threading API.
+   This option may only be set to ON when LIBCXX_ENABLE_THREADS=ON." OFF)
 
 # Misc options ----------------------------------------------------------------
 # FIXME: Turn -pedantic back ON. It is currently off because it warns
 # about #include_next which is used everywhere.
 option(LIBCXX_ENABLE_PEDANTIC "Compile with pedantic enabled." OFF)
 option(LIBCXX_ENABLE_WERROR "Fail and stop if a warning is triggered." OFF)
 option(LIBCXX_DISABLE_MACRO_CONFLICT_WARNINGS "Disable #warnings about conflicting macros." OFF)
 

                       " when LIBCXX_ENABLE_THREADS is also set to OFF.")
 endif()
 
 if(LIBCXX_HAS_PTHREAD_API AND NOT LIBCXX_ENABLE_THREADS)
   message(FATAL_ERROR "LIBCXX_HAS_PTHREAD_API can only be set to ON"
                       " when LIBCXX_ENABLE_THREADS is also set to ON.")
 endif()
 
+if(LIBCXX_HAS_EXTERNAL_THREAD_API AND NOT LIBCXX_ENABLE_THREADS)
+  message(FATAL_ERROR "LIBCXX_HAS_EXTERNAL_THREAD_API can only be set to ON"
+                      " when LIBCXX_ENABLE_THREADS is also set to ON.")
+endif()
+
 # Ensure LLVM_USE_SANITIZER is not specified when LIBCXX_GENERATE_COVERAGE
 # is ON.
 if (LLVM_USE_SANITIZER AND LIBCXX_GENERATE_COVERAGE)
   message(FATAL_ERROR "LLVM_USE_SANITIZER cannot be used with LIBCXX_GENERATE_COVERAGE")
 endif()
 
 # Set LIBCXX_BUILD_32_BITS to (LIBCXX_BUILD_32_BITS OR LLVM_BUILD_32_BITS)
 # and check that we can build with 32 bits if requested.

 # Exception flags =============================================================
 if (LIBCXX_ENABLE_EXCEPTIONS)
   # Catches C++ exceptions only and tells the compiler to assume that extern C
   # functions never throw a C++ exception.
   add_compile_flags_if_supported(-EHsc)
 else()
   add_definitions(-D_LIBCPP_NO_EXCEPTIONS)
   add_compile_flags_if_supported(-EHs- -EHa-)
   add_compile_flags_if_supported(-fno-exceptions)
 endif()
 
 # RTTI flags ==================================================================
 if (NOT LIBCXX_ENABLE_RTTI)
   add_definitions(-D_LIBCPP_NO_RTTI)
   add_compile_flags_if_supported(-GR-)
   add_compile_flags_if_supported(-fno-rtti)
 endif()
 
+# Threading flags =============================================================
+if (LIBCXX_HAS_EXTERNAL_THREAD_API AND LIBCXX_ENABLE_SHARED)
+  # Need to allow unresolved symbols if this is to work with shared library builds
+  if (APPLE)
+    add_link_flags("-undefined dynamic_lookup")
+  else()
+    # Relax this restriction from HandleLLVMOptions
+    string(REPLACE "-Wl,-z,defs" "" CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS}")
+  endif()
+endif()
+
 # Assertion flags =============================================================
 define_if(LIBCXX_ENABLE_ASSERTIONS -UNDEBUG)
 define_if_not(LIBCXX_ENABLE_ASSERTIONS -DNDEBUG)
 if (LIBCXX_ENABLE_ASSERTIONS)
   # MSVC doesn't like _DEBUG on release builds. See PR 4379.
   define_if_not(MSVC -D_DEBUG)
 endif()
 

 config_define_if_not(LIBCXX_ENABLE_GLOBAL_FILESYSTEM_NAMESPACE _LIBCPP_HAS_NO_GLOBAL_FILESYSTEM_NAMESPACE)
 config_define_if_not(LIBCXX_ENABLE_STDIN _LIBCPP_HAS_NO_STDIN)
 config_define_if_not(LIBCXX_ENABLE_STDOUT _LIBCPP_HAS_NO_STDOUT)
 config_define_if_not(LIBCXX_ENABLE_THREADS _LIBCPP_HAS_NO_THREADS)
 config_define_if_not(LIBCXX_ENABLE_MONOTONIC_CLOCK _LIBCPP_HAS_NO_MONOTONIC_CLOCK)
 config_define_if_not(LIBCXX_ENABLE_THREAD_UNSAFE_C_FUNCTIONS _LIBCPP_HAS_NO_THREAD_UNSAFE_C_FUNCTIONS)
 
 config_define_if(LIBCXX_HAS_PTHREAD_API _LIBCPP_HAS_THREAD_API_PTHREAD)
+config_define_if(LIBCXX_HAS_EXTERNAL_THREAD_API _LIBCPP_HAS_THREAD_API_EXTERNAL)
 config_define_if(LIBCXX_HAS_MUSL_LIBC _LIBCPP_HAS_MUSL_LIBC)
 
 if (LIBCXX_NEEDS_SITE_CONFIG)
   configure_file(
     include/__config_site.in
     ${LIBCXX_BINARY_DIR}/__config_site
     @ONLY)
   # Provide the config definitions by included the generated __config_site

include/__config

    (__GNUC_MINOR__ >= 3 || __GNUC_PATCHLEVEL__ >= 2))) && !defined(__GXX_RTTI)
 #    define _LIBCPP_NO_RTTI
 #  elif (defined(_MSC_VER) && !defined(__clang__)) && !defined(_CPPRTTI)
 #    define _LIBCPP_NO_RTTI
 #  endif
 #endif
 
 #ifndef _LIBCPP_WEAK
 #  define _LIBCPP_WEAK __attribute__((__weak__))
 #endif
 
 // Thread API
-#if !defined(_LIBCPP_HAS_NO_THREADS) && !defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
+#if !defined(_LIBCPP_HAS_NO_THREADS) && \
+    !defined(_LIBCPP_HAS_THREAD_API_PTHREAD) && \
+    !defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
 # if defined(__FreeBSD__) || \
     defined(__NetBSD__) || \
     defined(__linux__) || \
     defined(__APPLE__) || \
     defined(__CloudABI__) || \
     defined(__sun__)
 #  define _LIBCPP_HAS_THREAD_API_PTHREAD
 # else
 #  error "No thread API"
 # endif // _LIBCPP_HAS_THREAD_API
 #endif // _LIBCPP_HAS_NO_THREADS
 
 #if defined(_LIBCPP_HAS_NO_THREADS) && defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
 #  error _LIBCPP_HAS_THREAD_API_PTHREAD may only be defined when \
          _LIBCPP_HAS_NO_THREADS is not defined.
 #endif
 
+#if defined(_LIBCPP_HAS_NO_THREADS) && defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+#  error _LIBCPP_HAS_EXTERNAL_THREAD_API may not be defined when \
+         _LIBCPP_HAS_NO_THREADS is defined.
+#endif
+
 #if defined(_LIBCPP_HAS_NO_MONOTONIC_CLOCK) && !defined(_LIBCPP_HAS_NO_THREADS)
 #  error _LIBCPP_HAS_NO_MONOTONIC_CLOCK may only be defined when \
          _LIBCPP_HAS_NO_THREADS is defined.
 #endif
 
 // Systems that use capability-based security (FreeBSD with Capsicum,
 // Nuxi CloudABI) may only provide local filesystem access (using *at()).
 // Functions like open(), rename(), unlink() and stat() should not be

include/__config_site.in

 #cmakedefine _LIBCPP_HAS_NO_GLOBAL_FILESYSTEM_NAMESPACE
 #cmakedefine _LIBCPP_HAS_NO_STDIN
 #cmakedefine _LIBCPP_HAS_NO_STDOUT
 #cmakedefine _LIBCPP_HAS_NO_THREADS
 #cmakedefine _LIBCPP_HAS_NO_MONOTONIC_CLOCK
 #cmakedefine _LIBCPP_HAS_NO_THREAD_UNSAFE_C_FUNCTIONS
 #cmakedefine _LIBCPP_HAS_MUSL_LIBC
 #cmakedefine _LIBCPP_HAS_THREAD_API_PTHREAD
+#cmakedefine _LIBCPP_HAS_THREAD_API_EXTERNAL
 
 #endif // _LIBCPP_CONFIG_SITE

include/__threading_support

 #include <__config>
 
 #ifndef _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
 #pragma GCC system_header
 #endif
 
 #ifndef _LIBCPP_HAS_NO_THREADS
 
-#if defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
+#if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL) && (!defined(__has_include) || __has_include(<__external_threading>))
+#include <__external_threading>
+#else
 #include <pthread.h>
 #include <sched.h>
 #endif
 
-_LIBCPP_BEGIN_NAMESPACE_STD
+#if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
+#define _LIBCPP_THREAD_ABI_VISIBILITY _LIBCPP_FUNC_VIS
+#else
+#define _LIBCPP_THREAD_ABI_VISIBILITY inline _LIBCPP_INLINE_VISIBILITY
+#endif
 
-#if defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
+_LIBCPP_BEGIN_NAMESPACE_STD
 
 // Mutex
-#define _LIBCPP_MUTEX_INITIALIZER PTHREAD_MUTEX_INITIALIZER
 typedef pthread_mutex_t __libcpp_mutex_t;
+#define _LIBCPP_MUTEX_INITIALIZER PTHREAD_MUTEX_INITIALIZER
+
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_mutex_lock(__libcpp_mutex_t* __m);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_mutex_trylock(__libcpp_mutex_t* __m);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_mutex_unlock(__libcpp_mutex_t* __m);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_mutex_destroy(__libcpp_mutex_t* __m);
+
+// Condition variable
+typedef pthread_cond_t __libcpp_condvar_t;
+#define _LIBCPP_CONDVAR_INITIALIZER PTHREAD_COND_INITIALIZER
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_condvar_signal(__libcpp_condvar_t* __cv);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_condvar_broadcast(__libcpp_condvar_t* __cv);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_condvar_wait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_condvar_timedwait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m, timespec* __ts);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_condvar_destroy(__libcpp_condvar_t* __cv);
+
+// Thread id
+typedef pthread_t __libcpp_thread_id;
+_LIBCPP_THREAD_ABI_VISIBILITY
+bool __libcpp_thread_id_equal(__libcpp_thread_id t1, __libcpp_thread_id t2);
+_LIBCPP_THREAD_ABI_VISIBILITY
+bool __libcpp_thread_id_less(__libcpp_thread_id t1, __libcpp_thread_id t2);
+
+// Thread
+typedef pthread_t __libcpp_thread_t;
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg);
+_LIBCPP_THREAD_ABI_VISIBILITY
+__libcpp_thread_id __libcpp_thread_get_current_id();
+_LIBCPP_THREAD_ABI_VISIBILITY
+__libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t* __t);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_thread_join(__libcpp_thread_t* __t);
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_thread_detach(__libcpp_thread_t* __t);
+_LIBCPP_THREAD_ABI_VISIBILITY
+void __libcpp_thread_yield();
+
+// Thread local storage
+typedef pthread_key_t __libcpp_tls_key;
+_LIBCPP_THREAD_ABI_VISIBILITY
+int __libcpp_tls_create(__libcpp_tls_key* __key, void (*__at_exit)(void*));
+_LIBCPP_THREAD_ABI_VISIBILITY
+void* __libcpp_tls_get(__libcpp_tls_key __key);
+_LIBCPP_THREAD_ABI_VISIBILITY
+void __libcpp_tls_set(__libcpp_tls_key __key, void* __p);
+
+#if defined(_LIBCPP_HAS_THREAD_API_PTHREAD) || defined(_LIBCPP_BUILDING_EXTERNAL_THREADS)
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m)
 {
     pthread_mutexattr_t attr;
     int __ec = pthread_mutexattr_init(&attr);
     if (__ec) return __ec;
     __ec = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
     if (__ec)
     {
int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m)
     if (__ec)
     {
         pthread_mutex_destroy(__m);
         return __ec;
     }
     return 0;
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_mutex_lock(__libcpp_mutex_t* __m)
 {
     return pthread_mutex_lock(__m);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_mutex_trylock(__libcpp_mutex_t* __m)
 {
     return pthread_mutex_trylock(__m);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_mutex_unlock(__libcpp_mutex_t* __m)
 {
     return pthread_mutex_unlock(__m);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_mutex_destroy(__libcpp_mutex_t* __m)
 {
     return pthread_mutex_destroy(__m);
 }
 
 // Condition variable
-#define _LIBCPP_CONDVAR_INITIALIZER PTHREAD_COND_INITIALIZER
-typedef pthread_cond_t __libcpp_condvar_t;
-
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_condvar_signal(__libcpp_condvar_t* __cv)
 {
     return pthread_cond_signal(__cv);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_condvar_broadcast(__libcpp_condvar_t* __cv)
 {
     return pthread_cond_broadcast(__cv);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_condvar_wait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m)
 {
     return pthread_cond_wait(__cv, __m);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_condvar_timedwait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m, timespec* __ts)
 {
     return pthread_cond_timedwait(__cv, __m, __ts);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_condvar_destroy(__libcpp_condvar_t* __cv)
 {
     return pthread_cond_destroy(__cv);
 }
 
-// Thread id
-typedef pthread_t __libcpp_thread_id;
-
 // Returns non-zero if the thread ids are equal, otherwise 0
-inline _LIBCPP_ALWAYS_INLINE
 bool __libcpp_thread_id_equal(__libcpp_thread_id t1, __libcpp_thread_id t2)
 {
     return pthread_equal(t1, t2) != 0;
 }
 
 // Returns non-zero if t1 < t2, otherwise 0
-inline _LIBCPP_ALWAYS_INLINE
 bool __libcpp_thread_id_less(__libcpp_thread_id t1, __libcpp_thread_id t2)
 {
     return t1 < t2;
 }
 
 // Thread
-typedef pthread_t __libcpp_thread_t;
-
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg)
 {
     return pthread_create(__t, 0, __func, __arg);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 __libcpp_thread_id __libcpp_thread_get_current_id()
 {
     return pthread_self();
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 __libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t* __t)
 {
     return *__t;
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_thread_join(__libcpp_thread_t* __t)
 {
     return pthread_join(*__t, 0);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 int __libcpp_thread_detach(__libcpp_thread_t* __t)
 {
     return pthread_detach(*__t);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
 void __libcpp_thread_yield()
 {
     sched_yield();
 }
 
 // Thread local storage
-typedef pthread_key_t __libcpp_tl_key;
-
-inline _LIBCPP_ALWAYS_INLINE
-int __libcpp_tl_create(__libcpp_tl_key* __key, void (*__at_exit)(void*))
+int __libcpp_tls_create(__libcpp_tls_key* __key, void (*__at_exit)(void*))
 {
     return pthread_key_create(__key, __at_exit);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
-void* __libcpp_tl_get(__libcpp_tl_key __key)
+void* __libcpp_tls_get(__libcpp_tls_key __key)
 {
     return pthread_getspecific(__key);
 }
 
-inline _LIBCPP_ALWAYS_INLINE
-void __libcpp_tl_set(__libcpp_tl_key __key, void* __p)
+void __libcpp_tls_set(__libcpp_tls_key __key, void* __p)
 {
     pthread_setspecific(__key, __p);
 }
 
-#else // !_LIBCPP_HAS_THREAD_API_PTHREAD
-  #error "No thread API selected."
-#endif
+#endif // _LIBCPP_HAS_THREAD_API_PTHREAD || _LIBCPP_BUILDING_EXTERNAL_THREADS
 
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // _LIBCPP_HAS_NO_THREADS
 
 #endif // _LIBCPP_THREADING_SUPPORT

include/thread

 
     void notify_all_at_thread_exit(condition_variable*, mutex*);
     void __make_ready_at_thread_exit(__assoc_sub_state*);
 };
 
 template <class _Tp>
 class __thread_specific_ptr
 {
-    __libcpp_tl_key __key_;
+    __libcpp_tls_key __key_;
 
      // Only __thread_local_data() may construct a __thread_specific_ptr
      // and only with _Tp == __thread_struct.
     static_assert((is_same<_Tp, __thread_struct>::value), "");
     __thread_specific_ptr();
     friend _LIBCPP_FUNC_VIS __thread_specific_ptr<__thread_struct>& __thread_local_data();
 
     __thread_specific_ptr(const __thread_specific_ptr&);
     __thread_specific_ptr& operator=(const __thread_specific_ptr&);
 
     static void __at_thread_exit(void*);
 public:
     typedef _Tp* pointer;
 
     ~__thread_specific_ptr();
 
     _LIBCPP_INLINE_VISIBILITY
-    pointer get() const {return static_cast<_Tp*>(__libcpp_tl_get(__key_));}
+    pointer get() const {return static_cast<_Tp*>(__libcpp_tls_get(__key_));}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator*() const {return *get();}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator->() const {return get();}
     void set_pointer(pointer __p);
 };
 
 template <class _Tp>
 void
 __thread_specific_ptr<_Tp>::__at_thread_exit(void* __p)
 {
     delete static_cast<pointer>(__p);
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::__thread_specific_ptr()
 {
-    int __ec = __libcpp_tl_create(
+    int __ec = __libcpp_tls_create(
         &__key_,
         &__thread_specific_ptr::__at_thread_exit);
     if (__ec)
         __throw_system_error(__ec,
                            "__thread_specific_ptr construction failed");
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::~__thread_specific_ptr()
 {
     // __thread_specific_ptr is only created with a static storage duration
     // so this destructor is only invoked during program termination. Invoking
     // pthread_key_delete(__key_) may prevent other threads from deleting their
     // thread local data. For this reason we leak the key.
 }
 
 template <class _Tp>
 void
 __thread_specific_ptr<_Tp>::set_pointer(pointer __p)
 {
     _LIBCPP_ASSERT(get() == nullptr,
                    "Attempting to overwrite thread local data");
-    __libcpp_tl_set(__key_, __p);
+    __libcpp_tls_set(__key_, __p);
 }
 
 class _LIBCPP_TYPE_VIS thread;
 class _LIBCPP_TYPE_VIS __thread_id;
 
 namespace this_thread
 {
 

lib/CMakeLists.txt

   endif()
   set_target_properties(cxx_experimental
     PROPERTIES
       COMPILE_FLAGS "${experimental_flags}"
       OUTPUT_NAME   "c++experimental"
   )
 endif()
 
+if (LIBCXX_HAS_EXTERNAL_THREAD_API)
+  file(GLOB LIBCXX_EXTERNAL_THREADING_SUPPORT_SOURCES ../test/support/external_threads.cpp)
+
+  if (LIBCXX_ENABLE_SHARED)
+    add_library(cxx_external_threads SHARED ${LIBCXX_EXTERNAL_THREADING_SUPPORT_SOURCES})
+  else()
+    add_library(cxx_external_threads STATIC ${LIBCXX_EXTERNAL_THREADING_SUPPORT_SOURCES})
+  endif()
+
+  set_target_properties(cxx_external_threads
+    PROPERTIES
+      LINK_FLAGS    "${LIBCXX_LINK_FLAGS}"
+      COMPILE_FLAGS "${LIBCXX_COMPILE_FLAGS}"
+      OUTPUT_NAME   "c++external_threads"
+  )
+endif()
+
 # Generate a linker script inplace of a libc++.so symlink. Rerun this command
 # after cxx builds.
 if (LIBCXX_ENABLE_SHARED AND LIBCXX_ENABLE_ABI_LINKER_SCRIPT)
   # Get the name of the ABI library and handle the case where CXXABI_LIBNAME
   # is a target name and not a library. Ex cxxabi_shared.
   set(SCRIPT_ABI_LIBNAME "${LIBCXX_CXX_ABI_LIBRARY}")
   if (SCRIPT_ABI_LIBNAME STREQUAL "cxxabi_shared")
     set(SCRIPT_ABI_LIBNAME "c++abi")

test/CMakeLists.txt

 pythonize_bool(LIBCXX_ENABLE_RTTI)
 pythonize_bool(LIBCXX_ENABLE_SHARED)
 pythonize_bool(LIBCXX_BUILD_32_BITS)
 pythonize_bool(LIBCXX_GENERATE_COVERAGE)
 pythonize_bool(LIBCXXABI_ENABLE_SHARED)
 pythonize_bool(LIBCXXABI_USE_LLVM_UNWINDER)
 pythonize_bool(LIBCXX_HAS_ATOMIC_LIB)
 pythonize_bool(LIBCXX_HAVE_CXX_ATOMICS_WITH_LIB)
+pythonize_bool(LIBCXX_HAS_EXTERNAL_THREAD_API)
 
 # The tests shouldn't link to any ABI library when it has been linked into
 # libc++ statically or via a linker script.
 if (LIBCXX_ENABLE_STATIC_ABI_LIBRARY OR LIBCXX_ENABLE_ABI_LINKER_SCRIPT)
   set(LIBCXX_CXX_ABI_LIBNAME "none")
 endif()
 
 # By default, for non-standalone builds, libcxx and libcxxabi share a library
configure_file(
   ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
   ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
   @ONLY)
 
 if (LIBCXX_ENABLE_EXPERIMENTAL_LIBRARY)
   set(experimental_dep cxx_experimental)
 endif()
 
+if (LIBCXX_HAS_EXTERNAL_THREAD_API)
+  set(external_threads_dep cxx_external_threads)
+endif()
+
 add_lit_testsuite(check-cxx
   "Running libcxx tests"
   ${CMAKE_CURRENT_BINARY_DIR}
-  DEPENDS cxx ${experimental_dep})
+  DEPENDS cxx ${experimental_dep} ${external_threads_dep})
 
 add_custom_target(check-libcxx DEPENDS check-cxx)
 
 if (LIBCXX_GENERATE_COVERAGE)
   include(CodeCoverage)
   set(output_dir "${CMAKE_CURRENT_BINARY_DIR}/coverage")
   set(capture_dirs "${LIBCXX_LIB_CMAKEFILES_DIR}/cxx.dir/;${LIBCXX_LIB_CMAKEFILES_DIR}/cxx_experimental.dir/;${CMAKE_CURRENT_BINARY_DIR}")
   set(extract_dirs "${LIBCXX_SOURCE_DIR}/include;${LIBCXX_SOURCE_DIR}/src")
   setup_lcov_test_target_coverage("cxx" "${output_dir}" "${capture_dirs}" "${extract_dirs}")
 endif()

test/libcxx/test/config.py

             self.cxx.link_flags += ['-lc++']
         else:
             cxx_library_root = self.get_lit_conf('cxx_library_root')
             if cxx_library_root:
                 abs_path = os.path.join(cxx_library_root, 'libc++.a')
                 self.cxx.link_flags += [abs_path]
             else:
                 self.cxx.link_flags += ['-lc++']
+        # This needs to come after -lc++ as we want its unresolved thread-api symbols
+        # to be picked up from this one.
+        if self.get_lit_bool('libcxx_external_thread_api', default=False):
+            self.cxx.link_flags += ['-lc++external_threads']
 
     def configure_link_flags_abi_library(self):
         cxx_abi = self.get_lit_conf('cxx_abi', 'libcxxabi')
         if cxx_abi == 'libstdc++':
             self.cxx.link_flags += ['-lstdc++']
         elif cxx_abi == 'libsupc++':
             self.cxx.link_flags += ['-lsupc++']
         elif cxx_abi == 'libcxxabi':

test/libcxx/thread/thread.condition/thread.condition.condvar/native_handle.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <condition_variable>
 
 // class condition_variable;
 
 // typedef pthread_cond_t* native_handle_type;
 // native_handle_type native_handle();
 

test/libcxx/thread/thread.mutex/thread.mutex.requirements/thread.mutex.requirements.mutex/thread.mutex.class/native_handle.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <mutex>
 
 // class mutex;
 
 // typedef pthread_mutex_t* native_handle_type;
 // native_handle_type native_handle();
 

test/libcxx/thread/thread.mutex/thread.mutex.requirements/thread.mutex.requirements.mutex/thread.mutex.recursive/native_handle.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <mutex>
 
 // class recursive_mutex;
 
 // typedef pthread_mutex_t* native_handle_type;
 // native_handle_type native_handle();
 

test/libcxx/thread/thread.threads/thread.thread.class/thread.thread.member/native_handle.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <thread>
 
 // class thread
 
 // native_handle_type native_handle();
 
 #include <thread>

test/libcxx/thread/thread.threads/thread.thread.class/types.pass.cpp

 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// UNSUPPORTED: libcpp-has-no-threads
+// UNSUPPORTED: libcpp-has-no-threads, libcpp-has-thread-api-external
 
 // <thread>
 
 // class thread
 // {
 // public:
 //     typedef pthread_t native_handle_type;
 //     ...

test/lit.site.cfg.in

 config.generate_coverage        = "@LIBCXX_GENERATE_COVERAGE@"
 config.target_info              = "@LIBCXX_TARGET_INFO@"
 config.executor                 = "@LIBCXX_EXECUTOR@"
 config.llvm_unwinder            = "@LIBCXXABI_USE_LLVM_UNWINDER@"
 config.has_libatomic            = "@LIBCXX_HAS_ATOMIC_LIB@"
 config.use_libatomic            = "@LIBCXX_HAVE_CXX_ATOMICS_WITH_LIB@"
 
 config.libcxxabi_shared         = "@LIBCXXABI_ENABLE_SHARED@"
+config.libcxx_external_thread_api  = "@LIBCXX_HAS_EXTERNAL_THREAD_API@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@LIBCXX_SOURCE_DIR@/test/lit.cfg")

test/support/external_threads.cpp

+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+#define _LIBCPP_BUILDING_EXTERNAL_THREADS
+#include <__threading_support>