[libcxx] Refactor pthread usage - II

[rmaprath]

Summary

This is mostly D11781 with the final review comments addressed:

• Merged all the headers into a single __os_support header
• Moved all internal functions (those only used by the library sources) away from the headers. This required adding a few #ifdefs into the library sources to select between the thread API.

Note that this is only a refactoring, in that it isolates pthread usage of libcxx allowing anyone to easily plug-in a different thread implementation into libcxx sources.

The final goal of this work (at least for us) is a bit more involved: we want to allow libcxx users to plug-in their own thread implementation at compile time. I have a patch for this which builds on the current one, I will be uploading it soon for comments.

[rmaprath]

Added bit more context to the diff.

[rmaprath]

Minor cleanup + Gentle ping.

[bcraig]

src/mutex.cpp

31

Can't we just check for _LIBCPP_THREAD_API_PTHREAD once at the top of the file and #error as necessary there? I don't get the value of multiple #errors for the same condition, but sprinkled throughout the implementation. I do see the cost in the strategy, in that there are now lots of places that can bit-rot.

[rmaprath]

Addressing review comments from @bcraig:

In the earlier patch, I tried to keep the __os_support header to a minimum by not exposing the internal pthread dependencies (in library sources) in this header. This however blew up on me when externalizing the threading support in D19415, where I ended up sprinkling a lot of:

#if defined(_LIBCPP_THREAD_API_PTHREAD)
  // Do pthread thing
#elif defined(_LIBCPP_THREAD_API_EXTERNAL)
  // Do other thing
#else
  // Fault
#endif

Perhaps the mistake was to view these two threading APIs as unrelated. After hiding all pthread dependencies behind the corresponding __os_xxxx functions (as suggested), everything falls into place (D19415 trivially becomes the list of __os_xxxx function declarations).

I have addressed the remaining review comments along the way. Will update D19415 soon.

@bcraig: Hope this is much better?

[bcraig]

LGTM. You will still need to get approval from one of your original reviewers though.

[mclow.lists]

In general, I'm happy with this direction. I think we need to check that there are no ABI changes that happen here (I don't see any, but I'll keep looking), and I think the stuff in <__os_support> should be marked "always inline"

On a bikeshed note: is <__os_support> the right name? Or should it be something like <__thread> or <__threading_support>?

include/__os_support

32 ↗

(On Diff #55119)

These should all be marked with _LIBCPP_ALWAYS_INLINE.

src/algorithm.cpp

51

What happened to the initializer here?

src/memory.cpp

130

How does this array get initialized now?

src/mutex.cpp

201

These two variables used to be initialized. Now they're not.

[rmaprath]

On a bikeshed note: is <__os_support> the right name? Or should it be something like <__thread> or <__threading_support>?

I went with __os_support in case if we'd want to group further external dependencies (like, for example, if some non-c POSIX calls are used in the upcoming filesystem library). I can revert back to __threading_support if that's preferred.

Thanks.

/ Asiri

include/__os_support

32 ↗

(On Diff #55119)

Got it. Will update.

src/algorithm.cpp

51

I'm expecting the constructor of mutex to be run here at load time (before main). Note that this a libc++ mutex rather than a pthreads mutex (which does not required a constructor call like this). Would that be too much of an overhead?

src/memory.cpp

130

Same as above, but bigger (this now requires 16 constructor calls during load time). I can check if clang would figure out that this is a trivial construction (at least for the default pthreads implementation) and be able to avoid the constructor calls altogether, but may be relying on such behaviour is not good.

src/mutex.cpp

201

Same, relying on initialization at load time.

[bcraig]

src/algorithm.cpp

51

std::mutex's default ctor is constexpr. As long as the compiler supports constexpr, this initialization shouldn't require runtime code.

[rmaprath]

src/algorithm.cpp

51

Missed that!. The same applied to std::condition_variable (used below). From the sources it looks like we do cater for compilers that do not support constexpr, so for those compilers it would depend on how clever they are in optimizing out this kind of trivial constructor calls.

[rmaprath]

Addressing review comments by @mclow.lists:

• Switched to _LIBCPP_ALWAYS_INLINE from _LIBCPP_INLINE_VISIBILITY for all __os_xxx functions.

I've left the remaining points (naming of __os_support header and initialization of internal mutex / condition_variable instances) untouched pending further discussion.

/ Asiri

[bcraig]

In D19412#414374, @rmaprath wrote:

On a bikeshed note: is <__os_support> the right name? Or should it be something like <__thread> or <__threading_support>?

I went with __os_support in case if we'd want to group further external dependencies (like, for example, if some non-c POSIX calls are used in the upcoming filesystem library). I can revert back to __threading_support if that's preferred.

Thanks.

/ Asiri

I don't particularly care for <os_support> either. Either of <thread> or <__threading_support> are fine with me.

Note that there are already other OS specific calls that libcxx has to handle. A lot of the locale code wants to use BSD style foo_l functions that are present on Linux. I prefer to keep the locale code independent of the threading code, as the two don't overlap much.

For filesystem related items, I would expect those to go under <filesystem> or <filesystem_support>, or something similar.

[rmaprath]

Renamed __os_support to __threading_support as suggested by @mclow.lists and @bcraig. I've left the namespace name __libcpp_os_support untouched, can change it to __libcpp_threading_support if the latter is preferred.

Thanks!

/ Asiri

[bcraig]

LGTM. You'll still need to wait for one of the other reviewers though (probably @mclow.lists )

[EricWF]

The patch looks good for the most part. Nit picking I would rather have __libcpp_thread_foo instead of __libcpp_os_support::__os_thread_foo. IMO the namespace is undeeded.
@mclow.lists can you weigh in on this?

include/__mutex_base

43

Similar to the reason we choose __libcpp_mutex these macros should have a _LIBCPP prefix as well instead of an _OS one.

285

Shouldn't this be __cond_var_type?

[rmaprath]

Addressing review comments from @EricWF:

• Agree with all the suggested changes, I've applied them all.

Thanks!

/ Asiri

[EricWF]

OK. I *think* this is my last round of review comments except for one specific issue. I'm still looking into the changes to the static mutex's and condition_variables in memory.cpp and algorithm.cpp. In these cases I don't want to go from compile-time initialization to run-time initialization. This could introduce the static initialization order fiasco.

include/__mutex_base

37–38

Why not use __libcpp_mutex_t directly? Sorry for yet another renaming comment.

43

Only one underscore in _LIBCPP_MUTEX_INITIALIZER

include/__threading_support

12

_LIBCPP_THREADING_SUPPORT

14

#include <__config>

#ifndef _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
#pragma GCC system_header
#endif

31

Two comments:

1. This is only used to initialize a recursive mutex. Let's simply name this __libcpp_recursive_mutex_init and get rid of the bool parameter.
2. This is only used in recursive_mutex.cpp. It's OK to keep this in __threading_support for now but I'll probably want to move this out of the headers later.

34

Nit:

Let's invert the control flow here:

if (ec) return ec;
ec = pthread_mutexattr_settype(...);

Then at the end we can just return 0;

83

Nit: _LIBCPP_CONDVAR_INITIALIZER

120

Can we split this into __libcpp_thread_id_equal and __libcpp_thread_id_less?

130

This signature needs to work in C++03. Let's make the signature __libcpp_thread_create(__libcpp_thread_t* __t, _Func* __f, _Arg* __arg) because both _Func and _Arg should be pointers so there is no need to perfect forward them.

170

This signature needs to work in C++03. Make the signature __libcpp_tl_create(__libcpp_tl_key*, _Func*) since the function must be a pointer.

195

_LIBCPP_THREADING_SUPPORT

src/mutex.cpp

228

This change seems incorrect. There are some paths below where we manually unlock mut before a throw or return. Using unique_lock will result in a double unlock.

I think the fix is to change all mut.lock() and mut.unlock() calls into lk.lock()/lk.unlock() calls.

[rmaprath]

In D19412#416183, @EricWF wrote:

OK. I *think* this is my last round of review comments except for one specific issue. I'm still looking into the changes to the static mutex's and condition_variables in memory.cpp and algorithm.cpp. In these cases I don't want to go from compile-time initialization to run-time initialization. This could introduce the static initialization order fiasco.

So, as pointed out earlier by @bcraig, there won't be a runtime overhead for those compilers supporting constexr. For those that don't, this is still a trivial enough constructor call that is very likely to get optimized away. To elaborate, we can simplify this scenario to something like:

thread.h

struct thread_t {
  int state1;
  int *state2;
  bool state3;
};

#define THREAD_INITIALIZER  {0, 0, false}

test.cpp

#include "thread.h"

class Foo {
private:
  thread_t __t_;
public:
  Foo() {__t_ = (thread_t) THREAD_INITIALIZER;}
};

Foo f();

int main() {
  return 0;
}

Compiling this with either clang or gcc in -std=c++03 does not introduce an entry in the .init_array section. This can be observed with:

> clang++ -std=c++03 -c test.cpp
> objdump -j .init_array -dxs test.o

On the other hand, if you slightly change the constructor to take a parameter, or print something into std::cout, there will be an entry in the .init_array.

Moreover, even if some compiler naively emits a runtime initialization code for these constructors (mutex and condition_variable), I don't see how that can lead to problems related to static initialization order - all these constructors do is simply copy some memory, at worse, the compiler may emit a call to memcpy - but memcpy does not (AFAIK) depend on any other library state. In other words, these constructors don't require any other parts of the library having being initialized first.

I hope that clears it?

I have addressed the rest of you comments. Two minor nits:

• Can we keep __libcpp_mutex_t (used in __threading_support) and __mutex_type (used in __mutex_base) separate? I feel it's cleaner to make a distinction between the types of the threading API and the rest of the library. I don't have a strong opinion though, let me know if you'd like it changed still.
• I'd rather not move __libcpp_recursive_mutex_init back into recursive_mutex.cpp, as this would introduce pthread dependencies (pthread_mutexattr_xxx) back into the library sources. It's possible to workaround that by adding further __libcpp_mutexattr_xxx functions to the threading API, but those seem to be bit too pthread-specific. Again, no strong preference.

Thanks!

/ Asiri

[rmaprath]

Minor tweak: Got rid of the unnecessary template parameters on __libcpp_thread_create and __libcpp_tl_create.

[rmaprath]

Missed one: s/_LIBCPP_COND_INITIALIZER/_LIBCPP_CONDVAR_INITIALIZER

[rmaprath]

Added missing __confg header include.

[EricWF]

In D19412#416596, @rmaprath wrote:

In D19412#416183, @EricWF wrote:

OK. I *think* this is my last round of review comments except for one specific issue. I'm still looking into the changes to the static mutex's and condition_variables in memory.cpp and algorithm.cpp. In these cases I don't want to go from compile-time initialization to run-time initialization. This could introduce the static initialization order fiasco.

So, as pointed out earlier by @bcraig, there won't be a runtime overhead for those compilers supporting constexr. For those that don't, this is still a trivial enough constructor call that is very likely to get optimized away. To elaborate, we can simplify this scenario to something like:

thread.h

struct thread_t {
  int state1;
  int *state2;
  bool state3;
};

#define THREAD_INITIALIZER  {0, 0, false}

test.cpp

#include "thread.h"

class Foo {
private:
  thread_t __t_;
public:
  Foo() {__t_ = (thread_t) THREAD_INITIALIZER;}
};

Foo has a trivial destructor. Our mutex does not. I've already looked into the codegen for clang and in each case your change creates runtime code. The constructors may still be optimized away but it still has to register the destructors.

I'm just trying to figure out if that's going to be a problem.

Example here: https://godbolt.org/g/dJL29v

[rmaprath]

In D19412#417682, @EricWF wrote:

In D19412#416596, @rmaprath wrote:

In D19412#416183, @EricWF wrote:

OK. I *think* this is my last round of review comments except for one specific issue. I'm still looking into the changes to the static mutex's and condition_variables in memory.cpp and algorithm.cpp. In these cases I don't want to go from compile-time initialization to run-time initialization. This could introduce the static initialization order fiasco.

So, as pointed out earlier by @bcraig, there won't be a runtime overhead for those compilers supporting constexr. For those that don't, this is still a trivial enough constructor call that is very likely to get optimized away. To elaborate, we can simplify this scenario to something like:

thread.h

struct thread_t {
  int state1;
  int *state2;
  bool state3;
};

#define THREAD_INITIALIZER  {0, 0, false}

test.cpp

#include "thread.h"

class Foo {
private:
  thread_t __t_;
public:
  Foo() {__t_ = (thread_t) THREAD_INITIALIZER;}
};

Foo has a trivial destructor. Our mutex does not. I've already looked into the codegen for clang and in each case your change creates runtime code. The constructors may still be optimized away but it still has to register the destructors.

I'm just trying to figure out if that's going to be a problem.

Example here: https://godbolt.org/g/dJL29v

Hi Eric,

Thanks for the pointer (and also for godbolt!). I can see that that my code creates runtime code to register the destructors.

Apart from the overhead (registering the destructors at load time and then actually calling them when the DSO is unloaded), I can't see how this could lead to other issues. The ABI seem to be have a pretty well defined DSO destruction process [1].

Having to handle proper destruction of internal mutexes and condition variables sound like an OK thing for a standard library to do. But then, the following commit (which replaced mutex with the pthread native mutexes in memory.cpp originally) makes me think twice:

commit e33c2d1926f49221c9d72a353d797d135a810d77
Author: Howard Hinnant <[e-mail]>
Date:   Sat Mar 16 00:17:53 2013 +0000

    This should be nothing but a load-time optimization.  I'm trying to reduce load time initializers and this is a big one.  No visible functionality change intended.
    
    git-svn-id: https://llvm.org/svn/llvm-project/libcxx/trunk@177212 91177308-0d34-0410-b5e6-96231b3b80d8

diff --git a/src/memory.cpp b/src/memory.cpp
index 14084a5..98bcc86 100644
--- a/src/memory.cpp
+++ b/src/memory.cpp
@@ -122,7 +122,15 @@ __shared_weak_count::__get_deleter(const type_info&) const _NOEXCEPT
 #if __has_feature(cxx_atomic)
 
 static const std::size_t __sp_mut_count = 16;
-static mutex mut_back[__sp_mut_count];
+static pthread_mutex_t mut_back_imp[__sp_mut_count] =
+{
+    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
+    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
+    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
+    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER
+};
+
+static mutex* mut_back = reinterpret_cast<std::mutex*>(mut_back_imp);
 
 _LIBCPP_CONSTEXPR __sp_mut::__sp_mut(void* p) _NOEXCEPT
    : __lx(p)

So, perhaps it is best to leave these pthread mutexes alone, purely for performance reasons. We'll have to excuse a couple of #ifdef _LIBCPP_THREAD_API_XXXX conditionals in the library sources to allow the external threading API to function.

Does that sound like an OK compromise? @EricWF, @mclow.lists?

Thanks.

/ Asiri

[1] https://mentorembedded.github.io/cxx-abi/abi.html#dso-dtor

[EricWF]

In D19412#417729, @rmaprath wrote:

So, perhaps it is best to leave these pthread mutexes alone, purely for performance reasons. We'll have to excuse a couple of #ifdef _LIBCPP_THREAD_API_XXXX conditionals in the library sources to allow the external threading API to function.

Does that sound like an OK compromise? @EricWF, @mclow.lists?

Sounds good to me. Although we should probably change these from pthread_mutex_t to __libcpp_mutex_t.

[rmaprath]

As agreed, reverted back to using the native mutex / condition_variable types within library sources.

@EricWF: Good to go now?

Thanks.

/ Asiri

[rmaprath]

Added missing initializer (typo).

[rmaprath]

@mclow.lists, @EricWF: Gentle ping.

[EricWF]

LGTM after the minor fixes. Thank you for the patch.

include/__threading_support

39

ec -> __ec.

126

Let's make the return type bool so it's clear.

128

return pthread_equal(t1, t2) != 0;

133

Make the return type bool.

135

return t1 < t2;

include/thread

200

Nit: Use nullptr.

[rmaprath]

@EricWF: Thanks for taking the time to review! :)

Committed as r268734. The bots seem to be happy.

include/__config

 #    define _LIBCPP_NO_RTTI
 #  endif
 #endif
 
 #ifndef _LIBCPP_WEAK
 #  define _LIBCPP_WEAK __attribute__((__weak__))
 #endif
 
+// Thread API
+#ifndef _LIBCPP_HAS_NO_THREADS
+# if defined(__FreeBSD__) || \
+    defined(__NetBSD__) || \
+    defined(__linux__) || \
+    defined(__APPLE__) || \
+    defined(__CloudABI__)
+#  define _LIBCPP_THREAD_API_PTHREAD
+# else
+#  error "No thread API"
+# endif // _LIBCPP_THREAD_API
+#endif // _LIBCPP_HAS_NO_THREADS
+
+
 #if defined(_LIBCPP_HAS_NO_MONOTONIC_CLOCK) && !defined(_LIBCPP_HAS_NO_THREADS)
 #  error _LIBCPP_HAS_NO_MONOTONIC_CLOCK may only be defined when \
          _LIBCPP_HAS_NO_THREADS is defined.
 #endif
 
 // Systems that use capability-based security (FreeBSD with Capsicum,
 // Nuxi CloudABI) may only provide local filesystem access (using *at()).
 // Functions like open(), rename(), unlink() and stat() should not be

include/__mutex_base

 // -*- C++ -*-
 //===----------------------------------------------------------------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is dual licensed under the MIT and the University of Illinois Open
 // Source Licenses. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef _LIBCPP___MUTEX_BASE
 #define _LIBCPP___MUTEX_BASE
 
 #include <__config>
 #include <chrono>
 #include <system_error>
-#ifndef _LIBCPP_HAS_NO_THREADS
-#include <pthread.h>
-#endif
+#include <__threading_support>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
 #ifndef _LIBCPP_HAS_NO_THREADS
 
 #ifndef _LIBCPP_THREAD_SAFETY_ANNOTATION
 #  ifdef _LIBCPP_HAS_THREAD_SAFETY_ANNOTATIONS
 #    define _LIBCPP_THREAD_SAFETY_ANNOTATION(x) __attribute__((x))
 #  else
 #    define _LIBCPP_THREAD_SAFETY_ANNOTATION(x)
 #  endif
 #endif  // _LIBCPP_THREAD_SAFETY_ANNOTATION
 
 class _LIBCPP_TYPE_VIS _LIBCPP_THREAD_SAFETY_ANNOTATION(capability("mutex")) mutex
 {
-    pthread_mutex_t __m_;
+    typedef __libcpp_mutex_t __mutex_type;
+    __mutex_type __m_;
 
 public:
     _LIBCPP_INLINE_VISIBILITY
 #ifndef _LIBCPP_HAS_NO_CONSTEXPR
-     constexpr mutex() _NOEXCEPT : __m_(PTHREAD_MUTEX_INITIALIZER) {}
+    constexpr mutex() _NOEXCEPT : __m_(_LIBCPP_MUTEX_INITIALIZER) {}
 #else
-     mutex() _NOEXCEPT {__m_ = (pthread_mutex_t)PTHREAD_MUTEX_INITIALIZER;}
+    mutex() _NOEXCEPT {__m_ = (__mutex_type)_LIBCPP_MUTEX_INITIALIZER;}
 #endif
      ~mutex();
 
 private:
     mutex(const mutex&);// = delete;
     mutex& operator=(const mutex&);// = delete;
 
 public:
     void lock() _LIBCPP_THREAD_SAFETY_ANNOTATION(acquire_capability());
     bool try_lock() _NOEXCEPT _LIBCPP_THREAD_SAFETY_ANNOTATION(try_acquire_capability(true));
     void unlock() _NOEXCEPT _LIBCPP_THREAD_SAFETY_ANNOTATION(release_capability());
 
-    typedef pthread_mutex_t* native_handle_type;
+    typedef __mutex_type* native_handle_type;
     _LIBCPP_INLINE_VISIBILITY native_handle_type native_handle() {return &__m_;}
 };
 
 struct _LIBCPP_TYPE_VIS defer_lock_t {};
 struct _LIBCPP_TYPE_VIS try_to_lock_t {};
 struct _LIBCPP_TYPE_VIS adopt_lock_t {};
 
 #if defined(_LIBCPP_HAS_NO_CONSTEXPR) || defined(_LIBCPP_BUILDING_MUTEX)

 {
     no_timeout,
     timeout
 };
 _LIBCPP_DECLARE_STRONG_ENUM_EPILOG(cv_status)
 
 class _LIBCPP_TYPE_VIS condition_variable
 {
-    pthread_cond_t __cv_;
+    typedef __libcpp_condvar_t __cond_var_type;
+    __cond_var_type  __cv_;
 public:
     _LIBCPP_INLINE_VISIBILITY
 #ifndef _LIBCPP_HAS_NO_CONSTEXPR
-    constexpr condition_variable() : __cv_(PTHREAD_COND_INITIALIZER) {}
+    constexpr condition_variable() : __cv_(_LIBCPP_CONDVAR_INITIALIZER) {}
 #else
-    condition_variable() {__cv_ = (pthread_cond_t)PTHREAD_COND_INITIALIZER;}
+    condition_variable() {__cv_ = (__cond_var_type)_LIBCPP_CONDVAR_INITIALIZER;}
 #endif
     ~condition_variable();
 
 private:
     condition_variable(const condition_variable&); // = delete;
     condition_variable& operator=(const condition_variable&); // = delete;
 
 public:

 
     template <class _Rep, class _Period, class _Predicate>
         bool
         _LIBCPP_INLINE_VISIBILITY
         wait_for(unique_lock<mutex>& __lk,
                  const chrono::duration<_Rep, _Period>& __d,
                  _Predicate __pred);
 
-    typedef pthread_cond_t* native_handle_type;
+    typedef __cond_var_type* native_handle_type;
     _LIBCPP_INLINE_VISIBILITY native_handle_type native_handle() {return &__cv_;}
 
 private:
     void __do_timed_wait(unique_lock<mutex>& __lk,
        chrono::time_point<chrono::system_clock, chrono::nanoseconds>) _NOEXCEPT;
 };
 #endif // !_LIBCPP_HAS_NO_THREADS
 

include/__threading_support

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef _LIBCPP_THREADING_SUPPORT
+#define _LIBCPP_THREADING_SUPPORT
+
+#include <__config>
+
+#ifndef _LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER
+#pragma GCC system_header
+#endif
+
+#ifndef _LIBCPP_HAS_NO_THREADS
+
+#if defined(_LIBCPP_THREAD_API_PTHREAD)
+#include <pthread.h>
+#include <sched.h>
+#endif
+
+_LIBCPP_BEGIN_NAMESPACE_STD
+
+#if defined(_LIBCPP_THREAD_API_PTHREAD)
+
+// Mutex
+#define _LIBCPP_MUTEX_INITIALIZER PTHREAD_MUTEX_INITIALIZER
+typedef pthread_mutex_t __libcpp_mutex_t;
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_init(__libcpp_mutex_t* __m)
+{
+    pthread_mutexattr_t attr;
+    int ec = pthread_mutexattr_init(&attr);
+    if (ec) return ec;
+    ec = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
+    if (ec)
+    {
+        pthread_mutexattr_destroy(&attr);
+        return ec;
+    }
+    ec = pthread_mutex_init(__m, &attr);
+    if (ec)
+    {
+        pthread_mutexattr_destroy(&attr);
+        return ec;
+    }
+    ec = pthread_mutexattr_destroy(&attr);
+    if (ec)
+    {
+        pthread_mutex_destroy(__m);
+        return ec;
+    }
+    return 0;
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_lock(__libcpp_mutex_t* __m)
+{
+    return pthread_mutex_lock(__m);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_trylock(__libcpp_mutex_t* __m)
+{
+    return pthread_mutex_trylock(__m);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_unlock(__libcpp_mutex_t* __m)
+{
+    return pthread_mutex_unlock(__m);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_destroy(__libcpp_mutex_t* __m)
+{
+    return pthread_mutex_destroy(__m);
+}
+
+// Condition variable
+#define _LIBCPP_CONDVAR_INITIALIZER PTHREAD_COND_INITIALIZER
+typedef pthread_cond_t __libcpp_condvar_t;
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_signal(__libcpp_condvar_t* __cv)
+{
+    return pthread_cond_signal(__cv);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_broadcast(__libcpp_condvar_t* __cv)
+{
+    return pthread_cond_broadcast(__cv);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_wait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m)
+{
+    return pthread_cond_wait(__cv, __m);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_timedwait(__libcpp_condvar_t* __cv, __libcpp_mutex_t* __m, timespec* __ts)
+{
+    return pthread_cond_timedwait(__cv, __m, __ts);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_destroy(__libcpp_condvar_t* __cv)
+{
+    return pthread_cond_destroy(__cv);
+}
+
+// Thread id
+typedef pthread_t __libcpp_thread_id;
+
+// Returns non-zero if the thread ids are equal, otherwise 0
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_id_equal(__libcpp_thread_id t1, __libcpp_thread_id t2)
+{
+    return pthread_equal(t1, t2) != 0 ? 1 : 0;
+}
+
+// Returns non-zero if t1 < t2, otherwise 0
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_id_less(__libcpp_thread_id t1, __libcpp_thread_id t2)
+{
+    return t1 < t2 ? 1 : 0;
+}
+
+// Thread
+typedef pthread_t __libcpp_thread_t;
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_create(__libcpp_thread_t* __t, void* (*__func)(void*), void* __arg)
+{
+    return pthread_create(__t, 0, __func, __arg);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+__libcpp_thread_id __libcpp_thread_get_current_id()
+{
+    return pthread_self();
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+__libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t* __t)
+{
+    return *__t;
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_join(__libcpp_thread_t* __t)
+{
+    return pthread_join(*__t, 0);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_detach(__libcpp_thread_t* __t)
+{
+    return pthread_detach(*__t);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+void __libcpp_thread_yield()
+{
+    sched_yield();
+}
+
+// Thread local storage
+typedef pthread_key_t __libcpp_tl_key;
+
+inline _LIBCPP_ALWAYS_INLINE
+int __libcpp_tl_create(__libcpp_tl_key* __key, void (*__at_exit)(void*))
+{
+    return pthread_key_create(__key, __at_exit);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+void* __libcpp_tl_get(__libcpp_tl_key __key)
+{
+    return pthread_getspecific(__key);
+}
+
+inline _LIBCPP_ALWAYS_INLINE
+void __libcpp_tl_set(__libcpp_tl_key __key, void* __p)
+{
+    pthread_setspecific(__key, __p);
+}
+
+#else // !_LIBCPP_THREAD_API_PTHREAD
+  #error "No thread API selected."
+#endif
+
+_LIBCPP_END_NAMESPACE_STD
+
+#endif // _LIBCPP_HAS_NO_THREADS
+
+#endif // _LIBCPP_THREADING_SUPPORT

include/mutex

 
 #include <__config>
 #include <__mutex_base>
 #include <functional>
 #include <memory>
 #ifndef _LIBCPP_HAS_NO_VARIADICS
 #include <tuple>
 #endif
-#ifndef _LIBCPP_HAS_NO_THREADS
-#include <sched.h>
-#endif
+#include <__threading_support>
 
 #include <__undef_min_max>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
 #ifndef _LIBCPP_HAS_NO_THREADS
 
 class _LIBCPP_TYPE_VIS recursive_mutex
 {
-    pthread_mutex_t __m_;
+    typedef __libcpp_mutex_t __mutex_type;
+    __mutex_type __m_;
 
 public:
      recursive_mutex();
      ~recursive_mutex();
 
 private:
     recursive_mutex(const recursive_mutex&); // = delete;
     recursive_mutex& operator=(const recursive_mutex&); // = delete;
 
 public:
     void lock();
     bool try_lock() _NOEXCEPT;
     void unlock()  _NOEXCEPT;
 
-    typedef pthread_mutex_t* native_handle_type;
+    typedef __mutex_type* native_handle_type;
     _LIBCPP_INLINE_VISIBILITY
     native_handle_type native_handle() {return &__m_;}
 };
 
 class _LIBCPP_TYPE_VIS timed_mutex
 {
     mutex              __m_;
     condition_variable __cv_;

         __locked_ = true;
         return true;
     }
     return false;
 }
 
 class _LIBCPP_TYPE_VIS recursive_timed_mutex
 {
-    mutex              __m_;
-    condition_variable __cv_;
-    size_t             __count_;
-    pthread_t          __id_;
+    mutex                            __m_;
+    condition_variable               __cv_;
+    size_t                           __count_;
+    __libcpp_thread_id __id_;
 public:
      recursive_timed_mutex();
      ~recursive_timed_mutex();
 
 private:
     recursive_timed_mutex(const recursive_timed_mutex&); // = delete;
     recursive_timed_mutex& operator=(const recursive_timed_mutex&); // = delete;
 

     void unlock() _NOEXCEPT;
 };
 
 template <class _Clock, class _Duration>
 bool
 recursive_timed_mutex::try_lock_until(const chrono::time_point<_Clock, _Duration>& __t)
 {
     using namespace chrono;
-    pthread_t __id = pthread_self();
+    __libcpp_thread_id __id = __libcpp_thread_get_current_id();
     unique_lock<mutex> lk(__m_);
-    if (pthread_equal(__id, __id_))
+    if (__libcpp_thread_id_equal(__id, __id_))
     {
         if (__count_ == numeric_limits<size_t>::max())
             return false;
         ++__count_;
         return true;
     }
     bool no_timeout = _Clock::now() < __t;
     while (no_timeout && __count_ != 0)

         {
             unique_lock<_L0> __u0(__l0);
             if (__l1.try_lock())
             {
                 __u0.release();
                 break;
             }
         }
-        sched_yield();
+        __libcpp_thread_yield();
         {
             unique_lock<_L1> __u1(__l1);
             if (__l0.try_lock())
             {
                 __u1.release();
                 break;
             }
         }
-        sched_yield();
+        __libcpp_thread_yield();
     }
 }
 
 #ifndef _LIBCPP_HAS_NO_VARIADICS
 
 template <class _L0, class _L1, class _L2, class ..._L3>
 void
 __lock_first(int __i, _L0& __l0, _L1& __l1, _L2& __l2, _L3& ...__l3)
 {
     while (true)
     {
         switch (__i)
         {
         case 0:
             {
                 unique_lock<_L0> __u0(__l0);
                 __i = try_lock(__l1, __l2, __l3...);
                 if (__i == -1)
                 {
                     __u0.release();
                     return;
                 }
             }
             ++__i;
-            sched_yield();
+            __libcpp_thread_yield();
             break;
         case 1:
             {
                 unique_lock<_L1> __u1(__l1);
                 __i = try_lock(__l2, __l3..., __l0);
                 if (__i == -1)
                 {
                     __u1.release();
                     return;
                 }
             }
             if (__i == sizeof...(_L3) + 1)
                 __i = 0;
             else
                 __i += 2;
-            sched_yield();
+            __libcpp_thread_yield();
             break;
         default:
             __lock_first(__i - 2, __l2, __l3..., __l0, __l1);
             return;
         }
     }
 }
 

include/thread

 #include <functional>
 #include <memory>
 #include <system_error>
 #include <chrono>
 #include <__mutex_base>
 #ifndef _LIBCPP_HAS_NO_VARIADICS
 #include <tuple>
 #endif
-#include <pthread.h>
-#include <sched.h>
+#include <__threading_support>
 
 #if !defined(_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER)
 #pragma GCC system_header
 #endif
 
 #define __STDCPP_THREADS__ __cplusplus
 
 #ifdef _LIBCPP_HAS_NO_THREADS

 
     void notify_all_at_thread_exit(condition_variable*, mutex*);
     void __make_ready_at_thread_exit(__assoc_sub_state*);
 };
 
 template <class _Tp>
 class __thread_specific_ptr
 {
-    pthread_key_t __key_;
+    __libcpp_tl_key __key_;
 
      // Only __thread_local_data() may construct a __thread_specific_ptr
      // and only with _Tp == __thread_struct.
     static_assert((is_same<_Tp, __thread_struct>::value), "");
     __thread_specific_ptr();
     friend _LIBCPP_FUNC_VIS __thread_specific_ptr<__thread_struct>& __thread_local_data();
 
     __thread_specific_ptr(const __thread_specific_ptr&);
     __thread_specific_ptr& operator=(const __thread_specific_ptr&);
 
     static void __at_thread_exit(void*);
 public:
     typedef _Tp* pointer;
 
     ~__thread_specific_ptr();
 
     _LIBCPP_INLINE_VISIBILITY
-    pointer get() const {return static_cast<_Tp*>(pthread_getspecific(__key_));}
+    pointer get() const {return static_cast<_Tp*>(__libcpp_tl_get(__key_));}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator*() const {return *get();}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator->() const {return get();}
     pointer release();
     void reset(pointer __p = nullptr);
 };
 
 template <class _Tp>
 void
 __thread_specific_ptr<_Tp>::__at_thread_exit(void* __p)
 {
     delete static_cast<pointer>(__p);
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::__thread_specific_ptr()
 {
-    int __ec = pthread_key_create(&__key_, &__thread_specific_ptr::__at_thread_exit);
+    int __ec = __libcpp_tl_create(
+        &__key_,
+        &__thread_specific_ptr::__at_thread_exit);
 #ifndef _LIBCPP_NO_EXCEPTIONS
     if (__ec)
         throw system_error(error_code(__ec, system_category()),
                            "__thread_specific_ptr construction failed");
 #endif
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::~__thread_specific_ptr()
 {
     // __thread_specific_ptr is only created with a static storage duration
     // so this destructor is only invoked during program termination. Invoking
     // pthread_key_delete(__key_) may prevent other threads from deleting their
     // thread local data. For this reason we leak the key.
 }
 
 template <class _Tp>
 typename __thread_specific_ptr<_Tp>::pointer
 __thread_specific_ptr<_Tp>::release()
 {
     pointer __p = get();
-    pthread_setspecific(__key_, 0);
+    __libcpp_tl_set(__key_, 0);
     return __p;
 }
 
 template <class _Tp>
 void
 __thread_specific_ptr<_Tp>::reset(pointer __p)
 {
     pointer __p_old = get();
-    pthread_setspecific(__key_, __p);
+    __libcpp_tl_set(__key_, __p);
     delete __p_old;
 }
 
 class _LIBCPP_TYPE_VIS thread;
 class _LIBCPP_TYPE_VIS __thread_id;
 
 namespace this_thread
 {
 
 _LIBCPP_INLINE_VISIBILITY __thread_id get_id() _NOEXCEPT;
 
 }  // this_thread
 
 template<> struct hash<__thread_id>;
 
 class _LIBCPP_TYPE_VIS_ONLY __thread_id
 {
     // FIXME: pthread_t is a pointer on Darwin but a long on Linux.
     // NULL is the no-thread value on Darwin.  Someone needs to check
     // on other platforms.  We assume 0 works everywhere for now.
-    pthread_t __id_;
+    __libcpp_thread_id __id_;
 
 public:
     _LIBCPP_INLINE_VISIBILITY
     __thread_id() _NOEXCEPT : __id_(0) {}
 
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator==(__thread_id __x, __thread_id __y) _NOEXCEPT
-        {return __x.__id_ == __y.__id_;}
+        {return __libcpp_thread_id_equal(__x.__id_, __y.__id_);}
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator!=(__thread_id __x, __thread_id __y) _NOEXCEPT
         {return !(__x == __y);}
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator< (__thread_id __x, __thread_id __y) _NOEXCEPT
-        {return __x.__id_ < __y.__id_;}
+        {return  __libcpp_thread_id_less(__x.__id_, __y.__id_);}
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator<=(__thread_id __x, __thread_id __y) _NOEXCEPT
         {return !(__y < __x);}
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator> (__thread_id __x, __thread_id __y) _NOEXCEPT
         {return   __y < __x ;}
     friend _LIBCPP_INLINE_VISIBILITY
         bool operator>=(__thread_id __x, __thread_id __y) _NOEXCEPT
         {return !(__x < __y);}
 
     template<class _CharT, class _Traits>
     friend
     _LIBCPP_INLINE_VISIBILITY
     basic_ostream<_CharT, _Traits>&
     operator<<(basic_ostream<_CharT, _Traits>& __os, __thread_id __id)
         {return __os << __id.__id_;}
 
 private:
     _LIBCPP_INLINE_VISIBILITY
-    __thread_id(pthread_t __id) : __id_(__id) {}
+    __thread_id(__libcpp_thread_id __id) : __id_(__id) {}
 
     friend __thread_id this_thread::get_id() _NOEXCEPT;
     friend class _LIBCPP_TYPE_VIS thread;
     friend struct _LIBCPP_TYPE_VIS_ONLY hash<__thread_id>;
 };
 
 template<>
 struct _LIBCPP_TYPE_VIS_ONLY hash<__thread_id>
     : public unary_function<__thread_id, size_t>
 {
     _LIBCPP_INLINE_VISIBILITY
     size_t operator()(__thread_id __v) const
     {
-        return hash<pthread_t>()(__v.__id_);
+        return hash<__libcpp_thread_id>()(__v.__id_);
     }
 };
 
 namespace this_thread
 {
 
 inline _LIBCPP_INLINE_VISIBILITY
 __thread_id
 get_id() _NOEXCEPT
 {
-    return pthread_self();
+    return __libcpp_thread_get_current_id();
 }
 
 }  // this_thread
 
 class _LIBCPP_TYPE_VIS thread
 {
-    pthread_t __t_;
+    typedef __libcpp_thread_t __thread_type;
+    __thread_type __t_;
 
     thread(const thread&);
     thread& operator=(const thread&);
 public:
     typedef __thread_id id;
-    typedef pthread_t native_handle_type;
+    typedef __thread_type native_handle_type;
 
     _LIBCPP_INLINE_VISIBILITY
     thread() _NOEXCEPT : __t_(0) {}
 #ifndef _LIBCPP_HAS_NO_VARIADICS
     template <class _Fp, class ..._Args,
               class = typename enable_if
               <
                    !is_same<typename decay<_Fp>::type, thread>::value

     _LIBCPP_INLINE_VISIBILITY
     void swap(thread& __t) _NOEXCEPT {_VSTD::swap(__t_, __t.__t_);}
 
     _LIBCPP_INLINE_VISIBILITY
     bool joinable() const _NOEXCEPT {return __t_ != 0;}
     void join();
     void detach();
     _LIBCPP_INLINE_VISIBILITY
-    id get_id() const _NOEXCEPT {return __t_;}
+    id get_id() const _NOEXCEPT {return __libcpp_thread_get_id(&__t_);}
     _LIBCPP_INLINE_VISIBILITY
     native_handle_type native_handle() _NOEXCEPT {return __t_;}
 
     static unsigned hardware_concurrency() _NOEXCEPT;
 };
 
 #ifndef _LIBCPP_HAS_NO_VARIADICS
 

 {
     typedef unique_ptr<__thread_struct> _TSPtr;
     _TSPtr __tsp(new __thread_struct);
     typedef tuple<_TSPtr, typename decay<_Fp>::type, typename decay<_Args>::type...> _Gp;
     _VSTD::unique_ptr<_Gp> __p(
             new _Gp(std::move(__tsp),
                     __decay_copy(_VSTD::forward<_Fp>(__f)),
                     __decay_copy(_VSTD::forward<_Args>(__args))...));
-    int __ec = pthread_create(&__t_, 0, &__thread_proxy<_Gp>, __p.get());
+    int __ec = __libcpp_thread_create(&__t_, &__thread_proxy<_Gp>, __p.get());
     if (__ec == 0)
         __p.release();
     else
         __throw_system_error(__ec, "thread constructor failed");
 }
 
 #else  // _LIBCPP_HAS_NO_VARIADICS
 

 
 template <class _Fp>
 thread::thread(_Fp __f)
 {
 
     typedef __thread_invoke_pair<_Fp> _InvokePair;
     typedef std::unique_ptr<_InvokePair> _PairPtr;
     _PairPtr __pp(new _InvokePair(__f));
-    int __ec = pthread_create(&__t_, 0, &__thread_proxy_cxx03<_InvokePair>, __pp.get());
+    int __ec = __libcpp_thread_create(&__t_, &__thread_proxy_cxx03<_InvokePair>, __pp.get());
     if (__ec == 0)
         __pp.release();
     else
         __throw_system_error(__ec, "thread constructor failed");
 }
 
 #endif  // _LIBCPP_HAS_NO_VARIADICS
 

 void
 sleep_until(const chrono::time_point<chrono::steady_clock, _Duration>& __t)
 {
     using namespace chrono;
     sleep_for(__t - steady_clock::now());
 }
 
 inline _LIBCPP_INLINE_VISIBILITY
-void yield() _NOEXCEPT {sched_yield();}
+void yield() _NOEXCEPT {__libcpp_thread_yield();}
 
 }  // this_thread
 
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // !_LIBCPP_HAS_NO_THREADS
 
 #endif  // _LIBCPP_THREAD

src/algorithm.cpp

 template bool __insertion_sort_incomplete<__less<unsigned long long>&, unsigned long long*>(unsigned long long*, unsigned long long*, __less<unsigned long long>&);
 template bool __insertion_sort_incomplete<__less<float>&, float*>(float*, float*, __less<float>&);
 template bool __insertion_sort_incomplete<__less<double>&, double*>(double*, double*, __less<double>&);
 template bool __insertion_sort_incomplete<__less<long double>&, long double*>(long double*, long double*, __less<long double>&);
 
 template unsigned __sort5<__less<long double>&, long double*>(long double*, long double*, long double*, long double*, long double*, __less<long double>&);
 
 #ifndef _LIBCPP_HAS_NO_THREADS
-static pthread_mutex_t __rs_mut = PTHREAD_MUTEX_INITIALIZER;
+static mutex __rs_mut;
 #endif
 unsigned __rs_default::__c_ = 0;
 
 __rs_default::__rs_default()
 {
 #ifndef _LIBCPP_HAS_NO_THREADS
-    pthread_mutex_lock(&__rs_mut);
+    __rs_mut.lock();
 #endif
     __c_ = 1;
 }
 
 __rs_default::__rs_default(const __rs_default&)
 {
     ++__c_;
 }
 
 __rs_default::~__rs_default()
 {
 #ifndef _LIBCPP_HAS_NO_THREADS
     if (--__c_ == 0)
-        pthread_mutex_unlock(&__rs_mut);
+        __rs_mut.unlock();
 #else
     --__c_;
 #endif
 }
 
 __rs_default::result_type
 __rs_default::operator()()
 {

src/condition_variable.cpp

 #include "thread"
 #include "system_error"
 #include "cassert"
 
 _LIBCPP_BEGIN_NAMESPACE_STD
 
 condition_variable::~condition_variable()
 {
-    pthread_cond_destroy(&__cv_);
+    __libcpp_condvar_destroy(&__cv_);
 }
 
 void
 condition_variable::notify_one() _NOEXCEPT
 {
-    pthread_cond_signal(&__cv_);
+    __libcpp_condvar_signal(&__cv_);
 }
 
 void
 condition_variable::notify_all() _NOEXCEPT
 {
-    pthread_cond_broadcast(&__cv_);
+    __libcpp_condvar_broadcast(&__cv_);
 }
 
 void
 condition_variable::wait(unique_lock<mutex>& lk) _NOEXCEPT
 {
     if (!lk.owns_lock())
         __throw_system_error(EPERM,
                                   "condition_variable::wait: mutex not locked");
-    int ec = pthread_cond_wait(&__cv_, lk.mutex()->native_handle());
+    int ec = __libcpp_condvar_wait(&__cv_, lk.mutex()->native_handle());
     if (ec)
         __throw_system_error(ec, "condition_variable wait failed");
 }
 
 void
 condition_variable::__do_timed_wait(unique_lock<mutex>& lk,
      chrono::time_point<chrono::system_clock, chrono::nanoseconds> tp) _NOEXCEPT
 {

         ts.tv_sec = static_cast<ts_sec>(s.count());
         ts.tv_nsec = static_cast<decltype(ts.tv_nsec)>((d - s).count());
     }
     else
     {
         ts.tv_sec = ts_sec_max;
         ts.tv_nsec = giga::num - 1;
     }
-    int ec = pthread_cond_timedwait(&__cv_, lk.mutex()->native_handle(), &ts);
+    int ec = __libcpp_condvar_timedwait(&__cv_, lk.mutex()->native_handle(), &ts);
     if (ec != 0 && ec != ETIMEDOUT)
         __throw_system_error(ec, "condition_variable timed_wait failed");
 }
 
 void
 notify_all_at_thread_exit(condition_variable& cond, unique_lock<mutex> lk)
 {
     __thread_local_data()->notify_all_at_thread_exit(&cond, lk.release());
 }
 
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // !_LIBCPP_HAS_NO_THREADS

src/memory.cpp

     return 0;
 }
 
 #endif  // _LIBCPP_NO_RTTI
 
 #if defined(_LIBCPP_HAS_C_ATOMIC_IMP) && !defined(_LIBCPP_HAS_NO_THREADS)
 
 static const std::size_t __sp_mut_count = 16;
-static pthread_mutex_t mut_back_imp[__sp_mut_count] =
-{
-    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
-    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
-    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER,
-    PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER, PTHREAD_MUTEX_INITIALIZER
-};
-
-static mutex* mut_back = reinterpret_cast<std::mutex*>(mut_back_imp);
+static mutex mut_back[__sp_mut_count];
 
 _LIBCPP_CONSTEXPR __sp_mut::__sp_mut(void* p) _NOEXCEPT
    : __lx(p)
 {
 }
 
 void
 __sp_mut::lock() _NOEXCEPT

src/mutex.cpp

 #ifndef _LIBCPP_HAS_NO_THREADS
 
 const defer_lock_t  defer_lock = {};
 const try_to_lock_t try_to_lock = {};
 const adopt_lock_t  adopt_lock = {};
 
 mutex::~mutex()
 {
-    pthread_mutex_destroy(&__m_);
+    __libcpp_mutex_destroy(&__m_);
 }
 
 void
 mutex::lock()
 {
-    int ec = pthread_mutex_lock(&__m_);
+    int ec = __libcpp_mutex_lock(&__m_);
     if (ec)
         __throw_system_error(ec, "mutex lock failed");
 }
 
 bool
 mutex::try_lock() _NOEXCEPT
 {
-    return pthread_mutex_trylock(&__m_) == 0;
+    return __libcpp_mutex_trylock(&__m_) == 0;
 }
 
 void
 mutex::unlock() _NOEXCEPT
 {
-    int ec = pthread_mutex_unlock(&__m_);
+    int ec = __libcpp_mutex_unlock(&__m_);
     (void)ec;
     assert(ec == 0);
 }
 
 // recursive_mutex
 
 recursive_mutex::recursive_mutex()
 {
-    pthread_mutexattr_t attr;
-    int ec = pthread_mutexattr_init(&attr);
+    int ec = __libcpp_recursive_mutex_init(&__m_);
     if (ec)
-        goto fail;
-    ec = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE);
-    if (ec)
-    {
-        pthread_mutexattr_destroy(&attr);
-        goto fail;
-    }
-    ec = pthread_mutex_init(&__m_, &attr);
-    if (ec)
-    {
-        pthread_mutexattr_destroy(&attr);
-        goto fail;
-    }
-    ec = pthread_mutexattr_destroy(&attr);
-    if (ec)
-    {
-        pthread_mutex_destroy(&__m_);
-        goto fail;
-    }
-    return;
-fail:
-    __throw_system_error(ec, "recursive_mutex constructor failed");
+        __throw_system_error(ec, "recursive_mutex constructor failed");
 }
 
 recursive_mutex::~recursive_mutex()
 {
-    int e = pthread_mutex_destroy(&__m_);
+    int e = __libcpp_mutex_destroy(&__m_);
     (void)e;
     assert(e == 0);
 }
 
 void
 recursive_mutex::lock()
 {
-    int ec = pthread_mutex_lock(&__m_);
+    int ec = __libcpp_mutex_lock(&__m_);
     if (ec)
         __throw_system_error(ec, "recursive_mutex lock failed");
 }
 
 void
 recursive_mutex::unlock() _NOEXCEPT
 {
-    int e = pthread_mutex_unlock(&__m_);
+    int e = __libcpp_mutex_unlock(&__m_);
     (void)e;
     assert(e == 0);
 }
 
 bool
 recursive_mutex::try_lock() _NOEXCEPT
 {
-    return pthread_mutex_trylock(&__m_) == 0;
+    return __libcpp_mutex_trylock(&__m_) == 0;
 }
 
 // timed_mutex
 
 timed_mutex::timed_mutex()
     : __locked_(false)
 {
 }

 recursive_timed_mutex::~recursive_timed_mutex()
 {
     lock_guard<mutex> _(__m_);
 }
 
 void
 recursive_timed_mutex::lock()
 {
-    pthread_t id = pthread_self();
+    __libcpp_thread_id id = __libcpp_thread_get_current_id();
     unique_lock<mutex> lk(__m_);
-    if (pthread_equal(id, __id_))
+    if (__libcpp_thread_id_equal(id, __id_))
     {
         if (__count_ == numeric_limits<size_t>::max())
             __throw_system_error(EAGAIN, "recursive_timed_mutex lock limit reached");
         ++__count_;
         return;
     }
     while (__count_ != 0)
         __cv_.wait(lk);
     __count_ = 1;
     __id_ = id;
 }
 
 bool
 recursive_timed_mutex::try_lock() _NOEXCEPT
 {
-    pthread_t id = pthread_self();
+    __libcpp_thread_id id = __libcpp_thread_get_current_id();
     unique_lock<mutex> lk(__m_, try_to_lock);
-    if (lk.owns_lock() && (__count_ == 0 || pthread_equal(id, __id_)))
+    if (lk.owns_lock() && (__count_ == 0 || __libcpp_thread_id_equal(id, __id_)))
     {
         if (__count_ == numeric_limits<size_t>::max())
             return false;
         ++__count_;
         __id_ = id;
         return true;
     }
     return false;

 
 // If dispatch_once_f ever handles C++ exceptions, and if one can get to it
 // without illegal macros (unexpected macros not beginning with _UpperCase or
 // __lowercase), and if it stops spinning waiting threads, then call_once should
 // call into dispatch_once_f instead of here. Relevant radar this code needs to
 // keep in sync with:  7741191.
 
 #ifndef _LIBCPP_HAS_NO_THREADS
-static pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
-static pthread_cond_t  cv  = PTHREAD_COND_INITIALIZER;
+static mutex mut;
+static condition_variable cv;
 #endif
 
 /// NOTE: Changes to flag are done via relaxed atomic stores
 ///       even though the accesses are protected by a mutex because threads
 ///       just entering 'call_once` concurrently read from flag.
 void
 __call_once(volatile unsigned long& flag, void* arg, void(*func)(void*))
 {
 #if defined(_LIBCPP_HAS_NO_THREADS)
     if (flag == 0)

         catch (...)
         {
             flag = 0ul;
             throw;
         }
 #endif  // _LIBCPP_NO_EXCEPTIONS
     }
 #else // !_LIBCPP_HAS_NO_THREADS
-    pthread_mutex_lock(&mut);
+    unique_lock<mutex> lk(mut);
     while (flag == 1)
-        pthread_cond_wait(&cv, &mut);
+        cv.wait(lk);
     if (flag == 0)
     {
 #ifndef _LIBCPP_NO_EXCEPTIONS
         try
         {
 #endif  // _LIBCPP_NO_EXCEPTIONS
             __libcpp_relaxed_store(&flag, 1ul);
-            pthread_mutex_unlock(&mut);
+            lk.unlock();
             func(arg);
-            pthread_mutex_lock(&mut);
+            lk.lock();
             __libcpp_relaxed_store(&flag, ~0ul);
-            pthread_mutex_unlock(&mut);
-            pthread_cond_broadcast(&cv);
+            lk.unlock();
+            cv.notify_all();
 #ifndef _LIBCPP_NO_EXCEPTIONS
         }
         catch (...)
         {
-            pthread_mutex_lock(&mut);
+            lk.lock();
             __libcpp_relaxed_store(&flag, 0ul);
-            pthread_mutex_unlock(&mut);
-            pthread_cond_broadcast(&cv);
+            lk.unlock();
+            cv.notify_all();
             throw;
         }
 #endif  // _LIBCPP_NO_EXCEPTIONS
     }
     else
-        pthread_mutex_unlock(&mut);
+        lk.unlock();
 #endif // !_LIBCPP_HAS_NO_THREADS
 
 }
 
 _LIBCPP_END_NAMESPACE_STD

src/thread.cpp

 {
     if (__t_ != 0)
         terminate();
 }
 
 void
 thread::join()
 {
-    int ec = pthread_join(__t_, 0);
+    int ec = __libcpp_thread_join(&__t_);
 #ifndef _LIBCPP_NO_EXCEPTIONS
     if (ec)
         throw system_error(error_code(ec, system_category()), "thread::join failed");
 #else
     (void)ec;
 #endif  // _LIBCPP_NO_EXCEPTIONS
     __t_ = 0;
 }
 
 void
 thread::detach()
 {
     int ec = EINVAL;
     if (__t_ != 0)
     {
-        ec = pthread_detach(__t_);
+        ec = __libcpp_thread_detach(&__t_);
         if (ec == 0)
             __t_ = 0;
     }
 #ifndef _LIBCPP_NO_EXCEPTIONS
     if (ec)
         throw system_error(error_code(ec, system_category()), "thread::detach failed");
 #endif  // _LIBCPP_NO_EXCEPTIONS
 }