provide Win32 native threading

[compnerd]

Summary

Add an implementation for the Win32 threading model as a backing API for
the internal c++ threading interfaces. This uses the Fls* family for
the TLS (which has the support for adding termination callbacks),
CRITICAL_SECTIONs for the recursive mutex, and Slim Reader/Writer locks
(SRW locks) for non-recursive mutexes. These APIs should all be
available on Vista or newer.

[majnemer]

slim reader-writer locks are faster than critical sections, I'd recommend your implementation switch to those.

Also, why do you use Fls instead of Tls?

[EricWF]

Added inline comments.

The main issue is that this patch needs to use the same <__threading_support> forward declarations as every other supported threading API; Only new definitions should be provided.

include/__threading_support

44

Isn't checking _LIBCPP_HAS_THREAD_API_WIN32 enough?

57

The forward declarations of the __libcpp_ threading wrapper should be shared between all API's. Please don't add your own forward declarations for Windows.

src/algorithm.cpp

51 ↗

(On Diff #82821)

This makes me sad because not having constant initialization manifests itself as a static initialization order fiasco in libc++. If we do go this route please file a libc++ bug stating that Win32 mutex's do not provide constant initialization.

I think this should check _LIBCPP_HAS_THREAD_API_WIN32?

src/memory.cpp

158 ↗

(On Diff #82821)

I think this should check _LIBCPP_HAS_THREAD_API_WIN32?

src/mutex.cpp

198 ↗

(On Diff #82821)

I think this should check _LIBCPP_HAS_THREAD_API_WIN32?

[compnerd]

@majnemer Im using the Fls* APIs since they provide the thread termination callback, which Tls* doesn't. Good point about the SRW. Those are newer, but, we can always provide a fallback later.

[majnemer]

In D28220#633622, @compnerd wrote:

@majnemer Im using the Fls* APIs since they provide the thread termination callback, which Tls* doesn't. Good point about the SRW. Those are newer, but, we can always provide a fallback later.

I don't believe they are newer than the condition variable API so all should be fine.

[compnerd]

The SRW initializers avoid the initializer changes, so its one less set of changes that is necessary.

include/__threading_support

44

No, because the structure of the header is such that it requires redeclarations :-(.

[compnerd]

Use SRW locks, rebase for avoid redeclaration of interfaces, remove static initialization check removals.

[majnemer]

include/__threading_support

300–305

I don't think you can use slim rw locks for recursive locks. I think we will need to use CRITICAL_SECTION for those. std::recursive_mutex can't be used with std::condition_variable AFAIK so all you need (I think) is recursive versions of __libcpp_mutex_...

Recursive locks should be used far less frequently which makes it valuable, IMO, to use slim rw locks for the non-recursive mutex implementation.

355

I don't think it should be __libcpp_condvar_timedwait's problem. __libcpp_condvar_timedwait wraps pthread_cond_timedwait on POSIX platforms and the caller of __libcpp_condvar_wait properly handles spurious wakeups. The caller of __libcpp_condvar_timedwait probably should be audited.

[compnerd]

include/__threading_support

300–305

You are absolutely right. That was something that I looked at originally and went with the CS. However, the overhead of a tagged struct is 5 or 9 bytes (sizeof(void *) + 1) bytes (ignoring padding for MS ABI). Going with that should give the benefits of always being able to properly initialize the CS instead of the kludge.

[compnerd]

include/__threading_support

355

SG; seems that there is a single user in condition_variable.cpp

[compnerd]

switch between a CRITICAL_SECTION and SRWLOCK

[EricWF]

Could you upload this patch with more context?

[majnemer]

include/__threading_support

300–305

Er, isn't the overhead much more than that? IIRC, CRITICAL_SECTION is quite large. You'd be making all the users of std::mutex pay for the space of std::recursive_mutex...

[smeenai]

include/__threading_support

83

Why not a tagged union?

[smeenai]

include/__threading_support

300–305

Yeah, CRITICAL_SECTION is 24 bytes vs 4 bytes for a SRWLOCK.

[EricWF]

include/__threading_support

44

Do these definitions have any affect when <Windows.h> has already been included?
Also are these definitions required before including the header, or merely beneficial? If they are required this will make the <Windows.h> header a pain to use with modules.

306

The initial __libcpp_foo declarations specify the correct linkage and visibility attributes. Please don't decorate the definitions.

[compnerd]

update for separation of mutex and recursive_mutex.

[compnerd]

add more context

[compnerd]

include/__threading_support

44

No, they dont effect it once it has been included. They are beneficial since they reduce the amount of stuff that gets included (including things which, at least when I last checked, can cause clang to choke).

57

Yeah, restructed that. There is now the dependent patch for the type-erased mutex handling.

[EricWF]

include/__threading_support

44

And can users re-include <Windows.h> afterwards in the same TU and get all of the symbols?

[kimgr]

Re Eric's windows.h concern.

include/__threading_support

44

I don't think so.

We've recently switched to defining these two symbols in our build system, and I think that's basically the only way to make this work in a project composed of headers from various authors. I think you're right that libc++ should not define them.

[awson]

You don't need #define VC_EXTRA_LEAN since:

1. VC_EXTRA_LEAN is a mistype, only VC_EXTRALEAN has some meaning in Windows ecosystem;
2. VC_EXTRALEAN is used *only* in MFC headers which obviously aren't used in clang codebase.

Thus it would be better to remove this altogether, and (perhaps) replace with the relevant #define WIN32_LEAN_AND_MEAN.

[halyavin]

include/__threading_support

462

We can have integer overflow on cast to DWORD here and get unexpectedly small wait for large timeouts.

499

For other reviewers: CC means Calling Convention.

[compnerd]

Remove incorrect use of VC_EXTRALEAN. Fix signature for __libcpp_set_thread_specific. Provide CC adjustment thunks for the thread creation. Add an assertion for sleep timeout truncation. Implement exec-once interfaces.

[halyavin]

include/__threading_support

454

In posix, pthread_cond_timedwait uses absolute time but we use relative time here.

456

We need to include <chrono> to use types and functions from this namespace.

460

You have done it accidentally right. I realized there is a second problem here - if timeout equals to INFINITE, the thread will wait forever. INFINITE equals to (DWORD)-1, so the strict sign in the assert is required. But for sanity sake _VSTD::numeric_limits<DWORD>::max() should be replaced with INFINITE.

482

We need underscores for parameters and init_routine.

493

Underscores for parameters are missing.

523

Trampolines will never be inlined. Should we put them in support *.cpp instead? The downside is new public symbols which can't be changed without breaking backward compatibility. The upside is that we will have only one copy of each trampoline. What do you think?

527

Do we need #include <cstdlib> for free() and malloc() now? Can we use new/delete instead?

BTW, What is the purpose of _VSTD? I think it is used to prevent argument-dependent lookup and so is not needed for free and malloc.

528

Should we even try to pass thread exit code, given that sizeof(unsigned int) < sizeof(void*) on 64-bit system? std::thread doesn't support thread exit code anyway.

[rnk]

include/__threading_support

44

I don't think libc++ should not include windows.h in a public header. I'd rather write our own __threading_support_win.h that re-prototypes everything we need from windows.h, and then we can have some test in libc++ that validates that there are no mismatches when including windows.h before libc++ <mutex>. If we do this, we should sink as much win32 API usage as possible out of headers to reduce our duplication.

It's worth pointing out VS 2015's thread implementation also hides its win32 API usage.

523

Considering that libc++ already has a __thread_proxy trampoline, let's just give it the right CC and get rid of this trampoline.

589

Yeah, we need to fix that. We should find a way to make __thread_specific_ptr::__at_thread_exit(void*) have the right convention out of the box, rather than thunking. Something like:

#define __LIBCPP_TLS_CALLBACK_CC WINAPI
.. // else
#define __LIBCPP_TLS_CALLBACK_CC
... // <thread>
    static void __LIBCPP_TLS_CALLBACK_CC __at_thread_exit(void*);

[compnerd]

include/__threading_support

523

I think I prefer @rnk's solution here. Lets try to use the __thread_proxy to hide the thunk.

528

Im not sure what the cleanest way to address this is.

[compnerd]

Address a number of review comments.

[compnerd]

include/__threading_support

454

Good catch. I guess we have to convert the timespec to a relative time before using it. Slightly annoying, but thankfully, we can fallback on chrono :-).

[compnerd]

Fix __libcpp_condvar_timedwait parameter usage (absolute vs relative time)

[EricWF]

This LGTM. I'm sure we can flush out any bugs once we get the tests running.

[halyavin]

include/__threading_support

477

Since negative timeouts can't be avoided, we must make sure that timeout_ms.count() is at least zero.

479

It is >= INFINITE. _LIBCPP_ASSERT has 2 arguments and supports error message out of the box.

[compnerd]

include/__threading_support

477

Good point. I suppose that the assert takes care of that though.

479

Shouldnt this be, timeout_ms.count() > 0 which implicitly ensures that it is not INFINITE (-1)?

[halyavin]

include/__threading_support

477

Negative timeout_ms.count() is a normal case, we shouldn't fail on assertion. The reason is that program can always be slow enough for current timestamp (system_clock::now()) to pass any fixed point in time (__ts/abstime).

479

I just checked in the Windows SDK, INFINITE is defined as 0xFFFFFFFF and negative timeouts are legal anyway.

[EricWF]

This needs to be rebased following the <__threading_support> cleanup in r291275.

[compnerd]

rebase, address negative timeouts, fix thunking for __libcpp_tls_create

[compnerd]

rebase

[halyavin]

LGTM.

include/__threading_support

480

There is still problem with large timeout but let us fix it later.

[EricWF]

I'm seeing the following build errors on Windows:

C:\Users\Eric\workspace\libcxx\include\__threading_support(521,3):  warning: cannot delete expression with pointer-to-'void' type 'void *' [-Wdelete-incomplete]
  delete __data;
  ^      ~~~~~~
C:\Users\Eric\workspace\libcxx\include\__threading_support(530,18):  error: assigning to 'void *(*)(void *)' from incompatible type 'void (*)(void *)': different return type ('void *' vs 'void')
  data->__func = __func;
                 ^~~~~~
C:\Users\Eric\workspace\libcxx\include\__threading_support(572,5):  error: functions that differ only in their return type cannot be overloaded
int __libcpp_thread_yield()
~~~ ^
C:\Users\Eric\workspace\libcxx\include\__threading_support(178,6):  note: previous declaration is here
void __libcpp_thread_yield();
~~~~ ^
C:\Users\Eric\workspace\libcxx\include\__threading_support(589,5):  error: functions that differ only in their return type cannot be overloaded
int __libcpp_tls_get(__libcpp_tls_key __key)
~~~ ^
C:\Users\Eric\workspace\libcxx\include\__threading_support(185,7):  note: previous declaration is here
void *__libcpp_tls_get(__libcpp_tls_key __key);
~~~~~~^
C:\Users\Eric\workspace\libcxx\include\__threading_support(591,10):  error: cannot initialize return object of type 'int' with an rvalue of type 'PVOID' (aka 'void *')
  return FlsGetValue(__key);

[EricWF]

Just tried the updated patch and I still get one build error:

C:\Users\Eric\workspace\libcxx\src\thread.cpp(135,16):  error: use of undeclared identifier 'nanosleep'
        while (nanosleep(&ts, &ts) == -1 && errno == EINTR)

[EricWF]

include/__threading_support

33

I think we agreed that we cannot use this macro.

447

DO NOT ANNOTATE THESE DEFINITIONS!! The forward declarations are already properly declared.

[majnemer]

include/__threading_support

33–34

Can we do as Reid suggests and not expose users to windows.h?

[EricWF]

include/__threading_support

601

The _LIBCPP_TLS_DESTRUCTOR_CC needs to appear an the initial declaration as well.

[EricWF]

include/__threading_support

34

Can we do as Reid suggests and not expose users to windows.h?

I was about to ask the same question. These includes are dragging in the __deallocate macro and I would love to avoid that.

[compnerd]

remove WIN32_LEAN_AND_MEAN, fix decoration, remove inline decorations

[smeenai]

include/__threading_support

34

I feel like we would end up with a lot of duplication if we went down this route, since this is using a fair amount of Windows APIs. @rnk suggested having a test for prototype mismatches, but even with those checks there could be a high maintenance burden to the duplication.

Was the main objection to WIN32_LEAN_AND_MEAN that it would be problematic for modules? If we're including windows.h, it seems strictly preferable to include it with WIN32_LEAN_AND_MEAN than without, since we'll pull in a lot less that way. Including windows.h without WIN32_LEAN_AND_MEAN can also interact with other headers badly sometimes, e.g. [winsock2.h](https://msdn.microsoft.com/en-us/library/windows/desktop/ms737629%28v=vs.85%29.aspx).

[EricWF]

include/__threading_support

34

It seems that dragging in the __deallocate macro is inevitable :-(

I submitted a patch to work around __deallocate here: https://reviews.llvm.org/D28426

[EricWF]

include/__threading_support

34

Was the main objection to WIN32_LEAN_AND_MEAN that it would be problematic for modules? If we're including windows.h, it seems strictly preferable to include it with WIN32_LEAN_AND_MEAN than without, since we'll pull in a lot less that way. Including windows.h without WIN32_LEAN_AND_MEAN can also interact with other headers badly sometimes, e.g. winsock2.h.

The objection is that it breaks user code. For example:

#include <thread>
#include <Windows.h> // Windows.h already included as lean and mean.

typedef NonLeanAndMeanSymbol foo; // ERROR NonLeanAndMeanSymbol not defined

[smeenai]

include/__threading_support

34

But without the WIN32_LEAN_AND_MEAN, we're gonna break

#include <thread>
#include <winsock2.h>

(you could fix this by reordering the includes, which would also fix your example, or by defining WIN32_LEAN_AND_MEAN yourself, but it doesn't seem great either)

[EricWF]

include/__threading_support

34

I would much rather break that code. The fact that Windows.h doesn't play nice with other Windows headers is a problem for Windows not libc++.

[EricWF]

include/__threading_support

34

Although I think avoiding the Windows.h include all together would be better (if possible). However I think that can be fixed after committing this.

[EricWF]

LGTM. I just successfully built this on Windows.

[compnerd]

SVN r291333

include/__threading_support

33–34

I think that I would rather do that in a follow up. This will require a fair amount of duplication.

include/__config

 # if defined(__FreeBSD__) || \
     defined(__Fuchsia__) || \
     defined(__NetBSD__) || \
     defined(__linux__) || \
     defined(__APPLE__) || \
     defined(__CloudABI__) || \
     defined(__sun__)
 #  define _LIBCPP_HAS_THREAD_API_PTHREAD
+# elif defined(_LIBCPP_WIN32API)
+#  define _LIBCPP_HAS_THREAD_API_WIN32
 # else
 #  error "No thread API"
 # endif // _LIBCPP_HAS_THREAD_API
 #endif // _LIBCPP_HAS_NO_THREADS
 
 #if defined(_LIBCPP_HAS_NO_THREADS) && defined(_LIBCPP_HAS_THREAD_API_PTHREAD)
 #  error _LIBCPP_HAS_THREAD_API_PTHREAD may only be defined when \
          _LIBCPP_HAS_NO_THREADS is not defined.

include/__threading_support

#ifndef __libcpp_has_include
   #else
     #define __libcpp_has_include(x) __has_include(x)
   #endif
 #endif
 
 #if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL) && \
     !__libcpp_has_include(<__external_threading>)
 // If the <__external_threading> header is absent, build libc++ against a
 // pthread-oriented thread api but leave out its implementation. This setup
 // allows building+testing of an externally-threaded library variant (on any
 // platform that supports pthreads). Here, an 'externally-threaded' library
 // variant is one where the implementation of the libc++ thread api is provided
 // as a separate library.
 #define _LIBCPP_HAS_THREAD_API_EXTERNAL_PTHREAD
 #endif
 
 #if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL) && \
     __libcpp_has_include(<__external_threading>)
 #include <__external_threading>
 #else
 
 #if defined(_LIBCPP_HAS_THREAD_API_PTHREAD) || \
     defined(_LIBCPP_HAS_THREAD_API_EXTERNAL_PTHREAD)
 #include <pthread.h>
 #include <sched.h>
+#elif defined(_LIBCPP_HAS_THREAD_API_WIN32)
+#include <assert.h>
+#define WIN32_LEAN_AND_MEAN
+#include <Windows.h>
+#include <process.h>
+#include <fibersapi.h>
+
+#include <chrono>
 #endif
 
 #if defined(_LIBCPP_HAS_THREAD_API_EXTERNAL)
 #define _LIBCPP_THREAD_ABI_VISIBILITY _LIBCPP_FUNC_VIS
 #else
 #define _LIBCPP_THREAD_ABI_VISIBILITY inline _LIBCPP_INLINE_VISIBILITY
 #endif
 

 
 // Condition Variable
 typedef pthread_cond_t __libcpp_condvar_t;
 #define _LIBCPP_CONDVAR_INITIALIZER PTHREAD_COND_INITIALIZER
 
 // Execute once
 typedef pthread_once_t __libcpp_exec_once_flag;
 #define _LIBCPP_EXEC_ONCE_INITIALIZER PTHREAD_ONCE_INIT
 
 // Thread id
 typedef pthread_t __libcpp_thread_id;
 
 // Thread
 typedef pthread_t __libcpp_thread_t;
 
 // Thrad Local Storage
 typedef pthread_key_t __libcpp_tls_key;
+
+#define _LIBCPP_TLS_DESTRUCTOR_CC
+#else
+// Mutex
+typedef SRWLOCK __libcpp_mutex_t;
+#define _LIBCPP_MUTEX_INITIALIZER SRWLOCK_INIT
+
+typedef CRITICAL_SECTION __libcpp_recursive_mutex_t;
+
+// Condition Variable
+typedef CONDITION_VARIABLE __libcpp_condvar_t;
+#define _LIBCPP_CONDVAR_INITIALIZER CONDITION_VARIABLE_INIT
+
+// Execute Once
+typedef INIT_ONCE __libcpp_exec_once_flag;
+#define _LIBCPP_EXEC_ONCE_INITIALIZER INIT_ONCE_STATIC_INIT
+
+// Thread ID
+typedef DWORD __libcpp_thread_id;
+
+// Thread
+typedef HANDLE __libcpp_thread_t;
+
+// Thread Local Storage
+typedef DWORD __libcpp_tls_key;
+
+#define _LIBCPP_TLS_DESTRUCTOR_CC WINAPI
 #endif
 
 // Mutex
 _LIBCPP_THREAD_ABI_VISIBILITY
 int __libcpp_recursive_mutex_init(__libcpp_recursive_mutex_t *__m);
 
 _LIBCPP_THREAD_ABI_VISIBILITY
 int __libcpp_recursive_mutex_lock(__libcpp_recursive_mutex_t *__m);

 {
   return pthread_cond_wait(__cv, __m);
 }
 
 int __libcpp_condvar_timedwait(__libcpp_condvar_t *__cv, __libcpp_mutex_t *__m,
                                timespec *__ts)
 {
   return pthread_cond_timedwait(__cv, __m, __ts);
 }
 
 int __libcpp_condvar_destroy(__libcpp_condvar_t *__cv)
 {
   return pthread_cond_destroy(__cv);
 }
 
 // Execute once
 int __libcpp_execute_once(__libcpp_exec_once_flag *flag,
                           void (*init_routine)(void)) {
   return pthread_once(flag, init_routine);
 }
 
 // Thread id
 #if defined(__APPLE__) && !defined(__arm__)

 }
 
 int __libcpp_thread_join(__libcpp_thread_t *__t)
 {
   return pthread_join(*__t, 0);
 }
 
 int __libcpp_thread_detach(__libcpp_thread_t *__t)
 {
   return pthread_detach(*__t);
 }
 
 void __libcpp_thread_yield()
 {
   sched_yield();
 }
 
 // Thread local storage
 int __libcpp_tls_create(__libcpp_tls_key *__key, void (*__at_exit)(void *))
 {
   return pthread_key_create(__key, __at_exit);
 }
 
 void *__libcpp_tls_get(__libcpp_tls_key __key)
 {
   return pthread_getspecific(__key);
 }
 
 int __libcpp_tls_set(__libcpp_tls_key __key, void *__p)
 {
     return pthread_setspecific(__key, __p);
 }
 
+#elif defined(_LIBCPP_HAS_THREAD_API_WIN32)
+
+// Mutex
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_init(__libcpp_recursive_mutex_t *__m)
+{
+  InitializeCriticalSection(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_lock(__libcpp_recursive_mutex_t *__m)
+{
+  EnterCriticalSection(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_trylock(__libcpp_recursive_mutex_t *__m)
+{
+  TryEnterCriticalSection(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_unlock(__libcpp_recursive_mutex_t *__m)
+{
+  LeaveCriticalSection(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_recursive_mutex_destroy(__libcpp_recursive_mutex_t *__m)
+{
+  static_cast<void>(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_lock(__libcpp_mutex_t *__m)
+{
+  AcquireSRWLockExclusive(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_trylock(__libcpp_mutex_t *__m)
+{
+  TryAcquireSRWLockExclusive(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_unlock(__libcpp_mutex_t *__m)
+{
+  ReleaseSRWLockExclusive(__m);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_mutex_destroy(__libcpp_mutex_t *__m)
+{
+  static_cast<void>(__m);
+  return 0;
+}
+
+// Condition Variable
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_signal(__libcpp_condvar_t *__cv)
+{
+  WakeConditionVariable(__cv);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_broadcast(__libcpp_condvar_t *__cv)
+{
+  WakeAllConditionVariable(__cv);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_wait(__libcpp_condvar_t *__cv, __libcpp_mutex_t *__m)
+{
+  SleepConditionVariableSRW(__cv, __m, INFINITE, 0);
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_timedwait(__libcpp_condvar_t *__cv, __libcpp_mutex_t *__m,
+                               timespec *__ts)
+{
+  using namespace _VSTD::chrono;
+
+  auto duration = seconds(__ts->tv_sec) + nanoseconds(__ts->tv_nsec);
+  auto abstime =
+      system_clock::time_point(duration_cast<system_clock::duration>(duration));
+  auto timeout_ms = duration_cast<milliseconds>(abstime - system_clock::now());
+
+  if (!SleepConditionVariableSRW(__cv, __m,
+                                 timeout_ms.count() > 0 ? timeout_ms.count()
+                                                        : 0,
+                                 0))
+    return GetLastError();
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_condvar_destroy(__libcpp_condvar_t *__cv)
+{
+  static_cast<void>(__cv);
+  return 0;
+}
+
+// Execute Once
+static inline _LIBCPP_ALWAYS_INLINE BOOL CALLBACK
+__libcpp_init_once_execute_once_thunk(PINIT_ONCE __init_once, PVOID __parameter,
+                                      PVOID *__context)
+{
+  static_cast<void>(__init_once);
+  static_cast<void>(__context);
+
+  void (*init_routine)(void) = reinterpret_cast<void (*)(void)>(__parameter);
+  init_routine();
+  return TRUE;
+}
+
+int __libcpp_execute_once(__libcpp_exec_once_flag *__flag,
+                          void (*__init_routine)(void))
+{
+  if (!InitOnceExecuteOnce(__flag, __libcpp_init_once_execute_once_thunk,
+                           reinterpret_cast<void *>(__init_routine), NULL))
+    return GetLastError();
+  return 0;
+}
+
+// Thread ID
+_LIBCPP_ALWAYS_INLINE
+bool __libcpp_thread_id_equal(__libcpp_thread_id __lhs,
+                              __libcpp_thread_id __rhs)
+{
+  return __lhs == __rhs;
+}
+
+_LIBCPP_ALWAYS_INLINE
+bool __libcpp_thread_id_less(__libcpp_thread_id __lhs, __libcpp_thread_id __rhs)
+{
+  return __lhs < __rhs;
+}
+
+// Thread
+struct __libcpp_beginthreadex_thunk_data
+{
+  void *(*__func)(void *);
+  void *__arg;
+};
+
+static inline _LIBCPP_ALWAYS_INLINE unsigned int WINAPI
+__libcpp_beginthreadex_thunk(void *__data)
+{
+  __libcpp_beginthreadex_thunk_data data =
+      *reinterpret_cast<__libcpp_beginthreadex_thunk_data *>(__data);
+  delete __data;
+  return reinterpret_cast<unsigned int>(data.__func(data.__arg));
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_create(__libcpp_thread_t *__t, void (*__func)(void *),
+                           void *__arg)
+{
+  auto *data = new __libcpp_beginthreadex_thunk_data;
+  data->__func = __func;
+  data->__arg = __arg;
+
+  *__t = reinterpret_cast<HANDLE>(_beginthreadex(NULL, 0,
+                                                 __libcpp_beginthreadex_thunk,
+                                                 data, 0, NULL));
+  if (*__t)
+    return 0;
+  return GetLastError();
+}
+
+_LIBCPP_ALWAYS_INLINE
+__libcpp_thread_id __libcpp_thread_get_current_id()
+{
+  return GetCurrentThreadId();
+}
+
+_LIBCPP_ALWAYS_INLINE
+__libcpp_thread_id __libcpp_thread_get_id(const __libcpp_thread_t *__t)
+{
+  return GetThreadId(*__t);
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_join(__libcpp_thread_t *__t)
+{
+  if (WaitForSingleObjectEx(*__t, INFINITE, FALSE) == WAIT_FAILED)
+    return GetLastError();
+  if (!CloseHandle(*__t))
+    return GetLastError();
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_detach(__libcpp_thread_t *__t)
+{
+  if (!CloseHandle(*__t))
+    return GetLastError();
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_thread_yield()
+{
+  SwitchToThread();
+}
+
+// Thread Local Storage
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_tls_create(__libcpp_tls_key* __key,
+                        void(_LIBCPP_TLS_DESTRUCTOR_CC* __at_exit)(void*))
+{
+  *__key = FlsAlloc(__at_exit);
+  if (*__key == FLS_OUT_OF_INDEXES)
+    return GetLastError();
+  return 0;
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_tls_get(__libcpp_tls_key __key)
+{
+  return FlsGetValue(__key);
+}
+
+_LIBCPP_ALWAYS_INLINE
+int __libcpp_tls_set(__libcpp_tls_key __key, void *__p)
+{
+  if (!FlsSetValue(__key, __p))
+    return GetLastError();
+  return 0;
+}
+
 #endif // _LIBCPP_HAS_THREAD_API_PTHREAD
 
 _LIBCPP_END_NAMESPACE_STD
 
 #endif // !_LIBCPP_HAS_THREAD_API_EXTERNAL || !__libcpp_has_include(<__external_threading>)
 
 #endif // _LIBCPP_HAS_NO_THREADS
 
 #endif // _LIBCPP_THREADING_SUPPORT

include/thread

class __thread_specific_ptr
      // and only with _Tp == __thread_struct.
     static_assert((is_same<_Tp, __thread_struct>::value), "");
     __thread_specific_ptr();
     friend _LIBCPP_FUNC_VIS __thread_specific_ptr<__thread_struct>& __thread_local_data();
 
     __thread_specific_ptr(const __thread_specific_ptr&);
     __thread_specific_ptr& operator=(const __thread_specific_ptr&);
 
-    static void __at_thread_exit(void*);
+    static void _LIBCPP_TLS_DESTRUCTOR_CC __at_thread_exit(void*);
+
 public:
     typedef _Tp* pointer;
 
     ~__thread_specific_ptr();
 
     _LIBCPP_INLINE_VISIBILITY
     pointer get() const {return static_cast<_Tp*>(__libcpp_tls_get(__key_));}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator*() const {return *get();}
     _LIBCPP_INLINE_VISIBILITY
     pointer operator->() const {return get();}
     void set_pointer(pointer __p);
 };
 
 template <class _Tp>
-void
+void _LIBCPP_TLS_DESTRUCTOR_CC
 __thread_specific_ptr<_Tp>::__at_thread_exit(void* __p)
 {
     delete static_cast<pointer>(__p);
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::__thread_specific_ptr()
 {
-    int __ec = __libcpp_tls_create(
-        &__key_,
-        &__thread_specific_ptr::__at_thread_exit);
-    if (__ec)
-        __throw_system_error(__ec,
-                           "__thread_specific_ptr construction failed");
+  int __ec =
+      __libcpp_tls_create(&__key_, &__thread_specific_ptr::__at_thread_exit);
+    __throw_system_error(__ec, "__thread_specific_ptr construction failed");
 }
 
 template <class _Tp>
 __thread_specific_ptr<_Tp>::~__thread_specific_ptr()
 {
     // __thread_specific_ptr is only created with a static storage duration
     // so this destructor is only invoked during program termination. Invoking
     // pthread_key_delete(__key_) may prevent other threads from deleting their