Threading: use independent llvm::thread implementation on Apple platforms to increase stack size

[t.p.northover]

Summary

Darwin's libpthread creates threads with only 512KB of stack space, as opposed to the main thread's 8MB. 512KB often isn't enough to compile properly, so we get stack overflows during ThinLTO.

Unfortunately, with Posix threads the stack size can only be set during creation so there's no way to use the real std::thread to overcome this limit (even with the native handle).

So this patch adds a real implementation of llvm::thread (with the same interface as std::thread) when compiling for Apple that does request the full 8MB stack size. Most of the implementation is cribbed from libc++, with abstractions removed because we know the underlying implementation and target standard (and because we can tell users not to be daft and #define Thread 0).

[t.p.northover]

Actual main thread stack size is 8MB, but I'd accidentally implemented 2MB originally.

[dexonsmith]

I wonder, could this be used to simplify RunSafelyOnThread, and/or llvm_execute_on_thread_impl? I figure all that complexity is because of the same limitation, but maybe it's not something we can fix, since we exposed a libclang API (clang_executeOnThread) that allows specifying a stack size.

I guess one option would be to never use std::thread, and repurpose the guts of llvm_execute_on_thread_impl to implement a cross-platform llvm::thread that allows specifying the stack size as an optional constructor parameter (and then change llvm_execute_on_thread to use llvm::thread). WDYT?

[dexonsmith]

@dblaikie, wouldn't mind your thoughts on this idea (expanding the interface of llvm::thread to include an optional stack size parameter).

[t.p.northover]

I guess one option would be to never use std::thread, and repurpose the guts of llvm_execute_on_thread_impl to implement a cross-platform llvm::thread that allows specifying the stack size as an optional constructor parameter (and then change llvm_execute_on_thread to use llvm::thread). WDYT?

The llvm_execute_on_thread family only seems to have three users and is definitely a less appealing interface, I'd be very tempted round this llvm::thread out as you suggest and switch all users to it.

[dexonsmith]

In D103165#2786618, @t.p.northover wrote:

I guess one option would be to never use std::thread, and repurpose the guts of llvm_execute_on_thread_impl to implement a cross-platform llvm::thread that allows specifying the stack size as an optional constructor parameter (and then change llvm_execute_on_thread to use llvm::thread). WDYT?

The llvm_execute_on_thread family only seems to have three users and is definitely a less appealing interface, I'd be very tempted round this llvm::thread out as you suggest and switch all users to it.

SGTM; not sure if there'll be other opinions, but it does seem like a nice general cleanup. Also this way llvm::thread users that don't need full-sized stacks can continue getting the Darwin default of smaller pages.

[t.p.northover]

• Add llvm::thread for all potential platforms, allowing stack size to be specified.
• llvm::thread based on the Threading.inc where possible, else std::thread, else in-thread synchronous execution (with progressively more failures).
• Remove llvm_execute_on_thread* in favour of this new llvm::thread.

I've managed to test it on all 3 major platforms, and the fallbacks by fiddling #ifs. So it should at least compile everywhere.

Also this way llvm::thread users that don't need full-sized stacks can continue getting the Darwin default of smaller pages.

I'm not sure I've implemented this, since I've just made 8MB the Darwin default here. The case I care about is LTOBackend via ThreadPool. Would you prefer me to push an optional StackSize argument into ThreadPool too and make LTOBackend specify it?

I kind of think this has come up often enough (it's at least the second time I've had to fix it somewhere), and 8MB is small enough for anything running Clang that it's better to make sure it doesn't happen again.

[t.p.northover]

llvm/include/llvm/Support/thread.h

168

An alternative here would have been to inherit from std::thread but that seemed a bit icky to me. Happy to switch if the consensus is otherwise.

[t.p.northover]

Ping.

[dexonsmith]

This LGTM, with some whitespace nits inline.

In D103165#2790932, @t.p.northover wrote:

Also this way llvm::thread users that don't need full-sized stacks can continue getting the Darwin default of smaller pages.

I'm not sure I've implemented this, since I've just made 8MB the Darwin default here. The case I care about is LTOBackend via ThreadPool. Would you prefer me to push an optional StackSize argument into ThreadPool too and make LTOBackend specify it?

I kind of think this has come up often enough (it's at least the second time I've had to fix it somewhere), and 8MB is small enough for anything running Clang that it's better to make sure it doesn't happen again.

8MB by default is probably fine; I don't feel strongly either way.

llvm/include/llvm/Support/thread.h

107

Seems like you might as well run git-clang-format to pick up these changes...

261

Looks like it's missing a newline here at the end of the file.

[t.p.northover]

Thanks Duncan, committed as 727e1c9be3a5

llvm/include/llvm/Support/thread.h

107

Yes, I'd already done that on my local commit.

clang-tools-extra/clangd/support/Threading.cpp

 #include "support/Threading.h"
 #include "support/Trace.h"
 #include "llvm/ADT/ScopeExit.h"
 #include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/thread.h"
 #include "llvm/Support/Threading.h"
 #include <atomic>
 #include <thread>
 #ifdef __USE_POSIX
 #include <pthread.h>
 #elif defined(__APPLE__)
 #include <sys/resource.h>
 #elif defined(_WIN32)
 #include <windows.h>
auto Task = [Name = Name.str(), Action = std::move(Action),
                Cleanup = std::move(CleanupTask)]() mutable {
     llvm::set_thread_name(Name);
     Action();
     // Make sure function stored by ThreadFunc is destroyed before Cleanup runs.
     Action = nullptr;
   };
 
   // Ensure our worker threads have big enough stacks to run clang.
-  llvm::llvm_execute_on_thread_async(std::move(Task),
-                                     /*clang::DesiredStackSize*/ 8 << 20);
+  llvm::thread Thread(
+      /*clang::DesiredStackSize*/ llvm::Optional<unsigned>(8 << 20),
+      std::move(Task));
+  Thread.detach();
 }
 
 Deadline timeoutSeconds(llvm::Optional<double> Seconds) {
   using namespace std::chrono;
   if (!Seconds)
     return Deadline::infinity();
   return steady_clock::now() +
          duration_cast<steady_clock::duration>(duration<double>(*Seconds));

clang/tools/libclang/CIndex.cpp

 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/SaveAndRestore.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/TargetSelect.h"
+#include "llvm/Support/thread.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/Support/Timer.h"
 #include "llvm/Support/raw_ostream.h"
 #include <mutex>
 
 #if LLVM_ENABLE_THREADS != 0 && defined(__APPLE__)
 #define USE_DARWIN_THREADS
 #endif
 
 #ifdef USE_DARWIN_THREADS
 #include <pthread.h>

 
 void clang_enableStackTraces(void) {
   // FIXME: Provide an argv0 here so we can find llvm-symbolizer.
   llvm::sys::PrintStackTraceOnErrorSignal(StringRef());
 }
 
 void clang_executeOnThread(void (*fn)(void *), void *user_data,
                            unsigned stack_size) {
-  llvm::llvm_execute_on_thread(fn, user_data,
-                               stack_size == 0
-                                   ? clang::DesiredStackSize
-                                   : llvm::Optional<unsigned>(stack_size));
+  llvm::thread Thread(stack_size == 0 ? clang::DesiredStackSize
+                                      : llvm::Optional<unsigned>(stack_size),
+                      fn, user_data);
+  Thread.join();
 }
 
 //===----------------------------------------------------------------------===//
 // Token-based Operations.
 //===----------------------------------------------------------------------===//
 
 /* CXToken layout:
  *   int_data[0]: a CXTokenKind

llvm/include/llvm/Support/CrashRecoveryContext.h

public:
   bool RunSafely(void (*Fn)(void*), void *UserData) {
     return RunSafely([&]() { Fn(UserData); });
   }
 
   /// Execute the provide callback function (with the given arguments) in
   /// a protected context which is run in another thread (optionally with a
   /// requested stack size).
   ///
-  /// See RunSafely() and llvm_execute_on_thread().
+  /// See RunSafely().
   ///
   /// On Darwin, if PRIO_DARWIN_BG is set on the calling thread, it will be
   /// propagated to the new thread as well.
   bool RunSafelyOnThread(function_ref<void()>, unsigned RequestedStackSize = 0);
   bool RunSafelyOnThread(void (*Fn)(void*), void *UserData,
                          unsigned RequestedStackSize = 0) {
     return RunSafelyOnThread([&]() { Fn(UserData); }, RequestedStackSize);
   }

llvm/include/llvm/Support/Threading.h

 
 namespace llvm {
 class Twine;
 
 /// Returns true if LLVM is compiled with support for multi-threading, and
 /// false otherwise.
 bool llvm_is_multithreaded();
 
-/// Execute the given \p UserFn on a separate thread, passing it the provided \p
-/// UserData and waits for thread completion.
-///
-/// This function does not guarantee that the code will actually be executed
-/// on a separate thread or honoring the requested stack size, but tries to do
-/// so where system support is available.
-///
-/// \param UserFn - The callback to execute.
-/// \param UserData - An argument to pass to the callback function.
-/// \param StackSizeInBytes - A requested size (in bytes) for the thread stack
-/// (or None for default)
-void llvm_execute_on_thread(
-    void (*UserFn)(void *), void *UserData,
-    llvm::Optional<unsigned> StackSizeInBytes = llvm::None);
-
-/// Schedule the given \p Func for execution on a separate thread, then return
-/// to the caller immediately. Roughly equivalent to
-/// `std::thread(Func).detach()`, except it allows requesting a specific stack
-/// size, if supported for the platform.
-///
-/// This function would report a fatal error if it can't execute the code
-/// on a separate thread.
-///
-/// \param Func - The callback to execute.
-/// \param StackSizeInBytes - A requested size (in bytes) for the thread stack
-/// (or None for default)
-void llvm_execute_on_thread_async(
-    llvm::unique_function<void()> Func,
-    llvm::Optional<unsigned> StackSizeInBytes = llvm::None);
-
 #if LLVM_THREADING_USE_STD_CALL_ONCE
 
   typedef std::once_flag once_flag;
 
 #else
 
   enum InitStatus { Uninitialized = 0, Wait = 1, Done = 2 };
 

llvm/include/llvm/Support/thread.h

 // which is either a typedef of std::thread or a replacement that calls the
 // function synchronously depending on the value of LLVM_ENABLE_THREADS.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_SUPPORT_THREAD_H
 #define LLVM_SUPPORT_THREAD_H
 
+#include "llvm/ADT/Optional.h"
 #include "llvm/Config/llvm-config.h"
 
+#ifdef _WIN32
+#include "llvm/Support/Windows/WindowsSupport.h"
+#endif
+
 #if LLVM_ENABLE_THREADS
 
 #include <thread>
 
 namespace llvm {
-typedef std::thread thread;
+
+# if LLVM_ON_UNIX || _WIN32
+
+/// LLVM thread following std::thread interface with added constructor to
+/// specify stack size.
+class thread {
+  template<typename FPtr, typename ...Args, size_t ...Indices>
+  static void Apply(std::tuple<FPtr, Args...> &Callee, std::index_sequence<Indices...>) {
+    std::move(std::get<0>(Callee))(std::move(std::get<Indices + 1>(Callee))...);
+  }
+
+  template<typename CalleeTuple>
+  static void GenericThreadProxy(void *Ptr) {
+    std::unique_ptr<CalleeTuple> Callee(static_cast<CalleeTuple *>(Ptr));
+
+    // FIXME: use std::apply when C++17 is allowed.
+    std::make_index_sequence<std::tuple_size<CalleeTuple>() - 1> Indices{};
+    Apply(*Callee.get(), Indices);
+  }
+public:
+#if LLVM_ON_UNIX
+  using native_handle_type = pthread_t;
+  using id = pthread_t;
+  using start_routine_type = void *(*)(void *);
+
+  template <typename CalleeTuple> static void *ThreadProxy(void *Ptr) {
+    GenericThreadProxy<CalleeTuple>(Ptr);
+    return nullptr;
+  }
+#elif _WIN32
+  using native_handle_type = HANDLE;
+  using id = HANDLE;
+  using start_routine_type = unsigned (__stdcall *)(void *);
+
+  template <typename CalleeTuple> static unsigned __stdcall ThreadProxy(void *Ptr) {
+    GenericThreadProxy<CalleeTuple>(Ptr);
+    return 0;
+  }
+#endif
+
+#if defined(__APPLE__)
+  // Darwin's default stack size for threads except the main one is only 512KB,
+  // which is not enough for some/many normal LLVM compilations. This implements
+  // the same interface as std::thread but requests the same stack size as the
+  // main thread (8MB) before creation.
+  static const constexpr llvm::Optional<unsigned> DefaultStackSize =
+      8 * 1024 * 1024;
+#else
+  static const constexpr llvm::Optional<unsigned> DefaultStackSize = None;
+#endif
+
+  thread() : Thread(native_handle_type()) {}
+  thread(thread &&Other) noexcept
+      : Thread(std::exchange(Other.Thread, native_handle_type())) {}
+
+  template <class Function, class... Args>
+  explicit thread(Function &&f, Args &&...args)
+      : thread(DefaultStackSize, f, args...) {}
+
+  template <class Function, class... Args>
+  explicit thread(llvm::Optional<unsigned> StackSizeInBytes, Function &&f,
+                  Args &&...args);
+  thread(const thread &) = delete;
+
+  ~thread() {
+    if (joinable())
+      std::terminate();
+  }
+
+  thread &operator=(thread &&Other) noexcept {
+    if (joinable())
+      std::terminate();
+    Thread = std::exchange(Other.Thread, native_handle_type());
+    return *this;
+  }
+
+  bool joinable() const noexcept {
+    return Thread != native_handle_type();
+  }
+
+  id get_id() const noexcept {
+    return Thread;
+  }
+
+  native_handle_type native_handle() const noexcept {
+    return Thread;
+  }
+
+  static unsigned hardware_concurrency() {
+    return std::thread::hardware_concurrency();
+  };
+
+  inline void join();
+  inline void detach();
+
+  void swap(llvm::thread &Other) noexcept {
+    std::swap(Thread, Other.Thread);
+  }
+private:
+  native_handle_type Thread;
+};
+
+thread::native_handle_type
+llvm_execute_on_thread_impl(thread::start_routine_type ThreadFunc, void *Arg,
+                            llvm::Optional<unsigned> StackSizeInBytes);
+void llvm_thread_join_impl(thread::native_handle_type Thread);
+void llvm_thread_detach_impl(thread::native_handle_type Thread);
+
+template <class Function, class... Args>
+thread::thread(llvm::Optional<unsigned> StackSizeInBytes, Function &&f,
+               Args &&...args) {
+  typedef std::tuple<typename std::decay<Function>::type,
+                     typename std::decay<Args>::type...>
+      CalleeTuple;
+  std::unique_ptr<CalleeTuple> Callee(
+      new CalleeTuple(std::forward<Function>(f), std::forward<Args>(args)...));
+
+  Thread = llvm_execute_on_thread_impl(ThreadProxy<CalleeTuple>, Callee.get(),
+                                       StackSizeInBytes);
+  if (Thread != native_handle_type())
+    Callee.release();
 }
 
+void thread::join() {
+  llvm_thread_join_impl(Thread);
+  Thread = native_handle_type();
+}
+
+void thread::detach() {
+  llvm_thread_detach_impl(Thread);
+  Thread = native_handle_type();
+}
+
+#else // !LLVM_ON_UNIX && !_WIN32
+
+/// std::thread backed implementation of llvm::thread interface that ignores the
+/// stack size request.
+class thread {
+public:
+  using native_handle_type = std::thread::native_handle_type;
+  using id = std::thread::id;
+
+  thread() : Thread(std::thread()) {}
+  thread(thread &&Other) noexcept
+      : Thread(std::exchange(Other.Thread, std::thread())) {}
+
+  template <class Function, class... Args>
+  explicit thread(llvm::Optional<unsigned> StackSizeInBytes, Function &&f,
+                  Args &&...args)
+      : Thread(std::forward<Function>(f), std::forward<Args>(args)...) {}
+
+  template <class Function, class... Args>
+  explicit thread(Function &&f, Args &&...args) : Thread(f, args...) {}
+
+  thread(const thread &) = delete;
+
+  ~thread() {}
+
+  thread &operator=(thread &&Other) noexcept {
+    Thread = std::exchange(Other.Thread, std::thread());
+    return *this;
+  }
+
+  bool joinable() const noexcept {
+    return Thread.joinable();
+  }
+
+  id get_id() const noexcept {
+    return Thread.get_id();
+  }
+
+  native_handle_type native_handle() noexcept {
+    return Thread.native_handle();
+  }
+
+  static unsigned hardware_concurrency() {
+    return std::thread::hardware_concurrency();
+  };
+
+  inline void join() {
+    Thread.join();
+  }
+  inline void detach() {
+    Thread.detach();
+  }
+
+  void swap(llvm::thread &Other) noexcept {
+    std::swap(Thread, Other.Thread);
+  }
+private:
+  std::thread Thread;
+};
+
+#endif // LLVM_ON_UNIX || _WIN32
+
+} // namespace llvm
+
 #else // !LLVM_ENABLE_THREADS
 
 #include <utility>
 
 namespace llvm {
 
 struct thread {
   thread() {}
   thread(thread &&other) {}
   template <class Function, class... Args>
+  explicit thread(llvm::Optional<unsigned> StackSizeInBytes, Function &&f,
+                  Args &&...args) {
+    f(std::forward<Args>(args)...);
+  }
+  template <class Function, class... Args>
   explicit thread(Function &&f, Args &&... args) {
     f(std::forward<Args>(args)...);
   }
   thread(const thread &) = delete;
 
+  void detach() {
+    report_fatal_error("Detaching from a thread does not make sense with no "
+                       "threading support");
+  }
   void join() {}
   static unsigned hardware_concurrency() { return 1; };
 };
 
 }
 
 #endif // LLVM_ENABLE_THREADS
 
-#endif
+#endif // LLVM_SUPPORT_THREAD_H
 No newline at end of file

llvm/lib/Support/CrashRecoveryContext.cpp

 //===--- CrashRecoveryContext.cpp - Crash Recovery ------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/Support/CrashRecoveryContext.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ExitCodes.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Signals.h"
+#include "llvm/Support/thread.h"
 #include "llvm/Support/ThreadLocal.h"
 #include <mutex>
 #include <setjmp.h>
 
 using namespace llvm;
 
 namespace {
 
 struct CrashRecoveryContextImpl;
if (Info->UseBackgroundPriority)
     setThreadBackgroundPriority();
 
   Info->Result = Info->CRC->RunSafely(Info->Fn);
 }
 bool CrashRecoveryContext::RunSafelyOnThread(function_ref<void()> Fn,
                                              unsigned RequestedStackSize) {
   bool UseBackgroundPriority = hasThreadBackgroundPriority();
   RunSafelyOnThreadInfo Info = { Fn, this, UseBackgroundPriority, false };
-  llvm_execute_on_thread(RunSafelyOnThread_Dispatch, &Info,
-                         RequestedStackSize == 0
-                             ? llvm::None
-                             : llvm::Optional<unsigned>(RequestedStackSize));
+  llvm::thread Thread(RequestedStackSize == 0
+                          : llvm::Optional<unsigned>(RequestedStackSize),
+                      RunSafelyOnThread_Dispatch, &Info);
+  Thread.join();
+
   if (CrashRecoveryContextImpl *CRC = (CrashRecoveryContextImpl *)Impl)
     CRC->setSwitchedThread();
   return Info.Result;
 }

llvm/lib/Support/Threading.cpp

 //===-- llvm/Support/Threading.cpp- Control multithreading mode --*- C++ -*-==//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 //
 // This file defines helper functions for running LLVM in a multi-threaded
 // environment.
 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/Support/thread.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/Config/config.h"
 #include "llvm/Support/Host.h"
 
 #include <cassert>
 #include <errno.h>
 #include <stdlib.h>
 #include <string.h>
 
 using namespace llvm;
 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only TRULY operating system
 //===          independent code.
 //===----------------------------------------------------------------------===//
 
 bool llvm::llvm_is_multithreaded() {
 #if LLVM_ENABLE_THREADS != 0
   return true;
 #else
   return false;
 #endif
 }
 
 #if LLVM_ENABLE_THREADS == 0 ||                                                \
     (!defined(_WIN32) && !defined(HAVE_PTHREAD_H))
-// Support for non-Win32, non-pthread implementation.
-void llvm::llvm_execute_on_thread(void (*Fn)(void *), void *UserData,
-                                  llvm::Optional<unsigned> StackSizeInBytes) {
-  (void)StackSizeInBytes;
-  Fn(UserData);
-}
-
 uint64_t llvm::get_threadid() { return 0; }
 
 uint32_t llvm::get_max_thread_name_length() { return 0; }
 
 void llvm::set_thread_name(const Twine &Name) {}
 
 void llvm::get_thread_name(SmallVectorImpl<char> &Name) { Name.clear(); }
 
 llvm::BitVector llvm::get_thread_affinity_mask() { return {}; }
 
 unsigned llvm::ThreadPoolStrategy::compute_thread_count() const {
   // When threads are disabled, ensure clients will loop at least once.
   return 1;
 }
 
-#if LLVM_ENABLE_THREADS == 0
-void llvm::llvm_execute_on_thread_async(
-    llvm::unique_function<void()> Func,
-    llvm::Optional<unsigned> StackSizeInBytes) {
-  (void)Func;
-  (void)StackSizeInBytes;
-  report_fatal_error("Spawning a detached thread doesn't make sense with no "
-                     "threading support");
-}
-#else
-// Support for non-Win32, non-pthread implementation.
-void llvm::llvm_execute_on_thread_async(
-    llvm::unique_function<void()> Func,
-    llvm::Optional<unsigned> StackSizeInBytes) {
-  (void)StackSizeInBytes;
-  std::thread(std::move(Func)).detach();
-}
-#endif
-
 #else
 
 int computeHostNumHardwareThreads();
 
 unsigned llvm::ThreadPoolStrategy::compute_thread_count() const {
   int MaxThreadCount = UseHyperThreads ? computeHostNumHardwareThreads()
                                        : sys::getHostNumPhysicalCores();
   if (MaxThreadCount <= 0)
     MaxThreadCount = 1;
   if (ThreadsRequested == 0)
     return MaxThreadCount;
   if (!Limit)
     return ThreadsRequested;
   return std::min((unsigned)MaxThreadCount, ThreadsRequested);
 }
 
-namespace {
-struct SyncThreadInfo {
-  void (*UserFn)(void *);
-  void *UserData;
-};
-
-using AsyncThreadInfo = llvm::unique_function<void()>;
-
-enum class JoiningPolicy { Join, Detach };
-} // namespace
-
 // Include the platform-specific parts of this class.
 #ifdef LLVM_ON_UNIX
 #include "Unix/Threading.inc"
 #endif
 #ifdef _WIN32
 #include "Windows/Threading.inc"
 #endif
 
-void llvm::llvm_execute_on_thread(void (*Fn)(void *), void *UserData,
-                                  llvm::Optional<unsigned> StackSizeInBytes) {
-
-  SyncThreadInfo Info = {Fn, UserData};
-  llvm_execute_on_thread_impl(threadFuncSync, &Info, StackSizeInBytes,
-                              JoiningPolicy::Join);
-}
-
-void llvm::llvm_execute_on_thread_async(
-    llvm::unique_function<void()> Func,
-    llvm::Optional<unsigned> StackSizeInBytes) {
-  llvm_execute_on_thread_impl(&threadFuncAsync,
-                              new AsyncThreadInfo(std::move(Func)),
-                              StackSizeInBytes, JoiningPolicy::Detach);
-}
-
 #endif
 
 Optional<ThreadPoolStrategy>
 llvm::get_threadpool_strategy(StringRef Num, ThreadPoolStrategy Default) {
   if (Num == "all")
     return llvm::hardware_concurrency();
   if (Num.empty())
     return Default;

llvm/lib/Support/Unix/Threading.inc

 #endif
 
 #if defined(__linux__)
 #include <sched.h>       // For sched_getaffinity
 #include <sys/syscall.h> // For syscall codes
 #include <unistd.h>      // For syscall()
 #endif
 
-static void *threadFuncSync(void *Arg) {
-  SyncThreadInfo *TI = static_cast<SyncThreadInfo *>(Arg);
-  TI->UserFn(TI->UserData);
-  return nullptr;
-}
-
-static void *threadFuncAsync(void *Arg) {
-  std::unique_ptr<AsyncThreadInfo> Info(static_cast<AsyncThreadInfo *>(Arg));
-  (*Info)();
-  return nullptr;
-}
-
-static void
-llvm_execute_on_thread_impl(void *(*ThreadFunc)(void *), void *Arg,
-                            llvm::Optional<unsigned> StackSizeInBytes,
-                            JoiningPolicy JP) {
+pthread_t
+llvm::llvm_execute_on_thread_impl(void *(*ThreadFunc)(void *), void *Arg,
+                                  llvm::Optional<unsigned> StackSizeInBytes) {
   int errnum;
 
   // Construct the attributes object.
   pthread_attr_t Attr;
   if ((errnum = ::pthread_attr_init(&Attr)) != 0) {
     ReportErrnumFatal("pthread_attr_init failed", errnum);
   }
 
if (StackSizeInBytes) {
     }
   }
 
   // Construct and execute the thread.
   pthread_t Thread;
   if ((errnum = ::pthread_create(&Thread, &Attr, ThreadFunc, Arg)) != 0)
     ReportErrnumFatal("pthread_create failed", errnum);
 
-  if (JP == JoiningPolicy::Join) {
-    // Wait for the thread
-    if ((errnum = ::pthread_join(Thread, nullptr)) != 0) {
-      ReportErrnumFatal("pthread_join failed", errnum);
-    }
-  } else if (JP == JoiningPolicy::Detach) {
-    if ((errnum = ::pthread_detach(Thread)) != 0) {
-      ReportErrnumFatal("pthread_detach failed", errnum);
-    }
-  }
+  return Thread;
+
+void llvm::llvm_thread_detach_impl(pthread_t Thread) {
+  int errnum;
+
+
+void llvm::llvm_thread_join_impl(pthread_t Thread) {
+  int errnum;
+
+  if ((errnum = ::pthread_join(Thread, nullptr)) != 0) {
+    ReportErrnumFatal("pthread_join failed", errnum);
+  }
 }
 
 uint64_t llvm::get_threadid() {
 #if defined(__APPLE__)
   // Calling "mach_thread_self()" bumps the reference count on the thread
   // port, so we need to deallocate it. mach_task_self() doesn't bump the ref
   // count.
   thread_port_t Self = mach_thread_self();
   mach_port_deallocate(mach_task_self(), Self);
   return Self;
 #elif defined(__FreeBSD__)
#elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
   return 16;
 #elif defined(__OpenBSD__)
   return 32;
 #else
   return 0;
 #endif
 }
 
 uint32_t llvm::get_max_thread_name_length() {
   return get_max_thread_name_length_impl();
 }
 
 void llvm::set_thread_name(const Twine &Name) {
   // Make sure the input is null terminated.
   SmallString<64> Storage;
   StringRef NameStr = Name.toNullTerminatedStringRef(Storage);
 
   // Truncate from the beginning, not the end, if the specified name is too
   // long.  For one, this ensures that the resulting string is still null
   // terminated, but additionally the end of a long thread name will usually
   // be more unique than the beginning, since a common pattern is for similar

 #elif defined(__NetBSD__)
   ::pthread_setname_np(::pthread_self(), "%s",
     const_cast<char *>(NameStr.data()));
 #elif defined(__APPLE__)
   ::pthread_setname_np(NameStr.data());
 #endif
 }
 
 void llvm::get_thread_name(SmallVectorImpl<char> &Name) {
   Name.clear();
 
 #if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
   int pid = ::getpid();
   uint64_t tid = get_threadid();
 
   struct kinfo_proc *kp = nullptr, *nkp;
   size_t len = 0;
#if HAVE_PTHREAD_GETNAME_NP
   constexpr uint32_t len = get_max_thread_name_length_impl();
   char Buffer[len] = {'\0'};  // FIXME: working around MSan false positive.
   if (0 == ::pthread_getname_np(::pthread_self(), Buffer, len))
     Name.append(Buffer, Buffer + strlen(Buffer));
 #endif
 #endif
 }
 
 SetThreadPriorityResult llvm::set_thread_priority(ThreadPriority Priority) {
 #if defined(__linux__) && defined(SCHED_IDLE)
   // Some *really* old glibcs are missing SCHED_IDLE.
   // http://man7.org/linux/man-pages/man3/pthread_setschedparam.3.html
   // http://man7.org/linux/man-pages/man2/sched_setscheduler.2.html
   sched_param priority;
   // For each of the above policies, param->sched_priority must be 0.
   priority.sched_priority = 0;
   // SCHED_IDLE    for running very low priority background jobs.
   // SCHED_OTHER   the standard round-robin time-sharing policy;
   return !pthread_setschedparam(
              pthread_self(),
              Priority == ThreadPriority::Background ? SCHED_IDLE : SCHED_OTHER,
              &priority)
              ? SetThreadPriorityResult::SUCCESS
              : SetThreadPriorityResult::FAILURE;
 #elif defined(__APPLE__)
   // https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/getpriority.2.html
   // When setting a thread into background state the scheduling priority is set
   // to lowest value, disk and network IO are throttled. Network IO will be
   // throttled for any sockets the thread opens after going into background
   // state. Any previously opened sockets are not affected.
 
   // https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/getiopolicy_np.3.html
   // I/Os with THROTTLE policy are called THROTTLE I/Os. If a THROTTLE I/O
   // request occurs within a small time window (usually a fraction of a second)
   // of another NORMAL I/O request, the thread that issues the THROTTLE I/O is
   // forced to sleep for a certain interval. This slows down the thread that
   // issues the THROTTLE I/O so that NORMAL I/Os can utilize most of the disk
   // I/O bandwidth.
   return !setpriority(PRIO_DARWIN_THREAD, 0,
                       Priority == ThreadPriority::Background ? PRIO_DARWIN_BG
                                                              : 0)
              ? SetThreadPriorityResult::SUCCESS
              : SetThreadPriorityResult::FAILURE;
 #endif
   return SetThreadPriorityResult::FAILURE;
 }
 
 #include <thread>
 
 int computeHostNumHardwareThreads() {
 #if defined(__FreeBSD__)
   cpuset_t mask;
   CPU_ZERO(&mask);
   if (cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TID, -1, sizeof(mask),
                          &mask) == 0)
     return CPU_COUNT(&mask);
 #elif defined(__linux__)
   cpu_set_t Set;
   if (sched_getaffinity(0, sizeof(Set), &Set) == 0)
     return CPU_COUNT(&Set);
 #endif
   // Guard against std::thread::hardware_concurrency() returning 0.
   if (unsigned Val = std::thread::hardware_concurrency())
     return Val;
   return 1;
 }
 
 void llvm::ThreadPoolStrategy::apply_thread_strategy(
     unsigned ThreadPoolNum) const {}
 
 llvm::BitVector llvm::get_thread_affinity_mask() {
   // FIXME: Implement
   llvm_unreachable("Not implemented!");
 }
 
 unsigned llvm::get_cpus() { return 1; }

llvm/lib/Support/Windows/Threading.inc

 
 #include <bitset>
 
 // Windows will at times define MemoryFence.
 #ifdef MemoryFence
 #undef MemoryFence
 #endif
 
-static unsigned __stdcall threadFuncSync(void *Arg) {
-  SyncThreadInfo *TI = static_cast<SyncThreadInfo *>(Arg);
-  TI->UserFn(TI->UserData);
-  return 0;
-}
-
-static unsigned __stdcall threadFuncAsync(void *Arg) {
-  std::unique_ptr<AsyncThreadInfo> Info(static_cast<AsyncThreadInfo *>(Arg));
-  (*Info)();
-  return 0;
-}
-
-static void
-llvm_execute_on_thread_impl(unsigned (__stdcall *ThreadFunc)(void *), void *Arg,
-                            llvm::Optional<unsigned> StackSizeInBytes,
-                            JoiningPolicy JP) {
+HANDLE
+llvm::llvm_execute_on_thread_impl(unsigned(__stdcall *ThreadFunc)(void *),
+                                  void *Arg,
+                                  llvm::Optional<unsigned> StackSizeInBytes) {
   HANDLE hThread = (HANDLE)::_beginthreadex(
       NULL, StackSizeInBytes.getValueOr(0), ThreadFunc, Arg, 0, NULL);
 
   if (!hThread) {
     ReportLastErrorFatal("_beginthreadex failed");
   }
 
-  if (JP == JoiningPolicy::Join) {
-    if (::WaitForSingleObject(hThread, INFINITE) == WAIT_FAILED) {
-      ReportLastErrorFatal("WaitForSingleObject failed");
-    }
-  }
+  return hThread;
+}
+
+void llvm::llvm_thread_join_impl(HANDLE hThread) {
+
+void llvm::llvm_thread_detach_impl(HANDLE hThread) {
   if (::CloseHandle(hThread) == FALSE) {
     ReportLastErrorFatal("CloseHandle failed");
   }
 }
 
 uint64_t llvm::get_threadid() {
   return uint64_t(::GetCurrentThreadId());
 }

llvm/unittests/Support/Threading.cpp

TEST(Threading, RunOnThreadSyncAsync) {
   Notification ThreadStarted, ThreadAdvanced, ThreadFinished;
 
   auto ThreadFunc = [&] {
     ThreadStarted.notify();
     ASSERT_TRUE(ThreadAdvanced.wait());
     ThreadFinished.notify();
   };
 
-  llvm::llvm_execute_on_thread_async(ThreadFunc);
+  llvm::thread Thread(ThreadFunc);
+  Thread.detach();
   ASSERT_TRUE(ThreadStarted.wait());
   ThreadAdvanced.notify();
   ASSERT_TRUE(ThreadFinished.wait());
 }
 
 TEST(Threading, RunOnThreadSync) {
   std::atomic_bool Executed(false);
-  llvm::llvm_execute_on_thread(
+  llvm::thread Thread(
       [](void *Arg) { *static_cast<std::atomic_bool *>(Arg) = true; },
       &Executed);
+  Thread.join();
   ASSERT_EQ(Executed, true);
 }
+
+#if defined(__APPLE__)
+TEST(Threading, AppleStackSize) {
+  llvm::thread Thread([] {
+    volatile unsigned char Var[8 * 1024 * 1024 - 1024];
+    Var[0] = 0xff;
+    ASSERT_EQ(Var[0], 0xff);
+  });
+  Thread.join();
+}
+#endif
 #endif
 
 } // end anon namespace