[Support] Access threadIndex via a wrapper function

[mstorsjo]

Summary

On Unix platforms, this wrapper function is inline, so it should
expand to the same direct access to the thread local variable. On
Windows, it's a non-inline function within Parallel.cpp, allowing
making the thread_local variable static.

Windows Native TLS doesn't support direct access to thread local
variables in a different DLL, and GCC/binutils on Windows occasionally
has problems with non-static thread local variables too.

This fixes mingw dylib builds with native TLS.

[andrewng]

I think this looks fine to unblock the MinGW dylib build but I wonder if we could reduce the number of calls to getThreadIndex() by storing the index in the RelocationScanner? Although this would require quite a few changes to the existing interfaces.

[andrewng]

A couple of suggestions in response to the comment on D133003 regarding no TLS support.

llvm/include/llvm/Support/Parallel.h

31–40

Perhaps place this block within:

#if LLVM_ENABLE_THREADS
<HERE>
#else
inline unsigned getThreadIndex() { return 0; }
#endif

Which along with the other change, should fix any builds with LLVM_ENABLE_THREADS=0 that do not support TLS.

llvm/lib/Support/Parallel.cpp

21–27

Move this block down to below the #if LLVM_ENABLE_THREADS (line 31) and adjust for the namespaces.

[mstorsjo]

Move all use of the TLS variables into within #if LLVM_ENABLE_THREADS as suggested.

[andrewng]

LGTM, although might want to wait to see what @MaskRay thinks.

[MaskRay]

Interesting. I did not know this limitation.
lld is identical with the change on Linux x86-64.

[mstorsjo]

In D133759#3788555, @MaskRay wrote:

Interesting. I did not know this limitation.

It's of course a very subtle area; for "proper" DLL linkage, you'd use dllexport/dllimport, and if you combine dllexport/import with thread_local, then MSVC errors out with this error:

error C2492: 'tlsVar': data with thread storage duration may not have dll interface

Clang gives this error:

error: 'tlsVar' cannot be thread local when declared 'dllexport'

(And the same for dllimport too.) Clang does give these same errors also if using -femulated-tls, even if the linkage technically would work there. GCC doesn't error out on either of them (and the linkage does work out in the end, as it uses emulated TLS - but there's other potential pitfalls wrt TLS across object files there, due to their handling of weak linkage).

However, the mingw dylib build doesn't rely on explicit dllexport/imports, but on implicit autoimports, and then we don't get errors at that level at all, but only at linkage time:

ld.lld: error: unable to automatically import from _ZN4llvm8parallel11threadIndexE with relocation type IMAGE_REL_AMD64_SECREL in liblldELF.a(Relocations.cpp.obj)

lld/ELF/Relocations.cpp

static void addRelativeReloc(InputSectionBase &isec, uint64_t offsetInSec,
   // relocation offset is guaranteed to be even, add the relocation to
   // the relrDyn section, otherwise add it to the relaDyn section.
   // relrDyn sections don't support odd offsets. Also, relrDyn sections
   // don't store the addend values, so we must write it to the relocated
   // address.
   if (part.relrDyn && isec.alignment >= 2 && offsetInSec % 2 == 0) {
     isec.relocations.push_back({expr, type, offsetInSec, addend, &sym});
     if (shard)
-      part.relrDyn->relocsVec[parallel::threadIndex].push_back(
+      part.relrDyn->relocsVec[parallel::getThreadIndex()].push_back(
           {&isec, offsetInSec});
     else
       part.relrDyn->relocs.push_back({&isec, offsetInSec});
     return;
   }
   part.relaDyn->addRelativeReloc<shard>(target->relativeRel, isec, offsetInSec,
                                         sym, addend, type, expr);
 }
auto fn = [f]() {
       }
     };
     if (serial)
       fn();
     else
       tg.execute(fn);
   }
 
-  // Both the main thread and thread pool index 0 use threadIndex==0. Be
+  // Both the main thread and thread pool index 0 use getThreadIndex()==0. Be
   // careful that they don't concurrently run scanSections. When serial is
   // true, fn() has finished at this point, so running execute is safe.
   tg.execute([] {
     RelocationScanner scanner;
     for (Partition &part : partitions) {
       for (EhInputSection *sec : part.ehFrame->sections)
         scanner.template scanSection<ELFT>(*sec);
       if (part.armExidx && part.armExidx->isLive())

lld/ELF/SyntheticSections.h

protected:
   // will be moved into relocs by mergeRel().
   SmallVector<SmallVector<DynamicReloc, 0>, 0> relocsVec;
   size_t numRelativeRelocs = 0; // used by -z combreloc
   bool combreloc;
 };
 
 template <>
 inline void RelocationBaseSection::addReloc<true>(const DynamicReloc &reloc) {
-  relocsVec[llvm::parallel::threadIndex].push_back(reloc);
+  relocsVec[llvm::parallel::getThreadIndex()].push_back(reloc);
 }
 
 template <class ELFT>
 class RelocationSection final : public RelocationBaseSection {
   using Elf_Rel = typename ELFT::Rel;
   using Elf_Rela = typename ELFT::Rela;
 
 public:

llvm/include/llvm/Support/Parallel.h

 namespace llvm {
 
 namespace parallel {
 
 // Strategy for the default executor used by the parallel routines provided by
 // this file. It defaults to using all hardware threads and should be
 // initialized before the first use of parallel routines.
 extern ThreadPoolStrategy strategy;
+
+#if LLVM_ENABLE_THREADS
+#ifdef _WIN32
+// Direct access to thread_local variables from a different DLL isn't
+// possible with Windows Native TLS.
+unsigned getThreadIndex();
+#else
+// Don't access this directly, use the getThreadIndex wrapper.
 extern thread_local unsigned threadIndex;
 
+inline unsigned getThreadIndex() { return threadIndex; }
+#endif
+#else
+inline unsigned getThreadIndex() { return 0; }
+#endif
+
 namespace detail {
 class Latch {
   uint32_t Count;
   mutable std::mutex Mutex;
   mutable std::condition_variable Cond;
 
 public:
   explicit Latch(uint32_t Count = 0) : Count(Count) {}

llvm/lib/Support/Parallel.cpp

 
 #include <atomic>
 #include <future>
 #include <stack>
 #include <thread>
 #include <vector>
 
 llvm::ThreadPoolStrategy llvm::parallel::strategy;
-thread_local unsigned llvm::parallel::threadIndex;
 
 namespace llvm {
 namespace parallel {
 #if LLVM_ENABLE_THREADS
+
+#ifdef _WIN32
+static thread_local unsigned threadIndex;
+
+unsigned getThreadIndex() { return threadIndex; }
+#else
+thread_local unsigned threadIndex;
+#endif
+
 namespace detail {
 
 namespace {
 
 /// An abstract class that takes closures and runs them asynchronously.
 class Executor {
 public:
   virtual ~Executor() = default;