WIP: llvm-buildozer

[aganea]

Summary

This is a very early prototype for llvm-buildozer, a tool I've presented last year at the LLVM conference: https://www.youtube.com/watch?v=usPL_DROn4k

The overall goal is to switch the source code build model from a pool of processes, to a pool of threads.

Previously, a tool like ninja would both invoke the compiler and the linker .EXEs, based on a list of build actions, which are in turn built from a .ninja script containing a list of TUs and targets to compile. This model has the adverse effect of starting a new EXEs for each TU, having to call the initialization and compilation environnement detection, loading or mmap'ing of source files, etc. for each EXE instance. This comes with a high execution cost on Windows.

Our llvm-buildozer tool insteads takes a modified .CDB as an input, loads all necessary EXEs as DLLs once, and calls the build action internally, in a pool of threads, without dropping the EXEs, re-initializing the compiler or the heap. This evidently requires the compiler & link EXEs to be thread-safe, which represents the vast majority of our changes in this patch. With our llvm-buildozer tool, the rebuild times are decreased by 60%, tested on a 6-core, on a real-world game project: 20 min before -> 11 min 50 sec after.

As future work, if this idea passes RFC/community consensus, the goal after is to start working on inter-thread communication, such as sharing the state of the SourceManager or the tokenizer for a starter. Also a in-memory program repo, along the lines of SN-System's ideas could further avoid generating duplicate section/optimizations accross TUs.

This is an old prototype untouched since the LLVM conference, pasting this here only for reference purposes. I'm planning to get back to work on it at some point - or feel free in the meanwhile to pick anything interesting from this patch.

[tschuett]

When you extended the thread pool for high cores counts for Windows, I thought the other way around. I want to you use the distributed thinlto locally. Then each ninja job is single threaded and ninja can schedule everything. This probably won't be a desirable solution for Windows.

If you can get benefits out of inter-thread communication, then I am getting interested again.

[aganea]

In D86351#2230941, @tschuett wrote:

When you extended the thread pool for high cores counts for Windows, I thought the other way around. I want to you use the distributed thinlto locally. Then each ninja job is single threaded and ninja can schedule everything. This probably won't be a desirable solution for Windows.

This has been suggested by Bruce Dawson here: https://github.com/ninja-build/ninja/issues/1638 - although I haven't seen the prototype he mentions nor any figures. I feel that the direct and indirect costs for context switching between a pool of processes would be higher vs. a pool of threads in a single ongoing process, and that would increase with the number of cores. But I remain open to any idea, as long as it improves build times on Windows.

[aganea]

Rebase.

[arsenm]

llvm/lib/Analysis/MemorySSA.cpp

89 ↗

(On Diff #351663)

Do command line flags like this really need to be thread local? Wouldn't ever cl::opt need to be if this is?

[aganea]

llvm/lib/Analysis/MemorySSA.cpp

89 ↗

(On Diff #351663)

This was only for the PoC.

Values pointed by cl::opts -- that is either the internal cl::opt data or the static pointed by cl::location -- should be part of a context of some sort. LLD now has CommonLinkerContext plus a derived context per-driver. We probably need a "context" class along the same lines for the LLVM libs. Should it be part of LLVMContextImpl? Some options are read a bit earlier, before the LLVMContext is created.
The cl::opts themselves can remain static.

Unless we assume cl::opts are just for debugging, and move to tablegen any such globals that affect multi-threaded compilation?

CMakeLists.txt

 option(LLVM_ENABLE_PEDANTIC "Compile with pedantic enabled." ON)
 option(LLVM_ENABLE_WERROR "Fail and stop if a warning is triggered." OFF)
 
+if (MSVC)
+   option(LLVM_ENABLE_INCREMENTAL_LINK "Link incrementally. Enabling it might produce slower executables." OFF)
+endif()
+
 option(LLVM_ENABLE_DUMP "Enable dump functions even when assertions are disabled" OFF)
 
 if( NOT uppercase_CMAKE_BUILD_TYPE STREQUAL "DEBUG" )

cmake/modules/ChooseMSVCCRT.cmake

   set(${flagsvar} "${${flagsvar}}" CACHE STRING "${flagsvar_docs}" FORCE)
 endmacro(set_flag_in_var)
 
+macro(disable_MT_if_LLDB build message)
+  if (LLVM_TOOL_LLDB_BUILD)
+    if ((NOT ${build} STREQUAL "DEBUG") AND (LLVM_USE_CRT_${build} STREQUAL "MT"))
+      if (LLVM_TOOL_CLANG_BUILD OR LLVM_TOOL_LLD_BUILD)
+        set(performance " This might impact runtime performance for Clang or LLD. Preferably build them separately.")
+      endif()
+      message(WARNING "${message}.${performance}")
+      set(LLVM_USE_CRT_${build} "MD")
+    endif()
+  endif()
+endmacro(disable_MT_if_LLDB)
 
 macro(choose_msvc_crt MSVC_CRT)
   if(LLVM_USE_CRT)

       get_current_crt(LLVM_USE_CRT_${build}
         MSVC_CRT_REGEX
         CMAKE_CXX_FLAGS_${build})
+
+      # Make /MT the default in Release builds to make them faster
+      # and avoid the DLL function thunking.
+      if ((${build} STREQUAL "MINSIZEREL") OR
+          (${build} STREQUAL "RELEASE") OR
+          (${build} STREQUAL "RELWITHDEBINFO"))
+          set(LLVM_USE_CRT_${build} "MT")
+      endif()
+      if (${build} STREQUAL "DEBUG")
+          set(LLVM_USE_CRT_${build} "MTd")
+      endif()
+
+      disable_MT_if_LLDB(${build} "Using /MD as required by LLDB")
+
       set(LLVM_USE_CRT_${build}
         "${LLVM_USE_CRT_${build}}"
         CACHE STRING "Specify VC++ CRT to use for ${build_type} configurations."

         FORCE)
       set_property(CACHE LLVM_USE_CRT_${build}
         PROPERTY STRINGS ;${${MSVC_CRT}})
-    endif(NOT LLVM_USE_CRT_${build})
+    else()
+      disable_MT_if_LLDB(${build} "Disabling /MT as required by LLDB")
+    endif()
   endforeach(build_type)
 
   foreach(build_type ${CMAKE_CONFIGURATION_TYPES} ${CMAKE_BUILD_TYPE})

cmake/modules/HandleLLVMOptions.cmake

   # "Enforce type conversion rules".
   append("/Zc:rvalueCast" CMAKE_CXX_FLAGS)
 
+  if (CMAKE_CXX_COMPILER_ID MATCHES "MSVC" AND NOT LLVM_ENABLE_INCREMENTAL_LINK)
+    foreach(CONFIG RELEASE RELWITHDEBINFO MINSIZEREL)
+      foreach(FLAG EXE MODULE SHARED STATIC)
+        string(REGEX REPLACE "[-/](INCREMENTAL:YES|INCREMENTAL:NO|INCREMENTAL)" "/INCREMENTAL:NO" CMAKE_${FLAG}_LINKER_FLAGS_${CONFIG} "${CMAKE_${FLAG}_LINKER_FLAGS_${CONFIG}}")
+      endforeach()
+    endforeach()
+  endif()
+
   if (CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND NOT LLVM_ENABLE_LTO)
     # clang-cl and cl by default produce non-deterministic binaries because
     # link.exe /incremental requires a timestamp in the .obj file.  clang-cl

include/llvm/ADT/GlobalTypeDenseMap.h

+//===- llvm/ADT/GlobalTypeDenseMap.h - Dense probed hash table ------------*-
+// C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the GlobalTypeDenseMap class.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H
+#define LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H
+
+#include "llvm/DebugInfo/CodeView/TypeHashing.h"
+#include "llvm/Support/AlignOf.h"
+#include "llvm/Support/Compiler.h"
+#include "llvm/Support/MathExtras.h"
+#include "llvm/Support/Memory.h"
+#include "llvm/Support/ReverseIteration.h"
+#include "llvm/Support/type_traits.h"
+#include <algorithm>
+#include <cassert>
+#include <cstddef>
+#include <cstring>
+#include <initializer_list>
+#include <iterator>
+#include <new>
+#include <type_traits>
+#include <utility>
+
+namespace llvm {
+namespace codeview {
+
+struct GloballyHashedInfo {
+  static unsigned getHash(uint64_t Key, unsigned BucketMask) {
+    return (Key >> 32) & BucketMask;
+  }
+  void packKeyValue(uint64_t Key, uint32_t Value,
+                    unsigned BucketNo, unsigned BucketMask) {
+    KV = (Key & ~((uint64_t)BucketMask << 32)) | ((uint64_t)(Value & BucketMask) << 32);
+    //auto T = extractKeyValue(BucketNo, BucketMask); 
+    //assert((T == std::pair<uint64_t, uint32_t>(Key, Value)));
+  }
+  std::pair<uint64_t, uint32_t> extractKeyValue(unsigned BucketNo,
+                                                unsigned BucketMask) const {
+    uint64_t K =
+        (KV & ~((uint64_t)BucketMask << 32)) | ((uint64_t)BucketNo << 32);
+    uint32_t V = (KV >> 32) & BucketMask;
+    return {K, V};
+  }
+  bool isEmpty() const { return !KV; }
+  unsigned isEqualOrEmpty(const uint64_t Other, unsigned BucketNo,
+                          unsigned BucketMask) const {
+    if (!KV)
+      return 2; // empty
+    auto Extracted = extractKeyValue(BucketNo, BucketMask);
+    if (Extracted.first == Other)
+      return 1; // equal
+    return 0;
+  }
+  uint64_t KV;
+};
+
+template <typename KeyT = GloballyHashedType, typename ValueT = uint32_t,
+          typename BucketT = GloballyHashedInfo>
+class GlobalTypeDenseMap {
+public:
+  LLVM_NODISCARD bool empty() const { return getNumEntries() == 0; }
+  unsigned size() const { return getNumEntries(); }
+
+  uint64_t getBucketsMask() const { return getNumBuckets() - 1; }
+
+  struct iterator {
+    iterator(unsigned O, BucketT *B, unsigned M)
+        : Offset(O), Bucket(B), Mask(M) {}
+
+    unsigned Offset; /*hashed location (offset)*/
+    BucketT *Bucket; /*real key-value location (with collisions)*/
+    unsigned Mask;
+
+    ValueT getValue() {
+      auto KV = Bucket->extractKeyValue(Offset, Mask);
+      return KV.second;
+    }
+    void setValue(ValueT V) {
+      auto KV = Bucket->extractKeyValue(Offset, Mask);
+      Bucket->packKeyValue(KV.first, V, Offset, Mask);
+    }
+  };
+
+  // Inserts key,value pair into the map if the key isn't already in the map.
+  // The value is constructed in-place if the key is not in the map, otherwise
+  // it is not moved.
+  template <typename RehashFunc>
+  std::pair<iterator, bool>
+  try_emplace(const KeyT &Key, const uint32_t &Value,
+              RehashFunc Rehash) {
+    uint64_t InsertKey = *(uint64_t *)&Key;
+
+    auto R = LookupBucketFor(InsertKey);
+    if (std::get<2>(R) /*already there*/)
+      return {{std::get<0>(R), std::get<1>(R), (unsigned)getBucketsMask()},
+              false /*not inserted*/};
+
+    // Otherwise, insert the new element.
+    auto I = InsertIntoBucket({std::get<0>(R), std::get<1>(R)}, InsertKey, Value, Rehash);
+    return {{std::get<0>(I), std::get<1>(I), (unsigned)getBucketsMask()},
+            true /*inserted*/};
+  }
+
+protected:
+
+  /// Returns the number of buckets to allocate to ensure that the
+  /// GlobalTypeDenseMap can accommodate \p NumEntries without need to grow().
+  unsigned getMinBucketToReserveForEntries(unsigned NumEntries) {
+    // Ensure that "NumEntries * 4 < NumBuckets * 3"
+    if (NumEntries == 0)
+      return 0;
+    // +1 is required because of the strict equality.
+    // For example if NumEntries is 48, we need to return 401.
+    return NextPowerOf2(NumEntries * 4 / 3 + 1);
+  }
+
+private:
+
+  void incrementNumEntries() { setNumEntries(getNumEntries() + 1); }
+
+  void decrementNumEntries() { setNumEntries(getNumEntries() - 1); }
+
+  BucketT *getBucketsEnd() { return getBuckets() + getNumBuckets(); }
+
+  const BucketT *getBucketsEnd() const {
+    return getBuckets() + getNumBuckets();
+  }
+
+  template <typename RehashFunc>
+  std::pair<unsigned, BucketT *>
+  InsertIntoBucket(std::pair<unsigned, BucketT *> TheBucket,
+                   const uint64_t &Key, const uint32_t &Value,
+                   RehashFunc Rehash) {
+    // If the load of the hash table is more than 3/4, grow the table.
+    //
+    // The later case is tricky.  For example, if we had one empty bucket with
+    // tons of tombstones, failing lookups (e.g. for insertion) would have to
+    // probe almost the entire table until it found the empty bucket.  If the
+    // table completely filled with tombstones, no lookup would ever succeed,
+    // causing infinite loops in lookup.
+    unsigned NewNumEntries = getNumEntries() + 1;
+    unsigned NumBuckets = getNumBuckets();
+    if (LLVM_UNLIKELY(NewNumEntries * 4 >= NumBuckets * 3)) {
+      NumEntries = 0;
+      grow(NumBuckets * 2, Rehash);
+      auto R = LookupBucketFor(Key);
+      assert(!std::get<2>(R));
+      TheBucket = {std::get<0>(R), std::get<1>(R)};
+    }
+    assert(std::get<1>(TheBucket));
+
+    // Only update the state after we've grown our bucket space appropriately
+    // so that when growing buckets we have self-consistent entry count.
+    incrementNumEntries();
+
+    std::get<1>(TheBucket)->packKeyValue(Key, Value, std::get<0>(TheBucket),
+                                         getBucketsMask());
+    return TheBucket;
+  }
+
+  /// LookupBucketFor - Lookup the appropriate bucket for Val, returning it in
+  /// FoundBucket.  If the bucket contains the key and a value, this returns
+  /// true, otherwise it returns a bucket with an empty marker or tombstone and
+  /// returns false.
+  std::tuple<unsigned /*hashed location (offset)*/,
+             BucketT * /*real key-value location (with collisions)*/,
+             bool /*already there*/>
+  LookupBucketFor(const uint64_t &Key) {
+    if (getNumBuckets() == 0)
+      return {0, nullptr, false};
+
+    assert(Key && "Empty value shouldn't be inserted into map!");
+
+    BucketT *BucketsPtr = getBuckets();
+    const unsigned BucketsMask = getBucketsMask();
+    unsigned BucketNo = BucketT::getHash(Key, BucketsMask);
+    unsigned InitialBucketNo = BucketNo;
+    unsigned ProbeAmt = 1;
+    //BucketT *LastBucket = nullptr;
+    //unsigned CacheLines = 0;
+    HashTableStats::State S;
+    while (true) {
+      BucketT *ThisBucket = BucketsPtr + BucketNo;
+
+      /*if (ProbeAmt > MaxProbes)
+        MaxProbes = ProbeAmt;
+      if (ProbeAmt <= MAX_PROBES)
+        Probes[ProbeAmt - 1]++;
+      if (((size_t)ThisBucket & ~63) != ((size_t)LastBucket & ~63)) {
+        CacheLines++;
+        if (CacheLines > MaxCacheProbes)
+          MaxCacheProbes = CacheLines;
+      }
+      if (CacheLines <= MAX_CACHE_PROBES)
+        CacheProbes[CacheLines - 1]++;*/
+      Stats.updateOnAccess(S, ProbeAmt, ThisBucket);
+
+      auto R = ThisBucket->isEqualOrEmpty(Key, InitialBucketNo, BucketsMask);
+      if (LLVM_LIKELY(R)) {
+        return {InitialBucketNo, ThisBucket,
+                R == 1 ? true /*bucket equals provided Key*/
+                       : false /*bucket empty*/};
+      }
+
+      //LastBucket = ThisBucket;
+      // Otherwise, it's a hash collision, continue quadratic probing.
+      BucketNo += ProbeAmt++;
+      BucketNo &= BucketsMask;
+    }
+  }
+
+public:
+  /// Return the approximate size (in bytes) of the actual map.
+  /// This is just the raw memory used by GlobalTypeDenseMap.
+  /// If entries are pointers to objects, the size of the referenced objects
+  /// are not included.
+  size_t getMemorySize() const { return getNumBuckets() * sizeof(BucketT); }
+
+private:
+
+  struct HashTableStats {
+    struct State {
+      unsigned CacheLines{};
+      void *PrevBucket{};
+    };
+    void updateOnAccess(State &S, unsigned ProbeAmt, void *Bucket) {
+      if (ProbeAmt > MaxProbes)
+        MaxProbes = ProbeAmt;
+      if (ProbeAmt <= MAX_PROBES)
+        Probes[ProbeAmt - 1]++;
+      if (((size_t)Bucket & ~63) != ((size_t)S.PrevBucket & ~63)) {
+        S.CacheLines++;
+        if (S.CacheLines > MaxCacheProbes)
+          MaxCacheProbes = S.CacheLines;
+      }
+      if (S.CacheLines <= MAX_CACHE_PROBES)
+        CacheProbes[S.CacheLines - 1]++;
+      S.PrevBucket = Bucket;
+    }
+
+    unsigned MaxProbes = 0;
+    enum { MAX_PROBES = 256 };
+    unsigned Probes[MAX_PROBES]{};
+    unsigned MaxCacheProbes = 0;
+    enum { MAX_CACHE_PROBES = 128 };
+    unsigned CacheProbes[MAX_CACHE_PROBES]{};
+  };
+
+  BucketT *Buckets = nullptr;
+  unsigned NumEntries = 0;
+  unsigned NumBuckets = 0;
+  llvm::sys::MemoryBlock MB;
+  HashTableStats Stats;
+
+  /*unsigned MaxProbes = 0;
+  enum { MAX_PROBES = 256 };
+  unsigned Probes[MAX_PROBES]{};
+  unsigned MaxCacheProbes = 0;
+  enum { MAX_CACHE_PROBES = 128 };
+  unsigned CacheProbes[MAX_CACHE_PROBES]{};*/
+
+public:
+  GlobalTypeDenseMap() { init(0); }
+
+  ~GlobalTypeDenseMap() {
+    deleteBuckets(MB);
+    Buckets = nullptr;
+    NumBuckets = 0;
+    NumEntries = 0;
+  }
+
+  void init(unsigned InitNumEntries) {
+    auto InitBuckets = getMinBucketToReserveForEntries(InitNumEntries);
+    allocateBuckets(InitBuckets);
+  }
+
+  template <typename RehashFunc>
+  void grow(unsigned AtLeast, RehashFunc Rehash) {
+    BucketT *OldBuckets = Buckets;
+    unsigned NewNumBuckets = std::max<unsigned>(
+        64, static_cast<unsigned>(NextPowerOf2(AtLeast - 1)));
+    allocateBuckets(NewNumBuckets);
+    assert(Buckets);
+    if (OldBuckets)
+      Rehash();
+  }
+
+private:
+  unsigned getNumEntries() const { return NumEntries; }
+
+  void setNumEntries(unsigned Num) { NumEntries = Num; }
+
+  BucketT *getBuckets() const { return Buckets; }
+
+  unsigned getNumBuckets() const { return NumBuckets; }
+
+  bool allocateBuckets(unsigned Num) {
+    if (Num == 0)
+      return false;
+
+    size_t BlockSize = sizeof(BucketT) * Num;
+
+    if (MB.allocatedSize()) {
+      BucketT *MBEnd = (BucketT *)((uint8_t *)MB.base() + MB.allocatedSize());
+      if ((Buckets + NumBuckets + Num) <= MBEnd) {
+        Buckets += NumBuckets;
+        NumBuckets = Num;
+        return true;
+      }
+      if (BlockSize <= MB.allocatedSize()) {
+        ::memset(MB.base(), 0, BlockSize);
+        Buckets = (BucketT *)MB.base();
+        NumBuckets = Num;
+        return true;
+      }
+    }
+
+    deleteBuckets(MB);
+
+    const unsigned TwoMegs = 1 << 21;
+    const unsigned Nb2MegPages = (BlockSize + TwoMegs - 1) / TwoMegs;
+
+    const unsigned Flags =
+        sys::Memory::MF_READ | sys::Memory::MF_WRITE | sys::Memory::MF_HUGE_HINT;
+
+    std::error_code EC;
+    MB = llvm::sys::Memory::allocateMappedMemory(Nb2MegPages * TwoMegs, nullptr,
+                                                 Flags, EC);
+    Buckets = static_cast<BucketT *>(MB.base());
+    assert(Buckets);
+    NumBuckets = Num;
+    return true;
+  }
+  static void deleteBuckets(llvm::sys::MemoryBlock &MB) {
+    if (MB.allocatedSize()) {
+      llvm::sys::Memory::releaseMappedMemory(MB);
+    }
+  }
+};
+} // end namespace codeview
+} // end namespace llvm
+
+#endif // LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H

include/llvm/ADT/StringMap.h

 
 namespace llvm {
 
+struct HashTableStats {
+  struct State {
+    unsigned CacheLines{};
+    void* PrevBucket{};
+  };
+  void updateOnAccess(State& S, unsigned ProbeAmt, void* Bucket) {
+    if (ProbeAmt > MaxProbes)
+      MaxProbes = ProbeAmt;
+    if (ProbeAmt <= MAX_PROBES)
+      Probes[ProbeAmt - 1]++;
+    if (((size_t)Bucket & ~63) != ((size_t)S.PrevBucket & ~63)) {
+      S.CacheLines++;
+      if (S.CacheLines > MaxCacheProbes)
+        MaxCacheProbes = S.CacheLines;
+    }
+    if (S.CacheLines <= MAX_CACHE_PROBES)
+      CacheProbes[S.CacheLines - 1]++;
+    S.PrevBucket = Bucket;
+  }
+
+  unsigned MaxProbes = 0;
+  enum { MAX_PROBES = 256 };
+  unsigned Probes[MAX_PROBES]{};
+  unsigned MaxCacheProbes = 0;
+  enum { MAX_CACHE_PROBES = 128 };
+  unsigned CacheProbes[MAX_CACHE_PROBES]{};
+};
+
 template<typename ValueTy> class StringMapConstIterator;
 template<typename ValueTy> class StringMapIterator;
 template<typename ValueTy> class StringMapKeyIterator;

   unsigned NumItems = 0;
   unsigned NumTombstones = 0;
   unsigned ItemSize;
+  static HashTableStats Stats;
 
 protected:
   explicit StringMapImpl(unsigned itemSize)

include/llvm/ADT/iterator_range.h

 #ifndef LLVM_ADT_ITERATOR_RANGE_H
 #define LLVM_ADT_ITERATOR_RANGE_H
 
+#include <llvm/ADT/Optional.h>
 #include <iterator>
 #include <utility>
 

 ///
 /// This just wraps two iterators into a range-compatible interface. Nothing
 /// fancy at all.
-template <typename IteratorT>
-class iterator_range {
+template <typename IteratorT> class iterator_range {
   IteratorT begin_iterator, end_iterator;
+  size_t Length{};
+  bool LengthSet{false};
 
 public:
-  //TODO: Add SFINAE to test that the Container's iterators match the range's
+  // TODO: Add SFINAE to test that the Container's iterators match the range's
   //      iterators.
   template <typename Container>
   iterator_range(Container &&c)
-  //TODO: Consider ADL/non-member begin/end calls.
+      // TODO: Consider ADL/non-member begin/end calls.
       : begin_iterator(c.begin()), end_iterator(c.end()) {}
-  iterator_range(IteratorT begin_iterator, IteratorT end_iterator)
+  iterator_range(IteratorT begin_iterator, IteratorT end_iterator,
+                 Optional<size_t> RangeLength = None)
       : begin_iterator(std::move(begin_iterator)),
-        end_iterator(std::move(end_iterator)) {}
+        end_iterator(std::move(end_iterator)) {
+    if (RangeLength) {
+      Length = *RangeLength;
+      LengthSet = true;
+    }
+  }
 
   IteratorT begin() const { return begin_iterator; }
   IteratorT end() const { return end_iterator; }
+
+  size_t size() const {
+    assert(LengthSet && "Range length not set for this iterator! Use iteration "
+                        "instead to determine the size!");
+    return Length;
+  }
+  bool empty() const {
+    if (LengthSet)
+      return !Length;
+    return begin() == end();
+  }
 };
 
 /// Convenience function for iterating over sub-ranges.

 ///
 /// This provides a bit of syntactic sugar to make using sub-ranges
 /// in for loops a bit easier. Analogous to std::make_pair().
-template <class T> iterator_range<T> make_range(T x, T y) {
-  return iterator_range<T>(std::move(x), std::move(y));
+template <class T>
+iterator_range<T> make_range(T x, T y, Optional<size_t> RangeLength = None) {
+  return iterator_range<T>(std::move(x), std::move(y), RangeLength);
 }
 
-template <typename T> iterator_range<T> make_range(std::pair<T, T> p) {
-  return iterator_range<T>(std::move(p.first), std::move(p.second));
+template <typename T>
+iterator_range<T> make_range(std::pair<T, T> p,
+                             Optional<size_t> RangeLength = None) {
+  return iterator_range<T>(std::move(p.first), std::move(p.second),
+                           RangeLength);
 }
 
 template <typename T>

                                                                   int n) {
   return make_range(std::next(adl_begin(t), n), adl_end(t));
 }
-}
+} // namespace llvm
 
 #endif

include/llvm/Analysis/LoopAccessAnalysis.h

   static const unsigned MaxVectorWidth;
 
   /// VF as overridden by the user.
-  static unsigned VectorizationFactor;
+  static LLVM_THREAD_LOCAL unsigned VectorizationFactor;
   /// Interleave factor as overridden by the user.
-  static unsigned VectorizationInterleave;
+  static LLVM_THREAD_LOCAL unsigned VectorizationInterleave;
   /// True if force-vector-interleave was specified by the user.
   static bool isInterleaveForced();
 
   /// \When performing memory disambiguation checks at runtime do not
   /// make more than this number of comparisons.
-  static unsigned RuntimeMemoryCheckThreshold;
+  static LLVM_THREAD_LOCAL unsigned RuntimeMemoryCheckThreshold;
 };
 
 /// Checks memory dependences among accesses to the same underlying

include/llvm/Analysis/ObjCARCAnalysisUtils.h

 namespace objcarc {
 
 /// A handy option to enable/disable all ARC Optimizations.
-extern bool EnableARCOpts;
+extern LLVM_THREAD_LOCAL bool EnableARCOpts;
 
 /// Test if the given module looks interesting to run ARC optimization
 /// on.

include/llvm/Analysis/RegionInfo.h

   RegionInfoBase(const RegionInfoBase &) = delete;
   RegionInfoBase &operator=(const RegionInfoBase &) = delete;
 
-  static bool VerifyRegionInfo;
-  static typename RegionT::PrintStyle printStyle;
+  static LLVM_THREAD_LOCAL bool VerifyRegionInfo;
+  static LLVM_THREAD_LOCAL typename RegionT::PrintStyle printStyle;
 
   void print(raw_ostream &OS) const;
 #if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)

include/llvm/Analysis/RegionInfoImpl.h

 
 #ifdef EXPENSIVE_CHECKS
 template <class Tr>
-bool RegionInfoBase<Tr>::VerifyRegionInfo = true;
+LLVM_THREAD_LOCAL bool RegionInfoBase<Tr>::VerifyRegionInfo = true;
 #else
 template <class Tr>
-bool RegionInfoBase<Tr>::VerifyRegionInfo = false;
+LLVM_THREAD_LOCAL bool RegionInfoBase<Tr>::VerifyRegionInfo = false;
 #endif
 
 template <class Tr>
-typename Tr::RegionT::PrintStyle RegionInfoBase<Tr>::printStyle =
-    RegionBase<Tr>::PrintNone;
+LLVM_THREAD_LOCAL
+    typename Tr::RegionT::PrintStyle RegionInfoBase<Tr>::printStyle =
+        RegionBase<Tr>::PrintNone;
 
 template <class Tr>
 void RegionInfoBase<Tr>::print(raw_ostream &OS) const {

include/llvm/BinaryFormat/Magic.h

     wasm_object,         ///< WebAssembly Object file
     pdb,                 ///< Windows PDB debug info file
     tapi_file,           ///< Text-based Dynamic Library Stub file
+    clang_pch,           ///< Clang PCH precompiled headers file
   };
 
   bool is_object() const { return V != unknown; }

include/llvm/DebugInfo/CodeView/DebugHashesSection.h

+//===- DebugHashesSection.h -------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_CODEVIEW_DEBUGHASHESSECTION_H
+#define LLVM_DEBUGINFO_CODEVIEW_DEBUGHASHESSECTION_H
+
+namespace llvm {
+namespace codeview {
+
+} // namespace codeview
+} // namespace llvm
+
+#endif

include/llvm/IR/PassTimingInfo.h

 /// If the user specifies the -time-passes argument on an LLVM tool command line
 /// then the value of this boolean will be true, otherwise false.
 /// This is the storage for the -time-passes option.
-extern bool TimePassesIsEnabled;
+extern LLVM_THREAD_LOCAL bool TimePassesIsEnabled;
 
 /// This class implements -time-passes functionality for new pass manager.
 /// It provides the pass-instrumentation callbacks that measure the pass

include/llvm/Pass.h

 /// If the user specifies the -time-passes argument on an LLVM tool command line
 /// then the value of this boolean will be true, otherwise false.
 /// This is the storage for the -time-passes option.
-extern bool TimePassesIsEnabled;
+extern LLVM_THREAD_LOCAL bool TimePassesIsEnabled;
 
 } // end namespace llvm
 

include/llvm/Support/CommandLine.h

 #define LLVM_SUPPORT_COMMANDLINE_H
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallPtrSet.h"
 #include "llvm/ADT/SmallVector.h"

 #include "llvm/ADT/iterator_range.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <climits>

 };
 
 // A special subcommand representing no subcommand
-extern ManagedStatic<SubCommand> TopLevelSubCommand;
+//extern ManagedStatic<SubCommand> TopLevelSubCommand;
 
 // A special subcommand that can be used to put an option into all subcommands.
-extern ManagedStatic<SubCommand> AllSubCommands;
+//extern ManagedStatic<SubCommand> AllSubCommands;
 
 //===----------------------------------------------------------------------===//
 // Option Base class

     return getNumOccurrencesFlag() == cl::ConsumeAfter;
   }
 
-  bool isInAllSubCommands() const {
-    return any_of(Subs, [](const SubCommand *SC) {
-      return SC == &*AllSubCommands;
-    });
-  }
+  bool isInAllSubCommands() const;
 
   //-------------------------------------------------------------------------===
   // Accessor functions set by OptionModifiers

 
   inline int getNumOccurrences() const { return NumOccurrences; }
   void reset();
+
+  virtual unsigned getSize() const = 0;
+  virtual void duplicateInto(Option *O) = 0;
+  virtual void postRemap(DenseMap<Option *, Option *> &RemapOpt) {}
 };
 
 //===----------------------------------------------------------------------===//

 
   LocationClass(Ty &L) : Loc(L) {}
 
-  template <class Opt> void apply(Opt &O) const { O.setLocation(O, Loc); }
+  template <class Opt> void apply(Opt &O) const {
+    size_t TLsOffset = sys::Process::GetOffsetInTLS(&Loc);
+    O.setLocation(O, Loc, TLsOffset);
+  }
 };
 
 template <class Ty> LocationClass<Ty> location(Ty &L) {

   return LocationClass<Ty>(L);
 }
 
-// cat - Specifiy the Option category for the command line argument to belong
+// cat - Specify the Option category for the command line argument to belong
 // to.
 struct cat {
   OptionCategory &Category;

   template <class Opt> void apply(Opt &O) const { O.addSubCommand(Sub); }
 };
 
+struct allSub : public sub {
+  allSub();
+};
+
+struct topLevel : public sub {
+  topLevel();
+};
+
 //===----------------------------------------------------------------------===//
 // OptionValue class
 

 class opt_storage {
   DataType *Location = nullptr; // Where to store the object...
   OptionValue<DataType> Default;
+  unsigned TLSOffset{};
 
   void check_location() const {
     assert(Location && "cl::location(...) not specified for a command "

                        "or cl::init specified before cl::location()!!");
   }
 
+protected:
+  void remapLocationToTLS() {
+    Location = (DataType *)sys::Process::RemapIntoTLS(Location, TLSOffset);
+  }
+
 public:
   opt_storage() = default;
 
-  bool setLocation(Option &O, DataType &L) {
+  bool setLocation(Option &O, DataType &L, size_t Off) {
     if (Location)
       return O.error("cl::location(x) specified more than once!");
     Location = &L;
     Default = L;
+    TLSOffset = Off;
     return false;
   }
 

   const DataType &getValue() const { return *this; }
 
   const OptionValue<DataType> &getDefault() const { return Default; }
+
+protected:
+  void remapLocationToTLS() {}
 };
 
 // Define a partial specialization to handle things we cannot inherit from.  In

 
   // If the datatype is a pointer, support -> on it.
   DataType operator->() const { return Value; }
+
+protected:
+  void remapLocationToTLS() {}
 };
 
 //===----------------------------------------------------------------------===//

                                std::is_class<DataType>::value> {
   ParserClass Parser;
 
+  unsigned getSize() const override { return sizeof(*this); }
+
+  void duplicateInto(Option *O) override {
+    new ((void *)O) opt<DataType, ExternalStorage, ParserClass>(*this);
+
+    if (ExternalStorage)
+      ((opt<DataType, ExternalStorage, ParserClass> *)O)->remapLocationToTLS();
+  }
+
   bool handleOccurrence(unsigned pos, StringRef ArgName,
                         StringRef Arg) override {
     typename ParserClass::parser_data_type Val =

 
 public:
   // Command line options should not be copyable
-  opt(const opt &) = delete;
-  opt &operator=(const opt &) = delete;
+  //opt(const opt &) = delete;
+  //opt &operator=(const opt &) = delete;
 
   // setInitialValue - Used by the cl::init modifier...
   void setInitialValue(const DataType &V) { this->setValue(V, true); }

 template <class DataType, class StorageClass> class list_storage {
   StorageClass *Location = nullptr; // Where to store the object...
 
+protected:
+  void remapLocationToTLS() {}
+
 public:
   list_storage() = default;
 
   void clear() {}
 
-  bool setLocation(Option &O, StorageClass &L) {
+  bool setLocation(Option &O, StorageClass &L, size_t Off) {
     if (Location)
       return O.error("cl::location(x) specified more than once!");
     Location = &L;

                             "line option with external storage!");
     Location->push_back(V);
   }
+
+  using iterator = typename StorageClass::iterator;
+
+  iterator begin() const { return Location->begin(); }
+  iterator end() const { return Location->end(); }
 };
 
 // Define how to hold a class type object, such as a string.

 template <class DataType> class list_storage<DataType, bool> {
   std::vector<DataType> Storage;
 
+protected:
+  void remapLocationToTLS() {}
+
 public:
   using iterator = typename std::vector<DataType>::iterator;
 

   reference front() { return Storage.front(); }
   const_reference front() const { return Storage.front(); }
 
+  const DataType* data() const { return Storage.data(); }
+
   operator std::vector<DataType>&() { return Storage; }
   operator ArrayRef<DataType>() { return Storage; }
   std::vector<DataType> *operator&() { return &Storage; }

   std::vector<unsigned> Positions;
   ParserClass Parser;
 
+  unsigned getSize() const override { return sizeof(*this); }
+
+  void duplicateInto(Option *O) override {
+    new ((void *)O) list<DataType, StorageClass, ParserClass>(*this);
+
+    if (!std::is_base_of<StorageClass, bool>::value) {
+      ((list<DataType, StorageClass, ParserClass> *)O)->remapLocationToTLS();
+      /*for (const DataType &D : *this) {
+        llvm::sys::Process::RemapIntoTLS((void**)&D);
+      }*/
+    }
+  }
+
   enum ValueExpected getValueExpectedFlagDefault() const override {
     return Parser.getValueExpectedFlagDefault();
   }

 
 public:
   // Command line options should not be copyable
-  list(const list &) = delete;
-  list &operator=(const list &) = delete;
+  //list(const list &) = delete;
+  //list &operator=(const list &) = delete;
 
   ParserClass &getParser() { return Parser; }
 

 //
 template <class DataType, class StorageClass> class bits_storage {
   unsigned *Location = nullptr; // Where to store the bits...
+  size_t TLSOffset{};
 
   template <class T> static unsigned Bit(const T &V) {
     unsigned BitPos = reinterpret_cast<unsigned>(V);

 public:
   bits_storage() = default;
 
-  bool setLocation(Option &O, unsigned &L) {
+  bool setLocation(Option &O, unsigned &L, size_t Off) {
     if (Location)
       return O.error("cl::location(x) specified more than once!");
     Location = &L;
+    TLSOffset = Off;
     return false;
   }
 

     *Location |= Bit(V);
   }
 
-  unsigned getBits() { return *Location; }
+  unsigned getBits() const { return *Location; }
 
+  unsigned *getBitsPtr() const { return Location; }
+
   template <class T> bool isSet(const T &V) {
     return (*Location & Bit(V)) != 0;
   }

 public:
   template <class T> void addValue(const T &V) { Bits |= Bit(V); }
 
-  unsigned getBits() { return Bits; }
+  unsigned getBits() const { return Bits; }
 
+  unsigned *getBitsPtr() const { return &Bits; }
+
   template <class T> bool isSet(const T &V) { return (Bits & Bit(V)) != 0; }
 };
 

   std::vector<unsigned> Positions;
   ParserClass Parser;
 
+  unsigned getSize() const override { return sizeof(*this); }
+
+  void duplicateInto(Option *O) override {
+    new ((void *)O) bits<DataType, Storage, ParserClass>(*this);
+  }
+
   enum ValueExpected getValueExpectedFlagDefault() const override {
     return Parser.getValueExpectedFlagDefault();
   }

 class alias : public Option {
   Option *AliasFor;
 
+  unsigned getSize() const override { return sizeof(*this); }
+
+  void duplicateInto(Option *O) override { new ((void *)O) alias(*this); }
+
+  void postRemap(DenseMap<Option *, Option *> &RemapOpt) override {
+    AliasFor = RemapOpt[AliasFor];
+  }
+
   bool handleOccurrence(unsigned pos, StringRef /*ArgName*/,
                         StringRef Arg) override {
     return AliasFor->handleOccurrence(pos, AliasFor->ArgStr, Arg);

 
 public:
   // Command line options should not be copyable
-  alias(const alias &) = delete;
-  alias &operator=(const alias &) = delete;
+  //alias(const alias &) = delete;
+  //alias &operator=(const alias &) = delete;
 
   void setAliasFor(Option &O) {
     if (AliasFor)

 /// Hopefully this API can be deprecated soon. Any situation where options need
 /// to be modified by tools or libraries should be handled by sane APIs rather
 /// than just handing around a global list.
-StringMap<Option *> &getRegisteredOptions(SubCommand &Sub = *TopLevelSubCommand);
+StringMap<Option *> &getRegisteredOptions();
 
 /// Use this to get all registered SubCommands from the provided parser.
 ///

 iterator_range<typename SmallPtrSet<SubCommand *, 4>::iterator>
 getRegisteredSubcommands();
 
+bool isTopLevel(SubCommand *SC);
+
 //===----------------------------------------------------------------------===//
 // Standalone command line processing utilities.
 //

 /// Some tools (like clang-format) like to be able to hide all options that are
 /// not specific to the tool. This function allows a tool to specify a single
 /// option category to display in the -help output.
-void HideUnrelatedOptions(cl::OptionCategory &Category,
-                          SubCommand &Sub = *TopLevelSubCommand);
+void HideUnrelatedOptions(cl::OptionCategory &Category);
 
 /// Mark all options not part of the categories as cl::ReallyHidden.
 ///

 /// Some tools (like clang-format) like to be able to hide all options that are
 /// not specific to the tool. This function allows a tool to specify a single
 /// option category to display in the -help output.
-void HideUnrelatedOptions(ArrayRef<const cl::OptionCategory *> Categories,
-                          SubCommand &Sub = *TopLevelSubCommand);
+void HideUnrelatedOptions(ArrayRef<const cl::OptionCategory *> Categories);
 
 /// Reset all command line options to a state that looks as if they have
 /// never appeared on the command line.  This is useful for being able to parse

include/llvm/Support/CrashRecoveryContext.h

   /// Explicitly trigger a crash recovery in the current process, and
   /// return failure from RunSafely(). This function does not return.
   void HandleCrash();
+
+  /// In case of a crash, this is the crash identifier
+  int RetCode{};
+
+  /// Selects whether the global exception handler should be called. When this
+  /// is active, a crash has the same side-effect as uncatched code (callstack
+  /// print and minidump), except that here we recover the execution flow after
+  /// the call to RunSafely().
+  bool EnableExceptionHandler{};
 };
 
 /// Abstract base class of cleanup handlers.

include/llvm/Support/Debug.h

 //
 //===----------------------------------------------------------------------===//
 
+#include "llvm/Support/Compiler.h"
+
 #ifndef LLVM_SUPPORT_DEBUG_H
 #define LLVM_SUPPORT_DEBUG_H
 

 /// is specified.  This should probably not be referenced directly, instead, use
 /// the DEBUG macro below.
 ///
-extern bool DebugFlag;
+extern LLVM_THREAD_LOCAL bool DebugFlag;
 
 /// \name Verification flags.
 ///

 
 /// Enables verification of dominator trees.
 ///
-extern bool VerifyDomInfo;
+extern LLVM_THREAD_LOCAL bool VerifyDomInfo;
 
 /// Enables verification of loop info.
 ///
-extern bool VerifyLoopInfo;
+extern LLVM_THREAD_LOCAL bool VerifyLoopInfo;
 
 /// Enables verification of MemorySSA.
 ///
-extern bool VerifyMemorySSA;
+extern LLVM_THREAD_LOCAL bool VerifyMemorySSA;
 
 ///\}
 

include/llvm/Support/DebugCounter.h

   /// Returns a reference to the singleton instance.
   static DebugCounter &instance();
 
-  // Used by the command line option parser to push a new value it parsed.
-  void push_back(const std::string &);
-
   // Register a counter with the specified name.
   //
   // FIXME: Currently, counter registration is required to happen before command

 
   // Iterate through the registered counters
   typedef UniqueVector<std::string> CounterVector;
-  CounterVector::const_iterator begin() const {
-    return RegisteredCounters.begin();
-  }
-  CounterVector::const_iterator end() const { return RegisteredCounters.end(); }
+  const CounterVector &getCounterVector() const { return RegisteredCounters; }
 
   // Force-enables counting all DebugCounters.
   //

   // Whether we should do DebugCounting at all. DebugCounters aren't
   // thread-safe, so this should always be false in multithreaded scenarios.
   bool Enabled = false;
+
+public:
+  // Used by the command line option parser to push a new value it parsed.
+  void push_back(const std::string &);
+
+  struct iterator {
+    DebugCounter *This;
+    DenseMap<unsigned, CounterInfo>::iterator It;
+
+    bool operator!=(const iterator &RHS) const { return It != RHS.It; }
+
+    iterator &operator++() /*preincrement*/ {
+      ++It;
+      return *this;
+    }
+
+    const std::string &operator*() const {
+      return This->RegisteredCounters[It->first];
+    }
+  };
+  iterator begin() { return {this, Counters.begin()}; }
+  iterator end() { return { this, Counters.end() }; }
 };
 
 #define DEBUG_COUNTER(VARNAME, COUNTERNAME, DESC)                              \

include/llvm/Support/Error.h

   }
 
 private:
-  void checkError(Error Err) const {
-    if (Err) {
-      int ExitCode = GetExitCode(Err);
-      logAllUnhandledErrors(std::move(Err), errs(), Banner);
-      exit(ExitCode);
-    }
-  }
+  void checkError(Error Err) const;
 
   std::string Banner;
   std::function<int(const Error &)> GetExitCode;

include/llvm/Support/InitLLVM.h

 
   ~InitLLVM();
 
+  // This is necessary to properly initialize the CRT for Clang/LLVM EXE loaded
+  // as DLLs inside Buildozer.
+  static void InitFromBuildozer(void *InitEvent, void *ExitEvent);
+
+  static void SetCurrentModuleBackdoor(void *ModuleHandle);
+
+  // Don't register crash handlers, it is assumed someone else does it.
+  static void SkipInitLLVM();
+
+  static bool RunningInBuildozer();
+
 private:
   BumpPtrAllocator Alloc;
   SmallVector<const char *, 0> Args;

include/llvm/Support/Process.h

   /// Get the result of a process wide random number generator. The
   /// generator will be automatically seeded in non-deterministic fashion.
   static unsigned GetRandomNumber();
+
+  [[noreturn]]
+  static void ExitCurrentProcess(int code);
+
+  static bool IsInTLS(const void* Data);
+
+  static unsigned GetOffsetInTLS(const void *Data);
+
+  static void *RemapIntoTLS(void *Data, unsigned TLSOffset);
 };
-
 }
 }
 

lib/Analysis/LoopAccessAnalysis.cpp

 VectorizationFactor("force-vector-width", cl::Hidden,
                     cl::desc("Sets the SIMD width. Zero is autoselect."),
                     cl::location(VectorizerParams::VectorizationFactor));
-unsigned VectorizerParams::VectorizationFactor;
+LLVM_THREAD_LOCAL unsigned VectorizerParams::VectorizationFactor;
 
 static cl::opt<unsigned, true>
 VectorizationInterleave("force-vector-interleave", cl::Hidden,

                                  "Zero is autoselect."),
                         cl::location(
                             VectorizerParams::VectorizationInterleave));
-unsigned VectorizerParams::VectorizationInterleave;
+LLVM_THREAD_LOCAL unsigned VectorizerParams::VectorizationInterleave;
 
 static cl::opt<unsigned, true> RuntimeMemoryCheckThreshold(
     "runtime-memory-check-threshold", cl::Hidden,

     cl::desc("When performing memory disambiguation checks at runtime do not "
              "generate more than this number of comparisons (default = 8)."),
     cl::location(VectorizerParams::RuntimeMemoryCheckThreshold), cl::init(8));
-unsigned VectorizerParams::RuntimeMemoryCheckThreshold;
+LLVM_THREAD_LOCAL unsigned VectorizerParams::RuntimeMemoryCheckThreshold;
 
 /// The maximum iterations used to merge memory checks
 static cl::opt<unsigned> MemoryCheckMergeThreshold(

lib/Analysis/LoopInfo.cpp

 
 // Always verify loopinfo if expensive checking is enabled.
 #ifdef EXPENSIVE_CHECKS
-bool llvm::VerifyLoopInfo = true;
+LLVM_THREAD_LOCAL bool llvm::VerifyLoopInfo = true;
 #else
-bool llvm::VerifyLoopInfo = false;
+LLVM_THREAD_LOCAL bool llvm::VerifyLoopInfo = false;
 #endif
 static cl::opt<bool, true>
     VerifyLoopInfoX("verify-loop-info", cl::location(VerifyLoopInfo),

lib/Analysis/MemorySSA.cpp

 
 // Always verify MemorySSA if expensive checking is enabled.
 #ifdef EXPENSIVE_CHECKS
-bool llvm::VerifyMemorySSA = true;
+LLVM_THREAD_LOCAL bool llvm::VerifyMemorySSA = true;
 #else
-bool llvm::VerifyMemorySSA = false;
+LLVM_THREAD_LOCAL bool llvm::VerifyMemorySSA = false;
 #endif
 /// Enables memory ssa as a dependency for loop passes in legacy pass manager.
 cl::opt<bool> llvm::EnableMSSALoopDependency(

lib/Analysis/ModuleSummaryAnalysis.cpp

 
 // Option to force edges cold which will block importing when the
 // -import-cold-multiplier is set to 0. Useful for debugging.
-FunctionSummary::ForceSummaryHotnessType ForceSummaryEdgesCold =
+LLVM_THREAD_LOCAL FunctionSummary::ForceSummaryHotnessType ForceSummaryEdgesCold =
     FunctionSummary::FSHT_None;
 cl::opt<FunctionSummary::ForceSummaryHotnessType, true> FSEC(
     "force-summary-edges-cold", cl::Hidden, cl::location(ForceSummaryEdgesCold),

lib/Analysis/ObjCARCAnalysisUtils.cpp

 using namespace llvm::objcarc;
 
 /// A handy option to enable/disable all ARC Optimizations.
-bool llvm::objcarc::EnableARCOpts;
+LLVM_THREAD_LOCAL bool llvm::objcarc::EnableARCOpts;
 static cl::opt<bool, true> EnableARCOptimizations(
     "enable-objc-arc-opts", cl::desc("enable/disable all ARC Optimizations"),
     cl::location(EnableARCOpts), cl::init(true), cl::Hidden);

lib/BinaryFormat/Magic.cpp

       return file_magic::tapi_file;
     break;
 
+  case 'C':
+    if (startswith(Magic, "CPCH"))
+      return file_magic::clang_pch;
+    break;
+
   default:
     break;
   }

lib/CodeGen/AsmPrinter/DwarfDebug.cpp

     // If we're splitting the dwarf out now that we've got the entire
     // CU then add the dwo id to it.
     auto *SkCU = TheCU.getSkeleton();
-    if (useSplitDwarf() && !empty(TheCU.getUnitDie().children())) {
+    if (useSplitDwarf() && !TheCU.getUnitDie().children().empty()) {
       finishUnitAttributes(TheCU.getCUNode(), TheCU);
       TheCU.addString(TheCU.getUnitDie(), dwarf::DW_AT_GNU_dwo_name,
                       Asm->TM.Options.MCOptions.SplitDwarfFile);

     // is a bit pessimistic under LTO.
     if (!AddrPool.isEmpty() &&
         (getDwarfVersion() >= 5 ||
-         (SkCU && !empty(TheCU.getUnitDie().children()))))
+         (SkCU && !TheCU.getUnitDie().children().empty())))
       U.addAddrTableBase();
 
     if (getDwarfVersion() >= 5) {

lib/CodeGen/MachineDominators.cpp

 
 // Always verify dominfo if expensive checking is enabled.
 #ifdef EXPENSIVE_CHECKS
-static bool VerifyMachineDomInfo = true;
+static LLVM_THREAD_LOCAL bool VerifyMachineDomInfo = true;
 #else
-static bool VerifyMachineDomInfo = false;
+static LLVM_THREAD_LOCAL bool VerifyMachineDomInfo = false;
 #endif
 static cl::opt<bool, true> VerifyMachineDomInfoX(
     "verify-machine-dom-info", cl::location(VerifyMachineDomInfo), cl::Hidden,

lib/CodeGen/RegAllocBase.h

 
 public:
   /// VerifyEnabled - True when -verify-regalloc is given.
-  static bool VerifyEnabled;
+  static LLVM_THREAD_LOCAL bool VerifyEnabled;
 
 private:
   void seedLiveRegs();

lib/CodeGen/RegAllocBase.cpp

 
 const char RegAllocBase::TimerGroupName[] = "regalloc";
 const char RegAllocBase::TimerGroupDescription[] = "Register Allocation";
-bool RegAllocBase::VerifyEnabled = false;
+LLVM_THREAD_LOCAL bool RegAllocBase::VerifyEnabled = false;
 
 //===----------------------------------------------------------------------===//
 //                         RegAllocBase Implementation

lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp

 
 /// LimitFloatPrecision - Generate low-precision inline sequences for
 /// some float libcalls (6, 8 or 12 bits).
-static unsigned LimitFloatPrecision;
+static LLVM_THREAD_LOCAL unsigned LimitFloatPrecision;
 
 static cl::opt<unsigned, true>
     LimitFPPrecision("limit-float-precision",

lib/DebugInfo/DWARF/DWARFVerifier.cpp

 
   if (Pos != End) {
     if (Pos->intersects(R))
-      return Pos;
+      return std::move(Pos);
     if (Pos != Begin) {
       auto Iter = Pos - 1;
       if (Iter->intersects(R))
-        return Iter;
+        return std::move(Iter);
     }
   }
 

lib/ExecutionEngine/Interpreter/Execution.cpp

 #include "llvm/Support/Debug.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MathExtras.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/raw_ostream.h"
 #include <algorithm>
 #include <cmath>

   // the stack before interpreting atexit handlers.
   ECStack.clear();
   runAtExitHandlers();
-  exit(GV.IntVal.zextOrTrunc(32).getZExtValue());
+  sys::Process::ExitCurrentProcess(GV.IntVal.zextOrTrunc(32).getZExtValue());
 }
 
 /// Pop the last stack frame off of ECStack and then copy the result

lib/ExecutionEngine/TargetSelect.cpp

   if (TheTriple.getTriple().empty())
     TheTriple.setTriple(sys::getProcessTriple());
 
-  // Adjust the triple to match what the user requested.
-  const Target *TheTarget = nullptr;
-  if (!MArch.empty()) {
-    auto I = find_if(TargetRegistry::targets(),
-                     [&](const Target &T) { return MArch == T.getName(); });
+  std::string Error;
+  const Target *TheTarget =
+      TargetRegistry::lookupTarget(MArch, TheTriple, Error);
+  if (!MArch.empty() && !TheTarget)
+    Error = "No available targets are compatible with -march=" + MArch.str() +
+            ", "
+            "see -version for the available targets.\n";
+  if (ErrorStr)
+    *ErrorStr = Error;
+  if (!TheTarget)
+    return nullptr;
 
-    if (I == TargetRegistry::targets().end()) {
-      if (ErrorStr)
-        *ErrorStr = "No available targets are compatible with this -march, "
-                    "see -version for the available targets.\n";
-      return nullptr;
-    }
-
-    TheTarget = &*I;
-
-    // Adjust the triple to match (if known), otherwise stick with the
-    // requested/host triple.
-    Triple::ArchType Type = Triple::getArchTypeForLLVMName(MArch);
-    if (Type != Triple::UnknownArch)
-      TheTriple.setArch(Type);
-  } else {
-    std::string Error;
-    TheTarget = TargetRegistry::lookupTarget(TheTriple.getTriple(), Error);
-    if (!TheTarget) {
-      if (ErrorStr)
-        *ErrorStr = Error;
-      return nullptr;
-    }
-  }
-
   // Package up features to be passed to target/subtarget
   std::string FeaturesStr;
   if (!MAttrs.empty()) {

lib/IR/DiagnosticHandler.cpp

   }
 };
 
-static PassRemarksOpt PassRemarksPassedOptLoc;
-static PassRemarksOpt PassRemarksMissedOptLoc;
-static PassRemarksOpt PassRemarksAnalysisOptLoc;
+static LLVM_THREAD_LOCAL PassRemarksOpt PassRemarksPassedOptLoc;
+static LLVM_THREAD_LOCAL PassRemarksOpt PassRemarksMissedOptLoc;
+static LLVM_THREAD_LOCAL PassRemarksOpt PassRemarksAnalysisOptLoc;
 
 // -pass-remarks
 //    Command line flag to enable emitOptimizationRemark()

lib/IR/Dominators.cpp

 #include <algorithm>
 using namespace llvm;
 
-bool llvm::VerifyDomInfo = false;
+static LLVM_THREAD_LOCAL bool llvm::VerifyDomInfo = false;
 static cl::opt<bool, true>
     VerifyDomInfoX("verify-dom-info", cl::location(VerifyDomInfo), cl::Hidden,
                    cl::desc("Verify dominator info (time consuming)"));

lib/IR/LLVMContext.cpp

 #include "llvm/IR/RemarkStreamer.h"
 #include "llvm/Support/Casting.h"
 #include "llvm/Support/ErrorHandling.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <cstdlib>

   DI.print(DP);
   errs() << "\n";
   if (DI.getSeverity() == DS_Error)
-    exit(1);
+    sys::Process::ExitCurrentProcess(1);
 }
 
 void LLVMContext::emitError(unsigned LocCookie, const Twine &ErrorStr) {

lib/IR/PassTimingInfo.cpp

 
 namespace llvm {
 
-bool TimePassesIsEnabled = false;
+LLVM_THREAD_LOCAL bool TimePassesIsEnabled = false;
 
 static cl::opt<bool, true> EnableTiming(
     "time-passes", cl::location(TimePassesIsEnabled), cl::Hidden,

lib/IRReader/IRReader.cpp

 using namespace llvm;
 
 namespace llvm {
-  extern bool TimePassesIsEnabled;
+  extern LLVM_THREAD_LOCAL bool TimePassesIsEnabled;
 }
 
 static const char *const TimeIRParsingGroupName = "irparse";

lib/LTO/LTOBackend.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Program.h"
 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/Support/ThreadPool.h"

 LLVM_ATTRIBUTE_NORETURN static void reportOpenError(StringRef Path, Twine Msg) {
   errs() << "failed to open " << Path << ": " << Msg << '\n';
   errs().flush();
-  exit(1);
+  sys::Process::ExitCurrentProcess(1);
 }
 
 Error Config::addSaveTemps(std::string OutputFileName,

lib/MC/MCContext.cpp

 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/SourceMgr.h"
 #include "llvm/Support/raw_ostream.h"

   // to make sure any special cleanups get done, in particular that we remove
   // files registered with RemoveFileOnSignal.
   sys::RunInterruptHandlers();
-  exit(1);
+  sys::Process::ExitCurrentProcess(1);
 }

lib/Object/Binary.cpp

     return errorCodeToError(object_error::invalid_file_type);
   case file_magic::unknown:
   case file_magic::coff_cl_gl_object:
+  case file_magic::clang_pch:
     // Unrecognized object file format.
     return errorCodeToError(object_error::invalid_file_type);
   case file_magic::minidump:

lib/Object/ObjectFile.cpp

   case file_magic::windows_resource:
   case file_magic::pdb:
   case file_magic::minidump:
-    return errorCodeToError(object_error::invalid_file_type);
+  case file_magic::clang_pch:
   case file_magic::tapi_file:
     return errorCodeToError(object_error::invalid_file_type);
   case file_magic::elf:

lib/Object/SymbolicFile.cpp

   case file_magic::pdb:
   case file_magic::minidump:
   case file_magic::tapi_file:
+  case file_magic::clang_pch:
     return errorCodeToError(object_error::invalid_file_type);
   case file_magic::elf:
   case file_magic::elf_executable:

lib/ObjectYAML/ELFEmitter.cpp

 #include "llvm/ObjectYAML/yaml2obj.h"
 #include "llvm/Support/EndianStream.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/WithColor.h"
 #include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"

       if (!SN2I.lookup(Sym.Section, Index)) {
         WithColor::error() << "Unknown section referenced: '" << Sym.Section
                            << "' by YAML symbol " << Sym.Name << ".\n";
-        exit(1);
+        sys::Process::ExitCurrentProcess(1);
       }
       Symbol.st_shndx = Index;
     } else if (Sym.Index) {

                                           : Twine("`DynamicSymbols`")) +
                                 " for symbol table section '"
                          << RawSec->Name << "'.\n";
-    exit(1);
+    sys::Process::ExitCurrentProcess(1);
   }
 
   zero(SHeader);

       if (!SN2I.lookup(SecName.Section, Index)) {
         WithColor::error() << "Unknown section referenced: '" << SecName.Section
                            << "' by program header.\n";
-        exit(1);
+        sys::Process::ExitCurrentProcess(1);
       }
       Sections.push_back(&SHeaders[Index]);
     }

lib/Support/CMakeLists.txt

 if( MSVC OR MINGW )
   # libuuid required for FOLDERID_Profile usage in lib/Support/Windows/Path.inc.
   # advapi32 required for CryptAcquireContextW in lib/Support/Windows/Path.inc.
-  set(system_libs ${system_libs} psapi shell32 ole32 uuid advapi32)
+  # dbghelp for llvm::sys::Process::IsInTLS in CommandLine.h
+  set(system_libs ${system_libs} psapi shell32 ole32 uuid advapi32 dbghelp)
+#  set(system_libs ${system_libs} psapi shell32 ole32 uuid advapi32 libucrt)
 elseif( CMAKE_HOST_UNIX )
   if( HAVE_LIBRT )
     set(system_libs ${system_libs} rt)

lib/Support/CommandLine.cpp

 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Host.h"
+#include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"

   SmallPtrSet<SubCommand *, 4> RegisteredSubCommands;
 
   CommandLineParser() : ActiveSubCommand(nullptr) {
-    registerSubCommand(&*TopLevelSubCommand);
-    registerSubCommand(&*AllSubCommands);
+    registerSubCommand(&TopLevelSubCommand);
+    registerSubCommand(&AllSubCommands);
   }
 
+  CommandLineParser(const CommandLineParser& Other);
+
   void ResetAllOptionOccurrences();
 
   bool ParseCommandLineOptions(int argc, const char *const *argv,

 
     // If we're adding this to all sub-commands, add it to the ones that have
     // already been registered.
-    if (SC == &*AllSubCommands) {
+    if (SC == &AllSubCommands) {
       for (const auto &Sub : RegisteredSubCommands) {
         if (SC == Sub)
           continue;

 
   void addLiteralOption(Option &Opt, StringRef Name) {
     if (Opt.Subs.empty())
-      addLiteralOption(Opt, &*TopLevelSubCommand, Name);
+      addLiteralOption(Opt, &TopLevelSubCommand, Name);
     else {
       for (auto SC : Opt.Subs)
         addLiteralOption(Opt, SC, Name);

 
     // If we're adding this to all sub-commands, add it to the ones that have
     // already been registered.
-    if (SC == &*AllSubCommands) {
+    if (SC == &AllSubCommands) {
       for (const auto &Sub : RegisteredSubCommands) {
         if (SC == Sub)
           continue;

     }
 
     if (O->Subs.empty()) {
-      addOption(O, &*TopLevelSubCommand);
+      addOption(O, &TopLevelSubCommand);
     } else {
       for (auto SC : O->Subs)
         addOption(O, SC);

 
   void removeOption(Option *O) {
     if (O->Subs.empty())
-      removeOption(O, &*TopLevelSubCommand);
+      removeOption(O, &TopLevelSubCommand);
     else {
       if (O->isInAllSubCommands()) {
         for (auto SC : RegisteredSubCommands)

 
   void updateArgStr(Option *O, StringRef NewName) {
     if (O->Subs.empty())
-      updateArgStr(O, NewName, &*TopLevelSubCommand);
+      updateArgStr(O, NewName, &TopLevelSubCommand);
     else {
       if (O->isInAllSubCommands()) {
         for (auto SC : RegisteredSubCommands)

 
     // For all options that have been registered for all subcommands, add the
     // option to this subcommand now.
-    if (sub != &*AllSubCommands) {
-      for (auto &E : AllSubCommands->OptionsMap) {
+    if (sub != &AllSubCommands) {
+      for (auto &E : AllSubCommands.OptionsMap) {
         Option *O = E.second;
         if ((O->isPositional() || O->isSink() || O->isConsumeAfter()) ||
             O->hasArgStr())

     ResetAllOptionOccurrences();
     RegisteredSubCommands.clear();
 
-    TopLevelSubCommand->reset();
-    AllSubCommands->reset();
-    registerSubCommand(&*TopLevelSubCommand);
-    registerSubCommand(&*AllSubCommands);
+    TopLevelSubCommand.reset();
+    AllSubCommands.reset();
+    registerSubCommand(&TopLevelSubCommand);
+    registerSubCommand(&AllSubCommands);
 
     DefaultOptions.clear();
   }
 
+  // A special subcommand representing no subcommand.
+  SubCommand TopLevelSubCommand;
+
+  // A special subcommand that can be used to put an option into all subcommands.
+  SubCommand AllSubCommands;
+
 private:
   SubCommand *ActiveSubCommand;
 

 
 } // namespace
 
-static ManagedStatic<CommandLineParser> GlobalParser;
+static ManagedStatic<CommandLineParser> StaticParser;
 
+static CommandLineParser *GetParser() {
+  if (InitLLVM::RunningInBuildozer()) {
+    static LLVM_THREAD_LOCAL CommandLineParser Local(*StaticParser);
+    return &Local;
+  }
+  return &*StaticParser;
+}
+
 void cl::AddLiteralOption(Option &O, StringRef Name) {
-  GlobalParser->addLiteralOption(O, Name);
+  GetParser()->addLiteralOption(O, Name);
 }
 
 extrahelp::extrahelp(StringRef Help) : morehelp(Help) {
-  GlobalParser->MoreHelp.push_back(Help);
+  GetParser()->MoreHelp.push_back(Help);
 }
 
 void Option::addArgument() {
-  GlobalParser->addOption(this);
+  GetParser()->addOption(this);
   FullyInitialized = true;
 }
 
-void Option::removeArgument() { GlobalParser->removeOption(this); }
+void Option::removeArgument() { GetParser()->removeOption(this); }
 
+bool Option::isInAllSubCommands() const {
+  return any_of(Subs,
+                [](const SubCommand *SC) { return SC == &(GetParser()->AllSubCommands); });
+}
+
 void Option::setArgStr(StringRef S) {
   if (FullyInitialized)
-    GlobalParser->updateArgStr(this, S);
+    GetParser()->updateArgStr(this, S);
   assert((S.empty() || S[0] != '-') && "Option can't start with '-");
   ArgStr = S;
   if (ArgStr.size() == 1)

     removeArgument();
 }
 
-// Initialise the general option category.
+// Initialize the general option category.
 OptionCategory llvm::cl::GeneralCategory("General options");
 
 void OptionCategory::registerCategory() {
-  GlobalParser->registerCategory(this);
+  GetParser()->registerCategory(this);
 }
 
-// A special subcommand representing no subcommand. It is particularly important
-// that this ManagedStatic uses constant initailization and not dynamic
-// initialization because it is referenced from cl::opt constructors, which run
-// dynamically in an arbitrary order.
-LLVM_REQUIRE_CONSTANT_INITIALIZATION
-ManagedStatic<SubCommand> llvm::cl::TopLevelSubCommand;
-
-// A special subcommand that can be used to put an option into all subcommands.
-ManagedStatic<SubCommand> llvm::cl::AllSubCommands;
-
 void SubCommand::registerSubCommand() {
-  GlobalParser->registerSubCommand(this);
+  GetParser()->registerSubCommand(this);
 }
 
 void SubCommand::unregisterSubCommand() {
-  GlobalParser->unregisterSubCommand(this);
+  GetParser()->unregisterSubCommand(this);
 }
 
 void SubCommand::reset() {

 }
 
 SubCommand::operator bool() const {
-  return (GlobalParser->getActiveSubCommand() == this);
+  return (GetParser()->getActiveSubCommand() == this);
 }
 
 //===----------------------------------------------------------------------===//

 // Basic, shared command line option processing machinery.
 //
 
+// Duplicate into a self-contained Parser
+CommandLineParser::CommandLineParser(const CommandLineParser &Other) {
+  RegisteredOptionCategories = Other.RegisteredOptionCategories;
+
+  auto computeSize = [&Other]() -> unsigned {
+    DenseSet<Option *> Opt;
+
+    for (Option *O : Other.DefaultOptions)
+      Opt.insert(O);
+    for (SubCommand *C : Other.RegisteredSubCommands) {
+      for (Option *O : C->PositionalOpts)
+        Opt.insert(O);
+      for (Option *O : C->SinkOpts)
+        Opt.insert(O);
+      for (auto &P : C->OptionsMap)
+        Opt.insert(P.getValue());
+    }
+
+    unsigned Size{};
+    for (Option *O : Opt)
+      Size += O->getSize();
+    return Size;
+  };
+
+  static unsigned TotalOptionsSize = computeSize();
+  static LLVM_THREAD_LOCAL std::unique_ptr<char[]> OptStorage(
+      new char[TotalOptionsSize]);
+
+  // Now remap all point from the source
+
+  DenseMap<Option *, Option *> RemapOpt;
+  char *Next = OptStorage.get();
+
+  DenseMap<SubCommand *, SubCommand *> RemapCommands;
+
+  auto FindNewO = [&](Option *O) -> Option * {
+    auto R = RemapOpt.insert({O, (Option *)Next});
+    if (!R.second)
+      return R.first->second;
+    // Ensure external location are in .tls storage, to accommodate for
+    // Buildozer launching several thread of the Clang driver
+    //O->checkLocation();
+    Option *NewO = (Option *)Next;
+    unsigned Size = O->getSize();
+    O->duplicateInto(NewO);
+
+    NewO->Subs.clear();
+    for (auto S : O->Subs) {
+      SubCommand *NewS = RemapCommands[S];
+      assert(NewS);
+      NewO->Subs.insert(NewS);
+    }
+
+    Next += Size;
+    return NewO;
+  };
+
+  registerSubCommand(&TopLevelSubCommand);
+  registerSubCommand(&AllSubCommands);
+
+  for (SubCommand *C : Other.RegisteredSubCommands) {
+    if (C == &Other.TopLevelSubCommand) {
+      RemapCommands[C] = &TopLevelSubCommand;
+      continue;
+    }
+    if (C == &Other.AllSubCommands) {
+      RemapCommands[C] = &AllSubCommands;
+      continue;
+    }
+    assert(false);
+  }
+
+  for (Option *O : Other.DefaultOptions) {
+    Option *NewO = FindNewO(O);
+    DefaultOptions.push_back(NewO);
+  }
+
+  for (SubCommand *C : Other.RegisteredSubCommands) {
+    SubCommand *ThisC = RemapCommands[C];
+    for (Option *O : C->PositionalOpts) {
+      Option *NewO = FindNewO(O);
+      ThisC->PositionalOpts.push_back(NewO);
+    }
+    for (Option *O : C->SinkOpts) {
+      Option *NewO = FindNewO(O);
+      ThisC->SinkOpts.push_back(NewO);
+    }
+    for (auto &P : C->OptionsMap) {
+      Option *NewO = FindNewO(P.getValue());
+      ThisC->OptionsMap[P.getKey()] = NewO;
+    }
+  }
+
+  for (auto &Pair : RemapOpt)
+    Pair.second->postRemap(RemapOpt);
+}
+
 /// LookupOption - Lookup the option specified by the specified option on the
 /// command line.  If there is a value specified (after an equal sign) return
 /// that as well.  This assumes that leading dashes have already been stripped.

   // Reject all dashes.
   if (Arg.empty())
     return nullptr;
-  assert(&Sub != &*AllSubCommands);
+  assert(&Sub != &AllSubCommands);
 
   size_t EqualPos = Arg.find('=');
 

 
 SubCommand *CommandLineParser::LookupSubCommand(StringRef Name) {
   if (Name.empty())
-    return &*TopLevelSubCommand;
+    return &TopLevelSubCommand;
   for (auto S : RegisteredSubCommands) {
-    if (S == &*AllSubCommands)
+    if (S == &AllSubCommands)
       continue;
     if (S->getName().empty())
       continue;

     if (StringRef(S->getName()) == StringRef(Name))
       return S;
   }
-  return &*TopLevelSubCommand;
+  return &TopLevelSubCommand;
 }
 
 /// LookupNearestOption - Lookup the closest match to the option specified by

   int NewArgc = static_cast<int>(NewArgv.size());
 
   // Parse all options.
-  return GlobalParser->ParseCommandLineOptions(NewArgc, &NewArgv[0], Overview,
+  return GetParser()->ParseCommandLineOptions(NewArgc, &NewArgv[0], Overview,
                                                Errs, LongOptionsUseDoubleDash);
 }
 

   bool HasUnlimitedPositionals = false;
 
   int FirstArg = 1;
-  SubCommand *ChosenSubCommand = &*TopLevelSubCommand;
+  SubCommand *ChosenSubCommand = &TopLevelSubCommand;
   if (argc >= 2 && argv[FirstArg][0] != '-') {
     // If the first argument specifies a valid subcommand, start processing
     // options from the second argument.
     ChosenSubCommand = LookupSubCommand(StringRef(argv[FirstArg]));
-    if (ChosenSubCommand != &*TopLevelSubCommand)
+    if (ChosenSubCommand != &TopLevelSubCommand)
       FirstArg = 2;
   }
-  GlobalParser->ActiveSubCommand = ChosenSubCommand;
+  GetParser()->ActiveSubCommand = ChosenSubCommand;
 
   assert(ChosenSubCommand);
   auto &ConsumeAfterOpt = ChosenSubCommand->ConsumeAfterOpt;

   // If we had an error processing our arguments, don't let the program execute
   if (ErrorParsing) {
     if (!IgnoreErrors)
-      exit(1);
+      sys::Process::ExitCurrentProcess(1);
     return false;
   }
   return true;

   if (ArgName.empty())
     Errs << HelpStr; // Be nice for positional arguments
   else
-    Errs << GlobalParser->ProgramName << ": for the " << PrintArg(ArgName);
+    Errs << GetParser()->ProgramName << ": for the " << PrintArg(ArgName);
 
   Errs << " option: " << Message << "\n";
   return true;

     printHelp();
 
     // Halt the program since help information was printed
-    exit(0);
+    sys::Process::ExitCurrentProcess(0);
   }
 
   void printHelp() {
-    SubCommand *Sub = GlobalParser->getActiveSubCommand();
+    SubCommand *Sub = GetParser()->getActiveSubCommand();
     auto &OptionsMap = Sub->OptionsMap;
     auto &PositionalOpts = Sub->PositionalOpts;
     auto &ConsumeAfterOpt = Sub->ConsumeAfterOpt;

     sortOpts(OptionsMap, Opts, ShowHidden);
 
     StrSubCommandPairVector Subs;
-    sortSubCommands(GlobalParser->RegisteredSubCommands, Subs);
+    sortSubCommands(GetParser()->RegisteredSubCommands, Subs);
 
-    if (!GlobalParser->ProgramOverview.empty())
-      outs() << "OVERVIEW: " << GlobalParser->ProgramOverview << "\n";
+    if (!GetParser()->ProgramOverview.empty())
+      outs() << "OVERVIEW: " << GetParser()->ProgramOverview << "\n";
 
-    if (Sub == &*TopLevelSubCommand) {
-      outs() << "USAGE: " << GlobalParser->ProgramName;
+    if (Sub == &(GetParser()->TopLevelSubCommand)) {
+      outs() << "USAGE: " << GetParser()->ProgramName;
       if (Subs.size() > 2)
         outs() << " [subcommand]";
       outs() << " [options]";

         outs() << "SUBCOMMAND '" << Sub->getName()
                << "': " << Sub->getDescription() << "\n\n";
       }
-      outs() << "USAGE: " << GlobalParser->ProgramName << " " << Sub->getName()
+      outs() << "USAGE: " << GetParser()->ProgramName << " " << Sub->getName()
              << " [options]";
     }
 

     if (ConsumeAfterOpt)
       outs() << " " << ConsumeAfterOpt->HelpStr;
 
-    if (Sub == &*TopLevelSubCommand && !Subs.empty()) {
+    if (Sub == &(GetParser()->TopLevelSubCommand) && !Subs.empty()) {
       // Compute the maximum subcommand length...
       size_t MaxSubLen = 0;
       for (size_t i = 0, e = Subs.size(); i != e; ++i)

       outs() << "SUBCOMMANDS:\n\n";
       printSubCommands(Subs, MaxSubLen);
       outs() << "\n";
-      outs() << "  Type \"" << GlobalParser->ProgramName
+      outs() << "  Type \"" << GetParser()->ProgramName
              << " <subcommand> --help\" to get more help on a specific "
                 "subcommand";
     }

     printOptions(Opts, MaxArgLen);
 
     // Print any extra help the user has declared.
-    for (auto I : GlobalParser->MoreHelp)
+    for (auto I : GetParser()->MoreHelp)
       outs() << I;
-    GlobalParser->MoreHelp.clear();
+    GetParser()->MoreHelp.clear();
   }
 };
 

 
     // Collect registered option categories into vector in preparation for
     // sorting.
-    for (auto I = GlobalParser->RegisteredOptionCategories.begin(),
-              E = GlobalParser->RegisteredOptionCategories.end();
+    for (auto I = GetParser()->RegisteredOptionCategories.begin(),
+              E = GetParser()->RegisteredOptionCategories.end();
          I != E; ++I) {
       SortedCategories.push_back(*I);
     }

 
 // Declare the four HelpPrinter instances that are used to print out help, or
 // help-hidden as an uncategorized list or in categories.
-static HelpPrinter UncategorizedNormalPrinter(false);
-static HelpPrinter UncategorizedHiddenPrinter(true);
-static CategorizedHelpPrinter CategorizedNormalPrinter(false);
-static CategorizedHelpPrinter CategorizedHiddenPrinter(true);
+static LLVM_THREAD_LOCAL HelpPrinter UncategorizedNormalPrinter(false);
+static LLVM_THREAD_LOCAL HelpPrinter UncategorizedHiddenPrinter(true);
+static LLVM_THREAD_LOCAL CategorizedHelpPrinter CategorizedNormalPrinter(false);
+static LLVM_THREAD_LOCAL CategorizedHelpPrinter CategorizedHiddenPrinter(true);
 
 // Declare HelpPrinter wrappers that will decide whether or not to invoke
 // a categorizing help printer
-static HelpPrinterWrapper WrappedNormalPrinter(UncategorizedNormalPrinter,
-                                               CategorizedNormalPrinter);
-static HelpPrinterWrapper WrappedHiddenPrinter(UncategorizedHiddenPrinter,
-                                               CategorizedHiddenPrinter);
+static LLVM_THREAD_LOCAL HelpPrinterWrapper
+    WrappedNormalPrinter(UncategorizedNormalPrinter, CategorizedNormalPrinter);
+static LLVM_THREAD_LOCAL HelpPrinterWrapper
+    WrappedHiddenPrinter(UncategorizedHiddenPrinter, CategorizedHiddenPrinter);
 
 // Define a category for generic options that all tools should have.
 static cl::OptionCategory GenericCategory("Generic Options");

     "help-list",
     cl::desc("Display list of available options (--help-list-hidden for more)"),
     cl::location(UncategorizedNormalPrinter), cl::Hidden, cl::ValueDisallowed,
-    cl::cat(GenericCategory), cl::sub(*AllSubCommands));
+    cl::cat(GenericCategory), cl::allSub());
 
 static cl::opt<HelpPrinter, true, parser<bool>>
     HLHOp("help-list-hidden", cl::desc("Display list of all available options"),
           cl::location(UncategorizedHiddenPrinter), cl::Hidden,
           cl::ValueDisallowed, cl::cat(GenericCategory),
-          cl::sub(*AllSubCommands));
+          cl::allSub());
 
 // Define uncategorized/categorized help printers. These printers change their
 // behaviour at runtime depending on whether one or more Option categories have

 static cl::opt<HelpPrinterWrapper, true, parser<bool>>
     HOp("help", cl::desc("Display available options (--help-hidden for more)"),
         cl::location(WrappedNormalPrinter), cl::ValueDisallowed,
-        cl::cat(GenericCategory), cl::sub(*AllSubCommands));
+        cl::cat(GenericCategory), cl::allSub());
 
 static cl::alias HOpA("h", cl::desc("Alias for --help"), cl::aliasopt(HOp),
                       cl::DefaultOption);

 static cl::opt<HelpPrinterWrapper, true, parser<bool>>
     HHOp("help-hidden", cl::desc("Display all available options"),
          cl::location(WrappedHiddenPrinter), cl::Hidden, cl::ValueDisallowed,
-         cl::cat(GenericCategory), cl::sub(*AllSubCommands));
+         cl::cat(GenericCategory), cl::allSub());
 
 static cl::opt<bool> PrintOptions(
     "print-options",
     cl::desc("Print non-default options after command line parsing"),
     cl::Hidden, cl::init(false), cl::cat(GenericCategory),
-    cl::sub(*AllSubCommands));
+    cl::allSub());
 
 static cl::opt<bool> PrintAllOptions(
     "print-all-options",
     cl::desc("Print all option values after command line parsing"), cl::Hidden,
-    cl::init(false), cl::cat(GenericCategory), cl::sub(*AllSubCommands));
+    cl::init(false), cl::cat(GenericCategory), cl::allSub());
 
 void HelpPrinterWrapper::operator=(bool Value) {
   if (!Value)

   // Decide which printer to invoke. If more than one option category is
   // registered then it is useful to show the categorized help instead of
   // uncategorized help.
-  if (GlobalParser->RegisteredOptionCategories.size() > 1) {
+  if (GetParser()->RegisteredOptionCategories.size() > 1) {
     // unhide --help-list option so user can have uncategorized output if they
     // want it.
     HLOp.setHiddenFlag(NotHidden);

 }
 
 // Print the value of each option.
-void cl::PrintOptionValues() { GlobalParser->printOptionValues(); }
+void cl::PrintOptionValues() { GetParser()->printOptionValues(); }
 
 void CommandLineParser::printOptionValues() {
   if (!PrintOptions && !PrintAllOptions)

 
     if (OverrideVersionPrinter != nullptr) {
       OverrideVersionPrinter(outs());
-      exit(0);
+      sys::Process::ExitCurrentProcess(0);
     }
     print();
 

         I(outs());
     }
 
-    exit(0);
+    sys::Process::ExitCurrentProcess(0);
   }
 };
 } // End anonymous namespace
 
 // Define the --version option that prints out the LLVM version for the tool
-static VersionPrinter VersionPrinterInstance;
+static LLVM_THREAD_LOCAL VersionPrinter VersionPrinterInstance;
 
 static cl::opt<VersionPrinter, true, parser<bool>>
     VersOp("version", cl::desc("Display the version of this program"),

   ExtraVersionPrinters->push_back(func);
 }
 
-StringMap<Option *> &cl::getRegisteredOptions(SubCommand &Sub) {
-  auto &Subs = GlobalParser->RegisteredSubCommands;
+StringMap<Option *> &cl::getRegisteredOptions() {
+  SubCommand &Sub = GetParser()->TopLevelSubCommand;
+  auto &Subs = GetParser()->RegisteredSubCommands;
   (void)Subs;
   assert(is_contained(Subs, &Sub));
   return Sub.OptionsMap;

 
 iterator_range<typename SmallPtrSet<SubCommand *, 4>::iterator>
 cl::getRegisteredSubcommands() {
-  return GlobalParser->getRegisteredSubcommands();
+  return GetParser()->getRegisteredSubcommands();
 }
 
-void cl::HideUnrelatedOptions(cl::OptionCategory &Category, SubCommand &Sub) {
+bool cl::isTopLevel(SubCommand *SC) {
+  return &(GetParser()->TopLevelSubCommand) == SC;
+}
+
+void cl::HideUnrelatedOptions(cl::OptionCategory &Category) {
+  SubCommand &Sub = GetParser()->TopLevelSubCommand;
   for (auto &I : Sub.OptionsMap) {
     for (auto &Cat : I.second->Categories) {
       if (Cat != &Category &&

   }
 }
 
-void cl::HideUnrelatedOptions(ArrayRef<const cl::OptionCategory *> Categories,
-                              SubCommand &Sub) {
+void cl::HideUnrelatedOptions(ArrayRef<const cl::OptionCategory *> Categories) {
+  SubCommand &Sub = GetParser()->TopLevelSubCommand;
   for (auto &I : Sub.OptionsMap) {
     for (auto &Cat : I.second->Categories) {
       if (find(Categories, Cat) == Categories.end() && Cat != &GenericCategory)

   }
 }
 
-void cl::ResetCommandLineParser() { GlobalParser->reset(); }
+void cl::ResetCommandLineParser() { GetParser()->reset(); }
 void cl::ResetAllOptionOccurrences() {
-  GlobalParser->ResetAllOptionOccurrences();
+  GetParser()->ResetAllOptionOccurrences();
 }
 
 void LLVMParseCommandLineOptions(int argc, const char *const *argv,

   llvm::cl::ParseCommandLineOptions(argc, argv, StringRef(Overview),
                                     &llvm::nulls());
 }
+
+cl::allSub::allSub() : sub(GetParser()->AllSubCommands) {
+}
+
+cl::topLevel::topLevel() : sub(GetParser()->TopLevelSubCommand) {
+}

lib/Support/CrashRecoveryContext.cpp

 #include "llvm/Support/ThreadLocal.h"
 #include <mutex>
 #include <setjmp.h>
+
+#if defined(_MSC_VER)
+#include <windows.h>
+#endif
+
 using namespace llvm;
 
 namespace {

 #endif
   }
 
-  void HandleCrash() {
+  void HandleCrash(int retCode) {
     // Eliminate the current context entry, to avoid re-entering in case the
     // cleanup code crashes.
     CurrentContext->set(Next);

     assert(!Failed && "Crash recovery context already failed!");
     Failed = true;
 
+    CRC->RetCode = retCode;
+
     // FIXME: Stash the backtrace.
 
     // Jump back to the RunSafely we were called under.

 static void installExceptionOrSignalHandlers() {}
 static void uninstallExceptionOrSignalHandlers() {}
 
+// In Signals.inc
+LONG WINAPI LLVMUnhandledExceptionFilter(LPEXCEPTION_POINTERS ep);
+
 bool CrashRecoveryContext::RunSafely(function_ref<void()> Fn) {
   if (!gCrashRecoveryEnabled) {
     Fn();

   }
 
   bool Result = true;
+  // error: cannot compile this 'this' captured by SEH yet
+  CrashRecoveryContext *This = this;
   __try {
     Fn();
-  } __except (1) { // Catch any exception.
+  } __except (This->EnableExceptionHandler
+                  ? LLVMUnhandledExceptionFilter(GetExceptionInformation())
+                  : 1) {
+    // assert(false);
+    This->RetCode = GetExceptionCode();
     Result = false;
   }
   return Result;

   // implementation if we so choose.
 
   // Handle the crash
-  const_cast<CrashRecoveryContextImpl*>(CRCI)->HandleCrash();
+  const_cast<CrashRecoveryContextImpl *>(CRCI)->HandleCrash(
+      (int)ExceptionInfo->ExceptionRecord->ExceptionCode);
 
   // Note that we don't actually get here because HandleCrash calls
   // longjmp, which means the HandleCrash function never returns.

   sigprocmask(SIG_UNBLOCK, &SigMask, nullptr);
 
   if (CRCI)
-    const_cast<CrashRecoveryContextImpl*>(CRCI)->HandleCrash();
+    const_cast<CrashRecoveryContextImpl*>(CRCI)->HandleCrash(Signal);
 }
 
 static void installExceptionOrSignalHandlers() {

 void CrashRecoveryContext::HandleCrash() {
   CrashRecoveryContextImpl *CRCI = (CrashRecoveryContextImpl *) Impl;
   assert(CRCI && "Crash recovery context never initialized!");
-  CRCI->HandleCrash();
+  CRCI->HandleCrash(-1);
 }
 
 // FIXME: Portability.

lib/Support/Debug.cpp

 // built without NDEBUG can depend on via the llvm/Support/Debug.h header.
 namespace llvm {
 /// Exported boolean set by the -debug option.
-bool DebugFlag = false;
+LLVM_THREAD_LOCAL bool DebugFlag = false;
 
 static ManagedStatic<std::vector<std::string>> CurrentDebugType;
 

 
 }
 
-static DebugOnlyOpt DebugOnlyOptLoc;
+static LLVM_THREAD_LOCAL DebugOnlyOpt DebugOnlyOptLoc;
 
 static cl::opt<DebugOnlyOpt, true, cl::parser<std::string> >
 DebugOnly("debug-only", cl::desc("Enable a specific type of debug output (comma separated list of types)"),

lib/Support/DebugCounter.cpp

 #include "llvm/Support/DebugCounter.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Format.h"
+#include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Options.h"
 

     // width, so we do the same.
     Option::printHelpStr(HelpStr, GlobalWidth, ArgStr.size() + 6);
     const auto &CounterInstance = DebugCounter::instance();
-    for (auto Name : CounterInstance) {
+    for (auto Name : CounterInstance.getCounterVector()) {
       const auto Info =
           CounterInstance.getCounterInfo(CounterInstance.getCounterId(Name));
       size_t NumSpaces = GlobalWidth - Info.first.size() - 8;

     print(dbgs());
 }
 
-DebugCounter &DebugCounter::instance() { return *DC; }
+DebugCounter &DebugCounter::instance() {
+  if (InitLLVM::RunningInBuildozer()) {
+    static LLVM_THREAD_LOCAL DebugCounter Local;
+    return Local;
+  }
+  return *DC;
+}
 
 // This is called by the command line parser when it sees a value for the
 // debug-counter option defined above.

lib/Support/Error.cpp

 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/ErrorHandling.h"
 #include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/Process.h"
 #include <system_error>
 
 using namespace llvm;

   return reinterpret_cast<void *>(&StringError::ID);
 }
 
+void ExitOnError::checkError(Error Err) const {
+  if (Err) {
+    int ExitCode = GetExitCode(Err);
+    logAllUnhandledErrors(std::move(Err), errs(), Banner);
+    sys::Process::ExitCurrentProcess(ExitCode);
+  }
+}
+
 #ifndef _MSC_VER
 namespace llvm {
 

lib/Support/ErrorHandling.cpp

 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/Error.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/Threading.h"
 #include "llvm/Support/WindowsError.h"

 
 using namespace llvm;
 
-static fatal_error_handler_t ErrorHandler = nullptr;
-static void *ErrorHandlerUserData = nullptr;
+static LLVM_THREAD_LOCAL fatal_error_handler_t ErrorHandler = nullptr;
+static LLVM_THREAD_LOCAL void *ErrorHandlerUserData = nullptr;
 
-static fatal_error_handler_t BadAllocErrorHandler = nullptr;
-static void *BadAllocErrorHandlerUserData = nullptr;
+static LLVM_THREAD_LOCAL fatal_error_handler_t BadAllocErrorHandler = nullptr;
+static LLVM_THREAD_LOCAL void *BadAllocErrorHandlerUserData = nullptr;
 
 #if LLVM_ENABLE_THREADS == 1
 // Mutexes to synchronize installing error handlers and calling error handlers.

 // STL and hide all of its symbols with 'opt -internalize'. To reduce size, it
 // cuts out the threading portions of the hermetic copy of libc++ that it
 // builds. We can remove these ifdefs if that script goes away.
-static std::mutex ErrorHandlerMutex;
-static std::mutex BadAllocErrorHandlerMutex;
+//static LLVM_THREAD_LOCAL std::mutex ErrorHandlerMutex;
+//static LLVM_THREAD_LOCAL std::mutex BadAllocErrorHandlerMutex;
 #endif
 
 void llvm::install_fatal_error_handler(fatal_error_handler_t handler,
                                        void *user_data) {
 #if LLVM_ENABLE_THREADS == 1
-  std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
+  //std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
 #endif
   assert(!ErrorHandler && "Error handler already registered!\n");
   ErrorHandler = handler;

 
 void llvm::remove_fatal_error_handler() {
 #if LLVM_ENABLE_THREADS == 1
-  std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
+  //std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
 #endif
   ErrorHandler = nullptr;
   ErrorHandlerUserData = nullptr;

     // Only acquire the mutex while reading the handler, so as not to invoke a
     // user-supplied callback under a lock.
 #if LLVM_ENABLE_THREADS == 1
-    std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
+    //std::lock_guard<std::mutex> Lock(ErrorHandlerMutex);
 #endif
     handler = ErrorHandler;
     handlerData = ErrorHandlerUserData;

   // files registered with RemoveFileOnSignal.
   sys::RunInterruptHandlers();
 
-  exit(1);
+  sys::Process::ExitCurrentProcess(1);
 }
 
 void llvm::install_bad_alloc_error_handler(fatal_error_handler_t handler,
                                            void *user_data) {
 #if LLVM_ENABLE_THREADS == 1
-  std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
+  //std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
 #endif
   assert(!ErrorHandler && "Bad alloc error handler already registered!\n");
   BadAllocErrorHandler = handler;

 
 void llvm::remove_bad_alloc_error_handler() {
 #if LLVM_ENABLE_THREADS == 1
-  std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
+  //std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
 #endif
   BadAllocErrorHandler = nullptr;
   BadAllocErrorHandlerUserData = nullptr;

     // Only acquire the mutex while reading the handler, so as not to invoke a
     // user-supplied callback under a lock.
 #if LLVM_ENABLE_THREADS == 1
-    std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
+    //std::lock_guard<std::mutex> Lock(BadAllocErrorHandlerMutex);
 #endif
     Handler = BadAllocErrorHandler;
     HandlerData = BadAllocErrorHandlerUserData;

lib/Support/InitLLVM.cpp

-//===-- InitLLVM.cpp -----------------------------------------------------===//
-//
-// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
-// See https://llvm.org/LICENSE.txt for license information.
-// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-//
-//===----------------------------------------------------------------------===//
-
-#include "llvm/Support/InitLLVM.h"
-#include "llvm/Support/Error.h"
-#include "llvm/Support/ManagedStatic.h"
-#include "llvm/Support/PrettyStackTrace.h"
-#include "llvm/Support/Process.h"
-#include "llvm/Support/Signals.h"
-#include <string>
-
-#ifdef _WIN32
-#include "Windows/WindowsSupport.h"
-#endif
-
-using namespace llvm;
-using namespace llvm::sys;
-
-InitLLVM::InitLLVM(int &Argc, const char **&Argv) : StackPrinter(Argc, Argv) {
-  sys::PrintStackTraceOnErrorSignal(Argv[0]);
-  install_out_of_memory_new_handler();
-
-#ifdef _WIN32
-  // We use UTF-8 as the internal character encoding. On Windows,
-  // arguments passed to main() may not be encoded in UTF-8. In order
-  // to reliably detect encoding of command line arguments, we use an
-  // Windows API to obtain arguments, convert them to UTF-8, and then
-  // write them back to the Argv vector.
-  //
-  // There's probably other way to do the same thing (e.g. using
-  // wmain() instead of main()), but this way seems less intrusive
-  // than that.
-  std::string Banner = std::string(Argv[0]) + ": ";
-  ExitOnError ExitOnErr(Banner);
-
-  ExitOnErr(errorCodeToError(windows::GetCommandLineArguments(Args, Alloc)));
-
-  // GetCommandLineArguments doesn't terminate the vector with a
-  // nullptr.  Do it to make it compatible with the real argv.
-  Args.push_back(nullptr);
-
-  Argc = Args.size() - 1;
-  Argv = Args.data();
-#endif
-}
-
-InitLLVM::~InitLLVM() { llvm_shutdown(); }
+//===-- InitLLVM.cpp -----------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/Error.h"
+#include "llvm/Support/ManagedStatic.h"
+#include "llvm/Support/PrettyStackTrace.h"
+#include "llvm/Support/Process.h"
+#include "llvm/Support/Signals.h"
+#include <string>
+
+#ifdef _WIN32
+#include "Windows/WindowsSupport.h"
+#include <process.h>
+#endif
+
+using namespace llvm;
+using namespace llvm::sys;
+
+// Needed to initialize under-the-hood from Buildozer
+static LLVM_THREAD_LOCAL void* CRTInitEvent{};
+static LLVM_THREAD_LOCAL void* CRTExitEvent{};
+static bool ShouldSkipInitLLVM{};
+LLVM_THREAD_LOCAL HANDLE CurrentHandle{};
+
+InitLLVM::InitLLVM(int &Argc, const char **&Argv) : StackPrinter(Argc, Argv) {
+  if (CRTInitEvent) {
+#ifdef _WIN32
+    SetEvent((HANDLE)CRTInitEvent);
+    WaitForSingleObject(CRTExitEvent, INFINITE);
+    llvm_shutdown(); // clean up allocations
+    _cexit();
+    sys::Process::ExitCurrentProcess(0);
+#endif
+  }
+  if (ShouldSkipInitLLVM)
+    return;
+
+  sys::PrintStackTraceOnErrorSignal(Argv[0]);
+  install_out_of_memory_new_handler();
+
+#ifdef _WIN32
+  // We use UTF-8 as the internal character encoding. On Windows,
+  // arguments passed to main() may not be encoded in UTF-8. In order
+  // to reliably detect encoding of command line arguments, we use an
+  // Windows API to obtain arguments, convert them to UTF-8, and then
+  // write them back to the Argv vector.
+  //
+  // There's probably other way to do the same thing (e.g. using
+  // wmain() instead of main()), but this way seems less intrusive
+  // than that.
+  std::string Banner = std::string(Argv[0]) + ": ";
+  ExitOnError ExitOnErr(Banner);
+
+  ExitOnErr(errorCodeToError(windows::GetCommandLineArguments(Args, Alloc)));
+
+  // GetCommandLineArguments doesn't terminate the vector with a
+  // nullptr.  Do it to make it compatible with the real argv.
+  Args.push_back(nullptr);
+
+  Argc = Args.size() - 1;
+  Argv = Args.data();
+#endif
+}
+
+InitLLVM::~InitLLVM() {
+  if (ShouldSkipInitLLVM)
+    return;
+  llvm_shutdown();
+}
+
+__attribute__((used)) void InitLLVM::InitFromBuildozer(void *InitEvent,
+                                                       void *ExitEvent) {
+  CRTInitEvent = InitEvent;
+  CRTExitEvent = ExitEvent;
+}
+
+__attribute__((used)) void
+InitLLVM::SetCurrentModuleBackdoor(void *ModuleHandle) {
+  CurrentHandle = ModuleHandle;
+}
+
+__attribute__((used))
+void InitLLVM::SkipInitLLVM() { ShouldSkipInitLLVM = true; }
+
+bool InitLLVM::RunningInBuildozer() { return ShouldSkipInitLLVM; }
+
+//#pragma comment(linker, "/INCLUDE:?WillBlockMain@InitLLVM@llvm@@SAXPEAX@Z")

lib/Support/Process.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/Program.h"
+#include "llvm/Support/InitLLVM.h"
 
 using namespace llvm;
 using namespace sys;

lib/Support/Signals.cpp

 using namespace llvm;
 
 // Use explicit storage to avoid accessing cl::opt in a signal handler.
-static bool DisableSymbolicationFlag = false;
+static LLVM_THREAD_LOCAL bool DisableSymbolicationFlag = false;
 static cl::opt<bool, true>
     DisableSymbolication("disable-symbolication",
                          cl::desc("Disable symbolizing crash backtraces."),

lib/Support/StringMap.cpp

 
 using namespace llvm;
 
+HashTableStats StringMapImpl::Stats;
+
 /// Returns the number of buckets to allocate to ensure that the DenseMap can
 /// accommodate \p NumEntries without need to grow().
 static unsigned getMinBucketToReserveForEntries(unsigned NumEntries) {

   unsigned FullHashValue = djbHash(Name, 0);
   unsigned BucketNo = FullHashValue & (HTSize-1);
   unsigned *HashTable = (unsigned *)(TheTable + NumBuckets + 1);
-
+  HashTableStats::State S;
   unsigned ProbeAmt = 1;
   int FirstTombstone = -1;
   while (true) {
     StringMapEntryBase *BucketItem = TheTable[BucketNo];
+
+    Stats.updateOnAccess(S, ProbeAmt, TheTable+BucketNo);
+
     // If we found an empty bucket, this key isn't in the table yet, return it.
     if (LLVM_LIKELY(!BucketItem)) {
       // If we found a tombstone, we want to reuse the tombstone instead of an

   unsigned FullHashValue = djbHash(Key, 0);
   unsigned BucketNo = FullHashValue & (HTSize-1);
   unsigned *HashTable = (unsigned *)(TheTable + NumBuckets + 1);
-
+  HashTableStats::State S;
   unsigned ProbeAmt = 1;
   while (true) {
     StringMapEntryBase *BucketItem = TheTable[BucketNo];
+
+    Stats.updateOnAccess(S, ProbeAmt, TheTable+BucketNo);
+
     // If we found an empty bucket, this key isn't in the table yet, return.
     if (LLVM_LIKELY(!BucketItem))
       return -1;

lib/Support/TargetRegistry.cpp

 #include "llvm/Support/TargetRegistry.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
+#include "llvm/Support/RWMutex.h"
 #include "llvm/Support/raw_ostream.h"
 #include <cassert>
 #include <vector>

 using namespace llvm;
 
 // Clients are responsible for avoid race conditions in registration.
-static Target *FirstTarget = nullptr;
+static Target *FirstTarget{};
+static unsigned TargetsCount{};
+static sys::SmartRWMutex<true> TargetsMutex;
 
 iterator_range<TargetRegistry::iterator> TargetRegistry::targets() {
-  return make_range(iterator(FirstTarget), iterator());
+  return make_range(iterator(FirstTarget), iterator(), TargetsCount);
 }
 
 const Target *TargetRegistry::lookupTarget(const std::string &ArchName,

   // name, because it might be a backend that has no mapping to a target triple.
   const Target *TheTarget = nullptr;
   if (!ArchName.empty()) {
+    sys::SmartScopedReader<true> Lock(TargetsMutex);
     auto I = find_if(targets(),
                      [&](const Target &T) { return ArchName == T.getName(); });
 

 
 const Target *TargetRegistry::lookupTarget(const std::string &TT,
                                            std::string &Error) {
+  sys::SmartScopedReader<true> Lock(TargetsMutex);
   // Provide special warning when no targets are initialized.
   if (targets().begin() == targets().end()) {
     Error = "Unable to find target for this triple (no targets are registered)";

   assert(Name && ShortDesc && ArchMatchFn &&
          "Missing required target information!");
 
+  sys::SmartScopedWriter<true> Lock(TargetsMutex);
+
   // Check if this target has already been initialized, we allow this as a
   // convenience to some clients.
   if (T.Name)

   // Add to the list of targets.
   T.Next = FirstTarget;
   FirstTarget = &T;
+  ++TargetsCount;
 
   T.Name = Name;
   T.ShortDesc = ShortDesc;

 void TargetRegistry::printRegisteredTargetsForVersion(raw_ostream &OS) {
   std::vector<std::pair<StringRef, const Target*> > Targets;
   size_t Width = 0;
-  for (const auto &T : TargetRegistry::targets()) {
-    Targets.push_back(std::make_pair(T.getName(), &T));
-    Width = std::max(Width, Targets.back().first.size());
+  {
+    sys::SmartScopedReader<true> Lock(TargetsMutex);
+    
+    auto Range = TargetRegistry::targets();
+    Targets.reserve(Range.size());
+    for (const auto &T : Range) {
+      Targets.push_back(std::make_pair(T.getName(), &T));
+      Width = std::max(Width, Targets.back().first.size());
+    }
   }
   array_pod_sort(Targets.begin(), Targets.end(), TargetArraySortFn);
 

lib/Support/Timer.cpp

 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
+#include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/ManagedStatic.h"
 #include "llvm/Support/Mutex.h"
 #include "llvm/Support/Process.h"

 // it is needed and never destroying it.
 static ManagedStatic<std::string> LibSupportInfoOutputFilename;
 static std::string &getLibSupportInfoOutputFilename() {
+  if (InitLLVM::RunningInBuildozer()) {
+    static LLVM_THREAD_LOCAL std::string Local;
+    return Local;
+  }
   return *LibSupportInfoOutputFilename;
 }
 

lib/Support/Windows/DynamicLibrary.inc

     return &DynamicLibrary::Invalid;
   }
 
-  HMODULE Handle = LoadLibraryW(FileUnicode.data());
+  HMODULE Handle = ::LoadLibraryW(FileUnicode.data());
   if (Handle == NULL) {
     MakeErrMsg(Err, std::string(File) + ": Can't open");
     return &DynamicLibrary::Invalid;

lib/Support/Windows/Memory.inc

 
 } // namespace sys
 } // namespace llvm
+
+#include "C:\Users\aganea\Downloads\rpmalloc-1.4.0\rpmalloc\rpmalloc.c"
+
+typedef void *(__thiscall *MALLOC)(size_t);
+typedef void(__thiscall *FREE)(void *);
+typedef void *(__thiscall *CALLOC)(size_t, size_t);
+typedef void *(__thiscall *REALLOC)(void *, size_t);
+
+static uintptr_t AllocatorsTable[4]{
+    reinterpret_cast<uintptr_t>(&rpmalloc),
+    reinterpret_cast<uintptr_t>(&rpfree),
+    reinterpret_cast<uintptr_t>(&rpcalloc),
+    reinterpret_cast<uintptr_t>(&rprealloc),
+};
+
+__attribute__((used)) void HookAllocators(uintptr_t *Table) {
+  memcpy(AllocatorsTable, Table, sizeof(AllocatorsTable));
+}
+
+extern "C" {
+_ACRTIMP _CRTRESTRICT void *malloc(size_t size) {
+  return reinterpret_cast<MALLOC>(AllocatorsTable[0])(size);
+}
+
+_ACRTIMP void free(void *p) { reinterpret_cast<FREE>(AllocatorsTable[1])(p); }
+
+_ACRTIMP _CRTRESTRICT void *calloc(size_t n, size_t elem_size) {
+  return reinterpret_cast<CALLOC>(AllocatorsTable[2])(n, elem_size);
+}
+_ACRTIMP _CRTRESTRICT void *realloc(void *ptr, size_t size) {
+  return reinterpret_cast<REALLOC>(AllocatorsTable[3])(ptr, size);
+}
+}
+
+// Bypass CRT debug allocator
+#ifdef _DEBUG
+void *operator new(decltype(sizeof(0)) n) noexcept(false) { return malloc(n); }
+void __CRTDECL operator delete(void *const block) noexcept { free(block); }
+
+void *operator new[](std::size_t s) throw(std::bad_alloc) { return malloc(s); }
+void operator delete[](void *p) throw() { free(p); }
+#endif

lib/Support/Windows/Path.inc

   }
 }
 
+static LLVM_THREAD_LOCAL wchar_t CurrentDirectory[MAX_PATH * 2]{};
+static LLVM_THREAD_LOCAL unsigned CurrentDirectoryLen{};
+
+static unsigned InternalGetCurrentDirectory(unsigned len, wchar_t *data) {
+  if (!CurrentDirectoryLen)
+    return ::GetCurrentDirectoryW(len, data);
+
+  unsigned RealLen = CurrentDirectoryLen - 1; // to never reach zero
+  if (len == 0 || data == nullptr)
+    return RealLen;
+  if (len < RealLen)
+    return RealLen + 1;
+  wmemcpy_s(data, len, CurrentDirectory, RealLen);
+  data[RealLen] = 0;
+  return RealLen;
+}
+static bool InternalSetCurrentDirectory(const wchar_t *data) {
+  if (!CurrentDirectoryLen)
+    return !!::SetCurrentDirectoryW(data);
+
+  wcscpy_s(CurrentDirectory, MAX_PATH, data);
+  CurrentDirectoryLen = wcslen(CurrentDirectory) + 1; // +1 to never reach zero
+  CurrentDirectory[CurrentDirectoryLen - 1] = 0;
+  return true;
+}
+
+__attribute__((used))
+void SetCurrentDirectoryBackdoor(const wchar_t *data) {
+  wcscpy_s(CurrentDirectory, MAX_PATH, data);
+  unsigned Len = wcslen(CurrentDirectory);
+  if (CurrentDirectory[Len - 1] == L'\\') {
+    CurrentDirectory[Len - 1] = 0;
+    --Len;
+  }
+  CurrentDirectoryLen = Len + 1; // +1 to never reach zero
+}
+
+//#pragma comment(linker, "/INCLUDE:SetCurrentDirectoryBackdoor")
+
+static bool IsAbsoluteInternal(const wchar_t *data) {
+  if (!data || !data[0])
+    return false;
+  return (iswalpha(data[0]) && data[1] == L':') ||
+         (data[0] == L'\\' && data[0] == data[1]);
+}
+
+const wchar_t* PathAbsolutize(const wchar_t *data) {
+  if (!CurrentDirectoryLen || IsAbsoluteInternal(data))
+    return data;
+
+  CurrentDirectory[CurrentDirectoryLen - 1] = 0;
+  wchar_t *FullPath = CurrentDirectory + (CurrentDirectoryLen - 1);
+
+  if (data[0] != L'\\')
+    wcscat_s(FullPath, (MAX_PATH * 2) - CurrentDirectoryLen, L"\\");
+  wcscat_s(FullPath, (MAX_PATH * 2) - CurrentDirectoryLen, data);
+  // add an extra zero for remove_directories()
+  FullPath[wcslen(FullPath) + 1] = 0;
+  return CurrentDirectory;
+}
+
 namespace llvm {
 namespace sys  {
 namespace path {

   if (llvm::sys::path::is_absolute(Twine(Path8Str)))
     CurPathLen = 0; // No contribution from current_path needed.
   else {
-    CurPathLen = ::GetCurrentDirectoryW(0, NULL);
+    CurPathLen = InternalGetCurrentDirectory(0, NULL);
     if (CurPathLen == 0)
       return mapWindowsError(::GetLastError());
   }

 const file_t kInvalidFile = INVALID_HANDLE_VALUE;
 
 std::string getMainExecutable(const char *argv0, void *MainExecAddr) {
+  HMODULE hm;
+  GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS |
+                        GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
+                    (LPCWSTR)&getMainExecutable, &hm);
+
   SmallVector<wchar_t, MAX_PATH> PathName;
-  DWORD Size = ::GetModuleFileNameW(NULL, PathName.data(), PathName.capacity());
+  DWORD Size = ::GetModuleFileNameW(hm, PathName.data(), PathName.capacity());
 
   // A zero return value indicates a failure other than insufficient space.
   if (Size == 0)

 
   do {
     cur_path.reserve(len);
-    len = ::GetCurrentDirectoryW(cur_path.capacity(), cur_path.data());
+    len = InternalGetCurrentDirectory(cur_path.capacity(), cur_path.data());
 
     // A zero return value indicates a failure other than insufficient space.
     if (len == 0)

   if (std::error_code ec = widenPath(path, wide_path))
     return ec;
 
-  if (!::SetCurrentDirectoryW(wide_path.begin()))
+  if (!InternalSetCurrentDirectory(wide_path.begin()))
     return mapWindowsError(::GetLastError());
 
   return std::error_code();

   if (std::error_code ec = widenPath(path, path_utf16))
     return ec;
 
-  if (!::CreateDirectoryW(path_utf16.begin(), NULL)) {
+  if (!::CreateDirectoryW(PathAbsolutize(path_utf16.begin()), NULL)) {
     DWORD LastError = ::GetLastError();
     if (LastError != ERROR_ALREADY_EXISTS || !IgnoreExisting)
       return mapWindowsError(LastError);

   if (std::error_code ec = widenPath(to, wide_to))
     return ec;
 
-  if (!::CreateHardLinkW(wide_from.begin(), wide_to.begin(), NULL))
+  if (!::CreateHardLinkW(PathAbsolutize(wide_from.begin()),
+                         PathAbsolutize(wide_to.begin()), NULL))
     return mapWindowsError(::GetLastError());
 
   return std::error_code();

   // FILE_FLAG_BACKUP_SEMANTICS (which allows us to open directories), and
   // FILE_FLAG_OPEN_REPARSE_POINT (don't follow symlinks).
   ScopedFileHandle h(::CreateFileW(
-      c_str(path_utf16), DELETE,
+      PathAbsolutize(c_str(path_utf16)), DELETE,
       FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, NULL,
       OPEN_EXISTING,
       FILE_ATTRIBUTE_NORMAL | FILE_FLAG_BACKUP_SEMANTICS |

   size_t Len = 128;
   while (true) {
     VolumePath.resize(Len);
-    BOOL Success =
-        ::GetVolumePathNameW(Path.data(), VolumePath.data(), VolumePath.size());
-
+    BOOL Success = ::GetVolumePathNameW(PathAbsolutize(Path.data()),
+                                        VolumePath.data(), VolumePath.size());
     if (Success)
       break;
 

   if (auto EC = widenPath(To, ToWide))
     return EC;
 
+  const wchar_t* AbsToWide = PathAbsolutize(ToWide.begin());
+  unsigned AbsToWideLen = wcslen(AbsToWide);
+
   std::vector<char> RenameInfoBuf(sizeof(FILE_RENAME_INFO) - sizeof(wchar_t) +
-                                  (ToWide.size() * sizeof(wchar_t)));
+                                  (AbsToWideLen * sizeof(wchar_t)));
   FILE_RENAME_INFO &RenameInfo =
       *reinterpret_cast<FILE_RENAME_INFO *>(RenameInfoBuf.data());
   RenameInfo.ReplaceIfExists = ReplaceIfExists;
   RenameInfo.RootDirectory = 0;
-  RenameInfo.FileNameLength = ToWide.size() * sizeof(wchar_t);
-  std::copy(ToWide.begin(), ToWide.end(), &RenameInfo.FileName[0]);
+  RenameInfo.FileNameLength = AbsToWideLen * sizeof(wchar_t);
+  std::copy(AbsToWide, AbsToWide + AbsToWideLen, &RenameInfo.FileName[0]);
 
   SetLastError(ERROR_SUCCESS);
   if (!SetFileInformationByHandle(FromHandle, FileRenameInfo, &RenameInfo,

       SmallVector<wchar_t, MAX_PATH> WideFrom;
       if (std::error_code EC2 = realPathFromHandle(FromHandle, WideFrom))
         return EC2;
-      if (::MoveFileExW(WideFrom.begin(), WideTo.begin(),
-                        MOVEFILE_REPLACE_EXISTING))
+      if (::MoveFileExW(PathAbsolutize(WideFrom.begin()),
+                        PathAbsolutize(WideTo.begin()), MOVEFILE_REPLACE_EXISTING))
         return std::error_code();
       return mapWindowsError(GetLastError());
     }

     // to arrange for the destination file to be deleted when the other process
     // closes it.
     ScopedFileHandle ToHandle(
-        ::CreateFileW(WideTo.begin(), GENERIC_READ | DELETE,
+        ::CreateFileW(PathAbsolutize(WideTo.begin()), GENERIC_READ | DELETE,
                       FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
                       NULL, OPEN_EXISTING,
                       FILE_ATTRIBUTE_NORMAL | FILE_FLAG_DELETE_ON_CLOSE, NULL));

           // might have caused the permission denied error. If that was the
           // case, we don't need to move it ourselves.
           ScopedFileHandle ToHandle2(::CreateFileW(
-              WideTo.begin(), 0,
+              PathAbsolutize(WideTo.begin()), 0,
               FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, NULL,
               OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL));
           if (!ToHandle2) {

     if (Retry != 0)
       ::Sleep(10);
     FromHandle =
-        ::CreateFileW(WideFrom.begin(), GENERIC_READ | DELETE,
+        ::CreateFileW(PathAbsolutize(WideFrom.begin()), GENERIC_READ | DELETE,
                       FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
                       NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
     if (FromHandle)

   if (std::error_code EC = widenPath(Path, PathUtf16))
     return EC;
 
-  DWORD Attributes = ::GetFileAttributesW(PathUtf16.begin());
+  DWORD Attributes = ::GetFileAttributesW(PathAbsolutize(PathUtf16.begin()));
 
   if (Attributes == INVALID_FILE_ATTRIBUTES) {
     // See if the file didn't actually exist.

   if (std::error_code ec = widenPath(path8, path_utf16))
     return ec;
 
-  DWORD attr = ::GetFileAttributesW(path_utf16.begin());
+  DWORD attr = ::GetFileAttributesW(PathAbsolutize(path_utf16.begin()));
   if (attr == INVALID_FILE_ATTRIBUTES)
     return getStatus(INVALID_HANDLE_VALUE, result);
 

     Flags |= FILE_FLAG_OPEN_REPARSE_POINT;
 
   ScopedFileHandle h(
-      ::CreateFileW(path_utf16.begin(), 0, // Attributes only.
+      ::CreateFileW(PathAbsolutize(path_utf16.begin()), 0, // Attributes only.
                     FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE,
                     NULL, OPEN_EXISTING, Flags, 0));
   if (!h)

   if (std::error_code EC = widenPath(Path, PathUTF16))
     return EC;
 
-  DWORD Attributes = ::GetFileAttributesW(PathUTF16.begin());
+  DWORD Attributes = ::GetFileAttributesW(PathAbsolutize(PathUTF16.begin()));
   if (Attributes == INVALID_FILE_ATTRIBUTES)
     return mapWindowsError(GetLastError());
 

     Attributes &= ~FILE_ATTRIBUTE_NORMAL;
   }
 
-  if (!::SetFileAttributesW(PathUTF16.begin(), Attributes))
+  if (!::SetFileAttributesW(PathAbsolutize(PathUTF16.begin()), Attributes))
     return mapWindowsError(GetLastError());
 
   return std::error_code();

   //  Get the first directory entry.
   WIN32_FIND_DATAW FirstFind;
   ScopedFindHandle FindHandle(::FindFirstFileExW(
-      c_str(PathUTF16), FindExInfoBasic, &FirstFind, FindExSearchNameMatch,
-      NULL, FIND_FIRST_EX_LARGE_FETCH));
+      PathAbsolutize(c_str(PathUTF16)), FindExInfoBasic, &FirstFind,
+      FindExSearchNameMatch, NULL, FIND_FIRST_EX_LARGE_FETCH));
   if (!FindHandle)
     return mapWindowsError(::GetLastError());
 

   SA.bInheritHandle = Inherit;
 
   HANDLE H =
-      ::CreateFileW(PathUTF16.begin(), Access,
+      ::CreateFileW(PathAbsolutize(PathUTF16.begin()), Access,
                     FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, &SA,
                     Disp, Flags, NULL);
   if (H == INVALID_HANDLE_VALUE) {

 
   SHFILEOPSTRUCTW shfos = {};
   shfos.wFunc = FO_DELETE;
-  shfos.pFrom = Path16.data();
+  shfos.pFrom = PathAbsolutize(Path16.data());
   shfos.fFlags = FOF_NO_UI;
 
   int result = ::SHFileOperationW(&shfos);

lib/Support/Windows/Process.inc

 #include <io.h>
 #include <psapi.h>
 #include <shellapi.h>
+#include <winternl.h>
+#include <dbghelp.h>
 
 #if !defined(__MINGW32__)
  #pragma comment(lib, "psapi.lib")

 //===          and must not be UNIX code
 //===----------------------------------------------------------------------===//
 
+const wchar_t* PathAbsolutize(const wchar_t *data);
+
 #ifdef __MINGW32__
 // This ban should be lifted when MinGW 1.0+ has defined this value.
 #  define _HEAPOK (-2)

   // directory portion of the file path.  For example, it doesn't handle
   // "*\foo.c" nor "s?c\bar.cpp".
   WIN32_FIND_DATAW FileData;
-  HANDLE FindHandle = FindFirstFileW(ArgW.data(), &FileData);
+  HANDLE FindHandle = FindFirstFileW(PathAbsolutize(ArgW.data()), &FileData);
   if (FindHandle == INVALID_HANDLE_VALUE) {
     Args.push_back(Arg.data());
     return EC;

 std::error_code
 windows::GetCommandLineArguments(SmallVectorImpl<const char *> &Args,
                                  BumpPtrAllocator &Alloc) {
-  const wchar_t *CmdW = GetCommandLineW();
+  const wchar_t *CmdW = ::GetCommandLineW();
   assert(CmdW);
   std::error_code EC;
   SmallString<MAX_PATH> Cmd;

   // Windows 8 is version 6.2, service pack 0.
   return GetWindowsOSVersion() >= llvm::VersionTuple(6, 2, 0, 0);
 }
+
+[[noreturn]] void sys::Process::ExitCurrentProcess(int code) {
+  if (InitLLVM::RunningInBuildozer())
+    RaiseException(0xE0000000 | code, 0, 0, NULL);
+  exit(code);
+}
+
+extern LLVM_THREAD_LOCAL HANDLE CurrentHandle; // in InitLLVM.cpp
+
+bool sys::Process::IsInTLS(const void *Data) {
+  DWORD Size = 0;
+  PIMAGE_TLS_DIRECTORY64 TLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          CurrentHandle, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &Size);
+  if (!TLS)
+    return false;
+
+  PTEB TEB = NtCurrentTeb();
+
+  uint64_t **ProcessTLS =
+      (uint64_t **)TEB->Reserved1[11]; // ThreadLocalStoragePointer
+
+  char *ThreadTLS =
+      (char *)*(ProcessTLS + *(unsigned int *)TLS->AddressOfIndex);
+  unsigned TLSLen = TLS->EndAddressOfRawData - TLS->StartAddressOfRawData;
+
+  return Data >= ThreadTLS && Data < (ThreadTLS + TLSLen);
+}
+
+unsigned sys::Process::GetOffsetInTLS(const void *Data) {
+  DWORD Size = 0;
+  PIMAGE_TLS_DIRECTORY64 TLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          CurrentHandle, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &Size);
+  if (!TLS)
+    return 0;
+
+  PTEB TEB = NtCurrentTeb();
+
+  uint64_t **ProcessTLS =
+      (uint64_t **)TEB->Reserved1[11]; // ThreadLocalStoragePointer
+
+  char *ThreadTLS =
+      (char *)*(ProcessTLS + *(unsigned int *)TLS->AddressOfIndex);
+  unsigned TLSLen = TLS->EndAddressOfRawData - TLS->StartAddressOfRawData;
+
+  if (Data >= ThreadTLS && Data < (ThreadTLS + TLSLen))
+    return ((const char *)Data - ThreadTLS) + 1;
+  return 0;
+}
+
+void *sys::Process::RemapIntoTLS(void *Data, unsigned TLSOffset) {
+  if (!TLSOffset)
+    return Data;
+
+  DWORD Size = 0;
+  PIMAGE_TLS_DIRECTORY64 TLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          CurrentHandle, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &Size);
+  if (!TLS)
+    return Data;
+
+  PTEB TEB = NtCurrentTeb();
+
+  uint64_t **ProcessTLS =
+      (uint64_t **)TEB->Reserved1[11]; // ThreadLocalStoragePointer
+
+  char *ThreadTLS =
+      (char *)*(ProcessTLS + *(unsigned int *)TLS->AddressOfIndex);
+  unsigned TLSLen = TLS->EndAddressOfRawData - TLS->StartAddressOfRawData;
+  assert(TLSOffset <= TLSLen);
+  (void)TLSLen;
+
+  return ThreadTLS + (TLSOffset - 1);
+}

lib/Support/Windows/Program.inc

 //===          and must not be UNIX code
 //===----------------------------------------------------------------------===//
 
+const wchar_t* PathAbsolutize(const wchar_t *data);
+
 namespace llvm {
 
 ProcessInfo::ProcessInfo() : Pid(0), Process(0), ReturnCode(0) {}

               windows::UTF8ToUTF16(Twine(Name + Ext).str(), U16NameExt))
         return EC;
 
-      Len = ::SearchPathW(Path, c_str(U16NameExt), nullptr,
-                          U16Result.capacity(), U16Result.data(), nullptr);
+      Len = ::SearchPathW(Path ? PathAbsolutize(Path) : nullptr,
+                          c_str(U16NameExt), nullptr, U16Result.capacity(),
+                          U16Result.data(), nullptr);
     } while (Len > U16Result.capacity());
 
     if (Len != 0)

     if (path::widenPath(fname, fnameUnicode))
       return INVALID_HANDLE_VALUE;
   }
-  h = CreateFileW(fnameUnicode.data(), fd ? GENERIC_WRITE : GENERIC_READ,
-                  FILE_SHARE_READ, &sa, fd == 0 ? OPEN_EXISTING : CREATE_ALWAYS,
-                  FILE_ATTRIBUTE_NORMAL, NULL);
+  h = ::CreateFileW(PathAbsolutize(fnameUnicode.data()),
+                    fd ? GENERIC_WRITE : GENERIC_READ, FILE_SHARE_READ, &sa,
+                    fd == 0 ? OPEN_EXISTING : CREATE_ALWAYS,
+                    FILE_ATTRIBUTE_NORMAL, NULL);
   if (h == INVALID_HANDLE_VALUE) {
     MakeErrMsg(ErrMsg, fname + ": Can't open file for " +
         (fd ? "input" : "output"));

   // Assign the process to a job if a memory limit is defined.
   ScopedJobHandle hJob;
   if (MemoryLimit != 0) {
-    hJob = CreateJobObjectW(0, 0);
+    hJob = ::CreateJobObjectW(0, 0);
     bool success = false;
     if (hJob) {
       JOBOBJECT_EXTENDED_LIMIT_INFORMATION jeli;

lib/Support/Windows/Signals.inc

 using namespace llvm;
 
 // Forward declare.
-static LONG WINAPI LLVMUnhandledExceptionFilter(LPEXCEPTION_POINTERS ep);
+LONG WINAPI LLVMUnhandledExceptionFilter(LPEXCEPTION_POINTERS ep);
 static BOOL WINAPI LLVMConsoleCtrlHandler(DWORD dwCtrlType);
 
 // The function to call if ctrl-c is pressed.

 extern "C" VOID WINAPI RtlCaptureContext(PCONTEXT ContextRecord);
 #endif
 
-void llvm::sys::PrintStackTrace(raw_ostream &OS) {
-  STACKFRAME64 StackFrame = {};
-  CONTEXT Context = {};
-  ::RtlCaptureContext(&Context);
+static void LocalPrintStackTrace(raw_ostream &OS, PCONTEXT C) {
+  STACKFRAME64 StackFrame{};
+  CONTEXT Context{};
+  if (!C) {
+    ::RtlCaptureContext(&Context);
+    C = &Context;
+  }
 #if defined(_M_X64)
   StackFrame.AddrPC.Offset = Context.Rip;
   StackFrame.AddrStack.Offset = Context.Rsp;

   StackFrame.AddrStack.Mode = AddrModeFlat;
   StackFrame.AddrFrame.Mode = AddrModeFlat;
   PrintStackTraceForThread(OS, GetCurrentProcess(), GetCurrentThread(),
-                           StackFrame, &Context);
+                           StackFrame, C);
 }
 
+void llvm::sys::PrintStackTrace(raw_ostream &OS) {
+  LocalPrintStackTrace(OS, nullptr);
+}
 
 void llvm::sys::SetInterruptFunction(void (*IF)()) {
   RegisterHandler();

   return std::error_code();
 }
 
-static LONG WINAPI LLVMUnhandledExceptionFilter(LPEXCEPTION_POINTERS ep) {
+LONG WINAPI LLVMUnhandledExceptionFilter(LPEXCEPTION_POINTERS ep) {
   Cleanup();
 
   // We'll automatically write a Minidump file here to help diagnose

                    << "\n";
   }
 
-  // Initialize the STACKFRAME structure.
-  STACKFRAME64 StackFrame = {};
+  LocalPrintStackTrace(llvm::errs(), ep->ContextRecord);
 
-#if defined(_M_X64)
-  StackFrame.AddrPC.Offset = ep->ContextRecord->Rip;
-  StackFrame.AddrPC.Mode = AddrModeFlat;
-  StackFrame.AddrStack.Offset = ep->ContextRecord->Rsp;
-  StackFrame.AddrStack.Mode = AddrModeFlat;
-  StackFrame.AddrFrame.Offset = ep->ContextRecord->Rbp;
-  StackFrame.AddrFrame.Mode = AddrModeFlat;
-#elif defined(_M_IX86)
-  StackFrame.AddrPC.Offset = ep->ContextRecord->Eip;
-  StackFrame.AddrPC.Mode = AddrModeFlat;
-  StackFrame.AddrStack.Offset = ep->ContextRecord->Esp;
-  StackFrame.AddrStack.Mode = AddrModeFlat;
-  StackFrame.AddrFrame.Offset = ep->ContextRecord->Ebp;
-  StackFrame.AddrFrame.Mode = AddrModeFlat;
-#elif defined(_M_ARM64) || defined(_M_ARM)
-  StackFrame.AddrPC.Offset = ep->ContextRecord->Pc;
-  StackFrame.AddrPC.Mode = AddrModeFlat;
-  StackFrame.AddrStack.Offset = ep->ContextRecord->Sp;
-  StackFrame.AddrStack.Mode = AddrModeFlat;
-#if defined(_M_ARM64)
-  StackFrame.AddrFrame.Offset = ep->ContextRecord->Fp;
-#else
-  StackFrame.AddrFrame.Offset = ep->ContextRecord->R11;
-#endif
-  StackFrame.AddrFrame.Mode = AddrModeFlat;
-#endif
-
-  HANDLE hProcess = GetCurrentProcess();
-  HANDLE hThread = GetCurrentThread();
-  PrintStackTraceForThread(llvm::errs(), hProcess, hThread, StackFrame,
-                           ep->ContextRecord);
-
-  _exit(ep->ExceptionRecord->ExceptionCode);
+  return EXCEPTION_EXECUTE_HANDLER;
 }
 
 static BOOL WINAPI LLVMConsoleCtrlHandler(DWORD dwCtrlType) {

lib/Support/raw_ostream.cpp

 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Format.h"
 #include "llvm/Support/FormatVariadic.h"
+#include "llvm/Support/InitLLVM.h"
 #include "llvm/Support/MathExtras.h"
 #include "llvm/Support/NativeFormatting.h"
 #include "llvm/Support/Process.h"

 //  outs(), errs(), nulls()
 //===----------------------------------------------------------------------===//
 
+struct LocalStd {
+  bool Initialized;
+  std::string Str;
+  raw_string_ostream S{Str};
+};
+static LLVM_THREAD_LOCAL char LocalStream[2][sizeof(LocalStd)]{};
+
+#define RETURN_LOCAL_STREAM(I)                                                 \
+  if (InitLLVM::RunningInBuildozer()) {                                        \
+    LocalStd &S = (LocalStd &)LocalStream[I];                                  \
+    if (!S.Initialized) {                                                      \
+      new (LocalStream[I]) LocalStd;                                           \
+      S.Initialized = true;                                                    \
+    }                                                                          \
+    return S.S;                                                                \
+  }
+
+__attribute__((used)) void CaptureLocalOutput(llvm::StringRef &StdErr,
+                                              llvm::StringRef &StdOut) {
+  auto& Outs = (raw_string_ostream &)llvm::outs();
+  StdOut = Outs.str();
+  auto& Err = (raw_string_ostream &)llvm::errs();
+  StdErr = Err.str();
+}
+
 /// outs() - This returns a reference to a raw_ostream for standard output.
 /// Use it like: outs() << "foo" << "bar";
 raw_ostream &llvm::outs() {
+  RETURN_LOCAL_STREAM(0)
   // Set buffer settings to model stdout behavior.
   std::error_code EC;
   static raw_fd_ostream S("-", EC, sys::fs::OF_None);

 /// errs() - This returns a reference to a raw_ostream for standard error.
 /// Use it like: errs() << "foo" << "bar";
 raw_ostream &llvm::errs() {
+  RETURN_LOCAL_STREAM(1)
   // Set standard error to be unbuffered by default.
   static raw_fd_ostream S(STDERR_FILENO, false, true);
   return S;

lib/TableGen/Error.cpp

 
 #include "llvm/TableGen/Error.h"
 #include "llvm/ADT/Twine.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Support/Signals.h"
 #include "llvm/Support/WithColor.h"
 #include "llvm/Support/raw_ostream.h"

   PrintError(Msg);
   // The following call runs the file cleanup handlers.
   sys::RunInterruptHandlers();
-  std::exit(1);
+  sys::Process::ExitCurrentProcess(1);
 }
 
 void PrintFatalError(ArrayRef<SMLoc> ErrorLoc, const Twine &Msg) {

   PrintError(ErrorLoc, Msg);
   // The following call runs the file cleanup handlers.
   sys::RunInterruptHandlers();
-  std::exit(1);
+  sys::Process::ExitCurrentProcess(1);
 }
 
 } // end namespace llvm

lib/Target/AMDGPU/AMDGPULibFunc.cpp

   return false;
 }
 
-static inline bool isDigit(char c) { return c >= '0' && c <= '9'; }
-
 static int eatNumber(StringRef& s) {
   size_t const savedSize = s.size();
   int n = 0;

lib/Target/AMDGPU/AMDGPUTargetMachine.h

   StringRef getFeatureString(const Function &F) const;
 
 public:
-  static bool EnableLateStructurizeCFG;
-  static bool EnableFunctionCalls;
+  static LLVM_THREAD_LOCAL bool EnableLateStructurizeCFG;
+  static LLVM_THREAD_LOCAL bool EnableFunctionCalls;
 
   AMDGPUTargetMachine(const Target &T, const Triple &TT, StringRef CPU,
                       StringRef FS, TargetOptions Options,

lib/Target/AMDGPU/AMDGPUTargetMachine.cpp

   initAsmInfo();
 }
 
-bool AMDGPUTargetMachine::EnableLateStructurizeCFG = false;
-bool AMDGPUTargetMachine::EnableFunctionCalls = false;
+bool LLVM_THREAD_LOCAL AMDGPUTargetMachine::EnableLateStructurizeCFG = false;
+bool LLVM_THREAD_LOCAL AMDGPUTargetMachine::EnableFunctionCalls = false;
 
 AMDGPUTargetMachine::~AMDGPUTargetMachine() = default;
 

lib/Target/BPF/BPFAbstractMemberAccess.cpp

   LLVM_DEBUG(dbgs() << "********** Abstract Member Accesses **********\n");
 
   // Bail out if no debug info.
-  if (empty(M.debug_compile_units()))
+  if (M.debug_compile_units().empty())
     return false;
 
   return doTransformation(M);

lib/Target/BPF/BPFAsmPrinter.cpp

   AsmPrinter::doInitialization(M);
 
   // Only emit BTF when debuginfo available.
-  if (MAI->doesSupportDebugInformation() && !empty(M.debug_compile_units())) {
+  if (MAI->doesSupportDebugInformation() && !M.debug_compile_units().empty()) {
     BTF = new BTFDebug(this);
     Handlers.push_back(HandlerInfo(std::unique_ptr<BTFDebug>(BTF), "emit",
                                    "Debug Info Emission", "BTF",

lib/Target/TargetMachineC.cpp

 }
 
 LLVMTargetRef LLVMGetFirstTarget() {
-  if (TargetRegistry::targets().begin() == TargetRegistry::targets().end()) {
+  if (TargetRegistry::targets().empty()) {
     return nullptr;
   }
 

lib/ToolDrivers/llvm-lib/LibDriver.cpp

     return;
   handleAllErrors(std::move(E), [&](const llvm::ErrorInfoBase &EIB) {
     llvm::errs() << "error opening '" << File << "': " << EIB.message() << '\n';
-    exit(1);
+    sys::Process::ExitCurrentProcess(1);
   });
 }
 

lib/Transforms/IPO/PartialInlining.cpp

   if (PSI->isFunctionEntryCold(F))
     return {false, nullptr};
 
-  if (empty(F->users()))
+  if (F->users().empty())
     return {false, nullptr};
 
   OptimizationRemarkEmitter ORE(F);

     return false;
   }
 
-  assert(empty(Cloner.OrigFunc->users()) &&
+  assert(Cloner.OrigFunc->users().empty() &&
          "F's users should all be replaced!");
 
   std::vector<User *> Users(Cloner.ClonedFunc->user_begin(),

lib/Transforms/Instrumentation/ControlHeightReduction.cpp

 #include "llvm/IR/MDBuilder.h"
 #include "llvm/Support/BranchProbability.h"
 #include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/Process.h"
 #include "llvm/Transforms/Utils.h"
 #include "llvm/Transforms/Utils/BasicBlockUtils.h"
 #include "llvm/Transforms/Utils/Cloning.h"

     auto FileOrErr = MemoryBuffer::getFile(CHRFunctionList);
     if (!FileOrErr) {
       errs() << "Error: Couldn't read the chr-function-list file " << CHRFunctionList << "\n";
-      std::exit(1);
+      sys::Process::ExitCurrentProcess(1);
     }
     StringRef Buf = FileOrErr->get()->getBuffer();
     SmallVector<StringRef, 0> Lines;

lib/Transforms/Instrumentation/GCOVProfiling.cpp

     for (auto &F : M->functions()) {
       DISubprogram *SP = F.getSubprogram();
       if (!SP) continue;
-      if (!functionHasLines(F) || !isFunctionInstrumented(F))
+      if (SP->isArtificial() || !functionHasLines(F) ||
+          !isFunctionInstrumented(F))
         continue;
       // TODO: Functions using scope-based EH are currently not supported.
       if (isUsingScopeBasedEH(F)) continue;

     for (auto &F : M->functions()) {
       DISubprogram *SP = F.getSubprogram();
       if (!SP) continue;
-      if (!functionHasLines(F) || !isFunctionInstrumented(F))
+      if (SP->isArtificial() || !functionHasLines(F) ||
+          !isFunctionInstrumented(F))
         continue;
       // TODO: Functions using scope-based EH are currently not supported.
       if (isUsingScopeBasedEH(F)) continue;

lib/Transforms/Scalar/NewGVN.cpp

     return true;
   });
   // If we are left with no operands, it's dead.
-  if (empty(Filtered)) {
+  if (Filtered.empty()) {
     // If it has undef at this point, it means there are no-non-undef arguments,
     // and thus, the value of the phi node must be undef.
     if (HasUndef) {

lib/Transforms/Scalar/RewriteStatepointsForGC.cpp

                            cl::init(6));
 
 #ifdef EXPENSIVE_CHECKS
-static bool ClobberNonLive = true;
+static LLVM_THREAD_LOCAL bool ClobberNonLive = true;
 #else
-static bool ClobberNonLive = false;
+static LLVM_THREAD_LOCAL bool ClobberNonLive = false;
 #endif
 
 static cl::opt<bool, true> ClobberNonLiveOverride("rs4gc-clobber-non-live",

lib/Transforms/Utils/LCSSA.cpp

 STATISTIC(NumLCSSA, "Number of live out of a loop variables");
 
 #ifdef EXPENSIVE_CHECKS
-static bool VerifyLoopLCSSA = true;
+static LLVM_THREAD_LOCAL bool VerifyLoopLCSSA = true;
 #else
-static bool VerifyLoopLCSSA = false;
+static LLVM_THREAD_LOCAL bool VerifyLoopLCSSA = false;
 #endif
 static cl::opt<bool, true>
     VerifyLoopLCSSAFlag("verify-loop-lcssa", cl::location(VerifyLoopLCSSA),

lib/Transforms/Utils/PredicateInfo.cpp

     if (isa<PredicateWithEdge>(ValInfo)) {
       IRBuilder<> B(getBranchTerminator(ValInfo));
       Function *IF = getCopyDeclaration(F.getParent(), Op->getType());
-      if (empty(IF->users()))
+      if (IF->users().empty())
         CreatedDeclarations.insert(IF);
       CallInst *PIC =
           B.CreateCall(IF, Op, Op->getName() + "." + Twine(Counter++));

              "Should not have gotten here without it being an assume");
       IRBuilder<> B(PAssume->AssumeInst);
       Function *IF = getCopyDeclaration(F.getParent(), Op->getType());
-      if (empty(IF->users()))
+      if (IF->users().empty())
         CreatedDeclarations.insert(IF);
       CallInst *PIC = B.CreateCall(IF, Op);
       PredicateMap.insert({PIC, ValInfo});

lib/Transforms/Utils/SimplifyCFG.cpp

 
   // Figure out the corresponding result for each case value and phi node in the
   // common destination, as well as the min and max case values.
-  assert(!empty(SI->cases()));
+  assert(!SI->cases().empty());
   SwitchInst::CaseIt CI = SI->case_begin();
   ConstantInt *MinCaseVal = CI->getCaseValue();
   ConstantInt *MaxCaseVal = CI->getCaseValue();

tools/bugpoint/BugDriver.h

 
 class CC;
 
-extern bool DisableSimplifyCFG;
+extern LLVM_THREAD_LOCAL bool DisableSimplifyCFG;
 
 /// BugpointIsInterrupted - Set to true when the user presses ctrl-c.
 ///

tools/bugpoint/ExtractFunction.cpp

 #define DEBUG_TYPE "bugpoint"
 
 namespace llvm {
-bool DisableSimplifyCFG = false;
+static LLVM_THREAD_LOCAL bool DisableSimplifyCFG = false;
 extern cl::opt<std::string> OutputPrefix;
 } // End llvm namespace
 

tools/buildozer/CMakeLists.txt

+set(LLVM_LINK_COMPONENTS
+  Core
+  Support
+  )
+
+add_llvm_tool(buildozer
+  buildozer-main.cpp
+  LoadEXE.cpp
+  CompDB.cpp
+
+  DEPENDS
+  intrinsics_gen
+  )

tools/buildozer/CompDB.h

+//===---------- Buildozer - A Lightweight LLVM Build System ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/StringSaver.h"
+#include <vector>
+
+namespace buildozer {
+struct WorkUnit {
+  llvm::StringRef Directory;
+  llvm::StringRef File;
+  llvm::StringRef Output;
+  llvm::SmallVector<const char *, 0> Arguments;
+};
+typedef std::vector<WorkUnit> CompilationSeq;
+
+struct CompilationDB {
+  CompilationSeq Build;
+  llvm::BumpPtrAllocator BumpAlloc;
+  llvm::StringSaver Saver{BumpAlloc};
+};
+void ReadCompilationDatabase(llvm::StringRef FilePath, CompilationDB &DB);
+} // namespace buildozer

tools/buildozer/CompDB.cpp

+//===- buildozer-main.cpp - A Lightweight LLVM Build System ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "CompDB.h"
+#include "llvm/Support/MemoryBuffer.h"
+#include "llvm/Support/JSON.h"
+
+namespace buildozer {
+void ReadCompilationDatabase(llvm::StringRef FilePath, CompilationDB &DB) {
+  auto F = llvm::MemoryBuffer::getFile(FilePath);
+  if (auto E = F.getError())
+    return; // errorCodeToError(err);
+
+  auto Doc = llvm::json::parse((*F)->getBuffer());
+  if (!Doc) {
+    consumeError(Doc.takeError());
+    return;
+  }
+
+  llvm::json::Array *All = Doc->getAsArray();
+  for (llvm::json::Value &E : *All) {
+    llvm::json::Object *O = E.getAsObject();
+    DB.Build.emplace_back();
+    WorkUnit &W = DB.Build.back();
+
+    W.Directory = DB.Saver.save(*O->getString("directory")).data();
+    auto File = O->getString("file");
+    if (File)
+      W.File = DB.Saver.save(*File).data();
+    W.Output = DB.Saver.save(*O->getString("output")).data();
+
+    llvm::json::Array *Args = O->getArray("arguments");
+    for (llvm::json::Value &Arg : *Args) {
+      llvm::StringRef S = *Arg.getAsString();
+      W.Arguments.push_back(DB.Saver.save(S).data());
+    }
+  }
+}
+} // namespace buildozer

tools/buildozer/LLVMBuild.txt

+;===- ./tools/buildozer/LLVMBuild.txt --------------------------*- Conf -*--===;
+;
+; Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+; See https://llvm.org/LICENSE.txt for license information.
+; SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+;
+;===------------------------------------------------------------------------===;
+;
+; This is an LLVMBuild description file for the components in this subdirectory.
+;
+; For more information on the LLVMBuild system, please see:
+;
+;   http://llvm.org/docs/LLVMBuild.html
+;
+;===------------------------------------------------------------------------===;
+
+[component_0]
+type = Tool
+name = Buildozer
+parent = Tools
+required_libraries = Core Support

tools/buildozer/LoadEXE.h

+//===---------- Buildozer - A Lightweight LLVM Build System ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/StringRef.h"
+
+namespace buildozer {
+// returns a ModuleIndex
+int ImportEXE(llvm::StringRef EXE);
+void Launch(int ModuleIndex, llvm::StringRef CurDir,
+            const llvm::SmallVectorImpl<const char *> &Argv);
+void Terminate(int ModuleIndex, std::string &StdErr, std::string &StdOut);
+void TerminateAll();
+} // namespace buildozer

tools/buildozer/LoadEXE.cpp

+//===- buildozer-main.cpp - A Lightweight LLVM Build System ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/ConvertUTF.h"
+#include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/StringSaver.h"
+
+#include <windows.h>
+#include <dbghelp.h>
+#include <winnt.h>
+#include <winternl.h>
+
+#include <thread>
+#include <vector>
+
+#pragma comment(lib, "dbghelp.lib")
+
+struct ExecModule {
+  llvm::StringRef FullPath;
+  llvm::StringRef Name;
+  HANDLE Handle{};
+  void *TLSInitBuffer{}; // .tls buffer after CRT initialization
+  HANDLE ExitEvent{};
+  HANDLE TLSThreadHandle{};
+
+  typedef int(__thiscall *MAIN)(int, const char **);
+  MAIN Main;
+
+  typedef void(__thiscall *SETCURDIR)(const wchar_t *);
+  SETCURDIR SD;
+
+  typedef void(__thiscall *SETMODULE)(void * /*ModuleHandle*/);
+  SETMODULE SMod;
+
+  typedef void(WINAPI *DYNTLSINIT)(PVOID, DWORD /*dwReason*/, LPVOID);
+  DYNTLSINIT InitTLS;
+
+  typedef void(WINAPI *DYNTLSDTOR)(HANDLE, DWORD /*dwReason*/, LPVOID);
+  DYNTLSDTOR DestroyTLS;
+
+  typedef void(__thiscall *CAPTUREOUT)(llvm::StringRef &StdErr,
+                                       llvm::StringRef &StdOut);
+  CAPTUREOUT CaptureOutput;
+
+  static ExecModule &get(int ModuleIndex);
+};
+
+std::vector<ExecModule> LoadedModules;
+llvm::BumpPtrAllocator Alloc;
+llvm::StringSaver Saver(Alloc);
+
+ExecModule &ExecModule::get(int ModuleIndex) {
+  assert(ModuleIndex > 0 && ModuleIndex <= (int)LoadedModules.size());
+  ExecModule &M = LoadedModules[ModuleIndex - 1];
+  return M;
+}
+
+// ----------------------------------------------------------------------------
+
+typedef __kernel_entry NTSTATUS(__stdcall *NTQUERYINFORMATIONPROCESS)(
+    IN HANDLE ProcessHandle, IN PROCESSINFOCLASS ProcessInformationClass,
+    OUT PVOID ProcessInformation, IN ULONG ProcessInformationLength,
+    OUT PULONG ReturnLength);
+
+typedef NTSTATUS(__thiscall *LDRPHANDLETLSDATA)(
+    __inout PLDR_DATA_TABLE_ENTRY ModuleEntry);
+
+struct IATRESULTS {
+  enum class FAILUREREASON {
+    SUCCESS = 0,
+    OTHER = 1,
+    NOTFOUND = 2,
+    CANNOTPATCH = 3,
+  };
+  struct FUNCTIONINFO {
+    std::string name;
+    size_t ord = 0;
+    FAILUREREASON f = FAILUREREASON::SUCCESS;
+  };
+  struct MODULEINFO {
+    std::string name;
+    HINSTANCE handle = 0;
+    FAILUREREASON f = FAILUREREASON::SUCCESS;
+    std::vector<FUNCTIONINFO> functions;
+  };
+
+  std::vector<MODULEINFO> modules;
+};
+
+static bool PatchProcessBytes(void *Target, llvm::StringRef Data) {
+  HANDLE H = GetCurrentProcess();
+  DWORD R =
+      WriteProcessMemory(H, (LPVOID *)Target, Data.data(), Data.size(), NULL);
+  if (R)
+    return true;
+  if (GetLastError() != ERROR_NOACCESS)
+    return false;
+
+  DWORD dwOldProtect;
+  if (!VirtualProtect((LPVOID)Target, Data.size(), PAGE_WRITECOPY,
+                      &dwOldProtect))
+    return false;
+
+  R = WriteProcessMemory(H, (LPVOID *)Target, Data.data(), Data.size(), NULL);
+
+  if (!VirtualProtect((LPVOID)Target, Data.size(), dwOldProtect, &dwOldProtect))
+    return false;
+
+  return R;
+}
+
+static void RemapIAT(HINSTANCE H, IATRESULTS *res = nullptr) {
+  // Get IAT size
+  DWORD ulsize = 0;
+  PIMAGE_IMPORT_DESCRIPTOR pImportDesc =
+      (PIMAGE_IMPORT_DESCRIPTOR)ImageDirectoryEntryToData(
+          H, TRUE, IMAGE_DIRECTORY_ENTRY_IMPORT, &ulsize);
+  if (!pImportDesc)
+    return;
+
+  // Loop names
+  for (; pImportDesc->Name; pImportDesc++) {
+    PSTR pszModName = (PSTR)((PBYTE)H + pImportDesc->Name);
+    if (!pszModName)
+      break;
+
+    IATRESULTS::MODULEINFO m;
+
+    m.name = pszModName;
+
+    HINSTANCE hImportDLL = LoadLibraryA(pszModName);
+    if (!hImportDLL) {
+      m.f = IATRESULTS::FAILUREREASON::NOTFOUND;
+      if (res)
+        res->modules.push_back(m);
+      continue;
+    }
+    m.handle = hImportDLL;
+    m.f = IATRESULTS::FAILUREREASON::SUCCESS;
+
+    // Get caller's import address table (IAT) for the callee's functions
+    PIMAGE_THUNK_DATA pThunk =
+        (PIMAGE_THUNK_DATA)((PBYTE)H + pImportDesc->FirstThunk);
+
+    // Replace current function address with new function address
+    for (; pThunk->u1.Function; pThunk++) {
+      IATRESULTS::FUNCTIONINFO fu;
+
+      FARPROC pfnNew = 0;
+      size_t rva = 0;
+#ifdef _WIN64
+      if (pThunk->u1.Ordinal & IMAGE_ORDINAL_FLAG64)
+#else
+      if (pThunk->u1.Ordinal & IMAGE_ORDINAL_FLAG32)
+#endif
+      {
+        // Ordinal
+#ifdef _WIN64
+        size_t ord = IMAGE_ORDINAL64(pThunk->u1.Ordinal);
+#else
+        size_t ord = IMAGE_ORDINAL32(pThunk->u1.Ordinal);
+#endif
+
+        fu.ord = ord;
+        m.functions.push_back(fu);
+        PROC *ppfn = (PROC *)&pThunk->u1.Function;
+        if (!ppfn) {
+          fu.f = IATRESULTS::FAILUREREASON::NOTFOUND;
+          m.functions.push_back(fu);
+          continue;
+        }
+        rva = (size_t)pThunk;
+
+        char fe[100] = {0};
+        sprintf_s(fe, 100, "#%zu", ord);
+        pfnNew = GetProcAddress(hImportDLL, (LPCSTR)ord);
+        if (!pfnNew) {
+          fu.f = IATRESULTS::FAILUREREASON::NOTFOUND;
+          m.functions.push_back(fu);
+          continue;
+        }
+      } else {
+        // Get the address of the function address
+        PROC *ppfn = (PROC *)&pThunk->u1.Function;
+        if (!ppfn) {
+          fu.f = IATRESULTS::FAILUREREASON::NOTFOUND;
+          m.functions.push_back(fu);
+          continue;
+        }
+        rva = (size_t)pThunk;
+        PSTR fName = (PSTR)H;
+        fName += pThunk->u1.Function;
+        fName += 2;
+        if (!fName)
+          break;
+        fu.name = fName;
+        pfnNew = GetProcAddress(hImportDLL, fName);
+        if (!pfnNew) {
+          fu.f = IATRESULTS::FAILUREREASON::NOTFOUND;
+          m.functions.push_back(fu);
+          continue;
+        }
+      }
+
+      if (!PatchProcessBytes((LPVOID *)rva, llvm::StringRef((char *)&pfnNew,
+                                                            sizeof(pfnNew)))) {
+        fu.f = IATRESULTS::FAILUREREASON::CANNOTPATCH;
+        continue;
+      }
+      m.functions.push_back(fu);
+    }
+    if (res)
+      res->modules.push_back(m);
+  }
+}
+
+static void *GetSymbolFromDebInfo(llvm::StringRef Sym) {
+  ULONG64 buffer[(sizeof(SYMBOL_INFO) + MAX_SYM_NAME * sizeof(TCHAR) +
+                  sizeof(ULONG64) - 1) /
+                 sizeof(ULONG64)];
+  PSYMBOL_INFO pSymbol = (PSYMBOL_INFO)buffer;
+
+  pSymbol->SizeOfStruct = sizeof(SYMBOL_INFO);
+  pSymbol->MaxNameLen = MAX_SYM_NAME;
+
+  if (!SymFromName(GetCurrentProcess(), Sym.data(), pSymbol)) {
+    DWORD error = GetLastError();
+    (void)error;
+    return nullptr;
+  }
+
+  return (void *)pSymbol->Address;
+}
+
+static void InitDebInfo(llvm::StringRef SearchPath) {
+  static bool SymbolsInitialized{};
+  if (!SymbolsInitialized) {
+    SymSetOptions(0);
+
+    std::string Path("C:\\symbols\\;");
+    Path += SearchPath.str();
+
+    if (!SymInitialize(GetCurrentProcess(), Path.c_str(), TRUE)) {
+      DWORD error = GetLastError();
+      printf("SymInitialize returned error : %lu\n", error);
+      return;
+    }
+    SymbolsInitialized = true;
+  }
+}
+
+static NTQUERYINFORMATIONPROCESS GetNtQueryInformationProcess() {
+  static NTQUERYINFORMATIONPROCESS pNtQueryInformationProcess;
+  if (!pNtQueryInformationProcess) {
+    HMODULE hNtDll = LoadLibraryA("NtDll.dll");
+    pNtQueryInformationProcess = (NTQUERYINFORMATIONPROCESS)GetProcAddress(
+        hNtDll, "NtQueryInformationProcess");
+    FreeLibrary(hNtDll);
+  }
+  return pNtQueryInformationProcess;
+}
+
+static void PatchRPMalloc(ExecModule &M) {
+
+  // Suckiness to have only one instance of RPMalloc across all modules
+
+  typedef void*(__thiscall * MALLOC)(size_t);
+  typedef void(__thiscall * FREE)(void *);
+  typedef void*(__thiscall * CALLOC)(size_t, size_t);
+  typedef void*(__thiscall * REALLOC)(void *, size_t);
+  typedef void(__thiscall * HOOKTABLE)(uintptr_t*);
+
+  std::string ModuleName = M.Name;
+
+  // Get the target module alloc functions
+
+  /*MALLOC Malloc = (MALLOC)GetSymbolFromDebInfo(ModuleName + "!malloc");
+  FREE Free = (FREE)GetSymbolFromDebInfo(ModuleName + "!free");
+  REALLOC Realloc = (REALLOC)GetSymbolFromDebInfo(ModuleName + "!realloc");
+  CALLOC Calloc = (CALLOC)GetSymbolFromDebInfo(ModuleName + "!calloc");
+
+  MALLOC OtherRpMalloc = (MALLOC)GetSymbolFromDebInfo(ModuleName + "!rpmalloc");
+  FREE OtherRpFree = (FREE)GetSymbolFromDebInfo(ModuleName + "!rpfree");
+  REALLOC OtherRpRealloc =
+      (REALLOC)GetSymbolFromDebInfo(ModuleName + "!rprealloc");
+  CALLOC OtherRpCalloc = (CALLOC)GetSymbolFromDebInfo(ModuleName + "!rpcalloc");*/
+
+  HOOKTABLE Hook = (HOOKTABLE)GetSymbolFromDebInfo(ModuleName + "!HookAllocators");
+
+  // Get Buildozer RPMalloc functions
+
+  MALLOC RpMalloc = (MALLOC)GetSymbolFromDebInfo("buildozer!rpmalloc");
+  FREE RpFree = (FREE)GetSymbolFromDebInfo("buildozer!rpfree");
+  REALLOC RpRealloc = (REALLOC)GetSymbolFromDebInfo("buildozer!rprealloc");
+  CALLOC RpCalloc = (CALLOC)GetSymbolFromDebInfo("buildozer!rpcalloc");
+
+  uintptr_t Table[4]{
+    reinterpret_cast<uintptr_t>(RpMalloc),
+    reinterpret_cast<uintptr_t>(RpFree),
+    reinterpret_cast<uintptr_t>(RpCalloc),
+    reinterpret_cast<uintptr_t>(RpRealloc),
+  };
+  Hook(Table);
+
+  /*auto PatchJmp = [&](void *Dest, void *Other, void *Target) {
+    char *I = (char *)Dest;
+    while (true) {
+      unsigned OtherOffset = (char *)Other - ((char *)I + 5);
+      if (*(unsigned char *)I == 0xE8 &&
+          *(DWORD *)((char *)I + 1) == OtherOffset) {
+
+        DWORD NewOffset = (char *)Target - ((char *)I + 5);
+        PatchProcessBytes(I + 1,
+                          llvm::StringRef((char *)&NewOffset, sizeof(DWORD)));
+        break;
+      }
+      ++I;
+    }
+  };
+
+  PatchJmp(reinterpret_cast<void *>(Malloc),
+           reinterpret_cast<void *>(OtherRpMalloc),
+           reinterpret_cast<void *>(RpMalloc));
+  PatchJmp(reinterpret_cast<void *>(Free),
+           reinterpret_cast<void *>(OtherRpFree),
+           reinterpret_cast<void *>(RpFree));
+  PatchJmp(reinterpret_cast<void *>(Realloc),
+           reinterpret_cast<void *>(OtherRpRealloc),
+           reinterpret_cast<void *>(RpRealloc));
+  PatchJmp(reinterpret_cast<void *>(Calloc),
+           reinterpret_cast<void *>(OtherRpCalloc),
+           reinterpret_cast<void *>(RpCalloc));*/
+}
+
+static void InitializeStaticTLS(HINSTANCE h) {
+  DWORD ulsize = 0;
+  PIMAGE_TLS_DIRECTORY64 pTLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          h, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &ulsize);
+  if (!pTLS)
+    return;
+
+  static LDRPHANDLETLSDATA handleTLS;
+  if (!handleTLS) {
+    handleTLS = (LDRPHANDLETLSDATA)GetSymbolFromDebInfo("LdrpHandleTlsData");
+  }
+
+  NTQUERYINFORMATIONPROCESS pNtQueryInformationProcess =
+      GetNtQueryInformationProcess();
+
+  auto hp = GetCurrentProcess();
+
+  PROCESS_BASIC_INFORMATION BasicInfo;
+  DWORD dwReturnLength = 0;
+  NTSTATUS Status = pNtQueryInformationProcess(
+      hp, ProcessBasicInformation, &BasicInfo,
+      sizeof(PROCESS_BASIC_INFORMATION), &dwReturnLength);
+  (void)Status;
+
+  PPEB pPeb = BasicInfo.PebBaseAddress;
+
+  PIMAGE_TLS_DIRECTORY64 thisTLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          pPeb->Reserved3[1], TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &ulsize);
+  if (!thisTLS)
+    return;
+
+  PTEB pTEB = NtCurrentTeb();
+  (void)pTEB;
+
+  PLIST_ENTRY CurrentEntry = pPeb->Ldr->InMemoryOrderModuleList.Flink;
+  PLDR_DATA_TABLE_ENTRY Current = NULL;
+
+  while (CurrentEntry != &pPeb->Ldr->InMemoryOrderModuleList &&
+         CurrentEntry != NULL) {
+    Current = CONTAINING_RECORD(CurrentEntry, LDR_DATA_TABLE_ENTRY,
+                                InMemoryOrderLinks);
+    if (Current->DllBase == h) {
+      handleTLS(Current); // <--------------- the magic happens here
+      break;
+    }
+
+    CurrentEntry = CurrentEntry->Flink;
+  }
+}
+
+static void *SaveTLS(HANDLE H) {
+  DWORD Size = 0;
+  PIMAGE_TLS_DIRECTORY64 TLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          H, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &Size);
+  if (!TLS)
+    return nullptr;
+
+  unsigned Len = TLS->EndAddressOfRawData - TLS->StartAddressOfRawData;
+  void *Buf = llvm::safe_malloc(Len);
+
+  PTEB TEB = NtCurrentTeb();
+  (void)TEB;
+
+  uint64_t **ProcessTLS =
+      (uint64_t **)TEB->Reserved1[11]; // ThreadLocalStoragePointer
+
+  // Copy from current Process TLS static storage, not from PE's .tls
+  memcpy(Buf, *(ProcessTLS + *(unsigned int *)TLS->AddressOfIndex), Len);
+
+  return Buf;
+}
+
+static void ResetTLS(HANDLE H, void *Buffer = nullptr) {
+  DWORD Size = 0;
+  PIMAGE_TLS_DIRECTORY64 TLS =
+      (PIMAGE_TLS_DIRECTORY64)ImageDirectoryEntryToData(
+          H, TRUE, IMAGE_DIRECTORY_ENTRY_TLS, &Size);
+  if (!TLS)
+    return;
+
+  /*enum { ProcessTlsInformation = 35 };
+
+  typedef struct _THREAD_TLS_INFORMATION
+  {
+      ULONG Flags;
+      PVOID NewTlsData;
+      PVOID OldTlsData;
+      HANDLE ThreadId;
+  } THREAD_TLS_INFORMATION, *PTHREAD_TLS_INFORMATION;
+
+  typedef struct _PROCESS_TLS_INFORMATION
+  {
+      ULONG Flags;
+      ULONG OperationType;
+      ULONG ThreadDataCount;
+      ULONG TlsIndex;
+      ULONG PreviousCount;
+      THREAD_TLS_INFORMATION ThreadData[1];
+  } PROCESS_TLS_INFORMATION, *PPROCESS_TLS_INFORMATION;
+
+  auto hp = GetCurrentProcess();
+
+  enum : uint32_t { STATUS_INFO_LENGTH_MISMATCH = 0xC0000004L };
+
+  NTQUERYINFORMATIONPROCESS pNtQueryInformationProcess =
+  GetNtQueryInformationProcess();
+
+  DWORD dwReturnLength = 0;
+  NTSTATUS Status = pNtQueryInformationProcess(hp,
+  (PROCESSINFOCLASS)ProcessTlsInformation, nullptr, 0, &dwReturnLength); if
+  (Status != STATUS_INFO_LENGTH_MISMATCH) return;
+
+  void *ProcInfo = malloc(dwReturnLength);
+
+  Status = pNtQueryInformationProcess(hp,
+  (PROCESSINFOCLASS)ProcessTlsInformation, ProcInfo, dwReturnLength,
+  &dwReturnLength);*/
+
+  PTEB TEB = NtCurrentTeb();
+
+  uint64_t **ProcessTLS =
+      (uint64_t **)TEB->Reserved1[11]; // ThreadLocalStoragePointer
+
+  unsigned Len = TLS->EndAddressOfRawData - TLS->StartAddressOfRawData;
+  void *SrcBuf = Buffer ? Buffer : (void *)TLS->StartAddressOfRawData;
+
+  // Copy to current Process TLS static storage
+  memcpy(*(ProcessTLS + *(unsigned int *)TLS->AddressOfIndex), SrcBuf, Len);
+
+  // free(ProcInfo);
+}
+
+static BOOL EnumModulesCallback(PCSTR ModuleName, DWORD64 BaseOfDll,
+                                PVOID UserContext) {
+  return TRUE;
+}
+
+static void InitializeCRT(ExecModule &M) {
+  if (!SymRefreshModuleList(GetCurrentProcess())) {
+    DWORD error = GetLastError();
+    printf("SymLoadModuleEx returned error : %lu\n", error);
+  }
+
+  if (!SymEnumerateModules(GetCurrentProcess(), &EnumModulesCallback,
+                           nullptr)) {
+    DWORD error = GetLastError();
+    printf("EnumerateLoadedModules returned error : %lu\n", error);
+  }
+
+  std::string ModuleName = M.Name;
+
+  typedef int(__stdcall * MAINCRT)();
+  MAINCRT Main = (MAINCRT)GetSymbolFromDebInfo(ModuleName + "!mainCRTStartup");
+
+  typedef void(__thiscall * INITBUILDOZER)(void * /*InitEvent*/,
+                                           void * /*ExitEvent*/);
+  INITBUILDOZER INIT = (INITBUILDOZER)GetSymbolFromDebInfo(
+      ModuleName + "!llvm::InitLLVM::InitFromBuildozer");
+
+  HANDLE IE = CreateEvent(0, 0, 0, 0);
+  HANDLE EE = CreateEvent(0, 0, 0, 0);
+  std::thread T([&]() {
+    llvm::CrashRecoveryContext CRC;
+    CRC.RunSafely([&] {
+      INIT(IE, EE);
+      Main();
+    });
+  });
+  WaitForSingleObject(IE, INFINITE);
+  llvm::SmallVector<wchar_t, MAX_PATH> WThreadName;
+  std::string ThreadName = "CRT Init for " + ModuleName;
+  llvm::sys::windows::UTF8ToUTF16(ThreadName.c_str(), WThreadName);
+  SetThreadDescription(T.native_handle(), WThreadName.data());
+  SuspendThread(T.native_handle());
+
+  M.ExitEvent = EE;
+  M.TLSThreadHandle = T.native_handle();
+}
+
+void FindEntryPoints(ExecModule &M) {
+  std::string ModuleName = M.Name;
+
+  typedef void(__thiscall * SKIPINIT)();
+  SKIPINIT SI = (SKIPINIT)GetSymbolFromDebInfo(ModuleName +
+                                               "!llvm::InitLLVM::SkipInitLLVM");
+  SI();
+
+  M.Main = (ExecModule::MAIN)GetSymbolFromDebInfo(ModuleName + "!main");
+
+  M.SD = (ExecModule::SETCURDIR)GetSymbolFromDebInfo(
+      ModuleName + "!SetCurrentDirectoryBackdoor");
+
+  M.SMod = (ExecModule::SETMODULE)GetSymbolFromDebInfo(
+      ModuleName + "!llvm::InitLLVM::SetCurrentModuleBackdoor");
+
+  M.InitTLS = (ExecModule::DYNTLSINIT)GetSymbolFromDebInfo(ModuleName +
+                                                           "!__dyn_tls_init");
+  M.DestroyTLS = (ExecModule::DYNTLSDTOR)GetSymbolFromDebInfo(
+      ModuleName + "!__dyn_tls_dtor");
+
+  M.CaptureOutput = (ExecModule::CAPTUREOUT)GetSymbolFromDebInfo(
+      ModuleName + "!CaptureLocalOutput");
+
+  // Save in-memory static TLS storage, after CRT init, so that we can restore
+  // it on every call to the module's main()
+  M.TLSInitBuffer = SaveTLS(M.Handle);
+}
+
+namespace buildozer {
+int ImportEXE(llvm::StringRef EXE) {
+  ExecModule M;
+  llvm::StringRef N = llvm::sys::path::filename(EXE);
+  M.Name = Saver.save(N.substr(0, N.rfind('.')));
+  M.FullPath = Saver.save(llvm::sys::path::parent_path(EXE));
+
+  HINSTANCE H = LoadLibraryA(EXE.str().c_str());
+  if (!H)
+    return 0;
+  M.Handle = H;
+
+  RemapIAT(H);
+  InitDebInfo(M.FullPath);
+  PatchRPMalloc(M);
+  InitializeStaticTLS(H);
+  InitializeCRT(M);
+  FindEntryPoints(M);
+
+  LoadedModules.push_back(M);
+  return LoadedModules.size();
+}
+
+void Launch(int ModuleIndex, llvm::StringRef CurDir,
+            const llvm::SmallVectorImpl<const char *> &Argv) {
+  ExecModule &M = ExecModule::get(ModuleIndex);
+
+  ResetTLS(M.Handle, M.TLSInitBuffer);
+
+  M.InitTLS(nullptr, DLL_THREAD_ATTACH, nullptr);
+
+  llvm::SmallVector<wchar_t, MAX_PATH> WPath;
+  llvm::sys::path::widenPath(CurDir, WPath);
+
+  M.SD(WPath.begin());
+
+  M.SMod(M.Handle);
+
+  M.Main(Argv.size(), const_cast<const char **>(Argv.data()));
+}
+
+void Terminate(int ModuleIndex, std::string &StdErr, std::string &StdOut) {
+  ExecModule &M = ExecModule::get(ModuleIndex);
+
+  llvm::StringRef Err, Out;
+  M.CaptureOutput(Err, Out);
+  StdErr = Err.str();
+  StdOut = Out.str();
+
+  M.DestroyTLS(nullptr, DLL_THREAD_DETACH, nullptr);
+}
+
+void TerminateAll() {
+  for (auto &M : LoadedModules) {
+    ResumeThread(M.TLSThreadHandle);
+    SetEvent(M.ExitEvent);
+    // wait for thread termination
+    WaitForSingleObject(M.TLSThreadHandle, INFINITE);
+  }
+  LoadedModules.clear();
+}
+} // namespace buildozer

tools/buildozer/buildozer-main.cpp

+//===- buildozer-main.cpp - A Lightweight LLVM Build System ---------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "CompDB.h"
+#include "LoadEXE.h"
+#include "llvm/ADT/StringSet.h"
+#include "llvm/Support/CrashRecoveryContext.h"
+#include "llvm/Support/InitLLVM.h"
+#include "llvm/Support/Path.h"
+#include "llvm/Support/ThreadPool.h"
+#include "llvm/Support/Threading.h"
+#include "llvm/Support/WithColor.h"
+#include "llvm/Support/raw_ostream.h"
+#include "llvm/Support/ErrorOr.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/RandomNumberGenerator.h"
+#include "llvm/Support/xxhash.h"
+#include "llvm/ADT/GlobalTypeDenseMap.h"
+#include <mutex>
+#include <map>
+#include "C:\Users\aganea\Downloads\rpmalloc-1.4.0\rpmalloc\rpmalloc.h"
+
+static int ClangM{};
+static int LLDM{};
+static int ScanDepsM{};
+
+void LoadPlugins(llvm::StringRef DirPath) {
+  // To ensure the CRT init is enclosed into the CRC
+  llvm::CrashRecoveryContext::Enable();
+
+  llvm::SmallString<128> ExeFullPath = DirPath;
+  llvm::sys::path::append(ExeFullPath, "clang-cl.exe");
+  ClangM = buildozer::ImportEXE(ExeFullPath);
+  //LLDM = buildozer::ImportEXE("lld-link.exe");
+  //ScanDepsM = buildozer::ImportEXE("clang-scan-deps.exe");
+  // int TblGen = buildozer::ImportEXE("llvm-tblgen.exe");
+  // int CTblGen =
+  // buildozer::ImportEXE("clang-tblgen.exe");
+}
+
+static std::deque<size_t> BuildQueue(const buildozer::CompilationDB &DB,
+                                     llvm::StringSet<> &PCHInProgress) {
+  std::deque<size_t> Q;
+  for (unsigned I = 0; I < DB.Build.size(); ++I) {
+    const buildozer::WorkUnit &WU = DB.Build.at(I);
+    if (WU.Output.endswith(".pch")) {
+      Q.push_front(I);
+      PCHInProgress.insert(llvm::sys::path::filename(WU.Output));
+    } else {
+      Q.push_back(I);
+    }
+  }
+  return Q;
+}
+
+static buildozer::WorkUnit *PopNext(std::deque<size_t> &Q,
+                                    buildozer::CompilationDB &DB,
+                                    llvm::StringSet<> &PCHInProgress) {
+  if (Q.empty())
+    return nullptr;
+  buildozer::WorkUnit *WU{};
+  for (std::deque<size_t>::iterator I = Q.begin(); I != Q.end(); ++I) {
+    WU = &DB.Build.at(*I);
+
+    auto Fp = find_if(WU->Arguments, [](auto &A) {
+      return llvm::StringRef(A).startswith("/Fp");
+    });
+    if (Fp != WU->Arguments.end()) {
+      auto Yu = find_if(WU->Arguments, [](auto &A) {
+        return llvm::StringRef(A).startswith("/Yu");
+      });
+      if (Yu != WU->Arguments.end()) {
+        llvm::StringRef File = llvm::sys::path::filename(llvm::StringRef(*Fp));
+        // If the corresponding PCH is still building, skip this file and treat
+        // it later
+        if (PCHInProgress.count(File))
+          continue;
+      }
+    }
+    Q.erase(I);
+    break;
+  }
+  return WU;
+}
+
+static void FinishPCH(buildozer::WorkUnit *WU,
+                      llvm::StringSet<> &PCHInProgress) {
+  if (!WU->Output.endswith(".pch"))
+    return;
+  auto It = PCHInProgress.find(llvm::sys::path::filename(WU->Output));
+  PCHInProgress.erase(It);
+}
+
+int ExecuteDatabase(buildozer::CompilationDB &DB) {
+  unsigned NumWorkers = llvm::hardware_concurrency() + 2;
+  std::vector<std::thread> WorkerThreads;
+  std::atomic<int> Error{};
+  std::mutex Lock;
+  llvm::StringSet<> PCHInProgress;
+  std::deque<size_t> ActiveQueue = BuildQueue(DB, PCHInProgress);
+  std::mutex PrintLock;
+
+  for (unsigned I = 0; I < NumWorkers; ++I) {
+    auto Worker = [&Lock, &PCHInProgress, &ActiveQueue, &DB, &Error,
+                   &PrintLock]() {
+      while (true) {
+        buildozer::WorkUnit *WU{};
+        {
+          std::unique_lock<std::mutex> LockGuard(Lock);
+          WU = PopNext(ActiveQueue, DB, PCHInProgress);
+          if (!WU)
+            break;
+
+          //////// TEST - only process first CPP
+          //ActiveQueue.clear();
+          /*static unsigned Count{};
+          if (Count >= 2)
+            break;
+          ++Count;*/
+        }
+
+        int Mod{};
+        llvm::StringRef Exe(WU->Arguments[0]);
+        if (Exe.contains_lower("clang-cl"))
+          Mod = ClangM;
+        // if (Exe.contains_lower("lld-link"))
+        //  Mod = LLDM;
+        else
+          continue;
+
+        if (Mod == LLDM) {
+          WU->Arguments.push_back("/threads:no");
+        }
+
+        llvm::CrashRecoveryContext CRC;
+        // CRC.EnableExceptionHandler = true;
+        if (!CRC.RunSafely([&] {
+              buildozer::Launch(Mod, WU->Directory, WU->Arguments);
+            })) {
+          unsigned ExceptionType = CRC.RetCode & 0xF0000000;
+          if (ExceptionType == 0xE0000000)
+            Error = CRC.RetCode & ~0xF0000000; // internal LLVM error
+          else if (ExceptionType == 0x80000000 || ExceptionType == 0xC0000000)
+            Error = 1; // don't propagate crash
+          else
+            Error = 1;
+
+          {
+            std::unique_lock<std::mutex> LockGuard(Lock);
+            ActiveQueue.clear(); // stop the machines
+          }
+        }
+        std::string StdErr, StdOut;
+        buildozer::Terminate(Mod, StdErr, StdOut);
+
+        FinishPCH(WU, PCHInProgress);
+
+        if (!StdErr.empty() || !StdOut.empty()) {
+          std::unique_lock<std::mutex> LockGuard(PrintLock);
+          llvm::outs() << "--> " << WU->Output << "\n";
+          llvm::outs() << StdOut;
+          llvm::errs() << StdErr;
+        }
+      }
+      //buildozer::TerminateThread(/*which mod?*/);
+    };
+    WorkerThreads.emplace_back(std::move(Worker));
+  }
+  for (auto &W : WorkerThreads)
+    W.join();
+  return Error;
+}
+
+void TestStringMap(buildozer::CompilationDB &DB);
+void TestGlobalMap(buildozer::CompilationDB &DB);
+
+int main(int argc, const char **argv) {
+  llvm::InitLLVM X(argc, argv);
+
+  LoadPlugins(llvm::sys::path::parent_path(argv[0]));
+
+  auto *DBName = "compile_commands.json";
+
+  buildozer::CompilationDB DB;
+  buildozer::ReadCompilationDatabase(DBName, DB);
+
+  //TestStringMap(DB);
+  //TestGlobalMap(DB);
+
+  ExecuteDatabase(DB);
+
+  buildozer::TerminateAll();
+
+  //rpmalloc_finalize();
+
+  return 0;
+}
+
+/*void TestStringMap(buildozer::CompilationDB &DB) {
+  class DirectoryEntry {
+    llvm::StringRef Name;
+  };
+  llvm::StringMap<llvm::ErrorOr<DirectoryEntry *>, llvm::BumpPtrAllocator>
+      SeenDirEntries;
+
+  for (unsigned I = 0; I < DB.Build.size(); ++I) {
+    const buildozer::WorkUnit &WU = DB.Build.at(I);
+
+    auto SeenDirInsertResult =
+        SeenDirEntries.insert({WU.File, std::errc::no_such_file_or_directory});
+    if (!SeenDirInsertResult.second)
+      continue; // already there
+
+    auto &NamedDirEnt = *SeenDirInsertResult.first;
+    NamedDirEnt.second = nullptr; // point to something
+  }
+}
+
+void TestGlobalMap(buildozer::CompilationDB &DB) {
+  llvm::BumpPtrAllocator Alloc;
+  llvm::StringSaver Saver{Alloc};
+  //llvm::SmallVector<llvm::StringRef, 2> SeenStrings;
+  llvm::SmallVector<uint64_t, 2> SeenHashes;
+  llvm::codeview::GlobalTypeDenseMap<uint64_t, uint32_t> SeenDirEntries;
+
+  auto ReHasher = [&]() {
+    for (unsigned I = 0; I < SeenHashes.size(); ++I) {
+      SeenDirEntries.try_emplace(SeenHashes[I], I, [&]() {});
+    }
+  };
+
+  for (unsigned I = 0; I < DB.Build.size(); ++I) {
+    const buildozer::WorkUnit &WU = DB.Build.at(I);
+
+    uint64_t X = llvm::xxHash64(WU.File);
+    auto R = SeenDirEntries.try_emplace(X, SeenStrings.size(),
+                                        [&]() { ReHasher(); });
+    if (!R.second)
+      continue; // already there
+
+    auto &NamedDirEnt = *SeenDirInsertResult.first;
+    NamedDirEnt.second = nullptr; // point to something
+
+    //SeenStrings.push_back(Saver.save(WU.Output)); // some other file
+    SeenHashes.push_back(X);
+  }
+}*/

tools/llc/llc.cpp

 };
 }
 
-static RunPassOption RunPassOpt;
+static LLVM_THREAD_LOCAL RunPassOption RunPassOpt;
 
 static cl::opt<RunPassOption, true, cl::parser<std::string>> RunPass(
     "run-pass",

tools/llvm-cfi-verify/lib/FileAnalysis.h

 
 struct GraphResult;
 
-extern bool IgnoreDWARFFlag;
+extern LLVM_THREAD_LOCAL bool IgnoreDWARFFlag;
 
 enum class CFIProtectionStatus {
   // This instruction is protected by CFI.

tools/llvm-cfi-verify/lib/FileAnalysis.cpp

 namespace llvm {
 namespace cfi_verify {
 
-bool IgnoreDWARFFlag;
+LLVM_THREAD_LOCAL bool IgnoreDWARFFlag;
 
 static cl::opt<bool, true> IgnoreDWARFArg(
     "ignore-dwarf",

tools/llvm-cfi-verify/lib/GraphBuilder.h

 namespace llvm {
 namespace cfi_verify {
 
-extern uint64_t SearchLengthForUndef;
-extern uint64_t SearchLengthForConditionalBranch;
+extern LLVM_THREAD_LOCAL uint64_t SearchLengthForUndef;
+extern LLVM_THREAD_LOCAL uint64_t SearchLengthForConditionalBranch;
 
 struct ConditionalBranchNode {
   uint64_t Address;

tools/llvm-cfi-verify/lib/GraphBuilder.cpp

 namespace llvm {
 namespace cfi_verify {
 
-uint64_t SearchLengthForUndef;
-uint64_t SearchLengthForConditionalBranch;
+LLVM_THREAD_LOCAL uint64_t SearchLengthForUndef;
+LLVM_THREAD_LOCAL uint64_t SearchLengthForConditionalBranch;
 
 static cl::opt<uint64_t, true> SearchLengthForUndefArg(
     "search-length-undef",

tools/llvm-xray/llvm-xray.cpp

     if (*SC) {
       // If no subcommand was provided, we need to explicitly check if this is
       // the top-level subcommand.
-      if (SC == &*cl::TopLevelSubCommand) {
+      if (cl::isTopLevel(SC)) {
         cl::PrintHelpMessage(false, true);
         return 0;
       }

unittests/Support/CommandLineTest.cpp

   static const char ArgString[] = "new-test-option";
   static const char ValueString[] = "Integer";
 
-  StringMap<cl::Option *> &Map =
-      cl::getRegisteredOptions(*cl::TopLevelSubCommand);
+  StringMap<cl::Option *> &Map = cl::getRegisteredOptions();
 
   ASSERT_TRUE(Map.count("test-option") == 1) <<
     "Could not find option in map.";

   ASSERT_EQ(cl::NotHidden, TestOption2.getOptionHiddenFlag())
       << "Hid extra option that should be visable.";
 
-  StringMap<cl::Option *> &Map =
-      cl::getRegisteredOptions(*cl::TopLevelSubCommand);
+  StringMap<cl::Option *> &Map = cl::getRegisteredOptions();
   ASSERT_EQ(cl::NotHidden, Map["help"]->getOptionHiddenFlag())
       << "Hid default option that should be visable.";
 }

   ASSERT_EQ(cl::NotHidden, TestOption3.getOptionHiddenFlag())
       << "Hid extra option that should be visable.";
 
-  StringMap<cl::Option *> &Map =
-      cl::getRegisteredOptions(*cl::TopLevelSubCommand);
+  StringMap<cl::Option *> &Map = cl::getRegisteredOptions();
   ASSERT_EQ(cl::NotHidden, Map["help"]->getOptionHiddenFlag())
       << "Hid default option that should be visable.";
 }

   cl::ResetCommandLineParser();
 
   StackSubCommand SC1("sc1", "First subcommand");
-  StackOption<bool> AllOpt("everywhere", cl::sub(*cl::AllSubCommands),
+  StackOption<bool> AllOpt("everywhere", cl::allSub(),
                            cl::init(false));
   StackSubCommand SC2("sc2", "Second subcommand");
 

 TEST(CommandLineTest, ReparseCommandLineOptions) {
   cl::ResetCommandLineParser();
 
-  StackOption<bool> TopLevelOpt("top-level", cl::sub(*cl::TopLevelSubCommand),
-                                cl::init(false));
+  StackOption<bool> TopLevelOpt("top-level", cl::topLevel(), cl::init(false));
 
   const char *args[] = {"prog", "-top-level"};
 

 TEST(CommandLineTest, RemoveFromTopLevelSubCommand) {
   cl::ResetCommandLineParser();
 
-  StackOption<bool> TopLevelRemove(
-      "top-level-remove", cl::sub(*cl::TopLevelSubCommand), cl::init(false));
-  StackOption<bool> TopLevelKeep(
-      "top-level-keep", cl::sub(*cl::TopLevelSubCommand), cl::init(false));
+  StackOption<bool> TopLevelRemove("top-level-remove", cl::topLevel(),
+                                   cl::init(false));
+  StackOption<bool> TopLevelKeep("top-level-keep", cl::topLevel(),
+                                 cl::init(false));
 
   const char *args[] = {"prog", "-top-level-remove"};
 

 
   StackSubCommand SC1("sc1", "First Subcommand");
   StackSubCommand SC2("sc2", "Second Subcommand");
-  StackOption<bool> RemoveOption("remove-option", cl::sub(*cl::AllSubCommands),
+  StackOption<bool> RemoveOption("remove-option", cl::allSub(),
                                  cl::init(false));
-  StackOption<bool> KeepOption("keep-option", cl::sub(*cl::AllSubCommands),
+  StackOption<bool> KeepOption("keep-option", cl::allSub(),
                                cl::init(false));
 
   const char *args0[] = {"prog", "-remove-option"};

 TEST(CommandLineTest, DefaultOptions) {
   cl::ResetCommandLineParser();
 
-  StackOption<std::string> Bar("bar", cl::sub(*cl::AllSubCommands),
+  StackOption<std::string> Bar("bar", cl::allSub(),
                                cl::DefaultOption);
   StackOption<std::string, cl::alias> Bar_Alias(
       "b", cl::desc("Alias for -bar"), cl::aliasopt(Bar), cl::DefaultOption);
 
-  StackOption<bool> Foo("foo", cl::init(false), cl::sub(*cl::AllSubCommands),
+  StackOption<bool> Foo("foo", cl::init(false), cl::allSub(),
                         cl::DefaultOption);
   StackOption<bool, cl::alias> Foo_Alias("f", cl::desc("Alias for -foo"),
                                          cl::aliasopt(Foo), cl::DefaultOption);

 
   cl::ResetAllOptionOccurrences();
 
-  for (auto &OM : cl::getRegisteredOptions(*cl::TopLevelSubCommand)) {
+  for (auto &OM : cl::getRegisteredOptions()) {
     cl::Option *O = OM.second;
     if (O->ArgStr == "opt2") {
       continue;

utils/TableGen/TableGen.cpp

 namespace llvm {
 /// Storage for TimeRegionsOpt as a global so that backends aren't required to
 /// include CommandLine.h
-bool TimeRegions = false;
+LLVM_THREAD_LOCAL bool TimeRegions = false;
 } // end namespace llvm
 
 namespace {