RFC: [LLD][COFF] Parallel GHASH generation at link-time -- NOT FOR SUBMIT

[aganea]

Summary

This is just a proof-of-concept, just to demonstrate generating GHASHes at link-time (in parallel).
In my test, all the source OBJs are from MSVC, so there's no prior GHASH stream.

All in all, things are about 5sec faster (in my large DLL test), even if we're generating GHASHes.
The end-gain comes also from the Types/IDs hash tables, which are much faster with GHASHes

I've also optimized the Type hash table (through the new GlobalTypeDenseMap class) by making the buckets smaller (8 bytes vs. 12 bytes for regular GHASH). This makes merging about 35% faster.

I've also thown in parallel sorting the globals stream which makes this pass 2x faster.

Before patch, with regular Type merging:

-------------------------------------------------
  Input File Reading:          1658 ms (  4.7%)  
  Code Layout:                  621 ms (  1.8%)  
  PDB Emission (Cumulative):  30380 ms ( 86.7%)  
    Add Objects:              22615 ms ( 64.6%)  
                                                 
      Type Merging:           19205 ms ( 54.8%)  
      Symbol Merging:          3385 ms (  9.7%)  
    TPI Stream Layout:          897 ms (  2.6%)  
    Globals Stream Layout:     1418 ms (  4.1%)  
    Commit to Disk:            4559 ms ( 13.0%)  
  Commit Output File:          1717 ms (  4.9%)  
-------------------------------------------------
Total Link Time:              35021 ms (100.0%)

With this patch, GHASH-only merging:

------------------------------------------------
  Input File Reading:          1647 ms (  5.4%)
  Code Layout:                  576 ms (  1.9%)
  PDB Emission (Cumulative):  27537 ms ( 89.6%)
    Add Objects:              21088 ms ( 68.6%)
      Global hashing:         10723 ms ( 34.9%)  <<<< parallel
      Type Merging:            7419 ms ( 24.1%)  <<<< 12-byte buckets
      Symbol Merging:          2861 ms (  9.3%)
    TPI Stream Layout:          941 ms (  3.1%)
    Globals Stream Layout:     1545 ms (  5.0%)  <<<< no parallel
    Commit to Disk:            3184 ms ( 10.4%)
  Commit Output File:           353 ms (  1.1%)
------------------------------------------------
Total Link Time:              30728 ms (100.0%)

With this patch, GHASH-only merging:

  Input File Reading:          1620 ms (  5.6%)
  Code Layout:                  598 ms (  2.1%)
  PDB Emission (Cumulative):  23715 ms ( 81.6%)
    Add Objects:              17933 ms ( 61.7%)
      Global hashing:          9734 ms ( 33.5%)  <<<< parallel
      Type Merging:            5293 ms ( 18.2%)  <<<< 8-byte buckets
      Symbol Merging:          2823 ms (  9.7%)
    TPI Stream Layout:          900 ms (  3.1%)
    Globals Stream Layout:      953 ms (  3.3%)  <<<< parallel
    Commit to Disk:            3161 ms ( 10.9%)
  Commit Output File:          2512 ms (  8.6%)
-------------------------------------------------
Total Link Time:              29067 ms (100.0%)

Sorry for the messy patch, I am just looking for overall advice. If this is the right direction, I'll split down the patch in smaller pieces.

[aganea]

The source OBJ/LIBs are ~10GB.
Total records across all OBJs: 88,932,692
Merged types: 3,539,517 - indices: 1,194,366
The final DLL is 141MB, the PDB is ~1GB.

Timings are on a 6-core Intel Xeon Haswell 3.5Ghz.

[rnk]

Neat! I didn't have a chance to review with a fine toothed comb, but I like the idea, and here's how I analyze it. We've already talked about how merging a type record without ghash is O(size of type), and with a content hash, you can make that O(size of hash), i.e. look at 8 or 20 bytes instead of up to 64K. But, computing the hash requires reading the type record data, so we made the compiler do it and put it in .debug$H.
If one can't do that because MSVC is being used, parallelizing the hash lookup makes it so that the linker only takes O(size of all types / n cores) wall time to do it, and then it does O(# types) (not size of types) work to deduplicate them.

So it looks like this is the list of things to do:

1. custom ghash densemap: this is probably the most valuable, but probably the most complex and needs the most work.
2. parallelize publics sorting: small enough to send and commit separately ASAP
3. parallelize ghash fetch or computation: actually small in isolation as well, let's do it, I like the code for it

Another thing about the custom hash table is, we might end up ripping it out if we try to parallelize type merging.

lld/trunk/COFF/PDB.cpp

345–1

Wow, I'm surprised that matters. Feel free to send it separately, it seems like a small win.

382

All work up to here (except for dependency loading) can be kicked off from SymbolTable::addFile, which is the earliest point that we know we are including an object file in the link. The way it is written here, it's easy to understand that we are computing or fetching hashes in parallel. That clarify is worth something. Do you think it's worth restructuring things so that addFile is responsible for starting ghash computation and loading PDB and PCH dependencies? I think it might help reduce link time further by overlapping independent work, but it might make the unnecessarily complicated. Maybe it's best to keep things this way for now.

387–401

We need to find a way to parallelize type merging... but that is future work.

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

78

I haven't read this implementation yet, it's quite long, but broadly I'm in favor of having a custom hash table here. This is the most performance critical thing LLD does.

This seems like a good separable change, since this map isn't actually used in parallel.

343

How much do huge pages matter relative to the custom hash table?

[rnk]

One more thing I just realized that was probably obvious to you: Computing ghash in parallel completely eliminates the need for the /debug:ghash flag, and we can eliminate the non-ghash codepath. That's great.

[aganea]

In D55585#1339859, @rnk wrote:

One more thing I just realized that was probably obvious to you: Computing ghash in parallel completely eliminates the need for the /debug:ghash flag, and we can eliminate the non-ghash codepath. That's great.

Yes, and even further it might eliminate the need for GHASHes streams at all, if porting this step works out well on GPU (currently still WIP).

lld/trunk/COFF/PDB.cpp

345–1

Ok - I'll send another patch. Some of our large EXEs can take up to 3.5sec in this "Globals" step.

382

I was thinking about that. Maybe as a subsequent change, once all this lands?

387–401

I already have a good idea of what to do to make GlobalTypeDenseMap thread-safe and lock-free. It's more a matter of finding the time to do it :-)

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

78

It's mostly a copy of DenseMap, but without tombstones, and with a different DenseHashInfo API.
I was wondering if a insert-only DenseMap would be useful in other parts of LLVM/Clang?

[aganea]

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

343

It's quite significant:

without 2MB pages	Type Merging: 6588 ms ( 23.0%)
with 2MB pages	Type Merging: 4856 ms ( 19.3%)

I only removed the flag sys::Memory::MF_HUGE for this test.

Here are some stats for the data used:

                                    Summary
--------------------------------------------------------------------------------
            156 Input OBJ files (expanded from all cmd-line inputs)
              0 Dependent PDB files
              1 Dependent PCH OBJ files
       81556098 Input type records (across all OBJ and dependencies)
     5108516032 Input type records bytes (across all OBJ and dependencies)
        4588516 Output merged type records
       10067321 Output merged symbol records
          23157 Output PDB strings

This is the perfect use-case for large pages: a large contiguous structure, used with random accesses, in a tight loop. The GlobalTypeDenseMap fits just perfectly with 2MB pages. In this precise testcase, the hashtable is 64MB (32x 2MB pages), which also happen to fit perfectly the max DTLB slots on modern Intel CPUs. If my understanding is correct, TLB slots for large pages come in addition to 4KB pages (at least for L1 DTLB).

I think I'll make this sys::Memory::MF_HUGE flag indicate a hint . On many OSes, you need to manually enable large pages (at least on W10 and Linux), so this might not be available by default. And even at that, on Windows at least, large pages are physical-only (not swappable). When specifying this flag, Memory::allocateMappedMemory should only "try" to use large pages, and fallback to regular (4KB) pages instead.

[aganea]

Rebased on r350764.

I've played around a bit with different hashing algorithms. I've added a flag /hasher:(sha1|md5|cityhash|xxhash|crc) and /summary along the way.
Figures are on a 3.5 GHz, 6-core Intel Xeon Haswell:

Dataset 1, large DLL, the resulting PDB is 950 MB:

(at r350764) lld-link		Type Merging: 20816 ms ( 59.4%)	Total Link Time: 35065 ms
(this patch) lld-link ... /hasher:sha1 (default)	Global hashing: 9635 ms ( 34.3%)	Type Merging: 4814 ms ( 17.1%)	Total Link Time: 28128 ms
(this patch) lld-link ... /hasher:md5	Global hashing: 5658 ms ( 23.4%)	Type Merging: 4813 ms ( 19.9%)	Total Link Time: 24137 ms
(this patch) lld-link ... /hasher:cityhash	Global hashing: 3640 ms ( 16.5%)	Type Merging: 4822 ms ( 21.8%)	Total Link Time: 22120 ms
(this patch) lld-link ... /hasher:xxhash	Global hashing: 3594 ms ( 16.4%)	Type Merging: 4730 ms ( 21.6%)	Total Link Time: 22896 ms
(this patch) lld-link ... /hasher:crc	Global hashing: 3276 ms ( 14.5%)	Type Merging: 4886 ms ( 21.6%)	Total Link Time: 22626 ms

                                    Summary
--------------------------------------------------------------------------------
            156 Input OBJ files (expanded from all cmd-line inputs)
              0 Dependent PDB files
              1 Dependent PCH OBJ files
       81556098 Input type records (across all OBJ and dependencies)
     5108516032 Input type records bytes (across all OBJ and dependencies)
        4588516 Output merged type records
       10067321 Output merged symbol records
          23157 Output PDB strings

---

And with a slightly larger input the difference becomes more apparent.
Dataset 2, large EXE, the resulting PDB is 2 GB:

(r350764) lld-link		Type Merging: 38413 ms ( 56.8%)	Total Link Time: 67609 ms
(this patch) lld-link ... /hasher:sha1 (default)	Global hashing: 15807 ms ( 29.7%)	Type Merging: 9906 ms ( 18.6%)	Total Link Time: 53135 ms
(this patch) lld-link ... /hasher:md5	Global hashing: 8354 ms ( 17.9%)	Type Merging: 9977 ms ( 21.3%)	Total Link Time: 46745 ms
(this patch) lld-link ... /hasher:cityhash	Global hashing: 6077 ms ( 13.7%)	Type Merging: 9957 ms ( 22.5%)	Total Link Time: 43291 ms
(this patch) lld-link ... /hasher:xxhash	Global hashing: 5783 ms ( 13.2%)	Type Merging: 9912 ms ( 22.7%)	Total Link Time: 43695 ms
(this patch) lld-link ... /hasher:crc	Global hashing: 5587 ms ( 12.9%)	Type Merging: 9938 ms ( 23.0%)	Total Link Time: 43206 ms

                                    Summary
--------------------------------------------------------------------------------
           4768 Input OBJ files (expanded from all cmd-line inputs)
             70 Dependent PDB files
             27 Dependent PCH OBJ files
      142150698 Input type records (across all OBJ and dependencies)
     8623310584 Input type records bytes (across all OBJ and dependencies)
        9699343 Output merged type records
       33727100 Output merged symbol records
          48382 Output PDB strings

To the light of all this, does it still makes sense to compute and emit GHASH streams in clang? I'm pretty sure that the cost for serialization and I/O for those streams would be much higher that just computing GHASHes on-the-fly in the LLD. The only marginal benefit would be for incrementally linking, however even that is debatable.

If you have no major concerns over all this, I'll start sending smaller patches.

[aganea]

Just for reference, added timings for /hasher:(xxhash|crc) in the comment above.

xxHash is using llvm::xxHash().
crc is a naive implementation with two concatenated crc32 values, using the SSE4.2 intrinsic _mm_crc32_u32 - thus using only two instructions per 4 bytes. However this creates about 0.0001% hash collisions, which would not be suitable for this application.

Also providing timings for a 2.3 GHz, 36-cores Intel Xeon Skylake:

Dataset 2, large EXE, the resulting PDB is 2 GB:

(trunk) lld-link ...		Type Merging: 34582 ms ( 52.7%)	Total Link Time: 65572 ms (100.0%)
(this patch) lld-link ... /hasher:sha1 (default)	Global hashing: 6910 ms ( 14.8%)	Type Merging: 12573 ms ( 26.9%)	Total Link Time: 46768 ms
(this patch) lld-link ... /hasher:md5	Global hashing: 5785 ms ( 12.6%)	Type Merging: 12479 ms ( 27.2%)	Total Link Time: 45902 ms
(this patch) lld-link ... /hasher:cityhash	Global hashing: 5153 ms ( 11.5%)	Type Merging: 12008 ms ( 26.8%)	Total Link Time: 44728 ms
(this patch) lld-link ... /hasher:xxhash	Global hashing: 5041 ms ( 10.9%)	Type Merging: 11917 ms ( 25.9%)	Total Link Time: 46057 ms
(this patch) lld-link ... /hasher:crc	Global hashing: 4994 ms ( 11.1%)	Type Merging: 12089 ms ( 26.8%)	Total Link Time: 45076 ms

Even though the global hashing is faster than the 6-cores, the single-threaded parts drag timings down because of the slower CPU frequency.

[ruiu]

Overall this is a huge patch, and I think it needs much more comments or a large file comment that gives the big picture what you are doing. I'm worried that that would not be able to understood once people who wrote or reviewed left the project.

lld/trunk/COFF/Driver.cpp

608–613

What is the point of making it selectable to users? It feels to me that you should pick up the one that you think the best and just use it.

614

parse() functions should return a value instead of mutating a variable as a side effect.

lld/trunk/COFF/PDB.cpp

96–97

A dependency represented as an "input file" feels a odd concept. Only an object file and the like should inherit InputFile.

936

Formatting.

lld/trunk/Common/Summary.cpp

20–21

Please don't use llvm::Any. Pass StringRef. If you need, define your own lld::toString to stringize an object you need to apss.

26

Please don't use formatv

42

Please use clang-format to format the entire patch.

[rnk]

In D55585#1354878, @aganea wrote:

To the light of all this, does it still makes sense to compute and emit GHASH streams in clang? I'm pretty sure that the cost for serialization and I/O for those streams would be much higher that just computing GHASHes on-the-fly in the LLD. The only marginal benefit would be for incrementally linking, however even that is debatable.

I'd definitely consider it. At the very least, with this data, I don't think it ever makes sense to enable -gcodeview-ghash by default in clang-cl. Looking at your data, it looks like pre-computing hashes into the obj would still improve link time by 12-20%, and our measurements show it costs 6% object file size. It seems like it could still be worth it.

If you have no major concerns over all this, I'll start sending smaller patches.

Looking forward to them. :)

lld/trunk/COFF/Driver.cpp

608–613

I think it's important to at least have the flexibility for developers to test different hashing schemes. We don't have to document all these flags. Users don't have to know about them, and we can remove them later.

lld/trunk/COFF/PDB.cpp

936

@ruiu This is still a proof of concept patch mostly to express ideas, not quite ready for review.
@aganea There is a bunch of unreachable, break-after-return in this switch that we won't want in the long run.

[thakis]

I thought this was demo code, but I was confused by rnk's comment about the hash flag below, so replying to that.

lld/trunk/COFF/Driver.cpp

608–613

Having this while working on this patch makes sense, but the experiment show that xhash is fast enough, so we wouldn't want to check this flag in, right? (And an implementation of e.g. cityhash just for this flag even though cityhash doesn't have any advantages over hashes using code we already have in tree.)

[santagada]

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

78

I know this patch is old, but I was reading it again and I don't understand how you reconstruct the key if there were a collision on the bucket while inserting... as the bucketno will not be the same bits that were stripped when the key got packed.

In other words: LookupBucketFor starts looking for an empty bucket at the extracted bits from the key, but might have probed ahead of that number by the time we insert it in the bucket. At that time it seems that any other searches for the key on the map will fail as the key computed from that bucket doesn't have the correct value were the bucketmask is.

Also bucketmask needs to be all the lower bits of key, else there is information loss there.

Or maybe I'm just very confused on how this packing of having half the key as bucketno and half in the KV value is happening.

[aganea]

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

78

You're right, collision is the biggest challange here. I ran with this for a while, with both the old map and this new map side-by-side, asserting if there was a divergence, and there was not. In practice, what I am proposing here increases the chances of a key collision, and thus of a hash collision in the table. However, even with very large inputs, in the range of 1 billions type records from .OBJs, I couldn't see a single collision. However, that doesn't mean it couldn't happen :-)

I wanted to do this to make reading & writing to a bucket an atomic operation. I'm not very comftable yet with this change, I think in the long run it'd be better to rely on 128-bit data per bucket (64-bit for the key and 64-bit for the index), and do two atomic operations. Which makes things a bit more challenging for writing a truly lock-free and wait-free hash table that can resize.

[aganea]

Superseded by rG5519e4da83d1abc66620334692394749eceb0e50

lld/trunk/COFF/Config.h

struct Configuration {
   bool DoGC = true;
   bool DoICF = true;
   bool TailMerge;
   bool Relocatable = true;
   bool ForceMultiple = false;
   bool ForceUnresolved = false;
   bool Debug = false;
   bool DebugDwarf = false;
-  bool DebugGHashes = false;
+  bool DebugGHashes = true;
   bool DebugSymtab = false;
   bool ShowTiming = false;
+  bool ShowSummary = false;
   unsigned DebugTypes = static_cast<unsigned>(DebugType::None);
   std::vector<std::string> NatvisFiles;
   llvm::SmallString<128> PDBAltPath;
   llvm::SmallString<128> PDBPath;
   llvm::SmallString<128> PDBSourcePath;
   std::vector<llvm::StringRef> Argv;
 
   // Symbols in this set are considered as live by the garbage collector.
struct Configuration {
   bool AllowIsolation = true;
   bool TerminalServerAware = true;
   bool LargeAddressAware = false;
   bool HighEntropyVA = false;
   bool AppContainer = false;
   bool MinGW = false;
   bool WarnMissingOrderSymbol = true;
   bool WarnLocallyDefinedImported = true;
+  bool WarnDebugInfoUnusable = true;
   bool Incremental = true;
   bool IntegrityCheck = false;
   bool KillAt = false;
   bool Repro = false;
 };
 
 extern Configuration *Config;
 
 } // namespace coff
 } // namespace lld
 
 #endif

lld/trunk/COFF/Driver.cpp

 #include "MinGW.h"
 #include "SymbolTable.h"
 #include "Symbols.h"
 #include "Writer.h"
 #include "lld/Common/Args.h"
 #include "lld/Common/Driver.h"
 #include "lld/Common/ErrorHandler.h"
 #include "lld/Common/Memory.h"
+#include "lld/Common/Threads.h" 
 #include "lld/Common/Timer.h"
+#include "lld/Common/Summary.h"
 #include "lld/Common/Version.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/BinaryFormat/Magic.h"
 #include "llvm/Object/ArchiveWriter.h"
 #include "llvm/Object/COFFImportFile.h"
 #include "llvm/Object/COFFModuleDefinition.h"
 #include "llvm/Option/Arg.h"
static unsigned parseDebugTypes(const opt::InputArgList &Args) {
   if (Args.hasArg(OPT_driver))
     DebugTypes |= static_cast<unsigned>(DebugType::PData);
   if (Args.hasArg(OPT_profile))
     DebugTypes |= static_cast<unsigned>(DebugType::Fixup);
 
   return DebugTypes;
 }
 
+static void parseHasher(const opt::InputArgList &Args) {
+  auto *Arg = Args.getLastArg(OPT_hasher);
+  if (!Arg)
+    return;
+  std::string ArgL = StringRef(Arg->getValue()).lower();
+
+  auto Hasher = codeview::GloballyHashedType::H;
+  if (ArgL == "md5")
+    Hasher = codeview::GloballyHashedType::HashType::MD5;
+  else if (ArgL == "sha1")
+    Hasher = codeview::GloballyHashedType::HashType::SHA1;
+  else if (ArgL == "cityhash")
+    Hasher = codeview::GloballyHashedType::HashType::CityHash;
+  codeview::GloballyHashedType::H = Hasher;
+}
+
 static std::string getMapFile(const opt::InputArgList &Args) {
   auto *Arg = Args.getLastArg(OPT_lldmap, OPT_lldmap_file);
   if (!Arg)
     return "";
   if (Arg->getOption().getID() == OPT_lldmap_file)
     return Arg->getValue();
 
   assert(Arg->getOption().getID() == OPT_lldmap);
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   if (Args.hasArg(OPT_help)) {
     printHelp(ArgsArr[0]);
     return;
   }
 
   if (Args.hasArg(OPT_show_timing))
     Config->ShowTiming = true;
 
+  if (Args.hasArg(OPT_summary))
+    Config->ShowSummary = true;
+
   ScopedTimer T(Timer::root());
   // Handle --version, which is an lld extension. This option is a bit odd
   // because it doesn't start with "/", but we deliberately chose "--" to
   // avoid conflict with /version and for compatibility with clang-cl.
   if (Args.hasArg(OPT_dash_dash_version)) {
     outs() << getLLDVersion() << "\n";
     return;
   }
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   // Construct search path list.
   SearchPaths.push_back("");
   for (auto *Arg : Args.filtered(OPT_libpath))
     SearchPaths.push_back(Arg->getValue());
   addLibSearchPaths();
 
   // Handle /ignore
   for (auto *Arg : Args.filtered(OPT_ignore)) {
-    if (StringRef(Arg->getValue()) == "4037")
+    SmallVector<StringRef, 8> Vec;
+    StringRef(Arg->getValue()).split(Vec, ',');
+    for (StringRef S : Vec) {
+      if (S == "4037")
         Config->WarnMissingOrderSymbol = false;
-    else if (StringRef(Arg->getValue()) == "4217")
+      else if (S == "4099")
+        Config->WarnDebugInfoUnusable = false;
+      else if (S == "4217")
         Config->WarnLocallyDefinedImported = false;
       // Other warning numbers are ignored.
     }
+  }
 
   // Handle /out
   if (auto *Arg = Args.getLastArg(OPT_out))
     Config->OutputFile = Arg->getValue();
 
   // Handle /verbose
   if (Args.hasArg(OPT_verbose))
     Config->Verbose = true;
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
   // Handle /debug
   DebugKind Debug = parseDebugKind(Args);
   if (Debug == DebugKind::Full || Debug == DebugKind::Dwarf ||
       Debug == DebugKind::GHash) {
     Config->Debug = true;
     Config->Incremental = true;
   }
 
+  // Handle /hasher
+  parseHasher(Args);
+
   // Handle /debugtype
   Config->DebugTypes = parseDebugTypes(Args);
 
   // Handle /pdb
   bool ShouldCreatePDB =
       (Debug == DebugKind::Full || Debug == DebugKind::GHash);
   if (ShouldCreatePDB) {
     if (auto *Arg = Args.getLastArg(OPT_pdb))
Config->Incremental =
                    !Config->DoGC && !Config->DoICF && !Args.hasArg(OPT_order) &&
                        !Args.hasArg(OPT_profile));
   Config->IntegrityCheck =
       Args.hasFlag(OPT_integritycheck, OPT_integritycheck_no, false);
   Config->NxCompat = Args.hasFlag(OPT_nxcompat, OPT_nxcompat_no, true);
   Config->TerminalServerAware =
       !Config->DLL && Args.hasFlag(OPT_tsaware, OPT_tsaware_no, true);
   Config->DebugDwarf = Debug == DebugKind::Dwarf;
-  Config->DebugGHashes = Debug == DebugKind::GHash;
+  //Config->DebugGHashes = Debug == DebugKind::GHash;
   Config->DebugSymtab = Debug == DebugKind::Symtab;
 
+  lld::ThreadsEnabled = Args.hasFlag(OPT_threads, OPT_threads_no, true);
+
   Config->MapFile = getMapFile(Args);
 
   if (Config->Incremental && Args.hasArg(OPT_profile)) {
     warn("ignoring '/incremental' due to '/profile' specification");
     Config->Incremental = false;
   }
 
   if (Config->Incremental && Args.hasArg(OPT_order)) {
void LinkerDriver::link(ArrayRef<const char *> ArgsArr) {
 
   // Write the result.
   writeResult();
 
   // Stop early so we can print the results.
   Timer::root().stop();
   if (Config->ShowTiming)
     Timer::root().print();
+  if (Config->ShowSummary)
+    Summary::print(Config->ShowTiming);
 }
 
 } // namespace coff
 } // namespace lld

lld/trunk/COFF/InputFiles.h

 #ifndef LLD_COFF_INPUT_FILES_H
 #define LLD_COFF_INPUT_FILES_H
 
 #include "Config.h"
 #include "lld/Common/LLVM.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseMap.h"
 #include "llvm/ADT/DenseSet.h"
+#include "llvm/DebugInfo/CodeView/TypeHashing.h" 
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/LTO/LTO.h"
 #include "llvm/Object/Archive.h"
 #include "llvm/Object/COFF.h"
 #include "llvm/Support/StringSaver.h"
 #include <memory>
 #include <set>
 #include <vector>
 
 namespace llvm {
 namespace pdb {
 class DbiModuleDescriptorBuilder;
 }
-}
+} // namespace llvm
 
 namespace lld {
 namespace coff {
 
 std::vector<MemoryBufferRef> getArchiveMembers(llvm::object::Archive *File);
 
 using llvm::COFF::IMAGE_FILE_MACHINE_UNKNOWN;
 using llvm::COFF::MachineTypes;
 using llvm::object::Archive;
-using llvm::object::COFFObjectFile;
-using llvm::object::COFFSymbolRef;
 using llvm::object::coff_import_header;
 using llvm::object::coff_section;
+using llvm::object::COFFObjectFile;
+using llvm::object::COFFSymbolRef;
 
 class Chunk;
 class Defined;
 class DefinedImportData;
 class DefinedImportThunk;
 class Lazy;
 class SectionChunk;
 class Symbol;
 class Undefined;
 
 // The root class of input files.
 class InputFile {
 public:
-  enum Kind { ArchiveKind, ObjectKind, ImportKind, BitcodeKind };
+  enum Kind {
+    ArchiveKind,
+    ObjectKind,
+    ImportKind,
+    BitcodeKind,
+    PDBKind,
+    PCHKind
+  };
   Kind kind() const { return FileKind; }
   virtual ~InputFile() {}
 
   // Returns the filename.
   StringRef getName() const { return MB.getBufferIdentifier(); }
 
   // Reads a file (the constructor doesn't do that).
   virtual void parse() = 0;
private:
   std::unique_ptr<Archive> File;
   std::string Filename;
   llvm::DenseSet<uint64_t> Seen;
 };
 
 // .obj or .o file. This may be a member of an archive file.
 class ObjFile : public InputFile {
 public:
+  explicit ObjFile(Kind K, MemoryBufferRef M) : InputFile(K, M) {}
   explicit ObjFile(MemoryBufferRef M) : InputFile(ObjectKind, M) {}
   static bool classof(const InputFile *F) { return F->kind() == ObjectKind; }
   void parse() override;
   MachineTypes getMachineType() override;
   ArrayRef<Chunk *> getChunks() { return Chunks; }
   ArrayRef<SectionChunk *> getDebugChunks() { return DebugChunks; }
   ArrayRef<SectionChunk *> getSXDataChunks() { return SXDataChunks; }
   ArrayRef<SectionChunk *> getGuardFidChunks() { return GuardFidChunks; }
   ArrayRef<SectionChunk *> getGuardLJmpChunks() { return GuardLJmpChunks; }
   ArrayRef<Symbol *> getSymbols() { return Symbols; }
 
   // Returns a Symbol object for the SymbolIndex'th symbol in the
   // underlying object file.
-  Symbol *getSymbol(uint32_t SymbolIndex) {
-    return Symbols[SymbolIndex];
-  }
+  Symbol *getSymbol(uint32_t SymbolIndex) { return Symbols[SymbolIndex]; }
 
   // Returns the underlying COFF file.
   COFFObjectFile *getCOFFObj() { return COFFObj.get(); }
 
   // Whether the object was already merged into the final PDB or not
   bool wasProcessedForPDB() const { return !!ModuleDBI; }
 
   static std::vector<ObjFile *> Instances;
public:
 
   const coff_section *AddrsigSec = nullptr;
 
   // When using Microsoft precompiled headers, this is the PCH's key.
   // The same key is used by both the precompiled object, and objects using the
   // precompiled object. Any difference indicates out-of-date objects.
   llvm::Optional<uint32_t> PCHSignature;
 
+  std::vector<llvm::codeview::GloballyHashedType> OwnedHashes;
+
 private:
   void initializeChunks();
   void initializeSymbols();
 
   SectionChunk *
   readSection(uint32_t SectionNumber,
               const llvm::object::coff_aux_section_definition *Def,
               StringRef LeaderName);

lld/trunk/COFF/Options.td

def rsp_quoting : Joined<["--"], "rsp-quoting=">,
   HelpText<"Quoting style for response files, 'windows' (default) or 'posix'">;
 def dash_dash_version : Flag<["--"], "version">,
   HelpText<"Print version information">;
 
 // Flags for debugging
 def lldmap : F<"lldmap">;
 def lldmap_file : Joined<["/", "-"], "lldmap:">;
 def show_timing : F<"time">;
+def summary : F<"summary">;
+
+def hasher : P<"hasher", "Select the hashing algorithm for PDB types dedup; either 'MD5', 'SHA1', 'CityHash'">;
+
+defm threads: B<"threads",
+    "Run the linker multi-threaded (default)",
+    "Do not run the linker multi-threaded">;
 
 //==============================================================================
 // The flags below do nothing. They are defined only for link.exe compatibility.
 //==============================================================================
 
 class QF<string name> : Joined<["/", "-", "-?"], name#":">;
 
 multiclass QB<string name> {

lld/trunk/COFF/PDB.cpp

 #include "PDB.h"
 #include "Chunks.h"
 #include "Config.h"
 #include "Driver.h"
 #include "SymbolTable.h"
 #include "Symbols.h"
 #include "Writer.h"
 #include "lld/Common/ErrorHandler.h"
+#include "lld/Common/Summary.h"
+#include "lld/Common/Threads.h"
 #include "lld/Common/Timer.h"
 #include "llvm/DebugInfo/CodeView/DebugFrameDataSubsection.h"
 #include "llvm/DebugInfo/CodeView/DebugSubsectionRecord.h"
 #include "llvm/DebugInfo/CodeView/GlobalTypeTableBuilder.h"
 #include "llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h"
 #include "llvm/DebugInfo/CodeView/MergingTypeTableBuilder.h"
 #include "llvm/DebugInfo/CodeView/RecordName.h"
 #include "llvm/DebugInfo/CodeView/SymbolDeserializer.h"

 #include "llvm/DebugInfo/PDB/PDB.h"
 #include "llvm/Object/COFF.h"
 #include "llvm/Object/CVDebugRecord.h"
 #include "llvm/Support/BinaryByteStream.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/Errc.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/JamCRC.h"
-#include "llvm/Support/Parallel.h"
+#include "llvm/Support/Mutex.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include <memory>
 
 using namespace lld;
 using namespace lld::coff;
 using namespace llvm;
 using namespace llvm::codeview;
 
 using llvm::object::coff_section;
 
 static ExitOnError ExitOnErr;
 
 static Timer TotalPdbLinkTimer("PDB Emission (Cumulative)", Timer::root());
 
 static Timer AddObjectsTimer("Add Objects", TotalPdbLinkTimer);
+static Timer GlobalHashTimer("Global hashing", AddObjectsTimer);
 static Timer TypeMergingTimer("Type Merging", AddObjectsTimer);
+static Timer HashtableLookupTimer("Hashtable lookup", TypeMergingTimer);
 static Timer SymbolMergingTimer("Symbol Merging", AddObjectsTimer);
 static Timer GlobalsLayoutTimer("Globals Stream Layout", TotalPdbLinkTimer);
 static Timer TpiStreamLayoutTimer("TPI Stream Layout", TotalPdbLinkTimer);
 static Timer DiskCommitTimer("Commit to Disk", TotalPdbLinkTimer);
 
+void startGlobalHashTimer() { HashtableLookupTimer.start(); }
+void stopGlobalHashTimer() { HashtableLookupTimer.stop(); }
+
 namespace {
 /// Map from type index and item index in a type server PDB to the
 /// corresponding index in the destination PDB.
 struct CVIndexMap {
   SmallVector<TypeIndex, 0> TPIMap;
   SmallVector<TypeIndex, 0> IPIMap;
   bool IsTypeServerMap = false;
   bool IsPrecompiledTypeMap = false;
 };
 
+// A PDB type server, which might be a dependency of another OBJ
+class PDBDependency : public InputFile {
+public:
+  explicit PDBDependency(MemoryBufferRef M) : InputFile(PDBKind, M) {
+    // Mark this map as a type server map.
+    IndexMap.IsTypeServerMap = true;
+  }
+  static bool classof(const InputFile *F) { return F->kind() == PDBKind; }
+
+  void parse() override {}
+
+  static std::map<llvm::codeview::GUID, PDBDependency *> Instances;
+
+  std::unique_ptr<llvm::pdb::NativeSession> Session;
+  CVIndexMap IndexMap;
+
+  std::vector<llvm::codeview::GloballyHashedType> TpiHashes;
+  std::vector<llvm::codeview::GloballyHashedType> IpiHashes;
+};
+
+std::map<llvm::codeview::GUID, PDBDependency *> PDBDependency::Instances;
+
+// A PCH (precompiled header OBJ, which might be a dependency of another OBJ
+class PCHDependency : public InputFile {
+public:
+  explicit PCHDependency(ObjFile *O) : InputFile(PCHKind, O->MB) {
+    RefObj = O;
+    // Mark this map as a precompiled types map.
+    IndexMap.IsPrecompiledTypeMap = true;
+  }
+  static bool classof(const InputFile *F) { return F->kind() == PCHKind; }
+
+  void parse() override {}
+
+  ObjFile *refObj() const { return RefObj; }
+
+  static std::map<uint32_t /*signature*/, PCHDependency *> Instances;
+
+  ObjFile *RefObj;
+  CVIndexMap IndexMap;
+};
+
+std::map<uint32_t /*signature*/, PCHDependency *> PCHDependency::Instances;
+
 class DebugSHandler;
 
 class PDBLinker {
   friend DebugSHandler;
 
 public:
   PDBLinker(SymbolTable *Symtab)
       : Alloc(), Symtab(Symtab), Builder(Alloc), TypeTable(Alloc),
         IDTable(Alloc), GlobalTypeTable(Alloc), GlobalIDTable(Alloc) {
     // This isn't strictly necessary, but link.exe usually puts an empty string
     // as the first "valid" string in the string table, so we do the same in
     // order to maintain as much byte-for-byte compatibility as possible.
     PDBStrTab.insert("");
   }
 
   /// Emit the basic PDB structure: initial streams, headers, etc.
   void initialize(llvm::codeview::DebugInfo *BuildId);
 
   /// Add natvis files specified on the command line.
   void addNatvisFiles();
 
   /// Link CodeView from each object file in the symbol table into the PDB.
   void addObjectsToPDB();
 
+  Error mergeAllOBJ();
+
   /// Link CodeView from a single object file into the target (output) PDB.
   /// When a precompiled headers object is linked, its TPI map might be provided
   /// externally.
-  void addObjFile(ObjFile *File, CVIndexMap *ExternIndexMap = nullptr);
+  void mergeSymbols(ObjFile *File, CVIndexMap *IndexMap);
 
   /// Produce a mapping from the type and item indices used in the object
   /// file to those in the destination PDB.
   ///
   /// If the object file uses a type server PDB (compiled with /Zi), merge TPI
   /// and IPI from the type server PDB and return a map for it. Each unique type
   /// server PDB is merged at most once, so this may return an existing index
   /// mapping.
   ///
   /// If the object does not use a type server PDB (compiled with /Z7), we merge
   /// all the type and item records from the .debug$S stream and fill in the
   /// caller-provided ObjectIndexMap.
-  Expected<const CVIndexMap &> mergeDebugT(ObjFile *File,
-                                           CVIndexMap *ObjectIndexMap);
+
+  Expected<unsigned> computeHash(InputFile *File);
+
+  Error openDependencies(InputFile *File);
+
+  Expected<CVIndexMap *> mergeTypes(InputFile *File, CVIndexMap &TIStorage);
+
+  void mergeTypeStream(ObjFile *File, CVTypeArray &Types,
+                       CVIndexMap &ObjectIndexMap);
+
+  template <typename Range>
+  unsigned computeTypeHashes(
+      ObjFile *File, Range &&Types,
+      ArrayRef<llvm::codeview::GloballyHashedType> PrecompHashes = {});
 
   /// Reads and makes available a PDB.
-  Expected<const CVIndexMap &> maybeMergeTypeServerPDB(ObjFile *File,
-                                                       const CVType &FirstType);
+  Expected<PDBDependency *> openPDB(ObjFile *File, const CVType &FirstType);
+  unsigned computePDBHashes(PDBDependency *D);
+  void mergePDB(PDBDependency *D);
 
   /// Merges a precompiled headers TPI map into the current TPI map. The
   /// precompiled headers object will also be loaded and remapped in the
   /// process.
-  Expected<const CVIndexMap &>
-  mergeInPrecompHeaderObj(ObjFile *File, const CVType &FirstType,
-                          CVIndexMap *ObjectIndexMap);
+  Error mergeInPCH(ObjFile *File, CVTypeArray &Stream,
+                   CVIndexMap &ObjectIndexMap);
 
   /// Reads and makes available a precompiled headers object.
   ///
   /// This is a requirement for objects compiled with cl.exe /Yu. In that
   /// case, the referenced object (which was compiled with /Yc) has to be loaded
   /// first. This is mainly because the current object's TPI stream has external
   /// references to the precompiled headers object.
   ///
   /// If the precompiled headers object was already loaded, this function will
   /// simply return its (remapped) TPI map.
-  Expected<const CVIndexMap &> aquirePrecompObj(ObjFile *File,
-                                                PrecompRecord Precomp);
-
-  /// Adds a precompiled headers object signature -> TPI mapping.
-  std::pair<CVIndexMap &, bool /*already there*/>
-  registerPrecompiledHeaders(uint32_t Signature);
+  Expected<PCHDependency *> openPCH(ObjFile *File);
 
-  void mergeSymbolRecords(ObjFile *File, const CVIndexMap &IndexMap,
+  void mergeSymbolRecords(ObjFile *File, const CVIndexMap *IndexMap,
                           std::vector<ulittle32_t *> &StringTableRefs,
                           BinaryStreamRef SymData);
 
   /// Add the section map and section contributions to the PDB.
   void addSections(ArrayRef<OutputSection *> OutputSections,
                    ArrayRef<uint8_t> SectionTable);
 
   /// Get the type table or the global type table if /DEBUG:GHASH is enabled.
private:
   GlobalTypeTableBuilder GlobalIDTable;
 
   /// PDBs use a single global string table for filenames in the file checksum
   /// table.
   DebugStringTableSubsection PDBStrTab;
 
   llvm::SmallString<128> NativePath;
 
-  /// A list of other PDBs which are loaded during the linking process and which
-  /// we need to keep around since the linking operation may reference pointers
-  /// inside of these PDBs.
-  llvm::SmallVector<std::unique_ptr<pdb::NativeSession>, 2> LoadedPDBs;
-
   std::vector<pdb::SecMapEntry> SectionMap;
 
-  /// Type index mappings of type server PDBs that we've loaded so far.
-  std::map<codeview::GUID, CVIndexMap> TypeServerIndexMappings;
+  std::atomic<uint64_t> TotalTypesRecords{};
+  std::atomic<uint64_t> TotalTypesRecordsSize{};
 
-  /// Type index mappings of precompiled objects type map that we've loaded so
-  /// far.
-  std::map<uint32_t, CVIndexMap> PrecompTypeIndexMappings;
-
-  /// List of TypeServer PDBs which cannot be loaded.
-  /// Cached to prevent repeated load attempts.
-  std::map<codeview::GUID, std::string> MissingTypeServerPDBs;
+  std::atomic<unsigned> GlobalSymbolsCount{};
+  std::atomic<unsigned> ModulesSymbolsCount{};
 };
 
 class DebugSHandler {
   PDBLinker &Linker;
 
   /// The object file whose .debug$S sections we're processing.
   ObjFile &File;
 
   /// The result of merging type indices.
-  const CVIndexMap &IndexMap;
+  const CVIndexMap *IndexMap;
 
   /// The DEBUG_S_STRINGTABLE subsection.  These strings are referred to by
   /// index from other records in the .debug$S section.  All of these strings
   /// need to be added to the global PDB string table, and all references to
   /// these strings need to have their indices re-written to refer to the
   /// global PDB string table.
   DebugStringTableSubsectionRef CVStrTab;
 
class DebugSHandler {
   /// that need to be re-written.  We first process all .debug$S subsections
   /// to ensure that we can handle subsections written in any order, building
   /// up this list as we go.  At the end, we use the string table (which must
   /// have been discovered by now else it is an error) to re-write these
   /// references.
   std::vector<ulittle32_t *> StringTableReferences;
 
 public:
-  DebugSHandler(PDBLinker &Linker, ObjFile &File, const CVIndexMap &IndexMap)
+  DebugSHandler(PDBLinker &Linker, ObjFile &File, const CVIndexMap *IndexMap)
       : Linker(Linker), File(File), IndexMap(IndexMap) {}
 
   void handleDebugS(lld::coff::SectionChunk &DebugS);
   void finish();
 };
-}
+} // namespace
 
 // Visual Studio's debugger requires absolute paths in various places in the
 // PDB to work without additional configuration:
 // https://docs.microsoft.com/en-us/visualstudio/debugger/debug-source-files-common-properties-solution-property-pages-dialog-box
 static void pdbMakeAbsolute(SmallVectorImpl<char> &FileName) {
   // The default behavior is to produce paths that are valid within the context
   // of the machine that you perform the link on.  If the linker is running on
   // a POSIX system, we will output absolute POSIX paths.  If the linker is

 static ArrayRef<uint8_t> getDebugSection(ObjFile *File, StringRef SecName) {
   if (SectionChunk *Sec = findByName(File->getDebugChunks(), SecName))
     return consumeDebugMagic(Sec->getContents(), SecName);
   return {};
 }
 
 // A COFF .debug$H section is currently a clang extension.  This function checks
 // if a .debug$H section is in a format that we expect / understand, so that we
 // can ignore any sections which are coincidentally also named .debug$H but do
 // not contain a format we recognize.
 static bool canUseDebugH(ArrayRef<uint8_t> DebugH) {
   if (DebugH.size() < sizeof(object::debug_h_header))
     return false;
   auto *Header =
       reinterpret_cast<const object::debug_h_header *>(DebugH.data());
   DebugH = DebugH.drop_front(sizeof(object::debug_h_header));
   return Header->Magic == COFF::DEBUG_HASHES_SECTION_MAGIC &&
          Header->Version == 0 &&
          Header->HashAlgorithm == uint16_t(GlobalTypeHashAlg::SHA1_8) &&
          (DebugH.size() % 8 == 0);
 }
 
 static Optional<ArrayRef<uint8_t>> getDebugH(ObjFile *File) {
   SectionChunk *Sec = findByName(File->getDebugChunks(), ".debug$H");
   if (!Sec)
     return llvm::None;
   ArrayRef<uint8_t> Contents = Sec->getContents();
   if (!canUseDebugH(Contents))
     return None;
   return Contents;
 }
 
 static ArrayRef<GloballyHashedType>
 getHashesFromDebugH(ArrayRef<uint8_t> DebugH) {
   assert(canUseDebugH(DebugH));
 
static void addTypeInfo(pdb::TpiStreamBuilder &TpiBuilder,
   });
 }
 
 // OBJs usually start their symbol stream with a S_OBJNAME record. This record
 // also contains the signature/key of the current PCH session. The signature
 // must be same for all objects which depend on the precompiled object.
 // Recompiling the precompiled headers will generate a new PCH key and thus
 // invalidate all the dependent objects.
-static uint32_t extractPCHSignature(ObjFile *File) {
+static void extractPCHSignature(ObjFile *File) {
+  if (File->PCHSignature)
+    return;
+
   auto DbgIt = find_if(File->getDebugChunks(), [](SectionChunk *C) {
     return C->getSectionName() == ".debug$S";
   });
-  if (!DbgIt)
-    return 0;
+  if (DbgIt == File->getDebugChunks().end())
+    return;
 
   ArrayRef<uint8_t> Contents =
       consumeDebugMagic((*DbgIt)->getContents(), ".debug$S");
   DebugSubsectionArray Subsections;
   BinaryStreamReader Reader(Contents, support::little);
   ExitOnErr(Reader.readArray(Subsections, Contents.size()));
 
   for (const DebugSubsectionRecord &SS : Subsections) {
     if (SS.kind() != DebugSubsectionKind::Symbols)
       continue;
 
     // If it's there, the S_OBJNAME record shall come first in the stream.
     Expected<CVSymbol> Sym = readSymbolFromStream(SS.getRecordData(), 0);
     if (!Sym) {
       consumeError(Sym.takeError());
       continue;
     }
-    if (auto ObjName = SymbolDeserializer::deserializeAs<ObjNameSym>(Sym.get()))
-      return ObjName->Signature;
+    if (auto ObjName =
+            SymbolDeserializer::deserializeAs<ObjNameSym>(Sym.get())) {
+      File->PCHSignature.emplace(ObjName->Signature);
+      return;
     }
-  return 0;
   }
-
-Expected<const CVIndexMap &>
-PDBLinker::mergeDebugT(ObjFile *File, CVIndexMap *ObjectIndexMap) {
-  ScopedTimer T(TypeMergingTimer);
-
-  bool IsPrecompiledHeader = false;
-
-  ArrayRef<uint8_t> Data = getDebugSection(File, ".debug$T");
-  if (Data.empty()) {
-    // Try again, Microsoft precompiled headers use .debug$P instead of
-    // .debug$T
-    Data = getDebugSection(File, ".debug$P");
-    IsPrecompiledHeader = true;
-  }
-  if (Data.empty())
-    return *ObjectIndexMap; // no debug info
+  return;
 
-  // Precompiled headers objects need to save the index map for further
-  // reference by other objects which use the precompiled headers.
-  if (IsPrecompiledHeader) {
-    uint32_t PCHSignature = extractPCHSignature(File);
-    if (PCHSignature == 0)
-      fatal("No signature found for the precompiled headers OBJ (" +
-            File->getName() + ")");
+template <typename Range>
+unsigned PDBLinker::computeTypeHashes(
+    ObjFile *File, Range &&Types,
+    ArrayRef<llvm::codeview::GloballyHashedType> PrecompHashes) {
+  if (Config->DebugGHashes) {
+    Optional<ArrayRef<uint8_t>> DebugH = getDebugH(File);
+    if (!DebugH) {
+      assert(File->OwnedHashes.empty());
+      uint64_t LocalRecordsBytes{};
 
-    // When a precompiled headers object comes first on the command-line, we
-    // update the mapping here. Otherwise, if an object referencing the
-    // precompiled headers object comes first, the mapping is created in
-    // aquirePrecompObj(), thus we would skip this block.
-    if (!ObjectIndexMap->IsPrecompiledTypeMap) {
-      auto R = registerPrecompiledHeaders(PCHSignature);
-      if (R.second)
-        fatal(
-            "A precompiled headers OBJ with the same signature was already "
-            "provided! (" +
-            File->getName() + ")");
+      File->OwnedHashes = GloballyHashedType::hashTypes(
+          Types, LocalRecordsBytes, PrecompHashes);
       
-      ObjectIndexMap = &R.first;
-    }
-  }
+      TotalTypesRecordsSize += LocalRecordsBytes;
+
+      return File->OwnedHashes.size() - PrecompHashes.size();
+    }
+  return 0;
 
-  BinaryByteStream Stream(Data, support::little);
+namespace {
+enum TypeStreamKind { NoTypeStream, RegularOBJ, PCH, UsingPCH, PDB, UsingPDB };
+
+struct Ident {
+  TypeStreamKind Kind;
   CVTypeArray Types;
-  BinaryStreamReader Reader(Stream);
+};
+} // namespace
+
+static Ident identifyTypeStream(InputFile *File) {
+  if (isa<PDBDependency>(File))
+    return {PDB, {}};
+  if (isa<PCHDependency>(File))
+    return {PCH, identifyTypeStream(cast<PCHDependency>(File)->refObj()).Types};
+
+  ObjFile *Obj = cast<ObjFile>(File);
+
+  extractPCHSignature(Obj);
+
+  bool IsPCH = false;
+
+  auto Data = getDebugSection(Obj, ".debug$P");
+  if (!Data.empty())
+    IsPCH = true;
+  else
+    Data = getDebugSection(Obj, ".debug$T");
+
+  if (Data.empty())
+    return {NoTypeStream, {}};
+
+  CVTypeArray Types;
+  BinaryStreamReader Reader(Data, support::little);
   if (auto EC = Reader.readArray(Types, Reader.getLength()))
     fatal("Reader::readArray failed: " + toString(std::move(EC)));
 
   auto FirstType = Types.begin();
   if (FirstType == Types.end())
-    return *ObjectIndexMap;
+    return {NoTypeStream, {}};
 
-  if (FirstType->kind() == LF_TYPESERVER2) {
-    // Look through type servers. If we've already seen this type server,
-    // don't merge any type information.
-    return maybeMergeTypeServerPDB(File, *FirstType);
+  if (IsPCH) {
+    return {PCH, Types};
+  } else if (FirstType->kind() == LF_TYPESERVER2) {
+    return {UsingPDB, Types};
   } else if (FirstType->kind() == LF_PRECOMP) {
-    // This object was compiled with /Yu, so process the corresponding
-    // precompiled headers object (/Yc) first. Some type indices in the current
-    // object are referencing data in the precompiled headers object, so we need
-    // both to be loaded.
-    auto E = mergeInPrecompHeaderObj(File, *FirstType, ObjectIndexMap);
+    return {UsingPCH, Types};
+  }
+  return {RegularOBJ, Types};
+}
+
+Error PDBLinker::openDependencies(InputFile *File) {
+  auto Stream = identifyTypeStream(File);
+  if (Stream.Kind == UsingPDB) {
+    auto E = openPDB(cast<ObjFile>(File), *Stream.Types.begin());
     if (!E)
       return E.takeError();
+  } else if (Stream.Kind == PCH) {
+    auto E = openPCH(cast<ObjFile>(File));
+    if (!E)
+      return E.takeError();
+  }
+  return Error::success();
+}
+
+Expected<CVIndexMap *> PDBLinker::mergeTypes(InputFile *File,
+                                             CVIndexMap &TIStorage) {
+  ScopedTimer T1(TypeMergingTimer);
+
+  auto Stream = identifyTypeStream(File);
+  if (Stream.Kind == NoTypeStream)
+    return nullptr;
 
-    // Drop LF_PRECOMP record from the input stream, as it needs to be replaced
-    // with the precompiled headers object type stream.
-    // Note that we can't just call Types.drop_front(), as we explicitly want to
-    // rebase the stream.
-    Types.setUnderlyingStream(
-        Types.getUnderlyingStream().drop_front(FirstType->RecordData.size()));
-  }
+  if (Stream.Kind == UsingPCH) {
+    if (auto E = mergeInPCH(cast<ObjFile>(File), Stream.Types, TIStorage))
+      return std::move(E);
+    mergeTypeStream(cast<ObjFile>(File), Stream.Types, TIStorage);
+  } else if (Stream.Kind == RegularOBJ) {
+    mergeTypeStream(cast<ObjFile>(File), Stream.Types, TIStorage);
+  } else if (Stream.Kind == PCH) {
+    PCHDependency *D = cast<PCHDependency>(File);
+    mergeTypeStream(D->refObj(), Stream.Types, D->IndexMap);
+    return &D->IndexMap;
+  } else if (Stream.Kind == PDB) {
+    PDBDependency *D = cast<PDBDependency>(File);
+    mergePDB(D);
+    return &D->IndexMap;
+  } else if (Stream.Kind == UsingPDB) {
+    auto E = openPDB(cast<ObjFile>(File), *Stream.Types.begin());
+    if (!E)
+      return E.takeError();
+    PDBDependency *D = *E;
+    if (D)
+      return &D->IndexMap;
+  }
+  return &TIStorage;
 
-  // Fill in the temporary, caller-provided ObjectIndexMap.
+void PDBLinker::mergeTypeStream(ObjFile *File, CVTypeArray &Types,
+                                CVIndexMap &ObjectIndexMap) {
   if (Config->DebugGHashes) {
     ArrayRef<GloballyHashedType> Hashes;
-    std::vector<GloballyHashedType> OwnedHashes;
     if (Optional<ArrayRef<uint8_t>> DebugH = getDebugH(File))
       Hashes = getHashesFromDebugH(*DebugH);
     else {
-      OwnedHashes = GloballyHashedType::hashTypes(Types);
-      Hashes = OwnedHashes;
+      Hashes = File->OwnedHashes;
     }
 
     if (auto Err = mergeTypeAndIdRecords(GlobalIDTable, GlobalTypeTable,
-                                         ObjectIndexMap->TPIMap, Types, Hashes,
+                                         ObjectIndexMap.TPIMap, Types, Hashes,
                                          File->PCHSignature))
       fatal("codeview::mergeTypeAndIdRecords failed: " +
             toString(std::move(Err)));
+
   } else {
     if (auto Err =
-            mergeTypeAndIdRecords(IDTable, TypeTable, ObjectIndexMap->TPIMap,
+            mergeTypeAndIdRecords(IDTable, TypeTable, ObjectIndexMap.TPIMap,
                                   Types, File->PCHSignature))
       fatal("codeview::mergeTypeAndIdRecords failed: " +
             toString(std::move(Err)));
   }
-  return *ObjectIndexMap;
 }
 
-static Expected<std::unique_ptr<pdb::NativeSession>>
-tryToLoadPDB(const codeview::GUID &GuidFromObj, StringRef TSPath) {
+static Expected<PDBDependency *>
+tryToLoadPDB(const llvm::codeview::GUID &GuidFromObj, StringRef TSPath) {
   // Ensure the file exists before anything else. We want to return ENOENT,
   // "file not found", even if the path points to a removable device (in which
   // case the return message would be EAGAIN, "resource unavailable try again")
   if (!llvm::sys::fs::exists(TSPath))
     return errorCodeToError(std::error_code(ENOENT, std::generic_category()));
 
   ErrorOr<std::unique_ptr<MemoryBuffer>> MBOrErr = MemoryBuffer::getFile(
       TSPath, /*FileSize=*/-1, /*RequiresNullTerminator=*/false);
   if (!MBOrErr)
     return errorCodeToError(MBOrErr.getError());
 
+  MemoryBufferRef Buffer((*MBOrErr)->getMemBufferRef());
   std::unique_ptr<pdb::IPDBSession> ThisSession;
   if (auto EC = pdb::NativeSession::createFromPdb(
           MemoryBuffer::getMemBuffer(Driver->takeBuffer(std::move(*MBOrErr)),
                                      /*RequiresNullTerminator=*/false),
           ThisSession))
     return std::move(EC);
 
   std::unique_ptr<pdb::NativeSession> NS(
       static_cast<pdb::NativeSession *>(ThisSession.release()));
   pdb::PDBFile &File = NS->getPDBFile();
   auto ExpectedInfo = File.getPDBInfoStream();
   // All PDB Files should have an Info stream.
   if (!ExpectedInfo)
     return ExpectedInfo.takeError();
 
   // Just because a file with a matching name was found and it was an actual
   // PDB file doesn't mean it matches.  For it to match the InfoStream's GUID
   // must match the GUID specified in the TypeServer2 record.
   if (ExpectedInfo->getGuid() != GuidFromObj)
-    return make_error<pdb::PDBError>(pdb::pdb_error_code::signature_out_of_date);
+    return make_error<pdb::PDBError>(
+        pdb::pdb_error_code::signature_out_of_date);
 
-  return std::move(NS);
+  PDBDependency *D = make<PDBDependency>(Buffer);
+  D->Session = std::move(NS);
+  return D;
 }
 
-Expected<const CVIndexMap &>
-PDBLinker::maybeMergeTypeServerPDB(ObjFile *File, const CVType &FirstType) {
+// Look through type servers. If we've already seen this type server,
+// don't merge any type information.
+Expected<PDBDependency *> PDBLinker::openPDB(ObjFile *DependentFile,
+                                             const CVType &FirstType) {
   TypeServer2Record TS;
   if (auto EC =
           TypeDeserializer::deserializeAs(const_cast<CVType &>(FirstType), TS))
     fatal("error reading record: " + toString(std::move(EC)));
 
-  const codeview::GUID &TSId = TS.getGuid();
+  const auto &TSId = TS.getGuid();
   StringRef TSPath = TS.getName();
 
-  // First, check if the PDB has previously failed to load.
-  auto PrevErr = MissingTypeServerPDBs.find(TSId);
-  if (PrevErr != MissingTypeServerPDBs.end())
-    return createFileError(
-        TSPath,
-        make_error<StringError>(PrevErr->second, inconvertibleErrorCode()));
-
-  // Second, check if we already loaded a PDB with this GUID. Return the type
-  // index mapping if we have it.
-  auto Insertion = TypeServerIndexMappings.insert({TSId, CVIndexMap()});
-  CVIndexMap &IndexMap = Insertion.first->second;
-  if (!Insertion.second)
-    return IndexMap;
-
-  // Mark this map as a type server map.
-  IndexMap.IsTypeServerMap = true;
+  // Check if we already loaded a PDB with this GUID.
+  auto I = PDBDependency::Instances.find(TSId);
+  if (I != PDBDependency::Instances.end())
+    return I->second;
 
   // Check for a PDB at:
   // 1. The given file path
   // 2. Next to the object file or archive file
-  auto ExpectedSession = handleExpected(
+  auto ExpectedPDB = handleExpected(
       tryToLoadPDB(TSId, TSPath),
       [&]() {
-        StringRef LocalPath =
-            !File->ParentName.empty() ? File->ParentName : File->getName();
+        StringRef LocalPath = !DependentFile->ParentName.empty()
+                                  ? DependentFile->ParentName
+                                  : DependentFile->getName();
         SmallString<128> Path = sys::path::parent_path(LocalPath);
         // Currently, type server PDBs are only created by cl, which only runs
         // on Windows, so we can assume type server paths are Windows style.
         sys::path::append(
             Path, sys::path::filename(TSPath, sys::path::Style::windows));
         return tryToLoadPDB(TSId, Path);
       },
       [&](std::unique_ptr<ECError> EC) -> Error {
         auto SysErr = EC->convertToErrorCode();
         // Only re-try loading if the previous error was "No such file or
         // directory"
         if (SysErr.category() == std::generic_category() &&
             SysErr.value() == ENOENT)
           return Error::success();
         return Error(std::move(EC));
       });
 
-  if (auto E = ExpectedSession.takeError()) {
-    TypeServerIndexMappings.erase(TSId);
-
-    // Flatten the error to a string, for later display, if the error occurs
-    // again on the same PDB.
-    std::string ErrMsg;
-    raw_string_ostream S(ErrMsg);
-    S << E;
-    MissingTypeServerPDBs.emplace(TSId, S.str());
-
+  if (auto E = ExpectedPDB.takeError()) {
+    PDBDependency::Instances.emplace(TSId, nullptr);
     return createFileError(TSPath, std::move(E));
   }
 
-  pdb::NativeSession *Session = ExpectedSession->get();
+  PDBDependency *D = *ExpectedPDB;
+  PDBDependency::Instances.insert({TSId, D});
 
-  // Keep a strong reference to this PDB, so that it's safe to hold pointers
-  // into the file.
-  LoadedPDBs.push_back(std::move(*ExpectedSession));
-
-  auto ExpectedTpi = Session->getPDBFile().getPDBTpiStream();
+  auto ExpectedTpi = D->Session->getPDBFile().getPDBTpiStream();
   if (auto E = ExpectedTpi.takeError())
     fatal("Type server does not have TPI stream: " + toString(std::move(E)));
-  auto ExpectedIpi = Session->getPDBFile().getPDBIpiStream();
+  auto ExpectedIpi = D->Session->getPDBFile().getPDBIpiStream();
   if (auto E = ExpectedIpi.takeError())
     fatal("Type server does not have TPI stream: " + toString(std::move(E)));
 
+  return D;
+}
+
+unsigned PDBLinker::computePDBHashes(PDBDependency *D) {
   if (Config->DebugGHashes) {
+    auto ExpectedTpi = D->Session->getPDBFile().getPDBTpiStream();
+    auto ExpectedIpi = D->Session->getPDBFile().getPDBIpiStream();
     // PDBs do not actually store global hashes, so when merging a type server
     // PDB we have to synthesize global hashes.  To do this, we first synthesize
     // global hashes for the TPI stream, since it is independent, then we
     // synthesize hashes for the IPI stream, using the hashes for the TPI stream
     // as inputs.
-    auto TpiHashes = GloballyHashedType::hashTypes(ExpectedTpi->typeArray());
-    auto IpiHashes =
-        GloballyHashedType::hashIds(ExpectedIpi->typeArray(), TpiHashes);
+    uint64_t LocalTypesRecordsBytes{};
+    uint64_t LocalIDsRecordsBytes{};
+
+    D->TpiHashes = GloballyHashedType::hashTypes(ExpectedTpi->typeArray(),
+                                                 LocalTypesRecordsBytes);
+    D->IpiHashes = GloballyHashedType::hashIds(
+        ExpectedIpi->typeArray(), D->TpiHashes, LocalIDsRecordsBytes);
 
-    Optional<uint32_t> EndPrecomp;
+    TotalTypesRecordsSize += LocalTypesRecordsBytes + LocalIDsRecordsBytes;
+
+    return D->TpiHashes.size() + D->IpiHashes.size();
+  }
+  return 0;
+}
+
+void PDBLinker::mergePDB(PDBDependency *D) {
+  auto ExpectedTpi = D->Session->getPDBFile().getPDBTpiStream();
+  auto ExpectedIpi = D->Session->getPDBFile().getPDBIpiStream();
+  if (Config->DebugGHashes) {
+    Optional<uint32_t> PCHSignature;
     // Merge TPI first, because the IPI stream will reference type indices.
-    if (auto Err = mergeTypeRecords(GlobalTypeTable, IndexMap.TPIMap,
-                                    ExpectedTpi->typeArray(), TpiHashes, EndPrecomp))
+    if (auto Err = mergeTypeRecords(GlobalTypeTable, D->IndexMap.TPIMap,
+                                    ExpectedTpi->typeArray(), D->TpiHashes,
+                                    PCHSignature))
       fatal("codeview::mergeTypeRecords failed: " + toString(std::move(Err)));
 
     // Merge IPI.
-    if (auto Err =
-            mergeIdRecords(GlobalIDTable, IndexMap.TPIMap, IndexMap.IPIMap,
-                           ExpectedIpi->typeArray(), IpiHashes))
+    if (auto Err = mergeIdRecords(GlobalIDTable, D->IndexMap.TPIMap,
+                                  D->IndexMap.IPIMap, ExpectedIpi->typeArray(),
+                                  D->IpiHashes))
       fatal("codeview::mergeIdRecords failed: " + toString(std::move(Err)));
   } else {
     // Merge TPI first, because the IPI stream will reference type indices.
-    if (auto Err = mergeTypeRecords(TypeTable, IndexMap.TPIMap,
+    if (auto Err = mergeTypeRecords(TypeTable, D->IndexMap.TPIMap,
                                     ExpectedTpi->typeArray()))
       fatal("codeview::mergeTypeRecords failed: " + toString(std::move(Err)));
 
     // Merge IPI.
-    if (auto Err = mergeIdRecords(IDTable, IndexMap.TPIMap, IndexMap.IPIMap,
-                                  ExpectedIpi->typeArray()))
+    if (auto Err = mergeIdRecords(IDTable, D->IndexMap.TPIMap,
+                                  D->IndexMap.IPIMap, ExpectedIpi->typeArray()))
       fatal("codeview::mergeIdRecords failed: " + toString(std::move(Err)));
   }
+}
 
-  return IndexMap;
+static bool equals_path(StringRef path1, StringRef path2) {
+#if defined(_WIN32)
+  return path1.equals_lower(path2);
+#else
+  return path1.equals(path2);
+#endif
 }
 
-Expected<const CVIndexMap &>
-PDBLinker::mergeInPrecompHeaderObj(ObjFile *File, const CVType &FirstType,
-                                   CVIndexMap *ObjectIndexMap) {
+Expected<std::pair<PCHDependency *, PrecompRecord>>
+findPCH(ObjFile *File, CVTypeArray &Stream) {
+  uint32_t Signature = File->PCHSignature.getValueOr(0);
+  if (!Signature)
+    fatal("No signature found for the PCH-dependent OBJ (" + File->getName() +
+          ")");
+
   PrecompRecord Precomp;
-  if (auto EC = TypeDeserializer::deserializeAs(const_cast<CVType &>(FirstType),
-                                                Precomp))
+  if (auto EC = TypeDeserializer::deserializeAs(
+          const_cast<CVType &>(*Stream.begin()), Precomp))
     fatal("error reading record: " + toString(std::move(EC)));
 
-  auto E = aquirePrecompObj(File, Precomp);
-  if (!E)
-    return E.takeError();
+  if (File->PCHSignature.getValue() != Precomp.getSignature())
+    fatal("Corrupted file, the signature does not match the precomp record (" +
+          File->getName() + ")");
 
-  const CVIndexMap &PrecompIndexMap = *E;
-  assert(PrecompIndexMap.IsPrecompiledTypeMap);
+  // link.exe requires that the PCH OBJ must always be provided
+  // on the command-line, even if its path could be inferred from the precomp
+  // record.
+  auto PCH = PCHDependency::Instances.find(Precomp.getSignature());
+  if (PCH == PCHDependency::Instances.end()) {
+    
+    // Cross-compile warning: given that Clang doesn't generate LF_PRECOMP
+    // records, we assume the OBJ comes from MSVC. Thusly, the paths embedded in
+    // the OBJs are in the Windows format.
+    StringRef PrecompFileName = sys::path::filename(
+        Precomp.getPrecompFilePath(), sys::path::Style::windows);
+
+    auto MismatchedPCH = find_if(PCHDependency::Instances, [&](auto KV)
+    {
+      StringRef PCHFilename = sys::path::filename(
+          KV.second->getName(), sys::path::Style::windows);
+
+      // Compare based solely on the file name (link.exe behavior)
+      return equals_path(PCHFilename, PrecompFileName);
+    });
+
+    pdb::pdb_error_code EC = MismatchedPCH != PCHDependency::Instances.end()
+                                 ? pdb::pdb_error_code::signature_out_of_date
+                                 : pdb::pdb_error_code::external_cmdline_ref;
+    return createFileError(Precomp.getPrecompFilePath().str(),
+                           make_error<pdb::PDBError>(EC));
+  }
+  return std::pair<PCHDependency *, PrecompRecord>{PCH->second, Precomp};
+}
+
+// This object was compiled with /Yu, the corresponding precompiled headers
+// object (/Yc) has to be already loaded. Some type indices in the current
+// object are referencing data in the precompiled headers object, so we need
+// both to be in memory.
+Error PDBLinker::mergeInPCH(ObjFile *File, CVTypeArray &Stream,
+                            CVIndexMap &ObjectIndexMap) {
+  auto PCH = findPCH(File, Stream);
+  if (!PCH)
+    return PCH.takeError();
+
+  // Drop LF_PRECOMP record from the input stream, as it needs to be
+  // replaced with the PCH OBJ type stream below.
+  Stream.drop_front();
+
+  PCHDependency *D = PCH->first;
+  PrecompRecord &Precomp = PCH->second;
 
-  if (PrecompIndexMap.TPIMap.empty())
-    return PrecompIndexMap;
+  const CVIndexMap &PrecompIndexMap = D->IndexMap;
+  assert(PrecompIndexMap.IsPrecompiledTypeMap);
 
   assert(Precomp.getStartTypeIndex() == TypeIndex::FirstNonSimpleIndex);
   assert(Precomp.getTypesCount() <= PrecompIndexMap.TPIMap.size());
+
   // Use the previously remapped index map from the precompiled headers.
-  ObjectIndexMap->TPIMap.append(PrecompIndexMap.TPIMap.begin(),
-                                PrecompIndexMap.TPIMap.begin() +
-                                    Precomp.getTypesCount());
-  return *ObjectIndexMap;
+  ObjectIndexMap.TPIMap.append(PrecompIndexMap.TPIMap.begin(),
+  return Error::success();
 }
 
-static bool equals_path(StringRef path1, StringRef path2) {
+/*static bool equals_path(StringRef path1, StringRef path2) {
 #if defined(_WIN32)
   return path1.equals_lower(path2);
 #else
   return path1.equals(path2);
 #endif
-}
+}*/
 
 // Find by name an OBJ provided on the command line
-static ObjFile *findObjByName(StringRef FileNameOnly) {
+/*static ObjFile *findObjByName(StringRef FileNameOnly) {
   SmallString<128> CurrentPath;
 
   for (ObjFile *F : ObjFile::Instances) {
     StringRef CurrentFileName = sys::path::filename(F->getName());
 
     // Compare based solely on the file name (link.exe behavior)
     if (equals_path(CurrentFileName, FileNameOnly))
       return F;
   }
   return nullptr;
-}
+}*/
 
-std::pair<CVIndexMap &, bool /*already there*/>
-PDBLinker::registerPrecompiledHeaders(uint32_t Signature) {
-  auto Insertion = PrecompTypeIndexMappings.insert({Signature, CVIndexMap()});
-  CVIndexMap &IndexMap = Insertion.first->second;
-  if (!Insertion.second)
-    return {IndexMap, true};
-  // Mark this map as a precompiled types map.
-  IndexMap.IsPrecompiledTypeMap = true;
-  return {IndexMap, false};
+Expected<PCHDependency *> PDBLinker::openPCH(ObjFile *File) {
+  uint32_t Signature = File->PCHSignature.getValueOr(0);
+  if (!Signature)
+    fatal("No signature found for the precompiled headers OBJ (" +
+          File->getName() + ")");
+
+  /*if (File->PCHSignature.getValue() != Precomp.getSignature())
+    fatal("Corrupted file, the signature does not match the precomp record (" +
+          File->getName() + ")");*/
+
+  auto I = PCHDependency::Instances.find(Signature);
+  if (I != PCHDependency::Instances.end()) {
+    if (I->second->refObj()->getName() != File->getName())
+      fatal("A precompiled headers OBJ with the same signature was already "
+            "provided! (" +
+            File->getName() + ")");
+    return I->second;
   }
 
-Expected<const CVIndexMap &>
-PDBLinker::aquirePrecompObj(ObjFile *File, PrecompRecord Precomp) {
-  // First, check if we already loaded the precompiled headers object with this
-  // signature. Return the type index mapping if we've already seen it.
-  auto R = registerPrecompiledHeaders(Precomp.getSignature());
-  if (R.second)
-    return R.first;
+  PCHDependency *D = make<PCHDependency>(File);
+  PCHDependency::Instances[Signature] = D;
+
+  auto P = find(ObjFile::Instances, File);
+  ObjFile::Instances.erase(P);
 
-  CVIndexMap &IndexMap = R.first;
+  return D;
+
+  /*CVIndexMap &IndexMap = R.first;
 
   // Cross-compile warning: given that Clang doesn't generate LF_PRECOMP
   // records, we assume the OBJ comes from a Windows build of cl.exe. Thusly,
   // the paths embedded in the OBJs are in the Windows format.
   SmallString<128> PrecompFileName = sys::path::filename(
       Precomp.getPrecompFilePath(), sys::path::Style::windows);
 
   // link.exe requires that a precompiled headers object must always be provided
   // on the command-line, even if that's not necessary.
   auto PrecompFile = findObjByName(PrecompFileName);
   if (!PrecompFile)
     return createFileError(
         PrecompFileName.str(),
         make_error<pdb::PDBError>(pdb::pdb_error_code::external_cmdline_ref));
 
   addObjFile(PrecompFile, &IndexMap);
 
-  if (!PrecompFile->PCHSignature)
+  if (!PrecompFile->EndPrecomp)
     fatal(PrecompFile->getName() + " is not a precompiled headers object");
 
-  if (Precomp.getSignature() != PrecompFile->PCHSignature.getValueOr(0))
+  if (Precomp.getSignature() != PrecompFile->EndPrecomp->getSignature())
     return createFileError(
         Precomp.getPrecompFilePath().str(),
         make_error<pdb::PDBError>(pdb::pdb_error_code::signature_out_of_date));
 
-  return IndexMap;
+  return IndexMap;*/
+}
+
+Expected<unsigned> PDBLinker::computeHash(InputFile *File) {
+  auto Stream = identifyTypeStream(File);
+  switch (Stream.Kind) {
+  case PDB:
+    return computePDBHashes(cast<PDBDependency>(File));
+    break;
+  case PCH: {
+    return computeTypeHashes(cast<PCHDependency>(File)->refObj(), Stream.Types);
+  } break;
+  case UsingPCH: {
+    auto PCH = findPCH(cast<ObjFile>(File), Stream.Types);
+    if (!PCH)
+      return PCH.takeError();
+    auto PrecompStream = identifyTypeStream(PCH->first);
+
+    // We need to merge the previously computed PCH's hashes, however only the
+    // ones in the range [0, Precomp.TypesCount], where Precomp is the first
+    // LF_PRECOMP record read from the current OBJ.
+    auto PrecompHashes = makeArrayRef(PCH->first->refObj()->OwnedHashes.data(),
+                                      PCH->second.getTypesCount());
+
+    // Don't take into account LF_PRECOMP, because it is replaced (virtually in
+    // this case) by the PCH's stream
+    Stream.Types.drop_front();
+
+    return computeTypeHashes(cast<ObjFile>(File), Stream.Types, PrecompHashes);
+  } break;
+  case RegularOBJ:
+    return computeTypeHashes(cast<ObjFile>(File), Stream.Types);
+    break;
+  case UsingPDB:
+  case NoTypeStream:
+    break;
+  }
+  return 0;
 }
 
 static bool remapTypeIndex(TypeIndex &TI, ArrayRef<TypeIndex> TypeIndexMap) {
   if (TI.isSimple())
     return true;
   if (TI.toArrayIndex() >= TypeIndexMap.size())
     return false;
   TI = TypeIndexMap[TI.toArrayIndex()];
   return true;
 }
 
 static void remapTypesInSymbolRecord(ObjFile *File, SymbolKind SymKind,
                                      MutableArrayRef<uint8_t> RecordBytes,
-                                     const CVIndexMap &IndexMap,
+                                     const CVIndexMap *IndexMap,
                                      ArrayRef<TiReference> TypeRefs) {
   MutableArrayRef<uint8_t> Contents =
       RecordBytes.drop_front(sizeof(RecordPrefix));
   for (const TiReference &Ref : TypeRefs) {
-    unsigned ByteSize = Ref.Count * sizeof(TypeIndex);
-    if (Contents.size() < Ref.Offset + ByteSize)
+    unsigned ByteSize = Ref.count() * sizeof(TypeIndex);
+    if (Contents.size() < Ref.offset() + ByteSize)
       fatal("symbol record too short");
 
+    if (!IndexMap) {
+      log("Cannot merge symbol record of kind 0x" + utohexstr(SymKind) + " in " +
+          File->getName() + "; No type stream found in the file.");
+      continue;
+    }
+
     // This can be an item index or a type index. Choose the appropriate map.
-    ArrayRef<TypeIndex> TypeOrItemMap = IndexMap.TPIMap;
-    bool IsItemIndex = Ref.Kind == TiRefKind::IndexRef;
-    if (IsItemIndex && IndexMap.IsTypeServerMap)
-      TypeOrItemMap = IndexMap.IPIMap;
+    ArrayRef<TypeIndex> TypeOrItemMap = IndexMap->TPIMap;
+    bool IsItemIndex = Ref.kind() == TiRefKind::IndexRef;
+    if (IsItemIndex && IndexMap->IsTypeServerMap)
+      TypeOrItemMap = IndexMap->IPIMap;
 
     MutableArrayRef<TypeIndex> TIs(
-        reinterpret_cast<TypeIndex *>(Contents.data() + Ref.Offset), Ref.Count);
+        reinterpret_cast<TypeIndex *>(Contents.data() + Ref.offset()),
+        Ref.count());
     for (TypeIndex &TI : TIs) {
       if (!remapTypeIndex(TI, TypeOrItemMap)) {
         log("ignoring symbol record of kind 0x" + utohexstr(SymKind) + " in " +
             File->getName() + " with bad " + (IsItemIndex ? "item" : "type") +
             " index 0x" + utohexstr(TI.getIndex()));
         TI = TypeIndex(SimpleTypeKind::NotTranslated);
         continue;
       }
static void translateIdSymbols(MutableArrayRef<uint8_t> &RecordData,
   // symbol that refers to the type stream index space.  So we remap again from
   // ID index space to type index space.
   if (Kind == SymbolKind::S_GPROC32_ID || Kind == SymbolKind::S_LPROC32_ID) {
     SmallVector<TiReference, 1> Refs;
     auto Content = RecordData.drop_front(sizeof(RecordPrefix));
     CVSymbol Sym(Kind, RecordData);
     discoverTypeIndicesInSymbol(Sym, Refs);
     assert(Refs.size() == 1);
-    assert(Refs.front().Count == 1);
+    assert(Refs.front().count() == 1);
 
     TypeIndex *TI =
-        reinterpret_cast<TypeIndex *>(Content.data() + Refs[0].Offset);
+        reinterpret_cast<TypeIndex *>(Content.data() + Refs[0].offset());
     // `TI` is the index of a FuncIdRecord or MemberFuncIdRecord which lives in
     // the IPI stream, whose `FunctionType` member refers to the TPI stream.
     // Note that LF_FUNC_ID and LF_MEMFUNC_ID have the same record layout, and
     // in both cases we just need the second type index.
     if (!TI->isSimple() && !TI->isNoneType()) {
       CVType FuncIdData = IDTable.getType(*TI);
       SmallVector<TypeIndex, 2> Indices;
       discoverTypeIndices(FuncIdData, Indices);
       assert(Indices.size() == 2);
       *TI = Indices[1];
     }
 
     Kind = (Kind == SymbolKind::S_GPROC32_ID) ? SymbolKind::S_GPROC32
                                               : SymbolKind::S_LPROC32;
     Prefix->RecordKind = uint16_t(Kind);
   }
 }
 
 /// Copy the symbol record. In a PDB, symbol records must be 4 byte aligned.
 /// The object file may not be aligned.
 static MutableArrayRef<uint8_t>
 copyAndAlignSymbol(const CVSymbol &Sym, MutableArrayRef<uint8_t> &AlignedMem) {
-  size_t Size = alignTo(Sym.length(), alignOf(CodeViewContainer::Pdb));
+  size_t Size = alignTo(Sym.size(), alignOf(CodeViewContainer::Pdb));
   assert(Size >= 4 && "record too short");
   assert(Size <= MaxRecordLength && "record too long");
   assert(AlignedMem.size() >= Size && "didn't preallocate enough");
 
   // Copy the symbol record and zero out any padding bytes.
   MutableArrayRef<uint8_t> NewData = AlignedMem.take_front(Size);
   AlignedMem = AlignedMem.drop_front(Size);
   memcpy(NewData.data(), Sym.data().data(), Sym.length());
case SymbolKind::S_LPROC32: {
     Builder.addGlobalSymbol(PS);
     break;
   }
   default:
     llvm_unreachable("Invalid symbol kind!");
   }
 }
 
-void PDBLinker::mergeSymbolRecords(ObjFile *File, const CVIndexMap &IndexMap,
+void PDBLinker::mergeSymbolRecords(ObjFile *File, const CVIndexMap *IndexMap,
                                    std::vector<ulittle32_t *> &StringTableRefs,
                                    BinaryStreamRef SymData) {
   ArrayRef<uint8_t> SymsBuffer;
   cantFail(SymData.readBytes(0, SymData.getLength(), SymsBuffer));
   SmallVector<SymbolScope, 4> Scopes;
 
   // Iterate every symbol to check if any need to be realigned, and if so, how
   // much space we need to allocate for them.
cantFail(forEachCodeViewRecord<CVSymbol>(
           scopeStackOpen(Scopes, CurSymOffset, Sym);
         else if (symbolEndsScope(Sym.kind()))
           scopeStackClose(Scopes, CurSymOffset, File);
 
         // Add the symbol to the globals stream if necessary.  Do this before
         // adding the symbol to the module since we may need to get the next
         // symbol offset, and writing to the module's symbol stream will update
         // that offset.
-        if (symbolGoesInGlobalsStream(Sym, Scopes.empty()))
+        if (symbolGoesInGlobalsStream(Sym, Scopes.empty())) {
           addGlobalSymbol(Builder.getGsiBuilder(),
                           File->ModuleDBI->getModuleIndex(), CurSymOffset, Sym);
+          GlobalSymbolsCount++;
+        }
 
         if (symbolGoesInModuleStream(Sym, Scopes.empty())) {
           // Add symbols to the module in bulk. If this symbol is contiguous
           // with the previous run of symbols to add, combine the ranges. If
           // not, close the previous range of symbols and start a new one.
           if (Sym.data().data() == BulkSymbols.end()) {
             BulkSymbols = makeArrayRef(BulkSymbols.data(),
                                        BulkSymbols.size() + Sym.length());
           } else {
             File->ModuleDBI->addSymbolsInBulk(BulkSymbols);
             BulkSymbols = RecordBytes;
           }
           CurSymOffset += Sym.length();
+          ModulesSymbolsCount++;
         }
         return Error::success();
       }));
 
   // Add any remaining symbols we've accumulated.
   File->ModuleDBI->addSymbolsInBulk(BulkSymbols);
 }
 
for (FileChecksumEntry &FC : Checksums) {
     pdbMakeAbsolute(FileName);
     ExitOnErr(Linker.Builder.getDbiBuilder().addModuleSourceFile(
         *File.ModuleDBI, FileName));
     NewChecksums->addChecksum(FileName, FC.Kind, FC.Checksum);
   }
   File.ModuleDBI->addDebugSubsection(std::move(NewChecksums));
 }
 
-void PDBLinker::addObjFile(ObjFile *File, CVIndexMap *ExternIndexMap) {
-  if (File->wasProcessedForPDB())
+struct ExitStep : ErrorInfo<ExitStep> {
+  static char ID;
+};
+
+StringRef getFileNamePath(InputFile *File, SmallString<128> &Path) {
+  bool InArchive = !File->ParentName.empty();
+  Path = InArchive ? File->ParentName : File->getName();
+  pdbMakeAbsolute(Path);
+  StringRef Name = InArchive ? File->getName() : StringRef(Path);
+  return Name;
+}
+
+/*void PDBLinker::mergeTypes(InputFile *File) {
+  // Before we can process symbol substreams from .debug$S, we need to process
+  // type information, file checksums, and the string table.  Add type info to
+  // the PDB first, so that we can get the map from object file type and item
+  // indices to PDB type and item indices.
+  CVIndexMap ObjectIndexMap;
+  auto IndexMapResult = mergeDebugT(File, &ObjectIndexMap);
+
+  if (IndexMapResult && !*IndexMapResult)
     return;
+
+  // if (!IndexMapResult && IndexMapResult.errorIsA<ExitStep>())
+  //  return; // probably a step that requests exiting
+
+  // If the .debug$T sections fail to merge, assume there is no debug info.
+  if (!IndexMapResult) {
+    if (!Config->WarnDebugInfoUnusable)
+      return;
+    StringRef FileName = sys::path::filename(Path);
+    warn("Cannot use debug info for '" + FileName + "' [LNK4099]\n" +
+         ">>> failed to load reference " +
+         StringRef(toString(IndexMapResult.takeError())));
+    return;
+  }
+}*/
+
+static void createModuleDescriptor(pdb::PDBFileBuilder &Builder, ObjFile *File) {
+  SmallString<128> Path;
+  StringRef Name = getFileNamePath(File, Path);
+
   // Add a module descriptor for every object file. We need to put an absolute
   // path to the object into the PDB. If this is a plain object, we make its
   // path absolute. If it's an object in an archive, we make the archive path
   // absolute.
-  bool InArchive = !File->ParentName.empty();
-  SmallString<128> Path = InArchive ? File->ParentName : File->getName();
-  pdbMakeAbsolute(Path);
-  StringRef Name = InArchive ? File->getName() : StringRef(Path);
-
   pdb::DbiStreamBuilder &DbiBuilder = Builder.getDbiBuilder();
   File->ModuleDBI = &ExitOnErr(DbiBuilder.addModuleInfo(Name));
   File->ModuleDBI->setObjFileName(Path);
 
   auto Chunks = File->getChunks();
   uint32_t Modi = File->ModuleDBI->getModuleIndex();
   for (Chunk *C : Chunks) {
     auto *SecChunk = dyn_cast<SectionChunk>(C);
     if (!SecChunk || !SecChunk->Live)
       continue;
     pdb::SectionContrib SC = createSectionContrib(SecChunk, Modi);
     File->ModuleDBI->setFirstSectionContrib(SC);
     break;
   }
-
-  // Before we can process symbol substreams from .debug$S, we need to process
-  // type information, file checksums, and the string table.  Add type info to
-  // the PDB first, so that we can get the map from object file type and item
-  // indices to PDB type and item indices.
-  CVIndexMap ObjectIndexMap;
-  auto IndexMapResult =
-      mergeDebugT(File, ExternIndexMap ? ExternIndexMap : &ObjectIndexMap);
-
-  // If the .debug$T sections fail to merge, assume there is no debug info.
-  if (!IndexMapResult) {
-    auto FileName = sys::path::filename(Path);
-    warn("Cannot use debug info for '" + FileName + "'\n" +
-         ">>> failed to load reference " +
-         StringRef(toString(IndexMapResult.takeError())));
-    return;
-  }
 
+void PDBLinker::mergeSymbols(ObjFile *File, CVIndexMap *IndexMap) {
+
   ScopedTimer T(SymbolMergingTimer);
 
-  DebugSHandler DSH(*this, *File, *IndexMapResult);
+  pdb::DbiStreamBuilder &DbiBuilder = Builder.getDbiBuilder();
+  DebugSHandler DSH(*this, *File, IndexMap);
   // Now do all live .debug$S and .debug$F sections.
   for (SectionChunk *DebugChunk : File->getDebugChunks()) {
     if (!DebugChunk->Live || DebugChunk->getSize() == 0)
       continue;
 
     if (DebugChunk->getSectionName() == ".debug$S") {
       DSH.handleDebugS(*DebugChunk);
       continue;
static PublicSym32 createPublic(Defined *Def) {
 
   OutputSection *OS = Def->getChunk()->getOutputSection();
   assert(OS && "all publics should be in final image");
   Pub.Offset = Def->getRVA() - OS->getRVA();
   Pub.Segment = OS->SectionIndex;
   return Pub;
 }
 
+static void warnUnusable(InputFile *F, Error E) {
+  if (!Config->WarnDebugInfoUnusable)
+    return;
+  StringRef FileName = sys::path::filename(F->getName());
+  warn("Cannot use debug info for '" + FileName + "' [LNK4099]\n" +
+         ">>> failed to load reference " + StringRef(toString(std::move(E))));
+}
+
+template <typename Container, typename ElemT>
+void eraseElem(Container &C, ElemT E) {
+  erase_if(C, [&](ElemT E_) { return E_ == E; });
+}
+
+Error PDBLinker::mergeAllOBJ() {
+
+  Summary::addLine(ObjFile::Instances.size(), "Input OBJ files (expanded from all cmd-line inputs)");
+
+  // Create module descriptors
+  for_each(ObjFile::Instances,
+           [&](ObjFile *Obj) { createModuleDescriptor(Builder, Obj); });
+
+  // Gather dependencies
+  for (ObjFile *Obj : std::vector<ObjFile *>(ObjFile::Instances)) {
+    if (auto E = openDependencies(Obj)) {
+      warnUnusable(Obj, std::move(E));
+      eraseElem(ObjFile::Instances, Obj);
+    }
+  }
+
+  Summary::addLine(PDBDependency::Instances.size(), "Dependent PDB files");
+  Summary::addLine(PCHDependency::Instances.size(), "Dependent PCH OBJ files");
+
+  std::vector<InputFile *> Depends;
+  for_each(PDBDependency::Instances,
+           [&](auto KV) { Depends.push_back(KV.second); });
+  for_each(PCHDependency::Instances,
+           [&](auto KV) { Depends.push_back(KV.second); });
+
+  // Compute hashes
+  sys::SmartMutex<true> InvalidFilesMutex;
+  std::vector<InputFile *> InvalidFiles;
+
+  auto ComputeH = [&](InputFile *F) {
+    if (!F)
+      return;
+    auto E = computeHash(F);
+    if (!E) {
+      warnUnusable(F, E.takeError());
+      sys::SmartScopedLock<true> Lock(InvalidFilesMutex);
+      InvalidFiles.push_back(F);
+    } else {
+      TotalTypesRecords += *E;
+    }
+  };
+
+  GlobalHashTimer.start();
+  // Compute Type GHASH-es for all indirect PDB/PCH files
+  parallelForEach(Depends, ComputeH);
+
+  // Compute Type GHASH-es for all remaining OBJ files
+  parallelForEach(ObjFile::Instances, ComputeH);
+  GlobalHashTimer.stop();
+
+  for (InputFile *F : InvalidFiles) {
+    eraseElem(ObjFile::Instances, F);
+    eraseElem(Depends, F);
+  }
+
+  // Merge Types in PDB/PCH files
+  for (InputFile *File : Depends) {
+    if (!File)
+      continue;
+    CVIndexMap TIStorage;
+    auto R = mergeTypes(File, TIStorage);
+    if (!R)
+      return R.takeError();
+
+    if (isa<PCHDependency>(File))
+      mergeSymbols(cast<PCHDependency>(File)->refObj(), *R);
+  }
+
+  // Merge Types and Symbols in all remaining OBJ files
+  for (ObjFile *File : ObjFile::Instances) {
+    CVIndexMap TIStorage;
+    auto R = mergeTypes(File, TIStorage);
+    if (!R)
+      return R.takeError();
+
+    mergeSymbols(File, *R);
+  }
+
+  // Statistics
+  Summary::addLine(TotalTypesRecords.load(),
+                   "Input type records (across all OBJ and dependencies)");
+  Summary::addLine(TotalTypesRecordsSize.load(),
+                   "Input type records bytes (across all OBJ and dependencies)");
+  Summary::addLine(GlobalTypeTable.size() + GlobalIDTable.size(),
+                   "Output merged type records");
+  Summary::addLine(GlobalSymbolsCount.load() + ModulesSymbolsCount.load(),
+                   "Output merged symbol records");
+  Summary::addLine(PDBStrTab.size(), "Output PDB strings");
+
+  return Error::success();
+}
+
 // Add all object files to the PDB. Merge .debug$T sections into IpiData and
 // TpiData.
 void PDBLinker::addObjectsToPDB() {
+
   ScopedTimer T1(AddObjectsTimer);
-  for (ObjFile *File : ObjFile::Instances)
-    addObjFile(File);
+  auto E = mergeAllOBJ();
+  if (E)
+    fatal("Error merging OBJ: " + toString(std::move(E)));
 
   Builder.getStringTableBuilder().setStrings(PDBStrTab);
   T1.stop();
 
   // Construct TPI and IPI stream contents.
   ScopedTimer T2(TpiStreamLayoutTimer);
   addTypeInfo(Builder.getTpiBuilder(), getTypeTable());
   addTypeInfo(Builder.getIpiBuilder(), getIDTable());
   T2.stop();
 
   ScopedTimer T3(GlobalsLayoutTimer);
   // Compute the public and global symbols.
   auto &GsiBuilder = Builder.getGsiBuilder();
   std::vector<PublicSym32> Publics;
   Symtab->forEachSymbol([&Publics](Symbol *S) {
     // Only emit defined, live symbols that have a chunk.
     auto *Def = dyn_cast<Defined>(S);
     if (Def && Def->isLive() && Def->getChunk())
       Publics.push_back(createPublic(Def));
   });
 
   if (!Publics.empty()) {
     // Sort the public symbols and add them to the stream.
-    sort(parallel::par, Publics.begin(), Publics.end(),
+    parallelSort(Publics,
          [](const PublicSym32 &L, const PublicSym32 &R) {
            return L.Name < R.Name;
          });
     for (const PublicSym32 &Pub : Publics)
       GsiBuilder.addPublicSymbol(Pub);
   }
 }
 

lld/trunk/Common/CMakeLists.txt

 if(NOT LLD_BUILT_STANDALONE)
   set(tablegen_deps intrinsics_gen)
 endif()
 
 add_lld_library(lldCommon
   Args.cpp
   ErrorHandler.cpp
   Memory.cpp
   Reproduce.cpp
   Strings.cpp
+  Summary.cpp
   TargetOptionsCommandFlags.cpp
   Threads.cpp
   Timer.cpp
   Version.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLD_INCLUDE_DIR}/lld/Common
 

lld/trunk/Common/Summary.cpp

+//===- Summary.cpp ----------------------------------------------------------===//
+//
+//                             The LLVM Linker
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "lld/Common/ErrorHandler.h"
+#include "lld/Common/Summary.h"
+#include "llvm/ADT/Any.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/Format.h"
+#include "llvm/Support/FormatAdapters.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace lld;
+using namespace llvm;
+
+static void printLine(Any Val, StringRef S) {
+  if (Val.isEqual(0))
+    return;
+  SmallString<80> Str;
+  llvm::raw_svector_ostream Stream(Str);
+  Stream << formatv("{0} {1}", fmt_align(Val, AlignStyle::Right, 15), S.data());
+
+  message(Str);
+}
+
+namespace lld {
+namespace Summary {
+static std::vector<std::pair<Any, std::string>> Lines;
+void addLine(Any Val, StringRef S) { Lines.push_back({Val, S.str()}); };
+void print(bool Banner) {
+  if (Banner) {
+    message("");
+    message(formatv("{0}", fmt_align("Summary", AlignStyle::Center, 80)));
+    message(std::string(80, '-'));
+  }
+  for (auto& L : Lines)
+  {
+    printLine(L.first, L.second);
+  }
+}
+} // namespace Summary
+} // namespace lld

lld/trunk/include/lld/Common/Summary.h

+//===- Summary.h ----------------------------------------------*- C++ -*-===//
+//
+//                             The LLVM Linker
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLD_COMMON_SUMMARY_H
+#define LLD_COMMON_SUMMARY_H
+
+#include "llvm/ADT/StringRef.h"
+  
+namespace lld {
+namespace Summary {
+void addLine(llvm::Any Value, llvm::StringRef S);
+void print(bool Banner);
+}
+} // namespace lld
+
+#endif

lld/trunk/include/lld/Common/Threads.h

 inline void parallelForEachN(size_t Begin, size_t End,
                              llvm::function_ref<void(size_t)> Fn) {
   if (ThreadsEnabled)
     for_each_n(llvm::parallel::par, Begin, End, Fn);
   else
     for_each_n(llvm::parallel::seq, Begin, End, Fn);
 }
 
+template <typename R, class FuncTy> void parallelSort(R &&Range, FuncTy Fn) {
+  if (ThreadsEnabled)
+    sort(llvm::parallel::par, std::begin(Range), std::end(Range), Fn);
+  else
+    sort(llvm::parallel::seq, std::begin(Range), std::end(Range), Fn);
+}
+
 } // namespace lld
 
 #endif

llvm/trunk/include/llvm/ADT/Any.h

 
 class Any {
   template <typename T> struct TypeId { static const char Id; };
 
   struct StorageBase {
     virtual ~StorageBase() = default;
     virtual std::unique_ptr<StorageBase> clone() const = 0;
     virtual const void *id() const = 0;
+    virtual size_t size() const = 0;
+    virtual const void *data() const = 0;
   };
 
   template <typename T> struct StorageImpl : public StorageBase {
     explicit StorageImpl(const T &Value) : Value(Value) {}
 
     explicit StorageImpl(T &&Value) : Value(std::move(Value)) {}
 
     std::unique_ptr<StorageBase> clone() const override {
       return llvm::make_unique<StorageImpl<T>>(Value);
     }
 
     const void *id() const override { return &TypeId<T>::Id; }
 
+    size_t size() const override { return sizeof(T); }
+    const void* data() const override { return &Value; }
+
     T Value;
 
   private:
     StorageImpl &operator=(const StorageImpl &Other) = delete;
     StorageImpl(const StorageImpl &Other) = delete;
   };
 
 public:
template <
               // converted to the parameter type because in that case, this
               // constructor may combine with that conversion during overload
               // resolution for determining copy constructibility, and then
               // when we try to determine copy constructibility below we may
               // infinitely recurse. This is being evaluated by the standards
               // committee as a potential DR in `std::any` as well, but we're
               // going ahead and adopting it to work-around usage of `Any` with
               // types that need to be implicitly convertible from an `Any`.
-              llvm::negation<std::is_convertible<Any, typename std::decay<T>::type>>,
+              llvm::negation<
+                  std::is_convertible<Any, typename std::decay<T>::type>>,
               std::is_copy_constructible<typename std::decay<T>::type>>::value,
           int>::type = 0>
   Any(T &&Value) {
     using U = typename std::decay<T>::type;
     Storage = llvm::make_unique<StorageImpl<U>>(std::forward<T>(Value));
   }
 
   Any(Any &&Other) : Storage(std::move(Other.Storage)) {}
 
   Any &swap(Any &Other) {
     std::swap(Storage, Other.Storage);
     return *this;
   }
 
   Any &operator=(Any Other) {
     Storage = std::move(Other.Storage);
     return *this;
   }
 
   bool hasValue() const { return !!Storage; }
 
   void reset() { Storage.reset(); }
 
+  template <typename T>
+  bool isEqual(const T &Value) {
+    assert(hasValue());
+    if (sizeof(T) != Storage->size())
+      return false;
+    return memcmp(&Value, Storage->data(), sizeof(T)) == 0;
+  }
+
 private:
   template <class T> friend T any_cast(const Any &Value);
   template <class T> friend T any_cast(Any &Value);
   template <class T> friend T any_cast(Any &&Value);
   template <class T> friend const T *any_cast(const Any *Value);
   template <class T> friend T *any_cast(Any *Value);
   template <typename T> friend bool any_isa(const Any &Value);
 
   std::unique_ptr<StorageBase> Storage;
 };
 
 template <typename T> const char Any::TypeId<T>::Id = 0;
 
-
 template <typename T> bool any_isa(const Any &Value) {
   if (!Value.Storage)
     return false;
   using U =
       typename std::remove_cv<typename std::remove_reference<T>::type>::type;
   return Value.Storage->id() == &Any::TypeId<U>::Id;
 }
 

llvm/trunk/include/llvm/ADT/Hashing.h

 /// keeps 56 bytes of arbitrary state.
 struct hash_state {
   uint64_t h0, h1, h2, h3, h4, h5, h6;
 
   /// Create a new hash_state structure and initialize it based on the
   /// seed and the first 64-byte chunk.
   /// This effectively performs the initial mix.
   static hash_state create(const char *s, uint64_t seed) {
-    hash_state state = {
-      0, seed, hash_16_bytes(seed, k1), rotate(seed ^ k1, 49),
-      seed * k1, shift_mix(seed), 0 };
+    hash_state S = create(seed);
+    S.mix(s);
+    return S;
+  }
+
+  static hash_state create(uint64_t seed) {
+    hash_state state = {0,
+                        seed,
+                        hash_16_bytes(seed, k1),
+                        rotate(seed ^ k1, 49),
+                        seed * k1,
+                        shift_mix(seed),
+                        0};
     state.h6 = hash_16_bytes(state.h4, state.h5);
-    state.mix(s);
     return state;
   }
 
   /// Mix 32-bytes from the input sequence into the 16-bytes of 'a'
   /// and 'b', including whatever is already in 'a' and 'b'.
   static void mix_32_bytes(const char *s, uint64_t &a, uint64_t &b) {
     a += fetch64(s);
     uint64_t c = fetch64(s + 24);

llvm/trunk/include/llvm/DebugInfo/CodeView/CVRecord.h

 public:
   CVRecord() : Type(static_cast<Kind>(0)) {}
 
   CVRecord(Kind K, ArrayRef<uint8_t> Data) : Type(K), RecordData(Data) {}
 
   bool valid() const { return Type != static_cast<Kind>(0); }
 
   uint32_t length() const { return RecordData.size(); }
+  uint32_t size() const { return RecordData.size(); }
   Kind kind() const { return Type; }
   ArrayRef<uint8_t> data() const { return RecordData; }
   StringRef str_data() const {
     return StringRef(reinterpret_cast<const char *>(RecordData.data()),
                      RecordData.size());
   }
 
   ArrayRef<uint8_t> content() const {

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h

+//===- llvm/ADT/GlobalTypeDenseMap.h - Dense probed hash table ------------*-
+// C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+//
+// This file defines the GlobalTypeDenseMap class.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H
+#define LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H
+
+#include "llvm/DebugInfo/CodeView/TypeHashing.h"
+#include "llvm/Support/AlignOf.h"
+#include "llvm/Support/Compiler.h"
+#include "llvm/Support/MathExtras.h"
+#include "llvm/Support/Memory.h"
+#include "llvm/Support/ReverseIteration.h"
+#include "llvm/Support/type_traits.h"
+#include <algorithm>
+#include <cassert>
+#include <cstddef>
+#include <cstring>
+#include <initializer_list>
+#include <iterator>
+#include <new>
+#include <type_traits>
+#include <utility>
+
+namespace llvm {
+namespace codeview {
+
+struct GloballyHashedInfo {
+  static unsigned getHash(uint64_t Key, unsigned BucketMask) {
+    return (Key >> 32) & BucketMask;
+  }
+  void packKeyValue(uint64_t Key, uint32_t Value,
+                    unsigned BucketNo, unsigned BucketMask) {
+    KV = (Key & ~((uint64_t)BucketMask << 32)) | ((uint64_t)Value << 32);
+    //auto T = extractKeyValue(BucketNo, BucketMask); 
+    //assert((T == std::pair<uint64_t, uint32_t>(Key, Value)));
+  }
+  std::pair<uint64_t, uint32_t> extractKeyValue(unsigned BucketNo,
+                                                unsigned BucketMask) const {
+    // SHA1 generates keys padded to 0 if the record is too short. The topmost
+    // 32-bits are reliable.
+    uint64_t K =
+        (KV & ~((uint64_t)BucketMask << 32)) | ((uint64_t)BucketNo << 32);
+    uint32_t V = (KV >> 32) & BucketMask;
+    return {K, V};
+  }
+  bool isEmpty() const { return !KV; }
+  unsigned isEqualOrEmpty(const uint64_t Other, unsigned BucketNo,
+                          unsigned BucketMask) const {
+    if (!KV)
+      return 2; // empty
+    auto Extracted = extractKeyValue(BucketNo, BucketMask);
+    if (Extracted.first == Other)
+      return 1; // equal
+    return 0;
+  }
+  uint64_t KV;
+};
+
+template <typename KeyT = GloballyHashedType, typename ValueT = uint32_t,
+          typename BucketT = GloballyHashedInfo>
+class GlobalTypeDenseMap {
+public:
+  using size_type = unsigned;
+  using key_type = KeyT;
+  using mapped_type = ValueT;
+  using value_type = BucketT;
+
+  LLVM_NODISCARD bool empty() const { return getNumEntries() == 0; }
+  unsigned size() const { return getNumEntries(); }
+
+  uint64_t getBucketsMask() const { return getNumBuckets() - 1; }
+
+  // Inserts key,value pair into the map if the key isn't already in the map.
+  // The value is constructed in-place if the key is not in the map, otherwise
+  // it is not moved.
+  template <typename RehashFunc>
+  std::pair<uint32_t, bool> try_emplace(const GloballyHashedType &Key,
+                                               const uint32_t &Value,
+                                               RehashFunc Rehash) {
+    uint64_t InsertKey = *(uint64_t*)const_cast<uint8_t *>(Key.Hash.data());
+    
+    auto R = LookupBucketFor(InsertKey);
+    if (std::get<2>(R)) {
+      auto KV =
+          std::get<1>(R)->extractKeyValue(std::get<0>(R), getBucketsMask());
+      return {KV.second, false /*already there*/};
+    }
+
+    // Otherwise, insert the new element.
+    InsertIntoBucket({std::get<0>(R), std::get<1>(R)}, InsertKey, Value, Rehash);
+    return {Value, true /*inserted*/};
+  }
+
+protected:
+
+  /// Returns the number of buckets to allocate to ensure that the
+  /// GlobalTypeDenseMap can accommodate \p NumEntries without need to grow().
+  unsigned getMinBucketToReserveForEntries(unsigned NumEntries) {
+    // Ensure that "NumEntries * 4 < NumBuckets * 3"
+    if (NumEntries == 0)
+      return 0;
+    // +1 is required because of the strict equality.
+    // For example if NumEntries is 48, we need to return 401.
+    return NextPowerOf2(NumEntries * 4 / 3 + 1);
+  }
+
+private:
+
+  void incrementNumEntries() { setNumEntries(getNumEntries() + 1); }
+
+  void decrementNumEntries() { setNumEntries(getNumEntries() - 1); }
+
+  BucketT *getBucketsEnd() { return getBuckets() + getNumBuckets(); }
+
+  const BucketT *getBucketsEnd() const {
+    return getBuckets() + getNumBuckets();
+  }
+
+  template <typename RehashFunc>
+  void InsertIntoBucket(std::pair<unsigned, BucketT *> TheBucket,
+                        const uint64_t &Key, const uint32_t &Value,
+                        RehashFunc Rehash) {
+    // If the load of the hash table is more than 3/4, grow the table.
+    //
+    // The later case is tricky.  For example, if we had one empty bucket with
+    // tons of tombstones, failing lookups (e.g. for insertion) would have to
+    // probe almost the entire table until it found the empty bucket.  If the
+    // table completely filled with tombstones, no lookup would ever succeed,
+    // causing infinite loops in lookup.
+    unsigned NewNumEntries = getNumEntries() + 1;
+    unsigned NumBuckets = getNumBuckets();
+    if (LLVM_UNLIKELY(NewNumEntries * 4 >= NumBuckets * 3)) {
+      NumEntries = 0;
+      grow(NumBuckets * 2, Rehash);
+      auto R = LookupBucketFor(Key);
+      assert(!std::get<2>(R));
+      TheBucket = {std::get<0>(R), std::get<1>(R)};
+    }
+    assert(std::get<1>(TheBucket));
+
+    // Only update the state after we've grown our bucket space appropriately
+    // so that when growing buckets we have self-consistent entry count.
+    incrementNumEntries();
+
+    std::get<1>(TheBucket)->packKeyValue(Key, Value, std::get<0>(TheBucket),
+                                         getBucketsMask());
+  }
+
+  /// LookupBucketFor - Lookup the appropriate bucket for Val, returning it in
+  /// FoundBucket.  If the bucket contains the key and a value, this returns
+  /// true, otherwise it returns a bucket with an empty marker or tombstone and
+  /// returns false.
+  std::tuple<unsigned /*hashed location (offset)*/,
+             BucketT * /*real key-value location (with collisions)*/,
+             bool /*already there*/>
+  LookupBucketFor(const uint64_t &Key) {
+    if (getNumBuckets() == 0)
+      return {0, nullptr, false};
+
+    assert(Key && "Empty value shouldn't be inserted into map!");
+
+    BucketT *BucketsPtr = getBuckets();
+    const unsigned BucketsMask = getBucketsMask();
+    unsigned BucketNo = BucketT::getHash(Key, BucketsMask);
+    unsigned InitialBucketNo = BucketNo;
+    unsigned ProbeAmt = 1;
+    while (true) {
+      BucketT *ThisBucket = BucketsPtr + BucketNo;
+
+      if (ProbeAmt > MaxProbes)
+        MaxProbes = ProbeAmt;
+      if (ProbeAmt <= MAX_PROBES)
+        Probes[ProbeAmt - 1]++;
+
+      auto R = ThisBucket->isEqualOrEmpty(Key, InitialBucketNo, BucketsMask);
+      if (LLVM_LIKELY(R)) {
+        return {InitialBucketNo, ThisBucket,
+                R == 1 ? true /*bucket equals provided Key*/
+                       : false /*bucket empty*/};
+      }
+
+      // Otherwise, it's a hash collision, continue quadratic probing.
+      BucketNo += ProbeAmt++;
+      BucketNo &= BucketsMask;
+    }
+  }
+
+public:
+  /// Return the approximate size (in bytes) of the actual map.
+  /// This is just the raw memory used by GlobalTypeDenseMap.
+  /// If entries are pointers to objects, the size of the referenced objects
+  /// are not included.
+  size_t getMemorySize() const { return getNumBuckets() * sizeof(BucketT); }
+
+private:
+
+  BucketT *Buckets = nullptr;
+  unsigned NumEntries = 0;
+  unsigned NumBuckets = 0;
+  llvm::sys::MemoryBlock MB;
+  unsigned MaxProbes = 0;
+  enum { MAX_PROBES = 256 };
+  unsigned Probes[MAX_PROBES]{};
+
+public:
+  GlobalTypeDenseMap() { init(0); }
+
+  ~GlobalTypeDenseMap() {
+    deleteBuckets(MB);
+    Buckets = nullptr;
+    NumBuckets = 0;
+    NumEntries = 0;
+  }
+
+  void init(unsigned InitNumEntries) {
+    auto InitBuckets = getMinBucketToReserveForEntries(InitNumEntries);
+    allocateBuckets(InitBuckets);
+  }
+
+  template <typename RehashFunc>
+  void grow(unsigned AtLeast, RehashFunc Rehash) {
+    BucketT *OldBuckets = Buckets;
+    unsigned NewNumBuckets = std::max<unsigned>(
+        64, static_cast<unsigned>(NextPowerOf2(AtLeast - 1)));
+    allocateBuckets(NewNumBuckets);
+    assert(Buckets);
+    if (OldBuckets)
+      Rehash();
+  }
+
+private:
+  unsigned getNumEntries() const { return NumEntries; }
+
+  void setNumEntries(unsigned Num) { NumEntries = Num; }
+
+  BucketT *getBuckets() const { return Buckets; }
+
+  unsigned getNumBuckets() const { return NumBuckets; }
+
+  bool allocateBuckets(unsigned Num) {
+    if (Num == 0)
+      return false;
+
+    size_t BlockSize = sizeof(BucketT) * Num;
+
+    if (MB.size()) {
+      BucketT *MBEnd = (BucketT *)((uint8_t *)MB.base() + MB.size());
+      if ((Buckets + NumBuckets + Num) <= MBEnd) {
+        Buckets += NumBuckets;
+        NumBuckets = Num;
+        return true;
+      }
+      if (BlockSize <= MB.size()) {
+        ::memset(MB.base(), 0, BlockSize);
+        Buckets = (BucketT *)MB.base();
+        NumBuckets = Num;
+        return true;
+      }
+    }
+
+    deleteBuckets(MB);
+
+    const unsigned TwoMegs = 1 << 21;
+    const unsigned Nb2MegPages = (BlockSize + TwoMegs - 1) / TwoMegs;
+
+    const unsigned Flags =
+        sys::Memory::MF_READ | sys::Memory::MF_WRITE | sys::Memory::MF_HUGE;
+
+    std::error_code EC;
+    MB = llvm::sys::Memory::allocateMappedMemory(Nb2MegPages * TwoMegs, nullptr,
+                                                 Flags, EC);
+    Buckets = static_cast<BucketT *>(MB.base());
+    NumBuckets = Num;
+    return true;
+  }
+  static void deleteBuckets(llvm::sys::MemoryBlock &MB) {
+    if (MB.size()) {
+      llvm::sys::Memory::releaseMappedMemory(MB);
+    }
+  }
+};
+} // end namespace codeview
+} // end namespace llvm
+
+#endif // LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPEDENSEMAP_H

llvm/trunk/include/llvm/DebugInfo/CodeView/GlobalTypeTableBuilder.h

 //===- GlobalTypeTableBuilder.h ----------------------------------*- C++-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPETABLEBUILDER_H
 #define LLVM_DEBUGINFO_CODEVIEW_GLOBALTYPETABLEBUILDER_H
 
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/GlobalTypeDenseMap.h" 
 #include "llvm/DebugInfo/CodeView/SimpleTypeSerializer.h"
 #include "llvm/DebugInfo/CodeView/TypeCollection.h"
 #include "llvm/DebugInfo/CodeView/TypeHashing.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/Support/Allocator.h"
 #include <cassert>
 #include <cstdint>
 #include <memory>
 #include <vector>
 
+extern void startGlobalHashTimer();
+extern void stopGlobalHashTimer();
+
 namespace llvm {
 namespace codeview {
 
 class ContinuationRecordBuilder;
 
+#define FAST_HASH
+
 class GlobalTypeTableBuilder : public TypeCollection {
   /// Storage for records.  These need to outlive the TypeTableBuilder.
   BumpPtrAllocator &RecordStorage;
 
   /// A serializer that can write non-continuation leaf types.  Only used as
   /// a convenience function so that we can provide an interface method to
   /// write an unserialized record.
   SimpleTypeSerializer SimpleSerializer;
 
   /// Hash table.
-  DenseMap<GloballyHashedType, TypeIndex> HashedRecords;
+#ifdef FAST_HASH
+  GlobalTypeDenseMap<> HashedRecords;
+#else
+  DenseMap<GloballyHashedType, TypeIndex> HashedRecords2;
+#endif
 
   /// Contains a list of all records indexed by TypeIndex.toArrayIndex().
   SmallVector<ArrayRef<uint8_t>, 2> SeenRecords;
 
   /// Contains a list of all hash values inexed by TypeIndex.toArrayIndex().
   SmallVector<GloballyHashedType, 2> SeenHashes;
 
 public:
   explicit GlobalTypeTableBuilder(BumpPtrAllocator &Storage);
   ~GlobalTypeTableBuilder();
 
   // TypeTableCollection overrides
   Optional<TypeIndex> getFirst() override;
   Optional<TypeIndex> getNext(TypeIndex Prev) override;
   CVType getType(TypeIndex Index) override;
   StringRef getTypeName(TypeIndex Index) override;
   bool contains(TypeIndex Index) override;
   uint32_t size() override;
   uint32_t capacity() override;
 
   // public interface
-  void reset();
   TypeIndex nextTypeIndex() const;
 
   BumpPtrAllocator &getAllocator() { return RecordStorage; }
 
   ArrayRef<ArrayRef<uint8_t>> records() const;
   ArrayRef<GloballyHashedType> hashes() const;
 
   template <typename CreateFunc>
   TypeIndex insertRecordAs(GloballyHashedType Hash, size_t RecordSize,
                            CreateFunc Create) {
-    auto Result = HashedRecords.try_emplace(Hash, nextTypeIndex());
+    // startGlobalHashTimer();
+#ifdef FAST_HASH
+    auto R = HashedRecords.try_emplace(Hash, SeenRecords.size(),
+                                       [&]() { rehashMap(); });
+#else
+    auto R = HashedRecords2.try_emplace(Hash, nextTypeIndex());
+    // assert(R.first == R2.first->second.getIndex());
+    // assert(R.second == R2.second);
+#endif
+    // stopGlobalHashTimer();
 
-    if (LLVM_UNLIKELY(Result.second)) {
+    if (LLVM_UNLIKELY(R.second)) {
       uint8_t *Stable = RecordStorage.Allocate<uint8_t>(RecordSize);
       MutableArrayRef<uint8_t> Data(Stable, RecordSize);
       SeenRecords.push_back(Create(Data));
       SeenHashes.push_back(Hash);
     }
-
-    // Update the caller's copy of Record to point a stable copy.
-    return Result.first->second;
+#ifdef FAST_HASH
+    return TypeIndex::fromArrayIndex(R.first);
+#else
+    return R.first->second;
+#endif
   }
 
   TypeIndex insertRecordBytes(ArrayRef<uint8_t> Data);
   TypeIndex insertRecord(ContinuationRecordBuilder &Builder);
 
   template <typename T> TypeIndex writeLeafType(T &Record) {
     ArrayRef<uint8_t> Data = SimpleSerializer.serialize(Record);
     return insertRecordBytes(Data);
   }
+
+private:
+  void rehashMap();
 };
 
 } // end namespace codeview
 } // end namespace llvm
 
 #endif // LLVM_DEBUGINFO_CODEVIEW_MERGINGTYPETABLEBUILDER_H

llvm/trunk/include/llvm/DebugInfo/CodeView/RecordSerialization.h

 
 /// Limit on the size of all codeview symbol and type records, including the
 /// RecordPrefix. MSVC does not emit any records larger than this.
 enum : unsigned { MaxRecordLength = 0xFF00 };
 
 struct RecordPrefix {
   ulittle16_t RecordLen;  // Record length, starting from &RecordKind.
   ulittle16_t RecordKind; // Record kind enum (SymRecordKind or TypeRecordKind)
+
+  unsigned size() const { return RecordLen + sizeof(RecordKind); }
 };
 
 /// Reinterpret a byte array as an array of characters. Does not interpret as
 /// a C string, as StringRef has several helpers (split) that make that easy.
 StringRef getBytesAsCharacters(ArrayRef<uint8_t> LeafData);
 StringRef getBytesAsCString(ArrayRef<uint8_t> LeafData);
 
 inline Error consume(BinaryStreamReader &Reader) { return Error::success(); }

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeHashing.h

static GloballyHashedType hashType(CVType Type,
                                      ArrayRef<GloballyHashedType> PreviousTypes,
                                      ArrayRef<GloballyHashedType> PreviousIds) {
     return hashType(Type.RecordData, PreviousTypes, PreviousIds);
   }
 
   /// Given a sequence of combined type and ID records, compute global hashes
   /// for each of them, returning the results in a vector of hashed types.
   template <typename Range>
-  static std::vector<GloballyHashedType> hashTypes(Range &&Records) {
+  static std::vector<GloballyHashedType>
+  hashTypes(Range &&Records, uint64_t &TotalRecordsBytes,
+            ArrayRef<GloballyHashedType> PrevTypeHashes = {}) {
     std::vector<GloballyHashedType> Hashes;
-    for (const auto &R : Records)
+    Hashes.insert(Hashes.end(), PrevTypeHashes.begin(), PrevTypeHashes.end());
+    for (const auto &R : Records) {
       Hashes.push_back(hashType(R, Hashes, Hashes));
-
+      TotalRecordsBytes += R.size();
+    }
     return Hashes;
   }
 
+  template <typename Range>
+  static std::vector<GloballyHashedType>
+  hashTypes(Range &&Records, ArrayRef<GloballyHashedType> PrevTypeHashes = {}) {
+    uint64_t TotalRecordsBytes{};
+    return hashTypes(Records, TotalRecordsBytes, PrevTypeHashes);
+  }
+
   /// Given a sequence of combined type and ID records, compute global hashes
   /// for each of them, returning the results in a vector of hashed types.
   template <typename Range>
   static std::vector<GloballyHashedType>
-  hashIds(Range &&Records, ArrayRef<GloballyHashedType> TypeHashes) {
+  hashIds(Range &&Records, ArrayRef<GloballyHashedType> TypeHashes,
+          uint64_t &TotalRecordsBytes) {
     std::vector<GloballyHashedType> IdHashes;
-    for (const auto &R : Records)
+    for (const auto &R : Records) {
       IdHashes.push_back(hashType(R, TypeHashes, IdHashes));
-
+      TotalRecordsBytes += R.size();
+    }
     return IdHashes;
   }
 
+  template <typename Range>
+  static std::vector<GloballyHashedType>
+  hashIds(Range &&Records, ArrayRef<GloballyHashedType> TypeHashes) {
+    uint64_t TotalRecordsBytes{};
+    return hashIds(Records, TypeHashes, TotalRecordsBytes);  
+  }
+
   static std::vector<GloballyHashedType>
   hashTypeCollection(TypeCollection &Types) {
     std::vector<GloballyHashedType> Hashes;
     Types.ForEachRecord([&Hashes](TypeIndex TI, const CVType &Type) {
       Hashes.push_back(hashType(Type.RecordData, Hashes, Hashes));
     });
     return Hashes;
   }
+
+  enum HashType
+  {
+    CityHash,
+    SHA1,
+    MD5
+  };
+  static HashType H;
 };
 #if defined(_MSC_VER)
 // is_trivially_copyable is not available in older versions of libc++, but it is
 // available in all supported versions of MSVC, so at least this gives us some
 // coverage.
 static_assert(std::is_trivially_copyable<GloballyHashedType>::value,
               "GloballyHashedType must be trivially copyable so that we can "
               "reinterpret_cast arrays of hash data to arrays of "

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeIndexDiscovery.h

 #include "llvm/DebugInfo/CodeView/SymbolRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace codeview {
 enum class TiRefKind { TypeRef, IndexRef };
 struct TiReference {
-  TiRefKind Kind;
-  uint32_t Offset;
-  uint32_t Count;
+  TiReference(TiRefKind K, unsigned O, unsigned C) {
+    // Offset cannot be more than llvm::codeview::MaxRecordLength
+    Data = ((unsigned)K & 1) | ((C & 0x7FFF) << 1) | ((O & 0xFFFF) << 16);
+  }
+  unsigned offset() const { return Data >> 16; }
+  unsigned count() const { return (Data >> 1) & 0x7FFF; }
+  TiRefKind kind() const { return (TiRefKind)(Data & 1); }
+
+private:
+  uint32_t Data;
 };
 
 void discoverTypeIndices(ArrayRef<uint8_t> RecordData,
                          SmallVectorImpl<TiReference> &Refs);
 void discoverTypeIndices(const CVType &Type,
                          SmallVectorImpl<TiReference> &Refs);
 void discoverTypeIndices(const CVType &Type,
                          SmallVectorImpl<TypeIndex> &Indices);
 void discoverTypeIndices(ArrayRef<uint8_t> RecordData,
                          SmallVectorImpl<TypeIndex> &Indices);
 
 /// Discover type indices in symbol records. Returns false if this is an unknown
 /// record.
 bool discoverTypeIndicesInSymbol(const CVSymbol &Symbol,
                                  SmallVectorImpl<TiReference> &Refs);
 bool discoverTypeIndicesInSymbol(ArrayRef<uint8_t> RecordData,
                                  SmallVectorImpl<TiReference> &Refs);
 bool discoverTypeIndicesInSymbol(ArrayRef<uint8_t> RecordData,
                                  SmallVectorImpl<TypeIndex> &Indices);
-}
-}
+} // namespace codeview
+} // namespace llvm
 
 #endif

llvm/trunk/include/llvm/Support/BinaryStreamArray.h

public:
   BinaryStreamRef getUnderlyingStream() const { return Stream; }
   void setUnderlyingStream(BinaryStreamRef S, uint32_t Skew = 0) {
     Stream = S;
     this->Skew = Skew;
   }
 
   void drop_front() { Skew += begin()->length(); }
 
+  ArrayRef<uint8_t> getBytes() const {
+    ArrayRef<uint8_t> Buffer;
+    cantFail(Stream.readBytes(Skew, Stream.getLength() - Skew, Buffer));
+    return Buffer;
+  }
+
 private:
   BinaryStreamRef Stream;
   Extractor E;
   uint32_t Skew;
 };
 
 template <typename ValueType, typename Extractor>
 class VarStreamArrayIterator

llvm/trunk/include/llvm/Support/CityHash.h

+//==- SHA1.h - SHA1 implementation for LLVM                     --*- C++ -*-==//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_SUPPORT_CITYHASH_H
+#define LLVM_SUPPORT_CITYHASH_H
+
+#include "llvm/ADT/ArrayRef.h"
+
+namespace llvm {
+
+/// A class that wraps LLVM's version of the CityHash algorithm.
+class CityHash {
+public:
+  CityHash() { init(); }
+
+  /// Reinitialize the internal state
+  void init();
+
+  /// Digest more data.
+  void update(ArrayRef<uint8_t> Data);
+
+  /// Digest more data.
+  void update(StringRef Str) {
+    update(ArrayRef<uint8_t>((uint8_t *)const_cast<char *>(Str.data()),
+                             Str.size()));
+  }
+
+  StringRef final();
+
+  /// Computes the hash for a given bytes.
+  static std::array<uint8_t, 8> hash(ArrayRef<uint8_t> Data);
+
+private:
+  void internalUpdateState();
+
+private:
+  hashing::detail::hash_state S;
+  uint8_t Buffer[64]{};
+  unsigned BufferFree = 64;
+  size_t TotalLength{};
+  uint64_t Result{};
+  bool Created = false;
+};
+
+std::array<uint8_t, 8> CityHash::hash(ArrayRef<uint8_t> Data) {
+  hash_code H = hash_value(Data);
+  return *(std::array<uint8_t, 8> *)&H;
+}
+
+void CityHash::init() {}
+
+void CityHash::update(ArrayRef<uint8_t> Data) {
+  const uint8_t *Src = Data.data();
+  size_t SrcCount = Data.size();
+  TotalLength += SrcCount;
+  do {
+    if (BufferFree > 0) {
+      unsigned Copied = std::min(BufferFree, (unsigned)SrcCount);
+      memcpy(Buffer, Src, Copied);
+      BufferFree -= Copied;
+      SrcCount -= Copied;
+      Src += Copied;
+    }
+    if (BufferFree == 0) {
+      internalUpdateState();
+    }
+  } while (SrcCount > 0);
+}
+
+StringRef CityHash::final() {
+  if (TotalLength <= 64) {
+    Result = hashing::detail::hash_short((const char*)Buffer, TotalLength,
+                                         hashing::detail::get_execution_seed());
+  } else {
+    internalUpdateState();
+    Result = S.finalize(TotalLength);
+  }
+  return StringRef((const char *)&Result, 8);
+}
+
+void CityHash::internalUpdateState() {
+  if (BufferFree == 64)
+    return;
+  if (!Created) {
+    Created = true;
+    S = hashing::detail::hash_state::create(
+        hashing::detail::get_execution_seed());
+  }
+  if (BufferFree > 0)
+    memset(Buffer + 64 - BufferFree, 0, BufferFree);
+
+  S.mix((const char *)Buffer);
+  BufferFree = 64;
+}
+} // namespace llvm
+
+#endif

llvm/trunk/include/llvm/Support/FormatProviders.h

 // This file implements format providers for many common LLVM types, for example
 // allowing precision and width specifiers for scalar and string types.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_SUPPORT_FORMATPROVIDERS_H
 #define LLVM_SUPPORT_FORMATPROVIDERS_H
 
+#include "llvm/ADT/Any.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/ADT/Twine.h"
 #include "llvm/Support/FormatVariadicDetails.h"
 #include "llvm/Support/NativeFormatting.h"
 
 #include <type_traits>
while (Begin != End) {
       Stream << Sep;
       auto Adapter =
           detail::build_format_adapter(std::forward<reference>(*Begin));
       Adapter.format(Stream, ArgStyle);
       ++Begin;
     }
   }
 };
+
+/// Implementation of format_provider<T> for llvm::Any.
+
+#define IMPLEMENT_ANY(T) \
+  if (any_isa<T>(V)) { \
+    format_provider<T>::format(llvm::any_cast<T>(V), Stream, Style); \
+    return; \
+  }
+
+template <> struct format_provider<llvm::Any> {
+  static void format(const Any &V, llvm::raw_ostream &Stream,
+                     StringRef Style) {
+    IMPLEMENT_ANY(uint32_t);
+    IMPLEMENT_ANY(int32_t);
+    IMPLEMENT_ANY(uint64_t);
+    IMPLEMENT_ANY(int64_t);
+    IMPLEMENT_ANY(float);
+    IMPLEMENT_ANY(double);
   }
+};
+
+#undef IMPLEMENT_ANY
+
+} // namespace llvm
 
 #endif

llvm/trunk/include/llvm/Support/MD5.h

public:
   void update(ArrayRef<uint8_t> Data);
 
   /// Updates the hash for the StringRef provided.
   void update(StringRef Str);
 
   /// Finishes off the hash and puts the result in result.
   void final(MD5Result &Result);
 
+  StringRef final() {
+    final(Result);
+    return {StringRef((char *)Result.Bytes.data(), Result.Bytes.size())};
+  }
+
   /// Translates the bytes in \p Res to a hex string that is
   /// deposited into \p Str. The result will be of length 32.
   static void stringifyResult(MD5Result &Result, SmallString<32> &Str);
 
   /// Computes the hash for a given bytes.
   static std::array<uint8_t, 16> hash(ArrayRef<uint8_t> Data);
 
 private:
   const uint8_t *body(ArrayRef<uint8_t> Data);
+
+  MD5Result Result;
 };
 
 inline bool operator==(const MD5::MD5Result &LHS, const MD5::MD5Result &RHS) {
   return LHS.Bytes == RHS.Bytes;
 }
 
 /// Helper to compute and return lower 64 bits of the given string's MD5 hash.
 inline uint64_t MD5Hash(StringRef Str) {

llvm/trunk/include/llvm/Support/Memory.h

namespace sys {
 
   /// This class provides various memory handling functions that manipulate
   /// MemoryBlock instances.
   /// @since 1.4
   /// An abstraction for memory operations.
   class Memory {
   public:
     enum ProtectionFlags {
-      MF_READ  = 0x1000000,
-      MF_WRITE = 0x2000000,
-      MF_EXEC  = 0x4000000
+      MF_READ     = 0x1000000,
+      MF_WRITE    = 0x2000000,
+      MF_EXEC     = 0x4000000,
+      MF_RWE_MASK = 0x7000000,
+
+      MF_HUGE     = 0x0000001
     };
 
     /// This method allocates a block of memory that is suitable for loading
     /// dynamically generated code (e.g. JIT). An attempt to allocate
     /// \p NumBytes bytes of virtual memory is made.
     /// \p NearBlock may point to an existing allocation in which case
     /// an attempt is made to allocate more memory near the existing block.
     /// The actual allocated address is not guaranteed to be near the requested

llvm/trunk/lib/DebugInfo/CodeView/GlobalTypeTableBuilder.cpp

 ArrayRef<ArrayRef<uint8_t>> GlobalTypeTableBuilder::records() const {
   return SeenRecords;
 }
 
 ArrayRef<GloballyHashedType> GlobalTypeTableBuilder::hashes() const {
   return SeenHashes;
 }
 
-void GlobalTypeTableBuilder::reset() {
-  HashedRecords.clear();
-  SeenRecords.clear();
-}
-
 TypeIndex GlobalTypeTableBuilder::insertRecordBytes(ArrayRef<uint8_t> Record) {
   GloballyHashedType GHT =
       GloballyHashedType::hashType(Record, SeenHashes, SeenHashes);
   return insertRecordAs(GHT, Record.size(),
                         [Record](MutableArrayRef<uint8_t> Data) {
                           assert(Data.size() == Record.size());
                           ::memcpy(Data.data(), Record.data(), Record.size());
                           return Data;
                         });
 }
 
 TypeIndex
 GlobalTypeTableBuilder::insertRecord(ContinuationRecordBuilder &Builder) {
   TypeIndex TI;
   auto Fragments = Builder.end(nextTypeIndex());
   assert(!Fragments.empty());
   for (auto C : Fragments)
     TI = insertRecordBytes(C.RecordData);
   return TI;
 }
+
+void GlobalTypeTableBuilder::rehashMap() {
+#ifdef FAST_HASH
+  static bool Reentrance = false;
+  assert(!Reentrance);
+  Reentrance = true;
+  for (size_t I = 0; I < SeenHashes.size(); ++I) {
+    auto R = HashedRecords.try_emplace(SeenHashes[I], I, []() {});
+    (void)R;
+    assert(R.second /*inserted*/);
+  }
+  Reentrance = false;
+#endif
+}
 No newline at end of file

llvm/trunk/lib/DebugInfo/CodeView/TypeHashing.cpp

 //===- TypeHashing.cpp -------------------------------------------*- C++-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/CodeView/TypeHashing.h"
 
 #include "llvm/DebugInfo/CodeView/TypeIndexDiscovery.h"
+#include "llvm/Support/CityHash.h"
+#include "llvm/Support/MD5.h"
 #include "llvm/Support/SHA1.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 
 LocallyHashedType DenseMapInfo<LocallyHashedType>::Empty{0, {}};
 LocallyHashedType DenseMapInfo<LocallyHashedType>::Tombstone{hash_code(-1), {}};
 
 static std::array<uint8_t, 8> EmptyHash = {
     {0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}};
 static std::array<uint8_t, 8> TombstoneHash = {
     {0xFF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}};
 
 GloballyHashedType DenseMapInfo<GloballyHashedType>::Empty{EmptyHash};
 GloballyHashedType DenseMapInfo<GloballyHashedType>::Tombstone{TombstoneHash};
 
 LocallyHashedType LocallyHashedType::hashType(ArrayRef<uint8_t> RecordData) {
   return {llvm::hash_value(RecordData), RecordData};
 }
 
+GloballyHashedType::HashType GloballyHashedType::H = GloballyHashedType::HashType::SHA1;
+
+/*namespace {
+struct Hasher {
+  void update(ArrayRef<uint8_t> Data) {
+    switch (GloballyHashedType::H) {
+    case GloballyHashedType::HashType::CityHash:
+      H1.update(Data);
+      break;
+    case GloballyHashedType::HashType::SHA1:
+      H2.update(Data);
+      break;
+    case GloballyHashedType::HashType::MD5:
+      H3.update(Data);
+      break;
+    }
+  }
+  StringRef final() {
+    switch (GloballyHashedType::H) {
+    case GloballyHashedType::HashType::CityHash:
+      return H1.final();
+    case GloballyHashedType::HashType::SHA1:
+      return H2.final();
+    case GloballyHashedType::HashType::MD5:
+      return H3.final();
+    }
+    return{};
+  }
+  CityHash H1;
+  SHA1 H2;
+  MD5 H3;
+};
+} // namespace*/
+
+static uint32_t MaxDiscover = 0;
+
+template <typename R, typename ArrayT> void append(ArrayT &A, R &&Range) {
+  A.append(adl_begin(Range), adl_end(Range));
+}
+
 GloballyHashedType
 GloballyHashedType::hashType(ArrayRef<uint8_t> RecordData,
                              ArrayRef<GloballyHashedType> PreviousTypes,
                              ArrayRef<GloballyHashedType> PreviousIds) {
-  SmallVector<TiReference, 4> Refs;
+  SmallVector<TiReference, 2048> Refs;
   discoverTypeIndices(RecordData, Refs);
-  SHA1 S;
-  S.init();
+  if (Refs.size() > MaxDiscover)
+    MaxDiscover = Refs.size();
+  SmallVector<uint8_t, codeview::MaxRecordLength> Data;
   uint32_t Off = 0;
-  S.update(RecordData.take_front(sizeof(RecordPrefix)));
+  append(Data, RecordData.take_front(sizeof(RecordPrefix)));
   RecordData = RecordData.drop_front(sizeof(RecordPrefix));
   for (const auto &Ref : Refs) {
     // Hash any data that comes before this TiRef.
-    uint32_t PreLen = Ref.Offset - Off;
+    uint32_t PreLen = Ref.offset() - Off;
     ArrayRef<uint8_t> PreData = RecordData.slice(Off, PreLen);
-    S.update(PreData);
-    auto Prev = (Ref.Kind == TiRefKind::IndexRef) ? PreviousIds : PreviousTypes;
+    append(Data, PreData);
+    auto Prev = (Ref.kind() == TiRefKind::IndexRef) ? PreviousIds : PreviousTypes;
 
-    auto RefData = RecordData.slice(Ref.Offset, Ref.Count * sizeof(TypeIndex));
+    auto RefData = RecordData.slice(Ref.offset(), Ref.count() * sizeof(TypeIndex));
     // For each type index referenced, add in the previously computed hash
     // value of that type.
     ArrayRef<TypeIndex> Indices(
-        reinterpret_cast<const TypeIndex *>(RefData.data()), Ref.Count);
+        reinterpret_cast<const TypeIndex *>(RefData.data()), Ref.count());
     for (TypeIndex TI : Indices) {
       ArrayRef<uint8_t> BytesToHash;
       if (TI.isSimple() || TI.isNoneType() || TI.toArrayIndex() >= Prev.size()) {
         const uint8_t *IndexBytes = reinterpret_cast<const uint8_t *>(&TI);
         BytesToHash = makeArrayRef(IndexBytes, sizeof(TypeIndex));
       } else {
         BytesToHash = Prev[TI.toArrayIndex()].Hash;
       }
-      S.update(BytesToHash);
+      append(Data, BytesToHash);
     }
 
-    Off = Ref.Offset + Ref.Count * sizeof(TypeIndex);
+    Off = Ref.offset() + Ref.count() * sizeof(TypeIndex);
   }
 
   // Don't forget to add in any trailing bytes.
   auto TrailingBytes = RecordData.drop_front(Off);
-  S.update(TrailingBytes);
+  append(Data, TrailingBytes);
 
-  return {S.final().take_back(8)};
+  switch (GloballyHashedType::H) {
+  case GloballyHashedType::HashType::CityHash:
+    return makeArrayRef(CityHash::hash(Data).data(), 8);
+  case GloballyHashedType::HashType::SHA1:
+    return makeArrayRef(SHA1::hash(Data).data(), 8);
+  case GloballyHashedType::HashType::MD5:
+    return makeArrayRef(MD5::hash(Data).data(), 8);
+  }
+  return {};
 }

llvm/trunk/lib/DebugInfo/CodeView/TypeIndexDiscovery.cpp

static void resolveTypeIndexReferences(ArrayRef<uint8_t> RecordData,
 
   if (Refs.empty())
     return;
 
   RecordData = RecordData.drop_front(sizeof(RecordPrefix));
 
   BinaryStreamReader Reader(RecordData, support::little);
   for (const auto &Ref : Refs) {
-    Reader.setOffset(Ref.Offset);
+    Reader.setOffset(Ref.offset());
     FixedStreamArray<TypeIndex> Run;
-    cantFail(Reader.readArray(Run, Ref.Count));
+    cantFail(Reader.readArray(Run, Ref.count()));
     Indices.append(Run.begin(), Run.end());
   }
 }
 
 void llvm::codeview::discoverTypeIndices(const CVType &Type,
                                          SmallVectorImpl<TypeIndex> &Indices) {
   return discoverTypeIndices(Type.RecordData, Indices);
 }

llvm/trunk/lib/DebugInfo/CodeView/TypeStreamMerger.cpp

Error TypeStreamMerger::doit(const CVTypeArray &Types) {
   }
 
   if (LastError)
     return std::move(*LastError);
   return Error::success();
 }
 
 Error TypeStreamMerger::remapAllTypes(const CVTypeArray &Types) {
-  BinaryStreamRef Stream = Types.getUnderlyingStream();
-  ArrayRef<uint8_t> Buffer;
-  cantFail(Stream.readBytes(0, Stream.getLength(), Buffer));
-
   return forEachCodeViewRecord<CVType>(
-      Buffer, [this](const CVType &T) { return remapType(T); });
+      Types.getBytes(), [this](const CVType &T) { return remapType(T); });
 }
 
 Error TypeStreamMerger::remapType(const CVType &Type) {
   auto R = shouldRemapType(Type);
   if (!R)
     return R.takeError();
 
   TypeIndex DestIdx = Untranslated;
Error TypeStreamMerger::remapType(const CVType &Type) {
   assert((IsSecondPass || IndexMap.size() == slotForIndex(CurIndex)) &&
          "visitKnownRecord should add one index map entry");
   return Error::success();
 }
 
 ArrayRef<uint8_t>
 TypeStreamMerger::remapIndices(const CVType &OriginalType,
                                MutableArrayRef<uint8_t> Storage) {
-  SmallVector<TiReference, 4> Refs;
+  SmallVector<TiReference, 2048> Refs;
   discoverTypeIndices(OriginalType.RecordData, Refs);
   if (Refs.empty())
     return OriginalType.RecordData;
 
   ::memcpy(Storage.data(), OriginalType.RecordData.data(),
            OriginalType.RecordData.size());
 
   uint8_t *DestContent = Storage.data() + sizeof(RecordPrefix);
 
   for (auto &Ref : Refs) {
     TypeIndex *DestTIs =
-        reinterpret_cast<TypeIndex *>(DestContent + Ref.Offset);
+        reinterpret_cast<TypeIndex *>(DestContent + Ref.offset());
 
-    for (size_t I = 0; I < Ref.Count; ++I) {
+    for (size_t I = 0; I < Ref.count(); ++I) {
       TypeIndex &TI = DestTIs[I];
-      bool Success = (Ref.Kind == TiRefKind::IndexRef) ? remapItemIndex(TI)
+      bool Success = (Ref.kind() == TiRefKind::IndexRef) ? remapItemIndex(TI)
                                                        : remapTypeIndex(TI);
       if (LLVM_UNLIKELY(!Success))
         return {};
     }
   }
   return Storage;
 }
 

llvm/trunk/lib/Support/Windows/Memory.inc

 #include "llvm/Support/WindowsError.h"
 
 // The Windows.h header must be the last one included.
 #include "WindowsSupport.h"
 
 namespace {
 
 DWORD getWindowsProtectionFlags(unsigned Flags) {
-  switch (Flags) {
+  switch (Flags & llvm::sys::Memory::MF_RWE_MASK) {
   // Contrary to what you might expect, the Windows page protection flags
   // are not a bitwise combination of RWX values
   case llvm::sys::Memory::MF_READ:
     return PAGE_READONLY;
   case llvm::sys::Memory::MF_WRITE:
     // Note: PAGE_WRITE is not supported by VirtualProtect
     return PAGE_READWRITE;
   case llvm::sys::Memory::MF_READ|llvm::sys::Memory::MF_WRITE:
size_t getAllocationGranularity() {
   SYSTEM_INFO  Info;
   ::GetSystemInfo(&Info);
   if (Info.dwPageSize > Info.dwAllocationGranularity)
     return Info.dwPageSize;
   else
     return Info.dwAllocationGranularity;
 }
 
+size_t getLargePageSize() {
+  HANDLE Token = 0;
+  size_t LargePageMin = GetLargePageMinimum();
+  if (LargePageMin)
+    OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY,
+                     &Token);
+  if (Token) {
+    LUID Luid;
+    if (LookupPrivilegeValue(0, SE_LOCK_MEMORY_NAME, &Luid)) {
+      TOKEN_PRIVILEGES TP{};
+      TP.PrivilegeCount = 1;
+      TP.Privileges[0].Luid = Luid;
+      TP.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
+      if (AdjustTokenPrivileges(Token, FALSE, &TP, 0, 0, 0)) {
+        DWORD E = GetLastError();
+        if (E == ERROR_SUCCESS) {
+          return LargePageMin;
+        }
+      }
+    }
+    CloseHandle(Token);
+  }
+  return 0;
+}
+
 } // namespace
 
 namespace llvm {
 namespace sys {
 
 //===----------------------------------------------------------------------===//
 //=== WARNING: Implementation here must contain only Win32 specific code
 //===          and must not be UNIX code
MemoryBlock Memory::allocateMappedMemory(size_t NumBytes,
   // aren't thread safe with MSVC.
   static volatile size_t GranularityCached;
   size_t Granularity = GranularityCached;
   if (Granularity == 0) {
     Granularity = getAllocationGranularity();
     GranularityCached = Granularity;
   }
 
-  const size_t NumBlocks = (NumBytes+Granularity-1)/Granularity;
+  static size_t LargePageSize = getLargePageSize();
+  unsigned HugePages = ((Flags & MF_HUGE) && LargePageSize) ? MEM_LARGE_PAGES : 0;
+  if (HugePages) {
+    Granularity = LargePageSize;
+  }
+       
+  size_t NumBlocks = (NumBytes+Granularity-1)/Granularity;
 
   uintptr_t Start = NearBlock ? reinterpret_cast<uintptr_t>(NearBlock->base()) +
                                 NearBlock->size()
                            : 0;
 
   // If the requested address is not aligned to the allocation granularity,
   // round up to get beyond NearBlock. VirtualAlloc would have rounded down.
   if (Start && Start % Granularity != 0)
     Start += Granularity - Start % Granularity;
 
   DWORD Protect = getWindowsProtectionFlags(Flags);
   
   void *PA = ::VirtualAlloc(reinterpret_cast<void*>(Start),
                             NumBlocks*Granularity,
-                            MEM_RESERVE | MEM_COMMIT, Protect);
+                            MEM_RESERVE | MEM_COMMIT | HugePages, Protect);
   if (PA == NULL) {
     if (NearBlock) {
       // Try again without the NearBlock hint
       return allocateMappedMemory(NumBytes, NULL, Flags, EC);
     }
     EC = mapWindowsError(::GetLastError());
     return MemoryBlock();
   }

llvm/trunk/tools/llvm-pdbutil/DumpOutputStyle.cpp

Error DumpOutputStyle::dumpGSIRecords() {
   SymbolVisitorCallbackPipeline Pipeline;
   SymbolDeserializer Deserializer(nullptr, CodeViewContainer::Pdb);
   MinimalSymbolDumper Dumper(P, opts::dump::DumpSymRecordBytes, Ids, Types);
 
   Pipeline.addCallbackToPipeline(Deserializer);
   Pipeline.addCallbackToPipeline(Dumper);
   CVSymbolVisitor Visitor(Pipeline);
 
-  BinaryStreamRef SymStream = Records.getSymbolArray().getUnderlyingStream();
   if (auto E = Visitor.visitSymbolStream(Records.getSymbolArray(), 0))
     return E;
   return Error::success();
 }
 
 Error DumpOutputStyle::dumpGlobals() {
   printHeader(P, "Global Symbols");
 
if (HashExtras) {
     SymbolVisitorCallbackPipeline Pipeline;
     SymbolDeserializer Deserializer(nullptr, CodeViewContainer::Pdb);
     MinimalSymbolDumper Dumper(P, opts::dump::DumpSymRecordBytes, Ids, Types);
 
     Pipeline.addCallbackToPipeline(Deserializer);
     Pipeline.addCallbackToPipeline(Dumper);
     CVSymbolVisitor Visitor(Pipeline);
 
-
     BinaryStreamRef SymStream =
         ExpectedSyms->getSymbolArray().getUnderlyingStream();
     for (uint32_t PubSymOff : Table) {
       Expected<CVSymbol> Sym = readSymbolFromStream(SymStream, PubSymOff);
       if (!Sym)
         return Sym.takeError();
       if (auto E = Visitor.visitSymbolRecord(*Sym, PubSymOff))
         return E;

llvm/trunk/unittests/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp

protected:
 
   std::unique_ptr<AppendingTypeTableBuilder> TTB;
 
 private:
   uint32_t countRefs(uint32_t RecordIndex) const {
     auto &R = Refs[RecordIndex];
     uint32_t Count = 0;
     for (auto &Ref : R) {
-      Count += Ref.Count;
+      Count += Ref.count();
     }
     return Count;
   }
 
   bool checkOneTypeReference(uint32_t RecordIndex, ArrayRef<uint8_t> RecordData,
                              TypeIndex TI) const {
     RecordData = RecordData.drop_front(sizeof(RecordPrefix));
     auto &RefList = Refs[RecordIndex];
     for (auto &Ref : RefList) {
-      uint32_t Offset = Ref.Offset;
+      uint32_t Offset = Ref.offset();
       ArrayRef<uint8_t> Loc = RecordData.drop_front(Offset);
       ArrayRef<TypeIndex> Indices(
-          reinterpret_cast<const TypeIndex *>(Loc.data()), Ref.Count);
+          reinterpret_cast<const TypeIndex *>(Loc.data()), Ref.count());
       if (llvm::any_of(Indices,
                        [TI](const TypeIndex &Other) { return Other == TI; }))
         return true;
     }
     return false;
   }
 
   template <typename... Indices>