[PDB] Enable parallel ghash type merging by default

[rnk]

Summary

Ghashing is probably going to be faster in most cases, even without
precomputed ghashes in object files.

Here is my table of results linking clang.pdb:

threads	GHASH	NOGHASH
j1	51.031s	25.141s
j2	31.079s	22.109s
j4	18.609s	23.156s
j8	11.938s	21.984s
j28	8.375s	18.391s

This shows that ghashing is faster if at least four cores are available.
This may make the linker slower if most cores are busy in the middle of
a build, but in that case, the linker probably isn't on the critical
path of the build. Incremental build performance is arguably more
important than highly contended batch build link performance.

The -time output indicates that ghash computation is the dominant
factor:

  Input File Reading:             924 ms (  1.8%)
  GC:                             689 ms (  1.3%)
  ICF:                            527 ms (  1.0%)
  Code Layout:                    414 ms (  0.8%)
  Commit Output File:              24 ms (  0.0%)
  PDB Emission (Cumulative):    49938 ms ( 94.8%)
    Add Objects:                46783 ms ( 88.8%)
      Global Type Hashing:      38983 ms ( 74.0%)
      GHash Type Merging:        5640 ms ( 10.7%)
      Symbol Merging:            2154 ms (  4.1%)
    Publics Stream Layout:        188 ms (  0.4%)
    TPI Stream Layout:             18 ms (  0.0%)
    Commit to Disk:              2818 ms (  5.4%)
--------------------------------------------------
Total Link Time:                52669 ms (100.0%)

We can speed that up with a faster content hash (not SHA1).

Depends on D102885

[aganea]

We can speed that up with a faster content hash (not SHA1).

Definitely. xxHash in the LLVM tree gives quite good results, see https://reviews.llvm.org/D55585#1354878 Probably integrating the latest version would improve the figures (also it supports hardware vector instructions). https://github.com/Cyan4973/xxHash

lld/test/COFF/pdb-type-server-simple.test

23

I must confess I intuitively like better -debug:noghash because it's searchable & unique, and it's harder to spot the 'minus' in a large block of text. But there are maybe arguments both ways? :)

[rnk]

• use /debug:noghash

[rnk]

lld/test/COFF/pdb-type-server-simple.test

23

Yeah, I guess I agree.

[aganea]

LGTM. But perhaps @thakis and @mstorsjo might want to take a second look?

[mstorsjo]

In D102888#2772631, @aganea wrote:

LGTM. But perhaps @thakis and @mstorsjo might want to take a second look?

Sorry, I have little to no clue about PDB things, so I can't really give any meningful comment on this - but the patch overall looks reasonable.

[aganea]

In D102888#2773139, @mstorsjo wrote:

Sorry, I have little to no clue about PDB things, so I can't really give any meningful comment on this - but the patch overall looks reasonable.

Sorry I should have been more specific: I was wondering if you had an opinion for -debug:ghash- vs. -debug:noghash.

[aganea]

@rnk as a next step, you would probably want to re-do D43881 so that -gcodeview-ghash is enabled by default when building LLVM?

[rnk]

In D102888#2773545, @aganea wrote:

@rnk as a next step, you would probably want to re-do D43881 so that -gcodeview-ghash is enabled by default when building LLVM?

Oh right, I guess I switched to gn on Windows since then. You can see I did the equivalent here:

$ git grep -B1 gcodeview-ghash llvm
llvm/utils/gn/build/BUILD.gn-      if (use_lld && is_clang) {
llvm/utils/gn/build/BUILD.gn:        cflags += [ "-gcodeview-ghash" ]

I think at the time that I was using cmake, I just kept adding -gcodeview-ghash to CMAKE_CXX_FLAGS.

[rnk]

I suppose I should've updated D43881, but I made a new one at https://reviews.llvm.org/D103287. Oh well.

lld/COFF/Driver.cpp

static std::string createResponseFile(const opt::InputArgList &args,
   }
 
   for (StringRef path : filePaths)
     os << quote(relativeToRoot(path)) << "\n";
 
   return std::string(data.str());
 }
 
-enum class DebugKind { Unknown, None, Full, FastLink, GHash, Dwarf, Symtab };
+enum class DebugKind {
+  Unknown,
+  None,
+  Full,
+  FastLink,
+  GHash,
+  NoGHash,
+  Dwarf,
+  Symtab
+};
 
 static DebugKind parseDebugKind(const opt::InputArgList &args) {
   auto *a = args.getLastArg(OPT_debug, OPT_debug_opt);
   if (!a)
     return DebugKind::None;
   if (a->getNumValues() == 0)
     return DebugKind::Full;
 
   DebugKind debug = StringSwitch<DebugKind>(a->getValue())
                         .CaseLower("none", DebugKind::None)
                         .CaseLower("full", DebugKind::Full)
                         .CaseLower("fastlink", DebugKind::FastLink)
                         // LLD extensions
                         .CaseLower("ghash", DebugKind::GHash)
+                        .CaseLower("noghash", DebugKind::NoGHash)
                         .CaseLower("dwarf", DebugKind::Dwarf)
                         .CaseLower("symtab", DebugKind::Symtab)
                         .Default(DebugKind::Unknown);
 
   if (debug == DebugKind::FastLink) {
     warn("/debug:fastlink unsupported; using /debug:full");
     return DebugKind::Full;
   }
   if (debug == DebugKind::Unknown) {
     error("/debug: unknown option: " + Twine(a->getValue()));
     return DebugKind::None;
void LinkerDriver::linkerMain(ArrayRef<const char *> argsArr) {
 
   // Handle /force or /force:multipleres
   if (args.hasArg(OPT_force, OPT_force_multipleres))
     config->forceMultipleRes = true;
 
   // Handle /debug
   DebugKind debug = parseDebugKind(args);
   if (debug == DebugKind::Full || debug == DebugKind::Dwarf ||
-      debug == DebugKind::GHash) {
+      debug == DebugKind::GHash || debug == DebugKind::NoGHash) {
     config->debug = true;
     config->incremental = true;
   }
 
   // Handle /demangle
   config->demangle = args.hasFlag(OPT_demangle, OPT_demangle_no);
 
   // Handle /debugtype
   config->debugTypes = parseDebugTypes(args);
 
   // Handle /driver[:uponly|:wdm].
   config->driverUponly = args.hasArg(OPT_driver_uponly) ||
                          args.hasArg(OPT_driver_uponly_wdm) ||
                          args.hasArg(OPT_driver_wdm_uponly);
   config->driverWdm = args.hasArg(OPT_driver_wdm) ||
                       args.hasArg(OPT_driver_uponly_wdm) ||
                       args.hasArg(OPT_driver_wdm_uponly);
   config->driver =
       config->driverUponly || config->driverWdm || args.hasArg(OPT_driver);
 
   // Handle /pdb
   bool shouldCreatePDB =
-      (debug == DebugKind::Full || debug == DebugKind::GHash);
+      (debug == DebugKind::Full || debug == DebugKind::GHash ||
+       debug == DebugKind::NoGHash);
   if (shouldCreatePDB) {
     if (auto *arg = args.getLastArg(OPT_pdb))
       config->pdbPath = arg->getValue();
     if (auto *arg = args.getLastArg(OPT_pdbaltpath))
       config->pdbAltPath = arg->getValue();
     if (args.hasArg(OPT_natvis))
       config->natvisFiles = args.getAllArgValues(OPT_natvis);
     if (args.hasArg(OPT_pdbstream)) {
config->integrityCheck =
       args.hasFlag(OPT_integritycheck, OPT_integritycheck_no, false);
   config->cetCompat = args.hasFlag(OPT_cetcompat, OPT_cetcompat_no, false);
   config->nxCompat = args.hasFlag(OPT_nxcompat, OPT_nxcompat_no, true);
   for (auto *arg : args.filtered(OPT_swaprun))
     parseSwaprun(arg->getValue());
   config->terminalServerAware =
       !config->dll && args.hasFlag(OPT_tsaware, OPT_tsaware_no, true);
   config->debugDwarf = debug == DebugKind::Dwarf;
-  config->debugGHashes = debug == DebugKind::GHash;
+  config->debugGHashes = debug == DebugKind::GHash || debug == DebugKind::Full;
   config->debugSymtab = debug == DebugKind::Symtab;
   config->autoImport =
       args.hasFlag(OPT_auto_import, OPT_auto_import_no, config->mingw);
   config->pseudoRelocs = args.hasFlag(
       OPT_runtime_pseudo_reloc, OPT_runtime_pseudo_reloc_no, config->mingw);
   config->callGraphProfileSort = args.hasFlag(
       OPT_call_graph_profile_sort, OPT_call_graph_profile_sort_no, true);
 

lld/test/COFF/pdb-type-server-simple.test

 $ cl -c a.c b.c -Zi -Fdts.pdb
 
 $ lld-link a.obj b.obj -debug -entry:main -nodefaultlib -out:t.exe
 
 RUN: rm -rf %t && mkdir -p %t && cd %t
 RUN: yaml2obj %S/Inputs/pdb-type-server-simple-a.yaml -o a.obj
 RUN: yaml2obj %S/Inputs/pdb-type-server-simple-b.yaml -o b.obj
 RUN: llvm-pdbutil yaml2pdb %S/Inputs/pdb-type-server-simple-ts.yaml -pdb ts.pdb
-RUN: lld-link a.obj b.obj -entry:main -debug -out:t.exe -pdb:t.pdb -nodefaultlib -summary | FileCheck %s -check-prefix SUMMARY
+RUN: lld-link a.obj b.obj -entry:main -debug:noghash -out:t.exe -pdb:t.pdb -nodefaultlib -summary | FileCheck %s -check-prefix SUMMARY
 RUN: llvm-pdbutil dump -symbols -types -ids -globals %t/t.pdb | FileCheck %s
 
 Re-run with /DEBUG:GHASH
 RUN: lld-link a.obj b.obj -entry:main -debug:ghash -out:t.exe -pdb:t.pdb -nodefaultlib -summary -verbose
 RUN: llvm-pdbutil dump -symbols -types -ids -globals %t/t.pdb | FileCheck %s
 
 
 CHECK-LABEL: Types (TPI Stream)