lld-link: Set PDB GUID to hash of PDB contents instead of to a random byte sequence.

[thakis]

Summary

Previously, lld-link would use a random byte sequence as the PDB GUID. Instead, use a hash of the PDB file contents.

Naively computing the hash after the PDB data has been generated is in practice as fast as other approaches I tried. I also tried online-computing the hash as parts of the PDB were written out (https://reviews.llvm.org/D51887; that's also where all the measuring data is) and computing the hash in parallel (https://reviews.llvm.org/D51957). This approach here is simplest, without being slower.

To not disturb llvm-pdbutil pdb2yaml, make the hash generation an opt-in feature on InfoStreamBuilder and let ldb/COFF/PDB.cpp always set it.

Since writing the PDB computes this ID which also goes in the exe, the PDB writing code now must be called before writeBuildId(). writeBuildId() for that reason is no longer included in the "Code Layout" timer.

Since the PDB GUID is now a function of the PDB contents, the PDB Age is always set to 1. There was a long comment above loadExistingBuildId (now gone) about how not changing the GUID and only incrementing the age was important, but according to the discussion in PR35914 that comment was incorrect.

[thakis]

This replaces https://reviews.llvm.org/D51957.

[thakis]

Actually, this replaces https://reviews.llvm.org/D51887

[thakis]

rebase, minor cleanups

Please take a look!

[ruiu]

lld/COFF/PDB.cpp

128–129

It's return type is void. I'd change the comment or change the return type.

llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp

28

As long as you are using a non-crypto hash function, there is a risk of generating the same build id, and the probability is not negligible if you have a lot of executables due to the birthday problem. Is this okay?

[thakis]

lld/COFF/PDB.cpp

128–129

Will do.

llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp

28

The 8 byte hash still gives decent hash collision resistance for up to 2**32 different pdb files, and since pdbs are keyed by executable name on the symbol server that's per binary. Projects tend to have far fewer revisions than 4 billion. Does that make sense?

[thakis]

improve comment

[ruiu]

llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp

28

Maybe it is safe. But what could happen if two executables have the same hash? Since xxhash is not cryptographically-safe, you could easily generate two executables having the same ID. Is there any security risks or something caused by that possibility? If the probability is small and the result of hash collision is not that bad, xxhash is probably okay.

[thakis]

llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp

28

The main use case for this guid is to an executable to its pdb file. The common workflow is that a build server builds an executable and its pdb, then uploads both to a symbol server (under the namespace of the exe, the exe in a subdir containing the exe's pe timestamp and size, and the pdb under the guid). If the executable crashes, it produces a minidump. From the minidump, crash infrastructure can obtain the full executable and the pdb.

Since nothing guarantees that the pdb guid is a hash of the pdb data, I can't think of anything where being able to produce a pdb with a given uuid that is an xxhash buys you anything: Since nothing forces the guid to be a hash, you can just produce a pdb and set its guid field to whatever you want anyways.

[llvm-commits]

The symptoms of a collision are just going to be you can’t debug the
program, so not very severe imo, especially since it would almost certainly
be resolved on the next incremental build

• msg-23510-146.txt162 BDownload

[thakis]

In D51956#1235313, @llvm-commits wrote:

The symptoms of a collision are just going to be you can’t debug the
program, so not very severe imo, especially since it would almost certainly
be resolved on the next incremental build

Can you explain how it would lead to you not being able to debug the program?

Do you mean for local builds? If so, if two back-to-back builds end up with the same pdb guid in the exe and pdb by chance even though the pdb changes, the debugger should still load the new pdb off disk fine (?)

Do you mean if a build server produces PDBs with the same guid for different builds? If so, that would probably produce an error during pdb upload and make the build fail, not debugging (?)

[zturner]

If you’re uploading to build server, i don’t think it would be an error, it
would either overwrite or not. If it does overwrite, debugging the exe
matching the pdb that was there before wouldn’t work, and if it did not
overwrite debugging the new exe would fail.

That said, my point was mainly that the probability of this being an issue
in practice is negligible

[ruiu]

LGTM

It sounds like I worried a bit too much about hash collisions.

[thakis]

https://reviews.llvm.org/rLLD342334 too. Thanks!

lld/COFF/PDB.h

 namespace coff {
 class OutputSection;
 class SectionChunk;
 class SymbolTable;
 
 void createPDB(SymbolTable *Symtab,
                llvm::ArrayRef<OutputSection *> OutputSections,
                llvm::ArrayRef<uint8_t> SectionTable,
-               const llvm::codeview::DebugInfo &BuildId);
+               llvm::codeview::DebugInfo *BuildId);
 
 std::pair<llvm::StringRef, uint32_t> getFileLine(const SectionChunk *C,
                                                  uint32_t Addr);
 }
 }
 
 #endif

lld/COFF/PDB.cpp

PDBLinker(SymbolTable *Symtab)
         IDTable(Alloc), GlobalTypeTable(Alloc), GlobalIDTable(Alloc) {
     // This isn't strictly necessary, but link.exe usually puts an empty string
     // as the first "valid" string in the string table, so we do the same in
     // order to maintain as much byte-for-byte compatibility as possible.
     PDBStrTab.insert("");
   }
 
   /// Emit the basic PDB structure: initial streams, headers, etc.
-  void initialize(const llvm::codeview::DebugInfo &BuildId);
+  void initialize(llvm::codeview::DebugInfo *BuildId);
 
   /// Add natvis files specified on the command line.
   void addNatvisFiles();
 
   /// Link CodeView from each object file in the symbol table into the PDB.
   void addObjectsToPDB();
 
   /// Link CodeView from a single object file into the PDB.
public:
 
   Expected<const CVIndexMap&> maybeMergeTypeServerPDB(ObjFile *File,
                                                       TypeServer2Record &TS);
 
   /// Add the section map and section contributions to the PDB.
   void addSections(ArrayRef<OutputSection *> OutputSections,
                    ArrayRef<uint8_t> SectionTable);
 
-  /// Write the PDB to disk.
-  void commit();
+  /// Write the PDB to disk and return the Guid generated for it.
+  void commit(codeview::GUID *Guid);
 
 private:
   BumpPtrAllocator Alloc;
 
   SymbolTable *Symtab;
 
   pdb::PDBFileBuilder Builder;
 
tryToLoadPDB(const GUID &GuidFromObj, StringRef TSPath) {
   // PDB file doesn't mean it matches.  For it to match the InfoStream's GUID
   // must match the GUID specified in the TypeServer2 record.
   if (ExpectedInfo->getGuid() != GuidFromObj)
     return make_error<pdb::PDBError>(pdb::pdb_error_code::signature_out_of_date);
 
   return std::move(NS);
 }
 
-Expected<const CVIndexMap&> PDBLinker::maybeMergeTypeServerPDB(ObjFile *File,
-                                                               TypeServer2Record &TS) {
+Expected<const CVIndexMap &>
+PDBLinker::maybeMergeTypeServerPDB(ObjFile *File, TypeServer2Record &TS) {
   const GUID &TSId = TS.getGuid();
   StringRef TSPath = TS.getName();
 
   // First, check if the PDB has previously failed to load.
   auto PrevErr = MissingTypeServerPDBs.find(TSId);
   if (PrevErr != MissingTypeServerPDBs.end())
     return createFileError(
         TSPath,
void PDBLinker::addObjFile(ObjFile *File) {
   }
 
   // Make a new file checksum table that refers to offsets in the PDB-wide
   // string table. Generally the string table subsection appears after the
   // checksum table, so we have to do this after looping over all the
   // subsections.
   auto NewChecksums = make_unique<DebugChecksumsSubsection>(PDBStrTab);
   for (FileChecksumEntry &FC : Checksums) {
-    SmallString<128> FileName = ExitOnErr(CVStrTab.getString(FC.FileNameOffset));
+    SmallString<128> FileName =
+        ExitOnErr(CVStrTab.getString(FC.FileNameOffset));
     if (!sys::path::is_absolute(FileName) &&
         !Config->PDBSourcePath.empty()) {
       SmallString<128> AbsoluteFileName = Config->PDBSourcePath;
       sys::path::append(AbsoluteFileName, FileName);
       sys::path::native(AbsoluteFileName);
       sys::path::remove_dots(AbsoluteFileName, /*remove_dot_dots=*/true);
       FileName = std::move(AbsoluteFileName);
     }
static void addLinkerModuleSectionSymbol(pdb::DbiModuleDescriptorBuilder &Mod,
   Mod.addSymbol(codeview::SymbolSerializer::writeOneSymbol(
       Sym, Allocator, CodeViewContainer::Pdb));
 }
 
 // Creates a PDB file.
 void coff::createPDB(SymbolTable *Symtab,
                      ArrayRef<OutputSection *> OutputSections,
                      ArrayRef<uint8_t> SectionTable,
-                     const llvm::codeview::DebugInfo &BuildId) {
+                     llvm::codeview::DebugInfo *BuildId) {
   ScopedTimer T1(TotalPdbLinkTimer);
   PDBLinker PDB(Symtab);
 
   PDB.initialize(BuildId);
   PDB.addObjectsToPDB();
   PDB.addSections(OutputSections, SectionTable);
   PDB.addNatvisFiles();
 
   ScopedTimer T2(DiskCommitTimer);
-  PDB.commit();
+  codeview::GUID Guid;
+  PDB.commit(&Guid);
+  memcpy(&BuildId->PDB70.Signature, &Guid, 16);
 }
 
-void PDBLinker::initialize(const llvm::codeview::DebugInfo &BuildId) {
+void PDBLinker::initialize(llvm::codeview::DebugInfo *BuildId) {
   ExitOnErr(Builder.initialize(4096)); // 4096 is blocksize
 
+  BuildId->Signature.CVSignature = OMF::Signature::PDB70;
+  // Signature is set to a hash of the PDB contents when the PDB is done.
+  memset(BuildId->PDB70.Signature, 0, 16);
+  BuildId->PDB70.Age = 1;
+
   // Create streams in MSF for predefined streams, namely
   // PDB, TPI, DBI and IPI.
   for (int I = 0; I < (int)pdb::kSpecialStreamCount; ++I)
     ExitOnErr(Builder.getMsfBuilder().addStream(0));
 
   // Add an Info stream.
   auto &InfoBuilder = Builder.getInfoBuilder();
-  GUID uuid;
-  memcpy(&uuid, &BuildId.PDB70.Signature, sizeof(uuid));
-  InfoBuilder.setAge(BuildId.PDB70.Age);
-  InfoBuilder.setGuid(uuid);
   InfoBuilder.setVersion(pdb::PdbRaw_ImplVer::PdbImplVC70);
+  InfoBuilder.setHashPDBContentsToGUID(true);
 
   // Add an empty DBI stream.
   pdb::DbiStreamBuilder &DbiBuilder = Builder.getDbiBuilder();
-  DbiBuilder.setAge(BuildId.PDB70.Age);
+  DbiBuilder.setAge(BuildId->PDB70.Age);
   DbiBuilder.setVersionHeader(pdb::PdbDbiV70);
   DbiBuilder.setMachineType(Config->Machine);
   // Technically we are not link.exe 14.11, but there are known cases where
   // debugging tools on Windows expect Microsoft-specific version numbers or
   // they fail to work at all.  Since we know we produce PDBs that are
   // compatible with LINK 14.11, we set that version number here.
   DbiBuilder.setBuildNumber(14, 11);
 }
void PDBLinker::addSections(ArrayRef<OutputSection *> OutputSections,
   SectionMap = pdb::DbiStreamBuilder::createSectionMap(Sections);
   DbiBuilder.setSectionMap(SectionMap);
 
   // Add COFF section header stream.
   ExitOnErr(
       DbiBuilder.addDbgStream(pdb::DbgHeaderType::SectionHdr, SectionTable));
 }
 
-void PDBLinker::commit() {
+void PDBLinker::commit(codeview::GUID *Guid) {
   // Write to a file.
-  ExitOnErr(Builder.commit(Config->PDBPath));
+  ExitOnErr(Builder.commit(Config->PDBPath, Guid));
 }
 
 static Expected<StringRef>
 getFileName(const DebugStringTableSubsectionRef &Strings,
             const DebugChecksumsSubsectionRef &Checksums, uint32_t FileID) {
   auto Iter = Checksums.getArray().at(FileID);
   if (Iter == Checksums.getArray().end())
     return make_error<CodeViewError>(cv_error_code::no_records);

lld/COFF/Writer.cpp

private:
   IdataContents Idata;
   DelayLoadContents DelayIdata;
   EdataContents Edata;
   bool SetNoSEHCharacteristic = false;
 
   DebugDirectoryChunk *DebugDirectory = nullptr;
   std::vector<Chunk *> DebugRecords;
   CVDebugRecordChunk *BuildId = nullptr;
-  Optional<codeview::DebugInfo> PreviousBuildId;
   ArrayRef<uint8_t> SectionTable;
 
   uint64_t FileSize;
   uint32_t PointerToSymbolTable = 0;
   uint64_t SizeOfImage;
   uint64_t SizeOfHeaders;
 
   OutputSection *TextSec;
if (StringTableOff) {
     strncpy(Hdr->Name, Name.data(),
             std::min(Name.size(), (size_t)COFF::NameSize));
   }
 }
 
 } // namespace coff
 } // namespace lld
 
-// PDBs are matched against executables using a build id which consists of three
-// components:
-//   1. A 16-bit GUID
-//   2. An age
-//   3. A time stamp.
-//
-// Debuggers and symbol servers match executables against debug info by checking
-// each of these components of the EXE/DLL against the corresponding value in
-// the PDB and failing a match if any of the components differ.  In the case of
-// symbol servers, symbols are cached in a folder that is a function of the
-// GUID.  As a result, in order to avoid symbol cache pollution where every
-// incremental build copies a new PDB to the symbol cache, we must try to re-use
-// the existing GUID if one exists, but bump the age.  This way the match will
-// fail, so the symbol cache knows to use the new PDB, but the GUID matches, so
-// it overwrites the existing item in the symbol cache rather than making a new
-// one.
-static Optional<codeview::DebugInfo> loadExistingBuildId(StringRef Path) {
-  // We don't need to incrementally update a previous build id if we're not
-  // writing codeview debug info.
-  if (!Config->Debug)
-    return None;
-
-  auto ExpectedBinary = llvm::object::createBinary(Path);
-  if (!ExpectedBinary) {
-    consumeError(ExpectedBinary.takeError());
-    return None;
-  }
-
-  auto Binary = std::move(*ExpectedBinary);
-  if (!Binary.getBinary()->isCOFF())
-    return None;
-
-  std::error_code EC;
-  COFFObjectFile File(Binary.getBinary()->getMemoryBufferRef(), EC);
-  if (EC)
-    return None;
-
-  // If the machine of the binary we're outputting doesn't match the machine
-  // of the existing binary, don't try to re-use the build id.
-  if (File.is64() != Config->is64() || File.getMachine() != Config->Machine)
-    return None;
-
-  for (const auto &DebugDir : File.debug_directories()) {
-    if (DebugDir.Type != IMAGE_DEBUG_TYPE_CODEVIEW)
-      continue;
-
-    const codeview::DebugInfo *ExistingDI = nullptr;
-    StringRef PDBFileName;
-    if (auto EC = File.getDebugPDBInfo(ExistingDI, PDBFileName)) {
-      (void)EC;
-      return None;
-    }
-    // We only support writing PDBs in v70 format.  So if this is not a build
-    // id that we recognize / support, ignore it.
-    if (ExistingDI->Signature.CVSignature != OMF::Signature::PDB70)
-      return None;
-    return *ExistingDI;
-  }
-  return None;
-}
-
 // The main function of the writer.
 void Writer::run() {
   ScopedTimer T1(CodeLayoutTimer);
 
   createSections();
   createMiscChunks();
   createImportTables();
   createExportTable();
   mergeSections();
   assignAddresses();
   removeEmptySections();
   setSectionPermissions();
   createSymbolAndStringTable();
 
   if (FileSize > UINT32_MAX)
     fatal("image size (" + Twine(FileSize) + ") " +
         "exceeds maximum allowable size (" + Twine(UINT32_MAX) + ")");
 
-  // We must do this before opening the output file, as it depends on being able
-  // to read the contents of the existing output file.
-  PreviousBuildId = loadExistingBuildId(Config->OutputFile);
   openFile(Config->OutputFile);
   if (Config->is64()) {
     writeHeader<pe32plus_header>();
   } else {
     writeHeader<pe32_header>();
   }
   writeSections();
   sortExceptionTable();
-  writeBuildId();
 
   T1.stop();
 
   if (!Config->PDBPath.empty() && Config->Debug) {
     assert(BuildId);
-    createPDB(Symtab, OutputSections, SectionTable, *BuildId->BuildId);
+    createPDB(Symtab, OutputSections, SectionTable, BuildId->BuildId);
   }
+  writeBuildId();
 
   writeMapFile(OutputSections);
 
   ScopedTimer T2(DiskCommitTimer);
   if (auto E = Buffer->commit())
     fatal("failed to write the output file: " + toString(std::move(E)));
 }
 

 }
 
 void Writer::writeBuildId() {
   // There are two important parts to the build ID.
   // 1) If building with debug info, the COFF debug directory contains a
   //    timestamp as well as a Guid and Age of the PDB.
   // 2) In all cases, the PE COFF file header also contains a timestamp.
   // For reproducibility, instead of a timestamp we want to use a hash of the
-  // binary, however when building with debug info the hash needs to take into
-  // account the debug info, since it's possible to add blank lines to a file
-  // which causes the debug info to change but not the generated code.
-  //
-  // To handle this, we first set the Guid and Age in the debug directory (but
-  // only if we're doing a debug build).  Then, we hash the binary (thus causing
-  // the hash to change if only the debug info changes, since the Age will be
-  // different).  Finally, we write that hash into the debug directory (if
-  // present) as well as the COFF file header (always).
+  // PE contents.
   if (Config->Debug) {
     assert(BuildId && "BuildId is not set!");
-    if (PreviousBuildId.hasValue()) {
-      *BuildId->BuildId = *PreviousBuildId;
-      BuildId->BuildId->PDB70.Age = BuildId->BuildId->PDB70.Age + 1;
-    } else {
-      BuildId->BuildId->Signature.CVSignature = OMF::Signature::PDB70;
-      BuildId->BuildId->PDB70.Age = 1;
-      llvm::getRandomBytes(BuildId->BuildId->PDB70.Signature, 16);
-    }
+    // BuildId->BuildId was filled in when the PDB was written.
   }
 
   // At this point the only fields in the COFF file which remain unset are the
   // "timestamp" in the COFF file header, and the ones in the coff debug
   // directory.  Now we can hash the file and write that hash to the various
   // timestamp fields in the file.
   StringRef OutputFileData(
       reinterpret_cast<const char *>(Buffer->getBufferStart()),

lld/test/COFF/rsds.test

 # RUN: yaml2obj %s > %t.obj
 
 # RUN: rm -f %t.dll %t.pdb
-# RUN: lld-link /debug /pdbaltpath:test1.pdb /dll /out:%t.dll /entry:DllMain %t.obj
+# RUN: lld-link /debug /pdbaltpath:test.pdb /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll > %t.1.txt
-# RUN: lld-link /debug /pdbaltpath:test2.pdb /dll /out:%t.dll /entry:DllMain %t.obj
+# RUN: lld-link /debug /pdbaltpath:test.pdb /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll > %t.2.txt
 # RUN: cat %t.1.txt %t.2.txt | FileCheck %s
 
 # RUN: rm -f %t.dll %t.pdb
 # RUN: lld-link /debug /pdb:%t1.pdb /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll > %t.3.txt
 # RUN: lld-link /debug /pdb:%t2.pdb /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll > %t.4.txt
-# RUN: cat %t.3.txt %t.4.txt | FileCheck %s
+# RUN: cat %t.3.txt %t.4.txt | FileCheck --check-prefix TWOPDBS %s
 
 # RUN: rm -f %t.dll %t.pdb
 # RUN: lld-link /Brepro /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll | FileCheck --check-prefix REPRO %s
 
 # RUN: rm -f %t.dll %t.pdb
 # RUN: lld-link /Brepro /debug /dll /out:%t.dll /entry:DllMain %t.obj
 # RUN: llvm-readobj -coff-debug-directory %t.dll | FileCheck --check-prefix REPRODEBUG %s
 
 # CHECK: File: [[FILE:.*]].dll
 # CHECK: DebugDirectory [
 # CHECK:   DebugEntry {
 # CHECK:     Characteristics: 0x0
 # CHECK:     TimeDateStamp: 
 # CHECK:     MajorVersion: 0x0
 # CHECK:     MinorVersion: 0x0
 # CHECK:     Type: CodeView (0x2)
 # CHECK:     SizeOfData: 0x{{[^0]}}
 # CHECK:     AddressOfRawData: 0x{{[^0]}}
 # CHECK:     PointerToRawData: 0x{{[^0]}}
 # CHECK:     PDBInfo {
 # CHECK:       PDBSignature: 0x53445352
 # CHECK:       PDBGUID: [[GUID:\(([A-Za-z0-9]{2} ?){16}\)]]
 # CHECK:       PDBAge: 1
-# CHECK:       PDBFileName: {{.*}}1.pdb
+# CHECK:       PDBFileName: {{.*}}.pdb
 # CHECK:     }
 # CHECK:   }
 # CHECK: ]
 # CHECK: File: [[FILE]].dll
 # CHECK: DebugDirectory [
 # CHECK:   DebugEntry {
 # CHECK:     Characteristics: 0x0
 # CHECK:     TimeDateStamp: 
 # CHECK:     MajorVersion: 0x0
 # CHECK:     MinorVersion: 0x0
 # CHECK:     Type: CodeView (0x2)
 # CHECK:     SizeOfData: 0x{{[^0]}}
 # CHECK:     AddressOfRawData: 0x{{[^0]}}
 # CHECK:     PointerToRawData: 0x{{[^0]}}
 # CHECK:     PDBInfo {
 # CHECK:       PDBSignature: 0x53445352
 # CHECK:       PDBGUID: [[GUID]]
-# CHECK:       PDBAge: 2
-# CHECK:       PDBFileName: {{.*}}2.pdb
+# CHECK:       PDBAge: 1
+# CHECK:       PDBFileName: {{.*}}.pdb
 # CHECK:     }
 # CHECK:   }
 # CHECK: ]
 
+# TWOPDBS: File: [[FILE:.*]].dll
+# TWOPDBS: DebugDirectory [
+# TWOPDBS:   DebugEntry {
+# TWOPDBS:     Characteristics: 0x0
+# TWOPDBS:     TimeDateStamp: 
+# TWOPDBS:     MajorVersion: 0x0
+# TWOPDBS:     MinorVersion: 0x0
+# TWOPDBS:     Type: CodeView (0x2)
+# TWOPDBS:     SizeOfData: 0x{{[^0]}}
+# TWOPDBS:     AddressOfRawData: 0x{{[^0]}}
+# TWOPDBS:     PointerToRawData: 0x{{[^0]}}
+# TWOPDBS:     PDBInfo {
+# TWOPDBS:       PDBSignature: 0x53445352
+# TWOPDBS:       PDBGUID: [[GUID:\(([A-Za-z0-9]{2} ?){16}\)]]
+# TWOPDBS:       PDBAge: 1
+# TWOPDBS:       PDBFileName: {{.*}}.pdb
+# TWOPDBS:     }
+# TWOPDBS:   }
+# TWOPDBS: ]
+# TWOPDBS: File: [[FILE]].dll
+# TWOPDBS: DebugDirectory [
+# TWOPDBS:   DebugEntry {
+# TWOPDBS:     Characteristics: 0x0
+# TWOPDBS:     TimeDateStamp: 
+# TWOPDBS:     MajorVersion: 0x0
+# TWOPDBS:     MinorVersion: 0x0
+# TWOPDBS:     Type: CodeView (0x2)
+# TWOPDBS:     SizeOfData: 0x{{[^0]}}
+# TWOPDBS:     AddressOfRawData: 0x{{[^0]}}
+# TWOPDBS:     PointerToRawData: 0x{{[^0]}}
+# TWOPDBS:     PDBInfo {
+# TWOPDBS:       PDBSignature: 0x53445352
+# TWOPDBS-NOT:       PDBGUID: [[GUID]]
+# TWOPDBS:       PDBAge: 1
+# TWOPDBS:       PDBFileName: {{.*}}.pdb
+# TWOPDBS:     }
+# TWOPDBS:   }
+# TWOPDBS: ]
+
 # REPRO: File: {{.*}}.dll
 # REPRO: DebugDirectory [
 # REPRO:   DebugEntry {
 # REPRO:     Characteristics: 0x0
 # REPRO:     TimeDateStamp: 
 # REPRO:     MajorVersion: 0x0
 # REPRO:     MinorVersion: 0x0
 # REPRO:     Type: Repro (0x10)

llvm/include/llvm/DebugInfo/PDB/Native/InfoStreamBuilder.h

 
 class InfoStreamBuilder {
 public:
   InfoStreamBuilder(msf::MSFBuilder &Msf, NamedStreamMap &NamedStreams);
   InfoStreamBuilder(const InfoStreamBuilder &) = delete;
   InfoStreamBuilder &operator=(const InfoStreamBuilder &) = delete;
 
   void setVersion(PdbRaw_ImplVer V);
+  void addFeature(PdbRaw_FeatureSig Sig);
+
+  // If this is true, the PDB contents are hashed and this hash is used as
+  // PDB GUID and as Signature. The age is always 1.
+  void setHashPDBContentsToGUID(bool B);
+
+  // These only have an effect if hashPDBContentsToGUID() is false.
   void setSignature(uint32_t S);
   void setAge(uint32_t A);
   void setGuid(codeview::GUID G);
-  void addFeature(PdbRaw_FeatureSig Sig);
 
+  bool hashPDBContentsToGUID() const { return HashPDBContentsToGUID; }
   uint32_t getAge() const { return Age; }
   codeview::GUID getGuid() const { return Guid; }
   Optional<uint32_t> getSignature() const { return Signature; }
 
   uint32_t finalize();
 
   Error finalizeMsfLayout();
 
   Error commit(const msf::MSFLayout &Layout,
                WritableBinaryStreamRef Buffer) const;
 
 private:
   msf::MSFBuilder &Msf;
 
   std::vector<PdbRaw_FeatureSig> Features;
   PdbRaw_ImplVer Ver;
   uint32_t Age;
   Optional<uint32_t> Signature;
   codeview::GUID Guid;
 
+  bool HashPDBContentsToGUID = false;
+
   NamedStreamMap &NamedStreams;
 };
 }
 }
 
 #endif

llvm/include/llvm/DebugInfo/PDB/Native/PDBFileBuilder.h

public:
   msf::MSFBuilder &getMsfBuilder();
   InfoStreamBuilder &getInfoBuilder();
   DbiStreamBuilder &getDbiBuilder();
   TpiStreamBuilder &getTpiBuilder();
   TpiStreamBuilder &getIpiBuilder();
   PDBStringTableBuilder &getStringTableBuilder();
   GSIStreamBuilder &getGsiBuilder();
 
-  Error commit(StringRef Filename);
+  // If HashPDBContentsToGUID is true on the InfoStreamBuilder, Guid is filled
+  // with the computed PDB GUID on return.
+  Error commit(StringRef Filename, codeview::GUID *Guid);
 
   Expected<uint32_t> getNamedStreamIndex(StringRef Name) const;
   Error addNamedStream(StringRef Name, StringRef Data);
   void addInjectedSource(StringRef Name, std::unique_ptr<MemoryBuffer> Buffer);
 
 private:
   struct InjectedSourceDescriptor {
     // The full name of the stream that contains the contents of this injected

llvm/lib/DebugInfo/PDB/Native/GSIStreamBuilder.cpp

 }
 
 Error GSIStreamBuilder::commitPublicsHashStream(
     WritableBinaryStreamRef Stream) {
   BinaryStreamWriter Writer(Stream);
   PublicsStreamHeader Header;
 
   // FIXME: Fill these in. They are for incremental linking.
+  Header.SymHash = PSH->calculateSerializedLength();
+  Header.AddrMap = PSH->Records.size() * 4;
   Header.NumThunks = 0;
   Header.SizeOfThunk = 0;
   Header.ISectThunkTable = 0;
+  memset(Header.Padding, 0, sizeof(Header.Padding));
   Header.OffThunkTable = 0;
   Header.NumSections = 0;
-  Header.SymHash = PSH->calculateSerializedLength();
-  Header.AddrMap = PSH->Records.size() * 4;
   if (auto EC = Writer.writeObject(Header))
     return EC;
 
   if (auto EC = PSH->commit(Writer))
     return EC;
 
   std::vector<ulittle32_t> AddrMap = computeAddrMap(PSH->Records);
   if (auto EC = Writer.writeArray(makeArrayRef(AddrMap)))

llvm/lib/DebugInfo/PDB/Native/InfoStreamBuilder.cpp

InfoStreamBuilder::InfoStreamBuilder(msf::MSFBuilder &Msf,
                                      NamedStreamMap &NamedStreams)
     : Msf(Msf), Ver(PdbRaw_ImplVer::PdbImplVC70), Age(0),
       NamedStreams(NamedStreams) {
   ::memset(&Guid, 0, sizeof(Guid));
 }
 
 void InfoStreamBuilder::setVersion(PdbRaw_ImplVer V) { Ver = V; }
 
+void InfoStreamBuilder::addFeature(PdbRaw_FeatureSig Sig) {
+  Features.push_back(Sig);
+}
+
+void InfoStreamBuilder::setHashPDBContentsToGUID(bool B) {
+  HashPDBContentsToGUID = B;
+}
+
 void InfoStreamBuilder::setAge(uint32_t A) { Age = A; }
 
 void InfoStreamBuilder::setSignature(uint32_t S) { Signature = S; }
 
 void InfoStreamBuilder::setGuid(GUID G) { Guid = G; }
 
-void InfoStreamBuilder::addFeature(PdbRaw_FeatureSig Sig) {
-  Features.push_back(Sig);
-}
 
 Error InfoStreamBuilder::finalizeMsfLayout() {
   uint32_t Length = sizeof(InfoStreamHeader) +
                     NamedStreams.calculateSerializedLength() +
                     (Features.size() + 1) * sizeof(uint32_t);
   if (auto EC = Msf.setStreamSize(StreamPDB, Length))
     return EC;
   return Error::success();

llvm/lib/DebugInfo/PDB/Native/PDBFileBuilder.cpp

 #include "llvm/DebugInfo/PDB/Native/PDBStringTableBuilder.h"
 #include "llvm/DebugInfo/PDB/Native/RawError.h"
 #include "llvm/DebugInfo/PDB/Native/TpiStream.h"
 #include "llvm/DebugInfo/PDB/Native/TpiStreamBuilder.h"
 #include "llvm/Support/BinaryStream.h"
 #include "llvm/Support/BinaryStreamWriter.h"
 #include "llvm/Support/JamCRC.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/xxhash.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 using namespace llvm::msf;
 using namespace llvm::pdb;
 using namespace llvm::support;
 
 PDBFileBuilder::PDBFileBuilder(BumpPtrAllocator &Allocator)
auto SourceStream = WritableMappedBlockStream::createIndexedStream(
         Layout, MsfBuffer, SN, Allocator);
     BinaryStreamWriter SourceWriter(*SourceStream);
     assert(SourceWriter.bytesRemaining() == IS.Content->getBufferSize());
     cantFail(SourceWriter.writeBytes(
         arrayRefFromStringRef(IS.Content->getBuffer())));
   }
 }
 
-Error PDBFileBuilder::commit(StringRef Filename) {
+Error PDBFileBuilder::commit(StringRef Filename, codeview::GUID *Guid) {
   assert(!Filename.empty());
   if (auto EC = finalizeMsfLayout())
     return EC;
 
   MSFLayout Layout;
-  auto ExpectedMsfBuffer = Msf->commit(Filename, Layout);
+  Expected<FileBufferByteStream> ExpectedMsfBuffer =
+      Msf->commit(Filename, Layout);
   if (!ExpectedMsfBuffer)
     return ExpectedMsfBuffer.takeError();
   FileBufferByteStream Buffer = std::move(*ExpectedMsfBuffer);
 
   auto ExpectedSN = getNamedStreamIndex("/names");
   if (!ExpectedSN)
     return ExpectedSN.takeError();
 
uint64_t InfoStreamFileOffset =
       blockToOffset(InfoStreamBlocks.front(), Layout.SB->BlockSize);
   InfoStreamHeader *H = reinterpret_cast<InfoStreamHeader *>(
       Buffer.getBufferStart() + InfoStreamFileOffset);
 
   commitInjectedSources(Buffer, Layout);
 
   // Set the build id at the very end, after every other byte of the PDB
   // has been written.
-  // FIXME: Use a hash of the PDB rather than time(nullptr) for the signature.
+  if (Info->hashPDBContentsToGUID()) {
+    // Compute a hash of all sections of the output file.
+    uint8_t *Start = Buffer.getBufferStart();
+    uint8_t *End = Buffer.getBufferEnd();
+    ArrayRef<uint8_t> D{Start, End};
+    uint64_t Digest = xxHash64(D);
+
+    H->Age = 1;
+
+    memcpy(H->Guid.Guid, &Digest, 8);
+    // xxhash only gives us 8 bytes, so put some fixed data in the other half.
+    memcpy(H->Guid.Guid + 8, "LLD PDB.", 8);
+
+    // Return GUID to caller.
+    memcpy(Guid, H->Guid.Guid, 16);
+  } else {
     H->Age = Info->getAge();
     H->Guid = Info->getGuid();
+  }
+
+  // FIXME: Use a hash of the PDB rather than time(nullptr) for the signature.
+  // XXX: change this too
   Optional<uint32_t> Sig = Info->getSignature();
   H->Signature = Sig.hasValue() ? *Sig : time(nullptr);
 
   return Buffer.commit();
 }

llvm/tools/llvm-pdbutil/llvm-pdbutil.cpp

static void yamlToPdb(StringRef Path) {
   IpiBuilder.setVersionHeader(Ipi.Version);
   for (const auto &R : Ipi.Records) {
     CVType Type = R.toCodeViewRecord(TS);
     IpiBuilder.addTypeRecord(Type.RecordData, None);
   }
 
   Builder.getStringTableBuilder().setStrings(*Strings.strings());
 
-  ExitOnErr(Builder.commit(opts::yaml2pdb::YamlPdbOutputFile));
+  codeview::GUID IgnoredOutGuid;
+  ExitOnErr(Builder.commit(opts::yaml2pdb::YamlPdbOutputFile, &IgnoredOutGuid));
 }
 
 static PDBFile &loadPDB(StringRef Path, std::unique_ptr<IPDBSession> &Session) {
   ExitOnErr(loadDataForPDB(PDB_ReaderType::Native, Path, Session));
 
   NativeSession *NS = static_cast<NativeSession *>(Session.get());
   return NS->getPDBFile();
 }
static void mergePdbs() {
   });
   Builder.getInfoBuilder().addFeature(PdbRaw_FeatureSig::VC140);
 
   SmallString<64> OutFile(opts::merge::PdbOutputFile);
   if (OutFile.empty()) {
     OutFile = opts::merge::InputFilenames[0];
     llvm::sys::path::replace_extension(OutFile, "merged.pdb");
   }
-  ExitOnErr(Builder.commit(OutFile));
+
+  codeview::GUID IgnoredOutGuid;
+  ExitOnErr(Builder.commit(OutFile, &IgnoredOutGuid));
 }
 
 static void explain() {
   std::unique_ptr<IPDBSession> Session;
   InputFile IF =
       ExitOnErr(InputFile::open(opts::explain::InputFilename.front(), true));
 
   for (uint64_t Off : opts::explain::Offsets) {