Add support for writing TPI hash values

[zturner]

Summary

This only adds support for writing hash values of the record types we currently understand. For all other record types, we write a hash value of 0. A quick look at a PDB generated by Microsoft confirms that they have actual values for every single recofd type, so we will likely need to do some more work to figure out how to hash these other record types. It's probably as trivial as just applying one of these hashing functions to the entire record contents, mostly just need to try it and see.

[majnemer]

include/llvm/DebugInfo/PDB/Raw/TpiRecordHashVisitor.h

26 ↗

(On Diff #70774)

TpiRecordHashUpdateVisitor() = default; ?

56–59 ↗

(On Diff #70774)

Is this clang-formatted?

lib/DebugInfo/PDB/Raw/TpiStream.cpp

105–107 ↗

(On Diff #70774)

Couldn't you just do std::vector<ulittle32_t> HashValueList(HashValueList.begin(), HashValueList.end());

test/DebugInfo/PDB/pdbdump-readwrite.test

13–18 ↗

(On Diff #70774)

Is this diff correct?

[zturner]

include/llvm/DebugInfo/PDB/Raw/TpiRecordHashVisitor.h

56–59 ↗

(On Diff #70774)

It's clang-formatted. I agree this looks odd, I don't know why it did it formatted it that way. I'll run it through clang-format again just to make sure.

lib/DebugInfo/PDB/Raw/TpiStream.cpp

105–107 ↗

(On Diff #70774)

I tried but it didn't work because HashValueList.begin() isn't a normal iterator, it's a FixedStreamArrayIterator<>. There might be some magic to get it to work, but I don't know what the specific requirements are for an iterator to be usable in that context.

test/DebugInfo/PDB/pdbdump-readwrite.test

13–18 ↗

(On Diff #70774)

Yes, I removed these intentionally. Previously we were re-creating the stream directory and stream size list exactly, even though we weren't filling all the streams with any data. I removed the code to do that because it makes it difficult / impossible when we need to allocate a non-special stream. Specifically, in this patch we go and allocate space for the hash value stream. Which stream was that from the stream table? If we're trying to map the MSF layout exactly to the same stream indices etc, then we would need to look at the yaml, figure out what stream was the hash value stream, and assign the same index to it. The logic starts getting complicated for no tangible benefit.

So instead, I decided that I would only reserve special streams, and reserve other dynamic streams as needed. See all the code in llvm-pdbdump.cpp that was deleted. Because of all this, the directory, number of streams, etc is no longer going to match up. Someday in the future when we can round-trip a PDB byte for byte, we can add these back.

[zturner]

ping

[rnk]

include/llvm/DebugInfo/PDB/Raw/TpiRecordHashVisitor.h

24 ↗

(On Diff #70774)

Five words in a class name feels like too much. At least, it's not consistent with the rest of LLVM. How do these names sound?
TpiHashing.h
TpiHashUpdater
TpiHashVerifier

The first class computes the hash, the second one verifies an existing hash.

I don't feel like Record or Visitor are helping readability that much.

lib/DebugInfo/PDB/Raw/TpiRecordHashVisitor.cpp

44 ↗

(On Diff #70774)

Typo on "Souce"

[zturner]

Updated with suggestions from rnk

[rnk]

lgtm thanks!

llvm/trunk/include/llvm/DebugInfo/CodeView/CVRecord.h

 //===- RecordIterator.h -----------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_CODEVIEW_RECORDITERATOR_H
 #define LLVM_DEBUGINFO_CODEVIEW_RECORDITERATOR_H
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/DebugInfo/CodeView/CodeViewError.h"
 #include "llvm/DebugInfo/CodeView/RecordSerialization.h"
 #include "llvm/DebugInfo/MSF/StreamReader.h"
 #include "llvm/DebugInfo/MSF/StreamRef.h"
 #include "llvm/Support/Endian.h"
 
 namespace llvm {
 namespace codeview {
 
 template <typename Kind> struct CVRecord {
   uint32_t Length;
   Kind Type;
   ArrayRef<uint8_t> Data;
   ArrayRef<uint8_t> RawData;
+  Optional<uint32_t> Hash;
 };
 }
 namespace msf {
 
 template <typename Kind>
 struct VarStreamArrayExtractor<codeview::CVRecord<Kind>> {
   Error operator()(ReadableStreamRef Stream, uint32_t &Len,
                    codeview::CVRecord<Kind> &Item) const {

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeSerializationVisitor.h

virtual Error visitTypeBegin(CVType &Record) override {
     if (Record.Type == TypeLeafKind::LF_FIELDLIST)
       FieldListBuilder.reset();
     return Error::success();
   }
 
   virtual Error visitTypeEnd(CVType &Record) override {
     // Since this visitor's purpose is to serialize the record, fill out the
     // fields of `Record` with the bytes of the record.
-    if (Record.Type == TypeLeafKind::LF_FIELDLIST)
+    if (Record.Type == TypeLeafKind::LF_FIELDLIST) {
       TypeTableBuilder.writeFieldList(FieldListBuilder);
+      updateCVRecord(Record);
+    }
 
-    StringRef S = TypeTableBuilder.getRecords().back();
-    ArrayRef<uint8_t> Data(S.bytes_begin(), S.bytes_end());
-    Record.RawData = Data;
-    Record.Data = Record.RawData.drop_front(sizeof(RecordPrefix));
-    Record.Length = Data.size() - sizeof(ulittle16_t);
     return Error::success();
   }
 
 #define TYPE_RECORD(EnumName, EnumVal, Name)                                   \
-  virtual Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,                  \
-                                 Name##Record &Record) override {              \
-    visitKnownRecordImpl(Record);                                              \
+  virtual Error visitKnownRecord(CVType &CVR, Name##Record &Record) override { \
+    visitKnownRecordImpl(CVR, Record);                                         \
     return Error::success();                                                   \
   }
 #define TYPE_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
 #define MEMBER_RECORD(EnumName, EnumVal, Name)                                 \
-  virtual Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,                  \
-                                 Name##Record &Record) override {              \
+  virtual Error visitKnownRecord(CVType &CVR, Name##Record &Record) override { \
     visitMemberRecordImpl(Record);                                             \
     return Error::success();                                                   \
   }
 #define MEMBER_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
 #include "llvm/DebugInfo/CodeView/TypeRecords.def"
 
 private:
-  template <typename RecordKind> void visitKnownRecordImpl(RecordKind &Record) {
+  void updateCVRecord(CVType &Record) {
+    StringRef S = TypeTableBuilder.getRecords().back();
+    ArrayRef<uint8_t> Data(S.bytes_begin(), S.bytes_end());
+    Record.RawData = Data;
+    Record.Data = Record.RawData.drop_front(sizeof(RecordPrefix));
+    Record.Length = Data.size() - sizeof(ulittle16_t);
+  }
+  template <typename RecordKind>
+  void visitKnownRecordImpl(CVType &CVR, RecordKind &Record) {
     TypeTableBuilder.writeKnownType(Record);
+    updateCVRecord(CVR);
   }
   template <typename RecordKind>
   void visitMemberRecordImpl(RecordKind &Record) {
     FieldListBuilder.writeMemberType(Record);
   }
 
-  void visitKnownRecordImpl(FieldListRecord &FieldList) {}
+  void visitKnownRecordImpl(CVType &CVR, FieldListRecord &FieldList) {}
 
   FieldListRecordBuilder &FieldListBuilder;
   MemoryTypeTableBuilder &TypeTableBuilder;
 };
 }
 }
 
 #endif

llvm/trunk/include/llvm/DebugInfo/MSF/MSFBuilder.h

public:
   void setFreePageMap(uint32_t Fpm);
   void setUnknown1(uint32_t Unk1);
 
   /// Add a stream to the MSF file with the given size, occupying the given
   /// list of blocks.  This is useful when reading a MSF file and you want a
   /// particular stream to occupy the original set of blocks.  If the given
   /// blocks are already allocated, or if the number of blocks specified is
   /// incorrect for the given stream size, this function will return an Error.
-  Error addStream(uint32_t Size, ArrayRef<uint32_t> Blocks);
+  Expected<uint32_t> addStream(uint32_t Size, ArrayRef<uint32_t> Blocks);
 
   /// Add a stream to the MSF file with the given size, occupying any available
   /// blocks that the builder decides to use.  This is useful when building a
   /// new PDB file from scratch and you don't care what blocks a stream occupies
   /// but you just want it to work.
-  Error addStream(uint32_t Size);
+  Expected<uint32_t> addStream(uint32_t Size);
 
   /// Update the size of an existing stream.  This will allocate or deallocate
   /// blocks as needed to match the requested size.  This can fail if `CanGrow`
   /// was set to false when initializing the `MSFBuilder`.
   Error setStreamSize(uint32_t Idx, uint32_t Size);
 
   /// Get the total number of streams in the MSF layout.  This should return 1
   /// for every call to `addStream`.
public:
 
   /// Check whether a particular block is allocated or free.
   bool isBlockFree(uint32_t Idx) const;
 
   /// Finalize the layout and build the headers and structures that describe the
   /// MSF layout and can be written directly to the MSF file.
   Expected<MSFLayout> build();
 
+  BumpPtrAllocator &getAllocator() { return Allocator; }
+
 private:
   MSFBuilder(uint32_t BlockSize, uint32_t MinBlockCount, bool CanGrow,
              BumpPtrAllocator &Allocator);
 
   Error allocateBlocks(uint32_t NumBlocks, MutableArrayRef<uint32_t> Blocks);
   uint32_t computeDirectoryByteSize() const;
 
   typedef std::vector<uint32_t> BlockList;

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/DbiStreamBuilder.h

 
 #include "llvm/DebugInfo/MSF/ByteStream.h"
 #include "llvm/DebugInfo/MSF/StreamReader.h"
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/RawConstants.h"
 
 namespace llvm {
+namespace msf {
+class MSFBuilder;
+}
 namespace pdb {
 class DbiStream;
 struct DbiStreamHeader;
 class PDBFile;
 
 class DbiStreamBuilder {
 public:
-  DbiStreamBuilder(BumpPtrAllocator &Allocator);
+  DbiStreamBuilder(msf::MSFBuilder &Msf);
 
   DbiStreamBuilder(const DbiStreamBuilder &) = delete;
   DbiStreamBuilder &operator=(const DbiStreamBuilder &) = delete;
 
   void setVersionHeader(PdbRaw_DbiVer V);
   void setAge(uint32_t A);
   void setBuildNumber(uint16_t B);
   void setPdbDllVersion(uint16_t V);
   void setPdbDllRbld(uint16_t R);
   void setFlags(uint16_t F);
   void setMachineType(PDB_Machine M);
 
   uint32_t calculateSerializedLength() const;
 
   Error addModuleInfo(StringRef ObjFile, StringRef Module);
   Error addModuleSourceFile(StringRef Module, StringRef File);
 
+  Error finalizeMsfLayout();
+
   Expected<std::unique_ptr<DbiStream>> build(PDBFile &File,
                                              const msf::WritableStream &Buffer);
   Error commit(const msf::MSFLayout &Layout,
                const msf::WritableStream &Buffer);
 
 private:
   Error finalize();
   uint32_t calculateModiSubstreamSize() const;
   uint32_t calculateFileInfoSubstreamSize() const;
   uint32_t calculateNamesBufferSize() const;
 
   Error generateModiSubstream();
   Error generateFileInfoSubstream();
 
   struct ModuleInfo {
     std::vector<StringRef> SourceFiles;
     StringRef Obj;
     StringRef Mod;
   };
 
+  msf::MSFBuilder &Msf;
   BumpPtrAllocator &Allocator;
 
   Optional<PdbRaw_DbiVer> VerHeader;
   uint32_t Age;
   uint16_t BuildNumber;
   uint16_t PdbDllVersion;
   uint16_t PdbDllRbld;
   uint16_t Flags;

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/InfoStreamBuilder.h

 
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
 #include "llvm/DebugInfo/PDB/Raw/NameMapBuilder.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/RawConstants.h"
 
 namespace llvm {
 namespace msf {
+class MSFBuilder;
 class StreamWriter;
 }
 namespace pdb {
 class PDBFile;
 
 class InfoStreamBuilder {
 public:
-  InfoStreamBuilder();
+  InfoStreamBuilder(msf::MSFBuilder &Msf);
   InfoStreamBuilder(const InfoStreamBuilder &) = delete;
   InfoStreamBuilder &operator=(const InfoStreamBuilder &) = delete;
 
   void setVersion(PdbRaw_ImplVer V);
   void setSignature(uint32_t S);
   void setAge(uint32_t A);
   void setGuid(PDB_UniqueId G);
 
   NameMapBuilder &getNamedStreamsBuilder();
 
   uint32_t calculateSerializedLength() const;
 
+  Error finalizeMsfLayout();
+
   Expected<std::unique_ptr<InfoStream>>
   build(PDBFile &File, const msf::WritableStream &Buffer);
 
   Error commit(const msf::MSFLayout &Layout,
                const msf::WritableStream &Buffer) const;
 
 private:
+  msf::MSFBuilder &Msf;
+
   PdbRaw_ImplVer Ver;
   uint32_t Sig;
   uint32_t Age;
   PDB_UniqueId Guid;
 
   NameMapBuilder NamedStreams;
 };
 }
 }
 
 #endif

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/RawConstants.h

enum SpecialStream : uint32_t {
   // We are not currently using it, but technically if we find the main
   // MSF is corrupted, we could fallback to it.
   OldMSFDirectory = 0,
 
   StreamPDB = 1,
   StreamTPI = 2,
   StreamDBI = 3,
   StreamIPI = 4,
+
+  kSpecialStreamCount
 };
 
 enum class DbgHeaderType : uint16_t {
   FPO,
   Exception,
   Fixup,
   OmapToSrc,
   OmapFromSrc,

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/TpiHashing.h

+//===- TpiHashing.h ---------------------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_PDB_TPIHASHING_H
+#define LLVM_DEBUGINFO_PDB_TPIHASHING_H
+
+#include "llvm/ADT/Optional.h"
+#include "llvm/DebugInfo/CodeView/TypeIndex.h"
+#include "llvm/DebugInfo/CodeView/TypeRecord.h"
+#include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
+#include "llvm/DebugInfo/MSF/StreamArray.h"
+#include "llvm/DebugInfo/PDB/Raw/RawError.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Error.h"
+
+namespace llvm {
+namespace pdb {
+class TpiHashUpdater : public codeview::TypeVisitorCallbacks {
+public:
+  TpiHashUpdater() {}
+
+#define TYPE_RECORD(EnumName, EnumVal, Name)                                   \
+  virtual Error visitKnownRecord(codeview::CVType &CVR,                        \
+                                 codeview::Name##Record &Record) override {    \
+    visitKnownRecordImpl(CVR, Record);                                         \
+    return Error::success();                                                   \
+  }
+#define TYPE_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#define MEMBER_RECORD(EnumName, EnumVal, Name)
+#define MEMBER_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#include "llvm/DebugInfo/CodeView/TypeRecords.def"
+
+private:
+  template <typename RecordKind>
+  void visitKnownRecordImpl(codeview::CVType &CVR, RecordKind &Record) {
+    CVR.Hash = 0;
+  }
+
+  void visitKnownRecordImpl(codeview::CVType &CVR,
+                            codeview::UdtSourceLineRecord &Rec);
+  void visitKnownRecordImpl(codeview::CVType &CVR,
+                            codeview::UdtModSourceLineRecord &Rec);
+  void visitKnownRecordImpl(codeview::CVType &CVR, codeview::ClassRecord &Rec);
+  void visitKnownRecordImpl(codeview::CVType &CVR, codeview::EnumRecord &Rec);
+  void visitKnownRecordImpl(codeview::CVType &CVR, codeview::UnionRecord &Rec);
+};
+
+class TpiHashVerifier : public codeview::TypeVisitorCallbacks {
+public:
+  TpiHashVerifier(msf::FixedStreamArray<support::ulittle32_t> &HashValues,
+                  uint32_t NumHashBuckets)
+      : HashValues(HashValues), NumHashBuckets(NumHashBuckets) {}
+
+  Error visitKnownRecord(codeview::CVType &CVR,
+                         codeview::UdtSourceLineRecord &Rec) override;
+  Error visitKnownRecord(codeview::CVType &CVR,
+                         codeview::UdtModSourceLineRecord &Rec) override;
+  Error visitKnownRecord(codeview::CVType &CVR,
+                         codeview::ClassRecord &Rec) override;
+  Error visitKnownRecord(codeview::CVType &CVR,
+                         codeview::EnumRecord &Rec) override;
+  Error visitKnownRecord(codeview::CVType &CVR,
+                         codeview::UnionRecord &Rec) override;
+  Error visitTypeBegin(codeview::CVType &CVR) override;
+
+private:
+  Error verifySourceLine(codeview::TypeIndex TI);
+
+  Error errorInvalidHash() {
+    return make_error<RawError>(
+        raw_error_code::invalid_tpi_hash,
+        "Type index is 0x" +
+            utohexstr(codeview::TypeIndex::FirstNonSimpleIndex + Index));
+  }
+
+  msf::FixedStreamArray<support::ulittle32_t> HashValues;
+  codeview::CVType RawRecord;
+  uint32_t NumHashBuckets;
+  uint32_t Index = -1;
+};
+}
+}
+
+#endif

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/TpiStream.h

 private:
   Error verifyHashValues();
 
   const PDBFile &Pdb;
   std::unique_ptr<msf::MappedBlockStream> Stream;
 
   codeview::CVTypeArray TypeRecords;
 
-  std::unique_ptr<msf::MappedBlockStream> HashStream;
+  std::unique_ptr<msf::ReadableStream> HashStream;
   msf::FixedStreamArray<support::ulittle32_t> HashValues;
   msf::FixedStreamArray<TypeIndexOffset> TypeIndexOffsets;
   msf::FixedStreamArray<TypeIndexOffset> HashAdjustments;
 
   const TpiStreamHeader *Header;
 };
 }
 }
 
 #endif

llvm/trunk/include/llvm/DebugInfo/PDB/Raw/TpiStreamBuilder.h

 
 #include <vector>
 
 namespace llvm {
 namespace codeview {
 class TypeRecord;
 }
 namespace msf {
+class ByteStream;
+class MSFBuilder;
 struct MSFLayout;
 class ReadableStreamRef;
 class WritableStream;
 
 template <> struct SequencedItemTraits<llvm::codeview::CVType> {
   static size_t length(const codeview::CVType &Item) {
     return Item.RawData.size();
   }
   static ArrayRef<uint8_t> bytes(const codeview::CVType &Item) {
     return Item.RawData;
   }
 };
 }
 namespace pdb {
 class PDBFile;
 class TpiStream;
 struct TpiStreamHeader;
 
 class TpiStreamBuilder {
 public:
-  explicit TpiStreamBuilder(BumpPtrAllocator &Allocator);
+  explicit TpiStreamBuilder(msf::MSFBuilder &Msf);
   ~TpiStreamBuilder();
 
   TpiStreamBuilder(const TpiStreamBuilder &) = delete;
   TpiStreamBuilder &operator=(const TpiStreamBuilder &) = delete;
 
   void setVersionHeader(PdbRaw_TpiVer Version);
   void addTypeRecord(const codeview::CVType &Record);
 
+  Error finalizeMsfLayout();
+
   Expected<std::unique_ptr<TpiStream>> build(PDBFile &File,
                                              const msf::WritableStream &Buffer);
 
   Error commit(const msf::MSFLayout &Layout, const msf::WritableStream &Buffer);
 
   uint32_t calculateSerializedLength() const;
 
 private:
+  uint32_t calculateHashBufferSize() const;
   Error finalize();
 
+  msf::MSFBuilder &Msf;
   BumpPtrAllocator &Allocator;
 
   Optional<PdbRaw_TpiVer> VerHeader;
   std::vector<codeview::CVType> TypeRecords;
   msf::SequencedItemStream<codeview::CVType> TypeRecordStream;
+  uint32_t HashStreamIndex = kInvalidStreamIndex;
+  std::unique_ptr<msf::ByteStream> HashValueStream;
 
   const TpiStreamHeader *Header;
 };
 }
 }
 
 #endif

llvm/trunk/lib/DebugInfo/MSF/MSFBuilder.cpp

 }
 
 uint32_t MSFBuilder::getNumFreeBlocks() const { return FreeBlocks.count(); }
 
 uint32_t MSFBuilder::getTotalBlockCount() const { return FreeBlocks.size(); }
 
 bool MSFBuilder::isBlockFree(uint32_t Idx) const { return FreeBlocks[Idx]; }
 
-Error MSFBuilder::addStream(uint32_t Size, ArrayRef<uint32_t> Blocks) {
+Expected<uint32_t> MSFBuilder::addStream(uint32_t Size,
+                                         ArrayRef<uint32_t> Blocks) {
   // Add a new stream mapped to the specified blocks.  Verify that the specified
   // blocks are both necessary and sufficient for holding the requested number
   // of bytes, and verify that all requested blocks are free.
   uint32_t ReqBlocks = bytesToBlocks(Size, BlockSize);
   if (ReqBlocks != Blocks.size())
     return make_error<MSFError>(
         msf_error_code::invalid_format,
         "Incorrect number of blocks for requested stream size");
   for (auto Block : Blocks) {
     if (Block >= FreeBlocks.size())
       FreeBlocks.resize(Block + 1, true);
 
     if (!FreeBlocks.test(Block))
       return make_error<MSFError>(
           msf_error_code::unspecified,
           "Attempt to re-use an already allocated block");
   }
   // Mark all the blocks occupied by the new stream as not free.
   for (auto Block : Blocks) {
     FreeBlocks.reset(Block);
   }
   StreamData.push_back(std::make_pair(Size, Blocks));
-  return Error::success();
+  return StreamData.size() - 1;
 }
 
-Error MSFBuilder::addStream(uint32_t Size) {
+Expected<uint32_t> MSFBuilder::addStream(uint32_t Size) {
   uint32_t ReqBlocks = bytesToBlocks(Size, BlockSize);
   std::vector<uint32_t> NewBlocks;
   NewBlocks.resize(ReqBlocks);
   if (auto EC = allocateBlocks(ReqBlocks, NewBlocks))
-    return EC;
+    return std::move(EC);
   StreamData.push_back(std::make_pair(Size, NewBlocks));
-  return Error::success();
+  return StreamData.size() - 1;
 }
 
 Error MSFBuilder::setStreamSize(uint32_t Idx, uint32_t Size) {
   uint32_t OldSize = getStreamSize(Idx);
   if (OldSize == Size)
     return Error::success();
 
   uint32_t NewBlocks = bytesToBlocks(Size, BlockSize);

llvm/trunk/lib/DebugInfo/PDB/CMakeLists.txt

add_pdb_impl_folder(Raw
   Raw/NameMap.cpp
   Raw/NameMapBuilder.cpp
   Raw/PDBFile.cpp
   Raw/PDBFileBuilder.cpp
   Raw/PublicsStream.cpp
   Raw/RawError.cpp
   Raw/RawSession.cpp
   Raw/SymbolStream.cpp
+  Raw/TpiHashing.cpp
   Raw/TpiStream.cpp
   Raw/TpiStreamBuilder.cpp)
 
 list(APPEND LIBPDB_ADDITIONAL_HEADER_DIRS "${LLVM_MAIN_INCLUDE_DIR}/llvm/DebugInfo/PDB/Raw")
 list(APPEND LIBPDB_ADDITIONAL_HEADER_DIRS "${LLVM_MAIN_INCLUDE_DIR}/llvm/DebugInfo/PDB")
 
 add_llvm_library(LLVMDebugInfoPDB
   GenericError.cpp

llvm/trunk/lib/DebugInfo/PDB/Raw/DbiStreamBuilder.cpp

 //===- DbiStreamBuilder.cpp - PDB Dbi Stream Creation -----------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/PDB/Raw/DbiStreamBuilder.h"
 
+#include "llvm/DebugInfo/MSF/MSFBuilder.h"
 #include "llvm/DebugInfo/MSF/MappedBlockStream.h"
 #include "llvm/DebugInfo/MSF/StreamWriter.h"
 #include "llvm/DebugInfo/PDB/Raw/DbiStream.h"
 #include "llvm/DebugInfo/PDB/Raw/RawError.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 using namespace llvm::msf;
 using namespace llvm::pdb;
 
 namespace {
 class ModiSubstreamBuilder {};
 }
 
-DbiStreamBuilder::DbiStreamBuilder(BumpPtrAllocator &Allocator)
-    : Allocator(Allocator), Age(1), BuildNumber(0), PdbDllVersion(0),
-      PdbDllRbld(0), Flags(0), MachineType(PDB_Machine::x86), Header(nullptr) {}
+DbiStreamBuilder::DbiStreamBuilder(msf::MSFBuilder &Msf)
+    : Msf(Msf), Allocator(Msf.getAllocator()), Age(1), BuildNumber(0),
+      PdbDllVersion(0), PdbDllRbld(0), Flags(0), MachineType(PDB_Machine::x86),
+      Header(nullptr) {}
 
 void DbiStreamBuilder::setVersionHeader(PdbRaw_DbiVer V) { VerHeader = V; }
 
 void DbiStreamBuilder::setAge(uint32_t A) { Age = A; }
 
 void DbiStreamBuilder::setBuildNumber(uint16_t B) { BuildNumber = B; }
 
 void DbiStreamBuilder::setPdbDllVersion(uint16_t V) { PdbDllVersion = V; }
Error DbiStreamBuilder::finalize() {
   H->PublicSymbolStreamIndex = kInvalidStreamIndex;
   H->MFCTypeServerIndex = kInvalidStreamIndex;
   H->GlobalSymbolStreamIndex = kInvalidStreamIndex;
 
   Header = H;
   return Error::success();
 }
 
+Error DbiStreamBuilder::finalizeMsfLayout() {
+  uint32_t Length = calculateSerializedLength();
+  if (auto EC = Msf.setStreamSize(StreamDBI, Length))
+    return EC;
+  return Error::success();
+}
+
 Expected<std::unique_ptr<DbiStream>>
 DbiStreamBuilder::build(PDBFile &File, const msf::WritableStream &Buffer) {
   if (!VerHeader.hasValue())
     return make_error<RawError>(raw_error_code::unspecified,
                                 "Missing DBI Stream Version");
   if (auto EC = finalize())
     return std::move(EC);
 

llvm/trunk/lib/DebugInfo/PDB/Raw/InfoStreamBuilder.cpp

 //===- InfoStreamBuilder.cpp - PDB Info Stream Creation ---------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/PDB/Raw/InfoStreamBuilder.h"
 
+#include "llvm/DebugInfo/MSF/MSFBuilder.h"
 #include "llvm/DebugInfo/MSF/MappedBlockStream.h"
 #include "llvm/DebugInfo/MSF/StreamWriter.h"
 #include "llvm/DebugInfo/PDB/Raw/InfoStream.h"
 #include "llvm/DebugInfo/PDB/Raw/RawError.h"
 #include "llvm/DebugInfo/PDB/Raw/RawTypes.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 using namespace llvm::msf;
 using namespace llvm::pdb;
 
-InfoStreamBuilder::InfoStreamBuilder()
-    : Ver(PdbRaw_ImplVer::PdbImplVC70), Sig(-1), Age(0) {}
+InfoStreamBuilder::InfoStreamBuilder(msf::MSFBuilder &Msf)
+    : Msf(Msf), Ver(PdbRaw_ImplVer::PdbImplVC70), Sig(-1), Age(0) {}
 
 void InfoStreamBuilder::setVersion(PdbRaw_ImplVer V) { Ver = V; }
 
 void InfoStreamBuilder::setSignature(uint32_t S) { Sig = S; }
 
 void InfoStreamBuilder::setAge(uint32_t A) { Age = A; }
 
 void InfoStreamBuilder::setGuid(PDB_UniqueId G) { Guid = G; }
 
 NameMapBuilder &InfoStreamBuilder::getNamedStreamsBuilder() {
   return NamedStreams;
 }
 
 uint32_t InfoStreamBuilder::calculateSerializedLength() const {
   return sizeof(InfoStreamHeader) + NamedStreams.calculateSerializedLength();
 }
 
+Error InfoStreamBuilder::finalizeMsfLayout() {
+  uint32_t Length = calculateSerializedLength();
+  if (auto EC = Msf.setStreamSize(StreamPDB, Length))
+    return EC;
+  return Error::success();
+}
+
 Expected<std::unique_ptr<InfoStream>>
 InfoStreamBuilder::build(PDBFile &File, const msf::WritableStream &Buffer) {
   auto StreamData = MappedBlockStream::createIndexedStream(File.getMsfLayout(),
                                                            Buffer, StreamPDB);
   auto Info = llvm::make_unique<InfoStream>(std::move(StreamData));
   Info->Version = Ver;
   Info->Signature = Sig;
   Info->Age = Age;

llvm/trunk/lib/DebugInfo/PDB/Raw/PDBFileBuilder.cpp

Error PDBFileBuilder::initialize(const msf::SuperBlock &Super) {
   Msf->setUnknown1(Super.Unknown1);
   return Error::success();
 }
 
 MSFBuilder &PDBFileBuilder::getMsfBuilder() { return *Msf; }
 
 InfoStreamBuilder &PDBFileBuilder::getInfoBuilder() {
   if (!Info)
-    Info = llvm::make_unique<InfoStreamBuilder>();
+    Info = llvm::make_unique<InfoStreamBuilder>(*Msf);
   return *Info;
 }
 
 DbiStreamBuilder &PDBFileBuilder::getDbiBuilder() {
   if (!Dbi)
-    Dbi = llvm::make_unique<DbiStreamBuilder>(Allocator);
+    Dbi = llvm::make_unique<DbiStreamBuilder>(*Msf);
   return *Dbi;
 }
 
 TpiStreamBuilder &PDBFileBuilder::getTpiBuilder() {
   if (!Tpi)
-    Tpi = llvm::make_unique<TpiStreamBuilder>(Allocator);
+    Tpi = llvm::make_unique<TpiStreamBuilder>(*Msf);
   return *Tpi;
 }
 
 Expected<msf::MSFLayout> PDBFileBuilder::finalizeMsfLayout() const {
   if (Info) {
-    uint32_t Length = Info->calculateSerializedLength();
-    if (auto EC = Msf->setStreamSize(StreamPDB, Length))
+    if (auto EC = Info->finalizeMsfLayout())
       return std::move(EC);
   }
   if (Dbi) {
-    uint32_t Length = Dbi->calculateSerializedLength();
-    if (auto EC = Msf->setStreamSize(StreamDBI, Length))
+    if (auto EC = Dbi->finalizeMsfLayout())
       return std::move(EC);
   }
   if (Tpi) {
-    uint32_t Length = Tpi->calculateSerializedLength();
-    if (auto EC = Msf->setStreamSize(StreamTPI, Length))
+    if (auto EC = Tpi->finalizeMsfLayout())
       return std::move(EC);
   }
 
   return Msf->build();
 }
 
 Expected<std::unique_ptr<PDBFile>>
 PDBFileBuilder::build(std::unique_ptr<msf::WritableStream> PdbFileBuffer) {
Error PDBFileBuilder::commit(const msf::WritableStream &Buffer) {
 
   if (Tpi) {
     if (auto EC = Tpi->commit(Layout, Buffer))
       return EC;
   }
 
   return Buffer.commit();
 }
 No newline at end of file

llvm/trunk/lib/DebugInfo/PDB/Raw/TpiHashing.cpp

+//===- TpiHashing.cpp -----------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DebugInfo/PDB/Raw/TpiHashing.h"
+
+#include "llvm/DebugInfo/PDB/Raw/Hash.h"
+#include "llvm/DebugInfo/PDB/Raw/RawError.h"
+
+using namespace llvm;
+using namespace llvm::codeview;
+using namespace llvm::pdb;
+
+// Corresponds to `fUDTAnon`.
+template <typename T> static bool isAnonymous(T &Rec) {
+  StringRef Name = Rec.getName();
+  return Name == "<unnamed-tag>" || Name == "__unnamed" ||
+         Name.endswith("::<unnamed-tag>") || Name.endswith("::__unnamed");
+}
+
+// Computes a hash for a given TPI record.
+template <typename T>
+static uint32_t getTpiHash(T &Rec, ArrayRef<uint8_t> FullRecord) {
+  auto Opts = static_cast<uint16_t>(Rec.getOptions());
+
+  bool ForwardRef =
+      Opts & static_cast<uint16_t>(ClassOptions::ForwardReference);
+  bool Scoped = Opts & static_cast<uint16_t>(ClassOptions::Scoped);
+  bool UniqueName = Opts & static_cast<uint16_t>(ClassOptions::HasUniqueName);
+  bool IsAnon = UniqueName && isAnonymous(Rec);
+
+  if (!ForwardRef && !Scoped && !IsAnon)
+    return hashStringV1(Rec.getName());
+  if (!ForwardRef && UniqueName && !IsAnon)
+    return hashStringV1(Rec.getUniqueName());
+  return hashBufferV8(FullRecord);
+}
+
+template <typename T> static uint32_t getSourceLineHash(T &Rec) {
+  char Buf[4];
+  support::endian::write32le(Buf, Rec.getUDT().getIndex());
+  return hashStringV1(StringRef(Buf, 4));
+}
+
+void TpiHashUpdater::visitKnownRecordImpl(CVType &CVR,
+                                          UdtSourceLineRecord &Rec) {
+  CVR.Hash = getSourceLineHash(Rec);
+}
+
+void TpiHashUpdater::visitKnownRecordImpl(CVType &CVR,
+                                          UdtModSourceLineRecord &Rec) {
+  CVR.Hash = getSourceLineHash(Rec);
+}
+
+void TpiHashUpdater::visitKnownRecordImpl(CVType &CVR, ClassRecord &Rec) {
+  CVR.Hash = getTpiHash(Rec, CVR.RawData);
+}
+
+void TpiHashUpdater::visitKnownRecordImpl(CVType &CVR, EnumRecord &Rec) {
+  CVR.Hash = getTpiHash(Rec, CVR.RawData);
+}
+
+void TpiHashUpdater::visitKnownRecordImpl(CVType &CVR, UnionRecord &Rec) {
+  CVR.Hash = getTpiHash(Rec, CVR.RawData);
+}
+
+Error TpiHashVerifier::visitKnownRecord(CVType &CVR, UdtSourceLineRecord &Rec) {
+  return verifySourceLine(Rec.getUDT());
+}
+
+Error TpiHashVerifier::visitKnownRecord(CVType &CVR,
+                                        UdtModSourceLineRecord &Rec) {
+  return verifySourceLine(Rec.getUDT());
+}
+
+Error TpiHashVerifier::visitKnownRecord(CVType &CVR, ClassRecord &Rec) {
+  if (getTpiHash(Rec, CVR.RawData) % NumHashBuckets != HashValues[Index])
+    return errorInvalidHash();
+  return Error::success();
+}
+Error TpiHashVerifier::visitKnownRecord(CVType &CVR, EnumRecord &Rec) {
+  if (getTpiHash(Rec, CVR.RawData) % NumHashBuckets != HashValues[Index])
+    return errorInvalidHash();
+  return Error::success();
+}
+Error TpiHashVerifier::visitKnownRecord(CVType &CVR, UnionRecord &Rec) {
+  if (getTpiHash(Rec, CVR.RawData) % NumHashBuckets != HashValues[Index])
+    return errorInvalidHash();
+  return Error::success();
+}
+
+Error TpiHashVerifier::verifySourceLine(codeview::TypeIndex TI) {
+  char Buf[4];
+  support::endian::write32le(Buf, TI.getIndex());
+  uint32_t Hash = hashStringV1(StringRef(Buf, 4));
+  if (Hash % NumHashBuckets != HashValues[Index])
+    return errorInvalidHash();
+  return Error::success();
+}
+
+Error TpiHashVerifier::visitTypeBegin(CVType &Rec) {
+  ++Index;
+  RawRecord = Rec;
+  return Error::success();
+}

llvm/trunk/lib/DebugInfo/PDB/Raw/TpiStream.cpp

 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/CodeView.h"
 #include "llvm/DebugInfo/CodeView/TypeDeserializer.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbackPipeline.h"
 #include "llvm/DebugInfo/MSF/MappedBlockStream.h"
 #include "llvm/DebugInfo/MSF/StreamReader.h"
-#include "llvm/DebugInfo/PDB/Raw/Hash.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/RawConstants.h"
 #include "llvm/DebugInfo/PDB/Raw/RawError.h"
 #include "llvm/DebugInfo/PDB/Raw/RawTypes.h"
+#include "llvm/DebugInfo/PDB/Raw/TpiHashing.h"
 
 #include "llvm/Support/Endian.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 using namespace llvm::support;
 using namespace llvm::msf;
 using namespace llvm::pdb;
 
 TpiStream::TpiStream(const PDBFile &File,
                      std::unique_ptr<MappedBlockStream> Stream)
     : Pdb(File), Stream(std::move(Stream)) {}
 
 TpiStream::~TpiStream() {}
 
-// Corresponds to `fUDTAnon`.
-template <typename T> static bool isAnonymous(T &Rec) {
-  StringRef Name = Rec.getName();
-  return Name == "<unnamed-tag>" || Name == "__unnamed" ||
-      Name.endswith("::<unnamed-tag>") || Name.endswith("::__unnamed");
-}
-
-// Computes a hash for a given TPI record.
-template <typename T>
-static uint32_t getTpiHash(T &Rec, const CVRecord<TypeLeafKind> &RawRec) {
-  auto Opts = static_cast<uint16_t>(Rec.getOptions());
-
-  bool ForwardRef =
-      Opts & static_cast<uint16_t>(ClassOptions::ForwardReference);
-  bool Scoped = Opts & static_cast<uint16_t>(ClassOptions::Scoped);
-  bool UniqueName = Opts & static_cast<uint16_t>(ClassOptions::HasUniqueName);
-  bool IsAnon = UniqueName && isAnonymous(Rec);
-
-  if (!ForwardRef && !Scoped && !IsAnon)
-    return hashStringV1(Rec.getName());
-  if (!ForwardRef && UniqueName && !IsAnon)
-    return hashStringV1(Rec.getUniqueName());
-  return hashBufferV8(RawRec.RawData);
-}
-
-namespace {
-class TpiHashVerifier : public TypeVisitorCallbacks {
-public:
-  TpiHashVerifier(FixedStreamArray<support::ulittle32_t> &HashValues,
-                  uint32_t NumHashBuckets)
-      : HashValues(HashValues), NumHashBuckets(NumHashBuckets) {}
-
-  Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,
-                         UdtSourceLineRecord &Rec) override {
-    return verifySourceLine(Rec);
-  }
-
-  Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,
-                         UdtModSourceLineRecord &Rec) override {
-    return verifySourceLine(Rec);
-  }
-
-  Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,
-                         ClassRecord &Rec) override {
-    return verify(Rec);
-  }
-  Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,
-                         EnumRecord &Rec) override {
-    return verify(Rec);
-  }
-  Error visitKnownRecord(CVRecord<TypeLeafKind> &CVR,
-                         UnionRecord &Rec) override {
-    return verify(Rec);
-  }
-
-  Error visitTypeBegin(CVRecord<TypeLeafKind> &Rec) override {
-    ++Index;
-    RawRecord = Rec;
-    return Error::success();
-  }
-
-private:
-  template <typename T> Error verify(T &Rec) {
-    uint32_t Hash = getTpiHash(Rec, RawRecord);
-    if (Hash % NumHashBuckets != HashValues[Index])
-      return errorInvalidHash();
-    return Error::success();
-  }
-
-  template <typename T> Error verifySourceLine(T &Rec) {
-    char Buf[4];
-    support::endian::write32le(Buf, Rec.getUDT().getIndex());
-    uint32_t Hash = hashStringV1(StringRef(Buf, 4));
-    if (Hash % NumHashBuckets != HashValues[Index])
-      return errorInvalidHash();
-    return Error::success();
-  }
-
-  Error errorInvalidHash() {
-    return make_error<RawError>(
-        raw_error_code::invalid_tpi_hash,
-        "Type index is 0x" + utohexstr(TypeIndex::FirstNonSimpleIndex + Index));
-  }
-
-  FixedStreamArray<support::ulittle32_t> HashValues;
-  CVRecord<TypeLeafKind> RawRecord;
-  uint32_t NumHashBuckets;
-  uint32_t Index = -1;
-};
-}
-
 // Verifies that a given type record matches with a given hash value.
 // Currently we only verify SRC_LINE records.
 Error TpiStream::verifyHashValues() {
   TpiHashVerifier Verifier(HashValues, Header->NumHashBuckets);
   TypeDeserializer Deserializer;
 
   TypeVisitorCallbackPipeline Pipeline;
   Pipeline.addCallbackToPipeline(Deserializer);
uint32_t NumHashValues =
         Header->HashValueBuffer.Length / sizeof(ulittle32_t);
     if (NumHashValues != NumTypeRecords())
       return make_error<RawError>(
           raw_error_code::corrupt_file,
           "TPI hash count does not match with the number of type records.");
     HSR.setOffset(Header->HashValueBuffer.Off);
     if (auto EC = HSR.readArray(HashValues, NumHashValues))
       return EC;
+    std::vector<ulittle32_t> HashValueList;
+    for (auto I : HashValues)
+      HashValueList.push_back(I);
 
     HSR.setOffset(Header->IndexOffsetBuffer.Off);
     uint32_t NumTypeIndexOffsets =
         Header->IndexOffsetBuffer.Length / sizeof(TypeIndexOffset);
     if (auto EC = HSR.readArray(TypeIndexOffsets, NumTypeIndexOffsets))
       return EC;
 
     HSR.setOffset(Header->HashAdjBuffer.Off);

llvm/trunk/lib/DebugInfo/PDB/Raw/TpiStreamBuilder.cpp

 #include "llvm/DebugInfo/PDB/Raw/TpiStreamBuilder.h"
 
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
+#include "llvm/DebugInfo/MSF/MSFBuilder.h"
 #include "llvm/DebugInfo/MSF/MappedBlockStream.h"
 #include "llvm/DebugInfo/MSF/StreamWriter.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/RawError.h"
 #include "llvm/DebugInfo/PDB/Raw/TpiStream.h"
 #include "llvm/Support/Allocator.h"
 
 using namespace llvm;
 using namespace llvm::msf;
 using namespace llvm::pdb;
 using namespace llvm::support;
 
-TpiStreamBuilder::TpiStreamBuilder(BumpPtrAllocator &Allocator)
-    : Allocator(Allocator), Header(nullptr) {}
+TpiStreamBuilder::TpiStreamBuilder(MSFBuilder &Msf)
+    : Msf(Msf), Allocator(Msf.getAllocator()), Header(nullptr) {}
 
 TpiStreamBuilder::~TpiStreamBuilder() {}
 
 void TpiStreamBuilder::setVersionHeader(PdbRaw_TpiVer Version) {
   VerHeader = Version;
 }
 
 void TpiStreamBuilder::addTypeRecord(const codeview::CVType &Record) {
   TypeRecords.push_back(Record);
   TypeRecordStream.setItems(TypeRecords);
 }
 
 Error TpiStreamBuilder::finalize() {
   if (Header)
     return Error::success();
 
   TpiStreamHeader *H = Allocator.Allocate<TpiStreamHeader>();
 
   uint32_t Count = TypeRecords.size();
+  uint32_t HashBufferSize = calculateHashBufferSize();
 
   H->Version = *VerHeader;
   H->HeaderSize = sizeof(TpiStreamHeader);
   H->TypeIndexBegin = codeview::TypeIndex::FirstNonSimpleIndex;
   H->TypeIndexEnd = H->TypeIndexBegin + Count;
   H->TypeRecordBytes = TypeRecordStream.getLength();
 
-  H->HashStreamIndex = kInvalidStreamIndex;
+  H->HashStreamIndex = HashStreamIndex;
   H->HashAuxStreamIndex = kInvalidStreamIndex;
   H->HashKeySize = sizeof(ulittle32_t);
   H->NumHashBuckets = MinTpiHashBuckets;
 
-  H->HashValueBuffer.Length = 0;
+  // Recall that hash values go into a completely different stream identified by
+  // the `HashStreamIndex` field of the `TpiStreamHeader`.  Therefore, the data
+  // begins at offset 0 of this independent stream.
+  H->HashValueBuffer.Off = 0;
+  H->HashValueBuffer.Length = HashBufferSize;
+  H->HashAdjBuffer.Off = H->HashValueBuffer.Off + H->HashValueBuffer.Length;
   H->HashAdjBuffer.Length = 0;
+  H->IndexOffsetBuffer.Off = H->HashAdjBuffer.Off + H->HashAdjBuffer.Length;
   H->IndexOffsetBuffer.Length = 0;
 
   Header = H;
   return Error::success();
 }
 
 uint32_t TpiStreamBuilder::calculateSerializedLength() const {
-  return sizeof(TpiStreamHeader) + TypeRecordStream.getLength();
+  return sizeof(TpiStreamHeader) + TypeRecordStream.getLength() +
+         calculateHashBufferSize();
+}
+
+uint32_t TpiStreamBuilder::calculateHashBufferSize() const {
+  if (TypeRecords.empty() || !TypeRecords[0].Hash.hasValue())
+    return 0;
+  return TypeRecords.size() * sizeof(ulittle32_t);
+}
+
+Error TpiStreamBuilder::finalizeMsfLayout() {
+  uint32_t Length = calculateSerializedLength();
+  if (auto EC = Msf.setStreamSize(StreamTPI, Length))
+    return EC;
+
+  uint32_t HashBufferSize = calculateHashBufferSize();
+
+  if (HashBufferSize == 0)
+    return Error::success();
+
+  auto ExpectedIndex = Msf.addStream(HashBufferSize);
+  if (!ExpectedIndex)
+    return ExpectedIndex.takeError();
+  HashStreamIndex = *ExpectedIndex;
+  ulittle32_t *H = Allocator.Allocate<ulittle32_t>(TypeRecords.size());
+  MutableArrayRef<ulittle32_t> HashBuffer(H, TypeRecords.size());
+  for (uint32_t I = 0; I < TypeRecords.size(); ++I) {
+    HashBuffer[I] = *TypeRecords[I].Hash % MinTpiHashBuckets;
+  }
+  ArrayRef<uint8_t> Bytes(reinterpret_cast<const uint8_t *>(HashBuffer.data()),
+                          HashBufferSize);
+  HashValueStream = llvm::make_unique<ByteStream>(Bytes);
+  return Error::success();
 }
 
 Expected<std::unique_ptr<TpiStream>>
 TpiStreamBuilder::build(PDBFile &File, const msf::WritableStream &Buffer) {
   if (!VerHeader.hasValue())
     return make_error<RawError>(raw_error_code::unspecified,
                                 "Missing TPI Stream Version");
   if (auto EC = finalize())
     return std::move(EC);
 
   auto StreamData = MappedBlockStream::createIndexedStream(File.getMsfLayout(),
                                                            Buffer, StreamTPI);
   auto Tpi = llvm::make_unique<TpiStream>(File, std::move(StreamData));
   Tpi->Header = Header;
   Tpi->TypeRecords = VarStreamArray<codeview::CVType>(TypeRecordStream);
+  if (HashValueStream) {
+    Tpi->HashStream = std::move(HashValueStream);
+    StreamReader HSR(*Tpi->HashStream);
+    if (auto EC = HSR.readArray(Tpi->HashValues, TypeRecords.size()))
+      return std::move(EC);
+  }
   return std::move(Tpi);
 }
 
 Error TpiStreamBuilder::commit(const msf::MSFLayout &Layout,
                                const msf::WritableStream &Buffer) {
   if (auto EC = finalize())
     return EC;
 
   auto InfoS =
       WritableMappedBlockStream::createIndexedStream(Layout, Buffer, StreamTPI);
 
   StreamWriter Writer(*InfoS);
   if (auto EC = Writer.writeObject(*Header))
     return EC;
 
   auto RecordArray = VarStreamArray<codeview::CVType>(TypeRecordStream);
   if (auto EC = Writer.writeArray(RecordArray))
     return EC;
 
+  if (HashStreamIndex != kInvalidStreamIndex) {
+    auto HVS = WritableMappedBlockStream::createIndexedStream(Layout, Buffer,
+                                                              HashStreamIndex);
+    StreamWriter HW(*HVS);
+    if (auto EC = HW.writeStreamRef(*HashValueStream))
+      return EC;
+  }
+
   return Error::success();
 }

llvm/trunk/test/DebugInfo/PDB/pdbdump-readwrite.test

 RUN: llvm-pdbdump pdb2yaml -dbi-module-info -dbi-module-source-info \
 RUN:   -dbi-stream -pdb-stream -tpi-stream -stream-directory \
 RUN:   -stream-metadata %p/Inputs/empty.pdb > %t.1
 RUN: llvm-pdbdump yaml2pdb -pdb=%t.2 %t.1
 
 RUN: llvm-pdbdump raw -headers -tpi-records %p/Inputs/empty.pdb | FileCheck %s
 RUN: llvm-pdbdump raw -headers -tpi-records %t.2 | FileCheck %s
 
 CHECK:      FileHeaders {
 CHECK-NEXT:   BlockSize: 4096
 CHECK-NEXT:   FreeBlockMap: 2
 CHECK-NEXT:   NumBlocks: 25
-CHECK-NEXT:   NumDirectoryBytes: 136
+CHECK-NEXT:   NumDirectoryBytes:
 CHECK-NEXT:   Unknown1: 0
-CHECK-NEXT:   BlockMapAddr: 24
+CHECK-NEXT:   BlockMapAddr:
 CHECK-NEXT:   NumDirectoryBlocks: 1
-CHECK-NEXT:   DirectoryBlocks: [23]
-CHECK-NEXT:   NumStreams: 17
+CHECK-NEXT:   DirectoryBlocks:
+CHECK-NEXT:   NumStreams:
 CHECK-NEXT: }
 CHECK:      PDB Stream {
 CHECK-NEXT:   Version: 20000404
 CHECK-NEXT:   Signature: 0x54E507E2
 CHECK-NEXT:   Age: 1
 CHECK-NEXT:   Guid: {0B355641-86A0-A249-896F-9988FAE52FF0}
 CHECK-NEXT: }
 CHECK:      Type Info Stream (TPI) {

llvm/trunk/test/DebugInfo/PDB/pdbdump-write.test

 ; This testcase checks to make sure that we can write PDB files.  It
 ; works by first reading a known good PDB file and dumping the contents
 ; to YAML.  Then it tries to reconstruct as much of the original PDB as
 ; possible, although depending on what flags are specified when generating
 ; the YAML, the PDB might be missing data required for any standard tool
 ; to recognize it.  Finally, it dumps the same set of fields from the newly
 ; constructed PDB to YAML, and verifies that the YAML is the same as the
 ; original YAML generated from the good PDB.  Note that when doing the
 ; final comparison it must dump the original and the new pdb without any
 ; stream metadata, since the layout of the MSF file might be different
 ; (for example if we don't write the entire stream)
 ;
 ; RUN: llvm-pdbdump pdb2yaml -stream-metadata -stream-directory -pdb-stream -tpi-stream %p/Inputs/empty.pdb > %t.1
 ; RUN: llvm-pdbdump yaml2pdb -pdb=%t.2 %t.1
-; RUN: llvm-pdbdump pdb2yaml -pdb-stream -tpi-stream %p/Inputs/empty.pdb > %t.3
-; RUN: llvm-pdbdump pdb2yaml -pdb-stream -tpi-stream %t.2 > %t.4
+; RUN: llvm-pdbdump pdb2yaml -pdb-stream -tpi-stream -no-file-headers %p/Inputs/empty.pdb > %t.3
+; RUN: llvm-pdbdump pdb2yaml -pdb-stream -tpi-stream -no-file-headers %t.2 > %t.4
 ; RUN: diff %t.3 %t.4

llvm/trunk/tools/llvm-pdbdump/PdbYaml.cpp

 
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/TypeDeserializer.h"
 #include "llvm/DebugInfo/CodeView/TypeSerializationVisitor.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbackPipeline.h"
 #include "llvm/DebugInfo/PDB/PDBExtras.h"
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
+#include "llvm/DebugInfo/PDB/Raw/TpiHashing.h"
 
 using namespace llvm;
 using namespace llvm::pdb;
 using namespace llvm::pdb::yaml;
 using namespace llvm::yaml;
 
 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(uint32_t)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::StringRef)

 
 void MappingContextTraits<PdbTpiRecord, pdb::yaml::SerializationContext>::
     mapping(IO &IO, pdb::yaml::PdbTpiRecord &Obj,
             pdb::yaml::SerializationContext &Context) {
   codeview::TypeVisitorCallbackPipeline Pipeline;
   codeview::TypeDeserializer Deserializer;
   codeview::TypeSerializationVisitor Serializer(Context.FieldListBuilder,
                                                 Context.TypeTableBuilder);
+  pdb::TpiHashUpdater Hasher;
 
   if (IO.outputting()) {
     // For PDB to Yaml, deserialize into a high level record type, then dump it.
     Pipeline.addCallbackToPipeline(Deserializer);
     Pipeline.addCallbackToPipeline(Context.Dumper);
   } else {
     // For Yaml to PDB, extract from the high level record type, then write it
     // to bytes.
     Pipeline.addCallbackToPipeline(Context.Dumper);
     Pipeline.addCallbackToPipeline(Serializer);
+    Pipeline.addCallbackToPipeline(Hasher);
   }
 
   codeview::CVTypeVisitor Visitor(Pipeline);
   consumeError(Visitor.visitTypeRecord(Obj.Record));
 }

llvm/trunk/tools/llvm-pdbdump/llvm-pdbdump.cpp

if (OutFileOrError.getError())
     ExitOnErr(make_error<GenericError>(generic_error_code::invalid_path,
                                        opts::yaml2pdb::YamlPdbOutputFile));
 
   auto FileByteStream =
       llvm::make_unique<FileBufferByteStream>(std::move(*OutFileOrError));
   PDBFileBuilder Builder(Allocator);
 
   ExitOnErr(Builder.initialize(YamlObj.Headers->SuperBlock));
-  ExitOnErr(Builder.getMsfBuilder().setDirectoryBlocksHint(
-      YamlObj.Headers->DirectoryBlocks));
-  if (!YamlObj.StreamSizes.hasValue()) {
-    ExitOnErr(make_error<GenericError>(
-        generic_error_code::unspecified,
-        "Cannot generate a PDB when stream sizes are not known"));
-  }
-
-  if (YamlObj.StreamMap.hasValue()) {
-    if (YamlObj.StreamMap->size() != YamlObj.StreamSizes->size()) {
-      ExitOnErr(make_error<GenericError>(generic_error_code::unspecified,
-                                         "YAML specifies different number of "
-                                         "streams in stream sizes and stream "
-                                         "map"));
-    }
-
-    auto &Sizes = *YamlObj.StreamSizes;
-    auto &Map = *YamlObj.StreamMap;
-    for (uint32_t I = 0; I < Sizes.size(); ++I) {
-      uint32_t Size = Sizes[I];
-      std::vector<uint32_t> Blocks;
-      for (auto E : Map[I].Blocks)
-        Blocks.push_back(E);
-      ExitOnErr(Builder.getMsfBuilder().addStream(Size, Blocks));
-    }
-  } else {
-    auto &Sizes = *YamlObj.StreamSizes;
-    for (auto S : Sizes) {
-      ExitOnErr(Builder.getMsfBuilder().addStream(S));
-    }
-  }
+  // Add each of the reserved streams.  We ignore stream metadata in the
+  // yaml, because we will reconstruct our own view of the streams.  For
+  // example, the YAML may say that there were 20 streams in the original
+  // PDB, but maybe we only dump a subset of those 20 streams, so we will
+  // have fewer, and the ones we do have may end up with different indices
+  // than the ones in the original PDB.  So we just start with a clean slate.
+  for (uint32_t I = 0; I < kSpecialStreamCount; ++I)
+    ExitOnErr(Builder.getMsfBuilder().addStream(0));
 
   if (YamlObj.PdbStream.hasValue()) {
     auto &InfoBuilder = Builder.getInfoBuilder();
     InfoBuilder.setAge(YamlObj.PdbStream->Age);
     InfoBuilder.setGuid(YamlObj.PdbStream->Guid);
     InfoBuilder.setSignature(YamlObj.PdbStream->Signature);
     InfoBuilder.setVersion(YamlObj.PdbStream->Version);
     for (auto &NM : YamlObj.PdbStream->NamedStreams)