[llvm-pdbdump] Add the beginning of PDB diffing support

[zturner]

Summary

For now we only support diffing the MSF Super Block and the Stream Directory. Subsequent patches will drill down into individual streams.

[ruiu]

Compared to dumping two files using pdb2yaml and diffing them, what is the advantage of this tool?

llvm/tools/llvm-pdbdump/Diff.cpp

103 ↗

(On Diff #91613)

No newline at end?

[zturner]

In D30908#699783, @ruiu wrote:

Compared to dumping two files using pdb2yaml and diffing them, what is the advantage of this tool?

Well, as an extreme (but obviously hypothetical) example, consider the case where pdb2yaml outputs a constant yaml file no matter what input it gets. The diff would always be the same, but the underlying pdbs could be entirely unrelated.

This is obviously contrived, but the point is that if there are bugs in tool X, it doesn't make sense to use the output of tool X to verify that tool X is working correctly.

Beyond that though, pdb2yaml is only intended to give you a textual representation of the contents. It's useful to be able to understand how 2 PDB files differ at the byte level. It's entirely possible that two PDBs which differ in subtle ways could produce the same YAML. Also, even with a 50MB pdb, pdb2yaml takes an extremely long time to run. If we wanted to do it on something the order of chrome.dll.pdb which is maybe 1GB, it would be impractical.

[zturner]

Minor updates.

As one more point about "why not just diff the YAML", I just ran pdb2yaml and yaml2pdb on a random PDB on my drive. Then I ran llvm-pdbdump diff on it. With just the code in this patch, I get the following output:

Found 8 common streams.  Searching for intra-stream differences.
  DBI Stream (Left: 283128 bytes, Right: 160570 bytes)
  Named Stream "/names" (Left: 58244 bytes, Right: 55870 bytes)
  Old MSF Directory (Left: 40 bytes, Right: 0 bytes)
  PDB Stream (Left: 118 bytes, Right: 114 bytes)
  TPI Stream (Left: 382784 bytes, Right: 382632 bytes)

Note that the TPI stream on the left (the original file) is 152 bytes larger. If I use pdb2yaml again on the new file and diff their contents, nothing appears different about the TPI stream. So we need other tools to figure this kind of thing out.

[rnk]

It's also possible to have two PDBs with equivalent type streams but different type index numberings. This diffing tool would make it possible to look past those superficial differences.

[rnk]

lgtm

[amccarth]

llvm/tools/llvm-pdbdump/Diff.cpp

117 ↗

(On Diff #91613)

Should you report if P.size() != Q.size()?

122 ↗

(On Diff #91613)

Should that be !=? (Next line as well.)

150 ↗

(On Diff #91613)

Is the Comparator necessary? Isn't the default comparator std::less, which should do exactly the same thing?

166 ↗

(On Diff #91613)

In practice, I suspect .data() will work, but .begin() would seem more principled, since these algorithms work with pairs of iterators. The fact that SmallVector iterators are pointers (as returned by .data()) is an implementation detail I wouldn't rely on.

llvm/tools/llvm-pdbdump/StreamUtil.h

20 ↗

(On Diff #91613)

Could you add a comment clarifying what this method does?

[zturner]

llvm/tools/llvm-pdbdump/Diff.cpp

117 ↗

(On Diff #91613)

That should get reported indirectly (because we will just report that one file has streams not present in the other file).

122 ↗

(On Diff #91613)

Yes, fixed in latest diff.

150 ↗

(On Diff #91613)

PI is the value type of the enumerator range, which in this case is something like result_pair<SmallString<32>>. Even if that weren't the case and it were just std::pair<size_t, std::string> though, this comparator is specifically designed to only look at the second item of the pair (the name) and not the index.

166 ↗

(On Diff #91613)

begin() would actually be wrong here. I've only called reserve, not resize, so size() == 0. SmallVector<> has this optimization so that you can copy objects into it without default constructing them first, and then calling set_size() (which is different than resize()) after you write into its internal buffer.

[amccarth]

llvm/tools/llvm-pdbdump/Diff.cpp

166 ↗

(On Diff #91613)

I understand you've only reserved space using this SmallVector feature. I would expect begin on an empty SmallVector with a reservation to return an iterator at the beginning of the buffer and for end to return the same thing.

In fact, the implementation of SmallVector::data() is to return begin() coerced to the container's pointer type (which happens to be identical to its iterator type).

/// Return a pointer to the vector's buffer, even if empty().
pointer data() { return pointer(begin()); }

I normally wouldn't go diving into a container's implementation details, but it's the only place I can find that documents this feature of using the reserved buffer of an empty container.

Mixing pointers and iterators of a SmallVector works because they're interchangeable, but it seems to make this code fragile if someone were ever to come along and select a different container type, where the same effect could be achieved with a reserve and a back_insert_iterator.

llvm/trunk/tools/llvm-pdbdump/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   DebugInfoCodeView
   DebugInfoMSF
   DebugInfoPDB
   Object
   Support
   )
 
 add_llvm_tool(llvm-pdbdump
   Analyze.cpp
+  Diff.cpp
   CompactTypeDumpVisitor.cpp
   llvm-pdbdump.cpp
   YamlSymbolDumper.cpp
   YamlTypeDumper.cpp
   LinePrinter.cpp
   LLVMOutputStyle.cpp
   PdbYaml.cpp
   PrettyBuiltinDumper.cpp
   PrettyClassDefinitionDumper.cpp
   PrettyCompilandDumper.cpp
   PrettyEnumDumper.cpp
   PrettyExternalSymbolDumper.cpp
   PrettyFunctionDumper.cpp
   PrettyTypeDumper.cpp
   PrettyTypedefDumper.cpp
   PrettyVariableDumper.cpp
+  StreamUtil.cpp
   YAMLOutputStyle.cpp
   )
 
 if(LLVM_USE_SANITIZE_COVERAGE)
   add_subdirectory(fuzzer)
 endif()

llvm/trunk/tools/llvm-pdbdump/Diff.h

+//===- Diff.h - PDB diff utility --------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVMPDBDUMP_DIFF_H
+#define LLVM_TOOLS_LLVMPDBDUMP_DIFF_H
+
+#include "OutputStyle.h"
+
+namespace llvm {
+namespace pdb {
+class PDBFile;
+class DiffStyle : public OutputStyle {
+public:
+  explicit DiffStyle(PDBFile &File1, PDBFile &File2);
+
+  Error dump() override;
+
+private:
+  Error diffSuperBlock();
+  Error diffStreamDirectory();
+  Error diffStringTable();
+  Error diffFreePageMap();
+  Error diffInfoStream();
+  Error diffDbiStream();
+  Error diffSectionContribs();
+  Error diffSectionMap();
+  Error diffFpoStream();
+  Error diffTpiStream(int Index);
+  Error diffModuleInfoStream(int Index);
+  Error diffPublics();
+  Error diffGlobals();
+
+  PDBFile &File1;
+  PDBFile &File2;
+};
+}
+}
+
+#endif

llvm/trunk/tools/llvm-pdbdump/Diff.cpp

+//===- Diff.cpp - PDB diff utility ------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "Diff.h"
+
+#include "StreamUtil.h"
+#include "llvm-pdbdump.h"
+
+#include "llvm/DebugInfo/PDB/Native/PDBFile.h"
+#include "llvm/DebugInfo/PDB/Native/RawConstants.h"
+
+#include "llvm/Support/FormatAdapters.h"
+#include "llvm/Support/FormatVariadic.h"
+
+using namespace llvm;
+using namespace llvm::pdb;
+
+DiffStyle::DiffStyle(PDBFile &File1, PDBFile &File2)
+    : File1(File1), File2(File2) {}
+
+Error DiffStyle::dump() {
+  if (auto EC = diffSuperBlock())
+    return EC;
+
+  if (auto EC = diffFreePageMap())
+    return EC;
+
+  if (auto EC = diffStreamDirectory())
+    return EC;
+
+  if (auto EC = diffStringTable())
+    return EC;
+
+  if (auto EC = diffInfoStream())
+    return EC;
+
+  if (auto EC = diffDbiStream())
+    return EC;
+
+  if (auto EC = diffSectionContribs())
+    return EC;
+
+  if (auto EC = diffSectionMap())
+    return EC;
+
+  if (auto EC = diffFpoStream())
+    return EC;
+
+  if (auto EC = diffTpiStream(StreamTPI))
+    return EC;
+
+  if (auto EC = diffTpiStream(StreamIPI))
+    return EC;
+
+  if (auto EC = diffPublics())
+    return EC;
+
+  if (auto EC = diffGlobals())
+    return EC;
+
+  return Error::success();
+}
+
+template <typename T>
+static bool diffAndPrint(StringRef Label, PDBFile &File1, PDBFile &File2, T V1,
+                         T V2) {
+  if (V1 != V2) {
+    outs().indent(2) << Label << "\n";
+    outs().indent(4) << formatv("{0}: {1}\n", File1.getFilePath(), V1);
+    outs().indent(4) << formatv("{0}: {1}\n", File2.getFilePath(), V2);
+  }
+  return (V1 != V2);
+}
+
+Error DiffStyle::diffSuperBlock() {
+  outs() << "MSF Super Block: Searching for differences...\n";
+  bool Diffs = false;
+
+  Diffs |= diffAndPrint("Block Size", File1, File2, File1.getBlockSize(),
+                        File2.getBlockSize());
+  Diffs |= diffAndPrint("Block Count", File1, File2, File1.getBlockCount(),
+                        File2.getBlockCount());
+  Diffs |= diffAndPrint("Unknown 1", File1, File2, File1.getUnknown1(),
+                        File2.getUnknown1());
+
+  if (opts::diff::Pedantic) {
+    Diffs |= diffAndPrint("Free Block Map", File1, File2,
+                          File1.getFreeBlockMapBlock(),
+                          File2.getFreeBlockMapBlock());
+    Diffs |= diffAndPrint("Directory Size", File1, File2,
+                          File1.getNumDirectoryBytes(),
+                          File2.getNumDirectoryBytes());
+    Diffs |= diffAndPrint("Block Map Addr", File1, File2,
+                          File1.getBlockMapOffset(), File2.getBlockMapOffset());
+  }
+  if (!Diffs)
+    outs() << "MSF Super Block: No differences detected...\n";
+  return Error::success();
+}
+
+Error DiffStyle::diffStreamDirectory() {
+  SmallVector<std::string, 32> P;
+  SmallVector<std::string, 32> Q;
+  discoverStreamPurposes(File1, P);
+  discoverStreamPurposes(File2, Q);
+  outs() << "Stream Directory: Searching for differences...\n";
+
+  bool HasDifferences = false;
+  if (opts::diff::Pedantic) {
+    size_t Min = std::min(P.size(), Q.size());
+    for (size_t I = 0; I < Min; ++I) {
+      StringRef Names[] = {P[I], Q[I]};
+      uint32_t Sizes[] = {File1.getStreamByteSize(I),
+                          File2.getStreamByteSize(I)};
+      bool NamesDiffer = Names[0] != Names[1];
+      bool SizesDiffer = Sizes[0] != Sizes[1];
+      if (NamesDiffer) {
+        HasDifferences = true;
+        outs().indent(2) << formatv("Stream {0} - {1}: {2}, {3}: {4}\n", I,
+                                    File1.getFilePath(), Names[0],
+                                    File2.getFilePath(), Names[1]);
+        continue;
+      }
+      if (SizesDiffer) {
+        HasDifferences = true;
+        outs().indent(2) << formatv(
+            "Stream {0} ({1}): {2}: {3} bytes, {4}: {5} bytes\n", I, Names[0],
+            File1.getFilePath(), Sizes[0], File2.getFilePath(), Sizes[1]);
+        continue;
+      }
+    }
+
+    ArrayRef<std::string> MaxNames = (P.size() > Q.size() ? P : Q);
+    size_t Max = std::max(P.size(), Q.size());
+    PDBFile &MaxFile = (P.size() > Q.size() ? File1 : File2);
+    StringRef MinFileName =
+        (P.size() < Q.size() ? File1.getFilePath() : File2.getFilePath());
+    for (size_t I = Min; I < Max; ++I) {
+      HasDifferences = true;
+      StringRef StreamName = MaxNames[I];
+
+      outs().indent(2) << formatv(
+          "Stream {0} - {1}: <not present>, {2}: Index {3}, {4} bytes\n",
+          StreamName, MinFileName, MaxFile.getFilePath(), I,
+          MaxFile.getStreamByteSize(I));
+    }
+    if (!HasDifferences)
+      outs() << "Stream Directory: No differences detected...\n";
+  } else {
+    auto PI = to_vector<32>(enumerate(P));
+    auto QI = to_vector<32>(enumerate(Q));
+
+    typedef decltype(PI) ContainerType;
+    typedef typename ContainerType::value_type value_type;
+
+    auto Comparator = [](const value_type &I1, const value_type &I2) {
+      return I1.value() < I2.value();
+    };
+    std::sort(PI.begin(), PI.end(), Comparator);
+    std::sort(QI.begin(), QI.end(), Comparator);
+
+    decltype(PI) OnlyP;
+    decltype(QI) OnlyQ;
+    decltype(PI) Common;
+
+    OnlyP.reserve(P.size());
+    OnlyQ.reserve(Q.size());
+    Common.reserve(Q.size());
+
+    auto PEnd = std::set_difference(PI.begin(), PI.end(), QI.begin(), QI.end(),
+                                    OnlyP.begin(), Comparator);
+    auto QEnd = std::set_difference(QI.begin(), QI.end(), PI.begin(), PI.end(),
+                                    OnlyQ.begin(), Comparator);
+    auto ComEnd = std::set_intersection(PI.begin(), PI.end(), QI.begin(),
+                                        QI.end(), Common.begin(), Comparator);
+    OnlyP.set_size(std::distance(OnlyP.begin(), PEnd));
+    OnlyQ.set_size(std::distance(OnlyQ.begin(), QEnd));
+    Common.set_size(std::distance(Common.begin(), ComEnd));
+
+    if (!OnlyP.empty()) {
+      HasDifferences = true;
+      outs().indent(2) << formatv("{0} Stream(s) only in ({1})\n", OnlyP.size(),
+                                  File1.getFilePath());
+      for (auto &Item : OnlyP) {
+        outs().indent(4) << formatv("Stream {0} - {1}\n", Item.index(),
+                                    Item.value());
+      }
+    }
+
+    if (!OnlyQ.empty()) {
+      HasDifferences = true;
+      outs().indent(2) << formatv("{0} Streams(s) only in ({1})\n",
+                                  OnlyQ.size(), File2.getFilePath());
+      for (auto &Item : OnlyQ) {
+        outs().indent(4) << formatv("Stream {0} - {1}\n", Item.index(),
+                                    Item.value());
+      }
+    }
+    if (!Common.empty()) {
+      outs().indent(2) << formatv("Found {0} common streams.  Searching for "
+                                  "intra-stream differences.\n",
+                                  Common.size());
+      bool HasCommonDifferences = false;
+      for (const auto &Left : Common) {
+        // Left was copied from the first range so its index refers to a stream
+        // index in the first file.  Find the corresponding stream index in the
+        // second file.
+        auto Range =
+            std::equal_range(QI.begin(), QI.end(), Left,
+                             [](const value_type &L, const value_type &R) {
+                               return L.value() < R.value();
+                             });
+        const auto &Right = *Range.first;
+        assert(Left.value() == Right.value());
+        uint32_t LeftSize = File1.getStreamByteSize(Left.index());
+        uint32_t RightSize = File2.getStreamByteSize(Right.index());
+        if (LeftSize != RightSize) {
+          HasDifferences = true;
+          HasCommonDifferences = true;
+          outs().indent(4) << formatv("{0} ({1}: {2} bytes, {3}: {4} bytes)\n",
+                                      Left.value(), File1.getFilePath(),
+                                      LeftSize, File2.getFilePath(), RightSize);
+        }
+      }
+      if (!HasCommonDifferences)
+        outs().indent(2) << "Common Streams:  No differences detected!\n";
+    }
+    if (!HasDifferences)
+      outs() << "Stream Directory: No differences detected!\n";
+  }
+
+  return Error::success();
+}
+
+Error DiffStyle::diffStringTable() { return Error::success(); }
+
+Error DiffStyle::diffFreePageMap() { return Error::success(); }
+
+Error DiffStyle::diffInfoStream() { return Error::success(); }
+
+Error DiffStyle::diffDbiStream() { return Error::success(); }
+
+Error DiffStyle::diffSectionContribs() { return Error::success(); }
+
+Error DiffStyle::diffSectionMap() { return Error::success(); }
+
+Error DiffStyle::diffFpoStream() { return Error::success(); }
+
+Error DiffStyle::diffTpiStream(int Index) { return Error::success(); }
+
+Error DiffStyle::diffModuleInfoStream(int Index) { return Error::success(); }
+
+Error DiffStyle::diffPublics() { return Error::success(); }
+
+Error DiffStyle::diffGlobals() { return Error::success(); }

llvm/trunk/tools/llvm-pdbdump/LLVMOutputStyle.h

 //===- LLVMOutputStyle.h -------------------------------------- *- C++ --*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_TOOLS_LLVMPDBDUMP_LLVMOUTPUTSTYLE_H
 #define LLVM_TOOLS_LLVMPDBDUMP_LLVMOUTPUTSTYLE_H
 
 #include "OutputStyle.h"
 
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/DebugInfo/CodeView/TypeDatabase.h"
 #include "llvm/Support/ScopedPrinter.h"
 
+#include <string>
+
 namespace llvm {
 class BitVector;
 namespace pdb {
 class LLVMOutputStyle : public OutputStyle {
 public:
   LLVMOutputStyle(PDBFile &File);
 
   Error dump() override;
 
 private:
-  void discoverStreamPurposes();
-
   Error dumpFileHeaders();
   Error dumpStreamSummary();
   Error dumpFreePageMap();
   Error dumpBlockRanges();
   Error dumpGlobalsStream();
   Error dumpStreamBytes();
   Error dumpStreamBlocks();
   Error dumpStringTable();
   Error dumpInfoStream();
   Error dumpTpiStream(uint32_t StreamIdx);
   Error dumpDbiStream();
   Error dumpSectionContribs();
   Error dumpSectionMap();
   Error dumpPublicsStream();
   Error dumpSectionHeaders();
   Error dumpFpoStream();
 
   void dumpBitVector(StringRef Name, const BitVector &V);
 
   void flush();
 
   PDBFile &File;
   ScopedPrinter P;
   codeview::TypeDatabase TypeDB;
-  std::vector<std::string> StreamPurposes;
+  SmallVector<std::string, 32> StreamPurposes;
 };
 }
 }
 
 #endif

llvm/trunk/tools/llvm-pdbdump/LLVMOutputStyle.cpp

 //===- LLVMOutputStyle.cpp ------------------------------------ *- C++ --*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "LLVMOutputStyle.h"
 
 #include "CompactTypeDumpVisitor.h"
+#include "StreamUtil.h"
 #include "llvm-pdbdump.h"
 
 #include "llvm/DebugInfo/CodeView/CVTypeDumper.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/EnumTables.h"
 #include "llvm/DebugInfo/CodeView/ModuleSubstreamVisitor.h"
 #include "llvm/DebugInfo/CodeView/SymbolDumper.h"
 #include "llvm/DebugInfo/CodeView/TypeDatabaseVisitor.h"
Error LLVMOutputStyle::dumpFileHeaders() {
   // The directory is not contiguous.  Instead, the block map contains a
   // contiguous list of block numbers whose contents, when concatenated in
   // order, make up the directory.
   P.printList("DirectoryBlocks", File.getDirectoryBlockArray());
   P.printNumber("NumStreams", File.getNumStreams());
   return Error::success();
 }
 
-void LLVMOutputStyle::discoverStreamPurposes() {
-  if (!StreamPurposes.empty())
-    return;
-
-  // It's OK if we fail to load some of these streams, we still attempt to print
-  // what we can.
-  auto Dbi = File.getPDBDbiStream();
-  auto Tpi = File.getPDBTpiStream();
-  auto Ipi = File.getPDBIpiStream();
-  auto Info = File.getPDBInfoStream();
-
-  uint32_t StreamCount = File.getNumStreams();
-  std::unordered_map<uint16_t, const ModuleInfoEx *> ModStreams;
-  std::unordered_map<uint16_t, std::string> NamedStreams;
-
-  if (Dbi) {
-    for (auto &ModI : Dbi->modules()) {
-      uint16_t SN = ModI.Info.getModuleStreamIndex();
-      ModStreams[SN] = &ModI;
-    }
-  }
-  if (Info) {
-    for (auto &NSE : Info->named_streams()) {
-      NamedStreams[NSE.second] = NSE.first();
-    }
-  }
-
-  StreamPurposes.resize(StreamCount);
-  for (uint16_t StreamIdx = 0; StreamIdx < StreamCount; ++StreamIdx) {
-    std::string Value;
-    if (StreamIdx == OldMSFDirectory)
-      Value = "Old MSF Directory";
-    else if (StreamIdx == StreamPDB)
-      Value = "PDB Stream";
-    else if (StreamIdx == StreamDBI)
-      Value = "DBI Stream";
-    else if (StreamIdx == StreamTPI)
-      Value = "TPI Stream";
-    else if (StreamIdx == StreamIPI)
-      Value = "IPI Stream";
-    else if (Dbi && StreamIdx == Dbi->getGlobalSymbolStreamIndex())
-      Value = "Global Symbol Hash";
-    else if (Dbi && StreamIdx == Dbi->getPublicSymbolStreamIndex())
-      Value = "Public Symbol Hash";
-    else if (Dbi && StreamIdx == Dbi->getSymRecordStreamIndex())
-      Value = "Public Symbol Records";
-    else if (Tpi && StreamIdx == Tpi->getTypeHashStreamIndex())
-      Value = "TPI Hash";
-    else if (Tpi && StreamIdx == Tpi->getTypeHashStreamAuxIndex())
-      Value = "TPI Aux Hash";
-    else if (Ipi && StreamIdx == Ipi->getTypeHashStreamIndex())
-      Value = "IPI Hash";
-    else if (Ipi && StreamIdx == Ipi->getTypeHashStreamAuxIndex())
-      Value = "IPI Aux Hash";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Exception))
-      Value = "Exception Data";
-    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Fixup))
-      Value = "Fixup Data";
-    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::FPO))
-      Value = "FPO Data";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::NewFPO))
-      Value = "New FPO Data";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::OmapFromSrc))
-      Value = "Omap From Source Data";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::OmapToSrc))
-      Value = "Omap To Source Data";
-    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Pdata))
-      Value = "Pdata";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::SectionHdr))
-      Value = "Section Header Data";
-    else if (Dbi &&
-             StreamIdx ==
-                 Dbi->getDebugStreamIndex(DbgHeaderType::SectionHdrOrig))
-      Value = "Section Header Original Data";
-    else if (Dbi &&
-             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::TokenRidMap))
-      Value = "Token Rid Data";
-    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Xdata))
-      Value = "Xdata";
-    else {
-      auto ModIter = ModStreams.find(StreamIdx);
-      auto NSIter = NamedStreams.find(StreamIdx);
-      if (ModIter != ModStreams.end()) {
-        Value = "Module \"";
-        Value += ModIter->second->Info.getModuleName().str();
-        Value += "\"";
-      } else if (NSIter != NamedStreams.end()) {
-        Value = "Named Stream \"";
-        Value += NSIter->second;
-        Value += "\"";
-      } else {
-        Value = "???";
-      }
-    }
-    StreamPurposes[StreamIdx] = Value;
-  }
-
-  // Consume errors from missing streams.
-  if (!Dbi)
-    consumeError(Dbi.takeError());
-  if (!Tpi)
-    consumeError(Tpi.takeError());
-  if (!Ipi)
-    consumeError(Ipi.takeError());
-  if (!Info)
-    consumeError(Info.takeError());
-}
-
 Error LLVMOutputStyle::dumpStreamSummary() {
   if (!opts::raw::DumpStreamSummary)
     return Error::success();
 
-  discoverStreamPurposes();
+  if (StreamPurposes.empty())
+    discoverStreamPurposes(File, StreamPurposes);
 
   uint32_t StreamCount = File.getNumStreams();
 
   ListScope L(P, "Streams");
   for (uint16_t StreamIdx = 0; StreamIdx < StreamCount; ++StreamIdx) {
     std::string Label("Stream ");
     Label += to_string(StreamIdx);
 
Error LLVMOutputStyle::dumpBlockRanges() {
 
   return Error::success();
 }
 
 Error LLVMOutputStyle::dumpStreamBytes() {
   if (opts::raw::DumpStreamData.empty())
     return Error::success();
 
-  discoverStreamPurposes();
+  if (StreamPurposes.empty())
+    discoverStreamPurposes(File, StreamPurposes);
 
   DictScope D(P, "Stream Data");
   for (uint32_t SI : opts::raw::DumpStreamData) {
     if (SI >= File.getNumStreams())
       return make_error<RawError>(raw_error_code::no_stream);
 
     auto S = MappedBlockStream::createIndexedStream(File.getMsfLayout(),
                                                     File.getMsfBuffer(), SI);

llvm/trunk/tools/llvm-pdbdump/StreamUtil.h

+//===- Streamutil.h - PDB stream utilities ----------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_TOOLS_LLVMPDBDUMP_STREAMUTIL_H
+#define LLVM_TOOLS_LLVMPDBDUMP_STREAMUTIL_H
+
+#include "llvm/ADT/SmallVector.h"
+
+#include <string>
+
+namespace llvm {
+namespace pdb {
+class PDBFile;
+void discoverStreamPurposes(PDBFile &File,
+                            SmallVectorImpl<std::string> &Purposes);
+}
+}
+
+#endif

llvm/trunk/tools/llvm-pdbdump/StreamUtil.cpp

+//===- StreamUtil.cpp - PDB stream utilities --------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "StreamUtil.h"
+
+#include "llvm/ADT/DenseMap.h"
+#include "llvm/ADT/DenseMapInfo.h"
+#include "llvm/DebugInfo/PDB/Native/DbiStream.h"
+#include "llvm/DebugInfo/PDB/Native/InfoStream.h"
+#include "llvm/DebugInfo/PDB/Native/ModInfo.h"
+#include "llvm/DebugInfo/PDB/Native/PDBFile.h"
+#include "llvm/DebugInfo/PDB/Native/TpiStream.h"
+
+namespace llvm {
+namespace pdb {
+void discoverStreamPurposes(PDBFile &File,
+                            SmallVectorImpl<std::string> &Purposes) {
+
+  // It's OK if we fail to load some of these streams, we still attempt to print
+  // what we can.
+  auto Dbi = File.getPDBDbiStream();
+  auto Tpi = File.getPDBTpiStream();
+  auto Ipi = File.getPDBIpiStream();
+  auto Info = File.getPDBInfoStream();
+
+  uint32_t StreamCount = File.getNumStreams();
+  DenseMap<uint16_t, const ModuleInfoEx *> ModStreams;
+  DenseMap<uint16_t, std::string> NamedStreams;
+
+  if (Dbi) {
+    for (auto &ModI : Dbi->modules()) {
+      uint16_t SN = ModI.Info.getModuleStreamIndex();
+      if (SN != kInvalidStreamIndex)
+        ModStreams[SN] = &ModI;
+    }
+  }
+  if (Info) {
+    for (auto &NSE : Info->named_streams()) {
+      if (NSE.second != kInvalidStreamIndex)
+        NamedStreams[NSE.second] = NSE.first();
+    }
+  }
+
+  Purposes.resize(StreamCount);
+  for (uint16_t StreamIdx = 0; StreamIdx < StreamCount; ++StreamIdx) {
+    std::string Value;
+    if (StreamIdx == OldMSFDirectory)
+      Value = "Old MSF Directory";
+    else if (StreamIdx == StreamPDB)
+      Value = "PDB Stream";
+    else if (StreamIdx == StreamDBI)
+      Value = "DBI Stream";
+    else if (StreamIdx == StreamTPI)
+      Value = "TPI Stream";
+    else if (StreamIdx == StreamIPI)
+      Value = "IPI Stream";
+    else if (Dbi && StreamIdx == Dbi->getGlobalSymbolStreamIndex())
+      Value = "Global Symbol Hash";
+    else if (Dbi && StreamIdx == Dbi->getPublicSymbolStreamIndex())
+      Value = "Public Symbol Hash";
+    else if (Dbi && StreamIdx == Dbi->getSymRecordStreamIndex())
+      Value = "Public Symbol Records";
+    else if (Tpi && StreamIdx == Tpi->getTypeHashStreamIndex())
+      Value = "TPI Hash";
+    else if (Tpi && StreamIdx == Tpi->getTypeHashStreamAuxIndex())
+      Value = "TPI Aux Hash";
+    else if (Ipi && StreamIdx == Ipi->getTypeHashStreamIndex())
+      Value = "IPI Hash";
+    else if (Ipi && StreamIdx == Ipi->getTypeHashStreamAuxIndex())
+      Value = "IPI Aux Hash";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Exception))
+      Value = "Exception Data";
+    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Fixup))
+      Value = "Fixup Data";
+    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::FPO))
+      Value = "FPO Data";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::NewFPO))
+      Value = "New FPO Data";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::OmapFromSrc))
+      Value = "Omap From Source Data";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::OmapToSrc))
+      Value = "Omap To Source Data";
+    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Pdata))
+      Value = "Pdata";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::SectionHdr))
+      Value = "Section Header Data";
+    else if (Dbi &&
+             StreamIdx ==
+                 Dbi->getDebugStreamIndex(DbgHeaderType::SectionHdrOrig))
+      Value = "Section Header Original Data";
+    else if (Dbi &&
+             StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::TokenRidMap))
+      Value = "Token Rid Data";
+    else if (Dbi && StreamIdx == Dbi->getDebugStreamIndex(DbgHeaderType::Xdata))
+      Value = "Xdata";
+    else {
+      auto ModIter = ModStreams.find(StreamIdx);
+      auto NSIter = NamedStreams.find(StreamIdx);
+      if (ModIter != ModStreams.end()) {
+        Value = "Module \"";
+        Value += ModIter->second->Info.getModuleName().str();
+        Value += "\"";
+      } else if (NSIter != NamedStreams.end()) {
+        Value = "Named Stream \"";
+        Value += NSIter->second;
+        Value += "\"";
+      } else {
+        Value = "???";
+      }
+    }
+    Purposes[StreamIdx] = Value;
+  }
+
+  // Consume errors from missing streams.
+  if (!Dbi)
+    consumeError(Dbi.takeError());
+  if (!Tpi)
+    consumeError(Tpi.takeError());
+  if (!Ipi)
+    consumeError(Ipi.takeError());
+  if (!Info)
+    consumeError(Info.takeError());
+}
+}
+}

llvm/trunk/tools/llvm-pdbdump/llvm-pdbdump.h

 extern llvm::cl::opt<bool> DumpLineInfo;
 extern llvm::cl::opt<bool> DumpSectionMap;
 extern llvm::cl::opt<bool> DumpSymRecordBytes;
 extern llvm::cl::opt<bool> DumpSectionHeaders;
 extern llvm::cl::opt<bool> DumpFpo;
 extern llvm::cl::opt<bool> DumpStringTable;
 }
 
+namespace diff {
+extern llvm::cl::opt<bool> Pedantic;
+}
+
 namespace pdb2yaml {
 extern llvm::cl::opt<bool> NoFileHeaders;
 extern llvm::cl::opt<bool> StreamMetadata;
 extern llvm::cl::opt<bool> StreamDirectory;
 extern llvm::cl::opt<bool> StringTable;
 extern llvm::cl::opt<bool> PdbStream;
 extern llvm::cl::opt<bool> DbiStream;
 extern llvm::cl::opt<bool> DbiModuleInfo;

llvm/trunk/tools/llvm-pdbdump/llvm-pdbdump.cpp

 //===- llvm-pdbdump.cpp - Dump debug info from a PDB file -------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
-// Dumps debug information present in PDB files.  This utility makes use of
-// the Microsoft Windows SDK, so will not compile or run on non-Windows
-// platforms.
+// Dumps debug information present in PDB files.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm-pdbdump.h"
 
 #include "Analyze.h"
+#include "Diff.h"
 #include "LLVMOutputStyle.h"
 #include "LinePrinter.h"
 #include "OutputStyle.h"
 #include "PrettyCompilandDumper.h"
 #include "PrettyExternalSymbolDumper.h"
 #include "PrettyFunctionDumper.h"
 #include "PrettyTypeDumper.h"
 #include "PrettyVariableDumper.h"

 using namespace llvm::pdb;
 
 namespace opts {
 
 cl::SubCommand RawSubcommand("raw", "Dump raw structure of the PDB file");
 cl::SubCommand
     PrettySubcommand("pretty",
                      "Dump semantic information about types and symbols");
+
+cl::SubCommand DiffSubcommand("diff", "Diff the contents of 2 PDB files");
+
 cl::SubCommand
     YamlToPdbSubcommand("yaml2pdb",
                         "Generate a PDB file from a YAML description");
 cl::SubCommand
     PdbToYamlSubcommand("pdb2yaml",
                         "Generate a detailed YAML description of a PDB File");
 
 cl::SubCommand

 cl::opt<bool> NoClassDefs("no-class-definitions",
                           cl::desc("Don't display full class definitions"),
                           cl::cat(FilterCategory), cl::sub(PrettySubcommand));
 cl::opt<bool> NoEnumDefs("no-enum-definitions",
                          cl::desc("Don't display full enum definitions"),
                          cl::cat(FilterCategory), cl::sub(PrettySubcommand));
 }
 
+namespace diff {
+cl::opt<bool> Pedantic("pedantic",
+                       cl::desc("Finds all differences (even structural ones "
+                                "that produce otherwise identical PDBs)"),
+                       cl::sub(DiffSubcommand));
+
+cl::list<std::string> InputFilenames(cl::Positional,
+                                     cl::desc("<first> <second>"),
+                                     cl::OneOrMore, cl::sub(DiffSubcommand));
+}
+
 namespace raw {
 
 cl::OptionCategory MsfOptions("MSF Container Options");
 cl::OptionCategory TypeOptions("Type Record Options");
 cl::OptionCategory FileOptions("Module & File Options");
 cl::OptionCategory SymbolOptions("Symbol Options");
 cl::OptionCategory MiscOptions("Miscellaneous Options");
 
if (YamlObj.IpiStream.hasValue()) {
     IpiBuilder.setVersionHeader(YamlObj.IpiStream->Version);
     for (const auto &R : YamlObj.IpiStream->Records)
       IpiBuilder.addTypeRecord(R.Record);
   }
 
   ExitOnErr(Builder.commit(opts::yaml2pdb::YamlPdbOutputFile));
 }
 
+static PDBFile &loadPDB(StringRef Path, std::unique_ptr<IPDBSession> &Session) {
+  ExitOnErr(loadDataForPDB(PDB_ReaderType::Native, Path, Session));
+
+  NativeSession *NS = static_cast<NativeSession *>(Session.get());
+  return NS->getPDBFile();
+}
+
 static void pdb2Yaml(StringRef Path) {
   std::unique_ptr<IPDBSession> Session;
-  ExitOnErr(loadDataForPDB(PDB_ReaderType::Native, Path, Session));
+  auto &File = loadPDB(Path, Session);
 
-  NativeSession *RS = static_cast<NativeSession *>(Session.get());
-  PDBFile &File = RS->getPDBFile();
   auto O = llvm::make_unique<YAMLOutputStyle>(File);
   O = llvm::make_unique<YAMLOutputStyle>(File);
 
   ExitOnErr(O->dump());
 }
 
 static void dumpRaw(StringRef Path) {
   std::unique_ptr<IPDBSession> Session;
-  ExitOnErr(loadDataForPDB(PDB_ReaderType::Native, Path, Session));
+  auto &File = loadPDB(Path, Session);
 
-  NativeSession *RS = static_cast<NativeSession *>(Session.get());
-  PDBFile &File = RS->getPDBFile();
   auto O = llvm::make_unique<LLVMOutputStyle>(File);
 
   ExitOnErr(O->dump());
 }
 
 static void dumpAnalysis(StringRef Path) {
   std::unique_ptr<IPDBSession> Session;
-  ExitOnErr(loadDataForPDB(PDB_ReaderType::Native, Path, Session));
-
-  NativeSession *NS = static_cast<NativeSession *>(Session.get());
-  PDBFile &File = NS->getPDBFile();
+  auto &File = loadPDB(Path, Session);
   auto O = llvm::make_unique<AnalysisStyle>(File);
 
   ExitOnErr(O->dump());
 }
 
+static void diff(StringRef Path1, StringRef Path2) {
+  std::unique_ptr<IPDBSession> Session1;
+  std::unique_ptr<IPDBSession> Session2;
+
+  auto &File1 = loadPDB(Path1, Session1);
+  auto &File2 = loadPDB(Path2, Session2);
+
+  auto O = llvm::make_unique<DiffStyle>(File1, File2);
+
+  ExitOnErr(O->dump());
+}
+
 static void dumpPretty(StringRef Path) {
   std::unique_ptr<IPDBSession> Session;
 
   ExitOnErr(loadDataForPDB(PDB_ReaderType::DIA, Path, Session));
 
   if (opts::pretty::LoadAddress)
     Session->setLoadAddress(opts::pretty::LoadAddress);
 
if (opts::pretty::ExcludeSystemLibraries) {
       opts::pretty::ExcludeCompilands.push_back(
           "d:\\\\th.obj.x86fre\\\\minkernel");
     }
     std::for_each(opts::pretty::InputFilenames.begin(),
                   opts::pretty::InputFilenames.end(), dumpPretty);
   } else if (opts::RawSubcommand) {
     std::for_each(opts::raw::InputFilenames.begin(),
                   opts::raw::InputFilenames.end(), dumpRaw);
+  } else if (opts::DiffSubcommand) {
+    if (opts::diff::InputFilenames.size() != 2) {
+      errs() << "diff subcommand expects exactly 2 arguments.\n";
+      exit(1);
+    }
+    diff(opts::diff::InputFilenames[0], opts::diff::InputFilenames[1]);
   }
 
   outs().flush();
   return 0;
 }