[PDB] Parse module information from the DBI stream

[zturner]

Summary

With this patch we extract basic module information from the PDB file. A module is what is referred to in the DIA SDK and DebugInfoPDB as a "compiland". So we are essentially enumerating all compilands and dumping some of their metadata. In particular we get the compiland name and contributing object files, as well as various indices that will point at more detailed symbol and type information to be used in subsequent patches.

With this, we are close to being able to implement PDBSymbolExe and PDBSymbolCompiland in terms of the raw interface.

[zturner]

Fixed the post-increment operator on the stream iterator.

[ruiu]

include/llvm/DebugInfo/PDB/Raw/ModInfo.h

59

Isn't it a bit confusing that this Bytes is actually bytes but Bytes in ModInfo is a ModInfoBytes object.

62

Can you use iterator_range<ModInfoStreamIterator> instead of defining a new class?

lib/DebugInfo/PDB/Raw/ModInfo.cpp

49

Do you mean ulittle32_t by long?

[zturner]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

51

I don't think so, but honestly I don't know. We're reading this value from a file, and it doesn't make sense for a pointer to be in a file, because it would be meaningless in the address space of the process reading the file. So far in the limited experiments I've run, the value of this pointer (and the other pointer earlier in the class definition) are always 0 when reading them in. My best guess is that they use these as bookkeeping during the life of the PDB, but the value that goes to disk is unimportant. I guess it's something we will figure out in time as we study the format more closely.

[ruiu]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

51

Since you are not using this field at least at this moment, why don't you use uint32_t or ulittle32_t? I don't think this code will pass tests on LP64 machines because on that machine long* is 64-bit wide.

[zturner]

Fixed review issues.

[ruiu]

LGTM

[majnemer]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

54–57

Won't it be 64-bit in size on a 64-bit system?

65

Incrementing an iterator past end looks suspicious.

[zturner]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

54–57

Not as far as I can tell. They have 2 different structures that are almost the same, one using a uint32 and one using a pointer. The former is the file layout format, and the latter is an in-memory format. But since we aren't using these fields yet anyway, it doesn't make sense to add the additional complexity, as we may find a better less convoluted way to do the same thing, or we may end up just not needing those fields anyway.

65

In the general case I agree, but in this case we have two null terminated strings stored back to back in a variable length structure, so this seems like the most elegant way to express that. Otherwise it's going to end up just being equivalent code but more verbose, since I will have to save off the module name into a temporary, then call module.data() + module.length() + 1.

[ruiu]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

65

Yeah, I agree with Zach. This is where we intentionally overrun the end of the array to access stuff next to the string. This makes me wonder if C++ supports flexible array members.

[zturner]

Fixed alignment issue by adding 2 bytes of padding. When I was using a pointer it was getting 4-byte aligned, but when I changed to a ulittle32_t it was getting only 2-byte aligned, messing up the rest of the structure. This should be correct now. Alternatively I could have used an aligned_ulittle32_t, but the name is so long it made the comment formatting really ugly.

[ruiu]

I have no further comments. David?

[zturner]

Also fixed a place where I was accidentally using raw integer types instead of endian types in the section contributions.

[zturner]

Gah, is it better to use the aligned types, or to keep manually adding padding? The latest update also breaks the definition of SCBytes because:

uint16_t x;
uint32_t y;

is different from

ulittle16_t x;
ulittle32_t y;

Should I just switch to using aligned types everywhere, keeping in mind that the original definition is:

unsigned short;
unsigned int;

and assumes MSVC.

[ruiu]

I'd personally prefer adding padding field because it is explicit and it is I believe more common than implicit padding when describing data structure in file.

By the way, we don't care about endianness of padding, so I think

uint8_t Pad1[4];

is more common than

ulittle16_t Pad1;

[zturner]

Fix padding issues everywhere.

David, are you satisfied with the explanation about incrementing past the end of the iterator earlier, or do you have a suggestion for a better way to accomplish the same thing?

include/llvm/DebugInfo/PDB/Raw/ModInfo.h

+//===- ModInfo.h - PDB module information -----------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_PDB_RAW_MODINFO_H
+#define LLVM_DEBUGINFO_PDB_RAW_MODINFO_H
+
+#include "llvm/ADT/StringRef.h"
+
+#include <stdint.h>
+
+namespace llvm {
+class PDBFile;
+
+class ModInfo {
+private:
+  friend class ModInfoStreamIterator;
+  struct ModInfoBytes;
+
+public:
+  ModInfo(const uint8_t *InfoBytes);
+  ~ModInfo();
+
+  bool hasECInfo() const;
+  uint16_t getTypeServerIndex() const;
+  uint16_t getModuleStreamIndex() const;
+  uint32_t getSymbolDebugInfoByteSize() const;
+  uint32_t getLineInfoByteSize() const;
+  uint32_t getC13LineInfoByteSize() const;
+  uint32_t getNumberOfFiles() const;
+  uint32_t getSourceFileNameIndex() const;
+  uint32_t getPdbFilePathNameIndex() const;
+
+  llvm::StringRef getModuleName() const;
+  llvm::StringRef getObjFileName() const;
+
+private:
+  const ModInfoBytes *Bytes;
+};
+
+class ModInfoStreamIterator {
+public:
+  ModInfoStreamIterator(const uint8_t *Stream);
+  ModInfoStreamIterator(const ModInfoStreamIterator &Other);
+
+  ModInfo operator*();
+  ModInfoStreamIterator &operator++();
+  ModInfoStreamIterator operator++(int);
+  bool operator==(const ModInfoStreamIterator &Other);
+  bool operator!=(const ModInfoStreamIterator &Other);
+  ModInfoStreamIterator &operator=(const ModInfoStreamIterator &Other);
+
+private:
+  const uint8_t *Bytes;
+};
+
+class ModInfoRange {
+public:
+  ModInfoRange(const uint8_t *Stream, uint32_t ByteLength)
+      : Begin(Stream), Size(ByteLength) {}
+
+  ModInfoStreamIterator begin() { return ModInfoStreamIterator(Begin); }
+  ModInfoStreamIterator end() { return ModInfoStreamIterator(Begin + Size); }
+
+private:
+  const uint8_t *Begin;
+  uint32_t Size;
+};
+}
+
+#endif

include/llvm/DebugInfo/PDB/Raw/PDBDbiStream.h

 #define LLVM_DEBUGINFO_PDB_RAW_PDBDBISTREAM_H
 
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBRawConstants.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBStream.h"
 #include "llvm/Support/Endian.h"

 
   PDB_Machine getMachineType() const;
 
+  ModInfoRange modules() const;
+
 private:
   PDBFile &Pdb;
   PDBStream Stream;
+
+  std::vector<uint8_t> ModInfoSubstream;
   std::unique_ptr<HeaderInfo> Header;
 };
 }

lib/DebugInfo/PDB/CMakeLists.txt

 endif()
 
 add_pdb_impl_folder(Raw
+  Raw/ModInfo.cpp
   Raw/PDBFile.cpp
   Raw/PDBDbiStream.cpp
   Raw/PDBInfoStream.cpp

lib/DebugInfo/PDB/Raw/ModInfo.cpp

+//===- ModInfo.cpp - PDB module information -------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
+#include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
+#include "llvm/Support/Endian.h"
+
+using namespace llvm;
+using namespace llvm::support;
+
+namespace {
+struct SCBytes {
+  ulittle16_t Section;
+  little32_t Offset;
+  little32_t Size;
+  uint32_t Characteristics;
+  ulittle16_t ModuleIndex;
+  uint32_t DataCrc;
+  uint32_t RelocCrc;
+};
+
+// struct Flags {
+//  uint16_t fWritten : 1;   // True if ModInfo is dirty
+//  uint16_t fECEnabled : 1; // Is EC symbolic info present?  (What is EC?)
+//  uint16_t unused : 6;     // Reserved
+//  uint16_t iTSM : 8;       // Type Server Index for this module
+//};
+const uint16_t HasECFlagMask = 0x2;
+
+const uint16_t TypeServerIndexMask = 0xFF00;
+const uint16_t TypeServerIndexShift = 8;
+}
+
+struct ModInfo::ModInfoBytes {
+  void *Mod;                 // Currently opened module
+  SCBytes SC;                // First section contribution of this module.
+  ulittle16_t Flags;         // See Flags definition.
+  ulittle16_t ModDiStream;   // Stream Number of module debug info
+  ulittle32_t SymBytes;      // Size of local symbol debug info in above stream
+  ulittle32_t LineBytes;     // Size of line number debug info in above stream
+  ulittle32_t C13Bytes;      // Size of C13 line number info in above stream
+  ulittle16_t NumFiles;      // Number of files contributing to this module
+  long *FileNameOffsets;     // array of [0..NumFiles) DBI name buffer offsets
+  ulittle32_t SrcFileNameNI; // Name Index for src file name
+  ulittle32_t PdbFilePathNI; // Name Index for path to compiler PDB
+  char VarInfo[1];           // Module name followed by Obj File Name
+
+  StringRef getModuleName() const { return StringRef(VarInfo); }
+
+  StringRef getObjectFileName() const {
+    return StringRef(getModuleName().end() + 1);
+  }
+};
+
+ModInfo::ModInfo(const uint8_t *InfoBytes)
+    : Bytes(reinterpret_cast<const ModInfoBytes *>(InfoBytes)) {}
+
+ModInfo::~ModInfo() {}
+
+bool ModInfo::hasECInfo() const { return (Bytes->Flags & HasECFlagMask) != 0; }
+
+uint16_t ModInfo::getTypeServerIndex() const {
+  return (Bytes->Flags & TypeServerIndexMask) >> TypeServerIndexShift;
+}
+
+uint16_t ModInfo::getModuleStreamIndex() const { return Bytes->ModDiStream; }
+
+uint32_t ModInfo::getSymbolDebugInfoByteSize() const { return Bytes->SymBytes; }
+
+uint32_t ModInfo::getLineInfoByteSize() const { return Bytes->LineBytes; }
+
+uint32_t ModInfo::getC13LineInfoByteSize() const { return Bytes->C13Bytes; }
+
+uint32_t ModInfo::getNumberOfFiles() const { return Bytes->NumFiles; }
+
+uint32_t ModInfo::getSourceFileNameIndex() const {
+  return Bytes->SrcFileNameNI;
+}
+
+uint32_t ModInfo::getPdbFilePathNameIndex() const {
+  return Bytes->PdbFilePathNI;
+}
+
+llvm::StringRef ModInfo::getModuleName() const {
+  return Bytes->getModuleName();
+}
+
+llvm::StringRef ModInfo::getObjFileName() const {
+  return Bytes->getObjectFileName();
+}
+
+ModInfoStreamIterator::ModInfoStreamIterator(const uint8_t *Stream)
+    : Bytes(Stream) {}
+
+ModInfoStreamIterator::ModInfoStreamIterator(const ModInfoStreamIterator &Other)
+    : Bytes(Other.Bytes) {}
+
+ModInfo ModInfoStreamIterator::operator*() { return ModInfo(Bytes); }
+
+ModInfoStreamIterator &ModInfoStreamIterator::operator++() {
+  StringRef Obj = ModInfo(Bytes).getObjFileName();
+  Bytes = Obj.bytes_end() + 1;
+  Bytes = reinterpret_cast<const uint8_t *>(llvm::alignAddr(Bytes, 4));
+
+  return *this;
+}
+
+ModInfoStreamIterator ModInfoStreamIterator::operator++(int) {
+  StringRef Obj = ModInfo(Bytes).getObjFileName();
+  const uint8_t *Next = Obj.bytes_end() + 1;
+  Next = reinterpret_cast<const uint8_t *>(llvm::alignAddr(Next, 4));
+
+  return ModInfoStreamIterator(Next);
+}
+
+bool ModInfoStreamIterator::operator==(const ModInfoStreamIterator &Other) {
+  return Bytes == Other.Bytes;
+}
+
+bool ModInfoStreamIterator::operator!=(const ModInfoStreamIterator &Other) {
+  return !(*this == Other);
+}
+
+ModInfoStreamIterator &ModInfoStreamIterator::
+operator=(const ModInfoStreamIterator &Other) {
+  Bytes = Other.Bytes;
+  return *this;
+}

lib/DebugInfo/PDB/Raw/PDBDbiStream.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/PDB/Raw/PDBDbiStream.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBInfoStream.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBRawConstants.h"

   if (Header->VersionSignature != -1)
     return std::make_error_code(std::errc::illegal_byte_sequence);
 
-  // Prior to VC50 an old style header was used.  We don't support this.
-  if (Header->VersionHeader < PdbDbiV50)
+  // Require at least version 7, which should be present in all PDBs
+  // produced in the last decade and allows us to avoid having to
+  // special case all kinds of complicated arcane formats.
+  if (Header->VersionHeader < PdbDbiV70)
     return std::make_error_code(std::errc::not_supported);
 
   if (Header->Age != Pdb.getPDBInfoStream().getAge())

           Header->OptionalDbgHdrSize + Header->ECSubstreamSize)
     return std::make_error_code(std::errc::illegal_byte_sequence);
 
+  if (Header->ModiSubstreamSize % sizeof(uint32_t) != 0)
+    return std::make_error_code(std::errc::illegal_byte_sequence);
+
+  ModInfoSubstream.resize(Header->ModiSubstreamSize);
+  if (auto EC =
+          Stream.readBytes(&ModInfoSubstream[0], Header->ModiSubstreamSize))
+    return EC;
+
   return std::error_code();
 }
 

   uint16_t Machine = Header->MachineType;
   return static_cast<PDB_Machine>(Machine);
 }
+
+ModInfoRange PDBDbiStream::modules() const {
+  return ModInfoRange(&ModInfoSubstream[0], Header->ModiSubstreamSize);
+}

test/DebugInfo/PDB/pdbdump-headers.test

 ; CHECK-NEXT: Number of Symbols: 8
 ; CHECK-NEXT: Toolchain Version: 12.0
 ; CHECK-NEXT: mspdb120.dll version: 12.0.31101
+
+; CHECK:      Modules:
+; CHECK-NEXT: d:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj
+; CHECK-NEXT:     Debug Stream Index: 12
+; CHECK-NEXT:     Object File: d:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj
+; CHECK-NEXT:     Num Files: 1
+; CHECK-NEXT:     Source File Name Idx: 0
+; CHECK-NEXT:     Pdb File Name Idx: 0
+; CHECK-NEXT:     Line Info Byte Size: 0
+; CHECK-NEXT:     C13 Line Info Byte Size: 88
+; CHECK-NEXT:     Symbol Byte Size: 208
+; CHECK-NEXT:     Type Server Index: 0
+; CHECK-NEXT:     Has EC Info: 0
+; CHECK-NEXT: * Linker *
+; CHECK-NEXT:     Debug Stream Index: 14
+; CHECK-NEXT:     Object File:
+; CHECK-NEXT:     Num Files: 0
+; CHECK-NEXT:     Source File Name Idx: 0
+; CHECK-NEXT:     Pdb File Name Idx: 1
+; CHECK-NEXT:     Line Info Byte Size: 0
+; CHECK-NEXT:     C13 Line Info Byte Size: 0
+; CHECK-NEXT:     Symbol Byte Size: 516
+; CHECK-NEXT:     Type Server Index: 0
+; CHECK-NEXT:     Has EC Info: 0

tools/llvm-pdbdump/llvm-pdbdump.cpp

 #include "llvm/DebugInfo/PDB/PDBSymbolExe.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolFunc.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolThunk.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBDbiStream.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBInfoStream.h"

   outs() << "Toolchain Version: " << Major << "." << Minor << '\n';
   outs() << "mspdb" << Major << Minor << ".dll version: " << Major << "."
          << Minor << "." << DbiStream.getPdbDllVersion() << '\n';
+
+  outs() << "Modules: \n";
+  for (auto Modi : DbiStream.modules()) {
+    outs() << Modi.getModuleName() << '\n';
+    outs().indent(4) << "Debug Stream Index: " << Modi.getModuleStreamIndex()
+                     << '\n';
+    outs().indent(4) << "Object File: " << Modi.getObjFileName() << '\n';
+    outs().indent(4) << "Num Files: " << Modi.getNumberOfFiles() << '\n';
+    outs().indent(4) << "Source File Name Idx: "
+                     << Modi.getSourceFileNameIndex() << '\n';
+    outs().indent(4) << "Pdb File Name Idx: " << Modi.getPdbFilePathNameIndex()
+                     << '\n';
+    outs().indent(4) << "Line Info Byte Size: " << Modi.getLineInfoByteSize()
+                     << '\n';
+    outs().indent(4) << "C13 Line Info Byte Size: "
+                     << Modi.getC13LineInfoByteSize() << '\n';
+    outs().indent(4) << "Symbol Byte Size: "
+                     << Modi.getSymbolDebugInfoByteSize() << '\n';
+    outs().indent(4) << "Type Server Index: " << Modi.getTypeServerIndex()
+                     << '\n';
+    outs().indent(4) << "Has EC Info: " << Modi.hasECInfo() << '\n';
+  }
 }
 
 static void dumpInput(StringRef Path) {