[PDB] Parse module information from the DBI stream

[zturner]

Summary

With this patch we extract basic module information from the PDB file. A module is what is referred to in the DIA SDK and DebugInfoPDB as a "compiland". So we are essentially enumerating all compilands and dumping some of their metadata. In particular we get the compiland name and contributing object files, as well as various indices that will point at more detailed symbol and type information to be used in subsequent patches.

With this, we are close to being able to implement PDBSymbolExe and PDBSymbolCompiland in terms of the raw interface.

[zturner]

Fixed the post-increment operator on the stream iterator.

[ruiu]

include/llvm/DebugInfo/PDB/Raw/ModInfo.h

60

Isn't it a bit confusing that this Bytes is actually bytes but Bytes in ModInfo is a ModInfoBytes object.

63

Can you use iterator_range<ModInfoStreamIterator> instead of defining a new class?

lib/DebugInfo/PDB/Raw/ModInfo.cpp

50

Do you mean ulittle32_t by long?

[zturner]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

51

I don't think so, but honestly I don't know. We're reading this value from a file, and it doesn't make sense for a pointer to be in a file, because it would be meaningless in the address space of the process reading the file. So far in the limited experiments I've run, the value of this pointer (and the other pointer earlier in the class definition) are always 0 when reading them in. My best guess is that they use these as bookkeeping during the life of the PDB, but the value that goes to disk is unimportant. I guess it's something we will figure out in time as we study the format more closely.

[ruiu]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

51

Since you are not using this field at least at this moment, why don't you use uint32_t or ulittle32_t? I don't think this code will pass tests on LP64 machines because on that machine long* is 64-bit wide.

[zturner]

Fixed review issues.

[ruiu]

LGTM

[majnemer]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

54–57

Won't it be 64-bit in size on a 64-bit system?

65

Incrementing an iterator past end looks suspicious.

[zturner]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

54–57

Not as far as I can tell. They have 2 different structures that are almost the same, one using a uint32 and one using a pointer. The former is the file layout format, and the latter is an in-memory format. But since we aren't using these fields yet anyway, it doesn't make sense to add the additional complexity, as we may find a better less convoluted way to do the same thing, or we may end up just not needing those fields anyway.

65

In the general case I agree, but in this case we have two null terminated strings stored back to back in a variable length structure, so this seems like the most elegant way to express that. Otherwise it's going to end up just being equivalent code but more verbose, since I will have to save off the module name into a temporary, then call module.data() + module.length() + 1.

[ruiu]

lib/DebugInfo/PDB/Raw/ModInfo.cpp

65

Yeah, I agree with Zach. This is where we intentionally overrun the end of the array to access stuff next to the string. This makes me wonder if C++ supports flexible array members.

[zturner]

Fixed alignment issue by adding 2 bytes of padding. When I was using a pointer it was getting 4-byte aligned, but when I changed to a ulittle32_t it was getting only 2-byte aligned, messing up the rest of the structure. This should be correct now. Alternatively I could have used an aligned_ulittle32_t, but the name is so long it made the comment formatting really ugly.

[ruiu]

I have no further comments. David?

[zturner]

Also fixed a place where I was accidentally using raw integer types instead of endian types in the section contributions.

[zturner]

Gah, is it better to use the aligned types, or to keep manually adding padding? The latest update also breaks the definition of SCBytes because:

uint16_t x;
uint32_t y;

is different from

ulittle16_t x;
ulittle32_t y;

Should I just switch to using aligned types everywhere, keeping in mind that the original definition is:

unsigned short;
unsigned int;

and assumes MSVC.

[ruiu]

I'd personally prefer adding padding field because it is explicit and it is I believe more common than implicit padding when describing data structure in file.

By the way, we don't care about endianness of padding, so I think

uint8_t Pad1[4];

is more common than

ulittle16_t Pad1;

[zturner]

Fix padding issues everywhere.

David, are you satisfied with the explanation about incrementing past the end of the iterator earlier, or do you have a suggestion for a better way to accomplish the same thing?

include/llvm/DebugInfo/PDB/Raw/ModInfo.h

+//===- ModInfo.h - PDB module information -----------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_PDB_RAW_MODINFO_H
+#define LLVM_DEBUGINFO_PDB_RAW_MODINFO_H
+
+#include "llvm/ADT/StringRef.h"
+
+#include <stdint.h>
+
+namespace llvm {
+class PDBFile;
+
+class ModInfo {
+private:
+  struct FileLayout;
+
+public:
+  ModInfo(const uint8_t *Bytes);
+  ~ModInfo();
+
+  bool hasECInfo() const;
+  uint16_t getTypeServerIndex() const;
+  uint16_t getModuleStreamIndex() const;
+  uint32_t getSymbolDebugInfoByteSize() const;
+  uint32_t getLineInfoByteSize() const;
+  uint32_t getC13LineInfoByteSize() const;
+  uint32_t getNumberOfFiles() const;
+  uint32_t getSourceFileNameIndex() const;
+  uint32_t getPdbFilePathNameIndex() const;
+
+  llvm::StringRef getModuleName() const;
+  llvm::StringRef getObjFileName() const;
+
+private:
+  const FileLayout *Layout;
+};
+
+class ModInfoIterator {
+public:
+  ModInfoIterator(const uint8_t *Stream);
+  ModInfoIterator(const ModInfoIterator &Other);
+
+  ModInfo operator*();
+  ModInfoIterator &operator++();
+  ModInfoIterator operator++(int);
+  bool operator==(const ModInfoIterator &Other);
+  bool operator!=(const ModInfoIterator &Other);
+  ModInfoIterator &operator=(const ModInfoIterator &Other);
+
+private:
+  const uint8_t *Bytes;
+};
+}
+
+#endif

include/llvm/DebugInfo/PDB/Raw/PDBDbiStream.h

 #define LLVM_DEBUGINFO_PDB_RAW_PDBDBISTREAM_H
 
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBRawConstants.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBStream.h"
 #include "llvm/Support/Endian.h"

 
   PDB_Machine getMachineType() const;
 
+  llvm::iterator_range<ModInfoIterator> modules() const;
+
 private:
   PDBFile &Pdb;
   PDBStream Stream;
+
+  std::vector<uint8_t> ModInfoSubstream;
   std::unique_ptr<HeaderInfo> Header;
 };
 }

lib/DebugInfo/PDB/CMakeLists.txt

 endif()
 
 add_pdb_impl_folder(Raw
+  Raw/ModInfo.cpp
   Raw/PDBFile.cpp
   Raw/PDBDbiStream.cpp
   Raw/PDBInfoStream.cpp

lib/DebugInfo/PDB/Raw/ModInfo.cpp

+//===- ModInfo.cpp - PDB module information -------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
+#include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
+#include "llvm/Support/Endian.h"
+
+using namespace llvm;
+using namespace llvm::support;
+
+namespace {
+struct SCBytes {
+  ulittle16_t Section;
+  char Padding1[2];
+  little32_t Offset;
+  little32_t Size;
+  ulittle32_t Characteristics;
+  ulittle16_t ModuleIndex;
+  char Padding2[2];
+  ulittle32_t DataCrc;
+  ulittle32_t RelocCrc;
+};
+
+// struct Flags {
+//  uint16_t fWritten : 1;   // True if ModInfo is dirty
+//  uint16_t fECEnabled : 1; // Is EC symbolic info present?  (What is EC?)
+//  uint16_t unused : 6;     // Reserved
+//  uint16_t iTSM : 8;       // Type Server Index for this module
+//};
+const uint16_t HasECFlagMask = 0x2;
+
+const uint16_t TypeServerIndexMask = 0xFF00;
+const uint16_t TypeServerIndexShift = 8;
+}
+
+struct ModInfo::FileLayout {
+  ulittle32_t Mod;           // Currently opened module.  This field is a
+                             // pointer in the reference implementation, but
+                             // that won't work on 64-bit systems, and anyway
+                             // it doesn't make sense to read a pointer from a
+                             // file.  For now it is unused, so just ignore it.
+  SCBytes SC;                // First section contribution of this module.
+  ulittle16_t Flags;         // See Flags definition.
+  ulittle16_t ModDiStream;   // Stream Number of module debug info
+  ulittle32_t SymBytes;      // Size of local symbol debug info in above stream
+  ulittle32_t LineBytes;     // Size of line number debug info in above stream
+  ulittle32_t C13Bytes;      // Size of C13 line number info in above stream
+  ulittle16_t NumFiles;      // Number of files contributing to this module
+  char Padding1[2];          // Padding so the next field is 4-byte aligned.
+  ulittle32_t FileNameOffs;  // array of [0..NumFiles) DBI name buffer offsets.
+                             // This field is a pointer in the reference
+                             // implementation, but as with `Mod`, we ignore it
+                             // for now since it is unused.
+  ulittle32_t SrcFileNameNI; // Name Index for src file name
+  ulittle32_t PdbFilePathNI; // Name Index for path to compiler PDB
+  char VarInfo[1];           // Module name followed by Obj File Name
+
+  StringRef getModuleName() const { return StringRef(VarInfo); }
+
+  StringRef getObjectFileName() const {
+    return StringRef(getModuleName().end() + 1);
+  }
+};
+
+ModInfo::ModInfo(const uint8_t *Bytes)
+    : Layout(reinterpret_cast<const FileLayout *>(Bytes)) {}
+
+ModInfo::~ModInfo() {}
+
+bool ModInfo::hasECInfo() const { return (Layout->Flags & HasECFlagMask) != 0; }
+
+uint16_t ModInfo::getTypeServerIndex() const {
+  return (Layout->Flags & TypeServerIndexMask) >> TypeServerIndexShift;
+}
+
+uint16_t ModInfo::getModuleStreamIndex() const { return Layout->ModDiStream; }
+
+uint32_t ModInfo::getSymbolDebugInfoByteSize() const {
+  return Layout->SymBytes;
+}
+
+uint32_t ModInfo::getLineInfoByteSize() const { return Layout->LineBytes; }
+
+uint32_t ModInfo::getC13LineInfoByteSize() const { return Layout->C13Bytes; }
+
+uint32_t ModInfo::getNumberOfFiles() const { return Layout->NumFiles; }
+
+uint32_t ModInfo::getSourceFileNameIndex() const {
+  return Layout->SrcFileNameNI;
+}
+
+uint32_t ModInfo::getPdbFilePathNameIndex() const {
+  return Layout->PdbFilePathNI;
+}
+
+llvm::StringRef ModInfo::getModuleName() const {
+  return Layout->getModuleName();
+}
+
+llvm::StringRef ModInfo::getObjFileName() const {
+  return Layout->getObjectFileName();
+}
+
+ModInfoIterator::ModInfoIterator(const uint8_t *Stream) : Bytes(Stream) {}
+
+ModInfoIterator::ModInfoIterator(const ModInfoIterator &Other)
+    : Bytes(Other.Bytes) {}
+
+ModInfo ModInfoIterator::operator*() { return ModInfo(Bytes); }
+
+ModInfoIterator &ModInfoIterator::operator++() {
+  StringRef Obj = ModInfo(Bytes).getObjFileName();
+  Bytes = Obj.bytes_end() + 1;
+  Bytes = reinterpret_cast<const uint8_t *>(llvm::alignAddr(Bytes, 4));
+
+  return *this;
+}
+
+ModInfoIterator ModInfoIterator::operator++(int) {
+  ModInfoIterator Copy(*this);
+  ++(*this);
+  return Copy;
+}
+
+bool ModInfoIterator::operator==(const ModInfoIterator &Other) {
+  return Bytes == Other.Bytes;
+}
+
+bool ModInfoIterator::operator!=(const ModInfoIterator &Other) {
+  return !(*this == Other);
+}
+
+ModInfoIterator &ModInfoIterator::operator=(const ModInfoIterator &Other) {
+  Bytes = Other.Bytes;
+  return *this;
+}

lib/DebugInfo/PDB/Raw/PDBDbiStream.cpp

 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/PDB/Raw/PDBDbiStream.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBInfoStream.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBRawConstants.h"

   if (Header->VersionSignature != -1)
     return std::make_error_code(std::errc::illegal_byte_sequence);
 
-  // Prior to VC50 an old style header was used.  We don't support this.
-  if (Header->VersionHeader < PdbDbiV50)
+  // Require at least version 7, which should be present in all PDBs
+  // produced in the last decade and allows us to avoid having to
+  // special case all kinds of complicated arcane formats.
+  if (Header->VersionHeader < PdbDbiV70)
     return std::make_error_code(std::errc::not_supported);
 
   if (Header->Age != Pdb.getPDBInfoStream().getAge())

           Header->OptionalDbgHdrSize + Header->ECSubstreamSize)
     return std::make_error_code(std::errc::illegal_byte_sequence);
 
+  if (Header->ModiSubstreamSize % sizeof(uint32_t) != 0)
+    return std::make_error_code(std::errc::illegal_byte_sequence);
+
+  ModInfoSubstream.resize(Header->ModiSubstreamSize);
+  if (auto EC =
+          Stream.readBytes(&ModInfoSubstream[0], Header->ModiSubstreamSize))
+    return EC;
+
   return std::error_code();
 }
 

   uint16_t Machine = Header->MachineType;
   return static_cast<PDB_Machine>(Machine);
 }
+
+llvm::iterator_range<ModInfoIterator> PDBDbiStream::modules() const {
+  return llvm::make_range(ModInfoIterator(&ModInfoSubstream.front()),
+                          ModInfoIterator(&ModInfoSubstream.back() + 1));
+}

test/DebugInfo/PDB/pdbdump-headers.test

 ; CHECK-NEXT: Number of Symbols: 8
 ; CHECK-NEXT: Toolchain Version: 12.0
 ; CHECK-NEXT: mspdb120.dll version: 12.0.31101
+
+; CHECK:      Modules:
+; CHECK-NEXT: d:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj
+; CHECK-NEXT:     Debug Stream Index: 12
+; CHECK-NEXT:     Object File: d:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj
+; CHECK-NEXT:     Num Files: 1
+; CHECK-NEXT:     Source File Name Idx: 0
+; CHECK-NEXT:     Pdb File Name Idx: 0
+; CHECK-NEXT:     Line Info Byte Size: 0
+; CHECK-NEXT:     C13 Line Info Byte Size: 88
+; CHECK-NEXT:     Symbol Byte Size: 208
+; CHECK-NEXT:     Type Server Index: 0
+; CHECK-NEXT:     Has EC Info: 0
+; CHECK-NEXT: * Linker *
+; CHECK-NEXT:     Debug Stream Index: 14
+; CHECK-NEXT:     Object File:
+; CHECK-NEXT:     Num Files: 0
+; CHECK-NEXT:     Source File Name Idx: 0
+; CHECK-NEXT:     Pdb File Name Idx: 1
+; CHECK-NEXT:     Line Info Byte Size: 0
+; CHECK-NEXT:     C13 Line Info Byte Size: 0
+; CHECK-NEXT:     Symbol Byte Size: 516
+; CHECK-NEXT:     Type Server Index: 0
+; CHECK-NEXT:     Has EC Info: 0

tools/llvm-pdbdump/llvm-pdbdump.cpp

 #include "llvm/DebugInfo/PDB/PDBSymbolExe.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolFunc.h"
 #include "llvm/DebugInfo/PDB/PDBSymbolThunk.h"
+#include "llvm/DebugInfo/PDB/Raw/ModInfo.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBDbiStream.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Raw/PDBInfoStream.h"

   outs() << "Toolchain Version: " << Major << "." << Minor << '\n';
   outs() << "mspdb" << Major << Minor << ".dll version: " << Major << "."
          << Minor << "." << DbiStream.getPdbDllVersion() << '\n';
+
+  outs() << "Modules: \n";
+  for (auto Modi : DbiStream.modules()) {
+    outs() << Modi.getModuleName() << '\n';
+    outs().indent(4) << "Debug Stream Index: " << Modi.getModuleStreamIndex()
+                     << '\n';
+    outs().indent(4) << "Object File: " << Modi.getObjFileName() << '\n';
+    outs().indent(4) << "Num Files: " << Modi.getNumberOfFiles() << '\n';
+    outs().indent(4) << "Source File Name Idx: "
+                     << Modi.getSourceFileNameIndex() << '\n';
+    outs().indent(4) << "Pdb File Name Idx: " << Modi.getPdbFilePathNameIndex()
+                     << '\n';
+    outs().indent(4) << "Line Info Byte Size: " << Modi.getLineInfoByteSize()
+                     << '\n';
+    outs().indent(4) << "C13 Line Info Byte Size: "
+                     << Modi.getC13LineInfoByteSize() << '\n';
+    outs().indent(4) << "Symbol Byte Size: "
+                     << Modi.getSymbolDebugInfoByteSize() << '\n';
+    outs().indent(4) << "Type Server Index: " << Modi.getTypeServerIndex()
+                     << '\n';
+    outs().indent(4) << "Has EC Info: " << Modi.hasECInfo() << '\n';
+  }
 }
 
 static void dumpInput(StringRef Path) {