[PDB] Add documentation for the PDB DBI Stream.

[amccarth]

Good content. I have just a couple wording nits.

docs/PDB/DbiStream.rst

17

"containing" here is redundant and a tad confusing.

80

Consistency nit: Elsewhere, you've been using e.g. and i.e. without a comma.

87

Bit-field syntax is ambiguous because bit-fields can be allocated left-to-right or right-to-left (implementation defined). There are also implementation-defined padding and alignment issues.

129

There's a wording problem here that makes this confusing. My first instinct would be to delete the of, but I'm not sure if that's exactly what you meant.

Also, you probably should have a hyphens in fixed-size and variable-length.

161

hyphen: variable-length

[zturner]

Updated

[zturner]

docs/PDB/DbiStream.rst

87

While true, I need a concise way to express this since there are multiple bitfields defined throughout this file and in other files (some of which have not yet been written). If it was easy to insert images into RestructuredText, I could just draw it out, but I was hoping this could be understood to mean "the first 8 bits are X, the next 7 bits are Y, the final bit is Z", etc.

[ruiu]

docs/PDB/DbiStream.rst

87

Why is this uint16? I wonder if this can be written as

uint8_t MinorVersion;
uint8_t MajorVersin : 7;
uint8_t NewVersionFormat : 1;

[zturner]

docs/PDB/DbiStream.rst

87

No, because the bytes of the file are in little endian. If you run this code on a big endian machine, then one ulittle16_t is not the same as two uint8_t's. We read it into a ulittle16_t and then perform the bitwise operations on that.

[ruiu]

docs/PDB/DbiStream.rst

87

Ah, right. But I found it a bit confusing (as I was confused) to use both uint16_t and ulittle16_t to mean little endian 16-bit integer. Maybe you might want to say at the beginning of the document that all values are in little endian and use uint{16,32}_t?

87

Or, something like ulittle16_t X : 1, though it's not syntactically correct.

[zturner]

Add a note that all numeric values are little endian, and then use standard C types instead of endian types in struct definitions.

[ruiu]

LGTM

[amccarth]

LGTM

docs/PDB/DbiStream.rst

-=====================================
-The PDB DBI (Debug Info) Stream
-=====================================
+=====================================
+The PDB DBI (Debug Info) Stream
+=====================================
+
+.. contents::
+   :local:
+
+.. _dbi_intro:
+
+Introduction
+============
+
+The PDB DBI Stream (Index 3) is one of the largest and most important streams
+in a PDB file.  It contains information about how the program was compiled,
+(e.g. compilation flags, etc), the compilands (e.g. object files) that
+were used to link together the program, the source files which were used
+to build the program, as well as references to other streams that contain more
+detailed information about each compiland, such as the CodeView symbol records
+contained within each compiland and the source and line information for
+functions and other symbols within each compiland.
+
+
+.. _dbi_header:
+
+Stream Header
+=============
+At offset 0 of the DBI Stream is a header with the following layout:
+
+
+.. code-block:: c++
+
+  struct DbiStreamHeader {
+    little32_t VersionSignature;
+    ulittle32_t VersionHeader;
+    ulittle32_t Age;
+    ulittle16_t GlobalStreamIndex;
+    ulittle16_t BuildNumber;
+    ulittle16_t PublicStreamIndex;
+    ulittle16_t PdbDllVersion;
+    ulittle16_t SymRecordStream;
+    ulittle16_t PdbDllRbld;
+    little32_t ModInfoSize;
+    little32_t SectionContributionSize;
+    little32_t SectionMapSize;
+    little32_t SourceInfoSize;
+    little32_t TypeServerSize;
+    ulittle32_t MFCTypeServerIndex;
+    little32_t OptionalDbgHeaderSize;
+    little32_t ECSubstreamSize;
+    ulittle16_t Flags;
+    ulittle16_t Machine;
+    ulittle32_t Padding;
+  };
+  
+- **VersionSignature** - Unknown meaning.  Appears to always be ``-1``.
+
+- **VersionHeader** - A value from the following enum.
+
+.. code-block:: c++
+
+  enum class DbiStreamVersion : uint32_t {
+    VC41 = 930803,
+    V50 = 19960307,
+    V60 = 19970606,
+    V70 = 19990903,
+    V110 = 20091201
+  };
+
+Similar to the :doc:`PDB Stream <PdbStream>`, this value always appears to be
+``V70``, and it is not clear what the other values are for.
+
+- **Age** - The number of times the PDB has been written.  Equal to the same
+  field from the :ref:`PDB Stream header <pdb_stream_header>`.
+  
+- **GlobalStreamIndex** - The index of the :doc:`Global Symbol Stream <GlobalStream>`,
+  which contains CodeView symbol records for all global symbols.  Actual records
+  are stored in the symbol record stream, and are referenced from this stream.
+  
+- **BuildNumber** - A bitfield containing values representing the major and minor
+  version number of the toolchain (e.g. 12.0 for MSVC 2013) used to build the
+  program, with the following layout:
+
+.. code-block:: c++
+
+  uint16_t MinorVersion : 8;
+  uint16_t MajorVersion : 7;
+  uint16_t NewVersionFormat : 1;
+
+For the purposes of LLVM, we assume ``NewVersionFormat`` to be always ``true``.
+If it is ``false``, the layout above does not apply and the reader should consult
+the `Microsoft Source Code <https://github.com/Microsoft/microsoft-pdb>`__ for
+further guidance.
+  
+- **PublicStreamIndex** - The index of the :doc:`Public Symbol Stream <PublicStream>`,
+  which contains CodeView symbol records for all public symbols.  Actual records
+  are stored in the symbol record stream, and are referenced from this stream.
+  
+- **PdbDllVersion** - The version number of ``mspdbXXXX.dll`` used to produce this
+  PDB.  Note this obviously does not apply for LLVM as LLVM does not use ``mspdb.dll``.
+  
+- **SymRecordStream** - The stream containing all CodeView symbol records used
+  by the program.  This is used for deduplication, so that many different
+  compilands can refer to the same symbols without having to include the full record
+  content inside of each module stream.
+  
+- **PdbDllRbld** - Unknown
+
+- **MFCTypeServerIndex** - The length of the :ref:dbi_mfc_type_server_substream
+
+- **Flags** - A bitfield with the following layout, containing various
+  information about how the program was built:
+  
+.. code-block:: c++
+
+  uint16_t WasIncrementallyLinked : 1;
+  uint16_t ArePrivateSymbolsStripped : 1;
+  uint16_t HasConflictingTypes : 1;
+  uint16_t Reserved : 13;
+
+The only one of these that is not self-explanatory is ``HasConflictingTypes``.
+Although undocumented, ``link.exe`` contains a hidden flag ``/DEBUG:CTYPES``.
+If it is passed to ``link.exe``, this field will be set.  Otherwise it will
+not be set.  It is unclear what this flag does, although it seems to have
+subtle implications on the algorithm used to look up type records.
+
+- **Machine** - A value from the `CV_CPU_TYPE_e <https://msdn.microsoft.com/en-us/library/b2fc64ek.aspx>`__
+  enumeration.  Common values are ``0x8664`` (x86-64) and ``0x14C`` (x86).
+
+Immediately after the fixed-size DBI Stream header are ``7`` variable-length
+`substreams`.  The following ``7`` fields of the DBI Stream header specify the
+number of bytes of the corresponding substream.  Each substream's contents will
+be described in detail :ref:`below <dbi_substreams>`.  The length of the entire
+DBI Stream should equal ``64`` (the length of the header above) plus the value
+of each of the following ``7`` fields.
+
+- **ModInfoSize** - The length of the :ref:`dbi_mod_info_substream`.
+  
+- **SectionContributionSize** - The length of the :ref:`dbi_sec_contr_substream`.
+
+- **SectionMapSize** - The length of the :ref:`dbi_section_map_substream`.
+
+- **SourceInfoSize** - The length of the :ref:`dbi_file_info_substream`.
+
+- **TypeServerSize** - The length of the :ref:`dbi_type_server_substream`. 
+
+- **OptionalDbgHeaderSize** - The length of the :ref:`dbi_optional_dbg_stream`.
+
+- **ECSubstreamSize** - The length of the :ref:`dbi_ec_substream`.
+
+.. _dbi_substreams:
+
+Substreams
+==========
+
+.. _dbi_mod_info_substream:
+
+Module Info Substream
+^^^^^^^^^^^^^^^^^^^^^
+
+Begins at offset ``0`` immediately after the :ref:`header <dbi_header>`.  The
+module info substream is an array of variable-length records, each one
+describing a single module (e.g. object file) linked into the program.  Each
+record in the array has the format:
+  
+.. code-block:: c++
+
+  struct SectionContribEntry {
+    ulittle16_t Section;
+    char Padding1[2];
+    little32_t Offset;
+    little32_t Size;
+    ulittle32_t Characteristics;
+    ulittle16_t ModuleIndex;
+    char Padding2[2];
+    ulittle32_t DataCrc;
+    ulittle32_t RelocCrc;
+  };
+  
+While most of these are self-explanatory, the ``Characteristics`` field
+warrants some elaboration.  It corresponds to the ``Characteristics``
+field of the `IMAGE_SECTION_HEADER <https://msdn.microsoft.com/en-us/library/windows/desktop/ms680341(v=vs.85).aspx>`__
+structure.
+  
+.. code-block:: c++
+
+  struct ModInfo {
+    ulittle32_t Unused1;
+    SectionContribEntry SectionContr;
+    ulittle16_t Flags;
+    ulittle16_t ModuleSymStream;
+    ulittle32_t SymByteSize;
+    ulittle32_t C11ByteSize;
+    ulittle32_t C13ByteSize;
+    ulittle16_t SourceFileCount;
+    char Padding[2];
+    ulittle32_t Unused2;
+    ulittle32_t SourceFileNameIndex;
+    ulittle32_t PdbFilePathNameIndex;
+    char ModuleName[];
+    char ObjFileName[];
+  };
+  
+- **SectionContr** - Describes the properties of the section in the final binary
+  which contain the code and data from this module.
+
+- **Flags** - A bitfield with the following format:
+  
+.. code-block:: c++
+
+  uint16_t Dirty : 1;  // ``true`` if this ModInfo has been written since reading the PDB.
+  uint16_t EC : 1;     // ``true`` if EC information is present for this module. It is unknown what EC actually is.
+  uint16_t Unused : 6;
+  uint16_t TSM : 8;    // Type Server Index for this module.  It is unknown what this is used for, but it is not used by LLVM.
+  
+
+- **ModuleSymStream** - The index of the stream that contains symbol information
+  for this module.  This includes CodeView symbol information as well as source
+  and line information.
+
+- **SymByteSize** - The number of bytes of data from the stream identified by
+  ``ModuleSymStream`` that represent CodeView symbol records.
+
+- **C11ByteSize** - The number of bytes of data from the stream identified by
+  ``ModuleSymStream`` that represent C11-style CodeView line information.
+
+- **C13ByteSize** - The number of bytes of data from the stream identified by
+  ``ModuleSymStream`` that represent C13-style CodeView line information.  At
+  most one of ``C11ByteSize`` and ``C13ByteSize`` will be non-zero.
+
+- **SourceFileCount** - The number of source files that contributed to this
+  module during compilation.
+
+- **SourceFileNameIndex** - The offset in the names buffer of the primary
+  translation unit used to build this module.  All PDB files observed to date
+  always have this value equal to 0.
+
+- **PdbFilePathNameIndex** - The offset in the names buffer of the PDB file
+  containing this module's symbol information.  This has only been observed
+  to be non-zero for the special ``* Linker *`` module.
+
+- **ModuleName** - The module name.  This is usually either a full path to an
+  object file (either directly passed to ``link.exe`` or from an archive) or
+  a string of the form ``Import:<dll name>``.
+
+- **ObjFileName** - The object file name.  In the case of an module that is
+  linked directly passed to ``link.exe``, this is the same as **ModuleName**.
+  In the case of a module that comes from an archive, this is usually the full
+  path to the archive.
+
+.. _dbi_sec_contr_substream:
+
+Section Contribution Substream
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_mod_info_substream` ends,
+and consumes ``Header->SectionContributionSize`` bytes.  This substream begins
+with a single ``ulittle32_t`` which will be one of the following values:
+  
+.. code-block:: c++
+
+  enum class SectionContrSubstreamVersion : uint32_t {
+    Ver60 = 0xeffe0000 + 19970605,
+    V2 = 0xeffe0000 + 20140516
+  };
+  
+``Ver60`` is the only value which has been observed in a PDB so far.  Following
+this ``4`` byte field is an array of fixed-length structures.  If the version
+is ``Ver60``, it is an array of ``SectionContribEntry`` structures.  If the
+version is ``V2``, it is an array of ``SectionContribEntry2`` structures,
+defined as follows:
+  
+.. code-block:: c++
+
+  struct SectionContribEntry2 {
+    SectionContribEntry SC;
+    ulittle32_t ISectCoff;
+  };
+  
+The purpose of the second field is not well understood.
+  
+
+.. _dbi_section_map_substream:
+
+Section Map Substream
+^^^^^^^^^^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_sec_contr_substream` ends,
+and consumes ``Header->SectionMapSize`` bytes.  This substream begins with an ``8``
+byte header followed by an array of fixed-length records.  The header and records
+have the following layout:
+  
+.. code-block:: c++
+
+  struct SectionMapHeader {
+    ulittle16_t Count;    // Number of segment descriptors
+    ulittle16_t LogCount; // Number of logical segment descriptors
+  };
+  
+  struct SectionMapEntry {
+    ulittle16_t Flags;         // See the SectionMapEntryFlags enum below.
+    ulittle16_t Ovl;           // Logical overlay number
+    ulittle16_t Group;         // Group index into descriptor array.
+    ulittle16_t Frame;
+    ulittle16_t SectionName;   // Byte index of segment / group name in string table, or 0xFFFF.
+    ulittle16_t ClassName;     // Byte index of class in string table, or 0xFFFF.
+    ulittle32_t Offset;        // Byte offset of the logical segment within physical segment.  If group is set in flags, this is the offset of the group.
+    ulittle32_t SectionLength; // Byte count of the segment or group.
+  };
+  
+  enum class SectionMapEntryFlags : uint16_t {
+    Read = 1 << 0,              // Segment is readable.
+    Write = 1 << 1,             // Segment is writable.
+    Execute = 1 << 2,           // Segment is executable.
+    AddressIs32Bit = 1 << 3,    // Descriptor describes a 32-bit linear address.
+    IsSelector = 1 << 8,        // Frame represents a selector.
+    IsAbsoluteAddress = 1 << 9, // Frame represents an absolute address.
+    IsGroup = 1 << 10           // If set, descriptor represents a group.
+  };
+  
+Many of these fields are not well understood, so will not be discussed further.
+
+.. _dbi_file_info_substream:
+
+File Info Substream
+^^^^^^^^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_section_map_substream` ends,
+and consumes ``Header->SourceInfoSize`` bytes.  This substream defines the mapping
+from module to the source files that contribute to that module.  Since multiple
+modules can use the same source file (for example, a header file), this substream
+uses a string table to store each unique file name only once, and then have each
+module use offsets into the string table rather than embedding the string's value
+directly.  The format of this substream is as follows:
+  
+.. code-block:: c++
+
+  struct FileInfoSubstream {
+    ulittle16_t NumModules;
+    ulittle16_t NumSourceFiles;
+    
+    ulittle16_t ModIndices[NumModules];
+    ulittle16_t ModFileCounts[NumModules];
+    ulittle32_t FileNameOffsets[NumSourceFiles];
+    char NamesBuffer[][NumSourceFiles];
+  };
+
+**NumModules** - The number of modules for which source file information is
+contained within this substream.  Should match the corresponding value from the
+ref:`dbi_header`.
+
+**NumSourceFiles**: In theory this is supposed to contain the number of source
+files for which this substream contains information.  But that would present a
+problem in that the width of this field being ``16``-bits would prevent one from
+having more than 64K source files in a program.  In early versions of the file
+format, this seems to have been the case.  In order to support more than this, this
+field of the is simply ignored, and computed dynamically by summing up the values of
+the ``ModFileCounts`` array (discussed below).  In short, this value should be
+ignored.
+
+**ModIndices** - This array is present, but does not appear to be useful.
+
+**ModFileCountArray** - An array of ``NumModules`` integers, each one containing
+the number of source files which contribute to the module at the specified index.
+While each individual module is limited to 64K contributing source files, the
+union of all modules' source files may be greater than 64K.  The real number of
+source files is thus computed by summing this array.  Note that summing this array
+does not give the number of `unique` source files, only the total number of source
+file contributions to modules.
+
+**FileNameOffsets** - An array of **NumSourceFiles** integers (where **NumSourceFiles**
+here refers to the 32-bit value obtained from summing **ModFileCountArray**), where
+each integer is an offset into **NamesBuffer** pointing to a null terminated string.
+
+**NamesBuffer** - An array of null terminated strings containing the actual source
+file names.
+
+.. _dbi_type_server_substream:
+
+Type Server Substream
+^^^^^^^^^^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_file_info_substream` ends,
+and consumes ``Header->TypeServerSize`` bytes.  Neither the purpose nor the layout
+of this substream is understood, although it is assumed to related somehow to the
+usage of ``/Zi`` and ``mspdbsrv.exe``.  This substream will not be discussed further.
+
+.. _dbi_ec_substream:
+
+EC Substream
+^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_type_server_substream` ends,
+and consumes ``Header->ECSubstreamSize`` bytes.  Neither the purpose nor the layout
+of this substream is understood, and it will not be discussed further.
+
+.. _dbi_optional_dbg_stream:
+
+Optional Debug Header Stream
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Begins at offset ``0`` immediately after the :ref:`dbi_ec_substream` ends, and
+consumes ``Header->OptionalDbgHeaderSize`` bytes.  This field is an array of
+stream indices (e.g. ``ulittle16_t``'s), each of which identifies a stream
+index in the larger MSF file which contains some additional debug information.
+Each position of this array has a special meaning, allowing one to determine
+what kind of debug information is at the referenced stream.  ``11`` indices
+are currently understood, although it's possible there may be more.  The
+layout of each stream generally corresponds exactly to a particular type
+of debug data directory from the PE/COFF file.  The format of these fields
+can be found in the `Microsoft PE/COFF Specification <https://www.microsoft.com/en-us/download/details.aspx?id=19509>`__.
+
+**FPO Data** - ``DbgStreamArray[0]``.  The data in the referenced stream is a
+debug data directory of type ``IMAGE_DEBUG_TYPE_FPO``
+
+**Exception Data** - ``DbgStreamArray[1]``.  The data in the referenced stream
+is a debug data directory of type ``IMAGE_DEBUG_TYPE_EXCEPTION``.
+
+**Fixup Data** - ``DbgStreamArray[2]``.  The data in the referenced stream is a
+debug data directory of type ``IMAGE_DEBUG_TYPE_FIXUP``.
+
+**Omap To Src Data** - ``DbgStreamArray[3]``.  The data in the referenced stream
+is a debug data directory of type ``IMAGE_DEBUG_TYPE_OMAP_TO_SRC``.  This 
+is used for mapping addresses between instrumented and uninstrumented code.
+
+**Omap From Src Data** - ``DbgStreamArray[4]``.  The data in the referenced stream
+is a debug data directory of type ``IMAGE_DEBUG_TYPE_OMAP_FROM_SRC``.  This 
+is used for mapping addresses between instrumented and uninstrumented code.
+
+**Section Header Data** - ``DbgStreamArray[5]``.  A dump of all section headers from
+the original executable.
+
+**Token / RID Map** - ``DbgStreamArray[6]``.  The layout of this stream is not
+understood, but it is assumed to be a mapping from ``CLR Token`` to 
+``CLR Record ID``.  Refer to `ECMA 335 <http://www.ecma-international.org/publications/standards/Ecma-335.htm>`__
+for more information.
+
+**Xdata** - ``DbgStreamArray[7]``.  A copy of the ``.xdata`` section from the
+executable.
+
+**Pdata** - ``DbgStreamArray[8]``. This is assumed to be a copy of the ``.pdata``
+section from the executable, but that would make it identical to
+``DbgStreamArray[1]``.  The difference between these two indices is not well
+understood.
+
+**New FPO Data** - ``DbgStreamArray[9]``.  The data in the referenced stream is a
+debug data directory of type ``IMAGE_DEBUG_TYPE_FPO``.  It is not clear how this
+differs from ``DbgStreamArray[0]``, but in practice all observed PDB files have
+used the "new" format rather than the "old" format.
+
+**Original Section Header Data** - ``DbgStreamArray[10]``.  Assumed to be similar
+to ``DbgStreamArray[5]``, but has not been observed in practice.