[CodeView] Allow Debug Subsections to be read/written in any order

[zturner]

Summary

Debug Subsections (Checksums, Line Info, Frame Data, etc) present an interesting challenge when it comes to reading and writing, especially to and from YAML. Some of these challenges are:

1. Depending on whether the CodeView container is a PDB or an Object File, these show up in slightly different ways. For example, in an Object File, string tables appear in the .debug$S section as a "debug subsection", while in a PDB file the string table is global to the entire file and exists
2. They can appear in any order, even though some subsections reference other subsections. For example, you might have a Lines subsection followed by a Checksums subsection followed by a String Table subsection. But the Lines subsection references the Checksums subsection, and the Checksum subsection references the String Table subsection. So you can't interpret the data until you've read everything.

When going from Obj to Yaml, you might read (for example) a File Checksums subsection, which references a String Table subsection which you haven't actually read yet. But in Pdb to Yaml, you'll already have the string table section because it's global to the file.

Prior to this patch, we handled in this in sort of a crude way. When going from PDB to YAML, we would find the checksums subsection and write it to YAML first, then write subsequent sections later. In a sense mandating a topological sorting similar to type record streams. But this has a drawback. It means that we can't easily test our ability to work with arbitrary files, because the only files we can produce are laid out a specific way. Furthermore, it means that round-tripping will end up producing a different PDB than what you started with, because some of the subsections will be in a different order.

Finally, it makes things even more complicated when we start trying to do Obj -> Yaml -> Obj, because now these have a String Table subsection and various other subsections with cross subsection dependencies, and maintaining the topological sorting becomes cumbersome. So I consider this work a precursor to getting CodeView Obj <-> Yaml support working.

In short, if it is a valid object (or PDB) file, we would like to be able to produce it.

This patch allows subsections to be specified in any order. While this complicates some of the internal Yaml <-> Native data structure conversion logic, it greatly simplifies the portion of the library which actually writes subsections out, because you no longer have to have complicated sanitization checks to ensure things are written out in a specified order. You can just write whatever you want and it will just work.

[inglorion]

Happy you're doing this, but see inline comments for a few requested changes.

llvm/lib/DebugInfo/CodeView/DebugStringTableSubsection.cpp

79 ↗

(On Diff #101127)

This will call StringToId.find(S) twice when asserts are enabled. Since that returns an iterator, can we instead do

auto It = StringToId.find(S);
assert(It != StringToId.end() && "getStringId called with string that does not exist in the string table");
return It->second;

and save the second lookup?

llvm/lib/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.cpp

42 ↗

(On Diff #101127)

Do we need this? I thought we made it so that SymbolByteSize is always a multiple of 4 when writing PDBs. Also, is this related to the other changes here or a holdover from your earlier alignment fixes?

llvm/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp

93 ↗

(On Diff #101127)

If I'm reading the code in ModuleDebugStreamRef::reload() correctly, we read the subsections before it makes sense to get to this code. Given that, can we store which one is the FileChecksums subsection at reload() time so that we don't have to iterate over all the subsections to find it?

[zturner]

llvm/lib/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.cpp

42 ↗

(On Diff #101127)

I originally started investigating the alignment issue on the same branch as I was doing this patch. Then when I committed the other one and rebased on top of trunk, these changes were still here. I need to find all these remnants and remove them.

[zturner]

llvm/lib/DebugInfo/CodeView/DebugStringTableSubsection.cpp

79 ↗

(On Diff #101127)

For a debug build I think readability should take priority over performance unless there's a good reason not to. Which is why I used exists(). That said, I also think it's not a big deal and it's a minor preference, so I don't mind changing it.

llvm/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp

93 ↗

(On Diff #101127)

This is a ModuleDebugStreamRef, which literally contains no information about the subsections other than a big flat buffer consisting of their data. In order to do what you suggest, we'd have to iterate them up front during the reload, and then you would suffer that performance hit even in situations where you never accessed the module debug streams. I fixed a similar performance problem in pdbdump a few weeks ago where we were iterating all types up front to compute their hashes during the reload operatino, and then nobody ended up looking at the hash values. Usually this isn't noticeable, but when I tried it on a 1.5GB PDB file, it took several minutes.

I could see doing something like caching the checksums subsection in the ModuleDebugStreamRef class and only doing the full scan the first time this function is called, but short of that I don't think we should change it.

[zturner]

Updated based on previous feedback.

[zturner]

Minor update. Removed dead function, renamed variable back to its original name, and changed assertion as suggested previously.

[zturner]

More minor cleanup. Deleted an unused class and file

[inglorion]

lgtm

llvm/trunk/include/llvm/DebugInfo/CodeView/DebugChecksumsSubsection.h

static bool classof(const DebugSubsectionRef *S) {
     return S->kind() == DebugSubsectionKind::FileChecksums;
   }
 
   bool valid() const { return Checksums.valid(); }
 
   Error initialize(BinaryStreamReader Reader);
   Error initialize(BinaryStreamRef Stream);
 
-  Iterator begin() { return Checksums.begin(); }
-  Iterator end() { return Checksums.end(); }
+  Iterator begin() const { return Checksums.begin(); }
+  Iterator end() const { return Checksums.end(); }
 
   const FileChecksumArray &getArray() const { return Checksums; }
 
 private:
   FileChecksumArray Checksums;
 };
 
 class DebugChecksumsSubsection final : public DebugSubsection {

llvm/trunk/include/llvm/DebugInfo/CodeView/DebugInlineeLinesSubsection.h

 
 private:
   InlineeLinesSignature Signature;
   VarStreamArray<InlineeSourceLine> Lines;
 };
 
 class DebugInlineeLinesSubsection final : public DebugSubsection {
 public:
+  struct Entry {
+    std::vector<support::ulittle32_t> ExtraFiles;
+    InlineeSourceLineHeader Header;
+  };
+
   DebugInlineeLinesSubsection(DebugChecksumsSubsection &Checksums,
-                              bool HasExtraFiles);
+                              bool HasExtraFiles = false);
 
   static bool classof(const DebugSubsection *S) {
     return S->kind() == DebugSubsectionKind::InlineeLines;
   }
 
   Error commit(BinaryStreamWriter &Writer) const override;
   uint32_t calculateSerializedSize() const override;
 
   void addInlineSite(TypeIndex FuncId, StringRef FileName, uint32_t SourceLine);
   void addExtraFile(StringRef FileName);
 
+  bool hasExtraFiles() const { return HasExtraFiles; }
+  void setHasExtraFiles(bool Has) { HasExtraFiles = Has; }
+
+  std::vector<Entry>::const_iterator begin() const { return Entries.begin(); }
+  std::vector<Entry>::const_iterator end() const { return Entries.end(); }
+
 private:
   DebugChecksumsSubsection &Checksums;
 
   bool HasExtraFiles = false;
   uint32_t ExtraFileCount = 0;
 
-  struct Entry {
-    std::vector<support::ulittle32_t> ExtraFiles;
-    InlineeSourceLineHeader Header;
-  };
   std::vector<Entry> Entries;
 };
 }
 }
 
 #endif

llvm/trunk/include/llvm/DebugInfo/CodeView/DebugSubsectionRecord.h

 private:
   CodeViewContainer Container;
   DebugSubsectionKind Kind;
   BinaryStreamRef Data;
 };
 
 class DebugSubsectionRecordBuilder {
 public:
-  DebugSubsectionRecordBuilder(DebugSubsectionKind Kind, DebugSubsection &Frag,
+  DebugSubsectionRecordBuilder(std::unique_ptr<DebugSubsection> Subsection,
                                CodeViewContainer Container);
   uint32_t calculateSerializedLength();
   Error commit(BinaryStreamWriter &Writer);
 
 private:
+  std::unique_ptr<DebugSubsection> Subsection;
   CodeViewContainer Container;
-  DebugSubsectionKind Kind;
-  DebugSubsection &Frag;
 };
 
 } // namespace codeview
 
 template <> struct VarStreamArrayExtractor<codeview::DebugSubsectionRecord> {
   typedef void ContextType;
 
   static Error extract(BinaryStreamRef Stream, uint32_t &Length,

llvm/trunk/include/llvm/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.h

public:
 
   DbiModuleDescriptorBuilder(const DbiModuleDescriptorBuilder &) = delete;
   DbiModuleDescriptorBuilder &
   operator=(const DbiModuleDescriptorBuilder &) = delete;
 
   void setObjFileName(StringRef Name);
   void addSymbol(codeview::CVSymbol Symbol);
 
-  void addC13Fragment(std::unique_ptr<codeview::DebugLinesSubsection> Lines);
-  void addC13Fragment(
-      std::unique_ptr<codeview::DebugInlineeLinesSubsection> Inlinees);
-  void setC13FileChecksums(
-      std::unique_ptr<codeview::DebugChecksumsSubsection> Checksums);
+  void
+  addDebugSubsection(std::unique_ptr<codeview::DebugSubsection> Subsection);
 
   uint16_t getStreamIndex() const;
   StringRef getModuleName() const { return ModuleName; }
   StringRef getObjFileName() const { return ObjFileName; }
 
   ArrayRef<std::string> source_files() const {
     return makeArrayRef(SourceFiles);
   }
private:
   msf::MSFBuilder &MSF;
 
   uint32_t SymbolByteSize = 0;
   std::string ModuleName;
   std::string ObjFileName;
   std::vector<std::string> SourceFiles;
   std::vector<codeview::CVSymbol> Symbols;
 
-  std::unique_ptr<codeview::DebugChecksumsSubsection> ChecksumInfo;
-  std::vector<std::unique_ptr<codeview::DebugLinesSubsection>> LineInfo;
-  std::vector<std::unique_ptr<codeview::DebugInlineeLinesSubsection>> Inlinees;
-
   std::vector<std::unique_ptr<codeview::DebugSubsectionRecordBuilder>>
       C13Builders;
 
   ModuleInfoHeader Layout;
 };
 
 } // end namespace pdb
 
 } // end namespace llvm
 
 #endif // LLVM_DEBUGINFO_PDB_RAW_DBIMODULEDESCRIPTORBUILDER_H

llvm/trunk/include/llvm/DebugInfo/PDB/Native/ModuleDebugStream.h

 //===- ModuleDebugStream.h - PDB Module Info Stream Access ----------------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_PDB_RAW_MODULEDEBUGSTREAM_H
 #define LLVM_DEBUGINFO_PDB_RAW_MODULEDEBUGSTREAM_H
 
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/DebugInfo/CodeView/CVRecord.h"
+#include "llvm/DebugInfo/CodeView/DebugChecksumsSubsection.h"
 #include "llvm/DebugInfo/CodeView/DebugSubsectionRecord.h"
 #include "llvm/DebugInfo/CodeView/SymbolRecord.h"
 #include "llvm/DebugInfo/MSF/MappedBlockStream.h"
 #include "llvm/Support/BinaryStreamArray.h"
 #include "llvm/Support/BinaryStreamRef.h"
 #include "llvm/Support/Error.h"
 
 namespace llvm {
 namespace pdb {
 class PDBFile;
 class DbiModuleDescriptor;
 
 class ModuleDebugStreamRef {
-  typedef codeview::DebugSubsectionArray::Iterator LinesAndChecksumsIterator;
+  typedef codeview::DebugSubsectionArray::Iterator DebugSubsectionIterator;
 
 public:
   ModuleDebugStreamRef(const DbiModuleDescriptor &Module,
                        std::unique_ptr<msf::MappedBlockStream> Stream);
   ~ModuleDebugStreamRef();
 
   Error reload();
 
   uint32_t signature() const { return Signature; }
 
   iterator_range<codeview::CVSymbolArray::Iterator>
   symbols(bool *HadError) const;
 
-  llvm::iterator_range<LinesAndChecksumsIterator> linesAndChecksums() const;
+  llvm::iterator_range<DebugSubsectionIterator> subsections() const;
 
-  bool hasLineInfo() const;
+  bool hasDebugSubsections() const;
 
   Error commit();
 
+  Expected<codeview::DebugChecksumsSubsectionRef>
+  findChecksumsSubsection() const;
+
 private:
   const DbiModuleDescriptor &Mod;
 
   uint32_t Signature;
 
   std::unique_ptr<msf::MappedBlockStream> Stream;
 
   codeview::CVSymbolArray SymbolsSubstream;
   BinaryStreamRef C11LinesSubstream;
   BinaryStreamRef C13LinesSubstream;
   BinaryStreamRef GlobalRefsSubstream;
 
-  codeview::DebugSubsectionArray LinesAndChecksums;
+  codeview::DebugSubsectionArray Subsections;
 };
 }
 }
 
 #endif

llvm/trunk/include/llvm/DebugInfo/PDB/Native/PDBStringTable.h

public:
   uint32_t getHashVersion() const;
   uint32_t getSignature() const;
 
   Expected<StringRef> getStringForID(uint32_t ID) const;
   Expected<uint32_t> getIDForString(StringRef Str) const;
 
   FixedStreamArray<support::ulittle32_t> name_ids() const;
 
+  codeview::DebugStringTableSubsectionRef getStringTable() const;
+
 private:
   Error readHeader(BinaryStreamReader &Reader);
   Error readStrings(BinaryStreamReader &Reader);
   Error readHashTable(BinaryStreamReader &Reader);
   Error readEpilogue(BinaryStreamReader &Reader);
 
   const PDBStringTableHeader *Header = nullptr;
   codeview::DebugStringTableSubsectionRef Strings;

llvm/trunk/include/llvm/ObjectYAML/CodeViewYAMLDebugSections.h

 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_OBJECTYAML_CODEVIEWYAMLDEBUGSECTIONS_H
 #define LLVM_OBJECTYAML_CODEVIEWYAMLDEBUGSECTIONS_H
 
 #include "llvm/DebugInfo/CodeView/CodeView.h"
 #include "llvm/DebugInfo/CodeView/DebugSubsection.h"
+#include "llvm/DebugInfo/CodeView/DebugSubsectionRecord.h"
 #include "llvm/ObjectYAML/YAML.h"
 
 namespace llvm {
+
+namespace codeview {
+class DebugStringTableSubsection;
+class DebugStringTableSubsectionRef;
+class DebugChecksumsSubsectionRef;
+}
 namespace CodeViewYAML {
+
 namespace detail {
-struct C13FragmentBase;
+struct YAMLSubsectionBase;
 }
 
 struct SourceLineEntry {
   uint32_t Offset;
   uint32_t LineStart;
   uint32_t EndDelta;
   bool IsStatement;
 };
struct InlineeSite {
   std::vector<StringRef> ExtraFiles;
 };
 
 struct InlineeInfo {
   bool HasExtraFiles;
   std::vector<InlineeSite> Sites;
 };
 
-struct SourceFileInfo {
-  std::vector<SourceFileChecksumEntry> FileChecksums;
-  std::vector<SourceLineInfo> LineFragments;
-  std::vector<InlineeInfo> Inlinees;
+struct YAMLDebugSubsection {
+  static Expected<YAMLDebugSubsection>
+  fromCodeViewSubection(const codeview::DebugStringTableSubsectionRef &Strings,
+                        const codeview::DebugChecksumsSubsectionRef &Checksums,
+                        const codeview::DebugSubsectionRecord &SS);
+
+  std::shared_ptr<detail::YAMLSubsectionBase> Subsection;
 };
 
-struct C13DebugSection {
-  std::vector<detail::C13FragmentBase> Fragments;
-};
+Expected<std::vector<std::unique_ptr<codeview::DebugSubsection>>>
+convertSubsectionList(ArrayRef<YAMLDebugSubsection> Subsections,
+                      codeview::DebugStringTableSubsection &Strings);
+
 } // namespace CodeViewYAML
 } // namespace llvm
 
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceFileInfo)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::YAMLDebugSubsection)
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(CodeViewYAML::YAMLDebugSubsection)
 
 #endif

llvm/trunk/lib/DebugInfo/CodeView/DebugStringTableSubsection.cpp

Error DebugStringTableSubsection::commit(BinaryStreamWriter &Writer) const {
   Writer.setOffset(MaxOffset);
   assert(Writer.bytesRemaining() == 0);
   return Error::success();
 }
 
 uint32_t DebugStringTableSubsection::size() const { return Strings.size(); }
 
 uint32_t DebugStringTableSubsection::getStringId(StringRef S) const {
-  auto P = Strings.find(S);
-  assert(P != Strings.end());
-  return P->second;
+  auto Iter = Strings.find(S);
+  assert(Iter != Strings.end());
+  return Iter->second;
 }

llvm/trunk/lib/DebugInfo/CodeView/DebugSubsectionRecord.cpp

uint32_t DebugSubsectionRecord::getRecordLength() const {
   return Result;
 }
 
 DebugSubsectionKind DebugSubsectionRecord::kind() const { return Kind; }
 
 BinaryStreamRef DebugSubsectionRecord::getRecordData() const { return Data; }
 
 DebugSubsectionRecordBuilder::DebugSubsectionRecordBuilder(
-    DebugSubsectionKind Kind, DebugSubsection &Frag,
-    CodeViewContainer Container)
-    : Container(Container), Kind(Kind), Frag(Frag) {}
+    std::unique_ptr<DebugSubsection> Subsection, CodeViewContainer Container)
+    : Subsection(std::move(Subsection)), Container(Container) {}
 
 uint32_t DebugSubsectionRecordBuilder::calculateSerializedLength() {
-  uint32_t Size = sizeof(DebugSubsectionHeader) +
-                  alignTo(Frag.calculateSerializedSize(), alignOf(Container));
+  uint32_t Size =
+      sizeof(DebugSubsectionHeader) +
+      alignTo(Subsection->calculateSerializedSize(), alignOf(Container));
   return Size;
 }
 
 Error DebugSubsectionRecordBuilder::commit(BinaryStreamWriter &Writer) {
   assert(Writer.getOffset() % alignOf(Container) == 0 &&
          "Debug Subsection not properly aligned");
 
   DebugSubsectionHeader Header;
-  Header.Kind = uint32_t(Kind);
+  Header.Kind = uint32_t(Subsection->kind());
   Header.Length = calculateSerializedLength() - sizeof(DebugSubsectionHeader);
 
   if (auto EC = Writer.writeObject(Header))
     return EC;
-  if (auto EC = Frag.commit(Writer))
+  if (auto EC = Subsection->commit(Writer))
     return EC;
   if (auto EC = Writer.padToAlignment(alignOf(Container)))
     return EC;
 
   return Error::success();
 }

llvm/trunk/lib/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.cpp

if (Layout.ModDiStream != kInvalidStreamIndex) {
     if (auto EC = SymbolWriter.writeInteger<uint32_t>(0))
       return EC;
     if (SymbolWriter.bytesRemaining() > 0)
       return make_error<RawError>(raw_error_code::stream_too_long);
   }
   return Error::success();
 }
 
-void DbiModuleDescriptorBuilder::addC13Fragment(
-    std::unique_ptr<DebugLinesSubsection> Lines) {
-  DebugLinesSubsection &Frag = *Lines;
-
-  // File Checksums have to come first, so push an empty entry on if this
-  // is the first.
-  if (C13Builders.empty())
-    C13Builders.push_back(nullptr);
-
-  this->LineInfo.push_back(std::move(Lines));
-  C13Builders.push_back(llvm::make_unique<DebugSubsectionRecordBuilder>(
-      Frag.kind(), Frag, CodeViewContainer::Pdb));
-}
-
-void DbiModuleDescriptorBuilder::addC13Fragment(
-    std::unique_ptr<codeview::DebugInlineeLinesSubsection> Inlinees) {
-  DebugInlineeLinesSubsection &Frag = *Inlinees;
-
-  // File Checksums have to come first, so push an empty entry on if this
-  // is the first.
-  if (C13Builders.empty())
-    C13Builders.push_back(nullptr);
-
-  this->Inlinees.push_back(std::move(Inlinees));
+void DbiModuleDescriptorBuilder::addDebugSubsection(
+    std::unique_ptr<DebugSubsection> Subsection) {
+  assert(Subsection);
   C13Builders.push_back(llvm::make_unique<DebugSubsectionRecordBuilder>(
-      Frag.kind(), Frag, CodeViewContainer::Pdb));
-}
-
-void DbiModuleDescriptorBuilder::setC13FileChecksums(
-    std::unique_ptr<DebugChecksumsSubsection> Checksums) {
-  assert(!ChecksumInfo && "Can't have more than one checksum info!");
-
-  if (C13Builders.empty())
-    C13Builders.push_back(nullptr);
-
-  ChecksumInfo = std::move(Checksums);
-  C13Builders[0] = llvm::make_unique<DebugSubsectionRecordBuilder>(
-      ChecksumInfo->kind(), *ChecksumInfo, CodeViewContainer::Pdb);
+      std::move(Subsection), CodeViewContainer::Pdb));
 }

llvm/trunk/lib/DebugInfo/PDB/Native/ModuleDebugStream.cpp

Error ModuleDebugStreamRef::reload() {
   if (auto EC = Reader.readArray(SymbolsSubstream, SymbolSize - 4))
     return EC;
 
   if (auto EC = Reader.readStreamRef(C11LinesSubstream, C11Size))
     return EC;
   if (auto EC = Reader.readStreamRef(C13LinesSubstream, C13Size))
     return EC;
 
-  BinaryStreamReader LineReader(C13LinesSubstream);
-  if (auto EC =
-          LineReader.readArray(LinesAndChecksums, LineReader.bytesRemaining()))
+  BinaryStreamReader SubsectionsReader(C13LinesSubstream);
+  if (auto EC = SubsectionsReader.readArray(Subsections,
+                                            SubsectionsReader.bytesRemaining()))
     return EC;
 
   uint32_t GlobalRefsSize;
   if (auto EC = Reader.readInteger(GlobalRefsSize))
     return EC;
   if (auto EC = Reader.readStreamRef(GlobalRefsSubstream, GlobalRefsSize))
     return EC;
   if (Reader.bytesRemaining() > 0)
     return make_error<RawError>(raw_error_code::corrupt_file,
                                 "Unexpected bytes in module stream.");
 
   return Error::success();
 }
 
 iterator_range<codeview::CVSymbolArray::Iterator>
 ModuleDebugStreamRef::symbols(bool *HadError) const {
   return make_range(SymbolsSubstream.begin(HadError), SymbolsSubstream.end());
 }
 
-llvm::iterator_range<ModuleDebugStreamRef::LinesAndChecksumsIterator>
-ModuleDebugStreamRef::linesAndChecksums() const {
-  return make_range(LinesAndChecksums.begin(), LinesAndChecksums.end());
+llvm::iterator_range<ModuleDebugStreamRef::DebugSubsectionIterator>
+ModuleDebugStreamRef::subsections() const {
+  return make_range(Subsections.begin(), Subsections.end());
 }
 
-bool ModuleDebugStreamRef::hasLineInfo() const {
+bool ModuleDebugStreamRef::hasDebugSubsections() const {
   return C13LinesSubstream.getLength() > 0;
 }
 
 Error ModuleDebugStreamRef::commit() { return Error::success(); }
+
+Expected<codeview::DebugChecksumsSubsectionRef>
+ModuleDebugStreamRef::findChecksumsSubsection() const {
+  for (const auto &SS : subsections()) {
+    if (SS.kind() != DebugSubsectionKind::FileChecksums)
+      continue;
+
+    codeview::DebugChecksumsSubsectionRef Result;
+    if (auto EC = Result.initialize(SS.getRecordData()))
+      return std::move(EC);
+    return Result;
+  }
+  return make_error<RawError>(raw_error_code::no_entry);
+}

llvm/trunk/lib/DebugInfo/PDB/Native/PDBStringTable.cpp

return joinErrors(std::move(EC),
                       make_error<RawError>(raw_error_code::corrupt_file,
                                            "Invalid hash table byte length"));
   }
 
   assert(Reader.bytesRemaining() == 0);
   return Error::success();
 }
 
+codeview::DebugStringTableSubsectionRef PDBStringTable::getStringTable() const {
+  return Strings;
+}
+
 Error PDBStringTable::readHashTable(BinaryStreamReader &Reader) {
   const support::ulittle32_t *HashCount;
   if (auto EC = Reader.readObject(HashCount))
     return EC;
 
   if (auto EC = Reader.readArray(IDs, *HashCount)) {
     return joinErrors(std::move(EC),
                       make_error<RawError>(raw_error_code::corrupt_file,

llvm/trunk/lib/ObjectYAML/CodeViewYAMLDebugSections.cpp

 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ObjectYAML/CodeViewYAMLDebugSections.h"
 
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/ADT/StringSwitch.h"
 #include "llvm/DebugInfo/CodeView/CodeViewError.h"
+#include "llvm/DebugInfo/CodeView/DebugChecksumsSubsection.h"
+#include "llvm/DebugInfo/CodeView/DebugInlineeLinesSubsection.h"
+#include "llvm/DebugInfo/CodeView/DebugLinesSubsection.h"
+#include "llvm/DebugInfo/CodeView/DebugStringTableSubsection.h"
+#include "llvm/DebugInfo/CodeView/DebugSubsectionVisitor.h"
 #include "llvm/DebugInfo/CodeView/EnumTables.h"
 #include "llvm/DebugInfo/CodeView/SymbolRecord.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 using namespace llvm::CodeViewYAML;
 using namespace llvm::CodeViewYAML::detail;
 using namespace llvm::yaml;
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceFileChecksumEntry)
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceLineEntry)
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceColumnEntry)
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceLineBlock)
 LLVM_YAML_IS_SEQUENCE_VECTOR(SourceLineInfo)
 LLVM_YAML_IS_SEQUENCE_VECTOR(InlineeSite)
 LLVM_YAML_IS_SEQUENCE_VECTOR(InlineeInfo)
 LLVM_YAML_IS_SEQUENCE_VECTOR(StringRef)
 
 LLVM_YAML_DECLARE_SCALAR_TRAITS(HexFormattedString, false)
+LLVM_YAML_DECLARE_ENUM_TRAITS(DebugSubsectionKind)
 LLVM_YAML_DECLARE_ENUM_TRAITS(FileChecksumKind)
 LLVM_YAML_DECLARE_BITSET_TRAITS(LineFlags)
 
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceLineEntry)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceColumnEntry)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceFileChecksumEntry)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceLineInfo)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::SourceLineBlock)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::InlineeInfo)
-LLVM_YAML_DECLARE_MAPPING_TRAITS(CodeViewYAML::InlineeSite)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(SourceLineEntry)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(SourceColumnEntry)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(SourceFileChecksumEntry)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(SourceLineBlock)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(InlineeSite)
+
+namespace llvm {
+namespace CodeViewYAML {
+namespace detail {
+struct YAMLSubsectionBase {
+  explicit YAMLSubsectionBase(DebugSubsectionKind Kind) : Kind(Kind) {}
+  DebugSubsectionKind Kind;
+  virtual ~YAMLSubsectionBase() {}
+
+  virtual void map(IO &IO) = 0;
+  virtual std::unique_ptr<DebugSubsection>
+  toCodeViewSubsection(DebugStringTableSubsection *UseStrings,
+                       DebugChecksumsSubsection *UseChecksums) const = 0;
+};
+}
+}
+}
+
+namespace {
+struct YAMLChecksumsSubsection : public YAMLSubsectionBase {
+  YAMLChecksumsSubsection()
+      : YAMLSubsectionBase(DebugSubsectionKind::FileChecksums) {}
+
+  void map(IO &IO) override;
+  std::unique_ptr<DebugSubsection>
+  toCodeViewSubsection(DebugStringTableSubsection *Strings,
+                       DebugChecksumsSubsection *Checksums) const override;
+  static Expected<std::shared_ptr<YAMLChecksumsSubsection>>
+  fromCodeViewSubsection(const DebugStringTableSubsectionRef &Strings,
+                         const DebugChecksumsSubsectionRef &FC);
+
+  std::vector<SourceFileChecksumEntry> Checksums;
+};
+
+struct YAMLLinesSubsection : public YAMLSubsectionBase {
+  YAMLLinesSubsection() : YAMLSubsectionBase(DebugSubsectionKind::Lines) {}
+
+  void map(IO &IO) override;
+  std::unique_ptr<DebugSubsection>
+  toCodeViewSubsection(DebugStringTableSubsection *Strings,
+                       DebugChecksumsSubsection *Checksums) const override;
+  static Expected<std::shared_ptr<YAMLLinesSubsection>>
+  fromCodeViewSubsection(const DebugStringTableSubsectionRef &Strings,
+                         const DebugChecksumsSubsectionRef &Checksums,
+                         const DebugLinesSubsectionRef &Lines);
+
+  SourceLineInfo Lines;
+};
+
+struct YAMLInlineeLinesSubsection : public YAMLSubsectionBase {
+  YAMLInlineeLinesSubsection()
+      : YAMLSubsectionBase(DebugSubsectionKind::InlineeLines) {}
+
+  void map(IO &IO) override;
+  std::unique_ptr<DebugSubsection>
+  toCodeViewSubsection(DebugStringTableSubsection *Strings,
+                       DebugChecksumsSubsection *Checksums) const override;
+  static Expected<std::shared_ptr<YAMLInlineeLinesSubsection>>
+  fromCodeViewSubsection(const DebugStringTableSubsectionRef &Strings,
+                         const DebugChecksumsSubsectionRef &Checksums,
+                         const DebugInlineeLinesSubsectionRef &Lines);
+
+  InlineeInfo InlineeLines;
+};
+}
 
 void ScalarBitSetTraits<LineFlags>::bitset(IO &io, LineFlags &Flags) {
   io.bitSetCase(Flags, "HasColumnInfo", LF_HaveColumns);
   io.enumFallback<Hex16>(Flags);
 }
 
 void ScalarEnumerationTraits<FileChecksumKind>::enumeration(
     IO &io, FileChecksumKind &Kind) {

 
 void MappingTraits<SourceFileChecksumEntry>::mapping(
     IO &IO, SourceFileChecksumEntry &Obj) {
   IO.mapRequired("FileName", Obj.FileName);
   IO.mapRequired("Kind", Obj.Kind);
   IO.mapRequired("Checksum", Obj.ChecksumBytes);
 }
 
-void MappingTraits<SourceLineInfo>::mapping(IO &IO, SourceLineInfo &Obj) {
-  IO.mapRequired("CodeSize", Obj.CodeSize);
-
-  IO.mapRequired("Flags", Obj.Flags);
-  IO.mapRequired("RelocOffset", Obj.RelocOffset);
-  IO.mapRequired("RelocSegment", Obj.RelocSegment);
-  IO.mapRequired("Blocks", Obj.Blocks);
-}
-
-void MappingTraits<SourceFileInfo>::mapping(IO &IO, SourceFileInfo &Obj) {
-  IO.mapOptional("Checksums", Obj.FileChecksums);
-  IO.mapOptional("Lines", Obj.LineFragments);
-  IO.mapOptional("InlineeLines", Obj.Inlinees);
-}
-
 void MappingTraits<InlineeSite>::mapping(IO &IO, InlineeSite &Obj) {
   IO.mapRequired("FileName", Obj.FileName);
   IO.mapRequired("LineNum", Obj.SourceLineNum);
   IO.mapRequired("Inlinee", Obj.Inlinee);
   IO.mapOptional("ExtraFiles", Obj.ExtraFiles);
 }
 
-void MappingTraits<InlineeInfo>::mapping(IO &IO, InlineeInfo &Obj) {
-  IO.mapRequired("HasExtraFiles", Obj.HasExtraFiles);
-  IO.mapRequired("Sites", Obj.Sites);
+void YAMLChecksumsSubsection::map(IO &IO) {
+  IO.mapTag("!FileChecksums", true);
+  IO.mapRequired("Checksums", Checksums);
+}
+
+void YAMLLinesSubsection::map(IO &IO) {
+  IO.mapTag("!Lines", true);
+  IO.mapRequired("CodeSize", Lines.CodeSize);
+
+  IO.mapRequired("Flags", Lines.Flags);
+  IO.mapRequired("RelocOffset", Lines.RelocOffset);
+  IO.mapRequired("RelocSegment", Lines.RelocSegment);
+  IO.mapRequired("Blocks", Lines.Blocks);
+}
+
+void YAMLInlineeLinesSubsection::map(IO &IO) {
+  IO.mapTag("!InlineeLines", true);
+  IO.mapRequired("HasExtraFiles", InlineeLines.HasExtraFiles);
+  IO.mapRequired("Sites", InlineeLines.Sites);
+}
+
+void MappingTraits<YAMLDebugSubsection>::mapping(
+    IO &IO, YAMLDebugSubsection &Subsection) {
+  if (!IO.outputting()) {
+    if (IO.mapTag("!FileChecksums")) {
+      auto SS = std::make_shared<YAMLChecksumsSubsection>();
+      Subsection.Subsection = SS;
+    } else if (IO.mapTag("!Lines")) {
+      Subsection.Subsection = std::make_shared<YAMLLinesSubsection>();
+    } else if (IO.mapTag("!InlineeLines")) {
+      Subsection.Subsection = std::make_shared<YAMLInlineeLinesSubsection>();
+    } else {
+      llvm_unreachable("Unexpected subsection tag!");
+    }
+  }
+  Subsection.Subsection->map(IO);
+}
+
+static Expected<const YAMLChecksumsSubsection &>
+findChecksums(ArrayRef<YAMLDebugSubsection> Subsections) {
+  for (const auto &SS : Subsections) {
+    if (SS.Subsection->Kind == DebugSubsectionKind::FileChecksums) {
+      return static_cast<const YAMLChecksumsSubsection &>(*SS.Subsection);
+    }
+  }
+  return make_error<CodeViewError>(cv_error_code::no_records);
+}
+
+std::unique_ptr<DebugSubsection> YAMLChecksumsSubsection::toCodeViewSubsection(
+    DebugStringTableSubsection *UseStrings,
+    DebugChecksumsSubsection *UseChecksums) const {
+  assert(UseStrings && !UseChecksums);
+  auto Result = llvm::make_unique<DebugChecksumsSubsection>(*UseStrings);
+  for (const auto &CS : Checksums) {
+    Result->addChecksum(CS.FileName, CS.Kind, CS.ChecksumBytes.Bytes);
+  }
+  return std::move(Result);
+}
+
+std::unique_ptr<DebugSubsection> YAMLLinesSubsection::toCodeViewSubsection(
+    DebugStringTableSubsection *UseStrings,
+    DebugChecksumsSubsection *UseChecksums) const {
+  assert(UseStrings && UseChecksums);
+  auto Result =
+      llvm::make_unique<DebugLinesSubsection>(*UseChecksums, *UseStrings);
+  Result->setCodeSize(Lines.CodeSize);
+  Result->setRelocationAddress(Lines.RelocSegment, Lines.RelocOffset);
+  Result->setFlags(Lines.Flags);
+  for (const auto &LC : Lines.Blocks) {
+    Result->createBlock(LC.FileName);
+    if (Result->hasColumnInfo()) {
+      for (const auto &Item : zip(LC.Lines, LC.Columns)) {
+        auto &L = std::get<0>(Item);
+        auto &C = std::get<1>(Item);
+        uint32_t LE = L.LineStart + L.EndDelta;
+        Result->addLineAndColumnInfo(L.Offset,
+                                     LineInfo(L.LineStart, LE, L.IsStatement),
+                                     C.StartColumn, C.EndColumn);
+      }
+    } else {
+      for (const auto &L : LC.Lines) {
+        uint32_t LE = L.LineStart + L.EndDelta;
+        Result->addLineInfo(L.Offset, LineInfo(L.LineStart, LE, L.IsStatement));
+      }
+    }
+  }
+  return Result;
+}
+
+std::unique_ptr<DebugSubsection>
+YAMLInlineeLinesSubsection::toCodeViewSubsection(
+    DebugStringTableSubsection *UseStrings,
+    DebugChecksumsSubsection *UseChecksums) const {
+  assert(UseChecksums);
+  auto Result = llvm::make_unique<DebugInlineeLinesSubsection>(
+      *UseChecksums, InlineeLines.HasExtraFiles);
+
+  for (const auto &Site : InlineeLines.Sites) {
+    Result->addInlineSite(TypeIndex(Site.Inlinee), Site.FileName,
+                          Site.SourceLineNum);
+    if (!InlineeLines.HasExtraFiles)
+      continue;
+
+    for (auto EF : Site.ExtraFiles) {
+      Result->addExtraFile(EF);
+    }
+  }
+  return Result;
+}
+
+static Expected<SourceFileChecksumEntry>
+convertOneChecksum(const DebugStringTableSubsectionRef &Strings,
+                   const FileChecksumEntry &CS) {
+  auto ExpectedString = Strings.getString(CS.FileNameOffset);
+  if (!ExpectedString)
+    return ExpectedString.takeError();
+
+  SourceFileChecksumEntry Result;
+  Result.ChecksumBytes.Bytes = CS.Checksum;
+  Result.Kind = CS.Kind;
+  Result.FileName = *ExpectedString;
+  return Result;
+}
+
+static Expected<StringRef>
+getFileName(const DebugStringTableSubsectionRef &Strings,
+            const DebugChecksumsSubsectionRef &Checksums, uint32_t FileID) {
+  auto Iter = Checksums.getArray().at(FileID);
+  if (Iter == Checksums.getArray().end())
+    return make_error<CodeViewError>(cv_error_code::no_records);
+  uint32_t Offset = Iter->FileNameOffset;
+  return Strings.getString(Offset);
+}
+
+Expected<std::shared_ptr<YAMLChecksumsSubsection>>
+YAMLChecksumsSubsection::fromCodeViewSubsection(
+    const DebugStringTableSubsectionRef &Strings,
+    const DebugChecksumsSubsectionRef &FC) {
+  auto Result = std::make_shared<YAMLChecksumsSubsection>();
+
+  for (const auto &CS : FC) {
+    auto ConvertedCS = convertOneChecksum(Strings, CS);
+    if (!ConvertedCS)
+      return ConvertedCS.takeError();
+    Result->Checksums.push_back(*ConvertedCS);
+  }
+  return Result;
+}
+
+Expected<std::shared_ptr<YAMLLinesSubsection>>
+YAMLLinesSubsection::fromCodeViewSubsection(
+    const DebugStringTableSubsectionRef &Strings,
+    const DebugChecksumsSubsectionRef &Checksums,
+    const DebugLinesSubsectionRef &Lines) {
+  auto Result = std::make_shared<YAMLLinesSubsection>();
+  Result->Lines.CodeSize = Lines.header()->CodeSize;
+  Result->Lines.RelocOffset = Lines.header()->RelocOffset;
+  Result->Lines.RelocSegment = Lines.header()->RelocSegment;
+  Result->Lines.Flags = static_cast<LineFlags>(uint16_t(Lines.header()->Flags));
+  for (const auto &L : Lines) {
+    SourceLineBlock Block;
+    auto EF = getFileName(Strings, Checksums, L.NameIndex);
+    if (!EF)
+      return EF.takeError();
+    Block.FileName = *EF;
+    if (Lines.hasColumnInfo()) {
+      for (const auto &C : L.Columns) {
+        SourceColumnEntry SCE;
+        SCE.EndColumn = C.EndColumn;
+        SCE.StartColumn = C.StartColumn;
+        Block.Columns.push_back(SCE);
+      }
+    }
+    for (const auto &LN : L.LineNumbers) {
+      SourceLineEntry SLE;
+      LineInfo LI(LN.Flags);
+      SLE.Offset = LN.Offset;
+      SLE.LineStart = LI.getStartLine();
+      SLE.EndDelta = LI.getLineDelta();
+      SLE.IsStatement = LI.isStatement();
+      Block.Lines.push_back(SLE);
+    }
+    Result->Lines.Blocks.push_back(Block);
+  }
+  return Result;
+}
+
+Expected<std::shared_ptr<YAMLInlineeLinesSubsection>>
+YAMLInlineeLinesSubsection::fromCodeViewSubsection(
+    const DebugStringTableSubsectionRef &Strings,
+    const DebugChecksumsSubsectionRef &Checksums,
+    const DebugInlineeLinesSubsectionRef &Lines) {
+  auto Result = std::make_shared<YAMLInlineeLinesSubsection>();
+
+  Result->InlineeLines.HasExtraFiles = Lines.hasExtraFiles();
+  for (const auto &IL : Lines) {
+    InlineeSite Site;
+    auto ExpF = getFileName(Strings, Checksums, IL.Header->FileID);
+    if (!ExpF)
+      return ExpF.takeError();
+    Site.FileName = *ExpF;
+    Site.Inlinee = IL.Header->Inlinee.getIndex();
+    Site.SourceLineNum = IL.Header->SourceLineNum;
+    if (Lines.hasExtraFiles()) {
+      for (const auto EF : IL.ExtraFiles) {
+        auto ExpF2 = getFileName(Strings, Checksums, EF);
+        if (!ExpF2)
+          return ExpF2.takeError();
+        Site.ExtraFiles.push_back(*ExpF2);
+      }
+    }
+    Result->InlineeLines.Sites.push_back(Site);
+  }
+  return Result;
+}
+
+Expected<std::vector<std::unique_ptr<DebugSubsection>>>
+llvm::CodeViewYAML::convertSubsectionList(
+    ArrayRef<YAMLDebugSubsection> Subsections,
+    DebugStringTableSubsection &Strings) {
+  std::vector<std::unique_ptr<DebugSubsection>> Result;
+  if (Subsections.empty())
+    return Result;
+
+  auto Checksums = findChecksums(Subsections);
+  if (!Checksums)
+    return Checksums.takeError();
+  auto ChecksumsBase = Checksums->toCodeViewSubsection(&Strings, nullptr);
+  DebugChecksumsSubsection &CS =
+      llvm::cast<DebugChecksumsSubsection>(*ChecksumsBase);
+  for (const auto &SS : Subsections) {
+    // We've already converted the checksums subsection, don't do it
+    // twice.
+    std::unique_ptr<DebugSubsection> CVS;
+    if (SS.Subsection->Kind == DebugSubsectionKind::FileChecksums)
+      CVS = std::move(ChecksumsBase);
+    else
+      CVS = SS.Subsection->toCodeViewSubsection(&Strings, &CS);
+    Result.push_back(std::move(CVS));
+  }
+  return std::move(Result);
+}
+
+namespace {
+struct SubsectionConversionVisitor : public DebugSubsectionVisitor {
+  explicit SubsectionConversionVisitor(
+      const DebugStringTableSubsectionRef &Strings,
+      const DebugChecksumsSubsectionRef &Checksums)
+      : Strings(Strings), Checksums(Checksums) {}
+
+  Error visitUnknown(DebugUnknownSubsectionRef &Unknown) override;
+  Error visitLines(DebugLinesSubsectionRef &Lines) override;
+  Error visitFileChecksums(DebugChecksumsSubsectionRef &Checksums) override;
+  Error visitInlineeLines(DebugInlineeLinesSubsectionRef &Inlinees) override;
+
+  YAMLDebugSubsection Subsection;
+
+private:
+  const DebugStringTableSubsectionRef &Strings;
+  const DebugChecksumsSubsectionRef &Checksums;
+};
+
+Error SubsectionConversionVisitor::visitUnknown(
+    DebugUnknownSubsectionRef &Unknown) {
+  return make_error<CodeViewError>(cv_error_code::operation_unsupported);
+}
+
+Error SubsectionConversionVisitor::visitLines(DebugLinesSubsectionRef &Lines) {
+  auto Result =
+      YAMLLinesSubsection::fromCodeViewSubsection(Strings, Checksums, Lines);
+  if (!Result)
+    return Result.takeError();
+  Subsection.Subsection = *Result;
+  return Error::success();
+}
+
+Error SubsectionConversionVisitor::visitFileChecksums(
+    DebugChecksumsSubsectionRef &Checksums) {
+  auto Result =
+      YAMLChecksumsSubsection::fromCodeViewSubsection(Strings, Checksums);
+  if (!Result)
+    return Result.takeError();
+  Subsection.Subsection = *Result;
+  return Error::success();
+}
+
+Error SubsectionConversionVisitor::visitInlineeLines(
+    DebugInlineeLinesSubsectionRef &Inlinees) {
+  auto Result = YAMLInlineeLinesSubsection::fromCodeViewSubsection(
+      Strings, Checksums, Inlinees);
+  if (!Result)
+    return Result.takeError();
+  Subsection.Subsection = *Result;
+  return Error::success();
+}
+}
+
+Expected<YAMLDebugSubsection> YAMLDebugSubsection::fromCodeViewSubection(
+    const DebugStringTableSubsectionRef &Strings,
+    const DebugChecksumsSubsectionRef &Checksums,
+    const DebugSubsectionRecord &SS) {
+  SubsectionConversionVisitor V(Strings, Checksums);
+  if (auto EC = visitDebugSubsection(SS, V))
+    return std::move(EC);
+
+  return V.Subsection;
 }

llvm/trunk/test/DebugInfo/PDB/Inputs/simple-line-info.yaml

 ---
 DbiStream:
   Modules:
     - Module:          'd:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj'
       ObjFile:         'd:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj'
       SourceFiles:
         - 'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
-      LineInfo:
+      Subsections:
+        - !FileChecksums
           Checksums:
             - FileName:        'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
               Kind:            MD5
               Checksum:        A0A5BD0D3ECD93FC29D19DE826FBF4BC
             - FileName:        'f:\dd\externalapis\windows\10\sdk\inc\winerror.h'
               Kind:            MD5
               Checksum:        1154D69F5B2650196E1FC34F4134E56B
-        Lines:
-          - CodeSize:        10
-            Flags:           [  ]
-            RelocOffset:     16
-            RelocSegment:    1
-            Blocks:
-              - FileName:        'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
-                Lines:
-                  - Offset:          0
-                    LineStart:       5
-                    IsStatement:     true
-                    EndDelta:        0
-                  - Offset:          3
-                    LineStart:       6
-                    IsStatement:     true
-                    EndDelta:        0
-                  - Offset:          8
-                    LineStart:       7
-                    IsStatement:     true
-                    EndDelta:        0
-                Columns:
-        InlineeLines:    
-          - HasExtraFiles:   false
-            Sites:           
-              - FileName:        'f:\dd\externalapis\windows\10\sdk\inc\winerror.h'
-                LineNum:         26950
-                Inlinee:         22767
+        - !Lines
+          CodeSize:        10
+        - !InlineeLines
+          HasExtraFiles:   false
 ...

llvm/trunk/test/DebugInfo/PDB/pdbdump-yaml-lineinfo.test

 YAML:   PdbDllRbld:      0
 YAML:   Flags:           1
 YAML:   MachineType:     x86
 YAML:   Modules:
 YAML:   - Module:          'd:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj'
 YAML:     ObjFile:         'd:\src\llvm\test\DebugInfo\PDB\Inputs\empty.obj'
 YAML:     SourceFiles:
 YAML:       - 'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
-YAML:     LineInfo:
-YAML:       Checksums:
-YAML:         - FileName:        'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
-YAML:           Kind:            MD5
-YAML:           Checksum:        A0A5BD0D3ECD93FC29D19DE826FBF4BC
-YAML:       Lines:
+YAML:     Subsections:
+YAML:       - !Lines
 YAML:         CodeSize:        10
 YAML:         Flags:           [  ]
 YAML:         RelocOffset:     16
 YAML:         RelocSegment:    1
 YAML:         Blocks:
 YAML:           - FileName:        'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
 YAML:             Lines:
 YAML:               - Offset:          0
 YAML:                 LineStart:       5
 YAML:                 IsStatement:     true
 YAML:                 EndDelta:        0
 YAML:               - Offset:          3
 YAML:                 LineStart:       6
 YAML:                 IsStatement:     true
 YAML:                 EndDelta:        0
 YAML:               - Offset:          8
 YAML:                 LineStart:       7
 YAML:                 IsStatement:     true
 YAML:                 EndDelta:        0
 YAML:             Columns:
+YAML:       - !FileChecksums
+YAML:         Checksums:
+YAML:           - FileName:        'd:\src\llvm\test\debuginfo\pdb\inputs\empty.cpp'
+YAML:             Kind:            MD5
+YAML:             Checksum:        A0A5BD0D3ECD93FC29D19DE826FBF4BC
 YAML:  - Module:          '* Linker *'
 YAML:    ObjFile:         ''
 YAML: ...
 No newline at end of file

llvm/trunk/tools/llvm-pdbdump/LLVMOutputStyle.cpp

for (uint32_t I = 0; I < Modules.getModuleCount(); ++I) {
         }
         if (opts::raw::DumpLineInfo) {
           ListScope SS(P, "LineInfo");
           auto ExpectedTypes = initializeTypeDatabase(StreamIPI);
           if (!ExpectedTypes)
             return ExpectedTypes.takeError();
           auto &IpiItems = *ExpectedTypes;
           C13RawVisitor V(P, File, IpiItems);
-          if (auto EC =
-                  codeview::visitDebugSubsections(ModS.linesAndChecksums(), V))
+          if (auto EC = codeview::visitDebugSubsections(ModS.subsections(), V))
             return EC;
         }
       }
     }
   }
   return Error::success();
 }
 

llvm/trunk/tools/llvm-pdbdump/PdbYaml.h

 #include "llvm/ObjectYAML/CodeViewYAMLSymbols.h"
 #include "llvm/ObjectYAML/CodeViewYAMLTypes.h"
 #include "llvm/Support/Endian.h"
 #include "llvm/Support/YAMLTraits.h"
 
 #include <vector>
 
 namespace llvm {
+namespace codeview {
+class DebugStringTableSubsection;
+}
 namespace pdb {
 
 namespace yaml {
 struct SerializationContext;
 
 struct MSFHeaders {
   msf::SuperBlock SuperBlock;
   uint32_t NumDirectoryBlocks = 0;
struct PdbModiStream {
   uint32_t Signature;
   std::vector<CodeViewYAML::SymbolRecord> Symbols;
 };
 
 struct PdbDbiModuleInfo {
   StringRef Obj;
   StringRef Mod;
   std::vector<StringRef> SourceFiles;
-  Optional<CodeViewYAML::SourceFileInfo> FileLineInfo;
+  std::vector<CodeViewYAML::YAMLDebugSubsection> Subsections;
   Optional<PdbModiStream> Modi;
 };
 
 struct PdbDbiStream {
   PdbRaw_DbiVer VerHeader = PdbDbiV70;
   uint32_t Age = 1;
   uint16_t BuildNumber = 0;
   uint32_t PdbDllVersion = 0;

llvm/trunk/tools/llvm-pdbdump/PdbYaml.cpp

 //===- PdbYAML.cpp -------------------------------------------- *- C++ --*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "PdbYaml.h"
 
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/DebugInfo/CodeView/CVSymbolVisitor.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
+#include "llvm/DebugInfo/CodeView/DebugStringTableSubsection.h"
 #include "llvm/DebugInfo/CodeView/SymbolDeserializer.h"
 #include "llvm/DebugInfo/CodeView/SymbolVisitorCallbackPipeline.h"
 #include "llvm/DebugInfo/CodeView/TypeSerializer.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbackPipeline.h"
 #include "llvm/DebugInfo/PDB/Native/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Native/RawTypes.h"
 #include "llvm/DebugInfo/PDB/Native/TpiHashing.h"
 #include "llvm/DebugInfo/PDB/PDBExtras.h"
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
+#include "llvm/ObjectYAML/CodeViewYAMLDebugSections.h"
 #include "llvm/ObjectYAML/CodeViewYAMLTypes.h"
 
 using namespace llvm;
 using namespace llvm::pdb;
 using namespace llvm::pdb::yaml;
 using namespace llvm::yaml;
 
 LLVM_YAML_IS_FLOW_SEQUENCE_VECTOR(uint32_t)
void MappingTraits<PdbModiStream>::mapping(IO &IO, PdbModiStream &Obj) {
   IO.mapOptional("Signature", Obj.Signature, 4U);
   IO.mapRequired("Records", Obj.Symbols);
 }
 
 void MappingTraits<PdbDbiModuleInfo>::mapping(IO &IO, PdbDbiModuleInfo &Obj) {
   IO.mapRequired("Module", Obj.Mod);
   IO.mapOptional("ObjFile", Obj.Obj, Obj.Mod);
   IO.mapOptional("SourceFiles", Obj.SourceFiles);
-  IO.mapOptional("LineInfo", Obj.FileLineInfo);
+  IO.mapOptional("Subsections", Obj.Subsections);
   IO.mapOptional("Modi", Obj.Modi);
 }

llvm/trunk/tools/llvm-pdbdump/YAMLOutputStyle.h

 
 class YAMLOutputStyle : public OutputStyle {
 public:
   YAMLOutputStyle(PDBFile &File);
 
   Error dump() override;
 
 private:
-  Expected<Optional<CodeViewYAML::SourceFileInfo>>
-  getFileLineInfo(const pdb::ModuleDebugStreamRef &ModS);
-
   Error dumpStringTable();
   Error dumpFileHeaders();
   Error dumpStreamMetadata();
   Error dumpStreamDirectory();
   Error dumpPDBStream();
   Error dumpDbiStream();
   Error dumpTpiStream();
   Error dumpIpiStream();

llvm/trunk/tools/llvm-pdbdump/YAMLOutputStyle.cpp

Error YAMLOutputStyle::dump() {
 
   if (auto EC = dumpIpiStream())
     return EC;
 
   flush();
   return Error::success();
 }
 
-namespace {
-class C13YamlVisitor : public C13DebugFragmentVisitor {
-public:
-  C13YamlVisitor(CodeViewYAML::SourceFileInfo &Info, PDBFile &F)
-      : C13DebugFragmentVisitor(F), Info(Info) {}
-
-  Error handleFileChecksums() override {
-    for (const auto &C : *Checksums) {
-      CodeViewYAML::SourceFileChecksumEntry Entry;
-      if (auto Result = getNameFromStringTable(C.FileNameOffset))
-        Entry.FileName = *Result;
-      else
-        return Result.takeError();
-
-      Entry.Kind = C.Kind;
-      Entry.ChecksumBytes.Bytes = C.Checksum;
-      Info.FileChecksums.push_back(Entry);
-    }
-    return Error::success();
-  }
-
-  Error handleLines() override {
-    for (const auto &LF : Lines) {
-      Info.LineFragments.emplace_back();
-      auto &Fragment = Info.LineFragments.back();
-
-      Fragment.CodeSize = LF.header()->CodeSize;
-      Fragment.Flags =
-          static_cast<codeview::LineFlags>(uint16_t(LF.header()->Flags));
-      Fragment.RelocOffset = LF.header()->RelocOffset;
-      Fragment.RelocSegment = LF.header()->RelocSegment;
-
-      for (const auto &L : LF) {
-        Fragment.Blocks.emplace_back();
-        auto &Block = Fragment.Blocks.back();
-
-        if (auto Result = getNameFromChecksumsBuffer(L.NameIndex))
-          Block.FileName = *Result;
-        else
-          return Result.takeError();
-
-        for (const auto &N : L.LineNumbers) {
-          CodeViewYAML::SourceLineEntry Line;
-          Line.Offset = N.Offset;
-          codeview::LineInfo LI(N.Flags);
-          Line.LineStart = LI.getStartLine();
-          Line.EndDelta = LI.getLineDelta();
-          Line.IsStatement = LI.isStatement();
-          Block.Lines.push_back(Line);
-        }
-
-        if (LF.hasColumnInfo()) {
-          for (const auto &C : L.Columns) {
-            CodeViewYAML::SourceColumnEntry Column;
-            Column.StartColumn = C.StartColumn;
-            Column.EndColumn = C.EndColumn;
-            Block.Columns.push_back(Column);
-          }
-        }
-      }
-    }
-    return Error::success();
-  }
-
-  Error handleInlineeLines() override {
-    for (const auto &ILF : InlineeLines) {
-      Info.Inlinees.emplace_back();
-      auto &Inlinee = Info.Inlinees.back();
-
-      Inlinee.HasExtraFiles = ILF.hasExtraFiles();
-      for (const auto &IL : ILF) {
-        Inlinee.Sites.emplace_back();
-        auto &Site = Inlinee.Sites.back();
-        if (auto Result = getNameFromChecksumsBuffer(IL.Header->FileID))
-          Site.FileName = *Result;
-        else
-          return Result.takeError();
-
-        Site.Inlinee = IL.Header->Inlinee.getIndex();
-        Site.SourceLineNum = IL.Header->SourceLineNum;
-        if (ILF.hasExtraFiles()) {
-          for (const auto &EF : IL.ExtraFiles) {
-            if (auto Result = getNameFromChecksumsBuffer(EF))
-              Site.ExtraFiles.push_back(*Result);
-            else
-              return Result.takeError();
-          }
-        }
-      }
-    }
-    return Error::success();
-  }
-
-private:
-  CodeViewYAML::SourceFileInfo &Info;
-};
-}
-
-Expected<Optional<CodeViewYAML::SourceFileInfo>>
-YAMLOutputStyle::getFileLineInfo(const pdb::ModuleDebugStreamRef &ModS) {
-  if (!ModS.hasLineInfo())
-    return None;
-
-  CodeViewYAML::SourceFileInfo Info;
-  C13YamlVisitor Visitor(Info, File);
-  if (auto EC =
-          codeview::visitDebugSubsections(ModS.linesAndChecksums(), Visitor))
-    return std::move(EC);
-
-  return Info;
-}
 
 Error YAMLOutputStyle::dumpFileHeaders() {
   if (opts::pdb2yaml::NoFileHeaders)
     return Error::success();
 
   yaml::MSFHeaders Headers;
   Obj.Headers.emplace();
   Obj.Headers->SuperBlock.NumBlocks = File.getBlockCount();
   Obj.Headers->SuperBlock.BlockMapAddr = File.getBlockMapIndex();
   Obj.Headers->SuperBlock.BlockSize = File.getBlockSize();
   auto Blocks = File.getDirectoryBlockArray();
   Obj.Headers->DirectoryBlocks.assign(Blocks.begin(), Blocks.end());
   Obj.Headers->NumDirectoryBlocks = File.getNumDirectoryBlocks();
   Obj.Headers->SuperBlock.NumDirectoryBytes = File.getNumDirectoryBytes();
   Obj.Headers->NumStreams =
       opts::pdb2yaml::StreamMetadata ? File.getNumStreams() : 0;
   Obj.Headers->SuperBlock.FreeBlockMapBlock = File.getFreeBlockMapBlock();
   Obj.Headers->SuperBlock.Unknown1 = File.getUnknown1();
   Obj.Headers->FileSize = File.getFileSize();
 
   return Error::success();
 }
 
 Error YAMLOutputStyle::dumpStringTable() {
-  if (!opts::pdb2yaml::StringTable)
+  bool RequiresStringTable = opts::pdb2yaml::DbiModuleSourceFileInfo ||
+                             opts::pdb2yaml::DbiModuleSourceLineInfo;
+  bool RequestedStringTable = opts::pdb2yaml::StringTable;
+  if (!RequiresStringTable && !RequestedStringTable)
     return Error::success();
 
-  Obj.StringTable.emplace();
   auto ExpectedST = File.getStringTable();
   if (!ExpectedST)
     return ExpectedST.takeError();
 
+  Obj.StringTable.emplace();
   const auto &ST = ExpectedST.get();
   for (auto ID : ST.name_ids()) {
     auto S = ST.getStringForID(ID);
     if (!S)
       return S.takeError();
     if (S->empty())
       continue;
     Obj.StringTable->push_back(*S);
for (uint32_t I = 0; I < Modules.getModuleCount(); ++I) {
 
       auto ModStreamData = msf::MappedBlockStream::createIndexedStream(
           File.getMsfLayout(), File.getMsfBuffer(), ModiStream);
 
       pdb::ModuleDebugStreamRef ModS(MI, std::move(ModStreamData));
       if (auto EC = ModS.reload())
         return EC;
 
-      if (opts::pdb2yaml::DbiModuleSourceLineInfo) {
-        auto ExpectedInfo = getFileLineInfo(ModS);
-        if (!ExpectedInfo)
-          return ExpectedInfo.takeError();
-        DMI.FileLineInfo = *ExpectedInfo;
+      auto ExpectedST = File.getStringTable();
+      if (!ExpectedST)
+        return ExpectedST.takeError();
+      if (opts::pdb2yaml::DbiModuleSourceLineInfo &&
+          ModS.hasDebugSubsections()) {
+        auto ExpectedChecksums = ModS.findChecksumsSubsection();
+        if (!ExpectedChecksums)
+          return ExpectedChecksums.takeError();
+
+        for (const auto &SS : ModS.subsections()) {
+          auto Converted =
+              CodeViewYAML::YAMLDebugSubsection::fromCodeViewSubection(
+                  ExpectedST->getStringTable(), *ExpectedChecksums, SS);
+          if (!Converted)
+            return Converted.takeError();
+          DMI.Subsections.push_back(*Converted);
+        }
       }
 
       if (opts::pdb2yaml::DbiModuleSyms) {
         DMI.Modi.emplace();
 
         DMI.Modi->Signature = ModS.signature();
         bool HadError = false;
         for (auto &Sym : ModS.symbols(&HadError)) {

llvm/trunk/tools/llvm-pdbdump/llvm-pdbdump.cpp

static void yamlToPdb(StringRef Path) {
 
   if (ErrorOrBuffer.getError()) {
     ExitOnErr(make_error<GenericError>(generic_error_code::invalid_path, Path));
   }
 
   std::unique_ptr<MemoryBuffer> &Buffer = ErrorOrBuffer.get();
 
   llvm::yaml::Input In(Buffer->getBuffer());
-  In.setContext(&Allocator);
   pdb::yaml::PdbObject YamlObj(Allocator);
   In >> YamlObj;
 
   PDBFileBuilder Builder(Allocator);
 
   uint32_t BlockSize = 4096;
   if (YamlObj.Headers.hasValue())
     BlockSize = YamlObj.Headers->SuperBlock.BlockSize;
for (auto S : MI.SourceFiles)
       ExitOnErr(DbiBuilder.addModuleSourceFile(MI.Mod, S));
     if (MI.Modi.hasValue()) {
       const auto &ModiStream = *MI.Modi;
       for (auto Symbol : ModiStream.Symbols) {
         ModiBuilder.addSymbol(
             Symbol.toCodeViewSymbol(Allocator, CodeViewContainer::Pdb));
       }
     }
-    if (MI.FileLineInfo.hasValue()) {
-      const auto &FLI = *MI.FileLineInfo;
 
-      // File Checksums must be emitted before line information, because line
-      // info records use offsets into the checksum buffer to reference a file's
-      // source file name.
-      auto Checksums = llvm::make_unique<DebugChecksumsSubsection>(Strings);
-      auto &ChecksumRef = *Checksums;
-      if (!FLI.FileChecksums.empty()) {
-        for (auto &FC : FLI.FileChecksums)
-          Checksums->addChecksum(FC.FileName, FC.Kind, FC.ChecksumBytes.Bytes);
-      }
-      ModiBuilder.setC13FileChecksums(std::move(Checksums));
-
-      for (const auto &Fragment : FLI.LineFragments) {
-        auto Lines =
-            llvm::make_unique<DebugLinesSubsection>(ChecksumRef, Strings);
-        Lines->setCodeSize(Fragment.CodeSize);
-        Lines->setRelocationAddress(Fragment.RelocSegment,
-                                    Fragment.RelocOffset);
-        Lines->setFlags(Fragment.Flags);
-        for (const auto &LC : Fragment.Blocks) {
-          Lines->createBlock(LC.FileName);
-          if (Lines->hasColumnInfo()) {
-            for (const auto &Item : zip(LC.Lines, LC.Columns)) {
-              auto &L = std::get<0>(Item);
-              auto &C = std::get<1>(Item);
-              uint32_t LE = L.LineStart + L.EndDelta;
-              Lines->addLineAndColumnInfo(
-                  L.Offset, LineInfo(L.LineStart, LE, L.IsStatement),
-                  C.StartColumn, C.EndColumn);
-            }
-          } else {
-            for (const auto &L : LC.Lines) {
-              uint32_t LE = L.LineStart + L.EndDelta;
-              Lines->addLineInfo(L.Offset,
-                                 LineInfo(L.LineStart, LE, L.IsStatement));
-            }
-          }
-        }
-        ModiBuilder.addC13Fragment(std::move(Lines));
-      }
-
-      for (const auto &Inlinee : FLI.Inlinees) {
-        auto Inlinees = llvm::make_unique<DebugInlineeLinesSubsection>(
-            ChecksumRef, Inlinee.HasExtraFiles);
-        for (const auto &Site : Inlinee.Sites) {
-          Inlinees->addInlineSite(TypeIndex(Site.Inlinee), Site.FileName,
-                                  Site.SourceLineNum);
-          if (!Inlinee.HasExtraFiles)
-            continue;
-
-          for (auto EF : Site.ExtraFiles) {
-            Inlinees->addExtraFile(EF);
-          }
-        }
-        ModiBuilder.addC13Fragment(std::move(Inlinees));
-      }
+    auto CodeViewSubsections =
+        ExitOnErr(CodeViewYAML::convertSubsectionList(MI.Subsections, Strings));
+    for (auto &SS : CodeViewSubsections) {
+      ModiBuilder.addDebugSubsection(std::move(SS));
     }
   }
 
   auto &TpiBuilder = Builder.getTpiBuilder();
   const auto &Tpi = YamlObj.TpiStream.getValueOr(DefaultTpiStream);
   TpiBuilder.setVersionHeader(Tpi.Version);
   for (const auto &R : Tpi.Records) {
     CVType Type = R.toCodeViewRecord(Allocator);