[CodeView] Re-write TypeSerializer and TypeTableBuilder

[zturner]

Summary

The motivation behind this patch is that future directions require us to be able to compute the hash value of records independently of actually using them for de-duplication.

The current structure of TypeSerializer / TypeTableBuilder being a single entry point that takes an unserialized type record, and then hashes and de-duplicates it is not flexible enough to allow this.

At the same time, the existing TypeSerializer is already extremely complex for this very reason -- it tries to be too many things. In addition to serializing, hashing, and de-duplicating, ti also supports splitting up field list records and adding continuations. All of this functionality crammed into this one class makes it very complicated to work with and hard to maintain.

To solve all of these problems, I've re-written everything from scratch and split the functionality into separate pieces that can easily be reused. The end result is that one class TypeSerializer is turned into 3 new classes SimpleTypeSerializer, ContinuationRecordBuilder, and TypeTableBuilder, each of which in isolation is simple and straightforward.

A quick summary of these new classes and their responsibilities are:

• SimpleTypeSerializer : Turns a non-FieldList leaf type into a series of bytes. Does not do any hashing. Every time you call it, it will re-serialize and return bytes again. The same instance can be re-used over and over to avoid re-allocations, and in exchange for this optimization the bytes returned by the serializer only live until the caller attempts to serialize a new record.
• ContinuationRecordBuilder : Turns a FieldList-like record into a series of fragments. Does not do any hashing. Like SimpleTypeSerializer, returns references to privately owned bytes, so the storage is invalidated as soon as the caller tries to re-use the instance. Works equally well for LF_FIELDLIST as it does for LF_METHODLIST, solving a long-standing theoretical limitation of the previous implementation.
• TypeTableBuilder - Accepts sequences of bytes that the user has already serialized, and inserts them by de-duplicating with a hash table. For the sake of convenience and efficiency, this class internally stores a SimpleTypeSerializer so that it can accept unserialized records. The same is not true of ContinuationRecordBuilder. The user is required to create their own instance of ContinuationRecordBuilder.

Because of the separation of responsibilities, the proposed algorithm for content hashing now becomes straightforward to implement.

auto Hashes = readHashStream(Obj);
auto Types = readTypeStream(Obj);
for (int I=0; I < Hashes.size(); ++I) {
   TTB.insert_as(Types[I], Hashes[I]);
}

[rnk]

llvm/include/llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h

55 ↗

(On Diff #124455)

I see how this is supposed to work, but maybe stamping out overloads for all of the member types in the header and redirecting them to use a common template implementation in the .cpp file is a better way to do this.

Compilers sometimes warn when an implicit template instantiation is requested from a function template declaration with no body.

llvm/include/llvm/DebugInfo/CodeView/SimpleTypeSerializer.h

41 ↗

(On Diff #124455)

Ditto, re: stamping out overloads.

llvm/include/llvm/DebugInfo/CodeView/TypeTableBuilder.h

1–2 ↗

(On Diff #124455)

Fix that so it's on one line.

11 ↗

(On Diff #124455)

Fix the header guard macro name

llvm/lib/DebugInfo/CodeView/ContinuationRecordBuilder.cpp

8 ↗

(On Diff #124455)

I hate unicorn initialization =P

108 ↗

(On Diff #124455)

nit: reflow

114 ↗

(On Diff #124455)

nit: reflow

[zturner]

llvm/include/llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h

55 ↗

(On Diff #124455)

I had it like you suggested originally, but I actually did this on purpose because it's easier to look at a header file with a single declaration than it is to look at a bunch of template goo and list of 100 method overloads (or alternatively, the macro and #include goo that also works).

I explicitly instantiate every template in the implementation file using the macro / #include trick, but at least that way it's hidden in the implementation file. I thought explicit instantiations were correctly supported by all compilers?

[zturner]

llvm/lib/DebugInfo/CodeView/ContinuationRecordBuilder.cpp

8 ↗

(On Diff #124455)

If you mean the {} syntax, it's actually necessary in this case since it requires an implicit conversion. I couldn't get it to work with an = syntax.

[zturner]

Updated with suggestions.

[rnk]

lgtm

llvm/include/llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h

55 ↗

(On Diff #124455)

It'll work, but template declarations in headers without inline definitions below are kind of a code smell. At least, I start looking around going "how can this work". At least put a comment explaining that these are instantiated for all member types in the .cpp file. If it's warning free with that, sounds good, ship it.

llvm/lib/DebugInfo/CodeView/ContinuationRecordBuilder.cpp

8 ↗

(On Diff #124455)

Yeah, I figured. I'm just complaining.

llvm/trunk/include/llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h

+//===- ContinuationRecordBuilder.h ------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_CODEVIEW_CONTINUATIONRECORDBUILDER_H
+#define LLVM_DEBUGINFO_CODEVIEW_CONTINUATIONRECORDBUILDER_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
+#include "llvm/DebugInfo/CodeView/TypeIndex.h"
+#include "llvm/DebugInfo/CodeView/TypeRecord.h"
+#include "llvm/DebugInfo/CodeView/TypeRecordMapping.h"
+#include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/BinaryByteStream.h"
+#include "llvm/Support/BinaryStreamWriter.h"
+#include "llvm/Support/Error.h"
+#include <cassert>
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+namespace llvm {
+namespace codeview {
+enum class ContinuationRecordKind { FieldList, MethodOverloadList };
+
+class ContinuationRecordBuilder {
+  SmallVector<uint32_t, 4> SegmentOffsets;
+  Optional<ContinuationRecordKind> Kind;
+  AppendingBinaryByteStream Buffer;
+  BinaryStreamWriter SegmentWriter;
+  TypeRecordMapping Mapping;
+  ArrayRef<uint8_t> InjectedSegmentBytes;
+
+  uint32_t getCurrentSegmentLength() const;
+
+  void insertSegmentEnd(uint32_t Offset);
+  CVType createSegmentRecord(uint32_t OffBegin, uint32_t OffEnd,
+                             Optional<TypeIndex> RefersTo);
+
+public:
+  ContinuationRecordBuilder();
+  ~ContinuationRecordBuilder();
+
+  void begin(ContinuationRecordKind RecordKind);
+
+  // This template is explicitly instantiated in the implementation file for all
+  // supported types.  The method itself is ugly, so inlining it into the header
+  // file clutters an otherwise straightforward interface.
+  template <typename RecordType> void writeMemberType(RecordType &Record);
+
+  std::vector<CVType> end(TypeIndex Index);
+};
+} // namespace codeview
+} // namespace llvm
+
+#endif
 No newline at end of file

llvm/trunk/include/llvm/DebugInfo/CodeView/SimpleTypeSerializer.h

+//===- SimpleTypeSerializer.h -----------------------------------*- C++ -*-===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_DEBUGINFO_CODEVIEW_SIMPLETYPESERIALIZER_H
+#define LLVM_DEBUGINFO_CODEVIEW_SIMPLETYPESERIALIZER_H
+
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
+#include "llvm/DebugInfo/CodeView/TypeIndex.h"
+#include "llvm/DebugInfo/CodeView/TypeRecord.h"
+#include "llvm/DebugInfo/CodeView/TypeRecordMapping.h"
+#include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/BinaryByteStream.h"
+#include "llvm/Support/BinaryStreamWriter.h"
+#include "llvm/Support/Error.h"
+#include <cassert>
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+namespace llvm {
+namespace codeview {
+
+class SimpleTypeSerializer {
+  std::vector<uint8_t> ScratchBuffer;
+
+public:
+  SimpleTypeSerializer();
+  ~SimpleTypeSerializer();
+
+  // This template is explicitly instantiated in the implementation file for all
+  // supported types.  The method itself is ugly, so inlining it into the header
+  // file clutters an otherwise straightforward interface.
+  template <typename T> ArrayRef<uint8_t> serialize(T &Record);
+
+  // Don't allow serialization of field list records using this interface.
+  ArrayRef<uint8_t> serialize(const FieldListRecord &Record) = delete;
+};
+
+} // end namespace codeview
+} // end namespace llvm
+
+#endif // LLVM_DEBUGINFO_CODEVIEW_SIMPLETYPESERIALIZER_H

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeSerializer.h

-//===- TypeSerializer.h -----------------------------------------*- C++ -*-===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#ifndef LLVM_DEBUGINFO_CODEVIEW_TYPESERIALIZER_H
-#define LLVM_DEBUGINFO_CODEVIEW_TYPESERIALIZER_H
-
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/Optional.h"
-#include "llvm/ADT/SmallVector.h"
-#include "llvm/DebugInfo/CodeView/CodeView.h"
-#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
-#include "llvm/DebugInfo/CodeView/TypeIndex.h"
-#include "llvm/DebugInfo/CodeView/TypeRecord.h"
-#include "llvm/DebugInfo/CodeView/TypeRecordMapping.h"
-#include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
-#include "llvm/Support/Allocator.h"
-#include "llvm/Support/BinaryByteStream.h"
-#include "llvm/Support/BinaryStreamWriter.h"
-#include "llvm/Support/Error.h"
-#include <cassert>
-#include <cstdint>
-#include <memory>
-#include <vector>
-
-namespace llvm {
-namespace codeview {
-
-class TypeHasher;
-
-class TypeSerializer : public TypeVisitorCallbacks {
-  struct SubRecord {
-    SubRecord(TypeLeafKind K, uint32_t S) : Kind(K), Size(S) {}
-
-    TypeLeafKind Kind;
-    uint32_t Size = 0;
-  };
-  struct RecordSegment {
-    SmallVector<SubRecord, 16> SubRecords;
-
-    uint32_t length() const {
-      uint32_t L = sizeof(RecordPrefix);
-      for (const auto &R : SubRecords) {
-        L += R.Size;
-      }
-      return L;
-    }
-  };
-
-  using MutableRecordList = SmallVector<MutableArrayRef<uint8_t>, 2>;
-
-  static constexpr uint8_t ContinuationLength = 8;
-  BumpPtrAllocator &RecordStorage;
-  RecordSegment CurrentSegment;
-  MutableRecordList FieldListSegments;
-
-  Optional<TypeLeafKind> TypeKind;
-  Optional<TypeLeafKind> MemberKind;
-  std::vector<uint8_t> RecordBuffer;
-  MutableBinaryByteStream Stream;
-  BinaryStreamWriter Writer;
-  TypeRecordMapping Mapping;
-
-  /// Private type record hashing implementation details are handled here.
-  std::unique_ptr<TypeHasher> Hasher;
-
-  /// Contains a list of all records indexed by TypeIndex.toArrayIndex().
-  SmallVector<ArrayRef<uint8_t>, 2> SeenRecords;
-
-  /// Temporary storage that we use to copy a record's data while re-writing
-  /// its type indices.
-  SmallVector<uint8_t, 256> RemapStorage;
-
-  TypeIndex nextTypeIndex() const;
-
-  bool isInFieldList() const;
-  MutableArrayRef<uint8_t> getCurrentSubRecordData();
-  MutableArrayRef<uint8_t> getCurrentRecordData();
-  Error writeRecordPrefix(TypeLeafKind Kind);
-
-  Expected<MutableArrayRef<uint8_t>>
-  addPadding(MutableArrayRef<uint8_t> Record);
-
-public:
-  explicit TypeSerializer(BumpPtrAllocator &Storage, bool Hash = true);
-  ~TypeSerializer() override;
-
-  void reset();
-
-  BumpPtrAllocator &getAllocator() { return RecordStorage; }
-
-  ArrayRef<ArrayRef<uint8_t>> records() const;
-  TypeIndex insertRecordBytes(ArrayRef<uint8_t> &Record);
-  TypeIndex insertRecord(const RemappedType &Record);
-  Expected<TypeIndex> visitTypeEndGetIndex(CVType &Record);
-
-  using TypeVisitorCallbacks::visitTypeBegin;
-  Error visitTypeBegin(CVType &Record) override;
-  Error visitTypeEnd(CVType &Record) override;
-  Error visitMemberBegin(CVMemberRecord &Record) override;
-  Error visitMemberEnd(CVMemberRecord &Record) override;
-
-#define TYPE_RECORD(EnumName, EnumVal, Name)                                   \
-  virtual Error visitKnownRecord(CVType &CVR, Name##Record &Record) override { \
-    return visitKnownRecordImpl(CVR, Record);                                  \
-  }
-#define TYPE_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
-#define MEMBER_RECORD(EnumName, EnumVal, Name)                                 \
-  Error visitKnownMember(CVMemberRecord &CVR, Name##Record &Record) override { \
-    return visitKnownMemberImpl<Name##Record>(CVR, Record);                    \
-  }
-#define MEMBER_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
-#include "llvm/DebugInfo/CodeView/CodeViewTypes.def"
-
-private:
-  template <typename RecordKind>
-  Error visitKnownRecordImpl(CVType &CVR, RecordKind &Record) {
-    return Mapping.visitKnownRecord(CVR, Record);
-  }
-
-  template <typename RecordType>
-  Error visitKnownMemberImpl(CVMemberRecord &CVR, RecordType &Record) {
-    assert(CVR.Kind == static_cast<TypeLeafKind>(Record.getKind()));
-
-    if (auto EC = Writer.writeEnum(CVR.Kind))
-      return EC;
-
-    if (auto EC = Mapping.visitKnownMember(CVR, Record))
-      return EC;
-
-    // Get all the data that was just written and is yet to be committed to
-    // the current segment.  Then pad it to 4 bytes.
-    MutableArrayRef<uint8_t> ThisRecord = getCurrentSubRecordData();
-    auto ExpectedRecord = addPadding(ThisRecord);
-    if (!ExpectedRecord)
-      return ExpectedRecord.takeError();
-    ThisRecord = *ExpectedRecord;
-
-    CurrentSegment.SubRecords.emplace_back(CVR.Kind, ThisRecord.size());
-    CVR.Data = ThisRecord;
-
-    // Both the last subrecord and the total length of this segment should be
-    // multiples of 4.
-    assert(ThisRecord.size() % 4 == 0);
-    assert(CurrentSegment.length() % 4 == 0);
-
-    return Error::success();
-  }
-};
-
-} // end namespace codeview
-} // end namespace llvm
-
-#endif // LLVM_DEBUGINFO_CODEVIEW_TYPESERIALIZER_H

llvm/trunk/include/llvm/DebugInfo/CodeView/TypeTableBuilder.h

-//===- TypeTableBuilder.h ---------------------------------------*- C++ -*-===//
+//===- TypeTableBuilder.h ----------------------------------------*- C++-*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef LLVM_DEBUGINFO_CODEVIEW_TYPETABLEBUILDER_H
 #define LLVM_DEBUGINFO_CODEVIEW_TYPETABLEBUILDER_H
 
 #include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/Optional.h"
+#include "llvm/ADT/SmallVector.h"
 #include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
+#include "llvm/DebugInfo/CodeView/SimpleTypeSerializer.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
-#include "llvm/DebugInfo/CodeView/TypeSerializer.h"
+#include "llvm/DebugInfo/CodeView/TypeRecordMapping.h"
+#include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
 #include "llvm/Support/Allocator.h"
+#include "llvm/Support/BinaryByteStream.h"
+#include "llvm/Support/BinaryStreamWriter.h"
 #include "llvm/Support/Error.h"
-#include <algorithm>
 #include <cassert>
 #include <cstdint>
-#include <type_traits>
+#include <memory>
+#include <vector>
 
 namespace llvm {
 namespace codeview {
 
-class TypeTableBuilder {
-private:
-  TypeIndex handleError(Error EC) const {
-    assert(false && "Couldn't write Type!");
-    consumeError(std::move(EC));
-    return TypeIndex();
-  }
-
-  BumpPtrAllocator &Allocator;
-  TypeSerializer Serializer;
+class ContinuationRecordBuilder;
+class TypeHasher;
 
-public:
-  explicit TypeTableBuilder(BumpPtrAllocator &Allocator,
-                            bool WriteUnique = true)
-      : Allocator(Allocator), Serializer(Allocator, WriteUnique) {}
-  TypeTableBuilder(const TypeTableBuilder &) = delete;
-  TypeTableBuilder &operator=(const TypeTableBuilder &) = delete;
-
-  bool empty() const { return Serializer.records().empty(); }
-
-  BumpPtrAllocator &getAllocator() const { return Allocator; }
-
-  template <typename T> TypeIndex writeKnownType(T &Record) {
-    static_assert(!std::is_same<T, FieldListRecord>::value,
-                  "Can't serialize FieldList!");
-
-    CVType Type;
-    Type.Type = static_cast<TypeLeafKind>(Record.getKind());
-    if (auto EC = Serializer.visitTypeBegin(Type))
-      return handleError(std::move(EC));
-    if (auto EC = Serializer.visitKnownRecord(Type, Record))
-      return handleError(std::move(EC));
-
-    auto ExpectedIndex = Serializer.visitTypeEndGetIndex(Type);
-    if (!ExpectedIndex)
-      return handleError(ExpectedIndex.takeError());
+class TypeTableBuilder : public TypeVisitorCallbacks {
 
-    return *ExpectedIndex;
-  }
+  BumpPtrAllocator &RecordStorage;
+  SimpleTypeSerializer SimpleSerializer;
 
-  TypeIndex writeSerializedRecord(ArrayRef<uint8_t> Record) {
-    return Serializer.insertRecordBytes(Record);
-  }
+  /// Private type record hashing implementation details are handled here.
+  std::unique_ptr<TypeHasher> Hasher;
 
-  TypeIndex writeSerializedRecord(const RemappedType &Record) {
-    return Serializer.insertRecord(Record);
-  }
+  /// Contains a list of all records indexed by TypeIndex.toArrayIndex().
+  SmallVector<ArrayRef<uint8_t>, 2> SeenRecords;
 
-  template <typename TFunc> void ForEachRecord(TFunc Func) {
-    uint32_t Index = TypeIndex::FirstNonSimpleIndex;
+  /// Temporary storage that we use to copy a record's data while re-writing
+  /// its type indices.
+  SmallVector<uint8_t, 256> RemapStorage;
 
-    for (auto Record : Serializer.records()) {
-      Func(TypeIndex(Index), Record);
-      ++Index;
-    }
-  }
+public:
+  explicit TypeTableBuilder(BumpPtrAllocator &Storage, bool Hash = true);
+  ~TypeTableBuilder() override;
 
-  ArrayRef<ArrayRef<uint8_t>> records() const { return Serializer.records(); }
-};
+  void reset();
 
-class FieldListRecordBuilder {
-  TypeTableBuilder &TypeTable;
-  BumpPtrAllocator Allocator;
-  TypeSerializer TempSerializer;
-  CVType Type;
+  bool empty() const { return SeenRecords.empty(); }
 
-public:
-  explicit FieldListRecordBuilder(TypeTableBuilder &TypeTable)
-      : TypeTable(TypeTable), TempSerializer(Allocator, false) {
-    Type.Type = TypeLeafKind::LF_FIELDLIST;
-  }
+  TypeIndex nextTypeIndex() const;
 
-  void begin() {
-    TempSerializer.reset();
+  BumpPtrAllocator &getAllocator() { return RecordStorage; }
 
-    if (auto EC = TempSerializer.visitTypeBegin(Type))
-      consumeError(std::move(EC));
-  }
+  ArrayRef<ArrayRef<uint8_t>> records() const;
+  TypeIndex insertRecordBytes(ArrayRef<uint8_t> &Record);
+  TypeIndex insertRecord(const RemappedType &Record);
+  TypeIndex insertRecord(ContinuationRecordBuilder &Builder);
 
-  template <typename T> void writeMemberType(T &Record) {
-    CVMemberRecord CVMR;
-    CVMR.Kind = static_cast<TypeLeafKind>(Record.getKind());
-    if (auto EC = TempSerializer.visitMemberBegin(CVMR))
-      consumeError(std::move(EC));
-    if (auto EC = TempSerializer.visitKnownMember(CVMR, Record))
-      consumeError(std::move(EC));
-    if (auto EC = TempSerializer.visitMemberEnd(CVMR))
-      consumeError(std::move(EC));
+  template <typename T> TypeIndex writeLeafType(T &Record) {
+    ArrayRef<uint8_t> Data = SimpleSerializer.serialize(Record);
+    return insertRecordBytes(Data);
   }
 
-  TypeIndex end(bool Write) {
-    TypeIndex Index;
-    if (auto EC = TempSerializer.visitTypeEnd(Type)) {
-      consumeError(std::move(EC));
-      return TypeIndex();
-    }
+  template <typename TFunc> void ForEachRecord(TFunc Func) {
+    uint32_t Index = TypeIndex::FirstNonSimpleIndex;
 
-    if (Write) {
-      for (auto Record : TempSerializer.records())
-        Index = TypeTable.writeSerializedRecord(Record);
+    for (auto Record : SeenRecords) {
+      Func(TypeIndex(Index), Record);
+      ++Index;
     }
-
-    return Index;
   }
 };
 
 } // end namespace codeview
 } // end namespace llvm
 
 #endif // LLVM_DEBUGINFO_CODEVIEW_TYPETABLEBUILDER_H

llvm/trunk/include/llvm/ObjectYAML/CodeViewYAMLTypes.h

 #include "llvm/Support/YAMLTraits.h"
 #include <cstdint>
 #include <memory>
 #include <vector>
 
 namespace llvm {
 
 namespace codeview {
-
 class TypeTableBuilder;
-
-} // end namespace codeview
+}
 
 namespace CodeViewYAML {
 
 namespace detail {
 
 struct LeafRecordBase;
 struct MemberRecordBase;
 
 } // end namespace detail
 
 struct MemberRecord {
   std::shared_ptr<detail::MemberRecordBase> Member;
 };
 
 struct LeafRecord {
   std::shared_ptr<detail::LeafRecordBase> Leaf;
 
-  codeview::CVType toCodeViewRecord(BumpPtrAllocator &Allocator) const;
-  codeview::CVType toCodeViewRecord(codeview::TypeTableBuilder &TS) const;
+  codeview::CVType
+  toCodeViewRecord(codeview::TypeTableBuilder &Serializer) const;
   static Expected<LeafRecord> fromCodeViewRecord(codeview::CVType Type);
 };
 
 std::vector<LeafRecord> fromDebugT(ArrayRef<uint8_t> DebugT);
 ArrayRef<uint8_t> toDebugT(ArrayRef<LeafRecord>, BumpPtrAllocator &Alloc);
 
 } // end namespace CodeViewYAML
 

llvm/trunk/include/llvm/Support/BinaryByteStream.h

private:
   MutableArrayRef<uint8_t> Data;
   BinaryByteStream ImmutableStream;
 };
 
 /// \brief An implementation of WritableBinaryStream which can write at its end
 /// causing the underlying data to grow.  This class owns the underlying data.
 class AppendingBinaryByteStream : public WritableBinaryStream {
   std::vector<uint8_t> Data;
-  llvm::support::endianness Endian;
+  llvm::support::endianness Endian = llvm::support::little;
 
 public:
   AppendingBinaryByteStream() = default;
   AppendingBinaryByteStream(llvm::support::endianness Endian)
       : Endian(Endian) {}
 
   void clear() { Data.clear(); }
 
   llvm::support::endianness getEndian() const override { return Endian; }
 
   Error readBytes(uint32_t Offset, uint32_t Size,
                   ArrayRef<uint8_t> &Buffer) override {
     if (auto EC = checkOffsetForWrite(Offset, Buffer.size()))
       return EC;
 
     Buffer = makeArrayRef(Data).slice(Offset, Size);
     return Error::success();
   }
 
+  void insert(uint32_t Offset, ArrayRef<uint8_t> Bytes) {
+    Data.insert(Data.begin() + Offset, Bytes.begin(), Bytes.end());
+  }
+
   Error readLongestContiguousChunk(uint32_t Offset,
                                    ArrayRef<uint8_t> &Buffer) override {
     if (auto EC = checkOffsetForWrite(Offset, 1))
       return EC;
 
     Buffer = makeArrayRef(Data).slice(Offset);
     return Error::success();
   }

llvm/trunk/lib/CodeGen/AsmPrinter/CodeViewDebug.cpp

 #include "llvm/CodeGen/MachineOperand.h"
 #include "llvm/CodeGen/TargetFrameLowering.h"
 #include "llvm/CodeGen/TargetLoweringObjectFile.h"
 #include "llvm/CodeGen/TargetRegisterInfo.h"
 #include "llvm/CodeGen/TargetSubtargetInfo.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h"
 #include "llvm/DebugInfo/CodeView/DebugInlineeLinesSubsection.h"
 #include "llvm/DebugInfo/CodeView/Line.h"
 #include "llvm/DebugInfo/CodeView/SymbolRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeDumpVisitor.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeTableCollection.h"
 #include "llvm/IR/Constants.h"
TypeIndex CodeViewDebug::getScopeIndex(const DIScope *Scope) {
   // Check if we've already translated this scope.
   auto I = TypeIndices.find({Scope, nullptr});
   if (I != TypeIndices.end())
     return I->second;
 
   // Build the fully qualified name of the scope.
   std::string ScopeName = getFullyQualifiedName(Scope);
   StringIdRecord SID(TypeIndex(), ScopeName);
-  auto TI = TypeTable.writeKnownType(SID);
+  auto TI = TypeTable.writeLeafType(SID);
   return recordTypeIndexForDINode(Scope, TI);
 }
 
 TypeIndex CodeViewDebug::getFuncIdForSubprogram(const DISubprogram *SP) {
   assert(SP);
 
   // Check if we've already translated this subprogram.
   auto I = TypeIndices.find({SP, nullptr});
   if (I != TypeIndices.end())
     return I->second;
 
   // The display name includes function template arguments. Drop them to match
   // MSVC.
   StringRef DisplayName = SP->getName().split('<').first;
 
   const DIScope *Scope = SP->getScope().resolve();
   TypeIndex TI;
   if (const auto *Class = dyn_cast_or_null<DICompositeType>(Scope)) {
     // If the scope is a DICompositeType, then this must be a method. Member
     // function types take some special handling, and require access to the
     // subprogram.
     TypeIndex ClassType = getTypeIndex(Class);
     MemberFuncIdRecord MFuncId(ClassType, getMemberFunctionType(SP, Class),
                                DisplayName);
-    TI = TypeTable.writeKnownType(MFuncId);
+    TI = TypeTable.writeLeafType(MFuncId);
   } else {
     // Otherwise, this must be a free function.
     TypeIndex ParentScope = getScopeIndex(Scope);
     FuncIdRecord FuncId(ParentScope, getTypeIndex(SP->getType()), DisplayName);
-    TI = TypeTable.writeKnownType(FuncId);
+    TI = TypeTable.writeLeafType(FuncId);
   }
 
   return recordTypeIndexForDINode(SP, TI);
 }
 
 TypeIndex CodeViewDebug::getMemberFunctionType(const DISubprogram *SP,
                                                const DICompositeType *Class) {
   // Always use the method declaration as the key for the function type. The
for (int i = Elements.size() - 1; i >= 0; --i) {
 
     // If this is the outermost array, use the size from the array. It will be
     // more accurate if we had a VLA or an incomplete element type size.
     uint64_t ArraySize =
         (i == 0 && ElementSize == 0) ? Ty->getSizeInBits() / 8 : ElementSize;
 
     StringRef Name = (i == 0) ? Ty->getName() : "";
     ArrayRecord AR(ElementTypeIndex, IndexType, ArraySize, Name);
-    ElementTypeIndex = TypeTable.writeKnownType(AR);
+    ElementTypeIndex = TypeTable.writeLeafType(AR);
   }
 
   return ElementTypeIndex;
 }
 
 TypeIndex CodeViewDebug::lowerTypeBasic(const DIBasicType *Ty) {
   TypeIndex Index;
   dwarf::TypeKind Kind;
case dwarf::DW_TAG_rvalue_reference_type:
     PM = PointerMode::RValueReference;
     break;
   }
   // FIXME: MSVC folds qualifiers into PointerOptions in the context of a method
   // 'this' pointer, but not normal contexts. Figure out what we're supposed to
   // do.
   PointerOptions PO = PointerOptions::None;
   PointerRecord PR(PointeeTI, PK, PM, PO, Ty->getSizeInBits() / 8);
-  return TypeTable.writeKnownType(PR);
+  return TypeTable.writeLeafType(PR);
 }
 
 static PointerToMemberRepresentation
 translatePtrToMemberRep(unsigned SizeInBytes, bool IsPMF, unsigned Flags) {
   // SizeInBytes being zero generally implies that the member pointer type was
   // incomplete, which can happen if it is part of a function prototype. In this
   // case, use the unknown model instead of the general model.
   if (IsPMF) {
TypeIndex CodeViewDebug::lowerTypeMemberPointer(const DIDerivedType *Ty) {
   PointerMode PM = IsPMF ? PointerMode::PointerToMemberFunction
                          : PointerMode::PointerToDataMember;
   PointerOptions PO = PointerOptions::None; // FIXME
   assert(Ty->getSizeInBits() / 8 <= 0xff && "pointer size too big");
   uint8_t SizeInBytes = Ty->getSizeInBits() / 8;
   MemberPointerInfo MPI(
       ClassTI, translatePtrToMemberRep(SizeInBytes, IsPMF, Ty->getFlags()));
   PointerRecord PR(PointeeTI, PK, PM, PO, SizeInBytes, MPI);
-  return TypeTable.writeKnownType(PR);
+  return TypeTable.writeLeafType(PR);
 }
 
 /// Given a DWARF calling convention, get the CodeView equivalent. If we don't
 /// have a translation, use the NearC convention.
 static CallingConvention dwarfCCToCodeView(unsigned DwarfCC) {
   switch (DwarfCC) {
   case dwarf::DW_CC_normal:             return CallingConvention::NearC;
   case dwarf::DW_CC_BORLAND_msfastcall: return CallingConvention::NearFast;
default:
       IsModifier = false;
       break;
     }
     if (IsModifier)
       BaseTy = cast<DIDerivedType>(BaseTy)->getBaseType().resolve();
   }
   TypeIndex ModifiedTI = getTypeIndex(BaseTy);
   ModifierRecord MR(ModifiedTI, Mods);
-  return TypeTable.writeKnownType(MR);
+  return TypeTable.writeLeafType(MR);
 }
 
 TypeIndex CodeViewDebug::lowerTypeFunction(const DISubroutineType *Ty) {
   SmallVector<TypeIndex, 8> ReturnAndArgTypeIndices;
   for (DITypeRef ArgTypeRef : Ty->getTypeArray())
     ReturnAndArgTypeIndices.push_back(getTypeIndex(ArgTypeRef));
 
   TypeIndex ReturnTypeIndex = TypeIndex::Void();
   ArrayRef<TypeIndex> ArgTypeIndices = None;
   if (!ReturnAndArgTypeIndices.empty()) {
     auto ReturnAndArgTypesRef = makeArrayRef(ReturnAndArgTypeIndices);
     ReturnTypeIndex = ReturnAndArgTypesRef.front();
     ArgTypeIndices = ReturnAndArgTypesRef.drop_front();
   }
 
   ArgListRecord ArgListRec(TypeRecordKind::ArgList, ArgTypeIndices);
-  TypeIndex ArgListIndex = TypeTable.writeKnownType(ArgListRec);
+  TypeIndex ArgListIndex = TypeTable.writeLeafType(ArgListRec);
 
   CallingConvention CC = dwarfCCToCodeView(Ty->getCC());
 
   ProcedureRecord Procedure(ReturnTypeIndex, CC, FunctionOptions::None,
                             ArgTypeIndices.size(), ArgListIndex);
-  return TypeTable.writeKnownType(Procedure);
+  return TypeTable.writeLeafType(Procedure);
 }
 
 TypeIndex CodeViewDebug::lowerTypeMemberFunction(const DISubroutineType *Ty,
                                                  const DIType *ClassTy,
                                                  int ThisAdjustment,
                                                  bool IsStaticMethod) {
   // Lower the containing class type.
   TypeIndex ClassType = getTypeIndex(ClassTy);
TypeIndex CodeViewDebug::lowerTypeMemberFunction(const DISubroutineType *Ty,
   }
   TypeIndex ThisTypeIndex;
   if (!IsStaticMethod && !ArgTypeIndices.empty()) {
     ThisTypeIndex = ArgTypeIndices.front();
     ArgTypeIndices = ArgTypeIndices.drop_front();
   }
 
   ArgListRecord ArgListRec(TypeRecordKind::ArgList, ArgTypeIndices);
-  TypeIndex ArgListIndex = TypeTable.writeKnownType(ArgListRec);
+  TypeIndex ArgListIndex = TypeTable.writeLeafType(ArgListRec);
 
   CallingConvention CC = dwarfCCToCodeView(Ty->getCC());
 
   // TODO: Need to use the correct values for FunctionOptions.
   MemberFunctionRecord MFR(ReturnTypeIndex, ClassType, ThisTypeIndex, CC,
                            FunctionOptions::None, ArgTypeIndices.size(),
                            ArgListIndex, ThisAdjustment);
-  TypeIndex TI = TypeTable.writeKnownType(MFR);
-
-  return TI;
+  return TypeTable.writeLeafType(MFR);
 }
 
 TypeIndex CodeViewDebug::lowerTypeVFTableShape(const DIDerivedType *Ty) {
   unsigned VSlotCount =
       Ty->getSizeInBits() / (8 * Asm->MAI->getCodePointerSize());
   SmallVector<VFTableSlotKind, 4> Slots(VSlotCount, VFTableSlotKind::Near);
 
   VFTableShapeRecord VFTSR(Slots);
-  return TypeTable.writeKnownType(VFTSR);
+  return TypeTable.writeLeafType(VFTSR);
 }
 
 static MemberAccess translateAccessFlags(unsigned RecordTag, unsigned Flags) {
   switch (Flags & DINode::FlagAccessibility) {
   case DINode::FlagPrivate:   return MemberAccess::Private;
   case DINode::FlagPublic:    return MemberAccess::Public;
   case DINode::FlagProtected: return MemberAccess::Protected;
   case 0:

 TypeIndex CodeViewDebug::lowerTypeEnum(const DICompositeType *Ty) {
   ClassOptions CO = getCommonClassOptions(Ty);
   TypeIndex FTI;
   unsigned EnumeratorCount = 0;
 
   if (Ty->isForwardDecl()) {
     CO |= ClassOptions::ForwardReference;
   } else {
-    FieldListRecordBuilder FLRB(TypeTable);
-
-    FLRB.begin();
+    ContinuationRecordBuilder ContinuationBuilder;
+    ContinuationBuilder.begin(ContinuationRecordKind::FieldList);
     for (const DINode *Element : Ty->getElements()) {
       // We assume that the frontend provides all members in source declaration
       // order, which is what MSVC does.
       if (auto *Enumerator = dyn_cast_or_null<DIEnumerator>(Element)) {
         EnumeratorRecord ER(MemberAccess::Public,
                             APSInt::getUnsigned(Enumerator->getValue()),
                             Enumerator->getName());
-        FLRB.writeMemberType(ER);
+        ContinuationBuilder.writeMemberType(ER);
         EnumeratorCount++;
       }
     }
-    FTI = FLRB.end(true);
+    FTI = TypeTable.insertRecord(ContinuationBuilder);
   }
 
   std::string FullName = getFullyQualifiedName(Ty);
 
   EnumRecord ER(EnumeratorCount, CO, FTI, FullName, Ty->getIdentifier(),
                 getTypeIndex(Ty->getBaseType()));
-  return TypeTable.writeKnownType(ER);
+  return TypeTable.writeLeafType(ER);
 }
 
 //===----------------------------------------------------------------------===//
 // ClassInfo
 //===----------------------------------------------------------------------===//
 
 struct llvm::ClassInfo {
   struct MemberInfo {
TypeIndex CodeViewDebug::lowerTypeClass(const DICompositeType *Ty) {
   // First, construct the forward decl.  Don't look into Ty to compute the
   // forward decl options, since it might not be available in all TUs.
   TypeRecordKind Kind = getRecordKind(Ty);
   ClassOptions CO =
       ClassOptions::ForwardReference | getCommonClassOptions(Ty);
   std::string FullName = getFullyQualifiedName(Ty);
   ClassRecord CR(Kind, 0, CO, TypeIndex(), TypeIndex(), TypeIndex(), 0,
                  FullName, Ty->getIdentifier());
-  TypeIndex FwdDeclTI = TypeTable.writeKnownType(CR);
+  TypeIndex FwdDeclTI = TypeTable.writeLeafType(CR);
   if (!Ty->isForwardDecl())
     DeferredCompleteTypes.push_back(Ty);
   return FwdDeclTI;
 }
 
 TypeIndex CodeViewDebug::lowerCompleteTypeClass(const DICompositeType *Ty) {
   // Construct the field list and complete type record.
   TypeRecordKind Kind = getRecordKind(Ty);
if (ContainsNestedClass)
     CO |= ClassOptions::ContainsNestedClass;
 
   std::string FullName = getFullyQualifiedName(Ty);
 
   uint64_t SizeInBytes = Ty->getSizeInBits() / 8;
 
   ClassRecord CR(Kind, FieldCount, CO, FieldTI, TypeIndex(), VShapeTI,
                  SizeInBytes, FullName, Ty->getIdentifier());
-  TypeIndex ClassTI = TypeTable.writeKnownType(CR);
+  TypeIndex ClassTI = TypeTable.writeLeafType(CR);
 
   if (const auto *File = Ty->getFile()) {
     StringIdRecord SIDR(TypeIndex(0x0), getFullFilepath(File));
-    TypeIndex SIDI = TypeTable.writeKnownType(SIDR);
+    TypeIndex SIDI = TypeTable.writeLeafType(SIDR);
+
     UdtSourceLineRecord USLR(ClassTI, SIDI, Ty->getLine());
-    TypeTable.writeKnownType(USLR);
+    TypeTable.writeLeafType(USLR);
   }
 
   addToUDTs(Ty);
 
   return ClassTI;
 }
 
 TypeIndex CodeViewDebug::lowerTypeUnion(const DICompositeType *Ty) {
   ClassOptions CO =
       ClassOptions::ForwardReference | getCommonClassOptions(Ty);
   std::string FullName = getFullyQualifiedName(Ty);
   UnionRecord UR(0, CO, TypeIndex(), 0, FullName, Ty->getIdentifier());
-  TypeIndex FwdDeclTI = TypeTable.writeKnownType(UR);
+  TypeIndex FwdDeclTI = TypeTable.writeLeafType(UR);
   if (!Ty->isForwardDecl())
     DeferredCompleteTypes.push_back(Ty);
   return FwdDeclTI;
 }
 
 TypeIndex CodeViewDebug::lowerCompleteTypeUnion(const DICompositeType *Ty) {
   ClassOptions CO = ClassOptions::Sealed | getCommonClassOptions(Ty);
   TypeIndex FieldTI;
   unsigned FieldCount;
   bool ContainsNestedClass;
   std::tie(FieldTI, std::ignore, FieldCount, ContainsNestedClass) =
       lowerRecordFieldList(Ty);
 
   if (ContainsNestedClass)
     CO |= ClassOptions::ContainsNestedClass;
 
   uint64_t SizeInBytes = Ty->getSizeInBits() / 8;
   std::string FullName = getFullyQualifiedName(Ty);
 
   UnionRecord UR(FieldCount, CO, FieldTI, SizeInBytes, FullName,
                  Ty->getIdentifier());
-  TypeIndex UnionTI = TypeTable.writeKnownType(UR);
+  TypeIndex UnionTI = TypeTable.writeLeafType(UR);
 
   StringIdRecord SIR(TypeIndex(0x0), getFullFilepath(Ty->getFile()));
-  TypeIndex SIRI = TypeTable.writeKnownType(SIR);
+  TypeIndex SIRI = TypeTable.writeLeafType(SIR);
+
   UdtSourceLineRecord USLR(UnionTI, SIRI, Ty->getLine());
-  TypeTable.writeKnownType(USLR);
+  TypeTable.writeLeafType(USLR);
 
   addToUDTs(Ty);
 
   return UnionTI;
 }
 
 std::tuple<TypeIndex, TypeIndex, unsigned, bool>
 CodeViewDebug::lowerRecordFieldList(const DICompositeType *Ty) {
   // Manually count members. MSVC appears to count everything that generates a
   // field list record. Each individual overload in a method overload group
   // contributes to this count, even though the overload group is a single field
   // list record.
   unsigned MemberCount = 0;
   ClassInfo Info = collectClassInfo(Ty);
-  FieldListRecordBuilder FLBR(TypeTable);
-  FLBR.begin();
+  ContinuationRecordBuilder ContinuationBuilder;
+  ContinuationBuilder.begin(ContinuationRecordKind::FieldList);
 
   // Create base classes.
   for (const DIDerivedType *I : Info.Inheritance) {
     if (I->getFlags() & DINode::FlagVirtual) {
       // Virtual base.
       // FIXME: Emit VBPtrOffset when the frontend provides it.
       unsigned VBPtrOffset = 0;
       // FIXME: Despite the accessor name, the offset is really in bytes.
       unsigned VBTableIndex = I->getOffsetInBits() / 4;
       auto RecordKind = (I->getFlags() & DINode::FlagIndirectVirtualBase) == DINode::FlagIndirectVirtualBase
                             ? TypeRecordKind::IndirectVirtualBaseClass
                             : TypeRecordKind::VirtualBaseClass;
       VirtualBaseClassRecord VBCR(
           RecordKind, translateAccessFlags(Ty->getTag(), I->getFlags()),
           getTypeIndex(I->getBaseType()), getVBPTypeIndex(), VBPtrOffset,
           VBTableIndex);
 
-      FLBR.writeMemberType(VBCR);
+      ContinuationBuilder.writeMemberType(VBCR);
     } else {
       assert(I->getOffsetInBits() % 8 == 0 &&
              "bases must be on byte boundaries");
       BaseClassRecord BCR(translateAccessFlags(Ty->getTag(), I->getFlags()),
                           getTypeIndex(I->getBaseType()),
                           I->getOffsetInBits() / 8);
-      FLBR.writeMemberType(BCR);
+      ContinuationBuilder.writeMemberType(BCR);
     }
   }
 
   // Create members.
   for (ClassInfo::MemberInfo &MemberInfo : Info.Members) {
     const DIDerivedType *Member = MemberInfo.MemberTypeNode;
     TypeIndex MemberBaseType = getTypeIndex(Member->getBaseType());
     StringRef MemberName = Member->getName();
     MemberAccess Access =
         translateAccessFlags(Ty->getTag(), Member->getFlags());
 
     if (Member->isStaticMember()) {
       StaticDataMemberRecord SDMR(Access, MemberBaseType, MemberName);
-      FLBR.writeMemberType(SDMR);
+      ContinuationBuilder.writeMemberType(SDMR);
       MemberCount++;
       continue;
     }
 
     // Virtual function pointer member.
     if ((Member->getFlags() & DINode::FlagArtificial) &&
         Member->getName().startswith("_vptr$")) {
       VFPtrRecord VFPR(getTypeIndex(Member->getBaseType()));
-      FLBR.writeMemberType(VFPR);
+      ContinuationBuilder.writeMemberType(VFPR);
       MemberCount++;
       continue;
     }
 
     // Data member.
     uint64_t MemberOffsetInBits =
         Member->getOffsetInBits() + MemberInfo.BaseOffset;
     if (Member->isBitField()) {
       uint64_t StartBitOffset = MemberOffsetInBits;
       if (const auto *CI =
               dyn_cast_or_null<ConstantInt>(Member->getStorageOffsetInBits())) {
         MemberOffsetInBits = CI->getZExtValue() + MemberInfo.BaseOffset;
       }
       StartBitOffset -= MemberOffsetInBits;
       BitFieldRecord BFR(MemberBaseType, Member->getSizeInBits(),
                          StartBitOffset);
-      MemberBaseType = TypeTable.writeKnownType(BFR);
+      MemberBaseType = TypeTable.writeLeafType(BFR);
     }
     uint64_t MemberOffsetInBytes = MemberOffsetInBits / 8;
     DataMemberRecord DMR(Access, MemberBaseType, MemberOffsetInBytes,
                          MemberName);
-    FLBR.writeMemberType(DMR);
+    ContinuationBuilder.writeMemberType(DMR);
     MemberCount++;
   }
 
   // Create methods
   for (auto &MethodItr : Info.Methods) {
     StringRef Name = MethodItr.first->getString();
 
     std::vector<OneMethodRecord> Methods;
     for (const DISubprogram *SP : MethodItr.second) {
       TypeIndex MethodType = getMemberFunctionType(SP, Ty);
       bool Introduced = SP->getFlags() & DINode::FlagIntroducedVirtual;
 
       unsigned VFTableOffset = -1;
       if (Introduced)
         VFTableOffset = SP->getVirtualIndex() * getPointerSizeInBytes();
 
       Methods.push_back(OneMethodRecord(
           MethodType, translateAccessFlags(Ty->getTag(), SP->getFlags()),
           translateMethodKindFlags(SP, Introduced),
           translateMethodOptionFlags(SP), VFTableOffset, Name));
       MemberCount++;
     }
     assert(!Methods.empty() && "Empty methods map entry");
     if (Methods.size() == 1)
-      FLBR.writeMemberType(Methods[0]);
+      ContinuationBuilder.writeMemberType(Methods[0]);
     else {
+      // FIXME: Make this use its own ContinuationBuilder so that
+      // MethodOverloadList can be split correctly.
       MethodOverloadListRecord MOLR(Methods);
-      TypeIndex MethodList = TypeTable.writeKnownType(MOLR);
+      TypeIndex MethodList = TypeTable.writeLeafType(MOLR);
+
       OverloadedMethodRecord OMR(Methods.size(), MethodList, Name);
-      FLBR.writeMemberType(OMR);
+      ContinuationBuilder.writeMemberType(OMR);
     }
   }
 
   // Create nested classes.
   for (const DIType *Nested : Info.NestedTypes) {
     NestedTypeRecord R(getTypeIndex(DITypeRef(Nested)), Nested->getName());
-    FLBR.writeMemberType(R);
+    ContinuationBuilder.writeMemberType(R);
     MemberCount++;
   }
 
-  TypeIndex FieldTI = FLBR.end(true);
+  TypeIndex FieldTI = TypeTable.insertRecord(ContinuationBuilder);
   return std::make_tuple(FieldTI, Info.VShapeTI, MemberCount,
                          !Info.NestedTypes.empty());
 }
 
 TypeIndex CodeViewDebug::getVBPTypeIndex() {
   if (!VBPType.getIndex()) {
     // Make a 'const int *' type.
     ModifierRecord MR(TypeIndex::Int32(), ModifierOptions::Const);
-    TypeIndex ModifiedTI = TypeTable.writeKnownType(MR);
+    TypeIndex ModifiedTI = TypeTable.writeLeafType(MR);
 
     PointerKind PK = getPointerSizeInBytes() == 8 ? PointerKind::Near64
                                                   : PointerKind::Near32;
     PointerMode PM = PointerMode::Pointer;
     PointerOptions PO = PointerOptions::None;
     PointerRecord PR(ModifiedTI, PK, PM, PO, getPointerSizeInBytes());
-
-    VBPType = TypeTable.writeKnownType(PR);
+    VBPType = TypeTable.writeLeafType(PR);
   }
 
   return VBPType;
 }
 
 TypeIndex CodeViewDebug::getTypeIndex(DITypeRef TypeRef, DITypeRef ClassTyRef) {
   const DIType *Ty = TypeRef.resolve();
   const DIType *ClassTy = ClassTyRef.resolve();

 
 TypeIndex CodeViewDebug::getTypeIndexForReferenceTo(DITypeRef TypeRef) {
   DIType *Ty = TypeRef.resolve();
   PointerRecord PR(getTypeIndex(Ty),
                    getPointerSizeInBytes() == 8 ? PointerKind::Near64
                                                 : PointerKind::Near32,
                    PointerMode::LValueReference, PointerOptions::None,
                    Ty->getSizeInBits() / 8);
-  return TypeTable.writeKnownType(PR);
+  return TypeTable.writeLeafType(PR);
 }
 
 TypeIndex CodeViewDebug::getCompleteTypeIndex(DITypeRef TypeRef) {
   const DIType *Ty = TypeRef.resolve();
 
   // The null DIType is the void type. Don't try to hash it.
   if (!Ty)
     return TypeIndex::Void();

llvm/trunk/lib/DebugInfo/CodeView/CMakeLists.txt

 add_llvm_library(LLVMDebugInfoCodeView
   CodeViewError.cpp
   CodeViewRecordIO.cpp
+  ContinuationRecordBuilder.cpp
   CVSymbolVisitor.cpp
   CVTypeVisitor.cpp
   DebugChecksumsSubsection.cpp
   DebugCrossExSubsection.cpp
   DebugCrossImpSubsection.cpp
   DebugFrameDataSubsection.cpp
   DebugInlineeLinesSubsection.cpp
   DebugLinesSubsection.cpp
   DebugStringTableSubsection.cpp
   DebugSubsection.cpp
   DebugSubsectionRecord.cpp
   DebugSubsectionVisitor.cpp
   DebugSymbolRVASubsection.cpp
   DebugSymbolsSubsection.cpp
   EnumTables.cpp
   Formatters.cpp
   LazyRandomTypeCollection.cpp
   Line.cpp
   RecordName.cpp
   RecordSerialization.cpp
+  SimpleTypeSerializer.cpp
   StringsAndChecksums.cpp
   SymbolRecordMapping.cpp
   SymbolDumper.cpp
   SymbolSerializer.cpp
   TypeDumpVisitor.cpp
   TypeIndex.cpp
   TypeIndexDiscovery.cpp
   TypeRecordMapping.cpp
-  TypeSerializer.cpp
+  TypeTableBuilder.cpp
   TypeStreamMerger.cpp
   TypeTableCollection.cpp
 
   ADDITIONAL_HEADER_DIRS
   ${LLVM_MAIN_INCLUDE_DIR}/llvm/DebugInfo/CodeView
   )

llvm/trunk/lib/DebugInfo/CodeView/ContinuationRecordBuilder.cpp

+#include "llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h"
+
+using namespace llvm;
+using namespace llvm::codeview;
+
+namespace {
+struct ContinuationRecord {
+  ulittle16_t Kind{uint16_t(TypeLeafKind::LF_INDEX)};
+  ulittle16_t Size{0};
+  ulittle32_t IndexRef{0xB0C0B0C0};
+};
+
+struct SegmentInjection {
+  SegmentInjection(TypeLeafKind Kind) { Prefix.RecordKind = Kind; }
+
+  ContinuationRecord Cont;
+  RecordPrefix Prefix;
+};
+} // namespace
+
+static void addPadding(BinaryStreamWriter &Writer) {
+  uint32_t Align = Writer.getOffset() % 4;
+  if (Align == 0)
+    return;
+
+  int PaddingBytes = 4 - Align;
+  while (PaddingBytes > 0) {
+    uint8_t Pad = static_cast<uint8_t>(LF_PAD0 + PaddingBytes);
+    cantFail(Writer.writeInteger(Pad));
+    --PaddingBytes;
+  }
+}
+
+static SegmentInjection InjectFieldList(TypeLeafKind::LF_FIELDLIST);
+static SegmentInjection InjectMethodOverloadList(TypeLeafKind::LF_METHODLIST);
+
+static constexpr uint32_t ContinuationLength = sizeof(ContinuationRecord);
+static constexpr uint32_t MaxSegmentLength =
+    MaxRecordLength - ContinuationLength;
+
+static inline TypeLeafKind getTypeLeafKind(ContinuationRecordKind CK) {
+  return (CK == ContinuationRecordKind::FieldList) ? LF_FIELDLIST
+                                                   : LF_METHODLIST;
+}
+
+ContinuationRecordBuilder::ContinuationRecordBuilder()
+    : SegmentWriter(Buffer), Mapping(SegmentWriter) {}
+
+ContinuationRecordBuilder::~ContinuationRecordBuilder() {}
+
+void ContinuationRecordBuilder::begin(ContinuationRecordKind RecordKind) {
+  assert(!Kind.hasValue());
+  Kind = RecordKind;
+  Buffer.clear();
+  SegmentWriter.setOffset(0);
+  SegmentOffsets.clear();
+  SegmentOffsets.push_back(0);
+  assert(SegmentWriter.getOffset() == 0);
+  assert(SegmentWriter.getLength() == 0);
+
+  const SegmentInjection *FLI =
+      (RecordKind == ContinuationRecordKind::FieldList)
+          ? &InjectFieldList
+          : &InjectMethodOverloadList;
+  const uint8_t *FLIB = reinterpret_cast<const uint8_t *>(FLI);
+  InjectedSegmentBytes =
+      ArrayRef<uint8_t>(FLIB, FLIB + sizeof(SegmentInjection));
+
+  CVType Type;
+  Type.Type = getTypeLeafKind(RecordKind);
+  cantFail(Mapping.visitTypeBegin(Type));
+
+  // Seed the first trecord with an appropriate record prefix.
+  RecordPrefix Prefix;
+  Prefix.RecordLen = 0;
+  Prefix.RecordKind = Type.Type;
+  cantFail(SegmentWriter.writeObject(Prefix));
+}
+
+template <typename RecordType>
+void ContinuationRecordBuilder::writeMemberType(RecordType &Record) {
+  assert(Kind.hasValue());
+
+  uint32_t OriginalOffset = SegmentWriter.getOffset();
+  CVMemberRecord CVMR;
+  CVMR.Kind = static_cast<TypeLeafKind>(Record.getKind());
+
+  // Member Records aren't length-prefixed, they only have a 2-byte TypeLeafKind
+  // at the beginning.
+  cantFail(SegmentWriter.writeEnum(CVMR.Kind));
+
+  // Let the Mapping handle the rest.
+  cantFail(Mapping.visitMemberBegin(CVMR));
+  cantFail(Mapping.visitKnownMember(CVMR, Record));
+  cantFail(Mapping.visitMemberEnd(CVMR));
+
+  // Make sure it's padded to 4 bytes.
+  addPadding(SegmentWriter);
+  assert(getCurrentSegmentLength() % 4 == 0);
+
+  // The maximum length of a single segment is 64KB minus the size to insert a
+  // continuation.  So if we are over that, inject a continuation between the
+  // previous member and the member that was just written, then end the previous
+  // segment after the continuation and begin a new one with the just-written
+  // member.
+  if (getCurrentSegmentLength() > MaxSegmentLength) {
+    // We need to inject some bytes before the member we just wrote but after
+    // the previous member.  Save off the length of the member we just wrote so
+    // that we can do some sanity checking on it.
+    uint32_t MemberLength = SegmentWriter.getOffset() - OriginalOffset;
+    insertSegmentEnd(OriginalOffset);
+    // Since this member now becomes a new top-level record, it should have
+    // gotten a RecordPrefix injected, and that RecordPrefix + the member we
+    // just wrote should now constitute the entirety of the current "new"
+    // segment.
+    assert(getCurrentSegmentLength() == MemberLength + sizeof(RecordPrefix));
+  }
+
+  assert(getCurrentSegmentLength() % 4 == 0);
+  assert(getCurrentSegmentLength() <= MaxSegmentLength);
+}
+
+uint32_t ContinuationRecordBuilder::getCurrentSegmentLength() const {
+  return SegmentWriter.getOffset() - SegmentOffsets.back();
+}
+
+void ContinuationRecordBuilder::insertSegmentEnd(uint32_t Offset) {
+  uint32_t SegmentBegin = SegmentOffsets.back();
+  assert(Offset > SegmentBegin);
+  assert(Offset - SegmentBegin <= MaxSegmentLength);
+
+  // We need to make space for the continuation record.  For now we can't fill
+  // out the length or the TypeIndex of the back-reference, but we need the
+  // space to at least be there.
+  Buffer.insert(Offset, InjectedSegmentBytes);
+
+  uint32_t NewSegmentBegin = Offset + ContinuationLength;
+  uint32_t SegmentLength = NewSegmentBegin - SegmentOffsets.back();
+
+  assert(SegmentLength % 4 == 0);
+  assert(SegmentLength <= MaxRecordLength);
+  SegmentOffsets.push_back(NewSegmentBegin);
+
+  // Seek to the end so that we can keep writing against the new segment.
+  SegmentWriter.setOffset(SegmentWriter.getLength());
+  assert(SegmentWriter.bytesRemaining() == 0);
+}
+
+CVType ContinuationRecordBuilder::createSegmentRecord(
+    uint32_t OffBegin, uint32_t OffEnd, Optional<TypeIndex> RefersTo) {
+  assert(OffEnd - OffBegin <= USHRT_MAX);
+
+  MutableArrayRef<uint8_t> Data = Buffer.data();
+  Data = Data.slice(OffBegin, OffEnd - OffBegin);
+
+  CVType Type;
+  Type.Type = getTypeLeafKind(*Kind);
+  Type.RecordData = Data;
+
+  // Write the length to the RecordPrefix, making sure it does not include
+  // sizeof(RecordPrefix.Length)
+  RecordPrefix *Prefix = reinterpret_cast<RecordPrefix *>(Data.data());
+  assert(Prefix->RecordKind == Type.Type);
+  Prefix->RecordLen = Data.size() - sizeof(RecordPrefix::RecordLen);
+
+  if (RefersTo.hasValue()) {
+    auto Continuation = Data.take_back(ContinuationLength);
+    ContinuationRecord *CR =
+        reinterpret_cast<ContinuationRecord *>(Continuation.data());
+    assert(CR->Kind == TypeLeafKind::LF_INDEX);
+    assert(CR->IndexRef == 0xB0C0B0C0);
+    CR->IndexRef = RefersTo->getIndex();
+  }
+
+  return Type;
+}
+
+std::vector<CVType> ContinuationRecordBuilder::end(TypeIndex Index) {
+  CVType Type;
+  Type.Type = getTypeLeafKind(*Kind);
+  cantFail(Mapping.visitTypeEnd(Type));
+
+  // We're now done, and we have a series of segments each beginning at an
+  // offset specified in the SegmentOffsets array.  We now need to iterate
+  // over each segment and post-process them in the following two ways:
+  // 1) Each top-level record has a RecordPrefix whose type is either
+  //    LF_FIELDLIST or LF_METHODLIST, but the Length field is still 0.
+  //    Those should all be set to the correct length now.
+  // 2) Each continuation record has an IndexRef field which we set to the
+  //    magic value 0xB0C0B0C0.  Now that the caller has told us the TypeIndex
+  //    they want this sequence to start from, we can go through and update
+  //    each one.
+  //
+  // Logically, the sequence of records we've built up looks like this:
+  //
+  // SegmentOffsets[0]:   <Length>                    (Initially: uninitialized)
+  // SegmentOffsets[0]+2: LF_FIELDLIST
+  // SegmentOffsets[0]+4: Member[0]
+  // SegmentOffsets[0]+?: ...
+  // SegmentOffsets[0]+?: Member[4]
+  // SegmentOffsets[1]-8: LF_INDEX
+  // SegmentOffsets[1]-6: 0
+  // SegmentOffsets[1]-4: <Type Index of Next Record> (Initially: 0xB0C0B0C0)
+  //
+  // SegmentOffsets[1]:   <Length>                    (Initially: uninitialized)
+  // SegmentOffsets[1]+2: LF_FIELDLIST
+  // SegmentOffsets[1]+4: Member[0]
+  // SegmentOffsets[1]+?: ...
+  // SegmentOffsets[1]+?: Member[s]
+  // SegmentOffsets[2]-8: LF_INDEX
+  // SegmentOffsets[2]-6: 0
+  // SegmentOffsets[2]-4: <Type Index of Next Record> (Initially: 0xB0C0B0C0)
+  //
+  // ...
+  //
+  // SegmentOffsets[N]:   <Length>                    (Initially: uninitialized)
+  // SegmentOffsets[N]+2: LF_FIELDLIST
+  // SegmentOffsets[N]+4: Member[0]
+  // SegmentOffsets[N]+?: ...
+  // SegmentOffsets[N]+?: Member[t]
+  //
+  // And this is the way we have laid them out in the serialization buffer.  But
+  // we cannot actually commit them to the underlying stream this way, due to
+  // the topological sorting requirement of a type stream (specifically,
+  // TypeIndex references can only point backwards, not forwards).  So the
+  // sequence that we return to the caller contains the records in reverse
+  // order, which is the proper order for committing the serialized records.
+
+  std::vector<CVType> Types;
+  Types.reserve(SegmentOffsets.size());
+
+  auto SO = makeArrayRef(SegmentOffsets);
+
+  uint32_t End = SegmentWriter.getOffset();
+
+  Optional<TypeIndex> RefersTo;
+  for (uint32_t Offset : reverse(SO)) {
+    Types.push_back(createSegmentRecord(Offset, End, RefersTo));
+
+    End = Offset;
+    RefersTo = Index++;
+  }
+
+  Kind.reset();
+  return Types;
+}
+
+// Explicitly instantiate the member function for each known type so that we can
+// implement this in the cpp file.
+#define TYPE_RECORD(EnumName, EnumVal, Name)
+#define TYPE_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#define MEMBER_RECORD(EnumName, EnumVal, Name)                                 \
+  template void llvm::codeview::ContinuationRecordBuilder::writeMemberType(    \
+      Name##Record &Record);
+#define MEMBER_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#include "llvm/DebugInfo/CodeView/CodeViewTypes.def"

llvm/trunk/lib/DebugInfo/CodeView/SimpleTypeSerializer.cpp

+#include "llvm/DebugInfo/CodeView/SimpleTypeSerializer.h"
+
+using namespace llvm;
+using namespace llvm::codeview;
+
+static void writeRecordPrefix(BinaryStreamWriter &Writer, TypeLeafKind Kind) {
+  RecordPrefix Prefix;
+  Prefix.RecordKind = Kind;
+  Prefix.RecordLen = 0;
+  cantFail(Writer.writeObject(Prefix));
+}
+
+static void addPadding(BinaryStreamWriter &Writer) {
+  uint32_t Align = Writer.getOffset() % 4;
+  if (Align == 0)
+    return;
+
+  int PaddingBytes = 4 - Align;
+  while (PaddingBytes > 0) {
+    uint8_t Pad = static_cast<uint8_t>(LF_PAD0 + PaddingBytes);
+    cantFail(Writer.writeInteger(Pad));
+    --PaddingBytes;
+  }
+}
+
+SimpleTypeSerializer::SimpleTypeSerializer() : ScratchBuffer(MaxRecordLength) {}
+
+SimpleTypeSerializer::~SimpleTypeSerializer() {}
+
+template <typename T>
+ArrayRef<uint8_t> SimpleTypeSerializer::serialize(T &Record) {
+  BinaryStreamWriter Writer(ScratchBuffer, support::little);
+  TypeRecordMapping Mapping(Writer);
+
+  CVType CVT;
+  CVT.Type = static_cast<TypeLeafKind>(Record.getKind());
+
+  writeRecordPrefix(Writer, CVT.Type);
+
+  cantFail(Mapping.visitTypeBegin(CVT));
+  cantFail(Mapping.visitKnownRecord(CVT, Record));
+  cantFail(Mapping.visitTypeEnd(CVT));
+
+  addPadding(Writer);
+
+  RecordPrefix *Prefix = reinterpret_cast<RecordPrefix *>(ScratchBuffer.data());
+
+  Prefix->RecordKind = CVT.kind();
+  Prefix->RecordLen = Writer.getOffset() - sizeof(uint16_t);
+
+  return {ScratchBuffer.data(), Writer.getOffset()};
+}
+
+// Explicitly instantiate the member function for each known type so that we can
+// implement this in the cpp file.
+#define TYPE_RECORD(EnumName, EnumVal, Name)                                   \
+  template ArrayRef<uint8_t> llvm::codeview::SimpleTypeSerializer::serialize(  \
+      Name##Record &Record);
+#define TYPE_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#define MEMBER_RECORD(EnumName, EnumVal, Name)
+#define MEMBER_RECORD_ALIAS(EnumName, EnumVal, Name, AliasName)
+#include "llvm/DebugInfo/CodeView/CodeViewTypes.def"

llvm/trunk/lib/DebugInfo/CodeView/TypeRecordMapping.cpp

Error TypeRecordMapping::visitKnownMember(CVMemberRecord &CVR,
   error(IO.mapInteger(Record.MethodList));
   error(IO.mapStringZ(Record.Name));
 
   return Error::success();
 }
 
 Error TypeRecordMapping::visitKnownMember(CVMemberRecord &CVR,
                                           OneMethodRecord &Record) {
-  MapOneMethodRecord Mapper(false);
+  const bool IsFromOverloadList = (TypeKind == LF_METHODLIST);
+  MapOneMethodRecord Mapper(IsFromOverloadList);
   return Mapper(IO, Record);
 }
 
 Error TypeRecordMapping::visitKnownMember(CVMemberRecord &CVR,
                                           NestedTypeRecord &Record) {
   uint16_t Padding = 0;
   error(IO.mapInteger(Padding));
   error(IO.mapInteger(Record.Type));

llvm/trunk/lib/DebugInfo/CodeView/TypeSerializer.cpp

-//===- TypeSerialzier.cpp -------------------------------------------------===//
-//
-//                     The LLVM Compiler Infrastructure
-//
-// This file is distributed under the University of Illinois Open Source
-// License. See LICENSE.TXT for details.
-//
-//===----------------------------------------------------------------------===//
-
-#include "llvm/DebugInfo/CodeView/TypeSerializer.h"
-#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/DenseSet.h"
-#include "llvm/ADT/STLExtras.h"
-#include "llvm/DebugInfo/CodeView/CodeView.h"
-#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
-#include "llvm/DebugInfo/CodeView/TypeIndex.h"
-#include "llvm/Support/Allocator.h"
-#include "llvm/Support/BinaryByteStream.h"
-#include "llvm/Support/BinaryStreamWriter.h"
-#include "llvm/Support/Endian.h"
-#include "llvm/Support/Error.h"
-#include <algorithm>
-#include <cassert>
-#include <cstdint>
-#include <cstring>
-
-using namespace llvm;
-using namespace llvm::codeview;
-
-namespace {
-
-struct HashedType {
-  uint64_t Hash;
-  const uint8_t *Data;
-  unsigned Size; // FIXME: Go to uint16_t?
-  TypeIndex Index;
-};
-
-/// Wrapper around a poitner to a HashedType. Hash and equality operations are
-/// based on data in the pointee.
-struct HashedTypePtr {
-  HashedTypePtr() = default;
-  HashedTypePtr(HashedType *Ptr) : Ptr(Ptr) {}
-
-  HashedType *Ptr = nullptr;
-};
-
-} // end anonymous namespace
-
-namespace llvm {
-
-template <> struct DenseMapInfo<HashedTypePtr> {
-  static inline HashedTypePtr getEmptyKey() { return HashedTypePtr(nullptr); }
-
-  static inline HashedTypePtr getTombstoneKey() {
-    return HashedTypePtr(reinterpret_cast<HashedType *>(1));
-  }
-
-  static unsigned getHashValue(HashedTypePtr Val) {
-    assert(Val.Ptr != getEmptyKey().Ptr && Val.Ptr != getTombstoneKey().Ptr);
-    return Val.Ptr->Hash;
-  }
-
-  static bool isEqual(HashedTypePtr LHSP, HashedTypePtr RHSP) {
-    HashedType *LHS = LHSP.Ptr;
-    HashedType *RHS = RHSP.Ptr;
-    if (RHS == getEmptyKey().Ptr || RHS == getTombstoneKey().Ptr)
-      return LHS == RHS;
-    if (LHS->Hash != RHS->Hash || LHS->Size != RHS->Size)
-      return false;
-    return ::memcmp(LHS->Data, RHS->Data, LHS->Size) == 0;
-  }
-};
-
-} // end namespace llvm
-
-/// Private implementation so that we don't leak our DenseMap instantiations to
-/// users.
-class llvm::codeview::TypeHasher {
-private:
-  /// Storage for type record provided by the caller. Records will outlive the
-  /// hasher object, so they should be allocated here.
-  BumpPtrAllocator &RecordStorage;
-
-  /// Storage for hash keys. These only need to live as long as the hashing
-  /// operation.
-  BumpPtrAllocator KeyStorage;
-
-  /// Hash table. We really want a DenseMap<ArrayRef<uint8_t>, TypeIndex> here,
-  /// but DenseMap is inefficient when the keys are long (like type records)
-  /// because it recomputes the hash value of every key when it grows. This
-  /// value type stores the hash out of line in KeyStorage, so that table
-  /// entries are small and easy to rehash.
-  DenseSet<HashedTypePtr> HashedRecords;
-
-public:
-  TypeHasher(BumpPtrAllocator &RecordStorage) : RecordStorage(RecordStorage) {}
-
-  void reset() { HashedRecords.clear(); }
-
-  /// Takes the bytes of type record, inserts them into the hash table, saves
-  /// them, and returns a pointer to an identical stable type record along with
-  /// its type index in the destination stream.
-  TypeIndex getOrCreateRecord(ArrayRef<uint8_t> &Record, TypeIndex TI);
-};
-
-TypeIndex TypeHasher::getOrCreateRecord(ArrayRef<uint8_t> &Record,
-                                        TypeIndex TI) {
-  assert(Record.size() < UINT32_MAX && "Record too big");
-  assert(Record.size() % 4 == 0 && "Record is not aligned to 4 bytes!");
-
-  // Compute the hash up front so we can store it in the key.
-  HashedType TempHashedType = {hash_value(Record), Record.data(),
-                               unsigned(Record.size()), TI};
-  auto Result = HashedRecords.insert(HashedTypePtr(&TempHashedType));
-  HashedType *&Hashed = Result.first->Ptr;
-
-  if (Result.second) {
-    // This was a new type record. We need stable storage for both the key and
-    // the record. The record should outlive the hashing operation.
-    Hashed = KeyStorage.Allocate<HashedType>();
-    *Hashed = TempHashedType;
-
-    uint8_t *Stable = RecordStorage.Allocate<uint8_t>(Record.size());
-    memcpy(Stable, Record.data(), Record.size());
-    Hashed->Data = Stable;
-    assert(Hashed->Size == Record.size());
-  }
-
-  // Update the caller's copy of Record to point a stable copy.
-  Record = ArrayRef<uint8_t>(Hashed->Data, Hashed->Size);
-  return Hashed->Index;
-}
-
-TypeIndex TypeSerializer::nextTypeIndex() const {
-  return TypeIndex::fromArrayIndex(SeenRecords.size());
-}
-
-bool TypeSerializer::isInFieldList() const {
-  return TypeKind.hasValue() && *TypeKind == TypeLeafKind::LF_FIELDLIST;
-}
-
-MutableArrayRef<uint8_t> TypeSerializer::getCurrentSubRecordData() {
-  assert(isInFieldList());
-  return getCurrentRecordData().drop_front(CurrentSegment.length());
-}
-
-MutableArrayRef<uint8_t> TypeSerializer::getCurrentRecordData() {
-  return MutableArrayRef<uint8_t>(RecordBuffer).take_front(Writer.getOffset());
-}
-
-Error TypeSerializer::writeRecordPrefix(TypeLeafKind Kind) {
-  RecordPrefix Prefix;
-  Prefix.RecordKind = Kind;
-  Prefix.RecordLen = 0;
-  if (auto EC = Writer.writeObject(Prefix))
-    return EC;
-  return Error::success();
-}
-
-Expected<MutableArrayRef<uint8_t>>
-TypeSerializer::addPadding(MutableArrayRef<uint8_t> Record) {
-  uint32_t Align = Record.size() % 4;
-  if (Align == 0)
-    return Record;
-
-  int PaddingBytes = 4 - Align;
-  int N = PaddingBytes;
-  while (PaddingBytes > 0) {
-    uint8_t Pad = static_cast<uint8_t>(LF_PAD0 + PaddingBytes);
-    if (auto EC = Writer.writeInteger(Pad))
-      return std::move(EC);
-    --PaddingBytes;
-  }
-  return MutableArrayRef<uint8_t>(Record.data(), Record.size() + N);
-}
-
-TypeSerializer::TypeSerializer(BumpPtrAllocator &Storage, bool Hash)
-    : RecordStorage(Storage), RecordBuffer(MaxRecordLength * 2),
-      Stream(RecordBuffer, support::little), Writer(Stream),
-      Mapping(Writer) {
-  // RecordBuffer needs to be able to hold enough data so that if we are 1
-  // byte short of MaxRecordLen, and then we try to write MaxRecordLen bytes,
-  // we won't overflow.
-  if (Hash)
-    Hasher = llvm::make_unique<TypeHasher>(Storage);
-}
-
-TypeSerializer::~TypeSerializer() = default;
-
-ArrayRef<ArrayRef<uint8_t>> TypeSerializer::records() const {
-  return SeenRecords;
-}
-
-void TypeSerializer::reset() {
-  if (Hasher)
-    Hasher->reset();
-  Writer.setOffset(0);
-  CurrentSegment = RecordSegment();
-  FieldListSegments.clear();
-  TypeKind.reset();
-  MemberKind.reset();
-  SeenRecords.clear();
-}
-
-TypeIndex TypeSerializer::insertRecordBytes(ArrayRef<uint8_t> &Record) {
-  assert(!TypeKind.hasValue() && "Already in a type mapping!");
-  assert(Writer.getOffset() == 0 && "Stream has data already!");
-
-  if (Hasher) {
-    TypeIndex ActualTI = Hasher->getOrCreateRecord(Record, nextTypeIndex());
-    if (nextTypeIndex() == ActualTI)
-      SeenRecords.push_back(Record);
-    return ActualTI;
-  }
-
-  TypeIndex NewTI = nextTypeIndex();
-  uint8_t *Stable = RecordStorage.Allocate<uint8_t>(Record.size());
-  memcpy(Stable, Record.data(), Record.size());
-  Record = ArrayRef<uint8_t>(Stable, Record.size());
-  SeenRecords.push_back(Record);
-  return NewTI;
-}
-
-TypeIndex TypeSerializer::insertRecord(const RemappedType &Record) {
-  assert(!TypeKind.hasValue() && "Already in a type mapping!");
-  assert(Writer.getOffset() == 0 && "Stream has data already!");
-
-  TypeIndex TI;
-  ArrayRef<uint8_t> OriginalData = Record.OriginalRecord.RecordData;
-  if (Record.Mappings.empty()) {
-    // This record did not remap any type indices.  Just write it.
-    return insertRecordBytes(OriginalData);
-  }
-
-  // At least one type index was remapped.  Before we can hash it we have to
-  // copy the full record bytes, re-write each type index, then hash the copy.
-  // We do this in temporary storage since only the DenseMap can decide whether
-  // this record already exists, and if it does we don't want the memory to
-  // stick around.
-  RemapStorage.resize(OriginalData.size());
-  ::memcpy(&RemapStorage[0], OriginalData.data(), OriginalData.size());
-  uint8_t *ContentBegin = RemapStorage.data() + sizeof(RecordPrefix);
-  for (const auto &M : Record.Mappings) {
-    // First 4 bytes of every record are the record prefix, but the mapping
-    // offset is relative to the content which starts after.
-    *(TypeIndex *)(ContentBegin + M.first) = M.second;
-  }
-  auto RemapRef = makeArrayRef(RemapStorage);
-  return insertRecordBytes(RemapRef);
-}
-
-Error TypeSerializer::visitTypeBegin(CVType &Record) {
-  assert(!TypeKind.hasValue() && "Already in a type mapping!");
-  assert(Writer.getOffset() == 0 && "Stream has data already!");
-
-  if (auto EC = writeRecordPrefix(Record.kind()))
-    return EC;
-
-  TypeKind = Record.kind();
-  if (auto EC = Mapping.visitTypeBegin(Record))
-    return EC;
-
-  return Error::success();
-}
-
-Expected<TypeIndex> TypeSerializer::visitTypeEndGetIndex(CVType &Record) {
-  assert(TypeKind.hasValue() && "Not in a type mapping!");
-  if (auto EC = Mapping.visitTypeEnd(Record))
-    return std::move(EC);
-
-  // Update the record's length and fill out the CVType members to point to
-  // the stable memory holding the record's data.
-  auto ThisRecordData = getCurrentRecordData();
-  auto ExpectedData = addPadding(ThisRecordData);
-  if (!ExpectedData)
-    return ExpectedData.takeError();
-  ThisRecordData = *ExpectedData;
-
-  RecordPrefix *Prefix =
-      reinterpret_cast<RecordPrefix *>(ThisRecordData.data());
-  Prefix->RecordLen = ThisRecordData.size() - sizeof(uint16_t);
-
-  Record.Type = *TypeKind;
-  Record.RecordData = ThisRecordData;
-
-  // insertRecordBytes assumes we're not in a mapping, so do this first.
-  TypeKind.reset();
-  Writer.setOffset(0);
-
-  TypeIndex InsertedTypeIndex = insertRecordBytes(Record.RecordData);
-
-  // Write out each additional segment in reverse order, and update each
-  // record's continuation index to point to the previous one.
-  for (auto X : reverse(FieldListSegments)) {
-    auto CIBytes = X.take_back(sizeof(uint32_t));
-    support::ulittle32_t *CI =
-        reinterpret_cast<support::ulittle32_t *>(CIBytes.data());
-    assert(*CI == 0xB0C0B0C0 && "Invalid TypeIndex placeholder");
-    *CI = InsertedTypeIndex.getIndex();
-    InsertedTypeIndex = insertRecordBytes(X);
-  }
-
-  FieldListSegments.clear();
-  CurrentSegment.SubRecords.clear();
-
-  return InsertedTypeIndex;
-}
-
-Error TypeSerializer::visitTypeEnd(CVType &Record) {
-  auto ExpectedIndex = visitTypeEndGetIndex(Record);
-  if (!ExpectedIndex)
-    return ExpectedIndex.takeError();
-  return Error::success();
-}
-
-Error TypeSerializer::visitMemberBegin(CVMemberRecord &Record) {
-  assert(isInFieldList() && "Not in a field list!");
-  assert(!MemberKind.hasValue() && "Already in a member record!");
-  MemberKind = Record.Kind;
-
-  if (auto EC = Mapping.visitMemberBegin(Record))
-    return EC;
-
-  return Error::success();
-}
-
-Error TypeSerializer::visitMemberEnd(CVMemberRecord &Record) {
-  if (auto EC = Mapping.visitMemberEnd(Record))
-    return EC;
-
-  // Check if this subrecord makes the current segment not fit in 64K minus
-  // the space for a continuation record (8 bytes). If the segment does not
-  // fit, insert a continuation record.
-  if (Writer.getOffset() > MaxRecordLength - ContinuationLength) {
-    MutableArrayRef<uint8_t> Data = getCurrentRecordData();
-    SubRecord LastSubRecord = CurrentSegment.SubRecords.back();
-    uint32_t CopySize = CurrentSegment.length() - LastSubRecord.Size;
-    auto CopyData = Data.take_front(CopySize);
-    auto LeftOverData = Data.drop_front(CopySize);
-    assert(LastSubRecord.Size == LeftOverData.size());
-
-    // Allocate stable storage for the record and copy the old record plus
-    // continuation over.
-    uint16_t LengthWithSize = CopySize + ContinuationLength;
-    assert(LengthWithSize <= MaxRecordLength);
-    RecordPrefix *Prefix = reinterpret_cast<RecordPrefix *>(CopyData.data());
-    Prefix->RecordLen = LengthWithSize - sizeof(uint16_t);
-
-    uint8_t *SegmentBytes = RecordStorage.Allocate<uint8_t>(LengthWithSize);
-    auto SavedSegment = MutableArrayRef<uint8_t>(SegmentBytes, LengthWithSize);
-    MutableBinaryByteStream CS(SavedSegment, support::little);
-    BinaryStreamWriter CW(CS);
-    if (auto EC = CW.writeBytes(CopyData))
-      return EC;
-    if (auto EC = CW.writeEnum(TypeLeafKind::LF_INDEX))
-      return EC;
-    if (auto EC = CW.writeInteger<uint16_t>(0))
-      return EC;
-    if (auto EC = CW.writeInteger<uint32_t>(0xB0C0B0C0))
-      return EC;
-    FieldListSegments.push_back(SavedSegment);
-
-    // Write a new placeholder record prefix to mark the start of this new
-    // top-level record.
-    Writer.setOffset(0);
-    if (auto EC = writeRecordPrefix(TypeLeafKind::LF_FIELDLIST))
-      return EC;
-
-    // Then move over the subrecord that overflowed the old segment to the
-    // beginning of this segment.  Note that we have to use memmove here
-    // instead of Writer.writeBytes(), because the new and old locations
-    // could overlap.
-    ::memmove(Stream.data().data() + sizeof(RecordPrefix), LeftOverData.data(),
-              LeftOverData.size());
-    // And point the segment writer at the end of that subrecord.
-    Writer.setOffset(LeftOverData.size() + sizeof(RecordPrefix));
-
-    CurrentSegment.SubRecords.clear();
-    CurrentSegment.SubRecords.push_back(LastSubRecord);
-  }
-
-  // Update the CVMemberRecord since we may have shifted around or gotten
-  // padded.
-  Record.Data = getCurrentSubRecordData();
-
-  MemberKind.reset();
-  return Error::success();
-}

llvm/trunk/lib/DebugInfo/CodeView/TypeStreamMerger.cpp

private:
   Error errorCorruptRecord() const {
     return llvm::make_error<CodeViewError>(cv_error_code::corrupt_record);
   }
 
   Error writeRecord(TypeTableBuilder &Dest, const RemappedType &Record,
                     bool RemapSuccess) {
     TypeIndex DestIdx = Untranslated;
     if (RemapSuccess)
-      DestIdx = Dest.writeSerializedRecord(Record);
+      DestIdx = Dest.insertRecord(Record);
     addMapping(DestIdx);
     return Error::success();
   }
 
   Optional<Error> LastError;
 
   bool IsSecondPass = false;
 

llvm/trunk/lib/DebugInfo/CodeView/TypeTableBuilder.cpp

+//===- TypeSerialzier.cpp -------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is distributed under the University of Illinois Open Source
+// License. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+#include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
+#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseSet.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/DebugInfo/CodeView/CodeView.h"
+#include "llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h"
+#include "llvm/DebugInfo/CodeView/RecordSerialization.h"
+#include "llvm/DebugInfo/CodeView/TypeIndex.h"
+#include "llvm/Support/Allocator.h"
+#include "llvm/Support/BinaryByteStream.h"
+#include "llvm/Support/BinaryStreamWriter.h"
+#include "llvm/Support/Endian.h"
+#include "llvm/Support/Error.h"
+#include <algorithm>
+#include <cassert>
+#include <cstdint>
+#include <cstring>
+
+using namespace llvm;
+using namespace llvm::codeview;
+
+namespace {
+
+struct HashedType {
+  uint64_t Hash;
+  const uint8_t *Data;
+  unsigned Size; // FIXME: Go to uint16_t?
+  TypeIndex Index;
+};
+
+/// Wrapper around a poitner to a HashedType. Hash and equality operations are
+/// based on data in the pointee.
+struct HashedTypePtr {
+  HashedTypePtr() = default;
+  HashedTypePtr(HashedType *Ptr) : Ptr(Ptr) {}
+
+  HashedType *Ptr = nullptr;
+};
+
+} // end anonymous namespace
+
+namespace llvm {
+
+template <> struct DenseMapInfo<HashedTypePtr> {
+  static inline HashedTypePtr getEmptyKey() { return HashedTypePtr(nullptr); }
+
+  static inline HashedTypePtr getTombstoneKey() {
+    return HashedTypePtr(reinterpret_cast<HashedType *>(1));
+  }
+
+  static unsigned getHashValue(HashedTypePtr Val) {
+    assert(Val.Ptr != getEmptyKey().Ptr && Val.Ptr != getTombstoneKey().Ptr);
+    return Val.Ptr->Hash;
+  }
+
+  static bool isEqual(HashedTypePtr LHSP, HashedTypePtr RHSP) {
+    HashedType *LHS = LHSP.Ptr;
+    HashedType *RHS = RHSP.Ptr;
+    if (RHS == getEmptyKey().Ptr || RHS == getTombstoneKey().Ptr)
+      return LHS == RHS;
+    if (LHS->Hash != RHS->Hash || LHS->Size != RHS->Size)
+      return false;
+    return ::memcmp(LHS->Data, RHS->Data, LHS->Size) == 0;
+  }
+};
+
+} // end namespace llvm
+
+/// Private implementation so that we don't leak our DenseMap instantiations to
+/// users.
+class llvm::codeview::TypeHasher {
+private:
+  /// Storage for type record provided by the caller. Records will outlive the
+  /// hasher object, so they should be allocated here.
+  BumpPtrAllocator &RecordStorage;
+
+  /// Storage for hash keys. These only need to live as long as the hashing
+  /// operation.
+  BumpPtrAllocator KeyStorage;
+
+  /// Hash table. We really want a DenseMap<ArrayRef<uint8_t>, TypeIndex> here,
+  /// but DenseMap is inefficient when the keys are long (like type records)
+  /// because it recomputes the hash value of every key when it grows. This
+  /// value type stores the hash out of line in KeyStorage, so that table
+  /// entries are small and easy to rehash.
+  DenseSet<HashedTypePtr> HashedRecords;
+
+public:
+  TypeHasher(BumpPtrAllocator &RecordStorage) : RecordStorage(RecordStorage) {}
+
+  void reset() { HashedRecords.clear(); }
+
+  /// Takes the bytes of type record, inserts them into the hash table, saves
+  /// them, and returns a pointer to an identical stable type record along with
+  /// its type index in the destination stream.
+  TypeIndex getOrCreateRecord(ArrayRef<uint8_t> &Record, TypeIndex TI);
+};
+
+TypeIndex TypeHasher::getOrCreateRecord(ArrayRef<uint8_t> &Record,
+                                        TypeIndex TI) {
+  assert(Record.size() < UINT32_MAX && "Record too big");
+  assert(Record.size() % 4 == 0 && "Record is not aligned to 4 bytes!");
+
+  // Compute the hash up front so we can store it in the key.
+  HashedType TempHashedType = {hash_value(Record), Record.data(),
+                               unsigned(Record.size()), TI};
+  auto Result = HashedRecords.insert(HashedTypePtr(&TempHashedType));
+  HashedType *&Hashed = Result.first->Ptr;
+
+  if (Result.second) {
+    // This was a new type record. We need stable storage for both the key and
+    // the record. The record should outlive the hashing operation.
+    Hashed = KeyStorage.Allocate<HashedType>();
+    *Hashed = TempHashedType;
+
+    uint8_t *Stable = RecordStorage.Allocate<uint8_t>(Record.size());
+    memcpy(Stable, Record.data(), Record.size());
+    Hashed->Data = Stable;
+    assert(Hashed->Size == Record.size());
+  }
+
+  // Update the caller's copy of Record to point a stable copy.
+  Record = ArrayRef<uint8_t>(Hashed->Data, Hashed->Size);
+  return Hashed->Index;
+}
+
+TypeIndex TypeTableBuilder::nextTypeIndex() const {
+  return TypeIndex::fromArrayIndex(SeenRecords.size());
+}
+
+TypeTableBuilder::TypeTableBuilder(BumpPtrAllocator &Storage, bool Hash)
+    : RecordStorage(Storage) {
+  if (Hash)
+    Hasher = llvm::make_unique<TypeHasher>(Storage);
+}
+
+TypeTableBuilder::~TypeTableBuilder() = default;
+
+ArrayRef<ArrayRef<uint8_t>> TypeTableBuilder::records() const {
+  return SeenRecords;
+}
+
+void TypeTableBuilder::reset() {
+  if (Hasher)
+    Hasher->reset();
+  SeenRecords.clear();
+}
+
+TypeIndex TypeTableBuilder::insertRecordBytes(ArrayRef<uint8_t> &Record) {
+  if (Hasher) {
+    TypeIndex ActualTI = Hasher->getOrCreateRecord(Record, nextTypeIndex());
+    if (nextTypeIndex() == ActualTI)
+      SeenRecords.push_back(Record);
+    return ActualTI;
+  }
+
+  TypeIndex NewTI = nextTypeIndex();
+  uint8_t *Stable = RecordStorage.Allocate<uint8_t>(Record.size());
+  memcpy(Stable, Record.data(), Record.size());
+  Record = ArrayRef<uint8_t>(Stable, Record.size());
+  SeenRecords.push_back(Record);
+  return NewTI;
+}
+
+TypeIndex TypeTableBuilder::insertRecord(const RemappedType &Record) {
+  TypeIndex TI;
+  ArrayRef<uint8_t> OriginalData = Record.OriginalRecord.RecordData;
+  if (Record.Mappings.empty()) {
+    // This record did not remap any type indices.  Just write it.
+    return insertRecordBytes(OriginalData);
+  }
+
+  // At least one type index was remapped.  Before we can hash it we have to
+  // copy the full record bytes, re-write each type index, then hash the copy.
+  // We do this in temporary storage since only the DenseMap can decide whether
+  // this record already exists, and if it does we don't want the memory to
+  // stick around.
+  RemapStorage.resize(OriginalData.size());
+  ::memcpy(&RemapStorage[0], OriginalData.data(), OriginalData.size());
+  uint8_t *ContentBegin = RemapStorage.data() + sizeof(RecordPrefix);
+  for (const auto &M : Record.Mappings) {
+    // First 4 bytes of every record are the record prefix, but the mapping
+    // offset is relative to the content which starts after.
+    *(TypeIndex *)(ContentBegin + M.first) = M.second;
+  }
+  auto RemapRef = makeArrayRef(RemapStorage);
+  return insertRecordBytes(RemapRef);
+}
+
+TypeIndex TypeTableBuilder::insertRecord(ContinuationRecordBuilder &Builder) {
+  TypeIndex TI;
+  auto Fragments = Builder.end(nextTypeIndex());
+  assert(!Fragments.empty());
+  for (auto C : Fragments)
+    TI = insertRecordBytes(C.RecordData);
+  return TI;
+}

llvm/trunk/lib/DebugInfo/CodeView/TypeTableCollection.cpp

 //===- TypeTableCollection.cpp -------------------------------- *- C++ --*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/CodeView/TypeTableCollection.h"
 
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/RecordName.h"
-#include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
 #include "llvm/Support/BinaryByteStream.h"
 #include "llvm/Support/BinaryStreamReader.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 
 TypeTableCollection::TypeTableCollection(ArrayRef<ArrayRef<uint8_t>> Records)
     : NameStorage(Allocator), Records(Records) {

llvm/trunk/lib/ObjectYAML/CodeViewYAMLTypes.cpp

 #include "llvm/ObjectYAML/CodeViewYAMLTypes.h"
 #include "llvm/ADT/APSInt.h"
 #include "llvm/ADT/ArrayRef.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/BinaryFormat/COFF.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/CodeView.h"
 #include "llvm/DebugInfo/CodeView/CodeViewError.h"
+#include "llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h"
 #include "llvm/DebugInfo/CodeView/TypeDeserializer.h"
 #include "llvm/DebugInfo/CodeView/TypeIndex.h"
 #include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/BinaryStreamReader.h"
 #include "llvm/Support/BinaryStreamWriter.h"
 #include "llvm/Support/Endian.h"

 
 struct LeafRecordBase {
   TypeLeafKind Kind;
 
   explicit LeafRecordBase(TypeLeafKind K) : Kind(K) {}
   virtual ~LeafRecordBase() = default;
 
   virtual void map(yaml::IO &io) = 0;
-  virtual CVType toCodeViewRecord(TypeTableBuilder &TTB) const = 0;
+  virtual CVType toCodeViewRecord(TypeTableBuilder &TS) const = 0;
   virtual Error fromCodeViewRecord(CVType Type) = 0;
 };
 
 template <typename T> struct LeafRecordImpl : public LeafRecordBase {
   explicit LeafRecordImpl(TypeLeafKind K)
       : LeafRecordBase(K), Record(static_cast<TypeRecordKind>(K)) {}
 
   void map(yaml::IO &io) override;
 
   Error fromCodeViewRecord(CVType Type) override {
     return TypeDeserializer::deserializeAs<T>(Type, Record);
   }
 
-  CVType toCodeViewRecord(TypeTableBuilder &TTB) const override {
-    TTB.writeKnownType(Record);
-    return CVType(Kind, TTB.records().back());
+  CVType toCodeViewRecord(TypeTableBuilder &TS) const override {
+    TS.writeLeafType(Record);
+    return CVType(Kind, TS.records().back());
   }
 
   mutable T Record;
 };
 
 template <> struct LeafRecordImpl<FieldListRecord> : public LeafRecordBase {
   explicit LeafRecordImpl(TypeLeafKind K) : LeafRecordBase(K) {}
 
   void map(yaml::IO &io) override;
-  CVType toCodeViewRecord(TypeTableBuilder &TTB) const override;
+  CVType toCodeViewRecord(TypeTableBuilder &TS) const override;
   Error fromCodeViewRecord(CVType Type) override;
 
   std::vector<MemberRecord> Members;
 };
 
 struct MemberRecordBase {
   TypeLeafKind Kind;
 
   explicit MemberRecordBase(TypeLeafKind K) : Kind(K) {}
   virtual ~MemberRecordBase() = default;
 
   virtual void map(yaml::IO &io) = 0;
-  virtual void writeTo(FieldListRecordBuilder &FLRB) = 0;
+  virtual void writeTo(ContinuationRecordBuilder &CRB) = 0;
 };
 
 template <typename T> struct MemberRecordImpl : public MemberRecordBase {
   explicit MemberRecordImpl(TypeLeafKind K)
       : MemberRecordBase(K), Record(static_cast<TypeRecordKind>(K)) {}
 
   void map(yaml::IO &io) override;
 
-  void writeTo(FieldListRecordBuilder &FLRB) override {
-    FLRB.writeMemberType(Record);
+  void writeTo(ContinuationRecordBuilder &CRB) override {
+    CRB.writeMemberType(Record);
   }
 
   mutable T Record;
 };
 
 } // end namespace detail
 } // end namespace CodeViewYAML
 } // end namespace llvm

 } // end anonymous namespace
 
 Error LeafRecordImpl<FieldListRecord>::fromCodeViewRecord(CVType Type) {
   MemberRecordConversionVisitor V(Members);
   return visitMemberRecordStream(Type.content(), V);
 }
 
 CVType
-LeafRecordImpl<FieldListRecord>::toCodeViewRecord(TypeTableBuilder &TTB) const {
-  FieldListRecordBuilder FLRB(TTB);
-  FLRB.begin();
+LeafRecordImpl<FieldListRecord>::toCodeViewRecord(TypeTableBuilder &TS) const {
+  ContinuationRecordBuilder CRB;
+  CRB.begin(ContinuationRecordKind::FieldList);
   for (const auto &Member : Members) {
-    Member.Member->writeTo(FLRB);
+    Member.Member->writeTo(CRB);
   }
-  FLRB.end(true);
-  return CVType(Kind, TTB.records().back());
+  TS.insertRecord(CRB);
+  return CVType(Kind, TS.records().back());
 }
 
 void MappingTraits<OneMethodRecord>::mapping(IO &io, OneMethodRecord &Record) {
   io.mapRequired("Type", Record.Type);
   io.mapRequired("Attrs", Record.Attrs.Attrs);
   io.mapRequired("VFTableOffset", Record.VFTableOffset);
   io.mapRequired("Name", Record.Name);
 }
#define MEMBER_RECORD_ALIAS(EnumName, EnumVal, AliasName, ClassName)
   switch (Type.kind()) {
 #include "llvm/DebugInfo/CodeView/CodeViewTypes.def"
   default:
       llvm_unreachable("Unknown leaf kind!");
   }
   return make_error<CodeViewError>(cv_error_code::corrupt_record);
 }
 
-CVType LeafRecord::toCodeViewRecord(BumpPtrAllocator &Alloc) const {
-  TypeTableBuilder TTB(Alloc);
-  return Leaf->toCodeViewRecord(TTB);
-}
-
-CVType LeafRecord::toCodeViewRecord(TypeTableBuilder &TTB) const {
-  return Leaf->toCodeViewRecord(TTB);
+CVType LeafRecord::toCodeViewRecord(TypeTableBuilder &Serializer) const {
+  return Leaf->toCodeViewRecord(Serializer);
 }
 
 namespace llvm {
 namespace yaml {
 
 template <> struct MappingTraits<LeafRecordBase> {
   static void mapping(IO &io, LeafRecordBase &Record) { Record.map(io); }
 };
for (const auto &T : Types) {
     auto CVT = Err(LeafRecord::fromCodeViewRecord(T));
     Result.push_back(CVT);
   }
   return Result;
 }
 
 ArrayRef<uint8_t> llvm::CodeViewYAML::toDebugT(ArrayRef<LeafRecord> Leafs,
                                                BumpPtrAllocator &Alloc) {
-  TypeTableBuilder TTB(Alloc, false);
+  TypeTableBuilder TS(Alloc, false);
   uint32_t Size = sizeof(uint32_t);
   for (const auto &Leaf : Leafs) {
-    CVType T = Leaf.toCodeViewRecord(TTB);
+    CVType T = Leaf.Leaf->toCodeViewRecord(TS);
     Size += T.length();
     assert(T.length() % 4 == 0 && "Improper type record alignment!");
   }
   uint8_t *ResultBuffer = Alloc.Allocate<uint8_t>(Size);
   MutableArrayRef<uint8_t> Output(ResultBuffer, Size);
   BinaryStreamWriter Writer(Output, support::little);
   ExitOnError Err("Error writing type record to .debug$T section");
   Err(Writer.writeInteger<uint32_t>(COFF::DEBUG_SECTION_MAGIC));
-  for (const auto &R : TTB.records()) {
+  for (const auto &R : TS.records()) {
     Err(Writer.writeBytes(R));
   }
   assert(Writer.bytesRemaining() == 0 && "Didn't write all type record bytes!");
   return Output;
 }

llvm/trunk/test/DebugInfo/COFF/big-type.ll

 ; RUN: llc -filetype=obj < %s | llvm-readobj - -codeview | FileCheck %s
 
 ; We need three continuation records for this.
 
 ; CHECK-LABEL: FieldList (0x1000)
 ; CHECK-NEXT: TypeLeafKind: LF_FIELDLIST (0x1203)
 ; CHECK-NEXT: Enumerator {
 ; CHECK-NEXT:   TypeLeafKind: LF_ENUMERATE (0x1502)
 ; CHECK-NEXT:   AccessSpecifier: Public (0x3)
 ; CHECK-NEXT:   EnumValue: 5436
 ; CHECK-NEXT:   Name: EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE5437
 ; CHECK-NEXT: }
+; CHECK:        EnumValue: 5695
+; CHECK-NEXT:   Name: EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE5696
+; CHECK-NEXT: }
 ; CHECK-NOT: ContinuationIndex
 
 ; CHECK-LABEL: FieldList (0x1001)
 ; CHECK-NEXT: TypeLeafKind: LF_FIELDLIST (0x1203)
 ; CHECK-NEXT: Enumerator {
 ; CHECK-NEXT:   TypeLeafKind: LF_ENUMERATE (0x1502)
 ; CHECK-NEXT:   AccessSpecifier: Public (0x3)
 ; CHECK-NEXT:   EnumValue: 4077

 ; CHECK-NEXT: Enumerator {
 ; CHECK-NEXT:   TypeLeafKind: LF_ENUMERATE (0x1502)
 ; CHECK-NEXT:   AccessSpecifier: Public (0x3)
 ; CHECK-NEXT:   EnumValue: 0
 ; CHECK-NEXT:   Name: EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE1
 ; CHECK-NEXT: }
 ; CHECK: ContinuationIndex: <field list> (0x1003)
 
+; CHECK-LABEL: Enum (0x1005) {
+; CHECK-NEXT:    TypeLeafKind: LF_ENUM (0x1507)
+; CHECK-NEXT:    NumEnumerators: 5696
+; CHECK-NEXT:    Properties [ (0x200)
+; CHECK-NEXT:      HasUniqueName (0x200)
+; CHECK-NEXT:    ]
+; CHECK-NEXT:    UnderlyingType: int (0x74)
+; CHECK-NEXT:    FieldListType: <field list> (0x1004)
+; CHECK-NEXT:    Name: BigThing
+; CHECK-NEXT:    LinkageName: .?AW4BigThing@@
+; CHECK-NEXT:  }
+
 ; ModuleID = 't.cpp'
 source_filename = "t.cpp"
 target datalayout = "e-m:w-i64:64-f80:128-n8:16:32:64-S128"
 target triple = "x86_64-pc-windows-msvc19.0.23918"
 
 @"\01?x@@3W4BigThing@@A" = global i32 0, align 4, !dbg !0
 
 !llvm.dbg.cu = !{!2}

llvm/trunk/tools/llvm-pdbutil/PdbYaml.cpp

 #include "PdbYaml.h"
 
 #include "llvm/ADT/StringExtras.h"
 #include "llvm/DebugInfo/CodeView/CVSymbolVisitor.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/DebugStringTableSubsection.h"
 #include "llvm/DebugInfo/CodeView/SymbolDeserializer.h"
 #include "llvm/DebugInfo/CodeView/SymbolVisitorCallbackPipeline.h"
-#include "llvm/DebugInfo/CodeView/TypeSerializer.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbackPipeline.h"
 #include "llvm/DebugInfo/PDB/Native/PDBFile.h"
 #include "llvm/DebugInfo/PDB/Native/RawTypes.h"
 #include "llvm/DebugInfo/PDB/Native/TpiHashing.h"
 #include "llvm/DebugInfo/PDB/PDBExtras.h"
 #include "llvm/DebugInfo/PDB/PDBTypes.h"
 #include "llvm/ObjectYAML/CodeViewYAMLDebugSections.h"
 #include "llvm/ObjectYAML/CodeViewYAMLTypes.h"

llvm/trunk/tools/llvm-pdbutil/llvm-pdbutil.cpp

for (const auto &MI : Dbi.ModInfos) {
     for (auto &SS : CodeViewSubsections) {
       ModiBuilder.addDebugSubsection(SS);
     }
   }
 
   auto &TpiBuilder = Builder.getTpiBuilder();
   const auto &Tpi = YamlObj.TpiStream.getValueOr(DefaultTpiStream);
   TpiBuilder.setVersionHeader(Tpi.Version);
+  TypeTableBuilder TS(Allocator);
   for (const auto &R : Tpi.Records) {
-    CVType Type = R.toCodeViewRecord(Allocator);
+    CVType Type = R.toCodeViewRecord(TS);
     TpiBuilder.addTypeRecord(Type.RecordData, None);
   }
 
   const auto &Ipi = YamlObj.IpiStream.getValueOr(DefaultIpiStream);
   auto &IpiBuilder = Builder.getIpiBuilder();
   IpiBuilder.setVersionHeader(Ipi.Version);
   for (const auto &R : Ipi.Records) {
-    CVType Type = R.toCodeViewRecord(Allocator);
+    CVType Type = R.toCodeViewRecord(TS);
     IpiBuilder.addTypeRecord(Type.RecordData, None);
   }
 
   Builder.getStringTableBuilder().setStrings(*Strings.strings());
 
   ExitOnErr(Builder.commit(opts::yaml2pdb::YamlPdbOutputFile));
 }
 

llvm/trunk/unittests/DebugInfo/CodeView/RandomAccessVisitorTest.cpp

 //===- llvm/unittest/DebugInfo/CodeView/RandomAccessVisitorTest.cpp -------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ADT/SmallBitVector.h"
 #include "llvm/DebugInfo/CodeView/CVTypeVisitor.h"
 #include "llvm/DebugInfo/CodeView/LazyRandomTypeCollection.h"
 #include "llvm/DebugInfo/CodeView/TypeRecord.h"
 #include "llvm/DebugInfo/CodeView/TypeRecordMapping.h"
-#include "llvm/DebugInfo/CodeView/TypeSerializer.h"
 #include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
 #include "llvm/DebugInfo/CodeView/TypeVisitorCallbacks.h"
 #include "llvm/DebugInfo/PDB/Native/RawTypes.h"
 #include "llvm/Support/Allocator.h"
 #include "llvm/Support/BinaryItemStream.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Testing/Support/Error.h"
 
for (int I = 0; I < 11; ++I) {
       AR.ElementType = TypeIndex::Int32();
       AR.IndexType = TypeIndex::UInt32();
       AR.Size = I;
       std::string Name;
       raw_string_ostream Stream(Name);
       Stream << "Array [" << I << "]";
       AR.Name = GlobalState->Strings.save(Stream.str());
       GlobalState->Records.push_back(AR);
-      GlobalState->Indices.push_back(Builder.writeKnownType(AR));
+      GlobalState->Indices.push_back(Builder.writeLeafType(AR));
 
       CVType Type(TypeLeafKind::LF_ARRAY, Builder.records().back());
       GlobalState->TypeVector.push_back(Type);
 
       GlobalState->AllOffsets.push_back(
           {GlobalState->Indices.back(), ulittle32_t(Offset)});
       Offset += Type.length();
     }
TEST_F(RandomAccessVisitorTest, CrossChunkName) {
   ClassRecord Class(TypeRecordKind::Class);
   Class.Name = "FooClass";
   Class.Options = ClassOptions::None;
   Class.MemberCount = 0;
   Class.Size = 4U;
   Class.DerivationList = TypeIndex::fromArrayIndex(0);
   Class.FieldList = TypeIndex::fromArrayIndex(0);
   Class.VTableShape = TypeIndex::fromArrayIndex(0);
-  TypeIndex IndexZero = Builder.writeKnownType(Class);
+  TypeIndex IndexZero = Builder.writeLeafType(Class);
 
   // TypeIndex 1 refers to type index 0.
   ModifierRecord Modifier(TypeRecordKind::Modifier);
   Modifier.ModifiedType = TypeIndex::fromArrayIndex(0);
   Modifier.Modifiers = ModifierOptions::Const;
-  TypeIndex IndexOne = Builder.writeKnownType(Modifier);
+  TypeIndex IndexOne = Builder.writeLeafType(Modifier);
 
   // set up a type stream that refers to the above two serialized records.
   std::vector<CVType> TypeArray;
   TypeArray.push_back(
       CVType(static_cast<TypeLeafKind>(Class.Kind), Builder.records()[0]));
   TypeArray.push_back(
       CVType(static_cast<TypeLeafKind>(Modifier.Kind), Builder.records()[1]));
   BinaryItemStream<CVType> ItemStream(llvm::support::little);

llvm/trunk/unittests/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp

 //===- llvm/unittest/DebugInfo/CodeView/TypeIndexDiscoveryTest.cpp --------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 
 #include "llvm/DebugInfo/CodeView/TypeIndexDiscovery.h"
 
-#include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
+#include "llvm/DebugInfo/CodeView/ContinuationRecordBuilder.h"
 #include "llvm/DebugInfo/CodeView/SymbolSerializer.h"
+#include "llvm/DebugInfo/CodeView/TypeTableBuilder.h"
 #include "llvm/Support/Allocator.h"
 
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
 
 using namespace llvm;
 using namespace llvm::codeview;
 
 class TypeIndexIteratorTest : public testing::Test {
 public:
   TypeIndexIteratorTest() {}
 
   void SetUp() override {
     Refs.clear();
     TTB = make_unique<TypeTableBuilder>(Storage);
-    FLRB = make_unique<FieldListRecordBuilder>(*TTB);
+    CRB = make_unique<ContinuationRecordBuilder>();
     Symbols.clear();
   }
 
   void TearDown() override {
-    FLRB.reset();
+    CRB.reset();
     TTB.reset();
   }
 
 protected:
   template <typename... Indices>
   bool checkTypeReferences(uint32_t RecordIndex, Indices &&... TIs) const {
     EXPECT_EQ(sizeof...(Indices), countRefs(RecordIndex));
 
     // Choose between type or symbol records. The checking code doesn't care
     // which we have.
     std::vector<ArrayRef<uint8_t>> CVRecords;
     if (Symbols.empty()) {
       CVRecords = TTB->records();
     } else {
       for (const CVSymbol &S : Symbols)
         CVRecords.push_back(S.data());
     }
 
     return checkTypeReferencesImpl(RecordIndex, CVRecords,
                                    std::forward<Indices>(TIs)...);
   }
 
   template <typename... T> void writeFieldList(T &&... MemberRecords) {
-    FLRB->begin();
+    CRB->begin(ContinuationRecordKind::FieldList);
     writeFieldListImpl(std::forward<T>(MemberRecords)...);
-    FLRB->end(true);
-    ASSERT_EQ(1u, TTB->records().size());
+    auto Records = CRB->end(TTB->nextTypeIndex());
+    ASSERT_EQ(1u, Records.size());
+    TTB->insertRecordBytes(Records.front().RecordData);
     discoverAllTypeIndices();
   }
 
   template <typename... T> void writeTypeRecords(T &&... Records) {
     writeTypeRecordsImpl(std::forward<T>(Records)...);
     ASSERT_EQ(sizeof...(T), TTB->records().size());
     discoverAllTypeIndices();
   }
private:
   }
 
   // Helper function to write out a field list record with the given list
   // of member records.
   void writeFieldListImpl() {}
 
   template <typename RecType, typename... Rest>
   void writeFieldListImpl(RecType &&Record, Rest &&... Records) {
-    FLRB->writeMemberType(Record);
+    CRB->writeMemberType(Record);
     writeFieldListImpl(std::forward<Rest>(Records)...);
   }
 
   // Helper function to write out a list of type records.
   void writeTypeRecordsImpl() {}
 
   template <typename RecType, typename... Rest>
   void writeTypeRecordsImpl(RecType &&Record, Rest &&... Records) {
-    TTB->writeKnownType(Record);
+    TTB->writeLeafType(Record);
     writeTypeRecordsImpl(std::forward<Rest>(Records)...);
   }
 
   // Helper function to write out a list of symbol records.
   void writeSymbolRecordsImpl() {}
 
   template <typename RecType, typename... Rest>
   void writeSymbolRecordsImpl(RecType &&Record, Rest &&... Records) {
     Symbols.push_back(SymbolSerializer::writeOneSymbol(Record, Storage,
                                                        CodeViewContainer::Pdb));
     writeSymbolRecordsImpl(std::forward<Rest>(Records)...);
   }
 
   std::vector<SmallVector<TiReference, 4>> Refs;
-  std::unique_ptr<FieldListRecordBuilder> FLRB;
+  std::unique_ptr<ContinuationRecordBuilder> CRB;
   std::vector<CVSymbol> Symbols;
   BumpPtrAllocator Storage;
 };
 
 namespace leafs {
 static FuncIdRecord FuncId(TypeIndex(1), TypeIndex(2), "FuncId");
 static MemberFuncIdRecord MemFuncId(TypeIndex(3), TypeIndex(4), "FuncId");
 static StringIdRecord StringId(TypeIndex(5), "TheString");