MinidumpYAML: Add support for the memory info list stream

[labath]

Summary

The implementation is fairly straight-forward and uses the same patterns
as the existing streams. The yaml form does not attempt to preserve the
data in the "gaps" that can be created by setting a larger-than-required
header or entry size in the stream header, because the existing consumer
(lldb) does not make use of the information in the gap in any way, and
attempting to preserve that would make the implementation more
complicated.

[JosephTremoulet]

Nit: Title says "thread" rather than "memory info"

[labath]

In D68645#1699705, @JosephTremoulet wrote:

Nit: Title says "thread" rather than "memory info"

Woops, sorry about that. Should be fixed now.

[clayborg]

LGTM

[jhenderson]

FYI, I'm going to be away for 2 and a half weeks from the end of work today, so won't have time to look at these if I don't get to them later today. I have no issues with other people reviewing them. You might want to add @grimar and @MaskRay to the reviews as they've been doing a lot of work in obj2yaml/yaml2obj recently.

[grimar]

include/llvm/ObjectYAML/MinidumpYAML.h

111

Maybe be more explicit here, i.e.

std::vector<minidump::MemoryInfo> &&Infos

?

lib/ObjectYAML/MinidumpEmitter.cpp

166

Probably just

minidump::MemoryInfoListHeader Header = {
    (support::ulittle32_t)sizeof(minidump::MemoryInfoListHeader),
    (support::ulittle32_t)sizeof(minidump::MemoryInfo),
    (support::ulittle64_t)InfoList.Infos.size()};

?

Or perhaps it could have a constructor.

[labath]

Address review comments

lib/ObjectYAML/MinidumpEmitter.cpp

166

The thought of a constructor has crossed my mind too. I've now added it.

[grimar]

LGTM

[MaskRay]

include/llvm/ObjectYAML/MinidumpYAML.h

111

Or let the constructor take iterator_range<MinidumpFile::MemoryInfoIterator> as the argument, and change the call site below.

116

Move default constructor above.

[labath]

include/llvm/ObjectYAML/MinidumpYAML.h

116

Done, I've also moved the constructor of SystemInfoStream for consistency.

include/llvm/BinaryFormat/Minidump.h

struct MemoryDescriptor {
   LocationDescriptor Memory;
 };
 static_assert(sizeof(MemoryDescriptor) == 16, "");
 
 struct MemoryInfoListHeader {
   support::ulittle32_t SizeOfHeader;
   support::ulittle32_t SizeOfEntry;
   support::ulittle64_t NumberOfEntries;
+
+  MemoryInfoListHeader() = default;
+  MemoryInfoListHeader(uint32_t SizeOfHeader, uint32_t SizeOfEntry,
+                       uint64_t NumberOfEntries)
+      : SizeOfHeader(SizeOfHeader), SizeOfEntry(SizeOfEntry),
+        NumberOfEntries(NumberOfEntries) {}
 };
 static_assert(sizeof(MemoryInfoListHeader) == 16, "");
 
 enum class MemoryProtection : uint32_t {
 #define HANDLE_MDMP_PROTECT(CODE, NAME, NATIVENAME) NAME = CODE,
 #include "llvm/BinaryFormat/MinidumpConstants.def"
   LLVM_MARK_AS_BITMASK_ENUM(/*LargestValue=*/0xffffffffu),
 };
 
 enum class MemoryState : uint32_t {
 #define HANDLE_MDMP_MEMSTATE(CODE, NAME, NATIVENAME) NAME = CODE,
 #include "llvm/BinaryFormat/MinidumpConstants.def"
+  LLVM_MARK_AS_BITMASK_ENUM(/*LargestValue=*/0xffffffffu),
 };
 
 enum class MemoryType : uint32_t {
 #define HANDLE_MDMP_MEMTYPE(CODE, NAME, NATIVENAME) NAME = CODE,
 #include "llvm/BinaryFormat/MinidumpConstants.def"
+  LLVM_MARK_AS_BITMASK_ENUM(/*LargestValue=*/0xffffffffu),
 };
 
 struct MemoryInfo {
   support::ulittle64_t BaseAddress;
   support::ulittle64_t AllocationBase;
   support::little_t<MemoryProtection> AllocationProtect;
   support::ulittle32_t Reserved0;
   support::ulittle64_t RegionSize;

include/llvm/ObjectYAML/MinidumpYAML.h

 /// corresponds to the Stream Type field in the minidump file. The "Kind" field
 /// specifies how are we going to treat it. For highly specialized streams (e.g.
 /// SystemInfo), there is a 1:1 mapping between Types and Kinds, but in general
 /// one stream Kind can be used to represent multiple stream Types (e.g. any
 /// unrecognised stream Type will be handled via RawContentStream). The mapping
 /// from Types to Kinds is fixed and given by the static getKind function.
 struct Stream {
   enum class StreamKind {
+    MemoryInfoList,
     MemoryList,
     ModuleList,
     RawContent,
     SystemInfo,
     TextContent,
     ThreadList,
   };
 
struct ParsedMemoryDescriptor {
   yaml::BinaryRef Content;
 };
 } // namespace detail
 
 using ModuleListStream = detail::ListStream<detail::ParsedModule>;
 using ThreadListStream = detail::ListStream<detail::ParsedThread>;
 using MemoryListStream = detail::ListStream<detail::ParsedMemoryDescriptor>;
 
+/// A structure containing the list of MemoryInfo entries comprising a
+/// MemoryInfoList stream.
+struct MemoryInfoListStream : public Stream {
+  std::vector<minidump::MemoryInfo> Infos;
+
+  explicit MemoryInfoListStream(std::vector<minidump::MemoryInfo> &&Infos)
+      : Stream(StreamKind::MemoryInfoList,
+               minidump::StreamType::MemoryInfoList),
+        Infos(std::move(Infos)) {}
+
+  MemoryInfoListStream()
+      : Stream(StreamKind::MemoryInfoList,
+               minidump::StreamType::MemoryInfoList) {}
+
+  static bool classof(const Stream *S) {
+    return S->Kind == StreamKind::MemoryInfoList;
+  }
+};
+
 /// A minidump stream represented as a sequence of hex bytes. This is used as a
 /// fallback when no other stream kind is suitable.
 struct RawContentStream : public Stream {
   yaml::BinaryRef Content;
   yaml::Hex32 Size;
 
   RawContentStream(minidump::StreamType Type, ArrayRef<uint8_t> Content = {})
       : Stream(StreamKind::RawContent, Type), Content(Content),
template <> struct MappingContextTraits<minidump::MemoryDescriptor, BinaryRef> {
   static void mapping(IO &IO, minidump::MemoryDescriptor &Memory,
                       BinaryRef &Content);
 };
 
 } // namespace yaml
 
 } // namespace llvm
 
+LLVM_YAML_DECLARE_BITSET_TRAITS(llvm::minidump::MemoryProtection)
+LLVM_YAML_DECLARE_BITSET_TRAITS(llvm::minidump::MemoryState)
+LLVM_YAML_DECLARE_BITSET_TRAITS(llvm::minidump::MemoryType)
+
 LLVM_YAML_DECLARE_ENUM_TRAITS(llvm::minidump::ProcessorArchitecture)
 LLVM_YAML_DECLARE_ENUM_TRAITS(llvm::minidump::OSPlatform)
 LLVM_YAML_DECLARE_ENUM_TRAITS(llvm::minidump::StreamType)
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::minidump::CPUInfo::ArmInfo)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::minidump::CPUInfo::OtherInfo)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::minidump::CPUInfo::X86Info)
+LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::minidump::MemoryInfo)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::minidump::VSFixedFileInfo)
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(
     llvm::MinidumpYAML::MemoryListStream::entry_type)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(
     llvm::MinidumpYAML::ModuleListStream::entry_type)
 LLVM_YAML_DECLARE_MAPPING_TRAITS(
     llvm::MinidumpYAML::ThreadListStream::entry_type)
 
 LLVM_YAML_IS_SEQUENCE_VECTOR(std::unique_ptr<llvm::MinidumpYAML::Stream>)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::MinidumpYAML::MemoryListStream::entry_type)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::MinidumpYAML::ModuleListStream::entry_type)
 LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::MinidumpYAML::ThreadListStream::entry_type)
+LLVM_YAML_IS_SEQUENCE_VECTOR(llvm::minidump::MemoryInfo)
 
 LLVM_YAML_DECLARE_MAPPING_TRAITS(llvm::MinidumpYAML::Object)
 
 #endif // LLVM_OBJECTYAML_MINIDUMPYAML_H

lib/ObjectYAML/MinidumpEmitter.cpp

 }
 
 static Directory layout(BlobAllocator &File, Stream &S) {
   Directory Result;
   Result.Type = S.Type;
   Result.Location.RVA = File.tell();
   Optional<size_t> DataEnd;
   switch (S.Kind) {
+  case Stream::StreamKind::MemoryInfoList: {
+    MemoryInfoListStream &InfoList = cast<MemoryInfoListStream>(S);
+    File.allocateNewObject<minidump::MemoryInfoListHeader>(
+        sizeof(minidump::MemoryInfoListHeader), sizeof(minidump::MemoryInfo),
+        InfoList.Infos.size());
+    File.allocateArray(makeArrayRef(InfoList.Infos));
+    break;
+  }
   case Stream::StreamKind::MemoryList:
     DataEnd = layout(File, cast<MemoryListStream>(S));
     break;
   case Stream::StreamKind::ModuleList:
     DataEnd = layout(File, cast<ModuleListStream>(S));
     break;
   case Stream::StreamKind::RawContent: {
     RawContentStream &Raw = cast<RawContentStream>(S);

lib/ObjectYAML/MinidumpYAML.cpp

static inline void mapOptionalHex(yaml::IO &IO, const char *Key,
                                   typename EndianType::value_type Default) {
   mapOptionalAs<typename HexType<EndianType>::type>(IO, Key, Val, Default);
 }
 
 Stream::~Stream() = default;
 
 Stream::StreamKind Stream::getKind(StreamType Type) {
   switch (Type) {
+  case StreamType::MemoryInfoList:
+    return StreamKind::MemoryInfoList;
   case StreamType::MemoryList:
     return StreamKind::MemoryList;
   case StreamType::ModuleList:
     return StreamKind::ModuleList;
   case StreamType::SystemInfo:
     return StreamKind::SystemInfo;
   case StreamType::LinuxCPUInfo:
   case StreamType::LinuxProcStatus:
   case StreamType::LinuxLSBRelease:
   case StreamType::LinuxCMDLine:
   case StreamType::LinuxMaps:
   case StreamType::LinuxProcStat:
   case StreamType::LinuxProcUptime:
     return StreamKind::TextContent;
   case StreamType::ThreadList:
     return StreamKind::ThreadList;
   default:
     return StreamKind::RawContent;
   }
 }
 
 std::unique_ptr<Stream> Stream::create(StreamType Type) {
   StreamKind Kind = getKind(Type);
   switch (Kind) {
+  case StreamKind::MemoryInfoList:
+    return std::make_unique<MemoryInfoListStream>();
   case StreamKind::MemoryList:
     return std::make_unique<MemoryListStream>();
   case StreamKind::ModuleList:
     return std::make_unique<ModuleListStream>();
   case StreamKind::RawContent:
     return std::make_unique<RawContentStream>(Type);
   case StreamKind::SystemInfo:
     return std::make_unique<SystemInfoStream>();
   case StreamKind::TextContent:
     return std::make_unique<TextContentStream>(Type);
   case StreamKind::ThreadList:
     return std::make_unique<ThreadListStream>();
   }
   llvm_unreachable("Unhandled stream kind!");
 }
 
+void yaml::ScalarBitSetTraits<MemoryProtection>::bitset(
+    IO &IO, MemoryProtection &Protect) {
+#define HANDLE_MDMP_PROTECT(CODE, NAME, NATIVENAME)                            \
+  IO.bitSetCase(Protect, #NATIVENAME, MemoryProtection::NAME);
+#include "llvm/BinaryFormat/MinidumpConstants.def"
+}
+
+void yaml::ScalarBitSetTraits<MemoryState>::bitset(IO &IO, MemoryState &State) {
+#define HANDLE_MDMP_MEMSTATE(CODE, NAME, NATIVENAME)                           \
+  IO.bitSetCase(State, #NATIVENAME, MemoryState::NAME);
+#include "llvm/BinaryFormat/MinidumpConstants.def"
+}
+
+void yaml::ScalarBitSetTraits<MemoryType>::bitset(IO &IO, MemoryType &Type) {
+#define HANDLE_MDMP_MEMTYPE(CODE, NAME, NATIVENAME)                            \
+  IO.bitSetCase(Type, #NATIVENAME, MemoryType::NAME);
+#include "llvm/BinaryFormat/MinidumpConstants.def"
+}
+
 void yaml::ScalarEnumerationTraits<ProcessorArchitecture>::enumeration(
     IO &IO, ProcessorArchitecture &Arch) {
 #define HANDLE_MDMP_ARCH(CODE, NAME)                                           \
   IO.enumCase(Arch, #NAME, ProcessorArchitecture::NAME);
 #include "llvm/BinaryFormat/MinidumpConstants.def"
   IO.enumFallback<Hex16>(Arch);
 }
 
void yaml::MappingTraits<CPUInfo::X86Info>::mapping(IO &IO,
   FixedSizeString<sizeof(Info.VendorID)> VendorID(Info.VendorID);
   IO.mapRequired("Vendor ID", VendorID);
 
   mapRequiredHex(IO, "Version Info", Info.VersionInfo);
   mapRequiredHex(IO, "Feature Info", Info.FeatureInfo);
   mapOptionalHex(IO, "AMD Extended Features", Info.AMDExtendedFeatures, 0);
 }
 
+void yaml::MappingTraits<MemoryInfo>::mapping(IO &IO, MemoryInfo &Info) {
+  mapRequiredHex(IO, "Base Address", Info.BaseAddress);
+  mapOptionalHex(IO, "Allocation Base", Info.AllocationBase, Info.BaseAddress);
+  mapRequiredAs<MemoryProtection>(IO, "Allocation Protect",
+                                  Info.AllocationProtect);
+  mapOptionalHex(IO, "Reserved0", Info.Reserved0, 0);
+  mapRequiredHex(IO, "Region Size", Info.RegionSize);
+  mapRequiredAs<MemoryState>(IO, "State", Info.State);
+  mapOptionalAs<MemoryProtection>(IO, "Protect", Info.Protect,
+                                  Info.AllocationProtect);
+  mapRequiredAs<MemoryType>(IO, "Type", Info.Type);
+  mapOptionalHex(IO, "Reserved1", Info.Reserved1, 0);
+}
+
 void yaml::MappingTraits<VSFixedFileInfo>::mapping(IO &IO,
                                                    VSFixedFileInfo &Info) {
   mapOptionalHex(IO, "Signature", Info.Signature, 0);
   mapOptionalHex(IO, "Struct Version", Info.StructVersion, 0);
   mapOptionalHex(IO, "File Version High", Info.FileVersionHigh, 0);
   mapOptionalHex(IO, "File Version Low", Info.FileVersionLow, 0);
   mapOptionalHex(IO, "Product Version High", Info.ProductVersionHigh, 0);
   mapOptionalHex(IO, "Product Version Low", Info.ProductVersionLow, 0);

 }
 
 void yaml::MappingTraits<MemoryListStream::entry_type>::mapping(
     IO &IO, MemoryListStream::entry_type &Range) {
   MappingContextTraits<MemoryDescriptor, yaml::BinaryRef>::mapping(
       IO, Range.Entry, Range.Content);
 }
 
+static void streamMapping(yaml::IO &IO, MemoryInfoListStream &Stream) {
+  IO.mapRequired("Memory Ranges", Stream.Infos);
+}
+
 static void streamMapping(yaml::IO &IO, MemoryListStream &Stream) {
   IO.mapRequired("Memory Ranges", Stream.Entries);
 }
 
 static void streamMapping(yaml::IO &IO, ModuleListStream &Stream) {
   IO.mapRequired("Modules", Stream.Entries);
 }
 
void yaml::MappingTraits<std::unique_ptr<Stream>>::mapping(
   StreamType Type;
   if (IO.outputting())
     Type = S->Type;
   IO.mapRequired("Type", Type);
 
   if (!IO.outputting())
     S = MinidumpYAML::Stream::create(Type);
   switch (S->Kind) {
+  case MinidumpYAML::Stream::StreamKind::MemoryInfoList:
+    streamMapping(IO, llvm::cast<MemoryInfoListStream>(*S));
+    break;
   case MinidumpYAML::Stream::StreamKind::MemoryList:
     streamMapping(IO, llvm::cast<MemoryListStream>(*S));
     break;
   case MinidumpYAML::Stream::StreamKind::ModuleList:
     streamMapping(IO, llvm::cast<ModuleListStream>(*S));
     break;
   case MinidumpYAML::Stream::StreamKind::RawContent:
     streamMapping(IO, llvm::cast<RawContentStream>(*S));
void yaml::MappingTraits<std::unique_ptr<Stream>>::mapping(
   }
 }
 
 StringRef yaml::MappingTraits<std::unique_ptr<Stream>>::validate(
     yaml::IO &IO, std::unique_ptr<MinidumpYAML::Stream> &S) {
   switch (S->Kind) {
   case MinidumpYAML::Stream::StreamKind::RawContent:
     return streamValidate(cast<RawContentStream>(*S));
+  case MinidumpYAML::Stream::StreamKind::MemoryInfoList:
   case MinidumpYAML::Stream::StreamKind::MemoryList:
   case MinidumpYAML::Stream::StreamKind::ModuleList:
   case MinidumpYAML::Stream::StreamKind::SystemInfo:
   case MinidumpYAML::Stream::StreamKind::TextContent:
   case MinidumpYAML::Stream::StreamKind::ThreadList:
     return "";
   }
   llvm_unreachable("Fully covered switch above!");
 }
 
 void yaml::MappingTraits<Object>::mapping(IO &IO, Object &O) {
   IO.mapTag("!minidump", true);
   mapOptionalHex(IO, "Signature", O.Header.Signature, Header::MagicSignature);
   mapOptionalHex(IO, "Version", O.Header.Version, Header::MagicVersion);
   mapOptionalHex(IO, "Flags", O.Header.Flags, 0);
   IO.mapRequired("Streams", O.Streams);
 }
 
 Expected<std::unique_ptr<Stream>>
 Stream::create(const Directory &StreamDesc, const object::MinidumpFile &File) {
   StreamKind Kind = getKind(StreamDesc.Type);
   switch (Kind) {
+  case StreamKind::MemoryInfoList: {
+    auto ExpectedList = File.getMemoryInfoList();
+    if (!ExpectedList)
+      return ExpectedList.takeError();
+    return std::make_unique<MemoryInfoListStream>(
+        std::vector<MemoryInfo>(ExpectedList->begin(), ExpectedList->end()));
+  }
   case StreamKind::MemoryList: {
     auto ExpectedList = File.getMemoryList();
     if (!ExpectedList)
       return ExpectedList.takeError();
     std::vector<MemoryListStream::entry_type> Ranges;
     for (const MemoryDescriptor &MD : *ExpectedList) {
       auto ExpectedContent = File.getRawData(MD.Memory);
       if (!ExpectedContent)

test/tools/obj2yaml/basic-minidump.yaml

Threads:
         Context:           '7C7D7E7F80818283'
         Stack:
           Start of Memory Range: 0x6C6D6E6F70717273
           Content:               '7475767778797A7B'
   - Type:            MemoryList
     Memory Ranges:   
       - Start of Memory Range: 0x7C7D7E7F80818283
         Content:               '8485868788'
+  - Type:            MemoryInfoList
+    Memory Ranges:
+      - Base Address:    0x0000000000000000
+        Allocation Protect: [  ]
+        Region Size:     0x0000000000010000
+        State:           [ MEM_FREE ]
+        Protect:         [ PAGE_NO_ACCESS ]
+        Type:            [  ]
+      - Base Address:    0x0000000000010000
+        Allocation Protect: [ PAGE_READ_WRITE ]
+        Region Size:     0x0000000000010000
+        State:           [ MEM_COMMIT ]
+        Type:            [ MEM_MAPPED ]
+      - Base Address:    0x0000000000020000
+        Allocation Base: 0x0000000000000000
+        Allocation Protect: [ PAGE_READ_WRITE, PAGE_WRITECOMBINE ]
+        Reserved0:       0xDEADBEEF
+        Region Size:     0x0000000000010000
+        State:           [ MEM_COMMIT, MEM_FREE ]
+        Type:            [ MEM_PRIVATE, MEM_MAPPED ]
+        Reserved1:       0xBAADF00D
 ...
 
 # CHECK:      --- !minidump
 # CHECK-NEXT: Streams:
 # CHECK-NEXT:   - Type:            SystemInfo
 # CHECK-NEXT:     Processor Arch:  ARM64
 # CHECK-NEXT:     Platform ID:     Linux
 # CHECK-NEXT:     CSD Version:     Linux 3.13.0-91-generic

 # CHECK-NEXT:         Context:           7C7D7E7F80818283
 # CHECK-NEXT:         Stack:
 # CHECK-NEXT:           Start of Memory Range: 0x6C6D6E6F70717273
 # CHECK-NEXT:           Content:               7475767778797A7B
 # CHECK-NEXT:   - Type:            MemoryList
 # CHECK-NEXT:     Memory Ranges:   
 # CHECK-NEXT:       - Start of Memory Range: 0x7C7D7E7F80818283
 # CHECK-NEXT:         Content:               '8485868788'
+# CHECK-NEXT:   - Type:            MemoryInfoList
+# CHECK-NEXT:     Memory Ranges:
+# CHECK-NEXT:       - Base Address:       0x0000000000000000
+# CHECK-NEXT:         Allocation Protect: [  ]
+# CHECK-NEXT:         Region Size:        0x0000000000010000
+# CHECK-NEXT:         State:              [ MEM_FREE ]
+# CHECK-NEXT:         Protect:            [ PAGE_NO_ACCESS ]
+# CHECK-NEXT:         Type:               [  ]
+# CHECK-NEXT:       - Base Address:       0x0000000000010000
+# CHECK-NEXT:         Allocation Protect: [ PAGE_READ_WRITE ]
+# CHECK-NEXT:         Region Size:        0x0000000000010000
+# CHECK-NEXT:         State:              [ MEM_COMMIT ]
+# CHECK-NEXT:         Type:               [ MEM_MAPPED ]
+# CHECK-NEXT:       - Base Address:       0x0000000000020000
+# CHECK-NEXT:         Allocation Base:    0x0000000000000000
+# CHECK-NEXT:         Allocation Protect: [ PAGE_READ_WRITE, PAGE_WRITECOMBINE ]
+# CHECK-NEXT:         Reserved0:          0xDEADBEEF
+# CHECK-NEXT:         Region Size:        0x0000000000010000
+# CHECK-NEXT:         State:              [ MEM_COMMIT, MEM_FREE ]
+# CHECK-NEXT:         Type:               [ MEM_PRIVATE, MEM_MAPPED ]
+# CHECK-NEXT:         Reserved1:          0xBAADF00D
 # CHECK-NEXT: ...