Introduce core2yaml tool

[labath]

Summary

This patch introduces the core2yaml tool, whose purpose is to dump core
files in a human readable (and, ideally, editable) format. It's very
similar to llvm's obj2yaml, except that it works with core files.

Currently, I only add basic support for dumping minidump files, but in
the future, the goal is to be able to dump ELF core files as well (hence
the generic name). Most of the code is implemented as a library (to
enable it being used from unit tests), with the actual executable only
being a very thin wrapper around that.

This patch only sets up the basic plumbing, and implements enough
functionality to be able to dump one of the simple minidump files we
have lying around. More functionality (including the tool for doing the
opposite conversion) will come in subsequent patches.

[clayborg]

Do we want this in LLVM instead of lldb?

[labath]

In D58976#1418596, @clayborg wrote:

Do we want this in LLVM instead of lldb?

The thought has crossed my mind, but for that I would have to also move the minidump parser into llvm. It's already pretty standalone, so it shouldn't be a problem technically, but it's not clear to me whether there is any use case for it in llvm. This is a core file format, and I don't know of any other llvm tool/project working with core files.

[clayborg]

include/lldb/Formats/MinidumpParser.h

105

I worry about this going stale when the owner of the data lets it go and we crash now that we don't have strong ownership. If this is common in LLVM, then we need to document this in the header file.

[clayborg]

Strong ownership is needed for this class IMHO because it hands out pointers to native data

[zturner]

In D58976#1418621, @clayborg wrote:

Strong ownership is needed for this class IMHO because it hands out pointers to native data

I disagree here, see my previous comment. Binaries grow large very quickly, and if we always copy data around when we want to hand out some internal pointer, memory usage would explode. Luckily, the scenario that attempts to prevent is very rare. Specifically, it only addresses the scenario where you open a file, parse a bunch of data, close the file, then still expect to be able to do something with the file's internal data structures. I haven't ever seen this be a problem in practice, as the "natural" order is to open a file, process it, then close the file. And in that case this is fine.

I don't mind having it be documented in the class header, but given that the pattern is fairly pervasive in LLVM (e.g. all of lib/Object works this way, among other things), I'm also fine with just letting it be implicitly understood.

include/lldb/Formats/MinidumpParser.h

105

This is a fairly pervasive, as well as being an important optimization. We don't want to be copying data around from binary files because the amount of data that ends up being copied could quickly spiral out of control since binaries get quite large. The semantics are that the data is valid as long as the backing file remains opened. Anyone using the class needs to be aware of this, and if they want a lifetime that is not tied to the life of the backing file (which is a fairly uncommon scenario), they need to explicitly copy any data they need to outlive the file.

[labath]

instead of a fresh tool, minidump support will be added to obj2yaml.

include/lldb/Formats/MinidumpParser.h

struct Range {
   llvm::ArrayRef<uint8_t> range_ref;
 
   Range(lldb::addr_t start, llvm::ArrayRef<uint8_t> range_ref)
       : start(start), range_ref(range_ref) {}
 };
 
 class MinidumpParser {
 public:
-  static llvm::Expected<MinidumpParser>
-  Create(const lldb::DataBufferSP &data_buf_sp);
+  static llvm::Expected<MinidumpParser> Create(llvm::ArrayRef<uint8_t> data);
 
-  llvm::ArrayRef<uint8_t> GetData();
+  llvm::ArrayRef<uint8_t> GetData() { return m_data; }
 
   llvm::ArrayRef<uint8_t> GetStream(MinidumpStreamType stream_type);
 
   llvm::Optional<std::string> GetMinidumpString(uint32_t rva);
 
   UUID GetModuleUUID(const MinidumpModule* module);
 
   llvm::ArrayRef<MinidumpThread> GetThreads();
public:
 
   static llvm::StringRef GetStreamTypeAsString(uint32_t stream_type);
 
   const llvm::DenseMap<uint32_t, MinidumpLocationDescriptor> &
   GetDirectoryMap() const {
     return m_directory_map;
   }
 
+  const MinidumpHeader &GetHeader() { return *m_header; }
+
 private:
   MinidumpParser(
-      lldb::DataBufferSP data_sp,
+      llvm::ArrayRef<uint8_t> data, const MinidumpHeader &header,
       llvm::DenseMap<uint32_t, MinidumpLocationDescriptor> directory_map);
 
   MemoryRegionInfo FindMemoryRegion(lldb::addr_t load_addr) const;
 
 private:
-  lldb::DataBufferSP m_data_sp;
+  llvm::ArrayRef<uint8_t> m_data;
+  const MinidumpHeader *m_header;
   llvm::DenseMap<uint32_t, MinidumpLocationDescriptor> m_directory_map;
   ArchSpec m_arch;
   MemoryRegionInfos m_regions;
   bool m_parsed_regions = false;
 };
 
 } // end namespace minidump
 } // end namespace lldb_private
 
 #endif // LLDB_BINARYFORMAT_MINIDUMPPARSER_H

include/lldb/Formats/MinidumpTypes.h

struct MinidumpThread {
   ParseThreadList(llvm::ArrayRef<uint8_t> &data);
 };
 static_assert(sizeof(MinidumpThread) == 48,
               "sizeof MinidumpThread is not correct!");
 
 // Reference:
 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms680396(v=vs.85).aspx
 union MinidumpCPUInfo {
-  struct {
-    llvm::support::ulittle32_t vendor_id[3];        /* cpuid 0: ebx, edx, ecx */
+  struct X86CPUInfo {
+    char vendor_id[12];                             /* cpuid 0: ebx, edx, ecx */
     llvm::support::ulittle32_t version_information; /* cpuid 1: eax */
     llvm::support::ulittle32_t feature_information; /* cpuid 1: edx */
     llvm::support::ulittle32_t
         amd_extended_cpu_features; /* cpuid 0x80000001, ebx */
   } x86_cpu_info;
-  struct {
+  struct ArmCPUInfo{
     llvm::support::ulittle32_t cpuid;
     llvm::support::ulittle32_t elf_hwcaps; /* linux specific, 0 otherwise */
   } arm_cpu_info;
-  struct {
-    llvm::support::ulittle64_t processor_features[2];
+  struct OtherCPUInfo {
+    uint8_t processor_features[16];
   } other_cpu_info;
 };
 static_assert(sizeof(MinidumpCPUInfo) == 24,
               "sizeof MinidumpCPUInfo is not correct!");
 
 // Reference:
 // https://msdn.microsoft.com/en-us/library/windows/desktop/ms680396(v=vs.85).aspx
 struct MinidumpSystemInfo {

include/lldb/Formats/MinidumpYAML.h

+//===-- MinidumpYAML.h ------------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLDB_BINARYFORMAT_MINIDUMPYAML_H
+#define LLDB_BINARYFORMAT_MINIDUMPYAML_H
+
+#include "lldb/Formats/MinidumpTypes.h"
+#include "llvm/ObjectYAML/YAML.h"
+#include "llvm/Support/YAMLTraits.h"
+
+namespace lldb_private {
+namespace minidump {
+
+class Stream;
+class MinidumpParser;
+
+/// The top level structure representing a minidump object, consisting of a
+/// minidump header, and zero or more streams. To construct an Object from a
+/// minidump file, use the static create function. To serialize to/from yaml,
+/// use the appropriate streaming operator on a yaml stream.
+/// TODO: Add minidump serialization support.
+struct Object {
+  Object(const MinidumpHeader &Header,
+         std::vector<std::unique_ptr<Stream>> Streams)
+      : Header(Header), Streams(std::move(Streams)) {}
+
+  Object(const Object &) = delete;
+  Object &operator=(const Object &) = delete;
+  Object(Object &&) = default;
+  Object &operator=(Object &&) = default;
+
+  /// The minidump header.
+  MinidumpHeader Header;
+
+  /// The list of streams in this minidump object.
+  std::vector<std::unique_ptr<Stream>> Streams;
+
+  /// Construct a minidump object from the minidump file in the given parser.
+  /// The returned object contains references to the data in the minidump file,
+  /// so the parser instance should outlive the minidump object.
+  static llvm::Expected<Object> create(MinidumpParser &Parser);
+};
+
+/// The base class for all minidump streams. The "Type" of the stream
+/// corresponds to the Stream Type field in the minidump file. The "Kind" field
+/// specifies how are we going to treat it. For highly specialized streams (e.g.
+/// SystemInfo), there is a 1:1 mapping between Types and Kinds, but in general
+/// one stream Kind can be used to represent multiple stream Types (e.g. any
+/// unrecognised stream Type will be handled via HexStream). The mapping from
+/// Types to Kinds is fixed and given by the static getKind function.
+struct Stream {
+  enum StreamKind {
+    EnvironmentKind,
+    HexKind,
+    SystemInfoKind,
+    TextKind,
+  };
+
+  Stream(StreamKind Kind, MinidumpStreamType Type) : Kind(Kind), Type(Type) {}
+  virtual ~Stream(); // anchor
+
+  const StreamKind Kind;
+  const MinidumpStreamType Type;
+
+  /// Get the stream Kind used for representing streams of a given Type.
+  static StreamKind getKind(MinidumpStreamType Type);
+
+  /// Construct a stream of the given Type, using the data in the minidump file.
+  static llvm::Expected<std::unique_ptr<Stream>> create(MinidumpStreamType Type,
+                                                        MinidumpParser &Parser);
+
+  /// Create an empty stream of the given Type.
+  static std::unique_ptr<Stream> create(MinidumpStreamType Type);
+};
+
+/// A stream containing the execution environment of the process.
+struct EnvironmentStream: public Stream {
+  llvm::SmallVector<llvm::StringRef, 0> Variables;
+
+  explicit EnvironmentStream(
+      llvm::SmallVector<llvm::StringRef, 0> Variables = {})
+      : Stream(EnvironmentKind, MinidumpStreamType::LinuxEnviron),
+        Variables(std::move(Variables)) {}
+
+  static bool classof(const Stream *S) { return S->Kind == EnvironmentKind; }
+};
+
+/// A minidump stream represented as a sequence of hex bytes. This is used as a
+/// fallback when no other stream kind is suitable.
+struct HexStream : public Stream {
+  llvm::yaml::BinaryRef Content;
+  llvm::yaml::Hex32 Size;
+
+  HexStream(MinidumpStreamType Type, llvm::ArrayRef<uint8_t> Content = {})
+      : Stream(HexKind, Type), Content(Content),
+        Size(Content.size()) {}
+
+  static bool classof(const Stream *S) { return S->Kind == HexKind; }
+};
+
+/// SystemInfo minidump stream.
+struct SystemInfoStream: public Stream {
+  MinidumpSystemInfo Info;
+
+  explicit SystemInfoStream(const MinidumpSystemInfo &Info)
+      : Stream(SystemInfoKind, MinidumpStreamType::SystemInfo), Info(Info) {}
+
+  SystemInfoStream() : Stream(SystemInfoKind, MinidumpStreamType::SystemInfo) {
+    memset(&Info, 0, sizeof(Info));
+  }
+
+  static bool classof(const Stream *S) { return S->Kind == SystemInfoKind; }
+};
+
+/// A StringRef, which is printed using YAML block notation.
+LLVM_YAML_STRONG_TYPEDEF(llvm::StringRef, BlockStringRef)
+
+/// A minidump stream containing textual data (typically, the contents of a
+/// /proc/<pid> file on linux).
+struct TextStream : public Stream {
+  BlockStringRef Text;
+
+  TextStream(MinidumpStreamType Type, llvm::StringRef Text = {})
+      : Stream(TextKind, Type), Text(Text) {}
+
+  static bool classof(const Stream *S) { return S->Kind == TextKind; }
+};
+
+} // namespace minidump
+} // namespace lldb_private
+
+LLVM_YAML_IS_SEQUENCE_VECTOR(std::unique_ptr<lldb_private::minidump::Stream>)
+
+namespace llvm {
+namespace yaml {
+
+template <> struct MappingTraits<lldb_private::minidump::Object> {
+  static void mapping(yaml::IO &IO, lldb_private::minidump::Object &O);
+};
+
+template <>
+struct MappingTraits<std::unique_ptr<lldb_private::minidump::Stream>> {
+  static void mapping(yaml::IO &IO,
+                      std::unique_ptr<lldb_private::minidump::Stream> &S);
+};
+
+template <> struct BlockScalarTraits<lldb_private::minidump::BlockStringRef> {
+  static void output(const lldb_private::minidump::BlockStringRef &Text, void *,
+                     raw_ostream &OS) {
+    OS << Text;
+  }
+
+  static StringRef input(StringRef Scalar, void *,
+                         lldb_private::minidump::BlockStringRef &Text) {
+    Text = Scalar;
+    return "";
+  }
+};
+
+template <>
+struct ScalarEnumerationTraits<lldb_private::minidump::MinidumpStreamType> {
+  static void enumeration(IO &IO,
+                          lldb_private::minidump::MinidumpStreamType &Type);
+};
+
+template <>
+struct ScalarEnumerationTraits<
+    lldb_private::minidump::MinidumpCPUArchitecture> {
+  static void
+  enumeration(IO &IO, lldb_private::minidump::MinidumpCPUArchitecture &Type);
+};
+
+template <>
+struct ScalarEnumerationTraits<lldb_private::minidump::MinidumpOSPlatform> {
+  static void enumeration(IO &IO,
+                          lldb_private::minidump::MinidumpOSPlatform &Plat);
+};
+
+template <>
+struct MappingTraits<lldb_private::minidump::MinidumpCPUInfo::ArmCPUInfo> {
+  static void
+  mapping(IO &IO, lldb_private::minidump::MinidumpCPUInfo::ArmCPUInfo &Info);
+};
+
+template <>
+struct MappingTraits<lldb_private::minidump::MinidumpCPUInfo::OtherCPUInfo> {
+  static void
+  mapping(IO &IO, lldb_private::minidump::MinidumpCPUInfo::OtherCPUInfo &Info);
+};
+
+template <>
+struct MappingTraits<lldb_private::minidump::MinidumpCPUInfo::X86CPUInfo> {
+  static void
+  mapping(IO &IO, lldb_private::minidump::MinidumpCPUInfo::X86CPUInfo &Info);
+};
+
+} // namespace yaml
+} // namespace llvm
+#endif // LLDB_BINARYFORMAT_MINIDUMPYAML_H

lit/tools/core2yaml/basic-minidump.test

+RUN: core2yaml %S/Inputs/basic-minidump.dmp | FileCheck %s
+
+CHECK:      --- !minidump
+CHECK-NEXT: Streams:         
+CHECK-NEXT:   - Type:            SystemInfo
+CHECK-NEXT:     Processor Arch:  ARM64
+CHECK-NEXT:     Platform ID:     Linux
+CHECK-NEXT:     CSD Version RVA: 0x00000288
+CHECK-NEXT:     CPU:             
+CHECK-NEXT:       CPUID:           0x00000000
+CHECK-NEXT:   - Type:            LinuxEnviron
+CHECK-NEXT:     Variables:       
+CHECK-NEXT:       - '/proc/<pid>/environ output'
+CHECK-NEXT:   - Type:            LinuxMaps
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       400d9000-400db000 r-xp 00000000 b3:04 227        /system/bin/app_process
+CHECK-NEXT:       400db000-400dc000 r--p 00001000 b3:04 227        /system/bin/app_process
+CHECK-NEXT:       400dc000-400dd000 rw-p 00000000 00:00 0 
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxProcStatus
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       /proc/<pid>/status output
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxProcStat
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       /proc/<pid>/stat output
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxCMDLine
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       /proc/<pid>/cmdline output
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxProcFD
+CHECK-NEXT:     Content:         2F70726F632F3C7069643E2F6664206F757470757400
+CHECK-NEXT:   - Type:            LinuxAuxv
+CHECK-NEXT:     Content:         2F70726F632F3C7069643E2F61757876206F757470757400
+CHECK-NEXT:   - Type:            MiscInfo
+CHECK-NEXT:     Content:         00000000010000007B000000000000000000000000000000
+CHECK-NEXT:   - Type:            LinuxCPUInfo
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       cpu info output
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxLSBRelease
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       /etc/lsb-release output
+CHECK-EMPTY:
+CHECK-NEXT:   - Type:            LinuxProcUptime
+CHECK-NEXT:     Text:             |
+CHECK-NEXT:       uptime output
+CHECK-EMPTY:
+CHECK-NEXT: ...

source/Formats/CMakeLists.txt

 add_lldb_library(lldbFormats
   LinuxProcMaps.cpp
   MinidumpParser.cpp
   MinidumpTypes.cpp
+  MinidumpYAML.cpp
 
   LINK_LIBS
     lldbUtility
   LINK_COMPONENTS
+    ObjectYAML
     Support
   )

source/Formats/MinidumpParser.cpp

 using namespace minidump;
 
 static llvm::Error stringError(llvm::StringRef Err) {
   return llvm::make_error<llvm::StringError>(Err,
                                              llvm::inconvertibleErrorCode());
 }
 
 llvm::Expected<MinidumpParser>
-MinidumpParser::Create(const lldb::DataBufferSP &data_sp) {
-  if (data_sp->GetByteSize() < sizeof(MinidumpHeader))
+MinidumpParser::Create(llvm::ArrayRef<uint8_t> data) {
+  const size_t file_size = data.size();
+
+  if (file_size < sizeof(MinidumpHeader))
     return stringError("Buffer too small.");
 
-  llvm::ArrayRef<uint8_t> header_data(data_sp->GetBytes(),
-                                      sizeof(MinidumpHeader));
+  llvm::ArrayRef<uint8_t> header_data = data.take_front(sizeof(MinidumpHeader));
   const MinidumpHeader *header = MinidumpHeader::Parse(header_data);
   if (!header)
     return stringError("invalid minidump: can't parse the header");
 
   // A minidump without at least one stream is clearly ill-formed
   if (header->streams_count == 0)
     return stringError("invalid minidump: no streams present");
 
   struct FileRange {
     uint32_t offset = 0;
     uint32_t size = 0;
 
     FileRange(uint32_t offset, uint32_t size) : offset(offset), size(size) {}
     uint32_t end() const { return offset + size; }
   };
 
-  const uint32_t file_size = data_sp->GetByteSize();
-
   // Build a global minidump file map, checking for:
   // - overlapping streams/data structures
   // - truncation (streams pointing past the end of file)
   std::vector<FileRange> minidump_map;
 
   minidump_map.emplace_back(0, sizeof(MinidumpHeader));
 
   // Add the directory entries to the file map
   FileRange directory_range(header->stream_directory_rva,
                             header->streams_count * sizeof(MinidumpDirectory));
   if (directory_range.end() > file_size)
     return stringError("invalid minidump: truncated streams directory");
   minidump_map.push_back(directory_range);
 
   llvm::DenseMap<uint32_t, MinidumpLocationDescriptor> directory_map;
 
   // Parse stream directory entries
-  llvm::ArrayRef<uint8_t> directory_data(
-      data_sp->GetBytes() + directory_range.offset, directory_range.size);
+  llvm::ArrayRef<uint8_t> directory_data =
+      data.slice(directory_range.offset, directory_range.size);
   for (uint32_t i = 0; i < header->streams_count; ++i) {
     const MinidumpDirectory *directory_entry = nullptr;
     Status error = consumeObject(directory_data, directory_entry);
     if (error.Fail())
       return error.ToError();
     if (directory_entry->stream_type == 0) {
       // Ignore dummy streams (technically ill-formed, but a number of
       // existing minidumps seem to contain such streams)
if (prev_range.end() > minidump_map[i].offset)
       return stringError("invalid minidump: overlapping streams");
   }
 
   // Check for streams past the end of file
   const auto &last_range = minidump_map.back();
   if (last_range.end() > file_size)
     return stringError("invalid minidump: truncated stream");
 
-  return MinidumpParser(std::move(data_sp), std::move(directory_map));
+  return MinidumpParser(data, *header, std::move(directory_map));
 }
 
 MinidumpParser::MinidumpParser(
-    lldb::DataBufferSP data_sp,
+    llvm::ArrayRef<uint8_t> data, const MinidumpHeader &header,
     llvm::DenseMap<uint32_t, MinidumpLocationDescriptor> directory_map)
-    : m_data_sp(std::move(data_sp)), m_directory_map(std::move(directory_map)) {
-}
-
-llvm::ArrayRef<uint8_t> MinidumpParser::GetData() {
-  return llvm::ArrayRef<uint8_t>(m_data_sp->GetBytes(),
-                                 m_data_sp->GetByteSize());
-}
+    : m_data(data), m_header(&header),
+      m_directory_map(std::move(directory_map)) {}
 
 llvm::ArrayRef<uint8_t>
 MinidumpParser::GetStream(MinidumpStreamType stream_type) {
   auto iter = m_directory_map.find(static_cast<uint32_t>(stream_type));
   if (iter == m_directory_map.end())
     return {};
 
   // check if there is enough data
-  if (iter->second.rva + iter->second.data_size > m_data_sp->GetByteSize())
+  if (iter->second.rva + iter->second.data_size > m_data.size())
     return {};
 
-  return llvm::ArrayRef<uint8_t>(m_data_sp->GetBytes() + iter->second.rva,
-                                 iter->second.data_size);
+  return m_data.slice(iter->second.rva, iter->second.data_size);
 }
 
 llvm::Optional<std::string> MinidumpParser::GetMinidumpString(uint32_t rva) {
-  auto arr_ref = m_data_sp->GetData();
-  if (rva > arr_ref.size())
+  if (rva > m_data.size())
     return llvm::None;
-  arr_ref = arr_ref.drop_front(rva);
+  auto arr_ref = m_data.drop_front(rva);
   return parseMinidumpString(arr_ref);
 }
 
 UUID MinidumpParser::GetModuleUUID(const MinidumpModule *module) {
   auto cv_record =
       GetData().slice(module->CV_record.rva, module->CV_record.data_size);
 
   // Read the CV record signature

source/Formats/MinidumpYAML.cpp

+//===-- MinidumpYAML.cpp -----------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Formats/MinidumpYAML.h"
+#include "lldb/Formats/MinidumpParser.h"
+
+using namespace lldb_private::minidump;
+namespace yaml = llvm::yaml;
+
+template <typename EndianType>
+static inline void mapOptional(yaml::IO &IO, const char *Key, EndianType &Val,
+                               typename EndianType::value_type Default) {
+  EndianType EndianDefault(Default);
+  IO.mapOptional(Key, Val, EndianDefault);
+}
+
+static inline void mapOptional(yaml::IO &IO, const char *Key, uint8_t &Val,
+                               uint8_t Default) {
+  IO.mapOptional(Key, Val, Default);
+}
+
+/// Yaml-map an endian-aware type EndianType as some other type MapType.
+template <typename MapType, typename EndianType>
+static inline void mapAs(yaml::IO &IO, const char *Key, EndianType &Val) {
+  MapType Mapped = static_cast<MapType>(typename EndianType::value_type(Val));
+  IO.mapRequired(Key, Mapped);
+  Val = static_cast<typename EndianType::value_type>(Mapped);
+}
+
+/// Perform an optional yaml-mapping of an endian-aware type EndianType as some
+/// other type MapType.
+template <typename MapType, typename EndianType>
+static inline void mapOptionalAs(yaml::IO &IO, const char *Key, EndianType &Val,
+                                 MapType Default) {
+  MapType Mapped = static_cast<MapType>(typename EndianType::value_type(Val));
+  IO.mapOptional(Key, Mapped, Default);
+  Val = static_cast<typename EndianType::value_type>(Mapped);
+}
+
+namespace {
+/// Return the appropriate yaml Hex type for a given endian-aware type.
+template <typename EndianType> struct HexType;
+template <> struct HexType<llvm::support::ulittle16_t> {
+  using type = yaml::Hex16;
+};
+template <> struct HexType<llvm::support::ulittle32_t> {
+  using type = yaml::Hex32;
+};
+template <> struct HexType<llvm::support::ulittle64_t> {
+  using type = yaml::Hex64;
+};
+} // namespace
+
+/// Yaml-map an endian-aware type as an appropriately-sized hex value.
+template <typename EndianType>
+static void mapHex(yaml::IO &IO, const char *Key, EndianType &Val) {
+  mapAs<typename HexType<EndianType>::type>(IO, Key, Val);
+}
+
+/// Perform an optionam yaml-mapping of an endian-aware type as an
+/// appropriately-sized hex value.
+template <typename EndianType>
+static void mapOptionalHex(yaml::IO &IO, const char *Key, EndianType &Val,
+                           typename EndianType::value_type Default) {
+  mapOptionalAs<typename HexType<EndianType>::type>(IO, Key, Val, Default);
+}
+
+
+llvm::Expected<Object> Object::create(MinidumpParser &Parser) {
+  std::vector<std::unique_ptr<minidump::Stream>> Streams;
+  Streams.reserve(Parser.GetDirectoryMap().size());
+  for (const auto &KV: Parser.GetDirectoryMap()) {
+    auto ExpectedStream = Stream::create(MinidumpStreamType(KV.first), Parser);
+    if (!ExpectedStream)
+      return ExpectedStream.takeError();
+    Streams.push_back(std::move(*ExpectedStream));
+  }
+  return Object(Parser.GetHeader(), std::move(Streams));
+}
+
+Stream::~Stream() = default;
+
+Stream::StreamKind Stream::getKind(MinidumpStreamType Type) {
+  switch (Type) {
+  case MinidumpStreamType::LinuxEnviron:
+    return EnvironmentKind;
+  case MinidumpStreamType::SystemInfo:
+    return SystemInfoKind;
+  case MinidumpStreamType::LinuxCPUInfo:
+  case MinidumpStreamType::LinuxProcStatus:
+  case MinidumpStreamType::LinuxLSBRelease:
+  case MinidumpStreamType::LinuxCMDLine:
+  case MinidumpStreamType::LinuxMaps:
+  case MinidumpStreamType::LinuxProcStat:
+  case MinidumpStreamType::LinuxProcUptime:
+    return TextKind;
+  default:
+    return HexKind;
+  }
+}
+
+llvm::Expected<std::unique_ptr<Stream>> Stream::create(MinidumpStreamType Type,
+                                                       MinidumpParser &Parser) {
+  StreamKind Kind = getKind(Type);
+  switch (Kind) {
+  case HexKind:
+    return llvm::make_unique<HexStream>(Type, Parser.GetStream(Type));
+  case EnvironmentKind: {
+    llvm::StringRef Data = toStringRef(Parser.GetStream(Type));
+    llvm::SmallVector<llvm::StringRef, 0> Variables;
+    if (!Data.empty()) {
+      if (Data.back() != '\0')
+        return llvm::createStringError(llvm::inconvertibleErrorCode(),
+                                       "Environment is not null terminated.");
+      Data.split(Variables, '\0');
+      Variables.pop_back();
+    }
+    return llvm::make_unique<EnvironmentStream>(std::move(Variables));
+  }
+  case SystemInfoKind:
+    if (const MinidumpSystemInfo *Info = Parser.GetSystemInfo())
+      return llvm::make_unique<SystemInfoStream>(*Info);
+    return llvm::createStringError(llvm::inconvertibleErrorCode(),
+                                   "Minidump SystemInfo stream corrupted.");
+  case TextKind:
+    return llvm::make_unique<TextStream>(Type,
+                                         toStringRef(Parser.GetStream(Type)));
+  }
+  llvm_unreachable("Unhandled stream kind!");
+}
+
+std::unique_ptr<Stream> Stream::create(MinidumpStreamType Type) {
+  StreamKind Kind = getKind(Type);
+  switch (Kind) {
+  case HexKind:
+    return llvm::make_unique<HexStream>(Type);
+  case EnvironmentKind:
+    return llvm::make_unique<EnvironmentStream>();
+  case SystemInfoKind:
+    return llvm::make_unique<SystemInfoStream>();
+  case TextKind:
+    return llvm::make_unique<TextStream>(Type);
+  }
+  llvm_unreachable("Unhandled stream kind!");
+}
+
+void yaml::MappingTraits<Object>::mapping(IO &IO, Object &O) {
+  IO.mapTag("!minidump", true);
+  mapOptionalHex(IO, "Signature", O.Header.signature,
+                 uint32_t(MinidumpHeaderConstants::Signature));
+  mapOptionalHex(IO, "Version", O.Header.version,
+                 uint32_t(MinidumpHeaderConstants::Version));
+  mapOptionalHex(IO, "Flags", O.Header.flags, 0);
+  IO.mapRequired("Streams", O.Streams);
+}
+
+static void streamMapping(yaml::IO &IO, HexStream &Stream) {
+  IO.mapOptional("Content", Stream.Content);
+  IO.mapOptional("Size", Stream.Size, yaml::Hex32(Stream.Content.binary_size()));
+}
+
+static void streamMapping(yaml::IO &IO, TextStream &Stream) {
+  IO.mapOptional("Text", Stream.Text);
+}
+
+static void streamMapping(yaml::IO &IO, EnvironmentStream &Stream) {
+  IO.mapRequired("Variables", Stream.Variables);
+}
+
+static void streamMapping(yaml::IO &IO, SystemInfoStream &Stream) {
+  MinidumpSystemInfo &Info = Stream.Info;
+  mapAs<MinidumpCPUArchitecture>(IO, "Processor Arch", Info.processor_arch);
+  mapOptional(IO, "Processor Level", Info.processor_level, 0);
+  mapOptional(IO, "Processor Revision", Info.processor_revision, 0);
+  mapOptional(IO, "Number of Processors", Info.number_of_processors, 0);
+  mapOptional(IO, "Product type", Info.product_type, 0);
+  mapOptional(IO, "Major Version", Info.major_version, 0);
+  mapOptional(IO, "Minor Version", Info.minor_version, 0);
+  mapOptional(IO, "Build Number", Info.build_number, 0);
+  mapAs<MinidumpOSPlatform>(IO, "Platform ID", Info.platform_id);
+  mapOptionalHex(IO, "CSD Version RVA", Info.csd_version_rva, 0);
+  mapOptionalHex(IO, "Suite Mask", Info.suit_mask, 0);
+  mapOptionalHex(IO, "Reserved2", Info.reserved2, 0);
+  switch (MinidumpCPUArchitecture(uint16_t(Info.processor_arch))) {
+  case MinidumpCPUArchitecture::X86:
+  case MinidumpCPUArchitecture::AMD64:
+    IO.mapOptional("CPU", Info.cpu.x86_cpu_info);
+    break;
+  case MinidumpCPUArchitecture::ARM:
+  case MinidumpCPUArchitecture::ARM64:
+    IO.mapOptional("CPU", Info.cpu.arm_cpu_info);
+    break;
+  default:
+    IO.mapOptional("CPU", Info.cpu.other_cpu_info);
+    break;
+  }
+}
+
+void yaml::MappingTraits<std::unique_ptr<lldb_private::minidump::Stream>>::
+    mapping(IO &IO, std::unique_ptr<lldb_private::minidump::Stream> &S) {
+  MinidumpStreamType Type;
+  if (IO.outputting())
+    Type = S->Type;
+  IO.mapRequired("Type", Type);
+
+  if (!IO.outputting())
+    S = lldb_private::minidump::Stream::create(Type);
+  switch (S->Kind) {
+  case lldb_private::minidump::Stream::HexKind:
+    streamMapping(IO, llvm::cast<HexStream>(*S));
+    break;
+  case lldb_private::minidump::Stream::EnvironmentKind:
+    streamMapping(IO, llvm::cast<EnvironmentStream>(*S));
+    break;
+  case lldb_private::minidump::Stream::SystemInfoKind:
+    streamMapping(IO, llvm::cast<SystemInfoStream>(*S));
+    break;
+  case lldb_private::minidump::Stream::TextKind:
+    streamMapping(IO, llvm::cast<TextStream>(*S));
+    break;
+  }
+}
+
+void yaml::ScalarEnumerationTraits<MinidumpStreamType>::enumeration(
+    IO &IO, MinidumpStreamType &Type) {
+#define LLDB_HANDLE_STREAM_TYPE(CODE, NAME)                                    \
+  IO.enumCase(Type, #NAME, MinidumpStreamType::NAME);
+#include "lldb/Formats/MinidumpEnums.def"
+  IO.enumFallback<Hex32>(Type);
+}
+
+void yaml::ScalarEnumerationTraits<MinidumpCPUArchitecture>::enumeration(
+    IO &IO, MinidumpCPUArchitecture &Arch) {
+#define LLDB_HANDLE_ARCH(CODE, NAME)                                           \
+  IO.enumCase(Arch, #NAME, MinidumpCPUArchitecture::NAME);
+#include "lldb/Formats/MinidumpEnums.def"
+  IO.enumFallback<Hex16>(Arch);
+}
+
+void yaml::ScalarEnumerationTraits<MinidumpOSPlatform>::enumeration(
+    IO &IO, MinidumpOSPlatform &Plat) {
+#define LLDB_HANDLE_PLATFORM(CODE, NAME)                                       \
+  IO.enumCase(Plat, #NAME, MinidumpOSPlatform::NAME);
+#include "lldb/Formats/MinidumpEnums.def"
+  IO.enumFallback<Hex32>(Plat);
+}
+
+void yaml::MappingTraits<MinidumpCPUInfo::ArmCPUInfo>::mapping(
+    IO &IO, MinidumpCPUInfo::ArmCPUInfo &Info) {
+  mapHex(IO, "CPUID", Info.cpuid);
+  mapOptionalHex(IO, "ELF hwcaps", Info.elf_hwcaps, 0);
+}
+
+void yaml::MappingTraits<MinidumpCPUInfo::OtherCPUInfo>::mapping(
+    IO &IO, MinidumpCPUInfo::OtherCPUInfo &Info) {
+  llvm::MutableArrayRef<uint8_t> Features(Info.processor_features);
+  yaml::BinaryRef FeaturesRef(Features);
+
+  IO.mapRequired("Features", FeaturesRef);
+
+  llvm::SmallString<16> FeaturesStorage;
+  llvm::raw_svector_ostream FeaturesStream(FeaturesStorage);
+  FeaturesRef.writeAsBinary(FeaturesStream);
+  std::fill(Features.begin(), Features.end(), 0);
+  std::memcpy(Features.begin(), FeaturesStorage.begin(),
+              std::min(FeaturesStorage.size(), Features.size()));
+}
+
+void yaml::MappingTraits<MinidumpCPUInfo::X86CPUInfo>::mapping(
+    IO &IO, MinidumpCPUInfo::X86CPUInfo &Info) {
+  llvm::StringRef VendorID(Info.vendor_id, sizeof(Info.vendor_id));
+  IO.mapRequired("Vendor ID", VendorID);
+  std::fill_n(Info.vendor_id, sizeof(Info.vendor_id), 0);
+  std::copy_n(VendorID.begin(),
+              std::min(VendorID.size(), sizeof(Info.vendor_id)),
+              Info.vendor_id);
+
+  mapHex(IO, "Version Info", Info.feature_information);
+  mapHex(IO, "Feature Info", Info.feature_information);
+  mapOptionalHex(IO, "AMD Extended CPU Features",
+                 Info.amd_extended_cpu_features, 0);
+}

source/Plugins/Process/minidump/ProcessMinidump.cpp

void ProcessMinidump::Initialize() {
   });
 }
 
 void ProcessMinidump::Terminate() {
   PluginManager::UnregisterPlugin(ProcessMinidump::CreateInstance);
 }
 
 Status ProcessMinidump::DoLoadCore() {
-  auto expected_parser = MinidumpParser::Create(m_core_data);
+  auto expected_parser = MinidumpParser::Create(m_core_data->GetData());
   if (!expected_parser)
     return Status(expected_parser.takeError());
   m_minidump_parser = std::move(*expected_parser);
 
   Status error;
 
   // Do we support the minidump's architecture?
   ArchSpec arch = GetArchitecture();

tools/CMakeLists.txt

 add_subdirectory(argdumper)
+add_subdirectory(core2yaml)
 add_subdirectory(driver)
 add_subdirectory(intel-features)
 add_subdirectory(lldb-instr)
 add_subdirectory(lldb-mi)
 add_subdirectory(lldb-test)
 add_subdirectory(lldb-vscode)
 
 if (CMAKE_SYSTEM_NAME MATCHES "Darwin")
   add_subdirectory(darwin-debug)
   add_subdirectory(debugserver)
 endif()
 
 if (LLDB_CAN_USE_LLDB_SERVER AND NOT SKIP_LLDB_SERVER_BUILD)
   add_subdirectory(lldb-server)
 endif()

tools/core2yaml/CMakeLists.txt

+add_lldb_tool(core2yaml
+  core2yaml.cpp
+
+  LINK_LIBS
+    lldbUtility
+    lldbFormats
+
+  LINK_COMPONENTS
+    Support
+  )

tools/core2yaml/core2yaml.cpp

+//===-- core2yaml.cpp --------------------------------------------*- C++-*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Formats/MinidumpParser.h"
+#include "lldb/Formats/MinidumpTypes.h"
+#include "lldb/Formats/MinidumpYAML.h"
+#include "lldb/Utility/DataBufferHeap.h"
+#include "llvm/Support/CommandLine.h"
+#include "llvm/Support/Error.h"
+
+using namespace lldb_private;
+using namespace lldb_private::minidump;
+namespace yaml = llvm::yaml;
+namespace cl = llvm::cl;
+
+
+static llvm::Error dumpInput(llvm::StringRef File) {
+  auto BufferOrErr = llvm::MemoryBuffer::getFileOrSTDIN(
+      File, /*FileSize*/ -1, /*RequiresNullTerminator*/ false);
+  if (!BufferOrErr)
+    return llvm::errorCodeToError(BufferOrErr.getError());
+  llvm::Expected<MinidumpParser> Parser = MinidumpParser::Create(
+      arrayRefFromStringRef(BufferOrErr->get()->getBuffer()));
+  if (!Parser)
+    return Parser.takeError();
+
+  yaml::Output yout(llvm::outs());
+  auto ExpectedModel = Object::create(*Parser);
+  if (!ExpectedModel)
+    return ExpectedModel.takeError();
+  yout << *ExpectedModel;
+  return llvm::Error::success();
+}
+
+static void reportError(llvm::StringRef Input, llvm::Error Err) {
+  if (Input == "-")
+    Input = "<stdin>";
+  llvm::errs() << "Error reading file: " << Input << ": ";
+  logAllUnhandledErrors(std::move(Err), llvm::errs());
+}
+
+static cl::opt<std::string>
+    InputFilename(cl::Positional, cl::desc("<input file>"), cl::init("-"));
+
+int main(int argc, char *argv[]) {
+  cl::ParseCommandLineOptions(argc, argv);
+
+  if (llvm::Error Err = dumpInput(InputFilename)) {
+    reportError(InputFilename, std::move(Err));
+    return 1;
+  }
+
+  return 0;
+}

unittests/Formats/MinidumpParserTest.cpp

 class MinidumpParserTest : public testing::Test {
 public:
   void SetUp() override { FileSystem::Initialize(); }
 
   void TearDown() override { FileSystem::Terminate(); }
 
   void SetUpData(const char *minidump_filename) {
     std::string filename = GetInputFilePath(minidump_filename);
-    auto BufferPtr = FileSystem::Instance().CreateDataBuffer(filename, -1, 0);
-    ASSERT_NE(BufferPtr, nullptr);
+    data_sp = FileSystem::Instance().CreateDataBuffer(filename, -1, 0);
+    ASSERT_NE(data_sp, nullptr);
     llvm::Expected<MinidumpParser> expected_parser =
-        MinidumpParser::Create(BufferPtr);
+        MinidumpParser::Create(data_sp->GetData());
     ASSERT_THAT_EXPECTED(expected_parser, llvm::Succeeded());
     parser = std::move(*expected_parser);
     ASSERT_GT(parser->GetData().size(), 0UL);
   }
 
   void InvalidMinidump(const char *minidump_filename, uint64_t load_size) {
     std::string filename = GetInputFilePath(minidump_filename);
     auto BufferPtr =
         FileSystem::Instance().CreateDataBuffer(filename, load_size, 0);
     ASSERT_NE(BufferPtr, nullptr);
 
-    EXPECT_THAT_EXPECTED(MinidumpParser::Create(BufferPtr), llvm::Failed());
+    EXPECT_THAT_EXPECTED(MinidumpParser::Create(BufferPtr->GetData()),
+                         llvm::Failed());
   }
 
+  lldb::DataBufferSP data_sp;
   llvm::Optional<MinidumpParser> parser;
 };
 
 TEST_F(MinidumpParserTest, GetThreadsAndGetThreadContext) {
   SetUpData("linux-x86_64.dmp");
   llvm::ArrayRef<MinidumpThread> thread_list;
 
   thread_list = parser->GetThreads();