Introduce the "Formats" module and move LinuxProcMaps parser to it

[labath]

Summary

This patch introduces a "Formats" library, whose goal is to contain
serialization and deserialization code for various textual and binary
formats that lldb needs to work with. The motivation is to enable
building various tools on top of this, which do not need to depend on
the entire lldb codebase. (I have written a slightly more elaborate
version of this is given in the lldb architecture docs, which I update
to mention the new library).

LinuxProcMaps is just the beginning here. As the next immediate step I
plan to move the Minidump parsing code (currently burried in
source/Plugins/Process/minidump) here as well, but in the future I think
this library could contain the gdb-remote protocol code (currently in
source/Plugins/Process/gdb-remote) and the debug info protocol code
(currently in Zach's head).

[zturner]

Great idea, this is a nice analogue to LLVM/BinaryFormat.

But, "Formats" is a little generic. What about FileFormats or something?

[labath]

FileFormats wouldn't be generic enough to capture the gdb-remote or debug-info-server protocol code. However, even calling gdb-remote protocol a "format" is still a bit of a stretch. I think there is enough similarity between gdb-remote and minidump code for them to be in the same module (both convert some (and often the same) data structures into some (file or wire) format), but they're also sufficiently different for them to live in different places. So yeah, if we want the protocol code to live in a different library (Protocols ?), then FileFormats is a better name here.

[JDevlieghere]

In D58972#1418931, @labath wrote:

FileFormats wouldn't be generic enough to capture the gdb-remote or debug-info-server protocol code. However, even calling gdb-remote protocol a "format" is still a bit of a stretch. I think there is enough similarity between gdb-remote and minidump code for them to be in the same module (both convert some (and often the same) data structures into some (file or wire) format), but they're also sufficiently different for them to live in different places. So yeah, if we want the protocol code to live in a different library (Protocols ?), then FileFormats is a better name here.

I was going to propose Protocols for consideration but that doesn't really work for minidump. Thinking a little about this, Formats, although generic, is probably the best.

[labath]

I'm going to wait to see how the llvm option pans out. If minidump parsing code ends up there, we lose the biggest customer of this and so we may revisit the name and/or scope of the new module.

[labath]

minidump parser will go into llvm, which removes the main incentive for creating this new module. It's possible we may need something like this in the future, but should wait until there is a real use case for it.

docs/use/architecture.rst

 The expression parser uses a full instance of the Clang compiler in order to
 accurately evaluate expressions. Hooks have been put into Clang so that the
 compiler knows to ask about identifiers it doesn't know about. Once expressions
 have be compiled into an AST, we can then traverse this AST and either generate
 a DWARF expression that contains simple opcodes that can be quickly
 re-evaluated each time an expression needs to be evaluated, or JIT'ed up into
 code that can be run on the process being debugged.
 
+Formats
+-------
+
+The Formats module contains the code for handling various types of data formats
+that LLDB needs to work with. This includes serialization and deserialization
+code for all kinds of textual and binary formats, ranging from the contents of
+the linux /proc/PID/maps file, to windows minidump files. The goal of this
+library is to present the data in an easily accessible fashion. However, it does
+not connect this data to higher level lldb concepts (Targets, Modules, ...) --
+that's the job of individual plugins. The goal of this is so that another
+application (or lldb-server) can use the (de)serialization code from this
+library without depending on the lldb object model, or being constrained by it.
+
 Host
 ----
 
 LLDB tries to abstract itself from the host upon which it is currently running
 by providing a host abstraction layer. This layer includes functionality, whose
 implementation varies wildly from host to host.
 
 Host functionality includes abstraction layers for:

include/lldb/Formats/LinuxProcMaps.h

 //===-- LinuxProcMaps.h -----------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_LinuxProcMaps_H_
 #define liblldb_LinuxProcMaps_H_
 
 #include "lldb/lldb-forward.h"
 #include "llvm/ADT/StringRef.h"
 #include <functional>
 
 
 namespace lldb_private {
 
 typedef std::function<bool(const lldb_private::MemoryRegionInfo &,
                            const lldb_private::Status &)> LinuxMapCallback;
 
 void ParseLinuxMapRegions(llvm::StringRef linux_map,
                           LinuxMapCallback const &callback);
 
 } // namespace lldb_private
 
 #endif // liblldb_LinuxProcMaps_H_

include/lldb/module.modulemap

 
 module lldb_API {
   requires cplusplus
 
   umbrella "API"
   module * { export * }
 }
 
+module lldb_Formats {
+  requires cplusplus
+
+  umbrella "Formats"
+  module * { export * }
+}
+
 module lldb_Host {
   requires cplusplus
 
   // Because we have OS-specific headers in Host, we just list
   // all OS-independent headers here that will include the correct
   // OS-specific header for us.
   module ConnectionFileDescriptor { header "Host/ConnectionFileDescriptor.h" export * }
   module Debug { header "Host/Debug.h" export * }

source/CMakeLists.txt

add_lldb_library(lldbBase
   ${lldbBase_SOURCES}
   )
 
 add_subdirectory(Breakpoint)
 add_subdirectory(Commands)
 add_subdirectory(Core)
 add_subdirectory(DataFormatters)
 add_subdirectory(Expression)
+add_subdirectory(Formats)
 add_subdirectory(Host)
 add_subdirectory(Initialization)
 add_subdirectory(Interpreter)
 add_subdirectory(Plugins)
 add_subdirectory(Symbol)
 add_subdirectory(Target)
 add_subdirectory(Utility)
 
 # Build API last.  Since liblldb needs to link against every other target, it needs
 # those targets to have already been created.
 add_subdirectory(API)

source/Formats/CMakeLists.txt

+add_lldb_library(lldbFormats
+  LinuxProcMaps.cpp
+
+  LINK_LIBS
+    lldbUtility
+  LINK_COMPONENTS
+    Support
+  )

source/Formats/LinuxProcMaps.cpp

 //===-- LinuxProcMaps.cpp ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
-#include "LinuxProcMaps.h"
+#include "lldb/Formats/LinuxProcMaps.h"
 #include "lldb/Utility/MemoryRegionInfo.h"
 #include "lldb/Utility/Status.h"
 #include "lldb/Utility/StringExtractor.h"
 #include "llvm/ADT/StringRef.h"
 
 using namespace lldb_private;
 
 static Status

source/Plugins/Process/Linux/CMakeLists.txt

 add_lldb_library(lldbPluginProcessLinux PLUGIN
   NativeProcessLinux.cpp
   NativeRegisterContextLinux.cpp
   NativeRegisterContextLinux_arm.cpp
   NativeRegisterContextLinux_arm64.cpp
   NativeRegisterContextLinux_mips64.cpp
   NativeRegisterContextLinux_ppc64le.cpp
   NativeRegisterContextLinux_s390x.cpp
   NativeRegisterContextLinux_x86_64.cpp
   NativeThreadLinux.cpp
   ProcessorTrace.cpp
   SingleStepCheck.cpp
 
   LINK_LIBS
     lldbCore
+    lldbFormats
     lldbHost
     lldbSymbol
     lldbTarget
     lldbUtility
     lldbPluginProcessPOSIX
     lldbPluginProcessUtility
   LINK_COMPONENTS
     Support
   )

source/Plugins/Process/Linux/NativeProcessLinux.cpp

 #include "lldb/Utility/Status.h"
 #include "lldb/Utility/StringExtractor.h"
 #include "llvm/Support/Errno.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Threading.h"
 
 #include "NativeThreadLinux.h"
 #include "Plugins/Process/POSIX/ProcessPOSIXLog.h"
-#include "Plugins/Process/Utility/LinuxProcMaps.h"
+#include "lldb/Formats/LinuxProcMaps.h"
 #include "Procfs.h"
 
 #include <linux/unistd.h>
 #include <sys/socket.h>
 #include <sys/syscall.h>
 #include <sys/types.h>
 #include <sys/user.h>
 #include <sys/wait.h>

source/Plugins/Process/Utility/CMakeLists.txt

 add_lldb_library(lldbPluginProcessUtility PLUGIN
   DynamicRegisterInfo.cpp
   FreeBSDSignals.cpp
   GDBRemoteSignals.cpp
   HistoryThread.cpp
   HistoryUnwind.cpp
   InferiorCallPOSIX.cpp
-  LinuxProcMaps.cpp
   LinuxSignals.cpp
   MipsLinuxSignals.cpp
   NativeRegisterContextRegisterInfo.cpp
   NetBSDSignals.cpp
   RegisterContextDarwin_arm.cpp
   RegisterContextDarwin_arm64.cpp
   RegisterContextDarwin_i386.cpp
   RegisterContextDarwin_x86_64.cpp

source/Plugins/Process/minidump/CMakeLists.txt

 add_lldb_library(lldbPluginProcessMinidump PLUGIN
   MinidumpTypes.cpp
   MinidumpParser.cpp
   RegisterContextMinidump_ARM.cpp
   RegisterContextMinidump_ARM64.cpp
   RegisterContextMinidump_x86_32.cpp
   RegisterContextMinidump_x86_64.cpp
   ProcessMinidump.cpp
   ThreadMinidump.cpp
 
   LINK_LIBS
     lldbCore
+    lldbFormats
     lldbTarget
     lldbUtility
     lldbPluginProcessUtility
     lldbPluginProcessElfCore
   LINK_COMPONENTS
     Support
   )

source/Plugins/Process/minidump/MinidumpParser.cpp

 //===-- MinidumpParser.cpp ---------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "MinidumpParser.h"
 #include "NtStructures.h"
 #include "RegisterContextMinidump_x86_32.h"
 
 #include "lldb/Utility/LLDBAssert.h"
-#include "Plugins/Process/Utility/LinuxProcMaps.h"
+#include "lldb/Formats/LinuxProcMaps.h"
 
 // C includes
 // C++ includes
 #include <algorithm>
 #include <map>
 #include <vector>
 #include <utility>
 

unittests/CMakeLists.txt

 endfunction()
 
 add_subdirectory(TestingSupport)
 add_subdirectory(Breakpoint)
 add_subdirectory(Core)
 add_subdirectory(Disassembler)
 add_subdirectory(Editline)
 add_subdirectory(Expression)
+add_subdirectory(Formats)
 add_subdirectory(Host)
 add_subdirectory(Interpreter)
 add_subdirectory(Language)
 add_subdirectory(ObjectFile)
 add_subdirectory(Platform)
 add_subdirectory(Process)
 add_subdirectory(ScriptInterpreter)
 add_subdirectory(Signals)

unittests/Formats/CMakeLists.txt

+add_lldb_unittest(LLDBFormatsTests
+  LinuxProcMapsTest.cpp
+
+  LINK_LIBS
+    lldbFormats
+  LINK_COMPONENTS
+    Support
+  )

unittests/Formats/LinuxProcMapsTest.cpp

+//===-- LinuxProcMapsTest.cpp -----------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "lldb/Formats/LinuxProcMaps.h"
+#include "lldb/Utility/MemoryRegionInfo.h"
+#include "lldb/Utility/Status.h"
+#include "gtest/gtest.h"
+
+using namespace lldb_private;
+
+TEST(LinuxProcMaps, ParseLinuxMapRegions) {
+  std::vector<MemoryRegionInfo> Infos;
+  ParseLinuxMapRegions("557bdd004000-557bdd0bc000 r-xp 00000000 fd:01 3012381  "
+                       "                  /bin/bash",
+                       [&](const MemoryRegionInfo &Info, const Status &ST) {
+                         EXPECT_TRUE(ST.Success());
+                         Infos.push_back(Info);
+                         return true;
+                       });
+  ASSERT_EQ(1u, Infos.size());
+  EXPECT_EQ(0x557bdd004000u, Infos[0].GetRange().GetRangeBase());
+  EXPECT_EQ(0x557bdd0bc000u, Infos[0].GetRange().GetRangeEnd());
+  EXPECT_EQ(MemoryRegionInfo::eYes, Infos[0].GetReadable());
+  EXPECT_EQ(MemoryRegionInfo::eNo, Infos[0].GetWritable());
+  EXPECT_EQ(MemoryRegionInfo::eYes, Infos[0].GetExecutable());
+  EXPECT_STREQ("/bin/bash", Infos[0].GetName().GetCString());
+}

www/architecture/index.html

<div id="middle">
            <p>LLDB has many code groupings that makeup the source base:</p>
                    <ul>
                <li><a href="#api">API</a></li>
                 <li><a href="#breakpoint">Breakpoint</a></li>
                 <li><a href="#commands">Commands</a></li>
                 <li><a href="#core">Core</a></li>
                 <li><a href="#dataformatters">DataFormatters</a></li>
                  <li><a href="#expression">Expression</a></li>
+                 <li><a href="#formats">Formats</a></li>
                  <li><a href="#host">Host</a></li>
                  <li><a href="#interpreter">Interpreter</a></li>
                  <li><a href="#symbol">Symbol</a></li>
                  <li><a href="#targ">Target</a></li>
                  <li><a href="#utility">Utility</a></li>
               </ul>
         </div>
         <div class="postfooter"></div>
<div id="middle">
                 expressions have be compiled into an AST, we can then
                 traverse this AST and either generate a DWARF expression
                 that contains simple opcodes that can be quickly re-evaluated
                 each time an expression needs to be evaluated, or JIT'ed
                 up into code that can be run on the process being debugged.</p>
         </div>
         <div class="postfooter"></div>
       </div>
+      <a name="formats"></a>
+      <div class="post">
+        <h1 class ="postheader">Formats</h1>
+        <div class="postcontent">
+                                   <p>
+                                   The Formats module contains the code for
+                                   handling various types of data formats that
+                                   LLDB needs to work with. This includes
+                                   serialization and deserialization code for
+                                   all kinds of textual and binary formats,
+                                   ranging from the contents of the linux
+                                   /proc/PID/maps file, to windows minidump
+                                   files. The goal of this library is to present
+                                   the data in an easily accessible fashion.
+                                   However, it does not connect this data to
+                                   higher level lldb concepts (Targets, Modules,
+                                   ...) -- that's the job of individual plugins.
+                                   The goal of this is so that another
+                                   application (or lldb-server) can use the
+                                   (de)serialization code from this library
+                                   without depending on the lldb object model,
+                                   or being constrained by it.
+                                   </p>
+        </div>
+        <div class="postfooter"></div>
+      </div>
       <a name="host"></a>
       <div class="post">
         <h1 class ="postheader">Host</h1>
         <div class="postcontent">
           <p>
           LLDB tries to abstract itself from the host upon which
           it is currently running by providing a host abstraction
           layer. This layer includes functionality, whose