[lldb/Reproducers] Support multiple GDB remotes

[JDevlieghere]

Summary

When running the test suite with always capture on, a handful of tests are failing because they have multiple targets and therefore multiple GDB remote connections. The current reproducer infrastructure is capable of dealing with that.

This patch reworks the GDB remote provider to support multiple GDB remote connections, similar to how the reproducers support shadowing multiple command interpreter inputs. The provider now keeps a list of packet recorders which deal with a single GDB remote connection. During replay we rely on the order of creation to match the number of packets to the GDB remote connection.

[JDevlieghere]

This patch is still missing a test and could use another pass over the PacketRecorder and ProcessGDBRemoteProvider to eliminate code duplication with DataRecorder and the CommandProvider on which they're based respectively. I wasn't able to get around to that today but figured I'd upload the patch anyway to give you a change to look at the approach.

[labath]

Looks pretty straight-forward, but I am still not happy about how more and more of ProcessGDBRemote is infiltrating its way into the Utility module. I think that the reproducer code specific to gdb-remote should live next to the code it's capturing, and Utility/Reproducer.cpp should just provide the general architecture...

lldb/include/lldb/Utility/Reproducer.h

429

is this used anywhere?

lldb/source/Commands/CommandObjectReproducer.cpp

441

static? Is it not possible to dump multiple reproducer files in one session ?

447

maybe while ((gdb_file = multi_loader->GetNextFile())) (with an optional != None) ?

[JDevlieghere]

• Address CR feedback.
• Introduce AbstractRecorder to reuse code between the PacketRecorder and the DataRecorder.
• Move GDB related reproducer code into Utility/GDBRemote.h.
• Rename ProcessGDBRemoteProvider to GDBRemoteProvider.

[labath]

This looks better. Ideally I wouldn't even have Utility/GDBRemote.h, but as Process/gdb-remote is not a very good place for it either (it messes with lldb-server dependencies), we can live with it for now.

lldb/include/lldb/Utility/Reproducer.h

23

I guess this should not be needed anymore?

lldb/source/Commands/CommandObjectReproducer.cpp

439

So what about this static thing?

[JDevlieghere]

lldb/source/Commands/CommandObjectReproducer.cpp

439

This is totally bogus, I don't even remember why or how it ended up there.

[JDevlieghere]

In D71105#1774936, @labath wrote:

This looks better. Ideally I wouldn't even have Utility/GDBRemote.h, but as Process/gdb-remote is not a very good place for it either (it messes with lldb-server dependencies), we can live with it for now.

Yeah, the other problem is the command object which needs to be able to deserialize the packets to dump them. Layering is hard :(

[JDevlieghere]

Add test

lldb/include/lldb/Utility/GDBRemote.h

 #ifndef liblldb_GDBRemote_h_
 #define liblldb_GDBRemote_h_
 
+#include "lldb/Utility/FileSpec.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/StreamString.h"
 #include "lldb/lldb-enumerations.h"
 #include "lldb/lldb-public.h"

     std::string data;
   };
 
-  void Serialize(llvm::raw_ostream &strm) const;
   void Dump(Stream &strm) const;
 
   BinaryData packet;

   llvm::StringRef GetTypeStr() const;
 };
 
+namespace repro {
+class PacketRecorder : public AbstractRecorder {
+public:
+  PacketRecorder(const FileSpec &filename, std::error_code &ec)
+      : AbstractRecorder(filename, ec) {}
+
+  static llvm::Expected<std::unique_ptr<PacketRecorder>>
+  Create(const FileSpec &filename);
+
+  void Record(const GDBRemotePacket &packet);
+};
+
+class GDBRemoteProvider : public repro::Provider<GDBRemoteProvider> {
+public:
+  struct Info {
+    static const char *name;
+    static const char *file;
+  };
+
+  GDBRemoteProvider(const FileSpec &directory) : Provider(directory) {}
+
+  llvm::raw_ostream *GetHistoryStream();
+  PacketRecorder *GetNewPacketRecorder();
+
+  void SetCallback(std::function<void()> callback) {
+    m_callback = std::move(callback);
+  }
+
+  void Keep() override;
+  void Discard() override;
+
+  static char ID;
+
+private:
+  std::function<void()> m_callback;
+  std::unique_ptr<llvm::raw_fd_ostream> m_stream_up;
+  std::vector<std::unique_ptr<PacketRecorder>> m_packet_recorders;
+};
+
+} // namespace repro
 } // namespace lldb_private
 
 LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(lldb_private::GDBRemotePacket)

lldb/include/lldb/Utility/Reproducer.h

   static char ID;
 };
 
-class DataRecorder {
-public:
-  DataRecorder(const FileSpec &filename, std::error_code &ec)
+class AbstractRecorder {
+protected:
+  AbstractRecorder(const FileSpec &filename, std::error_code &ec)
       : m_filename(filename.GetFilename().GetStringRef()),
         m_os(filename.GetPath(), ec, llvm::sys::fs::OF_Text), m_record(true) {}
 
-  static llvm::Expected<std::unique_ptr<DataRecorder>>
-  Create(const FileSpec &filename);
-
-  template <typename T> void Record(const T &t, bool newline = false) {
-    if (!m_record)
-      return;
-    m_os << t;
-    if (newline)
-      m_os << '\n';
-    m_os.flush();
-  }
-
+public:
   const FileSpec &GetFilename() { return m_filename; }
 
   void Stop() {

 
 private:
   FileSpec m_filename;
+
+protected:
   llvm::raw_fd_ostream m_os;
   bool m_record;
 };
 
+class DataRecorder : public AbstractRecorder {
+public:
+  DataRecorder(const FileSpec &filename, std::error_code &ec)
+      : AbstractRecorder(filename, ec) {}
+
+  static llvm::Expected<std::unique_ptr<DataRecorder>>
+  Create(const FileSpec &filename);
+
+  template <typename T> void Record(const T &t, bool newline = false) {
+    if (!m_record)
+      return;
+    m_os << t;
+    if (newline)
+      m_os << '\n';
+    m_os.flush();
+  }
+};
+
 class CommandProvider : public Provider<CommandProvider> {
 public:
   struct Info {

   std::vector<std::unique_ptr<DataRecorder>> m_data_recorders;
 };
 
-class ProcessGDBRemoteProvider
-    : public repro::Provider<ProcessGDBRemoteProvider> {
-public:
-  struct Info {
-    static const char *name;
-    static const char *file;
-  };
-
-  ProcessGDBRemoteProvider(const FileSpec &directory) : Provider(directory) {}
-
-  llvm::raw_ostream *GetHistoryStream();
-
-  void SetCallback(std::function<void()> callback) {
-    m_callback = std::move(callback);
-  }
-
-  void Keep() override { m_callback(); }
-  void Discard() override { m_callback(); }
-
-  static char ID;
-
-private:
-  std::function<void()> m_callback;
-  std::unique_ptr<llvm::raw_fd_ostream> m_stream_up;
-};
-
 /// The generator is responsible for the logic needed to generate a
 /// reproducer. For doing so it relies on providers, who serialize data that
 /// is necessary for reproducing  a failure.

   mutable std::mutex m_mutex;
 };
 
-/// Helper class for replaying commands through the reproducer.
-class CommandLoader {
+template <typename T> class MultiLoader {
 public:
-  CommandLoader(std::vector<std::string> files) : m_files(files) {}
+  MultiLoader(std::vector<std::string> files) : m_files(files) {}
+
+  static std::unique_ptr<MultiLoader> Create(Loader *loader) {
+    if (!loader)
+      return {};
+
+    FileSpec file = loader->GetFile<typename T::Info>();
+    if (!file)
+      return {};
+
+    auto error_or_file = llvm::MemoryBuffer::getFile(file.GetPath());
+    if (auto err = error_or_file.getError())
+      return {};
 
-  static std::unique_ptr<CommandLoader> Create(Loader *loader);
-  llvm::Optional<std::string> GetNextFile();
+    std::vector<std::string> files;
+    llvm::yaml::Input yin((*error_or_file)->getBuffer());
+    yin >> files;
+
+    if (auto err = yin.error())
+      return {};
+
+    for (auto &file : files) {
+      FileSpec absolute_path =
+          loader->GetRoot().CopyByAppendingPathComponent(file);
+      file = absolute_path.GetPath();
+    }
+
+    return std::make_unique<MultiLoader<T>>(std::move(files));
+  }
+
+  llvm::Optional<std::string> GetNextFile() {
+    if (m_index >= m_files.size())
+      return {};
+    return m_files[m_index++];
+  }
 
 private:
   std::vector<std::string> m_files;

lldb/source/API/SBDebugger.cpp

 
   FileSP file_sp = file.m_opaque_sp;
 
-  static std::unique_ptr<repro::CommandLoader> loader =
-      repro::CommandLoader::Create(repro::Reproducer::Instance().GetLoader());
+  static std::unique_ptr<repro::MultiLoader<repro::CommandProvider>> loader =
+      repro::MultiLoader<repro::CommandProvider>::Create(
+          repro::Reproducer::Instance().GetLoader());
   if (loader) {
     llvm::Optional<std::string> nextfile = loader->GetNextFile();
     FILE *fh = nextfile ? FileSystem::Instance().Fopen(nextfile->c_str(), "r")

lldb/source/Commands/CommandObjectReproducer.cpp

       return true;
     }
     case eReproducerProviderCommands: {
-      // Create a new command loader.
-      std::unique_ptr<repro::CommandLoader> command_loader =
-          repro::CommandLoader::Create(loader);
-      if (!command_loader) {
+      std::unique_ptr<repro::MultiLoader<repro::CommandProvider>> multi_loader =
+          repro::MultiLoader<repro::CommandProvider>::Create(loader);
+      if (!multi_loader) {
         SetError(result,
                  make_error<StringError>(llvm::inconvertibleErrorCode(),
                                          "Unable to create command loader."));

       }
 
       // Iterate over the command files and dump them.
-      while (true) {
-        llvm::Optional<std::string> command_file =
-            command_loader->GetNextFile();
+      llvm::Optional<std::string> command_file;
+      while ((command_file = multi_loader->GetNextFile())) {
         if (!command_file)
           break;
 

       return true;
     }
     case eReproducerProviderGDB: {
-      FileSpec gdb_file = loader->GetFile<ProcessGDBRemoteProvider::Info>();
-      auto error_or_file = MemoryBuffer::getFile(gdb_file.GetPath());
-      if (auto err = error_or_file.getError()) {
-        SetError(result, errorCodeToError(err));
-        return false;
-      }
+      std::unique_ptr<repro::MultiLoader<repro::GDBRemoteProvider>>
+          multi_loader =
+              repro::MultiLoader<repro::GDBRemoteProvider>::Create(loader);
+      llvm::Optional<std::string> gdb_file;
+      while ((gdb_file = multi_loader->GetNextFile())) {
+        auto error_or_file = MemoryBuffer::getFile(*gdb_file);
+        if (auto err = error_or_file.getError()) {
+          SetError(result, errorCodeToError(err));
+          return false;
+        }
 
         std::vector<GDBRemotePacket> packets;
         yaml::Input yin((*error_or_file)->getBuffer());
         yin >> packets;
 
         if (auto err = yin.error()) {
           SetError(result, errorCodeToError(err));
           return false;
         }
 
         for (GDBRemotePacket &packet : packets) {
           packet.Dump(result.GetOutputStream());
+        }
       }
 
       result.SetStatus(eReturnStatusSuccessFinishResult);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.h

 #include "lldb/lldb-public.h"
 
 namespace lldb_private {
+namespace repro {
+class PacketRecorder;
+}
 namespace process_gdb_remote {
 
 enum GDBStoppointType {

                          // fork/exec to avoid having to connect/accept
 
   void DumpHistory(Stream &strm);
-  void SetHistoryStream(llvm::raw_ostream *strm);
+
+  void SetPacketRecorder(repro::PacketRecorder *recorder);
 
   static llvm::Error ConnectLocally(GDBRemoteCommunication &client,
                                     GDBRemoteCommunication &server);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp

 #include "lldb/Utility/FileSpec.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/RegularExpression.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/StreamString.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/ScopedPrinter.h"

 
 void GDBRemoteCommunication::DumpHistory(Stream &strm) { m_history.Dump(strm); }
 
-void GDBRemoteCommunication::SetHistoryStream(llvm::raw_ostream *strm) {
-  m_history.SetStream(strm);
+void GDBRemoteCommunication::SetPacketRecorder(
+    repro::PacketRecorder *recorder) {
+  m_history.SetRecorder(recorder);
 }
 
 llvm::Error

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationHistory.h

 #include <vector>
 
 #include "lldb/Utility/GDBRemote.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/lldb-public.h"
 #include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace lldb_private {
+namespace repro {
+class PacketRecorder;
+}
 namespace process_gdb_remote {
 
 /// The history keeps a circular buffer of GDB remote packets. The history is

   void Dump(Log *log) const;
   bool DidDumpToLog() const { return m_dumped_to_log; }
 
-  void SetStream(llvm::raw_ostream *strm) { m_stream = strm; }
+  void SetRecorder(repro::PacketRecorder *recorder) { m_recorder = recorder; }
 
 private:
   uint32_t GetFirstSavedPacketIndex() const {

   uint32_t m_curr_idx;
   uint32_t m_total_packet_count;
   mutable bool m_dumped_to_log;
-  llvm::raw_ostream *m_stream = nullptr;
+  repro::PacketRecorder *m_recorder = nullptr;
 };
 
 } // namespace process_gdb_remote

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationHistory.cpp

   m_packets[idx].bytes_transmitted = bytes_transmitted;
   m_packets[idx].packet_idx = m_total_packet_count;
   m_packets[idx].tid = llvm::get_threadid();
-  if (m_stream)
-    m_packets[idx].Serialize(*m_stream);
+  if (m_recorder)
+    m_recorder->Record(m_packets[idx]);
 }
 
 void GDBRemoteCommunicationHistory::AddPacket(const std::string &src,

   m_packets[idx].bytes_transmitted = bytes_transmitted;
   m_packets[idx].packet_idx = m_total_packet_count;
   m_packets[idx].tid = llvm::get_threadid();
-  if (m_stream)
-    m_packets[idx].Serialize(*m_stream);
+  if (m_recorder)
+    m_recorder->Record(m_packets[idx]);
 }
 
 void GDBRemoteCommunicationHistory::Dump(Stream &strm) const {

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp

                                    "async thread did exit");
 
   if (repro::Generator *g = repro::Reproducer::Instance().GetGenerator()) {
-    repro::ProcessGDBRemoteProvider &provider =
-        g->GetOrCreate<repro::ProcessGDBRemoteProvider>();
-    // Set the history stream to the stream owned by the provider.
-    m_gdb_comm.SetHistoryStream(provider.GetHistoryStream());
-    // Make sure to clear the stream again when we're finished.
-    provider.SetCallback([&]() { m_gdb_comm.SetHistoryStream(nullptr); });
+    repro::GDBRemoteProvider &provider =
+        g->GetOrCreate<repro::GDBRemoteProvider>();
+    m_gdb_comm.SetPacketRecorder(provider.GetNewPacketRecorder());
   }
 
   Log *log(ProcessGDBRemoteLog::GetLogIfAllCategoriesSet(GDBR_LOG_ASYNC));

   if (!loader)
     return Status("No loader provided.");
 
-  // Construct replay history path.
-  FileSpec history_file =
-      loader->GetFile<repro::ProcessGDBRemoteProvider::Info>();
-  if (!history_file)
-    return Status("No provider for gdb-remote.");
+  static std::unique_ptr<repro::MultiLoader<repro::GDBRemoteProvider>>
+      multi_loader = repro::MultiLoader<repro::GDBRemoteProvider>::Create(
+          repro::Reproducer::Instance().GetLoader());
 
-  // Enable replay mode.
-  m_replay_mode = true;
+  if (!multi_loader)
+    return Status("No gdb remote provider found.");
+
+  llvm::Optional<std::string> history_file = multi_loader->GetNextFile();
+  if (!history_file)
+    return Status("No gdb remote packet log found.");
 
   // Load replay history.
-  if (auto error = m_gdb_replay_server.LoadReplayHistory(history_file))
+  if (auto error =
+          m_gdb_replay_server.LoadReplayHistory(FileSpec(*history_file)))
     return Status("Unable to load replay history");
 
   // Make a local connection.

                                                           m_gdb_replay_server))
     return Status("Unable to connect to replay server");
 
+  // Enable replay mode.
+  m_replay_mode = true;
+
   // Start server thread.
   m_gdb_replay_server.StartAsyncThread();
 

lldb/source/Utility/GDBRemote.cpp

 #include <stdio.h>
 
 using namespace lldb;
+using namespace lldb_private::repro;
 using namespace lldb_private;
 using namespace llvm;
 

   return bytes_written;
 }
 
-void GDBRemotePacket::Serialize(raw_ostream &strm) const {
-  yaml::Output yout(strm);
-  yout << const_cast<GDBRemotePacket &>(*this);
-  strm.flush();
-}
-
 llvm::StringRef GDBRemotePacket::GetTypeStr() const {
   switch (type) {
   case GDBRemotePacket::ePacketTypeSend:

 
   return {};
 }
+
+void GDBRemoteProvider::Keep() {
+  std::vector<std::string> files;
+  for (auto &recorder : m_packet_recorders) {
+    files.push_back(recorder->GetFilename().GetPath());
+  }
+
+  FileSpec file = GetRoot().CopyByAppendingPathComponent(Info::file);
+  std::error_code ec;
+  llvm::raw_fd_ostream os(file.GetPath(), ec, llvm::sys::fs::OF_Text);
+  if (ec)
+    return;
+  yaml::Output yout(os);
+  yout << files;
+}
+
+void GDBRemoteProvider::Discard() { m_packet_recorders.clear(); }
+
+llvm::Expected<std::unique_ptr<PacketRecorder>>
+PacketRecorder::Create(const FileSpec &filename) {
+  std::error_code ec;
+  auto recorder = std::make_unique<PacketRecorder>(std::move(filename), ec);
+  if (ec)
+    return llvm::errorCodeToError(ec);
+  return std::move(recorder);
+}
+
+PacketRecorder *GDBRemoteProvider::GetNewPacketRecorder() {
+  std::size_t i = m_packet_recorders.size() + 1;
+  std::string filename = (llvm::Twine(Info::name) + llvm::Twine("-") +
+                          llvm::Twine(i) + llvm::Twine(".yaml"))
+                             .str();
+  auto recorder_or_error =
+      PacketRecorder::Create(GetRoot().CopyByAppendingPathComponent(filename));
+  if (!recorder_or_error) {
+    llvm::consumeError(recorder_or_error.takeError());
+    return nullptr;
+  }
+
+  m_packet_recorders.push_back(std::move(*recorder_or_error));
+  return m_packet_recorders.back().get();
+}
+
+void PacketRecorder::Record(const GDBRemotePacket &packet) {
+  if (!m_record)
+    return;
+  yaml::Output yout(m_os);
+  yout << const_cast<GDBRemotePacket &>(packet);
+  m_os.flush();
+}
+
+llvm::raw_ostream *GDBRemoteProvider::GetHistoryStream() {
+  FileSpec history_file = GetRoot().CopyByAppendingPathComponent(Info::file);
+
+  std::error_code EC;
+  m_stream_up = std::make_unique<raw_fd_ostream>(history_file.GetPath(), EC,
+                                                 sys::fs::OpenFlags::OF_Text);
+  return m_stream_up.get();
+}
+
+char GDBRemoteProvider::ID = 0;
+const char *GDBRemoteProvider::Info::file = "gdb-remote.yaml";
+const char *GDBRemoteProvider::Info::name = "gdb-remote";

lldb/source/Utility/Reproducer.cpp

 DataRecorder *CommandProvider::GetNewDataRecorder() {
   std::size_t i = m_data_recorders.size() + 1;
   std::string filename = (llvm::Twine(Info::name) + llvm::Twine("-") +
-                          llvm::Twine(i) + llvm::Twine(".txt"))
+                          llvm::Twine(i) + llvm::Twine(".yaml"))
                              .str();
   auto recorder_or_error =
       DataRecorder::Create(GetRoot().CopyByAppendingPathComponent(filename));

   os << m_cwd << "\n";
 }
 
-llvm::raw_ostream *ProcessGDBRemoteProvider::GetHistoryStream() {
-  FileSpec history_file = GetRoot().CopyByAppendingPathComponent(Info::file);
-
-  std::error_code EC;
-  m_stream_up = std::make_unique<raw_fd_ostream>(history_file.GetPath(), EC,
-                                                 sys::fs::OpenFlags::OF_Text);
-  return m_stream_up.get();
-}
-
-std::unique_ptr<CommandLoader> CommandLoader::Create(Loader *loader) {
-  if (!loader)
-    return {};
-
-  FileSpec file = loader->GetFile<repro::CommandProvider::Info>();
-  if (!file)
-    return {};
-
-  auto error_or_file = llvm::MemoryBuffer::getFile(file.GetPath());
-  if (auto err = error_or_file.getError())
-    return {};
-
-  std::vector<std::string> files;
-  llvm::yaml::Input yin((*error_or_file)->getBuffer());
-  yin >> files;
-
-  if (auto err = yin.error())
-    return {};
-
-  for (auto &file : files) {
-    FileSpec absolute_path =
-        loader->GetRoot().CopyByAppendingPathComponent(file);
-    file = absolute_path.GetPath();
-  }
-
-  return std::make_unique<CommandLoader>(std::move(files));
-}
-
-llvm::Optional<std::string> CommandLoader::GetNextFile() {
-  if (m_index >= m_files.size())
-    return {};
-  return m_files[m_index++];
-}
-
 void ProviderBase::anchor() {}
 char CommandProvider::ID = 0;
 char FileProvider::ID = 0;
-char ProcessGDBRemoteProvider::ID = 0;
 char ProviderBase::ID = 0;
 char VersionProvider::ID = 0;
 char WorkingDirectoryProvider::ID = 0;

 const char *CommandProvider::Info::name = "command-interpreter";
 const char *FileProvider::Info::file = "files.yaml";
 const char *FileProvider::Info::name = "files";
-const char *ProcessGDBRemoteProvider::Info::file = "gdb-remote.yaml";
-const char *ProcessGDBRemoteProvider::Info::name = "gdb-remote";
 const char *VersionProvider::Info::file = "version.txt";
 const char *VersionProvider::Info::name = "version";
 const char *WorkingDirectoryProvider::Info::file = "cwd.txt";