[lldb/Reproducers] Support multiple GDB remotes

[JDevlieghere]

Summary

When running the test suite with always capture on, a handful of tests are failing because they have multiple targets and therefore multiple GDB remote connections. The current reproducer infrastructure is capable of dealing with that.

This patch reworks the GDB remote provider to support multiple GDB remote connections, similar to how the reproducers support shadowing multiple command interpreter inputs. The provider now keeps a list of packet recorders which deal with a single GDB remote connection. During replay we rely on the order of creation to match the number of packets to the GDB remote connection.

[JDevlieghere]

This patch is still missing a test and could use another pass over the PacketRecorder and ProcessGDBRemoteProvider to eliminate code duplication with DataRecorder and the CommandProvider on which they're based respectively. I wasn't able to get around to that today but figured I'd upload the patch anyway to give you a change to look at the approach.

[labath]

Looks pretty straight-forward, but I am still not happy about how more and more of ProcessGDBRemote is infiltrating its way into the Utility module. I think that the reproducer code specific to gdb-remote should live next to the code it's capturing, and Utility/Reproducer.cpp should just provide the general architecture...

lldb/include/lldb/Utility/Reproducer.h

384

is this used anywhere?

lldb/source/Commands/CommandObjectReproducer.cpp

448–450

static? Is it not possible to dump multiple reproducer files in one session ?

454

maybe while ((gdb_file = multi_loader->GetNextFile())) (with an optional != None) ?

[JDevlieghere]

• Address CR feedback.
• Introduce AbstractRecorder to reuse code between the PacketRecorder and the DataRecorder.
• Move GDB related reproducer code into Utility/GDBRemote.h.
• Rename ProcessGDBRemoteProvider to GDBRemoteProvider.

[labath]

This looks better. Ideally I wouldn't even have Utility/GDBRemote.h, but as Process/gdb-remote is not a very good place for it either (it messes with lldb-server dependencies), we can live with it for now.

lldb/include/lldb/Utility/Reproducer.h

23

I guess this should not be needed anymore?

lldb/source/Commands/CommandObjectReproducer.cpp

448–450

So what about this static thing?

[JDevlieghere]

lldb/source/Commands/CommandObjectReproducer.cpp

448–450

This is totally bogus, I don't even remember why or how it ended up there.

[JDevlieghere]

In D71105#1774936, @labath wrote:

This looks better. Ideally I wouldn't even have Utility/GDBRemote.h, but as Process/gdb-remote is not a very good place for it either (it messes with lldb-server dependencies), we can live with it for now.

Yeah, the other problem is the command object which needs to be able to deserialize the packets to dump them. Layering is hard :(

[JDevlieghere]

Add test

lldb/include/lldb/Utility/GDBRemote.h

 //===-- GDBRemote.h ----------------------------------------------*- C++-*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_GDBRemote_h_
 #define liblldb_GDBRemote_h_
 
+#include "lldb/Utility/FileSpec.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/StreamString.h"
 #include "lldb/lldb-enumerations.h"
 #include "lldb/lldb-public.h"
 #include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"
 
 #include <stddef.h>
 #include <stdint.h>
void Clear() {
     packet_idx = 0;
     tid = LLDB_INVALID_THREAD_ID;
   }
 
   struct BinaryData {
     std::string data;
   };
 
-  void Serialize(llvm::raw_ostream &strm) const;
   void Dump(Stream &strm) const;
 
   BinaryData packet;
   Type type;
   uint32_t bytes_transmitted;
   uint32_t packet_idx;
   lldb::tid_t tid;
 
 private:
   llvm::StringRef GetTypeStr() const;
 };
 
+namespace repro {
+class PacketRecorder : public AbstractRecorder {
+public:
+  PacketRecorder(const FileSpec &filename, std::error_code &ec)
+      : AbstractRecorder(filename, ec) {}
+
+  static llvm::Expected<std::unique_ptr<PacketRecorder>>
+  Create(const FileSpec &filename);
+
+  void Record(const GDBRemotePacket &packet);
+};
+
+class GDBRemoteProvider : public repro::Provider<GDBRemoteProvider> {
+public:
+  struct Info {
+    static const char *name;
+    static const char *file;
+  };
+
+  GDBRemoteProvider(const FileSpec &directory) : Provider(directory) {}
+
+  llvm::raw_ostream *GetHistoryStream();
+  PacketRecorder *GetNewPacketRecorder();
+
+  void SetCallback(std::function<void()> callback) {
+    m_callback = std::move(callback);
+  }
+
+  void Keep() override;
+  void Discard() override;
+
+  static char ID;
+
+private:
+  std::function<void()> m_callback;
+  std::unique_ptr<llvm::raw_fd_ostream> m_stream_up;
+  std::vector<std::unique_ptr<PacketRecorder>> m_packet_recorders;
+};
+
+} // namespace repro
 } // namespace lldb_private
 
 LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(lldb_private::GDBRemotePacket)
 LLVM_YAML_IS_DOCUMENT_LIST_VECTOR(std::vector<lldb_private::GDBRemotePacket>)
 
 namespace llvm {
 namespace yaml {
 

lldb/include/lldb/Utility/Reproducer.h

 #include "llvm/Support/FileCollector.h"
 #include "llvm/Support/YAMLTraits.h"
 
 #include <mutex>
 #include <string>
 #include <vector>
 
 namespace lldb_private {
 namespace repro {
 
 class Reproducer;
 
 enum class ReproducerMode {
   Capture,
   Replay,
   Off,
 };
struct Info {
     static const char *name;
     static const char *file;
   };
   void Keep() override;
   std::string m_cwd;
   static char ID;
 };
 
-class DataRecorder {
-public:
-  DataRecorder(const FileSpec &filename, std::error_code &ec)
+class AbstractRecorder {
+protected:
+  AbstractRecorder(const FileSpec &filename, std::error_code &ec)
       : m_filename(filename.GetFilename().GetStringRef()),
         m_os(filename.GetPath(), ec, llvm::sys::fs::OF_Text), m_record(true) {}
 
-  static llvm::Expected<std::unique_ptr<DataRecorder>>
-  Create(const FileSpec &filename);
-
-  template <typename T> void Record(const T &t, bool newline = false) {
-    if (!m_record)
-      return;
-    m_os << t;
-    if (newline)
-      m_os << '\n';
-    m_os.flush();
-  }
-
+public:
   const FileSpec &GetFilename() { return m_filename; }
 
   void Stop() {
     assert(m_record);
     m_record = false;
   }
 
 private:
   FileSpec m_filename;
+
+protected:
   llvm::raw_fd_ostream m_os;
   bool m_record;
 };
 
+class DataRecorder : public AbstractRecorder {
+public:
+  DataRecorder(const FileSpec &filename, std::error_code &ec)
+      : AbstractRecorder(filename, ec) {}
+
+  static llvm::Expected<std::unique_ptr<DataRecorder>>
+  Create(const FileSpec &filename);
+
+  template <typename T> void Record(const T &t, bool newline = false) {
+    if (!m_record)
+      return;
+    m_os << t;
+    if (newline)
+      m_os << '\n';
+    m_os.flush();
+  }
+};
+
 class CommandProvider : public Provider<CommandProvider> {
 public:
   struct Info {
     static const char *name;
     static const char *file;
   };
 
   CommandProvider(const FileSpec &directory) : Provider(directory) {}
 
   DataRecorder *GetNewDataRecorder();
 
   void Keep() override;
   void Discard() override;
 
   static char ID;
 
 private:
   std::vector<std::unique_ptr<DataRecorder>> m_data_recorders;
 };
 
-class ProcessGDBRemoteProvider
-    : public repro::Provider<ProcessGDBRemoteProvider> {
-public:
-  struct Info {
-    static const char *name;
-    static const char *file;
-  };
-
-  ProcessGDBRemoteProvider(const FileSpec &directory) : Provider(directory) {}
-
-  llvm::raw_ostream *GetHistoryStream();
-
-  void SetCallback(std::function<void()> callback) {
-    m_callback = std::move(callback);
-  }
-
-  void Keep() override { m_callback(); }
-  void Discard() override { m_callback(); }
-
-  static char ID;
-
-private:
-  std::function<void()> m_callback;
-  std::unique_ptr<llvm::raw_fd_ostream> m_stream_up;
-};
-
 /// The generator is responsible for the logic needed to generate a
 /// reproducer. For doing so it relies on providers, who serialize data that
 /// is necessary for reproducing  a failure.
 class Generator final {
 
 public:
   Generator(FileSpec root);
   ~Generator();
private:
   static llvm::Optional<Reproducer> &InstanceImpl();
 
   llvm::Optional<Generator> m_generator;
   llvm::Optional<Loader> m_loader;
 
   mutable std::mutex m_mutex;
 };
 
-/// Helper class for replaying commands through the reproducer.
-class CommandLoader {
+template <typename T> class MultiLoader {
 public:
-  CommandLoader(std::vector<std::string> files) : m_files(files) {}
+  MultiLoader(std::vector<std::string> files) : m_files(files) {}
+
+  static std::unique_ptr<MultiLoader> Create(Loader *loader) {
+    if (!loader)
+      return {};
+
+    FileSpec file = loader->GetFile<typename T::Info>();
+    if (!file)
+      return {};
+
+    auto error_or_file = llvm::MemoryBuffer::getFile(file.GetPath());
+    if (auto err = error_or_file.getError())
+      return {};
+
+    std::vector<std::string> files;
+    llvm::yaml::Input yin((*error_or_file)->getBuffer());
+    yin >> files;
 
-  static std::unique_ptr<CommandLoader> Create(Loader *loader);
-  llvm::Optional<std::string> GetNextFile();
+    if (auto err = yin.error())
+      return {};
+
+    for (auto &file : files) {
+      FileSpec absolute_path =
+          loader->GetRoot().CopyByAppendingPathComponent(file);
+      file = absolute_path.GetPath();
+    }
+
+    return std::make_unique<MultiLoader<T>>(std::move(files));
+  }
+
+  llvm::Optional<std::string> GetNextFile() {
+    if (m_index >= m_files.size())
+      return {};
+    return m_files[m_index++];
+  }
 
 private:
   std::vector<std::string> m_files;
   unsigned m_index = 0;
 };
 
 } // namespace repro
 } // namespace lldb_private
 
 #endif // LLDB_UTILITY_REPRODUCER_H

lldb/source/API/SBDebugger.cpp

SBError SBDebugger::SetInputFile(SBFile file) {
   }
 
   repro::DataRecorder *recorder = nullptr;
   if (repro::Generator *g = repro::Reproducer::Instance().GetGenerator())
     recorder = g->GetOrCreate<repro::CommandProvider>().GetNewDataRecorder();
 
   FileSP file_sp = file.m_opaque_sp;
 
-  static std::unique_ptr<repro::CommandLoader> loader =
-      repro::CommandLoader::Create(repro::Reproducer::Instance().GetLoader());
+  static std::unique_ptr<repro::MultiLoader<repro::CommandProvider>> loader =
+      repro::MultiLoader<repro::CommandProvider>::Create(
+          repro::Reproducer::Instance().GetLoader());
   if (loader) {
     llvm::Optional<std::string> nextfile = loader->GetNextFile();
     FILE *fh = nextfile ? FileSystem::Instance().Fopen(nextfile->c_str(), "r")
                         : nullptr;
     // FIXME Jonas Devlieghere: shouldn't this error be propagated out to the
     // reproducer somehow if fh is NULL?
     if (fh) {
       file_sp = std::make_shared<NativeFile>(fh, true);

lldb/source/Commands/CommandObjectReproducer.cpp

case eReproducerProviderWorkingDirectory: {
         SetError(result, cwd.takeError());
         return false;
       }
       result.AppendMessage(*cwd);
       result.SetStatus(eReturnStatusSuccessFinishResult);
       return true;
     }
     case eReproducerProviderCommands: {
-      // Create a new command loader.
-      std::unique_ptr<repro::CommandLoader> command_loader =
-          repro::CommandLoader::Create(loader);
-      if (!command_loader) {
+      std::unique_ptr<repro::MultiLoader<repro::CommandProvider>> multi_loader =
+          repro::MultiLoader<repro::CommandProvider>::Create(loader);
+      if (!multi_loader) {
         SetError(result,
                  make_error<StringError>(llvm::inconvertibleErrorCode(),
                                          "Unable to create command loader."));
         return false;
       }
 
       // Iterate over the command files and dump them.
-      while (true) {
-        llvm::Optional<std::string> command_file =
-            command_loader->GetNextFile();
+      llvm::Optional<std::string> command_file;
+      while ((command_file = multi_loader->GetNextFile())) {
         if (!command_file)
           break;
 
         auto command_buffer = llvm::MemoryBuffer::getFile(*command_file);
         if (auto err = command_buffer.getError()) {
           SetError(result, errorCodeToError(err));
           return false;
         }
         result.AppendMessage((*command_buffer)->getBuffer());
       }
 
       result.SetStatus(eReturnStatusSuccessFinishResult);
       return true;
     }
     case eReproducerProviderGDB: {
-      FileSpec gdb_file = loader->GetFile<ProcessGDBRemoteProvider::Info>();
-      auto error_or_file = MemoryBuffer::getFile(gdb_file.GetPath());
+      std::unique_ptr<repro::MultiLoader<repro::GDBRemoteProvider>>
+          multi_loader =
+              repro::MultiLoader<repro::GDBRemoteProvider>::Create(loader);
+      llvm::Optional<std::string> gdb_file;
+      while ((gdb_file = multi_loader->GetNextFile())) {
+        auto error_or_file = MemoryBuffer::getFile(*gdb_file);
         if (auto err = error_or_file.getError()) {
           SetError(result, errorCodeToError(err));
           return false;
         }
 
         std::vector<GDBRemotePacket> packets;
         yaml::Input yin((*error_or_file)->getBuffer());
         yin >> packets;
 
         if (auto err = yin.error()) {
           SetError(result, errorCodeToError(err));
           return false;
         }
 
         for (GDBRemotePacket &packet : packets) {
           packet.Dump(result.GetOutputStream());
         }
+      }
 
       result.SetStatus(eReturnStatusSuccessFinishResult);
       return true;
     }
     case eReproducerProviderNone:
       result.SetError("No valid provider specified.");
       return false;
     }

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.h

 #include "lldb/Host/HostThread.h"
 #include "lldb/Utility/Args.h"
 #include "lldb/Utility/Listener.h"
 #include "lldb/Utility/Predicate.h"
 #include "lldb/Utility/StringExtractorGDBRemote.h"
 #include "lldb/lldb-public.h"
 
 namespace lldb_private {
+namespace repro {
+class PacketRecorder;
+}
 namespace process_gdb_remote {
 
 enum GDBStoppointType {
   eStoppointInvalid = -1,
   eBreakpointSoftware = 0,
   eBreakpointHardware,
   eWatchpointWrite,
   eWatchpointRead,
Status StartDebugserverProcess(
       const char *url,
       Platform *platform, // If non nullptr, then check with the platform for
                           // the GDB server binary if it can't be located
       ProcessLaunchInfo &launch_info, uint16_t *port, const Args *inferior_args,
       int pass_comm_fd); // Communication file descriptor to pass during
                          // fork/exec to avoid having to connect/accept
 
   void DumpHistory(Stream &strm);
-  void SetHistoryStream(llvm::raw_ostream *strm);
+
+  void SetPacketRecorder(repro::PacketRecorder *recorder);
 
   static llvm::Error ConnectLocally(GDBRemoteCommunication &client,
                                     GDBRemoteCommunication &server);
 
 protected:
   std::chrono::seconds m_packet_timeout;
   uint32_t m_echo_number;
   LazyBool m_supports_qEcho;

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp

 #include "lldb/Host/ThreadLauncher.h"
 #include "lldb/Host/common/TCPSocket.h"
 #include "lldb/Host/posix/ConnectionFileDescriptorPosix.h"
 #include "lldb/Target/Platform.h"
 #include "lldb/Utility/Event.h"
 #include "lldb/Utility/FileSpec.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/RegularExpression.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/StreamString.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/ScopedPrinter.h"
 
 #include "ProcessGDBRemoteLog.h"
 
 #if defined(__APPLE__)
 #define DEBUGSERVER_BASENAME "debugserver"
LLDB_LOGF(log, "GDBRemoteCommunication::%s() failed: %s", __FUNCTION__,
               error.AsCString());
   }
 
   return error;
 }
 
 void GDBRemoteCommunication::DumpHistory(Stream &strm) { m_history.Dump(strm); }
 
-void GDBRemoteCommunication::SetHistoryStream(llvm::raw_ostream *strm) {
-  m_history.SetStream(strm);
+void GDBRemoteCommunication::SetPacketRecorder(
+    repro::PacketRecorder *recorder) {
+  m_history.SetRecorder(recorder);
 }
 
 llvm::Error
 GDBRemoteCommunication::ConnectLocally(GDBRemoteCommunication &client,
                                        GDBRemoteCommunication &server) {
   const bool child_processes_inherit = false;
   const int backlog = 5;
   TCPSocket listen_socket(true, child_processes_inherit);

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationHistory.h

 //===-- GDBRemoteCommunicationHistory.h--------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_GDBRemoteCommunicationHistory_h_
 #define liblldb_GDBRemoteCommunicationHistory_h_
 
 #include <string>
 #include <vector>
 
 #include "lldb/Utility/GDBRemote.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/lldb-public.h"
 #include "llvm/Support/YAMLTraits.h"
 #include "llvm/Support/raw_ostream.h"
 
 namespace lldb_private {
+namespace repro {
+class PacketRecorder;
+}
 namespace process_gdb_remote {
 
 /// The history keeps a circular buffer of GDB remote packets. The history is
 /// used for logging and replaying GDB remote packets.
 class GDBRemoteCommunicationHistory {
 public:
   friend llvm::yaml::MappingTraits<GDBRemoteCommunicationHistory>;
 
   GDBRemoteCommunicationHistory(uint32_t size = 0);
 
   ~GDBRemoteCommunicationHistory();
 
   // For single char packets for ack, nack and /x03
   void AddPacket(char packet_char, GDBRemotePacket::Type type,
                  uint32_t bytes_transmitted);
 
   void AddPacket(const std::string &src, uint32_t src_len,
                  GDBRemotePacket::Type type, uint32_t bytes_transmitted);
 
   void Dump(Stream &strm) const;
   void Dump(Log *log) const;
   bool DidDumpToLog() const { return m_dumped_to_log; }
 
-  void SetStream(llvm::raw_ostream *strm) { m_stream = strm; }
+  void SetRecorder(repro::PacketRecorder *recorder) { m_recorder = recorder; }
 
 private:
   uint32_t GetFirstSavedPacketIndex() const {
     if (m_total_packet_count < m_packets.size())
       return 0;
     else
       return m_curr_idx + 1;
   }
private:
   uint32_t NormalizeIndex(uint32_t i) const {
     return m_packets.empty() ? 0 : i % m_packets.size();
   }
 
   std::vector<GDBRemotePacket> m_packets;
   uint32_t m_curr_idx;
   uint32_t m_total_packet_count;
   mutable bool m_dumped_to_log;
-  llvm::raw_ostream *m_stream = nullptr;
+  repro::PacketRecorder *m_recorder = nullptr;
 };
 
 } // namespace process_gdb_remote
 } // namespace lldb_private
 
 #endif // liblldb_GDBRemoteCommunicationHistory_h_

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationHistory.cpp

if (size == 0)
     return;
 
   const uint32_t idx = GetNextIndex();
   m_packets[idx].packet.data.assign(1, packet_char);
   m_packets[idx].type = type;
   m_packets[idx].bytes_transmitted = bytes_transmitted;
   m_packets[idx].packet_idx = m_total_packet_count;
   m_packets[idx].tid = llvm::get_threadid();
-  if (m_stream)
-    m_packets[idx].Serialize(*m_stream);
+  if (m_recorder)
+    m_recorder->Record(m_packets[idx]);
 }
 
 void GDBRemoteCommunicationHistory::AddPacket(const std::string &src,
                                               uint32_t src_len,
                                               GDBRemotePacket::Type type,
                                               uint32_t bytes_transmitted) {
   const size_t size = m_packets.size();
   if (size == 0)
     return;
 
   const uint32_t idx = GetNextIndex();
   m_packets[idx].packet.data.assign(src, 0, src_len);
   m_packets[idx].type = type;
   m_packets[idx].bytes_transmitted = bytes_transmitted;
   m_packets[idx].packet_idx = m_total_packet_count;
   m_packets[idx].tid = llvm::get_threadid();
-  if (m_stream)
-    m_packets[idx].Serialize(*m_stream);
+  if (m_recorder)
+    m_recorder->Record(m_packets[idx]);
 }
 
 void GDBRemoteCommunicationHistory::Dump(Stream &strm) const {
   const uint32_t size = GetNumPacketsInHistory();
   const uint32_t first_idx = GetFirstSavedPacketIndex();
   const uint32_t stop_idx = m_curr_idx + size;
   for (uint32_t i = first_idx; i < stop_idx; ++i) {
     const uint32_t idx = NormalizeIndex(i);

lldb/source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp

ProcessGDBRemote::ProcessGDBRemote(lldb::TargetSP target_sp,
   m_async_broadcaster.SetEventName(eBroadcastBitAsyncThreadShouldExit,
                                    "async thread should exit");
   m_async_broadcaster.SetEventName(eBroadcastBitAsyncContinue,
                                    "async thread continue");
   m_async_broadcaster.SetEventName(eBroadcastBitAsyncThreadDidExit,
                                    "async thread did exit");
 
   if (repro::Generator *g = repro::Reproducer::Instance().GetGenerator()) {
-    repro::ProcessGDBRemoteProvider &provider =
-        g->GetOrCreate<repro::ProcessGDBRemoteProvider>();
-    // Set the history stream to the stream owned by the provider.
-    m_gdb_comm.SetHistoryStream(provider.GetHistoryStream());
-    // Make sure to clear the stream again when we're finished.
-    provider.SetCallback([&]() { m_gdb_comm.SetHistoryStream(nullptr); });
+    repro::GDBRemoteProvider &provider =
+        g->GetOrCreate<repro::GDBRemoteProvider>();
+    m_gdb_comm.SetPacketRecorder(provider.GetNewPacketRecorder());
   }
 
   Log *log(ProcessGDBRemoteLog::GetLogIfAllCategoriesSet(GDBR_LOG_ASYNC));
 
   const uint32_t async_event_mask =
       eBroadcastBitAsyncContinue | eBroadcastBitAsyncThreadShouldExit;
 
   if (m_async_listener_sp->StartListeningForEvents(
if (!m_gdb_comm.SendAsyncSignal(signo))
     error.SetErrorStringWithFormat("failed to send signal %i", signo);
   return error;
 }
 
 Status ProcessGDBRemote::ConnectToReplayServer(repro::Loader *loader) {
   if (!loader)
     return Status("No loader provided.");
 
-  // Construct replay history path.
-  FileSpec history_file =
-      loader->GetFile<repro::ProcessGDBRemoteProvider::Info>();
-  if (!history_file)
-    return Status("No provider for gdb-remote.");
+  static std::unique_ptr<repro::MultiLoader<repro::GDBRemoteProvider>>
+      multi_loader = repro::MultiLoader<repro::GDBRemoteProvider>::Create(
+          repro::Reproducer::Instance().GetLoader());
 
-  // Enable replay mode.
-  m_replay_mode = true;
+  if (!multi_loader)
+    return Status("No gdb remote provider found.");
+
+  llvm::Optional<std::string> history_file = multi_loader->GetNextFile();
+  if (!history_file)
+    return Status("No gdb remote packet log found.");
 
   // Load replay history.
-  if (auto error = m_gdb_replay_server.LoadReplayHistory(history_file))
+  if (auto error =
+          m_gdb_replay_server.LoadReplayHistory(FileSpec(*history_file)))
     return Status("Unable to load replay history");
 
   // Make a local connection.
   if (auto error = GDBRemoteCommunication::ConnectLocally(m_gdb_comm,
                                                           m_gdb_replay_server))
     return Status("Unable to connect to replay server");
 
+  // Enable replay mode.
+  m_replay_mode = true;
+
   // Start server thread.
   m_gdb_replay_server.StartAsyncThread();
 
   // Start client thread.
   StartAsyncThread();
 
   // Do the usual setup.
   return ConnectToDebugserver("");

lldb/source/Utility/GDBRemote.cpp

 //===-- GDBRemote.cpp -----------------------------------------------------===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #include "lldb/Utility/GDBRemote.h"
 
 #include "lldb/Utility/Flags.h"
 #include "lldb/Utility/Stream.h"
 
 #include <stdio.h>
 
 using namespace lldb;
+using namespace lldb_private::repro;
 using namespace lldb_private;
 using namespace llvm;
 
 StreamGDBRemote::StreamGDBRemote() : StreamString() {}
 
 StreamGDBRemote::StreamGDBRemote(uint32_t flags, uint32_t addr_size,
                                  ByteOrder byte_order)
     : StreamString(flags, addr_size, byte_order) {}
while (src_len) {
     }
     bytes_written += PutChar(byte);
   };
   if (binary_is_set)
     m_flags.Set(eBinary);
   return bytes_written;
 }
 
-void GDBRemotePacket::Serialize(raw_ostream &strm) const {
-  yaml::Output yout(strm);
-  yout << const_cast<GDBRemotePacket &>(*this);
-  strm.flush();
-}
-
 llvm::StringRef GDBRemotePacket::GetTypeStr() const {
   switch (type) {
   case GDBRemotePacket::ePacketTypeSend:
     return "send";
   case GDBRemotePacket::ePacketTypeRecv:
     return "read";
   case GDBRemotePacket::ePacketTypeInvalid:
     return "invalid";

 StringRef
 yaml::MappingTraits<GDBRemotePacket>::validate(IO &io,
                                                GDBRemotePacket &Packet) {
   if (Packet.bytes_transmitted != Packet.packet.data.size())
     return "BinaryData size doesn't match bytes transmitted";
 
   return {};
 }
+
+void GDBRemoteProvider::Keep() {
+  std::vector<std::string> files;
+  for (auto &recorder : m_packet_recorders) {
+    files.push_back(recorder->GetFilename().GetPath());
+  }
+
+  FileSpec file = GetRoot().CopyByAppendingPathComponent(Info::file);
+  std::error_code ec;
+  llvm::raw_fd_ostream os(file.GetPath(), ec, llvm::sys::fs::OF_Text);
+  if (ec)
+    return;
+  yaml::Output yout(os);
+  yout << files;
+}
+
+void GDBRemoteProvider::Discard() { m_packet_recorders.clear(); }
+
+llvm::Expected<std::unique_ptr<PacketRecorder>>
+PacketRecorder::Create(const FileSpec &filename) {
+  std::error_code ec;
+  auto recorder = std::make_unique<PacketRecorder>(std::move(filename), ec);
+  if (ec)
+    return llvm::errorCodeToError(ec);
+  return std::move(recorder);
+}
+
+PacketRecorder *GDBRemoteProvider::GetNewPacketRecorder() {
+  std::size_t i = m_packet_recorders.size() + 1;
+  std::string filename = (llvm::Twine(Info::name) + llvm::Twine("-") +
+                          llvm::Twine(i) + llvm::Twine(".yaml"))
+                             .str();
+  auto recorder_or_error =
+      PacketRecorder::Create(GetRoot().CopyByAppendingPathComponent(filename));
+  if (!recorder_or_error) {
+    llvm::consumeError(recorder_or_error.takeError());
+    return nullptr;
+  }
+
+  m_packet_recorders.push_back(std::move(*recorder_or_error));
+  return m_packet_recorders.back().get();
+}
+
+void PacketRecorder::Record(const GDBRemotePacket &packet) {
+  if (!m_record)
+    return;
+  yaml::Output yout(m_os);
+  yout << const_cast<GDBRemotePacket &>(packet);
+  m_os.flush();
+}
+
+llvm::raw_ostream *GDBRemoteProvider::GetHistoryStream() {
+  FileSpec history_file = GetRoot().CopyByAppendingPathComponent(Info::file);
+
+  std::error_code EC;
+  m_stream_up = std::make_unique<raw_fd_ostream>(history_file.GetPath(), EC,
+                                                 sys::fs::OpenFlags::OF_Text);
+  return m_stream_up.get();
+}
+
+char GDBRemoteProvider::ID = 0;
+const char *GDBRemoteProvider::Info::file = "gdb-remote.yaml";
+const char *GDBRemoteProvider::Info::name = "gdb-remote";

lldb/source/Utility/Reproducer.cpp

DataRecorder::Create(const FileSpec &filename) {
   if (ec)
     return llvm::errorCodeToError(ec);
   return std::move(recorder);
 }
 
 DataRecorder *CommandProvider::GetNewDataRecorder() {
   std::size_t i = m_data_recorders.size() + 1;
   std::string filename = (llvm::Twine(Info::name) + llvm::Twine("-") +
-                          llvm::Twine(i) + llvm::Twine(".txt"))
+                          llvm::Twine(i) + llvm::Twine(".yaml"))
                              .str();
   auto recorder_or_error =
       DataRecorder::Create(GetRoot().CopyByAppendingPathComponent(filename));
   if (!recorder_or_error) {
     llvm::consumeError(recorder_or_error.takeError());
     return nullptr;
   }
 
void WorkingDirectoryProvider::Keep() {
   FileSpec file = GetRoot().CopyByAppendingPathComponent(Info::file);
   std::error_code ec;
   llvm::raw_fd_ostream os(file.GetPath(), ec, llvm::sys::fs::OF_Text);
   if (ec)
     return;
   os << m_cwd << "\n";
 }
 
-llvm::raw_ostream *ProcessGDBRemoteProvider::GetHistoryStream() {
-  FileSpec history_file = GetRoot().CopyByAppendingPathComponent(Info::file);
-
-  std::error_code EC;
-  m_stream_up = std::make_unique<raw_fd_ostream>(history_file.GetPath(), EC,
-                                                 sys::fs::OpenFlags::OF_Text);
-  return m_stream_up.get();
-}
-
-std::unique_ptr<CommandLoader> CommandLoader::Create(Loader *loader) {
-  if (!loader)
-    return {};
-
-  FileSpec file = loader->GetFile<repro::CommandProvider::Info>();
-  if (!file)
-    return {};
-
-  auto error_or_file = llvm::MemoryBuffer::getFile(file.GetPath());
-  if (auto err = error_or_file.getError())
-    return {};
-
-  std::vector<std::string> files;
-  llvm::yaml::Input yin((*error_or_file)->getBuffer());
-  yin >> files;
-
-  if (auto err = yin.error())
-    return {};
-
-  for (auto &file : files) {
-    FileSpec absolute_path =
-        loader->GetRoot().CopyByAppendingPathComponent(file);
-    file = absolute_path.GetPath();
-  }
-
-  return std::make_unique<CommandLoader>(std::move(files));
-}
-
-llvm::Optional<std::string> CommandLoader::GetNextFile() {
-  if (m_index >= m_files.size())
-    return {};
-  return m_files[m_index++];
-}
-
 void ProviderBase::anchor() {}
 char CommandProvider::ID = 0;
 char FileProvider::ID = 0;
-char ProcessGDBRemoteProvider::ID = 0;
 char ProviderBase::ID = 0;
 char VersionProvider::ID = 0;
 char WorkingDirectoryProvider::ID = 0;
 const char *CommandProvider::Info::file = "command-interpreter.yaml";
 const char *CommandProvider::Info::name = "command-interpreter";
 const char *FileProvider::Info::file = "files.yaml";
 const char *FileProvider::Info::name = "files";
-const char *ProcessGDBRemoteProvider::Info::file = "gdb-remote.yaml";
-const char *ProcessGDBRemoteProvider::Info::name = "gdb-remote";
 const char *VersionProvider::Info::file = "version.txt";
 const char *VersionProvider::Info::name = "version";
 const char *WorkingDirectoryProvider::Info::file = "cwd.txt";
 const char *WorkingDirectoryProvider::Info::name = "cwd";

lldb/test/Shell/Reproducer/Inputs/MultipleTargetsCapture.in

+target select 0
+breakpoint set -f simple.c -l 12
+run
+target select 1
+breakpoint set -f simple.c -l 16
+run
+target select 0
+cont
+target select 1
+cont
+reproducer status
+reproducer generate

lldb/test/Shell/Reproducer/TestMultipleTargets.test

+# UNSUPPORTED: system-windows, system-freebsd
+
+# This tests the replaying with multiple targets.
+
+# RUN: %clang_host %S/Inputs/simple.c -g -o %t.out
+
+# RUN: rm -rf %t.repro
+# RUN: %lldb -x -b --capture --capture-path %t.repro -o 'target create %t.out' -o 'target create %t.out' -s %S/Inputs/MultipleTargetsCapture.in  | FileCheck %s --check-prefix CHECK --check-prefix CAPTURE
+# RUN: env FOO=BAR %lldb --replay %t.repro | FileCheck %s --check-prefix CHECK --check-prefix REPLAY
+
+# CHECK: Process [[TARGET0:[0-9]+]] stopped
+# CHECK: stop reason = breakpoint 1.1
+# CHECK: simple.c:12:5
+# CHECK: Process [[TARGET1:[0-9]+]] stopped
+# CHECK: stop reason = breakpoint 1.1
+# CHECK: simple.c:16:5
+# CHECK: Process [[TARGET0]] resuming
+# CHECK: Process [[TARGET0]] exited
+# CHECK: Process [[TARGET1]] resuming
+# CHECK: Process [[TARGET1]] exited
+
+# CAPTURE: Reproducer is in capture mode.
+# CAPTURE: Reproducer written