[Reproducers] Add command provider

[JDevlieghere]

Summary

Add a provider to the command interpreter. Essentially this writes all the commands to a file which is used during replay as input to the command interpreter.

[aprantl]

lldb/include/lldb/Interpreter/CommandInterpreter.h

31 ↗

(On Diff #188014)

Doxygen comments?

615 ↗

(On Diff #188014)

///

lldb/source/Interpreter/CommandInterpreter.cpp

121 ↗

(On Diff #188014)

Any toplevel Doxygen comments that would be useful to add here?

132 ↗

(On Diff #188014)

At least from the outside it's not quite obvious what this function does. Could you perhaps add a high-level comment?

[JDevlieghere]

Add comments

[JDevlieghere]

• Add testcase.
• Don't log sourced commands twice.

For some reason I thought that we no longer needed to differentiate between commands being sourced from a file. I remember an earlier discussing with Pavel on this topic where we considered doing this at a lower level, but there we face the same issue, without an easy way to differentiate.

[labath]

I am sorry that I won't have much time to review this in the next couple of weeks, but I don't think this is a good direction here. I don't see how this will interact with the SB API recorder, specifically with things like SBCommandInterpreter::HandleCommand, and ::HandleCommandsFromFile. The thing I would expect to see is that SB recorder captures the input of those commands (for a somewhat broad interpretation of "capture") during recording, and then substitute this during replay. That way the CommandInterpreter class would not need (almost?) any modifications.

With this approach (shoving all commands into a single stream in the CommandInterpreter) it becomes impossible to replay the API calls above. If you want to proceed with this, then go ahead (it's your feature), but I believe you'll run into some problems down the line.

[JDevlieghere]

In D58564#1410213, @labath wrote:

I am sorry that I won't have much time to review this in the next couple of weeks, but I don't think this is a good direction here. I don't see how this will interact with the SB API recorder, specifically with things like SBCommandInterpreter::HandleCommand, and ::HandleCommandsFromFile. The thing I would expect to see is that SB recorder captures the input of those commands (for a somewhat broad interpretation of "capture") during recording, and then substitute this during replay. That way the CommandInterpreter class would not need (almost?) any modifications.

It’s been a while but wasn’t this exactly what you proposed in the other differential? How would you capture commands that are entered interactively (through RunCommqndInterpreter)?

Anyway, I don’t believe this is a concern. The provider here only capture what’s entered interactively, hence the flag. Replaying the API call should work exactly as expected. I’ll double check later today.

With this approach (shoving all commands into a single stream in the CommandInterpreter) it becomes impossible to replay the API calls above. If you want to proceed with this, then go ahead (it's your feature), but I believe you'll run into some problems down the line.

[labath]

In D58564#1410729, @JDevlieghere wrote:

In D58564#1410213, @labath wrote:

I am sorry that I won't have much time to review this in the next couple of weeks, but I don't think this is a good direction here. I don't see how this will interact with the SB API recorder, specifically with things like SBCommandInterpreter::HandleCommand, and ::HandleCommandsFromFile. The thing I would expect to see is that SB recorder captures the input of those commands (for a somewhat broad interpretation of "capture") during recording, and then substitute this during replay. That way the CommandInterpreter class would not need (almost?) any modifications.

It’s been a while but wasn’t this exactly what you proposed in the other differential? How would you capture commands that are entered interactively (through RunCommqndInterpreter)?

Anyway, I don’t believe this is a concern. The provider here only capture what’s entered interactively, hence the flag. Replaying the API call should work exactly as expected. I’ll double check later today.

I have to admit I haven't looked at this in detail, but the thing I'm missing here is the connection between commands and API calls. If we take RunCommandInterpreter, for instance, you can see that the lldb driver invokes this function three times. How do you ensure the "right" commands get replayed as a part of the API call?

If you look at the driver more closely, you'll see that each call to RunCommandInterpreter is preceeded by a call to SetInputFileHandle. I don't have this idea fully baked, but the way I'd try to approach this is to have each SetInputFileHandle create a new buffer where the commands will be stored in. Then as the commands are being processed (in RunCommandInterpreter), they would be added into this buffer. Then, when replaying you would know that you only should replay the commands from the given buffer.

I'd also probably try to capture these commands at a slightly lower level, because I am hoping that this will allow us to get rid of the add_to_reproducer flag. Ideally, this should fall out naturally due to the different source the commands are coming from -- the commands executed through the HandleCommand API would be captured at the SB boundary, and the "interactive" commands would be captured by whoever invokes the command interpreter in interactive mode.

(I have no idea how easy it is to achieve this, but that's how I'd approach this.)

[JDevlieghere]

Pavel made a good point that with the previous implementation, the first call to RunCommandInterpreter would replay every recorded commands. This is indeed incorrect, because it's possible and likely that the state of the debugger has changed between different runs of the commands interpreter.

Now every call to RunCommandInterpreter gets its own buffer. During replay, we will change the input file handler before invocation of "RunCommandInterpreter". This works because this function is only called through the SB layer.

I'm not convinced doing the recorded at a lower level has any benefits. I investigated this route before and the IOHandler seems basically the same things as the command interpreter. We would still need to make the distinction between things that should and shouldn't be recorded (e.g. sourcing a file vs every command in the file). This would be a lot harder to do there, because we have less information.

[labath]

In D58564#1412674, @JDevlieghere wrote:

Pavel made a good point that with the previous implementation, the first call to RunCommandInterpreter would replay every recorded commands. This is indeed incorrect, because it's possible and likely that the state of the debugger has changed between different runs of the commands interpreter.

Now every call to RunCommandInterpreter gets its own buffer. During replay, we will change the input file handler before invocation of "RunCommandInterpreter". This works because this function is only called through the SB layer.

I'm not convinced doing the recorded at a lower level has any benefits. I investigated this route before and the IOHandler seems basically the same things as the command interpreter. We would still need to make the distinction between things that should and shouldn't be recorded (e.g. sourcing a file vs every command in the file). This would be a lot harder to do there, because we have less information.

In my imagination, this "information" would come down from SBDebugger::SetInputFileHandle, together with the FILE* we actually read the commands from. So, a really crude prototype could be something like:

SBDebugger::SetInputFileHandle(FILE *in) {
if (recording) m_debugger->SetInputFileHandle(in, /*new argument!!!*/ new FileShadowRecorder(...));
else if (replaying) m_debugger->SetInputFileHandle(GetRecordedFile(...), nullptr);
}

The FILE* would trickle down into where you read the commands (this could be the IOHandler, or it could be the CommandInterpreter object), where you would do something like:

auto stuff = read_stuff(in);
if (shadow_recorder)
  shadow_recorder->record_stuff(stuff);

Now when you're sourcing an external file, you just pass in a nullptr for the recorder when you're setting the input handle for the command interpreter.

So, in essence the shadow recorder would kind of serve the same purpose as your add_to_reproducer flag, but IMO this would be better because it would come straight from the source (SetInputFileHandle) and you wouldn't need to rely on comments like "This works because this function is only called through the SB layer". Another benefit would be that this would work out-of-the-box in case you have multiple SBDebugger objects around, each with it's own command interpreter (right now this wouldn't work because the StartNewBuffer thingies would step on each others toes). I don't know when or if you plan to support that, but right now that tells me that this is a better design.

lldb/source/Interpreter/CommandInterpreter.cpp

144 ↗

(On Diff #188617)

This is incorrect usage of the Twine class. The temporary Twine objects will be destroyed before you get a chance to stringify them.

[JDevlieghere]

Thanks Pavel. I've updated the patch with your suggestion. I agree it's a lot better :-)

I implemented the logic in the Debugger rather than the SBDebugger because I think the latter should be a thin wrapper, but let me know if you had a particular reason for this.

[labath]

In D58564#1414410, @JDevlieghere wrote:

Thanks Pavel. I've updated the patch with your suggestion. I agree it's a lot better :-)

I implemented the logic in the Debugger rather than the SBDebugger because I think the latter should be a thin wrapper, but let me know if you had a particular reason for this.

Thanks Jonas. This looks even simpler than I anticipated. :)

While I agree that SB layer should be as thin as possible, I would say this deserves an exception. My reasoning behind that is that this command capture mechanism is kind of an extended arm of the SB recorder. Like, if we had an oracle, and were able to "instantly" capture the contents of the FILE* at the SB layer, we would just do that, and avoid this shadowing dance altogether. Unfortunately, we cannot see into the future, so we have to have this recorder "shadow" follow the input FILE* whereever it goes and capture the input as it is being read. The cleanest way to me seems to be to have that shadow start tracking the value as soon as it crosses the SB boundary.

Or, looking at it from a different angle, if somebody other than the SBDebugger calls Debugger::SetInputFileHandle, then we probably don't want to capture that input because it did not come from the outside world. (I have no idea why anybody would want to do that, but it does not seem completely out of the question.). This FILE* then most likely came from the filesystem, in which case it would be captured by the FileSystem recorder. Or if it did come through the SB API, but through a different function, then it should have been shadowed as soon it entered that function.

Apart from this I have some inline comments about error handling, but overall, I am very happy with how this is turning out.

lldb/include/lldb/Utility/Reproducer.h

120 ↗

(On Diff #188813)

Remove operator bool and the embedded error code. Instead have a static factory function which checks the error and only returns a DataRecorder if the file was created correctly. I'm thinking of something like

static std::unique_ptr<DataRecorder> /*or Expected<...> ?*/ Create(FileSpec f) { 
  std::error_code ec;
  auto rec = make_unique<DataRecorder>(f, ec);
  if (!ec)
    return rec;
  return nullptr; /*or the error code*/
}

lldb/source/Core/Debugger.cpp

933 ↗

(On Diff #188813)

Why not store this as a field? This way you still can have problems if two debugger objects are active concurrently.

lldb/source/Utility/Reproducer.cpp

228–229 ↗

(On Diff #188813)

So what happens if creating the file fails here? We abort when the assert in Debugger::GetInputRecorder fails? If we're going to crash, it's probably better to do it here, as that's closer to where the actual failure happened (alternatively, we could log a message and exit slightly more gracefully; or log a message and continue without recording).

[JDevlieghere]

• Moved the logic into SBDebugger.
• Created DataRecorder factory.
• Stored DataRecorder as a field in Debugger.

[labath]

I have a couple of more comments, including some things I missed on the previous pass, but I don't want to hold this up any more. Feel free to commit after taking the last batch into consideration.

lldb/include/lldb/Utility/Reproducer.h

116 ↗

(On Diff #188932)

The error_code will need to be a reference argument for this to have any effect.

lldb/source/API/SBDebugger.cpp

63 ↗

(On Diff #188932)

Use a factory function here too?

74 ↗

(On Diff #188932)

It might be nice to log these errors even if we don't plan to do anything about them.

lldb/source/Core/Debugger.cpp

887 ↗

(On Diff #188932)

Sticking with the "shadow" idea, I think it would be better if the input recorder is set as an extra argument to this function (instead of as a separate call), because they should always be changed together. Setting one without the other is almost certainly a mistake.

[JDevlieghere]

Addressed in commit. Thank you Pavel!

lldb/trunk/include/lldb/Core/Debugger.h

 class Stream;
 }
 namespace lldb_private {
 class SymbolContext;
 }
 namespace lldb_private {
 class Target;
 }
+namespace lldb_private {
+namespace repro {
+class DataRecorder;
+}
+} // namespace lldb_private
 namespace llvm {
 class raw_ostream;
 }
 
 namespace lldb_private {
 
 //----------------------------------------------------------------------
 /// @class Debugger Debugger.h "lldb/Core/Debugger.h"
public:
   void SetAsyncExecution(bool async);
 
   lldb::StreamFileSP GetInputFile() { return m_input_file_sp; }
 
   lldb::StreamFileSP GetOutputFile() { return m_output_file_sp; }
 
   lldb::StreamFileSP GetErrorFile() { return m_error_file_sp; }
 
-  void SetInputFileHandle(FILE *fh, bool tranfer_ownership);
+  repro::DataRecorder *GetInputRecorder();
+
+  void SetInputFileHandle(FILE *fh, bool tranfer_ownership,
+                          repro::DataRecorder *recorder = nullptr);
 
   void SetOutputFileHandle(FILE *fh, bool tranfer_ownership);
 
   void SetErrorFileHandle(FILE *fh, bool tranfer_ownership);
 
   void SaveInputTerminalState();
 
   void RestoreInputTerminalState();
protected:
   }
 
   void InstanceInitialize();
 
   lldb::StreamFileSP m_input_file_sp;
   lldb::StreamFileSP m_output_file_sp;
   lldb::StreamFileSP m_error_file_sp;
 
+  /// Used for shadowing the input file when capturing a reproducer.
+  repro::DataRecorder *m_input_recorder;
+
   lldb::BroadcasterManagerSP m_broadcaster_manager_sp; // The debugger acts as a
                                                        // broadcaster manager of
                                                        // last resort.
   // It needs to get constructed before the target_list or any other member
   // that might want to broadcast through the debugger.
 
   TerminalState m_terminal_state;
   TargetList m_target_list;

lldb/trunk/include/lldb/Core/IOHandler.h

 //===-- IOHandler.h ---------------------------------------------*- C++ -*-===//
 //
 // Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 // See https://llvm.org/LICENSE.txt for license information.
 // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 //
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_IOHandler_h_
 #define liblldb_IOHandler_h_
 
 #include "lldb/Core/ValueObjectList.h"
 #include "lldb/Utility/ConstString.h"
 #include "lldb/Utility/Flags.h"
 #include "lldb/Utility/Predicate.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/Stream.h"
 #include "lldb/Utility/StringList.h"
 #include "lldb/lldb-defines.h"
 #include "lldb/lldb-forward.h"
 #include "llvm/ADT/StringRef.h"
 
 #include <memory>
 #include <mutex>
enum class Type {
     Other
   };
 
   IOHandler(Debugger &debugger, IOHandler::Type type);
 
   IOHandler(Debugger &debugger, IOHandler::Type type,
             const lldb::StreamFileSP &input_sp,
             const lldb::StreamFileSP &output_sp,
-            const lldb::StreamFileSP &error_sp, uint32_t flags);
+            const lldb::StreamFileSP &error_sp, uint32_t flags,
+            repro::DataRecorder *data_recorder);
 
   virtual ~IOHandler();
 
   // Each IOHandler gets to run until it is done. It should read data from the
   // "in" and place output into "out" and "err and return when done.
   virtual void Run() = 0;
 
   // Called when an input reader should relinquish its control so another can
virtual void PrintAsync(Stream *stream, const char *s, size_t len) {
     stream->Flush();
   }
 
 protected:
   Debugger &m_debugger;
   lldb::StreamFileSP m_input_sp;
   lldb::StreamFileSP m_output_sp;
   lldb::StreamFileSP m_error_sp;
+  repro::DataRecorder *m_data_recorder;
   Predicate<bool> m_popped;
   Flags m_flags;
   Type m_type;
   void *m_user_data;
   bool m_done;
   bool m_active;
 
 private:

 public:
   IOHandlerEditline(Debugger &debugger, IOHandler::Type type,
                     const char *editline_name, // Used for saving history files
                     llvm::StringRef prompt, llvm::StringRef continuation_prompt,
                     bool multi_line, bool color_prompts,
                     uint32_t line_number_start, // If non-zero show line numbers
                                                 // starting at
                                                 // 'line_number_start'
-                    IOHandlerDelegate &delegate);
+                    IOHandlerDelegate &delegate,
+                    repro::DataRecorder *data_recorder);
 
   IOHandlerEditline(Debugger &debugger, IOHandler::Type type,
                     const lldb::StreamFileSP &input_sp,
                     const lldb::StreamFileSP &output_sp,
                     const lldb::StreamFileSP &error_sp, uint32_t flags,
                     const char *editline_name, // Used for saving history files
                     llvm::StringRef prompt, llvm::StringRef continuation_prompt,
                     bool multi_line, bool color_prompts,
                     uint32_t line_number_start, // If non-zero show line numbers
                                                 // starting at
                                                 // 'line_number_start'
-                    IOHandlerDelegate &delegate);
+                    IOHandlerDelegate &delegate,
+                    repro::DataRecorder *data_recorder);
 
   IOHandlerEditline(Debugger &, IOHandler::Type, const char *, const char *,
                     const char *, bool, bool, uint32_t,
                     IOHandlerDelegate &) = delete;
 
   IOHandlerEditline(Debugger &, IOHandler::Type, const lldb::StreamFileSP &,
                     const lldb::StreamFileSP &, const lldb::StreamFileSP &,
                     uint32_t, const char *, const char *, const char *, bool,

lldb/trunk/include/lldb/Utility/Reproducer.h

public:
   }
 
   static char ID;
 
 private:
   FileCollector m_collector;
 };
 
+class DataRecorder {
+public:
+  DataRecorder(FileSpec filename, std::error_code &ec)
+      : m_filename(std::move(filename)),
+        m_os(m_filename.GetPath(), ec, llvm::sys::fs::F_Text) {}
+
+  static llvm::Expected<std::unique_ptr<DataRecorder>>
+  Create(FileSpec filename);
+
+  template <typename T> void Record(const T &t, bool newline = false) {
+    m_os << t;
+    if (newline)
+      m_os << '\n';
+  }
+
+  const FileSpec &GetFilename() { return m_filename; }
+
+private:
+  FileSpec m_filename;
+  llvm::raw_fd_ostream m_os;
+};
+
+struct CommandInfo {
+  static const char *name;
+  static const char *file;
+};
+
+class CommandProvider : public Provider<CommandProvider> {
+public:
+  typedef CommandInfo info;
+
+  CommandProvider(const FileSpec &directory) : Provider(directory) {}
+
+  DataRecorder *GetNewDataRecorder();
+
+  void Keep() override;
+  void Discard() override;
+
+  static char ID;
+
+private:
+  std::vector<std::unique_ptr<DataRecorder>> m_data_recorders;
+};
+
 /// The generator is responsible for the logic needed to generate a
 /// reproducer. For doing so it relies on providers, who serialize data that
 /// is necessary for reproducing  a failure.
 class Generator final {
 public:
   Generator(const FileSpec &root);
   ~Generator();
 

lldb/trunk/source/API/SBDebugger.cpp

 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ManagedStatic.h"
 
 using namespace lldb;
 using namespace lldb_private;
 
+/// Helper class for replaying commands through the reproducer.
+class CommandLoader {
+public:
+  CommandLoader(std::vector<std::string> files) : m_files(files) {}
+
+  static std::unique_ptr<CommandLoader> Create() {
+    repro::Loader *loader = repro::Reproducer::Instance().GetLoader();
+    if (!loader)
+      return {};
+
+    FileSpec file = loader->GetFile<repro::CommandInfo>();
+    if (!file)
+      return {};
+
+    auto error_or_file = llvm::MemoryBuffer::getFile(file.GetPath());
+    if (auto err = error_or_file.getError())
+      return {};
+
+    std::vector<std::string> files;
+    llvm::yaml::Input yin((*error_or_file)->getBuffer());
+    yin >> files;
+
+    if (auto err = yin.error())
+      return {};
+
+    return llvm::make_unique<CommandLoader>(std::move(files));
+  }
+
+  FILE *GetNextFile() {
+    if (m_index >= m_files.size())
+      return nullptr;
+    return FileSystem::Instance().Fopen(m_files[m_index++].c_str(), "r");
+  }
+
+private:
+  std::vector<std::string> m_files;
+  unsigned m_index = 0;
+};
+
 static llvm::sys::DynamicLibrary LoadPlugin(const lldb::DebuggerSP &debugger_sp,
                                             const FileSpec &spec,
                                             Status &error) {
   llvm::sys::DynamicLibrary dynlib =
       llvm::sys::DynamicLibrary::getPermanentLibrary(spec.GetPath().c_str());
   if (dynlib.isValid()) {
     typedef bool (*LLDBCommandPluginInit)(lldb::SBDebugger & debugger);
 
void SBDebugger::SetInputFileHandle(FILE *fh, bool transfer_ownership) {
   Log *log(GetLogIfAllCategoriesSet(LIBLLDB_LOG_API));
 
   if (log)
     log->Printf(
         "SBDebugger(%p)::SetInputFileHandle (fh=%p, transfer_ownership=%i)",
         static_cast<void *>(m_opaque_sp.get()), static_cast<void *>(fh),
         transfer_ownership);
 
-  if (m_opaque_sp)
-    m_opaque_sp->SetInputFileHandle(fh, transfer_ownership);
+  if (!m_opaque_sp)
+    return;
+
+  repro::DataRecorder *recorder = nullptr;
+  if (repro::Generator *g = repro::Reproducer::Instance().GetGenerator())
+    recorder = g->GetOrCreate<repro::CommandProvider>().GetNewDataRecorder();
+
+  static std::unique_ptr<CommandLoader> loader = CommandLoader::Create();
+  if (loader)
+    fh = loader->GetNextFile();
+
+  m_opaque_sp->SetInputFileHandle(fh, transfer_ownership, recorder);
 }
 
 void SBDebugger::SetOutputFileHandle(FILE *fh, bool transfer_ownership) {
   Log *log(GetLogIfAllCategoriesSet(LIBLLDB_LOG_API));
 
   if (log)
     log->Printf(
         "SBDebugger(%p)::SetOutputFileHandle (fh=%p, transfer_ownership=%i)",

lldb/trunk/source/Commands/CommandObjectCommands.cpp

if (argc == 1) {
       const bool multiple_lines = true; // Get multiple lines
       IOHandlerSP io_handler_sp(new IOHandlerEditline(
           debugger, IOHandler::Type::Other,
           "lldb-regex",          // Name of input reader for history
           llvm::StringRef("> "), // Prompt
           llvm::StringRef(),     // Continuation prompt
           multiple_lines, color_prompt,
           0, // Don't show line numbers
-          *this));
+          *this, nullptr));
 
       if (io_handler_sp) {
         debugger.PushIOHandler(io_handler_sp);
         result.SetStatus(eReturnStatusSuccessFinishNoResult);
       }
     } else {
       for (auto &entry : command.entries().drop_front()) {
         bool check_only = false;

lldb/trunk/source/Commands/CommandObjectExpression.cpp

SetHelpLong(
       R"(
 Single and multi-line expressions:
 
 )"
       "    The expression provided on the command line must be a complete expression \
 with no newlines.  To evaluate a multi-line expression, \
 hit a return after an empty expression, and lldb will enter the multi-line expression editor. \
 Hit return on an empty line to end the multi-line expression."
-      
       R"(
 
 Timeouts:
 
 )"
       "    If the expression can be evaluated statically (without running code) then it will be.  \
 Otherwise, by default the expression will run on the current thread with a short timeout: \
 currently .25 seconds.  If it doesn't return in that time, the evaluation will be interrupted \
void CommandObjectExpression::GetMultilineExpression() {
   const bool multiple_lines = true; // Get multiple lines
   IOHandlerSP io_handler_sp(
       new IOHandlerEditline(debugger, IOHandler::Type::Expression,
                             "lldb-expr", // Name of input reader for history
                             llvm::StringRef(), // No prompt
                             llvm::StringRef(), // Continuation prompt
                             multiple_lines, color_prompt,
                             1, // Show line numbers starting at 1
-                            *this));
+                            *this, nullptr));
 
   StreamFileSP output_sp(io_handler_sp->GetOutputStreamFile());
   if (output_sp) {
     output_sp->PutCString(
         "Enter expressions, then terminate with an empty line to evaluate:\n");
     output_sp->Flush();
   }
   debugger.PushIOHandler(io_handler_sp);

lldb/trunk/source/Core/Debugger.cpp

 }
 
 Debugger::Debugger(lldb::LogOutputCallback log_callback, void *baton)
     : UserID(g_unique_id++),
       Properties(std::make_shared<OptionValueProperties>()),
       m_input_file_sp(std::make_shared<StreamFile>(stdin, false)),
       m_output_file_sp(std::make_shared<StreamFile>(stdout, false)),
       m_error_file_sp(std::make_shared<StreamFile>(stderr, false)),
+      m_input_recorder(nullptr),
       m_broadcaster_manager_sp(BroadcasterManager::MakeBroadcasterManager()),
       m_terminal_state(), m_target_list(*this), m_platform_list(),
       m_listener_sp(Listener::MakeListener("lldb.Debugger")),
       m_source_manager_up(), m_source_file_cache(),
       m_command_interpreter_up(llvm::make_unique<CommandInterpreter>(
           *this, eScriptLanguageDefault, false)),
       m_input_reader_stack(), m_instance_name(), m_loaded_plugins(),
       m_event_handler_thread(), m_io_handler_thread(),

 bool Debugger::GetAsyncExecution() {
   return !m_command_interpreter_up->GetSynchronous();
 }
 
 void Debugger::SetAsyncExecution(bool async_execution) {
   m_command_interpreter_up->SetSynchronous(!async_execution);
 }
 
-void Debugger::SetInputFileHandle(FILE *fh, bool tranfer_ownership) {
+repro::DataRecorder *Debugger::GetInputRecorder() { return m_input_recorder; }
+
+void Debugger::SetInputFileHandle(FILE *fh, bool tranfer_ownership,
+                                  repro::DataRecorder *recorder) {
+  m_input_recorder = recorder;
   if (m_input_file_sp)
     m_input_file_sp->GetFile().SetStream(fh, tranfer_ownership);
   else
     m_input_file_sp = std::make_shared<StreamFile>(fh, tranfer_ownership);
 
   File &in_file = m_input_file_sp->GetFile();
   if (!in_file.IsValid())
     in_file.SetStream(stdin, true);

lldb/trunk/source/Core/IOHandler.cpp

 using namespace lldb;
 using namespace lldb_private;
 
 IOHandler::IOHandler(Debugger &debugger, IOHandler::Type type)
     : IOHandler(debugger, type,
                 StreamFileSP(), // Adopt STDIN from top input reader
                 StreamFileSP(), // Adopt STDOUT from top input reader
                 StreamFileSP(), // Adopt STDERR from top input reader
-                0)              // Flags
-{}
+                0,              // Flags
+                nullptr         // Shadow file recorder
+      ) {}
 
 IOHandler::IOHandler(Debugger &debugger, IOHandler::Type type,
                      const lldb::StreamFileSP &input_sp,
                      const lldb::StreamFileSP &output_sp,
-                     const lldb::StreamFileSP &error_sp, uint32_t flags)
+                     const lldb::StreamFileSP &error_sp, uint32_t flags,
+                     repro::DataRecorder *data_recorder)
     : m_debugger(debugger), m_input_sp(input_sp), m_output_sp(output_sp),
-      m_error_sp(error_sp), m_popped(false), m_flags(flags), m_type(type),
-      m_user_data(nullptr), m_done(false), m_active(false) {
+      m_error_sp(error_sp), m_data_recorder(data_recorder), m_popped(false),
+      m_flags(flags), m_type(type), m_user_data(nullptr), m_done(false),
+      m_active(false) {
   // If any files are not specified, then adopt them from the top input reader.
   if (!m_input_sp || !m_output_sp || !m_error_sp)
     debugger.AdoptTopIOHandlerFilesIfInvalid(m_input_sp, m_output_sp,
                                              m_error_sp);
 }
 
 IOHandler::~IOHandler() = default;
 
IOHandlerConfirm::IOHandlerConfirm(Debugger &debugger, llvm::StringRef prompt,
                                    bool default_response)
     : IOHandlerEditline(
           debugger, IOHandler::Type::Confirm,
           nullptr, // nullptr editline_name means no history loaded/saved
           llvm::StringRef(), // No prompt
           llvm::StringRef(), // No continuation prompt
           false,             // Multi-line
           false, // Don't colorize the prompt (i.e. the confirm message.)
-          0, *this),
+          0, *this, nullptr),
       m_default_response(default_response), m_user_response(default_response) {
   StreamString prompt_stream;
   prompt_stream.PutCString(prompt);
   if (m_default_response)
     prompt_stream.Printf(": [Y/n] ");
   else
     prompt_stream.Printf(": [y/N] ");
 
int IOHandlerDelegate::IOHandlerComplete(
   return 0;
 }
 
 IOHandlerEditline::IOHandlerEditline(
     Debugger &debugger, IOHandler::Type type,
     const char *editline_name, // Used for saving history files
     llvm::StringRef prompt, llvm::StringRef continuation_prompt,
     bool multi_line, bool color_prompts, uint32_t line_number_start,
-    IOHandlerDelegate &delegate)
+    IOHandlerDelegate &delegate, repro::DataRecorder *data_recorder)
     : IOHandlerEditline(debugger, type,
                         StreamFileSP(), // Inherit input from top input reader
                         StreamFileSP(), // Inherit output from top input reader
                         StreamFileSP(), // Inherit error from top input reader
                         0,              // Flags
                         editline_name,  // Used for saving history files
                         prompt, continuation_prompt, multi_line, color_prompts,
-                        line_number_start, delegate) {}
+                        line_number_start, delegate, data_recorder) {}
 
 IOHandlerEditline::IOHandlerEditline(
     Debugger &debugger, IOHandler::Type type,
     const lldb::StreamFileSP &input_sp, const lldb::StreamFileSP &output_sp,
     const lldb::StreamFileSP &error_sp, uint32_t flags,
     const char *editline_name, // Used for saving history files
     llvm::StringRef prompt, llvm::StringRef continuation_prompt,
     bool multi_line, bool color_prompts, uint32_t line_number_start,
-    IOHandlerDelegate &delegate)
-    : IOHandler(debugger, type, input_sp, output_sp, error_sp, flags),
+    IOHandlerDelegate &delegate, repro::DataRecorder *data_recorder)
+    : IOHandler(debugger, type, input_sp, output_sp, error_sp, flags,
+                data_recorder),
 #ifndef LLDB_DISABLE_LIBEDIT
       m_editline_up(),
 #endif
       m_delegate(delegate), m_prompt(), m_continuation_prompt(),
       m_current_lines_ptr(nullptr), m_base_line_number(line_number_start),
       m_curr_line_idx(UINT32_MAX), m_multi_line(multi_line),
       m_color_prompts(color_prompts), m_interrupt_exits(true),
       m_editing(false) {

 void IOHandlerEditline::Deactivate() {
   IOHandler::Deactivate();
   m_delegate.IOHandlerDeactivated(*this);
 }
 
 bool IOHandlerEditline::GetLine(std::string &line, bool &interrupted) {
 #ifndef LLDB_DISABLE_LIBEDIT
   if (m_editline_up) {
-    return m_editline_up->GetLine(line, interrupted);
+    bool b = m_editline_up->GetLine(line, interrupted);
+    if (m_data_recorder)
+      m_data_recorder->Record(line, true);
+    return b;
   } else {
 #endif
     line.clear();
 
     FILE *in = GetInputFILE();
     if (in) {
       if (GetIsInteractive()) {
         const char *prompt = nullptr;
if (in) {
                 break;
               last_char = buffer[buffer_len - 1];
             }
           }
           line.append(buffer, buffer_len);
         }
       }
       m_editing = false;
+      if (m_data_recorder && got_line)
+        m_data_recorder->Record(line, true);
       // We might have gotten a newline on a line by itself make sure to return
       // true in this case.
       return got_line;
     } else {
       // No more input file, we are done...
       SetIsDone(true);
     }
     return false;

lldb/trunk/source/Expression/REPL.cpp

std::string REPL::GetSourcePath() {
   return tmpdir_file_spec.GetPath();
 }
 
 lldb::IOHandlerSP REPL::GetIOHandler() {
   if (!m_io_handler_sp) {
     Debugger &debugger = m_target.GetDebugger();
     m_io_handler_sp = std::make_shared<IOHandlerEditline>(
         debugger, IOHandler::Type::REPL,
-                              "lldb-repl", // Name of input reader for history
-                              llvm::StringRef("> "), // prompt
-                              llvm::StringRef(". "), // Continuation prompt
-                              true,                  // Multi-line
-                              true, // The REPL prompt is always colored
-                              1,    // Line number
-        *this);
+        "lldb-repl",           // Name of input reader for history
+        true,                  // The REPL prompt is always colored
+        1,                     // Line number
+        *this, nullptr);
 
     // Don't exit if CTRL+C is pressed
     static_cast<IOHandlerEditline *>(m_io_handler_sp.get())
         ->SetInterruptExits(false);
 
     if (m_io_handler_sp->GetIsInteractive() &&
         m_io_handler_sp->GetIsRealTerminal()) {
       m_indent_str.assign(debugger.GetTabSize(), ' ');

lldb/trunk/source/Interpreter/CommandInterpreter.cpp

IOHandlerSP io_handler_sp(new IOHandlerEditline(
                        // input reader output stream
       empty_stream_sp, // Pass in an empty stream so we inherit the top
                        // input reader error stream
       flags,
       nullptr, // Pass in NULL for "editline_name" so no history is saved,
                // or written
       debugger.GetPrompt(), llvm::StringRef(),
       false, // Not multi-line
-      debugger.GetUseColor(), 0, *this));
+      debugger.GetUseColor(), 0, *this, nullptr));
   const bool old_async_execution = debugger.GetAsyncExecution();
 
   // Set synchronous execution if we are not stopping on continue
   if ((flags & eHandleCommandFlagStopOnContinue) == 0)
     debugger.SetAsyncExecution(false);
 
   m_command_source_depth++;
 
void CommandInterpreter::GetLLDBCommandsFromIOHandler(
   Debugger &debugger = GetDebugger();
   IOHandlerSP io_handler_sp(
       new IOHandlerEditline(debugger, IOHandler::Type::CommandList,
                             "lldb", // Name of input reader for history
                             llvm::StringRef::withNullAsEmpty(prompt), // Prompt
                             llvm::StringRef(), // Continuation prompt
                             true,              // Get multiple lines
                             debugger.GetUseColor(),
-                            0,          // Don't show line numbers
-                            delegate)); // IOHandlerDelegate
+                            0,         // Don't show line numbers
+                            delegate,  // IOHandlerDelegate
+                            nullptr)); // FileShadowCollector
 
   if (io_handler_sp) {
     io_handler_sp->SetUserData(baton);
     if (asynchronously)
       debugger.PushIOHandler(io_handler_sp);
     else
       debugger.RunIOHandler(io_handler_sp);
   }
 }
 
 void CommandInterpreter::GetPythonCommandsFromIOHandler(
     const char *prompt, IOHandlerDelegate &delegate, bool asynchronously,
     void *baton) {
   Debugger &debugger = GetDebugger();
   IOHandlerSP io_handler_sp(
       new IOHandlerEditline(debugger, IOHandler::Type::PythonCode,
                             "lldb-python", // Name of input reader for history
                             llvm::StringRef::withNullAsEmpty(prompt), // Prompt
                             llvm::StringRef(), // Continuation prompt
                             true,              // Get multiple lines
                             debugger.GetUseColor(),
-                            0,          // Don't show line numbers
-                            delegate)); // IOHandlerDelegate
+                            0,         // Don't show line numbers
+                            delegate,  // IOHandlerDelegate
+                            nullptr)); // FileShadowCollector
 
   if (io_handler_sp) {
     io_handler_sp->SetUserData(baton);
     if (asynchronously)
       debugger.PushIOHandler(io_handler_sp);
     else
       debugger.RunIOHandler(io_handler_sp);
   }
if (force_create || !m_command_io_handler_sp) {
 
     m_command_io_handler_sp = std::make_shared<IOHandlerEditline>(
         m_debugger, IOHandler::Type::CommandInterpreter,
         m_debugger.GetInputFile(), m_debugger.GetOutputFile(),
         m_debugger.GetErrorFile(), flags, "lldb", m_debugger.GetPrompt(),
         llvm::StringRef(), // Continuation prompt
         false, // Don't enable multiple line input, just single line commands
         m_debugger.GetUseColor(),
-        0, // Don't show line numbers
-        *this);
+        0,     // Don't show line numbers
+        *this, // IOHandlerDelegate
+        GetDebugger().GetInputRecorder());
   }
   return m_command_io_handler_sp;
 }
 
 void CommandInterpreter::RunCommandInterpreter(
     bool auto_handle_events, bool spawn_thread,
     CommandInterpreterRunOptions &options) {
   // Always re-create the command interpreter when we run it in case any file

lldb/trunk/source/Utility/Reproducer.cpp

 }
 
 bool Loader::HasFile(StringRef file) {
   assert(m_loaded);
   auto it = std::lower_bound(m_files.begin(), m_files.end(), file.str());
   return (it != m_files.end()) && (*it == file);
 }
 
+llvm::Expected<std::unique_ptr<DataRecorder>>
+DataRecorder::Create(FileSpec filename) {
+  std::error_code ec;
+  auto recorder = llvm::make_unique<DataRecorder>(std::move(filename), ec);
+  if (ec)
+    return llvm::errorCodeToError(ec);
+  return recorder;
+}
+
+DataRecorder *CommandProvider::GetNewDataRecorder() {
+  std::size_t i = m_data_recorders.size() + 1;
+  std::string filename = (llvm::Twine(info::name) + llvm::Twine("-") +
+                          llvm::Twine(i) + llvm::Twine(".txt"))
+                             .str();
+  auto recorder_or_error =
+      DataRecorder::Create(GetRoot().CopyByAppendingPathComponent(filename));
+  if (!recorder_or_error) {
+    llvm::consumeError(recorder_or_error.takeError());
+    return nullptr;
+  }
+
+  m_data_recorders.push_back(std::move(*recorder_or_error));
+  return m_data_recorders.back().get();
+}
+
+void CommandProvider::Keep() {
+  std::vector<std::string> files;
+  for (auto &recorder : m_data_recorders)
+    files.push_back(recorder->GetFilename().GetPath());
+
+  FileSpec file = GetRoot().CopyByAppendingPathComponent(info::file);
+  std::error_code ec;
+  llvm::raw_fd_ostream os(file.GetPath(), ec, llvm::sys::fs::F_Text);
+  if (ec)
+    return;
+  yaml::Output yout(os);
+  yout << files;
+
+  m_data_recorders.clear();
+}
+
+void CommandProvider::Discard() { m_data_recorders.clear(); }
+
 void ProviderBase::anchor() {}
 char ProviderBase::ID = 0;
 char FileProvider::ID = 0;
+char CommandProvider::ID = 0;
 const char *FileInfo::name = "files";
 const char *FileInfo::file = "files.yaml";
+const char *CommandInfo::name = "command-interpreter";
+const char *CommandInfo::file = "command-interpreter.yaml";