[Reproducers] Add command reproducer

[JDevlieghere]

Summary

Add support for reproducing to the command interpreter. In capture mode we capture all the command to file and in reproducer mode we replay this.

[labath]

My main question is why do we need the separate SBDebugger::RunReplay API for this. Shouldn't the record/replay be as transparent as possible? I would expect that SBDebugger::RunCommandInterpreter will detect that it is running in replay mode and use the recorded input command stream instead of the real one.

source/Interpreter/CommandInterpreter.cpp

131

This seems pretty hacky, and easy to break in multiple ways:

• alias rep reproducer; rep generate
• file /tmp/reproducer/bin/lldb

Couldn't you just have the actual reproducer CommandObject check what mode we're in, and behave like a nop if there is nothing to do? (So, the commands would still be captured, but they wouldn't do anything when replaying.)

2151

Is this correct? What if the command originally has ended with an error (perhaps because the user made a typo)? Will that abort the replay here?

[JDevlieghere]

Address Pavel's comments.

[JDevlieghere]

Skip driver logic when replaying, otherwise commands are executed or sourced twice, once in the driver and once by replaying the commands.

[JDevlieghere]

When sourcing a file we should also ignore the individual commands otherwise they end up getting executed twice, once as part of the command source and once for every individual command in the file.

[labath]

I am very sorry about this, but I am having some second thoughts about the RunReplay thingy. I don't think I've correctly expressed what is bothering me about it (and that's probably because I wasn't clear with myself about what is bothering me). I'll try to formulate it better this time

What I was thinking about were the boundaries at which we want to record/replay events. If I understand correctly, one of those boundaries should be the SB layer (is that correct?). So, with that in mind, it makes sense for the lldb driver to do something "special" in replay mode, and invoke some other function which would replay the SB calls it would have normally made. That function might as well be called RunReplay.

Now, if we imagine we have this SB record/replay capability, during recording it would detect that the driver made a call to SBDebugger::RunCommandInterpreter. Then, during recording it would again make a call to SBDebugger::RunCommandInterpreter, but ensure that it behaves the same way as it did the first time (this is the auto-detection part I was speaking about).

The reason this came out all wrong is because I was also thinking about a (somewhat) unrelated thing, and that is the layer at which we do the capturing of the commands. I was thinking that maybe the command interpreter doesn't need to know anything about the record/replay taking place. If this SB replay engine substituted the stdin of the debugger with a recorded version of it, then it could go normally go about it's business and things would "just work" (you'd still have to make sure that libedit doesn't do something funny, but that could hopefully be done at the IOHandler level by substituting the editline IO handler).

So, I mean, that's how I'd do it, but this is your feature, and where you wan't to draw the record/replay line ultimately depends on what you want to get out of the feature. However, I do see some advantages to doing the r/r at a lower level that I'd like to try to sell to you:

• better possibility for capturing ^C. I think ^C is fairly important, particularly in console debugging, as it's the only way to interrupt a program without it voluntarily hitting a breakpoint. It will also be fairly tricky to reproduce. With the current framework, I'm not sure how you would manage to capture that. If you worked with IO handlers and was capturing stdin, I think you'd have a better chance of getting it right (since both stdin and ^C come from the terminal).
• no need for special handling of command sourcing. As you would capture almost-raw stdin, you wouldn't need to do anything special to the avoid sourced commands being captured twice. The filesystem would just provide the captured file, and stdin-recorder would capture the stdin. (which I think would make the stdin-capturer fit nicely with the Filesystem one, as they're both fairly low-level).
• right now you're merging commands coming from multiple sources (stdin via RunCommandInterpreter, command strings from HandleCommand) into one monolithic command stream. I think this means that once you have an SB recorder, it will also have to special case command APIs to avoid doing things twice. If you work at a lower level, this won't happen, because HandleCommand would be naturally captured as part of the SB stream, and stdin commands would be captured by the stdin-recorder (which would probably also be initialized and managed by the SB recorder, since it needs to be hooked into RunCommandInterpreter)

So, basically, my point is:

• let's bring back RunReplay. I don't think it makes sense to force this to go through SBDebugger::RunCommandInterpreter in the current setup, since both the caller and the callee special-case replay mode anyway.
• I'd urge you to consider whether you really want to do the capturing at this level

[JDevlieghere]

In D55582#1329479, @labath wrote:

I am very sorry about this, but I am having some second thoughts about the RunReplay thingy. I don't think I've correctly expressed what is bothering me about it (and that's probably because I wasn't clear with myself about what is bothering me). I'll try to formulate it better this time

No worries, I appreciate you thinking this over :-)

What I was thinking about were the boundaries at which we want to record/replay events. If I understand correctly, one of those boundaries should be the SB layer (is that correct?). So, with that in mind, it makes sense for the lldb driver to do something "special" in replay mode, and invoke some other function which would replay the SB calls it would have normally made. That function might as well be called RunReplay.

Yep, the SB layer is an important boundary. Everything that happens in Xcode goes through it. It's probably going to be one of the harder parts of the feature. Every API call needs to be able to capture and replay while keeping things like references consistent. We have spent some time thinking about this, but I'm sure a bunch of complexities will pop up once we start doing the actual work.

Now, if we imagine we have this SB record/replay capability, during recording it would detect that the driver made a call to SBDebugger::RunCommandInterpreter. Then, during recording it would again make a call to SBDebugger::RunCommandInterpreter, but ensure that it behaves the same way as it did the first time (this is the auto-detection part I was speaking about).

I see what you mean now, and it's a fair point/concern. So far I've been postponing dealing with the SB layer until I could get reproducers to work with the command driver. Basically I'll have to audit all the SB endpoints anyway so I figured patching this up was going to be negligible compared to that effort. However, you're totally right that we shouldn't ignore its impact on design decisions. Moving it into a separate function will make it easier down the road, as we can just do "the right" thing in replay mode for that particular function.

The reason this came out all wrong is because I was also thinking about a (somewhat) unrelated thing, and that is the layer at which we do the capturing of the commands. I was thinking that maybe the command interpreter doesn't need to know anything about the record/replay taking place. If this SB replay engine substituted the stdin of the debugger with a recorded version of it, then it could go normally go about it's business and things would "just work" (you'd still have to make sure that libedit doesn't do something funny, but that could hopefully be done at the IOHandler level by substituting the editline IO handler).

I didn't consider that idea but I like the sound of it :-)

So, I mean, that's how I'd do it, but this is your feature, and where you wan't to draw the record/replay line ultimately depends on what you want to get out of the feature. However, I do see some advantages to doing the r/r at a lower level that I'd like to try to sell to you:

• better possibility for capturing ^C. I think ^C is fairly important, particularly in console debugging, as it's the only way to interrupt a program without it voluntarily hitting a breakpoint. It will also be fairly tricky to reproduce. With the current framework, I'm not sure how you would manage to capture that. If you worked with IO handlers and was capturing stdin, I think you'd have a better chance of getting it right (since both stdin and ^C come from the terminal).
• no need for special handling of command sourcing. As you would capture almost-raw stdin, you wouldn't need to do anything special to the avoid sourced commands being captured twice. The filesystem would just provide the captured file, and stdin-recorder would capture the stdin. (which I think would make the stdin-capturer fit nicely with the Filesystem one, as they're both fairly low-level).
• right now you're merging commands coming from multiple sources (stdin via RunCommandInterpreter, command strings from HandleCommand) into one monolithic command stream. I think this means that once you have an SB recorder, it will also have to special case command APIs to avoid doing things twice. If you work at a lower level, this won't happen, because HandleCommand would be naturally captured as part of the SB stream, and stdin commands would be captured by the stdin-recorder (which would probably also be initialized and managed by the SB recorder, since it needs to be hooked into RunCommandInterpreter)

+1 on all these points. I'll look into doing it this way today.

So, basically, my point is:

• let's bring back RunReplay. I don't think it makes sense to force this to go through SBDebugger::RunCommandInterpreter in the current setup, since both the caller and the callee special-case replay mode anyway.
• I'd urge you to consider whether you really want to do the capturing at this level

Thanks!

[JDevlieghere]

I played around with this today:

• I'm totally convinced that we should replay by swapping out stdin with a file from the reproducer. This makes this better overall.
• I'm not convinced that capturing just stdin is an improvement. What I like about the current approach is that we don't have to bother with the -o/-S commands when replaying. We already have a mechanism to deal with them so I don't think it's unreasonable to piggyback on it, especially because this is how we display it to the user. On the other hand, I generally try to keep the code path the same between the regular and reproducer case, and processing the command line options in the reproducer case would reuse more of the existing code. Finally, we'd have to figure out a way to test this, because currently we rely on the commands provided/sourced through the command line to test the replay. How would we test this functionality when only capturing from stdin?

[labath]

In D55582#1330715, @JDevlieghere wrote:

I played around with this today:

• I'm totally convinced that we should replay by swapping out stdin with a file from the reproducer. This makes this better overall.
• I'm not convinced that capturing just stdin is an improvement. What I like about the current approach is that we don't have to bother with the -o/-S commands when replaying. We already have a mechanism to deal with them so I don't think it's unreasonable to piggyback on it, especially because this is how we display it to the user.

Hm.. so the way, I'd imagine this working is that -o and friends (as well as pretty much every other command line argument) would be captured by the SB recorder, once they cross the SB boundary. For the -o commands that would be the moment the driver calls m_debugger.SetInputFileHandle https://github.com/llvm-mirror/lldb/blob/master/tools/driver/Driver.cpp#L697 , followed by m_debugger.RunCommandInterpreter a couple of lines lower. So, the SB recorder would notice that we're changing what the debugger considers as it's stdin, and direct the "stdin" recorder (or create a new instance of it, or whatever), to capture it.

Then, during replay, the SB replayer would notice that the driver called SBDebugger::SetInputFileHandle, so it would create a fake FILE* object (or maybe just open the file containing the captured commands from the disk?), and pass *that* as the stdin handle. Then it could just call SBDebugger::RunCommandInterperer the same way as the driver did. And I am hoping that pretty much everything below RunCommandInterpreter could run oblivious to the fact that replay is taking place.

On the other hand, I generally try to keep the code path the same between the regular and reproducer case, and processing the command line options in the reproducer case would reuse more of the existing code.

In this (imaginary) world, the recording/replay would only deal with the things that happen below the SB API. Anything that happens above it could only be glimpsed through it's effects on the SB. So, when initializing replay, as soon the driver notices the --replay argument it would just load that file, call RunReplay and be done. There wouldn't be a need for any command-line-argument recorder, or making sure the command line arguments match in some other way.

Finally, we'd have to figure out a way to test this, because currently we rely on the commands provided/sourced through the command line to test the replay. How would we test this functionality when only capturing from stdin?

By "capturing stdin" I meant capturing anything which the RunCommandInterpreter considers as it's "stdin", i.e., everything where the debugger gets its commands from.

So, if we assume everything I said above is true, then capturing the command stream produced by processing the command-line arguments should work the same way as capturing the commands given interactively. So, perhaps the first step would be to make sure you can capture the commands issued via command-line args. This should be easily testable via something like:

RUN: %lldb --record %t -b -o cmd1 -o cmd2 -O cmd3 ...
RUN: %lldb --replay %t | FileCheck %s
CHECK: make sure cmd1, cmd2, cmd3 got executed

Of course, capturing *real* stdin will come with additional challenges, both for recording and testing. For recording, we'll have to decide whether to try to capture raw stdin with enough fidelity so libedit can be driven through it, or we go one level higher, and try to capture the "cooked" command stream coming out of libedit. The testing strategy will depend on decision made here. In case we capture cooked commands, the situation should be very similar to command line options, so it might be enough to test by just passing the commands to the driver via stdin (%lldb <%t/my_commands.txt) instead of via command line args. In case of low-level recordings, we may need to test using something which can provide a realistic-looking fake terminal such as pexpect (yikes). But even in that case, that should be only necessary to test the handling of funny line-editing operations (such as backspaces, cursor keys, etc.), and for general functionality redirecting stdin from file should be enough.

[JDevlieghere]

In D55582#1331145, @labath wrote:

Hm.. so the way, I'd imagine this working is that -o and friends (as well as pretty much every other command line argument) would be captured by the SB recorder, once they cross the SB boundary. For the -o commands that would be the moment the driver calls m_debugger.SetInputFileHandle https://github.com/llvm-mirror/lldb/blob/master/tools/driver/Driver.cpp#L697 , followed by m_debugger.RunCommandInterpreter a couple of lines lower. So, the SB recorder would notice that we're changing what the debugger considers as it's stdin, and direct the "stdin" recorder (or create a new instance of it, or whatever), to capture it.

Then, during replay, the SB replayer would notice that the driver called SBDebugger::SetInputFileHandle, so it would create a fake FILE* object (or maybe just open the file containing the captured commands from the disk?), and pass *that* as the stdin handle. Then it could just call SBDebugger::RunCommandInterperer the same way as the driver did. And I am hoping that pretty much everything below RunCommandInterpreter could run oblivious to the fact that replay is taking place.

Yes, this is how SB record/replay will work. I don't disagree with this
approach, it's just that, as I said in my earlier reply, I was hoping to get
the driver working without the need for the SB reproducers. Maybe I should give
up on that?

In this (imaginary) world, the recording/replay would only deal with the things that happen below the SB API. Anything that happens above it could only be glimpsed through it's effects on the SB. So, when initializing replay, as soon the driver notices the --replay argument it would just load that file, call RunReplay and be done. There wouldn't be a need for any command-line-argument recorder, or making sure the command line arguments match in some other way.

By "capturing stdin" I meant capturing anything which the RunCommandInterpreter considers as it's "stdin", i.e., everything where the debugger gets its commands from.

This is the tricky part though. Where and how are you going to do this? In my
opinion this shouldn't be the responsibility of the SB recorder. The latter
should be concerned with "serializing" and "deserializing" its input and
tracking what SB layer calls are being made. It should be relatively agnostic
of what the debugger does with that at a lower level.

The FILE* is particularly annoying. For the sake of argument let's say that
"serializing" a FILE* means writing its content to the reproducer, how would
we accomplish this? Maybe there's an API I don't know about, but wouldn't we
have to do this at the fgets level? For the command interpreter this is
either in libedit or in the IO handler (depending on the fidelity you touched
on below).

So, if we assume everything I said above is true, then capturing the command stream produced by processing the command-line arguments should work the same way as capturing the commands given interactively. So, perhaps the first step would be to make sure you can capture the commands issued via command-line args. This should be easily testable via something like:

RUN: %lldb --record %t -b -o cmd1 -o cmd2 -O cmd3 ...
RUN: %lldb --replay %t | FileCheck %s
CHECK: make sure cmd1, cmd2, cmd3 got executed

Of course, capturing *real* stdin will come with additional challenges, both for recording and testing. For recording, we'll have to decide whether to try to capture raw stdin with enough fidelity so libedit can be driven through it, or we go one level higher, and try to capture the "cooked" command stream coming out of libedit. The testing strategy will depend on decision made here. In case we capture cooked commands, the situation should be very similar to command line options, so it might be enough to test by just passing the commands to the driver via stdin (%lldb <%t/my_commands.txt) instead of via command line args. In case of low-level recordings, we may need to test using something which can provide a realistic-looking fake terminal such as pexpect (yikes). But even in that case, that should be only necessary to test the handling of funny line-editing operations (such as backspaces, cursor keys, etc.), and for general functionality redirecting stdin from file should be enough.

I don't think we really differentiate between *real* stdin and a file. Even
when sourcing a file go through the IOHandler, so that should be pretty
transparent, unless you actually want to differentiate which, if I understand
correctly, wouldn't be needed with your proposed approach?

[labath]

In D55582#1331333, @JDevlieghere wrote:

Yes, this is how SB record/replay will work. I don't disagree with this
approach, it's just that, as I said in my earlier reply, I was hoping to get
the driver working without the need for the SB reproducers. Maybe I should give
up on that?

Hm... I don't know.. as this patch shows, it seems it is possible to get some results without the SB reproducers. My fear is that this will produce a solution which is very specialized to this particular use case, and which will have to be redone once you want more functionality out of it. Maybe that doesn't matter? I'm not sure..

In this (imaginary) world, the recording/replay would only deal with the things that happen below the SB API. Anything that happens above it could only be glimpsed through it's effects on the SB. So, when initializing replay, as soon the driver notices the --replay argument it would just load that file, call RunReplay and be done. There wouldn't be a need for any command-line-argument recorder, or making sure the command line arguments match in some other way.

By "capturing stdin" I meant capturing anything which the RunCommandInterpreter considers as it's "stdin", i.e., everything where the debugger gets its commands from.

This is the tricky part though. Where and how are you going to do this? In my
opinion this shouldn't be the responsibility of the SB recorder. The latter
should be concerned with "serializing" and "deserializing" its input and
tracking what SB layer calls are being made. It should be relatively agnostic
of what the debugger does with that at a lower level.

The FILE* is particularly annoying. For the sake of argument let's say that
"serializing" a FILE* means writing its content to the reproducer, how would
we accomplish this? Maybe there's an API I don't know about, but wouldn't we
have to do this at the fgets level? For the command interpreter this is
either in libedit or in the IO handler (depending on the fidelity you touched
on below).

Yes, the FILE* is quite tricky, and it might be the trickiest part of capturing the SB interface. I would actually argue that capturing this *is* in the scope of the SB recorder, which I would define as capturing the data that flows across the SB boundary. For a FILE*, that obviously doesn't mean the pointer value, because that's going to be meaningless in subsequent runs. It has do do a "deep" capture of the FILE* so that it can later be recreated with reasonable fidelity.

In an ideal world, it would just take a snapshot of the file, and then move on, but that is of course not possible, because the FILE* may be a stream, and it would have to look into the future to capture it instantaneously. So, it will have to capture it incrementally, as it is being read. Which will require some cooperation from the lower layers of the debugger.

I'd imagine this could be done by adding a level of indirection, similar to how you've made all filesystem accesses go through the FileSystem class. Instead of FILE*. everything would deal with some proxy class, which would record it's interaction into the log in record mode and replay it from saved data in replay mode. Most of the code would still be oblivious to the fact that anything funny is going on, but it would have to be changed to deal with the proxy class instead of the raw APIs.

Of course, capturing *real* stdin will come with additional challenges, both for recording and testing. For recording, we'll have to decide whether to try to capture raw stdin with enough fidelity so libedit can be driven through it, or we go one level higher, and try to capture the "cooked" command stream coming out of libedit. The testing strategy will depend on decision made here. In case we capture cooked commands, the situation should be very similar to command line options, so it might be enough to test by just passing the commands to the driver via stdin (%lldb <%t/my_commands.txt) instead of via command line args. In case of low-level recordings, we may need to test using something which can provide a realistic-looking fake terminal such as pexpect (yikes). But even in that case, that should be only necessary to test the handling of funny line-editing operations (such as backspaces, cursor keys, etc.), and for general functionality redirecting stdin from file should be enough.

I don't think we really differentiate between *real* stdin and a file. Even
when sourcing a file go through the IOHandler, so that should be pretty
transparent, unless you actually want to differentiate which, if I understand
correctly, wouldn't be needed with your proposed approach?

I am afraid you lost me there. I am not sure what you wanted to say, but yes, I am saying that (for the most part) we don't have to differentiate between a file and real (tty) stdin.

[JDevlieghere]

Abandoning this in favor of D56322

include/lldb/Core/Debugger.h

public:
   bool IsForwardingEvents();
 
   void EnableForwardEvents(const lldb::ListenerSP &listener_sp);
 
   void CancelForwardEvents(const lldb::ListenerSP &listener_sp);
 
   bool IsHandlingEvents() const { return m_event_handler_thread.IsJoinable(); }
 
+  void RunReplay();
+
   Status RunREPL(lldb::LanguageType language, const char *repl_options);
 
   // This is for use in the command interpreter, when you either want the
   // selected target, or if no target is present you want to prime the dummy
   // target with entities that will be copied over to new targets.
   Target *GetSelectedOrDummyTarget(bool prefer_dummy = false);
   Target *GetDummyTarget();
 

include/lldb/Interpreter/CommandInterpreter.h

 //===-- CommandInterpreter.h ------------------------------------*- C++ -*-===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
-//
 //===----------------------------------------------------------------------===//
 
 #ifndef liblldb_CommandInterpreter_h_
 #define liblldb_CommandInterpreter_h_
 
 #include <mutex>
 #include "lldb/Core/Broadcaster.h"
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/Event.h"
 #include "lldb/Core/IOHandler.h"
 #include "lldb/Interpreter/CommandAlias.h"
 #include "lldb/Interpreter/CommandHistory.h"
 #include "lldb/Interpreter/CommandObject.h"
 #include "lldb/Interpreter/ScriptInterpreter.h"
 #include "lldb/Utility/Args.h"
 #include "lldb/Utility/CompletionRequest.h"
 #include "lldb/Utility/Log.h"
 #include "lldb/Utility/StringList.h"
 #include "lldb/lldb-forward.h"
 #include "lldb/lldb-private.h"
 
 namespace lldb_private {
 
+class CommandProvider;
+
 class CommandInterpreterRunOptions {
 public:
   //------------------------------------------------------------------
   /// Construct a CommandInterpreterRunOptions object. This class is used to
   /// control all the instances where we run multiple commands, e.g.
   /// HandleCommands, HandleCommandsFromFile, RunCommandInterpreter.
   ///
   /// The meanings of the options in this object are:
public:
   }
 
   bool GetAddToHistory() const { return DefaultToYes(m_add_to_history); }
 
   void SetAddToHistory(bool add_to_history) {
     m_add_to_history = add_to_history ? eLazyBoolYes : eLazyBoolNo;
   }
 
+  bool GetAddToReproducer() const { return DefaultToYes(m_add_to_reproducer); }
+
+  void SetAddToReproducer(bool add_to_reproducer) {
+    m_add_to_reproducer = add_to_reproducer ? eLazyBoolYes : eLazyBoolNo;
+  }
+
   LazyBool m_stop_on_continue;
   LazyBool m_stop_on_error;
   LazyBool m_stop_on_crash;
   LazyBool m_echo_commands;
   LazyBool m_echo_comment_commands;
   LazyBool m_print_results;
   LazyBool m_add_to_history;
+  LazyBool m_add_to_reproducer;
 
 private:
   static bool DefaultToYes(LazyBool flag) {
     switch (flag) {
     case eLazyBoolNo:
       return false;
     default:
       return true;
public:
   // These two functions fill out the Broadcaster interface:
 
   static ConstString &GetStaticBroadcasterClass();
 
   ConstString &GetBroadcasterClass() const override {
     return GetStaticBroadcasterClass();
   }
 
+  void ReplayCommands(CommandReturnObject &result);
+
   void SourceInitFile(bool in_cwd, CommandReturnObject &result);
 
   bool AddCommand(llvm::StringRef name, const lldb::CommandObjectSP &cmd_sp,
                   bool can_replace);
 
   bool AddUserCommand(llvm::StringRef name, const lldb::CommandObjectSP &cmd_sp,
                       bool can_replace);
 
CommandObject *BuildAliasResult(llvm::StringRef alias_name,
                                   std::string &raw_input_string,
                                   std::string &alias_result,
                                   CommandReturnObject &result);
 
   bool HandleCommand(const char *command_line, LazyBool add_to_history,
                      CommandReturnObject &result,
                      ExecutionContext *override_context = nullptr,
                      bool repeat_on_empty_command = true,
-                     bool no_context_switching = false);
+                     bool no_context_switching = false,
+                     bool add_to_reproducer = false);
 
   bool WasInterrupted() const;
 
   //------------------------------------------------------------------
   /// Execute a list of commands in sequence.
   ///
   /// @param[in] commands
   ///    The list of commands to execute.
ChildrenTruncatedWarningStatus m_truncation_warning; // Whether we truncated
                                                        // children and whether
                                                        // the user has been told
   uint32_t m_command_source_depth;
   std::vector<uint32_t> m_command_source_flags;
   uint32_t m_num_errors;
   bool m_quit_requested;
   bool m_stopped_for_crash;
 
+  // Reproducer provider.
+  CommandProvider *m_provider = nullptr;
+
   // The exit code the user has requested when calling the 'quit' command.
   // No value means the user hasn't set a custom exit code so far.
   llvm::Optional<int> m_quit_exit_code;
   // If the driver is accepts custom exit codes for the 'quit' command.
   bool m_allow_exit_code = false;
 };
 
 } // namespace lldb_private
 
 #endif // liblldb_CommandInterpreter_h_

lit/Reproducer/Inputs/CommandCapture.in

+breakpoint set -f simple.c -l 13
+run
+bt
+cont
+reproducer status
+reproducer generate

lit/Reproducer/TestCommandRepro.test

+# UNSUPPORTED: system-windows, system-freebsd
+
+# RUN: rm -rf %T/commands
+# RUN: %clang %S/Inputs/simple.c -g -o %t.out
+
+# This tests the replaying of commands.
+#
+# RUN: %lldb -x -b -o 'breakpoint set -f simple.c -l 13' -o 'run' -o 'bt' -o 'cont' -o 'reproducer status' -o 'reproducer generate' --capture %T/commands -- %t.out | FileCheck %s --check-prefix CHECK --check-prefix CAPTURE
+# RUN: %lldb -x -b --replay %T/commands -- %t.out | FileCheck %s --check-prefix CHECK --check-prefix REPLAY
+# RUN: %lldb -x -b --replay %T/commands | FileCheck %s --check-prefix CHECK --check-prefix REPLAY
+
+# RUN: rm -rf %T/commands
+
+# Test that sourced files are not captured double.
+#
+# RUN: %lldb -x -b -s %S/Inputs/CommandCapture.in --capture %T/commands -- %t.out | FileCheck %s --check-prefix CHECK --check-prefix CAPTURE
+# RUN: %lldb -x -b --replay %T/commands -- %t.out | FileCheck %s --check-prefix CHECK --check-prefix REPLAY
+
+# This tests that errors are printed.
+#
+# RUN: echo "bogus" >> %T/commands/command-interpreter.txt
+# RUN: %lldb -x -b --replay %T/commands 2>&1 | FileCheck %s --check-prefix CHECK --check-prefix REPLAY --check-prefix BOGUS
+
+# CHECK: Breakpoint 1
+# CHECK: Process {{.*}} stopped
+# CHECK: Process {{.*}} launched
+# CHECK: thread {{.*}} stop reason = breakpoint
+# CHECK: frame {{.*}} simple.c
+
+# CAPTURE: testing
+# REPLAY-NOT: testing
+
+# CHECK: Process {{.*}} resuming
+# CHECK: Process {{.*}} exited
+
+# CAPTURE: Reproducer is in capture mode.
+# CAPTURE: Reproducer written
+
+# BOGUS: error: 'bogus' is not a valid command.

source/API/SBDebugger.cpp

 #include "lldb/Host/XML.h"
 #include "lldb/Initialization/SystemLifetimeManager.h"
 #include "lldb/Interpreter/CommandInterpreter.h"
 #include "lldb/Interpreter/OptionArgParser.h"
 #include "lldb/Interpreter/OptionGroupPlatform.h"
 #include "lldb/Target/Process.h"
 #include "lldb/Target/TargetList.h"
 #include "lldb/Utility/Args.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/State.h"
 
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/DynamicLibrary.h"
 #include "llvm/Support/ManagedStatic.h"
 
 using namespace lldb;
if (m_opaque_sp)
     m_opaque_sp->DispatchInputEndOfFile();
 }
 
 void SBDebugger::PushInputReader(SBInputReader &reader) {}
 
 void SBDebugger::RunCommandInterpreter(bool auto_handle_events,
                                        bool spawn_thread) {
   if (m_opaque_sp) {
-    CommandInterpreterRunOptions options;
+    // If there's a loader we're in replay mode.
+    if (repro::Reproducer::Instance().GetLoader()) {
+      m_opaque_sp->RunReplay();
+      return;
+    }
 
+    CommandInterpreterRunOptions options;
     m_opaque_sp->GetCommandInterpreter().RunCommandInterpreter(
         auto_handle_events, spawn_thread, options);
   }
 }
 
 void SBDebugger::RunCommandInterpreter(bool auto_handle_events,
                                        bool spawn_thread,
                                        SBCommandInterpreterRunOptions &options,
                                        int &num_errors, bool &quit_requested,
                                        bool &stopped_for_crash)
 
 {
   if (m_opaque_sp) {
     CommandInterpreter &interp = m_opaque_sp->GetCommandInterpreter();
+
+    // If there's a loader we're in replay mode.
+    if (repro::Reproducer::Instance().GetLoader()) {
+      m_opaque_sp->RunReplay();
+      return;
+    }
+
     interp.RunCommandInterpreter(auto_handle_events, spawn_thread,
                                  options.ref());
     num_errors = interp.GetNumErrors();
     quit_requested = interp.GetQuitRequested();
     stopped_for_crash = interp.GetStoppedForCrash();
   }
 }
 

source/Commands/CommandObjectCommands.cpp

bool DoExecute(Args &command, CommandReturnObject &result) override {
     // If any options were set, then use them
     if (m_options.m_stop_on_error.OptionWasSet() ||
         m_options.m_silent_run.OptionWasSet() ||
         m_options.m_stop_on_continue.OptionWasSet()) {
       // Use user set settings
       CommandInterpreterRunOptions options;
       options.SetStopOnContinue(m_options.m_stop_on_continue.GetCurrentValue());
       options.SetStopOnError(m_options.m_stop_on_error.GetCurrentValue());
+      options.SetAddToReproducer(false);
 
       // Individual silent setting is override for global command echo settings.
       if (m_options.m_silent_run.GetCurrentValue()) {
         options.SetSilent(true);
       } else {
         options.SetPrintResults(true);
         options.SetEchoCommands(m_interpreter.GetEchoCommands());
         options.SetEchoCommentCommands(m_interpreter.GetEchoCommentCommands());
       }
 
       m_interpreter.HandleCommandsFromFile(cmd_file, exe_ctx, options, result);
     } else {
       // No options were set, inherit any settings from nested "command source"
       // commands, or set to sane default settings...
       CommandInterpreterRunOptions options;
+      options.SetAddToReproducer(false);
       m_interpreter.HandleCommandsFromFile(cmd_file, exe_ctx, options, result);
     }
     return result.Succeeded();
   }
 
   CommandOptions m_options;
 };
 

source/Commands/CommandObjectReproducer.cpp

if (!command.empty()) {
       result.AppendErrorWithFormat("'%s' takes no arguments",
                                    m_cmd_name.c_str());
       return false;
     }
 
     auto &r = repro::Reproducer::Instance();
     if (auto generator = r.GetGenerator()) {
       generator->Keep();
+    } else if (r.GetLoader()) {
+      // Make this operation a NOP in replay mode.
+      result.SetStatus(eReturnStatusSuccessFinishNoResult);
+      return result.Succeeded();
     } else {
       result.AppendErrorWithFormat("Unable to get the reproducer generator");
+      result.SetStatus(eReturnStatusFailed);
       return false;
     }
 
     result.GetOutputStream()
         << "Reproducer written to '" << r.GetReproducerPath() << "'\n";
 
     result.SetStatus(eReturnStatusSuccessFinishResult);
     return result.Succeeded();

source/Core/Debugger.cpp

 #include "lldb/DataFormatters/DataVisualization.h"
 #include "lldb/Expression/REPL.h"
 #include "lldb/Host/File.h"
 #include "lldb/Host/FileSystem.h"
 #include "lldb/Host/HostInfo.h"
 #include "lldb/Host/Terminal.h"
 #include "lldb/Host/ThreadLauncher.h"
 #include "lldb/Interpreter/CommandInterpreter.h"
+#include "lldb/Interpreter/CommandReturnObject.h"
 #include "lldb/Interpreter/OptionValue.h"
 #include "lldb/Interpreter/OptionValueProperties.h"
 #include "lldb/Interpreter/OptionValueSInt64.h"
 #include "lldb/Interpreter/OptionValueString.h"
 #include "lldb/Interpreter/Property.h"
 #include "lldb/Interpreter/ScriptInterpreter.h"
 #include "lldb/Symbol/Function.h"
 #include "lldb/Symbol/Symbol.h"
if (!prefer_dummy) {
     target = m_target_list.GetSelectedTarget().get();
     if (target)
       return target;
   }
 
   return GetDummyTarget();
 }
 
+void Debugger::RunReplay() {
+  // Create a command return object and hook up its streams.
+  CommandReturnObject result;
+  result.SetImmediateOutputStream(GetOutputFile());
+  result.SetImmediateErrorStream(GetErrorFile());
+
+  // Replay commands from reproducer.
+  GetCommandInterpreter().ReplayCommands(result);
+}
+
 Status Debugger::RunREPL(LanguageType language, const char *repl_options) {
   Status err;
   FileSpec repl_executable;
 
   if (language == eLanguageTypeUnknown) {
     std::set<LanguageType> repl_languages;
 
     Language::GetLanguagesSupportingREPLs(repl_languages);

source/Interpreter/CommandInterpreter.cpp

 #include "Commands/CommandObjectType.h"
 #include "Commands/CommandObjectVersion.h"
 #include "Commands/CommandObjectWatchpoint.h"
 
 #include "lldb/Core/Debugger.h"
 #include "lldb/Core/PluginManager.h"
 #include "lldb/Core/StreamFile.h"
 #include "lldb/Utility/Log.h"
+#include "lldb/Utility/Reproducer.h"
 #include "lldb/Utility/State.h"
 #include "lldb/Utility/Stream.h"
 #include "lldb/Utility/Timer.h"
 
 #ifndef LLDB_DISABLE_LIBEDIT
 #include "lldb/Host/Editline.h"
 #endif
 #include "lldb/Host/Host.h"

 
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallString.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/PrettyStackTrace.h"
 
 using namespace lldb;
 using namespace lldb_private;
+using namespace llvm;
 
 static const char *k_white_space = " \t\v";
 
 static constexpr bool NoGlobalSetting = true;
 static constexpr uintptr_t DefaultValueTrue = true;
 static constexpr uintptr_t DefaultValueFalse = false;
 static constexpr const char *NoCStrDefault = nullptr;
 
enum {
   ePropertyExpandRegexAliases = 0,
   ePropertyPromptOnQuit = 1,
   ePropertyStopCmdSourceOnError = 2,
   eSpaceReplPrompts = 3,
   eEchoCommands = 4,
   eEchoCommentCommands = 5
 };
 
+class lldb_private::CommandProvider
+    : public repro::Provider<lldb_private::CommandProvider> {
+public:
+  CommandProvider(const FileSpec &directory) : Provider(directory) {
+    m_info.name = "command-interpreter";
+    m_info.files.push_back("command-interpreter.txt");
+  }
+
+  void CaptureCommand(std::string command) {
+    m_commands.push_back(std::move(command));
+  }
+
+  void Keep() override {
+    FileSpec file =
+        GetRoot().CopyByAppendingPathComponent("command-interpreter.txt");
+
+    std::error_code ec;
+    llvm::raw_fd_ostream os(file.GetPath(), ec, llvm::sys::fs::F_Text);
+
+    if (ec)
+      return;
+
+    for (auto &command : m_commands)
+      os << command << '\n';
+  }
+
+  void Discard() override {}
+
+  static char ID;
+
+private:
+  std::vector<std::string> m_commands;
+};
+
+char CommandProvider::ID = 0;
+
 ConstString &CommandInterpreter::GetStaticBroadcasterClass() {
   static ConstString class_name("lldb.commandInterpreter");
   return class_name;
 }
 
 CommandInterpreter::CommandInterpreter(Debugger &debugger,
                                        ScriptLanguage script_language,
                                        bool synchronous_execution)
: Broadcaster(debugger.GetBroadcasterManager(),
       m_command_source_depth(0), m_num_errors(0), m_quit_requested(false),
       m_stopped_for_crash(false) {
   debugger.SetScriptLanguage(script_language);
   SetEventName(eBroadcastBitThreadShouldExit, "thread-should-exit");
   SetEventName(eBroadcastBitResetPrompt, "reset-prompt");
   SetEventName(eBroadcastBitQuitCommandReceived, "quit");
   CheckInWithManager();
   m_collection_sp->Initialize(g_properties);
+
+  if (repro::Generator *g = repro::Reproducer::Instance().GetGenerator())
+    m_provider = &g->GetOrCreate<CommandProvider>();
 }
 
 bool CommandInterpreter::GetExpandRegexAliases() const {
   const uint32_t idx = ePropertyExpandRegexAliases;
   return m_collection_sp->GetPropertyAtIndexAsBoolean(
       nullptr, idx, g_properties[idx].default_uint_value != 0);
 }
 
Status CommandInterpreter::PreprocessCommand(std::string &command) {
   return error;
 }
 
 bool CommandInterpreter::HandleCommand(const char *command_line,
                                        LazyBool lazy_add_to_history,
                                        CommandReturnObject &result,
                                        ExecutionContext *override_context,
                                        bool repeat_on_empty_command,
-                                       bool no_context_switching)
+                                       bool no_context_switching,
+                                       bool add_to_reproducer)
 
 {
 
   std::string command_string(command_line);
   std::string original_command_string(command_line);
 
   Log *log(lldb_private::GetLogIfAllCategoriesSet(LIBLLDB_LOG_COMMANDS));
   llvm::PrettyStackTraceFormat stack_trace("HandleCommand(command = \"%s\")",
if (empty_command) {
     }
   } else if (comment_command) {
     result.SetStatus(eReturnStatusSuccessFinishNoResult);
     return true;
   }
 
   Status error(PreprocessCommand(command_string));
 
+  if (m_provider && add_to_reproducer)
+    m_provider->CaptureCommand(original_command_string);
+
   if (error.Fail()) {
     result.AppendError(error.AsCString());
     result.SetStatus(eReturnStatusFailed);
     return false;
   }
 
   // Phase 1.
 
if (isdigit(cptr[0])) {
       if (cptr[0] == '\0')
         position = atoi(start);
     }
   }
 
   return position;
 }
 
+void CommandInterpreter::ReplayCommands(CommandReturnObject &result) {
+  repro::Loader *loader = repro::Reproducer::Instance().GetLoader();
+  if (!loader) {
+    result.SetStatus(eReturnStatusSuccessFinishNoResult);
+    return;
+  }
+
+  auto provider_info = loader->GetProviderInfo("command-interpreter");
+  if (!provider_info) {
+    result.AppendErrorWithFormat("no provider for command interpreter.");
+    result.SetStatus(eReturnStatusFailed);
+    return;
+  }
+
+  if (provider_info->files.empty()) {
+    result.AppendErrorWithFormat(
+        "provider for command interpreter contains no files.");
+    result.SetStatus(eReturnStatusFailed);
+    return;
+  }
+
+  FileSpec command_file = loader->GetRoot().CopyByAppendingPathComponent(
+      provider_info->files.front());
+
+  // Remember current batch mode.
+  const bool saved_batch = SetBatchCommandMode(true);
+
+  CommandInterpreterRunOptions options;
+  options.SetSilent(false);
+  options.SetStopOnError(false);
+  options.SetStopOnContinue(false);
+
+  // Run the commands in a new execution context.
+  HandleCommandsFromFile(command_file, nullptr, options, result);
+
+  // Reset current batch mode.
+  SetBatchCommandMode(saved_batch);
+}
+
 void CommandInterpreter::SourceInitFile(bool in_cwd,
                                         CommandReturnObject &result) {
   FileSpec init_file;
   if (in_cwd) {
     ExecutionContext exe_ctx(GetExecutionContext());
     Target *target = exe_ctx.GetTargetPtr();
     if (target) {
       // In the current working directory we don't load any program specific
void CommandInterpreter::SourceInitFile(bool in_cwd,
   // CommandObjectSource::Execute for more details).
 
   if (FileSystem::Instance().Exists(init_file)) {
     const bool saved_batch = SetBatchCommandMode(true);
     CommandInterpreterRunOptions options;
     options.SetSilent(true);
     options.SetStopOnError(false);
     options.SetStopOnContinue(true);
+    options.SetAddToReproducer(false);
 
     HandleCommandsFromFile(init_file,
                            nullptr, // Execution context
                            options, result);
     SetBatchCommandMode(saved_batch);
   } else {
     // nothing to be done if the file doesn't exist
     result.SetStatus(eReturnStatusSuccessFinishNoResult);
for (size_t idx = 0; idx < num_lines && !WasInterrupted(); idx++) {
     // add_to_history will get lost.  This m_command_source_depth dingus is the
     // way we turn off adding to the history in that case, so set it up here.
     if (!options.GetAddToHistory())
       m_command_source_depth++;
     bool success =
         HandleCommand(cmd, options.m_add_to_history, tmp_result,
                       nullptr, /* override_context */
                       true,    /* repeat_on_empty_command */
-                      override_context != nullptr /* no_context_switching */);
+                      override_context != nullptr /* no_context_switching */,
+                      options.GetAddToReproducer());
     if (!options.GetAddToHistory())
       m_command_source_depth--;
 
     if (options.GetPrintResults()) {
       if (tmp_result.Succeeded())
         result.AppendMessage(tmp_result.GetOutputData());
     }
 

 // Make flags that we can pass into the IOHandler so our delegates can do the
 // right thing
 enum {
   eHandleCommandFlagStopOnContinue = (1u << 0),
   eHandleCommandFlagStopOnError = (1u << 1),
   eHandleCommandFlagEchoCommand = (1u << 2),
   eHandleCommandFlagEchoCommentCommand = (1u << 3),
   eHandleCommandFlagPrintResult = (1u << 4),
-  eHandleCommandFlagStopOnCrash = (1u << 5)
+  eHandleCommandFlagStopOnCrash = (1u << 5),
+  eHandleCommandFlagAddToReproducer = (1u << 6)
 };
 
 void CommandInterpreter::HandleCommandsFromFile(
     FileSpec &cmd_file, ExecutionContext *context,
     CommandInterpreterRunOptions &options, CommandReturnObject &result) {
   if (FileSystem::Instance().Exists(cmd_file)) {
     StreamFileSP input_file_sp(new StreamFile());
 
if (error.Success()) {
         } else if (m_command_source_flags.back() &
                    eHandleCommandFlagPrintResult) {
           flags |= eHandleCommandFlagPrintResult;
         }
       } else if (options.m_print_results == eLazyBoolYes) {
         flags |= eHandleCommandFlagPrintResult;
       }
 
+      if (options.GetAddToReproducer()) {
+        flags |= eHandleCommandFlagAddToReproducer;
+      }
+
       if (flags & eHandleCommandFlagPrintResult) {
         debugger.GetOutputFile()->Printf("Executing commands in '%s'.\n",
                                          cmd_file_path.c_str());
       }
 
       // Used for inheriting the right settings when "command source" might
       // have nested "command source" commands
       lldb::StreamFileSP empty_stream_sp;
if (is_interactive == false) {
     if (EchoCommandNonInteractive(line, io_handler.GetFlags()))
       io_handler.GetOutputStreamFile()->Printf("%s%s\n", io_handler.GetPrompt(),
                                                line.c_str());
   }
 
   StartHandlingCommand();
 
   lldb_private::CommandReturnObject result;
-  HandleCommand(line.c_str(), eLazyBoolCalculate, result);
+  HandleCommand(line.c_str(), eLazyBoolCalculate, result, nullptr, true, false,
+                io_handler.GetFlags().Test(eHandleCommandFlagAddToReproducer));
 
   // Now emit the command output text from the command we just executed
   if (io_handler.GetFlags().Test(eHandleCommandFlagPrintResult)) {
     // Display any STDOUT/STDERR _prior_ to emitting the command result text
     GetProcessOutput();
 
     if (!result.GetImmediateOutputStream()) {
       llvm::StringRef output = result.GetOutputData();
if (options) {
       if (options->m_stop_on_crash == eLazyBoolYes)
         flags |= eHandleCommandFlagStopOnCrash;
       if (options->m_echo_commands != eLazyBoolNo)
         flags |= eHandleCommandFlagEchoCommand;
       if (options->m_echo_comment_commands != eLazyBoolNo)
         flags |= eHandleCommandFlagEchoCommentCommand;
       if (options->m_print_results != eLazyBoolNo)
         flags |= eHandleCommandFlagPrintResult;
+      if (options->m_add_to_reproducer != eLazyBoolNo)
+        flags |= eHandleCommandFlagAddToReproducer;
     } else {
       flags = eHandleCommandFlagEchoCommand | eHandleCommandFlagPrintResult;
     }
 
     m_command_io_handler_sp.reset(new IOHandlerEditline(
         m_debugger, IOHandler::Type::CommandInterpreter,
         m_debugger.GetInputFile(), m_debugger.GetOutputFile(),
         m_debugger.GetErrorFile(), flags, "lldb", m_debugger.GetPrompt(),

tools/driver/Driver.h

struct OptionData {
 
     bool m_debug_mode = false;
     bool m_source_quietly = false;
     bool m_print_version = false;
     bool m_print_python_path = false;
     bool m_wait_for = false;
     bool m_repl = false;
     bool m_batch = false;
+    bool m_replay = false;
 
     // FIXME: When we have set/show variables we can remove this from here.
     bool m_use_external_editor = false;
 
     typedef std::set<char> OptionSet;
     OptionSet m_seen_options;
   };
 

tools/driver/Driver.cpp

if (!file.Exists()) {
       error.SetErrorStringWithFormat(
           "file specified in --core (-c) option doesn't exist: '%s'",
           arg_value);
       return error;
     }
     m_option_data.m_core_file = arg_value;
   }
 
+  if (args.hasArg(OPT_replay)) {
+    m_option_data.m_replay = true;
+  }
+
   if (args.hasArg(OPT_editor)) {
     m_option_data.m_use_external_editor = true;
   }
 
   if (args.hasArg(OPT_no_lldbinit)) {
     m_debugger.SkipLLDBInitFiles(true);
     m_debugger.SkipAppInitFiles(true);
   }
if (m_option_data.m_repl) {
     SBError error(m_debugger.RunREPL(m_option_data.m_repl_lang, repl_options));
     if (error.Fail()) {
       const char *error_cstr = error.GetCString();
       if (error_cstr && error_cstr[0])
         WithColor::error() << error_cstr << '\n';
       else
         WithColor::error() << error.GetError() << '\n';
     }
+  } else if (m_option_data.m_replay) {
+    m_debugger.RunCommandInterpreter(handle_events, spawn_thread);
   } else {
     // Check if we have any data in the commands stream, and if so, save it to a
     // temp file
     // so we can then run the command interpreter using the file contents.
     const char *commands_data = commands_stream.GetData();
     const size_t commands_size = commands_stream.GetSize();
 
     // The command file might have requested that we quit, this variable will