[lldb] Protect the debugger's output and error stream

[JDevlieghere]

Summary

This patch introduces a small utility called ThreadSafeStream to protect the debugger's output and error stream. Nothing prevents someone from bypassing this and going straight through the debugger's Get{Output,Error}File. I don't think we want to get rid of that, but at least now we have something that allows us to access these streams in a safe way.

This fixes a real world race between the main thread and the default event handler thread:

Write of size 8 at 0x000106c01bc0 by thread T1 (mutexes: write M4069, write M2753):
  #0 lldb_private::IOHandlerEditline::PrintAsync(lldb_private::Stream*, char const*, unsigned long) IOHandler.cpp:636 (liblldb.15.0.0git.dylib:arm64+0x3ef4a4)
  #1 lldb_private::IOHandlerStack::PrintAsync(lldb_private::Stream*, char const*, unsigned long) IOHandler.cpp:129 (liblldb.15.0.0git.dylib:arm64+0x3ec84c)
  #2 lldb_private::Debugger::PrintAsync(char const*, unsigned long, bool) Debugger.cpp:1074 (liblldb.15.0.0git.dylib:arm64+0x3cb03c)
  #3 lldb_private::StreamAsynchronousIO::~StreamAsynchronousIO() StreamAsynchronousIO.cpp:21 (liblldb.15.0.0git.dylib:arm64+0x44749c)
  #4 std::__1::__shared_ptr_emplace<lldb_private::StreamAsynchronousIO, std::__1::allocator<lldb_private::StreamAsynchronousIO> >::__on_zero_shared() shared_ptr.h:315 (liblldb.15.0.0git.dylib:arm64+0x3d0f80)
  #5 lldb_private::Debugger::HandleThreadEvent(std::__1::shared_ptr<lldb_private::Event> const&) Debugger.cpp:1591 (liblldb.15.0.0git.dylib:arm64+0x3ce2cc)
  #6 lldb_private::Debugger::DefaultEventHandler() Debugger.cpp:1659 (liblldb.15.0.0git.dylib:arm64+0x3ce720)
  #7 std::__1::__function::__func<lldb_private::Debugger::StartEventHandlerThread()::$_2, std::__1::allocator<lldb_private::Debugger::StartEventHandlerThread()::$_2>, void* ()>::operator()() function.h:352 (liblldb.15.0.0git.dylib:arm64+0x3d145c)
  #8 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) HostNativeThreadBase.cpp:62 (liblldb.15.0.0git.dylib:arm64+0x4c67a4)
  #9 lldb_private::HostThreadMacOSX::ThreadCreateTrampoline(void*) HostThreadMacOSX.mm:18 (liblldb.15.0.0git.dylib:arm64+0x29cebac)

Previous write of size 8 at 0x000106c01bc0 by main thread (mutexes: write M2754):
  #0 lldb_private::CommandInterpreter::PrintCommandOutput(lldb_private::Stream&, llvm::StringRef) CommandInterpreter.cpp:2992 (liblldb.15.0.0git.dylib:arm64+0x502888)
  #1 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp:3060 (liblldb.15.0.0git.dylib:arm64+0x502fec)
  #2 non-virtual thunk to lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp (liblldb.15.0.0git.dylib:arm64+0x503410)
  #3 lldb_private::IOHandlerEditline::Run() IOHandler.cpp (liblldb.15.0.0git.dylib:arm64+0x3ef2f4)
  #4 lldb_private::Debugger::RunIOHandlerSync(std::__1::shared_ptr<lldb_private::IOHandler> const&) Debugger.cpp:1037 (liblldb.15.0.0git.dylib:arm64+0x3cabc0)
  #5 lldb_private::CommandInterpreter::HandleCommandsFromFile(lldb_private::FileSpec&, lldb_private::CommandInterpreterRunOptions const&, lldb_private::CommandReturnObject&) CommandInterpreter.cpp:2748 (liblldb.15.0.0git.dylib:arm64+0x5000d4)
  #6 CommandObjectCommandsSource::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) CommandObjectCommands.cpp:187 (liblldb.15.0.0git.dylib:arm64+0x284d0b4)
  #7 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) CommandObject.cpp:998 (liblldb.15.0.0git.dylib:arm64+0x50bf04)
  #8 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&) CommandInterpreter.cpp:1981 (liblldb.15.0.0git.dylib:arm64+0x4feaf0)
  #9 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp:3049 (liblldb.15.0.0git.dylib:arm64+0x502ee0)
  #10 non-virtual thunk to lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp (liblldb.15.0.0git.dylib:arm64+0x503410)
  #11 lldb_private::IOHandlerEditline::Run() IOHandler.cpp (liblldb.15.0.0git.dylib:arm64+0x3ef2f4)
  #12 lldb_private::Debugger::RunIOHandlers() Debugger.cpp:1008 (liblldb.15.0.0git.dylib:arm64+0x3ca8f8)
  #13 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) CommandInterpreter.cpp:3299 (liblldb.15.0.0git.dylib:arm64+0x5048a4)
  #14 lldb::SBDebugger::RunCommandInterpreter(lldb::SBCommandInterpreterRunOptions const&) SBDebugger.cpp:1203 (liblldb.15.0.0git.dylib:arm64+0x523f8)
  #15 Driver::MainLoop() Driver.cpp:575 (lldb:arm64+0x10000606c)
  #16 main Driver.cpp:851 (lldb:arm64+0x100007388)

[JDevlieghere]

Rename utility class to ThreadSafeStream.

[jingham]

LGTM. It might be worthwhile to audit the uses of Debugger::GetErrorFile & GetOutputFile to see which of them really should also be using the lockable stream instead. That would be fine as a separate commit, however.

[JDevlieghere]

In D121500#3376497, @jingham wrote:

LGTM. It might be worthwhile to audit the uses of Debugger::GetErrorFile & GetOutputFile to see which of them really should also be using the lockable stream instead. That would be fine as a separate commit, however.

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

[labath]

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object. If we don't want to bother with that then, we might as well ditch the StreamAsynchronousIO completely.

In D121500#3376501, @JDevlieghere wrote:

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

[JDevlieghere]

In D121500#3377321, @labath wrote:

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

I'm happy to update the patch if you think this is the way to go. Or maybe you had something else in mind?

If we don't want to bother with that then, we might as well ditch the StreamAsynchronousIO completely.

In D121500#3376501, @JDevlieghere wrote:

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

I don't think this patch and using StreamAsynchronousIO for the progress updates is mutually exclusive. It sounds like the right thing to use for the progress events regardless, so I'll update D121502 to use that.

[JDevlieghere]

In D121500#3377434, @JDevlieghere wrote:

In D121500#3377321, @labath wrote:

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

https://reviews.llvm.org/D121537 shows what that would look like

[labath]

In D121500#3377434, @JDevlieghere wrote:

In D121500#3377321, @labath wrote:

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

I agree with your analysis. I haven't inspected the callers, but I'd say the right solution is to cut down on the number of places that call these functions. Most of the time, this is not the right function to call. We could try to make that more explicit, by adding some comments, or even reducing the scope/visibility of those functions.

In D121500#3376501, @JDevlieghere wrote:

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

I don't think this patch and using StreamAsynchronousIO for the progress updates is mutually exclusive.

Depends how you look at it. Without StreamAsynchronousIO, this becomes the ~only way to ensure thread safety. If we move everything to StreamAsynchronousIO, then this could still be useful as an extra line of defence, but one could also say that the lack of the mutex (and the resulting tsan errors) are an extra line of defence against people misusing the output streams. I would put myself in the second camp.

(btw, you did not update the setters in Debuggers::Set(Output|Error)File)

lldb/include/lldb/Utility/Stream.h

 #include "lldb/lldb-enumerations.h"
 #include "llvm/ADT/StringRef.h"
 #include "llvm/Support/FormatVariadic.h"
 #include "llvm/Support/raw_ostream.h"
 
 #include <cstdarg>
 #include <cstddef>
 #include <cstdint>
+#include <mutex>
 #include <type_traits>
 
 namespace lldb_private {
 
 /// \class Stream Stream.h "lldb/Utility/Stream.h"
 /// A stream class that can stream formatted output to a file.
 class Stream {
 public:
public:
   /// \example
   ///   ByteDelta delta(*this);
   ///   WriteDataToStream("foo");
   ///   return *delta;
   class ByteDelta {
     Stream *m_stream;
     /// Bytes we have written so far when ByteDelta was created.
     size_t m_start;
-
   public:
     ByteDelta(Stream &s) : m_stream(&s), m_start(s.GetWrittenBytes()) {}
     /// Returns the number of bytes written to the given Stream since this
     /// ByteDelta object was created.
     size_t operator*() const { return m_stream->GetWrittenBytes() - m_start; }
   };
 
   /// Construct with flags and address size and byte order.
public:
     RawOstreamForward(Stream &target, bool colors = false)
         : llvm::raw_ostream(/*unbuffered*/ true), m_target(target) {
       enable_colors(colors);
     }
   };
   RawOstreamForward m_forwarder;
 };
 
+template <typename T, typename M = std::mutex> class LockableStream : public T {
+public:
+  using T::T;
+  ~LockableStream() = default;
+
+  void Flush() override {
+    std::lock_guard<M> guard(m_mutex);
+    T::Flush();
+  }
+
+protected:
+  size_t WriteImpl(const void *s, size_t length) override {
+    std::lock_guard<M> guard(m_mutex);
+    return T::WriteImpl(s, length);
+  }
+
+private:
+  M m_mutex;
+};
+
 /// Output an address value to this stream.
 ///
 /// Put an address \a addr out to the stream with optional \a prefix and \a
 /// suffix strings.
 ///
 /// \param[in] s
 ///     The output stream.
 ///

lldb/source/Core/Debugger.cpp

ConstString Debugger::GetStaticBroadcasterClass() {
   static ConstString class_name("lldb.debugger");
   return class_name;
 }
 
 Debugger::Debugger(lldb::LogOutputCallback log_callback, void *baton)
     : UserID(g_unique_id++),
       Properties(std::make_shared<OptionValueProperties>()),
       m_input_file_sp(std::make_shared<NativeFile>(stdin, false)),
-      m_output_stream_sp(std::make_shared<StreamFile>(stdout, false)),
-      m_error_stream_sp(std::make_shared<StreamFile>(stderr, false)),
+      m_output_stream_sp(
+          std::make_shared<LockableStream<StreamFile>>(stdout, false)),
+      m_error_stream_sp(
+          std::make_shared<LockableStream<StreamFile>>(stderr, false)),
       m_input_recorder(nullptr),
       m_broadcaster_manager_sp(BroadcasterManager::MakeBroadcasterManager()),
       m_terminal_state(), m_target_list(*this), m_platform_list(),
       m_listener_sp(Listener::MakeListener("lldb.Debugger")),
       m_source_manager_up(), m_source_file_cache(),
       m_command_interpreter_up(
           std::make_unique<CommandInterpreter>(*this, false)),
       m_io_handler_stack(), m_instance_name(), m_loaded_plugins(),