This is an archive of the discontinued LLVM Phabricator instance.

[lldb] Protect the debugger's output and error stream
ClosedPublic

Authored by JDevlieghere on Mar 11 2022, 4:51 PM.

Details

Summary

This patch introduces a small utility called ThreadSafeStream to protect the debugger's output and error stream. Nothing prevents someone from bypassing this and going straight through the debugger's Get{Output,Error}File. I don't think we want to get rid of that, but at least now we have something that allows us to access these streams in a safe way.

This fixes a real world race between the main thread and the default event handler thread:

Write of size 8 at 0x000106c01bc0 by thread T1 (mutexes: write M4069, write M2753):
  #0 lldb_private::IOHandlerEditline::PrintAsync(lldb_private::Stream*, char const*, unsigned long) IOHandler.cpp:636 (liblldb.15.0.0git.dylib:arm64+0x3ef4a4)
  #1 lldb_private::IOHandlerStack::PrintAsync(lldb_private::Stream*, char const*, unsigned long) IOHandler.cpp:129 (liblldb.15.0.0git.dylib:arm64+0x3ec84c)
  #2 lldb_private::Debugger::PrintAsync(char const*, unsigned long, bool) Debugger.cpp:1074 (liblldb.15.0.0git.dylib:arm64+0x3cb03c)
  #3 lldb_private::StreamAsynchronousIO::~StreamAsynchronousIO() StreamAsynchronousIO.cpp:21 (liblldb.15.0.0git.dylib:arm64+0x44749c)
  #4 std::__1::__shared_ptr_emplace<lldb_private::StreamAsynchronousIO, std::__1::allocator<lldb_private::StreamAsynchronousIO> >::__on_zero_shared() shared_ptr.h:315 (liblldb.15.0.0git.dylib:arm64+0x3d0f80)
  #5 lldb_private::Debugger::HandleThreadEvent(std::__1::shared_ptr<lldb_private::Event> const&) Debugger.cpp:1591 (liblldb.15.0.0git.dylib:arm64+0x3ce2cc)
  #6 lldb_private::Debugger::DefaultEventHandler() Debugger.cpp:1659 (liblldb.15.0.0git.dylib:arm64+0x3ce720)
  #7 std::__1::__function::__func<lldb_private::Debugger::StartEventHandlerThread()::$_2, std::__1::allocator<lldb_private::Debugger::StartEventHandlerThread()::$_2>, void* ()>::operator()() function.h:352 (liblldb.15.0.0git.dylib:arm64+0x3d145c)
  #8 lldb_private::HostNativeThreadBase::ThreadCreateTrampoline(void*) HostNativeThreadBase.cpp:62 (liblldb.15.0.0git.dylib:arm64+0x4c67a4)
  #9 lldb_private::HostThreadMacOSX::ThreadCreateTrampoline(void*) HostThreadMacOSX.mm:18 (liblldb.15.0.0git.dylib:arm64+0x29cebac)

Previous write of size 8 at 0x000106c01bc0 by main thread (mutexes: write M2754):
  #0 lldb_private::CommandInterpreter::PrintCommandOutput(lldb_private::Stream&, llvm::StringRef) CommandInterpreter.cpp:2992 (liblldb.15.0.0git.dylib:arm64+0x502888)
  #1 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp:3060 (liblldb.15.0.0git.dylib:arm64+0x502fec)
  #2 non-virtual thunk to lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp (liblldb.15.0.0git.dylib:arm64+0x503410)
  #3 lldb_private::IOHandlerEditline::Run() IOHandler.cpp (liblldb.15.0.0git.dylib:arm64+0x3ef2f4)
  #4 lldb_private::Debugger::RunIOHandlerSync(std::__1::shared_ptr<lldb_private::IOHandler> const&) Debugger.cpp:1037 (liblldb.15.0.0git.dylib:arm64+0x3cabc0)
  #5 lldb_private::CommandInterpreter::HandleCommandsFromFile(lldb_private::FileSpec&, lldb_private::CommandInterpreterRunOptions const&, lldb_private::CommandReturnObject&) CommandInterpreter.cpp:2748 (liblldb.15.0.0git.dylib:arm64+0x5000d4)
  #6 CommandObjectCommandsSource::DoExecute(lldb_private::Args&, lldb_private::CommandReturnObject&) CommandObjectCommands.cpp:187 (liblldb.15.0.0git.dylib:arm64+0x284d0b4)
  #7 lldb_private::CommandObjectParsed::Execute(char const*, lldb_private::CommandReturnObject&) CommandObject.cpp:998 (liblldb.15.0.0git.dylib:arm64+0x50bf04)
  #8 lldb_private::CommandInterpreter::HandleCommand(char const*, lldb_private::LazyBool, lldb_private::CommandReturnObject&) CommandInterpreter.cpp:1981 (liblldb.15.0.0git.dylib:arm64+0x4feaf0)
  #9 lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp:3049 (liblldb.15.0.0git.dylib:arm64+0x502ee0)
  #10 non-virtual thunk to lldb_private::CommandInterpreter::IOHandlerInputComplete(lldb_private::IOHandler&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) CommandInterpreter.cpp (liblldb.15.0.0git.dylib:arm64+0x503410)
  #11 lldb_private::IOHandlerEditline::Run() IOHandler.cpp (liblldb.15.0.0git.dylib:arm64+0x3ef2f4)
  #12 lldb_private::Debugger::RunIOHandlers() Debugger.cpp:1008 (liblldb.15.0.0git.dylib:arm64+0x3ca8f8)
  #13 lldb_private::CommandInterpreter::RunCommandInterpreter(lldb_private::CommandInterpreterRunOptions&) CommandInterpreter.cpp:3299 (liblldb.15.0.0git.dylib:arm64+0x5048a4)
  #14 lldb::SBDebugger::RunCommandInterpreter(lldb::SBCommandInterpreterRunOptions const&) SBDebugger.cpp:1203 (liblldb.15.0.0git.dylib:arm64+0x523f8)
  #15 Driver::MainLoop() Driver.cpp:575 (lldb:arm64+0x10000606c)
  #16 main Driver.cpp:851 (lldb:arm64+0x100007388)

Diff Detail

Event Timeline

JDevlieghere created this revision.Mar 11 2022, 4:51 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 11 2022, 4:51 PM
JDevlieghere requested review of this revision.Mar 11 2022, 4:51 PM
JDevlieghere edited the summary of this revision. (Show Details)

Rename utility class to ThreadSafeStream.

jingham accepted this revision.Mar 11 2022, 5:40 PM

LGTM. It might be worthwhile to audit the uses of Debugger::GetErrorFile & GetOutputFile to see which of them really should also be using the lockable stream instead. That would be fine as a separate commit, however.

This revision is now accepted and ready to land.Mar 11 2022, 5:40 PM

LGTM. It might be worthwhile to audit the uses of Debugger::GetErrorFile & GetOutputFile to see which of them really should also be using the lockable stream instead. That would be fine as a separate commit, however.

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object. If we don't want to bother with that then, we might as well ditch the StreamAsynchronousIO completely.

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

JDevlieghere added a comment.EditedMar 12 2022, 1:55 PM

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

I'm happy to update the patch if you think this is the way to go. Or maybe you had something else in mind?

If we don't want to bother with that then, we might as well ditch the StreamAsynchronousIO completely.

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

I don't think this patch and using StreamAsynchronousIO for the progress updates is mutually exclusive. It sounds like the right thing to use for the progress events regardless, so I'll update D121502 to use that.

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

https://reviews.llvm.org/D121537 shows what that would look like

Are you sure we don't want to handle this at a higher level? The way I understand it, the main reason for the existence of PrintAsync and StreamAsynchronousIO machinery is to provide precise control about when the various bits of output get printed. That's why the asynchronous output gets plumbed through the topmost iohandler and everything. In that setup, I think the right solution would be to set up some synchronization inside/near the iohandler object.

Yes, I considered doing the synchronization in the IO handler. We could have the IOHandler class hold onto a mutex. Locking it in PrintAsync takes care of the asynchronous output from the event thread. But everyone else is using the File, StreamFile, FILE* or file descriptor directly, so you need to hand out the mutex every time, in order to avoid colliding with the asynchronous output. That's certainly doable, but how long before someone forgets to lock the mutex? The benefit of doing it at a lower level is that as long as you're using the Stream, you're (somewhat) safe. Somewhat because you could still run into trouble when doing something that results in multiple calls to Stream::WriteImpl. The disadvantage is that we're still in the same boat for whoever is using the other accessors. I guess it's even slightly worse now, because there's no way to get the mutex.

I agree with your analysis. I haven't inspected the callers, but I'd say the right solution is to cut down on the number of places that call these functions. Most of the time, this is not the right function to call. We could try to make that more explicit, by adding some comments, or even reducing the scope/visibility of those functions.

Yup, that's exactly what I started doing. I'm aware of at least one offender (https://reviews.llvm.org/D121502) because I just added that. But I'll see if there are other places that can be migrated to use the Stream instead of the File.

Yeah, and I actually came very close to bringing that up (that it should be using StreamAsynchronousIO), but the patch was already submitted, and I did not want to bother with that. But if we're going to start introducing new concepts to make that work, then my calculus changes...

I don't think this patch and using StreamAsynchronousIO for the progress updates is mutually exclusive.

Depends how you look at it. Without StreamAsynchronousIO, this becomes the ~only way to ensure thread safety. If we move everything to StreamAsynchronousIO, then this could still be useful as an extra line of defence, but one could also say that the lack of the mutex (and the resulting tsan errors) are an extra line of defence against people misusing the output streams. I would put myself in the second camp.

(btw, you did not update the setters in Debuggers::Set(Output|Error)File)

This revision was automatically updated to reflect the committed changes.
Herald added a project: Restricted Project. · View Herald TranscriptMar 15 2022, 9:33 AM