This is an archive of the discontinued LLVM Phabricator instance.

Improve LLDB prompt handling
ClosedPublic

Authored by labath on May 18 2015, 5:22 AM.

Details

Summary

There is an issue in lldb where the command prompt can appear at the wrong time. The partial fix
we have in for this is not working all the time and is introducing unnecessary delays. This
change does:

  • Change Process:SyncIOHandler to use integer start id's for synchronization to avoid it being confused by quick start-stop cycles. I picked this up from a suggested patch by Greg to lldb-dev.
  • coordinates printing of asynchronous text with the iohandlers. This is also based on a (different) Greg's patch, but I have added stronger synchronization to it to avoid races.

Together, these changes solve the prompt problem for me on linux (both with and without libedit).
I think they should behave similarly on Mac and FreeBSD and I think they will not make matters
worse for windows.

Diff Detail

Event Timeline

labath updated this revision to Diff 25955.May 18 2015, 5:22 AM
labath retitled this revision from to Improve LLDB prompt handling.
labath updated this object.
labath edited the test plan for this revision. (Show Details)
labath added reviewers: clayborg, emaste, zturner.
labath added a subscriber: Unknown Object (MLST).
emaste edited edge metadata.May 18 2015, 7:02 AM

Unfortunately I still get the prompt at the wrong time with the patch:

feynman% bin/lldb /bin/ls
(lldb) target create "/bin/ls"
Current executable set to '/bin/ls' (x86_64).
(lldb) b main
Breakpoint 1: where = ls`main + 33 at ls.c:163, address = 0x00000000004023f1
(lldb) run
Process 80336 launching
Process 80336 launched: '/bin/ls' (x86_64)
(lldb) Process 80336 stopped
* thread #1: tid = 103527, 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe3d0) + 33 at ls.c:163, stop reason = breakpoint 1.1
    frame #0: 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe3d0) + 33 at ls.c:163
   160  #ifdef COLORLS
   161          char termcapbuf[1024];  /* termcap definition buffer */
   162          char tcapbuf[512];      /* capability buffer */
-> 163          char *bp = tcapbuf;
   164  #endif
   165 
   166          (void)setlocale(LC_ALL, "");

dang. :(

Do you compile your LLDB with libedit support?

dang. :(

Do you compile your LLDB with libedit support?

Yes - I can try it without as well if you like, but I just noticed something.

I have seen prompt issues at different points in the debugee lifecycle in the past, and perhaps they are separate issues. There's the one I pasted above, where the prompt comes out between launched and stopped, and one where the prompt is emitted at the wrong time when stepping, stopping at breakpoints, etc. I haven't been able to reproduce the second case with this patch, only the one at startup.

Thus, I think it might be that this solves the hard part of the problem (with thread races). The remaining issue may well be different, and it feels like it will be fairly straightforward (and independent of libedit support).

Oops. I see what's the problem now. I have accidentally committed a line which hard-disables libedit (I used it for testing). I have put up a new version of this patch. Could you test this one please. I have been mostly focusing on the libedit case and I hope this one is nailed.

Getting the output right without libedit is harder. With libedit, I just erase the command line when I want to output text and then redraw it. Redrawing the prompt could be done i guess (not sure if it's windows-compatible though), but redrawing the user-entered text is hard. That's why I don't try to do that now and instead rely on the improved SyncIOHandler trick to make sure the prompt doesn't get drawn in the first place. It seems that is not working for some reason. The new version of this also includes a bunch of new logging statements. Could you also try this out without libedit once again with enabled logging ("log enable -f lldb.log lldb process") and send me the file. We need to figure out why SyncIOHandler is failing.

thanks,
pl

source/Commands/CommandObjectProcess.cpp
273

This should block until we install the process handler.

source/Core/IOHandler.cpp
390

Oops.

source/Target/Process.cpp
4441

This should install the process handler.

labath updated this revision to Diff 25969.May 18 2015, 8:23 AM
labath edited edge metadata.
  • Add missing line.
  • Add logging statements

Indeed, this appears to be fixed for me with libedit.

Log for the non-libedit failing case is here:
https://people.freebsd.org/~emaste/lldb/no-libedit-prompt-issue.log

created as follows:

feynman% bin/lldb /bin/ls
(lldb) target create "/bin/ls"
Current executable set to '/bin/ls' (x86_64).
(lldb) log enable -f no-libedit-prompt-issue.log lldb process
(lldb) b main
Breakpoint 1: where = ls`main + 33 at ls.c:163, address = 0x00000000004023f1
(lldb) run
Process 87568 launching
Process 87568 launched: '/bin/ls' (x86_64)
(lldb) Process 87568 stopped
* thread #1: tid = 104753, 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe588) + 33 at ls.c:163, stop reason = breakpoint 1.1
    frame #0: 0x00000000004023f1 ls`main(argc=1, argv=0x00007fffffffe588) + 33 at ls.c:163
   160  #ifdef COLORLS
   161          char termcapbuf[1024];  /* termcap definition buffer */
   162          char tcapbuf[512];      /* capability buffer */
-> 163          char *bp = tcapbuf;
   164  #endif
   165 
   166          (void)setlocale(LC_ALL, "");
quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y
feynman%
clayborg requested changes to this revision.May 18 2015, 10:09 AM
clayborg edited edge metadata.

Looks mostly good. See my inline comments so we can avoid calling Activate() and calling Deactivate() on IOHandlers in the push and pop functions. Our Debugger::ExecuteIOHandlers() should handle those for us and Debugger::ExecuteIOHandlers() will do it at the right time.

source/Core/Debugger.cpp
1041

We shouldn't have to call Activate() here. Once the top IOHandler exits it's Run() function, the Debugger::ExecuteIOHandlers() will call it for us. See the code below:

void
Debugger::ExecuteIOHandlers()
{
    
    while (1)
    {
        IOHandlerSP reader_sp(m_input_reader_stack.Top());
        if (!reader_sp)
            break;

        reader_sp->Activate();
        reader_sp->Run();
        reader_sp->Deactivate();

        // Remove all input readers that are done from the top of the stack
        while (1)
        {
            IOHandlerSP top_reader_sp = m_input_reader_stack.Top();
            if (top_reader_sp && top_reader_sp->GetIsDone())
                m_input_reader_stack.Pop();
            else
                break;
        }
    }
    ClearIOHandlers();
}
1046–1049

We should probably just call Cancel and don't call Deactivate(). The Debugger::ExecuteIOHandlers() will deactivate the top IOHandler when it exits Run().

This revision now requires changes to proceed.May 18 2015, 10:09 AM
labath updated this revision to Diff 25980.May 18 2015, 10:10 AM
labath edited edge metadata.
  • Don't bump sync var when state is launching

Thanks for all the help Ed. The log file has helped me to trace the problem to the handling of the eStateLaunching state (which I guess is not used on linux). I have put up an updated version of this patch. Could you give it one more go?

This now works on FreeBSD with and without libedit.

source/Target/Process.cpp
4441

Oops!

ribrdb added a subscriber: ribrdb.May 18 2015, 6:11 PM
ribrdb added inline comments.
source/Target/Process.cpp
5256

TSAN shows a race here, because m_process_input_reader.reset() is not atomic and is called from a different thread.
This shows up in Process::Destroy(), so maybe that's a separate issue from what you're trying to fix with this change.

labath added inline comments.May 19 2015, 1:26 AM
source/Core/Debugger.cpp
1041

Note that I have removed the Activate/Deactivate calls from ExecuteIOHandlers. A more detailed explanation is below.

1046–1049

It's not clear to me what is the main purpose of the Activate/Deactivate functions. For example, for IOHandlerEditline, Deactivate() simply clears the m_active flag. IOHandlerEditline::Run() contains a while(m_active) { ... } loop. Therefore, it makes no sense to call Deactivate after Run() has exited, since the flag will be cleared at that point anyway. In fact, we have to (and we did) call Deactivate during pushing/popping to make sure that the Run function exits. I have moved the activation function here for symmetry. If you think they should be called from ExecuteIOHandlers, I can put them there, but then I will need to change the Cancel() function and have it clear the m_active flag also. What do you think?

source/Target/Process.cpp
5256

Thanks. This looks like a separate issue, so I will not fix it now, but I will definitely keep it in mind for the future. I have noticed a number of tsan issues in lldb during teardown and I would like to weed them out.

Friendly ping. :)

Greg, I need your opinion on the position of the Activate() calls (RunIOHandlers vs. Push/PopIOHandler).
The current implementation cannot be in RunIOHandlers as "m_active = true" in Activate() is racing with the Deactivate() in PopIOHandler. If Activate() happens after Deactivate(), the editline handler will get stuck in the Run() loop and the cancelation attempt will fail. Putting it in PushIOHandler avoids this as these are serialized by the input reader stack mutex. If we want the activation logic to be in RunIOHandlers, we will need to change the cancelation logic to make sure it does not race. What do you think?

clayborg accepted this revision.May 26 2015, 3:03 PM
clayborg edited edge metadata.

Lets try what you have an iterate on that. You seem to have it working well enough for your current needs. It would be great to get rid of SyncIOHandler eventually, but this improved version should be good for a first patch.

I question if we really need activate and deactivate at all. We should probably remove these if they cause problems and only allow the IOHandler::Run() to be the sole function. Then we can use Interrupt() or Cancel() when needed.

This revision is now accepted and ready to land.May 26 2015, 3:03 PM
This revision was automatically updated to reflect the committed changes.

I've committed this, and the buildbots seem to like it. I completely agree with you on the Activate/Deactivate issue and I will try to rig up a separate change to remove them.