@jingham, there may be some reconciliation needed between here and swift-lldb.

Looks reasonable to me.

This one needs more analysis.

I think this is a great approach. LGTM!

Thanks for working up the alternate solution, Mehdi!

LGTM! All tests passed on macOS 10.12.2 public beta with Xcode 8.1.

I'm going to test this one now, stacked on top of the final macOS-working version of D26698. Tell me now if you want it tested independently of D26698.

Here is the adjusted patch that fixes the issue I was seeing on TestTerminal.py. My first set of changes had a lifetime issue where I needed a const char* that was synthesized on the fly and went away by the time I needed it.

TestTerminal.py's test_launch_in_terminal() is timing out consistently with the current change + my diff attached above.

Hi Zachary,

I can give this one a run though on macOS in the morning.

I can run this through in the morning on macOS.

LGTM. Built and passed all existing tests.

I will give this a run a bit later this morning. Should be no later than early afternoon.

Looks fine to me pending that one question on the TSAN check (i.e. if the existing TSAN check is as good as your new ASAN check, LGTM).

Adding Jim, as I just want one of him or Greg to review.

Thanks, Jim!

Looks good, @beanz!

Adjusted web docs.

Making changes to the text...

Looks good, Tim!

@zturner wrote:
Can I have some background? What is the linking problem?

I'll let Greg comment on the public ABI expansion (i.e. including llvm of a specific version, which may differ from LLDB.framework clients that contain different versions of LLVM). My guess is this is not going to work for us, since we have long-lived frameworks shipped with Xcode that get used by clients where we cannot assume what they are or are not using. However, we could address that by adding the reverse flag, which would be "don't export LLVM", and have that be exported by default.

Looks reasonable.

In D26093#583358, @labath wrote:

The snippets you showed are pretty-much expected behavior. The backtrace that gets printed as a part of the log messages is coming from the backtrace(3) library, which has pretty limited backtracing capabilities -- it only looks at the symbols in the .dynsym section (because that's the only thing that is loaded into memory). I am pretty sure you would have problems backtracing if you compiled with -fomit-frame-pointer as well.

I think you fix of just making sure that all symbols show up in the .dynsym section would be fine, were it not for that fact that we have an inconsistent linking policy, which means you get wildly different links depending on whether you export something or not. If we get that straight that this would just be a size optimization (that was the reason we introduced it), with no impact on the behavior, that we could even consider turning off by default (I personally have never used the -S option of logging, but I can see how it could be useful in some workflows). I'll get back to the link policy in a separate email.

That said, you mentioned you also had some problems with setting breakpoints and stuff. Now, if this was true, that would be extremely worrying, but am not able to reproduce that on my side -- breakpoints everything else works fine. The reason for that is that lldb reads the .symtab section (if it is present, which it will be unless you strip the executable), and these linker options do not affect that section. If that is not the case, then we definitely need to figure that out (I suggest stepping through ObjectFileELF::GetSymtab() to get an initial idea of what is going on).

FWIW, this is how a debug session looks like for me. The backtrace() output has addresses only, but lldb is able to symbolicate it with no problem:

$ bin/lldb -- bin/lldb /bin/ls
(lldb) target create "bin/lldb"
Current executable set to 'bin/lldb' (x86_64).
(lldb) settings set -- target.run-args  "/bin/ls"
(lldb) br set -n Log::VAPrintf
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) pr la
Process 87438 launched: '/usr/local/google/home/labath/ll/build/dbg/bin/lldb' (x86_64)
1 location added to breakpoint 1
Process 87438 stopped and restarted: thread 1 received signal: SIGCHLD
(lldb) target create "/bin/ls"
Current executable set to '/bin/ls' (x86_64).
(lldb) log enable -S lldb process
(lldb) pr la
Process 87438 stopped
* thread #1: tid = 87438, 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s at least one of stdin/stdout/stderr was not set, evaluating default handling", args=0x00007fffffffc1a0) + 27 at Log.cpp:73, name = 'lldb', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s at least one of stdin/stdout/stderr was not set, evaluating default handling", args=0x00007fffffffc1a0) + 27 at Log.cpp:73
   70  	void Log::VAPrintf(const char *format, va_list args) {
   71  	  // Make a copy of our stream shared pointer in case someone disables our
   72  	  // log while we are logging and releases the stream
-> 73  	  StreamSP stream_sp(m_stream_sp);
   74  	  if (stream_sp) {
   75  	    static uint32_t g_sequence_id = 0;
   76  	    StreamString header;
(lldb) c
Process 87438 resuming
ProcessLaunchInfo::FinalizeFileActions at least one of stdin/stdout/stderr was not set, evaluating default handling
0  liblldb.so.40 0x00007ffff4d5a36f
1  liblldb.so.40 0x00007ffff0d1d49f
2  liblldb.so.40 0x00007ffff0d1cfbb
3  liblldb.so.40 0x00007ffff0fff4c4
4  liblldb.so.40 0x00007ffff103ad63
5  liblldb.so.40 0x00007ffff15c4f9f
6  liblldb.so.40 0x00007ffff0e33fe0
7  liblldb.so.40 0x00007ffff0e20317
8  liblldb.so.40 0x00007ffff0e25c0d
9  liblldb.so.40 0x00007ffff0e265a7
10 liblldb.so.40 0x00007ffff0cf5a60
11 liblldb.so.40 0x00007ffff0cb5c3c
12 liblldb.so.40 0x00007ffff0e26fdf
13 liblldb.so.40 0x00007ffff09d8d21 lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 129
14 lldb          0x0000000000407c15
15 lldb          0x0000000000408257
16 libc.so.6     0x00007fffeeeb8f45 __libc_start_main + 245
17 lldb          0x00000000004038d9
Process 87438 stopped
* thread #1: tid = 87438, 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s target stdin='%s', target stdout='%s', stderr='%s'", args=0x00007fffffffc1a0) + 27 at Log.cpp:73, name = 'lldb', stop reason = breakpoint 1.1
    frame #0: 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s target stdin='%s', target stdout='%s', stderr='%s'", args=0x00007fffffffc1a0) + 27 at Log.cpp:73
   70  	void Log::VAPrintf(const char *format, va_list args) {
   71  	  // Make a copy of our stream shared pointer in case someone disables our
   72  	  // log while we are logging and releases the stream
-> 73  	  StreamSP stream_sp(m_stream_sp);
   74  	  if (stream_sp) {
   75  	    static uint32_t g_sequence_id = 0;
   76  	    StreamString header;
(lldb) bt
* thread #1: tid = 87438, 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s target stdin='%s', target stdout='%s', stderr='%s'", args=0x00007fffffffc1a0) + 27 at Log.cpp:73, name = 'lldb', stop reason = breakpoint 1.1
  * frame #0: 0x00007ffff0d1cfeb liblldb.so.40`lldb_private::Log::VAPrintf(this=0x000000000059a560, format="ProcessLaunchInfo::%s target stdin='%s', target stdout='%s', stderr='%s'", args=0x00007fffffffc1a0) + 27 at Log.cpp:73
    frame #1: 0x00007ffff0d1cfbb liblldb.so.40`lldb_private::Log::Printf(this=0x000000000059a560, format="ProcessLaunchInfo::%s target stdin='%s', target stdout='%s', stderr='%s'") + 347 at Log.cpp:61
    frame #2: 0x00007ffff0fff970 liblldb.so.40`lldb_private::ProcessLaunchInfo::FinalizeFileActions(this=0x00000000004a0170, target=0x000000000058bea0, default_to_use_pty=true) + 1376 at ProcessLaunchInfo.cpp:260
    frame #3: 0x00007ffff103ad63 liblldb.so.40`lldb_private::Target::Launch(this=0x000000000058bea0, launch_info=0x00000000004a0170, stream=0x00007fffffffcaa8) + 1315 at Target.cpp:2835
    frame #4: 0x00007ffff15c4f9f liblldb.so.40`CommandObjectProcessLaunch::DoExecute(this=0x00000000004a0030, launch_args=0x00007fffffffcd68, result=0x00007fffffffd4d8) + 1647 at CommandObjectProcess.cpp:233
    frame #5: 0x00007ffff0e33fe0 liblldb.so.40`lldb_private::CommandObjectParsed::Execute(this=0x00000000004a0030, args_string="", result=0x00007fffffffd4d8) + 1344 at CommandObject.cpp:1008
    frame #6: 0x00007ffff0e20317 liblldb.so.40`lldb_private::CommandInterpreter::HandleCommand(this=0x0000000000482750, command_line="pr la", lazy_add_to_history=eLazyBoolCalculate, result=0x00007fffffffd4d8, override_context=0x0000000000000000, repeat_on_empty_command=true, no_context_switching=false) + 3527 at CommandInterpreter.cpp:1679
    frame #7: 0x00007ffff0e25c0d liblldb.so.40`lldb_private::CommandInterpreter::IOHandlerInputComplete(this=0x0000000000482750, io_handler=0x0000000000583190, line="pr la") + 301 at CommandInterpreter.cpp:2713
    frame #8: 0x00007ffff0e265a7 liblldb.so.40`non-virtual thunk to lldb_private::CommandInterpreter::IOHandlerInputComplete(this=0x0000000000482750, io_handler=0x0000000000583190, line="pr la") + 55 at CommandInterpreter.cpp:2689
    frame #9: 0x00007ffff0cf5a60 liblldb.so.40`lldb_private::IOHandlerEditline::Run(this=0x0000000000583190) + 608 at IOHandler.cpp:552
    frame #10: 0x00007ffff0cb5c3c liblldb.so.40`lldb_private::Debugger::ExecuteIOHandlers(this=0x0000000000480e90) + 140 at Debugger.cpp:899
    frame #11: 0x00007ffff0e26fdf liblldb.so.40`lldb_private::CommandInterpreter::RunCommandInterpreter(this=0x0000000000482750, auto_handle_events=true, spawn_thread=false, options=0x00007fffffffd7d8) + 223 at CommandInterpreter.cpp:2910
    frame #12: 0x00007ffff09d8d21 liblldb.so.40`lldb::SBDebugger::RunCommandInterpreter(this=0x00007fffffffdb48, auto_handle_events=true, spawn_thread=false) + 129 at SBDebugger.cpp:814
    frame #13: 0x0000000000407c15 lldb`Driver::MainLoop(this=0x00007fffffffdb28) + 4341 at Driver.cpp:1156
    frame #14: 0x0000000000408257 lldb`main(argc=2, argv=0x00007fffffffdd28) + 471 at Driver.cpp:1254
    frame #15: 0x00007fffeeeb8f45 libc.so.6`__libc_start_main(main=(lldb`main at Driver.cpp:1218), argc=2, argv=0x00007fffffffdd28, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=0x00007fffffffdd18) + 245 at libc-start.c:287
    frame #16: 0x00000000004038d9 lldb`_start + 41

Here's what you get with the same codebase, but using this cmake line instead:

cmake -GNinja ../llvm -DCMAKE_BUILD_TYPE=Debug -DLLDB_EXPORT_ALL_SYMBOLS:BOOL=YES

Could you elaborate on this?

Sure, I'll see if I can reproduce the lack of backtrace symbol lookup that I was experiencing without exporting those symbols back in roughly Oct/Nov 2015. We've been using this flag ever since to address it.

I've been debugging lldb without this switch and everything seems to work fine

That definitely was not my experience when I added the flag. I am seeing if the old scenario holds true on 16.04 x86_64 with the current LLDB codebase. It is possible that the issue we saw manifest has been addressed.) I will make a best effort attempt this weekend, but it may be more like Monday when I will have dedicated time to repro it. If it is no longer an issue, I'd consider this flag deprecated as its initial purpose is no longer valid. If it is still an issue, it could be a config difference between stock Ubuntu and your setup (as we know is at least the case with kernel settings).

(modulo problems evaluating fancy expressions, but I don't think that is related to this. Or is it?).

At the time the issue was symbol lookup - we were somewhere short-circuiting so that we didn't consider debug symbols properly. While I didn't try it at the time as I wasn't aware of it, it is possible the issue may be related to needing the -fno-limit-debug-info flag. When I get to reproducing the original issue, if it does surface, I will try adding that flag as a second step to see if that addresses it.

For expressions that need to evaluate any symbols we were failing to resolve, I could imagine that being at least one factor. Not sure though without a specific case to look at.

More later when I have some results.

In D26093#582825, @labath wrote:

In D26093#582618, @tfiala wrote:

If it unblocks something it might be ok, but it doesn't actually fix the problem.

It allows debugging of lldb with lldb on Ubuntu. This broke recently, and the CL here re-enables the ability to do that.

Could you elaborate on this?

In D26093#582659, @beanz wrote:

In D26093#582618, @tfiala wrote:

The current usage of CommandLine via global static constructors in Debug.cpp seems like a poor choice, as it forces all users of llvm with NDEBUG not defined to get some command line behavior that they may never use. Maybe some aspect of that can be made lazy?

It can't be made lazy without completely re-designing how cl::opt works. Which is something I've had lengthy discussions with Chandler about. Currently the biggest issue with fixing it relates to how the new pass manager works. Specifically in order to provide meaningful -help output we need a way to register command line options in passes, which we have today, but won't with the new pass manager.

It is an open problem that we're trying to come up with a solution to.

In D26093#582583, @beanz wrote:

This patch ends up hiding the problem, not fixing it.

I'm at a loss for the workflow to close this. I originally commandeered it to do an abort on it, but that didn't work. I'm going to abandon it, and see if I can kill it then.

The reason why it doesn't blow up in the hidden case is that liblldb.so has already internally resolved the storage, doesn't involve dynamic resolution, and has its own data location. Ditto for lldb-mi's copy. So they are separate islands.

In D26093#582412, @tfiala wrote:

In D26093#582395, @labath wrote:

I am a bit surprised that this fixes the problem. I would have expected that the problem would occur in case when we *do* restrict exports,

This matches my expectations. When I was using the flag the exported all symbols indiscriminately from liblldb.so, it meant that the llvm symbols for the copy inside liblldb.so were public and known. That public element allows the dynamic linker to coalesce the data storage locations with the ones for the llvm in lldb-mi. Hiding them (as per the normal case, when we only export the 'lldb' public namespace) prevents the dynamic linker from doing the coalesce.

In D26093#582395, @labath wrote:

I am a bit surprised that this fixes the problem. I would have expected that the problem would occur in case when we *do* restrict exports,

I am not that excited by this, but I don't see a much better way to achieve the result. :)
Possibly I'd consider making the variable name positive (instead of CFLAGS_NO_DEBUG, have a INCLUDES var, and then the test can use $(INCLUDES) $(TRIPLE_CFLAGS)).

Looks good.

LGTM.

@beanz and I discussed. This isn't needed here at all. The issue is entirely in the current manifestation of the GitHub side of swift-lldb.

(It would be good to wait for feedback from the others, though).

LGTM.

LGTM.

This seems fine, but @labath might want to weigh in if Android tools have a hard-baked assumption anywhere on lldb-launcher.

Yep, looks right. We shouldn't be assuming the shell.

LGTM.

Okay. I think we need a minor tweak, to drop the REQUIRED on the Backtrace usage.

LGTM.

(Retro) Ah nice, I like the generator expression you put in there.

(Retro) LGTM.

(Retro) LGTM.

(Retro) yep, this looked fine.

Looks good!

I also just added a test case to validate we get a non-None answer to test instance self.id() calls. We use it in several places, so might as well make that explicit.

Much better. I see you found test.id(), which gets the module.class.test_method setup. That will be unique.

In D24988#559314, @fjricci wrote:

For an example of something that couldn't be disabled with the original implementation, consider a test like:

CreateDuringStepTestCase.test_step_inst

Disabling by method name (test_step_inst) would also disable CreateDuringInstructionStepTestCase.test_step_inst.

That works fine.

In D25099#559701, @zturner wrote:

I know what this is. It should be fixed in this patch, I guess I didn't have the newest patch uploaded.

I'm getting one test crash (segfault) in logging/TestLogging.py:

I will test this on macOS. I will have the results this afternoon.

@zturner , Greg is out this week (and was last Friday as well).

What is the motivation for this change? It looks like the existing code works based on file names, which are required to be unique in the system. It looks like you're attempting to move it over to a classname.method scheme. Is that right? If so, classname.method is not guaranteed to be unique, either. It would have to include the module name, which is essentially the file name, which is required to be unique. (Maybe when you said <TestCase>.<TestMethod>, your <TestCase> element is fully qualified, including the module name? If so, then that should be guaranteed to be unique).

LGTM.

BTW, regarding this part: