This implements the interactive trace start and stop methods.
This diff ended up being much larger than I anticipated because, by doing it, I found that I had implemented in the beginning many things in a non optimal way. In any case, the code is much better now.
There's a lot of boilerplate code due to the gdb-remote protocol, but the main changes are:
- New tracing packets: jLLDBTraceStop, jLLDBTraceStart, jLLDBTraceGetBinaryData. The gdb-remote packet definitions are quite comprehensive.
- Implementation of the "process trace start|stop" and "thread trace start|stop" commands.
- Implementation of an API in Trace.h to interact with live traces.
- Created an IntelPTDecoder for live threads, that use the debugger's stop id as checkpoint for its internal cache.
- Added a functionality to stop the process in case "process tracing" is enabled and a new thread can't traced.
- Reworked how traces are manager in lldb-server, including the removal of the old Intel code
- Renamed a bunch of things for consistency and better naming
- Added tests
I have some ideas to unify the code paths for post mortem and live threads, but I'll do that in another diff.
Example flow for a single threaded process:
(lldb) file /home/wallace/llvm-sand/external/llvm-project/lldb/test/API/commands/trace/intelpt-trace/a.out Current executable set to '/home/wallace/llvm-sand/external/llvm-project/lldb/test/API/commands/trace/intelpt-trace/a.out' (x86_64).
(lldb) b main
Breakpoint 1: where = a.out`main + 4 at main.cpp:2, address = 0x0000000000400511
(lldb) r
Process 1563054 launched: '/home/wallace/llvm-sand/external/llvm-project/lldb/test/API/commands/trace/intelpt-trace/a.out' (x86_64)
Process 1563054 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
frame #0: 0x0000000000400511 a.out`main at main.cpp:2
1 int main() {
-> 2 int ret = 0;
3
4 for (int i = 0; i < 4; i++)
5 ret ^= 1;
6
7 return ret;
(lldb) thread trace start
(lldb) b 7
Breakpoint 2: where = a.out`main + 34 at main.cpp:7, address = 0x000000000040052f
(lldb) c
Process 1563054 resuming
Process 1563054 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 2.1
frame #0: 0x000000000040052f a.out`main at main.cpp:7
4 for (int i = 0; i < 4; i++)
5 ret ^= 1;
6
-> 7 return ret;
8 }
(lldb) thread trace dump instructions
thread #1: tid = 1563054, total instructions = 21
a.out`main + 11 at main.cpp:4
[ 1] 0x0000000000400518 movl $0x0, -0x8(%rbp)
[ 2] 0x000000000040051f jmp 0x400529 ; <+28> at main.cpp:4
a.out`main + 28 at main.cpp:4
[ 3] 0x0000000000400529 cmpl $0x3, -0x8(%rbp)
[ 4] 0x000000000040052d jle 0x400521 ; <+20> at main.cpp:5
...
a.out`main + 28 at main.cpp:4
[19] 0x0000000000400529 cmpl $0x3, -0x8(%rbp)
[20] 0x000000000040052d jle 0x400521 ; <+20> at main.cpp:5Example flow for a multi-threaded process with "all" tracing:
(lldb) file ~/llvm-sand/build/Release/Linux-x86_64/llvm/lldb-test-build.noindex/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.testStartMultipleLiveThreadsWithAllStops/a.out
Current executable set to '/home/wallace/llvm-sand/build/Release/Linux-x86_64/llvm/lldb-test-build.noindex/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.testStartMultipleLiveThreadsWithAllStops/a.out' (x86_64).
(lldb) b main
Breakpoint 1: where = a.out`main + 15 at main.cpp:17:15, address = 0x0000000000400a6f
(lldb) b 6
Breakpoint 2: where = a.out`f3() + 4 at main.cpp:6:5, address = 0x00000000004009f4
(lldb) b 11
Breakpoint 3: where = a.out`f2() + 8 at main.cpp:11:5, address = 0x0000000000400a08
(lldb) r
Process 4030831 launched: '/home/wallace/llvm-sand/build/Release/Linux-x86_64/llvm/lldb-test-build.noindex/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.testStartMultipleLiveThreadsWithAllStops/a.out' (x86_64)
Process 4030831 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
frame #0: 0x0000000000400a6f a.out`main at main.cpp:17:15
14 }
15
16 int main() { // main
-> 17 std::thread t2(f2);
18 t2.join();
19 return 0;
20 }
(lldb) process trace start
(lldb) c
Process 4030831 resuming
Process 4030831 stopped
* thread #2, name = 'a.out', stop reason = breakpoint 3.1
frame #0: 0x0000000000400a08 a.out`f2() at main.cpp:11:5
8
9 void f2() {
10 int n;
-> 11 n = 1; // thread 2
12 std::thread t3(f3);
13 t3.join();
14 }
(lldb) thread trace dump instructions -c 5
thread #2: tid = 4032721, total instructions = 211
a.out`void (*&&std::forward<void (*)()>(std::remove_reference<void (*)()>::type&))() + 13 at move.h:75:7
[206] 0x0000000000400f5d retq
a.out`void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) + 21 at invoke.h:60:14
[207] 0x00000000004010b5 callq *(%rax)
a.out`f2() at main.cpp:9
[208] 0x0000000000400a00 pushq %rbp
[209] 0x0000000000400a01 movq %rsp, %rbp
[210] 0x0000000000400a04 subq $0x30, %rsp
(lldb) c
Process 4030831 resuming
Process 4030831 stopped
* thread #3, name = 'a.out', stop reason = breakpoint 2.1
frame #0: 0x00000000004009f4 a.out`f3() at main.cpp:6:5
3
4 void f3() {
5 int m;
-> 6 m = 2; // thread 3
7 }
8
9 void f2() {
(lldb) thread trace dump instructions -c 5
thread #3: tid = 4034431, total instructions = 210
a.out`void (*&&std::forward<void (*)()>(std::remove_reference<void (*)()>::type&))() + 12 at move.h:75:7
[205] 0x0000000000400f5c popq %rbp
[206] 0x0000000000400f5d retq
a.out`void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) + 21 at invoke.h:60:14
[207] 0x00000000004010b5 callq *(%rax)
a.out`f3() at main.cpp:4
[208] 0x00000000004009f0 pushq %rbp
[209] 0x00000000004009f1 movq %rsp, %rbpExample flow for a simple crash and its trace dump:
(lldb) file /tmp/test/a.out
Current executable set to '/tmp/test/a.out' (x86_64).
(lldb) b main
Breakpoint 1: where = a.out`main + 8 at main.cpp:6, address = 0x0000000000400845
(lldb) r
Process 4121521 launched: '/tmp/test/a.out' (x86_64)
Process 4121521 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
frame #0: 0x0000000000400845 a.out`main at main.cpp:6
3
4 int main() {
5 int x;
-> 6 cin >> x;
7 cout << 12 / x << endl;
8 return 0;
9 }
(lldb) process trace start
(lldb) c
Process 4121521 resuming
0
Process 4121521 stopped
* thread #1, name = 'a.out', stop reason = signal SIGFPE: integer divide by zero
frame #0: 0x000000000040085f a.out`main at main.cpp:7
4 int main() {
5 int x;
6 cin >> x;
-> 7 cout << 12 / x << endl;
8 return 0;
9 }
10
(lldb) thread trace dump instructions -c 5 thread #1: tid = 4121521, total instructions = 8704
libstdc++.so.6`std::istream::operator>>(int&) + 185
[8699] 0x00007ffff7b52ce9 popq %rbp
[8700] 0x00007ffff7b52cea retq
a.out`main + 25 at main.cpp:7
[8701] 0x0000000000400856 movl -0x4(%rbp), %ecx
[8702] 0x0000000000400859 movl $0xc, %eax
[8703] 0x000000000040085e cltdExample of a failure due to insufficient process buffer
(lldb) b main
Breakpoint 1: where = a.out`main + 15 at main.cpp:17:15, address = 0x0000000000400a6f
(lldb) r
Process 2098567 launched: '/home/wallace/llvm-sand/build/Debug/fbcode-x86_64/llvm/lldb-test-build.noindex/commands/trace/multiple-threads/TestTraceStartStopMultipleThreads.testStartMultipleLiveThreadsWithAllStops/a.out' (x86_64)
Process 2098567 stopped
* thread #1, name = 'a.out', stop reason = breakpoint 1.1
frame #0: 0x0000000000400a6f a.out`main at main.cpp:17:15
14 }
15
16 int main() { // main
-> 17 std::thread t2(f2);
18 t2.join();
19 return 0;
20 }
(lldb) process trace start -l 5000 (lldb) c
Process 2098567 resuming
Process 2098567 stopped
* thread #2, name = 'a.out', stop reason = Thread 2099442 can't be traced as the process trace size limit has been reached. Consider retracing with a higher limit.
frame #0: 0x00007ffff70cf931 libc.so.6`__clone + 49
libc.so.6`__clone:
-> 0x7ffff70cf931 <+49>: testq %rax, %rax
0x7ffff70cf934 <+52>: jl 0x7ffff70cf975 ; <+0>
0x7ffff70cf936 <+54>: je 0x7ffff70cf939 ; <+57>
0x7ffff70cf938 <+56>: retq
Do we want to return a list of types here in case we have more than one option?