This is an archive of the discontinued LLVM Phabricator instance.

[trace] [intel pt] Create a "process trace save" command
ClosedPublic

Authored by hanbingwang on Aug 6 2021, 2:48 PM.

Details

Summary

added new command "process trace save -d <directory>".
-it saves a JSON file as <directory>/trace.json, with the main properties of the trace session.
-it saves binary Intel-pt trace as <directory>/thread_id.trace; each file saves each thread.
-it saves modules to the directory <directory>/modules .
-it only works for live process and it only support Intel-pt right now.

Example:

b main
run
process trace start
n
process trace save -d /tmp/mytrace

A file named trace.json and xxx.trace should be generated in /tmp/mytrace. To load the trace that was just saved:

trace load /tmp/mytrace
thread trace dump instructions

You should see the instructions of the trace got printed.

To run a test:

cd ~/llvm-sand/build/Release/fbcode-x86_64/toolchain
ninja lldb-dotest
./bin/lldb-dotest -p TestTraceSave

Diff Detail

Event Timeline

hanbingwang created this revision.Aug 6 2021, 2:48 PM
hanbingwang requested review of this revision.Aug 6 2021, 2:48 PM
Herald added a project: Restricted Project. · View Herald TranscriptAug 6 2021, 2:48 PM
hanbingwang retitled this revision from Create "process trace save" command to [trace] [intel pt] Create a "process trace save" command.Aug 6 2021, 2:52 PM
hanbingwang edited the summary of this revision. (Show Details)
hanbingwang edited the summary of this revision. (Show Details)Aug 6 2021, 2:52 PM

fix a typo. change a string "the trace of ..." from "the trace or ..."

clayborg requested changes to this revision.Aug 9 2021, 6:45 PM
clayborg added inline comments.
lldb/include/lldb/Target/Trace.h
76

This class will already have a process in "Process *m_live_process" member variable, so no need to pass in the process. This will be NULL if we loaded a trace from disk already, but that should be ok because it was already loaded from something on disk.

lldb/source/Commands/CommandObjectProcess.cpp
1690–1696
1711

No need for the process as mentioned in above inline comment.

lldb/source/Plugins/Trace/intel-pt/IntelPTDecoder.cpp
267 ↗(On Diff #364893)

llvm::ArrayRef

297 ↗(On Diff #364893)

llvm::ArrayRef

lldb/source/Plugins/Trace/intel-pt/IntelPTDecoder.h
40 ↗(On Diff #364893)

This data is quite large, can we just get a reference to the data without making a copy by using llvm::ArrayRef?

70 ↗(On Diff #364893)

llvm::ArrayRef?

88 ↗(On Diff #364893)

llvm::ArrayRef

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp
72

remove process_sp and use "m_live_process"

78

llvm::ArrayRef

lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.h
26

remove process

28

llvm::ArrayRef

This revision now requires changes to proceed.Aug 9 2021, 6:45 PM
wallace added inline comments.Aug 9 2021, 7:03 PM
lldb/include/lldb/Target/Trace.h
76

Fair enough. Then better rename this method to SaveLiveProcessTraceToDisk or SaveLiveTraceToDisk, and make a comment mentioning that this will return an error if this is not tracing a live process.

lldb/source/Plugins/Trace/intel-pt/IntelPTDecoder.h
36 ↗(On Diff #364893)

thr -> the

40 ↗(On Diff #364893)

Sadly this can't work. LiveThreadDecoder::GetRawTrace() gets the raw trace from lldb-server and puts it in a vector without any object owning it. So the actual vector will need to be returned because no one wants to own it. It's not that bad though, because Expected just moves the inner data without creating copies.

lldb/test/API/commands/trace/TestTraceSave.py
39–44

use self.getBuildDir() instead of self.getSourceDir() for the files were you will store data

70–71

you don't need to do this. Every time the test runs all the previous data will be wiped out

wallace added inline comments.Aug 9 2021, 7:38 PM
lldb/source/Plugins/Trace/intel-pt/IntelPTDecoder.cpp
267–281 ↗(On Diff #364893)

Given that now we will just make it work for live processes, you can revert these changes and just invoke TraceIntelPT::GetLiveThreadBuffer directly

277 ↗(On Diff #364893)

result.reserve(trace_data.size())

lldb/source/Plugins/Trace/intel-pt/IntelPTDecoder.h
40 ↗(On Diff #364893)

Another option is to store the raw trace in the ThreadDecoder itself, though might not be that useful because decoding happens only once and saving to disk as well.

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.cpp
2

this shuold have the same length as line 7

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionFileSaver.cpp
33–48 ↗(On Diff #364893)

remove braces for one-line if statements

50 ↗(On Diff #364893)

remove the word struct

65–66 ↗(On Diff #364893)

for checking errors while writing, do this:

os << out;
os.close();
if (!os)
  return createStringError(unconvertibleErrorCode(), formatv("couldn't write to the file {0}", trace_path.getPath());
return Error::success();

then you have to make this method return llvm::Error

72–74 ↗(On Diff #364893)

remove braces

76–82 ↗(On Diff #364893)

remove the keyword struct. It's very old-style

93–95 ↗(On Diff #364893)

remove braces for one-liners

99 ↗(On Diff #364893)

ditto

102 ↗(On Diff #364893)

invoke BuildModulesSection(process_sp) here directly to avoid an unnecessary copy

wallace added inline comments.Aug 9 2021, 7:38 PM
lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionFileSaver.cpp
76 ↗(On Diff #364893)

it'd be better to create a constructor of JSONTraceIntelPTCPUInfo that receives a pt_cpu object and then does this conversion inside

118–119 ↗(On Diff #364893)

json_path is a bad name, call it raw_trace_path

126–128 ↗(On Diff #364893)

braces

132 ↗(On Diff #364893)

do the same thing here of closing the stream and then checking if it's correct

147–163 ↗(On Diff #364893)

remove braces for one-liners

164 ↗(On Diff #364893)

you should copy the file to the directory as well. The reason is that if the original file changes, then decoding will fail, so we need to make this self-contained.
You can use the function https://llvm.org/doxygen/namespacellvm_1_1sys_1_1fs.html#abe768b38d21bfc2bc91a1c1d09cd84de to copy files

You can follow this idea:

If a module exists in the path /usr/lib/foo/libbar.so, then you should copy it to <directory>/modules/usr/lib/foo/libbar.so.

Then the "path" entry is the json file should be "modules/usr/lib/foo/libbar.so", and the "systemPath" entry should be "/usr/lib/foo/libbar.so"

Then, update all the comments mentioning that the original modules will be copied over to the <directory/modules> folder with the goal of making it self-contained. A cool benefit if this is that you could zip this folder, send it over to another person, and they could see the same exact trace as you.

lldb/test/API/commands/trace/TestTraceSave.py
11

nice test

35–37

now that we are changing to live processes, this won't work. Instead, trace one of the existing binary files in the test folder, run it, get the first and last 20 instructions, save the contents in strings, then save the trace, then load it and dump the instructions again, finally assert that the outputs are the same

wallace added inline comments.Aug 9 2021, 10:12 PM
lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionFileSaver.cpp
19 ↗(On Diff #364893)

this doesn't seem relevant

22 ↗(On Diff #364893)

leave a black line before this

31 ↗(On Diff #364893)

Better rename this class to TraceIntelPTSessionSaver, because this is not just working with a file, it's instead dealing with a directory of many files

56 ↗(On Diff #364893)

To make these useful functions more generic so that they can be reused by other Trace technologies, you can create a file TraceSessionSaver.h/cpp in Plugins/Trace/common.
Then you can move this function to that file an call it WriteTraceSession(llvm::Value session_json, FileSpec directory)

87 ↗(On Diff #364893)

you could move BuildProcessesSection, BuildThreadsSection and BuildModulesSection to TraceSessionSaver.h/cpp if you provide a callback that receives a thread_id and returns a raw trace. Then almost all the code would be reusable by other Trace plug-ins

hanbingwang marked 36 inline comments as done.Aug 11 2021, 5:38 PM
hanbingwang marked an inline comment as done.

*trace.h:
-rename SaveToDisk() to SaveLiveTraceToDisk()

*IntelPTDecoder.h, IntelPTDecoder.cpp:
-removed GetRawTrace()

*TraceIntelPT.h, TraceIntelPT.cpp:
-removed GetThreadBuffer()
-in function SaveToDisk(), replace "process_sp" with "m_live_process"

*TraceIntelPTSessionFileSaver.h, TraceIntelPTSessionFileSaver.cpp:
-renamed to TraceIntelPTSessionSaver.h and TraceIntelPTSessionSaver.cpp
-the function BuildModulesSection() copies modules inside <directory>/modules
-create new constructors for struct JSONTraceIntelPTCPUInfo
-error checking for writing stream to file

hanbingwang added inline comments.Aug 11 2021, 6:03 PM
lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionFileSaver.cpp
87 ↗(On Diff #364893)

Sorry I did not make changes here. A bit unsure how to "provide a callback that receives a thread_id and returns a raw trace" and how would it work?

lldb/test/API/commands/trace/TestTraceSave.py
35–37

I'm not sure how to "save the contents in strings"?

wallace requested changes to this revision.Aug 11 2021, 10:22 PM

discussed offline of some improvements to make

This revision now requires changes to proceed.Aug 11 2021, 10:22 PM
hanbingwang edited the summary of this revision. (Show Details)

*TraceSessionSaver.h, TraceSessionSaver.cpp:
-move BuildModulesSection(), BuildThreadsSection(), BuildProcessesSection(), WriteSessionToFile() from TraceIntelPTSessionSaver.cpp to this file. These functions are generic for all trace plug-ins.

*TestTraceSave.py:
-new test case for "process trace save" command

*TraceIntelPT.h, Trace.h

  • change IsTraced(const Thread &thread) to IsTraced(lldb::tid_t tid)

*TraceIntelPTJSONStructs.h
-rename JSONTraceIntelPTSchema to JSONTraceIntelPTSession

hanbingwang edited the summary of this revision. (Show Details)Aug 12 2021, 5:42 PM
wallace requested changes to this revision.Aug 13 2021, 11:22 PM

Almost there. Many cosmetic changes but the logic is all good.

lldb/include/lldb/Target/Trace.h
176–177

The id of the thread in question

lldb/source/Commands/Options.td
747

remove this line

756

remove this line

lldb/source/Plugins/Trace/common/TraceJSONStructs.cpp
29–30

merge into one single line

46–48

delete braces

52–54

same here

62–64

ditto

lldb/source/Plugins/Trace/common/TraceJSONStructs.h
29–34

you can omit these constructors because they don't have anything custom

43–47

same here, just delete these lines

53–58

similarly, delete these

70–73

delete

lldb/source/Plugins/Trace/common/TraceSessionSaver.cpp
28

trace_session_description

35

move line 30 here to avoid that unnecessary out variable

51

json_session_description

59–61

delete braces

78

rename this to json_threads

87

avoid this. Instead of creating first an empty json_thread with invalid values, wait until this moment to create a valid JSONThread

json_threads.push_back(JSONThread { thread_sp->GetID(), raw_trace_path.GetPath().c_str() });

that way all the code that you write is valid at all times

95–97

delete braces

115

json_modules

123–125

delete braces

128–130

delete braces

lldb/source/Plugins/Trace/common/TraceSessionSaver.h
20

Save the trace session description JSON object inside the given directory as a file named \a trace.json.

23

trace_session_description is more readable. Reading 'jo' is not intuitive and confuses a bit

32

delete this line

34

trace_session_description_json

37

Build the processes section of the trace session description file. Besides returning the processes information, this method saves to disk all modules and raw traces corresponding to the traced threads of the given process.

43–46
50

The directory where files will be saved when building the processes section.

52–54
/// \return
///      The processes section or an error in case of failures.
54

delete

62

Build the threads ... of the trace session description file.

63–64

For each traced thread, its raw trace is also written to the file thread_id_.trace inside of the given directory.

70–73

copy the comment from above

76–77

The directory where files will be saved when building the threads section.

79–80

You are making a lot of mistakes at formatting the comments. Make sure to look for existing examples as reference.

/// \return
///      The threads section or an error in case of failures.
81

delete

90

Build the modules sub-section of the trace session description file.

93

Copying the modules has the benefit of making these trace session directories self-contained, as the raw traces and modules are part of the output directory and can be sent to another machine, where lldb can load them and replicate exactly the same trace session.

107–109
lldb/source/Plugins/Trace/intel-pt/TraceIntelPT.cpp
75

Saving a trace requires...

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTJSONStructs.h
36–39

delete

46–50

delete

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionSaver.cpp
61–62

Use {} constructors

JSONTraceIntelPTSession json_trace_intel_pt_session {
      json_trace_intel_pt_trace.get(), json_trace_session_base.get() };

that way you don't need to specify your own constructor

lldb/source/Plugins/Trace/intel-pt/TraceIntelPTSessionSaver.h
32–34

you don't need to pass this variable. You should be able to get the live_process directly from trace_ipt. You can create a method in TraceIntelPT for this if needed

41–55

in the first return you are using the correct formatting, but in the second you don't. Try to be more careful

44

delete

56

delete

lldb/test/API/commands/trace/TestTraceSave.py
39–47

this a little bit weird, you can instead place a breakpoint at the line that you want and stop just there

76

Also add a test in which you pass an invalid directory

This revision now requires changes to proceed.Aug 13 2021, 11:22 PM
hanbingwang marked 51 inline comments as done.Aug 16 2021, 10:35 AM

*TraceIntelPT.h, TraceIntelPT.cpp:
-add new function GetLiveProcess()
*TraceIntelPTSessionSaver.h, TraceIntelPTSessionSaver.cpp

  • the function SaveToDisk() no longer requires "m_live_process" as input param.

*TestTraceSave.py:
-added new test cases.

hanbingwang marked an inline comment as not done.

merge into one line: "json_module["loadAddress"] = oss.str();"

wallace accepted this revision.Aug 16 2021, 2:02 PM

good job

This revision was not accepted when it landed; it landed in state Needs Review.Aug 27 2021, 9:34 AM
This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.