[lldb/crashlog] Load inlined symbol into interactive crashlog

[mib]

Summary

Sometimes, crash reports come with inlined symbols. These provide the
exact stacktrace from the user binary.

However, when investigating a crash, it's very likely that the images related
to the crashed thread are not available on the debugging user system or
that the versions don't match. This causes interactive crashlog to show
a degraded backtrace in lldb.

This patch aims to address that issue, by parsing the inlined symbols
from the crash report and load them into lldb's target.

This patch is a follow-up to 27f27d1, focusing on inlined symbols
loading from legacy (non-json) crash reports.

To do so, it updates the stack frame regular expression to make the
capture groups more granular, to be able to extract the symbol name, the
offset and the source location if available, while making it more
maintainable.

So now, when parsing the crash report, we build a data structure
containing all the symbol information for each stackframe. Then, after
launching the scripted process for interactive mode, we write a JSON
symbol file for each module, only containing the symbols that it contains.

Finally, we load the json symbol file into lldb, before showing the user
the process status and backtrace.

rdar://97345586

Signed-off-by: Med Ismail Bennani <[e-mail]>

[mib]

lldb/examples/python/crashlog.py

552–557

Same problem here I guess: IIUC if pc = image['base'] + frame_offset, then I believe frame_offset is not the address of the beginning of the function, but rather an address in the middle of the function (either at a callsite, or at the crash site).

669

@kastiglione may be you have a better idea how to handle symbol + offset where + offset might be optional.

880

I'm not sure if I should subtract the offset from the frame_addr so the symbol is at the right address.

Currently, the symbol name is symbol + offset, so I didn't have to do anything subtraction.

[kastiglione]

lldb/examples/python/crashlog.py

669

symbol is always present?

[bulbazord]

lldb/examples/python/crashlog.py

540

is image not the same as json_image from a few lines above this?

552–557

Yeah so pc is definitely not the right address of the symbol. It should be pc - frame_offset right?

606–611

What is the difference between frame_ofs and frame_offset?

1175–1176

Are these indented too much?

1194–1195

Won't this string look like "couldn't import crash report inlined symbols for %s (%s)"? You could probably just do string concatenation instead.

[mib]

lldb/examples/python/crashlog.py

540

+1

606–611

frame_ofs is actually the frame symbol name. I guess I should rename this for more clarity.

Some stack frame might have a symbol name and a offset, other might know have a symbol name.

Now that I'm thinking about this, I'm not sure if a crash report could be already symbolicated (i.e. symbol:line:column). In that case I think it will be pretty tricky to get the symbol address from the source info.

669

Technically, in the past the offs (symbol + offset) was optional:

r'(?: +(.*))?'

So I guess symbol could be missing.

[kastiglione]

lldb/examples/python/crashlog.py

669

Breaking it down, there is:

1. a symbol (maybe)
2. the literal text " + " followed by a number (also maybe)

I'll start start with 2:

\+ (\d+)

For the symbol, .*, it should instead have at least a length of 1. I'd use + instead of *. And, it shouldn't be _any_ character. At the very least it should be non-space, which is \S.

To combine these at a high level it looks like:

(?:(<symbol>)(?:<offset>)?)?

Filling in these two with symbol='\S+' and offset=' \+ (\d+)', it becomes:

(?:(\S+)(?: \+ (\d+))?)?

Here's some python real session that minimally exercises this code:

% python
>>> import re
>>> pat = re.compile(r"(?:(\S+)(?: \+ (\d+))?)?")
>>> pat.match("func + 123").groups()
('func', '123')
>>> pat.match("func").groups()
('func', None)
>>> pat.match("").groups()
(None, None)
>>>

[JDevlieghere]

lldb/examples/python/crashlog.py

1187–1192

We shouldn't keep those temp files around once they've been loaded by LLDB. We should use a tempfile.NamedTemporaryFile for this with the UUID as part of its name pattern.

lldb/include/lldb/API/SBModuleSpec.h

78

We don't need the uuid_len, the string should be NULL terminated. That matches other FromString SB APIs.

lldb/source/API/SBModuleSpec.cpp

136

newline after LLDB_INSTRUMENT

[mib]

Address some of the comments.

[kastiglione]

lldb/examples/python/crashlog.py

669

(?:(?: +\+ +)([0-9]+))

Non-capturing groups are only needed when you use a quantifier, like ?, *, or +. These regex has two non-capturing groups that don't use a quantifier, which means you can remove the groups:

+\+ +([0-9]+)

[mib]

lldb/examples/python/crashlog.py

669

Thanks for the tips! Regarding replacing the .+ by \S+ doesn't work because the symbol name could have a space in it (for objc symbols).

I'll try to break down these regex in smaller ones to make it easier to debug in the future.

[mib]

Address feedbacks:

• Simplify and Improve regex
• Add test

[bulbazord]

Looks good! A few minor comments.

lldb/examples/python/crashlog.py

618–619

I think this comment better describes parse_thread rather than parse_asi_backtrace no?

624–625

Does it make sense for self.images to be a dictionary instead of a list? Something that maps image_name -> image_info. That way you could do something like self.images[frame_img_name].symbols[frame_symbol] = { ... } instead of iterating over every image (with appropriate error checking in the event that frame_img_name isn't in self.images yet).

629

if frame_offset is None, you're doing int(None) which will give you a TypeError. Above you can do something like frame_offset_value = 0 and in the if frame_offset: block you can do frame_offset_value = int(frame_offset).

688

typo: infor -> info

691

What does garbage mean in this case? I assume it's boilerplate of some kind.

917

921

Same situation here as above, frame_offset might be None and then you'll crash with a TypeError.

[mib]

lldb/examples/python/crashlog.py

618–619

+1, that's an oversight.

624–625

I thought about that but the problem is that JSONCrashLogParser uses image index and TextCrashLogParser uses image name as dictionary keys. It would be pretty convoluted to make the dictionary that is the same for both parser so I'd rather keep a list here.

629

Good point!

691

When I broke down the regex, I really to isolate the capture groups that we cared the most about.
By "garbage" I mean text that are required to parse the whole thing but that isn't captured.

[mib]

Address @bulbazord comments

[bulbazord]

Ok, looks good to me now. Thanks!

[kastiglione]

lldb/examples/python/crashlog.py

683

this is just "spaces", not "spaces and + sign"

685

this is "space and + sign" not "spaces"

690

: does not need to be escaped

[mib]

Address @kastiglione comments and reformat.

lldb/bindings/python/python-typemaps.swig

%typemap(argout) (char *dst_or_null, size_t dst_len) {
   $result = string.release();
   free($1);
 }
 
 
 // typemap for an outgoing buffer
 // See also SBEvent::SBEvent(uint32_t event, const char *cstr, uint32_t cstr_len).
 // Ditto for SBProcess::PutSTDIN(const char *src, size_t src_len).
+// Ditto for SBModuleSpec::SetUUIDFromString(const char *uuid, size_t uuid_len).
 %typemap(in) (const char *cstr, uint32_t cstr_len),
-             (const char *src, size_t src_len) {
+             (const char *src, size_t src_len),
+             (const char *uuid, size_t uuid_len) {
   if (PythonString::Check($input)) {
     PythonString str(PyRefType::Borrowed, $input);
     $1 = (char *)str.GetString().data();
     $2 = str.GetSize();
   } else if (PythonByteArray::Check($input)) {
     PythonByteArray bytearray(PyRefType::Borrowed, $input);
     $1 = (char *)bytearray.GetBytes().data();
     $2 = bytearray.GetSize();

lldb/examples/python/crashlog.py

 import os
 import platform
 import plistlib
 import re
 import shlex
 import string
 import subprocess
 import sys
+import tempfile
 import threading
 import time
 import uuid
 
 
 print_lock = threading.RLock()
 
 try:
def create(debugger, path, verbose):
             return parser
         else:
             return TextCrashLogParser(debugger, path, verbose)
 
     def __init__(self, debugger, path, verbose):
         self.path = os.path.expanduser(path)
         self.verbose = verbose
         self.crashlog = CrashLog(debugger, self.path, self.verbose)
+        self.symbol_data = {}
 
     @abc.abstractmethod
     def parse(self):
         pass
 
 
 class JSONCrashLogParser(CrashLogParser):
     @staticmethod
def parse_frames(self, thread, json_frames):
         for json_frame in json_frames:
             image_id = int(json_frame['imageIndex'])
             json_image = self.get_used_image(image_id)
             ident = json_image['name'] if 'name' in json_image else ''
             thread.add_ident(ident)
             if ident not in self.crashlog.idents:
                 self.crashlog.idents.append(ident)
 
+            image = self.get_used_image(image_id)
             frame_offset = int(json_frame['imageOffset'])
-            image_addr = self.get_used_image(image_id)['base']
-            pc = image_addr + frame_offset
+            pc = image['base'] + frame_offset
+
+            if 'symbol' in json_frame and pc != 0:
+                image_uuid = image['uuid']
+                if not image_uuid in self.symbol_data:
+                    self.symbol_data[image_uuid] = {
+                        "symbols" : list(),
+                        "uuid": image_uuid,
+                        "triple": None
+                    }
+                self.symbol_data[image_uuid]["symbols"].append({
+                    "name": json_frame['symbol'],
+                    "type": "code",
+                    "size": 0,
+                    "address": pc,
+                })
             thread.frames.append(self.crashlog.Frame(idx, pc, frame_offset))
 
             # on arm64 systems, if it jump through a null function pointer,
             # we end up at address 0 and the crash reporter unwinder
             # misses the frame that actually faulted.
             # But $lr can tell us where the last BL/BLR instruction used
             # was at, so insert that address as the caller stack frame.
             if idx == 0 and pc == 0 and "lr" in thread.registers:
class JSONCrashLogParser(CrashLogParser):
 
     def parse_asi_backtrace(self, thread, bt):
         for line in bt.split('\n'):
             frame_match = TextCrashLogParser.frame_regex.search(line)
             if not frame_match:
                 print("error: can't parse application specific backtrace.")
                 return False
 
+            if len(frame_match.groups()) == 4:
                 (frame_id, frame_img_name, frame_addr,
                     frame_ofs) = frame_match.groups()
+            else:
+                (frame_id, frame_img_name, frame_addr,
+                    frame_ofs, frame_symbol, frame_offset) = frame_match.groups()
 
             thread.add_ident(frame_img_name)
             if frame_img_name not in self.crashlog.idents:
                 self.crashlog.idents.append(frame_img_name)
             thread.frames.append(self.crashlog.Frame(int(frame_id), int(
                 frame_addr, 0), frame_ofs))
 
         return True
 
     def parse_app_specific_backtraces(self, json_app_specific_bts):
         for idx, backtrace in enumerate(json_app_specific_bts):
             thread = self.crashlog.Thread(idx, True)
             thread.queue = "Application Specific Backtrace"
             if self.parse_asi_backtrace(thread, backtrace):
                 self.crashlog.threads.append(thread)
 
     def parse_thread_registers(self, json_thread_state, prefix=None):
         registers = dict()
         for key, state in json_thread_state.items():
             if key == "rosetta":
                 registers.update(self.parse_thread_registers(state))
                 continue
             if key == "x":
                 gpr_dict = { str(idx) : reg for idx,reg in enumerate(state) }
                 registers.update(self.parse_thread_registers(gpr_dict, key))
                 continue

 
 class TextCrashLogParser(CrashLogParser):
     parent_process_regex = re.compile(r'^Parent Process:\s*(.*)\[(\d+)\]')
     thread_state_regex = re.compile(r'^Thread \d+ crashed with')
     thread_instrs_regex = re.compile(r'^Thread \d+ instruction stream')
     thread_regex = re.compile(r'^Thread (\d+).*:')
     app_backtrace_regex = re.compile(r'^Application Specific Backtrace (\d+).*:')
     version = r'\(.+\)|(?:arm|x86_)[0-9a-z]+'
-    frame_regex = re.compile(r'^(\d+)\s+'              # id
-                             r'(.+?)\s+'               # img_name
-                             r'(?:' +version+ r'\s+)?' # img_version
-                             r'(0x[0-9a-fA-F]{4,})'    # addr (4 chars or more)
-                             r'(?: +(.*))?'            # offs
+    frame_regex = re.compile(r'^(\d+)\s+'                           # id
+                             r'(.+?)\s+'                            # img_name
+                             r'(?:' +version+ r'\s+)?'              # img_version
+                             r'(0x[0-9a-fA-F]{4,}) +'               # addr (4 chars or more)
+                             r'((.*)(?:(?: +\+ +)([0-9]+))|[^\s]+)' # symbol + offset
                             )
     null_frame_regex = re.compile(r'^\d+\s+\?\?\?\s+0{4,} +')
     image_regex_uuid = re.compile(r'(0x[0-9a-fA-F]+)'          # img_lo
                                   r'\s+-\s+'                   #   -
                                   r'(0x[0-9a-fA-F]+)\s+'       # img_hi
                                   r'[+]?(.+?)\s+'              # img_name
                                   r'(?:(' +version+ r')\s+)?'  # img_version
                                   r'(?:<([-0-9a-fA-F]+)>\s+)?' # img_uuid
                                   r'(\?+|/.*)'                 # img_path
                                  )
     exception_type_regex = re.compile(r'^Exception Type:\s+(EXC_[A-Z_]+)(?:\s+\((.*)\))?')
     exception_codes_regex = re.compile(r'^Exception Codes:\s+(0x[0-9a-fA-F]+),\s*(0x[0-9a-fA-F]+)')
     exception_extra_regex = re.compile(r'^Exception\s+.*:\s+(.*)')
 
     def __init__(self, debugger, path, verbose):
         super().__init__(debugger, path, verbose)
         self.thread = None
         self.app_specific_backtrace = False
         self.parse_mode = CrashLogParseMode.NORMAL
         self.parsers = {
             CrashLogParseMode.NORMAL : self.parse_normal,
             CrashLogParseMode.THREAD : self.parse_thread,
             CrashLogParseMode.IMAGES : self.parse_images,
             CrashLogParseMode.THREGS : self.parse_thread_registers,
             CrashLogParseMode.SYSTEM : self.parse_system,
             CrashLogParseMode.INSTRS : self.parse_instructions,
         }
 
     def parse(self):
         with open(self.path,'r', encoding='utf-8') as f:
class TextCrashLogParser(CrashLogParser):
     def parse_thread(self, line):
         if line.startswith('Thread'):
             return
         if self.null_frame_regex.search(line):
             print('warning: thread parser ignored null-frame: "%s"' % line)
             return
         frame_match = self.frame_regex.search(line)
         if frame_match:
+            if len(frame_match.groups()) == 4:
+                # Get the image UUID from the frame image name.
                 (frame_id, frame_img_name, frame_addr,
                     frame_ofs) = frame_match.groups()
+            else:
+                (frame_id, frame_img_name, frame_addr,
+                    frame_ofs, frame_symbol, frame_offset) = frame_match.groups()
             ident = frame_img_name
             self.thread.add_ident(ident)
             if ident not in self.crashlog.idents:
                 self.crashlog.idents.append(ident)
+
+            # Because of the Textual Crash Report layout, the thread section
+            # gets parsed before the binary images section. This means we can
+            # get the image's UUID and triple right away.
+            # This is why we need to do a second pass when parsing the binary
+            # images section to update these fields.
+            if not frame_ofs or int(frame_addr) != 0:
+                if frame_img_name not in self.symbol_data:
+                    self.symbol_data[frame_img_name] = {
+                        "symbols" : list(),
+                        "uuid": uuid.UUID(int=0),
+                        "triple": None
+                    }
+                self.symbol_data[frame_img_name]['symbols'].append({
+                    "name": frame_ofs,
+                    "type": "code",
+                    "size": 0,
+                    "address": frame_addr,
+                })
+
             self.thread.frames.append(self.crashlog.Frame(int(frame_id), int(
                 frame_addr, 0), frame_ofs))
         else:
             print('error: frame regex failed for line: "%s"' % line)
 
     def parse_images(self, line):
         image_match = self.image_regex_uuid.search(line)
         if image_match:
             (img_lo, img_hi, img_name, img_version,
                 img_uuid, img_path) = image_match.groups()
+
+            # Now that we've parsed the image, we can update its symbol_data UUID
+            if img_name in self.symbol_data:
+                self.symbol_data[frame_img_name]['uuid'] = image_uuid
+
             image = self.crashlog.DarwinImage(int(img_lo, 0), int(img_hi, 0),
                                             img_name.strip(),
                                             img_version.strip()
                                             if img_version else "",
                                             uuid.UUID(img_uuid), img_path,
                                             self.verbose)
             self.crashlog.images.append(image)
         else:
             print("error: image regex failed for: %s" % line)
 
 
     def parse_thread_registers(self, line):
         # "r12: 0x00007fff6b5939c8  r13: 0x0000000007000006  r14: 0x0000000000002a03  r15: 0x0000000000000c00"
         reg_values = re.findall('([a-z0-9]+): (0x[0-9a-f]+)', line, re.I)
         for reg, value in reg_values:
             self.thread.registers[reg] = int(value, 16)
 
     def parse_system(self, line):
         self.crashlog.system_profile.append(line)
 
     def parse_instructions(self, line):
         pass
 
 
 def usage():
     print("Usage: lldb-symbolicate.py [-n name] executable-image")
     sys.exit(0)
 
 
 def save_crashlog(debugger, command, exe_ctx, result, dict):
     usage = "usage: %prog [options] <output-path>"
     description = '''Export the state of current target into a crashlog file'''
if crash_log.errors:
         for error in crash_log.errors:
             print(error)
 
 def load_crashlog_in_scripted_process(debugger, crash_log_file, options, result):
     crashlog_path = os.path.expanduser(crash_log_file)
     if not os.path.exists(crashlog_path):
         raise InteractiveCrashLogException("crashlog file %s does not exist" % crashlog_path)
 
-    crashlog = CrashLogParser.create(debugger, crashlog_path, False).parse()
+    crashlog_parser = CrashLogParser.create(debugger, crashlog_path, False)
+    crashlog = crashlog_parser.parse()
 
     target = lldb.SBTarget()
     # 1. Try to use the user-provided target
     if options.target_path:
         target = debugger.CreateTarget(options.target_path)
         if not target:
             raise InteractiveCrashLogException("couldn't create target provided by the user (%s)" % options.target_path)
 
def load_crashlog_in_scripted_process(debugger, crash_log_file, options, result):
     launch_info.SetScriptedProcessClassName("crashlog_scripted_process.CrashLogScriptedProcess")
     launch_info.SetScriptedProcessDictionary(structured_data)
     error = lldb.SBError()
     process = target.Launch(launch_info, error)
 
     if not process or error.Fail():
         raise InteractiveCrashLogException("couldn't launch Scripted Process", error)
 
+    tmp_dir = tempfile.TemporaryDirectory()
+    for image, symbol_data in crashlog_parser.symbol_data.items():
+        module_spec = lldb.SBModuleSpec()
+        if crashlog_parser.__class__ == JSONCrashLogParser:
+                u = uuid.UUID(image)
+                module_spec.SetUUIDFromString(u.hex)
+        else:
+            module_spec.SetObjectName(image)
+        module = target.FindModule(module_spec)
+        if module.IsValid():
+            if not symbol_data['triple']:
+                symbol_data["triple"] = module.GetTriple()
+        else:
+            print("Skipping loading inlined symbols for module %s" % image)
+            continue
+
+        symbol_file = os.path.join(tmp_dir.name, module.GetUUIDString() + ".json")
+        with open(symbol_file, 'w') as f:
+            f.write(json.dumps(symbol_data))
+
+        ci.HandleCommand("target symbols add -s '%s' %s" %
+                         (module.GetFileSpec().fullpath, symbol_file), result)
+        if not result.Succeeded():
+            raise InteractiveCrashLogException("couldn't import crash report \
+                                               inlined symbols for %s (%s)" %
+                                               (module.file.basename,
+                                                module.GetUUIDString()))
+
+
     if not options.skip_status:
         @contextlib.contextmanager
         def synchronous(debugger):
             async_state = debugger.GetAsync()
             debugger.SetAsync(False)
             try:
                 yield
             finally:

lldb/include/lldb/API/SBModuleSpec.h

public:
   const char *GetTriple();
 
   void SetTriple(const char *triple);
 
   const uint8_t *GetUUIDBytes();
 
   size_t GetUUIDLength();
 
+  bool SetUUIDFromString(const char *uuid, size_t uuid_len);
+
   bool SetUUIDBytes(const uint8_t *uuid, size_t uuid_len);
 
   bool GetDescription(lldb::SBStream &description);
 
 private:
   friend class SBModuleSpecList;
   friend class SBModule;
   friend class SBTarget;

lldb/include/lldb/API/SBTarget.h

#endif
   lldb::SBModule GetModuleAtIndex(uint32_t idx);
 
   bool RemoveModule(lldb::SBModule module);
 
   lldb::SBDebugger GetDebugger() const;
 
   lldb::SBModule FindModule(const lldb::SBFileSpec &file_spec);
 
+  lldb::SBModule FindModule(const lldb::SBModuleSpec &module_spec);
+
   /// Find compile units related to *this target and passed source
   /// file.
   ///
   /// \param[in] sb_file_spec
   ///     A lldb::SBFileSpec object that contains source file
   ///     specification.
   ///
   /// \return

lldb/source/API/SBModuleSpec.cpp

 }
 
 size_t SBModuleSpec::GetUUIDLength() {
   LLDB_INSTRUMENT_VA(this);
 
   return m_opaque_up->GetUUID().GetBytes().size();
 }
 
+bool SBModuleSpec::SetUUIDFromString(const char *uuid, size_t uuid_len) {
+  LLDB_INSTRUMENT_VA(this, uuid, uuid_len)
+  m_opaque_up->GetUUID().SetFromStringRef(llvm::StringRef(uuid, uuid_len));
+  return m_opaque_up->GetUUID().IsValid();
+}
+
 bool SBModuleSpec::SetUUIDBytes(const uint8_t *uuid, size_t uuid_len) {
   LLDB_INSTRUMENT_VA(this, uuid, uuid_len)
   m_opaque_up->GetUUID() = UUID(uuid, uuid_len);
   return m_opaque_up->GetUUID().IsValid();
 }
 
 bool SBModuleSpec::GetDescription(lldb::SBStream &description) {
   LLDB_INSTRUMENT_VA(this, description);

lldb/source/API/SBTarget.cpp

SBModule SBTarget::FindModule(const SBFileSpec &sb_file_spec) {
   if (target_sp && sb_file_spec.IsValid()) {
     ModuleSpec module_spec(*sb_file_spec);
     // The module list is thread safe, no need to lock
     sb_module.SetSP(target_sp->GetImages().FindFirstModule(module_spec));
   }
   return sb_module;
 }
 
+SBModule SBTarget::FindModule(const SBModuleSpec &sb_module_spec) {
+  LLDB_INSTRUMENT_VA(this, sb_module_spec);
+
+  SBModule sb_module;
+  TargetSP target_sp(GetSP());
+  if (target_sp && sb_module_spec.IsValid()) {
+    // The module list is thread safe, no need to lock
+    sb_module.SetSP(
+        target_sp->GetImages().FindFirstModule(*sb_module_spec.m_opaque_up));
+  }
+  return sb_module;
+}
+
 SBSymbolContextList SBTarget::FindCompileUnits(const SBFileSpec &sb_file_spec) {
   LLDB_INSTRUMENT_VA(this, sb_file_spec);
 
   SBSymbolContextList sb_sc_list;
   const TargetSP target_sp(GetSP());
   if (target_sp && sb_file_spec.IsValid())
     target_sp->GetImages().FindCompileUnits(*sb_file_spec, *sb_sc_list);
   return sb_sc_list;