[lldb] Use ObjectFileJSON to create modules for interactive crashlogs

[JDevlieghere]

Summary

Create an artificial module using a JSON object file when we can't locate the module and dSYM through dsymForUUID (or however locate_module_and_debug_symbols is implemented). By parsing the symbols from the crashlog and making them part of the JSON object file, LLDB can symbolicate frames it otherwise wouldn't be able to, as there is no module for it. For non-interactive crashlogs, that never was a problem because we could simply show the "pre-symbolicated" frame from the input. For interactive crashlogs, we need a way to pass the symbol information to LLDB so that it can symbolicate the frames, which is what motivated the ObjectFileJSON format.

[JDevlieghere]

Add missing CHECK line in skipped_status_interactive_crashlog.test.

[mib]

lldb/examples/python/crashlog.py

460–464

Why do we need this ?

515

What do we use idx for ?

526–527

Seems like we're doing the same things for both self.images and self.crashlog.images ... In that case, is self.images really necessary ?

lldb/examples/python/scripted_process/crashlog_scripted_process.py

25–45

Nice!

lldb/examples/python/symbolication.py

413

Fancy! Can't we just do this and remove the extra import ?

428–429

This should work, right ?

[mib]

lldb/examples/python/symbolication.py

425

Since this field is optional in the ObjectFileJSON, I think it should be included here. We should just initialize it to 0 in lldb if the user didn't provide it.

[JDevlieghere]

lldb/examples/python/crashlog.py

460–464

The parser now stores a list of images sorted by their index. We can't use the list of images in the crashlog (self.crashlog.images) because they have the main module at index 0.

515

You're right, this isn't necessary anymore.

526–527

Yes, see my previous comment.

lldb/examples/python/symbolication.py

428–429

Apparently not: values() returns a view object rather than a list: https://docs.python.org/3/library/stdtypes.html#dict-views

[JDevlieghere]

Address @mib's code review feedback

[mib]

lldb/examples/python/crashlog.py

515

I'm really not a big fan of having very similar image lists ... may be we could use the from the crashlog object and skip the first entry (since we know it's the main executable).
What do you think ?

555

Does this needs to be initialized ?

lldb/examples/python/symbolication.py

427–428

What about :

[mib]

lldb/examples/python/crashlog.py

515

Otherwise, we could hoist the main executable image from the image list and handle it separately.

[JDevlieghere]

lldb/examples/python/crashlog.py

515

I understand the concern. To be fair, I didn't check whether the main executable coming first is something we rely on, but I'm pretty sure we are: we'll need it to create the target. I didn't want to mess with that and risk introducing a bug that way. It took me quite some time to figure out this was an issue when parsing the symbol data. If we don't want to break that assumption, there's nothing more efficient than keeping a second list of references. I also think it makes sense to keep that in the JSON parser, because the index of (parsed) image is only something that makes sense for that format because it cross references images based on their index. That's not the case in the textual or parser crashlogs.

FWIW this is the code that moves the main image to the top, invalidating the image indexes of every image before it:

def set_main_image(self, identifier):
    for i, image in enumerate(self.images):
        if image.identifier == identifier:
            self.images.insert(0, self.images.pop(i))
            break

[mib]

LGTM!

lldb/examples/python/crashlog.py

515

Fair

lldb/examples/python/crashlog.py

 
 class CrashLog(symbolication.Symbolicator):
     class Thread:
         """Class that represents a thread in a darwin crash log"""
 
         def __init__(self, index, app_specific_backtrace):
             self.index = index
             self.id = index
+            self.images = list()
             self.frames = list()
             self.idents = list()
             self.registers = dict()
             self.reason = None
             self.name = None
             self.queue = None
             self.crashed = False
             self.app_specific_backtrace = app_specific_backtrace
def is_valid_json(path):
 
         with open(path, 'r', encoding='utf-8') as f:
             buffer = f.read()
         try:
             return parse_json(buffer)
         except:
             return None
 
+    def __init__(self, debugger, path, verbose):
+        super().__init__(debugger, path, verbose)
+        # List of DarwinImages sorted by their index.
+        self.images = list()
+
     def parse(self):
         try:
             self.parse_process_info(self.data)
             self.parse_images(self.data['usedImages'])
             self.parse_main_image(self.data)
             self.parse_threads(self.data['threads'])
             if 'asi' in self.data:
                 self.crashlog.asi = self.data['asi']
def parse_crash_reason(self, json_exception):
         elif 'subtype' in json_exception:
             exception_extra = " ({})".format(json_exception['subtype'])
         else:
             exception_extra = ""
         return "{}{}{}".format(exception_type, exception_signal,
                                   exception_extra)
 
     def parse_images(self, json_images):
-        idx = 0
-        for json_image in json_images:
+        for idx, json_image in enumerate(json_images):
             img_uuid = uuid.UUID(json_image['uuid'])
             low = int(json_image['base'])
             high = low + int(
                 json_image['size']) if 'size' in json_image else low
             name = json_image['name'] if 'name' in json_image else ''
             path = json_image['path'] if 'path' in json_image else ''
             version = ''
             darwin_image = self.crashlog.DarwinImage(low, high, name, version,
                                                      img_uuid, path,
                                                      self.verbose)
+            self.images.append(darwin_image)
             self.crashlog.images.append(darwin_image)
-            idx += 1
 
     def parse_main_image(self, json_data):
         if 'procName' in json_data:
             proc_name = json_data['procName']
             self.crashlog.set_main_image(proc_name)
 
     def parse_frames(self, thread, json_frames):
         idx = 0
         for json_frame in json_frames:
             image_id = int(json_frame['imageIndex'])
             json_image = self.get_used_image(image_id)
             ident = json_image['name'] if 'name' in json_image else ''
             thread.add_ident(ident)
             if ident not in self.crashlog.idents:
                 self.crashlog.idents.append(ident)
 
             frame_offset = int(json_frame['imageOffset'])
             image_addr = self.get_used_image(image_id)['base']
             pc = image_addr + frame_offset
+
+            if 'symbol' in json_frame:
+                symbol = json_frame['symbol']
+                location = int(json_frame['symbolLocation'])
+                image = self.images[image_id]
+                image.symbols[symbol] = {
+                    "name": symbol,
+                    "type": "code",
+                    "size": 0x0,
+                    "address": frame_offset - location
+                }
+
             thread.frames.append(self.crashlog.Frame(idx, pc, frame_offset))
 
             # on arm64 systems, if it jump through a null function pointer,
             # we end up at address 0 and the crash reporter unwinder
             # misses the frame that actually faulted.
             # But $lr can tell us where the last BL/BLR instruction used
             # was at, so insert that address as the caller stack frame.
             if idx == 0 and pc == 0 and "lr" in thread.registers:

lldb/examples/python/scripted_process/crashlog_scripted_process.py

def set_crashlog(self, crashlog):
         self.loaded_images = []
         self.exception = self.crashlog.exception
         self.app_specific_thread = None
         if hasattr(self.crashlog, 'asi'):
             self.metadata['asi'] = self.crashlog.asi
         if hasattr(self.crashlog, 'asb'):
             self.extended_thread_info = self.crashlog.asb
 
-        def load_images(self, images):
-            #TODO: Add to self.loaded_images and load images in lldb
-            if images:
-                for image in images:
-                    if image not in self.loaded_images:
-                        if image.uuid == uuid.UUID(int=0):
-                            continue
-                        err = image.add_module(self.target)
-                        if err:
-                            # Append to SBCommandReturnObject
-                            print(err)
-                        else:
-                            self.loaded_images.append(image)
+        if self.load_all_images:
+            for image in self.crashlog.images:
+                image.resolve = True
+        else:
+            for thread in self.crashlog.threads:
+                if thread.did_crash():
+                    for ident in thread.idents:
+                        for image in self.crashlog.find_images_with_identifier(ident):
+                            image.resolve = False
+
+        for image in self.crashlog.images:
 
         for thread in self.crashlog.threads:
-            if self.load_all_images:
-                load_images(self, self.crashlog.images)
-            elif thread.did_crash():
-                for ident in thread.idents:
-                    load_images(self, self.crashlog.find_images_with_identifier(ident))
-
             if hasattr(thread, 'app_specific_backtrace') and thread.app_specific_backtrace:
                 # We don't want to include the Application Specific Backtrace
                 # Thread into the Scripted Process' Thread list.
                 # Instead, we will try to extract the stackframe pcs from the
                 # backtrace and inject that as the extended thread info.
                 self.app_specific_thread = thread
                 continue
 

lldb/examples/python/symbolication.py

 import optparse
 import os
 import plistlib
 import re
 import shlex
 import sys
 import time
 import uuid
+import json
+import tempfile
+import pathlib
 
 
 class Address:
     """Class that represents an address that will be symbolicated"""
 
     def __init__(self, target, load_addr):
         self.target = target
         self.load_addr = load_addr  # The load address that this object represents

 
 
 class Image:
     """A class that represents an executable image and any associated data"""
 
     def __init__(self, path, uuid=None):
         self.path = path
         self.resolved_path = None
+        self.resolve = False
         self.resolved = False
         self.unavailable = False
         self.uuid = uuid
         self.section_infos = list()
         self.identifier = None
         self.version = None
         self.arch = None
         self.module = None
         self.symfile = None
         self.slide = None
+        self.symbols = dict()
 
     @classmethod
     def InitWithSBTargetAndSBModule(cls, target, module):
         '''Initialize this Image object with a module from a target.'''
         obj = cls(module.file.fullpath, module.uuid)
         obj.resolved_path = module.platform_file.fullpath
         obj.resolved = True
         for section in module.sections:
class Image:
     def add_module(self, target):
         '''Add the Image described in this object to "target" and load the sections if "load" is True.'''
         if target:
             # Try and find using UUID only first so that paths need not match
             # up
             uuid_str = self.get_normalized_uuid_string()
             if uuid_str:
                 self.module = target.AddModule(None, None, uuid_str)
-            if not self.module:
+            if not self.module and self.resolve:
                 self.locate_module_and_debug_symbols()
                 if self.unavailable:
                     return None
                 resolved_path = self.get_resolved_path()
                 self.module = target.AddModule(
                     resolved_path, None, uuid_str, self.symfile)
+            if not self.module and self.section_infos:
+                name = pathlib.Path(self.path).name
+                with tempfile.NamedTemporaryFile(suffix='.' + name) as tf:
+                    data = {
+                        'triple': target.triple,
+                        'uuid': uuid_str,
+                        'type': 'sharedlibrary',
+                        'sections': list(),
+                        'symbols': list()
+                    }
+                    for section in self.section_infos:
+                        data['sections'].append({
+                            'name' : section.name,
+                            'address': 0,
+                            'size': section.end_addr - section.start_addr
+                            })
+                    for symbol in self.symbols.values():
+                        data['symbols'].append(symbol)
+                    with open(tf.name, 'w') as f:
+                        f.write(json.dumps(data, indent=4))
+                    self.module = target.AddModule(tf.name, None, uuid_str)
             if not self.module:
                 return 'error: unable to get module for (%s) "%s"' % (
                     self.arch, self.get_resolved_path())
             if self.has_section_load_info():
                 return self.load_module(target)
             else:
                 return None  # No sections, the module was added to the target, so success
         else:
             return 'error: invalid target'
 
     def locate_module_and_debug_symbols(self):
         # By default, just use the paths that were supplied in:
         # self.path
         # self.resolved_path
         # self.module
         # self.symfile
         # Subclasses can inherit from this class and override this function
         self.resolved = True
         return True
 
     def get_uuid(self):
         if not self.uuid and self.module:
             self.uuid = uuid.UUID(self.module.GetUUIDString())
         return self.uuid
 
     def get_normalized_uuid_string(self):
         if self.uuid:
             return str(self.uuid).upper()
         return None