[asan_symbolize] Teach `asan_symbolize.py` to symbolicate partially symbolicated ASan reports.

[delcypher]

Summary

The use case here is to be able symbolicate ASan reports that might be
partially symbolicated, in particular where the function name is known but no source
location is available. This can be caused by missing debug info. Previously we
would only try to symbolicate completely unsymbolicated reports.

The code currently contains an unfortunate quirk to handle a darwin
specific bug (rdar://problem/49784442) in the way partially symbolicated
reports are emitted when the source location is missing.

rdar://problem/49476995

[Eugene.Zelenko]

test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cc

30 ↗

(On Diff #194570)

Asserts are not used.

31 ↗

(On Diff #194570)

Please use cstdlib for C++ code.

[delcypher]

• Use cstdlib include instead of stdlib.h
• Remove unused assert.h include

[vitalybuka]

LGTM

[delcypher]

In D60533#1463092, @vitalybuka wrote:

LGTM

Thanks. I need to land the other patches in the stack first though. This is because asan_symbolize.py needs to use the module map to symbolize on darwin when the atos symbolizer is used (the default).

[vitalybuka]

Can you make the tests Posix?

[delcypher]

In D60533#1477310, @vitalybuka wrote:

Can you make the tests Posix?

To clarify, do you mean

1. Split the test up into the Posix and Darwin specific part (use Module map)
2. Disable the Posix test for Darwin (it won't work until we fix a bug in the Symbolizer implementation for Darwin).

If, yes then this seems like a good idea.

[vitalybuka]

Thanks, You already explained that module-map is Darwin specific on another patch.
You are still welcome to extract Posix stuff if you see how.

[delcypher]

@vitalybuka Sorry I forgot about this. I tried to make the test to work in a POSIX use case and I can't make it work. The problem is that when I use %M with stack_trace_format the module name gets stripped (i.e. an absolute path to the binary just becomes the file name of the binary) due to the call to StripModuleName(). This results in us not being able to symbolicate the report from the python script because we don't have the absolute path to the binaries we need to access to perform symbolication.

I have no idea why for %M we strip the module name but when %L is in fallback mode, we don't. This seems oddly inconsistent.

Anyway. Given that I can't make the posix version of the test right now I'll land this as it is.

[vitalybuka]

compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py

469

why do we need isabs here? should this be os.path.isfile ?

compiler-rt/trunk/lib/asan/scripts/asan_symbolize.py

else:
       self.system = os.uname()[0]
       if self.system not in ['Linux', 'Darwin', 'FreeBSD', 'NetBSD','SunOS']:
         raise Exception('Unknown system')
       self.llvm_symbolizers = {}
       self.last_llvm_symbolizer = None
       self.dsym_hints = set([])
       self.frame_no = 0
       self.process_line = self.process_line_posix
+      self.using_module_map = plugin_proxy.has_plugin(ModuleMapPlugIn.get_name())
 
   def symbolize_address(self, addr, binary, offset, arch):
     # On non-Darwin (i.e. on platforms without .dSYM debug info) always use
     # a single symbolizer binary.
     # On Darwin, if the dsym hint producer is present:
     #  1. check whether we've seen this binary already; if so,
     #     use |llvm_symbolizers[binary]|, which has already loaded the debug
     #     info for this binary (might not be the case for
for line in logfile:
       processed = self.process_line(line)
       print('\n'.join(processed))
 
   def process_line_echo(self, line):
     return [line.rstrip()]
 
   def process_line_posix(self, line):
     self.current_line = line.rstrip()
-    #0 0x7f6e35cf2e45  (/blah/foo.so+0x11fe45)
+    # Unsymbolicated:
+    # #0 0x7f6e35cf2e45  (/blah/foo.so+0x11fe45)
+    # Partially symbolicated:
+    # #0 0x7f6e35cf2e45 in foo (foo.so+0x11fe45)
+    # NOTE: We have to very liberal with symbol
+    # names in the regex because it could be an
+    # Objective-C or C++ demangled name.
     stack_trace_line_format = (
-        '^( *#([0-9]+) *)(0x[0-9a-f]+) *\((.*)\+(0x[0-9a-f]+)\)')
+        '^( *#([0-9]+) *)(0x[0-9a-f]+) *(?:in *.+)? *\((.*)\+(0x[0-9a-f]+)\)')
     match = re.match(stack_trace_line_format, line)
     if not match:
       return [self.current_line]
     logging.debug(line)
     _, frameno_str, addr, binary, offset = match.groups()
+    if not self.using_module_map and not os.path.isabs(binary):
+      # Do not try to symbolicate if the binary is just the module file name
+      # and a module map is unavailable.
+      # FIXME(dliew): This is currently necessary for reports on Darwin that are
+      # partially symbolicated by `atos`.
+      return [self.current_line]
     arch = ""
     # Arch can be embedded in the filename, e.g.: "libabc.dylib:x86_64h"
     colon_pos = binary.rfind(":")
     if colon_pos != -1:
       maybe_arch = binary[colon_pos+1:]
       if is_valid_arch(maybe_arch):
         arch = maybe_arch
         binary = binary[0:colon_pos]

compiler-rt/trunk/test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cc

+// When `external_symbolizer_path` is empty on Darwin we fallback on using
+// dladdr as the symbolizer which means we get the symbol name
+// but no source location. The current implementation also doesn't try to
+// change the module name so we end up with the full name so we actually don't
+// need the module map here.
+
+// RUN: %clangxx_asan -O0 -g %s -o %t.executable
+// RUN: %env_asan_opts=symbolize=1,print_module_map=0,external_symbolizer_path= not %run %t.executable > %t2.log 2>&1
+// RUN: FileCheck -input-file=%t2.log -check-prefix=CHECK-PS %s
+// RUN: %asan_symbolize --force-system-symbolizer < %t2.log > %t2.fully_symbolized
+// RUN: FileCheck -input-file=%t2.fully_symbolized -check-prefix=CHECK-FS %s
+
+// Due a quirk in the way atos reports module names we have to use the module
+// map here, otherwise we don't know what the full path to the module is.
+
+// FIXME(dliew): We currently have to use module map for this test due to the atos
+// symbolizer changing the module name from an absolute path to just the file name.
+// rdar://problem/49784442
+//
+// Simulate partial symbolication (can happen with %L specifier) by printing
+// out %L's fallback which will print the module name and offset instead of a
+// source location.
+// RUN: %clangxx_asan -O0 -g %s -o %t2.executable
+// RUN: %env_asan_opts=symbolize=1,print_module_map=1,stack_trace_format='"    #%%n %%p %%F %%M"' not %run %t.executable > %t2.log 2>&1
+// RUN: FileCheck -input-file=%t2.log -check-prefix=CHECK-PS %s
+// Now try to full symbolicate using the module map.
+// RUN: %asan_symbolize --module-map %t2.log --force-system-symbolizer < %t2.log > %t2.fully_symbolized
+// RUN: FileCheck -input-file=%t2.fully_symbolized -check-prefix=CHECK-FS %s
+
+#include <cstdlib>
+
+// Partially symbolicated back-trace where symbol is available but
+// source location is not and instead module name and offset are
+// printed.
+// CHECK-PS: WRITE of size 4
+// CHECK-PS: #0 0x{{.+}} in foo ({{.+}}.executable:{{.+}}+0x{{.+}})
+// CHECK-PS: #1 0x{{.+}} in main ({{.+}}.executable:{{.+}}+0x{{.+}})
+
+// CHECK-FS: WRITE of size 4
+
+extern "C" void foo(int* a) {
+  // CHECK-FS: #0 0x{{.+}} in foo {{.*}}asan-symbolize-partial-report-with-module-map.cc:[[@LINE+1]]
+  *a = 5;
+}
+
+int main() {
+  int* a = (int*) malloc(sizeof(int));
+  if (!a)
+    return 0;
+  free(a);
+  // CHECK-FS: #1 0x{{.+}} in main {{.*}}asan-symbolize-partial-report-with-module-map.cc:[[@LINE+1]]
+  foo(a);
+  return 0;
+}