This is an archive of the discontinued LLVM Phabricator instance.

Paths

Table of Contentst

-
compiler-rt/
-
lib/hwasan/scripts/
-
hwasan/
-
scripts/
-
hwasan_symbolize
-
test/hwasan/TestCases/
-
hwasan/
-
TestCases/
6/6
hwasan_symbolize.cpp

Differential D124950

[HWASan] Allow to linkify symbolizer output.
ClosedPublic

Authored by fmayer on May 4 2022, 11:27 AM.

Download Raw Diff

Details

Reviewers

eugenis

Commits

rGd8564dcbcf72: [HWASan] Allow to linkify symbolizer output.
rG4af9392e13a2: [HWASan] Allow to linkify symbolizer output.

Diff Detail

Repository: rG LLVM Github Monorepo

Event Timeline

fmayer created this revision.May 4 2022, 11:27 AM

Herald added a project: Restricted Project. · View Herald TranscriptMay 4 2022, 11:27 AM

style

fix

Harbormaster completed remote builds in B162764: Diff 427095.May 4 2022, 12:27 PM

fmayer published this revision for review.May 4 2022, 2:39 PM

fmayer added a reviewer: eugenis.

Herald added a project: Restricted Project. · View Herald TranscriptMay 4 2022, 2:40 PM

Herald added a subscriber: Restricted Project. · View Herald Transcript

This script is starting to get complex. Perhaps we should write tests?

In D124950#3492520, @eugenis wrote:

This script is starting to get complex. Perhaps we should write tests?

Yes, that is on my TODO list. Happy to do that before submission, but wanted to make sure this approach is fine before doing so.

The approach looks fine. Should we also put a <html></html> pair around the output, or do you want to treat it as a snippet?

In D124950#3492544, @eugenis wrote:

The approach looks fine. Should we also put a <html></html> pair around the output, or do you want to treat it as a snippet?

Generally I don't think this would be used a standalone input, so maybe a snippet is better and the consumer can add whatever they want around. For my use-case this gets piped into pandoc to be converted to Markdown.

Add test

In D124950#3492520, @eugenis wrote:

This script is starting to get complex. Perhaps we should write tests?

done. added a test for general symbolization and for linkifying

Harbormaster completed remote builds in B162828: Diff 427193.May 4 2022, 10:01 PM

fmayer added a child revision: D125032: [HWASan] Clean up hwasan_symbolize..May 5 2022, 11:54 AM

eugenis added inline comments.May 5 2022, 2:46 PM

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
3	trying to think if this will run correctly on android, when the test binary runs on the device, but FileCheck runs on the host. Also, I think we push llvm-symbolizer to the device for online symbolization, you may need to disable it through HWASAN_OPTIONS.

fmayer added inline comments.May 5 2022, 2:48 PM

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
3	It does. What happens is that %t becomes the wrapper script (and the real binary becomes %t.real) that runs on the device. The whole pipeline specified here happens on the host. I ran this on Android.

fmayer added inline comments.May 5 2022, 2:49 PM

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
3	Good point on the HWASAN_OPTIONS. That makes it more resilient.

LGTM

This revision is now accepted and ready to land.May 5 2022, 2:55 PM

address comment

Harbormaster completed remote builds in B163022: Diff 427466.May 5 2022, 3:26 PM

Closed by commit rG4af9392e13a2: [HWASan] Allow to linkify symbolizer output. (authored by fmayer). · Explain WhyMay 5 2022, 4:40 PM

This revision was automatically updated to reflect the committed changes.

fmayer added a commit: rG4af9392e13a2: [HWASan] Allow to linkify symbolizer output..

Hi, the test you added, hwasan_symbolize.cpp seems to be causing a failure on the sanitizer buildbot (https://lab.llvm.org/buildbot/#/builders/37/builds/13022):

FAIL: HWAddressSanitizer-x86_64 :: TestCases/hwasan_symbolize.cpp (71 of 86)
******************** TEST 'HWAddressSanitizer-x86_64 :: TestCases/hwasan_symbolize.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';      /b/sanitizer-x86_64-linux/build/clang_build/./bin/clang   -m64  -gline-tables-only -fsanitize=hwaddress -fuse-ld=lld -fsanitize-hwaddress-experimental-aliasing -mllvm -hwasan-generate-tags-with-calls=1 -mllvm -hwasan-globals -mllvm -hwasan-use-short-granules -mllvm -hwasan-instrument-landing-pads=0 -mllvm -hwasan-instrument-personality-functions -Wl,--build-id -g /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp -o /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp
: 'RUN: at line 2';   echo '[{"prefix": "'"$(realpath $(dirname /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp)/../../../../../../)"'/", "link": "http://test.invalid/{file}:{line}"}]' > /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp.linkify
: 'RUN: at line 3';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:symbolize=0 not  /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp 2>&1 | hwasan_symbolize --html --symbols $(dirname /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp) --index | FileCheck /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
: 'RUN: at line 4';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:symbolize=0 not  /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp 2>&1 | hwasan_symbolize --html --linkify /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp.linkify --symbols $(dirname /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp) --index | FileCheck --check-prefixes=CHECK,LINKIFY /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
: 'RUN: at line 5';   env HWASAN_OPTIONS=disable_allocator_tagging=1:random_tags=0:fail_without_syscall_abi=0:symbolize=0 not  /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp 2>&1 | hwasan_symbolize --symbols $(dirname /b/sanitizer-x86_64-linux/build/clang_build/projects/compiler-rt/test/hwasan/X86_64/TestCases/Output/hwasan_symbolize.cpp.tmp) --index | FileCheck /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
--
Exit Code: 2
Command Output (stderr):
--
Traceback (most recent call last):
  File "/b/sanitizer-x86_64-linux/build/clang_build/./bin/hwasan_symbolize", line 19, in <module>
    import html
ImportError: No module named html
FileCheck error: '<stdin>' is empty.
FileCheck command line:  FileCheck /b/sanitizer-x86_64-linux/build/llvm-project/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
--

And also our internal linux buildbot is reporting a failure after this commit:

/home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp:17:14: error: LINKIFY: expected string not found in input
 // LINKIFY: <a href="http://test.invalid/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp:[[@LINE-1]]">
             ^
<stdin>:1:1: note: scanning from here
==27214==ERROR: HWAddressSanitizer: tag-mismatch on address 0x680400000064 at pc 0x55d1c8f6e76b<br/>
^
<stdin>:1:1: note: with "@LINE-1" equal to "16"
==27214==ERROR: HWAddressSanitizer: tag-mismatch on address 0x680400000064 at pc 0x55d1c8f6e76b<br/>
^
<stdin>:6:68: note: possible intended match here
&nbsp;&nbsp;&nbsp;&nbsp;#1 main in /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp:16:10<br/>
                                                                   ^

Input file: <stdin>
Check file: /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: ==27214==ERROR: HWAddressSanitizer: tag-mismatch on address 0x680400000064 at pc 0x55d1c8f6e76b<br/> 
check:17'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
check:17'1                                                                                                           with "@LINE-1" equal to "16"
            2: READ of size 1 at 0x680400000064 tags: 03/00 (ptr/mem) in thread T0<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            3: &nbsp;&nbsp;&nbsp;&nbsp;#0 SigTrap&lt;0&gt; in [hwasan_rtl]:0<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            4: &nbsp;&nbsp;&nbsp;&nbsp;-&gt; CheckAddress&lt;(__hwasan::ErrorAction)0, (__hwasan::AccessType)0, 0&gt; in [hwasan_rtl]:0<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            5: &nbsp;&nbsp;&nbsp;&nbsp;-&gt; __hwasan_load1 in /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/lib/hwasan/hwasan.cpp:445:0<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            6: &nbsp;&nbsp;&nbsp;&nbsp;#1 main in /home/siadmin/jenkins/w/opensource/opensource_build/working_dir/compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp:16:10<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:17'2                                                                        ?                                                                                                  possible intended match
            7: &nbsp;&nbsp;&nbsp;&nbsp;#2 0x7fbc32907bf6 (/lib/x86_64-linux-gnu/libc.so.6+0x21bf6) (BuildId: ce450eb01a5e5acc7ce7b8c2633b02cc1093339e)<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            8: &nbsp;&nbsp;&nbsp;&nbsp;#3 _start in ??:0:0<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            9: <br/> 
check:17'0     ~~~~~~
           10: [0x680400000060,0x680400000080) is a small unallocated heap chunk; size: 32 offset: 4<br/> 
check:17'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           11: <br/> 
check:17'0     ~~~~~~
            .
            .
            .
>>>>>>

Can you please take a look and revert if it will take some time to fix?

And also our internal linux buildbot is reporting a failure after this commit:

*snip*

I'm seeing this same failure when I run ninja check-hwasan in my local, fairly vanilla, cmake build.

I'm also seeing it on http://45.33.8.238/linux/75461/step_10.txt . Given the other report of breakage, I'll revert this for now until things are sorted out.

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
6	Do you know why this works? Normally (in clang and lld and llvm), you have to call `llvm_config.add_tool_substitutions()` for the binaries that you want lit to find in the build dir. compiler-rt doesn't seem to do this, and hwasan_symbolize seems to be found fine so it works somehow, but I don't know how. Could you explain it to me?

thakis added a reverting change: rGc9faea04b1f8: Revert "[HWASan] Allow to linkify symbolizer output.".May 6 2022, 6:54 AM

Here's another bot this failed on, also doing a fairly vanilla cmake build: https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket/8814940711487302321/+/u/package_clang/stdout?format=raw

In D124950#3496660, @thakis wrote:

And also our internal linux buildbot is reporting a failure after this commit:

*snip*

I'm seeing this same failure when I run ninja check-hwasan in my local, fairly vanilla, cmake build.

I'm also seeing it on http://45.33.8.238/linux/75461/step_10.txt . Given the other report of breakage, I'll revert this for now until things are sorted out.

Which version of Python are you running? html is apparently around since 3.2.

EDIT: oh, I misunderstood. "this error" being the FileCheck, not the import. Correct?

I'm running python 3.9.10.

As far as I can tell, there are 2 distinct errors:

the import html error on https://lab.llvm.org/buildbot/#/builders/37/builds/13022
FileCheck error seen on @dyung's internal bot, on my machine, and on my bot

In D124950#3496884, @thakis wrote:

I'm running python 3.9.10.

As far as I can tell, there are 2 distinct errors:

the import html error on https://lab.llvm.org/buildbot/#/builders/37/builds/13022

FileCheck error seen on @dyung's internal bot, on my machine, and on my bot

Thanks. Sorry about that, thanks for reverting.

fmayer reopened this revision.May 6 2022, 11:14 AM

This revision is now accepted and ready to land.May 6 2022, 11:14 AM

fmayer planned changes to this revision.May 6 2022, 11:14 AM

fix test.

This revision is now accepted and ready to land.May 6 2022, 3:13 PM

fmayer added inline comments.May 6 2022, 3:16 PM

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp
6	When I run `env` in the test, `build/bin` is the first entry in path, so it makes sense that this works.

Harbormaster completed remote builds in B163242: Diff 427768.May 6 2022, 3:38 PM

fmayer added inline comments.May 6 2022, 3:43 PM

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp

That seems to be due to this in our lit cfg

# Tweak PATH to include llvm tools dir.
if (not config.llvm_tools_dir) or (not os.path.exists(config.llvm_tools_dir)):
  lit_config.fatal("Invalid llvm_tools_dir config attribute: %r" % config.llvm_tools_dir)
path = os.path.pathsep.join((config.llvm_tools_dir, config.environment['PATH']))
config.environment['PATH'] = path

Closed by commit rGd8564dcbcf72: [HWASan] Allow to linkify symbolizer output. (authored by fmayer). · Explain WhyMay 6 2022, 3:44 PM

This revision was automatically updated to reflect the committed changes.

fmayer added a commit: rGd8564dcbcf72: [HWASan] Allow to linkify symbolizer output..

Revision Contents

Path

Size

compiler-rt/

lib/

hwasan/

scripts/

hwasan_symbolize

87 lines

test/

hwasan/

TestCases/

hwasan_symbolize.cpp

27 lines

Diff 427775

compiler-rt/lib/hwasan/scripts/hwasan_symbolize

#!/usr/bin/env python		#!/usr/bin/env python3
#===- lib/hwasan/scripts/hwasan_symbolize ----------------------------------===#		#===- lib/hwasan/scripts/hwasan_symbolize ----------------------------------===#
#		#
# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.		# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
# See https:#llvm.org/LICENSE.txt for license information.		# See https:#llvm.org/LICENSE.txt for license information.
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception		# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#		#
#===------------------------------------------------------------------------===#		#===------------------------------------------------------------------------===#
#		#
# HWAddressSanitizer offline symbolization script.		# HWAddressSanitizer offline symbolization script.
#		#
#===------------------------------------------------------------------------===#		#===------------------------------------------------------------------------===#

from __future__ import print_function		from __future__ import print_function
from __future__ import unicode_literals		from __future__ import unicode_literals

import argparse		import argparse
import glob		import glob
		import html
		import json
import mmap		import mmap
import os		import os
import re		import re
import struct		import struct
import subprocess		import subprocess
import sys		import sys

if sys.version_info.major < 3:		if sys.version_info.major < 3:
▲ Show 20 Lines • Show All 74 Lines • ▼ Show 20 Lines	class Symbolizer:
def __init__(self, path, binary_prefixes, paths_to_cut):		def __init__(self, path, binary_prefixes, paths_to_cut):
self.__pipe = None		self.__pipe = None
self.__path = path		self.__path = path
self.__binary_prefixes = binary_prefixes		self.__binary_prefixes = binary_prefixes
self.__paths_to_cut = paths_to_cut		self.__paths_to_cut = paths_to_cut
self.__log = False		self.__log = False
self.__warnings = set()		self.__warnings = set()
self.__index = {}		self.__index = {}
		self.__link_prefixes = []
		self.__html = False

		def enable_html(self, enable):
		self.__html = enable

def enable_logging(self, enable):		def enable_logging(self, enable):
self.__log = enable		self.__log = enable

		def maybe_escape(self, text):
		if self.__html:
		# We need to manually use   for leading spaces, html.escape does
		# not do that, and HTML ignores them.
		spaces = 0
		for i, c in enumerate(text):
		spaces = i
		if c != ' ':
		break
		text = text[spaces:]
		return spaces * ' ' + html.escape(text)
		return text

		def print(self, line, escape=True):
		if escape:
		line = self.maybe_escape(line)
		if self.__html:
		line += '<br/>'
		print(line)

		def read_linkify(self, filename):
		with open(filename, 'r') as fd:
		data = json.load(fd)
		self.__link_prefixes = [(e["prefix"], e["link"]) for e in data]

def __open_pipe(self):		def __open_pipe(self):
if not self.__pipe:		if not self.__pipe:
opt = {}		opt = {}
if sys.version_info.major > 2:		if sys.version_info.major > 2:
opt['encoding'] = 'utf-8'		opt['encoding'] = 'utf-8'
self.__pipe = subprocess.Popen([self.__path, "--inlining", "--functions"],		self.__pipe = subprocess.Popen([self.__path, "--inlining", "--functions"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE,		stdin=subprocess.PIPE, stdout=subprocess.PIPE,
**opt)		**opt)
▲ Show 20 Lines • Show All 81 Lines • ▼ Show 20 Lines	try:
while True:		while True:
function_name = self.__read()		function_name = self.__read()
file_line = self.__read()		file_line = self.__read()
file_line = self.__process_source_path(file_line)		file_line = self.__process_source_path(file_line)
yield (function_name, file_line)		yield (function_name, file_line)
except Symbolizer.__EOF:		except Symbolizer.__EOF:
pass		pass

		def maybe_linkify(self, file_line):
		if not self.__html or not self.__link_prefixes:
		return file_line
		filename, line_col = file_line.split(':', 1)
		if not line_col:
		line = '0' # simplify the link generation
		else:
		line = line_col.split(':')[0]
		longest_prefix = max((
		(prefix, link) for prefix, link in self.__link_prefixes
		if filename.startswith(prefix)),
		key=lambda x: len(x[0]), default=None)
		if longest_prefix is None:
		return file_line
		else:
		prefix, link = longest_prefix
		return '<a href="{}">{}</a>'.format(
		html.escape(link.format(file=filename[len(prefix):], line=line,
		file_line=file_line, prefix=prefix)), file_line)

def build_index(self):		def build_index(self):
for p in self.__binary_prefixes:		for p in self.__binary_prefixes:
for dname, _, fnames in os.walk(p):		for dname, _, fnames in os.walk(p):
for fn in fnames:		for fn in fnames:
filename = os.path.join(dname, fn)		filename = os.path.join(dname, fn)
bid = get_buildid(filename)		bid = get_buildid(filename)
if bid is not None:		if bid is not None:
self.__index[bid] = filename		self.__index[bid] = filename

def symbolize_line(line, symbolizer_path):		def symbolize_line(line, symbolizer_path):
#0 0x7f6e35cf2e45 (/blah/foo.so+0x11fe45) (BuildId: 4abce4cd41ea5c2f34753297b7e774d9)		#0 0x7f6e35cf2e45 (/blah/foo.so+0x11fe45) (BuildId: 4abce4cd41ea5c2f34753297b7e774d9)
match = re.match(r'^(.?)#([0-9]+)( )(0x[0-9a-f]) $(.*)\+(0x[0-9a-f]+)$'		match = re.match(r'^(.?)#([0-9]+)( )(0x[0-9a-f]) $(.*)\+(0x[0-9a-f]+)$'
r'(?:\s*$BuildId: ([0-9a-f]+)$)?', line, re.UNICODE)		r'(?:\s*$BuildId: ([0-9a-f]+)$)?', line, re.UNICODE)
if match:		if match:
frameno = match.group(2)		frameno = match.group(2)
binary = match.group(5)		binary = match.group(5)
addr = int(match.group(6), 16)		addr = int(match.group(6), 16)
buildid = match.group(7)		buildid = match.group(7)

frames = list(symbolizer.iter_call_stack(binary, buildid, addr))		frames = list(symbolizer.iter_call_stack(binary, buildid, addr))

if len(frames) > 0:		if len(frames) > 0:
print("%s#%s%s%s in %s" % (match.group(1), match.group(2),		symbolizer.print(
match.group(3), frames[0][0], frames[0][1]))		symbolizer.maybe_escape(
		"%s#%s%s%s in " % (match.group(1), match.group(2), match.group(3),
		frames[0][0])
		) + symbolizer.maybe_linkify(frames[0][1]),
		escape=False)
for i in range(1, len(frames)):		for i in range(1, len(frames)):
space1 = ' ' * match.end(1)		space1 = ' ' * match.end(1)
space2 = ' ' * (match.start(4) - match.end(1) - 2)		space2 = ' ' * (match.start(4) - match.end(1) - 2)
print("%s->%s%s in %s" % (space1, space2, frames[i][0], frames[i][1]))		symbolizer.print(
		symbolizer.maybe_escape("%s->%s%s in " % (space1, space2, frames[i][0]))
		+ symbolizer.maybe_linkify(frames[i][1]), escape=False)
else:		else:
print(line.rstrip())		symbolizer.print(line.rstrip())
else:		else:
print(line.rstrip())		symbolizer.print(line.rstrip())

def save_access_address(line):		def save_access_address(line):
global last_access_address, last_access_tag		global last_access_address, last_access_tag
match = re.match(r'^(.*?)HWAddressSanitizer: tag-mismatch on address (0x[0-9a-f]+) ', line, re.UNICODE)		match = re.match(r'^(.*?)HWAddressSanitizer: tag-mismatch on address (0x[0-9a-f]+) ', line, re.UNICODE)
if match:		if match:
last_access_address = int(match.group(2), 16)		last_access_address = int(match.group(2), 16)
match = re.match(r'^(.?) of size [0-9]+ at 0x[0-9a-f] tags: ([0-9a-f]+)/[0-9a-f]+ $ptr/mem$', line, re.UNICODE)		match = re.match(r'^(.?) of size [0-9]+ at 0x[0-9a-f] tags: ([0-9a-f]+)/[0-9a-f]+ $ptr/mem$', line, re.UNICODE)
if match:		if match:
Show All 25 Lines	for local in symbolizer.iter_locals(binary, addr, buildid):
if frame_offset is None or size is None:		if frame_offset is None or size is None:
continue		continue
obj_offset = (last_access_address - fp - frame_offset) & fp_mask		obj_offset = (last_access_address - fp - frame_offset) & fp_mask
if obj_offset >= size:		if obj_offset >= size:
continue		continue
tag_offset = local[5]		tag_offset = local[5]
if not ignore_tags and (tag_offset is None or base_tag ^ tag_offset != last_access_tag):		if not ignore_tags and (tag_offset is None or base_tag ^ tag_offset != last_access_tag):
continue		continue
print('')		symbolizer.print('')
print('Potentially referenced stack object:')		symbolizer.print('Potentially referenced stack object:')
print(' %d bytes inside variable "%s" in stack frame of function "%s"' % (obj_offset, local[2], local[0]))		symbolizer.print(' %d bytes inside variable "%s" in stack frame of function "%s"' % (obj_offset, local[2], local[0]))
print(' at %s' % (local[1],))		symbolizer.print(' at %s' % (local[1],))
return True		return True
return False		return False

parser = argparse.ArgumentParser()		parser = argparse.ArgumentParser()
parser.add_argument('-d', action='store_true')		parser.add_argument('-d', action='store_true')
parser.add_argument('-v', action='store_true')		parser.add_argument('-v', action='store_true')
parser.add_argument('--ignore-tags', action='store_true')		parser.add_argument('--ignore-tags', action='store_true')
parser.add_argument('--symbols', action='append')		parser.add_argument('--symbols', action='append')
parser.add_argument('--source', action='append')		parser.add_argument('--source', action='append')
parser.add_argument('--index', action='store_true')		parser.add_argument('--index', action='store_true')
parser.add_argument('--symbolizer')		parser.add_argument('--symbolizer')
		parser.add_argument('--linkify', type=str)
		parser.add_argument('--html', action='store_true')
parser.add_argument('args', nargs=argparse.REMAINDER)		parser.add_argument('args', nargs=argparse.REMAINDER)
args = parser.parse_args()		args = parser.parse_args()

# Unstripped binaries location.		# Unstripped binaries location.
binary_prefixes = args.symbols or []		binary_prefixes = args.symbols or []
if not binary_prefixes:		if not binary_prefixes:
if 'ANDROID_PRODUCT_OUT' in os.environ:		if 'ANDROID_PRODUCT_OUT' in os.environ:
product_out = os.path.join(os.environ['ANDROID_PRODUCT_OUT'], 'symbols')		product_out = os.path.join(os.environ['ANDROID_PRODUCT_OUT'], 'symbols')
▲ Show 20 Lines • Show All 69 Lines • ▼ Show 20 Lines	for s in binary_prefixes:
print(" %s" % (s,))		print(" %s" % (s,))
print("Stripping source path prefixes:")		print("Stripping source path prefixes:")
for s in paths_to_cut:		for s in paths_to_cut:
print(" %s" % (s,))		print(" %s" % (s,))
print("Using llvm-symbolizer binary in:\n %s" % (symbolizer_path,))		print("Using llvm-symbolizer binary in:\n %s" % (symbolizer_path,))
print()		print()

symbolizer = Symbolizer(symbolizer_path, binary_prefixes, paths_to_cut)		symbolizer = Symbolizer(symbolizer_path, binary_prefixes, paths_to_cut)
		symbolizer.enable_html(args.html)
symbolizer.enable_logging(args.d)		symbolizer.enable_logging(args.d)
if args.index:		if args.index:
symbolizer.build_index()		symbolizer.build_index()

		if args.linkify:
		if not args.html:
		print('Need --html to --linkify', file=sys.stderr)
		sys.exit(1)
		symbolizer.read_linkify(args.linkify)

for line in sys.stdin:		for line in sys.stdin:
if sys.version_info.major < 3:		if sys.version_info.major < 3:
line = line.decode('utf-8')		line = line.decode('utf-8')
save_access_address(line)		save_access_address(line)
if process_stack_history(line, symbolizer, ignore_tags=args.ignore_tags):		if process_stack_history(line, symbolizer, ignore_tags=args.ignore_tags):
continue		continue
symbolize_line(line, symbolizer_path)		symbolize_line(line, symbolizer_path)

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp

This file was added.

				// RUN: %clang_hwasan -Wl,--build-id -g %s -o %t
				// RUN: echo '[{"prefix": "'"$(realpath $(dirname %s))"'/", "link": "http://test.invalid/{file}:{line}"}]' > %t.linkify
				// RUN: %env_hwasan_opts=symbolize=0 not %run %t 2>&1 \| hwasan_symbolize --html --symbols $(dirname %t) --index \| FileCheck %s
				eugenisUnsubmitted Done Reply Inline Actions trying to think if this will run correctly on android, when the test binary runs on the device, but FileCheck runs on the host. Also, I think we push llvm-symbolizer to the device for online symbolization, you may need to disable it through HWASAN_OPTIONS. eugenis: trying to think if this will run correctly on android, when the test binary runs on the device…
				fmayerAuthorUnsubmitted Done Reply Inline Actions It does. What happens is that %t becomes the wrapper script (and the real binary becomes %t.real) that runs on the device. The whole pipeline specified here happens on the host. I ran this on Android. fmayer: It does. What happens is that %t becomes the wrapper script (and the real binary becomes %t.
				fmayerAuthorUnsubmitted Done Reply Inline Actions Good point on the HWASAN_OPTIONS. That makes it more resilient. fmayer: Good point on the HWASAN_OPTIONS. That makes it more resilient.
				// RUN: %env_hwasan_opts=symbolize=0 not %run %t 2>&1 \| hwasan_symbolize --html --linkify %t.linkify --symbols $(dirname %t) --index \| FileCheck --check-prefixes=CHECK,LINKIFY %s
				// RUN: %env_hwasan_opts=symbolize=0 not %run %t 2>&1 \| hwasan_symbolize --symbols $(dirname %t) --index \| FileCheck %s

				thakisUnsubmitted Done Reply Inline Actions Do you know why this works? Normally (in clang and lld and llvm), you have to call `llvm_config.add_tool_substitutions()` for the binaries that you want lit to find in the build dir. compiler-rt doesn't seem to do this, and hwasan_symbolize seems to be found fine so it works somehow, but I don't know how. Could you explain it to me? thakis: Do you know why this works? Normally (in clang and lld and llvm), you have to call `llvm_config.
				fmayerAuthorUnsubmitted Done Reply Inline Actions When I run `env` in the test, `build/bin` is the first entry in path, so it makes sense that this works. fmayer: When I run `env` in the test, `build/bin` is the first entry in path, so it makes sense that…
				fmayerAuthorUnsubmitted Done Reply Inline Actions That seems to be due to this in our lit cfg # Tweak PATH to include llvm tools dir. if (not config.llvm_tools_dir) or (not os.path.exists(config.llvm_tools_dir)): lit_config.fatal("Invalid llvm_tools_dir config attribute: %r" % config.llvm_tools_dir) path = os.path.pathsep.join((config.llvm_tools_dir, config.environment['PATH'])) config.environment['PATH'] = path fmayer: That seems to be due to this in our lit cfg ``` # Tweak PATH to include llvm tools dir. if…
				// There are currently unrelated problems on x86_64, so skipping for now.
				// REQUIRES: android
				// REQUIRES: stable-runtime

				#include <sanitizer/hwasan_interface.h>
				#include <stdlib.h>

				static volatile char sink;

				int main(int argc, char **argv) {
				__hwasan_enable_allocator_tagging();
				char volatile x = (char )malloc(10);
				sink = x[100];
				// LINKIFY: <a href="http://test.invalid/hwasan_symbolize.cpp:[[@LINE-1]]">
				// CHECK: hwasan_symbolize.cpp:[[@LINE-2]]
				// CHECK: Cause: heap-buffer-overflow
				// CHECK: allocated here:
				// LINKIFY: <a href="http://test.invalid/hwasan_symbolize.cpp:[[@LINE-6]]">
				// CHECK: hwasan_symbolize.cpp:[[@LINE-7]]
				return 0;
				}

This is an archive of the discontinued LLVM Phabricator instance.

[HWASan] Allow to linkify symbolizer output.ClosedPublic

Details

Diff Detail

Event Timeline

Revision Contents

Diff 427775

compiler-rt/lib/hwasan/scripts/hwasan_symbolize

compiler-rt/test/hwasan/TestCases/hwasan_symbolize.cpp

[HWASan] Allow to linkify symbolizer output.
ClosedPublic