This is an archive of the discontinued LLVM Phabricator instance.

Differential D73011

[opt viewer] Python compat - decode/encode string
ClosedPublic

Authored by ychen on Jan 19 2020, 5:57 PM.

Details

Reviewers
anemet
serge-sans-paille
Commits
rG43d9f2d1e8ae: [opt viewer] Python compat - decode/encode string
Summary

Use io.open instead of codecs.open according to here
https://stackoverflow.com/questions/10971033/backporting-python-3-openencoding-utf-8-to-python-2

Add u prefix to string literal to make them utf-8 in python2.

Diff Detail

Event Timeline

ychen created this revision.Jan 19 2020, 5:57 PM
Herald added a project: Restricted Project. · View Herald TranscriptJan 19 2020, 5:57 PM
Herald added a subscriber: llvm-commits. · View Herald Transcript
MaskRay added subscribers: thakis, MaskRay.Jan 19 2020, 6:07 PM

#!/usr/bin/env python -> python3 will be simpler, but I know @thakis could be unhappy.

merge_guards_bot added a subscriber: merge_guards_bot.Jan 19 2020, 6:24 PM

Unit tests: pass. 62002 tests passed, 0 failed and 783 were skipped.

clang-tidy: unknown.

clang-format: pass.

Build artifacts: diff.json, clang-format.patch, CMakeCache.txt, console-log.txt, test-results.xml

Harbormaster completed remote builds in B44373: Diff 239016.Jan 19 2020, 6:28 PM
ychen added a comment.Jan 19 2020, 10:36 PM

@MaskRay Yeah, that would be ideal. I'm also in favor of supporting python3 only. The change for open to add encoding parameter is still needed though since the default value is platform-dependent.

serge-sans-paille added a comment.Jan 20 2020, 12:54 AM

I'm also in favor of a loose consensus such as « as long as we can keep py2/3 compat, keep it. »

RHEL7 still ships Python2 by default, but it has reached « End of Full Support ». (see https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle)
Latest OSX version also explicitly obsoletes Python2 (see https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes#3318257)

So I'd advocate to keep the compatibility until end of 2020? That's probably a subject that should be discussed on the mlist though.

ychen added a comment.Jan 24 2020, 10:35 AM

ping?

serge-sans-paille added a comment.Jan 29 2020, 2:30 AM

@ychen I summarized the situation in http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html , but according to the documentation, https://llvm.org/docs/GettingStarted.html#id7 we still support python2.7 ...

ychen added a comment.Jan 29 2020, 7:36 AM
In D73011#1846419, @serge-sans-paille wrote:

@ychen I summarized the situation in http://lists.llvm.org/pipermail/llvm-dev/2020-January/138730.html , but according to the documentation, https://llvm.org/docs/GettingStarted.html#id7 we still support python2.7 ...

@serge-sans-paille io.open is supported since 2.6 https://docs.python.org/2/library/io.html#module-interface. I did test that the patch works for both 2.7 and >3. I saw some failures with the Unicode problem without this change.

serge-sans-paille accepted this revision.Jan 29 2020, 12:18 PM

@serge-sans-paille io.open is supported since 2.6 https://docs.python.org/2/library/io.html#module-interface. I did test that the patch works for both 2.7 and >3. I saw some failures with the Unicode problem without this change.

Good lord - you're absolutely right!

The discussion diverged because of the python3 shebang, but the actual diff is portable. LGTM then :-)

This revision is now accepted and ready to land.Jan 29 2020, 12:18 PM
ychen added a comment.Jan 29 2020, 2:37 PM

@serge-sans-paille Thank you!

Closed by commit rG43d9f2d1e8ae: [opt viewer] Python compat - decode/encode string (authored by Yuanfang Chen <yuanfang.chen@sony.com>). · Explain WhyJan 29 2020, 2:53 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents

PathSize
llvm/
tools/
opt-viewer/
23 lines
3 lines

Diff 239016

llvm/tools/opt-viewer/opt-viewer.py

#!/usr/bin/env python #!/usr/bin/env python
from __future__ import print_function from __future__ import print_function
import argparse import argparse
import cgi import cgi
import codecs
import errno import errno
import functools import functools
import io
from multiprocessing import cpu_count from multiprocessing import cpu_count
import os.path import os.path
import re import re
import shutil import shutil
import sys import sys
from pygments import highlight from pygments import highlight
from pygments.lexers.c_cpp import CppLexer from pygments.lexers.c_cpp import CppLexer
Show All 31 Lines def __init__(self, source_dir, output_dir, filename, no_highlight):
if os.path.exists(filename): if os.path.exists(filename):
existing_filename = filename existing_filename = filename
else: else:
fn = os.path.join(source_dir, filename) fn = os.path.join(source_dir, filename)
if os.path.exists(fn): if os.path.exists(fn):
existing_filename = fn existing_filename = fn
self.no_highlight = no_highlight self.no_highlight = no_highlight
self.stream = codecs.open(os.path.join(output_dir, optrecord.html_file_name(filename)), 'w', encoding='utf-8') self.stream = io.open(os.path.join(output_dir, optrecord.html_file_name(filename)), 'w', encoding='utf-8')
if existing_filename: if existing_filename:
self.source_stream = open(existing_filename) self.source_stream = io.open(existing_filename, encoding='utf-8')
else: else:
self.source_stream = None self.source_stream = None
print(''' print(u'''
<html> <html>
<h1>Unable to locate file {}</h1> <h1>Unable to locate file {}</h1>
</html> </html>
'''.format(filename), file=self.stream) '''.format(filename), file=self.stream)
self.html_formatter = HtmlFormatter(encoding='utf-8') self.html_formatter = HtmlFormatter(encoding='utf-8')
self.cpp_lexer = CppLexer(stripnl=False) self.cpp_lexer = CppLexer(stripnl=False)
def render_source_lines(self, stream, line_remarks): def render_source_lines(self, stream, line_remarks):
file_text = stream.read() file_text = stream.read()
if self.no_highlight: if self.no_highlight:
if sys.version_info.major >= 3:
html_highlighted = file_text html_highlighted = file_text
else: else:
html_highlighted = file_text.decode('utf-8')
else:
html_highlighted = highlight( html_highlighted = highlight(
file_text, file_text,
self.cpp_lexer, self.cpp_lexer,
self.html_formatter) self.html_formatter)
# Note that the API is different between Python 2 and 3. On # Note that the API is different between Python 2 and 3. On
# Python 3, pygments.highlight() returns a bytes object, so we # Python 3, pygments.highlight() returns a bytes object, so we
# have to decode. On Python 2, the output is str but since we # have to decode. On Python 2, the output is str but since we
▲ Show 20 Lines • Show All 54 Lines • ▼ Show 20 Lines
<td><pre style="display:inline">{indent}</pre><span class=\"column-entry-yellow\">{expand_link} {message}&nbsp;</span>{expand_message}</td> <td><pre style="display:inline">{indent}</pre><span class=\"column-entry-yellow\">{expand_link} {message}&nbsp;</span>{expand_message}</td>
<td class=\"column-entry-yellow\">{inlining_context}</td> <td class=\"column-entry-yellow\">{inlining_context}</td>
</tr>'''.format(**locals()), file=self.stream) </tr>'''.format(**locals()), file=self.stream)
def render(self, line_remarks): def render(self, line_remarks):
if not self.source_stream: if not self.source_stream:
return return
print(''' print(u'''
<html> <html>
<title>{}</title> <title>{}</title>
<meta charset="utf-8" /> <meta charset="utf-8" />
<head> <head>
<link rel='stylesheet' type='text/css' href='style.css'> <link rel='stylesheet' type='text/css' href='style.css'>
<script type="text/javascript"> <script type="text/javascript">
/* Simple helper to show/hide the expanded message of a remark. */ /* Simple helper to show/hide the expanded message of a remark. */
function toggleExpandedMessage(e) {{ function toggleExpandedMessage(e) {{
Show All 22 Lines
<th style="width: 10%">Optimization</td> <th style="width: 10%">Optimization</td>
<th style="width: 70%">Source</td> <th style="width: 70%">Source</td>
<th style="width: 15%">Inline Context</td> <th style="width: 15%">Inline Context</td>
</tr> </tr>
</thead> </thead>
<tbody>'''.format(os.path.basename(self.filename)), file=self.stream) <tbody>'''.format(os.path.basename(self.filename)), file=self.stream)
self.render_source_lines(self.source_stream, line_remarks) self.render_source_lines(self.source_stream, line_remarks)
print(''' print(u'''
</tbody> </tbody>
</table> </table>
</body> </body>
</html>''', file=self.stream) </html>''', file=self.stream)
class IndexRenderer: class IndexRenderer:
def __init__(self, output_dir, should_display_hotness, max_hottest_remarks_on_index): def __init__(self, output_dir, should_display_hotness, max_hottest_remarks_on_index):
self.stream = codecs.open(os.path.join(output_dir, 'index.html'), 'w', encoding='utf-8') self.stream = io.open(os.path.join(output_dir, 'index.html'), 'w', encoding='utf-8')
self.should_display_hotness = should_display_hotness self.should_display_hotness = should_display_hotness
self.max_hottest_remarks_on_index = max_hottest_remarks_on_index self.max_hottest_remarks_on_index = max_hottest_remarks_on_index
def render_entry(self, r, odd): def render_entry(self, r, odd):
escaped_name = cgi.escape(r.DemangledFunctionName) escaped_name = cgi.escape(r.DemangledFunctionName)
print(u''' print(u'''
<tr> <tr>
<td class=\"column-entry-{odd}\"><a href={r.Link}>{r.DebugLocString}</a></td> <td class=\"column-entry-{odd}\"><a href={r.Link}>{r.DebugLocString}</a></td>
<td class=\"column-entry-{odd}\">{r.RelativeHotness}</td> <td class=\"column-entry-{odd}\">{r.RelativeHotness}</td>
<td class=\"column-entry-{odd}\">{escaped_name}</td> <td class=\"column-entry-{odd}\">{escaped_name}</td>
<td class=\"column-entry-{r.color}\">{r.PassWithDiffPrefix}</td> <td class=\"column-entry-{r.color}\">{r.PassWithDiffPrefix}</td>
</tr>'''.format(**locals()), file=self.stream) </tr>'''.format(**locals()), file=self.stream)
def render(self, all_remarks): def render(self, all_remarks):
print(''' print(u'''
<html> <html>
<meta charset="utf-8" /> <meta charset="utf-8" />
<head> <head>
<link rel='stylesheet' type='text/css' href='style.css'> <link rel='stylesheet' type='text/css' href='style.css'>
</head> </head>
<body> <body>
<div class="centered"> <div class="centered">
<table> <table>
<tr> <tr>
<td>Source Location</td> <td>Source Location</td>
<td>Hotness</td> <td>Hotness</td>
<td>Function</td> <td>Function</td>
<td>Pass</td> <td>Pass</td>
</tr>''', file=self.stream) </tr>''', file=self.stream)
max_entries = None max_entries = None
if self.should_display_hotness: if self.should_display_hotness:
max_entries = self.max_hottest_remarks_on_index max_entries = self.max_hottest_remarks_on_index
for i, remark in enumerate(all_remarks[:max_entries]): for i, remark in enumerate(all_remarks[:max_entries]):
if not suppress(remark): if not suppress(remark):
self.render_entry(remark, i % 2) self.render_entry(remark, i % 2)
print(''' print(u'''
</table> </table>
</body> </body>
</html>''', file=self.stream) </html>''', file=self.stream)
def _render_file(source_dir, output_dir, ctx, no_highlight, entry, filter_): def _render_file(source_dir, output_dir, ctx, no_highlight, entry, filter_):
global context global context
context = ctx context = ctx
▲ Show 20 Lines • Show All 138 Lines • Show Last 20 Lines

llvm/tools/opt-viewer/optrecord.py

#!/usr/bin/env python #!/usr/bin/env python
from __future__ import print_function from __future__ import print_function
import io
import yaml import yaml
# Try to use the C parser. # Try to use the C parser.
try: try:
from yaml import CLoader as Loader from yaml import CLoader as Loader
except ImportError: except ImportError:
print("For faster parsing, you may want to install libYAML for PyYAML") print("For faster parsing, you may want to install libYAML for PyYAML")
from yaml import Loader from yaml import Loader
▲ Show 20 Lines • Show All 254 Lines • ▼ Show 20 Lines
class Failure(Missed): class Failure(Missed):
yaml_tag = '!Failure' yaml_tag = '!Failure'
def get_remarks(input_file, filter_=None): def get_remarks(input_file, filter_=None):
max_hotness = 0 max_hotness = 0
all_remarks = dict() all_remarks = dict()
file_remarks = defaultdict(functools.partial(defaultdict, list)) file_remarks = defaultdict(functools.partial(defaultdict, list))
with open(input_file) as f: with io.open(input_file, encoding = 'utf-8') as f:
docs = yaml.load_all(f, Loader=Loader) docs = yaml.load_all(f, Loader=Loader)
filter_e = None filter_e = None
if filter_: if filter_:
filter_e = re.compile(filter_) filter_e = re.compile(filter_)
for remark in docs: for remark in docs:
remark.canonicalize() remark.canonicalize()
# Avoid remarks withoug debug location or if they are duplicated # Avoid remarks withoug debug location or if they are duplicated
▲ Show 20 Lines • Show All 62 Lines • Show Last 20 Lines