[opt-viewer] Don't Decode HTML bytes for Python 2

[lebedev.ri]

Summary

D36624 added some python3 compatibility. But that fix has a problem..

With python2 (which is specified by #!/usr/bin/env python2.7), if the env variables do not specify the UTF8,
and the source file is UTF8 (contains non-ASCII symbols), then the .decode('utf-8') causes the following exception:

Reading YAML files...
Rendering HTML files...
        8 of 41Traceback (most recent call last):
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 277, in <module>
    print_progress)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 213, in generate_report
    should_print_progress)
  File "/build/llvm/tools/opt-viewer/optpmap.py", line 45, in pmap
    result = map(_wrapped_func, func_and_args, *args, **kwargs)
  File "/build/llvm/tools/opt-viewer/optpmap.py", line 25, in _wrapped_func
    return func(argument)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 174, in _render_file
    SourceFileRenderer(source_dir, output_dir, filename).render(remarks)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 125, in render
    self.render_source_lines(self.source_stream, line_remarks)
  File "/build/llvm/tools/opt-viewer/opt-viewer.py", line 79, in render_source_lines
    </tr>'''.format(**locals()), file=self.stream)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf4' in position 47: ordinal not in range(128)

This is similar to https://bugs.llvm.org/show_bug.cgi?id=33548, which was fixed by https://reviews.llvm.org/D37661

Unlike that fix, here, *removing* .decode('utf-8') actually fixes it.

Since i assume that the original fix is needed, i simply made
that fix conditional, since for python2 it actually breaks things.

[lebedev.ri]

Ping.

[lebedev.ri]

Ping.

[anemet]

Sorry about the delay! I remembered something similar also for Python2 so I wanted to doublecheck. Turns out that was https://reviews.llvm.org/D29802 which is unrelated.

LGTM with the additional comment.

@modocache, any opinion about this?

tools/opt-viewer/opt-viewer.py

69 ↗

(On Diff #116696)

Please add a comment:

On Python 3, pygments.highlight() returns a bytes object, not
a str.

[lebedev.ri]

In D38289#893294, @anemet wrote:

Sorry about the delay! I remembered something similar also for Python2 so I wanted to doublecheck. Turns out that was https://reviews.llvm.org/D29802 which is unrelated.

LGTM with the additional comment.

Thank you!

@modocache, any opinion about this?

Since the change is trivial i'll proceed to commit it now.

[anemet]

HI @lebedev.ri,

I had to further tweak this in rL320725. Let me know if you see any issues.

Thanks,
Adam

llvm/trunk/tools/opt-viewer/opt-viewer.py

 #!/usr/bin/env python2.7
 
 from __future__ import print_function
 
 import argparse
 import cgi
 import errno
 import functools
 from multiprocessing import cpu_count
 import os.path
 import re
 import shutil
+import sys
 
 from pygments import highlight
 from pygments.lexers.c_cpp import CppLexer
 from pygments.formatters import HtmlFormatter
 
 import optpmap
 import optrecord
 
</html>
         self.html_formatter = HtmlFormatter(encoding='utf-8')
         self.cpp_lexer = CppLexer(stripnl=False)
 
     def render_source_lines(self, stream, line_remarks):
         file_text = stream.read()
         html_highlighted = highlight(
             file_text,
             self.cpp_lexer,
-            self.html_formatter).decode('utf-8')
+            self.html_formatter)
+
+        # On Python 3, pygments.highlight() returns a bytes object, not a str.
+        if sys.version_info >= (3, 0):
+          html_highlighted = html_highlighted.decode('utf-8')
 
         # Take off the header and footer, these must be
         #   reapplied line-wise, within the page structure
         html_highlighted = html_highlighted.replace('<div class="highlight"><pre>', '')
         html_highlighted = html_highlighted.replace('</pre></div>', '')
 
         for (linenum, html_line) in enumerate(html_highlighted.split('\n'), start=1):
             print('''