New utility to visualize optimization records

[anemet]

Summary

This is a new tool built on top of the new YAML ouput generated from
optimization remarks. It produces HTML for easy navigation and
visualization.

The tool assumes that hotness information for the remarks is available
(the YAML file was produced with PGO). It uses hotness to list the
remark prioritized by the hotness on the index page. Clicking the
source location of the remark in the list takes you the source where the
remarks are rendedered inlined in the source.

For now the tool is meant as prototype.

It's written in Python. It uses PyYAML to parse the input.

[anemet]

A small example:

example.tar8 KBDownload

[hfinkel]

This is neat, thanks!

So long as we're using Python, any though on using something like http://pygments.org/ to do syntax highlighting?

In any case, I suspect we'll want to change the interface a lot, but I'm happy for you to commit this and we can iterate in-tree.

utils/opt-viewer/opt-viewer.py

19

FWIW, we have an llvm-cxxfilt tool, perhaps we should use it?

85

<a tag here is missing its closing >

[anemet]

In D25348#564004, @hfinkel wrote:

This is neat, thanks!

Thank you.

So long as we're using Python, any though on using something like http://pygments.org/ to do syntax highlighting?

Yes, that was probably my main motivation too for going with Python ;). I'll take a look in a follow-up.

In any case, I suspect we'll want to change the interface a lot, but I'm happy for you to commit this and we can iterate in-tree.

Absolutely.

utils/opt-viewer/opt-viewer.py

19

I was thinking the same thing but I don't think that we currently install that in the macOS toolchain :(

85

Yeah, I am very loose with those. Let me fix them before committing.

utils/opt-viewer/opt-viewer.py

+#!/usr/bin/env python2.7
+
+from __future__ import print_function
+
+desc = '''Generate HTML output to visualize optimization records from the YAML files
+generated with -fsave-optimization-record and -fdiagnostics-show-hotness.'''
+
+import yaml
+import argparse
+import os.path
+import subprocess
+import shutil
+
+parser = argparse.ArgumentParser(description=desc)
+parser.add_argument('yaml_files', nargs='+')
+parser.add_argument('output_dir')
+args = parser.parse_args()
+
+p = subprocess.Popen(['c++filt', '-n'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
+def demangle(name):
+    p.stdin.write(name + '\n')
+    return p.stdout.readline().rstrip()
+
+class Remark(yaml.YAMLObject):
+    @property
+    def File(self):
+        return self.DebugLoc['File']
+
+    @property
+    def Line(self):
+        return int(self.DebugLoc['Line'])
+
+    @property
+    def Column(self):
+        return self.DebugLoc['Column']
+
+    def getDebugLoc(self):
+        return "{}:{}:{}".format(self.File, self.Line, self.Column)
+
+    def getLink(self):
+        return "{}#L{}".format(SourceFileRenderer.html_file_name(self.File), self.Line)
+
+    def getArgString(self, pair):
+        if pair[0] == 'Callee' or pair[0] == 'Caller':
+            return demangle(pair[1])
+        return pair[1]
+
+    @property
+    def message(self):
+        # Args is a list of mappings (dictionaries) with each dictionary with
+        # exactly one key-value pair.
+        values = [self.getArgString(mapping.items()[0]) for mapping in self.Args]
+        return demangle("".join(values))
+
+class Analysis(Remark):
+    yaml_tag = '!Analysis'
+
+    @property
+    def color(self): return "white"
+
+class AnalysisFPCommute(Analysis):
+    yaml_tag = '!AnalysisFPCommute'
+
+class AnalysisAliasing(Analysis):
+    yaml_tag = '!AnalysisAliasing'
+
+class Passed(Remark):
+    yaml_tag = '!Passed'
+
+    @property
+    def color(self): return "green"
+
+class Missed(Remark):
+    yaml_tag = '!Missed'
+
+    @property
+    def color(self): return "red"
+
+class SourceFileRenderer:
+    def __init__(self, filename):
+        self.source_stream = open(filename)
+        self.stream = open(os.path.join(args.output_dir, SourceFileRenderer.html_file_name(filename)), 'w')
+
+    def render_source_line(self, linenum, line):
+        print("<tr><td><a name=\"L{linenum}\"{linenum}</a><td><td><td><pre>{line}</tr>".format(
+            **locals()), file=self.stream)
+
+    def render_inline_remarks(self, r):
+        print("<tr><td><td>{r.Hotness}<td class=\"column-entry-{r.color}\">{r.Pass}<td class=\"column-entry-yellow\">{r.message}</tr>".format(
+            **locals()), file=self.stream)
+
+    def render(self, line_remarks):
+        print('''
+<html>
+<head>
+<link rel='stylesheet' type='text/css' href='style.css'>
+</head>
+<body>
+<div class="centered">
+<table>
+<tr>
+<td>Line
+<td>Hotness
+<td>Optimization
+<td>Source</tr>''', file=self.stream)
+        for (linenum, line) in enumerate(self.source_stream.readlines(), start=1):
+            self.render_source_line(linenum, line)
+            for remark in line_remarks.get(linenum, []):
+                self.render_inline_remarks(remark)
+        print('''
+</table>
+</body>
+</html>''', file=self.stream)
+
+    @classmethod
+    def html_file_name(cls, filename):
+        return filename.replace('/', '_') + ".html"
+
+class IndexRenderer:
+    def __init__(self):
+        self.stream = open(os.path.join(args.output_dir, 'index.html'), 'w')
+
+    def render_entry(self, remark):
+        html = SourceFileRenderer.html_file_name(remark.File)
+        link = "<a href={}>{}</a>".format(remark.getLink(), remark.getDebugLoc())
+
+        dem_name = demangle(remark.Function)
+        print("<tr><td>{}<td>{}<td>{}<td class=\"column-entry-{}\">{}</tr>".format(
+            link,
+            remark.Hotness, dem_name, remark.color, remark.Pass), file=self.stream)
+
+    def render(self, all_remarks):
+        print('''
+<html>
+<head>
+<link rel='stylesheet' type='text/css' href='style.css'>
+</head>
+<body>
+<div class="centered">
+<table>
+<tr>
+<td>Source Location
+<td>Hotness
+<td>Function
+<td>Pass</tr>''', file=self.stream)
+        for remark in all_remarks:
+            self.render_entry(remark)
+        print('''
+</table>
+</body>
+</html>''', file=self.stream)
+
+
+all_remarks = []
+file_remarks  = dict()
+
+for input_file in args.yaml_files:
+    f = open(input_file)
+    docs = yaml.load_all(f)
+    for remark in docs:
+        if hasattr(remark, 'Hotness'):
+            file_remarks.setdefault(remark.File, dict()).setdefault(remark.Line, []).append(remark);
+            all_remarks.append(remark)
+
+all_remarks = sorted(all_remarks, key=lambda r: r.Hotness, reverse=True)
+
+if not os.path.exists(args.output_dir):
+    os.mkdir(args.output_dir)
+
+for (filename, remarks) in file_remarks.iteritems():
+    SourceFileRenderer(filename).render(remarks)
+
+IndexRenderer().render(all_remarks)
+
+shutil.copy(os.path.join(os.path.dirname(os.path.realpath(__file__)), "style.css"), args.output_dir)

utils/opt-viewer/style.css

+.red {
+  background-color: #ffd0d0;
+}
+.cyan {
+  background-color: cyan;
+}
+body {
+  font-family: -apple-system, sans-serif;
+}
+pre {
+  margin-top: 0px !important;
+  margin-bottom: 0px !important;
+}
+.source-name-title {
+  padding: 5px 10px;
+  border-bottom: 1px solid #dbdbdb;
+  background-color: #eee;
+  line-height: 35px;
+}
+.centered {
+  display: table;
+  margin-left: left;
+  margin-right: auto;
+  border: 1px solid #dbdbdb;
+  border-radius: 3px;
+}
+.expansion-view {
+  background-color: rgba(0, 0, 0, 0);
+  margin-left: 0px;
+  margin-top: 5px;
+  margin-right: 5px;
+  margin-bottom: 5px;
+  border: 1px solid #dbdbdb;
+  border-radius: 3px;
+}
+table {
+  border-collapse: collapse;
+}
+.light-row {
+  background: #ffffff;
+  border: 1px solid #dbdbdb;
+}
+.column-entry {
+  text-align: right;
+}
+.column-entry-left {
+  text-align: left;
+}
+.column-entry-white {
+  text-align: right;
+  background-color: #ffffff;
+}
+.column-entry-red {
+  text-align: right;
+  background-color: #ffd0d0;
+}
+.column-entry-green {
+  text-align: right;
+  background-color: #d0ffd0;
+}
+.column-entry-yellow {
+  text-align: left;
+  background-color: #ffe1a6;
+}
+.line-number {
+  text-align: right;
+  color: #aaa;
+}
+.covered-line {
+  text-align: right;
+  color: #0080ff;
+}
+.uncovered-line {
+  text-align: right;
+  color: #ff3300;
+}
+.tooltip {
+  position: relative;
+  display: inline;
+  background-color: #b3e6ff;
+  text-decoration: none;
+}
+.tooltip span.tooltip-content {
+  position: absolute;
+  width: 100px;
+  margin-left: -50px;
+  color: #FFFFFF;
+  background: #000000;
+  height: 30px;
+  line-height: 30px;
+  text-align: center;
+  visibility: hidden;
+  border-radius: 6px;
+}
+.tooltip span.tooltip-content:after {
+  content: '';
+  position: absolute;
+  top: 100%;
+  left: 50%;
+  margin-left: -8px;
+  width: 0; height: 0;
+  border-top: 8px solid #000000;
+  border-right: 8px solid transparent;
+  border-left: 8px solid transparent;
+}
+:hover.tooltip span.tooltip-content {
+  visibility: visible;
+  opacity: 0.8;
+  bottom: 30px;
+  left: 50%;
+  z-index: 999;
+}
+th, td {
+  vertical-align: top;
+  padding: 2px 5px;
+  border-collapse: collapse;
+  border-right: solid 1px #eee;
+  border-left: solid 1px #eee;
+}
+td:first-child {
+  border-left: none;
+}
+td:last-child {
+  border-right: none;
+}