[llvm-locstats] Add the --compare option

[djtodoro]

Summary

Draw a plot showing the difference in debug loc coverage on two
files provided.

[djtodoro]

An example of the plot on a GDB binary built with LLVM Trunk using the entry values vs. no entry values.

[dstenb]

Some nits.

llvm/utils/llvm-locstats/llvm-locstats.py

125

spaces around equals for the width parameter.

129

spaces around equals for the width parameter.

333

whoing -> showing ?

[aprantl]

How ubiquitous is matplotlib? I wonder if it might be better to textually print out a gnuplot file or something similar. OTOH the surface area in the code seems to be really small, so this may be fine.

[djtodoro]

@aprantl Thanks for the comment, sorry for the late response, I was on the vacation.

In D71870#1810823, @aprantl wrote:

How ubiquitous is matplotlib? I wonder if it might be better to textually print out a gnuplot file or something similar. OTOH the surface area in the code seems to be really small, so this may be fine.

I see the matplotlib is used within some of the LLVM Python tools, but I can investigate this further in order to see if the gnuplot is better option for this.

llvm/utils/llvm-locstats/llvm-locstats.py

333

Thanks!

[djtodoro]

-Addressing comments

[djtodoro]

In D71870#1816443, @djtodoro wrote:

@aprantl Thanks for the comment, sorry for the late response, I was on the vacation.

In D71870#1810823, @aprantl wrote:

How ubiquitous is matplotlib? I wonder if it might be better to textually print out a gnuplot file or something similar. OTOH the surface area in the code seems to be really small, so this may be fine.

I see the matplotlib is used within some of the LLVM Python tools, but I can investigate this further in order to see if the gnuplot is better option for this.

@aprantl For this option I would keep using the matplotlib (there is a help message saying we need the library, please see e.g. clang/utils/analyzer/CmpRuns.py).
Although, I think the gnuplot could be useful as well, but I don't see any obstacle that both ways of plotting coexist here, so we can add another feature to the llvm-locstats tool generating a gnuplot text file. WDYT?

[aprantl]

In D71870#1819207, @djtodoro wrote:

In D71870#1816443, @djtodoro wrote:

@aprantl Thanks for the comment, sorry for the late response, I was on the vacation.

In D71870#1810823, @aprantl wrote:

How ubiquitous is matplotlib? I wonder if it might be better to textually print out a gnuplot file or something similar. OTOH the surface area in the code seems to be really small, so this may be fine.

I see the matplotlib is used within some of the LLVM Python tools, but I can investigate this further in order to see if the gnuplot is better option for this.

@aprantl For this option I would keep using the matplotlib (there is a help message saying we need the library, please see e.g. clang/utils/analyzer/CmpRuns.py).
Although, I think the gnuplot could be useful as well, but I don't see any obstacle that both ways of plotting coexist here, so we can add another feature to the llvm-locstats tool generating a gnuplot text file. WDYT?

Gnuplot also has a weird license and I don't necessarily want to introduce it as a dependency either. But some form of dumping the raw data that is being graphed into some sort of machine-readable output (perhaps CSV?) could be useful for importing it into other tools like spreadsheets, pgfplot, gnuplot and so on.

[djtodoro]

In D71870#1820417, @aprantl wrote:

In D71870#1819207, @djtodoro wrote:

In D71870#1816443, @djtodoro wrote:

@aprantl Thanks for the comment, sorry for the late response, I was on the vacation.

In D71870#1810823, @aprantl wrote:

How ubiquitous is matplotlib? I wonder if it might be better to textually print out a gnuplot file or something similar. OTOH the surface area in the code seems to be really small, so this may be fine.

I see the matplotlib is used within some of the LLVM Python tools, but I can investigate this further in order to see if the gnuplot is better option for this.

@aprantl For this option I would keep using the matplotlib (there is a help message saying we need the library, please see e.g. clang/utils/analyzer/CmpRuns.py).
Although, I think the gnuplot could be useful as well, but I don't see any obstacle that both ways of plotting coexist here, so we can add another feature to the llvm-locstats tool generating a gnuplot text file. WDYT?

Gnuplot also has a weird license and I don't necessarily want to introduce it as a dependency either. But some form of dumping the raw data that is being graphed into some sort of machine-readable output (perhaps CSV?) could be useful for importing it into other tools like spreadsheets, pgfplot, gnuplot and so on.

That sounds like a very good idea! Having something like that will be useful, and we should make a proposal for that. Thanks for the review!

llvm/docs/CommandGuide/llvm-locstats.rst

.. option:: --ignore-debug-entry-values
             ignore the location statistics on locations containing the
             debug entry values DWARF operation
 
 .. option:: --draw-plot
 
             make histogram of location buckets generated (requires
             matplotlib)
 
+.. option:: --compare
+
+            compare the debug location coverage on two files provided, and draw
+            a plot showing the difference (requires matplotlib)
+
 EXIT STATUS
 -----------
 
 :program:`llvm-locstats` returns 0 if the input file were parsed
 successfully. Otherwise, it returns 1.
 
 EXAMPLE 1
 --------------

 
 .. code-block:: none
 
   llvm-locstats --draw-plot file1.out
 
 .. image:: locstats-draw-plot.png
   :align: center
 
+EXAMPLE 3
+--------------
+
+Generate a plot as an image file showing the difference in the debug location
+coverage.
+
+.. code-block:: none
+
+  llvm-locstats --compare file1.out file1.withentryvals.out
+
+.. image:: locstats-compare.png
+  :align: center
+
 SEE ALSO
 --------
 
 :manpage:`llvm-dwarfdump(1)`

llvm/utils/llvm-locstats/llvm-locstats.py

 #!/usr/bin/env python
 #
 # This is a tool that works like debug location coverage calculator.
 # It parses the llvm-dwarfdump --statistics output by reporting it
 # in a more human readable way.
 #
 
 from __future__ import print_function
 import argparse
 import os
 import sys
 from json import loads
 from math import ceil
 from collections import OrderedDict
 from subprocess import Popen, PIPE
 
+# Initialize the plot.
+def init_plot(plt):
+  plt.title('Debug Location Statistics', fontweight='bold')
+  plt.xlabel('location buckets')
+  plt.ylabel('number of variables in the location buckets')
+  plt.xticks(rotation=45, fontsize='x-small')
+  plt.yticks()
+
+# Finalize the plot.
+def finish_plot(plt):
+  plt.legend()
+  plt.grid(color='grey', which='major', axis='y', linestyle='-', linewidth=0.3)
+  plt.savefig('locstats.png')
+  print('The plot was saved within "locstats.png".')
+
 # Holds the debug location statistics.
 class LocationStats:
   def __init__(self, file_name, variables_total, variables_total_locstats,
     variables_with_loc, variables_scope_bytes_covered, variables_scope_bytes,
     variables_coverage_map):
     self.file_name = file_name
     self.variables_total = variables_total
     self.variables_total_locstats = variables_total_locstats
if self.variables_total and self.variables_with_loc:
       print (' -------------------------------------------------')
       print (' -total availability: ' + str(total_availability) + '%')
     print (' =================================================')
 
     return 0
 
   # Draw a plot representing the location buckets.
   def draw_plot(self):
-    try:
-      import matplotlib
-    except ImportError:
-      print('error: matplotlib not found.')
-      sys.exit(1)
-
     from matplotlib import pyplot as plt
 
     buckets = range(len(self.variables_coverage_map))
     plt.figure(figsize=(12, 8))
-    plt.title('Debug Location Statistics', fontweight='bold')
-    plt.xlabel('location buckets')
-    plt.ylabel('number of variables in the location buckets')
+    init_plot(plt)
     plt.bar(buckets, self.variables_coverage_map.values(), align='center',
             tick_label=self.variables_coverage_map.keys(),
             label='variables of {}'.format(self.file_name))
-    plt.xticks(rotation=45, fontsize='x-small')
-    plt.yticks()
 
     # Place the text box with the coverage info.
     pc_ranges_covered = self.get_pc_coverage()
     props = dict(boxstyle='round', facecolor='wheat', alpha=0.5)
     plt.text(0.02, 0.90, 'PC ranges covered: {}%'.format(pc_ranges_covered),
              transform=plt.gca().transAxes, fontsize=12,
              verticalalignment='top', bbox=props)
-    plt.legend()
-    plt.grid(color='grey', which='major', axis='y', linestyle='-', linewidth=0.3)
 
-    plt.savefig('locstats.png')
-    print('The plot was saved within "locstats.png".')
+    finish_plot(plt)
+
+  # Compare the two LocationStats objects and draw a plot showing
+  # the difference.
+  def draw_location_diff(self, locstats_to_compare):
+    from matplotlib import pyplot as plt
+
+    pc_ranges_covered = self.get_pc_coverage()
+    pc_ranges_covered_to_compare = locstats_to_compare.get_pc_coverage()
+
+    buckets = range(len(self.variables_coverage_map))
+    buckets_to_compare = range(len(locstats_to_compare.variables_coverage_map))
+
+    fig = plt.figure(figsize=(12, 8))
+    ax = fig.add_subplot(111)
+    init_plot(plt)
+
+    ax.bar(buckets, self.variables_coverage_map.values(), align='edge',
+           tick_label=self.variables_coverage_map.keys(), width=0.4,
+           label='variables of {}'.format(self.file_name))
+    ax.bar(buckets_to_compare,
+           locstats_to_compare.variables_coverage_map.values(),
+           color='r', align='edge', width=-0.4,
+           tick_label=locstats_to_compare.variables_coverage_map.keys(),
+           label='variables of {}'.format(locstats_to_compare.file_name))
+
+    props = dict(boxstyle='round', facecolor='wheat', alpha=0.5)
+    plt.text(0.02, 0.88,
+             '{} PC ranges covered: {}%'. \
+             format(self.file_name, pc_ranges_covered),
+             transform=plt.gca().transAxes, fontsize=12,
+             verticalalignment='top', bbox=props)
+    plt.text(0.02, 0.83,
+             '{} PC ranges covered: {}%'. \
+             format(locstats_to_compare.file_name,
+                    pc_ranges_covered_to_compare),
+             transform=plt.gca().transAxes, fontsize=12,
+             verticalalignment='top', bbox=props)
+
+    finish_plot(plt)
 
 # Define the location buckets.
 def coverage_buckets():
   yield '0%'
   yield '(0%,10%)'
   for start in range(10, 91, 10):
     yield '[{0}%,{1}%)'.format(start, start + 10)
   yield '100%'
parser.add_argument('--only-formal-parameters', action='store_true',
             help='calculate the location statistics only for formal parameters')
   parser.add_argument('--ignore-debug-entry-values', action='store_true',
             default=False,
             help='ignore the location statistics on locations with '
                  'entry values')
   parser.add_argument('--draw-plot', action='store_true', default=False,
             help='show histogram of location buckets generated (requires '
                  'matplotlib)')
-  parser.add_argument('file_name', type=str, help='file to process')
+  parser.add_argument('--compare', action='store_true', default=False,
+            help='compare the debug location coverage on two files provided, '
+                 'and draw a plot showing the difference  (requires '
+                 'matplotlib)')
+  parser.add_argument('file_names', nargs='+', type=str, help='file to process')
 
   return parser.parse_args()
 
 # Verify that the program inputs meet the requirements.
 def verify_program_inputs(opts):
   if len(sys.argv) < 2:
     print ('error: Too few arguments.')
     return False
 
   if opts.only_variables and opts.only_formal_parameters:
     print ('error: Please use just one --only* option.')
     return False
 
+  if not opts.compare and len(opts.file_names) != 1:
+    print ('error: Please specify only one file to process.')
+    return False
+
+  if opts.compare and len(opts.file_names) != 2:
+    print ('error: Please specify two files to process.')
+    return False
+
+  if opts.draw_plot or opts.compare:
+    try:
+      import matplotlib
+    except ImportError:
+      print('error: matplotlib not found.')
+      return False
+
   return True
 
 def Main():
   parser = argparse.ArgumentParser()
   opts = parse_program_args(parser)
 
   if not verify_program_inputs(opts):
     parser.print_help()
     sys.exit(1)
 
-  binary = opts.file_name
-  locstats = parse_locstats(opts, binary)
+  binary_file = opts.file_names[0]
+  locstats = parse_locstats(opts, binary_file)
 
+  if not opts.compare:
     if opts.draw_plot:
       # Draw a histogram representing the location buckets.
       locstats.draw_plot()
     else:
       # Pretty print collected info on the standard output.
       if locstats.pretty_print() == -1:
         sys.exit(0)
+  else:
+    binary_file_to_compare = opts.file_names[1]
+    locstats_to_compare = parse_locstats(opts, binary_file_to_compare)
+    # Draw a plot showing the difference in debug location coverage between
+    # two files.
+    locstats.draw_location_diff(locstats_to_compare)
 
 if __name__ == '__main__':
   Main()
   sys.exit(0)