[utils] Add the llvm-locstats tool

[djtodoro]

Summary

As we discussed on the D65585, this patch introduces the llvm-locstats tool.

The line 0 shows the number (and the percentage) of DIEs with no location information, but the line 100 shows the number (and the percentage) of DIEs where there is location information in all code section bytes (where the variable or parameter is in the scope). The line 51..59 shows the number (and the percentage) of DIEs where the location information is in between 51 and 59 percentage of its scope covered. There are options for considering only local variables or formal parameters, and also there is an option for reporting the stats by ignoring the debug entry values generated.

Using the tool

llvm-locstats test.o

=================================================
          Debug Location Statistics
=================================================
      cov%          samples       percentage(~)
-------------------------------------------------
        0                 1              16%
     1..9                 0               0%
   10..19                 0               0%
   20..29                 0               0%
   30..39                 0               0%
   40..49                 0               0%
   50..99                 1              16%
   60..69                 0               0%
   70..79                 0               0%
   80..89                 1              16%
   90..99                 0               0%
      100                 3              50%
=================================================
-the number of debug variables processed: 6
-PC ranges covered: 81%
-------------------------------------------------
-total availability: 83%
=================================================

Or,

llvm-locstats -ignore-debug-entry-values test.o

=================================================
           Debug Location Statistics
=================================================
      cov%          samples       percentage(~)
-------------------------------------------------
        0                 1              16%
     1..9                 0               0%
   10..19                 0               0%
   20..29                 0               0%
   30..39                 0               0%
   40..49                 0               0%
   50..99                 2              33%
   60..69                 0               0%
   70..79                 0               0%
   80..89                 1              16%
   90..99                 0               0%
      100                 2              33%
=================================================
-the number of debug variables processed: 6
-PC ranges covered: 76%
-------------------------------------------------
-total availability: 83%
=================================================

[vsk]

I really like this approach. Thanks for doing this!

llvm/utils/llvm-locstats/llvm-locstats.py

108

Please use subprocess.call or Popen along with PIPE to avoid round-tripping to the filesystem, leaving stale files around by accident, etc.

110

Please use os.path.join to support Windows.

[Orlando]

llvm/utils/llvm-locstats/llvm-locstats.py

108

Would it be worth using tempfile.NamedTemporaryFile instead of hard coding the temporary file name here?

[djtodoro]

@vsk

I really like this approach. Thanks for doing this!

Thanks!

llvm/utils/llvm-locstats/llvm-locstats.py

108

Please use subprocess.call or Popen along with PIPE to avoid round-tripping to the filesystem, leaving stale files around by accident, etc.

Sure.

Would it be worth using tempfile.NamedTemporaryFile instead of hard coding the temporary file name here?

I think so, thanks!

110

Sure.

[djtodoro]

-Rebase
-Use the tempfile.TemporaryFile
-Use subprocess.calll
-Use os.path.join

[vsk]

llvm/utils/llvm-locstats/llvm-locstats.py

168

Still not sure why this needs to touch the filesystem. Why not use a pipe? Like:

subproc = Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE, universal_newlines = True)
cmd_stdout, cmd_stderr = subproc.communicate()

214

I think it'd be great to simplify this. Note that "1-9%" is covered, as is "11-19%", but that "10%" is missed. It should be possible to write a helper to get the buckets, like:

def coverage_buckets():
  yield '0%'
  for start in range(1, 92, 10):
    yield '{0}%-{1}%'.format(start, start + 9)
  yield '100%'

You can use a helper like that to use a for-loop to extract all of the "vars with $bucket of its scope covered" fields, etc.

255

Ditto here and everywhere else we need to look up coverage buckets within json_parsed.

[djtodoro]

• Simplify the code
• Rebase

@vsk Thanks for your comments!

[vsk]

llvm/utils/llvm-locstats/llvm-locstats.py

8

I think this either needs '#!/usr/bin/env python3' or 'from future import print_function'.

176

What's the difference between this ('total variables procesed by location statistics') and 'total vars procesed by location statistics' above?

[djtodoro]

llvm/utils/llvm-locstats/llvm-locstats.py

8

I see.. Thanks!

The other LLVM python scripts uses the print function from future package, so I will use that as well.

176

I just followed the existing name convention within llvm-dwarfdump statistics. Reporting stats with "variables" applies to both local variables and formal parameters, "vars" applies only to local variables and "formal parameters" obviously applies only to formal parameters.

[djtodoro]

-Use `print_function ` from `__future__`

[vsk]

Thanks, this looks good to me. Perhaps you can get more feedback by dropping the WIP label. @aprantl, any thoughts?

llvm/utils/llvm-locstats/llvm-locstats.py

176

Ok. Perhaps we can revisit that & pick a clearer naming convention later.

[djtodoro]

llvm/utils/llvm-locstats/llvm-locstats.py

176

I agree with that.
Thanks a lot for the review and comments! I really appreciate it!

[djtodoro]

-Add the documentation
-Add the test case
-Adjust the CMake

[djtodoro]

Is this OK to go?

@aprantl Do you have any additional comments ? :)

[djtodoro]

This was reverted with the rL371527 due to test failure on Windows platform.

[djtodoro]

-Adjust the CMake
-Adjust the tool to work on Windows (skip the backslash character from JSON string)
-Adjust the test case to work on Windows

[vsk]

utils/llvm-locstats/llvm-locstats.py

120 ↗

(On Diff #219933)

I'm a bit confused by this. Mind sharing a link / description for the error?

Did llvm-dwarfdump produce invalid (unescaped) JSON, thus causing python's JSON parser to fail?

[djtodoro]

utils/llvm-locstats/llvm-locstats.py

120 ↗

(On Diff #219933)

Sure.

When try to load the JSON from llvm-dwarfdump output with loads() from json package it trows an exception on Windows platform.

Invalid \escape: line 1 column 23 (char 23)

The reason of the exception is special character '\' from Windows path. I think that llvm-dwarfdump should handle that, so this won't be needed here.

[djtodoro]

utils/llvm-locstats/llvm-locstats.py

120 ↗

(On Diff #219933)

@vsk Do you think we should handle this on the llvm-dwarfdump side, since we have the JSON that breaks the specification (the JSON spec)?

[vsk]

utils/llvm-locstats/llvm-locstats.py

120 ↗

(On Diff #219933)

Absolutely, llvm-dwarfdump needs to produce valid json, as I'm sure there are clients that rely on that. I understand that may be more work than you signed up for :). If you don't have the cycles for that, let me know.

[djtodoro]

utils/llvm-locstats/llvm-locstats.py

120 ↗

(On Diff #219933)

This will be a small fix. Thanks for your comments and reviews! :)

[djtodoro]

-Rebase

[djtodoro]

@vsk I believe this can go again, according to the comment from D67699?

[vsk]

Yes, please go for it :)

[djtodoro]

@vsk Thanks! :)

[gchatelet]

This patch seems to have broken the build

llvm-lit: /redacted/git/llvm-project/llvm/utils/lit/lit/llvm/config.py:340: note: using clang: /tmp/llvm-project_dbg_compiled-with-clang/bin/clang
FAIL: LLVM :: tools/llvm-dwarfdump/X86/locstats.ll (47789 of 49472)
******************** TEST 'LLVM :: tools/llvm-dwarfdump/X86/locstats.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /tmp/llvm-project_dbg_compiled-with-clang/bin/llc -debug-entry-values /redacted/git/llvm-project/llvm/test/tools/llvm-dwarfdump/X86/locstats.ll -o - -filetype=obj    | /tmp/llvm-project_dbg_compiled-with-clang/bin/llvm-dwarfdump -statistics - | /tmp/llvm-project_dbg_compiled-with-clang/bin/FileCheck /redacted/git/llvm-project/llvm/test/tools/llvm-dwarfdump/X86/locstats.ll
: 'RUN: at line 4';   /tmp/llvm-project_dbg_compiled-with-clang/bin/llc -debug-entry-values /redacted/git/llvm-project/llvm/test/tools/llvm-dwarfdump/X86/locstats.ll -o /tmp/llvm-project_dbg_compiled-with-clang/test/tools/llvm-dwarfdump/X86/Output/locstats.ll.tmp0.o -filetype=obj    | '/usr/bin/python' /tmp/llvm-project_dbg_compiled-with-clang/./bin/llvm-locstats /tmp/llvm-project_dbg_compiled-with-clang/test/tools/llvm-dwarfdump/X86/Output/locstats.ll.tmp0.o | /tmp/llvm-project_dbg_compiled-with-clang/bin/FileCheck /redacted/git/llvm-project/llvm/test/tools/llvm-dwarfdump/X86/locstats.ll --check-prefix=LOCSTATS
--
Exit Code: 2

Command Output (stderr):
--
/usr/bin/python: can't open file '/tmp/llvm-project_dbg_compiled-with-clang/./bin/llvm-locstats': [Errno 2] No such file or directory
FileCheck error: '-' is empty.
FileCheck command line:  /tmp/llvm-project_dbg_compiled-with-clang/bin/FileCheck /redacted/git/llvm-project/llvm/test/tools/llvm-dwarfdump/X86/locstats.ll --check-prefix=LOCSTATS

--

[djtodoro]

@gchatelet What build is this about? Since I can not see the failure on the http://lab.llvm.org:8011/console.

[gchatelet]

In D66526#1678598, @djtodoro wrote:

@gchatelet What build is this about? Since I can not see the failure on the http://lab.llvm.org:8011/console.

@djtodoro my apologies I think it's a local cmake caching problem. Sorry for the noise.

[djtodoro]

@gchatelet No problem :) I am happy if that is the case!

[djtodoro]

I found a build failure on the sanitizer-x86_64-linux-fast. I will investigate that..

[gchatelet]

In D66526#1678628, @djtodoro wrote:

I found a build failure on the sanitizer-x86_64-linux-fast. I will investigate that..

Yes even with a clean build it's still failing, here is my reproducer:

cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=/usr/local/bin/clang -DCMAKE_ASM_COMPILER=/usr/local/bin/clang -DCMAKE_CXX_COMPILER=/usr/local/bin/clang++ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DBUILD_SHARED_LIBS=ON -DLLVM_USE_SPLIT_DWARF=ON -DLLVM_OPTIMIZED_TABLEGEN=ON '-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=ARC;AVR' -DLLVM_ENABLE_PROJECTS=clang -DLLVM_BUILD_TESTS=ON -DLLVM_ENABLE_WERROR=ON -H/redacted/llvm-project/llvm -B/tmp/llvm-project_dbg_compiled-with-clang -GNinja

ninja -C /tmp/llvm-project_dbg_compiled-with-clang check-all

[djtodoro]

This is reverted with the rL372580 while investigating.

The problem is within CMake changes. In some configurations the tool is not copied into the right directory, so the test fails due to the reason.

[djtodoro]

-Make the testcase unsupported when needed

@vsk WDYT?

[vsk]

This is checking for the existence of llvm-locstats before running the test? Seems reasonable to me, please go ahead and feel free to fix forwards. Thanks!

[djtodoro]

This is checking for the existence of llvm-locstats before running the test?

Yes.

Thanks again!!

[rupprecht]

I see this is reverted again; a couple comments while trying to import this:

llvm/trunk/test/tools/llvm-locstats/locstats.ll

1–2 ↗

(On Diff #222356)

The pipe should be deleted; llc is generating %t0.o as an intermediate object file and llvm-locstats is parsing that, not stdin. The stdout of llc is empty.

(If you want to support stdin, that should be a different test case)

llvm/trunk/utils/llvm-locstats/llvm-locstats.py

107–109 ↗

(On Diff #222356)

This assumes that llvm-dwarfdump exists in the same directory as llvm-locstats, which may not be true, e.g. for build systems that build dwardump and locstats in separate build trees. Can this be configurable? Possibly a regular $PATH lookup makes sense here (with cwd as a preferred location), although allowing a flag pointing to the location of dwarfdump would be the most flexible.

[rupprecht]

llvm/trunk/utils/llvm-locstats/llvm-locstats.py

107–109 ↗

(On Diff #222356)

(Note: feel free to reland this bit as-is, I can suggest this in the form of a patch once this landing has stuck).

[djtodoro]

llvm/trunk/test/tools/llvm-locstats/locstats.ll

1–2 ↗

(On Diff #222356)

Thanks! I have missed this..

llvm/trunk/utils/llvm-locstats/llvm-locstats.py

107–109 ↗

(On Diff #222356)

Sure... A custom build system could make situation like that, I also thought about some improvements, but that can go as separate patches.

(Note: feel free to reland this bit as-is, I can suggest this in the form of a patch once this landing has stuck).

Let's do that way. Please feel free to post such patch! :)

llvm/CMakeLists.txt

if (LLVM_INCLUDE_BENCHMARKS)
   set(BENCHMARK_DOWNLOAD_DEPENDENCIES OFF CACHE BOOL "Don't download dependencies" FORCE)
   set(BENCHMARK_ENABLE_GTEST_TESTS OFF CACHE BOOL "Disable Google Test in benchmark" FORCE)
   # Since LLVM requires C++11 it is safe to assume that std::regex is available.
   set(HAVE_STD_REGEX ON CACHE BOOL "OK" FORCE)
 
   add_subdirectory(utils/benchmark)
   add_subdirectory(benchmarks)
 endif()
+
+if (LLVM_INCLUDE_UTILS)
+  add_subdirectory(utils/llvm-locstats)
+endif()

llvm/tools/llvm-dwarfdump/CMakeLists.txt

 set(LLVM_LINK_COMPONENTS
   DebugInfoDWARF
   AllTargetsDescs
   AllTargetsInfos
   MC
   Object
   Support
   )
 
 add_llvm_tool(llvm-dwarfdump
   Statistics.cpp
   llvm-dwarfdump.cpp
   )
 
 add_subdirectory(fuzzer)
+add_dependencies(llvm-dwarfdump llvm-locstats)

llvm/utils/llvm-locstats/CMakeLists.txt

+add_custom_target(llvm-locstats
+  COMMAND  ${CMAKE_COMMAND} -E copy ${LLVM_MAIN_SRC_DIR}/utils/llvm-locstats/llvm-locstats.py ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/llvm-locstats
+  )
+set_target_properties(llvm-locstats PROPERTIES FOLDER "Utils")

llvm/utils/llvm-locstats/llvm-locstats.py

+#!/usr/bin/env python
+#
+# This is a tool that works like debug location coverage calculator.
+# It parses the llvm-dwarfdump --statistics output by reporting it
+# in a more human readable way.
+#
+
+from __future__ import print_function
+import argparse
+import os
+import sys
+from json import loads
+from math import ceil
+from subprocess import Popen, PIPE
+
+def coverage_buckets():
+  yield '0%'
+  yield '1-9%'
+  for start in range(10, 91, 10):
+    yield '{0}-{1}%'.format(start, start + 9)
+  yield '100%'
+
+def locstats_output(
+  variables_total,
+  variables_total_locstats,
+  variables_with_loc,
+  scope_bytes_covered,
+  scope_bytes_from_first_def,
+  variables_coverage_map
+  ):
+
+  pc_ranges_covered = int(ceil(scope_bytes_covered * 100.0)
+              / scope_bytes_from_first_def)
+  variables_coverage_per_map = {}
+  for cov_bucket in coverage_buckets():
+    variables_coverage_per_map[cov_bucket] = \
+      int(ceil(variables_coverage_map[cov_bucket] * 100.0) \
+               / variables_total_locstats)
+
+  print (' =================================================')
+  print ('            Debug Location Statistics       ')
+  print (' =================================================')
+  print ('     cov%          samples          percentage(~)  ')
+  print (' -------------------------------------------------')
+  for cov_bucket in coverage_buckets():
+    print ('   {0:6}        {1:8d}             {2:3d}%'. \
+      format(cov_bucket, variables_coverage_map[cov_bucket], \
+             variables_coverage_per_map[cov_bucket]))
+  print (' =================================================')
+  print (' -the number of debug variables processed: ' \
+    + str(variables_total_locstats))
+  print (' -PC ranges covered: ' + str(pc_ranges_covered) + '%')
+
+  # Only if we are processing all the variables output the total
+  # availability.
+  if variables_total and variables_with_loc:
+    total_availability = int(ceil(variables_with_loc * 100.0) \
+                                  / variables_total)
+    print (' -------------------------------------------------')
+    print (' -total availability: ' + str(total_availability) + '%')
+  print (' =================================================')
+
+def parse_program_args(parser):
+  parser.add_argument('-only-variables', action='store_true',
+            default=False,
+            help='calculate the location statistics only for '
+               'local variables'
+            )
+  parser.add_argument('-only-formal-parameters', action='store_true',
+            default=False,
+            help='calculate the location statistics only for '
+               'formal parameters'
+            )
+  parser.add_argument('-ignore-debug-entry-values', action='store_true',
+            default=False,
+            help='ignore the location statistics on locations with '
+               'entry values'
+            )
+  parser.add_argument('file_name', type=str, help='File to process')
+  return parser.parse_args()
+
+
+def Main():
+  parser = argparse.ArgumentParser()
+  results = parse_program_args(parser)
+
+  if len(sys.argv) < 2:
+    print ('error: Too few arguments.')
+    parser.print_help()
+    sys.exit(1)
+
+  if results.only_variables and results.only_formal_parameters:
+    print ('error: Please use just one only* option.')
+    parser.print_help()
+    sys.exit(1)
+
+  # These will be different due to different options enabled.
+  variables_total = None
+  variables_total_locstats = None
+  variables_with_loc = None
+  variables_scope_bytes_covered = None
+  variables_scope_bytes_from_first_def = None
+  variables_scope_bytes_entry_values = None
+  variables_coverage_map = {}
+  binary = results.file_name
+
+  # Get the directory of the LLVM tools.
+  llvm_dwarfdump_cmd = os.path.join(os.path.dirname(__file__), \
+                                    "llvm-dwarfdump")
+  # The statistics llvm-dwarfdump option.
+  llvm_dwarfdump_stats_opt = "--statistics"
+
+  subproc = Popen([llvm_dwarfdump_cmd, llvm_dwarfdump_stats_opt, binary], \
+                  stdin=PIPE, stdout=PIPE, stderr=PIPE, \
+                  universal_newlines = True)
+  cmd_stdout, cmd_stderr = subproc.communicate()
+
+  # Get the JSON and parse it.
+  json_parsed = None
+
+  try:
+    json_parsed = loads(cmd_stdout)
+  except:
+    print ('error: No valid llvm-dwarfdump statistics found.')
+    sys.exit(1)
+
+  if results.only_variables:
+    # Read the JSON only for local variables.
+    variables_total_locstats = \
+      json_parsed['total vars procesed by location statistics']
+    variables_scope_bytes_covered = \
+      json_parsed['vars scope bytes covered']
+    variables_scope_bytes_from_first_def = \
+      json_parsed['vars scope bytes total']
+    if not results.ignore_debug_entry_values:
+      for cov_bucket in coverage_buckets():
+        cov_category = "vars with {} of its scope covered".format(cov_bucket)
+        variables_coverage_map[cov_bucket] = json_parsed[cov_category]
+    else:
+      variables_scope_bytes_entry_values = \
+        json_parsed['vars entry value scope bytes covered']
+      variables_scope_bytes_covered = variables_scope_bytes_covered \
+         - variables_scope_bytes_entry_values
+      for cov_bucket in coverage_buckets():
+        cov_category = \
+          "vars (excluding the debug entry values) " \
+          "with {} of its scope covered".format(cov_bucket)
+        variables_coverage_map[cov_bucket] = json_parsed[cov_category]
+  elif results.only_formal_parameters:
+    # Read the JSON only for formal parameters.
+    variables_total_locstats = \
+      json_parsed['total params procesed by location statistics']
+    variables_scope_bytes_covered = \
+      json_parsed['formal params scope bytes covered']
+    variables_scope_bytes_from_first_def = \
+      json_parsed['formal params scope bytes total']
+    if not results.ignore_debug_entry_values:
+      for cov_bucket in coverage_buckets():
+        cov_category = "params with {} of its scope covered".format(cov_bucket)
+        variables_coverage_map[cov_bucket] = json_parsed[cov_category]
+    else:
+      variables_scope_bytes_entry_values = \
+        json_parsed['formal params entry value scope bytes covered']
+      variables_scope_bytes_covered = variables_scope_bytes_covered \
+        - variables_scope_bytes_entry_values
+      for cov_bucket in coverage_buckets():
+        cov_category = \
+          "params (excluding the debug entry values) " \
+          "with {} of its scope covered".format(cov_bucket)
+  else:
+    # Read the JSON for both local variables and formal parameters.
+    variables_total = \
+      json_parsed['source variables']
+    variables_with_loc = json_parsed['variables with location']
+    variables_total_locstats = \
+      json_parsed['total variables procesed by location statistics']
+    variables_scope_bytes_covered = \
+      json_parsed['scope bytes covered']
+    variables_scope_bytes_from_first_def = \
+      json_parsed['scope bytes total']
+    if not results.ignore_debug_entry_values:
+      for cov_bucket in coverage_buckets():
+        cov_category = "variables with {} of its scope covered". \
+                       format(cov_bucket)
+        variables_coverage_map[cov_bucket] = json_parsed[cov_category]
+    else:
+      variables_scope_bytes_entry_values = \
+        json_parsed['entry value scope bytes covered']
+      variables_scope_bytes_covered = variables_scope_bytes_covered \
+        - variables_scope_bytes_entry_values
+      for cov_bucket in coverage_buckets():
+        cov_category = "variables with {} of its scope covered". \
+                       format(cov_bucket)
+        variables_coverage_map[cov_bucket] = json_parsed[cov_category]
+  # Pretty print collected info.
+  locstats_output(
+    variables_total,
+    variables_total_locstats,
+    variables_with_loc,
+    variables_scope_bytes_covered,
+    variables_scope_bytes_from_first_def,
+    variables_coverage_map
+    )
+
+if __name__ == '__main__':
+  Main()