[debuginfo-tests][dexter] Add a test tool --calculate-average option

[Pierre-vh]

Summary

During our experiments with DExTer, we found that it lacked a way to calculate the average score of every test ran inside a directory. For instance, if I have a directory full of DExTer tests and I run them all using something like
python3 dexter.py /foo/mytests --builder clang --debugger lldb --cflags="-g -O2", I have no way to know the average score of all the tests that I just ran. So, we implemented this feature under`--calculate-average`, and we now propose to upstream it:

When using that option, DExTer will calculate the average score of every test run and print it after all the tests have run,.

Example output:

$ python3 dexter/dexter.py test ~/dexter/tests/nostdlib/llvm_passes/GVN --builder clang --debugger lldb --calculate-average

NumGVNEqProp: (0.1429)
NumGVNLoad: (0.1429)
NumGVNPRE: (0.1613)
NumPRELoad: (0.1549)

@avg: (0.1505)

[Orlando]

LGTM. Thanks for the patch, It's great to see someone using and contributing to dexter!

FYI It will help visibility if you add the debug-info tag to these patches.

debuginfo-tests/dexter/dex/tools/test/Tool.py

 # DExTer : Debugging Experience Tester
 # ~~~~~~   ~         ~~         ~   ~~
 #
 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 """Test tool."""
 
+import math
 import os
 import csv
 import pickle
 import shutil
 
 from dex.builder import run_external_build_script
 from dex.debugger.Debuggers import get_debugger_steps
 from dex.heuristic import Heuristic
class Tool(TestToolBase):
 
     def add_tool_arguments(self, parser, defaults):
         parser.add_argument('--fail-lt',
                             type=float,
                             default=0.0, # By default TEST always succeeds.
                             help='exit with status FAIL(2) if the test result'
                                 ' is less than this value.',
                             metavar='<float>')
+        parser.add_argument('--calculate-average',
+                            action="store_true",
+                            help='calculate the average score of every test run')
         super(Tool, self).add_tool_arguments(parser, defaults)
 
     def _build_test_case(self):
         """Build an executable from the test source with the given --builder
         script and flags (--cflags, --ldflags) in the working directory.
         Or, if the --binary option has been given, copy the executable provided
         into the working directory and rename it to match the --builder output.
         """
class Tool(TestToolBase):
 
     def _handle_results(self) -> ReturnCode:
         return_code = ReturnCode.OK
         options = self.context.options
 
         if not options.verbose:
             self.context.o.auto('\n')
 
+        if options.calculate_average:
+            # Calculate and print the average score
+            score_sum = 0.0
+            num_tests = 0
+            for test_case in self._test_cases:
+                score = test_case.score
+                if not test_case.error and not math.isnan(score):
+                    score_sum += test_case.score
+                    num_tests += 1
+
+            if num_tests != 0:
+                print("@avg: ({:.4f})".format(score_sum/num_tests))
+
         summary_path = os.path.join(options.results_directory, 'summary.csv')
         with open(summary_path, mode='w', newline='') as fp:
             writer = csv.writer(fp, delimiter=',')
             writer.writerow(['Test Case', 'Score', 'Error'])
 
             for test_case in self._test_cases:
                 if (test_case.score < options.fail_lt or
                         test_case.error is not None):
                     return_code = ReturnCode.FAIL
 
                 writer.writerow([
                     test_case.name, '{:.4f}'.format(test_case.score),
                     test_case.error
                 ])
 
         return return_code