[LNT][test-suite] Implement a new option: --single-result

[jmolloy]

Summary

The idea behind this option is to ease bisecting using tools like 'llvmlab bisect'. With --only-test we can already narrow down the tests we want to run to one test, however we still produce a report.json and exit success no matter if the test passed or failed.

With --single-result, we only perform one test (it sets --only-test) but the exit status of LNT is determined by a predicate which allows it to be used directly with llvmlab exec / llvmlab bisect.

The predicate is set with --single-result-predicate, and is a python expression. It is evaluated in a context that contains 'status', a boolean representing the pass or fail status of the test, and all of the metrics exposed by LIT. Where metrics have different names in LIT and LNT parlance (for example 'exec' in LNT and 'exec_time' in LIT) both names are exposed. This has a sideeffect of working around a feature of Python - 'exec' is a keyword so cannot be used as a variable!

The default predicate is simply "status". This causes the exit status of LNT to correspond to the pass/fail status of the test - useful for conformance testing. Predicates such as "status and exec_time > 6.0" allow for simple performance bisections.

We've been using this feature internally for a week or so now and have found our bisections have become a lot easier. An example bisect command would be:

$ llvmlab bisect -b ARM-aarch32 --min-rev=261265 --max-rev=261369 \
    lnt runtest test-suite \
    --cc '%(path)s/bin/clang' \
    --sandbox SANDBOX \
    --test-suite /work/llvm-test-suite \
    --use-lit lit \
    --run-under 'taskset -c 5' \
    --cflags '-O3 -mthumb -mcpu=cortex-a57' \
    --single-result MultiSource/Benchmarks/TSVC/Expansion-flt/Expansion-flt.test \
    --single-result-predicate 'exec_time > 8.0'

[kristof.beyls]

lnt/tests/test_suite.py

154–158

The help text doesn't indicate that this argument expects a fully-qualified test name.
Also, it seems like this argument changes test execution - so is it best placed in the "Test Execution" group, or here in the "Output Options" group?

All in all, the help text is a bit long. Maybe it would be better to shorten the help text here and write some more elaborate documentation with an example for this in one of the rst files under docs, so that how to use this becomes part of the LNT documentation?
Should the documentation for the bisecter (which I guess lives in zorg) be updated to have an example making use of these options?

230–232

Don't we have directories with multiple tests in the same directory? e.g. test-suite/SingleSource/Regression/C
Does this mean that at most a full directory can be specified, not a single test?
Would using this to bisect a single test in a directory with multiple tests still work?

[MatzeB]

I believe this was committed in r263445, accepting & closing.

[MatzeB]

I believe this was committed in r263445, accepting & closing.

lnt/tests/test_suite.py

-import subprocess, tempfile, json, os, shlex, platform, pipes
+import subprocess, tempfile, json, os, shlex, platform, pipes, sys
 
 from optparse import OptionParser, OptionGroup
 
 import lnt.testing
 import lnt.testing.util.compilers
 from lnt.testing.util.misc import timestamp
 from lnt.testing.util.commands import note, warning, fatal
 from lnt.testing.util.commands import capture, mkdir_p, which
def run_test(self, name, args):
         group.add_option("-v", "--verbose", dest="verbose",
                          help="show verbose test results",
                          action="store_true", default=False)
         group.add_option("", "--exclude-stat-from-submission",
                          dest="exclude_stat_from_submission",
                          help="Do not submit the stat of this type [%default]",
                          action='append', choices=KNOWN_SAMPLE_KEYS,
                          type='choice', default=[])
+        group.add_option("", "--single-result", dest="single_result",
+                         help=("only execute one test and apply "
+                               "--single-result-predicate to calculate the "
+                               "exit status. Do not submit to a LNT instance "
+                               "or create a JSON report (useful for bisection)"))
+        group.add_option("", "--single-result-predicate",
+                         dest="single_result_predicate",
+                         help=("the predicate to apply to calculate the exit "
+                               "status (with --single-result). This is a Python "
+                               "expression that will be eval()ed with all metrics "
+                               "available as variables and a boolean 'status'. "
+                               "Example: \n"
+                               "  status and exec_time < 3.0"),
+                         default="status")
         parser.add_option_group(group)
 
         group = OptionGroup(parser, "Test tools")
         group.add_option("", "--use-cmake", dest="cmake", metavar="PATH",
                          type=str, default="cmake",
                          help="Path to CMake [cmake]")
         group.add_option("", "--use-make", dest="make", metavar="PATH",
                          type=str, default="make",
def run_test(self, name, args):
                 opts.test_suite_root))
 
         if opts.test_suite_externals:
             if not os.path.exists(opts.test_suite_externals):
                 parser.error(
                     "invalid --test-externals argument, does not exist: %r" % (
                         opts.test_suite_externals,))
 
+        if self.opts.single_result:
+            self.opts.only_test = os.path.dirname(self.opts.single_result)
+                
         opts.cmake = resolve_command_path(opts.cmake)
         if not isexecfile(opts.cmake):
             parser.error("CMake tool not found (looked for %s)" % opts.cmake)
         opts.make = resolve_command_path(opts.make)
         if not isexecfile(opts.make):
             parser.error("Make tool not found (looked for %s)" % opts.make)
         opts.lit = resolve_command_path(opts.lit)
         if not isexecfile(opts.lit):
def _parse_lit_output(self, path, data, only_test=False):
         ignore = self.opts.exclude_stat_from_submission
         if only_test:
             ignore.append('compile')
 
         for test_data in data['tests']:
             raw_name = test_data['name'].split(' :: ', 1)[1]
             name = 'nts.' + raw_name.rsplit('.test', 1)[0]
             is_pass = self._is_pass_code(test_data['code'])
+
+            # If --single-result is given, exit based on --single-result-predicate
+            if self.opts.single_result and \
+               raw_name == self.opts.single_result+'.test':
+                env = {'status': is_pass}
+                if 'metrics' in test_data:
+                    for k,v in test_data['metrics'].items():
+                        env[k] = v
+                        if k in LIT_METRIC_TO_LNT:
+                            env[LIT_METRIC_TO_LNT[k]] = v
+                status = eval(self.opts.single_result_predicate, {}, env)
+                sys.exit(0 if status else 1)
+
             if 'metrics' in test_data:
                 for k,v in test_data['metrics'].items():
                     if k not in LIT_METRIC_TO_LNT or LIT_METRIC_TO_LNT[k] in ignore:
                         continue
                     test_samples.append(
                         lnt.testing.TestSamples(name + '.' + LIT_METRIC_TO_LNT[k],
                                                 [v],
                                                 test_info,
                                                 LIT_METRIC_CONV_FN[k]))
 
             if self._test_failed_to_compile(raw_name, path):
                 test_samples.append(
                     lnt.testing.TestSamples(name + '.compile.status',
                                             [lnt.testing.FAIL],
                                             test_info))
 
             elif not is_pass:
                 test_samples.append(
                     lnt.testing.TestSamples(name + '.exec.status',
                                             [self._get_lnt_code(test_data['code'])],
                                             test_info))
 
+        if self.opts.single_result:
+            # If we got this far, the result we were looking for didn't exist.
+            raise RuntimeError("Result %s did not exist!" %
+                               self.opts.single_result)
 
         # FIXME: Add more machine info!
         run_info = {
             'tag': 'nts'
         }
         run_info.update(self._get_cc_info())
         run_info['run_order'] = run_info['inferred_run_order']
         if self.opts.run_order: