[test-suite] Implement a new option: --single-result


[test-suite] Implement a new option: --single-result

The idea behind this option is to ease bisecting using tools like 'llvmlab bisect'. With --only-test we can already narrow down the tests we want to run to one test, however we still produce a report.json and exit success no matter if the test passed or failed.

With --single-result, we only perform one test (it sets --only-test) but the exit status of LNT is determined by a predicate which allows it to be used directly with llvmlab exec / llvmlab bisect.

The predicate is set with --single-result-predicate, and is a python expression. It is evaluated in a context that contains 'status', a boolean representing the pass or fail status of the test, and all of the metrics exposed by LIT. Where metrics have different names in LIT and LNT parlance (for example 'exec' in LNT and 'exec_time' in LIT) both names are exposed. This has a sideeffect of working around a feature of Python - 'exec' is a keyword so cannot be used as a variable!

The default predicate is simply "status". This causes the exit status of LNT to correspond to the pass/fail status of the test - useful for conformance testing. Predicates such as "status and exec_time > 6.0" allow for simple performance bisections.

We've been using this feature internally for a few weeks now and have found our bisections have become a lot easier.


jamesmMar 14 2016, 9:50 AM
rL263444: [mips] MIPS32R6 compact branch support