[LIT] Change the semantics of the LIT options '-s' and '-v' and modify when information is output.

[EricWF]

Summary

Currently I have benchmark tests that I run with LIT that have meaningful output even when the test passes. The aim of this patch is to modify the existing LIT output options in such a way that they can be used to always print the output of a test with minimal changes to how output is currently handled.

The main change in this patch is that -v can be specified multiple times to increase the amount of information that is output. Using lit -vv will cause LIT to report a passing tests output (if any is present).

This patch has a secondary drive-by change that changes when metrics are output. Currently metrics are reported when:

1. The test failed.
2. The test passed and neither '-s' nor '-q' are specified.

With this patch metrics are reported ONLY when '-s' is not specified and one of the following is true:

1. The test failed.
2. The test passed and neither '-s' nor '-q' are specified and '-vv' is not specified.
3. The test passed and '-vv' is specified.

[• ddunbar]

Hi Eric,

I like the feature change, but I find the implementation makes it harder to
see what the conditions are. I would like:

0. Use the label TEST ... OUTPUT when showing the output with no failures.

1. Keep the metrics output in their own section, and with the current

conditions. We can change that separately if necessary.

This should leave two boolean conditions to check, both of which can be
computed prior to "if shouldShow:..." and then we continue if both are
False.

Also, I would like to see a test that sanity checks for the expected
output, and an update to the docs describing the option.

• Daniel

[EricWF]

Ah, Sorry I never saw your response. I'll make the requested changes soon. Thank!

[EricWF]

Address @ddunbar's comments. I'm not quite sure what semantics you want for showing metrics on passing tests. Could you clarify?

1. Remove *most* changes regarding metrics. Metrics are now shown on passing tests with '-vv' as well.
2. Add tests for change.
3. Update doc for --verbose. (Wording suggestions welcome).

[EricWF]

Abandoning. I no longer need this.

docs/CommandGuide/lit.rst

 
 .. option:: -v, --verbose
 
- Show more information on test failures, for example the entire test output
- instead of just the test result.
+ Show more information about test results, for example the entire test output
+ instead of just the test result. This option can be specified more than once.
+ If ``--verbose`` is specified only once more information will be shown about
+ failing tests. If ``--verbose`` is specified more than once more information
+ will be shown for passing tests as well.
 
 .. option:: --no-progress-bar
 

utils/lit/lit/main.py

             self.progressBar.update(float(self.completed)/self.numTests,
                                     test.getFullName())
 
-        shouldShow = test.result.code.isFailure or \
+        shouldShow = test.result.code.isFailure or self.opts.showOutput > 1 or \
             (not self.opts.quiet and not self.opts.succinct)
         if not shouldShow:
             return

         print('%s: %s (%d of %d)' % (test.result.code.name, test_name,
                                      self.completed, self.numTests))
 
+        showPassingTest = self.opts.showOutput > 1 and test.result.output
+        showFailingTest = test.result.code.isFailure and self.opts.showOutput
         # Show the test failure output, if requested.
-        if test.result.code.isFailure and self.opts.showOutput:
-            print("%s TEST '%s' FAILED %s" % ('*'*20, test.getFullName(),
-                                              '*'*20))
+        if showFailingTest or showPassingTest:
+            print("%s TEST '%s' %s %s" %
+                  ('*'*20, test.getFullName(),
+                   'FAILED' if showFailingTest else 'OUTPUT', '*'*20))
             print(test.result.output)
             print("*" * 20)
 

                      action="store_true", default=False)
     group.add_option("-v", "--verbose", dest="showOutput",
                      help="Show all test output",
-                     action="store_true", default=False)
+                     action="count", default=0)
     group.add_option("-o", "--output", dest="output_path",
                      help="Write test results to the provided path",
                      action="store", type=str, metavar="PATH")

         for suite_name, suite in by_suite.items():
             safe_suite_name = suite_name.replace(".", "-")
             xunit_output_file.write("<testsuite name='" + safe_suite_name + "'")
-            xunit_output_file.write(" tests='" + str(suite['passes'] + 
+            xunit_output_file.write(" tests='" + str(suite['passes'] +
               suite['failures']) + "'")
-            xunit_output_file.write(" failures='" + str(suite['failures']) + 
+            xunit_output_file.write(" failures='" + str(suite['failures']) +
               "'>\n")
             for result_test in suite['tests']:
                 xunit_output_file.write(result_test.getJUnitXML() + "\n")

utils/lit/tests/Inputs/test-data/lit.cfg

         result = lit.Test.Result(getattr(lit.Test, result_code),
                                  result_output)
 
+        if not cfg.has_section('results'):
+            return result
+
         # Load additional metrics.
         for key,value_str in cfg.items('results'):
             value = eval(value_str)

utils/lit/tests/Inputs/test-data/metrics.ini

-[global]
-result_code = PASS
-result_output = Test passed.
-
-[results]
-value0 = 1
-value1 = 2.3456
 

utils/lit/tests/Inputs/test-data/test-metrics.ini

+[global]
+result_code = PASS
+result_output = Test passed.
+
+[results]
+value0 = 1
+value1 = 2.3456

utils/lit/tests/Inputs/test-data/test-verbose-failed.ini

+[global]
+result_code = FAIL
+result_output = This is the FAILING output.
+
+[results]
+value0 = 1
+value1 = 2.3456

utils/lit/tests/Inputs/test-data/test-verbose.ini

+[global]
+result_code = PASS
+result_output = This is the passing output.
+
+[results]
+value0 = 1
+value1 = 2.3456

utils/lit/tests/test-output.py

-# RUN: %{lit} -j 1 -v %{inputs}/test-data --output %t.results.out > %t.out
-# RUN: FileCheck < %t.results.out %s
+# RUN: %{lit} -v %{inputs}/test-data/test-metrics.ini --output %t.results.out > %t.out
+# RUN: FileCheck --check-prefix=CHECK < %t.results.out %s
 
 # CHECK: {
 # CHECK: "__version__"

 # CHECK-NEXT:       "value0": 1,
 # CHECK-NEXT:       "value1": 2.3456
 # CHECK-NEXT:     }
-# CHECK-NEXT:     "name": "test-data :: metrics.ini",
+# CHECK-NEXT:     "name": "test-data :: test-metrics.ini",
 # CHECK-NEXT:     "output": "Test passed."
 # CHECK-NEXT:   }
 # CHECK-NEXT: ]
 # CHECK-NEXT: }
+
+# RUN: %{lit} -j 1 -vv %{inputs}/test-data/test-verbose.ini > %t.out
+# RUN: FileCheck -check-prefix=CHECK-VERBOSE %s < %t.out
+
+# CHECK-VERBOSE: {{\*+}} TEST 'test-data :: test-verbose.ini' OUTPUT {{\*+}}
+# CHECK-VERBOSE-NEXT: This is the passing output.
+# CHECK-VERBOSE-NEXT: {{\*+}}
+# CHECK-VERBOSE-NEXT: {{\*+}} TEST 'test-data :: test-verbose.ini' RESULTS {{\*+}}
+
+# RUN: %{lit} -j 1 -v %{inputs}/test-data/test-verbose.ini > %t.out
+# RUN: FileCheck -check-prefix=CHECK-NONVERBOSE %s < %t.out
+
+# CHECK-NONVERBOSE-NOT: {{\*+}} TEST 'test-data :: test-verbose.ini' OUTPUT {{\*+}}
+# CHECK-NONVERBOSE-NOT: This is the passing output.
+# CHECK-NONVERBOSE: {{\*+}} TEST 'test-data :: test-verbose.ini' RESULTS {{\*+}}
+
+# RUN: not %{lit} -j 1 -vv %{inputs}/test-data/test-verbose-failed.ini > %t.out
+# RUN: FileCheck -check-prefix=CHECK-VERBOSE-FAIL %s < %t.out
+
+# CHECK-VERBOSE-FAIL: {{\*+}} TEST 'test-data :: test-verbose-failed.ini' FAILED {{\*+}}
+# CHECK-VERBOSE-FAIL-NEXT: This is the FAILING output.
+# CHECK-VERBOSE-FAIL-NEXT: {{\*+}}
+# CHECK-VERBOSE-FAIL-NEXT: {{\*+}} TEST 'test-data :: test-verbose-failed.ini' RESULTS {{\*+}}
+
+# RUN: not %{lit} -j 1 -v %{inputs}/test-data/test-verbose-failed.ini > %t.out
+# RUN: FileCheck -check-prefix=CHECK-NONVERBOSE-FAIL %s < %t.out
+
+# CHECK-NONVERBOSE-FAIL: {{\*+}} TEST 'test-data :: test-verbose-failed.ini' FAILED {{\*+}}
+# CHECK-NONVERBOSE-FAIL-NEXT: This is the FAILING output.
+# CHECK-NONVERBOSE-FAIL-NEXT: {{\*+}}
+# CHECK-NONVERBOSE-FAIL-NEXT: {{\*+}} TEST 'test-data :: test-verbose-failed.ini' RESULTS {{\*+}}