[lit] Provide extension API for custom result categories

[yln]

Summary

The lnt test suite defines custom result codes [1]. Support those via
an extension API instead of "by accident", which should offer the
advantage of properly handling them when we print test results.

[1] https://reviews.llvm.org/D77986

[jdoerfert]

Thanks, I'll check if I can run add_result_category from the test suite to resolve my crash :)

[yln]

Did you have an opportunity to try this? @jdoerfert

[Meinersbur]

AFAIU, test-suite would need to call add_result_category for its categories to resolve this? Is there anything more blocking this than being tried-out using test-suite?

[Meinersbur]

Note that this occurs problem occurs in our buildbots such as: http://lab.llvm.org:8011/builders/clang-native-arm-lnt-perf

[yln]

In D78164#2006362, @Meinersbur wrote:

AFAIU, test-suite would need to call add_result_category for its categories to resolve this? Is there anything more blocking this than being tried-out using test-suite?

No, I am only waiting for confirmation that this patch will work for test-suite. Friendly ping to @jdoerfert

[Meinersbur]

Confirming this together with D79064 works with the test suite.

[yln]

Can I get an official "Accept" so I can land this without a "Landed without approval" warning (and without accepting my own change). Thanks!

[abhinavgaba]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

Using MyFormat() here is causing check-lit to hang indefinitely on WIndows x86-64 (Python 2.7.6).

Changing this to use config.test_format = lit.format/ShTest() "fixes" the hang (although then the test fails, which is expected).

Is that a known issue?

[yln]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

I disabled the test on Windows in 3610fd8c5c6039c096d61c376b37a724a81c4d35. Would you be able to help me debug this? (I don't have access to a Windows machine.)

My best guess is that this has something to do with loading the class from the SITE_DIR:

import site
site.addsitedir(os.path.dirname(__file__))

https://github.com/llvm/llvm-project/blob/6cecd3c3dbef48eca6c4cf2dcc2df3290ab91488/llvm/utils/lit/tests/lit.cfg#L46
https://docs.python.org/3.9/library/site.html#site.USER_SITE

Can you also try Python3?

[abhinavgaba]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

Thanks. It does seem like a configuration issue. I'll try with python 3 (and a newer version of 2.7).

Another data point: with everything as is, this change fixes the hang as well:

diff --git a/llvm/utils/lit/tests/Inputs/custom-result-category/format.py b/llvm/utils/lit/tests/Inputs/custom-result-category/format.py
index b0c97ec7..a3258d3 100644
--- a/llvm/utils/lit/tests/Inputs/custom-result-category/format.py
+++ b/llvm/utils/lit/tests/Inputs/custom-result-category/format.py
@@ -8,11 +8,4 @@ lit.main.add_result_category(CUSTOM_PASS, "My Passed")
 lit.main.add_result_category(CUSTOM_FAILURE, "My Failed")


-class MyFormat(lit.formats.ShTest):
-    def execute(self, test, lit_config):
-        result = super(MyFormat, self).execute(test, lit_config)
-        if result.code.isFailure:
-            result.code = CUSTOM_FAILURE
-        else:
-            result.code = CUSTOM_PASS
-        return result
+MyFormat = lit.formats.ShTest

[abhinavgaba]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

I tried Python 2.7.12 and 3.7.4. It passes with 3.7.4 but hangs with 2.7.12.

[yln]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

The "fix" is interesting. This means that's it is not the error I suspected: loading the separate Python file via site.addsitedir() generally works. But then defining a class in it doesn't?! Strange.

I don't really have an idea how to narrow this down further. Considering that this is pretty obscure (maybe even a Python bug?) and seems to work in Python 3, I propose to keep this test disabled on Windows until we require Python3 to run lit itself.

[abhinavgaba]

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

9

Considering that this is pretty obscure (maybe even a Python bug?) and seems to work in Python 3, I propose to keep this test disabled on Windows until we require Python3 to run lit itself.

That sounds reasonable to me. Thanks.

llvm/utils/lit/lit/main.py

finally:
                 lit_config.warning("Failed to delete temp directory '%s'" % tmp_dir)
 
 
 def print_histogram(tests):
     test_times = [(t.getFullName(), t.result.elapsed) for t in tests]
     lit.util.printHistogram(test_times, title='Tests')
 
 
-# Status code, summary label, group label
-failure_codes = [
-    (lit.Test.UNRESOLVED,  'Unresolved Tests',    'Unresolved'),
-    (lit.Test.TIMEOUT,     'Individual Timeouts', 'Timed Out'),
-    (lit.Test.FAIL,        'Unexpected Failures', 'Failing'),
-    (lit.Test.XPASS,       'Unexpected Passes',   'Unexpected Passing')
-]
+def add_result_category(result_code, label):
+    assert isinstance(result_code, lit.Test.ResultCode)
+    category = (result_code, "%s Tests" % label, label)
+    result_codes.append(category)
+
 
-all_codes = [
-    (lit.Test.SKIPPED,     'Skipped Tests',     'Skipped'),
-    (lit.Test.UNSUPPORTED, 'Unsupported Tests', 'Unsupported'),
-    (lit.Test.PASS,        'Expected Passes',   ''),
-    (lit.Test.FLAKYPASS,   'Passes With Retry', ''),
-    (lit.Test.XFAIL,       'Expected Failures', 'Expected Failing'),
-] + failure_codes
+# Status code, summary label, group label
+result_codes = [
+    # Passes
+    (lit.Test.SKIPPED,     'Skipped Tests',       'Skipped'),
+    (lit.Test.UNSUPPORTED, 'Unsupported Tests',   'Unsupported'),
+    (lit.Test.PASS,        'Expected Passes',     ''),
+    (lit.Test.FLAKYPASS,   'Passes With Retry',   ''),
+    (lit.Test.XFAIL,       'Expected Failures',   'Expected Failing'),
+    # Failures
+    (lit.Test.UNRESOLVED,  'Unresolved Tests',    'Unresolved'),
+    (lit.Test.TIMEOUT,     'Individual Timeouts', 'Timed Out'),
+    (lit.Test.FAIL,        'Unexpected Failures', 'Failing'),
+    (lit.Test.XPASS,       'Unexpected Passes',   'Unexpected Passing')
+]
 
 
 def print_results(tests, elapsed, opts):
-    tests_by_code = {code: [] for (code, _, _) in all_codes}
+    tests_by_code = {code: [] for (code, _, _) in result_codes}
     for test in tests:
         tests_by_code[test.result.code].append(test)
 
-    for (code, _, group_label) in all_codes:
+    for (code, _, group_label) in result_codes:
         print_group(code, group_label, tests_by_code[code], opts)
 
     print_summary(tests_by_code, opts.quiet, elapsed)
 
 
 def print_group(code, label, tests, opts):
     if not tests:
         return
for test in tests:
         print('  %s' % test.getFullName())
     sys.stdout.write('\n')
 
 
 def print_summary(tests_by_code, quiet, elapsed):
     if not quiet:
         print('\nTesting Time: %.2fs' % elapsed)
 
-    codes = failure_codes if quiet else all_codes
+    codes = [c for c in result_codes if not quiet or c.isFailure]
     groups = [(label, len(tests_by_code[code])) for code, label, _ in codes]
     groups = [(label, count) for label, count in groups if count]
     if not groups:
         return
 
     max_label_len = max(len(label) for label, _ in groups)
     max_count_len = max(len(str(count)) for _, count in groups)
 

llvm/utils/lit/tests/Inputs/custom-result-category/format.py

+import lit
+import lit.formats
+
+CUSTOM_PASS = lit.Test.ResultCode('CUSTOM_PASS', False)
+CUSTOM_FAILURE = lit.Test.ResultCode('CUSTOM_FAILURE', True)
+
+lit.main.add_result_category(CUSTOM_PASS, "My Passed")
+lit.main.add_result_category(CUSTOM_FAILURE, "My Failed")
+
+
+class MyFormat(lit.formats.ShTest):
+    def execute(self, test, lit_config):
+        result = super(MyFormat, self).execute(test, lit_config)
+        if result.code.isFailure:
+            result.code = CUSTOM_FAILURE
+        else:
+            result.code = CUSTOM_PASS
+        return result

llvm/utils/lit/tests/Inputs/custom-result-category/lit.cfg

+import lit
+import site
+site.addsitedir(os.path.dirname(__file__))
+import format
+
+config.name = 'custom-result-category'
+config.suffixes = ['.txt']
+config.test_format = format.MyFormat()
+config.test_source_root = None
+config.test_exec_root = None

llvm/utils/lit/tests/Inputs/custom-result-category/test1.txt

+# RUN: true

llvm/utils/lit/tests/Inputs/custom-result-category/test2.txt

+# RUN: false

llvm/utils/lit/tests/Inputs/shtest-inject/lit.cfg

 import lit
 
 preamble_commands = [
     'echo "THIS WAS"',
     'echo "INJECTED"'
-];
+]
 
 config.name = 'shtest-inject'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest(preamble_commands=preamble_commands)
 config.test_source_root = None
 config.test_exec_root = None

llvm/utils/lit/tests/custom-result-category.py

+# Test lit.main.add_result_category() extension API.
+
+# RUN: not %{lit} -j 1 %{inputs}/custom-result-category | FileCheck %s
+
+# CHECK: CUSTOM_PASS: custom-result-category :: test1.txt
+# CHECK: CUSTOM_FAILURE: custom-result-category :: test2.txt
+
+# TODO(yln): Passing tests shouldn't be printed by default.
+# CHECK: My Passed Tests (1)
+# CHECK: My Failed Tests (1)
+
+# CHECK: My Passed Tests: 1
+# CHECK: My Failed Tests: 1