[test-suite] Register result codes with lit.

[Meinersbur]

Summary

lit has a hard-coded set of result codes, but their semantics don't
match with what we want to express in the test suite:

• NOCHANGE: Executable Unchanged
• NOEXE: Executable Missing

5c244115c98a simplified registration of custom result codes. Calling
add_result_category() is no longer necessary.

While this does not fix any buildbot (such as http://lab.llvm.org:8011/builders/clang-native-arm-lnt-perf), it at least should allow lit to display which tests have failed at the end.

[Meinersbur]

• Missing Executables -> Executable Missing

When reporting test codes, lit appends " Test" to the category name.
I think "Executable Missing Tests" is better than "Missing Executables Tests".

[yln]

litsupport/test.py

16–17

Actually, I thought that the semantics of lit's skipped differ from test-suite's skipped?!
https://reviews.llvm.org/D77986#1979402

[Meinersbur]

• Rename SKIPPED to UNCHANGED

[Meinersbur]

Note I could not confirm that the regression tests are working. litsupport-tests has a run.sh which does not work. I changed its references to SKIPPED anyway.

I can make lit emit "Executable Unchanged Tests" during a regular run (using -Dprevious=/path/to/lit-output.json), but running tests seem to been serialized, the llvm-lit processes using tens of GBs of memory and as a result being slower than just running the tests.

[Meinersbur]

ping

[Meinersbur]

ping

[jdenny]

I don't have significant experience with LNT or test-suite, so I doubt I'm the right person to respond to your issues here.

Given that this patch is currently accepted, I wonder if clicking "Request Review" in phabricator would help to attract the attention of other reviewers.

[jdoerfert]

LGTM. Thanks for fixing this @Meinersbur and @yln !

[paquette]

Just as a heads up, it seems that this commit breaks anyone using the lit in pypi. (https://pypi.org/project/lit/)

If you try using pypi's lit with this commit (say by running the test suite with LNT), you'll get

AttributeError: 'function' object has no attribute 'add_result_category'

Would it be possible to temporarily back this out until that lit has been updated? (cc: @tstellar @ddunbar)

[tstellar]

In D79064#2035470, @paquette wrote:

Just as a heads up, it seems that this commit breaks anyone using the lit in pypi. (https://pypi.org/project/lit/)

If you try using pypi's lit with this commit (say by running the test suite with LNT), you'll get

AttributeError: 'function' object has no attribute 'add_result_category'

Would it be possible to temporarily back this out until that lit has been updated? (cc: @tstellar @ddunbar)

What change do we need to pull into the pypi lit to fix this?

[paquette]

In D79064#2035472, @tstellar wrote:

In D79064#2035470, @paquette wrote:

Just as a heads up, it seems that this commit breaks anyone using the lit in pypi. (https://pypi.org/project/lit/)

If you try using pypi's lit with this commit (say by running the test suite with LNT), you'll get

AttributeError: 'function' object has no attribute 'add_result_category'

Would it be possible to temporarily back this out until that lit has been updated? (cc: @tstellar @ddunbar)

What change do we need to pull into the pypi lit to fix this?

Looks like rGfbdcfcd4c392c487c411632b951f4aada351154e should do the trick.

[tstellar]

In D79064#2035476, @paquette wrote:

In D79064#2035472, @tstellar wrote:

In D79064#2035470, @paquette wrote:

Just as a heads up, it seems that this commit breaks anyone using the lit in pypi. (https://pypi.org/project/lit/)

If you try using pypi's lit with this commit (say by running the test suite with LNT), you'll get

AttributeError: 'function' object has no attribute 'add_result_category'

Would it be possible to temporarily back this out until that lit has been updated? (cc: @tstellar @ddunbar)

What change do we need to pull into the pypi lit to fix this?

Looks like rGfbdcfcd4c392c487c411632b951f4aada351154e should do the trick.

I was hoping this would be a bug fix, but it looks more like a feature. We were planning to update lit on pypi for the 10.0.1 release, but this doesn't look like a good backport candidate to me. Can we work-around this in the test suite somehow?

[yln]

ResultCode has a self-registering constructor now, calling
add_result_category() is no longer necessary. See 5c244115c98a.

[Meinersbur]

In D79064#2098947, @yln wrote:

ResultCode has a self-registering constructor now, calling
add_result_category() is no longer necessary. See 5c244115c98a.

Given the change:

-    def __init__(self, name, isFailure):
+    def __init__(self, name, label, isFailure):

every program using the two-argument ResultCode constructor will fail now? I also see the add_result_category has been removed entirely, and llvm-test-suite has not received an update.

[yln]

In D79064#2099087, @Meinersbur wrote:

and llvm-test-suite has not received an update.

Can we land this change or should I revert the lit change?

[yln]

Obsoleted.

litsupport-tests/hash/check/lit.out

-CHECK-DAG: SKIPPED: test-suite :: tests/skip.test
+CHECK-DAG: NOCHANGE: test-suite :: tests/unchanged.test
 CHECK-DAG: PASS: test-suite :: tests/normal.test
 CHECK: Expected Passes

litsupport-tests/hash/tests/unchanged.test

 RUN: tests/foo.sh

litsupport/test.py

 """
 Main integration for llvm-lit: This defines a lit test format.
 Also contains logic to load benchmark modules.
 """
 import lit
 import lit.TestRunner
 import lit.util
 import lit.formats
 import litsupport.modules
 import litsupport.modules.hash
 import litsupport.testfile
 import litsupport.testplan
 import os
 
 
-SKIPPED = lit.Test.ResultCode('SKIPPED', False)
-NOEXE = lit.Test.ResultCode('NOEXE', True)
+NOCHANGE = lit.Test.ResultCode('NOCHANGE', 'Executable Unchanged', False)
+NOEXE = lit.Test.ResultCode('NOEXE', 'Executable Missing', True)
 
 
 class TestSuiteTest(lit.formats.ShTest):
     def __init__(self):
         super(TestSuiteTest, self).__init__()
 
     def execute(self, test, litConfig):
         config = test.config
def execute(self, test, litConfig):
             return lit.Test.Result(NOEXE, "Executable '%s' is missing" %
                                    context.executable)
 
         # Skip unchanged tests
         if config.previous_results:
             litsupport.modules.hash.compute(context)
             if litsupport.modules.hash.same_as_previous(context):
                 result = lit.Test.Result(
-                        SKIPPED, 'Executable identical to previous run')
+                        NOCHANGE, 'Executable identical to previous run')
                 val = lit.Test.toMetricValue(context.executable_hash)
                 result.addMetric('hash', val)
                 return result
 
         # Let test modules modify the test plan.
         for modulename in config.test_modules:
             module = litsupport.modules.modules.get(modulename)
             if module is None:
                 raise Exception("Unknown testmodule '%s'" % modulename)
             module.mutatePlan(context, plan)
 
         # Execute Test plan
         result = litsupport.testplan.executePlanTestResult(context, plan)
 
         return result