[lit] Hack lit to allow a test suite to request that it is run "early".

[chandlerc]

Summary

This lets us for example start running the unit test suite early. For
'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!!

It's pretty ugly. I barely know how to write Python, so feel free to
just tell me how I should write it instead. =D

[filcab]

utils/lit/lit/main.py

370 ↗

(On Diff #50443)

Maybe run.tests.sort(key = lambda x: (x.isEarlyTest(), x.getFullName())?
The manual says: "tuples are compared lexicographically; the first items are compared; if they are the same then the second items are compared, and so on.". You'd be able to avoid that nasty string concatenation.

[chandlerc]

Use a tuple instead of hideous string concatenation tricks.

[chandlerc]

utils/lit/lit/main.py

370 ↗

(On Diff #50446)

Fantastic suggestion (but needed a "not")!

[mehdi_amini]

FWIW I'm fine with this. It is a bit "hacky", but it helps a real problem. And almost 25% reduction on the test-suite time is welcome for every bots around.

[• rafael]

Nice!

I can also reproduce the speedup, from 35.66s to 29.46s in a Release+Asserts!

[chandlerc]

Cool, since folks seem happy with this approach, I'll land it. Also happy to follow up with any other tweaks or solutions folks want.

llvm/trunk/test/Unit/lit.cfg

 # -*- Python -*-
 
 # Configuration file for the 'lit' test runner.
 
 import os
 
 import lit.formats
 
 # name: The name of this test suite.
 config.name = 'LLVM-Unit'
 
 # suffixes: A list of file extensions to treat as test files.
 config.suffixes = []
 
+# is_early; Request to run this suite early.
+config.is_early = True
+
 # test_source_root: The root path where tests are located.
 # test_exec_root: The root path where tests should be run.
 llvm_obj_root = getattr(config, 'llvm_obj_root', None)
 if llvm_obj_root is not None:
     config.test_exec_root = os.path.join(llvm_obj_root, 'unittests')
     config.test_source_root = config.test_exec_root
 
 # testFormat: The test format to use to interpret tests.

llvm/trunk/utils/lit/lit/Test.py

def isExpectedToFail(self):
                 return True
 
             # If this is a part of the target triple, it fails.
             if item and item in self.suite.config.target_triple:
                 return True
 
         return False
 
+    def isEarlyTest(self):
+        """
+        isEarlyTest() -> bool
+
+        Check whether this test should be executed early in a particular run.
+        This can be used for test suites with long running tests to maximize
+        parallelism or where it is desirable to surface their failures early.
+        """
+        return self.suite.config.is_early
 
     def getJUnitXML(self):
         test_name = self.path_in_suite[-1]
         test_path = self.path_in_suite[:-1]
         safe_test_path = [x.replace(".","_") for x in test_path]
         safe_name = self.suite.name.replace(".","-")
 
         if safe_test_path:

llvm/trunk/utils/lit/lit/TestingConfig.py

def load_from_path(self, path, litConfig):
                 'unable to parse config file %r, traceback: %s' % (
                     path, traceback.format_exc()))
 
         self.finish(litConfig)
 
     def __init__(self, parent, name, suffixes, test_format,
                  environment, substitutions, unsupported,
                  test_exec_root, test_source_root, excludes,
-                 available_features, pipefail, limit_to_features = []):
+                 available_features, pipefail, limit_to_features = [],
+                 is_early = False):
         self.parent = parent
         self.name = str(name)
         self.suffixes = set(suffixes)
         self.test_format = test_format
         self.environment = dict(environment)
         self.substitutions = list(substitutions)
         self.unsupported = unsupported
         self.test_exec_root = test_exec_root
         self.test_source_root = test_source_root
         self.excludes = set(excludes)
         self.available_features = set(available_features)
         self.pipefail = pipefail
         # This list is used by TestRunner.py to restrict running only tests that
         # require one of the features in this list if this list is non-empty.
         # Configurations can set this list to restrict the set of tests to run.
         self.limit_to_features = set(limit_to_features)
+        # Whether the suite should be tested early in a given run.
+        self.is_early = bool(is_early)
 
     def finish(self, litConfig):
         """finish() - Finish this config object, after loading is complete."""
 
         self.name = str(self.name)
         self.suffixes = set(self.suffixes)
         self.environment = dict(self.environment)
         self.substitutions = list(self.substitutions)

llvm/trunk/utils/lit/lit/main.py

if opts.filter:
                      if rex.search(result_test.getFullName())]
 
     # Then select the order.
     if opts.shuffle:
         random.shuffle(run.tests)
     elif opts.incremental:
         sort_by_incremental_cache(run)
     else:
-        run.tests.sort(key = lambda result_test: result_test.getFullName())
+        run.tests.sort(key = lambda t: (not t.isEarlyTest(), t.getFullName()))
 
     # Finally limit the number of tests, if desired.
     if opts.maxTests is not None:
         run.tests = run.tests[:opts.maxTests]
 
     # Don't create more threads than tests.
     opts.numThreads = min(len(run.tests), opts.numThreads)