[lit][compiler-rt] add multi-cfgd test suite discovery

[ychen]

Summary

Currently llvm-lit /path/to/src does not work for many compiler-rt tests which is not developer-friendly.

The reason for this is compiler-rt tests (asan, ubsan etc..) are really multi-config'd tests that have configurations for different architectures and dynamic/static linking. Currently we generate one TestSuite for each configuration at build directory side which could be invoked like llvm-lit /path/to/build/config-linux-dynamic. We could not do llvm-lit /path/to/src for all configurations because in LIT, class TestSuite has only one config object.

This patch

• make class TestSuite own more than one TestConfig and update clients accordingly.
• a multi-config'd TestSuite is recognized by text files with suffix '.multi.cfg'. The '.multi.cfg' file contents is <test suite name> /path/to/cfg_dir1/lit.site.cfg /path/to/cfg_dir2/lit.site.cfg ...
• currently there exists testSuiteCache to avoid reloading same TestConfig twice when exec dir is in tree.(See llvm\utils\lit\tests\Inputs\exec-discovery-in-tree for examples). This patch add a testConfigCache to solve the same issue since class TestSuite is able to own multiple TestConfig now.
• changes in utils/lit/lit/main.py handles the printing of discovered test-suite properly.
• this is needed by D73981

[merge_guards_bot]

Unit tests: unknown.

clang-tidy: pass.

clang-format: pass.

Build artifacts: diff.json, clang-tidy.txt, clang-format.patch, CMakeCache.txt, console-log.txt

Pre-merge checks is in beta. Report issue. Please join beta or enable it for your project.

[yln]

Hi @ychen,

the code changes look fine. My understanding of your patch is that it allows test suites to be subdivided into parts that can be configured differently.
Before proceeding, can you please explain which problem you are trying to solve and why the existing facilities don't fit your needs?

Thanks!

[ychen]

In D73980#1862032, @yln wrote:

Hi @ychen,

the code changes look fine. My understanding of your patch is that it allows test suites to be subdivided into parts that can be configured differently.
Before proceeding, can you please explain which problem you are trying to solve and why the existing facilities don't fit your needs?

Thanks!

Hi @yln, this is mostly for compiler-rt where one test could be tested in many configurations. Without this patch, llvm-lit /path/to/test/src/ does not work since the config_map maps test source code to only one configuration currently. This patch adds a third test suite marker file (lit.site.multi.cfg), config_map could map to and invoke all the test configs applicable to the test.

D73981 shows the way it is used in the CMake file. I also updated the lit.rst doc.

[ychen]

Hello, friendly ping for some attention here. I've run LIT unit test and most of the regression tests in the monorepo. I didn't see any regressions.

[delcypher]

@ychen I need convincing that this feature is actually needed. Today the problem of multiple configurations is solved at the CMake level in compiler-rt where we have CMake generate multiple config directories that point to the same source tree. Each config is generated differently depending on the use case. Why isn't that sufficient for the problem you are trying to solve.

The description of this change is also very incomplete.

[ychen]

In D73980#1888537, @delcypher wrote:

@ychen I need convincing that this feature is actually needed. Today the problem of multiple configurations is solved at the CMake level in compiler-rt where we have CMake generate multiple config directories that point to the same source tree. Each config is generated differently depending on the use case. Why isn't that sufficient for the problem you are trying to solve.

The description of this change is also very incomplete.

Hi @delcypher, thanks for chime in. This patch is a pre-requisite of D73981, to solve the problem/inconvenience of not being able to do llvm-lit /path/to/compiler-rt/x.test. I will update the description.

[ychen]

• rebase

[ychen]

hello, I'm still looking to get this in.

[rnk]

-self, load shedding

[ychen]

ping?

[ychen]

ping ...

[delcypher]

I've done another pass.

llvm/docs/CommandGuide/lit.rst

65

Is the second

:file:`{NAME}.site.multi.cfg`

supposed to be

:file:`lit.site.multi.cfg`

?

254

This format doesn't seem like to it would handle future changes very well. Could this be a more structured format, e.g. JSON?

llvm/utils/lit/CMakeLists.txt

11 ↗

(On Diff #261951)

This might not work on Windows due to the slashes.

I don't like that we're doing something special here just for this test. This means that the test doesn't even exist if someone tries to run the tests in the source tree.
Can't we have the multicfg format support relative paths so

• that additional copying is unnecessary
• The test gets executed if lit is tested in the source tree.

llvm/utils/lit/lit/discovery.py

84–107

Parsing of the file format probably belongs in its own function.

133

If you're going to rename ts perhaps we should use a more helpful name, e.g. testConfig.

248

Why is this here? It deserves a comment at the very least.

llvm/utils/lit/lit/reports.py

13

This change is an API break. Is anyone using the old by_suite_and_test_path from their lit configuration files?

[delcypher]

@yln Any opinions on this? You touch lit much more than I do these days.

[ychen]

llvm/docs/CommandGuide/lit.rst

65

Yes.

llvm/utils/lit/CMakeLists.txt

11 ↗

(On Diff #261951)

• Added relative path support
• Removed the code here: the test works like others in the lit/test directory

llvm/utils/lit/lit/discovery.py

248

I should have removed this. It was intended to address the same issue the above code is addressing now. It could've been removed because when if sub_ts.configs[0] in ts.configs is true, if sub_ts is ts would also be true during the recursive call of getTestsInSuite (line 248).

if sub_ts is ts:
    continue

llvm/utils/lit/lit/reports.py

13

It seems no in-tree/lnt/test-suite/zorg clients are using this.

[ychen]

• support relative path in multicfg file
• use json for multicfg file
• update lit.rst
• address comments

[ychen]

ping..

[ychen]

Gentle ping.

Definitely agree that this change is on the large side. For the most part, it just removes the assumption that class TestSuite only has one config. It is not a design change or anything. I'll follow up quickly if there are any issues.

[MaskRay]

The description seems long and I haven't tried understanding what this patch does, but still wanted to make one comment.

If /tmp/ReleaseA is a build directory, for a test file compiler-rt/test/profile/gcov-basic.c, you can run /tmp/ReleaseA/bin/llvm-lit /tmp/ReleaseA/projects/compiler-rt/test/profile/Profile-x86_64/gcov-basic.c

(I create a shell alias for /tmp/ReleaseA/bin/llvm-lit)

llvm/docs/CommandGuide/lit.rst

 
 .. option:: -j N, --workers=N
 
  Run ``N`` tests in parallel.  By default, this is automatically chosen to
  match the number of detected available CPUs.
 
 .. option:: --config-prefix=NAME
 
- Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for
- test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`.
+ Search for :file:`{NAME}.cfg`, :file:`{NAME}.site.cfg` and
+ :file:`{NAME}.site.multi.cfg` when searching for test suites, instead of
+ :file:`lit.cfg`, :file:`lit.site.cfg` and :file:`lit.site.multi.cfg`.
 
 .. option:: -D NAME[=VALUE], --param NAME[=VALUE]
 
  Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty
  string if not given).  The meaning and use of these parameters is test suite
  dependent.
 
 .. _output-options:

 
 The inputs passed to :program:`lit` can be either individual tests, or entire
 directories or hierarchies of tests to run.  When :program:`lit` starts up, the
 first thing it does is convert the inputs into a complete list of tests to run
 as part of *test discovery*.
 
 In the :program:`lit` model, every test must exist inside some *test suite*.
 :program:`lit` resolves the inputs specified on the command line to test suites
-by searching upwards from the input path until it finds a :file:`lit.cfg` or
-:file:`lit.site.cfg` file.  These files serve as both a marker of test suites
-and as configuration files which :program:`lit` loads in order to understand
-how to find and run the tests inside the test suite.
+by searching upwards from the input path until it finds a :file:`lit.cfg`,
+:file:`lit.site.cfg` or :file:`lit.site.multi.cfg` file. These files serve as
+both a marker of test suites and as configuration files which :program:`lit`
+loads in order to understand how to find and run the tests inside the test suite.
+
+:file:`lit.site.multi.cfg` file is used when the test suite has multiple
+configurations. The file is in JSON format with two key-value pairs. The format
+looks like the example below. The sub-testsuite path could be either a absolute
+filepath or a relative filepath to the path of the :file:`lit.site.multi.cfg` file.
+
+.. code-block:: json
+
+  {
+      "name": "<test suite name>",
+      "test suites": [
+          "<relpath/to/cfg_dir1/lit.site.cfg>", # or "</abspath/to/cfg_dir1/lit.site.cfg>"
+          "<relpath/to/cfg_dir2/lit.site.cfg>"  # or "</abspath/to/cfg_dir2/lit.site.cfg>"
+      ]
+  }
 
 Once :program:`lit` has mapped the inputs into test suites it traverses the
 list of inputs adding tests for individual files and recursively searching for
 tests in directories.
 
 This behavior makes it easy to specify a subset of tests to run, while still
 allowing the test suite configuration to control exactly how tests are
 interpreted.  In addition, :program:`lit` always identifies tests by the test

 
 TEST SUITES
 ~~~~~~~~~~~
 
 As described in :ref:`test-discovery`, tests are always located inside a *test
 suite*.  Test suites serve to define the format of the tests they contain, the
 logic for finding those tests, and any additional information to run the tests.
 
-:program:`lit` identifies test suites as directories containing ``lit.cfg`` or
-``lit.site.cfg`` files (see also :option:`--config-prefix`).  Test suites are
-initially discovered by recursively searching up the directory hierarchy for
-all the input files passed on the command line.  You can use
+:program:`lit` identifies test suites as directories containing ``lit.cfg``,
+``lit.site.cfg`` or ``lit.site.multi.cfg`` files (see also :option:`--config-prefix`).
+Test suites are initially discovered by recursively searching up the directory
+hierarchy for all the input files passed on the command line.  You can use
 :option:`--show-suites` to display the discovered test suites at startup.
 
 Once a test suite is discovered, its config file is loaded.  Config files
 themselves are Python modules which will be executed.  When the config file is
 executed, two important global variables are predefined:
 
 **lit_config**
 

llvm/utils/lit/lit/LitConfig.py

def __init__(self, progname, path, quiet,
         self.debug = debug
         self.isWindows = bool(isWindows)
         self.params = dict(params)
         self.bashPath = None
 
         # Configuration files to look for when discovering test suites.
         self.config_prefix = config_prefix or 'lit'
         self.suffixes = ['cfg.py', 'cfg']
+        self.multiconfig_suffixes = ['multi.cfg']
         self.config_names = ['%s.%s' % (self.config_prefix,x) for x in self.suffixes]
-        self.site_config_names = ['%s.site.%s' % (self.config_prefix,x) for x in self.suffixes]
+        # Only exec side test suites could be multi-config'd.
+        self.site_config_names = ['%s.site.%s' % (self.config_prefix,x) for x in self.suffixes + self.multiconfig_suffixes]
         self.local_config_names = ['%s.local.%s' % (self.config_prefix,x) for x in self.suffixes]
 
         self.numErrors = 0
         self.numWarnings = 0
 
         self.valgrindArgs = []
         if self.useValgrind:
             self.valgrindArgs = ['valgrind', '-q', '--run-libc-freeres=no',

llvm/utils/lit/lit/Test.py

 
 
 # Test classes.
 
 class TestSuite:
     """TestSuite - Information on a group of tests.
 
     A test suite groups together a set of logically related tests.
+    This set of tests could be configured in different ways.
     """
 
-    def __init__(self, name, source_root, exec_root, config):
+    def __init__(self, name, configs, is_multi_cfg):
         self.name = name
-        self.source_root = source_root
-        self.exec_root = exec_root
-        # The test suite configuration.
-        self.config = config
-
-    def getSourcePath(self, components):
-        return os.path.join(self.source_root, *components)
-
-    def getExecPath(self, components):
-        return os.path.join(self.exec_root, *components)
+        self.configs = configs
+        self.is_multi_cfg = is_multi_cfg
 
 class Test:
     """Test - Information on a single test instance."""
 
     def __init__(self, suite, path_in_suite, config, file_path = None):
         self.suite = suite
         self.path_in_suite = path_in_suite
         self.config = config
def setResult(self, result):
                     result.code = XFAIL
         self.result = result
 
     def isFailure(self):
         assert self.result
         return self.result.code.isFailure
 
     def getFullName(self):
-        return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
+        return self.config.name + ' :: ' + '/'.join(self.path_in_suite)
 
     def getFilePath(self):
         if self.file_path:
             return self.file_path
         return self.getSourcePath()
 
     def getSourcePath(self):
-        return self.suite.getSourcePath(self.path_in_suite)
+        return self.config.getSourcePath(self.path_in_suite)
 
     def getExecPath(self):
-        return self.suite.getExecPath(self.path_in_suite)
+        return self.config.getExecPath(self.path_in_suite)
 
     def isExpectedToFail(self):
         """
         isExpectedToFail() -> bool
 
         Check whether this test is expected to fail in the current
         configuration. This check relies on the test xfails property which by
         some test formats may not be computed until the test has first been
         executed.
         Throws ValueError if an XFAIL line has a syntax error.
         """
 
         features = self.config.available_features
-        triple = getattr(self.suite.config, 'target_triple', "")
+        triple = getattr(self.config, 'target_triple', "")
 
         # Check if any of the xfails match an available feature or the target.
         for item in self.xfails:
             # If this is the wildcard, it always fails.
             if item == '*':
                 return True
 
             # If this is a True expression of features and target triple parts,
def getUnsupportedFeatures(self):
         getUnsupportedFeatures() -> list of strings
 
         Returns a list of features from UNSUPPORTED that are present
         in the test configuration's features or target triple.
         Throws ValueError if an UNSUPPORTED line has a syntax error.
         """
 
         features = self.config.available_features
-        triple = getattr(self.suite.config, 'target_triple', "")
+        triple = getattr(self.config, 'target_triple', "")
 
         try:
             return [item for item in self.unsupported
                     if BooleanExpression.evaluate(item, features, triple)]
         except ValueError as e:
             raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))
 
     def isEarlyTest(self):
         """
         isEarlyTest() -> bool
 
         Check whether this test should be executed early in a particular run.
         This can be used for test suites with long running tests to maximize
         parallelism or where it is desirable to surface their failures early.
         """
-        return self.suite.config.is_early
+        return self.config.is_early

llvm/utils/lit/lit/TestingConfig.py

def finish(self, litConfig):
             # files. Should we distinguish them?
             self.test_exec_root = str(self.test_exec_root)
         if self.test_source_root is not None:
             # FIXME: This should really only be suite in test suite config
             # files. Should we distinguish them?
             self.test_source_root = str(self.test_source_root)
         self.excludes = set(self.excludes)
 
+    def getSourcePath(self, components):
+        return os.path.join(self.test_source_root, *components)
+
+    def getExecPath(self, components):
+        return os.path.join(self.test_exec_root, *components)
+
     @property
     def root(self):
         """root attribute - The root configuration for the test suite."""
         if self.parent is None:
             return self
         else:
             return self.parent.root
 

llvm/utils/lit/lit/discovery.py

 """
 Test discovery functions.
 """
 
 import copy
+import json
 import os
 import sys
 
 from lit.TestingConfig import TestingConfig
 from lit import LitConfig, Test
 
 def chooseConfigFileFromDir(dir, config_names):
     for name in config_names:
         p = os.path.join(dir, name)
         if os.path.exists(p):
             return p
     return None
 
 def dirContainsTestSuite(path, lit_config):
     cfgpath = chooseConfigFileFromDir(path, lit_config.site_config_names)
     if not cfgpath:
         cfgpath = chooseConfigFileFromDir(path, lit_config.config_names)
     return cfgpath
 
-def getTestSuite(item, litConfig, cache):
-    """getTestSuite(item, litConfig, cache) -> (suite, relative_path)
+def getTestSuiteConfigs(cfgpath, litConfig):
+    ts_name = ''
+    cfgpaths = []
+    ts_multi = any(
+        cfgpath.endswith(s) for s in litConfig.multiconfig_suffixes)
+    if ts_multi:
+        # Read multi-cfg test suite
+        with open(cfgpath, 'r') as f:
+            data = json.load(f)
+            ts_name = data['name']
+            for ln in data['test suites']:
+                ln = ln.strip()
+                if ln:
+                    ln = os.path.normpath(ln)
+                    if not os.path.isabs(ln):
+                        ln = os.path.join(cfgpath, '..', ln)
+                    cfgpaths.append(ln)
+    else:
+        cfgpaths.append(cfgpath)
+    return cfgpaths, ts_multi, ts_name
+
+def getTestSuite(item, litConfig, testSuiteCache, testConfigCache):
+    """getTestSuite(item, litConfig, testSuiteCache,
+                    testConfigCache) -> (suite, relative_path)
 
     Find the test suite containing @arg item.
 
     @retval (None, ...) - Indicates no test suite contains @arg item.
     @retval (suite, relative_path) - The suite that @arg item is in, and its
     relative path inside that suite.
     """
     def search1(path):
def search1(path):
         config_map = litConfig.params.get('config_map')
         if config_map:
             cfgpath = os.path.realpath(cfgpath)
             cfgpath = os.path.normcase(cfgpath)
             target = config_map.get(cfgpath)
             if target:
                 cfgpath = target
 
-        # We found a test suite, create a new config for it and load it.
-        if litConfig.debug:
-            litConfig.note('loading suite config %r' % cfgpath)
+        cfgpaths, ts_multi, ts_name = getTestSuiteConfigs(cfgpath, litConfig)
 
+        cfgs = []
+        for cp in cfgpaths:
+            cp = os.path.realpath(cp)
+            cp = os.path.normpath(cp)
+            cfg = testConfigCache.get(cp)
+            if cfg is None:
+                # We found a test suite config, create a TestingConfig and load it.
+                if litConfig.debug:
+                    litConfig.note('loading suite config %r' % cp)
                 cfg = TestingConfig.fromdefaults(litConfig)
-        cfg.load_from_path(cfgpath, litConfig)
-        source_root = os.path.realpath(cfg.test_source_root or path)
-        exec_root = os.path.realpath(cfg.test_exec_root or path)
-        return Test.TestSuite(cfg.name, source_root, exec_root, cfg), ()
+                cfg.load_from_path(cp, litConfig)
+                cfg.test_exec_root = os.path.realpath(cfg.test_exec_root
+                                                      or path)
+                cfg.test_source_root = os.path.realpath(cfg.test_source_root
+                                                        or path)
+                testConfigCache[cp] = cfg
+            cfgs.append(cfg)
+
+            if not ts_multi:
+                ts_name = cfg.name
+
+        return Test.TestSuite(ts_name, cfgs, ts_multi), ()
 
     def search(path):
         # Check for an already instantiated test suite.
         real_path = os.path.realpath(path)
-        res = cache.get(real_path)
+        res = testSuiteCache.get(real_path)
         if res is None:
-            cache[real_path] = res = search1(path)
+            testSuiteCache[real_path] = res = search1(path)
         return res
 
     # Canonicalize the path.
     item = os.path.normpath(os.path.join(os.getcwd(), item))
 
     # Skip files and virtual components.
     components = []
     while not os.path.isdir(item):
         parent,base = os.path.split(item)
         if parent == item:
             return (None, ())
         components.append(base)
         item = parent
     components.reverse()
 
     ts, relative = search(item)
     return ts, tuple(relative + tuple(components))
 
-def getLocalConfig(ts, path_in_suite, litConfig, cache):
+def getLocalConfig(testConfig, path_in_suite, litConfig, cache):
     def search1(path_in_suite):
         # Get the parent config.
         if not path_in_suite:
-            parent = ts.config
+            parent = testConfig
         else:
             parent = search(path_in_suite[:-1])
 
         # Check if there is a local configuration file.
-        source_path = ts.getSourcePath(path_in_suite)
-        cfgpath = chooseConfigFileFromDir(source_path, litConfig.local_config_names)
+        source_path = testConfig.getSourcePath(path_in_suite)
+        cfgpath = chooseConfigFileFromDir(source_path,
+                                          litConfig.local_config_names)
 
         # If not, just reuse the parent config.
         if not cfgpath:
             return parent
 
         # Otherwise, copy the current config and load the local configuration
         # file into it.
         config = copy.deepcopy(parent)
         if litConfig.debug:
             litConfig.note('loading local config %r' % cfgpath)
         config.load_from_path(cfgpath, litConfig)
+        # config without parent is a test suite config, not a local config
+        if config.parent is None:
+            config.parent = parent
         return config
 
     def search(path_in_suite):
-        key = (ts, path_in_suite)
+        key = (testConfig, path_in_suite)
         res = cache.get(key)
         if res is None:
             cache[key] = res = search1(path_in_suite)
         return res
 
     return search(path_in_suite)
 
-def getTests(path, litConfig, testSuiteCache, localConfigCache):
+
+def getTests(path, litConfig, testSuiteCache, testConfigCache,
+             localConfigCache):
     # Find the test suite for this input and its relative path.
-    ts,path_in_suite = getTestSuite(path, litConfig, testSuiteCache)
+    ts,path_in_suite = getTestSuite(path, litConfig, testSuiteCache,
+                                    testConfigCache)
     if ts is None:
         litConfig.warning('unable to find test suite for %r' % path)
         return (),()
 
+    tests = []
+    for tc in ts.configs:
         if litConfig.debug:
-        litConfig.note('resolved input %r to %r::%r' % (path, ts.name,
-                                                        path_in_suite))
-
-    return ts, getTestsInSuite(ts, path_in_suite, litConfig,
-                               testSuiteCache, localConfigCache)
+            litConfig.note('resolved input %r to %r::%r' %
+                           (path, tc.name, path_in_suite))
+        tests.extend(
+            getTestsInSuite(ts, tc, path_in_suite, litConfig, testSuiteCache,
+                            testConfigCache, localConfigCache))
+    return tests
 
-def getTestsInSuite(ts, path_in_suite, litConfig,
-                    testSuiteCache, localConfigCache):
+def getTestsInSuite(ts, tc, path_in_suite, litConfig, testSuiteCache,
+                    testConfigCache, localConfigCache):
     # Check that the source path exists (errors here are reported by the
     # caller).
-    source_path = ts.getSourcePath(path_in_suite)
+    source_path = tc.getSourcePath(path_in_suite)
     if not os.path.exists(source_path):
         return
 
     # Check if the user named a test directly.
     if not os.path.isdir(source_path):
-        lc = getLocalConfig(ts, path_in_suite[:-1], litConfig, localConfigCache)
+        lc = getLocalConfig(tc, path_in_suite[:-1], litConfig, localConfigCache)
         yield Test.Test(ts, path_in_suite, lc)
         return
 
     # Otherwise we have a directory to search for tests, start by getting the
     # local configuration.
-    lc = getLocalConfig(ts, path_in_suite, litConfig, localConfigCache)
+    lc = getLocalConfig(tc, path_in_suite, litConfig, localConfigCache)
 
     # Search for tests.
     if lc.test_format is not None:
         for res in lc.test_format.getTestsInDirectory(ts, path_in_suite,
                                                       litConfig, lc):
             yield res
 
     # Search subdirectories.
     for filename in os.listdir(source_path):
         # FIXME: This doesn't belong here?
         if filename in ('Output', '.svn', '.git') or filename in lc.excludes:
             continue
 
         # Ignore non-directories.
         file_sourcepath = os.path.join(source_path, filename)
         if not os.path.isdir(file_sourcepath):
             continue
 
         # Check for nested test suites, first in the execpath in case there is a
         # site configuration and then in the source path.
         subpath = path_in_suite + (filename,)
-        file_execpath = ts.getExecPath(subpath)
+        file_execpath = tc.getExecPath(subpath)
         if dirContainsTestSuite(file_execpath, litConfig):
             sub_ts, subpath_in_suite = getTestSuite(file_execpath, litConfig,
-                                                    testSuiteCache)
+                                                    testSuiteCache,
+                                                    testConfigCache)
         elif dirContainsTestSuite(file_sourcepath, litConfig):
             sub_ts, subpath_in_suite = getTestSuite(file_sourcepath, litConfig,
-                                                    testSuiteCache)
+                                                    testSuiteCache,
+                                                    testConfigCache)
         else:
             sub_ts = None
 
         # If the this directory recursively maps back to the current test suite,
         # disregard it (this can happen if the exec root is located inside the
         # current test suite, for example).
         if sub_ts is ts:
             continue
 
         # Otherwise, load from the nested test suite, if present.
-        if sub_ts is not None:
-            subiter = getTestsInSuite(sub_ts, subpath_in_suite, litConfig,
-                                      testSuiteCache, localConfigCache)
-        else:
-            subiter = getTestsInSuite(ts, subpath, litConfig, testSuiteCache,
+        subts = ts if sub_ts is None else sub_ts
+        subpath = subpath if sub_ts is None else subpath_in_suite
+        for sub_tc in subts.configs:
+            subiter = getTestsInSuite(subts, sub_tc, subpath, litConfig,
+                                      testSuiteCache, testConfigCache,
                                       localConfigCache)
-
             N = 0
             for res in subiter:
                 N += 1
                 yield res
+
             if sub_ts and not N:
-            litConfig.warning('test suite %r contained no tests' % sub_ts.name)
+                litConfig.warning('test suite %r contained no tests' %
+                                  sub_tc.name)
+
 
 def find_tests_for_inputs(lit_config, inputs):
     """
     find_tests_for_inputs(lit_config, inputs) -> [Test]
 
     Given a configuration object and a list of input specifiers, find all the
     tests to execute.
     """
for input in inputs:
             finally:
                 f.close()
         else:
             actual_inputs.append(input)
 
     # Load the tests from the inputs.
     tests = []
     test_suite_cache = {}
+    test_config_cache = {}  # Avoid reloading the same test config.
     local_config_cache = {}
     for input in actual_inputs:
         prev = len(tests)
-        tests.extend(getTests(input, lit_config,
-                              test_suite_cache, local_config_cache)[1])
+        tests.extend(
+            getTests(input, lit_config, test_suite_cache, test_config_cache,
+                     local_config_cache))
         if prev == len(tests):
             lit_config.warning('input %r contained no tests' % input)
 
     # If there were any errors during test discovery, exit now.
     if lit_config.numErrors:
         sys.stderr.write('%d errors, exiting.\n' % lit_config.numErrors)
         sys.exit(2)
 
     return tests

llvm/utils/lit/lit/formats/base.py

 from __future__ import absolute_import
 import os
 
 import lit.Test
 import lit.util
 
 class TestFormat(object):
     pass
 
 ###
 
 class FileBasedTest(TestFormat):
     def getTestsInDirectory(self, testSuite, path_in_suite,
                             litConfig, localConfig):
-        source_path = testSuite.getSourcePath(path_in_suite)
+        source_path = localConfig.getSourcePath(path_in_suite)
         for filename in os.listdir(source_path):
             # Ignore dot files and excluded tests.
             if (filename.startswith('.') or
                 filename in localConfig.excludes):
                 continue
 
             filepath = os.path.join(source_path, filename)
             if not os.path.isdir(filepath):
def __init__(self, command, dir, recursive=False,
         self.recursive = bool(recursive)
         self.pattern = re.compile(pattern)
         self.useTempInput = useTempInput
 
     def getTestsInDirectory(self, testSuite, path_in_suite,
                             litConfig, localConfig):
         dir = self.dir
         if dir is None:
-            dir = testSuite.getSourcePath(path_in_suite)
+            dir = localConfig.getSourcePath(path_in_suite)
 
         for dirname,subdirs,filenames in os.walk(dir):
             if not self.recursive:
                 subdirs[:] = []
 
             subdirs[:] = [d for d in subdirs
                           if (d != '.svn' and
                               d not in localConfig.excludes)]

llvm/utils/lit/lit/formats/googletest.py

def getGTestTests(self, path, litConfig, localConfig):
                 # Gtest will internally skip these tests. No need to launch a
                 # child process for it.
                 continue
             else:
                 yield ''.join(nested_tests) + ln
 
     def getTestsInDirectory(self, testSuite, path_in_suite,
                             litConfig, localConfig):
-        source_path = testSuite.getSourcePath(path_in_suite)
+        source_path = localConfig.getSourcePath(path_in_suite)
         for subdir in self.test_sub_dirs:
             dir_path = os.path.join(source_path, subdir)
             if not os.path.isdir(dir_path):
                 continue
             for fn in lit.util.listdir_files(dir_path,
                                              suffixes=self.test_suffixes):
                 # Discover the tests in this executable.
                 execpath = os.path.join(source_path, subdir, fn)

llvm/utils/lit/lit/main.py

def parse(p):
         return p.split('=', 1) if '=' in p else (p, '')
 
     params = dict(builtin_params)
     params.update([parse(p) for p in user_params])
     return params
 
 
 def print_discovered(tests, show_suites, show_tests):
-    tests.sort(key=lit.reports.by_suite_and_test_path)
+    tests.sort(key=lit.reports.by_suite_and_config_and_test_path)
 
     if show_suites:
         import itertools
         tests_by_suite = itertools.groupby(tests, lambda t: t.suite)
         print('-- Test Suites --')
         for suite, test_iter in tests_by_suite:
-            test_count = sum(1 for _ in test_iter)
-            print('  %s - %d tests' % (suite.name, test_count))
-            print('    Source Root: %s' % suite.source_root)
-            print('    Exec Root  : %s' % suite.exec_root)
-            features = ' '.join(sorted(suite.config.available_features))
+            tests_by_config = itertools.groupby(test_iter, lambda t: lit.reports.getRoot(t.config))
+            for _, test_iter_config in tests_by_config:
+                config = next(test_iter_config).config
+                test_count = sum(1 for _ in test_iter_config) + 1
+                if suite.is_multi_cfg:
+                    print('  %s :: %s - %d tests' %
+                          (suite.name, config.name, test_count))
+                else:
+                    print('  %s - %d tests' % (config.name, test_count))
+                print('    Source Root: %s' % config.test_source_root)
+                print('    Exec Root  : %s' % config.test_exec_root)
+                features = ' '.join(sorted(config.available_features))
                 print('    Available Features: %s' % features)
-            substitutions = sorted(suite.config.substitutions)
+                substitutions = sorted(config.substitutions)
                 substitutions = ('%s => %s' % (x, y) for (x, y) in substitutions)
                 substitutions = '\n'.ljust(30).join(substitutions)
                 print('    Available Substitutions: %s' % substitutions)
 
     if show_tests:
         print('-- Available Tests --')
         for t in tests:
             print('  %s' % t.getFullName())
 
 
 def determine_order(tests, order):

llvm/utils/lit/lit/reports.py

 import itertools
 import json
 
 from xml.sax.saxutils import quoteattr as quo
 
 import lit.Test
 
+def getRoot(config):
+    if not config.parent:
+        return config
+    return getRoot(config.parent)
 
-def by_suite_and_test_path(test):
+def by_suite_and_config_and_test_path(test):
     # Suite names are not necessarily unique.  Include object identity in sort
     # key to avoid mixing tests of different suites.
-    return (test.suite.name, id(test.suite), test.path_in_suite)
+    return (test.suite.name, id(test.suite), getRoot(test.config).name,
+            id(getRoot(test.config)), test.path_in_suite)
 
 
 class JsonReport(object):
     def __init__(self, output_file):
         self.output_file = output_file
 
     def write_results(self, tests, elapsed):
         assert not any(t.result.code in {lit.Test.EXCLUDED, lit.Test.SKIPPED} for t in tests)
class XunitReport(object):
     def __init__(self, output_file):
         self.output_file = output_file
         self.skipped_codes = {lit.Test.EXCLUDED,
                               lit.Test.SKIPPED, lit.Test.UNSUPPORTED}
 
     # TODO(yln): elapsed unused, put it somewhere?
     def write_results(self, tests, elapsed):
         assert not any(t.result.code in {lit.Test.EXCLUDED, lit.Test.SKIPPED} for t in tests)
-        tests.sort(key=by_suite_and_test_path)
+        tests.sort(key=by_suite_and_config_and_test_path)
         tests_by_suite = itertools.groupby(tests, lambda t: t.suite)
 
         with open(self.output_file, 'w') as file:
             file.write('<?xml version="1.0" encoding="UTF-8"?>\n')
             file.write('<testsuites>\n')
             for suite, test_iter in tests_by_suite:
                 self._write_testsuite(file, suite, list(test_iter))
             file.write('</testsuites>\n')
 
     def _write_testsuite(self, file, suite, tests):
         skipped = sum(1 for t in tests if t.result.code in self.skipped_codes)
         failures = sum(1 for t in tests if t.isFailure())
 
-        name = suite.config.name.replace('.', '-')
+        name = suite.name.replace('.', '-')
         # file.write(f'<testsuite name={quo(name)} tests="{len(tests)}" failures="{failures}" skipped="{skipped}">\n')
         file.write('<testsuite name={name} tests="{tests}" failures="{failures}" skipped="{skipped}">\n'.format(
             name=quo(name), tests=len(tests), failures=failures, skipped=skipped))
         for test in tests:
             self._write_test(file, test, name)
         file.write('</testsuite>\n')
 
     def _write_test(self, file, test, suite_name):
         path = '/'.join(test.path_in_suite[:-1]).replace('.', '_')
-        # class_name = f'{suite_name}.{path or suite_name}'
-        class_name = suite_name + '.' + (path or suite_name)
+        class_name = suite_name
+        if test.suite.is_multi_cfg:
+            class_name += '.' + getRoot(test.config).name
+        class_name += '.' + (path or suite_name)
         name = test.path_in_suite[-1]
         time = test.result.elapsed or 0.0
         # file.write(f'<testcase classname={quo(class_name)} name={quo(name)} time="{time:.2f}"')
         file.write('<testcase classname={class_name} name={name} time="{time:.2f}"'.format(
             class_name=quo(class_name), name=quo(name), time=time))
 
         if test.isFailure():
             file.write('>\n  <failure><![CDATA[')

llvm/utils/lit/tests/Inputs/discovery-multicfg/lit.site.multi.cfg

+{
+    "name": "test-multi-cfg-test-suite",
+    "test suites": [
+        "../exec-discovery/lit.site.cfg",
+        "../exec-discovery-in-tree/obj/lit.site.cfg"
+    ]
+}
 No newline at end of file

llvm/utils/lit/tests/discovery.py

 #
 # CHECK-BASIC-OUT: -- Available Tests --
 # CHECK-BASIC-OUT: sub-suite :: test-one
 # CHECK-BASIC-OUT: sub-suite :: test-two
 # CHECK-BASIC-OUT: top-level-suite :: subdir/test-three
 # CHECK-BASIC-OUT: top-level-suite :: test-one
 # CHECK-BASIC-OUT: top-level-suite :: test-two
 
+# Check the discovery process for multi-configured test suite.
+#
+# RUN: %{lit} %{inputs}/discovery-multicfg \
+# RUN:   -j 1 --debug --show-tests --show-suites \
+# RUN:   -v > %t.out 2> %t.err
+# RUN: FileCheck --check-prefix=CHECK-MULTI-OUT < %t.out %s
+# RUN: FileCheck --check-prefix=CHECK-MULTI-ERR < %t.err %s
+#
+# CHECK-MULTI-OUT: -- Test Suites --
+# CHECK-MULTI-OUT:   sub-suite - 2 tests
+# CHECK-MULTI-OUT:     Source Root: {{.*[/\\]discovery[/\\]subsuite$}}
+# CHECK-MULTI-OUT:     Exec Root  : {{.*[/\\]discovery[/\\]subsuite$}}
+# CHECK-MULTI-OUT:   test-multi-cfg-test-suite :: exec-discovery-in-tree-suite - 1 tests
+# CHECK-MULTI-OUT:     Source Root: {{.*[/\\]exec-discovery-in-tree$}}
+# CHECK-MULTI-OUT:     Exec Root  : {{.*[/\\]exec-discovery-in-tree[/\\]obj$}}
+# CHECK-MULTI-OUT:   test-multi-cfg-test-suite :: top-level-suite - 3 tests
+# CHECK-MULTI-OUT:     Source Root: {{.*[/\\]discovery$}}
+# CHECK-MULTI-OUT:     Exec Root  : {{.*[/\\]exec-discovery$}}
+# CHECK-MULTI-OUT: -- Available Tests --
+# CHECK-MULTI-OUT: sub-suite :: test-one
+# CHECK-MULTI-OUT: sub-suite :: test-two
+# CHECK-MULTI-OUT: exec-discovery-in-tree-suite :: test-one
+# CHECK-MULTI-OUT: top-level-suite :: subdir/test-three
+# CHECK-MULTI-OUT: top-level-suite :: test-one
+# CHECK-MULTI-OUT: top-level-suite :: test-two
+#
+# CHECK-MULTI-ERR: loading suite config '{{.*(/|\\\\)exec-discovery(/|\\\\)lit.site.cfg}}'
+# CHECK-MULTI-ERR-DAG: loading suite config '{{.*(/|\\\\)exec-discovery-in-tree(/|\\\\)obj(/|\\\\)lit.site.cfg}}'
+# CHECK-MULTI-ERR-DAG: loading local config '{{.*(/|\\\\)discovery(/|\\\\)subdir(/|\\\\)lit.local.cfg}}'
+# CHECK-MULTI-ERR-DAG: loading suite config '{{.*(/|\\\\)discovery(/|\\\\)subsuite(/|\\\\)lit.cfg}}'
+
 # Check discovery when providing the special builtin 'config_map'
 # RUN: %{python} %{inputs}/config-map-discovery/driver.py \
 # RUN:           %{inputs}/config-map-discovery/main-config/lit.cfg \
 # RUN:           %{inputs}/config-map-discovery/lit.alt.cfg \
 # RUN:           --workers=1 --debug --show-tests --show-suites > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-CONFIG-MAP-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-CONFIG-MAP-ERR < %t.err %s