[LIT] Allow suite configs to be preloaded allowing for better support of out-of-dir builds.

[EricWF]

Summary

This patch was inspired by the following use case when testing libc++:

• The lit.site.cfg is in build-libcxx/test/lit.site.cfg
• Your working out of libcxx/test

Currently the only solution is to copy or symlink the generated lit.site.cfg into the source tree and then teach your VCS to ignore it. This patch allows the user to manually specify configuration files to load. It works by pre-loading the suite configuration file cache. The configuration files can be specified one of two ways:

• using the --config command line parameter
• using the LIT_CONFIG environment variable.

The configuration files specified using --config are loaded before the ones in LIT_CONFIG. configuration files with the same path or the same test source path are not loaded twice.

@ddunbar If you disagree with this direction please let me know.

[EricWF]

I really don't like this patch. There was some suggestion that this feature would go nicely is LIT but I think it's better implemented on a test-suite basis.

[• ddunbar]

I sympathize with the use case, and in fact if you look at LLVM and Clang's
config files they have code which tries to do a trick where if it finds it
is running in the source without a lit.site.cfg, then it tries to infer the
location of the actual exec_root by seeing if you are the kind of user who
puts such things in your path. It's messy and only works for some users and
is complicated, so I'm ultimately not a big fan of that solution, but it
was the first thing I came up with.

The next thing I came up with was the idea of building a tool that baked in
the path to the config file, that is llvm-lit. I'm not all that happy with
that approach either.r

Why did you decide you didn't like this patch?

• Daniel

[EricWF]

It just seemed like an ugly patch and the wrong way to do things. As you mentioned both clang and LLVM already provide similar functionality at almost no extra cost compared to this patch. For that reason it doesn't make sense to dirty up lit to achieve these goals.

docs/CommandGuide/lit.rst

  Run ``N`` tests in parallel.  By default, this is automatically chosen to
  match the number of detected available CPUs.
 
+.. option:: --config=FILE
+
+ Add a test suite configuration to be preloaded into the test suite cache during
+ test discovery. This option can be used to load configuration files that are
+ generated by out-of-dir builds.
+
 .. option:: --config-prefix=NAME
 
  Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for

 
 :program:`lit` identifies test suites as directories containing ``lit.cfg`` or
 ``lit.site.cfg`` files (see also :option:`--config-prefix`).  Test suites are
-initially discovered by recursively searching up the directory hierarchy for
-all the input files passed on the command line.  You can use
-:option:`--show-suites` to display the discovered test suites at startup.
+initially discovered in two ways.
+
+1. By loading the suite configurations specified in the :option:`--config`
+option and the ``LIT_CONFIG`` enviroment variable. Where ``LIT_CONFIG`` is a
+colon separated list of configuration files.
+
+2. By recursively searching up the directory hierarchy for all the input files
+passed on the command line.
+
+
+You can use :option:`--show-suites` to display the discovered test suites at
+startup. (Note: Only suites containing tests will be displayed).
 
 Once a test suite is discovered, its config file is loaded.  Config files
 themselves are Python modules which will be executed.  When the config file is

utils/lit/lit/discovery.py

         if sub_ts and not N:
             litConfig.warning('test suite %r contained no tests' % sub_ts.name)
 
-def find_tests_for_inputs(lit_config, inputs):
+def find_tests_for_inputs(lit_config, inputs, suite_inputs=[]):
     """
     find_tests_for_inputs(lit_config, inputs) -> [Test]
 

                 f.close()
         else:
             actual_inputs.append(input)
-                    
     # Load the tests from the inputs.
     tests = []
     test_suite_cache = {}
+    # Preload the test suite cache by filling it with explicitly specified test
+    # suites. Make sure each test suite is only loaded once by checking that it
+    # doesn't already exist in the cache and an extra list of seen test suites.
+    # NOTE: The path that is used as a key in the cache can differ from the
+    #  path that was used to load the suites.
+    test_suites_seen = set()
+    for cfgpath in suite_inputs:
+        cfgpath = os.path.realpath(cfgpath)
+        if not os.path.isfile(cfgpath):
+            lit_config.warning('Invalid config file: %s' % cfgpath)
+            break
+        if cfgpath in test_suites_seen or \
+           cfgpath in test_suite_cache.keys():
+            break
+        if lit_config.debug:
+            lit_config.note('Loading config file: %s' % cfgpath)
+        test_suites_seen.add(cfgpath)
+        cfg = TestingConfig.fromdefaults(lit_config)
+        cfg.load_from_path(cfgpath, lit_config)
+        path = os.path.dirname(cfgpath)
+        source_root = os.path.realpath(cfg.test_source_root or path)
+        exec_root = os.path.realpath(cfg.test_exec_root or path)
+        test_suite_cache[source_root] = Test.TestSuite(cfg.name, source_root, exec_root, cfg), ()
+
     local_config_cache = {}
     for input in actual_inputs:
         prev = len(tests)

utils/lit/lit/main.py

     parser.add_option("-j", "--threads", dest="numThreads", metavar="N",
                       help="Number of testing threads",
                       type=int, action="store", default=None)
+    parser.add_option('--config', dest="suiteConfigs",
+                      metavar='FILE', help='Extra test suite configs',
+                      type=str, action="append", default=[])
     parser.add_option("", "--config-prefix", dest="configPrefix",
                       metavar="NAME", help="Prefix for 'lit' config files",
                       action="store", default=None)

         params = userParams,
         config_prefix = opts.configPrefix)
 
-    # Perform test discovery.
+    # Perform test discovery. Read in an extra list of configuration files
+    # from the enviroment and the command line and pass that along with the
+    # tests to do test discovery.
+    suite_configs = list(opts.suiteConfigs)
+    env_suites = os.environ.get('LIT_CONFIG')
+    if env_suites is not None:
+        env_suites = re.split(r'[:;]', env_suites)
+        suite_configs += [s.strip() for s in env_suites if s.strip()]
     run = lit.run.Run(litConfig,
-                      lit.discovery.find_tests_for_inputs(litConfig, inputs))
+                      lit.discovery.find_tests_for_inputs(
+                        litConfig, inputs, suite_configs))
 
     if opts.showSuites or opts.showTests:
         # Aggregate the tests by suite.