[lit] Use path as test suite name

[modocache]

Summary

This resolves an item from utils/lit/TODO, which suggested that the
path to a test suite be used in lit output, instead of the test suite
name specified in the lit config. This makes re-running lit test
suites easier, because users are able to copy-and-paste the test suite
name.

Changing the lit output for LLVM may have unintended consequences, so
to stage the rollout of this feature, add an option to use the old
behavior, --use-test-suite-names. (Swift and other projects that use
LLVM will use the new behavior by default once they update their LLVM
checkouts).

I've tested these changes by running the utils/lit/tests (all passing),
as well as check-llvm with the option enabled and disabled.

[modocache]

utils/lit/TODO

44 ↗

(On Diff #74039)

@ddunbar @delcypher FYI I'm planning on submitting LLVM PRs for each item in this TODO file this weekend, then submitting a diff to delete it.

[modocache]

Update lit documentation to reflect name no longer necessary.

[modocache]

utils/lit/TODO

44 ↗

(On Diff #74039)

Submitted a patch for this, in D25496.

[kastiglione]

utils/lit/tests/discovery.py

14

Can %s or some substitution be used to avoid repetition, and make file renaming more straight forward?

[modocache]

Rebased onto recent lit changes.

[modocache]

utils/lit/tests/discovery.py

14

Good suggestion, but I believe that substitutions are only available for lit RUN commands, not for input to FileCheck. I might be wrong, but I don't think we'll be able to use something like %{inputs}/discovery here.

[modocache]

@delcypher @ddunbar Friendly ping! Let me know if this diff is too big, I could try to split it up. The majority of the changes are to lit's own test suite.

[delcypher]

Sorry for taking so long to look at this. I've taken a quick glance over the code changes. They look reasonable to me. I'm not sure about the output of lit though (see my other comments).

utils/lit/tests/discovery.py

14

@modocache I believe you are correct that the substitutions are only available in RUN: lines. You could use FileCheck's variable support (http://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-variables ) to remove some duplication if you wanted to but I prefer the clarity of how the test is written now.

22–26

I'm not sure if this is the output that @ddunbar intended. One of the motivations for making this change is so that paths to tests could be copied and pasted and given to lit. If I'm reading these test correctly there's :: in between the root path to the test and path to the test in the test suite.

utils/lit/tests/xunit-output.py

8

You should probably talk to some of the consumers of this format about this change. A quick git blame suggestes that talking to Chris Matthews and David Chisnall about this would be a good place to start.

[modocache]

Thanks for the review, @delcypher! I'm going to remove the extraneous :: and update the tests.

utils/lit/tests/discovery.py

22–26

Exactly right! I spoke with @ddunbar today and indeed he suggested I remove this :: . Patch incoming.

utils/lit/tests/xunit-output.py

8

Great idea, thanks! Chris, David, this diff changes lit to output the absolute path to each test file, instead of TestSuiteName :: relative/path/to/test. As a result, this xunit format will become slightly longer. Is this alright?

/cc @cmatthews @theraven

[modocache]

Don't use double colon for test suites using the absolute path instead of test names.

[delcypher]

@modocache Thanks for updating.

ping: @cmathews @theraven

[cmatthews]

It would be a good idea to test this one with a Jenkins instance. The xunit reports were designed to split the test names in a way that seemed natural in the Jenkins test reports.

For example:

http://lab.llvm.org:8080/green/job/clang-stage1-configure-RA_check/24377/testReport/

The test-suite name is mapped into the Package name, all the intermediate folder names are the class name, and the file name gets mapped to the test name. This is to keep the tree to 3 levels.

It may be the case that the buildbots parse these names from the process output, not the xunit file. For example in zorg's LitTestCommand, I believe it is looking for the :: to generate the buildbot test results. Galina might know more about that.

[gkistanova]

LitTestCommand parses LIT printout, looking for the ::, then uses the built "path" to make temp files with log chunks.

I have done a change to skip some intermediate folder names, as it was running out of the MAX_PATH on bots. If you want to have the absolute path there, we should have a reliable way to tell the root then, which could be stripped for these purposes. Ideas?

If you want to change the printout, we would need to change the LitTestCommand in advance, otherwise you would get a lot of bot failures.

I'll see if I can change the LitTestCommand to support both formats during the transition. Unless this is something you plan to do yourself.

[modocache]

Sorry for the noise bringing up this change, I don't think I'll be pursuing this after all.

CMakeLists.txt

 # platforms, where supported.
 if( CMAKE_SIZEOF_VOID_P EQUAL 8 AND NOT WIN32 )
   # TODO: support other platforms and toolchains.
   option(LLVM_BUILD_32_BITS "Build 32 bits executables and libraries." OFF)
 endif()
 
 # Define the default arguments to use with 'lit', and an option for the user to
 # override.
-set(LIT_ARGS_DEFAULT "-sv")
+# FIXME(modocache): The --use-test-suite-names option exists purely to stage
+#                   the migration to using paths as test suite names. It will
+#                   be removed.
+set(LIT_ARGS_DEFAULT "-sv --use-test-suite-names")
 if (MSVC_IDE OR XCODE)
   set(LIT_ARGS_DEFAULT "${LIT_ARGS_DEFAULT} --no-progress-bar")
 endif()
 set(LLVM_LIT_ARGS "${LIT_ARGS_DEFAULT}" CACHE STRING "Default options for lit")
 
 # On Win32 hosts, provide an option to specify the path to the GnuWin32 tools.
 if( WIN32 AND NOT CYGWIN )
   set(LLVM_LIT_TOOLS_DIR "" CACHE PATH "Path to GnuWin32 tools")

docs/CommandGuide/lit.rst

 
 **config**
 
  This is the config object (a *TestingConfig* instance) for the test suite,
  which the config file is expected to populate.  The following variables are also
  available on the *config* object, some of which must be set by the config and
  others are optional or predefined:
 
- **name** *[required]* The name of the test suite, for use in reports and
- diagnostics.
-
  **test_format** *[required]* The test format object which will be used to
  discover and run tests in the test suite.  Generally this will be a builtin test
  format available from the *lit.formats* module.
 
  **test_source_root** The filesystem path to the test suite root.  For out-of-dir
  builds this is the directory that will be scanned for tests.
 
  **test_exec_root** For out-of-dir builds, the path to the test suite root inside

utils/lit/lit/LitConfig.py

class LitConfig(object):
     easily.
     """
 
     def __init__(self, progname, path, quiet,
                  useValgrind, valgrindLeakCheck, valgrindArgs,
                  noExecute, debug, isWindows,
                  params, config_prefix = None,
                  maxIndividualTestTime = 0,
-                 maxFailures = None):
+                 maxFailures = None,
+                 useTestSuiteNames = False):
         # The name of the test runner.
         self.progname = progname
         # The items to add to the PATH environment variable.
         self.path = [str(p) for p in path]
         self.quiet = bool(quiet)
         self.useValgrind = bool(useValgrind)
         self.valgrindLeakCheck = bool(valgrindLeakCheck)
         self.valgrindUserArgs = list(valgrindArgs)
def __init__(self, progname, path, quiet,
                 self.valgrindArgs.append('--leak-check=full')
             else:
                 # The default is 'summary'.
                 self.valgrindArgs.append('--leak-check=no')
             self.valgrindArgs.extend(self.valgrindUserArgs)
 
         self.maxIndividualTestTime = maxIndividualTestTime
         self.maxFailures = maxFailures
+        self.useTestSuiteNames = useTestSuiteNames
 
     @property
     def maxIndividualTestTime(self):
         """
             Interface for getting maximum time to spend executing
             a single test
         """
         return self._maxIndividualTestTime

utils/lit/lit/Test.py

def setResult(self, result):
         # Apply the XFAIL handling to resolve the result exit code.
         if self.isExpectedToFail():
             if self.result.code == PASS:
                 self.result.code = XPASS
             elif self.result.code == FAIL:
                 self.result.code = XFAIL
         
     def getFullName(self):
-        return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
+        return self.suite.config.name + self.suite.config.separator + \
+            '/'.join(self.path_in_suite)
 
     def getFilePath(self):
         if self.file_path:
             return self.file_path
         return self.getSourcePath()
 
     def getSourcePath(self):
         return self.suite.getSourcePath(self.path_in_suite)

utils/lit/lit/TestingConfig.py

 import os
 import sys
 
 
 class TestingConfig:
     """"
     TestingConfig - Information on the tests inside a suite.
     """
 
+    # The name used for test suites with lit configs that don't
+    # specify a name, and are run with --use-test-suite-names.
+    _unnamed_test_suite_name = '<unnamed>'
+
     @staticmethod
     def fromdefaults(litConfig):
         """
         fromdefaults(litConfig) -> TestingConfig
 
         Create a TestingConfig object with default values.
         """
         # Set the environment based on the command line arguments.
def fromdefaults(litConfig):
 
         # Set the default available features based on the LitConfig.
         available_features = []
         if litConfig.useValgrind:
             available_features.append('valgrind')
             if litConfig.valgrindLeakCheck:
                 available_features.append('vg_leak')
 
+        # FIXME(modocache): The double colon separator exists purely to stage
+        #                   the migration to using paths as test suite names.
+        #                   It will be removed.
+        if litConfig.useTestSuiteNames:
+            test_suite_name_separator = ' :: '
+        else:
+            test_suite_name_separator = '/'
+
         return TestingConfig(None,
-                             name = '<unnamed>',
+                             name = TestingConfig._unnamed_test_suite_name,
+                             separator = test_suite_name_separator,
                              suffixes = set(),
                              test_format = None,
                              environment = environment,
                              substitutions = [],
                              unsupported = False,
                              test_exec_root = None,
                              test_source_root = None,
                              excludes = [],
def load_from_path(self, path, litConfig):
 
         # Execute the config script to initialize the object.
         cfg_globals = dict(globals())
         cfg_globals['config'] = self
         cfg_globals['lit_config'] = litConfig
         cfg_globals['__file__'] = path
         try:
             exec(compile(data, path, 'exec'), cfg_globals, None)
+
+            if not litConfig.useTestSuiteNames:
+                if self.name != self.__class__._unnamed_test_suite_name:
+                    litConfig.note(
+                        'Test suite names have been deprecated. The name '
+                        'specified by the the configuration, "{}", will be '
+                        'ignored.'.format(self.name))
+                self.name = os.path.dirname(path)
+
             if litConfig.debug:
                 litConfig.note('... loaded config %r' % path)
         except SystemExit:
             e = sys.exc_info()[1]
             # We allow normal system exit inside a config file to just
             # return control without error.
             if e.args:
                 raise
         except:
             import traceback
             litConfig.fatal(
                 'unable to parse config file %r, traceback: %s' % (
                     path, traceback.format_exc()))
 
         self.finish(litConfig)
 
-    def __init__(self, parent, name, suffixes, test_format,
+    def __init__(self, parent, name, separator, suffixes, test_format,
                  environment, substitutions, unsupported,
                  test_exec_root, test_source_root, excludes,
                  available_features, pipefail, limit_to_features = [],
                  is_early = False):
         self.parent = parent
         self.name = str(name)
+        self.separator = separator
         self.suffixes = set(suffixes)
         self.test_format = test_format
         self.environment = dict(environment)
         self.substitutions = list(substitutions)
         self.unsupported = unsupported
         self.test_exec_root = test_exec_root
         self.test_source_root = test_source_root
         self.excludes = set(excludes)

utils/lit/lit/main.py

debug_group.add_argument("--debug",
                       help="Enable debugging (for 'lit' development)",
                       action="store_true", default=False)
     debug_group.add_argument("--show-suites", dest="showSuites",
                       help="Show discovered test suites",
                       action="store_true", default=False)
     debug_group.add_argument("--show-tests", dest="showTests",
                       help="Show all discovered tests",
                       action="store_true", default=False)
+    debug_group.add_argument(
+        "--use-test-suite-names",
+        help="Display the test suite name specified in the 'lit' config " +
+             "file, instead of the path to that config file. This is " +
+             "deprecated behavior that will be removed in an upcoming " +
+             "version of 'lit'",
+        action="store_true",
+        dest="useTestSuiteNames")
     debug_group.add_argument("--use-processes", dest="useProcesses",
                       help="Run tests in parallel with processes (not threads)",
                       action="store_true", default=True)
     debug_group.add_argument("--use-threads", dest="useProcesses",
                       help="Run tests in parallel with threads (not processes)",
                       action="store_false", default=True)
 
     opts = parser.parse_args()
litConfig = lit.LitConfig.LitConfig(
         valgrindLeakCheck = opts.valgrindLeakCheck,
         valgrindArgs = opts.valgrindArgs,
         noExecute = opts.noExecute,
         debug = opts.debug,
         isWindows = isWindows,
         params = userParams,
         config_prefix = opts.configPrefix,
         maxIndividualTestTime = maxIndividualTestTime,
-        maxFailures = opts.maxFailures)
+        maxFailures = opts.maxFailures,
+        useTestSuiteNames = opts.useTestSuiteNames)
 
     # Perform test discovery.
     run = lit.run.Run(litConfig,
                       lit.discovery.find_tests_for_inputs(litConfig, inputs))
 
     # After test discovery the configuration might have changed
     # the maxIndividualTestTime. If we explicitly set this on the
     # command line then override what was set in the test configuration

utils/lit/tests/Inputs/discovery/lit.cfg

 import lit.formats
-config.name = 'top-level-suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 
 # We intentionally don't set the source root or exec root directories here,
 # because this suite gets reused for testing the exec root behavior (in
 # ../exec-discovery).
 #
 #config.test_source_root = None
 #config.test_exec_root = None
 
 # Check that arbitrary config values are copied (tested by subdir/lit.local.cfg).
 config.an_extra_variable = False

utils/lit/tests/Inputs/discovery/subsuite/lit.cfg

 import lit.formats
-config.name = 'sub-suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None

utils/lit/tests/Inputs/exec-discovery-in-tree/lit.cfg

 import lit.formats
 
 # Verify that the site configuration was loaded.
 if config.test_source_root is None or config.test_exec_root is None:
     lit_config.fatal("No site specific configuration")
 
-config.name = 'exec-discovery-in-tree-suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()

utils/lit/tests/Inputs/googletest-format/lit.cfg

 import lit.formats
-config.name = 'googletest-format'
 config.test_format = lit.formats.GoogleTest('DummySubDir', 'Test')

utils/lit/tests/Inputs/googletest-timeout/lit.cfg

 import lit.formats
-config.name = 'googletest-timeout'
 config.test_format = lit.formats.GoogleTest('DummySubDir', 'Test')
 
 configSetTimeout = lit_config.params.get('set_timeout', '0')
 
 if configSetTimeout == '1':
     # Try setting the max individual test time in the configuration
     lit_config.maxIndividualTestTime = 1

utils/lit/tests/Inputs/googletest-upstream-format/lit.cfg

 import lit.formats
-config.name = 'googletest-upstream-format'
 config.test_format = lit.formats.GoogleTest('DummySubDir', 'Test')

utils/lit/tests/Inputs/progress-bar/lit.cfg

 import lit.formats
-config.name = 'shtest-shell'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None

utils/lit/tests/Inputs/shtest-format/lit.cfg

 import lit.formats
-config.name = 'shtest-format'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None
 config.target_triple = 'x86_64-unknown-unknown'
 config.available_features.add('a-present-feature')

utils/lit/tests/Inputs/shtest-output-printing/lit.cfg

 import lit.formats
-config.name = 'shtest-output-printing'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest(execute_external=False)

utils/lit/tests/Inputs/shtest-shell/lit.cfg

 import lit.formats
-config.name = 'shtest-shell'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None

utils/lit/tests/Inputs/shtest-timeout/lit.cfg

 # -*- Python -*-
 import os
 import sys
 
 import lit.formats
 
-config.name = 'per_test_timeout'
-
 shellType = lit_config.params.get('external', '1')
 
 if shellType == '0':
     lit_config.note('Using internal shell')
     externalShell = False
 else:
     lit_config.note('Using external shell')
     externalShell = True

utils/lit/tests/Inputs/test-data/lit.cfg

def execute(self, test, lit_config):
             elif isinstance(value, float):
                 metric = lit.Test.RealMetricValue(value)
             else:
                 raise RuntimeError("unsupported result type")
             result.addMetric(key, metric)
 
         return result
 
-config.name = 'test-data'
 config.suffixes = ['.ini']
 config.test_format = DummyFormat()
 config.test_source_root = None
 config.test_exec_root = None
 config.target_triple = None

utils/lit/tests/Inputs/unittest-adaptor/lit.cfg

 import lit.formats
-config.name = 'unittest-adaptor'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None

utils/lit/tests/Inputs/use-test-suite-names/named/lit.cfg

 import lit.formats
-config.name = 'shtest-shell'
+config.name = 'A named test suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None
+

utils/lit/tests/Inputs/use-test-suite-names/named/test-one.txt

+# RUN: true
+

utils/lit/tests/Inputs/use-test-suite-names/unnamed/lit.cfg

 import lit.formats
-config.name = 'sub-suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 config.test_source_root = None
 config.test_exec_root = None
+

utils/lit/tests/Inputs/use-test-suite-names/unnamed/test-one.txt

+# RUN: true
+

utils/lit/tests/discovery.py

 # Check the basic discovery process, including a sub-suite.
 #
 # RUN: %{lit} %{inputs}/discovery \
 # RUN:   -j 1 --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-BASIC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-BASIC-ERR < %t.err %s
 #
 # CHECK-BASIC-ERR: loading suite config '{{.*}}/discovery/lit.cfg'
 # CHECK-BASIC-ERR-DAG: loading suite config '{{.*}}/discovery/subsuite/lit.cfg'
 # CHECK-BASIC-ERR-DAG: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 #
 # CHECK-BASIC-OUT: -- Test Suites --
-# CHECK-BASIC-OUT:   sub-suite - 2 tests
-# CHECK-BASIC-OUT:     Source Root: {{.*/discovery/subsuite$}}
-# CHECK-BASIC-OUT:     Exec Root  : {{.*/discovery/subsuite$}}
-# CHECK-BASIC-OUT:   top-level-suite - 3 tests
+# CHECK-BASIC-OUT:   {{.*/discovery}} - 3 tests
 # CHECK-BASIC-OUT:     Source Root: {{.*/discovery$}}
 # CHECK-BASIC-OUT:     Exec Root  : {{.*/discovery$}}
+# CHECK-BASIC-OUT:   {{.*/discovery/subsuite}} - 2 tests
+# CHECK-BASIC-OUT:     Source Root: {{.*/discovery/subsuite$}}
+# CHECK-BASIC-OUT:     Exec Root  : {{.*/discovery/subsuite$}}
 #
 # CHECK-BASIC-OUT: -- Available Tests --
-# CHECK-BASIC-OUT: sub-suite :: test-one
-# CHECK-BASIC-OUT: sub-suite :: test-two
-# CHECK-BASIC-OUT: top-level-suite :: subdir/test-three
-# CHECK-BASIC-OUT: top-level-suite :: test-one
-# CHECK-BASIC-OUT: top-level-suite :: test-two
+# CHECK-BASIC-OUT: {{.*/discovery/subdir/test-three}}
+# CHECK-BASIC-OUT: {{.*/discovery/test-one}}
+# CHECK-BASIC-OUT: {{.*/discovery/test-two}}
+# CHECK-BASIC-OUT: {{.*/discovery/subsuite/test-one}}
+# CHECK-BASIC-OUT: {{.*/discovery/subsuite/test-two}}
 
 
 # Check discovery when exact test names are given.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/discovery/subdir/test-three.py \
 # RUN:     %{inputs}/discovery/subsuite/test-one.txt \
 # RUN:   -j 1 --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-EXACT-TEST < %t.out %s
 #
 # CHECK-EXACT-TEST: -- Available Tests --
-# CHECK-EXACT-TEST: sub-suite :: test-one
-# CHECK-EXACT-TEST: top-level-suite :: subdir/test-three
+# CHECK-EXACT-TEST: {{.*/discovery/subdir/test-three}}
+# CHECK-EXACT-TEST: {{.*/discovery/subsuite/test-one}}
 
 
 # Check discovery when using an exec path.
 #
 # RUN: %{lit} %{inputs}/exec-discovery \
 # RUN:   -j 1 --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-ERR < %t.err %s
 #
 # CHECK-ASEXEC-ERR: loading suite config '{{.*}}/exec-discovery/lit.site.cfg'
 # CHECK-ASEXEC-ERR: load_config from '{{.*}}/discovery/lit.cfg'
 # CHECK-ASEXEC-ERR: loaded config '{{.*}}/discovery/lit.cfg'
 # CHECK-ASEXEC-ERR: loaded config '{{.*}}/exec-discovery/lit.site.cfg'
 # CHECK-ASEXEC-ERR-DAG: loading suite config '{{.*}}/discovery/subsuite/lit.cfg'
 # CHECK-ASEXEC-ERR-DAG: loading local config '{{.*}}/discovery/subdir/lit.local.cfg'
 #
 # CHECK-ASEXEC-OUT: -- Test Suites --
-# CHECK-ASEXEC-OUT:   sub-suite - 2 tests
+# CHECK-ASEXEC-OUT:   {{.*/discovery/subsuite}} - 2 tests
 # CHECK-ASEXEC-OUT:     Source Root: {{.*/discovery/subsuite$}}
 # CHECK-ASEXEC-OUT:     Exec Root  : {{.*/discovery/subsuite$}}
-# CHECK-ASEXEC-OUT:   top-level-suite - 3 tests
+# CHECK-ASEXEC-OUT:   {{.*/exec-discovery}} - 3 tests
 # CHECK-ASEXEC-OUT:     Source Root: {{.*/discovery$}}
 # CHECK-ASEXEC-OUT:     Exec Root  : {{.*/exec-discovery$}}
 #
 # CHECK-ASEXEC-OUT: -- Available Tests --
-# CHECK-ASEXEC-OUT: sub-suite :: test-one
-# CHECK-ASEXEC-OUT: sub-suite :: test-two
-# CHECK-ASEXEC-OUT: top-level-suite :: subdir/test-three
-# CHECK-ASEXEC-OUT: top-level-suite :: test-one
-# CHECK-ASEXEC-OUT: top-level-suite :: test-two
+# CHECK-ASEXEC-OUT: {{.*/discovery/subsuite/test-one}}
+# CHECK-ASEXEC-OUT: {{.*/discovery/subsuite/test-two}}
+# CHECK-ASEXEC-OUT: {{.*/exec-discovery/subdir/test-three}}
+# CHECK-ASEXEC-OUT: {{.*/exec-discovery/test-one}}
+# CHECK-ASEXEC-OUT: {{.*/exec-discovery/test-two}}
 
 # Check discovery when exact test names are given.
 #
 # FIXME: Note that using a path into a subsuite doesn't work correctly here.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/exec-discovery/subdir/test-three.py \
 # RUN:   -j 1 --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-EXACT-TEST < %t.out %s
 #
 # CHECK-ASEXEC-EXACT-TEST: -- Available Tests --
-# CHECK-ASEXEC-EXACT-TEST: top-level-suite :: subdir/test-three
+# CHECK-ASEXEC-EXACT-TEST: {{.*/exec-discovery/subdir/test-three}}
 
 
 # Check that we don't recurse infinitely when loading an site specific test
 # suite located inside the test source root.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/exec-discovery-in-tree/obj/ \
 # RUN:   -j 1 --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-INTREE < %t.out %s
 #
-#      CHECK-ASEXEC-INTREE:   exec-discovery-in-tree-suite - 1 tests
+#      CHECK-ASEXEC-INTREE:   {{.*/exec-discovery-in-tree/obj}} - 1 tests
 # CHECK-ASEXEC-INTREE-NEXT:     Source Root: {{.*/exec-discovery-in-tree$}}
 # CHECK-ASEXEC-INTREE-NEXT:     Exec Root  : {{.*/exec-discovery-in-tree/obj$}}
 # CHECK-ASEXEC-INTREE-NEXT: -- Available Tests --
-# CHECK-ASEXEC-INTREE-NEXT: exec-discovery-in-tree-suite :: test-one
+# CHECK-ASEXEC-INTREE-NEXT: {{.*/exec-discovery-in-tree/obj/test-one}}

utils/lit/tests/googletest-format.py

 # Check the various features of the GoogleTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
-# CHECK: PASS: googletest-format :: DummySubDir/OneTest/FirstTest.subTestA
-# CHECK: FAIL: googletest-format :: DummySubDir/OneTest/FirstTest.subTestB
-# CHECK-NEXT: *** TEST 'googletest-format :: DummySubDir/OneTest/FirstTest.subTestB' FAILED ***
+# CHECK: PASS: {{.*/googletest-format/DummySubDir/OneTest/FirstTest.subTestA}}
+# CHECK: FAIL: {{.*/googletest-format/DummySubDir/OneTest/FirstTest.subTestB}}
+# CHECK-NEXT: *** TEST '{{.*/googletest-format/DummySubDir/OneTest/FirstTest.subTestB}}' FAILED ***
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
-# CHECK: PASS: googletest-format :: DummySubDir/OneTest/ParameterizedTest/0.subTest
-# CHECK: PASS: googletest-format :: DummySubDir/OneTest/ParameterizedTest/1.subTest
+# CHECK: PASS: {{.*/googletest-format/DummySubDir/OneTest/ParameterizedTest/0.subTest}}
+# CHECK: PASS: {{.*/googletest-format/DummySubDir/OneTest/ParameterizedTest/1.subTest}}
 # CHECK: Failing Tests (1)
 # CHECK: Expected Passes    : 3
 # CHECK: Unexpected Failures: 1
 

utils/lit/tests/googletest-timeout.py

 # REQUIRES: python-psutil
 
 # Check that the per test timeout is enforced when running GTest tests.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout --timeout=1 > %t.cmd.out
 # RUN: FileCheck < %t.cmd.out %s
 
 # Check that the per test timeout is enforced when running GTest tests via
 # the configuration file
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
 # RUN: --param set_timeout=1 > %t.cfgset.out 2> %t.cfgset.err
 # RUN: FileCheck < %t.cfgset.out %s
 
 # CHECK: -- Testing:
-# CHECK: PASS: googletest-timeout :: DummySubDir/OneTest/FirstTest.subTestA
-# CHECK: TIMEOUT: googletest-timeout :: DummySubDir/OneTest/FirstTest.subTestB
-# CHECK: TIMEOUT: googletest-timeout :: DummySubDir/OneTest/FirstTest.subTestC
+# CHECK: PASS: {{.*/googletest-timeout/DummySubDir/OneTest/FirstTest.subTestA}}
+# CHECK: TIMEOUT: {{.*/googletest-timeout/DummySubDir/OneTest/FirstTest.subTestB}}
+# CHECK: TIMEOUT: {{.*/googletest-timeout/DummySubDir/OneTest/FirstTest.subTestC}}
 # CHECK: Expected Passes    : 1
 # CHECK: Individual Timeouts: 2
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
 # RUN: --param set_timeout=1 --timeout=2 > %t.cmdover.out 2> %t.cmdover.err
 # RUN: FileCheck < %t.cmdover.out %s
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.cmdover.err %s
 
 # CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 2 seconds

utils/lit/tests/googletest-upstream-format.py

 # Check the various features of the GoogleTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-upstream-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
-# CHECK: PASS: googletest-upstream-format :: DummySubDir/OneTest/FirstTest.subTestA
-# CHECK: FAIL: googletest-upstream-format :: DummySubDir/OneTest/FirstTest.subTestB
-# CHECK-NEXT: *** TEST 'googletest-upstream-format :: DummySubDir/OneTest/FirstTest.subTestB' FAILED ***
+# CHECK: PASS: {{.*/googletest-upstream-format/DummySubDir/OneTest/FirstTest.subTestA}}
+# CHECK: FAIL: {{.*/googletest-upstream-format/DummySubDir/OneTest/FirstTest.subTestB}}
+# CHECK-NEXT: *** TEST '{{.*/googletest-upstream-format/DummySubDir/OneTest/FirstTest.subTestB}}' FAILED ***
 # CHECK-NEXT: Running main() from gtest_main.cc
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
-# CHECK: PASS: googletest-upstream-format :: DummySubDir/OneTest/ParameterizedTest/0.subTest
-# CHECK: PASS: googletest-upstream-format :: DummySubDir/OneTest/ParameterizedTest/1.subTest
+# CHECK: PASS: {{.*/googletest-upstream-format/DummySubDir/OneTest/ParameterizedTest/0.subTest}}
+# CHECK: PASS: {{.*/googletest-upstream-format/DummySubDir/OneTest/ParameterizedTest/1.subTest}}
 # CHECK: Failing Tests (1)
 # CHECK: Expected Passes    : 3
 # CHECK: Unexpected Failures: 1

utils/lit/tests/lit.cfg

 # -*- Python -*-
 
 import os
 import sys
 
 import lit.formats
 
 # Configuration file for the 'lit' test runner.
 
-# name: The name of this test suite.
-config.name = 'lit'
-
 # testFormat: The test format to use to interpret tests.
 config.test_format = lit.formats.ShTest(execute_external=False)
 
 # suffixes: A list of file extensions to treat as test files.
 config.suffixes = ['.py']
 
 # excludes: A list of individual files to exclude.
 config.excludes = ['Inputs']

utils/lit/tests/progress-bar.py

 # Check the simple progress bar.
 #
 # RUN: not %{lit} -j 1 -s %{inputs}/progress-bar > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # CHECK: Testing: 0 .. 10.. 20
-# CHECK: FAIL: shtest-shell :: test-1.txt (1 of 4)
+# CHECK: FAIL: {{.*/progress-bar/test-1.txt}} (1 of 4)
 # CHECK: Testing: 0 .. 10.. 20.. 30.. 40.. 
-# CHECK: FAIL: shtest-shell :: test-2.txt (2 of 4)
+# CHECK: FAIL: {{.*/progress-bar/test-2.txt}} (2 of 4)
 # CHECK: Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70
-# CHECK: FAIL: shtest-shell :: test-3.txt (3 of 4)
+# CHECK: FAIL: {{.*/progress-bar/test-3.txt}} (3 of 4)
 # CHECK: Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
-# CHECK: FAIL: shtest-shell :: test-4.txt (4 of 4)
+# CHECK: FAIL: {{.*/progress-bar/test-4.txt}} (4 of 4)

utils/lit/tests/shtest-format.py

 # Check the various features of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
-# CHECK: PASS: shtest-format :: argv0.txt
-# CHECK: FAIL: shtest-format :: external_shell/fail.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail.txt' FAILED ***
+# CHECK: PASS: {{.*/shtest-format/argv0.txt}}
+# CHECK: FAIL: {{.*/shtest-format/external_shell/fail.txt}}
+# CHECK-NEXT: *** TEST '{{.*/shtest-format/external_shell/fail.txt}}' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 # CHECK: Command Output (stderr):
 # CHECK-NEXT: --
 # CHECK-NEXT: cat: does-not-exist: No such file or directory
 # CHECK: --
 
-# CHECK: FAIL: shtest-format :: external_shell/fail_with_bad_encoding.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail_with_bad_encoding.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-format/external_shell/fail_with_bad_encoding.txt}}
+# CHECK-NEXT: *** TEST '{{.*/shtest-format/external_shell/fail_with_bad_encoding.txt}}' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: a line with bad encoding:
 # CHECK: --
 
-# CHECK: PASS: shtest-format :: external_shell/pass.txt
+# CHECK: PASS: {{.*/shtest-format/external_shell/pass.txt}}
 
-# CHECK: FAIL: shtest-format :: fail.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: fail.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-format/fail.txt}}
+# CHECK-NEXT: *** TEST '{{.*/shtest-format/fail.txt}}' FAILED ***
 # CHECK-NEXT: Script:
 # CHECK-NEXT: --
 # CHECK-NEXT: printf "line 1
 # CHECK-NEXT: false
 # CHECK-NEXT: --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: $ "printf"
 # CHECK-NEXT: # command output:
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 
-# CHECK: UNRESOLVED: shtest-format :: no-test-line.txt
-# CHECK: PASS: shtest-format :: pass.txt
-# CHECK: UNSUPPORTED: shtest-format :: requires-any-missing.txt
-# CHECK: PASS: shtest-format :: requires-any-present.txt
-# CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
-# CHECK: PASS: shtest-format :: requires-present.txt
-# CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
-# CHECK: XFAIL: shtest-format :: xfail-feature.txt
-# CHECK: XFAIL: shtest-format :: xfail-target.txt
-# CHECK: XFAIL: shtest-format :: xfail.txt
-# CHECK: XPASS: shtest-format :: xpass.txt
-# CHECK-NEXT: *** TEST 'shtest-format :: xpass.txt' FAILED ***
+# CHECK: UNRESOLVED: {{.*/shtest-format/no-test-line.txt}}
+# CHECK: PASS: {{.*/shtest-format/pass.txt}}
+# CHECK: UNSUPPORTED: {{.*/shtest-format/requires-any-missing.txt}}
+# CHECK: PASS: {{.*/shtest-format/requires-any-present.txt}}
+# CHECK: UNSUPPORTED: {{.*/shtest-format/requires-missing.txt}}
+# CHECK: PASS: {{.*/shtest-format/requires-present.txt}}
+# CHECK: UNSUPPORTED: {{.*/shtest-format/unsupported_dir/some-test.txt}}
+# CHECK: XFAIL: {{.*/shtest-format/xfail-feature.txt}}
+# CHECK: XFAIL: {{.*/shtest-format/xfail-target.txt}}
+# CHECK: XFAIL: {{.*/shtest-format/xfail.txt}}
+# CHECK: XPASS: {{.*/shtest-format/xpass.txt}}
+# CHECK-NEXT: *** TEST '{{.*/shtest-format/xpass.txt}}' FAILED ***
 # CHECK-NEXT: Script
 # CHECK-NEXT: --
 # CHECK-NEXT: true
 # CHECK-NEXT: --
 # CHECK: Testing Time
 
 # CHECK: Unexpected Passing Tests (1)
-# CHECK: shtest-format :: xpass.txt
+# CHECK: {{.*/shtest-format/xpass.txt}}
 
 # CHECK: Failing Tests (3)
-# CHECK: shtest-format :: external_shell/fail.txt
-# CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
-# CHECK: shtest-format :: fail.txt
+# CHECK: {{.*/shtest-format/external_shell/fail.txt}}
+# CHECK: {{.*/shtest-format/external_shell/fail_with_bad_encoding.txt}}
+# CHECK: {{.*/shtest-format/fail.txt}}
 
 # CHECK: Expected Passes    : 5
 # CHECK: Expected Failures  : 3
 # CHECK: Unsupported Tests  : 3
 # CHECK: Unresolved Tests   : 1
 # CHECK: Unexpected Passes  : 1
 # CHECK: Unexpected Failures: 3

utils/lit/tests/shtest-output-printing.py

 # Check the various features of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-output-printing > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
-# CHECK: FAIL: shtest-output-printing :: basic.txt
-# CHECK-NEXT: *** TEST 'shtest-output-printing :: basic.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-output-printing/basic.txt}}
+# CHECK-NEXT: *** TEST '{{.*/shtest-output-printing/basic.txt}}' FAILED ***
 # CHECK-NEXT: Script:
 # CHECK-NEXT: --
 # CHECK:      --
 # CHECK-NEXT: Exit Code: 1
 #
 # CHECK:      Command Output
 # CHECK-NEXT: --
 # CHECK-NEXT: $ "true"

utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
-# CHECK: FAIL: shtest-shell :: error-0.txt
-# CHECK: *** TEST 'shtest-shell :: error-0.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-shell/error-0.txt}}
+# CHECK: *** TEST '{{.*/shtest-shell/error-0.txt}}' FAILED ***
 # CHECK: $ "not-a-real-command"
 # CHECK: # command stderr:
 # CHECK: 'not-a-real-command': command not found
 # CHECK: error: command failed with exit status: 127
 # CHECK: ***
 
 # FIXME: The output here sucks.
 #
-# CHECK: FAIL: shtest-shell :: error-1.txt
-# CHECK: *** TEST 'shtest-shell :: error-1.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-shell/error-1.txt}}
+# CHECK: *** TEST '{{.*/shtest-shell/error-1.txt}}' FAILED ***
 # CHECK: shell parser error on: 'echo "missing quote'
 # CHECK: ***
 
-# CHECK: FAIL: shtest-shell :: error-2.txt
-# CHECK: *** TEST 'shtest-shell :: error-2.txt' FAILED ***
+# CHECK: FAIL: {{.*/shtest-shell/error-2.txt}}
+# CHECK: *** TEST '{{.*/shtest-shell/error-2.txt}}' FAILED ***
 # CHECK: Unsupported redirect:
 # CHECK: ***
 
-# CHECK: PASS: shtest-shell :: redirects.txt
-# CHECK: PASS: shtest-shell :: sequencing-0.txt
-# CHECK: XFAIL: shtest-shell :: sequencing-1.txt
+# CHECK: PASS: {{.*/shtest-shell/redirects.txt}}
+# CHECK: PASS: {{.*/shtest-shell/sequencing-0.txt}}
+# CHECK: XFAIL: {{.*/shtest-shell/sequencing-1.txt}}
 # CHECK: Failing Tests (3)

utils/lit/tests/shtest-timeout.py

 # RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --timeout 1 --param external=0 > %t.intsh.out 2> %t.intsh.err
 # RUN: FileCheck  --check-prefix=CHECK-OUT-COMMON < %t.intsh.out %s
 # RUN: FileCheck --check-prefix=CHECK-INTSH-OUT < %t.intsh.out %s
 # RUN: FileCheck --check-prefix=CHECK-INTSH-ERR < %t.intsh.err %s
 
-# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: infinite_loop.py
+# CHECK-INTSH-OUT: TIMEOUT: {{.*/shtest-timeout/infinite_loop.py}}
 # CHECK-INTSH-OUT: command output:
 # CHECK-INTSH-OUT-NEXT: Running infinite loop
 # CHECK-INTSH-OUT: command reached timeout: True
 
-# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: quick_then_slow.py
+# CHECK-INTSH-OUT: TIMEOUT: {{.*/shtest-timeout/quick_then_slow.py}}
 # CHECK-INTSH-OUT: Timeout: Reached timeout of 1 seconds
 # CHECK-INTSH-OUT: Command Output
 # CHECK-INTSH-OUT: command output:
 # CHECK-INTSH-OUT-NEXT: Running in quick mode
 # CHECK-INTSH-OUT: command reached timeout: False
 # CHECK-INTSH-OUT: command output:
 # CHECK-INTSH-OUT-NEXT: Running in slow mode
 # CHECK-INTSH-OUT: command reached timeout: True
 
-# CHECK-INTSH-OUT: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-INTSH-OUT: TIMEOUT: {{.*/shtest-timeout/slow.py}}
 # CHECK-INTSH-OUT: command output:
 # CHECK-INTSH-OUT-NEXT: Running slow program
 # CHECK-INTSH-OUT: command reached timeout: True
 
 # CHECK-INTSH-ERR: Using internal shell
 
 # Test per test timeout set via a config file rather than on the command line
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
 # RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --param external=0 \
 # RUN: --param set_timeout=1 > %t.cfgset.out 2> %t.cfgset.err
 # RUN: FileCheck --check-prefix=CHECK-OUT-COMMON  < %t.cfgset.out %s
 # RUN: FileCheck --check-prefix=CHECK-CFGSET-ERR < %t.cfgset.err %s
 #
 # CHECK-CFGSET-ERR: Using internal shell
 
-# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: infinite_loop.py
+# CHECK-OUT-COMMON: TIMEOUT: {{.*/shtest-timeout/infinite_loop.py}}
 # CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
 # CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
 # CHECK-OUT-COMMON: Running infinite loop
 
-# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: quick_then_slow.py
+# CHECK-OUT-COMMON: TIMEOUT: {{.*/shtest-timeout/quick_then_slow.py}}
 # CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
 # CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
 # CHECK-OUT-COMMON: Running in quick mode
 # CHECK-OUT-COMMON: Running in slow mode
 
-# CHECK-OUT-COMMON: PASS: per_test_timeout :: short.py
+# CHECK-OUT-COMMON: PASS: {{.*/shtest-timeout/short.py}}
 
-# CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-OUT-COMMON: TIMEOUT: {{.*/shtest-timeout/slow.py}}
 # CHECK-OUT-COMMON: Timeout: Reached timeout of 1 seconds
 # CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
 # CHECK-OUT-COMMON: Running slow program
 
 # CHECK-OUT-COMMON: Expected Passes{{ *}}: 1
 # CHECK-OUT-COMMON: Individual Timeouts{{ *}}: 3
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
 # RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --param external=0 \
 # RUN: --param set_timeout=1 --timeout=2 > %t.cmdover.out 2> %t.cmdover.err
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-OUT  < %t.cmdover.out %s
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.cmdover.err %s
 
 # CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 2 seconds
 
-# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: infinite_loop.py
+# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: {{.*/shtest-timeout/infinite_loop.py}}
 # CHECK-CMDLINE-OVERRIDE-OUT: Timeout: Reached timeout of 2 seconds
 # CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
 # CHECK-CMDLINE-OVERRIDE-OUT: Running infinite loop
 
-# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: quick_then_slow.py
+# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: {{.*/shtest-timeout/quick_then_slow.py}}
 # CHECK-CMDLINE-OVERRIDE-OUT: Timeout: Reached timeout of 2 seconds
 # CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
 # CHECK-CMDLINE-OVERRIDE-OUT: Running in quick mode
 # CHECK-CMDLINE-OVERRIDE-OUT: Running in slow mode
 
-# CHECK-CMDLINE-OVERRIDE-OUT: PASS: per_test_timeout :: short.py
+# CHECK-CMDLINE-OVERRIDE-OUT: PASS: {{.*/shtest-timeout/short.py}}
 
-# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: slow.py
+# CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: {{.*/shtest-timeout/slow.py}}
 # CHECK-CMDLINE-OVERRIDE-OUT: Timeout: Reached timeout of 2 seconds
 # CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
 # CHECK-CMDLINE-OVERRIDE-OUT: Running slow program
 
 # CHECK-CMDLINE-OVERRIDE-OUT: Expected Passes{{ *}}: 1
 # CHECK-CMDLINE-OVERRIDE-OUT: Individual Timeouts{{ *}}: 3

utils/lit/tests/test-data.py

 # Test features related to formats which support reporting additional test data.
 
 # RUN: %{lit} -j 1 -v %{inputs}/test-data > %t.out
 # RUN: FileCheck < %t.out %s
 
 # CHECK: -- Testing:
 
-# CHECK: PASS: test-data :: metrics.ini
-# CHECK-NEXT: *** TEST 'test-data :: metrics.ini' RESULTS ***
+# CHECK: PASS: {{.*/test-data/metrics.ini}}
+# CHECK-NEXT: *** TEST '{{.*/test-data/metrics.ini}}' RESULTS ***
 # CHECK-NEXT: value0: 1
 # CHECK-NEXT: value1: 2.3456
 # CHECK-NEXT: ***

utils/lit/tests/test-output.py

 # RUN: %{lit} -j 1 -v %{inputs}/test-data --output %t.results.out > %t.out
 # RUN: FileCheck < %t.results.out %s
 
 # CHECK: {
 # CHECK: "__version__"
 # CHECK: "elapsed"
 # CHECK-NEXT: "tests": [
 # CHECK-NEXT:   {
 # CHECK-NEXT:     "code": "PASS",
 # CHECK-NEXT:     "elapsed": {{[0-9.]+}},
 # CHECK-NEXT:     "metrics": {
 # CHECK-NEXT:       "value0": 1,
 # CHECK-NEXT:       "value1": 2.3456
 # CHECK-NEXT:     }
-# CHECK-NEXT:     "name": "test-data :: metrics.ini",
+# CHECK-NEXT:     "name": "{{.*/test-data/metrics.ini}}",
 # CHECK-NEXT:     "output": "Test passed."
 # CHECK-NEXT:   }
 # CHECK-NEXT: ]
 # CHECK-NEXT: }

utils/lit/tests/unittest-adaptor.py

 # Check the lit adaption to run under unittest.
 #
 # RUN: %{python} %s %{inputs}/unittest-adaptor 2> %t.err
 # RUN: FileCheck < %t.err %s
 #
-# CHECK-DAG: unittest-adaptor :: test-two.txt ... FAIL
-# CHECK-DAG: unittest-adaptor :: test-one.txt ... ok
+# CHECK-DAG: unittest-adaptor/test-two.txt ... FAIL
+# CHECK-DAG: unittest-adaptor/test-one.txt ... ok
 
 import unittest
 import sys
 
 import lit
 import lit.discovery
 
 input_path = sys.argv[1]
 unittest_suite = lit.discovery.load_test_suite([input_path])
 runner = unittest.TextTestRunner(verbosity=2)
 runner.run(unittest_suite)

utils/lit/tests/use-test-suite-names.py

+# Check the behavior of the --use-test-suite-names option.
+
+# When --use-test-suite-names is not specified, the path to the test
+# suite is used instead of its name, and a note is printed to stderr.
+
+# RUN: %{lit} -j 1 -v %{inputs}/use-test-suite-names/named > %t.named.out 2> %t.named.err
+# RUN: FileCheck < %t.named.out %s
+# RUN: FileCheck < %t.named.err %s --check-prefix=CHECK-ERR
+# CHECK: {{.*/use-test-suite-names/named/test-one.txt}}
+# CHECK-ERR: Test suite names have been deprecated.{{.*}}"A named test suite"
+
+# When --use-test-suite-names is specified, the name is used instead of
+# the path to the suite. Nothing is printed to stderr.
+
+# RUN: %{lit} -j 1 -v --use-test-suite-names %{inputs}/use-test-suite-names/named 2>&1 > %t.named.usename.out
+# RUN: FileCheck < %t.named.usename.out %s --check-prefix=CHECK-USE-NAME
+# CHECK-USE-NAME: A named test suite :: test-one.txt
+# CHECK-USE-NAME-NOT: Test suite names have been deprecated.
+
+# When --use-test-suite-names is specified, but no name is given,
+# the suite is named "<unnamed>". Nothing is printed to stderr.
+
+# RUN: %{lit} -j 1 -v --use-test-suite-names %{inputs}/use-test-suite-names/unnamed 2>&1 > %t.unnamed.usename.out
+# RUN: FileCheck < %t.unnamed.usename.out %s --check-prefix=CHECK-UNNAMED-USE-NAME
+# CHECK-UNNAMED-USE-NAME: <unnamed> :: test-one.txt
+# CHECK-UNNAMED-USE-NAME-NOT: Test suite names have been deprecated.
+

utils/lit/tests/xunit-output.py

 # Check xunit output
 # RUN: %{lit} --xunit-xml-output %t.xunit.xml %{inputs}/test-data
 # RUN: FileCheck < %t.xunit.xml %s
 
 # CHECK: <?xml version="1.0" encoding="UTF-8" ?>
 # CHECK: <testsuites>
-# CHECK: <testsuite name='test-data' tests='1' failures='0'>
-# CHECK: <testcase classname='test-data.test-data' name='metrics.ini' time='0.{{[0-9]+}}'/>
+# CHECK: <testsuite name='{{.*/test-data}}' tests='1' failures='0'>
+# CHECK: <testcase classname='{{.*/test-data}}.{{.*/test-data}}' name='metrics.ini' time='0.{{[0-9]+}}'/>
 # CHECK: </testsuite>
 # CHECK: </testsuites>