[llvm] [lit] Support forcing lexical test order

[mgorny]

Summary

Add a new --order option to choose between available test orders:
the default "smart" order, predictable "lexical" order or "random"
order. Default to using lexical order and one job in the lit test
suite.

[jdenny]

Thanks for working on this!

The patch doesn't spell it consistently: sometimes --lexical and sometimes --lexical-order.

Shouldn't we put --lexical-order directly into lit's test suite's %{lit} substitution? That way, if another of lit's current or future tests is susceptible to this problem, no one has to waste time trying to reproduce the race and figure out what's happening. Blindly using %{lit} in lit's test suite would just work.

As far as I know, reorder.py is the only test in lit's test suite that requires the reordering. Either it can use a %{lit-reordered} substitution that drops --lexical-order, or it can use a new command-line option that selects the default order and overrides any previous --lexical-order or --shuffle.

llvm/utils/lit/tests/shtest-shell.py

10–11

Any reason to keep this? It might be worthwhile to grep for lit_test_times, if you didn't already.

[jdenny]

Also, the docs should be updated: https://llvm.org/docs/CommandGuide/lit.html

[mgorny]

In D107695#2932751, @jdenny wrote:

Thanks for working on this!

The patch doesn't spell it consistently: sometimes --lexical and sometimes --lexical-order.

Sorry, I've forgotten how I named it :-D.

Shouldn't we put --lexical-order directly into lit's test suite's %{lit} substitution? That way, if another of lit's current or future tests is susceptible to this problem, no one has to waste time trying to reproduce the race and figure out what's happening. Blindly using %{lit} in lit's test suite would just work.

I don't mind that. However, should we also put -j 1 then?

As far as I know, reorder.py is the only test in lit's test suite that requires the reordering. Either it can use a %{lit-reordered} substitution that drops --lexical-order, or it can use a new command-line option that selects the default order and overrides any previous --lexical-order or --shuffle.

I suppose I could do that, and then replace the mutually exclusive group approach with 'last option wins'.

[mgorny]

llvm/utils/lit/tests/shtest-shell.py

10–11

I was grepping for order FIXMEs ;-). Fixed now.

[mgorny]

Add --default-order. Last option specified takes precedence. Include --lexical-order in %{lit} substitution.

[jdenny]

Thanks for continuing to work on all this.

In D107695#2932775, @mgorny wrote:

Shouldn't we put --lexical-order directly into lit's test suite's %{lit} substitution? That way, if another of lit's current or future tests is susceptible to this problem, no one has to waste time trying to reproduce the race and figure out what's happening. Blindly using %{lit} in lit's test suite would just work.

I don't mind that. However, should we also put -j 1 then?

Good point. I haven't looked to see whether the default value of -j is ever expected in lit's test suite. Anyway, maybe that should be a different patch to make reviewing/debugging easier.

In D107695#2932760, @jdenny wrote:

Also, the docs should be updated: https://llvm.org/docs/CommandGuide/lit.html

Please don't forget this.

llvm/utils/lit/lit/cl_arguments.py

148

Instead of --default-order, I'd prefer something that describes the order. I'm not sure of the best name, especially given that it's always possible we'll think of other tweaks to the default order in the future. Perhaps something like --auto-order or --smart-order to at least suggest it's something clever. What do you think?

llvm/utils/lit/tests/lit.cfg

65–67

I think it would be helpful to extend the above comments to explain why --lexical-order is needed.

llvm/utils/lit/tests/reorder.py

4

As far as I can tell, we now have no test that --default-order is actually the default. Sorry, I didn't think of that when I made that suggestion. If we ever broke that behavior (for example, someone could reorganize the arg specification in a subtly broken way), I wonder how long it would be before anyone realized that test suites started taking much longer than necessary.

What if we have something like a %{lit-no-order-opt} that doesn't specify any order options? reorder.py could then run lit twice, once to ensure --default-order is indeed the default, and once to ensure it overrides --lexical-order and --shuffle when specified afterward.

I guess that means --default-order isn't so useful in lit's own test suite. It's just another feature to test. However, as I mentioned in D107427, I'd like to use it in clang/test/utils too for testing test-suite-generation scripts.

[jdenny]

llvm/utils/lit/tests/reorder.py

4

I guess that means --default-order isn't so useful in lit's own test suite. It's just another feature to test. However, as I mentioned in D107427, I'd like to use it in clang/test/utils too for testing test-suite-generation scripts.

Well, that actually makes no sense. I want --lexical-order there not --default-order.

Hmm. I'm not opposed to keeping something like --default-order for completeness, but I'm now having trouble seeing a real use case. What do others think?

[mgorny]

llvm/utils/lit/tests/reorder.py

4

To be honest, I don't really see a use case for testing which order is the default. After all, setting the default order is a matter of changing the parameter to default=… A test for this sounds like asking people to change the default in two places.

[jdenny]

llvm/utils/lit/tests/reorder.py

4

I don't agree with the argument that a behavior whose current implementation looks simple doesn't need a regression test, and I've previously found that the apparent simplicity of an argparse config can be deceptive. If we're going to implement these options, then I think we should test them fully, especially that the intended default behavior is actually the default. If the default ever changes accidentally, it could quietly affect many people. Moreover, testing it is straight-forward: we just need another lit substitution to use in reorder.py.

After further thought, I do think having a different option for each possible order is likely to prove useful, so I vote we keep --default-order (or whatever we end up calling it).

[yln]

Thanks for working on this! :)

llvm/utils/lit/lit/cl_arguments.py

11

While we are at it, can we also give DEFAULT a more descriptive name here?

Order predicate: (not t.previous_failure, -t.previous_elapsed, t.getFullName())

1. Failures first
2. Longest tests first
3. Then by name

Since putting all of the above in an enum name is quite a mouthful, I am advocating for smart as suggested by Joel.

Together with my other comment (see below) the help text could then be:
--order=(smart|aaa|bbb|...) [default: smart]

This would align the names used for options and names in the code.

148

I would also prefer descriptive names instead of "default".

If we already put the effort i to improve ordering, how about: --order=(aaa|bbb|...) and make the old ones aliases for the new ones: --shuffle --> --order=random.

I think argparse spells out all possible options and even shows which one is the default when printing the help: --order=(smart|aaa|bbb|...) [default: smart]

So it shows the user the available options, which option is the default, and the explanation of these options is right next to it.
Having all options documented right next to each other also improves discoverability and makes contrasting the differences between options easier.

[yln]

llvm/utils/lit/lit/cl_arguments.py

148

If it wasn't clear, I am advocating for using parser.add_argument('--order', choices=['smart', aaa', 'bbb']) here.
https://docs.python.org/3/library/argparse.html#choices

[mgorny]

Switched to choice-based --order= option. Renamed 'default' to 'smart'. Added a new lit subst not to include order args. Made %{lit} include -j1 as well. Added the comment.

I think this addresses all the comments.

I have also removed the enum since it seemed to make little sense now... but I have a new idea, so I might reintroduce it in a minute ;-).

[mgorny]

Restore the TestOrder enum. It doesn't change much logically but gives some abstraction that'll hopefully make uncaught typos less likely in the future.

[jdenny]

--order is nice.

Also, the docs should be updated: https://llvm.org/docs/CommandGuide/lit.html

This is still needed.

[mgorny]

Updated docs.

[mgorny]

Ping.

[jdenny]

LGTM. Thanks.

[MatzeB]

Hey, I don't know if this has anything to do with this change.

But I see the "llvm-clang-x86_64-sie-win" buildbot (https://lab.llvm.org/buildbot/#/builders/216) failing the reorder.py testcase randomly every 10-20 builds.

See https://lab.llvm.org/buildbot/#/builders/216 for the list of recent builds.

I didn't look deeply into the root cause here, but thought it would be good to ping here in case anyone has an idea to get the bot more stable.

[dyung]

In D107695#3617159, @MatzeB wrote:

Hey, I don't know if this has anything to do with this change.

But I see the "llvm-clang-x86_64-sie-win" buildbot (https://lab.llvm.org/buildbot/#/builders/216) failing the reorder.py testcase randomly every 10-20 builds.

See https://lab.llvm.org/buildbot/#/builders/216 for the list of recent builds.

I didn't look deeply into the root cause here, but thought it would be good to ping here in case anyone has an idea to get the bot more stable.

I'm the owner of that buildbot, but unfortunately have not been able to get the failure to reproduce outside of the buildbot initiated build environment, so I haven't been able to really look into the issue. Any ideas how to possibly debug and/or fix the issue would be greatly appreciated as I've been a bit stuck on how to get it more stable!

[ychen]

In D107695#3620822, @dyung wrote:

In D107695#3617159, @MatzeB wrote:

Hey, I don't know if this has anything to do with this change.

But I see the "llvm-clang-x86_64-sie-win" buildbot (https://lab.llvm.org/buildbot/#/builders/216) failing the reorder.py testcase randomly every 10-20 builds.

See https://lab.llvm.org/buildbot/#/builders/216 for the list of recent builds.

I didn't look deeply into the root cause here, but thought it would be good to ping here in case anyone has an idea to get the bot more stable.

I'm the owner of that buildbot, but unfortunately have not been able to get the failure to reproduce outside of the buildbot initiated build environment, so I haven't been able to really look into the issue. Any ideas how to possibly debug and/or fix the issue would be greatly appreciated as I've been a bit stuck on how to get it more stable!

It's kind of concerning that the test (reorder.py) writes to the source-controlled input file. I'm not sure if using a temporary directory to save intermediate files (like most other tests do) would fix the issue.
Alternatively, as the last resort, I'd suggest temporarily inserting some extra RUN lines in the test to help debug the random failure.

llvm/utils/lit/lit/cl_arguments.py

 import argparse
 import enum
 import os
 import shlex
 import sys
 
 import lit.reports
 import lit.util
 
 
-class TestOrder(enum.Enum):
-    DEFAULT = enum.auto()
-    RANDOM = enum.auto()
-
-
 def parse_args():
     parser = argparse.ArgumentParser(prog='lit')
     parser.add_argument('test_paths',
             nargs='+',
             metavar="TEST_PATH",
             help='File or path to include in the test suite')
 
     parser.add_argument('--version',
             action='version',
selection_group.add_argument("--max-tests",
             metavar="N",
             help="Maximum number of tests to run",
             type=_positive_int)
     selection_group.add_argument("--max-time",
             dest="timeout",
             metavar="N",
             help="Maximum time to spend testing (in seconds)",
             type=_positive_int)
+    selection_group.add_argument("--order",
+            choices=["lexical", "random", "smart"],
+            default="smart",
+            help="Test order to use (default: smart)")
     selection_group.add_argument("--shuffle",
-            help="Run tests in random order",
-            action="store_true")
+            dest="order",
+            help="Run tests in random order (DEPRECATED: use --order=random)",
+            action="store_const",
+            const="random")
     selection_group.add_argument("-i", "--incremental",
-            help="Run failed tests first (DEPRECATED: now always enabled)",
+            help="Run failed tests first (DEPRECATED: use --order=smart)",
             action="store_true")
     selection_group.add_argument("--filter",
             metavar="REGEX",
             type=_case_insensitive_regex,
             help="Only run tests with paths matching the given regular expression",
             default=os.environ.get("LIT_FILTER", ".*"))
     selection_group.add_argument("--filter-out",
             metavar="REGEX",
def parse_args():
 
     # Validate command line options
     if opts.echoAllCommands:
         opts.showOutput = True
 
     if opts.incremental:
         print('WARNING: --incremental is deprecated. Failing tests now always run first.')
 
-    if opts.shuffle:
-        opts.order = TestOrder.RANDOM
-    else:
-        opts.order = TestOrder.DEFAULT
-
     if opts.numShards or opts.runShard:
         if not opts.numShards or not opts.runShard:
             parser.error("--num-shards and --run-shard must be used together")
         if opts.runShard > opts.numShards:
             parser.error("--run-shard must be between 1 and --num-shards (inclusive)")
         opts.shard = (opts.runShard, opts.numShards)
     else:
         opts.shard = None

llvm/utils/lit/lit/main.py

def print_discovered(tests, show_suites, show_tests):
 
     if show_tests:
         print('-- Available Tests --')
         for t in tests:
             print('  %s' % t.getFullName())
 
 
 def determine_order(tests, order):
-    from lit.cl_arguments import TestOrder
-    if order == TestOrder.RANDOM:
+    if order == "random":
         import random
         random.shuffle(tests)
+    elif order == "lexical":
+        tests.sort(key=lambda t: t.getFullName())
     else:
-        assert order == TestOrder.DEFAULT, 'Unknown TestOrder value'
+        assert order == "smart", 'Unknown TestOrder value'
         tests.sort(key=lambda t: (not t.previous_failure, -t.previous_elapsed, t.getFullName()))
 
 
 def filter_by_shard(tests, run, shards, lit_config):
     test_ixs = range(run - 1, len(tests), shards)
     selected_tests = [tests[i] for i in test_ixs]
 
     # For clarity, generate a preview of the first few test indices in the shard

llvm/utils/lit/tests/allow-retries.py

 # Check the behavior of the ALLOW_RETRIES keyword.
 
 # This test uses a file that's stable across retries of the test to fail and
 # only succeed the fourth time it is retried.
 #
 # RUN: rm -f %t.counter
-# RUN: %{lit} -j 1 %{inputs}/allow-retries/succeeds-within-limit.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST1 %s
+# RUN: %{lit} %{inputs}/allow-retries/succeeds-within-limit.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST1 %s
 # CHECK-TEST1: Passed With Retry: 1
 
 # Test that a per-file ALLOW_RETRIES overwrites the config-wide test_retry_attempts property, if any.
 #
 # RUN: rm -f %t.counter
-# RUN: %{lit} -j 1 %{inputs}/allow-retries/succeeds-within-limit.py -Dtest_retry_attempts=2 -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST2 %s
+# RUN: %{lit} %{inputs}/allow-retries/succeeds-within-limit.py -Dtest_retry_attempts=2 -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST2 %s
 # CHECK-TEST2: Passed With Retry: 1
 
 # This test does not succeed within the allowed retry limit
 #
-# RUN: not %{lit} -j 1 %{inputs}/allow-retries/does-not-succeed-within-limit.py | FileCheck --check-prefix=CHECK-TEST3 %s
+# RUN: not %{lit} %{inputs}/allow-retries/does-not-succeed-within-limit.py | FileCheck --check-prefix=CHECK-TEST3 %s
 # CHECK-TEST3: Failed Tests (1):
 # CHECK-TEST3: allow-retries :: does-not-succeed-within-limit.py
 
 # This test should be UNRESOLVED since it has more than one ALLOW_RETRIES
 # lines, and that is not allowed.
 #
-# RUN: not %{lit} -j 1 %{inputs}/allow-retries/more-than-one-allow-retries-lines.py | FileCheck --check-prefix=CHECK-TEST4 %s
+# RUN: not %{lit} %{inputs}/allow-retries/more-than-one-allow-retries-lines.py | FileCheck --check-prefix=CHECK-TEST4 %s
 # CHECK-TEST4: Unresolved Tests (1):
 # CHECK-TEST4: allow-retries :: more-than-one-allow-retries-lines.py
 
 # This test does not provide a valid integer to the ALLOW_RETRIES keyword.
 # It should be unresolved.
 #
-# RUN: not %{lit} -j 1 %{inputs}/allow-retries/not-a-valid-integer.py | FileCheck --check-prefix=CHECK-TEST5 %s
+# RUN: not %{lit} %{inputs}/allow-retries/not-a-valid-integer.py | FileCheck --check-prefix=CHECK-TEST5 %s
 # CHECK-TEST5: Unresolved Tests (1):
 # CHECK-TEST5: allow-retries :: not-a-valid-integer.py
 
 # This test checks that the config-wide test_retry_attempts property is used
 # when no ALLOW_RETRIES keyword is present.
 #
 # RUN: rm -f %t.counter
-# RUN: %{lit} -j 1 %{inputs}/test_retry_attempts/test.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST6 %s
+# RUN: %{lit} %{inputs}/test_retry_attempts/test.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST6 %s
 # CHECK-TEST6: Passed With Retry: 1

llvm/utils/lit/tests/custom-result-category.py

 # UNSUPPORTED: system-windows
 # Test lit.main.add_result_category() extension API.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/custom-result-category/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 %{inputs}/custom-result-category | FileCheck %s
+# RUN: not %{lit} %{inputs}/custom-result-category | FileCheck %s
 
 # CHECK: CUSTOM_PASS: custom-result-category :: test1.txt
 # CHECK: CUSTOM_FAILURE: custom-result-category :: test2.txt
 
 # CHECK-NOT: My Passed Tests (1)
 # CHECK-NOT:   custom-result-category :: test1.txt
 # CHECK:     My Failed Tests (1)
 # CHECK:       custom-result-category :: test2.txt
 
 # CHECK: My Passed: 1
 # CHECK: My Failed: 1

llvm/utils/lit/tests/discovery.py

 # Check the basic discovery process, including a sub-suite.
 #
 # RUN: %{lit} %{inputs}/discovery \
-# RUN:   -j 1 --debug --show-tests --show-suites \
+# RUN:   --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-BASIC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-BASIC-ERR < %t.err %s
 #
 # CHECK-BASIC-ERR: loading suite config '{{.*(/|\\\\)discovery(/|\\\\)lit.cfg}}'
 # CHECK-BASIC-ERR-DAG: loading suite config '{{.*(/|\\\\)discovery(/|\\\\)subsuite(/|\\\\)lit.cfg}}'
 # CHECK-BASIC-ERR-DAG: loading local config '{{.*(/|\\\\)discovery(/|\\\\)subdir(/|\\\\)lit.local.cfg}}'
 #

 # CHECK-CONFIG-MAP-ERR: resolved input '{{.*(/|\\\\)config-map-discovery(/|\\\\)main-config}}' to 'config-map'::()
 
 
 # Check discovery when tests are named directly.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/discovery/subdir/test-three.py \
 # RUN:     %{inputs}/discovery/subsuite/test-one.txt \
-# RUN:   -j 1 --show-tests --show-suites -v > %t.out
+# RUN:   --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-DIRECT-TEST < %t.out %s
 #
 # CHECK-DIRECT-TEST: -- Available Tests --
 # CHECK-DIRECT-TEST: sub-suite :: test-one
 # CHECK-DIRECT-TEST: top-level-suite :: subdir/test-three
 
 # Check discovery when config files end in .py
 # RUN: %{lit} %{inputs}/py-config-discovery \
-# RUN:   -j 1 --debug --show-tests --show-suites \
+# RUN:   --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-PYCONFIG-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-PYCONFIG-ERR < %t.err %s
 #
 # CHECK-PYCONFIG-ERR: loading suite config '{{.*(/|\\\\)py-config-discovery(/|\\\\)lit.site.cfg.py}}'
 # CHECK-PYCONFIG-ERR: load_config from '{{.*(/|\\\\)discovery(/|\\\\)lit.cfg}}'
 # CHECK-PYCONFIG-ERR: loaded config '{{.*(/|\\\\)discovery(/|\\\\)lit.cfg}}'
 # CHECK-PYCONFIG-ERR: loaded config '{{.*(/|\\\\)py-config-discovery(/|\\\\)lit.site.cfg.py}}'

 # CHECK-PYCONFIG-OUT: sub-suite :: test-two
 # CHECK-PYCONFIG-OUT: top-level-suite :: subdir/test-three
 # CHECK-PYCONFIG-OUT: top-level-suite :: test-one
 # CHECK-PYCONFIG-OUT: top-level-suite :: test-two
 
 # Check discovery when using an exec path.
 #
 # RUN: %{lit} %{inputs}/exec-discovery \
-# RUN:   -j 1 --debug --show-tests --show-suites \
+# RUN:   --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-ERR < %t.err %s
 #
 # CHECK-ASEXEC-ERR: loading suite config '{{.*(/|\\\\)exec-discovery(/|\\\\)lit.site.cfg}}'
 # CHECK-ASEXEC-ERR: load_config from '{{.*(/|\\\\)discovery(/|\\\\)lit.cfg}}'
 # CHECK-ASEXEC-ERR: loaded config '{{.*(/|\\\\)discovery(/|\\\\)lit.cfg}}'
 # CHECK-ASEXEC-ERR: loaded config '{{.*(/|\\\\)exec-discovery(/|\\\\)lit.site.cfg}}'

 # CHECK-ASEXEC-OUT: top-level-suite :: test-two
 
 # Check discovery when tests are named directly.
 #
 # FIXME: Note that using a path into a subsuite doesn't work correctly here.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/exec-discovery/subdir/test-three.py \
-# RUN:   -j 1 --show-tests --show-suites -v > %t.out
+# RUN:   --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-DIRECT-TEST < %t.out %s
 #
 # CHECK-ASEXEC-DIRECT-TEST: -- Available Tests --
 # CHECK-ASEXEC-DIRECT-TEST: top-level-suite :: subdir/test-three
 
 # Check an error is emitted when the directly named test would not be run
 # indirectly (e.g. when the directory containing the test is specified).
 #
 # RUN: not %{lit} \
-# RUN:     %{inputs}/discovery/test.not-txt -j 1 2>%t.err
+# RUN:     %{inputs}/discovery/test.not-txt 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-ERROR-INDIRECT-RUN-CHECK < %t.err %s
 #
 # CHECK-ERROR-INDIRECT-RUN-CHECK: error: 'top-level-suite :: test.not-txt' would not be run indirectly
 
 # Check that no error is emitted with --no-indirectly-run-check.
 #
 # RUN: %{lit} \
-# RUN:     %{inputs}/discovery/test.not-txt -j 1 --no-indirectly-run-check
+# RUN:     %{inputs}/discovery/test.not-txt --no-indirectly-run-check
 
 # Check that a standalone test with no suffixes set is run without any errors.
 #
-# RUN: %{lit} %{inputs}/standalone-tests/true.txt -j 1 > %t.out
+# RUN: %{lit} %{inputs}/standalone-tests/true.txt > %t.out
 # RUN: FileCheck --check-prefix=CHECK-STANDALONE < %t.out %s
 #
 # CHECK-STANDALONE: PASS: Standalone tests :: true.txt
 
 # Check that an error is produced if suffixes variable is set for a suite with
 # standalone tests.
 #
-# RUN: not %{lit} %{inputs}/standalone-tests-with-suffixes -j 1 2> %t.err
+# RUN: not %{lit} %{inputs}/standalone-tests-with-suffixes 2> %t.err
 # RUN: FileCheck --check-prefixes=CHECK-STANDALONE-SUFFIXES,CHECK-STANDALONE-DISCOVERY < %t.err %s
 #
 # CHECK-STANDALONE-SUFFIXES: standalone_tests set {{.*}} but suffixes
 
 # Check that an error is produced if excludes variable is set for a suite with
 # standalone tests.
 #
-# RUN: not %{lit} %{inputs}/standalone-tests-with-excludes -j 1 2> %t.err
+# RUN: not %{lit} %{inputs}/standalone-tests-with-excludes 2> %t.err
 # RUN: FileCheck --check-prefixes=CHECK-STANDALONE-EXCLUDES,CHECK-STANDALONE-DISCOVERY < %t.err %s
 #
 # CHECK-STANDALONE-EXCLUDES: standalone_tests set {{.*}} but {{.*}} excludes
 
 # Check that no discovery is done for testsuite with standalone tests.
 #
-# RUN: not %{lit} %{inputs}/standalone-tests -j 1 2>%t.err
+# RUN: not %{lit} %{inputs}/standalone-tests 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-STANDALONE-DISCOVERY < %t.err %s
 #
 # CHECK-STANDALONE-DISCOVERY: error: did not discover any tests for provided path(s)
 
 # Check that we don't recurse infinitely when loading an site specific test
 # suite located inside the test source root.
 #
 # RUN: %{lit} \
 # RUN:     %{inputs}/exec-discovery-in-tree/obj/ \
-# RUN:   -j 1 --show-tests --show-suites -v > %t.out
+# RUN:   --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-INTREE < %t.out %s
 #
 # Try it again after cd'ing into the test suite using a short relative path.
 #
 # RUN: cd %{inputs}/exec-discovery-in-tree/obj/
 # RUN: %{lit} . \
-# RUN:   -j 1 --show-tests --show-suites -v > %t.out
+# RUN:   --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-INTREE < %t.out %s
 #
 #      CHECK-ASEXEC-INTREE:   exec-discovery-in-tree-suite - 1 tests
 # CHECK-ASEXEC-INTREE-NEXT:     Source Root: {{.*[/\\]exec-discovery-in-tree$}}
 # CHECK-ASEXEC-INTREE-NEXT:     Exec Root  : {{.*[/\\]exec-discovery-in-tree[/\\]obj$}}
 # CHECK-ASEXEC-INTREE-NEXT:     Available Features:
 # CHECK-ASEXEC-INTREE-NEXT:     Available Substitutions:
 # CHECK-ASEXEC-INTREE-NEXT: -- Available Tests --
 # CHECK-ASEXEC-INTREE-NEXT: exec-discovery-in-tree-suite :: test-one

llvm/utils/lit/tests/googletest-discovery-failed.py

 # Check for correct error message when discovery of tests fails.
 #
-# RUN: not %{lit} -j 1 -v %{inputs}/googletest-discovery-failed > %t.cmd.out
+# RUN: not %{lit} -v %{inputs}/googletest-discovery-failed > %t.cmd.out
 # RUN: FileCheck < %t.cmd.out %s
 
 
 # CHECK: -- Testing:
 # CHECK: Failed Tests (1):
 # CHECK:   googletest-discovery-failed :: subdir/OneTest.py/failed_to_discover_tests_from_gtest
 # CHECK: Failed: 1

llvm/utils/lit/tests/googletest-format.py

 # Check the various features of the GoogleTest format.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/googletest-format/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -v %{inputs}/googletest-format > %t.out
+# RUN: not %{lit} -v %{inputs}/googletest-format > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:

llvm/utils/lit/tests/googletest-timeout.py

 # REQUIRES: lit-max-individual-test-time
 
 ###############################################################################
 # Check tests can hit timeout when set
 ###############################################################################
 
 # Check that the per test timeout is enforced when running GTest tests.
 #
-# RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
+# RUN: not %{lit} -v %{inputs}/googletest-timeout \
 # RUN:   --filter=InfiniteLoopSubTest --timeout=1 > %t.cmd.out
 # RUN: FileCheck --check-prefix=CHECK-INF < %t.cmd.out %s
 
 # Check that the per test timeout is enforced when running GTest tests via
 # the configuration file
 #
-# RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
+# RUN: not %{lit} -v %{inputs}/googletest-timeout \
 # RUN:  --filter=InfiniteLoopSubTest  --param set_timeout=1 \
 # RUN:  > %t.cfgset.out
 # RUN: FileCheck --check-prefix=CHECK-INF < %t.cfgset.out %s
 
 # CHECK-INF: -- Testing:
 # CHECK-INF: TIMEOUT: googletest-timeout :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/[[TEST:T\.InfiniteLoopSubTest]]
 # CHECK-INF-NEXT: ******************** TEST 'googletest-timeout :: [[PATH]][[FILE]]/[[TEST]]' FAILED ********************
 # CHECK-INF-NEXT: Script:
 # CHECK-INF-NEXT: --
 # CHECK-INF-NEXT: [[FILE]] --gtest_filter=[[TEST]]
 # CHECK-INF-NEXT: --
 # CHECK-INF: Timed Out: 1
 
 ###############################################################################
 # Check tests can complete with a timeout set
 #
 # `QuickSubTest` should execute quickly so we shouldn't wait anywhere near the
 # 3600 second timeout.
 ###############################################################################
 
-# RUN: %{lit} -j 1 -v %{inputs}/googletest-timeout \
+# RUN: %{lit} -v %{inputs}/googletest-timeout \
 # RUN:   --filter=QuickSubTest --timeout=3600 > %t.cmd.out
 # RUN: FileCheck --check-prefix=CHECK-QUICK < %t.cmd.out %s
 
 # CHECK-QUICK: PASS: googletest-timeout :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/T.QuickSubTest
 # CHECK-QUICK: Passed : 1
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
-# RUN: %{lit} -j 1 -v %{inputs}/googletest-timeout --filter=QuickSubTest \
+# RUN: %{lit} -v %{inputs}/googletest-timeout --filter=QuickSubTest \
 # RUN:   --param set_timeout=1 --timeout=3600 \
 # RUN:   > %t.cmdover.out 2> %t.cmdover.err
 # RUN: FileCheck --check-prefix=CHECK-QUICK < %t.cmdover.out %s
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.cmdover.err %s
 
 # CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 3600 seconds

llvm/utils/lit/tests/googletest-upstream-format.py

 # Check the various features of the GoogleTest format.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/googletest-upstream-format/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -v %{inputs}/googletest-upstream-format > %t.out
+# RUN: not %{lit} -v %{inputs}/googletest-upstream-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 # CHECK: PASS: googletest-upstream-format :: [[PATH:[Dd]ummy[Ss]ub[Dd]ir/]][[FILE:OneTest\.py]]/FirstTest.subTestA
 # CHECK: FAIL: googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST:FirstTest\.subTestB]]
 # CHECK-NEXT: *** TEST 'googletest-upstream-format :: [[PATH]][[FILE]]/[[TEST]]' FAILED ***

llvm/utils/lit/tests/ignore-fail.py

 # Check that --ignore-fail produces exit status 0 despite various kinds of
 # test failures but doesn't otherwise suppress those failures.
 
-# RUN: not %{lit} -j 1 %{inputs}/ignore-fail | FileCheck %s
-# RUN: %{lit} -j 1 --ignore-fail %{inputs}/ignore-fail | FileCheck %s
+# RUN: not %{lit} %{inputs}/ignore-fail | FileCheck %s
+# RUN: %{lit} --ignore-fail %{inputs}/ignore-fail | FileCheck %s
 
 # END.
 
 # CHECK-DAG: FAIL: ignore-fail :: fail.txt
 # CHECK-DAG: UNRESOLVED: ignore-fail :: unresolved.txt
 # CHECK-DAG: XFAIL: ignore-fail :: xfail.txt
 # CHECK-DAG: XPASS: ignore-fail :: xpass.txt
 
 #      CHECK: Testing Time:
 # CHECK-NEXT:   Expectedly Failed : 1
 # CHECK-NEXT:   Unresolved : 1
 # CHECK-NEXT:   Failed : 1
 # CHECK-NEXT:   Unexpectedly Passed: 1
 #  CHECK-NOT: {{.}}

llvm/utils/lit/tests/lit-opts.py

 # Check cases where LIT_OPTS has no effect.
 #
-# RUN:                 %{lit} -j 1 -s %{inputs}/lit-opts | FileCheck %s
-# RUN: env LIT_OPTS=   %{lit} -j 1 -s %{inputs}/lit-opts | FileCheck %s
-# RUN: env LIT_OPTS=-s %{lit} -j 1 -s %{inputs}/lit-opts | FileCheck %s
+# RUN:                 %{lit} -s %{inputs}/lit-opts | FileCheck %s
+# RUN: env LIT_OPTS=   %{lit} -s %{inputs}/lit-opts | FileCheck %s
+# RUN: env LIT_OPTS=-s %{lit} -s %{inputs}/lit-opts | FileCheck %s
 
 # Check that LIT_OPTS can override command-line options.
 #
 # RUN: env LIT_OPTS=-a \
-# RUN: %{lit} -j 1 -s %{inputs}/lit-opts \
+# RUN: %{lit} -s %{inputs}/lit-opts \
 # RUN: | FileCheck -check-prefix=SHOW-ALL -DVAR= %s
 
 # Check that LIT_OPTS understands multiple options with arbitrary spacing.
 #
 # RUN: env LIT_OPTS='-a -v  -Dvar=foobar' \
-# RUN: %{lit} -j 1 -s %{inputs}/lit-opts \
+# RUN: %{lit} -s %{inputs}/lit-opts \
 # RUN: | FileCheck -check-prefix=SHOW-ALL -DVAR=foobar %s
 
 # Check that LIT_OPTS parses shell-like quotes and escapes.
 #
 # RUN: env LIT_OPTS='-a   -v -Dvar="foo bar"\ baz' \
-# RUN: %{lit} -j 1 -s %{inputs}/lit-opts \
+# RUN: %{lit} -s %{inputs}/lit-opts \
 # RUN: | FileCheck -check-prefix=SHOW-ALL -DVAR="foo bar baz" %s
 
 # CHECK:      Testing: 1 tests
 # CHECK-NOT:  PASS
 # CHECK:      Passed: 1
 
 # SHOW-ALL:     Testing: 1 tests
 # SHOW-ALL:     PASS: lit-opts :: test.txt (1 of 1)
 # SHOW-ALL:     {{^}}[[VAR]]
 # SHOW-ALL-NOT: PASS
 # SHOW-ALL:     Passed: 1

llvm/utils/lit/tests/lit.cfg

for attribute in ('llvm_tools_dir', 'lit_tools_dir'):
     directory = getattr(config, attribute, None)
     if directory:
         llvm_config.with_environment('PATH', directory, append_path=True)
 
 # This test suite calls %{lit} to test lit's behavior for the sample test
 # suites in %{inputs}.  This test suite's results are then determined in part
 # by %{lit}'s textual output, which includes the output of FileCheck calls
 # within %{inputs}'s test suites.  Thus, %{lit} clears environment variables
-# that can affect FileCheck's output.
+# that can affect FileCheck's output.  It also includes "--order=lexical -j1"
+# to ensure predictable test order, as it is often required for FileCheck
+# matches.
 config.substitutions.append(('%{inputs}', os.path.join(
     config.test_source_root, 'Inputs')))
-config.substitutions.append(('%{lit}',
-    "{env} %{{python}} {lit}".format(
+config.substitutions.append(('%{lit}', '%{lit-no-order-opt} --order=lexical'))
+config.substitutions.append(('%{lit-no-order-opt}',
+    "{env} %{{python}} {lit} -j1".format(
         env="env -u FILECHECK_OPTS",
         lit=os.path.join(lit_path, 'lit.py'))))
 config.substitutions.append(('%{python}', '"%s"' % (sys.executable)))
 
 # Enable coverage.py reporting, assuming the coverage module has been installed
 # and sitecustomize.py in the virtualenv has been modified appropriately.
 if lit_config.params.get('check-coverage', None):
     config.environment['COVERAGE_PROCESS_START'] = os.path.join(

llvm/utils/lit/tests/max-failures.py

 # UNSUPPORTED: system-windows
 
 # Check the behavior of --max-failures option.
 #
-# RUN: not %{lit}                  -j 1 %{inputs}/max-failures >  %t.out 2>&1
-# RUN: not %{lit} --max-failures=1 -j 1 %{inputs}/max-failures >> %t.out 2>&1
-# RUN: not %{lit} --max-failures=2 -j 1 %{inputs}/max-failures >> %t.out 2>&1
-# RUN: not %{lit} --max-failures=0 -j 1 %{inputs}/max-failures 2>> %t.out
+# RUN: not %{lit}                  %{inputs}/max-failures >  %t.out 2>&1
+# RUN: not %{lit} --max-failures=1 %{inputs}/max-failures >> %t.out 2>&1
+# RUN: not %{lit} --max-failures=2 %{inputs}/max-failures >> %t.out 2>&1
+# RUN: not %{lit} --max-failures=0 %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 
 # CHECK-NOT: reached maximum number of test failures
 # CHECK-NOT: Skipped
 # CHECK: Failed: 3
 
 # CHECK: reached maximum number of test failures, skipping remaining tests
 # CHECK: Skipped: 2
 # CHECK: Failed : 1
 
 # CHECK: reached maximum number of test failures, skipping remaining tests
 # CHECK: Skipped: 1
 # CHECK: Failed : 2
 
 # CHECK: error: argument --max-failures: requires positive integer, but found '0'

llvm/utils/lit/tests/progress-bar.py

 # Check the simple progress bar.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/progress-bar/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -s %{inputs}/progress-bar > %t.out
+# RUN: not %{lit} -s %{inputs}/progress-bar > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # CHECK: Testing:
 # CHECK: FAIL: progress-bar :: test-1.txt (1 of 4)
 # CHECK: Testing:  0.. 10.. 20
 # CHECK: FAIL: progress-bar :: test-2.txt (2 of 4)
 # CHECK: Testing:  0.. 10.. 20.. 30.. 40..
 # CHECK: FAIL: progress-bar :: test-3.txt (3 of 4)
 # CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70
 # CHECK: FAIL: progress-bar :: test-4.txt (4 of 4)
 # CHECK: Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..

llvm/utils/lit/tests/reorder.py

 ## Check that we can reorder test runs.
 
 # RUN: cp %{inputs}/reorder/.lit_test_times.txt %{inputs}/reorder/.lit_test_times.txt.orig
-# RUN: %{lit} -j1 %{inputs}/reorder > %t.out
+# RUN: %{lit-no-order-opt} %{inputs}/reorder > %t.out
 # RUN: cp %{inputs}/reorder/.lit_test_times.txt %{inputs}/reorder/.lit_test_times.txt.new
 # RUN: cp %{inputs}/reorder/.lit_test_times.txt.orig %{inputs}/reorder/.lit_test_times.txt
 # RUN: not diff %{inputs}/reorder/.lit_test_times.txt.new %{inputs}/reorder/.lit_test_times.txt.orig
 # RUN: FileCheck --check-prefix=TIMES --implicit-check-not= < %{inputs}/reorder/.lit_test_times.txt.new %s
 # RUN: FileCheck < %t.out %s
 # END.
 
 # TIMES: not-executed.txt

llvm/utils/lit/tests/shtest-env.py

 # Check the env command
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-env/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -a -v %{inputs}/shtest-env \
+# RUN: not %{lit} -a -v %{inputs}/shtest-env \
 # RUN: | FileCheck -match-full-lines %s
 #
 # END.
 
 # Make sure env commands are included in printed commands.
 
 # CHECK: -- Testing: 16 tests{{.*}}
 

llvm/utils/lit/tests/shtest-format-argv0.py

 # Check that we route argv[0] as it was written, instead of the resolved
 # path. This is important for some tools, in particular '[' which at least on OS
 # X only recognizes that it is in '['-mode when its argv[0] is exactly
 # '['. Otherwise it will refuse to accept the trailing closing bracket.
 #
 # This test is not supported on AIX since `[` is only available as a shell builtin
 # and is not installed under PATH by default.
 # UNSUPPORTED: system-aix
 #
-# RUN: %{lit} -j 1 -v %{inputs}/shtest-format-argv0 | FileCheck %s
+# RUN: %{lit} -v %{inputs}/shtest-format-argv0 | FileCheck %s
 
 # CHECK: -- Testing:
 # CHECK: PASS: shtest-format-argv0 :: argv0.txt

llvm/utils/lit/tests/shtest-format.py

 # Check the various features of the ShTest format.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-format/.lit_test_times.txt
-
 # RUN: rm -f %t.xml
-# RUN: not %{lit} -j 1 -v %{inputs}/shtest-format --xunit-xml-output %t.xml > %t.out
+# RUN: not %{lit} -v %{inputs}/shtest-format --xunit-xml-output %t.xml > %t.out
 # RUN: FileCheck < %t.out %s
 # RUN: FileCheck --check-prefix=XUNIT < %t.xml %s
 
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-format :: external_shell/fail.txt

llvm/utils/lit/tests/shtest-inject.py

 # Check that we can inject commands at the beginning of a ShTest.
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-empty.txt --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
+# RUN: %{lit} %{inputs}/shtest-inject/test-empty.txt --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
 #
 # CHECK-TEST1: Script:
 # CHECK-TEST1: --
 # CHECK-TEST1: echo "THIS WAS"
 # CHECK-TEST1: echo "INJECTED"
 # CHECK-TEST1: --
 #
 # CHECK-TEST1: THIS WAS
 # CHECK-TEST1: INJECTED
 #
 # CHECK-TEST1: Passed: 1
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-one.txt --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
+# RUN: %{lit} %{inputs}/shtest-inject/test-one.txt --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
 #
 # CHECK-TEST2: Script:
 # CHECK-TEST2: --
 # CHECK-TEST2: echo "THIS WAS"
 # CHECK-TEST2: echo "INJECTED"
 # CHECK-TEST2: echo "IN THE FILE"
 # CHECK-TEST2: --
 #
 # CHECK-TEST2: THIS WAS
 # CHECK-TEST2: INJECTED
 # CHECK-TEST2: IN THE FILE
 #
 # CHECK-TEST2: Passed: 1
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-many.txt --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
+# RUN: %{lit} %{inputs}/shtest-inject/test-many.txt --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
 #
 # CHECK-TEST3: Script:
 # CHECK-TEST3: --
 # CHECK-TEST3: echo "THIS WAS"
 # CHECK-TEST3: echo "INJECTED"
 # CHECK-TEST3: echo "IN THE FILE"
 # CHECK-TEST3: echo "IF IT WORKS"
 # CHECK-TEST3: echo "AS EXPECTED"

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-keyword-parse-errors/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-keyword-parse-errors > %t.out
+# RUN: not %{lit} -vv %{inputs}/shtest-keyword-parse-errors > %t.out
 # RUN: FileCheck -input-file %t.out %s
 #
 # END.
 
 # CHECK: Testing: 3 tests
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: empty.txt
 # CHECK:       {{^}}Test has no 'RUN:' line{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: multiple-allow-retries.txt
 # CHECK:       {{^}}Test has more than one ALLOW_RETRIES lines{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: unterminated-run.txt
 # CHECK:       {{^}}Test has unterminated 'RUN:' lines (with '\'){{$}}

llvm/utils/lit/tests/shtest-not.py

 # Check the not command
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-not/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -a -v %{inputs}/shtest-not \
+# RUN: not %{lit} -a -v %{inputs}/shtest-not \
 # RUN: | FileCheck -match-full-lines %s
 #
 # END.
 
 # Make sure not and env commands are included in printed commands.
 
 # CHECK: -- Testing: 17 tests{{.*}}
 

llvm/utils/lit/tests/shtest-output-printing.py

 # Check the various features of the ShTest format.
 #
-# RUN: not %{lit} -j 1 -v %{inputs}/shtest-output-printing > %t.out
+# RUN: not %{lit} -v %{inputs}/shtest-output-printing > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-output-printing :: basic.txt
 # CHECK-NEXT: *** TEST 'shtest-output-printing :: basic.txt' FAILED ***

llvm/utils/lit/tests/shtest-recursive-substitution.py

 # Check that the config.recursiveExpansionLimit is picked up and will cause
 # lit substitutions to be expanded recursively.
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-recursive-substitution/substitutes-within-limit --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
+# RUN: %{lit} %{inputs}/shtest-recursive-substitution/substitutes-within-limit --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
 # CHECK-TEST1: PASS: substitutes-within-limit :: test.py
 # CHECK-TEST1: $ "echo" "STOP"
 
-# RUN: not %{lit} -j 1 %{inputs}/shtest-recursive-substitution/does-not-substitute-within-limit --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
+# RUN: not %{lit} %{inputs}/shtest-recursive-substitution/does-not-substitute-within-limit --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
 # CHECK-TEST2: UNRESOLVED: does-not-substitute-within-limit :: test.py
 # CHECK-TEST2: ValueError: Recursive substitution of
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-recursive-substitution/does-not-substitute-no-limit --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
+# RUN: %{lit} %{inputs}/shtest-recursive-substitution/does-not-substitute-no-limit --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
 # CHECK-TEST3: PASS: does-not-substitute-no-limit :: test.py
 # CHECK-TEST3: $ "echo" "%rec4"
 
-# RUN: not %{lit} -j 1 %{inputs}/shtest-recursive-substitution/not-an-integer --show-all 2>&1 | FileCheck --check-prefix=CHECK-TEST4 %s
+# RUN: not %{lit} %{inputs}/shtest-recursive-substitution/not-an-integer --show-all 2>&1 | FileCheck --check-prefix=CHECK-TEST4 %s
 # CHECK-TEST4: recursiveExpansionLimit must be either None or an integer
 
-# RUN: not %{lit} -j 1 %{inputs}/shtest-recursive-substitution/negative-integer --show-all 2>&1 | FileCheck --check-prefix=CHECK-TEST5 %s
+# RUN: not %{lit} %{inputs}/shtest-recursive-substitution/negative-integer --show-all 2>&1 | FileCheck --check-prefix=CHECK-TEST5 %s
 # CHECK-TEST5: recursiveExpansionLimit must be a non-negative integer
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-recursive-substitution/set-to-none --show-all | FileCheck --check-prefix=CHECK-TEST6 %s
+# RUN: %{lit} %{inputs}/shtest-recursive-substitution/set-to-none --show-all | FileCheck --check-prefix=CHECK-TEST6 %s
 # CHECK-TEST6: PASS: set-to-none :: test.py
 
-# RUN: %{lit} -j 1 %{inputs}/shtest-recursive-substitution/escaping --show-all | FileCheck --check-prefix=CHECK-TEST7 %s
+# RUN: %{lit} %{inputs}/shtest-recursive-substitution/escaping --show-all | FileCheck --check-prefix=CHECK-TEST7 %s
 # CHECK-TEST7: PASS: escaping :: test.py
 # CHECK-TEST7: $ "echo" "%s" "%s" "%%s"

llvm/utils/lit/tests/shtest-run-at-line.py

 # Check that -vv makes the line number of the failing RUN command clear.
 # (-v is actually sufficient in the case of the internal shell.)
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-run-at-line/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -vv %{inputs}/shtest-run-at-line > %t.out
+# RUN: not %{lit} -vv %{inputs}/shtest-run-at-line > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 
 # CHECK: Testing: 4 tests
 
 

llvm/utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 
-# FIXME: this test depends on order of tests
-# RUN: rm -f %{inputs}/shtest-shell/.lit_test_times.txt
-
-# RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
+# RUN: not %{lit} -v %{inputs}/shtest-shell > %t.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # Test again in non-UTF shell to catch potential errors with python 2 seen
 # on stdout-encoding.txt
-# FIXME: lit's testing sets source_root == exec_root which complicates running lit more than once per test.
-# RUN: rm -f %{inputs}/shtest-shell/.lit_test_times.txt
-# RUN: env PYTHONIOENCODING=ascii not %{lit} -j 1 -a %{inputs}/shtest-shell > %t.ascii.out
+# RUN: env PYTHONIOENCODING=ascii not %{lit} -a %{inputs}/shtest-shell > %t.ascii.out
 # FIXME: Temporarily dump test output so we can debug failing tests on
 # buildbots.
 # RUN: cat %t.ascii.out
 # RUN: FileCheck --input-file %t.ascii.out %s
 #
 # END.
 
 # CHECK: -- Testing:

llvm/utils/lit/tests/test-data-micro.py

 # Test features related to formats which support reporting additional test data.
 # and multiple test results.
 
-# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro | FileCheck %s
+# RUN: %{lit} -v %{inputs}/test-data-micro | FileCheck %s
 
 # CHECK: -- Testing:
 
 # CHECK: PASS: test-data-micro :: micro-tests.ini
 # CHECK-NEXT: *** TEST 'test-data-micro :: micro-tests.ini' RESULTS ***
 # CHECK-NEXT: value0: 1
 # CHECK-NEXT: value1: 2.3456
 # CHECK-NEXT: ***

llvm/utils/lit/tests/test-data.py

 # Test features related to formats which support reporting additional test data.
 
-# RUN: %{lit} -j 1 -v %{inputs}/test-data > %t.out
+# RUN: %{lit} -v %{inputs}/test-data > %t.out
 # RUN: FileCheck < %t.out %s
 
 # CHECK: -- Testing:
 
 # CHECK: PASS: test-data :: metrics.ini
 # CHECK-NEXT: *** TEST 'test-data :: metrics.ini' RESULTS ***
 # CHECK-NEXT: value0: 1
 # CHECK-NEXT: value1: 2.3456
 # CHECK-NEXT: ***

llvm/utils/lit/tests/test-output-micro.py

-# RUN: %{lit} -j 1 -v %{inputs}/test-data-micro --output %t.results.out
+# RUN: %{lit} -v %{inputs}/test-data-micro --output %t.results.out
 # RUN: FileCheck < %t.results.out %s
 # RUN: rm %t.results.out
 
 
 # CHECK: {
 # CHECK: "__version__"
 # CHECK: "elapsed"
 # CHECK-NEXT: "tests": [

llvm/utils/lit/tests/test-output.py

-# RUN: %{lit} -j 1 -v %{inputs}/test-data --output %t.results.out > %t.out
+# RUN: %{lit} -v %{inputs}/test-data --output %t.results.out > %t.out
 # RUN: FileCheck < %t.results.out %s
 
 # CHECK: {
 # CHECK: "__version__"
 # CHECK: "elapsed"
 # CHECK-NEXT: "tests": [
 # CHECK-NEXT:   {
 # CHECK-NEXT:     "code": "PASS",