[lit] Improve naming of test result categories

[yln]

Summary

Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines. For
example, "Failing Tests"/"Unexpected Failures". This commit changes lit
to label things consistently.

Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous. Some labels contain the word "Tests" and some don't.
Let's simplify the names.

Before:

Failing Tests (1):
  ...

Expected Passes    : 3
Unexpected Failures: 1

After:

Failed Tests (1):
  ...

Passed: 3
Failed: 1

[yln]

Happy to bikeshed the names here.

[ldionne]

I like this.

Did you check whether other parts of LLVM had tests relying on this output? Otherwise, LGTM.

[rnk]

I kind of like "Passes", "Failures", "Expected Failures" better than Pass-*ing* Fail-*ing* etc. But I do think we can drop "Expected" from "Expected Passes".

To be consistent, you could change the text "Failing Tests (1):" to "Test Failures (1):"

[jdenny]

In D77708#1969638, @rnk wrote:

I kind of like "Passes", "Failures", "Expected Failures" better than Pass-*ing* Fail-*ing* etc. But I do think we can drop "Expected" from "Expected Passes".

I like "Expected Passes" to clarify it does not include unexpected passes. This would mostly be helpful for developers used to test harnesses without an XFAIL concept.

To be consistent, you could change the text "Failing Tests (1):" to "Test Failures (1):"

I do like this kind of improved consistency.

By the way, are there any bots or other runners that grep for the current keywords when presenting results?

[yln]

I am happy to hear that other people also value improving consistency! :)

In D77708#1969638, @rnk wrote:

I kind of like "Passes", "Failures", "Expected Failures" better than Pass-*ing* Fail-*ing* etc. But I do think we can drop "Expected" from "Expected Passes".

To be consistent, you could change the text "Failing Tests (1):" to "Test Failures (1):"

I feel the same about "Passes"/"Failures" vs. "Passing"/"Failing". I can't even give a good reason other than that to me it sounds better. Maybe because I am just used to it by now.
The issue with the old naming is that some labels are nouns and some are adjectives. If everything follows the same scheme we can print the label in the summary and use "<label> Tests" as the group headings. How about "Passed"/"Failed"?

In D77708#1969689, @jdenny wrote:

In D77708#1969638, @rnk wrote:

I kind of like "Passes", "Failures", "Expected Failures" better than Pass-*ing* Fail-*ing* etc. But I do think we can drop "Expected" from "Expected Passes".

I like "Expected Passes" to clarify it does not include unexpected passes. This would mostly be helpful for developers used to test harnesses without an XFAIL concept.

I think for new developers and people who can be blissfully ignorant of lit's XFAIL/XPASS the following is better

Passing: 3
Failing: 1

than this:

Expected Passes    : 3
Unexpected Failures: 1

For devs who need to care the "normal" is short and the "odd" is long and ugly to draw attention:

Expectedly Failing:   1
Unexpectedly Passing: 1

In D77708#1969436, @ldionne wrote:

Did you check whether other parts of LLVM had tests relying on this output? Otherwise, LGTM.

I will run ninja check-all and grep for the old names before landing this.

[jdenny]

In D77708#1970510, @yln wrote:

For devs who need to care the "normal" is short and the "odd" is long and ugly to draw attention:

Expectedly Failing:   1
Unexpectedly Passing: 1

I'm not sure what you're saying here. Could you explain a bit more?

[jdenny]

In D77708#1970685, @jdenny wrote:

In D77708#1970510, @yln wrote:

For devs who need to care the "normal" is short and the "odd" is long and ugly to draw attention:

Expectedly Failing:   1
Unexpectedly Passing: 1

I'm not sure what you're saying here. Could you explain a bit more?

Ah, nevermind. After a second look, I got your meaning.

People can certainly learn that, by Passes, lit doesn't really mean all passes. I'd prefer less ambiguous terminology, but I'm in the minority so far, and it's not a big deal.

[ldionne]

Are we moving forward with some version of this patch?

[yln]

Rebase onto master. Rename Passing -> Passed and Failing -> Failed.

[yln]

In D77708#2038628, @ldionne wrote:

Are we moving forward with some version of this patch?

Yes, I would still like to if we can settle on a naming scheme. I think that most of the preference here comes from familiarity/what sounds best, which of course is a bit subjective.

I renamed "Passing -> Passed" and "Failing -> Failed". All category headers (the header printed before the list of tests in that group) are now adjective + "Test". For summary counts we use just the adjective describing the category. Currently, @rnk still prefers to old names. @rnk: is the updated scheme acceptable?

ninja check-all shows no failing tests due to this change for me.

After we settle on a final name, I will do a search for the changed strings in the entire repository.

[yln]

I have noticed that Reid's profile says that he will be away "between June 2020 and October 2020", so I don't think we want to wait on him. I will give people another week to chime in on the current naming scheme and then land this.

[yln]

Landing now; standing by the next few hours for any fallout.

clang/www/hacking.html

</pre>
   progress indications while the tests are running.</p>
 
   <p>Your output might look something like this:</p>
 
   <pre>lit.py: lit.cfg:152: note: using clang: 'C:\Tools\llvm\bin\Release\clang.EXE'
 -- Testing: Testing: 2534 tests, 4 threads --
 Testing: 0 .. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
 Testing Time: 81.52s
-  Expected Passes    : 2503
-  Expected Failures  : 28
-  Unsupported Tests  : 3
+  Passed           : 2503
+  Expectedly Failed:   28
+  Unsupported      :    3
 </pre>
 
-  <p>The statistic, "Unexpected Failures" (not shown if all tests pass), is the important one.</p>
+  <p>The statistic, "Failed" (not shown if all tests pass), is the important one.</p>
 
   <!--=====================================================================-->
   <h2 id="patches">Creating Patch Files</h2>
   <!--=====================================================================-->
 
   <p>To contribute changes to Clang see
     <a href="https://llvm.org/docs/GettingStarted.html#sending-patches">LLVM's Getting Started page</a></p>
 

llvm/utils/lit/lit/main.py

def print_histogram(tests):
     test_times = [(t.getFullName(), t.result.elapsed)
                   for t in tests if t.result.elapsed]
     if test_times:
         lit.util.printHistogram(test_times, title='Tests')
 
 
 def add_result_category(result_code, label):
     assert isinstance(result_code, lit.Test.ResultCode)
-    category = (result_code, "%s Tests" % label, label)
+    category = (result_code, label)
     result_codes.append(category)
 
 
-# Status code, summary label, group label
 result_codes = [
     # Passes
-    (lit.Test.EXCLUDED,    'Excluded Tests',      'Excluded'),
-    (lit.Test.SKIPPED,     'Skipped Tests',       'Skipped'),
-    (lit.Test.UNSUPPORTED, 'Unsupported Tests',   'Unsupported'),
-    (lit.Test.PASS,        'Expected Passes',     ''),
-    (lit.Test.FLAKYPASS,   'Passes With Retry',   ''),
-    (lit.Test.XFAIL,       'Expected Failures',   'Expected Failing'),
+    (lit.Test.EXCLUDED,    'Excluded'),
+    (lit.Test.SKIPPED,     'Skipped'),
+    (lit.Test.UNSUPPORTED, 'Unsupported'),
+    (lit.Test.PASS,        'Passed'),
+    (lit.Test.FLAKYPASS,   'Passed With Retry'),
+    (lit.Test.XFAIL,       'Expectedly Failed'),
     # Failures
-    (lit.Test.UNRESOLVED,  'Unresolved Tests',    'Unresolved'),
-    (lit.Test.TIMEOUT,     'Individual Timeouts', 'Timed Out'),
-    (lit.Test.FAIL,        'Unexpected Failures', 'Failing'),
-    (lit.Test.XPASS,       'Unexpected Passes',   'Unexpected Passing')
+    (lit.Test.UNRESOLVED,  'Unresolved'),
+    (lit.Test.TIMEOUT,     'Timed Out'),
+    (lit.Test.FAIL,        'Failed'),
+    (lit.Test.XPASS,       'Unexpectedly Passed')
 ]
 
 
 def print_results(tests, elapsed, opts):
-    tests_by_code = {code: [] for (code, _, _) in result_codes}
+    tests_by_code = {code: [] for code, _ in result_codes}
     for test in tests:
         tests_by_code[test.result.code].append(test)
 
-    for (code, _, group_label) in result_codes:
-        print_group(code, group_label, tests_by_code[code], opts)
+    for (code, label) in result_codes:
+        print_group(code, label, tests_by_code[code], opts)
 
     print_summary(tests_by_code, opts.quiet, elapsed)
 
 
 def print_group(code, label, tests, opts):
     if not tests:
         return
     # TODO(yln): FLAKYPASS? Make this more consistent!
def print_group(code, label, tests, opts):
     sys.stdout.write('\n')
 
 
 def print_summary(tests_by_code, quiet, elapsed):
     if not quiet:
         print('\nTesting Time: %.2fs' % elapsed)
 
     codes = [c for c in result_codes if not quiet or c.isFailure]
-    groups = [(label, len(tests_by_code[code])) for code, label, _ in codes]
+    groups = [(label, len(tests_by_code[code])) for code, label in codes]
     groups = [(label, count) for label, count in groups if count]
     if not groups:
         return
 
     max_label_len = max(len(label) for label, _ in groups)
     max_count_len = max(len(str(count)) for _, count in groups)
 
     for (label, count) in groups:
         label = label.ljust(max_label_len)
         count = str(count).rjust(max_count_len)
         print('  %s: %s' % (label, count))

llvm/utils/lit/tests/allow-retries.py

 # Check the behavior of the ALLOW_RETRIES keyword.
 
 # This test uses a file that's stable across retries of the test to fail and
 # only succeed the fourth time it is retried.
 #
 # RUN: rm -f %t.counter
 # RUN: %{lit} -j 1 %{inputs}/allow-retries/succeeds-within-limit.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST1 %s
-# CHECK-TEST1: Passes With Retry: 1
+# CHECK-TEST1: Passed With Retry: 1
 
 # Test that a per-file ALLOW_RETRIES overwrites the config-wide test_retry_attempts property, if any.
 #
 # RUN: rm -f %t.counter
 # RUN: %{lit} -j 1 %{inputs}/allow-retries/succeeds-within-limit.py -Dtest_retry_attempts=2 -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST2 %s
-# CHECK-TEST2: Passes With Retry: 1
+# CHECK-TEST2: Passed With Retry: 1
 
 # This test does not succeed within the allowed retry limit
 #
 # RUN: not %{lit} -j 1 %{inputs}/allow-retries/does-not-succeed-within-limit.py | FileCheck --check-prefix=CHECK-TEST3 %s
-# CHECK-TEST3: Failing Tests (1):
+# CHECK-TEST3: Failed Tests (1):
 # CHECK-TEST3: allow-retries :: does-not-succeed-within-limit.py
 
 # This test should be UNRESOLVED since it has more than one ALLOW_RETRIES
 # lines, and that is not allowed.
 #
 # RUN: not %{lit} -j 1 %{inputs}/allow-retries/more-than-one-allow-retries-lines.py | FileCheck --check-prefix=CHECK-TEST4 %s
 # CHECK-TEST4: Unresolved Tests (1):
 # CHECK-TEST4: allow-retries :: more-than-one-allow-retries-lines.py
 
 # This test does not provide a valid integer to the ALLOW_RETRIES keyword.
 # It should be unresolved.
 #
 # RUN: not %{lit} -j 1 %{inputs}/allow-retries/not-a-valid-integer.py | FileCheck --check-prefix=CHECK-TEST5 %s
 # CHECK-TEST5: Unresolved Tests (1):
 # CHECK-TEST5: allow-retries :: not-a-valid-integer.py
 
 # This test checks that the config-wide test_retry_attempts property is used
 # when no ALLOW_RETRIES keyword is present.
 #
 # RUN: rm -f %t.counter
 # RUN: %{lit} -j 1 %{inputs}/test_retry_attempts/test.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST6 %s
-# CHECK-TEST6: Passes With Retry: 1
+# CHECK-TEST6: Passed With Retry: 1

llvm/utils/lit/tests/custom-result-category.py

 # UNSUPPORTED: system-windows
 # Test lit.main.add_result_category() extension API.
 
 # RUN: not %{lit} -j 1 %{inputs}/custom-result-category | FileCheck %s
 
 # CHECK: CUSTOM_PASS: custom-result-category :: test1.txt
 # CHECK: CUSTOM_FAILURE: custom-result-category :: test2.txt
 
-# TODO(yln): Passing tests shouldn't be printed by default.
+# TODO(yln): Passed tests shouldn't be printed by default.
 # CHECK: My Passed Tests (1)
 # CHECK: My Failed Tests (1)
 # CHECK:   custom-result-category :: test2.txt
 
-# CHECK: My Passed Tests: 1
-# CHECK: My Failed Tests: 1
+# CHECK: My Passed: 1
+# CHECK: My Failed: 1

llvm/utils/lit/tests/googletest-discovery-failed.py

 # Check for correct error message when discovery of tests fails.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-discovery-failed > %t.cmd.out
 # RUN: FileCheck < %t.cmd.out %s
 
 
 # CHECK: -- Testing:
-# CHECK: Failing Tests (1):
+# CHECK: Failed Tests (1):
 # CHECK:   googletest-discovery-failed :: subdir/OneTest.py/failed_to_discover_tests_from_gtest
-# CHECK: Unexpected Failures: 1
+# CHECK: Failed: 1

llvm/utils/lit/tests/googletest-format.py

 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestA
 # CHECK: FAIL: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
 # CHECK-NEXT: *** TEST 'googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB' FAILED ***
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/0.subTest
 # CHECK: PASS: googletest-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/1.subTest
-# CHECK: Failing Tests (1)
-# CHECK: Expected Passes    : 3
-# CHECK: Unexpected Failures: 1
+# CHECK: Failed Tests (1)
+# CHECK: Passed: 3
+# CHECK: Failed: 1
 

llvm/utils/lit/tests/googletest-timeout.py

 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
 # RUN: --param set_timeout=1 > %t.cfgset.out 2> %t.cfgset.err
 # RUN: FileCheck < %t.cfgset.out %s
 
 # CHECK: -- Testing:
 # CHECK: PASS: googletest-timeout :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestA
 # CHECK: TIMEOUT: googletest-timeout :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
 # CHECK: TIMEOUT: googletest-timeout :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestC
-# CHECK: Expected Passes    : 1
-# CHECK: Individual Timeouts: 2
+# CHECK: Passed   : 1
+# CHECK: Timed Out: 2
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout \
 # RUN: --param set_timeout=1 --timeout=2 > %t.cmdover.out 2> %t.cmdover.err
 # RUN: FileCheck < %t.cmdover.out %s
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.cmdover.err %s
 
 # CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 2 seconds

llvm/utils/lit/tests/googletest-upstream-format.py

 # CHECK: FAIL: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB
 # CHECK-NEXT: *** TEST 'googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/FirstTest.subTestB' FAILED ***
 # CHECK-NEXT: Running main() from gtest_main.cc
 # CHECK-NEXT: I am subTest B, I FAIL
 # CHECK-NEXT: And I have two lines of output
 # CHECK: ***
 # CHECK: PASS: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/0.subTest
 # CHECK: PASS: googletest-upstream-format :: {{[Dd]ummy[Ss]ub[Dd]ir}}/OneTest.py/ParameterizedTest/1.subTest
-# CHECK: Failing Tests (1)
-# CHECK: Expected Passes    : 3
-# CHECK: Unexpected Failures: 1
+# CHECK: Failed Tests (1)
+# CHECK: Passed: 3
+# CHECK: Failed: 1

llvm/utils/lit/tests/lit-opts.py

 # Check that LIT_OPTS parses shell-like quotes and escapes.
 #
 # RUN: env LIT_OPTS='-a   -v -Dvar="foo bar"\ baz' \
 # RUN: %{lit} -j 1 -s %{inputs}/lit-opts \
 # RUN: | FileCheck -check-prefix=SHOW-ALL -DVAR="foo bar baz" %s
 
 # CHECK:      Testing: 1 tests
 # CHECK-NOT:  PASS
-# CHECK:      Expected Passes: 1
+# CHECK:      Passed: 1
 
 # SHOW-ALL:     Testing: 1 tests
 # SHOW-ALL:     PASS: lit-opts :: test.txt (1 of 1)
 # SHOW-ALL:     {{^}}[[VAR]]
 # SHOW-ALL-NOT: PASS
-# SHOW-ALL:     Expected Passes: 1
+# SHOW-ALL:     Passed: 1

llvm/utils/lit/tests/max-failures.py

 # UNSUPPORTED: system-windows
 
 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit}                  -j 1 %{inputs}/max-failures >  %t.out 2>&1
 # RUN: not %{lit} --max-failures=1 -j 1 %{inputs}/max-failures >> %t.out 2>&1
 # RUN: not %{lit} --max-failures=2 -j 1 %{inputs}/max-failures >> %t.out 2>&1
 # RUN: not %{lit} --max-failures=0 -j 1 %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 
 # CHECK-NOT: reached maximum number of test failures
-# CHECK-NOT: Skipped Tests
-# CHECK: Unexpected Failures: 3
+# CHECK-NOT: Skipped
+# CHECK: Failed: 3
 
 # CHECK: reached maximum number of test failures, skipping remaining tests
-# CHECK: Skipped Tests      : 2
-# CHECK: Unexpected Failures: 1
+# CHECK: Skipped: 2
+# CHECK: Failed : 1
 
 # CHECK: reached maximum number of test failures, skipping remaining tests
-# CHECK: Skipped Tests      : 1
-# CHECK: Unexpected Failures: 2
+# CHECK: Skipped: 1
+# CHECK: Failed : 2
 
 # CHECK: error: argument --max-failures: requires positive integer, but found '0'

llvm/utils/lit/tests/max-time.py

 # UNSUPPORTED: system-windows
 
 # Test overall lit timeout (--max-time).
 #
 # RUN: %{lit} %{inputs}/max-time --max-time=5 2>&1  |  FileCheck %s
 
 # CHECK: reached timeout, skipping remaining tests
-# CHECK: Skipped Tests  : 1
-# CHECK: Expected Passes: 1
+# CHECK: Skipped: 1
+# CHECK: Passed : 1

llvm/utils/lit/tests/parallelism-groups.py

 #   config.parallelism_group = 'my_group'
 #
 
 # RUN: %{lit} -j2 %{inputs}/parallelism-groups | FileCheck %s
 
 # CHECK:     -- Testing: 2 tests, 2 workers --
 # CHECK-DAG: PASS: parallelism-groups :: test1.txt
 # CHECK-DAG: PASS: parallelism-groups :: test2.txt
-# CHECK:     Expected Passes: 2
+# CHECK:     Passed: 2

llvm/utils/lit/tests/selecting.py

 # CHECK-BAD-FILTER-ALLOW: Suppressing error because '--allow-empty-runs' was specified.
 
 # Check that regex-filtering works, is case-insensitive, and can be configured via env var.
 #
 # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # RUN: %{lit} --filter 'O[A-Z]E' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # RUN: env LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # CHECK-FILTER: Testing: 2 of 5 tests
-# CHECK-FILTER: Excluded Tests : 3
+# CHECK-FILTER: Excluded: 3
 
 
 # Check that maximum counts work
 #
 # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck --check-prefix=CHECK-MAX %s
 # CHECK-MAX: Testing: 3 of 5 tests
-# CHECK-MAX: Excluded Tests : 2
+# CHECK-MAX: Excluded: 2
 
 
 # Check that sharding partitions the testsuite in a way that distributes the
 # rounding error nicely (i.e. 5/3 => 2 2 1, not 1 1 3 or whatever)
 #
 # RUN: %{lit} --num-shards 3 --run-shard 1 %{inputs}/discovery >%t.out 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-SHARD0-ERR < %t.err %s
 # RUN: FileCheck --check-prefix=CHECK-SHARD0-OUT < %t.out %s
 # CHECK-SHARD0-ERR: note: Selecting shard 1/3 = size 2/5 = tests #(3*k)+1 = [1, 4]
 # CHECK-SHARD0-OUT: Testing: 2 of 5 tests
-# CHECK-SHARD0-OUT: Excluded Tests : 3
+# CHECK-SHARD0-OUT: Excluded: 3
 #
 # RUN: %{lit} --num-shards 3 --run-shard 2 %{inputs}/discovery >%t.out 2>%t.err
 # RUN: FileCheck --check-prefix=CHECK-SHARD1-ERR < %t.err %s
 # RUN: FileCheck --check-prefix=CHECK-SHARD1-OUT < %t.out %s
 # CHECK-SHARD1-ERR: note: Selecting shard 2/3 = size 2/5 = tests #(3*k)+2 = [2, 5]
 # CHECK-SHARD1-OUT: Testing: 2 of 5 tests
 #
 # RUN: %{lit} --num-shards 3 --run-shard 3 %{inputs}/discovery >%t.out 2>%t.err

llvm/utils/lit/tests/shtest-env.py

 # CHECK: $ "env" "A_FOO=1" "B_BAR=2" "C_OOF=3" "{{[^"]*}}" "print_environment.py"
 # CHECK-NOT: ${{.*}}print_environment.py
 
 # CHECK: PASS: shtest-env :: mixed.txt ({{[^)]*}})
 # CHECK: $ "env" "A_FOO=999" "-u" "FOO" "{{[^"]*}}" "print_environment.py"
 # CHECK: $ "env" "A_FOO=1" "-u" "FOO" "B_BAR=2" "-u" "BAR" "C_OOF=3" "{{[^"]*}}" "print_environment.py"
 # CHECK-NOT: ${{.*}}print_environment.py
 
-# CHECK: Expected Passes : 4
-# CHECK: Unexpected Failures: 12
+# CHECK: Passed:  4
+# CHECK: Failed: 12
 # CHECK-NOT: {{.}}

llvm/utils/lit/tests/shtest-format.py

 # CHECK: XFAIL: shtest-format :: xfail.txt
 # CHECK: XPASS: shtest-format :: xpass.txt
 # CHECK-NEXT: *** TEST 'shtest-format :: xpass.txt' FAILED ***
 # CHECK-NEXT: Script
 # CHECK-NEXT: --
 # CHECK-NEXT: true
 # CHECK-NEXT: --
 
-# CHECK: Failing Tests (3)
+# CHECK: Failed Tests (3)
 # CHECK: shtest-format :: external_shell/fail.txt
 # CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK: shtest-format :: fail.txt
 
-# CHECK: Unexpected Passing Tests (1)
+# CHECK: Unexpectedly Passed Tests (1)
 # CHECK: shtest-format :: xpass.txt
 
 # CHECK: Testing Time:
-# CHECK: Unsupported Tests  : 4
-# CHECK: Expected Passes    : 7
-# CHECK: Expected Failures  : 4
-# CHECK: Unresolved Tests   : 3
-# CHECK: Unexpected Failures: 3
-# CHECK: Unexpected Passes  : 1
+# CHECK: Unsupported        : 4
+# CHECK: Passed             : 7
+# CHECK: Expectedly Failed  : 4
+# CHECK: Unresolved         : 3
+# CHECK: Failed             : 3
+# CHECK: Unexpectedly Passed: 1
 
 
 # XUNIT: <?xml version="1.0" encoding="UTF-8"?>
 # XUNIT-NEXT: <testsuites>
 # XUNIT-NEXT: <testsuite name="shtest-format" tests="22" failures="7" skipped="4">
 
 # XUNIT: <testcase classname="shtest-format.shtest-format" name="argv0.txt" time="{{[0-9]+\.[0-9]+}}"/>
 

llvm/utils/lit/tests/shtest-inject.py

 # Check that we can inject commands at the beginning of a ShTest.
 
 # RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-empty.txt --show-all | FileCheck --check-prefix=CHECK-TEST1 %s
 #
 # CHECK-TEST1: Script:
 # CHECK-TEST1: --
 # CHECK-TEST1: echo "THIS WAS"
 # CHECK-TEST1: echo "INJECTED"
 # CHECK-TEST1: --
 #
 # CHECK-TEST1: THIS WAS
 # CHECK-TEST1: INJECTED
 #
-# CHECK-TEST1: Expected Passes: 1
+# CHECK-TEST1: Passed: 1
 
 # RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-one.txt --show-all | FileCheck --check-prefix=CHECK-TEST2 %s
 #
 # CHECK-TEST2: Script:
 # CHECK-TEST2: --
 # CHECK-TEST2: echo "THIS WAS"
 # CHECK-TEST2: echo "INJECTED"
 # CHECK-TEST2: echo "IN THE FILE"
 # CHECK-TEST2: --
 #
 # CHECK-TEST2: THIS WAS
 # CHECK-TEST2: INJECTED
 # CHECK-TEST2: IN THE FILE
 #
-# CHECK-TEST2: Expected Passes: 1
+# CHECK-TEST2: Passed: 1
 
 # RUN: %{lit} -j 1 %{inputs}/shtest-inject/test-many.txt --show-all | FileCheck --check-prefix=CHECK-TEST3 %s
 #
 # CHECK-TEST3: Script:
 # CHECK-TEST3: --
 # CHECK-TEST3: echo "THIS WAS"
 # CHECK-TEST3: echo "INJECTED"
 # CHECK-TEST3: echo "IN THE FILE"
 # CHECK-TEST3: echo "IF IT WORKS"
 # CHECK-TEST3: echo "AS EXPECTED"
 # CHECK-TEST3: --
 #
 # CHECK-TEST3: THIS WAS
 # CHECK-TEST3: INJECTED
 # CHECK-TEST3: IN THE FILE
 # CHECK-TEST3: IF IT WORKS
 # CHECK-TEST3: AS EXPECTED
 #
-# CHECK-TEST3: Expected Passes: 1
+# CHECK-TEST3: Passed: 1

llvm/utils/lit/tests/shtest-not.py

 # CHECK: error: command failed with exit status: {{.*}}
 
 # CHECK: FAIL: shtest-not :: not-calls-rm.txt {{.*}}
 # CHECK: $ "not" "rm" {{.*}}
 # CHECK: $ "not" "--crash" "rm" "foobar"
 # CHECK: Error: 'not --crash' cannot call 'rm'
 # CHECK: error: command failed with exit status: {{.*}}
 
-# CHECK: Expected Passes : 1
-# CHECK: Unexpected Failures: 12
+# CHECK: Passed:  1
+# CHECK: Failed: 12
 # CHECK-NOT: {{.}}

llvm/utils/lit/tests/shtest-shell.py

 # CHECK-NEXT: {{^.f.o.o.$}}
 # CHECK-NEXT: {{^.b.a.r..}}
 # CHECK-NEXT: {{^.b.a.z.$}}
 # CHECK-NOT: error
 # CHECK: $ "false"
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (35)
+# CHECK: Failed Tests (35)

llvm/utils/lit/tests/shtest-timeout.py

 # CHECK-CFGSET-ERR: Using internal shell
 
 # CHECK-OUT-COMMON: TIMEOUT: per_test_timeout :: infinite_loop.py
 # CHECK-OUT-COMMON: Timeout: Reached timeout of 2 seconds
 # CHECK-OUT-COMMON: Command {{([0-9]+ )?}}Output
 
 # CHECK-OUT-COMMON: PASS: per_test_timeout :: short.py
 
-# CHECK-OUT-COMMON: Expected Passes{{ *}}: 1
-# CHECK-OUT-COMMON: Individual Timeouts{{ *}}: 1
+# CHECK-OUT-COMMON: Passed   : 1
+# CHECK-OUT-COMMON: Timed Out: 1
 
 # Test per test timeout via a config file and on the command line.
 # The value set on the command line should override the config file.
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: -j 1 -v --debug --param external=0 \
 # RUN: --param set_timeout=1 --timeout=2 > %t.cmdover.out 2> %t.cmdover.err
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-OUT  < %t.cmdover.out %s
 # RUN: FileCheck --check-prefix=CHECK-CMDLINE-OVERRIDE-ERR < %t.cmdover.err %s
 
 # CHECK-CMDLINE-OVERRIDE-ERR: Forcing timeout to be 2 seconds
 
 # CHECK-CMDLINE-OVERRIDE-OUT: TIMEOUT: per_test_timeout :: infinite_loop.py
 # CHECK-CMDLINE-OVERRIDE-OUT: Timeout: Reached timeout of 2 seconds
 # CHECK-CMDLINE-OVERRIDE-OUT: Command {{([0-9]+ )?}}Output
 
 # CHECK-CMDLINE-OVERRIDE-OUT: PASS: per_test_timeout :: short.py
 
-# CHECK-CMDLINE-OVERRIDE-OUT: Expected Passes{{ *}}: 1
-# CHECK-CMDLINE-OVERRIDE-OUT: Individual Timeouts{{ *}}: 1
+# CHECK-CMDLINE-OVERRIDE-OUT: Passed   : 1
+# CHECK-CMDLINE-OVERRIDE-OUT: Timed Out: 1

mlir/test/Examples/standalone/test.toy

 # RUN: %cmake %mlir_src_root/examples/standalone -DCMAKE_CXX_COMPILER=%host_cxx -DCMAKE_C_COMPILER=%host_cc -DMLIR_DIR=%llvm_lib_dir/cmake/mlir ; %cmake --build . --target check-standalone | tee %t | FileCheck %s
 
-# CHECK: Expected Passes: 3
+# CHECK: Passed: 3
 # UNSUPPORTED: windows, android