[lit] Print substitutions with --show-suites

[ldionne]

Summary

We already print available features, and it can be useful to print
substitutions as well since those are a pretty fundamental part of
a test suite. We could also consider printing other things like the
test environment, however the need doesn't appear to be as strong.

Before:

$ lit -sv libcxx/test --show-suites
-- Test Suites --
  libc++ - 6350 tests
    Source Root: [...]
    Exec Root  : [...]
    Available Features : -faligned-allocation -fsized-deallocation [...]

After:

$ lit -sv libcxx/test --show-suites
-- Test Suites --
  libc++ - 6350 tests
    Source Root: [...]
    Exec Root  : [...]
    Available Features : -faligned-allocation -fsized-deallocation [...]
    Available Substitutions : %{build_module} => [...]
                              %{build} => %{cxx} -o [...]

[ldionne]

I thought this was useful, but I do admit it makes the output of --show-suites a lot more verbose because substitutions are larger than available features, and they are one per line. I still think this is useful, but I'm open to discussions.

[yln]

I think --show-suites is already a debugging feature (to figure out what has been discovered) so a bit more verbosity shouldn't hurt.

LGTM, with nits.

llvm/utils/lit/lit/main.py

139–140

Let's change this to always print available_features and substitutions. Not having any is rare and that is information as well.

140–145

Let's remove the space before the colon here. For Source/Exec Root it aligns the output, but for features and substitutions I don't think that it improves legibility.

142

I think tuples already have a good natural sort.

145

Maybe use ljust here?

llvm/utils/lit/tests/Inputs/discovery/lit.cfg

17

Thanks for adding a test for an existing feature! :)

20

Maybe provide unsorted entries here (also for features) so that the test proves that we sort it.

[ldionne]

Address review comments.

[jdenny]

Other than the comment I added, LGTM. Thanks.

llvm/utils/lit/lit/main.py

139–140

Please mention the change for available_features in the patch summary.

llvm/utils/lit/lit/main.py

def print_discovered(tests, show_suites, show_tests):
     if show_suites:
         import itertools
         tests_by_suite = itertools.groupby(tests, lambda t: t.suite)
         print('-- Test Suites --')
         for suite, suite_iter in tests_by_suite:
             test_count = sum(1 for _ in suite_iter)
             print('  %s - %d tests' % (suite.name, test_count))
             print('    Source Root: %s' % suite.source_root)
             print('    Exec Root  : %s' % suite.exec_root)
-            if suite.config.available_features:
-                features = ' '.join(sorted(suite.config.available_features))
-                print('    Available Features : %s' % features)
+            print('    Available Features: %s' % features)
+            substitutions = sorted(suite.config.substitutions)
+            substitutions = ('%s => %s' % (x, y) for (x, y) in substitutions)
+            substitutions = '\n'.ljust(30).join(substitutions)
+            print('    Available Substitutions: %s' % substitutions)
 
     if show_tests:
         print('-- Available Tests --')
         for t in tests:
             print('  %s' % t.getFullName())
 
 
 def determine_order(tests, order):

llvm/utils/lit/tests/Inputs/discovery/lit.cfg

 import lit.formats
 config.name = 'top-level-suite'
 config.suffixes = ['.txt']
 config.test_format = lit.formats.ShTest()
 
 # We intentionally don't set the source root or exec root directories here,
 # because this suite gets reused for testing the exec root behavior (in
 # ../exec-discovery).
 #
 #config.test_source_root = None
 #config.test_exec_root = None
 
 # Check that arbitrary config values are copied (tested by subdir/lit.local.cfg).
 config.an_extra_variable = False
+
+# Check that available_features are printed by --show-suites (and in the right order)
+config.available_features = ['feature2', 'feature1']
+
+# Check that substitutions are printed by --show-suites (and in the right order)
+config.substitutions = [('%key2', 'value2'), ('%key1', 'value1')]

llvm/utils/lit/tests/discovery.py

 #
 # CHECK-BASIC-OUT: -- Test Suites --
 # CHECK-BASIC-OUT:   sub-suite - 2 tests
 # CHECK-BASIC-OUT:     Source Root: {{.*[/\\]discovery[/\\]subsuite$}}
 # CHECK-BASIC-OUT:     Exec Root  : {{.*[/\\]discovery[/\\]subsuite$}}
 # CHECK-BASIC-OUT:   top-level-suite - 3 tests
 # CHECK-BASIC-OUT:     Source Root: {{.*[/\\]discovery$}}
 # CHECK-BASIC-OUT:     Exec Root  : {{.*[/\\]discovery$}}
+# CHECK-BASIC-OUT:     Available Features: feature1 feature2
+# CHECK-BASIC-OUT:     Available Substitutions: %key1 => value1
+# CHECK-BASIC-OUT:                              %key2 => value2
 #
 # CHECK-BASIC-OUT: -- Available Tests --
 # CHECK-BASIC-OUT: sub-suite :: test-one
 # CHECK-BASIC-OUT: sub-suite :: test-two
 # CHECK-BASIC-OUT: top-level-suite :: subdir/test-three
 # CHECK-BASIC-OUT: top-level-suite :: test-one
 # CHECK-BASIC-OUT: top-level-suite :: test-two
 

 # RUN: cd %{inputs}/exec-discovery-in-tree/obj/
 # RUN: %{lit} . \
 # RUN:   -j 1 --show-tests --show-suites -v > %t.out
 # RUN: FileCheck --check-prefix=CHECK-ASEXEC-INTREE < %t.out %s
 #
 #      CHECK-ASEXEC-INTREE:   exec-discovery-in-tree-suite - 1 tests
 # CHECK-ASEXEC-INTREE-NEXT:     Source Root: {{.*[/\\]exec-discovery-in-tree$}}
 # CHECK-ASEXEC-INTREE-NEXT:     Exec Root  : {{.*[/\\]exec-discovery-in-tree[/\\]obj$}}
+# CHECK-ASEXEC-INTREE-NEXT:     Available Features:
+# CHECK-ASEXEC-INTREE-NEXT:     Available Substitutions:
 # CHECK-ASEXEC-INTREE-NEXT: -- Available Tests --
 # CHECK-ASEXEC-INTREE-NEXT: exec-discovery-in-tree-suite :: test-one