Modify LIT to accept environment variable LIT_FILTER to select tests.

[george.karpenkov]

Summary

This is especially useful when lit is invoked indirectly by the build
system, and additional arguments can not be easily specified.

[• ddunbar]

LGTM

[mgorny]

I'm sorry to crash the party but the added test fails on Linux:

$ "LIT_FILTER=o[a-z]e" "/usr/src/llvm/utils/lit/_venv/bin/python" "/usr/src/llvm/utils/lit/tests/../lit.py" "/usr/src/llvm/utils/lit/tests/../tests/Inputs/discovery"
# command stderr:
'LIT_FILTER=o[a-z]e': command not found
error: command failed with exit status: 127

It looks that lit does not expect shell-style variable substitutions here. I would suggest using env ... but I suppose that wouldn't be portable to Windows (not that I expect the current syntax to work any better there).

[george.karpenkov]

@mgorny I can fix this on Linux. I will not be able to do testing on Windows though.
TBH I haven't been able to run tests: how did you do that? I've even opened a bug about this: https://bugs.llvm.org/show_bug.cgi?id=33704
Is there a buildbot checking for those?

[mgorny]

TBH I haven't been able to run tests: how did you do that? I've even opened a bug about this: https://bugs.llvm.org/show_bug.cgi?id=33704

./lit.py -v tests ;-)

Is there a buildbot checking for those?

I'd like to pursue one once we get all the current failures fixed.

[george.karpenkov]

@mgorny I still can't run them:

1. No class called FileBasedTest: where is it?
2. From which directory do you run it? If I run it from util/lit it misses FileCheck/etc. Or do you just modify $PATH?
3. Can we document it somewhere?

[mgorny]

In D35091#820739, @george.karpenkov wrote:

@mgorny I still can't run them:

1. No class called FileBasedTest: where is it?

It's a breakage from another commit. I'm currently working on finding a proper solution for it.

2. From which directory do you run it? If I run it from util/lit it misses FileCheck/etc. Or do you just modify $PATH?

I have it installed in /usr/bin. However, modifying PATH should work as well.

3. Can we document it somewhere?

Sure, please do ;-). Or maybe even add a CMake target for it ;-).

[modocache]

Running tests is documented in utils/lit/README.txt, in the section Contributing to lit.

[modocache]

As for a CMake target for lit tests, @delcypher added one in rL257221, but it was reverted because the tests failed on Windows.

[george.karpenkov]

@modocache

Running tests is documented

Yeah, but they were failing, so I thought that maybe the documentation is out of date.

but it was reverted because the tests failed on Windows

that's a shame, can't we simply if/def around it for MSVC?

[modocache]

Yeah, but they were failing, so I thought that maybe the documentation is out of date.

Ah, whoops, I hadn't read your comment above before posting -- sorry!

that's a shame, can't we simply if/def around it for MSVC?

I agree, I think this would be a good thing to do as soon as possible. Unfortunately part of the test suite fails on all host machines at the moment. I commented on https://reviews.llvm.org/D35857#820783 with how I think would be a good way to proceed -- sorry for the scattered discussion.

docs/CommandGuide/lit.rst

 
 .. option:: --run-shard=N
 
  Select which shard to run, assuming the ``--num-shards=M`` option was
  provided. The two options must be used together, and the value of ``N``
  must be in the range ``1..M``. The environment variable
  ``LIT_RUN_SHARD`` can also be used in place of this option.
 
+.. option:: --filter=REGEXP
+
+  Run only those tests whose name matches the regular expression specified in
+  ``REGEXP``. The environment variable ``LIT_FILTER`` can be also used in place
+  of this option, which is especially useful in environments where the call
+  to ``lit`` is issued indirectly.
+
 ADDITIONAL OPTIONS
 ------------------
 
 .. option:: --debug
 
  Run :program:`lit` in debug mode, for debugging configuration issues and
  :program:`lit` itself.
 

utils/lit/lit/main.py

selection_group.add_argument("--shuffle",
                      action="store_true", default=False)
     selection_group.add_argument("-i", "--incremental",
                      help="Run modified and failing tests first (updates "
                      "mtimes)",
                      action="store_true", default=False)
     selection_group.add_argument("--filter", metavar="REGEX",
                      help=("Only run tests with paths matching the given "
                            "regular expression"),
-                     action="store", default=None)
+                     action="store",
+                     default=os.environ.get("LIT_FILTER"))
     selection_group.add_argument("--num-shards", dest="numShards", metavar="M",
                      help="Split testsuite into M pieces and only run one",
                      action="store", type=int,
                      default=os.environ.get("LIT_NUM_SHARDS"))
     selection_group.add_argument("--run-shard", dest="runShard", metavar="N",
                      help="Run shard #N of the testsuite",
                      action="store", type=int,
                      default=os.environ.get("LIT_RUN_SHARD"))

utils/lit/tests/selecting.py

 # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC %s
 # CHECK-BASIC: Testing: 5 tests
 
 
 # Check that regex-filtering works
 #
 # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # CHECK-FILTER: Testing: 2 of 5 tests
 
+# Check that regex-filtering based on environment variables work.
+#
+# RUN: LIT_FILTER='o[a-z]e' %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER-ENV %s
+# CHECK-FILTER-ENV: Testing: 2 of 5 tests
+
 
 # Check that maximum counts work
 #
 # RUN: %{lit} --max-tests 3 %{inputs}/discovery | FileCheck --check-prefix=CHECK-MAX %s
 # CHECK-MAX: Testing: 3 of 5 tests
 
 
 # Check that sharding partitions the testsuite in a way that distributes the