[lit][googletest] Don't choke on non-executables

[modocache]

Summary

lit is usually forgiving when it comes to files that aren't tests, but
happen to match the test suffix being searched for. For example, even if I
specify a suffix of ".py", lit will not fail the test suite if it finds
a Python file that doesn't contain any lit commands.

lit's googletest format, on the other hand, is not as forgiving. If a
normal, non-executable file matching the test suffix is found, lit
attempts to execute it:

NotATest --gtest_list_tests

This fails, and causes lit to fail the test suite.

To mirror the lit's lenient behavior with other test formats, instead
of failing the test, have lit simply emit a warning instead. Add a test
that verifies the new behavior.

[• ddunbar]

In cases where this is trigger, wouldn't the warning always fire and the
proper way to suppress (so developers don't regularly see warnings) to
exclude that path? If so, then it seems like erroring out is ok, it just
forces resolution of that issue immediately.

[modocache]

I think this situation does signify a problem that should be remedied, but I think I disagree with the severity with which lit reports the problem. When running lit with ShTest on a directory containing no tests, for example, lit emits a warning but still exits successfully. Why should the GoogleTest format emit an error, as opposed to a warning?

My problem stems from the fact that I am running lit tests using a third-party project's lit.cfg. I'd rather not fork that project just to modify its test suffix setting, or to add an lit_config.excludes. Ideally I'd be able to use lit.py --filter, but that filter only applies to which tests are run, not which tests are discovered. And since GoogleTest fails during the discovery process here, --filter doesn't help.

My problem would also be solved by adding something like a top-level lit.py --excludes option, but this approach seemed like much less work. :)

[modocache]

@ddunbar I've submitted https://github.com/apple/swift/pull/2054 to apple/swift based on our discussion. Thanks!

[modocache]

Abandoned in favor of https://github.com/apple/swift/pull/2054, which scratches my itch. Thanks!

utils/lit/lit/formats/googletest.py

def getGTestTests(self, path, litConfig, localConfig):
 
         try:
             lines = lit.util.capture([path, '--gtest_list_tests'],
                                      env=localConfig.environment)
             if kIsWindows:
               lines = lines.replace('\r', '')
             lines = lines.split('\n')
         except:
-            litConfig.error("unable to discover google-tests in %r" % path)
-            raise StopIteration
+            litConfig.warning("unable to discover google-tests in %r" % path)
+            lines = []
 
         nested_tests = []
         for ln in lines:
             if not ln.strip():
                 continue
 
             prefix = ''
             index = 0

utils/lit/tests/Inputs/googletest-format/DummySubDir/NotAGoogleTest

+This file is not an executable, but it happens to have the same test suffix as
+as the googletest executables in this directory. lit should emit a warning in
+this case, but it should not fail the test suite.