[lit] Mark several of lit's tests XFAIL on Windows

[modocache]

Summary

rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.

In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.

Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:

utils/lit/lit.py \
    --path /path/to/your/llvm/build/bin \
    utils/lit/tests

Verify that the test suite is run and a successful exit code is
returned.

[modocache]

The change to utils/lit/tests/shtest-encoding.py is shown in Phabricator as a change to a binary file because that test file contains a non-UTF8 character. Like the other files, I just added the following lines:

diff --git a/utils/lit/tests/shtest-encoding.py b/utils/lit/tests/shtest-encoding.py
index dfc987f6df7..9ccd897275b 100644
--- a/utils/lit/tests/shtest-encoding.py
+++ b/utils/lit/tests/shtest-encoding.py
@@ -1,3 +1,6 @@
 # RUN: true

+# PR33936
+# XFAIL: windows
+

[delcypher]

Looks fine. The only bit I'm not sure about is the interaction of config.available_features and XFAIL.

utils/lit/tests/lit.cfg

57

I've only ever used config.available_features with the REQUIRES: declaration. Does it work with XFAIL: too? I thought XFAIL: just took a target architecture?

[delcypher]

utils/lit/tests/lit.cfg

57

Hmm looks like things have changed a lot since I last looked.

https://reviews.llvm.org/D18185

I was half right. In Test.isExpectedToFail() (lit/Test.py) I can see that the boolean expression evaluator takes the features and the target triple.

So I guess XFAIL: windows is fine.

[mgorny]

Technically LGTM but I don't have a Windows setup to test.

[rnk]

Looks good, thanks!

utils/lit/tests/discovery.py

 # Check the basic discovery process, including a sub-suite.
 #
+# PR33932
+# XFAIL: windows
+#
 # RUN: %{lit} %{inputs}/discovery \
 # RUN:   -j 1 --debug --show-tests --show-suites \
 # RUN:   -v > %t.out 2> %t.err
 # RUN: FileCheck --check-prefix=CHECK-BASIC-OUT < %t.out %s
 # RUN: FileCheck --check-prefix=CHECK-BASIC-ERR < %t.err %s
 #
 # CHECK-BASIC-ERR: loading suite config '{{.*}}/discovery/lit.cfg'
 # CHECK-BASIC-ERR-DAG: loading suite config '{{.*}}/discovery/subsuite/lit.cfg'

utils/lit/tests/googletest-format.py

 # Check the various features of the GoogleTest format.
 #
+# PR33933
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 # CHECK: PASS: googletest-format :: DummySubDir/OneTest/FirstTest.subTestA
 # CHECK: FAIL: googletest-format :: DummySubDir/OneTest/FirstTest.subTestB

utils/lit/tests/googletest-timeout.py

 # REQUIRES: python-psutil
 
+# PR33934
+# XFAIL: windows
+
 # Check that the per test timeout is enforced when running GTest tests.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-timeout --timeout=1 > %t.cmd.out
 # RUN: FileCheck < %t.cmd.out %s
 
 # Check that the per test timeout is enforced when running GTest tests via
 # the configuration file
 #

utils/lit/tests/googletest-upstream-format.py

 # Check the various features of the GoogleTest format.
 #
+# PR33935
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/googletest-upstream-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 # CHECK: PASS: googletest-upstream-format :: DummySubDir/OneTest/FirstTest.subTestA
 # CHECK: FAIL: googletest-upstream-format :: DummySubDir/OneTest/FirstTest.subTestB

utils/lit/tests/lit.cfg

 # Add a feature to detect if psutil is available
 try:
     import psutil
     lit_config.note('Found python psutil module')
     config.available_features.add("python-psutil")
 except ImportError:
     lit_config.warning('Could not import psutil. Some tests will be skipped and'
                        ' the --timeout command line argument will not work.')
+
+if sys.platform.startswith('win') or sys.platform.startswith('cygwin'):
+    config.available_features.add('windows')

utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
+# PR33941
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/shtest-shell >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/shtest-shell >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/shtest-shell 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: Failing Tests (3)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: Setting --max-failures to 0 does not have any effect.

utils/lit/tests/selecting.py

 # RUN: %{lit} %{inputs}/discovery | FileCheck --check-prefix=CHECK-BASIC %s
 # CHECK-BASIC: Testing: 5 tests
 
+# PR33937
+# XFAIL: windows
 
 # Check that regex-filtering works
 #
 # RUN: %{lit} --filter 'o[a-z]e' %{inputs}/discovery | FileCheck --check-prefix=CHECK-FILTER %s
 # CHECK-FILTER: Testing: 2 of 5 tests
 
 # Check that regex-filtering based on environment variables work.
 #

utils/lit/tests/shtest-format.py

 # Check the various features of the ShTest format.
 #
+# PR33939
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-format > %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: PASS: shtest-format :: argv0.txt

utils/lit/tests/shtest-output-printing.py

 # Check the various features of the ShTest format.
 #
+# PR33938
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-output-printing > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-output-printing :: basic.txt

utils/lit/tests/shtest-shell.py

 # Check the internal shell handling component of the ShTest format.
 #
+# PR33940
+# XFAIL: windows
+#
 # RUN: not %{lit} -j 1 -v %{inputs}/shtest-shell > %t.out
 # RUN: FileCheck --input-file %t.out %s
 #
 # END.
 
 # CHECK: -- Testing:
 
 # CHECK: FAIL: shtest-shell :: error-0.txt

utils/lit/tests/shtest-timeout.py

 # REQUIRES: python-psutil
 
+# PR33944
+# XFAIL: windows
+
 # Test per test timeout using external shell
 # RUN: not %{lit} \
 # RUN: %{inputs}/shtest-timeout/infinite_loop.py \
 # RUN: %{inputs}/shtest-timeout/quick_then_slow.py \
 # RUN: %{inputs}/shtest-timeout/short.py \
 # RUN: %{inputs}/shtest-timeout/slow.py \
 # RUN: -j 1 -v --debug --timeout 1 --param external=1 > %t.extsh.out 2> %t.extsh.err
 # RUN: FileCheck --check-prefix=CHECK-OUT-COMMON < %t.extsh.out %s

utils/lit/tests/unit/TestRunner.py

 # RUN: %{python} %s
 #
+# PR33942
+# XFAIL: windows
+#
 # END.
 
 
 import unittest
 import platform
 import os.path
 import tempfile
 

utils/lit/tests/unittest-adaptor.py

 # Check the lit adaption to run under unittest.
 #
+# PR33943
+# XFAIL: windows
+#
 # RUN: %{python} %s %{inputs}/unittest-adaptor 2> %t.err
 # RUN: FileCheck < %t.err %s
 #
 # CHECK-DAG: unittest-adaptor :: test-two.txt ... FAIL
 # CHECK-DAG: unittest-adaptor :: test-one.txt ... ok
 
 import unittest
 import sys
 
 import lit
 import lit.discovery
 
 input_path = sys.argv[1]
 unittest_suite = lit.discovery.load_test_suite([input_path])
 runner = unittest.TextTestRunner(verbosity=2)
 runner.run(unittest_suite)