Make shtest-format.py CHECK lines more flexible

[dyung]

Summary

I've noticed that when run on Windows, the lit test shtest-format.py seems to fail quite often on my machine, seemingly randomly. From an examination of the error output, in my case it seemed that it was due to an extra line being printed out which caused the test to fail because it was using CHECK-NEXT. If I change to use CHECK instead of CHECK-NEXT, it should still maintain the intent of the test while making it flexible enough to handle the "random" error that I see often.

For example, here is the output that the test expects:

Command Output (stderr): 
-- 
cat: does-not-exist: No such file or directory 
 
--

However, on my machine, for reasons I haven't been able to figure out and/or fix, the bash.exe that is used to execute the test emits a warning about a missing /tmp directory to stderr which appears first and causes the CHECK-NEXT sequence to fail. The output looks like this:

Command Output (stderr): 
-- 
bash.exe: warning: could not find /tmp, please create! 
cat: does-not-exist: No such file or directory 
 
--

I strongly suspect that when the test passes on my machine, it is because the bash warning is emitted AFTER the expected line in the stderr stream, so it doesn't fail the test because after matching the cat error, the test only uses CHECK to detect the "--" at the end of the test rather than CHECK-NEXT.

The fix I'm proposing here still checks for the expected output, but doesn't require the lines to immediately follow one-another making it more flexible and should preserve the intent of the test.

The test random failures can also be seen on our public PS4 Windows bot: https://lab.llvm.org/staging/#/builders/204

[dyung]

This is a simple change that could make Windows testing of this test more reliable I think. Could I get a quick review?

[probinson]

llvm/utils/lit/tests/shtest-format.py

20–21

I think we want to have the test continue to guarantee the cat line comes from the correct command, without requiring it to be the first output line. This should do the trick:

CHECK-NEXT: --
CHECK-NOT: --
CHECK: cat{{(_64) yada yada

[dyung]

Update change based on @probinson 's feedback.

[probinson]

LGTM

llvm/utils/lit/tests/shtest-format.py

 # CHECK: FAIL: shtest-format :: external_shell/fail.txt
 # CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail.txt' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: line 1: failed test output on stdout
 # CHECK-NEXT: line 2: failed test output on stdout
 # CHECK: Command Output (stderr):
 # CHECK-NEXT: --
-# CHECK-NEXT: cat{{(_64)?(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
+# CHECK-NOT: --
+# CHECK: cat{{(_64)?(\.exe)?}}: {{cannot open does-not-exist|does-not-exist: No such file or directory}}
 # CHECK: --
 
 # CHECK: FAIL: shtest-format :: external_shell/fail_with_bad_encoding.txt
 # CHECK-NEXT: *** TEST 'shtest-format :: external_shell/fail_with_bad_encoding.txt' FAILED ***
 # CHECK: Command Output (stdout):
 # CHECK-NEXT: --
 # CHECK-NEXT: a line with bad encoding:
 # CHECK: --