[PATCH] [llvm] [lit] Fix inconsistent test order in shtest-keyword-parse-errors

[mgorny]

Summary

Remove test times when running shtest-keyword-parse-errors test,
in order to prevent the previous executions from impacting subtest
order and therefore causing FileCheck to fail.

[jdenny]

Other than a tiny nit, LGTM.

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

1

@lebedev.ri has been marking these lines with a FIXME: ef4b3a4571e2.

Perhaps it's good to do that consistently until someone implements a more comprehensive fix (like putting the cleanup or some command-line option in %{lit}).

[mgorny]

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

1

Will do, thanks!

[jdenny]

Thanks for the fix!

[mgorny]

You're welcome. I'd love to try to figure out how to make the tests not depend on subtest order but my TODO is too long already.

[jdenny]

In D107427#2928786, @mgorny wrote:

You're welcome. I'd love to try to figure out how to make the tests not depend on subtest order but my TODO is too long already.

Yes, it's not my highest priority either.

I think some sort of command-line option to disable this behavior is the right solution, and it can be embedded in %{lit} so that no test is affected accidentally. (I'd like to use that in clang/test/utils as well.) For tests that specifically check this behavior (reorder.py), an alternate substitution (e.g., %{lit-reordered}) can be provided.

[mgorny]

Indeed. In fact, the first thing I did was to see --help if there's a way to disable these files. To be honest, I don't really like the idea of lit leaving cache files all over the place.

[jdenny]

In D107427#2928843, @mgorny wrote:

To be honest, I don't really like the idea of lit leaving cache files all over the place.

I think you're saying that users of lit (as opposed to maintainers of lit's test suite) might like such a command-line option too.

[mgorny]

Exactly. Or simply a new lexical test order option.

[mgorny]

Ok, I've decided to spend 10 minutes and implement a lexical test order option: D107695.

llvm/utils/lit/tests/shtest-keyword-parse-errors.py

+# RUN: rm -f %{inputs}/shtest-keyword-parse-errors/.lit_test_times.txt
 # RUN: not %{lit} -j 1 -vv %{inputs}/shtest-keyword-parse-errors > %t.out
 # RUN: FileCheck -input-file %t.out %s
 #
 # END.
 
 # CHECK: Testing: 3 tests
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: empty.txt
 # CHECK:       {{^}}Test has no 'RUN:' line{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: multiple-allow-retries.txt
 # CHECK:       {{^}}Test has more than one ALLOW_RETRIES lines{{$}}
 
 # CHECK-LABEL: UNRESOLVED: shtest-keyword-parse-errors :: unterminated-run.txt
 # CHECK:       {{^}}Test has unterminated 'RUN:' lines (with '\'){{$}}