[cmake][lit] Add check-lit CMake target

[modocache]

Summary

Use add_lit_testsuite to add a check-lit CMake target, which runs lit's
test suite. Unlike LLVM's test/lit.cfg, lit's lit.cfg does not include logic
to configure a PATH that includes not and FileCheck, so the CMake
target specifies that as an argument.

Running lit's tests results in several "Output" directories being
generated within the lit tests directory. Add these to a .gitignore
to prevent them from being added to source control.

[modocache]

Oops, wrong target name in commit title. Fixed.

[jroelofs]

I could have sworn I'd already committed (or reviewed?) a patch to make this happen.

Running lit's tests results in several "Output" directories being
generated within the lit tests directory. Add these to a .gitignore
to prevent them from being added to source control.

It'd be better to rework the tests so that they dump their output in %T, rather than in the build directory. Having builds that don't modify the source directory is a good goal, and I'd rather see us move toward that rather than away from it.

[jroelofs]

In D24995#555291, @jroelofs wrote:

I could have sworn I'd already committed (or reviewed?) a patch to make this happen.

Ohh, turns out that it was r257221, but that got reverted for some reason?

Running lit's tests results in several "Output" directories being
generated within the lit tests directory. Add these to a .gitignore
to prevent them from being added to source control.

It'd be better to rework the tests so that they dump their output in %T, rather than in the build directory. Having builds that don't modify the source directory is a good goal, and I'd rather see us move toward that rather than away from it.

Oops, I meant: s/in the build directory/in the source directory/

r257221 solved this problem by copying all the tests to the build directory, FWIW.

[beanz]

@jroelofs, thanks for digging out that patch. I thought we had tried to do this recently and something went wrong.

The revert in rL257268 said that the tests were not working correctly under windows and the lit per-test timeout tests failed on the bots.

Not sure if @delcypher has looked at this anymore since January, but his input might be useful here.

[delcypher]

In D24995#555563, @beanz wrote:

@jroelofs, thanks for digging out that patch. I thought we had tried to do this recently and something went wrong.

The revert in rL257268 said that the tests were not working correctly under windows and the lit per-test timeout tests failed on the bots.

Not sure if @delcypher has looked at this anymore since January, but his input might be useful here.

Yes I reverted the patch for the reasons stated above and I haven't found time to look into the issue since. @modocache you should probably take a look at old patch and see if there are any ideas you want to take from it. I think copying the tests to build directory and running there is cleaner. I really don't want build files ending up in the source tree.

I think the lit per-test timeouts were probably fixed by later patch in rL257616 but I never investigated re-running on the build bots.

[modocache]

Awesome, thanks @delcypher, will do! One question: is there a way I can test on the Windows buildbots before committing a change? Alternatively, I could prepare a Windows environment and try building and running the tests there, but it would be great if I could do a test run on the actual bots.

[beanz]

@modocache, sadly we do not have pre-commit testing on LLVM.

[modocache]

I still need to double-check this works on Windows, sorry for the delay!

[modocache]

Abandoning this in favor of D35880, which un-reverts @delcypher's previous work.

test/CMakeLists.txt

 add_lit_testsuite(check-llvm "Running the LLVM regression tests"
   ${CMAKE_CURRENT_BINARY_DIR}
   PARAMS llvm_site_config=${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
          llvm_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg
   DEPENDS ${LLVM_TEST_DEPENDS}
   )
 set_target_properties(check-llvm PROPERTIES FOLDER "Tests")
 
+add_lit_testsuite(check-lit "Running lit's own test suite"
+  ${LLVM_MAIN_SRC_DIR}/utils/lit/tests
+  DEPENDS ${LLVM_TEST_DEPENDS}
+  ARGS
+    # lit's own test.cfg doesn't have logic to find the LLVM
+    # utlities it depends on, like 'not' and 'FileCheck',
+    # so manually specify the PATH here.
+    --path=${LLVM_BINARY_DIR}/bin
+  )
+
 add_lit_testsuites(LLVM ${CMAKE_CURRENT_SOURCE_DIR}
   PARAMS llvm_site_config=${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
          llvm_unit_site_config=${CMAKE_CURRENT_BINARY_DIR}/Unit/lit.site.cfg
   DEPENDS ${LLVM_TEST_DEPENDS}
   )
 
 # Setup a legacy alias for 'check-llvm'. This will likely change to be an
 # alias for 'check-all' at some point in the future.
 add_custom_target(check)
 add_dependencies(check check-llvm)
 set_target_properties(check PROPERTIES FOLDER "Tests")

utils/lit/tests/.gitignore

+Output
+