[test] Add dotest wrapper

[JDevlieghere]

Summary

This adds a wrapper around dotest, similar to llvm-lit in llvm. The
wrapper is created in the binary directory, next to LLDB and allows you
to invoke dotest without having to pass any of the configuration
arguments yourself. I think this could also be useful for re-running a
particular test case when it fails, as an alternative to "Command
Invoked".

The motivation for this is that I'd like to replace the driver part of
dotest with lit. As a first step, I'd like to have lit invoke dotest,
which would just run the complete test suite, completely identical to
what the CMake target does today. Once this is in place, we can have lit
run dotest for the different test directories, and ultimately once per
python file. Along the way we can strip out driver functionality from
dotest where appropriate.

Please let me know what you think about this!

[aprantl]

Thanks for working on this! Have you thought about how this would work for a bot that wants to run the dotest testsuite against multiple compilers (clang trunk, clang-6.0, clang-5.0, , ...)?

[JDevlieghere]

In D44002#1025433, @aprantl wrote:

Thanks for working on this! Have you thought about how this would work for a bot that wants to run the dotest testsuite against multiple compilers (clang trunk, clang-6.0, clang-5.0, , ...)?

I must admit I haven't. I'm guessing you mean specifically for doing that in combination with lit?

If it's just this one variable that's undecided at compile time, I think we can work around this. Lit itself supports you to pass variables with the -D prefix, which could be used to pass a different compiler to dotest. The wrapper itself would just forward that argument. I assume there is (and if there isn't it's trivial to add) support for overwriting the compiler?

[JDevlieghere]

FWIW I've been using this all day and so far it has been nothing but really convenient, even if I say so myself 🙂

[labath]

I've been wanting to implement some thing like this myself, but I did not attempt it because of the issue of integration with the xcode project. If you can handle that part, then I'm all for it.

Having just a dotest frontend with hardcoded default options is fine, but when we start a more deeper integrations we will need to think about the different config issue that Adrian pointed out. BTW, the compiler is not the only config option that is variable. For local builds there is at least the "--arch" option (so you can test 32-bit capabilities). Then for remote testing you also have the --platform-*** family of options. And our bots also use the --skip-category option selectively (to skip watchpoint tests, as some devices don't support them).

test/dotest.in

1–3

I think this should be a python file, for the sake of windows folks.

[aprantl]

I must admit I haven't. I'm guessing you mean specifically for doing that in combination with lit?

I haven't either :-)
But I would really like to have a bot the builds lldb, runs (what we call today) lit tests and the unittests, and then runs the dotest tests in a number of different configurations with different compilers.

I assume there is (and if there isn't it's trivial to add) support for overwriting the compiler?

Which tool are you referring to, lit?

We could take this even one step further, by either elevating the big categories (dwarf, dsym, dwo) on the same level as the compiler choice (e.g., run the testsuite for (clang-6.0, dwo)), *or* by sinking the compiler choice into test matrix inside of dotest (basically dotest would accept a list of compilers).

[labath]

In D44002#1025558, @aprantl wrote:

We could take this even one step further, by either elevating the big categories (dwarf, dsym, dwo) on the same level as the compiler choice (e.g., run the testsuite for (clang-6.0, dwo)), *or* by sinking the compiler choice into test matrix inside of dotest (basically dotest would accept a list of compilers).

At one point in the past, dotest accepted a list of compilers and architectures and then ran each test for the cartesian product. However, this was broken for a long time (it operated on a completely different level than the debug flavour thingy) and so it was removed.

[vsk]

The basic approach looks good -- lgtm with Pavel's comment addressed. Thanks!

[davide]

In D44002#1025573, @labath wrote:

In D44002#1025558, @aprantl wrote:

We could take this even one step further, by either elevating the big categories (dwarf, dsym, dwo) on the same level as the compiler choice (e.g., run the testsuite for (clang-6.0, dwo)), *or* by sinking the compiler choice into test matrix inside of dotest (basically dotest would accept a list of compilers).

At one point in the past, dotest accepted a list of compilers and architectures and then ran each test for the cartesian product. However, this was broken for a long time (it operated on a completely different level than the debug flavour thingy) and so it was removed.

That's a good thing. I'm really not in favor of having this cartesian explosion. If people want to run multiple configuration, then it needs to happen in some principled manner (if anything, the cmdline should be generated by a textual configuration representation, YAML or JSON or something else).

About the patch itself, LGTM. I wonder whether we should call the binary lldb-dotest. But I don't feel really strong about it, so go for whatever you like.

[JDevlieghere]

Thanks for the feedback everyone! I committed this in r326687.

test/CMakeLists.txt

   "Testing LLDB (parallel execution, with a separate subprocess per test)"
   )
 
+# Generate a wrapper for dotest.py in the bin directory.
+string (REPLACE ";" " " LLDB_DOTEST_ARGS_STR  "${LLDB_DOTEST_ARGS}")
+# We need this to substitute variables.
+configure_file(
+  dotest.in
+  ${CMAKE_CURRENT_BINARY_DIR}/dotest.configured
+  )
+# We need this to expand the generator expressions.
+file(GENERATE
+  OUTPUT
+  $<TARGET_FILE_DIR:lldb>/dotest
+  INPUT
+  ${CMAKE_CURRENT_BINARY_DIR}/dotest.configured
+  )
+
 # If we're building with an in-tree clang, then list clang as a dependency
 # to run tests.
 if (TARGET clang)

test/dotest.in

+#!/usr/bin/env sh
+
+@LLDB_SOURCE_DIR@/test/dotest.py -q @LLDB_DOTEST_ARGS_STR@ $@