[lldb] Add more documentation on test variants

[labath]

Summary

This formalizes some of the discussion on D121631.

[JDevlieghere]

LGTM. Thanks!

lldb/docs/resources/test.rst

 that runs the test suite under different configurations, with older host
 compilers and different DWARF versions.
 
 As you can imagine, this quickly lead to combinatorial explosion in the number
 of variants. It's very tempting to add more variants because it's an easy way
 to increase test coverage. It doesn't scale. It's easy to set up, but increases
 the runtime of the tests and has a large ongoing cost.
 
-The key take away is that the different variants don't obviate the need for
-focused tests. So relying on it to test say DWARF5 is a really bad idea.
-Instead you should write tests that check the specific DWARF5 feature, and have
-the variant as a nice-to-have.
+The test variants are most useful when developing a larger feature (e.g. support
+for a new DWARF version). The test suite contains a large number of fairly
+generic tests, so running the test suite with the feature enabled is a good way
+to gain confidence that you haven't missed an important aspect. However, this
+genericness makes them poor regression tests. Because it's not clear what a
+specific test covers, a random modification to the test case can make it start
+(or stop) testing a completely different part of your feature. And since these
+tests tend to look very similar, it's easy for a simple bug to cause hundreds of
+tests to fail in the same way.
+
+For this reason, we recommend using test variants only while developing a new
+feature. This can often be done by running the test suite with different
+arguments -- without any modifications to the code. You can create a focused
+test for any bug found that way. Often, there will be many tests failing, but a
+lot of then will have the same root cause.  These tests will be easier to debug
+and will not put undue burden on all other bots and developers.
 
 In conclusion, you'll want to opt for an API test to test the API itself or
 when you need the expressivity, either for the test case itself or for the
 program being debugged. The fact that the API tests work with different
 variants mean that more general tests should be API tests, so that they can be
 run against the different variants.
 
 Guidelines for API tests