Add and link to a testing philosophy document in the developer documentation.

[echristo]

Summary

Here's an RFC with patch on documentation explaining the how and why of splitting up tests in the testsuite. Effectively: "Why we don’t emit assembly in the clang testsuite”. It largely elaborates on things already in the developer documentation, but provides a (hopefully) useful example and answers a few questions I've gotten a lot over the years. I wrote this a while back after having to explain for the Nth time why we split things up :)

I've added a few people that probably care about this sort of thing to the review line.

This patch:

• Adds a link to a testing philosophy document in the developer documentation.
• Creates a set of explanatory text in another file explaining our philosophy.

[MatzeB]

Thanks for writing this! And LGTM in it's current form already.

I wonder if we should mention unittests here... As the whole llc/opt per-pass testing thing works nice for the majority of our testing. But we still have llvm/unittests for the cases where it doesn't...

docs/TestingPolicy.rst

42

I hate making a simple example more complicated... But I would add -disable-O0-optnone to the clang commandline as many developers get biten by that nowadays.

67–69

as you mention testing a single-pass in opt you should also mention:

`llc -o - a.mir -run-pass=peephole-opt | FileCheck %s`

93–94

see above

109–110

The first sentence seems a bit too much tongue in cheek...

You could also switch the perspective here: "You should make it easy for others to fix your bugs".

[MatzeB]

docs/TestingPolicy.rst

42

Actually maybe using -O3 -mllvm -disable-llvm-optzns -emit-llvm would be the way to go here ?

[rnk]

docs/TestingPolicy.rst

36

Is it worth mentioning Sema tests? This is targeted at backend / ISA people, but oftentimes attributes and target features need diagnostic tests. We could mention it briefly here and link to the VerifyDiagnosticConsumer doxygen to explain the other test format: https://clang.llvm.org/doxygen/classclang_1_1VerifyDiagnosticConsumer.html

42

Clang CodeGen RUN lines use -cc1 and look more like:
%clang_cc1 -triple x86_64-linux-gnu -emit-llvm %s -o - | FileCheck %s

I don't think -disable-llvm-passes / -disable-O0-optnone is necessary. It is necessary when attempting to translate C input to LLVM IR for opt tests in the next step. I'd document it just below here.

Also, I prefer -O1 -Xclang -disable-llvm-passes, i.e. give me lifetime annotations, and any other stuff that Clang would put in the actual "optimizations enabled" IR. I remove them manually when reducing the LLVM IR test case.

126

Personally, I don't like this tool. We end up with golden tests or pixel tests that have to be regenerated when you breathe on the compiler, rather than targeted tests that are constructed so that the relevant code sequence can be easily pattern matched.

docs/DeveloperPolicy.rst

   entire failing program into ``llvm/test`` as this creates a *time-to-test*
   burden on all developers. Please keep them short.
 
+* More testing details and discussion can be found in
+  :doc:`Testing Policy <TestingPolicy>`.
+
 Note that llvm/test and clang/test are designed for regression and small feature
 tests only. More extensive test cases (e.g., entire applications, benchmarks,
 etc) should be added to the ``llvm-test`` test suite.  The llvm-test suite is

docs/TestingPolicy.rst

+===================
+LLVM Testing Policy
+===================
+
+.. contents::
+   :local:
+
+Introduction
+============
+
+LLVM was designed from the beginning to enable fine grained testing of compiler
+passes and code emission. While the project has expanded to include compilers,
+assemblers, linkers, and debuggers the general testing philosophy continues:
+
+**“Test fine grained function rather than full integration tests wherever
+possible.”**
+
+This means that tests involving clang and the backend are decoupled from each
+other to make sure that we're testing the right part each time.
+
+Example
+=======
+
+Let’s take as an example a C++ file that you’d like to check the assembly output
+in order to get an idea of the probable performance impact of an individual
+function.
+
+You could add something in llvm/tools/clang/test/CodeGen that looks like:
+
+``clang -S -O3 -target my_favorite_cpu-linux-gnu a.c -o - | FileCheck %s``
+
+that checks the output of your favorite test compiled by clang with all of the
+optimization you desired. This is just the kind of test we’d like to avoid.
+Instead look at adding tests in all of these places:
+
+* ``llvm/tools/clang/test/CodeGen``
+* ``llvm/test/Transforms/``
+* ``llvm/test/CodeGen/my_favorite_cpu``
+
+The first test will look something like:
+
+``clang -O0 -mllvm -disable-llvm-optzns -S a.c -o - -emit-llvm | FileCheck %s``
+
+And verify that the IR emitted out of clang is both correct and a reasonable
+place to begin optimization.
+
+The second test (or set of tests) will look something like:
+
+``opt -O3 a.ll -S -o - | FileCheck %s``
+
+or even better:
+
+``opt -instcombine a.ll -S -o - | FileCheck %s``
+
+This second set of tests takes the llvm IR generated by the first test and
+runs some amount of optimization on it - making sure that the optimization
+passes are able to take the expected incoming IR patterns and generate
+appropriate IR for the code generator. The first test will run a lot of
+optimizations, the second one is even more targeted to a single run of a pass.
+The second form is preferable here if you can arrange it, we’ll discuss why
+more later.
+
+The third test will then look something like:
+
+``llc -O3 a.ll -S -o - | FileCheck %s``
+
+or even:
+
+``llc a.ll -S -o - | FileCheck %s``
+
+This last test (or set of tests) takes IR coming out of the mid-level optimizers
+that you’ve saved and tidied up by hand and runs it through the backend to
+generate assembly code that you can then check for particular set of assembly
+instructions you want to make sure are generated.
+
+Look at how all of these tests work as unit tests on different parts of the llvm
+infrastructure.
+
+* .cc tests in clang (that produce IR) are used to test the correctness of the
+  IR that comes out of clang.
+* opt tests (in llvm) that produce IR are designed to test the effectiveness of
+  the optimizer.
+* .ll (or MIR) tests are used to test the effectiveness of the code generator.
+
+Each of these has its part in how we want to test code generation from a .cc
+file in llvm.
+
+FAQ/Additional Comments
+=======================
+
+#. Passes can differ between clang and opt.
+
+   ``clang++  -O0 -mllvm -disable-llvm-optzns -S a.cc -o - -emit-llvm | opt -O3
+   -S | llc -O3``
+
+   Can be different from
+
+   ``clang++ -O3 -S a.cc``
+
+   Due to differences in the set of passes and additional infrastructure that
+   can be difficult to figure out when invoking clang versus the developer
+   tools. The developer list (or IRC) is useful for trying to analyze this, but
+   sometimes you’ll need to modify the intermediate steps in your test by hand
+   or add additional command line flags you weren’t aware of the ensure the
+   behavior you were expecting.
+
+#. “But I just want to check that my integration all worked!”
+
+   You’re a compiler developer now! We want to be more able to pinpoint bugs
+   when something fails under testing.
+
+#. “... but how can I ensure that the integration test actually works?”
+
+   Comments in test files are great! One thing that has proven useful over the
+   years is taking the original source and adding it as a comment with
+   reproduction instructions on how you generated the reduced testcase so that
+   when someone wants to verify that the original test works it makes it easier,
+   and they’ve still got a reduced testcase to make development easier. You
+   can also add an execution test to ``llvm/projects/test-suite`` to ensure the
+   runtime correctness of your testcase.
+
+#. “OK, this sounds great, but I have a lot of tests?”
+
+   There are a number of tools in the repository that can help you with scripts
+   to make updating and writing your tests much easier. Take a look at
+   ``update_test_checks.py`` and the similar ones for llc and mir to make your
+   life as a compiler developer much easier.