LNT Documentation Cleanup

[MatzeB]

Summary

Putting this up for review as there are various removals in here.

docs/developer_guide:

• Remove pip install psycopg2 note: The package is part of requirements.server.txt anyway
• Slightly improve wording.
• Fix the 2nd and 3rd sample invocation of the tests.

docs/intro:

• Remove the development part of the intro section, as there is a whole developer_guide section nowadays which is less outdated.

docs/tests:

• Move runtest test-suite part first
• Add deprecation notice to runtest nt

docs/importing_data.rst:

• Create subsections instead of toplevel ones

docs/todo:

Remove the whole TODO section:
- Maintaining a TODO in the documentation feels slightly out of place.
- It is really outdated and nobody seems to update it.

docs/changes:

• Remove sorry excuse of a changelog

[MatzeB]

ping

[grosser]

Thanks for pushing this forward!

CHANGELOG.txt

-0.4.2
-=====
-
-We did a really bad job of incrementing this number for years. Lots of changes!
-
-
-0.4.1
-=====
-
-...
-
-0.4.0
-=====
-
-* Major redesign of database schema, primarily for performance reasons.
-
-  * New design allows for definition of independent "test suites" which define
-    the exact kinds of test data that are stored in the database. This greatly
-    increased LNT's utility for testing things other than just the LLVM
-    test-suite.
-
-0.3.0
-=====
-
-* Initial release.

docs/api.rst

 +---------------------------+------------------------------------------------------------------------------------------+
 
 Write Operations
 ----------------
 
 The machines, orders and runs endpoints also support the DELETE http method.  The user must include a http header called
 "AuthToken" which has the API auth token set in the LNT instance configuration.
 
-The API Auth token can be set by adding `api_auth_token` to the instnaces lnt.cfg config file::
+The API Auth token can be set by adding `api_auth_token` to the instances lnt.cfg config file::
 
     # API Auth Token
     api_auth_token = "SomeSecret"
 
 Example::
 
     curl --request DELETE --header "AuthToken: SomeSecret" http://localhost:8000/api/db_default/v4/nts/runs/1

docs/changes.rst

-:tocdepth: 2
-
-.. _changes:
-
-Changelog
-*********
-
-.. include:: ../CHANGELOG.txt

docs/contents.rst

 .. _contents:
 
 Contents
 ========
 
 .. toctree::
    :maxdepth: 2
 
    intro
 
    quickstart
 
    tools
 
    tests
 
-   changes
-
    concepts
 
    api
 
    importing_data
 
    developer_guide
 
    profiles
    
-   todo
-
 Indices and tables
 ==================
 
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
 
 
 Module Listing
 --------------
 
 .. toctree::
    :maxdepth: 2
 
    modules/testing
    

docs/developer_guide.rst

 .. _developer_guide:
 
 Developer Guide
 ===============
 
 This developer guide aims to get you started with developing LNT. At the
 moment, a lot of detailed info is missing, but hopefully this will get you
 started.
 
 Installation
 ------------
 
 See :ref:`quickstart` for setting up an installation. Use the "develop" option
 when running ~/lnt/setup.py.
 
-You may have to install psycopg2 into your python virtualenv to run the
-unit tests against a postgres database:
-
-     pip install psycopg2
-
-
 Running LNT's Regression Tests
 ------------------------------
 
 LNT has a growing body of regression tests that makes it easier to improve LNT
 without accidentally breaking existing functionality. Just like when developing
 most other LLVM sub-projects, you should consider adding regression tests for
 every feature you add or every bug you fix. The regression tests must pass at
 all times, therefore you should run the regression tests as part of your
 development work-flow, just like you do when developing on other LLVM
 sub-projects.
 
 The LNT regression tests make use of lit and other tools like FileCheck. At
 the moment, probably the easiest way to get them installed is to compile LLVM
 and use the binaries that are generated there. Assuming you've build LLVM
 into $LLVMBUILD, and installed lnt in $LNTINSTALL you can run the regression
 tests using the following command::
 
      PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv ./tests
 
 If you don't like temporary files being created in your LNT source directory,
 you can run the tests in a different directory too::
 
      mkdir ../run_lnt_tests
      cd ../run_lnt_tests
-     PATH=$LLVMBUILD/bin:$PATH llvm-lit -sv ../lnt/tests
+     PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv ../lnt/tests
 
 For simple changes, adding a regression test and making sure all regression
 tests pass, is often a good enough testing approach. For some changes, the
 existing regression tests aren't good enough at the moment, and manual testing
 will be needed.
 
 For any changes that touch on the LNT database design, you'll need to run tests
 on at least sqlite and postgres database engines.  By default the regression
-tests uses sqlite. To run the regression tests against a postgress database,
-use a command like the following::
+tests uses sqlite. To enable additional regression tests against a postgress
+database, use a command like the following::
 
-     PATH=$LLVMBUILD/bin:$PATH llvm-lit -sv -Dpostgres=1 ../lnt/tests
+     PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv -Dpostgres=1 ../lnt/tests
 
 You'll need to use at least postgres version 9.2 to run the regression tests.

docs/importing_data.rst

 .. _importing_data:
 
+Importing Data
+==============
+
 Importing Data in a Text File
-=============================
+-----------------------------
 
 The LNT importreport command will import data in a simple text file format. The
 command takes a space separated key value file and creates an LNT report file,
 which can be submitted to a LNT server.  Example input file::
 
     foo.exec 123
     bar.size 456
     foo/bar/baz.size 789
 
 The format is "test-name.metric", so exec and size are valid metrics for the
 test suite you are submitting to.
 
 Example::
 
     echo -n "foo.exec 25\nbar.score 24.2\nbar/baz.size 110.0\n" > results.txt
     lnt importreport --machine=my-machine-name --order=1234 --testsuite=nts results.txt report.json
     lnt submit http://mylnt.com/default/submitRun --commit report.json
 
 .. _json_format:
 
 LNT Report File Format
-======================
+----------------------
 
 The lnt importreport tool is an easy way to import data into LNTs test format.
 You can also create LNTs report data directly for additional flexibility.
 
 First, make sure you've understood the underlying :ref:`concepts` used by LNT.
 
 .. code-block:: none
 

 Given how simple it is to make your own results and send them to LNT,
 it is common to not use the LNT client application at all, and just have a
 custom script run your tests and submit the data to the LNT server. Details
 on how to do this are in :mod:`lnt.testing`
 
 .. _nts_suite:
 
 Default Test Suite (NTS)
-========================
+------------------------
 
 The default test-suite schema is called NTS. It was originally designed for
 nightly test runs of the llvm test-suite. However it should fit many other
 benchmark suites as well. The following metrics are supported for a test:
 
 * ``execution_time``: Execution time in seconds; lower is better.
 * ``score``: Benchmarking score; higher is better.
 * ``compile_time``: Compiling time in seconds; lower is better.
 * ``execution_status``: A non zero value represents an execution failure.
 * ``compilation_status``: A non zero value represents a compilation failure.
 * ``hash_status``: A non zero value represents a failure computing the
   executable hash.
 * ``mem_byts``: Memory usage in bytes during execution; lower is better.
 * ``code_size``: Code size (usually the size of the text segment) in bytes;
   lower is better.
 
 The `run` information is expected to contain this:
 
 * ``llvm_project_revision``: The revision or version of the compiler
   used for the tests. Used to sort runs.
 
 .. _custom_testsuites:
 
 Custom Test Suites
-==================
+------------------
 
 LNTs test suites are derived from a set of metadata definitions for each suite.
 Simply put, suites are a collections of metrics that are collected for each run.
 You can define your own test-suites if the schema in a different suite does not
 already meet your needs.
 
 To create a schema place a yaml file into the schemas directory of your lnt
 instance. Example:

docs/intro.rst

  2. Then create LNT installation::
 
       lnt create path/to/install-dir --db-dir postgresql://user@host
 
  3. Run server normally::
 
       lnt runserver path/to/install-dir
 
-Development
------------
-
-Developing LNT should be done under a virtualenv (most likely in 'develop'
-mode). Currently, the tests require:
-
- 1. 'lit', the LLVM test runner, is available.
-
- 2. If inside a virtualenv, the virtualenv will need to be activated because the
- tests want to run Python directly. This should be fixed to substitute in the
- Python that was used to run lit, so the venv doesn't have to be active.
-
-To run the tests use, e.g.::
-
-  lit -sv $ROOT/tests
-
-This requires the 'lit' testing tool be installed. You can also use::
-
-  python setup.py test
-
-if you prefer 'unittest' style output (this still requires that 'lit' be
-installed).
-
 Architecture
 ------------
 
 The LNT web app is currently implemented as a Flask WSGI web app, with Jinja2
 for the templating engine. My hope is to eventually move to a more AJAXy web
 interface.
 
 The database layer uses SQLAlchemy for its ORM, and is typically backed by
 SQLite, although I have tested on MySQL on the past, and supporting other
 databases should be trivial. My plan is to always support SQLite as this allows
 the possibility of developers easily running their own LNT installation for
 viewing nightly test results, and to run with whatever DB makes the most sense
 on the server.

docs/tests.rst

``lnt runtest [<run options>] <test name> ... test arguments ...``
     The remainder of the options are passed to the test tool itself. The options
     are specific to the test, but well behaved tests should respond to ``lnt
     runtest <test name> --help``. The following section provides specific
     documentation on the built-in tests.
 
 Built-in Tests
 --------------
 
+LLVM CMake test-suite
+~~~~~~~~~~~~~~~~~~~~~
+
+The llvm test-suite can be run with the ``test-suite``` built-in test.
+
+Running the test-suite via CMake and lit uses a different LNT test::
+
+  $ rm -rf /tmp/BAR
+  $ lnt runtest test-suite \
+       --sandbox /tmp/BAR \
+       --cc ~/llvm.obj.64/Release+Asserts/bin/clang \
+       --cxx ~/llvm.obj.64/Release+Asserts/bin/clang++ \
+       --use-cmake=/usr/local/bin/cmake \
+       --use-lit=~/llvm/utils/lit/lit.py \
+       --test-suite ~/llvm-test-suite \
+       --cmake-cache Release
+
+Since the CMake test-suite uses lit to run the tests and compare their output,
+LNT needs to know the path to your LLVM lit installation.  The test-suite Holds
+some common common configurations in CMake caches. The ``--cmake-cache`` flag
+and the ``--cmake-define`` flag allow you to change how LNT configures cmake
+for the test-suite run.
+
+
 LLVM Makefile test-suite (aka LLVM Nightly Test)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+.. note:: The Makefile test-suite is deprecated.
+   Consider using the cmake based ``lnt runtest test-suite`` mode instead.
+   It is actively maintained, collects additional metrics such as code size
+   and has extra features such as producing and using PGO data.
+
 The ``nt`` built-in test runs the LLVM test-suite execution and performance
 tests, in the "nightly test" configuration. This test allows running many
 different applications and benchmarks (e.g., SPEC), with various compile
 options, and in several different configurations (for example, using an LLVM
 compiler like ``clang`` or ``llvm-gcc``, running under the LLVM JIT compiler
 using the LLVM ``lli`` bit-code interpreter, or testing new code generator
 passes).
 

  * ``EXECUTION_ENVIRONMENT_OVERRIDES`` [optional] - If given, a ``env`` style list of
    environment overrides to use when executing tests.
 
 For more information, see the example tests in the LLVM test-suite repository
 under the ``LNT/Examples`` directory.
 
 
 
-LLVM CMake test-suite
-~~~~~~~~~~~~~~~~~~~~~
-
-The LLVM test-suite also has a new CMake driver.  It can run more tests in
-more configurations than the Make based system. It also collects more
-metrics than the Make system, for example code size.
-
-Running the test-suite via CMake and lit uses a different LNT test::
-
-  $ rm -rf /tmp/BAR
-  $ lnt runtest test-suite \
-       --sandbox /tmp/BAR \
-       --cc ~/llvm.obj.64/Release+Asserts/bin/clang \
-       --cxx ~/llvm.obj.64/Release+Asserts/bin/clang++ \
-       --use-cmake=/usr/local/bin/cmake \
-       --use-lit=~/llvm/utils/lit/lit.py \
-       --test-suite ~/llvm-test-suite \
-       --cmake-cache Release
-     
-Since the CMake test-suite uses lit to run the tests and compare their output,
-LNT needs to know the path to your LLVM lit installation.  The test-suite Holds
-some common common configurations in CMake caches. The ``--cmake-cache`` flag
-and the ``--cmake-define`` flag allow you to change how LNT configures cmake
-for the test-suite run.
-
-
 Capturing Linux perf profile info
 +++++++++++++++++++++++++++++++++
 
 When using the CMake driver in the test-suite, LNT can also capture profile
 information using linux perf. This can then be explored through the LNT webUI
 as demonstrated at
 http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html .
 

docs/todo.rst

-.. _todo:
-
-TODO
-====
-
-This is a TODO list of major and minor features for LNT.
-
-Major Features
---------------
-
-Too many to name!
-
-Minor Features
---------------
-
-Optimize test distribution format for common cases.
-
- 1. We should left the test info higher in the format, so that it can easily be
- shared by a large number of samples.
-
- 2. We should specify test samples in an array instead of objects, to avoid
- requiring repetitive 'Name' and 'Data' keys.
-
- 3. We should support [test, sample] in addition to [test, [sample, ...]].
-
- 4. If we changed the .success marker to be .failure, then having [test] be a
- shortcut for [test, 0] would be fairly nice, and in the visualization we would
- automatically get the right defaulting for absent tests.
-
-These changes would significantly compact the archive format, which improves
-performance across the board.
-
-Other stuff:
-
- 1. We should find ways to manage the SQLite databases better. Currently we:
-
-    o Could benefit from having LNT manage when to run ANALYZE.
-
-    o Could benefit from making LNT handle setting some of the page size
-      pragmas, at some point.
-
-    o Could benefit from finding a way to have LNT VACUUM, although this can be
-      very expensive.
-
- 2. We should get an LNT buildbot up somewhere.
-
- 3. We should start defining performance tests for LNT itself, and have LNT
-    monitor them (of course). Probably best integrated into the LNT buildbot.
-
-v0.4 Redesign
--------------
-
- - Kill parameter sets.
-
-   o Is this actual worth doing? If we just bring up dual models for a while
-     there is no reason to spend time on the old model.
-
-   o Counter argument is that it might make migrating code more simple.
-
-   o Another counter argument is that we want to remove these from some places
-     that just migrating the schema won't touch (like the submission format).
-
- - Add schema version to test submission format (for future future proofing).
-
- - Schema redesign:
-
-   o Plan on doing a dual submission model for bringup purposes.
-
-   o Part of the planned schema design is to have proper attributes for the rows
-     in the run table. Should we moved to a typed model?
-
-   o See below.
-
- - Open question: UI rewrite at the same time? We are going to have to do a
-   metric ton of rewrite work to adapt to the new schema anyway.
-
- - Open question: Introduce Mongo dependency? I would rather not, from a
-   dependency perspective, but I also would really like to have mongo
-   available. Also I am almost certainly going to want to move llvmlab to using
-   mongo so it is an effective dependency on the server. Having it as a
-   dependency for local use seems sucky though. Shame there isn't a local
-   isolated mongo-as-a-library client.
-
-   o CONCLUSION: Not for v0.4.
-
- - Run order redesign?
-
-   o The current run order design is really unfortunate. It would be much better
-     if we could provide a way for users to report all of the important
-     revisions. Unfortunately, this places a lot of requirements on the server
-     for understanding all the repositories, especially the git based ones.
-
-Schema Resign
-~~~~~~~~~~~~~
-
-Primary Purpose
-+++++++++++++++
-
-The primary purpose of the schema redesign is to eliminate what Bill Karwin
-calls the "Entity-Attribute-Value" anti-pattern in his book SQL
-Antipatterns. This is where we basically store arbitrary dictionaries of
-attributes in the various Info tables, which I have since discovered is a
-terrible idea for all the reasons Karwin articulates.
-
-The particular difficulty with following this within the other LNT design goals
-is that we still have the desire to allow users to report very flexible
-structured data.
-
-Currently, the best idea to have to resolve this conflict is that we will
-construct tables on the fly. I suspect many SQL experts might also regard that
-as an anti- or scary- pattern, but it seems like the best option to me. If one
-thinks of LNT as trying to be a general purpose product, then the idea of
-creating tables is notionally related to the specialization (instantiation) of
-the general purpose product for one's particular test suites.
-
-However, we certainly also want to limit the degree to which we create or modify
-tables. Having the test submission mechanism having to modify the table any time
-a user reported a new key would certainly be superflous.
-
-Thus, my current plan is to follow what Karwin calls the "Semistructured Data"
-pattern. What we will do is add an arbitrary blob field (to be JSON or perhaps
-BSON data). We will basically expect that any fields that are required (or
-almost always used) to be put in the actual table schema, but any time we see
-additional fields we can handle them by just placing them in the BLOB field.
-
-We will probably allow users to migrate fields to and from the schema. This
-gives us a good amount of flexibility (and an easy path to eliminate the JSON
-field if need be). We may require users to do this before they can do anything
-but just see the data associated with a run. For example, if they want to use
-one of the reported fields as an axis.
-
-Secondary Purpose
-+++++++++++++++++
-
-One additional painful part of the current schema design is that we use separate
-tests to represent the status aspect of other tests. This is nice and flexible,
-but makes the UI code very painful. Especially, some things like making a graph
-of the test values for all tests which passed become incredibly complex.
-
-The plan is to handle this problem by also constructing the Sample tables on the
-fly, and allowing the test suite to define the keys that go into a sample. Thus,
-any one sample will reflect *all* of the statistics that were reported for that
-test.
-
-This has many advantages:
- * We can start using types for the columns (e.g., easy to start reporting hash
-   of produced binaries, for example).
- * The performance impact of adding new sample values should be much lower than
-   in the old schema.
- * The database explicitly models the fact that sample values were produced from
-   a single run, whereas before sample values and status could not technically
-   be correlated correctly.
- * We eliminate the need to mangle subtest/test result key information into the
-   test name, which is a big win.
-
-The has some disadvantages, however:
- * Poorly models suites where different tests reported different test results.
- * Poorly models situations where we want to support a large number of test
-   results and that set might change frequently or dynamically (e.g., suppose we
-   reported one test for each LLVM Statistic counter).
-
-However, at least for our current usage this scheme should work well enough and
-be **substantially** faster than the old scheme.
-
-This will probably mean that we have to do a bit of work (similar to what we had
-to do for parameter sets) to handle what the UI for this should look
-like. However, we should have better infrastructure for defining how the UI
-should handle things in the metadata.
-
-Other Antipatterns In Use
-+++++++++++++++++++++++++
-
-The "status kind" field uses and suffers from a view of the problems mentioned
-in Chapter 11. 31 Flavors. It would probably be good to move being foreign key
-references into an auxiliary table. This also reduces some of my reservations
-about making that field required / part of every test.
-
-Conveniently, this can also be done without actually changing the status kind
-values, which makes migration easy.
-
-Proposed Concepts
-~~~~~~~~~~~~~~~~~
-
-Test Suite
-++++++++++
-
-The major high level concept in the new schema is that of a test suite. This is
-designed to correspond to some group of tests which users would browse
-independently. Examples would be things like "LLVM Test Suite" or "PlumHall" or
-"GCC Test Suite".
-
-The test suite is the place that defines information about what is being tested
-and the metadata on what information is reported by the runs and the tests.
-
-Parameter Sets
-++++++++++++++
-
-This concept will be removed. Instead, the idea is that all the information
-about how a test was run lives at the Run level. This corresponds much more to
-how LNT is currently primarily used in production. Although there were tests
-like the Clang tests which made use of parameter sets, the theory is that we
-should only have one place for parameters, and a lot of them have to be in the
-Run. The goal is that the UI will be enhanced to better support situations when
-one group of tests was split up into multiple Runs. We should also eventually
-support submitting multiple runs in one submission.
-
-Proposed Migration Path
-~~~~~~~~~~~~~~~~~~~~~~~
-
-I would prefer to not do any coordinated changes to the non-DB side of things
-while effecting the database changes (and generally, I like the test submission
-format to be fairly stable).
-
-For the most part, I think this can be done relatively easily, but there are a
-few places that will require special care.
-
- - For parameter sets (TestInfo), we just discard them and reject any attempts
-   to use them.
-
- - For MachineInfo, we just turn them into the Parameters BLOB or put them in
-   the appropriate column.
-
- - For RunInfo, we just turn them into the Parameters BLOB or put them in the
-   appropriate column.
-
-   We will need to extract the run_order value and put it into the order table.
-
-   This points out that we probably want the order table to be UNIQUE across all
-   entries. Can we do that in SQLite3?
-
- - For the sample status field, we will need to convert the existing format,
-   which encodes samples via multiple tests, into the new format.
-
-   This is the one area where I really don't want to change the test data
-   serialization format, so maybe this is even the right long term approach.
-
-   In the new model of collapsing samples into a single row, this is going to
-   mean that we will need to assume that tests mangled into subtest names are
-   specified in the same order.
-
-Unaddressed Issues
-~~~~~~~~~~~~~~~~~~
-
-There are couple design problems in the current system which I *am not*
-intending to address as part of the v0.4 changes.
-
-Machine Naming
-++++++++++++++
-
-LNT currently allows a "name" for machines, which is very arbitrary. It would be
-nice to eliminate this field completely, but we should probably eliminate the
-name from the UI completely first, and make sure that is workable.

docs/tools.rst

``lnt checkformat [<file>]``
 
     If run without arguments, this expects to read the input file from ``stdin``.
 
   ``lnt convert <input path> [<output path>]``
     Convert between LNT test report formats. By default, this will convert to
     the property list format. You can use ``-`` for either the input (to read
     from ``stdin) or the output (to write to ``stdout``).
 
   ``lnt importreport <input path> [<output path>]``
     Convert text based key value pairs into a LNT json report file.
 
   ``lnt submit [--commit] <server url> <file>+``
     Submits one or more files to the given server. The ``<server url>`` should
     be the url to the actual ``submitRun`` page on the server; the database
     being submitted to is effectively a part of this URL.
 
     By default, this only submits the report to the server but does not actually