[LNT] Some more docs for the LNT json file format.

[kristof.beyls]

Summary

A few people recently have asked me how to be able to quickly test out LNT on their own data.
For most of them, the quickest way to do so is to create a script to translate their data format to the JSON file to be submitted to the LNT server. The quickest way to do so is probably just seeing an example and a simple description of what the json file structure is, rather than use the APIs in lnt.testing.

After the evaluation of the LNT server and web-ui shows it's a useful tool, the recommended way to produce lnt json files remains using the lnt.testing api.

[jgreenhalgh]

docs/concepts.rst

63–65 ↗

(On Diff #53530)

Or use the lnt import command line tool.

docs/importing_data.rst

17 ↗

(On Diff #53530)

I would prefer a formal specification alongside this example data. As you'll see from the rest of the review - using just the sample text leaves some things ambiguous.

25–26 ↗

(On Diff #53530)

It is worth pointing out that the date *must* be in "%Y-%m-%d %H:%M:%S" format. I tripped up on this trying to use dates in the format in which I had recorded them.

29 ↗

(On Diff #53530)

You could mention that arbitrary String->String fields can be added here, and will be displayed in the Run Info in the interface.

52 ↗

(On Diff #53530)

What is the difference between adding a seperate test with the same name, and adding multiple data items to "Data"?

[cmatthews]

Kristof, thank you for writing this! We need to do this kind of stuff more for LNT! I have sent people raw json files and said "make it like that" way too many times!!

docs/concepts.rst

63–65 ↗

(On Diff #53530)

An example with curl would be handy.

docs/importing_data.rst

29 ↗

(On Diff #53530)

This should be emphasized, we use lots of extra fields and they are very handy.

Since the the data is parsed duing the render, you can use it in the templates, so it is essentially a 1 line change to render extra run and machine data. We locally customize LNT like this all the time.

52 ↗

(On Diff #53530)

I have never tested that, but I have a feeling it would not matter, either way a series of samples would be created linking back to the benchmark name.

[cmatthews]

On this topic, one thing I thought would be handy is a LNT command line interface for doing this. Something where it would be simple enough to invoke from a shell script. Not like this, but I was thinking:

lnt submitdata http://foo --machine bar --order 1234 --data nts.foo.compile=10 --data nts.foo.exec=20 --data nts.bar.exec=30

I'd use it all the time if it existed. There are many places where we generate data but don't put it into LNT because of the overhead of submission.

[kristof.beyls]

In D19053#399954, @cmatthews wrote:

On this topic, one thing I thought would be handy is a LNT command line interface for doing this. Something where it would be simple enough to invoke from a shell script. Not like this, but I was thinking:

lnt submitdata http://foo --machine bar --order 1234 --data nts.foo.compile=10 --data nts.foo.exec=20 --data nts.bar.exec=30

I'd use it all the time if it existed. There are many places where we generate data but don't put it into LNT because of the overhead of submission.

I haven't seen demand for this (yet) - but am not opposed to having the feature. Thanks for sharing your thoughts!

[kristof.beyls]

docs/concepts.rst

63–65 ↗

(On Diff #53530)

I agree. I'm afraid I have never used the HTTP POST method to submit runs before. I just tried using curl, and it wasn't immediately obvious to me which parameters to specify - so I'm not going to write documentation about this right now.

lnt/trunk/docs/concepts.rst

 The Nightly Test Suite (NTS) (which is run far more often than nightly now), 
 collects 6 metrics per test: compile time, compile status, execution time, execution
 status, score and size.  The Compile (compile) Test Suite, is focused on metrics
 for compile quality: wall, system and user compile time, compile memory usage
 and code size.  Other test suites can be added to LNT if these sets of metrics
 don't mactch your needs.
 
 Any program can submit results data to LNT, and specify any test suite.  The
-data format is a simple JSON file, and that file needs to be HTTP POSTed to the
-submitRun URL.
+data format is a simple JSON file, and that file needs to be submitted to the
+server using either the lnt import or submit commands, see :ref:`tools`, or
+HTTP POSTed to the submitRun URL.
 
 The most common program to submit data to LNT is the LNT client application
 itself.  The ``lnt runtest nt`` command can run the LLVM test suite, and submit
 data under the NTS test suite. Likewise the ``lnt runtest compile`` command
 can run a set of compile time benchmarks and submit to the Compile test suite.
-
-Given how simple it is to make your own results and send them to LNT,
-it is common to not use the LNT client application at all, and just have a
-custom script run your tests and submit the data to the LNT server. Details
-on how to do this are in :mod:`lnt.testing`

lnt/trunk/docs/contents.rst

.. toctree::
 
    quickstart
 
    tools
 
    tests
 
    changes
-   
    concepts
 
+   importing_data
+
    developer_guide
 
    todo
 
 Indices and tables
 ==================
 
 * :ref:`genindex`

lnt/trunk/docs/importing_data.rst

+.. _importing_data:
+
+Importing Data from Other Test Systems
+======================================
+
+First, make sure you've understood the underlying :ref:`concepts` used by LNT.
+
+Given how simple it is to make your own results and send them to LNT,
+it is common to not use the LNT client application at all, and just have a
+custom script run your tests and submit the data to the LNT server. Details
+on how to do this are in :mod:`lnt.testing`
+
+If for some reason you prefer to generate the json file more directly, the
+current format looks like below. It remains recommended to use the APIs in
+:mod:`lnt.testing` to be better protected against future changes to the json
+format::
+
+  {
+     "Machine": {
+        "Info": {
+          (_String_: _String_)* // optional extra info about the machine.
+        },
+        "Name": _String_ // machine name, mandatory
+     },
+     "Run": {
+        "End Time": "%Y-%m-%d %H:%M:%S", // mandatory
+        "Start Time": "%Y-%m-%d %H:%M:%S", // mandatory
+        "Info": {
+          "run_order": _String_, // mandatory
+          "tag": "nts" // mandatory
+          (_String_: _String_)* // optional extra info about the run.
+        }
+     },
+     "Tests": [
+          {
+              "Data": [ (float+) ],
+              "Info": {},
+              "Name": "nts._ProgramName_._metric_"
+          }+
+      ]
+   }
+
+
+A concrete small example is::
+
+  {
+     "Machine": {
+        "Info": {
+        },
+        "Name": "LNT-AArch64-A53-O3__clang_DEV__aarch64"
+     },
+     "Run": {
+        "End Time": "2016-04-07 14:25:52",
+        "Start Time": "2016-04-07 09:33:48",
+        "Info": {
+          "run_order": "265649",
+          "tag": "nts"
+        }
+     },
+     "Tests": [
+          {
+              "Data": [
+                  0.1056,
+                  0.1055
+              ],
+              "Info": {},
+              "Name": "nts.suite1/program1.exec"
+          },
+          {
+              "Data": [
+                  0.2136
+              ],
+              "Info": {},
+              "Name": "nts.suite2/program1.exec"
+          }
+      ]
+   }
+
+Make sure that:
+ * The Run.Info.tag value is "nts".
+ * The test names always start with "nts.".
+ * The extension of the test name indicate what kind of data is recorded.
+   Currently accepted extensions in the NTS database are:
+
+   * ".exec": represents execution time - a lower number is better.
+   * ".score": represent a benchmark score - a higher number is better.
+   * ".hash": represents a hash of the binary program being executed. This is
+     used to detect if between compiler versions, the generated code has
+     changed.
+   * ".compile": represents the compile time of the program.
+   All of these metrics are optional.