This is an archive of the discontinued LLVM Phabricator instance.

[LNT] Some more docs for the LNT json file format.
ClosedPublic

Authored by kristof.beyls on Apr 13 2016, 3:56 AM.

Details

Summary

A few people recently have asked me how to be able to quickly test out LNT on their own data.
For most of them, the quickest way to do so is to create a script to translate their data format to the JSON file to be submitted to the LNT server. The quickest way to do so is probably just seeing an example and a simple description of what the json file structure is, rather than use the APIs in lnt.testing.

After the evaluation of the LNT server and web-ui shows it's a useful tool, the recommended way to produce lnt json files remains using the lnt.testing api.

Diff Detail

Repository
rL LLVM

Event Timeline

kristof.beyls retitled this revision from to [LNT] Some more docs for the LNT json file format..
kristof.beyls updated this object.
kristof.beyls added a subscriber: llvm-commits.
jgreenhalgh added inline comments.
docs/concepts.rst
63–65 ↗(On Diff #53530)

Or use the lnt import command line tool.

docs/importing_data.rst
17 ↗(On Diff #53530)

I would prefer a formal specification alongside this example data. As you'll see from the rest of the review - using just the sample text leaves some things ambiguous.

25–26 ↗(On Diff #53530)

It is worth pointing out that the date *must* be in "%Y-%m-%d %H:%M:%S" format. I tripped up on this trying to use dates in the format in which I had recorded them.

29 ↗(On Diff #53530)

You could mention that arbitrary String->String fields can be added here, and will be displayed in the Run Info in the interface.

52 ↗(On Diff #53530)

What is the difference between adding a seperate test with the same name, and adding multiple data items to "Data"?

cmatthews accepted this revision.Apr 13 2016, 9:31 AM
cmatthews edited edge metadata.

Kristof, thank you for writing this! We need to do this kind of stuff more for LNT! I have sent people raw json files and said "make it like that" way too many times!!

docs/concepts.rst
63–65 ↗(On Diff #53530)

An example with curl would be handy.

docs/importing_data.rst
29 ↗(On Diff #53530)

This should be emphasized, we use lots of extra fields and they are very handy.

Since the the data is parsed duing the render, you can use it in the templates, so it is essentially a 1 line change to render extra run and machine data. We locally customize LNT like this all the time.

52 ↗(On Diff #53530)

I have never tested that, but I have a feeling it would not matter, either way a series of samples would be created linking back to the benchmark name.

This revision is now accepted and ready to land.Apr 13 2016, 9:31 AM

On this topic, one thing I thought would be handy is a LNT command line interface for doing this. Something where it would be simple enough to invoke from a shell script. Not like this, but I was thinking:

lnt submitdata http://foo --machine bar --order 1234 --data nts.foo.compile=10 --data nts.foo.exec=20 --data nts.bar.exec=30

I'd use it all the time if it existed. There are many places where we generate data but don't put it into LNT because of the overhead of submission.

On this topic, one thing I thought would be handy is a LNT command line interface for doing this. Something where it would be simple enough to invoke from a shell script. Not like this, but I was thinking:

lnt submitdata http://foo --machine bar --order 1234 --data nts.foo.compile=10 --data nts.foo.exec=20 --data nts.bar.exec=30

I'd use it all the time if it existed. There are many places where we generate data but don't put it into LNT because of the overhead of submission.

I haven't seen demand for this (yet) - but am not opposed to having the feature. Thanks for sharing your thoughts!

kristof.beyls added inline comments.Apr 14 2016, 8:23 AM
docs/concepts.rst
63–65 ↗(On Diff #53530)

I agree. I'm afraid I have never used the HTTP POST method to submit runs before. I just tried using curl, and it wasn't immediately obvious to me which parameters to specify - so I'm not going to write documentation about this right now.

This revision was automatically updated to reflect the committed changes.