[LNT] Add support for v2 fmt in test data library

[thopre]

Summary

Add support for generating LNT JSON report file format in version 2 in
the test data creation library. All unit tests introduced in earlier
patch for existing code pass with this new code.

[thopre]

Rebase on top of latest changes

[thopre]

Ping?

[cmatthews]

Looks good.

lnt/testing/__init__.py

52 ↗

(On Diff #213460)

Could you add a comment or assertion text explaining why this check is needed.

211 ↗

(On Diff #213460)

info = dict(self.info)

[thopre]

Add comments explaining version checks

[thopre]

Reduce amount of dictionary update done

[thopre]

lnt/testing/__init__.py

52 ↗

(On Diff #213460)

Are these comments appropriate?

[thopre]

Putting back under review to get confirmation changes made are ok.

[danilaml]

What does this mean for the existing v1 users? Will they continue using v1 format, i.e. will LNT keep using whatever it was using before this patch or will it switch to v2? Btw, is there some place/short summary of the rationale behind v2? Not really related to this patch per se, just want to understand the implication of the new format for the downstream users.

[thopre]

In D65751#1634999, @danilaml wrote:

What does this mean for the existing v1 users? Will they continue using v1 format, i.e. will LNT keep using whatever it was using before this patch or will it switch to v2? Btw, is there some place/short summary of the rationale behind v2? Not really related to this patch per se, just want to understand the implication of the new format for the downstream users.

Yes existing users will still use the v1 format. The API is just extended to be able create a v2 format. As to the rationale I guess https://reviews.llvm.org/D34584 should contain some info.

[danilaml]

I see. It also looks like "run_order" was switched to llvm_project_revision for some reason (I couldn't quite get why - is it simply because the old schema used it? Seems not worth it even if it's hard to correct it in LNT), but if it won't change the behavior (i.e. produced reports) for existing LNT users then it's fine. There seems to be some confusion around what run order is supposed to be already.

[lebedev.ri]

In D65751#1635200, @danilaml wrote:

I see. It also looks like "run_order" was switched to llvm_project_revision for some reason (I couldn't quite get why - is it simply because the old schema used it? Seems not worth it even if it's hard to correct it in LNT), but if it won't change the behavior (i.e. produced reports) for existing LNT users then it's fine. There seems to be some confusion around what run order is supposed to be already.

+1, i'm late to the party, but this part seems really unfortunate.

[thopre]

In D65751#1635200, @danilaml wrote:

I see. It also looks like "run_order" was switched to llvm_project_revision for some reason (I couldn't quite get why - is it simply because the old schema used it? Seems not worth it even if it's hard to correct it in LNT), but if it won't change the behavior (i.e. produced reports) for existing LNT users then it's fine. There seems to be some confusion around what run order is supposed to be already.

I agree that run_order is a nicer name since one might want to use lnt for something else than llvm. But that ought to be done as a separate patch, this just allows producer to generate a v2 format.

[thopre]

In D65751#1633869, @thopre wrote:

Putting back under review to get confirmation changes made are ok.

Ping?

[thopre]

lnt/testing/__init__.py

52 ↗

(On Diff #213460)

Ping?

[cmatthews]

lnt/testing/__init__.py

52 ↗

(On Diff #213460)

assert self.report_version <= 2, "This library only supports v2 and newer."

When someone hits an assertion, it is nice to see the assertion message, instead of having to navigate to the code.

[thopre]

Add failure message to asserts

[thopre]

lnt/testing/__init__.py

52 ↗

(On Diff #213460)

How about now?

[cmatthews]

Thanks!

[thopre]

Make code more python3-compatible

lnt/trunk/lnt/testing/__init__.py

 
 
 class Report:
     """Information on a single testing run.
 
     In the LNT test model, every test run should define exactly one
     machine and run, and any number of test samples.
     """
-    def __init__(self, machine, run, tests):
+    def __init__(self, machine, run, tests, report_version=1):
+        """Construct a LNT report file format in the given format version."""
         self.machine = machine
         self.run = run
         self.tests = list(tests)
-        self.report_version = '1'
+        self.report_version = report_version
         self.check()
 
     def check(self):
         """Check that object members are adequate to generate an LNT
-        json report file when rendering that instance.
+        json report file of the version specified at construction when
+        rendering that instance.
         """
-        assert isinstance(self.machine, Machine)
-        assert isinstance(self.run, Run)
+        # Check requested report version is supported by this library
+        assert self.report_version <= 2, "Only v2 or older LNT report format supported."
+
+        assert isinstance(self.machine, Machine), "Unexpected type for machine."
+        assert (
+            self.machine.report_version == self.report_version
+        ), "Mismatch between machine and report version."
+
+        assert isinstance(self.run, Run), "Unexpected type for run."
+        assert (
+            self.run.report_version == self.report_version
+        ), "Mismatch between run and report version."
+
         for t in self.tests:
-            assert isinstance(t, TestSamples)
+            if self.report_version == 2:
+                assert isinstance(t, Test), "Unexpected type for test"
+                assert (
+                    t.report_version == self.report_version
+                ), "Mismatch between test and report version."
+            else:
+                assert isinstance(t, TestSamples), "Unexpected type for test samples."
 
-    def update_report(self, new_samples):
+    def update_report(self, new_tests_samples, end_time=None):
         """Add extra samples to this report, and update the end time of
         the run.
         """
         self.check()
-        self.tests.extend(new_samples)
-        self.run.update_endtime()
+        self.tests.extend(new_tests_samples)
+        self.run.update_endtime(end_time)
         self.check()
 
     def render(self, indent=4):
-        """Return a LNT json report file format as a string, where each
-        object is indented by indent spaces compared to its parent.
+        """Return a LNT json report file format of the version specified
+        at construction as a string, where each object is indented by
+        indent spaces compared to its parent.
         """
         # Note that we specifically override the encoding to avoid the
         # possibility of encoding errors. Clients which care about the
         # text encoding should supply unicode string objects.
+        if self.report_version == 2:
+            return json.dumps({'format_version': str(self.report_version),
+                               'machine': self.machine.render(),
+                               'run': self.run.render(),
+                               'tests': [t.render() for t in self.tests]},
+                              sort_keys=True, indent=indent, encoding='latin-1')
+        else:
             return json.dumps({'Machine': self.machine.render(),
                                'Run': self.run.render(),
                                'Tests': [t.render() for t in self.tests]},
                               sort_keys=True, indent=indent, encoding='latin-1')
 
 
 class Machine:
     """Information on the machine the test was run on.
 
     The info dictionary can be used to describe additional information
     about the machine, for example the hardware resources or the
     operating environment.
 
     Machines entries in the database are uniqued by their name and the
     entire contents of the info dictionary.
     """
-    def __init__(self, name, info={}):
+    def __init__(self, name, info={}, report_version=1):
         self.name = str(name)
         self.info = dict((str(key), str(value))
                          for key, value in info.items())
+        self.report_version = report_version
+        self.check()
+
+    def check(self):
+        """Check object members are adequate to generate an LNT json
+        report file of the version specified at construction when
+        rendering that instance.
+        """
+        # Check requested version is supported by this library
+        assert (
+            self.report_version <= 2
+        ), "Only v2 or older supported for LNT report format Machine objects."
 
     def render(self):
         """Return info from this instance in a dictionary that respects
-        the LNT report format when printed as json.
+        the LNT report format in the version specified at construction
+        when printed as json.
         """
+        if self.report_version == 2:
+            d = dict(self.info)
+            d['Name'] = self.name
+            return d
+        else:
             return {'Name': self.name,
                     'Info': self.info}
 
 
 class Run:
     """Information on the particular test run.
 
-    The start and end time should always be supplied with the run.
-    Currently, the server uses these to order runs. In the future we
-    will support additional ways to order runs (for example, by a source
-    revision).
+    At least one parameter must be supplied and is used as ordering
+    among several runs. When generating a report in format 1 or earlier,
+    both start_time and end_time are used for that effect and the
+    current date is used if their value is None.
 
     As with Machine, the info dictionary can be used to describe
     additional information on the run. This dictionary should be used to
     describe information on the software-under-test that is constant
     across the test run, for example the revision number being tested.
     It can also be used to describe information about the current state
     which could be useful in analysis, for example the current machine
     load.
     """
-    def __init__(self, start_time, end_time, info={}):
+    def __init__(self, start_time=None, end_time=None, info={}, report_version=1):
+        if report_version <= 1:
             if start_time is None:
                 start_time = datetime.datetime.utcnow()
             if end_time is None:
                 end_time = datetime.datetime.utcnow()
-
-        self.start_time = normalize_time(start_time)
-        self.end_time = normalize_time(end_time)
+        self.start_time = normalize_time(start_time) if start_time is not None else None
+        self.end_time = normalize_time(end_time) if end_time is not None else None
         self.info = dict()
         # Convert keys/values that are not json encodable to strings.
         for key, value in info.items():
             key = str(key)
             value = str(value)
             self.info[key] = value
-        if '__report_version__' in self.info:
+        self.report_version = report_version
+        if self.report_version <= 1:
+            if 'tag' not in self.info:
+                raise ValueError("Missing 'tag' entry in 'info' dictionary")
+            if 'run_order' not in self.info:
+                raise ValueError("Missing 'run_order' entry in 'info' dictionary")
+        else:
+            if 'llvm_project_revision' not in self.info:
+                raise ValueError("Missing 'llvm_project_revision' entry in 'info' dictionary")
+        if '__report_version__' in info:
             raise ValueError("'__report_version__' key is reserved")
-        # TODO: Convert to version 2
+        if report_version == 1:
             self.info['__report_version__'] = '1'
+        self.check()
+
+    def check(self):
+        """Check object members are adequate to generate an LNT json
+        report file of the version specified at construction when
+        rendering that instance.
+        """
+        # Check requested version is supported by this library
+        assert (
+            self.report_version <= 2
+        ), "Only v2 or older supported for LNT report format Run objects."
+        if self.start_time is None and self.end_time is None and not bool(self.info):
+            raise ValueError("No data defined in this Run")
 
     def update_endtime(self, end_time=None):
         """Update the end time of this run."""
-        if end_time is None:
+        if self.report_version <= 1 and end_time is None:
             end_time = datetime.datetime.utcnow()
-        self.end_time = normalize_time(end_time)
+        self.end_time = normalize_time(end_time) if end_time else None
+        self.check()
 
     def render(self):
         """Return info from this instance in a dictionary that respects
-        the LNT report format when printed as json.
+        the LNT report format in the version specified at construction
+        when printed as json.
         """
+        if self.report_version == 2:
+            d = dict(self.info)
+            if self.start_time is not None:
+                d['start_time'] = self.start_time
+            if self.end_time is not None:
+                d['end_time'] = self.end_time
+            return d
+        else:
+            info = dict(self.info)
+            if self.report_version == 1:
+                info['__report_version__'] = '1'
             return {'Start Time': self.start_time,
                     'End Time': self.end_time,
-                'Info': self.info}
+                    'Info': info}
+
+
+class Test:
+    """Information on a particular test in the run and its associated
+    samples.
+
+    The server automatically creates test database objects whenever a
+    new test name is seen. Test should be used to generate report in
+    version 2 or later of LNT JSON report file format.
+
+    Test names are intended to be a persistent, recognizable identifier
+    for what is being executed. Currently, most formats use some form of
+    dotted notation for the test name, and this may become enshrined in
+    the format in the future. In general, the test names should be
+    independent of the software-under-test and refer to some known
+    quantity, for example the software under test. For example,
+    'CINT2006.403_gcc' is a meaningful test name.
+
+    The test info dictionary is intended to hold information on the
+    particular permutation of the test that was run. This might include
+    variables specific to the software-under-test . This could include,
+    for example, the compile flags the test was built with, or the
+    runtime parameters that were used. As a general rule, if two test
+    samples are meaningfully and directly comparable, then they should
+    have the same test name but different info paramaters.
+    """
+
+    def __init__(self, name, samples, info={}, report_version=2):
+        self.name = name
+        self.samples = samples
+        self.info = dict()
+        # Convert keys/values that are not json encodable to strings.
+        for key, value in list(info.items()):
+            key = str(key)
+            value = str(value)
+            self.info[key] = value
+        self.report_version = report_version
+        self.check()
+
+    def check(self):
+        """Check object members are adequate to generate an LNT json
+        report file of the version specified at construction when
+        rendering that instance.
+        """
+        # Check requested version is supported by this library and is
+        # valid for this object.
+        assert (
+            self.report_version == 2
+        ), "Only v2 supported for LNT report format Test objects."
+        for s in self.samples:
+            assert isinstance(s, MetricSamples), "Unexpected type for metric sample."
+            assert (
+                s.report_version == self.report_version
+            ), "Mismatch between test and metric samples."
+
+    def render(self):
+        """Return info from this instance in a dictionary that respects
+        the LNT report format in the version specified at construction
+        when printed as json.
+        """
+        d = dict(self.info)
+        d.update([s.render().popitem() for s in self.samples])
+        d['Name'] = self.name
+        return d
 
 
 class TestSamples:
-    """Test sample data.
+    """Information on a given test and its associated samples data.
 
-    The test sample data defines both the tests that were run and their
-    values. The server automatically creates test database objects
-    whenever a new test name is seen.
+    Samples data must all relate to the same metric. When several
+    metrics are available for a given test, the convention is to have
+    one TestSamples per metric and to encode the metric into the name,
+    e.g. Benchmark1.exec. The server automatically creates test database
+    objects whenever a new test name is seen. TestSamples should only be
+    used to generate report in version 1 or earlier of LNT JSON report
+    file format.
 
     Test names are intended to be a persistent, recognizable identifier
     for what is being executed. Currently, most formats use some form of
     dotted notation for the test name, and this may become enshrined in
     the format in the future. In general, the test names should be
     independent of the software-under-test and refer to some known
     quantity, for example the software under test. For example,
     'CINT2006.403_gcc' is a meaningful test name.
def __init__(self, name, data, info={}, conv_f=float):
         """
         self.name = str(name)
         self.info = dict((str(key), str(value))
                          for key, value in info.items())
         self.data = list(map(conv_f, data))
 
     def render(self):
         """Return info from this instance in a dictionary that respects
-        the LNT report format when printed as json.
+        the LNT report format in the version specified at construction
+        when printed as json.
         """
         return {'Name': self.name,
                 'Info': self.info,
                 'Data': self.data}
 
     def __repr__(self):
         # TODO remove this
         return "TestSample({}): {} - {}".format(self.name,
                                                 self.data,
                                                 self.info)
 
 
+class MetricSamples:
+    """Samples data for a given metric of a given test.
+
+    An arbitrary number of samples for a given metric is allowed for
+    situations where the same metric is obtained several time for a
+    given test to gather statistical data.
+
+    MetricSamples should be used to generate report in version 2 or
+    later of LNT JSON report file format.
+    """
+
+    def __init__(self, metric, data, conv_f=float, report_version=2):
+        self.metric = str(metric)
+        self.data = list(map(conv_f, data))
+        self.report_version = report_version
+        self.check()
+
+    def check(self):
+        """Check object members are adequate to generate an LNT json
+        report file of the version specified at construction when
+        rendering that instance.
+        """
+        # Check requested version is supported by this library and is
+        # valid for this object.
+        assert (
+            self.report_version == 2
+        ), "Only v2 supported for LNT report format MetricSamples objects."
+
+    def add_samples(self, new_samples, conv_f=float):
+        """Add samples for this metric, converted to float by calling
+        function conv_f.
+        """
+        self.data.extend(list(map(conv_f, new_samples)))
+
+    def render(self):
+        """Return info from this instance in a dictionary that respects
+        the LNT report format in the version specified at construction
+        when printed as json.
+        """
+        return {self.metric: self.data if len(self.data) > 1 else self.data[0]}
+
+
 ###
 # Format Versioning
 
 # We record information on the report "version" to allow the server to support
 # some level of auto-upgrading data from submissions of older reports.
 #
 # We recorder the report version as a reserved key in the run information
 # (primarily so that it can be accessed post-import on the server).

lnt/trunk/tests/testing/TestingTest.py

 # RUN: python %s
 
 import unittest
 import logging
 import copy
 import re
 import sys
 from datetime import datetime
-from lnt.testing import TestSamples, Run, Machine, Report
+from lnt.testing import MetricSamples, Test, TestSamples, Run, Machine, Report
 
 logging.basicConfig(level=logging.DEBUG)
 
 
+class MetricSamplesTest(unittest.TestCase):
+    def setUp(self):
+        self.samples_list_float = MetricSamples('execution_time', [21.4, 3.2])
+
+    def test_constructor(self):
+        # Check implicit float conversion from string.
+        samples_list_float_str = MetricSamples('execution_time',
+                                               ['6.7', '30.4'])
+        self.assertEqual(samples_list_float_str.metric, 'execution_time')
+        self.assertListEqual(samples_list_float_str.data, [6.7, 30.4])
+        self.assertEqual(samples_list_float_str.report_version, 2)
+
+        # Check explicit float conversion.
+        float_samples_list_float_str = MetricSamples('execution_time',
+                                                     ['4.7', '32.4'], float)
+        self.assertEqual(float_samples_list_float_str.metric, 'execution_time')
+        self.assertListEqual(float_samples_list_float_str.data, [4.7, 32.4])
+        self.assertEqual(float_samples_list_float_str.report_version, 2)
+
+        # Check nop implicit float conversion from float.
+        self.assertEqual(self.samples_list_float.metric, 'execution_time')
+        self.assertListEqual(self.samples_list_float.data, [21.4, 3.2])
+        self.assertEqual(self.samples_list_float.report_version, 2)
+
+        # Check implicit float conversion from integer.
+        samples_list_int = MetricSamples('execution_time', [6, 11])
+        self.assertEqual(samples_list_int.metric, 'execution_time')
+        self.assertIsInstance(samples_list_int.data[0], float)
+        self.assertIsInstance(samples_list_int.data[1], float)
+        self.assertListEqual(samples_list_int.data, [6.0, 11.0])
+        self.assertEqual(samples_list_int.report_version, 2)
+
+        # Check non-float conversion from float.
+        int_samples_list_float = MetricSamples('execution_time', [21.4, 3.2],
+                                               int)
+        self.assertEqual(int_samples_list_float.metric, 'execution_time')
+        self.assertIsInstance(int_samples_list_float.data[0], int)
+        self.assertIsInstance(int_samples_list_float.data[1], int)
+        self.assertListEqual(int_samples_list_float.data, [21, 3])
+        self.assertEqual(int_samples_list_float.report_version, 2)
+
+        # Check non-float conversion from same input type.
+        int_samples_list_int = MetricSamples('execution_time', [6, 11], int)
+        self.assertEqual(int_samples_list_int.metric, 'execution_time')
+        self.assertIsInstance(int_samples_list_int.data[0], int)
+        self.assertIsInstance(int_samples_list_int.data[1], int)
+        self.assertListEqual(int_samples_list_int.data, [6, 11])
+        self.assertEqual(int_samples_list_int.report_version, 2)
+
+        # Check explicit version.
+        samples_list_float_version = MetricSamples('execution_time',
+                                                   [22.4, 5.2],
+                                                   report_version=2)
+        self.assertEqual(samples_list_float_version.metric, 'execution_time')
+        self.assertListEqual(samples_list_float_version.data, [22.4, 5.2])
+        self.assertEqual(samples_list_float_version.report_version, 2)
+
+        # Check call to check().
+        self.assertRaises(AssertionError, MetricSamples, 'execution_time',
+                          [22.4, 5.2], report_version=1)
+
+    def test_check(self):
+        # Check valid instance.
+        self.samples_list_float.report_version = 2
+        self.samples_list_float.check()
+
+        # Check too small version.
+        self.samples_list_float.report_version = 1
+        self.assertRaises(AssertionError, self.samples_list_float.check)
+
+        # Check too big version.
+        self.samples_list_float.report_version = 3
+        self.assertRaises(AssertionError, self.samples_list_float.check)
+
+    def test_add_samples(self):
+        # Check nop implicit float conversion from float.
+        self.samples_list_float.add_samples([9.9])
+        self.assertListEqual(self.samples_list_float.data, [21.4, 3.2, 9.9])
+
+        # Check implicit float conversion from string.
+        self.samples_list_float.add_samples(['4.4'])
+        self.assertListEqual(self.samples_list_float.data,
+                             [21.4, 3.2, 9.9, 4.4])
+
+        # Check explicit float conversion from integer.
+        self.samples_list_float.add_samples([2])
+        self.assertIsInstance(self.samples_list_float.data[-1], float)
+        self.assertListEqual(self.samples_list_float.data,
+                             [21.4, 3.2, 9.9, 4.4, 2.0])
+
+        # Check int conversion from float.
+        self.samples_list_float.add_samples([11.6], int)
+        self.assertListEqual(self.samples_list_float.data,
+                             [21.4, 3.2, 9.9, 4.4, 2.0, 11])
+
+    def test_render(self):
+        # Check rendering with several samples.
+        self.assertDictEqual(self.samples_list_float.render(),
+                             dict(execution_time=[21.4, 3.2]))
+
+        # Check rendering with a single sample.
+        samples_list_one_float = MetricSamples('execution_time', [7.3])
+        self.assertDictEqual(samples_list_one_float.render(),
+                             dict(execution_time=7.3))
+
+
+class TestTest(unittest.TestCase):
+    def setUp(self):
+        self.samples = [MetricSamples('execution_time', [21.4, 3.2])]
+        self.test_noinfo = Test('Test1', self.samples)
+        self.test_info = Test('Test2', self.samples, {'nb_files': 2})
+
+    def test_constructor(self):
+        # Check default version, no extra info.
+        self.assertEqual(self.test_noinfo.name, 'Test1')
+        self.assertListEqual(self.test_noinfo.samples, self.samples)
+        self.assertDictEqual(self.test_noinfo.info, dict())
+        self.assertEqual(self.test_noinfo.report_version, 2)
+
+        # Check default version, extra info.
+        self.assertEqual(self.test_info.name, 'Test2')
+        self.assertListEqual(self.test_info.samples, self.samples)
+        self.assertDictEqual(self.test_info.info, dict(nb_files='2'))
+        self.assertEqual(self.test_info.report_version, 2)
+
+        # Check explicit version, no extra info.
+        test_noinfo_version = Test('Test3', self.samples, report_version=2)
+        self.assertListEqual(test_noinfo_version.samples, self.samples)
+        self.assertDictEqual(test_noinfo_version.info, dict())
+        self.assertEqual(test_noinfo_version.report_version, 2)
+
+        # Check call to check().
+        self.assertRaises(AssertionError, Test, 'Test4', self.samples,
+                          report_version=1)
+
+    def test_check(self):
+        # Check too small version.
+        self.test_noinfo.report_version = 1
+        self.assertRaises(AssertionError, self.test_noinfo.check)
+
+        # Check too big version.
+        self.test_noinfo.report_version = 3
+        self.assertRaises(AssertionError, self.test_noinfo.check)
+
+        # Check valid instance.
+        self.test_noinfo.report_version = 2
+        self.test_noinfo.check()
+
+        # Check wrong instance for tests.
+        self.test_noinfo.samples = [self.samples[0], 2]
+        self.assertRaises(AssertionError, self.test_noinfo.check)
+
+    def test_render(self):
+        # Check rendering with no info.
+        d1 = {'Name': 'Test1',
+              'execution_time': [21.4, 3.2]}
+        self.assertDictEqual(self.test_noinfo.render(), d1)
+
+        # Check rendering with info.
+        d2 = {'Name': 'Test2',
+              'execution_time': [21.4, 3.2],
+              'nb_files': '2'}
+        self.assertDictEqual(self.test_info.render(), d2)
+
+
 class TestTestSamples(unittest.TestCase):
     def setUp(self):
         self.test_samples_int_list_noinfo = TestSamples('Test1', [1, 2])
         self.test_samples_str_float_list_noinfo = TestSamples('Test3',
                                                               ['2.3', '5.8'])
         self.test_samples_int_list_info = TestSamples('Test7', [1.7, 2.8],
                                                       {'nb_files': 2})
 
def test_render(self):
         d2 = {'Name': 'Test7',
               'Info': {'nb_files': '2'},
               'Data': [1.7, 2.8]}
         self.assertDictEqual(self.test_samples_int_list_info.render(), d2)
 
 
 class TestRun(unittest.TestCase):
     def setUp(self):
-        self.info = {'tag': 'nts', 'run_order': 18246}
-        self.run_float_start = Run(0.0, None, self.info)
+        self.info_v1 = {'tag': 'nts', 'run_order': 18246}
+        self.run_float_start_v1 = Run(0.0, None, self.info_v1)
+        self.run_float_end_v1 = Run(None, 0.0, self.info_v1)
+
+        self.info_v2 = {'llvm_project_revision': 18246}
+        self.run_float_start_v2 = Run(0.0, info=self.info_v2, report_version=2)
+        self.run_float_end_v2 = Run(end_time=0.0, info=self.info_v2,
+                                    report_version=2)
 
     def test_constructor(self):
         info = {'__report_version__': '1',
                 'tag': 'nts',
                 'run_order': '18246'}
 
         # Check time normalization of end time from float.
-        self.assertEqual(self.run_float_start.start_time,
+        self.assertEqual(self.run_float_start_v1.start_time,
                          '1970-01-01 00:00:00')
-        self.assertTrue(self.run_float_start.end_time)
-        self.assertNotEqual(self.run_float_start.end_time,
-                            self.run_float_start.start_time)
-        self.assertTrue(datetime.strptime(self.run_float_start.end_time,
+        self.assertTrue(self.run_float_start_v1.end_time)
+        self.assertNotEqual(self.run_float_start_v1.end_time,
+                            self.run_float_start_v1.start_time)
+        self.assertTrue(datetime.strptime(self.run_float_start_v1.end_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(self.run_float_start.info, info)
+        self.assertDictEqual(self.run_float_start_v1.info, info)
+        self.assertEqual(self.run_float_start_v1.report_version, 1)
 
         # Check time normalization of end time from datetime.
-        run_str_start = Run('2019-07-01 01:02:03', None, info=self.info)
-        self.assertEqual(run_str_start.start_time, '2019-07-01 01:02:03')
-        self.assertTrue(run_str_start.end_time)
-        self.assertNotEqual(run_str_start.end_time, run_str_start.start_time)
-        self.assertTrue(datetime.strptime(run_str_start.end_time,
+        run_str_start_v1 = Run('2019-07-01 01:02:03', None, info=self.info_v1)
+        self.assertEqual(run_str_start_v1.start_time, '2019-07-01 01:02:03')
+        self.assertTrue(run_str_start_v1.end_time)
+        self.assertNotEqual(run_str_start_v1.end_time,
+                            run_str_start_v1.start_time)
+        self.assertTrue(datetime.strptime(run_str_start_v1.end_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(run_str_start.info, info)
+        self.assertDictEqual(run_str_start_v1.info, info)
+        self.assertEqual(run_str_start_v1.report_version, 1)
 
         # Check time normalization of end time from string.
-        run_datetime_start = Run(datetime(2019, 7, 2), None, info=self.info)
-        self.assertEqual(run_datetime_start.start_time, '2019-07-02 00:00:00')
-        self.assertTrue(run_datetime_start.end_time)
-        self.assertNotEqual(run_datetime_start.end_time,
-                            run_datetime_start.start_time)
-        self.assertTrue(datetime.strptime(run_datetime_start.end_time,
+        run_datetime_start_v1 = Run(datetime(2019, 7, 2), None,
+                                    info=self.info_v1)
+        self.assertEqual(run_datetime_start_v1.start_time,
+                         '2019-07-02 00:00:00')
+        self.assertTrue(run_datetime_start_v1.end_time)
+        self.assertNotEqual(run_datetime_start_v1.end_time,
+                            run_datetime_start_v1.start_time)
+        self.assertTrue(datetime.strptime(run_datetime_start_v1.end_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(run_datetime_start.info, info)
+        self.assertDictEqual(run_datetime_start_v1.info, info)
+        self.assertEqual(run_datetime_start_v1.report_version, 1)
 
         # Check time normalization of start time from float.
-        run_float_end = Run(None, 0.0, self.info)
-        self.assertEqual(run_float_end.end_time, '1970-01-01 00:00:00')
-        self.assertTrue(run_float_end.start_time)
-        self.assertNotEqual(run_float_end.start_time, run_float_end.end_time)
-        self.assertTrue(datetime.strptime(run_float_end.start_time,
+        run_float_end_v1 = Run(None, 0.0, self.info_v1)
+        self.assertEqual(run_float_end_v1.end_time, '1970-01-01 00:00:00')
+        self.assertTrue(run_float_end_v1.start_time)
+        self.assertNotEqual(run_float_end_v1.start_time,
+                            run_float_end_v1.end_time)
+        self.assertTrue(datetime.strptime(run_float_end_v1.start_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(run_float_end.info, info)
+        self.assertDictEqual(run_float_end_v1.info, info)
+        self.assertEqual(run_float_end_v1.report_version, 1)
 
         # Check time normalization of start time from datetime.
-        run_str_end = Run(None, '2019-07-01 01:02:03', self.info)
-        self.assertEqual(run_str_end.end_time, '2019-07-01 01:02:03')
-        self.assertTrue(run_str_end.start_time)
-        self.assertNotEqual(run_str_end.start_time, run_str_end.end_time)
-        self.assertTrue(datetime.strptime(run_str_end.start_time,
+        run_str_end_v1 = Run(None, '2019-07-01 01:02:03', self.info_v1)
+        self.assertEqual(run_str_end_v1.end_time, '2019-07-01 01:02:03')
+        self.assertTrue(run_str_end_v1.start_time)
+        self.assertNotEqual(run_str_end_v1.start_time, run_str_end_v1.end_time)
+        self.assertTrue(datetime.strptime(run_str_end_v1.start_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(run_str_end.info, info)
+        self.assertDictEqual(run_str_end_v1.info, info)
+        self.assertEqual(run_str_end_v1.report_version, 1)
 
         # Check time normalization of start time from string.
-        run_datetime_end = Run(None, datetime(2019, 7, 2), self.info)
-        self.assertEqual(run_datetime_end.end_time, '2019-07-02 00:00:00')
-        self.assertTrue(run_datetime_end.start_time)
-        self.assertNotEqual(run_datetime_end.start_time,
-                            run_datetime_end.end_time)
-        self.assertTrue(datetime.strptime(run_datetime_end.start_time,
+        run_datetime_end_v1 = Run(None, datetime(2019, 7, 2), self.info_v1)
+        self.assertEqual(run_datetime_end_v1.end_time, '2019-07-02 00:00:00')
+        self.assertTrue(run_datetime_end_v1.start_time)
+        self.assertNotEqual(run_datetime_end_v1.start_time,
+                            run_datetime_end_v1.end_time)
+        self.assertTrue(datetime.strptime(run_datetime_end_v1.start_time,
                                           '%Y-%m-%d %H:%M:%S'))
-        self.assertDictEqual(run_datetime_end.info, info)
+        self.assertDictEqual(run_datetime_end_v1.info, info)
+        self.assertEqual(run_datetime_end_v1.report_version, 1)
 
         # Check failure when info contains __report_version__ key.
         self.assertRaisesRegexp(ValueError, '__report_version__.*reserved',
                                 Run, None, None, info)
 
+        # Check missing tag entry in info for format version 1.
+        self.assertRaisesRegexp(ValueError,
+                                "Missing 'tag' entry in 'info' dictionary",
+                                Run, info={'run_order': 40385})
+
+        # Check missing run_order entry in info for format version 1.
+        self.assertRaisesRegexp(ValueError,
+                                "Missing 'run_order' entry in 'info'"
+                                " dictionary", Run, info={'tag': 'nts'})
+
+        # Test empty start and end time in format version 2
+        self.assertEqual(self.run_float_start_v2.start_time,
+                         '1970-01-01 00:00:00')
+        self.assertIsNone(self.run_float_start_v2.end_time)
+        self.assertDictEqual(self.run_float_start_v2.info,
+                             {'llvm_project_revision': '18246'})
+        self.assertEqual(self.run_float_start_v2.report_version, 2)
+
+        # Check missing llvm_project_revision entry in info for format
+        # version 2.
+        self.assertRaisesRegexp(ValueError,
+                                "Missing 'llvm_project_revision' entry in"
+                                " 'info' dictionary", Run, 0.0, info={},
+                                report_version=2)
+
+        # Check call to check()
+        self.assertRaises(AssertionError, Run, info=self.info_v2,
+                          report_version=3)
+
+    def test_check(self):
+        # Check valid v1 instance.
+        self.run_float_start_v1.check()
+
+        # Check too big version.
+        self.run_float_start_v2.report_version = 3
+        self.assertRaises(AssertionError, self.run_float_start_v2.check)
+
+        # Check valid v2 instance.
+        self.run_float_start_v2.report_version = 2
+        self.run_float_start_v2.start_time = None
+        self.run_float_start_v2.check()
+
+        # Check no time or info.
+        self.run_float_start_v2.info = {}
+        self.assertRaisesRegexp(ValueError, 'No data defined in this Run',
+                                self.run_float_start_v2.check)
+
     def test_update(self):
         # Check update with a supplied end time.
-        end_time_updated_run_float_start = copy.deepcopy(self.run_float_start)
-        end_time_updated_run_float_start.update_endtime(datetime(2019, 8, 2))
-        self.assertEqual(end_time_updated_run_float_start.end_time,
+        end_time_updated_run_float_start_v1 = (
+            copy.deepcopy(self.run_float_start_v1))
+        end_time_updated_run_float_start_v1.update_endtime(
+            datetime(2019, 8, 2))
+        self.assertEqual(end_time_updated_run_float_start_v1.end_time,
                          '2019-08-02 00:00:00')
 
-        # Check update with default (=now) end time.
-        updated_run_float_start = (
-            copy.deepcopy(end_time_updated_run_float_start))
-        updated_run_float_start.update_endtime()
-        self.assertTrue(updated_run_float_start.end_time)
-        self.assertNotEqual(updated_run_float_start.end_time,
-                            updated_run_float_start.start_time)
-        self.assertNotEqual(updated_run_float_start.end_time,
-                            end_time_updated_run_float_start.end_time)
+        # Check update with default end time in format v1: end time =
+        # now.
+        updated_run_float_start_v1 = (
+            copy.deepcopy(end_time_updated_run_float_start_v1))
+        updated_run_float_start_v1.update_endtime()
+        self.assertTrue(updated_run_float_start_v1.end_time)
+        self.assertNotEqual(updated_run_float_start_v1.end_time,
+                            updated_run_float_start_v1.start_time)
+        self.assertNotEqual(updated_run_float_start_v1.end_time,
+                            end_time_updated_run_float_start_v1.end_time)
+
+        # Check update with default end time in format v2: end time =
+        # None.
+        updated_run_float_end_v2 = copy.deepcopy(self.run_float_end_v2)
+        updated_run_float_end_v2.update_endtime()
+        self.assertEqual(updated_run_float_end_v2.start_time,
+                         updated_run_float_end_v2.start_time)
+        self.assertIsNone(updated_run_float_end_v2.end_time)
 
     def test_render(self):
-        d = {'Start Time': '1970-01-01 00:00:00',
-             'End Time': self.run_float_start.end_time,
+        # Check rendering of format v1.
+        d1 = {'Start Time': '1970-01-01 00:00:00',
+              'End Time': self.run_float_start_v1.end_time,
               'Info': {'__report_version__': '1',
                        'run_order': '18246',
                        'tag': 'nts'}}
-        self.assertDictEqual(self.run_float_start.render(), d)
+        self.assertDictEqual(self.run_float_start_v1.render(), d1)
+
+        # Check rendering of format v2 with no end time.
+        d2 = {'start_time': '1970-01-01 00:00:00',
+              'llvm_project_revision': '18246'}
+        self.assertDictEqual(self.run_float_start_v2.render(), d2)
+
+        # Check rendering of format v2 with no start time.
+        d3 = {'end_time': '1970-01-01 00:00:00',
+              'llvm_project_revision': '18246'}
+        self.assertDictEqual(self.run_float_end_v2.render(), d3)
 
 
 class TestMachine(unittest.TestCase):
     def setUp(self):
-        self.machine_noinfo = Machine('Machine1')
-        self.machine = Machine('Machine2', {'CPUs': 2})
+        self.machine_v1_noinfo = Machine('Machine1')
+        self.machine_v1 = Machine('Machine2', {'CPUs': 2})
+        self.machine_v2 = Machine('Machine3', {'CPUs': 2}, 2)
 
     def test_constructor(self):
-        # Check constructor with no info.
-        self.assertEqual(self.machine_noinfo.name, 'Machine1')
-        self.assertDictEqual(self.machine_noinfo.info, {})
-
-        # Check constructor with info.
-        self.assertEqual(self.machine.name, 'Machine2')
-        self.assertDictEqual(self.machine.info, {'CPUs': '2'})
+        # Check constructor with no info and default version (v1).
+        self.assertEqual(self.machine_v1_noinfo.name, 'Machine1')
+        self.assertDictEqual(self.machine_v1_noinfo.info, {})
+        self.assertEqual(self.machine_v1_noinfo.report_version, 1)
+
+        # Check constructor with info and default version (v1).
+        self.assertEqual(self.machine_v1.name, 'Machine2')
+        self.assertDictEqual(self.machine_v1.info, {'CPUs': '2'})
+        self.assertEqual(self.machine_v1.report_version, 1)
+
+        # Check v2 constructor with info.
+        self.assertEqual(self.machine_v2.name, 'Machine3')
+        self.assertDictEqual(self.machine_v2.info, {'CPUs': '2'})
+        self.assertEqual(self.machine_v2.report_version, 2)
+
+    def test_check(self):
+        # Check valid v1 instance.
+        self.machine_v1.check()
+
+        # Check valid v2 instance.
+        self.machine_v2.check()
+
+        # Check too big version.
+        self.machine_v2.report_version = 3
+        self.assertRaises(AssertionError, self.machine_v2.check)
 
     def test_render(self):
-        # Check rendering with no info.
+        # Check v1 rendering with no info.
         d1 = {'Name': 'Machine1',
               'Info': {}}
-        self.assertDictEqual(self.machine_noinfo.render(), d1)
+        self.assertDictEqual(self.machine_v1_noinfo.render(), d1)
 
-        # Check rendering with info.
+        # Check v1 rendering with info.
         d2 = {'Name': 'Machine2',
               'Info': {'CPUs': '2'}}
-        self.assertDictEqual(self.machine.render(), d2)
+        self.assertDictEqual(self.machine_v1.render(), d2)
+
+        # Check v2 rendering with no info.
+        d3 = {'Name': 'Machine3',
+              'CPUs': '2'}
+        self.assertDictEqual(self.machine_v2.render(), d3)
+
+        # Check v2 rendering with info.
+        self.machine_v2.info = {}
+        d4 = {'Name': 'Machine3'}
+        self.assertDictEqual(self.machine_v2.render(), d4)
 
 
 class TestReport(unittest.TestCase):
     maxDiff = None
 
     def setUp(self):
-        self.machine = Machine('Machine', info={'nb_cpus': 2})
-        self.run = Run(0.0, '1982-01-01 00:00:00', {'tag': 'nts',
+        self.machine_v1 = Machine('Machine', info={'nb_cpus': 2})
+        self.run_v1 = Run(0.0, '1982-01-01 00:00:00', {'tag': 'nts',
                                                        'run_order': 18246})
         self.tests_samples = [TestSamples('Test.exec', [1.7, 2.8],
                                           {'nb_files': 2})]
-        self.report = Report(self.machine, self.run, self.tests_samples)
+        self.report_v1 = Report(self.machine_v1, self.run_v1,
+                                self.tests_samples)
+
+        self.machine_v2 = Machine('Machine', info={'nb_cpus': 2},
+                                  report_version=2)
+        self.run_v2 = Run(0.0, info={'llvm_project_revision': 18246},
+                          report_version=2)
+        samples = MetricSamples('execution_time', [21.4, 3.2])
+        self.tests = [Test('Test', [samples], {'nb_files': 2})]
+        self.report_v2 = Report(self.machine_v2, self.run_v2, self.tests, 2)
 
     def test_constructor(self):
-        # Check successful constructor call.
-        self.assertEqual(self.report.machine, self.machine)
-        self.assertEqual(self.report.run, self.run)
-        self.assertListEqual(self.report.tests, self.tests_samples)
+        # Check successful constructor call with default version.
+        self.assertEqual(self.report_v1.machine, self.machine_v1)
+        self.assertEqual(self.report_v1.run, self.run_v1)
+        self.assertListEqual(self.report_v1.tests, self.tests_samples)
+        self.assertEqual(self.report_v1.report_version, 1)
+
+        # Check successful constructor call with explicit version.
+        self.assertEqual(self.report_v2.machine, self.machine_v2)
+        self.assertEqual(self.report_v2.run, self.run_v2)
+        self.assertListEqual(self.report_v2.tests, self.tests)
+        self.assertEqual(self.report_v2.report_version, 2)
 
         # Check call to check().
-        self.assertRaises(AssertionError, Report, [], self.run,
+        self.assertRaises(AssertionError, Report, [], self.run_v1,
                           self.tests_samples)
 
     def test_check(self):
-        # Check valid report.
-        self.report.check()
+        # Check wrong version.
+        self.report_v2.report_version = 3
+        self.assertRaises(AssertionError, self.report_v2.check)
+
+        # Check valid v2 report.
+        self.report_v2.report_version = 2
+        self.report_v2.check()
 
         # Check type test for machine.
-        report_machine_list = copy.deepcopy(self.report)
+        report_machine_list = copy.deepcopy(self.report_v1)
         report_machine_list.machine = []
         self.assertRaises(AssertionError, report_machine_list.check)
 
+        # Check version mismatch between machine and report.
+        self.report_v1.machine.report_version = 2
+        self.assertRaises(AssertionError, self.report_v1.check)
+
+        # Check valid v1 report.
+        self.report_v1.machine.report_version = 1
+        self.report_v1.check()
+
         # Check type test for run.
-        report_run_list = copy.deepcopy(self.report)
+        report_run_list = copy.deepcopy(self.report_v1)
         report_run_list.run = []
         self.assertRaises(AssertionError, report_run_list.check)
 
-        # Check type test for all tests.
-        report_run_list = copy.deepcopy(self.report)
-        report_tests_int_list = copy.deepcopy(self.report)
-        report_tests_int_list.tests = [2]
-        self.assertRaises(AssertionError, report_tests_int_list.check)
+        # Check version mismatch between run and report.
+        self.report_v1.run.report_version = 2
+        self.assertRaises(AssertionError, self.report_v1.check)
+
+        self.report_v1.run.report_version = 1
+
+        # Check type test for all v1 tests.
+        report_v1_tests_int_list = copy.deepcopy(self.report_v1)
+        report_v1_tests_int_list.tests = [2]
+        self.assertRaises(AssertionError, report_v1_tests_int_list.check)
+
+        # Check type test for all v2 tests.
+        report_v2_tests_int_list = copy.deepcopy(self.report_v2)
+        report_v2_tests_int_list.tests = [2]
+        self.assertRaises(AssertionError, report_v2_tests_int_list.check)
+
+        # Check version mismatch between one of the tests and report.
+        self.report_v2.tests[0].report_version = 1
+        self.assertRaises(AssertionError, self.report_v2.check)
 
     def test_update_report(self):
-        orig_end_time = self.report.run.end_time
+        # Check update with default (=now) end time.
+        orig_end_time = self.report_v1.run.end_time
         new_tests_samples = [TestSamples('Test2.exec', [56.5])]
-        self.report.update_report(new_tests_samples)
+        self.report_v1.update_report(new_tests_samples)
+        self.tests_samples.extend(new_tests_samples)
+        self.assertListEqual(self.report_v1.tests, self.tests_samples)
+        self.assertNotEqual(self.report_v1.run.end_time, orig_end_time)
+
+        # Check update with supplied end time.
+        new_tests_samples = [TestSamples('Test3.exec', [18.3])]
+        self.report_v1.update_report(new_tests_samples, '1990-07-07 00:00:00')
         self.tests_samples.extend(new_tests_samples)
-        self.assertListEqual(self.report.tests, self.tests_samples)
-        self.assertNotEqual(self.report.run.end_time, orig_end_time)
+        self.assertListEqual(self.report_v1.tests, self.tests_samples)
+        self.assertEqual(self.report_v1.run.end_time, '1990-07-07 00:00:00')
 
     def test_render(self):
-        # Check rendering with default indentation.
-        self.assertMultiLineEqual(re.sub(r' +\n', '\n', self.report.render()),
-                                  """\
+        # Check v1 format rendering with default indentation.
+        self.assertMultiLineEqual(re.sub(r' +\n', '\n',
+                                         self.report_v1.render()), """\
 {
     "Machine": {
         "Info": {
             "nb_cpus": "2"
         },
         "Name": "Machine"
     },
     "Run": {
"Tests": [
             "Info": {
                 "nb_files": "2"
             },
             "Name": "Test.exec"
         }
     ]
 }""")
 
-        # Check rendering with supplied indentation.
+        # Check v1 format rendering with supplied indentation.
         self.assertMultiLineEqual(re.sub(r' +\n', '\n',
-                                         self.report.render(indent=2)), """\
+                                         self.report_v1.render(indent=2)), """\
 {
   "Machine": {
     "Info": {
       "nb_cpus": "2"
     },
     "Name": "Machine"
   },
   "Run": {
"Tests": [
       "Info": {
         "nb_files": "2"
       },
       "Name": "Test.exec"
     }
   ]
 }""")
 
+        # Check v2 format rendering with default indentation.
+        self.assertMultiLineEqual(re.sub(r' +\n', '\n',
+                                         self.report_v2.render()), """\
+{
+    "format_version": "2",
+    "machine": {
+        "Name": "Machine",
+        "nb_cpus": "2"
+    },
+    "run": {
+        "llvm_project_revision": "18246",
+        "start_time": "1970-01-01 00:00:00"
+    },
+    "tests": [
+        {
+            "Name": "Test",
+            "execution_time": [
+                21.4,
+                3.2
+            ],
+            "nb_files": "2"
+        }
+    ]
+}""")
+
+        # Check v2 format rendering with supplied indentation.
+        self.assertMultiLineEqual(re.sub(r' +\n', '\n',
+                                         self.report_v2.render(indent=2)), """\
+{
+  "format_version": "2",
+  "machine": {
+    "Name": "Machine",
+    "nb_cpus": "2"
+  },
+  "run": {
+    "llvm_project_revision": "18246",
+    "start_time": "1970-01-01 00:00:00"
+  },
+  "tests": [
+    {
+      "Name": "Test",
+      "execution_time": [
+        21.4,
+        3.2
+      ],
+      "nb_files": "2"
+    }
+  ]
+}""")
+
+        # Check v2 format rendering with single sample for a metric and
+        # default indentation.
+        self.report_v2.tests[0].samples[0].data.pop()
+        self.assertMultiLineEqual(re.sub(r' +\n', '\n',
+                                         self.report_v2.render()), """\
+{
+    "format_version": "2",
+    "machine": {
+        "Name": "Machine",
+        "nb_cpus": "2"
+    },
+    "run": {
+        "llvm_project_revision": "18246",
+        "start_time": "1970-01-01 00:00:00"
+    },
+    "tests": [
+        {
+            "Name": "Test",
+            "execution_time": 21.4,
+            "nb_files": "2"
+        }
+    ]
+}""")
+
 
 if __name__ == '__main__':
     unittest.main(argv=[sys.argv[0], ])