Introduce new JSON import format

[MatzeB]

Summary

This redesigns to get a new simpler import format for LNT. With plans to
use the same format for exporting.

• Go for lowercase keys without spaces.
• Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.
• Drop the renaming where we would use different names for import ("info_key") than inside the lnt database. Just expect the lnt database names.
• Simplify the format by flattening Machine/Info into Machine and Run/Info into Run.

Previously:

"Machine": {
  "name": "mymachine",
  "Info": {
    "hardware": "HAL 9000"
  }
},
"Run": {
  "Start Time": "2017-05-16 21:31:42",
  "End Time": "2017-05-16 21:31:42",
  "Info": {
    "__report_version__": "1",
    "tag": "nts",
    "run_order": "123456"
  }
}

Now:

"format_version": "2",
"machine": {
  "name": "mymachine",
  "hardware": "HAL 9000"
},
"run": {
  "start_time": "2017-05-16 21:31:42",
  "end_time": "2017-05-16 21:31:42",
  "llvm_project_revision": "123456"
}

• Reorganize test data, so we have 1 record for each tests containing all the metrics.
• Drop the "Info" record for tests (the v4db would refuse to take non-empty Info records anyway).
• Again no renaming of test metrics from "info_key" anymore.

Previously:

"Tests": [
   { "Data": [ 7, 42 ],
     "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.compile",
     "Info": {}
   },
   { "Data": [ 13 ],
     "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
     "Info": {}
   },
   { "Data": [ 11 ],
     "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
     "Info": {}
   },
   { "Data": [ "49333a87d501b0aea2191830b66b5eec" ],
     "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.hash",
     "Info": {}
   }
]

Now:

"test": [
    {
        "name": "SingleSource/Benchmarks/Adobe-C++/functionobjects",
        "compile_time": [ 7, 42 ],
        "execution_time": [ 13, 11 ],
        "hash": "49333a87d501b0aea2191830b66b5eec"
    }
]

• Implements upgrade logic for submissions in the old format
• I will submit a separate patch to remove the "info_key" columns from the LNT database.
• I will submit a separate patch to have the runtest tests submit in the new format.
• This has been discussed with Chris Matthews to make sure the new REST export APIs will spit out the same format, so users only need to learn one format and it becomes easier to move data around servers.

[MatzeB]

Add missing file

[kristof.beyls]

Thank you so much for this, Matthias!

Some questions/remarks inline:

This redesigns to get a new simpler import format for LNT. With plans to
use the same format for exporting.

Go for lowercase keys without spaces.

Makes sense to me.

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

Drop the renaming where we would use different names for import ("info_key") than inside the lnt database. Just expect the lnt database names.

Again, I don't understand fully why info_key used to be needed, but this seem a good change to me.

Simplify the format by flattening Machine/Info into Machine and Run/Info into Run.

Looks sensible.

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

}
Reorganize test data, so we have 1 record for each tests containing all the metrics.

Thanks, that will make the json easier to read by a human, when that is needed.

Drop the "Info" record for tests (the v4db would refuse to take non-empty Info records anyway).
Again no renaming of test metrics from "info_key" anymore.
Previously:

"Tests": [

{ "Data": [ 7, 42 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.compile",
  "Info": {}
},
{ "Data": [ 13 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
  "Info": {}
},
{ "Data": [ 11 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
  "Info": {}
},
{ "Data": [ "49333a87d501b0aea2191830b66b5eec" ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.hash",
  "Info": {}
}

]
Now:

"test": [

{
    "name": "SingleSource/Benchmarks/Adobe-C++/functionobjects",
    "compile_time": [ 7, 42 ],
    "execution_time": [ 13, 11 ],
    "hash": "49333a87d501b0aea2191830b66b5eec"
}

]
Implements upgrade logic for submissions in the old format

I guess we can encounter both clients submitting the new format to older servers and older clients submitting the old format to newer servers.
Out of these combinations, I guess we can only support old format submitted to newer servers, not the other way around, right?
If so, it'll be important to make sure the lnt.llvm.org is operating well and all public bots currently submitting data to llvm.org/perf have switched to submitting lnt.llvm.org before this lands as otherwise the bots will start submitting this new format and the very out-of-date LNT server at llvm.org/perf will not be able to cope?
Although that being said, it seems the llvm.org/perf service is down since last weekend's maintenance works on llvm.org, and maybe we just have to switch submitting to lnt.llvm.org anyway.
@cmatthews: Do you think the version of the server at lnt.llvm.org could be upgraded pretty much immediately after this lands to make sure we don't miss performance reports from bots that automatically upgrade to ToT LNT?

I will submit a separate patch to remove the "info_key" columns from the LNT database.
I will submit a separate patch to have the runtest tests submit in the new format.

Ah right, here we can leave some time between the server accepting the new format and the clients starting to submit the new format, to allow servers to be upgraded.

This has been discussed with Chris Matthews to make sure the new REST export APIs will spit out the same format, so users only need to learn one format and it becomes easier to move data around servers.

docs/importing_data.rst

46 ↗

(On Diff #103809)

As said in the long general comment, I'd suggest changing "llvm_project_revision" to "project_revision". Or maybe just "revision"?

109–116 ↗

(On Diff #103809)

I think the documentation of the different kinds of data that is interpreted by LNT should remain in the documentation in one form or another. It is essential information to transform test data into this format.

[MatzeB]

In D34584#791693, @kristof.beyls wrote:

Thank you so much for this, Matthias!

Some questions/remarks inline:

This redesigns to get a new simpler import format for LNT. With plans to
use the same format for exporting.

Go for lowercase keys without spaces.

Makes sense to me.

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

The patch works fine. People using non-nts testsuites (which I have a strong suspicion only exist within apple), may need to supply a -s MySchema flag for the commandline tools or use db_default/v4/MySchema/submitRun instead of db_default/submitRun to express they don't use the 'nts' default.

This is probably not the right patch to discuss LNT database design. But some points anyway:

• IMO making SQL tables too generic (like having one row with MetricType, Test, Run and Value; so we could create new MetricTypes on demand) is a sure way to kill performance. As long as we use SQL we should have 1 table column for each metric.
• A schema is nice when you need to decide bits like the unit of a metric, whether bigger is better or getting a human readable metric name for the website. However we could make good arguments that we only need that on the presentation/analysis side and not as part of the data storage model.

Drop the renaming where we would use different names for import ("info_key") than inside the lnt database. Just expect the lnt database names.

Again, I don't understand fully why info_key used to be needed, but this seem a good change to me.

Same here.

Simplify the format by flattening Machine/Info into Machine and Run/Info into Run.

Looks sensible.

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

I talked a bit with Chris about this. And I think we want to persue a somwhat weaker version scheme were tools add something like "generated": "LNT version 0.4.1" or "generated": "llvm test-suite runner 0.1", similar in spirit to the http user agent header.

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

• The name for the field is determined by the schema in LNT, so not necessarily part of this patch.
• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

}
Reorganize test data, so we have 1 record for each tests containing all the metrics.

Thanks, that will make the json easier to read by a human, when that is needed.

Drop the "Info" record for tests (the v4db would refuse to take non-empty Info records anyway).
Again no renaming of test metrics from "info_key" anymore.
Previously:

"Tests": [

{ "Data": [ 7, 42 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.compile",
  "Info": {}
},
{ "Data": [ 13 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
  "Info": {}
},
{ "Data": [ 11 ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.exec",
  "Info": {}
},
{ "Data": [ "49333a87d501b0aea2191830b66b5eec" ],
  "Name": "nts.SingleSource/Benchmarks/Adobe-C++/functionobjects.hash",
  "Info": {}
}

]
Now:

"test": [

{
    "name": "SingleSource/Benchmarks/Adobe-C++/functionobjects",
    "compile_time": [ 7, 42 ],
    "execution_time": [ 13, 11 ],
    "hash": "49333a87d501b0aea2191830b66b5eec"
}

]
Implements upgrade logic for submissions in the old format

I guess we can encounter both clients submitting the new format to older servers and older clients submitting the old format to newer servers.
Out of these combinations, I guess we can only support old format submitted to newer servers, not the other way around, right?

Yes.

If so, it'll be important to make sure the lnt.llvm.org is operating well and all public bots currently submitting data to llvm.org/perf have switched to submitting lnt.llvm.org before this lands as otherwise the bots will start submitting this new format and the very out-of-date LNT server at llvm.org/perf will not be able to cope?

• We are in luck here: The reporting part haven't changed with this patch so lnt runtest still produces the old format. (I planed to change that separately)
• I hope we have ways to upgrade the llvm.org LNT instance...

Although that being said, it seems the llvm.org/perf service is down since last weekend's maintenance works on llvm.org, and maybe we just have to switch submitting to lnt.llvm.org anyway.
@cmatthews: Do you think the version of the server at lnt.llvm.org could be upgraded pretty much immediately after this lands to make sure we don't miss performance reports from bots that automatically upgrade to ToT LNT?

I wasn't even aware there are two public instances. Is this like the greendragon/buildbot differences?
We are in no hurry to land this; we can probably sync this with upcoming Rest API changes.

I will submit a separate patch to remove the "info_key" columns from the LNT database.
I will submit a separate patch to have the runtest tests submit in the new format.

Ah right, here we can leave some time between the server accepting the new format and the clients starting to submit the new format, to allow servers to be upgraded.

yep.

This has been discussed with Chris Matthews to make sure the new REST export APIs will spit out the same format, so users only need to learn one format and it becomes easier to move data around servers.

[kristof.beyls]

In D34584#792859, @MatzeB wrote:

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

The patch works fine. People using non-nts testsuites (which I have a strong suspicion only exist within apple), may need to supply a -s MySchema flag for the commandline tools or use db_default/v4/MySchema/submitRun instead of db_default/submitRun to express they don't use the 'nts' default.

This is probably not the right patch to discuss LNT database design. But some points anyway:

• IMO making SQL tables too generic (like having one row with MetricType, Test, Run and Value; so we could create new MetricTypes on demand) is a sure way to kill performance. As long as we use SQL we should have 1 table column for each metric.
• A schema is nice when you need to decide bits like the unit of a metric, whether bigger is better or getting a human readable metric name for the website. However we could make good arguments that we only need that on the presentation/analysis side and not as part of the data storage model.

Agreed that not losing performance will be the key challenge for any redesign of the SQL schema to avoid having test-suite-specific tables. Anyway, it's indeed a discussion for another time...

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

I talked a bit with Chris about this. And I think we want to persue a somwhat weaker version scheme were tools add something like "generated": "LNT version 0.4.1" or "generated": "llvm test-suite runner 0.1", similar in spirit to the http user agent header.

I'm not sure I fully get this. I know of at least 2 groups having custom scripts to translate data coming out of their custom benchmarking system into something that can be imported into LNT. I know of yet another group planning to do something similar. I think it won't be possible for LNT to understand all possible producers of json files by name. So, with my limited insight here, I think that just a numerical version number or something similar indicating which version of the specification this json follows would be better than a string naming the producer?
In other words, I think this json format is rapidly becoming one of the key public interfaces of LNT - maybe the single most important one. If LNT keeps on growing in popularity in teams outside of LLVM, I think we'll end up having to specify this interface, and version control of it, a bit more strictly.

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

• The name for the field is determined by the schema in LNT, so not necessarily part of this patch.

OK, I agree it doesn't have to be part of this patch. But taking into account my remark above about the json file rapidly becoming one of the most important interfaces of LNT, I think that it may not be tenable in the long run to have the json fields map exactly to names of fields in the database schema. It is of course a nice to have; but the field names in the json file are on the publicly-visible interface of LNT, and the database field names are an implementation detail from a user point-of-view. I can foresee reasons why both might start deviating.
Since my gut feel is that it will be harder/more work to change the "json format specification" in the future, I thought that maybe it could be part of the change that is going to happen as part of this patch. Anyway, I don't have too strong a feeling about the exact name of this field.

• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

Agreed, it's not the best possible name. It should be the revision of the software for which you're tracking the performance in my mind. That software doesn't have to be a "compiler". Not sure what a better name would be though.

If so, it'll be important to make sure the lnt.llvm.org is operating well and all public bots currently submitting data to llvm.org/perf have switched to submitting lnt.llvm.org before this lands as otherwise the bots will start submitting this new format and the very out-of-date LNT server at llvm.org/perf will not be able to cope?

• We are in luck here: The reporting part haven't changed with this patch so lnt runtest still produces the old format. (I planed to change that separately)
• I hope we have ways to upgrade the llvm.org LNT instance...

Although that being said, it seems the llvm.org/perf service is down since last weekend's maintenance works on llvm.org, and maybe we just have to switch submitting to lnt.llvm.org anyway.

I think we ended up doing this within the past 12 hours...

[MatzeB]

Before I dive in the meta discussion, are there any issues with the patch itself?

In D34584#793766, @kristof.beyls wrote:

In D34584#792859, @MatzeB wrote:

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

The patch works fine. People using non-nts testsuites (which I have a strong suspicion only exist within apple), may need to supply a -s MySchema flag for the commandline tools or use db_default/v4/MySchema/submitRun instead of db_default/submitRun to express they don't use the 'nts' default.

This is probably not the right patch to discuss LNT database design. But some points anyway:

• IMO making SQL tables too generic (like having one row with MetricType, Test, Run and Value; so we could create new MetricTypes on demand) is a sure way to kill performance. As long as we use SQL we should have 1 table column for each metric.
• A schema is nice when you need to decide bits like the unit of a metric, whether bigger is better or getting a human readable metric name for the website. However we could make good arguments that we only need that on the presentation/analysis side and not as part of the data storage model.

Agreed that not losing performance will be the key challenge for any redesign of the SQL schema to avoid having test-suite-specific tables. Anyway, it's indeed a discussion for another time...

Again unrelated to this patch,

Note that LNT is already designed to be extensible. Schemas like 'nts' or 'compile' switch the metrics available or what fields are considered for identifying/ordering runs. It just fails being user friendly so that people would actually create new schemes or modifying existing when they setup new benchmarks. IMO the current database design is good enough and we rather need a patch that makes creating and installing new schemes easy. I imagine a world where testsuite come with lnt.schema.json which describes the names/bigger is better/etc. for a testsuite and then when you setup a server you just reference those schemas in your lnt.cfg and get a server that matches your testsuite.

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

I talked a bit with Chris about this. And I think we want to persue a somwhat weaker version scheme were tools add something like "generated": "LNT version 0.4.1" or "generated": "llvm test-suite runner 0.1", similar in spirit to the http user agent header.

I'm not sure I fully get this. I know of at least 2 groups having custom scripts to translate data coming out of their custom benchmarking system into something that can be imported into LNT. I know of yet another group planning to do something similar. I think it won't be possible for LNT to understand all possible producers of json files by name. So, with my limited insight here, I think that just a numerical version number or something similar indicating which version of the specification this json follows would be better than a string naming the producer?
In other words, I think this json format is rapidly becoming one of the key public interfaces of LNT - maybe the single most important one. If LNT keeps on growing in popularity in teams outside of LLVM, I think we'll end up having to specify this interface, and version control of it, a bit more strictly.

Ok, I'll add a field 'format_version' to the toplevel.

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

• The name for the field is determined by the schema in LNT, so not necessarily part of this patch.

OK, I agree it doesn't have to be part of this patch. But taking into account my remark above about the json file rapidly becoming one of the most important interfaces of LNT, I think that it may not be tenable in the long run to have the json fields map exactly to names of fields in the database schema. It is of course a nice to have; but the field names in the json file are on the publicly-visible interface of LNT, and the database field names are an implementation detail from a user point-of-view. I can foresee reasons why both might start deviating.
Since my gut feel is that it will be harder/more work to change the "json format specification" in the future, I thought that maybe it could be part of the change that is going to happen as part of this patch. Anyway, I don't have too strong a feeling about the exact name of this field.

Yes and no; As see above about my vision of actually using the LNT feature of different benchmark suites instead of pushing everything into NTS :)

• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

Agreed, it's not the best possible name. It should be the revision of the software for which you're tracking the performance in my mind. That software doesn't have to be a "compiler". Not sure what a better name would be though.

I'd be fine changing it to "compiler_revision" right away, if we need more discussion about how to call the field, then I'd suggest doing that in another patch to not delay this patch.

If so, it'll be important to make sure the lnt.llvm.org is operating well and all public bots currently submitting data to llvm.org/perf have switched to submitting lnt.llvm.org before this lands as otherwise the bots will start submitting this new format and the very out-of-date LNT server at llvm.org/perf will not be able to cope?

• We are in luck here: The reporting part haven't changed with this patch so lnt runtest still produces the old format. (I planed to change that separately)
• I hope we have ways to upgrade the llvm.org LNT instance...

Although that being said, it seems the llvm.org/perf service is down since last weekend's maintenance works on llvm.org, and maybe we just have to switch submitting to lnt.llvm.org anyway.

I think we ended up doing this within the past 12 hours...

[MatzeB]

• Rebase to ToT
• Add "format_version": "2" to the toplevel dictionary.

[MatzeB]

Fix typo in docu.

[kristof.beyls]

In D34584#794519, @MatzeB wrote:

Before I dive in the meta discussion, are there any issues with the patch itself?

I'm not up to speed on many parts of LNT that this code touches, but reading through it, I only added some optional nitpicks, apart from 1 issue that I think needs to be addressed before committing:
The patch as is will drop documentation on what field names to use in the json representation, making it impossible for people to write a converter from data from other testing/benchmarking systems to the json-format to import into LNT, without going through a lot of effort trying to reverse engineer the format from source code. I think the documentation of the fields with meaning for "nts" must be retained.
My experience is that most of such users of LNT don't grasp the concept of multiple "test-suite formats" such as "nts" or "compile" in the LNT DB, which is why I didn't bother trying to explain that well in the current documentation.
As you suggest later, maybe the json format should become self-describing (I'm maybe pushing your suggestion further than you intended here), but until that happens, I think the documentation on the nts field names that matter should stay.

Once that documentation issue is addressed, I have no objections for this patch to be committed.

In D34584#793766, @kristof.beyls wrote:

In D34584#792859, @MatzeB wrote:

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

The patch works fine. People using non-nts testsuites (which I have a strong suspicion only exist within apple), may need to supply a -s MySchema flag for the commandline tools or use db_default/v4/MySchema/submitRun instead of db_default/submitRun to express they don't use the 'nts' default.

This is probably not the right patch to discuss LNT database design. But some points anyway:

• IMO making SQL tables too generic (like having one row with MetricType, Test, Run and Value; so we could create new MetricTypes on demand) is a sure way to kill performance. As long as we use SQL we should have 1 table column for each metric.
• A schema is nice when you need to decide bits like the unit of a metric, whether bigger is better or getting a human readable metric name for the website. However we could make good arguments that we only need that on the presentation/analysis side and not as part of the data storage model.

Agreed that not losing performance will be the key challenge for any redesign of the SQL schema to avoid having test-suite-specific tables. Anyway, it's indeed a discussion for another time...

Again unrelated to this patch,

Note that LNT is already designed to be extensible. Schemas like 'nts' or 'compile' switch the metrics available or what fields are considered for identifying/ordering runs. It just fails being user friendly so that people would actually create new schemes or modifying existing when they setup new benchmarks. IMO the current database design is good enough and we rather need a patch that makes creating and installing new schemes easy. I imagine a world where testsuite come with lnt.schema.json which describes the names/bigger is better/etc. for a testsuite and then when you setup a server you just reference those schemas in your lnt.cfg and get a server that matches your testsuite.

Right, seems like a reasonable way forward to me. I'm wondering if adding extra fields to a test-suite becoming as simple as just describing the extra fields in the json file you import, rather than at server configuration time, would even be nicer? Anyway - a discussion for the future.

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

I talked a bit with Chris about this. And I think we want to persue a somwhat weaker version scheme were tools add something like "generated": "LNT version 0.4.1" or "generated": "llvm test-suite runner 0.1", similar in spirit to the http user agent header.

I'm not sure I fully get this. I know of at least 2 groups having custom scripts to translate data coming out of their custom benchmarking system into something that can be imported into LNT. I know of yet another group planning to do something similar. I think it won't be possible for LNT to understand all possible producers of json files by name. So, with my limited insight here, I think that just a numerical version number or something similar indicating which version of the specification this json follows would be better than a string naming the producer?
In other words, I think this json format is rapidly becoming one of the key public interfaces of LNT - maybe the single most important one. If LNT keeps on growing in popularity in teams outside of LLVM, I think we'll end up having to specify this interface, and version control of it, a bit more strictly.

Ok, I'll add a field 'format_version' to the toplevel.

Thanks, sounds good.

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

• The name for the field is determined by the schema in LNT, so not necessarily part of this patch.

OK, I agree it doesn't have to be part of this patch. But taking into account my remark above about the json file rapidly becoming one of the most important interfaces of LNT, I think that it may not be tenable in the long run to have the json fields map exactly to names of fields in the database schema. It is of course a nice to have; but the field names in the json file are on the publicly-visible interface of LNT, and the database field names are an implementation detail from a user point-of-view. I can foresee reasons why both might start deviating.
Since my gut feel is that it will be harder/more work to change the "json format specification" in the future, I thought that maybe it could be part of the change that is going to happen as part of this patch. Anyway, I don't have too strong a feeling about the exact name of this field.

Yes and no; As see above about my vision of actually using the LNT feature of different benchmark suites instead of pushing everything into NTS :)

Right. As you said earlier, probably the only teams using different benchmark suite schemas may well be in Apple, so me and others using LNT elsewhere probably don't have the best feel for this.
I'm not against keeping the DB scheme as is. As you rightfully indicated, the key is making it easier to add fields for other test-suites.

• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

Agreed, it's not the best possible name. It should be the revision of the software for which you're tracking the performance in my mind. That software doesn't have to be a "compiler". Not sure what a better name would be though.

I'd be fine changing it to "compiler_revision" right away, if we need more discussion about how to call the field, then I'd suggest doing that in another patch to not delay this patch.

Let's change it to "compiler_revision" then, as all the other projects I know of so far track performance of a piece of software that could be claimed to be a compiler of sorts.

docs/importing_data.rst

109–116 ↗

(On Diff #103809)

This is the one comment I feel needs to be addressed before committing this patch.

lnt/server/db/testsuitedb.py

683 ↗

(On Diff #104547)

s/to/so ?

888 ↗

(On Diff #104547)

This is an opportunity to put a space after the comma in "machine,inserted" to make this PEP8 clean?

891 ↗

(On Diff #104547)

Another opportunity to make this PEP8-clean?

lnt/testing/__init__.py

40 ↗

(On Diff #104547)

I'm not sure, as I don't understand this code well, but there's a chance this should now be renamed to self.format_version instead of self.report_version?

109–112 ↗

(On Diff #104547)

I guess the conversion of the Run class to follow format_version 2, rather than 1 will fit more naturally with one of the follow-up patches? Or as a stand-alone technical-debt-removal patch?

[MatzeB]

In D34584#794921, @kristof.beyls wrote:

In D34584#794519, @MatzeB wrote:

Before I dive in the meta discussion, are there any issues with the patch itself?

I'm not up to speed on many parts of LNT that this code touches, but reading through it, I only added some optional nitpicks, apart from 1 issue that I think needs to be addressed before committing:
The patch as is will drop documentation on what field names to use in the json representation, making it impossible for people to write a converter from data from other testing/benchmarking systems to the json-format to import into LNT, without going through a lot of effort trying to reverse engineer the format from source code. I think the documentation of the fields with meaning for "nts" must be retained.
My experience is that most of such users of LNT don't grasp the concept of multiple "test-suite formats" such as "nts" or "compile" in the LNT DB, which is why I didn't bother trying to explain that well in the current documentation.
As you suggest later, maybe the json format should become self-describing (I'm maybe pushing your suggestion further than you intended here), but until that happens, I think the documentation on the nts field names that matter should stay.

Once that documentation issue is addressed, I have no objections for this patch to be committed.

In D34584#793766, @kristof.beyls wrote:

In D34584#792859, @MatzeB wrote:

Drop Run/Info/tag: json itself is a schemaless format so it feels more natural without one. The LNT testsuite type is now determined by the submit URL used, or the '-s' commandline flag on the commandline. Both default to 'nts' if omitted.

I don't fully understand how LNT reacts to this, but on the surface makes sense to me. If someone ever had time to do a major redesign of the database schema, there is a chance we might also be able to drop the need for differences in the database schema per test suite, like "nts"? Anyway, that is definitely far out of scope for this patch.

The patch works fine. People using non-nts testsuites (which I have a strong suspicion only exist within apple), may need to supply a -s MySchema flag for the commandline tools or use db_default/v4/MySchema/submitRun instead of db_default/submitRun to express they don't use the 'nts' default.

This is probably not the right patch to discuss LNT database design. But some points anyway:

• IMO making SQL tables too generic (like having one row with MetricType, Test, Run and Value; so we could create new MetricTypes on demand) is a sure way to kill performance. As long as we use SQL we should have 1 table column for each metric.
• A schema is nice when you need to decide bits like the unit of a metric, whether bigger is better or getting a human readable metric name for the website. However we could make good arguments that we only need that on the presentation/analysis side and not as part of the data storage model.

Agreed that not losing performance will be the key challenge for any redesign of the SQL schema to avoid having test-suite-specific tables. Anyway, it's indeed a discussion for another time...

Again unrelated to this patch,

Note that LNT is already designed to be extensible. Schemas like 'nts' or 'compile' switch the metrics available or what fields are considered for identifying/ordering runs. It just fails being user friendly so that people would actually create new schemes or modifying existing when they setup new benchmarks. IMO the current database design is good enough and we rather need a patch that makes creating and installing new schemes easy. I imagine a world where testsuite come with lnt.schema.json which describes the names/bigger is better/etc. for a testsuite and then when you setup a server you just reference those schemas in your lnt.cfg and get a server that matches your testsuite.

Right, seems like a reasonable way forward to me. I'm wondering if adding extra fields to a test-suite becoming as simple as just describing the extra fields in the json file you import, rather than at server configuration time, would even be nicer? Anyway - a discussion for the future.

Previously:

"Machine": {

"name": "mymachine",
"Info": {
  "hardware": "HAL 9000"
}

},
"Run": {

"Start Time": "2017-05-16 21:31:42",
"End Time": "2017-05-16 21:31:42",
"Info": {
  "__report_version__": "1",

I don't know what report_version is used for today, but would keep on having some kind of "report_version" key still help if in the future we'd want to change this format again? Having a version string in there could make it easier to support multiple versions?

I talked a bit with Chris about this. And I think we want to persue a somwhat weaker version scheme were tools add something like "generated": "LNT version 0.4.1" or "generated": "llvm test-suite runner 0.1", similar in spirit to the http user agent header.

I'm not sure I fully get this. I know of at least 2 groups having custom scripts to translate data coming out of their custom benchmarking system into something that can be imported into LNT. I know of yet another group planning to do something similar. I think it won't be possible for LNT to understand all possible producers of json files by name. So, with my limited insight here, I think that just a numerical version number or something similar indicating which version of the specification this json follows would be better than a string naming the producer?
In other words, I think this json format is rapidly becoming one of the key public interfaces of LNT - maybe the single most important one. If LNT keeps on growing in popularity in teams outside of LLVM, I think we'll end up having to specify this interface, and version control of it, a bit more strictly.

Ok, I'll add a field 'format_version' to the toplevel.

Thanks, sounds good.

  "tag": "nts",
  "run_order": "123456"
}

}
Now:

"machine": {

"name": "mymachine",
"hardware": "HAL 9000"

},
"run": {

"start_time": "2017-05-16 21:31:42",
"end_time": "2017-05-16 21:31:42",
"llvm_project_revision": "123456"

Since I see LNT being used also for tracking performance for non-LLVM projects, I think it'd be best to drop "llvm_" from "llvm_project_revision".

• The name for the field is determined by the schema in LNT, so not necessarily part of this patch.

OK, I agree it doesn't have to be part of this patch. But taking into account my remark above about the json file rapidly becoming one of the most important interfaces of LNT, I think that it may not be tenable in the long run to have the json fields map exactly to names of fields in the database schema. It is of course a nice to have; but the field names in the json file are on the publicly-visible interface of LNT, and the database field names are an implementation detail from a user point-of-view. I can foresee reasons why both might start deviating.
Since my gut feel is that it will be harder/more work to change the "json format specification" in the future, I thought that maybe it could be part of the change that is going to happen as part of this patch. Anyway, I don't have too strong a feeling about the exact name of this field.

Yes and no; As see above about my vision of actually using the LNT feature of different benchmark suites instead of pushing everything into NTS :)

Right. As you said earlier, probably the only teams using different benchmark suite schemas may well be in Apple, so me and others using LNT elsewhere probably don't have the best feel for this.
I'm not against keeping the DB scheme as is. As you rightfully indicated, the key is making it easier to add fields for other test-suites.

• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

Agreed, it's not the best possible name. It should be the revision of the software for which you're tracking the performance in my mind. That software doesn't have to be a "compiler". Not sure what a better name would be though.

I'd be fine changing it to "compiler_revision" right away, if we need more discussion about how to call the field, then I'd suggest doing that in another patch to not delay this patch.

Let's change it to "compiler_revision" then, as all the other projects I know of so far track performance of a piece of software that could be claimed to be a compiler of sorts.

Hmpf, I should have tried this before making promises: Turns out sqlite does not support renaming a column. This means I cannot easily provide migration code (adding a new column and copying doesn't work nicely either as removing the old column isn't supported either). So I fear this is best addressed by creating a new LNT schema.

[kristof.beyls]

• "project_revision" seems to generic to me (I wouldn't know whether it's the compiler or the benchmark suites revision). I could probably live with "compiler_revision", but we should have a separate patch for that.

Agreed, it's not the best possible name. It should be the revision of the software for which you're tracking the performance in my mind. That software doesn't have to be a "compiler". Not sure what a better name would be though.

I'd be fine changing it to "compiler_revision" right away, if we need more discussion about how to call the field, then I'd suggest doing that in another patch to not delay this patch.

Let's change it to "compiler_revision" then, as all the other projects I know of so far track performance of a piece of software that could be claimed to be a compiler of sorts.

Hmpf, I should have tried this before making promises: Turns out sqlite does not support renaming a column. This means I cannot easily provide migration code (adding a new column and copying doesn't work nicely either as removing the old column isn't supported either). So I fear this is best addressed by creating a new LNT schema.

Right, let's leave it for later then.

[MatzeB]

Thanks for the review!

lnt/testing/__init__.py

40 ↗

(On Diff #104547)

This is the version used for the reports produced by things like lnt runtest. Those still produce the old format. Which I guess for now is a feature as we can still submit to old servers for a while and can do this switch separately.

109–112 ↗

(On Diff #104547)

Yep.

lnt/trunk/docs/importing_data.rst

 test suite you are submitting to.
 
 Example::
 
     echo -n "foo.exec 25\nbar.score 24.2\nbar/baz.size 110.0\n" > results.txt
     lnt importreport --machine=my-machine-name --order=1234 --testsuite=nts results.txt report.json
     lnt submit http://mylnt.com/default/submitRun --commit=1 report.json
 
+.. _json_format:
 
-Importing Data from Other Test Systems
-======================================
+LNT Report File Format
+======================
 
 The lnt importreport tool is an easy way to import data into LNTs test format.
-Or you can write your own importer.
+You can also create LNTs report data directly for additional flexibility.
 
 First, make sure you've understood the underlying :ref:`concepts` used by LNT.
 
-Given how simple it is to make your own results and send them to LNT,
-it is common to not use the LNT client application at all, and just have a
-custom script run your tests and submit the data to the LNT server. Details
-on how to do this are in :mod:`lnt.testing`
-
-If for some reason you prefer to generate the json file more directly, the
-current format looks like below. It remains recommended to use the APIs in
-:mod:`lnt.testing` to be better protected against future changes to the json
-format::
+.. code-block:: none
 
   {
-     "Machine": {
-        "Info": {
-          (_String_: _String_)* // optional extra info about the machine.
-        },
-        "Name": _String_ // machine name, mandatory
-     },
-     "Run": {
-        "End Time": "%Y-%m-%d %H:%M:%S", // mandatory
-        "Start Time": "%Y-%m-%d %H:%M:%S", // mandatory
-        "Info": {
-          "run_order": _String_, // mandatory
-          "tag": "nts" // mandatory
+      "format_version": "2",
+      "machine": {
+          "name": _String_      // machine name, mandatory
+          (_String_: _String_)* // optional extra info
+      },
+      "run": {
+          "start_time": "%Y-%m-%d %H:%M:%S", // mandatory
+          "end_time": "%Y-%m-%d %H:%M:%S",   // mandatory
           (_String_: _String_)* // optional extra info about the run.
-        }
+          // At least one of the extra fields is used as ordering and is
+          // mandatory. For the 'nts' and 'Compile' schemas this is the
+          // 'llvm_project_revision' field.
       },
-     "Tests": [
+      "tests": [
           {
-              "Data": [ (float+) ],
-              "Info": {},
-              "Name": "nts._ProgramName_._metric_"
+              "name": _String_,   // test name mandatory
+              (_String_: _Data_)* // List of metrics, _Data_ allows:
+                                  // number, string or list of numbers
           }+
       ]
-   }
 
+A concrete small example is
 
-A concrete small example is::
+.. literalinclude:: report-example.json
+    :language: json
 
-  {
-     "Machine": {
-        "Info": {
-        },
-        "Name": "LNT-AArch64-A53-O3__clang_DEV__aarch64"
-     },
-     "Run": {
-        "End Time": "2016-04-07 14:25:52",
-        "Start Time": "2016-04-07 09:33:48",
-        "Info": {
-          "run_order": "265649",
-          "tag": "nts"
-        }
-     },
-     "Tests": [
-          {
-              "Data": [
-                  0.1056,
-                  0.1055
-              ],
-              "Info": {},
-              "Name": "nts.suite1/program1.exec"
-          },
-          {
-              "Data": [
-                  0.2136
-              ],
-              "Info": {},
-              "Name": "nts.suite2/program1.exec"
-          }
-      ]
-   }
 
-Make sure that:
- * The Run.Info.tag value is "nts".
- * The test names always start with "nts.".
- * The extension of the test name indicate what kind of data is recorded.
-   Currently accepted extensions in the NTS database are:
-
-   * ".exec": represents execution time - a lower number is better.
-   * ".exec.status": a non zero value represents a program failure.
-   * ".score": represent a benchmark score - a higher number is better.
-   * ".code_size": represents the code size of a program.
-   * ".hash": represents a hash of the binary program being executed. This is
-     used to detect if between compiler versions, the generated code has
-     changed.
-   * ".compile": represents the compile time of the program.
+Given how simple it is to make your own results and send them to LNT,
+it is common to not use the LNT client application at all, and just have a
+custom script run your tests and submit the data to the LNT server. Details
+on how to do this are in :mod:`lnt.testing`
+
+.. _nts_suite:
+
+Default Test Suite (NTS)
+========================
+
+The default test-suite schema is called NTS. It was originally designed for
+nightly test runs of the llvm test-suite. However it should fit many other
+benchmark suites as well. The following metrics are supported for a test:
+
+* ``execution_time``: Execution time in seconds; lower is better.
+* ``score``: Benchmarking score; higher is better.
+* ``compile_time``: Compiling time in seconds; lower is better.
+* ``execution_status``: A non zero value represents an execution failure.
+* ``compilation_status``: A non zero value represents a compilation failure.
+* ``hash_status``: A non zero value represents a failure computing the
+  executable hash.
+* ``mem_byts``: Memory usage in bytes during execution; lower is better.
+* ``code_size``: Code size (usually the size of the text segment) in bytes;
+  lower is better.
 
- All of these metrics are optional.
+The `run` information is expected to contain this:
 
+* ``llvm_project_revision``: The revision or version of the compiler
+  used for the tests. Used to sort runs.
 
 .. _custom_testsuites:
 
 Custom Test Suites
 ==================
 
 LNTs test suites are derived from a set of metadata definitions for each suite.
 Simply put, suites are a collections of metrics that are collected for each run.

lnt/trunk/docs/report-example.json

+{
+    "format_version": "2",
+    "machine": {
+       "name": "LNT-AArch64-A53-O3__clang_DEV__aarch64",
+       "hardware": "HAL 9000"
+    },
+    "run": {
+       "end_time": "2016-04-07 14:25:52",
+       "start_time": "2016-04-07 09:33:48",
+       "llvm_project_revision": "265649",
+       "compiler_version": "clang 4.0"
+    },
+    "tests": [
+       {
+           "name": "benchmark1",
+           "execution_time": [ 0.1056, 0.1055 ],
+           "hash": "49333a87d501b0aea2191830b66b5eec"
+       },
+       {
+           "name": "benchmark2",
+           "compile_time": 13.12,
+           "execution_time": 0.2135,
+           "hash": "c321727e7e0dfef279548efdb8ab2ea6"
+       }
+    ]
+}

lnt/trunk/lnt/lnttool/import_data.py

@click.option("--database", default="default", show_default=True,
               help="database to modify")
 @click.option("--format", "output_format", show_default=True,
               type=click.Choice(lnt.formats.format_names + ['<auto>']),
               default='<auto>', help="input format")
 @click.option("--commit", type=int, help="commit changes to the database")
 @click.option("--show-sql", is_flag=True, help="show SQL statements")
 @click.option("--show-sample-count", is_flag=True)
 @click.option("--show-raw-result", is_flag=True)
[e-mail]("--testsuite", "-s", default='nts')
 @click.option("--verbose", "-v", is_flag=True,
               help="show verbose test results")
 @click.option("--quiet", "-q", is_flag=True, help="don't show test results")
 @click.option("--no-email", is_flag=True, help="don't send e-mail")
 @click.option("--no-report", is_flag=True, help="don't generate report")
 def action_import(instance_path, files, database, output_format, commit,
-                  show_sql, show_sample_count, show_raw_result, verbose,
-                  quiet, no_email, no_report):
+                  show_sql, show_sample_count, show_raw_result, testsuite,
+                  verbose, quiet, no_email, no_report):
     """import test data into a database"""
 
     # Load the LNT instance.
     instance = lnt.server.instance.Instance.frompath(instance_path)
     config = instance.config
 
     # Get the database.
     with contextlib.closing(config.get_database(database,
                                                 echo=show_sql)) as db:
         # Load the database.
         success = True
         for file_name in files:
             result = lnt.util.ImportData.import_and_report(
                 config, database, db, file_name,
-                output_format, commit, show_sample_count,
+                output_format, testsuite, commit, show_sample_count,
                 no_email, no_report)
 
             success &= result.get('success', False)
             if quiet:
                 continue
 
             if show_raw_result:
                 pprint.pprint(result)
             else:
                 lnt.util.ImportData.print_report_result(result, sys.stdout,
                                                         sys.stderr,
                                                         verbose)
 
         if not success:
             raise SystemExit, 1

lnt/trunk/lnt/lnttool/main.py

else:
                 use_reloader=reloader,
                 use_debugger=debugger,
                 threaded=threaded,
                 processes=processes)
 
 
 @click.command("checkformat")
 @click.argument("file", "input_file", nargs=-1, type=click.Path(exists=True))
-def action_checkformat(input_file):
[e-mail]("--testsuite", "-s", default='nts')
+def action_checkformat(input_file, testsuite):
     """check the format of an LNT test report file"""
 
     import lnt.server.db.v4db
     import lnt.server.config
     db = lnt.server.db.v4db.V4DB('sqlite:///:memory:',
                                  lnt.server.config.Config.dummy_instance())
     result = lnt.util.ImportData.import_and_report(
-        None, None, db, input_file, 'json', commit=True)
+        None, None, db, input_file, 'json', testsuite, commit=True)
     lnt.util.ImportData.print_report_result(result, sys.stdout, sys.stderr,
                                             verbose=True)
 
 
 def _print_result_url(results, verbose):
     result_url = results.get('result_url')
     if result_url is not None:
         if verbose:

lnt/trunk/lnt/lnttool/viewcomparison.py

 @click.argument("report_a", type=click.Path(exists=True))
 @click.argument("report_b", type=click.Path(exists=True))
 @click.option("--hostname", default="localhost", show_default=True,
               help="host interface to use")
 @click.option("--port", default=8000, show_default=True,
               help="local port to use")
 @click.option("--dry-run", is_flag=True,
               help="do a dry run through the comparison")
-def action_view_comparison(report_a, report_b, hostname, port, dry_run):
[e-mail]("--testsuite", "-s", default='nts')
+def action_view_comparison(report_a, report_b, hostname, port, dry_run,
+                           testsuite):
     """view a report comparison using a temporary server"""
 
     import lnt.server.instance
     import lnt.server.ui.app
     import lnt.server.db.migrate
 
     # Set up the default logger.
     logger = logging.getLogger("lnt")
try:
                                           None)
         instance = lnt.server.instance.Instance(None, config)
 
         # Create the database.
         lnt.server.db.migrate.update_path(db_path)
 
         # Import the two reports.
         with contextlib.closing(config.get_database('default')) as db:
-            r = import_and_report(
-                config, 'default', db, report_a, '<auto>', commit=True)
-            import_and_report(
-                config, 'default', db, report_b, '<auto>', commit=True)
+            r = import_and_report(config, 'default', db, report_a, '<auto>',
+                                  testsuite, commit=True)
+            import_and_report(config, 'default', db, report_b, '<auto>',
+                              testsuite, commit=True)
 
             # Dispatch another thread to start the webbrowser.
             comparison_url = '%s/v4/nts/2?compare_to=1' % (url,)
             note("opening comparison view: %s" % (comparison_url,))
 
             if not dry_run:
                 thread.start_new_thread(start_browser, (comparison_url, True))
 

lnt/trunk/lnt/server/db/migrations/upgrade_11_to_12.py

+# Rename 'name' field in machine parameters to 'hostname' to avoid name clash
+# in import/export file format.
+import sqlalchemy
+import json
+
+def update_testsuite(engine, db_key_name):
+    class Machine(object):
+        pass
+
+    session = sqlalchemy.orm.sessionmaker(engine)()
+    meta = sqlalchemy.MetaData(bind=engine)
+
+    machine_table = sqlalchemy.Table("%s_Machine" % db_key_name, meta, autoload=True)
+    sqlalchemy.orm.mapper(Machine, machine_table)
+
+    all_machines = session.query(Machine)
+    for machine in all_machines:
+        info = dict(json.loads(machine.Parameters))
+        name = info.pop('name', None)
+        if name is not None:
+            info['hostname'] = name
+        machine.Parameters = json.dumps(sorted(info.items()))
+
+    session.commit()
+    session.close()
+
+def upgrade(engine):
+    update_testsuite(engine, 'NT')
+    update_testsuite(engine, 'Compile')

lnt/trunk/lnt/server/db/testsuitedb.py

def _getOrCreateMachine(self, machine_data):
         """
 
         # Convert the machine data into a machine record. We construct the query
         # to look for any existing machine at the same time as we build up the
         # record to possibly add.
         #
         # FIXME: This feels inelegant, can't SA help us out here?
         query = self.query(self.Machine).\
-            filter(self.Machine.name == machine_data['Name'])
-        machine = self.Machine(machine_data['Name'])
-        machine_parameters = machine_data['Info'].copy()
+            filter(self.Machine.name == machine_data['name'])
+        machine = self.Machine(machine_data['name'])
+        machine_parameters = machine_data.copy()
+        machine_parameters.pop('name')
+        # Ignore incoming ids; we will create our own.
+        # TODO: Add some API/result so we can send a warning back to the user
+        # that we ignore the id.
+        machine_parameters.pop('id', None)
 
         # First, extract all of the specified machine fields.
         for item in self.machine_fields:
-            if item.info_key in machine_parameters:
-                value = machine_parameters.pop(item.info_key)
-            else:
-                # For now, insert empty values for any missing fields. We don't
-                # want to insert NULLs, so we should probably allow the test
-                # suite to define defaults.
-                value = ''
+            default_value = ''
 
+            value = machine_parameters.pop(item.name, default_value)
             query = query.filter(item.column == value)
             machine.set_field(item, value)
 
         # Convert any remaining machine_parameters into a JSON encoded blob. We
         # encode this as an array to avoid a potential ambiguity on the key
         # ordering.
         machine.parameters = machine_parameters
         query = query.filter(self.Machine.parameters_data ==
def _getOrCreateOrder(self, run_parameters):
         or not.
         """
 
         query = self.query(self.Order)
         order = self.Order()
 
         # First, extract all of the specified order fields.
         for item in self.order_fields:
-            if item.info_key in run_parameters:
-                value = run_parameters.pop(item.info_key)
-            else:
+            value = run_parameters.pop(item.name, None)
+            if value is None:
                 # We require that all of the order fields be present.
                 raise ValueError,"""\
 supplied run is missing required run parameter: %r""" % (
-                    item.info_key)
+                    item.name)
 
             query = query.filter(item.column == value)
             order.set_field(item, value)
 
         # Execute the query to see if we already have this order.
         try:
             return query.one(),False
         except sqlalchemy.orm.exc.NoResultFound:
def _getOrCreateRun(self, run_data, machine):
         Add a new Run record from the given data (as recorded by the test
         interchange format).
 
         The boolean result indicates whether the returned record was constructed
         or not.
         """
 
         # Extra the run parameters that define the order.
-        run_parameters = run_data['Info'].copy()
-
-        # The tag has already been used to dispatch to the appropriate database.
-        run_parameters.pop('tag')
+        run_parameters = run_data.copy()
+        # Ignore incoming ids; we will create our own
+        run_parameters.pop('id', None)
 
         # Find the order record.
         order,inserted = self._getOrCreateOrder(run_parameters)
-        start_time = datetime.datetime.strptime(run_data['Start Time'],
+        start_time = datetime.datetime.strptime(run_data['start_time'],
                                                 "%Y-%m-%d %H:%M:%S")
-        end_time = datetime.datetime.strptime(run_data['End Time'],
+        end_time = datetime.datetime.strptime(run_data['end_time'],
                                               "%Y-%m-%d %H:%M:%S")
+        run_parameters.pop('start_time')
+        run_parameters.pop('end_time')
 
         # Convert the rundata into a run record. As with Machines, we construct
         # the query to look for any existingrun at the same time as we build up
         # the record to possibly add.
         #
         # FIXME: This feels inelegant, can't SA help us out here?
         query = self.query(self.Run).\
             filter(self.Run.machine_id == machine.id).\
             filter(self.Run.order_id == order.id).\
             filter(self.Run.start_time == start_time).\
             filter(self.Run.end_time == end_time)
         run = self.Run(machine, order, start_time, end_time)
 
         # First, extract all of the specified run fields.
         for item in self.run_fields:
-            if item.info_key in run_parameters:
-                value = run_parameters.pop(item.info_key)
-            else:
-                # For now, insert empty values for any missing fields. We don't
-                # want to insert NULLs, so we should probably allow the test
-                # suite to define defaults.
-                value = ''
+            default_value = ''
 
+            value = run_parameters.pop(item.name, default_value)
             query = query.filter(item.column == value)
             run.set_field(item, value)
 
         # Any remaining parameters are saved as a JSON encoded array.
         run.parameters = run_parameters
         query = query.filter(self.Run.parameters_data == run.parameters_data)
 
         # Execute the query to see if we already have this run.
         try:
             return query.one(),False
         except sqlalchemy.orm.exc.NoResultFound:
             # If not, add the run.
             self.add(run)
 
             return run,True
 
-    def _importSampleValues(self, tests_data, run, tag, commit, config):
-        # We now need to transform the old schema data (composite samples split
-        # into multiple tests with mangling) into the V4DB format where each
-        # sample is a complete record.
-        tag_dot = "%s." % tag
-        tag_dot_len = len(tag_dot)
-
+    def _importSampleValues(self, tests_data, run, commit, config):
         # Load a map of all the tests, which we will extend when we find tests
         # that need to be added.
         test_cache = dict((test.name, test)
                           for test in self.query(self.Test))
 
-        # First, we aggregate all of the samples by test name. The schema allows
-        # reporting multiple values for a test in two ways, one by multiple
-        # samples and the other by multiple test entries with the same test
-        # name. We need to handle both.
-        tests_values = {}
+        profiles = dict()
+        field_dict = dict([(f.name, f) for f in self.sample_fields])
         for test_data in tests_data:
-            if test_data['Info']:
-                raise ValueError,"""\
-test parameter sets are not supported by V4DB databases"""
-
-            name = test_data['Name']
-            if not name.startswith(tag_dot):
-                raise ValueError,"""\
-test %r is misnamed for reporting under schema %r""" % (
-                    name, tag)
-            name = name[tag_dot_len:]
-
-            # Add all the values.
-            values = tests_values.get(name)
-            if values is None:
-                tests_values[name] = values = []
-
-            values.extend(test_data['Data'])
-
-        # Next, build a map of test name to sample values, by scanning all the
-        # tests. This is complicated by the interchange's support of multiple
-        # values, which we cannot properly aggregate. We handle this by keying
-        # off of the test name and the sample index.
-        sample_records = {}
-        profiles = {}
-        for name,test_samples in tests_values.items():
-            # Map this reported test name into a test name and a sample field.
-            #
-            # FIXME: This is really slow.
-            if name.endswith('.profile'):
-                test_name = name[:-len('.profile')]
-                sample_field = 'profile'
-            else:
-                for item in self.sample_fields:
-                    if name.endswith(item.info_key):
-                        test_name = name[:-len(item.info_key)]
-                        sample_field = item
-                        break
-                else:
-                    # Disallow tests which do not map to a sample field.
-                    raise ValueError,"""\
-    test %r does not map to a sample field in the reported suite""" % (
-                        name)
-
-            # Get or create the test.
-            test = test_cache.get(test_name)
+            name = test_data['name']
+            test = test_cache.get(name)
             if test is None:
-                test_cache[test_name] = test = self.Test(test_name)
+                test = self.Test(test_data['name'])
+                test_cache[name] = test
                 self.add(test)
 
-            for i, value in enumerate(test_samples):
-                record_key = (test_name, i)
-                sample = sample_records.get(record_key)
-                if sample is None:
-                    sample_records[record_key] = sample = self.Sample(run, test)
+            samples = []
+            for key, values in test_data.items():
+                if key == 'name':
+                    continue
+                field = field_dict.get(key)
+                if field is None and key != 'profile':
+                    raise ValueError, \
+    "test %r metric %r does not map to a sample field in the reported suite" % \
+                    (name, key)
+
+                if not isinstance(values, list):
+                    values = [values]
+                while len(samples) < len(values):
+                    sample = self.Sample(run, test)
                     self.add(sample)
-
-                if sample_field != 'profile':
-                    sample.set_field(sample_field, value)
-                else:
+                    samples.append(sample)
+                for sample, value in zip(samples, values):
+                    if key == 'profile':
                         sample.profile = profiles.get(hash(value),
-                                                  self.Profile(value, config,
-                                                               test_name))
+                                            self.Profile(value, config, name))
+                    else:
+                        sample.set_field(field, value)
 
     def importDataFromDict(self, data, commit, config=None):
         """
         importDataFromDict(data) -> bool, Run
 
         Import a new run from the provided test interchange data, and return the
         constructed Run record.
 
         The boolean result indicates whether the returned record was constructed
         or not (i.e., whether the data was a duplicate submission).
         """
 
         # Construct the machine entry.
-        machine,inserted = self._getOrCreateMachine(data['Machine'])
+        machine, inserted = self._getOrCreateMachine(data['machine'])
 
         # Construct the run entry.
-        run,inserted = self._getOrCreateRun(data['Run'], machine)
-
-        # Get the schema tag.
-        tag = data['Run']['Info']['tag']
-        
+        run, inserted = self._getOrCreateRun(data['run'], machine)
         # If we didn't construct a new run, this is a duplicate
         # submission. Return the prior Run.
         if not inserted:
             return False, run
         
-        self._importSampleValues(data['Tests'], run, tag, commit, config)
+        self._importSampleValues(data['tests'], run, commit, config)
 
         return True, run
 
     # Simple query support (mostly used by templates)
 
     def machines(self, name=None):
         q = self.query(self.Machine)
         if name:

lnt/trunk/lnt/server/db/v4db.py

def getNumRuns(self):
                     for ts in self.testsuite.values()])
     def getNumSamples(self):
         return sum([ts.query(ts.Sample).count()
                     for ts in self.testsuite.values()])
     def getNumTests(self):
         return sum([ts.query(ts.Test).count()
                     for ts in self.testsuite.values()])
 
-    def importDataFromDict(self, data, commit, config=None):
+    def importDataFromDict(self, data, commit, testsuite_schema, config=None):
         # Select the database to import into.
-        #
-        # FIXME: Promote this to a top-level field in the data.
-        db_name = data['Run']['Info'].get('tag')
-        if db_name is None:
-            raise ValueError, "unknown database target (no tag field)"
+        data_schema = data.get('schema')
+        if data_schema is not None and data_schema != testsuite_schema:
+            raise ValueError, "Tried to import '%s' data into schema '%s'" % \
+                              (data_schema, testsuite_schema)
 
-        db = self.testsuite.get(db_name)
+        db = self.testsuite.get(testsuite_schema)
         if db is None:
             raise ValueError, "test suite %r not present in this database!" % (
-                db_name)
+                db_schema)
 
         return db.importDataFromDict(data, commit, config)

lnt/trunk/lnt/server/ui/templates/reporting/runs.txt

 {%- endif %}
 {%- endfor %}
 
 {%- endif %}
 {%- endmacro -%}
 
 {{ report_url }}
 Nickname: {{ machine.name }}:{{ machine.id }}
-{% if 'name' in machine_parameters -%}
-{{ "Name: %s" | format(machine_parameters['name']) }}
+{% if 'hostname' in machine_parameters -%}
+{{ "Hostname: %s" | format(machine_parameters['hostname']) }}
 {%- endif %}
 Comparing:
      Run: {{ run.id }}, Order: {{ run.order.llvm_project_revision }}, Start Time: {{ run.start_time }}, End Time: {{ run.end_time }}
 {%- if compare_to %}
       To: {{ compare_to.id }}, Order: {{ compare_to.order.llvm_project_revision }}, Start Time: {{ compare_to.start_time }}, End Time: {{ compare_to.end_time }}
      {%- if run.machine != compare_to.machine %}
 *** WARNING ***:
 comparision is against a different machine

lnt/trunk/lnt/server/ui/views.py

 
 @db_route('/', only_v3 = False)
 def index():
     return render_template("index.html")
 
 ###
 # Database Actions
 
-
-@db_route('/submitRun', only_v3=False, methods=('GET', 'POST'))
-def submit_run():
+def _do_submit():
     if request.method == 'GET':
         return render_template("submit_run.html")
 
     assert request.method == 'POST'
     input_file = request.files.get('file')
     input_data = request.form.get('input_data')
     commit = int(request.form.get('commit', 0))
 
def _do_submit():
     db = request.get_db()
 
     # Import the data.
     #
     # FIXME: Gracefully handle formats failures and DOS attempts. We
     # should at least reject overly large inputs.
 
     result = lnt.util.ImportData.import_and_report(
-        current_app.old_config, g.db_name, db, path, '<auto>', commit)
+        current_app.old_config, g.db_name, db, path, '<auto>',
+        ts_name=g.testsuite_name, commit=commit)
 
     # It is nice to have a full URL to the run, so fixup the request URL
     # here were we know more about the flask instance.
     if result.get('result_url'):
         result['result_url'] = request.url_root + result['result_url']
 
     return flask.jsonify(**result)
 
 
+@db_route('/submitRun', only_v3=False, methods=('GET', 'POST'))
+def submit_run():
+    """Compatibility url that hardcodes testsuite to 'nts'"""
+    g.testsuite_name = 'nts'
+    return _do_submit()
+
+
+@v4_route('/submitRun', methods=('GET', 'POST'))
+def submit_run_ts():
+    return _do_submit()
 
 ###
 # V4 Schema Viewer
 
 @v4_route("/")
 def v4_overview():
     return render_template("v4_overview.html",
                            testsuite_name=g.testsuite_name)

lnt/trunk/lnt/testing/__init__.py

 """
 Utilities for working with the LNT test format.
 
 Clients can easily generate LNT test format data by creating Report objects for
 the runs they wish to submit, and using Report.render to convert them to JSON
 data suitable for submitting to the server.
 """
 
 import datetime
 import re
+import logging
 
 try:
     import json
 except ImportError:
     import simplejson as json
 
 # We define the following constants for use as sample values by convention.
 PASS = 0
class Report:
 
     In the LNT test model, every test run should define exactly one machine and
     run, and any number of test samples.
     """
     def __init__(self, machine, run, tests):
         self.machine = machine
         self.run = run
         self.tests = list(tests)
-        self.report_version = current_version
+        self.report_version = '1'
         self.check()
 
     def check(self):
         assert isinstance(self.machine, Machine)
         assert isinstance(self.run, Run)
         for t in self.tests:
             assert isinstance(t, TestSamples)
 
def __init__(self, start_time, end_time, info={}):
             end_time = datetime.datetime.utcnow()
 
         self.start_time = normalize_time(start_time)
         self.end_time = normalize_time(end_time)
         self.info = dict((str(key),str(value))
                          for key,value in info.items())
         if '__report_version__' in self.info:
             raise ValueError("'__report_version__' key is reserved")
-        self.info['__report_version__'] = str(current_version)
+        # TODO: Convert to version 2
+        self.info['__report_version__'] = '1'
 
     def update_endtime(self, end_time=None):
         if end_time is None:
             end_time = datetime.datetime.utcnow()
         self.end_time = normalize_time(end_time)
 
     def render(self):
         return { 'Start Time' : self.start_time,
def render(self):
                 'Data': self.data}
 
     def __repr__(self):
         # TODO remove this
         return "TestSample({}): {} - {}".format(self.name,
                                                 self.data,
                                                 self.info)
 ###
-# Report Versioning
+# Format Versioning
 
 # We record information on the report "version" to allow the server to support
 # some level of auto-upgrading data from submissions of older reports.
 #
 # We recorder the report version as a reserved key in the run information
 # (primarily so that it can be accessed post-import on the server).
 #
 # Version 0 --           : initial (and unversioned).
 #
 # Version 1 -- 2012-04-12: run_order was changed to not be padded, and allow
 # non-integral values.
-current_version = 1
+#
+# Version 2 -- 2017-06:  Revamped json format
+#    - Directly uses lnt names (no 'info_key' names anymore)
+#    - Flatten Machine.Info and Run.Info into the Machine and Run records
+#    - One record for each test (not one record for test+metric) with one entry
+#      for each metric.
+def _get_format_version(data):
+    format_version = data.get('format_version')
+    if format_version is not None:
+        return int(format_version)
+
+    # Older versions had a Run.Info.__report_version__ field
+    run = data.get('Run')
+    if run is not None:
+        info = run.get('Info')
+        if info is not None:
+            report_version = info.get('__report_version__', '0')
+            return int(report_version)
+
+    return None
+
 
 def upgrade_0_to_1(data):
     # We recompute the run_order here if it looks like this run_order was
     # derived (we presume from sniffing a compiler).
     run_info = data['Run']['Info']
     run_order = run_info.get('run_order')
     inferred_run_order = run_info.get('inferred_run_order')
 
if 'clang' in run_info.get('cc_name','') and \
         if not m:
             return
 
         cc_name,cc_version_num,cc_build_string,cc_extra = m.groups()
         m = re.search('clang-([0-9.]*)', cc_build_string)
         if m:
             run_info['run_order'] = run_info['inferred_run_order'] = \
                 run_info['cc_src_tag'] = m.group(1)
+    data['Run']['Info']['__report_version__'] = "1"
+    return data
 
-def upgrade_report(data):
-    # Get the report version.
-    report_version = int(data['Run']['Info'].get('__report_version__', 0))
 
-    # Check if the report is current.
-    if report_version == current_version:
-        return data
+# Upgrading from version 1 to version 2 needs some schema in place
+class _UpgradeSchema(object):
+    def __init__(self, metric_rename, machine_param_rename, run_param_rename):
+        self.metric_rename = metric_rename
+        self.machine_param_rename = machine_param_rename
+        self.run_param_rename = run_param_rename
+
+_nts_upgrade = _UpgradeSchema(
+    metric_rename={
+        '.code_size': 'code_size',
+        '.compile': 'compile_time',
+        '.compile.status': 'compile_status',
+        '.exec': 'execution_time',
+        '.exec.status': 'execution_status',
+        '.hash': 'hash',
+        '.hash.status': 'hash_status',
+        '.mem': 'mem_bytes',
+        '.score': 'score',
+    }, machine_param_rename = {
+        'name': 'hostname', # Avoid name clash with actual machine name.
+    }, run_param_rename = {
+        'run_order': 'llvm_project_revision',
+    }
+)
+_compile_upgrade = _UpgradeSchema(
+    metric_rename={
+        '.mem.bytes': 'mem_bytes',
+        '.mem.status': 'mem_status',
+        '.size.bytes': 'size_bytes',
+        '.size.status': 'size_status',
+        '.sys.status': 'sys_status',
+        '.user.status': 'user_status',
+        '.user.time': 'sys_time',
+        '.user.time': 'user_time',
+        '.wall.status': 'wall_status',
+        '.wall.time': 'wall_time',
+    }, machine_param_rename = {
+        'hw.model': 'hardware',
+        'kern.version': 'os_version',
+    }, run_param_rename = {
+        'run_order': 'llvm_project_revision',
+    }
+)
+_upgrades = {
+    'nts': _nts_upgrade,
+    'compile': _compile_upgrade
+}
+
+def upgrade_1_to_2(data, ts_name):
+    result = dict()
+
+    # Pull version and database schema to toplevel
+    result['format_version'] = '2'
+    report_version = data['Run']['Info'].pop('__report_version__', '1')
+    # We should not be in upgrade_1_to_2 for other versions
+    assert(report_version == '1')
+    tag = data['Run']['Info'].pop('tag', None)
+    if tag is not None and tag != ts_name:
+        raise ValueError("Importing '%s' data into '%s' testsuite" %
+                         (tag, ts_name))
+
+    upgrade = _upgrades.get(tag)
+    if upgrade is None:
+        logging.warn("No upgrade schema known for '%s'\n" % tag)
+        upgrade = _UpgradeSchema({}, {}, {})
+
+    # Flatten Machine.Info into machine
+    Machine = data['Machine']
+    result_machine = {'name': Machine['Name']}
+    for key, value in Machine['Info'].items():
+        newname = upgrade.machine_param_rename.get(key, key)
+        if newname in result_machine:
+            raise ValueError("Name clash for machine info '%s'" % newname)
+        result_machine[newname] = value
+    result['machine'] = result_machine
+
+    # Flatten Result.Info into result
+    Run = data['Run']
+    result_run = {
+        'end_time': Run['End Time'],
+        'start_time': Run['Start Time'],
+    }
+    for key, value in Run['Info'].items():
+        newname = upgrade.run_param_rename.get(key, key)
+        if newname in result_run:
+            raise ValueError("Name clash for run info '%s'" % newname)
+        result_run[newname] = value
+    result['run'] = result_run
+
+    # Merge tests
+    result_tests = list()
+    result_tests_dict = dict()
+    Tests = data['Tests']
+    for test in Tests:
+        test_Name = test['Name']
+
+        # Old testnames always started with 'tag.', split that part.
+        if len(test['Info']) != 0:
+            # The Info record didn't work with the v4 database anyway...
+            raise ValueError("Tests/%s: cannot convert non-empty Info record" %
+                             test_Name)
+        tag_dot = '%s.' % ts_name
+        if not test_Name.startswith(tag_dot):
+            raise ValueError("Tests/%s: test name does not start with '%s'" %
+                             (test_Name, tag_dot))
+        name_metric = test_Name[len(tag_dot):]
+
+        found_metric = False
+        for oldname, newname in upgrade.metric_rename.items():
+            assert(oldname.startswith('.'))
+            if name_metric.endswith(oldname):
+                name = name_metric[:-len(oldname)]
+                metric = newname
+                found_metric = True
+                break
+        if not found_metric:
+            # Fallback logic for unknown metrics: Assume they are '.xxxx'
+            name, dot, metric = name_metric.rpartition('.')
+            if dot != '.':
+                raise ValueError("Tests/%s: test name does not end in .metric" %
+                                 test_Name)
+            logging.warning("Found unknown metric '%s'" % metric)
+            upgrade.metric_rename['.'+metric] = '.'+metric
+
+        result_test = result_tests_dict.get(name)
+        if result_test is None:
+            result_test = {'name': name}
+            result_tests_dict[name] = result_test
+            result_tests.append(result_test)
+
+        data = test['Data']
+        if metric not in result_test:
+            # Do not construct a list for the very common case of just a
+            # single datum.
+            if len(data) == 1:
+                data = data[0]
+            result_test[metric] = data
+        elif len(data) > 0:
+            # Transform the test data into a list
+            if not isinstance(result_test[metric], list):
+                result_test[metric] = [ result_test[metric] ]
+            result_test[metric] += data
+
+    result['tests'] = result_tests
+    return result
+
+def upgrade_report(data, ts_name):
+    # Get the report version. V2 has it at the top level, older version
+    # in Run.Info.
+    format_version = _get_format_version(data)
+    if format_version is None:
+        format_version = 2
+
+    if format_version==0:
+        data = upgrade_0_to_1(data)
+        format_version=1
+    if format_version==1:
+        data = upgrade_1_to_2(data, ts_name)
+        format_version=2
+    assert(format_version==2)
 
-    # Check if the version is out-of-range.
-    if report_version > current_version:
-        raise ValueError("unknown report version: %r" % (report_version,))
-
-    # Otherwise, we need to upgrade it.
-    for version in range(report_version, current_version):
-        upgrade_method = globals().get('upgrade_%d_to_%d' % (
-                version, version+1))
-        upgrade_method(data)
-        data['Run']['Info']['__report_version__'] = str(version + 1)
-
 __all__ = ['Report', 'Machine', 'Run', 'TestSamples']

lnt/trunk/lnt/tests/builtintest.py

def print_report(report, output):
         if output == '-':
             output_stream = sys.stdout
         else:
             output_stream = open(output, 'w')
         print >> output_stream, report.render()
         if output_stream is not sys.stdout:
             output_stream.close()
 
-    def submit(self, report_path, config, commit=True):
+    def submit(self, report_path, config, ts_name=None, commit=True):
         """Submit the results file to the server.  If no server
         was specified, use a local mock server.
 
         report_path is the location of the json report file.  config
         holds options for submission url, and verbosity.  When commit
         is true, results will be saved in the server, otherwise you
         will just get back a report but server state is not altered.
 
         Returns the report from the server.
         """
         assert os.path.exists(report_path), "Failed report should have" \
             " never gotten here!"
+        assert ts_name is not None
 
         server_report = None
         if config.submit_url is not None:
             self.log("submitting result to %r" % (config.submit_url,))
             server_report = ServerUtil.submitFile(config.submit_url,
                                                   report_path,
                                                   commit,
                                                   config.verbose)
         else:
             # Simulate a submission to retrieve the results report.
 
             # Construct a temporary database and import the result.
             self.log("submitting result to dummy instance")
 
             db = lnt.server.db.v4db.V4DB("sqlite:///:memory:",
                                          server_config.Config.dummy_instance())
             server_report = ImportData.import_and_report(
-                None, None, db, report_path, 'json', commit)
+                None, None, db, report_path, 'json', ts_name, commit)
 
         assert server_report is not None, "Results were not submitted."
 
         ImportData.print_report_result(server_report, sys.stdout, sys.stderr,
                                        config.verbose)
         return server_report
 
     @staticmethod
     def show_results_url(server_results):
         """Print the result URL"""
         if server_results.get('result_url'):
             print "Results available at:", server_results['result_url']
         else:
             print "Results available at: no URL available"

lnt/trunk/lnt/tests/compile.py

def run_test(self, opts):
 
         # Save report to disk for submission.
         self.print_report(report, lnt_report_path)
 
         # Then, also print to screen if requested.
         if opts.output is not None:
             self.print_report(report, opts.output)
 
-        server_report = self.submit(lnt_report_path, opts)
+        server_report = self.submit(lnt_report_path, opts, ts_name='compile')
 
         return server_report

lnt/trunk/lnt/tests/nt.py

def submit_helper(self, config, commit=False):
             # Construct a temporary database and import the result.
             self.log("submitting result to dummy instance")
 
             import lnt.server.db.v4db
             import lnt.server.config
             db = lnt.server.db.v4db.V4DB("sqlite:///:memory:",
                                          lnt.server.config.Config.dummy_instance())
             result = lnt.util.ImportData.import_and_report(
-                None, None, db, report_path, 'json', commit)
+                None, None, db, report_path, 'json', 'nts', commit)
 
         if result is None:
             fatal("Results were not obtained from submission.")
 
         return result
 
 
 def _tools_check():

lnt/trunk/lnt/tests/test_suite.py

def run_test(self, opts):
             fd.write(str_template)
 
         csv_report_path = os.path.join(self._base_path,
                                        'test-results.csv')
         str_template = _lit_json_to_csv(json_reports)
         with open(csv_report_path, 'w') as fd:
             fd.write(str_template)
 
-        return self.submit(report_path, self.opts, commit=True)
+        return self.submit(report_path, self.opts, 'nts', commit=True)
 
     def _configure_if_needed(self):
         mkdir_p(self._base_path)
         if not self.configured:
             self._configure(self._base_path)
             self._clean(self._base_path)
             self.configured = True
 

lnt/trunk/lnt/util/ImportData.py

 import os, re, time
 import collections
 import lnt.testing
 import lnt.formats
 import lnt.server.reporting.analysis
 from lnt.testing.util.commands import note
 from lnt.util import NTEmailReport
 from lnt.util import async_ops
 
-def import_and_report(config, db_name, db, file, format, commit=False,
-                      show_sample_count=False, disable_email=False,
-                      disable_report=False):
+def import_and_report(config, db_name, db, file, format, ts_name,
+                      commit=False, show_sample_count=False,
+                      disable_email=False, disable_report=False):
     """
-    import_and_report(config, db_name, db, file, format,
+    import_and_report(config, db_name, db, file, format, ts_name,
                       [commit], [show_sample_count],
                       [disable_email]) -> ... object ...
 
     Import a test data file into an LNT server and generate a test report. On
     success, run is the newly imported run. Note that success is uneffected by
     the value of commit, this merely changes whether the run (on success) is
     committed to the database.
 
def import_and_report(config, db_name, db, file, format, ts_name,
     except:
         import traceback
         result['error'] = "load failure: %s" % traceback.format_exc()
         return result
 
     result['load_time'] = time.time() - startTime
 
     # Auto-upgrade the data, if necessary.
-    lnt.testing.upgrade_report(data)
+    data = lnt.testing.upgrade_report(data, ts_name)
 
     # Find the database config, if we have a configuration object.
     if config:
         db_config = config.databases[db_name]
     else:
         db_config = None
 
     # Find the email address for this machine's results.
     toAddress = email_config = None
     if db_config and not disable_email:
         email_config = db_config.email_config
         if email_config.enabled:
             # Find the machine name.
             machineName = str(data.get('Machine',{}).get('Name'))
             toAddress = email_config.get_to_address(machineName)
             if toAddress is None:
                 result['error'] = ("unable to match machine name "
                                    "for test results email address!")
                 return result
 
     importStartTime = time.time()
     try:
-        success, run = db.importDataFromDict(data, commit, config=db_config)
+        success, run = db.importDataFromDict(data, commit, ts_name,
+                                             config=db_config)
     except KeyboardInterrupt:
         raise
     except:
         raise
         import traceback
         result['error'] = "import failure: %s" % traceback.format_exc()
         return result
 
def import_and_report(config, db_name, db, file, format, ts_name,
     result['added_machines'] = db.getNumMachines() - numMachines
     result['added_runs'] = db.getNumRuns() - numRuns
     result['added_tests'] = db.getNumTests() - numTests
     if show_sample_count:
         result['added_samples'] = db.getNumSamples() - numSamples
 
     result['committed'] = commit
     result['run_id'] = run.id
-    ts_name = data['Run']['Info'].get('tag')
     if commit:
         db.commit()
         if db_config:
             #  If we are not in a dummy instance, also run background jobs.
             #  We have to have a commit before we run, so subprocesses can
             #  see the submitted data.
             ts = db.testsuite.get(ts_name)
             async_ops.async_fieldchange_calc(db_name, ts, run, config)
if config and config.databases[db_name].shadow_import:
         shadow_name = db_config.shadow_import
         with closing(config.get_database(shadow_name)) as shadow_db:
             if shadow_db is None:
                 raise ValueError, ("invalid configuration, shadow import "
                                    "database %r does not exist") % shadow_name
 
             # Perform the shadow import.
             shadow_result = import_and_report(config, shadow_name,
-                                              shadow_db, file, format, commit,
-                                              show_sample_count, disable_email,
-                                              disable_report)
+                                              shadow_db, file, format, ts_name,
+                                              commit, show_sample_count,
+                                              disable_email, disable_report)
 
             # Append the shadow result to the result.
             result['shadow_result'] = shadow_result
 
     result['success'] = True
     return result
 
 def print_report_result(result, out, err, verbose = True):

lnt/trunk/lnt/util/ServerUtil.py

def submitFileToInstance(path, file, commit):
     # in the instance.
     instance = lnt.server.instance.Instance.frompath(path)
     config = instance.config
     db_name = 'default'
     with contextlib.closing(config.get_database(db_name)) as db:
         if db is None:
             raise ValueError("no default database in instance: %r" % (path,))
         return lnt.util.ImportData.import_and_report(
-            config, db_name, db, file, format='<auto>', commit=commit)
+            config, db_name, db, file, format='<auto>', ts_name='nts',
+            commit=commit)
 
 
 def submitFile(url, file, commit, verbose):
     # If this is a real url, submit it using urllib.
     if '://' in url:
         result = submitFileToServer(url, file, commit)
         if result is None:
             return

lnt/trunk/tests/lnttool/email_tools.py

 # RUN: --host localhost %t.instance 1 2 | FileCheck %s --check-prefix CHECK0
 #
 # CHECK0: From: [e-mail]
 # CHECK0: To: [e-mail]
 # CHECK0: Subject: localhost__clang_DEV__x86_64 test results
 # CHECK0: === text/plain report
 # CHECK0: http://localhost/perf/v4/nts/2?compare_to=1&baseline=2
 # CHECK0: Nickname: localhost__clang_DEV__x86_64:1
-# CHECK0: Name: localhost
+# CHECK0: Hostname: localhost
 # CHECK0: Comparing:
 # CHECK0:      Run: 2, Order: 152289, Start Time: 2012-04-11 21:13:53, End Time: 2012-04-11 21:14:49
 # CHECK0:      To: 1, Order: 154331, Start Time: 2012-04-11 16:28:23, End Time: 2012-04-11 16:28:58
 # CHECK0: Baseline: 2, Order: 152289, Start Time: 2012-04-11 21:13:53, End Time: 2012-04-11 21:14:49
 # CHECK0: Tests Summary
 # ...
 # CHECK0: Unchanged Tests: 10 (10 on baseline)
 # CHECK0: Total Tests: 10

lnt/trunk/tests/lnttool/submit.py

 # CHECK-VERBOSE: Added Machines: 1
 # CHECK-VERBOSE: Added Runs    : 1
 # CHECK-VERBOSE: Added Tests   : 2
 #
 # CHECK-VERBOSE: Results
 # CHECK-VERBOSE: ----------------
 # CHECK-VERBOSE: PASS : 10
 # CHECK-VERBOSE: Results available at: http://localhost:9091/db_default/v4/nts/3
+
+# RUN: rm -rf %t.instance
+# RUN: python %{shared_inputs}/create_temp_instance.py \
+# RUN:   %s %{shared_inputs}/SmallInstance %t.instance
+# RUN: %{shared_inputs}/server_wrapper.sh %t.instance 9091 \
+# RUN:    lnt submit "http://localhost:9091/db_default/submitRun" --commit=1 \
+# RUN:       %{src_root}/docs/report-example.json -v | \
+# RUN:    FileCheck %s --check-prefix=CHECK-NEWFORMAT
+#
+# CHECK-NEWFORMAT: Import succeeded.
+# CHECK-NEWFORMAT: --- Tested: 10 tests --
+#
+# CHECK-NEWFORMAT: Imported Data
+# CHECK-NEWFORMAT: -------------
+# CHECK-NEWFORMAT: Added Machines: 1
+# CHECK-NEWFORMAT: Added Runs    : 1
+# CHECK-NEWFORMAT: Added Tests   : 2
+#
+# CHECK-NEWFORMAT: Results
+# CHECK-NEWFORMAT: ----------------
+# CHECK-NEWFORMAT: PASS : 10
+# CHECK-NEWFORMAT: Results available at: http://localhost:9091/db_default/v4/nts/3

lnt/trunk/tests/server/db/ImportV4TestSuiteInstance.py

 assert run_a.machine is machine
 assert run_b.machine is machine
 assert run_a.order is order_a
 assert run_b.order is order_b
 assert run_a.imported_from.endswith("sample-a-small.plist")
 assert run_b.imported_from.endswith("sample-b-small.plist")
 assert run_a.start_time == datetime.datetime(2009, 11, 17, 2, 12, 25)
 assert run_a.end_time == datetime.datetime(2009, 11, 17, 3, 44, 48)
-assert tuple(sorted(run_a.parameters.items())) == \
-    (('__report_version__', '1'), ('inferred_run_order', '1'))
-assert tuple(sorted(run_b.parameters.items())) == \
-    (('__report_version__', '1'), ('inferred_run_order', '2'))
+assert sorted(run_a.parameters.items()) == [('inferred_run_order', '1')]
+assert sorted(run_b.parameters.items()) == [('inferred_run_order', '2')]
 
 # Validate the samples.
 samples = list(ts.query(ts.Sample))
 assert len(samples) == 3
 sample_a_0,sample_a_1,sample_b = samples
 assert sample_a_0.run is run_a
 assert sample_a_1.run is run_a
 assert sample_b.run is run_b

lnt/trunk/tests/server/db/search.py

def setUp(self):
                 data = open(os.path.join(base_path, 'Inputs/report.json.in')) \
                     .read() \
                     .replace('@@MACHINE@@', r[0]) \
                     .replace('@@ORDER@@', r[1])
                 open(f.name, 'w').write(data)
     
                 result = lnt.util.ImportData.import_and_report(
                     None, 'default', self.db, f.name,
-                    '<auto>', True, False,
+                    '<auto>', 'nts', True, False,
                     True, True)
 
                 success &= result.get('success', False)
 
         assert success
 
     def _mangleResults(self, rs):
         return [(r.machine.name, str(r.order.llvm_project_revision))

lnt/trunk/tests/server/ui/test_api.py

 from V4Pages import check_json
 
 logging.basicConfig(level=logging.DEBUG)
 
 machines_expected_response = [{u'hardware': u'x86_64',
                                u'os': u'Darwin 11.3.0',
                                u'id': 1,
                                u'name': u'localhost__clang_DEV__x86_64',
+                               u'hostname': u'localhost',
                                u'uname': u'Darwin localhost 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12'
                                          u' 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64'},
                               {u'hardware': u'AArch64',
                                u'os': u'linux',
                                u'id': 2,
                                u'name': u'machine2'},
                               {u'hardware': u'AArch64',
                                u'os': u'linux',
possible_run_keys = {u'__report_version__',
                      u'TEST',
                      u'LLC_OPTFLAGS',
                      u'machine_id',
                      u'order',
                      u'cc_exec_hash',
                      }
 
 possible_machine_keys = {u'name',
+                         u'hostname',
                          u'hardware',
                          u'os',
                          u'id',
                          u'uname'}
 
 
 class JSONAPITester(unittest.TestCase):
     """Test the REST api."""