lit: Add StringMetricValue for TestResult.addMetric()

[MatzeB]

Summary

The test-suite will use this to attach additional infos like the md5sum
of the executable to the test result. We should not use the existing
addMetric() interface for this as an md5sum is not really a metric.

[MatzeB]

ping

[• ddunbar]

I would just use addMetric for this. I see a binary signature as just
another kind of metric data, sometimes people even use the term "measure"
to the process of computing a hash signature for a binary.

If you really dislike the term metric, I would probably rather just change
the one method name than have two methods that basically do the same job.

Also, +1 on getting test suite support for reporting binary hashes!

• Daniel

[MatzeB]

In D16871#349237, @ddunbar wrote:

I would just use addMetric for this. I see a binary signature as just
another kind of metric data, sometimes people even use the term "measure"
to the process of computing a hash signature for a binary.

I'm sure mathematicians will cry seeing the terms used like this, or they get nerd sniped into defining a metric space on md5sums :)

Anyway my thinking was mainly that analysis software can know which things we can compute deltas between runs and plot graphs and which information does not fit this category. Then again we can just check whether the datatype is a number or not, so I'll go with addMetric() then if an addData() is deemed unnecessary.

PS: For the record: lnt already collects md5sums, I just think that it would be more natural to collect such data with the test-suite lit runner (also because I like to run custom analysis scripts on the test-suite/lit output).

[MatzeB]

I just tried changing my code to use addMetric to store the md5sum, but that fails: TypeError: unexpected metric value: '577ee3bb6917176a1130303424ed1aea' because only subclasses of IntMetricValue and RealMetricValue are accepted.

So this patch seems the only way to store those, it is ok to push it?

[EricWF]

In D16871#354314, @MatzeB wrote:

I just tried changing my code to use addMetric to store the md5sum, but that fails: TypeError: unexpected metric value: '577ee3bb6917176a1130303424ed1aea' because only subclasses of IntMetricValue and RealMetricValue are accepted.

So this patch seems the only way to store those, it is ok to push it?

Why don't you define a MetricValue subclass that works for your data?

[• ddunbar]

Right, that is what I expected, something like MetricValueSignature that
clients would anticipate is a unique but otherwise meaningless signature
for some part of the test (and display accordingly, maybe by highlighting
when it changes).

[MatzeB]

Turns out you don't need a special StringMetricValue as the default fallback JSONMetricValue works as well here. Discovered because James
independently created a pretty similar patch for the testsuite (http://reviews.llvm.org/rL262220) in the meantime...

utils/lit/lit/Test.py

 import os
 from xml.sax.saxutils import escape
 from json import JSONEncoder
+try:
+    basestring
+except NameError:
+    basestring = str # python3 compatibility
 
 # Test result codes.
 
 class ResultCode(object):
     """Test result codes."""
 
     # We override __new__ and __getnewargs__ to ensure that pickling still
     # provides unique ResultCode objects in any particular instance.
def __init__(self, value):
         self.value = value
 
     def format(self):
         return '%.4f' % self.value
 
     def todata(self):
         return self.value
 
+class StringMetricValue(MetricValue):
+    def __init__(self, value):
+        self.value = value
+
+    def format(self):
+        return self.value
+
+    def todata(self):
+        return self.value
+
 class JSONMetricValue(MetricValue):
     """
         JSONMetricValue is used for types that are representable in the output
         but that are otherwise uninterpreted.
     """
     def __init__(self, value):
         # Ensure the value is a serializable by trying to encode it.
         # WARNING: The value may change before it is encoded again, and may

 
 def toMetricValue(value):
     if isinstance(value, MetricValue):
         return value
     elif isinstance(value, int) or isinstance(value, long):
         return IntMetricValue(value)
     elif isinstance(value, float):
         return RealMetricValue(value)
+    elif isinstance(value, basestring):
+        return StringMetricValue(value)
     else:
         # Try to create a JSONMetricValue and let the constructor throw
         # if value is not a valid type.
         return JSONMetricValue(value)
 
 
 # Test results.