[XRay] [test-suite] Add litsupport module for microbenchmarks

[eizan]

Summary

• add microbenchmark litsupport plugin
• use microbenchmark litsupport plugin for XRay benchmarks

The litsuport module was originally written by @MatzeB provided in https://reviews.llvm.org/D37421 with some minor edits to the test output name string (note: s/exec_time/microbenchmark_time_ns/)

[hfinkel]

litsupport/modules/microbenchmark.py

35

The summary might make sense in some contexts, but why not just return multiple metrics? I'd really like to move to a model where we collect all of the microbenchmark timings individually.

[MatzeB]

MicroBenchmarks/XRay/lit.local.cfg

2

I think we need some more logic here, like this:

test_modules = config.test_modules
if 'run' in test_modules:
    # Insert microbenchmarks module behind 'run'
    test_modules.insert(test_modules.index('run')+1, 'microbenchmarks')
    # Timeit results are not useful for microbenchmarks
    if 'timeit' in test_modules:
        test_modules.remove('timeit')

• I think we should add the microbenchmarks module behind 'run' so that 'remote' or 'run_under' can come later and modify the commandline including the redirections and flags for the microbenchmarks.
• It's probably best to remove the 'timeit' module as the number isn't really useful for microbenchmarks.

litsupport/modules/microbenchmark.py

35

I'd like to see individual timings too; but I think putting the benchmark name as prefix to the metric name is not the right direction. LNT unfortunately has a fixed schema of metrics that are recorded. The LNT UI and homebrewn tools are probably designed to well known metrics too (some information like 'bigger/smaller is better' is derived from the name today).

I think adding something to lit that allows to report multiple benchmarks/tests from a single test file can't be that hard.

[hfinkel]

litsupport/modules/microbenchmark.py

35

I think adding something to lit that allows to report multiple benchmarks/tests from a single test file can't be that hard.

I don't object to doing this, but it seems like the multiple-metrics-per-test is a direction in which we need to go anyway, no? I expect that we will need to update the LNT UI regardless of what we do (otherwise we could easily end up with a single huge list that makes the interface clunky). What are we going to do with all of the binary-size information that we collect?

[MatzeB]

litsupport/modules/microbenchmark.py

35

I don't object to doing this, but it seems like the multiple-metrics-per-test is a direction in which we need to go anyway, no? I expect that we will need to update the LNT UI regardless of what we do (otherwise we could easily end up with a single huge list that makes the interface clunky).

Sure and we already have multiple metrics per test of course; I think it's compiletime, exec_time, hash and size today. However given that this is somewhat two dimensional data today and we have one dimension along test names and one along the metrics, I really think the "subbenchmarks" should end up on the test-name dimenson and not the metrics one.

What are we going to do with all of the binary-size information that we collect?

LNT throws all the detailed codesize metrics away and only keeps size.__text I think, which probably is macOS only :-(

Though if you run the benchmarks yourself you can simply not use LNT and view results with something like test-suite/utils/compare.py.

[hfinkel]

litsupport/modules/microbenchmark.py

35

Sure and we already have multiple metrics per test of course; I think it's compiletime, exec_time, hash and size today. However given that this is somewhat two dimensional data today and we have one dimension along test names and one along the metrics, I really think the "subbenchmarks" should end up on the test-name dimenson and not the metrics one.

Okay. At some point we may want something more hierarchical, but we can also cross that bridge if/when we come to it.

LNT throws all the detailed codesize metrics away and only keeps size.__text I think, which probably is macOS only :-(

Yes. And that's certainly something else that can be improved.

[eizan]

add microbenchmark module after run and remove timeit from XRay/lit.local.cfg

[MatzeB]

Thanks, LGTM

[dberris]

LGTM, thanks @eizan!

[eizan]

This has been submitted: https://reviews.llvm.org/rL317310

MicroBenchmarks/XRay/lit.local.cfg

 config.environment['XRAY_OPTIONS'] = 'patch_premain=false xray_naive_log=false'
+test_modules = config.test_modules
+if 'run' in test_modules:
+    # Insert microbenchmark module behind 'run'
+    test_modules.insert(test_modules.index('run')+1, 'microbenchmark')
+    # Timeit results are not useful for microbenchmarks
+    if 'timeit' in test_modules:
+        test_modules.remove('timeit')

litsupport/modules/microbenchmark.py

+'''Test module to collect google benchmark results.'''
+from litsupport import shellcommand
+from litsupport import testplan
+import csv
+import lit.Test
+
+
+def _mutateCommandLine(context, commandline):
+    cmd = shellcommand.parse(commandline)
+    cmd.arguments.append("--benchmark_format=csv")
+    # We need stdout outself to get the benchmark csv data.
+    if cmd.stdout is not None:
+        raise Exception("Rerouting stdout not allowed for microbenchmarks")
+    benchfile = context.tmpBase + '.bench.csv'
+    cmd.stdout = benchfile
+    context.microbenchfiles.append(benchfile)
+
+    return cmd.toCommandline()
+
+
+def _mutateScript(context, script):
+    return testplan.mutateScript(context, script, _mutateCommandLine)
+
+
+def _collectMicrobenchmarkTime(context, microbenchfiles):
+    result = 0.0
+    for f in microbenchfiles:
+        with open(f) as inp:
+            lines = csv.reader(inp)
+            # First line: "name,iterations,real_time,cpu_time,time_unit..."
+            for line in lines:
+                if line[0] == 'name':
+                    continue
+                # Note that we cannot create new tests here, so for now we just
+                # add up all the numbers here.
+                result += float(line[3])
+    return {'microbenchmark_time_ns': lit.Test.toMetricValue(result)}
+
+
+def mutatePlan(context, plan):
+    context.microbenchfiles = []
+    plan.runscript = _mutateScript(context, plan.runscript)
+    plan.metric_collectors.append(
+        lambda context: _collectMicrobenchmarkTime(context,
+                                                   context.microbenchfiles)
+    )