[LNT] Combine perf data metrics from several files

[kpdev42]

Summary

Metrics are recorded to files test.perf_data, test.perf_data1, …
Parse all these files into a single dictionary.

OS Laboratory. Huawei Russian Research Institute. Saint-Petersburg

[kpdev42]

Ping

[cmatthews]

lnt/testing/profile/perf.py

16

It would be nice to add a type annotation here, so we can find issues before hitting "assert False" in production.

16

I think this should be "merge_recursivly", to be pep8 compliant.

Does this pass flake8?

Can you add a doc comment.

Can you add a small unit test for this.

24

I don't like assert False, especially with no extra logging. It will be hard to debug if we hit in production. How about raise a ValueError or TypeError or NotImplementedYet, and in the message. mention the types responsible for the error ?

55

This functionality needs a unit test.

55

Could this glob be any more specific?

[kpdev42]

Addressed some comments

[kpdev42]

lnt/testing/profile/perf.py

55

Could you please clarify a little bit - how this test should looks like? Because glob.glob("%s*" % f) where f - is (e.g.) somefile.perf_data just returns all somefile.perf_data* files

55

Could this glob be any more specific?

I think no, because name of profile file can be quite different

[kpdev42]

Ping

[kpdev42]

@cmatthews Hi, could you please review latest changes?

[cmatthews]

LGTM now. Thanks!

lnt/testing/profile/perf.py

 from __future__ import absolute_import
 from lnt.util import logger
 from .profile import ProfileImpl
 from .profilev1impl import ProfileV1
 
 import os
 import traceback
+import glob
 
 try:
     from . import cPerf  # type: ignore  # mypy cannot process Cython modules
 except Exception:
     pass
 
 
+def merge_recursively(dct1, dct2):
+    # type: (dict, dict) -> None
+    """Add the content of dct2 to dct1.
+    :param dct1: merge to.
+    :param dct2: merge from.
+    """
+    for k, v in dct2.items():
+        if k in dct1:
+            if isinstance(dct1[k], dict) and isinstance(v, dict):
+                merge_recursively(dct1[k], v)
+            elif isinstance(dct1[k], list) and isinstance(v, list):
+                dct1[k].extend(v)
+            else:
+                raise TypeError("Values for the key {} must be the same type (dict or list), "
+                                "but got {} and {}".format(k, type(dct1[k]), type(v)))
+        else:
+            dct1[k] = v
+
+
 class LinuxPerfProfile(ProfileImpl):
     def __init__(self):
         pass
 
     @staticmethod
     def checkFile(fn):
         with open(fn, 'rb') as f:
             return f.read(8) == b'PERFILE2'
 
     @staticmethod
     def deserialize(f, objdump='objdump', propagateExceptions=False,
                     binaryCacheRoot=''):
         f = f.name
 
         if os.path.getsize(f) == 0:
             # Empty file - exit early.
             return None
 
         try:
-            data = cPerf.importPerf(f, objdump, binaryCacheRoot)
+            data = {}
+            for fname in glob.glob("%s*" % f):
+                cur_data = cPerf.importPerf(fname, objdump, binaryCacheRoot)
+                merge_recursively(data, cur_data)
 
             # Go through the data and convert counter values to percentages.
             for f in data['functions'].values():
                 fc = f['counters']
                 for inst_info in f['data']:
                     for k, v in inst_info[0].items():
                         inst_info[0][k] = 100.0 * float(v) / fc[k]
                 for k, v in fc.items():