[LNT][RFC] Introduce Latest Runs Report.

[marxin]

Summary

We've started using LNT with about 80 machines and we noticed it's crucial to check regressions and progress on a single place.
Daily Report looked very promising tool, but I found some limitations:

• we have machines that very rare frequency (once a month), so that only way to see a progress is to use num_days=big_value
• you need to have a Run that ends precisely in Day 0, it's not much handy if you want to see recent activity
• the Daily Report does quite some juggling with Orders and selects the recent one in a day frame
• that makes the code very complex (240 LOC)

Thus I made a simple report, called it for now Latest Runs. It jsut takes last N runs of each machine and does comparison. From implementation
point of view it consists of only 80 LOC and is very simple.

I also factored out common code shared in between Daily Report and the new report. Hope it can help
acceptance of the patch.

Static example of the report: https://drive.google.com/file/d/1PPFYugVg4Q3lhYxG6q2J4BUifj91JQkI/view?usp=sharing

[MatzeB]

The new report and its implementation look good to me! However before committing this needs at least a basic unit test (tests/server/ui/V4Pages.py) to check that the page renders without exception and produces valid html.

[cmatthews]

Testcases?

lnt/server/reporting/dailyreport.py

3 ↗

(On Diff #163022)

Please use explicit imports.

lnt/server/reporting/latestrunsreport.py

3 ↗

(On Diff #163022)

no * imports

lnt/server/ui/templates/reporting/latest_runs_report.html

9 ↗

(On Diff #163022)

Does something else close this body tag, or do you need to?

lnt/server/ui/views.py

12 ↗

(On Diff #163022)

Are you using these in your patch? I don't see them.

1532 ↗

(On Diff #163022)

+ comment on why this can happen.

1533 ↗

(On Diff #163022)

400 is "the server could not understand the request due to invalid syntax." This endpoint has no arguments, so that does not make sense. What is the error condition this is responding to?

[cmatthews]

lnt/server/reporting/latestrunsreport.py

101 ↗

(On Diff #163022)

no space around keywords.

lnt/server/ui/views.py

1521 ↗

(On Diff #163022)

Not a redirect (http 302).

1526 ↗

(On Diff #163022)

Sort of obvious comment.

1527 ↗

(On Diff #163022)

It might be handy to have the report length as a url parameter and default to 10. Sometimes the reports can be very slow, so it is nice to tweak how much data they look at.

[cmatthews]

This is a great idea. Thanks for setting it up!

[MatzeB]

lnt/server/reporting/dailyreport.py

3 ↗

(On Diff #163022)

+1

434–452 ↗

(On Diff #163022)

where did this block of code go to and where is report_css_styles coming from, is there a file missing?

[marxin]

Changes from previous version:

• I removed try, except in v4_latest_runs_report
• I added at least check_html test, once installed I add some real checking content
• accordion toggle added to HTML report, it's easier to navigate now
• all nits you mentioned should be fixed now

lnt/server/reporting/dailyreport.py

434–452 ↗

(On Diff #163022)

I isolated styles into report_css_styles which is now shared by both reports.

[marxin]

One more update where I add accordion for easier orientation on the report page.

[marxin]

May I please ping this.

[marxin]

May I please remind this review..

[cmatthews]

I bet some of these report queries are going to be really slow on big databases. But we can fix them up over time if that is the case. LGTM.

lnt/trunk/lnt/server/reporting/dailyreport.py

 from collections import namedtuple
 from lnt.server.reporting.analysis import REGRESSED, UNCHANGED_FAIL
+from lnt.server.reporting.report import RunResult, RunResults, report_css_styles, pairs, OrderAndHistory
 from lnt.util import multidict
-import colorsys
 import datetime
 import lnt.server.reporting.analysis
 import lnt.server.ui.app
 import re
 import sqlalchemy.sql
 import urllib
 
-OrderAndHistory = namedtuple('OrderAndHistory', ['max_order', 'recent_orders'])
-
-
-def _pairs(list):
-    return zip(list[:-1], list[1:])
-
-
-# The hash color palette avoids green and red as these colours are already used
-# in quite a few places to indicate "good" or "bad".
-_hash_color_palette = (
-    colorsys.hsv_to_rgb(h=45. / 360, s=0.3, v=0.9999),  # warm yellow
-    colorsys.hsv_to_rgb(h=210. / 360, s=0.3, v=0.9999),  # blue cyan
-    colorsys.hsv_to_rgb(h=300. / 360, s=0.3, v=0.9999),  # mid magenta
-    colorsys.hsv_to_rgb(h=150. / 360, s=0.3, v=0.9999),  # green cyan
-    colorsys.hsv_to_rgb(h=225. / 360, s=0.3, v=0.9999),  # cool blue
-    colorsys.hsv_to_rgb(h=180. / 360, s=0.3, v=0.9999),  # mid cyan
-)
-
-
-def _clamp(v, minVal, maxVal):
-    return min(max(v, minVal), maxVal)
-
-
-def _toColorString(col):
-    r, g, b = [_clamp(int(v * 255), 0, 255)
-               for v in col]
-    return "#%02x%02x%02x" % (r, g, b)
-
-
-def _get_rgb_colors_for_hashes(hash_strings):
-    hash2color = {}
-    unique_hash_counter = 0
-    for hash_string in hash_strings:
-        if hash_string is not None:
-            if hash_string in hash2color:
-                continue
-            hash2color[hash_string] = _hash_color_palette[unique_hash_counter]
-            unique_hash_counter += 1
-            if unique_hash_counter >= len(_hash_color_palette):
-                break
-    result = []
-    for hash_string in hash_strings:
-        if hash_string is None:
-            result.append(None)
-        else:
-            # If not one of the first N hashes, return rgb value 0,0,0 which is
-            # white.
-            rgb = hash2color.get(hash_string, (0.999, 0.999, 0.999))
-            result.append(_toColorString(rgb))
-    return result
-
-
-# Helper classes to make the sparkline chart construction easier in the jinja
-# template.
-class DayResult:
-    def __init__(self, comparisonResult):
-        self.cr = comparisonResult
-        self.hash = self.cr.cur_hash
-        self.samples = self.cr.samples
-        if self.samples is None:
-            self.samples = []
-
-
-class DayResults:
-    """
-    DayResults contains pre-processed data to easily construct the HTML for
-    a single row in the results table, showing how one test on one board
-    evolved over a number of runs/days.
-    """
-    def __init__(self):
-        self.day_results = []
-        self._complete = False
-        self.min_sample = None
-        self.max_sample = None
-
-    def __getitem__(self, i):
-        return self.day_results[i]
-
-    def __len__(self):
-        return len(self.day_results)
-
-    def append(self, day_result):
-        assert not self._complete
-        self.day_results.append(day_result)
-
-    def complete(self):
-        """
-        complete() needs to be called after all appends to this object, but
-        before the data is used the jinja template.
-        """
-        self._complete = True
-        all_samples = []
-        for dr in self.day_results:
-            if dr is None:
-                continue
-            if dr.cr.samples is not None and not dr.cr.failed:
-                all_samples.extend(dr.cr.samples)
-        if len(all_samples) > 0:
-            self.min_sample = min(all_samples)
-            self.max_sample = max(all_samples)
-        hashes = []
-        for dr in self.day_results:
-            if dr is None:
-                hashes.append(None)
-            else:
-                hashes.append(dr.hash)
-        rgb_colors = _get_rgb_colors_for_hashes(hashes)
-        for i, dr in enumerate(self.day_results):
-            if dr is not None:
-                dr.hash_rgb_color = rgb_colors[i]
-
-
 class DailyReport(object):
     def __init__(self, ts, year, month, day, num_prior_days_to_include=3,
                  day_start_offset_hours=16, for_mail=False,
                  filter_machine_regex=None):
         self.ts = ts
         self.hash_of_binary_field = self.ts.Sample.get_hash_of_binary_field()
         self.num_prior_days_to_include = num_prior_days_to_include
         self.year = year
def build(self, session):
         self.prior_days = [(datetime.datetime.fromordinal(day_ordinal - i) +
                             self.day_start_offset)
                            for i in range(self.num_prior_days_to_include + 1)]
 
         # Find all the runs that occurred for each day slice.
         prior_runs = [session.query(ts.Run).
                       filter(ts.Run.start_time > prior_day).
                       filter(ts.Run.start_time <= day).all()
-                      for day, prior_day in _pairs(self.prior_days)]
+                      for day, prior_day in pairs(self.prior_days)]
 
         if self.filter_machine_re is not None:
             prior_runs = [[run for run in runs
                            if self.filter_machine_re.search(run.machine.name)]
                           for runs in prior_runs]
 
         # For every machine, we only want to report on the last run order that
         # was reported for that machine for the particular day range.
def build(self, session):
                         day_runs, prev_runs, test.id, field,
                         self.hash_of_binary_field)
 
                     # If the result is not "interesting", ignore this machine.
                     if not cr.is_result_interesting():
                         continue
 
                     # Otherwise, compute the results for all the days.
-                    day_results = DayResults()
-                    day_results.append(DayResult(cr))
+                    day_results = RunResults()
+                    day_results.append(RunResult(cr))
                     for i in range(1, self.num_prior_days_to_include):
                         day_runs = machine_runs.get((machine.id, i), ())
                         if len(day_runs) == 0:
                             day_results.append(None)
                             continue
 
                         prev_day_index = find_most_recent_run_with_samples(i)
                         prev_runs = self.machine_past_runs.get(
                                        (machine.id, prev_day_index), ())
                         cr = sri.get_comparison_result(
                             day_runs, prev_runs, test.id, field,
                             self.hash_of_binary_field)
-                        day_results.append(DayResult(cr))
+                        day_results.append(RunResult(cr))
 
                     day_results.complete()
 
                     # Append the result for the machine.
                     visible_results.append((machine, day_results))
 
                 # If there are visible results for this test, append it to the
                 # view.
class DailyReport(object):
     def render(self, ts_url, only_html_body=True):
         # Strip any trailing slash on the testsuite URL.
         if ts_url.endswith('/'):
             ts_url = ts_url[:-1]
 
         env = lnt.server.ui.app.create_jinja_environment()
         template = env.get_template('reporting/daily_report.html')
 
-        # Compute static CSS styles for elements. We use the style directly on
-        # elements instead of via a stylesheet to support major email clients
-        # (like Gmail) which can't deal with embedded style sheets.
-        #
-        # These are derived from the static style.css file we use elsewhere.
-        styles = {
-            "body": ("color:#000000; background-color:#ffffff; "
-                     "font-family: Helvetica, sans-serif; font-size:9pt"),
-            "table": ("font-size:9pt; border-spacing: 0px; "
-                      "border: 1px solid black"),
-            "th": (
-                "background-color:#eee; color:#666666; font-weight: bold; "
-                "cursor: default; text-align:center; font-weight: bold; "
-                "font-family: Verdana; padding:5px; padding-left:8px"),
-            "td": "padding:5px; padding-left:8px",
-        }
-
         return template.render(
-            report=self, styles=styles, analysis=lnt.server.reporting.analysis,
+            report=self, styles=report_css_styles, analysis=lnt.server.reporting.analysis,
             ts_url=ts_url, only_html_body=only_html_body)

lnt/trunk/lnt/server/ui/templates/layout.html

<div id="wrap">
                 {% if g.testsuite_name is defined %}
                     <ul class="nav">
                         <li class="dropdown">
                             <a href="#" class="dropdown-toggle" data-toggle="dropdown">{{g.testsuite_name}}<b class="caret"></b></a>
                             <ul class="dropdown-menu">
                               <li><a href="{{ v4_url_for('.v4_recent_activity') }}">Recent Activity</a></li>
                               <li><a href="{{ v4_url_for('.v4_global_status') }}">Global Status</a></li>
                               <li><a href="{{ v4_url_for('.v4_daily_report_overview') }}">Daily Report</a></li>
+                              <li><a href="{{ v4_url_for('.v4_latest_runs_report') }}">Latest Runs Report</a></li>
                               <li><a href="{{ v4_url_for('.v4_machines') }}">All Machines</a></li>
                                <li class="divider"></li>
                               <li class="disabled"><a href="#">Changes</a></li>
                               <li><a href="{{ v4_url_for('.v4_regression_list', state=0) }}">Detected</a></li>
                               <li><a href="{{ v4_url_for('.v4_regression_list', state=1) }}">Staged</a></li>
                               <li><a href="{{ v4_url_for('.v4_regression_list', state=10) }}">Active</a></li>
                               <li><a href="{{ v4_url_for('.v4_regression_list', state=20) }}">NTBF</a></li>
                               <li><a href="{{ v4_url_for('.v4_regression_list', state=21) }}">Ignored</a></li>

lnt/trunk/lnt/server/ui/templates/reporting/daily_report.html

 {% import "utils.html" as utils %}
+{% import "reportutils.html" as reportutils %}
 {% if not only_html_body %}
 <html>
 <head>
    <title>Daily Report: {{
      '%04d-%02d-%02d' % (report.year, report.month, report.day) }}</title>
 </head>
 <body style="{{ styles['body'] }}">
 {% endif %}
<td class="machine-name" style="{{ styles.td }}">{{machine.name}}</td>
             >{{nr_tests}}</td>
         {% set nr_tests_prev = nr_tests %}
     {% endfor %}
   </tr>
   {% endfor %}
   </tbody>
 </table>
 
-{% macro get_initial_cell_value(day_result) %}
-{%- set cr = day_result.cr -%}
-{%- set test_status = cr.get_test_status() -%}
-
-{%- if (test_status == analysis.REGRESSED or
-       test_status == analysis.UNCHANGED_FAIL) %}
-    <td style="{{ styles.td }}; background-color:#e98080">FAIL</td>
-{%- else %}
-    <td style="{{ styles.td }}; background-color:#d2d2d2"> {#- -#}
-      {{ ("%.4f" % cr.current) if cr.current != none else "N/A" }}</td>
-{%- endif -%}
-
-{% endmacro %}
-
-{% macro get_cell_value(day_result) %}
-{%- set cr = day_result.cr -%}
-{%- set test_status = cr.get_test_status() -%}
-{%- set value_status = cr.get_value_status() -%}
-{%- if (test_status == analysis.REGRESSED or
-       test_status == analysis.UNCHANGED_FAIL) %}
-    <td style="{{ styles.td }}; background-color:#e98080">FAIL</td>
-{%- elif test_status == analysis.IMPROVED %}
-    <td style="{{ styles.td }}; background-color:#8fdf5f">PASS</td>
-{%- else -%}
-{%- if (value_status == analysis.REGRESSED or
-       value_status == analysis.IMPROVED) %}
-    {{ cr.pct_delta|aspctcell(reverse=cr.bigger_is_better)|safe }}
-{%- else %}
-    <td style="{{ styles.td }}">-</td>
-{%- endif -%}
-{%- endif -%}
-{% endmacro %}
-
 {# Generate the table showing the raw sample data. #}
 
 {# If the report is for mail, we put the table header on each test. This is
    ugly, however it makes it much easier to inline comments in replies to the
    email. Given that is the primary purpose of the email, we trade off wasted
    space for convenience. #}
 {% macro result_header() %}
 <table border="1" style="{{ styles.table }}; width: {{
                             400 + report.num_prior_days_to_include*80}}px;">
   <thead>
     <tr>
       <th style="{{ styles.th }}; width: 100px;">Test Name</th>
       <th style="{{ styles.th }}; width: 300px;">Machine Name</th>
 {%- for i in range(report.num_prior_days_to_include)|reverse %}
       <th style="{{ styles.th }}">Day - {{i}}</th>
 {%- endfor %}
       <th style="{{ styles.th }}">Sparkline</th>
     </tr>
   </thead>
 {% endmacro %}
 
-{% macro spark_plot(day_results) %}
-{%- set x_border_size = 5 %}
-{%- set y_border_size = 2 %}
-{%- set height = 18 %}
-{%- set full_height = height + 2*y_border_size %}
-{%- set x_day_spacing = 10 %}
-{%- set sample_fuzzing = 0.5 %}
-{%- set nr_days = day_results|length %}
-{%- set width = x_day_spacing * nr_days + 2*x_border_size %}
-{%- if day_results.max_sample != day_results.min_sample %}
-  {%- set scaling_factor = (1.0*height)
-         / (day_results.max_sample-day_results.min_sample) -%}
-{%- else %}
-  {%- set scaling_factor = 1.0 -%}
-{%- endif %}
-{%- macro spark_y_coord(day_nr, value) -%}
-  {{ (value-day_results.min_sample) * scaling_factor + y_border_size }}
-{%- endmacro -%}
-{%- macro spark_x_coord(day_nr) -%}
-  {{ (nr_days - day_nr) * x_day_spacing + x_border_size }}
-{%- endmacro -%}
-{%- macro spark_hash_background(day_nr, dr) -%}
-  {%- if dr.cr.cur_hash is not none -%}
-    {%- set style = "fill: "+dr.hash_rgb_color+";" -%}
-  {%- else -%}
-    {%- set style = "fill: none;" -%}
-  {%- endif -%}
-    <rect x="{{(nr_days-day_nr-0.5) * x_day_spacing + x_border_size}}" y="0"
-          width="{{x_day_spacing}}" height="{{full_height}}" style="{{style}}"/>
-{%- endmacro -%}
-    <span>
-      <svg width="{{width}}" height="{{full_height}}">
-      <rect width="{{width}}" height="{{full_height}}" fill="#f7f7f7"/>
-{#- Make y-axis go upwards instead of downwards: #}
-      <g transform="translate(0, {{full_height}}) scale(1, -1) ">
-{%- for dr in day_results -%}
-  {%- if dr is not none and not dr.cr.failed -%}
-    {%- set day_nr = loop.index %}
-    {%- set nr_samples_for_day = dr.samples|length %}
-        {{ spark_hash_background(day_nr, dr) }}
-    {%- for sample in dr.samples -%}
-      {# fuzz the x-coordinate slightly so that multiple samples with the same
-         value can be noticed #}
-      {%- set sample_fuzz = (-sample_fuzzing*1.25) +
-                       (2.0*sample_fuzzing/nr_samples_for_day) * loop.index %}
-        <circle cx="{{ spark_x_coord(day_nr)|float + sample_fuzz }}" {# -#}
-                cy="{{ spark_y_coord(day_nr, sample) }}" r="1"
-                stroke-width="1" stroke="black" fill="black" />
-    {%- endfor -%}
-  {%- endif -%}
-{%- endfor %}
-        <polyline points="
-  {%- for dr in day_results -%}
-    {%- if dr is not none -%}
-      {%- set cr = dr.cr -%}
-      {%- set day_nr = loop.index -%}
-      {%- if not cr.failed and cr.current is not none %}
-          {{ spark_x_coord(day_nr) }} {{ spark_y_coord(day_nr, cr.current) }}
-      {%- endif -%}
-    {%- endif -%}
-  {%- endfor -%}
-          " fill="none" stroke="red" stroke-width="1"/>
-        </g>
-      </svg>
-    </span>
-{%- endmacro %}
-
 {% for field,field_results in report.result_table|reverse %}
 {%- if field_results -%}
 <h3>Result Table ({{ field.display_name }})</h3>
 {{ result_header()+"\n  <tbody>" if not report.for_mail }}
 {%- for test,visible_results in field_results -%}
 {{ result_header()+"\n  <tbody>" if report.for_mail }}
   <tr>
     <td style="{{ styles.td }}" colspan="2"> <b>{{test.name}}</b></td>
     <td style="{{ styles.td }}" colspan="{{report.num_prior_days_to_include + 1}}">&nbsp;</td>
   </tr>
 {%-   for machine,day_results in visible_results -%}
 {%-   set key_run = report.get_key_run(machine, 0) %}
   <tr class="searchable">
     <td style="{{ styles.td }}">&nbsp;</td>
     <td class="machine-name" style="{{ styles.td }}"><a href="{{
       ts_url}}/graph?plot.0={{machine.id}}.{{test.id}}.{{
         report.ts.get_field_index(field)}}&amp;highlight_run={{key_run.id}}">{{
           machine.name}}</a></td>
 {%-    set first_result_shown = false -%}
 {%-    for day_result in day_results|reverse -%}
 {%-      if day_result is none %}
     <td style="{{ styles.td }}">-</td>
 {%-      else -%}
 {%-        if first_result_shown -%}
-    {{ get_cell_value(day_result) }}
+    {{ reportutils.get_cell_value(day_result, analysis, styles) }}
 {%-        else -%}
 {%-        set first_result_shown = true -%}
-    {{ get_initial_cell_value(day_result) }}
+    {{ reportutils.get_initial_cell_value(day_result, analysis, styles) }}
 {%-        endif -%}
 {%-      endif -%}
 {%-    endfor %}
-    <td>{{ spark_plot(day_results) }}</td>
+    <td>{{ reportutils.spark_plot(day_results) }}</td>
   </tr>
 {%-  endfor %}
 {{ "</tbody></table><p>" if report.for_mail }}
 {%- endfor %}
 {{ "</tbody></table>" if not report.for_mail }}
 {%- else -%}
 <h3>No significant {{ field.display_name }} changes found.</h3>
 {%- endif -%}
 {%- endfor -%}
 
 {% endif %}
 
 {% if not only_html_body %}
 </body>
 </html>
 {% endif %}

lnt/trunk/lnt/server/ui/views.py

 from typing import List, Optional, Text, Union
 from wtforms import SelectField, StringField, SubmitField
 from wtforms.validators import DataRequired, Length
 
 import lnt.server.db.rules_manager
 import lnt.server.db.search
 import lnt.server.reporting.analysis
 import lnt.server.reporting.dailyreport
+import lnt.server.reporting.latestrunsreport
 import lnt.server.reporting.runs
 import lnt.server.reporting.summaryreport
 import lnt.server.ui.util
 import lnt.util
 import lnt.util.ImportData
 import lnt.util.stats
 from lnt.external.stats import stats as ext_stats
 from lnt.server.reporting.analysis import ComparisonResult, calc_geomean
for ts in testsuites:
             all_orders.add(name)
     all_machines = sorted(all_machines)
     all_orders = sorted(all_orders, key=to_key)
 
     return render_template("v4_summary_report_ui.html",
                            config=config, all_machines=all_machines,
                            all_orders=all_orders, **ts_data(ts))
 
+@v4_route("/latest_runs_report")
+def v4_latest_runs_report():
+    session = request.session
+    ts = request.get_testsuite()
+
+    num_runs_str = request.args.get('num_runs')
+    if num_runs_str is not None:
+        num_runs = int(num_runs_str)
+    else:
+        num_runs = 10
+
+    report = lnt.server.reporting.latestrunsreport.LatestRunsReport(ts, num_runs)
+    report.build(request.session)
+
+    return render_template("v4_latest_runs_report.html", report=report,
+                           analysis=lnt.server.reporting.analysis,
+                           **ts_data(ts))
 
 @db_route("/summary_report")
 def v4_summary_report():
     session = request.session
     # Load the summary report configuration.
     config_path = get_summary_config_path()
     if not os.path.exists(config_path):
         return render_template("error.html", message="""\

lnt/trunk/tests/server/ui/V4Pages.py

def main():
     assert 4 == nr_sample_points, \
         "Expected 4 sample points, found %d" % nr_sample_points
     background_colors = extract_background_colors(sparkline_mhashonrun_xml, 3)
     assert len(background_colors) == 3
     color1, color2, color3 = background_colors
     assert color2 is None
     assert color1 != color3
 
+    # Check some variations of the latest runs report work.
+    check_html(client, '/v4/nts/latest_runs_report')
+
     check_redirect(client, '/db_default/submitRun',
                    '/db_default/v4/nts/submitRun')
     check_html(client, '/db_default/v4/nts/submitRun')
 
     check_html(client, '/v4/nts/global_status')
 
     check_html(client, '/v4/nts/recent_activity')