[LNT] Come up with MIN_PERCENTAGE_CHANGE value

[marxin]

Summary

Currently it's hard-coded what's considered as minimal visible change. I would like to factor it out to a constant
that one can adjust. What about putting that constant to lnt.conf?

[dmgreen]

This looks like something that would be useful for us, where some of our benchmarks are very low noise. I currently have a couple of test-ish LNT instances with changes similar to this, either changing this value or setting ignore_small=False from the daily report page.

Making it configurable sounds like a great idea.

lnt/server/reporting/analysis.py

188 ↗

(On Diff #169198)

This comment looks like it needs a reflow, or maybe a rewrite?

[cmatthews]

LGTM.

lnt/trunk/lnt/server/db/rules/rule_update_fixed_regressions.py

 """Check if a regression is fixed, and move to differnt sate.
 Detcted + fixed -> Ignored
 Staged or Active + fixed -> Verify
 """
 from sqlalchemy.orm.session import Session
 
 from lnt.server.db.regression import RegressionState
 from lnt.server.db.regression import get_cr_for_field_change, get_ris
 from lnt.server.db.testsuitedb import TestSuiteDB
 from lnt.testing.util.commands import timed
 from lnt.util import logger
-
+from lnt.server.reporting.analysis import MIN_PERCENTAGE_CHANGE
 
 def _fixed_rind(session, ts, rind):
     """Is this regression indicator fixed?"""
     fc = rind.field_change
     if fc is None:
         return False
     current_cr, _, _ = get_cr_for_field_change(session, ts, fc, current=True)
-    if current_cr.pct_delta < 0.01:
+    if current_cr.pct_delta < MIN_PERCENTAGE_CHANGE:
         return True
     else:
         return False
 
 
 def is_fixed(session, ts, regression):
     """Comparing the current value to the regression, is this regression now
     fixed?

lnt/trunk/lnt/server/reporting/analysis.py

 REGRESSED = 'REGRESSED'
 IMPROVED = 'IMPROVED'
 UNCHANGED_PASS = 'UNCHANGED_PASS'
 UNCHANGED_FAIL = 'UNCHANGED_FAIL'
 
 # The smallest measureable change we can detect in seconds.
 MIN_VALUE_PRECISION = 0.0001
 
+# Minimal percentage difference that is visible in reports
+MIN_PERCENTAGE_CHANGE = .01
 
 def absmin_diff(current, prevs):
     """Min of differences between current sample and all previous samples.
     Given more than one min, use the last one detected which is probably a
     newer value. Returns (difference, prev used)
     """
     diffs = [abs(current-prev) for prev in prevs]
     smallest_pos = 0
def get_value_status(self, confidence_interval=2.576,
         # which flop to failure then back. What would be nice to do here is to
         # find the last value in a passing run, or to move to using proper
         # keyed reference runs.
         if self.failed:
             return UNCHANGED_FAIL
         elif self.prev_failed:
             return UNCHANGED_PASS
 
-        # Always ignore percentage changes below 1%, for now, we just don't
+        # Always ignore percentage changes below MIN_PERCENTAGE_CHANGE %, for now, we just don't
         # have enough time to investigate that level of stuff.
-        if ignore_small and abs(self.pct_delta) < .01:
+        if ignore_small and abs(self.pct_delta) < MIN_PERCENTAGE_CHANGE:
             return UNCHANGED_PASS
 
         # Always ignore changes with small deltas. There is no mathematical
         # basis for this, it should be obviated by appropriate statistical
         # checks, but practical evidence indicates what we currently have isn't
         # good enough (for reasons I do not yet understand).
-        if ignore_small and abs(self.delta) < .01:
+        if ignore_small and abs(self.delta) < MIN_PERCENTAGE_CHANGE:
             return UNCHANGED_PASS
 
         # Ignore tests whose delta is too small relative to the precision we
         # can sample at; otherwise quantization means that we can't measure the
         # standard deviation with enough accuracy.
         if abs(self.delta) <= 2 * value_precision * confidence_interval:
             return UNCHANGED_PASS