Keep sqlalchemy session separate from database objects

[MatzeB]

Summary

SQLAlchemy sessions should be handled independently of the lifetimes of
objects such as V4DB and TestSuiteDB (which basically represent sqlalchemy models).

See also the SQLAlchemy documentation:
http://docs.sqlalchemy.org/en/latest/orm/session_basics.html esp. the
"When do I construct a Session, when do I commit it, and when do I close
it?"; "As a general rule, keep the lifecycle of the session
separate and external from functions and objects that access and/or
manipulate database data. This will greatly help with achieving a
predictable and consistent transactional scope."

This requires a bunch of mechanical code changes to make sure the
session object is passed around separately and a session needs to be
created explicitely now rather than getting a long lived session as part
of V4DB.

Advantages:

• We can construct the DB object and the testsuite table descriptions once when LNT starts. This avoids a bunch of database queries and sqlalchemy model building logic running again for each request.
• We can switch session.rollback() to session.close() in an upcoming commit, avoid countless unnecessary database queries happening on rollback().
• Avoids a bunch of obscure caching effects with long lived sessions helping to clean up LNT code (the cleanup already happened in r312059)
• I hope this will help some of the server instabilities we experience (it's hard to pinpoint the cause though, it may be something different).

[kristof.beyls]

I have felt before that LNT's connection and interaction with the database is too obscure in the code.
Therefore, I like this proposal, even if it means passing around session objects more.

I've even wondered before if we shouldn't just get rid of SqlAlchemy and write sql directly to allow us to speed up some of the heavy queries.
That would make the interaction with the database in LNT's code clearer again, removing an obfuscating layer of indirection.
However, I do see that with now supporting 3 database engines (sqlite, postgres, mysql), we may not want to have to reinvent the wheel on handling the database differences that SqlAlchemy already handles.
Anyway, the SqlAlchemy thought is clearly independent of this particular change.

I've been out of the loop on python & database interactions for too many years to be able to properly review this, but as said above, I do like the direction this is taking LNT's design/code base.

[grosser]

I agree with Kristof. Even though I did not write DB code since a while, the reasoning sounds very good. Hope this fixes some of the errors we see!

[MatzeB]

In D37360#858537, @kristof.beyls wrote:

I have felt before that LNT's connection and interaction with the database is too obscure in the code.
Therefore, I like this proposal, even if it means passing around session objects more.

I've even wondered before if we shouldn't just get rid of SqlAlchemy and write sql directly to allow us to speed up some of the heavy queries.
That would make the interaction with the database in LNT's code clearer again, removing an obfuscating layer of indirection.
However, I do see that with now supporting 3 database engines (sqlite, postgres, mysql), we may not want to have to reinvent the wheel on handling the database differences that SqlAlchemy already handles.
Anyway, the SqlAlchemy thought is clearly independent of this particular change.

I've been out of the loop on python & database interactions for too many years to be able to properly review this, but as said above, I do like the direction this is taking LNT's design/code base.

Thanks for the review. Chris wanted to see this change too when I talked with him yesterday so I'm going ahead and commit it.

As for the rest: I feel the same way. I also think we would be better off with a set of helper functions to construct SQL queries rather than a full blown ORM layer. While doing simple things with sqlalchemy is simple enough, getting really accustomed to it and knowing how to debug and create performant queries takes weeks. This is unfortunate as most potential LNT developers will have a background in compiler technology and not in database programming.

That said I won't be the person rewriting all the code to use plain SQL (or some lower level abstraction as sqlalchemies DDL without using the full blown ORM); I'm also sure we can get the existing code working well so sqlalchemy is mainly a higher entry barriers for developing LNT (at least if someone wants to make fundamental changes).

lnt/trunk/lnt/lnttool/import_data.py

init_logger(logging.INFO if show_sql else logging.WARNING,
                 show_sql=show_sql)
 
     # Load the LNT instance.
     instance = lnt.server.instance.Instance.frompath(instance_path)
     config = instance.config
 
     # Get the database.
     with contextlib.closing(config.get_database(database)) as db:
+        session = db.make_session()
         # Load the database.
         success = True
         for file_name in files:
             result = lnt.util.ImportData.import_and_report(
-                config, database, db, file_name,
+                config, database, db, session, file_name,
                 output_format, testsuite, show_sample_count, no_email,
                 no_report, updateMachine=update_machine, mergeRun=merge)
 
             success &= result.get('success', False)
             if quiet:
                 continue
 
             if show_raw_result:
                 pprint.pprint(result)
             else:
                 lnt.util.ImportData.print_report_result(result, sys.stdout,
                                                         sys.stderr,
                                                         verbose)
 
         if not success:
             raise SystemExit(1)

lnt/trunk/lnt/lnttool/main.py

 @click.option("--testsuite", "-s", default='nts')
 def action_checkformat(files, testsuite):
     """check the format of LNT test report files"""
     import lnt.server.config
     import lnt.server.db.v4db
     import lnt.util.ImportData
     db = lnt.server.db.v4db.V4DB('sqlite:///:memory:',
                                  lnt.server.config.Config.dummy_instance())
+    session = db.make_session()
     for file in files:
         result = lnt.util.ImportData.import_and_report(
-            None, None, db, file, '<auto>', testsuite)
+            None, None, db, session, file, '<auto>', testsuite)
         lnt.util.ImportData.print_report_result(result, sys.stdout,
                                                 sys.stderr, verbose=True)
 
 
 @click.command("check-no-errors")
 @click.argument("files", nargs=-1, type=click.Path(exists=True))
 def action_check_no_errors(files):
     '''Check that report contains "no_error": true.'''
def action_send_daily_report(instance_path, address, database, testsuite, host,
     import smtplib
 
     # Load the LNT instance.
     instance = lnt.server.instance.Instance.frompath(instance_path)
     config = instance.config
 
     # Get the database.
     with contextlib.closing(config.get_database(database)) as db:
+        session = db.make_session()
 
         # Get the testsuite.
         ts = db.testsuite[testsuite]
 
         if today:
             date = datetime.datetime.utcnow()
         else:
             # Get a timestamp to use to derive the daily report to generate.
-            latest = ts.query(ts.Run).\
+            latest = session.query(ts.Run).\
                 order_by(ts.Run.start_time.desc()).limit(1).first()
 
             # If we found a run, use it's start time (rounded up to the next
             # hour, so we make sure it gets included).
             if latest:
                 date = latest.start_time + datetime.timedelta(hours=1)
             else:
                 # Otherwise, just use now.
                 date = datetime.datetime.utcnow()
 
         # Generate the daily report.
         logger.info("building report data...")
         report = lnt.server.reporting.dailyreport.DailyReport(
             ts, year=date.year, month=date.month, day=date.day,
             day_start_offset_hours=date.hour, for_mail=True,
             num_prior_days_to_include=days,
             filter_machine_regex=filter_machine_regex)
-        report.build()
+        report.build(session)
 
         logger.info("generating HTML report...")
         ts_url = "%s/db_%s/v4/%s" \
             % (config.zorgURL, database, testsuite)
         subject = "Daily Report: %04d-%02d-%02d" % (
             report.year, report.month, report.day)
         html_report = report.render(ts_url, only_html_body=False)
 
def action_send_run_comparison(instance_path, run_a_id, run_b_id, database,
     init_logger(logging.ERROR)
 
     # Load the LNT instance.
     instance = lnt.server.instance.Instance.frompath(instance_path)
     config = instance.config
 
     # Get the database.
     with contextlib.closing(config.get_database(database)) as db:
+        session = db.make_session()
 
         # Get the testsuite.
         ts = db.testsuite[testsuite]
 
         # Lookup the two runs.
         run_a_id = int(run_a_id)
         run_b_id = int(run_b_id)
-        run_a = ts.query(ts.Run).\
+        run_a = session.query(ts.Run).\
             filter_by(id=run_a_id).first()
-        run_b = ts.query(ts.Run).\
+        run_b = session.query(ts.Run).\
             filter_by(id=run_b_id).first()
         if run_a is None:
             logger.error("invalid run ID %r (not in database)" % (run_a_id,))
         if run_b is None:
             logger.error("invalid run ID %r (not in database)" % (run_b_id,))
 
         # Generate the report.
         data = lnt.server.reporting.runs.generate_run_data(
-            run_b, baseurl=config.zorgURL, result=None, compare_to=run_a,
-            baseline=None, aggregation_fn=min)
+            session, run_b, baseurl=config.zorgURL, result=None,
+            compare_to=run_a, baseline=None, aggregation_fn=min)
 
         env = lnt.server.ui.app.create_jinja_environment()
         text_template = env.get_template('reporting/run_report.txt')
         text_report = text_template.render(data)
         html_template = env.get_template('reporting/run_report.html')
         html_report = html_template.render(data)
 
         subject = data['subject']

lnt/trunk/lnt/lnttool/updatedb.py

def action_updatedb(instance_path, database, testsuite, tmp_dir, show_sql,
     init_logger(logging.INFO if show_sql else logging.WARNING,
                 show_sql=show_sql)
 
     # Load the instance.
     instance = lnt.server.instance.Instance.frompath(instance_path)
 
     # Get the database and test suite.
     with contextlib.closing(instance.get_database(database)) as db:
+        session = db.make_session()
         ts = db.testsuite[testsuite]
         order = None
         # Compute a list of all the runs to delete.
         if delete_order:
-            runs = ts.query(ts.Run).join(ts.Order) \
+            runs = session.query(ts.Run).join(ts.Order) \
                 .filter(ts.Order.id == delete_order).all()
         else:
-            runs = ts.query(ts.Run).filter(ts.Run.id.in_(delete_runs)).all()
+            runs = session.query(ts.Run) \
+                .filter(ts.Run.id.in_(delete_runs)).all()
         for run in runs:
-            ts.delete(run)
+            session.delete(run)
 
         if delete_machines:
-            machines = ts.query(ts.Machine) \
+            machines = session.query(ts.Machine) \
                 .filter(ts.Machine.name.in_(delete_machines)).all()
             for machine in machines:
-                ts.delete(machine)
+                session.delete(machine)
 
-        db.commit()
+        session.commit()

lnt/trunk/lnt/lnttool/viewcomparison.py

try:
                                           None)
         instance = lnt.server.instance.Instance(None, config)
 
         # Create the database.
         lnt.server.db.migrate.update_path(db_path)
 
         # Import the two reports.
         with contextlib.closing(config.get_database('default')) as db:
-            r = import_and_report(config, 'default', db, report_a, '<auto>',
-                                  testsuite, updateMachine=True)
-            import_and_report(config, 'default', db, report_b, '<auto>',
-                              testsuite, updateMachine=True)
+            session = db.make_session()
+            r = import_and_report(config, 'default', db, session, report_a,
+                                  '<auto>', testsuite, updateMachine=True)
+            import_and_report(config, 'default', db, session, report_b,
+                              '<auto>', testsuite, updateMachine=True)
 
             # Dispatch another thread to start the webbrowser.
             comparison_url = '%s/v4/nts/2?compare_to=1' % (url,)
             logger.info("opening comparison view: %s" % (comparison_url,))
 
             if not dry_run:
                 thread.start_new_thread(_start_browser, (comparison_url, True))
 

lnt/trunk/lnt/server/db/fieldchange.py

 from sqlalchemy import or_
 from lnt.server.db import rules_manager as rules
 # How many runs backwards to use in the previous run set.
 # More runs are slower (more DB access), but may provide
 # more accurate results.
 FIELD_CHANGE_LOOKBACK = 10
 
 
-def post_submit_tasks(ts, run_id):
-    regenerate_fieldchanges_for_run(ts, run_id)
+def post_submit_tasks(session, ts, run_id):
+    regenerate_fieldchanges_for_run(ts, session, run_id)
 
 
-def delete_fieldchange(ts, change):
+def delete_fieldchange(session, ts, change):
     """Delete this field change.  Since it might be attahed to a regression
     via regression indicators, fix those up too.  If this orphans a regression
     delete it as well."""
     # Find the indicators.
-    indicators = ts.query(ts.RegressionIndicator). \
+    indicators = session.query(ts.RegressionIndicator). \
         filter(ts.RegressionIndicator.field_change_id == change.id). \
         all()
     # And all the related regressions.
     regression_ids = [r.regression_id for r in indicators]
 
     # Remove the idicators that point to this change.
     for ind in indicators:
-        ts.delete(ind)
+        session.delete(ind)
 
     # Now we can remove the change, itself.
-    ts.delete(change)
+    session.delete(change)
 
     # We might have just created a regression with no changes.
     # If so, delete it as well.
     deleted_ids = []
     for r in regression_ids:
-        remaining = ts.query(ts.RegressionIndicator). \
+        remaining = session.query(ts.RegressionIndicator). \
             filter(ts.RegressionIndicator.regression_id == r). \
             all()
         if len(remaining) == 0:
-            r = ts.query(ts.Regression).get(r)
+            r = session.query(ts.Regression).get(r)
             logger.info("Deleting regression because it has not changes:" +
                         repr(r))
-            ts.delete(r)
+            session.delete(r)
             deleted_ids.append(r)
-    ts.commit()
+    session.commit()
     return deleted_ids
 
 
 @timed
-def regenerate_fieldchanges_for_run(ts, run_id):
+def regenerate_fieldchanges_for_run(ts, session, run_id):
     """Regenerate the set of FieldChange objects for the given run.
     """
     # Allow for potentially a few different runs, previous_runs, next_runs
     # all with the same order_id which we will aggregate together to make
     # our comparison result.
-    run = ts.getRun(run_id)
-    runs = ts.query(ts.Run). \
+    run = ts.getRun(session, run_id)
+    runs = session.query(ts.Run). \
         filter(ts.Run.order_id == run.order_id). \
         filter(ts.Run.machine_id == run.machine_id). \
         all()
 
-    previous_runs = ts.get_previous_runs_on_machine(run, FIELD_CHANGE_LOOKBACK)
-    next_runs = ts.get_next_runs_on_machine(run, FIELD_CHANGE_LOOKBACK)
+    previous_runs = ts.get_previous_runs_on_machine(session, run,
+                                                    FIELD_CHANGE_LOOKBACK)
+    next_runs = ts.get_next_runs_on_machine(session, run,
+                                            FIELD_CHANGE_LOOKBACK)
 
     # Find our start/end order.
     if previous_runs != []:
         start_order = previous_runs[0].order
     else:
         start_order = run.order
     if next_runs != []:
         end_order = next_runs[-1].order
     else:
         end_order = run.order
 
     # Load our run data for the creation of the new fieldchanges.
     runs_to_load = [r.id for r in (runs + previous_runs)]
 
     # When the same rev is submitted many times, the database accesses here
     # can be huge, and it is almost always an error to have the same rev
     # be used in so many runs.
     run_size = len(runs_to_load)
     if run_size > 50:
         logger.warning("Generating field changes for {} runs."
                        "That will be very slow.".format(run_size))
-    runinfo = lnt.server.reporting.analysis.RunInfo(ts, runs_to_load)
+    runinfo = lnt.server.reporting.analysis.RunInfo(session, ts, runs_to_load)
 
     # Only store fieldchanges for "metric" samples like execution time;
     # not for fields with other data, e.g. hash of a binary
     for field in list(ts.Sample.get_metric_fields()):
         for test_id in runinfo.test_ids:
             f = None
             result = runinfo.get_comparison_result(
                 runs, previous_runs, test_id, field,
                 ts.Sample.get_hash_of_binary_field())
             # Try and find a matching FC and update, else create one.
             try:
-                f = ts.query(ts.FieldChange) \
+                f = session.query(ts.FieldChange) \
                     .filter(ts.FieldChange.start_order == start_order) \
                     .filter(ts.FieldChange.end_order == end_order) \
                     .filter(ts.FieldChange.test_id == test_id) \
                     .filter(ts.FieldChange.machine == run.machine) \
-                    .filter(ts.FieldChange.field == field) \
+                    .filter(ts.FieldChange.field_id == field.id) \
                     .one()
             except sqlalchemy.orm.exc.NoResultFound:
                 f = None
 
             if not result.is_result_performance_change() and f:
                 # With more data, its not a regression. Kill it!
                 logger.info("Removing field change: {}".format(f.id))
-                deleted = delete_fieldchange(ts, f)
+                deleted = delete_fieldchange(session, ts, f)
                 continue
 
             if result.is_result_performance_change() and not f:
-                test = ts.query(ts.Test).filter(ts.Test.id == test_id).one()
+                test = session.query(ts.Test) \
+                    .filter(ts.Test.id == test_id) \
+                    .one()
                 f = ts.FieldChange(start_order=start_order,
                                    end_order=run.order,
                                    machine=run.machine,
                                    test=test,
-                                   field=field)
+                                   field_id=field.id)
                 # Check the rules to see if this change matters.
-                if rules.is_useful_change(ts, f):
-                    ts.add(f)
+                if rules.is_useful_change(session, ts, f):
+                    session.add(f)
                     try:
-                        found, new_reg = identify_related_changes(ts, f)
+                        found, new_reg = identify_related_changes(session, ts,
+                                                                  f)
                     except ObjectDeletedError:
                         # This can happen from time to time.
                         # So, lets retry once.
-                        found, new_reg = identify_related_changes(ts, f)
+                        found, new_reg = identify_related_changes(session, ts,
+                                                                  f)
 
                     if found:
                         logger.info("Found field change: {}".format(
                                     run.machine))
 
             # Always update FCs with new values.
             if f:
                 f.old_value = result.previous
                 f.new_value = result.current
                 f.run = run
-    ts.commit()
+    session.commit()
 
-    regressions = ts.query(ts.Regression).all()[::-1]
-    rules.post_submission_hooks(ts, regressions)
+    regressions = session.query(ts.Regression).all()[::-1]
+    rules.post_submission_hooks(session, ts, regressions)
 
 
 def is_overlaping(fc1, fc2):
     """"Returns true if these two orders intersect. """
     try:
         r1_min = fc1.start_order
         r1_max = fc1.end_order
         r2_min = fc2.start_order
def percent_similar(a, b):
     :param b: second string
     """
     # type: (str, str) -> float
     s = difflib.SequenceMatcher(lambda x: x.isdigit(), a, b)
     return s.ratio()
 
 
 @timed
-def identify_related_changes(ts, fc):
+def identify_related_changes(session, ts, fc):
     """Can we find a home for this change in some existing regression? If a
     match is found add a regression indicator adding this change to that
     regression, otherwise create a new regression for this change.
 
     Regression matching looks for regressions that happen in overlapping order
     ranges. Then looks for changes that are similar.
 
     """
-    regressions = ts.query(ts.Regression.id) \
+    regressions = session.query(ts.Regression.id) \
         .filter(or_(ts.Regression.state == RegressionState.DETECTED,
                 ts.Regression.state == RegressionState.DETECTED_FIXED)) \
         .all()
 
     for regression_packed in regressions:
         regression_id = regression_packed[0]
-        regression_indicators = get_ris(ts, regression_id)
+        regression_indicators = get_ris(session, ts, regression_id)
         print "RIs:", regression_indicators
         for change in regression_indicators:
             regression_change = change.field_change
             if is_overlaping(regression_change, fc):
                 confidence = 0.0
 
                 confidence += percent_similar(regression_change.machine.name,
                                               fc.machine.name)
                 confidence += percent_similar(regression_change.test.name,
                                               fc.test.name)
 
-                if regression_change.field == fc.field:
+                if regression_change.field_id == fc.field_id:
                     confidence += 1.0
 
                 if confidence >= 2.0:
                     # Matching
                     MSG = "Found a match: {} with score {}."
-                    regression = ts.query(ts.Regression).get(regression_id)
+                    regression = session.query(ts.Regression) \
+                        .get(regression_id)
                     logger.info(MSG.format(str(regression),
                                            confidence))
                     ri = ts.RegressionIndicator(regression, fc)
-                    ts.add(ri)
+                    session.add(ri)
                     # Update the default title if needed.
-                    rebuild_title(ts, regression)
-                    ts.commit()
+                    rebuild_title(session, ts, regression)
+                    session.commit()
                     return True, regression
     logger.info("Could not find a partner, creating new Regression for change")
-    new_reg = new_regression(ts, [fc.id])
+    new_reg = new_regression(session, ts, [fc.id])
     return False, new_reg

lnt/trunk/lnt/server/db/regression.py

names = {DETECTED: u'Detected',
              DETECTED_FIXED: u'Verify',
              FIXED: u'Fixed'
              }
 
 ChangeRuns = namedtuple("ChangeRuns", ["before", "after"])
 ChangeData = namedtuple("ChangeData", ["ri", "cr", "run", "latest_cr"])
 
 
-def new_regression(ts, field_changes):
+def new_regression(session, ts, field_changes):
     """Make a new regression and add to DB."""
     today = datetime.date.today()
     MSG = "Regression of 0 benchmarks"
     title = MSG
     regression = ts.Regression(title, "", RegressionState.DETECTED)
-    ts.add(regression)
+    session.add(regression)
     for fc_id in field_changes:
-        fc = get_fieldchange(ts, fc_id)
+        fc = get_fieldchange(session, ts, fc_id)
         ri1 = ts.RegressionIndicator(regression, fc)
-        ts.add(ri1)
-    rebuild_title(ts, regression)
-    ts.commit()
+        session.add(ri1)
+    rebuild_title(session, ts, regression)
+    session.commit()
     return regression
 
 
-def rebuild_title(ts, regression):
+def rebuild_title(session, ts, regression):
     """Update the title of a regresson."""
     if re.match("Regression of \d+ benchmarks.*", regression.title):
-        old_changes = ts.query(ts.RegressionIndicator) \
+        old_changes = session.query(ts.RegressionIndicator) \
             .filter(ts.RegressionIndicator.regression_id == regression.id) \
             .all()
         new_size = len(old_changes)
         benchmarks = set()
         for ri in old_changes:
             fc = ri.field_change
             benchmarks.add(shortname(fc.test.name))
         FMT = "Regression of {} benchmarks: {}"
         title = FMT.format(new_size, ', '.join(benchmarks))
         # Crop long titles.
         title = (title[:120] + '...') if len(title) > 120 else title
         regression.title = title
     return regression
 
 
-def get_all_orders_for_machine(ts, machine):
+def get_all_orders_for_machine(session, ts, machine):
     """Get all the oredrs for this sa machine."""
-    return ts.query(ts.Order) \
+    return session.query(ts.Order) \
         .join(ts.Run) \
         .filter(ts.Run.machine_id == machine) \
         .order_by(asc(ts.Order.llvm_project_revision)) \
         .all()
 
 
-def get_ris(ts, regression_id):
-    return ts.query(ts.RegressionIndicator) \
+def get_ris(session, ts, regression_id):
+    return session.query(ts.RegressionIndicator) \
         .filter(ts.RegressionIndicator.regression_id == regression_id) \
         .all()
 
 
-def get_runs_for_order_and_machine(ts, order_id, machine_id):
+def get_runs_for_order_and_machine(session, ts, order_id, machine_id):
     """Collect all the runs for a particular order/machine combo."""
-    runs = ts.query(ts.Run) \
+    runs = session.query(ts.Run) \
         .filter(ts.Run.machine_id == machine_id) \
         .filter(ts.Run.order_id == order_id) \
         .all()
     return runs
 
 
-def get_runs_of_fieldchange(ts, fc):
-    before_runs = get_runs_for_order_and_machine(ts, fc.start_order_id,
+def get_runs_of_fieldchange(session, ts, fc):
+    before_runs = get_runs_for_order_and_machine(session, ts,
+                                                 fc.start_order_id,
                                                  fc.machine_id)
-    after_runs = get_runs_for_order_and_machine(ts, fc.end_order_id,
+    after_runs = get_runs_for_order_and_machine(session, ts, fc.end_order_id,
                                                 fc.machine_id)
     return ChangeRuns(before_runs, after_runs)
 
 
-def get_current_runs_of_fieldchange(ts, fc):
-    before_runs = get_runs_for_order_and_machine(ts, fc.start_order_id,
+def get_current_runs_of_fieldchange(session, ts, fc):
+    before_runs = get_runs_for_order_and_machine(session, ts,
+                                                 fc.start_order_id,
                                                  fc.machine_id)
-    newest_order = get_all_orders_for_machine(ts, fc.machine_id)[-1]
+    newest_order = get_all_orders_for_machine(session, ts, fc.machine_id)[-1]
 
-    after_runs = get_runs_for_order_and_machine(ts, newest_order.id,
+    after_runs = get_runs_for_order_and_machine(session, ts, newest_order.id,
                                                 fc.machine_id)
     return ChangeRuns(before_runs, after_runs)
 
 
-def get_first_runs_of_fieldchange(ts, fc):
-    run = ts.query(ts.Run) \
+def get_first_runs_of_fieldchange(session, ts, fc):
+    run = session.query(ts.Run) \
         .filter(ts.Run.machine_id == fc.machine_id) \
         .filter(ts.Run.order_id == fc.end_order_id) \
         .first()
     return run
 
 
-def get_cr_for_field_change(ts, field_change, current=False):
+def get_cr_for_field_change(session, ts, field_change, current=False):
     """Given a filed_change, calculate a comparison result for that change.
     And the last run."""
     if current:
-        runs = get_current_runs_of_fieldchange(ts, field_change)
+        runs = get_current_runs_of_fieldchange(session, ts, field_change)
     else:
-        runs = get_runs_of_fieldchange(ts, field_change)
+        runs = get_runs_of_fieldchange(session, ts, field_change)
     runs_all = list(runs.before)
     runs_all.extend(runs.after)
-    ri = RunInfo(ts, [r.id for r in runs_all],
+    ri = RunInfo(session, ts, [r.id for r in runs_all],
                  only_tests=[field_change.test_id])
     cr = ri.get_comparison_result(runs.after, runs.before,
                                   field_change.test.id, field_change.field,
                                   ts.Sample.get_hash_of_binary_field())
     return cr, runs.after[0], runs_all
 
 
-def get_fieldchange(ts, fc_id):
+def get_fieldchange(session, ts, fc_id):
     """Get a fieldchange given an ID."""
-    return ts.query(ts.FieldChange).filter(ts.FieldChange.id == fc_id).one()
+    return session.query(ts.FieldChange) \
+        .filter(ts.FieldChange.id == fc_id).one()

lnt/trunk/lnt/server/db/rules/rule_blacklist_benchmarks_by_name.py

if path and os.path.isfile(path):
         logger.info("Loading blacklist file: {}".format(path))
         with open(path, 'r') as f:
             for l in f.readlines():
                 ignored.append(re.compile(l.strip()))
     else:
         logger.warning("Ignoring blacklist file: {}".format(path))
 
 
-def filter_by_benchmark_name(ts, field_change):
+def filter_by_benchmark_name(session, ts, field_change):
     """Is this a fieldchanges we care about?
     """
     if ignored is None:
         _populate_blacklist()
     benchmark_name = field_change.test.name
     ts_name = ts.name
     full_name = '.'.join([ts_name,
                           field_change.machine.name,

lnt/trunk/lnt/server/db/rules/rule_update_fixed_regressions.py

 """Check if a regression is fixed, and move to differnt sate.
 Detcted + fixed -> Ignored
 Staged or Active + fixed -> Verify
 """
 from lnt.server.db.regression import RegressionState
 from lnt.server.db.regression import get_cr_for_field_change, get_ris
 from lnt.util import logger
 from lnt.testing.util.commands import timed
 
 
-def _fixed_rind(ts, rind):
+def _fixed_rind(session, ts, rind):
     """Is this regression indicator fixed?"""
     fc = rind.field_change
     if fc is None:
         return False
-    current_cr, _, _ = get_cr_for_field_change(ts, fc, current=True)
+    current_cr, _, _ = get_cr_for_field_change(session, ts, fc, current=True)
     if current_cr.pct_delta < 0.01:
         return True
     else:
         return False
 
 
-def is_fixed(ts, regression):
+def is_fixed(session, ts, regression):
     """Comparing the current value to the regression, is this regression now
     fixed?
     """
-    r_inds = get_ris(ts, regression.id)
-    fixes = [_fixed_rind(ts, x) for x in r_inds]
+    r_inds = get_ris(session, ts, regression.id)
+    fixes = [_fixed_rind(session, ts, x) for x in r_inds]
     return all(fixes)
 
 
 @timed
-def regression_evolution(ts, run_id):
+def regression_evolution(session, ts, run_id):
     """Analyse regressions. If they have changes, process them.
     Look at each regression in state detect.  Move to ignore if it is fixed.
     Look at each regression in state stage. Move to verify if fixed.
     Look at regressions in detect, do they match our policy? If no, move to
     NTBF.
     """
     logger.info("Running regression evolution")
     changed = 0
     evolve_states = [RegressionState.DETECTED, RegressionState.STAGED,
                      RegressionState.ACTIVE]
-    regressions = ts.query(ts.Regression) \
+    regressions = session.query(ts.Regression) \
         .filter(ts.Regression.state.in_(evolve_states)) \
         .all()
 
     detects = [r for r in regressions if r.state == RegressionState.DETECTED]
     staged = [r for r in regressions if r.state == RegressionState.STAGED]
     active = [r for r in regressions if r.state == RegressionState.ACTIVE]
 
     for regression in detects:
-        if is_fixed(ts, regression):
+        if is_fixed(session, ts, regression):
             logger.info("Detected fixed regression" + str(regression))
             regression.state = RegressionState.IGNORED
             regression.title = regression.title + " [Detected Fixed]"
             changed += 1
 
     for regression in staged:
-        if is_fixed(ts, regression):
+        if is_fixed(session, ts, regression):
             logger.info("Staged fixed regression" + str(regression))
             regression.state = RegressionState.DETECTED_FIXED
             regression.title = regression.title + " [Detected Fixed]"
             changed += 1
 
     for regression in active:
-        if is_fixed(ts, regression):
+        if is_fixed(session, ts, regression):
             logger.info("Active fixed regression" + str(regression))
             regression.state = RegressionState.DETECTED_FIXED
             regression.title = regression.title + " [Detected Fixed]"
             changed += 1
-    ts.commit()
+    session.commit()
     logger.info("Changed the state of {} regressions".format(changed))
 
 post_submission_hook = regression_evolution

lnt/trunk/lnt/server/db/rules/rule_update_profile_stats.py

 """
 Post submission hook to write the current state of the profiles directory. This
 gets fed into the profile/admin page.
 """
 import datetime
 import glob
 import json
 import os
 import subprocess
 import time
 
 
-def update_profile_stats(ts, run_id):
+def update_profile_stats(session, ts, run_id):
     config = ts.v4db.config
 
     history_path = os.path.join(config.profileDir, '_profile-history.json')
     age_path = os.path.join(config.profileDir, '_profile-age.json')
     profile_path = config.profileDir
 
     if not os.path.exists(profile_path):
         return

lnt/trunk/lnt/server/db/rules_manager.py

for name, path in load_rules().items():
         execfile(path, globals)
         DESCRIPTIONS[name] = globals['__doc__']
         for hook_name in HOOKS.keys():
             if hook_name in globals:
                 HOOKS[hook_name].append(globals[hook_name])
     return HOOKS
 
 
-def post_submission_hooks(ts, run_id):
+def post_submission_hooks(session, ts, run_id):
     """Run all the post submission hooks on the submitted run."""
     for func in HOOKS['post_submission_hook']:
-        func(ts, run_id)
+        func(session, ts, run_id)
 
 
-def is_useful_change(ts, field_change):
+def is_useful_change(session, ts, field_change):
     """Run all the change filters. If any are false, drop this change."""
     all_filters = []
     for func in HOOKS['is_useful_change']:
-        decision = func(ts, field_change)
+        decision = func(session, ts, field_change)
         all_filters.append(decision)
     if len(all_filters) == 0:
         return True
     else:
         #  If any filter ignores, we ignore.
         return all(all_filters)

lnt/trunk/lnt/server/db/search.py

 import re
 
 
-def _naive_search_for_run(ts, query, num_results, default_machine):
+def _naive_search_for_run(session, ts, query, num_results, default_machine):
     """
     This 'naive' search doesn't rely on any indexes so can be used without
     full-text search enabled. This does make it less clever however.
 
     It is able to match queries for machine names and order numbers
     (specifically llvm_project_revision numbers). The revision numbers may be
     partial and may be preceded by '#' or 'r'. Any other non-integer tokens are
     considered to be partial matches for a machine name; any machine that
if not machine_queries and not default_machine:
         # No machines to query: no matches. We can't query all machines, we'd
         # end up doing a full table scan and that is not scalable.
         return []
 
     machines = []
     if not machine_queries:
         machines = [default_machine]
     else:
-        for m in ts.query(ts.Machine).all():
+        for m in session.query(ts.Machine).all():
             if all(q in m.name for q in machine_queries):
                 machines.append(m.id)
 
     if not machines:
         return []
 
     llvm_project_revision_idx = [i
                                  for i, f in enumerate(ts.Order.fields)
                                  if f.name == 'llvm_project_revision'][0]
     llvm_project_revision_col = \
         ts.Order.fields[llvm_project_revision_idx].column
 
-    q = ts.query(ts.Run) \
+    q = session.query(ts.Run) \
           .filter(ts.Run.machine_id.in_(machines)) \
           .filter(ts.Run.order_id == ts.Order.id) \
           .filter(llvm_project_revision_col.isnot(None))
     if order_queries:
         oq = '%' + str(order_queries[0]) + '%'
         q = q.filter(llvm_project_revision_col.like(oq))
 
     return q.order_by(ts.Run.id.desc()).limit(num_results).all()
 
 
-def search(ts, query,
+def search(session, ts, query,
            num_results=8, default_machine=None):
     """
     Performs a textual search for a run. The exact syntax supported depends on
     the engine used to perform the search; see _naive_search_for_run for the
     minimum supported syntax.
 
     ts: TestSuite object
     query: Textual query string
     num_results: Number of results to return
     default_machine: If no machines were specified (only orders), return
     results from this machine.
 
     Returns a list of Run objects.
     """
 
-    return _naive_search_for_run(ts, query,
+    return _naive_search_for_run(session, ts, query,
                                  num_results, default_machine)

lnt/trunk/lnt/server/db/testsuitedb.py

 """
 Database models for the TestSuite databases themselves.
 
 These are a bit magical because the models themselves are driven by the test
 suite metadata, so we only create the classes at runtime.
 """
 
 import datetime
 import json
 import os
 
 import aniso8601
 import sqlalchemy
-from flask import session
+import flask
 from sqlalchemy import *
 from sqlalchemy.orm import relation
 from sqlalchemy.orm.exc import ObjectDeletedError
 from typing import List
 
 import testsuite
 import lnt.testing.profile.profile as profile
 import lnt
def __init__(self, v4db, name, test_suite, create_tables=False):
                     value = data_dict.pop(field.name, None)
                     self.set_field(field, value)
 
         db_key_name = self.test_suite.db_key_name
 
         class Machine(self.base, ParameterizedMixin):
             __tablename__ = db_key_name + '_Machine'
 
-            DEFAULT_BASELINE_REVISION = self.v4db.baseline_revision
+            DEFAULT_BASELINE_REVISION = v4db.baseline_revision
 
             fields = self.machine_fields
             id = Column("ID", Integer, primary_key=True)
             name = Column("Name", String(256), index=True)
 
             # The parameters blob is used to store any additional information
             # reported by the run but not promoted into the machine record.
             # Such data is stored as a JSON encoded blob.
def __init__(self, v4db, name, test_suite, create_tables=False):
             def parameters(self):
                 """dictionary access to the BLOB encoded parameters data"""
                 return dict(json.loads(self.parameters_data))
 
             @parameters.setter
             def parameters(self, data):
                 self.parameters_data = json.dumps(sorted(data.items()))
 
-            def get_baseline_run(self):
+            def get_baseline_run(self, session):
                 ts = Machine.testsuite
-                user_baseline = ts.get_users_baseline()
+                user_baseline = ts.get_users_baseline(session)
                 if user_baseline:
                     return self.get_closest_previously_reported_run(
-                        user_baseline.order)
+                        session, user_baseline.order)
                 else:
                     mach_base = Machine.DEFAULT_BASELINE_REVISION
                     # If we have an int, convert it to a proper string.
                     if isinstance(mach_base, int):
                         mach_base = '% 7d' % mach_base
                     return self.get_closest_previously_reported_run(
-                        ts.Order(llvm_project_revision=mach_base))
+                        session, ts.Order(llvm_project_revision=mach_base))
 
-            def get_closest_previously_reported_run(self, order_to_find):
+            def get_closest_previously_reported_run(self, session,
+                                                    order_to_find):
                 """
                 Find the closest previous run to the requested order, for which
                 this machine also reported.
                 """
 
                 # FIXME: Scalability! Pretty fast in practice, but still.
                 ts = Machine.testsuite
                 # Search for best order.
                 best_order = None
-                for order in ts.query(ts.Order).\
+                for order in session.query(ts.Order).\
                         join(ts.Run).\
                         filter(ts.Run.machine_id == self.id).distinct():
                     if order >= order_to_find and \
                           (best_order is None or order < best_order):
                         best_order = order
 
                 # Find the most recent run on this machine that used
                 # that order.
                 closest_run = None
                 if best_order:
-                    closest_run = ts.query(ts.Run)\
+                    closest_run = session.query(ts.Run)\
                         .filter(ts.Run.machine_id == self.id)\
                         .filter(ts.Run.order_id == best_order.id)\
                         .order_by(ts.Run.start_time.desc()).first()
 
                 return closest_run
 
             def set_from_dict(self, data):
                 data_name = data.pop('name', None)
def __init__(self, v4db, name, test_suite, create_tables=False):
             def get_metric_fields():
                 """
                 get_metric_fields() -> [SampleField*]
 
                 Get the sample fields which represent some kind of metric, i.e.
                 those which have a value that can be interpreted as better or
                 worse than other potential values for this field.
                 """
-                for field in self.Sample.fields:
+                for field in Sample.fields:
                     if field.type.name in ['Real', 'Integer']:
                         yield field
 
             @staticmethod
             def get_hash_of_binary_field():
                 """
                 get_hash_of_binary_field() -> SampleField
 
def __init__(self, v4db, name, test_suite, create_tables=False):
             old_value = Column("OldValue", Float)
             new_value = Column("NewValue", Float)
             start_order_id = Column("StartOrderID", Integer,
                                     ForeignKey(Order.id))
             end_order_id = Column("EndOrderID", Integer, ForeignKey(Order.id))
             test_id = Column("TestID", Integer, ForeignKey(Test.id))
             machine_id = Column("MachineID", Integer, ForeignKey(Machine.id))
             field_id = Column("FieldID", Integer,
-                              ForeignKey(self.v4db.SampleField.id))
+                              ForeignKey(testsuite.SampleField.id))
             # Could be from many runs, but most recent one is interesting.
             run_id = Column("RunID", Integer, ForeignKey(Run.id))
 
             start_order = relation(Order, primaryjoin='FieldChange.'
                                    'start_order_id==Order.id')
             end_order = relation(Order, primaryjoin='FieldChange.'
                                  'end_order_id==Order.id')
             test = relation(Test)
             machine = relation(Machine)
-            field = relation(self.v4db.SampleField,
-                             primaryjoin=(self.v4db.SampleField.id ==
-                                          field_id))
+            field = relation(testsuite.SampleField)
             run = relation(Run)
 
             def __init__(self, start_order, end_order, machine,
-                         test, field):
+                         test, field_id):
                 self.start_order = start_order
                 self.end_order = end_order
                 self.machine = machine
-                self.field = field
                 self.test = test
+                self.field_id = field_id
 
             def __repr__(self):
                 return '%s_%s%r' % (db_key_name, self.__class__.__name__,
                                     (self.start_order, self.end_order,
                                      self.test, self.machine, self.field))
 
             def __json__(self):
                 return {
def __init__(self, v4db, name, test_suite, create_tables=False):
         self.RegressionIndicator = RegressionIndicator
         self.ChangeIgnore = ChangeIgnore
         self.Baseline = Baseline
 
         # Create the compound index we cannot declare inline.
         sqlalchemy.schema.Index("ix_%s_Sample_RunID_TestID" % db_key_name,
                                 Sample.run_id, Sample.test_id)
 
-        # Add several shortcut aliases, similar to the ones on the v4db.
-        self.session = self.v4db.session
-        self.add = self.v4db.add
-        self.delete = self.v4db.delete
-        self.commit = self.v4db.commit
-        self.query = self.v4db.query
-        self.rollback = self.v4db.rollback
-
         if create_tables:
             self.base.metadata.create_all(v4db.engine)
 
-    def get_baselines(self):
-        return self.query(self.Baseline).all()
+    def get_baselines(self, session):
+        return session.query(self.Baseline).all()
 
-    def get_users_baseline(self):
+    def get_users_baseline(self, session):
         try:
             baseline_key = lnt.server.ui.util.baseline_key(self.name)
-            session_baseline = session.get(baseline_key)
+            session_baseline = flask.session.get(baseline_key)
         except RuntimeError:
             # Sometimes this is called from outside the app context.
             # In that case, don't get the user's session baseline.
             return None
         if session_baseline:
-            return self.query(self.Baseline).get(session_baseline)
+            return session.query(self.Baseline).get(session_baseline)
 
         return None
 
-    def _getOrCreateMachine(self, machine_data, forceUpdate):
+    def _getOrCreateMachine(self, session, machine_data, forceUpdate):
         """
         _getOrCreateMachine(data, forceUpdate) -> Machine
 
         Add or create (and insert) a Machine record from the given machine data
         (as recorded by the test interchange format).
         """
 
         # Convert the machine data into a machine record.
         machine_parameters = machine_data.copy()
         name = machine_parameters.pop('name')
         machine = self.Machine(name)
         machine_parameters.pop('id', None)
         for item in self.machine_fields:
             value = machine_parameters.pop(item.name, None)
             machine.set_field(item, value)
         machine.parameters = machine_parameters
 
         # Look for an existing machine.
-        existing_machines = self.query(self.Machine) \
+        existing_machines = session.query(self.Machine) \
             .filter(self.Machine.name == name) \
             .order_by(self.Machine.id.desc()) \
             .all()
         if len(existing_machines) == 0:
-            self.add(machine)
+            session.add(machine)
             return machine
 
         existing = existing_machines[0]
 
         # Unfortunately previous LNT versions allowed multiple machines
         # with the same name to exist, so we should choose the one that
         # matches best.
         if len(existing_machines) > 1:
def _getOrCreateMachine(self, session, machine_data, forceUpdate):
             if existing_value is None or forceUpdate:
                 existing_parameters[key] = value
             else:
                 raise MachineInfoChanged("'%s' on machine '%s' changed." %
                                          (key, name))
         existing.parameters = existing_parameters
         return existing
 
-    def _getOrCreateOrder(self, run_parameters):
+    def _getOrCreateOrder(self, session, run_parameters):
         """
         _getOrCreateOrder(data) -> Order
 
         Add or create (and insert) an Order record based on the given run
         parameters (as recorded by the test interchange format).
 
         The run parameters that define the order will be removed from the
         provided ddata argument.
         """
 
-        query = self.query(self.Order)
+        query = session.query(self.Order)
         order = self.Order()
 
         # First, extract all of the specified order fields.
         for item in self.order_fields:
             value = run_parameters.pop(item.name, None)
             if value is None:
                 # We require that all of the order fields be present.
                 raise ValueError("Supplied run is missing parameter: %r" %
                                  (item.name))
 
             query = query.filter(item.column == value)
             order.set_field(item, value)
 
         # Execute the query to see if we already have this order.
         existing = query.first()
         if existing is not None:
             return existing
 
         # If not, then we need to insert this order into the total ordering
         # linked list.
 
         # Add the new order and commit, to assign an ID.
-        self.add(order)
-        self.v4db.session.commit()
+        session.add(order)
+        session.commit()
 
         # Load all the orders.
-        orders = list(self.query(self.Order))
+        orders = list(session.query(self.Order))
 
         # Sort the objects to form the total ordering.
         orders.sort()
 
         # Find the order we just added.
         index = orders.index(order)
 
         # Insert this order into the linked list which forms the total
         # ordering.
         if index > 0:
             previous_order = orders[index - 1]
             previous_order.next_order_id = order.id
             order.previous_order_id = previous_order.id
         if index + 1 < len(orders):
             next_order = orders[index + 1]
             next_order.previous_order_id = order.id
             order.next_order_id = next_order.id
 
         return order
 
-    def _getOrCreateRun(self, run_data, machine, merge):
+    def _getOrCreateRun(self, session, run_data, machine, merge):
         """
-        _getOrCreateRun(run_data, machine, merge) -> Run, bool
+        _getOrCreateRun(session, run_data, machine, merge) -> Run, bool
 
         Add a new Run record from the given data (as recorded by the test
         interchange format).
 
         merge comes into play when there is already a run with the same order
         fields:
         - 'reject': Reject submission (raise ValueError).
         - 'replace': Remove the existing submission(s), then add the new one.
def _getOrCreateRun(self, session, run_data, machine, merge):
         # Added by REST API, we will replace as well.
         run_parameters.pop('order_by', None)
         run_parameters.pop('order_id', None)
         run_parameters.pop('machine_id', None)
         run_parameters.pop('imported_from', None)
         run_parameters.pop('simple_run_id', None)
 
         # Find the order record.
-        order = self._getOrCreateOrder(run_parameters)
+        order = self._getOrCreateOrder(session, run_parameters)
 
         if merge != 'append':
-            existing_runs = self.query(self.Run) \
+            existing_runs = session.query(self.Run) \
                 .filter(self.Run.machine_id == machine.id) \
                 .filter(self.Run.order_id == order.id) \
                 .all()
             if len(existing_runs) > 0:
                 if merge == 'reject':
                     raise ValueError("Duplicate submission for '%s'" %
                                      order.name)
                 elif merge == 'replace':
                     for run in existing_runs:
-                        self.delete(run)
+                        session.delete(run)
                 else:
                     raise ValueError('Invalid Run mergeStrategy %r' % merge)
 
         # We'd like ISO8061 timestamps, but will also accept the old format.
         try:
             start_time = aniso8601.parse_datetime(run_data['start_time'])
         except ValueError:
             start_time = datetime.datetime.strptime(run_data['start_time'],
def _getOrCreateRun(self, session, run_data, machine, merge):
 
         # First, extract all of the specified run fields.
         for item in self.run_fields:
             value = run_parameters.pop(item.name, None)
             run.set_field(item, value)
 
         # Any remaining parameters are saved as a JSON encoded array.
         run.parameters = run_parameters
-        self.add(run)
+        session.add(run)
         return run
 
-    def _importSampleValues(self, tests_data, run, config):
+    def _importSampleValues(self, session, tests_data, run, config):
         # Load a map of all the tests, which we will extend when we find tests
         # that need to be added.
         # Downcast to str, so we match on MySQL.
         test_cache = dict((str(test.name), test)
-                          for test in self.query(self.Test))
+                          for test in session.query(self.Test))
 
         profiles = dict()
         field_dict = dict([(f.name, f) for f in self.sample_fields])
         for test_data in tests_data:
             name = str(test_data['name'])
             test = test_cache.get(name)
             if test is None:
                 test = self.Test(test_data['name'])
                 test_cache[name] = test
-                self.add(test)
+                session.add(test)
 
             samples = []
             for key, values in test_data.items():
                 if key == 'name' or key == "id" or key.endswith("_id"):
                     continue
                 field = field_dict.get(key)
                 if field is None and key != 'profile':
                     raise ValueError("test %r: Metric %r unknown in suite " %
                                      (name, key))
 
                 if not isinstance(values, list):
                     values = [values]
                 while len(samples) < len(values):
                     sample = self.Sample(run, test)
-                    self.add(sample)
+                    session.add(sample)
                     samples.append(sample)
                 for sample, value in zip(samples, values):
                     if key == 'profile':
                         profile = self.Profile(value, config, name)
                         sample.profile = profiles.get(hash(value), profile)
                     else:
                         sample.set_field(field, value)
 
-    def importDataFromDict(self, data, config, updateMachine, mergeRun):
+    def importDataFromDict(self, session, data, config, updateMachine,
+                           mergeRun):
         """
         importDataFromDict(data, config, updateMachine, mergeRun)
             -> Run  (or throws ValueError exception)
 
         Import a new run from the provided test interchange data, and return
         the constructed Run record. May throw ValueError exceptions in cases
         like mismatching machine data or duplicate run submission with
         mergeRun == 'reject'.
         """
-        machine = self._getOrCreateMachine(data['machine'], updateMachine)
-        run = self._getOrCreateRun(data['run'], machine, mergeRun)
-        self._importSampleValues(data['tests'], run, config)
+        machine = self._getOrCreateMachine(session, data['machine'],
+                                           updateMachine)
+        run = self._getOrCreateRun(session, data['run'], machine, mergeRun)
+        self._importSampleValues(session, data['tests'], run, config)
         return run
 
     # Simple query support (mostly used by templates)
 
-    def machines(self, name=None):
-        q = self.query(self.Machine)
+    def machines(self, session, name=None):
+        q = session.query(self.Machine)
         if name:
             q = q.filter_by(name=name)
         return q
 
-    def getMachine(self, id):
-        return self.query(self.Machine).filter_by(id=id).one()
+    def getMachine(self, session, id):
+        return session.query(self.Machine).filter_by(id=id).one()
 
-    def getRun(self, id):
-        return self.query(self.Run).filter_by(id=id).one()
+    def getRun(self, session, id):
+        return session.query(self.Run).filter_by(id=id).one()
 
-    def get_adjacent_runs_on_machine(self, run, N, direction=-1):
+    def get_adjacent_runs_on_machine(self, session, run, N, direction=-1):
         """
         get_adjacent_runs_on_machine(run, N, direction=-1) -> [Run*]
 
         Return the N runs which have been submitted to the same machine and are
         adjacent to the given run.
 
         The actual number of runs returned may be greater than N in situations
         where multiple reports were received for the same order.
def get_adjacent_runs_on_machine(self, session, run, N, direction=-1):
         # better in the worst cast, and I prefer that response times be
         # uniform. In practice, this appears to perform fine even for quite
         # large (~1GB, ~20k runs) databases.
 
         # Find all the orders on this machine, then sort them.
         #
         # FIXME: Scalability! However, pretty fast in practice, see elaborate
         # explanation above.
-        all_machine_orders = self.query(self.Order).\
+        all_machine_orders = session.query(self.Order).\
             join(self.Run).\
             filter(self.Run.machine == run.machine).distinct().all()
         all_machine_orders.sort()
 
         # Find the index of the current run.
         index = all_machine_orders.index(run.order)
 
         # Gather the next N orders.
         if direction == -1:
             orders_to_return = all_machine_orders[max(0, index - N):index]
         else:
             orders_to_return = all_machine_orders[index+1:index+N]
 
         # Get all the runs for those orders on this machine in a single query.
         ids_to_fetch = [o.id
                         for o in orders_to_return]
         if not ids_to_fetch:
             return []
 
-        runs = self.query(self.Run).\
+        runs = session.query(self.Run).\
             filter(self.Run.machine == run.machine).\
             filter(self.Run.order_id.in_(ids_to_fetch)).all()
 
         # Sort the result by order, accounting for direction to satisfy our
         # requirement of returning the runs in adjacency order.
         #
         # Even though we already know the right order, this is faster than
         # issueing separate queries.
         runs.sort(key=lambda r: r.order, reverse=(direction == -1))
 
         return runs
 
-    def get_previous_runs_on_machine(self, run, N):
-        return self.get_adjacent_runs_on_machine(run, N, direction=-1)
+    def get_previous_runs_on_machine(self, session, run, N):
+        return self.get_adjacent_runs_on_machine(session, run, N, direction=-1)
 
-    def get_next_runs_on_machine(self, run, N):
-        return self.get_adjacent_runs_on_machine(run, N, direction=1)
+    def get_next_runs_on_machine(self, session, run, N):
+        return self.get_adjacent_runs_on_machine(session, run, N, direction=1)
 
     def __repr__(self):
-        return "{} (on {})".format(self.name, self.v4db.path)
+        return "TestSuiteDB('%s')" % self.name
 
-    def getNumMachines(self):
-        return self.query(self.Machine).count()
+    def getNumMachines(self, session):
+        return session.query(self.Machine).count()
 
-    def getNumRuns(self):
-        return self.query(self.Run).count()
+    def getNumRuns(self, session):
+        return session.query(self.Run).count()
 
-    def getNumSamples(self):
-        return self.query(self.Sample).count()
+    def getNumSamples(self, session):
+        return session.query(self.Sample).count()
 
-    def getNumTests(self):
-        return self.query(self.Test).count()
+    def getNumTests(self, session):
+        return session.query(self.Test).count()
 
     def get_field_index(self, sample_field):
         return self.sample_field_indexes[sample_field.name]

lnt/trunk/lnt/server/db/v4db.py

 
 import lnt.testing
 
 import lnt.server.db.testsuitedb
 import lnt.server.db.migrate
 
 from lnt.util import logger
 from lnt.server.db import testsuite
+from sqlalchemy.orm import joinedload, subqueryload
 import lnt.server.db.util
 
 
 class V4DB(object):
     """
     Wrapper object for LNT v0.4+ databases.
     """
-
-    _db_updated = set()
     _engine_lock = threading.Lock()
-    _engine = {}
-
+    _engines = []
     def _load_schema_file(self, session, schema_file):
         with open(schema_file) as schema_fd:
             data = yaml.load(schema_fd)
         suite = testsuite.TestSuite.from_json(data)
         testsuite.check_testsuite_schema_changes(session, suite)
         suite = testsuite.sync_testsuite_with_metatables(session, suite)
         session.commit()
 
def _load_schemas(self, session):
         schemasDir = self.config.schemasDir
         for schema_file in glob.glob('%s/*.yaml' % schemasDir):
             try:
                 self._load_schema_file(session, schema_file)
             except Exception as e:
                 fatal("Could not load schema '%s': %s\n" % (schema_file, e))
 
         # Load schemas from database (deprecated)
-        ts_list = session.query(testsuite.TestSuite).all()
+        ts_list = session.query(testsuite.TestSuite) \
+            .options(subqueryload(testsuite.TestSuite.sample_fields)
+                     .joinedload(testsuite.SampleField.status_field)) \
+            .options(joinedload(testsuite.TestSuite.order_fields)) \
+            .options(joinedload(testsuite.TestSuite.run_fields)) \
+            .options(joinedload(testsuite.TestSuite.machine_fields)) \
+            .all()
         for suite in ts_list:
             name = suite.name
             if name in self.testsuite:
                 continue
             ts = lnt.server.db.testsuitedb.TestSuiteDB(self, name, suite,
                                                        create_tables=False)
             self.testsuite[name] = ts
 
     def __init__(self, path, config, baseline_revision=0):
         # If the path includes no database type, assume sqlite.
         if lnt.server.db.util.path_has_no_database_type(path):
             path = 'sqlite:///' + path
 
         self.path = path
         self.config = config
         self.baseline_revision = baseline_revision
-        with V4DB._engine_lock:
-            if path not in V4DB._engine:
-                connect_args = {}
-                if path.startswith("sqlite://"):
-                    # Some of the background tasks keep database transactions
-                    # open for a long time. Make it less likely to hit
-                    # "(OperationalError) database is locked" because of that.
-                    connect_args['timeout'] = 30
-                engine = sqlalchemy.create_engine(path,
-                                                  connect_args=connect_args)
-                V4DB._engine[path] = engine
-        self.engine = V4DB._engine[path]
+        self.engine = sqlalchemy.create_engine(path,
+        with V4DB._engine_lock:
+            V4DB._engines.append(self.engine)
 
         # Update the database to the current version, if necessary. Only check
         # this once per path.
-        if path not in V4DB._db_updated:
-            lnt.server.db.migrate.update(self.engine)
-            V4DB._db_updated.add(path)
-
-        self.session = sqlalchemy.orm.sessionmaker(self.engine)()
 
-        # Add several shortcut aliases.
-        self.add = self.session.add
-        self.delete = self.session.delete
-        self.commit = self.session.commit
-        self.query = self.session.query
-        self.rollback = self.session.rollback
-
-        # For parity with the usage of TestSuiteDB, we make our primary model
-        # classes available as instance variables.
-        self.SampleType = testsuite.SampleType
-        self.StatusKind = testsuite.StatusKind
-        self.TestSuite = testsuite.TestSuite
-        self.SampleField = testsuite.SampleField
+        self.sessionmaker = sqlalchemy.orm.sessionmaker(self.engine)
+        session = self.make_session()
 
         self.testsuite = dict()
-        self._load_schemas(self.session)
+        self._load_schemas(session)
+        session.expunge_all()
+        session.close()
 
     def close(self):
-        if self.session is not None:
-            self.session.close()
+        self.engine.dispose()
 
     @staticmethod
     def close_all_engines():
+        """Hack for async_ops. Do not use for anything else."""
         with V4DB._engine_lock:
-            for key, engine in V4DB._engine.items():
+            for engine in _engines:
                 engine.dispose()
-            V4DB._engine = {}
-            V4DB._db_updated = set()
+            V4DB._engines = []
+
+    def make_session(self):
+        return self.sessionmaker()
 
     def settings(self):
         """All the setting needed to recreate this instnace elsewhere."""
         return {
             'path': self.path,
             'config': self.config,
             'baseline_revision': self.baseline_revision,
         }

lnt/trunk/lnt/server/instance.py

def frompath(path):
         config = lnt.server.config.Config.from_data(config_path, config_data)
 
         return Instance(config_path, config, tmpdir)
 
     def __init__(self, config_path, config, tmpdir=None):
         self.config_path = config_path
         self.config = config
         self.tmpdir = tmpdir
+        self.databases = dict()
+        for name in self.config.get_database_names():
+            self.databases[name] = self.config.get_database(name)
 
     def __del__(self):
         # If we have a temporary dir, clean it up now.
         if self.tmpdir is not None:
             shutil.rmtree(self.tmpdir)
 
-    def get_database(self, *args, **kwargs):
-        return self.config.get_database(*args, **kwargs)
+    def get_database(self, name):
+        return self.databases.get(name, None)

lnt/trunk/lnt/server/reporting/analysis.py

def get_value_status(self, confidence_interval=2.576,
                 return REGRESSED if self.bigger_is_better else IMPROVED
             else:
                 return IMPROVED if self.bigger_is_better else REGRESSED
         else:
             return UNCHANGED_PASS
 
 
 class RunInfo(object):
-    def __init__(self, testsuite, runs_to_load,
+    def __init__(self, session, testsuite, runs_to_load,
                  aggregation_fn=stats.safe_min, confidence_lv=.05,
                  only_tests=None):
         """Get all the samples needed to build a CR.
         runs_to_load are the run IDs of the runs to get the samples from.
         if only_tests is passed, only samples form those test IDs are fetched.
         """
         self.testsuite = testsuite
         self.aggregation_fn = aggregation_fn
         self.confidence_lv = confidence_lv
 
         self.sample_map = multidict.multidict()
         self.profile_map = dict()
         self.loaded_run_ids = set()
 
-        self._load_samples_for_runs(runs_to_load, only_tests)
+        self._load_samples_for_runs(session, runs_to_load, only_tests)
 
     @property
     def test_ids(self):
         return set(key[1] for key in self.sample_map.keys())
 
-    def get_sliding_runs(self, run, compare_run, num_comparison_runs=0):
+    def get_sliding_runs(self, session, run, compare_run,
+                         num_comparison_runs=0):
         """
         Get num_comparison_runs most recent runs,
         This query is expensive.
         """
         runs = [run]
         runs_prev = self.testsuite \
-            .get_previous_runs_on_machine(run, num_comparison_runs)
+            .get_previous_runs_on_machine(session, run, num_comparison_runs)
         runs += runs_prev
 
         if compare_run is not None:
             compare_runs = [compare_run]
             comp_prev = self.testsuite \
-                .get_previous_runs_on_machine(compare_run, num_comparison_runs)
+                .get_previous_runs_on_machine(session, compare_run,
+                                              num_comparison_runs)
             compare_runs += comp_prev
         else:
             compare_runs = []
 
         return runs, compare_runs
 
     def get_run_comparison_result(self, run, compare_to, test_id, field,
                                   hash_of_binary_field):
def get_geomean_comparison_result(self, run, compare_to, field, tests):
                                 prev_failed=not bool(prev_values),
                                 samples=run_values,
                                 prev_samples=prev_values,
                                 cur_hash=cur_hash,
                                 prev_hash=prev_hash,
                                 confidence_lv=0,
                                 bigger_is_better=field.bigger_is_better)
 
-    def _load_samples_for_runs(self, run_ids, only_tests):
+    def _load_samples_for_runs(self, session, run_ids, only_tests):
         # Find the set of new runs to load.
         to_load = set(run_ids) - self.loaded_run_ids
         if not to_load:
             return
 
         # Batch load all of the samples for the needed runs.
         #
         # We speed things up considerably by loading the column data directly
         # here instead of requiring SA to materialize Sample objects.
         columns = [self.testsuite.Sample.run_id,
                    self.testsuite.Sample.test_id,
                    self.testsuite.Sample.profile_id]
         columns.extend(f.column for f in self.testsuite.sample_fields)
-        q = self.testsuite.query(*columns)
+        q = session.query(*columns)
         if only_tests:
             q = q.filter(self.testsuite.Sample.test_id.in_(only_tests))
         q = q.filter(self.testsuite.Sample.run_id.in_(to_load))
         for data in q:
             run_id = data[0]
             test_id = data[1]
             profile_id = data[2]
             sample_values = data[3:]
             self.sample_map[(run_id, test_id)] = sample_values
             if profile_id is not None:
                 self.profile_map[(run_id, test_id)] = profile_id
 
         self.loaded_run_ids |= to_load

lnt/trunk/lnt/server/reporting/dailyreport.py

def get_key_run(self, machine, day_index):
 
         runs = self.machine_runs.get((machine.id, day_index))
         if runs is None:
             return None
 
         # Select a key run arbitrarily.
         return runs[0]
 
-    def build(self):
+    def build(self, session):
         ts = self.ts
 
         # Construct datetime instances for the report range.
         day_ordinal = datetime.datetime(self.year, self.month,
                                         self.day).toordinal()
 
         # Adjust the dates time component.  As we typically want to do runs
         # overnight, we define "daily" to really mean "daily plus some
         # offset". The offset should generally be whenever the last run
         # finishes on today's date.
 
         self.next_day = (datetime.datetime.fromordinal(day_ordinal + 1) +
                          self.day_start_offset)
         self.prior_days = [(datetime.datetime.fromordinal(day_ordinal - i) +
                             self.day_start_offset)
                            for i in range(self.num_prior_days_to_include + 1)]
 
         # Find all the runs that occurred for each day slice.
-        prior_runs = [ts.query(ts.Run).
+        prior_runs = [session.query(ts.Run).
                       filter(ts.Run.start_time > prior_day).
                       filter(ts.Run.start_time <= day).all()
                       for day, prior_day in _pairs(self.prior_days)]
 
         if self.filter_machine_re is not None:
             prior_runs = [[run for run in runs
                            if self.filter_machine_re.search(run.machine.name)]
                           for runs in prior_runs]
def build(self, session):
 
         # If there are no relevant runs, just stop processing (the report will
         # generate an error).
         if not relevant_run_ids:
             self.error = "no runs to display in selected date range"
             return
 
         # Get the set all tests reported in the recent runs.
-        self.reporting_tests = ts.query(ts.Test).filter(
+        self.reporting_tests = session.query(ts.Test).filter(
             sqlalchemy.sql.exists('*', sqlalchemy.sql.and_(
                     ts.Sample.run_id.in_(relevant_run_ids),
                     ts.Sample.test_id == ts.Test.id))).all()
         self.reporting_tests.sort(key=lambda t: t.name)
 
         run_ids_to_load = list(relevant_run_ids) + \
             [r.id for r in less_relevant_runs]
 
         # Create a run info object.
-        sri = lnt.server.reporting.analysis.RunInfo(ts, run_ids_to_load)
+        sri = lnt.server.reporting.analysis.RunInfo(session, ts,
+                                                    run_ids_to_load)
 
         # Build the result table of tests with interesting results.
         def compute_visible_results_priority(visible_results):
             # We just use an ad hoc priority that favors showing tests with
             # failures and large changes. We do this by computing the priority
             # as tuple of whether or not there are any failures, and then sum
             # of the mean percentage changes.
             test, results = visible_results

lnt/trunk/lnt/server/reporting/runs.py

 """
 Report functionality centered around individual runs.
 """
 
 import time
 import lnt.server.reporting.analysis
 import lnt.server.ui.app
 import lnt.util.stats
 
 
-def generate_run_data(run, baseurl, num_comparison_runs=0, result=None,
-                      compare_to=None, baseline=None,
+def generate_run_data(session, run, baseurl, num_comparison_runs=0,
+                      result=None, compare_to=None, baseline=None,
                       aggregation_fn=lnt.util.stats.safe_min,
                       confidence_lv=.05, styles=dict(), classes=dict()):
     """
     Generate raw data for a report on the results of the given individual
     run. They are meant as inputs to jinja templates which could create
     email reports or presentations on a web page.
     """
     assert num_comparison_runs >= 0
 
     start_time = time.time()
 
     ts = run.testsuite
     machine = run.machine
     machine_parameters = machine.parameters
 
     if baseline is None:
         # If a baseline has not been given, look up the run closest to
         # the default baseline revision for which this machine also
         # reported.
-        baseline = machine.get_baseline_run()
+        baseline = machine.get_baseline_run(session)
 
     # If the baseline is the same as the comparison run, ignore it.
     visible_note = None
     if baseline is compare_to:
         visible_note = "Baseline and compare_to are the same: " \
                        "disabling baseline."
         baseline = None
 
     # Gather the runs to use for statistical data.
     comparison_start_run = compare_to or run
     comparison_window = list(ts.get_previous_runs_on_machine(
-            comparison_start_run, num_comparison_runs))
+            session, comparison_start_run, num_comparison_runs))
     if baseline:
         baseline_window = list(ts.get_previous_runs_on_machine(
-                baseline, num_comparison_runs))
+                session, baseline, num_comparison_runs))
     else:
         baseline_window = []
 
     # If we don't have an explicit baseline run or a comparison run, use the
     # previous run.
     if compare_to is None and comparison_window:
         compare_to = comparison_window[0]
 
     # Create the run info analysis object.
     runs_to_load = set(r.id for r in comparison_window)
     for r in baseline_window:
         runs_to_load.add(r.id)
     runs_to_load.add(run.id)
     if compare_to:
         runs_to_load.add(compare_to.id)
     if baseline:
         runs_to_load.add(baseline.id)
     sri = lnt.server.reporting.analysis.RunInfo(
-        ts, runs_to_load, aggregation_fn, confidence_lv)
+        session, ts, runs_to_load, aggregation_fn, confidence_lv)
 
     # Get the test names, metric fields and total test counts.
-    test_names = ts.query(ts.Test.name, ts.Test.id).\
+    test_names = session.query(ts.Test.name, ts.Test.id).\
         order_by(ts.Test.name).\
         filter(ts.Test.id.in_(sri.test_ids)).all()
     metric_fields = list(ts.Sample.get_metric_fields())
     num_total_tests = len(metric_fields) * len(test_names)
 
     # Gather the run-over-run changes to report, organized by field and then
     # collated by change type.
     run_to_run_info, test_results = _get_changes_by_type(

lnt/trunk/lnt/server/reporting/summaryreport.py

def __init__(self, db, report_orders, report_machine_names,
 
         self.data_table = None
         self.requested_machine_ids = None
         self.requested_machines = None
         self.runs_at_index = None
 
         self.warnings = []
 
-    def build(self):
+    def build(self, session):
         # Build a per-testsuite list of the machines that match the specified
         # patterns.
         def should_be_in_report(machine):
             if machine.name in self.report_machine_names:
                 return True
             for rex in self.report_machine_rexes:
                 if rex.match(machine.name):
                     return True
-        self.requested_machines = dict((ts, filter(should_be_in_report,
-                                                   ts.query(ts.Machine).all()))
-                                       for ts in self.testsuites)
+        self.requested_machines = dict(
+            (ts, filter(should_be_in_report, session.query(ts.Machine).all()))
         self.requested_machine_ids = dict(
             (ts, [m.id for m in machines])
             for ts, machines in self.requested_machines.items()
         )
 
         # First, collect all the runs to summarize on, for each index in the
         # report orders.
         self.runs_at_index = []
         for _, orders in self.report_orders:
             # For each test suite...
             runs = []
             for ts in self.testsuites:
                 # Find all the orders that match.
-                result = ts.query(ts.Order.id).\
+                result = session.query(ts.Order.id).\
                     filter(ts.Order.llvm_project_revision.in_(
                         orders)).all()
                 ts_order_ids = [id for id, in result]
 
                 # Find all the runs that matchs those orders.
                 if not ts_order_ids:
                     ts_runs = []
                 else:
-                    ts_runs = ts.query(ts.Run).\
+                    ts_runs = session.query(ts.Run).\
                         filter(ts.Run.order_id.in_(ts_order_ids)).\
                         filter(ts.Run.machine_id.in_(
                             self.requested_machine_ids[ts])).all()
 
                 if not ts_runs:
                     self.warnings.append(
                         'no runs for test suite %r in orders %r' % (
                             ts.name, orders))
 
                 runs.append((ts_runs, ts_order_ids))
             self.runs_at_index.append(runs)
 
         # Load the tests for each testsuite.
         self.tests = dict((ts, dict((test.id, test)
-                                    for test in ts.query(ts.Test)))
+                                    for test in session.query(ts.Test)))
                           for ts in self.testsuites)
 
         # Compute the base table for aggregation.
         #
         # The table is indexed by a test name and test features, which are
         # either extracted from the test name or from the test run (depending
         # on the suite).
         #
def _build_data_table(self):
 
                 # Preload the run parameters.
                 run_parameters_map = dict((r.id, r.parameters)
                                           for r in ts_runs)
 
                 # Load all the samples for all runs we are interested in.
                 columns = [ts.Sample.run_id, ts.Sample.test_id]
                 columns.extend(f.column for f in ts.sample_fields)
-                samples = ts.query(*columns).filter(
+                samples = session.query(*columns).filter(
                     ts.Sample.run_id.in_(run_id_map.keys()))
                 for sample in samples:
                     run = run_id_map[sample[0]]
                     datapoints = list()
                     for key, value in get_datapoints_for_sample(ts, sample):
                         items = self.data_table.get(key)
                         if items is None:
                             items = [[]

lnt/trunk/lnt/server/ui/api.py

def in_db(func):
     def wrap(*args, **kwargs):
         db = kwargs.pop('db')
         ts = kwargs.pop('ts')
         g.db_name = db
         g.testsuite_name = ts
         g.db_info = current_app.old_config.databases.get(g.db_name)
         if g.db_info is None:
             abort(404, message="Invalid database.")
+        request.db = current_app.instance.get_database(g.db_name)
+        request.session = request.db.make_session()
         # Compute result.
         result = func(*args, **kwargs)
 
         # Make sure that any transactions begun by this request are finished.
-        request.get_db().rollback()
+        request.session.rollback()
         return result
 
     return wrap
 
 
 def requires_auth_token(f):
     @wraps(f)
     def decorated(*args, **kwargs):

 
 class Machines(Resource):
     """List all the machines and give summary information."""
     method_decorators = [in_db]
 
     @staticmethod
     def get():
         ts = request.get_testsuite()
-        machines = ts.query(ts.Machine).all()
+        session = request.session
+        machines = session.query(ts.Machine).all()
 
         result = common_fields_factory()
         result['machines'] = machines
         return result
 
 
 class Machine(Resource):
     """Detailed results about a particular machine, including runs on it."""
     method_decorators = [in_db]
 
     @staticmethod
     def _get_machine(machine_spec):
         ts = request.get_testsuite()
+        session = request.session
         # Assume id number if machine_spec is numeric, otherwise a name.
         if machine_spec.isdigit():
-            machine = ts.query(ts.Machine) \
+            machine = session.query(ts.Machine) \
                 .filter(ts.Machine.id == machine_spec).first()
         else:
-            machines = ts.query(ts.Machine) \
+            machines = session.query(ts.Machine) \
                 .filter(ts.Machine.name == machine_spec).all()
             if len(machines) == 0:
                 machine = None
             elif len(machines) > 1:
                 abort(404, msg="Name '%s' is ambiguous; specify machine id" %
                       (machine_spec))
         if machine is None:
             abort(404, msg="Did not find machine '%s'" % (machine_spec,))
         return machine
 
     @staticmethod
     def get(machine_spec):
         ts = request.get_testsuite()
+        session = request.session
         machine = Machine._get_machine(machine_spec)
-        machine_runs = ts.query(ts.Run) \
+        machine_runs = session.query(ts.Run) \
             .filter(ts.Run.machine_id == machine.id) \
             .options(joinedload(ts.Run.order)) \
             .all()
 
         runs = [run.__json__(flatten_order=True) for run in machine_runs]
 
         result = common_fields_factory()
         result['machine'] = machine
         result['runs'] = runs
         return result
 
     @staticmethod
     @requires_auth_token
     def delete(machine_spec):
         ts = request.get_testsuite()
+        session = request.session
         machine = Machine._get_machine(machine_spec)
 
-        # Just saying ts.session.delete(machine) takes a long time and risks
+        # Just saying session.delete(machine) takes a long time and risks
         # running into OOM or timeout situations for machines with a hundreds
         # of runs. So instead remove machine runs in chunks.
         def perform_delete(ts, machine):
-            count = ts.query(ts.Run) \
+            count = session.query(ts.Run) \
                 .filter(ts.Run.machine_id == machine.id).count()
             at = 0
             while True:
-                runs = ts.query(ts.Run) \
+                runs = session.query(ts.Run) \
                     .filter(ts.Run.machine_id == machine.id) \
                     .options(joinedload(ts.Run.samples)) \
                     .options(joinedload(ts.Run.fieldchanges)) \
                     .order_by(ts.Run.id).limit(10).all()
                 if len(runs) == 0:
                     break
                 at += len(runs)
                 msg = "Deleting runs %s (%d/%d)" % \
                     (" ".join([str(run.id) for run in runs]), at, count)
                 logger.info(msg)
                 yield msg + '\n'
                 for run in runs:
-                    ts.session.delete(run)
-                ts.commit()
+                    session.delete(run)
+                session.commit()
 
             machine_name = "%s:%s" % (machine.name, machine.id)
-            ts.session.delete(machine)
-            ts.commit()
+            session.delete(machine)
+            session.commit()
             msg = "Deleted machine %s" % machine_name
             logger.info(msg)
             yield msg + '\n'
 
         stream = stream_with_context(perform_delete(ts, machine))
         return Response(stream, mimetype="text/plain")
 
     @staticmethod
     @requires_auth_token
     def put(machine_spec):
         machine = Machine._get_machine(machine_spec)
         machine_name = "%s:%s" % (machine.name, machine.id)
         data = json.loads(request.data)
         machine_data = data['machine']
         machine.set_from_dict(machine_data)
+        session = request.session
         ts = request.get_testsuite()
-        ts.commit()
+        session.commit()
 
     @staticmethod
     @requires_auth_token
     def post(machine_spec):
+        session = request.session
         ts = request.get_testsuite()
         machine = Machine._get_machine(machine_spec)
         machine_name = "%s:%s" % (machine.name, machine.id)
 
         action = request.values.get('action', None)
         if action is None:
             abort(400, msg="No 'action' specified")
         elif action == 'rename':
             name = request.values.get('name', None)
             if name is None:
                 abort(400, msg="Expected 'name' for rename request")
-            existing = ts.query(ts.Machine).filter(ts.Machine.name == name) \
+            existing = session.query(ts.Machine) \
+                .filter(ts.Machine.name == name) \
                 .first()
             if existing is not None:
                 abort(400, msg="Machine with name '%s' already exists" % name)
             machine.name = name
-            ts.session.commit()
+            session.commit()
             logger.info("Renamed machine %s to %s" % (machine_name, name))
         elif action == 'merge':
             into_id = request.values.get('into', None)
             if into_id is None:
                 abort(400, msg="Expected 'into' for merge request")
             into = Machine._get_machine(into_id)
             into_name = "%s:%s" % (into.name, into.id)
-            ts.query(ts.Run) \
+            session.query(ts.Run) \
                 .filter(ts.Run.machine_id == machine.id) \
                 .update({ts.Run.machine_id: into.id},
                         synchronize_session=False)
-            ts.session.expire_all()  # be safe after synchronize_session==False
+            session.expire_all()  # be safe after synchronize_session==False
             # re-query Machine so we can delete it.
             machine = Machine._get_machine(machine_spec)
-            ts.delete(machine)
-            ts.session.commit()
+            session.delete(machine)
+            session.commit()
             logger.info("Merged machine %s into %s" %
                         (machine_name, into_name))
             logger.info("Deleted machine %s" % machine_name)
         else:
             abort(400, msg="Unknown action '%s'" % action)
 
 
 class Run(Resource):
     method_decorators = [in_db]
 
     @staticmethod
     def get(run_id):
+        session = request.session
         ts = request.get_testsuite()
         try:
-            run = ts.query(ts.Run) \
+            run = session.query(ts.Run) \
                 .filter(ts.Run.id == run_id) \
                 .options(joinedload(ts.Run.machine)) \
                 .options(joinedload(ts.Run.order)) \
                 .one()
         except sqlalchemy.orm.exc.NoResultFound:
             abort(404, msg="Did not find run " + str(run_id))
 
         to_get = [ts.Sample.id, ts.Sample.run_id, ts.Test.name]
         for f in ts.sample_fields:
             to_get.append(f.column)
 
-        sample_query = ts.query(*to_get) \
+        sample_query = session.query(*to_get) \
             .join(ts.Test) \
             .filter(ts.Sample.run_id == run_id) \
             .all()
         # TODO: Handle multiple samples for a single test?
 
         # noinspection PyProtectedMember
         samples = [row._asdict() for row in sample_query]
 
         result = common_fields_factory()
         result['run'] = run
         result['machine'] = run.machine
         result['tests'] = samples
         return result
 
     @staticmethod
     @requires_auth_token
     def delete(run_id):
+        session = request.session
         ts = request.get_testsuite()
-        run = ts.query(ts.Run).filter(ts.Run.id == run_id).first()
+        run = session.query(ts.Run).filter(ts.Run.id == run_id).first()
         if run is None:
             abort(404, msg="Did not find run " + str(run_id))
-        ts.delete(run)
-        ts.commit()
+        session.delete(run)
+        session.commit()
         logger.info("Deleted run %s" % (run_id,))
 
 
 class Runs(Resource):
     """Detailed results about a particular machine, including runs on it."""
     method_decorators = [in_db]
 
     @staticmethod
     @requires_auth_token
     def post():
         """Add a new run into the lnt database"""
+        session = request.session
         db = request.get_db()
         data = request.data
         updateMachine = request.values.get('update_machine', False)
         merge = request.values.get('merge', 'replace')
         result = lnt.util.ImportData.import_from_string(
-            current_app.old_config, g.db_name, db, g.testsuite_name, data,
-            updateMachine=updateMachine, mergeRun=merge)
+            current_app.old_config, g.db_name, db, session, g.testsuite_name,
+            data, updateMachine=updateMachine, mergeRun=merge)
 
         error = result['error']
         if error is not None:
             response = jsonify(result)
             response.status = '400'
             logger.warning("%s: Submission rejected: %s" %
                            (request.url, error))
             return response
 
         new_url = ('%sapi/db_%s/v4/%s/runs/%s' %
                    (request.url_root, g.db_name, g.testsuite_name,
                     result['run_id']))
         result['result_url'] = new_url
         response = jsonify(result)
         response.status = '301'
         response.headers.add('Location', new_url)
         return response
 
 
 class Order(Resource):
     method_decorators = [in_db]
 
     @staticmethod
     def get(order_id):
+        session = request.session
         ts = request.get_testsuite()
         try:
-            order = ts.query(ts.Order).filter(ts.Order.id == order_id).one()
+            order = session.query(ts.Order) \
+                .filter(ts.Order.id == order_id).one()
         except NoResultFound:
             abort(404, message="Invalid order.")
         result = common_fields_factory()
         result['orders'] = [order]
         return result
 
 
 class SampleData(Resource):
     method_decorators = [in_db]
 
     @staticmethod
     def get(sample_id):
+        session = request.session
         ts = request.get_testsuite()
         try:
-            sample = ts.query(ts.Sample) \
+            sample = session.query(ts.Sample) \
                 .filter(ts.Sample.id == sample_id) \
                 .one()
         except NoResultFound:
             abort(404, message="Invalid order.")
         result = common_fields_factory()
         result['samples'] = [sample]
         return result
 
 
 class SamplesData(Resource):
     """List all the machines and give summary information."""
     method_decorators = [in_db]
 
     @staticmethod
     def get():
         """Get the data for a particular line in a graph."""
+        session = request.session
         ts = request.get_testsuite()
         args = request.args.to_dict(flat=False)
         # Maybe we don't need to do this?
         run_ids = [int(r) for r in args.get('runid', [])]
 
         if not run_ids:
             abort(400, msg='No runids found in args. '
                            'Should be "samples?runid=1&runid=2" etc.')
 
         to_get = [ts.Sample.id,
                   ts.Sample.run_id,
                   ts.Test.name,
                   ts.Order.fields[0].column]
 
         for f in ts.sample_fields:
             to_get.append(f.column)
 
-        q = ts.query(*to_get) \
+        q = session.query(*to_get) \
             .join(ts.Test) \
             .join(ts.Run) \
             .join(ts.Order) \
             .filter(ts.Sample.run_id.in_(run_ids))
         result = common_fields_factory()
         # noinspection PyProtectedMember
         result['samples'] = [{k: v for k, v in sample.items() if v is not None}
                              for sample in [sample._asdict()
                                             for sample in q.all()]]
 
         return result
 
 
 class Graph(Resource):
     """List all the machines and give summary information."""
     method_decorators = [in_db]
 
     @staticmethod
     def get(machine_id, test_id, field_index):
         """Get the data for a particular line in a graph."""
+        session = request.session
         ts = request.get_testsuite()
         # Maybe we don't need to do this?
         try:
-            machine = ts.query(ts.Machine) \
+            machine = session.query(ts.Machine) \
                 .filter(ts.Machine.id == machine_id) \
                 .one()
-            test = ts.query(ts.Test) \
+            test = session.query(ts.Test) \
                 .filter(ts.Test.id == test_id) \
                 .one()
             field = ts.sample_fields[field_index]
         except NoResultFound:
             abort(404)
 
-        q = ts.query(field.column, ts.Order.llvm_project_revision,
+        q = session.query(field.column, ts.Order.llvm_project_revision,
                      ts.Run.start_time, ts.Run.id) \
             .join(ts.Run) \
             .join(ts.Order) \
             .filter(ts.Run.machine_id == machine.id) \
             .filter(ts.Sample.test == test) \
             .filter(field.column.isnot(None)) \
             .order_by(ts.Order.llvm_project_revision.desc())
 

 
 class Regression(Resource):
     """List all the machines and give summary information."""
     method_decorators = [in_db]
 
     @staticmethod
     def get(machine_id, test_id, field_index):
         """Get the regressions for a particular line in a graph."""
+        session = request.session
         ts = request.get_testsuite()
         field = ts.sample_fields[field_index]
         # Maybe we don't need to do this?
-        fcs = ts.query(ts.FieldChange) \
+        fcs = session.query(ts.FieldChange) \
             .filter(ts.FieldChange.machine_id == machine_id) \
             .filter(ts.FieldChange.test_id == test_id) \
             .filter(ts.FieldChange.field_id == field.id) \
             .all()
         fc_ids = [x.id for x in fcs]
         fc_mappings = dict(
             [(x.id, (x.end_order.as_ordered_string(), x.new_value))
              for x in fcs])
         if len(fcs) == 0:
             # If we don't find anything, lets see if we are even looking
             # for a valid thing to provide a nice error.
             try:
-                ts.query(ts.Machine) \
+                session.query(ts.Machine) \
                     .filter(ts.Machine.id == machine_id) \
                     .one()
-                ts.query(ts.Test) \
+                session.query(ts.Test) \
                     .filter(ts.Test.id == test_id) \
                     .one()
                 _ = ts.sample_fields[field_index]
             except (NoResultFound, IndexError):
                 abort(404)
             # I think we found nothing.
             return []
-        regressions = ts.query(ts.Regression.title, ts.Regression.id,
+        regressions = session.query(ts.Regression.title, ts.Regression.id,
                                     ts.RegressionIndicator.field_change_id,
                                     ts.Regression.state) \
             .join(ts.RegressionIndicator) \
             .filter(ts.RegressionIndicator.field_change_id.in_(fc_ids)) \
             .all()
         results = [
             {
                 'title': r.title,
                 'id': r.id,
                 'state': r.state,

lnt/trunk/lnt/server/ui/app.py

def __init__(self, *args, **kwargs):
         self.db = None
         self.testsuite = None
 
     def elapsed_time(self):
         return time.time() - self.request_time
 
     # Utility Methods
     def get_db(self):
-        """
-        get_db() -> <db instance>
-
-        Get the active database and add a logging handler if part of the
-        request arguments.
-        """
-
-        if self.db is None:
-            try:
-                self.db = current_app.old_config.get_database(g.db_name)
-            except DatabaseError:
-                self.db = current_app.old_config.get_database(g.db_name)
-            # Enable SQL logging with db_log.
-            #
-            # FIXME: Conditionalize on an is_production variable.
-            show_sql = bool(self.args.get('db_log') or self.form.get('db_log'))
-            if show_sql:
-                g.db_log = StringIO.StringIO()
-                logger = logging.getLogger("sqlalchemy")
-                logger.addHandler(logging.StreamHandler(g.db_log))
-
+        assert self.db is not None
         return self.db
 
     def get_testsuite(self):
         """
         get_testsuite() -> server.db.testsuite.TestSuite
 
         Get the active testsuite.
         """
 
         if self.testsuite is None:
-            testsuites = self.get_db().testsuite
+            testsuites = self.db.testsuite
             if g.testsuite_name not in testsuites:
                 flask.abort(404)
 
             self.testsuite = testsuites[g.testsuite_name]
 
         return self.testsuite
 
     def close(self):

lnt/trunk/lnt/server/ui/decorators.py

def db_route(rule, **options):
     """
     def decorator(f):
         def wrap(db_name=None, **args):
             # Initialize the database parameters on the app globals object.
             g.db_name = db_name or "default"
             g.db_info = current_app.old_config.databases.get(g.db_name)
             if g.db_info is None:
                 abort(404)
+            request.db = current_app.instance.get_database(g.db_name)
+            request.session = request.db.make_session()
 
             # Compute result.
             result = f(**args)
 
             # Make sure that any transactions begun by this request are
             # finished.
-            request.get_db().rollback()
+            request.session.rollback()
 
             # Return result.
             return result
 
         frontend.add_url_rule(rule, f.__name__, wrap, **options)
         frontend.add_url_rule("/db_<db_name>" + rule,
                               f.__name__, wrap, **options)
 
def decorator(f):
             # Initialize the test suite parameters on the app globals object.
             g.testsuite_name = testsuite_name
 
             # Initialize the database parameters on the app globals object.
             g.db_name = db_name or "default"
             g.db_info = current_app.old_config.databases.get(g.db_name)
             if g.db_info is None:
                 abort(404)
+            request.db = current_app.instance.get_database(g.db_name)
+            request.session = request.db.make_session()
 
             # Compute result.
             result = f(**args)
 
             # Make sure that any transactions begun by this request are
             # finished.
-            request.get_db().rollback()
+            request.session.rollback()
 
             # Return result.
             return result
 
         frontend.add_url_rule("/v4/<testsuite_name>" + rule,
                               f.__name__, wrap, **options)
         frontend.add_url_rule("/db_<db_name>/v4/<testsuite_name>" + rule,
                               f.__name__, wrap, **options)
 
         return wrap
     return decorator

lnt/trunk/lnt/server/ui/profile_views.py

def profile_admin():
     age = [[k * bucket_size, hist[k]] for k in sorted(hist.keys())]
 
     return render_template("profile_admin.html",
                            history=history, age=age, bucket_size=bucket_size)
 
 
 @v4_route("/profile/ajax/getFunctions")
 def v4_profile_ajax_getFunctions():
+    session = request.session
     ts = request.get_testsuite()
     runid = request.args.get('runid')
     testid = request.args.get('testid')
 
     profileDir = current_app.old_config.profileDir
 
     idx = 0
     tlc = {}
-    sample = ts.query(ts.Sample) \
+    sample = session.query(ts.Sample) \
                .filter(ts.Sample.run_id == runid) \
                .filter(ts.Sample.test_id == testid).first()
     if sample and sample.profile:
         p = sample.profile.load(profileDir)
         return json.dumps([[n, f] for n, f in p.getFunctions().items()])
     else:
         abort(404)
 
 
 @v4_route("/profile/ajax/getTopLevelCounters")
 def v4_profile_ajax_getTopLevelCounters():
+    session = request.session
     ts = request.get_testsuite()
     runids = request.args.get('runids').split(',')
     testid = request.args.get('testid')
 
     profileDir = current_app.old_config.profileDir
 
     idx = 0
     tlc = {}
     for rid in runids:
-        sample = ts.query(ts.Sample) \
+        sample = session.query(ts.Sample) \
                    .filter(ts.Sample.run_id == rid) \
                    .filter(ts.Sample.test_id == testid).first()
         if sample and sample.profile:
             p = sample.profile.load(profileDir)
             for k, v in p.getTopLevelCounters().items():
                 tlc.setdefault(k, [None]*len(runids))[idx] = v
         idx += 1
 
     # If the 1'th counter is None for all keys, truncate the list.
     if all(len(k) > 1 and k[1] is None for k in tlc.values()):
         tlc = {k: [v[0]] for k, v in tlc.items()}
 
     return json.dumps(tlc)
 
 
 @v4_route("/profile/ajax/getCodeForFunction")
 def v4_profile_ajax_getCodeForFunction():
+    session = request.session
     ts = request.get_testsuite()
     runid = request.args.get('runid')
     testid = request.args.get('testid')
     f = request.args.get('f')
 
     profileDir = current_app.old_config.profileDir
 
-    sample = ts.query(ts.Sample) \
+    sample = session.query(ts.Sample) \
                .filter(ts.Sample.run_id == runid) \
                .filter(ts.Sample.test_id == testid).first()
     if not sample or not sample.profile:
         abort(404)
 
     p = sample.profile.load(profileDir)
     return json.dumps([x for x in p.getCodeForFunction(f)])
 
 
 @v4_route("/profile/<int:testid>/<int:run1_id>")
 def v4_profile_fwd(testid, run1_id):
     return v4_profile(testid, run1_id)
 
 
 @v4_route("/profile/<int:testid>/<int:run1_id>/<int:run2_id>")
 def v4_profile_fwd2(testid, run1_id, run2_id=None):
     return v4_profile(testid, run1_id, run2_id)
 
 
 def v4_profile(testid, run1_id, run2_id=None):
+    session = request.session
     ts = request.get_testsuite()
     profileDir = current_app.old_config.profileDir
 
     try:
-        test = ts.query(ts.Test).filter(ts.Test.id == testid).one()
-        run1 = ts.query(ts.Run).filter(ts.Run.id == run1_id).one()
-        sample1 = ts.query(ts.Sample) \
+        test = session.query(ts.Test).filter(ts.Test.id == testid).one()
+        run1 = session.query(ts.Run).filter(ts.Run.id == run1_id).one()
+        sample1 = session.query(ts.Sample) \
                     .filter(ts.Sample.run_id == run1_id) \
                     .filter(ts.Sample.test_id == testid).first()
         if run2_id is not None:
-            run2 = ts.query(ts.Run).filter(ts.Run.id == run2_id).one()
-            sample2 = ts.query(ts.Sample) \
+            run2 = session.query(ts.Run).filter(ts.Run.id == run2_id).one()
+            sample2 = session.query(ts.Sample) \
                         .filter(ts.Sample.run_id == run2_id) \
                         .filter(ts.Sample.test_id == testid).first()
         else:
             run2 = None
             sample2 = None
     except NoResultFound:
         # FIXME: Make this a nicer error page.
         abort(404)

lnt/trunk/lnt/server/ui/regression_views.py

class MultiCheckboxField(SelectMultipleField):
     option_widget = widgets.CheckboxInput()
 
 
 class TriagePageSelectedForm(Form):
     field_changes = MultiCheckboxField("Changes", coerce=int)
     name = StringField('name', validators=[DataRequired()])
 
 
-def get_fieldchange(ts, id):
-    return ts.query(ts.FieldChange).filter(ts.FieldChange.id == id).one()
+def get_fieldchange(session, ts, id):
+    return session.query(ts.FieldChange) \
+        .filter(ts.FieldChange.id == id) \
+        .one()
 
 
 @v4_route("/regressions/new", methods=["GET", "POST"])
 def v4_new_regressions():
     form = TriagePageSelectedForm(request.form)
+    session = request.session
     ts = request.get_testsuite()
     if request.method == 'POST' and \
             request.form['btn'] == "Create New Regression":
-        regression = new_regression(ts, form.field_changes.data)
+        regression = new_regression(session, ts, form.field_changes.data)
         flash("Created " + regression.title, FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_list",
                         highlight=regression.id))
     if request.method == 'POST' and request.form['btn'] == "Ignore Changes":
         msg = "Ignoring changes: "
         ignored = []
         for fc_id in form.field_changes.data:
             ignored.append(str(fc_id))
-            fc = get_fieldchange(ts, fc_id)
+            fc = get_fieldchange(session, ts, fc_id)
             ignored_change = ts.ChangeIgnore(fc)
-            ts.add(ignored_change)
-        ts.commit()
+            session.add(ignored_change)
+        session.commit()
         flash(msg + ", ".join(ignored), FLASH_SUCCESS)
 
 #    d = datetime.datetime.now()
 #    two_weeks_ago = d - datetime.timedelta(days=14)
-    recent_fieldchange = ts.query(ts.FieldChange) \
+    recent_fieldchange = session.query(ts.FieldChange) \
         .join(ts.Test) \
         .outerjoin(ts.ChangeIgnore) \
         .filter(ts.ChangeIgnore.id.is_(None)) \
         .outerjoin(ts.RegressionIndicator) \
         .filter(ts.RegressionIndicator.id.is_(None)) \
         .order_by(desc(ts.FieldChange.id)) \
         .limit(500) \
         .all()
     crs = []
 
     form.field_changes.choices = list()
     for fc in recent_fieldchange:
         if fc.old_value is None:
-            cr, key_run, _ = get_cr_for_field_change(ts, fc)
+            cr, key_run, _ = get_cr_for_field_change(session, ts, fc)
         else:
             cr = PrecomputedCR(fc.old_value, fc.new_value,
                                fc.field.bigger_is_better)
             key_run = get_first_runs_of_fieldchange(ts, fc)
-        current_cr, _, _ = get_cr_for_field_change(ts, fc, current=True)
+        current_cr, _, _ = get_cr_for_field_change(session, ts, fc,
+                                                   current=True)
         crs.append(ChangeData(fc, cr, key_run, current_cr))
         form.field_changes.choices.append((fc.id, 1,))
     return render_template("v4_new_regressions.html",
                            testsuite_name=g.testsuite_name,
                            changes=crs, analysis=lnt.server.reporting.analysis,
                            form=form, **ts_data(ts))
 
 
-def calc_impact(ts, fcs):
+def calc_impact(session, ts, fcs):
     crs = []
     for fc in fcs:
         if fc is None:
             continue
         if fc.old_value is None:
-            cr, _, _ = get_cr_for_field_change(ts, fc)
+            cr, _, _ = get_cr_for_field_change(session, ts, fc)
         else:
             cr = PrecomputedCR(fc.old_value, fc.new_value,
                                fc.field.bigger_is_better)
         crs.append(cr)
     if crs:
         olds = sum([x.previous for x in crs if x.previous])
         news = sum([x.current for x in crs if x.current])
         if olds and news:
def isoformat(self):
         return "-"
 
     def strftime(self, _):
         return "0"
 
 
 @v4_route("/regressions/", methods=["GET", "POST"])
 def v4_regression_list():
-
+    session = request.session
     ts = request.get_testsuite()
     form = MergeRegressionForm(request.form)
     machine_filter = request.args.get('machine_filter')
     state_filter = int(request.args.get('state', RegressionState.ACTIVE))
     # Merge requested regressions.
     if request.method == 'POST' and \
        request.form['merge_btn'] == "Merge Regressions":
-        reg_inds, regressions = _get_regressions_from_selected_form(form, ts)
+        reg_inds, regressions = _get_regressions_from_selected_form(session,
+                                                                    form, ts)
         links = []
         target = 0
         for i, r in enumerate(regressions):
             if r.bug:
                 target = i
                 links.append(r.bug)
 
-        new_regress = new_regression(ts, [x.field_change_id for x in reg_inds])
+        new_regress = new_regression(session, ts,
+                                     [x.field_change_id for x in reg_inds])
         new_regress.state = regressions[target].state
         new_regress.title = regressions[target].title
         new_regress.bug = ' '.join(links)
         for r in regressions:
             r.bug = v4_url_for(".v4_regression_detail", id=new_regress.id)
             r.title = "Merged into Regression " + str(new_regress.id)
             r.state = RegressionState.IGNORED
-        [ts.delete(x) for x in reg_inds]
+        [session.delete(x) for x in reg_inds]
 
-        ts.commit()
+        session.commit()
         flash("Created: " + new_regress.title, FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_detail", id=new_regress.id))
     # Delete requested regressions.
     if request.method == 'POST' and \
             request.form['merge_btn'] == "Delete Regressions":
-        reg_inds, regressions = _get_regressions_from_selected_form(form, ts)
+        reg_inds, regressions = _get_regressions_from_selected_form(session,
+                                                                    form, ts)
         titles = [r.title for r in regressions]
         for res_ind in reg_inds:
-            ts.delete(res_ind)
+            session.delete(res_ind)
         for reg in regressions:
-            ts.delete(reg)
-        ts.commit()
+            session.delete(reg)
+        session.commit()
         flash(' Deleted: '.join(titles), FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_list", state=state_filter))
 
-    q = ts.query(ts.Regression)
+    q = session.query(ts.Regression)
     title = "All Regressions"
     if state_filter != -1:
         q = q.filter(ts.Regression.state == state_filter)
         title = RegressionState.names[state_filter]
     regression_info = q.all()[::-1]
 
     form.regression_checkboxes.choices = list()
     regression_sizes = []
     impacts = []
     ages = []
 
     filtered_regressions = []
     for regression in regression_info:
-        reg_inds = ts.query(ts.RegressionIndicator) \
+        reg_inds = session.query(ts.RegressionIndicator) \
             .filter(ts.RegressionIndicator.regression_id ==
                     regression.id) \
             .all()
         if machine_filter:
             machine_names = \
                 set([x.field_change.machine.name for x in reg_inds])
             if machine_filter in machine_names:
                 filtered_regressions.append(regression)
             else:
                 continue
         else:
             filtered_regressions.append(regression)
         form.regression_checkboxes.choices.append((regression.id, 1,))
 
         regression_sizes.append(len(reg_inds))
-        impacts.append(calc_impact(ts, [x.field_change for x in reg_inds]))
+        impacts.append(calc_impact(session, ts,
+                                   [x.field_change for x in reg_inds]))
         # Now guess the regression age:
         if len(reg_inds) and reg_inds[0].field_change and \
                 reg_inds[0].field_change.run:
             age = reg_inds[0].field_change.run.end_time
         else:
             age = EmptyDate()
         ages.append(age)
 
     return render_template("v4_regression_list.html",
                            testsuite_name=g.testsuite_name,
                            regressions=filtered_regressions,
                            highlight=request.args.get('highlight'),
                            title=title,
                            RegressionState=RegressionState,
                            state_filter=state_filter,
                            form=form,
                            sizes=regression_sizes,
                            impacts=impacts,
                            ages=ages,
                            analysis=lnt.server.reporting.analysis,
                            **ts_data(ts))
 
 
-def _get_regressions_from_selected_form(form, ts):
+def _get_regressions_from_selected_form(session, form, ts):
     regressions_id_to_merge = form.regression_checkboxes.data
-    regressions = ts.query(ts.Regression) \
+    regressions = session.query(ts.Regression) \
         .filter(ts.Regression.id.in_(regressions_id_to_merge)).all()
-    reg_inds = ts.query(ts.RegressionIndicator) \
+    reg_inds = session.query(ts.RegressionIndicator) \
         .filter(ts.RegressionIndicator.regression_id.in_(
             regressions_id_to_merge)) \
         .all()
     return reg_inds, regressions
 
 
 class EditRegressionForm(Form):
     title = StringField(u'Title', validators=[DataRequired()])
def default(self, obj):
             return
         if name(obj) == "SampleField":
             return obj.name
         return flask.json.JSONEncoder.default(self, obj)
 
 
 @v4_route("/regressions/<int:id>", methods=["GET", "POST"])
 def v4_regression_detail(id):
+    session = request.session
     ts = request.get_testsuite()
     form = EditRegressionForm(request.form)
 
     try:
-        regression_info = ts.query(ts.Regression) \
+        regression_info = session.query(ts.Regression) \
             .filter(ts.Regression.id == id) \
             .one()
     except NoResultFound as e:
         abort(404)
     if request.method == 'POST' and request.form['save_btn'] == "Save Changes":
         regression_info.title = form.title.data
         regression_info.bug = form.bug.data
         regression_info.state = form.state.data
-        ts.commit()
+        session.commit()
         flash("Updated " + regression_info.title, FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_list",
                         highlight=regression_info.id,
                         state=int(form.edit_state.data)))
     if request.method == 'POST' and \
             request.form['save_btn'] == "Split Regression":
         # For each of the regression indicators, grab their field ids.
-        res_inds = ts.query(ts.RegressionIndicator) \
+        res_inds = session.query(ts.RegressionIndicator) \
             .filter(ts.RegressionIndicator.field_change_id.in_(
                 form.field_changes.data)) \
             .all()
         fc_ids = [x.field_change_id for x in res_inds]
-        second_regression = new_regression(ts, fc_ids)
+        second_regression = new_regression(session, ts, fc_ids)
         second_regression.state = regression_info.state
 
         # Now remove our links to this regression.
         for res_ind in res_inds:
-            ts.delete(res_ind)
+            session.delete(res_ind)
         lnt.server.db.fieldchange.rebuild_title(ts, regression_info)
-        ts.commit()
+        session.commit()
         flash("Split " + second_regression.title, FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_list",
                         highlight=second_regression.id,
                         state=int(form.edit_state.data)))
     if request.method == 'POST' and request.form['save_btn'] == "Delete":
         # For each of the regression indicators, grab their field ids.
         title = regression_info.title
-        res_inds = ts.query(ts.RegressionIndicator) \
+        res_inds = session.query(ts.RegressionIndicator) \
             .filter(
                 ts.RegressionIndicator.regression_id == regression_info.id) \
             .all()
         # Now remove our links to this regression.
         for res_ind in res_inds:
-            ts.delete(res_ind)
-        ts.delete(regression_info)
-        ts.commit()
+            session.delete(res_ind)
+        session.delete(regression_info)
+        session.commit()
         flash("Deleted " + title, FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_regression_list",
                         state=int(form.edit_state.data)))
     form.field_changes.choices = list()
     form.state.default = regression_info.state
     form.process()
     form.edit_state.data = regression_info.state
     form.title.data = regression_info.title
     form.bug.data = regression_info.bug
-    regression_indicators = ts.query(ts.RegressionIndicator) \
+    regression_indicators = session.query(ts.RegressionIndicator) \
         .filter(ts.RegressionIndicator.regression_id == id) \
         .all()
 
     crs = []
 
     test_suite_versions = set()
     form.field_changes.choices = list()
     # If we have more than 10 regressions, don't graph any by default.
     checkbox_state = 1
     if len(regression_indicators) >= 10:
         checkbox_state = 0
 
     for regression in regression_indicators:
         fc = regression.field_change
         if fc is None:
             continue
         if fc.old_value is None:
-            cr, key_run, all_runs = get_cr_for_field_change(ts, fc)
+            cr, key_run, all_runs = get_cr_for_field_change(session, ts, fc)
         else:
             cr = PrecomputedCR(fc.old_value, fc.new_value,
                                fc.field.bigger_is_better)
             key_run = get_first_runs_of_fieldchange(ts, fc)
-        current_cr, _, all_runs = get_cr_for_field_change(ts, fc, current=True)
+        current_cr, _, all_runs = get_cr_for_field_change(session, ts, fc,
+                                                          current=True)
         crs.append(ChangeData(fc, cr, key_run, current_cr))
         form.field_changes.choices.append((fc.id, checkbox_state,))
         for run in all_runs:
             ts_rev = key_run.parameters.get('test_suite_revision')
             if ts_rev and ts_rev != u'None':
                 test_suite_versions.add(ts_rev)
 
     if len(test_suite_versions) > 1:
return render_template("v4_regression_detail.html",
                            regression=regression_info, changes=crs,
                            form=form, analysis=lnt.server.reporting.analysis,
                            check_all=checkbox_state,
                            **ts_data(ts))
 
 
 @v4_route("/hook", methods=["GET"])
 def v4_hook():
+    session = request.session
     ts = request.get_testsuite()
-    rule_hooks.post_submission_hooks(ts, 0)
+    rule_hooks.post_submission_hooks(session, ts, 0)
     abort(400)
 
 
 @v4_route("/regressions/new_from_graph/<int:machine_id>/<int:test_id>"
           "/<int:field_index>/<int:run_id>", methods=["GET"])
 def v4_make_regression(machine_id, test_id, field_index, run_id):
     """This function is called to make a new regression from a graph data point.
 
     It is not nessessarly the case that there will be a real change there,
     so we must create a regression, bypassing the normal analysis.
 
     """
+    session = request.session
     ts = request.get_testsuite()
     field = ts.sample_fields[field_index]
     new_regression_id = 0
-    run = ts.query(ts.Run).get(run_id)
+    run = session.query(ts.Run).get(run_id)
 
-    runs = ts.query(ts.Run). \
+    runs = session.query(ts.Run). \
         filter(ts.Run.order_id == run.order_id). \
         filter(ts.Run.machine_id == run.machine_id). \
         all()
 
     if len(runs) == 0:
         abort(404)
 
-    previous_runs = ts.get_previous_runs_on_machine(run, 1)
+    previous_runs = ts.get_previous_runs_on_machine(session, run, 1)
 
     # Find our start/end order.
     if previous_runs != []:
         start_order = previous_runs[0].order
     else:
         start_order = run.order
     end_order = run.order
 
     # Load our run data for the creation of the new fieldchanges.
     runs_to_load = [r.id for r in (runs + previous_runs)]
 
-    runinfo = lnt.server.reporting.analysis.RunInfo(ts, runs_to_load)
+    runinfo = lnt.server.reporting.analysis.RunInfo(session, ts, runs_to_load)
 
     result = runinfo.get_comparison_result(
         runs, previous_runs, test_id, field,
         ts.Sample.get_hash_of_binary_field())
 
     # Try and find a matching FC and update, else create one.
-    f = None
-
     try:
-        f = ts.query(ts.FieldChange) \
+        f = session.query(ts.FieldChange) \
             .filter(ts.FieldChange.start_order == start_order) \
             .filter(ts.FieldChange.end_order == end_order) \
             .filter(ts.FieldChange.test_id == test_id) \
             .filter(ts.FieldChange.machine == run.machine) \
-            .filter(ts.FieldChange.field == field) \
+            .filter(ts.FieldChange.field_id == field.id) \
             .one()
     except sqlalchemy.orm.exc.NoResultFound:
-            f = None
-
-    if not f:
-        test = ts.query(ts.Test).filter(ts.Test.id == test_id).one()
+        # Create one
+        test = session.query(ts.Test).filter(ts.Test.id == test_id).one()
         f = ts.FieldChange(start_order=start_order,
                            end_order=run.order,
                            machine=run.machine,
                            test=test,
-                           field=field)
-        ts.add(f)
+                           field_id=field.id)
+        session.add(f)
+
     # Always update FCs with new values.
     if f:
         f.old_value = result.previous
         f.new_value = result.current
         f.run = run
-    ts.commit()
+    session.commit()
 
     # Make new regressions.
-    regression = new_regression(ts, [f.id])
+    regression = new_regression(session, ts, [f.id])
     regression.state = RegressionState.ACTIVE
 
-    ts.commit()
+    session.commit()
     logger.info("Manually created new regressions: {}".format(regression.id))
     flash("Created " + regression.title, FLASH_SUCCESS)
 
     return redirect(v4_url_for(".v4_regression_detail", id=regression.id))

lnt/trunk/lnt/server/ui/templates/v4_run.html

$('.profile-prev-only').tooltip();
   {% set graph_base=v4_url_for('.v4_graph', highlight_run=run.id) %}
   <form id="graph_selection_form" method="GET" action="{{ graph_base }}">
 
     {# Report one table for each primary field. #}
     {% for field in metric_fields %}
       {% set field_index = ts.get_field_index(field) %}
       <section id="{{ field.name }}">
           {% set tests = [] %}
-          {% set (runs, compare_runs) = request_info.sri.get_sliding_runs(run, compare_to, request_info.num_comparison_runs) %}
+          {% set (runs, compare_runs) = request_info.sri.get_sliding_runs(session, run, compare_to, request_info.num_comparison_runs) %}
           {% for test_name,test_id in test_info %}
             {% set cr = request_info.sri.get_comparison_result(
                 runs, compare_runs, test_id, field, hash_field) %}
             {% if cr.previous is not none or cr.current is not none %}
               {% if cr.current is none or cr.current >= test_min_value_filter %}
                 {% if tests.append((test_name, test_id, cr)) %}{% endif %}
               {% endif %}
             {% endif %}

lnt/trunk/lnt/server/ui/views.py

 from flask import abort
 from flask import current_app
 from flask import flash
 from flask import g
 from flask import make_response
 from flask import redirect
 from flask import render_template
 from flask import request, url_for
-from flask import session
 from flask_wtf import Form
 from sqlalchemy.orm import joinedload
 from sqlalchemy.orm.exc import NoResultFound
 from typing import List, Optional
 from wtforms import SelectField, StringField, SubmitField
 from wtforms.validators import DataRequired, Length
 
 import lnt.server.db.rules_manager
if g.testsuite_name is None:
                 if Info is not None:
                     g.testsuite_name = Info.get('tag')
         except:
             pass
     if g.testsuite_name is None:
         g.testsuite_name = 'nts'
 
     # Get a DB connection.
+    session = request.session
     db = request.get_db()
 
     result = lnt.util.ImportData.import_from_string(
-        current_app.old_config, g.db_name, db, g.testsuite_name, data_value,
-        updateMachine=updateMachine, mergeRun=merge)
+        current_app.old_config, g.db_name, db, session, g.testsuite_name,
+        data_value, updateMachine=updateMachine, mergeRun=merge)
 
     # It is nice to have a full URL to the run, so fixup the request URL
     # here were we know more about the flask instance.
     if result.get('result_url'):
         result['result_url'] = request.url_root + result['result_url']
 
     response = flask.jsonify(**result)
     error = result['error']
     if error is not None:
         response.status_code = 400
         logger.warning("%s: Submission rejected: %s" % (request.url, error))
     return response
 
 
 def ts_data(ts):
     """Data about the current testsuite used by layout.html which should be
     present in most templates."""
-    baseline_id = session.get(baseline_key())
-    baselines = ts.query(ts.Baseline).all()
+    baseline_id = flask.session.get(baseline_key())
+    baselines = request.session.query(ts.Baseline).all()
     return {
         'baseline_id': baseline_id,
         'baselines': baselines,
         'ts': ts
     }
 
 
 @db_route('/submitRun', methods=('GET', 'POST'))

 def v4_overview():
     ts = request.get_testsuite()
     return render_template("v4_overview.html", testsuite_name=g.testsuite_name,
                            **ts_data(ts))
 
 
 @v4_route("/recent_activity")
 def v4_recent_activity():
+    session = request.session
     ts = request.get_testsuite()
 
     # Get the most recent runs in this tag, we just arbitrarily limit to
     # looking at the last 100 submission.
-    recent_runs = ts.query(ts.Run) \
+    recent_runs = session.query(ts.Run) \
         .options(joinedload(ts.Run.order)) \
         .options(joinedload(ts.Run.machine)) \
         .order_by(ts.Run.start_time.desc()).limit(100)
     recent_runs = recent_runs.all()
 
     # Compute the active machine list.
     active_machines = dict((run.machine.name, run)
                            for run in recent_runs[::-1])
return render_template("v4_recent_activity.html",
                            **ts_data(ts))
 
 
 @v4_route("/machine/")
 def v4_machines():
     # Compute the list of associated runs, grouped by order.
 
     # Gather all the runs on this machine.
+    session = request.session
     ts = request.get_testsuite()
-    machines = ts.query(ts.Machine)
+    machines = session.query(ts.Machine)
 
     return render_template("all_machines.html", machines=machines,
                            **ts_data(ts))
 
 
 @v4_route("/machine/<int:machine_id>/latest")
 def v4_machine_latest(machine_id):
     """Return the most recent run on this machine."""
+    session = request.session
     ts = request.get_testsuite()
 
-    run = ts.query(ts.Run) \
+    run = session.query(ts.Run) \
         .filter(ts.Run.machine_id == machine_id) \
         .order_by(ts.Run.start_time.desc()) \
         .first()
     return redirect(v4_url_for('.v4_run', id=run.id, **request.args))
 
 
 @v4_route("/machine/<int:machine_id>/compare")
 def v4_machine_compare(machine_id):
     """Return the most recent run on this machine."""
+    session = request.session
     ts = request.get_testsuite()
     machine_compare_to_id = int(request.args['compare_to_id'])
-    machine_1_run = ts.query(ts.Run) \
+    machine_1_run = session.query(ts.Run) \
         .filter(ts.Run.machine_id == machine_id) \
         .order_by(ts.Run.start_time.desc()) \
         .first()
 
-    machine_2_run = ts.query(ts.Run) \
+    machine_2_run = session.query(ts.Run) \
         .filter(ts.Run.machine_id == machine_compare_to_id) \
         .order_by(ts.Run.start_time.desc()) \
         .first()
 
     return redirect(v4_url_for('.v4_run', id=machine_1_run.id,
                                compare_to=machine_2_run.id))
 
 
 @v4_route("/machine/<int:id>")
 def v4_machine(id):
 
     # Compute the list of associated runs, grouped by order.
     from lnt.server.ui import util
 
     # Gather all the runs on this machine.
+    session = request.session
     ts = request.get_testsuite()
 
     associated_runs = multidict.multidict(
         (run_order, r)
-        for r, run_order in (ts.query(ts.Run, ts.Order)
+        for r, run_order in (session.query(ts.Run, ts.Order)
                              .join(ts.Order)
                              .filter(ts.Run.machine_id == id)
                              .order_by(ts.Run.start_time.desc())))
     associated_runs = associated_runs.items()
     associated_runs.sort()
 
     try:
-        machine = ts.query(ts.Machine).filter(ts.Machine.id == id).one()
+        machine = session.query(ts.Machine).filter(ts.Machine.id == id).one()
     except NoResultFound:
         abort(404)
 
     if request.args.get('json'):
         json_obj = dict()
         json_obj['name'] = machine.name
         json_obj['id'] = machine.id
         json_obj['runs'] = []
         for order in associated_runs:
             rev = order[0].llvm_project_revision
             for run in order[1]:
                 json_obj['runs'].append((run.id, rev,
                                          run.start_time.isoformat(),
                                          run.end_time.isoformat()))
         return flask.jsonify(**json_obj)
 
-    machines = ts.query(ts.Machine).all()
+    machines = session.query(ts.Machine).all()
     relatives = [m for m in machines if m.name == machine.name]
     return render_template("v4_machine.html",
                            testsuite_name=g.testsuite_name,
                            id=id,
                            associated_runs=associated_runs,
                            machine=machine,
                            machines=machines,
                            relatives=relatives,
                            **ts_data(ts))
 
 
 class V4RequestInfo(object):
     def __init__(self, run_id):
+        session = request.session
         self.db = request.get_db()
+        self.session = session
         self.ts = ts = request.get_testsuite()
-        self.run = run = ts.query(ts.Run).filter_by(id=run_id).first()
+        self.run = run = session.query(ts.Run).filter_by(id=run_id).first()
         if run is None:
             abort(404)
 
         # Get the aggregation function to use.
         aggregation_fn_name = request.args.get('aggregation_fn')
         self.aggregation_fn = {'min': lnt.util.stats.safe_min,
                                'median': lnt.util.stats.median}.get(
             aggregation_fn_name, lnt.util.stats.safe_min)
 
         # Get the MW confidence level.
         try:
             confidence_lv = float(request.args.get('MW_confidence_lv'))
         except (TypeError, ValueError):
             confidence_lv = .05
         self.confidence_lv = confidence_lv
 
         # Find the neighboring runs, by order.
-        prev_runs = list(ts.get_previous_runs_on_machine(run, N=3))
-        next_runs = list(ts.get_next_runs_on_machine(run, N=3))
+        prev_runs = list(ts.get_previous_runs_on_machine(session, run, N=3))
+        next_runs = list(ts.get_next_runs_on_machine(session, run, N=3))
         self.neighboring_runs = next_runs[::-1] + [self.run] + prev_runs
 
         # Select the comparison run as either the previous run, or a user
         # specified comparison run.
         compare_to_str = request.args.get('compare_to')
         if compare_to_str:
             compare_to_id = int(compare_to_str)
-            compare_to = ts.query(ts.Run).filter_by(id=compare_to_id).first()
+            compare_to = session.query(ts.Run) \
+                .filter_by(id=compare_to_id) \
+                .first()
             if compare_to is None:
                 flash("Comparison Run is invalid: " + compare_to_str,
                       FLASH_DANGER)
             else:
                 self.comparison_neighboring_runs = (
-                    list(ts.get_next_runs_on_machine(compare_to, N=3))[::-1] +
+                    list(ts.get_next_runs_on_machine(session, compare_to,
+                                                     N=3))[::-1] +
                     [compare_to] +
-                    list(ts.get_previous_runs_on_machine(compare_to, N=3)))
+                    list(ts.get_previous_runs_on_machine(session, compare_to,
+                                                         N=3)))
         else:
             if prev_runs:
                 compare_to = prev_runs[0]
             else:
                 compare_to = None
             self.comparison_neighboring_runs = self.neighboring_runs
 
         try:
             self.num_comparison_runs = int(
                 request.args.get('num_comparison_runs'))
         except:
             self.num_comparison_runs = 0
 
         # Find the baseline run, if requested.
         baseline_str = request.args.get('baseline')
         if baseline_str:
             baseline_id = int(baseline_str)
-            baseline = ts.query(ts.Run).filter_by(id=baseline_id).first()
+            baseline = session.query(ts.Run).filter_by(id=baseline_id).first()
             if baseline is None:
                 flash("Could not find baseline " + baseline_str, FLASH_DANGER)
         else:
             baseline = None
 
         # Gather the runs to use for statistical data.
         comparison_start_run = compare_to or self.run
 
         # We're going to render this on a real webpage with CSS support, so
         # override the default styles and provide bootstrap class names for
         # the tables.
         styles = {
             'body': '', 'td': '',
             'h1': 'font-size: 14pt',
             'table': 'width: initial; font-size: 9pt;',
             'th': 'text-align: center;'
         }
         classes = {
             'table': 'table table-striped table-condensed table-hover'
         }
 
         self.data = lnt.server.reporting.runs.generate_run_data(
-            self.run, baseurl=db_url_for('.index', _external=True),
+            session, self.run, baseurl=db_url_for('.index', _external=True),
             result=None, compare_to=compare_to, baseline=baseline,
             num_comparison_runs=self.num_comparison_runs,
             aggregation_fn=self.aggregation_fn, confidence_lv=confidence_lv,
             styles=styles, classes=classes)
         self.sri = self.data['sri']
         note = self.data['visible_note']
         if note:
             flash(note, FLASH_INFO)

 # Compatilibity route for old run pages.
 @db_route("/simple/<tag>/<int:id>/")
 def simple_run(tag, id):
     # Get the expected test suite.
     db = request.get_db()
     ts = db.testsuite[tag]
 
     # Look for a matched run.
-    matched_run = ts.query(ts.Run).\
+    matched_run = session.query(ts.Run).\
         filter(ts.Run.simple_run_id == id).\
         first()
 
     # If we found one, redirect to it's report.
     if matched_run is not None:
         return redirect(db_url_for(".v4_run", testsuite_name=tag,
                                    id=matched_run.id))
 
     # Otherwise, report an error.
     return render_template("error.html", message="""\
 Unable to find a run for this ID. Please use the native v4 URL interface
 (instead of the /simple/... URL schema).""")
 
 
 @v4_route("/<int:id>")
 def v4_run(id):
     info = V4RequestInfo(id)
 
+    session = info.session
     ts = info.ts
     run = info.run
 
     # Parse the view options.
     options = {}
     options['show_delta'] = bool(request.args.get('show_delta'))
     options['show_previous'] = bool(request.args.get('show_previous'))
     options['show_stddev'] = bool(request.args.get('show_stddev'))
def v4_run(id):
     if test_min_value_filter_str != '':
         test_min_value_filter = float(test_min_value_filter_str)
     else:
         test_min_value_filter = 0.0
 
     options['aggregation_fn'] = request.args.get('aggregation_fn', 'min')
 
     # Get the test names.
-    test_info = ts.query(ts.Test.name, ts.Test.id).\
+    test_info = session.query(ts.Test.name, ts.Test.id).\
         order_by(ts.Test.name).all()
 
     # Filter the list of tests by name, if requested.
     if test_filter_re:
         test_info = [test
                      for test in test_info
                      if test_filter_re.search(test[0])]
 
     if request.args.get('json'):
         json_obj = dict()
 
-        sri = lnt.server.reporting.analysis.RunInfo(ts, [id])
-        reported_tests = ts.query(ts.Test.name, ts.Test.id).\
+        sri = lnt.server.reporting.analysis.RunInfo(session, ts, [id])
+        reported_tests = session.query(ts.Test.name, ts.Test.id).\
             filter(ts.Run.id == id).\
             filter(ts.Test.id.in_(sri.test_ids)).all()
         order = run.order.as_ordered_string()
 
         for test_name, test_id in reported_tests:
             test = dict(test_name=test_name, test_id=test_id,
                         order=order, machine=run.machine.name)
             for sample_field in ts.sample_fields:
class PromoteOrderToBaseline(Form):
     update = SubmitField('Update')
     demote = SubmitField('Demote')
 
 
 @v4_route("/order/<int:id>", methods=['GET', 'POST'])
 def v4_order(id):
     """Order page details order information, as well as runs that are in this
     order as well setting this run as a baseline."""
+    session = request.session
     ts = request.get_testsuite()
     form = PromoteOrderToBaseline()
 
     if form.validate_on_submit():
         try:
-            baseline = ts.query(ts.Baseline) \
+            baseline = session.query(ts.Baseline) \
                 .filter(ts.Baseline.order_id == id) \
                 .one()
         except NoResultFound:
             baseline = ts.Baseline()
 
         if form.demote.data:
-            ts.session.delete(baseline)
-            ts.session.commit()
+            session.delete(baseline)
+            session.commit()
 
             flash("Baseline demoted.", FLASH_SUCCESS)
         else:
             baseline.name = form.name.data
             baseline.comment = form.description.data
             baseline.order_id = id
-            ts.session.add(baseline)
-            ts.session.commit()
+            session.add(baseline)
+            session.commit()
 
             flash("Baseline {} updated.".format(baseline.name), FLASH_SUCCESS)
         return redirect(v4_url_for(".v4_order", id=id))
     else:
         print form.errors
 
     try:
-        baseline = ts.query(ts.Baseline) \
+        baseline = session.query(ts.Baseline) \
             .filter(ts.Baseline.order_id == id) \
             .one()
         form.name.data = baseline.name
         form.description.data = baseline.comment
     except NoResultFound:
         pass
 
     # Get the order.
-    order = ts.query(ts.Order).filter(ts.Order.id == id).first()
+    order = session.query(ts.Order).filter(ts.Order.id == id).first()
     if order is None:
         abort(404)
 
     previous_order = None
     if order.previous_order_id:
-        previous_order = ts.query(ts.Order) \
+        previous_order = session.query(ts.Order) \
             .filter(ts.Order.id == order.previous_order_id).one()
     next_order = None
     if order.next_order_id:
-        next_order = ts.query(ts.Order) \
+        next_order = session.query(ts.Order) \
             .filter(ts.Order.id == order.next_order_id).one()
 
-    runs = ts.query(ts.Run) \
+    runs = session.query(ts.Run) \
         .filter(ts.Run.order_id == id) \
         .options(joinedload(ts.Run.machine)) \
         .all()
     num_runs = len(runs)
 
     return render_template("v4_order.html", order=order, form=form,
                            previous_order=previous_order,
                            next_order=next_order, runs=runs, num_runs=num_runs,
                            **ts_data(ts))
 
 
 @v4_route("/set_baseline/<int:id>")
 def v4_set_baseline(id):
     """Update the baseline stored in the user's session."""
+    session = request.session
     ts = request.get_testsuite()
-    base = ts.query(ts.Baseline).get(id)
+    base = session.query(ts.Baseline).get(id)
     if not base:
         return abort(404)
     flash("Baseline set to " + base.name, FLASH_SUCCESS)
-    session[baseline_key()] = id
+    flask.session[baseline_key()] = id
 
     return redirect(get_redirect_target())
 
 
 @v4_route("/all_orders")
 def v4_all_orders():
     # Get the testsuite.
+    session = request.session
     ts = request.get_testsuite()
 
     # Get the orders.
-    orders = ts.query(ts.Order).all()
+    orders = session.query(ts.Order).all()
 
     # Order the runs totally.
     orders.sort()
 
     return render_template("v4_all_orders.html", orders=orders, **ts_data(ts))
 
 
 @v4_route("/<int:id>/graph")
 def v4_run_graph(id):
     # This is an old style endpoint that treated graphs as associated with
     # runs. Redirect to the new endpoint.
 
+    session = request.session
     ts = request.get_testsuite()
-    run = ts.query(ts.Run).filter_by(id=id).first()
+    run = session.query(ts.Run).filter_by(id=id).first()
     if run is None:
         abort(404)
 
     # Convert the old style test parameters encoding.
     args = {'highlight_run': id}
     plot_number = 0
     for name, value in request.args.items():
         # If this isn't a test specification, just forward it.
def v4_graph_for_sample(sample_id, field_name):
     page, for that sample.  Extra args are passed through, to allow the caller
     to customize the graph page displayed, with for example run highlighting.
 
     :param sample_id: the sample ID from the database, obtained from the API.
     :param field_name: the name of the field.
     :return: a redirect to the graph page for that sample and field.
 
     """
+    session = request.session
     ts = request.get_testsuite()
-    target_sample = ts.query(ts.Sample).get(sample_id)
+    target_sample = session.query(ts.Sample).get(sample_id)
     if not target_sample:
         abort(404, "Could not find sample id: {}".format(sample_id))
 
     # Get the field index we are interested in.
     field_index = None
     for idx, f in enumerate(ts.sample_fields):
         if f.name == field_name:
             field_index = idx

 
 @v4_route("/graph")
 def v4_graph():
     from lnt.server.ui import util
     from lnt.testing import PASS
     from lnt.util import stats
     from lnt.external.stats import stats as ext_stats
 
+    session = request.session
     ts = request.get_testsuite()
     switch_min_mean_local = False
 
-    if 'switch_min_mean_session' not in session:
-        session['switch_min_mean_session'] = False
+    if 'switch_min_mean_session' not in flask.session:
+        flask.session['switch_min_mean_session'] = False
     # Parse the view options.
     options = {'min_mean_checkbox': 'min()'}
     if 'submit' in request.args:  # user pressed a button
         if 'switch_min_mean' in request.args:  # user checked mean() checkbox
-            session['switch_min_mean_session'] = options['switch_min_mean'] = \
+            flask.session['switch_min_mean_session'] = \
+                options['switch_min_mean'] = \
                 bool(request.args.get('switch_min_mean'))
-            switch_min_mean_local = session['switch_min_mean_session']
+            switch_min_mean_local = flask.session['switch_min_mean_session']
         else:  # mean() check box is not checked
-            session['switch_min_mean_session'] = options['switch_min_mean'] = \
+            flask.session['switch_min_mean_session'] = \
+                options['switch_min_mean'] = \
                 bool(request.args.get('switch_min_mean'))
-            switch_min_mean_local = session['switch_min_mean_session']
+            switch_min_mean_local = flask.session['switch_min_mean_session']
     else:  # new page was loaded by clicking link, not submit button
         options['switch_min_mean'] = switch_min_mean_local = \
-            session['switch_min_mean_session']
+            flask.session['switch_min_mean_session']
 
     options['hide_lineplot'] = bool(request.args.get('hide_lineplot'))
     show_lineplot = not options['hide_lineplot']
     options['show_mad'] = show_mad = bool(request.args.get('show_mad'))
     options['show_stddev'] = show_stddev = \
         bool(request.args.get('show_stddev'))
     options['hide_all_points'] = hide_all_points = bool(
         request.args.get('hide_all_points'))
for name, value in request.args.items():
         except:
             return abort(400)
 
         if not (0 <= field_index < len(ts.sample_fields)):
             return abort(404)
 
         try:
             machine = \
-                ts.query(ts.Machine).filter(ts.Machine.id == machine_id).one()
-            test = ts.query(ts.Test).filter(ts.Test.id == test_id).one()
+                session.query(ts.Machine) \
+                    .filter(ts.Machine.id == machine_id) \
+                    .one()
+            test = session.query(ts.Test).filter(ts.Test.id == test_id).one()
             field = ts.sample_fields[field_index]
         except NoResultFound:
             return abort(404)
         graph_parameters.append((machine, test, field, field_index))
 
     # Order the plots by machine name, test name and then field.
     graph_parameters.sort(key=lambda (m, t, f, _): (m.name, t.name, f.name, _))
 
for name, value in request.args.items():
             field_index = int(field_index_str)
         except ValueError:
             return abort(400)
 
         if not (0 <= field_index < len(ts.sample_fields)):
             return abort(404)
 
         try:
-            machine = \
-                ts.query(ts.Machine).filter(ts.Machine.id == machine_id).one()
+            machine = session.query(ts.Machine) \
+                .filter(ts.Machine.id == machine_id) \
+                .one()
         except NoResultFound:
             return abort(404)
         field = ts.sample_fields[field_index]
 
         mean_parameter = (machine, field)
 
     # Sanity check the arguments.
     if not graph_parameters and not mean_parameter:
for name, value in request.args.items():
 
         run_id_str = value
         try:
             run_id = int(run_id_str)
         except:
             return abort(400)
 
         try:
-            run = ts.query(ts.Run) \
+            run = session.query(ts.Run) \
                 .options(joinedload(ts.Run.machine)) \
                 .filter(ts.Run.id == run_id) \
                 .one()
         except:
             err_msg = ("The run {} was not found in the database."
                        .format(run_id))
             return render_template("error.html",
                                    message=err_msg)
 
         baseline_parameters.append((run, baseline_title))
 
     # Create region of interest for run data region if we are performing a
     # comparison.
     revision_range = None
     highlight_run_id = request.args.get('highlight_run')
     if show_highlight and highlight_run_id and highlight_run_id.isdigit():
-        highlight_run = ts.query(ts.Run).filter_by(
+        highlight_run = session.query(ts.Run).filter_by(
             id=int(highlight_run_id)).first()
         if highlight_run is None:
             abort(404)
 
         # Find the neighboring runs, by order.
-        prev_runs = list(ts.get_previous_runs_on_machine(highlight_run, N=1))
+        prev_runs = list(ts.get_previous_runs_on_machine(session,
+                                                         highlight_run, N=1))
         if prev_runs:
             start_rev = prev_runs[0].order.llvm_project_revision
             end_rev = highlight_run.order.llvm_project_revision
             revision_range = {
                 "start": convert_revision(start_rev),
                 "end": convert_revision(end_rev),
             }
 
for i, (machine, test, field, field_index) in enumerate(graph_parameters):
                                  url))
 
         # Load all the field values for this test on the same machine.
         #
         # FIXME: Don't join to Order here, aggregate this across all the tests
         # we want to load. Actually, we should just make this a single query.
         #
         # FIXME: Don't hard code field name.
-        q = ts.query(field.column, ts.Order.llvm_project_revision,
+        q = session.query(field.column, ts.Order.llvm_project_revision,
                      ts.Run.start_time, ts.Run.id) \
             .join(ts.Run).join(ts.Order) \
             .filter(ts.Run.machine_id == machine.id) \
             .filter(ts.Sample.test == test) \
             .filter(field.column.isnot(None))
 
         # Unless all samples requested, filter out failing tests.
         if not show_failures:
             if field.status_field:
                 q = q.filter((field.status_field.column == PASS) |
                              (field.status_field.column.is_(None)))
 
         # Aggregate by revision.
         data = multidict.multidict((rev, (val, date, run_id))
                                    for val, rev, date, run_id in q).items()
         data.sort(key=lambda sample: convert_revision(sample[0]))
 
         graph_datum.append((test.name, data, col, field, url))
 
         # Get baselines for this line
         num_baselines = len(baseline_parameters)
         for baseline_id, (baseline, baseline_title) in \
                 enumerate(baseline_parameters):
-            q_baseline = ts.query(field.column, ts.Order.llvm_project_revision,
+            q_baseline = session.query(field.column,
+                                       ts.Order.llvm_project_revision,
                                        ts.Run.start_time, ts.Machine.name) \
                          .join(ts.Run).join(ts.Order).join(ts.Machine) \
                          .filter(ts.Run.id == baseline.id) \
                          .filter(ts.Sample.test == test) \
                          .filter(field.column.isnot(None))
             # In the event of many samples, use the mean of the samples as the
             # baseline.
             samples = []
             for sample in q_baseline:
def v4_graph():
     # Draw mean trend if requested.
     if mean_parameter:
         machine, field = mean_parameter
         test_name = 'Geometric Mean'
 
         col = (0, 0, 0)
         legend.append(LegendItem(machine, test_name, field.name, col, None))
 
-        q = ts.query(sqlalchemy.sql.func.min(field.column),
+        q = session.query(sqlalchemy.sql.func.min(field.column),
                           ts.Order.llvm_project_revision,
                           sqlalchemy.sql.func.min(ts.Run.start_time)) \
               .join(ts.Run).join(ts.Order).join(ts.Test) \
               .filter(ts.Run.machine_id == machine.id) \
               .filter(field.column.isnot(None)) \
               .group_by(ts.Order.llvm_project_revision, ts.Test)
 
         # Calculate geomean of each revision.
         data = multidict.multidict(
                     ((rev, date), val) for val, rev, date in q).items()
return render_template("v4_graph.html", options=options,
                            baseline_plots=baseline_plots,
                            **ts_data(ts))
 
 
 @v4_route("/global_status")
 def v4_global_status():
     from lnt.server.ui import util
 
+    session = request.session
     ts = request.get_testsuite()
     metric_fields = sorted(list(ts.Sample.get_metric_fields()),
                            key=lambda f: f.name)
     fields = dict((f.name, f) for f in metric_fields)
 
     # Get the latest run.
-    latest = ts.query(ts.Run.start_time).\
+    latest = session.query(ts.Run.start_time).\
         order_by(ts.Run.start_time.desc()).first()
 
     # If we found an entry, use that.
     if latest is not None:
         latest_date, = latest
     else:
         # Otherwise, just use today.
         latest_date = datetime.date.today()
 
     # Create a datetime for the day before the most recent run.
     yesterday = latest_date - datetime.timedelta(days=1)
 
     # Get arguments.
     revision = int(request.args.get('revision',
                                     ts.Machine.DEFAULT_BASELINE_REVISION))
     field = fields.get(request.args.get('field', None), metric_fields[0])
 
     # Get the list of all runs we might be interested in.
-    recent_runs = ts.query(ts.Run).filter(ts.Run.start_time > yesterday).all()
+    recent_runs = session.query(ts.Run) \
+        .filter(ts.Run.start_time > yesterday) \
+        .all()
 
     # Aggregate the runs by machine.
     recent_runs_by_machine = multidict.multidict()
     for run in recent_runs:
         recent_runs_by_machine[run.machine] = run
 
     # Get a sorted list of recent machines.
     recent_machines = sorted(recent_runs_by_machine.keys(),
def v4_global_status():
     # over.
     machine_run_info = []
     reported_run_ids = []
 
     for machine in recent_machines:
         runs = recent_runs_by_machine[machine]
 
         # Get the baseline run for this machine.
-        baseline = machine.get_closest_previously_reported_run(revision)
+        baseline = machine.get_closest_previously_reported_run(session,
+                                                               revision)
 
         # Choose the "best" run to report on. We want the most recent one with
         # the most recent order.
         run = max(runs, key=lambda r: (r.order, r.start_time))
         if baseline:
             machine_run_info.append((baseline, run))
             reported_run_ids.append(baseline.id)
         reported_run_ids.append(run.id)
 
     # Get the set all tests reported in the recent runs.
-    reported_tests = ts.query(ts.Test.id, ts.Test.name).filter(
+    reported_tests = session.query(ts.Test.id, ts.Test.name).filter(
         sqlalchemy.sql.exists('*', sqlalchemy.sql.and_(
             ts.Sample.run_id.in_(reported_run_ids),
             ts.Sample.test_id == ts.Test.id))).all()
 
     # Load all of the runs we are interested in.
-    runinfo = lnt.server.reporting.analysis.RunInfo(ts, reported_run_ids)
+    runinfo = lnt.server.reporting.analysis.RunInfo(session, ts,
+                                                    reported_run_ids)
 
     # Build the test matrix. This is a two dimensional table index by
     # (machine-index, test-index), where each entry is the percent change.
     test_table = []
     for i, (test_id, test_name) in enumerate(reported_tests):
         # Create the row, starting with the test name and worst entry.
         row = [(test_id, test_name), None]
 
return render_template("v4_global_status.html",
                            selected_revision=revision,
                            **ts_data(ts))
 
 
 @v4_route("/daily_report")
 def v4_daily_report_overview():
     # Redirect to the report for the most recent submitted run's date.
 
+    session = request.session
     ts = request.get_testsuite()
 
     # Get the latest run.
-    latest = ts.query(ts.Run).\
+    latest = session.query(ts.Run).\
         order_by(ts.Run.start_time.desc()).limit(1).first()
 
     # If we found a run, use it's start time.
     if latest:
         date = latest.start_time
     else:
         # Otherwise, just use today.
         date = datetime.date.today()
def v4_daily_report(year, month, day):
     day_start_str = request.args.get('day_start')
     if day_start_str is not None:
         day_start = int(day_start_str)
     else:
         day_start = 16
 
     filter_machine_regex = request.args.get('filter-machine-regex')
 
+    session = request.session
     ts = request.get_testsuite()
 
     # Create the report object.
     report = lnt.server.reporting.dailyreport.DailyReport(
         ts, year, month, day, num_days, day_start,
         filter_machine_regex=filter_machine_regex)
 
     # Build the report.
     try:
-        report.build()
+        report.build(request.session)
     except ValueError:
         return abort(400)
 
     return render_template("v4_daily_report.html", report=report,
                            analysis=lnt.server.reporting.analysis,
                            **ts_data(ts))
 
 ###
def v4_summary_report_ui():
     def to_key(name):
         first = name.split('.', 1)[0]
         if first.isdigit():
             return (int(first), name)
         return (first, name)
     all_machines = set()
     all_orders = set()
     for ts in testsuites:
-        for name, in ts.query(ts.Machine.name):
+        for name, in session.query(ts.Machine.name):
             all_machines.add(name)
-        for name, in ts.query(ts.Order.llvm_project_revision):
+        for name, in session.query(ts.Order.llvm_project_revision):
             all_orders.add(name)
     all_machines = sorted(all_machines)
     all_orders = sorted(all_orders, key=to_key)
 
     return render_template("v4_summary_report_ui.html",
                            config=config, all_machines=all_machines,
-                           all_orders=all_orders)
+                           all_orders=all_orders, **ts_data(ts))
 
 
 @db_route("/summary_report")
 def v4_summary_report():
     # Load the summary report configuration.
     config_path = get_summary_config_path()
     if not os.path.exists(config_path):
         return render_template("error.html", message="""\

 def v4_search():
     def _isint(i):
         try:
             int(i)
             return True
         except:
             return False
 
+    session = request.session
     ts = request.get_testsuite()
     query = request.args.get('q')
     l = request.args.get('l', 8)
     default_machine = request.args.get('m', None)
 
     assert query
-    results = lnt.server.db.search.search(ts, query, num_results=l,
+    results = lnt.server.db.search.search(session, ts, query, num_results=l,
                                           default_machine=default_machine)
 
     return json.dumps(
         [('%s #%s' % (r.machine.name, r.order.llvm_project_revision),
           r.id)
          for r in results])
 
 
class MatrixOptions(Form):
     limit = SelectField('Size', choices=MATRIX_LIMITS)
 
 
 def baseline():
     # type: () -> Optional[testsuitedb.Baseline]
     """Get the baseline object from the user's current session baseline value
     or None if one is not defined.
     """
+    session = request.session
     ts = request.get_testsuite()
-    base_id = session.get(baseline_key())
+    base_id = flask.session.get(baseline_key())
     if not base_id:
         return None
     try:
-        base = ts.query(ts.Baseline).get(base_id)
+        base = session.query(ts.Baseline).get(base_id)
     except NoResultFound:
         return None
     return base
 
 
 @v4_route("/matrix", methods=['GET', 'POST'])
 def v4_matrix():
     """A table view for Run sample data, because *some* people really
     like to be able to see results textually.
     request.args.limit limits the number of samples.
     for each dataset to add, there will be a "plot.n=.m.b.f" where m is machine
     ID, b is benchmark ID and f os field kind offset. "n" is used to unique
     the paramters, and is ignored.
 
     """
+    session = request.session
     ts = request.get_testsuite()
     # Load the matrix request parameters.
     form = MatrixOptions(request.form)
     if request.method == 'POST':
         post_limit = form.limit.data
     else:
         post_limit = MATRIX_LIMITS[0][0]
     data_parameters = []  # type: List[MatrixDataRequest]
for name, value in request.args.items():
         except ValueError:
             err_msg = "data {} was malformed. {} must be int.int.int"
             return abort(400, err_msg.format(name, value))
 
         if not (0 <= field_index < len(ts.sample_fields)):
             return abort(404, "Invalid field index: {}".format(field_index))
 
         try:
-            machine = \
-                ts.query(ts.Machine).filter(ts.Machine.id == machine_id).one()
+            machine = session.query(ts.Machine) \
+                .filter(ts.Machine.id == machine_id) \
+                .one()
         except NoResultFound:
             return abort(404, "Invalid machine ID: {}".format(machine_id))
         try:
-            test = ts.query(ts.Test).filter(ts.Test.id == test_id).one()
+            test = session.query(ts.Test).filter(ts.Test.id == test_id).one()
         except NoResultFound:
             return abort(404, "Invalid test ID: {}".format(test_id))
         try:
             field = ts.sample_fields[field_index]
         except NoResultFound:
             return abort(404, "Invalid field_index: {}".format(field_index))
 
         valid_request = MatrixDataRequest(machine, test, field)
def v4_matrix():
 
     # It is nice for the columns to be sorted by name.
     data_parameters.sort(key=lambda x: x.test.name),
 
     # Now lets get the data.
     all_orders = set()
     order_to_id = {}
     for req in data_parameters:
-        q = ts.query(req.field.column, ts.Order.llvm_project_revision,
+        q = session.query(req.field.column, ts.Order.llvm_project_revision,
                           ts.Order.id) \
             .join(ts.Run) \
             .join(ts.Order) \
             .filter(ts.Run.machine_id == req.machine.id) \
             .filter(ts.Sample.test == req.test) \
             .filter(req.field.column.isnot(None)) \
             .order_by(ts.Order.llvm_project_revision.desc())
 
         limit = request.args.get('limit', post_limit)
if user_baseline:
         all_orders.add(user_baseline.order.llvm_project_revision)
         baseline_rev = user_baseline.order.llvm_project_revision
         baseline_name = user_baseline.name
     else:
         baseline_rev = backup_baseline
         baseline_name = backup_baseline
 
     for req in data_parameters:
-        q_baseline = ts.query(req.field.column, ts.Order.llvm_project_revision,
+        q_baseline = session.query(req.field.column,
+                                   ts.Order.llvm_project_revision,
                                    ts.Order.id) \
                        .join(ts.Run) \
                        .join(ts.Order) \
                        .filter(ts.Run.machine_id == req.machine.id) \
                        .filter(ts.Sample.test == req.test) \
                        .filter(req.field.column.isnot(None)) \
                        .filter(ts.Order.llvm_project_revision == baseline_rev)
         baseline_data = q_baseline.all()
         if baseline_data:
for order in all_orders:
             order_to_geomean[order] = cr
         else:
             # There will be no change here, but display current val.
             if curr_geomean:
                 order_to_geomean[order] = PrecomputedCR(curr_geomean,
                                                         curr_geomean,
                                                         False)
     # Calculate the date of each order.
-    runs = ts.query(ts.Run.start_time, ts.Order.llvm_project_revision) \
+    runs = session.query(ts.Run.start_time, ts.Order.llvm_project_revision) \
              .join(ts.Order) \
              .filter(ts.Order.llvm_project_revision.in_(all_orders)) \
              .all()
 
     order_to_date = dict([(x[1], x[0]) for x in runs])
 
     class FakeOptions(object):
         show_small_diff = False

lnt/trunk/lnt/tests/builtintest.py

 """
 Base class for builtin-in tests.
 """
 
 import sys
 import os
 
 from lnt.testing.util.misc import timestamp
 
 import lnt.util.ServerUtil as ServerUtil
 import lnt.util.ImportData as ImportData
 import lnt.server.config as server_config
-import lnt.server.db.v4db
 
 
 class OptsContainer(object):
     pass
 
 
 class BuiltinTest(object):
     def __init__(self):
def submit(self, report_path, config, ts_name=None):
 
         server_report = None
         if config.submit_url is not None:
             self.log("submitting result to %r" % (config.submit_url,))
             server_report = ServerUtil.submitFile(
                 config.submit_url, report_path, config.verbose,
                 updateMachine=config.update_machine, mergeRun=config.merge)
         else:
-            server_report = lnt.util.ImportData.no_submit()
+            server_report = ImportData.no_submit()
         if server_report:
             ImportData.print_report_result(server_report, sys.stdout, sys.stderr,
                                            config.verbose)
         return server_report
 
     @staticmethod
     def show_results_url(server_results):
         """Print the result URL"""
         result_url = server_results.get('result_url', None)
         if result_url is not None:
             print "Results available at:", server_results['result_url']

lnt/trunk/lnt/tests/nt.py

def submit_helper(self, config):
             # Simulate a submission to retrieve the results report.
             # Construct a temporary database and import the result.
             self.log("submitting result to dummy instance")
 
             import lnt.server.db.v4db
             import lnt.server.config
             db = lnt.server.db.v4db.V4DB("sqlite:///:memory:",
                                          lnt.server.config.Config.dummy_instance())
+            session = db.make_session()
             result = lnt.util.ImportData.import_and_report(
-                None, None, db, report_path, 'json', 'nts')
+                None, None, db, session, report_path, 'json', 'nts')
 
         if result is None:
             fatal("Results were not obtained from submission.")
 
         return result
 
 
 def _tools_check():

lnt/trunk/lnt/util/ImportData.py

 from lnt.util import NTEmailReport
 from lnt.util import async_ops
 from lnt.util import logger
 import collections
 import datetime
 import lnt.formats
 import lnt.server.reporting.analysis
 import lnt.testing
 import os
 import re
 import tempfile
 import time
 
 
-def import_and_report(config, db_name, db, file, format, ts_name,
+def import_and_report(config, db_name, db, session, file, format, ts_name,
                       show_sample_count=False, disable_email=False,
                       disable_report=False, updateMachine=False,
                       mergeRun='replace'):
     """
     import_and_report(config, db_name, db, file, format, ts_name,
                       [show_sample_count], [disable_email]) -> ... object ...
 
     Import a test data file into an LNT server and generate a test report. On
     success, run is the newly imported run.
 
     The result object is a dictionary containing information on the imported
     run and its comparison to the previous run.
     """
     result = {
         'success': False,
         'error': None,
         'import_file': file,
     }
 
-    ts = db.testsuite.get(ts_name)
+    ts = db.testsuite.get(ts_name, None)
     if ts is None:
         result['error'] = "Unknown test suite '%s'!" % ts_name
         return result
-    numMachines = ts.getNumMachines()
-    numRuns = ts.getNumRuns()
-    numTests = ts.getNumTests()
+    numMachines = ts.getNumMachines(session)
+    numRuns = ts.getNumRuns(session)
+    numTests = ts.getNumTests(session)
 
     # If the database gets fragmented, count(*) in SQLite can get really
     # slow!?!
     if show_sample_count:
-        numSamples = ts.getNumSamples()
+        numSamples = ts.getNumSamples(session)
 
     startTime = time.time()
     try:
         data = lnt.formats.read_any(file, format)
     except KeyboardInterrupt:
         raise
     except:
         import traceback
def import_and_report(config, db_name, db, session, file, format, ts_name,
     importStartTime = time.time()
     try:
         data_schema = data.get('schema')
         if data_schema is not None and data_schema != ts_name:
             result['error'] = ("Importing '%s' data into test suite '%s'" %
                                (data_schema, ts_name))
             return result
 
-        run = ts.importDataFromDict(data, config=db_config,
+        run = ts.importDataFromDict(session, data, config=db_config,
                                     updateMachine=updateMachine,
                                     mergeRun=mergeRun)
     except KeyboardInterrupt:
         raise
     except Exception as e:
         import traceback
         result['error'] = "import failure: %s" % e.message
         result['message'] = traceback.format_exc()
def import_and_report(config, db_name, db, session, file, format, ts_name,
     if config:
         report_url = "%s/db_%s/" % (config.zorgURL, db_name)
     else:
         report_url = "localhost"
 
     if not disable_report:
         #  This has the side effect of building the run report for
         #  this result.
-        NTEmailReport.emailReport(result, db, run, report_url, email_config,
-                                  toAddress, True)
+        NTEmailReport.emailReport(result, session, run, report_url,
+                                  email_config, toAddress, True)
 
-    result['added_machines'] = ts.getNumMachines() - numMachines
-    result['added_runs'] = ts.getNumRuns() - numRuns
-    result['added_tests'] = ts.getNumTests() - numTests
+    result['added_machines'] = ts.getNumMachines(session) - numMachines
+    result['added_runs'] = ts.getNumRuns(session) - numRuns
+    result['added_tests'] = ts.getNumTests(session) - numTests
     if show_sample_count:
-        result['added_samples'] = ts.getNumSamples() - numSamples
+        result['added_samples'] = ts.getNumSamples(session) - numSamples
 
     result['committed'] = True
     result['run_id'] = run.id
-    ts.commit()
+    session.commit()
     if db_config:
         #  If we are not in a dummy instance, also run background jobs.
         #  We have to have a commit before we run, so subprocesses can
         #  see the submitted data.
         async_ops.async_fieldchange_calc(db_name, ts, run, config)
 
     # Add a handy relative link to the submitted run.
     result['result_url'] = "db_{}/v4/{}/{}".format(db_name, ts_name, run.id)
     result['report_time'] = time.time() - importStartTime
     result['total_time'] = time.time() - startTime
     logger.info("Successfully created {}".format(result['result_url']))
     # If this database has a shadow import configured, import the run into that
     # database as well.
     if config and config.databases[db_name].shadow_import:
         # Load the shadow database to import into.
         db_config = config.databases[db_name]
         shadow_name = db_config.shadow_import
         with closing(config.get_database(shadow_name)) as shadow_db:
             if shadow_db is None:
                 raise ValueError("invalid configuration, shadow import "
                                  "database %r does not exist" % shadow_name)
 
             # Perform the shadow import.
+            shadow_session = shadow_db.make_session()
             shadow_result = import_and_report(config, shadow_name,
-                                              shadow_db, file, format, ts_name,
+                                              shadow_db, shadow_session, file,
+                                              format, ts_name,
                                               show_sample_count, disable_email,
                                               disable_report, updateMachine)
 
             # Append the shadow result to the result.
             result['shadow_result'] = shadow_result
 
     result['success'] = True
     return result
if total_added:
             print >>out, "Added Samples : %d" % result['added_samples']
         print >>out
     print >>out, "Results"
     print >>out, "----------------"
     for kind, count in result_kinds.items():
         print >>out, kind, ":", count
 
 
-def import_from_string(config, db_name, db, ts_name, data, updateMachine=False,
-                       mergeRun='replace'):
+def import_from_string(config, db_name, db, session, ts_name, data,
+                       updateMachine=False, mergeRun='replace'):
     # Stash a copy of the raw submission.
     #
     # To keep the temporary directory organized, we keep files in
     # subdirectories organized by (database, year-month).
     utcnow = datetime.datetime.utcnow()
     tmpdir = os.path.join(config.tempDir, db_name,
                           "%04d-%02d" % (utcnow.year, utcnow.month))
     try:
def import_from_string(config, db_name, db, session, ts_name, data,
     os.close(fd)
 
     # Import the data.
     #
     # FIXME: Gracefully handle formats failures and DOS attempts. We
     # should at least reject overly large inputs.
 
     result = lnt.util.ImportData.import_and_report(
-        config, db_name, db, path, '<auto>', ts_name,
+        config, db_name, db, session, path, '<auto>', ts_name,
         updateMachine=updateMachine, mergeRun=mergeRun)
     return result

lnt/trunk/lnt/util/NTEmailReport.py

 import os
 import smtplib
 import sys
 import urllib
 
 import StringIO
-import lnt.server.db.v4db
 import lnt.server.reporting.runs
 
 
-def emailReport(result, db, run, baseurl, email_config, to, was_added=True):
+def emailReport(result, session, run, baseurl, email_config, to,
+                was_added=True):
     import email.mime.multipart
     import email.mime.text
 
-    subject, report, html_report = _getReport(result, db, run, baseurl,
+    subject, report, html_report = _getReport(result, session, run, baseurl,
                                               was_added)
 
     # Ignore if no to address was given, we do things this way because of the
     # awkward way we collect result information as part of generating the email
     # report.
     if email_config is None or to is None:
         return
 
else:
         msg.attach(email.mime.text.MIMEText(report, 'plain'))
         msg.attach(email.mime.text.MIMEText(html_report, 'html'))
 
     s = smtplib.SMTP(email_config.host)
     s.sendmail(email_config.from_address, [to], msg.as_string())
     s.quit()
 
 
-def _getReport(result, db, run, baseurl, was_added, compare_to=None):
-    assert isinstance(db, lnt.server.db.v4db.V4DB)
-
+def _getReport(result, session, run, baseurl, was_added, compare_to=None):
     data = lnt.server.reporting.runs.generate_run_data(
-        run, baseurl=baseurl, result=result, compare_to=compare_to,
+        session, run, baseurl=baseurl, result=result, compare_to=compare_to,
         num_comparison_runs=10)
 
     env = lnt.server.ui.app.create_jinja_environment()
     text_template = env.get_template('reporting/run_report.txt')
     text_report = text_template.render(data)
     html_template = env.get_template('reporting/run_report.html')
     html_report = html_template.render(data)
     return data['subject'], text_report, html_report

lnt/trunk/lnt/util/ServerUtil.py

def submitFileToInstance(path, file, updateMachine=False, mergeRun='replace'):
     # Otherwise, assume it is a local url and submit to the default database
     # in the instance.
     instance = lnt.server.instance.Instance.frompath(path)
     config = instance.config
     db_name = 'default'
     with contextlib.closing(config.get_database(db_name)) as db:
         if db is None:
             raise ValueError("no default database in instance: %r" % (path,))
+        session = db.make_session()
         return lnt.util.ImportData.import_and_report(
-            config, db_name, db, file, format='<auto>', ts_name='nts',
+            config, db_name, db, session, file, format='<auto>', ts_name='nts',
             updateMachine=updateMachine, mergeRun=mergeRun)
 
 
 def submitFile(url, file, verbose, updateMachine=False, mergeRun='replace'):
     # If this is a real url, submit it using urllib.
     if '://' in url:
         result = submitFileToServer(url, file, updateMachine, mergeRun)
     else:

lnt/trunk/lnt/util/async_ops.py

job = Process(target=async_wrapper,
                   args=[job, args, func_args])
 
     # Set this to make sure when parent dies, children are killed.
     job.daemon = True
 
     job.start()
     JOBS.append(job)
 
+
 # Flag to track if we have disposed of the parents database connections in
 # this subprocess.
 clean_db = False
 
 
 def async_wrapper(job, ts_args, func_args):
     """Setup test-suite in this subprocess and run something.
 
     Because of multipocessing, capture excptions and log messages,
     and return them.
     """
     global clean_db
     try:
         start_time = time.time()
 
         if not clean_db:
             lnt.server.db.v4db.V4DB.close_all_engines()
             clean_db = True
+
         sleep(3)
         logger.info("Running async wrapper: {} ".format(job.__name__) +
                     str(os.getpid()))
         config = ts_args['db_info']
-        _v4db = config.get_database(ts_args['db'])
-        # with contextlib.closing(_v4db) as db:
-        ts = _v4db.testsuite[ts_args['tsname']]
-        nothing = job(ts, **func_args)
+        db = config.get_database(ts_args['db'])
+        with contextlib.closing(db):
+            session = db.make_session()
+            ts = db.testsuite[ts_args['tsname']]
+            nothing = job(ts, session, **func_args)
             assert nothing is None
+            session.close()
         end_time = time.time()
         delta = end_time-start_time
         msg = "Finished: {name} in {time:.2f}s ".format(name=job.__name__,
                                                         time=delta)
         if delta < 100:
             logger.info(msg)
         else:
             logger.warning(msg)

lnt/trunk/tests/server/db/CreateV4TestSuite.py

 # This test just checks that we can construct and manipulate the test suite
 # model itself. The heavy lifting of constructing a test suite's databases,
 # etc. is checked in CreateV4TestSuiteInstance.
 #
 # RUN: rm -f %t.db
 # RUN: python %s %t.db
 
 import sys
 from lnt.server.config import Config
 from lnt.server.db import testsuite
 from lnt.server.db import v4db
 
 # Create an in memory database.
 db = v4db.V4DB("sqlite:///:memory:", Config.dummy_instance())
+session = db.make_session()
 
 # We expect exactly the NTS test suite.
-test_suites = list(db.query(testsuite.TestSuite))
+test_suites = list(session.query(testsuite.TestSuite))
 assert len(test_suites) == 1
 
 # Check the NTS test suite.
-ts = db.query(testsuite.TestSuite).filter_by(name="nts").first()
+ts = session.query(testsuite.TestSuite).filter_by(name="nts").first()
 assert ts.name == "nts"
 assert ts.db_key_name == "NT"
 assert len(ts.machine_fields) == 2
 assert len(ts.order_fields) == 1
 assert len(ts.run_fields) == 0
 
 assert ts.machine_fields[0].name == "hardware"
 assert ts.machine_fields[1].name == "os"
 
 assert ts.order_fields[0].name == "llvm_project_revision"
 
 assert ts.machine_fields[0].test_suite is ts
 assert ts.order_fields[0].test_suite is ts

lnt/trunk/tests/server/db/CreateV4TestSuiteInstance.py

 # Check the model bindings for test suite instances.
 #
 # RUN: rm -f %t.db
 # RUN: python %s %t.db
 
 import datetime
 
 from lnt.server.config import Config
 from lnt.server.db import testsuite
 from lnt.server.db import v4db
 from lnt.server.db.fieldchange import RegressionState
 
 # Create an in memory database.
 db = v4db.V4DB("sqlite:///:memory:", Config.dummy_instance())
+session = db.make_session()
 
 # Get the test suite wrapper.
 ts_db = db.testsuite['nts']
 
 # Check that we can construct and access all of the primary fields for the test
 # suite database objects.
 
 # Create the objects.

 
 run = ts_db.Run(machine, order, start_time, end_time)
 test = ts_db.Test("test-a")
 
 sample = ts_db.Sample(run, test, compile_time=1.0, score=4.2, code_size=100)
 sample.mem_bytes = 58093568
 
 # Add and commit.
-ts_db.add(machine)
-ts_db.add(order)
-ts_db.add(order2)
-ts_db.add(order3)
+session.add(machine)
+session.add(order)
+session.add(order2)
+session.add(order3)
 
 
-ts_db.add(run)
-ts_db.add(test)
-ts_db.add(sample)
+session.add(run)
+session.add(test)
+session.add(sample)
 field_change = ts_db.FieldChange(order, order2, machine, test,
-                                 list(sample.get_primary_fields())[0])
+                                 list(sample.get_primary_fields())[0].id)
 
-ts_db.add(field_change)
+session.add(field_change)
 
 field_change2 = ts_db.FieldChange(order2, order3, machine, test,
-                                  list(sample.get_primary_fields())[1])
-ts_db.add(field_change2)
+                                  list(sample.get_primary_fields())[1].id)
+session.add(field_change2)
 
 TEST_TITLE = "Some regression title"
 
 regression = ts_db.Regression(TEST_TITLE, "PR1234", RegressionState.DETECTED)
-ts_db.add(regression)
+session.add(regression)
 
 regression_indicator1 = ts_db.RegressionIndicator(regression, field_change)
 regression_indicator2 = ts_db.RegressionIndicator(regression, field_change2)
 
-ts_db.add(regression_indicator1)
-ts_db.add(regression_indicator2)
+session.add(regression_indicator1)
+session.add(regression_indicator2)
 
-ts_db.commit()
+session.commit()
 
 del machine, order, run, test, sample
 
 # Fetch the added objects.
-machines = ts_db.query(ts_db.Machine).all()
+machines = session.query(ts_db.Machine).all()
 assert len(machines) == 1
 machine = machines[0]
 
-orders = ts_db.query(ts_db.Order).all()
+orders = session.query(ts_db.Order).all()
 assert len(orders) == 3
 order = orders[0]
 
-runs = ts_db.query(ts_db.Run).all()
+runs = session.query(ts_db.Run).all()
 assert len(runs) == 1
 run = runs[0]
 
-tests = ts_db.query(ts_db.Test).all()
+tests = session.query(ts_db.Test).all()
 assert len(tests) == 1
 test = tests[0]
 
-samples = ts_db.query(ts_db.Sample).all()
+samples = session.query(ts_db.Sample).all()
 assert len(samples) == 1
 sample = samples[0]
 
 assert sample.code_size == 100
 
-regression_indicators = ts_db.query(ts_db.RegressionIndicator).all()
+regression_indicators = session.query(ts_db.RegressionIndicator).all()
 assert len(regression_indicators) == 2
 ri = regression_indicators[0]
 
 assert ri.regression.title == TEST_TITLE
 
 # Audit the various fields.
 assert machine.name == "test-machine"
 assert machine.os == "test-os"

lnt/trunk/tests/server/db/ImportV4TestSuiteInstance.py

 
 import lnt.testing
 from lnt.server.config import Config
 from lnt.server.db import testsuite
 from lnt.server.db import v4db
 
 # Load the test database.
 db = v4db.V4DB("sqlite:///%s" % sys.argv[1], Config.dummy_instance())
+session = db.make_session()
 
 # Get the status kinds, and validate the IDs align with the testing IDs.
-pass_kind = db.query(db.StatusKind).filter_by(id = lnt.testing.PASS).one()
+pass_kind = session.query(testsuite.StatusKind) \
+    .filter_by(id = lnt.testing.PASS).one()
 assert pass_kind.name == "PASS"
-fail_kind = db.query(db.StatusKind).filter_by(id = lnt.testing.FAIL).one()
+fail_kind = session.query(testsuite.StatusKind) \
+    .filter_by(id = lnt.testing.FAIL).one()
 assert fail_kind.name == "FAIL"
-xfail_kind = db.query(db.StatusKind).filter_by(id = lnt.testing.XFAIL).one()
+xfail_kind = session.query(testsuite.StatusKind) \
+    .filter_by(id = lnt.testing.XFAIL).one()
 assert xfail_kind.name == "XFAIL"
 
 # Load the imported test suite.
 ts = db.testsuite['nts']
 
 # Validate the machine.
-machines = list(ts.query(ts.Machine))
+machines = list(session.query(ts.Machine))
 assert len(machines) == 1
 machine = machines[0]
 assert machine.name == 'LNT SAMPLE MACHINE'
 assert machine.hardware == "x86_64"
 assert machine.os == "SAMPLE OS"
 parameters = machine.parameters
 assert len(parameters) == 1
 assert parameters['extrakey'] == u'extravalue'
 
 # Validate the tests.
-tests = list(ts.query(ts.Test))
+tests = list(session.query(ts.Test))
 assert len(tests) == 1
 test = tests[0]
 assert tests[0].name == 'sampletest'
 
 # Validate the orders.
-orders = list(ts.query(ts.Order).order_by(ts.Order.llvm_project_revision))
+orders = list(session.query(ts.Order).order_by(ts.Order.llvm_project_revision))
 assert len(orders) == 2
 order_a,order_b = orders
 print order_a
 print order_b
 assert order_a.previous_order_id is None
 assert order_a.next_order_id is order_b.id
 assert order_a.llvm_project_revision == '1'
 assert order_b.previous_order_id is order_a.id
 assert order_b.next_order_id is None
 assert order_b.llvm_project_revision == '2'
 
 # Validate the runs.
-runs = list(ts.query(ts.Run).order_by(ts.Run.order_id))
+runs = list(session.query(ts.Run).order_by(ts.Run.order_id))
 assert len(runs) == 2
 run_a,run_b = runs
 assert run_a.machine is machine
 assert run_b.machine is machine
 assert run_a.order is order_a
 assert run_b.order is order_b
 assert run_a.imported_from.endswith("sample-a-small.plist")
 assert run_b.imported_from.endswith("sample-b-small.plist")
 assert run_a.start_time == datetime.datetime(2009, 11, 17, 2, 12, 25)
 assert run_a.end_time == datetime.datetime(2009, 11, 17, 3, 44, 48)
 assert sorted(run_a.parameters.items()) == [('inferred_run_order', '1')]
 assert sorted(run_b.parameters.items()) == [('inferred_run_order', '2')]
 
 # Validate the samples.
-samples = list(ts.query(ts.Sample)\
+samples = list(session.query(ts.Sample)\
     .join(ts.Run) \
     .order_by(ts.Run.order_id, ts.Sample.id))
 assert len(samples) == 3
 sample_a_0,sample_a_1,sample_b = samples
 assert sample_a_0.run is run_a
 assert sample_a_1.run is run_a
 assert sample_b.run is run_b
 assert sample_a_0.test is test

lnt/trunk/tests/server/db/blacklist.py

 here = os.path.dirname(__file__)
 
 logging.basicConfig(level=logging.DEBUG)
 
 
 class BlacklistProcessingTest(unittest.TestCase):
     """Test the Rules facility."""
 
-    def _mkorder(self, ts, rev):
+    def _mkorder(self, session, ts, rev):
         order = ts.Order()
         order.llvm_project_revision = rev
-        ts.add(order)
+        session.add(order)
         return order
         
     def setUp(self):
         _, instance_path = sys.argv
 
         # Create the application instance.
         app = lnt.server.ui.app.App.create_standalone(instance_path)
         app.old_config.blacklist = here + "/blacklist"
         app.app_context().push()
         # Get the test suite wrapper.
         with app.test_request_context('/db_default/nts/foo') as r:
             app.preprocess_request()
             r.g.db_name = "default"
             r.g.testsuite_name = "nts"
+            r.request.db = app.instance.get_database(r.g.db_name)
+            r.request.session = r.request.db.make_session()
             self.ts = r.request.get_testsuite()
             self.ts_db = self.ts
+            self.session = r.request.session
+        session = self.session
         ts_db = self.ts_db
-        order1234 = self.order1234 = self._mkorder(ts_db, "1234")
-        order1236 = self.order1236 = self._mkorder(ts_db, "1236")
+        order1234 = self.order1234 = self._mkorder(session, ts_db, "1234")
+        order1236 = self.order1236 = self._mkorder(session, ts_db, "1236")
 
         machine = self.machine = ts_db.Machine("test-machine")
-        ts_db.add(machine)
+        session.add(machine)
 
         a_field = ts_db.Sample.fields[0]
 
-        ts_db.commit()
+        session.commit()
         
         test = self.test = ts_db.Test("Foo")
         test2 = self.test2 = ts_db.Test("SingleSource/Foo/Bar/baz")
         test3 = self.test3 = ts_db.Test("SingleSource/UnitTests/Bar/baz")
         test4 = self.test4 = ts_db.Test("MultiSource/Benchmarks/Ptrdist/ks/ks")
 
         self.field_change1 = ts_db.FieldChange(order1234,
                                                order1236,
                                                machine,
                                                test,
-                                               a_field)
+                                               a_field.id)
         self.field_change2 = ts_db.FieldChange(order1234,
                                                order1236,
                                                machine,
                                                test2,
-                                               a_field)
+                                               a_field.id)
         self.field_change3 = ts_db.FieldChange(order1234,
                                                order1236,
                                                machine,
                                                test3,
-                                               a_field)
+                                               a_field.id)
         self.field_change4 = ts_db.FieldChange(order1234,
                                                order1236,
                                                machine,
                                                test4,
-                                               a_field)
-        ts_db.add(self.field_change1)
-        ts_db.add(self.field_change2)
-        ts_db.add(self.field_change3)
-        ts_db.add(self.field_change4)
+                                               a_field.id)
+        session.add(self.field_change1)
+        session.add(self.field_change2)
+        session.add(self.field_change3)
+        session.add(self.field_change4)
 
-        ts_db.commit()
+        session.commit()
 
     def test_blacklist(self):
         """Check we filter by benchmark name correctly."""
+        session = self.session
         ts = self.ts_db
         fc1 = self.field_change1
         fc2 = self.field_change2
         fc3 = self.field_change3
         fc4 = self.field_change4
 
-        valid = blacklist.filter_by_benchmark_name(ts, fc1)
+        valid = blacklist.filter_by_benchmark_name(session, ts, fc1)
         self.assertTrue(valid, "Expect this to not be filtered.")
-        valid = blacklist.filter_by_benchmark_name(ts, fc2)
+        valid = blacklist.filter_by_benchmark_name(session, ts, fc2)
         self.assertTrue(valid, "Expect this to not be filtered.")
-        bad = blacklist.filter_by_benchmark_name(ts, fc3)
+        bad = blacklist.filter_by_benchmark_name(session, ts, fc3)
         self.assertFalse(bad, "Expect this to be filtered by regex.")
-        bad = blacklist.filter_by_benchmark_name(ts, fc4)
+        bad = blacklist.filter_by_benchmark_name(session, ts, fc4)
         self.assertFalse(bad, "Expect this to be filtered by blacklist.")
 
 if __name__ == '__main__':
     unittest.main(argv=[sys.argv[0], ])

lnt/trunk/tests/server/db/search.py

def setUp(self):
                          ('machine3', '6512'),
                          ('machine3', '7623'),
                          ('machine3', '11324'),
                          ('supermachine', '1324'),
                          ('supermachine', '7623')]
         
         # Get the database.
         self.db = config.get_database('default')
+        self.session = self.db.make_session()
         # Load the database.
         for r in imported_runs:
             with tempfile.NamedTemporaryFile() as f:
                 data = open(os.path.join(base_path, 'Inputs/report.json.in')) \
                     .read() \
                     .replace('@@MACHINE@@', r[0]) \
                     .replace('@@ORDER@@', r[1])
                 open(f.name, 'w').write(data)
     
                 result = lnt.util.ImportData.import_and_report(
-                    None, 'default', self.db, f.name,
+                    None, 'default', self.db, self.session, f.name,
                     format='<auto>', ts_name='nts', show_sample_count=False,
                     disable_email=True, disable_report=True,
                     updateMachine=False, mergeRun='reject')
 
                 assert result.get('success', False)
 
     def _mangleResults(self, rs):
         return [(r.machine.name, str(r.order.llvm_project_revision))
                 for r in rs]
         
     def test_specific(self):
+        session = self.session
         ts = self.db.testsuite.get('nts')
 
-        results = self._mangleResults(search(ts, 'machine1 #5625'))
+        results = self._mangleResults(search(session, ts, 'machine1 #5625'))
         self.assertEqual(results, [
             ('machine1', '5625')
         ])
 
-        results = self._mangleResults(search(ts, 'machine1 #5624'))
+        results = self._mangleResults(search(session, ts, 'machine1 #5624'))
         self.assertEqual(results, [
             ('machine1', '5624')
         ])
 
     def test_multiple_orders(self):
+        session = self.session
         ts = self.db.testsuite.get('nts')
 
-        results = self._mangleResults(search(ts, 'machine1 #56'))
+        results = self._mangleResults(search(session, ts, 'machine1 #56'))
         self.assertEqual(results, [
             ('machine1', '5625'), ('machine1', '5624')
         ])
 
     def test_nohash(self):
+        session = self.session
         ts = self.db.testsuite.get('nts')
 
-        results = self._mangleResults(search(ts, 'machine1 r56'))
+        results = self._mangleResults(search(session, ts, 'machine1 r56'))
         self.assertEqual(results, [
             ('machine1', '5625'), ('machine1', '5624')
         ])
 
-        results = self._mangleResults(search(ts, 'machine1 56'))
+        results = self._mangleResults(search(session, ts, 'machine1 56'))
         self.assertEqual(results, [
             ('machine1', '5625'), ('machine1', '5624')
         ])
 
     def test_default_order(self):
+        session = self.session
         ts = self.db.testsuite.get('nts')
 
-        results = self._mangleResults(search(ts, 'machi ne3'))
+        results = self._mangleResults(search(session, ts, 'machi ne3'))
         self.assertEqual(results, [
             ('machine3', '11324'),
             ('machine3', '7623'),
             ('machine3', '6512'),
             ('machine3', '65')
         ])
         
     def test_default_machine(self):
+        session = self.session
         ts = self.db.testsuite.get('nts')
 
-        results = self._mangleResults(search(ts, '65', default_machine=3))
+        results = self._mangleResults(search(session, ts, '65',
+                                             default_machine=3))
         self.assertEqual(results, [
             ('machine2', '6512')
         ])
 
 if __name__ == '__main__':
     if len(sys.argv) > 1:
         base_path = sys.argv[1]
     unittest.main(argv=[sys.argv[0], ])

lnt/trunk/tests/server/ui/change_processing.py

 from lnt.server.db.fieldchange import delete_fieldchange
 from lnt.server.db.fieldchange import is_overlaping, identify_related_changes
 from lnt.server.db.regression import rebuild_title, RegressionState
 from lnt.server.db.rules import rule_update_fixed_regressions
 
 logging.basicConfig(level=logging.DEBUG)
 
 
-def _mkorder(ts, rev):
+def _mkorder(session, ts, rev):
     order = ts.Order()
     order.llvm_project_revision = rev
-    ts.add(order)
+    session.add(order)
     return order
 
 
 class ChangeProcessingTests(unittest.TestCase):
     """Test fieldchange and regression building."""
 
     def setUp(self):
         """Bind to the LNT test instance."""
 
         self.db = v4db.V4DB("sqlite:///:memory:", Config.dummy_instance())
+        session = self.session = self.db.make_session()
 
         # Get the test suite wrapper.
         ts_db = self.ts_db = self.db.testsuite['nts']
 
-        order1234 = self.order1234 = _mkorder(ts_db, "1234")
-        order1235 = self.order1235 = _mkorder(ts_db, "1235")
-        order1236 = self.order1236 = _mkorder(ts_db, "1236")
-        order1237 = self.order1237 = _mkorder(ts_db, "1237")
-        order1238 = self.order1238 = _mkorder(ts_db, "1238")
+        order1234 = self.order1234 = _mkorder(session, ts_db, "1234")
+        order1235 = self.order1235 = _mkorder(session, ts_db, "1235")
+        order1236 = self.order1236 = _mkorder(session, ts_db, "1236")
+        order1237 = self.order1237 = _mkorder(session, ts_db, "1237")
+        order1238 = self.order1238 = _mkorder(session, ts_db, "1238")
 
         start_time = end_time = datetime.datetime.utcnow()
         machine = self.machine = ts_db.Machine("test-machine")
-        ts_db.add(machine)
+        session.add(machine)
 
         test = self.test = ts_db.Test("foo")
-        ts_db.add(test)
+        session.add(test)
 
         machine2 = self.machine2 = ts_db.Machine("test-machine2")
-        ts_db.add(machine2)
+        session.add(machine2)
 
         test2 = self.test2 = ts_db.Test("bar")
-        ts_db.add(test2)
+        session.add(test2)
 
         run = self.run = ts_db.Run(machine, order1235, start_time,
                                    end_time)
-        ts_db.add(run)
+        session.add(run)
 
         run2 = self.run2 = ts_db.Run(machine2, order1235, start_time,
                                      end_time)
-        ts_db.add(run2)
+        session.add(run2)
 
         sample = ts_db.Sample(run, test, compile_time=1.0,
                               score=4.2)
-        ts_db.add(sample)
+        session.add(sample)
 
         a_field = self.a_field = list(sample.get_primary_fields())[0]
         self.a_field2 = list(sample.get_primary_fields())[1]
 
         field_change = self.field_change = ts_db.FieldChange(order1234,
                                                              order1236,
                                                              machine,
                                                              test,
-                                                             a_field)
+                                                             a_field.id)
         field_change.run = run
-        ts_db.add(field_change)
+        session.add(field_change)
 
         fc_mach2 = ts_db.FieldChange(order1234,
                                      order1236,
                                      machine2,
                                      test,
-                                     a_field)
+                                     a_field.id)
         fc_mach2.run = run2
-        ts_db.add(fc_mach2)
+        session.add(fc_mach2)
 
         field_change2 = self.field_change2 = ts_db.FieldChange(order1235, order1236, machine,
                                                                test,
-                                                               a_field)
+                                                               a_field.id)
 
         field_change2.run = run
-        ts_db.add(field_change2)
+        session.add(field_change2)
 
         field_change3 = self.field_change3 = ts_db.FieldChange(order1237, order1238, machine,
                                                                test,
-                                                               a_field)
-        ts_db.add(field_change3)
+                                                               a_field.id)
+        session.add(field_change3)
 
         regression = self.regression = ts_db.Regression("Regression of 1 benchmarks:", "PR1234",
                                                         RegressionState.DETECTED)
-        ts_db.add(self.regression)
+        session.add(self.regression)
 
         self.regression_indicator1 = ts_db.RegressionIndicator(regression,
                                                                field_change)
         self.regression_indicator2 = ts_db.RegressionIndicator(regression,
                                                                field_change2)
 
-        ts_db.add(self.regression_indicator1)
-        ts_db.add(self.regression_indicator2)
+        session.add(self.regression_indicator1)
+        session.add(self.regression_indicator2)
 
         # All the regressions we detected.
         self.regressions = [regression]
-        ts_db.commit()
+        session.commit()
 
     def tearDown(self):
-        self.db.close_all_engines()
+        self.db.close()
 
     def test_startup(self):
         pass
 
     def test_change_grouping_criteria(self):
+        session = self.session
         ts_db = self.ts_db
 
         # Check simple overlap checks work.
         self.assertTrue(is_overlaping(self.field_change, self.field_change2),
                         "Should be overlapping")
         self.assertFalse(is_overlaping(self.field_change, self.field_change3),
                          "Should not be overlapping")
 
         # Check non-overlapping changes are always False.
-        ret, reg = identify_related_changes(ts_db, self.field_change3)
+        ret, reg = identify_related_changes(session, ts_db, self.field_change3)
 
         self.assertFalse(ret, "Ranges don't overlap, should not match")
         self.regressions.append(reg)
         # Check a regression matches if all fields match.
-        ret, _ = identify_related_changes(ts_db, self.field_change)
+        ret, _ = identify_related_changes(session, ts_db, self.field_change)
         self.assertTrue(ret, "Should Match.")
 
         field_change7 = ts_db.FieldChange(self.order1234,
                                           self.order1235,
                                           self.machine2,
                                           self.test2,
-                                          self.a_field)
-        ts_db.add(field_change7)
-        ret, reg = identify_related_changes(ts_db, field_change7)
+                                          self.a_field.id)
+        session.add(field_change7)
+        ret, reg = identify_related_changes(session, ts_db, field_change7)
         self.assertNotEquals(self.regression, reg)
         self.assertFalse(ret, "No match with different machine and tests.")
         self.regressions.append(reg)
         field_change4 = ts_db.FieldChange(self.order1234,
                                           self.order1235,
                                           self.machine2,
                                           self.test,
-                                          self.a_field)
+                                          self.a_field.id)
 
         # Check a regression matches if all fields match.
-        ret, _ = identify_related_changes(ts_db, field_change4)
+        ret, _ = identify_related_changes(session, ts_db, field_change4)
         self.assertTrue(ret, "Should Match with different machine.")
 
         field_change5 = ts_db.FieldChange(self.order1234,
                                           self.order1235,
                                           self.machine,
                                           self.test2,
-                                          self.a_field)
+                                          self.a_field.id)
 
         # Check a regression matches if all fields match.
-        ret, _ = identify_related_changes(ts_db, field_change5)
+        ret, _ = identify_related_changes(session, ts_db, field_change5)
         self.assertTrue(ret, "Should Match with different tests.")
         field_change6 = ts_db.FieldChange(self.order1234,
                                           self.order1235,
                                           self.machine,
                                           self.test,
-                                          self.a_field2)
+                                          self.a_field2.id)
 
         # Check a regression matches if all fields match.
-        ret, _ = identify_related_changes(ts_db, field_change6)
+        ret, _ = identify_related_changes(session, ts_db, field_change6)
         self.assertTrue(ret, "Should Match with different fields.")
 
-        ts_db.commit()
+        session.commit()
 
-        r2 = rebuild_title(ts_db, self.regression)
+        r2 = rebuild_title(session, ts_db, self.regression)
         expected_title = "Regression of 6 benchmarks: foo, bar"
         self.assertEquals(r2.title, expected_title)
 
     def test_regression_evolution(self):
+        session = self.session
         ts_db = self.ts_db
-        rule_update_fixed_regressions.regression_evolution(ts_db, self.regressions)
+        rule_update_fixed_regressions.regression_evolution(
+            session, ts_db, self.regressions)
 
     def test_fc_deletion(self):
-        delete_fieldchange(self.ts_db, self.field_change)
-        delete_fieldchange(self.ts_db, self.field_change2)
-        delete_fieldchange(self.ts_db, self.field_change3)
+        session = self.session
+        ts_db = self.ts_db
+        delete_fieldchange(session, ts_db, self.field_change)
+        delete_fieldchange(session, ts_db, self.field_change2)
+        delete_fieldchange(session, ts_db, self.field_change3)
 
     def test_run_deletion(self):
         """Do the FC and RIs get cleaned up when runs are deleted?"""
+        session = self.session
         ts_db = self.ts_db
-        run_idsq = ts_db.query(ts_db.Run.id).all()
-        fc_ids = ts_db.query(ts_db.FieldChange.id).all()
-        ri_ids = ts_db.query(ts_db.RegressionIndicator.id).all()
+        run_idsq = session.query(ts_db.Run.id).all()
+        fc_ids = session.query(ts_db.FieldChange.id).all()
+        ri_ids = session.query(ts_db.RegressionIndicator.id).all()
 
         run_ids = [row[0] for row in run_idsq]
-        runs = ts_db.query(ts_db.Run).filter(ts_db.Run.id.in_(run_ids)).all()
+        runs = session.query(ts_db.Run).filter(ts_db.Run.id.in_(run_ids)).all()
         for run in runs:
-            ts_db.delete(run)
+            session.delete(run)
 
-        run_ids_new = ts_db.query(ts_db.Run.id).all()
-        fc_ids_new = ts_db.query(ts_db.FieldChange.id).all()
-        ri_ids_new = ts_db.query(ts_db.RegressionIndicator.id).all()
+        run_ids_new = session.query(ts_db.Run.id).all()
+        fc_ids_new = session.query(ts_db.FieldChange.id).all()
+        ri_ids_new = session.query(ts_db.RegressionIndicator.id).all()
         # Make sure there was some runs.
         self.assertNotEqual(len(run_idsq), 0)
         self.assertNotEqual(len(fc_ids), 0)
         self.assertNotEqual(len(ri_ids), 0)
 
         # Now make sure there were all deleted.
         self.assertEqual(len(run_ids_new), 0)
 
         # Not all the FCs are covered by the runs.
         self.assertEqual(len(fc_ids_new), 1)
 
         self.assertEqual(len(ri_ids_new), 0)
 
 
 if __name__ == '__main__':
     unittest.main(argv=[sys.argv[0], ])