Add lnt commandline REST client

[MatzeB]

Summary

Note: I am mainly putting the review here in case we want to bikeshed the names of the commands or the names/style of configuration. I will work on the unittests while this review is going on.

This adds the following commands to lnt, so users can perform some data cleanup on LNT servers through the REST api without having ssh+database access on the server. (REST authentication works through an API token, that server administrators can optionally set).

$ > bin/lnt admin --help
...
Commands:
  create-config
  get-machine
  list-machines
  rename-machine
  rm-machine
  rm-run

The commands attempt to load lntadmin.yaml from the current directory, so you can prepare this configuration with all your server information, authentication and api tokens and don't need to retype it with every command.

[cmatthews]

This is a wonderful idea!

[kristof.beyls]

In D35501#811783, @cmatthews wrote:

This is a wonderful idea!

Agreed!
I'm wondering if it'd be useful to also write a bit of documentation about this? For example, being a very inexperienced server admin, I'm not sure where I'd start looking to set up authentication tokens. Maybe a pointer in a section documenting this could be useful?

Thanks!

[MatzeB]

• Add unittests
• Add documentation

[MatzeB]

Reuploading with some updates:

• Rebased to ToT
• get-run, post-run support an arbitrary number of arguments now to download/post multiple
• Added new merge-machine-into mode
• Show status updates during rm-machine

[MatzeB]

ping

[grosser]

I really like the idea and would like to see this going in. I am fine with the name of the commands and believe the overall patch looks good, but did not check each line.

[grosser]

I mark this as LGTM. Depending on your confidence you may wait for other reviews, but from my side this is fine.

lnt/trunk/docs/api.rst

 +---------------------------+------------------------------------------------------------------------------------------+
 | /samples?runid=1&runid=2  | Retrieve all the sample data for a list of run ids.  Run IDs should be pass as args.     |
 |                           | Will return sample data in the samples section, as a list of dicts, with a key for       |
 |                           | each metric type. Empty samples are not sent.                                            |
 +---------------------------+------------------------------------------------------------------------------------------+
 | /samples/`id`             | Get all non-empty sample info for Sample `id`.                                           |
 +---------------------------+------------------------------------------------------------------------------------------+
 
+.. _auth_tokens:
+
 Write Operations
 ----------------
 
 The machines, orders and runs endpoints also support the DELETE http method.  The user must include a http header called
 "AuthToken" which has the API auth token set in the LNT instance configuration.
 
 The API Auth token can be set by adding `api_auth_token` to the instances lnt.cfg config file::
 
     # API Auth Token
     api_auth_token = "SomeSecret"
 
 Example::
 
     curl --request DELETE --header "AuthToken: SomeSecret" http://localhost:8000/api/db_default/v4/nts/runs/1

lnt/trunk/docs/tools.rst

-----------------
   ``lnt showtests``
     List available built-in tests. See the :ref:`tests` documentation for more
     details on this tool.
 
   ``lnt runtest [<run options>] <test name> ... test arguments ...``
     Run a built-in test. See the :ref:`tests` documentation for more
     details on this tool.
 
+
+Server Administration
+~~~~~~~~~~~~~~~~~~~~~
+
+The ``lnt admin`` tool allows connecting to a server through LNTs REST API and
+perform data queries and modifications. Data modification is only possible with
+an authentication mechanism specified in the `lntadmin.cfg` file.  See
+:ref:`auth_tokens` for details.
+
+  ``lnt admin create-config``
+  Create a `lntadmin.cfg` configuration file in the current directory. The file
+  describes the URL, authentication settings and default database and
+  test-suite settings for an LNT server. The other admin commands read this
+  file if it exists.
+
+  ``lnt admin list-machines``
+  List machines and their id numbers
+
+  ``lnt admin get-machine <machine>``
+  Download machine information and run list for a specific machine.
+
+  ``lnt admin rm-machine <machine>``
+  Removes the specified machine and related runs and samples.
+
+  ``lnt admin rename-machine <machine> <new-name>``
+  Renames the specified machine.
+
+  ``lnt admin merge-machine-into <machine> <merge-into-machine>``
+  Move all runs from one machine to another machine and delete the machine.
+
+  ``lnt admin list-runs <machine>``
+  List all runs for the specified machine.
+
+  ``lnt admin get-run <run>+``
+  Download the specified runs.
+
+  ``lnt admin post-run <filename>+``
+  Post the specified report files as a new runs to the server.
+
+  ``lnt admin rm-run <run>+``
+  Remove the specified runs and related samples.
+
+
 Server-Side Tools
 -----------------
 
 The following tools are used to interact with an LNT server:
 
   ``lnt create <path>``
     Creates a new LNT server instance. This command has a number of parameters
     to tweak the generated server, but they can all be modified after the fact

lnt/trunk/lnt/lnttool/admin.py

+#!/usr/bin/env python
+import click
+
+_config_filename = 'lntadmin.yaml'
+
+
+def _load_dependencies():
+    global yaml, sys, requests, json, os, httplib
+    import yaml
+    import sys
+    import requests
+    import json
+    import os
+    import httplib
+
+
+def _error(msg):
+    sys.stderr.write('%s\n' % msg)
+
+
+def _fatal(msg):
+    _error(msg)
+    sys.exit(1)
+
+
+def _check_normalize_config(config, need_auth_token):
+    '''Verify whether config is correct and complete. Also normalizes the
+    server URL if necessary.'''
+    lnt_url = config.get('lnt_url', None)
+    if lnt_url is None:
+        _fatal('No lnt_url specified in config or commandline\n'
+               'Tip: Use `create-config` for an example configuration')
+    if lnt_url.endswith('/'):
+        lnt_url = lnt_url[:-1]
+        config['lnt_url'] = lnt_url
+    database = config.get('database', None)
+    if database is None:
+        _fatal('No database specified in config or commandline')
+    testsuite = config.get('testsuite', None)
+    if testsuite is None:
+        config['testsuite'] = 'nts'
+
+    session = requests.Session()
+    user = config.get('user', None)
+    password = config.get('password', None)
+    if user is not None and password is not None:
+        session.auth = (user, password)
+
+    auth_token = config.get('auth_token', None)
+    if need_auth_token and auth_token is None:
+        _fatal('No auth_token specified in config')
+    else:
+        session.headers.update({'AuthToken': auth_token})
+    config['session'] = session
+
+
+def _make_config(kwargs, need_auth_token=False):
+    '''Load configuration from yaml file, merges it with the commandline
+    options and verifies the resulting configuration.'''
+    verbose = kwargs.get('verbose', False)
+    # Load config file
+    config = {}
+    try:
+        config = yaml.load(open(_config_filename))
+    except IOError:
+        if verbose:
+            _error("Could not load configuration file '%s'\n" %
+                   _config_filename)
+    for key, value in kwargs.items():
+        if value is None:
+            continue
+        config[key] = value
+    _check_normalize_config(config, need_auth_token=need_auth_token)
+    return config
+
+
+def _check_response(response):
+    '''Check given response. If it is not a 200 response print an error message
+    and quit.'''
+    status_code = response.status_code
+    if 200 <= status_code and status_code < 400:
+        return
+
+    sys.stderr.write("%d: %s\n" %
+                     (status_code, httplib.responses.get(status_code, '')))
+    sys.stderr.write("\n%s\n" % response.text)
+    sys.exit(1)
+
+
+def _print_machine_info(machine, indent=''):
+    for key, value in machine.items():
+        sys.stdout.write('%s%s: %s\n' % (indent, key, value))
+
+
+def _print_run_info(run, indent=''):
+    for key, value in run.items():
+        sys.stdout.write('%s%s: %s\n' % (indent, key, value))
+
+
+def _common_options(func):
+    func = click.option("--lnt-url", help="URL of LNT server")(func)
+    func = click.option("--database", help="database to use")(func)
+    func = click.option("--testsuite", help="testsuite to use")(func)
+    func = click.option("--verbose", "-v", is_flag=True,
+                        help="verbose output")(func)
+    return func
+
+
[e-mail]("list-machines")
+@_common_options
+def action_list_machines(**kwargs):
+    """List machines and their id numbers."""
+    config = _make_config(kwargs)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines'
+           .format(**config))
+    session = config['session']
+    response = session.get(url)
+    _check_response(response)
+    data = json.loads(response.text)
+    for machine in data['machines']:
+        id = machine.get('id', None)
+        name = machine.get('name', None)
+        sys.stdout.write("%s:%s\n" % (name, id))
+        if config['verbose']:
+            _print_machine_info(machine, indent='\t')
+
+
[e-mail]("get-machine")
[e-mail]("machine")
+@_common_options
+def action_get_machine(**kwargs):
+    """Download machine information and run list."""
+    config = _make_config(kwargs)
+
+    filename = 'machine_%s.json' % config['machine']
+    if os.path.exists(filename):
+        _fatal("'%s' already exists" % filename)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines/{machine}'
+           .format(**config))
+    session = config['session']
+    response = session.get(url)
+    _check_response(response)
+    data = json.loads(response.text)
+    assert len(data['machines']) == 1
+    machine = data['machines'][0]
+
+    result = {
+        'machine': machine
+    }
+    runs = data.get('runs', None)
+    if runs is not None:
+        result['runs'] = runs
+    with open(filename, "w") as destfile:
+        json.dump(result, destfile, indent=2)
+    sys.stdout.write("%s created.\n" % filename)
+
+
[e-mail]("rm-machine")
[e-mail]("machine")
+@_common_options
+def action_rm_machine(**kwargs):
+    """Remove machine and related data."""
+    config = _make_config(kwargs, need_auth_token=True)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines/{machine}'
+           .format(**config))
+    session = config['session']
+    response = session.delete(url, stream=True)
+    _check_response(response)
+    for line in response.iter_lines():
+        sys.stdout.write(line + '\n')
+        sys.stdout.flush()
+
+
[e-mail]("rename-machine")
[e-mail]("machine")
[e-mail]("new-name")
+@_common_options
+def action_rename_machine(**kwargs):
+    """Rename machine."""
+    config = _make_config(kwargs, need_auth_token=True)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines/{machine}'
+           .format(**config))
+    session = config['session']
+    response = session.post(url, data=(('action', 'rename'),
+                                       ('name', config['new_name'])))
+    _check_response(response)
+
+
[e-mail]("merge-machine-into")
[e-mail]("machine")
[e-mail]("into")
+@_common_options
+def action_merge_machine_into(**kwargs):
+    """Merge machine into another machine."""
+    config = _make_config(kwargs, need_auth_token=True)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines/{machine}'
+           .format(**config))
+    session = config['session']
+    response = session.post(url, data=(('action', 'merge'),
+                                       ('into', config['into'])))
+    _check_response(response)
+
+
[e-mail]("list-runs")
[e-mail]("machine")
+@_common_options
+def action_list_runs(**kwargs):
+    """List runs of a machine."""
+    config = _make_config(kwargs)
+
+    url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/machines/{machine}'
+           .format(**config))
+    session = config['session']
+    response = session.get(url)
+    _check_response(response)
+    data = json.loads(response.text)
+    runs = data['runs']
+    if config['verbose']:
+        sys.stdout.write("order run-id\n")
+        sys.stdout.write("------------\n")
+    for run in runs:
+        order_by = [x.strip() for x in run['order_by'].split(',')]
+        orders = []
+        for field in order_by:
+            orders.append("%s=%s" % (field, run[field]))
+        sys.stdout.write("%s %s\n" % (";".join(orders), run['id']))
+        if config['verbose']:
+            _print_run_info(run, indent='\t')
+
+
[e-mail]("get-run")
[e-mail]("runs", nargs=-1, required=True)
+@_common_options
+def action_get_run(**kwargs):
+    """Download runs and save as report files."""
+    config = _make_config(kwargs)
+
+    runs = config['runs']
+    for run in runs:
+        filename = 'run_%s.json' % run
+        if os.path.exists(filename):
+            _fatal("'%s' already exists" % filename)
+
+    session = config['session']
+    for run in runs:
+        url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/runs/{run}'
+               .format(run=run, **config))
+        response = session.get(url)
+        _check_response(response)
+
+        data = json.loads(response.text)
+        filename = 'run_%s.json' % run
+        with open(filename, "w") as destfile:
+            json.dump(data, destfile, indent=2)
+        sys.stdout.write("%s created.\n" % filename)
+
+
[e-mail]("rm-run")
[e-mail]("runs", nargs=-1, required=True)
+@_common_options
+def action_rm_run(**kwargs):
+    """Remove runs and related data."""
+    config = _make_config(kwargs, need_auth_token=True)
+
+    session = config['session']
+    runs = config['runs']
+    for run in runs:
+        url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/runs/{run}'
+               .format(run=run, **config))
+        response = session.delete(url)
+        _check_response(response)
+
+
[e-mail]("post-run")
[e-mail]("datafiles", nargs=-1, type=click.Path(exists=True),
+                required=True)
+@_common_options
[e-mail]("--update-machine", is_flag=True, help="Update machine fields")
[e-mail]("--merge", default="replace", show_default=True,
+              type=click.Choice(['reject', 'replace', 'merge']),
+              help="Merge strategy when run already exists")
+def action_post_run(**kwargs):
+    """Submit report files to server."""
+    config = _make_config(kwargs, need_auth_token=True)
+
+    session = config['session']
+    datafiles = config['datafiles']
+    for datafile in datafiles:
+        with open(datafile, "r") as datafile:
+            data = datafile.read()
+
+        url = ('{lnt_url}/api/db_{database}/v4/{testsuite}/runs'
+               .format(**config))
+        url_params = {
+            'update_machine': 1 if config['update_machine'] else 0,
+            'merge': config['merge'],
+        }
+        response = session.post(url, params=url_params, data=data,
+                                allow_redirects=False)
+        _check_response(response)
+        if response.status_code == 301:
+            sys.stdout.write(response.headers.get('Location') + '\n')
+        if config['verbose']:
+            try:
+                response_data = json.loads(response.text)
+                json.dump(response_data, sys.stderr, response_data, indent=2)
+            except:
+                sys.stderr.write(response.text)
+            sys.stderr.write('\n')
+
+
[e-mail]('create-config')
+def action_create_config():
+    """Create example configuration."""
+    if os.path.exists(_config_filename):
+        _fatal("'%s' already exists" % _config_filename)
+    with open(_config_filename, "w") as out:
+        out.write('''\
+lnt_url: "http://localhost:8000"
+database: default
+testsuite: nts
+# user: 'http_user'
+# password: 'http_password'
+# auth_token: 'secret'
+''')
+    sys.stderr.write("Created '%s'\n" % _config_filename)
+
+
+class AdminCLI(click.MultiCommand):
+    '''Admin subcommands. Put into this class so we can lazily import
+    dependencies.'''
+    _commands = [
+        action_create_config,
+        action_get_machine,
+        action_get_run,
+        action_list_machines,
+        action_list_runs,
+        action_merge_machine_into,
+        action_post_run,
+        action_rename_machine,
+        action_rm_machine,
+        action_rm_run,
+    ]
+    def list_commands(self, ctx):
+        return [command.name for command in self._commands]
+
+    def get_command(self, ctx, name):
+        _load_dependencies()
+        for command in self._commands:
+            if command.name == name:
+                return command
+        raise ValueError("Request unknown command '%s'" % name)
+
+
[e-mail]("admin", cls=AdminCLI, no_args_is_help=True)
+def group_admin():
+    """LNT server admin client."""

lnt/trunk/lnt/lnttool/main.py

 """Implement the command line 'lnt' tool."""
 from .common import init_logger
 from .convert import action_convert
 from .create import action_create
 from .import_data import action_import
 from .import_report import action_importreport
 from .updatedb import action_updatedb
 from .viewcomparison import action_view_comparison
+from .admin import group_admin
 from lnt.util import logger
 import click
 import logging
 import sys
 
 
 @click.command("runserver", short_help="start a new development server")
 @click.argument("instance_path", type=click.UNPROCESSED)
class RunTestCLI(click.MultiCommand):
 
     def get_command(self, ctx, name):
         import lnt.tests
         return lnt.tests.get_module(name).cli_action
 
 
 @click.group("runtest", cls=RunTestCLI, context_settings=dict(
     ignore_unknown_options=True, allow_extra_args=True,))
-def action_runtest():
+def group_runtest():
     """run a builtin test application"""
     init_logger(logging.INFO)
 
 
 @click.command("showtests")
 def action_showtests():
     """show the available built-in tests"""
     import lnt.tests
@click.option('--version', is_flag=True, callback=show_version,
               expose_value=False, is_eager=True, help=show_version.__doc__)
 def cli():
     """LNT command line tool
 
 \b
 Use ``lnt <command> --help`` for more information on a specific command.
     """
 cli.add_command(action_checkformat)
-cli.add_command(action_create)
 cli.add_command(action_convert)
+cli.add_command(action_create)
 cli.add_command(action_import)
 cli.add_command(action_importreport)
 cli.add_command(action_profile)
 cli.add_command(action_runserver)
-cli.add_command(action_runtest)
 cli.add_command(action_send_daily_report)
 cli.add_command(action_send_run_comparison)
 cli.add_command(action_showtests)
 cli.add_command(action_submit)
 cli.add_command(action_update)
 cli.add_command(action_updatedb)
 cli.add_command(action_view_comparison)
+cli.add_command(group_admin)
+cli.add_command(group_runtest)
 
 
 def main():
     _version_check()
     # Change deprecated `--commit=1` and `--commit 1` options to new ones.
     for i in range(1, len(sys.argv)):
         arg = sys.argv[i]
         if arg == '--commit=1':

lnt/trunk/tests/lnttool/admin.shtest

+# RUN: rm -f %T/lntadmin.yaml
+# RUN: cd %T ; lnt admin create-config
+# RUN: FileCheck %s < %T/lntadmin.yaml --check-prefix=CREATE_CONFIG
+# CREATE_CONFIG: lnt_url: "http://localhost:8000"
+# CREATE_CONFIG-NEXT: database: default
+# CREATE_CONFIG-NEXT: testsuite: nts
+# CREATE_CONFIG-NEXT: # user: 'http_user'
+# CREATE_CONFIG-NEXT: # password: 'http_password'
+# CREATE_CONFIG-NEXT: # auth_token: 'secret'
+
+# RUN: rm -rf %t.instance
+# RUN: python %{shared_inputs}/create_temp_instance.py \
+# RUN:   %s %{shared_inputs}/SmallInstance %t.instance
+# RUN: %{shared_inputs}/server_wrapper.sh %t.instance 9092 /bin/sh %s %T %{shared_inputs}
+
+DIR="$1"
+SHARED_INPUTS="$2"
+cd "$DIR"
+cat > lntadmin.yaml << '__EOF__'
+lnt_url: "http://localhost:9092"
+database: default
+testsuite: nts
+auth_token: test_token
+__EOF__
+
+lnt admin post-run "${SHARED_INPUTS}/sample-a-small.plist" > post_run.stdout
+# RUN: FileCheck %s --check-prefix=POST_RN < %T/post_run.stdout
+# POST_RN: http://localhost:9092/api/db_default/v4/nts/runs/3
+
+rm -rf run_3.json
+lnt admin get-run 3
+# RUN: FileCheck %s --check-prefix=GET_RN < %T/run_3.json
+# GET_RN: {
+# GET_RN:   "machine": {
+# ...
+# GET_RN:   },
+# GET_RN:   "tests": [
+#...
+# GET_RN:   ],
+# GET_RN:   "run": {
+# GET_RN:     "start_time": "2009-11-17T02:12:25"
+# GET_RN:     "end_time": "2009-11-17T03:44:48"
+# GET_RN:     "id": 3
+#...
+# GET_RN:   },
+# GET_RN:   "generated_by": "LNT Server v0.4.2dev"
+# GET_RN: }
+
+lnt admin list-machines > list_machines.stdout
+# RUN: FileCheck %s --check-prefix=LIST_MACHINES < %T/list_machines.stdout
+# LIST_MACHINES: localhost__clang_DEV__x86_64:1
+# LIST_MACHINES-NEXT: LNT SAMPLE MACHINE:2
+
+lnt admin list-runs 1 > list_runs.stdout
+# RUN: FileCheck %s --check-prefix=LIST_RUNS < %T/list_runs.stdout
+# LIST_RUNS: llvm_project_revision=154331 1
+# LIST_RUNS: llvm_project_revision=152289 2
+
+lnt admin rm-machine 1
+
+lnt admin list-machines > list_machines2.stdout
+# RUN: FileCheck %s --check-prefix=LIST_MACHINES2 < %T/list_machines2.stdout
+# LIST_MACHINES2-NOT: localhost__clang_DEV__x86_64:1
+# LIST_MACHINES2: LNT SAMPLE MACHINE:2
+
+lnt admin rename-machine 2 hal9000
+
+lnt admin list-machines > list_machines3.stdout
+# RUN: FileCheck %s --check-prefix=LIST_MACHINES3 < %T/list_machines3.stdout
+# LIST_MACHINES3: hal9000:2