Local path implementation of swig-bot.py

[zturner]

Summary

This version supports local generation only.  It's intentionally
stupid, and does not support any kind of dependency checking.
If you run the script, it's going to call SWIG.  Period.  While this is
a slow process, we are going to combine the use of the swig bot
with checked in static bindings, meaning that it won't be terribly
important to only regenerate the bindings when the input files
have actually changed.  I'm open to the possibility of adding
dep checking, but I want to make sure there's a clear use case
for it first.

A side benefit of this is that the implementation is drastically
simpler.

Since depending on how you call the script, swig can be run on
either the client or the server, it means both the client and server
need to be able to reuse the code that physically exec's swig.  This
logic is therefore contained in the file `local.py`, which is used by
by both `client.py` and (later) `server.py`.

This is all experimental at the moment, but it duplicates a lot
of the logic currently found in prepare_bindings.py.  There was
not a good way to reuse some of the logic without invasive changes
on that script, and since this script is still experimental, it
makes sense to just copy them over, and if / when this becomes
more mature, we can get rid of the other ones.

One thing that differs fundamentally from previous implementations
is that there is no need for a custom `_python.py` version of the script.
AFAICT you can generate multiple languages with a single swig invocation
so there is no reason to invoke swig multiple times, which is what the old
system would do.  So this new method just contains logic to build up a
single command line which can generate everything at once.

The modify step is not included here, because again the idea here is that
this will be run on a remote server.  So you will eventually do the modify
locally after receiving the result from the server.

[zturner]

Fixed a few minor issues

[zturner]

Note: I was wrong in my initial post, you can't actually generate multiple language bindings in a single invocation. So the second patchset loops over the language list and invokes swig multiple times.

Note that we have some occurrences of %pythoncode in the swig files, so languages other than python don't work anyway atm. But I don't want to make it harder than it needs to be for someone in the future to make this work.

[tfiala]

Nice, I'm looking forward to looking at this in depth.

I'm out (not around a computer) until after Thanksgiving. I may be a little laggy on feedback.

[tfiala]

Basic idea looks fine. (I saw you put this in not long after, but I'm finally getting to look at the actual review).

I'm not sure if you're intending to cover all the use cases of the generation, but I think the prepare_bindings.py will also have swig generate the compiler dependency .d file that some of the compilers will use.

[tfiala]

Can this be closed out?

scripts/swig_bot.py

+#!/usr/bin/env python
+
+# Python modules
+import sys
+
+# LLDB modules
+import use_lldb_suite
+
+if __name__ == "__main__":
+    from swig_bot_lib import client
+    client.run(sys.argv[1:])

scripts/swig_bot_lib/client.py

+#!/usr/bin/env python
+
+# Future imports
+from __future__ import absolute_import
+from __future__ import print_function
+
+# Python modules
+import argparse
+import logging
+import os
+import sys
+
+# LLDB modules
+import use_lldb_suite
+from lldbsuite.support import fs
+
+def process_args(args):
+    """Returns options processed from the provided command line.
+
+    @param args the command line to process.
+    """
+
+    class FindLocalSwigAction(argparse.Action):
+        def __init__(self, option_strings, dest, **kwargs):
+            super(FindLocalSwigAction, self).__init__(option_strings, dest, nargs='?', **kwargs)
+        def __call__(self, parser, namespace, values, option_string=None):
+            swig_exe = None
+            if values is None:
+                swig_exe = fs.find_executable('swig')
+            else:
+                swig_exe = values
+            setattr(namespace, self.dest, os.path.normpath(swig_exe))
+
+    # Setup the parser arguments that are accepted.
+    parser = argparse.ArgumentParser(
+        description='Generate SWIG bindings.')
+
+    # Arguments to control logging verbosity.
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        default=False,
+        help="Increase logging verbosity level.")
+
+    parser.add_argument(
+        "--local",
+        action=FindLocalSwigAction,
+        dest="swig_executable",
+        help=(
+            "Run the copy of swig at the specified location, or search PATH"
+            "if the location is omitted"))
+
+    parser.add_argument(
+        "--remote",
+        action="store",
+        help=(
+            "Use the given connection string to connect to a remote "
+            "generation service"))
+
+    parser.add_argument(
+        "--src-root",
+        required=True,
+        help="The root folder of the LLDB source tree.")
+
+    parser.add_argument(
+        "--target-dir",
+        default=os.getcwd(),
+        help=(
+            "Specifies the build dir where the language binding "
+            "should be placed"))
+
+    parser.add_argument(
+        "--language",
+        dest="languages",
+        action="append",
+        help="Specifies the language to generate bindings for")
+
+    # Process args.
+    options = parser.parse_args(args)
+
+    if options.languages is None:
+        options.languages = ['python']
+
+    if options.remote is None and options.swig_executable is None:
+        logging.error("Must specify either --local or --remote")
+        sys.exit(-3)
+
+    # Set logging level based on verbosity count.
+    if options.verbose:
+        log_level = logging.DEBUG
+    else:
+        log_level = logging.NOTSET
+    logging.basicConfig(level=log_level)
+    logging.info("logging is using level: %d", log_level)
+
+    return options
+
+
+def run(args):
+    options = process_args(args)
+
+    if options.remote is None:
+        if not os.path.isfile(options.swig_executable):
+            logging.error("Swig executable '%s' does not exist." % options.swig_executable)
+        from . import local
+        local.generate(options)
+    else:
+        logging.error("Remote path is not yet implemented!")

scripts/swig_bot_lib/local.py

+#!/usr/bin/env python
+
+# Future imports
+from __future__ import absolute_import
+from __future__ import print_function
+
+# Python modules
+import argparse
+import imp
+import logging
+import os
+import subprocess
+import sys
+
+# LLDB modules
+import use_lldb_suite
+
+def generate(options):
+    include_folder = os.path.join(options.src_root, "include")
+    in_file = os.path.join(options.src_root, "scripts", "lldb.swig")
+    include_folder = os.path.normcase(include_folder)
+
+    for lang in options.languages:
+        lang = lang.lower()
+        out_dir = os.path.join(options.target_dir, lang.title())
+        if not os.path.exists(out_dir):
+            os.makedirs(out_dir)
+        out_file = os.path.join(out_dir, "LLDBWrap{}".format(lang.title()))
+        swig_command = [
+            options.swig_executable,
+            "-c++",
+        ]
+        swig_command.append("-" + lang)
+        if lang == "python":
+            swig_command.append("-threads")
+
+        swig_command.extend([
+            "-I" + include_folder,
+            "-D__STDC_LIMIT_MACROS",
+            "-D__STDC_CONSTANT_MACROS",
+            "-outdir", out_dir,
+            "-o", out_file,
+            in_file
+        ])
+
+        logging.info("generating swig {} bindings into {}".format(lang, out_dir))
+        logging.debug("swig command line: {}".format(swig_command))
+        try:
+            # Execute swig
+            swig_output = subprocess.check_output(
+                swig_command, stderr=subprocess.STDOUT, universal_newlines=True)
+
+            logging.info("swig generation succeeded")
+            if swig_output is not None and len(swig_output) > 0:
+                logging.info("swig output: %s", swig_output)
+        except subprocess.CalledProcessError as e:
+            logging.error("An error occurred executing swig.  returncode={}".format(e.returncode))
+            logging.error(e.output)

scripts/swig_bot_lib/server.py

+#!/usr/bin/env python
+
+# Future imports
+from __future__ import absolute_import
+from __future__ import print_function
+