[LLVM][TableGen] Add jupyter kernel for llvm-tblgen

[DavidSpickett]

Summary

This is based on the MLIR opt kernel:
https://github.com/llvm/llvm-project/tree/main/mlir/utils/jupyter

The inent of this is to enable experimentation and the creation
of interactive tutorials for the basics of tablegen.

Noteable changes from that:

• Removed the codemirror mode settings since those won't exist for tablegen.
• Added "%args" "magic" to control arguments sent to llvm-tblgen.

(magics are directives, see
https://ipython.readthedocs.io/en/stable/interactive/magics.html)

For example the following:

%args --print-detailed-records
class Stuff {}

def water_bottle : Stuff {}

Produces:

DETAILED RECORDS for file -

-------------------- Global Variables (0) --------------------

-------------------- Classes (1) --------------------

Stuff  |<stdin>:1|
  Template args: (none)
  Superclasses: (none)
  Fields: (none)

-------------------- Records (1) --------------------

water_bottle  |<stdin>:3|
  Superclasses: Stuff
  Fields: (none)

[DavidSpickett]

First obvious thing, I just chose the closest folder to where MLIR put their kernel. It can go elsewhere of course.

[DavidSpickett]

I'd like to put a notebook next to this that walks through https://llvm.org/docs/TableGen/ProgRef.html with examples but I figure the kernel has some use on its own too.

[DavidSpickett]

@jpienaar If you have any feedback since you wrote the kernel for MLIR (and thanks for doing that!).

[DavidSpickett]

Correct module name in install command.

[DavidSpickett]

• Added doctests.
• Added example/testing notebook.
• Added markdown output for that notebook.

Probably easier to view the markdown at https://github.com/DavidSpickett/llvm-project/blob/5eb2366310b125fca846c9ce339d2d7386c2a434/llvm/utils/TableGen/jupyter/LLVM%20TableGen.md.

[jpienaar]

Nice, I've found a variant where one just uses Python magic to do most of the same. The pro is not requireing any installation of jupyter kernel, the con is a little bit less output flexibility and error reporting I felt wasn't as nice.

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

82

Can one interleave magic like this in jupyter? I thought one had cell and line magic but both required to be only at start. (Well required is strong word, expected perhaps better)

[DavidSpickett]

Nice, I've found a variant where one just uses Python magic to do most of the same.

Another notebook do you mean? Can you post a link to it?

One idea I had was to be able to run python scripts on the JSON tablegen output, so I might end up coming back around to Python.

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

82

Well you already know more than me then :) I did this based on skimming the iPython kernel's docs so I'm still learning the terminology.

So far I guess this would be called "cell magic" and you're right it doesn't make sense for that to be anywhere but the first (or first set of) lines. Since you can't apply cell magic half way through a cell.

I will make this more strict.

[DavidSpickett]

Restrict magic that effects the whole cell to only be in the first lines,
or a continuous set of magic lines starting from the first line.

Interepreted as two magic lines:

%args
%args
// Code

Only one because the run of magic was broken:

%args
// Code
%args

[DavidSpickett]

Adding Paul as the current TableGen code owner.

[awarzynski]

Thanks for doing this!

I've only had the time go over and to run "LLVM TableGen.md" - everything worked fine and the examples make sense to me. If only there was no empty space in the filename ;-) Btw, why do you need an *.md and *.ipynb file?

llvm/utils/TableGen/jupyter/README.md

4

?

32

llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py

5

I think that I am right, but I might be wrong ;-) (https://www.pristineword.com/grammar-an-l/)

[DavidSpickett]

• Address review comments.
• Explain what the jupyter console line does.

[DavidSpickett]

Btw, why do you need an *.md and *.ipynb file?

The ipynb is the notebook itself which contains the cells and copies of the last output from those cells. This is the one you can re-run and edit in jupyter.

The markdown is a static export of the resulting notebook so you can see what the kernel can do without running it. Also though Phabricator does have support for diffing ipynb, github doesn't and markdown makes it easier to tell what changed without running it locally.

[DavidSpickett]

• Note that there is a markdown file for anyone who just wants to see the results.

[awarzynski]

LGTM, thank you for doing this!

[jpienaar]

In D132378#3748229, @DavidSpickett wrote:

Nice, I've found a variant where one just uses Python magic to do most of the same.

Another notebook do you mean? Can you post a link to it?

Sorry for delay, waa OOO. I'll need to clean it up (well sanitize). I hit a restriction where i couldn't use a custom runtime and then made it as a quick workaround. I'll try and do that shortly (will try today but depends on amount of email I have to sift through :-)). But don't take that as a blocker from me here. (You could always use cell magic to refer to previous cells and so combine this kernel with that).

[6yearold]

llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json

10

Is tablgen a typo? I don't see any TableGen modes at https://codemirror.net/5/mode/

[DavidSpickett]

llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json

10

Yes it is. The binary is llvm-tblgen but the language is TableGen, it's endlessly confusing.

And yes there's no style for it in codemirror at this time. I think there is some way to add new languages but I haven't got into it myself.

[DavidSpickett]

Correct typo.

llvm/utils/TableGen/jupyter/.gitignore

+__pycache__
+*.pyc
+build/
+dist/
+MANIFEST

llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb

+# LLVM TableGen Kernel
+This notebook is running `llvm-tblgen`.
+In [1]:// This is some tablegen
+class Foo {}
+------------- Classes -----------------
class Foo {
}
------------- Defs -----------------
+Errors printed to stderr are shown.
+In [2]:This is not tablegen.
+<stdin>:1:1: error: Unexpected token at top level
This is not tablegen.
^
+Add some classes to get some output.
+In [3]:class Stuff {}
+def thing : Stuff {}
+------------- Classes -----------------
class Stuff {
}
------------- Defs -----------------
def thing {	// Stuff
}
+Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one.
+In [4]:def other_thing : Stuff {}
+<stdin>:1:19: error: Couldn't find class 'Stuff'
def other_thing : Stuff {}
                  ^
+There is a "magic" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.
+For example, here we have some code that shows a warning.
+In [5]:class Thing <int A, int B> {
+    int num = A;
+}
+<stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {
                        ^
+We can pass an argument to ignore that warning.
+In [6]:%args --no-warn-on-unused-template-args
+class Thing <int A, int B> {
+    int num = A;
+}
+------------- Classes -----------------
class Thing<int Thing:A = ?, int Thing:B = ?> {
  int num = Thing:A;
}
------------- Defs -----------------
+The last `%args` in a cell will be the arguments used.
+In [7]:%args --no-warn-on-unused-template-args
+%args
+class Thing <int A, int B> {
+    int num = A;
+}
+<stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {
                        ^

llvm/utils/TableGen/jupyter/LLVM_TableGen.md

+# LLVM TableGen Kernel
+
+This notebook is running `llvm-tblgen`.
+
+
+```tablegen
+// This is some tablegen
+class Foo {}
+```
+
+    ------------- Classes -----------------
+    class Foo {
+    }
+    ------------- Defs -----------------
+
+
+Errors printed to stderr are shown.
+
+
+```tablegen
+This is not tablegen.
+```
+
+    <stdin>:1:1: error: Unexpected token at top level
+    This is not tablegen.
+    ^
+
+
+Add some classes to get some output.
+
+
+```tablegen
+class Stuff {}
+def thing : Stuff {}
+```
+
+    ------------- Classes -----------------
+    class Stuff {
+    }
+    ------------- Defs -----------------
+    def thing { // Stuff
+    }
+
+
+Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one.
+
+
+```tablegen
+def other_thing : Stuff {}
+```
+
+    <stdin>:1:19: error: Couldn't find class 'Stuff'
+    def other_thing : Stuff {}
+                      ^
+
+
+There is a "magic" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.
+
+For example, here we have some code that shows a warning.
+
+
+```tablegen
+class Thing <int A, int B> {
+    int num = A;
+}
+```
+
+    <stdin>:1:25: warning: unused template argument: Thing:B
+    class Thing <int A, int B> {
+                            ^
+
+
+We can pass an argument to ignore that warning.
+
+
+```tablegen
+%args --no-warn-on-unused-template-args
+class Thing <int A, int B> {
+    int num = A;
+}
+```
+
+    ------------- Classes -----------------
+    class Thing<int Thing:A = ?, int Thing:B = ?> {
+      int num = Thing:A;
+    }
+    ------------- Defs -----------------
+
+
+The last `%args` in a cell will be the arguments used.
+
+
+```tablegen
+%args --no-warn-on-unused-template-args
+%args
+class Thing <int A, int B> {
+    int num = A;
+}
+```
+
+    <stdin>:1:25: warning: unused template argument: Thing:B
+    class Thing <int A, int B> {
+                            ^
+
+
+
+```tablegen
+
+```

llvm/utils/TableGen/jupyter/README.md

+A Jupyter kernel for TableGen (llvm-tblgen)
+
+To use the kernel, first install it into jupyter:
+
+    python3 -m tablegen_kernel.install
+
+Then put this folder on your PYTHONPATH so jupyter can find it:
+
+```shell
+    export PYTHONPATH=$PYTHONPATH:<path to this dir>
+```
+
+Then run one of:
+
+```shell
+    jupyter notebook
+    # Then in the notebook interface, select 'LLVM TableGen' from the 'New' menu.
+
+    # To run the example notebook in this folder.
+    jupyter notebook LLVM_Tablegen.ipynb
+
+    # To use the kernel from the command line.
+    jupyter console --kernel tablegen
+```
+
+If you just want to see the results of the notebook, you can read
+`LLVM_Tablegen.md` instead.
+
+`llvm-tblgen` is expected to be either in the `PATH` or you can set
+the environment variable `LLVM_TBLGEN_EXECUTABLE` to point to it directly.
+
+To run the kernel's doctests do:
+
+```shell
+    python3 tablegen_kernel/kernel.py
+```

llvm/utils/TableGen/jupyter/tablegen_kernel/__init__.py

+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+"""An llvm-tblgen kernel for Jupyter"""
+
+from .kernel import __version__

llvm/utils/TableGen/jupyter/tablegen_kernel/__main__.py

+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+from ipykernel.kernelapp import IPKernelApp
+from .kernel import TableGenKernel
+
+IPKernelApp.launch_instance(kernel_class=TableGenKernel)

llvm/utils/TableGen/jupyter/tablegen_kernel/assets/kernel.json

+{
+    "argv": [
+        "python3", "-m", "tablegen_kernel", "-f", "{connection_file}"
+    ],
+    "display_name": "LLVM TableGen",
+    "language": "tablegen",
+    "language_info": {
+        "name": "tablegen",
+        "codemirror_mode": "tablegen",
+        "mimetype": "text/x-tablegen",
+        "file_extension": ".td",
+        "pygments_lexer": "text"
+    }
+}

llvm/utils/TableGen/jupyter/tablegen_kernel/install.py

+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import os
+import argparse
+from jupyter_client.kernelspec import KernelSpecManager
+
+
+def install_my_kernel_spec(user=True, prefix=None):
+    """Install the kernel spec for user in given prefix."""
+    print("Installing llvm-tblgen IPython kernel spec")
+    pkgroot = os.path.dirname(__file__)
+    KernelSpecManager().install_kernel_spec(
+        os.path.join(pkgroot, "assets"), "tablegen", user=user, prefix=prefix
+    )
+
+
+def _is_root():
+    """Returns whether the current user is root."""
+    try:
+        return os.geteuid() == 0
+    except AttributeError:
+        # Return false wherever unknown.
+        return False
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(
+        description="Install KernelSpec for LLVM TableGen Kernel"
+    )
+    prefix_locations = parser.add_mutually_exclusive_group()
+
+    prefix_locations.add_argument(
+        "--user", help="Install in user home directory", action="store_true"
+    )
+    prefix_locations.add_argument(
+        "--prefix", help="Install directory prefix", default=None
+    )
+
+    args = parser.parse_args(argv)
+
+    user = args.user or not _is_root()
+    prefix = args.prefix
+
+    install_my_kernel_spec(user=user, prefix=prefix)
+
+
+if __name__ == "__main__":
+    main()

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+import os
+import shutil
+import subprocess
+import tempfile
+from ipykernel.kernelbase import Kernel
+
+__version__ = "0.0.1"
+
+
+class TableGenKernel(Kernel):
+    """Kernel using llvm-tblgen inside jupyter.
+
+    All input is treated as TableGen unless the first non whitespace character
+    is "%" in which case it is a "magic" line.
+
+    The only magic line supported is "%args". The rest of the line is the
+    arguments passed to llvm-tblgen.
+
+    This is "cell magic" meaning it applies to the whole cell. Therefore
+    it must be the first line, or part of a run of magic lines starting
+    from the first line.
+
+    ```tablgen
+    %args
+    %args --print-records --print-detailed-records
+    class Stuff {
+      string Name;
+    }
+
+    def a_thing : Stuff {}
+    ```
+
+    """
+
+    implementation = "tablegen"
+    implementation_version = __version__
+
+    language_version = __version__
+    language = "tablegen"
+    language_info = {
+        "name": "tablegen",
+        "mimetype": "text/x-tablegen",
+        "file_extension": ".td",
+        "pygments_lexer": "text",
+    }
+
+    def __init__(self, **kwargs):
+        Kernel.__init__(self, **kwargs)
+        self._executable = None
+
+    @property
+    def banner(self):
+        return "llvm-tblgen kernel %s" % __version__
+
+    @property
+    def executable(self):
+        """If this is the first run, search for llvm-tblgen.
+        Otherwise return the cached path to it."""
+        if self._executable is None:
+            path = os.environ.get("LLVM_TBLGEN_EXECUTABLE")
+            if path is not None and os.path.isfile(path) and os.access(path, os.X_OK):
+                self._executable = path
+            else:
+                path = shutil.which("llvm-tblgen")
+                if path is None:
+                    raise OSError("llvm-tblgen not found, please see README")
+                self._executable = path
+
+        return self._executable
+
+    def get_magic(self, code):
+        """Given a block of code remove the magic lines from it and return
+        a tuple of the code lines (newline joined) and a list of magic lines
+        with their leading spaces removed.
+
+        Currently we only look for "cell magic" which must be at the start of
+        the cell. Meaning the first line, or a set of lines beginning with %
+        that come before the first non-magic line.
+
+        >>> k.get_magic("")
+        ('', [])
+        >>> k.get_magic("Hello World.\\nHello again.")
+        ('Hello World.\\nHello again.', [])
+        >>> k.get_magic("   %foo a b c")
+        ('', ['%foo a b c'])
+        >>> k.get_magic("   %foo a b c\\n%foo\\nFoo")
+        ('Foo', ['%foo a b c', '%foo'])
+        >>> k.get_magic("Foo\\n%foo\\nFoo")
+        ('Foo\\n%foo\\nFoo', [])
+        >>> k.get_magic("%foo\\n   Foo\\n%foo")
+        ('   Foo\\n%foo', ['%foo'])
+        >>> k.get_magic("%foo\\n\\n%foo")
+        ('\\n%foo', ['%foo'])
+        >>> k.get_magic("%foo\\n \\n%foo")
+        (' \\n%foo', ['%foo'])
+        """
+        magic_lines = []
+        code_lines = []
+
+        lines = code.splitlines()
+        while lines:
+            line = lines.pop(0)
+            possible_magic = line.lstrip()
+            if possible_magic.startswith("%"):
+                magic_lines.append(possible_magic)
+            else:
+                code_lines = [line, *lines]
+                break
+
+        return "\n".join(code_lines), magic_lines
+
+    def do_execute(
+        self, code, silent, store_history=True, user_expressions=None, allow_stdin=False
+    ):
+        """Execute user code using llvm-tblgen binary."""
+        code, magic = self.get_magic(code)
+
+        extra_args = []
+        for m in magic:
+            if m.startswith("%args"):
+                # Last one in wins
+                extra_args = m.split()[1:]
+
+        with tempfile.TemporaryFile("w+") as f:
+            f.write(code)
+            f.seek(0)
+            got = subprocess.run(
+                [self.executable, *extra_args],
+                stdin=f,
+                stderr=subprocess.PIPE,
+                stdout=subprocess.PIPE,
+                universal_newlines=True,
+            )
+
+        if not silent:
+            if got.stderr:
+                self.send_response(
+                    self.iopub_socket, "stream", {"name": "stderr", "text": got.stderr}
+                )
+            else:
+                self.send_response(
+                    self.iopub_socket, "stream", {"name": "stdout", "text": got.stdout}
+                )
+
+        return {
+            "status": "ok",
+            "execution_count": self.execution_count,
+            "payload": [],
+            "user_expressions": {},
+        }
+
+
+if __name__ == "__main__":
+    import doctest
+
+    doctest.testmod(extraglobs={"k": TableGenKernel()})