[LLVM][TableGen] Add JSON magic directive to Jupyter kernel

[DavidSpickett]

Summary

This allows you to see the records as pretty printed JSON,
which can give you a better idea of the structure.

This is equivalent to %args --dump-json except that
the kernel will format it before returning the result.

(and if you did both, it would still work)

Without formatting, the ouput is all one line.

An example is added to the end of the notebook
and is shown in the Markdown export.

Depends on D132646

[DavidSpickett]

Rebase

[DavidSpickett]

I am going to rethink this once the basic tutorials are in place.

One idea is to use jq (https://stedolan.github.io/jq/) to give us a way to filter the JSON without making up our own language (and python as a one liner gets awkward). Also the current output is pretty lean, so it might need extra fields.

[DavidSpickett]

I am rethinking the next steps for the jupyter kernel, probably ones that don't involve showing the raw JSON.

If anyone is here looking for that option, you can do what the SQL backend turorial does instead: https://github.com/llvm/llvm-project/blob/main/llvm/utils/TableGen/jupyter/sql_query_backend.ipynb

llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb

 class Thing <int A, int B> {}
 <stdin>:1:18: warning: unused template argument: Thing:A
class Thing <int A, int B> {}
                 ^
<stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {}
                        ^
+The `%json` magic can be used to show the records as JSON. This is equivalent to `%args --dump-json` with the added benefit that we format the output for you.
+In [11]:%reset
+%json
+class Root {}
+def thing : Root {}
+{
    "!instanceof": {
        "Root": [
            "thing"
        ]
    },
    "!tablegen_json_version": 1,
    "thing": {
        "!anonymous": false,
        "!fields": [],
        "!name": "thing",
        "!superclasses": [
            "Root"
        ]
    }
}

llvm/utils/TableGen/jupyter/LLVM_TableGen.md

```
 
     <stdin>:1:18: warning: unused template argument: Thing:A
     class Thing <int A, int B> {}
                      ^
     <stdin>:1:25: warning: unused template argument: Thing:B
     class Thing <int A, int B> {}
                             ^
 
+
+The `%json` magic can be used to show the records as JSON. This is equivalent to `%args --dump-json` with the added benefit that we format the output for you.
+
+
+```tablegen
+%reset
+%json
+class Root {}
+def thing : Root {}
+```
+
+    {
+        "!instanceof": {
+            "Root": [
+                "thing"
+            ]
+        },
+        "!tablegen_json_version": 1,
+        "thing": {
+            "!anonymous": false,
+            "!fields": [],
+            "!name": "thing",
+            "!superclasses": [
+                "Root"
+            ]
+        }
+    }

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

 # Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
+import json
 import os
 import shutil
 import subprocess
 import tempfile
 from ipykernel.kernelbase import Kernel
 
 __version__ = "0.0.1"
 
 
 class TableGenKernel(Kernel):
     """Kernel using llvm-tblgen inside jupyter.
 
     All input is treated as TableGen unless the first non whitespace character
     is "%" in which case it is a "magic" line.
 
     The supported magic is:
     * %args - to set the arguments passed to llvm-tblgen.
     * %reset - to reset the cached code and magic state.
+    * %json - to emit records as JSON.
 
     These are "cell magic" meaning it applies to the whole cell. Therefore
     it must be the first line, or part of a run of magic lines starting
     from the first line.
 
     ```tablgen
     %args
     %reset
def get_magic(self, code):
                 # Key has the % removed
                 magic[magic_parts[0][1:]] = magic_parts[1:]
             else:
                 code_lines = [line, *lines]
                 break
 
         return "\n".join(code_lines), magic
 
-    def get_code_and_args(self, new_code):
-        """Get the code that do_execute should use, taking into account
-        the code from any cached cells.
-
-        Returns the code to compile and the arguments to use to do so.
+    def get_code_and_magic(self, new_code):
+        """Get the code that do_execute should use, and the magic
+        to run it with.
 
         >>> k._previous_code = ""
         >>> k._previous_magic = {}
-        >>> k.get_code_and_args("")
-        ('', [])
-        >>> k.get_code_and_args("%args 1\\nSome code")
-        ('Some code', ['1'])
-        >>> k.get_code_and_args("%args 2\\nSome more code")
-        ('Some code\\nSome more code', ['2'])
-        >>> k.get_code_and_args("%reset\\n%args 3 4\\nSome new code")
-        ('Some new code', ['3', '4'])
-        >>> k.get_code_and_args("%reset\\nSome new code")
-        ('Some new code', [])
+        >>> k.get_code_and_magic("")
+        ('', {})
+        >>> k.get_code_and_magic("%args 1\\nSome code")
+        ('Some code', {'args': ['1']})
+        >>> k.get_code_and_magic("%args 2\\nSome more code")
+        ('Some code\\nSome more code', {'args': ['2']})
+        >>> k.get_code_and_magic("")
+        ('Some code\\nSome more code\\n', {'args': ['2']})
+        >>> k.get_code_and_magic("%reset\\n%args 3 4\\nSome new code")
+        ('Some new code', {'args': ['3', '4']})
+        >>> k.get_code_and_magic("%reset\\nSome new code")
+        ('Some new code', {})
         """
         new_code, new_magic = self.get_magic(new_code)
 
         # Only a reset in the newest cell actually does a reset.
         if new_magic.get("reset") is not None:
             self._previous_code = new_code
             self._previous_magic = new_magic
         else:
             self._previous_code += ("\n" if self._previous_code else "") + new_code
             self._previous_magic.update(new_magic)
 
-        return self._previous_code, self._previous_magic.get("args", [])
+        return self._previous_code, self._previous_magic
 
     def do_execute(
         self, code, silent, store_history=True, user_expressions=None, allow_stdin=False
     ):
         """Execute user code using llvm-tblgen binary."""
-        all_code, args = self.get_code_and_args(code)
+        all_code, magic = self.get_code_and_magic(code)
+
+        args = magic.get("args", [])
+        if magic.get("json") is not None:
+            args.append("--dump-json")
 
         with tempfile.TemporaryFile("w+") as f:
             f.write(all_code)
             f.seek(0)
             got = subprocess.run(
                 [self.executable, *args],
                 stdin=f,
                 stderr=subprocess.PIPE,
                 stdout=subprocess.PIPE,
                 universal_newlines=True,
             )
 
         if not silent:
             if got.stderr:
                 self.send_response(
                     self.iopub_socket, "stream", {"name": "stderr", "text": got.stderr}
                 )
             else:
+                out = got.stdout
+                if magic.get("json") is not None:
+                    j = json.loads(out)
+                    out = json.dumps(j, indent=4)
+
                 self.send_response(
-                    self.iopub_socket, "stream", {"name": "stdout", "text": got.stdout}
+                    self.iopub_socket,
+                    "stream",
+                    {"name": "stdout", "text": out},
                 )
 
         return {
             "status": "ok",
             "execution_count": self.execution_count,
             "payload": [],
             "user_expressions": {},
         }
 
 
 if __name__ == "__main__":
     import doctest
 
     doctest.testmod(extraglobs={"k": TableGenKernel()})