[LLVM][TableGen] Support combined cells in jupyter kernel

[DavidSpickett]

Summary

This changes the default mode to cache the code blocks we're
asked to compile until we see the new %reset magic to clear that cache.

This means that if you run several cells in sequence, at the end you're
compiling the code from all the cells at once.

This emulates what the ipython kernel does where it uses a persistent
interpreter state by default.

%reset will only be acted on when it's in the cell we're asked to run
(the newest code).

%args we will use the most recent value we have cached.

The example notebook has been updated to explain that.

Depends on D132378

[DavidSpickett]

Rebase

[jpienaar]

This looks nicer and I can see appeal for literate style. One question comes to mind: what happens when you click runs out of order? E.g., click 1,2,3,2,2 ? In python one ends up rendering and clobbering, what would one get here?

[DavidSpickett]

Yes it behaves like the Python kernel, the order that you run cells in matters.

[jpienaar]

Mostly looks good, not sure about collapsing magic at top, but beyond that.

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

86

Wouldn't this collapse all magic to top rather than allowing different magic per section? And in the examples, it seems we have magic at the top only per sections of code.

[DavidSpickett]

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

86

By section do you mean block of code _within_ a cell? If so you are right and this is how I intend that to work, for now at least. This is what I meant by the comment further down about only supporting "cell magic".

If you're wondering if magic in the middle of a cell will it be sort of moved up and made to apply to the whole cell, it won't. As shown in this test:

>>> k.get_magic("%foo a b\\n \\n%foo c d")
        (' \\n%foo c d', {'foo': ['a', 'b']})

After you break the run of lines beginning with % we assume it's all source code.

If you're instead talking about what happens across cells then it goes like this.

If the cell does a reset then the magic is cleaned and it can set new values as needed. Otherwise we'll merge the magic from the most recent cell with the set of cached magic. For example:

%args --disable-the-warning
// some code that only compiles with a warning disabled.

% args
// some more code, which will be added to the previous cell content

The first cell will compile the second one won't.

Since the model is based around giving llvm-tblgen one source file and one set of arguments.

Hopefully that makes sense, if the code gives a different impression I'll fix it and make it clearer.

156

Here is where the magic from the most recent cell updates the cached magic.

[jpienaar]

LGTM

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

86

Ah I thought it collect all arguments and collect all source and pass in to one invocation. Rather, it concatenates source and overrides args (if provided) as you go.

[DavidSpickett]

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

86

Exactly.

So you'd need to restate things sometimes. If you had done %args --foo and want to pass --bar you'd need to do %args --foo --bar. Given that arguments to llvm-tblgen are last value wins, we could append to a list of arguments instead and avoid the need to do that.

Either way, there aren't too many arguments you'd want to pass anyway. So it's not a big burden to restate them.

llvm/utils/TableGen/jupyter/LLVM_TableGen.ipynb

 # LLVM TableGen Kernel
 This notebook is running `llvm-tblgen`.
+In [1]:%reset
 // This is some tablegen
 class Foo {}
 ------------- Classes -----------------
class Foo {
}
------------- Defs -----------------
 Errors printed to stderr are shown.
+In [2]:%reset
 This is not tablegen.
 <stdin>:1:1: error: Unexpected token at top level
This is not tablegen.
^
 Add some classes to get some output.
+In [3]:%reset
 class Stuff {}
 def thing : Stuff {}
 ------------- Classes -----------------
class Stuff {
}
------------- Defs -----------------
def thing {	// Stuff
}
-Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one.
+By default cells are connected. Meaning that we cache the code and magic directives from the previously run cells.
+This means that the next cell still sees the `Stuff` class.
+In [4]:def other_thing : Stuff {}
+------------- Classes -----------------
class Stuff {
}
------------- Defs -----------------
def other_thing {	// Stuff
}
def thing {	// Stuff
}
+You can use the magic `%reset` to clear this cache and start fresh.
+In [5]:%reset
 def other_thing : Stuff {}
 <stdin>:1:19: error: Couldn't find class 'Stuff'
def other_thing : Stuff {}
                  ^
 There is a "magic" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.
 For example, here we have some code that shows a warning.
+In [6]:%reset
 class Thing <int A, int B> {
     int num = A;

 <stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {
                        ^
 We can pass an argument to ignore that warning.
-In [6]:In [6]:%args --no-warn-on-unused-template-args
-class Thing <int A, int B> {
-    int num = A;
-}
+In [7]:In [7]:%args --no-warn-on-unused-template-args
+------------- Classes -----------------
class Thing<int Thing:A = ?, int Thing:B = ?> {
  int num = Thing:A;
}
------------- Defs -----------------
+If you have a run of cells without a `%reset`, the most recent `%args` is used.
+In [8]:In [8]:// This passes --no-warn-on-unused-template-args
 ------------- Classes -----------------
class Thing<int Thing:A = ?, int Thing:B = ?> {
  int num = Thing:A;
}
------------- Defs -----------------
-The last `%args` in a cell will be the arguments used.
-In [7]:%args --no-warn-on-unused-template-args
 In [9]:%args
-class Thing <int A, int B> {
-    int num = A;
-}
+// Now we're not passing the argument so the warning comes back.
 <stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {
                        ^
+If there are many `%args` in a cell, the last one is used.
+In [10]:%reset
+%args --no-warn-on-unused-template-args
+%args
+class Thing <int A, int B> {}
+<stdin>:1:18: warning: unused template argument: Thing:A
class Thing <int A, int B> {}
                 ^
<stdin>:1:25: warning: unused template argument: Thing:B
class Thing <int A, int B> {}
                        ^

llvm/utils/TableGen/jupyter/LLVM_TableGen.md

 # LLVM TableGen Kernel
 
 This notebook is running `llvm-tblgen`.
 
 
 ```tablegen
+%reset
 // This is some tablegen
 class Foo {}
 ```
 
     ------------- Classes -----------------
     class Foo {
     }
     ------------- Defs -----------------
 
 
 Errors printed to stderr are shown.
 
 
 ```tablegen
+%reset
 This is not tablegen.
 ```
 
     <stdin>:1:1: error: Unexpected token at top level
     This is not tablegen.
     ^
 
 
 Add some classes to get some output.
 
 
 ```tablegen
+%reset
 class Stuff {}
 def thing : Stuff {}
 ```
 
     ------------- Classes -----------------
     class Stuff {
     }
     ------------- Defs -----------------
     def thing { // Stuff
     }
 
 
-Currently cells are not connected. Meaning that this next cell doesn't have the class from the previous one.
+By default cells are connected. Meaning that we cache the code and magic directives from the previously run cells.
+
+This means that the next cell still sees the `Stuff` class.
+
+
+```tablegen
+def other_thing : Stuff {}
+```
+
+    ------------- Classes -----------------
+    class Stuff {
+    }
+    ------------- Defs -----------------
+    def other_thing { // Stuff
+    }
+    def thing { // Stuff
+    }
+
+
+You can use the magic `%reset` to clear this cache and start fresh.
 
 
 ```tablegen
+%reset
 def other_thing : Stuff {}
 ```
 
     <stdin>:1:19: error: Couldn't find class 'Stuff'
     def other_thing : Stuff {}
                       ^
 
 
 There is a "magic" directive `%args` that you can use to send command line arguments to `llvm-tblgen`.
 
 For example, here we have some code that shows a warning.
 
 
 ```tablegen
+%reset
 class Thing <int A, int B> {
     int num = A;
 }
 ```
 
     <stdin>:1:25: warning: unused template argument: Thing:B
     class Thing <int A, int B> {
                             ^
 
 
 We can pass an argument to ignore that warning.
 
 
 ```tablegen
 %args --no-warn-on-unused-template-args
-class Thing <int A, int B> {
-    int num = A;
-}
 ```
 
     ------------- Classes -----------------
     class Thing<int Thing:A = ?, int Thing:B = ?> {
       int num = Thing:A;
     }
     ------------- Defs -----------------
 
 
-The last `%args` in a cell will be the arguments used.
+If you have a run of cells without a `%reset`, the most recent `%args` is used.
 
 
 ```tablegen
-%args --no-warn-on-unused-template-args
-%args
-class Thing <int A, int B> {
-    int num = A;
+// This passes --no-warn-on-unused-template-args
+```
+
+    ------------- Classes -----------------
+    class Thing<int Thing:A = ?, int Thing:B = ?> {
+      int num = Thing:A;
     }
+    ------------- Defs -----------------
+
+
+
+```tablegen
+%args
+// Now we're not passing the argument so the warning comes back.
 ```
 
     <stdin>:1:25: warning: unused template argument: Thing:B
     class Thing <int A, int B> {
                             ^
 
 
+If there are many `%args` in a cell, the last one is used.
 
-```tablegen
 
+```tablegen
+%reset
+%args --no-warn-on-unused-template-args
+%args
+class Thing <int A, int B> {}
 ```
+
+    <stdin>:1:18: warning: unused template argument: Thing:A
+    class Thing <int A, int B> {}
+                     ^
+    <stdin>:1:25: warning: unused template argument: Thing:B
+    class Thing <int A, int B> {}
+                            ^
+

llvm/utils/TableGen/jupyter/tablegen_kernel/kernel.py

 
 
 class TableGenKernel(Kernel):
     """Kernel using llvm-tblgen inside jupyter.
 
     All input is treated as TableGen unless the first non whitespace character
     is "%" in which case it is a "magic" line.
 
-    The only magic line supported is "%args". The rest of the line is the
-    arguments passed to llvm-tblgen.
+    The supported magic is:
+    * %args - to set the arguments passed to llvm-tblgen.
+    * %reset - to reset the cached code and magic state.
 
-    This is "cell magic" meaning it applies to the whole cell. Therefore
+    These are "cell magic" meaning it applies to the whole cell. Therefore
     it must be the first line, or part of a run of magic lines starting
     from the first line.
 
     ```tablgen
     %args
+    %reset
     %args --print-records --print-detailed-records
     class Stuff {
       string Name;
     }
 
     def a_thing : Stuff {}
     ```
 
language_info = {
         "mimetype": "text/x-tablegen",
         "file_extension": ".td",
         "pygments_lexer": "text",
     }
 
     def __init__(self, **kwargs):
         Kernel.__init__(self, **kwargs)
         self._executable = None
+        # A list of (code, magic) tuples.
+        # All the previous cell's code we have run since the last reset.
+        # This emulates a persistent state like a Python interpreter would have.
+        self._previous_code = ""
+        # The most recent set of magic since the last reset.
+        self._previous_magic = {}
 
     @property
     def banner(self):
         return "llvm-tblgen kernel %s" % __version__
 
     @property
     def executable(self):
         """If this is the first run, search for llvm-tblgen.
         Otherwise return the cached path to it."""
         if self._executable is None:
             path = os.environ.get("LLVM_TBLGEN_EXECUTABLE")
             if path is not None and os.path.isfile(path) and os.access(path, os.X_OK):
                 self._executable = path
             else:
                 path = shutil.which("llvm-tblgen")
                 if path is None:
                     raise OSError("llvm-tblgen not found, please see README")
                 self._executable = path
 
         return self._executable
 
     def get_magic(self, code):
-        """Given a block of code remove the magic lines from it and return
-        a tuple of the code lines (newline joined) and a list of magic lines
-        with their leading spaces removed.
+        """Given a block of code remove the magic lines from it.
+        Returns a tuple of newline joined code lines and a dictionary of magic.
+        Where the key is the magic name (minus the %) and the values are lists
+        of the arguments to the magic.
 
         Currently we only look for "cell magic" which must be at the start of
         the cell. Meaning the first line, or a set of lines beginning with %
         that come before the first non-magic line.
 
         >>> k.get_magic("")
-        ('', [])
+        ('', {})
         >>> k.get_magic("Hello World.\\nHello again.")
-        ('Hello World.\\nHello again.', [])
+        ('Hello World.\\nHello again.', {})
         >>> k.get_magic("   %foo a b c")
-        ('', ['%foo a b c'])
-        >>> k.get_magic("   %foo a b c\\n%foo\\nFoo")
-        ('Foo', ['%foo a b c', '%foo'])
+        ('', {'foo': ['a', 'b', 'c']})
+        >>> k.get_magic("%foo\\n   %foo a b c\\nFoo")
+        ('Foo', {'foo': ['a', 'b', 'c']})
+        >>> k.get_magic("%foo\\n%bar\\nFoo")
+        ('Foo', {'foo': [], 'bar': []})
         >>> k.get_magic("Foo\\n%foo\\nFoo")
-        ('Foo\\n%foo\\nFoo', [])
-        >>> k.get_magic("%foo\\n   Foo\\n%foo")
-        ('   Foo\\n%foo', ['%foo'])
-        >>> k.get_magic("%foo\\n\\n%foo")
-        ('\\n%foo', ['%foo'])
-        >>> k.get_magic("%foo\\n \\n%foo")
-        (' \\n%foo', ['%foo'])
+        ('Foo\\n%foo\\nFoo', {})
+        >>> k.get_magic("%bar\\n\\n%foo")
+        ('\\n%foo', {'bar': []})
+        >>> k.get_magic("%foo a b\\n   Foo\\n%foo c d")
+        ('   Foo\\n%foo c d', {'foo': ['a', 'b']})
+        >>> k.get_magic("%foo a b\\n \\n%foo c d")
+        (' \\n%foo c d', {'foo': ['a', 'b']})
         """
-        magic_lines = []
+        magic = {}
         code_lines = []
 
         lines = code.splitlines()
         while lines:
             line = lines.pop(0)
             possible_magic = line.lstrip()
             if possible_magic.startswith("%"):
-                magic_lines.append(possible_magic)
+                magic_parts = possible_magic.split()
+                # Key has the % removed
+                magic[magic_parts[0][1:]] = magic_parts[1:]
             else:
                 code_lines = [line, *lines]
                 break
 
-        return "\n".join(code_lines), magic_lines
+        return "\n".join(code_lines), magic
+
+    def get_code_and_args(self, new_code):
+        """Get the code that do_execute should use, taking into account
+        the code from any cached cells.
+
+        Returns the code to compile and the arguments to use to do so.
+
+        >>> k._previous_code = ""
+        >>> k._previous_magic = {}
+        >>> k.get_code_and_args("")
+        ('', [])
+        >>> k.get_code_and_args("%args 1\\nSome code")
+        ('Some code', ['1'])
+        >>> k.get_code_and_args("%args 2\\nSome more code")
+        ('Some code\\nSome more code', ['2'])
+        >>> k.get_code_and_args("%reset\\n%args 3 4\\nSome new code")
+        ('Some new code', ['3', '4'])
+        >>> k.get_code_and_args("%reset\\nSome new code")
+        ('Some new code', [])
+        """
+        new_code, new_magic = self.get_magic(new_code)
+
+        # Only a reset in the newest cell actually does a reset.
+        if new_magic.get("reset") is not None:
+            self._previous_code = new_code
+            self._previous_magic = new_magic
+        else:
+            self._previous_code += ("\n" if self._previous_code else "") + new_code
+            self._previous_magic.update(new_magic)
+
+        return self._previous_code, self._previous_magic.get("args", [])
 
     def do_execute(
         self, code, silent, store_history=True, user_expressions=None, allow_stdin=False
     ):
         """Execute user code using llvm-tblgen binary."""
-        code, magic = self.get_magic(code)
-
-        extra_args = []
-        for m in magic:
-            if m.startswith("%args"):
-                # Last one in wins
-                extra_args = m.split()[1:]
+        all_code, args = self.get_code_and_args(code)
 
         with tempfile.TemporaryFile("w+") as f:
-            f.write(code)
+            f.write(all_code)
             f.seek(0)
             got = subprocess.run(
-                [self.executable, *extra_args],
+                [self.executable, *args],
                 stdin=f,
                 stderr=subprocess.PIPE,
                 stdout=subprocess.PIPE,
                 universal_newlines=True,
             )
 
         if not silent:
             if got.stderr: