[TableGen] Add a new json textmate description for syntax highlighting

[rriddle]

Summary

There isn't really a good pre-existing syntax highlighter for tablegen, so this
commit adds a textmate version that covers nearly everything in the current
spec.

[rriddle]

Added a few reviewers that I've been able to bamboozle into reviewing tablegen related things. Feel free to comment on the spec itself, but the thing I wanted some thoughts on was the currently existing tablegen textmate bundle. It hasn't been touched in ~10 years and is written in xml, should I just delete all of the existing stuff? (https://github.com/llvm/llvm-project/tree/main/llvm/utils/textmate) I don't use the macos textmate editor referenced there, so I'm not sure how to update that README.

Some context for this change is that I'm working on integrating tablegen language features into the MLIR vscode extension, and syntax highlighting is a big part of that.

[lattner]

I'm not a competent reviewer for this patch, but I love tblgen getting better tooling :-)

[rriddle]

Yeah, I'm not sure who would be good to review the actual json itself. Mostly just wondering if I should care about the existing xml package or not, happy to land this and not care about the other thing.

Here are some vscode screen grabs comparing the "LLVM TableGen" extension (which is what I think most people use for vscode, and is semi-based on the current grammar in-tree) with this (in the MLIR extension):

Before | After

[lattner]

Very cool, if there is no good reviewer for this, I'd say "land it; advertise it; react to feedback from users" :-)

[sebastian-ne]

The highlighting looks nice with this.
I didn’t see anything blatantly wrong or missing, so thumbs up from me.

FWIW, there’s a tree-sitter grammar here: https://github.com/Flakebi/tree-sitter-tablegen
which also lists some interesting edge cases like nested comments or identifiers that start with digits: https://github.com/Flakebi/tree-sitter-tablegen/blob/master/test/corpus/regressions.td

For fun facts, 2var is a valid identifier in tablegen, but in LLVM IR %2var is actually a variable and a keyword (yes, %2 = load i8, i8* %1store i8 %2, i8* %1 is valid IR).

llvm/utils/textmate/tablegen.json

44–51

I’m not familiar with textmate parsers, so not sure if this is handled here (or can be handled), but tablegen supports nested block comments. I.e.

/* nested /*
comment */
still a comment
*/

(and for the record, only tablegen supports nested comments, LLVM IR or MIR doesn’t)

218

field is also a keyword (although deprecated)

[jpienaar]

In D125427#3511309, @sebastian-ne wrote:

FWIW, there’s a tree-sitter grammar here: https://github.com/Flakebi/tree-sitter-tablegen

Ooh nice, I was looking for one for an experiment.

In D125427#3510845, @lattner wrote:

Very cool, if there is no good reviewer for this, I'd say "land it; advertise it; react to feedback from users" :-)

+1, the output looks good, structuring seems good and we can iterate.

[rriddle]

llvm/utils/textmate/tablegen.json

44–51

Oooh thanks for pointing this out. I just need to add recursion, but this slipped my mind.

218

Yep thanks, I originally dropped it because I wasn't sure how widespread it was actually still being used (despite the "deprecated" tagline). We have never used it in MLIR, but seems like it's still heavily used in older LLVM .td files.

[rriddle]

Thanks all for the review!

llvm/utils/textmate/tablegen.json

+{
+  "name": "TableGen",
+  "fileTypes": [
+    "td"
+  ],
+  "patterns": [
+    {
+      "include": "#comments"
+    },
+    {
+      "include": "#preprocessor"
+    },
+    {
+      "include": "#strings"
+    },
+    {
+      "include": "#integer"
+    },
+    {
+      "include": "#builtin_types"
+    },
+    {
+      "include": "#keywords"
+    },
+    {
+      "include": "#classes"
+    },
+    {
+      "include": "#defs"
+    },
+    {
+      "include": "#values"
+    }
+  ],
+  "repository": {
+    "comments": {
+      "patterns": [
+        {
+          "match": "\/\/.*$",
+          "name": "comment.line.double-slash.tablegen"
+        },
+        {
+          "begin": "/\\*",
+          "end": "\\*/",
+          "captures": {
+            "0": {
+              "name": "comment.block.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "include": "#comments"
+            }
+          ],
+          "name": "comment.block.tablegen"
+        }
+      ]
+    },
+    "preprocessor": {
+      "patterns": [
+        {
+          "match": "^#ifdef\\s+(\\w+)",
+          "name": "keyword.control.ifdef.tablegen",
+          "captures": {
+            "1": {
+              "name": "entity.name.function.preprocessor.tablegen"
+            }
+          }
+        },
+        {
+          "match": "^#ifndef\\s+(\\w+)",
+          "name": "keyword.control.ifndef.tablegen",
+          "captures": {
+            "1": {
+              "name": "entity.name.function.preprocessor.tablegen"
+            }
+          }
+        },
+        {
+          "match": "^#else",
+          "name": "keyword.control.else.tablegen"
+        },
+        {
+          "match": "#endif",
+          "name": "keyword.control.endif.tablegen"
+        },
+        {
+          "match": "#define\\s+(\\w+)",
+          "name": "keyword.control.ifndef.tablegen",
+          "captures": {
+            "1": {
+              "name": "entity.name.function.preprocessor.tablegen"
+            }
+          }
+        },
+        {
+          "match": "^include\\s+",
+          "name": "keyword.control.include.tablegen"
+        }
+      ]
+    },
+    "strings": {
+      "patterns": [
+        {
+          "name": "string.quoted.double.tablegen",
+          "begin": "\"",
+          "end": "\"",
+          "beginCaptures": {
+            "0": {
+              "name": "punctuation.definition.string.begin.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "match": "\\\\['nt\"]",
+              "name": "constant.character.escape.tablegen"
+            }
+          ],
+          "endCaptures": {
+            "0": {
+              "name": "punctuation.definition.string.end.tablegen"
+            }
+          }
+        },
+        {
+          "name": "string.quoted.triple.tablegen",
+          "begin": "\\[{",
+          "end": "}]",
+          "beginCaptures": {
+            "0": {
+              "name": "punctuation.definition.string.begin.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "match": "\\\\[nt\"]",
+              "name": "constant.character.escape.tablegen"
+            }
+          ],
+          "endCaptures": {
+            "0": {
+              "name": "punctuation.definition.string.end.tablegen"
+            }
+          }
+        }
+      ]
+    },
+    "integer": {
+      "patterns": [
+        {
+          "match": "\\b\\d+\\b",
+          "name": "constant.numeric.tablegen"
+        },
+        {
+          "match": "\\b0[xX][a-fA-F0-9]+\\b",
+          "name": "constant.numeric.hex.tablegen"
+        },
+        {
+          "match": "\\b0b[01]+\\b",
+          "name": "constant.numeric.binary.tablegen"
+        }
+      ]
+    },
+    "builtin_types": {
+      "patterns": [
+        {
+          "match": "\\b(?:bit|code|dag|int|string)\\b",
+          "name": "entity.name.type.tablegen"
+        },
+        {
+          "begin": "\\b(bits)\\<",
+          "beginCaptures": {
+            "1": {
+              "name": "entity.name.type.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "include": "#integer"
+            }
+          ],
+          "end": "\\>"
+        },
+        {
+          "begin": "\\b(list)\\<",
+          "beginCaptures": {
+            "1": {
+              "name": "entity.name.type.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "include": "#types"
+            }
+          ],
+          "end": "\\>"
+        }
+      ]
+    },
+    "types": {
+      "patterns": [
+        {
+          "include": "#builtin_types"
+        },
+        {
+          "match": "\\b(\\w+)\\b",
+          "name": "entity.name.type.tablegen"
+        }
+      ]
+    },
+    "bang_operators": {
+      "patterns": [
+        {
+          "begin": "(\\!cast|\\!getdagop|\\!isa)\\<",
+          "beginCaptures": {
+            "1": {
+              "name": "entity.name.function"
+            }
+          },
+          "patterns": [
+            {
+              "include": "#types"
+            }
+          ],
+          "end": "\\>"
+        },
+        {
+          "match": "(\\!\\w+)\\b",
+          "name": "entity.name.function"
+        }
+      ]
+    },
+    "keywords": {
+      "patterns": [
+        {
+          "match": "\\b(assert|else|foreach|defset|defvar|field|if|in|let|then)\\b",
+          "name": "keyword.other.tablegen"
+        }
+      ]
+    },
+    "record_body": {
+      "patterns": [
+        {
+          "begin": "\\:",
+          "patterns": [
+            {
+              "match": "\\b(\\w+)\\b",
+              "name": "entity.name.type.tablegen"
+            },
+            {
+              "begin": "\\<",
+              "patterns": [
+                {
+                  "include": "#values"
+                }
+              ],
+              "end": "\\>"
+            }
+          ],
+          "end": "(?=\\;|\\{)"
+        },
+        {
+          "begin": "\\{",
+          "patterns": [
+            {
+              "include": "#keywords"
+            },
+            {
+              "include": "#builtin_types"
+            },
+            {
+              "match": "\\b(\\w+\\s+)?\\b(\\w+)\\b\\s+\\=",
+              "captures": {
+                "1": {
+                  "name": "entity.name.type.tablegen"
+                },
+                "2": {
+                  "name": "variable.other.tablegen"
+                }
+              }
+            },
+            {
+              "include": "#values"
+            }
+          ],
+          "end": "(?=\\})"
+        }
+      ]
+    },
+    "classes": {
+      "begin": "\\b(multiclass|class)\\b\\s+(\\w+)\\b",
+      "beginCaptures": {
+        "1": {
+          "name": "keyword.other.class.tablegen"
+        },
+        "2": {
+          "name": "entity.name.type.tablegen"
+        }
+      },
+      "patterns": [
+        {
+          "begin": "\\<",
+          "patterns": [
+            {
+              "include": "#builtin_types"
+            },
+            {
+              "match": "\\b(\\w+\\s+)?\\b(\\w+)\\b",
+              "captures": {
+                "1": {
+                  "name": "entity.name.type.tablegen"
+                },
+                "2": {
+                  "name": "variable.other.tablegen"
+                }
+              }
+            },
+            {
+              "begin": "=",
+              "patterns": [
+                {
+                  "include": "#values"
+                }
+              ],
+              "end": "(?=\\>|\\,)"
+            }
+          ],
+          "end": "\\>"
+        },
+        {
+          "include": "#record_body"
+        }
+      ],
+      "end": "(\\;|\\})"
+    },
+    "defs": {
+      "begin": "\\b(defm|def)\\b",
+      "beginCaptures": {
+        "1": {
+          "name": "keyword.other.def.tablegen"
+        }
+      },
+      "patterns": [
+        {
+          "include": "#record_body"
+        },
+        {
+          "include": "#values"
+        }
+      ],
+      "end": "(\\;|\\})"
+    },
+    "values": {
+      "patterns": [
+        {
+          "include": "#builtin_types"
+        },
+        {
+          "include": "#bang_operators"
+        },
+        {
+          "include": "#classes"
+        },
+        {
+          "include": "#defs"
+        },
+        {
+          "include": "#comments"
+        },
+        {
+          "include": "#integer"
+        },
+        {
+          "include": "#strings"
+        },
+        {
+          "match": "\\b(?:false|true)\\b",
+          "name": "keyword.other.tablegen"
+        },
+        {
+          "match": "(\\?)",
+          "name": "keyword.other.tablegen"
+        },
+        {
+          "begin": "\\{",
+          "patterns": [
+            {
+              "include": "#values"
+            }
+          ],
+          "end": "\\}"
+        },
+        {
+          "begin": "\\[",
+          "patterns": [
+            {
+              "include": "#values"
+            }
+          ],
+          "end": "(?=\\])"
+        },
+        {
+          "begin": "\\]\\s*\\<",
+          "patterns": [
+            {
+              "include": "#types"
+            }
+          ],
+          "end": "\\>"
+        },
+        {
+          "begin": "\\(",
+          "patterns": [
+            {
+              "include": "#values"
+            }
+          ],
+          "end": "\\)"
+        },
+        {
+          "include": "#keywords"
+        },
+        {
+          "begin": "\\b(\\w+)\\<",
+          "beginCaptures": {
+            "1": {
+              "name": "entity.name.type.tablegen"
+            }
+          },
+          "patterns": [
+            {
+              "include": "#values"
+            }
+          ],
+          "end": "\\>"
+        },
+        {
+          "match": "(\\$\\w+)\\b",
+          "name": "variable.other.enummember.tablegen"
+        },
+        {
+          "match": "\\b(\\w+)\\b",
+          "name": "variable.other.tablegen"
+        },
+        {
+          "include": "#preprocessor"
+        }
+      ]
+    }
+  },
+  "scopeName": "source.tablegen"
+}
 No newline at end of file