[gn build] Add template for running llvm-tblgen and use it to add build file for llvm/lib/IR.

[thakis]

Summary

Also adds a boring build file for llvm/lib/BinaryFormat (needed by llvm/lib/IR).

lib/IR marks Attributes and IntrinsicsEnum as public_deps (because IR's public headers include the generated .inc files), so projects depending on lib/IR will implicitly depend on them being generated. As a consequence, most targets won't have to explicitly list a dependency on these tablegen steps (contrast with intrinsics_gen in the cmake build).

This doesn't yet have the optimization where tablegen's output is only updated if it's changed.

[thakis]

git ls-files '*.gn' '*.gni' | xargs -n 1 gn format

[thakis]

…wait, putting the copy-if-different logic in the tablegen runner as I did doesn't actually work, since tablegen will write the name of the temp file in the depfile this way. I'll just put a FIXME for adding write-if-different back into tablegen itself (aka reland r330742) and revert the runner script to the 2 lines it originally was. I'll upload a new patch set in an hour or two; if you look earlier please ignore run_tablegen.py (the rest will stay as-is).

[thakis]

remove not functional "write if different" attempt

[phosek]

llvm/utils/gn/build/run_tablegen.py

9

Chromium version does more fancy error checking on Windows, shall we do the same?

llvm/utils/gn/secondary/llvm/utils/TableGen/tablegen.gni

28

Have you considered copying Chromium's [compiled_action.gni](https://chromium.googlesource.com/chromium/src/+/master/build/compiled_action.gni) and then implementing tablegen in terms of that? It seems generally useful as I suspect you'll need the same functionality for Clang tablegen as well in the future.

38

You should probably forward testonly as well`.

[thakis]

Thanks for the review!

llvm/utils/gn/build/run_tablegen.py

9

That's somewhat recent (https://chromium.googlesource.com/chromium/src/+/8c50b3ed049c736406943ac24dc40fc8ac1e782c). Since this script is only used to run tblgen, shorter code seemed preferable over the small utility gain.

llvm/utils/gn/secondary/llvm/utils/TableGen/tablegen.gni

28

I did consider it! Right now this is only used for tablegen though, and for upstreaming I figured I'd make clang_tblgen call this template and make tblgen_target something the invoker can set, and then clang_tblgen would look like (pseudocode):

template("clang_tblgen") {

tablegen($target_name) {
  tblgen_target = "//clang/utils/TableGen:clang-tblgen"
}

}

(in my prototyping branch, clang-tblgen is just a copy-pasted version of this file so I'm not 100% sure if this is going to pan out, but I think it should.)

38

I haven't needed it yet -- are you aware of any test-only tblgen invocations?

[phosek]

LGTM

llvm/utils/gn/secondary/llvm/utils/TableGen/tablegen.gni

38

Nope, but it's a general pattern to always forward visibility and testonly.

llvm/utils/gn/build/run_tablegen.py

+#!/usr/bin/env python
+"""Runs tablegen."""
+
+# This makes llvm-tblgen write the output to a temp file and only copies
+# it to the final location if the output is different from last run.
+# FIXME: llvm-tblgen should do this itself. r330742 tried this, but it caused
+# problems. Fix those and reland, then rename this script to run_host_binary.py
+# and make it only do 
+# `sys.exit(subprocess.call(['./' + sys.argv[1]] + sys.argv[2:]))`
+
+from __future__ import print_function
+
+import os
+import subprocess
+import sys
+import tempfile
+
+
+def main():
+    output = None
+    for i in range(1, len(sys.argv)):
+        if sys.argv[i] == '-o':
+            output_index = i + 1
+            output = sys.argv[i + 1]
+            break
+    if not output:
+        print('failed to find -o argument', file=sys.stderr)
+        return 1
+    
+    # Run tablegen, but write to a temp file.
+    temp_file = tempfile.NamedTemporaryFile(delete=False)
+    temp_file.close()  # So that llvm-tblgen can write to it on Windows.
+
+    # Prefix with ./ to run built binary, not arbitrary stuff from PATH.
+    cmd = ['./' + sys.argv[1]] + sys.argv[2:]
+    cmd[output_index - 1] = temp_file.name
+    subprocess.check_call(cmd)
+
+    # Open new output, and overwrite if it's different from old output.
+    with open(temp_file.name, 'rb') as temp_file_handle:
+        new_output = temp_file_handle.read()
+    os.remove(temp_file.name)
+
+    try:
+        with open(output, 'rb') as output_handle:
+            should_write = output_handle.read() != new_output
+    except IOError as e:
+        if error.errno != errno.ENOENT:
+            raise
+        should_write = True
+
+    if should_write:
+        with open(output, 'wb') as output_handle:
+            output_handle.write(new_output)
+
+
+if __name__ == '__main__':
+    sys.exit(main())

llvm/utils/gn/secondary/BUILD.gn

 group("default") {
   deps = [
-    "//llvm/utils/TableGen:llvm-tblgen",
+    "//llvm/lib/IR",
     "//llvm/tools/llvm-undname",
+    "//llvm/utils/TableGen:llvm-tblgen",
   ]
 }
 
 # A pool called "console" in the root BUILD.gn is magic and represents ninja's
 # built-in console pool. (Requires a GN with `gn --version` >= 552353.)
 pool("console") {
   depth = 1
 }

llvm/utils/gn/secondary/llvm/include/llvm/IR/BUILD.gn

+import("//llvm/utils/TableGen/tablegen.gni")
+
+tablegen("IntrinsicEnums") {
+  visibility = [ "//llvm/lib/IR" ]
+  args = [ "-gen-intrinsic-enums" ]
+  td_file = "Intrinsics.td"
+}
+
+tablegen("IntrinsicImpl") {
+  visibility = [ "//llvm/lib/IR" ]
+  args = [ "-gen-intrinsic-impl" ]
+  td_file = "Intrinsics.td"
+}
+
+tablegen("Attributes") {
+  visibility = [ "//llvm/lib/IR" ]
+  args = [ "-gen-attrs" ]
+}

llvm/utils/gn/secondary/llvm/lib/BinaryFormat/BUILD.gn

+static_library("BinaryFormat") {
+  output_name = "LLVMBinaryFormat"
+  deps = [
+    "//llvm/lib/Support",
+  ]
+  sources = [
+    "Dwarf.cpp",
+    "Magic.cpp",
+    "MsgPackReader.cpp",
+    "MsgPackWriter.cpp",
+    "Wasm.cpp",
+  ]
+}

llvm/utils/gn/secondary/llvm/lib/IR/BUILD.gn

+import("//llvm/utils/TableGen/tablegen.gni")
+
+tablegen("AttributesCompatFunc") {
+  visibility = [ ":IR" ]
+  args = [ "-gen-attrs" ]
+}
+
+static_library("IR") {
+  output_name = "LLVMCore"
+  public_deps = [
+    # Must be public_dep because IR's public headers include llvm-config.h.
+    "//llvm/include/llvm/Config:llvm-config",
+
+    # Must be public_dep because IR's public headers include Attributes.inc.
+    "//llvm/include/llvm/IR:Attributes",
+
+    # Must be public_dep because IR's public headers include IntrinsicEnums.inc.
+    "//llvm/include/llvm/IR:IntrinsicEnums",
+  ]
+  deps = [
+    ":AttributesCompatFunc",
+    "//llvm/include/llvm/IR:IntrinsicImpl",
+    "//llvm/lib/BinaryFormat",
+    "//llvm/lib/Support",
+  ]
+  sources = [
+    "AsmWriter.cpp",
+    "Attributes.cpp",
+    "AutoUpgrade.cpp",
+    "BasicBlock.cpp",
+    "Comdat.cpp",
+    "ConstantFold.cpp",
+    "ConstantRange.cpp",
+    "Constants.cpp",
+    "Core.cpp",
+    "DIBuilder.cpp",
+    "DataLayout.cpp",
+    "DebugInfo.cpp",
+    "DebugInfoMetadata.cpp",
+    "DebugLoc.cpp",
+    "DiagnosticHandler.cpp",
+    "DiagnosticInfo.cpp",
+    "DiagnosticPrinter.cpp",
+    "DomTreeUpdater.cpp",
+    "Dominators.cpp",
+    "Function.cpp",
+    "GVMaterializer.cpp",
+    "Globals.cpp",
+    "IRBuilder.cpp",
+    "IRPrintingPasses.cpp",
+    "InlineAsm.cpp",
+    "Instruction.cpp",
+    "Instructions.cpp",
+    "IntrinsicInst.cpp",
+    "LLVMContext.cpp",
+    "LLVMContextImpl.cpp",
+    "LegacyPassManager.cpp",
+    "MDBuilder.cpp",
+    "Mangler.cpp",
+    "Metadata.cpp",
+    "Module.cpp",
+    "ModuleSummaryIndex.cpp",
+    "Operator.cpp",
+    "OptBisect.cpp",
+    "Pass.cpp",
+    "PassInstrumentation.cpp",
+    "PassManager.cpp",
+    "PassRegistry.cpp",
+    "PassTimingInfo.cpp",
+    "ProfileSummary.cpp",
+    "SafepointIRVerifier.cpp",
+    "Statepoint.cpp",
+    "Type.cpp",
+    "TypeFinder.cpp",
+    "Use.cpp",
+    "User.cpp",
+    "Value.cpp",
+    "ValueSymbolTable.cpp",
+    "Verifier.cpp",
+  ]
+}

llvm/utils/gn/secondary/llvm/utils/TableGen/tablegen.gni

+# This file defines a template for running llvm-tblgen.
+#
+# Parameters:
+#
+#   args (required)
+#       [list of strings] Flags to pass to llvm-tblgen.
+#
+#   output_name (optional)
+#       Basename of the generated output file.
+#       Defaults to target name with ".inc" appended.
+#
+#   td_file (roptional)
+#       The .td file to pass to llvm-tblgen.
+#       Defaults to target name with ".td" appended.
+#
+#   visibility (optional)
+#       GN's regular visibility attribute, see `gn help visibility`.
+#
+# Example use:
+#
+#   tablegen("attributes_compat_func_gen") {
+#     visibility = [ ":IR" ]
+#     args = [ "-gen-attrs" ]
+#     td_file = "AttributesCompatFunc.td"
+#   }
+
+template("tablegen") {
+  assert(defined(invoker.args), "args must be defined for $target_name")
+
+  config_name = "${target_name}_config"
+  config(config_name) {
+    visibility = [ ":$target_name" ]
+    include_dirs = [ target_gen_dir ]
+  }
+
+  action(target_name) {
+    forward_variables_from(invoker, [ "visibility" ])
+
+    # FIXME: In cross builds, this should depend on the host binary.
+    tblgen_target = "//llvm/utils/TableGen:llvm-tblgen"
+    tblgen_executable = get_label_info(tblgen_target, "root_out_dir") +
+                        "/bin/" + get_label_info(tblgen_target, "name")
+    deps = [
+      tblgen_target,
+    ]
+    if (defined(invoker.td_file)) {
+      td_file = invoker.td_file
+    } else {
+      td_file = "$target_name.td"
+    }
+    sources = [
+      td_file,
+    ]
+    script = "//llvm/utils/gn/build/run_tablegen.py"
+    if (defined(invoker.output_name)) {
+      gen_output = "$target_gen_dir/" + invoker.output_name
+    } else {
+      gen_output = "$target_gen_dir/$target_name.inc"
+    }
+    depfile = "$gen_output.d"
+    td_file = rebase_path(td_file, root_build_dir)
+    args = [
+             rebase_path(tblgen_executable, root_build_dir),
+             "-I",
+             rebase_path("//llvm/include", root_build_dir),
+
+             # FIXME: Rather than duplicating this -I flag in both the CMake and
+             # the gn build, let tablegen add input dir to include search path
+             # instead?
+             "-I",
+             get_path_info(td_file, "dir"),
+             "-d",
+             rebase_path(depfile, root_build_dir),
+             "-o",
+             rebase_path(gen_output, root_build_dir),
+             td_file,
+           ] + invoker.args
+    outputs = [
+      gen_output,
+    ]
+
+    # Let targets depending on this find the generated file.
+    public_configs = [ ":$config_name" ]
+  }
+}