[Bazel] Make td_library usable as data

[GMNGeoffrey]

Summary

This patch makes it possible to list a td_library as a rule's data
attribute and get its source files and all its transitive dependencies
at runtime. This is useful for, e.g. shell tests running tblgen.

Note that this is a bit different from how a "normal" (e.g. C++) library
rule would work because those have actual library outputs and the
td_library rule just bundles some source files and includes. If someone
wanted to make use of the includes, they would have to access the TdInfo
provider, but this keeps simple things simple.

[jpienaar]

utils/bazel/llvm-project-overlay/mlir/tblgen.bzl

105

"to produce on the output" is difficult to parse for me

267

High-level comment here too?

[GMNGeoffrey]

• Fix typo

utils/bazel/llvm-project-overlay/mlir/tblgen.bzl

105

That's because it makes no sense :-D Fixed

267

I was less sure on this one since here we're gathering the files that gentbl_test itself needs to execute, which is much more normal. The gathering of runfiles from dependencies is just following the advice at https://docs.bazel.build/versions/main/skylark/rules.html#runfiles.

[GMNGeoffrey]

Rebase on main

[GMNGeoffrey]

Ping on this? Any concerns? I'm happy to add a comment if you'd like Jacques :-D

[jpienaar]

Works for me, thanks!

utils/bazel/llvm-project-overlay/mlir/tblgen.bzl

def _resolve_includes(ctx, includes):
     return resolved_includes
 
 def _td_library_impl(ctx):
     trans_srcs = _get_transitive_srcs(ctx.files.srcs, ctx.attr.deps)
     trans_includes = _get_transitive_includes(
         _resolve_includes(ctx, ctx.attr.includes),
         ctx.attr.deps,
     )
+
+    # Note that we include srcs in runfiles. A td_library doesn't compile to
+    # produce an output: it's just a depset of source files and include
+    # directories. So if it is needed for execution of some rule (likely
+    # something running tblgen as a test action), the files needed are the same
+    # as the source files.
+    # Note: not using merge_all, as that is not available in Bazel 4.0
+    runfiles = ctx.runfiles(ctx.files.srcs)
+    for src in ctx.attr.srcs:
+        runfiles = runfiles.merge(src[DefaultInfo].default_runfiles)
+    for dep in ctx.attr.deps:
+        runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)
+
     return [
-        DefaultInfo(files = trans_srcs),
+        DefaultInfo(files = trans_srcs, runfiles = runfiles),
         TdInfo(
             transitive_sources = trans_srcs,
             transitive_includes = trans_includes,
         ),
     ]
 
 td_library = rule(
     _td_library_impl,
attrs = {
         ),
     },
 )
 
 # TODO(gcmn): Figure out how to reduce duplication with _gentbl_rule_impl
 def _gentbl_test_impl(ctx):
     td_file = ctx.file.td_file
 
-    trans_srcs = _get_transitive_srcs(
-        ctx.files.td_srcs + [td_file],
-        ctx.attr.deps,
-    )
-
     # Note that we have two types of includes here. The deprecated ones expanded
     # only by "_prefix_roots" are already relative to the execution root, i.e.
     # may contain an `external/<workspace_name>` prefix if the current workspace
     # is not the main workspace (where workspace_name is something configured
     # per-project and therefore generally not known). Note that dirname also
     # already includes this prefix. The new style of includes have it prepended
     # automatically by `_resolve_includes` to avoid BUILD files having to depend
     # on project specific configurations and Bazel implementation details.
def _gentbl_test_impl(ctx):
     test_args.extend(["-o", "/dev/null"])
 
     ctx.actions.write(
         ctx.outputs.executable,
         content = " ".join(test_args),
         is_executable = True,
     )
 
+    # Note: not using merge_all, as that is not available in Bazel 4.0
+    runfiles = ctx.runfiles(
+        files = [ctx.executable.tblgen],
+        transitive_files = _get_transitive_srcs(
+            ctx.files.td_srcs + [td_file],
+            ctx.attr.deps,
+        ),
+    )
+    for src in ctx.attr.td_srcs:
+        runfiles = runfiles.merge(src[DefaultInfo].default_runfiles)
+    for dep in ctx.attr.deps:
+        runfiles = runfiles.merge(dep[DefaultInfo].default_runfiles)
+
     return [
         coverage_common.instrumented_files_info(
             ctx,
             source_attributes = ["td_file", "td_srcs"],
             dependency_attributes = ["tblgen", "deps"],
         ),
-        DefaultInfo(
-            runfiles = ctx.runfiles(
-                [ctx.executable.tblgen],
-                transitive_files = trans_srcs,
-            ),
-        ),
+        DefaultInfo(runfiles = runfiles),
     ]
 
 gentbl_test = rule(
     _gentbl_test_impl,
     test = True,
     doc = "A shell test that tests the given TablegGen invocation. Note" +
           " that unlike gentbl_rule, this builds and invokes `tblgen` in the" +
           " target configuration. Takes all the same arguments as gentbl_rule" +