[Bazel] Add bzlmod support

[aaronmondal]

Summary

Introduces a build configuration for bzlmod.

The version introduced in this module is importable in MODULE.bazel files via

python
bazel_dep(name="llvm-project-overlay", version="16.0.0-bcr.0")

llvm_project_overlay = use_extension(
    "@llvm-project-overlay//utils/bazel:extensions.bzl",
    "llvm_project_overlay",
)

use_repo(llvm_project_overlay, "llvm-project")

Optionally, users may specify custom commits, targets and patches by overriding llvm_project_overlay.configure:

python
llvm_project_overlay.configure(
    targets = ["X86"],
    commit = "<some_commit>",
    sha256 = "<sha_of_commit_archive>",
    patches = ["<some_patch>"],
)

[aaronmondal]

Continuation of https://reviews.llvm.org/D137007.

[aaronmondal]

A few notes:

The --registry=https://github.com/eomii/bazel-eomii-registry/commit/04d3d96f0fa35e38b056853db3bae8dd903a4bbd line in the example is only meant for testing and should not be part of the final version of this diff.

I "delayed" the custom commit example to https://reviews.llvm.org/D137009 since it will require an LLVM commit that contains the changes in this commit.

The`utils/bazel/MODULE.bazel` file is the file used if someone (most likely a CI system) would run a bzlmod build from within a cloned version of LLVM. The MODULE.bazel file that most users will actually interact with is the one patched in by bzlmod_compatibility_patch.patch.

Something I'm still figuring out is how to add WORKSPACE, MODULE.bazel and BUILD.bazel dynamically when building in-tree. When using the http_archive variant the files can just be patched in, and bazel registries will have to contain a copy of bzlmod_compatibility_patch.patch anyways (unless we add these files to the real top level llvm-project directory, which is probably not possible). When building from within utils/bazel, patching may hurt reproducibility as @fmeum mentioned in a previous discussion. I'll investigate this further.

[aaronmondal]

Remove that hacky patch. It is not necessary and will usually be supplied by a bazel registry.

Also fixes the logic to determine whether or not to use a custom commit or the default sources.

[matts1]

utils/bazel/extensions.bzl

35

FYI, since I wasn't aware of your implementation when I was working on this yesterday, I wrote my own implementation. I'll add the code below, so feel free to steal whatever seems good from it.

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_file")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "new_git_repository")
load("//repos:non_bcr_deps.bzl", "non_bcr_deps")
load("//:configure.bzl", "llvm_disable_optional_support_deps", "llvm_use_system_support_deps", _llvm_configure_repo = "llvm_configure")
load("//:terminfo.bzl", "llvm_terminfo_disable", "llvm_terminfo_system")
load("//:vulkan_sdk.bzl", "vulkan_sdk_setup")
load("//:zlib.bzl", "llvm_zlib_disable", "llvm_zlib_external", "llvm_zlib_system")

_ZLIB_NAME = "llvm_zlib"
_TERMINFO_NAME = "llvm_terminfo"

def _llvm_configure_impl(module_ctx):
    root = module_ctx.modules[0]
    tag = root.tags.configure[0]

    non_bcr_deps()

    if tag.zlib_strategy == "external":
        llvm_zlib_external(name = _ZLIB_NAME, external_zlib = "@zlib")
    elif tag.zlib_strategy == "system":
        llvm_zlib_system(name = _ZLIB_NAME)
    elif tag.zlib_strategy == "disabled":
        llvm_zlib_disable(name = _ZLIB_NAME)
    else:
        fail("zlib_strategy must be one of 'external', 'system', and 'disabled'")

    if tag.terminfo_strategy == "system":
        llvm_terminfo_system(name = _TERMINFO_NAME)
    elif tag.terminfo_strategy == "disabled":
        llvm_terminfo_disable(name = _TERMINFO_NAME)
    else:
        fail("terminfo_strategy must be one of 'disabled' and 'system'")

    vulkan_sdk_setup(name = "vulkan_sdk")

    _llvm_configure_repo(name = tag.name)

_llvm_configure_tag = tag_class(attrs = dict(
    name = attr.string(mandatory = True),
    zlib_strategy = attr.string(mandatory = True, default = "disabled"),
    terminfo_strategy = attr.string(default = "disabled"),
))

llvm_configure = module_extension(
    implementation = _llvm_configure_impl,
    tag_classes = dict(configure = _llvm_configure_tag),
)

[matts1]

utils/bazel/examples/bzlmod/.bazelrc

5

I'll add a comment here so you don't forget to delete before submitting

[aaronmondal]

utils/bazel/extensions.bzl

35

Thanks a lot for this! After the changes to zlib we con't need the custom zlib logic anymore, and if we remove terminfo from the external deps entirely we can also get rid of that. I feel like your patch at https://reviews.llvm.org/D150641 was the missing piece to make all of this fall into place quite nicely :) I propose that we first remove terminfo, then get you patch in and then move forward with the patch here to finally add bzlmod support natively.

utils/bazel/MODULE.bazel

+module(
+    name = "llvm-project-overlay",
+    version = "16.0.0-bcr.0",
+    compatibility_level = 0,
+)
+
+bazel_dep(name = "bazel_skylib", version = "1.3.0")
+bazel_dep(name = "platforms", version = "0.0.4")
+bazel_dep(name = "rules_cc", version = "0.0.2")
+bazel_dep(name = "zlib", version = "1.2.12")
+
+llvm_project_overlay = use_extension(
+    "//:extensions.bzl",
+    "llvm_project_overlay",
+)
+
+llvm_project_overlay.configure(
+    targets = [
+        "AArch64",
+        "AMDGPU",
+        "ARM",
+        "AVR",
+        "BPF",
+        "Hexagon",
+        "Lanai",
+        "Mips",
+        "MSP430",
+        "NVPTX",
+        "PowerPC",
+        "RISCV",
+        "Sparc",
+        "SystemZ",
+        "VE",
+        "WebAssembly",
+        "X86",
+        "XCore",
+    ],
+)
+
+use_repo(
+    llvm_project_overlay,
+    "llvm-project",
+    "llvm-raw",
+)

utils/bazel/bzlmod_compatibility_patch.patch

+diff --git a/BUILD.bazel b/BUILD.bazel
+new file mode 100755
+index 000000000000..996cee9ad835
+--- /dev/null
++++ b/BUILD.bazel
+@@ -0,0 +1 @@
++# Empty.
+diff --git a/MODULE.bazel b/MODULE.bazel
+new file mode 100644
+index 000000000000..d2c45773034f
+--- /dev/null
++++ b/MODULE.bazel
+@@ -0,0 +1,23 @@
++module(
++    name = "llvm-project-overlay",
++    version = "16.0.0-bcr.0",
++    compatibility_level = 0,
++)
++
++bazel_dep(name = "bazel_skylib", version = "1.3.0")
++bazel_dep(name = "platforms", version = "0.0.4")
++bazel_dep(name = "rules_cc", version = "0.0.2")
++bazel_dep(name = "zlib", version = "1.2.12")
++
++llvm_project_overlay = use_extension(
++    "//utils/bazel:extensions.bzl",
++    "llvm_project_overlay",
++)
++
++llvm_project_overlay.configure()
++
++use_repo(
++    llvm_project_overlay,
++    "llvm-project",
++    "llvm-raw",
++)
+diff --git a/WORKSPACE b/WORKSPACE
+new file mode 100755
+index 000000000000..996cee9ad835
+--- /dev/null
++++ b/WORKSPACE
+@@ -0,0 +1 @@
++# Empty.

utils/bazel/examples/bzlmod/.bazelrc

+common --repo_env=BAZEL_CXXOPTS='-std=c++17:-O3'
+common --experimental_enable_bzlmod
+
+# TODO(aaronmondal): Only for testing during review. Remove before merging this patch.
+common --registry=https://raw.githubusercontent.com/eomii/bazel-eomii-registry/llvm-project-overlay

utils/bazel/examples/bzlmod/.bazelversion

+5.3.2

utils/bazel/examples/bzlmod/MODULE.bazel

+bazel_dep(name = "llvm-project-overlay", version = "16.0.0-bcr.0")
+
+llvm_project_overlay = use_extension(
+    "@llvm-project-overlay//utils/bazel:extensions.bzl",
+    "llvm_project_overlay",
+)
+
+use_repo(llvm_project_overlay, "llvm-project")

utils/bazel/examples/bzlmod/WORKSPACE.bazel

+# Empty.

utils/bazel/extensions.bzl

+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+load(":vulkan_sdk.bzl", "vulkan_sdk_setup")
+load(":configure.bzl", "llvm_configure", "DEFAULT_TARGETS")
+load(":terminfo.bzl", "llvm_terminfo_disable")
+load(":zlib.bzl", "llvm_zlib_external")
+
+def _llvm_in_tree_sources_impl(repository_ctx):
+    """Create a non-overlaid version of the llvm-project repository.
+
+    Adds MODULE.bazel, WORKSPACE.bazel and BUILD.bazel to the top level
+    directory, similar to `native.local_repository` which cannot be called from
+    within a module extension implementation.
+    """
+    repo_name = repository_ctx.path(
+        Label("@llvm-project-overlay//:MODULE.bazel"),
+    ).dirname.basename
+
+    # If we are building from within the /utils/bazel subdirectory in the LLVM
+    # repo, symlink to "llvm-raw". Otherwise, keep using the dynamically
+    # determined llvm-project-overlay module version and workspace to get the
+    # correct path to the sources.
+    if repo_name == "bazel":
+        repo_name = "llvm-raw"
+
+    repository_ctx.symlink("../../external/{}".format(repo_name), "")
+
+    # repository_ctx.file("WORKSPACE", content = "# Empty.",)
+
+    # repository_ctx.file("BUILD.bazel", content = "# Empty.")
+
+
+llvm_in_tree_sources = repository_rule(
+    implementation = _llvm_in_tree_sources_impl,
+    attrs = {
+        "patches": attr.label_list(),
+    },
+)
+
+def _llvm_configure_extension_impl(ctx):
+    targets = []
+    patches = []
+    commit = ""
+    sha256 = ""
+
+    for module in ctx.modules:
+
+        # Aggregate targets and patches across imports.
+        for config in module.tags.configure:
+            for target in config.targets:
+                if target not in targets:
+                    targets.append(target)
+            for patch in config.patches:
+                if patch not in patches:
+                    patches.append(patch)
+
+        # Use the commit/sha configuration from the top-level import.
+        if module.tags.configure != []:
+            commit = module.tags.configure[0].commit
+            sha256 = module.tags.configure[0].sha256
+
+    if commit == "":
+        if patches != []:
+            fail("""Cannot apply patches when `commit` is unspecified. Patches
+                 would modify the original LLVM sources, making the build
+                 irreproducible.
+                 """,
+            )
+        llvm_in_tree_sources(name = "llvm-raw")
+    else:
+        http_archive(
+            name = "llvm-raw",
+            build_file_content = "# Empty.",
+            sha256 = sha256,
+            strip_prefix = "llvm-project-" + commit,
+            urls = [
+                "https://github.com/llvm/llvm-project/archive/{}.tar.gz".format(
+                    commit,
+                ),
+            ],
+            patches = patches + [
+                ":bzlmod_compatibility_patch.patch",
+            ],
+            patch_args = ["-p1"],
+        )
+
+    # Fall back to the default targets if all configurations of this extension
+    # omit the `target` attribute.
+    if targets == []:
+        targets = DEFAULT_TARGETS
+
+    llvm_configure(name = "llvm-project", targets = targets)
+
+    llvm_zlib_external(name = "llvm_zlib", external_zlib = "@zlib")
+
+    llvm_terminfo_disable(name = "llvm_terminfo")
+
+    http_archive(
+        name = "vulkan_headers",
+        build_file = "@llvm-raw//utils/bazel/third_party_build:vulkan_headers.BUILD",
+        sha256 = "19f491784ef0bc73caff877d11c96a48b946b5a1c805079d9006e3fbaa5c1895",
+        strip_prefix = "Vulkan-Headers-9bd3f561bcee3f01d22912de10bb07ce4e23d378",
+        urls = [
+            "https://github.com/KhronosGroup/Vulkan-Headers/archive/9bd3f561bcee3f01d22912de10bb07ce4e23d378.tar.gz",
+        ],
+    )
+
+    vulkan_sdk_setup(name = "vulkan_sdk")
+
+llvm_project_overlay = module_extension(
+    doc = """Configure the llvm-project.
+
+    Tags:
+        targets: List of targets which Clang should support.
+        commit: An optional LLVM commit. If specified, the LLVM project will be
+            redownloaded from that commit (i.e.) it is downloaded once for the
+            bzlmod dependency and once for the specified commit.
+            Defaults to None, in which case the sources specified in the BCR
+            module are used.
+        sha256: Hash for verifying the custom archive if `commit` is not `None`.
+        patches: Optional list of patches to apply to the LLVM sources. May not
+            be used if `commit` is unspecified.
+            """,
+    implementation = _llvm_configure_extension_impl,
+    tag_classes = {
+        "configure": tag_class(
+            attrs = {
+                "commit": attr.string(),
+                "sha256": attr.string(),
+                "targets": attr.string_list(),
+                "patches": attr.label_list(),
+            },
+        ),
+    },
+)