Simplify setting up LLVM as bazel external repo

[csigg]

Summary

Only require one intermediate repository instead of two.
Fewer parameters in llvm_config.
Remove bazel_skylib dependency.

[csigg]

Remove no longer needed attr.

[csigg]

Update WORKSPACE file.

[csigg]

Add back bazel_skylib, it's needed by the tests.

[goncharov]

Hi @csigg ! LGTM, and tests also pass. I don't have much experience with bazel though :)

[GMNGeoffrey]

Nice! This is much better. I think some of the workspace munging came from when this rule was more general (overlay any directory over any other). I still think such functionality is useful, but not at the cost of making this API really hard to use.

A few nits though. Sorry I was out and couldn't review :-)

utils/bazel/configure.bzl

36–40

I think this could just be repository_ctx.path(".")

utils/bazel/examples/http_archive/WORKSPACE

12

We should update the commit here so it's one that includes this change. For future reference, I think it would've been better to leave this example as-is, since it is still a working example for the given LLVM commit and then follow up with an update for your commit (since it's impossible to know the commit hash before it's committed)

13

bazel_skylib is still a required dependency for tblgen, although we could go back to manual path manipulation to avoid the dependency

[csigg]

Undo example changes, they should be submitted separately.

[csigg]

Revert back to repository_ctx.path(Label("//:WORKSPACE")).dirname.

repository_ctx.path(".") simply resolves to ".".
Adding .realpath doesn't help either, because that then resolves to the root of the repository we are creating (e.g. llvm-project), instead of the source repo (e.g. llvm-archive).

[GMNGeoffrey]

I actually just tested this further and it breaks resolution of deps_impl. This is yet another case where Bazel makes it really hard to understand what labels are being resolved relative to...

~/src/llvm-project/utils/bazel/examples/http_archive$ bazel build @llvm-project//llvm:llvm-symbolizer
INFO: Invocation ID: b5fb0254-073a-4ef5-b9f3-6dcaad9ac98f
INFO: Repository llvm_zlib instantiated at:
  /usr/local/google/home/gcmn/src/llvm-project/utils/bazel/examples/http_archive/WORKSPACE:45:35: in <toplevel>
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/configure.bzl:93:10: in llvm_disable_optional_support_deps
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule llvm_zlib_disable defined at:
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl:72:36: in <toplevel>
INFO: Repository llvm_terminfo instantiated at:
  /usr/local/google/home/gcmn/src/llvm-project/utils/bazel/examples/http_archive/WORKSPACE:45:35: in <toplevel>
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/configure.bzl:98:10: in llvm_disable_optional_support_deps
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule llvm_terminfo_disable defined at:
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/terminfo.bzl:35:40: in <toplevel>
ERROR: An error occurred during the fetch of repository 'llvm_zlib':
   Traceback (most recent call last):
        File "/usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl", line 66, column 28, in _llvm_zlib_disable_impl
                repository_ctx.template(
Error in template: Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: Error fetching repository: Traceback (most recent call last):
        File "/usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl", line 66, column 28, in _llvm_zlib_disable_impl
                repository_ctx.template(
Error in template: Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-project/llvm/BUILD.bazel:170:11: @llvm-project//llvm:Support depends on @llvm_zlib//:zlib in repository @llvm_zlib which failed to fetch. no such package '@llvm_zlib//': Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: Analysis of target '@llvm-project//llvm:llvm-symbolizer' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.309s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (15 packages loaded, 1015 targets configured)

utils/bazel/WORKSPACE

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
 
-SKYLIB_VERSION = "1.0.3"
-
-http_archive(
-    name = "bazel_skylib",
-    sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-    ],
+new_local_repository(
+    name = "llvm-archive",
+    path = "../../",
+    build_file_content = "# empty",
 )
 
-load(":configure.bzl", "llvm_configure")
+load("@llvm-archive//utils/bazel:configure.bzl", "llvm_configure")
 
-llvm_configure(
-    name = "llvm-project",
-    overlay_path = "llvm-project-overlay",
-    src_path = "../..",
-)
+llvm_configure(name = "llvm-project")
 
 load(":terminfo.bzl", "llvm_terminfo_from_env")
 
 maybe(
     llvm_terminfo_from_env,
     name = "llvm_terminfo",
 )
 

utils/bazel/configure.bzl

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 """Helper macros to configure the LLVM overlay project."""
 
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
-load("@bazel_skylib//lib:paths.bzl", "paths")
 load(":zlib.bzl", "llvm_zlib_disable", "llvm_zlib_system")
 load(":terminfo.bzl", "llvm_terminfo_disable", "llvm_terminfo_system")
 
 # Directory of overlay files relative to WORKSPACE
 DEFAULT_OVERLAY_PATH = "llvm-project-overlay"
 
 DEFAULT_TARGETS = [
     "AArch64",
DEFAULT_TARGETS = [
     "Sparc",
     "SystemZ",
     "WebAssembly",
     "X86",
     "XCore",
 ]
 
 def _overlay_directories(repository_ctx):
-    src_workspace_path = str(repository_ctx.path(
-        repository_ctx.attr.src_workspace,
-    ).dirname)
-
-    src_path = paths.join(src_workspace_path, repository_ctx.attr.src_path)
-
-    overlay_workspace_path = str(repository_ctx.path(
-        repository_ctx.attr.overlay_workspace,
-    ).dirname)
-    overlay_path = paths.join(
-        overlay_workspace_path,
-        repository_ctx.attr.overlay_path,
-    )
 
-    overlay_script = repository_ctx.path(
-        repository_ctx.attr._overlay_script,
-    )
+    src_path = repository_ctx.path(Label("//:WORKSPACE")).dirname
+    bazel_path = src_path.get_child("utils").get_child("bazel")
+    overlay_path = bazel_path.get_child("llvm-project-overlay")
+    script_path = bazel_path.get_child("overlay_directories.py")
+
     python_bin = repository_ctx.which("python3")
     if not python_bin:
         # Windows typically just defines "python" as python3. The script itself
         # contains a check to ensure python3.
         python_bin = repository_ctx.which("python")
 
     if not python_bin:
         fail("Failed to find python3 binary")
 
     cmd = [
         python_bin,
-        overlay_script,
+        script_path,
         "--src",
         src_path,
         "--overlay",
         overlay_path,
         "--target",
         ".",
     ]
     exec_result = repository_ctx.execute(cmd, timeout = 20)
repository_ctx.file(
         executable = False,
     )
 
 llvm_configure = repository_rule(
     implementation = _llvm_configure_impl,
     local = True,
     configure = True,
     attrs = {
-        "_overlay_script": attr.label(
-            default = Label("//:overlay_directories.py"),
-            allow_single_file = True,
-        ),
-        "overlay_workspace": attr.label(default = Label("//:WORKSPACE")),
-        "overlay_path": attr.string(default = DEFAULT_OVERLAY_PATH),
-        "src_workspace": attr.label(default = Label("//:WORKSPACE")),
-        "src_path": attr.string(mandatory = True),
         "targets": attr.string_list(default = DEFAULT_TARGETS),
     },
 )
 
 def llvm_disable_optional_support_deps():
     maybe(
         llvm_zlib_disable,
         name = "llvm_zlib",

utils/bazel/examples/http_archive/WORKSPACE

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 """An example WORKSPACE for configuring LLVM using http_archive."""
 
 workspace(name = "http_archive_example")
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
-SKYLIB_VERSION = "1.0.3"
-
-http_archive(
-    name = "bazel_skylib",
-    sha256 = "97e70364e9249702246c0e9444bccdc4b847bed1eb03c5a3ece4f83dfe6abc44",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-    ],
-)
-
 # Replace with the LLVM commit you want to use.
 LLVM_COMMIT = "09ac97ce350316b95b8cddb796d52f71b6f68296"
 
 # The easiest way to calculate this for a new commit is to set it to empty and
 # then run a bazel build and it will report the digest necessary to cache the
 # archive and make the build reproducible.
 LLVM_SHA256 = "2fb1aa06d12f8db349a27426cb0ced062987c5c2a75143c69f4284929e2750ff"
 
-# FIXME: It shouldn't be necessary to use http_archive twice here. Caching
-# should mean that this isn't too expensive though.
-
 http_archive(
-    name = "llvm-project-raw",
-    build_file_content = "#empty",
+    name = "llvm-archive",
+    build_file_content = "# empty",
     sha256 = LLVM_SHA256,
     strip_prefix = "llvm-project-" + LLVM_COMMIT,
     urls = ["https://github.com/llvm/llvm-project/archive/{commit}.tar.gz".format(commit = LLVM_COMMIT)],
 )
 
-http_archive(
-    name = "llvm-bazel",
-    sha256 = LLVM_SHA256,
-    strip_prefix = "llvm-project-{}/utils/bazel".format(LLVM_COMMIT),
-    urls = ["https://github.com/llvm/llvm-project/archive/{commit}.tar.gz".format(commit = LLVM_COMMIT)],
-)
-
-load("@llvm-bazel//:configure.bzl", "llvm_configure", "llvm_disable_optional_support_deps")
+load("@llvm-archive//utils/bazel:configure.bzl", "llvm_configure")
 
 llvm_configure(
     name = "llvm-project",
-    src_path = ".",
-    src_workspace = "@llvm-project-raw//:WORKSPACE",
 )
 
 # Disables optional dependencies for Support like zlib and terminfo. You may
 # instead want to configure them using the macros in the corresponding bzl
 # files.
 llvm_disable_optional_support_deps()

utils/bazel/examples/submodule/WORKSPACE

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 """An example WORKSPACE for configuring LLVM using a git submodule."""
 
 workspace(name = "submodule_example")
 
-SKYLIB_VERSION = "1.0.3"
-
-http_archive(
-    name = "bazel_skylib",
-    sha256 = "97e70364e9249702246c0e9444bccdc4b847bed1eb03c5a3ece4f83dfe6abc44",
-    urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-    ],
-)
-
+new_local_repository(
+    name = "llvm-archive",
+    build_file_content = "# empty",
     # Or wherever your submodule is located.
-SUBMODULE_PATH = "third_party/llvm-project"
-
-local_repository(
-    name = "llvm-bazel",
-    path = SUBMODULE_PATH + "/utils/bazel",
+    path = "third_party/llvm-project",
 )
 
-load("@llvm-bazel//:configure.bzl", "llvm_configure", "llvm_disable_optional_support_deps")
+load("@llvm-archive//utils/bazel:configure.bzl", "llvm_configure")
 
-llvm_configure(
-    name = "llvm-project",
-    src_path = SUBMODULE_PATH,
-    src_workspace = "@submodule_example//:WORKSPACE",
-)
+llvm_configure(name = "llvm-project")
 
 # Disables optional dependencies for Support like zlib and terminfo. You may
 # instead want to configure them using the macros in the corresponding bzl
 # files.
 llvm_disable_optional_support_deps()