Simplify setting up LLVM as bazel external repo

[csigg]

Summary

Only require one intermediate repository instead of two.
Fewer parameters in llvm_config.
Remove bazel_skylib dependency.

[csigg]

Remove no longer needed attr.

[csigg]

Update WORKSPACE file.

[csigg]

Add back bazel_skylib, it's needed by the tests.

[goncharov]

Hi @csigg ! LGTM, and tests also pass. I don't have much experience with bazel though :)

[GMNGeoffrey]

Nice! This is much better. I think some of the workspace munging came from when this rule was more general (overlay any directory over any other). I still think such functionality is useful, but not at the cost of making this API really hard to use.

A few nits though. Sorry I was out and couldn't review :-)

utils/bazel/configure.bzl

39–40

I think this could just be repository_ctx.path(".")

utils/bazel/examples/http_archive/WORKSPACE

13

bazel_skylib is still a required dependency for tblgen, although we could go back to manual path manipulation to avoid the dependency

23

We should update the commit here so it's one that includes this change. For future reference, I think it would've been better to leave this example as-is, since it is still a working example for the given LLVM commit and then follow up with an update for your commit (since it's impossible to know the commit hash before it's committed)

[csigg]

Undo example changes, they should be submitted separately.

[csigg]

Revert back to repository_ctx.path(Label("//:WORKSPACE")).dirname.

repository_ctx.path(".") simply resolves to ".".
Adding .realpath doesn't help either, because that then resolves to the root of the repository we are creating (e.g. llvm-project), instead of the source repo (e.g. llvm-archive).

[GMNGeoffrey]

I actually just tested this further and it breaks resolution of deps_impl. This is yet another case where Bazel makes it really hard to understand what labels are being resolved relative to...

~/src/llvm-project/utils/bazel/examples/http_archive$ bazel build @llvm-project//llvm:llvm-symbolizer
INFO: Invocation ID: b5fb0254-073a-4ef5-b9f3-6dcaad9ac98f
INFO: Repository llvm_zlib instantiated at:
  /usr/local/google/home/gcmn/src/llvm-project/utils/bazel/examples/http_archive/WORKSPACE:45:35: in <toplevel>
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/configure.bzl:93:10: in llvm_disable_optional_support_deps
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule llvm_zlib_disable defined at:
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl:72:36: in <toplevel>
INFO: Repository llvm_terminfo instantiated at:
  /usr/local/google/home/gcmn/src/llvm-project/utils/bazel/examples/http_archive/WORKSPACE:45:35: in <toplevel>
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/configure.bzl:98:10: in llvm_disable_optional_support_deps
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule llvm_terminfo_disable defined at:
  /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/terminfo.bzl:35:40: in <toplevel>
ERROR: An error occurred during the fetch of repository 'llvm_zlib':
   Traceback (most recent call last):
        File "/usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl", line 66, column 28, in _llvm_zlib_disable_impl
                repository_ctx.template(
Error in template: Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: Error fetching repository: Traceback (most recent call last):
        File "/usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-archive/utils/bazel/zlib.bzl", line 66, column 28, in _llvm_zlib_disable_impl
                repository_ctx.template(
Error in template: Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: /usr/local/google/home/gcmn/.cache/bazel/_bazel_gcmn/862d4f35edba4a3b7a8ffb6f5ccce305/external/llvm-project/llvm/BUILD.bazel:170:11: @llvm-project//llvm:Support depends on @llvm_zlib//:zlib in repository @llvm_zlib which failed to fetch. no such package '@llvm_zlib//': Unable to load package for @llvm-archive//deps_impl:zlib_disable.BUILD: BUILD file not found in directory 'deps_impl' of external repository @llvm-archive. Add a BUILD file to a directory to mark it as a package.
ERROR: Analysis of target '@llvm-project//llvm:llvm-symbolizer' failed; build aborted: Analysis failed
INFO: Elapsed time: 3.309s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (15 packages loaded, 1015 targets configured)

utils/bazel/WORKSPACE

http_archive(
     name = "bazel_skylib",
     sha256 = "1c531376ac7e5a180e0237938a2536de0c54d93f5c278634818e0efc952dd56c",
     urls = [
         "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
         "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
     ],
 )
 
-load(":configure.bzl", "llvm_configure")
-
-llvm_configure(
-    name = "llvm-project",
-    overlay_path = "llvm-project-overlay",
-    src_path = "../..",
+new_local_repository(
+    name = "llvm-archive",
+    path = "../../",
+    build_file_content = "# empty",
 )
 
+load("@llvm-archive//utils/bazel:configure.bzl", "llvm_configure")
+
+llvm_configure(name = "llvm-project")
+
 load(":terminfo.bzl", "llvm_terminfo_from_env")
 
 maybe(
     llvm_terminfo_from_env,
     name = "llvm_terminfo",
 )
 
 maybe(

utils/bazel/configure.bzl

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 """Helper macros to configure the LLVM overlay project."""
 
 load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
-load("@bazel_skylib//lib:paths.bzl", "paths")
 load(":zlib.bzl", "llvm_zlib_disable", "llvm_zlib_system")
 load(":terminfo.bzl", "llvm_terminfo_disable", "llvm_terminfo_system")
 
 # Directory of overlay files relative to WORKSPACE
 DEFAULT_OVERLAY_PATH = "llvm-project-overlay"
 
 DEFAULT_TARGETS = [
     "AArch64",
DEFAULT_TARGETS = [
     "Sparc",
     "SystemZ",
     "WebAssembly",
     "X86",
     "XCore",
 ]
 
 def _overlay_directories(repository_ctx):
-    src_workspace_path = str(repository_ctx.path(
-        repository_ctx.attr.src_workspace,
-    ).dirname)
-
-    src_path = paths.join(src_workspace_path, repository_ctx.attr.src_path)
-
-    overlay_workspace_path = str(repository_ctx.path(
-        repository_ctx.attr.overlay_workspace,
-    ).dirname)
-    overlay_path = paths.join(
-        overlay_workspace_path,
-        repository_ctx.attr.overlay_path,
-    )
+    src_path = repository_ctx.path(Label("//:WORKSPACE")).dirname
+    bazel_path = src_path.get_child("utils").get_child("bazel")
+    overlay_path = bazel_path.get_child("llvm-project-overlay")
+    script_path = bazel_path.get_child("overlay_directories.py")
 
-    overlay_script = repository_ctx.path(
-        repository_ctx.attr._overlay_script,
-    )
     python_bin = repository_ctx.which("python3")
     if not python_bin:
         # Windows typically just defines "python" as python3. The script itself
         # contains a check to ensure python3.
         python_bin = repository_ctx.which("python")
 
     if not python_bin:
         fail("Failed to find python3 binary")
 
     cmd = [
         python_bin,
-        overlay_script,
+        script_path,
         "--src",
         src_path,
         "--overlay",
         overlay_path,
         "--target",
         ".",
     ]
     exec_result = repository_ctx.execute(cmd, timeout = 20)
repository_ctx.file(
         executable = False,
     )
 
 llvm_configure = repository_rule(
     implementation = _llvm_configure_impl,
     local = True,
     configure = True,
     attrs = {
-        "_overlay_script": attr.label(
-            default = Label("//:overlay_directories.py"),
-            allow_single_file = True,
-        ),
-        "overlay_workspace": attr.label(default = Label("//:WORKSPACE")),
-        "overlay_path": attr.string(default = DEFAULT_OVERLAY_PATH),
-        "src_workspace": attr.label(default = Label("//:WORKSPACE")),
-        "src_path": attr.string(mandatory = True),
         "targets": attr.string_list(default = DEFAULT_TARGETS),
     },
 )
 
 def llvm_disable_optional_support_deps():
     maybe(
         llvm_zlib_disable,
         name = "llvm_zlib",

utils/bazel/examples/http_archive/WORKSPACE

 # This file is licensed under the Apache License v2.0 with LLVM Exceptions.
 # See https://llvm.org/LICENSE.txt for license information.
 # SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
 
 """An example WORKSPACE for configuring LLVM using http_archive."""
 
 workspace(name = "http_archive_example")
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
 
 SKYLIB_VERSION = "1.0.3"
 
 http_archive(
     name = "bazel_skylib",
     sha256 = "97e70364e9249702246c0e9444bccdc4b847bed1eb03c5a3ece4f83dfe6abc44",
     urls = [
-        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
-        "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version=SKYLIB_VERSION),
+        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version = SKYLIB_VERSION),
+        "https://github.com/bazelbuild/bazel-skylib/releases/download/{version}/bazel-skylib-{version}.tar.gz".format(version = SKYLIB_VERSION),
     ],
 )
 
 # Replace with the LLVM commit you want to use.
 LLVM_COMMIT = "09ac97ce350316b95b8cddb796d52f71b6f68296"
 
 # The easiest way to calculate this for a new commit is to set it to empty and
 # then run a bazel build and it will report the digest necessary to cache the
 # archive and make the build reproducible.
 LLVM_SHA256 = "2fb1aa06d12f8db349a27426cb0ced062987c5c2a75143c69f4284929e2750ff"
 
 # FIXME: It shouldn't be necessary to use http_archive twice here. Caching
 # should mean that this isn't too expensive though.