This is an archive of the discontinued LLVM Phabricator instance.

[cmake] Provide CURRENT_TOOLS_DIR centrally, replacing CLANG_TOOLS_DIR
ClosedPublic

Authored by sammccall on Mar 15 2022, 7:37 PM.

Details

Summary

CLANG_TOOLS_DIR holds the the current bin/ directory, maybe with a %(build_mode)
placeholder. It is used to add the just-built binaries to $PATH for lit tests.
In most cases it equals LLVM_TOOLS_DIR, which is used for the same purpose.
But for a standalone build of clang, CLANG_TOOLS_DIR points at the build tree
and LLVM_TOOLS_DIR points at the provided LLVM binaries.

Currently CLANG_TOOLS_DIR is set in clang/test/, clang-tools-extra/test/, and
other things always built with clang. This is a few cryptic lines of CMake in
each place. Meanwhile LLVM_TOOLS_DIR is provided by configure_site_lit_cfg().

This patch moves CLANG_TOOLS_DIR to configure_site_lit_cfg() and renames it:

  • there's nothing clang-specific about the value
  • it will also replace LLD_TOOLS_DIR, LLDB_TOOLS_DIR etc (not in this patch)

It also defines CURRENT_LIBS_DIR. While I removed the last usage of
CLANG_LIBS_DIR in e4cab4e24d1, there are LLD_LIBS_DIR usages etc that
may be live, and I'd like to mechanically update them in a followup patch.

Diff Detail

Event Timeline

sammccall created this revision.Mar 15 2022, 7:37 PM
Herald added a project: Restricted Project. · View Herald TranscriptMar 15 2022, 7:37 PM
sammccall requested review of this revision.Mar 15 2022, 7:37 PM
Herald added projects: Restricted Project, Restricted Project, Restricted Project. · View Herald TranscriptMar 15 2022, 7:37 PM

Trolling for reviewers:
@hokein: this is another -10 lines per test suite!
@thakis: the GN changes seem trivial (no standalone build) but wonder if you have opinions
@mgorny: because I don't truly understand the standalone build, and only hope I'm getting this right

I think this patch is probably doing too much: 1. move CLANG_TOOLS_DIR to configure_site_lit_cfg() 2. rename.

Can we split it into two patches? Land the first part, and the second part afterwards.

llvm/cmake/modules/AddLLVM.cmake
1640

the change looks good, but I'm not sure the new name CURRENT_TOOLS_DIR is good, the name seems vague and not significantly better, since it always points at the build tree, maybe LLVM_BUILD_DIR or something similar? (The existing naming pattern looks like project-specific variables are prefixed with the project name LLVM_, CLANG_).

llvm/utils/gn/secondary/clang-tools-extra/clangd/test/BUILD.gn
31

nit: this seems not needed anymore, as we have removed the CLANG_LIBS_DIR from the CMake file already.

I think this patch is probably doing too much: 1. move CLANG_TOOLS_DIR to configure_site_lit_cfg() 2. rename.

Can we split it into two patches? Land the first part, and the second part afterwards.

I don't think it can reasonably be called CLANG_TOOLS_DIR in that macro, the var has no connection to clang, and is set even when clang is not enabled.

llvm/cmake/modules/AddLLVM.cmake
1640

I agree the new name isn't good. (OTOH *all* these variables seem to have terrible names, so maybe it's just hard). But I think LLVM_BUILD_DIR is worse.

The name needs to communicate a lot of things:

  1. that this is parallel to LLVM_TOOLS_DIR, and the successor to $PROJECT_TOOLS_DIR
  2. that when LLVM and $PROJECT are built in different paths, this is the dir for $PROJECT and not for LLVM
  3. That this is not a concrete directory, but may contain a %(build_mode)% placeholder
  4. this variable is part of llvm-project, as opposed to some other CMake project it may be mixed with.

CLANG_TOOLS_DIR passes 1, 2, 4 (and fails 3, which is really hard).
CURRENT_TOOLS_DIR passes 1 & 2 only - it lacks namespacing.
LLVM_BUILD_DIR passes 4 only, I think - it's very unclear how it relates to LLVM_TOOLS_DIR.
LLVM_CURRENT_TOOLS_DIR would pass 1 and 4, but muddies the LLVM-vs-subproject distinction a bit.

What do you think of these criteria, are there some I missed? Any more name ideas? :-)

llvm/utils/gn/secondary/clang-tools-extra/clangd/test/BUILD.gn
31

Agree, I'll make a separate commit to avoid muddying the waters here.

hokein added inline comments.Mar 16 2022, 7:49 AM
llvm/cmake/modules/AddLLVM.cmake
1640

The name needs to communicate a lot of things:

  1. that this is parallel to LLVM_TOOLS_DIR, and the successor to $PROJECT_TOOLS_DIR
  2. that when LLVM and $PROJECT are built in different paths, this is the dir for $PROJECT and not for LLVM
  3. That this is not a concrete directory, but may contain a %(build_mode)% placeholder
  4. this variable is part of llvm-project, as opposed to some other CMake project it may be mixed with.

Thanks, these make sense, I were not aware of all of these and didn't get the full picture. I suppose it is being used for exposing various config.{project}_tools_dir variables for different tools (e.g. config.clang_tools_dir, config.lldb_tools_dir etc).

CURRENT_TOOLS_DIR passes 1 & 2 only - it lacks namespacing.

hmm, I think the new name is somewhat weak on 2 (particularly the PROJECT bit) -- the Current was unclear to me, I guess it means the current project right? I think it'd be better to have the PROJECT in the variable name, LLVM_PROJECT_TOOLS_DIR or shorter PROJECT_TOOLS_DIR. WDYT?

sammccall added inline comments.Mar 16 2022, 8:05 AM
llvm/cmake/modules/AddLLVM.cmake
1640

The name needs to communicate a lot of things:

  1. that this is parallel to LLVM_TOOLS_DIR, and the successor to $PROJECT_TOOLS_DIR
  2. that when LLVM and $PROJECT are built in different paths, this is the dir for $PROJECT and not for LLVM
  3. That this is not a concrete directory, but may contain a %(build_mode)% placeholder
  4. this variable is part of llvm-project, as opposed to some other CMake project it may be mixed with.

Thanks, these make sense, I were not aware of all of these and didn't get the full picture. I suppose it is being used for exposing various config.{project}_tools_dir variables for different tools (e.g. config.clang_tools_dir, config.lldb_tools_dir etc).

That's right. Ultimately it's being used to make sure that when you run lit tests, clang resolves to the clang you just built and not something else.
(It's more complicated than that, there are several conflicting mechanisms, but that's the essence).

CURRENT_TOOLS_DIR passes 1 & 2 only - it lacks namespacing.

hmm, I think the new name is somewhat weak on 2 (particularly the PROJECT bit) -- the Current was unclear to me, I guess it means the current project right?

It really means the current cmake invocation.

In a "standalone build" of clang (rooted at llvm-project/clang, with a separate LLVM tree), it is build/bin where LLVM_TOOLS_DIR is path/to/llvm/bin, and was produced by a *previous* cmake invocation (or an install step).

In a normal unified build of llvm+clang+lld (rooted at llvm-project/llvm with LLVM_ENABLE_PROJECTS), both variables are set to build/bin.

There's really no fixed connection with a subproject at all, and certainly its value doesn't vary across the various subprojects of llvm-project.

I think it'd be better to have the PROJECT in the variable name, LLVM_PROJECT_TOOLS_DIR

I think this name is misleading because of the connection to the repo llvm-project.
It suggests LLVM_PROJECT_TOOLS_DIR is a top-level thing when it's really the opposite, a subproject thing. (And while LLVM_TOOLS_DIR *does* correspond to llvm-project/llvm the tools are often things people think of as "top-level").

or shorter PROJECT_TOOLS_DIR. WDYT?

I can live with it, but I'm not really sure whether PROJECT communicates anything that's true :-)
I'm tempted by just TOOLS_DIR but I worry that such a short name with a lack of namespacing is asking for trouble. On the other hand, there's SHLIBDIR, which also points at shared libs in the current build tree...

hokein accepted this revision.Mar 16 2022, 8:24 AM
hokein added inline comments.
llvm/cmake/modules/AddLLVM.cmake
1640

Thanks for the explanations, now I'm convinced, Let's stick with the current name :)

This revision is now accepted and ready to land.Mar 16 2022, 8:24 AM

Rebase. Also update LLD since it's simple and similar to clang.

This revision was landed with ongoing or failed builds.Mar 25 2022, 12:22 PM
This revision was automatically updated to reflect the committed changes.