This is an archive of the discontinued LLVM Phabricator instance.

LLVM Driver Multicall tool
ClosedPublic

Authored by abrachet on Sep 17 2021, 9:36 AM.

Details

Summary

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER option to the add_llvm_executable CMake
call, and changing the tool's main function to a canonicalized
tool_name_main format (i.e. llvm_ar_main, clang_main, etc...).

As currently implemented llvm-driver contains dsymutil, llvm-ar,
llvm-cxxfilt, llvm-objcopy, and clang (if clang is included in the
build).

llvm-driver can be enabled from builds by setting
LLVM_TOOL_LLVM_DRIVER_BUILD=On.

There are several limitations in the current implementation, which can
be addressed in subsequent patches:

(1) the multicall binary cannot currently properly handle
multi-dispatch tools. This means symlinking llvm-ranlib to llvm-driver
will not properly result in llvm-ar's main being called.
(2) the multicall binary cannot be comprised of tools containing
conflicting cl::opt options as the global cl::opt option list cannot
contain duplicates.

These limitations can be addressed in subsequent patches.

Diff Detail

Event Timeline

beanz created this revision.Sep 17 2021, 9:36 AM
beanz requested review of this revision.Sep 17 2021, 9:36 AM
Herald added projects: Restricted Project, Restricted Project. · View Herald TranscriptSep 17 2021, 9:36 AM
Herald added a subscriber: cfe-commits. · View Herald Transcript
aganea added a subscriber: aganea.Sep 17 2021, 9:41 AM

That's pretty nice! Have you thought about looking into a lit option (triggered by a cmake flag maybe) that would change the substitution for the tools that are enabled by llvm-driver to use the latter instead when running the tests?

phosek added inline comments.Sep 17 2021, 10:23 AM
llvm/tools/llvm-driver/CMakeLists.txt
18

As was already suggested on D104686, I'd prefer naming this just llvm so you can invoke tools with llvm name which doesn't require any additional typing compared to llvm-name.

That's pretty nice! Have you thought about looking into a lit option (triggered by a cmake flag maybe) that would change the substitution for the tools that are enabled by llvm-driver to use the latter instead when running the tests?

That's an awesome idea. I had been thinking about having a CMake option to generate symlinks instead of the tools, but you could totally bake this into the lit substitutions. I'll take a look at that.

llvm/tools/llvm-driver/CMakeLists.txt
18

Can do

beanz updated this revision to Diff 373300.Sep 17 2021, 12:04 PM

Renaming llvm-driver binary to llvm.

This is really a great change, thanks @beanz!

llvm/tools/llvm-objcopy/llvm-objcopy.cpp
226

Shouldn't we say:

int objcopy_main(int argc, char **argv) {

here and the other places + all supporting code, if we want llvm objcopy (without the dash) like @phosek suggests?

beanz added inline comments.Sep 17 2021, 2:51 PM
llvm/tools/llvm-objcopy/llvm-objcopy.cpp
226

I had a different thought for that. I think we want the tools to respond to llvm-objcopy since we will want them to exist in parallel to binutils tools just like the current tools do today.

Many of the current tools also support symlink-redirection, to support that we'll need to have a multiplex where multiple tool names point to the same main function.

Handling that was my point (1) in the main commit message, and I intended to work on it in a follow-on commit.

@beanz Is this ready to land or do you plan on making any more changes?

Sorry for disappearing on this. I'm working right now on adding support for multi-entry tools (like objcopy). I'll get an update in today.

beanz updated this revision to Diff 404114.Jan 28 2022, 11:37 AM

Adding support for multiple subcommands mapping to the same entry.

Hope that someone familiar with CMake can take a look.

As currently implemented llvm-driver contains dsymutil, llvm-ar, llvm-cxxfilt, llvm-objcopy, and clang (if clang is included in the build).

I think either (bundled clang+binary utilities) or (bundled lld+binary utilities) is fine.
Some folks are interested in PGO builds for clang and lld. Not bundling clang and lld together can make both optimized.
The performance of binary utilities likely does not matter that much, so bundling them with either clang or lld should be fine.


I have tried the latest diff but -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;flang;lldb;lld;compiler-rt;openmp;mlir;cross-project-tests' -DLLVM_TOOL_LLVM_DRIVER_BUILD=on gives me:

CMake Error at cmake/modules/LLVM-Config.cmake:110 (target_link_libraries):                 
  The keyword signature for target_link_libraries has already been used with                
  the target "obj.dsymutil".  All uses of target_link_libraries with a target               
  must be either all-keyword or all-plain.                                                  
                                                                                            
  The uses of the keyword signature are here:                                                                                                 
                                                                                            
   * cmake/modules/AddLLVM.cmake:897 (target_link_libraries)                                
                                                                                            
Call Stack (most recent call first):                                                        
  cmake/modules/LLVM-Config.cmake:95 (explicit_llvm_config)                                 
  cmake/modules/AddLLVM.cmake:898 (llvm_config)                                                                                               
  cmake/modules/AddLLVM.cmake:1264 (add_llvm_executable)                                                                                      
  tools/dsymutil/CMakeLists.txt:21 (add_llvm_tool)
MaskRay added a comment.EditedMar 12 2022, 10:36 AM

I have tried the latest diff but -DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;flang;lldb;lld;compiler-rt;openmp;mlir;cross-project-tests' -DLLVM_TOOL_LLVM_DRIVER_BUILD=on gives me:

CMake Error at cmake/modules/LLVM-Config.cmake:110 (target_link_libraries):                 
  The keyword signature for target_link_libraries has already been used with                
  the target "obj.dsymutil".  All uses of target_link_libraries with a target               
  must be either all-keyword or all-plain.                                                  
                                                                                            
  The uses of the keyword signature are here:                                                                                                 
                                                                                            
   * cmake/modules/AddLLVM.cmake:897 (target_link_libraries)                                
                                                                                            
Call Stack (most recent call first):                                                        
  cmake/modules/LLVM-Config.cmake:95 (explicit_llvm_config)                                 
  cmake/modules/AddLLVM.cmake:898 (llvm_config)                                                                                               
  cmake/modules/AddLLVM.cmake:1264 (add_llvm_executable)                                                                                      
  tools/dsymutil/CMakeLists.txt:21 (add_llvm_tool)

@beanz ⬆️

Herald added a project: Restricted Project. · View Herald TranscriptMar 12 2022, 10:36 AM

Another potential future improvement is error reporting for subcommands:

$ ./bin/llvm clang       
llvm: error: no input files
$ ./bin/clang    
clang-15: error: no input files

Ideally, the multicall tool would produce the same error message.

llvm/cmake/modules/AddLLVM.cmake
900

The error that @MaskRay reported is because we use PUBLIC with target_link_libraries() here, but we use plain form of target_link_libraries() later in explicit_llvm_config that is invoked llvm_config (see https://github.com/llvm/llvm-project/blob/1643f01232b41b93e5f3a21d89b111efab0e378a/llvm/cmake/modules/LLVM-Config.cmake#L110). Omitting PUBLIC here appears to be sufficient to avoid the error.

llvm/tools/CMakeLists.txt
59

Should this be guarded by LLVM_TOOL_LLVM_DRIVER_BUILD to behave as the commit message says? Currently the llvm-driver appears to be built regardless of LLVM_TOOL_LLVM_DRIVER_BUILD.

llvm/tools/llvm-driver/llvm-driver.cpp
44

When the tool is invoked without any arguments (that is, simply as ./bin/llvm) this will lead to out-of-bounds array access. We should handle this case explicitly.

@beanz Let me know if you need help, I'm happy to commandeer the change if you're too busy.

beanz added a comment.Mar 24 2022, 9:32 AM

@beanz Let me know if you need help, I'm happy to commandeer the change if you're too busy.

Please feel free to commandeer it. I'm really sorry I've held this up so long and I'm totally swamped right now :(.

abrachet commandeered this revision.Jun 4 2022, 10:45 AM
abrachet added a reviewer: beanz.
abrachet added a subscriber: abrachet.

I spoke to @beanz offline to make sure he is still ok with the patch being commandeered. Thanks for the patch @beanz, very excited about this :)

abrachet updated this revision to Diff 434277.Jun 4 2022, 10:47 AM
  • llvm llvm-$tool -> llvm $tool
  • Error messages will now show $tool: error: ... instead of error: ...
abrachet marked 5 inline comments as done.Jun 4 2022, 10:51 AM

Another potential future improvement is error reporting for subcommands:

$ ./bin/llvm clang       
llvm: error: no input files
$ ./bin/clang    
clang-15: error: no input files

Ideally, the multicall tool would produce the same error message.

It's difficult to make the error message the same, ie clang-15, but hopefully the name the tool was invoked with is enough.

llvm/tools/llvm-driver/llvm-driver.cpp
44

This will now print the help message

llvm/tools/llvm-objcopy/llvm-objcopy.cpp
226

Shouldn't we say:

int objcopy_main(int argc, char **argv) {

here and the other places + all supporting code, if we want llvm objcopy (without the dash) like @phosek suggests?

I've kept name of the function as is, but llvm objcopy is now supported, and llvm llvm-objcopy is not

MaskRay added a comment.EditedJun 4 2022, 11:17 PM

Thanks for picking up the change. I confirm that I can build llvm without an error and it appears to work fine.

llvm-driver can be disabled from builds by setting LLVM_TOOL_LLVM_DRIVER_BUILD=Off.

I think this should be opt-in. The new llvm executable takes a lot of space and not needed by many developers/build bots.
It's useful to some groups (distributions) but they can specify the option themselves.
I think the modified code is quite stable, so don't worry about regressions just because this is not opt-in.

llvm/tools/llvm-driver/llvm-driver.cpp
24

static int unknownMain

25

Prefer OptTable to cl:: for user-facing utilities. If simple, just open coding the help message.

37
50

Some distributions may want to use something like llvm-15. See some binary utilities how the version is handled.

58

If only used once, don't add a typedef (using is preferred)

MaskRay added a comment.EditedJun 4 2022, 11:19 PM
% /tmp/out/custom1/bin/llvm --help
OVERVIEW: llvm compiler driver   ##### llvm toolchain driver ?

USAGE: llvm [subcommand] [options]

SUBCOMMANDS:

  ar                - ar  ###### should be llvm-ar
  bitcode-strip     - bitcode-strip   ### should be llvm-bitcode-strip
  clang             - clang
  clang++           - clang++
  clang-cl          - clang-cl
  clang-cpp         - clang-cpp
  cxxfilt           - cxxfilt   ### ditto
  dlltool           - dlltool
  dsymutil          - dsymutil
  install-name-tool - install-name-tool
  lib               - lib  ##### this is confusing. llvm-lib
  objcopy           - objcopy
  ranlib            - ranlib
  strip             - strip

  Type "llvm <subcommand> --help" to get more help on a specific subcommand

OPTIONS:

Generic Options:

  --help      - Display available options (--help-hidden for more) 
  --help-list - Display list of available options (--help-list-hidden for more)  #### this from cl:: is not useful
  --version   - Display the version of this program
abrachet updated this revision to Diff 434352.Jun 5 2022, 2:12 PM
abrachet marked an inline comment as done.
abrachet edited the summary of this revision. (Show Details)

Better handling of names with version, ie llvm-cxxfilt-15 -> cxxfilt

abrachet marked 5 inline comments as done.Jun 5 2022, 2:15 PM

Thanks for picking up the change. I confirm that I can build llvm without an error and it appears to work fine.

llvm-driver can be disabled from builds by setting LLVM_TOOL_LLVM_DRIVER_BUILD=Off.

I think this should be opt-in. The new llvm executable takes a lot of space and not needed by many developers/build bots.
It's useful to some groups (distributions) but they can specify the option themselves.
I think the modified code is quite stable, so don't worry about regressions just because this is not opt-in.

I didn't update the commits message, now it properly says that it can be opted in by setting LLVM_TOOL_LLVM_DRIVER_BUILD to On

% /tmp/out/custom1/bin/llvm --help
...

The output now looks like:

OVERVIEW: llvm toolchain driver

USAGE: llvm [subcommand] [options]

SUBCOMMANDS:

  ar
  clang
  dsymutil
  cxxfilt
  objcopy
  ranlib
  lib
  dlltool
  clang++
  clang-cl
  clang-cpp
  install-name-tool
  bitcode-strip
  strip

  Type "llvm <subcommand> --help" to get more help on a specific subcommand

OPTIONS:

  --help - Display this message

What do you think?

llvm/tools/llvm-driver/llvm-driver.cpp
50

Thank's I've taken this from objcopy's code

MaskRay accepted this revision.Jun 5 2022, 2:21 PM

LGTM from my perspective, but this needs someone else's review.

llvm/tools/llvm-driver/llvm-driver.cpp
50

The format of cl::PrintVersionMessage(); is not so good for user-facing tools. Consider omitting it.

llvm-objcopy --version should probably use a style similar to clang --version but the priority isn't high.

This revision is now accepted and ready to land.Jun 5 2022, 2:21 PM
phosek accepted this revision.Jun 5 2022, 5:50 PM

LGTM

llvm/cmake/modules/AddLLVM.cmake
2030

Alternative substitution for + would be x which is used elsewhere in LLVM, for example libcxx, libcxxabi or cxxfilt.

abrachet updated this revision to Diff 434370.Jun 5 2022, 7:49 PM
abrachet marked an inline comment as done.
  • Fix bazel build, llvm-driver can't be built by bazel, but the existing tools which lost their main will still work.
  • Remove --version
  • Fix dsymutil not depending on DsymutilTableGen which caused a dependency ordering problem on the bots
abrachet marked 2 inline comments as done.Jun 5 2022, 7:52 PM
abrachet added inline comments.
llvm/cmake/modules/AddLLVM.cmake
2030

Upon closer inspection we are no longer using that parameter of the macro, so I have removed it.

llvm/tools/llvm-driver/llvm-driver.cpp
50

I've just removed --version completely for now. I don't think it is that pressing.

This revision was landed with ongoing or failed builds.Jun 5 2022, 9:28 PM
Closed by commit rGf06abbb39380: LLVM Driver Multicall tool (authored by beanz, committed by abrachet). · Explain Why
This revision was automatically updated to reflect the committed changes.
abrachet marked 2 inline comments as done.
thakis added a subscriber: thakis.Jun 6 2022, 3:43 AM
thakis added inline comments.
llvm/test/lit.cfg.py
142

This causes llvm-lit: /Users/thakis/src/llvm-project/llvm/utils/lit/lit/llvm/subst.py:126: note: Did not find llvm in /Users/thakis/src/llvm-build-2/bin if llvm-driver isn't enabled (which it isn't by default). Any reason not to add this one only if config.have_llvm_driver?

abrachet marked an inline comment as done.Jun 6 2022, 7:31 AM
abrachet added inline comments.
llvm/test/lit.cfg.py
142
chapuni added a subscriber: chapuni.Jun 6 2022, 5:17 PM

An interesting improvement. Thanks.

llvm/cmake/modules/AddLLVM.cmake
890

Do you have a plan to export the template?

utils/bazel/llvm-project-overlay/clang/BUILD.bazel
1932 ↗(On Diff #434384)

Could we unify generated names.cpp between cmake and bazel?

I am running a builder to compare each object file between cmake and bazel. Such a difference appears in a few sections, at least in elf.

thakis added inline comments.Jun 6 2022, 6:36 PM
llvm/test/lit.cfg.py
142

Thank you!

nikic added a subscriber: nikic.Jun 9 2022, 12:56 AM
nikic added inline comments.
llvm/cmake/modules/AddLLVM.cmake
890

Unsurprisingly, this breaks standalone builds. However, I think that the multicall tool is fundamentally incompatible with standalone builds, in the sense that all tools included in it must be part of the initial build configuration -- you can't first build the LLVM multicall tool and then add clang to it afterwards.

So I think we just need to guard this code by LLVM_TOOL_LLVM_DRIVER_BUILD, so we don't try to read a non-exported driver-template.cpp.in file even if we're not actually building the multicall tool.

nikic added inline comments.Jun 9 2022, 1:06 AM
llvm/cmake/modules/AddLLVM.cmake
890

Ah, I misunderstood how this works. Generating the driver file is necessary for normal usage, not for the mutlicall tool, so it's not possible to just guard this code. We do need to export driver-template.cpp.in as part of the LLVM cmake installation.

nikic added inline comments.Jun 9 2022, 1:54 AM
llvm/cmake/modules/AddLLVM.cmake
890

I've put up https://reviews.llvm.org/D127384 to fix this.

This seems to have broken standalone builds for Gentoo.

While building Clang, I get:

CMake Error at /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:138 (message):
  Target Sparc is not in the set of libraries.
Call Stack (most recent call first):
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:263 (llvm_expand_pseudo_components)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:102 (llvm_map_components_to_libnames)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:95 (explicit_llvm_config)
  /usr/lib/llvm/15/lib/cmake/llvm/AddLLVM.cmake:901 (llvm_config)
  cmake/modules/AddClang.cmake:146 (add_llvm_executable)
  cmake/modules/AddClang.cmake:156 (add_clang_executable)
  tools/driver/CMakeLists.txt:25 (add_clang_tool)

(or a similar message for another target)

abrachet marked an inline comment as done.Jul 14 2022, 7:52 AM

This seems to have broken standalone builds for Gentoo.

While building Clang, I get:

CMake Error at /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:138 (message):
  Target Sparc is not in the set of libraries.
Call Stack (most recent call first):
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:263 (llvm_expand_pseudo_components)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:102 (llvm_map_components_to_libnames)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:95 (explicit_llvm_config)
  /usr/lib/llvm/15/lib/cmake/llvm/AddLLVM.cmake:901 (llvm_config)
  cmake/modules/AddClang.cmake:146 (add_llvm_executable)
  cmake/modules/AddClang.cmake:156 (add_clang_executable)
  tools/driver/CMakeLists.txt:25 (add_clang_tool)

(or a similar message for another target)

Could I have a cmake invocation or a bot link?

beanz added a comment.Jul 14 2022, 8:00 AM

This seems to have broken standalone builds for Gentoo.

While building Clang, I get:

CMake Error at /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:138 (message):
  Target Sparc is not in the set of libraries.
Call Stack (most recent call first):
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:263 (llvm_expand_pseudo_components)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:102 (llvm_map_components_to_libnames)
  /usr/lib/llvm/15/lib/cmake/llvm/LLVM-Config.cmake:95 (explicit_llvm_config)
  /usr/lib/llvm/15/lib/cmake/llvm/AddLLVM.cmake:901 (llvm_config)
  cmake/modules/AddClang.cmake:146 (add_llvm_executable)
  cmake/modules/AddClang.cmake:156 (add_clang_executable)
  tools/driver/CMakeLists.txt:25 (add_clang_tool)

(or a similar message for another target)

Could I have a cmake invocation or a bot link?

@mgorny Have you updated your LLVM with this change?

If not, I suspect this breakage is expected.

@mgorny is building against an installed llvm-15 that if it doesn't have the changes from this patch, mixing that with a ToT clang which does will not work.

In that case the solution is that @mgorny will need to update the LLVM install to handle the revlock.

LLVM & Clang do rev lock from time to time during trunk development, usually when LLVM APIs change, but the build system can cause rev locks too since Clang reuses LLVM's CMake modules.

Could I have a cmake invocation or a bot link?

The exact CMake invocation for clang is:

cmake -C /var/tmp/portage/sys-devel/clang-15.0.0.9999/work/x/y/clang-abi_x86_64.amd64/gentoo_common_config.cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DLLVM_CMAKE_PATH=/usr/lib/llvm/15/lib64/cmake/llvm -DCMAKE_INSTALL_PREFIX=/usr/lib/llvm/15 -DCMAKE_INSTALL_MANDIR=/usr/lib/llvm/15/share/man -DCLANG_RESOURCE_DIR=../../../../lib/clang/15.0.0 -DBUILD_SHARED_LIBS=OFF -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_DISTRIBUTION_COMPONENTS="clang-cmake-exports;clang-headers;clang-resource-headers;libclang-headers;clang-cpp;libclang;bash-autocomplete;libclang-python-bindings;c-index-test;clang;clang-format;clang-offload-bundler;clang-offload-wrapper;clang-refactor;clang-repl;clang-rename;clang-scan-deps;diagtool;hmaptool;clang-apply-replacements;clang-change-namespace;clang-doc;clang-include-fixer;clang-move;clang-query;clang-reorder-fields;clang-tidy;clang-tidy-headers;clangd;find-all-symbols;modularize;pp-trace;docs-clang-man;docs-clang-tools-man;clang-check;clang-extdef-mapping;scan-build;scan-build-py;scan-view;" -DLLVM_TARGETS_TO_BUILD="SystemZ;WebAssembly;BPF;Hexagon;VE;XCore;X86;Sparc;NVPTX;ARM;AVR;Lanai;AMDGPU;Mips;AArch64;PowerPC;MSP430;RISCV" -DLLVM_BUILD_TESTS=no -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_RTTI=ON -DCMAKE_DISABLE_FIND_PACKAGE_LibXml2=no -DCLANG_DEFAULT_OPENMP_RUNTIME=libomp -DCMAKE_DISABLE_FIND_PACKAGE_CUDA=ON -DCLANG_DEFAULT_CXX_STDLIB= -DCLANG_DEFAULT_RTLIB= -DCLANG_DEFAULT_LINKER= -DCLANG_DEFAULT_PIE_ON_LINUX=yes -DCLANG_DEFAULT_UNWINDLIB= -DCLANG_ENABLE_ARCMT=yes -DCLANG_ENABLE_STATIC_ANALYZER=yes -DPython3_EXECUTABLE=/usr/bin/python3.10 -DLLVM_BUILD_DOCS=ON -DLLVM_ENABLE_SPHINX=ON -DCLANG_INSTALL_SPHINX_HTML_DIR=/usr/share/doc/clang-15.0.0.9999/html -DCLANG-TOOLS_INSTALL_SPHINX_HTML_DIR=/usr/share/doc/clang-15.0.0.9999/tools-extra -DSPHINX_WARNINGS_AS_ERRORS=OFF -DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=/var/tmp/portage/sys-devel/clang-15.0.0.9999/work/clang-tools-extra -DCLANG_INCLUDE_DOCS=ON -DCLANG_TOOLS_EXTRA_INCLUDE_DOCS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/sys-devel/clang-15.0.0.9999/work/x/y/clang-abi_x86_64.amd64/gentoo_toolchain.cmake /var/tmp/portage/sys-devel/clang-15.0.0.9999/work/clang

Though I can reproduce it with much shorter:

mkdir build
cd build
cmake ../clang -G Ninja

(matching installed version of LLVM 15 must be on PATH)

@mgorny Have you updated your LLVM with this change?

Obviously. I always build matching git commit of all LLVM components when testing the most recent state.

mceier added a subscriber: mceier.Jul 14 2022, 3:06 PM

Could I have a cmake invocation or a bot link?

The exact CMake invocation for clang is:

cmake -C /var/tmp/portage/sys-devel/clang-15.0.0.9999/work/x/y/clang-abi_x86_64.amd64/gentoo_common_config.cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DLLVM_CMAKE_PATH=/usr/lib/llvm/15/lib64/cmake/llvm -DCMAKE_INSTALL_PREFIX=/usr/lib/llvm/15 -DCMAKE_INSTALL_MANDIR=/usr/lib/llvm/15/share/man -DCLANG_RESOURCE_DIR=../../../../lib/clang/15.0.0 -DBUILD_SHARED_LIBS=OFF -DCLANG_LINK_CLANG_DYLIB=ON -DLLVM_DISTRIBUTION_COMPONENTS="clang-cmake-exports;clang-headers;clang-resource-headers;libclang-headers;clang-cpp;libclang;bash-autocomplete;libclang-python-bindings;c-index-test;clang;clang-format;clang-offload-bundler;clang-offload-wrapper;clang-refactor;clang-repl;clang-rename;clang-scan-deps;diagtool;hmaptool;clang-apply-replacements;clang-change-namespace;clang-doc;clang-include-fixer;clang-move;clang-query;clang-reorder-fields;clang-tidy;clang-tidy-headers;clangd;find-all-symbols;modularize;pp-trace;docs-clang-man;docs-clang-tools-man;clang-check;clang-extdef-mapping;scan-build;scan-build-py;scan-view;" -DLLVM_TARGETS_TO_BUILD="SystemZ;WebAssembly;BPF;Hexagon;VE;XCore;X86;Sparc;NVPTX;ARM;AVR;Lanai;AMDGPU;Mips;AArch64;PowerPC;MSP430;RISCV" -DLLVM_BUILD_TESTS=no -DLLVM_ENABLE_EH=ON -DLLVM_ENABLE_RTTI=ON -DCMAKE_DISABLE_FIND_PACKAGE_LibXml2=no -DCLANG_DEFAULT_OPENMP_RUNTIME=libomp -DCMAKE_DISABLE_FIND_PACKAGE_CUDA=ON -DCLANG_DEFAULT_CXX_STDLIB= -DCLANG_DEFAULT_RTLIB= -DCLANG_DEFAULT_LINKER= -DCLANG_DEFAULT_PIE_ON_LINUX=yes -DCLANG_DEFAULT_UNWINDLIB= -DCLANG_ENABLE_ARCMT=yes -DCLANG_ENABLE_STATIC_ANALYZER=yes -DPython3_EXECUTABLE=/usr/bin/python3.10 -DLLVM_BUILD_DOCS=ON -DLLVM_ENABLE_SPHINX=ON -DCLANG_INSTALL_SPHINX_HTML_DIR=/usr/share/doc/clang-15.0.0.9999/html -DCLANG-TOOLS_INSTALL_SPHINX_HTML_DIR=/usr/share/doc/clang-15.0.0.9999/tools-extra -DSPHINX_WARNINGS_AS_ERRORS=OFF -DLLVM_EXTERNAL_CLANG_TOOLS_EXTRA_SOURCE_DIR=/var/tmp/portage/sys-devel/clang-15.0.0.9999/work/clang-tools-extra -DCLANG_INCLUDE_DOCS=ON -DCLANG_TOOLS_EXTRA_INCLUDE_DOCS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_TOOLCHAIN_FILE=/var/tmp/portage/sys-devel/clang-15.0.0.9999/work/x/y/clang-abi_x86_64.amd64/gentoo_toolchain.cmake /var/tmp/portage/sys-devel/clang-15.0.0.9999/work/clang

This won't work for me because I will need gentoo_common_config.cmake, though I would rather work with the shorter reproducer you have provided.

Though I can reproduce it with much shorter:

mkdir build
cd build
cmake ../clang -G Ninja

(matching installed version of LLVM 15 must be on PATH)

Are you sure it should be in PATH? I have added installed LLVM from ToT to a directory, put that in path yet CMake still finds /usr/lib/llvm-13/cmake/AddLLVM.cmake instead. I can't find any documentation on the standalone build, unfortunately.

Though I can reproduce it with much shorter:

mkdir build
cd build
cmake ../clang -G Ninja

(matching installed version of LLVM 15 must be on PATH)

Are you sure it should be in PATH? I have added installed LLVM from ToT to a directory, put that in path yet CMake still finds /usr/lib/llvm-13/cmake/AddLLVM.cmake instead. I can't find any documentation on the standalone build, unfortunately.

I'm sorry, what does "ToT" mean here? Also, what's /usr/lib/llvm-13/cmake? Perhaps it's picking it up from some other variable. In my case, it's finding the installed LLVM 14 because that's the first PATH entry. Note that CMake looks into ../lib*/cmake and ../share/cmake relative to PATH entries. FWICS it also looks at CMAKE_MODULE_PATH.

MaskRay added a comment.EditedJul 18 2022, 12:47 PM

Though I can reproduce it with much shorter:

mkdir build
cd build
cmake ../clang -G Ninja

(matching installed version of LLVM 15 must be on PATH)

Are you sure it should be in PATH? I have added installed LLVM from ToT to a directory, put that in path yet CMake still finds /usr/lib/llvm-13/cmake/AddLLVM.cmake instead. I can't find any documentation on the standalone build, unfortunately.

I'm sorry, what does "ToT" mean here? Also, what's i? Perhaps it's picking it up from some other variable. In my case, it's finding the installed LLVM 14 because that's the first PATH entry. Note that CMake looks into ../lib*/cmake and ../share/cmake relative to PATH entries. FWICS it also looks at CMAKE_MODULE_PATH.

"Top of Trunk" (or tip?) (or tree?) which indicates the latest commit of origin/main . "Trunk" was a term inherited from the old subversion days.

Debian installs different llvm packages to /usr/lib/llvm-*:

% ls /usr/lib/llvm-* 
/usr/lib/llvm-11:
bin/  build/  cmake@  include/  lib/  share/

/usr/lib/llvm-13:
bin/  build/  cmake@  include/  lib/  share/

/usr/lib/llvm-9:
bin/  build/  cmake@  include/  lib/  share/
% dpkg -S /usr/lib/llvm-13/lib/cmake/llvm/LLVMConfig.cmake 
llvm-13-dev: /usr/lib/llvm-13/lib/cmake/llvm/LLVMConfig.cmake

For downstream reports (e.g. Debian, Gentoo), it is worth stripping the distribution specific knowledge so that a patch author doesn't need to install a particular distribution to reproduce an issue.

Personally I really wish that the build system issue can be fixed in the upcoming 15.0.0 release and encourage distributions to use it to decrease executable sizes.

Thanks @mgorny I was able to get CMake to run. Although I didn't end up with the same error as you. I simply had warnings:

CMake Warning (dev) at <redacted>/usr/local/lib/cmake/llvm/TableGen.cmake:103 (add_custom_command):
  Policy CMP0116 is not set: Ninja generators transform DEPFILEs from
  add_custom_command().  Run "cmake --help-policy CMP0116" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.
Call Stack (most recent call first):
  cmake/modules/AddClang.cmake:25 (tablegen)
  include/clang/AST/CMakeLists.txt:1 (clang_tablegen)
This warning is for project developers.  Use -Wno-dev to suppress it.

When running just cmake ../clang -G Ninja did you delete CMakeCache.txt? If you didn't then perhaps something from the full command you gave stayed around and caused the error. I suspect then I will need the full command you have supplied above including gentoo_common_config.cmake. Likely I will not need gentoo_toolchain.cmake

Personally I really wish that the build system issue can be fixed in the upcoming 15.0.0 release and encourage distributions to use it to decrease executable sizes.

Me too. For these tools to be installable as symlinks to the llvm driver, D127800 will need to land, as well as a follow up after that patch. Maybe I could entice you to take a look :)

Personally I really wish that the build system issue can be fixed in the upcoming 15.0.0 release and encourage distributions to use it to decrease executable sizes.

Me too, not because of multicall but because right now it's impossible to even build clang with multicall disabled. It's a complete 100% blocker for us.

When running just cmake ../clang -G Ninja did you delete CMakeCache.txt?

I'm pretty sure I did.

If you didn't then perhaps something from the full command you gave stayed around and caused the error. I suspect then I will need the full command you have supplied above including gentoo_common_config.cmake. Likely I will not need gentoo_toolchain.cmake

I'm going to start by trying against with today's main, maybe (hopefully) something changed. I just hoped I wouldn't have to spend the time I don't really have on this but I guess I have no other option :-(.

Hmm, I have another idea. Do you have individual LLVM static (or shared) libraries installed? On Gentoo we install only the dylib and a few unavoidable static libs:

/usr/lib/llvm/15/lib64/libRemarks.so
/usr/lib/llvm/15/lib64/libRemarks.so.15git
/usr/lib/llvm/15/lib64/libLLVM-15git.so
/usr/lib/llvm/15/lib64/libLLVM.so
/usr/lib/llvm/15/lib64/libLLVM-15.0.0git.so
/usr/lib/llvm/15/lib64/libLTO.so
/usr/lib/llvm/15/lib64/libLTO.so.15git
/usr/lib/llvm/15/lib64/libLLVMTableGen.a
/usr/lib/llvm/15/lib64/libLLVMSupport.a
/usr/lib/llvm/15/lib64/libLLVMDemangle.a

On my end, I'm going to check if installing full set of static libs changes anything.

Unfortunately, that doesn't seem to be it. I think at this point it would be easier for both of us if I tried to figure it out myself on Gentoo rather than trying to make it reproducible elsewhere. If you want, you could try grabbing e.g. a Gentoo stage3 image (e.g. via Docker or systemd-nspawn) and trying ACCEPT_KEYWORDS=** emerge -v clang:15 there but I suppose you'd have to figure out some Gentoo basics anyway (and then you'd realize it's the best distro in the world and waste years of your life developing it :-P).

Ok, I think the key to reproducing it is -DLLVM_LINK_LLVM_DYLIB=ON. We pass that while building LLVM, so it gets passed on to clang but I suppose passing it to clang build might be sufficient to reproduce the problem.

The actual problem is that you're passing ${USE_SHARED} to llvm_config() (in add_llvm_executable) before it's defined. Moving the definition earlier fixed this error but uncovers another problem:

CMake Error at cmake/modules/AddClang.cmake:188 (target_link_libraries):
  The keyword signature for target_link_libraries has already been used with
  the target "obj.clang".  All uses of target_link_libraries with a target
  must be either all-keyword or all-plain.

  The uses of the keyword signature are here:

   * /usr/lib/llvm/15/lib64/cmake/llvm/LLVM-Config.cmake:92 (target_link_libraries)

Call Stack (most recent call first):
  tools/driver/CMakeLists.txt:37 (clang_target_link_libraries)

Now, if you move USE_SHARED definition using the following patch:

diff --git a/llvm/cmake/modules/AddLLVM.cmake b/llvm/cmake/modules/AddLLVM.cmake
index 8e1385e90b82..c37c1c3aa59d 100644
--- a/llvm/cmake/modules/AddLLVM.cmake
+++ b/llvm/cmake/modules/AddLLVM.cmake
@@ -890,6 +890,10 @@ macro(add_llvm_executable name)
     set_target_properties(${obj_name} PROPERTIES FOLDER "Object Libraries")
   endif()
 
+  if (LLVM_LINK_LLVM_DYLIB AND NOT ARG_DISABLE_LLVM_LINK_LLVM_DYLIB)
+    set(USE_SHARED USE_SHARED)
+  endif()
+
   if (ARG_GENERATE_DRIVER)
     string(REPLACE "-" "_" TOOL_NAME ${name})
     foreach(path ${CMAKE_MODULE_PATH})
@@ -964,10 +968,6 @@ macro(add_llvm_executable name)
     add_llvm_symbol_exports( ${name} ${LLVM_EXPORTED_SYMBOL_FILE} )
   endif(LLVM_EXPORTED_SYMBOL_FILE)
 
-  if (LLVM_LINK_LLVM_DYLIB AND NOT ARG_DISABLE_LLVM_LINK_LLVM_DYLIB)
-    set(USE_SHARED USE_SHARED)
-  endif()
-
   set(EXCLUDE_FROM_ALL OFF)
   set_output_directory(${name} BINARY_DIR ${LLVM_RUNTIME_OUTPUT_INTDIR} LIBRARY_DIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
   llvm_config( ${name} ${USE_SHARED} ${LLVM_LINK_COMPONENTS} )

you should be able to reproduce the latter error using a regular in-tree build, e.g.:

cmake ../llvm -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DLLVM_ENABLE_PROJECTS='llvm;clang' -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
MaskRay added a comment.EditedJul 19 2022, 11:11 AM

Ok, I think the key to reproducing it is -DLLVM_LINK_LLVM_DYLIB=ON. We pass that while building LLVM, so it gets passed on to clang but I suppose passing it to clang build might be sufficient to reproduce the problem.
...

cmake -Sllvm -B/tmp/out/play -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS='llvm;clang' -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
ninja -C /tmp/out/play all

works on my Debian testing machine at main (without any patch).


If I enable -DLLVM_TOOL_LLVM_DRIVER_BUILD=on, there is an incompatibility due to some cl::opt options registered more than once.
This is a classical error. I think at this point -DLLVM_TOOL_LLVM_DRIVER_BUILD=on users are not supposed to use -DLLVM_BUILD_LLVM_DYLIB=ON,
but it'll be useful to fix this to get more size reduction.

% ninja -C /tmp/out/play bin/llvm
% /tmp/out/play/bin/llvm clang --help
: CommandLine Error: Option 'arm-implicit-it' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
[1]    160970 IOT instruction  /tmp/out/play/bin/llvm clang --help

LLVM_TOOL_LLVM_DRIVER_BUILD defaults to off, so this should be fine.
Interested distributors can help fix the issue so that they can possibly use LLVM_TOOL_LLVM_DRIVER_BUILD for the 16.0.0 release.

If you didn't apply my patch, then it works from in-tree build because it uses static libs instead of the dylib.

The actual problem is that you're passing ${USE_SHARED} to llvm_config() (in add_llvm_executable) before it's defined. Moving the definition earlier fixed this error but uncovers another problem:

Thanks for looking into it more. As for what @MaskRay said this mode won't work with LLVM_TOOL_LLVM_DRIVER_BUILD=On and it looks like because we were using this variable before setting, it was always empty anyway. For the reproducer that you gave the following diff works. Do you want to try this diff with the full gentoo build that was breaking? If it works then we can commit this.

diff --git a/llvm/cmake/modules/AddLLVM.cmake b/llvm/cmake/modules/AddLLVM.cmake
index 8e1385e90b82..5563a7848ec3 100644
--- a/llvm/cmake/modules/AddLLVM.cmake
+++ b/llvm/cmake/modules/AddLLVM.cmake
@@ -909,7 +909,7 @@ macro(add_llvm_executable name)
     
     set_property(GLOBAL APPEND PROPERTY LLVM_DRIVER_TOOLS ${name})
     target_link_libraries(${obj_name} ${LLVM_PTHREAD_LIB})
-    llvm_config(${obj_name} ${USE_SHARED} ${LLVM_LINK_COMPONENTS} )
+    llvm_config(${obj_name} ${LLVM_LINK_COMPONENTS})
   endif()
 
   add_windows_version_resource_file(ALL_FILES ${ALL_FILES})

Unsurprisingly, it fails just the same because using an undefined variable is equivalent to not using it at all, so once again you're trying to link to static libraries that are *not there*. When dylib linking is used, you can't link to static libraries here.

Ok, I've made D130158 as an absolutely minimal change that seems to fix it for me. Basically, I'm skipping all the driver-related logic that's not strictly necessary when LLVM_TOOL_LLVM_DRIVER_BUILD is off. The driver remains broken when dylib is used but at least it shouldn't affect people who don't enable it now.