This is an archive of the discontinued LLVM Phabricator instance.

CMAKE: Add flags to skip building NATIVE tools
AbandonedPublic

Authored by martell on Dec 26 2017, 3:09 AM.

Details

Reviewers
compnerd
rnk
pcc
Summary

When cross compiling llvm and clang there is a native step where we build
llvm-config llvm-tblgen and clang-tblgen for the host compiler.

When someone in doing this in a stage process they may already have these tools and wish to opt to use the existing ones instead.
This adds support for 3 new optional cmake flags to disable building of the native steps with the expectation of having them on the PATH already.

HAVE_NATIVE_LLVM_CONFIG
HAVE_NATIVE_LLVM_TABLEGEN
HAVE_NATIVE_CLANG_TABLEGEN

Example.

Stage -1 - install gcc or clang for host
Stage 0 - trust untrust
Stage 1 - build host clang and llvm and cross compile a crt, compiler-rt, libunwind, libcxx etc for a different target than the host using compiler we just built
Stage 2 - cross compile llvm and clang using the new target support added in stage 1 and using the native tools built in stage 1.

These defines come in handy in stage 2, removing the need to build these tools twice.

Diff Detail

Repository
rL LLVM

Event Timeline

martell created this revision.Dec 26 2017, 3:09 AM
martell edited the summary of this revision. (Show Details)Dec 26 2017, 3:19 AM
martell edited the summary of this revision. (Show Details)Dec 26 2017, 3:20 AM

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

It might also be nice to add a test for all three before setting LLVM_USE_HOST_TOOLS which will prevent configuring NATIVE tools altogether if not needed.

cmake/modules/TableGen.cmake
150

This line already tests whether or not to build a new NATIVE tblgen, so I don't think you need to change this file. Passing -D(CLANG|LLVM)_TABLEGEN should just work here -- or explicitly adding it to the cache in your toolchain file.

tools/llvm-config/CMakeLists.txt
67

Instead of a new flag, can't you just add AND NOT DEFINED ${project}_LLVM_CONFIG_EXE?

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

I was originally trying to use -D(CLANG|LLVM)_TABLEGEN but clang would complain about no rule to build a target.
This either was fixed very very recently (like in the last day or two) or the following set of extra flags fix this

-DCMAKE_SYSTEM_PROGRAM_PATH=$TOOLCHAIN_PREFIX/bin
-DCMAKE_FIND_ROOT_PATH=$TOOLCHAIN_PREFIX
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY

I can probably drop that part of the patch now, though I would like to know how it was fixed.

When building llvm project out of tree you are able to specify -DLLVM_CONFIG_PATH, this is true for both lld and clang on a quick look.
It would probably be better to use LLVM_CONFIG_PATH to keep the flags consistent across projects rather than using _LLVM_CONFIG_EXE

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

I was originally trying to use -D(CLANG|LLVM)_TABLEGEN but clang would complain about no rule to build a target.
This either was fixed very very recently (like in the last day or two) or the following set of extra flags fix this

-DCMAKE_SYSTEM_PROGRAM_PATH=$TOOLCHAIN_PREFIX/bin
-DCMAKE_FIND_ROOT_PATH=$TOOLCHAIN_PREFIX
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY

I can probably drop that part of the patch now, though I would like to know how it was fixed.

I don't think any of this has changed recently, and can't reproduce the problem you mention. However, the check uses STREQUAL, so you must add a path when setting (CLANG|LLVM)_TABLEGEN. If you passed llvm-config, it would succeed and build the NATIVE version. That bit of code could be refactored to respect the -D flag.

Also, the CMAKE_FIND_* variables only affect how the find_xxx* functions work, so I don't think they make any difference in this context.

When building llvm project out of tree you are able to specify -DLLVM_CONFIG_PATH, this is true for both lld and clang on a quick look.
It would probably be better to use LLVM_CONFIG_PATH to keep the flags consistent across projects rather than using _LLVM_CONFIG_EXE

I don't see LLVM_CONFIG_PATH being used anywhere. I see examples of where it's set, but not used. Could you give me a pointer to where it is used?

Btw, use of ${project} variable in llvm/tools/llvm-config/CMakeLists.txt looks like a bug to me, in the sense that it is never defined and just melts away. Perhaps removing it and the _ prefix from _LLVM_CONFIG_EXE would simplify everything.

Btw, I just submitted a separate patch, D41806, that removes the LLVM_CONFIG_EXE completely since it's unused.

martell abandoned this revision.Mar 23 2018, 7:56 PM
martell added inline comments.
tools/llvm-config/CMakeLists.txt
67

This seems better handled in D41580

67
shi-yan added a subscriber: shi-yan.Aug 5 2019, 9:03 AM

I set both ${${project}_TABLEGEN when building clang, but it still tries to build the native tools.

cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_TOOLCHAIN_FILE=/work/emsdk/fastcomp/emscripten/cmake/Modules/Platform/Emscripten.cmake -DLLVM_ENABLE_DUMP=OFF -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_TARGETS_TO_BUILD= -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly -DLLVM_BUILD_TOOLS=OFF -DLLVM_ENABLE_THREADS=OFF -DLLVM_BUILD_LLVM_DYLIB=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_ENABLE_EXPENSIVE_CHECKS=OFF -DLLVM_ENABLE_BACKTRACES=OFF -DLLVM_TABLEGEN=/work/llvm-native/build/bin/llvm-tblgen -DCLANG_TABLEGEN=/work/llvm-native/build/bin/clang-tblgen -DLLVM_USE_HOST_TOOLS=on -DLLVM_BUILD_UTILS=off -G "Unix Makefiles" ../llvm
Herald added a project: Restricted Project. · View Herald TranscriptAug 5 2019, 9:03 AM