CMAKE: Add flags to skip building NATIVE tools

[martell]

Summary

When cross compiling llvm and clang there is a native step where we build
llvm-config llvm-tblgen and clang-tblgen for the host compiler.

When someone in doing this in a stage process they may already have these tools and wish to opt to use the existing ones instead.
This adds support for 3 new optional cmake flags to disable building of the native steps with the expectation of having them on the PATH already.

HAVE_NATIVE_LLVM_CONFIG
HAVE_NATIVE_LLVM_TABLEGEN
HAVE_NATIVE_CLANG_TABLEGEN

Example.

Stage -1 - install gcc or clang for host
Stage 0 - trust untrust
Stage 1 - build host clang and llvm and cross compile a crt, compiler-rt, libunwind, libcxx etc for a different target than the host using compiler we just built
Stage 2 - cross compile llvm and clang using the new target support added in stage 1 and using the native tools built in stage 1.

These defines come in handy in stage 2, removing the need to build these tools twice.

[hintonda]

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

It might also be nice to add a test for all three before setting LLVM_USE_HOST_TOOLS which will prevent configuring NATIVE tools altogether if not needed.

cmake/modules/TableGen.cmake

150

This line already tests whether or not to build a new NATIVE tblgen, so I don't think you need to change this file. Passing -D(CLANG|LLVM)_TABLEGEN should just work here -- or explicitly adding it to the cache in your toolchain file.

tools/llvm-config/CMakeLists.txt

67

Instead of a new flag, can't you just add AND NOT DEFINED ${project}_LLVM_CONFIG_EXE?

[martell]

In D41580#964147, @hintonda wrote:

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

I was originally trying to use -D(CLANG|LLVM)_TABLEGEN but clang would complain about no rule to build a target.
This either was fixed very very recently (like in the last day or two) or the following set of extra flags fix this

-DCMAKE_SYSTEM_PROGRAM_PATH=$TOOLCHAIN_PREFIX/bin
-DCMAKE_FIND_ROOT_PATH=$TOOLCHAIN_PREFIX
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY

I can probably drop that part of the patch now, though I would like to know how it was fixed.

When building llvm project out of tree you are able to specify -DLLVM_CONFIG_PATH, this is true for both lld and clang on a quick look.
It would probably be better to use LLVM_CONFIG_PATH to keep the flags consistent across projects rather than using _LLVM_CONFIG_EXE

[hintonda]

In D41580#964323, @martell wrote:

In D41580#964147, @hintonda wrote:

I like your idea, but can't you do this without adding new flags?

You already have to pass -D(CLANG|LLVM)_TABLEGEN which prevents them from getting built. llvm-config is a little different in that you need to pass -D_LLVM_CONFIG_EXE since project is undefined in tools/llvm-config, but adding a test for ${project}_LLVM_CONFIG_EXE is consistent and should work as well. Perhaps this needs to be refactored as well, so you can pass -DLLVM_CONFIG_EXE instead of -D_LLVM_CONFIG_EXE.

I was originally trying to use -D(CLANG|LLVM)_TABLEGEN but clang would complain about no rule to build a target.
This either was fixed very very recently (like in the last day or two) or the following set of extra flags fix this

-DCMAKE_SYSTEM_PROGRAM_PATH=$TOOLCHAIN_PREFIX/bin
-DCMAKE_FIND_ROOT_PATH=$TOOLCHAIN_PREFIX
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY

I can probably drop that part of the patch now, though I would like to know how it was fixed.

I don't think any of this has changed recently, and can't reproduce the problem you mention. However, the check uses STREQUAL, so you must add a path when setting (CLANG|LLVM)_TABLEGEN. If you passed llvm-config, it would succeed and build the NATIVE version. That bit of code could be refactored to respect the -D flag.

Also, the CMAKE_FIND_* variables only affect how the find_xxx* functions work, so I don't think they make any difference in this context.

When building llvm project out of tree you are able to specify -DLLVM_CONFIG_PATH, this is true for both lld and clang on a quick look.
It would probably be better to use LLVM_CONFIG_PATH to keep the flags consistent across projects rather than using _LLVM_CONFIG_EXE

I don't see LLVM_CONFIG_PATH being used anywhere. I see examples of where it's set, but not used. Could you give me a pointer to where it is used?

Btw, use of ${project} variable in llvm/tools/llvm-config/CMakeLists.txt looks like a bug to me, in the sense that it is never defined and just melts away. Perhaps removing it and the _ prefix from _LLVM_CONFIG_EXE would simplify everything.

[hintonda]

Btw, I just submitted a separate patch, D41806, that removes the LLVM_CONFIG_EXE completely since it's unused.

[martell]

tools/llvm-config/CMakeLists.txt

67

This seems better handled in D41580

67

D41806

[shi-yan]

I set both ${${project}_TABLEGEN when building clang, but it still tries to build the native tools.

cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_TOOLCHAIN_FILE=/work/emsdk/fastcomp/emscripten/cmake/Modules/Platform/Emscripten.cmake -DLLVM_ENABLE_DUMP=OFF -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_TARGETS_TO_BUILD= -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD=WebAssembly -DLLVM_BUILD_TOOLS=OFF -DLLVM_ENABLE_THREADS=OFF -DLLVM_BUILD_LLVM_DYLIB=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_ENABLE_EXPENSIVE_CHECKS=OFF -DLLVM_ENABLE_BACKTRACES=OFF -DLLVM_TABLEGEN=/work/llvm-native/build/bin/llvm-tblgen -DCLANG_TABLEGEN=/work/llvm-native/build/bin/clang-tblgen -DLLVM_USE_HOST_TOOLS=on -DLLVM_BUILD_UTILS=off -G "Unix Makefiles" ../llvm

cmake/modules/TableGen.cmake

if(${LLVM_TABLEGEN} STREQUAL tblgen)
           FORCE)
     endif()
   endif()
 
   # Effective tblgen executable to be used:
   set(${project}_TABLEGEN_EXE ${${project}_TABLEGEN} PARENT_SCOPE)
   set(${project}_TABLEGEN_TARGET ${${project}_TABLEGEN} PARENT_SCOPE)
 
-  if(LLVM_USE_HOST_TOOLS)
+  if(LLVM_USE_HOST_TOOLS AND NOT HAVE_NATIVE_${project}_TABLEGEN)
     if( ${${project}_TABLEGEN} STREQUAL "${target}" )
       if (NOT CMAKE_CONFIGURATION_TYPES)
         set(${project}_TABLEGEN_EXE "${LLVM_NATIVE_BUILD}/bin/${target}")
       else()
         set(${project}_TABLEGEN_EXE "${LLVM_NATIVE_BUILD}/Release/bin/${target}")
       endif()
       set(${project}_TABLEGEN_EXE ${${project}_TABLEGEN_EXE} PARENT_SCOPE)
 
       llvm_ExternalProject_BuildCmd(tblgen_build_cmd ${target}

tools/llvm-config/CMakeLists.txt

if(LLVM_ENABLE_MODULES)
   target_compile_options(llvm-config PUBLIC
     "-fmodules-ignore-macro=CMAKE_CFG_INTDIR"
     )
 endif()
 
 # Add the dependency on the generation step.
 add_file_dependencies(${CMAKE_CURRENT_SOURCE_DIR}/llvm-config.cpp ${BUILDVARIABLES_OBJPATH})
 
-if(CMAKE_CROSSCOMPILING)
+if(CMAKE_CROSSCOMPILING AND NOT HAVE_NATIVE_LLVM_CONFIG)
   set(${project}_LLVM_CONFIG_EXE "${LLVM_NATIVE_BUILD}/bin/llvm-config")
   set(${project}_LLVM_CONFIG_EXE ${${project}_LLVM_CONFIG_EXE} PARENT_SCOPE)
 
   add_custom_command(OUTPUT "${${project}_LLVM_CONFIG_EXE}"
     COMMAND ${CMAKE_COMMAND} --build . --target llvm-config --config $<CONFIGURATION>
     DEPENDS ${LLVM_NATIVE_BUILD}/CMakeCache.txt
     WORKING_DIRECTORY ${LLVM_NATIVE_BUILD}
     COMMENT "Building native llvm-config..."
     USES_TERMINAL)
   add_custom_target(${project}NativeLLVMConfig DEPENDS ${${project}_LLVM_CONFIG_EXE})
   add_dependencies(${project}NativeLLVMConfig CONFIGURE_LLVM_NATIVE)
 
   add_dependencies(llvm-config ${project}NativeLLVMConfig)
 endif(CMAKE_CROSSCOMPILING)