[cmake] Convert the NATIVE llvm build process to be project agnostic

[lanza]

Summary

lldb recently added a tablegen tool. In order to properly cross compile
lldb standalone there needs to be a mechanism to generate the native
lldb build, analgous to what's done for the NATIVE llvm build. Thus,
we can simply modify this setup to allow for any project to be used.

[lanza]

Still testing the builds, just figured I'd put it up early.

[smeenai]

cmake/modules/CrossCompile.cmake

92–98

Does it hurt anything if these flags are included for other projects?

[lanza]

update

[compnerd]

I'm pretty sure that this doesn't work for Visual Studio (where you do not have a cross-compiler).

[lanza]

update

[lanza]

title

[lanza]

I'm pretty sure that this doesn't work for Visual Studio (where you do not have a cross-compiler).

It already doesn't work for Visual Studio. This change doesn't change that.

[lanza]

format

[smeenai]

This just seems like it's replacing all instances of LLVM with ${CMAKE_PROJECT_NAME}?

I think it'd be cleaner and more explicit to pass an additional project name parameter to llvm_create_cross_target instead of implicitly using ${CMAKE_PROJECT_NAME}. There's some prior art for that, e.g. in https://reviews.llvm.org/diffusion/L/browse/llvm/trunk/cmake/modules/AddLLVM.cmake;366290$1009?as=source&blame=off

[smeenai]

cmake/modules/CrossCompile.cmake

63–64

You'll wanna change this one to read from a project-specific cache variable too. You'll also wanna preserve backwards compatibility for the people using the old spelling.

[lanza]

In D64837#1588826, @smeenai wrote:

This just seems like it's replacing all instances of LLVM with ${CMAKE_PROJECT_NAME}?

I think it'd be cleaner and more explicit to pass an additional project name parameter to llvm_create_cross_target instead of implicitly using ${CMAKE_PROJECT_NAME}. There's some prior art for that, e.g. in https://reviews.llvm.org/diffusion/L/browse/llvm/trunk/cmake/modules/AddLLVM.cmake;366290$1009?as=source&blame=off

That's what I did originally. However, the users of build_natve_tool wouldn't have the proper context to know what to use. It's only non-llvm specific usage is within add_tablegen which would then have to learn how to decide what project_name variable to pass in. e.g. if lldb is part of ENABLE_PROJECTS then add_tablegen should call llvm_native_build(LLVM NATIVE ...) else llvm_native_build(LLDB NATIVE ...). So we'd have to propagate that to all users of add_tablegen where we'd have to figure out whether the build is standalone or not via checking CMAKE_PROJECT_NAME anyways. e.g.

 if (${CMAKE_PROJECT_NAME} MATCHES lldb)
    add_tablegen(lldb lldb lldb-tblgen)
else()
    add_tablegen(llvm lldb lldb-tblgen)
endif()

I'm not convinced that adding the granularity to choose your llvm_create_cross_target project is worth propagating the configuration outwards given that there are no users that don't satisfy CMAKE_PROJECT_NAME == project_name.

An alternative would be to have some global variable llvm_cross_targets list that build_native_tool iterates over. But this is making an already clumsy solution more clumsy.

[plotfi]

@smeenai I think if this change works then for now we can just land it because we have several bots to turn green in the short term. Would you be fine with allowing for this fix in the near term?

[beanz]

cmake/modules/CrossCompile.cmake

63–64

I think you probably want to make CROSS_TOOLCHAIN_FLAGS_${target_name} generic, not project specific, and only add a project-specific variant if it is needed.

It is entirely possible to use the cross-compile code to generate multiple different cross-projects for the same target where you do want the same target flags, because those flags relate to the target not the project you're building.

[lanza]

update

[lanza]

typo

[lanza]

adjust for beanz comments

[smeenai]

In D64837#1589983, @lanza wrote:

In D64837#1588826, @smeenai wrote:

This just seems like it's replacing all instances of LLVM with ${CMAKE_PROJECT_NAME}?

I think it'd be cleaner and more explicit to pass an additional project name parameter to llvm_create_cross_target instead of implicitly using ${CMAKE_PROJECT_NAME}. There's some prior art for that, e.g. in https://reviews.llvm.org/diffusion/L/browse/llvm/trunk/cmake/modules/AddLLVM.cmake;366290$1009?as=source&blame=off

That's what I did originally. However, the users of build_natve_tool wouldn't have the proper context to know what to use. It's only non-llvm specific usage is within add_tablegen which would then have to learn how to decide what project_name variable to pass in. e.g. if lldb is part of ENABLE_PROJECTS then add_tablegen should call llvm_native_build(LLVM NATIVE ...) else llvm_native_build(LLDB NATIVE ...). So we'd have to propagate that to all users of add_tablegen where we'd have to figure out whether the build is standalone or not via checking CMAKE_PROJECT_NAME anyways. e.g.

 if (${CMAKE_PROJECT_NAME} MATCHES lldb)
    add_tablegen(lldb lldb lldb-tblgen)
else()
    add_tablegen(llvm lldb lldb-tblgen)
endif()

I'm not convinced that adding the granularity to choose your llvm_create_cross_target project is worth propagating the configuration outwards given that there are no users that don't satisfy CMAKE_PROJECT_NAME == project_name.

An alternative would be to have some global variable llvm_cross_targets list that build_native_tool iterates over. But this is making an already clumsy solution more clumsy.

I missed the build_native_tool thing. In that case, I would have been fine with sticking to ${CMAKE_PROJECT_NAME} inside llvm_create_cross_target as well, but it looks like you've already changed everything to use a project name variable :)

cmake/modules/CrossCompile.cmake

27

Are you missing an underscore here? Should the variable be CROSS_TOOLCHAIN_FLAGS_${project_name}_${target_name}?

63–64

Makes sense. I can see there being a need for project-specific variables as well.

@lanza you should definitely keep the original ${CROSS_TOOLCHAIN_FLAGS_${target_name}} spelling here. You may also want to add the ${CROSS_TOOLCHAIN_FLAGS_${project_name}_${target_name}} spelling if e.g. there's some LLDB-specific configurations you'll want to pass? (You could just use the generic ${CROSS_TOOLCHAIN_FLAGS_${target_name}} for project-specific configs as well; it's slightly ugly, but as long as the configs don't cause issues in other projects it should be fine.)

[lanza]

typo

[smeenai]

LGTM

CMakeLists.txt

 
 set(CMAKE_INCLUDE_CURRENT_DIR ON)
 
 include_directories( ${LLVM_INCLUDE_DIR} ${LLVM_MAIN_INCLUDE_DIR})
 
 # when crosscompiling import the executable targets from a file
 if(LLVM_USE_HOST_TOOLS)
   include(CrossCompile)
+  llvm_create_cross_target(LLVM NATIVE "" Release)
 endif(LLVM_USE_HOST_TOOLS)
 if(LLVM_TARGET_IS_CROSSCOMPILE_HOST)
 # Dummy use to avoid CMake Warning: Manually-specified variables were not used
 # (this is a variable that CrossCompile sets on recursive invocations)
 endif()
 
 if(${CMAKE_SYSTEM_NAME} MATCHES "(FreeBSD|DragonFly)")
   # On FreeBSD, /usr/local/* is not used by default. In order to build LLVM

cmake/modules/CrossCompile.cmake

 include(LLVMExternalProjectUtils)
 
-function(llvm_create_cross_target_internal target_name toolchain buildtype)
 
-  if(NOT DEFINED LLVM_${target_name}_BUILD)
-    set(LLVM_${target_name}_BUILD "${CMAKE_BINARY_DIR}/${target_name}")
-    set(LLVM_${target_name}_BUILD ${LLVM_${target_name}_BUILD} PARENT_SCOPE)
-    message(STATUS "Setting native build dir to " ${LLVM_${target_name}_BUILD})
-  endif(NOT DEFINED LLVM_${target_name}_BUILD)
+function(llvm_create_cross_target project_name target_name toolchain buildtype)
+
+  if(NOT DEFINED ${project_name}_${target_name}_BUILD)
+    set(${project_name}_${target_name}_BUILD
+      "${CMAKE_BINARY_DIR}/${target_name}")
+    set(${project_name}_${target_name}_BUILD
+      ${${project_name}_${target_name}_BUILD} PARENT_SCOPE)
+    message(STATUS "Setting native build dir to " ${${project_name}_${target_name}_BUILD})
+  endif(NOT DEFINED ${project_name}_${target_name}_BUILD)
 
   if (EXISTS ${LLVM_MAIN_SRC_DIR}/cmake/platforms/${toolchain}.cmake)
     set(CROSS_TOOLCHAIN_FLAGS_INIT
       -DCMAKE_TOOLCHAIN_FILE=\"${LLVM_MAIN_SRC_DIR}/cmake/platforms/${toolchain}.cmake\")
   else()
     set(CROSS_TOOLCHAIN_FLAGS_INIT
       -DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
       -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
       )
   endif()
   set(CROSS_TOOLCHAIN_FLAGS_${target_name} ${CROSS_TOOLCHAIN_FLAGS_INIT}
     CACHE STRING "Toolchain configuration for ${target_name}")
 
+  # project specific version of the flags up above
+  set(CROSS_TOOLCHAIN_FLAGS_${project_name}_${target_name} ""
+    CACHE STRING "Toolchain configuration for ${Pproject_name}_${target_name}")
+
   if (buildtype)
     set(build_type_flags "-DCMAKE_BUILD_TYPE=${buildtype}")
   endif()
   if (LLVM_USE_LINKER AND NOT CMAKE_CROSSCOMPILING)
     set(linker_flag "-DLLVM_USE_LINKER=${LLVM_USE_LINKER}")
   endif()
   if (LLVM_EXTERNAL_CLANG_SOURCE_DIR)
     # Propagate LLVM_EXTERNAL_CLANG_SOURCE_DIR so that clang-tblgen can be built
     set(external_clang_dir "-DLLVM_EXTERNAL_CLANG_SOURCE_DIR=${LLVM_EXTERNAL_CLANG_SOURCE_DIR}")
   endif()
 
-  add_custom_command(OUTPUT ${LLVM_${target_name}_BUILD}
-    COMMAND ${CMAKE_COMMAND} -E make_directory ${LLVM_${target_name}_BUILD}
-    COMMENT "Creating ${LLVM_${target_name}_BUILD}...")
+  add_custom_command(OUTPUT ${${project_name}_${target_name}_BUILD}
+    COMMAND ${CMAKE_COMMAND} -E make_directory ${${project_name}_${target_name}_BUILD}
+    COMMENT "Creating ${${project_name}_${target_name}_BUILD}...")
 
-  add_custom_target(CREATE_LLVM_${target_name}
-                    DEPENDS ${LLVM_${target_name}_BUILD})
+  add_custom_target(CREATE_${project_name}_${target_name}
+    DEPENDS ${${project_name}_${target_name}_BUILD})
 
   # Escape semicolons in the targets list so that cmake doesn't expand
   # them to spaces.
   string(REPLACE ";" "$<SEMICOLON>" targets_to_build_arg
          "${LLVM_TARGETS_TO_BUILD}")
   string(REPLACE ";" "$<SEMICOLON>" experimental_targets_to_build_arg
          "${LLVM_EXPERIMENTAL_TARGETS_TO_BUILD}")
 
   string(REPLACE ";" "$<SEMICOLON>" llvm_enable_projects_arg
          "${LLVM_ENABLE_PROJECTS}")
   string(REPLACE ";" "$<SEMICOLON>" llvm_external_projects_arg
          "${LLVM_EXTERNAL_PROJECTS}")
 
-  add_custom_command(OUTPUT ${LLVM_${target_name}_BUILD}/CMakeCache.txt
+  add_custom_command(OUTPUT ${${project_name}_${target_name}_BUILD}/CMakeCache.txt
     COMMAND ${CMAKE_COMMAND} -G "${CMAKE_GENERATOR}"
         -DCMAKE_MAKE_PROGRAM="${CMAKE_MAKE_PROGRAM}"
         ${CROSS_TOOLCHAIN_FLAGS_${target_name}} ${CMAKE_SOURCE_DIR}
+        ${CROSS_TOOLCHAIN_FLAGS_${project_name}_${target_name}}
         -DLLVM_TARGET_IS_CROSSCOMPILE_HOST=TRUE
         -DLLVM_TARGETS_TO_BUILD="${targets_to_build_arg}"
         -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="${experimental_targets_to_build_arg}"
         -DLLVM_DEFAULT_TARGET_TRIPLE="${TARGET_TRIPLE}"
         -DLLVM_TARGET_ARCH="${LLVM_TARGET_ARCH}"
         -DLLVM_ENABLE_PROJECTS="${llvm_enable_projects_arg}"
         -DLLVM_EXTERNAL_PROJECTS="${llvm_external_projects_arg}"
         -DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN="${LLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN}"
         ${build_type_flags} ${linker_flag} ${external_clang_dir}
-    WORKING_DIRECTORY ${LLVM_${target_name}_BUILD}
-    DEPENDS CREATE_LLVM_${target_name}
-    COMMENT "Configuring ${target_name} LLVM...")
+        ${ARGN}
+    WORKING_DIRECTORY ${${project_name}_${target_name}_BUILD}
+    DEPENDS CREATE_${project_name}_${target_name}
+    COMMENT "Configuring ${target_name} ${project_name}...")
 
-  add_custom_target(CONFIGURE_LLVM_${target_name}
-                    DEPENDS ${LLVM_${target_name}_BUILD}/CMakeCache.txt)
+  add_custom_target(CONFIGURE_${project_name}_${target_name}
+    DEPENDS ${${project_name}_${target_name}_BUILD}/CMakeCache.txt)
 
 endfunction()
 
-function(llvm_create_cross_target target_name sysroot)
-  llvm_create_cross_target_internal(${target_name} ${sysroot} ${CMAKE_BUILD_TYPE})
-endfunction()
-
 # Sets up a native build for a tool, used e.g. for cross-compilation and
 # LLVM_OPTIMIZED_TABLEGEN. Always builds in Release.
 # - target: The target to build natively
 # - output_path_var: A variable name which receives the path to the built target
 # - DEPENDS: Any additional dependencies for the target
 function(build_native_tool target output_path_var)
   cmake_parse_arguments(ARG "" "" "DEPENDS" ${ARGN})
 
   if(CMAKE_CONFIGURATION_TYPES)
-    set(output_path "${LLVM_NATIVE_BUILD}/Release/bin/${target}")
+    set(output_path "${${CMAKE_PROJECT_NAME}_NATIVE_BUILD}/Release/bin/${target}")
   else()
-    set(output_path "${LLVM_NATIVE_BUILD}/bin/${target}")
+    set(output_path "${${CMAKE_PROJECT_NAME}_NATIVE_BUILD}/bin/${target}")
   endif()
 
-  llvm_ExternalProject_BuildCmd(build_cmd ${target} ${LLVM_NATIVE_BUILD}
+  llvm_ExternalProject_BuildCmd(build_cmd ${target} ${${CMAKE_PROJECT_NAME}_NATIVE_BUILD}
                                 CONFIGURATION Release)
   add_custom_command(OUTPUT "${output_path}"
                      COMMAND ${build_cmd}
-                     DEPENDS CONFIGURE_LLVM_NATIVE ${ARG_DEPENDS}
-                     WORKING_DIRECTORY "${LLVM_NATIVE_BUILD}"
+                     DEPENDS CONFIGURE_${CMAKE_PROJECT_NAME}_NATIVE ${ARG_DEPENDS}
+                     WORKING_DIRECTORY "${${CMAKE_PROJECT_NAME}_NATIVE_BUILD}"
                      COMMENT "Building native ${target}..."
                      USES_TERMINAL)
   set(${output_path_var} "${output_path}" PARENT_SCOPE)
 endfunction()
-
-llvm_create_cross_target_internal(NATIVE "" Release)