[CMake] Support multi-target runtimes build

[phosek]

Summary

This changes adds support for building runtimes for multiple different targets using LLVM runtimes directory.

The implementation follow the model used already by the builtins build which already supports this option. To specify the runtimes targets to be built, use the LLVM_RUNTIME_TARGETS variable, where the valuae is the list of targets to build runtimes for. To pass a per target variable to the runtimes build, you can set RUNTIMES_<target>_<variable> where <variable> will be passed to the runtimes build for <target>.

[phosek]

Example use case of this feature is D32817.

[phosek]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

I'm not sure on what's the best way to handle the install prefix. When building runtimes for multiple targets, only one of them can use the "default" install prefix, others have to use a different one otherwise they'll overwrite each other files.

[phosek]

ping

[jroelofs]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

I'd shove them all under: lib/clang-runtimes/$triple/{lib,include}, and specifically not use a "default" install prefix.

[aoli]

Hi I did some tests about this patch and I got one question.

Due to include(HandleLLVMOptions) compiling runtimes will require libc++ for targets. If we don't have libc++ the test in cmake/modules/CheckCompilerVersion.cmake: 38 will fail. Since compiling compiler-rt does not require libc++ do we really need to include this file here? This not a problem for building runtimes for host because libc++ is always installed on hosts. But targets may not have libc++.

[aoli]

runtimes/CMakeLists.txt

329

Shall we also pass CMAKE_BUILD_TYPE here?

[jroelofs]

runtimes/CMakeLists.txt

329

@aoli making the runtimes have the same build type as the compiler? No. That should only be passed via RUNTIMES_${target}_CMAKE_BUILD_TYPE=

[phosek]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

Now they will be installed in $CMAKE_INSTALL_PREFIX/$target, but I'd be happy to change that to lib/runtimes/$target. One issue I've noticed is that sanitizer and xray headers end up in lib/runtimes/$target/lib/clang/5.0.0/include. I don't know if those headers are target specific, but if not it would be better to install them in lib/clang/5.0.0/include same as for compiler headers.

[beanz]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

The sanitizer headers are not (to my knowledge) sanitizer specific, so we should be able to install them once to /lib/clang/$version/include.

For the general target library path I think we need a more complicated solution if we want the install rules to be useful.

Static archives from compiler-rt should install to /lib/clang/$version/lib/$sys/, and shared libraries (and static archives from other runtime projects) should install to /lib/$target/, except in the case where $target == host, in which case it should just be /lib.

I'm not entirely sure how complicated this would be to implement, but I expect it will require changes to compiler-rt's build system to differentiate install paths of static and shared libraries.

[phosek]

I realized that the current solution isn't going to work for compiler-rt since Clang driver expects the sanitizer libraries to end up in /lib/clang/$version/lib/$sys/. The issues is that compiler-rt already has it's own way of handling different targets, so we should use it rather than having to modify both the compiler-rt build and Clang driver. However, we need a solution for other libraries,: libunwind, libc++abi and libc++. One option would be to build compiler-rt runtimes separately, similarly to builtins build and then build the rest of libraries using the runtimes/CMakeLists.txt file (what this change does) and use the CMAKE_INSTALL_PREFIX to specify the installation target. Alternative would be to add a new variable to libunwind's, libc++abi's and libc++'s CMakeLists.txt that will allow overriding the installation path; compiler-rt already uses such variable: COMPILER_RT_INSTALL_PATH so we could probably use <RUNTIME>_INSTALL_PATH. Do you have any preference or opinion about any of these solutions?

[phosek]

I have implemented the latter solution, adding <RUNTIME>_INSTALL_PATH to libunwind, libc++abi, libc++ and it's working. The headers in this case are installed to /lib/$target/include and libraries to /lib/$target/lib, except for compiler-rt where headers are installed to /lib/clang/$version/include and libraries to /lib/clang/$version/lib as today. The changes required to Clang drivers are going to be similar to D32613. It's little strange for headers to be /lib, but this is already the case for compiler-rt today, so this solution follows the same pattern. I'd be happy to upload all the patches if this solution is fine with you.

Splitting libraries and headers, e.g. into /lib/$target and /include/$target, would require introducing pair of variables to each runtime, e.g. <RUNTIME>_HEADER_INSTALL_PATH and <RUNTIME>_LIBRARY_INSTALL_PATH. Unfortunately, I don't think we can shared headers between targets because libc++ generates target specific __config file (all other headers should be the same).

[phosek]

Uploaded the changes to runtimes as D33760, D33761 and D33762.

[jroelofs]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

@beanz what about shared libs from the builtins build? shouldn't those still end up in /lib/clang/$version/lib/$sys/, since they're tied to the specific version of the compiler, unlike the rest of the runtime bits?

[beanz]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

Presently we don't support shared libs from the builtins build at all. If we did, you'd likely want them to behave like the shared libraries from other runtime projects because you'd want them to be part of the sysroot for the cross target. At least, that's how I would assume you'd want them to work.

[phosek]

I've added a proper handling of check- and install- targets from subcomponents which were broken in the previous version as @mcgrathr pointed out while testing this change. This version seems to be working fine in my testing when combined with D33760, D33761, D33762, D32817 and D32613.

[jroelofs]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

good point.

[phosek]

cmake/modules/LLVMExternalProjectUtils.cmake

185 ↗

(On Diff #97684)

I looked into different runtimes, libunwind, libc++abi and libc++ all define ${runtime}_LIBRARY_PATH which could be used to install libraries for different targets into /lib/$target. Most headers are not target-specific, the only exception being __config which for some targets is generated by concatenating generated __config_site with __config. There're ways around this. One would be to completely ignore such targets. The other would be to surround the generated __config_site content with #if defined(...)/#endif block. This is going to require some libc++ changes though. Separating static and shared libraries in compiler-rt should be relatively straightforward: all runtimes use add_compiler_rt_runtime CMake function which sets both ARCHIVE and LIBRARY destination in the install target to COMPILER_RT_LIBRARY_INSTALL_DIR, so we just need to introduce another variable, e.g. COMPILER_RT_ARCHIVE_INSTALL_DIR. We would also need to modify Clang ToolChain; getCompilerRT currently assumes that both static and shared libraries are in the same directory, so we would need to change that.

This solution would require more changes in different components than the current version, but most of them (except maybe the libc++ __config_site) are pretty straightforward. If you think this is a cleaner solution, I'd be happy to go ahead and implement those changes, but I'd like to finally make some progress on this change. Do you think it'd make sense to start a thread on llvm-dev to make sure everyone is on board?

[phosek]

@beanz do you have any comments about the CMake implementation bits?

[beanz]

One question inline below.

cmake/modules/LLVMExternalProjectUtils.cmake

198 ↗

(On Diff #102255)

I'm a little confused why this is needed. Isn't ARG_EXTRA_TARGETS a list? This code looks like it is expecting ARG_EXTRA_TARGETS to be a list of lists where the inner list is colon-separated instead of semi-colon separated.

[phosek]

cmake/modules/LLVMExternalProjectUtils.cmake

198 ↗

(On Diff #102255)

Yes, correct. The problem is that llvm_ExternalProject_Add adds a custom target with the ${target} name that triggers a build for external project's ${target} target. In a single target scenario, this is fine because both targets would be something like install-asan, but in multi-target case we need to use a different target name in the main project to avoid collision (e.g. x86_64-unknown-linux-gnu-install-asan) and different in the subproject (i.e. install-asan), so I use : to optionally separate subproject target name from the target name. I don't know what would be the best way to do this, I also considered using global variables (.e.g <target>_TARGET_NAME but that feels icky) or maybe an additional argument?

[beanz]

CMake stuff all looks good to me!

cmake/modules/LLVMExternalProjectUtils.cmake

198 ↗

(On Diff #102255)

Ok. I understand what you're doing here, and it makes sense.

[phosek]

Thanks! Would it be also possible to look at D33760, D33761 and D33762 which are needed for this change?

[phosek]

Updated the patch to use the relative <runtime>_INSTALL_PREFIX property.

[phosek]

Rebased on top of the latest runtimes build change.

runtimes/CMakeLists.txt

if(${CMAKE_SOURCE_DIR} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR})
 
   # LLVMConfig.cmake contains a bunch of CMake variables from the LLVM build.
   # This file is installed as part of LLVM distributions, so this can be used
   # either from a build directory or an installed LLVM.
   include(LLVMConfig)
 
   # Setting these variables will allow the sub-build to put their outputs into
   # the library and bin directories of the top-level build.
+  if(LLVM_RUNTIMES_TARGET)
+    set(LLVM_LIBRARY_OUTPUT_INTDIR "${LLVM_LIBRARY_DIR}/${LLVM_RUNTIMES_TARGET}")
+    set(LLVM_RUNTIME_OUTPUT_INTDIR "${LLVM_TOOLS_BINARY_DIR}/${LLVM_RUNTIMES_TARGET}")
+  else()
     set(LLVM_LIBRARY_OUTPUT_INTDIR ${LLVM_LIBRARY_DIR})
     set(LLVM_RUNTIME_OUTPUT_INTDIR ${LLVM_TOOLS_BINARY_DIR})
+  endif()
 
   # This variable makes sure that e.g. llvm-lit is found.
   set(LLVM_MAIN_SRC_DIR ${LLVM_BUILD_MAIN_SRC_DIR})
 
   if(APPLE)
     set(LLVM_ENABLE_LIBCXX ON CACHE BOOL "")
   endif()
 
   set(SAFE_CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS})
   set(SAFE_CMAKE_REQUIRED_LIBRARIES ${CMAKE_REQUIRED_LIBRARIES})
 
   include(CheckLibraryExists)
   include(CheckCCompilerFlag)
 
foreach(entry ${runtimes})
     get_filename_component(projName ${entry} NAME)
 
     # TODO: Clean this up as part of an interface standardization
     string(REPLACE "-" "_" canon_name ${projName})
     string(TOUPPER ${canon_name} canon_name)
     # The subdirectories need to treat this as standalone builds
     set(${canon_name}_STANDALONE_BUILD On)
 
+    if(LLVM_RUNTIMES_TARGET)
+      if(NOT "${entry}" MATCHES "compiler-rt")
+        set(${canon_name}_INSTALL_PATH "${CMAKE_INSTALL_PREFIX}/lib/${LLVM_RUNTIMES_TARGET}" CACHE PATH "" FORCE)
+      endif()
+    endif()
+
     # Setting a variable to let sub-projects detect which other projects
     # will be included under here.
     set(HAVE_${canon_name} On)
   endforeach()
 
   # We do this in two loops so that HAVE_* is set for each runtime before the
   # other runtimes are added.
   foreach(entry ${runtimes})
if(NOT LLVM_BUILTIN_TARGETS)
                                           -DCMAKE_ASM_COMPILER_TARGET=${TARGET_TRIPLE}
                                PASSTHROUGH_PREFIXES COMPILER_RT
                                USE_TOOLCHAIN
                                ${EXTRA_ARGS})
     else()
       get_cmake_property(variableNames VARIABLES)
       add_custom_target(builtins)
       foreach(target ${LLVM_BUILTIN_TARGETS})
+        if(target STREQUAL "default")
+          set(target ${LLVM_DEFAULT_TARGET_TRIPLE})
+        endif()
+
         string(REPLACE "-" ";" builtin_target_list ${target})
         foreach(item ${builtin_target_list})
           string(TOLOWER "${item}" item_lower)
           if(item_lower MATCHES "darwin")
             message(FATAL_ERROR "LLVM_BUILTIN_TARGETS isn't implemented for Darwin platform!")
           endif()
         endforeach()
 
foreach(entry ${runtimes})
     string(FIND ${projName} "lib" LIB_IDX)
     if(LIB_IDX EQUAL 0)
       string(SUBSTRING ${projName} 3 -1 projName)
     endif()
     list(APPEND runtime_names ${projName})
   endforeach()
 
   if(runtimes)
-
+    # Create a runtimes target that uses this file as its top-level CMake file.
+    # The runtimes target is a configuration of all the runtime libraries
+    # together in a single CMake invocaiton.
+    if(NOT LLVM_RUNTIME_TARGETS)
       foreach(runtime_name ${runtime_names})
         list(APPEND extra_targets
           ${runtime_name}
           install-${runtime_name}
           check-${runtime_name})
       endforeach()
 
       if(LLVM_INCLUDE_TESTS)
         set(test_targets runtimes-test-depends check-runtimes)
         foreach(component ${SUB_COMPONENTS})
           list(APPEND SUB_COMPONENT_CHECK_TARGETS check-${component})
         endforeach()
       endif()
 
-    # Create a runtimes target that uses this file as its top-level CMake file.
-    # The runtimes target is a configuration of all the runtime libraries
-    # together in a single CMake invocaiton.
       llvm_ExternalProject_Add(runtimes
                                ${CMAKE_CURRENT_SOURCE_DIR}
                                DEPENDS ${deps}
                                # Builtins were built separately above
                                CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off
                                           -DLLVM_INCLUDE_TESTS=${LLVM_INCLUDE_TESTS}
                                PASSTHROUGH_PREFIXES ${prefixes}
                                EXTRA_TARGETS ${extra_targets}
                                               ${test_targets}
                                               ${SUB_COMPONENTS}
                                               ${SUB_COMPONENT_CHECK_TARGETS}
                                               ${SUB_INSTALL_TARGETS}
                                USE_TOOLCHAIN
                                ${EXTRA_ARGS})
-    
+    else()
+      get_cmake_property(variableNames VARIABLES)
+      add_custom_target(runtimes)
+      add_custom_target(install-runtimes)
+      if(LLVM_INCLUDE_TESTS)
+        add_custom_target(check-runtimes)
+        add_custom_target(runtimes-test-depends)
+      endif()
+      foreach(target ${LLVM_RUNTIME_TARGETS})
+        if(target STREQUAL "default")
+          set(target ${LLVM_DEFAULT_TARGET_TRIPLE})
+        endif()
+
+        foreach(runtime_name ${runtime_names})
+          list(APPEND ${target}_extra_targets
+            ${runtime_name}-${target}
+            install-${runtime_name}-${target}
+            check-${runtime_name}-${target})
+        endforeach()
+
+        if(LLVM_INCLUDE_TESTS)
+          set(${target}_test_targets runtimes-test-depends-${target} check-runtimes-${target})
+          foreach(component ${SUB_COMPONENTS})
+            list(APPEND SUB_COMPONENT_CHECK_TARGETS check-${component}-${target})
+          endforeach()
+        endif()
+
+        foreach(variableName ${variableNames})
+          if(variableName MATCHES "^RUNTIMES_${target}")
+            string(REPLACE "RUNTIMES_${target}_" "" new_name ${variableName})
+            list(APPEND ${target}_extra_args "-D${new_name}=${${variableName}}")
+          endif()
+        endforeach()
+
+        if(NOT target STREQUAL LLVM_DEFAULT_TARGET_TRIPLE)
+          list(APPEND ${target}_extra_args "-DLLVM_RUNTIMES_TARGET=${target}")
+        endif()
+
+        llvm_ExternalProject_Add(runtimes-${target}
+                                 ${CMAKE_CURRENT_SOURCE_DIR}
+                                 DEPENDS ${deps}
+                                 # Builtins were built separately above
+                                 CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off
+                                            -DLLVM_INCLUDE_TESTS=${LLVM_INCLUDE_TESTS}
+                                            -DCMAKE_C_COMPILER_TARGET=${target}
+                                            -DCMAKE_CXX_COMPILER_TARGET=${target}
+                                            -DCMAKE_ASM_COMPILER_TARGET=${target}
+                                            -DCMAKE_C_COMPILER_WORKS=ON
+                                            -DCMAKE_CXX_COMPILER_WORKS=ON
+                                            -DCMAKE_ASM_COMPILER_WORKS=ON
+                                            ${${target}_extra_args}
+                                 PASSTHROUGH_PREFIXES ${prefixes}
+                                 EXTRA_TARGETS ${${target}_extra_targets}
+                                               ${${target}_test_targets}
+                                               ${SUB_COMPONENTS}
+                                               ${SUB_COMPONENT_CHECK_TARGETS}
+                                               ${SUB_INSTALL_TARGETS}
+                                 USE_TOOLCHAIN
+                                 ${EXTRA_ARGS})
+
+        add_dependencies(runtimes runtimes-${target})
+        add_dependencies(install-runtimes install-runtimes-${target})
+        if(LLVM_INCLUDE_TESTS)
+          add_dependencies(check-runtimes check-runtimes-${target})
+          add_dependencies(runtimes-test-depends runtimes-test-depends-${target})
+        endif()
+      endforeach()
+    endif()
     # TODO: This is a hack needed because the libcxx headers are copied into the
     # build directory during configuration. Without that step the clang in the
     # build directory cannot find the C++ headers in certain configurations.
     # I need to build a mechanism for runtime projects to provide CMake code
     # that executes at LLVM configuration time to handle this case.
     if(NOT LLVM_BUILD_INSTRUMENTED AND CLANG_ENABLE_BOOTSTRAP)
       add_dependencies(clang-bootstrap-deps runtimes-configure)
     endif()
 
     if(LLVM_INCLUDE_TESTS)
       set_property(GLOBAL APPEND PROPERTY LLVM_ADDITIONAL_TEST_DEPENDS runtimes-test-depends)
       set_property(GLOBAL APPEND PROPERTY LLVM_ADDITIONAL_TEST_TARGETS check-runtimes)
     endif()
   endif()
 endif()