[PR48898][CMake] Support MinGW Toolchain tools in llvm_ExternalProject_Add

[zero9178]

Summary

Windows is in the unique position of having two drivers, clang-cl and normal GNU clang, depending on whether a GNU or MSVC target is used. The current implementation with the USE_TOOLCHAIN argument assumes that when CMAKE_SYSTEM_NAME is set to Windows that clang-cl should be used, which is the incorrect choice when targeting a GNU environment.

This patch solves the following:

• If the host is MinGW and USE_TOOLCHAIN is passed as an argument, the GNU style drivers should be selected instead of clang-cl
• If one of CMAKE_(C|ASM|CXX)_COMPILER_TARGET has been passed as cmake arguments it checks whether a windows-gnu or windows-msvc target has been requested and selects accordingly

I discovered these issues when trying to do builds of the runtimes using LLVM_ENABLE_RUNTIMES while having a MinGW target as one of the LLVM_RUNTIME_TARGETS.

[mstorsjo]

Windows is in the unique position of having two drivers, clang-cl and normal GNU clang, depending on whether a GNU or MSVC target is used.

FWIW, the normal GNU frontend also works for the MSVC target; some finer details are omitted in that case, but it rougly does work (and if the default target of the build is an msvc target, that's what targeted when calling that frontend too).

Also FWIW, I haven't really used the runtimes build mechanism at all, because I primarily bootstrap everything from scratch in a cross compile setup, i.e. with no preexisting build tools; first building the compiler, then building the base mingw crt, the compiler-rt builtins on top of that, and then other runtimes like libunwind, libcxxabi and libcxx on top of that. In such cases, afaik the runtimes build is a bit tricky, because I'd need to build other bits (e.g. the mingw crt) between building the compiler and the rest of the llvm provided runtimes. For now I just build all of those subprojects individually afterwards.

So therefore all of this is a bit of uncharted territory for me.

There's also a separate contender for how to build runtimes (still not entirely formally supported), as a middle ground - see https://github.com/llvm/llvm-project/blob/main/libcxx/utils/ci/runtimes/CMakeLists.txt. With that one, you configure building the runtimes just like a normal standalone project (i.e. you set the cmake configuration yourself, instead of instructing the machinery to figure it out based on a target triple), but you can add a series of projects to build as e.g. -DLLVM_ENABLE_PROJECTS="libunwind;libcxxabi;libcxx". That setup has worked fine for me in some initial testing.

llvm/cmake/modules/LLVMExternalProjectUtils.cmake

65

Technically, you can have triples in other forms too, like x86_64-win32; win32 is an alias for windows, and if the -msvc suffix is omitted it defaults to the msvc variant.

If this only deals with canonicalized triples it should be fine. And for other cases, I guess it might be fine as well (as you'd get proper results if you pass triples in the expected form at least and we can't expect to mimic all the quirks handled by the llvm internal triple parser).

85

I'm not quite sure I follow the logic of this condition (parentheses might help?) - is it not (aix or mingw) or (not aix) or mingw? And why doesn't the current default logic work for the mingw case? At worst it might build an unnecessary llvm-lib frontend?

[zero9178]

In D97451#2587089, @mstorsjo wrote:

Windows is in the unique position of having two drivers, clang-cl and normal GNU clang, depending on whether a GNU or MSVC target is used.

FWIW, the normal GNU frontend also works for the MSVC target; some finer details are omitted in that case, but it rougly does work (and if the default target of the build is an msvc target, that's what targeted when calling that frontend too).

Also FWIW, I haven't really used the runtimes build mechanism at all, because I primarily bootstrap everything from scratch in a cross compile setup, i.e. with no preexisting build tools; first building the compiler, then building the base mingw crt, the compiler-rt builtins on top of that, and then other runtimes like libunwind, libcxxabi and libcxx on top of that. In such cases, afaik the runtimes build is a bit tricky, because I'd need to build other bits (e.g. the mingw crt) between building the compiler and the rest of the llvm provided runtimes. For now I just build all of those subprojects individually afterwards.

So therefore all of this is a bit of uncharted territory for me.

There's also a separate contender for how to build runtimes (still not entirely formally supported), as a middle ground - see https://github.com/llvm/llvm-project/blob/main/libcxx/utils/ci/runtimes/CMakeLists.txt. With that one, you configure building the runtimes just like a normal standalone project (i.e. you set the cmake configuration yourself, instead of instructing the machinery to figure it out based on a target triple), but you can add a series of projects to build as e.g. -DLLVM_ENABLE_PROJECTS="libunwind;libcxxabi;libcxx". That setup has worked fine for me in some initial testing.

I use a MinGW sysroot that is already prebuilt (with GCC in my case) therefore only needing all the other runtimes to be built by clang. Additionally I also build sanitizers and others with clang-cl for i386,x86 and aarch64. I do all of this natively from Windows and being able to simply add LLVM_ENABLE_RUNTIMES to the normal cmake config makes it a lot easier and I only use a single cmake cache file containing all variables, nothing else.

I took heavy inspiration from some of Fuchsia build caches as seen here https://github.com/llvm/llvm-project/blob/d748908fa02b11c7840a7f03c7a52223126bdba9/clang/cmake/caches/Fuchsia.cmake.
I then created my own script (https://gist.github.com/zero9178/cb992f102597f591dc7ed91449f77743) allowing me to just do cmake .. -GNinja -C build.cmake and it'll build a vast array of configurations.

The fact that it worked in LLVM 11 and before was more luck than anything. Since LLVM 12 this has been broken for mingw triples however, so I am trying to fix it.

llvm/cmake/modules/LLVMExternalProjectUtils.cmake

65

I was checking around if there were any functions in other cmake modules or so that could parse or canonicalize triples. I was thinking that for now only supporting canonicalized triples should be fine. Other patterns could technically be added in the future. As far as I am aware, clang would also accept eg. GCC style triple for mingw as well I believe.

85

That's true, that was a mistake of mine. Not a lot that can go wrong here but I'll only conditionally add llvm-lib anyway for the sake of it. Will fix shortly

[zero9178]

Addressed reviewers comments:
Corrected appending the correct tools to ARG_TOOLCHAIN_TOOLS in the case when use_mingw_toolchain_tools is set.

[rnk]

llvm/cmake/modules/LLVMExternalProjectUtils.cmake

65

This string matching of the cmake argument list makes me feel like we should be passing the target as a proper function argument, and then have llvm_ExternalProject_Add be responsible for appending all three DCMAKE_(C|CXX|ASM)_COMPILER_TARGET variants to the argument list. Do you mind making that happen? It should be a net savings in lines of code. I see only the test-suite caller of llvm_ExternalProject_add doesn't pass these arguments.

I share the concern that cmake doesn't know how to normalize mingw/gnu/msvc triples, but let's factor the decision out into some kind of "is_msvc_triple" utility, and then later we can teach it to match any of the patterns really matter.

[zero9178]

Added new option argument to llvm_ExternalProject_Add called TARGET_TRIPLE. When set, passes the various CMAKE_<LANG>_COMPILER_TARGET as cmake argument.
Additionally a helper function called is_msvc_triple is used to determine whether the target triple (either set by TARGET_TRIPLE or LLVM_DEFAULT_TARGET_TRIPLE) refers to an msvc target. If that is the case and the just-built toolchain is to be used, it will select clang-cl as well as other MSVC specific tools over the GNU style ones.

Also updated usages of llvm_ExternProject_Add accordingly

llvm/cmake/modules/LLVMExternalProjectUtils.cmake

if (CMAKE_GENERATOR MATCHES "Make")
     # Use special command for Makefiles to support parallelism.
     set(${out_var} "$(MAKE)" "-C" "${bin_dir}" "${target}" PARENT_SCOPE)
   else()
     set(${out_var} ${CMAKE_COMMAND} --build ${bin_dir} --target ${target}
                                     --config ${ARG_CONFIGURATION} PARENT_SCOPE)
   endif()
 endfunction()
 
+# is_msvc_triple(out_var triple)
+#   Checks whether the passed triple refers to an MSVC environment
+function(is_msvc_triple out_var triple)
+  if (triple MATCHES ".*-windows-msvc.*")
+    set(${out_var} TRUE PARENT_SCOPE)
+  else()
+    set(${out_var} FALSE PARENT_SCOPE)
+  endif()
+endfunction()
+
+
 # llvm_ExternalProject_Add(name source_dir ...
 #   USE_TOOLCHAIN
 #     Use just-built tools (see TOOLCHAIN_TOOLS)
 #   EXCLUDE_FROM_ALL
 #     Exclude this project from the all target
 #   NO_INSTALL
 #     Don't generate install targets for this project
 #   ALWAYS_CLEAN
 #     Always clean the sub-project before building
 #   CMAKE_ARGS arguments...
 #     Optional cmake arguments to pass when configuring the project
 #   TOOLCHAIN_TOOLS targets...
 #     Targets for toolchain tools (defaults to clang;lld)
 #   DEPENDS targets...
 #     Targets that this project depends on
 #   EXTRA_TARGETS targets...
 #     Extra targets in the subproject to generate targets for
 #   PASSTHROUGH_PREFIXES prefix...
 #     Extra variable prefixes (name is always included) to pass down
 #   STRIP_TOOL path
 #     Use provided strip tool instead of the default one.
+#   TARGET_TRIPLE triple
+#     Optional target triple to pass to the compiler
 #   )
 function(llvm_ExternalProject_Add name source_dir)
   cmake_parse_arguments(ARG
     "USE_TOOLCHAIN;EXCLUDE_FROM_ALL;NO_INSTALL;ALWAYS_CLEAN"
     "SOURCE_DIR"
-    "CMAKE_ARGS;TOOLCHAIN_TOOLS;RUNTIME_LIBRARIES;DEPENDS;EXTRA_TARGETS;PASSTHROUGH_PREFIXES;STRIP_TOOL"
+    "CMAKE_ARGS;TOOLCHAIN_TOOLS;RUNTIME_LIBRARIES;DEPENDS;EXTRA_TARGETS;PASSTHROUGH_PREFIXES;STRIP_TOOL;TARGET_TRIPLE"
     ${ARGN})
   canonicalize_tool_name(${name} nameCanon)
 
   foreach(arg ${ARG_CMAKE_ARGS})
     if(arg MATCHES "^-DCMAKE_SYSTEM_NAME=")
       string(REGEX REPLACE "^-DCMAKE_SYSTEM_NAME=(.*)$" "\\1" _cmake_system_name "${arg}")
     endif()
   endforeach()
 
   # If CMAKE_SYSTEM_NAME is not set explicitly in the arguments passed to us,
   # reflect CMake's own default.
   if (NOT _cmake_system_name)
     set(_cmake_system_name "${CMAKE_HOST_SYSTEM_NAME}")
   endif()
 
+  if(NOT ARG_TARGET_TRIPLE)
+    set(target_triple ${LLVM_DEFAULT_TARGET_TRIPLE})
+  else()
+    set(target_triple ${ARG_TARGET_TRIPLE})
+  endif()
+
+  is_msvc_triple(is_msvc_target ${target_triple})
+
   if(NOT ARG_TOOLCHAIN_TOOLS)
     set(ARG_TOOLCHAIN_TOOLS clang)
     # AIX 64-bit XCOFF and big AR format is not yet supported in some of these tools.
     if(NOT _cmake_system_name STREQUAL AIX)
       list(APPEND ARG_TOOLCHAIN_TOOLS lld llvm-ar llvm-ranlib llvm-nm llvm-objdump)
       if(_cmake_system_name STREQUAL Darwin)
         list(APPEND ARG_TOOLCHAIN_TOOLS llvm-libtool-darwin llvm-lipo)
-      elseif(_cmake_system_name STREQUAL Windows)
+      elseif(is_msvc_target)
         list(APPEND ARG_TOOLCHAIN_TOOLS llvm-lib)
       else()
         # TODO: These tools don't fully support Mach-O format yet.
         list(APPEND ARG_TOOLCHAIN_TOOLS llvm-objcopy llvm-strip)
       endif()
     endif()
   endif()
   foreach(tool ${ARG_TOOLCHAIN_TOOLS})
foreach(variableName ${variableNames})
         list(APPEND PASSTHROUGH_VARIABLES
           -D${variableName}=${value})
       endif()
     endforeach()
   endforeach()
 
   if(ARG_USE_TOOLCHAIN AND NOT CMAKE_CROSSCOMPILING)
     if(CLANG_IN_TOOLCHAIN)
-      if(_cmake_system_name STREQUAL Windows)
+      if(is_msvc_target)
         set(compiler_args -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX}
                           -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX}
                           -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX})
       else()
         set(compiler_args -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang${CMAKE_EXECUTABLE_SUFFIX}
                           -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang++${CMAKE_EXECUTABLE_SUFFIX}
                           -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang${CMAKE_EXECUTABLE_SUFFIX})
       endif()
     endif()
     if(lld IN_LIST TOOLCHAIN_TOOLS)
-      if(_cmake_system_name STREQUAL Windows)
+      if(is_msvc_target)
         list(APPEND compiler_args -DCMAKE_LINKER=${LLVM_RUNTIME_OUTPUT_INTDIR}/lld-link${CMAKE_EXECUTABLE_SUFFIX})
       elseif(NOT _cmake_system_name STREQUAL Darwin)
         list(APPEND compiler_args -DCMAKE_LINKER=${LLVM_RUNTIME_OUTPUT_INTDIR}/ld.lld${CMAKE_EXECUTABLE_SUFFIX})
       endif()
     endif()
     if(llvm-ar IN_LIST TOOLCHAIN_TOOLS)
-      if(_cmake_system_name STREQUAL Windows)
+      if(is_msvc_target)
         list(APPEND compiler_args -DCMAKE_AR=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-lib${CMAKE_EXECUTABLE_SUFFIX})
       else()
         list(APPEND compiler_args -DCMAKE_AR=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-ar${CMAKE_EXECUTABLE_SUFFIX})
       endif()
     endif()
     if(llvm-libtool-darwin IN_LIST TOOLCHAIN_TOOLS)
       list(APPEND compiler_args -DCMAKE_LIBTOOL=${LLVM_RUNTIME_OUTPUT_INTDIR}/llvm-libtool-darwin${CMAKE_EXECUTABLE_SUFFIX})
     endif()
if(ARG_USE_TOOLCHAIN AND NOT CMAKE_CROSSCOMPILING)
     endif()
     list(APPEND ARG_DEPENDS ${TOOLCHAIN_TOOLS})
   endif()
 
   if(ARG_STRIP_TOOL)
     list(APPEND compiler_args -DCMAKE_STRIP=${ARG_STRIP_TOOL})
   endif()
 
+  if (ARG_TARGET_TRIPLE)
+    list(APPEND compiler_args -DCMAKE_C_COMPILER_TARGET=${ARG_TARGET_TRIPLE})
+    list(APPEND compiler_args -DCMAKE_CXX_COMPILER_TARGET=${ARG_TARGET_TRIPLE})
+    list(APPEND compiler_args -DCMAKE_ASM_COMPILER_TARGET=${ARG_TARGET_TRIPLE})
+  endif()
+
   add_custom_command(
     OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${name}-clobber-stamp
     DEPENDS ${ARG_DEPENDS}
     COMMAND ${CMAKE_COMMAND} -E touch ${BINARY_DIR}/CMakeCache.txt
     COMMAND ${CMAKE_COMMAND} -E touch ${STAMP_DIR}/${name}-mkdir
     COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/${name}-clobber-stamp
     COMMENT "Clobbering bootstrap build and stamp directories"
     )

llvm/runtimes/CMakeLists.txt

function(builtin_default_target compiler_rt_path)
 
   llvm_ExternalProject_Add(builtins
                            ${compiler_rt_path}/lib/builtins
                            DEPENDS ${ARG_DEPENDS}
                            CMAKE_ARGS -DLLVM_LIBRARY_OUTPUT_INTDIR=${LLVM_LIBRARY_DIR}
                                       -DLLVM_RUNTIME_OUTPUT_INTDIR=${LLVM_TOOLS_BINARY_DIR}
                                       -DLLVM_DEFAULT_TARGET_TRIPLE=${TARGET_TRIPLE}
                                       -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default}
-                                      -DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE}
-                                      -DCMAKE_ASM_COMPILER_TARGET=${TARGET_TRIPLE}
                                       -DCMAKE_C_COMPILER_WORKS=ON
                                       -DCMAKE_ASM_COMPILER_WORKS=ON
                                       ${BUILTINS_CMAKE_ARGS}
                            PASSTHROUGH_PREFIXES COMPILER_RT
                            USE_TOOLCHAIN
+                           TARGET_TRIPLE ${TARGET_TRIPLE}
                            ${EXTRA_ARGS})
 endfunction()
 
 function(builtin_register_target compiler_rt_path target)
   cmake_parse_arguments(ARG "" "" "DEPENDS" ${ARGN})
 
   check_apple_target(${target} builtin)
 
function(builtin_register_target compiler_rt_path target)
 
   llvm_ExternalProject_Add(builtins-${target}
                            ${compiler_rt_path}/lib/builtins
                            DEPENDS ${ARG_DEPENDS}
                            CMAKE_ARGS -DLLVM_LIBRARY_OUTPUT_INTDIR=${LLVM_LIBRARY_DIR}
                                       -DLLVM_RUNTIME_OUTPUT_INTDIR=${LLVM_TOOLS_BINARY_DIR}
                                       -DLLVM_DEFAULT_TARGET_TRIPLE=${target}
                                       -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON
-                                      -DCMAKE_C_COMPILER_TARGET=${target}
-                                      -DCMAKE_ASM_COMPILER_TARGET=${target}
                                       -DCMAKE_C_COMPILER_WORKS=ON
                                       -DCMAKE_ASM_COMPILER_WORKS=ON
                                       -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON
                                       ${${target}_extra_args}
                            USE_TOOLCHAIN
+                           TARGET_TRIPLE ${target}
                            ${EXTRA_ARGS})
 endfunction()
 
 # If compiler-rt is present we need to build the builtin libraries first. This
 # is required because the other runtimes need the builtin libraries present
 # before the just-built compiler can pass the configuration tests.
 get_compiler_rt_path(compiler_rt_path)
 if(compiler_rt_path)
llvm_ExternalProject_Add(runtimes
                            ${CMAKE_CURRENT_SOURCE_DIR}/../../runtimes
                            DEPENDS ${ARG_DEPENDS}
                            # Builtins were built separately above
                            CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off
                                       -DLLVM_INCLUDE_TESTS=${LLVM_INCLUDE_TESTS}
                                       -DLLVM_DEFAULT_TARGET_TRIPLE=${TARGET_TRIPLE}
                                       -DLLVM_ENABLE_PROJECTS_USED=${LLVM_ENABLE_PROJECTS_USED}
                                       -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default}
-                                      -DCMAKE_C_COMPILER_TARGET=${TARGET_TRIPLE}
-                                      -DCMAKE_CXX_COMPILER_TARGET=${TARGET_TRIPLE}
-                                      -DCMAKE_ASM_COMPILER_TARGET=${TARGET_TRIPLE}
                                       -DCMAKE_C_COMPILER_WORKS=ON
                                       -DCMAKE_CXX_COMPILER_WORKS=ON
                                       -DCMAKE_ASM_COMPILER_WORKS=ON
                                       ${RUNTIMES_CMAKE_ARGS}
                            PASSTHROUGH_PREFIXES LLVM_ENABLE_RUNTIMES
                                                 ${ARG_PREFIXES}
                            EXTRA_TARGETS ${extra_targets}
                                          ${test_targets}
                                          ${SUB_COMPONENTS}
                                          ${SUB_CHECK_TARGETS}
                                          ${SUB_INSTALL_TARGETS}
                            USE_TOOLCHAIN
+                           TARGET_TRIPLE ${TARGET_TRIPLE}
                            ${EXTRA_ARGS})
 endfunction()
 
 # runtime_register_target(target)
 #   Utility function to register external runtime target.
 function(runtime_register_target name target)
   cmake_parse_arguments(ARG "" "" "DEPENDS;CMAKE_ARGS" ${ARGN})
   include(${LLVM_BINARY_DIR}/runtimes/${name}/Components.cmake OPTIONAL)
llvm_ExternalProject_Add(runtimes-${name}
                            ${CMAKE_CURRENT_SOURCE_DIR}/../../runtimes
                            DEPENDS ${${name}_deps}
                            # Builtins were built separately above
                            CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off
                                       -DLLVM_INCLUDE_TESTS=${LLVM_INCLUDE_TESTS}
                                       -DLLVM_DEFAULT_TARGET_TRIPLE=${target}
                                       -DLLVM_ENABLE_PROJECTS_USED=${LLVM_ENABLE_PROJECTS_USED}
                                       -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON
-                                      -DCMAKE_C_COMPILER_TARGET=${target}
-                                      -DCMAKE_CXX_COMPILER_TARGET=${target}
-                                      -DCMAKE_ASM_COMPILER_TARGET=${target}
                                       -DCMAKE_C_COMPILER_WORKS=ON
                                       -DCMAKE_CXX_COMPILER_WORKS=ON
                                       -DCMAKE_ASM_COMPILER_WORKS=ON
                                       -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON
                                       -DLLVM_RUNTIMES_TARGET=${name}
                                       ${${name}_extra_args}
                            EXTRA_TARGETS ${${name}_extra_targets}
                                          ${${name}_test_targets}
                            USE_TOOLCHAIN
+                           TARGET_TRIPLE ${target}
                            ${EXTRA_ARGS})
 endfunction()
 
 if(runtimes)
   # Create a runtimes target that uses this file as its top-level CMake file.
   # The runtimes target is a configuration of all the runtime libraries
   # together in a single CMake invocaiton.
   if(NOT LLVM_RUNTIME_TARGETS)