[CMake] First pass at adding support for clang bootstrap builds to CMake

[beanz]

Summary

This patch adds a new CLANG_ENABLE_BOOTSTRAP option to CMake which adds targets for building a stage2 bootstrap compiler. The targets are:

bootstrap-configure
bootstrap-build
bootstrap-install
bootstrap (which does all 3).

If you are using 3.3.20150708 or greater it utilizes the ninja USES_TERMINAL_* settings on the external project so that the output is properly buffered.

[chandlerc]

This is pretty awesome.

A question -- how can we document better what this does? In particular, the fact that we have both 'bootstrap-install' and 'install' seems... confusing. Is it possible to make 'install' == 'bootstrap-install' in the case where the CMake option is set?

[chandlerc]

Other thoughts:

Can we get 'bootstrap-check'? 'bootstrap-ccheck-foo' for all the 'check-foo' variants?

I think 'bootstrap' shouldn't do 'bootstrap-install', it should be equivalent to 'bootstrap-check-all' IMO.

[filcab]

Thank you!
I was going to ask you to change the bootstrap-install name, but it seems it's a cmake thing.

CMakeLists.txt

565 ↗

(On Diff #31288)

Nit: not consistent with the other parens closing multi-line macros/directives.

[beanz]

Chandler,

I would eventually like to make install be bootstrap-install in cases where CLANG_ENABLE_BOOTSTRAP=On. Doing that will require a larger reworking of how clang's install targets are structured. It is on my todo list, but in the interest of sending out smaller patches I wanted to send this out early and expand it in the public tree.

We can document the existing behavior in the Building LLVM With CMake documentation under frequently used variables (http://llvm.org/docs/CMake.html#frequently-used-cmake-variables).

I'm not super familiar with the ExternalProject support, but since you can't import targets from the external project I don't think we'll want to do a bootstrap-check-foo for all foo. Unless someone from the CMake community has some better ideas how to do that. Part of my motivation for prototyping this up was to start getting to know ExternalProject so that I can take advantage of it for compiler-rt.

I don't want bootstrap to do bootstrap-install either, but I can't find a way to make bootstrap-install work without it being part of the bootstrap target.

If it is alright with you, I'd really like to land this mostly as is, and iterate on it in-tree. There should be no real risk to this because it is fully gated by an option that defaults off.

Thanks,
-Chris

CMakeLists.txt

565 ↗

(On Diff #31288)

I will fix this and update the patches.

[beanz]

Fixing the formatting as per filcab's request.

[beanz]

• Fixed a dependency tracking issue with figuring out when to clear out the bootstrap build
• Removed 'bootstrap-install' from the 'bootstrap' target
• Added 'bootstrap-check' and 'bootstrap-check-all' targets

[chandlerc]

Totally awesome man, ship it. I've left a nit-pick below, but this is definitely the right starting place.

CMakeLists.txt

615 ↗

(On Diff #31359)

nit pick: I'd call it 'check-llvm', 'check-clang', and 'check-all'. I'm OK if we have to hard code the list though, that seems a minor inconvenience.

cfe/trunk/CMakeLists.txt

else()
     set(LLVM_SHLIB_OUTPUT_INTDIR ${LLVM_LIBRARY_OUTPUT_INTDIR})
   endif()
 
   option(LLVM_INSTALL_TOOLCHAIN_ONLY
     "Only include toolchain files in the 'install' target." OFF)
 
   option(LLVM_FORCE_USE_OLD_HOST_TOOLCHAIN
     "Set to ON to force using an old, unsupported host toolchain." OFF)
+  option(CLANG_ENABLE_BOOTSTRAP "Generate the clang bootstrap target" OFF)
 
   include(AddLLVM)
   include(TableGen)
   include(HandleLLVMOptions)
 
   set(PACKAGE_VERSION "${LLVM_PACKAGE_VERSION}")
 
   if (NOT DEFINED LLVM_INCLUDE_TESTS)
if (CLANG_BUILT_STANDALONE)
 
   # Also copy ClangConfig.cmake to the build directory so that dependent projects
   # can build against a build directory of Clang more easily.
   configure_file(
     ${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules/ClangConfig.cmake
     ${CLANG_BINARY_DIR}/share/clang/cmake/ClangConfig.cmake
     COPYONLY)
 endif ()
+
+if (CLANG_ENABLE_BOOTSTRAP)
+  include(ExternalProject)
+
+  if(CMAKE_VERSION VERSION_LESS 3.3.20150708)
+    set(cmake_3_4_USES_TERMINAL_OPTIONS)
+  else()
+    set(cmake_3_4_USES_TERMINAL_OPTIONS
+      USES_TERMINAL_CONFIGURE 1
+      USES_TERMINAL_BUILD 1
+      USES_TERMINAL_INSTALL 1
+      )
+  endif()
+  
+  set(STAMP_DIR ${CMAKE_CURRENT_BINARY_DIR}/bootstrap-stamps/)
+  set(BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR}/bootstrap-bins/)
+
+  add_custom_target(bootstrap-clear
+    DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/bootstrap-cleared
+    )
+  add_custom_command(
+    OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/bootstrap-cleared
+    DEPENDS clang
+    COMMAND ${CMAKE_COMMAND} -E remove_directory ${BINARY_DIR}
+    COMMAND ${CMAKE_COMMAND} -E make_directory ${BINARY_DIR}
+    COMMAND ${CMAKE_COMMAND} -E remove_directory ${STAMP_DIR}
+    COMMAND ${CMAKE_COMMAND} -E make_directory ${STAMP_DIR}
+    COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/bootstrap-cleared
+    COMMENT "Clobberring bootstrap build and stamp directories"
+    )
+
+  ExternalProject_Add(bootstrap
+    DEPENDS clang
+    PREFIX bootstrap
+    SOURCE_DIR ${CMAKE_SOURCE_DIR}
+    STAMP_DIR ${STAMP_DIR}
+    BINARY_DIR ${BINARY_DIR}
+    CMAKE_ARGS
+                # We shouldn't need to set this here, but INSTALL_DIR doesn't
+                # seem to work, so instead I'm passing this through
+                -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
+                ${CLANG_BOOTSTRAP_CMAKE_ARGS}
+                -DCMAKE_CXX_COMPILER=${CMAKE_BINARY_DIR}/bin/clang++
+                -DCMAKE_C_COMPILER=${CMAKE_BINARY_DIR}/bin/clang
+    INSTALL_COMMAND ""
+    STEP_TARGETS configure build
+    ${cmake_3_4_USES_TERMINAL_OPTIONS}
+    )
+
+  # exclude really-install from main target
+  set_target_properties(bootstrap PROPERTIES _EP_really-install_EXCLUDE_FROM_MAIN On)
+  ExternalProject_Add_Step(bootstrap really-install
+    COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target install
+    COMMENT "Performing install step for 'bootstrap'"
+    DEPENDEES build
+  )
+  ExternalProject_Add_StepTargets(bootstrap really-install)
+  add_custom_target(bootstrap-install DEPENDS bootstrap-really-install)
+
+
+  set(ADDITIONAL_TARGETS_TO_ADD check-llvm check-clang check-all)
+  foreach(target ${ADDITIONAL_TARGETS_TO_ADD})
+    # exclude from main target
+    set_target_properties(bootstrap PROPERTIES _EP_${target}_EXCLUDE_FROM_MAIN On)
+
+    ExternalProject_Add_Step(bootstrap ${target}
+      COMMAND ${CMAKE_COMMAND} --build <BINARY_DIR> --target ${target}
+      COMMENT "Performing ${target} for 'bootstrap'"
+      DEPENDEES configure
+    )
+    ExternalProject_Add_StepTargets(bootstrap ${target})
+  endforeach()
+endif()