Harmonize cmake_policy() across standalone builds of all projects

[mgorny]

Summary

Move cmake_policy() settings from llvm/CMakeLists.txt into a shared
cmake/modules/CMakePolicy.cmake. Include it from all relevant
projects that support standalone builds, in order to ensure that
the policies are consistently set whether they are built in-tree
or stand-alone.

[mgorny]

Replace with one patch for all the projects.

[nickdesaulniers]

clang/CMakeLists.txt

12

Seems error prone. Does cmake have an include system for including fragments from elsewhere? If so, that seems like a more concise and less error prone approach.

https://cmake.org/cmake/help/latest/command/include.html

[mgorny]

clang/CMakeLists.txt

12

I suppose that makes sense. I think that at this point all these components require the top-level cmake directory from monorepo anyway.

[mgorny]

Use a shared policy file via include() instead.

[mgorny]

Thanks. Since this is not urgent, I'll let it wait for others to chime in a while more.

[mstorsjo]

This broke builds where clang/lld/lldb are symlinked into llvm/tools instead of specified via LLVM_ENABLE_PROJECTS - since ${CMAKE_CURRENT_SOURCE_DIR}/../cmake doesn't find anything in that context.

(The reason for wanting to stick to such a legacy setup, is that ccache caches can't be shared between workdirs for source files that are found outside of the base llvm-project/llvm directory - and I'm heavily reliant on ccache for keeping build times manageable in my dev workflow.)

[mstorsjo]

In D136572#3888177, @mstorsjo wrote:

This broke builds where clang/lld/lldb are symlinked into llvm/tools instead of specified via LLVM_ENABLE_PROJECTS - since ${CMAKE_CURRENT_SOURCE_DIR}/../cmake doesn't find anything in that context.

This aspect, for finding the shared cmake directory, is handled elsewhere by the main LLVM build setting the LLVM_COMMON_CMAKE_UTILS variable, and if not found, it's assumed to be in ${CMAKE_CURRENT_SOURCE_DIR}/... See e.g. https://github.com/llvm/llvm-project/blob/llvmorg-15.0.0/lld/CMakeLists.txt#L198-L206.

Since this is invoked very early in the per-project cmakefiles, I guess we can't assume to have all those variables set up yet, but e.g. something like this fixes the issue for me:

diff --git a/lld/CMakeLists.txt b/lld/CMakeLists.txt
index 964bc3fd8b17..530f52b05bac 100644
--- a/lld/CMakeLists.txt
+++ b/lld/CMakeLists.txt
@@ -1,6 +1,11 @@
 cmake_minimum_required(VERSION 3.13.4)
-include(${CMAKE_CURRENT_SOURCE_DIR}/../cmake/Modules/CMakePolicy.cmake
-  NO_POLICY_SCOPE)
+if(DEFINED LLVM_COMMON_CMAKE_UTILS)
+  include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+    NO_POLICY_SCOPE)
+else()
+  include(${CMAKE_CURRENT_SOURCE_DIR}/../cmake/Modules/CMakePolicy.cmake
+    NO_POLICY_SCOPE)
+endif()
 
 # If we are not building as a part of LLVM, build LLD as an
 # standalone project, using LLVM as an external library:

(And the same applied for all the other subprojects.) It's admittedly not very pretty though...

[mgorny]

I've reverted it right now, and I'll update the patch later today.

[mgorny]

Move LLVM_COMMON_CMAKE_UTILS earlier and use it for CMakePolicy path.

[mgorny]

@mstorsjo, could you check this version?

[MaskRay]

It'll be good if we have a wiki describing how downstream users invoke cmake, so that large cmake refactoring can be verified beforehand.
(like usage verification https://github.com/opencollab/llvm-toolchain-integration-test-suite, but for cmake invocations)

[mstorsjo]

In D136572#3888630, @mgorny wrote:

@mstorsjo, could you check this version?

LGTM, this seems to work for my usecase (with lld,lldb,clang enabled this way). Thanks for the quick revert and fix!

[mstorsjo]

Also for the record, I'd love to get rid of this symlink based setup, if I could make cmake produce project files where caching works in the same way. The main requirement for that would be to have the whole cmake build started from the toplevel llvm-project directory (so that all source from all subprojects are subdirectories to this), instead of the llvm-project/llvm directory.

[mgorny]

In D136572#3889317, @mstorsjo wrote:

Also for the record, I'd love to get rid of this symlink based setup, if I could make cmake produce project files where caching works in the same way. The main requirement for that would be to have the whole cmake build started from the toplevel llvm-project directory (so that all source from all subprojects are subdirectories to this), instead of the llvm-project/llvm directory.

To be honest, I also find it weird to start monorepo builds from within llvm subdirectory. Perhaps adding a top-level CMakeLists.txt that would work and wouldn't break people's workflows wouldn't be that hard, though.

[mstorsjo]

In D136572#3890835, @mgorny wrote:

In D136572#3889317, @mstorsjo wrote:

Also for the record, I'd love to get rid of this symlink based setup, if I could make cmake produce project files where caching works in the same way. The main requirement for that would be to have the whole cmake build started from the toplevel llvm-project directory (so that all source from all subprojects are subdirectories to this), instead of the llvm-project/llvm directory.

To be honest, I also find it weird to start monorepo builds from within llvm subdirectory. Perhaps adding a top-level CMakeLists.txt that would work and wouldn't break people's workflows wouldn't be that hard, though.

This has been discussed briefly before - the main issue is that LLVM does some cmake trickery to recreate the installed directory structure (with executables in $(pwd)/bin etc) directly in the build folder, and if llvm-project/llvm is included from a toplevel llvm-project/CMakeLists.txt, this all ends up in $(pwd)/llvm/bin as things stand right now. See e.g. https://discourse.llvm.org/t/rfc-llvm-build-system-future-direction/53430/18. Hopefully it wouldn't be too hard to resolve - I had a brief look at it (just creating a stub CMakeLists.txt at the root and including the llvm subdirectory) a couple years ago, but didn't spend further time on looking into how things are set up and what it would take to fix it.

clang/CMakeLists.txt

 cmake_minimum_required(VERSION 3.13.4)
 
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
+
 # If we are not building as a part of LLVM, build Clang as an
 # standalone project, using LLVM as an external library:
 if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
   project(Clang)
   set(CLANG_BUILT_STANDALONE TRUE)
 endif()
 
 # Must go below project(..)
 include(GNUInstallDirs)
 
 if(CLANG_BUILT_STANDALONE)
   set(CMAKE_CXX_STANDARD 17 CACHE STRING "C++ standard to conform to")
if(LLVM_INCLUDE_TESTS)
     else()
       set(LLVM_INCLUDE_TESTS OFF)
     endif()
 
     umbrella_lit_testsuite_begin(check-all)
   endif() # LLVM_INCLUDE_TESTS
 endif() # standalone
 
-if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
-  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
-endif()
-
 # Make sure that our source directory is on the current cmake module path so that
 # we can include cmake files from this directory.
 list(INSERT CMAKE_MODULE_PATH 0
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
   "${LLVM_COMMON_CMAKE_UTILS}/Modules"
   )
 
 # This allows disabling clang's XML dependency even if LLVM finds libxml2.

cmake/Modules/CMakePolicy.cmake

+# CMake policy settings shared between LLVM projects
+
+# CMP0114: ExternalProject step targets fully adopt their steps.
+# New in CMake 3.19: https://cmake.org/cmake/help/latest/policy/CMP0114.html
+if(POLICY CMP0114)
+  cmake_policy(SET CMP0114 OLD)
+endif()
+# CMP0116: Ninja generators transform `DEPFILE`s from `add_custom_command()`
+# New in CMake 3.20. https://cmake.org/cmake/help/latest/policy/CMP0116.html
+if(POLICY CMP0116)
+  cmake_policy(SET CMP0116 OLD)
+endif()

flang/CMakeLists.txt

 cmake_minimum_required(VERSION 3.13.4)
 
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
+
 set(CMAKE_BUILD_WITH_INSTALL_NAME_DIR ON)
 
 # Flang requires C++17.
 set(CMAKE_CXX_STANDARD 17)
 set(CMAKE_CXX_STANDARD_REQUIRED TRUE)
 set(CMAKE_CXX_EXTENSIONS OFF)
 
 set(FLANG_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR})

 endif()
 # Always build tco tool
 set(LLVM_BUILD_TOOLS ON)
 
 include_directories(BEFORE
   ${FLANG_BINARY_DIR}/include
   ${FLANG_SOURCE_DIR}/include)
 
-if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
-  set(LLVM_COMMON_CMAKE_UTILS ${FLANG_SOURCE_DIR}/../cmake)
-endif()
-
 # Add Flang-centric modules to cmake path.
 list(INSERT CMAKE_MODULE_PATH 0
   "${FLANG_SOURCE_DIR}/cmake/modules"
   "${LLVM_COMMON_CMAKE_UTILS}/Modules"
   )
 include(AddFlang)
 
 if (NOT DEFAULT_SYSROOT)

lld/CMakeLists.txt

 cmake_minimum_required(VERSION 3.13.4)
 
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
+
 # If we are not building as a part of LLVM, build LLD as an
 # standalone project, using LLVM as an external library:
 if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
   project(lld)
   set(LLD_BUILT_STANDALONE TRUE)
 endif()
 
 # Must go below project(..)

 if (CMAKE_SOURCE_DIR STREQUAL CMAKE_BINARY_DIR)
   message(FATAL_ERROR "In-source builds are not allowed. CMake would overwrite "
 "the makefiles distributed with LLVM. Please create a directory and run cmake "
 "from there, passing the path to this source directory as the last argument. "
 "This process created the file `CMakeCache.txt' and the directory "
 "`CMakeFiles'. Please delete them.")
 endif()
 
-if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
-  set(LLVM_COMMON_CMAKE_UTILS ${LLD_SOURCE_DIR}/../cmake)
-endif()
-
 # Add path for custom modules.
 list(INSERT CMAKE_MODULE_PATH 0
   "${LLD_SOURCE_DIR}/cmake/modules"
   "${LLVM_COMMON_CMAKE_UTILS}/Modules"
   )
 
 include(AddLLD)
 

lldb/CMakeLists.txt

 cmake_minimum_required(VERSION 3.13.4)
 
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
+
 # Add path for custom modules.
 set(CMAKE_MODULE_PATH
   ${CMAKE_MODULE_PATH}
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake"
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
   )
 
 # If we are not building as part of LLVM, build LLDB as a standalone project,

lldb/cmake/modules/LLDBStandalone.cmake

-# CMP0116: Ninja generators transform `DEPFILE`s from `add_custom_command()`
-# New in CMake 3.20. https://cmake.org/cmake/help/latest/policy/CMP0116.html
-if(POLICY CMP0116)
-  cmake_policy(SET CMP0116 OLD)
-endif()
-
 if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
   set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
 endif()
 
 list(APPEND CMAKE_MODULE_PATH "${LLVM_COMMON_CMAKE_UTILS}/Modules")
 
 option(LLVM_INSTALL_TOOLCHAIN_ONLY "Only include toolchain files in the 'install' target." OFF)
 

llvm/CMakeLists.txt

 # See docs/CMake.html for instructions about how to build LLVM with CMake.
 
 cmake_minimum_required(VERSION 3.13.4)
 
-# CMP0114: ExternalProject step targets fully adopt their steps.
-# New in CMake 3.19: https://cmake.org/cmake/help/latest/policy/CMP0114.html
-if(POLICY CMP0114)
-  cmake_policy(SET CMP0114 OLD)
-endif()
-# CMP0116: Ninja generators transform `DEPFILE`s from `add_custom_command()`
-# New in CMake 3.20. https://cmake.org/cmake/help/latest/policy/CMP0116.html
-if(POLICY CMP0116)
-  cmake_policy(SET CMP0116 OLD)
-endif()
+set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
 
 set(CMAKE_BUILD_WITH_INSTALL_NAME_DIR ON)
 
 if(NOT DEFINED LLVM_VERSION_MAJOR)
   set(LLVM_VERSION_MAJOR 16)
 endif()
 if(NOT DEFINED LLVM_VERSION_MINOR)
   set(LLVM_VERSION_MINOR 0)

 endif()
 
 option(LLVM_ENABLE_DAGISEL_COV "Debug: Prints tablegen patterns that were used for selecting" OFF)
 option(LLVM_ENABLE_GISEL_COV "Enable collection of GlobalISel rule coverage" OFF)
 if(LLVM_ENABLE_GISEL_COV)
   set(LLVM_GISEL_COV_PREFIX "${CMAKE_BINARY_DIR}/gisel-coverage-" CACHE STRING "Provide a filename prefix to collect the GlobalISel rule coverage")
 endif()
 
-set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
-
 # Add path for custom modules
 list(INSERT CMAKE_MODULE_PATH 0
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake"
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
   "${LLVM_COMMON_CMAKE_UTILS}/Modules"
   )
 
 # Generate a CompilationDatabase (compile_commands.json file) for our build,

mlir/CMakeLists.txt

 # MLIR project.
+cmake_minimum_required(VERSION 3.13.4)
+
+if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
+  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
+endif()
+include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
+  NO_POLICY_SCOPE)
 
 # Check if MLIR is built as a standalone project.
 if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
   project(mlir)
   set(MLIR_STANDALONE_BUILD TRUE)
 endif()
 
 # Must go below project(..)
 include(GNUInstallDirs)
 
 if(MLIR_STANDALONE_BUILD)
-  cmake_minimum_required(VERSION 3.13.4)
-
   find_package(LLVM CONFIG REQUIRED)
   set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${LLVM_CMAKE_DIR})
   include(HandleLLVMOptions)
   include(AddLLVM)
   include(TableGen)
 
   include_directories(${LLVM_INCLUDE_DIRS})
 

 set(MLIR_MAIN_SRC_DIR     ${CMAKE_CURRENT_SOURCE_DIR}  )
 set(MLIR_MAIN_INCLUDE_DIR ${MLIR_MAIN_SRC_DIR}/include )
 
 set(MLIR_SOURCE_DIR  ${CMAKE_CURRENT_SOURCE_DIR})
 set(MLIR_BINARY_DIR  ${CMAKE_CURRENT_BINARY_DIR})
 set(MLIR_INCLUDE_DIR ${CMAKE_CURRENT_BINARY_DIR}/include)
 set(MLIR_TOOLS_DIR   ${CMAKE_RUNTIME_OUTPUT_DIRECTORY})
 
-if(NOT DEFINED LLVM_COMMON_CMAKE_UTILS)
-  set(LLVM_COMMON_CMAKE_UTILS ${CMAKE_CURRENT_SOURCE_DIR}/../cmake)
-endif()
-
 # Make sure that our source directory is on the current cmake module path so
 # that we can include cmake files from this directory.
 list(INSERT CMAKE_MODULE_PATH 0
   "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
   "${LLVM_COMMON_CMAKE_UTILS}/Modules"
   )
 
 include(AddMLIR)