[CMake] Determine early on which projects are enabled

[zturner]

Summary

Some projects need to add conditional dependencies on other projects. compiler-rt is already doing this, and I attempted to add this to debuginfo-tests when I ran into the ordering problem, that you can't conditionally add a dependency unless that dependency's CMakeLists.txt has already been run (which would allow you to say if (TARGET foo).

The solution to this seems to be to determine very early on the entire set of projects which is enabled. This is complicated by the fact that there are multiple ways to enable projects, and different tree layouts (e.g. mono-repo, out of -tree, external, etc). This patch attempts to centralize all of this into one place, and then updates compiler-rt to demonstrate as a proof of concept how this can simplify code.

[beanz]

llvm/CMakeLists.txt

838 ↗

(On Diff #114409)

In general the goal is to move away from the <llvm>/projects directory in favor of <llvm>/runtimes or <llvm>/tools in the future. At the very least you'll need to support <llvm>/runtimes here too.

[zturner]

Updated to look in runtimes. Also +eugenis since I'm touching compiler-rt CMake, even though this is supposed to be NFC. Note that with this change, compiler-rt could in theory just delete its custom variables and use LLVM_PROJECT_LLD_ENABLED everywhere instead of COMPILER_RT_HAS_LLD etc. But that is a bit more invasive change which I don't want to make.

[eugenis]

compiler-rt changes look great!

llvm/cmake/modules/AddLLVM.cmake

929 ↗

(On Diff #114426)

The code in llvm/projects treats any subdirectory with CMakeLists.txt as a project. Could we do the same here, instead of listing all (or some?) known projects?

932 ↗

(On Diff #114426)

Might be a good idea to warn if a project is found in two places at once.

[zturner]

Updated to look for all projects in the runtimes and projects folder.

Also worth mentioning is that projects/CMakeLists.txt seems to have some hackish code that appears to be a workaround for the fact the feature I'm adding in this patch doesn't exist. Specifically, it first adds a bunch of known projects that don't depend on each other in a random order. Then it adds a couple of specific projects that do depend on each other in the right order so that they can use if (TARGET foo) to check for existence.

After this patch, the entire set of valid projects that are present in the tree can probably be added in any order, as long as each CMakeLists.txt is updated to use the proper LLVM_PROJECT_foo_ENABLED variable, rather than an if (TARGET foo) check.

[eugenis]

LGTM

[pdouglas]

Hi, since this change we're seeing a Cmake error (see below) when we run it with -DLLVM_TOOL_LLD_BUILD=OFF but with tools/lld checked out. If either that argument isn't passed or the lld directory isn't present, it runs fine.

-- Configuring done
CMake Error at cmake/modules/AddLLVM.cmake:1241 (add_dependencies):
  The dependency target "lld" of target "check-all" does not exist.
Call Stack (most recent call first):
  CMakeLists.txt:933 (add_lit_target)


CMake Error at cmake/modules/AddLLVM.cmake:1241 (add_dependencies):
  The dependency target "lld" of target "check-cfi" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1262 (add_lit_target)
  projects/compiler-rt/test/cfi/CMakeLists.txt:80 (add_lit_testsuite)


CMake Error at cmake/modules/AddLLVM.cmake:1241 (add_dependencies):
  The dependency target "lld" of target "check-cfi-and-supported" does not
  exist.
Call Stack (most recent call first):
  projects/compiler-rt/test/cfi/CMakeLists.txt:84 (add_lit_target)


CMake Error at cmake/modules/AddLLVM.cmake:1241 (add_dependencies):
  The dependency target "lld" of target "check-asan" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1262 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:161 (add_lit_testsuite)


CMake Error at cmake/modules/AddLLVM.cmake:1241 (add_dependencies):
  The dependency target "lld" of target "check-asan-dynamic" does not exist.
Call Stack (most recent call first):
  cmake/modules/AddLLVM.cmake:1262 (add_lit_target)
  projects/compiler-rt/test/asan/CMakeLists.txt:167 (add_lit_testsuite)

compiler-rt/trunk/CMakeLists.txt

 endif()
 
 append_list_if(COMPILER_RT_DEBUG -DSANITIZER_DEBUG=1 SANITIZER_COMMON_CFLAGS)
 
 # If we're using MSVC,
 # always respect the optimization flags set by CMAKE_BUILD_TYPE instead.
 if (NOT MSVC)
 
-  # Build with optimization, unless we're in debug mode. 
+  # Build with optimization, unless we're in debug mode.
   if(COMPILER_RT_DEBUG)
     list(APPEND SANITIZER_COMMON_CFLAGS -O0)
   else()
     list(APPEND SANITIZER_COMMON_CFLAGS -O3)
   endif()
 endif()
 
 # Determine if we should restrict stack frame sizes.
if (CMAKE_LINKER MATCHES "link.exe$")
   # it, but CMake doesn't seem to have a way to set linker flags for
   # individual static libraries, so we enable the suppression flag for
   # the whole compiler-rt project.
   append("/IGNORE:4221" CMAKE_STATIC_LINKER_FLAGS)
 endif()
 
 add_subdirectory(include)
 
-set(COMPILER_RT_LIBCXX_PATH ${LLVM_MAIN_SRC_DIR}/projects/libcxx)
-if(EXISTS ${COMPILER_RT_LIBCXX_PATH}/)
-  set(COMPILER_RT_HAS_LIBCXX_SOURCES TRUE)
-else()
-  set(COMPILER_RT_LIBCXX_PATH ${LLVM_MAIN_SRC_DIR}/../libcxx)
-  if(EXISTS ${COMPILER_RT_LIBCXX_PATH}/)
-    set(COMPILER_RT_HAS_LIBCXX_SOURCES TRUE)
-  else()
-    set(COMPILER_RT_HAS_LIBCXX_SOURCES FALSE)
-  endif()
+set(COMPILER_RT_HAS_LIBCXX_SOURCES ${LLVM_PROJECT_LIBCXX_ENABLED})
+if (LLVM_PROJECT_LIBCXX_ENABLED)
+  set(COMPILER_RT_LIBCXX_PATH ${LLVM_PROJECT_LIBCXX_SOURCE_DIR})
+  message("compiler-rt libcxx enabled at ${COMPILER_RT_LIBCXX_PATH}")
 endif()
 
-set(COMPILER_RT_LLD_PATH ${LLVM_MAIN_SRC_DIR}/tools/lld)
-if(EXISTS ${COMPILER_RT_LLD_PATH}/ AND LLVM_TOOL_LLD_BUILD)
-  set(COMPILER_RT_HAS_LLD TRUE)
-else()
-  set(COMPILER_RT_LLD_PATH ${LLVM_MAIN_SRC_DIR}/../lld)
-  if(EXISTS ${COMPILER_RT_LLD_PATH}/ AND LLVM_TOOL_LLD_BUILD)
-    set(COMPILER_RT_HAS_LLD TRUE)
-  else()
-    set(COMPILER_RT_HAS_LLD FALSE)
-  endif()
+set(COMPILER_RT_HAS_LLD ${LLVM_PROJECT_LLD_ENABLED})
+if (LLVM_PROJECT_LLD_ENABLED)
+  set(COMPILER_RT_LLD_PATH ${LLVM_PROJECT_LLD_SOURCE_DIR})
+  message("compiler-rt lld enabled at ${COMPILER_RT_LLD_PATH}")
 endif()
 pythonize_bool(COMPILER_RT_HAS_LLD)
 
 add_subdirectory(lib)
 
 if(COMPILER_RT_INCLUDE_TESTS)
   add_subdirectory(unittests)
   add_subdirectory(test)
 endif()

llvm/trunk/CMakeLists.txt

if(LLVM_PROFDATA_FILE AND EXISTS ${LLVM_PROFDATA_FILE})
   else()
     message(FATAL_ERROR "LLVM_PROFDATA_FILE can only be specified when compiling with clang")
   endif()
 endif()
 
 include(AddLLVM)
 include(TableGen)
 
+
+# Find all subprojects which are either enabled in a side-by-side layout, or
+# cloned into a non-side-by-side layout.  Do this before adding any
+# subdirectories so that any project can check for the existence of any other
+# project.  Each call takes priority over the next call, so any project which
+# is enabled via LLVM_ENABLE_PROJECTS will not have its location or enabled
+# status overwritten via a subsequent call.
+
+# First look for all projects explicitly enabled at the root.
+find_llvm_enabled_projects("${LLVM_SOURCE_DIR}/.." "${LLVM_ENABLE_PROJECTS}")
+
+# Then pick up any projects explicitly cloned into llvm/projects or llvm/runtimes
+find_llvm_enabled_projects("${LLVM_SOURCE_DIR}/runtimes")
+find_llvm_enabled_projects("${LLVM_SOURCE_DIR}/projects")
+
+# Then pick up a few specific projects which can be explicit cloned into llvm/tools
+find_llvm_enabled_projects("${LLVM_SOURCE_DIR}/tools" "clang;lldb;lld")
+
+
 if( MINGW )
   # People report that -O3 is unreliable on MinGW. The traditional
   # build also uses -O2 for that reason:
   llvm_replace_compiler_option(CMAKE_CXX_FLAGS_RELEASE "-O3" "-O2")
 endif()
 
 # Put this before tblgen. Else we have a circular dependence.
 add_subdirectory(lib/Demangle)

llvm/trunk/cmake/modules/AddLLVM.cmake

macro(add_llvm_executable name)
 
   add_windows_version_resource_file(ALL_FILES ${ALL_FILES})
 
   if(XCODE)
     # Note: the dummy.cpp source file provides no definitions. However,
     # it forces Xcode to properly link the static library.
     list(APPEND ALL_FILES "${LLVM_MAIN_SRC_DIR}/cmake/dummy.cpp")
   endif()
-  
   if( EXCLUDE_FROM_ALL )
     add_executable(${name} EXCLUDE_FROM_ALL ${ALL_FILES})
   else()
     add_executable(${name} ${ALL_FILES})
   endif()
 
   setup_dependency_debugging(${name} ${LLVM_COMMON_DEPENDS})
 

 
 function(canonicalize_tool_name name output)
   string(REPLACE "${CMAKE_CURRENT_SOURCE_DIR}/" "" nameStrip ${name})
   string(REPLACE "-" "_" nameUNDERSCORE ${nameStrip})
   string(TOUPPER ${nameUNDERSCORE} nameUPPER)
   set(${output} "${nameUPPER}" PARENT_SCOPE)
 endfunction(canonicalize_tool_name)
 
+macro(find_llvm_enabled_projects base_dir)
+  set(specific_project_list "${ARGN}")
+  if("${specific_project_list}" STREQUAL "")
+    file(GLOB entries "${base_dir}/*")
+    set(list_of_project_dirs "")
+    foreach(entry ${entries})
+      if(IS_DIRECTORY ${entry} AND EXISTS ${entry}/CMakeLists.txt)
+        get_filename_component(filename "${entry}" NAME)
+        list(APPEND specific_project_list "${filename}")
+      endif()
+    endforeach(entry)
+  endif()
+
+  foreach(proj ${specific_project_list})
+    canonicalize_tool_name(${proj} projUPPER)
+
+    if (${LLVM_PROJECT_${projUPPER}_ENABLED})
+      if (EXISTS "${base_dir}/${proj}")
+        message(WARNING "Project ${projUPPER} in ${base_dir}/${proj} previously found in ${LLVM_PROJECT_${projUPPER}_SOURCE_DIR}")
+      endif()
+      continue()
+    elseif(EXISTS "${LLVM_EXTERNAL_${projUPPER}_SOURCE_DIR}")
+      set(LLVM_PROJECT_${projUPPER}_ENABLED ON)
+      set(LLVM_PROJECT_${projUPPER}_SOURCE_DIR "${LLVM_EXTERNAL_${projUPPER}_SOURCE_DIR}")
+    elseif(EXISTS "${base_dir}/${proj}")
+      set(LLVM_PROJECT_${projUPPER}_ENABLED ON)
+      set(LLVM_PROJECT_${projUPPER}_SOURCE_DIR "${base_dir}/${proj}")
+    endif()
+  endforeach()
+endmacro()
+
 # Custom add_subdirectory wrapper
 # Takes in a project name (i.e. LLVM), the subdirectory name, and an optional
 # path if it differs from the name.
 macro(add_llvm_subdirectory project type name)
   set(add_llvm_external_dir "${ARGN}")
   if("${add_llvm_external_dir}" STREQUAL "")
     set(add_llvm_external_dir ${name})
   endif()
function(add_llvm_tool_symlink link_name target)
   set(dest_binary "$<TARGET_FILE:${target}>")
 
   # This got a bit gross... For multi-configuration generators the target
   # properties return the resolved value of the string, not the build system
   # expression. To reconstruct the platform-agnostic path we have to do some
   # magic. First we grab one of the types, and a type-specific path. Then from
   # the type-specific path we find the last occurrence of the type in the path,
   # and replace it with CMAKE_CFG_INTDIR. This allows the build step to be type
-  # agnostic again. 
+  # agnostic again.
   if(NOT ARG_OUTPUT_DIR)
     # If you're not overriding the OUTPUT_DIR, we can make the link relative in
     # the same directory.
     if(UNIX)
       set(dest_binary "$<TARGET_FILE_NAME:${target}>")
     endif()
     if(CMAKE_CONFIGURATION_TYPES)
       list(GET CMAKE_CONFIGURATION_TYPES 0 first_type)