Moved flang to LLVM_ALL_PROJECTS

[pscoro]

Summary

If my understanding is correct, when flang was added to llvm-project, it was separated from the other llvm projects (which are all handled in a list called LLVM_ALL_PROJECTS) because it had a higher C++ requirement (C++17 instead of C++11). Supposedly, this discrepancy has since been resolved and flang should be safe to be add to this list. From my testing this seems to have worked without any issues, but I am opening this up to the community to discuss if there are any issues with this move or if this change is wanted.

The motivation for doing this move now is the recent discussions about better supporting Flang's runtime libraries with existing LLVM runtime infrastructure. As a prerequisite, before we can consider stuff like adding flang's runtime to LLVM_ENABLE_RUNTIMES=all, it would make sense to first treat flang like one of "all the llvm projects" (if this can be done without issue)

[mehdi_amini]

llvm/CMakeLists.txt

121

Might be useful to keep the LLVM_EXTRA_PROJECTS here and below, even if empty for now? Adding a new project to LLVM could then just reuse this.

[nikic]

I don't think this is safe to do until the extremely high memory usage requirements for flang compilation are fixed. The last two times I tried enabling the flang project, I had to hard reset my computer afterwards.

As far as I can tell, for safety reasons you currently need to build flang separately, with a different build configuration like -j1.

[brad]

In D154109#4656845, @nikic wrote:

I don't think this is safe to do until the extremely high memory usage requirements for flang compilation are fixed. The last two times I tried enabling the flang project, I had to hard reset my computer afterwards.

As far as I can tell, for safety reasons you currently need to build flang separately, with a different build configuration like -j1.

I do various builds including Flang and on my VMs I have 24GB plus a bit of swap and 8 CPUs. I can blast through more or less all of the other sub-projects enabled with -j8 or -j9 no problem.

When I hit Flang and it starts running out of memory quick I have to step down to -j3 instead of -j4 to really be on the safe side. Watching the build a lot of the code is taking up 4.5 to 5.5GB of memory per file.

llvm/CMakeLists.txt

elseif(CMAKE_BUILD_TYPE)
     set(CMAKE_TRY_COMPILE_CONFIGURATION ${CMAKE_BUILD_TYPE})
   endif()
 endif()
 
 # Side-by-side subprojects layout: automatically set the
 # LLVM_EXTERNAL_${project}_SOURCE_DIR using LLVM_ALL_PROJECTS
 # This allows an easy way of setting up a build directory for llvm and another
 # one for llvm+clang+... using the same sources.
-set(LLVM_ALL_PROJECTS "bolt;clang;clang-tools-extra;compiler-rt;cross-project-tests;libc;libclc;lld;lldb;mlir;openmp;polly;pstl")
-# The flang project is not yet part of "all" projects (see C++ requirements)
-set(LLVM_EXTRA_PROJECTS "flang")
+set(LLVM_ALL_PROJECTS "bolt;clang;clang-tools-extra;compiler-rt;cross-project-tests;flang;libc;libclc;lld;lldb;mlir;openmp;polly;pstl")
 # List of all known projects in the mono repo
-set(LLVM_KNOWN_PROJECTS "${LLVM_ALL_PROJECTS};${LLVM_EXTRA_PROJECTS}")
 set(LLVM_ENABLE_PROJECTS "" CACHE STRING
-    "Semicolon-separated list of projects to build (${LLVM_KNOWN_PROJECTS}), or \"all\".")
+    "Semicolon-separated list of projects to build (${LLVM_ALL_PROJECTS}), or \"all\".")
 # Make sure expansion happens first to not handle "all" in rest of the checks.
 if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
   set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})
 endif()
 foreach(proj ${LLVM_ENABLE_PROJECTS})
-  if (NOT proj STREQUAL "llvm" AND NOT "${proj}" IN_LIST LLVM_KNOWN_PROJECTS)
-     MESSAGE(FATAL_ERROR "${proj} isn't a known project: ${LLVM_KNOWN_PROJECTS}. Did you mean to enable it as a runtime in LLVM_ENABLE_RUNTIMES?")
+  if (NOT proj STREQUAL "llvm" AND NOT "${proj}" IN_LIST LLVM_ALL_PROJECTS)
+      MESSAGE(FATAL_ERROR "${proj} isn't a known project: ${LLVM_ALL_PROJECTS}. Did you mean to enable it as a runtime in LLVM_ENABLE_RUNTIMES?")
   endif()
 endforeach()
 
 if ("flang" IN_LIST LLVM_ENABLE_PROJECTS)
   if (NOT "mlir" IN_LIST LLVM_ENABLE_PROJECTS)
     message(STATUS "Enabling MLIR as a dependency to flang")
     list(APPEND LLVM_ENABLE_PROJECTS "mlir")
   endif()

 # by setting `LLVM_ENABLE_PROJECTS` to an empty string. In that case we still
 # need to set the `LLVM_TOOL_${upper_proj}_BUILD` variables so that we disable
 # building all the projects that were previously enabled.
 set(LLVM_ENABLE_PROJECTS_USED OFF CACHE BOOL "")
 mark_as_advanced(LLVM_ENABLE_PROJECTS_USED)
 
 if (LLVM_ENABLE_PROJECTS_USED OR NOT LLVM_ENABLE_PROJECTS STREQUAL "")
   set(LLVM_ENABLE_PROJECTS_USED ON CACHE BOOL "" FORCE)
-  foreach(proj ${LLVM_KNOWN_PROJECTS} ${LLVM_EXTERNAL_PROJECTS})
+  foreach(proj ${LLVM_ALL_PROJECTS} ${LLVM_EXTERNAL_PROJECTS})
     string(TOUPPER "${proj}" upper_proj)
     string(REGEX REPLACE "-" "_" upper_proj ${upper_proj})
     if ("${proj}" IN_LIST LLVM_ENABLE_PROJECTS)
       message(STATUS "${proj} project is enabled")
       set(SHOULD_ENABLE_PROJECT TRUE)
       set(PROJ_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../${proj}")
       if(NOT EXISTS "${PROJ_DIR}" OR NOT IS_DIRECTORY "${PROJ_DIR}")
         message(FATAL_ERROR "LLVM_ENABLE_PROJECTS requests ${proj} but directory not found: ${PROJ_DIR}")