[Flang] add flang as a new subproject in cmake

[schweitz]

Summary

This patch is some minor prep work for merging the flang(f18) project into the monorepo. This patch adds "flang" as a supported target for the LLVM_ENABLE_PROJECTS option.

[jdoerfert]

It helps if you use tags in the title of commits (and reviews), e.g., [Flang] .... Maybe we can make the commit message a bit more specific: "[Flang] Add Flang as subproject in cmake"

[schweitz]

In D72416#1811239, @jdoerfert wrote:

It helps if you use tags in the title of commits (and reviews), e.g., [Flang] .... Maybe we can make the commit message a bit more specific: "[Flang] Add Flang as subproject in cmake"

Sure. Done.

[fhahn]

IIUC this is only needed after the flang merge, right? It will break things before the merge. The change is fine, after flang gets merged. IIUC this is not required as preparation for the merge, rather a follow up. Unless I am missing something, it would be good to clarify the title/commit message.

[schweitz]

Hmm. Yes, it seems this isn't exactly the "prep" patch I wanted w.r.t llvm/CMakeLists.txt. We made the "all" change locally, but it does not appear to be required. Removed it.

We're not going to want f18 to be picked up by the buildbots, since it requires everything get built with C++ 17.

[schweitz]

Ideally, this ought to go in around the same time that the flang sources get merged into the monorepo. It's not necessary until that time.

[DavidTruby]

My current plan is to add this as the first commit on top of the flang merge and push that all together. That way although there will be commits that don't build in the history (which is unavoidable) there won't actually be a time at which this causes build issues. Does that sound reasonable?

[schweitz]

That sounds like a good plan to me, David. Thank you.

[DavidTruby]

This doesn't seem to be sufficient to build Flang for me. My process was as follows:

git clone https://github.com/llvm/llvm-project
cd llvm-project
arc patch D72416
git clone https://github.com/pmccormick/f18 flang # The f18 branch with LLVM cmake infrastructure changes
mkdir build && cd build
cmake -GNinja -DLLVM_ENABLE_PROJECTS="mlir;flang" ../llvm
ninja

after which I get a correct build of llvm with no errors in the cmake step, but the flang binaries are not present.
My cursory reading of the cmake files suggests to me that we _do_ need flang in LLVM_ALL_PROJECTS, as it seems that LLVM_ENABLE_PROJECTS only affects things that are in LLVM_ALL_PROJECTS. If I add flang to LLVM_ALL_PROJECTS and run the above commands I get a correct in-tree build of flang.

The other option is to special case flang in the handling for LLVM_ENABLE_PRIOJECTS, but I think this is a less nice solution.

[schweitz]

Hmm. Interesting. Thanks for testing it again. I can add it back to ALL. No problem. (Locally, I already have, which may explain why I don't see any problem.)

[DavidTruby]

@hfinkel suggested we make an LLVM_EXTRA_PROJECTS and add that to the logic to search for projects. I have a patch to do that, I'll just link it here when I work out how to!

[schweitz]

In D72416#1912641, @DavidTruby wrote:

@hfinkel suggested we make an LLVM_EXTRA_PROJECTS and add that to the logic to search for projects. I have a patch to do that, I'll just link it here when I work out how to!

I agree with Hal. It makes sense to separate the "what's a valid project" logic from the "let's build everything" option.

[DavidTruby]

https://reviews.llvm.org/differential/diff/249126/ this works for me
Although I just spotted I've left an OR ${proj} in there which shouldn't be there. If you ignore that then it works!

[DavidTruby]

LGTM

llvm/CMakeLists.txt

if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
   set(CMAKE_BUILD_TYPE "Debug" CACHE STRING "Build type (default Debug)" FORCE)
 endif()
 
 # Side-by-side subprojects layout: automatically set the
 # LLVM_EXTERNAL_${project}_SOURCE_DIR using LLVM_ALL_PROJECTS
 # This allows an easy way of setting up a build directory for llvm and another
 # one for llvm+clang+... using the same sources.
 set(LLVM_ALL_PROJECTS "clang;clang-tools-extra;compiler-rt;debuginfo-tests;libc;libclc;libcxx;libcxxabi;libunwind;lld;lldb;mlir;openmp;parallel-libs;polly;pstl")
+# The flang project is not yet part of "all" projects (see C++ requirements)
+set(LLVM_EXTRA_PROJECTS "flang")
+# List of all known projects in the mono repo
+set(LLVM_KNOWN_PROJECTS "${LLVM_ALL_PROJECTS};${LLVM_EXTRA_PROJECTS}")
 set(LLVM_ENABLE_PROJECTS "" CACHE STRING
-  "Semicolon-separated list of projects to build (${LLVM_ALL_PROJECTS}), or \"all\".")
+  "Semicolon-separated list of projects to build (${LLVM_KNOWN_PROJECTS}), or \"all\".")
 if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
   set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})
 endif()
 
 # LLVM_ENABLE_PROJECTS_USED is `ON` if the user has ever used the
 # `LLVM_ENABLE_PROJECTS` CMake cache variable.  This exists for
 # several reasons:
 #
 # * As an indicator that the `LLVM_ENABLE_PROJECTS` list is now the single
 # source of truth for which projects to build. This means we will ignore user
 # supplied `LLVM_TOOL_<project>_BUILD` CMake cache variables and overwrite
 # them.
 #
 # * The case where the user previously had `LLVM_ENABLE_PROJECTS` set to a
 # non-empty list but now the user wishes to disable building all other projects
 # by setting `LLVM_ENABLE_PROJECTS` to an empty string. In that case we still
 # need to set the `LLVM_TOOL_${upper_proj}_BUILD` variables so that we disable
 # building all the projects that were previously enabled.
 set(LLVM_ENABLE_PROJECTS_USED OFF CACHE BOOL "")
 mark_as_advanced(LLVM_ENABLE_PROJECTS_USED)
 
 if (LLVM_ENABLE_PROJECTS_USED OR NOT LLVM_ENABLE_PROJECTS STREQUAL "")
   set(LLVM_ENABLE_PROJECTS_USED ON CACHE BOOL "" FORCE)
-  foreach(proj ${LLVM_ALL_PROJECTS} ${LLVM_EXTERNAL_PROJECTS})
+  foreach(proj ${LLVM_KNOWN_PROJECTS} ${LLVM_EXTERNAL_PROJECTS})
     string(TOUPPER "${proj}" upper_proj)
     string(REGEX REPLACE "-" "_" upper_proj ${upper_proj})
     if ("${proj}" IN_LIST LLVM_ENABLE_PROJECTS)
       message(STATUS "${proj} project is enabled")
       set(SHOULD_ENABLE_PROJECT TRUE)
       set(PROJ_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../${proj}")
       if(NOT EXISTS "${PROJ_DIR}" OR NOT IS_DIRECTORY "${PROJ_DIR}")
         message(FATAL_ERROR "LLVM_ENABLE_PROJECTS requests ${proj} but directory not found: ${PROJ_DIR}")

llvm/tools/CMakeLists.txt

 add_llvm_tool_subdirectory(llvm-profdata)
 
 # Projects supported via LLVM_EXTERNAL_*_SOURCE_DIR need to be explicitly
 # specified.
 add_llvm_external_project(clang)
 add_llvm_external_project(lld)
 add_llvm_external_project(lldb)
 add_llvm_external_project(mlir)
+# Flang depends on mlir, so place it afterward
+add_llvm_external_project(flang)
 
 # Automatically add remaining sub-directories containing a 'CMakeLists.txt'
 # file as external projects.
 add_llvm_implicit_projects()
 
 add_llvm_external_project(polly)
 
 # Add subprojects specified using LLVM_EXTERNAL_PROJECTS
 foreach(p ${LLVM_EXTERNAL_PROJECTS})
   add_llvm_external_project(${p})
 endforeach(p)
 
 set(LLVM_COMMON_DEPENDS ${LLVM_COMMON_DEPENDS} PARENT_SCOPE)