[RFC] Remove support for building libc++ with `LLVM_ENABLE_PROJECTS`

[Ericson2314]

Summary

This has been officially deprecated since D112724, meaning the
deprecation warning is present in released 14 and 15.

This makes me think that now, shortly after the 15 release is branched,
is a good time to pull the trigger.

[phosek]

LGTM

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

103

I'd consider renaming this to RUNTIMES.

[Ericson2314]

Do the rename @phosek suggested

[Ericson2314]

Ah I see email about sphinx jobs defined out of tree :/ I will take a look at that, see if it is easy to fix.

[Ericson2314]

In 952f90b72b3546d6b6b038d410f07ce520c59b48 I relented and made it a non-fatal error until the remaining jobs are figured out.

[MaskRay]

clang/lib/Driver/ToolChains/Linux.cpp

310

@mgorny FYI, this has been removed.

[MaskRay]

@vitalybuka -DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' in zorg/buildbot/builders/sanitizers/buildbot_functions.sh needs adjustment.

[vitalybuka]

In D132324#3738253, @MaskRay wrote:

@vitalybuka -DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' in zorg/buildbot/builders/sanitizers/buildbot_functions.sh needs adjustment.

If I do -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' now it builds parts of LLVM
problem is that with -DLLVM_USE_SANITIZER=Memory, it builds llvm instrumented as well, but we have no instrumented libc++, so we have false report

previously with DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' no LLVM tools, like tbl-gen where built

Fails like this: https://lab.llvm.org/buildbot/#/builders/74/builds/12976/steps/10/logs/stdio

cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF -DCMAKE_C_COMPILER=/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build0/bin/clang -DCMAKE_CXX_COMPILER=/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build0/bin/clang++ -DLLVM_USE_LINKER=lld '-DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi' -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory '-DCMAKE_C_FLAGS=-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-param-retval ' '-DCMAKE_CXX_FLAGS=-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-param-retval ' /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm

ninja cxx cxxabi

Worked with PROJECTS

[vitalybuka]

In D132324#3738430, @vitalybuka wrote:

In D132324#3738253, @MaskRay wrote:

@vitalybuka -DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' in zorg/buildbot/builders/sanitizers/buildbot_functions.sh needs adjustment.

If I do -DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi' now it builds parts of LLVM
problem is that with -DLLVM_USE_SANITIZER=Memory, it builds llvm instrumented as well, but we have no instrumented libc++, so we have false report

previously with DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi' no LLVM tools, like tbl-gen where built

Fails like this: https://lab.llvm.org/buildbot/#/builders/74/builds/12976/steps/10/logs/stdio

cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF -DCMAKE_C_COMPILER=/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build0/bin/clang -DCMAKE_CXX_COMPILER=/b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm_build0/bin/clang++ -DLLVM_USE_LINKER=lld '-DLLVM_ENABLE_RUNTIMES=libcxx;libcxxabi' -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory '-DCMAKE_C_FLAGS=-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-param-retval ' '-DCMAKE_CXX_FLAGS=-fsanitize=memory -fsanitize-memory-use-after-dtor -fsanitize-memory-param-retval ' /b/sanitizer-x86_64-linux-bootstrap-msan/build/llvm-project/llvm

ninja cxx cxxabi

Worked with PROJECTS

I guess now I need to use runtimes/ for that?

[aaron.ballman]

In D132324#3737976, @Ericson2314 wrote:

Ah I see email about sphinx jobs defined out of tree :/ I will take a look at that, see if it is easy to fix.

This still hasn't been fixed, so none of the documentation is being updated on the website: https://lab.llvm.org/buildbot/#/builders/89/builds/31614 -- if it's not easy, please revert until we have a path forward.

[Ericson2314]

I did a "soft revert" in rG952f90b72b3546d6b6b038d410f07ce520c59b48 which makes it a non-fatal error so everything keeps on working, but I can do a real revert too if that is preferred.

[Ericson2314]

https://reviews.llvm.org/rZORG3a209ca6c1b9 This is what I had started doing. but I am not sure it is a very good solution. I think I might write a discourse post with my thoughts.

[aaron.ballman]

In D132324#3739646, @Ericson2314 wrote:

I did a "soft revert" in rG952f90b72b3546d6b6b038d410f07ce520c59b48 which makes it a non-fatal error so everything keeps on working, but I can do a real revert too if that is preferred.

It doesn't seem to have helped the sphinx publish bot: https://lab.llvm.org/buildbot/#/builders/89/builds/31676

CMake Error at /home/buildbot/as-worker-4/publish-sphinx-docs/llvm-project/libcxxabi/CMakeLists.txt:138 (message):
  LIBCXXABI_LIBCXX_INCLUDES= is not a valid directory.  Please provide the
  path to where the libc++ headers have been installed.

[Ericson2314]

https://reviews.llvm.org/D132411 should fix the issue @aaron.ballman mentioned.

[ldionne]

Please don't change libc++ things like that without my approval first. This is a transition I've been working on for 2+ years, I had a local patch for it waiting to be published, and this patch is incomplete in several ways. I greatly appreciate the effort and wanting to chime into this transition, however concretely this was rushed in.

[aaron.ballman]

In D132324#3742394, @ldionne wrote:

Please don't change libc++ things like that without my approval first. This is a transition I've been working on for 2+ years, I had a local patch for it waiting to be published, and this patch is incomplete in several ways. I greatly appreciate the effort and wanting to chime into this transition, however concretely this was rushed in.

FWIW, I was also pretty surprised to see something with [RFC] in the title be proposed on a Saturday night and landed the next morning. While we had community agreement on the original deprecation, I think we usually would then follow up with an RFC (on Discourse) proposing to remove the deprecated functionality so people have more notice before the changes landed. (Just something to keep in mind for the future.)

For clarity, @ldionne are you requesting that these changes be reverted due to being incomplete, or do you prefer this be fixed forward?

[ldionne]

In D132324#3742406, @aaron.ballman wrote:

For clarity, @ldionne are you requesting that these changes be reverted due to being incomplete, or do you prefer this be fixed forward?

Sorry, I left no action item on my end after my first comment. I'm wrapping my head around this change and the other changes that followed D132298 and D132411 first, and I'll post here what I think we should do. I've also seen some internal fallout of this change and I suspect I'm not alone.

Like I said, I greatly appreciate that folks took the lead to push this removal forward, however it should have been under review for more than 24h on a week-end. Concretely, what I planned on doing was turn the message(WARNING into message(ERROR as a dye test without changing anything else, and then turn it back on to message(WARNING to give folks a bit of time to fix their up/downstream CI systems (to avoid them being red for too long).

I'll post here again when I've wrapped my head around this.

[ldionne]

This sucks, but I'm going to revert this series of patches. This is getting somewhat out of hands and I feel that we're rushing to fix all kinds of issues in various directions. I'm going to revert the following patches, in this order:

1. 952f90b72b35 [CMake] Weaken 176db3b3ab25ff8a9b2405f50ef5a8bd9304a6d5
2. e6a0800532bb [libcxxabi][cmake] Allow building without libcxx again
3. 176db3b3ab25 [RFC] Remove support for building C++ with LLVM_ENABLE_PROJECTS

Otherwise:

• I also think we don't want to move forward with D132411 -- it's going against this transition.
• https://reviews.llvm.org/rZORG3a209ca6c1b9 sounds like a good change we'd need to make no matter what, so that's great.
• D132454 seems like a good change regardless of this series of patches.

I think that's the whole set of changes/reviews associated to this in the past few days. If I've missed anything, please let me know ASAP.

This will have been extremely useful as a way to shake out places where the deprecated LLVM_ENABLE_PROJECTS build is still being used for libc++/libc++abi. However, we should take the time to fix these places first before landing the actual breaking change, otherwise things become too unstable and we're rushed into making potentially poor decisions.

Concretely, as an action item to move this transition forward, I will:

1. Grep for places where we still use LLVM_ENABLE_PROJECTS for libc++ and libc++abi in the LLVM monorepo
2. Do the same in the llvm-zorg repo
3. Do the same in our downstream bots (if other folks see this because of downstream breakage in their forks, be proactive and do this too, or you will get broken in the future!)
4. Re-land a patch that turns the WARNING into a FATAL_ERROR, and just that. Watch for more breakage and try fixing it iteratively.
5. Once we have FATAL_ERROR in place and things are green, we can land the rest of this patch (plus a few other things I had locally) to actually remove support for LLVM_ENABLE_PROJECTS with libc++/libc++abi.

[aaron.ballman]

In D132324#3742605, @ldionne wrote:

This sucks, but I'm going to revert this series of patches. This is getting somewhat out of hands and I feel that we're rushing to fix all kinds of issues in various directions. I'm going to revert the following patches, in this order:

1. 952f90b72b35 [CMake] Weaken 176db3b3ab25ff8a9b2405f50ef5a8bd9304a6d5
2. e6a0800532bb [libcxxabi][cmake] Allow building without libcxx again
3. 176db3b3ab25 [RFC] Remove support for building C++ with LLVM_ENABLE_PROJECTS

Otherwise:

• I also think we don't want to move forward with D132411 -- it's going against this transition.
• https://reviews.llvm.org/rZORG3a209ca6c1b9 sounds like a good change we'd need to make no matter what, so that's great.
• D132454 seems like a good change regardless of this series of patches.

I think that's the whole set of changes/reviews associated to this in the past few days. If I've missed anything, please let me know ASAP.

This will have been extremely useful as a way to shake out places where the deprecated LLVM_ENABLE_PROJECTS build is still being used for libc++/libc++abi. However, we should take the time to fix these places first before landing the actual breaking change, otherwise things become too unstable and we're rushed into making potentially poor decisions.

Concretely, as an action item to move this transition forward, I will:

1. Grep for places where we still use LLVM_ENABLE_PROJECTS for libc++ and libc++abi in the LLVM monorepo
2. Do the same in the llvm-zorg repo
3. Do the same in our downstream bots (if other folks see this because of downstream breakage in their forks, be proactive and do this too, or you will get broken in the future!)
4. Re-land a patch that turns the WARNING into a FATAL_ERROR, and just that. Watch for more breakage and try fixing it iteratively.
5. Once we have FATAL_ERROR in place and things are green, we can land the rest of this patch (plus a few other things I had locally) to actually remove support for LLVM_ENABLE_PROJECTS with libc++/libc++abi.

Thank you @ldionne -- I think this plan of action makes sense to me. The disruption from reverts is unfortunate, but I think it's better to take a clean run at this once we have everything lined up and ready in the other projects.

[Ericson2314]

That sounds good to me too.

Sorry I did unquestionably jump the gun here --- I suppose I was surprised to get a fairly simple approval when I put on the [RFC], and thought "hmm, maybe this isn't so controversial as I feared because indeed the deprecation cycle has been so thorough", but then again the simple approval was probably predicated on the [RFC] indicating I would wait long!

I also think we don't want to move forward with D132411 -- it's going against this transition.

I would like to discuss some of this sort of thing so more, but better to do that on the the discourse than my shoddy commits that need reverting!

[ldionne]

In D132324#3742700, @Ericson2314 wrote:

I also think we don't want to move forward with D132411 -- it's going against this transition.

I would like to discuss some of this sort of thing so more, but better to do that on the the discourse than my shoddy commits that need reverting!

Sure, I'd be curious to understand why that's needed.

After some thinking, I created the following stack of patches: D132478 => D132479 => D132480.

The last patch is basically a reboot of this patch, i.e. it's the one that will make folks who use LLVM_ENABLE_PROJECTS=libcxx start failing. However, I am also trying to keep LLVM_ENABLE_PROJECTS and LLVM_ENABLE_RUNTIMES consistent, i.e. it should be possible to use LLVM_ENABLE_PROJECTS=all and LLVM_ENABLE_RUNTIMES=all without ever being broken by any of our changes (except libcxx, libcxxabi and libunwind would suddenly start building with the bootstrapping build instead of the normal runtimes build).

clang/lib/Driver/ToolChains/Linux.cpp

#endif
   if (IsRISCV) {
     StringRef ABIName = tools::riscv::getRISCVABI(Args, Triple);
     addPathIfExists(D, concat(SysRoot, "/", OSLibDir, ABIName), Paths);
     addPathIfExists(D, concat(SysRoot, "/usr", OSLibDir, ABIName), Paths);
   }
 
   Generic_GCC::AddMultiarchPaths(D, SysRoot, OSLibDir, Paths);
 
-  // The deprecated -DLLVM_ENABLE_PROJECTS=libcxx configuration installs
-  // libc++.so in D.Dir+"/../lib/". Detect this path.
-  // TODO Remove once LLVM_ENABLE_PROJECTS=libcxx is unsupported.
-  if (StringRef(D.Dir).startswith(SysRoot) &&
-      D.getVFS().exists(D.Dir + "/../lib/libc++.so"))
-    addPathIfExists(D, D.Dir + "/../lib", Paths);
-
   addPathIfExists(D, concat(SysRoot, "/lib"), Paths);
   addPathIfExists(D, concat(SysRoot, "/usr/lib"), Paths);
 }
 
 ToolChain::RuntimeLibType Linux::GetDefaultRuntimeLibType() const {
   if (getTriple().isAndroid())
     return ToolChain::RLT_CompilerRT;
   return Generic_ELF::GetDefaultRuntimeLibType();

compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh

 CC=$CC CFLAGS="$FLAGS" RANLIB=/bin/true ./configure --static
 make -j${J} libz.a
 
 # Build and install libcxxabi and libcxx.
 if [[ ! -d ${LIBCXX_BUILD} ]]; then
   mkdir -p ${LIBCXX_BUILD}
   cd ${LIBCXX_BUILD}
   LIBCXX_FLAGS="${FLAGS} -Wno-macro-redefined"
-  PROJECTS=
+  RUNTIMES=
   if [[ ! -d $LLVM_SRC/projects/libcxxabi ]] ; then
-    PROJECTS="-DLLVM_ENABLE_PROJECTS='libcxx;libcxxabi'"
+    RUNTIMES="-DLLVM_ENABLE_RUNTIMES='libcxx;libcxxabi'"
   fi
   cmake -GNinja \
-    ${PROJECTS} \
+    ${RUNTIMES} \
     -DCMAKE_BUILD_TYPE=Release \
     -DCMAKE_C_COMPILER=$CC \
     -DCMAKE_CXX_COMPILER=$CXX \
     -DCMAKE_C_FLAGS_RELEASE="${LIBCXX_FLAGS}" \
     -DCMAKE_CXX_FLAGS_RELEASE="${LIBCXX_FLAGS}" \
     -DLIBCXXABI_ENABLE_ASSERTIONS=OFF \
     -DLIBCXXABI_ENABLE_EXCEPTIONS=OFF \
     -DLIBCXXABI_ENABLE_SHARED=OFF \

libcxx/utils/ci/buildkite-pipeline.yml

- label: "Legacy Lit configuration"
         queue: "libcxx-builders"
         os: "linux"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
-    - label: "Legacy LLVM_ENABLE_PROJECTS build"
-      command: "libcxx/utils/ci/run-buildbot legacy-project-build"
-      artifact_paths:
-        - "**/test-results.xml"
-        - "**/*.abilist"
-      agents:
-        queue: "libcxx-builders"
-        os: "linux"
-      retry:
-        automatic:
-          - exit_status: -1  # Agent was lost
-            limit: 2
-      timeout_in_minutes: 120
-
   # Tests with various build configurations.
   - label: "Static libraries"
     command: "libcxx/utils/ci/run-buildbot generic-static"
     artifact_paths:
       - "**/test-results.xml"
       - "**/*.abilist"
     agents:
       queue: "libcxx-builders"

libcxx/utils/ci/oss-fuzz.sh

 #!/bin/bash -eu
 
 #
 # This script runs the continuous fuzzing tests on OSS-Fuzz.
 #
 
 if [[ ${SANITIZER} = *undefined* ]]; then
   CXXFLAGS="${CXXFLAGS} -fsanitize=unsigned-integer-overflow -fsanitize-trap=unsigned-integer-overflow"
 fi
 
 BUILD=cxx_build_dir
 INSTALL=cxx_install_dir
 
 mkdir ${BUILD}
 cmake -S ${PWD} -B ${BUILD} \
-      -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi" \
+      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \
       -DCMAKE_BUILD_TYPE=RelWithDebInfo \
       -DCMAKE_INSTALL_PREFIX="${INSTALL}"
 cmake --build ${BUILD} --target install-cxx-headers
 
 for test in libcxx/test/libcxx/fuzzing/*.pass.cpp; do
     exe="$(basename ${test})"
     exe="${exe%.pass.cpp}"
     ${CXX} ${CXXFLAGS} \

libcxx/utils/ci/run-buildbot

 ;;
 legacy-test-config)
     clean
     generate-cmake -DLIBCXX_TEST_CONFIG="legacy.cfg.in" \
                    -DLIBCXXABI_TEST_CONFIG="${MONOREPO_ROOT}/libcxxabi/test/lit.site.cfg.in" \
                    -DLIBUNWIND_TEST_CONFIG="${MONOREPO_ROOT}/libunwind/test/lit.site.cfg.in"
     check-runtimes
 ;;
-legacy-project-build)
-    clean
-
-    echo "--- Generating CMake"
-    ${CMAKE} \
-          -S "${MONOREPO_ROOT}/llvm" \
-          -B "${BUILD_DIR}" \
-          -DLLVM_ENABLE_PROJECTS="libcxx;libunwind;libcxxabi" \
-          -GNinja -DCMAKE_MAKE_PROGRAM="${NINJA}" \
-          -DCMAKE_BUILD_TYPE=RelWithDebInfo \
-          -DCMAKE_INSTALL_PREFIX="${INSTALL_DIR}" \
-          -DLLVM_LIT_ARGS="-sv --show-unsupported --xunit-xml-output test-results.xml --timeout=1500" \
-          -DLIBCXX_CXX_ABI=libcxxabi
-    check-runtimes
-;;
 aarch64)
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/AArch64.cmake"
     check-runtimes
 ;;
 aarch64-noexceptions)
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/AArch64.cmake" \

llvm/CMakeLists.txt

set(LLVM_ENABLE_PROJECTS "" CACHE STRING
     "Semicolon-separated list of projects to build (${LLVM_KNOWN_PROJECTS}), or \"all\".")
 foreach(proj ${LLVM_ENABLE_PROJECTS})
   if (NOT proj STREQUAL "all" AND NOT proj STREQUAL "llvm" AND NOT "${proj}" IN_LIST LLVM_KNOWN_PROJECTS)
      MESSAGE(FATAL_ERROR "${proj} isn't a known project: ${LLVM_KNOWN_PROJECTS}")
   endif()
 endforeach()
 foreach(proj "libcxx" "libcxxabi" "libunwind")
   if (${proj} IN_LIST LLVM_ENABLE_PROJECTS)
-    message(WARNING "Using LLVM_ENABLE_PROJECTS=${proj} is deprecated now, please use -DLLVM_ENABLE_RUNTIMES=${proj} or "
-                    "see the instructions at https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.")
+    message(FATAL_ERROR
+      "Using LLVM_ENABLE_PROJECTS=${proj} is incorrect. Please use -DLLVM_ENABLE_RUNTIMES=${proj} or "
   endif()
 endforeach()
 
 if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
   set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})
 endif()
 
 if ("flang" IN_LIST LLVM_ENABLE_PROJECTS)