[libcxx] [ci] Add a test configuration with an incomplete sysroot

[mstorsjo]

Summary

When bringing up a new cross compiler from scratch, we build
libunwind/libcxx in a setup where the toolchain is incomplete and
unable to perform the normal linker checks; this requires a few
special cases in the CMake files.

We simulate that scenario by making a copy of the existing sysroot
and remove libunwind/libcxx from it, and point to this with the
--sysroot option passed via CMAKE_*_FLAGS_INIT.

We need to set CMAKE_CXX_COMPILER_WORKS since CMake fails to
probe the compiler. We need to set CMAKE_CXX_COMPILER_TARGET,
since LLVM's heuristics fail when CMake hasn't been able to
probe the environment properly. (This is normal; one has to
set those options when setting up such a toolchain from scratch.)

This adds CI coverage for these build scenarios, which otherwise
seldom are tested by some build flow (but are essential when setting
up a cross compiler from scratch).

[phosek]

libcxx/utils/ci/run-buildbot

675–676

Can you use CMAKE_SYSROOT?

679

I'd set CMAKE_C_COMPILER_TARGET as well for consistency.

[mstorsjo]

Use CMAKE_SYSROOT instead of CMAKE_*_FLAGS_INIT, set CMAKE_C_COMPILER_TARGET for consistency.

[phosek]

LGTM

[mstorsjo]

Ping @ldionne - I believe this should add extra CI coverage for situations that we otherwise don't have tested, which certainly would be useful for @phosek 's pending refactorings.

[ldionne]

libcxx/utils/ci/run-buildbot

648

Is there a reason why we're running this on MinGW? It would make more sense IMO to do this in our Docker image on Linux. At the end of the day, we are trying to test a general property of our CMake build, i.e. that it can be run even if you don't have access to a fully working C++ toolchain. This isn't Windows-specific, right?

675

I would set this unconditionally in runtimes/CMakeLists.txt with an appropriate comment. I don't think it ever makes sense for us to rely on the default CMake compiler checking.

681–683

Additionally, the sysroot flag from CMAKE_SYSROOT doesn't end up propagated into the tests

This shouldn't matter, since we're using -nostdlib++ and -nostdinc++ to avoid depending on any existing libc++ in the toolchain.

[mstorsjo]

libcxx/utils/ci/run-buildbot

648

Indeed it isn't Windows specific per se.

But in most cases on e.g. Linux, you'd already have a working toolchain with libgcc provided by the system. For the case of something that mostly behaves as a cross compiler (i.e. there's no such thing as a system provided libgcc or similar), it's easier to produce a stripped down sysroot and reach such a state.

The same goes for the actual bringup of a toolchain, where you'd have such an incomplete stage; when building a non-cross toolchain, you're usually already operating on top of a working environment, so you usually don't pass through that incomplete state.

(There's of course small nuances that do differ between systems wrt how they behave in an incomplete state, e.g. ELF usually is more tolerant against undefined symbols than what PE/COFF is.)

If configuring a toolchain where the compiler uses libunwind by default, and libunwind doesn't exist yet, then you'd have the same situation on Linux too. The main thing that is relevant is that you'd have a clang setup for a target where just invoking clang without any flags tries to link libunwind by default.

Regarding Docker - if I understood correctly, the current setup is that we're running Docker containers, which then run the buildkite agent within them, serving multiple build jobs within the same persistent Docker container. If the setup would be the reverse - each buildkite job spawning a new individual Docker container, then this setup would be easier to more properly simulate an incomplete situation. Instead of setting up and stripping down a sysroot, the actual system toolchain could be stripped in a matching fashion.

In other words: I was easily able to set this up, and this gives test coverage of the bringup phase of my toolchain. If soomeone else wants to try to set up a similar test scenario within the Linux test configs, feel free to, but for me, that'd be significantly more effort to set up.

The case that this covers is literally something that every one of @phosek's refactorings to runtimes/CMakeFiles.txt hits and needs to test.

675

Hmm, I wonder if it's possible (and generally advisable?) to do that. WDYT @phosek?

681–683

It's not so much about libc++ as it is about libunwind and all that. If the toolchain default sysroot is fully functional with a working libunwind, and the intentionally stripped down sysroot lacks libunwind, then testing without the --sysroot parameter loses that distinction, even if -nostdlib++ avoids using the default C++ library.

But in any case, this configuration is mostly interesting for compiling the runtimes, once built it should mostly behave as any other build.

[phosek]

libcxx/utils/ci/run-buildbot

675

We already set these in the bootstrapping build, see https://github.com/llvm/llvm-project/blob/e53da3eab42e6efd448c1c60c14668e1eb3d907c/llvm/runtimes/CMakeLists.txt#L239, but I'm not sure if setting this in runtimes/CMakeFiles.txt is the right approach since these should be only needed when building runtimes with an incomplete toolchain, and I'm worried if this could be a potential footgun.

[mstorsjo]

So, FWIW, my view is that this patch would add valuable test coverage as-is.

It should be theoretically possible to set up something similar on linux as well, but it's not necessarily as straightforward (as it'd require setting up an environment that hard defaults to libunwind etc, to get the same test coverage as this gets). If it's a hard requirement for adding this kind of test scenario to have it running on Linux and not Windows, then I'd defer it until some time later when I have time to try to set it up. But as is, this patch gives lots of extra valuable test coverage.

libcxx/utils/ci/buildkite-pipeline.yml

- label: "MinGW (Static, x86_64)"
       agents:
         queue: "windows"
       retry:
         automatic:
           - exit_status: -1  # Agent was lost
             limit: 2
       timeout_in_minutes: 120
 
+    - label: "MinGW (Incomplete Sysroot)"
+      command: "bash libcxx/utils/ci/run-buildbot mingw-incomplete-sysroot"
+      artifact_paths:
+        - "**/test-results.xml"
+        - "**/*.abilist"
+      agents:
+        queue: "windows"
+      retry:
+        automatic:
+          - exit_status: -1  # Agent was lost
+            limit: 2
+      timeout_in_minutes: 120
+
     - label: "MinGW (DLL, i686)"
       command: "bash libcxx/utils/ci/run-buildbot mingw-dll-i686"
       artifact_paths:
         - "**/test-results.xml"
         - "**/*.abilist"
       agents:
         queue: "windows"
       retry:

libcxx/utils/ci/run-buildbot

 mingw-dll-i686)
     clean
     generate-cmake \
           -DCMAKE_C_COMPILER=i686-w64-mingw32-clang \
           -DCMAKE_CXX_COMPILER=i686-w64-mingw32-clang++ \
           -C "${MONOREPO_ROOT}/libcxx/cmake/caches/MinGW.cmake"
     check-runtimes
 ;;
+mingw-incomplete-sysroot)
+    clean
+    # When bringing up a new cross compiler from scratch, we build
+    # libunwind/libcxx in a setup where the toolchain is incomplete and
+    # unable to perform the normal linker checks; this requires a few
+    # special cases in the CMake files.
+    #
+    # The CI environment uses prebuilt complete toolchains, but make a
+    # copy of a sysroot into the build dir, and remove libunwind/libcxx
+    # to simulate the real case of bringing up a new toolchain.
+    # Pass this sysroot to CMake via CMAKE_*_FLAGS_INIT to make sure it
+    # is used throughout all the early CMake probing stages.
+    mkdir -p "${BUILD_DIR}/sysroot"
+    TOOLCHAIN="$(dirname $(command -v x86_64-w64-mingw32-clang))/.."
+    cp -a "${TOOLCHAIN}/x86_64-w64-mingw32/lib" "${BUILD_DIR}/sysroot"
+    cp -a "${TOOLCHAIN}/include" "${BUILD_DIR}/sysroot"
+    rm -rf "${BUILD_DIR}"/sysroot/lib/lib{c++,unwind}*
+    rm -rf "${BUILD_DIR}"/sysroot/include/c++
+    # Building in an incomplete setup requires setting CMAKE_*_COMPILER_WORKS,
+    # as CMake fails to probe the compiler. This case also requires
+    # setting CMAKE_CXX_COMPILER_TARGET, as LLVM's heuristics for setting
+    # the triple fails when CMake hasn't been able to probe the environment.
+    # (This is what one has to do when building the initial libunwind/libcxx
+    # for a new toolchain.)
+    generate-cmake \
+          -DCMAKE_C_COMPILER=x86_64-w64-mingw32-clang \
+          -DCMAKE_CXX_COMPILER=x86_64-w64-mingw32-clang++ \
+          -DCMAKE_C_FLAGS_INIT="--sysroot=${BUILD_DIR}/sysroot" \
+          -DCMAKE_CXX_FLAGS_INIT="--sysroot=${BUILD_DIR}/sysroot" \
+          -DCMAKE_C_COMPILER_WORKS=TRUE \
+          -DCMAKE_CXX_COMPILER_WORKS=TRUE \
+          -DCMAKE_CXX_COMPILER_TARGET=x86_64-w64-windows-gnu \
+          -C "${MONOREPO_ROOT}/libcxx/cmake/caches/MinGW.cmake"
+    # Only test that building succeeds; there's no extra value in running the
+    # tests here, as it would be equivalent to the mingw-dll config above.
+    # Additionally, the sysroot flags from CMAKE_*_FLAGS/CMAKE_*_FLAGS_INIT
+    # don't even propagate into the tests at the moment.
+    ${NINJA} -vC "${BUILD_DIR}"
+;;
 aix)
     clean
     generate-cmake -C "${MONOREPO_ROOT}/libcxx/cmake/caches/AIX.cmake" \
                    -DLIBCXX_TEST_CONFIG="ibm-libc++-shared.cfg.in" \
                    -DLIBCXXABI_TEST_CONFIG="ibm-libc++abi-shared.cfg.in" \
                    -DLIBUNWIND_TEST_CONFIG="ibm-libunwind-shared.cfg.in"
     check-abi-list
     check-runtimes